2024-04-22T23:06:10.504 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-22T23:06:10.504 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --name adking-2024-04-22_22:45:33-orch:cephadm-wip-adk-testing-2024-04-22-1618-distro-default-smithi --archive /home/teuthworker/archive/adking-2024-04-22_22:45:33-orch:cephadm-wip-adk-testing-2024-04-22-1618-distro-default-smithi/7669190 --owner scheduled_adking@teuthology -- /home/teuthworker/archive/adking-2024-04-22_22:45:33-orch:cephadm-wip-adk-testing-2024-04-22-1618-distro-default-smithi/7669190/orig.config.yaml 2024-04-22T23:06:10.559 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T23:06:10.666 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/adking-2024-04-22_22:45:33-orch:cephadm-wip-adk-testing-2024-04-22-1618-distro-default-smithi/7669190 branch: wip-adk-testing-2024-04-22-1618 description: orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: adking@redhat.com first_in_suite: false job_id: '7669190' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/v18.2.1 ' name: adking-2024-04-22_22:45:33-orch:cephadm-wip-adk-testing-2024-04-22-1618-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-adk-testing-2024-04-22-1618 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: 43be020184947e53516056c9931e1ac5bdbbb1a5 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 43be020184947e53516056c9931e1ac5bdbbb1a5 kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-adk-testing-2024-04-22-1618 sha1: 43be020184947e53516056c9931e1ac5bdbbb1a5 owner: scheduled_adking@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 6551 sha1: 43be020184947e53516056c9931e1ac5bdbbb1a5 sleep_before_teardown: 0 subset: 1/16 suite: orch:cephadm suite_branch: wip-adk-testing-2024-04-22-1618 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_43be020184947e53516056c9931e1ac5bdbbb1a5/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 43be020184947e53516056c9931e1ac5bdbbb1a5 targets: smithi067.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ5hEEgSAMmspQN5NnJQsa+hsJAfHYTu3I5IW2nL2wQlYZzcUSeO0DE0hSGXpgY+MuX3h1ZRYwV4X9rxoiwDYN4= smithi134.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCEGBOx3MuAKppZlNWhmQElTN2uNaQUyAfgTd0/Cox0zmELbuVIEtalkaxr05afBn9GowE0EaPYgpdlHG/jSdAc= tasks: - install: exclude_packages: - ceph-volume tag: v18.2.1 - print: '**** done install task...' - cephadm: compiled_cephadm_branch: reef conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.io/ceph/ceph:v18.2.1 roleless: true - print: '**** done end installing v18.2.1 cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - ceph-fuse: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-22_22:45:33 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: adking verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-04-22T23:06:10.666 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_43be020184947e53516056c9931e1ac5bdbbb1a5/qa; will attempt to use it 2024-04-22T23:06:10.667 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_43be020184947e53516056c9931e1ac5bdbbb1a5/qa/tasks 2024-04-22T23:06:10.667 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-22T23:06:10.669 INFO:teuthology.task.internal:Checking packages... 2024-04-22T23:06:10.692 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '43be020184947e53516056c9931e1ac5bdbbb1a5' 2024-04-22T23:06:10.692 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-22T23:06:10.692 INFO:teuthology.packaging:ref: None 2024-04-22T23:06:10.692 INFO:teuthology.packaging:tag: None 2024-04-22T23:06:10.692 INFO:teuthology.packaging:branch: wip-adk-testing-2024-04-22-1618 2024-04-22T23:06:10.692 INFO:teuthology.packaging:sha1: 43be020184947e53516056c9931e1ac5bdbbb1a5 2024-04-22T23:06:10.693 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-adk-testing-2024-04-22-1618 2024-04-22T23:06:10.848 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3252.g43be0201 2024-04-22T23:06:10.850 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-22T23:06:10.858 INFO:teuthology.task.internal:no buildpackages task found 2024-04-22T23:06:10.858 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-22T23:06:10.892 INFO:teuthology.task.internal:Saving configuration 2024-04-22T23:06:10.934 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-22T23:06:10.942 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T23:06:10.967 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-04-22_22:45:33-orch:cephadm-wip-adk-testing-2024-04-22-1618-distro-default-smithi/7669190', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 23:01:16.930821', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ5hEEgSAMmspQN5NnJQsa+hsJAfHYTu3I5IW2nL2wQlYZzcUSeO0DE0hSGXpgY+MuX3h1ZRYwV4X9rxoiwDYN4='} 2024-04-22T23:06:10.993 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi134.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-04-22_22:45:33-orch:cephadm-wip-adk-testing-2024-04-22-1618-distro-default-smithi/7669190', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 23:01:16.932839', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCEGBOx3MuAKppZlNWhmQElTN2uNaQUyAfgTd0/Cox0zmELbuVIEtalkaxr05afBn9GowE0EaPYgpdlHG/jSdAc='} 2024-04-22T23:06:10.993 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-22T23:06:11.001 INFO:teuthology.task.internal:roles: ubuntu@smithi067.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-04-22T23:06:11.001 INFO:teuthology.task.internal:roles: ubuntu@smithi134.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-04-22T23:06:11.001 INFO:teuthology.run_tasks:Running task console_log... 2024-04-22T23:06:11.075 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa0e5a1d160>, signals=[15]) 2024-04-22T23:06:11.076 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-22T23:06:11.084 INFO:teuthology.task.internal:Opening connections... 2024-04-22T23:06:11.084 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi067.front.sepia.ceph.com 2024-04-22T23:06:11.086 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T23:06:11.164 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi134.front.sepia.ceph.com 2024-04-22T23:06:11.165 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T23:06:11.240 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-22T23:06:11.248 DEBUG:teuthology.orchestra.run.smithi067:> uname -m 2024-04-22T23:06:11.270 INFO:teuthology.orchestra.run.smithi067.stdout:x86_64 2024-04-22T23:06:11.270 DEBUG:teuthology.orchestra.run.smithi067:> cat /etc/os-release 2024-04-22T23:06:11.328 INFO:teuthology.orchestra.run.smithi067.stdout:NAME="CentOS Stream" 2024-04-22T23:06:11.328 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION="9" 2024-04-22T23:06:11.328 INFO:teuthology.orchestra.run.smithi067.stdout:ID="centos" 2024-04-22T23:06:11.328 INFO:teuthology.orchestra.run.smithi067.stdout:ID_LIKE="rhel fedora" 2024-04-22T23:06:11.328 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_ID="9" 2024-04-22T23:06:11.328 INFO:teuthology.orchestra.run.smithi067.stdout:PLATFORM_ID="platform:el9" 2024-04-22T23:06:11.328 INFO:teuthology.orchestra.run.smithi067.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T23:06:11.328 INFO:teuthology.orchestra.run.smithi067.stdout:ANSI_COLOR="0;31" 2024-04-22T23:06:11.328 INFO:teuthology.orchestra.run.smithi067.stdout:LOGO="fedora-logo-icon" 2024-04-22T23:06:11.328 INFO:teuthology.orchestra.run.smithi067.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T23:06:11.328 INFO:teuthology.orchestra.run.smithi067.stdout:HOME_URL="https://centos.org/" 2024-04-22T23:06:11.328 INFO:teuthology.orchestra.run.smithi067.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T23:06:11.328 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T23:06:11.328 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T23:06:11.329 INFO:teuthology.lock.ops:Updating smithi067.front.sepia.ceph.com on lock server 2024-04-22T23:06:11.356 DEBUG:teuthology.orchestra.run.smithi134:> uname -m 2024-04-22T23:06:11.374 INFO:teuthology.orchestra.run.smithi134.stdout:x86_64 2024-04-22T23:06:11.375 DEBUG:teuthology.orchestra.run.smithi134:> cat /etc/os-release 2024-04-22T23:06:11.432 INFO:teuthology.orchestra.run.smithi134.stdout:NAME="CentOS Stream" 2024-04-22T23:06:11.432 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION="9" 2024-04-22T23:06:11.432 INFO:teuthology.orchestra.run.smithi134.stdout:ID="centos" 2024-04-22T23:06:11.432 INFO:teuthology.orchestra.run.smithi134.stdout:ID_LIKE="rhel fedora" 2024-04-22T23:06:11.432 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION_ID="9" 2024-04-22T23:06:11.432 INFO:teuthology.orchestra.run.smithi134.stdout:PLATFORM_ID="platform:el9" 2024-04-22T23:06:11.432 INFO:teuthology.orchestra.run.smithi134.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T23:06:11.432 INFO:teuthology.orchestra.run.smithi134.stdout:ANSI_COLOR="0;31" 2024-04-22T23:06:11.432 INFO:teuthology.orchestra.run.smithi134.stdout:LOGO="fedora-logo-icon" 2024-04-22T23:06:11.432 INFO:teuthology.orchestra.run.smithi134.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T23:06:11.433 INFO:teuthology.orchestra.run.smithi134.stdout:HOME_URL="https://centos.org/" 2024-04-22T23:06:11.433 INFO:teuthology.orchestra.run.smithi134.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T23:06:11.433 INFO:teuthology.orchestra.run.smithi134.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T23:06:11.433 INFO:teuthology.orchestra.run.smithi134.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T23:06:11.433 INFO:teuthology.lock.ops:Updating smithi134.front.sepia.ceph.com on lock server 2024-04-22T23:06:11.456 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-22T23:06:11.465 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-22T23:06:11.472 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-22T23:06:11.472 DEBUG:teuthology.orchestra.run.smithi067:> test '!' -e /home/ubuntu/cephtest 2024-04-22T23:06:11.474 DEBUG:teuthology.orchestra.run.smithi134:> test '!' -e /home/ubuntu/cephtest 2024-04-22T23:06:11.490 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-22T23:06:11.497 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-22T23:06:11.497 DEBUG:teuthology.orchestra.run.smithi067:> test -z $(ls -A /var/lib/ceph) 2024-04-22T23:06:11.532 DEBUG:teuthology.orchestra.run.smithi134:> test -z $(ls -A /var/lib/ceph) 2024-04-22T23:06:11.574 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-22T23:06:11.627 INFO:teuthology.run_tasks:Running task kernel... 2024-04-22T23:06:11.642 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-22T23:06:11.642 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-22T23:06:11.643 DEBUG:teuthology.orchestra.run.smithi067:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T23:06:11.643 DEBUG:teuthology.orchestra.run.smithi134:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T23:06:11.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T23:06:11.659 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2024-04-22T23:06:11.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T23:06:11.661 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-04-22T23:06:11.716 INFO:teuthology.orchestra.run.smithi134.stdout:5.14.0-437.el9.x86_64 2024-04-22T23:06:11.717 INFO:teuthology.task.kernel:Running kernel on smithi134: 5.14.0-437.el9.x86_64 2024-04-22T23:06:11.717 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum install -y kernel 2024-04-22T23:06:11.719 INFO:teuthology.orchestra.run.smithi067.stdout:5.14.0-437.el9.x86_64 2024-04-22T23:06:11.719 INFO:teuthology.task.kernel:Running kernel on smithi067: 5.14.0-437.el9.x86_64 2024-04-22T23:06:11.719 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum install -y kernel 2024-04-22T23:06:14.165 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - BaseOS 9.9 MB/s | 8.0 MB 00:00 2024-04-22T23:06:14.466 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - BaseOS 7.7 MB/s | 8.0 MB 00:01 2024-04-22T23:06:15.916 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - AppStream 29 MB/s | 19 MB 00:00 2024-04-22T23:06:16.275 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-04-22T23:06:20.242 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - CRB 9.9 MB/s | 6.2 MB 00:00 2024-04-22T23:06:20.243 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.2 MB 00:00 2024-04-22T23:06:21.918 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 16 kB 00:00 2024-04-22T23:06:22.093 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-04-22T23:06:24.703 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 7.9 MB/s | 21 MB 00:02 2024-04-22T23:06:29.903 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-04-22T23:06:30.090 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:01 ago on Mon 22 Apr 2024 11:06:29 PM UTC. 2024-04-22T23:06:31.834 INFO:teuthology.orchestra.run.smithi067.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-22T23:06:31.834 INFO:teuthology.orchestra.run.smithi067.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-22T23:06:31.928 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-22T23:06:31.929 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-04-22T23:06:31.929 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-22T23:06:32.054 DEBUG:teuthology.orchestra.run.smithi067:> echo no | sudo yum reinstall kernel || true 2024-04-22T23:06:32.575 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 11:06:29 PM UTC. 2024-04-22T23:06:32.708 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-22T23:06:32.723 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:06:32.723 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-04-22T23:06:32.723 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:06:32.723 INFO:teuthology.orchestra.run.smithi067.stdout:Reinstalling: 2024-04-22T23:06:32.724 INFO:teuthology.orchestra.run.smithi067.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T23:06:32.724 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:06:32.724 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-22T23:06:32.724 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:06:32.724 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:06:32.724 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 255 k 2024-04-22T23:06:32.724 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 0 2024-04-22T23:06:32.724 INFO:teuthology.orchestra.run.smithi067.stderr:Operation aborted. 2024-04-22T23:06:32.782 INFO:teuthology.orchestra.run.smithi067.stdout:Is this ok [y/N]: 2024-04-22T23:06:32.782 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum reinstall -y kernel || true 2024-04-22T23:06:33.299 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:04 ago on Mon 22 Apr 2024 11:06:29 PM UTC. 2024-04-22T23:06:33.419 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-22T23:06:33.420 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:06:33.420 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-04-22T23:06:33.420 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:06:33.420 INFO:teuthology.orchestra.run.smithi067.stdout:Reinstalling: 2024-04-22T23:06:33.420 INFO:teuthology.orchestra.run.smithi067.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T23:06:33.420 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:06:33.420 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-22T23:06:33.420 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:06:33.420 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:06:33.420 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 255 k 2024-04-22T23:06:33.420 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 0 2024-04-22T23:06:33.420 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-22T23:06:34.022 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-5.14.0-437.el9.x86_64.rpm 787 kB/s | 255 kB 00:00 2024-04-22T23:06:34.022 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-22T23:06:34.022 INFO:teuthology.orchestra.run.smithi067.stdout:Total 424 kB/s | 255 kB 00:00 2024-04-22T23:06:34.038 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-22T23:06:34.226 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-22T23:06:34.227 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-22T23:06:34.232 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-22T23:06:34.232 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-22T23:06:34.364 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-22T23:06:34.447 INFO:teuthology.orchestra.run.smithi067.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T23:06:34.447 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T23:06:34.809 INFO:teuthology.orchestra.run.smithi134.stdout:Extra Packages for Enterprise Linux 1.7 MB/s | 21 MB 00:12 2024-04-22T23:06:35.147 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T23:06:35.147 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T23:06:35.817 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T23:06:35.817 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:06:35.817 INFO:teuthology.orchestra.run.smithi067.stdout:Reinstalled: 2024-04-22T23:06:35.817 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-22T23:06:35.817 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:06:35.817 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-22T23:06:35.984 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q kernel | sort -rV | head -n 1 2024-04-22T23:06:36.033 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-22T23:06:36.033 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-22T23:06:36.033 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-22T23:06:36.033 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-04-22T23:06:36.033 DEBUG:teuthology.orchestra.run.smithi067:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-22T23:06:36.060 INFO:teuthology.orchestra.run.smithi067.stdout:ttyS1 2024-04-22T23:06:36.079 DEBUG:teuthology.parallel:result is None 2024-04-22T23:06:39.926 INFO:teuthology.orchestra.run.smithi134.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-04-22T23:06:41.807 INFO:teuthology.orchestra.run.smithi134.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-22T23:06:41.807 INFO:teuthology.orchestra.run.smithi134.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-22T23:06:41.888 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-22T23:06:41.889 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-04-22T23:06:41.889 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-22T23:06:42.016 DEBUG:teuthology.orchestra.run.smithi134:> echo no | sudo yum reinstall kernel || true 2024-04-22T23:06:42.513 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 11:06:39 PM UTC. 2024-04-22T23:06:42.644 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-22T23:06:42.649 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:06:42.649 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-04-22T23:06:42.649 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:06:42.649 INFO:teuthology.orchestra.run.smithi134.stdout:Reinstalling: 2024-04-22T23:06:42.649 INFO:teuthology.orchestra.run.smithi134.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T23:06:42.649 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:06:42.649 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-04-22T23:06:42.649 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:06:42.649 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:06:42.650 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 255 k 2024-04-22T23:06:42.650 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 0 2024-04-22T23:06:42.650 INFO:teuthology.orchestra.run.smithi134.stderr:Operation aborted. 2024-04-22T23:06:42.723 INFO:teuthology.orchestra.run.smithi134.stdout:Is this ok [y/N]: 2024-04-22T23:06:42.724 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum reinstall -y kernel || true 2024-04-22T23:06:43.217 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:04 ago on Mon 22 Apr 2024 11:06:39 PM UTC. 2024-04-22T23:06:43.336 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-22T23:06:43.337 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:06:43.337 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-04-22T23:06:43.337 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:06:43.337 INFO:teuthology.orchestra.run.smithi134.stdout:Reinstalling: 2024-04-22T23:06:43.338 INFO:teuthology.orchestra.run.smithi134.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T23:06:43.338 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:06:43.338 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-04-22T23:06:43.338 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:06:43.338 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:06:43.338 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 255 k 2024-04-22T23:06:43.338 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 0 2024-04-22T23:06:43.338 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-04-22T23:06:43.836 INFO:teuthology.orchestra.run.smithi134.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.1 MB/s | 255 kB 00:00 2024-04-22T23:06:43.836 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-04-22T23:06:43.836 INFO:teuthology.orchestra.run.smithi134.stdout:Total 511 kB/s | 255 kB 00:00 2024-04-22T23:06:43.855 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-04-22T23:06:44.039 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-04-22T23:06:44.039 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-04-22T23:06:44.044 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-04-22T23:06:44.045 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-04-22T23:06:44.176 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-04-22T23:06:44.262 INFO:teuthology.orchestra.run.smithi134.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T23:06:44.262 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T23:06:45.019 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T23:06:45.019 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T23:06:45.690 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T23:06:45.691 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:06:45.691 INFO:teuthology.orchestra.run.smithi134.stdout:Reinstalled: 2024-04-22T23:06:45.691 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-22T23:06:45.691 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:06:45.691 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-22T23:06:45.872 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q kernel | sort -rV | head -n 1 2024-04-22T23:06:45.919 INFO:teuthology.orchestra.run.smithi134.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-22T23:06:45.919 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-22T23:06:45.920 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-22T23:06:45.920 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-04-22T23:06:45.920 DEBUG:teuthology.orchestra.run.smithi134:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-22T23:06:45.953 INFO:teuthology.orchestra.run.smithi134.stdout:ttyS1 2024-04-22T23:06:45.988 DEBUG:teuthology.parallel:result is None 2024-04-22T23:06:45.988 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-22T23:06:45.995 INFO:teuthology.task.internal:Creating test directory... 2024-04-22T23:06:45.995 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T23:06:45.998 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T23:06:46.043 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-22T23:06:46.049 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-22T23:06:46.060 INFO:teuthology.task.internal:Creating archive directory... 2024-04-22T23:06:46.060 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T23:06:46.062 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T23:06:46.111 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-22T23:06:46.117 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-22T23:06:46.117 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T23:06:46.131 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T23:06:46.159 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T23:06:46.170 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T23:06:46.178 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T23:06:46.187 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T23:06:46.188 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-22T23:06:46.196 INFO:teuthology.task.internal:Configuring sudo... 2024-04-22T23:06:46.196 DEBUG:teuthology.orchestra.run.smithi067:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T23:06:46.214 DEBUG:teuthology.orchestra.run.smithi134:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T23:06:46.252 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-22T23:06:46.261 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-22T23:06:46.261 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T23:06:46.284 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T23:06:46.306 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T23:06:46.383 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T23:06:46.436 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T23:06:46.521 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T23:06:46.564 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-22T23:06:46.564 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T23:06:46.630 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T23:06:46.669 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T23:06:46.713 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T23:06:46.796 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T23:06:46.833 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-22T23:06:46.833 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T23:06:46.896 DEBUG:teuthology.orchestra.run.smithi067:> sudo service rsyslog restart 2024-04-22T23:06:46.899 DEBUG:teuthology.orchestra.run.smithi134:> sudo service rsyslog restart 2024-04-22T23:06:46.932 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T23:06:46.967 INFO:teuthology.orchestra.run.smithi134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T23:06:47.330 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-22T23:06:47.339 INFO:teuthology.task.internal:Starting timer... 2024-04-22T23:06:47.340 INFO:teuthology.run_tasks:Running task pcp... 2024-04-22T23:06:47.357 INFO:teuthology.run_tasks:Running task selinux... 2024-04-22T23:06:47.366 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-22T23:06:47.366 DEBUG:teuthology.orchestra.run.smithi067:> sudo service auditd rotate 2024-04-22T23:06:47.438 INFO:teuthology.orchestra.run.smithi067.stdout:Rotating logs: 2024-04-22T23:06:47.440 DEBUG:teuthology.orchestra.run.smithi134:> sudo service auditd rotate 2024-04-22T23:06:47.502 INFO:teuthology.orchestra.run.smithi134.stdout:Rotating logs: 2024-04-22T23:06:47.504 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-22T23:06:47.504 DEBUG:teuthology.orchestra.run.smithi067:> /usr/sbin/getenforce 2024-04-22T23:06:47.532 INFO:teuthology.orchestra.run.smithi067.stdout:Permissive 2024-04-22T23:06:47.532 DEBUG:teuthology.orchestra.run.smithi134:> /usr/sbin/getenforce 2024-04-22T23:06:47.554 INFO:teuthology.orchestra.run.smithi134.stdout:Permissive 2024-04-22T23:06:47.554 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi067.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi134.front.sepia.ceph.com': 'permissive'} 2024-04-22T23:06:47.555 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T23:06:47.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T23:06:47.601 DEBUG:teuthology.orchestra.run.smithi134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T23:06:47.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T23:06:47.625 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-22T23:06:47.625 DEBUG:teuthology.orchestra.run.smithi067:> sudo /usr/sbin/setenforce permissive 2024-04-22T23:06:47.677 DEBUG:teuthology.orchestra.run.smithi134:> sudo /usr/sbin/setenforce permissive 2024-04-22T23:06:47.700 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-22T23:06:47.711 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-22T23:06:47.727 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-22T23:06:47.729 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi067.front.sepia.ceph.com,smithi134.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-22T23:10:16.315 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi067.front.sepia.ceph.com'), Remote(name='ubuntu@smithi134.front.sepia.ceph.com')] 2024-04-22T23:10:16.317 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-22T23:10:16.318 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T23:10:16.393 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-04-22T23:10:16.469 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-22T23:10:16.469 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-04-22T23:10:16.470 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T23:10:16.545 DEBUG:teuthology.orchestra.run.smithi134:> true 2024-04-22T23:10:16.632 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-04-22T23:10:16.633 INFO:teuthology.run_tasks:Running task clock... 2024-04-22T23:10:16.651 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-22T23:10:16.651 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T23:10:16.652 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T23:10:16.654 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T23:10:16.654 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T23:10:16.680 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T23:10:16.694 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T23:10:16.704 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T23:10:16.717 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: ntpd: command not found 2024-04-22T23:10:16.719 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T23:10:16.728 INFO:teuthology.orchestra.run.smithi067.stdout:506 Cannot talk to daemon 2024-04-22T23:10:16.741 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T23:10:16.743 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: ntpd: command not found 2024-04-22T23:10:16.754 INFO:teuthology.orchestra.run.smithi134.stdout:506 Cannot talk to daemon 2024-04-22T23:10:16.755 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T23:10:16.768 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T23:10:16.781 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T23:10:16.812 INFO:teuthology.orchestra.run.smithi067.stderr:bash: line 1: ntpq: command not found 2024-04-22T23:10:16.815 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T23:10:16.815 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2024-04-22T23:10:16.815 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T23:10:16.815 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T23:10:16.815 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T23:10:16.815 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T23:10:16.840 INFO:teuthology.orchestra.run.smithi134.stderr:bash: line 1: ntpq: command not found 2024-04-22T23:10:16.844 INFO:teuthology.orchestra.run.smithi134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T23:10:16.844 INFO:teuthology.orchestra.run.smithi134.stdout:=============================================================================== 2024-04-22T23:10:16.844 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T23:10:16.844 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T23:10:16.844 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T23:10:16.844 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T23:10:16.845 INFO:teuthology.run_tasks:Running task install... 2024-04-22T23:10:16.853 DEBUG:teuthology.task.install:project ceph 2024-04-22T23:10:16.853 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '43be020184947e53516056c9931e1ac5bdbbb1a5'}} 2024-04-22T23:10:16.853 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v18.2.1', 'flavor': 'default', 'sha1': '43be020184947e53516056c9931e1ac5bdbbb1a5'} 2024-04-22T23:10:16.853 INFO:teuthology.task.install:Using flavor: default 2024-04-22T23:10:16.859 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-22T23:10:16.859 INFO:teuthology.task.install:extra packages: [] 2024-04-22T23:10:16.860 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '43be020184947e53516056c9931e1ac5bdbbb1a5', 'tag': 'v18.2.1', 'wait_for_package': False} 2024-04-22T23:10:16.860 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-22T23:10:16.860 INFO:teuthology.packaging:ref: None 2024-04-22T23:10:16.860 INFO:teuthology.packaging:tag: v18.2.1 2024-04-22T23:10:16.860 INFO:teuthology.packaging:branch: None 2024-04-22T23:10:16.860 INFO:teuthology.packaging:sha1: 43be020184947e53516056c9931e1ac5bdbbb1a5 2024-04-22T23:10:17.297 DEBUG:teuthology.repo_utils:git ls-remote https://git.ceph.com/ceph-ci v18.2.1^{} -> None 2024-04-22T23:10:17.298 INFO:teuthology.packaging:Tag 'v18.2.1' not found in https://git.ceph.com/ceph-ci; will also look in https://git.ceph.com/ceph 2024-04-22T23:10:19.958 DEBUG:teuthology.repo_utils:git ls-remote https://git.ceph.com/ceph v18.2.1^{} -> 7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-04-22T23:10:19.959 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-04-22T23:10:19.961 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '43be020184947e53516056c9931e1ac5bdbbb1a5', 'tag': 'v18.2.1', 'wait_for_package': False} 2024-04-22T23:10:19.962 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-22T23:10:19.962 INFO:teuthology.packaging:ref: None 2024-04-22T23:10:19.962 INFO:teuthology.packaging:tag: v18.2.1 2024-04-22T23:10:19.962 INFO:teuthology.packaging:branch: None 2024-04-22T23:10:19.962 INFO:teuthology.packaging:sha1: 43be020184947e53516056c9931e1ac5bdbbb1a5 2024-04-22T23:10:19.962 INFO:teuthology.packaging:Tag 'v18.2.1' not found in https://git.ceph.com/ceph-ci; will also look in https://git.ceph.com/ceph 2024-04-22T23:10:19.962 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-04-22T23:10:20.186 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/ 2024-04-22T23:10:20.187 INFO:teuthology.task.install.rpm:Package version is 18.2.1-0 2024-04-22T23:10:20.199 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/ 2024-04-22T23:10:20.199 INFO:teuthology.task.install.rpm:Package version is 18.2.1-0 2024-04-22T23:10:20.233 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-22T23:10:20.234 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-22T23:10:20.234 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-22T23:10:20.245 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-22T23:10:20.246 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-22T23:10:20.246 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-22T23:10:20.266 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-22T23:10:20.266 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-22T23:10:20.266 INFO:teuthology.packaging:ref: None 2024-04-22T23:10:20.266 INFO:teuthology.packaging:tag: v18.2.1 2024-04-22T23:10:20.266 INFO:teuthology.packaging:branch: None 2024-04-22T23:10:20.266 INFO:teuthology.packaging:sha1: 43be020184947e53516056c9931e1ac5bdbbb1a5 2024-04-22T23:10:20.267 DEBUG:teuthology.orchestra.run.smithi067:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v18.2.1/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-22T23:10:20.276 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-22T23:10:20.276 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-22T23:10:20.276 INFO:teuthology.packaging:ref: None 2024-04-22T23:10:20.276 INFO:teuthology.packaging:tag: v18.2.1 2024-04-22T23:10:20.276 INFO:teuthology.packaging:branch: None 2024-04-22T23:10:20.276 INFO:teuthology.packaging:sha1: 43be020184947e53516056c9931e1ac5bdbbb1a5 2024-04-22T23:10:20.276 DEBUG:teuthology.orchestra.run.smithi134:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v18.2.1/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-22T23:10:20.336 DEBUG:teuthology.orchestra.run.smithi067:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-22T23:10:20.344 DEBUG:teuthology.orchestra.run.smithi134:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-22T23:10:20.414 DEBUG:teuthology.orchestra.run.smithi067:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-22T23:10:20.420 DEBUG:teuthology.orchestra.run.smithi134:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-22T23:10:20.481 INFO:teuthology.orchestra.run.smithi067.stdout:check_obsoletes = 1 2024-04-22T23:10:20.482 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum clean all 2024-04-22T23:10:20.483 INFO:teuthology.orchestra.run.smithi134.stdout:check_obsoletes = 1 2024-04-22T23:10:20.485 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum clean all 2024-04-22T23:10:20.718 INFO:teuthology.orchestra.run.smithi134.stdout:48 files removed 2024-04-22T23:10:20.719 INFO:teuthology.orchestra.run.smithi067.stdout:48 files removed 2024-04-22T23:10:20.746 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-radosgw 2024-04-22T23:10:20.749 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-radosgw 2024-04-22T23:10:21.177 INFO:teuthology.orchestra.run.smithi134.stdout:ceph packages for x86_64 496 kB/s | 76 kB 00:00 2024-04-22T23:10:21.178 INFO:teuthology.orchestra.run.smithi067.stdout:ceph packages for x86_64 473 kB/s | 76 kB 00:00 2024-04-22T23:10:21.386 INFO:teuthology.orchestra.run.smithi134.stdout:ceph noarch packages 57 kB/s | 9.4 kB 00:00 2024-04-22T23:10:21.428 INFO:teuthology.orchestra.run.smithi067.stdout:ceph noarch packages 46 kB/s | 9.4 kB 00:00 2024-04-22T23:10:21.570 INFO:teuthology.orchestra.run.smithi134.stdout:ceph source packages 15 kB/s | 2.2 kB 00:00 2024-04-22T23:10:21.604 INFO:teuthology.orchestra.run.smithi067.stdout:ceph source packages 15 kB/s | 2.2 kB 00:00 2024-04-22T23:10:23.145 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - BaseOS 5.2 MB/s | 8.0 MB 00:01 2024-04-22T23:10:23.263 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - BaseOS 5.0 MB/s | 8.0 MB 00:01 2024-04-22T23:10:27.087 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - AppStream 6.6 MB/s | 19 MB 00:02 2024-04-22T23:10:31.496 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - CRB 7.1 MB/s | 6.2 MB 00:00 2024-04-22T23:10:33.314 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - Extras packages 24 kB/s | 16 kB 00:00 2024-04-22T23:10:35.301 INFO:teuthology.orchestra.run.smithi134.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-04-22T23:10:40.296 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-04-22T23:10:40.381 INFO:teuthology.orchestra.run.smithi134.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-04-22T23:10:42.523 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-22T23:10:42.525 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:10:42.525 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-04-22T23:10:42.525 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:10:42.525 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-04-22T23:10:42.526 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw x86_64 2:18.2.1-0.el9 ceph 7.7 M 2024-04-22T23:10:42.526 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrading: 2024-04-22T23:10:42.526 INFO:teuthology.orchestra.run.smithi134.stdout: librados2 x86_64 2:18.2.1-0.el9 ceph 3.3 M 2024-04-22T23:10:42.526 INFO:teuthology.orchestra.run.smithi134.stdout: librbd1 x86_64 2:18.2.1-0.el9 ceph 3.0 M 2024-04-22T23:10:42.526 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-04-22T23:10:42.526 INFO:teuthology.orchestra.run.smithi134.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-22T23:10:42.526 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base x86_64 2:18.2.1-0.el9 ceph 5.2 M 2024-04-22T23:10:42.526 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common x86_64 2:18.2.1-0.el9 ceph 18 M 2024-04-22T23:10:42.526 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-selinux x86_64 2:18.2.1-0.el9 ceph 24 k 2024-04-22T23:10:42.526 INFO:teuthology.orchestra.run.smithi134.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-04-22T23:10:42.526 INFO:teuthology.orchestra.run.smithi134.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-22T23:10:42.526 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-22T23:10:42.526 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-22T23:10:42.526 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2 x86_64 2:18.2.1-0.el9 ceph 658 k 2024-04-22T23:10:42.527 INFO:teuthology.orchestra.run.smithi134.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-22T23:10:42.527 INFO:teuthology.orchestra.run.smithi134.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-22T23:10:42.527 INFO:teuthology.orchestra.run.smithi134.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-22T23:10:42.527 INFO:teuthology.orchestra.run.smithi134.stdout: libradosstriper1 x86_64 2:18.2.1-0.el9 ceph 474 k 2024-04-22T23:10:42.527 INFO:teuthology.orchestra.run.smithi134.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-22T23:10:42.527 INFO:teuthology.orchestra.run.smithi134.stdout: librgw2 x86_64 2:18.2.1-0.el9 ceph 4.5 M 2024-04-22T23:10:42.527 INFO:teuthology.orchestra.run.smithi134.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-22T23:10:42.527 INFO:teuthology.orchestra.run.smithi134.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-22T23:10:42.527 INFO:teuthology.orchestra.run.smithi134.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-22T23:10:42.527 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-argparse x86_64 2:18.2.1-0.el9 ceph 45 k 2024-04-22T23:10:42.527 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-common x86_64 2:18.2.1-0.el9 ceph 124 k 2024-04-22T23:10:42.527 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs x86_64 2:18.2.1-0.el9 ceph 161 k 2024-04-22T23:10:42.527 INFO:teuthology.orchestra.run.smithi134.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-22T23:10:42.527 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados x86_64 2:18.2.1-0.el9 ceph 321 k 2024-04-22T23:10:42.527 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd x86_64 2:18.2.1-0.el9 ceph 297 k 2024-04-22T23:10:42.528 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw x86_64 2:18.2.1-0.el9 ceph 99 k 2024-04-22T23:10:42.528 INFO:teuthology.orchestra.run.smithi134.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-22T23:10:42.528 INFO:teuthology.orchestra.run.smithi134.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-22T23:10:42.528 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:10:42.528 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-04-22T23:10:42.528 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:10:42.528 INFO:teuthology.orchestra.run.smithi134.stdout:Install 28 Packages 2024-04-22T23:10:42.528 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrade 2 Packages 2024-04-22T23:10:42.528 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:10:42.528 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 53 M 2024-04-22T23:10:42.529 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-04-22T23:10:43.066 INFO:teuthology.orchestra.run.smithi134.stdout:(1/30): ceph-base-18.2.1-0.el9.x86_64.rpm 30 MB/s | 5.2 MB 00:00 2024-04-22T23:10:43.108 INFO:teuthology.orchestra.run.smithi134.stdout:(2/30): ceph-selinux-18.2.1-0.el9.x86_64.rpm 580 kB/s | 24 kB 00:00 2024-04-22T23:10:43.150 INFO:teuthology.orchestra.run.smithi134.stdout:(3/30): libcephfs2-18.2.1-0.el9.x86_64.rpm 15 MB/s | 658 kB 00:00 2024-04-22T23:10:43.183 INFO:teuthology.orchestra.run.smithi134.stdout:(4/30): libradosstriper1-18.2.1-0.el9.x86_64.rp 14 MB/s | 474 kB 00:00 2024-04-22T23:10:43.275 INFO:teuthology.orchestra.run.smithi134.stdout:(5/30): librgw2-18.2.1-0.el9.x86_64.rpm 49 MB/s | 4.5 MB 00:00 2024-04-22T23:10:43.292 INFO:teuthology.orchestra.run.smithi134.stdout:(6/30): python3-ceph-argparse-18.2.1-0.el9.x86_ 2.7 MB/s | 45 kB 00:00 2024-04-22T23:10:43.384 INFO:teuthology.orchestra.run.smithi134.stdout:(7/30): ceph-radosgw-18.2.1-0.el9.x86_64.rpm 16 MB/s | 7.7 MB 00:00 2024-04-22T23:10:43.409 INFO:teuthology.orchestra.run.smithi134.stdout:(8/30): python3-ceph-common-18.2.1-0.el9.x86_64 1.0 MB/s | 124 kB 00:00 2024-04-22T23:10:43.434 INFO:teuthology.orchestra.run.smithi134.stdout:(9/30): python3-cephfs-18.2.1-0.el9.x86_64.rpm 3.1 MB/s | 161 kB 00:00 2024-04-22T23:10:43.460 INFO:teuthology.orchestra.run.smithi134.stdout:(10/30): python3-rados-18.2.1-0.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-04-22T23:10:43.495 INFO:teuthology.orchestra.run.smithi134.stdout:(11/30): python3-rbd-18.2.1-0.el9.x86_64.rpm 4.8 MB/s | 297 kB 00:00 2024-04-22T23:10:43.521 INFO:teuthology.orchestra.run.smithi134.stdout:(12/30): python3-rgw-18.2.1-0.el9.x86_64.rpm 1.6 MB/s | 99 kB 00:00 2024-04-22T23:10:43.738 INFO:teuthology.orchestra.run.smithi134.stdout:(13/30): ceph-common-18.2.1-0.el9.x86_64.rpm 22 MB/s | 18 MB 00:00 2024-04-22T23:10:43.913 INFO:teuthology.orchestra.run.smithi134.stdout:(14/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 259 kB/s | 45 kB 00:00 2024-04-22T23:10:43.938 INFO:teuthology.orchestra.run.smithi134.stdout:(15/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 384 kB/s | 160 kB 00:00 2024-04-22T23:10:43.987 INFO:teuthology.orchestra.run.smithi134.stdout:(16/30): boost-program-options-1.75.0-8.el9.x86 217 kB/s | 107 kB 00:00 2024-04-22T23:10:44.097 INFO:teuthology.orchestra.run.smithi134.stdout:(17/30): librdkafka-1.6.1-102.el9.x86_64.rpm 3.5 MB/s | 662 kB 00:00 2024-04-22T23:10:44.122 INFO:teuthology.orchestra.run.smithi134.stdout:(18/30): python3-prettytable-0.7.2-27.el9.noarc 314 kB/s | 42 kB 00:00 2024-04-22T23:10:44.147 INFO:teuthology.orchestra.run.smithi134.stdout:(19/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.4 MB/s | 292 kB 00:00 2024-04-22T23:10:44.481 INFO:teuthology.orchestra.run.smithi134.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 288 kB/s | 111 kB 00:00 2024-04-22T23:10:44.573 INFO:teuthology.orchestra.run.smithi134.stdout:(21/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 284 kB/s | 26 kB 00:00 2024-04-22T23:10:44.598 INFO:teuthology.orchestra.run.smithi134.stdout:(22/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 649 kB/s | 309 kB 00:00 2024-04-22T23:10:44.682 INFO:teuthology.orchestra.run.smithi134.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 448 kB/s | 49 kB 00:00 2024-04-22T23:10:44.723 INFO:teuthology.orchestra.run.smithi134.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 538 kB/s | 67 kB 00:00 2024-04-22T23:10:44.857 INFO:teuthology.orchestra.run.smithi134.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2024-04-22T23:10:45.207 INFO:teuthology.orchestra.run.smithi134.stdout:(26/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.6 MB/s | 838 kB 00:00 2024-04-22T23:10:45.275 INFO:teuthology.orchestra.run.smithi134.stdout:(27/30): librados2-18.2.1-0.el9.x86_64.rpm 49 MB/s | 3.3 MB 00:00 2024-04-22T23:10:45.317 INFO:teuthology.orchestra.run.smithi134.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 3.4 MB/s | 1.6 MB 00:00 2024-04-22T23:10:45.375 INFO:teuthology.orchestra.run.smithi134.stdout:(29/30): libarrow-9.0.0-10.el9.x86_64.rpm 3.6 MB/s | 4.4 MB 00:01 2024-04-22T23:10:45.434 INFO:teuthology.orchestra.run.smithi134.stdout:(30/30): librbd1-18.2.1-0.el9.x86_64.rpm 19 MB/s | 3.0 MB 00:00 2024-04-22T23:10:45.438 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-04-22T23:10:45.438 INFO:teuthology.orchestra.run.smithi134.stdout:Total 18 MB/s | 53 MB 00:02 2024-04-22T23:10:45.507 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-04-22T23:10:45.542 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-04-22T23:10:45.542 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-04-22T23:10:45.962 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-04-22T23:10:45.963 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-04-22T23:10:47.345 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-04-22T23:10:47.426 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-04-22T23:10:47.464 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-04-22T23:10:47.513 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-04-22T23:10:47.882 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-04-22T23:10:47.929 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : librados2-2:18.2.1-0.el9.x86_64 5/32 2024-04-22T23:10:48.013 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-2:18.2.1-0.el9.x86_64 5/32 2024-04-22T23:10:48.093 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rados-2:18.2.1-0.el9.x86_64 6/32 2024-04-22T23:10:48.114 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs2-2:18.2.1-0.el9.x86_64 7/32 2024-04-22T23:10:48.160 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephfs2-2:18.2.1-0.el9.x86_64 7/32 2024-04-22T23:10:48.189 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-04-22T23:10:48.232 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ceph-argparse-2:18.2.1-0.el9.x86_64 9/32 2024-04-22T23:10:48.298 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cephfs-2:18.2.1-0.el9.x86_64 10/32 2024-04-22T23:10:48.350 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libradosstriper1-2:18.2.1-0.el9.x86_64 11/32 2024-04-22T23:10:48.415 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libradosstriper1-2:18.2.1-0.el9.x86_64 11/32 2024-04-22T23:10:48.521 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-04-22T23:10:48.548 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-04-22T23:10:48.588 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-04-22T23:10:48.613 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-04-22T23:10:48.640 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-04-22T23:10:48.676 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-04-22T23:10:48.700 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-04-22T23:10:48.999 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-04-22T23:10:49.039 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : librbd1-2:18.2.1-0.el9.x86_64 20/32 2024-04-22T23:10:49.129 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-2:18.2.1-0.el9.x86_64 20/32 2024-04-22T23:10:49.159 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rbd-2:18.2.1-0.el9.x86_64 21/32 2024-04-22T23:10:49.195 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-04-22T23:10:49.596 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-04-22T23:10:49.624 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librgw2-2:18.2.1-0.el9.x86_64 24/32 2024-04-22T23:10:49.695 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librgw2-2:18.2.1-0.el9.x86_64 24/32 2024-04-22T23:10:49.763 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rgw-2:18.2.1-0.el9.x86_64 25/32 2024-04-22T23:10:49.801 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ceph-common-2:18.2.1-0.el9.x86_64 26/32 2024-04-22T23:10:51.876 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-04-22T23:10:51.906 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-04-22T23:10:51.946 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - CRB 792 kB/s | 6.2 MB 00:07 2024-04-22T23:10:52.438 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-04-22T23:10:52.476 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-base-2:18.2.1-0.el9.x86_64 28/32 2024-04-22T23:10:52.574 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-base-2:18.2.1-0.el9.x86_64 28/32 2024-04-22T23:10:52.574 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-22T23:10:52.574 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-22T23:10:52.574 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:10:52.602 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-selinux-2:18.2.1-0.el9.x86_64 29/32 2024-04-22T23:10:53.405 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - Extras packages 64 kB/s | 16 kB 00:00 2024-04-22T23:10:54.598 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-04-22T23:10:59.998 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 51 kB/s | 1.7 kB 00:00 2024-04-22T23:11:02.217 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-22T23:11:02.219 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:11:02.220 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-04-22T23:11:02.220 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:11:02.220 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-22T23:11:02.220 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw x86_64 2:18.2.1-0.el9 ceph 7.7 M 2024-04-22T23:11:02.220 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrading: 2024-04-22T23:11:02.220 INFO:teuthology.orchestra.run.smithi067.stdout: librados2 x86_64 2:18.2.1-0.el9 ceph 3.3 M 2024-04-22T23:11:02.220 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1 x86_64 2:18.2.1-0.el9 ceph 3.0 M 2024-04-22T23:11:02.220 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-04-22T23:11:02.220 INFO:teuthology.orchestra.run.smithi067.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-22T23:11:02.220 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base x86_64 2:18.2.1-0.el9 ceph 5.2 M 2024-04-22T23:11:02.220 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common x86_64 2:18.2.1-0.el9 ceph 18 M 2024-04-22T23:11:02.220 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux x86_64 2:18.2.1-0.el9 ceph 24 k 2024-04-22T23:11:02.220 INFO:teuthology.orchestra.run.smithi067.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-04-22T23:11:02.221 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-22T23:11:02.221 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-22T23:11:02.221 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-22T23:11:02.221 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2 x86_64 2:18.2.1-0.el9 ceph 658 k 2024-04-22T23:11:02.221 INFO:teuthology.orchestra.run.smithi067.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-22T23:11:02.221 INFO:teuthology.orchestra.run.smithi067.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-22T23:11:02.221 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-22T23:11:02.221 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1 x86_64 2:18.2.1-0.el9 ceph 474 k 2024-04-22T23:11:02.221 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-22T23:11:02.221 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2 x86_64 2:18.2.1-0.el9 ceph 4.5 M 2024-04-22T23:11:02.221 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-22T23:11:02.221 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-22T23:11:02.222 INFO:teuthology.orchestra.run.smithi067.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-22T23:11:02.222 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse x86_64 2:18.2.1-0.el9 ceph 45 k 2024-04-22T23:11:02.222 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common x86_64 2:18.2.1-0.el9 ceph 124 k 2024-04-22T23:11:02.222 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs x86_64 2:18.2.1-0.el9 ceph 161 k 2024-04-22T23:11:02.222 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-22T23:11:02.222 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados x86_64 2:18.2.1-0.el9 ceph 321 k 2024-04-22T23:11:02.222 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd x86_64 2:18.2.1-0.el9 ceph 297 k 2024-04-22T23:11:02.222 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw x86_64 2:18.2.1-0.el9 ceph 99 k 2024-04-22T23:11:02.222 INFO:teuthology.orchestra.run.smithi067.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-22T23:11:02.222 INFO:teuthology.orchestra.run.smithi067.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-22T23:11:02.222 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:11:02.222 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-22T23:11:02.222 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:11:02.222 INFO:teuthology.orchestra.run.smithi067.stdout:Install 28 Packages 2024-04-22T23:11:02.223 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrade 2 Packages 2024-04-22T23:11:02.223 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:11:02.223 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 53 M 2024-04-22T23:11:02.223 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-22T23:11:02.228 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-selinux-2:18.2.1-0.el9.x86_64 29/32 2024-04-22T23:11:02.229 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /sys 2024-04-22T23:11:02.229 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /proc 2024-04-22T23:11:02.229 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /mnt 2024-04-22T23:11:02.229 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /var/tmp 2024-04-22T23:11:02.229 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /home 2024-04-22T23:11:02.229 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /root 2024-04-22T23:11:02.229 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /tmp 2024-04-22T23:11:02.229 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:02.942 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-radosgw-2:18.2.1-0.el9.x86_64 30/32 2024-04-22T23:11:02.966 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-radosgw-2:18.2.1-0.el9.x86_64 30/32 2024-04-22T23:11:02.966 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T23:11:02.966 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-22T23:11:02.966 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T23:11:02.966 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T23:11:02.966 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:02.968 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-22T23:11:03.052 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-22T23:11:03.052 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-22T23:11:03.074 INFO:teuthology.orchestra.run.smithi067.stdout:(1/30): ceph-base-18.2.1-0.el9.x86_64.rpm 26 MB/s | 5.2 MB 00:00 2024-04-22T23:11:03.100 INFO:teuthology.orchestra.run.smithi067.stdout:(2/30): ceph-selinux-18.2.1-0.el9.x86_64.rpm 962 kB/s | 24 kB 00:00 2024-04-22T23:11:03.142 INFO:teuthology.orchestra.run.smithi067.stdout:(3/30): libcephfs2-18.2.1-0.el9.x86_64.rpm 16 MB/s | 658 kB 00:00 2024-04-22T23:11:03.175 INFO:teuthology.orchestra.run.smithi067.stdout:(4/30): libradosstriper1-18.2.1-0.el9.x86_64.rp 14 MB/s | 474 kB 00:00 2024-04-22T23:11:03.292 INFO:teuthology.orchestra.run.smithi067.stdout:(5/30): ceph-radosgw-18.2.1-0.el9.x86_64.rpm 18 MB/s | 7.7 MB 00:00 2024-04-22T23:11:03.317 INFO:teuthology.orchestra.run.smithi067.stdout:(6/30): python3-ceph-argparse-18.2.1-0.el9.x86_ 1.8 MB/s | 45 kB 00:00 2024-04-22T23:11:03.343 INFO:teuthology.orchestra.run.smithi067.stdout:(7/30): python3-ceph-common-18.2.1-0.el9.x86_64 4.9 MB/s | 124 kB 00:00 2024-04-22T23:11:03.368 INFO:teuthology.orchestra.run.smithi067.stdout:(8/30): python3-cephfs-18.2.1-0.el9.x86_64.rpm 6.3 MB/s | 161 kB 00:00 2024-04-22T23:11:03.393 INFO:teuthology.orchestra.run.smithi067.stdout:(9/30): python3-rados-18.2.1-0.el9.x86_64.rpm 12 MB/s | 321 kB 00:00 2024-04-22T23:11:03.452 INFO:teuthology.orchestra.run.smithi067.stdout:(10/30): librgw2-18.2.1-0.el9.x86_64.rpm 16 MB/s | 4.5 MB 00:00 2024-04-22T23:11:03.477 INFO:teuthology.orchestra.run.smithi067.stdout:(11/30): python3-rgw-18.2.1-0.el9.x86_64.rpm 3.9 MB/s | 99 kB 00:00 2024-04-22T23:11:03.502 INFO:teuthology.orchestra.run.smithi067.stdout:(12/30): python3-rbd-18.2.1-0.el9.x86_64.rpm 2.7 MB/s | 297 kB 00:00 2024-04-22T23:11:03.719 INFO:teuthology.orchestra.run.smithi067.stdout:(13/30): ceph-common-18.2.1-0.el9.x86_64.rpm 22 MB/s | 18 MB 00:00 2024-04-22T23:11:03.753 INFO:teuthology.orchestra.run.smithi067.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 388 kB/s | 107 kB 00:00 2024-04-22T23:11:03.811 INFO:teuthology.orchestra.run.smithi067.stdout:(15/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 519 kB/s | 160 kB 00:00 2024-04-22T23:11:03.837 INFO:teuthology.orchestra.run.smithi067.stdout:(16/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 388 kB/s | 45 kB 00:00 2024-04-22T23:11:03.887 INFO:teuthology.orchestra.run.smithi067.stdout:(17/30): librdkafka-1.6.1-102.el9.x86_64.rpm 4.8 MB/s | 662 kB 00:00 2024-04-22T23:11:03.912 INFO:teuthology.orchestra.run.smithi067.stdout:(18/30): python3-prettytable-0.7.2-27.el9.noarc 561 kB/s | 42 kB 00:00 2024-04-22T23:11:03.937 INFO:teuthology.orchestra.run.smithi067.stdout:(19/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.3 MB/s | 292 kB 00:00 2024-04-22T23:11:04.029 INFO:teuthology.orchestra.run.smithi067.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 780 kB/s | 111 kB 00:00 2024-04-22T23:11:04.055 INFO:teuthology.orchestra.run.smithi067.stdout:(21/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.1 MB/s | 309 kB 00:00 2024-04-22T23:11:04.080 INFO:teuthology.orchestra.run.smithi067.stdout:(22/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 517 kB/s | 26 kB 00:00 2024-04-22T23:11:04.105 INFO:teuthology.orchestra.run.smithi067.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 968 kB/s | 49 kB 00:00 2024-04-22T23:11:04.122 INFO:teuthology.orchestra.run.smithi067.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-04-22T23:11:04.180 INFO:teuthology.orchestra.run.smithi067.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-04-22T23:11:04.206 INFO:teuthology.orchestra.run.smithi067.stdout:(26/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.2 MB/s | 838 kB 00:00 2024-04-22T23:11:04.277 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-22T23:11:04.277 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-base-2:18.2.1-0.el9.x86_64 1/32 2024-04-22T23:11:04.277 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-common-2:18.2.1-0.el9.x86_64 2/32 2024-04-22T23:11:04.278 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-radosgw-2:18.2.1-0.el9.x86_64 3/32 2024-04-22T23:11:04.278 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-selinux-2:18.2.1-0.el9.x86_64 4/32 2024-04-22T23:11:04.278 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs2-2:18.2.1-0.el9.x86_64 5/32 2024-04-22T23:11:04.278 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libradosstriper1-2:18.2.1-0.el9.x86_64 6/32 2024-04-22T23:11:04.278 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librgw2-2:18.2.1-0.el9.x86_64 7/32 2024-04-22T23:11:04.278 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ceph-argparse-2:18.2.1-0.el9.x86_64 8/32 2024-04-22T23:11:04.278 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ceph-common-2:18.2.1-0.el9.x86_64 9/32 2024-04-22T23:11:04.278 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cephfs-2:18.2.1-0.el9.x86_64 10/32 2024-04-22T23:11:04.278 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rados-2:18.2.1-0.el9.x86_64 11/32 2024-04-22T23:11:04.278 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rbd-2:18.2.1-0.el9.x86_64 12/32 2024-04-22T23:11:04.278 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rgw-2:18.2.1-0.el9.x86_64 13/32 2024-04-22T23:11:04.278 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-04-22T23:11:04.279 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-04-22T23:11:04.279 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-04-22T23:11:04.279 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-04-22T23:11:04.279 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-04-22T23:11:04.279 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-04-22T23:11:04.279 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-04-22T23:11:04.279 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-04-22T23:11:04.279 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-04-22T23:11:04.279 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-04-22T23:11:04.279 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-04-22T23:11:04.279 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-04-22T23:11:04.280 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-04-22T23:11:04.280 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-04-22T23:11:04.280 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-04-22T23:11:04.280 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-2:18.2.1-0.el9.x86_64 29/32 2024-04-22T23:11:04.280 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-04-22T23:11:04.280 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-2:18.2.1-0.el9.x86_64 31/32 2024-04-22T23:11:04.289 INFO:teuthology.orchestra.run.smithi067.stdout:(27/30): librados2-18.2.1-0.el9.x86_64.rpm 39 MB/s | 3.3 MB 00:00 2024-04-22T23:11:04.356 INFO:teuthology.orchestra.run.smithi067.stdout:(28/30): libarrow-9.0.0-10.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-04-22T23:11:04.398 INFO:teuthology.orchestra.run.smithi067.stdout:(29/30): thrift-0.15.0-2.el9.x86_64.rpm 7.3 MB/s | 1.6 MB 00:00 2024-04-22T23:11:04.465 INFO:teuthology.orchestra.run.smithi067.stdout:(30/30): librbd1-18.2.1-0.el9.x86_64.rpm 17 MB/s | 3.0 MB 00:00 2024-04-22T23:11:04.466 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-22T23:11:04.467 INFO:teuthology.orchestra.run.smithi067.stdout:Total 24 MB/s | 53 MB 00:02 2024-04-22T23:11:04.536 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-22T23:11:04.574 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-22T23:11:04.574 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-22T23:11:05.020 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-22T23:11:05.021 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-22T23:11:05.052 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-04-22T23:11:05.052 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:05.052 INFO:teuthology.orchestra.run.smithi134.stdout:Upgraded: 2024-04-22T23:11:05.052 INFO:teuthology.orchestra.run.smithi134.stdout: librados2-2:18.2.1-0.el9.x86_64 librbd1-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:05.052 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-04-22T23:11:05.052 INFO:teuthology.orchestra.run.smithi134.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-22T23:11:05.052 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:05.052 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:05.052 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:05.052 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-selinux-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:05.052 INFO:teuthology.orchestra.run.smithi134.stdout: fmt-8.1.1-5.el9.x86_64 2024-04-22T23:11:05.053 INFO:teuthology.orchestra.run.smithi134.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-22T23:11:05.053 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-22T23:11:05.053 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-22T23:11:05.053 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:05.053 INFO:teuthology.orchestra.run.smithi134.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-22T23:11:05.053 INFO:teuthology.orchestra.run.smithi134.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-22T23:11:05.053 INFO:teuthology.orchestra.run.smithi134.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-22T23:11:05.053 INFO:teuthology.orchestra.run.smithi134.stdout: libradosstriper1-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:05.053 INFO:teuthology.orchestra.run.smithi134.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-22T23:11:05.054 INFO:teuthology.orchestra.run.smithi134.stdout: librgw2-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:05.054 INFO:teuthology.orchestra.run.smithi134.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-22T23:11:05.054 INFO:teuthology.orchestra.run.smithi134.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-22T23:11:05.054 INFO:teuthology.orchestra.run.smithi134.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-22T23:11:05.054 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-argparse-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:05.054 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-common-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:05.054 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:05.054 INFO:teuthology.orchestra.run.smithi134.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-22T23:11:05.054 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:05.054 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:05.054 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:05.054 INFO:teuthology.orchestra.run.smithi134.stdout: re2-1:20211101-3.el9.x86_64 2024-04-22T23:11:05.054 INFO:teuthology.orchestra.run.smithi134.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-22T23:11:05.055 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:05.055 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-22T23:11:05.332 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-test 2024-04-22T23:11:05.938 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:25 ago on Mon 22 Apr 2024 11:10:40 PM UTC. 2024-04-22T23:11:06.064 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-22T23:11:06.065 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:11:06.065 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-04-22T23:11:06.065 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:11:06.065 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-04-22T23:11:06.065 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test x86_64 2:18.2.1-0.el9 ceph 40 M 2024-04-22T23:11:06.065 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-04-22T23:11:06.065 INFO:teuthology.orchestra.run.smithi134.stdout: libcephsqlite x86_64 2:18.2.1-0.el9 ceph 165 k 2024-04-22T23:11:06.065 INFO:teuthology.orchestra.run.smithi134.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-22T23:11:06.065 INFO:teuthology.orchestra.run.smithi134.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-22T23:11:06.065 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:06.065 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-04-22T23:11:06.065 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:11:06.065 INFO:teuthology.orchestra.run.smithi134.stdout:Install 4 Packages 2024-04-22T23:11:06.065 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:06.066 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 40 M 2024-04-22T23:11:06.066 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 166 M 2024-04-22T23:11:06.066 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-04-22T23:11:06.295 INFO:teuthology.orchestra.run.smithi134.stdout:(1/4): libcephsqlite-18.2.1-0.el9.x86_64.rpm 1.4 MB/s | 165 kB 00:00 2024-04-22T23:11:06.439 INFO:teuthology.orchestra.run.smithi134.stdout:(2/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.2 MB/s | 305 kB 00:00 2024-04-22T23:11:06.441 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-22T23:11:06.467 INFO:teuthology.orchestra.run.smithi134.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 370 kB/s | 64 kB 00:00 2024-04-22T23:11:06.526 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-04-22T23:11:06.554 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-04-22T23:11:06.605 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-04-22T23:11:06.856 INFO:teuthology.orchestra.run.smithi134.stdout:(4/4): ceph-test-18.2.1-0.el9.x86_64.rpm 59 MB/s | 40 MB 00:00 2024-04-22T23:11:06.859 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-04-22T23:11:06.859 INFO:teuthology.orchestra.run.smithi134.stdout:Total 51 MB/s | 40 MB 00:00 2024-04-22T23:11:06.885 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-04-22T23:11:06.904 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-04-22T23:11:06.904 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-04-22T23:11:06.932 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-04-22T23:11:07.037 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librados2-2:18.2.1-0.el9.x86_64 5/32 2024-04-22T23:11:07.130 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-2:18.2.1-0.el9.x86_64 5/32 2024-04-22T23:11:07.212 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rados-2:18.2.1-0.el9.x86_64 6/32 2024-04-22T23:11:07.237 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-04-22T23:11:07.237 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-04-22T23:11:07.238 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs2-2:18.2.1-0.el9.x86_64 7/32 2024-04-22T23:11:07.301 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephfs2-2:18.2.1-0.el9.x86_64 7/32 2024-04-22T23:11:07.330 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-04-22T23:11:07.374 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-argparse-2:18.2.1-0.el9.x86_64 9/32 2024-04-22T23:11:07.441 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cephfs-2:18.2.1-0.el9.x86_64 10/32 2024-04-22T23:11:07.465 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libradosstriper1-2:18.2.1-0.el9.x86_64 11/32 2024-04-22T23:11:07.540 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libradosstriper1-2:18.2.1-0.el9.x86_64 11/32 2024-04-22T23:11:07.650 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-04-22T23:11:07.681 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-04-22T23:11:07.721 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-04-22T23:11:07.745 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-04-22T23:11:07.754 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-04-22T23:11:07.782 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-04-22T23:11:07.818 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-04-22T23:11:07.831 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-04-22T23:11:07.850 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-04-22T23:11:07.889 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-04-22T23:11:07.916 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephsqlite-2:18.2.1-0.el9.x86_64 3/4 2024-04-22T23:11:08.164 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-04-22T23:11:08.205 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librbd1-2:18.2.1-0.el9.x86_64 20/32 2024-04-22T23:11:08.296 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-2:18.2.1-0.el9.x86_64 20/32 2024-04-22T23:11:08.326 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rbd-2:18.2.1-0.el9.x86_64 21/32 2024-04-22T23:11:08.362 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-04-22T23:11:08.785 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-04-22T23:11:08.815 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librgw2-2:18.2.1-0.el9.x86_64 24/32 2024-04-22T23:11:08.887 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librgw2-2:18.2.1-0.el9.x86_64 24/32 2024-04-22T23:11:08.966 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rgw-2:18.2.1-0.el9.x86_64 25/32 2024-04-22T23:11:09.001 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-common-2:18.2.1-0.el9.x86_64 26/32 2024-04-22T23:11:11.203 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-04-22T23:11:11.239 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-04-22T23:11:11.627 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephsqlite-2:18.2.1-0.el9.x86_64 3/4 2024-04-22T23:11:11.739 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-test-2:18.2.1-0.el9.x86_64 4/4 2024-04-22T23:11:11.806 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:18.2.1-0.el9.x86_64 27/32 2024-04-22T23:11:11.844 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-base-2:18.2.1-0.el9.x86_64 28/32 2024-04-22T23:11:11.936 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-base-2:18.2.1-0.el9.x86_64 28/32 2024-04-22T23:11:11.936 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-22T23:11:11.936 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-22T23:11:11.936 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:11:11.962 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-selinux-2:18.2.1-0.el9.x86_64 29/32 2024-04-22T23:11:13.670 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-test-2:18.2.1-0.el9.x86_64 4/4 2024-04-22T23:11:13.670 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-test-2:18.2.1-0.el9.x86_64 1/4 2024-04-22T23:11:13.670 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephsqlite-2:18.2.1-0.el9.x86_64 2/4 2024-04-22T23:11:13.670 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-04-22T23:11:13.930 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-04-22T23:11:13.930 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:13.930 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-04-22T23:11:13.930 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test-2:18.2.1-0.el9.x86_64 libcephsqlite-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:13.930 INFO:teuthology.orchestra.run.smithi134.stdout: socat-1.7.4.1-5.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2024-04-22T23:11:13.930 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:13.931 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-22T23:11:14.200 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph 2024-04-22T23:11:14.717 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:34 ago on Mon 22 Apr 2024 11:10:40 PM UTC. 2024-04-22T23:11:14.841 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-22T23:11:14.843 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:11:14.844 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-04-22T23:11:14.844 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:11:14.844 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-04-22T23:11:14.844 INFO:teuthology.orchestra.run.smithi134.stdout: ceph x86_64 2:18.2.1-0.el9 ceph 6.4 k 2024-04-22T23:11:14.844 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-04-22T23:11:14.844 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds x86_64 2:18.2.1-0.el9 ceph 2.1 M 2024-04-22T23:11:14.844 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr x86_64 2:18.2.1-0.el9 ceph 1.4 M 2024-04-22T23:11:14.844 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-modules-core noarch 2:18.2.1-0.el9 ceph-noarch 242 k 2024-04-22T23:11:14.844 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon x86_64 2:18.2.1-0.el9 ceph 4.4 M 2024-04-22T23:11:14.844 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd x86_64 2:18.2.1-0.el9 ceph 18 M 2024-04-22T23:11:14.844 INFO:teuthology.orchestra.run.smithi134.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-22T23:11:14.844 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-22T23:11:14.844 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-22T23:11:14.845 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-22T23:11:14.845 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-22T23:11:14.845 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-22T23:11:14.845 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-22T23:11:14.845 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-22T23:11:14.845 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-22T23:11:14.845 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-22T23:11:14.845 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-22T23:11:14.845 INFO:teuthology.orchestra.run.smithi134.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-22T23:11:14.845 INFO:teuthology.orchestra.run.smithi134.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-22T23:11:14.845 INFO:teuthology.orchestra.run.smithi134.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-22T23:11:14.845 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-22T23:11:14.845 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-22T23:11:14.845 INFO:teuthology.orchestra.run.smithi134.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-22T23:11:14.845 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-22T23:11:14.846 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-22T23:11:14.846 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-22T23:11:14.846 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-22T23:11:14.846 INFO:teuthology.orchestra.run.smithi134.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-22T23:11:14.846 INFO:teuthology.orchestra.run.smithi134.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-22T23:11:14.846 INFO:teuthology.orchestra.run.smithi134.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-22T23:11:14.846 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-22T23:11:14.846 INFO:teuthology.orchestra.run.smithi134.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-22T23:11:14.846 INFO:teuthology.orchestra.run.smithi134.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-22T23:11:14.846 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:14.846 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-04-22T23:11:14.846 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:11:14.846 INFO:teuthology.orchestra.run.smithi134.stdout:Install 33 Packages 2024-04-22T23:11:14.846 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:14.847 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 30 M 2024-04-22T23:11:14.847 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 110 M 2024-04-22T23:11:14.847 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-04-22T23:11:15.724 INFO:teuthology.orchestra.run.smithi134.stdout:(1/33): ceph-18.2.1-0.el9.x86_64.rpm 59 kB/s | 6.4 kB 00:00 2024-04-22T23:11:15.766 INFO:teuthology.orchestra.run.smithi134.stdout:(2/33): ceph-mgr-18.2.1-0.el9.x86_64.rpm 9.6 MB/s | 1.4 MB 00:00 2024-04-22T23:11:15.816 INFO:teuthology.orchestra.run.smithi134.stdout:(3/33): ceph-mds-18.2.1-0.el9.x86_64.rpm 10 MB/s | 2.1 MB 00:00 2024-04-22T23:11:15.850 INFO:teuthology.orchestra.run.smithi134.stdout:(4/33): ceph-mgr-modules-core-18.2.1-0.el9.noar 7.1 MB/s | 242 kB 00:00 2024-04-22T23:11:15.942 INFO:teuthology.orchestra.run.smithi134.stdout:(5/33): ceph-mon-18.2.1-0.el9.x86_64.rpm 20 MB/s | 4.4 MB 00:00 2024-04-22T23:11:16.175 INFO:teuthology.orchestra.run.smithi134.stdout:(6/33): ceph-osd-18.2.1-0.el9.x86_64.rpm 43 MB/s | 18 MB 00:00 2024-04-22T23:11:16.242 INFO:teuthology.orchestra.run.smithi134.stdout:(7/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 646 kB/s | 253 kB 00:00 2024-04-22T23:11:16.284 INFO:teuthology.orchestra.run.smithi134.stdout:(8/33): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-04-22T23:11:16.318 INFO:teuthology.orchestra.run.smithi134.stdout:(9/33): python3-cryptography-36.0.1-4.el9.x86_6 3.3 MB/s | 1.2 MB 00:00 2024-04-22T23:11:16.343 INFO:teuthology.orchestra.run.smithi134.stdout:(10/33): python3-ply-3.11-14.el9.noarch.rpm 638 kB/s | 106 kB 00:00 2024-04-22T23:11:16.377 INFO:teuthology.orchestra.run.smithi134.stdout:(11/33): python3-requests-2.25.1-8.el9.noarch.r 1.3 MB/s | 125 kB 00:00 2024-04-22T23:11:16.402 INFO:teuthology.orchestra.run.smithi134.stdout:(12/33): python3-urllib3-1.26.5-5.el9.noarch.rp 2.5 MB/s | 215 kB 00:00 2024-04-22T23:11:16.644 INFO:teuthology.orchestra.run.smithi134.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 130 kB/s | 35 kB 00:00 2024-04-22T23:11:16.744 INFO:teuthology.orchestra.run.smithi134.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 149 kB/s | 51 kB 00:00 2024-04-22T23:11:16.845 INFO:teuthology.orchestra.run.smithi134.stdout:(15/33): python3-toml-0.10.2-6.el9.noarch.rpm 208 kB/s | 42 kB 00:00 2024-04-22T23:11:16.878 INFO:teuthology.orchestra.run.smithi134.stdout:(16/33): python3-mako-1.1.4-6.el9.noarch.rpm 322 kB/s | 172 kB 00:00 2024-04-22T23:11:17.037 INFO:teuthology.orchestra.run.smithi134.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 149 kB/s | 43 kB 00:00 2024-04-22T23:11:17.179 INFO:teuthology.orchestra.run.smithi134.stdout:(18/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.2 MB/s | 358 kB 00:00 2024-04-22T23:11:17.204 INFO:teuthology.orchestra.run.smithi134.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 478 kB/s | 172 kB 00:00 2024-04-22T23:11:17.221 INFO:teuthology.orchestra.run.smithi134.stdout:(20/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 58 kB/s | 11 kB 00:00 2024-04-22T23:11:17.246 INFO:teuthology.orchestra.run.smithi134.stdout:(21/33): python3-jaraco-classes-3.2.1-5.el9.noa 263 kB/s | 18 kB 00:00 2024-04-22T23:11:17.280 INFO:teuthology.orchestra.run.smithi134.stdout:(22/33): python3-jaraco-collections-3.0.0-8.el9 397 kB/s | 23 kB 00:00 2024-04-22T23:11:17.296 INFO:teuthology.orchestra.run.smithi134.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 258 kB/s | 19 kB 00:00 2024-04-22T23:11:17.313 INFO:teuthology.orchestra.run.smithi134.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 293 kB/s | 20 kB 00:00 2024-04-22T23:11:17.339 INFO:teuthology.orchestra.run.smithi134.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 787 kB/s | 46 kB 00:00 2024-04-22T23:11:17.372 INFO:teuthology.orchestra.run.smithi134.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-04-22T23:11:17.406 INFO:teuthology.orchestra.run.smithi134.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-04-22T23:11:17.423 INFO:teuthology.orchestra.run.smithi134.stdout:(28/33): python3-portend-3.1.0-2.el9.noarch.rpm 196 kB/s | 16 kB 00:00 2024-04-22T23:11:17.448 INFO:teuthology.orchestra.run.smithi134.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-04-22T23:11:17.465 INFO:teuthology.orchestra.run.smithi134.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 609 kB/s | 36 kB 00:00 2024-04-22T23:11:17.498 INFO:teuthology.orchestra.run.smithi134.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-04-22T23:11:17.540 INFO:teuthology.orchestra.run.smithi134.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-04-22T23:11:17.658 INFO:teuthology.orchestra.run.smithi134.stdout:(33/33): python3-zc-lockfile-2.0-10.el9.noarch. 217 kB/s | 20 kB 00:00 2024-04-22T23:11:17.659 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-04-22T23:11:17.659 INFO:teuthology.orchestra.run.smithi134.stdout:Total 11 MB/s | 30 MB 00:02 2024-04-22T23:11:17.673 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-04-22T23:11:17.710 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-04-22T23:11:17.710 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-04-22T23:11:17.993 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-04-22T23:11:17.994 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-04-22T23:11:19.173 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-04-22T23:11:19.227 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-04-22T23:11:19.259 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-04-22T23:11:19.307 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-04-22T23:11:19.400 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-04-22T23:11:19.479 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-04-22T23:11:19.528 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-04-22T23:11:19.563 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-04-22T23:11:19.593 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-04-22T23:11:19.627 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-04-22T23:11:19.661 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-04-22T23:11:19.701 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-04-22T23:11:19.736 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-04-22T23:11:19.762 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-04-22T23:11:19.792 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-04-22T23:11:19.865 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-04-22T23:11:20.020 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-04-22T23:11:20.099 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-04-22T23:11:20.153 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-04-22T23:11:20.191 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-04-22T23:11:20.240 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-04-22T23:11:20.295 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-04-22T23:11:20.506 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-04-22T23:11:20.562 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-04-22T23:11:20.640 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-04-22T23:11:20.729 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-04-22T23:11:20.769 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-04-22T23:11:20.884 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-04-22T23:11:21.038 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 28/33 2024-04-22T23:11:21.063 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-2:18.2.1-0.el9.x86_64 29/33 2024-04-22T23:11:21.090 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-2:18.2.1-0.el9.x86_64 29/33 2024-04-22T23:11:21.090 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T23:11:21.090 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-22T23:11:21.090 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T23:11:21.090 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T23:11:21.091 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:21.831 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-selinux-2:18.2.1-0.el9.x86_64 29/32 2024-04-22T23:11:21.831 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /sys 2024-04-22T23:11:21.831 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /proc 2024-04-22T23:11:21.831 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /mnt 2024-04-22T23:11:21.831 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /var/tmp 2024-04-22T23:11:21.831 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /home 2024-04-22T23:11:21.831 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /root 2024-04-22T23:11:21.831 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /tmp 2024-04-22T23:11:21.831 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:11:22.585 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-radosgw-2:18.2.1-0.el9.x86_64 30/32 2024-04-22T23:11:22.610 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-radosgw-2:18.2.1-0.el9.x86_64 30/32 2024-04-22T23:11:22.611 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T23:11:22.611 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-22T23:11:22.611 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T23:11:22.611 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T23:11:22.611 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:11:22.613 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-22T23:11:22.679 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-22T23:11:22.680 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-22T23:11:22.690 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-osd-2:18.2.1-0.el9.x86_64 30/33 2024-04-22T23:11:22.719 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-osd-2:18.2.1-0.el9.x86_64 30/33 2024-04-22T23:11:22.719 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T23:11:22.719 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-22T23:11:22.719 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T23:11:22.720 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T23:11:22.720 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:23.166 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mon-2:18.2.1-0.el9.x86_64 31/33 2024-04-22T23:11:23.189 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mon-2:18.2.1-0.el9.x86_64 31/33 2024-04-22T23:11:23.189 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T23:11:23.189 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-22T23:11:23.189 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T23:11:23.189 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T23:11:23.189 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:23.391 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mds-2:18.2.1-0.el9.x86_64 32/33 2024-04-22T23:11:23.415 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mds-2:18.2.1-0.el9.x86_64 32/33 2024-04-22T23:11:23.415 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T23:11:23.415 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-22T23:11:23.415 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T23:11:23.415 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T23:11:23.415 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:23.478 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-2:18.2.1-0.el9.x86_64 33/33 2024-04-22T23:11:23.836 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-22T23:11:23.836 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-base-2:18.2.1-0.el9.x86_64 1/32 2024-04-22T23:11:23.836 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-common-2:18.2.1-0.el9.x86_64 2/32 2024-04-22T23:11:23.836 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-radosgw-2:18.2.1-0.el9.x86_64 3/32 2024-04-22T23:11:23.836 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-selinux-2:18.2.1-0.el9.x86_64 4/32 2024-04-22T23:11:23.836 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs2-2:18.2.1-0.el9.x86_64 5/32 2024-04-22T23:11:23.836 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libradosstriper1-2:18.2.1-0.el9.x86_64 6/32 2024-04-22T23:11:23.837 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librgw2-2:18.2.1-0.el9.x86_64 7/32 2024-04-22T23:11:23.837 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-argparse-2:18.2.1-0.el9.x86_64 8/32 2024-04-22T23:11:23.837 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-common-2:18.2.1-0.el9.x86_64 9/32 2024-04-22T23:11:23.837 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cephfs-2:18.2.1-0.el9.x86_64 10/32 2024-04-22T23:11:23.837 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rados-2:18.2.1-0.el9.x86_64 11/32 2024-04-22T23:11:23.837 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rbd-2:18.2.1-0.el9.x86_64 12/32 2024-04-22T23:11:23.837 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rgw-2:18.2.1-0.el9.x86_64 13/32 2024-04-22T23:11:23.837 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-04-22T23:11:23.837 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-04-22T23:11:23.837 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-04-22T23:11:23.837 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-04-22T23:11:23.837 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-04-22T23:11:23.838 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-04-22T23:11:23.838 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-04-22T23:11:23.838 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-04-22T23:11:23.838 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-04-22T23:11:23.838 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-04-22T23:11:23.838 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-04-22T23:11:23.838 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-04-22T23:11:23.838 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-04-22T23:11:23.838 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-04-22T23:11:23.838 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-04-22T23:11:23.838 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-2:18.2.1-0.el9.x86_64 29/32 2024-04-22T23:11:23.838 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-04-22T23:11:23.839 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-2:18.2.1-0.el9.x86_64 31/32 2024-04-22T23:11:24.578 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-04-22T23:11:24.578 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:11:24.578 INFO:teuthology.orchestra.run.smithi067.stdout:Upgraded: 2024-04-22T23:11:24.578 INFO:teuthology.orchestra.run.smithi067.stdout: librados2-2:18.2.1-0.el9.x86_64 librbd1-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:24.578 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-22T23:11:24.579 INFO:teuthology.orchestra.run.smithi067.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-22T23:11:24.579 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:24.579 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:24.579 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:24.579 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:24.579 INFO:teuthology.orchestra.run.smithi067.stdout: fmt-8.1.1-5.el9.x86_64 2024-04-22T23:11:24.579 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-22T23:11:24.579 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-22T23:11:24.579 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-22T23:11:24.579 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:24.579 INFO:teuthology.orchestra.run.smithi067.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-22T23:11:24.579 INFO:teuthology.orchestra.run.smithi067.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-22T23:11:24.579 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-22T23:11:24.579 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:24.580 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-22T23:11:24.580 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:24.580 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-22T23:11:24.580 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-22T23:11:24.580 INFO:teuthology.orchestra.run.smithi067.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-22T23:11:24.580 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:24.580 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:24.580 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:24.580 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-22T23:11:24.580 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:24.580 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:24.580 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:24.580 INFO:teuthology.orchestra.run.smithi067.stdout: re2-1:20211101-3.el9.x86_64 2024-04-22T23:11:24.580 INFO:teuthology.orchestra.run.smithi067.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-22T23:11:24.580 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:11:24.580 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-22T23:11:24.850 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-test 2024-04-22T23:11:25.067 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-2:18.2.1-0.el9.x86_64 33/33 2024-04-22T23:11:25.068 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-2:18.2.1-0.el9.x86_64 1/33 2024-04-22T23:11:25.068 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mds-2:18.2.1-0.el9.x86_64 2/33 2024-04-22T23:11:25.068 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-2:18.2.1-0.el9.x86_64 3/33 2024-04-22T23:11:25.068 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mon-2:18.2.1-0.el9.x86_64 4/33 2024-04-22T23:11:25.068 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-osd-2:18.2.1-0.el9.x86_64 5/33 2024-04-22T23:11:25.068 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 6/33 2024-04-22T23:11:25.068 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-04-22T23:11:25.068 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-04-22T23:11:25.068 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-04-22T23:11:25.068 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-04-22T23:11:25.068 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-04-22T23:11:25.068 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-04-22T23:11:25.069 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-04-22T23:11:25.069 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-04-22T23:11:25.069 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-04-22T23:11:25.069 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-04-22T23:11:25.069 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-04-22T23:11:25.069 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-22T23:11:25.069 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-04-22T23:11:25.069 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-04-22T23:11:25.069 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-04-22T23:11:25.069 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-04-22T23:11:25.069 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-04-22T23:11:25.069 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-04-22T23:11:25.069 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-04-22T23:11:25.069 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-04-22T23:11:25.069 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-22T23:11:25.069 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-04-22T23:11:25.070 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-04-22T23:11:25.070 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-04-22T23:11:25.070 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-04-22T23:11:25.070 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-04-22T23:11:25.475 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:26 ago on Mon 22 Apr 2024 11:10:59 PM UTC. 2024-04-22T23:11:25.598 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-22T23:11:25.598 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:11:25.598 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-04-22T23:11:25.598 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:11:25.599 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-22T23:11:25.599 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test x86_64 2:18.2.1-0.el9 ceph 40 M 2024-04-22T23:11:25.599 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-04-22T23:11:25.599 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite x86_64 2:18.2.1-0.el9 ceph 165 k 2024-04-22T23:11:25.599 INFO:teuthology.orchestra.run.smithi067.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-22T23:11:25.599 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-22T23:11:25.599 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:11:25.599 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-22T23:11:25.599 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:11:25.599 INFO:teuthology.orchestra.run.smithi067.stdout:Install 4 Packages 2024-04-22T23:11:25.599 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:11:25.600 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 40 M 2024-04-22T23:11:25.600 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 166 M 2024-04-22T23:11:25.600 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-22T23:11:26.058 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-04-22T23:11:26.058 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:26.058 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-04-22T23:11:26.058 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:26.059 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:26.059 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:26.059 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 2024-04-22T23:11:26.059 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:26.059 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:26.059 INFO:teuthology.orchestra.run.smithi134.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-22T23:11:26.059 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-22T23:11:26.059 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-22T23:11:26.059 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-22T23:11:26.059 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-22T23:11:26.059 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-22T23:11:26.059 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-22T23:11:26.059 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-22T23:11:26.059 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-22T23:11:26.060 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-22T23:11:26.060 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-22T23:11:26.060 INFO:teuthology.orchestra.run.smithi134.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-22T23:11:26.060 INFO:teuthology.orchestra.run.smithi134.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-22T23:11:26.060 INFO:teuthology.orchestra.run.smithi134.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-22T23:11:26.060 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-22T23:11:26.060 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ply-3.11-14.el9.noarch 2024-04-22T23:11:26.060 INFO:teuthology.orchestra.run.smithi134.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-22T23:11:26.060 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-22T23:11:26.060 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-22T23:11:26.060 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-22T23:11:26.060 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-22T23:11:26.060 INFO:teuthology.orchestra.run.smithi134.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-22T23:11:26.060 INFO:teuthology.orchestra.run.smithi134.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-22T23:11:26.060 INFO:teuthology.orchestra.run.smithi134.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-22T23:11:26.061 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-22T23:11:26.061 INFO:teuthology.orchestra.run.smithi134.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-22T23:11:26.061 INFO:teuthology.orchestra.run.smithi134.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-22T23:11:26.061 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:26.061 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-22T23:11:26.063 INFO:teuthology.orchestra.run.smithi067.stdout:(1/4): libcephsqlite-18.2.1-0.el9.x86_64.rpm 882 kB/s | 165 kB 00:00 2024-04-22T23:11:26.419 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-base 2024-04-22T23:11:26.600 INFO:teuthology.orchestra.run.smithi067.stdout:(2/4): ceph-test-18.2.1-0.el9.x86_64.rpm 55 MB/s | 40 MB 00:00 2024-04-22T23:11:26.650 INFO:teuthology.orchestra.run.smithi067.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 109 kB/s | 64 kB 00:00 2024-04-22T23:11:26.939 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:46 ago on Mon 22 Apr 2024 11:10:40 PM UTC. 2024-04-22T23:11:26.945 INFO:teuthology.orchestra.run.smithi067.stdout:(4/4): socat-1.7.4.1-5.el9.x86_64.rpm 285 kB/s | 305 kB 00:01 2024-04-22T23:11:26.947 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-22T23:11:26.947 INFO:teuthology.orchestra.run.smithi067.stdout:Total 30 MB/s | 40 MB 00:01 2024-04-22T23:11:26.974 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-22T23:11:26.996 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-22T23:11:26.996 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-22T23:11:27.018 INFO:teuthology.orchestra.run.smithi134.stdout:Package ceph-base-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-22T23:11:27.058 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-22T23:11:27.060 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-04-22T23:11:27.060 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-22T23:11:27.131 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install cephadm 2024-04-22T23:11:27.329 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-22T23:11:27.329 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-22T23:11:27.640 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:47 ago on Mon 22 Apr 2024 11:10:40 PM UTC. 2024-04-22T23:11:27.760 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-22T23:11:27.760 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:11:27.761 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-04-22T23:11:27.761 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:11:27.761 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-04-22T23:11:27.761 INFO:teuthology.orchestra.run.smithi134.stdout: cephadm noarch 2:18.2.1-0.el9 ceph-noarch 221 k 2024-04-22T23:11:27.761 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:27.761 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-04-22T23:11:27.761 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:11:27.761 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-04-22T23:11:27.761 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:27.762 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 221 k 2024-04-22T23:11:27.762 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 213 k 2024-04-22T23:11:27.762 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-04-22T23:11:27.832 INFO:teuthology.orchestra.run.smithi134.stdout:cephadm-18.2.1-0.el9.noarch.rpm 3.1 MB/s | 221 kB 00:00 2024-04-22T23:11:27.833 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-04-22T23:11:27.833 INFO:teuthology.orchestra.run.smithi134.stdout:Total 3.0 MB/s | 221 kB 00:00 2024-04-22T23:11:27.833 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-04-22T23:11:27.838 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-04-22T23:11:27.838 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-04-22T23:11:27.840 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-04-22T23:11:27.841 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-04-22T23:11:27.866 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-22T23:11:27.910 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-04-22T23:11:27.961 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-04-22T23:11:28.009 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-04-22T23:11:28.036 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephsqlite-2:18.2.1-0.el9.x86_64 3/4 2024-04-22T23:11:28.368 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: cephadm-2:18.2.1-0.el9.noarch 1/1 2024-04-22T23:11:28.451 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : cephadm-2:18.2.1-0.el9.noarch 1/1 2024-04-22T23:11:28.782 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: cephadm-2:18.2.1-0.el9.noarch 1/1 2024-04-22T23:11:29.032 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : cephadm-2:18.2.1-0.el9.noarch 1/1 2024-04-22T23:11:29.032 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:29.032 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-04-22T23:11:29.032 INFO:teuthology.orchestra.run.smithi134.stdout: cephadm-2:18.2.1-0.el9.noarch 2024-04-22T23:11:29.032 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:29.032 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-22T23:11:29.219 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-immutable-object-cache 2024-04-22T23:11:29.729 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:49 ago on Mon 22 Apr 2024 11:10:40 PM UTC. 2024-04-22T23:11:29.849 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-22T23:11:29.850 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:11:29.850 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-04-22T23:11:29.850 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:11:29.850 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-04-22T23:11:29.850 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache x86_64 2:18.2.1-0.el9 ceph 142 k 2024-04-22T23:11:29.850 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:29.850 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-04-22T23:11:29.850 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:11:29.850 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-04-22T23:11:29.850 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:29.850 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 142 k 2024-04-22T23:11:29.850 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 395 k 2024-04-22T23:11:29.850 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-04-22T23:11:29.963 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-immutable-object-cache-18.2.1-0.el9.x86_64 1.2 MB/s | 142 kB 00:00 2024-04-22T23:11:29.963 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-04-22T23:11:29.964 INFO:teuthology.orchestra.run.smithi134.stdout:Total 1.2 MB/s | 142 kB 00:00 2024-04-22T23:11:29.964 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-04-22T23:11:29.972 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-04-22T23:11:29.972 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-04-22T23:11:30.031 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-04-22T23:11:30.031 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-04-22T23:11:30.180 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-04-22T23:11:30.264 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T23:11:30.288 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T23:11:30.288 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T23:11:30.288 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-22T23:11:30.289 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:31.128 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T23:11:31.128 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:31.129 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-04-22T23:11:31.129 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:31.129 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:31.129 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-22T23:11:31.333 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr 2024-04-22T23:11:31.846 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:51 ago on Mon 22 Apr 2024 11:10:40 PM UTC. 2024-04-22T23:11:31.927 INFO:teuthology.orchestra.run.smithi134.stdout:Package ceph-mgr-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-22T23:11:31.933 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephsqlite-2:18.2.1-0.el9.x86_64 3/4 2024-04-22T23:11:31.970 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-22T23:11:31.971 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-04-22T23:11:31.971 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-22T23:11:32.012 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-test-2:18.2.1-0.el9.x86_64 4/4 2024-04-22T23:11:32.034 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-dashboard 2024-04-22T23:11:32.543 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:52 ago on Mon 22 Apr 2024 11:10:40 PM UTC. 2024-04-22T23:11:32.663 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-22T23:11:32.663 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:11:32.664 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-04-22T23:11:32.664 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:11:32.664 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-04-22T23:11:32.664 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-dashboard noarch 2:18.2.1-0.el9 ceph-noarch 1.8 M 2024-04-22T23:11:32.664 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-04-22T23:11:32.664 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-grafana-dashboards noarch 2:18.2.1-0.el9 ceph-noarch 23 k 2024-04-22T23:11:32.664 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-prometheus-alerts noarch 2:18.2.1-0.el9 ceph-noarch 15 k 2024-04-22T23:11:32.664 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-04-22T23:11:32.664 INFO:teuthology.orchestra.run.smithi134.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-22T23:11:32.664 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-22T23:11:32.664 INFO:teuthology.orchestra.run.smithi134.stdout:Installing weak dependencies: 2024-04-22T23:11:32.664 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-04-22T23:11:32.664 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:32.664 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-04-22T23:11:32.665 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:11:32.665 INFO:teuthology.orchestra.run.smithi134.stdout:Install 7 Packages 2024-04-22T23:11:32.665 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:32.665 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 2.1 M 2024-04-22T23:11:32.665 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 8.5 M 2024-04-22T23:11:32.665 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-04-22T23:11:32.925 INFO:teuthology.orchestra.run.smithi134.stdout:(1/7): ceph-prometheus-alerts-18.2.1-0.el9.noar 126 kB/s | 15 kB 00:00 2024-04-22T23:11:32.950 INFO:teuthology.orchestra.run.smithi134.stdout:(2/7): ceph-grafana-dashboards-18.2.1-0.el9.noa 163 kB/s | 23 kB 00:00 2024-04-22T23:11:32.992 INFO:teuthology.orchestra.run.smithi134.stdout:(3/7): ceph-mgr-dashboard-18.2.1-0.el9.noarch.r 9.6 MB/s | 1.8 MB 00:00 2024-04-22T23:11:33.151 INFO:teuthology.orchestra.run.smithi134.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 40 kB/s | 9.0 kB 00:00 2024-04-22T23:11:33.285 INFO:teuthology.orchestra.run.smithi134.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 122 kB/s | 41 kB 00:00 2024-04-22T23:11:33.310 INFO:teuthology.orchestra.run.smithi134.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 97 kB/s | 31 kB 00:00 2024-04-22T23:11:33.502 INFO:teuthology.orchestra.run.smithi134.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 538 kB/s | 188 kB 00:00 2024-04-22T23:11:33.504 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-04-22T23:11:33.504 INFO:teuthology.orchestra.run.smithi134.stdout:Total 2.4 MB/s | 2.1 MB 00:00 2024-04-22T23:11:33.505 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-04-22T23:11:33.513 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-04-22T23:11:33.513 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-04-22T23:11:33.565 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-04-22T23:11:33.567 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-04-22T23:11:33.856 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-04-22T23:11:33.920 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-04-22T23:11:33.954 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-04-22T23:11:34.000 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-04-22T23:11:34.015 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-test-2:18.2.1-0.el9.x86_64 4/4 2024-04-22T23:11:34.015 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-test-2:18.2.1-0.el9.x86_64 1/4 2024-04-22T23:11:34.015 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephsqlite-2:18.2.1-0.el9.x86_64 2/4 2024-04-22T23:11:34.015 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-04-22T23:11:34.033 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-04-22T23:11:34.061 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 5/7 2024-04-22T23:11:34.297 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 6/7 2024-04-22T23:11:34.330 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 7/7 2024-04-22T23:11:34.402 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-04-22T23:11:34.402 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:11:34.402 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-22T23:11:34.402 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test-2:18.2.1-0.el9.x86_64 libcephsqlite-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:34.402 INFO:teuthology.orchestra.run.smithi067.stdout: socat-1.7.4.1-5.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2024-04-22T23:11:34.402 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:11:34.403 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-22T23:11:34.669 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph 2024-04-22T23:11:34.865 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 7/7 2024-04-22T23:11:34.865 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 1/7 2024-04-22T23:11:34.865 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 2/7 2024-04-22T23:11:34.865 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 3/7 2024-04-22T23:11:34.865 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-04-22T23:11:34.865 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-04-22T23:11:34.865 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-04-22T23:11:35.117 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-04-22T23:11:35.117 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:35.117 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-04-22T23:11:35.117 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 2024-04-22T23:11:35.117 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 2024-04-22T23:11:35.118 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 2024-04-22T23:11:35.118 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-04-22T23:11:35.118 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-04-22T23:11:35.118 INFO:teuthology.orchestra.run.smithi134.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-22T23:11:35.118 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-22T23:11:35.118 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:35.118 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-22T23:11:35.189 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:36 ago on Mon 22 Apr 2024 11:10:59 PM UTC. 2024-04-22T23:11:35.316 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-22T23:11:35.318 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:11:35.318 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-04-22T23:11:35.318 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:11:35.319 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-22T23:11:35.319 INFO:teuthology.orchestra.run.smithi067.stdout: ceph x86_64 2:18.2.1-0.el9 ceph 6.4 k 2024-04-22T23:11:35.319 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-04-22T23:11:35.319 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds x86_64 2:18.2.1-0.el9 ceph 2.1 M 2024-04-22T23:11:35.319 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr x86_64 2:18.2.1-0.el9 ceph 1.4 M 2024-04-22T23:11:35.319 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core noarch 2:18.2.1-0.el9 ceph-noarch 242 k 2024-04-22T23:11:35.319 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon x86_64 2:18.2.1-0.el9 ceph 4.4 M 2024-04-22T23:11:35.319 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd x86_64 2:18.2.1-0.el9 ceph 18 M 2024-04-22T23:11:35.319 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-22T23:11:35.319 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-22T23:11:35.319 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-22T23:11:35.319 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-22T23:11:35.320 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-22T23:11:35.320 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-22T23:11:35.320 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-22T23:11:35.320 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-22T23:11:35.320 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-22T23:11:35.320 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-22T23:11:35.320 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-22T23:11:35.320 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-22T23:11:35.320 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-22T23:11:35.320 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-22T23:11:35.320 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-22T23:11:35.320 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-22T23:11:35.320 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-22T23:11:35.320 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-22T23:11:35.320 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-22T23:11:35.321 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-22T23:11:35.321 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-22T23:11:35.321 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-22T23:11:35.321 INFO:teuthology.orchestra.run.smithi067.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-22T23:11:35.321 INFO:teuthology.orchestra.run.smithi067.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-22T23:11:35.321 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-22T23:11:35.321 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-22T23:11:35.321 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-22T23:11:35.321 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:11:35.321 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-22T23:11:35.322 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:11:35.322 INFO:teuthology.orchestra.run.smithi067.stdout:Install 33 Packages 2024-04-22T23:11:35.322 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:11:35.323 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 30 M 2024-04-22T23:11:35.323 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 110 M 2024-04-22T23:11:35.323 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-22T23:11:35.327 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-22T23:11:35.832 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:55 ago on Mon 22 Apr 2024 11:10:40 PM UTC. 2024-04-22T23:11:35.953 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-22T23:11:35.955 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:11:35.955 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-04-22T23:11:35.955 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:11:35.955 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-04-22T23:11:35.955 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.1-0.el9 ceph-noarch 7.4 M 2024-04-22T23:11:35.955 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-04-22T23:11:35.955 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-22T23:11:35.955 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-22T23:11:35.955 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-22T23:11:35.955 INFO:teuthology.orchestra.run.smithi134.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-22T23:11:35.955 INFO:teuthology.orchestra.run.smithi134.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-22T23:11:35.956 INFO:teuthology.orchestra.run.smithi134.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-22T23:11:35.956 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-22T23:11:35.956 INFO:teuthology.orchestra.run.smithi134.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-22T23:11:35.956 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-22T23:11:35.956 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-22T23:11:35.956 INFO:teuthology.orchestra.run.smithi134.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-22T23:11:35.956 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:35.956 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-04-22T23:11:35.956 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:11:35.956 INFO:teuthology.orchestra.run.smithi134.stdout:Install 12 Packages 2024-04-22T23:11:35.956 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:35.957 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 38 M 2024-04-22T23:11:35.957 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 209 M 2024-04-22T23:11:35.957 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-04-22T23:11:35.960 INFO:teuthology.orchestra.run.smithi067.stdout:(1/33): ceph-18.2.1-0.el9.x86_64.rpm 57 kB/s | 6.4 kB 00:00 2024-04-22T23:11:36.011 INFO:teuthology.orchestra.run.smithi067.stdout:(2/33): ceph-mgr-18.2.1-0.el9.x86_64.rpm 8.9 MB/s | 1.4 MB 00:00 2024-04-22T23:11:36.069 INFO:teuthology.orchestra.run.smithi067.stdout:(3/33): ceph-mds-18.2.1-0.el9.x86_64.rpm 9.5 MB/s | 2.1 MB 00:00 2024-04-22T23:11:36.094 INFO:teuthology.orchestra.run.smithi067.stdout:(4/33): ceph-mgr-modules-core-18.2.1-0.el9.noar 9.5 MB/s | 242 kB 00:00 2024-04-22T23:11:36.178 INFO:teuthology.orchestra.run.smithi067.stdout:(5/33): ceph-mon-18.2.1-0.el9.x86_64.rpm 20 MB/s | 4.4 MB 00:00 2024-04-22T23:11:36.470 INFO:teuthology.orchestra.run.smithi067.stdout:(6/33): ceph-osd-18.2.1-0.el9.x86_64.rpm 38 MB/s | 18 MB 00:00 2024-04-22T23:11:36.517 INFO:teuthology.orchestra.run.smithi134.stdout:(1/12): ceph-mgr-diskprediction-local-18.2.1-0. 43 MB/s | 7.4 MB 00:00 2024-04-22T23:11:36.562 INFO:teuthology.orchestra.run.smithi067.stdout:(7/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 542 kB/s | 253 kB 00:00 2024-04-22T23:11:36.608 INFO:teuthology.orchestra.run.smithi134.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 358 kB/s | 33 kB 00:00 2024-04-22T23:11:36.628 INFO:teuthology.orchestra.run.smithi067.stdout:(8/33): python3-pycparser-2.20-6.el9.noarch.rpm 2.0 MB/s | 135 kB 00:00 2024-04-22T23:11:36.634 INFO:teuthology.orchestra.run.smithi134.stdout:(3/12): libquadmath-11.4.1-3.el9.x86_64.rpm 675 kB/s | 195 kB 00:00 2024-04-22T23:11:36.659 INFO:teuthology.orchestra.run.smithi134.stdout:(4/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 803 kB 00:00 2024-04-22T23:11:36.679 INFO:teuthology.orchestra.run.smithi067.stdout:(9/33): python3-cryptography-36.0.1-4.el9.x86_6 2.4 MB/s | 1.2 MB 00:00 2024-04-22T23:11:36.693 INFO:teuthology.orchestra.run.smithi134.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 310 kB/s | 18 kB 00:00 2024-04-22T23:11:36.704 INFO:teuthology.orchestra.run.smithi067.stdout:(10/33): python3-ply-3.11-14.el9.noarch.rpm 457 kB/s | 106 kB 00:00 2024-04-22T23:11:36.710 INFO:teuthology.orchestra.run.smithi134.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 700 kB/s | 35 kB 00:00 2024-04-22T23:11:36.729 INFO:teuthology.orchestra.run.smithi067.stdout:(11/33): python3-requests-2.25.1-8.el9.noarch.r 1.2 MB/s | 125 kB 00:00 2024-04-22T23:11:36.755 INFO:teuthology.orchestra.run.smithi067.stdout:(12/33): python3-urllib3-1.26.5-5.el9.noarch.rp 2.8 MB/s | 215 kB 00:00 2024-04-22T23:11:36.802 INFO:teuthology.orchestra.run.smithi134.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 15 MB/s | 3.0 MB 00:00 2024-04-22T23:11:36.813 INFO:teuthology.orchestra.run.smithi067.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 416 kB/s | 35 kB 00:00 2024-04-22T23:11:36.838 INFO:teuthology.orchestra.run.smithi067.stdout:(14/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-04-22T23:11:36.844 INFO:teuthology.orchestra.run.smithi134.stdout:(8/12): python3-devel-3.9.18-3.el9.x86_64.rpm 1.8 MB/s | 244 kB 00:00 2024-04-22T23:11:36.855 INFO:teuthology.orchestra.run.smithi067.stdout:(15/33): python3-toml-0.10.2-6.el9.noarch.rpm 997 kB/s | 42 kB 00:00 2024-04-22T23:11:36.872 INFO:teuthology.orchestra.run.smithi067.stdout:(16/33): python3-pytz-2021.1-5.el9.noarch.rpm 435 kB/s | 51 kB 00:00 2024-04-22T23:11:36.952 INFO:teuthology.orchestra.run.smithi134.stdout:(9/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2024-04-22T23:11:37.003 INFO:teuthology.orchestra.run.smithi134.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 1.5 MB/s | 250 kB 00:00 2024-04-22T23:11:37.081 INFO:teuthology.orchestra.run.smithi067.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 179 kB/s | 43 kB 00:00 2024-04-22T23:11:37.119 INFO:teuthology.orchestra.run.smithi134.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 5.1 MB 00:00 2024-04-22T23:11:37.148 INFO:teuthology.orchestra.run.smithi067.stdout:(18/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-04-22T23:11:37.164 INFO:teuthology.orchestra.run.smithi067.stdout:(19/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 128 kB/s | 11 kB 00:00 2024-04-22T23:11:37.215 INFO:teuthology.orchestra.run.smithi067.stdout:(20/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 478 kB/s | 172 kB 00:00 2024-04-22T23:11:37.232 INFO:teuthology.orchestra.run.smithi067.stdout:(21/33): python3-jaraco-classes-3.2.1-5.el9.noa 211 kB/s | 18 kB 00:00 2024-04-22T23:11:37.382 INFO:teuthology.orchestra.run.smithi067.stdout:(22/33): python3-jaraco-collections-3.0.0-8.el9 106 kB/s | 23 kB 00:00 2024-04-22T23:11:37.407 INFO:teuthology.orchestra.run.smithi067.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 101 kB/s | 19 kB 00:00 2024-04-22T23:11:37.441 INFO:teuthology.orchestra.run.smithi067.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 336 kB/s | 20 kB 00:00 2024-04-22T23:11:37.483 INFO:teuthology.orchestra.run.smithi067.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 461 kB/s | 46 kB 00:00 2024-04-22T23:11:37.508 INFO:teuthology.orchestra.run.smithi067.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 786 kB/s | 79 kB 00:00 2024-04-22T23:11:37.558 INFO:teuthology.orchestra.run.smithi067.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-04-22T23:11:37.578 INFO:teuthology.orchestra.run.smithi134.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 26 MB/s | 16 MB 00:00 2024-04-22T23:11:37.579 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-04-22T23:11:37.579 INFO:teuthology.orchestra.run.smithi134.stdout:Total 23 MB/s | 38 MB 00:01 2024-04-22T23:11:37.584 INFO:teuthology.orchestra.run.smithi067.stdout:(28/33): python3-portend-3.1.0-2.el9.noarch.rpm 163 kB/s | 16 kB 00:00 2024-04-22T23:11:37.609 INFO:teuthology.orchestra.run.smithi067.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 895 kB/s | 90 kB 00:00 2024-04-22T23:11:37.634 INFO:teuthology.orchestra.run.smithi067.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 473 kB/s | 36 kB 00:00 2024-04-22T23:11:37.701 INFO:teuthology.orchestra.run.smithi067.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-04-22T23:11:37.726 INFO:teuthology.orchestra.run.smithi067.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-04-22T23:11:37.752 INFO:teuthology.orchestra.run.smithi067.stdout:(33/33): python3-zc-lockfile-2.0-10.el9.noarch. 171 kB/s | 20 kB 00:00 2024-04-22T23:11:37.759 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-22T23:11:37.759 INFO:teuthology.orchestra.run.smithi067.stdout:Total 12 MB/s | 30 MB 00:02 2024-04-22T23:11:37.876 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-22T23:11:37.916 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-22T23:11:37.917 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-22T23:11:37.927 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-04-22T23:11:37.953 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-04-22T23:11:37.953 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-04-22T23:11:38.220 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-22T23:11:38.220 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-22T23:11:38.361 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-04-22T23:11:38.362 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-04-22T23:11:39.122 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-04-22T23:11:39.191 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-22T23:11:39.234 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-22T23:11:39.335 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-22T23:11:39.377 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-22T23:11:39.400 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-22T23:11:39.463 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-04-22T23:11:39.496 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-04-22T23:11:39.529 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-04-22T23:11:39.545 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-22T23:11:39.576 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-22T23:11:39.625 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-04-22T23:11:39.652 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-22T23:11:39.701 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-04-22T23:11:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-04-22T23:11:39.791 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-04-22T23:11:39.821 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-04-22T23:11:39.855 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-04-22T23:11:39.888 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-04-22T23:11:39.929 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-04-22T23:11:39.973 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-04-22T23:11:40.007 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-04-22T23:11:40.045 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-04-22T23:11:40.122 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-04-22T23:11:40.273 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-04-22T23:11:40.290 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-22T23:11:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-04-22T23:11:40.413 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-22T23:11:40.415 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-04-22T23:11:40.453 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-04-22T23:11:40.512 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-04-22T23:11:40.576 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-04-22T23:11:40.771 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-04-22T23:11:40.816 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-04-22T23:11:40.898 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-04-22T23:11:40.985 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-04-22T23:11:41.023 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-04-22T23:11:41.146 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-04-22T23:11:41.307 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 28/33 2024-04-22T23:11:41.333 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-2:18.2.1-0.el9.x86_64 29/33 2024-04-22T23:11:41.358 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-2:18.2.1-0.el9.x86_64 29/33 2024-04-22T23:11:41.358 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T23:11:41.358 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-22T23:11:41.358 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T23:11:41.358 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T23:11:41.358 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:11:41.624 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-22T23:11:42.659 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-22T23:11:42.691 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 12/12 2024-04-22T23:11:42.968 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-osd-2:18.2.1-0.el9.x86_64 30/33 2024-04-22T23:11:42.998 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-osd-2:18.2.1-0.el9.x86_64 30/33 2024-04-22T23:11:42.998 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T23:11:42.998 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-22T23:11:42.998 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T23:11:42.999 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T23:11:42.999 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:11:43.469 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mon-2:18.2.1-0.el9.x86_64 31/33 2024-04-22T23:11:43.494 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mon-2:18.2.1-0.el9.x86_64 31/33 2024-04-22T23:11:43.494 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T23:11:43.494 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-22T23:11:43.494 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T23:11:43.495 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T23:11:43.495 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:11:43.703 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mds-2:18.2.1-0.el9.x86_64 32/33 2024-04-22T23:11:43.727 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mds-2:18.2.1-0.el9.x86_64 32/33 2024-04-22T23:11:43.727 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T23:11:43.727 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-22T23:11:43.728 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T23:11:43.728 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T23:11:43.728 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:11:43.793 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-2:18.2.1-0.el9.x86_64 33/33 2024-04-22T23:11:45.283 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 12/12 2024-04-22T23:11:45.283 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 1/12 2024-04-22T23:11:45.283 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-22T23:11:45.284 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-22T23:11:45.284 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-22T23:11:45.284 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-22T23:11:45.284 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-22T23:11:45.284 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-22T23:11:45.284 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-22T23:11:45.284 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-22T23:11:45.284 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-22T23:11:45.284 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-22T23:11:45.500 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-2:18.2.1-0.el9.x86_64 33/33 2024-04-22T23:11:45.500 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-2:18.2.1-0.el9.x86_64 1/33 2024-04-22T23:11:45.500 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mds-2:18.2.1-0.el9.x86_64 2/33 2024-04-22T23:11:45.501 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-2:18.2.1-0.el9.x86_64 3/33 2024-04-22T23:11:45.501 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mon-2:18.2.1-0.el9.x86_64 4/33 2024-04-22T23:11:45.501 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-osd-2:18.2.1-0.el9.x86_64 5/33 2024-04-22T23:11:45.501 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 6/33 2024-04-22T23:11:45.501 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-04-22T23:11:45.501 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-04-22T23:11:45.501 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-04-22T23:11:45.501 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-04-22T23:11:45.501 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-04-22T23:11:45.501 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-04-22T23:11:45.501 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-04-22T23:11:45.501 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-04-22T23:11:45.501 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-04-22T23:11:45.501 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-04-22T23:11:45.501 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-04-22T23:11:45.502 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-22T23:11:45.502 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-04-22T23:11:45.502 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-04-22T23:11:45.502 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-04-22T23:11:45.502 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-04-22T23:11:45.502 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-04-22T23:11:45.502 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-04-22T23:11:45.502 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-04-22T23:11:45.502 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-04-22T23:11:45.502 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-22T23:11:45.502 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-04-22T23:11:45.502 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-04-22T23:11:45.502 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-04-22T23:11:45.502 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-04-22T23:11:45.503 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-04-22T23:11:45.692 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-22T23:11:45.692 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:45.692 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-04-22T23:11:45.692 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarch 2024-04-22T23:11:45.693 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-22T23:11:45.693 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-22T23:11:45.693 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-22T23:11:45.693 INFO:teuthology.orchestra.run.smithi134.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-22T23:11:45.693 INFO:teuthology.orchestra.run.smithi134.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-22T23:11:45.693 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-22T23:11:45.693 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-22T23:11:45.693 INFO:teuthology.orchestra.run.smithi134.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-22T23:11:45.693 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-22T23:11:45.693 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-22T23:11:45.693 INFO:teuthology.orchestra.run.smithi134.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-22T23:11:45.693 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:45.693 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-22T23:11:46.094 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-rook 2024-04-22T23:11:46.347 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-04-22T23:11:46.347 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:11:46.347 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-22T23:11:46.347 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:46.347 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:46.348 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:46.348 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core-2:18.2.1-0.el9.noarch 2024-04-22T23:11:46.348 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:46.348 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:46.348 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-22T23:11:46.348 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-22T23:11:46.348 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-22T23:11:46.348 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-22T23:11:46.348 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-22T23:11:46.348 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-22T23:11:46.348 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-22T23:11:46.348 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-22T23:11:46.348 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-22T23:11:46.348 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-22T23:11:46.349 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-22T23:11:46.349 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-22T23:11:46.349 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-22T23:11:46.349 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-22T23:11:46.349 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-22T23:11:46.349 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ply-3.11-14.el9.noarch 2024-04-22T23:11:46.349 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-22T23:11:46.349 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-22T23:11:46.349 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-22T23:11:46.349 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-22T23:11:46.349 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-22T23:11:46.349 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-22T23:11:46.349 INFO:teuthology.orchestra.run.smithi067.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-22T23:11:46.349 INFO:teuthology.orchestra.run.smithi067.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-22T23:11:46.349 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-22T23:11:46.350 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-22T23:11:46.350 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-22T23:11:46.350 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:11:46.350 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-22T23:11:46.609 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:06 ago on Mon 22 Apr 2024 11:10:40 PM UTC. 2024-04-22T23:11:46.701 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-base 2024-04-22T23:11:46.729 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-22T23:11:46.730 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:11:46.730 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-04-22T23:11:46.730 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:11:46.731 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-04-22T23:11:46.731 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-rook noarch 2:18.2.1-0.el9 ceph-noarch 50 k 2024-04-22T23:11:46.731 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-04-22T23:11:46.731 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-22T23:11:46.731 INFO:teuthology.orchestra.run.smithi134.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-22T23:11:46.731 INFO:teuthology.orchestra.run.smithi134.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-22T23:11:46.731 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-22T23:11:46.731 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-22T23:11:46.731 INFO:teuthology.orchestra.run.smithi134.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-22T23:11:46.731 INFO:teuthology.orchestra.run.smithi134.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-22T23:11:46.731 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-22T23:11:46.731 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-22T23:11:46.731 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-22T23:11:46.732 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-22T23:11:46.732 INFO:teuthology.orchestra.run.smithi134.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-22T23:11:46.732 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:46.732 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-04-22T23:11:46.732 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:11:46.732 INFO:teuthology.orchestra.run.smithi134.stdout:Install 13 Packages 2024-04-22T23:11:46.732 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:46.733 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 2.2 M 2024-04-22T23:11:46.733 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 26 M 2024-04-22T23:11:46.733 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-04-22T23:11:46.933 INFO:teuthology.orchestra.run.smithi134.stdout:(1/13): ceph-mgr-rook-18.2.1-0.el9.noarch.rpm 893 kB/s | 50 kB 00:00 2024-04-22T23:11:47.245 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:48 ago on Mon 22 Apr 2024 11:10:59 PM UTC. 2024-04-22T23:11:47.325 INFO:teuthology.orchestra.run.smithi067.stdout:Package ceph-base-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-22T23:11:47.368 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-22T23:11:47.369 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-04-22T23:11:47.369 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-22T23:11:47.433 INFO:teuthology.orchestra.run.smithi134.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 34 kB/s | 19 kB 00:00 2024-04-22T23:11:47.445 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install cephadm 2024-04-22T23:11:47.458 INFO:teuthology.orchestra.run.smithi134.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 44 kB/s | 26 kB 00:00 2024-04-22T23:11:47.842 INFO:teuthology.orchestra.run.smithi134.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 244 kB/s | 222 kB 00:00 2024-04-22T23:11:47.980 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:48 ago on Mon 22 Apr 2024 11:10:59 PM UTC. 2024-04-22T23:11:48.105 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-22T23:11:48.106 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:11:48.107 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-04-22T23:11:48.107 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:11:48.107 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-22T23:11:48.107 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm noarch 2:18.2.1-0.el9 ceph-noarch 221 k 2024-04-22T23:11:48.107 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:11:48.107 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-22T23:11:48.107 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:11:48.107 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-04-22T23:11:48.107 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:11:48.107 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 221 k 2024-04-22T23:11:48.107 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 213 k 2024-04-22T23:11:48.107 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-22T23:11:48.185 INFO:teuthology.orchestra.run.smithi067.stdout:cephadm-18.2.1-0.el9.noarch.rpm 2.8 MB/s | 221 kB 00:00 2024-04-22T23:11:48.186 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-22T23:11:48.186 INFO:teuthology.orchestra.run.smithi067.stdout:Total 2.7 MB/s | 221 kB 00:00 2024-04-22T23:11:48.186 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-22T23:11:48.191 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-22T23:11:48.191 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-22T23:11:48.194 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-22T23:11:48.194 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-22T23:11:48.242 INFO:teuthology.orchestra.run.smithi134.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 196 kB/s | 159 kB 00:00 2024-04-22T23:11:48.267 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-22T23:11:48.326 INFO:teuthology.orchestra.run.smithi134.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 388 kB/s | 32 kB 00:00 2024-04-22T23:11:48.359 INFO:teuthology.orchestra.run.smithi134.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 449 kB/s | 15 kB 00:00 2024-04-22T23:11:48.443 INFO:teuthology.orchestra.run.smithi134.stdout:(8/13): python3-google-auth-2.28.2-1.el9.noarch 2.4 MB/s | 203 kB 00:00 2024-04-22T23:11:48.469 INFO:teuthology.orchestra.run.smithi134.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 276 kB/s | 279 kB 00:01 2024-04-22T23:11:48.485 INFO:teuthology.orchestra.run.smithi134.stdout:(10/13): python3-requests-oauthlib-1.3.0-12.el9 83 kB/s | 54 kB 00:00 2024-04-22T23:11:48.535 INFO:teuthology.orchestra.run.smithi134.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 11 MB/s | 1.0 MB 00:00 2024-04-22T23:11:48.552 INFO:teuthology.orchestra.run.smithi134.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 707 kB/s | 59 kB 00:00 2024-04-22T23:11:48.603 INFO:teuthology.orchestra.run.smithi134.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 767 kB/s | 90 kB 00:00 2024-04-22T23:11:48.604 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-04-22T23:11:48.604 INFO:teuthology.orchestra.run.smithi134.stdout:Total 1.2 MB/s | 2.2 MB 00:01 2024-04-22T23:11:48.673 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-04-22T23:11:48.689 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-04-22T23:11:48.689 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-04-22T23:11:48.705 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephadm-2:18.2.1-0.el9.noarch 1/1 2024-04-22T23:11:48.773 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-04-22T23:11:48.773 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-04-22T23:11:48.789 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephadm-2:18.2.1-0.el9.noarch 1/1 2024-04-22T23:11:49.094 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephadm-2:18.2.1-0.el9.noarch 1/1 2024-04-22T23:11:49.361 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephadm-2:18.2.1-0.el9.noarch 1/1 2024-04-22T23:11:49.361 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:11:49.361 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-22T23:11:49.361 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm-2:18.2.1-0.el9.noarch 2024-04-22T23:11:49.361 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:11:49.361 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-22T23:11:49.380 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-04-22T23:11:49.556 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-22T23:11:49.590 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-immutable-object-cache 2024-04-22T23:11:49.611 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-22T23:11:49.675 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-22T23:11:49.705 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-22T23:11:49.746 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-22T23:11:49.838 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-22T23:11:49.948 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-22T23:11:49.997 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-22T23:11:50.120 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:51 ago on Mon 22 Apr 2024 11:10:59 PM UTC. 2024-04-22T23:11:50.244 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-22T23:11:50.244 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:11:50.245 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2024-04-22T23:11:50.245 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:11:50.245 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-22T23:11:50.245 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache x86_64 2:18.2.1-0.el9 ceph 142 k 2024-04-22T23:11:50.245 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:11:50.245 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-22T23:11:50.245 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:11:50.245 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-04-22T23:11:50.245 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:11:50.245 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 142 k 2024-04-22T23:11:50.245 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 395 k 2024-04-22T23:11:50.245 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-22T23:11:50.310 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-immutable-object-cache-18.2.1-0.el9.x86_64 2.2 MB/s | 142 kB 00:00 2024-04-22T23:11:50.311 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-22T23:11:50.311 INFO:teuthology.orchestra.run.smithi067.stdout:Total 2.1 MB/s | 142 kB 00:00 2024-04-22T23:11:50.311 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-22T23:11:50.319 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-22T23:11:50.319 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-22T23:11:50.379 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-22T23:11:50.379 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-22T23:11:50.533 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-22T23:11:50.600 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T23:11:50.625 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T23:11:50.625 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T23:11:50.626 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-22T23:11:50.626 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:11:50.715 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-22T23:11:50.779 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-22T23:11:50.812 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-22T23:11:50.859 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-22T23:11:50.882 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-rook-2:18.2.1-0.el9.noarch 13/13 2024-04-22T23:11:51.537 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T23:11:51.537 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:11:51.537 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-22T23:11:51.537 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache-2:18.2.1-0.el9.x86_64 2024-04-22T23:11:51.537 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:11:51.537 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-22T23:11:51.662 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.1-0.el9.noarch 13/13 2024-04-22T23:11:51.662 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-rook-2:18.2.1-0.el9.noarch 1/13 2024-04-22T23:11:51.662 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-22T23:11:51.662 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-22T23:11:51.662 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-22T23:11:51.662 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-22T23:11:51.662 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-22T23:11:51.662 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-22T23:11:51.662 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-22T23:11:51.662 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-22T23:11:51.663 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-22T23:11:51.663 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-22T23:11:51.663 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-22T23:11:51.718 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr 2024-04-22T23:11:51.969 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-22T23:11:51.969 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:51.969 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-04-22T23:11:51.970 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-rook-2:18.2.1-0.el9.noarch 2024-04-22T23:11:51.970 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-22T23:11:51.970 INFO:teuthology.orchestra.run.smithi134.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-22T23:11:51.970 INFO:teuthology.orchestra.run.smithi134.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-22T23:11:51.970 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-22T23:11:51.970 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-22T23:11:51.970 INFO:teuthology.orchestra.run.smithi134.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-22T23:11:51.970 INFO:teuthology.orchestra.run.smithi134.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-22T23:11:51.970 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-22T23:11:51.970 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-22T23:11:51.970 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-22T23:11:51.970 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-22T23:11:51.970 INFO:teuthology.orchestra.run.smithi134.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-22T23:11:51.971 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:51.971 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-22T23:11:52.213 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-cephadm 2024-04-22T23:11:52.250 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:53 ago on Mon 22 Apr 2024 11:10:59 PM UTC. 2024-04-22T23:11:52.332 INFO:teuthology.orchestra.run.smithi067.stdout:Package ceph-mgr-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-22T23:11:52.378 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-22T23:11:52.378 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-04-22T23:11:52.378 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-22T23:11:52.462 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-dashboard 2024-04-22T23:11:52.725 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:12 ago on Mon 22 Apr 2024 11:10:40 PM UTC. 2024-04-22T23:11:52.843 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-22T23:11:52.843 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:11:52.844 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-04-22T23:11:52.844 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:11:52.844 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-04-22T23:11:52.844 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-cephadm noarch 2:18.2.1-0.el9 ceph-noarch 132 k 2024-04-22T23:11:52.844 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-04-22T23:11:52.844 INFO:teuthology.orchestra.run.smithi134.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-22T23:11:52.844 INFO:teuthology.orchestra.run.smithi134.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-22T23:11:52.844 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-22T23:11:52.844 INFO:teuthology.orchestra.run.smithi134.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-22T23:11:52.844 INFO:teuthology.orchestra.run.smithi134.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-22T23:11:52.845 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:52.845 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-04-22T23:11:52.845 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:11:52.845 INFO:teuthology.orchestra.run.smithi134.stdout:Install 6 Packages 2024-04-22T23:11:52.845 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:52.845 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 7.0 M 2024-04-22T23:11:52.846 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 33 M 2024-04-22T23:11:52.846 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-04-22T23:11:52.987 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:53 ago on Mon 22 Apr 2024 11:10:59 PM UTC. 2024-04-22T23:11:53.117 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-22T23:11:53.118 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:11:53.118 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-04-22T23:11:53.118 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:11:53.118 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-22T23:11:53.118 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard noarch 2:18.2.1-0.el9 ceph-noarch 1.8 M 2024-04-22T23:11:53.118 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-04-22T23:11:53.118 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards noarch 2:18.2.1-0.el9 ceph-noarch 23 k 2024-04-22T23:11:53.119 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts noarch 2:18.2.1-0.el9 ceph-noarch 15 k 2024-04-22T23:11:53.119 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-04-22T23:11:53.119 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-22T23:11:53.119 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-22T23:11:53.119 INFO:teuthology.orchestra.run.smithi067.stdout:Installing weak dependencies: 2024-04-22T23:11:53.119 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-04-22T23:11:53.119 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:11:53.119 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-22T23:11:53.119 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:11:53.119 INFO:teuthology.orchestra.run.smithi067.stdout:Install 7 Packages 2024-04-22T23:11:53.119 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:11:53.120 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 2.1 M 2024-04-22T23:11:53.120 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 8.5 M 2024-04-22T23:11:53.120 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-22T23:11:53.199 INFO:teuthology.orchestra.run.smithi134.stdout:(1/6): ceph-mgr-cephadm-18.2.1-0.el9.noarch.rpm 2.2 MB/s | 132 kB 00:00 2024-04-22T23:11:53.307 INFO:teuthology.orchestra.run.smithi134.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.5 MB/s | 248 kB 00:00 2024-04-22T23:11:53.403 INFO:teuthology.orchestra.run.smithi067.stdout:(1/7): ceph-grafana-dashboards-18.2.1-0.el9.noa 195 kB/s | 23 kB 00:00 2024-04-22T23:11:53.428 INFO:teuthology.orchestra.run.smithi067.stdout:(2/7): ceph-prometheus-alerts-18.2.1-0.el9.noar 102 kB/s | 15 kB 00:00 2024-04-22T23:11:53.478 INFO:teuthology.orchestra.run.smithi067.stdout:(3/7): ceph-mgr-dashboard-18.2.1-0.el9.noarch.r 9.1 MB/s | 1.8 MB 00:00 2024-04-22T23:11:53.533 INFO:teuthology.orchestra.run.smithi134.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-04-22T23:11:53.558 INFO:teuthology.orchestra.run.smithi134.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 231 kB/s | 58 kB 00:00 2024-04-22T23:11:53.616 INFO:teuthology.orchestra.run.smithi134.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.3 MB/s | 548 kB 00:00 2024-04-22T23:11:53.761 INFO:teuthology.orchestra.run.smithi067.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 25 kB/s | 9.0 kB 00:00 2024-04-22T23:11:53.841 INFO:teuthology.orchestra.run.smithi134.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 165 kB/s | 51 kB 00:00 2024-04-22T23:11:53.842 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-04-22T23:11:53.842 INFO:teuthology.orchestra.run.smithi134.stdout:Total 7.0 MB/s | 7.0 MB 00:00 2024-04-22T23:11:53.887 INFO:teuthology.orchestra.run.smithi067.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 89 kB/s | 41 kB 00:00 2024-04-22T23:11:53.912 INFO:teuthology.orchestra.run.smithi067.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 71 kB/s | 31 kB 00:00 2024-04-22T23:11:53.912 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-04-22T23:11:53.923 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-04-22T23:11:53.923 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-04-22T23:11:54.018 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-04-22T23:11:54.018 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-04-22T23:11:54.162 INFO:teuthology.orchestra.run.smithi067.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 471 kB/s | 188 kB 00:00 2024-04-22T23:11:54.165 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-22T23:11:54.166 INFO:teuthology.orchestra.run.smithi067.stdout:Total 2.0 MB/s | 2.1 MB 00:01 2024-04-22T23:11:54.166 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-22T23:11:54.175 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-22T23:11:54.175 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-22T23:11:54.231 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-22T23:11:54.232 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-22T23:11:54.319 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-04-22T23:11:54.490 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-22T23:11:54.528 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-22T23:11:54.551 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-22T23:11:54.588 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-04-22T23:11:54.615 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-04-22T23:11:54.663 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-04-22T23:11:54.701 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-04-22T23:11:54.729 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 5/7 2024-04-22T23:11:54.977 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 6/7 2024-04-22T23:11:55.007 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 7/7 2024-04-22T23:11:55.066 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-22T23:11:55.162 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-22T23:11:55.231 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-22T23:11:55.256 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 6/6 2024-04-22T23:11:55.516 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 7/7 2024-04-22T23:11:55.516 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 1/7 2024-04-22T23:11:55.516 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 2/7 2024-04-22T23:11:55.516 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 3/7 2024-04-22T23:11:55.516 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-04-22T23:11:55.516 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-04-22T23:11:55.516 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-04-22T23:11:55.762 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-04-22T23:11:55.762 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:11:55.762 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-22T23:11:55.762 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards-2:18.2.1-0.el9.noarch 2024-04-22T23:11:55.763 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard-2:18.2.1-0.el9.noarch 2024-04-22T23:11:55.763 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts-2:18.2.1-0.el9.noarch 2024-04-22T23:11:55.763 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-04-22T23:11:55.763 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-04-22T23:11:55.763 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-22T23:11:55.763 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-22T23:11:55.763 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:11:55.763 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-22T23:11:55.973 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-22T23:11:56.038 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 6/6 2024-04-22T23:11:56.038 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 1/6 2024-04-22T23:11:56.038 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-22T23:11:56.038 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-22T23:11:56.038 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-22T23:11:56.038 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-22T23:11:56.280 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-22T23:11:56.280 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:56.280 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-04-22T23:11:56.280 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 2024-04-22T23:11:56.280 INFO:teuthology.orchestra.run.smithi134.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-22T23:11:56.280 INFO:teuthology.orchestra.run.smithi134.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-22T23:11:56.280 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-22T23:11:56.280 INFO:teuthology.orchestra.run.smithi134.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-22T23:11:56.280 INFO:teuthology.orchestra.run.smithi134.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-22T23:11:56.280 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:56.280 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-22T23:11:56.499 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:57 ago on Mon 22 Apr 2024 11:10:59 PM UTC. 2024-04-22T23:11:56.505 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-fuse 2024-04-22T23:11:56.625 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-22T23:11:56.626 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:11:56.626 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-04-22T23:11:56.626 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:11:56.626 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-22T23:11:56.626 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.1-0.el9 ceph-noarch 7.4 M 2024-04-22T23:11:56.626 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-04-22T23:11:56.626 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-22T23:11:56.626 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-22T23:11:56.627 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-22T23:11:56.627 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-22T23:11:56.627 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-22T23:11:56.627 INFO:teuthology.orchestra.run.smithi067.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-22T23:11:56.627 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-22T23:11:56.627 INFO:teuthology.orchestra.run.smithi067.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-22T23:11:56.627 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-22T23:11:56.627 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-22T23:11:56.627 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-22T23:11:56.627 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:11:56.627 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-22T23:11:56.627 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:11:56.627 INFO:teuthology.orchestra.run.smithi067.stdout:Install 12 Packages 2024-04-22T23:11:56.627 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:11:56.628 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 38 M 2024-04-22T23:11:56.628 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 209 M 2024-04-22T23:11:56.628 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-22T23:11:57.011 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:17 ago on Mon 22 Apr 2024 11:10:40 PM UTC. 2024-04-22T23:11:57.129 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-22T23:11:57.130 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:11:57.130 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-04-22T23:11:57.130 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:11:57.130 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-04-22T23:11:57.130 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse x86_64 2:18.2.1-0.el9 ceph 839 k 2024-04-22T23:11:57.130 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-04-22T23:11:57.130 INFO:teuthology.orchestra.run.smithi134.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-22T23:11:57.130 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:57.131 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-04-22T23:11:57.131 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:11:57.131 INFO:teuthology.orchestra.run.smithi134.stdout:Install 2 Packages 2024-04-22T23:11:57.131 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:57.131 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 920 k 2024-04-22T23:11:57.132 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 2.7 M 2024-04-22T23:11:57.132 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-04-22T23:11:57.244 INFO:teuthology.orchestra.run.smithi067.stdout:(1/12): ceph-mgr-diskprediction-local-18.2.1-0. 42 MB/s | 7.4 MB 00:00 2024-04-22T23:11:57.419 INFO:teuthology.orchestra.run.smithi067.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 187 kB/s | 33 kB 00:00 2024-04-22T23:11:57.485 INFO:teuthology.orchestra.run.smithi134.stdout:(1/2): ceph-fuse-18.2.1-0.el9.x86_64.rpm 11 MB/s | 839 kB 00:00 2024-04-22T23:11:58.094 INFO:teuthology.orchestra.run.smithi134.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 117 kB/s | 80 kB 00:00 2024-04-22T23:11:58.095 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-04-22T23:11:58.095 INFO:teuthology.orchestra.run.smithi134.stdout:Total 953 kB/s | 920 kB 00:00 2024-04-22T23:11:58.102 INFO:teuthology.orchestra.run.smithi067.stdout:(3/12): libquadmath-11.4.1-3.el9.x86_64.rpm 189 kB/s | 195 kB 00:01 2024-04-22T23:11:58.107 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-04-22T23:11:58.130 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-04-22T23:11:58.130 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-04-22T23:11:58.207 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-04-22T23:11:58.207 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-04-22T23:11:58.219 INFO:teuthology.orchestra.run.smithi067.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 157 kB/s | 18 kB 00:00 2024-04-22T23:11:58.345 INFO:teuthology.orchestra.run.smithi067.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 282 kB/s | 35 kB 00:00 2024-04-22T23:11:58.395 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-04-22T23:11:58.436 INFO:teuthology.orchestra.run.smithi067.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.1 MB/s | 3.0 MB 00:00 2024-04-22T23:11:58.524 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-22T23:11:58.537 INFO:teuthology.orchestra.run.smithi067.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 2.4 MB/s | 244 kB 00:00 2024-04-22T23:11:58.590 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-fuse-2:18.2.1-0.el9.x86_64 2/2 2024-04-22T23:11:58.829 INFO:teuthology.orchestra.run.smithi067.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 456 kB/s | 803 kB 00:01 2024-04-22T23:11:59.054 INFO:teuthology.orchestra.run.smithi067.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.1 MB/s | 250 kB 00:00 2024-04-22T23:11:59.155 INFO:teuthology.orchestra.run.smithi067.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 8.3 MB/s | 5.1 MB 00:00 2024-04-22T23:11:59.288 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-fuse-2:18.2.1-0.el9.x86_64 2/2 2024-04-22T23:11:59.288 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-fuse-2:18.2.1-0.el9.x86_64 1/2 2024-04-22T23:11:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-22T23:11:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:59.531 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-04-22T23:11:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse-2:18.2.1-0.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-22T23:11:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:11:59.531 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-22T23:11:59.613 INFO:teuthology.orchestra.run.smithi067.stdout:(11/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 3.7 MB/s | 4.7 MB 00:01 2024-04-22T23:11:59.780 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librados-devel 2024-04-22T23:12:00.300 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:20 ago on Mon 22 Apr 2024 11:10:40 PM UTC. 2024-04-22T23:12:00.421 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-22T23:12:00.422 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:12:00.423 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-04-22T23:12:00.423 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:12:00.423 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-04-22T23:12:00.423 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel x86_64 2:18.2.1-0.el9 ceph 127 k 2024-04-22T23:12:00.423 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:12:00.423 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-04-22T23:12:00.423 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:12:00.423 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-04-22T23:12:00.423 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:12:00.423 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 127 k 2024-04-22T23:12:00.423 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 456 k 2024-04-22T23:12:00.423 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-04-22T23:12:00.500 INFO:teuthology.orchestra.run.smithi134.stdout:librados-devel-18.2.1-0.el9.x86_64.rpm 1.6 MB/s | 127 kB 00:00 2024-04-22T23:12:00.500 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-04-22T23:12:00.501 INFO:teuthology.orchestra.run.smithi134.stdout:Total 1.6 MB/s | 127 kB 00:00 2024-04-22T23:12:00.501 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-04-22T23:12:00.506 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-04-22T23:12:00.506 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-04-22T23:12:00.568 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-04-22T23:12:00.568 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-04-22T23:12:00.588 INFO:teuthology.orchestra.run.smithi067.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-04-22T23:12:00.589 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-22T23:12:00.590 INFO:teuthology.orchestra.run.smithi067.stdout:Total 9.6 MB/s | 38 MB 00:03 2024-04-22T23:12:00.722 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-04-22T23:12:00.878 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T23:12:00.954 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-22T23:12:00.982 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-22T23:12:00.982 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-22T23:12:01.265 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T23:12:01.415 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-22T23:12:01.416 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-22T23:12:01.489 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T23:12:01.489 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:12:01.489 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-04-22T23:12:01.489 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel-2:18.2.1-0.el9.x86_64 2024-04-22T23:12:01.489 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:12:01.489 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-22T23:12:01.709 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install libcephfs2 2024-04-22T23:12:02.193 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-22T23:12:02.236 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:22 ago on Mon 22 Apr 2024 11:10:40 PM UTC. 2024-04-22T23:12:02.260 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-22T23:12:02.303 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-22T23:12:02.316 INFO:teuthology.orchestra.run.smithi134.stdout:Package libcephfs2-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-22T23:12:02.357 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-22T23:12:02.358 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-04-22T23:12:02.358 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-22T23:12:02.410 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-22T23:12:02.446 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-22T23:12:02.451 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install libcephfs-devel 2024-04-22T23:12:02.632 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-22T23:12:02.661 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-22T23:12:02.740 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-22T23:12:02.964 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:22 ago on Mon 22 Apr 2024 11:10:40 PM UTC. 2024-04-22T23:12:03.082 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-22T23:12:03.082 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:12:03.082 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-04-22T23:12:03.082 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:12:03.082 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-04-22T23:12:03.082 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs-devel x86_64 2:18.2.1-0.el9 ceph 31 k 2024-04-22T23:12:03.083 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:12:03.083 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-04-22T23:12:03.083 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:12:03.083 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-04-22T23:12:03.083 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:12:03.084 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 31 k 2024-04-22T23:12:03.084 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 139 k 2024-04-22T23:12:03.084 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-04-22T23:12:03.146 INFO:teuthology.orchestra.run.smithi134.stdout:libcephfs-devel-18.2.1-0.el9.x86_64.rpm 498 kB/s | 31 kB 00:00 2024-04-22T23:12:03.147 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-04-22T23:12:03.147 INFO:teuthology.orchestra.run.smithi134.stdout:Total 485 kB/s | 31 kB 00:00 2024-04-22T23:12:03.147 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-04-22T23:12:03.150 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-04-22T23:12:03.150 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-04-22T23:12:03.152 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-04-22T23:12:03.153 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-04-22T23:12:03.237 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-04-22T23:12:03.358 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T23:12:03.445 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-22T23:12:03.563 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-22T23:12:03.717 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T23:12:03.920 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T23:12:03.920 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:12:03.920 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-04-22T23:12:03.920 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs-devel-2:18.2.1-0.el9.x86_64 2024-04-22T23:12:03.920 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:12:03.920 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-22T23:12:04.093 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librados2 2024-04-22T23:12:04.597 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:24 ago on Mon 22 Apr 2024 11:10:40 PM UTC. 2024-04-22T23:12:04.674 INFO:teuthology.orchestra.run.smithi134.stdout:Package librados2-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-22T23:12:04.714 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-22T23:12:04.715 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-04-22T23:12:04.716 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-22T23:12:04.785 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librbd1 2024-04-22T23:12:04.887 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-22T23:12:05.298 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:25 ago on Mon 22 Apr 2024 11:10:40 PM UTC. 2024-04-22T23:12:05.378 INFO:teuthology.orchestra.run.smithi134.stdout:Package librbd1-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-22T23:12:05.419 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-22T23:12:05.420 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-04-22T23:12:05.421 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-22T23:12:05.504 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rados 2024-04-22T23:12:05.960 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-22T23:12:05.985 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 12/12 2024-04-22T23:12:06.016 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:26 ago on Mon 22 Apr 2024 11:10:40 PM UTC. 2024-04-22T23:12:06.095 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rados-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-22T23:12:06.136 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-22T23:12:06.137 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-04-22T23:12:06.137 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-22T23:12:06.220 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rgw 2024-04-22T23:12:06.735 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:26 ago on Mon 22 Apr 2024 11:10:40 PM UTC. 2024-04-22T23:12:06.813 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rgw-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-22T23:12:06.853 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-22T23:12:06.854 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-04-22T23:12:06.855 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-22T23:12:06.937 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-cephfs 2024-04-22T23:12:07.442 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:27 ago on Mon 22 Apr 2024 11:10:40 PM UTC. 2024-04-22T23:12:07.520 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-cephfs-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-22T23:12:07.560 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-22T23:12:07.561 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-04-22T23:12:07.561 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-22T23:12:07.645 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rbd 2024-04-22T23:12:08.160 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:28 ago on Mon 22 Apr 2024 11:10:40 PM UTC. 2024-04-22T23:12:08.239 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rbd-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-22T23:12:08.280 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-22T23:12:08.281 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-04-22T23:12:08.281 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-22T23:12:08.355 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-fuse 2024-04-22T23:12:08.441 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 12/12 2024-04-22T23:12:08.441 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarc 1/12 2024-04-22T23:12:08.441 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-22T23:12:08.441 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-22T23:12:08.441 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-22T23:12:08.442 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-22T23:12:08.442 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-22T23:12:08.442 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-22T23:12:08.442 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-22T23:12:08.442 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-22T23:12:08.442 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-22T23:12:08.442 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-22T23:12:08.785 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-22T23:12:08.785 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:12:08.786 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-22T23:12:08.786 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local-2:18.2.1-0.el9.noarch 2024-04-22T23:12:08.786 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-22T23:12:08.786 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-22T23:12:08.786 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-22T23:12:08.786 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-22T23:12:08.786 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-22T23:12:08.786 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-22T23:12:08.786 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-22T23:12:08.786 INFO:teuthology.orchestra.run.smithi067.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-22T23:12:08.786 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-22T23:12:08.786 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-22T23:12:08.786 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-22T23:12:08.787 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:12:08.787 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-22T23:12:08.864 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:28 ago on Mon 22 Apr 2024 11:10:40 PM UTC. 2024-04-22T23:12:08.985 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-22T23:12:08.986 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:12:08.986 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-04-22T23:12:08.986 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:12:08.986 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-04-22T23:12:08.987 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse x86_64 2:18.2.1-0.el9 ceph 86 k 2024-04-22T23:12:08.987 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:12:08.987 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-04-22T23:12:08.987 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:12:08.987 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-04-22T23:12:08.987 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:12:08.987 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 86 k 2024-04-22T23:12:08.987 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 226 k 2024-04-22T23:12:08.988 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-04-22T23:12:09.057 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-fuse-18.2.1-0.el9.x86_64.rpm 1.2 MB/s | 86 kB 00:00 2024-04-22T23:12:09.058 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-04-22T23:12:09.058 INFO:teuthology.orchestra.run.smithi134.stdout:Total 1.2 MB/s | 86 kB 00:00 2024-04-22T23:12:09.058 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-04-22T23:12:09.065 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-04-22T23:12:09.065 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-04-22T23:12:09.128 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-04-22T23:12:09.129 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-04-22T23:12:09.151 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-rook 2024-04-22T23:12:09.266 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-04-22T23:12:09.361 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T23:12:09.685 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:10 ago on Mon 22 Apr 2024 11:10:59 PM UTC. 2024-04-22T23:12:09.753 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T23:12:09.811 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-22T23:12:09.812 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:12:09.812 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-04-22T23:12:09.812 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:12:09.812 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-22T23:12:09.812 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook noarch 2:18.2.1-0.el9 ceph-noarch 50 k 2024-04-22T23:12:09.812 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-04-22T23:12:09.813 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-22T23:12:09.813 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-22T23:12:09.813 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-22T23:12:09.813 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-22T23:12:09.813 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-22T23:12:09.813 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-22T23:12:09.813 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-22T23:12:09.813 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-22T23:12:09.813 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-22T23:12:09.813 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-22T23:12:09.813 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-22T23:12:09.813 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-22T23:12:09.813 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:12:09.813 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-22T23:12:09.814 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:12:09.814 INFO:teuthology.orchestra.run.smithi067.stdout:Install 13 Packages 2024-04-22T23:12:09.814 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:12:09.814 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 2.2 M 2024-04-22T23:12:09.814 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 26 M 2024-04-22T23:12:09.814 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-22T23:12:09.945 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T23:12:09.945 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:12:09.945 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-04-22T23:12:09.945 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse-2:18.2.1-0.el9.x86_64 2024-04-22T23:12:09.945 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:12:09.946 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-22T23:12:10.028 INFO:teuthology.orchestra.run.smithi067.stdout:(1/13): ceph-mgr-rook-18.2.1-0.el9.noarch.rpm 846 kB/s | 50 kB 00:00 2024-04-22T23:12:10.184 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-mirror 2024-04-22T23:12:10.191 INFO:teuthology.orchestra.run.smithi067.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 118 kB/s | 26 kB 00:00 2024-04-22T23:12:10.203 INFO:teuthology.orchestra.run.smithi067.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 82 kB/s | 19 kB 00:00 2024-04-22T23:12:10.287 INFO:teuthology.orchestra.run.smithi067.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 159 kB 00:00 2024-04-22T23:12:10.312 INFO:teuthology.orchestra.run.smithi067.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 782 kB/s | 222 kB 00:00 2024-04-22T23:12:10.337 INFO:teuthology.orchestra.run.smithi067.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.0 MB/s | 279 kB 00:00 2024-04-22T23:12:10.379 INFO:teuthology.orchestra.run.smithi067.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 583 kB/s | 54 kB 00:00 2024-04-22T23:12:10.497 INFO:teuthology.orchestra.run.smithi067.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 175 kB/s | 32 kB 00:00 2024-04-22T23:12:10.522 INFO:teuthology.orchestra.run.smithi067.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 95 kB/s | 15 kB 00:00 2024-04-22T23:12:10.622 INFO:teuthology.orchestra.run.smithi067.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 836 kB/s | 203 kB 00:00 2024-04-22T23:12:10.689 INFO:teuthology.orchestra.run.smithi067.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 5.3 MB/s | 1.0 MB 00:00 2024-04-22T23:12:10.701 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:30 ago on Mon 22 Apr 2024 11:10:40 PM UTC. 2024-04-22T23:12:10.714 INFO:teuthology.orchestra.run.smithi067.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 307 kB/s | 59 kB 00:00 2024-04-22T23:12:10.739 INFO:teuthology.orchestra.run.smithi067.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 769 kB/s | 90 kB 00:00 2024-04-22T23:12:10.744 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-22T23:12:10.744 INFO:teuthology.orchestra.run.smithi067.stdout:Total 2.4 MB/s | 2.2 MB 00:00 2024-04-22T23:12:10.806 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-22T23:12:10.822 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-22T23:12:10.822 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-22T23:12:10.823 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-22T23:12:10.823 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:12:10.823 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-04-22T23:12:10.823 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:12:10.823 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-04-22T23:12:10.824 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror x86_64 2:18.2.1-0.el9 ceph 3.0 M 2024-04-22T23:12:10.824 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:12:10.824 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-04-22T23:12:10.824 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:12:10.824 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-04-22T23:12:10.824 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:12:10.824 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 3.0 M 2024-04-22T23:12:10.824 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 12 M 2024-04-22T23:12:10.824 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-04-22T23:12:10.916 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-22T23:12:10.916 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-22T23:12:10.918 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-mirror-18.2.1-0.el9.x86_64.rpm 32 MB/s | 3.0 MB 00:00 2024-04-22T23:12:10.919 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-04-22T23:12:10.919 INFO:teuthology.orchestra.run.smithi134.stdout:Total 31 MB/s | 3.0 MB 00:00 2024-04-22T23:12:10.919 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-04-22T23:12:10.928 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-04-22T23:12:10.928 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-04-22T23:12:11.013 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-04-22T23:12:11.013 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-04-22T23:12:11.427 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-04-22T23:12:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-22T23:12:11.630 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T23:12:11.655 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T23:12:11.655 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T23:12:11.655 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-22T23:12:11.655 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T23:12:11.655 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T23:12:11.655 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:12:11.703 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-22T23:12:11.759 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-22T23:12:11.816 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-22T23:12:11.852 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-22T23:12:11.894 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-22T23:12:11.991 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-22T23:12:12.112 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-22T23:12:12.196 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-22T23:12:12.578 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T23:12:12.578 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:12:12.578 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-04-22T23:12:12.579 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror-2:18.2.1-0.el9.x86_64 2024-04-22T23:12:12.579 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:12:12.579 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-22T23:12:12.848 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-nbd 2024-04-22T23:12:12.932 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-22T23:12:12.975 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-22T23:12:13.008 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-22T23:12:13.057 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-22T23:12:13.086 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-rook-2:18.2.1-0.el9.noarch 13/13 2024-04-22T23:12:13.367 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:33 ago on Mon 22 Apr 2024 11:10:40 PM UTC. 2024-04-22T23:12:13.492 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-22T23:12:13.493 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:12:13.493 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-04-22T23:12:13.493 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:12:13.493 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-04-22T23:12:13.493 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd x86_64 2:18.2.1-0.el9 ceph 171 k 2024-04-22T23:12:13.493 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:12:13.493 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-04-22T23:12:13.493 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-22T23:12:13.493 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-04-22T23:12:13.493 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:12:13.493 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 171 k 2024-04-22T23:12:13.494 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 494 k 2024-04-22T23:12:13.494 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-04-22T23:12:13.557 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-nbd-18.2.1-0.el9.x86_64.rpm 2.7 MB/s | 171 kB 00:00 2024-04-22T23:12:13.558 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-04-22T23:12:13.558 INFO:teuthology.orchestra.run.smithi134.stdout:Total 2.6 MB/s | 171 kB 00:00 2024-04-22T23:12:13.558 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-04-22T23:12:13.565 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-04-22T23:12:13.565 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-04-22T23:12:13.628 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-04-22T23:12:13.628 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-04-22T23:12:13.774 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-04-22T23:12:13.870 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.1-0.el9.noarch 13/13 2024-04-22T23:12:13.871 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-rook-2:18.2.1-0.el9.noarch 1/13 2024-04-22T23:12:13.871 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-22T23:12:13.871 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-22T23:12:13.871 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-22T23:12:13.871 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-22T23:12:13.871 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-22T23:12:13.871 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-22T23:12:13.871 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-22T23:12:13.871 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-22T23:12:13.872 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-22T23:12:13.872 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-22T23:12:13.872 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-22T23:12:13.872 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T23:12:14.178 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-22T23:12:14.178 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:12:14.179 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-22T23:12:14.179 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook-2:18.2.1-0.el9.noarch 2024-04-22T23:12:14.179 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-22T23:12:14.179 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-22T23:12:14.179 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-22T23:12:14.179 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-22T23:12:14.179 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-22T23:12:14.179 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-22T23:12:14.179 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-22T23:12:14.179 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-22T23:12:14.180 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-22T23:12:14.180 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-22T23:12:14.180 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-22T23:12:14.180 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-22T23:12:14.180 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:12:14.180 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-22T23:12:14.280 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T23:12:14.415 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-cephadm 2024-04-22T23:12:14.511 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T23:12:14.511 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:12:14.511 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-04-22T23:12:14.511 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd-2:18.2.1-0.el9.x86_64 2024-04-22T23:12:14.512 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:12:14.512 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-22T23:12:14.778 DEBUG:teuthology.parallel:result is None 2024-04-22T23:12:14.949 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:15 ago on Mon 22 Apr 2024 11:10:59 PM UTC. 2024-04-22T23:12:15.073 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-22T23:12:15.074 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:12:15.074 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-04-22T23:12:15.074 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:12:15.074 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-22T23:12:15.074 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm noarch 2:18.2.1-0.el9 ceph-noarch 132 k 2024-04-22T23:12:15.074 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-04-22T23:12:15.074 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-22T23:12:15.074 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-22T23:12:15.075 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-22T23:12:15.075 INFO:teuthology.orchestra.run.smithi067.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-22T23:12:15.075 INFO:teuthology.orchestra.run.smithi067.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-22T23:12:15.075 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:12:15.075 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-22T23:12:15.075 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:12:15.075 INFO:teuthology.orchestra.run.smithi067.stdout:Install 6 Packages 2024-04-22T23:12:15.075 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:12:15.076 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 7.0 M 2024-04-22T23:12:15.076 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 33 M 2024-04-22T23:12:15.076 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-22T23:12:15.684 INFO:teuthology.orchestra.run.smithi067.stdout:(1/6): ceph-mgr-cephadm-18.2.1-0.el9.noarch.rpm 2.0 MB/s | 132 kB 00:00 2024-04-22T23:12:15.984 INFO:teuthology.orchestra.run.smithi067.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.8 MB/s | 548 kB 00:00 2024-04-22T23:12:16.026 INFO:teuthology.orchestra.run.smithi067.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.4 MB/s | 58 kB 00:00 2024-04-22T23:12:16.068 INFO:teuthology.orchestra.run.smithi067.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 1.2 MB/s | 51 kB 00:00 2024-04-22T23:12:16.552 INFO:teuthology.orchestra.run.smithi067.stdout:(5/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 266 kB/s | 248 kB 00:00 2024-04-22T23:12:23.727 INFO:teuthology.orchestra.run.smithi067.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 753 kB/s | 6.0 MB 00:08 2024-04-22T23:12:23.728 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-22T23:12:23.728 INFO:teuthology.orchestra.run.smithi067.stdout:Total 825 kB/s | 7.0 MB 00:08 2024-04-22T23:12:23.804 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-22T23:12:23.817 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-22T23:12:23.817 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-22T23:12:23.920 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-22T23:12:23.920 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-22T23:12:24.236 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-22T23:12:24.407 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-22T23:12:24.462 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-22T23:12:25.019 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-22T23:12:25.100 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-22T23:12:25.177 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-22T23:12:25.206 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 6/6 2024-04-22T23:12:25.920 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 6/6 2024-04-22T23:12:25.920 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 1/6 2024-04-22T23:12:25.920 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-22T23:12:25.920 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-22T23:12:25.920 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-22T23:12:25.920 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-22T23:12:26.183 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-22T23:12:26.183 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:12:26.183 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-22T23:12:26.183 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm-2:18.2.1-0.el9.noarch 2024-04-22T23:12:26.183 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-22T23:12:26.183 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-22T23:12:26.183 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-22T23:12:26.183 INFO:teuthology.orchestra.run.smithi067.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-22T23:12:26.183 INFO:teuthology.orchestra.run.smithi067.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-22T23:12:26.183 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:12:26.184 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-22T23:12:26.408 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-fuse 2024-04-22T23:12:26.942 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:27 ago on Mon 22 Apr 2024 11:10:59 PM UTC. 2024-04-22T23:12:27.068 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-22T23:12:27.069 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:12:27.069 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-04-22T23:12:27.069 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:12:27.069 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-22T23:12:27.069 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse x86_64 2:18.2.1-0.el9 ceph 839 k 2024-04-22T23:12:27.069 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-04-22T23:12:27.069 INFO:teuthology.orchestra.run.smithi067.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-22T23:12:27.069 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:12:27.069 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-22T23:12:27.069 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:12:27.069 INFO:teuthology.orchestra.run.smithi067.stdout:Install 2 Packages 2024-04-22T23:12:27.070 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:12:27.070 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 920 k 2024-04-22T23:12:27.070 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 2.7 M 2024-04-22T23:12:27.070 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-22T23:12:27.411 INFO:teuthology.orchestra.run.smithi067.stdout:(1/2): ceph-fuse-18.2.1-0.el9.x86_64.rpm 11 MB/s | 839 kB 00:00 2024-04-22T23:12:28.028 INFO:teuthology.orchestra.run.smithi067.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 117 kB/s | 80 kB 00:00 2024-04-22T23:12:28.029 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-22T23:12:28.029 INFO:teuthology.orchestra.run.smithi067.stdout:Total 958 kB/s | 920 kB 00:00 2024-04-22T23:12:28.043 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-22T23:12:28.070 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-22T23:12:28.070 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-22T23:12:28.155 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-22T23:12:28.155 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-22T23:12:28.345 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-22T23:12:28.488 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-22T23:12:28.552 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-fuse-2:18.2.1-0.el9.x86_64 2/2 2024-04-22T23:12:29.193 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-fuse-2:18.2.1-0.el9.x86_64 2/2 2024-04-22T23:12:29.193 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-fuse-2:18.2.1-0.el9.x86_64 1/2 2024-04-22T23:12:29.448 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-22T23:12:29.448 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:12:29.448 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-22T23:12:29.448 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse-2:18.2.1-0.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-22T23:12:29.448 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:12:29.448 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-22T23:12:29.694 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install librados-devel 2024-04-22T23:12:30.229 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:31 ago on Mon 22 Apr 2024 11:10:59 PM UTC. 2024-04-22T23:12:30.353 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-22T23:12:30.353 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:12:30.353 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-04-22T23:12:30.353 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:12:30.353 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-22T23:12:30.354 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel x86_64 2:18.2.1-0.el9 ceph 127 k 2024-04-22T23:12:30.354 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:12:30.354 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-22T23:12:30.354 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:12:30.354 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-04-22T23:12:30.354 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:12:30.354 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 127 k 2024-04-22T23:12:30.355 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 456 k 2024-04-22T23:12:30.355 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-22T23:12:30.415 INFO:teuthology.orchestra.run.smithi067.stdout:librados-devel-18.2.1-0.el9.x86_64.rpm 2.0 MB/s | 127 kB 00:00 2024-04-22T23:12:30.416 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-22T23:12:30.416 INFO:teuthology.orchestra.run.smithi067.stdout:Total 2.0 MB/s | 127 kB 00:00 2024-04-22T23:12:30.416 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-22T23:12:30.422 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-22T23:12:30.423 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-22T23:12:30.488 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-22T23:12:30.488 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-22T23:12:30.651 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-22T23:12:30.782 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T23:12:31.167 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T23:12:31.420 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T23:12:31.421 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:12:31.421 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-22T23:12:31.421 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel-2:18.2.1-0.el9.x86_64 2024-04-22T23:12:31.421 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:12:31.421 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-22T23:12:31.662 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install libcephfs2 2024-04-22T23:12:32.204 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:33 ago on Mon 22 Apr 2024 11:10:59 PM UTC. 2024-04-22T23:12:32.288 INFO:teuthology.orchestra.run.smithi067.stdout:Package libcephfs2-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-22T23:12:32.332 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-22T23:12:32.333 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-04-22T23:12:32.333 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-22T23:12:32.398 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install libcephfs-devel 2024-04-22T23:12:32.928 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:33 ago on Mon 22 Apr 2024 11:10:59 PM UTC. 2024-04-22T23:12:33.054 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-22T23:12:33.058 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:12:33.058 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-04-22T23:12:33.058 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:12:33.058 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-22T23:12:33.058 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel x86_64 2:18.2.1-0.el9 ceph 31 k 2024-04-22T23:12:33.058 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:12:33.059 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-22T23:12:33.059 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:12:33.059 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-04-22T23:12:33.059 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:12:33.059 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 31 k 2024-04-22T23:12:33.059 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 139 k 2024-04-22T23:12:33.059 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-22T23:12:33.138 INFO:teuthology.orchestra.run.smithi067.stdout:libcephfs-devel-18.2.1-0.el9.x86_64.rpm 376 kB/s | 31 kB 00:00 2024-04-22T23:12:33.139 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-22T23:12:33.139 INFO:teuthology.orchestra.run.smithi067.stdout:Total 371 kB/s | 31 kB 00:00 2024-04-22T23:12:33.139 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-22T23:12:33.143 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-22T23:12:33.143 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-22T23:12:33.145 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-22T23:12:33.145 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-22T23:12:33.239 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-22T23:12:33.353 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T23:12:33.706 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T23:12:33.915 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs-devel-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T23:12:33.915 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:12:33.915 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-22T23:12:33.915 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel-2:18.2.1-0.el9.x86_64 2024-04-22T23:12:33.916 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:12:33.916 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-22T23:12:34.094 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install librados2 2024-04-22T23:12:34.633 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:35 ago on Mon 22 Apr 2024 11:10:59 PM UTC. 2024-04-22T23:12:34.716 INFO:teuthology.orchestra.run.smithi067.stdout:Package librados2-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-22T23:12:34.760 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-22T23:12:34.761 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-04-22T23:12:34.761 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-22T23:12:34.835 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install librbd1 2024-04-22T23:12:35.365 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:36 ago on Mon 22 Apr 2024 11:10:59 PM UTC. 2024-04-22T23:12:35.447 INFO:teuthology.orchestra.run.smithi067.stdout:Package librbd1-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-22T23:12:35.490 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-22T23:12:35.491 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-04-22T23:12:35.491 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-22T23:12:35.567 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-rados 2024-04-22T23:12:36.108 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:37 ago on Mon 22 Apr 2024 11:10:59 PM UTC. 2024-04-22T23:12:36.191 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-rados-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-22T23:12:36.233 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-22T23:12:36.234 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-04-22T23:12:36.235 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-22T23:12:36.306 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-rgw 2024-04-22T23:12:36.841 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:37 ago on Mon 22 Apr 2024 11:10:59 PM UTC. 2024-04-22T23:12:36.922 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-rgw-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-22T23:12:36.964 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-22T23:12:36.965 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-04-22T23:12:36.966 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-22T23:12:37.031 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-cephfs 2024-04-22T23:12:37.561 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:38 ago on Mon 22 Apr 2024 11:10:59 PM UTC. 2024-04-22T23:12:37.643 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-cephfs-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-22T23:12:37.686 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-22T23:12:37.687 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-04-22T23:12:37.688 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-22T23:12:37.759 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-rbd 2024-04-22T23:12:38.294 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:39 ago on Mon 22 Apr 2024 11:10:59 PM UTC. 2024-04-22T23:12:38.376 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-rbd-2:18.2.1-0.el9.x86_64 is already installed. 2024-04-22T23:12:38.420 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-22T23:12:38.422 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-04-22T23:12:38.422 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-22T23:12:38.491 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install rbd-fuse 2024-04-22T23:12:39.022 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:40 ago on Mon 22 Apr 2024 11:10:59 PM UTC. 2024-04-22T23:12:39.145 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-22T23:12:39.146 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:12:39.146 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-04-22T23:12:39.146 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:12:39.146 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-22T23:12:39.146 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse x86_64 2:18.2.1-0.el9 ceph 86 k 2024-04-22T23:12:39.146 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:12:39.146 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-22T23:12:39.146 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:12:39.146 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-04-22T23:12:39.146 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:12:39.146 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 86 k 2024-04-22T23:12:39.146 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 226 k 2024-04-22T23:12:39.146 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-22T23:12:39.216 INFO:teuthology.orchestra.run.smithi067.stdout:rbd-fuse-18.2.1-0.el9.x86_64.rpm 1.2 MB/s | 86 kB 00:00 2024-04-22T23:12:39.217 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-22T23:12:39.217 INFO:teuthology.orchestra.run.smithi067.stdout:Total 1.2 MB/s | 86 kB 00:00 2024-04-22T23:12:39.217 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-22T23:12:39.225 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-22T23:12:39.225 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-22T23:12:39.291 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-22T23:12:39.291 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-22T23:12:39.437 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-22T23:12:39.531 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T23:12:39.877 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T23:12:40.132 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-fuse-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T23:12:40.132 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:12:40.132 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-22T23:12:40.132 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse-2:18.2.1-0.el9.x86_64 2024-04-22T23:12:40.132 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:12:40.132 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-22T23:12:40.369 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install rbd-mirror 2024-04-22T23:12:40.909 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:41 ago on Mon 22 Apr 2024 11:10:59 PM UTC. 2024-04-22T23:12:41.034 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-22T23:12:41.034 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:12:41.034 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-04-22T23:12:41.035 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:12:41.035 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-22T23:12:41.035 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror x86_64 2:18.2.1-0.el9 ceph 3.0 M 2024-04-22T23:12:41.035 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:12:41.035 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-22T23:12:41.035 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:12:41.035 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-04-22T23:12:41.035 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:12:41.035 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 3.0 M 2024-04-22T23:12:41.035 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 12 M 2024-04-22T23:12:41.035 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-22T23:12:41.136 INFO:teuthology.orchestra.run.smithi067.stdout:rbd-mirror-18.2.1-0.el9.x86_64.rpm 30 MB/s | 3.0 MB 00:00 2024-04-22T23:12:41.138 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-22T23:12:41.138 INFO:teuthology.orchestra.run.smithi067.stdout:Total 29 MB/s | 3.0 MB 00:00 2024-04-22T23:12:41.138 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-22T23:12:41.147 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-22T23:12:41.147 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-22T23:12:41.231 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-22T23:12:41.232 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-22T23:12:41.669 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-22T23:12:41.754 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T23:12:41.780 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T23:12:41.780 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T23:12:41.780 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-22T23:12:41.780 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T23:12:41.780 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T23:12:41.780 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:12:42.764 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-mirror-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T23:12:42.764 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:12:42.764 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-22T23:12:42.764 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror-2:18.2.1-0.el9.x86_64 2024-04-22T23:12:42.765 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:12:42.765 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-22T23:12:43.018 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install rbd-nbd 2024-04-22T23:12:43.560 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:44 ago on Mon 22 Apr 2024 11:10:59 PM UTC. 2024-04-22T23:12:43.688 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-22T23:12:43.688 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:12:43.689 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-04-22T23:12:43.689 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:12:43.689 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-22T23:12:43.689 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd x86_64 2:18.2.1-0.el9 ceph 171 k 2024-04-22T23:12:43.689 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:12:43.689 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-22T23:12:43.689 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-22T23:12:43.689 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-04-22T23:12:43.690 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:12:43.690 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 171 k 2024-04-22T23:12:43.690 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 494 k 2024-04-22T23:12:43.691 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-22T23:12:43.757 INFO:teuthology.orchestra.run.smithi067.stdout:rbd-nbd-18.2.1-0.el9.x86_64.rpm 2.5 MB/s | 171 kB 00:00 2024-04-22T23:12:43.757 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-22T23:12:43.757 INFO:teuthology.orchestra.run.smithi067.stdout:Total 2.4 MB/s | 171 kB 00:00 2024-04-22T23:12:43.758 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-22T23:12:43.766 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-22T23:12:43.766 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-22T23:12:43.833 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-22T23:12:43.833 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-22T23:12:43.988 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-22T23:12:44.097 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T23:12:44.462 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T23:12:44.724 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-nbd-2:18.2.1-0.el9.x86_64 1/1 2024-04-22T23:12:44.724 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:12:44.724 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-22T23:12:44.724 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd-2:18.2.1-0.el9.x86_64 2024-04-22T23:12:44.724 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:12:44.724 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-22T23:12:44.962 DEBUG:teuthology.parallel:result is None 2024-04-22T23:12:44.963 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-22T23:12:44.963 INFO:teuthology.packaging:ref: None 2024-04-22T23:12:44.963 INFO:teuthology.packaging:tag: v18.2.1 2024-04-22T23:12:44.963 INFO:teuthology.packaging:branch: None 2024-04-22T23:12:44.963 INFO:teuthology.packaging:sha1: 43be020184947e53516056c9931e1ac5bdbbb1a5 2024-04-22T23:12:44.963 INFO:teuthology.packaging:Tag 'v18.2.1' not found in https://git.ceph.com/ceph-ci; will also look in https://git.ceph.com/ceph 2024-04-22T23:12:44.963 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-04-22T23:12:45.189 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-22T23:12:45.211 INFO:teuthology.orchestra.run.smithi067.stdout:18.2.1-0.el9 2024-04-22T23:12:45.211 INFO:teuthology.packaging:The installed version of ceph is 18.2.1-0.el9 2024-04-22T23:12:45.212 INFO:teuthology.task.install:The correct ceph version 18.2.1-0 is installed. 2024-04-22T23:12:45.213 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-22T23:12:45.213 INFO:teuthology.packaging:ref: None 2024-04-22T23:12:45.213 INFO:teuthology.packaging:tag: v18.2.1 2024-04-22T23:12:45.213 INFO:teuthology.packaging:branch: None 2024-04-22T23:12:45.213 INFO:teuthology.packaging:sha1: 43be020184947e53516056c9931e1ac5bdbbb1a5 2024-04-22T23:12:45.214 INFO:teuthology.packaging:Tag 'v18.2.1' not found in https://git.ceph.com/ceph-ci; will also look in https://git.ceph.com/ceph 2024-04-22T23:12:45.214 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-04-22T23:12:45.429 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-22T23:12:45.453 INFO:teuthology.orchestra.run.smithi134.stdout:18.2.1-0.el9 2024-04-22T23:12:45.453 INFO:teuthology.packaging:The installed version of ceph is 18.2.1-0.el9 2024-04-22T23:12:45.454 INFO:teuthology.task.install:The correct ceph version 18.2.1-0 is installed. 2024-04-22T23:12:45.455 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-22T23:12:45.456 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-22T23:12:45.456 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-22T23:12:45.481 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-22T23:12:45.481 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-22T23:12:45.521 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-22T23:12:45.521 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-22T23:12:45.521 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/daemon-helper 2024-04-22T23:12:45.545 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-22T23:12:45.634 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-22T23:12:45.634 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/daemon-helper 2024-04-22T23:12:45.661 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-22T23:12:45.728 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-22T23:12:45.728 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-22T23:12:45.728 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-22T23:12:45.752 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-22T23:12:45.817 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-22T23:12:45.817 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-22T23:12:45.841 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-22T23:12:45.904 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-22T23:12:45.904 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-22T23:12:45.904 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/stdin-killer 2024-04-22T23:12:45.928 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-22T23:12:45.993 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-22T23:12:45.993 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/stdin-killer 2024-04-22T23:12:46.017 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-22T23:12:46.081 INFO:teuthology.run_tasks:Running task print... 2024-04-22T23:12:46.090 INFO:teuthology.task.print:**** done install task... 2024-04-22T23:12:46.091 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-22T23:12:46.235 INFO:tasks.cephadm:Config: {'compiled_cephadm_branch': 'reef', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}}, 'image': 'quay.io/ceph/ceph:v18.2.1', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': '43be020184947e53516056c9931e1ac5bdbbb1a5'} 2024-04-22T23:12:46.235 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v18.2.1 2024-04-22T23:12:46.236 INFO:tasks.cephadm:Cluster fsid is d46c346e-00fd-11ef-bc93-c7b262605968 2024-04-22T23:12:46.236 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-22T23:12:46.236 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-04-22T23:12:46.236 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi067': '172.21.15.67', 'mon.smithi134': '172.21.15.134'} 2024-04-22T23:12:46.236 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-22T23:12:46.236 DEBUG:teuthology.orchestra.run.smithi067:> sudo hostname $(hostname -s) 2024-04-22T23:12:46.262 DEBUG:teuthology.orchestra.run.smithi134:> sudo hostname $(hostname -s) 2024-04-22T23:12:46.286 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra for reef 2024-04-22T23:12:46.287 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=43be020184947e53516056c9931e1ac5bdbbb1a5 2024-04-22T23:12:46.510 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '43be020184947e53516056c9931e1ac5bdbbb1a5', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78807/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3252-g43be0201', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3252.g43be0201'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-22-1618/43be020184947e53516056c9931e1ac5bdbbb1a5/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-22 22:31:25.102991', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-adk-testing-2024-04-22-1618', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-adk-testing-2024-04-22-1618/43be020184947e53516056c9931e1ac5bdbbb1a5/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-22T23:12:46.815 INFO:tasks.util.chacra:got chacra host chacra.ceph.com, ref reef, sha1 9d98999acebbad2663cc4938e94c9ffd83485cbd from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&ref=reef 2024-04-22T23:12:46.816 INFO:tasks.cephadm:Discovered cachra url: https://chacra.ceph.com/binaries/ceph/reef/9d98999acebbad2663cc4938e94c9ffd83485cbd/centos/9/x86_64/flavors/default/cephadm 2024-04-22T23:12:46.816 DEBUG:teuthology.orchestra.run.smithi067:> curl --silent -L https://chacra.ceph.com/binaries/ceph/reef/9d98999acebbad2663cc4938e94c9ffd83485cbd/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-22T23:12:47.162 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r--. 1 ubuntu ubuntu 212578 Apr 22 23:12 /home/ubuntu/cephtest/cephadm 2024-04-22T23:12:47.162 DEBUG:teuthology.orchestra.run.smithi134:> curl --silent -L https://chacra.ceph.com/binaries/ceph/reef/9d98999acebbad2663cc4938e94c9ffd83485cbd/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-22T23:12:47.467 INFO:teuthology.orchestra.run.smithi134.stdout:-rw-r--r--. 1 ubuntu ubuntu 212578 Apr 22 23:12 /home/ubuntu/cephtest/cephadm 2024-04-22T23:12:47.467 DEBUG:teuthology.orchestra.run.smithi067:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-22T23:12:47.487 DEBUG:teuthology.orchestra.run.smithi134:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-22T23:12:47.515 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v18.2.1 on all hosts... 2024-04-22T23:12:47.515 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 pull 2024-04-22T23:12:47.530 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 pull 2024-04-22T23:12:48.923 INFO:teuthology.orchestra.run.smithi067.stderr:Pulling container image quay.io/ceph/ceph:v18.2.1... 2024-04-22T23:12:49.100 INFO:teuthology.orchestra.run.smithi134.stderr:Pulling container image quay.io/ceph/ceph:v18.2.1... 2024-04-22T23:13:07.929 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-04-22T23:13:07.929 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph_version": "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)", 2024-04-22T23:13:07.929 INFO:teuthology.orchestra.run.smithi134.stdout: "image_id": "5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf", 2024-04-22T23:13:07.929 INFO:teuthology.orchestra.run.smithi134.stdout: "repo_digests": [ 2024-04-22T23:13:07.929 INFO:teuthology.orchestra.run.smithi134.stdout: "quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3", 2024-04-22T23:13:07.929 INFO:teuthology.orchestra.run.smithi134.stdout: "quay.io/ceph/ceph@sha256:e8e55db8b4fd270dbec25bc764437a2a3abb707971c4dba5f559fb83018049dc" 2024-04-22T23:13:07.929 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-04-22T23:13:07.929 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-04-22T23:13:09.444 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:13:09.445 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_version": "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)", 2024-04-22T23:13:09.445 INFO:teuthology.orchestra.run.smithi067.stdout: "image_id": "5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf", 2024-04-22T23:13:09.445 INFO:teuthology.orchestra.run.smithi067.stdout: "repo_digests": [ 2024-04-22T23:13:09.445 INFO:teuthology.orchestra.run.smithi067.stdout: "quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3", 2024-04-22T23:13:09.445 INFO:teuthology.orchestra.run.smithi067.stdout: "quay.io/ceph/ceph@sha256:e8e55db8b4fd270dbec25bc764437a2a3abb707971c4dba5f559fb83018049dc" 2024-04-22T23:13:09.445 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-22T23:13:09.445 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:13:09.466 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /etc/ceph 2024-04-22T23:13:09.519 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /etc/ceph 2024-04-22T23:13:09.551 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 777 /etc/ceph 2024-04-22T23:13:09.576 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 777 /etc/ceph 2024-04-22T23:13:09.617 INFO:tasks.cephadm:Writing seed config... 2024-04-22T23:13:09.618 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-04-22T23:13:09.618 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-04-22T23:13:09.619 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-04-22T23:13:09.619 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-04-22T23:13:09.619 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-04-22T23:13:09.619 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-22T23:13:09.619 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-22T23:13:09.619 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-04-22T23:13:09.619 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-04-22T23:13:09.619 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-22T23:13:09.619 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-22T23:13:09.619 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-04-22T23:13:09.619 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-22T23:13:09.620 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-22T23:13:09.620 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-22T23:13:09.620 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-22T23:13:09.620 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-22T23:13:09.620 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-22T23:13:09.620 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-22T23:13:09.620 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-22T23:13:09.620 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-22T23:13:09.620 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-22T23:13:09.620 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-22T23:13:09.620 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-04-22T23:13:09.620 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-22T23:13:09.621 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-22T23:13:09.621 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-22T23:13:09.621 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-22T23:13:09.621 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-22T23:13:09.621 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-22T23:13:09.621 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-22T23:13:09.621 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-22T23:13:09.621 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-22T23:13:09.621 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-22T23:13:09.621 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-22T23:13:09.621 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-22T23:13:09.621 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-22T23:13:09.621 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-22T23:13:09.621 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-22T23:13:09.621 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-22T23:13:09.622 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-22T23:13:09.622 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-22T23:13:09.622 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-22T23:13:09.623 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-22T23:13:09.639 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = d46c346e-00fd-11ef-bc93-c7b262605968 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-22T23:13:09.639 DEBUG:teuthology.orchestra.run.smithi067:mon.smithi067> sudo journalctl -f -n 0 -u ceph-d46c346e-00fd-11ef-bc93-c7b262605968@mon.smithi067.service 2024-04-22T23:13:09.681 INFO:tasks.cephadm:Bootstrapping... 2024-04-22T23:13:09.681 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 -v bootstrap --fsid d46c346e-00fd-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.67 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-22T23:13:09.840 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-22T23:13:09.840 INFO:teuthology.orchestra.run.smithi067.stdout:cephadm ['--image', 'quay.io/ceph/ceph:v18.2.1', '-v', 'bootstrap', '--fsid', 'd46c346e-00fd-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.67', '--skip-admin-label'] 2024-04-22T23:13:09.866 INFO:teuthology.orchestra.run.smithi067.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-22T23:13:09.867 INFO:teuthology.orchestra.run.smithi067.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-22T23:13:09.867 INFO:teuthology.orchestra.run.smithi067.stdout:Verifying podman|docker is present... 2024-04-22T23:13:09.890 INFO:teuthology.orchestra.run.smithi067.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-22T23:13:09.890 INFO:teuthology.orchestra.run.smithi067.stdout:Verifying lvm2 is present... 2024-04-22T23:13:09.890 INFO:teuthology.orchestra.run.smithi067.stdout:Verifying time synchronization is in place... 2024-04-22T23:13:09.898 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-22T23:13:09.898 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-22T23:13:09.904 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-22T23:13:09.904 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout inactive 2024-04-22T23:13:09.910 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout enabled 2024-04-22T23:13:09.916 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout active 2024-04-22T23:13:09.916 INFO:teuthology.orchestra.run.smithi067.stdout:Unit chronyd.service is enabled and running 2024-04-22T23:13:09.917 INFO:teuthology.orchestra.run.smithi067.stdout:Repeating the final host check... 2024-04-22T23:13:09.940 INFO:teuthology.orchestra.run.smithi067.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-22T23:13:09.941 INFO:teuthology.orchestra.run.smithi067.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-22T23:13:09.941 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl is present 2024-04-22T23:13:09.941 INFO:teuthology.orchestra.run.smithi067.stdout:lvcreate is present 2024-04-22T23:13:09.947 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-22T23:13:09.947 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-22T23:13:09.953 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-22T23:13:09.953 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout inactive 2024-04-22T23:13:09.959 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout enabled 2024-04-22T23:13:09.966 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout active 2024-04-22T23:13:09.966 INFO:teuthology.orchestra.run.smithi067.stdout:Unit chronyd.service is enabled and running 2024-04-22T23:13:09.966 INFO:teuthology.orchestra.run.smithi067.stdout:Host looks OK 2024-04-22T23:13:09.966 INFO:teuthology.orchestra.run.smithi067.stdout:Cluster fsid: d46c346e-00fd-11ef-bc93-c7b262605968 2024-04-22T23:13:09.966 INFO:teuthology.orchestra.run.smithi067.stdout:Acquiring lock 139761415228624 on /run/cephadm/d46c346e-00fd-11ef-bc93-c7b262605968.lock 2024-04-22T23:13:09.966 INFO:teuthology.orchestra.run.smithi067.stdout:Lock 139761415228624 acquired on /run/cephadm/d46c346e-00fd-11ef-bc93-c7b262605968.lock 2024-04-22T23:13:09.967 INFO:teuthology.orchestra.run.smithi067.stdout:Verifying IP 172.21.15.67 port 3300 ... 2024-04-22T23:13:09.967 INFO:teuthology.orchestra.run.smithi067.stdout:Verifying IP 172.21.15.67 port 6789 ... 2024-04-22T23:13:09.967 INFO:teuthology.orchestra.run.smithi067.stdout:Base mon IP(s) is [172.21.15.67:3300, 172.21.15.67:6789], mon addrv is [v2:172.21.15.67:3300,v1:172.21.15.67:6789] 2024-04-22T23:13:09.970 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.67 metric 100 2024-04-22T23:13:09.970 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.67 metric 100 2024-04-22T23:13:09.973 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-22T23:13:09.973 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-04-22T23:13:09.973 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-04-22T23:13:09.975 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-22T23:13:09.975 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-22T23:13:09.975 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-22T23:13:09.976 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-22T23:13:09.976 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout inet6 fe80::b0a1:74d2:2efb:1383/64 scope link noprefixroute 2024-04-22T23:13:09.976 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-22T23:13:09.976 INFO:teuthology.orchestra.run.smithi067.stdout:Mon IP `172.21.15.67` is in CIDR network `172.21.0.0/20` 2024-04-22T23:13:09.976 INFO:teuthology.orchestra.run.smithi067.stdout:Mon IP `172.21.15.67` is in CIDR network `172.21.0.0/20` 2024-04-22T23:13:09.977 INFO:teuthology.orchestra.run.smithi067.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-22T23:13:09.977 INFO:teuthology.orchestra.run.smithi067.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-22T23:13:09.978 INFO:teuthology.orchestra.run.smithi067.stdout:Pulling container image quay.io/ceph/ceph:v18.2.1... 2024-04-22T23:13:10.713 INFO:teuthology.orchestra.run.smithi067.stdout:/bin/podman: stdout 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf 2024-04-22T23:13:10.713 INFO:teuthology.orchestra.run.smithi067.stdout:/bin/podman: stderr Trying to pull quay.io/ceph/ceph:v18.2.1... 2024-04-22T23:13:10.713 INFO:teuthology.orchestra.run.smithi067.stdout:/bin/podman: stderr Getting image source signatures 2024-04-22T23:13:10.713 INFO:teuthology.orchestra.run.smithi067.stdout:/bin/podman: stderr Copying blob sha256:a733d3c618b71f19c168ebecd1953429dce2c1631835ca182e9551c36dce5989 2024-04-22T23:13:10.713 INFO:teuthology.orchestra.run.smithi067.stdout:/bin/podman: stderr Copying blob sha256:7feca07754707458c3945cf0062cf4dabc512f6d90fe1a9a1370b362b6011124 2024-04-22T23:13:10.714 INFO:teuthology.orchestra.run.smithi067.stdout:/bin/podman: stderr Copying config sha256:5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf 2024-04-22T23:13:10.714 INFO:teuthology.orchestra.run.smithi067.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-22T23:13:11.601 INFO:teuthology.orchestra.run.smithi067.stdout:ceph: stdout ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable) 2024-04-22T23:13:11.602 INFO:teuthology.orchestra.run.smithi067.stdout:Ceph version: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable) 2024-04-22T23:13:11.602 INFO:teuthology.orchestra.run.smithi067.stdout:Extracting ceph user uid/gid from container image... 2024-04-22T23:13:12.750 INFO:teuthology.orchestra.run.smithi067.stdout:stat: stdout 167 167 2024-04-22T23:13:12.750 INFO:teuthology.orchestra.run.smithi067.stdout:Creating initial keys... 2024-04-22T23:13:13.656 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph-authtool: stdout AQAJ7yZmfsAjAxAAGdSYFYi52YYr8764fhD7/A== 2024-04-22T23:13:14.537 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph-authtool: stdout AQAJ7yZmnyDtOBAA1/phJTzJwMWv+zhwykVrOg== 2024-04-22T23:13:15.433 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph-authtool: stdout AQAK7yZmB19RMxAAMJxre5RQBeqFeuD80j6Blw== 2024-04-22T23:13:15.434 INFO:teuthology.orchestra.run.smithi067.stdout:Creating initial monmap... 2024-04-22T23:13:16.316 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-22T23:13:16.316 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-04-22T23:13:16.316 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to d46c346e-00fd-11ef-bc93-c7b262605968 2024-04-22T23:13:16.316 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-22T23:13:16.316 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool for smithi067 [v2:172.21.15.67:3300,v1:172.21.15.67:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-22T23:13:16.317 INFO:teuthology.orchestra.run.smithi067.stdout:setting min_mon_release = pacific 2024-04-22T23:13:16.317 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/monmaptool: set fsid to d46c346e-00fd-11ef-bc93-c7b262605968 2024-04-22T23:13:16.317 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-22T23:13:16.317 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:13:16.317 INFO:teuthology.orchestra.run.smithi067.stdout:Creating mon... 2024-04-22T23:13:17.210 INFO:teuthology.orchestra.run.smithi067.stdout:create mon.smithi067 on 2024-04-22T23:13:17.433 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-22T23:13:17.585 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-22T23:13:17.785 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-d46c346e-00fd-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-d46c346e-00fd-11ef-bc93-c7b262605968.target. 2024-04-22T23:13:17.785 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-d46c346e-00fd-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-d46c346e-00fd-11ef-bc93-c7b262605968.target. 2024-04-22T23:13:18.002 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-d46c346e-00fd-11ef-bc93-c7b262605968@mon.smithi067 2024-04-22T23:13:18.002 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Failed to reset failed state of unit ceph-d46c346e-00fd-11ef-bc93-c7b262605968@mon.smithi067.service: Unit ceph-d46c346e-00fd-11ef-bc93-c7b262605968@mon.smithi067.service not loaded. 2024-04-22T23:13:18.170 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-d46c346e-00fd-11ef-bc93-c7b262605968.target.wants/ceph-d46c346e-00fd-11ef-bc93-c7b262605968@mon.smithi067.service → /etc/systemd/system/ceph-d46c346e-00fd-11ef-bc93-c7b262605968@.service. 2024-04-22T23:13:18.742 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-22T23:13:18.742 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout disabled 2024-04-22T23:13:18.748 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-22T23:13:18.748 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout inactive 2024-04-22T23:13:18.748 INFO:teuthology.orchestra.run.smithi067.stdout:firewalld.service is not enabled 2024-04-22T23:13:18.748 INFO:teuthology.orchestra.run.smithi067.stdout:Not possible to enable service . firewalld.service is not available 2024-04-22T23:13:18.749 INFO:teuthology.orchestra.run.smithi067.stdout:Waiting for mon to start... 2024-04-22T23:13:18.749 INFO:teuthology.orchestra.run.smithi067.stdout:Waiting for mon... 2024-04-22T23:13:18.862 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:18 smithi067 systemd[1]: Started Ceph mon.smithi067 for d46c346e-00fd-11ef-bc93-c7b262605968. 2024-04-22T23:13:19.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:19 smithi067 ceph-mon[26080]: mon.smithi067 is new leader, mons smithi067 in quorum (ranks 0) 2024-04-22T23:13:19.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:19 smithi067 ceph-mon[26080]: monmap e1: 1 mons at {smithi067=[v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-22T23:13:19.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:19 smithi067 ceph-mon[26080]: fsmap 2024-04-22T23:13:19.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:19 smithi067 ceph-mon[26080]: osdmap e1: 0 total, 0 up, 0 in 2024-04-22T23:13:19.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:19 smithi067 ceph-mon[26080]: mgrmap e1: no daemons active 2024-04-22T23:13:19.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:19 smithi067 ceph-mon[26080]: from='client.? 172.21.15.67:0/2305597336' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-22T23:13:19.805 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout cluster: 2024-04-22T23:13:19.812 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout id: d46c346e-00fd-11ef-bc93-c7b262605968 2024-04-22T23:13:19.812 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-22T23:13:19.812 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-04-22T23:13:19.812 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout services: 2024-04-22T23:13:19.812 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi067 (age 0.540726s) 2024-04-22T23:13:19.812 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-22T23:13:19.812 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-22T23:13:19.812 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-04-22T23:13:19.812 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout data: 2024-04-22T23:13:19.812 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-22T23:13:19.812 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-22T23:13:19.812 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-22T23:13:19.813 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout pgs: 2024-04-22T23:13:19.813 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-04-22T23:13:19.813 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.148+0000 7f7f46792700 1 Processor -- start 2024-04-22T23:13:19.813 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.148+0000 7f7f46792700 1 -- start start 2024-04-22T23:13:19.813 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.148+0000 7f7f46792700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f40104bd0 0x7f7f40104ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:13:19.813 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.148+0000 7f7f46792700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f401055c0 con 0x7f7f40104bd0 2024-04-22T23:13:19.813 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.148+0000 7f7f3ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f40104bd0 0x7f7f40104ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:13:19.813 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.148+0000 7f7f3ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f40104bd0 0x7f7f40104ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36032/0 (socket says 172.21.15.67:36032) 2024-04-22T23:13:19.813 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.148+0000 7f7f3ffff700 1 -- 172.21.15.67:0/795792952 learned_addr learned my addr 172.21.15.67:0/795792952 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:13:19.813 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.149+0000 7f7f3ffff700 1 -- 172.21.15.67:0/795792952 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f40105dd0 con 0x7f7f40104bd0 2024-04-22T23:13:19.813 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.149+0000 7f7f3ffff700 1 --2- 172.21.15.67:0/795792952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f40104bd0 0x7f7f40104ff0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7f3000aa40 tx=0x7f7f3000ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6a6f5269bb848c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:13:19.813 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.149+0000 7f7f3effd700 1 -- 172.21.15.67:0/795792952 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f30004180 con 0x7f7f40104bd0 2024-04-22T23:13:19.813 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.149+0000 7f7f3effd700 1 -- 172.21.15.67:0/795792952 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f300042e0 con 0x7f7f40104bd0 2024-04-22T23:13:19.813 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.149+0000 7f7f3effd700 1 -- 172.21.15.67:0/795792952 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f3000c8f0 con 0x7f7f40104bd0 2024-04-22T23:13:19.814 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.150+0000 7f7f46792700 1 -- 172.21.15.67:0/795792952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f40104bd0 msgr2=0x7f7f40104ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:13:19.814 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.150+0000 7f7f46792700 1 --2- 172.21.15.67:0/795792952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f40104bd0 0x7f7f40104ff0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7f3000aa40 tx=0x7f7f3000ad50 comp rx=0 tx=0).stop 2024-04-22T23:13:19.814 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.150+0000 7f7f46792700 1 -- 172.21.15.67:0/795792952 shutdown_connections 2024-04-22T23:13:19.814 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.150+0000 7f7f46792700 1 --2- 172.21.15.67:0/795792952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f40104bd0 0x7f7f40104ff0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:13:19.814 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.150+0000 7f7f46792700 1 -- 172.21.15.67:0/795792952 >> 172.21.15.67:0/795792952 conn(0x7f7f40100170 msgr2=0x7f7f401025b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:13:19.814 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.150+0000 7f7f46792700 1 -- 172.21.15.67:0/795792952 shutdown_connections 2024-04-22T23:13:19.814 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.150+0000 7f7f46792700 1 -- 172.21.15.67:0/795792952 wait complete. 2024-04-22T23:13:19.814 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.150+0000 7f7f46792700 1 Processor -- start 2024-04-22T23:13:19.814 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.150+0000 7f7f46792700 1 -- start start 2024-04-22T23:13:19.814 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.151+0000 7f7f46792700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f40104bd0 0x7f7f401999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:13:19.814 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.151+0000 7f7f46792700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f40199ef0 con 0x7f7f40104bd0 2024-04-22T23:13:19.814 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.151+0000 7f7f3ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f40104bd0 0x7f7f401999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:13:19.814 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.151+0000 7f7f3ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f40104bd0 0x7f7f401999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36040/0 (socket says 172.21.15.67:36040) 2024-04-22T23:13:19.815 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.151+0000 7f7f3ffff700 1 -- 172.21.15.67:0/2305597336 learned_addr learned my addr 172.21.15.67:0/2305597336 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:13:19.815 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.151+0000 7f7f3ffff700 1 -- 172.21.15.67:0/2305597336 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f3000a6f0 con 0x7f7f40104bd0 2024-04-22T23:13:19.815 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.151+0000 7f7f3ffff700 1 --2- 172.21.15.67:0/2305597336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f40104bd0 0x7f7f401999b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7f30004000 tx=0x7f7f30004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:13:19.815 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.151+0000 7f7f3d7fa700 1 -- 172.21.15.67:0/2305597336 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f3001d070 con 0x7f7f40104bd0 2024-04-22T23:13:19.815 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.151+0000 7f7f3d7fa700 1 -- 172.21.15.67:0/2305597336 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f300038b0 con 0x7f7f40104bd0 2024-04-22T23:13:19.815 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.151+0000 7f7f3d7fa700 1 -- 172.21.15.67:0/2305597336 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f30003b70 con 0x7f7f40104bd0 2024-04-22T23:13:19.815 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.151+0000 7f7f46792700 1 -- 172.21.15.67:0/2305597336 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f4019a0f0 con 0x7f7f40104bd0 2024-04-22T23:13:19.815 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.152+0000 7f7f46792700 1 -- 172.21.15.67:0/2305597336 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f4019a590 con 0x7f7f40104bd0 2024-04-22T23:13:19.815 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.152+0000 7f7f3d7fa700 1 -- 172.21.15.67:0/2305597336 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7f30003cd0 con 0x7f7f40104bd0 2024-04-22T23:13:19.815 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.152+0000 7f7f3d7fa700 1 -- 172.21.15.67:0/2305597336 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7f30021410 con 0x7f7f40104bd0 2024-04-22T23:13:19.816 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.152+0000 7f7f46792700 1 -- 172.21.15.67:0/2305597336 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f40193670 con 0x7f7f40104bd0 2024-04-22T23:13:19.816 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.155+0000 7f7f3d7fa700 1 -- 172.21.15.67:0/2305597336 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f7f30030060 con 0x7f7f40104bd0 2024-04-22T23:13:19.816 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.226+0000 7f7f46792700 1 -- 172.21.15.67:0/2305597336 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f7f4002cc70 con 0x7f7f40104bd0 2024-04-22T23:13:19.816 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.227+0000 7f7f3d7fa700 1 -- 172.21.15.67:0/2305597336 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f7f30026070 con 0x7f7f40104bd0 2024-04-22T23:13:19.816 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.227+0000 7f7f46792700 1 -- 172.21.15.67:0/2305597336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f40104bd0 msgr2=0x7f7f401999b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:13:19.816 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.227+0000 7f7f46792700 1 --2- 172.21.15.67:0/2305597336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f40104bd0 0x7f7f401999b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7f30004000 tx=0x7f7f30004750 comp rx=0 tx=0).stop 2024-04-22T23:13:19.816 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.227+0000 7f7f46792700 1 -- 172.21.15.67:0/2305597336 shutdown_connections 2024-04-22T23:13:19.816 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.227+0000 7f7f46792700 1 --2- 172.21.15.67:0/2305597336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f40104bd0 0x7f7f401999b0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:13:19.816 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.227+0000 7f7f46792700 1 -- 172.21.15.67:0/2305597336 >> 172.21.15.67:0/2305597336 conn(0x7f7f40100170 msgr2=0x7f7f401907b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:13:19.816 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.227+0000 7f7f46792700 1 -- 172.21.15.67:0/2305597336 shutdown_connections 2024-04-22T23:13:19.816 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:19.228+0000 7f7f46792700 1 -- 172.21.15.67:0/2305597336 wait complete. 2024-04-22T23:13:19.816 INFO:teuthology.orchestra.run.smithi067.stdout:mon is available 2024-04-22T23:13:19.816 INFO:teuthology.orchestra.run.smithi067.stdout:Assimilating anything we can from ceph.conf... 2024-04-22T23:13:20.859 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-04-22T23:13:20.859 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout [global] 2024-04-22T23:13:20.859 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout fsid = d46c346e-00fd-11ef-bc93-c7b262605968 2024-04-22T23:13:20.859 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.67:3300,v1:172.21.15.67:6789] 2024-04-22T23:13:20.859 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-22T23:13:20.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-22T23:13:20.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-22T23:13:20.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-22T23:13:20.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-04-22T23:13:20.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-22T23:13:20.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-22T23:13:20.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-04-22T23:13:20.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout [osd] 2024-04-22T23:13:20.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-22T23:13:20.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-22T23:13:20.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-22T23:13:20.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-22T23:13:20.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-22T23:13:20.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.230+0000 7fbcda9a1700 1 Processor -- start 2024-04-22T23:13:20.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.230+0000 7fbcda9a1700 1 -- start start 2024-04-22T23:13:20.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.230+0000 7fbcda9a1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd4104a10 0x7fbcd4104e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:13:20.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.230+0000 7fbcda9a1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd4105400 con 0x7fbcd4104a10 2024-04-22T23:13:20.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.230+0000 7fbcd999f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd4104a10 0x7fbcd4104e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:13:20.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.230+0000 7fbcd999f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd4104a10 0x7fbcd4104e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36054/0 (socket says 172.21.15.67:36054) 2024-04-22T23:13:20.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.230+0000 7fbcd999f700 1 -- 172.21.15.67:0/514802881 learned_addr learned my addr 172.21.15.67:0/514802881 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:13:20.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.231+0000 7fbcd999f700 1 -- 172.21.15.67:0/514802881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd4105c10 con 0x7fbcd4104a10 2024-04-22T23:13:20.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.231+0000 7fbcd999f700 1 --2- 172.21.15.67:0/514802881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd4104a10 0x7fbcd4104e30 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fbcd00098a0 tx=0x7fbcd0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a672446c960e04c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:13:20.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.231+0000 7fbcd899d700 1 -- 172.21.15.67:0/514802881 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcd0004180 con 0x7fbcd4104a10 2024-04-22T23:13:20.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.231+0000 7fbcd899d700 1 -- 172.21.15.67:0/514802881 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcd00042e0 con 0x7fbcd4104a10 2024-04-22T23:13:20.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.231+0000 7fbcd899d700 1 -- 172.21.15.67:0/514802881 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcd00036a0 con 0x7fbcd4104a10 2024-04-22T23:13:20.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.232+0000 7fbcda9a1700 1 -- 172.21.15.67:0/514802881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd4104a10 msgr2=0x7fbcd4104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:13:20.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.232+0000 7fbcda9a1700 1 --2- 172.21.15.67:0/514802881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd4104a10 0x7fbcd4104e30 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fbcd00098a0 tx=0x7fbcd0009bb0 comp rx=0 tx=0).stop 2024-04-22T23:13:20.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.232+0000 7fbcda9a1700 1 -- 172.21.15.67:0/514802881 shutdown_connections 2024-04-22T23:13:20.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.232+0000 7fbcda9a1700 1 --2- 172.21.15.67:0/514802881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd4104a10 0x7fbcd4104e30 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:13:20.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.232+0000 7fbcda9a1700 1 -- 172.21.15.67:0/514802881 >> 172.21.15.67:0/514802881 conn(0x7fbcd40ffff0 msgr2=0x7fbcd4102410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:13:20.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.232+0000 7fbcda9a1700 1 -- 172.21.15.67:0/514802881 shutdown_connections 2024-04-22T23:13:20.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.232+0000 7fbcda9a1700 1 -- 172.21.15.67:0/514802881 wait complete. 2024-04-22T23:13:20.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.232+0000 7fbcda9a1700 1 Processor -- start 2024-04-22T23:13:20.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.232+0000 7fbcda9a1700 1 -- start start 2024-04-22T23:13:20.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.233+0000 7fbcda9a1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd4104a10 0x7fbcd419ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:13:20.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.233+0000 7fbcda9a1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd419bf50 con 0x7fbcd4104a10 2024-04-22T23:13:20.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.233+0000 7fbcd999f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd4104a10 0x7fbcd419ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:13:20.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.233+0000 7fbcd999f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd4104a10 0x7fbcd419ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36060/0 (socket says 172.21.15.67:36060) 2024-04-22T23:13:20.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.233+0000 7fbcd999f700 1 -- 172.21.15.67:0/2547907399 learned_addr learned my addr 172.21.15.67:0/2547907399 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:13:20.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.233+0000 7fbcd999f700 1 -- 172.21.15.67:0/2547907399 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd0009550 con 0x7fbcd4104a10 2024-04-22T23:13:20.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.233+0000 7fbcd999f700 1 --2- 172.21.15.67:0/2547907399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd4104a10 0x7fbcd419ba10 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fbcd0004000 tx=0x7fbcd0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:13:20.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.233+0000 7fbccaffd700 1 -- 172.21.15.67:0/2547907399 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcd0003ce0 con 0x7fbcd4104a10 2024-04-22T23:13:20.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.233+0000 7fbccaffd700 1 -- 172.21.15.67:0/2547907399 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcd0020070 con 0x7fbcd4104a10 2024-04-22T23:13:20.863 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.233+0000 7fbcda9a1700 1 -- 172.21.15.67:0/2547907399 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcd419c150 con 0x7fbcd4104a10 2024-04-22T23:13:20.863 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.233+0000 7fbccaffd700 1 -- 172.21.15.67:0/2547907399 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcd0011630 con 0x7fbcd4104a10 2024-04-22T23:13:20.863 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.233+0000 7fbcda9a1700 1 -- 172.21.15.67:0/2547907399 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcd419c5f0 con 0x7fbcd4104a10 2024-04-22T23:13:20.863 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.234+0000 7fbccaffd700 1 -- 172.21.15.67:0/2547907399 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fbcd0022030 con 0x7fbcd4104a10 2024-04-22T23:13:20.863 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.234+0000 7fbccaffd700 1 -- 172.21.15.67:0/2547907399 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fbcd0025d00 con 0x7fbcd4104a10 2024-04-22T23:13:20.863 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.234+0000 7fbcda9a1700 1 -- 172.21.15.67:0/2547907399 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcd4195710 con 0x7fbcd4104a10 2024-04-22T23:13:20.863 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.236+0000 7fbccaffd700 1 -- 172.21.15.67:0/2547907399 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fbcd0025480 con 0x7fbcd4104a10 2024-04-22T23:13:20.863 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.307+0000 7fbcda9a1700 1 -- 172.21.15.67:0/2547907399 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fbcd402ce00 con 0x7fbcd4104a10 2024-04-22T23:13:20.863 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.311+0000 7fbccaffd700 1 -- 172.21.15.67:0/2547907399 <== mon.0 v2:172.21.15.67:3300/0 7 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbcd0011c20 con 0x7fbcd4104a10 2024-04-22T23:13:20.863 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.311+0000 7fbccaffd700 1 -- 172.21.15.67:0/2547907399 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7fbcd0038590 con 0x7fbcd4104a10 2024-04-22T23:13:20.863 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.312+0000 7fbcda9a1700 1 -- 172.21.15.67:0/2547907399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd4104a10 msgr2=0x7fbcd419ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:13:20.863 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.312+0000 7fbcda9a1700 1 --2- 172.21.15.67:0/2547907399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd4104a10 0x7fbcd419ba10 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fbcd0004000 tx=0x7fbcd0004750 comp rx=0 tx=0).stop 2024-04-22T23:13:20.863 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.312+0000 7fbcda9a1700 1 -- 172.21.15.67:0/2547907399 shutdown_connections 2024-04-22T23:13:20.863 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.312+0000 7fbcda9a1700 1 --2- 172.21.15.67:0/2547907399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd4104a10 0x7fbcd419ba10 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:13:20.863 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.312+0000 7fbcda9a1700 1 -- 172.21.15.67:0/2547907399 >> 172.21.15.67:0/2547907399 conn(0x7fbcd40ffff0 msgr2=0x7fbcd4100c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:13:20.864 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.312+0000 7fbcda9a1700 1 -- 172.21.15.67:0/2547907399 shutdown_connections 2024-04-22T23:13:20.864 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:20.312+0000 7fbcda9a1700 1 -- 172.21.15.67:0/2547907399 wait complete. 2024-04-22T23:13:20.864 INFO:teuthology.orchestra.run.smithi067.stdout:Generating new minimal ceph.conf... 2024-04-22T23:13:20.976 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:20 smithi067 ceph-mon[26080]: from='client.? 172.21.15.67:0/2547907399' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-22T23:13:20.976 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:20 smithi067 ceph-mon[26080]: from='client.? 172.21.15.67:0/2547907399' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-22T23:13:21.908 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:21 smithi067 ceph-mon[26080]: from='client.? 172.21.15.67:0/2564758147' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:13:21.913 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.279+0000 7f9a6dcc4700 1 Processor -- start 2024-04-22T23:13:21.913 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.279+0000 7f9a6dcc4700 1 -- start start 2024-04-22T23:13:21.913 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.279+0000 7f9a6dcc4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a68104e00 0x7f9a68105220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:13:21.914 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.279+0000 7f9a6dcc4700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a68079e50 con 0x7f9a68104e00 2024-04-22T23:13:21.914 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.279+0000 7f9a677fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a68104e00 0x7f9a68105220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:13:21.914 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.279+0000 7f9a677fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a68104e00 0x7f9a68105220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36068/0 (socket says 172.21.15.67:36068) 2024-04-22T23:13:21.914 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.279+0000 7f9a677fe700 1 -- 172.21.15.67:0/2124019803 learned_addr learned my addr 172.21.15.67:0/2124019803 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:13:21.914 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.279+0000 7f9a677fe700 1 -- 172.21.15.67:0/2124019803 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a68105760 con 0x7f9a68104e00 2024-04-22T23:13:21.914 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.280+0000 7f9a677fe700 1 --2- 172.21.15.67:0/2124019803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a68104e00 0x7f9a68105220 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f9a580098a0 tx=0x7f9a58009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6a3d6e7c257f1a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:13:21.914 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.280+0000 7f9a667fc700 1 -- 172.21.15.67:0/2124019803 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a58004180 con 0x7f9a68104e00 2024-04-22T23:13:21.914 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.280+0000 7f9a667fc700 1 -- 172.21.15.67:0/2124019803 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9a580042e0 con 0x7f9a68104e00 2024-04-22T23:13:21.914 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.280+0000 7f9a667fc700 1 -- 172.21.15.67:0/2124019803 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a58011a50 con 0x7f9a68104e00 2024-04-22T23:13:21.914 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.280+0000 7f9a6dcc4700 1 -- 172.21.15.67:0/2124019803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a68104e00 msgr2=0x7f9a68105220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:13:21.914 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.280+0000 7f9a6dcc4700 1 --2- 172.21.15.67:0/2124019803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a68104e00 0x7f9a68105220 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f9a580098a0 tx=0x7f9a58009bb0 comp rx=0 tx=0).stop 2024-04-22T23:13:21.914 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.280+0000 7f9a6dcc4700 1 -- 172.21.15.67:0/2124019803 shutdown_connections 2024-04-22T23:13:21.914 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.280+0000 7f9a6dcc4700 1 --2- 172.21.15.67:0/2124019803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a68104e00 0x7f9a68105220 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:13:21.915 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.280+0000 7f9a6dcc4700 1 -- 172.21.15.67:0/2124019803 >> 172.21.15.67:0/2124019803 conn(0x7f9a68100a20 msgr2=0x7f9a68102e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:13:21.915 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.281+0000 7f9a6dcc4700 1 -- 172.21.15.67:0/2124019803 shutdown_connections 2024-04-22T23:13:21.915 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.281+0000 7f9a6dcc4700 1 -- 172.21.15.67:0/2124019803 wait complete. 2024-04-22T23:13:21.915 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.281+0000 7f9a6dcc4700 1 Processor -- start 2024-04-22T23:13:21.915 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.281+0000 7f9a6dcc4700 1 -- start start 2024-04-22T23:13:21.915 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.281+0000 7f9a6dcc4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a68104e00 0x7f9a6819c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:13:21.915 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.281+0000 7f9a6dcc4700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6819c920 con 0x7f9a68104e00 2024-04-22T23:13:21.915 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.282+0000 7f9a677fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a68104e00 0x7f9a6819c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:13:21.915 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.282+0000 7f9a677fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a68104e00 0x7f9a6819c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36082/0 (socket says 172.21.15.67:36082) 2024-04-22T23:13:21.915 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.282+0000 7f9a677fe700 1 -- 172.21.15.67:0/2564758147 learned_addr learned my addr 172.21.15.67:0/2564758147 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:13:21.915 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.282+0000 7f9a677fe700 1 -- 172.21.15.67:0/2564758147 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a58009550 con 0x7f9a68104e00 2024-04-22T23:13:21.915 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.282+0000 7f9a677fe700 1 --2- 172.21.15.67:0/2564758147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a68104e00 0x7f9a6819c3e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9a58003de0 tx=0x7f9a58011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:13:21.915 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.282+0000 7f9a64ff9700 1 -- 172.21.15.67:0/2564758147 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a58003930 con 0x7f9a68104e00 2024-04-22T23:13:21.915 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.282+0000 7f9a64ff9700 1 -- 172.21.15.67:0/2564758147 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9a58003a90 con 0x7f9a68104e00 2024-04-22T23:13:21.916 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.282+0000 7f9a64ff9700 1 -- 172.21.15.67:0/2564758147 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a58019730 con 0x7f9a68104e00 2024-04-22T23:13:21.916 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.282+0000 7f9a6dcc4700 1 -- 172.21.15.67:0/2564758147 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a6819cb20 con 0x7f9a68104e00 2024-04-22T23:13:21.916 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.282+0000 7f9a6dcc4700 1 -- 172.21.15.67:0/2564758147 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a6819cfc0 con 0x7f9a68104e00 2024-04-22T23:13:21.916 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.282+0000 7f9a64ff9700 1 -- 172.21.15.67:0/2564758147 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9a58019cb0 con 0x7f9a68104e00 2024-04-22T23:13:21.916 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.283+0000 7f9a64ff9700 1 -- 172.21.15.67:0/2564758147 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9a5802c430 con 0x7f9a68104e00 2024-04-22T23:13:21.916 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.283+0000 7f9a6dcc4700 1 -- 172.21.15.67:0/2564758147 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a68195cb0 con 0x7f9a68104e00 2024-04-22T23:13:21.916 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.285+0000 7f9a64ff9700 1 -- 172.21.15.67:0/2564758147 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f9a5801abc0 con 0x7f9a68104e00 2024-04-22T23:13:21.916 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.356+0000 7f9a6dcc4700 1 -- 172.21.15.67:0/2564758147 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f9a6802cc70 con 0x7f9a68104e00 2024-04-22T23:13:21.916 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.356+0000 7f9a64ff9700 1 -- 172.21.15.67:0/2564758147 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f9a58031030 con 0x7f9a68104e00 2024-04-22T23:13:21.916 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.357+0000 7f9a6dcc4700 1 -- 172.21.15.67:0/2564758147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a68104e00 msgr2=0x7f9a6819c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:13:21.916 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.357+0000 7f9a6dcc4700 1 --2- 172.21.15.67:0/2564758147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a68104e00 0x7f9a6819c3e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9a58003de0 tx=0x7f9a58011ab0 comp rx=0 tx=0).stop 2024-04-22T23:13:21.916 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.357+0000 7f9a6dcc4700 1 -- 172.21.15.67:0/2564758147 shutdown_connections 2024-04-22T23:13:21.917 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.357+0000 7f9a6dcc4700 1 --2- 172.21.15.67:0/2564758147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a68104e00 0x7f9a6819c3e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:13:21.917 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.357+0000 7f9a6dcc4700 1 -- 172.21.15.67:0/2564758147 >> 172.21.15.67:0/2564758147 conn(0x7f9a68100a20 msgr2=0x7f9a681909c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:13:21.917 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.357+0000 7f9a6dcc4700 1 -- 172.21.15.67:0/2564758147 shutdown_connections 2024-04-22T23:13:21.917 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:21.357+0000 7f9a6dcc4700 1 -- 172.21.15.67:0/2564758147 wait complete. 2024-04-22T23:13:21.917 INFO:teuthology.orchestra.run.smithi067.stdout:Restarting the monitor... 2024-04-22T23:13:22.169 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:21 smithi067 systemd[1]: Stopping Ceph mon.smithi067 for d46c346e-00fd-11ef-bc93-c7b262605968... 2024-04-22T23:13:22.435 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:22 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[26076]: 2024-04-22T23:13:22.167+0000 7f495aa62700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi067 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T23:13:22.435 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:22 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[26076]: 2024-04-22T23:13:22.167+0000 7f495aa62700 -1 mon.smithi067@0(leader) e1 *** Got Signal Terminated *** 2024-04-22T23:13:22.435 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:22 smithi067 podman[26323]: 2024-04-22 23:13:22.208821965 +0000 UTC m=+0.128011777 container died c62613c16c7dc0efef9fb8085e41c5a512f4379b583a6216e5600800be97a9da (image=quay.io/ceph/ceph:v18.2.1, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=HEAD, ceph=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-22T23:13:22.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:22 smithi067 podman[26323]: 2024-04-22 23:13:22.517103247 +0000 UTC m=+0.436293067 container cleanup c62613c16c7dc0efef9fb8085e41c5a512f4379b583a6216e5600800be97a9da (image=quay.io/ceph/ceph:v18.2.1, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067, GIT_CLEAN=True, io.buildah.version=1.29.1, RELEASE=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-04-22T23:13:22.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:22 smithi067 bash[26323]: ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067 2024-04-22T23:13:23.162 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:22 smithi067 podman[26337]: 2024-04-22 23:13:22.801212229 +0000 UTC m=+0.587139725 container remove c62613c16c7dc0efef9fb8085e41c5a512f4379b583a6216e5600800be97a9da (image=quay.io/ceph/ceph:v18.2.1, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, RELEASE=HEAD) 2024-04-22T23:13:23.163 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:22 smithi067 systemd[1]: ceph-d46c346e-00fd-11ef-bc93-c7b262605968@mon.smithi067.service: Deactivated successfully. 2024-04-22T23:13:23.163 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:22 smithi067 systemd[1]: Stopped Ceph mon.smithi067 for d46c346e-00fd-11ef-bc93-c7b262605968. 2024-04-22T23:13:23.163 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:22 smithi067 systemd[1]: Starting Ceph mon.smithi067 for d46c346e-00fd-11ef-bc93-c7b262605968... 2024-04-22T23:13:23.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:23 smithi067 podman[26434]: 2024-04-22 23:13:23.06530739 +0000 UTC m=+0.018454133 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph:v18.2.1 2024-04-22T23:13:23.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:23 smithi067 podman[26434]: 2024-04-22 23:13:23.237402524 +0000 UTC m=+0.190549308 container create 6789283703e0028c6f447f8504400ced49d85f3b57dea8cf1a25a7e126b8b58e (image=quay.io/ceph/ceph:v18.2.1, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_BRANCH=HEAD) 2024-04-22T23:13:25.296 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 podman[26434]: 2024-04-22 23:13:24.908297984 +0000 UTC m=+1.861444727 container init 6789283703e0028c6f447f8504400ced49d85f3b57dea8cf1a25a7e126b8b58e (image=quay.io/ceph/ceph:v18.2.1, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1) 2024-04-22T23:13:25.296 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 podman[26434]: 2024-04-22 23:13:24.912116342 +0000 UTC m=+1.865263085 container start 6789283703e0028c6f447f8504400ced49d85f3b57dea8cf1a25a7e126b8b58e (image=quay.io/ceph/ceph:v18.2.1, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux , io.buildah.version=1.29.1) 2024-04-22T23:13:25.296 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: set uid:gid to 167:167 (ceph:ceph) 2024-04-22T23:13:25.296 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable), process ceph-mon, pid 2 2024-04-22T23:13:25.296 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: pidfile_write: ignore empty --pid-file 2024-04-22T23:13:25.296 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: load: jerasure load: lrc 2024-04-22T23:13:25.296 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: RocksDB version: 7.9.2 2024-04-22T23:13:25.296 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Git sha 0 2024-04-22T23:13:25.296 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Compile date 2023-12-11 22:07:34 2024-04-22T23:13:25.296 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: DB SUMMARY 2024-04-22T23:13:25.296 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: DB Session ID: RMT86CPF87YNLW5GSVK7 2024-04-22T23:13:25.297 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: CURRENT file: CURRENT 2024-04-22T23:13:25.297 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: IDENTITY file: IDENTITY 2024-04-22T23:13:25.297 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-22T23:13:25.297 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi067/store.db dir, Total Num: 1, files: 000008.sst 2024-04-22T23:13:25.297 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi067/store.db: 000009.log size: 98361 ; 2024-04-22T23:13:25.297 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.error_if_exists: 0 2024-04-22T23:13:25.297 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.create_if_missing: 0 2024-04-22T23:13:25.297 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.paranoid_checks: 1 2024-04-22T23:13:25.297 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-22T23:13:25.297 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-22T23:13:25.297 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-22T23:13:25.297 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.env: 0x55c0bbb55720 2024-04-22T23:13:25.297 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.fs: PosixFileSystem 2024-04-22T23:13:25.297 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.info_log: 0x55c0bd7cd2e0 2024-04-22T23:13:25.298 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.max_file_opening_threads: 16 2024-04-22T23:13:25.298 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.statistics: (nil) 2024-04-22T23:13:25.298 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.use_fsync: 0 2024-04-22T23:13:25.298 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.max_log_file_size: 0 2024-04-22T23:13:25.298 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-22T23:13:25.298 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-22T23:13:25.298 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.keep_log_file_num: 1000 2024-04-22T23:13:25.298 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.recycle_log_file_num: 0 2024-04-22T23:13:25.298 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.allow_fallocate: 1 2024-04-22T23:13:25.298 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.allow_mmap_reads: 0 2024-04-22T23:13:25.298 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.allow_mmap_writes: 0 2024-04-22T23:13:25.298 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.use_direct_reads: 0 2024-04-22T23:13:25.298 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-22T23:13:25.298 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.create_missing_column_families: 0 2024-04-22T23:13:25.299 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.db_log_dir: 2024-04-22T23:13:25.299 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.wal_dir: 2024-04-22T23:13:25.299 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-22T23:13:25.299 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-22T23:13:25.299 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-22T23:13:25.299 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-22T23:13:25.299 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-22T23:13:25.299 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-22T23:13:25.299 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.advise_random_on_open: 1 2024-04-22T23:13:25.299 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.db_write_buffer_size: 0 2024-04-22T23:13:25.299 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.write_buffer_manager: 0x55c0bca3e140 2024-04-22T23:13:25.299 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-22T23:13:25.299 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-22T23:13:25.299 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-22T23:13:25.299 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.rate_limiter: (nil) 2024-04-22T23:13:25.300 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-22T23:13:25.300 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.wal_recovery_mode: 2 2024-04-22T23:13:25.300 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.enable_thread_tracking: 0 2024-04-22T23:13:25.300 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.enable_pipelined_write: 0 2024-04-22T23:13:25.300 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.unordered_write: 0 2024-04-22T23:13:25.300 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-22T23:13:25.300 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-22T23:13:25.300 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-22T23:13:25.300 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-22T23:13:25.300 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.row_cache: None 2024-04-22T23:13:25.300 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.wal_filter: None 2024-04-22T23:13:25.300 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-22T23:13:25.300 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.allow_ingest_behind: 0 2024-04-22T23:13:25.301 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.two_write_queues: 0 2024-04-22T23:13:25.302 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.manual_wal_flush: 0 2024-04-22T23:13:25.302 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.wal_compression: 0 2024-04-22T23:13:25.302 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.atomic_flush: 0 2024-04-22T23:13:25.302 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-22T23:13:25.302 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-22T23:13:25.302 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-22T23:13:25.302 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.log_readahead_size: 0 2024-04-22T23:13:25.302 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-22T23:13:25.302 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.best_efforts_recovery: 0 2024-04-22T23:13:25.302 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-22T23:13:25.302 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-22T23:13:25.302 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.allow_data_in_errors: 0 2024-04-22T23:13:25.302 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.db_host_id: __hostname__ 2024-04-22T23:13:25.303 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-22T23:13:25.303 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.max_background_jobs: 2 2024-04-22T23:13:25.303 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.max_background_compactions: -1 2024-04-22T23:13:25.303 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.max_subcompactions: 1 2024-04-22T23:13:25.303 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-22T23:13:25.303 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-22T23:13:25.303 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-22T23:13:25.303 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.max_total_wal_size: 0 2024-04-22T23:13:25.303 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-22T23:13:25.303 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-22T23:13:25.303 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-22T23:13:25.303 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-22T23:13:25.303 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.max_open_files: -1 2024-04-22T23:13:25.303 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.bytes_per_sync: 0 2024-04-22T23:13:25.303 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-22T23:13:25.304 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-22T23:13:25.304 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.compaction_readahead_size: 0 2024-04-22T23:13:25.304 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.max_background_flushes: -1 2024-04-22T23:13:25.304 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Compression algorithms supported: 2024-04-22T23:13:25.304 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-22T23:13:25.304 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: kZSTD supported: 0 2024-04-22T23:13:25.304 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: kXpressCompression supported: 0 2024-04-22T23:13:25.304 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: kLZ4HCCompression supported: 1 2024-04-22T23:13:25.304 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: kZlibCompression supported: 1 2024-04-22T23:13:25.304 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: kSnappyCompression supported: 1 2024-04-22T23:13:25.304 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: kLZ4Compression supported: 1 2024-04-22T23:13:25.304 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: kBZip2Compression supported: 0 2024-04-22T23:13:25.304 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-22T23:13:25.304 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-22T23:13:25.304 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi067/store.db/MANIFEST-000010 2024-04-22T23:13:25.305 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-22T23:13:25.305 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-22T23:13:25.305 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.merge_operator: 2024-04-22T23:13:25.305 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.compaction_filter: None 2024-04-22T23:13:25.305 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.compaction_filter_factory: None 2024-04-22T23:13:25.305 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.sst_partitioner_factory: None 2024-04-22T23:13:25.305 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-22T23:13:25.305 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-22T23:13:25.305 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c0bd7cd3e0) 2024-04-22T23:13:25.305 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: cache_index_and_filter_blocks: 1 2024-04-22T23:13:25.305 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-22T23:13:25.305 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-22T23:13:25.305 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: pin_top_level_index_and_filter: 1 2024-04-22T23:13:25.305 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: index_type: 0 2024-04-22T23:13:25.305 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: data_block_index_type: 0 2024-04-22T23:13:25.306 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: index_shortening: 1 2024-04-22T23:13:25.306 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-22T23:13:25.306 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: checksum: 4 2024-04-22T23:13:25.306 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: no_block_cache: 0 2024-04-22T23:13:25.306 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: block_cache: 0x55c0bcadf1f0 2024-04-22T23:13:25.306 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: block_cache_name: BinnedLRUCache 2024-04-22T23:13:25.306 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: block_cache_options: 2024-04-22T23:13:25.306 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: capacity : 536870912 2024-04-22T23:13:25.306 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: num_shard_bits : 4 2024-04-22T23:13:25.306 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: strict_capacity_limit : 0 2024-04-22T23:13:25.306 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: high_pri_pool_ratio: 0.000 2024-04-22T23:13:25.306 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: block_cache_compressed: (nil) 2024-04-22T23:13:25.306 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: persistent_cache: (nil) 2024-04-22T23:13:25.306 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: block_size: 4096 2024-04-22T23:13:25.306 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: block_size_deviation: 10 2024-04-22T23:13:25.307 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: block_restart_interval: 16 2024-04-22T23:13:25.307 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: index_block_restart_interval: 1 2024-04-22T23:13:25.307 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: metadata_block_size: 4096 2024-04-22T23:13:25.307 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: partition_filters: 0 2024-04-22T23:13:25.307 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: use_delta_encoding: 1 2024-04-22T23:13:25.307 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: filter_policy: bloomfilter 2024-04-22T23:13:25.307 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: whole_key_filtering: 1 2024-04-22T23:13:25.307 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: verify_compression: 0 2024-04-22T23:13:25.307 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: read_amp_bytes_per_bit: 0 2024-04-22T23:13:25.307 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: format_version: 5 2024-04-22T23:13:25.307 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: enable_index_compression: 1 2024-04-22T23:13:25.307 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: block_align: 0 2024-04-22T23:13:25.307 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: max_auto_readahead_size: 262144 2024-04-22T23:13:25.307 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: prepopulate_block_cache: 0 2024-04-22T23:13:25.307 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: initial_auto_readahead_size: 8192 2024-04-22T23:13:25.307 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: num_file_reads_for_auto_readahead: 2 2024-04-22T23:13:25.308 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.write_buffer_size: 33554432 2024-04-22T23:13:25.308 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.max_write_buffer_number: 2 2024-04-22T23:13:25.308 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.compression: NoCompression 2024-04-22T23:13:25.308 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.bottommost_compression: Disabled 2024-04-22T23:13:25.308 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.prefix_extractor: nullptr 2024-04-22T23:13:25.308 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-22T23:13:25.308 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.num_levels: 7 2024-04-22T23:13:25.308 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-22T23:13:25.308 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-22T23:13:25.308 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-22T23:13:25.308 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-22T23:13:25.308 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-22T23:13:25.308 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-22T23:13:25.308 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-22T23:13:25.308 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-22T23:13:25.309 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-22T23:13:25.309 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-22T23:13:25.309 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-22T23:13:25.309 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-22T23:13:25.309 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-22T23:13:25.309 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.compression_opts.level: 32767 2024-04-22T23:13:25.309 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.compression_opts.strategy: 0 2024-04-22T23:13:25.309 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-22T23:13:25.309 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-22T23:13:25.309 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-22T23:13:25.309 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-22T23:13:25.309 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.compression_opts.enabled: false 2024-04-22T23:13:25.309 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-22T23:13:25.309 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-22T23:13:25.309 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-22T23:13:25.310 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-22T23:13:25.310 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.target_file_size_base: 67108864 2024-04-22T23:13:25.310 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-22T23:13:25.310 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-22T23:13:25.310 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-22T23:13:25.310 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-22T23:13:25.310 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-22T23:13:25.310 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-22T23:13:25.310 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-22T23:13:25.310 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-22T23:13:25.310 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-22T23:13:25.310 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-22T23:13:25.310 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-22T23:13:25.311 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-22T23:13:25.311 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-22T23:13:25.311 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-22T23:13:25.311 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.arena_block_size: 1048576 2024-04-22T23:13:25.311 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-22T23:13:25.311 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-22T23:13:25.311 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.disable_auto_compactions: 0 2024-04-22T23:13:25.311 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-22T23:13:25.311 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-22T23:13:25.311 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-22T23:13:25.311 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-22T23:13:25.311 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-22T23:13:25.311 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-22T23:13:25.311 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-22T23:13:25.311 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-22T23:13:25.311 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-22T23:13:25.312 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-22T23:13:25.312 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.table_properties_collectors: 2024-04-22T23:13:25.312 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.inplace_update_support: 0 2024-04-22T23:13:25.312 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-22T23:13:25.312 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-22T23:13:25.312 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-22T23:13:25.312 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-22T23:13:25.312 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.bloom_locality: 0 2024-04-22T23:13:25.312 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.max_successive_merges: 0 2024-04-22T23:13:25.312 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-22T23:13:25.312 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.paranoid_file_checks: 0 2024-04-22T23:13:25.312 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.force_consistency_checks: 1 2024-04-22T23:13:25.312 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.report_bg_io_stats: 0 2024-04-22T23:13:25.312 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.ttl: 2592000 2024-04-22T23:13:25.312 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-22T23:13:25.313 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-22T23:13:25.313 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-22T23:13:25.313 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.enable_blob_files: false 2024-04-22T23:13:25.313 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.min_blob_size: 0 2024-04-22T23:13:25.313 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.blob_file_size: 268435456 2024-04-22T23:13:25.313 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-22T23:13:25.313 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-22T23:13:25.313 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-22T23:13:25.313 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-22T23:13:25.313 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-22T23:13:25.313 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.blob_file_starting_level: 0 2024-04-22T23:13:25.313 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-22T23:13:25.313 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi067/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-22T23:13:25.313 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-22T23:13:25.313 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 3f61ccfa-96d2-469e-9f4c-f1125e9e9400 2024-04-22T23:13:25.314 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713827604946256, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-22T23:13:25.314 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-22T23:13:25.314 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713827604947482, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 93605, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 314, "table_properties": {"data_size": 91627, "index_size": 260, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13576, "raw_average_key_size": 49, "raw_value_size": 84236, "raw_average_value_size": 308, "num_data_blocks": 11, "num_entries": 273, "num_filter_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713827604, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "3f61ccfa-96d2-469e-9f4c-f1125e9e9400", "db_session_id": "RMT86CPF87YNLW5GSVK7", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-22T23:13:25.314 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713827604947589, "job": 1, "event": "recovery_finished"} 2024-04-22T23:13:25.314 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-22T23:13:25.314 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi067/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-22T23:13:25.314 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55c0bcb7c000 2024-04-22T23:13:25.314 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: DB pointer 0x55c0bcb68000 2024-04-22T23:13:25.314 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-22T23:13:25.314 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-22T23:13:25.314 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: ** DB Stats ** 2024-04-22T23:13:25.314 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T23:13:25.315 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-22T23:13:25.315 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T23:13:25.315 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T23:13:25.315 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-22T23:13:25.315 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T23:13:25.315 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T23:13:25.315 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: 2024-04-22T23:13:25.315 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: ** Compaction Stats [default] ** 2024-04-22T23:13:25.315 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T23:13:25.315 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-22T23:13:25.315 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: L0 2/0 93.28 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 113.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T23:13:25.315 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Sum 2/0 93.28 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 113.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T23:13:25.315 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 113.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T23:13:25.315 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: 2024-04-22T23:13:25.316 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: ** Compaction Stats [default] ** 2024-04-22T23:13:25.316 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T23:13:25.316 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T23:13:25.316 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 113.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T23:13:25.316 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: 2024-04-22T23:13:25.316 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-22T23:13:25.316 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: 2024-04-22T23:13:25.316 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T23:13:25.316 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T23:13:25.316 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T23:13:25.316 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T23:13:25.316 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T23:13:25.316 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T23:13:25.316 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Cumulative compaction: 0.00 GB write, 24.27 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T23:13:25.316 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Interval compaction: 0.00 GB write, 24.27 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T23:13:25.317 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T23:13:25.317 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Block cache BinnedLRUCache@0x55c0bcadf1f0#2 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-04-22T23:13:25.317 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-04-22T23:13:25.317 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: 2024-04-22T23:13:25.317 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T23:13:25.317 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: starting mon.smithi067 rank 0 at public addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] at bind addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi067 fsid d46c346e-00fd-11ef-bc93-c7b262605968 2024-04-22T23:13:25.317 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: mon.smithi067@-1(???) e1 preinit fsid d46c346e-00fd-11ef-bc93-c7b262605968 2024-04-22T23:13:25.317 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: mon.smithi067@-1(???).mds e1 new map 2024-04-22T23:13:25.317 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: mon.smithi067@-1(???).mds e1 print_map 2024-04-22T23:13:25.317 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: e1 2024-04-22T23:13:25.317 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:13:25.317 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:13:25.317 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: legacy client fscid: -1 2024-04-22T23:13:25.317 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: 2024-04-22T23:13:25.318 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: No filesystems configured 2024-04-22T23:13:25.318 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: mon.smithi067@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-22T23:13:25.318 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: mon.smithi067@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T23:13:25.318 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: mon.smithi067@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T23:13:25.318 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: mon.smithi067@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T23:13:25.318 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:24 smithi067 ceph-mon[26453]: mon.smithi067@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-22T23:13:25.318 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:25 smithi067 ceph-mon[26453]: mon.smithi067 is new leader, mons smithi067 in quorum (ranks 0) 2024-04-22T23:13:25.318 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:25 smithi067 ceph-mon[26453]: monmap e1: 1 mons at {smithi067=[v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-22T23:13:25.318 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:25 smithi067 ceph-mon[26453]: fsmap 2024-04-22T23:13:25.318 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:25 smithi067 ceph-mon[26453]: osdmap e1: 0 total, 0 up, 0 in 2024-04-22T23:13:25.318 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:25 smithi067 ceph-mon[26453]: mgrmap e1: no daemons active 2024-04-22T23:13:25.847 INFO:teuthology.orchestra.run.smithi067.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-22T23:13:26.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:25 smithi067 bash[26434]: 6789283703e0028c6f447f8504400ced49d85f3b57dea8cf1a25a7e126b8b58e 2024-04-22T23:13:26.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:25 smithi067 systemd[1]: Started Ceph mon.smithi067 for d46c346e-00fd-11ef-bc93-c7b262605968. 2024-04-22T23:13:30.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:29 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/3267544133' entity='client.admin' 2024-04-22T23:13:35.183 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.859+0000 7fd4a8ad3700 1 Processor -- start 2024-04-22T23:13:35.183 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.859+0000 7fd4a8ad3700 1 -- start start 2024-04-22T23:13:35.183 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.860+0000 7fd4a8ad3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4a4106560 0x7fd4a4106980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:13:35.183 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.860+0000 7fd4a8ad3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a4106f50 con 0x7fd4a4106560 2024-04-22T23:13:35.183 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.860+0000 7fd4a259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4a4106560 0x7fd4a4106980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:13:35.184 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.860+0000 7fd4a259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4a4106560 0x7fd4a4106980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34096/0 (socket says 172.21.15.67:34096) 2024-04-22T23:13:35.184 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.860+0000 7fd4a259c700 1 -- 172.21.15.67:0/2734308602 learned_addr learned my addr 172.21.15.67:0/2734308602 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:13:35.184 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.860+0000 7fd4a259c700 1 -- 172.21.15.67:0/2734308602 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a4107760 con 0x7fd4a4106560 2024-04-22T23:13:35.184 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.861+0000 7fd4a259c700 1 --2- 172.21.15.67:0/2734308602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4a4106560 0x7fd4a4106980 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd4940098a0 tx=0x7fd494009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcc0e3b5c44bca02 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:13:35.184 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.861+0000 7fd4a159a700 1 -- 172.21.15.67:0/2734308602 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd494004180 con 0x7fd4a4106560 2024-04-22T23:13:35.184 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.861+0000 7fd4a159a700 1 -- 172.21.15.67:0/2734308602 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd4940042e0 con 0x7fd4a4106560 2024-04-22T23:13:35.184 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.861+0000 7fd4a159a700 1 -- 172.21.15.67:0/2734308602 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd494011a50 con 0x7fd4a4106560 2024-04-22T23:13:35.184 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.861+0000 7fd4a8ad3700 1 -- 172.21.15.67:0/2734308602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4a4106560 msgr2=0x7fd4a4106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:13:35.184 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.861+0000 7fd4a8ad3700 1 --2- 172.21.15.67:0/2734308602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4a4106560 0x7fd4a4106980 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd4940098a0 tx=0x7fd494009bb0 comp rx=0 tx=0).stop 2024-04-22T23:13:35.184 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.861+0000 7fd4a8ad3700 1 -- 172.21.15.67:0/2734308602 shutdown_connections 2024-04-22T23:13:35.184 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.861+0000 7fd4a8ad3700 1 --2- 172.21.15.67:0/2734308602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4a4106560 0x7fd4a4106980 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:13:35.184 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.861+0000 7fd4a8ad3700 1 -- 172.21.15.67:0/2734308602 >> 172.21.15.67:0/2734308602 conn(0x7fd4a4101b20 msgr2=0x7fd4a4103f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:13:35.184 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.862+0000 7fd4a8ad3700 1 -- 172.21.15.67:0/2734308602 shutdown_connections 2024-04-22T23:13:35.184 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.862+0000 7fd4a8ad3700 1 -- 172.21.15.67:0/2734308602 wait complete. 2024-04-22T23:13:35.185 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.862+0000 7fd4a8ad3700 1 Processor -- start 2024-04-22T23:13:35.185 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.862+0000 7fd4a8ad3700 1 -- start start 2024-04-22T23:13:35.185 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.862+0000 7fd4a8ad3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4a4106560 0x7fd4a419a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:13:35.185 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.863+0000 7fd4a8ad3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a419a6e0 con 0x7fd4a4106560 2024-04-22T23:13:35.185 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.863+0000 7fd4a259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4a4106560 0x7fd4a419a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:13:35.185 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.863+0000 7fd4a259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4a4106560 0x7fd4a419a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34108/0 (socket says 172.21.15.67:34108) 2024-04-22T23:13:35.185 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.863+0000 7fd4a259c700 1 -- 172.21.15.67:0/3267544133 learned_addr learned my addr 172.21.15.67:0/3267544133 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:13:35.185 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.863+0000 7fd4a259c700 1 -- 172.21.15.67:0/3267544133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd494009550 con 0x7fd4a4106560 2024-04-22T23:13:35.185 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.863+0000 7fd4a259c700 1 --2- 172.21.15.67:0/3267544133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4a4106560 0x7fd4a419a1a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd494000c00 tx=0x7fd494011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:13:35.185 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.863+0000 7fd4937fe700 1 -- 172.21.15.67:0/3267544133 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd494003b90 con 0x7fd4a4106560 2024-04-22T23:13:35.185 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.863+0000 7fd4937fe700 1 -- 172.21.15.67:0/3267544133 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fd4940248b0 con 0x7fd4a4106560 2024-04-22T23:13:35.185 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.863+0000 7fd4937fe700 1 -- 172.21.15.67:0/3267544133 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd49401a8e0 con 0x7fd4a4106560 2024-04-22T23:13:35.185 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.863+0000 7fd4a8ad3700 1 -- 172.21.15.67:0/3267544133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4a419a8e0 con 0x7fd4a4106560 2024-04-22T23:13:35.185 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.863+0000 7fd4a8ad3700 1 -- 172.21.15.67:0/3267544133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4a419ad80 con 0x7fd4a4106560 2024-04-22T23:13:35.186 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.864+0000 7fd4937fe700 1 -- 172.21.15.67:0/3267544133 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd49402d3f0 con 0x7fd4a4106560 2024-04-22T23:13:35.186 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.864+0000 7fd4937fe700 1 -- 172.21.15.67:0/3267544133 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd494019490 con 0x7fd4a4106560 2024-04-22T23:13:35.186 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.864+0000 7fd4a8ad3700 1 -- 172.21.15.67:0/3267544133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4a40623c0 con 0x7fd4a4106560 2024-04-22T23:13:35.186 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.867+0000 7fd4937fe700 1 -- 172.21.15.67:0/3267544133 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fd494024a20 con 0x7fd4a4106560 2024-04-22T23:13:35.186 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.937+0000 7fd4a8ad3700 1 -- 172.21.15.67:0/3267544133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fd4a410a9f0 con 0x7fd4a4106560 2024-04-22T23:13:35.186 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.939+0000 7fd4937fe700 1 -- 172.21.15.67:0/3267544133 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fd4940350c0 con 0x7fd4a4106560 2024-04-22T23:13:35.186 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.940+0000 7fd4a8ad3700 1 -- 172.21.15.67:0/3267544133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4a4106560 msgr2=0x7fd4a419a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:13:35.186 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.940+0000 7fd4a8ad3700 1 --2- 172.21.15.67:0/3267544133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4a4106560 0x7fd4a419a1a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd494000c00 tx=0x7fd494011ab0 comp rx=0 tx=0).stop 2024-04-22T23:13:35.186 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.940+0000 7fd4a8ad3700 1 -- 172.21.15.67:0/3267544133 shutdown_connections 2024-04-22T23:13:35.186 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.940+0000 7fd4a8ad3700 1 --2- 172.21.15.67:0/3267544133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4a4106560 0x7fd4a419a1a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:13:35.186 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.940+0000 7fd4a8ad3700 1 -- 172.21.15.67:0/3267544133 >> 172.21.15.67:0/3267544133 conn(0x7fd4a4101b20 msgr2=0x7fd4a4102840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:13:35.186 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.940+0000 7fd4a8ad3700 1 -- 172.21.15.67:0/3267544133 shutdown_connections 2024-04-22T23:13:35.186 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:28.940+0000 7fd4a8ad3700 1 -- 172.21.15.67:0/3267544133 wait complete. 2024-04-22T23:13:35.187 INFO:teuthology.orchestra.run.smithi067.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-22T23:13:35.207 INFO:teuthology.orchestra.run.smithi067.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-22T23:13:35.207 INFO:teuthology.orchestra.run.smithi067.stdout:Creating mgr... 2024-04-22T23:13:35.207 INFO:teuthology.orchestra.run.smithi067.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-22T23:13:35.208 INFO:teuthology.orchestra.run.smithi067.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-22T23:13:35.208 INFO:teuthology.orchestra.run.smithi067.stdout:Verifying port 0.0.0.0:8443 ... 2024-04-22T23:13:35.473 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-d46c346e-00fd-11ef-bc93-c7b262605968@mgr.smithi067.whdcsr 2024-04-22T23:13:35.473 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Failed to reset failed state of unit ceph-d46c346e-00fd-11ef-bc93-c7b262605968@mgr.smithi067.whdcsr.service: Unit ceph-d46c346e-00fd-11ef-bc93-c7b262605968@mgr.smithi067.whdcsr.service not loaded. 2024-04-22T23:13:35.626 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-d46c346e-00fd-11ef-bc93-c7b262605968.target.wants/ceph-d46c346e-00fd-11ef-bc93-c7b262605968@mgr.smithi067.whdcsr.service → /etc/systemd/system/ceph-d46c346e-00fd-11ef-bc93-c7b262605968@.service. 2024-04-22T23:13:36.184 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-22T23:13:36.184 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout disabled 2024-04-22T23:13:36.191 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-22T23:13:36.192 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout inactive 2024-04-22T23:13:36.192 INFO:teuthology.orchestra.run.smithi067.stdout:firewalld.service is not enabled 2024-04-22T23:13:36.192 INFO:teuthology.orchestra.run.smithi067.stdout:Not possible to enable service . firewalld.service is not available 2024-04-22T23:13:36.200 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-22T23:13:36.200 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout disabled 2024-04-22T23:13:36.208 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-22T23:13:36.208 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout inactive 2024-04-22T23:13:36.208 INFO:teuthology.orchestra.run.smithi067.stdout:firewalld.service is not enabled 2024-04-22T23:13:36.208 INFO:teuthology.orchestra.run.smithi067.stdout:Not possible to open ports <[9283, 8765, 8443]>. firewalld.service is not available 2024-04-22T23:13:36.208 INFO:teuthology.orchestra.run.smithi067.stdout:Waiting for mgr to start... 2024-04-22T23:13:36.208 INFO:teuthology.orchestra.run.smithi067.stdout:Waiting for mgr... 2024-04-22T23:13:37.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:36 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/46159857' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T23:13:37.322 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-04-22T23:13:37.323 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout { 2024-04-22T23:13:37.323 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "fsid": "d46c346e-00fd-11ef-bc93-c7b262605968", 2024-04-22T23:13:37.323 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "health": { 2024-04-22T23:13:37.323 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-22T23:13:37.323 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-22T23:13:37.323 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-22T23:13:37.323 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-04-22T23:13:37.323 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-22T23:13:37.323 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-22T23:13:37.323 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 0 2024-04-22T23:13:37.323 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-04-22T23:13:37.323 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-22T23:13:37.323 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "smithi067" 2024-04-22T23:13:37.323 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-04-22T23:13:37.323 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2024-04-22T23:13:37.324 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-22T23:13:37.324 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T23:13:37.324 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-22T23:13:37.324 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-22T23:13:37.324 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-04-22T23:13:37.324 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-22T23:13:37.324 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T23:13:37.324 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-22T23:13:37.324 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-22T23:13:37.324 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-22T23:13:37.332 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-22T23:13:37.332 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-22T23:13:37.332 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-22T23:13:37.332 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-04-22T23:13:37.332 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-22T23:13:37.332 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-22T23:13:37.332 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-22T23:13:37.333 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-22T23:13:37.333 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-22T23:13:37.333 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-22T23:13:37.333 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-22T23:13:37.333 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-22T23:13:37.333 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-22T23:13:37.333 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-04-22T23:13:37.333 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-22T23:13:37.333 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T23:13:37.333 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-22T23:13:37.333 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-22T23:13:37.333 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-04-22T23:13:37.333 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-22T23:13:37.334 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-22T23:13:37.334 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-22T23:13:37.334 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-22T23:13:37.334 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-22T23:13:37.334 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-22T23:13:37.334 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "restful" 2024-04-22T23:13:37.334 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-04-22T23:13:37.334 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T23:13:37.334 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-04-22T23:13:37.334 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-22T23:13:37.334 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T23:13:37.335 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "modified": "2024-04-22T23:13:18.688362+0000", 2024-04-22T23:13:37.335 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T23:13:37.335 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-04-22T23:13:37.335 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-22T23:13:37.335 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout } 2024-04-22T23:13:37.335 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.642+0000 7f48de67a700 1 Processor -- start 2024-04-22T23:13:37.335 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.642+0000 7f48de67a700 1 -- start start 2024-04-22T23:13:37.335 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.642+0000 7f48de67a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48d8107000 0x7f48d81093f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:13:37.335 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.642+0000 7f48de67a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d8079e50 con 0x7f48d8107000 2024-04-22T23:13:37.335 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.643+0000 7f48d7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48d8107000 0x7f48d81093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:13:37.335 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.643+0000 7f48d7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48d8107000 0x7f48d81093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34132/0 (socket says 172.21.15.67:34132) 2024-04-22T23:13:37.335 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.643+0000 7f48d7fff700 1 -- 172.21.15.67:0/4043167648 learned_addr learned my addr 172.21.15.67:0/4043167648 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:13:37.335 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.643+0000 7f48d7fff700 1 -- 172.21.15.67:0/4043167648 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d8109930 con 0x7f48d8107000 2024-04-22T23:13:37.336 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.643+0000 7f48d7fff700 1 --2- 172.21.15.67:0/4043167648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48d8107000 0x7f48d81093f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f48c80098a0 tx=0x7f48c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53deacdab0d364ef server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:13:37.336 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.643+0000 7f48d6ffd700 1 -- 172.21.15.67:0/4043167648 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48c8004180 con 0x7f48d8107000 2024-04-22T23:13:37.336 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.643+0000 7f48d6ffd700 1 -- 172.21.15.67:0/4043167648 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f48c80042e0 con 0x7f48d8107000 2024-04-22T23:13:37.336 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.643+0000 7f48d6ffd700 1 -- 172.21.15.67:0/4043167648 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48c8011a50 con 0x7f48d8107000 2024-04-22T23:13:37.336 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.644+0000 7f48de67a700 1 -- 172.21.15.67:0/4043167648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48d8107000 msgr2=0x7f48d81093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:13:37.336 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.644+0000 7f48de67a700 1 --2- 172.21.15.67:0/4043167648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48d8107000 0x7f48d81093f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f48c80098a0 tx=0x7f48c8009bb0 comp rx=0 tx=0).stop 2024-04-22T23:13:37.336 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.644+0000 7f48de67a700 1 -- 172.21.15.67:0/4043167648 shutdown_connections 2024-04-22T23:13:37.336 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.644+0000 7f48de67a700 1 --2- 172.21.15.67:0/4043167648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48d8107000 0x7f48d81093f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:13:37.336 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.644+0000 7f48de67a700 1 -- 172.21.15.67:0/4043167648 >> 172.21.15.67:0/4043167648 conn(0x7f48d81009b0 msgr2=0x7f48d8102e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:13:37.336 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.644+0000 7f48de67a700 1 -- 172.21.15.67:0/4043167648 shutdown_connections 2024-04-22T23:13:37.336 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.644+0000 7f48de67a700 1 -- 172.21.15.67:0/4043167648 wait complete. 2024-04-22T23:13:37.336 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.645+0000 7f48de67a700 1 Processor -- start 2024-04-22T23:13:37.336 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.645+0000 7f48de67a700 1 -- start start 2024-04-22T23:13:37.336 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.645+0000 7f48de67a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48d8107000 0x7f48d819a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:13:37.336 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.645+0000 7f48de67a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d819a710 con 0x7f48d8107000 2024-04-22T23:13:37.337 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.645+0000 7f48d7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48d8107000 0x7f48d819a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:13:37.337 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.645+0000 7f48d7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48d8107000 0x7f48d819a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34138/0 (socket says 172.21.15.67:34138) 2024-04-22T23:13:37.337 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.645+0000 7f48d7fff700 1 -- 172.21.15.67:0/46159857 learned_addr learned my addr 172.21.15.67:0/46159857 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:13:37.337 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.645+0000 7f48d7fff700 1 -- 172.21.15.67:0/46159857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c8009550 con 0x7f48d8107000 2024-04-22T23:13:37.337 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.646+0000 7f48d7fff700 1 --2- 172.21.15.67:0/46159857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48d8107000 0x7f48d819a1d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f48c8003f80 tx=0x7f48c8011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:13:37.337 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.646+0000 7f48d57fa700 1 -- 172.21.15.67:0/46159857 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48c8003a80 con 0x7f48d8107000 2024-04-22T23:13:37.337 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.646+0000 7f48d57fa700 1 -- 172.21.15.67:0/46159857 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f48c8003be0 con 0x7f48d8107000 2024-04-22T23:13:37.337 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.646+0000 7f48de67a700 1 -- 172.21.15.67:0/46159857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48d819a910 con 0x7f48d8107000 2024-04-22T23:13:37.337 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.646+0000 7f48d57fa700 1 -- 172.21.15.67:0/46159857 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48c8019820 con 0x7f48d8107000 2024-04-22T23:13:37.337 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.646+0000 7f48de67a700 1 -- 172.21.15.67:0/46159857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48d819adb0 con 0x7f48d8107000 2024-04-22T23:13:37.337 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.646+0000 7f48d57fa700 1 -- 172.21.15.67:0/46159857 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f48c802d440 con 0x7f48d8107000 2024-04-22T23:13:37.337 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.646+0000 7f48d57fa700 1 -- 172.21.15.67:0/46159857 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f48c80247c0 con 0x7f48d8107000 2024-04-22T23:13:37.337 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.647+0000 7f48de67a700 1 -- 172.21.15.67:0/46159857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48d80623c0 con 0x7f48d8107000 2024-04-22T23:13:37.337 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.649+0000 7f48d57fa700 1 -- 172.21.15.67:0/46159857 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f48c801acc0 con 0x7f48d8107000 2024-04-22T23:13:37.338 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.722+0000 7f48de67a700 1 -- 172.21.15.67:0/46159857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f48d819d770 con 0x7f48d8107000 2024-04-22T23:13:37.338 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.722+0000 7f48d57fa700 1 -- 172.21.15.67:0/46159857 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1247 (secure 0 0 0) 0x7f48c8035090 con 0x7f48d8107000 2024-04-22T23:13:37.338 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.723+0000 7f48de67a700 1 -- 172.21.15.67:0/46159857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48d8107000 msgr2=0x7f48d819a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:13:37.338 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.723+0000 7f48de67a700 1 --2- 172.21.15.67:0/46159857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48d8107000 0x7f48d819a1d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f48c8003f80 tx=0x7f48c8011ab0 comp rx=0 tx=0).stop 2024-04-22T23:13:37.338 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.723+0000 7f48de67a700 1 -- 172.21.15.67:0/46159857 shutdown_connections 2024-04-22T23:13:37.338 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.723+0000 7f48de67a700 1 --2- 172.21.15.67:0/46159857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48d8107000 0x7f48d819a1d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:13:37.338 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.723+0000 7f48de67a700 1 -- 172.21.15.67:0/46159857 >> 172.21.15.67:0/46159857 conn(0x7f48d81009b0 msgr2=0x7f48d8101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:13:37.338 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.724+0000 7f48de67a700 1 -- 172.21.15.67:0/46159857 shutdown_connections 2024-04-22T23:13:37.338 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:36.724+0000 7f48de67a700 1 -- 172.21.15.67:0/46159857 wait complete. 2024-04-22T23:13:37.338 INFO:teuthology.orchestra.run.smithi067.stdout:mgr not available, waiting (1/15)... 2024-04-22T23:13:40.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:39 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/156353249' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T23:13:41.050 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-04-22T23:13:41.050 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout { 2024-04-22T23:13:41.050 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "fsid": "d46c346e-00fd-11ef-bc93-c7b262605968", 2024-04-22T23:13:41.050 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "health": { 2024-04-22T23:13:41.050 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-22T23:13:41.050 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-22T23:13:41.050 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-22T23:13:41.050 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-04-22T23:13:41.051 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-22T23:13:41.051 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-22T23:13:41.051 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 0 2024-04-22T23:13:41.051 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-04-22T23:13:41.051 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-22T23:13:41.051 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "smithi067" 2024-04-22T23:13:41.051 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-04-22T23:13:41.051 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-04-22T23:13:41.051 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-22T23:13:41.051 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T23:13:41.051 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-22T23:13:41.051 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-22T23:13:41.051 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-04-22T23:13:41.052 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-22T23:13:41.052 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T23:13:41.052 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-22T23:13:41.052 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-22T23:13:41.052 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-22T23:13:41.052 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-22T23:13:41.052 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-22T23:13:41.052 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-22T23:13:41.052 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-04-22T23:13:41.052 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-22T23:13:41.052 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-22T23:13:41.052 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-22T23:13:41.052 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-22T23:13:41.052 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-22T23:13:41.052 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-22T23:13:41.053 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-22T23:13:41.053 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-22T23:13:41.053 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-22T23:13:41.053 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-04-22T23:13:41.053 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-22T23:13:41.053 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T23:13:41.053 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-22T23:13:41.053 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-22T23:13:41.053 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-04-22T23:13:41.053 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-22T23:13:41.053 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-22T23:13:41.053 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-22T23:13:41.053 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-22T23:13:41.053 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-22T23:13:41.054 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-22T23:13:41.054 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "restful" 2024-04-22T23:13:41.054 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-04-22T23:13:41.054 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T23:13:41.054 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-04-22T23:13:41.054 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-22T23:13:41.054 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T23:13:41.054 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "modified": "2024-04-22T23:13:18.688362+0000", 2024-04-22T23:13:41.054 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T23:13:41.054 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-04-22T23:13:41.054 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-22T23:13:41.054 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout } 2024-04-22T23:13:41.054 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.759+0000 7f2f11e5b700 1 Processor -- start 2024-04-22T23:13:41.055 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.759+0000 7f2f11e5b700 1 -- start start 2024-04-22T23:13:41.055 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.759+0000 7f2f11e5b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f0c106540 0x7f2f0c106960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:13:41.055 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.759+0000 7f2f11e5b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c106f30 con 0x7f2f0c106540 2024-04-22T23:13:41.055 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.760+0000 7f2f0b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f0c106540 0x7f2f0c106960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:13:41.055 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.760+0000 7f2f0b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f0c106540 0x7f2f0c106960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36840/0 (socket says 172.21.15.67:36840) 2024-04-22T23:13:41.055 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.760+0000 7f2f0b7fe700 1 -- 172.21.15.67:0/3245384736 learned_addr learned my addr 172.21.15.67:0/3245384736 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:13:41.055 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.760+0000 7f2f0b7fe700 1 -- 172.21.15.67:0/3245384736 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f0c107740 con 0x7f2f0c106540 2024-04-22T23:13:41.055 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.760+0000 7f2f0b7fe700 1 --2- 172.21.15.67:0/3245384736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f0c106540 0x7f2f0c106960 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2f000098a0 tx=0x7f2f00009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b7087a1a3f29ce server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:13:41.055 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.760+0000 7f2f0a7fc700 1 -- 172.21.15.67:0/3245384736 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f00004180 con 0x7f2f0c106540 2024-04-22T23:13:41.055 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.760+0000 7f2f0a7fc700 1 -- 172.21.15.67:0/3245384736 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2f000042e0 con 0x7f2f0c106540 2024-04-22T23:13:41.055 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.760+0000 7f2f0a7fc700 1 -- 172.21.15.67:0/3245384736 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f00011a50 con 0x7f2f0c106540 2024-04-22T23:13:41.055 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.761+0000 7f2f11e5b700 1 -- 172.21.15.67:0/3245384736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f0c106540 msgr2=0x7f2f0c106960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:13:41.055 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.761+0000 7f2f11e5b700 1 --2- 172.21.15.67:0/3245384736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f0c106540 0x7f2f0c106960 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2f000098a0 tx=0x7f2f00009bb0 comp rx=0 tx=0).stop 2024-04-22T23:13:41.056 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.761+0000 7f2f11e5b700 1 -- 172.21.15.67:0/3245384736 shutdown_connections 2024-04-22T23:13:41.056 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.761+0000 7f2f11e5b700 1 --2- 172.21.15.67:0/3245384736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f0c106540 0x7f2f0c106960 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:13:41.056 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.761+0000 7f2f11e5b700 1 -- 172.21.15.67:0/3245384736 >> 172.21.15.67:0/3245384736 conn(0x7f2f0c101b20 msgr2=0x7f2f0c103f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:13:41.056 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.761+0000 7f2f11e5b700 1 -- 172.21.15.67:0/3245384736 shutdown_connections 2024-04-22T23:13:41.056 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.761+0000 7f2f11e5b700 1 -- 172.21.15.67:0/3245384736 wait complete. 2024-04-22T23:13:41.056 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.762+0000 7f2f11e5b700 1 Processor -- start 2024-04-22T23:13:41.056 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.762+0000 7f2f11e5b700 1 -- start start 2024-04-22T23:13:41.056 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.762+0000 7f2f11e5b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f0c106540 0x7f2f0c19c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:13:41.056 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.762+0000 7f2f11e5b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c19c8e0 con 0x7f2f0c106540 2024-04-22T23:13:41.056 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.762+0000 7f2f0b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f0c106540 0x7f2f0c19c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:13:41.056 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.762+0000 7f2f0b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f0c106540 0x7f2f0c19c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36842/0 (socket says 172.21.15.67:36842) 2024-04-22T23:13:41.056 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.762+0000 7f2f0b7fe700 1 -- 172.21.15.67:0/156353249 learned_addr learned my addr 172.21.15.67:0/156353249 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:13:41.056 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.763+0000 7f2f0b7fe700 1 -- 172.21.15.67:0/156353249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f00009550 con 0x7f2f0c106540 2024-04-22T23:13:41.057 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.763+0000 7f2f0b7fe700 1 --2- 172.21.15.67:0/156353249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f0c106540 0x7f2f0c19c3a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2f00000c00 tx=0x7f2f00011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:13:41.057 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.763+0000 7f2f08ff9700 1 -- 172.21.15.67:0/156353249 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f00003b90 con 0x7f2f0c106540 2024-04-22T23:13:41.057 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.763+0000 7f2f08ff9700 1 -- 172.21.15.67:0/156353249 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2f000248b0 con 0x7f2f0c106540 2024-04-22T23:13:41.057 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.763+0000 7f2f08ff9700 1 -- 172.21.15.67:0/156353249 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f0001a8e0 con 0x7f2f0c106540 2024-04-22T23:13:41.057 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.763+0000 7f2f11e5b700 1 -- 172.21.15.67:0/156353249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f0c19cae0 con 0x7f2f0c106540 2024-04-22T23:13:41.057 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.763+0000 7f2f11e5b700 1 -- 172.21.15.67:0/156353249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f0c19cf80 con 0x7f2f0c106540 2024-04-22T23:13:41.057 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.763+0000 7f2f08ff9700 1 -- 172.21.15.67:0/156353249 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2f0002d3f0 con 0x7f2f0c106540 2024-04-22T23:13:41.057 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.763+0000 7f2f08ff9700 1 -- 172.21.15.67:0/156353249 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2f00019400 con 0x7f2f0c106540 2024-04-22T23:13:41.057 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.764+0000 7f2f11e5b700 1 -- 172.21.15.67:0/156353249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f0c0623c0 con 0x7f2f0c106540 2024-04-22T23:13:41.057 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.768+0000 7f2f08ff9700 1 -- 172.21.15.67:0/156353249 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f2f00024a20 con 0x7f2f0c106540 2024-04-22T23:13:41.057 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.840+0000 7f2f11e5b700 1 -- 172.21.15.67:0/156353249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f2f0c10a9f0 con 0x7f2f0c106540 2024-04-22T23:13:41.057 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.840+0000 7f2f08ff9700 1 -- 172.21.15.67:0/156353249 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1247 (secure 0 0 0) 0x7f2f00035090 con 0x7f2f0c106540 2024-04-22T23:13:41.057 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.842+0000 7f2f11e5b700 1 -- 172.21.15.67:0/156353249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f0c106540 msgr2=0x7f2f0c19c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:13:41.057 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.842+0000 7f2f11e5b700 1 --2- 172.21.15.67:0/156353249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f0c106540 0x7f2f0c19c3a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2f00000c00 tx=0x7f2f00011ab0 comp rx=0 tx=0).stop 2024-04-22T23:13:41.058 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.842+0000 7f2f11e5b700 1 -- 172.21.15.67:0/156353249 shutdown_connections 2024-04-22T23:13:41.058 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.842+0000 7f2f11e5b700 1 --2- 172.21.15.67:0/156353249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f0c106540 0x7f2f0c19c3a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:13:41.058 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.842+0000 7f2f11e5b700 1 -- 172.21.15.67:0/156353249 >> 172.21.15.67:0/156353249 conn(0x7f2f0c101b20 msgr2=0x7f2f0c102860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:13:41.058 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.842+0000 7f2f11e5b700 1 -- 172.21.15.67:0/156353249 shutdown_connections 2024-04-22T23:13:41.058 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:39.842+0000 7f2f11e5b700 1 -- 172.21.15.67:0/156353249 wait complete. 2024-04-22T23:13:41.058 INFO:teuthology.orchestra.run.smithi067.stdout:mgr not available, waiting (2/15)... 2024-04-22T23:13:43.396 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:43 smithi067 ceph-mon[26453]: Activating manager daemon smithi067.whdcsr 2024-04-22T23:13:43.396 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:43 smithi067 ceph-mon[26453]: mgrmap e2: smithi067.whdcsr(active, starting, since 0.00311098s) 2024-04-22T23:13:43.396 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:43 smithi067 ceph-mon[26453]: from='mgr.14100 172.21.15.67:0/1159860528' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T23:13:43.396 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:43 smithi067 ceph-mon[26453]: from='mgr.14100 172.21.15.67:0/1159860528' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T23:13:43.397 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:43 smithi067 ceph-mon[26453]: from='mgr.14100 172.21.15.67:0/1159860528' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T23:13:43.397 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:43 smithi067 ceph-mon[26453]: from='mgr.14100 172.21.15.67:0/1159860528' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-22T23:13:43.397 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:43 smithi067 ceph-mon[26453]: from='mgr.14100 172.21.15.67:0/1159860528' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mgr metadata", "who": "smithi067.whdcsr", "id": "smithi067.whdcsr"}]: dispatch 2024-04-22T23:13:43.397 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:43 smithi067 ceph-mon[26453]: Manager daemon smithi067.whdcsr is now available 2024-04-22T23:13:43.397 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:43 smithi067 ceph-mon[26453]: from='mgr.14100 172.21.15.67:0/1159860528' entity='mgr.smithi067.whdcsr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.whdcsr/mirror_snapshot_schedule"}]: dispatch 2024-04-22T23:13:43.397 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:43 smithi067 ceph-mon[26453]: from='mgr.14100 172.21.15.67:0/1159860528' entity='mgr.smithi067.whdcsr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.whdcsr/trash_purge_schedule"}]: dispatch 2024-04-22T23:13:43.397 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:43 smithi067 ceph-mon[26453]: from='mgr.14100 172.21.15.67:0/1159860528' entity='mgr.smithi067.whdcsr' 2024-04-22T23:13:43.397 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:43 smithi067 ceph-mon[26453]: from='mgr.14100 172.21.15.67:0/1159860528' entity='mgr.smithi067.whdcsr' 2024-04-22T23:13:43.397 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:43 smithi067 ceph-mon[26453]: from='mgr.14100 172.21.15.67:0/1159860528' entity='mgr.smithi067.whdcsr' 2024-04-22T23:13:44.165 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-04-22T23:13:44.165 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout { 2024-04-22T23:13:44.166 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "fsid": "d46c346e-00fd-11ef-bc93-c7b262605968", 2024-04-22T23:13:44.166 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "health": { 2024-04-22T23:13:44.166 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-22T23:13:44.166 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-22T23:13:44.166 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-22T23:13:44.166 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-04-22T23:13:44.166 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-22T23:13:44.166 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-22T23:13:44.166 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 0 2024-04-22T23:13:44.166 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-04-22T23:13:44.166 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-22T23:13:44.166 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "smithi067" 2024-04-22T23:13:44.166 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-04-22T23:13:44.167 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum_age": 18, 2024-04-22T23:13:44.167 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-22T23:13:44.167 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T23:13:44.167 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-22T23:13:44.167 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-22T23:13:44.167 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-04-22T23:13:44.167 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-22T23:13:44.167 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T23:13:44.167 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-22T23:13:44.167 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-22T23:13:44.167 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-22T23:13:44.177 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-22T23:13:44.177 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-22T23:13:44.177 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-22T23:13:44.177 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-04-22T23:13:44.177 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-22T23:13:44.177 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-22T23:13:44.177 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-22T23:13:44.177 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-22T23:13:44.177 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-22T23:13:44.177 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-22T23:13:44.177 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-22T23:13:44.177 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-22T23:13:44.178 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-22T23:13:44.178 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-04-22T23:13:44.178 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-22T23:13:44.178 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T23:13:44.178 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-22T23:13:44.178 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-22T23:13:44.178 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-04-22T23:13:44.178 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-22T23:13:44.178 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-22T23:13:44.178 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-22T23:13:44.178 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-22T23:13:44.178 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-22T23:13:44.178 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-22T23:13:44.178 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "restful" 2024-04-22T23:13:44.179 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-04-22T23:13:44.179 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T23:13:44.179 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-04-22T23:13:44.179 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-22T23:13:44.179 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T23:13:44.179 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "modified": "2024-04-22T23:13:18.688362+0000", 2024-04-22T23:13:44.179 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T23:13:44.179 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-04-22T23:13:44.179 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-22T23:13:44.179 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout } 2024-04-22T23:13:44.179 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.525+0000 7fa849bf2700 1 Processor -- start 2024-04-22T23:13:44.179 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.525+0000 7fa849bf2700 1 -- start start 2024-04-22T23:13:44.179 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.525+0000 7fa849bf2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa844107020 0x7fa844109410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:13:44.179 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.525+0000 7fa849bf2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa844074640 con 0x7fa844107020 2024-04-22T23:13:44.179 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.525+0000 7fa8437fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa844107020 0x7fa844109410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:13:44.180 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.525+0000 7fa8437fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa844107020 0x7fa844109410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36896/0 (socket says 172.21.15.67:36896) 2024-04-22T23:13:44.180 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.525+0000 7fa8437fe700 1 -- 172.21.15.67:0/4041188077 learned_addr learned my addr 172.21.15.67:0/4041188077 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:13:44.180 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.525+0000 7fa8437fe700 1 -- 172.21.15.67:0/4041188077 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa844109950 con 0x7fa844107020 2024-04-22T23:13:44.180 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.526+0000 7fa8437fe700 1 --2- 172.21.15.67:0/4041188077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa844107020 0x7fa844109410 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa8340098a0 tx=0x7fa834009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcb6e76ea54f81da server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:13:44.180 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.526+0000 7fa8427fc700 1 -- 172.21.15.67:0/4041188077 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa834004180 con 0x7fa844107020 2024-04-22T23:13:44.180 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.526+0000 7fa8427fc700 1 -- 172.21.15.67:0/4041188077 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa8340042e0 con 0x7fa844107020 2024-04-22T23:13:44.180 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.526+0000 7fa8427fc700 1 -- 172.21.15.67:0/4041188077 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa834011a50 con 0x7fa844107020 2024-04-22T23:13:44.180 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.526+0000 7fa849bf2700 1 -- 172.21.15.67:0/4041188077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa844107020 msgr2=0x7fa844109410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:13:44.180 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.526+0000 7fa849bf2700 1 --2- 172.21.15.67:0/4041188077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa844107020 0x7fa844109410 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa8340098a0 tx=0x7fa834009bb0 comp rx=0 tx=0).stop 2024-04-22T23:13:44.180 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.526+0000 7fa849bf2700 1 -- 172.21.15.67:0/4041188077 shutdown_connections 2024-04-22T23:13:44.180 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.526+0000 7fa849bf2700 1 --2- 172.21.15.67:0/4041188077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa844107020 0x7fa844109410 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:13:44.180 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.526+0000 7fa849bf2700 1 -- 172.21.15.67:0/4041188077 >> 172.21.15.67:0/4041188077 conn(0x7fa8441009f0 msgr2=0x7fa844102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:13:44.180 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.526+0000 7fa849bf2700 1 -- 172.21.15.67:0/4041188077 shutdown_connections 2024-04-22T23:13:44.180 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.526+0000 7fa849bf2700 1 -- 172.21.15.67:0/4041188077 wait complete. 2024-04-22T23:13:44.180 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.527+0000 7fa849bf2700 1 Processor -- start 2024-04-22T23:13:44.181 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.527+0000 7fa849bf2700 1 -- start start 2024-04-22T23:13:44.181 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.527+0000 7fa849bf2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa844107020 0x7fa84419a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:13:44.181 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.527+0000 7fa849bf2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84419a680 con 0x7fa844107020 2024-04-22T23:13:44.181 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.527+0000 7fa8437fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa844107020 0x7fa84419a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:13:44.181 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.527+0000 7fa8437fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa844107020 0x7fa84419a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36898/0 (socket says 172.21.15.67:36898) 2024-04-22T23:13:44.181 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.527+0000 7fa8437fe700 1 -- 172.21.15.67:0/1895426173 learned_addr learned my addr 172.21.15.67:0/1895426173 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:13:44.181 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.528+0000 7fa8437fe700 1 -- 172.21.15.67:0/1895426173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa834009550 con 0x7fa844107020 2024-04-22T23:13:44.181 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.528+0000 7fa8437fe700 1 --2- 172.21.15.67:0/1895426173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa844107020 0x7fa84419a140 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa8340119e0 tx=0x7fa834011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:13:44.181 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.528+0000 7fa840ff9700 1 -- 172.21.15.67:0/1895426173 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa834003b90 con 0x7fa844107020 2024-04-22T23:13:44.181 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.528+0000 7fa840ff9700 1 -- 172.21.15.67:0/1895426173 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fa83401a8a0 con 0x7fa844107020 2024-04-22T23:13:44.181 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.528+0000 7fa849bf2700 1 -- 172.21.15.67:0/1895426173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa84419a880 con 0x7fa844107020 2024-04-22T23:13:44.181 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.528+0000 7fa840ff9700 1 -- 172.21.15.67:0/1895426173 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa834019930 con 0x7fa844107020 2024-04-22T23:13:44.181 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.528+0000 7fa849bf2700 1 -- 172.21.15.67:0/1895426173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa84419ad20 con 0x7fa844107020 2024-04-22T23:13:44.181 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.529+0000 7fa840ff9700 1 -- 172.21.15.67:0/1895426173 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 2) v1 ==== 45039+0+0 (secure 0 0 0) 0x7fa834019bf0 con 0x7fa844107020 2024-04-22T23:13:44.181 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.529+0000 7fa849bf2700 1 -- 172.21.15.67:0/1895426173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa844062430 con 0x7fa844107020 2024-04-22T23:13:44.182 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.529+0000 7fa840ff9700 1 -- 172.21.15.67:0/1895426173 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa83404c290 con 0x7fa844107020 2024-04-22T23:13:44.182 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.532+0000 7fa840ff9700 1 -- 172.21.15.67:0/1895426173 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fa844062430 con 0x7fa844107020 2024-04-22T23:13:44.182 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.605+0000 7fa849bf2700 1 -- 172.21.15.67:0/1895426173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa84419d730 con 0x7fa844107020 2024-04-22T23:13:44.182 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.606+0000 7fa840ff9700 1 -- 172.21.15.67:0/1895426173 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1247 (secure 0 0 0) 0x7fa8340189a0 con 0x7fa844107020 2024-04-22T23:13:44.182 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.607+0000 7fa849bf2700 1 -- 172.21.15.67:0/1895426173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa844107020 msgr2=0x7fa84419a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:13:44.182 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.607+0000 7fa849bf2700 1 --2- 172.21.15.67:0/1895426173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa844107020 0x7fa84419a140 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa8340119e0 tx=0x7fa834011ac0 comp rx=0 tx=0).stop 2024-04-22T23:13:44.182 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.607+0000 7fa849bf2700 1 -- 172.21.15.67:0/1895426173 shutdown_connections 2024-04-22T23:13:44.182 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.607+0000 7fa849bf2700 1 --2- 172.21.15.67:0/1895426173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa844107020 0x7fa84419a140 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:13:44.182 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.607+0000 7fa849bf2700 1 -- 172.21.15.67:0/1895426173 >> 172.21.15.67:0/1895426173 conn(0x7fa8441009f0 msgr2=0x7fa844101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:13:44.182 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.607+0000 7fa849bf2700 1 -- 172.21.15.67:0/1895426173 shutdown_connections 2024-04-22T23:13:44.182 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:43.607+0000 7fa849bf2700 1 -- 172.21.15.67:0/1895426173 wait complete. 2024-04-22T23:13:44.182 INFO:teuthology.orchestra.run.smithi067.stdout:mgr not available, waiting (3/15)... 2024-04-22T23:13:44.384 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:44 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/1895426173' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T23:13:44.384 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:44 smithi067 ceph-mon[26453]: mgrmap e3: smithi067.whdcsr(active, since 1.00626s) 2024-04-22T23:13:47.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:46 smithi067 ceph-mon[26453]: mgrmap e4: smithi067.whdcsr(active, since 2s) 2024-04-22T23:13:47.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:46 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/2607114318' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T23:13:47.437 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-04-22T23:13:47.437 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout { 2024-04-22T23:13:47.437 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "fsid": "d46c346e-00fd-11ef-bc93-c7b262605968", 2024-04-22T23:13:47.437 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "health": { 2024-04-22T23:13:47.437 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-22T23:13:47.437 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-22T23:13:47.437 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-22T23:13:47.437 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-04-22T23:13:47.437 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-22T23:13:47.437 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-22T23:13:47.438 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 0 2024-04-22T23:13:47.438 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-04-22T23:13:47.438 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-22T23:13:47.438 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "smithi067" 2024-04-22T23:13:47.438 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-04-22T23:13:47.438 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum_age": 21, 2024-04-22T23:13:47.438 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-22T23:13:47.438 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T23:13:47.438 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-22T23:13:47.438 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-22T23:13:47.438 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-04-22T23:13:47.438 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-22T23:13:47.438 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T23:13:47.438 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-22T23:13:47.439 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-22T23:13:47.439 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-22T23:13:47.447 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-22T23:13:47.447 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-22T23:13:47.447 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-22T23:13:47.447 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-04-22T23:13:47.448 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-22T23:13:47.448 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-22T23:13:47.448 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-22T23:13:47.448 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-22T23:13:47.448 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-22T23:13:47.448 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-22T23:13:47.448 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-22T23:13:47.448 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-22T23:13:47.448 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-22T23:13:47.448 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-04-22T23:13:47.448 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-22T23:13:47.448 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T23:13:47.448 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-22T23:13:47.448 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-22T23:13:47.448 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-04-22T23:13:47.449 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-22T23:13:47.449 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-22T23:13:47.449 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-22T23:13:47.449 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-22T23:13:47.449 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-22T23:13:47.449 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-22T23:13:47.449 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "restful" 2024-04-22T23:13:47.449 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-04-22T23:13:47.449 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T23:13:47.449 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-04-22T23:13:47.449 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-22T23:13:47.449 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T23:13:47.449 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "modified": "2024-04-22T23:13:18.688362+0000", 2024-04-22T23:13:47.449 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T23:13:47.450 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-04-22T23:13:47.450 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-22T23:13:47.450 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout } 2024-04-22T23:13:47.450 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.579+0000 7f49645e2700 1 Processor -- start 2024-04-22T23:13:47.450 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.579+0000 7f49645e2700 1 -- start start 2024-04-22T23:13:47.450 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.579+0000 7f49645e2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f495c07be80 0x7f495c07a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:13:47.450 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.579+0000 7f49645e2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f495c07a8b0 con 0x7f495c07be80 2024-04-22T23:13:47.450 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.580+0000 7f496237e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f495c07be80 0x7f495c07a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:13:47.450 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.580+0000 7f496237e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f495c07be80 0x7f495c07a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36902/0 (socket says 172.21.15.67:36902) 2024-04-22T23:13:47.450 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.580+0000 7f496237e700 1 -- 172.21.15.67:0/276701935 learned_addr learned my addr 172.21.15.67:0/276701935 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:13:47.450 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.580+0000 7f496237e700 1 -- 172.21.15.67:0/276701935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f495c07a9f0 con 0x7f495c07be80 2024-04-22T23:13:47.450 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.580+0000 7f496237e700 1 --2- 172.21.15.67:0/276701935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f495c07be80 0x7f495c07a2e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f49580098a0 tx=0x7f4958009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e00766290b6e3ac9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:13:47.450 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.580+0000 7f496137c700 1 -- 172.21.15.67:0/276701935 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4958004180 con 0x7f495c07be80 2024-04-22T23:13:47.450 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.581+0000 7f496137c700 1 -- 172.21.15.67:0/276701935 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f49580042e0 con 0x7f495c07be80 2024-04-22T23:13:47.451 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.581+0000 7f496137c700 1 -- 172.21.15.67:0/276701935 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4958011a50 con 0x7f495c07be80 2024-04-22T23:13:47.451 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.581+0000 7f49645e2700 1 -- 172.21.15.67:0/276701935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f495c07be80 msgr2=0x7f495c07a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:13:47.451 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.581+0000 7f49645e2700 1 --2- 172.21.15.67:0/276701935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f495c07be80 0x7f495c07a2e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f49580098a0 tx=0x7f4958009bb0 comp rx=0 tx=0).stop 2024-04-22T23:13:47.451 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.581+0000 7f49645e2700 1 -- 172.21.15.67:0/276701935 shutdown_connections 2024-04-22T23:13:47.451 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.581+0000 7f49645e2700 1 --2- 172.21.15.67:0/276701935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f495c07be80 0x7f495c07a2e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:13:47.451 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.581+0000 7f49645e2700 1 -- 172.21.15.67:0/276701935 >> 172.21.15.67:0/276701935 conn(0x7f495c101b40 msgr2=0x7f495c103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:13:47.451 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.581+0000 7f49645e2700 1 -- 172.21.15.67:0/276701935 shutdown_connections 2024-04-22T23:13:47.451 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.581+0000 7f49645e2700 1 -- 172.21.15.67:0/276701935 wait complete. 2024-04-22T23:13:47.451 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.582+0000 7f49645e2700 1 Processor -- start 2024-04-22T23:13:47.451 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.582+0000 7f49645e2700 1 -- start start 2024-04-22T23:13:47.451 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.582+0000 7f49645e2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f495c07be80 0x7f495c19c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:13:47.451 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.582+0000 7f49645e2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f495c19c960 con 0x7f495c07be80 2024-04-22T23:13:47.451 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.582+0000 7f496237e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f495c07be80 0x7f495c19c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:13:47.451 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.582+0000 7f496237e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f495c07be80 0x7f495c19c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36906/0 (socket says 172.21.15.67:36906) 2024-04-22T23:13:47.452 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.582+0000 7f496237e700 1 -- 172.21.15.67:0/2607114318 learned_addr learned my addr 172.21.15.67:0/2607114318 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:13:47.452 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.583+0000 7f496237e700 1 -- 172.21.15.67:0/2607114318 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4958009550 con 0x7f495c07be80 2024-04-22T23:13:47.452 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.583+0000 7f496237e700 1 --2- 172.21.15.67:0/2607114318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f495c07be80 0x7f495c19c420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4958009520 tx=0x7f4958011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:13:47.452 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.583+0000 7f49537fe700 1 -- 172.21.15.67:0/2607114318 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4958003a80 con 0x7f495c07be80 2024-04-22T23:13:47.452 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.583+0000 7f49537fe700 1 -- 172.21.15.67:0/2607114318 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f4958003be0 con 0x7f495c07be80 2024-04-22T23:13:47.452 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.583+0000 7f49645e2700 1 -- 172.21.15.67:0/2607114318 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f495c19cb60 con 0x7f495c07be80 2024-04-22T23:13:47.452 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.583+0000 7f49537fe700 1 -- 172.21.15.67:0/2607114318 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49580197b0 con 0x7f495c07be80 2024-04-22T23:13:47.452 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.583+0000 7f49645e2700 1 -- 172.21.15.67:0/2607114318 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f495c19d000 con 0x7f495c07be80 2024-04-22T23:13:47.452 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.584+0000 7f49537fe700 1 -- 172.21.15.67:0/2607114318 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) v1 ==== 45272+0+0 (secure 0 0 0) 0x7f4958019a70 con 0x7f495c07be80 2024-04-22T23:13:47.452 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.584+0000 7f49537fe700 1 --2- 172.21.15.67:0/2607114318 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7f4948038280 0x7f494803a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:13:47.452 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.584+0000 7f49645e2700 1 -- 172.21.15.67:0/2607114318 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f495c10c8c0 con 0x7f495c07be80 2024-04-22T23:13:47.452 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.584+0000 7f49537fe700 1 -- 172.21.15.67:0/2607114318 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f495804c4d0 con 0x7f495c07be80 2024-04-22T23:13:47.452 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.584+0000 7f4961b7d700 1 --2- 172.21.15.67:0/2607114318 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7f4948038280 0x7f494803a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:13:47.452 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.585+0000 7f4961b7d700 1 --2- 172.21.15.67:0/2607114318 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7f4948038280 0x7f494803a740 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f494c006fd0 tx=0x7f494c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:13:47.453 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.589+0000 7f49537fe700 1 -- 172.21.15.67:0/2607114318 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f495801ac30 con 0x7f495c07be80 2024-04-22T23:13:47.453 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.867+0000 7f49645e2700 1 -- 172.21.15.67:0/2607114318 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f495c19fde0 con 0x7f495c07be80 2024-04-22T23:13:47.453 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.867+0000 7f49537fe700 1 -- 172.21.15.67:0/2607114318 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f4958023390 con 0x7f495c07be80 2024-04-22T23:13:47.453 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.869+0000 7f49645e2700 1 -- 172.21.15.67:0/2607114318 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7f4948038280 msgr2=0x7f494803a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:13:47.453 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.869+0000 7f49645e2700 1 --2- 172.21.15.67:0/2607114318 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7f4948038280 0x7f494803a740 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f494c006fd0 tx=0x7f494c006e40 comp rx=0 tx=0).stop 2024-04-22T23:13:47.453 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.869+0000 7f49645e2700 1 -- 172.21.15.67:0/2607114318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f495c07be80 msgr2=0x7f495c19c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:13:47.453 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.869+0000 7f49645e2700 1 --2- 172.21.15.67:0/2607114318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f495c07be80 0x7f495c19c420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4958009520 tx=0x7f4958011ab0 comp rx=0 tx=0).stop 2024-04-22T23:13:47.453 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.870+0000 7f49645e2700 1 -- 172.21.15.67:0/2607114318 shutdown_connections 2024-04-22T23:13:47.453 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.870+0000 7f49645e2700 1 --2- 172.21.15.67:0/2607114318 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7f4948038280 0x7f494803a740 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:13:47.453 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.870+0000 7f49645e2700 1 --2- 172.21.15.67:0/2607114318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f495c07be80 0x7f495c19c420 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:13:47.453 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.870+0000 7f49645e2700 1 -- 172.21.15.67:0/2607114318 >> 172.21.15.67:0/2607114318 conn(0x7f495c101b40 msgr2=0x7f495c1031a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:13:47.453 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.870+0000 7f49645e2700 1 -- 172.21.15.67:0/2607114318 shutdown_connections 2024-04-22T23:13:47.453 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:46.870+0000 7f49645e2700 1 -- 172.21.15.67:0/2607114318 wait complete. 2024-04-22T23:13:47.453 INFO:teuthology.orchestra.run.smithi067.stdout:mgr is available 2024-04-22T23:13:48.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:48 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/1674665675' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-22T23:13:48.592 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-04-22T23:13:48.592 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout [global] 2024-04-22T23:13:48.592 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout fsid = d46c346e-00fd-11ef-bc93-c7b262605968 2024-04-22T23:13:48.592 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-22T23:13:48.592 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-22T23:13:48.592 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-22T23:13:48.592 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-22T23:13:48.592 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-04-22T23:13:48.592 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-22T23:13:48.593 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-22T23:13:48.593 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-04-22T23:13:48.593 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout [osd] 2024-04-22T23:13:48.593 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-22T23:13:48.593 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-22T23:13:48.593 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-22T23:13:48.593 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-22T23:13:48.593 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-22T23:13:48.593 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.888+0000 7f5b8b319700 1 Processor -- start 2024-04-22T23:13:48.593 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.888+0000 7f5b8b319700 1 -- start start 2024-04-22T23:13:48.593 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.888+0000 7f5b8b319700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b84106f30 0x7f5b84109320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:13:48.593 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.888+0000 7f5b8b319700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b84074680 con 0x7f5b84106f30 2024-04-22T23:13:48.593 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.889+0000 7f5b890b5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b84106f30 0x7f5b84109320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:13:48.593 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.889+0000 7f5b890b5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b84106f30 0x7f5b84109320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36914/0 (socket says 172.21.15.67:36914) 2024-04-22T23:13:48.594 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.889+0000 7f5b890b5700 1 -- 172.21.15.67:0/857984468 learned_addr learned my addr 172.21.15.67:0/857984468 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:13:48.594 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.889+0000 7f5b890b5700 1 -- 172.21.15.67:0/857984468 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b84109860 con 0x7f5b84106f30 2024-04-22T23:13:48.594 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.889+0000 7f5b890b5700 1 --2- 172.21.15.67:0/857984468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b84106f30 0x7f5b84109320 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5b740098a0 tx=0x7f5b74009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0678828589d8be1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:13:48.594 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.889+0000 7f5b7bfff700 1 -- 172.21.15.67:0/857984468 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b74004180 con 0x7f5b84106f30 2024-04-22T23:13:48.594 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.889+0000 7f5b7bfff700 1 -- 172.21.15.67:0/857984468 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f5b740042e0 con 0x7f5b84106f30 2024-04-22T23:13:48.594 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.889+0000 7f5b7bfff700 1 -- 172.21.15.67:0/857984468 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b74011a50 con 0x7f5b84106f30 2024-04-22T23:13:48.594 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.890+0000 7f5b8b319700 1 -- 172.21.15.67:0/857984468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b84106f30 msgr2=0x7f5b84109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:13:48.594 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.890+0000 7f5b8b319700 1 --2- 172.21.15.67:0/857984468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b84106f30 0x7f5b84109320 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5b740098a0 tx=0x7f5b74009bb0 comp rx=0 tx=0).stop 2024-04-22T23:13:48.594 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.890+0000 7f5b8b319700 1 -- 172.21.15.67:0/857984468 shutdown_connections 2024-04-22T23:13:48.594 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.890+0000 7f5b8b319700 1 --2- 172.21.15.67:0/857984468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b84106f30 0x7f5b84109320 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:13:48.594 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.890+0000 7f5b8b319700 1 -- 172.21.15.67:0/857984468 >> 172.21.15.67:0/857984468 conn(0x7f5b84100940 msgr2=0x7f5b84102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:13:48.594 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.890+0000 7f5b8b319700 1 -- 172.21.15.67:0/857984468 shutdown_connections 2024-04-22T23:13:48.594 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.890+0000 7f5b8b319700 1 -- 172.21.15.67:0/857984468 wait complete. 2024-04-22T23:13:48.594 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.891+0000 7f5b8b319700 1 Processor -- start 2024-04-22T23:13:48.594 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.891+0000 7f5b8b319700 1 -- start start 2024-04-22T23:13:48.595 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.891+0000 7f5b8b319700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b84106f30 0x7f5b84106890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:13:48.595 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.891+0000 7f5b8b319700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b84106dd0 con 0x7f5b84106f30 2024-04-22T23:13:48.595 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.891+0000 7f5b890b5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b84106f30 0x7f5b84106890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:13:48.595 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.891+0000 7f5b890b5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b84106f30 0x7f5b84106890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36930/0 (socket says 172.21.15.67:36930) 2024-04-22T23:13:48.595 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.891+0000 7f5b890b5700 1 -- 172.21.15.67:0/1674665675 learned_addr learned my addr 172.21.15.67:0/1674665675 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:13:48.595 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.891+0000 7f5b890b5700 1 -- 172.21.15.67:0/1674665675 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b74009550 con 0x7f5b84106f30 2024-04-22T23:13:48.595 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.891+0000 7f5b890b5700 1 --2- 172.21.15.67:0/1674665675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b84106f30 0x7f5b84106890 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5b740119e0 tx=0x7f5b74011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:13:48.595 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.891+0000 7f5b7a7fc700 1 -- 172.21.15.67:0/1674665675 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b74003b90 con 0x7f5b84106f30 2024-04-22T23:13:48.595 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.892+0000 7f5b7a7fc700 1 -- 172.21.15.67:0/1674665675 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f5b7401a8a0 con 0x7f5b84106f30 2024-04-22T23:13:48.595 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.892+0000 7f5b8b319700 1 -- 172.21.15.67:0/1674665675 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b84104ee0 con 0x7f5b84106f30 2024-04-22T23:13:48.595 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.892+0000 7f5b7a7fc700 1 -- 172.21.15.67:0/1674665675 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b74019930 con 0x7f5b84106f30 2024-04-22T23:13:48.595 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.892+0000 7f5b8b319700 1 -- 172.21.15.67:0/1674665675 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b84105380 con 0x7f5b84106f30 2024-04-22T23:13:48.595 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.892+0000 7f5b7a7fc700 1 -- 172.21.15.67:0/1674665675 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) v1 ==== 45272+0+0 (secure 0 0 0) 0x7f5b74019ba0 con 0x7f5b84106f30 2024-04-22T23:13:48.595 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.893+0000 7f5b7a7fc700 1 --2- 172.21.15.67:0/1674665675 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7f5b700382e0 0x7f5b7003a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:13:48.596 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.893+0000 7f5b7a7fc700 1 -- 172.21.15.67:0/1674665675 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f5b7404c650 con 0x7f5b84106f30 2024-04-22T23:13:48.596 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.893+0000 7f5b888b4700 1 --2- 172.21.15.67:0/1674665675 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7f5b700382e0 0x7f5b7003a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:13:48.596 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.893+0000 7f5b8b319700 1 -- 172.21.15.67:0/1674665675 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b8410a5e0 con 0x7f5b84106f30 2024-04-22T23:13:48.596 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.893+0000 7f5b888b4700 1 --2- 172.21.15.67:0/1674665675 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7f5b700382e0 0x7f5b7003a7a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5b80006fd0 tx=0x7f5b80006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:13:48.596 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:47.898+0000 7f5b7a7fc700 1 -- 172.21.15.67:0/1674665675 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f5b74018530 con 0x7f5b84106f30 2024-04-22T23:13:48.596 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:48.093+0000 7f5b8b319700 1 -- 172.21.15.67:0/1674665675 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f5b8410a800 con 0x7f5b84106f30 2024-04-22T23:13:48.596 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:48.094+0000 7f5b7a7fc700 1 -- 172.21.15.67:0/1674665675 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f5b74018350 con 0x7f5b84106f30 2024-04-22T23:13:48.596 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:48.096+0000 7f5b8b319700 1 -- 172.21.15.67:0/1674665675 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7f5b700382e0 msgr2=0x7f5b7003a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:13:48.596 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:48.096+0000 7f5b8b319700 1 --2- 172.21.15.67:0/1674665675 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7f5b700382e0 0x7f5b7003a7a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5b80006fd0 tx=0x7f5b80006e40 comp rx=0 tx=0).stop 2024-04-22T23:13:48.596 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:48.096+0000 7f5b8b319700 1 -- 172.21.15.67:0/1674665675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b84106f30 msgr2=0x7f5b84106890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:13:48.596 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:48.096+0000 7f5b8b319700 1 --2- 172.21.15.67:0/1674665675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b84106f30 0x7f5b84106890 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5b740119e0 tx=0x7f5b74011ac0 comp rx=0 tx=0).stop 2024-04-22T23:13:48.596 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:48.096+0000 7f5b8b319700 1 -- 172.21.15.67:0/1674665675 shutdown_connections 2024-04-22T23:13:48.596 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:48.096+0000 7f5b8b319700 1 --2- 172.21.15.67:0/1674665675 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7f5b700382e0 0x7f5b7003a7a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:13:48.596 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:48.096+0000 7f5b8b319700 1 --2- 172.21.15.67:0/1674665675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b84106f30 0x7f5b84106890 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:13:48.596 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:48.096+0000 7f5b8b319700 1 -- 172.21.15.67:0/1674665675 >> 172.21.15.67:0/1674665675 conn(0x7f5b84100940 msgr2=0x7f5b841016a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:13:48.597 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:48.096+0000 7f5b8b319700 1 -- 172.21.15.67:0/1674665675 shutdown_connections 2024-04-22T23:13:48.597 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:48.096+0000 7f5b8b319700 1 -- 172.21.15.67:0/1674665675 wait complete. 2024-04-22T23:13:48.597 INFO:teuthology.orchestra.run.smithi067.stdout:Enabling cephadm module... 2024-04-22T23:13:49.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:49 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/887820889' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-22T23:13:49.918 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.026+0000 7fe7cf59e700 1 Processor -- start 2024-04-22T23:13:49.918 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.026+0000 7fe7cf59e700 1 -- start start 2024-04-22T23:13:49.918 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.026+0000 7fe7cf59e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7d01063a0 0x7fe7d01067c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:13:49.918 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.026+0000 7fe7cf59e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d0106d90 con 0x7fe7d01063a0 2024-04-22T23:13:49.918 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.026+0000 7fe7ce59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7d01063a0 0x7fe7d01067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:13:49.919 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.026+0000 7fe7ce59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7d01063a0 0x7fe7d01067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45158/0 (socket says 172.21.15.67:45158) 2024-04-22T23:13:49.919 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.026+0000 7fe7ce59c700 1 -- 172.21.15.67:0/3145187084 learned_addr learned my addr 172.21.15.67:0/3145187084 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:13:49.919 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.026+0000 7fe7ce59c700 1 -- 172.21.15.67:0/3145187084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d01075a0 con 0x7fe7d01063a0 2024-04-22T23:13:49.919 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.027+0000 7fe7ce59c700 1 --2- 172.21.15.67:0/3145187084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7d01063a0 0x7fe7d01067c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe7c00098a0 tx=0x7fe7c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbea9cc4000c7b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:13:49.919 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.027+0000 7fe7cd59a700 1 -- 172.21.15.67:0/3145187084 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7c0004180 con 0x7fe7d01063a0 2024-04-22T23:13:49.919 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.027+0000 7fe7cd59a700 1 -- 172.21.15.67:0/3145187084 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe7c00042e0 con 0x7fe7d01063a0 2024-04-22T23:13:49.919 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.027+0000 7fe7cd59a700 1 -- 172.21.15.67:0/3145187084 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7c0011a50 con 0x7fe7d01063a0 2024-04-22T23:13:49.919 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.027+0000 7fe7cf59e700 1 -- 172.21.15.67:0/3145187084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7d01063a0 msgr2=0x7fe7d01067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:13:49.919 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.027+0000 7fe7cf59e700 1 --2- 172.21.15.67:0/3145187084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7d01063a0 0x7fe7d01067c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe7c00098a0 tx=0x7fe7c0009bb0 comp rx=0 tx=0).stop 2024-04-22T23:13:49.919 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.027+0000 7fe7cf59e700 1 -- 172.21.15.67:0/3145187084 shutdown_connections 2024-04-22T23:13:49.919 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.027+0000 7fe7cf59e700 1 --2- 172.21.15.67:0/3145187084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7d01063a0 0x7fe7d01067c0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:13:49.919 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.027+0000 7fe7cf59e700 1 -- 172.21.15.67:0/3145187084 >> 172.21.15.67:0/3145187084 conn(0x7fe7d0101980 msgr2=0x7fe7d0103da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:13:49.919 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.027+0000 7fe7cf59e700 1 -- 172.21.15.67:0/3145187084 shutdown_connections 2024-04-22T23:13:49.919 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.027+0000 7fe7cf59e700 1 -- 172.21.15.67:0/3145187084 wait complete. 2024-04-22T23:13:49.920 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.028+0000 7fe7cf59e700 1 Processor -- start 2024-04-22T23:13:49.920 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.028+0000 7fe7cf59e700 1 -- start start 2024-04-22T23:13:49.920 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.028+0000 7fe7cf59e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7d01063a0 0x7fe7d019a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:13:49.920 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.028+0000 7fe7cf59e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d019a570 con 0x7fe7d01063a0 2024-04-22T23:13:49.920 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.028+0000 7fe7ce59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7d01063a0 0x7fe7d019a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:13:49.920 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.028+0000 7fe7ce59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7d01063a0 0x7fe7d019a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45164/0 (socket says 172.21.15.67:45164) 2024-04-22T23:13:49.920 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.028+0000 7fe7ce59c700 1 -- 172.21.15.67:0/887820889 learned_addr learned my addr 172.21.15.67:0/887820889 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:13:49.920 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.029+0000 7fe7ce59c700 1 -- 172.21.15.67:0/887820889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7c0009550 con 0x7fe7d01063a0 2024-04-22T23:13:49.920 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.029+0000 7fe7ce59c700 1 --2- 172.21.15.67:0/887820889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7d01063a0 0x7fe7d019a030 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe7c0009520 tx=0x7fe7c0011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:13:49.920 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.029+0000 7fe7bf7fe700 1 -- 172.21.15.67:0/887820889 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7c0003a80 con 0x7fe7d01063a0 2024-04-22T23:13:49.920 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.029+0000 7fe7bf7fe700 1 -- 172.21.15.67:0/887820889 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fe7c0003be0 con 0x7fe7d01063a0 2024-04-22T23:13:49.920 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.029+0000 7fe7cf59e700 1 -- 172.21.15.67:0/887820889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7d019a770 con 0x7fe7d01063a0 2024-04-22T23:13:49.920 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.029+0000 7fe7bf7fe700 1 -- 172.21.15.67:0/887820889 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7c00197b0 con 0x7fe7d01063a0 2024-04-22T23:13:49.920 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.029+0000 7fe7cf59e700 1 -- 172.21.15.67:0/887820889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7d019ac10 con 0x7fe7d01063a0 2024-04-22T23:13:49.921 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.030+0000 7fe7bf7fe700 1 -- 172.21.15.67:0/887820889 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) v1 ==== 45272+0+0 (secure 0 0 0) 0x7fe7c0019a20 con 0x7fe7d01063a0 2024-04-22T23:13:49.921 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.030+0000 7fe7cf59e700 1 -- 172.21.15.67:0/887820889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7d00623c0 con 0x7fe7d01063a0 2024-04-22T23:13:49.921 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.030+0000 7fe7bf7fe700 1 --2- 172.21.15.67:0/887820889 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7fe7b80382d0 0x7fe7b803a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:13:49.921 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.030+0000 7fe7bf7fe700 1 -- 172.21.15.67:0/887820889 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe7c004c630 con 0x7fe7d01063a0 2024-04-22T23:13:49.921 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.030+0000 7fe7cdd9b700 1 --2- 172.21.15.67:0/887820889 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7fe7b80382d0 0x7fe7b803a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:13:49.921 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.031+0000 7fe7cdd9b700 1 --2- 172.21.15.67:0/887820889 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7fe7b80382d0 0x7fe7b803a790 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4006fd0 tx=0x7fe7c4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:13:49.921 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.035+0000 7fe7bf7fe700 1 -- 172.21.15.67:0/887820889 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fe7c002b750 con 0x7fe7d01063a0 2024-04-22T23:13:49.921 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.292+0000 7fe7cf59e700 1 -- 172.21.15.67:0/887820889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fe7d019dda0 con 0x7fe7d01063a0 2024-04-22T23:13:49.921 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.345+0000 7fe7bf7fe700 1 -- 172.21.15.67:0/887820889 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 5) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fe7c0019cd0 con 0x7fe7d01063a0 2024-04-22T23:13:49.921 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.346+0000 7fe7bf7fe700 1 -- 172.21.15.67:0/887820889 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fe7c0023550 con 0x7fe7d01063a0 2024-04-22T23:13:49.921 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.348+0000 7fe7cf59e700 1 -- 172.21.15.67:0/887820889 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7fe7b80382d0 msgr2=0x7fe7b803a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:13:49.921 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.348+0000 7fe7cf59e700 1 --2- 172.21.15.67:0/887820889 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7fe7b80382d0 0x7fe7b803a790 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4006fd0 tx=0x7fe7c4006e40 comp rx=0 tx=0).stop 2024-04-22T23:13:49.921 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.348+0000 7fe7cf59e700 1 -- 172.21.15.67:0/887820889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7d01063a0 msgr2=0x7fe7d019a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:13:49.921 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.348+0000 7fe7cf59e700 1 --2- 172.21.15.67:0/887820889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7d01063a0 0x7fe7d019a030 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe7c0009520 tx=0x7fe7c0011ab0 comp rx=0 tx=0).stop 2024-04-22T23:13:49.922 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.349+0000 7fe7cf59e700 1 -- 172.21.15.67:0/887820889 shutdown_connections 2024-04-22T23:13:49.922 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.349+0000 7fe7cf59e700 1 --2- 172.21.15.67:0/887820889 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7fe7b80382d0 0x7fe7b803a790 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:13:49.922 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.349+0000 7fe7cf59e700 1 --2- 172.21.15.67:0/887820889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7d01063a0 0x7fe7d019a030 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:13:49.922 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.349+0000 7fe7cf59e700 1 -- 172.21.15.67:0/887820889 >> 172.21.15.67:0/887820889 conn(0x7fe7d0101980 msgr2=0x7fe7d0102720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:13:49.922 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.349+0000 7fe7cf59e700 1 -- 172.21.15.67:0/887820889 shutdown_connections 2024-04-22T23:13:49.922 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:49.349+0000 7fe7cf59e700 1 -- 172.21.15.67:0/887820889 wait complete. 2024-04-22T23:13:50.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:50 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/887820889' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-22T23:13:50.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:50 smithi067 ceph-mon[26453]: mgrmap e5: smithi067.whdcsr(active, since 6s) 2024-04-22T23:13:51.275 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout { 2024-04-22T23:13:51.276 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-22T23:13:51.276 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-22T23:13:51.276 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "active_name": "smithi067.whdcsr", 2024-04-22T23:13:51.276 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-22T23:13:51.276 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout } 2024-04-22T23:13:51.276 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.347+0000 7f9b1f78d700 1 Processor -- start 2024-04-22T23:13:51.276 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.347+0000 7f9b1f78d700 1 -- start start 2024-04-22T23:13:51.276 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.347+0000 7f9b1f78d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b18105470 0x7f9b18105890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:13:51.276 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.347+0000 7f9b1f78d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b18105e60 con 0x7f9b18105470 2024-04-22T23:13:51.276 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.348+0000 7f9b1d529700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b18105470 0x7f9b18105890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:13:51.276 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.348+0000 7f9b1d529700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b18105470 0x7f9b18105890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45188/0 (socket says 172.21.15.67:45188) 2024-04-22T23:13:51.276 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.348+0000 7f9b1d529700 1 -- 172.21.15.67:0/422321448 learned_addr learned my addr 172.21.15.67:0/422321448 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:13:51.277 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.348+0000 7f9b1d529700 1 -- 172.21.15.67:0/422321448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b18106670 con 0x7f9b18105470 2024-04-22T23:13:51.277 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.348+0000 7f9b1d529700 1 --2- 172.21.15.67:0/422321448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b18105470 0x7f9b18105890 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9b08010b60 tx=0x7f9b08010e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=233a4d11044db53 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:13:51.277 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.350+0000 7f9b0ffff700 1 -- 172.21.15.67:0/422321448 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b0800bca0 con 0x7f9b18105470 2024-04-22T23:13:51.277 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.350+0000 7f9b0ffff700 1 -- 172.21.15.67:0/422321448 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9b0800be00 con 0x7f9b18105470 2024-04-22T23:13:51.277 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.350+0000 7f9b0ffff700 1 -- 172.21.15.67:0/422321448 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b08018420 con 0x7f9b18105470 2024-04-22T23:13:51.277 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.350+0000 7f9b1f78d700 1 -- 172.21.15.67:0/422321448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b18105470 msgr2=0x7f9b18105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:13:51.277 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.350+0000 7f9b1f78d700 1 --2- 172.21.15.67:0/422321448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b18105470 0x7f9b18105890 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9b08010b60 tx=0x7f9b08010e70 comp rx=0 tx=0).stop 2024-04-22T23:13:51.277 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.350+0000 7f9b1f78d700 1 -- 172.21.15.67:0/422321448 shutdown_connections 2024-04-22T23:13:51.277 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.350+0000 7f9b1f78d700 1 --2- 172.21.15.67:0/422321448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b18105470 0x7f9b18105890 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:13:51.277 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.350+0000 7f9b1f78d700 1 -- 172.21.15.67:0/422321448 >> 172.21.15.67:0/422321448 conn(0x7f9b181009f0 msgr2=0x7f9b18102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:13:51.277 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.351+0000 7f9b1f78d700 1 -- 172.21.15.67:0/422321448 shutdown_connections 2024-04-22T23:13:51.277 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.351+0000 7f9b1f78d700 1 -- 172.21.15.67:0/422321448 wait complete. 2024-04-22T23:13:51.277 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.351+0000 7f9b1f78d700 1 Processor -- start 2024-04-22T23:13:51.277 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.351+0000 7f9b1f78d700 1 -- start start 2024-04-22T23:13:51.277 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.351+0000 7f9b1f78d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b18105470 0x7f9b18197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:13:51.278 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.351+0000 7f9b1f78d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b181984e0 con 0x7f9b18105470 2024-04-22T23:13:51.278 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.351+0000 7f9b1d529700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b18105470 0x7f9b18197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:13:51.278 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.351+0000 7f9b1d529700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b18105470 0x7f9b18197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45196/0 (socket says 172.21.15.67:45196) 2024-04-22T23:13:51.278 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.351+0000 7f9b1d529700 1 -- 172.21.15.67:0/541769405 learned_addr learned my addr 172.21.15.67:0/541769405 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:13:51.278 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.352+0000 7f9b1d529700 1 -- 172.21.15.67:0/541769405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b08010810 con 0x7f9b18105470 2024-04-22T23:13:51.278 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.352+0000 7f9b1d529700 1 --2- 172.21.15.67:0/541769405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b18105470 0x7f9b18197fa0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9b08006cb0 tx=0x7f9b0800bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:13:51.278 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.352+0000 7f9b0e7fc700 1 -- 172.21.15.67:0/541769405 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b08003f00 con 0x7f9b18105470 2024-04-22T23:13:51.278 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.352+0000 7f9b0e7fc700 1 -- 172.21.15.67:0/541769405 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f9b080044d0 con 0x7f9b18105470 2024-04-22T23:13:51.278 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.352+0000 7f9b1f78d700 1 -- 172.21.15.67:0/541769405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b181986e0 con 0x7f9b18105470 2024-04-22T23:13:51.278 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.352+0000 7f9b0e7fc700 1 -- 172.21.15.67:0/541769405 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b08022d50 con 0x7f9b18105470 2024-04-22T23:13:51.278 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.352+0000 7f9b1f78d700 1 -- 172.21.15.67:0/541769405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b18198b80 con 0x7f9b18105470 2024-04-22T23:13:51.278 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.353+0000 7f9b0e7fc700 1 -- 172.21.15.67:0/541769405 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f9b0802b3b0 con 0x7f9b18105470 2024-04-22T23:13:51.278 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.353+0000 7f9b0e7fc700 1 --2- 172.21.15.67:0/541769405 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7f9b040382d0 0x7f9b0403a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:13:51.278 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.353+0000 7f9b0e7fc700 1 -- 172.21.15.67:0/541769405 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9b0804fea0 con 0x7f9b18105470 2024-04-22T23:13:51.279 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.353+0000 7f9b1cd28700 1 -- 172.21.15.67:0/541769405 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7f9b040382d0 msgr2=0x7f9b0403a790 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/2451437776 2024-04-22T23:13:51.279 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.353+0000 7f9b1cd28700 1 --2- 172.21.15.67:0/541769405 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7f9b040382d0 0x7f9b0403a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T23:13:51.279 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.353+0000 7f9b1f78d700 1 -- 172.21.15.67:0/541769405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b18191d00 con 0x7f9b18105470 2024-04-22T23:13:51.279 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.358+0000 7f9b0e7fc700 1 -- 172.21.15.67:0/541769405 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f9b08027070 con 0x7f9b18105470 2024-04-22T23:13:51.279 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.554+0000 7f9b1cd28700 1 -- 172.21.15.67:0/541769405 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7f9b040382d0 msgr2=0x7f9b0403a790 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/2451437776 2024-04-22T23:13:51.279 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.554+0000 7f9b1cd28700 1 --2- 172.21.15.67:0/541769405 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7f9b040382d0 0x7f9b0403a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T23:13:51.279 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.646+0000 7f9b1f78d700 1 -- 172.21.15.67:0/541769405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f9b180738e0 con 0x7f9b18105470 2024-04-22T23:13:51.279 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.647+0000 7f9b0e7fc700 1 -- 172.21.15.67:0/541769405 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f9b08027030 con 0x7f9b18105470 2024-04-22T23:13:51.279 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.651+0000 7f9b1f78d700 1 -- 172.21.15.67:0/541769405 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7f9b040382d0 msgr2=0x7f9b0403a790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T23:13:51.279 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.651+0000 7f9b1f78d700 1 --2- 172.21.15.67:0/541769405 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7f9b040382d0 0x7f9b0403a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:13:51.279 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.651+0000 7f9b1f78d700 1 -- 172.21.15.67:0/541769405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b18105470 msgr2=0x7f9b18197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:13:51.279 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.651+0000 7f9b1f78d700 1 --2- 172.21.15.67:0/541769405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b18105470 0x7f9b18197fa0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9b08006cb0 tx=0x7f9b0800bd50 comp rx=0 tx=0).stop 2024-04-22T23:13:51.279 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.651+0000 7f9b1f78d700 1 -- 172.21.15.67:0/541769405 shutdown_connections 2024-04-22T23:13:51.280 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.651+0000 7f9b1f78d700 1 --2- 172.21.15.67:0/541769405 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7f9b040382d0 0x7f9b0403a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:13:51.280 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.651+0000 7f9b1f78d700 1 --2- 172.21.15.67:0/541769405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b18105470 0x7f9b18197fa0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:13:51.280 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.651+0000 7f9b1f78d700 1 -- 172.21.15.67:0/541769405 >> 172.21.15.67:0/541769405 conn(0x7f9b181009f0 msgr2=0x7f9b1818ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:13:51.280 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.653+0000 7f9b1f78d700 1 -- 172.21.15.67:0/541769405 shutdown_connections 2024-04-22T23:13:51.280 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:50.653+0000 7f9b1f78d700 1 -- 172.21.15.67:0/541769405 wait complete. 2024-04-22T23:13:51.280 INFO:teuthology.orchestra.run.smithi067.stdout:Waiting for the mgr to restart... 2024-04-22T23:13:51.280 INFO:teuthology.orchestra.run.smithi067.stdout:Waiting for mgr epoch 5... 2024-04-22T23:13:51.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:51 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/541769405' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-22T23:13:56.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:56 smithi067 ceph-mon[26453]: Active manager daemon smithi067.whdcsr restarted 2024-04-22T23:13:56.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:56 smithi067 ceph-mon[26453]: Activating manager daemon smithi067.whdcsr 2024-04-22T23:13:56.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:56 smithi067 ceph-mon[26453]: osdmap e2: 0 total, 0 up, 0 in 2024-04-22T23:13:56.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:56 smithi067 ceph-mon[26453]: mgrmap e6: smithi067.whdcsr(active, starting, since 0.00331414s) 2024-04-22T23:13:56.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:56 smithi067 ceph-mon[26453]: from='mgr.14120 172.21.15.67:0/1431842859' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-22T23:13:56.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:56 smithi067 ceph-mon[26453]: from='mgr.14120 172.21.15.67:0/1431842859' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mgr metadata", "who": "smithi067.whdcsr", "id": "smithi067.whdcsr"}]: dispatch 2024-04-22T23:13:56.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:56 smithi067 ceph-mon[26453]: from='mgr.14120 172.21.15.67:0/1431842859' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T23:13:56.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:56 smithi067 ceph-mon[26453]: from='mgr.14120 172.21.15.67:0/1431842859' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T23:13:56.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:56 smithi067 ceph-mon[26453]: from='mgr.14120 172.21.15.67:0/1431842859' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T23:13:56.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:56 smithi067 ceph-mon[26453]: Manager daemon smithi067.whdcsr is now available 2024-04-22T23:13:56.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:56 smithi067 ceph-mon[26453]: from='mgr.14120 172.21.15.67:0/1431842859' entity='mgr.smithi067.whdcsr' 2024-04-22T23:13:56.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:56 smithi067 ceph-mon[26453]: from='mgr.14120 172.21.15.67:0/1431842859' entity='mgr.smithi067.whdcsr' 2024-04-22T23:13:56.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:56 smithi067 ceph-mon[26453]: from='mgr.14120 172.21.15.67:0/1431842859' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:13:56.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:56 smithi067 ceph-mon[26453]: from='mgr.14120 172.21.15.67:0/1431842859' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:13:56.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:56 smithi067 ceph-mon[26453]: from='mgr.14120 172.21.15.67:0/1431842859' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:13:56.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:56 smithi067 ceph-mon[26453]: from='mgr.14120 172.21.15.67:0/1431842859' entity='mgr.smithi067.whdcsr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.whdcsr/mirror_snapshot_schedule"}]: dispatch 2024-04-22T23:13:56.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:56 smithi067 ceph-mon[26453]: from='mgr.14120 172.21.15.67:0/1431842859' entity='mgr.smithi067.whdcsr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.whdcsr/trash_purge_schedule"}]: dispatch 2024-04-22T23:13:57.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:57 smithi067 ceph-mon[26453]: Found migration_current of "None". Setting to last migration. 2024-04-22T23:13:57.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:57 smithi067 ceph-mon[26453]: from='mgr.14120 172.21.15.67:0/1431842859' entity='mgr.smithi067.whdcsr' 2024-04-22T23:13:57.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:57 smithi067 ceph-mon[26453]: from='mgr.14120 172.21.15.67:0/1431842859' entity='mgr.smithi067.whdcsr' 2024-04-22T23:13:57.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:57 smithi067 ceph-mon[26453]: mgrmap e7: smithi067.whdcsr(active, since 1.00521s) 2024-04-22T23:13:58.101 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout { 2024-04-22T23:13:58.108 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-22T23:13:58.108 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-22T23:13:58.108 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout } 2024-04-22T23:13:58.108 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.698+0000 7f3336bb1700 1 Processor -- start 2024-04-22T23:13:58.109 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.698+0000 7f3336bb1700 1 -- start start 2024-04-22T23:13:58.109 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.698+0000 7f3336bb1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3330106fe0 0x7f33301093d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:13:58.109 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.698+0000 7f3336bb1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3330074680 con 0x7f3330106fe0 2024-04-22T23:13:58.109 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.699+0000 7f333494d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3330106fe0 0x7f33301093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:13:58.109 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.699+0000 7f333494d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3330106fe0 0x7f33301093d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45212/0 (socket says 172.21.15.67:45212) 2024-04-22T23:13:58.109 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.699+0000 7f333494d700 1 -- 172.21.15.67:0/4256764050 learned_addr learned my addr 172.21.15.67:0/4256764050 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:13:58.109 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.699+0000 7f333494d700 1 -- 172.21.15.67:0/4256764050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3330109910 con 0x7f3330106fe0 2024-04-22T23:13:58.109 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.699+0000 7f333494d700 1 --2- 172.21.15.67:0/4256764050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3330106fe0 0x7f33301093d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f33240098a0 tx=0x7f3324009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bdcf4e4ff314d9c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:13:58.109 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.699+0000 7f332f7fe700 1 -- 172.21.15.67:0/4256764050 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3324004180 con 0x7f3330106fe0 2024-04-22T23:13:58.109 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.699+0000 7f332f7fe700 1 -- 172.21.15.67:0/4256764050 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f33240042e0 con 0x7f3330106fe0 2024-04-22T23:13:58.109 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.699+0000 7f332f7fe700 1 -- 172.21.15.67:0/4256764050 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3324011a50 con 0x7f3330106fe0 2024-04-22T23:13:58.109 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.700+0000 7f3336bb1700 1 -- 172.21.15.67:0/4256764050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3330106fe0 msgr2=0x7f33301093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:13:58.109 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.700+0000 7f3336bb1700 1 --2- 172.21.15.67:0/4256764050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3330106fe0 0x7f33301093d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f33240098a0 tx=0x7f3324009bb0 comp rx=0 tx=0).stop 2024-04-22T23:13:58.109 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.700+0000 7f3336bb1700 1 -- 172.21.15.67:0/4256764050 shutdown_connections 2024-04-22T23:13:58.109 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.700+0000 7f3336bb1700 1 --2- 172.21.15.67:0/4256764050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3330106fe0 0x7f33301093d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:13:58.110 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.700+0000 7f3336bb1700 1 -- 172.21.15.67:0/4256764050 >> 172.21.15.67:0/4256764050 conn(0x7f33301009f0 msgr2=0x7f3330102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:13:58.110 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.700+0000 7f3336bb1700 1 -- 172.21.15.67:0/4256764050 shutdown_connections 2024-04-22T23:13:58.110 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.700+0000 7f3336bb1700 1 -- 172.21.15.67:0/4256764050 wait complete. 2024-04-22T23:13:58.110 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.701+0000 7f3336bb1700 1 Processor -- start 2024-04-22T23:13:58.110 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.701+0000 7f3336bb1700 1 -- start start 2024-04-22T23:13:58.110 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.701+0000 7f3336bb1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3330106fe0 0x7f333019a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:13:58.110 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.701+0000 7f3336bb1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f333019a670 con 0x7f3330106fe0 2024-04-22T23:13:58.110 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.701+0000 7f333494d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3330106fe0 0x7f333019a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:13:58.110 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.701+0000 7f333494d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3330106fe0 0x7f333019a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45224/0 (socket says 172.21.15.67:45224) 2024-04-22T23:13:58.110 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.701+0000 7f333494d700 1 -- 172.21.15.67:0/2570164240 learned_addr learned my addr 172.21.15.67:0/2570164240 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:13:58.110 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.701+0000 7f333494d700 1 -- 172.21.15.67:0/2570164240 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3324009550 con 0x7f3330106fe0 2024-04-22T23:13:58.110 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.702+0000 7f333494d700 1 --2- 172.21.15.67:0/2570164240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3330106fe0 0x7f333019a130 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f33240119e0 tx=0x7f3324011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:13:58.110 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.702+0000 7f332dffb700 1 -- 172.21.15.67:0/2570164240 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3324003b90 con 0x7f3330106fe0 2024-04-22T23:13:58.111 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.702+0000 7f3336bb1700 1 -- 172.21.15.67:0/2570164240 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f333019a870 con 0x7f3330106fe0 2024-04-22T23:13:58.111 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.702+0000 7f332dffb700 1 -- 172.21.15.67:0/2570164240 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f332401a8a0 con 0x7f3330106fe0 2024-04-22T23:13:58.111 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.702+0000 7f332dffb700 1 -- 172.21.15.67:0/2570164240 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3324019930 con 0x7f3330106fe0 2024-04-22T23:13:58.111 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.702+0000 7f3336bb1700 1 -- 172.21.15.67:0/2570164240 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f333019ad10 con 0x7f3330106fe0 2024-04-22T23:13:58.111 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.703+0000 7f332dffb700 1 -- 172.21.15.67:0/2570164240 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f3324019bf0 con 0x7f3330106fe0 2024-04-22T23:13:58.111 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.703+0000 7f332dffb700 1 --2- 172.21.15.67:0/2570164240 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7f33200382e0 0x7f332003a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:13:58.111 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.703+0000 7f332dffb700 1 -- 172.21.15.67:0/2570164240 --> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f332003aeb0 con 0x7f33200382e0 2024-04-22T23:13:58.111 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.703+0000 7f332dffb700 1 -- 172.21.15.67:0/2570164240 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f332404c6e0 con 0x7f3330106fe0 2024-04-22T23:13:58.111 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.703+0000 7f332ffff700 1 -- 172.21.15.67:0/2570164240 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7f33200382e0 msgr2=0x7f332003a7a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/2451437776 2024-04-22T23:13:58.111 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.703+0000 7f332ffff700 1 --2- 172.21.15.67:0/2570164240 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7f33200382e0 0x7f332003a7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T23:13:58.111 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.904+0000 7f332ffff700 1 -- 172.21.15.67:0/2570164240 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7f33200382e0 msgr2=0x7f332003a7a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/2451437776 2024-04-22T23:13:58.111 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:51.904+0000 7f332ffff700 1 --2- 172.21.15.67:0/2570164240 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7f33200382e0 0x7f332003a7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T23:13:58.111 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:52.305+0000 7f332ffff700 1 -- 172.21.15.67:0/2570164240 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7f33200382e0 msgr2=0x7f332003a7a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/2451437776 2024-04-22T23:13:58.111 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:52.305+0000 7f332ffff700 1 --2- 172.21.15.67:0/2570164240 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7f33200382e0 0x7f332003a7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T23:13:58.111 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:53.106+0000 7f332ffff700 1 -- 172.21.15.67:0/2570164240 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7f33200382e0 msgr2=0x7f332003a7a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/2451437776 2024-04-22T23:13:58.112 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:53.106+0000 7f332ffff700 1 --2- 172.21.15.67:0/2570164240 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7f33200382e0 0x7f332003a7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T23:13:58.112 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:54.707+0000 7f332ffff700 1 -- 172.21.15.67:0/2570164240 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7f33200382e0 msgr2=0x7f332003a7a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/2451437776 2024-04-22T23:13:58.112 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:54.707+0000 7f332ffff700 1 --2- 172.21.15.67:0/2570164240 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7f33200382e0 0x7f332003a7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-22T23:13:58.112 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:56.303+0000 7f332dffb700 1 -- 172.21.15.67:0/2570164240 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mgrmap(e 6) v1 ==== 45050+0+0 (secure 0 0 0) 0x7f332402d620 con 0x7f3330106fe0 2024-04-22T23:13:58.112 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:56.303+0000 7f332dffb700 1 -- 172.21.15.67:0/2570164240 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7f33200382e0 msgr2=0x7f332003a7a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T23:13:58.112 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:56.303+0000 7f332dffb700 1 --2- 172.21.15.67:0/2570164240 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7f33200382e0 0x7f332003a7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:13:58.112 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:57.304+0000 7f332dffb700 1 -- 172.21.15.67:0/2570164240 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 7) v1 ==== 45177+0+0 (secure 0 0 0) 0x7f332401ab60 con 0x7f3330106fe0 2024-04-22T23:13:58.112 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:57.304+0000 7f332dffb700 1 --2- 172.21.15.67:0/2570164240 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f332003bc60 0x7f332003e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:13:58.112 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:57.304+0000 7f332dffb700 1 -- 172.21.15.67:0/2570164240 --> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f332003aeb0 con 0x7f332003bc60 2024-04-22T23:13:58.112 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:57.304+0000 7f332ffff700 1 --2- 172.21.15.67:0/2570164240 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f332003bc60 0x7f332003e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:13:58.112 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:57.305+0000 7f332ffff700 1 --2- 172.21.15.67:0/2570164240 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f332003bc60 0x7f332003e050 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f331c003a10 tx=0x7f331c009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:13:58.112 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:57.306+0000 7f332dffb700 1 -- 172.21.15.67:0/2570164240 <== mgr.14120 v2:172.21.15.67:6800/3568049827 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f332003aeb0 con 0x7f332003bc60 2024-04-22T23:13:58.112 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:57.312+0000 7f3336bb1700 1 -- 172.21.15.67:0/2570164240 --> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f333019b470 con 0x7f332003bc60 2024-04-22T23:13:58.112 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:57.312+0000 7f332dffb700 1 -- 172.21.15.67:0/2570164240 <== mgr.14120 v2:172.21.15.67:6800/3568049827 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f333019b470 con 0x7f332003bc60 2024-04-22T23:13:58.112 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:57.313+0000 7f3336bb1700 1 -- 172.21.15.67:0/2570164240 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f332003bc60 msgr2=0x7f332003e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:13:58.113 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:57.313+0000 7f3336bb1700 1 --2- 172.21.15.67:0/2570164240 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f332003bc60 0x7f332003e050 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f331c003a10 tx=0x7f331c009290 comp rx=0 tx=0).stop 2024-04-22T23:13:58.113 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:57.313+0000 7f3336bb1700 1 -- 172.21.15.67:0/2570164240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3330106fe0 msgr2=0x7f333019a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:13:58.113 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:57.313+0000 7f3336bb1700 1 --2- 172.21.15.67:0/2570164240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3330106fe0 0x7f333019a130 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f33240119e0 tx=0x7f3324011ac0 comp rx=0 tx=0).stop 2024-04-22T23:13:58.113 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:57.313+0000 7f3336bb1700 1 -- 172.21.15.67:0/2570164240 shutdown_connections 2024-04-22T23:13:58.113 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:57.313+0000 7f3336bb1700 1 --2- 172.21.15.67:0/2570164240 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f332003bc60 0x7f332003e050 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:13:58.113 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:57.313+0000 7f3336bb1700 1 --2- 172.21.15.67:0/2570164240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3330106fe0 0x7f333019a130 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:13:58.113 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:57.313+0000 7f3336bb1700 1 --2- 172.21.15.67:0/2570164240 >> [v2:172.21.15.67:6800/2451437776,v1:172.21.15.67:6801/2451437776] conn(0x7f33200382e0 0x7f332003a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:13:58.113 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:57.313+0000 7f3336bb1700 1 -- 172.21.15.67:0/2570164240 >> 172.21.15.67:0/2570164240 conn(0x7f33301009f0 msgr2=0x7f3330101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:13:58.113 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:57.313+0000 7f3336bb1700 1 -- 172.21.15.67:0/2570164240 shutdown_connections 2024-04-22T23:13:58.113 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:57.313+0000 7f3336bb1700 1 -- 172.21.15.67:0/2570164240 wait complete. 2024-04-22T23:13:58.113 INFO:teuthology.orchestra.run.smithi067.stdout:mgr epoch 5 is available 2024-04-22T23:13:58.113 INFO:teuthology.orchestra.run.smithi067.stdout:Setting orchestrator backend to cephadm... 2024-04-22T23:13:58.484 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:58 smithi067 ceph-mon[26453]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-22T23:13:58.484 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:58 smithi067 ceph-mon[26453]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-22T23:13:58.484 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:58 smithi067 ceph-mon[26453]: from='mgr.14120 172.21.15.67:0/1431842859' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:13:59.270 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.542+0000 7fac50655700 1 Processor -- start 2024-04-22T23:13:59.270 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.542+0000 7fac50655700 1 -- start start 2024-04-22T23:13:59.270 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.543+0000 7fac50655700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac48105460 0x7fac48105880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:13:59.270 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.543+0000 7fac50655700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac48105e50 con 0x7fac48105460 2024-04-22T23:13:59.270 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.543+0000 7fac4e3f1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac48105460 0x7fac48105880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:13:59.270 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.543+0000 7fac4e3f1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac48105460 0x7fac48105880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45292/0 (socket says 172.21.15.67:45292) 2024-04-22T23:13:59.270 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.543+0000 7fac4e3f1700 1 -- 172.21.15.67:0/3771092993 learned_addr learned my addr 172.21.15.67:0/3771092993 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:13:59.270 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.543+0000 7fac4e3f1700 1 -- 172.21.15.67:0/3771092993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac48106660 con 0x7fac48105460 2024-04-22T23:13:59.271 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.543+0000 7fac4e3f1700 1 --2- 172.21.15.67:0/3771092993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac48105460 0x7fac48105880 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fac3800d180 tx=0x7fac3800d490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a9afc3dffe20775 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:13:59.271 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.543+0000 7fac4d3ef700 1 -- 172.21.15.67:0/3771092993 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac38010070 con 0x7fac48105460 2024-04-22T23:13:59.271 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.543+0000 7fac4d3ef700 1 -- 172.21.15.67:0/3771092993 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fac3800dcf0 con 0x7fac48105460 2024-04-22T23:13:59.271 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.543+0000 7fac4d3ef700 1 -- 172.21.15.67:0/3771092993 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac38009950 con 0x7fac48105460 2024-04-22T23:13:59.271 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.544+0000 7fac50655700 1 -- 172.21.15.67:0/3771092993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac48105460 msgr2=0x7fac48105880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:13:59.271 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.544+0000 7fac50655700 1 --2- 172.21.15.67:0/3771092993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac48105460 0x7fac48105880 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fac3800d180 tx=0x7fac3800d490 comp rx=0 tx=0).stop 2024-04-22T23:13:59.271 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.544+0000 7fac50655700 1 -- 172.21.15.67:0/3771092993 shutdown_connections 2024-04-22T23:13:59.271 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.544+0000 7fac50655700 1 --2- 172.21.15.67:0/3771092993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac48105460 0x7fac48105880 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:13:59.271 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.544+0000 7fac50655700 1 -- 172.21.15.67:0/3771092993 >> 172.21.15.67:0/3771092993 conn(0x7fac48100a20 msgr2=0x7fac48102e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:13:59.271 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.544+0000 7fac50655700 1 -- 172.21.15.67:0/3771092993 shutdown_connections 2024-04-22T23:13:59.271 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.544+0000 7fac50655700 1 -- 172.21.15.67:0/3771092993 wait complete. 2024-04-22T23:13:59.271 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.545+0000 7fac50655700 1 Processor -- start 2024-04-22T23:13:59.271 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.545+0000 7fac50655700 1 -- start start 2024-04-22T23:13:59.271 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.545+0000 7fac50655700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac48105460 0x7fac4819c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:13:59.272 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.545+0000 7fac50655700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac4819c900 con 0x7fac48105460 2024-04-22T23:13:59.272 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.545+0000 7fac4e3f1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac48105460 0x7fac4819c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:13:59.272 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.545+0000 7fac4e3f1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac48105460 0x7fac4819c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45294/0 (socket says 172.21.15.67:45294) 2024-04-22T23:13:59.272 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.545+0000 7fac4e3f1700 1 -- 172.21.15.67:0/1533217283 learned_addr learned my addr 172.21.15.67:0/1533217283 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:13:59.272 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.545+0000 7fac4e3f1700 1 -- 172.21.15.67:0/1533217283 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac3800bd40 con 0x7fac48105460 2024-04-22T23:13:59.272 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.546+0000 7fac4e3f1700 1 --2- 172.21.15.67:0/1533217283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac48105460 0x7fac4819c3c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fac3800d910 tx=0x7fac3800b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:13:59.272 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.546+0000 7fac477fe700 1 -- 172.21.15.67:0/1533217283 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac38010050 con 0x7fac48105460 2024-04-22T23:13:59.272 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.546+0000 7fac50655700 1 -- 172.21.15.67:0/1533217283 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac4819cb00 con 0x7fac48105460 2024-04-22T23:13:59.272 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.546+0000 7fac477fe700 1 -- 172.21.15.67:0/1533217283 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fac380045c0 con 0x7fac48105460 2024-04-22T23:13:59.272 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.546+0000 7fac477fe700 1 -- 172.21.15.67:0/1533217283 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac3801fa10 con 0x7fac48105460 2024-04-22T23:13:59.272 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.546+0000 7fac50655700 1 -- 172.21.15.67:0/1533217283 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac4819d000 con 0x7fac48105460 2024-04-22T23:13:59.272 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.547+0000 7fac477fe700 1 -- 172.21.15.67:0/1533217283 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fac38029440 con 0x7fac48105460 2024-04-22T23:13:59.272 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.547+0000 7fac477fe700 1 --2- 172.21.15.67:0/1533217283 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fac300382d0 0x7fac3003a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:13:59.273 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.547+0000 7fac50655700 1 -- 172.21.15.67:0/1533217283 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac48191cf0 con 0x7fac48105460 2024-04-22T23:13:59.273 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.547+0000 7fac477fe700 1 -- 172.21.15.67:0/1533217283 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fac3804c750 con 0x7fac48105460 2024-04-22T23:13:59.273 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.547+0000 7fac4dbf0700 1 --2- 172.21.15.67:0/1533217283 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fac300382d0 0x7fac3003a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:13:59.273 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.547+0000 7fac4dbf0700 1 --2- 172.21.15.67:0/1533217283 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fac300382d0 0x7fac3003a790 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fac3c006fd0 tx=0x7fac3c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:13:59.273 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.552+0000 7fac477fe700 1 -- 172.21.15.67:0/1533217283 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fac38049e60 con 0x7fac48105460 2024-04-22T23:13:59.273 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.767+0000 7fac50655700 1 -- 172.21.15.67:0/1533217283 --> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fac480611d0 con 0x7fac300382d0 2024-04-22T23:13:59.273 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.776+0000 7fac477fe700 1 -- 172.21.15.67:0/1533217283 <== mgr.14120 v2:172.21.15.67:6800/3568049827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fac480611d0 con 0x7fac300382d0 2024-04-22T23:13:59.273 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.778+0000 7fac50655700 1 -- 172.21.15.67:0/1533217283 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fac300382d0 msgr2=0x7fac3003a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:13:59.273 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.778+0000 7fac50655700 1 --2- 172.21.15.67:0/1533217283 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fac300382d0 0x7fac3003a790 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fac3c006fd0 tx=0x7fac3c006e40 comp rx=0 tx=0).stop 2024-04-22T23:13:59.273 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.778+0000 7fac50655700 1 -- 172.21.15.67:0/1533217283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac48105460 msgr2=0x7fac4819c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:13:59.273 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.778+0000 7fac50655700 1 --2- 172.21.15.67:0/1533217283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac48105460 0x7fac4819c3c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fac3800d910 tx=0x7fac3800b3b0 comp rx=0 tx=0).stop 2024-04-22T23:13:59.273 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.778+0000 7fac50655700 1 -- 172.21.15.67:0/1533217283 shutdown_connections 2024-04-22T23:13:59.273 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.778+0000 7fac50655700 1 --2- 172.21.15.67:0/1533217283 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fac300382d0 0x7fac3003a790 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:13:59.274 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.778+0000 7fac50655700 1 --2- 172.21.15.67:0/1533217283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac48105460 0x7fac4819c3c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:13:59.274 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.778+0000 7fac50655700 1 -- 172.21.15.67:0/1533217283 >> 172.21.15.67:0/1533217283 conn(0x7fac48100a20 msgr2=0x7fac4818ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:13:59.274 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.778+0000 7fac50655700 1 -- 172.21.15.67:0/1533217283 shutdown_connections 2024-04-22T23:13:59.274 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:58.778+0000 7fac50655700 1 -- 172.21.15.67:0/1533217283 wait complete. 2024-04-22T23:13:59.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:59 smithi067 ceph-mon[26453]: [22/Apr/2024:23:13:57] ENGINE Bus STARTING 2024-04-22T23:13:59.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:59 smithi067 ceph-mon[26453]: [22/Apr/2024:23:13:57] ENGINE Serving on http://172.21.15.67:8765 2024-04-22T23:13:59.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:59 smithi067 ceph-mon[26453]: [22/Apr/2024:23:13:57] ENGINE Serving on https://172.21.15.67:7150 2024-04-22T23:13:59.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:59 smithi067 ceph-mon[26453]: [22/Apr/2024:23:13:57] ENGINE Bus STARTED 2024-04-22T23:13:59.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:59 smithi067 ceph-mon[26453]: mgrmap e8: smithi067.whdcsr(active, since 2s) 2024-04-22T23:13:59.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:59 smithi067 ceph-mon[26453]: from='mgr.14120 172.21.15.67:0/1431842859' entity='mgr.smithi067.whdcsr' 2024-04-22T23:13:59.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:13:59 smithi067 ceph-mon[26453]: from='mgr.14120 172.21.15.67:0/1431842859' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:14:00.477 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:00 smithi067 ceph-mon[26453]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:14:00.485 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-22T23:14:00.486 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.708+0000 7fee93d5b700 1 Processor -- start 2024-04-22T23:14:00.486 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.708+0000 7fee93d5b700 1 -- start start 2024-04-22T23:14:00.486 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.708+0000 7fee93d5b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee8c106560 0x7fee8c106980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:00.486 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.708+0000 7fee93d5b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8c106f50 con 0x7fee8c106560 2024-04-22T23:14:00.486 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.708+0000 7fee91af7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee8c106560 0x7fee8c106980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:00.486 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.708+0000 7fee91af7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee8c106560 0x7fee8c106980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49406/0 (socket says 172.21.15.67:49406) 2024-04-22T23:14:00.486 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.708+0000 7fee91af7700 1 -- 172.21.15.67:0/4257221312 learned_addr learned my addr 172.21.15.67:0/4257221312 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:00.486 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.709+0000 7fee91af7700 1 -- 172.21.15.67:0/4257221312 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee8c107760 con 0x7fee8c106560 2024-04-22T23:14:00.486 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.709+0000 7fee91af7700 1 --2- 172.21.15.67:0/4257221312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee8c106560 0x7fee8c106980 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fee84009e80 tx=0x7fee8400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b109067acaf016b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:00.486 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.709+0000 7fee90af5700 1 -- 172.21.15.67:0/4257221312 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee8400e070 con 0x7fee8c106560 2024-04-22T23:14:00.486 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.709+0000 7fee90af5700 1 -- 172.21.15.67:0/4257221312 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fee84003e90 con 0x7fee8c106560 2024-04-22T23:14:00.486 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.709+0000 7fee90af5700 1 -- 172.21.15.67:0/4257221312 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee84004380 con 0x7fee8c106560 2024-04-22T23:14:00.487 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.710+0000 7fee93d5b700 1 -- 172.21.15.67:0/4257221312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee8c106560 msgr2=0x7fee8c106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:00.487 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.710+0000 7fee93d5b700 1 --2- 172.21.15.67:0/4257221312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee8c106560 0x7fee8c106980 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fee84009e80 tx=0x7fee8400b330 comp rx=0 tx=0).stop 2024-04-22T23:14:00.487 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.710+0000 7fee93d5b700 1 -- 172.21.15.67:0/4257221312 shutdown_connections 2024-04-22T23:14:00.487 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.710+0000 7fee93d5b700 1 --2- 172.21.15.67:0/4257221312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee8c106560 0x7fee8c106980 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:00.487 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.710+0000 7fee93d5b700 1 -- 172.21.15.67:0/4257221312 >> 172.21.15.67:0/4257221312 conn(0x7fee8c101b40 msgr2=0x7fee8c103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:00.487 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.710+0000 7fee93d5b700 1 -- 172.21.15.67:0/4257221312 shutdown_connections 2024-04-22T23:14:00.487 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.710+0000 7fee93d5b700 1 -- 172.21.15.67:0/4257221312 wait complete. 2024-04-22T23:14:00.487 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.710+0000 7fee93d5b700 1 Processor -- start 2024-04-22T23:14:00.487 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.710+0000 7fee93d5b700 1 -- start start 2024-04-22T23:14:00.487 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.710+0000 7fee93d5b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee8c106560 0x7fee8c19a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:00.487 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.710+0000 7fee93d5b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8c19a710 con 0x7fee8c106560 2024-04-22T23:14:00.487 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.711+0000 7fee91af7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee8c106560 0x7fee8c19a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:00.487 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.711+0000 7fee91af7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee8c106560 0x7fee8c19a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49420/0 (socket says 172.21.15.67:49420) 2024-04-22T23:14:00.488 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.711+0000 7fee91af7700 1 -- 172.21.15.67:0/138468676 learned_addr learned my addr 172.21.15.67:0/138468676 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:00.488 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.711+0000 7fee91af7700 1 -- 172.21.15.67:0/138468676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee84009b30 con 0x7fee8c106560 2024-04-22T23:14:00.488 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.711+0000 7fee91af7700 1 --2- 172.21.15.67:0/138468676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee8c106560 0x7fee8c19a1d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fee8400d010 tx=0x7fee84004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:00.488 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.711+0000 7fee82ffd700 1 -- 172.21.15.67:0/138468676 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee8400e040 con 0x7fee8c106560 2024-04-22T23:14:00.488 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.711+0000 7fee82ffd700 1 -- 172.21.15.67:0/138468676 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fee84003c30 con 0x7fee8c106560 2024-04-22T23:14:00.488 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.711+0000 7fee93d5b700 1 -- 172.21.15.67:0/138468676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee8c19a910 con 0x7fee8c106560 2024-04-22T23:14:00.488 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.711+0000 7fee82ffd700 1 -- 172.21.15.67:0/138468676 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee8401b790 con 0x7fee8c106560 2024-04-22T23:14:00.488 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.711+0000 7fee93d5b700 1 -- 172.21.15.67:0/138468676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee8c19ae10 con 0x7fee8c106560 2024-04-22T23:14:00.488 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.712+0000 7fee82ffd700 1 -- 172.21.15.67:0/138468676 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fee84019040 con 0x7fee8c106560 2024-04-22T23:14:00.488 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.712+0000 7fee93d5b700 1 -- 172.21.15.67:0/138468676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee8c0623c0 con 0x7fee8c106560 2024-04-22T23:14:00.488 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.712+0000 7fee82ffd700 1 --2- 172.21.15.67:0/138468676 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fee780382e0 0x7fee7803a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:00.488 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.712+0000 7fee82ffd700 1 -- 172.21.15.67:0/138468676 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fee8404be30 con 0x7fee8c106560 2024-04-22T23:14:00.488 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.712+0000 7fee912f6700 1 --2- 172.21.15.67:0/138468676 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fee780382e0 0x7fee7803a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:00.489 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.713+0000 7fee912f6700 1 --2- 172.21.15.67:0/138468676 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fee780382e0 0x7fee7803a7a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fee7c006fd0 tx=0x7fee7c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:00.489 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.717+0000 7fee82ffd700 1 -- 172.21.15.67:0/138468676 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fee84012430 con 0x7fee8c106560 2024-04-22T23:14:00.489 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.919+0000 7fee93d5b700 1 -- 172.21.15.67:0/138468676 --> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fee8c102f90 con 0x7fee780382e0 2024-04-22T23:14:00.489 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.919+0000 7fee82ffd700 1 -- 172.21.15.67:0/138468676 <== mgr.14120 v2:172.21.15.67:6800/3568049827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fee8c102f90 con 0x7fee780382e0 2024-04-22T23:14:00.489 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.921+0000 7fee93d5b700 1 -- 172.21.15.67:0/138468676 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fee780382e0 msgr2=0x7fee7803a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:00.489 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.921+0000 7fee93d5b700 1 --2- 172.21.15.67:0/138468676 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fee780382e0 0x7fee7803a7a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fee7c006fd0 tx=0x7fee7c006e40 comp rx=0 tx=0).stop 2024-04-22T23:14:00.489 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.921+0000 7fee93d5b700 1 -- 172.21.15.67:0/138468676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee8c106560 msgr2=0x7fee8c19a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:00.489 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.921+0000 7fee93d5b700 1 --2- 172.21.15.67:0/138468676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee8c106560 0x7fee8c19a1d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fee8400d010 tx=0x7fee84004430 comp rx=0 tx=0).stop 2024-04-22T23:14:00.489 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.921+0000 7fee93d5b700 1 -- 172.21.15.67:0/138468676 shutdown_connections 2024-04-22T23:14:00.489 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.921+0000 7fee93d5b700 1 --2- 172.21.15.67:0/138468676 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fee780382e0 0x7fee7803a7a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:00.489 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.921+0000 7fee93d5b700 1 --2- 172.21.15.67:0/138468676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee8c106560 0x7fee8c19a1d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:00.489 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.921+0000 7fee93d5b700 1 -- 172.21.15.67:0/138468676 >> 172.21.15.67:0/138468676 conn(0x7fee8c101b40 msgr2=0x7fee8c102880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:00.489 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.922+0000 7fee93d5b700 1 -- 172.21.15.67:0/138468676 shutdown_connections 2024-04-22T23:14:00.489 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:13:59.922+0000 7fee93d5b700 1 -- 172.21.15.67:0/138468676 wait complete. 2024-04-22T23:14:00.489 INFO:teuthology.orchestra.run.smithi067.stdout:Generating ssh key... 2024-04-22T23:14:01.493 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:01 smithi067 ceph-mon[26453]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:14:02.049 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.954+0000 7fab3baf3700 1 Processor -- start 2024-04-22T23:14:02.050 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.954+0000 7fab3baf3700 1 -- start start 2024-04-22T23:14:02.050 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.954+0000 7fab3baf3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab34105460 0x7fab34105880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:02.050 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.954+0000 7fab3baf3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab34105e50 con 0x7fab34105460 2024-04-22T23:14:02.050 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.954+0000 7fab3988f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab34105460 0x7fab34105880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:02.050 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.954+0000 7fab3988f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab34105460 0x7fab34105880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49428/0 (socket says 172.21.15.67:49428) 2024-04-22T23:14:02.050 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.954+0000 7fab3988f700 1 -- 172.21.15.67:0/1166913577 learned_addr learned my addr 172.21.15.67:0/1166913577 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:02.050 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.955+0000 7fab3988f700 1 -- 172.21.15.67:0/1166913577 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab34106660 con 0x7fab34105460 2024-04-22T23:14:02.050 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.955+0000 7fab3988f700 1 --2- 172.21.15.67:0/1166913577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab34105460 0x7fab34105880 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fab300098a0 tx=0x7fab30009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30a5722e3941407e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:02.050 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.955+0000 7fab3888d700 1 -- 172.21.15.67:0/1166913577 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab30004180 con 0x7fab34105460 2024-04-22T23:14:02.050 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.955+0000 7fab3888d700 1 -- 172.21.15.67:0/1166913577 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fab300042e0 con 0x7fab34105460 2024-04-22T23:14:02.050 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.955+0000 7fab3888d700 1 -- 172.21.15.67:0/1166913577 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab30011a50 con 0x7fab34105460 2024-04-22T23:14:02.050 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.956+0000 7fab3baf3700 1 -- 172.21.15.67:0/1166913577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab34105460 msgr2=0x7fab34105880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:02.050 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.956+0000 7fab3baf3700 1 --2- 172.21.15.67:0/1166913577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab34105460 0x7fab34105880 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fab300098a0 tx=0x7fab30009bb0 comp rx=0 tx=0).stop 2024-04-22T23:14:02.051 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.956+0000 7fab3baf3700 1 -- 172.21.15.67:0/1166913577 shutdown_connections 2024-04-22T23:14:02.051 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.956+0000 7fab3baf3700 1 --2- 172.21.15.67:0/1166913577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab34105460 0x7fab34105880 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:02.051 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.956+0000 7fab3baf3700 1 -- 172.21.15.67:0/1166913577 >> 172.21.15.67:0/1166913577 conn(0x7fab34100a20 msgr2=0x7fab34102e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:02.051 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.956+0000 7fab3baf3700 1 -- 172.21.15.67:0/1166913577 shutdown_connections 2024-04-22T23:14:02.051 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.956+0000 7fab3baf3700 1 -- 172.21.15.67:0/1166913577 wait complete. 2024-04-22T23:14:02.051 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.956+0000 7fab3baf3700 1 Processor -- start 2024-04-22T23:14:02.051 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.956+0000 7fab3baf3700 1 -- start start 2024-04-22T23:14:02.051 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.957+0000 7fab3baf3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab34105460 0x7fab34190630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:02.051 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.957+0000 7fab3baf3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab34190b70 con 0x7fab34105460 2024-04-22T23:14:02.051 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.957+0000 7fab3988f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab34105460 0x7fab34190630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:02.051 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.957+0000 7fab3988f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab34105460 0x7fab34190630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49436/0 (socket says 172.21.15.67:49436) 2024-04-22T23:14:02.051 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.957+0000 7fab3988f700 1 -- 172.21.15.67:0/3221170773 learned_addr learned my addr 172.21.15.67:0/3221170773 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:02.051 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.957+0000 7fab3988f700 1 -- 172.21.15.67:0/3221170773 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab30009550 con 0x7fab34105460 2024-04-22T23:14:02.051 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.957+0000 7fab3988f700 1 --2- 172.21.15.67:0/3221170773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab34105460 0x7fab34190630 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fab30003de0 tx=0x7fab30011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:02.051 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.957+0000 7fab2affd700 1 -- 172.21.15.67:0/3221170773 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab30003930 con 0x7fab34105460 2024-04-22T23:14:02.052 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.957+0000 7fab2affd700 1 -- 172.21.15.67:0/3221170773 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fab30003a90 con 0x7fab34105460 2024-04-22T23:14:02.052 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.957+0000 7fab3baf3700 1 -- 172.21.15.67:0/3221170773 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab3418ec80 con 0x7fab34105460 2024-04-22T23:14:02.052 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.957+0000 7fab2affd700 1 -- 172.21.15.67:0/3221170773 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab300196c0 con 0x7fab34105460 2024-04-22T23:14:02.052 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.958+0000 7fab3baf3700 1 -- 172.21.15.67:0/3221170773 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab3418f120 con 0x7fab34105460 2024-04-22T23:14:02.052 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.958+0000 7fab2affd700 1 -- 172.21.15.67:0/3221170773 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fab3001a430 con 0x7fab34105460 2024-04-22T23:14:02.052 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.958+0000 7fab3baf3700 1 -- 172.21.15.67:0/3221170773 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab340623c0 con 0x7fab34105460 2024-04-22T23:14:02.052 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.958+0000 7fab2affd700 1 --2- 172.21.15.67:0/3221170773 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fab20038320 0x7fab2003a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:02.052 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.959+0000 7fab2affd700 1 -- 172.21.15.67:0/3221170773 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fab3004c3b0 con 0x7fab34105460 2024-04-22T23:14:02.052 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.959+0000 7fab3908e700 1 --2- 172.21.15.67:0/3221170773 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fab20038320 0x7fab2003a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:02.052 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.959+0000 7fab3908e700 1 --2- 172.21.15.67:0/3221170773 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fab20038320 0x7fab2003a7e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fab24006fd0 tx=0x7fab24006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:02.052 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:00.963+0000 7fab2affd700 1 -- 172.21.15.67:0/3221170773 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fab30010970 con 0x7fab34105460 2024-04-22T23:14:02.052 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:01.165+0000 7fab3baf3700 1 -- 172.21.15.67:0/3221170773 --> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fab34102bb0 con 0x7fab20038320 2024-04-22T23:14:02.053 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:01.503+0000 7fab2affd700 1 -- 172.21.15.67:0/3221170773 <== mgr.14120 v2:172.21.15.67:6800/3568049827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fab34102bb0 con 0x7fab20038320 2024-04-22T23:14:02.053 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:01.505+0000 7fab3baf3700 1 -- 172.21.15.67:0/3221170773 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fab20038320 msgr2=0x7fab2003a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:02.053 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:01.505+0000 7fab3baf3700 1 --2- 172.21.15.67:0/3221170773 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fab20038320 0x7fab2003a7e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fab24006fd0 tx=0x7fab24006e40 comp rx=0 tx=0).stop 2024-04-22T23:14:02.053 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:01.505+0000 7fab3baf3700 1 -- 172.21.15.67:0/3221170773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab34105460 msgr2=0x7fab34190630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:02.053 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:01.505+0000 7fab3baf3700 1 --2- 172.21.15.67:0/3221170773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab34105460 0x7fab34190630 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fab30003de0 tx=0x7fab30011ab0 comp rx=0 tx=0).stop 2024-04-22T23:14:02.053 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:01.505+0000 7fab3baf3700 1 -- 172.21.15.67:0/3221170773 shutdown_connections 2024-04-22T23:14:02.053 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:01.505+0000 7fab3baf3700 1 --2- 172.21.15.67:0/3221170773 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fab20038320 0x7fab2003a7e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:02.053 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:01.505+0000 7fab3baf3700 1 --2- 172.21.15.67:0/3221170773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab34105460 0x7fab34190630 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:02.053 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:01.505+0000 7fab3baf3700 1 -- 172.21.15.67:0/3221170773 >> 172.21.15.67:0/3221170773 conn(0x7fab34100a20 msgr2=0x7fab34079900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:02.053 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:01.505+0000 7fab3baf3700 1 -- 172.21.15.67:0/3221170773 shutdown_connections 2024-04-22T23:14:02.053 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:01.505+0000 7fab3baf3700 1 -- 172.21.15.67:0/3221170773 wait complete. 2024-04-22T23:14:02.476 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:02 smithi067 ceph-mon[26453]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:14:02.477 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:02 smithi067 ceph-mon[26453]: Generating ssh key... 2024-04-22T23:14:02.477 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:02 smithi067 ceph-mon[26453]: from='mgr.14120 172.21.15.67:0/1431842859' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:02.477 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:02 smithi067 ceph-mon[26453]: from='mgr.14120 172.21.15.67:0/1431842859' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:03.296 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwSNMj7/MPduD49iqXpH+MkRm1oc8wB9SeMsBzzaC4r0zh6bNpyWPmIIxpWNQ8fMhKWLKM0D3J8i3pAbJvdhTx38gvjl6FOGkKpr1DrnnbLdcBsp4kbOSQpoH/q9rVkOyggQpKOfAGanOjqP3dWUJNJjR8RkPT16b4itFLVjDZDLS5tBuAPu1DtCEs0g5xqLFpqf0Zfry6Krug7LsVWc5FnKi5Slo645rk9EMsp8Lplp+D4s6faMnjpSWuo7tngJUuYDm3vRrdvCqJLATWmVFJpnIfTerYAKgT88XZf0lU7HywvcLWVc/7K2PDnp2V/6xg0bPOMIWtYZKM0BV9y7CFVr1vgQenovZl+J8NxWHwx9VovWruiOEViYY5SmBwFLb8fkHbLlzdIg7DKib4MZtAMXlvt3ZRUZB0eAkr/reGVugBrD0w9tUD37Ch/uqEfSiegV9r1kCU3gQRgWb27yuoun8roytesWJf/63fPPRvpZtm2GaVM6wsPC+ZOvxn9nc= ceph-d46c346e-00fd-11ef-bc93-c7b262605968 2024-04-22T23:14:03.297 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.505+0000 7f62ee0cd700 1 Processor -- start 2024-04-22T23:14:03.297 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.505+0000 7f62ee0cd700 1 -- start start 2024-04-22T23:14:03.297 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.505+0000 7f62ee0cd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62e807be80 0x7f62e807a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:03.297 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.505+0000 7f62ee0cd700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e807a8b0 con 0x7f62e807be80 2024-04-22T23:14:03.297 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.505+0000 7f62e77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62e807be80 0x7f62e807a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:03.297 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.505+0000 7f62e77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62e807be80 0x7f62e807a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49440/0 (socket says 172.21.15.67:49440) 2024-04-22T23:14:03.297 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.505+0000 7f62e77fe700 1 -- 172.21.15.67:0/2769380151 learned_addr learned my addr 172.21.15.67:0/2769380151 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:03.297 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.506+0000 7f62e77fe700 1 -- 172.21.15.67:0/2769380151 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e807a9f0 con 0x7f62e807be80 2024-04-22T23:14:03.297 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.506+0000 7f62e77fe700 1 --2- 172.21.15.67:0/2769380151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62e807be80 0x7f62e807a2e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f62d80098a0 tx=0x7f62d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11a6503cb5227abe server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:03.297 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.506+0000 7f62e67fc700 1 -- 172.21.15.67:0/2769380151 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62d8004180 con 0x7f62e807be80 2024-04-22T23:14:03.297 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.506+0000 7f62e67fc700 1 -- 172.21.15.67:0/2769380151 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f62d80042e0 con 0x7f62e807be80 2024-04-22T23:14:03.298 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.506+0000 7f62e67fc700 1 -- 172.21.15.67:0/2769380151 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62d8011a50 con 0x7f62e807be80 2024-04-22T23:14:03.298 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.507+0000 7f62ee0cd700 1 -- 172.21.15.67:0/2769380151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62e807be80 msgr2=0x7f62e807a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:03.298 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.507+0000 7f62ee0cd700 1 --2- 172.21.15.67:0/2769380151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62e807be80 0x7f62e807a2e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f62d80098a0 tx=0x7f62d8009bb0 comp rx=0 tx=0).stop 2024-04-22T23:14:03.298 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.507+0000 7f62ee0cd700 1 -- 172.21.15.67:0/2769380151 shutdown_connections 2024-04-22T23:14:03.298 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.507+0000 7f62ee0cd700 1 --2- 172.21.15.67:0/2769380151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62e807be80 0x7f62e807a2e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:03.298 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.507+0000 7f62ee0cd700 1 -- 172.21.15.67:0/2769380151 >> 172.21.15.67:0/2769380151 conn(0x7f62e8101b00 msgr2=0x7f62e8103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:03.298 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.507+0000 7f62ee0cd700 1 -- 172.21.15.67:0/2769380151 shutdown_connections 2024-04-22T23:14:03.298 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.507+0000 7f62ee0cd700 1 -- 172.21.15.67:0/2769380151 wait complete. 2024-04-22T23:14:03.298 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.507+0000 7f62ee0cd700 1 Processor -- start 2024-04-22T23:14:03.298 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.507+0000 7f62ee0cd700 1 -- start start 2024-04-22T23:14:03.298 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.508+0000 7f62ee0cd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62e807be80 0x7f62e819c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:03.299 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.508+0000 7f62ee0cd700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e819c990 con 0x7f62e807be80 2024-04-22T23:14:03.299 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.508+0000 7f62e77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62e807be80 0x7f62e819c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:03.299 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.508+0000 7f62e77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62e807be80 0x7f62e819c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49454/0 (socket says 172.21.15.67:49454) 2024-04-22T23:14:03.299 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.508+0000 7f62e77fe700 1 -- 172.21.15.67:0/2072807538 learned_addr learned my addr 172.21.15.67:0/2072807538 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:03.299 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.508+0000 7f62e77fe700 1 -- 172.21.15.67:0/2072807538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d8009550 con 0x7f62e807be80 2024-04-22T23:14:03.299 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.508+0000 7f62e77fe700 1 --2- 172.21.15.67:0/2072807538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62e807be80 0x7f62e819c450 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f62d8003de0 tx=0x7f62d8011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:03.299 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.508+0000 7f62e4ff9700 1 -- 172.21.15.67:0/2072807538 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62d8003930 con 0x7f62e807be80 2024-04-22T23:14:03.299 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.508+0000 7f62e4ff9700 1 -- 172.21.15.67:0/2072807538 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f62d8003a90 con 0x7f62e807be80 2024-04-22T23:14:03.299 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.508+0000 7f62e4ff9700 1 -- 172.21.15.67:0/2072807538 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62d80196c0 con 0x7f62e807be80 2024-04-22T23:14:03.299 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.508+0000 7f62ee0cd700 1 -- 172.21.15.67:0/2072807538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62e819cb90 con 0x7f62e807be80 2024-04-22T23:14:03.299 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.509+0000 7f62ee0cd700 1 -- 172.21.15.67:0/2072807538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62e819d030 con 0x7f62e807be80 2024-04-22T23:14:03.299 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.509+0000 7f62e4ff9700 1 -- 172.21.15.67:0/2072807538 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f62d801abc0 con 0x7f62e807be80 2024-04-22T23:14:03.300 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.510+0000 7f62ee0cd700 1 -- 172.21.15.67:0/2072807538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62e810c8e0 con 0x7f62e807be80 2024-04-22T23:14:03.300 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.510+0000 7f62e4ff9700 1 --2- 172.21.15.67:0/2072807538 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f62d0038320 0x7f62d003a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:03.300 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.510+0000 7f62e4ff9700 1 -- 172.21.15.67:0/2072807538 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f62d804c470 con 0x7f62e807be80 2024-04-22T23:14:03.300 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.510+0000 7f62e6ffd700 1 --2- 172.21.15.67:0/2072807538 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f62d0038320 0x7f62d003a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:03.300 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.510+0000 7f62e6ffd700 1 --2- 172.21.15.67:0/2072807538 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f62d0038320 0x7f62d003a7e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f62dc006fd0 tx=0x7f62dc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:03.300 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.515+0000 7f62e4ff9700 1 -- 172.21.15.67:0/2072807538 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f62d8010380 con 0x7f62e807be80 2024-04-22T23:14:03.300 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.713+0000 7f62ee0cd700 1 -- 172.21.15.67:0/2072807538 --> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f62e8103e00 con 0x7f62d0038320 2024-04-22T23:14:03.300 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.713+0000 7f62e4ff9700 1 -- 172.21.15.67:0/2072807538 <== mgr.14120 v2:172.21.15.67:6800/3568049827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f62e8103e00 con 0x7f62d0038320 2024-04-22T23:14:03.300 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.715+0000 7f62ee0cd700 1 -- 172.21.15.67:0/2072807538 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f62d0038320 msgr2=0x7f62d003a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:03.300 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.715+0000 7f62ee0cd700 1 --2- 172.21.15.67:0/2072807538 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f62d0038320 0x7f62d003a7e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f62dc006fd0 tx=0x7f62dc006e40 comp rx=0 tx=0).stop 2024-04-22T23:14:03.300 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.715+0000 7f62ee0cd700 1 -- 172.21.15.67:0/2072807538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62e807be80 msgr2=0x7f62e819c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:03.300 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.715+0000 7f62ee0cd700 1 --2- 172.21.15.67:0/2072807538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62e807be80 0x7f62e819c450 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f62d8003de0 tx=0x7f62d8011ab0 comp rx=0 tx=0).stop 2024-04-22T23:14:03.300 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.715+0000 7f62ee0cd700 1 -- 172.21.15.67:0/2072807538 shutdown_connections 2024-04-22T23:14:03.300 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.715+0000 7f62ee0cd700 1 --2- 172.21.15.67:0/2072807538 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f62d0038320 0x7f62d003a7e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:03.301 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.715+0000 7f62ee0cd700 1 --2- 172.21.15.67:0/2072807538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62e807be80 0x7f62e819c450 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:03.301 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.715+0000 7f62ee0cd700 1 -- 172.21.15.67:0/2072807538 >> 172.21.15.67:0/2072807538 conn(0x7f62e8101b00 msgr2=0x7f62e81031e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:03.301 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.715+0000 7f62ee0cd700 1 -- 172.21.15.67:0/2072807538 shutdown_connections 2024-04-22T23:14:03.301 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:02.715+0000 7f62ee0cd700 1 -- 172.21.15.67:0/2072807538 wait complete. 2024-04-22T23:14:03.301 INFO:teuthology.orchestra.run.smithi067.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-22T23:14:03.301 INFO:teuthology.orchestra.run.smithi067.stdout:Adding key to root@localhost authorized_keys... 2024-04-22T23:14:03.301 INFO:teuthology.orchestra.run.smithi067.stdout:Adding host smithi067... 2024-04-22T23:14:03.724 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:03 smithi067 ceph-mon[26453]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:14:04.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:04 smithi067 ceph-mon[26453]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi067", "addr": "172.21.15.67", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:14:06.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:05 smithi067 ceph-mon[26453]: Deploying cephadm binary to smithi067 2024-04-22T23:14:07.012 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout Added host 'smithi067' with addr '172.21.15.67' 2024-04-22T23:14:07.012 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.767+0000 7fda34dfa700 1 Processor -- start 2024-04-22T23:14:07.012 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.767+0000 7fda34dfa700 1 -- start start 2024-04-22T23:14:07.012 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.767+0000 7fda34dfa700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda30106560 0x7fda30106980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:07.012 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.767+0000 7fda34dfa700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda30106f50 con 0x7fda30106560 2024-04-22T23:14:07.012 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.767+0000 7fda2e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda30106560 0x7fda30106980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:07.012 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.767+0000 7fda2e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda30106560 0x7fda30106980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49460/0 (socket says 172.21.15.67:49460) 2024-04-22T23:14:07.012 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.767+0000 7fda2e59c700 1 -- 172.21.15.67:0/1934858996 learned_addr learned my addr 172.21.15.67:0/1934858996 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:07.012 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.768+0000 7fda2e59c700 1 -- 172.21.15.67:0/1934858996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda30107760 con 0x7fda30106560 2024-04-22T23:14:07.012 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.768+0000 7fda2e59c700 1 --2- 172.21.15.67:0/1934858996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda30106560 0x7fda30106980 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fda200098a0 tx=0x7fda20009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd5c4c3ef7b516a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:07.013 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.768+0000 7fda2d59a700 1 -- 172.21.15.67:0/1934858996 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda20004180 con 0x7fda30106560 2024-04-22T23:14:07.013 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.768+0000 7fda2d59a700 1 -- 172.21.15.67:0/1934858996 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fda200042e0 con 0x7fda30106560 2024-04-22T23:14:07.013 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.768+0000 7fda2d59a700 1 -- 172.21.15.67:0/1934858996 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda20011a50 con 0x7fda30106560 2024-04-22T23:14:07.013 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.768+0000 7fda34dfa700 1 -- 172.21.15.67:0/1934858996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda30106560 msgr2=0x7fda30106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:07.013 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.768+0000 7fda34dfa700 1 --2- 172.21.15.67:0/1934858996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda30106560 0x7fda30106980 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fda200098a0 tx=0x7fda20009bb0 comp rx=0 tx=0).stop 2024-04-22T23:14:07.013 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.769+0000 7fda34dfa700 1 -- 172.21.15.67:0/1934858996 shutdown_connections 2024-04-22T23:14:07.013 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.769+0000 7fda34dfa700 1 --2- 172.21.15.67:0/1934858996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda30106560 0x7fda30106980 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:07.013 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.769+0000 7fda34dfa700 1 -- 172.21.15.67:0/1934858996 >> 172.21.15.67:0/1934858996 conn(0x7fda30101b40 msgr2=0x7fda30103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:07.013 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.769+0000 7fda34dfa700 1 -- 172.21.15.67:0/1934858996 shutdown_connections 2024-04-22T23:14:07.013 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.769+0000 7fda34dfa700 1 -- 172.21.15.67:0/1934858996 wait complete. 2024-04-22T23:14:07.013 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.769+0000 7fda34dfa700 1 Processor -- start 2024-04-22T23:14:07.013 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.769+0000 7fda34dfa700 1 -- start start 2024-04-22T23:14:07.013 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.770+0000 7fda34dfa700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda30106560 0x7fda3019a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:07.013 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.770+0000 7fda34dfa700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3019a730 con 0x7fda30106560 2024-04-22T23:14:07.014 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.770+0000 7fda2e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda30106560 0x7fda3019a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:07.014 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.770+0000 7fda2e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda30106560 0x7fda3019a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49466/0 (socket says 172.21.15.67:49466) 2024-04-22T23:14:07.014 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.770+0000 7fda2e59c700 1 -- 172.21.15.67:0/3882054595 learned_addr learned my addr 172.21.15.67:0/3882054595 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:07.014 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.770+0000 7fda2e59c700 1 -- 172.21.15.67:0/3882054595 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda20009550 con 0x7fda30106560 2024-04-22T23:14:07.014 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.770+0000 7fda2e59c700 1 --2- 172.21.15.67:0/3882054595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda30106560 0x7fda3019a1f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fda20009520 tx=0x7fda20011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:07.014 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.770+0000 7fda1f7fe700 1 -- 172.21.15.67:0/3882054595 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda20003a80 con 0x7fda30106560 2024-04-22T23:14:07.014 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.770+0000 7fda1f7fe700 1 -- 172.21.15.67:0/3882054595 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fda20003be0 con 0x7fda30106560 2024-04-22T23:14:07.014 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.770+0000 7fda34dfa700 1 -- 172.21.15.67:0/3882054595 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda3019a930 con 0x7fda30106560 2024-04-22T23:14:07.014 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.770+0000 7fda1f7fe700 1 -- 172.21.15.67:0/3882054595 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda200197b0 con 0x7fda30106560 2024-04-22T23:14:07.014 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.770+0000 7fda34dfa700 1 -- 172.21.15.67:0/3882054595 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda3019add0 con 0x7fda30106560 2024-04-22T23:14:07.014 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.771+0000 7fda1f7fe700 1 -- 172.21.15.67:0/3882054595 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fda20019a40 con 0x7fda30106560 2024-04-22T23:14:07.014 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.771+0000 7fda34dfa700 1 -- 172.21.15.67:0/3882054595 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda300623c0 con 0x7fda30106560 2024-04-22T23:14:07.014 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.772+0000 7fda1f7fe700 1 --2- 172.21.15.67:0/3882054595 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fda18038320 0x7fda1803a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:07.014 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.772+0000 7fda1f7fe700 1 -- 172.21.15.67:0/3882054595 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fda2004c680 con 0x7fda30106560 2024-04-22T23:14:07.015 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.772+0000 7fda2dd9b700 1 --2- 172.21.15.67:0/3882054595 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fda18038320 0x7fda1803a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:07.015 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.772+0000 7fda2dd9b700 1 --2- 172.21.15.67:0/3882054595 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fda18038320 0x7fda1803a7e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fda24006fd0 tx=0x7fda24006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:07.015 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.776+0000 7fda1f7fe700 1 -- 172.21.15.67:0/3882054595 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fda2002b750 con 0x7fda30106560 2024-04-22T23:14:07.015 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:03.977+0000 7fda34dfa700 1 -- 172.21.15.67:0/3882054595 --> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi067", "addr": "172.21.15.67", "target": ["mon-mgr", ""]}) v1 -- 0x7fda3010a960 con 0x7fda18038320 2024-04-22T23:14:07.015 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:06.430+0000 7fda1f7fe700 1 -- 172.21.15.67:0/3882054595 <== mgr.14120 v2:172.21.15.67:6800/3568049827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fda3010a960 con 0x7fda18038320 2024-04-22T23:14:07.015 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:06.432+0000 7fda34dfa700 1 -- 172.21.15.67:0/3882054595 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fda18038320 msgr2=0x7fda1803a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:07.015 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:06.432+0000 7fda34dfa700 1 --2- 172.21.15.67:0/3882054595 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fda18038320 0x7fda1803a7e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fda24006fd0 tx=0x7fda24006e40 comp rx=0 tx=0).stop 2024-04-22T23:14:07.015 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:06.432+0000 7fda34dfa700 1 -- 172.21.15.67:0/3882054595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda30106560 msgr2=0x7fda3019a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:07.015 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:06.432+0000 7fda34dfa700 1 --2- 172.21.15.67:0/3882054595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda30106560 0x7fda3019a1f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fda20009520 tx=0x7fda20011ab0 comp rx=0 tx=0).stop 2024-04-22T23:14:07.015 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:06.432+0000 7fda34dfa700 1 -- 172.21.15.67:0/3882054595 shutdown_connections 2024-04-22T23:14:07.015 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:06.432+0000 7fda34dfa700 1 --2- 172.21.15.67:0/3882054595 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fda18038320 0x7fda1803a7e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:07.015 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:06.432+0000 7fda34dfa700 1 --2- 172.21.15.67:0/3882054595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda30106560 0x7fda3019a1f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:07.015 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:06.432+0000 7fda34dfa700 1 -- 172.21.15.67:0/3882054595 >> 172.21.15.67:0/3882054595 conn(0x7fda30101b40 msgr2=0x7fda30102900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:07.015 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:06.432+0000 7fda34dfa700 1 -- 172.21.15.67:0/3882054595 shutdown_connections 2024-04-22T23:14:07.015 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:06.432+0000 7fda34dfa700 1 -- 172.21.15.67:0/3882054595 wait complete. 2024-04-22T23:14:07.016 INFO:teuthology.orchestra.run.smithi067.stdout:Deploying mon service with default placement... 2024-04-22T23:14:07.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:07 smithi067 ceph-mon[26453]: from='mgr.14120 172.21.15.67:0/1431842859' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:07.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:07 smithi067 ceph-mon[26453]: Added host smithi067 2024-04-22T23:14:07.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:07 smithi067 ceph-mon[26453]: from='mgr.14120 172.21.15.67:0/1431842859' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:14:08.328 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-22T23:14:08.328 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.434+0000 7f2b2c0cd700 1 Processor -- start 2024-04-22T23:14:08.328 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.435+0000 7f2b2c0cd700 1 -- start start 2024-04-22T23:14:08.328 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.435+0000 7f2b2c0cd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b24104850 0x7f2b24104c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:08.328 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.435+0000 7f2b2c0cd700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b24105240 con 0x7f2b24104850 2024-04-22T23:14:08.328 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.435+0000 7f2b29e69700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b24104850 0x7f2b24104c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:08.328 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.435+0000 7f2b29e69700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b24104850 0x7f2b24104c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49478/0 (socket says 172.21.15.67:49478) 2024-04-22T23:14:08.328 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.435+0000 7f2b29e69700 1 -- 172.21.15.67:0/2858763587 learned_addr learned my addr 172.21.15.67:0/2858763587 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:08.328 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.435+0000 7f2b29e69700 1 -- 172.21.15.67:0/2858763587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b24105380 con 0x7f2b24104850 2024-04-22T23:14:08.328 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.435+0000 7f2b29e69700 1 --2- 172.21.15.67:0/2858763587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b24104850 0x7f2b24104c70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2b200098a0 tx=0x7f2b20009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5957bc1ab89f930c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:08.329 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.436+0000 7f2b28e67700 1 -- 172.21.15.67:0/2858763587 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b20004180 con 0x7f2b24104850 2024-04-22T23:14:08.329 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.436+0000 7f2b28e67700 1 -- 172.21.15.67:0/2858763587 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2b200042e0 con 0x7f2b24104850 2024-04-22T23:14:08.329 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.436+0000 7f2b28e67700 1 -- 172.21.15.67:0/2858763587 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b20011a50 con 0x7f2b24104850 2024-04-22T23:14:08.329 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.436+0000 7f2b2c0cd700 1 -- 172.21.15.67:0/2858763587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b24104850 msgr2=0x7f2b24104c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:08.329 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.436+0000 7f2b2c0cd700 1 --2- 172.21.15.67:0/2858763587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b24104850 0x7f2b24104c70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2b200098a0 tx=0x7f2b20009bb0 comp rx=0 tx=0).stop 2024-04-22T23:14:08.329 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.436+0000 7f2b2c0cd700 1 -- 172.21.15.67:0/2858763587 shutdown_connections 2024-04-22T23:14:08.329 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.436+0000 7f2b2c0cd700 1 --2- 172.21.15.67:0/2858763587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b24104850 0x7f2b24104c70 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:08.329 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.436+0000 7f2b2c0cd700 1 -- 172.21.15.67:0/2858763587 >> 172.21.15.67:0/2858763587 conn(0x7f2b241001a0 msgr2=0x7f2b241025c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:08.329 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.436+0000 7f2b2c0cd700 1 -- 172.21.15.67:0/2858763587 shutdown_connections 2024-04-22T23:14:08.329 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.436+0000 7f2b2c0cd700 1 -- 172.21.15.67:0/2858763587 wait complete. 2024-04-22T23:14:08.329 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.437+0000 7f2b2c0cd700 1 Processor -- start 2024-04-22T23:14:08.329 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.437+0000 7f2b2c0cd700 1 -- start start 2024-04-22T23:14:08.329 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.437+0000 7f2b2c0cd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b24104850 0x7f2b2419c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:08.329 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.437+0000 7f2b2c0cd700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2419c950 con 0x7f2b24104850 2024-04-22T23:14:08.329 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.437+0000 7f2b29e69700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b24104850 0x7f2b2419c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:08.330 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.437+0000 7f2b29e69700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b24104850 0x7f2b2419c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49488/0 (socket says 172.21.15.67:49488) 2024-04-22T23:14:08.330 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.437+0000 7f2b29e69700 1 -- 172.21.15.67:0/4225706520 learned_addr learned my addr 172.21.15.67:0/4225706520 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:08.330 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.438+0000 7f2b29e69700 1 -- 172.21.15.67:0/4225706520 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b20009550 con 0x7f2b24104850 2024-04-22T23:14:08.330 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.438+0000 7f2b29e69700 1 --2- 172.21.15.67:0/4225706520 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b24104850 0x7f2b2419c410 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2b20003de0 tx=0x7f2b20011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:08.330 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.438+0000 7f2b1affd700 1 -- 172.21.15.67:0/4225706520 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b20003930 con 0x7f2b24104850 2024-04-22T23:14:08.330 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.438+0000 7f2b1affd700 1 -- 172.21.15.67:0/4225706520 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7f2b20003a90 con 0x7f2b24104850 2024-04-22T23:14:08.330 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.438+0000 7f2b1affd700 1 -- 172.21.15.67:0/4225706520 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b200196c0 con 0x7f2b24104850 2024-04-22T23:14:08.330 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.438+0000 7f2b2c0cd700 1 -- 172.21.15.67:0/4225706520 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b2419cb50 con 0x7f2b24104850 2024-04-22T23:14:08.330 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.438+0000 7f2b2c0cd700 1 -- 172.21.15.67:0/4225706520 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b2419cff0 con 0x7f2b24104850 2024-04-22T23:14:08.330 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.439+0000 7f2b1affd700 1 -- 172.21.15.67:0/4225706520 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f2b2001a430 con 0x7f2b24104850 2024-04-22T23:14:08.330 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.439+0000 7f2b2c0cd700 1 -- 172.21.15.67:0/4225706520 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b24196380 con 0x7f2b24104850 2024-04-22T23:14:08.330 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.439+0000 7f2b1affd700 1 --2- 172.21.15.67:0/4225706520 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f2b10038370 0x7f2b1003a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:08.330 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.439+0000 7f2b1affd700 1 -- 172.21.15.67:0/4225706520 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f2b2004c3d0 con 0x7f2b24104850 2024-04-22T23:14:08.330 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.439+0000 7f2b29668700 1 --2- 172.21.15.67:0/4225706520 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f2b10038370 0x7f2b1003a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:08.331 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.439+0000 7f2b29668700 1 --2- 172.21.15.67:0/4225706520 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f2b10038370 0x7f2b1003a830 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2b14006fd0 tx=0x7f2b14006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:08.331 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.444+0000 7f2b1affd700 1 -- 172.21.15.67:0/4225706520 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f2b20010380 con 0x7f2b24104850 2024-04-22T23:14:08.331 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.668+0000 7f2b2c0cd700 1 -- 172.21.15.67:0/4225706520 --> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b240611d0 con 0x7f2b10038370 2024-04-22T23:14:08.331 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.675+0000 7f2b1affd700 1 -- 172.21.15.67:0/4225706520 <== mgr.14120 v2:172.21.15.67:6800/3568049827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f2b240611d0 con 0x7f2b10038370 2024-04-22T23:14:08.331 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.677+0000 7f2b2c0cd700 1 -- 172.21.15.67:0/4225706520 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f2b10038370 msgr2=0x7f2b1003a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:08.331 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.677+0000 7f2b2c0cd700 1 --2- 172.21.15.67:0/4225706520 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f2b10038370 0x7f2b1003a830 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2b14006fd0 tx=0x7f2b14006e40 comp rx=0 tx=0).stop 2024-04-22T23:14:08.331 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.677+0000 7f2b2c0cd700 1 -- 172.21.15.67:0/4225706520 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b24104850 msgr2=0x7f2b2419c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:08.331 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.677+0000 7f2b2c0cd700 1 --2- 172.21.15.67:0/4225706520 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b24104850 0x7f2b2419c410 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2b20003de0 tx=0x7f2b20011ab0 comp rx=0 tx=0).stop 2024-04-22T23:14:08.331 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.677+0000 7f2b2c0cd700 1 -- 172.21.15.67:0/4225706520 shutdown_connections 2024-04-22T23:14:08.331 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.677+0000 7f2b2c0cd700 1 --2- 172.21.15.67:0/4225706520 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f2b10038370 0x7f2b1003a830 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:08.331 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.677+0000 7f2b2c0cd700 1 --2- 172.21.15.67:0/4225706520 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b24104850 0x7f2b2419c410 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:08.331 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.677+0000 7f2b2c0cd700 1 -- 172.21.15.67:0/4225706520 >> 172.21.15.67:0/4225706520 conn(0x7f2b241001a0 msgr2=0x7f2b24109ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:08.331 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.677+0000 7f2b2c0cd700 1 -- 172.21.15.67:0/4225706520 shutdown_connections 2024-04-22T23:14:08.331 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:07.677+0000 7f2b2c0cd700 1 -- 172.21.15.67:0/4225706520 wait complete. 2024-04-22T23:14:08.331 INFO:teuthology.orchestra.run.smithi067.stdout:Deploying mgr service with default placement... 2024-04-22T23:14:08.841 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:08 smithi067 ceph-mon[26453]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:14:08.841 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:08 smithi067 ceph-mon[26453]: Saving service mon spec with placement count:5 2024-04-22T23:14:08.842 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:08 smithi067 ceph-mon[26453]: from='mgr.14120 172.21.15.67:0/1431842859' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:10.163 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-22T23:14:10.163 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.840+0000 7fbe7843a700 1 Processor -- start 2024-04-22T23:14:10.163 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.840+0000 7fbe7843a700 1 -- start start 2024-04-22T23:14:10.163 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.841+0000 7fbe7843a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe70107040 0x7fbe70109430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:10.164 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.841+0000 7fbe7843a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe70074620 con 0x7fbe70107040 2024-04-22T23:14:10.164 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.841+0000 7fbe761d6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe70107040 0x7fbe70109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:10.164 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.841+0000 7fbe761d6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe70107040 0x7fbe70109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36410/0 (socket says 172.21.15.67:36410) 2024-04-22T23:14:10.164 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.841+0000 7fbe761d6700 1 -- 172.21.15.67:0/651204065 learned_addr learned my addr 172.21.15.67:0/651204065 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:10.164 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.841+0000 7fbe761d6700 1 -- 172.21.15.67:0/651204065 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe70109970 con 0x7fbe70107040 2024-04-22T23:14:10.164 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.841+0000 7fbe761d6700 1 --2- 172.21.15.67:0/651204065 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe70107040 0x7fbe70109430 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c0098a0 tx=0x7fbe6c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e394f533828bf6f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:10.164 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.841+0000 7fbe751d4700 1 -- 172.21.15.67:0/651204065 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe6c004180 con 0x7fbe70107040 2024-04-22T23:14:10.164 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.842+0000 7fbe751d4700 1 -- 172.21.15.67:0/651204065 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbe6c0042e0 con 0x7fbe70107040 2024-04-22T23:14:10.164 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.842+0000 7fbe751d4700 1 -- 172.21.15.67:0/651204065 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe6c011a50 con 0x7fbe70107040 2024-04-22T23:14:10.164 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.842+0000 7fbe7843a700 1 -- 172.21.15.67:0/651204065 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe70107040 msgr2=0x7fbe70109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:10.164 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.842+0000 7fbe7843a700 1 --2- 172.21.15.67:0/651204065 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe70107040 0x7fbe70109430 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c0098a0 tx=0x7fbe6c009bb0 comp rx=0 tx=0).stop 2024-04-22T23:14:10.164 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.842+0000 7fbe7843a700 1 -- 172.21.15.67:0/651204065 shutdown_connections 2024-04-22T23:14:10.164 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.842+0000 7fbe7843a700 1 --2- 172.21.15.67:0/651204065 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe70107040 0x7fbe70109430 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:10.164 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.842+0000 7fbe7843a700 1 -- 172.21.15.67:0/651204065 >> 172.21.15.67:0/651204065 conn(0x7fbe701009f0 msgr2=0x7fbe70102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:10.165 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.842+0000 7fbe7843a700 1 -- 172.21.15.67:0/651204065 shutdown_connections 2024-04-22T23:14:10.165 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.842+0000 7fbe7843a700 1 -- 172.21.15.67:0/651204065 wait complete. 2024-04-22T23:14:10.165 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.843+0000 7fbe7843a700 1 Processor -- start 2024-04-22T23:14:10.165 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.843+0000 7fbe7843a700 1 -- start start 2024-04-22T23:14:10.165 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.843+0000 7fbe7843a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe70107040 0x7fbe7019c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:10.165 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.843+0000 7fbe7843a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7019c870 con 0x7fbe70107040 2024-04-22T23:14:10.165 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.843+0000 7fbe761d6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe70107040 0x7fbe7019c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:10.165 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.843+0000 7fbe761d6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe70107040 0x7fbe7019c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36424/0 (socket says 172.21.15.67:36424) 2024-04-22T23:14:10.165 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.843+0000 7fbe761d6700 1 -- 172.21.15.67:0/1178093345 learned_addr learned my addr 172.21.15.67:0/1178093345 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:10.165 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.843+0000 7fbe761d6700 1 -- 172.21.15.67:0/1178093345 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe6c009550 con 0x7fbe70107040 2024-04-22T23:14:10.165 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.844+0000 7fbe761d6700 1 --2- 172.21.15.67:0/1178093345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe70107040 0x7fbe7019c330 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c000c00 tx=0x7fbe6c011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:10.165 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.844+0000 7fbe637fe700 1 -- 172.21.15.67:0/1178093345 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe6c003b90 con 0x7fbe70107040 2024-04-22T23:14:10.165 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.844+0000 7fbe637fe700 1 -- 172.21.15.67:0/1178093345 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1037+0+0 (secure 0 0 0) 0x7fbe6c0248b0 con 0x7fbe70107040 2024-04-22T23:14:10.165 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.844+0000 7fbe7843a700 1 -- 172.21.15.67:0/1178093345 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe7019ca70 con 0x7fbe70107040 2024-04-22T23:14:10.165 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.844+0000 7fbe637fe700 1 -- 172.21.15.67:0/1178093345 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe6c01a8e0 con 0x7fbe70107040 2024-04-22T23:14:10.166 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.844+0000 7fbe7843a700 1 -- 172.21.15.67:0/1178093345 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe7019cf10 con 0x7fbe70107040 2024-04-22T23:14:10.166 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.845+0000 7fbe637fe700 1 -- 172.21.15.67:0/1178093345 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fbe6c01ab50 con 0x7fbe70107040 2024-04-22T23:14:10.166 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.845+0000 7fbe7843a700 1 -- 172.21.15.67:0/1178093345 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe7010a690 con 0x7fbe70107040 2024-04-22T23:14:10.166 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.845+0000 7fbe637fe700 1 --2- 172.21.15.67:0/1178093345 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fbe5c038370 0x7fbe5c03a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:10.166 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.845+0000 7fbe637fe700 1 -- 172.21.15.67:0/1178093345 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fbe6c04c3e0 con 0x7fbe70107040 2024-04-22T23:14:10.166 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.845+0000 7fbe759d5700 1 --2- 172.21.15.67:0/1178093345 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fbe5c038370 0x7fbe5c03a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:10.166 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.845+0000 7fbe759d5700 1 --2- 172.21.15.67:0/1178093345 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fbe5c038370 0x7fbe5c03a830 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fbe64006fd0 tx=0x7fbe64006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:10.166 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:08.850+0000 7fbe637fe700 1 -- 172.21.15.67:0/1178093345 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fbe6c01ae00 con 0x7fbe70107040 2024-04-22T23:14:10.166 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:09.058+0000 7fbe7843a700 1 -- 172.21.15.67:0/1178093345 --> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe70101e50 con 0x7fbe5c038370 2024-04-22T23:14:10.166 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:09.063+0000 7fbe637fe700 1 -- 172.21.15.67:0/1178093345 <== mgr.14120 v2:172.21.15.67:6800/3568049827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fbe70101e50 con 0x7fbe5c038370 2024-04-22T23:14:10.166 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:09.065+0000 7fbe7843a700 1 -- 172.21.15.67:0/1178093345 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fbe5c038370 msgr2=0x7fbe5c03a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:10.166 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:09.065+0000 7fbe7843a700 1 --2- 172.21.15.67:0/1178093345 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fbe5c038370 0x7fbe5c03a830 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fbe64006fd0 tx=0x7fbe64006e40 comp rx=0 tx=0).stop 2024-04-22T23:14:10.166 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:09.065+0000 7fbe7843a700 1 -- 172.21.15.67:0/1178093345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe70107040 msgr2=0x7fbe7019c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:10.166 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:09.065+0000 7fbe7843a700 1 --2- 172.21.15.67:0/1178093345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe70107040 0x7fbe7019c330 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c000c00 tx=0x7fbe6c011ab0 comp rx=0 tx=0).stop 2024-04-22T23:14:10.167 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:09.065+0000 7fbe7843a700 1 -- 172.21.15.67:0/1178093345 shutdown_connections 2024-04-22T23:14:10.167 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:09.065+0000 7fbe7843a700 1 --2- 172.21.15.67:0/1178093345 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fbe5c038370 0x7fbe5c03a830 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:10.167 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:09.065+0000 7fbe7843a700 1 --2- 172.21.15.67:0/1178093345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe70107040 0x7fbe7019c330 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:10.167 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:09.065+0000 7fbe7843a700 1 -- 172.21.15.67:0/1178093345 >> 172.21.15.67:0/1178093345 conn(0x7fbe701009f0 msgr2=0x7fbe70101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:10.167 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:09.065+0000 7fbe7843a700 1 -- 172.21.15.67:0/1178093345 shutdown_connections 2024-04-22T23:14:10.167 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:09.065+0000 7fbe7843a700 1 -- 172.21.15.67:0/1178093345 wait complete. 2024-04-22T23:14:10.167 INFO:teuthology.orchestra.run.smithi067.stdout:Deploying crash service with default placement... 2024-04-22T23:14:10.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:10 smithi067 ceph-mon[26453]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:14:10.282 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:10 smithi067 ceph-mon[26453]: Saving service mgr spec with placement count:2 2024-04-22T23:14:10.282 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:10 smithi067 ceph-mon[26453]: from='mgr.14120 172.21.15.67:0/1431842859' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:10.282 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:10 smithi067 ceph-mon[26453]: from='mgr.14120 172.21.15.67:0/1431842859' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:11.452 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-04-22T23:14:11.453 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.601+0000 7fd26e32b700 1 Processor -- start 2024-04-22T23:14:11.453 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.601+0000 7fd26e32b700 1 -- start start 2024-04-22T23:14:11.453 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.601+0000 7fd26e32b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd268106fd0 0x7fd2681093c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:11.453 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.601+0000 7fd26e32b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd268079e90 con 0x7fd268106fd0 2024-04-22T23:14:11.453 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.601+0000 7fd267fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd268106fd0 0x7fd2681093c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:11.453 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.601+0000 7fd267fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd268106fd0 0x7fd2681093c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36430/0 (socket says 172.21.15.67:36430) 2024-04-22T23:14:11.453 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.601+0000 7fd267fff700 1 -- 172.21.15.67:0/2115552634 learned_addr learned my addr 172.21.15.67:0/2115552634 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:11.453 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.602+0000 7fd267fff700 1 -- 172.21.15.67:0/2115552634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd268109900 con 0x7fd268106fd0 2024-04-22T23:14:11.453 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.602+0000 7fd267fff700 1 --2- 172.21.15.67:0/2115552634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd268106fd0 0x7fd2681093c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd2580098a0 tx=0x7fd258009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8001674cb8465075 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:11.453 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.602+0000 7fd266ffd700 1 -- 172.21.15.67:0/2115552634 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd258004180 con 0x7fd268106fd0 2024-04-22T23:14:11.453 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.602+0000 7fd266ffd700 1 -- 172.21.15.67:0/2115552634 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd2580042e0 con 0x7fd268106fd0 2024-04-22T23:14:11.453 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.602+0000 7fd266ffd700 1 -- 172.21.15.67:0/2115552634 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd258011af0 con 0x7fd268106fd0 2024-04-22T23:14:11.453 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.602+0000 7fd26e32b700 1 -- 172.21.15.67:0/2115552634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd268106fd0 msgr2=0x7fd2681093c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:11.454 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.602+0000 7fd26e32b700 1 --2- 172.21.15.67:0/2115552634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd268106fd0 0x7fd2681093c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd2580098a0 tx=0x7fd258009bb0 comp rx=0 tx=0).stop 2024-04-22T23:14:11.454 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.603+0000 7fd26e32b700 1 -- 172.21.15.67:0/2115552634 shutdown_connections 2024-04-22T23:14:11.454 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.603+0000 7fd26e32b700 1 --2- 172.21.15.67:0/2115552634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd268106fd0 0x7fd2681093c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:11.454 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.603+0000 7fd26e32b700 1 -- 172.21.15.67:0/2115552634 >> 172.21.15.67:0/2115552634 conn(0x7fd268100a20 msgr2=0x7fd268102e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:11.454 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.603+0000 7fd26e32b700 1 -- 172.21.15.67:0/2115552634 shutdown_connections 2024-04-22T23:14:11.454 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.603+0000 7fd26e32b700 1 -- 172.21.15.67:0/2115552634 wait complete. 2024-04-22T23:14:11.454 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.603+0000 7fd26e32b700 1 Processor -- start 2024-04-22T23:14:11.454 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.603+0000 7fd26e32b700 1 -- start start 2024-04-22T23:14:11.454 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.603+0000 7fd26e32b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd268106fd0 0x7fd26819a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:11.454 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.603+0000 7fd26e32b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd26819a760 con 0x7fd268106fd0 2024-04-22T23:14:11.454 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.604+0000 7fd267fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd268106fd0 0x7fd26819a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:11.454 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.604+0000 7fd267fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd268106fd0 0x7fd26819a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36444/0 (socket says 172.21.15.67:36444) 2024-04-22T23:14:11.454 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.604+0000 7fd267fff700 1 -- 172.21.15.67:0/263948433 learned_addr learned my addr 172.21.15.67:0/263948433 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:11.454 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.604+0000 7fd267fff700 1 -- 172.21.15.67:0/263948433 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd258009550 con 0x7fd268106fd0 2024-04-22T23:14:11.455 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.604+0000 7fd267fff700 1 --2- 172.21.15.67:0/263948433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd268106fd0 0x7fd26819a220 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd258011670 tx=0x7fd258011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:11.455 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.604+0000 7fd2657fa700 1 -- 172.21.15.67:0/263948433 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd258003a40 con 0x7fd268106fd0 2024-04-22T23:14:11.455 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.604+0000 7fd2657fa700 1 -- 172.21.15.67:0/263948433 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd258003ba0 con 0x7fd268106fd0 2024-04-22T23:14:11.455 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.604+0000 7fd2657fa700 1 -- 172.21.15.67:0/263948433 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd258024820 con 0x7fd268106fd0 2024-04-22T23:14:11.455 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.604+0000 7fd26e32b700 1 -- 172.21.15.67:0/263948433 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd26819a960 con 0x7fd268106fd0 2024-04-22T23:14:11.455 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.604+0000 7fd26e32b700 1 -- 172.21.15.67:0/263948433 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd26819ae00 con 0x7fd268106fd0 2024-04-22T23:14:11.455 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.605+0000 7fd2657fa700 1 -- 172.21.15.67:0/263948433 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fd25801a430 con 0x7fd268106fd0 2024-04-22T23:14:11.455 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.605+0000 7fd26e32b700 1 -- 172.21.15.67:0/263948433 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd26810a6f0 con 0x7fd268106fd0 2024-04-22T23:14:11.455 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.606+0000 7fd2657fa700 1 --2- 172.21.15.67:0/263948433 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fd250038330 0x7fd25003a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:11.455 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.606+0000 7fd2657fa700 1 -- 172.21.15.67:0/263948433 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fd25804c290 con 0x7fd268106fd0 2024-04-22T23:14:11.455 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.606+0000 7fd2677fe700 1 --2- 172.21.15.67:0/263948433 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fd250038330 0x7fd25003a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:11.455 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.606+0000 7fd2677fe700 1 --2- 172.21.15.67:0/263948433 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fd250038330 0x7fd25003a7f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd25c006fd0 tx=0x7fd25c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:11.455 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.610+0000 7fd2657fa700 1 -- 172.21.15.67:0/263948433 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd25801a6e0 con 0x7fd268106fd0 2024-04-22T23:14:11.455 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.829+0000 7fd26e32b700 1 -- 172.21.15.67:0/263948433 --> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7fd268101ec0 con 0x7fd250038330 2024-04-22T23:14:11.456 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.837+0000 7fd2657fa700 1 -- 172.21.15.67:0/263948433 <== mgr.14120 v2:172.21.15.67:6800/3568049827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7fd268101ec0 con 0x7fd250038330 2024-04-22T23:14:11.456 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.840+0000 7fd26e32b700 1 -- 172.21.15.67:0/263948433 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fd250038330 msgr2=0x7fd25003a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:11.456 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.840+0000 7fd26e32b700 1 --2- 172.21.15.67:0/263948433 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fd250038330 0x7fd25003a7f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd25c006fd0 tx=0x7fd25c006e40 comp rx=0 tx=0).stop 2024-04-22T23:14:11.456 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.840+0000 7fd26e32b700 1 -- 172.21.15.67:0/263948433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd268106fd0 msgr2=0x7fd26819a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:11.456 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.840+0000 7fd26e32b700 1 --2- 172.21.15.67:0/263948433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd268106fd0 0x7fd26819a220 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd258011670 tx=0x7fd258011aa0 comp rx=0 tx=0).stop 2024-04-22T23:14:11.456 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.840+0000 7fd26e32b700 1 -- 172.21.15.67:0/263948433 shutdown_connections 2024-04-22T23:14:11.456 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.840+0000 7fd26e32b700 1 --2- 172.21.15.67:0/263948433 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fd250038330 0x7fd25003a7f0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:11.456 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.840+0000 7fd26e32b700 1 --2- 172.21.15.67:0/263948433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd268106fd0 0x7fd26819a220 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:11.456 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.840+0000 7fd26e32b700 1 -- 172.21.15.67:0/263948433 >> 172.21.15.67:0/263948433 conn(0x7fd268100a20 msgr2=0x7fd2681017b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:11.456 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.840+0000 7fd26e32b700 1 -- 172.21.15.67:0/263948433 shutdown_connections 2024-04-22T23:14:11.456 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:10.840+0000 7fd26e32b700 1 -- 172.21.15.67:0/263948433 wait complete. 2024-04-22T23:14:11.456 INFO:teuthology.orchestra.run.smithi067.stdout:Deploying ceph-exporter service with default placement... 2024-04-22T23:14:12.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:11 smithi067 ceph-mon[26453]: from='mgr.14120 172.21.15.67:0/1431842859' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:12.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:11 smithi067 ceph-mon[26453]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:14:12.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:11 smithi067 ceph-mon[26453]: Saving service crash spec with placement * 2024-04-22T23:14:12.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:11 smithi067 ceph-mon[26453]: from='mgr.14120 172.21.15.67:0/1431842859' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:12.877 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout Scheduled ceph-exporter update... 2024-04-22T23:14:12.877 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.858+0000 7f108d4f7700 1 Processor -- start 2024-04-22T23:14:12.877 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.858+0000 7f108d4f7700 1 -- start start 2024-04-22T23:14:12.878 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.858+0000 7f108d4f7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1088106560 0x7f1088106980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:12.878 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.858+0000 7f108d4f7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1088106f50 con 0x7f1088106560 2024-04-22T23:14:12.878 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.858+0000 7f1086ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1088106560 0x7f1088106980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:12.878 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.858+0000 7f1086ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1088106560 0x7f1088106980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36460/0 (socket says 172.21.15.67:36460) 2024-04-22T23:14:12.878 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.858+0000 7f1086ffd700 1 -- 172.21.15.67:0/3806457508 learned_addr learned my addr 172.21.15.67:0/3806457508 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:12.878 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.858+0000 7f1086ffd700 1 -- 172.21.15.67:0/3806457508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1088107760 con 0x7f1088106560 2024-04-22T23:14:12.878 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.859+0000 7f1086ffd700 1 --2- 172.21.15.67:0/3806457508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1088106560 0x7f1088106980 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f10780098a0 tx=0x7f1078009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3d152c517df8195 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:12.878 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.859+0000 7f1085ffb700 1 -- 172.21.15.67:0/3806457508 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1078004180 con 0x7f1088106560 2024-04-22T23:14:12.878 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.859+0000 7f1085ffb700 1 -- 172.21.15.67:0/3806457508 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f10780042e0 con 0x7f1088106560 2024-04-22T23:14:12.878 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.859+0000 7f1085ffb700 1 -- 172.21.15.67:0/3806457508 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1078011af0 con 0x7f1088106560 2024-04-22T23:14:12.878 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.859+0000 7f108d4f7700 1 -- 172.21.15.67:0/3806457508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1088106560 msgr2=0x7f1088106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:12.878 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.859+0000 7f108d4f7700 1 --2- 172.21.15.67:0/3806457508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1088106560 0x7f1088106980 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f10780098a0 tx=0x7f1078009bb0 comp rx=0 tx=0).stop 2024-04-22T23:14:12.878 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.859+0000 7f108d4f7700 1 -- 172.21.15.67:0/3806457508 shutdown_connections 2024-04-22T23:14:12.878 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.859+0000 7f108d4f7700 1 --2- 172.21.15.67:0/3806457508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1088106560 0x7f1088106980 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:12.879 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.859+0000 7f108d4f7700 1 -- 172.21.15.67:0/3806457508 >> 172.21.15.67:0/3806457508 conn(0x7f1088101b40 msgr2=0x7f1088103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:12.879 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.859+0000 7f108d4f7700 1 -- 172.21.15.67:0/3806457508 shutdown_connections 2024-04-22T23:14:12.879 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.860+0000 7f108d4f7700 1 -- 172.21.15.67:0/3806457508 wait complete. 2024-04-22T23:14:12.879 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.860+0000 7f108d4f7700 1 Processor -- start 2024-04-22T23:14:12.879 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.860+0000 7f108d4f7700 1 -- start start 2024-04-22T23:14:12.879 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.860+0000 7f108d4f7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1088106560 0x7f108819a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:12.879 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.860+0000 7f108d4f7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f108819a780 con 0x7f1088106560 2024-04-22T23:14:12.879 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.861+0000 7f1086ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1088106560 0x7f108819a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:12.879 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.861+0000 7f1086ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1088106560 0x7f108819a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36462/0 (socket says 172.21.15.67:36462) 2024-04-22T23:14:12.879 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.861+0000 7f1086ffd700 1 -- 172.21.15.67:0/3193796321 learned_addr learned my addr 172.21.15.67:0/3193796321 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:12.879 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.861+0000 7f1086ffd700 1 -- 172.21.15.67:0/3193796321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1078009550 con 0x7f1088106560 2024-04-22T23:14:12.879 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.861+0000 7f1086ffd700 1 --2- 172.21.15.67:0/3193796321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1088106560 0x7f108819a240 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1078011670 tx=0x7f1078011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:12.879 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.861+0000 7f106ffff700 1 -- 172.21.15.67:0/3193796321 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10780039c0 con 0x7f1088106560 2024-04-22T23:14:12.879 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.861+0000 7f106ffff700 1 -- 172.21.15.67:0/3193796321 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1078003b20 con 0x7f1088106560 2024-04-22T23:14:12.879 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.861+0000 7f108d4f7700 1 -- 172.21.15.67:0/3193796321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f108819a980 con 0x7f1088106560 2024-04-22T23:14:12.880 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.861+0000 7f106ffff700 1 -- 172.21.15.67:0/3193796321 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10780247f0 con 0x7f1088106560 2024-04-22T23:14:12.880 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.861+0000 7f108d4f7700 1 -- 172.21.15.67:0/3193796321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f108819ae20 con 0x7f1088106560 2024-04-22T23:14:12.880 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.862+0000 7f106ffff700 1 -- 172.21.15.67:0/3193796321 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f1078024ab0 con 0x7f1088106560 2024-04-22T23:14:12.880 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.862+0000 7f108d4f7700 1 -- 172.21.15.67:0/3193796321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10880623c0 con 0x7f1088106560 2024-04-22T23:14:12.880 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.862+0000 7f106ffff700 1 --2- 172.21.15.67:0/3193796321 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f10700382e0 0x7f107003a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:12.880 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.862+0000 7f106ffff700 1 -- 172.21.15.67:0/3193796321 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f107804c420 con 0x7f1088106560 2024-04-22T23:14:12.880 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.862+0000 7f10867fc700 1 --2- 172.21.15.67:0/3193796321 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f10700382e0 0x7f107003a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:12.880 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.863+0000 7f10867fc700 1 --2- 172.21.15.67:0/3193796321 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f10700382e0 0x7f107003a7a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f107c006fd0 tx=0x7f107c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:12.880 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:11.867+0000 7f106ffff700 1 -- 172.21.15.67:0/3193796321 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f1078024d60 con 0x7f1088106560 2024-04-22T23:14:12.880 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:12.087+0000 7f108d4f7700 1 -- 172.21.15.67:0/3193796321 --> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f108810a960 con 0x7f10700382e0 2024-04-22T23:14:12.880 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:12.091+0000 7f106ffff700 1 -- 172.21.15.67:0/3193796321 <== mgr.14120 v2:172.21.15.67:6800/3568049827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f108810a960 con 0x7f10700382e0 2024-04-22T23:14:12.880 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:12.093+0000 7f108d4f7700 1 -- 172.21.15.67:0/3193796321 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f10700382e0 msgr2=0x7f107003a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:12.880 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:12.093+0000 7f108d4f7700 1 --2- 172.21.15.67:0/3193796321 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f10700382e0 0x7f107003a7a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f107c006fd0 tx=0x7f107c006e40 comp rx=0 tx=0).stop 2024-04-22T23:14:12.880 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:12.093+0000 7f108d4f7700 1 -- 172.21.15.67:0/3193796321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1088106560 msgr2=0x7f108819a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:12.881 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:12.093+0000 7f108d4f7700 1 --2- 172.21.15.67:0/3193796321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1088106560 0x7f108819a240 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1078011670 tx=0x7f1078011aa0 comp rx=0 tx=0).stop 2024-04-22T23:14:12.881 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:12.093+0000 7f108d4f7700 1 -- 172.21.15.67:0/3193796321 shutdown_connections 2024-04-22T23:14:12.881 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:12.093+0000 7f108d4f7700 1 --2- 172.21.15.67:0/3193796321 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f10700382e0 0x7f107003a7a0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:12.881 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:12.093+0000 7f108d4f7700 1 --2- 172.21.15.67:0/3193796321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1088106560 0x7f108819a240 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:12.881 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:12.093+0000 7f108d4f7700 1 -- 172.21.15.67:0/3193796321 >> 172.21.15.67:0/3193796321 conn(0x7f1088101b40 msgr2=0x7f1088102920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:12.881 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:12.093+0000 7f108d4f7700 1 -- 172.21.15.67:0/3193796321 shutdown_connections 2024-04-22T23:14:12.881 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:12.093+0000 7f108d4f7700 1 -- 172.21.15.67:0/3193796321 wait complete. 2024-04-22T23:14:12.881 INFO:teuthology.orchestra.run.smithi067.stdout:Deploying prometheus service with default placement... 2024-04-22T23:14:13.122 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:13 smithi067 ceph-mon[26453]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:14:13.122 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:13 smithi067 ceph-mon[26453]: Saving service ceph-exporter spec with placement * 2024-04-22T23:14:13.122 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:13 smithi067 ceph-mon[26453]: from='mgr.14120 172.21.15.67:0/1431842859' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:14.168 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-04-22T23:14:14.168 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.317+0000 7fed97f84700 1 Processor -- start 2024-04-22T23:14:14.168 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.317+0000 7fed97f84700 1 -- start start 2024-04-22T23:14:14.168 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.317+0000 7fed97f84700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed9007be80 0x7fed9007a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:14.168 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.317+0000 7fed97f84700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9007a8b0 con 0x7fed9007be80 2024-04-22T23:14:14.168 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.317+0000 7fed95d20700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed9007be80 0x7fed9007a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:14.168 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.317+0000 7fed95d20700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed9007be80 0x7fed9007a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36472/0 (socket says 172.21.15.67:36472) 2024-04-22T23:14:14.168 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.317+0000 7fed95d20700 1 -- 172.21.15.67:0/1347094703 learned_addr learned my addr 172.21.15.67:0/1347094703 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:14.168 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.317+0000 7fed95d20700 1 -- 172.21.15.67:0/1347094703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed9007a9f0 con 0x7fed9007be80 2024-04-22T23:14:14.168 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.318+0000 7fed95d20700 1 --2- 172.21.15.67:0/1347094703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed9007be80 0x7fed9007a2e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fed8c0098a0 tx=0x7fed8c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8715bed2acf13674 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:14.168 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.318+0000 7fed94d1e700 1 -- 172.21.15.67:0/1347094703 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed8c004180 con 0x7fed9007be80 2024-04-22T23:14:14.169 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.318+0000 7fed94d1e700 1 -- 172.21.15.67:0/1347094703 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fed8c0042e0 con 0x7fed9007be80 2024-04-22T23:14:14.169 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.318+0000 7fed94d1e700 1 -- 172.21.15.67:0/1347094703 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed8c011af0 con 0x7fed9007be80 2024-04-22T23:14:14.169 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.318+0000 7fed97f84700 1 -- 172.21.15.67:0/1347094703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed9007be80 msgr2=0x7fed9007a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:14.169 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.318+0000 7fed97f84700 1 --2- 172.21.15.67:0/1347094703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed9007be80 0x7fed9007a2e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fed8c0098a0 tx=0x7fed8c009bb0 comp rx=0 tx=0).stop 2024-04-22T23:14:14.169 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.318+0000 7fed97f84700 1 -- 172.21.15.67:0/1347094703 shutdown_connections 2024-04-22T23:14:14.169 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.318+0000 7fed97f84700 1 --2- 172.21.15.67:0/1347094703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed9007be80 0x7fed9007a2e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:14.169 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.318+0000 7fed97f84700 1 -- 172.21.15.67:0/1347094703 >> 172.21.15.67:0/1347094703 conn(0x7fed90101b40 msgr2=0x7fed90103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:14.169 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.319+0000 7fed97f84700 1 -- 172.21.15.67:0/1347094703 shutdown_connections 2024-04-22T23:14:14.169 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.319+0000 7fed97f84700 1 -- 172.21.15.67:0/1347094703 wait complete. 2024-04-22T23:14:14.169 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.319+0000 7fed97f84700 1 Processor -- start 2024-04-22T23:14:14.169 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.319+0000 7fed97f84700 1 -- start start 2024-04-22T23:14:14.169 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.319+0000 7fed97f84700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed9007be80 0x7fed9019c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:14.169 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.319+0000 7fed97f84700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed9019c9a0 con 0x7fed9007be80 2024-04-22T23:14:14.169 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.319+0000 7fed95d20700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed9007be80 0x7fed9019c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:14.170 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.319+0000 7fed95d20700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed9007be80 0x7fed9019c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36484/0 (socket says 172.21.15.67:36484) 2024-04-22T23:14:14.170 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.319+0000 7fed95d20700 1 -- 172.21.15.67:0/3976226775 learned_addr learned my addr 172.21.15.67:0/3976226775 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:14.170 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.320+0000 7fed95d20700 1 -- 172.21.15.67:0/3976226775 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed8c009550 con 0x7fed9007be80 2024-04-22T23:14:14.170 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.320+0000 7fed95d20700 1 --2- 172.21.15.67:0/3976226775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed9007be80 0x7fed9019c460 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fed8c011670 tx=0x7fed8c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:14.170 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.320+0000 7fed86ffd700 1 -- 172.21.15.67:0/3976226775 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed8c0039c0 con 0x7fed9007be80 2024-04-22T23:14:14.170 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.320+0000 7fed86ffd700 1 -- 172.21.15.67:0/3976226775 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fed8c003b20 con 0x7fed9007be80 2024-04-22T23:14:14.170 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.320+0000 7fed97f84700 1 -- 172.21.15.67:0/3976226775 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed9019cba0 con 0x7fed9007be80 2024-04-22T23:14:14.170 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.320+0000 7fed86ffd700 1 -- 172.21.15.67:0/3976226775 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed8c0247f0 con 0x7fed9007be80 2024-04-22T23:14:14.170 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.320+0000 7fed97f84700 1 -- 172.21.15.67:0/3976226775 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed9019d040 con 0x7fed9007be80 2024-04-22T23:14:14.170 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.321+0000 7fed86ffd700 1 -- 172.21.15.67:0/3976226775 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fed8c024ab0 con 0x7fed9007be80 2024-04-22T23:14:14.170 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.321+0000 7fed86ffd700 1 --2- 172.21.15.67:0/3976226775 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fed7c0382e0 0x7fed7c03a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:14.170 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.321+0000 7fed97f84700 1 -- 172.21.15.67:0/3976226775 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed9010c920 con 0x7fed9007be80 2024-04-22T23:14:14.170 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.321+0000 7fed9551f700 1 --2- 172.21.15.67:0/3976226775 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fed7c0382e0 0x7fed7c03a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:14.171 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.321+0000 7fed86ffd700 1 -- 172.21.15.67:0/3976226775 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fed8c04c2e0 con 0x7fed9007be80 2024-04-22T23:14:14.171 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.321+0000 7fed9551f700 1 --2- 172.21.15.67:0/3976226775 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fed7c0382e0 0x7fed7c03a7a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fed80006fd0 tx=0x7fed80006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:14.171 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.326+0000 7fed86ffd700 1 -- 172.21.15.67:0/3976226775 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fed8c018740 con 0x7fed9007be80 2024-04-22T23:14:14.171 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.539+0000 7fed97f84700 1 -- 172.21.15.67:0/3976226775 --> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fed90103e00 con 0x7fed7c0382e0 2024-04-22T23:14:14.171 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.544+0000 7fed86ffd700 1 -- 172.21.15.67:0/3976226775 <== mgr.14120 v2:172.21.15.67:6800/3568049827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fed90103e00 con 0x7fed7c0382e0 2024-04-22T23:14:14.171 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.546+0000 7fed97f84700 1 -- 172.21.15.67:0/3976226775 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fed7c0382e0 msgr2=0x7fed7c03a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:14.171 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.546+0000 7fed97f84700 1 --2- 172.21.15.67:0/3976226775 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fed7c0382e0 0x7fed7c03a7a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fed80006fd0 tx=0x7fed80006e40 comp rx=0 tx=0).stop 2024-04-22T23:14:14.171 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.546+0000 7fed97f84700 1 -- 172.21.15.67:0/3976226775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed9007be80 msgr2=0x7fed9019c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:14.171 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.546+0000 7fed97f84700 1 --2- 172.21.15.67:0/3976226775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed9007be80 0x7fed9019c460 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fed8c011670 tx=0x7fed8c011aa0 comp rx=0 tx=0).stop 2024-04-22T23:14:14.171 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.546+0000 7fed97f84700 1 -- 172.21.15.67:0/3976226775 shutdown_connections 2024-04-22T23:14:14.171 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.546+0000 7fed97f84700 1 --2- 172.21.15.67:0/3976226775 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fed7c0382e0 0x7fed7c03a7a0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:14.171 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.546+0000 7fed97f84700 1 --2- 172.21.15.67:0/3976226775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed9007be80 0x7fed9019c460 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:14.171 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.546+0000 7fed97f84700 1 -- 172.21.15.67:0/3976226775 >> 172.21.15.67:0/3976226775 conn(0x7fed90101b40 msgr2=0x7fed901031a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:14.171 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.546+0000 7fed97f84700 1 -- 172.21.15.67:0/3976226775 shutdown_connections 2024-04-22T23:14:14.172 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:13.546+0000 7fed97f84700 1 -- 172.21.15.67:0/3976226775 wait complete. 2024-04-22T23:14:14.172 INFO:teuthology.orchestra.run.smithi067.stdout:Deploying grafana service with default placement... 2024-04-22T23:14:14.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:14 smithi067 ceph-mon[26453]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:14:14.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:14 smithi067 ceph-mon[26453]: Saving service prometheus spec with placement count:1 2024-04-22T23:14:14.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:14 smithi067 ceph-mon[26453]: from='mgr.14120 172.21.15.67:0/1431842859' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:14.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:14 smithi067 ceph-mon[26453]: from='mgr.14120 172.21.15.67:0/1431842859' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:15.490 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-04-22T23:14:15.491 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.637+0000 7f8703d27700 1 Processor -- start 2024-04-22T23:14:15.491 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.637+0000 7f8703d27700 1 -- start start 2024-04-22T23:14:15.491 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.637+0000 7f8703d27700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86fc106560 0x7f86fc106980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:15.491 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.637+0000 7f8703d27700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86fc106f50 con 0x7f86fc106560 2024-04-22T23:14:15.491 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.638+0000 7f8701ac3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86fc106560 0x7f86fc106980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:15.491 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.638+0000 7f8701ac3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86fc106560 0x7f86fc106980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36488/0 (socket says 172.21.15.67:36488) 2024-04-22T23:14:15.491 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.638+0000 7f8701ac3700 1 -- 172.21.15.67:0/3365375238 learned_addr learned my addr 172.21.15.67:0/3365375238 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:15.491 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.638+0000 7f8701ac3700 1 -- 172.21.15.67:0/3365375238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86fc107760 con 0x7f86fc106560 2024-04-22T23:14:15.491 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.638+0000 7f8701ac3700 1 --2- 172.21.15.67:0/3365375238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86fc106560 0x7f86fc106980 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f86f80098a0 tx=0x7f86f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69417fd5f15b2a12 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:15.491 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.638+0000 7f8700ac1700 1 -- 172.21.15.67:0/3365375238 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86f8004180 con 0x7f86fc106560 2024-04-22T23:14:15.491 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.638+0000 7f8700ac1700 1 -- 172.21.15.67:0/3365375238 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f86f80042e0 con 0x7f86fc106560 2024-04-22T23:14:15.491 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.638+0000 7f8700ac1700 1 -- 172.21.15.67:0/3365375238 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86f8011af0 con 0x7f86fc106560 2024-04-22T23:14:15.491 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.639+0000 7f8703d27700 1 -- 172.21.15.67:0/3365375238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86fc106560 msgr2=0x7f86fc106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:15.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.639+0000 7f8703d27700 1 --2- 172.21.15.67:0/3365375238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86fc106560 0x7f86fc106980 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f86f80098a0 tx=0x7f86f8009bb0 comp rx=0 tx=0).stop 2024-04-22T23:14:15.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.639+0000 7f8703d27700 1 -- 172.21.15.67:0/3365375238 shutdown_connections 2024-04-22T23:14:15.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.639+0000 7f8703d27700 1 --2- 172.21.15.67:0/3365375238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86fc106560 0x7f86fc106980 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:15.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.639+0000 7f8703d27700 1 -- 172.21.15.67:0/3365375238 >> 172.21.15.67:0/3365375238 conn(0x7f86fc101b40 msgr2=0x7f86fc103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:15.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.639+0000 7f8703d27700 1 -- 172.21.15.67:0/3365375238 shutdown_connections 2024-04-22T23:14:15.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.639+0000 7f8703d27700 1 -- 172.21.15.67:0/3365375238 wait complete. 2024-04-22T23:14:15.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.640+0000 7f8703d27700 1 Processor -- start 2024-04-22T23:14:15.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.640+0000 7f8703d27700 1 -- start start 2024-04-22T23:14:15.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.640+0000 7f8703d27700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86fc106560 0x7f86fc07bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:15.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.640+0000 7f8703d27700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86fc07c170 con 0x7f86fc106560 2024-04-22T23:14:15.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.640+0000 7f8701ac3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86fc106560 0x7f86fc07bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:15.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.640+0000 7f8701ac3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86fc106560 0x7f86fc07bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36490/0 (socket says 172.21.15.67:36490) 2024-04-22T23:14:15.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.640+0000 7f8701ac3700 1 -- 172.21.15.67:0/1609239772 learned_addr learned my addr 172.21.15.67:0/1609239772 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:15.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.641+0000 7f8701ac3700 1 -- 172.21.15.67:0/1609239772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f8009550 con 0x7f86fc106560 2024-04-22T23:14:15.492 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.641+0000 7f8701ac3700 1 --2- 172.21.15.67:0/1609239772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86fc106560 0x7f86fc07bc30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f86f8000c00 tx=0x7f86f8011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:15.493 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.641+0000 7f86f2ffd700 1 -- 172.21.15.67:0/1609239772 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86f8003a70 con 0x7f86fc106560 2024-04-22T23:14:15.493 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.641+0000 7f8703d27700 1 -- 172.21.15.67:0/1609239772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86fc07a280 con 0x7f86fc106560 2024-04-22T23:14:15.493 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.641+0000 7f86f2ffd700 1 -- 172.21.15.67:0/1609239772 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f86f8003bd0 con 0x7f86fc106560 2024-04-22T23:14:15.493 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.641+0000 7f86f2ffd700 1 -- 172.21.15.67:0/1609239772 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86f80238c0 con 0x7f86fc106560 2024-04-22T23:14:15.493 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.641+0000 7f8703d27700 1 -- 172.21.15.67:0/1609239772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86fc07a720 con 0x7f86fc106560 2024-04-22T23:14:15.493 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.642+0000 7f8703d27700 1 -- 172.21.15.67:0/1609239772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86fc0623c0 con 0x7f86fc106560 2024-04-22T23:14:15.493 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.642+0000 7f86f2ffd700 1 -- 172.21.15.67:0/1609239772 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f86f8023b80 con 0x7f86fc106560 2024-04-22T23:14:15.493 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.642+0000 7f86f2ffd700 1 --2- 172.21.15.67:0/1609239772 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f86e8038320 0x7f86e803a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:15.493 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.642+0000 7f86f2ffd700 1 -- 172.21.15.67:0/1609239772 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f86f804c350 con 0x7f86fc106560 2024-04-22T23:14:15.493 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.643+0000 7f87012c2700 1 --2- 172.21.15.67:0/1609239772 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f86e8038320 0x7f86e803a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:15.493 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.643+0000 7f87012c2700 1 --2- 172.21.15.67:0/1609239772 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f86e8038320 0x7f86e803a7e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f86ec006fd0 tx=0x7f86ec006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:15.493 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.648+0000 7f86f2ffd700 1 -- 172.21.15.67:0/1609239772 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f86f8010d10 con 0x7f86fc106560 2024-04-22T23:14:15.493 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.903+0000 7f8703d27700 1 -- 172.21.15.67:0/1609239772 --> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f86fc102fa0 con 0x7f86e8038320 2024-04-22T23:14:15.493 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.909+0000 7f86f2ffd700 1 -- 172.21.15.67:0/1609239772 <== mgr.14120 v2:172.21.15.67:6800/3568049827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f86fc102fa0 con 0x7f86e8038320 2024-04-22T23:14:15.494 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.911+0000 7f8703d27700 1 -- 172.21.15.67:0/1609239772 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f86e8038320 msgr2=0x7f86e803a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:15.494 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.911+0000 7f8703d27700 1 --2- 172.21.15.67:0/1609239772 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f86e8038320 0x7f86e803a7e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f86ec006fd0 tx=0x7f86ec006e40 comp rx=0 tx=0).stop 2024-04-22T23:14:15.494 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.911+0000 7f8703d27700 1 -- 172.21.15.67:0/1609239772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86fc106560 msgr2=0x7f86fc07bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:15.494 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.911+0000 7f8703d27700 1 --2- 172.21.15.67:0/1609239772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86fc106560 0x7f86fc07bc30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f86f8000c00 tx=0x7f86f8011b50 comp rx=0 tx=0).stop 2024-04-22T23:14:15.494 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.911+0000 7f8703d27700 1 -- 172.21.15.67:0/1609239772 shutdown_connections 2024-04-22T23:14:15.494 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.911+0000 7f8703d27700 1 --2- 172.21.15.67:0/1609239772 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f86e8038320 0x7f86e803a7e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:15.494 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.911+0000 7f8703d27700 1 --2- 172.21.15.67:0/1609239772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86fc106560 0x7f86fc07bc30 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:15.494 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.911+0000 7f8703d27700 1 -- 172.21.15.67:0/1609239772 >> 172.21.15.67:0/1609239772 conn(0x7f86fc101b40 msgr2=0x7f86fc102890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:15.494 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.912+0000 7f8703d27700 1 -- 172.21.15.67:0/1609239772 shutdown_connections 2024-04-22T23:14:15.494 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:14.912+0000 7f8703d27700 1 -- 172.21.15.67:0/1609239772 wait complete. 2024-04-22T23:14:15.494 INFO:teuthology.orchestra.run.smithi067.stdout:Deploying node-exporter service with default placement... 2024-04-22T23:14:15.794 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:15 smithi067 ceph-mon[26453]: from='mgr.14120 172.21.15.67:0/1431842859' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:15.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:15 smithi067 ceph-mon[26453]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:14:15.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:15 smithi067 ceph-mon[26453]: Saving service grafana spec with placement count:1 2024-04-22T23:14:15.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:15 smithi067 ceph-mon[26453]: from='mgr.14120 172.21.15.67:0/1431842859' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:17.082 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-04-22T23:14:17.082 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.942+0000 7ff9ee1f8700 1 Processor -- start 2024-04-22T23:14:17.082 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.942+0000 7ff9ee1f8700 1 -- start start 2024-04-22T23:14:17.082 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.942+0000 7ff9ee1f8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9e8107000 0x7ff9e81093f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:17.083 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.942+0000 7ff9ee1f8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e8079e30 con 0x7ff9e8107000 2024-04-22T23:14:17.083 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.942+0000 7ff9e77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9e8107000 0x7ff9e81093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:17.083 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.942+0000 7ff9e77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9e8107000 0x7ff9e81093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36494/0 (socket says 172.21.15.67:36494) 2024-04-22T23:14:17.083 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.942+0000 7ff9e77fe700 1 -- 172.21.15.67:0/3634161369 learned_addr learned my addr 172.21.15.67:0/3634161369 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:17.083 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.943+0000 7ff9e77fe700 1 -- 172.21.15.67:0/3634161369 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e8109930 con 0x7ff9e8107000 2024-04-22T23:14:17.083 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.943+0000 7ff9e77fe700 1 --2- 172.21.15.67:0/3634161369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9e8107000 0x7ff9e81093f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff9d80098a0 tx=0x7ff9d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ed8b885b112fe90 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:17.083 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.943+0000 7ff9e67fc700 1 -- 172.21.15.67:0/3634161369 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9d8004180 con 0x7ff9e8107000 2024-04-22T23:14:17.084 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.943+0000 7ff9e67fc700 1 -- 172.21.15.67:0/3634161369 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff9d80042e0 con 0x7ff9e8107000 2024-04-22T23:14:17.084 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.943+0000 7ff9e67fc700 1 -- 172.21.15.67:0/3634161369 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9d8011af0 con 0x7ff9e8107000 2024-04-22T23:14:17.084 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.944+0000 7ff9ee1f8700 1 -- 172.21.15.67:0/3634161369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9e8107000 msgr2=0x7ff9e81093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:17.084 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.944+0000 7ff9ee1f8700 1 --2- 172.21.15.67:0/3634161369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9e8107000 0x7ff9e81093f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff9d80098a0 tx=0x7ff9d8009bb0 comp rx=0 tx=0).stop 2024-04-22T23:14:17.084 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.944+0000 7ff9ee1f8700 1 -- 172.21.15.67:0/3634161369 shutdown_connections 2024-04-22T23:14:17.085 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.944+0000 7ff9ee1f8700 1 --2- 172.21.15.67:0/3634161369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9e8107000 0x7ff9e81093f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:17.085 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.944+0000 7ff9ee1f8700 1 -- 172.21.15.67:0/3634161369 >> 172.21.15.67:0/3634161369 conn(0x7ff9e81009b0 msgr2=0x7ff9e8102e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:17.085 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.944+0000 7ff9ee1f8700 1 -- 172.21.15.67:0/3634161369 shutdown_connections 2024-04-22T23:14:17.085 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.944+0000 7ff9ee1f8700 1 -- 172.21.15.67:0/3634161369 wait complete. 2024-04-22T23:14:17.085 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.945+0000 7ff9ee1f8700 1 Processor -- start 2024-04-22T23:14:17.085 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.945+0000 7ff9ee1f8700 1 -- start start 2024-04-22T23:14:17.085 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.945+0000 7ff9ee1f8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9e8107000 0x7ff9e819a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:17.085 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.945+0000 7ff9ee1f8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e819a6e0 con 0x7ff9e8107000 2024-04-22T23:14:17.085 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.946+0000 7ff9e77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9e8107000 0x7ff9e819a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:17.085 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.946+0000 7ff9e77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9e8107000 0x7ff9e819a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36508/0 (socket says 172.21.15.67:36508) 2024-04-22T23:14:17.085 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.946+0000 7ff9e77fe700 1 -- 172.21.15.67:0/2238328403 learned_addr learned my addr 172.21.15.67:0/2238328403 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:17.085 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.946+0000 7ff9e77fe700 1 -- 172.21.15.67:0/2238328403 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d8009550 con 0x7ff9e8107000 2024-04-22T23:14:17.085 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.946+0000 7ff9e77fe700 1 --2- 172.21.15.67:0/2238328403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9e8107000 0x7ff9e819a1a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8011ad0 tx=0x7ff9d8011bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:17.086 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.946+0000 7ff9e4ff9700 1 -- 172.21.15.67:0/2238328403 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9d8003ab0 con 0x7ff9e8107000 2024-04-22T23:14:17.086 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.946+0000 7ff9e4ff9700 1 -- 172.21.15.67:0/2238328403 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff9d8003c10 con 0x7ff9e8107000 2024-04-22T23:14:17.086 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.946+0000 7ff9ee1f8700 1 -- 172.21.15.67:0/2238328403 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e819a8e0 con 0x7ff9e8107000 2024-04-22T23:14:17.086 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.946+0000 7ff9e4ff9700 1 -- 172.21.15.67:0/2238328403 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9d80198e0 con 0x7ff9e8107000 2024-04-22T23:14:17.086 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.946+0000 7ff9ee1f8700 1 -- 172.21.15.67:0/2238328403 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e819ad80 con 0x7ff9e8107000 2024-04-22T23:14:17.086 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.947+0000 7ff9e4ff9700 1 -- 172.21.15.67:0/2238328403 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7ff9d8019b50 con 0x7ff9e8107000 2024-04-22T23:14:17.086 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.947+0000 7ff9ee1f8700 1 -- 172.21.15.67:0/2238328403 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e8062430 con 0x7ff9e8107000 2024-04-22T23:14:17.086 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.948+0000 7ff9e4ff9700 1 --2- 172.21.15.67:0/2238328403 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7ff9d0038370 0x7ff9d003a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:17.086 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.948+0000 7ff9e4ff9700 1 -- 172.21.15.67:0/2238328403 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff9d804d240 con 0x7ff9e8107000 2024-04-22T23:14:17.086 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.948+0000 7ff9e6ffd700 1 --2- 172.21.15.67:0/2238328403 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7ff9d0038370 0x7ff9d003a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:17.086 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.948+0000 7ff9e6ffd700 1 --2- 172.21.15.67:0/2238328403 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7ff9d0038370 0x7ff9d003a830 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc006fd0 tx=0x7ff9dc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:17.086 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:15.953+0000 7ff9e4ff9700 1 -- 172.21.15.67:0/2238328403 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff9e8062430 con 0x7ff9e8107000 2024-04-22T23:14:17.086 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:16.168+0000 7ff9ee1f8700 1 -- 172.21.15.67:0/2238328403 --> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7ff9e8062430 con 0x7ff9d0038370 2024-04-22T23:14:17.086 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:16.173+0000 7ff9e4ff9700 1 -- 172.21.15.67:0/2238328403 <== mgr.14120 v2:172.21.15.67:6800/3568049827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7ff9e8062430 con 0x7ff9d0038370 2024-04-22T23:14:17.087 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:16.175+0000 7ff9ee1f8700 1 -- 172.21.15.67:0/2238328403 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7ff9d0038370 msgr2=0x7ff9d003a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:17.087 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:16.175+0000 7ff9ee1f8700 1 --2- 172.21.15.67:0/2238328403 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7ff9d0038370 0x7ff9d003a830 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc006fd0 tx=0x7ff9dc006e40 comp rx=0 tx=0).stop 2024-04-22T23:14:17.087 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:16.175+0000 7ff9ee1f8700 1 -- 172.21.15.67:0/2238328403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9e8107000 msgr2=0x7ff9e819a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:17.087 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:16.175+0000 7ff9ee1f8700 1 --2- 172.21.15.67:0/2238328403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9e8107000 0x7ff9e819a1a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8011ad0 tx=0x7ff9d8011bb0 comp rx=0 tx=0).stop 2024-04-22T23:14:17.087 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:16.175+0000 7ff9ee1f8700 1 -- 172.21.15.67:0/2238328403 shutdown_connections 2024-04-22T23:14:17.087 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:16.176+0000 7ff9ee1f8700 1 --2- 172.21.15.67:0/2238328403 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7ff9d0038370 0x7ff9d003a830 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:17.087 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:16.176+0000 7ff9ee1f8700 1 --2- 172.21.15.67:0/2238328403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9e8107000 0x7ff9e819a1a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:17.087 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:16.176+0000 7ff9ee1f8700 1 -- 172.21.15.67:0/2238328403 >> 172.21.15.67:0/2238328403 conn(0x7ff9e81009b0 msgr2=0x7ff9e8101720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:17.087 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:16.176+0000 7ff9ee1f8700 1 -- 172.21.15.67:0/2238328403 shutdown_connections 2024-04-22T23:14:17.087 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:16.176+0000 7ff9ee1f8700 1 -- 172.21.15.67:0/2238328403 wait complete. 2024-04-22T23:14:17.087 INFO:teuthology.orchestra.run.smithi067.stdout:Deploying alertmanager service with default placement... 2024-04-22T23:14:17.396 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:17 smithi067 ceph-mon[26453]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:14:17.396 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:17 smithi067 ceph-mon[26453]: Saving service node-exporter spec with placement * 2024-04-22T23:14:17.397 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:17 smithi067 ceph-mon[26453]: from='mgr.14120 172.21.15.67:0/1431842859' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:17.397 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:17 smithi067 ceph-mon[26453]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:14:18.447 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-04-22T23:14:18.447 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.507+0000 7ff2843ca700 1 Processor -- start 2024-04-22T23:14:18.447 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.508+0000 7ff2843ca700 1 -- start start 2024-04-22T23:14:18.447 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.508+0000 7ff2843ca700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff27c104840 0x7ff27c104c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:18.447 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.508+0000 7ff2843ca700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27c105230 con 0x7ff27c104840 2024-04-22T23:14:18.447 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.508+0000 7ff282166700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff27c104840 0x7ff27c104c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:18.447 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.508+0000 7ff282166700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff27c104840 0x7ff27c104c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36524/0 (socket says 172.21.15.67:36524) 2024-04-22T23:14:18.447 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.508+0000 7ff282166700 1 -- 172.21.15.67:0/908129807 learned_addr learned my addr 172.21.15.67:0/908129807 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:18.447 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.508+0000 7ff282166700 1 -- 172.21.15.67:0/908129807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff27c1077d0 con 0x7ff27c104840 2024-04-22T23:14:18.447 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.509+0000 7ff282166700 1 --2- 172.21.15.67:0/908129807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff27c104840 0x7ff27c104c60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff2780098a0 tx=0x7ff278009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93c20b5181e45d2a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:18.447 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.509+0000 7ff281164700 1 -- 172.21.15.67:0/908129807 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff278004180 con 0x7ff27c104840 2024-04-22T23:14:18.448 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.509+0000 7ff281164700 1 -- 172.21.15.67:0/908129807 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff2780042e0 con 0x7ff27c104840 2024-04-22T23:14:18.448 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.509+0000 7ff281164700 1 -- 172.21.15.67:0/908129807 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff278011af0 con 0x7ff27c104840 2024-04-22T23:14:18.448 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.509+0000 7ff2843ca700 1 -- 172.21.15.67:0/908129807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff27c104840 msgr2=0x7ff27c104c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:18.448 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.509+0000 7ff2843ca700 1 --2- 172.21.15.67:0/908129807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff27c104840 0x7ff27c104c60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff2780098a0 tx=0x7ff278009bb0 comp rx=0 tx=0).stop 2024-04-22T23:14:18.448 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.509+0000 7ff2843ca700 1 -- 172.21.15.67:0/908129807 shutdown_connections 2024-04-22T23:14:18.448 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.509+0000 7ff2843ca700 1 --2- 172.21.15.67:0/908129807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff27c104840 0x7ff27c104c60 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:18.448 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.509+0000 7ff2843ca700 1 -- 172.21.15.67:0/908129807 >> 172.21.15.67:0/908129807 conn(0x7ff27c100170 msgr2=0x7ff27c1025b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:18.448 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.510+0000 7ff2843ca700 1 -- 172.21.15.67:0/908129807 shutdown_connections 2024-04-22T23:14:18.448 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.510+0000 7ff2843ca700 1 -- 172.21.15.67:0/908129807 wait complete. 2024-04-22T23:14:18.448 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.510+0000 7ff2843ca700 1 Processor -- start 2024-04-22T23:14:18.448 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.510+0000 7ff2843ca700 1 -- start start 2024-04-22T23:14:18.448 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.510+0000 7ff2843ca700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff27c104840 0x7ff27c19c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:18.448 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.510+0000 7ff2843ca700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27c19c9d0 con 0x7ff27c104840 2024-04-22T23:14:18.449 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.510+0000 7ff282166700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff27c104840 0x7ff27c19c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:18.449 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.510+0000 7ff282166700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff27c104840 0x7ff27c19c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36536/0 (socket says 172.21.15.67:36536) 2024-04-22T23:14:18.449 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.510+0000 7ff282166700 1 -- 172.21.15.67:0/1888747118 learned_addr learned my addr 172.21.15.67:0/1888747118 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:18.449 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.511+0000 7ff282166700 1 -- 172.21.15.67:0/1888747118 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff278009550 con 0x7ff27c104840 2024-04-22T23:14:18.449 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.511+0000 7ff282166700 1 --2- 172.21.15.67:0/1888747118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff27c104840 0x7ff27c19c490 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff278011670 tx=0x7ff278011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:18.449 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.511+0000 7ff26f7fe700 1 -- 172.21.15.67:0/1888747118 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2780039f0 con 0x7ff27c104840 2024-04-22T23:14:18.449 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.511+0000 7ff26f7fe700 1 -- 172.21.15.67:0/1888747118 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff278003b50 con 0x7ff27c104840 2024-04-22T23:14:18.449 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.511+0000 7ff2843ca700 1 -- 172.21.15.67:0/1888747118 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff27c19cbd0 con 0x7ff27c104840 2024-04-22T23:14:18.449 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.511+0000 7ff26f7fe700 1 -- 172.21.15.67:0/1888747118 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2780247c0 con 0x7ff27c104840 2024-04-22T23:14:18.449 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.511+0000 7ff2843ca700 1 -- 172.21.15.67:0/1888747118 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff27c19d070 con 0x7ff27c104840 2024-04-22T23:14:18.449 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.512+0000 7ff26f7fe700 1 -- 172.21.15.67:0/1888747118 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7ff27802d3f0 con 0x7ff27c104840 2024-04-22T23:14:18.449 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.512+0000 7ff2843ca700 1 -- 172.21.15.67:0/1888747118 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff27c0623c0 con 0x7ff27c104840 2024-04-22T23:14:18.449 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.512+0000 7ff26f7fe700 1 --2- 172.21.15.67:0/1888747118 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7ff268038370 0x7ff26803a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:18.449 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.512+0000 7ff26f7fe700 1 -- 172.21.15.67:0/1888747118 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff27804c5b0 con 0x7ff27c104840 2024-04-22T23:14:18.450 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.512+0000 7ff281965700 1 --2- 172.21.15.67:0/1888747118 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7ff268038370 0x7ff26803a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:18.450 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.513+0000 7ff281965700 1 --2- 172.21.15.67:0/1888747118 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7ff268038370 0x7ff26803a830 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff270006fd0 tx=0x7ff270006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:18.450 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.517+0000 7ff26f7fe700 1 -- 172.21.15.67:0/1888747118 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff278010970 con 0x7ff27c104840 2024-04-22T23:14:18.450 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.737+0000 7ff2843ca700 1 -- 172.21.15.67:0/1888747118 --> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7ff27c10a9d0 con 0x7ff268038370 2024-04-22T23:14:18.450 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.741+0000 7ff26f7fe700 1 -- 172.21.15.67:0/1888747118 <== mgr.14120 v2:172.21.15.67:6800/3568049827 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7ff27c10a9d0 con 0x7ff268038370 2024-04-22T23:14:18.450 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.743+0000 7ff2843ca700 1 -- 172.21.15.67:0/1888747118 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7ff268038370 msgr2=0x7ff26803a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:18.450 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.743+0000 7ff2843ca700 1 --2- 172.21.15.67:0/1888747118 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7ff268038370 0x7ff26803a830 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff270006fd0 tx=0x7ff270006e40 comp rx=0 tx=0).stop 2024-04-22T23:14:18.450 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.743+0000 7ff2843ca700 1 -- 172.21.15.67:0/1888747118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff27c104840 msgr2=0x7ff27c19c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:18.450 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.743+0000 7ff2843ca700 1 --2- 172.21.15.67:0/1888747118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff27c104840 0x7ff27c19c490 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff278011670 tx=0x7ff278011b50 comp rx=0 tx=0).stop 2024-04-22T23:14:18.450 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.743+0000 7ff2843ca700 1 -- 172.21.15.67:0/1888747118 shutdown_connections 2024-04-22T23:14:18.450 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.743+0000 7ff2843ca700 1 --2- 172.21.15.67:0/1888747118 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7ff268038370 0x7ff26803a830 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:18.450 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.743+0000 7ff2843ca700 1 --2- 172.21.15.67:0/1888747118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff27c104840 0x7ff27c19c490 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:18.451 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.743+0000 7ff2843ca700 1 -- 172.21.15.67:0/1888747118 >> 172.21.15.67:0/1888747118 conn(0x7ff27c100170 msgr2=0x7ff27c1017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:18.451 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.743+0000 7ff2843ca700 1 -- 172.21.15.67:0/1888747118 shutdown_connections 2024-04-22T23:14:18.451 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:17.743+0000 7ff2843ca700 1 -- 172.21.15.67:0/1888747118 wait complete. 2024-04-22T23:14:18.846 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:18 smithi067 ceph-mon[26453]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:14:18.846 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:18 smithi067 ceph-mon[26453]: Saving service alertmanager spec with placement count:1 2024-04-22T23:14:18.846 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:18 smithi067 ceph-mon[26453]: from='mgr.14120 172.21.15.67:0/1431842859' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:18.846 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:18 smithi067 ceph-mon[26453]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:14:19.632 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.868+0000 7fa7d9dfe700 1 Processor -- start 2024-04-22T23:14:19.632 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.868+0000 7fa7d9dfe700 1 -- start start 2024-04-22T23:14:19.632 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.868+0000 7fa7d9dfe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7d4106fe0 0x7fa7d41093d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:19.632 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.868+0000 7fa7d9dfe700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d4074680 con 0x7fa7d4106fe0 2024-04-22T23:14:19.632 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.869+0000 7fa7d37fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7d4106fe0 0x7fa7d41093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:19.632 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.869+0000 7fa7d37fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7d4106fe0 0x7fa7d41093d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33810/0 (socket says 172.21.15.67:33810) 2024-04-22T23:14:19.632 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.869+0000 7fa7d37fe700 1 -- 172.21.15.67:0/2542721705 learned_addr learned my addr 172.21.15.67:0/2542721705 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:19.632 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.869+0000 7fa7d37fe700 1 -- 172.21.15.67:0/2542721705 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d4109910 con 0x7fa7d4106fe0 2024-04-22T23:14:19.632 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.869+0000 7fa7d37fe700 1 --2- 172.21.15.67:0/2542721705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7d4106fe0 0x7fa7d41093d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa7c40098a0 tx=0x7fa7c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5232177764b2dbf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:19.632 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.869+0000 7fa7d27fc700 1 -- 172.21.15.67:0/2542721705 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7c4004180 con 0x7fa7d4106fe0 2024-04-22T23:14:19.632 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.869+0000 7fa7d27fc700 1 -- 172.21.15.67:0/2542721705 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa7c40042e0 con 0x7fa7d4106fe0 2024-04-22T23:14:19.633 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.869+0000 7fa7d27fc700 1 -- 172.21.15.67:0/2542721705 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7c4011af0 con 0x7fa7d4106fe0 2024-04-22T23:14:19.633 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.870+0000 7fa7d9dfe700 1 -- 172.21.15.67:0/2542721705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7d4106fe0 msgr2=0x7fa7d41093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:19.633 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.870+0000 7fa7d9dfe700 1 --2- 172.21.15.67:0/2542721705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7d4106fe0 0x7fa7d41093d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa7c40098a0 tx=0x7fa7c4009bb0 comp rx=0 tx=0).stop 2024-04-22T23:14:19.633 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.870+0000 7fa7d9dfe700 1 -- 172.21.15.67:0/2542721705 shutdown_connections 2024-04-22T23:14:19.633 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.870+0000 7fa7d9dfe700 1 --2- 172.21.15.67:0/2542721705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7d4106fe0 0x7fa7d41093d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:19.633 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.870+0000 7fa7d9dfe700 1 -- 172.21.15.67:0/2542721705 >> 172.21.15.67:0/2542721705 conn(0x7fa7d41009f0 msgr2=0x7fa7d4102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:19.633 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.870+0000 7fa7d9dfe700 1 -- 172.21.15.67:0/2542721705 shutdown_connections 2024-04-22T23:14:19.633 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.870+0000 7fa7d9dfe700 1 -- 172.21.15.67:0/2542721705 wait complete. 2024-04-22T23:14:19.633 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.870+0000 7fa7d9dfe700 1 Processor -- start 2024-04-22T23:14:19.633 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.871+0000 7fa7d9dfe700 1 -- start start 2024-04-22T23:14:19.633 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.871+0000 7fa7d9dfe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7d4106fe0 0x7fa7d4106940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:19.633 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.871+0000 7fa7d9dfe700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d4106e80 con 0x7fa7d4106fe0 2024-04-22T23:14:19.633 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.871+0000 7fa7d37fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7d4106fe0 0x7fa7d4106940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:19.634 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.871+0000 7fa7d37fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7d4106fe0 0x7fa7d4106940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33824/0 (socket says 172.21.15.67:33824) 2024-04-22T23:14:19.634 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.871+0000 7fa7d37fe700 1 -- 172.21.15.67:0/3490084621 learned_addr learned my addr 172.21.15.67:0/3490084621 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:19.634 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.871+0000 7fa7d37fe700 1 -- 172.21.15.67:0/3490084621 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c4009550 con 0x7fa7d4106fe0 2024-04-22T23:14:19.634 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.871+0000 7fa7d37fe700 1 --2- 172.21.15.67:0/3490084621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7d4106fe0 0x7fa7d4106940 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa7c4006ba0 tx=0x7fa7c4011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:19.634 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.872+0000 7fa7d0ff9700 1 -- 172.21.15.67:0/3490084621 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7c4003b00 con 0x7fa7d4106fe0 2024-04-22T23:14:19.634 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.872+0000 7fa7d0ff9700 1 -- 172.21.15.67:0/3490084621 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa7c401a8e0 con 0x7fa7d4106fe0 2024-04-22T23:14:19.636 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.872+0000 7fa7d9dfe700 1 -- 172.21.15.67:0/3490084621 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7d4104f90 con 0x7fa7d4106fe0 2024-04-22T23:14:19.636 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.872+0000 7fa7d0ff9700 1 -- 172.21.15.67:0/3490084621 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7c40199c0 con 0x7fa7d4106fe0 2024-04-22T23:14:19.636 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.872+0000 7fa7d9dfe700 1 -- 172.21.15.67:0/3490084621 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7d4105430 con 0x7fa7d4106fe0 2024-04-22T23:14:19.636 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.872+0000 7fa7d0ff9700 1 -- 172.21.15.67:0/3490084621 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fa7c4019c30 con 0x7fa7d4106fe0 2024-04-22T23:14:19.636 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.873+0000 7fa7d0ff9700 1 --2- 172.21.15.67:0/3490084621 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fa7bc038320 0x7fa7bc03a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:19.636 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.873+0000 7fa7d0ff9700 1 -- 172.21.15.67:0/3490084621 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa7c404c550 con 0x7fa7d4106fe0 2024-04-22T23:14:19.636 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.873+0000 7fa7d9dfe700 1 -- 172.21.15.67:0/3490084621 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7d410a700 con 0x7fa7d4106fe0 2024-04-22T23:14:19.636 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.873+0000 7fa7d2ffd700 1 --2- 172.21.15.67:0/3490084621 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fa7bc038320 0x7fa7bc03a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:19.637 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.873+0000 7fa7d2ffd700 1 --2- 172.21.15.67:0/3490084621 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fa7bc038320 0x7fa7bc03a7e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa7c8006fd0 tx=0x7fa7c8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:19.637 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:18.878+0000 7fa7d0ff9700 1 -- 172.21.15.67:0/3490084621 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa7c402d440 con 0x7fa7d4106fe0 2024-04-22T23:14:19.637 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:19.073+0000 7fa7d9dfe700 1 -- 172.21.15.67:0/3490084621 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fa7d4105d40 con 0x7fa7d4106fe0 2024-04-22T23:14:19.637 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:19.076+0000 7fa7d0ff9700 1 -- 172.21.15.67:0/3490084621 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fa7c40172b0 con 0x7fa7d4106fe0 2024-04-22T23:14:19.637 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:19.078+0000 7fa7d9dfe700 1 -- 172.21.15.67:0/3490084621 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fa7bc038320 msgr2=0x7fa7bc03a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:19.637 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:19.078+0000 7fa7d9dfe700 1 --2- 172.21.15.67:0/3490084621 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fa7bc038320 0x7fa7bc03a7e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa7c8006fd0 tx=0x7fa7c8006e40 comp rx=0 tx=0).stop 2024-04-22T23:14:19.637 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:19.078+0000 7fa7d9dfe700 1 -- 172.21.15.67:0/3490084621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7d4106fe0 msgr2=0x7fa7d4106940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:19.637 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:19.078+0000 7fa7d9dfe700 1 --2- 172.21.15.67:0/3490084621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7d4106fe0 0x7fa7d4106940 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa7c4006ba0 tx=0x7fa7c4011b50 comp rx=0 tx=0).stop 2024-04-22T23:14:19.637 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:19.078+0000 7fa7d9dfe700 1 -- 172.21.15.67:0/3490084621 shutdown_connections 2024-04-22T23:14:19.637 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:19.078+0000 7fa7d9dfe700 1 --2- 172.21.15.67:0/3490084621 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fa7bc038320 0x7fa7bc03a7e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:19.637 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:19.078+0000 7fa7d9dfe700 1 --2- 172.21.15.67:0/3490084621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7d4106fe0 0x7fa7d4106940 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:19.637 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:19.078+0000 7fa7d9dfe700 1 -- 172.21.15.67:0/3490084621 >> 172.21.15.67:0/3490084621 conn(0x7fa7d41009f0 msgr2=0x7fa7d4101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:19.637 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:19.079+0000 7fa7d9dfe700 1 -- 172.21.15.67:0/3490084621 shutdown_connections 2024-04-22T23:14:19.637 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:19.079+0000 7fa7d9dfe700 1 -- 172.21.15.67:0/3490084621 wait complete. 2024-04-22T23:14:20.248 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:20 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/3490084621' entity='client.admin' 2024-04-22T23:14:20.820 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.040+0000 7f904551d700 1 Processor -- start 2024-04-22T23:14:20.820 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.040+0000 7f904551d700 1 -- start start 2024-04-22T23:14:20.820 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.040+0000 7f904551d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9040104840 0x7f9040104c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:20.821 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.040+0000 7f904551d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9040105230 con 0x7f9040104840 2024-04-22T23:14:20.821 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.040+0000 7f903effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9040104840 0x7f9040104c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:20.821 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.040+0000 7f903effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9040104840 0x7f9040104c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33840/0 (socket says 172.21.15.67:33840) 2024-04-22T23:14:20.821 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.040+0000 7f903effd700 1 -- 172.21.15.67:0/2182611176 learned_addr learned my addr 172.21.15.67:0/2182611176 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:20.821 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.041+0000 7f903effd700 1 -- 172.21.15.67:0/2182611176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90401077d0 con 0x7f9040104840 2024-04-22T23:14:20.821 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.041+0000 7f903effd700 1 --2- 172.21.15.67:0/2182611176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9040104840 0x7f9040104c60 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f90340098a0 tx=0x7f9034009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1e37a55335e9569 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:20.821 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.041+0000 7f903dffb700 1 -- 172.21.15.67:0/2182611176 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9034004180 con 0x7f9040104840 2024-04-22T23:14:20.821 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.041+0000 7f903dffb700 1 -- 172.21.15.67:0/2182611176 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f90340042e0 con 0x7f9040104840 2024-04-22T23:14:20.821 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.041+0000 7f903dffb700 1 -- 172.21.15.67:0/2182611176 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9034011af0 con 0x7f9040104840 2024-04-22T23:14:20.821 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.041+0000 7f904551d700 1 -- 172.21.15.67:0/2182611176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9040104840 msgr2=0x7f9040104c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:20.821 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.041+0000 7f904551d700 1 --2- 172.21.15.67:0/2182611176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9040104840 0x7f9040104c60 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f90340098a0 tx=0x7f9034009bb0 comp rx=0 tx=0).stop 2024-04-22T23:14:20.821 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.042+0000 7f904551d700 1 -- 172.21.15.67:0/2182611176 shutdown_connections 2024-04-22T23:14:20.821 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.042+0000 7f904551d700 1 --2- 172.21.15.67:0/2182611176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9040104840 0x7f9040104c60 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:20.821 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.042+0000 7f904551d700 1 -- 172.21.15.67:0/2182611176 >> 172.21.15.67:0/2182611176 conn(0x7f9040100190 msgr2=0x7f90401025b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:20.822 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.042+0000 7f904551d700 1 -- 172.21.15.67:0/2182611176 shutdown_connections 2024-04-22T23:14:20.822 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.042+0000 7f904551d700 1 -- 172.21.15.67:0/2182611176 wait complete. 2024-04-22T23:14:20.822 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.042+0000 7f904551d700 1 Processor -- start 2024-04-22T23:14:20.822 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.042+0000 7f904551d700 1 -- start start 2024-04-22T23:14:20.822 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.043+0000 7f904551d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9040104840 0x7f904019c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:20.822 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.043+0000 7f904551d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f904019c9d0 con 0x7f9040104840 2024-04-22T23:14:20.822 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.043+0000 7f903effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9040104840 0x7f904019c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:20.822 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.043+0000 7f903effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9040104840 0x7f904019c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33850/0 (socket says 172.21.15.67:33850) 2024-04-22T23:14:20.822 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.043+0000 7f903effd700 1 -- 172.21.15.67:0/2913597858 learned_addr learned my addr 172.21.15.67:0/2913597858 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:20.822 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.043+0000 7f903effd700 1 -- 172.21.15.67:0/2913597858 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9034009550 con 0x7f9040104840 2024-04-22T23:14:20.822 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.043+0000 7f903effd700 1 --2- 172.21.15.67:0/2913597858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9040104840 0x7f904019c490 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9034011670 tx=0x7f9034011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:20.822 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.043+0000 7f9027fff700 1 -- 172.21.15.67:0/2913597858 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90340039c0 con 0x7f9040104840 2024-04-22T23:14:20.823 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.043+0000 7f904551d700 1 -- 172.21.15.67:0/2913597858 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f904019cbd0 con 0x7f9040104840 2024-04-22T23:14:20.823 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.043+0000 7f9027fff700 1 -- 172.21.15.67:0/2913597858 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9034003b20 con 0x7f9040104840 2024-04-22T23:14:20.823 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.043+0000 7f9027fff700 1 -- 172.21.15.67:0/2913597858 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90340247f0 con 0x7f9040104840 2024-04-22T23:14:20.823 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.043+0000 7f904551d700 1 -- 172.21.15.67:0/2913597858 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f904019d070 con 0x7f9040104840 2024-04-22T23:14:20.823 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.044+0000 7f9027fff700 1 -- 172.21.15.67:0/2913597858 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f9034024ab0 con 0x7f9040104840 2024-04-22T23:14:20.823 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.044+0000 7f904551d700 1 -- 172.21.15.67:0/2913597858 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90400623c0 con 0x7f9040104840 2024-04-22T23:14:20.823 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.044+0000 7f9027fff700 1 --2- 172.21.15.67:0/2913597858 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f90280382e0 0x7f902803a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:20.823 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.044+0000 7f9027fff700 1 -- 172.21.15.67:0/2913597858 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f903404c450 con 0x7f9040104840 2024-04-22T23:14:20.823 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.045+0000 7f903e7fc700 1 --2- 172.21.15.67:0/2913597858 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f90280382e0 0x7f902803a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:20.823 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.045+0000 7f903e7fc700 1 --2- 172.21.15.67:0/2913597858 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f90280382e0 0x7f902803a7a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9030006fd0 tx=0x7f9030006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:20.823 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.049+0000 7f9027fff700 1 -- 172.21.15.67:0/2913597858 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f9034024d60 con 0x7f9040104840 2024-04-22T23:14:20.823 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.247+0000 7f904551d700 1 -- 172.21.15.67:0/2913597858 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f904019ff70 con 0x7f9040104840 2024-04-22T23:14:20.823 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.253+0000 7f9027fff700 1 -- 172.21.15.67:0/2913597858 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f903402c350 con 0x7f9040104840 2024-04-22T23:14:20.824 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.255+0000 7f904551d700 1 -- 172.21.15.67:0/2913597858 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f90280382e0 msgr2=0x7f902803a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:20.824 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.255+0000 7f904551d700 1 --2- 172.21.15.67:0/2913597858 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f90280382e0 0x7f902803a7a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9030006fd0 tx=0x7f9030006e40 comp rx=0 tx=0).stop 2024-04-22T23:14:20.824 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.255+0000 7f904551d700 1 -- 172.21.15.67:0/2913597858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9040104840 msgr2=0x7f904019c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:20.824 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.255+0000 7f904551d700 1 --2- 172.21.15.67:0/2913597858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9040104840 0x7f904019c490 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9034011670 tx=0x7f9034011aa0 comp rx=0 tx=0).stop 2024-04-22T23:14:20.824 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.255+0000 7f904551d700 1 -- 172.21.15.67:0/2913597858 shutdown_connections 2024-04-22T23:14:20.824 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.255+0000 7f904551d700 1 --2- 172.21.15.67:0/2913597858 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f90280382e0 0x7f902803a7a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:20.824 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.255+0000 7f904551d700 1 --2- 172.21.15.67:0/2913597858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9040104840 0x7f904019c490 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:20.824 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.255+0000 7f904551d700 1 -- 172.21.15.67:0/2913597858 >> 172.21.15.67:0/2913597858 conn(0x7f9040100190 msgr2=0x7f90401017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:20.824 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.256+0000 7f904551d700 1 -- 172.21.15.67:0/2913597858 shutdown_connections 2024-04-22T23:14:20.824 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:20.256+0000 7f904551d700 1 -- 172.21.15.67:0/2913597858 wait complete. 2024-04-22T23:14:20.824 INFO:teuthology.orchestra.run.smithi067.stdout:Enabling the dashboard module... 2024-04-22T23:14:21.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:21 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/2913597858' entity='client.admin' 2024-04-22T23:14:21.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:21 smithi067 ceph-mon[26453]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:14:22.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:22 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/1681964469' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-22T23:14:22.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.210+0000 7f8f470e2700 1 Processor -- start 2024-04-22T23:14:22.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.210+0000 7f8f470e2700 1 -- start start 2024-04-22T23:14:22.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.210+0000 7f8f470e2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f40107010 0x7f8f40109400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:22.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.210+0000 7f8f470e2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f40079e50 con 0x7f8f40107010 2024-04-22T23:14:22.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.210+0000 7f8f44e7e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f40107010 0x7f8f40109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:22.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.210+0000 7f8f44e7e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f40107010 0x7f8f40109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33856/0 (socket says 172.21.15.67:33856) 2024-04-22T23:14:22.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.210+0000 7f8f44e7e700 1 -- 172.21.15.67:0/596235 learned_addr learned my addr 172.21.15.67:0/596235 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:22.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.210+0000 7f8f44e7e700 1 -- 172.21.15.67:0/596235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f40109940 con 0x7f8f40107010 2024-04-22T23:14:22.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.211+0000 7f8f44e7e700 1 --2- 172.21.15.67:0/596235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f40107010 0x7f8f40109400 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8f340098a0 tx=0x7f8f34009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6c227c70e7e135c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:22.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.211+0000 7f8f3f7fe700 1 -- 172.21.15.67:0/596235 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f34004180 con 0x7f8f40107010 2024-04-22T23:14:22.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.211+0000 7f8f3f7fe700 1 -- 172.21.15.67:0/596235 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8f340042e0 con 0x7f8f40107010 2024-04-22T23:14:22.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.211+0000 7f8f3f7fe700 1 -- 172.21.15.67:0/596235 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f34011af0 con 0x7f8f40107010 2024-04-22T23:14:22.860 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.211+0000 7f8f470e2700 1 -- 172.21.15.67:0/596235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f40107010 msgr2=0x7f8f40109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:22.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.211+0000 7f8f470e2700 1 --2- 172.21.15.67:0/596235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f40107010 0x7f8f40109400 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8f340098a0 tx=0x7f8f34009bb0 comp rx=0 tx=0).stop 2024-04-22T23:14:22.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.211+0000 7f8f470e2700 1 -- 172.21.15.67:0/596235 shutdown_connections 2024-04-22T23:14:22.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.211+0000 7f8f470e2700 1 --2- 172.21.15.67:0/596235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f40107010 0x7f8f40109400 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:22.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.211+0000 7f8f470e2700 1 -- 172.21.15.67:0/596235 >> 172.21.15.67:0/596235 conn(0x7f8f40100a20 msgr2=0x7f8f40102e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:22.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.211+0000 7f8f470e2700 1 -- 172.21.15.67:0/596235 shutdown_connections 2024-04-22T23:14:22.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.211+0000 7f8f470e2700 1 -- 172.21.15.67:0/596235 wait complete. 2024-04-22T23:14:22.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.212+0000 7f8f470e2700 1 Processor -- start 2024-04-22T23:14:22.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.212+0000 7f8f470e2700 1 -- start start 2024-04-22T23:14:22.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.212+0000 7f8f470e2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f40107010 0x7f8f4019c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:22.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.212+0000 7f8f470e2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4019c960 con 0x7f8f40107010 2024-04-22T23:14:22.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.212+0000 7f8f44e7e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f40107010 0x7f8f4019c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:22.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.212+0000 7f8f44e7e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f40107010 0x7f8f4019c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33858/0 (socket says 172.21.15.67:33858) 2024-04-22T23:14:22.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.212+0000 7f8f44e7e700 1 -- 172.21.15.67:0/1681964469 learned_addr learned my addr 172.21.15.67:0/1681964469 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:22.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.212+0000 7f8f44e7e700 1 -- 172.21.15.67:0/1681964469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f34009550 con 0x7f8f40107010 2024-04-22T23:14:22.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.213+0000 7f8f44e7e700 1 --2- 172.21.15.67:0/1681964469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f40107010 0x7f8f4019c420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8f34011670 tx=0x7f8f34011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:22.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.213+0000 7f8f3dffb700 1 -- 172.21.15.67:0/1681964469 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f34003a40 con 0x7f8f40107010 2024-04-22T23:14:22.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.213+0000 7f8f3dffb700 1 -- 172.21.15.67:0/1681964469 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8f34003ba0 con 0x7f8f40107010 2024-04-22T23:14:22.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.213+0000 7f8f470e2700 1 -- 172.21.15.67:0/1681964469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f4019cb60 con 0x7f8f40107010 2024-04-22T23:14:22.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.213+0000 7f8f3dffb700 1 -- 172.21.15.67:0/1681964469 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f34024820 con 0x7f8f40107010 2024-04-22T23:14:22.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.213+0000 7f8f470e2700 1 -- 172.21.15.67:0/1681964469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f4019d000 con 0x7f8f40107010 2024-04-22T23:14:22.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.214+0000 7f8f3dffb700 1 -- 172.21.15.67:0/1681964469 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f8f3401a430 con 0x7f8f40107010 2024-04-22T23:14:22.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.214+0000 7f8f3dffb700 1 --2- 172.21.15.67:0/1681964469 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f8f30038330 0x7f8f3003a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:22.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.214+0000 7f8f470e2700 1 -- 172.21.15.67:0/1681964469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f4010a6d0 con 0x7f8f40107010 2024-04-22T23:14:22.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.214+0000 7f8f3dffb700 1 -- 172.21.15.67:0/1681964469 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f8f3404c290 con 0x7f8f40107010 2024-04-22T23:14:22.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.214+0000 7f8f3ffff700 1 --2- 172.21.15.67:0/1681964469 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f8f30038330 0x7f8f3003a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:22.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.214+0000 7f8f3ffff700 1 --2- 172.21.15.67:0/1681964469 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f8f30038330 0x7f8f3003a7f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c006fd0 tx=0x7f8f2c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:22.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.219+0000 7f8f3dffb700 1 -- 172.21.15.67:0/1681964469 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f8f34018530 con 0x7f8f40107010 2024-04-22T23:14:22.862 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:21.473+0000 7f8f470e2700 1 -- 172.21.15.67:0/1681964469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f8f4019fde0 con 0x7f8f40107010 2024-04-22T23:14:22.863 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:22.262+0000 7f8f3dffb700 1 -- 172.21.15.67:0/1681964469 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 9) v1 ==== 45296+0+0 (secure 0 0 0) 0x7f8f3401ac70 con 0x7f8f40107010 2024-04-22T23:14:22.863 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:22.264+0000 7f8f3dffb700 1 -- 172.21.15.67:0/1681964469 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f8f3400c150 con 0x7f8f40107010 2024-04-22T23:14:22.863 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:22.266+0000 7f8f470e2700 1 -- 172.21.15.67:0/1681964469 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f8f30038330 msgr2=0x7f8f3003a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:22.863 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:22.266+0000 7f8f470e2700 1 --2- 172.21.15.67:0/1681964469 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f8f30038330 0x7f8f3003a7f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c006fd0 tx=0x7f8f2c006e40 comp rx=0 tx=0).stop 2024-04-22T23:14:22.863 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:22.267+0000 7f8f470e2700 1 -- 172.21.15.67:0/1681964469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f40107010 msgr2=0x7f8f4019c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:22.863 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:22.267+0000 7f8f470e2700 1 --2- 172.21.15.67:0/1681964469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f40107010 0x7f8f4019c420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8f34011670 tx=0x7f8f34011aa0 comp rx=0 tx=0).stop 2024-04-22T23:14:22.863 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:22.267+0000 7f8f470e2700 1 -- 172.21.15.67:0/1681964469 shutdown_connections 2024-04-22T23:14:22.863 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:22.267+0000 7f8f470e2700 1 --2- 172.21.15.67:0/1681964469 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f8f30038330 0x7f8f3003a7f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:22.863 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:22.267+0000 7f8f470e2700 1 --2- 172.21.15.67:0/1681964469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f40107010 0x7f8f4019c420 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:22.863 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:22.267+0000 7f8f470e2700 1 -- 172.21.15.67:0/1681964469 >> 172.21.15.67:0/1681964469 conn(0x7f8f40100a20 msgr2=0x7f8f401017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:22.863 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:22.267+0000 7f8f470e2700 1 -- 172.21.15.67:0/1681964469 shutdown_connections 2024-04-22T23:14:22.863 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:22.267+0000 7f8f470e2700 1 -- 172.21.15.67:0/1681964469 wait complete. 2024-04-22T23:14:23.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:23 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/1681964469' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-22T23:14:23.522 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:23 smithi067 ceph-mon[26453]: mgrmap e9: smithi067.whdcsr(active, since 25s) 2024-04-22T23:14:24.191 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout { 2024-04-22T23:14:24.191 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-22T23:14:24.191 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-22T23:14:24.191 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "active_name": "smithi067.whdcsr", 2024-04-22T23:14:24.191 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-22T23:14:24.192 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout } 2024-04-22T23:14:24.192 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.313+0000 7fc44c206700 1 Processor -- start 2024-04-22T23:14:24.192 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.313+0000 7fc44c206700 1 -- start start 2024-04-22T23:14:24.192 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.314+0000 7fc44c206700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc444106fe0 0x7fc4441093d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:24.192 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.314+0000 7fc44c206700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc444074680 con 0x7fc444106fe0 2024-04-22T23:14:24.192 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.314+0000 7fc449fa2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc444106fe0 0x7fc4441093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:24.192 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.314+0000 7fc449fa2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc444106fe0 0x7fc4441093d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33894/0 (socket says 172.21.15.67:33894) 2024-04-22T23:14:24.192 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.314+0000 7fc449fa2700 1 -- 172.21.15.67:0/4023093 learned_addr learned my addr 172.21.15.67:0/4023093 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:24.192 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.314+0000 7fc449fa2700 1 -- 172.21.15.67:0/4023093 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc444109910 con 0x7fc444106fe0 2024-04-22T23:14:24.192 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.314+0000 7fc449fa2700 1 --2- 172.21.15.67:0/4023093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc444106fe0 0x7fc4441093d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc4400098a0 tx=0x7fc440009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20d4bec75b1f9f7f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:24.192 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.315+0000 7fc448fa0700 1 -- 172.21.15.67:0/4023093 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc440004180 con 0x7fc444106fe0 2024-04-22T23:14:24.192 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.315+0000 7fc448fa0700 1 -- 172.21.15.67:0/4023093 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc4400042e0 con 0x7fc444106fe0 2024-04-22T23:14:24.192 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.315+0000 7fc448fa0700 1 -- 172.21.15.67:0/4023093 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc440011af0 con 0x7fc444106fe0 2024-04-22T23:14:24.193 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.315+0000 7fc44c206700 1 -- 172.21.15.67:0/4023093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc444106fe0 msgr2=0x7fc4441093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:24.193 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.315+0000 7fc44c206700 1 --2- 172.21.15.67:0/4023093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc444106fe0 0x7fc4441093d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc4400098a0 tx=0x7fc440009bb0 comp rx=0 tx=0).stop 2024-04-22T23:14:24.193 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.315+0000 7fc44c206700 1 -- 172.21.15.67:0/4023093 shutdown_connections 2024-04-22T23:14:24.193 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.315+0000 7fc44c206700 1 --2- 172.21.15.67:0/4023093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc444106fe0 0x7fc4441093d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:24.193 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.315+0000 7fc44c206700 1 -- 172.21.15.67:0/4023093 >> 172.21.15.67:0/4023093 conn(0x7fc4441009f0 msgr2=0x7fc444102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:24.193 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.316+0000 7fc44c206700 1 -- 172.21.15.67:0/4023093 shutdown_connections 2024-04-22T23:14:24.193 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.316+0000 7fc44c206700 1 -- 172.21.15.67:0/4023093 wait complete. 2024-04-22T23:14:24.193 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.316+0000 7fc44c206700 1 Processor -- start 2024-04-22T23:14:24.193 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.316+0000 7fc44c206700 1 -- start start 2024-04-22T23:14:24.193 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.317+0000 7fc44c206700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc444106fe0 0x7fc44419a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:24.193 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.317+0000 7fc44c206700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44419a6b0 con 0x7fc444106fe0 2024-04-22T23:14:24.193 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.317+0000 7fc449fa2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc444106fe0 0x7fc44419a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:24.193 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.317+0000 7fc449fa2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc444106fe0 0x7fc44419a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33900/0 (socket says 172.21.15.67:33900) 2024-04-22T23:14:24.193 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.317+0000 7fc449fa2700 1 -- 172.21.15.67:0/4136981999 learned_addr learned my addr 172.21.15.67:0/4136981999 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:24.194 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.317+0000 7fc449fa2700 1 -- 172.21.15.67:0/4136981999 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc440009550 con 0x7fc444106fe0 2024-04-22T23:14:24.194 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.317+0000 7fc449fa2700 1 --2- 172.21.15.67:0/4136981999 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc444106fe0 0x7fc44419a170 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc440006ba0 tx=0x7fc440011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:24.194 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.317+0000 7fc43affd700 1 -- 172.21.15.67:0/4136981999 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc440003b00 con 0x7fc444106fe0 2024-04-22T23:14:24.194 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.317+0000 7fc44c206700 1 -- 172.21.15.67:0/4136981999 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc44419a8b0 con 0x7fc444106fe0 2024-04-22T23:14:24.194 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.317+0000 7fc43affd700 1 -- 172.21.15.67:0/4136981999 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc44001a8e0 con 0x7fc444106fe0 2024-04-22T23:14:24.194 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.318+0000 7fc43affd700 1 -- 172.21.15.67:0/4136981999 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4400199c0 con 0x7fc444106fe0 2024-04-22T23:14:24.194 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.318+0000 7fc44c206700 1 -- 172.21.15.67:0/4136981999 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc44419ad50 con 0x7fc444106fe0 2024-04-22T23:14:24.194 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.318+0000 7fc43affd700 1 -- 172.21.15.67:0/4136981999 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 45296+0+0 (secure 0 0 0) 0x7fc440019c30 con 0x7fc444106fe0 2024-04-22T23:14:24.194 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.318+0000 7fc43affd700 1 --2- 172.21.15.67:0/4136981999 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fc4300383c0 0x7fc43003a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:24.194 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.319+0000 7fc43affd700 1 -- 172.21.15.67:0/4136981999 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc44001a430 con 0x7fc444106fe0 2024-04-22T23:14:24.194 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.319+0000 7fc4497a1700 1 -- 172.21.15.67:0/4136981999 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fc4300383c0 msgr2=0x7fc43003a880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/3568049827 2024-04-22T23:14:24.194 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.319+0000 7fc4497a1700 1 --2- 172.21.15.67:0/4136981999 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fc4300383c0 0x7fc43003a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T23:14:24.194 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.319+0000 7fc44c206700 1 -- 172.21.15.67:0/4136981999 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc44410a700 con 0x7fc444106fe0 2024-04-22T23:14:24.194 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.324+0000 7fc43affd700 1 -- 172.21.15.67:0/4136981999 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc440053060 con 0x7fc444106fe0 2024-04-22T23:14:24.195 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.520+0000 7fc4497a1700 1 -- 172.21.15.67:0/4136981999 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fc4300383c0 msgr2=0x7fc43003a880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/3568049827 2024-04-22T23:14:24.195 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.520+0000 7fc4497a1700 1 --2- 172.21.15.67:0/4136981999 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fc4300383c0 0x7fc43003a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T23:14:24.195 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.624+0000 7fc44c206700 1 -- 172.21.15.67:0/4136981999 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fc44419de40 con 0x7fc444106fe0 2024-04-22T23:14:24.195 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.624+0000 7fc43affd700 1 -- 172.21.15.67:0/4136981999 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7fc440023390 con 0x7fc444106fe0 2024-04-22T23:14:24.195 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.626+0000 7fc44c206700 1 -- 172.21.15.67:0/4136981999 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fc4300383c0 msgr2=0x7fc43003a880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T23:14:24.195 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.626+0000 7fc44c206700 1 --2- 172.21.15.67:0/4136981999 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fc4300383c0 0x7fc43003a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:24.195 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.626+0000 7fc44c206700 1 -- 172.21.15.67:0/4136981999 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc444106fe0 msgr2=0x7fc44419a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:24.195 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.626+0000 7fc44c206700 1 --2- 172.21.15.67:0/4136981999 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc444106fe0 0x7fc44419a170 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc440006ba0 tx=0x7fc440011b50 comp rx=0 tx=0).stop 2024-04-22T23:14:24.195 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.626+0000 7fc44c206700 1 -- 172.21.15.67:0/4136981999 shutdown_connections 2024-04-22T23:14:24.195 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.626+0000 7fc44c206700 1 --2- 172.21.15.67:0/4136981999 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7fc4300383c0 0x7fc43003a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:24.195 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.626+0000 7fc44c206700 1 --2- 172.21.15.67:0/4136981999 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc444106fe0 0x7fc44419a170 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:24.195 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.626+0000 7fc44c206700 1 -- 172.21.15.67:0/4136981999 >> 172.21.15.67:0/4136981999 conn(0x7fc4441009f0 msgr2=0x7fc444101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:24.195 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.626+0000 7fc44c206700 1 -- 172.21.15.67:0/4136981999 shutdown_connections 2024-04-22T23:14:24.195 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:23.626+0000 7fc44c206700 1 -- 172.21.15.67:0/4136981999 wait complete. 2024-04-22T23:14:24.196 INFO:teuthology.orchestra.run.smithi067.stdout:Waiting for the mgr to restart... 2024-04-22T23:14:24.196 INFO:teuthology.orchestra.run.smithi067.stdout:Waiting for mgr epoch 9... 2024-04-22T23:14:24.512 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:24 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/4136981999' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-22T23:14:29.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:29 smithi067 ceph-mon[26453]: Active manager daemon smithi067.whdcsr restarted 2024-04-22T23:14:29.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:29 smithi067 ceph-mon[26453]: Activating manager daemon smithi067.whdcsr 2024-04-22T23:14:29.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:29 smithi067 ceph-mon[26453]: osdmap e3: 0 total, 0 up, 0 in 2024-04-22T23:14:29.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:29 smithi067 ceph-mon[26453]: mgrmap e10: smithi067.whdcsr(active, starting, since 0.00378496s) 2024-04-22T23:14:29.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:29 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-22T23:14:29.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:29 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mgr metadata", "who": "smithi067.whdcsr", "id": "smithi067.whdcsr"}]: dispatch 2024-04-22T23:14:29.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:29 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T23:14:29.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:29 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T23:14:29.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:29 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T23:14:29.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:29 smithi067 ceph-mon[26453]: Manager daemon smithi067.whdcsr is now available 2024-04-22T23:14:29.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:29 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:14:29.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:29 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:14:29.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:29 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.whdcsr/mirror_snapshot_schedule"}]: dispatch 2024-04-22T23:14:29.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:29 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.whdcsr/trash_purge_schedule"}]: dispatch 2024-04-22T23:14:31.021 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout { 2024-04-22T23:14:31.021 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-22T23:14:31.021 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-22T23:14:31.021 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout } 2024-04-22T23:14:31.021 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.614+0000 7f58c280f700 1 Processor -- start 2024-04-22T23:14:31.021 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.614+0000 7f58c280f700 1 -- start start 2024-04-22T23:14:31.021 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.614+0000 7f58c280f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58bc107020 0x7f58bc109410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:31.021 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.614+0000 7f58c280f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc079e50 con 0x7f58bc107020 2024-04-22T23:14:31.021 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.614+0000 7f58bbfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58bc107020 0x7f58bc109410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:31.021 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.614+0000 7f58bbfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58bc107020 0x7f58bc109410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33910/0 (socket says 172.21.15.67:33910) 2024-04-22T23:14:31.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.614+0000 7f58bbfff700 1 -- 172.21.15.67:0/1811398813 learned_addr learned my addr 172.21.15.67:0/1811398813 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:31.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.614+0000 7f58bbfff700 1 -- 172.21.15.67:0/1811398813 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58bc109950 con 0x7f58bc107020 2024-04-22T23:14:31.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.615+0000 7f58bbfff700 1 --2- 172.21.15.67:0/1811398813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58bc107020 0x7f58bc109410 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f58ac0098a0 tx=0x7f58ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1eece9142bf4054 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:31.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.615+0000 7f58baffd700 1 -- 172.21.15.67:0/1811398813 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58ac004180 con 0x7f58bc107020 2024-04-22T23:14:31.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.615+0000 7f58baffd700 1 -- 172.21.15.67:0/1811398813 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f58ac0042e0 con 0x7f58bc107020 2024-04-22T23:14:31.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.615+0000 7f58baffd700 1 -- 172.21.15.67:0/1811398813 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58ac011af0 con 0x7f58bc107020 2024-04-22T23:14:31.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.616+0000 7f58c280f700 1 -- 172.21.15.67:0/1811398813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58bc107020 msgr2=0x7f58bc109410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:31.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.616+0000 7f58c280f700 1 --2- 172.21.15.67:0/1811398813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58bc107020 0x7f58bc109410 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f58ac0098a0 tx=0x7f58ac009bb0 comp rx=0 tx=0).stop 2024-04-22T23:14:31.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.616+0000 7f58c280f700 1 -- 172.21.15.67:0/1811398813 shutdown_connections 2024-04-22T23:14:31.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.616+0000 7f58c280f700 1 --2- 172.21.15.67:0/1811398813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58bc107020 0x7f58bc109410 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:31.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.616+0000 7f58c280f700 1 -- 172.21.15.67:0/1811398813 >> 172.21.15.67:0/1811398813 conn(0x7f58bc1009f0 msgr2=0x7f58bc102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:31.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.616+0000 7f58c280f700 1 -- 172.21.15.67:0/1811398813 shutdown_connections 2024-04-22T23:14:31.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.616+0000 7f58c280f700 1 -- 172.21.15.67:0/1811398813 wait complete. 2024-04-22T23:14:31.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.616+0000 7f58c280f700 1 Processor -- start 2024-04-22T23:14:31.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.616+0000 7f58c280f700 1 -- start start 2024-04-22T23:14:31.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.617+0000 7f58c280f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58bc107020 0x7f58bc19a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:31.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.617+0000 7f58c280f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc19a6f0 con 0x7f58bc107020 2024-04-22T23:14:31.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.617+0000 7f58bbfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58bc107020 0x7f58bc19a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:31.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.617+0000 7f58bbfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58bc107020 0x7f58bc19a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33922/0 (socket says 172.21.15.67:33922) 2024-04-22T23:14:31.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.617+0000 7f58bbfff700 1 -- 172.21.15.67:0/401437305 learned_addr learned my addr 172.21.15.67:0/401437305 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:31.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.617+0000 7f58bbfff700 1 -- 172.21.15.67:0/401437305 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ac009550 con 0x7f58bc107020 2024-04-22T23:14:31.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.617+0000 7f58bbfff700 1 --2- 172.21.15.67:0/401437305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58bc107020 0x7f58bc19a1b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f58ac006ba0 tx=0x7f58ac011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:31.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.617+0000 7f58b97fa700 1 -- 172.21.15.67:0/401437305 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58ac003b00 con 0x7f58bc107020 2024-04-22T23:14:31.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.617+0000 7f58b97fa700 1 -- 172.21.15.67:0/401437305 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f58ac01a8e0 con 0x7f58bc107020 2024-04-22T23:14:31.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.617+0000 7f58c280f700 1 -- 172.21.15.67:0/401437305 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58bc19a8f0 con 0x7f58bc107020 2024-04-22T23:14:31.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.617+0000 7f58b97fa700 1 -- 172.21.15.67:0/401437305 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58ac0199c0 con 0x7f58bc107020 2024-04-22T23:14:31.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.617+0000 7f58c280f700 1 -- 172.21.15.67:0/401437305 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58bc19ad90 con 0x7f58bc107020 2024-04-22T23:14:31.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.618+0000 7f58b97fa700 1 -- 172.21.15.67:0/401437305 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) v1 ==== 45296+0+0 (secure 0 0 0) 0x7f58ac019c30 con 0x7f58bc107020 2024-04-22T23:14:31.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.619+0000 7f58b97fa700 1 --2- 172.21.15.67:0/401437305 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f58a4038370 0x7f58a403a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:31.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.619+0000 7f58b97fa700 1 -- 172.21.15.67:0/401437305 --> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f58a403af40 con 0x7f58a4038370 2024-04-22T23:14:31.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.619+0000 7f58b97fa700 1 -- 172.21.15.67:0/401437305 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f58ac04c5b0 con 0x7f58bc107020 2024-04-22T23:14:31.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.619+0000 7f58bb7fe700 1 -- 172.21.15.67:0/401437305 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f58a4038370 msgr2=0x7f58a403a830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/3568049827 2024-04-22T23:14:31.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.619+0000 7f58bb7fe700 1 --2- 172.21.15.67:0/401437305 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f58a4038370 0x7f58a403a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T23:14:31.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.820+0000 7f58bb7fe700 1 -- 172.21.15.67:0/401437305 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f58a4038370 msgr2=0x7f58a403a830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/3568049827 2024-04-22T23:14:31.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:24.820+0000 7f58bb7fe700 1 --2- 172.21.15.67:0/401437305 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f58a4038370 0x7f58a403a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T23:14:31.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:25.221+0000 7f58bb7fe700 1 -- 172.21.15.67:0/401437305 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f58a4038370 msgr2=0x7f58a403a830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/3568049827 2024-04-22T23:14:31.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:25.221+0000 7f58bb7fe700 1 --2- 172.21.15.67:0/401437305 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f58a4038370 0x7f58a403a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T23:14:31.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:26.022+0000 7f58bb7fe700 1 -- 172.21.15.67:0/401437305 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f58a4038370 msgr2=0x7f58a403a830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/3568049827 2024-04-22T23:14:31.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:26.022+0000 7f58bb7fe700 1 --2- 172.21.15.67:0/401437305 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f58a4038370 0x7f58a403a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T23:14:31.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:27.623+0000 7f58bb7fe700 1 -- 172.21.15.67:0/401437305 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f58a4038370 msgr2=0x7f58a403a830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/3568049827 2024-04-22T23:14:31.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:27.623+0000 7f58bb7fe700 1 --2- 172.21.15.67:0/401437305 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f58a4038370 0x7f58a403a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-22T23:14:31.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:29.215+0000 7f58b97fa700 1 -- 172.21.15.67:0/401437305 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mgrmap(e 10) v1 ==== 45063+0+0 (secure 0 0 0) 0x7f58ac02c3a0 con 0x7f58bc107020 2024-04-22T23:14:31.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:29.215+0000 7f58b97fa700 1 -- 172.21.15.67:0/401437305 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f58a4038370 msgr2=0x7f58a403a830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T23:14:31.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:29.215+0000 7f58b97fa700 1 --2- 172.21.15.67:0/401437305 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f58a4038370 0x7f58a403a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:31.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:30.218+0000 7f58b97fa700 1 -- 172.21.15.67:0/401437305 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 11) v1 ==== 45190+0+0 (secure 0 0 0) 0x7f58ac02d960 con 0x7f58bc107020 2024-04-22T23:14:31.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:30.218+0000 7f58b97fa700 1 --2- 172.21.15.67:0/401437305 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f58a403bf00 0x7f58a403e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:31.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:30.218+0000 7f58b97fa700 1 -- 172.21.15.67:0/401437305 --> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f58a403af40 con 0x7f58a403bf00 2024-04-22T23:14:31.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:30.218+0000 7f58bb7fe700 1 --2- 172.21.15.67:0/401437305 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f58a403bf00 0x7f58a403e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:31.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:30.219+0000 7f58bb7fe700 1 --2- 172.21.15.67:0/401437305 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f58a403bf00 0x7f58a403e2f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f58b0003a10 tx=0x7f58b0009270 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:31.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:30.219+0000 7f58b97fa700 1 -- 172.21.15.67:0/401437305 <== mgr.14164 v2:172.21.15.67:6800/2590792187 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f58a403af40 con 0x7f58a403bf00 2024-04-22T23:14:31.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:30.226+0000 7f58c280f700 1 -- 172.21.15.67:0/401437305 --> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f58bc19b4f0 con 0x7f58a403bf00 2024-04-22T23:14:31.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:30.226+0000 7f58b97fa700 1 -- 172.21.15.67:0/401437305 <== mgr.14164 v2:172.21.15.67:6800/2590792187 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f58bc19b4f0 con 0x7f58a403bf00 2024-04-22T23:14:31.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:30.226+0000 7f58c280f700 1 -- 172.21.15.67:0/401437305 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f58a403bf00 msgr2=0x7f58a403e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:31.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:30.226+0000 7f58c280f700 1 --2- 172.21.15.67:0/401437305 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f58a403bf00 0x7f58a403e2f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f58b0003a10 tx=0x7f58b0009270 comp rx=0 tx=0).stop 2024-04-22T23:14:31.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:30.226+0000 7f58c280f700 1 -- 172.21.15.67:0/401437305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58bc107020 msgr2=0x7f58bc19a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:31.026 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:30.226+0000 7f58c280f700 1 --2- 172.21.15.67:0/401437305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58bc107020 0x7f58bc19a1b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f58ac006ba0 tx=0x7f58ac011b50 comp rx=0 tx=0).stop 2024-04-22T23:14:31.026 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:30.226+0000 7f58c280f700 1 -- 172.21.15.67:0/401437305 shutdown_connections 2024-04-22T23:14:31.026 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:30.226+0000 7f58c280f700 1 --2- 172.21.15.67:0/401437305 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f58a403bf00 0x7f58a403e2f0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:31.026 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:30.226+0000 7f58c280f700 1 --2- 172.21.15.67:0/401437305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58bc107020 0x7f58bc19a1b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:31.026 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:30.227+0000 7f58c280f700 1 --2- 172.21.15.67:0/401437305 >> [v2:172.21.15.67:6800/3568049827,v1:172.21.15.67:6801/3568049827] conn(0x7f58a4038370 0x7f58a403a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:31.026 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:30.227+0000 7f58c280f700 1 -- 172.21.15.67:0/401437305 >> 172.21.15.67:0/401437305 conn(0x7f58bc1009f0 msgr2=0x7f58bc101720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:31.026 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:30.227+0000 7f58c280f700 1 -- 172.21.15.67:0/401437305 shutdown_connections 2024-04-22T23:14:31.026 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:30.227+0000 7f58c280f700 1 -- 172.21.15.67:0/401437305 wait complete. 2024-04-22T23:14:31.026 INFO:teuthology.orchestra.run.smithi067.stdout:mgr epoch 9 is available 2024-04-22T23:14:31.026 INFO:teuthology.orchestra.run.smithi067.stdout:Generating a dashboard self-signed certificate... 2024-04-22T23:14:31.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:31 smithi067 ceph-mon[26453]: [22/Apr/2024:23:14:29] ENGINE Bus STARTING 2024-04-22T23:14:31.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:31 smithi067 ceph-mon[26453]: [22/Apr/2024:23:14:30] ENGINE Serving on https://172.21.15.67:7150 2024-04-22T23:14:31.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:31 smithi067 ceph-mon[26453]: [22/Apr/2024:23:14:30] ENGINE Serving on http://172.21.15.67:8765 2024-04-22T23:14:31.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:31 smithi067 ceph-mon[26453]: [22/Apr/2024:23:14:30] ENGINE Bus STARTED 2024-04-22T23:14:31.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:31 smithi067 ceph-mon[26453]: from='client.14168 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-22T23:14:31.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:31 smithi067 ceph-mon[26453]: mgrmap e11: smithi067.whdcsr(active, since 1.00596s) 2024-04-22T23:14:31.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:31 smithi067 ceph-mon[26453]: from='client.14168 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-22T23:14:31.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:31 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:32.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:32 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:32.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:32 smithi067 ceph-mon[26453]: mgrmap e12: smithi067.whdcsr(active, since 2s) 2024-04-22T23:14:32.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:32 smithi067 ceph-mon[26453]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:14:32.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:32 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:32.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:32 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:33.473 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-22T23:14:33.473 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.502+0000 7f23052d1700 1 Processor -- start 2024-04-22T23:14:33.474 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.502+0000 7f23052d1700 1 -- start start 2024-04-22T23:14:33.474 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.502+0000 7f23052d1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2300105470 0x7f2300105890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:33.474 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.502+0000 7f23052d1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2300105e60 con 0x7f2300105470 2024-04-22T23:14:33.474 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.502+0000 7f22feffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2300105470 0x7f2300105890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:33.474 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.503+0000 7f22feffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2300105470 0x7f2300105890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57084/0 (socket says 172.21.15.67:57084) 2024-04-22T23:14:33.474 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.503+0000 7f22feffd700 1 -- 172.21.15.67:0/4171974744 learned_addr learned my addr 172.21.15.67:0/4171974744 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:33.474 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.503+0000 7f22feffd700 1 -- 172.21.15.67:0/4171974744 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2300106670 con 0x7f2300105470 2024-04-22T23:14:33.474 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.503+0000 7f22feffd700 1 --2- 172.21.15.67:0/4171974744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2300105470 0x7f2300105890 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f22f000bb40 tx=0x7f22f000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7291c48d35b8b585 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:33.474 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.503+0000 7f22fdffb700 1 -- 172.21.15.67:0/4171974744 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22f000dc00 con 0x7f2300105470 2024-04-22T23:14:33.474 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.503+0000 7f22fdffb700 1 -- 172.21.15.67:0/4171974744 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f22f00044e0 con 0x7f2300105470 2024-04-22T23:14:33.474 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.503+0000 7f22fdffb700 1 -- 172.21.15.67:0/4171974744 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22f0010470 con 0x7f2300105470 2024-04-22T23:14:33.475 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.504+0000 7f23052d1700 1 -- 172.21.15.67:0/4171974744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2300105470 msgr2=0x7f2300105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:33.475 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.504+0000 7f23052d1700 1 --2- 172.21.15.67:0/4171974744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2300105470 0x7f2300105890 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f22f000bb40 tx=0x7f22f000be50 comp rx=0 tx=0).stop 2024-04-22T23:14:33.475 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.504+0000 7f23052d1700 1 -- 172.21.15.67:0/4171974744 shutdown_connections 2024-04-22T23:14:33.475 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.504+0000 7f23052d1700 1 --2- 172.21.15.67:0/4171974744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2300105470 0x7f2300105890 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:33.475 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.504+0000 7f23052d1700 1 -- 172.21.15.67:0/4171974744 >> 172.21.15.67:0/4171974744 conn(0x7f23001009f0 msgr2=0x7f2300102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:33.475 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.504+0000 7f23052d1700 1 -- 172.21.15.67:0/4171974744 shutdown_connections 2024-04-22T23:14:33.475 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.504+0000 7f23052d1700 1 -- 172.21.15.67:0/4171974744 wait complete. 2024-04-22T23:14:33.475 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.505+0000 7f23052d1700 1 Processor -- start 2024-04-22T23:14:33.475 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.505+0000 7f23052d1700 1 -- start start 2024-04-22T23:14:33.475 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.505+0000 7f23052d1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2300105470 0x7f230019c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:33.475 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.505+0000 7f23052d1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f230019c910 con 0x7f2300105470 2024-04-22T23:14:33.475 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.505+0000 7f22feffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2300105470 0x7f230019c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:33.475 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.505+0000 7f22feffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2300105470 0x7f230019c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57088/0 (socket says 172.21.15.67:57088) 2024-04-22T23:14:33.475 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.505+0000 7f22feffd700 1 -- 172.21.15.67:0/2262980248 learned_addr learned my addr 172.21.15.67:0/2262980248 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:33.476 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.506+0000 7f22feffd700 1 -- 172.21.15.67:0/2262980248 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f000b7f0 con 0x7f2300105470 2024-04-22T23:14:33.476 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.506+0000 7f22feffd700 1 --2- 172.21.15.67:0/2262980248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2300105470 0x7f230019c3d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f22f0003fc0 tx=0x7f22f00040a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:33.476 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.506+0000 7f22e7fff700 1 -- 172.21.15.67:0/2262980248 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22f000de20 con 0x7f2300105470 2024-04-22T23:14:33.476 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.506+0000 7f22e7fff700 1 -- 172.21.15.67:0/2262980248 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f22f0010be0 con 0x7f2300105470 2024-04-22T23:14:33.476 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.506+0000 7f23052d1700 1 -- 172.21.15.67:0/2262980248 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f230019cb10 con 0x7f2300105470 2024-04-22T23:14:33.476 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.506+0000 7f22e7fff700 1 -- 172.21.15.67:0/2262980248 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22f0021c50 con 0x7f2300105470 2024-04-22T23:14:33.476 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.506+0000 7f23052d1700 1 -- 172.21.15.67:0/2262980248 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f230019cf50 con 0x7f2300105470 2024-04-22T23:14:33.476 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.507+0000 7f22e7fff700 1 -- 172.21.15.67:0/2262980248 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) v1 ==== 45190+0+0 (secure 0 0 0) 0x7f22f00193f0 con 0x7f2300105470 2024-04-22T23:14:33.476 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.507+0000 7f23052d1700 1 -- 172.21.15.67:0/2262980248 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2300191d00 con 0x7f2300105470 2024-04-22T23:14:33.476 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.507+0000 7f22e7fff700 1 --2- 172.21.15.67:0/2262980248 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f22e80381f0 0x7f22e803a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:33.476 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.507+0000 7f22e7fff700 1 -- 172.21.15.67:0/2262980248 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f22f004c4f0 con 0x7f2300105470 2024-04-22T23:14:33.476 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.507+0000 7f22fe7fc700 1 --2- 172.21.15.67:0/2262980248 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f22e80381f0 0x7f22e803a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:33.476 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.508+0000 7f22fe7fc700 1 --2- 172.21.15.67:0/2262980248 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f22e80381f0 0x7f22e803a6b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f22f4006fd0 tx=0x7f22f4006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:33.477 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.512+0000 7f22e7fff700 1 -- 172.21.15.67:0/2262980248 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f22f000e950 con 0x7f2300105470 2024-04-22T23:14:33.477 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.621+0000 7f22e7fff700 1 -- 172.21.15.67:0/2262980248 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 12) v1 ==== 45296+0+0 (secure 0 0 0) 0x7f22f00203c0 con 0x7f2300105470 2024-04-22T23:14:33.477 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.733+0000 7f23052d1700 1 -- 172.21.15.67:0/2262980248 --> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f23000611d0 con 0x7f22e80381f0 2024-04-22T23:14:33.477 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.934+0000 7f22e7fff700 1 -- 172.21.15.67:0/2262980248 <== mgr.14164 v2:172.21.15.67:6800/2590792187 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f23000611d0 con 0x7f22e80381f0 2024-04-22T23:14:33.477 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.937+0000 7f23052d1700 1 -- 172.21.15.67:0/2262980248 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f22e80381f0 msgr2=0x7f22e803a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:33.478 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.937+0000 7f23052d1700 1 --2- 172.21.15.67:0/2262980248 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f22e80381f0 0x7f22e803a6b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f22f4006fd0 tx=0x7f22f4006e40 comp rx=0 tx=0).stop 2024-04-22T23:14:33.478 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.937+0000 7f23052d1700 1 -- 172.21.15.67:0/2262980248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2300105470 msgr2=0x7f230019c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:33.478 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.937+0000 7f23052d1700 1 --2- 172.21.15.67:0/2262980248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2300105470 0x7f230019c3d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f22f0003fc0 tx=0x7f22f00040a0 comp rx=0 tx=0).stop 2024-04-22T23:14:33.478 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.937+0000 7f23052d1700 1 -- 172.21.15.67:0/2262980248 shutdown_connections 2024-04-22T23:14:33.478 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.937+0000 7f23052d1700 1 --2- 172.21.15.67:0/2262980248 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f22e80381f0 0x7f22e803a6b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:33.478 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.937+0000 7f23052d1700 1 --2- 172.21.15.67:0/2262980248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2300105470 0x7f230019c3d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:33.478 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.937+0000 7f23052d1700 1 -- 172.21.15.67:0/2262980248 >> 172.21.15.67:0/2262980248 conn(0x7f23001009f0 msgr2=0x7f230018ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:33.478 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.937+0000 7f23052d1700 1 -- 172.21.15.67:0/2262980248 shutdown_connections 2024-04-22T23:14:33.478 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:31.937+0000 7f23052d1700 1 -- 172.21.15.67:0/2262980248 wait complete. 2024-04-22T23:14:33.478 INFO:teuthology.orchestra.run.smithi067.stdout:Creating initial admin user... 2024-04-22T23:14:34.397 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:34 smithi067 ceph-mon[26453]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:14:34.962 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$SFV3TeYdg6u62V58sDgBZ.8/60DuVOJin4yj9/8x7m0UIy2jdDsnu", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713827674, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-22T23:14:34.962 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.905+0000 7f4773586700 1 Processor -- start 2024-04-22T23:14:34.962 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.905+0000 7f4773586700 1 -- start start 2024-04-22T23:14:34.963 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.905+0000 7f4773586700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f476c07be60 0x7f476c07a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:34.963 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.905+0000 7f4773586700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f476c07a890 con 0x7f476c07be60 2024-04-22T23:14:34.963 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.905+0000 7f4771322700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f476c07be60 0x7f476c07a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:34.963 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.906+0000 7f4771322700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f476c07be60 0x7f476c07a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57096/0 (socket says 172.21.15.67:57096) 2024-04-22T23:14:34.963 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.906+0000 7f4771322700 1 -- 172.21.15.67:0/437524795 learned_addr learned my addr 172.21.15.67:0/437524795 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:34.963 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.906+0000 7f4771322700 1 -- 172.21.15.67:0/437524795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f476c07a9d0 con 0x7f476c07be60 2024-04-22T23:14:34.963 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.906+0000 7f4771322700 1 --2- 172.21.15.67:0/437524795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f476c07be60 0x7f476c07a2c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f47680098a0 tx=0x7f4768009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c1a4d37151e1706 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:34.963 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.906+0000 7f4763fff700 1 -- 172.21.15.67:0/437524795 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4768004180 con 0x7f476c07be60 2024-04-22T23:14:34.963 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.906+0000 7f4763fff700 1 -- 172.21.15.67:0/437524795 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f47680042e0 con 0x7f476c07be60 2024-04-22T23:14:34.963 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.906+0000 7f4763fff700 1 -- 172.21.15.67:0/437524795 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4768011af0 con 0x7f476c07be60 2024-04-22T23:14:34.963 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.907+0000 7f4773586700 1 -- 172.21.15.67:0/437524795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f476c07be60 msgr2=0x7f476c07a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:34.964 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.907+0000 7f4773586700 1 --2- 172.21.15.67:0/437524795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f476c07be60 0x7f476c07a2c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f47680098a0 tx=0x7f4768009bb0 comp rx=0 tx=0).stop 2024-04-22T23:14:34.964 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.907+0000 7f4773586700 1 -- 172.21.15.67:0/437524795 shutdown_connections 2024-04-22T23:14:34.964 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.907+0000 7f4773586700 1 --2- 172.21.15.67:0/437524795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f476c07be60 0x7f476c07a2c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:34.964 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.907+0000 7f4773586700 1 -- 172.21.15.67:0/437524795 >> 172.21.15.67:0/437524795 conn(0x7f476c101b40 msgr2=0x7f476c103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:34.964 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.907+0000 7f4773586700 1 -- 172.21.15.67:0/437524795 shutdown_connections 2024-04-22T23:14:34.964 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.907+0000 7f4773586700 1 -- 172.21.15.67:0/437524795 wait complete. 2024-04-22T23:14:34.964 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.907+0000 7f4773586700 1 Processor -- start 2024-04-22T23:14:34.964 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.908+0000 7f4773586700 1 -- start start 2024-04-22T23:14:34.964 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.908+0000 7f4773586700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f476c07be60 0x7f476c19c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:34.964 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.908+0000 7f4773586700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f476c19c970 con 0x7f476c07be60 2024-04-22T23:14:34.965 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.908+0000 7f4771322700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f476c07be60 0x7f476c19c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:34.965 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.908+0000 7f4771322700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f476c07be60 0x7f476c19c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57102/0 (socket says 172.21.15.67:57102) 2024-04-22T23:14:34.965 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.908+0000 7f4771322700 1 -- 172.21.15.67:0/2396197379 learned_addr learned my addr 172.21.15.67:0/2396197379 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:34.965 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.908+0000 7f4771322700 1 -- 172.21.15.67:0/2396197379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4768009550 con 0x7f476c07be60 2024-04-22T23:14:34.965 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.908+0000 7f4771322700 1 --2- 172.21.15.67:0/2396197379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f476c07be60 0x7f476c19c430 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4768011670 tx=0x7f4768011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:34.965 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.908+0000 7f47627fc700 1 -- 172.21.15.67:0/2396197379 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4768003a40 con 0x7f476c07be60 2024-04-22T23:14:34.965 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.909+0000 7f4773586700 1 -- 172.21.15.67:0/2396197379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f476c19cb70 con 0x7f476c07be60 2024-04-22T23:14:34.965 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.909+0000 7f47627fc700 1 -- 172.21.15.67:0/2396197379 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4768003ba0 con 0x7f476c07be60 2024-04-22T23:14:34.965 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.909+0000 7f47627fc700 1 -- 172.21.15.67:0/2396197379 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4768024790 con 0x7f476c07be60 2024-04-22T23:14:34.965 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.909+0000 7f4773586700 1 -- 172.21.15.67:0/2396197379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f476c19d010 con 0x7f476c07be60 2024-04-22T23:14:34.965 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.909+0000 7f47627fc700 1 -- 172.21.15.67:0/2396197379 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 12) v1 ==== 45296+0+0 (secure 0 0 0) 0x7f476802d3f0 con 0x7f476c07be60 2024-04-22T23:14:34.965 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.910+0000 7f47627fc700 1 --2- 172.21.15.67:0/2396197379 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f4758038370 0x7f475803a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:34.966 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.910+0000 7f47627fc700 1 -- 172.21.15.67:0/2396197379 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f476804c7a0 con 0x7f476c07be60 2024-04-22T23:14:34.966 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.910+0000 7f4773586700 1 -- 172.21.15.67:0/2396197379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f476c0623c0 con 0x7f476c07be60 2024-04-22T23:14:34.966 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.910+0000 7f4770b21700 1 --2- 172.21.15.67:0/2396197379 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f4758038370 0x7f475803a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:34.966 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.910+0000 7f4770b21700 1 --2- 172.21.15.67:0/2396197379 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f4758038370 0x7f475803a830 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f475c006fd0 tx=0x7f475c006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:34.966 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:33.915+0000 7f47627fc700 1 -- 172.21.15.67:0/2396197379 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f476802d740 con 0x7f476c07be60 2024-04-22T23:14:34.966 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:34.135+0000 7f4773586700 1 -- 172.21.15.67:0/2396197379 --> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f476c0621e0 con 0x7f4758038370 2024-04-22T23:14:34.966 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:34.396+0000 7f47627fc700 1 -- 172.21.15.67:0/2396197379 <== mgr.14164 v2:172.21.15.67:6800/2590792187 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f476c0621e0 con 0x7f4758038370 2024-04-22T23:14:34.966 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:34.398+0000 7f4773586700 1 -- 172.21.15.67:0/2396197379 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f4758038370 msgr2=0x7f475803a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:34.966 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:34.398+0000 7f4773586700 1 --2- 172.21.15.67:0/2396197379 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f4758038370 0x7f475803a830 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f475c006fd0 tx=0x7f475c006e40 comp rx=0 tx=0).stop 2024-04-22T23:14:34.966 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:34.398+0000 7f4773586700 1 -- 172.21.15.67:0/2396197379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f476c07be60 msgr2=0x7f476c19c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:34.966 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:34.398+0000 7f4773586700 1 --2- 172.21.15.67:0/2396197379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f476c07be60 0x7f476c19c430 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4768011670 tx=0x7f4768011b50 comp rx=0 tx=0).stop 2024-04-22T23:14:34.966 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:34.398+0000 7f4773586700 1 -- 172.21.15.67:0/2396197379 shutdown_connections 2024-04-22T23:14:34.966 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:34.398+0000 7f4773586700 1 --2- 172.21.15.67:0/2396197379 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f4758038370 0x7f475803a830 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:34.967 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:34.398+0000 7f4773586700 1 --2- 172.21.15.67:0/2396197379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f476c07be60 0x7f476c19c430 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:34.967 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:34.398+0000 7f4773586700 1 -- 172.21.15.67:0/2396197379 >> 172.21.15.67:0/2396197379 conn(0x7f476c101b40 msgr2=0x7f476c1031f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:34.967 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:34.398+0000 7f4773586700 1 -- 172.21.15.67:0/2396197379 shutdown_connections 2024-04-22T23:14:34.967 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:34.398+0000 7f4773586700 1 -- 172.21.15.67:0/2396197379 wait complete. 2024-04-22T23:14:34.967 INFO:teuthology.orchestra.run.smithi067.stdout:Fetching dashboard port number... 2024-04-22T23:14:35.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:35 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:36.160 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 8443 2024-04-22T23:14:36.161 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.369+0000 7f5b208fc700 1 Processor -- start 2024-04-22T23:14:36.161 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.369+0000 7f5b208fc700 1 -- start start 2024-04-22T23:14:36.161 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.369+0000 7f5b208fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b1c07bec0 0x7f5b1c07a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:36.161 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.369+0000 7f5b208fc700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c07a8f0 con 0x7f5b1c07bec0 2024-04-22T23:14:36.161 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.369+0000 7f5b1a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b1c07bec0 0x7f5b1c07a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:36.161 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.369+0000 7f5b1a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b1c07bec0 0x7f5b1c07a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57112/0 (socket says 172.21.15.67:57112) 2024-04-22T23:14:36.161 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.369+0000 7f5b1a59c700 1 -- 172.21.15.67:0/1676538448 learned_addr learned my addr 172.21.15.67:0/1676538448 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:36.161 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.370+0000 7f5b1a59c700 1 -- 172.21.15.67:0/1676538448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b1c07aa30 con 0x7f5b1c07bec0 2024-04-22T23:14:36.161 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.370+0000 7f5b1a59c700 1 --2- 172.21.15.67:0/1676538448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b1c07bec0 0x7f5b1c07a320 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c0098a0 tx=0x7f5b0c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b0f12a4e8e82ce1 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:36.161 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.370+0000 7f5b1959a700 1 -- 172.21.15.67:0/1676538448 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b0c004180 con 0x7f5b1c07bec0 2024-04-22T23:14:36.161 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.370+0000 7f5b1959a700 1 -- 172.21.15.67:0/1676538448 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5b0c0042e0 con 0x7f5b1c07bec0 2024-04-22T23:14:36.161 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.370+0000 7f5b1959a700 1 -- 172.21.15.67:0/1676538448 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b0c011af0 con 0x7f5b1c07bec0 2024-04-22T23:14:36.161 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.371+0000 7f5b208fc700 1 -- 172.21.15.67:0/1676538448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b1c07bec0 msgr2=0x7f5b1c07a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:36.162 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.371+0000 7f5b208fc700 1 --2- 172.21.15.67:0/1676538448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b1c07bec0 0x7f5b1c07a320 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c0098a0 tx=0x7f5b0c009bb0 comp rx=0 tx=0).stop 2024-04-22T23:14:36.162 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.371+0000 7f5b208fc700 1 -- 172.21.15.67:0/1676538448 shutdown_connections 2024-04-22T23:14:36.162 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.371+0000 7f5b208fc700 1 --2- 172.21.15.67:0/1676538448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b1c07bec0 0x7f5b1c07a320 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:36.162 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.371+0000 7f5b208fc700 1 -- 172.21.15.67:0/1676538448 >> 172.21.15.67:0/1676538448 conn(0x7f5b1c101b40 msgr2=0x7f5b1c103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:36.162 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.371+0000 7f5b208fc700 1 -- 172.21.15.67:0/1676538448 shutdown_connections 2024-04-22T23:14:36.162 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.371+0000 7f5b208fc700 1 -- 172.21.15.67:0/1676538448 wait complete. 2024-04-22T23:14:36.162 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.371+0000 7f5b208fc700 1 Processor -- start 2024-04-22T23:14:36.162 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.372+0000 7f5b208fc700 1 -- start start 2024-04-22T23:14:36.162 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.372+0000 7f5b208fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b1c07bec0 0x7f5b1c19c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:36.162 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.372+0000 7f5b208fc700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c19c980 con 0x7f5b1c07bec0 2024-04-22T23:14:36.162 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.372+0000 7f5b1a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b1c07bec0 0x7f5b1c19c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:36.162 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.372+0000 7f5b1a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b1c07bec0 0x7f5b1c19c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57124/0 (socket says 172.21.15.67:57124) 2024-04-22T23:14:36.162 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.372+0000 7f5b1a59c700 1 -- 172.21.15.67:0/1373893525 learned_addr learned my addr 172.21.15.67:0/1373893525 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:36.162 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.372+0000 7f5b1a59c700 1 -- 172.21.15.67:0/1373893525 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b0c009550 con 0x7f5b1c07bec0 2024-04-22T23:14:36.163 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.372+0000 7f5b1a59c700 1 --2- 172.21.15.67:0/1373893525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b1c07bec0 0x7f5b1c19c440 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c011670 tx=0x7f5b0c011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:36.163 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.372+0000 7f5b0b7fe700 1 -- 172.21.15.67:0/1373893525 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b0c0039f0 con 0x7f5b1c07bec0 2024-04-22T23:14:36.163 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.372+0000 7f5b0b7fe700 1 -- 172.21.15.67:0/1373893525 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5b0c003b50 con 0x7f5b1c07bec0 2024-04-22T23:14:36.163 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.372+0000 7f5b0b7fe700 1 -- 172.21.15.67:0/1373893525 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b0c0247c0 con 0x7f5b1c07bec0 2024-04-22T23:14:36.163 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.372+0000 7f5b208fc700 1 -- 172.21.15.67:0/1373893525 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b1c19cb80 con 0x7f5b1c07bec0 2024-04-22T23:14:36.163 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.372+0000 7f5b208fc700 1 -- 172.21.15.67:0/1373893525 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b1c19d020 con 0x7f5b1c07bec0 2024-04-22T23:14:36.163 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.373+0000 7f5b0b7fe700 1 -- 172.21.15.67:0/1373893525 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 12) v1 ==== 45296+0+0 (secure 0 0 0) 0x7f5b0c02d3f0 con 0x7f5b1c07bec0 2024-04-22T23:14:36.163 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.373+0000 7f5b208fc700 1 -- 172.21.15.67:0/1373893525 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b1c10c930 con 0x7f5b1c07bec0 2024-04-22T23:14:36.163 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.373+0000 7f5b0b7fe700 1 --2- 172.21.15.67:0/1373893525 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f5b040383c0 0x7f5b0403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:36.163 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.373+0000 7f5b0b7fe700 1 -- 172.21.15.67:0/1373893525 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5b0c04c5d0 con 0x7f5b1c07bec0 2024-04-22T23:14:36.163 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.374+0000 7f5b19d9b700 1 --2- 172.21.15.67:0/1373893525 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f5b040383c0 0x7f5b0403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:36.163 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.374+0000 7f5b19d9b700 1 --2- 172.21.15.67:0/1373893525 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f5b040383c0 0x7f5b0403a880 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5b10006fd0 tx=0x7f5b10006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:36.163 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.378+0000 7f5b0b7fe700 1 -- 172.21.15.67:0/1373893525 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f5b0c010970 con 0x7f5b1c07bec0 2024-04-22T23:14:36.163 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.573+0000 7f5b208fc700 1 -- 172.21.15.67:0/1373893525 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f5b1c10cb50 con 0x7f5b1c07bec0 2024-04-22T23:14:36.164 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.574+0000 7f5b0b7fe700 1 -- 172.21.15.67:0/1373893525 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f5b0c010380 con 0x7f5b1c07bec0 2024-04-22T23:14:36.164 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.576+0000 7f5b208fc700 1 -- 172.21.15.67:0/1373893525 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f5b040383c0 msgr2=0x7f5b0403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:36.164 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.576+0000 7f5b208fc700 1 --2- 172.21.15.67:0/1373893525 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f5b040383c0 0x7f5b0403a880 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5b10006fd0 tx=0x7f5b10006e40 comp rx=0 tx=0).stop 2024-04-22T23:14:36.164 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.576+0000 7f5b208fc700 1 -- 172.21.15.67:0/1373893525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b1c07bec0 msgr2=0x7f5b1c19c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:36.164 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.576+0000 7f5b208fc700 1 --2- 172.21.15.67:0/1373893525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b1c07bec0 0x7f5b1c19c440 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c011670 tx=0x7f5b0c011b50 comp rx=0 tx=0).stop 2024-04-22T23:14:36.164 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.576+0000 7f5b208fc700 1 -- 172.21.15.67:0/1373893525 shutdown_connections 2024-04-22T23:14:36.164 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.576+0000 7f5b208fc700 1 --2- 172.21.15.67:0/1373893525 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f5b040383c0 0x7f5b0403a880 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:36.164 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.576+0000 7f5b208fc700 1 --2- 172.21.15.67:0/1373893525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b1c07bec0 0x7f5b1c19c440 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:36.164 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.576+0000 7f5b208fc700 1 -- 172.21.15.67:0/1373893525 >> 172.21.15.67:0/1373893525 conn(0x7f5b1c101b40 msgr2=0x7f5b1c1031f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:36.164 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.576+0000 7f5b208fc700 1 -- 172.21.15.67:0/1373893525 shutdown_connections 2024-04-22T23:14:36.164 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:35.576+0000 7f5b208fc700 1 -- 172.21.15.67:0/1373893525 wait complete. 2024-04-22T23:14:36.169 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-22T23:14:36.169 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout disabled 2024-04-22T23:14:36.177 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-22T23:14:36.177 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout inactive 2024-04-22T23:14:36.177 INFO:teuthology.orchestra.run.smithi067.stdout:firewalld.service is not enabled 2024-04-22T23:14:36.177 INFO:teuthology.orchestra.run.smithi067.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-22T23:14:36.179 INFO:teuthology.orchestra.run.smithi067.stdout:Ceph Dashboard is now available at: 2024-04-22T23:14:36.179 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:14:36.179 INFO:teuthology.orchestra.run.smithi067.stdout: URL: https://smithi067.front.sepia.ceph.com:8443/ 2024-04-22T23:14:36.179 INFO:teuthology.orchestra.run.smithi067.stdout: User: admin 2024-04-22T23:14:36.179 INFO:teuthology.orchestra.run.smithi067.stdout: Password: f1mq5u20j1 2024-04-22T23:14:36.180 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:14:36.180 INFO:teuthology.orchestra.run.smithi067.stdout:Saving cluster configuration to /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config directory 2024-04-22T23:14:36.180 INFO:teuthology.orchestra.run.smithi067.stdout:Enabling autotune for osd_memory_target 2024-04-22T23:14:36.412 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:36 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/1373893525' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-22T23:14:36.412 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:36 smithi067 ceph-mon[26453]: mgrmap e13: smithi067.whdcsr(active, since 6s) 2024-04-22T23:14:37.407 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.640+0000 7f62b87da700 1 Processor -- start 2024-04-22T23:14:37.407 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.640+0000 7f62b87da700 1 -- start start 2024-04-22T23:14:37.407 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.640+0000 7f62b87da700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62b0106560 0x7f62b0106980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:37.408 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.640+0000 7f62b87da700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b0106f50 con 0x7f62b0106560 2024-04-22T23:14:37.408 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.640+0000 7f62b6576700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62b0106560 0x7f62b0106980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:37.408 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.640+0000 7f62b6576700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62b0106560 0x7f62b0106980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57134/0 (socket says 172.21.15.67:57134) 2024-04-22T23:14:37.408 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.640+0000 7f62b6576700 1 -- 172.21.15.67:0/1106337796 learned_addr learned my addr 172.21.15.67:0/1106337796 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:37.408 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.641+0000 7f62b6576700 1 -- 172.21.15.67:0/1106337796 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b0107760 con 0x7f62b0106560 2024-04-22T23:14:37.408 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.641+0000 7f62b6576700 1 --2- 172.21.15.67:0/1106337796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62b0106560 0x7f62b0106980 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f62ac0098a0 tx=0x7f62ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd10087f9aff683f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:37.408 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.641+0000 7f62b5574700 1 -- 172.21.15.67:0/1106337796 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62ac004180 con 0x7f62b0106560 2024-04-22T23:14:37.408 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.641+0000 7f62b5574700 1 -- 172.21.15.67:0/1106337796 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f62ac0042e0 con 0x7f62b0106560 2024-04-22T23:14:37.408 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.641+0000 7f62b5574700 1 -- 172.21.15.67:0/1106337796 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62ac011af0 con 0x7f62b0106560 2024-04-22T23:14:37.408 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.642+0000 7f62b87da700 1 -- 172.21.15.67:0/1106337796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62b0106560 msgr2=0x7f62b0106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:37.408 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.642+0000 7f62b87da700 1 --2- 172.21.15.67:0/1106337796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62b0106560 0x7f62b0106980 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f62ac0098a0 tx=0x7f62ac009bb0 comp rx=0 tx=0).stop 2024-04-22T23:14:37.408 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.642+0000 7f62b87da700 1 -- 172.21.15.67:0/1106337796 shutdown_connections 2024-04-22T23:14:37.408 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.642+0000 7f62b87da700 1 --2- 172.21.15.67:0/1106337796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62b0106560 0x7f62b0106980 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:37.408 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.642+0000 7f62b87da700 1 -- 172.21.15.67:0/1106337796 >> 172.21.15.67:0/1106337796 conn(0x7f62b0101b40 msgr2=0x7f62b0103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:37.409 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.642+0000 7f62b87da700 1 -- 172.21.15.67:0/1106337796 shutdown_connections 2024-04-22T23:14:37.409 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.642+0000 7f62b87da700 1 -- 172.21.15.67:0/1106337796 wait complete. 2024-04-22T23:14:37.409 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.642+0000 7f62b87da700 1 Processor -- start 2024-04-22T23:14:37.409 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.642+0000 7f62b87da700 1 -- start start 2024-04-22T23:14:37.409 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.643+0000 7f62b87da700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62b0106560 0x7f62b019a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:37.409 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.643+0000 7f62b87da700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b019a780 con 0x7f62b0106560 2024-04-22T23:14:37.409 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.643+0000 7f62b6576700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62b0106560 0x7f62b019a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:37.409 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.643+0000 7f62b6576700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62b0106560 0x7f62b019a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57140/0 (socket says 172.21.15.67:57140) 2024-04-22T23:14:37.409 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.643+0000 7f62b6576700 1 -- 172.21.15.67:0/1055411765 learned_addr learned my addr 172.21.15.67:0/1055411765 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:37.409 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.643+0000 7f62b6576700 1 -- 172.21.15.67:0/1055411765 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ac009550 con 0x7f62b0106560 2024-04-22T23:14:37.409 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.643+0000 7f62b6576700 1 --2- 172.21.15.67:0/1055411765 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62b0106560 0x7f62b019a240 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f62ac011670 tx=0x7f62ac011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:37.409 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.643+0000 7f62a77fe700 1 -- 172.21.15.67:0/1055411765 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62ac0039f0 con 0x7f62b0106560 2024-04-22T23:14:37.409 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.643+0000 7f62a77fe700 1 -- 172.21.15.67:0/1055411765 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f62ac003b50 con 0x7f62b0106560 2024-04-22T23:14:37.409 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.643+0000 7f62a77fe700 1 -- 172.21.15.67:0/1055411765 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62ac0247c0 con 0x7f62b0106560 2024-04-22T23:14:37.410 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.643+0000 7f62b87da700 1 -- 172.21.15.67:0/1055411765 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62b019a980 con 0x7f62b0106560 2024-04-22T23:14:37.410 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.644+0000 7f62b87da700 1 -- 172.21.15.67:0/1055411765 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62b019ae20 con 0x7f62b0106560 2024-04-22T23:14:37.410 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.645+0000 7f62a77fe700 1 -- 172.21.15.67:0/1055411765 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f62ac01a430 con 0x7f62b0106560 2024-04-22T23:14:37.410 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.645+0000 7f62b87da700 1 -- 172.21.15.67:0/1055411765 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62b00623c0 con 0x7f62b0106560 2024-04-22T23:14:37.410 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.645+0000 7f62a77fe700 1 --2- 172.21.15.67:0/1055411765 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f629c038460 0x7f629c03a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:37.410 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.645+0000 7f62a77fe700 1 -- 172.21.15.67:0/1055411765 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f62ac04c530 con 0x7f62b0106560 2024-04-22T23:14:37.410 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.645+0000 7f62b5d75700 1 --2- 172.21.15.67:0/1055411765 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f629c038460 0x7f629c03a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:37.410 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.645+0000 7f62b5d75700 1 --2- 172.21.15.67:0/1055411765 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f629c038460 0x7f629c03a920 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f62a0006fd0 tx=0x7f62a0006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:37.410 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.650+0000 7f62a77fe700 1 -- 172.21.15.67:0/1055411765 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f62ac018b40 con 0x7f62b0106560 2024-04-22T23:14:37.410 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.848+0000 7f62b87da700 1 -- 172.21.15.67:0/1055411765 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f62b010a960 con 0x7f62b0106560 2024-04-22T23:14:37.410 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.848+0000 7f62a77fe700 1 -- 172.21.15.67:0/1055411765 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f62ac018b40 con 0x7f62b0106560 2024-04-22T23:14:37.410 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.850+0000 7f62b87da700 1 -- 172.21.15.67:0/1055411765 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f629c038460 msgr2=0x7f629c03a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:37.410 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.850+0000 7f62b87da700 1 --2- 172.21.15.67:0/1055411765 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f629c038460 0x7f629c03a920 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f62a0006fd0 tx=0x7f62a0006e40 comp rx=0 tx=0).stop 2024-04-22T23:14:37.410 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.850+0000 7f62b87da700 1 -- 172.21.15.67:0/1055411765 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62b0106560 msgr2=0x7f62b019a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:37.411 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.850+0000 7f62b87da700 1 --2- 172.21.15.67:0/1055411765 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62b0106560 0x7f62b019a240 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f62ac011670 tx=0x7f62ac011b50 comp rx=0 tx=0).stop 2024-04-22T23:14:37.411 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.850+0000 7f62b87da700 1 -- 172.21.15.67:0/1055411765 shutdown_connections 2024-04-22T23:14:37.411 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.850+0000 7f62b87da700 1 --2- 172.21.15.67:0/1055411765 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f629c038460 0x7f629c03a920 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:37.411 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.850+0000 7f62b87da700 1 --2- 172.21.15.67:0/1055411765 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62b0106560 0x7f62b019a240 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:37.411 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.850+0000 7f62b87da700 1 -- 172.21.15.67:0/1055411765 >> 172.21.15.67:0/1055411765 conn(0x7f62b0101b40 msgr2=0x7f62b0102920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:37.411 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.850+0000 7f62b87da700 1 -- 172.21.15.67:0/1055411765 shutdown_connections 2024-04-22T23:14:37.411 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:36.850+0000 7f62b87da700 1 -- 172.21.15.67:0/1055411765 wait complete. 2024-04-22T23:14:38.706 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.836+0000 7f0caaec1700 1 Processor -- start 2024-04-22T23:14:38.706 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.836+0000 7f0caaec1700 1 -- start start 2024-04-22T23:14:38.706 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.837+0000 7f0caaec1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ca407be60 0x7f0ca407a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:38.706 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.837+0000 7f0caaec1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca407a890 con 0x7f0ca407be60 2024-04-22T23:14:38.706 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.837+0000 7f0ca8c5d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ca407be60 0x7f0ca407a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:38.706 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.837+0000 7f0ca8c5d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ca407be60 0x7f0ca407a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57154/0 (socket says 172.21.15.67:57154) 2024-04-22T23:14:38.706 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.837+0000 7f0ca8c5d700 1 -- 172.21.15.67:0/78119345 learned_addr learned my addr 172.21.15.67:0/78119345 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:38.706 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.837+0000 7f0ca8c5d700 1 -- 172.21.15.67:0/78119345 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca407a9d0 con 0x7f0ca407be60 2024-04-22T23:14:38.706 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.837+0000 7f0ca8c5d700 1 --2- 172.21.15.67:0/78119345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ca407be60 0x7f0ca407a2c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0c980098a0 tx=0x7f0c98009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ec59ef18576292c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:38.707 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.837+0000 7f0ca37fe700 1 -- 172.21.15.67:0/78119345 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c98004180 con 0x7f0ca407be60 2024-04-22T23:14:38.707 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.837+0000 7f0ca37fe700 1 -- 172.21.15.67:0/78119345 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0c980042e0 con 0x7f0ca407be60 2024-04-22T23:14:38.707 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.837+0000 7f0ca37fe700 1 -- 172.21.15.67:0/78119345 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c98011af0 con 0x7f0ca407be60 2024-04-22T23:14:38.707 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.838+0000 7f0caaec1700 1 -- 172.21.15.67:0/78119345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ca407be60 msgr2=0x7f0ca407a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:38.707 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.838+0000 7f0caaec1700 1 --2- 172.21.15.67:0/78119345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ca407be60 0x7f0ca407a2c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0c980098a0 tx=0x7f0c98009bb0 comp rx=0 tx=0).stop 2024-04-22T23:14:38.707 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.838+0000 7f0caaec1700 1 -- 172.21.15.67:0/78119345 shutdown_connections 2024-04-22T23:14:38.707 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.838+0000 7f0caaec1700 1 --2- 172.21.15.67:0/78119345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ca407be60 0x7f0ca407a2c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:38.707 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.838+0000 7f0caaec1700 1 -- 172.21.15.67:0/78119345 >> 172.21.15.67:0/78119345 conn(0x7f0ca4101b40 msgr2=0x7f0ca4103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:38.707 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.838+0000 7f0caaec1700 1 -- 172.21.15.67:0/78119345 shutdown_connections 2024-04-22T23:14:38.707 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.838+0000 7f0caaec1700 1 -- 172.21.15.67:0/78119345 wait complete. 2024-04-22T23:14:38.707 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.839+0000 7f0caaec1700 1 Processor -- start 2024-04-22T23:14:38.707 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.839+0000 7f0caaec1700 1 -- start start 2024-04-22T23:14:38.707 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.839+0000 7f0caaec1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ca407be60 0x7f0ca419c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:38.707 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.839+0000 7f0caaec1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca419c950 con 0x7f0ca407be60 2024-04-22T23:14:38.707 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.839+0000 7f0ca8c5d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ca407be60 0x7f0ca419c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:38.708 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.839+0000 7f0ca8c5d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ca407be60 0x7f0ca419c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57160/0 (socket says 172.21.15.67:57160) 2024-04-22T23:14:38.708 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.839+0000 7f0ca8c5d700 1 -- 172.21.15.67:0/627788374 learned_addr learned my addr 172.21.15.67:0/627788374 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:38.708 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.839+0000 7f0ca8c5d700 1 -- 172.21.15.67:0/627788374 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c98009550 con 0x7f0ca407be60 2024-04-22T23:14:38.708 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.839+0000 7f0ca8c5d700 1 --2- 172.21.15.67:0/627788374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ca407be60 0x7f0ca419c410 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0c98011670 tx=0x7f0c98011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:38.708 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.840+0000 7f0ca1ffb700 1 -- 172.21.15.67:0/627788374 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c98003a40 con 0x7f0ca407be60 2024-04-22T23:14:38.708 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.840+0000 7f0ca1ffb700 1 -- 172.21.15.67:0/627788374 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0c98003ba0 con 0x7f0ca407be60 2024-04-22T23:14:38.708 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.840+0000 7f0caaec1700 1 -- 172.21.15.67:0/627788374 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ca419cb50 con 0x7f0ca407be60 2024-04-22T23:14:38.708 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.840+0000 7f0ca1ffb700 1 -- 172.21.15.67:0/627788374 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c98024790 con 0x7f0ca407be60 2024-04-22T23:14:38.708 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.840+0000 7f0caaec1700 1 -- 172.21.15.67:0/627788374 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ca419cff0 con 0x7f0ca407be60 2024-04-22T23:14:38.708 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.841+0000 7f0ca1ffb700 1 -- 172.21.15.67:0/627788374 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f0c9802d3f0 con 0x7f0ca407be60 2024-04-22T23:14:38.708 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.841+0000 7f0caaec1700 1 -- 172.21.15.67:0/627788374 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ca410c940 con 0x7f0ca407be60 2024-04-22T23:14:38.708 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.841+0000 7f0ca1ffb700 1 --2- 172.21.15.67:0/627788374 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f0c94038460 0x7f0c9403a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:38.708 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.841+0000 7f0ca1ffb700 1 -- 172.21.15.67:0/627788374 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0c9804c740 con 0x7f0ca407be60 2024-04-22T23:14:38.708 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.841+0000 7f0ca3fff700 1 --2- 172.21.15.67:0/627788374 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f0c94038460 0x7f0c9403a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:38.709 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.842+0000 7f0ca3fff700 1 --2- 172.21.15.67:0/627788374 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f0c94038460 0x7f0c9403a920 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0c90006fd0 tx=0x7f0c90006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:38.709 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:37.846+0000 7f0ca1ffb700 1 -- 172.21.15.67:0/627788374 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f0c98010380 con 0x7f0ca407be60 2024-04-22T23:14:38.709 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:38.155+0000 7f0caaec1700 1 -- 172.21.15.67:0/627788374 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f0ca419fdd0 con 0x7f0ca407be60 2024-04-22T23:14:38.709 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:38.157+0000 7f0ca1ffb700 1 -- 172.21.15.67:0/627788374 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v34)=0 set mgr/dashboard/cluster/status v34) v1 ==== 153+0+0 (secure 0 0 0) 0x7f0c98010380 con 0x7f0ca407be60 2024-04-22T23:14:38.709 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:38.159+0000 7f0caaec1700 1 -- 172.21.15.67:0/627788374 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f0c94038460 msgr2=0x7f0c9403a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:38.709 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:38.159+0000 7f0caaec1700 1 --2- 172.21.15.67:0/627788374 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f0c94038460 0x7f0c9403a920 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0c90006fd0 tx=0x7f0c90006e40 comp rx=0 tx=0).stop 2024-04-22T23:14:38.709 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:38.159+0000 7f0caaec1700 1 -- 172.21.15.67:0/627788374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ca407be60 msgr2=0x7f0ca419c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:38.709 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:38.159+0000 7f0caaec1700 1 --2- 172.21.15.67:0/627788374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ca407be60 0x7f0ca419c410 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0c98011670 tx=0x7f0c98011b50 comp rx=0 tx=0).stop 2024-04-22T23:14:38.709 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:38.159+0000 7f0caaec1700 1 -- 172.21.15.67:0/627788374 shutdown_connections 2024-04-22T23:14:38.709 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:38.159+0000 7f0caaec1700 1 --2- 172.21.15.67:0/627788374 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f0c94038460 0x7f0c9403a920 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:38.709 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:38.159+0000 7f0caaec1700 1 --2- 172.21.15.67:0/627788374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ca407be60 0x7f0ca419c410 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:38.709 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:38.159+0000 7f0caaec1700 1 -- 172.21.15.67:0/627788374 >> 172.21.15.67:0/627788374 conn(0x7f0ca4101b40 msgr2=0x7f0ca41028d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:38.709 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:38.159+0000 7f0caaec1700 1 -- 172.21.15.67:0/627788374 shutdown_connections 2024-04-22T23:14:38.709 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-04-22T23:14:38.159+0000 7f0caaec1700 1 -- 172.21.15.67:0/627788374 wait complete. 2024-04-22T23:14:38.709 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-22T23:14:38.709 INFO:teuthology.orchestra.run.smithi067.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-22T23:14:38.710 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:14:38.710 INFO:teuthology.orchestra.run.smithi067.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-22T23:14:38.710 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:14:38.710 INFO:teuthology.orchestra.run.smithi067.stdout:Or, if you are only running a single cluster on this host: 2024-04-22T23:14:38.710 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:14:38.710 INFO:teuthology.orchestra.run.smithi067.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-22T23:14:38.710 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:14:38.710 INFO:teuthology.orchestra.run.smithi067.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-22T23:14:38.711 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:14:38.711 INFO:teuthology.orchestra.run.smithi067.stdout: ceph telemetry on 2024-04-22T23:14:38.711 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:14:38.711 INFO:teuthology.orchestra.run.smithi067.stdout:For more information see: 2024-04-22T23:14:38.711 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:14:38.711 INFO:teuthology.orchestra.run.smithi067.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-22T23:14:38.711 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:14:38.711 INFO:teuthology.orchestra.run.smithi067.stdout:Bootstrap complete. 2024-04-22T23:14:38.737 INFO:tasks.cephadm:Fetching config... 2024-04-22T23:14:38.737 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-22T23:14:38.737 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-22T23:14:38.754 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-22T23:14:38.754 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-22T23:14:38.755 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-22T23:14:38.810 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-22T23:14:38.810 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-22T23:14:38.810 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/keyring of=/dev/stdout 2024-04-22T23:14:38.875 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-22T23:14:38.875 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-22T23:14:38.875 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-22T23:14:38.930 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-22T23:14:38.930 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwSNMj7/MPduD49iqXpH+MkRm1oc8wB9SeMsBzzaC4r0zh6bNpyWPmIIxpWNQ8fMhKWLKM0D3J8i3pAbJvdhTx38gvjl6FOGkKpr1DrnnbLdcBsp4kbOSQpoH/q9rVkOyggQpKOfAGanOjqP3dWUJNJjR8RkPT16b4itFLVjDZDLS5tBuAPu1DtCEs0g5xqLFpqf0Zfry6Krug7LsVWc5FnKi5Slo645rk9EMsp8Lplp+D4s6faMnjpSWuo7tngJUuYDm3vRrdvCqJLATWmVFJpnIfTerYAKgT88XZf0lU7HywvcLWVc/7K2PDnp2V/6xg0bPOMIWtYZKM0BV9y7CFVr1vgQenovZl+J8NxWHwx9VovWruiOEViYY5SmBwFLb8fkHbLlzdIg7DKib4MZtAMXlvt3ZRUZB0eAkr/reGVugBrD0w9tUD37Ch/uqEfSiegV9r1kCU3gQRgWb27yuoun8roytesWJf/63fPPRvpZtm2GaVM6wsPC+ZOvxn9nc= ceph-d46c346e-00fd-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-22T23:14:39.003 INFO:teuthology.orchestra.run.smithi067.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwSNMj7/MPduD49iqXpH+MkRm1oc8wB9SeMsBzzaC4r0zh6bNpyWPmIIxpWNQ8fMhKWLKM0D3J8i3pAbJvdhTx38gvjl6FOGkKpr1DrnnbLdcBsp4kbOSQpoH/q9rVkOyggQpKOfAGanOjqP3dWUJNJjR8RkPT16b4itFLVjDZDLS5tBuAPu1DtCEs0g5xqLFpqf0Zfry6Krug7LsVWc5FnKi5Slo645rk9EMsp8Lplp+D4s6faMnjpSWuo7tngJUuYDm3vRrdvCqJLATWmVFJpnIfTerYAKgT88XZf0lU7HywvcLWVc/7K2PDnp2V/6xg0bPOMIWtYZKM0BV9y7CFVr1vgQenovZl+J8NxWHwx9VovWruiOEViYY5SmBwFLb8fkHbLlzdIg7DKib4MZtAMXlvt3ZRUZB0eAkr/reGVugBrD0w9tUD37Ch/uqEfSiegV9r1kCU3gQRgWb27yuoun8roytesWJf/63fPPRvpZtm2GaVM6wsPC+ZOvxn9nc= ceph-d46c346e-00fd-11ef-bc93-c7b262605968 2024-04-22T23:14:39.018 DEBUG:teuthology.orchestra.run.smithi134:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwSNMj7/MPduD49iqXpH+MkRm1oc8wB9SeMsBzzaC4r0zh6bNpyWPmIIxpWNQ8fMhKWLKM0D3J8i3pAbJvdhTx38gvjl6FOGkKpr1DrnnbLdcBsp4kbOSQpoH/q9rVkOyggQpKOfAGanOjqP3dWUJNJjR8RkPT16b4itFLVjDZDLS5tBuAPu1DtCEs0g5xqLFpqf0Zfry6Krug7LsVWc5FnKi5Slo645rk9EMsp8Lplp+D4s6faMnjpSWuo7tngJUuYDm3vRrdvCqJLATWmVFJpnIfTerYAKgT88XZf0lU7HywvcLWVc/7K2PDnp2V/6xg0bPOMIWtYZKM0BV9y7CFVr1vgQenovZl+J8NxWHwx9VovWruiOEViYY5SmBwFLb8fkHbLlzdIg7DKib4MZtAMXlvt3ZRUZB0eAkr/reGVugBrD0w9tUD37Ch/uqEfSiegV9r1kCU3gQRgWb27yuoun8roytesWJf/63fPPRvpZtm2GaVM6wsPC+ZOvxn9nc= ceph-d46c346e-00fd-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-22T23:14:39.057 INFO:teuthology.orchestra.run.smithi134.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwSNMj7/MPduD49iqXpH+MkRm1oc8wB9SeMsBzzaC4r0zh6bNpyWPmIIxpWNQ8fMhKWLKM0D3J8i3pAbJvdhTx38gvjl6FOGkKpr1DrnnbLdcBsp4kbOSQpoH/q9rVkOyggQpKOfAGanOjqP3dWUJNJjR8RkPT16b4itFLVjDZDLS5tBuAPu1DtCEs0g5xqLFpqf0Zfry6Krug7LsVWc5FnKi5Slo645rk9EMsp8Lplp+D4s6faMnjpSWuo7tngJUuYDm3vRrdvCqJLATWmVFJpnIfTerYAKgT88XZf0lU7HywvcLWVc/7K2PDnp2V/6xg0bPOMIWtYZKM0BV9y7CFVr1vgQenovZl+J8NxWHwx9VovWruiOEViYY5SmBwFLb8fkHbLlzdIg7DKib4MZtAMXlvt3ZRUZB0eAkr/reGVugBrD0w9tUD37Ch/uqEfSiegV9r1kCU3gQRgWb27yuoun8roytesWJf/63fPPRvpZtm2GaVM6wsPC+ZOvxn9nc= ceph-d46c346e-00fd-11ef-bc93-c7b262605968 2024-04-22T23:14:39.069 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-22T23:14:39.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:39 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/627788374' entity='client.admin' 2024-04-22T23:14:39.302 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:14:41.760 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:41 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:41.760 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:41 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:41.760 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:41 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "who": "osd/host:smithi067", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:14:41.761 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:41 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:41.761 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:41 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi067", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T23:14:41.761 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:41 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi067", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-22T23:14:41.761 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:41 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:14:41.761 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:41 smithi067 ceph-mon[26453]: Deploying daemon ceph-exporter.smithi067 on smithi067 2024-04-22T23:14:41.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:41.874+0000 7f144b195700 1 -- 172.21.15.67:0/586089216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1444101310 msgr2=0x7f1444103700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:41.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:41.874+0000 7f144b195700 1 --2- 172.21.15.67:0/586089216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1444101310 0x7f1444103700 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1438009910 tx=0x7f1438009c20 comp rx=0 tx=0).stop 2024-04-22T23:14:41.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:41.874+0000 7f144b195700 1 -- 172.21.15.67:0/586089216 shutdown_connections 2024-04-22T23:14:41.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:41.874+0000 7f144b195700 1 --2- 172.21.15.67:0/586089216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1444101310 0x7f1444103700 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:41.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:41.874+0000 7f144b195700 1 -- 172.21.15.67:0/586089216 >> 172.21.15.67:0/586089216 conn(0x7f14440fad20 msgr2=0x7f14440fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:41.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:41.874+0000 7f144b195700 1 -- 172.21.15.67:0/586089216 shutdown_connections 2024-04-22T23:14:41.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:41.874+0000 7f144b195700 1 -- 172.21.15.67:0/586089216 wait complete. 2024-04-22T23:14:41.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:41.875+0000 7f144b195700 1 Processor -- start 2024-04-22T23:14:41.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:41.875+0000 7f144b195700 1 -- start start 2024-04-22T23:14:41.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:41.875+0000 7f144b195700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1444101310 0x7f1444100c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:41.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:41.875+0000 7f144b195700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14441011b0 con 0x7f1444101310 2024-04-22T23:14:41.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:41.875+0000 7f1448f31700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1444101310 0x7f1444100c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:41.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:41.875+0000 7f1448f31700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1444101310 0x7f1444100c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52750/0 (socket says 172.21.15.67:52750) 2024-04-22T23:14:41.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:41.875+0000 7f1448f31700 1 -- 172.21.15.67:0/2866163106 learned_addr learned my addr 172.21.15.67:0/2866163106 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:41.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:41.876+0000 7f1448f31700 1 -- 172.21.15.67:0/2866163106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14380095f0 con 0x7f1444101310 2024-04-22T23:14:41.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:41.876+0000 7f1448f31700 1 --2- 172.21.15.67:0/2866163106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1444101310 0x7f1444100c70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1438005b40 tx=0x7f1438004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:41.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:41.876+0000 7f1441ffb700 1 -- 172.21.15.67:0/2866163106 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f143801d070 con 0x7f1444101310 2024-04-22T23:14:41.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:41.876+0000 7f1441ffb700 1 -- 172.21.15.67:0/2866163106 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f143800be10 con 0x7f1444101310 2024-04-22T23:14:41.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:41.876+0000 7f1441ffb700 1 -- 172.21.15.67:0/2866163106 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f143800f460 con 0x7f1444101310 2024-04-22T23:14:41.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:41.876+0000 7f144b195700 1 -- 172.21.15.67:0/2866163106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14440ff2c0 con 0x7f1444101310 2024-04-22T23:14:41.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:41.876+0000 7f144b195700 1 -- 172.21.15.67:0/2866163106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14440ff760 con 0x7f1444101310 2024-04-22T23:14:41.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:41.877+0000 7f1441ffb700 1 -- 172.21.15.67:0/2866163106 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f14380227e0 con 0x7f1444101310 2024-04-22T23:14:41.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:41.877+0000 7f1441ffb700 1 --2- 172.21.15.67:0/2866163106 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f14340383c0 0x7f143403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:41.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:41.877+0000 7f144b195700 1 -- 172.21.15.67:0/2866163106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14440623c0 con 0x7f1444101310 2024-04-22T23:14:41.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:41.877+0000 7f1441ffb700 1 -- 172.21.15.67:0/2866163106 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f143804c4c0 con 0x7f1444101310 2024-04-22T23:14:41.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:41.878+0000 7f1443fff700 1 --2- 172.21.15.67:0/2866163106 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f14340383c0 0x7f143403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:41.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:41.878+0000 7f1443fff700 1 --2- 172.21.15.67:0/2866163106 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f14340383c0 0x7f143403a880 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1430006fd0 tx=0x7f1430006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:41.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:41.883+0000 7f1441ffb700 1 -- 172.21.15.67:0/2866163106 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f1438027030 con 0x7f1444101310 2024-04-22T23:14:42.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:42.089+0000 7f144b195700 1 -- 172.21.15.67:0/2866163106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f144418f380 con 0x7f1444101310 2024-04-22T23:14:42.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:42.092+0000 7f1441ffb700 1 -- 172.21.15.67:0/2866163106 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f143802bba0 con 0x7f1444101310 2024-04-22T23:14:42.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:42.094+0000 7f144b195700 1 -- 172.21.15.67:0/2866163106 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f14340383c0 msgr2=0x7f143403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:42.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:42.094+0000 7f144b195700 1 --2- 172.21.15.67:0/2866163106 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f14340383c0 0x7f143403a880 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1430006fd0 tx=0x7f1430006e40 comp rx=0 tx=0).stop 2024-04-22T23:14:42.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:42.094+0000 7f144b195700 1 -- 172.21.15.67:0/2866163106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1444101310 msgr2=0x7f1444100c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:42.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:42.094+0000 7f144b195700 1 --2- 172.21.15.67:0/2866163106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1444101310 0x7f1444100c70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1438005b40 tx=0x7f1438004740 comp rx=0 tx=0).stop 2024-04-22T23:14:42.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:42.094+0000 7f144b195700 1 -- 172.21.15.67:0/2866163106 shutdown_connections 2024-04-22T23:14:42.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:42.094+0000 7f144b195700 1 --2- 172.21.15.67:0/2866163106 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f14340383c0 0x7f143403a880 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:42.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:42.094+0000 7f144b195700 1 --2- 172.21.15.67:0/2866163106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1444101310 0x7f1444100c70 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:42.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:42.094+0000 7f144b195700 1 -- 172.21.15.67:0/2866163106 >> 172.21.15.67:0/2866163106 conn(0x7f14440fad20 msgr2=0x7f14440fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:42.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:42.094+0000 7f144b195700 1 -- 172.21.15.67:0/2866163106 shutdown_connections 2024-04-22T23:14:42.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:42.094+0000 7f144b195700 1 -- 172.21.15.67:0/2866163106 wait complete. 2024-04-22T23:14:42.688 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-22T23:14:42.688 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-22T23:14:43.236 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:14:43.352 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:43 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/2866163106' entity='client.admin' 2024-04-22T23:14:45.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.613+0000 7ff9cf3aa700 1 -- 172.21.15.67:0/3660436059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9c8100880 msgr2=0x7ff9c8100ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:45.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.613+0000 7ff9cf3aa700 1 --2- 172.21.15.67:0/3660436059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9c8100880 0x7ff9c8100ca0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff9c4009910 tx=0x7ff9c4009c20 comp rx=0 tx=0).stop 2024-04-22T23:14:45.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.613+0000 7ff9cf3aa700 1 -- 172.21.15.67:0/3660436059 shutdown_connections 2024-04-22T23:14:45.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.613+0000 7ff9cf3aa700 1 --2- 172.21.15.67:0/3660436059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9c8100880 0x7ff9c8100ca0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:45.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.613+0000 7ff9cf3aa700 1 -- 172.21.15.67:0/3660436059 >> 172.21.15.67:0/3660436059 conn(0x7ff9c80fbe00 msgr2=0x7ff9c80fe260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:45.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.613+0000 7ff9cf3aa700 1 -- 172.21.15.67:0/3660436059 shutdown_connections 2024-04-22T23:14:45.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.613+0000 7ff9cf3aa700 1 -- 172.21.15.67:0/3660436059 wait complete. 2024-04-22T23:14:45.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.614+0000 7ff9cf3aa700 1 Processor -- start 2024-04-22T23:14:45.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.614+0000 7ff9cf3aa700 1 -- start start 2024-04-22T23:14:45.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.614+0000 7ff9cf3aa700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9c8100880 0x7ff9c81959d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:45.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.614+0000 7ff9cf3aa700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c8195f10 con 0x7ff9c8100880 2024-04-22T23:14:45.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.614+0000 7ff9cd146700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9c8100880 0x7ff9c81959d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:45.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.614+0000 7ff9cd146700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9c8100880 0x7ff9c81959d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52764/0 (socket says 172.21.15.67:52764) 2024-04-22T23:14:45.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.614+0000 7ff9cd146700 1 -- 172.21.15.67:0/3152928394 learned_addr learned my addr 172.21.15.67:0/3152928394 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:45.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.614+0000 7ff9cd146700 1 -- 172.21.15.67:0/3152928394 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c40095f0 con 0x7ff9c8100880 2024-04-22T23:14:45.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.615+0000 7ff9cd146700 1 --2- 172.21.15.67:0/3152928394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9c8100880 0x7ff9c81959d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff9c4005b40 tx=0x7ff9c4004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:45.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.615+0000 7ff9be7fc700 1 -- 172.21.15.67:0/3152928394 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9c401d070 con 0x7ff9c8100880 2024-04-22T23:14:45.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.615+0000 7ff9be7fc700 1 -- 172.21.15.67:0/3152928394 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff9c400be10 con 0x7ff9c8100880 2024-04-22T23:14:45.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.615+0000 7ff9be7fc700 1 -- 172.21.15.67:0/3152928394 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9c400f460 con 0x7ff9c8100880 2024-04-22T23:14:45.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.615+0000 7ff9cf3aa700 1 -- 172.21.15.67:0/3152928394 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9c8196110 con 0x7ff9c8100880 2024-04-22T23:14:45.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.615+0000 7ff9cf3aa700 1 -- 172.21.15.67:0/3152928394 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9c81965b0 con 0x7ff9c8100880 2024-04-22T23:14:45.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.616+0000 7ff9cf3aa700 1 -- 172.21.15.67:0/3152928394 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9c818f4a0 con 0x7ff9c8100880 2024-04-22T23:14:45.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.616+0000 7ff9be7fc700 1 -- 172.21.15.67:0/3152928394 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7ff9c400b960 con 0x7ff9c8100880 2024-04-22T23:14:45.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.616+0000 7ff9be7fc700 1 --2- 172.21.15.67:0/3152928394 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7ff9b4038410 0x7ff9b403a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:45.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.617+0000 7ff9be7fc700 1 -- 172.21.15.67:0/3152928394 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff9c404c510 con 0x7ff9c8100880 2024-04-22T23:14:45.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.617+0000 7ff9cc945700 1 --2- 172.21.15.67:0/3152928394 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7ff9b4038410 0x7ff9b403a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:45.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.617+0000 7ff9cc945700 1 --2- 172.21.15.67:0/3152928394 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7ff9b4038410 0x7ff9b403a8d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff9b8006fd0 tx=0x7ff9b8006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:45.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.624+0000 7ff9be7fc700 1 -- 172.21.15.67:0/3152928394 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff9c402bb60 con 0x7ff9c8100880 2024-04-22T23:14:45.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.832+0000 7ff9cf3aa700 1 -- 172.21.15.67:0/3152928394 --> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7ff9c8061260 con 0x7ff9b4038410 2024-04-22T23:14:45.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.836+0000 7ff9be7fc700 1 -- 172.21.15.67:0/3152928394 <== mgr.14164 v2:172.21.15.67:6800/2590792187 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff9c8061260 con 0x7ff9b4038410 2024-04-22T23:14:45.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.838+0000 7ff9cf3aa700 1 -- 172.21.15.67:0/3152928394 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7ff9b4038410 msgr2=0x7ff9b403a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:45.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.838+0000 7ff9cf3aa700 1 --2- 172.21.15.67:0/3152928394 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7ff9b4038410 0x7ff9b403a8d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff9b8006fd0 tx=0x7ff9b8006e40 comp rx=0 tx=0).stop 2024-04-22T23:14:45.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.838+0000 7ff9cf3aa700 1 -- 172.21.15.67:0/3152928394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9c8100880 msgr2=0x7ff9c81959d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:45.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.838+0000 7ff9cf3aa700 1 --2- 172.21.15.67:0/3152928394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9c8100880 0x7ff9c81959d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff9c4005b40 tx=0x7ff9c4004740 comp rx=0 tx=0).stop 2024-04-22T23:14:45.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.838+0000 7ff9cf3aa700 1 -- 172.21.15.67:0/3152928394 shutdown_connections 2024-04-22T23:14:45.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.838+0000 7ff9cf3aa700 1 --2- 172.21.15.67:0/3152928394 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7ff9b4038410 0x7ff9b403a8d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:45.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.838+0000 7ff9cf3aa700 1 --2- 172.21.15.67:0/3152928394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9c8100880 0x7ff9c81959d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:45.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.838+0000 7ff9cf3aa700 1 -- 172.21.15.67:0/3152928394 >> 172.21.15.67:0/3152928394 conn(0x7ff9c80fbe00 msgr2=0x7ff9c80fcae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:45.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.838+0000 7ff9cf3aa700 1 -- 172.21.15.67:0/3152928394 shutdown_connections 2024-04-22T23:14:45.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:45.838+0000 7ff9cf3aa700 1 -- 172.21.15.67:0/3152928394 wait complete. 2024-04-22T23:14:46.529 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi134 2024-04-22T23:14:46.530 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-22T23:14:46.530 DEBUG:teuthology.orchestra.run.smithi134:> dd of=/etc/ceph/ceph.conf 2024-04-22T23:14:46.549 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-22T23:14:46.549 DEBUG:teuthology.orchestra.run.smithi134:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T23:14:46.605 INFO:tasks.cephadm:Adding host smithi134 to orchestrator... 2024-04-22T23:14:46.606 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph orch host add smithi134 2024-04-22T23:14:46.880 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:14:47.003 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:46 smithi067 ceph-mon[26453]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:14:47.003 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:46 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:48.013 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:47 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:48.013 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:47 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:48.013 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:47 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:48.013 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:47 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:48.013 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:47 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi067", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T23:14:48.014 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:47 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi067", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-22T23:14:48.014 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:47 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:14:48.014 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:47 smithi067 ceph-mon[26453]: Deploying daemon crash.smithi067 on smithi067 2024-04-22T23:14:48.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:48.211+0000 7fc961edc700 1 -- 172.21.15.67:0/4286616272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc95c0ff780 msgr2=0x7fc95c0ffba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:48.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:48.211+0000 7fc961edc700 1 --2- 172.21.15.67:0/4286616272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc95c0ff780 0x7fc95c0ffba0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc94c00b0a0 tx=0x7fc94c00b3b0 comp rx=0 tx=0).stop 2024-04-22T23:14:48.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:48.211+0000 7fc961edc700 1 -- 172.21.15.67:0/4286616272 shutdown_connections 2024-04-22T23:14:48.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:48.211+0000 7fc961edc700 1 --2- 172.21.15.67:0/4286616272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc95c0ff780 0x7fc95c0ffba0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:48.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:48.211+0000 7fc961edc700 1 -- 172.21.15.67:0/4286616272 >> 172.21.15.67:0/4286616272 conn(0x7fc95c0fad20 msgr2=0x7fc95c0fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:48.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:48.211+0000 7fc961edc700 1 -- 172.21.15.67:0/4286616272 shutdown_connections 2024-04-22T23:14:48.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:48.211+0000 7fc961edc700 1 -- 172.21.15.67:0/4286616272 wait complete. 2024-04-22T23:14:48.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:48.212+0000 7fc961edc700 1 Processor -- start 2024-04-22T23:14:48.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:48.212+0000 7fc961edc700 1 -- start start 2024-04-22T23:14:48.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:48.212+0000 7fc961edc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc95c0ff780 0x7fc95c197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:48.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:48.212+0000 7fc961edc700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc95c1980e0 con 0x7fc95c0ff780 2024-04-22T23:14:48.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:48.212+0000 7fc95b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc95c0ff780 0x7fc95c197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:48.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:48.212+0000 7fc95b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc95c0ff780 0x7fc95c197ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52778/0 (socket says 172.21.15.67:52778) 2024-04-22T23:14:48.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:48.212+0000 7fc95b7fe700 1 -- 172.21.15.67:0/2144593871 learned_addr learned my addr 172.21.15.67:0/2144593871 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:48.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:48.212+0000 7fc95b7fe700 1 -- 172.21.15.67:0/2144593871 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc94c009d00 con 0x7fc95c0ff780 2024-04-22T23:14:48.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:48.213+0000 7fc95b7fe700 1 --2- 172.21.15.67:0/2144593871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc95c0ff780 0x7fc95c197ba0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc94c009fd0 tx=0x7fc94c00bb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:48.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:48.213+0000 7fc958ff9700 1 -- 172.21.15.67:0/2144593871 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc94c00e040 con 0x7fc95c0ff780 2024-04-22T23:14:48.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:48.213+0000 7fc958ff9700 1 -- 172.21.15.67:0/2144593871 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc94c00bdc0 con 0x7fc95c0ff780 2024-04-22T23:14:48.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:48.213+0000 7fc958ff9700 1 -- 172.21.15.67:0/2144593871 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc94c01da40 con 0x7fc95c0ff780 2024-04-22T23:14:48.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:48.213+0000 7fc961edc700 1 -- 172.21.15.67:0/2144593871 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc95c1982e0 con 0x7fc95c0ff780 2024-04-22T23:14:48.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:48.213+0000 7fc961edc700 1 -- 172.21.15.67:0/2144593871 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc95c198700 con 0x7fc95c0ff780 2024-04-22T23:14:48.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:48.214+0000 7fc958ff9700 1 -- 172.21.15.67:0/2144593871 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7fc94c01dc80 con 0x7fc95c0ff780 2024-04-22T23:14:48.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:48.214+0000 7fc961edc700 1 -- 172.21.15.67:0/2144593871 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc95c1917b0 con 0x7fc95c0ff780 2024-04-22T23:14:48.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:48.214+0000 7fc958ff9700 1 --2- 172.21.15.67:0/2144593871 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7fc948038410 0x7fc94803a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:48.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:48.214+0000 7fc958ff9700 1 -- 172.21.15.67:0/2144593871 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc94c04c7a0 con 0x7fc95c0ff780 2024-04-22T23:14:48.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:48.214+0000 7fc95affd700 1 --2- 172.21.15.67:0/2144593871 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7fc948038410 0x7fc94803a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:48.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:48.215+0000 7fc95affd700 1 --2- 172.21.15.67:0/2144593871 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7fc948038410 0x7fc94803a8d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc944006fd0 tx=0x7fc944006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:48.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:48.219+0000 7fc958ff9700 1 -- 172.21.15.67:0/2144593871 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc94c01c5c0 con 0x7fc95c0ff780 2024-04-22T23:14:48.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:48.429+0000 7fc961edc700 1 -- 172.21.15.67:0/2144593871 --> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi134", "target": ["mon-mgr", ""]}) v1 -- 0x7fc95c0611d0 con 0x7fc948038410 2024-04-22T23:14:50.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:50 smithi067 ceph-mon[26453]: from='client.14191 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi134", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:14:50.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:50 smithi067 ceph-mon[26453]: Deploying cephadm binary to smithi134 2024-04-22T23:14:50.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:50 smithi067 ceph-mon[26453]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:14:50.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:50 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:50.998 INFO:teuthology.orchestra.run.smithi067.stdout:Added host 'smithi134' with addr '172.21.15.134' 2024-04-22T23:14:50.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:50.996+0000 7fc958ff9700 1 -- 172.21.15.67:0/2144593871 <== mgr.14164 v2:172.21.15.67:6800/2590792187 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fc95c0611d0 con 0x7fc948038410 2024-04-22T23:14:50.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:50.998+0000 7fc961edc700 1 -- 172.21.15.67:0/2144593871 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7fc948038410 msgr2=0x7fc94803a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:50.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:50.998+0000 7fc961edc700 1 --2- 172.21.15.67:0/2144593871 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7fc948038410 0x7fc94803a8d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc944006fd0 tx=0x7fc944006e40 comp rx=0 tx=0).stop 2024-04-22T23:14:51.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:50.998+0000 7fc961edc700 1 -- 172.21.15.67:0/2144593871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc95c0ff780 msgr2=0x7fc95c197ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:51.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:50.998+0000 7fc961edc700 1 --2- 172.21.15.67:0/2144593871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc95c0ff780 0x7fc95c197ba0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc94c009fd0 tx=0x7fc94c00bb10 comp rx=0 tx=0).stop 2024-04-22T23:14:51.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:50.998+0000 7fc961edc700 1 -- 172.21.15.67:0/2144593871 shutdown_connections 2024-04-22T23:14:51.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:50.998+0000 7fc961edc700 1 --2- 172.21.15.67:0/2144593871 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7fc948038410 0x7fc94803a8d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:51.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:50.998+0000 7fc961edc700 1 --2- 172.21.15.67:0/2144593871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc95c0ff780 0x7fc95c197ba0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:51.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:50.998+0000 7fc961edc700 1 -- 172.21.15.67:0/2144593871 >> 172.21.15.67:0/2144593871 conn(0x7fc95c0fad20 msgr2=0x7fc95c068820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:51.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:50.999+0000 7fc961edc700 1 -- 172.21.15.67:0/2144593871 shutdown_connections 2024-04-22T23:14:51.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:50.999+0000 7fc961edc700 1 -- 172.21.15.67:0/2144593871 wait complete. 2024-04-22T23:14:51.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:51 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:51.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:51 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:51.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:51 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:51.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:51 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:51.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:51 smithi067 ceph-mon[26453]: Deploying daemon node-exporter.smithi067 on smithi067 2024-04-22T23:14:51.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:51 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:51.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:51 smithi067 ceph-mon[26453]: Added host smithi134 2024-04-22T23:14:51.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:51 smithi067 ceph-mon[26453]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:14:51.619 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-22T23:14:51.847 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:14:54.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.085+0000 7f11cc501700 1 -- 172.21.15.67:0/702576061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11c40ff780 msgr2=0x7f11c40ffba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:54.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.085+0000 7f11cc501700 1 --2- 172.21.15.67:0/702576061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11c40ff780 0x7f11c40ffba0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f11c0009910 tx=0x7f11c0009c20 comp rx=0 tx=0).stop 2024-04-22T23:14:54.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.085+0000 7f11cc501700 1 -- 172.21.15.67:0/702576061 shutdown_connections 2024-04-22T23:14:54.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.085+0000 7f11cc501700 1 --2- 172.21.15.67:0/702576061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11c40ff780 0x7f11c40ffba0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:54.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.085+0000 7f11cc501700 1 -- 172.21.15.67:0/702576061 >> 172.21.15.67:0/702576061 conn(0x7f11c40fad20 msgr2=0x7f11c40fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:54.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.085+0000 7f11cc501700 1 -- 172.21.15.67:0/702576061 shutdown_connections 2024-04-22T23:14:54.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.085+0000 7f11cc501700 1 -- 172.21.15.67:0/702576061 wait complete. 2024-04-22T23:14:54.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.086+0000 7f11cc501700 1 Processor -- start 2024-04-22T23:14:54.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.086+0000 7f11cc501700 1 -- start start 2024-04-22T23:14:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.086+0000 7f11cc501700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11c40ff780 0x7f11c418c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.086+0000 7f11cc501700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c418cca0 con 0x7f11c40ff780 2024-04-22T23:14:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.086+0000 7f11ca29d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11c40ff780 0x7f11c418c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.086+0000 7f11ca29d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11c40ff780 0x7f11c418c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33616/0 (socket says 172.21.15.67:33616) 2024-04-22T23:14:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.086+0000 7f11ca29d700 1 -- 172.21.15.67:0/772982040 learned_addr learned my addr 172.21.15.67:0/772982040 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.086+0000 7f11ca29d700 1 -- 172.21.15.67:0/772982040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c00095f0 con 0x7f11c40ff780 2024-04-22T23:14:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.086+0000 7f11ca29d700 1 --2- 172.21.15.67:0/772982040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11c40ff780 0x7f11c418c760 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f11c0005950 tx=0x7f11c0009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.087+0000 7f11bb7fe700 1 -- 172.21.15.67:0/772982040 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11c001d070 con 0x7f11c40ff780 2024-04-22T23:14:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.087+0000 7f11bb7fe700 1 -- 172.21.15.67:0/772982040 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f11c000be10 con 0x7f11c40ff780 2024-04-22T23:14:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.087+0000 7f11cc501700 1 -- 172.21.15.67:0/772982040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11c418adb0 con 0x7f11c40ff780 2024-04-22T23:14:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.087+0000 7f11bb7fe700 1 -- 172.21.15.67:0/772982040 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11c000f460 con 0x7f11c40ff780 2024-04-22T23:14:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.087+0000 7f11cc501700 1 -- 172.21.15.67:0/772982040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11c418b250 con 0x7f11c40ff780 2024-04-22T23:14:54.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.088+0000 7f11bb7fe700 1 -- 172.21.15.67:0/772982040 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f11c000f690 con 0x7f11c40ff780 2024-04-22T23:14:54.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.088+0000 7f11cc501700 1 -- 172.21.15.67:0/772982040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11c4062430 con 0x7f11c40ff780 2024-04-22T23:14:54.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.088+0000 7f11bb7fe700 1 --2- 172.21.15.67:0/772982040 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f11b00383c0 0x7f11b003a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:54.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.088+0000 7f11bb7fe700 1 -- 172.21.15.67:0/772982040 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f11c004c560 con 0x7f11c40ff780 2024-04-22T23:14:54.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.088+0000 7f11c9a9c700 1 --2- 172.21.15.67:0/772982040 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f11b00383c0 0x7f11b003a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:54.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.089+0000 7f11c9a9c700 1 --2- 172.21.15.67:0/772982040 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f11b00383c0 0x7f11b003a880 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f11b4006fd0 tx=0x7f11b4006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:54.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.093+0000 7f11bb7fe700 1 -- 172.21.15.67:0/772982040 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f11c4062430 con 0x7f11c40ff780 2024-04-22T23:14:54.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.299+0000 7f11cc501700 1 -- 172.21.15.67:0/772982040 --> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f11c4062430 con 0x7f11b00383c0 2024-04-22T23:14:54.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.300+0000 7f11bb7fe700 1 -- 172.21.15.67:0/772982040 <== mgr.14164 v2:172.21.15.67:6800/2590792187 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f11c4062430 con 0x7f11b00383c0 2024-04-22T23:14:54.301 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:14:54.301 INFO:teuthology.orchestra.run.smithi067.stdout:[{"addr": "172.21.15.67", "hostname": "smithi067", "labels": [], "status": ""}, {"addr": "172.21.15.134", "hostname": "smithi134", "labels": [], "status": ""}] 2024-04-22T23:14:54.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.302+0000 7f11cc501700 1 -- 172.21.15.67:0/772982040 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f11b00383c0 msgr2=0x7f11b003a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:54.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.302+0000 7f11cc501700 1 --2- 172.21.15.67:0/772982040 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f11b00383c0 0x7f11b003a880 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f11b4006fd0 tx=0x7f11b4006e40 comp rx=0 tx=0).stop 2024-04-22T23:14:54.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.302+0000 7f11cc501700 1 -- 172.21.15.67:0/772982040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11c40ff780 msgr2=0x7f11c418c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:54.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.302+0000 7f11cc501700 1 --2- 172.21.15.67:0/772982040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11c40ff780 0x7f11c418c760 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f11c0005950 tx=0x7f11c0009e60 comp rx=0 tx=0).stop 2024-04-22T23:14:54.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.302+0000 7f11cc501700 1 -- 172.21.15.67:0/772982040 shutdown_connections 2024-04-22T23:14:54.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.302+0000 7f11cc501700 1 --2- 172.21.15.67:0/772982040 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f11b00383c0 0x7f11b003a880 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:54.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.302+0000 7f11cc501700 1 --2- 172.21.15.67:0/772982040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11c40ff780 0x7f11c418c760 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:54.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.302+0000 7f11cc501700 1 -- 172.21.15.67:0/772982040 >> 172.21.15.67:0/772982040 conn(0x7f11c40fad20 msgr2=0x7f11c40fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:54.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.302+0000 7f11cc501700 1 -- 172.21.15.67:0/772982040 shutdown_connections 2024-04-22T23:14:54.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:54.302+0000 7f11cc501700 1 -- 172.21.15.67:0/772982040 wait complete. 2024-04-22T23:14:54.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:54 smithi067 ceph-mon[26453]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:14:54.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:54 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:55.221 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-22T23:14:55.221 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-22T23:14:55.408 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:55 smithi067 ceph-mon[26453]: from='client.14193 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T23:14:55.620 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:14:56.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:56 smithi067 ceph-mon[26453]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:14:56.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:56 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:56.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:56 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:56.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:56 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:56.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:56 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:57.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:57.464+0000 7f497cbe2700 1 -- 172.21.15.67:0/3238332195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4978074d40 msgr2=0x7f49780731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:57.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:57.464+0000 7f497cbe2700 1 --2- 172.21.15.67:0/3238332195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4978074d40 0x7f49780731a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f497000b0a0 tx=0x7f497000b3b0 comp rx=0 tx=0).stop 2024-04-22T23:14:57.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:57.464+0000 7f497cbe2700 1 -- 172.21.15.67:0/3238332195 shutdown_connections 2024-04-22T23:14:57.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:57.464+0000 7f497cbe2700 1 --2- 172.21.15.67:0/3238332195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4978074d40 0x7f49780731a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:57.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:57.464+0000 7f497cbe2700 1 -- 172.21.15.67:0/3238332195 >> 172.21.15.67:0/3238332195 conn(0x7f49780fbe20 msgr2=0x7f49780fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:57.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:57.464+0000 7f497cbe2700 1 -- 172.21.15.67:0/3238332195 shutdown_connections 2024-04-22T23:14:57.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:57.464+0000 7f497cbe2700 1 -- 172.21.15.67:0/3238332195 wait complete. 2024-04-22T23:14:57.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:57.464+0000 7f497cbe2700 1 Processor -- start 2024-04-22T23:14:57.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:57.464+0000 7f497cbe2700 1 -- start start 2024-04-22T23:14:57.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:57.465+0000 7f497cbe2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4978074d40 0x7f4978197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:57.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:57.465+0000 7f497cbe2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4978198100 con 0x7f4978074d40 2024-04-22T23:14:57.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:57.465+0000 7f497659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4978074d40 0x7f4978197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:57.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:57.465+0000 7f497659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4978074d40 0x7f4978197bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33632/0 (socket says 172.21.15.67:33632) 2024-04-22T23:14:57.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:57.465+0000 7f497659c700 1 -- 172.21.15.67:0/619091433 learned_addr learned my addr 172.21.15.67:0/619091433 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:14:57.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:57.465+0000 7f497659c700 1 -- 172.21.15.67:0/619091433 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4970009d00 con 0x7f4978074d40 2024-04-22T23:14:57.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:57.465+0000 7f497659c700 1 --2- 172.21.15.67:0/619091433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4978074d40 0x7f4978197bc0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4970009fd0 tx=0x7f497000bb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:57.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:57.465+0000 7f496f7fe700 1 -- 172.21.15.67:0/619091433 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f497000e040 con 0x7f4978074d40 2024-04-22T23:14:57.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:57.465+0000 7f497cbe2700 1 -- 172.21.15.67:0/619091433 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4978198300 con 0x7f4978074d40 2024-04-22T23:14:57.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:57.465+0000 7f496f7fe700 1 -- 172.21.15.67:0/619091433 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f497000bdc0 con 0x7f4978074d40 2024-04-22T23:14:57.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:57.465+0000 7f496f7fe700 1 -- 172.21.15.67:0/619091433 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f497001da40 con 0x7f4978074d40 2024-04-22T23:14:57.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:57.465+0000 7f497cbe2700 1 -- 172.21.15.67:0/619091433 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4978198720 con 0x7f4978074d40 2024-04-22T23:14:57.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:57.466+0000 7f496f7fe700 1 -- 172.21.15.67:0/619091433 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f497001dc80 con 0x7f4978074d40 2024-04-22T23:14:57.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:57.466+0000 7f497cbe2700 1 -- 172.21.15.67:0/619091433 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49781916c0 con 0x7f4978074d40 2024-04-22T23:14:57.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:57.466+0000 7f496f7fe700 1 --2- 172.21.15.67:0/619091433 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f4964038410 0x7f496403a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:14:57.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:57.467+0000 7f496f7fe700 1 -- 172.21.15.67:0/619091433 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f497004c7d0 con 0x7f4978074d40 2024-04-22T23:14:57.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:57.467+0000 7f4975d9b700 1 --2- 172.21.15.67:0/619091433 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f4964038410 0x7f496403a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:14:57.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:57.467+0000 7f4975d9b700 1 --2- 172.21.15.67:0/619091433 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f4964038410 0x7f496403a8d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4960006fd0 tx=0x7f4960006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:14:57.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:57.471+0000 7f496f7fe700 1 -- 172.21.15.67:0/619091433 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f4970025750 con 0x7f4978074d40 2024-04-22T23:14:57.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:57 smithi067 ceph-mon[26453]: Deploying daemon alertmanager.smithi067 on smithi067 2024-04-22T23:14:57.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:57.677+0000 7f497cbe2700 1 -- 172.21.15.67:0/619091433 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f497802cc70 con 0x7f4978074d40 2024-04-22T23:14:58.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:58.288+0000 7f496f7fe700 1 -- 172.21.15.67:0/619091433 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f497001b340 con 0x7f4978074d40 2024-04-22T23:14:58.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:58.290+0000 7f497cbe2700 1 -- 172.21.15.67:0/619091433 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f4964038410 msgr2=0x7f496403a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:58.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:58.290+0000 7f497cbe2700 1 --2- 172.21.15.67:0/619091433 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f4964038410 0x7f496403a8d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4960006fd0 tx=0x7f4960006e40 comp rx=0 tx=0).stop 2024-04-22T23:14:58.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:58.290+0000 7f497cbe2700 1 -- 172.21.15.67:0/619091433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4978074d40 msgr2=0x7f4978197bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:14:58.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:58.290+0000 7f497cbe2700 1 --2- 172.21.15.67:0/619091433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4978074d40 0x7f4978197bc0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4970009fd0 tx=0x7f497000bb10 comp rx=0 tx=0).stop 2024-04-22T23:14:58.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:58.290+0000 7f497cbe2700 1 -- 172.21.15.67:0/619091433 shutdown_connections 2024-04-22T23:14:58.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:58.290+0000 7f497cbe2700 1 --2- 172.21.15.67:0/619091433 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f4964038410 0x7f496403a8d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:58.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:58.290+0000 7f497cbe2700 1 --2- 172.21.15.67:0/619091433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4978074d40 0x7f4978197bc0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:14:58.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:58.290+0000 7f497cbe2700 1 -- 172.21.15.67:0/619091433 >> 172.21.15.67:0/619091433 conn(0x7f49780fbe20 msgr2=0x7f49780fcb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:14:58.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:58.290+0000 7f497cbe2700 1 -- 172.21.15.67:0/619091433 shutdown_connections 2024-04-22T23:14:58.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:14:58.290+0000 7f497cbe2700 1 -- 172.21.15.67:0/619091433 wait complete. 2024-04-22T23:14:58.292 INFO:teuthology.orchestra.run.smithi067.stderr:adjusted tunables profile to default 2024-04-22T23:14:58.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:58 smithi067 ceph-mon[26453]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:14:58.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:58 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/619091433' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-22T23:14:59.520 INFO:tasks.cephadm:Adding mon.smithi067 on smithi067 2024-04-22T23:14:59.520 INFO:tasks.cephadm:Adding mon.smithi134 on smithi134 2024-04-22T23:14:59.520 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph orch apply mon '2;smithi067:172.21.15.67=smithi067;smithi134:172.21.15.134=smithi134' 2024-04-22T23:14:59.544 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:59 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/619091433' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-22T23:14:59.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:59 smithi067 ceph-mon[26453]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T23:14:59.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:14:59 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:14:59.779 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T23:14:59.827 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T23:15:00.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:00 smithi067 ceph-mon[26453]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:15:01.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.332+0000 7f6dafb5b700 1 -- 172.21.15.134:0/19722649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6da8100880 msgr2=0x7f6da8100ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:01.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.332+0000 7f6dafb5b700 1 --2- 172.21.15.134:0/19722649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6da8100880 0x7f6da8100ca0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6da4009910 tx=0x7f6da4009c20 comp rx=0 tx=0).stop 2024-04-22T23:15:01.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.332+0000 7f6dafb5b700 1 -- 172.21.15.134:0/19722649 shutdown_connections 2024-04-22T23:15:01.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.332+0000 7f6dafb5b700 1 --2- 172.21.15.134:0/19722649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6da8100880 0x7f6da8100ca0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:01.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.332+0000 7f6dafb5b700 1 -- 172.21.15.134:0/19722649 >> 172.21.15.134:0/19722649 conn(0x7f6da80fbe60 msgr2=0x7f6da80fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:15:01.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.332+0000 7f6dafb5b700 1 -- 172.21.15.134:0/19722649 shutdown_connections 2024-04-22T23:15:01.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.332+0000 7f6dafb5b700 1 -- 172.21.15.134:0/19722649 wait complete. 2024-04-22T23:15:01.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.333+0000 7f6dafb5b700 1 Processor -- start 2024-04-22T23:15:01.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.333+0000 7f6dafb5b700 1 -- start start 2024-04-22T23:15:01.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.333+0000 7f6dafb5b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6da8100880 0x7f6da81959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:15:01.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.333+0000 7f6dafb5b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da8195ee0 con 0x7f6da8100880 2024-04-22T23:15:01.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.333+0000 7f6dad8f7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6da8100880 0x7f6da81959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:15:01.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.333+0000 7f6dad8f7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6da8100880 0x7f6da81959a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.134:58774/0 (socket says 172.21.15.134:58774) 2024-04-22T23:15:01.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.333+0000 7f6dad8f7700 1 -- 172.21.15.134:0/2187325090 learned_addr learned my addr 172.21.15.134:0/2187325090 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-04-22T23:15:01.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.333+0000 7f6dad8f7700 1 -- 172.21.15.134:0/2187325090 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6da40095f0 con 0x7f6da8100880 2024-04-22T23:15:01.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.334+0000 7f6dad8f7700 1 --2- 172.21.15.134:0/2187325090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6da8100880 0x7f6da81959a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6da4005b40 tx=0x7f6da4004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:15:01.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.334+0000 7f6d9effd700 1 -- 172.21.15.134:0/2187325090 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6da401d070 con 0x7f6da8100880 2024-04-22T23:15:01.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.334+0000 7f6d9effd700 1 -- 172.21.15.134:0/2187325090 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6da400be10 con 0x7f6da8100880 2024-04-22T23:15:01.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.334+0000 7f6d9effd700 1 -- 172.21.15.134:0/2187325090 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6da400f460 con 0x7f6da8100880 2024-04-22T23:15:01.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.334+0000 7f6dafb5b700 1 -- 172.21.15.134:0/2187325090 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6da81960e0 con 0x7f6da8100880 2024-04-22T23:15:01.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.334+0000 7f6dafb5b700 1 -- 172.21.15.134:0/2187325090 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6da8196580 con 0x7f6da8100880 2024-04-22T23:15:01.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.335+0000 7f6d9effd700 1 -- 172.21.15.134:0/2187325090 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f6da40227e0 con 0x7f6da8100880 2024-04-22T23:15:01.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.335+0000 7f6d9effd700 1 --2- 172.21.15.134:0/2187325090 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f6d94038410 0x7f6d9403a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:15:01.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.335+0000 7f6d9effd700 1 -- 172.21.15.134:0/2187325090 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6da404c3e0 con 0x7f6da8100880 2024-04-22T23:15:01.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.335+0000 7f6dafb5b700 1 -- 172.21.15.134:0/2187325090 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6da818f4a0 con 0x7f6da8100880 2024-04-22T23:15:01.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.335+0000 7f6dad0f6700 1 --2- 172.21.15.134:0/2187325090 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f6d94038410 0x7f6d9403a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:15:01.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.336+0000 7f6dad0f6700 1 --2- 172.21.15.134:0/2187325090 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f6d94038410 0x7f6d9403a8d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6d98006fd0 tx=0x7f6d98006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:15:01.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.340+0000 7f6d9effd700 1 -- 172.21.15.134:0/2187325090 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6da4027030 con 0x7f6da8100880 2024-04-22T23:15:01.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.537+0000 7f6dafb5b700 1 -- 172.21.15.134:0/2187325090 --> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi067:172.21.15.67=smithi067;smithi134:172.21.15.134=smithi134", "target": ["mon-mgr", ""]}) v1 -- 0x7f6da8061340 con 0x7f6d94038410 2024-04-22T23:15:01.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.543+0000 7f6d9effd700 1 -- 172.21.15.134:0/2187325090 <== mgr.14164 v2:172.21.15.67:6800/2590792187 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f6da8061340 con 0x7f6d94038410 2024-04-22T23:15:01.545 INFO:teuthology.orchestra.run.smithi134.stdout:Scheduled mon update... 2024-04-22T23:15:01.547 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.545+0000 7f6dafb5b700 1 -- 172.21.15.134:0/2187325090 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f6d94038410 msgr2=0x7f6d9403a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:01.547 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.545+0000 7f6dafb5b700 1 --2- 172.21.15.134:0/2187325090 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f6d94038410 0x7f6d9403a8d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6d98006fd0 tx=0x7f6d98006e40 comp rx=0 tx=0).stop 2024-04-22T23:15:01.547 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.545+0000 7f6dafb5b700 1 -- 172.21.15.134:0/2187325090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6da8100880 msgr2=0x7f6da81959a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:01.547 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.545+0000 7f6dafb5b700 1 --2- 172.21.15.134:0/2187325090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6da8100880 0x7f6da81959a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6da4005b40 tx=0x7f6da4004740 comp rx=0 tx=0).stop 2024-04-22T23:15:01.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.545+0000 7f6dafb5b700 1 -- 172.21.15.134:0/2187325090 shutdown_connections 2024-04-22T23:15:01.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.545+0000 7f6dafb5b700 1 --2- 172.21.15.134:0/2187325090 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f6d94038410 0x7f6d9403a8d0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:01.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.545+0000 7f6dafb5b700 1 --2- 172.21.15.134:0/2187325090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6da8100880 0x7f6da81959a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:01.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.545+0000 7f6dafb5b700 1 -- 172.21.15.134:0/2187325090 >> 172.21.15.134:0/2187325090 conn(0x7f6da80fbe60 msgr2=0x7f6da80fcad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:15:01.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.546+0000 7f6dafb5b700 1 -- 172.21.15.134:0/2187325090 shutdown_connections 2024-04-22T23:15:01.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:01.546+0000 7f6dafb5b700 1 -- 172.21.15.134:0/2187325090 wait complete. 2024-04-22T23:15:02.081 DEBUG:teuthology.orchestra.run.smithi134:mon.smithi134> sudo journalctl -f -n 0 -u ceph-d46c346e-00fd-11ef-bc93-c7b262605968@mon.smithi134.service 2024-04-22T23:15:02.084 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T23:15:02.084 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T23:15:02.305 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T23:15:02.351 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T23:15:02.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:02 smithi067 ceph-mon[26453]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:15:02.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:02 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:03.722 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:03 smithi067 ceph-mon[26453]: from='client.14197 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi067:172.21.15.67=smithi067;smithi134:172.21.15.134=smithi134", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:15:03.722 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:03 smithi067 ceph-mon[26453]: Saving service mon spec with placement smithi067:172.21.15.67=smithi067;smithi134:172.21.15.134=smithi134;count:2 2024-04-22T23:15:03.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:03.733+0000 7f4e0f3df700 1 -- 172.21.15.134:0/987154558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e08074d40 msgr2=0x7f4e080731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:03.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:03.733+0000 7f4e0f3df700 1 --2- 172.21.15.134:0/987154558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e08074d40 0x7f4e080731a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4df8009910 tx=0x7f4df8009c20 comp rx=0 tx=0).stop 2024-04-22T23:15:03.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:03.733+0000 7f4e0f3df700 1 -- 172.21.15.134:0/987154558 shutdown_connections 2024-04-22T23:15:03.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:03.733+0000 7f4e0f3df700 1 --2- 172.21.15.134:0/987154558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e08074d40 0x7f4e080731a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:03.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:03.733+0000 7f4e0f3df700 1 -- 172.21.15.134:0/987154558 >> 172.21.15.134:0/987154558 conn(0x7f4e080fbe20 msgr2=0x7f4e080fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:15:03.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:03.733+0000 7f4e0f3df700 1 -- 172.21.15.134:0/987154558 shutdown_connections 2024-04-22T23:15:03.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:03.733+0000 7f4e0f3df700 1 -- 172.21.15.134:0/987154558 wait complete. 2024-04-22T23:15:03.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:03.734+0000 7f4e0f3df700 1 Processor -- start 2024-04-22T23:15:03.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:03.734+0000 7f4e0f3df700 1 -- start start 2024-04-22T23:15:03.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:03.734+0000 7f4e0f3df700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e08074d40 0x7f4e08197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:15:03.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:03.734+0000 7f4e0f3df700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e081980e0 con 0x7f4e08074d40 2024-04-22T23:15:03.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:03.734+0000 7f4e0d17b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e08074d40 0x7f4e08197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:15:03.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:03.734+0000 7f4e0d17b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e08074d40 0x7f4e08197ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.134:58804/0 (socket says 172.21.15.134:58804) 2024-04-22T23:15:03.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:03.734+0000 7f4e0d17b700 1 -- 172.21.15.134:0/2738924500 learned_addr learned my addr 172.21.15.134:0/2738924500 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-04-22T23:15:03.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:03.735+0000 7f4e0d17b700 1 -- 172.21.15.134:0/2738924500 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df80095f0 con 0x7f4e08074d40 2024-04-22T23:15:03.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:03.735+0000 7f4e0d17b700 1 --2- 172.21.15.134:0/2738924500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e08074d40 0x7f4e08197ba0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4df8005b40 tx=0x7f4df8004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:15:03.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:03.735+0000 7f4e067fc700 1 -- 172.21.15.134:0/2738924500 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4df801d070 con 0x7f4e08074d40 2024-04-22T23:15:03.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:03.735+0000 7f4e067fc700 1 -- 172.21.15.134:0/2738924500 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4df800be10 con 0x7f4e08074d40 2024-04-22T23:15:03.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:03.735+0000 7f4e0f3df700 1 -- 172.21.15.134:0/2738924500 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e081982e0 con 0x7f4e08074d40 2024-04-22T23:15:03.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:03.735+0000 7f4e067fc700 1 -- 172.21.15.134:0/2738924500 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4df800f460 con 0x7f4e08074d40 2024-04-22T23:15:03.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:03.735+0000 7f4e0f3df700 1 -- 172.21.15.134:0/2738924500 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e08198780 con 0x7f4e08074d40 2024-04-22T23:15:03.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:03.736+0000 7f4e0f3df700 1 -- 172.21.15.134:0/2738924500 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e081916c0 con 0x7f4e08074d40 2024-04-22T23:15:03.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:03.736+0000 7f4e067fc700 1 -- 172.21.15.134:0/2738924500 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f4df80227e0 con 0x7f4e08074d40 2024-04-22T23:15:03.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:03.736+0000 7f4e067fc700 1 --2- 172.21.15.134:0/2738924500 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f4df00383c0 0x7f4df003a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:15:03.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:03.736+0000 7f4e067fc700 1 -- 172.21.15.134:0/2738924500 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4df804c580 con 0x7f4e08074d40 2024-04-22T23:15:03.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:03.736+0000 7f4e0c97a700 1 --2- 172.21.15.134:0/2738924500 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f4df00383c0 0x7f4df003a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:15:03.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:03.737+0000 7f4e0c97a700 1 --2- 172.21.15.134:0/2738924500 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f4df00383c0 0x7f4df003a880 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc006fd0 tx=0x7f4dfc006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:15:03.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:03.741+0000 7f4e067fc700 1 -- 172.21.15.134:0/2738924500 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f4df802bba0 con 0x7f4e08074d40 2024-04-22T23:15:04.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:04.004+0000 7f4e0f3df700 1 -- 172.21.15.134:0/2738924500 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4e0802cc70 con 0x7f4e08074d40 2024-04-22T23:15:04.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:04.004+0000 7f4e067fc700 1 -- 172.21.15.134:0/2738924500 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f4df8027020 con 0x7f4e08074d40 2024-04-22T23:15:04.005 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:15:04.006 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"d46c346e-00fd-11ef-bc93-c7b262605968","modified":"2024-04-22T23:13:15.759242Z","created":"2024-04-22T23:13:15.759242Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T23:15:04.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:04.006+0000 7f4e0f3df700 1 -- 172.21.15.134:0/2738924500 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f4df00383c0 msgr2=0x7f4df003a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:04.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:04.006+0000 7f4e0f3df700 1 --2- 172.21.15.134:0/2738924500 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f4df00383c0 0x7f4df003a880 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc006fd0 tx=0x7f4dfc006e40 comp rx=0 tx=0).stop 2024-04-22T23:15:04.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:04.006+0000 7f4e0f3df700 1 -- 172.21.15.134:0/2738924500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e08074d40 msgr2=0x7f4e08197ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:04.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:04.006+0000 7f4e0f3df700 1 --2- 172.21.15.134:0/2738924500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e08074d40 0x7f4e08197ba0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4df8005b40 tx=0x7f4df8004740 comp rx=0 tx=0).stop 2024-04-22T23:15:04.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:04.006+0000 7f4e0f3df700 1 -- 172.21.15.134:0/2738924500 shutdown_connections 2024-04-22T23:15:04.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:04.006+0000 7f4e0f3df700 1 --2- 172.21.15.134:0/2738924500 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f4df00383c0 0x7f4df003a880 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:04.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:04.006+0000 7f4e0f3df700 1 --2- 172.21.15.134:0/2738924500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e08074d40 0x7f4e08197ba0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:04.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:04.006+0000 7f4e0f3df700 1 -- 172.21.15.134:0/2738924500 >> 172.21.15.134:0/2738924500 conn(0x7f4e080fbe20 msgr2=0x7f4e080fcb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:15:04.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:04.007+0000 7f4e0f3df700 1 -- 172.21.15.134:0/2738924500 shutdown_connections 2024-04-22T23:15:04.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:04.007+0000 7f4e0f3df700 1 -- 172.21.15.134:0/2738924500 wait complete. 2024-04-22T23:15:04.009 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-04-22T23:15:04.491 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:04 smithi067 ceph-mon[26453]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:15:04.492 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:04 smithi067 ceph-mon[26453]: from='client.? 172.21.15.134:0/2738924500' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T23:15:05.552 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T23:15:05.552 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T23:15:05.752 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T23:15:05.798 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T23:15:06.296 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:06 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:06.296 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:06 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:06.296 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:06 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:06.297 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:06 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:06.297 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:06 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:06.297 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:06 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:06.297 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:06 smithi067 ceph-mon[26453]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-22T23:15:06.297 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:06 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:06.297 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:06 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:06.297 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:06 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-22T23:15:06.297 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:06 smithi067 ceph-mon[26453]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-22T23:15:06.298 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:06 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:06.298 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:06 smithi067 ceph-mon[26453]: Deploying daemon grafana.smithi067 on smithi067 2024-04-22T23:15:06.298 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:06 smithi067 ceph-mon[26453]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:15:07.254 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.253+0000 7f673dedf700 1 -- 172.21.15.134:0/4160091739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67380feb70 msgr2=0x7f67380fef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:07.393 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.253+0000 7f673dedf700 1 --2- 172.21.15.134:0/4160091739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67380feb70 0x7f67380fef90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6728009910 tx=0x7f6728009c20 comp rx=0 tx=0).stop 2024-04-22T23:15:07.393 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.253+0000 7f673dedf700 1 -- 172.21.15.134:0/4160091739 shutdown_connections 2024-04-22T23:15:07.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.253+0000 7f673dedf700 1 --2- 172.21.15.134:0/4160091739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67380feb70 0x7f67380fef90 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:07.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.253+0000 7f673dedf700 1 -- 172.21.15.134:0/4160091739 >> 172.21.15.134:0/4160091739 conn(0x7f67380fa490 msgr2=0x7f67380fc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:15:07.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.253+0000 7f673dedf700 1 -- 172.21.15.134:0/4160091739 shutdown_connections 2024-04-22T23:15:07.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.253+0000 7f673dedf700 1 -- 172.21.15.134:0/4160091739 wait complete. 2024-04-22T23:15:07.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.253+0000 7f673dedf700 1 Processor -- start 2024-04-22T23:15:07.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.254+0000 7f673dedf700 1 -- start start 2024-04-22T23:15:07.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.254+0000 7f673dedf700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67380feb70 0x7f6738101550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:15:07.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.254+0000 7f673dedf700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6738101a90 con 0x7f67380feb70 2024-04-22T23:15:07.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.254+0000 7f67377fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67380feb70 0x7f6738101550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:15:07.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.254+0000 7f67377fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67380feb70 0x7f6738101550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.134:58832/0 (socket says 172.21.15.134:58832) 2024-04-22T23:15:07.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.254+0000 7f67377fe700 1 -- 172.21.15.134:0/1638848932 learned_addr learned my addr 172.21.15.134:0/1638848932 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-04-22T23:15:07.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.254+0000 7f67377fe700 1 -- 172.21.15.134:0/1638848932 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67280095f0 con 0x7f67380feb70 2024-04-22T23:15:07.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.254+0000 7f67377fe700 1 --2- 172.21.15.134:0/1638848932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67380feb70 0x7f6738101550 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6728005b40 tx=0x7f6728004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:15:07.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.255+0000 7f6734ff9700 1 -- 172.21.15.134:0/1638848932 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f672801d070 con 0x7f67380feb70 2024-04-22T23:15:07.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.255+0000 7f6734ff9700 1 -- 172.21.15.134:0/1638848932 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f672800be10 con 0x7f67380feb70 2024-04-22T23:15:07.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.255+0000 7f673dedf700 1 -- 172.21.15.134:0/1638848932 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67380ffc00 con 0x7f67380feb70 2024-04-22T23:15:07.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.255+0000 7f6734ff9700 1 -- 172.21.15.134:0/1638848932 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f672800f460 con 0x7f67380feb70 2024-04-22T23:15:07.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.255+0000 7f673dedf700 1 -- 172.21.15.134:0/1638848932 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6738100100 con 0x7f67380feb70 2024-04-22T23:15:07.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.255+0000 7f6734ff9700 1 -- 172.21.15.134:0/1638848932 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f672800f5c0 con 0x7f67380feb70 2024-04-22T23:15:07.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.256+0000 7f6734ff9700 1 --2- 172.21.15.134:0/1638848932 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f6720038410 0x7f672003a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:15:07.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.256+0000 7f6734ff9700 1 -- 172.21.15.134:0/1638848932 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f672804c550 con 0x7f67380feb70 2024-04-22T23:15:07.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.256+0000 7f673dedf700 1 -- 172.21.15.134:0/1638848932 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6738062430 con 0x7f67380feb70 2024-04-22T23:15:07.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.256+0000 7f6736ffd700 1 --2- 172.21.15.134:0/1638848932 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f6720038410 0x7f672003a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:15:07.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.256+0000 7f6736ffd700 1 --2- 172.21.15.134:0/1638848932 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f6720038410 0x7f672003a8d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f672c006fd0 tx=0x7f672c006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:15:07.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.261+0000 7f6734ff9700 1 -- 172.21.15.134:0/1638848932 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6738062430 con 0x7f67380feb70 2024-04-22T23:15:07.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.521+0000 7f673dedf700 1 -- 172.21.15.134:0/1638848932 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f67380623c0 con 0x7f67380feb70 2024-04-22T23:15:07.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.522+0000 7f6734ff9700 1 -- 172.21.15.134:0/1638848932 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f6728027070 con 0x7f67380feb70 2024-04-22T23:15:07.523 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:15:07.523 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"d46c346e-00fd-11ef-bc93-c7b262605968","modified":"2024-04-22T23:13:15.759242Z","created":"2024-04-22T23:13:15.759242Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T23:15:07.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.524+0000 7f673dedf700 1 -- 172.21.15.134:0/1638848932 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f6720038410 msgr2=0x7f672003a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:07.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.524+0000 7f673dedf700 1 --2- 172.21.15.134:0/1638848932 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f6720038410 0x7f672003a8d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f672c006fd0 tx=0x7f672c006e40 comp rx=0 tx=0).stop 2024-04-22T23:15:07.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.524+0000 7f673dedf700 1 -- 172.21.15.134:0/1638848932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67380feb70 msgr2=0x7f6738101550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:07.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.524+0000 7f673dedf700 1 --2- 172.21.15.134:0/1638848932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67380feb70 0x7f6738101550 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6728005b40 tx=0x7f6728004740 comp rx=0 tx=0).stop 2024-04-22T23:15:07.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.524+0000 7f673dedf700 1 -- 172.21.15.134:0/1638848932 shutdown_connections 2024-04-22T23:15:07.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.524+0000 7f673dedf700 1 --2- 172.21.15.134:0/1638848932 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f6720038410 0x7f672003a8d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:07.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.524+0000 7f673dedf700 1 --2- 172.21.15.134:0/1638848932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67380feb70 0x7f6738101550 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:07.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.524+0000 7f673dedf700 1 -- 172.21.15.134:0/1638848932 >> 172.21.15.134:0/1638848932 conn(0x7f67380fa490 msgr2=0x7f67380fb170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:15:07.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.524+0000 7f673dedf700 1 -- 172.21.15.134:0/1638848932 shutdown_connections 2024-04-22T23:15:07.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:07.524+0000 7f673dedf700 1 -- 172.21.15.134:0/1638848932 wait complete. 2024-04-22T23:15:07.526 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-04-22T23:15:08.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:08 smithi067 ceph-mon[26453]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:15:08.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:08 smithi067 ceph-mon[26453]: from='client.? 172.21.15.134:0/1638848932' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T23:15:09.065 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T23:15:09.065 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T23:15:09.263 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T23:15:09.309 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T23:15:10.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:10 smithi067 ceph-mon[26453]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:15:10.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:10 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:10.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.651+0000 7f575076d700 1 -- 172.21.15.134:0/2049872184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57480ff780 msgr2=0x7f57480ffba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:10.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.651+0000 7f575076d700 1 --2- 172.21.15.134:0/2049872184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57480ff780 0x7f57480ffba0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5744009910 tx=0x7f5744009c20 comp rx=0 tx=0).stop 2024-04-22T23:15:10.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.651+0000 7f575076d700 1 -- 172.21.15.134:0/2049872184 shutdown_connections 2024-04-22T23:15:10.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.651+0000 7f575076d700 1 --2- 172.21.15.134:0/2049872184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57480ff780 0x7f57480ffba0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:10.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.651+0000 7f575076d700 1 -- 172.21.15.134:0/2049872184 >> 172.21.15.134:0/2049872184 conn(0x7f57480fad20 msgr2=0x7f57480fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:15:10.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.651+0000 7f575076d700 1 -- 172.21.15.134:0/2049872184 shutdown_connections 2024-04-22T23:15:10.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.651+0000 7f575076d700 1 -- 172.21.15.134:0/2049872184 wait complete. 2024-04-22T23:15:10.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.652+0000 7f575076d700 1 Processor -- start 2024-04-22T23:15:10.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.652+0000 7f575076d700 1 -- start start 2024-04-22T23:15:10.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.652+0000 7f575076d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57480ff780 0x7f57481948a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:15:10.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.652+0000 7f575076d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5748194de0 con 0x7f57480ff780 2024-04-22T23:15:10.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.652+0000 7f574e509700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57480ff780 0x7f57481948a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:15:10.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.652+0000 7f574e509700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57480ff780 0x7f57481948a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.134:44014/0 (socket says 172.21.15.134:44014) 2024-04-22T23:15:10.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.652+0000 7f574e509700 1 -- 172.21.15.134:0/3304431427 learned_addr learned my addr 172.21.15.134:0/3304431427 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-04-22T23:15:10.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.652+0000 7f574e509700 1 -- 172.21.15.134:0/3304431427 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57440095f0 con 0x7f57480ff780 2024-04-22T23:15:10.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.653+0000 7f574e509700 1 --2- 172.21.15.134:0/3304431427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57480ff780 0x7f57481948a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5744005b40 tx=0x7f5744004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:15:10.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.653+0000 7f573f7fe700 1 -- 172.21.15.134:0/3304431427 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f574401d070 con 0x7f57480ff780 2024-04-22T23:15:10.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.653+0000 7f573f7fe700 1 -- 172.21.15.134:0/3304431427 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f574400be10 con 0x7f57480ff780 2024-04-22T23:15:10.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.653+0000 7f573f7fe700 1 -- 172.21.15.134:0/3304431427 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f574400f460 con 0x7f57480ff780 2024-04-22T23:15:10.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.653+0000 7f575076d700 1 -- 172.21.15.134:0/3304431427 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5748194fe0 con 0x7f57480ff780 2024-04-22T23:15:10.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.653+0000 7f575076d700 1 -- 172.21.15.134:0/3304431427 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5748195480 con 0x7f57480ff780 2024-04-22T23:15:10.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.654+0000 7f573f7fe700 1 -- 172.21.15.134:0/3304431427 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f574400b960 con 0x7f57480ff780 2024-04-22T23:15:10.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.654+0000 7f575076d700 1 -- 172.21.15.134:0/3304431427 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f574818e3a0 con 0x7f57480ff780 2024-04-22T23:15:10.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.654+0000 7f573f7fe700 1 --2- 172.21.15.134:0/3304431427 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f5734038410 0x7f573403a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:15:10.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.654+0000 7f573f7fe700 1 -- 172.21.15.134:0/3304431427 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f574404c450 con 0x7f57480ff780 2024-04-22T23:15:10.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.654+0000 7f574dd08700 1 --2- 172.21.15.134:0/3304431427 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f5734038410 0x7f573403a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:15:10.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.655+0000 7f574dd08700 1 --2- 172.21.15.134:0/3304431427 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f5734038410 0x7f573403a8d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5738006fd0 tx=0x7f5738006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:15:10.660 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.659+0000 7f573f7fe700 1 -- 172.21.15.134:0/3304431427 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f574402bb60 con 0x7f57480ff780 2024-04-22T23:15:10.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.926+0000 7f575076d700 1 -- 172.21.15.134:0/3304431427 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f574802cc70 con 0x7f57480ff780 2024-04-22T23:15:10.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.927+0000 7f573f7fe700 1 -- 172.21.15.134:0/3304431427 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f5744027030 con 0x7f57480ff780 2024-04-22T23:15:10.928 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:15:10.928 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"d46c346e-00fd-11ef-bc93-c7b262605968","modified":"2024-04-22T23:13:15.759242Z","created":"2024-04-22T23:13:15.759242Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T23:15:10.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.929+0000 7f575076d700 1 -- 172.21.15.134:0/3304431427 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f5734038410 msgr2=0x7f573403a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:10.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.929+0000 7f575076d700 1 --2- 172.21.15.134:0/3304431427 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f5734038410 0x7f573403a8d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5738006fd0 tx=0x7f5738006e40 comp rx=0 tx=0).stop 2024-04-22T23:15:10.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.929+0000 7f575076d700 1 -- 172.21.15.134:0/3304431427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57480ff780 msgr2=0x7f57481948a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:10.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.929+0000 7f575076d700 1 --2- 172.21.15.134:0/3304431427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57480ff780 0x7f57481948a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5744005b40 tx=0x7f5744004740 comp rx=0 tx=0).stop 2024-04-22T23:15:10.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.929+0000 7f575076d700 1 -- 172.21.15.134:0/3304431427 shutdown_connections 2024-04-22T23:15:10.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.929+0000 7f575076d700 1 --2- 172.21.15.134:0/3304431427 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f5734038410 0x7f573403a8d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:10.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.929+0000 7f575076d700 1 --2- 172.21.15.134:0/3304431427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57480ff780 0x7f57481948a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:10.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.929+0000 7f575076d700 1 -- 172.21.15.134:0/3304431427 >> 172.21.15.134:0/3304431427 conn(0x7f57480fad20 msgr2=0x7f57480fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:15:10.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.929+0000 7f575076d700 1 -- 172.21.15.134:0/3304431427 shutdown_connections 2024-04-22T23:15:10.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:10.929+0000 7f575076d700 1 -- 172.21.15.134:0/3304431427 wait complete. 2024-04-22T23:15:10.931 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-04-22T23:15:11.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:11 smithi067 ceph-mon[26453]: from='client.? 172.21.15.134:0/3304431427' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T23:15:12.453 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T23:15:12.453 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T23:15:12.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:12 smithi067 ceph-mon[26453]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:15:12.651 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T23:15:12.697 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T23:15:13.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:13.985+0000 7f1e15805700 1 -- 172.21.15.134:0/3190771400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e100ff780 msgr2=0x7f1e100ffba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:13.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:13.985+0000 7f1e15805700 1 --2- 172.21.15.134:0/3190771400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e100ff780 0x7f1e100ffba0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1e00009910 tx=0x7f1e00009c20 comp rx=0 tx=0).stop 2024-04-22T23:15:13.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:13.985+0000 7f1e15805700 1 -- 172.21.15.134:0/3190771400 shutdown_connections 2024-04-22T23:15:13.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:13.985+0000 7f1e15805700 1 --2- 172.21.15.134:0/3190771400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e100ff780 0x7f1e100ffba0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:13.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:13.985+0000 7f1e15805700 1 -- 172.21.15.134:0/3190771400 >> 172.21.15.134:0/3190771400 conn(0x7f1e100fad20 msgr2=0x7f1e100fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:15:13.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:13.985+0000 7f1e15805700 1 -- 172.21.15.134:0/3190771400 shutdown_connections 2024-04-22T23:15:13.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:13.985+0000 7f1e15805700 1 -- 172.21.15.134:0/3190771400 wait complete. 2024-04-22T23:15:13.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:13.985+0000 7f1e15805700 1 Processor -- start 2024-04-22T23:15:13.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:13.986+0000 7f1e15805700 1 -- start start 2024-04-22T23:15:13.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:13.986+0000 7f1e15805700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e10069380 0x7f1e10198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:15:13.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:13.986+0000 7f1e15805700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e100697a0 con 0x7f1e10069380 2024-04-22T23:15:13.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:13.986+0000 7f1e0effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e10069380 0x7f1e10198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:15:13.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:13.986+0000 7f1e0effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e10069380 0x7f1e10198910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.134:44034/0 (socket says 172.21.15.134:44034) 2024-04-22T23:15:13.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:13.986+0000 7f1e0effd700 1 -- 172.21.15.134:0/3560553857 learned_addr learned my addr 172.21.15.134:0/3560553857 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-04-22T23:15:13.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:13.986+0000 7f1e0effd700 1 -- 172.21.15.134:0/3560553857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e000095f0 con 0x7f1e10069380 2024-04-22T23:15:13.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:13.986+0000 7f1e0effd700 1 --2- 172.21.15.134:0/3560553857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e10069380 0x7f1e10198910 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1e00005950 tx=0x7f1e00009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:15:13.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:13.987+0000 7f1e14803700 1 -- 172.21.15.134:0/3560553857 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e0001d070 con 0x7f1e10069380 2024-04-22T23:15:13.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:13.987+0000 7f1e15805700 1 -- 172.21.15.134:0/3560553857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e10198e50 con 0x7f1e10069380 2024-04-22T23:15:13.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:13.987+0000 7f1e14803700 1 -- 172.21.15.134:0/3560553857 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1e0000be10 con 0x7f1e10069380 2024-04-22T23:15:13.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:13.987+0000 7f1e14803700 1 -- 172.21.15.134:0/3560553857 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e00022e20 con 0x7f1e10069380 2024-04-22T23:15:13.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:13.987+0000 7f1e15805700 1 -- 172.21.15.134:0/3560553857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e10199290 con 0x7f1e10069380 2024-04-22T23:15:13.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:13.987+0000 7f1e14803700 1 -- 172.21.15.134:0/3560553857 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f1e00022700 con 0x7f1e10069380 2024-04-22T23:15:13.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:13.987+0000 7f1e15805700 1 -- 172.21.15.134:0/3560553857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e1018e3a0 con 0x7f1e10069380 2024-04-22T23:15:13.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:13.988+0000 7f1e14803700 1 --2- 172.21.15.134:0/3560553857 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f1df8038370 0x7f1df803a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:15:13.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:13.988+0000 7f1e14803700 1 -- 172.21.15.134:0/3560553857 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1e0004c2c0 con 0x7f1e10069380 2024-04-22T23:15:13.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:13.988+0000 7f1e0e7fc700 1 --2- 172.21.15.134:0/3560553857 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f1df8038370 0x7f1df803a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:15:13.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:13.988+0000 7f1e0e7fc700 1 --2- 172.21.15.134:0/3560553857 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f1df8038370 0x7f1df803a830 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1e04006fd0 tx=0x7f1e04006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:15:13.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:13.992+0000 7f1e14803700 1 -- 172.21.15.134:0/3560553857 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f1e0002b590 con 0x7f1e10069380 2024-04-22T23:15:14.254 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:14.254+0000 7f1e15805700 1 -- 172.21.15.134:0/3560553857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1e1002cc70 con 0x7f1e10069380 2024-04-22T23:15:14.255 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:14.254+0000 7f1e14803700 1 -- 172.21.15.134:0/3560553857 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f1e00027030 con 0x7f1e10069380 2024-04-22T23:15:14.255 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:15:14.255 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"d46c346e-00fd-11ef-bc93-c7b262605968","modified":"2024-04-22T23:13:15.759242Z","created":"2024-04-22T23:13:15.759242Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T23:15:14.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:14.256+0000 7f1e15805700 1 -- 172.21.15.134:0/3560553857 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f1df8038370 msgr2=0x7f1df803a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:14.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:14.256+0000 7f1e15805700 1 --2- 172.21.15.134:0/3560553857 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f1df8038370 0x7f1df803a830 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1e04006fd0 tx=0x7f1e04006e40 comp rx=0 tx=0).stop 2024-04-22T23:15:14.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:14.256+0000 7f1e15805700 1 -- 172.21.15.134:0/3560553857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e10069380 msgr2=0x7f1e10198910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:14.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:14.256+0000 7f1e15805700 1 --2- 172.21.15.134:0/3560553857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e10069380 0x7f1e10198910 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1e00005950 tx=0x7f1e00009f10 comp rx=0 tx=0).stop 2024-04-22T23:15:14.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:14.256+0000 7f1e15805700 1 -- 172.21.15.134:0/3560553857 shutdown_connections 2024-04-22T23:15:14.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:14.256+0000 7f1e15805700 1 --2- 172.21.15.134:0/3560553857 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f1df8038370 0x7f1df803a830 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:14.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:14.256+0000 7f1e15805700 1 --2- 172.21.15.134:0/3560553857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e10069380 0x7f1e10198910 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:14.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:14.256+0000 7f1e15805700 1 -- 172.21.15.134:0/3560553857 >> 172.21.15.134:0/3560553857 conn(0x7f1e100fad20 msgr2=0x7f1e100fba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:15:14.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:14.256+0000 7f1e15805700 1 -- 172.21.15.134:0/3560553857 shutdown_connections 2024-04-22T23:15:14.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:14.256+0000 7f1e15805700 1 -- 172.21.15.134:0/3560553857 wait complete. 2024-04-22T23:15:14.258 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-04-22T23:15:14.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:14 smithi067 ceph-mon[26453]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:15:14.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:14 smithi067 ceph-mon[26453]: from='client.? 172.21.15.134:0/3560553857' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T23:15:15.745 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T23:15:15.745 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T23:15:15.944 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T23:15:15.989 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T23:15:16.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:15 smithi067 ceph-mon[26453]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:15:17.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.377+0000 7f4ff9f27700 1 -- 172.21.15.134:0/1446132632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ff4100880 msgr2=0x7f4ff4100ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:17.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.377+0000 7f4ff9f27700 1 --2- 172.21.15.134:0/1446132632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ff4100880 0x7f4ff4100ca0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4fe4009910 tx=0x7f4fe4009c20 comp rx=0 tx=0).stop 2024-04-22T23:15:17.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.377+0000 7f4ff9f27700 1 -- 172.21.15.134:0/1446132632 shutdown_connections 2024-04-22T23:15:17.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.377+0000 7f4ff9f27700 1 --2- 172.21.15.134:0/1446132632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ff4100880 0x7f4ff4100ca0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:17.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.377+0000 7f4ff9f27700 1 -- 172.21.15.134:0/1446132632 >> 172.21.15.134:0/1446132632 conn(0x7f4ff40fbe00 msgr2=0x7f4ff40fe260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:15:17.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.377+0000 7f4ff9f27700 1 -- 172.21.15.134:0/1446132632 shutdown_connections 2024-04-22T23:15:17.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.377+0000 7f4ff9f27700 1 -- 172.21.15.134:0/1446132632 wait complete. 2024-04-22T23:15:17.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.378+0000 7f4ff9f27700 1 Processor -- start 2024-04-22T23:15:17.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.378+0000 7f4ff9f27700 1 -- start start 2024-04-22T23:15:17.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.378+0000 7f4ff9f27700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ff4100880 0x7f4ff41959d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:15:17.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.378+0000 7f4ff9f27700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff4195f10 con 0x7f4ff4100880 2024-04-22T23:15:17.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.378+0000 7f4ff37fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ff4100880 0x7f4ff41959d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:15:17.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.379+0000 7f4ff37fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ff4100880 0x7f4ff41959d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.134:44040/0 (socket says 172.21.15.134:44040) 2024-04-22T23:15:17.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.379+0000 7f4ff37fe700 1 -- 172.21.15.134:0/1156771103 learned_addr learned my addr 172.21.15.134:0/1156771103 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-04-22T23:15:17.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.379+0000 7f4ff37fe700 1 -- 172.21.15.134:0/1156771103 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe40095f0 con 0x7f4ff4100880 2024-04-22T23:15:17.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.379+0000 7f4ff37fe700 1 --2- 172.21.15.134:0/1156771103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ff4100880 0x7f4ff41959d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4fe4005b40 tx=0x7f4fe4004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:15:17.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.379+0000 7f4ff17fa700 1 -- 172.21.15.134:0/1156771103 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4fe401d070 con 0x7f4ff4100880 2024-04-22T23:15:17.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.379+0000 7f4ff9f27700 1 -- 172.21.15.134:0/1156771103 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ff4196110 con 0x7f4ff4100880 2024-04-22T23:15:17.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.379+0000 7f4ff17fa700 1 -- 172.21.15.134:0/1156771103 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4fe400be10 con 0x7f4ff4100880 2024-04-22T23:15:17.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.379+0000 7f4ff17fa700 1 -- 172.21.15.134:0/1156771103 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4fe400f460 con 0x7f4ff4100880 2024-04-22T23:15:17.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.379+0000 7f4ff9f27700 1 -- 172.21.15.134:0/1156771103 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ff41965b0 con 0x7f4ff4100880 2024-04-22T23:15:17.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.380+0000 7f4ff17fa700 1 -- 172.21.15.134:0/1156771103 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f4fe400b960 con 0x7f4ff4100880 2024-04-22T23:15:17.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.380+0000 7f4ff9f27700 1 -- 172.21.15.134:0/1156771103 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ff418f4a0 con 0x7f4ff4100880 2024-04-22T23:15:17.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.380+0000 7f4ff17fa700 1 --2- 172.21.15.134:0/1156771103 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f4fe00383c0 0x7f4fe003a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:15:17.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.380+0000 7f4ff17fa700 1 -- 172.21.15.134:0/1156771103 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4fe404c480 con 0x7f4ff4100880 2024-04-22T23:15:17.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.381+0000 7f4feaffd700 1 --2- 172.21.15.134:0/1156771103 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f4fe00383c0 0x7f4fe003a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:15:17.383 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.381+0000 7f4feaffd700 1 --2- 172.21.15.134:0/1156771103 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f4fe00383c0 0x7f4fe003a880 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc006fd0 tx=0x7f4fdc006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:15:17.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.385+0000 7f4ff17fa700 1 -- 172.21.15.134:0/1156771103 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f4fe402a360 con 0x7f4ff4100880 2024-04-22T23:15:17.647 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.646+0000 7f4ff9f27700 1 -- 172.21.15.134:0/1156771103 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4ff402cc70 con 0x7f4ff4100880 2024-04-22T23:15:17.648 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.647+0000 7f4ff17fa700 1 -- 172.21.15.134:0/1156771103 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f4fe4027030 con 0x7f4ff4100880 2024-04-22T23:15:17.648 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:15:17.648 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"d46c346e-00fd-11ef-bc93-c7b262605968","modified":"2024-04-22T23:13:15.759242Z","created":"2024-04-22T23:13:15.759242Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T23:15:17.650 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.649+0000 7f4ff9f27700 1 -- 172.21.15.134:0/1156771103 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f4fe00383c0 msgr2=0x7f4fe003a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:17.650 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.649+0000 7f4ff9f27700 1 --2- 172.21.15.134:0/1156771103 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f4fe00383c0 0x7f4fe003a880 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc006fd0 tx=0x7f4fdc006e40 comp rx=0 tx=0).stop 2024-04-22T23:15:17.650 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.649+0000 7f4ff9f27700 1 -- 172.21.15.134:0/1156771103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ff4100880 msgr2=0x7f4ff41959d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:17.650 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.649+0000 7f4ff9f27700 1 --2- 172.21.15.134:0/1156771103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ff4100880 0x7f4ff41959d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4fe4005b40 tx=0x7f4fe4004740 comp rx=0 tx=0).stop 2024-04-22T23:15:17.650 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.649+0000 7f4ff9f27700 1 -- 172.21.15.134:0/1156771103 shutdown_connections 2024-04-22T23:15:17.650 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.649+0000 7f4ff9f27700 1 --2- 172.21.15.134:0/1156771103 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f4fe00383c0 0x7f4fe003a880 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:17.650 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.649+0000 7f4ff9f27700 1 --2- 172.21.15.134:0/1156771103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ff4100880 0x7f4ff41959d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:17.650 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.649+0000 7f4ff9f27700 1 -- 172.21.15.134:0/1156771103 >> 172.21.15.134:0/1156771103 conn(0x7f4ff40fbe00 msgr2=0x7f4ff40fcae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:15:17.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.649+0000 7f4ff9f27700 1 -- 172.21.15.134:0/1156771103 shutdown_connections 2024-04-22T23:15:17.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:17.649+0000 7f4ff9f27700 1 -- 172.21.15.134:0/1156771103 wait complete. 2024-04-22T23:15:17.651 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-04-22T23:15:18.465 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:18 smithi067 ceph-mon[26453]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:15:18.465 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:18 smithi067 ceph-mon[26453]: from='client.? 172.21.15.134:0/1156771103' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T23:15:19.231 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T23:15:19.232 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T23:15:19.437 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T23:15:19.483 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T23:15:20.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:20 smithi067 ceph-mon[26453]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:15:20.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:20 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:20.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:20 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:20.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:20 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:20.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:20 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:20.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:20 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:20.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:20 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:20.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:20 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:20.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:20 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:20.802 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:20.801+0000 7f57d7fc6700 1 -- 172.21.15.134:0/2222739067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57d00ff780 msgr2=0x7f57d00ffba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:20.803 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:20.801+0000 7f57d7fc6700 1 --2- 172.21.15.134:0/2222739067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57d00ff780 0x7f57d00ffba0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f57cc009910 tx=0x7f57cc009c20 comp rx=0 tx=0).stop 2024-04-22T23:15:20.803 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:20.801+0000 7f57d7fc6700 1 -- 172.21.15.134:0/2222739067 shutdown_connections 2024-04-22T23:15:20.803 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:20.801+0000 7f57d7fc6700 1 --2- 172.21.15.134:0/2222739067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57d00ff780 0x7f57d00ffba0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:20.803 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:20.801+0000 7f57d7fc6700 1 -- 172.21.15.134:0/2222739067 >> 172.21.15.134:0/2222739067 conn(0x7f57d00fad20 msgr2=0x7f57d00fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:15:20.803 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:20.801+0000 7f57d7fc6700 1 -- 172.21.15.134:0/2222739067 shutdown_connections 2024-04-22T23:15:20.803 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:20.801+0000 7f57d7fc6700 1 -- 172.21.15.134:0/2222739067 wait complete. 2024-04-22T23:15:20.803 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:20.802+0000 7f57d7fc6700 1 Processor -- start 2024-04-22T23:15:20.803 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:20.802+0000 7f57d7fc6700 1 -- start start 2024-04-22T23:15:20.803 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:20.802+0000 7f57d7fc6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57d00ff780 0x7f57d00fccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:15:20.803 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:20.802+0000 7f57d7fc6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d00fd200 con 0x7f57d00ff780 2024-04-22T23:15:20.805 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:20.802+0000 7f57d5d62700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57d00ff780 0x7f57d00fccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:15:20.805 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:20.802+0000 7f57d5d62700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57d00ff780 0x7f57d00fccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.134:37566/0 (socket says 172.21.15.134:37566) 2024-04-22T23:15:20.805 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:20.802+0000 7f57d5d62700 1 -- 172.21.15.134:0/3455744832 learned_addr learned my addr 172.21.15.134:0/3455744832 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-04-22T23:15:20.805 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:20.802+0000 7f57d5d62700 1 -- 172.21.15.134:0/3455744832 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57cc0095f0 con 0x7f57d00ff780 2024-04-22T23:15:20.805 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:20.803+0000 7f57d5d62700 1 --2- 172.21.15.134:0/3455744832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57d00ff780 0x7f57d00fccc0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f57cc005b40 tx=0x7f57cc004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:15:20.805 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:20.803+0000 7f57c6ffd700 1 -- 172.21.15.134:0/3455744832 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57cc01d070 con 0x7f57d00ff780 2024-04-22T23:15:20.805 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:20.803+0000 7f57c6ffd700 1 -- 172.21.15.134:0/3455744832 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f57cc00be10 con 0x7f57d00ff780 2024-04-22T23:15:20.805 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:20.803+0000 7f57c6ffd700 1 -- 172.21.15.134:0/3455744832 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57cc00f460 con 0x7f57d00ff780 2024-04-22T23:15:20.805 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:20.803+0000 7f57d7fc6700 1 -- 172.21.15.134:0/3455744832 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57d00fb310 con 0x7f57d00ff780 2024-04-22T23:15:20.805 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:20.803+0000 7f57d7fc6700 1 -- 172.21.15.134:0/3455744832 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57d00fb7b0 con 0x7f57d00ff780 2024-04-22T23:15:20.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:20.804+0000 7f57c6ffd700 1 -- 172.21.15.134:0/3455744832 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f57cc00f5c0 con 0x7f57d00ff780 2024-04-22T23:15:20.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:20.804+0000 7f57c6ffd700 1 --2- 172.21.15.134:0/3455744832 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f57bc038410 0x7f57bc03a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:15:20.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:20.804+0000 7f57c6ffd700 1 -- 172.21.15.134:0/3455744832 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f57cc04c4a0 con 0x7f57d00ff780 2024-04-22T23:15:20.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:20.804+0000 7f57d7fc6700 1 -- 172.21.15.134:0/3455744832 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57d00623c0 con 0x7f57d00ff780 2024-04-22T23:15:20.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:20.804+0000 7f57d5561700 1 --2- 172.21.15.134:0/3455744832 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f57bc038410 0x7f57bc03a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:15:20.807 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:20.805+0000 7f57d5561700 1 --2- 172.21.15.134:0/3455744832 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f57bc038410 0x7f57bc03a8d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f57c0006fd0 tx=0x7f57c0006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:15:20.810 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:20.809+0000 7f57c6ffd700 1 -- 172.21.15.134:0/3455744832 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f57cc027070 con 0x7f57d00ff780 2024-04-22T23:15:21.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:21.078+0000 7f57d7fc6700 1 -- 172.21.15.134:0/3455744832 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f57d019b9f0 con 0x7f57d00ff780 2024-04-22T23:15:21.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:21.078+0000 7f57c6ffd700 1 -- 172.21.15.134:0/3455744832 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f57cc02c440 con 0x7f57d00ff780 2024-04-22T23:15:21.079 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:15:21.080 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"d46c346e-00fd-11ef-bc93-c7b262605968","modified":"2024-04-22T23:13:15.759242Z","created":"2024-04-22T23:13:15.759242Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T23:15:21.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:21.080+0000 7f57d7fc6700 1 -- 172.21.15.134:0/3455744832 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f57bc038410 msgr2=0x7f57bc03a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:21.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:21.080+0000 7f57d7fc6700 1 --2- 172.21.15.134:0/3455744832 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f57bc038410 0x7f57bc03a8d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f57c0006fd0 tx=0x7f57c0006e40 comp rx=0 tx=0).stop 2024-04-22T23:15:21.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:21.080+0000 7f57d7fc6700 1 -- 172.21.15.134:0/3455744832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57d00ff780 msgr2=0x7f57d00fccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:21.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:21.080+0000 7f57d7fc6700 1 --2- 172.21.15.134:0/3455744832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57d00ff780 0x7f57d00fccc0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f57cc005b40 tx=0x7f57cc004740 comp rx=0 tx=0).stop 2024-04-22T23:15:21.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:21.080+0000 7f57d7fc6700 1 -- 172.21.15.134:0/3455744832 shutdown_connections 2024-04-22T23:15:21.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:21.080+0000 7f57d7fc6700 1 --2- 172.21.15.134:0/3455744832 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f57bc038410 0x7f57bc03a8d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:21.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:21.080+0000 7f57d7fc6700 1 --2- 172.21.15.134:0/3455744832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57d00ff780 0x7f57d00fccc0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:21.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:21.080+0000 7f57d7fc6700 1 -- 172.21.15.134:0/3455744832 >> 172.21.15.134:0/3455744832 conn(0x7f57d00fad20 msgr2=0x7f57d0068820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:15:21.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:21.080+0000 7f57d7fc6700 1 -- 172.21.15.134:0/3455744832 shutdown_connections 2024-04-22T23:15:21.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:21.080+0000 7f57d7fc6700 1 -- 172.21.15.134:0/3455744832 wait complete. 2024-04-22T23:15:21.082 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-04-22T23:15:21.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:21 smithi067 ceph-mon[26453]: Deploying daemon prometheus.smithi067 on smithi067 2024-04-22T23:15:21.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:21 smithi067 ceph-mon[26453]: from='client.? 172.21.15.134:0/3455744832' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T23:15:22.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:22 smithi067 ceph-mon[26453]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:15:22.615 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T23:15:22.616 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T23:15:22.814 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T23:15:22.860 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T23:15:24.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.156+0000 7ff979dfa700 1 -- 172.21.15.134:0/4173638248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9741013b0 msgr2=0x7ff9741037a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:24.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.156+0000 7ff979dfa700 1 --2- 172.21.15.134:0/4173638248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9741013b0 0x7ff9741037a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff96c009910 tx=0x7ff96c009c20 comp rx=0 tx=0).stop 2024-04-22T23:15:24.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.156+0000 7ff979dfa700 1 -- 172.21.15.134:0/4173638248 shutdown_connections 2024-04-22T23:15:24.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.156+0000 7ff979dfa700 1 --2- 172.21.15.134:0/4173638248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9741013b0 0x7ff9741037a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:24.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.156+0000 7ff979dfa700 1 -- 172.21.15.134:0/4173638248 >> 172.21.15.134:0/4173638248 conn(0x7ff9740fad20 msgr2=0x7ff9740fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:15:24.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.156+0000 7ff979dfa700 1 -- 172.21.15.134:0/4173638248 shutdown_connections 2024-04-22T23:15:24.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.156+0000 7ff979dfa700 1 -- 172.21.15.134:0/4173638248 wait complete. 2024-04-22T23:15:24.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.157+0000 7ff979dfa700 1 Processor -- start 2024-04-22T23:15:24.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.157+0000 7ff979dfa700 1 -- start start 2024-04-22T23:15:24.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.157+0000 7ff979dfa700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9741013b0 0x7ff974195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:15:24.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.157+0000 7ff979dfa700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff974195ec0 con 0x7ff9741013b0 2024-04-22T23:15:24.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.157+0000 7ff9737fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9741013b0 0x7ff974195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:15:24.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.157+0000 7ff9737fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9741013b0 0x7ff974195980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.134:37582/0 (socket says 172.21.15.134:37582) 2024-04-22T23:15:24.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.158+0000 7ff9737fe700 1 -- 172.21.15.134:0/178526123 learned_addr learned my addr 172.21.15.134:0/178526123 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-04-22T23:15:24.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.158+0000 7ff9737fe700 1 -- 172.21.15.134:0/178526123 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff96c0095f0 con 0x7ff9741013b0 2024-04-22T23:15:24.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.158+0000 7ff9737fe700 1 --2- 172.21.15.134:0/178526123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9741013b0 0x7ff974195980 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff96c005b40 tx=0x7ff96c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:15:24.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.158+0000 7ff970ff9700 1 -- 172.21.15.134:0/178526123 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff96c01d070 con 0x7ff9741013b0 2024-04-22T23:15:24.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.158+0000 7ff979dfa700 1 -- 172.21.15.134:0/178526123 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9741960c0 con 0x7ff9741013b0 2024-04-22T23:15:24.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.158+0000 7ff970ff9700 1 -- 172.21.15.134:0/178526123 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff96c00be10 con 0x7ff9741013b0 2024-04-22T23:15:24.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.158+0000 7ff970ff9700 1 -- 172.21.15.134:0/178526123 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff96c00f460 con 0x7ff9741013b0 2024-04-22T23:15:24.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.158+0000 7ff979dfa700 1 -- 172.21.15.134:0/178526123 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff974196560 con 0x7ff9741013b0 2024-04-22T23:15:24.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.159+0000 7ff970ff9700 1 -- 172.21.15.134:0/178526123 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7ff96c00f6d0 con 0x7ff9741013b0 2024-04-22T23:15:24.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.159+0000 7ff979dfa700 1 -- 172.21.15.134:0/178526123 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff97418f4a0 con 0x7ff9741013b0 2024-04-22T23:15:24.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.159+0000 7ff970ff9700 1 --2- 172.21.15.134:0/178526123 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7ff95c0383c0 0x7ff95c03a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:15:24.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.159+0000 7ff970ff9700 1 -- 172.21.15.134:0/178526123 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff96c04c520 con 0x7ff9741013b0 2024-04-22T23:15:24.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.159+0000 7ff972ffd700 1 --2- 172.21.15.134:0/178526123 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7ff95c0383c0 0x7ff95c03a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:15:24.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.160+0000 7ff972ffd700 1 --2- 172.21.15.134:0/178526123 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7ff95c0383c0 0x7ff95c03a880 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff96800ad30 tx=0x7ff9680093f0 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:15:24.165 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.165+0000 7ff970ff9700 1 -- 172.21.15.134:0/178526123 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff96c02a360 con 0x7ff9741013b0 2024-04-22T23:15:24.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.428+0000 7ff979dfa700 1 -- 172.21.15.134:0/178526123 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff97402cc70 con 0x7ff9741013b0 2024-04-22T23:15:24.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.428+0000 7ff970ff9700 1 -- 172.21.15.134:0/178526123 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7ff96c027030 con 0x7ff9741013b0 2024-04-22T23:15:24.429 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:15:24.429 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"d46c346e-00fd-11ef-bc93-c7b262605968","modified":"2024-04-22T23:13:15.759242Z","created":"2024-04-22T23:13:15.759242Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T23:15:24.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.430+0000 7ff979dfa700 1 -- 172.21.15.134:0/178526123 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7ff95c0383c0 msgr2=0x7ff95c03a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:24.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.430+0000 7ff979dfa700 1 --2- 172.21.15.134:0/178526123 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7ff95c0383c0 0x7ff95c03a880 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff96800ad30 tx=0x7ff9680093f0 comp rx=0 tx=0).stop 2024-04-22T23:15:24.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.430+0000 7ff979dfa700 1 -- 172.21.15.134:0/178526123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9741013b0 msgr2=0x7ff974195980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:24.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.430+0000 7ff979dfa700 1 --2- 172.21.15.134:0/178526123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9741013b0 0x7ff974195980 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff96c005b40 tx=0x7ff96c004740 comp rx=0 tx=0).stop 2024-04-22T23:15:24.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.430+0000 7ff979dfa700 1 -- 172.21.15.134:0/178526123 shutdown_connections 2024-04-22T23:15:24.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.430+0000 7ff979dfa700 1 --2- 172.21.15.134:0/178526123 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7ff95c0383c0 0x7ff95c03a880 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:24.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.430+0000 7ff979dfa700 1 --2- 172.21.15.134:0/178526123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9741013b0 0x7ff974195980 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:24.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.430+0000 7ff979dfa700 1 -- 172.21.15.134:0/178526123 >> 172.21.15.134:0/178526123 conn(0x7ff9740fad20 msgr2=0x7ff9740fb9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:15:24.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.430+0000 7ff979dfa700 1 -- 172.21.15.134:0/178526123 shutdown_connections 2024-04-22T23:15:24.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:24.430+0000 7ff979dfa700 1 -- 172.21.15.134:0/178526123 wait complete. 2024-04-22T23:15:24.432 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-04-22T23:15:24.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:24 smithi067 ceph-mon[26453]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:15:24.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:24 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:25.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:25 smithi067 ceph-mon[26453]: from='client.? 172.21.15.134:0/178526123' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T23:15:26.010 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T23:15:26.010 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T23:15:26.210 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T23:15:26.255 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T23:15:26.543 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:26 smithi067 ceph-mon[26453]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:15:27.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.512+0000 7f07cbdb1700 1 -- 172.21.15.134:0/414134445 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07c4100d60 msgr2=0x7f07c4101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:27.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.512+0000 7f07cbdb1700 1 --2- 172.21.15.134:0/414134445 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07c4100d60 0x7f07c4101180 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f07c0009910 tx=0x7f07c0009c20 comp rx=0 tx=0).stop 2024-04-22T23:15:27.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.512+0000 7f07cbdb1700 1 -- 172.21.15.134:0/414134445 shutdown_connections 2024-04-22T23:15:27.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.513+0000 7f07cbdb1700 1 --2- 172.21.15.134:0/414134445 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07c4100d60 0x7f07c4101180 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:27.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.513+0000 7f07cbdb1700 1 -- 172.21.15.134:0/414134445 >> 172.21.15.134:0/414134445 conn(0x7f07c40fa4d0 msgr2=0x7f07c40fc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:15:27.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.513+0000 7f07cbdb1700 1 -- 172.21.15.134:0/414134445 shutdown_connections 2024-04-22T23:15:27.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.513+0000 7f07cbdb1700 1 -- 172.21.15.134:0/414134445 wait complete. 2024-04-22T23:15:27.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.513+0000 7f07cbdb1700 1 Processor -- start 2024-04-22T23:15:27.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.513+0000 7f07cbdb1700 1 -- start start 2024-04-22T23:15:27.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.513+0000 7f07cbdb1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07c4100d60 0x7f07c40ff430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:15:27.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.513+0000 7f07cbdb1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c41986a0 con 0x7f07c4100d60 2024-04-22T23:15:27.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.514+0000 7f07c9b4d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07c4100d60 0x7f07c40ff430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:15:27.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.514+0000 7f07c9b4d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07c4100d60 0x7f07c40ff430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.134:37598/0 (socket says 172.21.15.134:37598) 2024-04-22T23:15:27.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.514+0000 7f07c9b4d700 1 -- 172.21.15.134:0/4178656090 learned_addr learned my addr 172.21.15.134:0/4178656090 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-04-22T23:15:27.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.514+0000 7f07c9b4d700 1 -- 172.21.15.134:0/4178656090 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c00095f0 con 0x7f07c4100d60 2024-04-22T23:15:27.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.514+0000 7f07c9b4d700 1 --2- 172.21.15.134:0/4178656090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07c4100d60 0x7f07c40ff430 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f07c0005b40 tx=0x7f07c0004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:15:27.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.514+0000 7f07baffd700 1 -- 172.21.15.134:0/4178656090 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07c001d070 con 0x7f07c4100d60 2024-04-22T23:15:27.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.514+0000 7f07cbdb1700 1 -- 172.21.15.134:0/4178656090 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07c41987e0 con 0x7f07c4100d60 2024-04-22T23:15:27.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.514+0000 7f07baffd700 1 -- 172.21.15.134:0/4178656090 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f07c000be10 con 0x7f07c4100d60 2024-04-22T23:15:27.518 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.514+0000 7f07cbdb1700 1 -- 172.21.15.134:0/4178656090 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07c4198c80 con 0x7f07c4100d60 2024-04-22T23:15:27.518 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.514+0000 7f07baffd700 1 -- 172.21.15.134:0/4178656090 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07c000f460 con 0x7f07c4100d60 2024-04-22T23:15:27.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.515+0000 7f07baffd700 1 -- 172.21.15.134:0/4178656090 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) v1 ==== 45339+0+0 (secure 0 0 0) 0x7f07c000b960 con 0x7f07c4100d60 2024-04-22T23:15:27.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.515+0000 7f07cbdb1700 1 -- 172.21.15.134:0/4178656090 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07c4062430 con 0x7f07c4100d60 2024-04-22T23:15:27.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.515+0000 7f07baffd700 1 --2- 172.21.15.134:0/4178656090 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f07b00383c0 0x7f07b003a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:15:27.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.515+0000 7f07baffd700 1 -- 172.21.15.134:0/4178656090 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f07c004c4b0 con 0x7f07c4100d60 2024-04-22T23:15:27.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.516+0000 7f07c934c700 1 --2- 172.21.15.134:0/4178656090 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f07b00383c0 0x7f07b003a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:15:27.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.516+0000 7f07c934c700 1 --2- 172.21.15.134:0/4178656090 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f07b00383c0 0x7f07b003a880 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f07b4006fd0 tx=0x7f07b4006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:15:27.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.521+0000 7f07baffd700 1 -- 172.21.15.134:0/4178656090 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f07c4062430 con 0x7f07c4100d60 2024-04-22T23:15:27.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.785+0000 7f07cbdb1700 1 -- 172.21.15.134:0/4178656090 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f07c4062430 con 0x7f07c4100d60 2024-04-22T23:15:27.788 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.786+0000 7f07baffd700 1 -- 172.21.15.134:0/4178656090 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f07c4062430 con 0x7f07c4100d60 2024-04-22T23:15:27.788 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:15:27.788 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"d46c346e-00fd-11ef-bc93-c7b262605968","modified":"2024-04-22T23:13:15.759242Z","created":"2024-04-22T23:13:15.759242Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T23:15:27.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.788+0000 7f07cbdb1700 1 -- 172.21.15.134:0/4178656090 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f07b00383c0 msgr2=0x7f07b003a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:27.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.788+0000 7f07cbdb1700 1 --2- 172.21.15.134:0/4178656090 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f07b00383c0 0x7f07b003a880 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f07b4006fd0 tx=0x7f07b4006e40 comp rx=0 tx=0).stop 2024-04-22T23:15:27.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.788+0000 7f07cbdb1700 1 -- 172.21.15.134:0/4178656090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07c4100d60 msgr2=0x7f07c40ff430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:27.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.788+0000 7f07cbdb1700 1 --2- 172.21.15.134:0/4178656090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07c4100d60 0x7f07c40ff430 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f07c0005b40 tx=0x7f07c0004740 comp rx=0 tx=0).stop 2024-04-22T23:15:27.791 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.788+0000 7f07cbdb1700 1 -- 172.21.15.134:0/4178656090 shutdown_connections 2024-04-22T23:15:27.791 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.788+0000 7f07cbdb1700 1 --2- 172.21.15.134:0/4178656090 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f07b00383c0 0x7f07b003a880 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:27.791 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.788+0000 7f07cbdb1700 1 --2- 172.21.15.134:0/4178656090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07c4100d60 0x7f07c40ff430 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:27.791 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.788+0000 7f07cbdb1700 1 -- 172.21.15.134:0/4178656090 >> 172.21.15.134:0/4178656090 conn(0x7f07c40fa4d0 msgr2=0x7f07c40fb140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:15:27.791 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.788+0000 7f07cbdb1700 1 -- 172.21.15.134:0/4178656090 shutdown_connections 2024-04-22T23:15:27.791 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:27.788+0000 7f07cbdb1700 1 -- 172.21.15.134:0/4178656090 wait complete. 2024-04-22T23:15:27.791 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-04-22T23:15:28.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:28 smithi067 ceph-mon[26453]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:15:28.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:28 smithi067 ceph-mon[26453]: from='client.? 172.21.15.134:0/4178656090' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T23:15:29.361 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T23:15:29.361 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T23:15:29.573 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T23:15:29.620 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T23:15:30.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:30 smithi067 ceph-mon[26453]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:15:30.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:30 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:30.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:30 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:30.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:30 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:30.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:30 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-22T23:15:30.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:30.919+0000 7f7f5b54c700 1 -- 172.21.15.134:0/1691008114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f541013a0 msgr2=0x7f7f54103790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:30.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:30.919+0000 7f7f5b54c700 1 --2- 172.21.15.134:0/1691008114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f541013a0 0x7f7f54103790 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7f50009910 tx=0x7f7f50009c20 comp rx=0 tx=0).stop 2024-04-22T23:15:30.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:30.919+0000 7f7f5b54c700 1 -- 172.21.15.134:0/1691008114 shutdown_connections 2024-04-22T23:15:30.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:30.919+0000 7f7f5b54c700 1 --2- 172.21.15.134:0/1691008114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f541013a0 0x7f7f54103790 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:30.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:30.919+0000 7f7f5b54c700 1 -- 172.21.15.134:0/1691008114 >> 172.21.15.134:0/1691008114 conn(0x7f7f540fad30 msgr2=0x7f7f540fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:15:30.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:30.919+0000 7f7f5b54c700 1 -- 172.21.15.134:0/1691008114 shutdown_connections 2024-04-22T23:15:30.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:30.919+0000 7f7f5b54c700 1 -- 172.21.15.134:0/1691008114 wait complete. 2024-04-22T23:15:30.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:30.919+0000 7f7f5b54c700 1 Processor -- start 2024-04-22T23:15:30.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:30.920+0000 7f7f5b54c700 1 -- start start 2024-04-22T23:15:30.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:30.920+0000 7f7f5b54c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f541013a0 0x7f7f54100d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:15:30.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:30.920+0000 7f7f5b54c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f54101240 con 0x7f7f541013a0 2024-04-22T23:15:30.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:30.920+0000 7f7f592e8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f541013a0 0x7f7f54100d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:15:30.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:30.920+0000 7f7f592e8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f541013a0 0x7f7f54100d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.134:37642/0 (socket says 172.21.15.134:37642) 2024-04-22T23:15:30.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:30.920+0000 7f7f592e8700 1 -- 172.21.15.134:0/1305609621 learned_addr learned my addr 172.21.15.134:0/1305609621 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-04-22T23:15:30.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:30.920+0000 7f7f592e8700 1 -- 172.21.15.134:0/1305609621 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f500095f0 con 0x7f7f541013a0 2024-04-22T23:15:30.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:30.920+0000 7f7f592e8700 1 --2- 172.21.15.134:0/1305609621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f541013a0 0x7f7f54100d00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7f50005b40 tx=0x7f7f50004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:15:30.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:30.921+0000 7f7f4a7fc700 1 -- 172.21.15.134:0/1305609621 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f5001d070 con 0x7f7f541013a0 2024-04-22T23:15:30.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:30.921+0000 7f7f5b54c700 1 -- 172.21.15.134:0/1305609621 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f540ff350 con 0x7f7f541013a0 2024-04-22T23:15:30.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:30.921+0000 7f7f4a7fc700 1 -- 172.21.15.134:0/1305609621 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7f5000be10 con 0x7f7f541013a0 2024-04-22T23:15:30.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:30.921+0000 7f7f4a7fc700 1 -- 172.21.15.134:0/1305609621 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f5000f460 con 0x7f7f541013a0 2024-04-22T23:15:30.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:30.921+0000 7f7f5b54c700 1 -- 172.21.15.134:0/1305609621 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f540ff7f0 con 0x7f7f541013a0 2024-04-22T23:15:30.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:30.921+0000 7f7f5b54c700 1 -- 172.21.15.134:0/1305609621 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f540623c0 con 0x7f7f541013a0 2024-04-22T23:15:30.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:30.921+0000 7f7f4a7fc700 1 -- 172.21.15.134:0/1305609621 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 14) v1 ==== 45353+0+0 (secure 0 0 0) 0x7f7f5000b960 con 0x7f7f541013a0 2024-04-22T23:15:30.925 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:30.922+0000 7f7f4a7fc700 1 --2- 172.21.15.134:0/1305609621 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f7f40038410 0x7f7f4003a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:15:30.925 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:30.922+0000 7f7f4a7fc700 1 -- 172.21.15.134:0/1305609621 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7f5004c510 con 0x7f7f541013a0 2024-04-22T23:15:30.925 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:30.922+0000 7f7f58ae7700 1 -- 172.21.15.134:0/1305609621 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f7f40038410 msgr2=0x7f7f4003a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/2590792187 2024-04-22T23:15:30.925 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:30.922+0000 7f7f58ae7700 1 --2- 172.21.15.134:0/1305609621 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f7f40038410 0x7f7f4003a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T23:15:30.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:30.927+0000 7f7f4a7fc700 1 -- 172.21.15.134:0/1305609621 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f7f5002a360 con 0x7f7f541013a0 2024-04-22T23:15:31.125 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:31.123+0000 7f7f58ae7700 1 -- 172.21.15.134:0/1305609621 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f7f40038410 msgr2=0x7f7f4003a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/2590792187 2024-04-22T23:15:31.125 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:31.123+0000 7f7f58ae7700 1 --2- 172.21.15.134:0/1305609621 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f7f40038410 0x7f7f4003a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T23:15:31.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:31.191+0000 7f7f5b54c700 1 -- 172.21.15.134:0/1305609621 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7f5418ee90 con 0x7f7f541013a0 2024-04-22T23:15:31.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:31.192+0000 7f7f4a7fc700 1 -- 172.21.15.134:0/1305609621 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f7f50027070 con 0x7f7f541013a0 2024-04-22T23:15:31.194 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:15:31.194 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"d46c346e-00fd-11ef-bc93-c7b262605968","modified":"2024-04-22T23:13:15.759242Z","created":"2024-04-22T23:13:15.759242Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T23:15:31.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:31.194+0000 7f7f5b54c700 1 -- 172.21.15.134:0/1305609621 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f7f40038410 msgr2=0x7f7f4003a8d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T23:15:31.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:31.194+0000 7f7f5b54c700 1 --2- 172.21.15.134:0/1305609621 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f7f40038410 0x7f7f4003a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:31.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:31.194+0000 7f7f5b54c700 1 -- 172.21.15.134:0/1305609621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f541013a0 msgr2=0x7f7f54100d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:31.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:31.194+0000 7f7f5b54c700 1 --2- 172.21.15.134:0/1305609621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f541013a0 0x7f7f54100d00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7f50005b40 tx=0x7f7f50004740 comp rx=0 tx=0).stop 2024-04-22T23:15:31.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:31.194+0000 7f7f5b54c700 1 -- 172.21.15.134:0/1305609621 shutdown_connections 2024-04-22T23:15:31.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:31.194+0000 7f7f5b54c700 1 --2- 172.21.15.134:0/1305609621 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7f7f40038410 0x7f7f4003a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:31.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:31.194+0000 7f7f5b54c700 1 --2- 172.21.15.134:0/1305609621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f541013a0 0x7f7f54100d00 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:31.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:31.194+0000 7f7f5b54c700 1 -- 172.21.15.134:0/1305609621 >> 172.21.15.134:0/1305609621 conn(0x7f7f540fad30 msgr2=0x7f7f540fba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:15:31.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:31.194+0000 7f7f5b54c700 1 -- 172.21.15.134:0/1305609621 shutdown_connections 2024-04-22T23:15:31.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:31.194+0000 7f7f5b54c700 1 -- 172.21.15.134:0/1305609621 wait complete. 2024-04-22T23:15:31.197 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-04-22T23:15:31.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:31 smithi067 ceph-mon[26453]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-22T23:15:31.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:31 smithi067 ceph-mon[26453]: mgrmap e14: smithi067.whdcsr(active, since 61s) 2024-04-22T23:15:31.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:31 smithi067 ceph-mon[26453]: from='client.? 172.21.15.134:0/1305609621' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T23:15:32.800 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T23:15:32.801 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T23:15:33.001 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T23:15:33.047 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T23:15:34.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.323+0000 7faa0fd43700 1 -- 172.21.15.134:0/1179468471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa081008a0 msgr2=0x7faa08100cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:34.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.323+0000 7faa0fd43700 1 --2- 172.21.15.134:0/1179468471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa081008a0 0x7faa08100cc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7faa0000b0a0 tx=0x7faa0000b3b0 comp rx=0 tx=0).stop 2024-04-22T23:15:34.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.323+0000 7faa0fd43700 1 -- 172.21.15.134:0/1179468471 shutdown_connections 2024-04-22T23:15:34.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.323+0000 7faa0fd43700 1 --2- 172.21.15.134:0/1179468471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa081008a0 0x7faa08100cc0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:34.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.323+0000 7faa0fd43700 1 -- 172.21.15.134:0/1179468471 >> 172.21.15.134:0/1179468471 conn(0x7faa080fbe20 msgr2=0x7faa080fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:15:34.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.323+0000 7faa0fd43700 1 -- 172.21.15.134:0/1179468471 shutdown_connections 2024-04-22T23:15:34.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.323+0000 7faa0fd43700 1 -- 172.21.15.134:0/1179468471 wait complete. 2024-04-22T23:15:34.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.324+0000 7faa0fd43700 1 Processor -- start 2024-04-22T23:15:34.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.324+0000 7faa0fd43700 1 -- start start 2024-04-22T23:15:34.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.324+0000 7faa0fd43700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa081008a0 0x7faa081959f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:15:34.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.324+0000 7faa0fd43700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa08195f30 con 0x7faa081008a0 2024-04-22T23:15:34.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.324+0000 7faa0dadf700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa081008a0 0x7faa081959f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:15:34.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.324+0000 7faa0dadf700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa081008a0 0x7faa081959f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.134:37664/0 (socket says 172.21.15.134:37664) 2024-04-22T23:15:34.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.324+0000 7faa0dadf700 1 -- 172.21.15.134:0/3253880701 learned_addr learned my addr 172.21.15.134:0/3253880701 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-04-22T23:15:34.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.325+0000 7faa0dadf700 1 -- 172.21.15.134:0/3253880701 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa00009d00 con 0x7faa081008a0 2024-04-22T23:15:34.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.325+0000 7faa0dadf700 1 --2- 172.21.15.134:0/3253880701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa081008a0 0x7faa081959f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7faa00009fd0 tx=0x7faa0000bb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:15:34.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.325+0000 7faa06ffd700 1 -- 172.21.15.134:0/3253880701 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa0000e040 con 0x7faa081008a0 2024-04-22T23:15:34.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.325+0000 7faa0fd43700 1 -- 172.21.15.134:0/3253880701 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa08196130 con 0x7faa081008a0 2024-04-22T23:15:34.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.325+0000 7faa06ffd700 1 -- 172.21.15.134:0/3253880701 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7faa0000bdc0 con 0x7faa081008a0 2024-04-22T23:15:34.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.325+0000 7faa06ffd700 1 -- 172.21.15.134:0/3253880701 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa0001da40 con 0x7faa081008a0 2024-04-22T23:15:34.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.325+0000 7faa0fd43700 1 -- 172.21.15.134:0/3253880701 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa08196550 con 0x7faa081008a0 2024-04-22T23:15:34.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.326+0000 7faa06ffd700 1 -- 172.21.15.134:0/3253880701 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 14) v1 ==== 45353+0+0 (secure 0 0 0) 0x7faa0001dc80 con 0x7faa081008a0 2024-04-22T23:15:34.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.326+0000 7faa0fd43700 1 -- 172.21.15.134:0/3253880701 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa0818f4c0 con 0x7faa081008a0 2024-04-22T23:15:34.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.326+0000 7faa06ffd700 1 --2- 172.21.15.134:0/3253880701 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7fa9f0038460 0x7fa9f003a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:15:34.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.326+0000 7faa06ffd700 1 -- 172.21.15.134:0/3253880701 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7faa0004c910 con 0x7faa081008a0 2024-04-22T23:15:34.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.326+0000 7faa0d2de700 1 -- 172.21.15.134:0/3253880701 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7fa9f0038460 msgr2=0x7fa9f003a920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/2590792187 2024-04-22T23:15:34.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.326+0000 7faa0d2de700 1 --2- 172.21.15.134:0/3253880701 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7fa9f0038460 0x7fa9f003a920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T23:15:34.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.331+0000 7faa06ffd700 1 -- 172.21.15.134:0/3253880701 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7faa0001c5c0 con 0x7faa081008a0 2024-04-22T23:15:34.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.527+0000 7faa0d2de700 1 -- 172.21.15.134:0/3253880701 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7fa9f0038460 msgr2=0x7fa9f003a920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/2590792187 2024-04-22T23:15:34.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.527+0000 7faa0d2de700 1 --2- 172.21.15.134:0/3253880701 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7fa9f0038460 0x7fa9f003a920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T23:15:34.597 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.595+0000 7faa0fd43700 1 -- 172.21.15.134:0/3253880701 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faa080623c0 con 0x7faa081008a0 2024-04-22T23:15:34.597 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.595+0000 7faa06ffd700 1 -- 172.21.15.134:0/3253880701 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7faa0002c200 con 0x7faa081008a0 2024-04-22T23:15:34.598 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:15:34.598 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"d46c346e-00fd-11ef-bc93-c7b262605968","modified":"2024-04-22T23:13:15.759242Z","created":"2024-04-22T23:13:15.759242Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T23:15:34.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.598+0000 7faa0fd43700 1 -- 172.21.15.134:0/3253880701 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7fa9f0038460 msgr2=0x7fa9f003a920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T23:15:34.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.598+0000 7faa0fd43700 1 --2- 172.21.15.134:0/3253880701 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7fa9f0038460 0x7fa9f003a920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:34.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.598+0000 7faa0fd43700 1 -- 172.21.15.134:0/3253880701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa081008a0 msgr2=0x7faa081959f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:34.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.598+0000 7faa0fd43700 1 --2- 172.21.15.134:0/3253880701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa081008a0 0x7faa081959f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7faa00009fd0 tx=0x7faa0000bb10 comp rx=0 tx=0).stop 2024-04-22T23:15:34.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.598+0000 7faa0fd43700 1 -- 172.21.15.134:0/3253880701 shutdown_connections 2024-04-22T23:15:34.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.598+0000 7faa0fd43700 1 --2- 172.21.15.134:0/3253880701 >> [v2:172.21.15.67:6800/2590792187,v1:172.21.15.67:6801/2590792187] conn(0x7fa9f0038460 0x7fa9f003a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:34.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.598+0000 7faa0fd43700 1 --2- 172.21.15.134:0/3253880701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa081008a0 0x7faa081959f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:34.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.598+0000 7faa0fd43700 1 -- 172.21.15.134:0/3253880701 >> 172.21.15.134:0/3253880701 conn(0x7faa080fbe20 msgr2=0x7faa080fcb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:15:34.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.598+0000 7faa0fd43700 1 -- 172.21.15.134:0/3253880701 shutdown_connections 2024-04-22T23:15:34.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:34.598+0000 7faa0fd43700 1 -- 172.21.15.134:0/3253880701 wait complete. 2024-04-22T23:15:34.601 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-04-22T23:15:34.997 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:34 smithi067 ceph-mon[26453]: from='client.? 172.21.15.134:0/3253880701' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T23:15:36.200 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T23:15:36.201 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T23:15:36.399 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T23:15:36.445 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T23:15:37.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:37 smithi067 ceph-mon[26453]: Active manager daemon smithi067.whdcsr restarted 2024-04-22T23:15:37.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:37 smithi067 ceph-mon[26453]: Activating manager daemon smithi067.whdcsr 2024-04-22T23:15:37.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:37 smithi067 ceph-mon[26453]: osdmap e5: 0 total, 0 up, 0 in 2024-04-22T23:15:37.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:37 smithi067 ceph-mon[26453]: mgrmap e15: smithi067.whdcsr(active, starting, since 0.00333419s) 2024-04-22T23:15:37.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:37 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-22T23:15:37.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:37 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mgr metadata", "who": "smithi067.whdcsr", "id": "smithi067.whdcsr"}]: dispatch 2024-04-22T23:15:37.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:37 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T23:15:37.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:37 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T23:15:37.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:37 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T23:15:37.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:37 smithi067 ceph-mon[26453]: Manager daemon smithi067.whdcsr is now available 2024-04-22T23:15:37.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:37 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:15:37.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:37 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:15:37.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:37 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:37.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:37.980+0000 7fb47e977700 1 -- 172.21.15.134:0/835533673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4780ff780 msgr2=0x7fb4780ffba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:37.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:37.980+0000 7fb47e977700 1 --2- 172.21.15.134:0/835533673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4780ff780 0x7fb4780ffba0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb46c009910 tx=0x7fb46c009c20 comp rx=0 tx=0).stop 2024-04-22T23:15:37.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:37.980+0000 7fb47e977700 1 -- 172.21.15.134:0/835533673 shutdown_connections 2024-04-22T23:15:37.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:37.980+0000 7fb47e977700 1 --2- 172.21.15.134:0/835533673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4780ff780 0x7fb4780ffba0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:37.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:37.980+0000 7fb47e977700 1 -- 172.21.15.134:0/835533673 >> 172.21.15.134:0/835533673 conn(0x7fb4780fad00 msgr2=0x7fb4780fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:15:37.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:37.980+0000 7fb47e977700 1 -- 172.21.15.134:0/835533673 shutdown_connections 2024-04-22T23:15:37.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:37.980+0000 7fb47e977700 1 -- 172.21.15.134:0/835533673 wait complete. 2024-04-22T23:15:37.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:37.980+0000 7fb47e977700 1 Processor -- start 2024-04-22T23:15:37.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:37.980+0000 7fb47e977700 1 -- start start 2024-04-22T23:15:37.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:37.981+0000 7fb47e977700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4780ff780 0x7fb4781959b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:15:37.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:37.981+0000 7fb47e977700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb478195ef0 con 0x7fb4780ff780 2024-04-22T23:15:37.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:37.981+0000 7fb477fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4780ff780 0x7fb4781959b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:15:37.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:37.981+0000 7fb477fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4780ff780 0x7fb4781959b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.134:37690/0 (socket says 172.21.15.134:37690) 2024-04-22T23:15:37.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:37.981+0000 7fb477fff700 1 -- 172.21.15.134:0/3989777031 learned_addr learned my addr 172.21.15.134:0/3989777031 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-04-22T23:15:37.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:37.981+0000 7fb477fff700 1 -- 172.21.15.134:0/3989777031 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb46c0095f0 con 0x7fb4780ff780 2024-04-22T23:15:37.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:37.981+0000 7fb477fff700 1 --2- 172.21.15.134:0/3989777031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4780ff780 0x7fb4781959b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb46c005b40 tx=0x7fb46c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:15:37.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:37.982+0000 7fb4757fa700 1 -- 172.21.15.134:0/3989777031 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb46c01d070 con 0x7fb4780ff780 2024-04-22T23:15:37.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:37.982+0000 7fb47e977700 1 -- 172.21.15.134:0/3989777031 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4781960f0 con 0x7fb4780ff780 2024-04-22T23:15:37.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:37.982+0000 7fb4757fa700 1 -- 172.21.15.134:0/3989777031 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb46c00be10 con 0x7fb4780ff780 2024-04-22T23:15:37.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:37.982+0000 7fb4757fa700 1 -- 172.21.15.134:0/3989777031 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb46c00f460 con 0x7fb4780ff780 2024-04-22T23:15:37.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:37.982+0000 7fb47e977700 1 -- 172.21.15.134:0/3989777031 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb478196590 con 0x7fb4780ff780 2024-04-22T23:15:37.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:37.983+0000 7fb4757fa700 1 -- 172.21.15.134:0/3989777031 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 15) v1 ==== 45077+0+0 (secure 0 0 0) 0x7fb46c00f6d0 con 0x7fb4780ff780 2024-04-22T23:15:37.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:37.983+0000 7fb47e977700 1 -- 172.21.15.134:0/3989777031 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb47818f3b0 con 0x7fb4780ff780 2024-04-22T23:15:37.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:37.983+0000 7fb4757fa700 1 -- 172.21.15.134:0/3989777031 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fb46c04c0a0 con 0x7fb4780ff780 2024-04-22T23:15:37.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:37.988+0000 7fb4757fa700 1 -- 172.21.15.134:0/3989777031 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb46c00fdf0 con 0x7fb4780ff780 2024-04-22T23:15:38.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:38.265+0000 7fb47e977700 1 -- 172.21.15.134:0/3989777031 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb47802cc70 con 0x7fb4780ff780 2024-04-22T23:15:38.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:38.266+0000 7fb4757fa700 1 -- 172.21.15.134:0/3989777031 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fb46c027020 con 0x7fb4780ff780 2024-04-22T23:15:38.267 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:15:38.267 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"d46c346e-00fd-11ef-bc93-c7b262605968","modified":"2024-04-22T23:13:15.759242Z","created":"2024-04-22T23:13:15.759242Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T23:15:38.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:38.268+0000 7fb47e977700 1 -- 172.21.15.134:0/3989777031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4780ff780 msgr2=0x7fb4781959b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:38.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:38.268+0000 7fb47e977700 1 --2- 172.21.15.134:0/3989777031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4780ff780 0x7fb4781959b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb46c005b40 tx=0x7fb46c004740 comp rx=0 tx=0).stop 2024-04-22T23:15:38.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:38.268+0000 7fb47e977700 1 -- 172.21.15.134:0/3989777031 shutdown_connections 2024-04-22T23:15:38.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:38.268+0000 7fb47e977700 1 --2- 172.21.15.134:0/3989777031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4780ff780 0x7fb4781959b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:38.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:38.268+0000 7fb47e977700 1 -- 172.21.15.134:0/3989777031 >> 172.21.15.134:0/3989777031 conn(0x7fb4780fad00 msgr2=0x7fb4780690b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:15:38.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:38.268+0000 7fb47e977700 1 -- 172.21.15.134:0/3989777031 shutdown_connections 2024-04-22T23:15:38.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:38.268+0000 7fb47e977700 1 -- 172.21.15.134:0/3989777031 wait complete. 2024-04-22T23:15:38.269 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-04-22T23:15:38.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:38 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:15:38.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:38 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.whdcsr/mirror_snapshot_schedule"}]: dispatch 2024-04-22T23:15:38.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:38 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.whdcsr/trash_purge_schedule"}]: dispatch 2024-04-22T23:15:38.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:38 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:38.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:38 smithi067 ceph-mon[26453]: from='client.? 172.21.15.134:0/3989777031' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T23:15:38.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:38 smithi067 ceph-mon[26453]: mgrmap e16: smithi067.whdcsr(active, since 1.00563s) 2024-04-22T23:15:39.753 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T23:15:39.753 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T23:15:39.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:39 smithi067 ceph-mon[26453]: [22/Apr/2024:23:15:37] ENGINE Bus STARTING 2024-04-22T23:15:39.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:39 smithi067 ceph-mon[26453]: [22/Apr/2024:23:15:37] ENGINE Serving on https://172.21.15.67:7150 2024-04-22T23:15:39.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:39 smithi067 ceph-mon[26453]: [22/Apr/2024:23:15:38] ENGINE Serving on http://172.21.15.67:8765 2024-04-22T23:15:39.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:39 smithi067 ceph-mon[26453]: [22/Apr/2024:23:15:38] ENGINE Bus STARTED 2024-04-22T23:15:39.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:39 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:40.480 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T23:15:40.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:40 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:40.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:40 smithi067 ceph-mon[26453]: mgrmap e17: smithi067.whdcsr(active, since 2s) 2024-04-22T23:15:40.855 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-22T23:15:43.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:42 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:43.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:42 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:43.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:42 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:43.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:42 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:43.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:42 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:43.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:42 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:43.046 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:42 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "who": "osd/host:smithi134", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:15:43.046 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:42 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:43.046 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:42 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:43.046 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:42 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "who": "osd/host:smithi067", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:15:43.046 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:42 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:15:43.046 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:42 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:15:43.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.301+0000 7f20a8457700 1 -- 172.21.15.134:0/89389002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20a0100880 msgr2=0x7f20a0100ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:43.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.301+0000 7f20a8457700 1 --2- 172.21.15.134:0/89389002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20a0100880 0x7f20a0100ca0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2098009910 tx=0x7f2098009c20 comp rx=0 tx=0).stop 2024-04-22T23:15:43.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.302+0000 7f20a8457700 1 -- 172.21.15.134:0/89389002 shutdown_connections 2024-04-22T23:15:43.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.302+0000 7f20a8457700 1 --2- 172.21.15.134:0/89389002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20a0100880 0x7f20a0100ca0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:43.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.302+0000 7f20a8457700 1 -- 172.21.15.134:0/89389002 >> 172.21.15.134:0/89389002 conn(0x7f20a00fbe00 msgr2=0x7f20a00fe260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:15:43.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.302+0000 7f20a8457700 1 -- 172.21.15.134:0/89389002 shutdown_connections 2024-04-22T23:15:43.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.302+0000 7f20a8457700 1 -- 172.21.15.134:0/89389002 wait complete. 2024-04-22T23:15:43.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.302+0000 7f20a8457700 1 Processor -- start 2024-04-22T23:15:43.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.302+0000 7f20a8457700 1 -- start start 2024-04-22T23:15:43.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.302+0000 7f20a8457700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20a0100880 0x7f20a0074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:15:43.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.302+0000 7f20a8457700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a0075070 con 0x7f20a0100880 2024-04-22T23:15:43.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.303+0000 7f20a61f3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20a0100880 0x7f20a0074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:15:43.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.303+0000 7f20a61f3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20a0100880 0x7f20a0074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.134:47496/0 (socket says 172.21.15.134:47496) 2024-04-22T23:15:43.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.303+0000 7f20a61f3700 1 -- 172.21.15.134:0/2460354466 learned_addr learned my addr 172.21.15.134:0/2460354466 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-04-22T23:15:43.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.303+0000 7f20a61f3700 1 -- 172.21.15.134:0/2460354466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20980095f0 con 0x7f20a0100880 2024-04-22T23:15:43.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.303+0000 7f20a61f3700 1 --2- 172.21.15.134:0/2460354466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20a0100880 0x7f20a0074b30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2098005b40 tx=0x7f2098004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:15:43.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.303+0000 7f20977fe700 1 -- 172.21.15.134:0/2460354466 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f209801d070 con 0x7f20a0100880 2024-04-22T23:15:43.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.303+0000 7f20977fe700 1 -- 172.21.15.134:0/2460354466 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f209800be10 con 0x7f20a0100880 2024-04-22T23:15:43.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.303+0000 7f20a8457700 1 -- 172.21.15.134:0/2460354466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20a0073180 con 0x7f20a0100880 2024-04-22T23:15:43.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.303+0000 7f20977fe700 1 -- 172.21.15.134:0/2460354466 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f209800f460 con 0x7f20a0100880 2024-04-22T23:15:43.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.303+0000 7f20a8457700 1 -- 172.21.15.134:0/2460354466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20a0073620 con 0x7f20a0100880 2024-04-22T23:15:43.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.304+0000 7f20977fe700 1 -- 172.21.15.134:0/2460354466 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 45396+0+0 (secure 0 0 0) 0x7f209800f6d0 con 0x7f20a0100880 2024-04-22T23:15:43.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.304+0000 7f20a8457700 1 -- 172.21.15.134:0/2460354466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20a00623c0 con 0x7f20a0100880 2024-04-22T23:15:43.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.304+0000 7f20977fe700 1 --2- 172.21.15.134:0/2460354466 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f208c0384b0 0x7f208c03a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:15:43.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.304+0000 7f20977fe700 1 -- 172.21.15.134:0/2460354466 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f209804c780 con 0x7f20a0100880 2024-04-22T23:15:43.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.305+0000 7f20a59f2700 1 --2- 172.21.15.134:0/2460354466 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f208c0384b0 0x7f208c03a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:15:43.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.305+0000 7f20a59f2700 1 --2- 172.21.15.134:0/2460354466 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f208c0384b0 0x7f208c03a970 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2090006fd0 tx=0x7f2090006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:15:43.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.309+0000 7f20977fe700 1 -- 172.21.15.134:0/2460354466 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f209802a360 con 0x7f20a0100880 2024-04-22T23:15:43.587 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.586+0000 7f20a8457700 1 -- 172.21.15.134:0/2460354466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f20a018ee80 con 0x7f20a0100880 2024-04-22T23:15:43.588 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.587+0000 7f20977fe700 1 -- 172.21.15.134:0/2460354466 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f2098027030 con 0x7f20a0100880 2024-04-22T23:15:43.588 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:15:43.589 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"d46c346e-00fd-11ef-bc93-c7b262605968","modified":"2024-04-22T23:13:15.759242Z","created":"2024-04-22T23:13:15.759242Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T23:15:43.590 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.589+0000 7f20a8457700 1 -- 172.21.15.134:0/2460354466 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f208c0384b0 msgr2=0x7f208c03a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:43.590 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.589+0000 7f20a8457700 1 --2- 172.21.15.134:0/2460354466 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f208c0384b0 0x7f208c03a970 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2090006fd0 tx=0x7f2090006e40 comp rx=0 tx=0).stop 2024-04-22T23:15:43.590 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.589+0000 7f20a8457700 1 -- 172.21.15.134:0/2460354466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20a0100880 msgr2=0x7f20a0074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:43.591 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.589+0000 7f20a8457700 1 --2- 172.21.15.134:0/2460354466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20a0100880 0x7f20a0074b30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2098005b40 tx=0x7f2098004740 comp rx=0 tx=0).stop 2024-04-22T23:15:43.591 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.589+0000 7f20a8457700 1 -- 172.21.15.134:0/2460354466 shutdown_connections 2024-04-22T23:15:43.591 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.589+0000 7f20a8457700 1 --2- 172.21.15.134:0/2460354466 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f208c0384b0 0x7f208c03a970 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:43.591 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.589+0000 7f20a8457700 1 --2- 172.21.15.134:0/2460354466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20a0100880 0x7f20a0074b30 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:43.591 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.589+0000 7f20a8457700 1 -- 172.21.15.134:0/2460354466 >> 172.21.15.134:0/2460354466 conn(0x7f20a00fbe00 msgr2=0x7f20a00fcae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:15:43.591 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.589+0000 7f20a8457700 1 -- 172.21.15.134:0/2460354466 shutdown_connections 2024-04-22T23:15:43.591 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:43.589+0000 7f20a8457700 1 -- 172.21.15.134:0/2460354466 wait complete. 2024-04-22T23:15:43.591 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-04-22T23:15:44.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:43 smithi067 ceph-mon[26453]: Updating smithi067:/etc/ceph/ceph.conf 2024-04-22T23:15:44.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:43 smithi067 ceph-mon[26453]: Updating smithi134:/etc/ceph/ceph.conf 2024-04-22T23:15:44.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:43 smithi067 ceph-mon[26453]: Updating smithi067:/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T23:15:44.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:43 smithi067 ceph-mon[26453]: Updating smithi134:/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T23:15:44.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:43 smithi067 ceph-mon[26453]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-04-22T23:15:44.046 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:43 smithi067 ceph-mon[26453]: Updating smithi134:/etc/ceph/ceph.client.admin.keyring 2024-04-22T23:15:44.046 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:43 smithi067 ceph-mon[26453]: Updating smithi067:/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T23:15:44.046 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:43 smithi067 ceph-mon[26453]: Updating smithi134:/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T23:15:44.046 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:43 smithi067 ceph-mon[26453]: from='client.? 172.21.15.134:0/2460354466' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T23:15:44.046 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:43 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:44.046 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:43 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:44.046 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:43 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:44.046 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:43 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:44.046 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:43 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:44.046 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:43 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi134", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T23:15:44.046 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:43 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi134", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-22T23:15:44.046 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:43 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:15:45.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:44 smithi067 ceph-mon[26453]: Deploying daemon ceph-exporter.smithi134 on smithi134 2024-04-22T23:15:45.193 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T23:15:45.193 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T23:15:45.437 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T23:15:46.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:46.969+0000 7fa1df312700 1 -- 172.21.15.134:0/1771824762 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1d8105720 msgr2=0x7fa1d8105b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:46.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:46.969+0000 7fa1df312700 1 --2- 172.21.15.134:0/1771824762 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1d8105720 0x7fa1d8105b00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa1d4009910 tx=0x7fa1d4009c20 comp rx=0 tx=0).stop 2024-04-22T23:15:46.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:46.969+0000 7fa1df312700 1 -- 172.21.15.134:0/1771824762 shutdown_connections 2024-04-22T23:15:46.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:46.969+0000 7fa1df312700 1 --2- 172.21.15.134:0/1771824762 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1d8105720 0x7fa1d8105b00 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:46.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:46.969+0000 7fa1df312700 1 -- 172.21.15.134:0/1771824762 >> 172.21.15.134:0/1771824762 conn(0x7fa1d80fb250 msgr2=0x7fa1d80fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:15:46.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:46.969+0000 7fa1df312700 1 -- 172.21.15.134:0/1771824762 shutdown_connections 2024-04-22T23:15:46.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:46.969+0000 7fa1df312700 1 -- 172.21.15.134:0/1771824762 wait complete. 2024-04-22T23:15:46.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:46.970+0000 7fa1df312700 1 Processor -- start 2024-04-22T23:15:46.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:46.970+0000 7fa1df312700 1 -- start start 2024-04-22T23:15:46.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:46.970+0000 7fa1df312700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1d8105720 0x7fa1d8195d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:15:46.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:46.970+0000 7fa1df312700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d8196270 con 0x7fa1d8105720 2024-04-22T23:15:46.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:46.970+0000 7fa1dd0ae700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1d8105720 0x7fa1d8195d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:15:46.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:46.970+0000 7fa1dd0ae700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1d8105720 0x7fa1d8195d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.134:47512/0 (socket says 172.21.15.134:47512) 2024-04-22T23:15:46.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:46.970+0000 7fa1dd0ae700 1 -- 172.21.15.134:0/3937462797 learned_addr learned my addr 172.21.15.134:0/3937462797 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-04-22T23:15:46.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:46.970+0000 7fa1dd0ae700 1 -- 172.21.15.134:0/3937462797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1d40095f0 con 0x7fa1d8105720 2024-04-22T23:15:46.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:46.971+0000 7fa1dd0ae700 1 --2- 172.21.15.134:0/3937462797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1d8105720 0x7fa1d8195d30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa1d4005b40 tx=0x7fa1d4004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:15:46.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:46.971+0000 7fa1ce7fc700 1 -- 172.21.15.134:0/3937462797 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1d401d070 con 0x7fa1d8105720 2024-04-22T23:15:46.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:46.971+0000 7fa1df312700 1 -- 172.21.15.134:0/3937462797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1d8196470 con 0x7fa1d8105720 2024-04-22T23:15:46.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:46.971+0000 7fa1ce7fc700 1 -- 172.21.15.134:0/3937462797 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa1d40044d0 con 0x7fa1d8105720 2024-04-22T23:15:46.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:46.971+0000 7fa1ce7fc700 1 -- 172.21.15.134:0/3937462797 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1d4003910 con 0x7fa1d8105720 2024-04-22T23:15:46.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:46.971+0000 7fa1df312700 1 -- 172.21.15.134:0/3937462797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1d81968f0 con 0x7fa1d8105720 2024-04-22T23:15:46.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:46.972+0000 7fa1df312700 1 -- 172.21.15.134:0/3937462797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1d80623c0 con 0x7fa1d8105720 2024-04-22T23:15:46.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:46.972+0000 7fa1ce7fc700 1 -- 172.21.15.134:0/3937462797 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 45396+0+0 (secure 0 0 0) 0x7fa1d400b960 con 0x7fa1d8105720 2024-04-22T23:15:46.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:46.972+0000 7fa1ce7fc700 1 --2- 172.21.15.134:0/3937462797 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa1c40384b0 0x7fa1c403a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:15:46.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:46.972+0000 7fa1ce7fc700 1 -- 172.21.15.134:0/3937462797 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fa1d404c5a0 con 0x7fa1d8105720 2024-04-22T23:15:46.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:46.972+0000 7fa1dc8ad700 1 --2- 172.21.15.134:0/3937462797 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa1c40384b0 0x7fa1c403a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:15:46.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:46.973+0000 7fa1dc8ad700 1 --2- 172.21.15.134:0/3937462797 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa1c40384b0 0x7fa1c403a970 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8006fd0 tx=0x7fa1c8006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:15:46.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:46.977+0000 7fa1ce7fc700 1 -- 172.21.15.134:0/3937462797 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa1d402a540 con 0x7fa1d8105720 2024-04-22T23:15:47.256 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:47.255+0000 7fa1df312700 1 -- 172.21.15.134:0/3937462797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa1d819a860 con 0x7fa1d8105720 2024-04-22T23:15:47.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:47.256+0000 7fa1ce7fc700 1 -- 172.21.15.134:0/3937462797 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fa1d402a360 con 0x7fa1d8105720 2024-04-22T23:15:47.257 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:15:47.258 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"d46c346e-00fd-11ef-bc93-c7b262605968","modified":"2024-04-22T23:13:15.759242Z","created":"2024-04-22T23:13:15.759242Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T23:15:47.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:47.258+0000 7fa1df312700 1 -- 172.21.15.134:0/3937462797 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa1c40384b0 msgr2=0x7fa1c403a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:47.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:47.258+0000 7fa1df312700 1 --2- 172.21.15.134:0/3937462797 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa1c40384b0 0x7fa1c403a970 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8006fd0 tx=0x7fa1c8006e40 comp rx=0 tx=0).stop 2024-04-22T23:15:47.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:47.258+0000 7fa1df312700 1 -- 172.21.15.134:0/3937462797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1d8105720 msgr2=0x7fa1d8195d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:47.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:47.258+0000 7fa1df312700 1 --2- 172.21.15.134:0/3937462797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1d8105720 0x7fa1d8195d30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa1d4005b40 tx=0x7fa1d4004c80 comp rx=0 tx=0).stop 2024-04-22T23:15:47.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:47.258+0000 7fa1df312700 1 -- 172.21.15.134:0/3937462797 shutdown_connections 2024-04-22T23:15:47.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:47.258+0000 7fa1df312700 1 --2- 172.21.15.134:0/3937462797 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa1c40384b0 0x7fa1c403a970 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:47.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:47.258+0000 7fa1df312700 1 --2- 172.21.15.134:0/3937462797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1d8105720 0x7fa1d8195d30 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:47.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:47.258+0000 7fa1df312700 1 -- 172.21.15.134:0/3937462797 >> 172.21.15.134:0/3937462797 conn(0x7fa1d80fb250 msgr2=0x7fa1d8107af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:15:47.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:47.258+0000 7fa1df312700 1 -- 172.21.15.134:0/3937462797 shutdown_connections 2024-04-22T23:15:47.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:47.258+0000 7fa1df312700 1 -- 172.21.15.134:0/3937462797 wait complete. 2024-04-22T23:15:47.261 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-04-22T23:15:47.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:47 smithi067 ceph-mon[26453]: from='client.? 172.21.15.134:0/3937462797' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T23:15:49.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:48 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:49.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:48 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:49.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:48 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:49.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:48 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:49.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:48 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi134", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T23:15:49.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:48 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi134", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-22T23:15:49.046 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:48 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:15:49.046 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:48 smithi067 ceph-mon[26453]: Deploying daemon crash.smithi134 on smithi134 2024-04-22T23:15:49.073 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T23:15:49.073 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T23:15:49.286 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T23:15:50.642 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.639+0000 7fde98eef700 1 -- 172.21.15.134:0/3834960475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde94105720 msgr2=0x7fde94105b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:50.642 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.639+0000 7fde98eef700 1 --2- 172.21.15.134:0/3834960475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde94105720 0x7fde94105b00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fde84009910 tx=0x7fde84009c20 comp rx=0 tx=0).stop 2024-04-22T23:15:50.642 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.639+0000 7fde98eef700 1 -- 172.21.15.134:0/3834960475 shutdown_connections 2024-04-22T23:15:50.642 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.639+0000 7fde98eef700 1 --2- 172.21.15.134:0/3834960475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde94105720 0x7fde94105b00 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:50.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.639+0000 7fde98eef700 1 -- 172.21.15.134:0/3834960475 >> 172.21.15.134:0/3834960475 conn(0x7fde940fb230 msgr2=0x7fde940fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:15:50.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.639+0000 7fde98eef700 1 -- 172.21.15.134:0/3834960475 shutdown_connections 2024-04-22T23:15:50.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.639+0000 7fde98eef700 1 -- 172.21.15.134:0/3834960475 wait complete. 2024-04-22T23:15:50.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.640+0000 7fde98eef700 1 Processor -- start 2024-04-22T23:15:50.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.640+0000 7fde98eef700 1 -- start start 2024-04-22T23:15:50.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.640+0000 7fde98eef700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde94105720 0x7fde940fb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:15:50.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.640+0000 7fde98eef700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde940fd660 con 0x7fde94105720 2024-04-22T23:15:50.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.640+0000 7fde9259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde94105720 0x7fde940fb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:15:50.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.640+0000 7fde9259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde94105720 0x7fde940fb6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.134:35358/0 (socket says 172.21.15.134:35358) 2024-04-22T23:15:50.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.640+0000 7fde9259c700 1 -- 172.21.15.134:0/585726373 learned_addr learned my addr 172.21.15.134:0/585726373 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-04-22T23:15:50.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.641+0000 7fde9259c700 1 -- 172.21.15.134:0/585726373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde840095f0 con 0x7fde94105720 2024-04-22T23:15:50.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.641+0000 7fde9259c700 1 --2- 172.21.15.134:0/585726373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde94105720 0x7fde940fb6e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fde84005b40 tx=0x7fde84005000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:15:50.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.641+0000 7fde837fe700 1 -- 172.21.15.134:0/585726373 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde8401d070 con 0x7fde94105720 2024-04-22T23:15:50.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.641+0000 7fde98eef700 1 -- 172.21.15.134:0/585726373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde940fbc20 con 0x7fde94105720 2024-04-22T23:15:50.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.641+0000 7fde837fe700 1 -- 172.21.15.134:0/585726373 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fde840044d0 con 0x7fde94105720 2024-04-22T23:15:50.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.641+0000 7fde837fe700 1 -- 172.21.15.134:0/585726373 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde84003a40 con 0x7fde94105720 2024-04-22T23:15:50.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.641+0000 7fde98eef700 1 -- 172.21.15.134:0/585726373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde940fc0a0 con 0x7fde94105720 2024-04-22T23:15:50.645 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.642+0000 7fde98eef700 1 -- 172.21.15.134:0/585726373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde940623c0 con 0x7fde94105720 2024-04-22T23:15:50.645 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.642+0000 7fde837fe700 1 -- 172.21.15.134:0/585726373 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 45396+0+0 (secure 0 0 0) 0x7fde8400b960 con 0x7fde94105720 2024-04-22T23:15:50.645 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.642+0000 7fde837fe700 1 --2- 172.21.15.134:0/585726373 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fde7c0384b0 0x7fde7c03a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:15:50.645 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.643+0000 7fde837fe700 1 -- 172.21.15.134:0/585726373 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fde8404c410 con 0x7fde94105720 2024-04-22T23:15:50.645 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.643+0000 7fde91d9b700 1 --2- 172.21.15.134:0/585726373 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fde7c0384b0 0x7fde7c03a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:15:50.645 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.643+0000 7fde91d9b700 1 --2- 172.21.15.134:0/585726373 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fde7c0384b0 0x7fde7c03a970 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fde88006fd0 tx=0x7fde88006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:15:50.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.648+0000 7fde837fe700 1 -- 172.21.15.134:0/585726373 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fde8402a540 con 0x7fde94105720 2024-04-22T23:15:50.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.947+0000 7fde98eef700 1 -- 172.21.15.134:0/585726373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fde940fc410 con 0x7fde94105720 2024-04-22T23:15:50.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.947+0000 7fde837fe700 1 -- 172.21.15.134:0/585726373 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fde8402a360 con 0x7fde94105720 2024-04-22T23:15:50.948 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:15:50.949 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"d46c346e-00fd-11ef-bc93-c7b262605968","modified":"2024-04-22T23:13:15.759242Z","created":"2024-04-22T23:13:15.759242Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T23:15:50.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.949+0000 7fde98eef700 1 -- 172.21.15.134:0/585726373 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fde7c0384b0 msgr2=0x7fde7c03a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:50.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.949+0000 7fde98eef700 1 --2- 172.21.15.134:0/585726373 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fde7c0384b0 0x7fde7c03a970 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fde88006fd0 tx=0x7fde88006e40 comp rx=0 tx=0).stop 2024-04-22T23:15:50.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.949+0000 7fde98eef700 1 -- 172.21.15.134:0/585726373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde94105720 msgr2=0x7fde940fb6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:50.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.949+0000 7fde98eef700 1 --2- 172.21.15.134:0/585726373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde94105720 0x7fde940fb6e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fde84005b40 tx=0x7fde84005000 comp rx=0 tx=0).stop 2024-04-22T23:15:50.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.949+0000 7fde98eef700 1 -- 172.21.15.134:0/585726373 shutdown_connections 2024-04-22T23:15:50.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.949+0000 7fde98eef700 1 --2- 172.21.15.134:0/585726373 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fde7c0384b0 0x7fde7c03a970 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:50.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.950+0000 7fde98eef700 1 --2- 172.21.15.134:0/585726373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde94105720 0x7fde940fb6e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:50.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.950+0000 7fde98eef700 1 -- 172.21.15.134:0/585726373 >> 172.21.15.134:0/585726373 conn(0x7fde940fb230 msgr2=0x7fde94068880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:15:50.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.950+0000 7fde98eef700 1 -- 172.21.15.134:0/585726373 shutdown_connections 2024-04-22T23:15:50.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:50.950+0000 7fde98eef700 1 -- 172.21.15.134:0/585726373 wait complete. 2024-04-22T23:15:50.951 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-04-22T23:15:51.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:50 smithi067 ceph-mon[26453]: from='client.? 172.21.15.134:0/585726373' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T23:15:52.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:52 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:52.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:52 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:52.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:52 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:52.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:52 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:52.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:52 smithi067 ceph-mon[26453]: Deploying daemon node-exporter.smithi134 on smithi134 2024-04-22T23:15:52.596 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T23:15:52.596 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T23:15:52.806 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T23:15:53.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:53 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:53.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:53 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:15:55.507 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.502+0000 7f3a95901700 1 -- 172.21.15.134:0/3605079516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a90105780 msgr2=0x7f3a90105b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:55.507 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.502+0000 7f3a95901700 1 --2- 172.21.15.134:0/3605079516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a90105780 0x7f3a90105b60 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3a84009910 tx=0x7f3a84009c20 comp rx=0 tx=0).stop 2024-04-22T23:15:55.507 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.502+0000 7f3a95901700 1 -- 172.21.15.134:0/3605079516 shutdown_connections 2024-04-22T23:15:55.507 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.502+0000 7f3a95901700 1 --2- 172.21.15.134:0/3605079516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a90105780 0x7f3a90105b60 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:55.507 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.502+0000 7f3a95901700 1 -- 172.21.15.134:0/3605079516 >> 172.21.15.134:0/3605079516 conn(0x7f3a90075290 msgr2=0x7f3a900756a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:15:55.507 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.502+0000 7f3a95901700 1 -- 172.21.15.134:0/3605079516 shutdown_connections 2024-04-22T23:15:55.507 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.502+0000 7f3a95901700 1 -- 172.21.15.134:0/3605079516 wait complete. 2024-04-22T23:15:55.507 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.503+0000 7f3a95901700 1 Processor -- start 2024-04-22T23:15:55.507 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.503+0000 7f3a95901700 1 -- start start 2024-04-22T23:15:55.507 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.503+0000 7f3a95901700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a90105780 0x7f3a90195d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:15:55.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.503+0000 7f3a95901700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a901962d0 con 0x7f3a90105780 2024-04-22T23:15:55.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.504+0000 7f3a8effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a90105780 0x7f3a90195d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:15:55.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.504+0000 7f3a8effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a90105780 0x7f3a90195d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.134:35376/0 (socket says 172.21.15.134:35376) 2024-04-22T23:15:55.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.504+0000 7f3a8effd700 1 -- 172.21.15.134:0/4207089009 learned_addr learned my addr 172.21.15.134:0/4207089009 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-04-22T23:15:55.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.504+0000 7f3a8effd700 1 -- 172.21.15.134:0/4207089009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a840095f0 con 0x7f3a90105780 2024-04-22T23:15:55.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.504+0000 7f3a8effd700 1 --2- 172.21.15.134:0/4207089009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a90105780 0x7f3a90195d90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3a84005b40 tx=0x7f3a84004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:15:55.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.504+0000 7f3a948ff700 1 -- 172.21.15.134:0/4207089009 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a8401d070 con 0x7f3a90105780 2024-04-22T23:15:55.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.504+0000 7f3a95901700 1 -- 172.21.15.134:0/4207089009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a901964d0 con 0x7f3a90105780 2024-04-22T23:15:55.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.504+0000 7f3a948ff700 1 -- 172.21.15.134:0/4207089009 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3a840044d0 con 0x7f3a90105780 2024-04-22T23:15:55.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.504+0000 7f3a948ff700 1 -- 172.21.15.134:0/4207089009 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a84003930 con 0x7f3a90105780 2024-04-22T23:15:55.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.504+0000 7f3a95901700 1 -- 172.21.15.134:0/4207089009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a90196890 con 0x7f3a90105780 2024-04-22T23:15:55.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.505+0000 7f3a95901700 1 -- 172.21.15.134:0/4207089009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a900623c0 con 0x7f3a90105780 2024-04-22T23:15:55.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.505+0000 7f3a948ff700 1 -- 172.21.15.134:0/4207089009 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 45396+0+0 (secure 0 0 0) 0x7f3a84003bf0 con 0x7f3a90105780 2024-04-22T23:15:55.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.506+0000 7f3a948ff700 1 --2- 172.21.15.134:0/4207089009 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f3a780384b0 0x7f3a7803a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:15:55.509 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.506+0000 7f3a948ff700 1 -- 172.21.15.134:0/4207089009 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f3a8404c460 con 0x7f3a90105780 2024-04-22T23:15:55.509 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.506+0000 7f3a8e7fc700 1 --2- 172.21.15.134:0/4207089009 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f3a780384b0 0x7f3a7803a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:15:55.509 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.506+0000 7f3a8e7fc700 1 --2- 172.21.15.134:0/4207089009 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f3a780384b0 0x7f3a7803a970 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3a80006fd0 tx=0x7f3a80006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:15:55.512 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.511+0000 7f3a948ff700 1 -- 172.21.15.134:0/4207089009 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f3a8402a540 con 0x7f3a90105780 2024-04-22T23:15:55.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.794+0000 7f3a95901700 1 -- 172.21.15.134:0/4207089009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3a9019a880 con 0x7f3a90105780 2024-04-22T23:15:55.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.795+0000 7f3a948ff700 1 -- 172.21.15.134:0/4207089009 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f3a8402a360 con 0x7f3a90105780 2024-04-22T23:15:55.796 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:15:55.796 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"d46c346e-00fd-11ef-bc93-c7b262605968","modified":"2024-04-22T23:13:15.759242Z","created":"2024-04-22T23:13:15.759242Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T23:15:55.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.797+0000 7f3a95901700 1 -- 172.21.15.134:0/4207089009 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f3a780384b0 msgr2=0x7f3a7803a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:55.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.797+0000 7f3a95901700 1 --2- 172.21.15.134:0/4207089009 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f3a780384b0 0x7f3a7803a970 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3a80006fd0 tx=0x7f3a80006e40 comp rx=0 tx=0).stop 2024-04-22T23:15:55.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.797+0000 7f3a95901700 1 -- 172.21.15.134:0/4207089009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a90105780 msgr2=0x7f3a90195d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:15:55.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.797+0000 7f3a95901700 1 --2- 172.21.15.134:0/4207089009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a90105780 0x7f3a90195d90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3a84005b40 tx=0x7f3a84004c80 comp rx=0 tx=0).stop 2024-04-22T23:15:55.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.797+0000 7f3a95901700 1 -- 172.21.15.134:0/4207089009 shutdown_connections 2024-04-22T23:15:55.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.797+0000 7f3a95901700 1 --2- 172.21.15.134:0/4207089009 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f3a780384b0 0x7f3a7803a970 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:55.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.797+0000 7f3a95901700 1 --2- 172.21.15.134:0/4207089009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a90105780 0x7f3a90195d90 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:15:55.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.797+0000 7f3a95901700 1 -- 172.21.15.134:0/4207089009 >> 172.21.15.134:0/4207089009 conn(0x7f3a90075290 msgr2=0x7f3a90067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:15:55.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.798+0000 7f3a95901700 1 -- 172.21.15.134:0/4207089009 shutdown_connections 2024-04-22T23:15:55.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:15:55.798+0000 7f3a95901700 1 -- 172.21.15.134:0/4207089009 wait complete. 2024-04-22T23:15:55.799 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-04-22T23:15:56.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:56 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:56.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:56 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:56.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:56 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:56.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:56 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:56.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:56 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi134.ccxkcj", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T23:15:56.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:56 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi134.ccxkcj", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T23:15:56.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:56 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T23:15:56.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:56 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:15:56.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:56 smithi067 ceph-mon[26453]: Deploying daemon mgr.smithi134.ccxkcj on smithi134 2024-04-22T23:15:56.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:56 smithi067 ceph-mon[26453]: from='client.? 172.21.15.134:0/4207089009' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T23:15:57.395 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T23:15:57.395 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T23:15:58.139 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T23:15:58.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:58 smithi067 ceph-mon[26453]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:15:58.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:58 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:58.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:58 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:58.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:58 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:58.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:58 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:58.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:58 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:15:58.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:58 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T23:15:58.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:58 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:15:59.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:15:59 smithi067 ceph-mon[26453]: Deploying daemon mon.smithi134 on smithi134 2024-04-22T23:16:00.098 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.097+0000 7fa65f51c700 1 -- 172.21.15.134:0/2964187908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa658106820 msgr2=0x7fa658106c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:16:00.099 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.097+0000 7fa65f51c700 1 --2- 172.21.15.134:0/2964187908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa658106820 0x7fa658106c00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa648009910 tx=0x7fa648009c20 comp rx=0 tx=0).stop 2024-04-22T23:16:00.099 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.097+0000 7fa65f51c700 1 -- 172.21.15.134:0/2964187908 shutdown_connections 2024-04-22T23:16:00.099 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.097+0000 7fa65f51c700 1 --2- 172.21.15.134:0/2964187908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa658106820 0x7fa658106c00 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:16:00.099 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.097+0000 7fa65f51c700 1 -- 172.21.15.134:0/2964187908 >> 172.21.15.134:0/2964187908 conn(0x7fa658075290 msgr2=0x7fa6580756a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:16:00.099 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.098+0000 7fa65f51c700 1 -- 172.21.15.134:0/2964187908 shutdown_connections 2024-04-22T23:16:00.099 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.098+0000 7fa65f51c700 1 -- 172.21.15.134:0/2964187908 wait complete. 2024-04-22T23:16:00.100 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.098+0000 7fa65f51c700 1 Processor -- start 2024-04-22T23:16:00.100 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.098+0000 7fa65f51c700 1 -- start start 2024-04-22T23:16:00.101 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.098+0000 7fa65f51c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa658106820 0x7fa65819bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:16:00.101 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.098+0000 7fa65f51c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa65819c060 con 0x7fa658106820 2024-04-22T23:16:00.101 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.099+0000 7fa65d2b8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa658106820 0x7fa65819bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:16:00.101 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.099+0000 7fa65d2b8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa658106820 0x7fa65819bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.134:43668/0 (socket says 172.21.15.134:43668) 2024-04-22T23:16:00.101 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.099+0000 7fa65d2b8700 1 -- 172.21.15.134:0/1487104516 learned_addr learned my addr 172.21.15.134:0/1487104516 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-04-22T23:16:00.101 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.099+0000 7fa65d2b8700 1 -- 172.21.15.134:0/1487104516 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6480095f0 con 0x7fa658106820 2024-04-22T23:16:00.101 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.099+0000 7fa65d2b8700 1 --2- 172.21.15.134:0/1487104516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa658106820 0x7fa65819bb20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa648005b40 tx=0x7fa648004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:16:00.101 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.099+0000 7fa64e7fc700 1 -- 172.21.15.134:0/1487104516 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa64801d070 con 0x7fa658106820 2024-04-22T23:16:00.101 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.099+0000 7fa64e7fc700 1 -- 172.21.15.134:0/1487104516 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa6480044d0 con 0x7fa658106820 2024-04-22T23:16:00.101 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.099+0000 7fa65f51c700 1 -- 172.21.15.134:0/1487104516 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa65819c260 con 0x7fa658106820 2024-04-22T23:16:00.101 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.099+0000 7fa64e7fc700 1 -- 172.21.15.134:0/1487104516 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa648003910 con 0x7fa658106820 2024-04-22T23:16:00.101 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.100+0000 7fa65f51c700 1 -- 172.21.15.134:0/1487104516 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa658195b10 con 0x7fa658106820 2024-04-22T23:16:00.102 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.100+0000 7fa65f51c700 1 -- 172.21.15.134:0/1487104516 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6580623c0 con 0x7fa658106820 2024-04-22T23:16:00.102 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.100+0000 7fa64e7fc700 1 -- 172.21.15.134:0/1487104516 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 45396+0+0 (secure 0 0 0) 0x7fa64800b960 con 0x7fa658106820 2024-04-22T23:16:00.102 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.101+0000 7fa64e7fc700 1 --2- 172.21.15.134:0/1487104516 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa644038500 0x7fa64403a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:16:00.102 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.101+0000 7fa64e7fc700 1 -- 172.21.15.134:0/1487104516 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fa64804c530 con 0x7fa658106820 2024-04-22T23:16:00.103 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.101+0000 7fa65cab7700 1 --2- 172.21.15.134:0/1487104516 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa644038500 0x7fa64403a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:16:00.103 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.101+0000 7fa65cab7700 1 --2- 172.21.15.134:0/1487104516 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa644038500 0x7fa64403a9c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa654006fd0 tx=0x7fa654006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:16:00.107 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.106+0000 7fa64e7fc700 1 -- 172.21.15.134:0/1487104516 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa64802a540 con 0x7fa658106820 2024-04-22T23:16:00.383 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.382+0000 7fa65f51c700 1 -- 172.21.15.134:0/1487104516 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa6581965a0 con 0x7fa658106820 2024-04-22T23:16:00.383 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.382+0000 7fa64e7fc700 1 -- 172.21.15.134:0/1487104516 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fa64802a360 con 0x7fa658106820 2024-04-22T23:16:00.383 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:16:00.383 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":1,"fsid":"d46c346e-00fd-11ef-bc93-c7b262605968","modified":"2024-04-22T23:13:15.759242Z","created":"2024-04-22T23:13:15.759242Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T23:16:00.385 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.384+0000 7fa65f51c700 1 -- 172.21.15.134:0/1487104516 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa644038500 msgr2=0x7fa64403a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:16:00.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.384+0000 7fa65f51c700 1 --2- 172.21.15.134:0/1487104516 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa644038500 0x7fa64403a9c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa654006fd0 tx=0x7fa654006e40 comp rx=0 tx=0).stop 2024-04-22T23:16:00.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.384+0000 7fa65f51c700 1 -- 172.21.15.134:0/1487104516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa658106820 msgr2=0x7fa65819bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:16:00.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.384+0000 7fa65f51c700 1 --2- 172.21.15.134:0/1487104516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa658106820 0x7fa65819bb20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa648005b40 tx=0x7fa648004c80 comp rx=0 tx=0).stop 2024-04-22T23:16:00.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.384+0000 7fa65f51c700 1 -- 172.21.15.134:0/1487104516 shutdown_connections 2024-04-22T23:16:00.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.384+0000 7fa65f51c700 1 --2- 172.21.15.134:0/1487104516 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa644038500 0x7fa64403a9c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:16:00.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.384+0000 7fa65f51c700 1 --2- 172.21.15.134:0/1487104516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa658106820 0x7fa65819bb20 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:16:00.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.384+0000 7fa65f51c700 1 -- 172.21.15.134:0/1487104516 >> 172.21.15.134:0/1487104516 conn(0x7fa658075290 msgr2=0x7fa6580fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:16:00.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.385+0000 7fa65f51c700 1 -- 172.21.15.134:0/1487104516 shutdown_connections 2024-04-22T23:16:00.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:00.385+0000 7fa65f51c700 1 -- 172.21.15.134:0/1487104516 wait complete. 2024-04-22T23:16:00.386 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 1 2024-04-22T23:16:00.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:00 smithi067 ceph-mon[26453]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:00.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:00 smithi067 ceph-mon[26453]: from='client.? 172.21.15.134:0/1487104516' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T23:16:02.200 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-22T23:16:02.200 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T23:16:02.451 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-22T23:16:02.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:02 smithi067 ceph-mon[26453]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:03.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:03 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.254 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:04.252+0000 7fe60a180700 1 -- 172.21.15.134:0/1593540573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe604106930 msgr2=0x7fe604106d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:16:04.255 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:04.252+0000 7fe60a180700 1 --2- 172.21.15.134:0/1593540573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe604106930 0x7fe604106d10 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe5f4009910 tx=0x7fe5f4009c20 comp rx=0 tx=0).stop 2024-04-22T23:16:04.255 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:04.252+0000 7fe60a180700 1 -- 172.21.15.134:0/1593540573 shutdown_connections 2024-04-22T23:16:04.255 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:04.252+0000 7fe60a180700 1 --2- 172.21.15.134:0/1593540573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe604106930 0x7fe604106d10 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:16:04.255 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:04.252+0000 7fe60a180700 1 -- 172.21.15.134:0/1593540573 >> 172.21.15.134:0/1593540573 conn(0x7fe6040fc400 msgr2=0x7fe6040fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:16:04.255 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:04.252+0000 7fe60a180700 1 -- 172.21.15.134:0/1593540573 shutdown_connections 2024-04-22T23:16:04.255 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:04.252+0000 7fe60a180700 1 -- 172.21.15.134:0/1593540573 wait complete. 2024-04-22T23:16:04.255 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:04.253+0000 7fe60a180700 1 Processor -- start 2024-04-22T23:16:04.255 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:04.253+0000 7fe60a180700 1 -- start start 2024-04-22T23:16:04.255 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:04.253+0000 7fe60a180700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe604106930 0x7fe604195e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:16:04.255 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:04.253+0000 7fe60a180700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6041963a0 con 0x7fe604106930 2024-04-22T23:16:04.255 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:04.253+0000 7fe6037fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe604106930 0x7fe604195e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:16:04.255 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:04.253+0000 7fe6037fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe604106930 0x7fe604195e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.134:43698/0 (socket says 172.21.15.134:43698) 2024-04-22T23:16:04.255 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:04.253+0000 7fe6037fe700 1 -- 172.21.15.134:0/3251394088 learned_addr learned my addr 172.21.15.134:0/3251394088 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-04-22T23:16:04.256 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:04.254+0000 7fe6037fe700 1 -- 172.21.15.134:0/3251394088 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f40095f0 con 0x7fe604106930 2024-04-22T23:16:04.256 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:04.254+0000 7fe6037fe700 1 --2- 172.21.15.134:0/3251394088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe604106930 0x7fe604195e60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe5f4005b40 tx=0x7fe5f4004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:16:04.256 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:04.254+0000 7fe600ff9700 1 -- 172.21.15.134:0/3251394088 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5f401d070 con 0x7fe604106930 2024-04-22T23:16:04.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:04.254+0000 7fe60a180700 1 -- 172.21.15.134:0/3251394088 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6041965a0 con 0x7fe604106930 2024-04-22T23:16:04.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:04.254+0000 7fe600ff9700 1 -- 172.21.15.134:0/3251394088 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe5f40044d0 con 0x7fe604106930 2024-04-22T23:16:04.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:04.254+0000 7fe600ff9700 1 -- 172.21.15.134:0/3251394088 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5f4003910 con 0x7fe604106930 2024-04-22T23:16:04.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:04.254+0000 7fe60a180700 1 -- 172.21.15.134:0/3251394088 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe604196a20 con 0x7fe604106930 2024-04-22T23:16:04.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:04.255+0000 7fe60a180700 1 -- 172.21.15.134:0/3251394088 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6040623c0 con 0x7fe604106930 2024-04-22T23:16:04.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:04.255+0000 7fe600ff9700 1 -- 172.21.15.134:0/3251394088 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) v1 ==== 45396+0+0 (secure 0 0 0) 0x7fe5f4003b90 con 0x7fe604106930 2024-04-22T23:16:04.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:04.255+0000 7fe600ff9700 1 --2- 172.21.15.134:0/3251394088 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fe5ec0384b0 0x7fe5ec03a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:16:04.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:04.255+0000 7fe600ff9700 1 -- 172.21.15.134:0/3251394088 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fe5f404c4c0 con 0x7fe604106930 2024-04-22T23:16:04.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:04.256+0000 7fe602ffd700 1 --2- 172.21.15.134:0/3251394088 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fe5ec0384b0 0x7fe5ec03a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:16:04.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:04.256+0000 7fe602ffd700 1 --2- 172.21.15.134:0/3251394088 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fe5ec0384b0 0x7fe5ec03a970 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe5f8006fd0 tx=0x7fe5f8006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:16:04.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:04.260+0000 7fe600ff9700 1 -- 172.21.15.134:0/3251394088 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fe5f402a540 con 0x7fe604106930 2024-04-22T23:16:04.400 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: load: jerasure load: lrc 2024-04-22T23:16:04.400 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: RocksDB version: 7.9.2 2024-04-22T23:16:04.400 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Git sha 0 2024-04-22T23:16:04.400 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Compile date 2023-12-11 22:07:34 2024-04-22T23:16:04.400 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: DB SUMMARY 2024-04-22T23:16:04.400 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: DB Session ID: SLK4T9YYLUVD3JFXYQB6 2024-04-22T23:16:04.400 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: CURRENT file: CURRENT 2024-04-22T23:16:04.400 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: IDENTITY file: IDENTITY 2024-04-22T23:16:04.401 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-22T23:16:04.401 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi134/store.db dir, Total Num: 0, files: 2024-04-22T23:16:04.401 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi134/store.db: 000004.log size: 511 ; 2024-04-22T23:16:04.401 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.error_if_exists: 0 2024-04-22T23:16:04.401 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.create_if_missing: 0 2024-04-22T23:16:04.401 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.paranoid_checks: 1 2024-04-22T23:16:04.401 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-22T23:16:04.401 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-22T23:16:04.401 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-22T23:16:04.401 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.env: 0x55e2d890f720 2024-04-22T23:16:04.401 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.fs: PosixFileSystem 2024-04-22T23:16:04.401 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.info_log: 0x55e2da4c72e0 2024-04-22T23:16:04.401 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.max_file_opening_threads: 16 2024-04-22T23:16:04.401 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.statistics: (nil) 2024-04-22T23:16:04.401 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.use_fsync: 0 2024-04-22T23:16:04.402 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.max_log_file_size: 0 2024-04-22T23:16:04.402 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-22T23:16:04.402 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-22T23:16:04.402 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.keep_log_file_num: 1000 2024-04-22T23:16:04.402 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.recycle_log_file_num: 0 2024-04-22T23:16:04.402 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.allow_fallocate: 1 2024-04-22T23:16:04.402 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.allow_mmap_reads: 0 2024-04-22T23:16:04.402 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.allow_mmap_writes: 0 2024-04-22T23:16:04.402 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.use_direct_reads: 0 2024-04-22T23:16:04.402 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-22T23:16:04.402 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.create_missing_column_families: 0 2024-04-22T23:16:04.402 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.db_log_dir: 2024-04-22T23:16:04.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:04.430+0000 7fe600ff9700 1 -- 172.21.15.134:0/3251394088 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5f402a540 con 0x7fe604106930 2024-04-22T23:16:04.566 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:04.565+0000 7fe60a180700 1 -- 172.21.15.134:0/3251394088 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe6041047c0 con 0x7fe604106930 2024-04-22T23:16:04.678 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.wal_dir: 2024-04-22T23:16:04.678 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-22T23:16:04.678 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-22T23:16:04.678 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-22T23:16:04.679 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-22T23:16:04.679 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-22T23:16:04.679 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-22T23:16:04.679 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.advise_random_on_open: 1 2024-04-22T23:16:04.679 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.db_write_buffer_size: 0 2024-04-22T23:16:04.679 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.write_buffer_manager: 0x55e2d9738140 2024-04-22T23:16:04.679 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-22T23:16:04.679 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-22T23:16:04.679 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-22T23:16:04.679 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.rate_limiter: (nil) 2024-04-22T23:16:04.679 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-22T23:16:04.679 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.wal_recovery_mode: 2 2024-04-22T23:16:04.679 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.enable_thread_tracking: 0 2024-04-22T23:16:04.679 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.enable_pipelined_write: 0 2024-04-22T23:16:04.680 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.unordered_write: 0 2024-04-22T23:16:04.680 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-22T23:16:04.680 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-22T23:16:04.680 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-22T23:16:04.680 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-22T23:16:04.680 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.row_cache: None 2024-04-22T23:16:04.680 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.wal_filter: None 2024-04-22T23:16:04.680 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-22T23:16:04.680 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.allow_ingest_behind: 0 2024-04-22T23:16:04.680 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.two_write_queues: 0 2024-04-22T23:16:04.680 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.manual_wal_flush: 0 2024-04-22T23:16:04.680 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.wal_compression: 0 2024-04-22T23:16:04.680 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.atomic_flush: 0 2024-04-22T23:16:04.680 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-22T23:16:04.680 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-22T23:16:04.680 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-22T23:16:04.681 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.log_readahead_size: 0 2024-04-22T23:16:04.681 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-22T23:16:04.681 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.best_efforts_recovery: 0 2024-04-22T23:16:04.681 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-22T23:16:04.681 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-22T23:16:04.683 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.allow_data_in_errors: 0 2024-04-22T23:16:04.683 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.db_host_id: __hostname__ 2024-04-22T23:16:04.683 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-22T23:16:04.683 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.max_background_jobs: 2 2024-04-22T23:16:04.683 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.max_background_compactions: -1 2024-04-22T23:16:04.683 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.max_subcompactions: 1 2024-04-22T23:16:04.683 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-22T23:16:04.684 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-22T23:16:04.684 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-22T23:16:04.684 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.max_total_wal_size: 0 2024-04-22T23:16:04.684 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-22T23:16:04.684 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-22T23:16:04.684 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-22T23:16:04.684 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-22T23:16:04.684 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.max_open_files: -1 2024-04-22T23:16:04.684 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.bytes_per_sync: 0 2024-04-22T23:16:04.684 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-22T23:16:04.684 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-22T23:16:04.684 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.compaction_readahead_size: 0 2024-04-22T23:16:04.684 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.max_background_flushes: -1 2024-04-22T23:16:04.684 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Compression algorithms supported: 2024-04-22T23:16:04.684 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-22T23:16:04.685 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: kZSTD supported: 0 2024-04-22T23:16:04.685 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: kXpressCompression supported: 0 2024-04-22T23:16:04.685 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: kLZ4HCCompression supported: 1 2024-04-22T23:16:04.685 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: kZlibCompression supported: 1 2024-04-22T23:16:04.685 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: kSnappyCompression supported: 1 2024-04-22T23:16:04.685 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: kLZ4Compression supported: 1 2024-04-22T23:16:04.685 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: kBZip2Compression supported: 0 2024-04-22T23:16:04.685 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-22T23:16:04.685 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-22T23:16:04.685 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi134/store.db/MANIFEST-000005 2024-04-22T23:16:04.685 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-22T23:16:04.685 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-22T23:16:04.685 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.merge_operator: 2024-04-22T23:16:04.685 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.compaction_filter: None 2024-04-22T23:16:04.686 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.compaction_filter_factory: None 2024-04-22T23:16:04.686 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.sst_partitioner_factory: None 2024-04-22T23:16:04.686 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-22T23:16:04.686 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-22T23:16:04.686 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e2da4c73e0) 2024-04-22T23:16:04.686 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: cache_index_and_filter_blocks: 1 2024-04-22T23:16:04.686 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-22T23:16:04.686 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-22T23:16:04.686 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: pin_top_level_index_and_filter: 1 2024-04-22T23:16:04.686 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: index_type: 0 2024-04-22T23:16:04.686 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: data_block_index_type: 0 2024-04-22T23:16:04.686 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: index_shortening: 1 2024-04-22T23:16:04.686 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-22T23:16:04.686 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: checksum: 4 2024-04-22T23:16:04.686 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: no_block_cache: 0 2024-04-22T23:16:04.687 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: block_cache: 0x55e2d97d91f0 2024-04-22T23:16:04.687 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: block_cache_name: BinnedLRUCache 2024-04-22T23:16:04.687 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: block_cache_options: 2024-04-22T23:16:04.687 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: capacity : 536870912 2024-04-22T23:16:04.687 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: num_shard_bits : 4 2024-04-22T23:16:04.687 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: strict_capacity_limit : 0 2024-04-22T23:16:04.687 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: high_pri_pool_ratio: 0.000 2024-04-22T23:16:04.687 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: block_cache_compressed: (nil) 2024-04-22T23:16:04.687 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: persistent_cache: (nil) 2024-04-22T23:16:04.687 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: block_size: 4096 2024-04-22T23:16:04.687 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: block_size_deviation: 10 2024-04-22T23:16:04.687 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: block_restart_interval: 16 2024-04-22T23:16:04.687 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: index_block_restart_interval: 1 2024-04-22T23:16:04.687 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: metadata_block_size: 4096 2024-04-22T23:16:04.687 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: partition_filters: 0 2024-04-22T23:16:04.687 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: use_delta_encoding: 1 2024-04-22T23:16:04.688 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: filter_policy: bloomfilter 2024-04-22T23:16:04.688 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: whole_key_filtering: 1 2024-04-22T23:16:04.688 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: verify_compression: 0 2024-04-22T23:16:04.688 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: read_amp_bytes_per_bit: 0 2024-04-22T23:16:04.688 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: format_version: 5 2024-04-22T23:16:04.688 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: enable_index_compression: 1 2024-04-22T23:16:04.688 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: block_align: 0 2024-04-22T23:16:04.688 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: max_auto_readahead_size: 262144 2024-04-22T23:16:04.688 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: prepopulate_block_cache: 0 2024-04-22T23:16:04.688 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: initial_auto_readahead_size: 8192 2024-04-22T23:16:04.688 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: num_file_reads_for_auto_readahead: 2 2024-04-22T23:16:04.689 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.write_buffer_size: 33554432 2024-04-22T23:16:04.689 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.max_write_buffer_number: 2 2024-04-22T23:16:04.689 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.compression: NoCompression 2024-04-22T23:16:04.689 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.bottommost_compression: Disabled 2024-04-22T23:16:04.689 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.prefix_extractor: nullptr 2024-04-22T23:16:04.689 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-22T23:16:04.689 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.num_levels: 7 2024-04-22T23:16:04.689 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-22T23:16:04.689 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-22T23:16:04.689 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-22T23:16:04.689 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-22T23:16:04.690 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-22T23:16:04.690 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-22T23:16:04.690 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-22T23:16:04.690 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-22T23:16:04.690 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-22T23:16:04.690 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-22T23:16:04.690 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-22T23:16:04.690 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-22T23:16:04.690 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-22T23:16:04.690 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.compression_opts.level: 32767 2024-04-22T23:16:04.691 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.compression_opts.strategy: 0 2024-04-22T23:16:04.691 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-22T23:16:04.691 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-22T23:16:04.691 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-22T23:16:04.691 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-22T23:16:04.691 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.compression_opts.enabled: false 2024-04-22T23:16:04.691 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-22T23:16:04.691 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-22T23:16:04.691 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-22T23:16:04.691 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-22T23:16:04.691 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.target_file_size_base: 67108864 2024-04-22T23:16:04.691 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-22T23:16:04.691 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-22T23:16:04.691 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-22T23:16:04.691 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-22T23:16:04.692 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-22T23:16:04.692 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-22T23:16:04.692 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-22T23:16:04.692 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-22T23:16:04.692 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-22T23:16:04.692 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-22T23:16:04.692 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-22T23:16:04.692 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-22T23:16:04.692 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-22T23:16:04.692 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-22T23:16:04.692 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.arena_block_size: 1048576 2024-04-22T23:16:04.692 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-22T23:16:04.692 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-22T23:16:04.692 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.disable_auto_compactions: 0 2024-04-22T23:16:04.692 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-22T23:16:04.693 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-22T23:16:04.693 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-22T23:16:04.693 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-22T23:16:04.693 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-22T23:16:04.693 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-22T23:16:04.693 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-22T23:16:04.693 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-22T23:16:04.693 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-22T23:16:04.693 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-22T23:16:04.693 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.table_properties_collectors: 2024-04-22T23:16:04.693 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.inplace_update_support: 0 2024-04-22T23:16:04.693 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-22T23:16:04.693 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-22T23:16:04.693 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-22T23:16:04.694 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-22T23:16:04.694 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.bloom_locality: 0 2024-04-22T23:16:04.694 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.max_successive_merges: 0 2024-04-22T23:16:04.694 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-22T23:16:04.694 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.paranoid_file_checks: 0 2024-04-22T23:16:04.694 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.force_consistency_checks: 1 2024-04-22T23:16:04.694 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.report_bg_io_stats: 0 2024-04-22T23:16:04.694 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.ttl: 2592000 2024-04-22T23:16:04.694 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-22T23:16:04.694 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-22T23:16:04.694 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-22T23:16:04.694 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.enable_blob_files: false 2024-04-22T23:16:04.694 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.min_blob_size: 0 2024-04-22T23:16:04.695 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.blob_file_size: 268435456 2024-04-22T23:16:04.695 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-22T23:16:04.695 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-22T23:16:04.695 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-22T23:16:04.695 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-22T23:16:04.695 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-22T23:16:04.695 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.blob_file_starting_level: 0 2024-04-22T23:16:04.695 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-22T23:16:04.695 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi134/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-22T23:16:04.695 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-22T23:16:04.695 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 859ad799-3c2f-4498-9dd5-2d736f84a4e9 2024-04-22T23:16:04.695 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713827764400214, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-22T23:16:04.696 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-22T23:16:04.696 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713827764400676, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713827764, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "859ad799-3c2f-4498-9dd5-2d736f84a4e9", "db_session_id": "SLK4T9YYLUVD3JFXYQB6", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-22T23:16:04.696 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713827764400761, "job": 1, "event": "recovery_finished"} 2024-04-22T23:16:04.696 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-22T23:16:04.696 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi134/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-22T23:16:04.696 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55e2d9876000 2024-04-22T23:16:04.696 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: DB pointer 0x55e2d9862000 2024-04-22T23:16:04.696 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-22T23:16:04.696 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-22T23:16:04.696 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: ** DB Stats ** 2024-04-22T23:16:04.696 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T23:16:04.696 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-22T23:16:04.696 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T23:16:04.696 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T23:16:04.696 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-22T23:16:04.697 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T23:16:04.697 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T23:16:04.697 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: 2024-04-22T23:16:04.698 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: ** Compaction Stats [default] ** 2024-04-22T23:16:04.698 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T23:16:04.698 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-22T23:16:04.698 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-22T23:16:04.698 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-22T23:16:04.698 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-22T23:16:04.698 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: 2024-04-22T23:16:04.698 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: ** Compaction Stats [default] ** 2024-04-22T23:16:04.698 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T23:16:04.698 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T23:16:04.698 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-22T23:16:04.698 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: 2024-04-22T23:16:04.698 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-22T23:16:04.699 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: 2024-04-22T23:16:04.699 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T23:16:04.699 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T23:16:04.699 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T23:16:04.699 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T23:16:04.699 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T23:16:04.699 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T23:16:04.699 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Cumulative compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T23:16:04.699 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Interval compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T23:16:04.699 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T23:16:04.699 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Block cache BinnedLRUCache@0x55e2d97d91f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-04-22T23:16:04.699 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-22T23:16:04.699 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: 2024-04-22T23:16:04.699 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T23:16:04.699 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: mon.smithi134 does not exist in monmap, will attempt to join an existing cluster 2024-04-22T23:16:04.700 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: using public_addr v2:172.21.15.134:0/0 -> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-04-22T23:16:04.700 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: starting mon.smithi134 rank -1 at public addrs [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] at bind addrs [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi134 fsid d46c346e-00fd-11ef-bc93-c7b262605968 2024-04-22T23:16:04.700 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: mon.smithi134@-1(???) e0 preinit fsid d46c346e-00fd-11ef-bc93-c7b262605968 2024-04-22T23:16:04.700 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: mon.smithi134@-1(synchronizing).mds e1 new map 2024-04-22T23:16:04.700 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: mon.smithi134@-1(synchronizing).mds e1 print_map 2024-04-22T23:16:04.700 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: e1 2024-04-22T23:16:04.700 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:16:04.700 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:16:04.700 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: legacy client fscid: -1 2024-04-22T23:16:04.700 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: 2024-04-22T23:16:04.700 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: No filesystems configured 2024-04-22T23:16:04.700 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: mon.smithi134@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-22T23:16:04.700 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: mon.smithi134@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-22T23:16:04.700 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: mon.smithi134@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-22T23:16:04.700 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: mon.smithi134@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-22T23:16:04.701 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: mon.smithi134@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-22T23:16:04.701 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: mon.smithi134@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-22T23:16:04.701 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: mon.smithi134@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-04-22T23:16:04.701 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: mon.smithi134@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-22T23:16:04.701 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: mon.smithi134@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T23:16:04.701 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: mon.smithi134@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T23:16:04.701 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: mon.smithi134@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T23:16:04.701 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='client.14191 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi134", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:16:04.701 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: Deploying cephadm binary to smithi134 2024-04-22T23:16:04.701 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:04.701 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.701 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.701 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.701 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.701 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.702 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: Deploying daemon node-exporter.smithi067 on smithi067 2024-04-22T23:16:04.702 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.702 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: Added host smithi134 2024-04-22T23:16:04.702 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:04.702 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:04.702 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.702 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='client.14193 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T23:16:04.702 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:04.702 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.702 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.703 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.703 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.703 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: Deploying daemon alertmanager.smithi067 on smithi067 2024-04-22T23:16:04.703 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:04.703 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/619091433' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-22T23:16:04.703 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/619091433' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-22T23:16:04.703 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T23:16:04.703 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.703 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:04.703 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:04.703 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.703 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='client.14197 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi067:172.21.15.67=smithi067;smithi134:172.21.15.134=smithi134", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:16:04.703 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: Saving service mon spec with placement smithi067:172.21.15.67=smithi067;smithi134:172.21.15.134=smithi134;count:2 2024-04-22T23:16:04.703 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:04.703 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='client.? 172.21.15.134:0/2738924500' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T23:16:04.704 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.704 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.704 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.704 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.704 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.704 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.704 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-22T23:16:04.704 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.704 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.704 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-22T23:16:04.704 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-22T23:16:04.704 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.704 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: Deploying daemon grafana.smithi067 on smithi067 2024-04-22T23:16:04.704 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:04.704 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:04.704 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='client.? 172.21.15.134:0/1638848932' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T23:16:04.705 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:04.705 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.705 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='client.? 172.21.15.134:0/3304431427' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T23:16:04.705 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:04.705 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:04.705 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='client.? 172.21.15.134:0/3560553857' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T23:16:04.705 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:04.705 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:04.705 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='client.? 172.21.15.134:0/1156771103' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T23:16:04.705 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:04.705 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.705 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.705 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.705 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.705 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.706 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.706 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.706 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.706 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: Deploying daemon prometheus.smithi067 on smithi067 2024-04-22T23:16:04.706 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='client.? 172.21.15.134:0/3455744832' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T23:16:04.706 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:04.706 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:04.706 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.706 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='client.? 172.21.15.134:0/178526123' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T23:16:04.706 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:04.706 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:04.706 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='client.? 172.21.15.134:0/4178656090' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T23:16:04.706 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:04.706 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.707 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.707 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.707 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-22T23:16:04.707 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14164 172.21.15.67:0/2098111869' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-22T23:16:04.707 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: mgrmap e14: smithi067.whdcsr(active, since 61s) 2024-04-22T23:16:04.707 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='client.? 172.21.15.134:0/1305609621' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T23:16:04.707 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='client.? 172.21.15.134:0/3253880701' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T23:16:04.707 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: Active manager daemon smithi067.whdcsr restarted 2024-04-22T23:16:04.707 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: Activating manager daemon smithi067.whdcsr 2024-04-22T23:16:04.707 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: osdmap e5: 0 total, 0 up, 0 in 2024-04-22T23:16:04.707 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: mgrmap e15: smithi067.whdcsr(active, starting, since 0.00333419s) 2024-04-22T23:16:04.707 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-22T23:16:04.707 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mgr metadata", "who": "smithi067.whdcsr", "id": "smithi067.whdcsr"}]: dispatch 2024-04-22T23:16:04.707 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T23:16:04.707 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T23:16:04.707 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T23:16:04.708 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: Manager daemon smithi067.whdcsr is now available 2024-04-22T23:16:04.708 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:16:04.708 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:16:04.708 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.708 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:16:04.708 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.whdcsr/mirror_snapshot_schedule"}]: dispatch 2024-04-22T23:16:04.708 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.whdcsr/trash_purge_schedule"}]: dispatch 2024-04-22T23:16:04.708 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.708 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='client.? 172.21.15.134:0/3989777031' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T23:16:04.708 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: mgrmap e16: smithi067.whdcsr(active, since 1.00563s) 2024-04-22T23:16:04.709 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: [22/Apr/2024:23:15:37] ENGINE Bus STARTING 2024-04-22T23:16:04.709 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: [22/Apr/2024:23:15:37] ENGINE Serving on https://172.21.15.67:7150 2024-04-22T23:16:04.709 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: [22/Apr/2024:23:15:38] ENGINE Serving on http://172.21.15.67:8765 2024-04-22T23:16:04.709 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: [22/Apr/2024:23:15:38] ENGINE Bus STARTED 2024-04-22T23:16:04.709 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.709 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.709 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: mgrmap e17: smithi067.whdcsr(active, since 2s) 2024-04-22T23:16:04.709 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.709 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.709 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.709 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.709 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.709 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.709 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "who": "osd/host:smithi134", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:16:04.709 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.710 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.710 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "who": "osd/host:smithi067", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:16:04.710 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:16:04.710 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:16:04.710 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: Updating smithi067:/etc/ceph/ceph.conf 2024-04-22T23:16:04.710 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: Updating smithi134:/etc/ceph/ceph.conf 2024-04-22T23:16:04.710 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: Updating smithi067:/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T23:16:04.710 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: Updating smithi134:/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T23:16:04.710 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-04-22T23:16:04.710 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: Updating smithi134:/etc/ceph/ceph.client.admin.keyring 2024-04-22T23:16:04.710 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: Updating smithi067:/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T23:16:04.710 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: Updating smithi134:/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T23:16:04.710 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='client.? 172.21.15.134:0/2460354466' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T23:16:04.710 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.710 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.711 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.711 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.711 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.711 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi134", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T23:16:04.711 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi134", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-22T23:16:04.711 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:16:04.711 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: Deploying daemon ceph-exporter.smithi134 on smithi134 2024-04-22T23:16:04.711 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='client.? 172.21.15.134:0/3937462797' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T23:16:04.711 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.711 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.711 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.711 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.711 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi134", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T23:16:04.711 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi134", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-22T23:16:04.711 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:16:04.712 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: Deploying daemon crash.smithi134 on smithi134 2024-04-22T23:16:04.712 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='client.? 172.21.15.134:0/585726373' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T23:16:04.712 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.712 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.712 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.712 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.712 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: Deploying daemon node-exporter.smithi134 on smithi134 2024-04-22T23:16:04.712 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.712 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:16:04.712 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.712 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.712 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.712 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.712 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi134.ccxkcj", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T23:16:04.712 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi134.ccxkcj", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T23:16:04.712 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T23:16:04.713 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:16:04.713 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: Deploying daemon mgr.smithi134.ccxkcj on smithi134 2024-04-22T23:16:04.713 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='client.? 172.21.15.134:0/4207089009' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T23:16:04.713 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:04.713 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.713 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.713 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.713 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.713 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.713 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T23:16:04.713 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:16:04.713 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: Deploying daemon mon.smithi134 on smithi134 2024-04-22T23:16:04.713 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:04.713 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='client.? 172.21.15.134:0/1487104516' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T23:16:04.713 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:04.714 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:04.714 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: mon.smithi134@-1(synchronizing).paxosservice(auth 1..8) refresh upgraded, format 0 -> 3 2024-04-22T23:16:04.714 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: mon.smithi134@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-04-22T23:16:04.714 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: expand_channel_meta expand map: {default=false} 2024-04-22T23:16:04.714 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: expand_channel_meta from 'false' to 'false' 2024-04-22T23:16:04.714 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: expand_channel_meta expanded map: {default=false} 2024-04-22T23:16:04.714 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: expand_channel_meta expand map: {default=info} 2024-04-22T23:16:04.714 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: expand_channel_meta from 'info' to 'info' 2024-04-22T23:16:04.714 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: expand_channel_meta expanded map: {default=info} 2024-04-22T23:16:04.714 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: expand_channel_meta expand map: {default=daemon} 2024-04-22T23:16:04.714 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: expand_channel_meta from 'daemon' to 'daemon' 2024-04-22T23:16:04.714 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: expand_channel_meta expanded map: {default=daemon} 2024-04-22T23:16:04.714 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: expand_channel_meta expand map: {default=debug} 2024-04-22T23:16:04.714 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: expand_channel_meta from 'debug' to 'debug' 2024-04-22T23:16:04.714 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 ceph-mon[34609]: expand_channel_meta expanded map: {default=debug} 2024-04-22T23:16:04.714 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 bash[34562]: 2fb23795f7e3ee64e6f0dc5b6806658d3b8dcf49fa78aaa0b246fbc72bf339aa 2024-04-22T23:16:04.715 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:04 smithi134 systemd[1]: Started Ceph mon.smithi134 for d46c346e-00fd-11ef-bc93-c7b262605968. 2024-04-22T23:16:09.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:09.442+0000 7fe600ff9700 1 -- 172.21.15.134:0/3251394088 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1022 (secure 0 0 0) 0x7fe5f402a360 con 0x7fe604106930 2024-04-22T23:16:09.444 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-22T23:16:09.444 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":2,"fsid":"d46c346e-00fd-11ef-bc93-c7b262605968","modified":"2024-04-22T23:16:04.430584Z","created":"2024-04-22T23:13:15.759242Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi067","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi134","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:3300","nonce":0},{"type":"v1","addr":"172.21.15.134:6789","nonce":0}]},"addr":"172.21.15.134:6789/0","public_addr":"172.21.15.134:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-22T23:16:09.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:09.445+0000 7fe60a180700 1 -- 172.21.15.134:0/3251394088 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fe5ec0384b0 msgr2=0x7fe5ec03a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:16:09.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:09.445+0000 7fe60a180700 1 --2- 172.21.15.134:0/3251394088 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fe5ec0384b0 0x7fe5ec03a970 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe5f8006fd0 tx=0x7fe5f8006e40 comp rx=0 tx=0).stop 2024-04-22T23:16:09.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:09.445+0000 7fe60a180700 1 -- 172.21.15.134:0/3251394088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe604106930 msgr2=0x7fe604195e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:16:09.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:09.445+0000 7fe60a180700 1 --2- 172.21.15.134:0/3251394088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe604106930 0x7fe604195e60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe5f4005b40 tx=0x7fe5f4004c80 comp rx=0 tx=0).stop 2024-04-22T23:16:09.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:09.446+0000 7fe60a180700 1 -- 172.21.15.134:0/3251394088 shutdown_connections 2024-04-22T23:16:09.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:09.446+0000 7fe60a180700 1 --2- 172.21.15.134:0/3251394088 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fe5ec0384b0 0x7fe5ec03a970 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:16:09.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:09.446+0000 7fe60a180700 1 --2- 172.21.15.134:0/3251394088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe604106930 0x7fe604195e60 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:16:09.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:09.446+0000 7fe60a180700 1 -- 172.21.15.134:0/3251394088 >> 172.21.15.134:0/3251394088 conn(0x7fe6040fc400 msgr2=0x7fe604108d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:16:09.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:09.446+0000 7fe60a180700 1 -- 172.21.15.134:0/3251394088 shutdown_connections 2024-04-22T23:16:09.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:16:09.446+0000 7fe60a180700 1 -- 172.21.15.134:0/3251394088 wait complete. 2024-04-22T23:16:09.448 INFO:teuthology.orchestra.run.smithi134.stderr:dumped monmap epoch 2 2024-04-22T23:16:09.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:09 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-22T23:16:09.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:09 smithi067 ceph-mon[26453]: mon.smithi067 calling monitor election 2024-04-22T23:16:09.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:09 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-04-22T23:16:09.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:09 smithi067 ceph-mon[26453]: from='client.? 172.21.15.134:0/3251394088' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T23:16:09.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:09 smithi067 ceph-mon[26453]: from='mgr.? 172.21.15.134:0/4059094102' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi134.ccxkcj/crt"}]: dispatch 2024-04-22T23:16:09.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:09 smithi067 ceph-mon[26453]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:09.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:09 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-04-22T23:16:09.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:09 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-04-22T23:16:09.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:09 smithi067 ceph-mon[26453]: mon.smithi134 calling monitor election 2024-04-22T23:16:09.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:09 smithi067 ceph-mon[26453]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:09.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:09 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-04-22T23:16:09.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:09 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:16:09.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:09 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-04-22T23:16:09.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:09 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-04-22T23:16:09.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:09 smithi067 ceph-mon[26453]: mon.smithi067 is new leader, mons smithi067,smithi134 in quorum (ranks 0,1) 2024-04-22T23:16:09.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:09 smithi067 ceph-mon[26453]: monmap e2: 2 mons at {smithi067=[v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0],smithi134=[v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-22T23:16:09.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:09 smithi067 ceph-mon[26453]: fsmap 2024-04-22T23:16:09.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:09 smithi067 ceph-mon[26453]: osdmap e5: 0 total, 0 up, 0 in 2024-04-22T23:16:09.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:09 smithi067 ceph-mon[26453]: mgrmap e17: smithi067.whdcsr(active, since 32s) 2024-04-22T23:16:09.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:09 smithi067 ceph-mon[26453]: Standby manager daemon smithi134.ccxkcj started 2024-04-22T23:16:09.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:09 smithi067 ceph-mon[26453]: from='mgr.? 172.21.15.134:0/4059094102' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T23:16:09.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:09 smithi067 ceph-mon[26453]: overall HEALTH_OK 2024-04-22T23:16:09.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:09 smithi067 ceph-mon[26453]: from='mgr.? 172.21.15.134:0/4059094102' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi134.ccxkcj/key"}]: dispatch 2024-04-22T23:16:09.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:09 smithi067 ceph-mon[26453]: from='mgr.? 172.21.15.134:0/4059094102' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T23:16:09.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:09 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:09.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:09 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:09.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:09 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:09.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:09 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:09.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:09 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:09.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:09 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-22T23:16:09.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:09 smithi134 ceph-mon[34609]: mon.smithi067 calling monitor election 2024-04-22T23:16:09.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:09 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-04-22T23:16:09.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:09 smithi134 ceph-mon[34609]: from='client.? 172.21.15.134:0/3251394088' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T23:16:09.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:09 smithi134 ceph-mon[34609]: from='mgr.? 172.21.15.134:0/4059094102' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi134.ccxkcj/crt"}]: dispatch 2024-04-22T23:16:09.746 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:09 smithi134 ceph-mon[34609]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:09.746 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:09 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-04-22T23:16:09.746 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:09 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-04-22T23:16:09.746 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:09 smithi134 ceph-mon[34609]: mon.smithi134 calling monitor election 2024-04-22T23:16:09.746 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:09 smithi134 ceph-mon[34609]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:09.746 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:09 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-04-22T23:16:09.747 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:09 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:16:09.747 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:09 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-04-22T23:16:09.747 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:09 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-04-22T23:16:09.747 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:09 smithi134 ceph-mon[34609]: mon.smithi067 is new leader, mons smithi067,smithi134 in quorum (ranks 0,1) 2024-04-22T23:16:09.747 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:09 smithi134 ceph-mon[34609]: monmap e2: 2 mons at {smithi067=[v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0],smithi134=[v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-22T23:16:09.748 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:09 smithi134 ceph-mon[34609]: fsmap 2024-04-22T23:16:09.748 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:09 smithi134 ceph-mon[34609]: osdmap e5: 0 total, 0 up, 0 in 2024-04-22T23:16:09.748 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:09 smithi134 ceph-mon[34609]: mgrmap e17: smithi067.whdcsr(active, since 32s) 2024-04-22T23:16:09.748 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:09 smithi134 ceph-mon[34609]: Standby manager daemon smithi134.ccxkcj started 2024-04-22T23:16:09.748 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:09 smithi134 ceph-mon[34609]: from='mgr.? 172.21.15.134:0/4059094102' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T23:16:09.748 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:09 smithi134 ceph-mon[34609]: overall HEALTH_OK 2024-04-22T23:16:09.749 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:09 smithi134 ceph-mon[34609]: from='mgr.? 172.21.15.134:0/4059094102' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi134.ccxkcj/key"}]: dispatch 2024-04-22T23:16:09.749 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:09 smithi134 ceph-mon[34609]: from='mgr.? 172.21.15.134:0/4059094102' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T23:16:09.749 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:09 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:09.749 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:09 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:09.749 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:09 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:09.749 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:09 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:09.749 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:09 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:10.006 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-22T23:16:10.006 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-22T23:16:10.231 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:16:10.661 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:10 smithi067 ceph-mon[26453]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:10.662 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:10 smithi067 ceph-mon[26453]: mgrmap e18: smithi067.whdcsr(active, since 32s), standbys: smithi134.ccxkcj 2024-04-22T23:16:10.662 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:10 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mgr metadata", "who": "smithi134.ccxkcj", "id": "smithi134.ccxkcj"}]: dispatch 2024-04-22T23:16:10.662 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:10 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:16:10.662 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:10 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-04-22T23:16:10.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:10 smithi134 ceph-mon[34609]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:10.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:10 smithi134 ceph-mon[34609]: mgrmap e18: smithi067.whdcsr(active, since 32s), standbys: smithi134.ccxkcj 2024-04-22T23:16:10.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:10 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mgr metadata", "who": "smithi134.ccxkcj", "id": "smithi134.ccxkcj"}]: dispatch 2024-04-22T23:16:10.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:10 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:16:10.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:10 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-04-22T23:16:11.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.697+0000 7ff488e7e700 1 -- 172.21.15.67:0/1747983033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4841067a0 msgr2=0x7ff484106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:16:11.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.697+0000 7ff488e7e700 1 --2- 172.21.15.67:0/1747983033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4841067a0 0x7ff484106b80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff474009910 tx=0x7ff474009c20 comp rx=0 tx=0).stop 2024-04-22T23:16:11.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.697+0000 7ff488e7e700 1 -- 172.21.15.67:0/1747983033 shutdown_connections 2024-04-22T23:16:11.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.697+0000 7ff488e7e700 1 --2- 172.21.15.67:0/1747983033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4841067a0 0x7ff484106b80 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:16:11.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.697+0000 7ff488e7e700 1 -- 172.21.15.67:0/1747983033 >> 172.21.15.67:0/1747983033 conn(0x7ff4840fc2f0 msgr2=0x7ff4840fe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:16:11.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.697+0000 7ff488e7e700 1 -- 172.21.15.67:0/1747983033 shutdown_connections 2024-04-22T23:16:11.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.697+0000 7ff488e7e700 1 -- 172.21.15.67:0/1747983033 wait complete. 2024-04-22T23:16:11.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.698+0000 7ff488e7e700 1 Processor -- start 2024-04-22T23:16:11.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.698+0000 7ff488e7e700 1 -- start start 2024-04-22T23:16:11.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.698+0000 7ff488e7e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4841067a0 0x7ff484073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:16:11.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.698+0000 7ff488e7e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4840735d0 0x7ff484073a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:16:11.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.698+0000 7ff488e7e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4841a20c0 con 0x7ff4840735d0 2024-04-22T23:16:11.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.698+0000 7ff488e7e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4841a2200 con 0x7ff4841067a0 2024-04-22T23:16:11.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.698+0000 7ff481d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4840735d0 0x7ff484073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:16:11.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.698+0000 7ff481d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4840735d0 0x7ff484073a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44932/0 (socket says 172.21.15.67:44932) 2024-04-22T23:16:11.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.698+0000 7ff481d9b700 1 -- 172.21.15.67:0/612812767 learned_addr learned my addr 172.21.15.67:0/612812767 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:16:11.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.698+0000 7ff48259c700 1 --2- 172.21.15.67:0/612812767 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4841067a0 0x7ff484073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:16:11.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.699+0000 7ff48259c700 1 -- 172.21.15.67:0/612812767 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4841067a0 msgr2=0x7ff484073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-22T23:16:11.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.699+0000 7ff481d9b700 1 -- 172.21.15.67:0/612812767 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4841067a0 msgr2=0x7ff484073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:16:11.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.699+0000 7ff48259c700 1 -- 172.21.15.67:0/612812767 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4841067a0 msgr2=0x7ff484073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T23:16:11.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.699+0000 7ff48259c700 1 --2- 172.21.15.67:0/612812767 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4841067a0 0x7ff484073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T23:16:11.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.699+0000 7ff48259c700 1 --2- 172.21.15.67:0/612812767 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4841067a0 0x7ff484073090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T23:16:11.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.699+0000 7ff481d9b700 1 --2- 172.21.15.67:0/612812767 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4841067a0 0x7ff484073090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:16:11.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.699+0000 7ff481d9b700 1 -- 172.21.15.67:0/612812767 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4740095f0 con 0x7ff4840735d0 2024-04-22T23:16:11.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.699+0000 7ff481d9b700 1 --2- 172.21.15.67:0/612812767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4840735d0 0x7ff484073a50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff47800b730 tx=0x7ff47800baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:16:11.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.699+0000 7ff4737fe700 1 -- 172.21.15.67:0/612812767 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff478004200 con 0x7ff4840735d0 2024-04-22T23:16:11.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.699+0000 7ff4737fe700 1 -- 172.21.15.67:0/612812767 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff478004360 con 0x7ff4840735d0 2024-04-22T23:16:11.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.699+0000 7ff488e7e700 1 -- 172.21.15.67:0/612812767 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4841a2340 con 0x7ff4840735d0 2024-04-22T23:16:11.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.699+0000 7ff4737fe700 1 -- 172.21.15.67:0/612812767 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff478011be0 con 0x7ff4840735d0 2024-04-22T23:16:11.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.699+0000 7ff488e7e700 1 -- 172.21.15.67:0/612812767 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4841a2890 con 0x7ff4840735d0 2024-04-22T23:16:11.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.700+0000 7ff488e7e700 1 -- 172.21.15.67:0/612812767 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4840623c0 con 0x7ff4840735d0 2024-04-22T23:16:11.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.701+0000 7ff4737fe700 1 -- 172.21.15.67:0/612812767 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 90256+0+0 (secure 0 0 0) 0x7ff4780036a0 con 0x7ff4840735d0 2024-04-22T23:16:11.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.701+0000 7ff4737fe700 1 --2- 172.21.15.67:0/612812767 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ff46c06c5b0 0x7ff46c06ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:16:11.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.701+0000 7ff4737fe700 1 -- 172.21.15.67:0/612812767 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7ff47808a1f0 con 0x7ff4840735d0 2024-04-22T23:16:11.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.701+0000 7ff48259c700 1 --2- 172.21.15.67:0/612812767 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ff46c06c5b0 0x7ff46c06ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:16:11.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.702+0000 7ff48259c700 1 --2- 172.21.15.67:0/612812767 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ff46c06c5b0 0x7ff46c06ea70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff474006010 tx=0x7ff47400b540 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:16:11.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.707+0000 7ff4737fe700 1 -- 172.21.15.67:0/612812767 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff478055b30 con 0x7ff4840735d0 2024-04-22T23:16:11.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.928+0000 7ff488e7e700 1 -- 172.21.15.67:0/612812767 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7ff4841a2c60 con 0x7ff4840735d0 2024-04-22T23:16:11.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.929+0000 7ff4737fe700 1 -- 172.21.15.67:0/612812767 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+225 (secure 0 0 0) 0x7ff4780899b0 con 0x7ff4840735d0 2024-04-22T23:16:11.930 INFO:teuthology.orchestra.run.smithi067.stdout:# minimal ceph.conf for d46c346e-00fd-11ef-bc93-c7b262605968 2024-04-22T23:16:11.930 INFO:teuthology.orchestra.run.smithi067.stdout:[global] 2024-04-22T23:16:11.930 INFO:teuthology.orchestra.run.smithi067.stdout: fsid = d46c346e-00fd-11ef-bc93-c7b262605968 2024-04-22T23:16:11.930 INFO:teuthology.orchestra.run.smithi067.stdout: mon_host = [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] 2024-04-22T23:16:11.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.931+0000 7ff488e7e700 1 -- 172.21.15.67:0/612812767 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ff46c06c5b0 msgr2=0x7ff46c06ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:16:11.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.931+0000 7ff488e7e700 1 --2- 172.21.15.67:0/612812767 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ff46c06c5b0 0x7ff46c06ea70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff474006010 tx=0x7ff47400b540 comp rx=0 tx=0).stop 2024-04-22T23:16:11.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.931+0000 7ff488e7e700 1 -- 172.21.15.67:0/612812767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4840735d0 msgr2=0x7ff484073a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:16:11.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.931+0000 7ff488e7e700 1 --2- 172.21.15.67:0/612812767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4840735d0 0x7ff484073a50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff47800b730 tx=0x7ff47800baf0 comp rx=0 tx=0).stop 2024-04-22T23:16:11.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.931+0000 7ff488e7e700 1 -- 172.21.15.67:0/612812767 shutdown_connections 2024-04-22T23:16:11.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.931+0000 7ff488e7e700 1 --2- 172.21.15.67:0/612812767 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ff46c06c5b0 0x7ff46c06ea70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:16:11.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.931+0000 7ff488e7e700 1 --2- 172.21.15.67:0/612812767 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4841067a0 0x7ff484073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:16:11.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.931+0000 7ff488e7e700 1 --2- 172.21.15.67:0/612812767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4840735d0 0x7ff484073a50 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:16:11.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.931+0000 7ff488e7e700 1 -- 172.21.15.67:0/612812767 >> 172.21.15.67:0/612812767 conn(0x7ff4840fc2f0 msgr2=0x7ff4841022a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:16:11.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.931+0000 7ff488e7e700 1 -- 172.21.15.67:0/612812767 shutdown_connections 2024-04-22T23:16:11.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:11.932+0000 7ff488e7e700 1 -- 172.21.15.67:0/612812767 wait complete. 2024-04-22T23:16:12.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:12 smithi067 ceph-mon[26453]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:12.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:12 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:12.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:12 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:12.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:12 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:16:12.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:12 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:16:12.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:12 smithi067 ceph-mon[26453]: Updating smithi067:/etc/ceph/ceph.conf 2024-04-22T23:16:12.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:12 smithi067 ceph-mon[26453]: Updating smithi134:/etc/ceph/ceph.conf 2024-04-22T23:16:12.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:12 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/612812767' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:16:12.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:12 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:12.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:12 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:12.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:12 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:12.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:12 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:12.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:12 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:12.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:12 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T23:16:12.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:12 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T23:16:12.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:12 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:16:12.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:12 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:12.550 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-22T23:16:12.550 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-22T23:16:12.550 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.conf 2024-04-22T23:16:12.577 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-22T23:16:12.577 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T23:16:12.642 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-22T23:16:12.642 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/ceph/ceph.conf 2024-04-22T23:16:12.671 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-22T23:16:12.671 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T23:16:12.736 INFO:tasks.cephadm:Deploying OSDs... 2024-04-22T23:16:12.736 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-22T23:16:12.736 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T23:16:12.752 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T23:16:12.752 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_1 2024-04-22T23:16:12.809 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T23:16:12.810 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T23:16:12.810 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-22T23:16:12.810 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T23:16:12.810 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T23:16:12.810 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-04-22 23:15:28.772540133 +0000 2024-04-22T23:16:12.810 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-04-22 23:09:58.251006738 +0000 2024-04-22T23:16:12.810 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-04-22 23:09:58.251006738 +0000 2024-04-22T23:16:12.810 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-04-22 23:09:58.251006738 +0000 2024-04-22T23:16:12.810 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T23:16:12.876 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-22T23:16:12.876 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-22T23:16:12.876 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000256409 s, 2.0 MB/s 2024-04-22T23:16:12.877 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T23:16:12.934 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_2 2024-04-22T23:16:12.989 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T23:16:12.989 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T23:16:12.989 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-22T23:16:12.990 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T23:16:12.990 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T23:16:12.990 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-04-22 23:15:28.773540111 +0000 2024-04-22T23:16:12.990 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-04-22 23:09:58.609998811 +0000 2024-04-22T23:16:12.990 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-04-22 23:09:58.609998811 +0000 2024-04-22T23:16:12.990 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-04-22 23:09:58.609998811 +0000 2024-04-22T23:16:12.990 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T23:16:12.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:12 smithi134 ceph-mon[34609]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:12.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:12 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:12.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:12 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:12.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:12 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:16:12.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:12 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:16:12.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:12 smithi134 ceph-mon[34609]: Updating smithi067:/etc/ceph/ceph.conf 2024-04-22T23:16:12.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:12 smithi134 ceph-mon[34609]: Updating smithi134:/etc/ceph/ceph.conf 2024-04-22T23:16:12.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:12 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/612812767' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:16:12.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:12 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:12.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:12 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:12.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:12 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:12.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:12 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:12.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:12 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:12.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:12 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T23:16:12.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:12 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T23:16:12.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:12 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:16:12.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:12 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:13.053 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-22T23:16:13.053 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-22T23:16:13.053 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000204201 s, 2.5 MB/s 2024-04-22T23:16:13.054 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T23:16:13.110 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_3 2024-04-22T23:16:13.166 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T23:16:13.166 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T23:16:13.166 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-22T23:16:13.166 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T23:16:13.166 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T23:16:13.166 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-04-22 23:15:28.773540111 +0000 2024-04-22T23:16:13.167 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-04-22 23:09:58.939991524 +0000 2024-04-22T23:16:13.167 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-04-22 23:09:58.939991524 +0000 2024-04-22T23:16:13.167 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-04-22 23:09:58.939991524 +0000 2024-04-22T23:16:13.167 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T23:16:13.229 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-22T23:16:13.229 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-22T23:16:13.229 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000201751 s, 2.5 MB/s 2024-04-22T23:16:13.231 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T23:16:13.287 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_4 2024-04-22T23:16:13.343 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T23:16:13.343 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T23:16:13.343 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-22T23:16:13.343 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T23:16:13.343 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T23:16:13.343 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-04-22 23:15:28.773540111 +0000 2024-04-22T23:16:13.343 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-04-22 23:09:59.282983950 +0000 2024-04-22T23:16:13.343 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-04-22 23:09:59.282983950 +0000 2024-04-22T23:16:13.344 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-04-22 23:09:59.282983950 +0000 2024-04-22T23:16:13.344 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T23:16:13.407 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-22T23:16:13.407 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-22T23:16:13.407 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000175592 s, 2.9 MB/s 2024-04-22T23:16:13.408 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T23:16:13.465 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-22T23:16:13.465 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T23:16:13.485 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T23:16:13.485 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_1 2024-04-22T23:16:13.542 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T23:16:13.542 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T23:16:13.542 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-22T23:16:13.542 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T23:16:13.542 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T23:16:13.542 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-04-22 23:16:02.605160942 +0000 2024-04-22T23:16:13.542 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-04-22 23:09:54.697190671 +0000 2024-04-22T23:16:13.542 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-04-22 23:09:54.697190671 +0000 2024-04-22T23:16:13.542 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-04-22 23:09:54.697190671 +0000 2024-04-22T23:16:13.543 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T23:16:13.607 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-04-22T23:16:13.607 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-04-22T23:16:13.607 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000262713 s, 1.9 MB/s 2024-04-22T23:16:13.608 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T23:16:13.667 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_2 2024-04-22T23:16:13.723 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T23:16:13.724 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T23:16:13.724 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-22T23:16:13.724 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T23:16:13.724 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T23:16:13.724 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-04-22 23:16:02.605160942 +0000 2024-04-22T23:16:13.724 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-04-22 23:09:55.047182469 +0000 2024-04-22T23:16:13.724 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-04-22 23:09:55.047182469 +0000 2024-04-22T23:16:13.724 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-04-22 23:09:55.047182469 +0000 2024-04-22T23:16:13.724 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T23:16:13.786 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:13 smithi134 ceph-mon[34609]: Updating smithi134:/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T23:16:13.786 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:13 smithi134 ceph-mon[34609]: Updating smithi067:/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T23:16:13.786 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:13 smithi134 ceph-mon[34609]: Reconfiguring mon.smithi067 (unknown last config time)... 2024-04-22T23:16:13.786 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:13 smithi134 ceph-mon[34609]: Reconfiguring daemon mon.smithi067 on smithi067 2024-04-22T23:16:13.788 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-04-22T23:16:13.788 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-04-22T23:16:13.788 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000207996 s, 2.5 MB/s 2024-04-22T23:16:13.790 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T23:16:13.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:13 smithi067 ceph-mon[26453]: Updating smithi134:/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T23:16:13.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:13 smithi067 ceph-mon[26453]: Updating smithi067:/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T23:16:13.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:13 smithi067 ceph-mon[26453]: Reconfiguring mon.smithi067 (unknown last config time)... 2024-04-22T23:16:13.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:13 smithi067 ceph-mon[26453]: Reconfiguring daemon mon.smithi067 on smithi067 2024-04-22T23:16:13.847 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_3 2024-04-22T23:16:13.903 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T23:16:13.903 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T23:16:13.904 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-22T23:16:13.904 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T23:16:13.904 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T23:16:13.904 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-04-22 23:16:02.606160918 +0000 2024-04-22T23:16:13.904 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-04-22 23:09:55.379174688 +0000 2024-04-22T23:16:13.904 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-04-22 23:09:55.379174688 +0000 2024-04-22T23:16:13.904 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-04-22 23:09:55.379174688 +0000 2024-04-22T23:16:13.904 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T23:16:13.968 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-04-22T23:16:13.968 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-04-22T23:16:13.968 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000246836 s, 2.1 MB/s 2024-04-22T23:16:13.969 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T23:16:14.026 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_4 2024-04-22T23:16:14.082 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T23:16:14.083 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T23:16:14.083 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-04-22T23:16:14.083 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T23:16:14.083 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T23:16:14.083 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-04-22 23:16:02.606160918 +0000 2024-04-22T23:16:14.083 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-04-22 23:09:55.717166766 +0000 2024-04-22T23:16:14.083 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-04-22 23:09:55.717166766 +0000 2024-04-22T23:16:14.083 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-04-22 23:09:55.717166766 +0000 2024-04-22T23:16:14.084 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T23:16:14.146 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-04-22T23:16:14.146 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-04-22T23:16:14.146 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000238467 s, 2.1 MB/s 2024-04-22T23:16:14.147 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T23:16:14.204 INFO:tasks.cephadm:Deploying osd.0 on smithi067 with /dev/vg_nvme/lv_4... 2024-04-22T23:16:14.204 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-22T23:16:14.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:14 smithi067 ceph-mon[26453]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:14.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:14 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:14.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:14 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:14.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:14 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi067.whdcsr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T23:16:14.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:14 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T23:16:14.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:14 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:16:14.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:14 smithi134 ceph-mon[34609]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:14.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:14 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:14.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:14 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:14.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:14 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi067.whdcsr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T23:16:14.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:14 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T23:16:14.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:14 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:16:15.178 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:16:15.555 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:15 smithi067 ceph-mon[26453]: Reconfiguring mgr.smithi067.whdcsr (unknown last config time)... 2024-04-22T23:16:15.555 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:15 smithi067 ceph-mon[26453]: Reconfiguring daemon mgr.smithi067.whdcsr on smithi067 2024-04-22T23:16:15.555 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:15 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:15.555 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:15 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:15.555 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:15 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi067", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T23:16:15.556 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:15 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:16:15.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:15 smithi134 ceph-mon[34609]: Reconfiguring mgr.smithi067.whdcsr (unknown last config time)... 2024-04-22T23:16:15.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:15 smithi134 ceph-mon[34609]: Reconfiguring daemon mgr.smithi067.whdcsr on smithi067 2024-04-22T23:16:15.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:15 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:15.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:15 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:15.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:15 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi067", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T23:16:15.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:15 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:16:16.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:16 smithi067 ceph-mon[26453]: Reconfiguring ceph-exporter.smithi067 (monmap changed)... 2024-04-22T23:16:16.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:16 smithi067 ceph-mon[26453]: Reconfiguring daemon ceph-exporter.smithi067 on smithi067 2024-04-22T23:16:16.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:16 smithi067 ceph-mon[26453]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:16.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:16 smithi134 ceph-mon[34609]: Reconfiguring ceph-exporter.smithi067 (monmap changed)... 2024-04-22T23:16:16.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:16 smithi134 ceph-mon[34609]: Reconfiguring daemon ceph-exporter.smithi067 on smithi067 2024-04-22T23:16:16.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:16 smithi134 ceph-mon[34609]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:18.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:17 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:18.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:17 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:18.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:17 smithi067 ceph-mon[26453]: Reconfiguring crash.smithi067 (monmap changed)... 2024-04-22T23:16:18.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:17 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi067", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T23:16:18.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:17 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:16:18.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:17 smithi067 ceph-mon[26453]: Reconfiguring daemon crash.smithi067 on smithi067 2024-04-22T23:16:18.046 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:17 smithi067 ceph-mon[26453]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:18.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:17 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:18.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:17 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:18.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:17 smithi134 ceph-mon[34609]: Reconfiguring crash.smithi067 (monmap changed)... 2024-04-22T23:16:18.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:17 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi067", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T23:16:18.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:17 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:16:18.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:17 smithi134 ceph-mon[34609]: Reconfiguring daemon crash.smithi067 on smithi067 2024-04-22T23:16:18.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:17 smithi134 ceph-mon[34609]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:19.033 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-22T23:16:19.033 INFO:teuthology.orchestra.run.smithi067.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-22T23:16:19.033 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 10+0 records in 2024-04-22T23:16:19.033 INFO:teuthology.orchestra.run.smithi067.stderr:10+0 records out 2024-04-22T23:16:19.033 INFO:teuthology.orchestra.run.smithi067.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0114612 s, 915 MB/s 2024-04-22T23:16:19.033 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping successful for: 2024-04-22T23:16:19.388 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:19 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:19.388 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:19 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:19.388 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:19 smithi067 ceph-mon[26453]: Reconfiguring alertmanager.smithi067 (dependencies changed)... 2024-04-22T23:16:19.388 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:19 smithi067 ceph-mon[26453]: Reconfiguring daemon alertmanager.smithi067 on smithi067 2024-04-22T23:16:19.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:19 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:19.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:19 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:19.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:19 smithi134 ceph-mon[34609]: Reconfiguring alertmanager.smithi067 (dependencies changed)... 2024-04-22T23:16:19.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:19 smithi134 ceph-mon[34609]: Reconfiguring daemon alertmanager.smithi067 on smithi067 2024-04-22T23:16:20.098 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi067:vg_nvme/lv_4 2024-04-22T23:16:20.441 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:20 smithi067 ceph-mon[26453]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:20.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:20 smithi134 ceph-mon[34609]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:21.646 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:16:22.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:22 smithi134 ceph-mon[34609]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:22.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:22 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:16:22.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:22 smithi067 ceph-mon[26453]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:22.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:22 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:16:23.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:23.225+0000 7f4e92f3c700 1 -- 172.21.15.67:0/1045319555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8c100e10 msgr2=0x7f4e8c101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:16:23.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:23.225+0000 7f4e92f3c700 1 --2- 172.21.15.67:0/1045319555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8c100e10 0x7f4e8c101270 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4e78009910 tx=0x7f4e78009c20 comp rx=0 tx=0).stop 2024-04-22T23:16:23.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:23.225+0000 7f4e92f3c700 1 -- 172.21.15.67:0/1045319555 shutdown_connections 2024-04-22T23:16:23.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:23.225+0000 7f4e92f3c700 1 --2- 172.21.15.67:0/1045319555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8c100e10 0x7f4e8c101270 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:16:23.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:23.225+0000 7f4e92f3c700 1 --2- 172.21.15.67:0/1045319555 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e8c0ffc10 0x7f4e8c100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:16:23.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:23.225+0000 7f4e92f3c700 1 -- 172.21.15.67:0/1045319555 >> 172.21.15.67:0/1045319555 conn(0x7f4e8c0fb190 msgr2=0x7f4e8c0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:16:23.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:23.225+0000 7f4e92f3c700 1 -- 172.21.15.67:0/1045319555 shutdown_connections 2024-04-22T23:16:23.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:23.225+0000 7f4e92f3c700 1 -- 172.21.15.67:0/1045319555 wait complete. 2024-04-22T23:16:23.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:23.226+0000 7f4e92f3c700 1 Processor -- start 2024-04-22T23:16:23.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:23.226+0000 7f4e92f3c700 1 -- start start 2024-04-22T23:16:23.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:23.226+0000 7f4e92f3c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8c0ffc10 0x7f4e8c196480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:16:23.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:23.226+0000 7f4e92f3c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e8c100e10 0x7f4e8c1969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:16:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:23.226+0000 7f4e92f3c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8c196fe0 con 0x7f4e8c0ffc10 2024-04-22T23:16:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:23.226+0000 7f4e92f3c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8c197120 con 0x7f4e8c100e10 2024-04-22T23:16:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:23.226+0000 7f4e90cd8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8c0ffc10 0x7f4e8c196480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:16:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:23.226+0000 7f4e90cd8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8c0ffc10 0x7f4e8c196480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38866/0 (socket says 172.21.15.67:38866) 2024-04-22T23:16:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:23.226+0000 7f4e90cd8700 1 -- 172.21.15.67:0/3145491474 learned_addr learned my addr 172.21.15.67:0/3145491474 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:16:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:23.226+0000 7f4e90cd8700 1 -- 172.21.15.67:0/3145491474 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e8c100e10 msgr2=0x7f4e8c1969c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:16:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:23.226+0000 7f4e90cd8700 1 --2- 172.21.15.67:0/3145491474 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e8c100e10 0x7f4e8c1969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:16:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:23.226+0000 7f4e90cd8700 1 -- 172.21.15.67:0/3145491474 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e780095f0 con 0x7f4e8c0ffc10 2024-04-22T23:16:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:23.226+0000 7f4e90cd8700 1 --2- 172.21.15.67:0/3145491474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8c0ffc10 0x7f4e8c196480 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4e8000c8f0 tx=0x7f4e8000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:16:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:23.226+0000 7f4e89ffb700 1 -- 172.21.15.67:0/3145491474 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e80007a70 con 0x7f4e8c0ffc10 2024-04-22T23:16:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:23.227+0000 7f4e89ffb700 1 -- 172.21.15.67:0/3145491474 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4e80007bd0 con 0x7f4e8c0ffc10 2024-04-22T23:16:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:23.227+0000 7f4e92f3c700 1 -- 172.21.15.67:0/3145491474 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e8c19bbd0 con 0x7f4e8c0ffc10 2024-04-22T23:16:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:23.227+0000 7f4e89ffb700 1 -- 172.21.15.67:0/3145491474 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e80018630 con 0x7f4e8c0ffc10 2024-04-22T23:16:23.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:23.227+0000 7f4e92f3c700 1 -- 172.21.15.67:0/3145491474 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e8c19c120 con 0x7f4e8c0ffc10 2024-04-22T23:16:23.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:23.228+0000 7f4e92f3c700 1 -- 172.21.15.67:0/3145491474 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e8c108700 con 0x7f4e8c0ffc10 2024-04-22T23:16:23.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:23.228+0000 7f4e89ffb700 1 -- 172.21.15.67:0/3145491474 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 90256+0+0 (secure 0 0 0) 0x7f4e8000f450 con 0x7f4e8c0ffc10 2024-04-22T23:16:23.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:23.229+0000 7f4e89ffb700 1 --2- 172.21.15.67:0/3145491474 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f4e7c06c470 0x7f4e7c06e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:16:23.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:23.229+0000 7f4e89ffb700 1 -- 172.21.15.67:0/3145491474 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f4e8008a6f0 con 0x7f4e8c0ffc10 2024-04-22T23:16:23.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:23.229+0000 7f4e8bfff700 1 --2- 172.21.15.67:0/3145491474 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f4e7c06c470 0x7f4e7c06e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:16:23.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:23.229+0000 7f4e8bfff700 1 --2- 172.21.15.67:0/3145491474 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f4e7c06c470 0x7f4e7c06e930 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4e7800b5c0 tx=0x7f4e78005fd0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:16:23.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:23.234+0000 7f4e89ffb700 1 -- 172.21.15.67:0/3145491474 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f4e80059af0 con 0x7f4e8c0ffc10 2024-04-22T23:16:23.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:23.443+0000 7f4e92f3c700 1 -- 172.21.15.67:0/3145491474 --> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f4e8c068c90 con 0x7f4e7c06c470 2024-04-22T23:16:23.731 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:23 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T23:16:23.731 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:23 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T23:16:23.731 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:23 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:16:23.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:23 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T23:16:23.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:23 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T23:16:23.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:23 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:16:24.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:24 smithi067 ceph-mon[26453]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:24.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:24 smithi067 ceph-mon[26453]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:16:24.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:24 smithi134 ceph-mon[34609]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:24.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:24 smithi134 ceph-mon[34609]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:16:26.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:25 smithi134 ceph-mon[34609]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:26.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:25 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:26.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:25 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:26.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:25 smithi067 ceph-mon[26453]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:26.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:25 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:26.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:25 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:27.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:26 smithi134 ceph-mon[34609]: Reconfiguring grafana.smithi067 (dependencies changed)... 2024-04-22T23:16:27.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:26 smithi134 ceph-mon[34609]: Reconfiguring daemon grafana.smithi067 on smithi067 2024-04-22T23:16:27.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:26 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/1899706630' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "25c9acd1-5743-4dc0-aee0-c7bb8884c0a3"}]: dispatch 2024-04-22T23:16:27.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:26 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/1899706630' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "25c9acd1-5743-4dc0-aee0-c7bb8884c0a3"}]': finished 2024-04-22T23:16:27.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:26 smithi134 ceph-mon[34609]: osdmap e6: 1 total, 0 up, 1 in 2024-04-22T23:16:27.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:26 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T23:16:27.256 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:26 smithi067 ceph-mon[26453]: Reconfiguring grafana.smithi067 (dependencies changed)... 2024-04-22T23:16:27.257 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:26 smithi067 ceph-mon[26453]: Reconfiguring daemon grafana.smithi067 on smithi067 2024-04-22T23:16:27.257 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:26 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/1899706630' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "25c9acd1-5743-4dc0-aee0-c7bb8884c0a3"}]: dispatch 2024-04-22T23:16:27.257 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:26 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/1899706630' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "25c9acd1-5743-4dc0-aee0-c7bb8884c0a3"}]': finished 2024-04-22T23:16:27.257 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:26 smithi067 ceph-mon[26453]: osdmap e6: 1 total, 0 up, 1 in 2024-04-22T23:16:27.257 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:26 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T23:16:28.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:27 smithi067 ceph-mon[26453]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:28.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:27 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/2043033147' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T23:16:28.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:27 smithi134 ceph-mon[34609]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:28.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:27 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/2043033147' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T23:16:30.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:30 smithi134 ceph-mon[34609]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:30.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:30 smithi067 ceph-mon[26453]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:32.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:32 smithi134 ceph-mon[34609]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:32.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:32 smithi067 ceph-mon[26453]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:34.605 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:34 smithi067 ceph-mon[26453]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:34.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:34 smithi134 ceph-mon[34609]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:36.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:35 smithi134 ceph-mon[34609]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:36.276 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:35 smithi067 ceph-mon[26453]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:37.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:37 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:37.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:37 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:37.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:37 smithi134 ceph-mon[34609]: Reconfiguring prometheus.smithi067 (dependencies changed)... 2024-04-22T23:16:37.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:37 smithi134 ceph-mon[34609]: Reconfiguring daemon prometheus.smithi067 on smithi067 2024-04-22T23:16:37.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:37 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:37.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:37 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:37.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:37 smithi067 ceph-mon[26453]: Reconfiguring prometheus.smithi067 (dependencies changed)... 2024-04-22T23:16:37.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:37 smithi067 ceph-mon[26453]: Reconfiguring daemon prometheus.smithi067 on smithi067 2024-04-22T23:16:38.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:38 smithi067 ceph-mon[26453]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:38.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:38 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:16:38.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:38 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T23:16:38.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:38 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:16:38.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:38 smithi134 ceph-mon[34609]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:38.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:38 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:16:38.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:38 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T23:16:38.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:38 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:16:39.235 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:39 smithi067 ceph-mon[26453]: Deploying daemon osd.0 on smithi067 2024-04-22T23:16:39.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:39 smithi134 ceph-mon[34609]: Deploying daemon osd.0 on smithi067 2024-04-22T23:16:40.470 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:40 smithi067 ceph-mon[26453]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:40.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:40 smithi134 ceph-mon[34609]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:42.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:42 smithi067 ceph-mon[26453]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:42.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:42 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:42.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:42 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:42.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:42 smithi067 ceph-mon[26453]: Reconfiguring ceph-exporter.smithi134 (monmap changed)... 2024-04-22T23:16:42.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:42 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi134", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T23:16:42.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:42 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:16:42.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:42 smithi067 ceph-mon[26453]: Reconfiguring daemon ceph-exporter.smithi134 on smithi134 2024-04-22T23:16:42.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:42 smithi134 ceph-mon[34609]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:42.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:42 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:42.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:42 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:42.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:42 smithi134 ceph-mon[34609]: Reconfiguring ceph-exporter.smithi134 (monmap changed)... 2024-04-22T23:16:42.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:42 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi134", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T23:16:42.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:42 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:16:42.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:42 smithi134 ceph-mon[34609]: Reconfiguring daemon ceph-exporter.smithi134 on smithi134 2024-04-22T23:16:43.895 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:43 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:43.895 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:43 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:43.895 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:43 smithi134 ceph-mon[34609]: Reconfiguring crash.smithi134 (monmap changed)... 2024-04-22T23:16:43.895 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:43 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi134", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T23:16:43.895 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:43 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:16:43.895 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:43 smithi134 ceph-mon[34609]: Reconfiguring daemon crash.smithi134 on smithi134 2024-04-22T23:16:43.895 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:43 smithi134 ceph-mon[34609]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:44.035 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:43 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:44.035 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:43 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:44.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:43 smithi067 ceph-mon[26453]: Reconfiguring crash.smithi134 (monmap changed)... 2024-04-22T23:16:44.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:43 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi134", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T23:16:44.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:43 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:16:44.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:43 smithi067 ceph-mon[26453]: Reconfiguring daemon crash.smithi134 on smithi134 2024-04-22T23:16:44.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:43 smithi067 ceph-mon[26453]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:45.396 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:45 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:45.396 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:45 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:45.396 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:45 smithi067 ceph-mon[26453]: Reconfiguring mgr.smithi134.ccxkcj (monmap changed)... 2024-04-22T23:16:45.396 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:45 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi134.ccxkcj", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T23:16:45.396 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:45 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T23:16:45.396 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:45 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:16:45.396 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:45 smithi067 ceph-mon[26453]: Reconfiguring daemon mgr.smithi134.ccxkcj on smithi134 2024-04-22T23:16:45.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:45 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:45.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:45 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:45.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:45 smithi134 ceph-mon[34609]: Reconfiguring mgr.smithi134.ccxkcj (monmap changed)... 2024-04-22T23:16:45.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:45 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi134.ccxkcj", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T23:16:45.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:45 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T23:16:45.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:45 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:16:45.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:45 smithi134 ceph-mon[34609]: Reconfiguring daemon mgr.smithi134.ccxkcj on smithi134 2024-04-22T23:16:46.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:46 smithi067 ceph-mon[26453]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:46.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:46 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:46.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:46 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:46.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:46 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T23:16:46.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:46 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T23:16:46.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:46 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:16:46.313 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:46 smithi134 ceph-mon[34609]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:46.313 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:46 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:46.313 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:46 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:46.313 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:46 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T23:16:46.313 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:46 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T23:16:46.313 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:46 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:16:47.543 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:47 smithi134 ceph-mon[34609]: Reconfiguring mon.smithi134 (monmap changed)... 2024-04-22T23:16:47.544 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:47 smithi134 ceph-mon[34609]: Reconfiguring daemon mon.smithi134 on smithi134 2024-04-22T23:16:47.544 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:47 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:47.544 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:47 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:47.544 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:47 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:47.544 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:47 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:47.544 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:47 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T23:16:47.544 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:47 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi067.front.sepia.ceph.com:9093"}]: dispatch 2024-04-22T23:16:47.544 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:47 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:47.544 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:47 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T23:16:47.544 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:47 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi067.front.sepia.ceph.com:3000"}]: dispatch 2024-04-22T23:16:47.544 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:47 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:47.545 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:47 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T23:16:47.545 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:47 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi067.front.sepia.ceph.com:9095"}]: dispatch 2024-04-22T23:16:47.545 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:47 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:47.545 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:47 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:16:47.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:47 smithi067 ceph-mon[26453]: Reconfiguring mon.smithi134 (monmap changed)... 2024-04-22T23:16:47.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:47 smithi067 ceph-mon[26453]: Reconfiguring daemon mon.smithi134 on smithi134 2024-04-22T23:16:47.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:47 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:47.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:47 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:47.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:47 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:47.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:47 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:47.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:47 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T23:16:47.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:47 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi067.front.sepia.ceph.com:9093"}]: dispatch 2024-04-22T23:16:47.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:47 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:47.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:47 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T23:16:47.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:47 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi067.front.sepia.ceph.com:3000"}]: dispatch 2024-04-22T23:16:47.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:47 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:47.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:47 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T23:16:47.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:47 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi067.front.sepia.ceph.com:9095"}]: dispatch 2024-04-22T23:16:47.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:47 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:47.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:47 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:16:48.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:48 smithi067 ceph-mon[26453]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T23:16:48.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:48 smithi067 ceph-mon[26453]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi067.front.sepia.ceph.com:9093"}]: dispatch 2024-04-22T23:16:48.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:48 smithi067 ceph-mon[26453]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T23:16:48.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:48 smithi067 ceph-mon[26453]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi067.front.sepia.ceph.com:3000"}]: dispatch 2024-04-22T23:16:48.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:48 smithi067 ceph-mon[26453]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T23:16:48.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:48 smithi067 ceph-mon[26453]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi067.front.sepia.ceph.com:9095"}]: dispatch 2024-04-22T23:16:48.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:48 smithi067 ceph-mon[26453]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:48.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:48 smithi134 ceph-mon[34609]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T23:16:48.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:48 smithi134 ceph-mon[34609]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi067.front.sepia.ceph.com:9093"}]: dispatch 2024-04-22T23:16:48.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:48 smithi134 ceph-mon[34609]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T23:16:48.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:48 smithi134 ceph-mon[34609]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi067.front.sepia.ceph.com:3000"}]: dispatch 2024-04-22T23:16:48.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:48 smithi134 ceph-mon[34609]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T23:16:48.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:48 smithi134 ceph-mon[34609]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi067.front.sepia.ceph.com:9095"}]: dispatch 2024-04-22T23:16:48.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:48 smithi134 ceph-mon[34609]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:49.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:49.560+0000 7f4e89ffb700 1 -- 172.21.15.67:0/3145491474 <== mgr.14215 v2:172.21.15.67:6800/2589747850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f4e8c068c90 con 0x7f4e7c06c470 2024-04-22T23:16:49.563 INFO:teuthology.orchestra.run.smithi067.stdout:Created osd(s) 0 on host 'smithi067' 2024-04-22T23:16:49.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:49.562+0000 7f4e92f3c700 1 -- 172.21.15.67:0/3145491474 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f4e7c06c470 msgr2=0x7f4e7c06e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:16:49.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:49.562+0000 7f4e92f3c700 1 --2- 172.21.15.67:0/3145491474 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f4e7c06c470 0x7f4e7c06e930 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4e7800b5c0 tx=0x7f4e78005fd0 comp rx=0 tx=0).stop 2024-04-22T23:16:49.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:49.562+0000 7f4e92f3c700 1 -- 172.21.15.67:0/3145491474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8c0ffc10 msgr2=0x7f4e8c196480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:16:49.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:49.562+0000 7f4e92f3c700 1 --2- 172.21.15.67:0/3145491474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8c0ffc10 0x7f4e8c196480 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4e8000c8f0 tx=0x7f4e8000ccb0 comp rx=0 tx=0).stop 2024-04-22T23:16:49.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:49.562+0000 7f4e92f3c700 1 -- 172.21.15.67:0/3145491474 shutdown_connections 2024-04-22T23:16:49.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:49.562+0000 7f4e92f3c700 1 --2- 172.21.15.67:0/3145491474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8c0ffc10 0x7f4e8c196480 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:16:49.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:49.562+0000 7f4e92f3c700 1 --2- 172.21.15.67:0/3145491474 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f4e7c06c470 0x7f4e7c06e930 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:16:49.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:49.562+0000 7f4e92f3c700 1 --2- 172.21.15.67:0/3145491474 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e8c100e10 0x7f4e8c1969c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:16:49.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:49.562+0000 7f4e92f3c700 1 -- 172.21.15.67:0/3145491474 >> 172.21.15.67:0/3145491474 conn(0x7f4e8c0fb190 msgr2=0x7f4e8c068580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:16:49.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:49.562+0000 7f4e92f3c700 1 -- 172.21.15.67:0/3145491474 shutdown_connections 2024-04-22T23:16:49.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:49.562+0000 7f4e92f3c700 1 -- 172.21.15.67:0/3145491474 wait complete. 2024-04-22T23:16:50.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:49 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:50.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:49 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:50.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:49 smithi067 ceph-mon[26453]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:50.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:49 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:50.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:49 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:50.075 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:49 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:50.075 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:49 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:50.075 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:49 smithi134 ceph-mon[34609]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:50.075 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:49 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:50.075 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:49 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:50.093 DEBUG:teuthology.orchestra.run.smithi067:osd.0> sudo journalctl -f -n 0 -u ceph-d46c346e-00fd-11ef-bc93-c7b262605968@osd.0.service 2024-04-22T23:16:50.096 INFO:tasks.cephadm:Deploying osd.1 on smithi067 with /dev/vg_nvme/lv_3... 2024-04-22T23:16:50.096 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-22T23:16:50.350 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:16:50 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0[45701]: 2024-04-22T23:16:50.179+0000 7feb238e0640 -1 osd.0 0 log_to_monitors true 2024-04-22T23:16:50.888 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:16:51.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:50 smithi134 ceph-mon[34609]: from='osd.0 [v2:172.21.15.67:6802/268472413,v1:172.21.15.67:6803/268472413]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T23:16:51.251 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:50 smithi067 ceph-mon[26453]: from='osd.0 [v2:172.21.15.67:6802/268472413,v1:172.21.15.67:6803/268472413]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T23:16:52.295 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:16:52 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0[45701]: 2024-04-22T23:16:52.007+0000 7feb18743700 -1 osd.0 0 waiting for initial osdmap 2024-04-22T23:16:52.295 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:16:52 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0[45701]: 2024-04-22T23:16:52.014+0000 7feb10d31700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T23:16:52.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:52 smithi067 ceph-mon[26453]: from='osd.0 [v2:172.21.15.67:6802/268472413,v1:172.21.15.67:6803/268472413]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T23:16:52.296 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:52 smithi067 ceph-mon[26453]: osdmap e7: 1 total, 0 up, 1 in 2024-04-22T23:16:52.296 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:52 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T23:16:52.296 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:52 smithi067 ceph-mon[26453]: from='osd.0 [v2:172.21.15.67:6802/268472413,v1:172.21.15.67:6803/268472413]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-04-22T23:16:52.296 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:52 smithi067 ceph-mon[26453]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:52.362 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:52 smithi134 ceph-mon[34609]: from='osd.0 [v2:172.21.15.67:6802/268472413,v1:172.21.15.67:6803/268472413]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T23:16:52.362 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:52 smithi134 ceph-mon[34609]: osdmap e7: 1 total, 0 up, 1 in 2024-04-22T23:16:52.363 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:52 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T23:16:52.363 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:52 smithi134 ceph-mon[34609]: from='osd.0 [v2:172.21.15.67:6802/268472413,v1:172.21.15.67:6803/268472413]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-04-22T23:16:52.363 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:52 smithi134 ceph-mon[34609]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:53.063 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:53 smithi067 ceph-mon[26453]: from='osd.0 [v2:172.21.15.67:6802/268472413,v1:172.21.15.67:6803/268472413]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-04-22T23:16:53.063 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:53 smithi067 ceph-mon[26453]: osdmap e8: 1 total, 0 up, 1 in 2024-04-22T23:16:53.063 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:53 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T23:16:53.063 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:53 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T23:16:53.063 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:53 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:16:53.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:53 smithi134 ceph-mon[34609]: from='osd.0 [v2:172.21.15.67:6802/268472413,v1:172.21.15.67:6803/268472413]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-04-22T23:16:53.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:53 smithi134 ceph-mon[34609]: osdmap e8: 1 total, 0 up, 1 in 2024-04-22T23:16:53.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:53 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T23:16:53.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:53 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T23:16:53.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:53 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:16:54.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:54 smithi067 ceph-mon[26453]: purged_snaps scrub starts 2024-04-22T23:16:54.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:54 smithi067 ceph-mon[26453]: purged_snaps scrub ok 2024-04-22T23:16:54.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:54 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T23:16:54.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:54 smithi067 ceph-mon[26453]: osd.0 [v2:172.21.15.67:6802/268472413,v1:172.21.15.67:6803/268472413] boot 2024-04-22T23:16:54.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:54 smithi067 ceph-mon[26453]: osdmap e9: 1 total, 1 up, 1 in 2024-04-22T23:16:54.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:54 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T23:16:54.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:54 smithi067 ceph-mon[26453]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:54.457 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-22T23:16:54.457 INFO:teuthology.orchestra.run.smithi067.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-22T23:16:54.457 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 10+0 records in 2024-04-22T23:16:54.457 INFO:teuthology.orchestra.run.smithi067.stderr:10+0 records out 2024-04-22T23:16:54.457 INFO:teuthology.orchestra.run.smithi067.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0119219 s, 880 MB/s 2024-04-22T23:16:54.457 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 2024-04-22T23:16:54.457 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping successful for: 2024-04-22T23:16:54.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:54 smithi134 ceph-mon[34609]: purged_snaps scrub starts 2024-04-22T23:16:54.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:54 smithi134 ceph-mon[34609]: purged_snaps scrub ok 2024-04-22T23:16:54.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:54 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T23:16:54.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:54 smithi134 ceph-mon[34609]: osd.0 [v2:172.21.15.67:6802/268472413,v1:172.21.15.67:6803/268472413] boot 2024-04-22T23:16:54.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:54 smithi134 ceph-mon[34609]: osdmap e9: 1 total, 1 up, 1 in 2024-04-22T23:16:54.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:54 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T23:16:54.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:54 smithi134 ceph-mon[34609]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T23:16:55.033 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:55 smithi067 ceph-mon[26453]: osdmap e10: 1 total, 1 up, 1 in 2024-04-22T23:16:55.033 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:55 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:55.033 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:55 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:55.054 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi067:vg_nvme/lv_3 2024-04-22T23:16:55.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:55 smithi134 ceph-mon[34609]: osdmap e10: 1 total, 1 up, 1 in 2024-04-22T23:16:55.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:55 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:55.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:55 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:55.776 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:16:56.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:56 smithi067 ceph-mon[26453]: pgmap v37: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T23:16:56.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:56 smithi134 ceph-mon[34609]: pgmap v37: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T23:16:58.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:58.023+0000 7fc1d6a9e700 1 -- 172.21.15.67:0/2284395364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1d01040d0 msgr2=0x7fc1d01064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:16:58.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:58.023+0000 7fc1d6a9e700 1 --2- 172.21.15.67:0/2284395364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1d01040d0 0x7fc1d01064c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc009910 tx=0x7fc1bc009c20 comp rx=0 tx=0).stop 2024-04-22T23:16:58.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:58.023+0000 7fc1d6a9e700 1 -- 172.21.15.67:0/2284395364 shutdown_connections 2024-04-22T23:16:58.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:58.023+0000 7fc1d6a9e700 1 --2- 172.21.15.67:0/2284395364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1d01040d0 0x7fc1d01064c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:16:58.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:58.023+0000 7fc1d6a9e700 1 --2- 172.21.15.67:0/2284395364 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1d01017a0 0x7fc1d0103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:16:58.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:58.023+0000 7fc1d6a9e700 1 -- 172.21.15.67:0/2284395364 >> 172.21.15.67:0/2284395364 conn(0x7fc1d00fb190 msgr2=0x7fc1d00fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:16:58.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:58.023+0000 7fc1d6a9e700 1 -- 172.21.15.67:0/2284395364 shutdown_connections 2024-04-22T23:16:58.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:58.023+0000 7fc1d6a9e700 1 -- 172.21.15.67:0/2284395364 wait complete. 2024-04-22T23:16:58.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:58.023+0000 7fc1d6a9e700 1 Processor -- start 2024-04-22T23:16:58.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:58.023+0000 7fc1d6a9e700 1 -- start start 2024-04-22T23:16:58.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:58.024+0000 7fc1d6a9e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1d01017a0 0x7fc1d01964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:16:58.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:58.024+0000 7fc1d6a9e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1d01040d0 0x7fc1d0196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:16:58.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:58.024+0000 7fc1d6a9e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d0197030 con 0x7fc1d01040d0 2024-04-22T23:16:58.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:58.024+0000 7fc1d6a9e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d0197170 con 0x7fc1d01017a0 2024-04-22T23:16:58.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:58.024+0000 7fc1cffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1d01040d0 0x7fc1d0196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:16:58.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:58.024+0000 7fc1cffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1d01040d0 0x7fc1d0196a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49828/0 (socket says 172.21.15.67:49828) 2024-04-22T23:16:58.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:58.024+0000 7fc1cffff700 1 -- 172.21.15.67:0/1507491378 learned_addr learned my addr 172.21.15.67:0/1507491378 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:16:58.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:58.024+0000 7fc1d483a700 1 --2- 172.21.15.67:0/1507491378 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1d01017a0 0x7fc1d01964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:16:58.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:58.024+0000 7fc1cffff700 1 -- 172.21.15.67:0/1507491378 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1d01017a0 msgr2=0x7fc1d01964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:16:58.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:58.024+0000 7fc1cffff700 1 --2- 172.21.15.67:0/1507491378 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1d01017a0 0x7fc1d01964d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:16:58.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:58.024+0000 7fc1cffff700 1 -- 172.21.15.67:0/1507491378 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1bc0095f0 con 0x7fc1d01040d0 2024-04-22T23:16:58.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:58.024+0000 7fc1cffff700 1 --2- 172.21.15.67:0/1507491378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1d01040d0 0x7fc1d0196a10 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc00bd70 tx=0x7fc1bc00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:16:58.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:58.024+0000 7fc1cdffb700 1 -- 172.21.15.67:0/1507491378 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1bc01d070 con 0x7fc1d01040d0 2024-04-22T23:16:58.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:58.025+0000 7fc1d6a9e700 1 -- 172.21.15.67:0/1507491378 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1d019bbc0 con 0x7fc1d01040d0 2024-04-22T23:16:58.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:58.025+0000 7fc1cdffb700 1 -- 172.21.15.67:0/1507491378 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc1bc022470 con 0x7fc1d01040d0 2024-04-22T23:16:58.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:58.025+0000 7fc1cdffb700 1 -- 172.21.15.67:0/1507491378 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1bc00f670 con 0x7fc1d01040d0 2024-04-22T23:16:58.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:58.025+0000 7fc1d6a9e700 1 -- 172.21.15.67:0/1507491378 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1d019c0b0 con 0x7fc1d01040d0 2024-04-22T23:16:58.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:58.026+0000 7fc1d6a9e700 1 -- 172.21.15.67:0/1507491378 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1d0190690 con 0x7fc1d01040d0 2024-04-22T23:16:58.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:58.026+0000 7fc1cdffb700 1 -- 172.21.15.67:0/1507491378 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 90256+0+0 (secure 0 0 0) 0x7fc1bc022a50 con 0x7fc1d01040d0 2024-04-22T23:16:58.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:58.027+0000 7fc1cdffb700 1 --2- 172.21.15.67:0/1507491378 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fc1c006c460 0x7fc1c006e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:16:58.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:58.027+0000 7fc1cdffb700 1 -- 172.21.15.67:0/1507491378 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1982+0+0 (secure 0 0 0) 0x7fc1bc08bfc0 con 0x7fc1d01040d0 2024-04-22T23:16:58.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:58.027+0000 7fc1d483a700 1 --2- 172.21.15.67:0/1507491378 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fc1c006c460 0x7fc1c006e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:16:58.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:58.028+0000 7fc1d483a700 1 --2- 172.21.15.67:0/1507491378 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fc1c006c460 0x7fc1c006e920 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4006fd0 tx=0x7fc1c4008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:16:58.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:58.033+0000 7fc1cdffb700 1 -- 172.21.15.67:0/1507491378 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc1bc0bf0e0 con 0x7fc1d01040d0 2024-04-22T23:16:58.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:16:58.245+0000 7fc1d6a9e700 1 -- 172.21.15.67:0/1507491378 --> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fc1d00611d0 con 0x7fc1c006c460 2024-04-22T23:16:58.513 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:58 smithi067 ceph-mon[26453]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T23:16:58.513 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:58 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T23:16:58.513 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:58 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T23:16:58.513 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:58 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:16:58.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:58 smithi134 ceph-mon[34609]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T23:16:58.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:58 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T23:16:58.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:58 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T23:16:58.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:58 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:16:59.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:59 smithi134 ceph-mon[34609]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:16:59.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:59 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:59.746 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:59 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:59.746 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:59 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:16:59.746 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:59 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:59.746 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:59 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:16:59.746 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:59 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:16:59.746 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:59 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:59.746 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:16:59 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:16:59.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:59 smithi067 ceph-mon[26453]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:16:59.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:59 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:59.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:59 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:59.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:59 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:16:59.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:59 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:59.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:59 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:16:59.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:59 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:16:59.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:59 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:16:59.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:16:59 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:17:00.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:00 smithi134 ceph-mon[34609]: Detected new or changed devices on smithi067 2024-04-22T23:17:00.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:00 smithi134 ceph-mon[34609]: Adjusting osd_memory_target on smithi067 to 11840M 2024-04-22T23:17:00.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:00 smithi134 ceph-mon[34609]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T23:17:00.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:00 smithi067 ceph-mon[26453]: Detected new or changed devices on smithi067 2024-04-22T23:17:00.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:00 smithi067 ceph-mon[26453]: Adjusting osd_memory_target on smithi067 to 11840M 2024-04-22T23:17:00.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:00 smithi067 ceph-mon[26453]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T23:17:02.216 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:01 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:02.216 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:01 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:02.216 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:01 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/158028829' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8be83014-01af-44d3-ab2f-b27840506e7f"}]: dispatch 2024-04-22T23:17:02.216 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:01 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/158028829' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8be83014-01af-44d3-ab2f-b27840506e7f"}]': finished 2024-04-22T23:17:02.216 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:01 smithi067 ceph-mon[26453]: osdmap e11: 2 total, 1 up, 2 in 2024-04-22T23:17:02.216 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:01 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T23:17:02.216 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:01 smithi067 ceph-mon[26453]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T23:17:02.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:01 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:02.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:01 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:02.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:01 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/158028829' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8be83014-01af-44d3-ab2f-b27840506e7f"}]: dispatch 2024-04-22T23:17:02.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:01 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/158028829' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8be83014-01af-44d3-ab2f-b27840506e7f"}]': finished 2024-04-22T23:17:02.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:01 smithi134 ceph-mon[34609]: osdmap e11: 2 total, 1 up, 2 in 2024-04-22T23:17:02.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:01 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T23:17:02.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:01 smithi134 ceph-mon[34609]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T23:17:03.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:02 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/3500211189' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T23:17:03.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:02 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/3500211189' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T23:17:04.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:03 smithi134 ceph-mon[34609]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T23:17:04.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:03 smithi067 ceph-mon[26453]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T23:17:05.242 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:04 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:05.242 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:04 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:05.242 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:04 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:17:05.242 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:04 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:17:05.242 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:04 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:05.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:05.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:05.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:17:05.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:17:05.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:06.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:05 smithi067 ceph-mon[26453]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T23:17:06.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:05 smithi134 ceph-mon[34609]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T23:17:07.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:07 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:17:07.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:07 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:17:08.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:08 smithi067 ceph-mon[26453]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T23:17:08.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:08 smithi134 ceph-mon[34609]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T23:17:09.698 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:09 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T23:17:09.698 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:09 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:17:09.895 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:09 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T23:17:09.896 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:09 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:17:10.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:10 smithi067 ceph-mon[26453]: Deploying daemon osd.1 on smithi067 2024-04-22T23:17:10.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:10 smithi067 ceph-mon[26453]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T23:17:10.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:10 smithi134 ceph-mon[34609]: Deploying daemon osd.1 on smithi067 2024-04-22T23:17:10.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:10 smithi134 ceph-mon[34609]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T23:17:11.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:11 smithi067 ceph-mon[26453]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T23:17:11.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:11 smithi134 ceph-mon[34609]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T23:17:14.552 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:14 smithi067 ceph-mon[26453]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T23:17:14.552 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:14 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:14.552 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:14 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:14.552 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:14 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:17:14.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:14 smithi134 ceph-mon[34609]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T23:17:14.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:14 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:14.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:14 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:14.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:14 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:17:16.258 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:15 smithi067 ceph-mon[26453]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T23:17:16.259 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:16 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:16.259 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:16 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:16.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:16 smithi134 ceph-mon[34609]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T23:17:16.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:16 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:16.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:16 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:16.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:16.846+0000 7fc1cdffb700 1 -- 172.21.15.67:0/1507491378 <== mgr.14215 v2:172.21.15.67:6800/2589747850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc1d00611d0 con 0x7fc1c006c460 2024-04-22T23:17:16.848 INFO:teuthology.orchestra.run.smithi067.stdout:Created osd(s) 1 on host 'smithi067' 2024-04-22T23:17:16.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:16.848+0000 7fc1d6a9e700 1 -- 172.21.15.67:0/1507491378 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fc1c006c460 msgr2=0x7fc1c006e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:17:16.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:16.848+0000 7fc1d6a9e700 1 --2- 172.21.15.67:0/1507491378 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fc1c006c460 0x7fc1c006e920 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4006fd0 tx=0x7fc1c4008040 comp rx=0 tx=0).stop 2024-04-22T23:17:16.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:16.848+0000 7fc1d6a9e700 1 -- 172.21.15.67:0/1507491378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1d01040d0 msgr2=0x7fc1d0196a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:17:16.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:16.848+0000 7fc1d6a9e700 1 --2- 172.21.15.67:0/1507491378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1d01040d0 0x7fc1d0196a10 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc00bd70 tx=0x7fc1bc00be50 comp rx=0 tx=0).stop 2024-04-22T23:17:16.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:16.849+0000 7fc1d6a9e700 1 -- 172.21.15.67:0/1507491378 shutdown_connections 2024-04-22T23:17:16.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:16.849+0000 7fc1d6a9e700 1 --2- 172.21.15.67:0/1507491378 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fc1c006c460 0x7fc1c006e920 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:17:16.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:16.849+0000 7fc1d6a9e700 1 --2- 172.21.15.67:0/1507491378 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1d01017a0 0x7fc1d01964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:17:16.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:16.849+0000 7fc1d6a9e700 1 --2- 172.21.15.67:0/1507491378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1d01040d0 0x7fc1d0196a10 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:17:16.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:16.849+0000 7fc1d6a9e700 1 -- 172.21.15.67:0/1507491378 >> 172.21.15.67:0/1507491378 conn(0x7fc1d00fb190 msgr2=0x7fc1d00fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:17:16.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:16.849+0000 7fc1d6a9e700 1 -- 172.21.15.67:0/1507491378 shutdown_connections 2024-04-22T23:17:16.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:16.849+0000 7fc1d6a9e700 1 -- 172.21.15.67:0/1507491378 wait complete. 2024-04-22T23:17:17.525 DEBUG:teuthology.orchestra.run.smithi067:osd.1> sudo journalctl -f -n 0 -u ceph-d46c346e-00fd-11ef-bc93-c7b262605968@osd.1.service 2024-04-22T23:17:17.528 INFO:tasks.cephadm:Deploying osd.2 on smithi067 with /dev/vg_nvme/lv_2... 2024-04-22T23:17:17.528 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-22T23:17:17.971 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:17 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:17.972 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:17 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:17.972 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:17 smithi067 ceph-mon[26453]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T23:17:17.972 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:17 smithi067 ceph-mon[26453]: from='osd.1 [v2:172.21.15.67:6810/3382820294,v1:172.21.15.67:6811/3382820294]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T23:17:17.972 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:17:17 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1[53145]: 2024-04-22T23:17:17.687+0000 7f2c75e40640 -1 osd.1 0 log_to_monitors true 2024-04-22T23:17:18.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:17 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:18.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:17 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:18.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:17 smithi134 ceph-mon[34609]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T23:17:18.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:17 smithi134 ceph-mon[34609]: from='osd.1 [v2:172.21.15.67:6810/3382820294,v1:172.21.15.67:6811/3382820294]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T23:17:18.349 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:17:19.092 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:18 smithi067 ceph-mon[26453]: from='osd.1 [v2:172.21.15.67:6810/3382820294,v1:172.21.15.67:6811/3382820294]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T23:17:19.093 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:18 smithi067 ceph-mon[26453]: osdmap e12: 2 total, 1 up, 2 in 2024-04-22T23:17:19.093 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:18 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T23:17:19.093 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:18 smithi067 ceph-mon[26453]: from='osd.1 [v2:172.21.15.67:6810/3382820294,v1:172.21.15.67:6811/3382820294]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-04-22T23:17:19.093 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:17:18 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1[53145]: 2024-04-22T23:17:18.836+0000 7f2c6aca3700 -1 osd.1 0 waiting for initial osdmap 2024-04-22T23:17:19.093 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:17:18 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1[53145]: 2024-04-22T23:17:18.841+0000 7f2c65295700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T23:17:19.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:18 smithi134 ceph-mon[34609]: from='osd.1 [v2:172.21.15.67:6810/3382820294,v1:172.21.15.67:6811/3382820294]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T23:17:19.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:18 smithi134 ceph-mon[34609]: osdmap e12: 2 total, 1 up, 2 in 2024-04-22T23:17:19.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:18 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T23:17:19.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:18 smithi134 ceph-mon[34609]: from='osd.1 [v2:172.21.15.67:6810/3382820294,v1:172.21.15.67:6811/3382820294]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-04-22T23:17:19.981 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:19 smithi067 ceph-mon[26453]: from='osd.1 [v2:172.21.15.67:6810/3382820294,v1:172.21.15.67:6811/3382820294]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-04-22T23:17:19.981 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:19 smithi067 ceph-mon[26453]: osdmap e13: 2 total, 1 up, 2 in 2024-04-22T23:17:19.981 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:19 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T23:17:19.981 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:19 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T23:17:19.981 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:19 smithi067 ceph-mon[26453]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T23:17:19.981 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:19 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T23:17:20.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:19 smithi134 ceph-mon[34609]: from='osd.1 [v2:172.21.15.67:6810/3382820294,v1:172.21.15.67:6811/3382820294]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-04-22T23:17:20.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:19 smithi134 ceph-mon[34609]: osdmap e13: 2 total, 1 up, 2 in 2024-04-22T23:17:20.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:19 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T23:17:20.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:19 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T23:17:20.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:19 smithi134 ceph-mon[34609]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-22T23:17:20.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:19 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T23:17:20.786 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-22T23:17:20.786 INFO:teuthology.orchestra.run.smithi067.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-22T23:17:20.786 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 10+0 records in 2024-04-22T23:17:20.786 INFO:teuthology.orchestra.run.smithi067.stderr:10+0 records out 2024-04-22T23:17:20.786 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126557 s, 829 MB/s 2024-04-22T23:17:20.786 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping successful for: 2024-04-22T23:17:21.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:20 smithi067 ceph-mon[26453]: purged_snaps scrub starts 2024-04-22T23:17:21.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:20 smithi067 ceph-mon[26453]: purged_snaps scrub ok 2024-04-22T23:17:21.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:20 smithi067 ceph-mon[26453]: osd.1 [v2:172.21.15.67:6810/3382820294,v1:172.21.15.67:6811/3382820294] boot 2024-04-22T23:17:21.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:20 smithi067 ceph-mon[26453]: osdmap e14: 2 total, 2 up, 2 in 2024-04-22T23:17:21.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:20 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T23:17:21.144 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:20 smithi134 ceph-mon[34609]: purged_snaps scrub starts 2024-04-22T23:17:21.144 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:20 smithi134 ceph-mon[34609]: purged_snaps scrub ok 2024-04-22T23:17:21.144 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:20 smithi134 ceph-mon[34609]: osd.1 [v2:172.21.15.67:6810/3382820294,v1:172.21.15.67:6811/3382820294] boot 2024-04-22T23:17:21.144 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:20 smithi134 ceph-mon[34609]: osdmap e14: 2 total, 2 up, 2 in 2024-04-22T23:17:21.144 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:20 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T23:17:21.755 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi067:vg_nvme/lv_2 2024-04-22T23:17:22.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:22 smithi067 ceph-mon[26453]: osdmap e15: 2 total, 2 up, 2 in 2024-04-22T23:17:22.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:22 smithi067 ceph-mon[26453]: pgmap v55: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T23:17:22.377 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:17:22.403 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:22 smithi134 ceph-mon[34609]: osdmap e15: 2 total, 2 up, 2 in 2024-04-22T23:17:22.403 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:22 smithi134 ceph-mon[34609]: pgmap v55: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T23:17:23.175 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:23 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:17:23.175 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:23 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:23.175 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:23 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:23.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:23 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:17:23.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:23 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:23.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:23 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:23.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:23.928+0000 7f3f01137700 1 -- 172.21.15.67:0/626671641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3efc0fef80 msgr2=0x7f3efc0ff3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:17:23.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:23.928+0000 7f3f01137700 1 --2- 172.21.15.67:0/626671641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3efc0fef80 0x7f3efc0ff3a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3eec009910 tx=0x7f3eec009c20 comp rx=0 tx=0).stop 2024-04-22T23:17:23.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:23.928+0000 7f3f01137700 1 -- 172.21.15.67:0/626671641 shutdown_connections 2024-04-22T23:17:23.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:23.928+0000 7f3f01137700 1 --2- 172.21.15.67:0/626671641 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3efc102000 0x7f3efc102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:17:23.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:23.928+0000 7f3f01137700 1 --2- 172.21.15.67:0/626671641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3efc0fef80 0x7f3efc0ff3a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:17:23.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:23.928+0000 7f3f01137700 1 -- 172.21.15.67:0/626671641 >> 172.21.15.67:0/626671641 conn(0x7f3efc0fa8c0 msgr2=0x7f3efc0fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:17:23.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:23.928+0000 7f3f01137700 1 -- 172.21.15.67:0/626671641 shutdown_connections 2024-04-22T23:17:23.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:23.928+0000 7f3f01137700 1 -- 172.21.15.67:0/626671641 wait complete. 2024-04-22T23:17:23.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:23.929+0000 7f3f01137700 1 Processor -- start 2024-04-22T23:17:23.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:23.930+0000 7f3f01137700 1 -- start start 2024-04-22T23:17:23.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:23.930+0000 7f3f01137700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3efc0fef80 0x7f3efc101960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:17:23.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:23.930+0000 7f3f01137700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3efc102000 0x7f3efc0fffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:17:23.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:23.930+0000 7f3f01137700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3efc1004f0 con 0x7f3efc102000 2024-04-22T23:17:23.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:23.930+0000 7f3f01137700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3efc100630 con 0x7f3efc0fef80 2024-04-22T23:17:23.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:23.930+0000 7f3efa59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3efc102000 0x7f3efc0fffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:17:23.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:23.930+0000 7f3efa59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3efc102000 0x7f3efc0fffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36692/0 (socket says 172.21.15.67:36692) 2024-04-22T23:17:23.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:23.930+0000 7f3efa59c700 1 -- 172.21.15.67:0/1496977677 learned_addr learned my addr 172.21.15.67:0/1496977677 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:17:23.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:23.930+0000 7f3efad9d700 1 --2- 172.21.15.67:0/1496977677 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3efc0fef80 0x7f3efc101960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:17:23.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:23.930+0000 7f3efa59c700 1 -- 172.21.15.67:0/1496977677 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3efc0fef80 msgr2=0x7f3efc101960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:17:23.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:23.930+0000 7f3efa59c700 1 --2- 172.21.15.67:0/1496977677 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3efc0fef80 0x7f3efc101960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:17:23.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:23.930+0000 7f3efa59c700 1 -- 172.21.15.67:0/1496977677 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eec0095f0 con 0x7f3efc102000 2024-04-22T23:17:23.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:23.931+0000 7f3efa59c700 1 --2- 172.21.15.67:0/1496977677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3efc102000 0x7f3efc0fffb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3ef000e920 tx=0x7f3ef000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:17:23.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:23.931+0000 7f3ee3fff700 1 -- 172.21.15.67:0/1496977677 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ef000c9d0 con 0x7f3efc102000 2024-04-22T23:17:23.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:23.931+0000 7f3ee3fff700 1 -- 172.21.15.67:0/1496977677 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3ef0010430 con 0x7f3efc102000 2024-04-22T23:17:23.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:23.931+0000 7f3ee3fff700 1 -- 172.21.15.67:0/1496977677 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ef0018650 con 0x7f3efc102000 2024-04-22T23:17:23.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:23.931+0000 7f3f01137700 1 -- 172.21.15.67:0/1496977677 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3efc100910 con 0x7f3efc102000 2024-04-22T23:17:23.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:23.931+0000 7f3f01137700 1 -- 172.21.15.67:0/1496977677 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3efc100da0 con 0x7f3efc102000 2024-04-22T23:17:23.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:23.932+0000 7f3f01137700 1 -- 172.21.15.67:0/1496977677 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3efc109800 con 0x7f3efc102000 2024-04-22T23:17:23.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:23.933+0000 7f3ee3fff700 1 -- 172.21.15.67:0/1496977677 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) v1 ==== 90256+0+0 (secure 0 0 0) 0x7f3ef00105a0 con 0x7f3efc102000 2024-04-22T23:17:23.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:23.933+0000 7f3ee3fff700 1 --2- 172.21.15.67:0/1496977677 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f3ee406c460 0x7f3ee406e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:17:23.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:23.933+0000 7f3ee3fff700 1 -- 172.21.15.67:0/1496977677 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2414+0+0 (secure 0 0 0) 0x7f3ef0014070 con 0x7f3efc102000 2024-04-22T23:17:23.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:23.933+0000 7f3efad9d700 1 --2- 172.21.15.67:0/1496977677 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f3ee406c460 0x7f3ee406e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:17:23.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:23.934+0000 7f3efad9d700 1 --2- 172.21.15.67:0/1496977677 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f3ee406c460 0x7f3ee406e920 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3eec00b6f0 tx=0x7f3eec005940 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:17:23.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:23.938+0000 7f3ee3fff700 1 -- 172.21.15.67:0/1496977677 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f3ef00520e0 con 0x7f3efc102000 2024-04-22T23:17:24.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:24.148+0000 7f3f01137700 1 -- 172.21.15.67:0/1496977677 --> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f3efc106950 con 0x7f3ee406c460 2024-04-22T23:17:24.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:24 smithi067 ceph-mon[26453]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T23:17:24.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:24 smithi134 ceph-mon[34609]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T23:17:25.127 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:25 smithi067 ceph-mon[26453]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:17:25.127 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:25 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T23:17:25.127 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:25 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T23:17:25.127 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:25 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:17:25.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:25 smithi134 ceph-mon[34609]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:17:25.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:25 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T23:17:25.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:25 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T23:17:25.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:25 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:17:26.262 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:26 smithi067 ceph-mon[26453]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T23:17:26.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:26 smithi134 ceph-mon[34609]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T23:17:27.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:27 smithi134 ceph-mon[34609]: Detected new or changed devices on smithi067 2024-04-22T23:17:27.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:27 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:27.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:27 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:27.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:27 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:17:27.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:27 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:17:27.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:27 smithi134 ceph-mon[34609]: Adjusting osd_memory_target on smithi067 to 5920M 2024-04-22T23:17:27.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:27 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:27.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:27 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:17:27.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:27 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:17:27.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:27 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:27.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:27 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:17:27.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:27 smithi134 ceph-mon[34609]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T23:17:27.847 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:27 smithi067 ceph-mon[26453]: Detected new or changed devices on smithi067 2024-04-22T23:17:27.847 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:27 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:27.848 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:27 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:27.848 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:27 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:17:27.848 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:27 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:17:27.848 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:27 smithi067 ceph-mon[26453]: Adjusting osd_memory_target on smithi067 to 5920M 2024-04-22T23:17:27.848 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:27 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:27.848 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:27 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:17:27.848 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:27 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:17:27.848 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:27 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:27.848 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:27 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:17:27.848 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:27 smithi067 ceph-mon[26453]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T23:17:28.852 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:28 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/3937803386' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ab1cbcda-14dd-4c24-876b-25ad5d4d8563"}]: dispatch 2024-04-22T23:17:28.852 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:28 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/3937803386' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ab1cbcda-14dd-4c24-876b-25ad5d4d8563"}]': finished 2024-04-22T23:17:28.852 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:28 smithi067 ceph-mon[26453]: osdmap e16: 3 total, 2 up, 3 in 2024-04-22T23:17:28.853 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:28 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T23:17:28.853 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:28 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:28.853 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:28 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:28.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:28 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/3937803386' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ab1cbcda-14dd-4c24-876b-25ad5d4d8563"}]: dispatch 2024-04-22T23:17:28.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:28 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/3937803386' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ab1cbcda-14dd-4c24-876b-25ad5d4d8563"}]': finished 2024-04-22T23:17:28.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:28 smithi134 ceph-mon[34609]: osdmap e16: 3 total, 2 up, 3 in 2024-04-22T23:17:28.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:28 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T23:17:28.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:28 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:28.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:28 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:29.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:29 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/2436201279' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T23:17:29.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:29 smithi134 ceph-mon[34609]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T23:17:30.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:29 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/2436201279' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T23:17:30.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:29 smithi067 ceph-mon[26453]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T23:17:32.665 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:32 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:32.665 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:32 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:32.665 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:32 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:17:32.665 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:32 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:17:32.666 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:32 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:32.666 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:32 smithi067 ceph-mon[26453]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T23:17:32.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:32 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:32.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:32 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:32.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:32 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:17:32.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:32 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:17:32.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:32 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:32.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:32 smithi134 ceph-mon[34609]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T23:17:34.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:34 smithi067 ceph-mon[26453]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T23:17:34.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:34 smithi134 ceph-mon[34609]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T23:17:36.147 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:36 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T23:17:36.147 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:36 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:17:36.148 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:36 smithi067 ceph-mon[26453]: Deploying daemon osd.2 on smithi067 2024-04-22T23:17:36.148 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:36 smithi067 ceph-mon[26453]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T23:17:36.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:36 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T23:17:36.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:36 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:17:36.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:36 smithi134 ceph-mon[34609]: Deploying daemon osd.2 on smithi067 2024-04-22T23:17:36.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:36 smithi134 ceph-mon[34609]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T23:17:37.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:37 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:17:37.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:37 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:17:38.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:38 smithi067 ceph-mon[26453]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T23:17:38.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:38 smithi134 ceph-mon[34609]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T23:17:40.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:40 smithi067 ceph-mon[26453]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T23:17:40.826 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:40 smithi134 ceph-mon[34609]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T23:17:41.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:41 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:41.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:41 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:41.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:41 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:17:41.970 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:41 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:41.970 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:41 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:41.970 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:41 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:17:42.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:42 smithi067 ceph-mon[26453]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T23:17:42.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:42 smithi134 ceph-mon[34609]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T23:17:43.629 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:43 smithi067 ceph-mon[26453]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T23:17:43.681 INFO:teuthology.orchestra.run.smithi067.stdout:Created osd(s) 2 on host 'smithi067' 2024-04-22T23:17:43.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:43.679+0000 7f3ee3fff700 1 -- 172.21.15.67:0/1496977677 <== mgr.14215 v2:172.21.15.67:6800/2589747850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3efc106950 con 0x7f3ee406c460 2024-04-22T23:17:43.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:43.683+0000 7f3f01137700 1 -- 172.21.15.67:0/1496977677 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f3ee406c460 msgr2=0x7f3ee406e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:17:43.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:43.683+0000 7f3f01137700 1 --2- 172.21.15.67:0/1496977677 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f3ee406c460 0x7f3ee406e920 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3eec00b6f0 tx=0x7f3eec005940 comp rx=0 tx=0).stop 2024-04-22T23:17:43.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:43.683+0000 7f3f01137700 1 -- 172.21.15.67:0/1496977677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3efc102000 msgr2=0x7f3efc0fffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:17:43.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:43.683+0000 7f3f01137700 1 --2- 172.21.15.67:0/1496977677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3efc102000 0x7f3efc0fffb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3ef000e920 tx=0x7f3ef000ec30 comp rx=0 tx=0).stop 2024-04-22T23:17:43.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:43.683+0000 7f3f01137700 1 -- 172.21.15.67:0/1496977677 shutdown_connections 2024-04-22T23:17:43.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:43.683+0000 7f3f01137700 1 --2- 172.21.15.67:0/1496977677 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f3ee406c460 0x7f3ee406e920 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:17:43.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:43.683+0000 7f3f01137700 1 --2- 172.21.15.67:0/1496977677 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3efc0fef80 0x7f3efc101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:17:43.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:43.683+0000 7f3f01137700 1 --2- 172.21.15.67:0/1496977677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3efc102000 0x7f3efc0fffb0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:17:43.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:43.683+0000 7f3f01137700 1 -- 172.21.15.67:0/1496977677 >> 172.21.15.67:0/1496977677 conn(0x7f3efc0fa8c0 msgr2=0x7f3efc105230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:17:43.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:43.684+0000 7f3f01137700 1 -- 172.21.15.67:0/1496977677 shutdown_connections 2024-04-22T23:17:43.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:17:43.684+0000 7f3f01137700 1 -- 172.21.15.67:0/1496977677 wait complete. 2024-04-22T23:17:43.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:43 smithi134 ceph-mon[34609]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T23:17:44.314 DEBUG:teuthology.orchestra.run.smithi067:osd.2> sudo journalctl -f -n 0 -u ceph-d46c346e-00fd-11ef-bc93-c7b262605968@osd.2.service 2024-04-22T23:17:44.316 INFO:tasks.cephadm:Deploying osd.3 on smithi134 with /dev/vg_nvme/lv_4... 2024-04-22T23:17:44.316 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-22T23:17:44.525 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-22T23:17:44.545 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:17:44 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2[60972]: 2024-04-22T23:17:44.404+0000 7f93aee55640 -1 osd.2 0 log_to_monitors true 2024-04-22T23:17:44.981 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:44 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:44.981 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:44 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:44.981 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:44 smithi134 ceph-mon[34609]: from='osd.2 [v2:172.21.15.67:6818/3835340581,v1:172.21.15.67:6819/3835340581]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T23:17:45.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:44 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:45.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:44 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:45.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:44 smithi067 ceph-mon[26453]: from='osd.2 [v2:172.21.15.67:6818/3835340581,v1:172.21.15.67:6819/3835340581]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T23:17:45.962 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:45 smithi134 ceph-mon[34609]: from='osd.2 [v2:172.21.15.67:6818/3835340581,v1:172.21.15.67:6819/3835340581]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T23:17:45.962 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:45 smithi134 ceph-mon[34609]: osdmap e17: 3 total, 2 up, 3 in 2024-04-22T23:17:45.962 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:45 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T23:17:45.962 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:45 smithi134 ceph-mon[34609]: from='osd.2 [v2:172.21.15.67:6818/3835340581,v1:172.21.15.67:6819/3835340581]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-04-22T23:17:45.963 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:45 smithi134 ceph-mon[34609]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T23:17:46.045 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:17:45 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2[60972]: 2024-04-22T23:17:45.680+0000 7f93a3cb8700 -1 osd.2 0 waiting for initial osdmap 2024-04-22T23:17:46.045 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:17:45 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2[60972]: 2024-04-22T23:17:45.688+0000 7f939eaab700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T23:17:46.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:45 smithi067 ceph-mon[26453]: from='osd.2 [v2:172.21.15.67:6818/3835340581,v1:172.21.15.67:6819/3835340581]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T23:17:46.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:45 smithi067 ceph-mon[26453]: osdmap e17: 3 total, 2 up, 3 in 2024-04-22T23:17:46.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:45 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T23:17:46.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:45 smithi067 ceph-mon[26453]: from='osd.2 [v2:172.21.15.67:6818/3835340581,v1:172.21.15.67:6819/3835340581]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-04-22T23:17:46.046 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:45 smithi067 ceph-mon[26453]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T23:17:46.355 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-22T23:17:46.355 INFO:teuthology.orchestra.run.smithi134.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-22T23:17:46.355 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10+0 records in 2024-04-22T23:17:46.356 INFO:teuthology.orchestra.run.smithi134.stderr:10+0 records out 2024-04-22T23:17:46.356 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0116868 s, 897 MB/s 2024-04-22T23:17:46.356 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping successful for: 2024-04-22T23:17:46.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:46 smithi134 ceph-mon[34609]: from='osd.2 [v2:172.21.15.67:6818/3835340581,v1:172.21.15.67:6819/3835340581]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-04-22T23:17:46.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:46 smithi134 ceph-mon[34609]: osdmap e18: 3 total, 2 up, 3 in 2024-04-22T23:17:46.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:46 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T23:17:46.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:46 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T23:17:47.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:46 smithi067 ceph-mon[26453]: from='osd.2 [v2:172.21.15.67:6818/3835340581,v1:172.21.15.67:6819/3835340581]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-04-22T23:17:47.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:46 smithi067 ceph-mon[26453]: osdmap e18: 3 total, 2 up, 3 in 2024-04-22T23:17:47.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:46 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T23:17:47.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:46 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T23:17:47.084 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi134:vg_nvme/lv_4 2024-04-22T23:17:47.290 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-22T23:17:47.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:47 smithi134 ceph-mon[34609]: purged_snaps scrub starts 2024-04-22T23:17:47.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:47 smithi134 ceph-mon[34609]: purged_snaps scrub ok 2024-04-22T23:17:47.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:47 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T23:17:47.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:47 smithi134 ceph-mon[34609]: osd.2 [v2:172.21.15.67:6818/3835340581,v1:172.21.15.67:6819/3835340581] boot 2024-04-22T23:17:47.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:47 smithi134 ceph-mon[34609]: osdmap e19: 3 total, 3 up, 3 in 2024-04-22T23:17:47.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:47 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T23:17:47.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:47 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:47.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:47 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:47.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:47 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-22T23:17:48.039 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:47 smithi067 ceph-mon[26453]: purged_snaps scrub starts 2024-04-22T23:17:48.039 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:47 smithi067 ceph-mon[26453]: purged_snaps scrub ok 2024-04-22T23:17:48.040 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:47 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T23:17:48.040 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:47 smithi067 ceph-mon[26453]: osd.2 [v2:172.21.15.67:6818/3835340581,v1:172.21.15.67:6819/3835340581] boot 2024-04-22T23:17:48.040 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:47 smithi067 ceph-mon[26453]: osdmap e19: 3 total, 3 up, 3 in 2024-04-22T23:17:48.040 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:47 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T23:17:48.040 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:47 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:48.040 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:47 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:48.040 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:47 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-22T23:17:48.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:17:48.667+0000 7f5644a10700 1 -- 172.21.15.134:0/1503568581 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5640100dc0 msgr2=0x7f5640101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:17:48.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:17:48.667+0000 7f5644a10700 1 --2- 172.21.15.134:0/1503568581 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5640100dc0 0x7f5640101220 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f5628009910 tx=0x7f5628009c20 comp rx=0 tx=0).stop 2024-04-22T23:17:48.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:17:48.667+0000 7f5644a10700 1 -- 172.21.15.134:0/1503568581 shutdown_connections 2024-04-22T23:17:48.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:17:48.667+0000 7f5644a10700 1 --2- 172.21.15.134:0/1503568581 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5640100dc0 0x7f5640101220 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:17:48.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:17:48.667+0000 7f5644a10700 1 --2- 172.21.15.134:0/1503568581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56400ffbc0 0x7f56400fffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:17:48.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:17:48.667+0000 7f5644a10700 1 -- 172.21.15.134:0/1503568581 >> 172.21.15.134:0/1503568581 conn(0x7f56400fb160 msgr2=0x7f56400fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:17:48.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:17:48.667+0000 7f5644a10700 1 -- 172.21.15.134:0/1503568581 shutdown_connections 2024-04-22T23:17:48.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:17:48.667+0000 7f5644a10700 1 -- 172.21.15.134:0/1503568581 wait complete. 2024-04-22T23:17:48.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:17:48.667+0000 7f5644a10700 1 Processor -- start 2024-04-22T23:17:48.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:17:48.668+0000 7f5644a10700 1 -- start start 2024-04-22T23:17:48.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:17:48.668+0000 7f5644a10700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56400ffbc0 0x7f5640103b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:17:48.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:17:48.668+0000 7f5644a10700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5640100dc0 0x7f56401021a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:17:48.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:17:48.668+0000 7f5644a10700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56401026e0 con 0x7f56400ffbc0 2024-04-22T23:17:48.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:17:48.668+0000 7f5644a10700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5640102820 con 0x7f5640100dc0 2024-04-22T23:17:48.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:17:48.668+0000 7f5635d9b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5640100dc0 0x7f56401021a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:17:48.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:17:48.668+0000 7f5635d9b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5640100dc0 0x7f56401021a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:35784/0 (socket says 172.21.15.134:35784) 2024-04-22T23:17:48.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:17:48.668+0000 7f5635d9b700 1 -- 172.21.15.134:0/1240436529 learned_addr learned my addr 172.21.15.134:0/1240436529 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-04-22T23:17:48.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:17:48.668+0000 7f563e59c700 1 --2- 172.21.15.134:0/1240436529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56400ffbc0 0x7f5640103b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:17:48.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:17:48.668+0000 7f5635d9b700 1 -- 172.21.15.134:0/1240436529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56400ffbc0 msgr2=0x7f5640103b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:17:48.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:17:48.668+0000 7f5635d9b700 1 --2- 172.21.15.134:0/1240436529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56400ffbc0 0x7f5640103b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:17:48.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:17:48.668+0000 7f5635d9b700 1 -- 172.21.15.134:0/1240436529 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56280095f0 con 0x7f5640100dc0 2024-04-22T23:17:48.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:17:48.668+0000 7f5635d9b700 1 --2- 172.21.15.134:0/1240436529 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5640100dc0 0x7f56401021a0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f5628006010 tx=0x7f562800bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:17:48.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:17:48.668+0000 7f5637fff700 1 -- 172.21.15.134:0/1240436529 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f562801d070 con 0x7f5640100dc0 2024-04-22T23:17:48.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:17:48.668+0000 7f5644a10700 1 -- 172.21.15.134:0/1240436529 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5640102aa0 con 0x7f5640100dc0 2024-04-22T23:17:48.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:17:48.668+0000 7f5637fff700 1 -- 172.21.15.134:0/1240436529 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f562800f460 con 0x7f5640100dc0 2024-04-22T23:17:48.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:17:48.668+0000 7f5637fff700 1 -- 172.21.15.134:0/1240436529 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5628021640 con 0x7f5640100dc0 2024-04-22T23:17:48.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:17:48.668+0000 7f5644a10700 1 -- 172.21.15.134:0/1240436529 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5640102f90 con 0x7f5640100dc0 2024-04-22T23:17:48.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:17:48.669+0000 7f5644a10700 1 -- 172.21.15.134:0/1240436529 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5640068af0 con 0x7f5640100dc0 2024-04-22T23:17:48.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:17:48.670+0000 7f5637fff700 1 -- 172.21.15.134:0/1240436529 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 18) v1 ==== 90256+0+0 (secure 0 0 0) 0x7f562800fab0 con 0x7f5640100dc0 2024-04-22T23:17:48.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:17:48.670+0000 7f5637fff700 1 --2- 172.21.15.134:0/1240436529 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f562c06c460 0x7f562c06e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:17:48.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:17:48.670+0000 7f5637fff700 1 -- 172.21.15.134:0/1240436529 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3221+0+0 (secure 0 0 0) 0x7f562808c090 con 0x7f5640100dc0 2024-04-22T23:17:48.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:17:48.670+0000 7f563e59c700 1 --2- 172.21.15.134:0/1240436529 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f562c06c460 0x7f562c06e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:17:48.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:17:48.671+0000 7f563e59c700 1 --2- 172.21.15.134:0/1240436529 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f562c06c460 0x7f562c06e920 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5640100c20 tx=0x7f563000b410 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:17:48.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:17:48.675+0000 7f5637fff700 1 -- 172.21.15.134:0/1240436529 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f562805aca0 con 0x7f5640100dc0 2024-04-22T23:17:48.873 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:17:48.872+0000 7f5644a10700 1 -- 172.21.15.134:0/1240436529 --> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f5640069540 con 0x7f562c06c460 2024-04-22T23:17:49.181 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:48 smithi067 ceph-mon[26453]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T23:17:49.182 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:48 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-22T23:17:49.182 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:48 smithi067 ceph-mon[26453]: osdmap e20: 3 total, 3 up, 3 in 2024-04-22T23:17:49.182 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:48 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T23:17:49.182 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:48 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T23:17:49.182 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:48 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T23:17:49.182 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:48 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:17:49.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:48 smithi134 ceph-mon[34609]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-22T23:17:49.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:48 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-22T23:17:49.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:48 smithi134 ceph-mon[34609]: osdmap e20: 3 total, 3 up, 3 in 2024-04-22T23:17:49.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:48 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T23:17:49.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:48 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T23:17:49.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:48 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T23:17:49.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:48 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:17:50.545 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:17:50 smithi067 sudo[65422]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T23:17:50.545 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:17:50 smithi067 sudo[65422]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T23:17:50.545 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:17:50 smithi067 sudo[65422]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T23:17:50.545 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:17:50 smithi067 sudo[65422]: pam_unix(sudo:session): session closed for user root 2024-04-22T23:17:50.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:50 smithi067 ceph-mon[26453]: from='client.24135 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:17:50.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:50 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-22T23:17:50.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:50 smithi067 ceph-mon[26453]: osdmap e21: 3 total, 3 up, 3 in 2024-04-22T23:17:50.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:50 smithi067 ceph-mon[26453]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-04-22T23:17:50.561 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:50 smithi134 ceph-mon[34609]: from='client.24135 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:17:50.561 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:50 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-22T23:17:50.561 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:50 smithi134 ceph-mon[34609]: osdmap e21: 3 total, 3 up, 3 in 2024-04-22T23:17:50.561 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:50 smithi134 ceph-mon[34609]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-04-22T23:17:50.925 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:17:50 smithi067 sudo[65427]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T23:17:50.925 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:17:50 smithi067 sudo[65427]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T23:17:50.925 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:17:50 smithi067 sudo[65427]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T23:17:50.925 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:17:50 smithi067 sudo[65427]: pam_unix(sudo:session): session closed for user root 2024-04-22T23:17:51.266 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:17:51 smithi067 sudo[65529]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T23:17:51.266 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:17:51 smithi067 sudo[65529]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T23:17:51.266 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:17:51 smithi067 sudo[65529]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T23:17:51.266 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:17:51 smithi067 sudo[65529]: pam_unix(sudo:session): session closed for user root 2024-04-22T23:17:51.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:51 smithi067 ceph-mon[26453]: osdmap e22: 3 total, 3 up, 3 in 2024-04-22T23:17:51.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:51 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:51.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:51 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:51.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:51 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:51.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:51 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:17:51.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:51 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:17:51.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:51 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:17:51.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:51 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:51.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:51 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:17:51.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:51 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:17:51.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:51 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:51.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:51 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:17:51.731 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:51 smithi134 ceph-mon[34609]: osdmap e22: 3 total, 3 up, 3 in 2024-04-22T23:17:51.731 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:51 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:51.731 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:51 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:51.731 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:51 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:51.731 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:51 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:17:51.731 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:51 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:17:51.731 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:51 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:17:51.731 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:51 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:51.731 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:51 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:17:51.731 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:51 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:17:51.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:51 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:51.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:51 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:17:51.795 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:17:51 smithi067 sudo[65624]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T23:17:51.795 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:17:51 smithi067 sudo[65624]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T23:17:51.795 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:17:51 smithi067 sudo[65624]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T23:17:51.795 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:17:51 smithi067 sudo[65624]: pam_unix(sudo:session): session closed for user root 2024-04-22T23:17:52.294 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:17:51 smithi067 sudo[65688]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T23:17:52.295 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:17:51 smithi067 sudo[65688]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T23:17:52.295 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:17:51 smithi067 sudo[65688]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T23:17:52.295 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:17:52 smithi067 sudo[65688]: pam_unix(sudo:session): session closed for user root 2024-04-22T23:17:52.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:52 smithi067 ceph-mon[26453]: Detected new or changed devices on smithi067 2024-04-22T23:17:52.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:52 smithi067 ceph-mon[26453]: Adjusting osd_memory_target on smithi067 to 3946M 2024-04-22T23:17:52.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:52 smithi067 ceph-mon[26453]: from='client.? 172.21.15.134:0/4202388422' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a30b6c3-95bf-4b19-883b-1271ff359e39"}]: dispatch 2024-04-22T23:17:52.419 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:17:52.418+0000 7f5637fff700 1 -- 172.21.15.134:0/1240436529 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f56280575e0 con 0x7f5640100dc0 2024-04-22T23:17:52.545 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:17:52 smithi067 sudo[65768]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T23:17:52.545 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:17:52 smithi067 sudo[65768]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T23:17:52.545 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:17:52 smithi067 sudo[65768]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T23:17:52.545 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:17:52 smithi067 sudo[65768]: pam_unix(sudo:session): session closed for user root 2024-04-22T23:17:52.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:52 smithi067 ceph-mon[26453]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a30b6c3-95bf-4b19-883b-1271ff359e39"}]: dispatch 2024-04-22T23:17:52.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:52 smithi067 ceph-mon[26453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1a30b6c3-95bf-4b19-883b-1271ff359e39"}]': finished 2024-04-22T23:17:52.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:52 smithi067 ceph-mon[26453]: osdmap e23: 4 total, 3 up, 4 in 2024-04-22T23:17:52.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:52 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T23:17:52.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:52 smithi067 ceph-mon[26453]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-22T23:17:52.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:52 smithi067 ceph-mon[26453]: from='client.? 172.21.15.134:0/2925266932' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T23:17:52.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:52 smithi134 ceph-mon[34609]: Detected new or changed devices on smithi067 2024-04-22T23:17:52.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:52 smithi134 ceph-mon[34609]: Adjusting osd_memory_target on smithi067 to 3946M 2024-04-22T23:17:52.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:52 smithi134 ceph-mon[34609]: from='client.? 172.21.15.134:0/4202388422' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a30b6c3-95bf-4b19-883b-1271ff359e39"}]: dispatch 2024-04-22T23:17:52.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:52 smithi134 ceph-mon[34609]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a30b6c3-95bf-4b19-883b-1271ff359e39"}]: dispatch 2024-04-22T23:17:52.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:52 smithi134 ceph-mon[34609]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1a30b6c3-95bf-4b19-883b-1271ff359e39"}]': finished 2024-04-22T23:17:52.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:52 smithi134 ceph-mon[34609]: osdmap e23: 4 total, 3 up, 4 in 2024-04-22T23:17:52.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:52 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T23:17:52.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:52 smithi134 ceph-mon[34609]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-22T23:17:52.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:52 smithi134 ceph-mon[34609]: from='client.? 172.21.15.134:0/2925266932' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T23:17:53.217 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:52 smithi067 sudo[65817]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T23:17:53.218 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:52 smithi067 sudo[65817]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T23:17:53.218 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:52 smithi067 sudo[65817]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T23:17:53.218 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:52 smithi067 sudo[65817]: pam_unix(sudo:session): session closed for user root 2024-04-22T23:17:53.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:53 smithi067 sudo[65848]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T23:17:53.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:53 smithi067 sudo[65848]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T23:17:53.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:53 smithi067 sudo[65848]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T23:17:53.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:53 smithi067 sudo[65848]: pam_unix(sudo:session): session closed for user root 2024-04-22T23:17:53.873 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:53 smithi134 sudo[39070]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T23:17:53.873 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:53 smithi134 sudo[39070]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T23:17:53.873 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:53 smithi134 sudo[39070]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T23:17:53.873 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:53 smithi134 sudo[39070]: pam_unix(sudo:session): session closed for user root 2024-04-22T23:17:54.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:53 smithi067 ceph-mon[26453]: mgrmap e19: smithi067.whdcsr(active, since 2m), standbys: smithi134.ccxkcj 2024-04-22T23:17:54.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:53 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:17:54.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:53 smithi067 ceph-mon[26453]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T23:17:54.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:53 smithi067 ceph-mon[26453]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T23:17:54.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:53 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-22T23:17:54.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:53 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-04-22T23:17:54.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:53 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-22T23:17:54.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:53 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-04-22T23:17:54.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:53 smithi134 sudo[39258]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T23:17:54.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:53 smithi134 sudo[39258]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T23:17:54.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:53 smithi134 sudo[39258]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T23:17:54.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:53 smithi134 sudo[39258]: pam_unix(sudo:session): session closed for user root 2024-04-22T23:17:54.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:53 smithi134 ceph-mon[34609]: mgrmap e19: smithi067.whdcsr(active, since 2m), standbys: smithi134.ccxkcj 2024-04-22T23:17:54.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:53 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:17:54.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:53 smithi134 ceph-mon[34609]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T23:17:54.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:53 smithi134 ceph-mon[34609]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T23:17:54.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:53 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-22T23:17:54.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:53 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-04-22T23:17:54.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:53 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-22T23:17:54.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:53 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-04-22T23:17:55.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:54 smithi134 ceph-mon[34609]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T23:17:55.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:54 smithi134 ceph-mon[34609]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-22T23:17:55.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:54 smithi134 ceph-mon[34609]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T23:17:55.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:54 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:55.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:54 smithi067 ceph-mon[26453]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T23:17:55.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:54 smithi067 ceph-mon[26453]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-22T23:17:55.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:54 smithi067 ceph-mon[26453]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T23:17:55.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:54 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:56.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:56 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:56.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:56 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:56.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:56 smithi134 ceph-mon[34609]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-22T23:17:56.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:56 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:17:56.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:56 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:17:56.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:56 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:56.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:56 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:56.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:56 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:56.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:56 smithi067 ceph-mon[26453]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-22T23:17:56.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:56 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:17:56.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:56 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:17:56.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:56 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:17:58.637 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:58 smithi134 ceph-mon[34609]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 129 KiB/s wr, 9 op/s 2024-04-22T23:17:58.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:58 smithi067 ceph-mon[26453]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 129 KiB/s wr, 9 op/s 2024-04-22T23:17:59.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:59 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T23:17:59.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:17:59 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:17:59.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:59 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T23:17:59.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:17:59 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:18:00.671 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:00 smithi134 ceph-mon[34609]: Deploying daemon osd.3 on smithi134 2024-04-22T23:18:00.672 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:00 smithi134 ceph-mon[34609]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 97 KiB/s wr, 6 op/s 2024-04-22T23:18:00.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:00 smithi067 ceph-mon[26453]: Deploying daemon osd.3 on smithi134 2024-04-22T23:18:00.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:00 smithi067 ceph-mon[26453]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 97 KiB/s wr, 6 op/s 2024-04-22T23:18:02.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:02 smithi134 ceph-mon[34609]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-22T23:18:02.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:02 smithi067 ceph-mon[26453]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-22T23:18:04.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:04 smithi134 ceph-mon[34609]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-22T23:18:04.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:04.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:04.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:04 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:18:04.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:04 smithi067 ceph-mon[26453]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-22T23:18:04.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:04 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:04.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:04 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:04.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:04 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:18:06.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:06 smithi067 ceph-mon[26453]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-22T23:18:06.309 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:06 smithi134 ceph-mon[34609]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-22T23:18:06.384 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:06.383+0000 7f5637fff700 1 -- 172.21.15.134:0/1240436529 <== mgr.14215 v2:172.21.15.67:6800/2589747850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5640069540 con 0x7f562c06c460 2024-04-22T23:18:06.384 INFO:teuthology.orchestra.run.smithi134.stdout:Created osd(s) 3 on host 'smithi134' 2024-04-22T23:18:06.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:06.385+0000 7f5644a10700 1 -- 172.21.15.134:0/1240436529 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f562c06c460 msgr2=0x7f562c06e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:18:06.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:06.385+0000 7f5644a10700 1 --2- 172.21.15.134:0/1240436529 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f562c06c460 0x7f562c06e920 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5640100c20 tx=0x7f563000b410 comp rx=0 tx=0).stop 2024-04-22T23:18:06.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:06.385+0000 7f5644a10700 1 -- 172.21.15.134:0/1240436529 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5640100dc0 msgr2=0x7f56401021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:18:06.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:06.385+0000 7f5644a10700 1 --2- 172.21.15.134:0/1240436529 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5640100dc0 0x7f56401021a0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f5628006010 tx=0x7f562800bd70 comp rx=0 tx=0).stop 2024-04-22T23:18:06.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:06.385+0000 7f5644a10700 1 -- 172.21.15.134:0/1240436529 shutdown_connections 2024-04-22T23:18:06.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:06.385+0000 7f5644a10700 1 --2- 172.21.15.134:0/1240436529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56400ffbc0 0x7f5640103b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:18:06.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:06.385+0000 7f5644a10700 1 --2- 172.21.15.134:0/1240436529 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f562c06c460 0x7f562c06e920 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:18:06.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:06.385+0000 7f5644a10700 1 --2- 172.21.15.134:0/1240436529 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5640100dc0 0x7f56401021a0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:18:06.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:06.385+0000 7f5644a10700 1 -- 172.21.15.134:0/1240436529 >> 172.21.15.134:0/1240436529 conn(0x7f56400fb160 msgr2=0x7f5640068e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:18:06.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:06.385+0000 7f5644a10700 1 -- 172.21.15.134:0/1240436529 shutdown_connections 2024-04-22T23:18:06.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:06.385+0000 7f5644a10700 1 -- 172.21.15.134:0/1240436529 wait complete. 2024-04-22T23:18:07.402 DEBUG:teuthology.orchestra.run.smithi134:osd.3> sudo journalctl -f -n 0 -u ceph-d46c346e-00fd-11ef-bc93-c7b262605968@osd.3.service 2024-04-22T23:18:07.404 INFO:tasks.cephadm:Deploying osd.4 on smithi134 with /dev/vg_nvme/lv_3... 2024-04-22T23:18:07.404 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-22T23:18:07.617 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:07 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:07.617 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:07 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:07.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:07 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:07.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:07 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:07.909 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:18:07 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3[40519]: 2024-04-22T23:18:07.616+0000 7f0b2b7d8640 -1 osd.3 0 log_to_monitors true 2024-04-22T23:18:08.081 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-22T23:18:08.458 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:08 smithi134 ceph-mon[34609]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-22T23:18:08.458 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:08 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:18:08.458 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:08 smithi134 ceph-mon[34609]: from='osd.3 [v2:172.21.15.134:6800/2235786001,v1:172.21.15.134:6801/2235786001]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T23:18:08.458 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:08 smithi134 ceph-mon[34609]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T23:18:08.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:08 smithi067 ceph-mon[26453]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-22T23:18:08.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:08 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:18:08.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:08 smithi067 ceph-mon[26453]: from='osd.3 [v2:172.21.15.134:6800/2235786001,v1:172.21.15.134:6801/2235786001]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T23:18:08.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:08 smithi067 ceph-mon[26453]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T23:18:09.450 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:18:09 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3[40519]: 2024-04-22T23:18:09.350+0000 7f0b2063b700 -1 osd.3 0 waiting for initial osdmap 2024-04-22T23:18:09.450 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:18:09 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3[40519]: 2024-04-22T23:18:09.358+0000 7f0b1cc31700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T23:18:09.451 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:09 smithi134 ceph-mon[34609]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T23:18:09.451 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:09 smithi134 ceph-mon[34609]: osdmap e24: 4 total, 3 up, 4 in 2024-04-22T23:18:09.451 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:09 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T23:18:09.451 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:09 smithi134 ceph-mon[34609]: from='osd.3 [v2:172.21.15.134:6800/2235786001,v1:172.21.15.134:6801/2235786001]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-22T23:18:09.451 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:09 smithi134 ceph-mon[34609]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-22T23:18:09.451 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:09 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:09.451 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:09 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:09.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:09 smithi067 ceph-mon[26453]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T23:18:09.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:09 smithi067 ceph-mon[26453]: osdmap e24: 4 total, 3 up, 4 in 2024-04-22T23:18:09.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:09 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T23:18:09.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:09 smithi067 ceph-mon[26453]: from='osd.3 [v2:172.21.15.134:6800/2235786001,v1:172.21.15.134:6801/2235786001]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-22T23:18:09.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:09 smithi067 ceph-mon[26453]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-22T23:18:09.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:09 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:09.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:09 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:09.976 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-22T23:18:09.976 INFO:teuthology.orchestra.run.smithi134.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-22T23:18:09.976 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10+0 records in 2024-04-22T23:18:09.976 INFO:teuthology.orchestra.run.smithi134.stderr:10+0 records out 2024-04-22T23:18:09.976 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0162102 s, 647 MB/s 2024-04-22T23:18:09.976 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping successful for: 2024-04-22T23:18:10.427 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:10 smithi134 ceph-mon[34609]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-04-22T23:18:10.427 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:10 smithi134 ceph-mon[34609]: osdmap e25: 4 total, 3 up, 4 in 2024-04-22T23:18:10.427 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:10 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T23:18:10.428 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:10 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T23:18:10.428 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:10 smithi134 ceph-mon[34609]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-22T23:18:10.428 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:10 smithi134 ceph-mon[34609]: osd.3 [v2:172.21.15.134:6800/2235786001,v1:172.21.15.134:6801/2235786001] boot 2024-04-22T23:18:10.428 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:10 smithi134 ceph-mon[34609]: osdmap e26: 4 total, 4 up, 4 in 2024-04-22T23:18:10.428 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:10 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T23:18:10.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:10 smithi067 ceph-mon[26453]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-04-22T23:18:10.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:10 smithi067 ceph-mon[26453]: osdmap e25: 4 total, 3 up, 4 in 2024-04-22T23:18:10.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:10 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T23:18:10.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:10 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T23:18:10.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:10 smithi067 ceph-mon[26453]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-22T23:18:10.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:10 smithi067 ceph-mon[26453]: osd.3 [v2:172.21.15.134:6800/2235786001,v1:172.21.15.134:6801/2235786001] boot 2024-04-22T23:18:10.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:10 smithi067 ceph-mon[26453]: osdmap e26: 4 total, 4 up, 4 in 2024-04-22T23:18:10.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:10 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T23:18:11.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:11 smithi134 ceph-mon[34609]: purged_snaps scrub starts 2024-04-22T23:18:11.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:11 smithi134 ceph-mon[34609]: purged_snaps scrub ok 2024-04-22T23:18:11.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:11 smithi134 ceph-mon[34609]: osdmap e27: 4 total, 4 up, 4 in 2024-04-22T23:18:11.652 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi134:vg_nvme/lv_3 2024-04-22T23:18:11.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:11 smithi067 ceph-mon[26453]: purged_snaps scrub starts 2024-04-22T23:18:11.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:11 smithi067 ceph-mon[26453]: purged_snaps scrub ok 2024-04-22T23:18:11.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:11 smithi067 ceph-mon[26453]: osdmap e27: 4 total, 4 up, 4 in 2024-04-22T23:18:11.908 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-22T23:18:12.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:12 smithi134 ceph-mon[34609]: pgmap v92: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-22T23:18:12.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:12 smithi134 ceph-mon[34609]: osdmap e28: 4 total, 4 up, 4 in 2024-04-22T23:18:12.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:12 smithi067 ceph-mon[26453]: pgmap v92: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-22T23:18:12.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:12 smithi067 ceph-mon[26453]: osdmap e28: 4 total, 4 up, 4 in 2024-04-22T23:18:13.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:13.948+0000 7f6d03c25700 1 -- 172.21.15.134:0/1741569330 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6cfc100dc0 msgr2=0x7f6cfc101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:18:13.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:13.948+0000 7f6d03c25700 1 --2- 172.21.15.134:0/1741569330 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6cfc100dc0 0x7f6cfc101220 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6cec009910 tx=0x7f6cec009c20 comp rx=0 tx=0).stop 2024-04-22T23:18:13.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:13.948+0000 7f6d03c25700 1 -- 172.21.15.134:0/1741569330 shutdown_connections 2024-04-22T23:18:13.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:13.948+0000 7f6d03c25700 1 --2- 172.21.15.134:0/1741569330 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6cfc100dc0 0x7f6cfc101220 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:18:13.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:13.948+0000 7f6d03c25700 1 --2- 172.21.15.134:0/1741569330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cfc0ffbc0 0x7f6cfc0fffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:18:13.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:13.948+0000 7f6d03c25700 1 -- 172.21.15.134:0/1741569330 >> 172.21.15.134:0/1741569330 conn(0x7f6cfc0fb160 msgr2=0x7f6cfc0fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:18:13.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:13.948+0000 7f6d03c25700 1 -- 172.21.15.134:0/1741569330 shutdown_connections 2024-04-22T23:18:13.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:13.948+0000 7f6d03c25700 1 -- 172.21.15.134:0/1741569330 wait complete. 2024-04-22T23:18:13.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:13.949+0000 7f6d03c25700 1 Processor -- start 2024-04-22T23:18:13.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:13.949+0000 7f6d03c25700 1 -- start start 2024-04-22T23:18:13.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:13.949+0000 7f6d03c25700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6cfc0ffbc0 0x7f6cfc069970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:18:13.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:13.949+0000 7f6d03c25700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cfc100dc0 0x7f6cfc19b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:18:13.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:13.949+0000 7f6d03c25700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cfc19bc40 con 0x7f6cfc100dc0 2024-04-22T23:18:13.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:13.949+0000 7f6d03c25700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cfc19bd80 con 0x7f6cfc0ffbc0 2024-04-22T23:18:13.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:13.949+0000 7f6d019c1700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6cfc0ffbc0 0x7f6cfc069970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:18:13.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:13.949+0000 7f6d019c1700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6cfc0ffbc0 0x7f6cfc069970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:43416/0 (socket says 172.21.15.134:43416) 2024-04-22T23:18:13.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:13.949+0000 7f6d019c1700 1 -- 172.21.15.134:0/924278882 learned_addr learned my addr 172.21.15.134:0/924278882 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-04-22T23:18:13.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:13.949+0000 7f6d011c0700 1 --2- 172.21.15.134:0/924278882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cfc100dc0 0x7f6cfc19b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:18:13.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:13.949+0000 7f6d019c1700 1 -- 172.21.15.134:0/924278882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cfc100dc0 msgr2=0x7f6cfc19b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:18:13.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:13.949+0000 7f6d019c1700 1 --2- 172.21.15.134:0/924278882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cfc100dc0 0x7f6cfc19b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:18:13.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:13.949+0000 7f6d019c1700 1 -- 172.21.15.134:0/924278882 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cec0095f0 con 0x7f6cfc0ffbc0 2024-04-22T23:18:13.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:13.949+0000 7f6d019c1700 1 --2- 172.21.15.134:0/924278882 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6cfc0ffbc0 0x7f6cfc069970 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6cf800b4b0 tx=0x7f6cf800b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:18:13.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:13.950+0000 7f6cf2ffd700 1 -- 172.21.15.134:0/924278882 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6cf8011800 con 0x7f6cfc0ffbc0 2024-04-22T23:18:13.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:13.950+0000 7f6cf2ffd700 1 -- 172.21.15.134:0/924278882 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6cf8011e10 con 0x7f6cfc0ffbc0 2024-04-22T23:18:13.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:13.950+0000 7f6d03c25700 1 -- 172.21.15.134:0/924278882 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cfc19c060 con 0x7f6cfc0ffbc0 2024-04-22T23:18:13.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:13.950+0000 7f6cf2ffd700 1 -- 172.21.15.134:0/924278882 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6cf800f2e0 con 0x7f6cfc0ffbc0 2024-04-22T23:18:13.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:13.950+0000 7f6d03c25700 1 -- 172.21.15.134:0/924278882 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cfc19c5b0 con 0x7f6cfc0ffbc0 2024-04-22T23:18:13.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:13.950+0000 7f6d03c25700 1 -- 172.21.15.134:0/924278882 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cfc18f5d0 con 0x7f6cfc0ffbc0 2024-04-22T23:18:13.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:13.951+0000 7f6cf2ffd700 1 -- 172.21.15.134:0/924278882 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f6cf8010380 con 0x7f6cfc0ffbc0 2024-04-22T23:18:13.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:13.951+0000 7f6cf2ffd700 1 --2- 172.21.15.134:0/924278882 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f6ce806c4e0 0x7f6ce806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:18:13.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:13.952+0000 7f6cf2ffd700 1 -- 172.21.15.134:0/924278882 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 3774+0+0 (secure 0 0 0) 0x7f6cf808add0 con 0x7f6cfc0ffbc0 2024-04-22T23:18:13.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:13.952+0000 7f6d011c0700 1 --2- 172.21.15.134:0/924278882 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f6ce806c4e0 0x7f6ce806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:18:13.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:13.952+0000 7f6d011c0700 1 --2- 172.21.15.134:0/924278882 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f6ce806c4e0 0x7f6ce806e9a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc069ef0 tx=0x7f6cec0058e0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:18:13.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:13.956+0000 7f6cf2ffd700 1 -- 172.21.15.134:0/924278882 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6cf80596e0 con 0x7f6cfc0ffbc0 2024-04-22T23:18:14.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:14.156+0000 7f6d03c25700 1 -- 172.21.15.134:0/924278882 --> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f6cfc02d010 con 0x7f6ce806c4e0 2024-04-22T23:18:14.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:14 smithi134 ceph-mon[34609]: osdmap e29: 4 total, 4 up, 4 in 2024-04-22T23:18:14.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:14 smithi134 ceph-mon[34609]: Detected new or changed devices on smithi134 2024-04-22T23:18:14.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:14 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:14.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:14 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:14.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:14 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:18:14.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:14 smithi134 ceph-mon[34609]: Adjusting osd_memory_target on smithi134 to 14912M 2024-04-22T23:18:14.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:14 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:14.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:14 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:18:14.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:14 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:18:14.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:14 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:14.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:14 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:18:14.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:14 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:18:14.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:14 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:18:14.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:14 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:14.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:14 smithi134 ceph-mon[34609]: pgmap v95: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-22T23:18:14.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:14 smithi067 ceph-mon[26453]: osdmap e29: 4 total, 4 up, 4 in 2024-04-22T23:18:14.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:14 smithi067 ceph-mon[26453]: Detected new or changed devices on smithi134 2024-04-22T23:18:14.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:14 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:14.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:14 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:14.296 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:14 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:18:14.296 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:14 smithi067 ceph-mon[26453]: Adjusting osd_memory_target on smithi134 to 14912M 2024-04-22T23:18:14.296 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:14 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:14.296 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:14 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:18:14.296 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:14 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:18:14.296 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:14 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:14.296 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:14 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:18:14.296 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:14 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:18:14.296 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:14 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:18:14.296 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:14 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:14.296 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:14 smithi067 ceph-mon[26453]: pgmap v95: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-22T23:18:15.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:15 smithi067 ceph-mon[26453]: osdmap e30: 4 total, 4 up, 4 in 2024-04-22T23:18:15.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:15 smithi067 ceph-mon[26453]: from='client.24159 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:18:15.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:15 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T23:18:15.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:15 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T23:18:15.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:15 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:18:15.377 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:15 smithi134 ceph-mon[34609]: osdmap e30: 4 total, 4 up, 4 in 2024-04-22T23:18:15.377 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:15 smithi134 ceph-mon[34609]: from='client.24159 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:18:15.378 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:15 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T23:18:15.378 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:15 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T23:18:15.378 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:15 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:18:16.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:16 smithi067 ceph-mon[26453]: pgmap v97: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-22T23:18:16.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:16 smithi134 ceph-mon[34609]: pgmap v97: 1 pgs: 1 remapped+peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-22T23:18:17.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:17 smithi067 ceph-mon[26453]: from='client.? 172.21.15.134:0/1382384873' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "53368fc1-28fd-4622-bc9c-1d8901b06bd2"}]: dispatch 2024-04-22T23:18:17.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:17 smithi067 ceph-mon[26453]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "53368fc1-28fd-4622-bc9c-1d8901b06bd2"}]: dispatch 2024-04-22T23:18:17.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:17 smithi067 ceph-mon[26453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "53368fc1-28fd-4622-bc9c-1d8901b06bd2"}]': finished 2024-04-22T23:18:17.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:17 smithi067 ceph-mon[26453]: osdmap e31: 5 total, 4 up, 5 in 2024-04-22T23:18:17.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:17 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T23:18:17.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:17 smithi067 ceph-mon[26453]: from='client.? 172.21.15.134:0/1160244381' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T23:18:17.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:17 smithi134 ceph-mon[34609]: from='client.? 172.21.15.134:0/1382384873' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "53368fc1-28fd-4622-bc9c-1d8901b06bd2"}]: dispatch 2024-04-22T23:18:17.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:17 smithi134 ceph-mon[34609]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "53368fc1-28fd-4622-bc9c-1d8901b06bd2"}]: dispatch 2024-04-22T23:18:17.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:17 smithi134 ceph-mon[34609]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "53368fc1-28fd-4622-bc9c-1d8901b06bd2"}]': finished 2024-04-22T23:18:17.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:17 smithi134 ceph-mon[34609]: osdmap e31: 5 total, 4 up, 5 in 2024-04-22T23:18:17.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:17 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T23:18:17.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:17 smithi134 ceph-mon[34609]: from='client.? 172.21.15.134:0/1160244381' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T23:18:18.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:18 smithi067 ceph-mon[26453]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-04-22T23:18:18.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:18 smithi134 ceph-mon[34609]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-04-22T23:18:20.570 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:20 smithi134 ceph-mon[34609]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-22T23:18:20.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:20 smithi067 ceph-mon[26453]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-22T23:18:22.488 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:22 smithi134 ceph-mon[34609]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-22T23:18:22.488 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:22 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:18:22.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:22 smithi067 ceph-mon[26453]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-22T23:18:22.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:22 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:18:24.605 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:24 smithi134 ceph-mon[34609]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-22T23:18:24.605 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:24 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T23:18:24.605 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:24 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:18:24.606 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:24 smithi134 ceph-mon[34609]: Deploying daemon osd.4 on smithi134 2024-04-22T23:18:24.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:24 smithi067 ceph-mon[26453]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-22T23:18:24.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:24 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T23:18:24.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:24 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:18:24.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:24 smithi067 ceph-mon[26453]: Deploying daemon osd.4 on smithi134 2024-04-22T23:18:26.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:26 smithi067 ceph-mon[26453]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-22T23:18:26.324 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:26 smithi134 ceph-mon[34609]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-22T23:18:28.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:28 smithi134 ceph-mon[34609]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-22T23:18:28.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:28 smithi067 ceph-mon[26453]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-22T23:18:29.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:29 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:18:29.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:29 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:29.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:29 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:29.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:29 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:18:29.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:29 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:29.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:29 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:30.695 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:30 smithi134 ceph-mon[34609]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-22T23:18:30.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:30 smithi067 ceph-mon[26453]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-22T23:18:32.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:32.069+0000 7f6cf2ffd700 1 -- 172.21.15.134:0/924278882 <== mgr.14215 v2:172.21.15.67:6800/2589747850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f6cfc02d010 con 0x7f6ce806c4e0 2024-04-22T23:18:32.396 INFO:teuthology.orchestra.run.smithi134.stdout:Created osd(s) 4 on host 'smithi134' 2024-04-22T23:18:32.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:32.071+0000 7f6d03c25700 1 -- 172.21.15.134:0/924278882 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f6ce806c4e0 msgr2=0x7f6ce806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:18:32.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:32.071+0000 7f6d03c25700 1 --2- 172.21.15.134:0/924278882 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f6ce806c4e0 0x7f6ce806e9a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc069ef0 tx=0x7f6cec0058e0 comp rx=0 tx=0).stop 2024-04-22T23:18:32.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:32.071+0000 7f6d03c25700 1 -- 172.21.15.134:0/924278882 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6cfc0ffbc0 msgr2=0x7f6cfc069970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:18:32.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:32.071+0000 7f6d03c25700 1 --2- 172.21.15.134:0/924278882 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6cfc0ffbc0 0x7f6cfc069970 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6cf800b4b0 tx=0x7f6cf800b870 comp rx=0 tx=0).stop 2024-04-22T23:18:32.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:32.071+0000 7f6d03c25700 1 -- 172.21.15.134:0/924278882 shutdown_connections 2024-04-22T23:18:32.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:32.071+0000 7f6d03c25700 1 --2- 172.21.15.134:0/924278882 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f6ce806c4e0 0x7f6ce806e9a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:18:32.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:32.071+0000 7f6d03c25700 1 --2- 172.21.15.134:0/924278882 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6cfc0ffbc0 0x7f6cfc069970 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:18:32.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:32.071+0000 7f6d03c25700 1 --2- 172.21.15.134:0/924278882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cfc100dc0 0x7f6cfc19b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:18:32.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:32.071+0000 7f6d03c25700 1 -- 172.21.15.134:0/924278882 >> 172.21.15.134:0/924278882 conn(0x7f6cfc0fb160 msgr2=0x7f6cfc104170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:18:32.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:32.071+0000 7f6d03c25700 1 -- 172.21.15.134:0/924278882 shutdown_connections 2024-04-22T23:18:32.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:32.071+0000 7f6d03c25700 1 -- 172.21.15.134:0/924278882 wait complete. 2024-04-22T23:18:32.552 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:32 smithi134 ceph-mon[34609]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-22T23:18:32.552 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:32 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:32.552 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:32 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:32.552 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:32 smithi134 ceph-mon[34609]: from='osd.4 [v2:172.21.15.134:6808/668644810,v1:172.21.15.134:6809/668644810]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T23:18:32.552 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:32 smithi134 ceph-mon[34609]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T23:18:32.752 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:32 smithi067 ceph-mon[26453]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-22T23:18:32.752 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:32 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:32.752 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:32 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:32.752 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:32 smithi067 ceph-mon[26453]: from='osd.4 [v2:172.21.15.134:6808/668644810,v1:172.21.15.134:6809/668644810]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T23:18:32.752 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:32 smithi067 ceph-mon[26453]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T23:18:33.045 DEBUG:teuthology.orchestra.run.smithi134:osd.4> sudo journalctl -f -n 0 -u ceph-d46c346e-00fd-11ef-bc93-c7b262605968@osd.4.service 2024-04-22T23:18:33.048 INFO:tasks.cephadm:Deploying osd.5 on smithi134 with /dev/vg_nvme/lv_2... 2024-04-22T23:18:33.048 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-22T23:18:33.298 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-22T23:18:34.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:34 smithi067 ceph-mon[26453]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T23:18:34.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:34 smithi067 ceph-mon[26453]: osdmap e32: 5 total, 4 up, 5 in 2024-04-22T23:18:34.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:34 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T23:18:34.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:34 smithi067 ceph-mon[26453]: from='osd.4 [v2:172.21.15.134:6808/668644810,v1:172.21.15.134:6809/668644810]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-22T23:18:34.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:34 smithi067 ceph-mon[26453]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-22T23:18:34.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:34 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:34.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:34 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:34.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:34 smithi067 ceph-mon[26453]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-22T23:18:34.381 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:34 smithi134 ceph-mon[34609]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T23:18:34.381 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:34 smithi134 ceph-mon[34609]: osdmap e32: 5 total, 4 up, 5 in 2024-04-22T23:18:34.381 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:34 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T23:18:34.381 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:34 smithi134 ceph-mon[34609]: from='osd.4 [v2:172.21.15.134:6808/668644810,v1:172.21.15.134:6809/668644810]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-22T23:18:34.381 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:34 smithi134 ceph-mon[34609]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-22T23:18:34.381 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:34 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:34.381 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:34 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:34.382 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:34 smithi134 ceph-mon[34609]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-22T23:18:34.382 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:18:34 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4[47414]: 2024-04-22T23:18:34.036+0000 7f9e79cac700 -1 osd.4 0 waiting for initial osdmap 2024-04-22T23:18:34.382 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:18:34 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4[47414]: 2024-04-22T23:18:34.043+0000 7f9e73a9d700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T23:18:35.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:35 smithi134 ceph-mon[34609]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-04-22T23:18:35.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:35 smithi134 ceph-mon[34609]: osdmap e33: 5 total, 4 up, 5 in 2024-04-22T23:18:35.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:35 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T23:18:35.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:35 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T23:18:35.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:35 smithi067 ceph-mon[26453]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-04-22T23:18:35.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:35 smithi067 ceph-mon[26453]: osdmap e33: 5 total, 4 up, 5 in 2024-04-22T23:18:35.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:35 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T23:18:35.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:35 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T23:18:35.794 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-22T23:18:35.795 INFO:teuthology.orchestra.run.smithi134.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-22T23:18:35.795 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10+0 records in 2024-04-22T23:18:35.795 INFO:teuthology.orchestra.run.smithi134.stderr:10+0 records out 2024-04-22T23:18:35.795 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0121832 s, 861 MB/s 2024-04-22T23:18:35.795 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping successful for: 2024-04-22T23:18:36.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:36 smithi067 ceph-mon[26453]: purged_snaps scrub starts 2024-04-22T23:18:36.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:36 smithi067 ceph-mon[26453]: purged_snaps scrub ok 2024-04-22T23:18:36.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:36 smithi067 ceph-mon[26453]: osd.4 [v2:172.21.15.134:6808/668644810,v1:172.21.15.134:6809/668644810] boot 2024-04-22T23:18:36.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:36 smithi067 ceph-mon[26453]: osdmap e34: 5 total, 5 up, 5 in 2024-04-22T23:18:36.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:36 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T23:18:36.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:36 smithi067 ceph-mon[26453]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-22T23:18:36.452 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:36 smithi134 ceph-mon[34609]: purged_snaps scrub starts 2024-04-22T23:18:36.453 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:36 smithi134 ceph-mon[34609]: purged_snaps scrub ok 2024-04-22T23:18:36.453 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:36 smithi134 ceph-mon[34609]: osd.4 [v2:172.21.15.134:6808/668644810,v1:172.21.15.134:6809/668644810] boot 2024-04-22T23:18:36.453 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:36 smithi134 ceph-mon[34609]: osdmap e34: 5 total, 5 up, 5 in 2024-04-22T23:18:36.453 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:36 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T23:18:36.453 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:36 smithi134 ceph-mon[34609]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-22T23:18:36.475 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi134:vg_nvme/lv_2 2024-04-22T23:18:36.953 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-22T23:18:37.072 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:37 smithi134 ceph-mon[34609]: osdmap e35: 5 total, 5 up, 5 in 2024-04-22T23:18:37.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:37 smithi067 ceph-mon[26453]: osdmap e35: 5 total, 5 up, 5 in 2024-04-22T23:18:38.162 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:38 smithi134 ceph-mon[34609]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T23:18:38.162 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:38 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:18:38.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:38 smithi067 ceph-mon[26453]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T23:18:38.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:38 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:18:39.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:39.077+0000 7f1afa247700 1 -- 172.21.15.134:0/2538597780 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1af40ffc10 msgr2=0x7f1af4100030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:18:39.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:39.077+0000 7f1afa247700 1 --2- 172.21.15.134:0/2538597780 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1af40ffc10 0x7f1af4100030 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8009910 tx=0x7f1ae8009c20 comp rx=0 tx=0).stop 2024-04-22T23:18:39.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:39.077+0000 7f1afa247700 1 -- 172.21.15.134:0/2538597780 shutdown_connections 2024-04-22T23:18:39.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:39.077+0000 7f1afa247700 1 --2- 172.21.15.134:0/2538597780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1af4100e10 0x7f1af4101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:18:39.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:39.077+0000 7f1afa247700 1 --2- 172.21.15.134:0/2538597780 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1af40ffc10 0x7f1af4100030 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:18:39.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:39.077+0000 7f1afa247700 1 -- 172.21.15.134:0/2538597780 >> 172.21.15.134:0/2538597780 conn(0x7f1af40fb190 msgr2=0x7f1af40fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:18:39.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:39.077+0000 7f1afa247700 1 -- 172.21.15.134:0/2538597780 shutdown_connections 2024-04-22T23:18:39.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:39.077+0000 7f1afa247700 1 -- 172.21.15.134:0/2538597780 wait complete. 2024-04-22T23:18:39.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:39.077+0000 7f1afa247700 1 Processor -- start 2024-04-22T23:18:39.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:39.078+0000 7f1afa247700 1 -- start start 2024-04-22T23:18:39.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:39.078+0000 7f1afa247700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1af40ffc10 0x7f1af40690e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:18:39.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:39.078+0000 7f1afa247700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1af4100e10 0x7f1af419b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:18:39.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:39.078+0000 7f1afa247700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af419b880 con 0x7f1af4100e10 2024-04-22T23:18:39.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:39.078+0000 7f1afa247700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af419b9c0 con 0x7f1af40ffc10 2024-04-22T23:18:39.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:39.078+0000 7f1af37fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1af40ffc10 0x7f1af40690e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:18:39.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:39.078+0000 7f1af37fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1af40ffc10 0x7f1af40690e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:51718/0 (socket says 172.21.15.134:51718) 2024-04-22T23:18:39.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:39.078+0000 7f1af37fe700 1 -- 172.21.15.134:0/4084560214 learned_addr learned my addr 172.21.15.134:0/4084560214 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-04-22T23:18:39.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:39.078+0000 7f1af2ffd700 1 --2- 172.21.15.134:0/4084560214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1af4100e10 0x7f1af419b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:18:39.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:39.078+0000 7f1af37fe700 1 -- 172.21.15.134:0/4084560214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1af4100e10 msgr2=0x7f1af419b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:18:39.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:39.078+0000 7f1af37fe700 1 --2- 172.21.15.134:0/4084560214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1af4100e10 0x7f1af419b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:18:39.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:39.078+0000 7f1af37fe700 1 -- 172.21.15.134:0/4084560214 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae80095f0 con 0x7f1af40ffc10 2024-04-22T23:18:39.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:39.078+0000 7f1af37fe700 1 --2- 172.21.15.134:0/4084560214 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1af40ffc10 0x7f1af40690e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1ae800bdd0 tx=0x7f1ae800beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:18:39.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:39.078+0000 7f1af0ff9700 1 -- 172.21.15.134:0/4084560214 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ae801d070 con 0x7f1af40ffc10 2024-04-22T23:18:39.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:39.078+0000 7f1afa247700 1 -- 172.21.15.134:0/4084560214 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1af419bc40 con 0x7f1af40ffc10 2024-04-22T23:18:39.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:39.079+0000 7f1afa247700 1 -- 172.21.15.134:0/4084560214 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1af419c130 con 0x7f1af40ffc10 2024-04-22T23:18:39.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:39.079+0000 7f1af0ff9700 1 -- 172.21.15.134:0/4084560214 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1ae8004c50 con 0x7f1af40ffc10 2024-04-22T23:18:39.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:39.079+0000 7f1af0ff9700 1 -- 172.21.15.134:0/4084560214 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ae8017630 con 0x7f1af40ffc10 2024-04-22T23:18:39.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:39.080+0000 7f1afa247700 1 -- 172.21.15.134:0/4084560214 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ae0005320 con 0x7f1af40ffc10 2024-04-22T23:18:39.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:39.080+0000 7f1af0ff9700 1 -- 172.21.15.134:0/4084560214 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f1ae800f460 con 0x7f1af40ffc10 2024-04-22T23:18:39.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:39.081+0000 7f1af0ff9700 1 --2- 172.21.15.134:0/4084560214 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f1adc06c4e0 0x7f1adc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:18:39.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:39.081+0000 7f1af0ff9700 1 -- 172.21.15.134:0/4084560214 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4206+0+0 (secure 0 0 0) 0x7f1ae808c730 con 0x7f1af40ffc10 2024-04-22T23:18:39.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:39.081+0000 7f1af2ffd700 1 --2- 172.21.15.134:0/4084560214 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f1adc06c4e0 0x7f1adc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:18:39.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:39.081+0000 7f1af2ffd700 1 --2- 172.21.15.134:0/4084560214 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f1adc06c4e0 0x7f1adc06e9a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1ae4006fd0 tx=0x7f1ae4008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:18:39.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:39.085+0000 7f1af0ff9700 1 -- 172.21.15.134:0/4084560214 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f1ae805b060 con 0x7f1af40ffc10 2024-04-22T23:18:39.284 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:39.283+0000 7f1afa247700 1 -- 172.21.15.134:0/4084560214 --> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ae0000bf0 con 0x7f1adc06c4e0 2024-04-22T23:18:39.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:39 smithi134 ceph-mon[34609]: Detected new or changed devices on smithi134 2024-04-22T23:18:39.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:39 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:39.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:39 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:39.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:39 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:18:39.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:39 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:18:39.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:39 smithi134 ceph-mon[34609]: Adjusting osd_memory_target on smithi134 to 7456M 2024-04-22T23:18:39.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:39 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:39.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:39 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:18:39.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:39 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:18:39.496 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:39 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:39.496 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:39 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:18:39.496 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:39 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:18:39.496 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:39 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:18:39.496 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:39 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:39.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:39 smithi067 ceph-mon[26453]: Detected new or changed devices on smithi134 2024-04-22T23:18:39.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:39 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:39.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:39 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:39.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:39 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:18:39.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:39 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:18:39.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:39 smithi067 ceph-mon[26453]: Adjusting osd_memory_target on smithi134 to 7456M 2024-04-22T23:18:39.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:39 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:39.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:39 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:18:39.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:39 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:18:39.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:39 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:39.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:39 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:18:39.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:39 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:18:39.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:39 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:18:39.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:39 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:40.298 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:40 smithi134 ceph-mon[34609]: from='client.24177 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:18:40.299 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:40 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T23:18:40.299 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:40 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T23:18:40.299 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:40 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:18:40.299 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:40 smithi134 ceph-mon[34609]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T23:18:40.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:40 smithi067 ceph-mon[26453]: from='client.24177 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:18:40.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:40 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T23:18:40.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:40 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T23:18:40.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:40 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:18:40.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:40 smithi067 ceph-mon[26453]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T23:18:42.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:42 smithi067 ceph-mon[26453]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T23:18:42.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:42 smithi067 ceph-mon[26453]: from='client.? 172.21.15.134:0/3723586992' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31a885f7-8f91-4b6a-922a-065753f70a95"}]: dispatch 2024-04-22T23:18:42.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:42 smithi067 ceph-mon[26453]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31a885f7-8f91-4b6a-922a-065753f70a95"}]: dispatch 2024-04-22T23:18:42.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:42 smithi067 ceph-mon[26453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "31a885f7-8f91-4b6a-922a-065753f70a95"}]': finished 2024-04-22T23:18:42.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:42 smithi067 ceph-mon[26453]: osdmap e36: 6 total, 5 up, 6 in 2024-04-22T23:18:42.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:42 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T23:18:42.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:42 smithi067 ceph-mon[26453]: from='client.? 172.21.15.134:0/4213860665' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T23:18:42.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:42 smithi134 ceph-mon[34609]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T23:18:42.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:42 smithi134 ceph-mon[34609]: from='client.? 172.21.15.134:0/3723586992' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31a885f7-8f91-4b6a-922a-065753f70a95"}]: dispatch 2024-04-22T23:18:42.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:42 smithi134 ceph-mon[34609]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31a885f7-8f91-4b6a-922a-065753f70a95"}]: dispatch 2024-04-22T23:18:42.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:42 smithi134 ceph-mon[34609]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "31a885f7-8f91-4b6a-922a-065753f70a95"}]': finished 2024-04-22T23:18:42.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:42 smithi134 ceph-mon[34609]: osdmap e36: 6 total, 5 up, 6 in 2024-04-22T23:18:42.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:42 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T23:18:42.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:42 smithi134 ceph-mon[34609]: from='client.? 172.21.15.134:0/4213860665' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T23:18:44.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:44 smithi134 ceph-mon[34609]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T23:18:44.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:44 smithi067 ceph-mon[26453]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T23:18:46.198 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:46 smithi134 ceph-mon[34609]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T23:18:46.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:46 smithi067 ceph-mon[26453]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T23:18:48.661 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:48 smithi134 ceph-mon[34609]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T23:18:48.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:48 smithi067 ceph-mon[26453]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T23:18:49.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:49 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T23:18:49.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:49 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:18:49.738 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:49 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T23:18:49.738 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:49 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:18:50.642 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:50 smithi134 ceph-mon[34609]: Deploying daemon osd.5 on smithi134 2024-04-22T23:18:50.642 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:50 smithi134 ceph-mon[34609]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T23:18:50.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:50 smithi067 ceph-mon[26453]: Deploying daemon osd.5 on smithi134 2024-04-22T23:18:50.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:50 smithi067 ceph-mon[26453]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T23:18:52.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:52 smithi134 ceph-mon[34609]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T23:18:52.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:52 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:18:52.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:52 smithi067 ceph-mon[26453]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T23:18:52.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:52 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:18:54.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:54 smithi134 ceph-mon[34609]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T23:18:54.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:54 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:54.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:54 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:54.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:54 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:18:54.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:54 smithi067 ceph-mon[26453]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T23:18:54.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:54 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:54.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:54 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:54.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:54 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:18:56.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:56 smithi134 ceph-mon[34609]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T23:18:56.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:56 smithi067 ceph-mon[26453]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T23:18:57.900 INFO:teuthology.orchestra.run.smithi134.stdout:Created osd(s) 5 on host 'smithi134' 2024-04-22T23:18:57.901 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:57.899+0000 7f1af0ff9700 1 -- 172.21.15.134:0/4084560214 <== mgr.14215 v2:172.21.15.67:6800/2589747850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1ae0000bf0 con 0x7f1adc06c4e0 2024-04-22T23:18:57.903 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:57.901+0000 7f1afa247700 1 -- 172.21.15.134:0/4084560214 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f1adc06c4e0 msgr2=0x7f1adc06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:18:57.903 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:57.901+0000 7f1afa247700 1 --2- 172.21.15.134:0/4084560214 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f1adc06c4e0 0x7f1adc06e9a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1ae4006fd0 tx=0x7f1ae4008040 comp rx=0 tx=0).stop 2024-04-22T23:18:57.903 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:57.901+0000 7f1afa247700 1 -- 172.21.15.134:0/4084560214 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1af40ffc10 msgr2=0x7f1af40690e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:18:57.903 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:57.901+0000 7f1afa247700 1 --2- 172.21.15.134:0/4084560214 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1af40ffc10 0x7f1af40690e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1ae800bdd0 tx=0x7f1ae800beb0 comp rx=0 tx=0).stop 2024-04-22T23:18:57.903 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:57.902+0000 7f1afa247700 1 -- 172.21.15.134:0/4084560214 shutdown_connections 2024-04-22T23:18:57.903 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:57.902+0000 7f1afa247700 1 --2- 172.21.15.134:0/4084560214 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f1adc06c4e0 0x7f1adc06e9a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:18:57.903 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:57.902+0000 7f1afa247700 1 --2- 172.21.15.134:0/4084560214 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1af40ffc10 0x7f1af40690e0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:18:57.903 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:57.902+0000 7f1afa247700 1 --2- 172.21.15.134:0/4084560214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1af4100e10 0x7f1af419b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:18:57.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:57.902+0000 7f1afa247700 1 -- 172.21.15.134:0/4084560214 >> 172.21.15.134:0/4084560214 conn(0x7f1af40fb190 msgr2=0x7f1af4104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:18:57.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:57.902+0000 7f1afa247700 1 -- 172.21.15.134:0/4084560214 shutdown_connections 2024-04-22T23:18:57.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:18:57.902+0000 7f1afa247700 1 -- 172.21.15.134:0/4084560214 wait complete. 2024-04-22T23:18:58.668 DEBUG:teuthology.orchestra.run.smithi134:osd.5> sudo journalctl -f -n 0 -u ceph-d46c346e-00fd-11ef-bc93-c7b262605968@osd.5.service 2024-04-22T23:18:58.671 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-04-22T23:18:58.671 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-22T23:18:58.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:58 smithi134 ceph-mon[34609]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T23:18:58.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:58 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:58.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:58 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:58.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:58 smithi134 ceph-mon[34609]: from='osd.5 [v2:172.21.15.134:6816/3846690247,v1:172.21.15.134:6817/3846690247]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T23:18:58.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:58 smithi134 ceph-mon[34609]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T23:18:58.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:58 smithi067 ceph-mon[26453]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T23:18:58.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:58 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:58.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:58 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:18:58.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:58 smithi067 ceph-mon[26453]: from='osd.5 [v2:172.21.15.134:6816/3846690247,v1:172.21.15.134:6817/3846690247]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T23:18:58.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:58 smithi067 ceph-mon[26453]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T23:18:58.904 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:19:00.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:59 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:19:00.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:59 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:19:00.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:59 smithi067 ceph-mon[26453]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T23:19:00.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:59 smithi067 ceph-mon[26453]: osdmap e37: 6 total, 5 up, 6 in 2024-04-22T23:19:00.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:59 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T23:19:00.046 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:59 smithi067 ceph-mon[26453]: from='osd.5 [v2:172.21.15.134:6816/3846690247,v1:172.21.15.134:6817/3846690247]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-22T23:19:00.046 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:59 smithi067 ceph-mon[26453]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-22T23:19:00.046 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:18:59 smithi067 ceph-mon[26453]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T23:19:00.065 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:18:59 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[54248]: 2024-04-22T23:18:59.900+0000 7f8b01e45700 -1 osd.5 0 waiting for initial osdmap 2024-04-22T23:19:00.065 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:18:59 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[54248]: 2024-04-22T23:18:59.910+0000 7f8afe43b700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T23:19:00.066 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:59 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:19:00.066 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:59 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:19:00.066 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:59 smithi134 ceph-mon[34609]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T23:19:00.066 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:59 smithi134 ceph-mon[34609]: osdmap e37: 6 total, 5 up, 6 in 2024-04-22T23:19:00.066 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:59 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T23:19:00.066 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:59 smithi134 ceph-mon[34609]: from='osd.5 [v2:172.21.15.134:6816/3846690247,v1:172.21.15.134:6817/3846690247]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-22T23:19:00.066 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:59 smithi134 ceph-mon[34609]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-22T23:19:00.066 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:18:59 smithi134 ceph-mon[34609]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-22T23:19:00.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.677+0000 7fa821dec700 1 -- 172.21.15.67:0/166604678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa81c100cb0 msgr2=0x7fa81c1010d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:00.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.677+0000 7fa821dec700 1 --2- 172.21.15.67:0/166604678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa81c100cb0 0x7fa81c1010d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa810009910 tx=0x7fa810009c20 comp rx=0 tx=0).stop 2024-04-22T23:19:00.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.677+0000 7fa821dec700 1 -- 172.21.15.67:0/166604678 shutdown_connections 2024-04-22T23:19:00.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.677+0000 7fa821dec700 1 --2- 172.21.15.67:0/166604678 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa81c101eb0 0x7fa81c102310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:00.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.677+0000 7fa821dec700 1 --2- 172.21.15.67:0/166604678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa81c100cb0 0x7fa81c1010d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:00.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.677+0000 7fa821dec700 1 -- 172.21.15.67:0/166604678 >> 172.21.15.67:0/166604678 conn(0x7fa81c0fc230 msgr2=0x7fa81c0fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:00.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.677+0000 7fa821dec700 1 -- 172.21.15.67:0/166604678 shutdown_connections 2024-04-22T23:19:00.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.677+0000 7fa821dec700 1 -- 172.21.15.67:0/166604678 wait complete. 2024-04-22T23:19:00.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.677+0000 7fa821dec700 1 Processor -- start 2024-04-22T23:19:00.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.677+0000 7fa821dec700 1 -- start start 2024-04-22T23:19:00.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.678+0000 7fa821dec700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa81c100cb0 0x7fa81c074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:00.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.678+0000 7fa821dec700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa81c101eb0 0x7fa81c073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:00.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.678+0000 7fa821dec700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c0736c0 con 0x7fa81c101eb0 2024-04-22T23:19:00.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.678+0000 7fa821dec700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c073800 con 0x7fa81c100cb0 2024-04-22T23:19:00.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.678+0000 7fa81affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa81c101eb0 0x7fa81c073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:00.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.678+0000 7fa81affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa81c101eb0 0x7fa81c073180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40848/0 (socket says 172.21.15.67:40848) 2024-04-22T23:19:00.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.678+0000 7fa81affd700 1 -- 172.21.15.67:0/3433701517 learned_addr learned my addr 172.21.15.67:0/3433701517 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:19:00.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.678+0000 7fa81b7fe700 1 --2- 172.21.15.67:0/3433701517 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa81c100cb0 0x7fa81c074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:00.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.678+0000 7fa81affd700 1 -- 172.21.15.67:0/3433701517 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa81c100cb0 msgr2=0x7fa81c074b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:00.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.678+0000 7fa81affd700 1 --2- 172.21.15.67:0/3433701517 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa81c100cb0 0x7fa81c074b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:00.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.678+0000 7fa81affd700 1 -- 172.21.15.67:0/3433701517 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8100095f0 con 0x7fa81c101eb0 2024-04-22T23:19:00.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.678+0000 7fa81affd700 1 --2- 172.21.15.67:0/3433701517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa81c101eb0 0x7fa81c073180 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa80c00e920 tx=0x7fa80c00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:00.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.678+0000 7fa818ff9700 1 -- 172.21.15.67:0/3433701517 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa80c00c9d0 con 0x7fa81c101eb0 2024-04-22T23:19:00.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.678+0000 7fa821dec700 1 -- 172.21.15.67:0/3433701517 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa81c073ae0 con 0x7fa81c101eb0 2024-04-22T23:19:00.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.678+0000 7fa818ff9700 1 -- 172.21.15.67:0/3433701517 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa80c010430 con 0x7fa81c101eb0 2024-04-22T23:19:00.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.678+0000 7fa818ff9700 1 -- 172.21.15.67:0/3433701517 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa80c018650 con 0x7fa81c101eb0 2024-04-22T23:19:00.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.678+0000 7fa821dec700 1 -- 172.21.15.67:0/3433701517 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa81c19e520 con 0x7fa81c101eb0 2024-04-22T23:19:00.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.680+0000 7fa821dec700 1 -- 172.21.15.67:0/3433701517 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa81c109810 con 0x7fa81c101eb0 2024-04-22T23:19:00.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.680+0000 7fa818ff9700 1 -- 172.21.15.67:0/3433701517 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fa80c0105a0 con 0x7fa81c101eb0 2024-04-22T23:19:00.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.681+0000 7fa818ff9700 1 --2- 172.21.15.67:0/3433701517 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa80406c4e0 0x7fa80406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:00.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.681+0000 7fa81b7fe700 1 --2- 172.21.15.67:0/3433701517 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa80406c4e0 0x7fa80406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:00.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.681+0000 7fa818ff9700 1 -- 172.21.15.67:0/3433701517 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4358+0+0 (secure 0 0 0) 0x7fa80c014070 con 0x7fa81c101eb0 2024-04-22T23:19:00.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.681+0000 7fa81b7fe700 1 --2- 172.21.15.67:0/3433701517 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa80406c4e0 0x7fa80406e9a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa81000b6f0 tx=0x7fa810005940 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:00.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.685+0000 7fa818ff9700 1 -- 172.21.15.67:0/3433701517 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa80c0520e0 con 0x7fa81c101eb0 2024-04-22T23:19:00.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.885+0000 7fa821dec700 1 -- 172.21.15.67:0/3433701517 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fa81c19ece0 con 0x7fa81c101eb0 2024-04-22T23:19:00.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.886+0000 7fa818ff9700 1 -- 172.21.15.67:0/3433701517 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v38) v1 ==== 74+0+130 (secure 0 0 0) 0x7fa80c059d60 con 0x7fa81c101eb0 2024-04-22T23:19:00.888 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:19:00.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.888+0000 7fa821dec700 1 -- 172.21.15.67:0/3433701517 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa80406c4e0 msgr2=0x7fa80406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:00.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.888+0000 7fa821dec700 1 --2- 172.21.15.67:0/3433701517 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa80406c4e0 0x7fa80406e9a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa81000b6f0 tx=0x7fa810005940 comp rx=0 tx=0).stop 2024-04-22T23:19:00.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.888+0000 7fa821dec700 1 -- 172.21.15.67:0/3433701517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa81c101eb0 msgr2=0x7fa81c073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:00.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.888+0000 7fa821dec700 1 --2- 172.21.15.67:0/3433701517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa81c101eb0 0x7fa81c073180 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa80c00e920 tx=0x7fa80c00ec30 comp rx=0 tx=0).stop 2024-04-22T23:19:00.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.888+0000 7fa821dec700 1 -- 172.21.15.67:0/3433701517 shutdown_connections 2024-04-22T23:19:00.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.888+0000 7fa821dec700 1 --2- 172.21.15.67:0/3433701517 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa80406c4e0 0x7fa80406e9a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:00.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.888+0000 7fa821dec700 1 --2- 172.21.15.67:0/3433701517 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa81c100cb0 0x7fa81c074b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:00.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.888+0000 7fa821dec700 1 --2- 172.21.15.67:0/3433701517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa81c101eb0 0x7fa81c073180 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:00.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.888+0000 7fa821dec700 1 -- 172.21.15.67:0/3433701517 >> 172.21.15.67:0/3433701517 conn(0x7fa81c0fc230 msgr2=0x7fa81c1050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:00.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.888+0000 7fa821dec700 1 -- 172.21.15.67:0/3433701517 shutdown_connections 2024-04-22T23:19:00.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:00.888+0000 7fa821dec700 1 -- 172.21.15.67:0/3433701517 wait complete. 2024-04-22T23:19:00.942 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:00 smithi134 ceph-mon[34609]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-04-22T23:19:00.942 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:00 smithi134 ceph-mon[34609]: osdmap e38: 6 total, 5 up, 6 in 2024-04-22T23:19:00.942 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:00 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T23:19:00.942 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:00 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T23:19:00.942 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:00 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/3433701517' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T23:19:01.249 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:00 smithi067 ceph-mon[26453]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-04-22T23:19:01.249 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:00 smithi067 ceph-mon[26453]: osdmap e38: 6 total, 5 up, 6 in 2024-04-22T23:19:01.249 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:00 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T23:19:01.249 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:00 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T23:19:01.249 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:00 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/3433701517' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T23:19:01.582 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":38,"num_osds":6,"num_up_osds":5,"osd_up_since":1713827915,"num_in_osds":6,"osd_in_since":1713827921,"num_remapped_pgs":0} 2024-04-22T23:19:01.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:01 smithi134 ceph-mon[34609]: purged_snaps scrub starts 2024-04-22T23:19:01.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:01 smithi134 ceph-mon[34609]: purged_snaps scrub ok 2024-04-22T23:19:01.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:01 smithi134 ceph-mon[34609]: osd.5 [v2:172.21.15.134:6816/3846690247,v1:172.21.15.134:6817/3846690247] boot 2024-04-22T23:19:01.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:01 smithi134 ceph-mon[34609]: osdmap e39: 6 total, 6 up, 6 in 2024-04-22T23:19:01.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:01 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T23:19:01.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:01 smithi134 ceph-mon[34609]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:02.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:01 smithi067 ceph-mon[26453]: purged_snaps scrub starts 2024-04-22T23:19:02.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:01 smithi067 ceph-mon[26453]: purged_snaps scrub ok 2024-04-22T23:19:02.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:01 smithi067 ceph-mon[26453]: osd.5 [v2:172.21.15.134:6816/3846690247,v1:172.21.15.134:6817/3846690247] boot 2024-04-22T23:19:02.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:01 smithi067 ceph-mon[26453]: osdmap e39: 6 total, 6 up, 6 in 2024-04-22T23:19:02.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:01 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T23:19:02.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:01 smithi067 ceph-mon[26453]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:02.583 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-22T23:19:02.806 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:19:02.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:02 smithi134 ceph-mon[34609]: osdmap e40: 6 total, 6 up, 6 in 2024-04-22T23:19:02.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:02 smithi134 ceph-mon[34609]: Detected new or changed devices on smithi134 2024-04-22T23:19:02.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:02 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:19:02.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:02 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:19:02.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:02 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:19:02.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:02 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:19:02.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:02 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:19:02.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:02 smithi134 ceph-mon[34609]: Adjusting osd_memory_target on smithi134 to 4970M 2024-04-22T23:19:02.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:02 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:19:02.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:02 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:19:02.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:02 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:19:02.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:02 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:19:02.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:02 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:19:03.265 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:02 smithi067 ceph-mon[26453]: osdmap e40: 6 total, 6 up, 6 in 2024-04-22T23:19:03.265 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:02 smithi067 ceph-mon[26453]: Detected new or changed devices on smithi134 2024-04-22T23:19:03.265 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:02 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:19:03.265 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:02 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:19:03.265 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:02 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:19:03.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:02 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:19:03.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:02 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:19:03.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:02 smithi067 ceph-mon[26453]: Adjusting osd_memory_target on smithi134 to 4970M 2024-04-22T23:19:03.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:02 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:19:03.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:02 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:19:03.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:02 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:19:03.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:02 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:19:03.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:02 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:19:04.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:03 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:19:04.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:03 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:19:04.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:03 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:19:04.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:03 smithi134 ceph-mon[34609]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:04.252 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:03 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:19:04.252 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:03 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:19:04.252 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:03 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:19:04.252 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:03 smithi067 ceph-mon[26453]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:04.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.573+0000 7f33a5e13700 1 -- 172.21.15.67:0/1263735379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33a0101750 msgr2=0x7f33a0103b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:04.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.573+0000 7f33a5e13700 1 --2- 172.21.15.67:0/1263735379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33a0101750 0x7f33a0103b40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3390009910 tx=0x7f3390009c20 comp rx=0 tx=0).stop 2024-04-22T23:19:04.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.573+0000 7f33a5e13700 1 -- 172.21.15.67:0/1263735379 shutdown_connections 2024-04-22T23:19:04.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.573+0000 7f33a5e13700 1 --2- 172.21.15.67:0/1263735379 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f33a0104080 0x7f33a0106470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:04.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.573+0000 7f33a5e13700 1 --2- 172.21.15.67:0/1263735379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33a0101750 0x7f33a0103b40 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:04.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.573+0000 7f33a5e13700 1 -- 172.21.15.67:0/1263735379 >> 172.21.15.67:0/1263735379 conn(0x7f33a00fb160 msgr2=0x7f33a00fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:04.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.574+0000 7f33a5e13700 1 -- 172.21.15.67:0/1263735379 shutdown_connections 2024-04-22T23:19:04.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.574+0000 7f33a5e13700 1 -- 172.21.15.67:0/1263735379 wait complete. 2024-04-22T23:19:04.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.574+0000 7f33a5e13700 1 Processor -- start 2024-04-22T23:19:04.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.574+0000 7f33a5e13700 1 -- start start 2024-04-22T23:19:04.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.574+0000 7f33a5e13700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33a0101750 0x7f33a0196820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:04.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.574+0000 7f33a5e13700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f33a0104080 0x7f33a0196d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:04.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.574+0000 7f33a5e13700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a0197380 con 0x7f33a0101750 2024-04-22T23:19:04.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.574+0000 7f33a5e13700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a01974c0 con 0x7f33a0104080 2024-04-22T23:19:04.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.575+0000 7f339f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33a0101750 0x7f33a0196820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:04.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.575+0000 7f339f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33a0101750 0x7f33a0196820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40858/0 (socket says 172.21.15.67:40858) 2024-04-22T23:19:04.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.575+0000 7f339f7fe700 1 -- 172.21.15.67:0/4191332881 learned_addr learned my addr 172.21.15.67:0/4191332881 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:19:04.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.575+0000 7f339effd700 1 --2- 172.21.15.67:0/4191332881 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f33a0104080 0x7f33a0196d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:04.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.575+0000 7f339f7fe700 1 -- 172.21.15.67:0/4191332881 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f33a0104080 msgr2=0x7f33a0196d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:04.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.575+0000 7f339f7fe700 1 --2- 172.21.15.67:0/4191332881 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f33a0104080 0x7f33a0196d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:04.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.575+0000 7f339f7fe700 1 -- 172.21.15.67:0/4191332881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33900095f0 con 0x7f33a0101750 2024-04-22T23:19:04.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.575+0000 7f339f7fe700 1 --2- 172.21.15.67:0/4191332881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33a0101750 0x7f33a0196820 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3390006010 tx=0x7f339000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:04.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.575+0000 7f339cff9700 1 -- 172.21.15.67:0/4191332881 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f339001d070 con 0x7f33a0101750 2024-04-22T23:19:04.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.575+0000 7f339cff9700 1 -- 172.21.15.67:0/4191332881 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f339000f460 con 0x7f33a0101750 2024-04-22T23:19:04.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.575+0000 7f33a5e13700 1 -- 172.21.15.67:0/4191332881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33a019bf10 con 0x7f33a0101750 2024-04-22T23:19:04.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.575+0000 7f339cff9700 1 -- 172.21.15.67:0/4191332881 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3390021640 con 0x7f33a0101750 2024-04-22T23:19:04.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.575+0000 7f33a5e13700 1 -- 172.21.15.67:0/4191332881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33a019c380 con 0x7f33a0101750 2024-04-22T23:19:04.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.576+0000 7f33a5e13700 1 -- 172.21.15.67:0/4191332881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33a01906b0 con 0x7f33a0101750 2024-04-22T23:19:04.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.577+0000 7f339cff9700 1 -- 172.21.15.67:0/4191332881 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f339000fab0 con 0x7f33a0101750 2024-04-22T23:19:04.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.577+0000 7f339cff9700 1 --2- 172.21.15.67:0/4191332881 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f338806c4e0 0x7f338806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:04.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.577+0000 7f339cff9700 1 -- 172.21.15.67:0/4191332881 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f339008c8b0 con 0x7f33a0101750 2024-04-22T23:19:04.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.578+0000 7f339effd700 1 --2- 172.21.15.67:0/4191332881 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f338806c4e0 0x7f338806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:04.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.578+0000 7f339effd700 1 --2- 172.21.15.67:0/4191332881 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f338806c4e0 0x7f338806e9a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3394009a10 tx=0x7f3394009190 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:04.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.582+0000 7f339cff9700 1 -- 172.21.15.67:0/4191332881 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f339005ae60 con 0x7f33a0101750 2024-04-22T23:19:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.780+0000 7f33a5e13700 1 -- 172.21.15.67:0/4191332881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f33a002cfa0 con 0x7f33a0101750 2024-04-22T23:19:04.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.780+0000 7f339cff9700 1 -- 172.21.15.67:0/4191332881 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v40) v1 ==== 74+0+130 (secure 0 0 0) 0x7f339005a9f0 con 0x7f33a0101750 2024-04-22T23:19:04.782 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:19:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.782+0000 7f33a5e13700 1 -- 172.21.15.67:0/4191332881 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f338806c4e0 msgr2=0x7f338806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.782+0000 7f33a5e13700 1 --2- 172.21.15.67:0/4191332881 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f338806c4e0 0x7f338806e9a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3394009a10 tx=0x7f3394009190 comp rx=0 tx=0).stop 2024-04-22T23:19:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.782+0000 7f33a5e13700 1 -- 172.21.15.67:0/4191332881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33a0101750 msgr2=0x7f33a0196820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.782+0000 7f33a5e13700 1 --2- 172.21.15.67:0/4191332881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33a0101750 0x7f33a0196820 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3390006010 tx=0x7f339000bd70 comp rx=0 tx=0).stop 2024-04-22T23:19:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.783+0000 7f33a5e13700 1 -- 172.21.15.67:0/4191332881 shutdown_connections 2024-04-22T23:19:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.783+0000 7f33a5e13700 1 --2- 172.21.15.67:0/4191332881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33a0101750 0x7f33a0196820 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.783+0000 7f33a5e13700 1 --2- 172.21.15.67:0/4191332881 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f338806c4e0 0x7f338806e9a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.783+0000 7f33a5e13700 1 --2- 172.21.15.67:0/4191332881 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f33a0104080 0x7f33a0196d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.783+0000 7f33a5e13700 1 -- 172.21.15.67:0/4191332881 >> 172.21.15.67:0/4191332881 conn(0x7f33a00fb160 msgr2=0x7f33a00fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.783+0000 7f33a5e13700 1 -- 172.21.15.67:0/4191332881 shutdown_connections 2024-04-22T23:19:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:04.783+0000 7f33a5e13700 1 -- 172.21.15.67:0/4191332881 wait complete. 2024-04-22T23:19:05.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:04 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/4191332881' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T23:19:05.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:04 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/4191332881' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T23:19:05.413 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":40,"num_osds":6,"num_up_osds":6,"osd_up_since":1713827940,"num_in_osds":6,"osd_in_since":1713827921,"num_remapped_pgs":0} 2024-04-22T23:19:05.414 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T23:19:05.627 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:19:06.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:06 smithi067 ceph-mon[26453]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:06.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:06 smithi134 ceph-mon[34609]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:07.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.246+0000 7f6ac9e2a700 1 -- 172.21.15.67:0/3851011665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ac4100ea0 msgr2=0x7f6ac4101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:07.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.246+0000 7f6ac9e2a700 1 --2- 172.21.15.67:0/3851011665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ac4100ea0 0x7f6ac4101280 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4009910 tx=0x7f6ab4009c20 comp rx=0 tx=0).stop 2024-04-22T23:19:07.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.246+0000 7f6ac9e2a700 1 -- 172.21.15.67:0/3851011665 shutdown_connections 2024-04-22T23:19:07.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.246+0000 7f6ac9e2a700 1 --2- 172.21.15.67:0/3851011665 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ac4101850 0x7f6ac41058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:07.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.246+0000 7f6ac9e2a700 1 --2- 172.21.15.67:0/3851011665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ac4100ea0 0x7f6ac4101280 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:07.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.246+0000 7f6ac9e2a700 1 -- 172.21.15.67:0/3851011665 >> 172.21.15.67:0/3851011665 conn(0x7f6ac40fc750 msgr2=0x7f6ac40feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:07.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.246+0000 7f6ac9e2a700 1 -- 172.21.15.67:0/3851011665 shutdown_connections 2024-04-22T23:19:07.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.246+0000 7f6ac9e2a700 1 -- 172.21.15.67:0/3851011665 wait complete. 2024-04-22T23:19:07.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.246+0000 7f6ac9e2a700 1 Processor -- start 2024-04-22T23:19:07.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.247+0000 7f6ac9e2a700 1 -- start start 2024-04-22T23:19:07.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.247+0000 7f6ac9e2a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ac4100ea0 0x7f6ac4073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:07.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.247+0000 7f6ac9e2a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ac4101850 0x7f6ac40735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:07.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.247+0000 7f6ac9e2a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac4073b10 con 0x7f6ac4101850 2024-04-22T23:19:07.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.247+0000 7f6ac9e2a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac4073c50 con 0x7f6ac4100ea0 2024-04-22T23:19:07.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.247+0000 7f6ac2ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ac4101850 0x7f6ac40735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:07.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.247+0000 7f6ac2ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ac4101850 0x7f6ac40735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40864/0 (socket says 172.21.15.67:40864) 2024-04-22T23:19:07.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.247+0000 7f6ac2ffd700 1 -- 172.21.15.67:0/1904249078 learned_addr learned my addr 172.21.15.67:0/1904249078 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:19:07.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.247+0000 7f6ac37fe700 1 --2- 172.21.15.67:0/1904249078 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ac4100ea0 0x7f6ac4073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:07.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.247+0000 7f6ac2ffd700 1 -- 172.21.15.67:0/1904249078 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ac4100ea0 msgr2=0x7f6ac4073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:07.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.247+0000 7f6ac2ffd700 1 --2- 172.21.15.67:0/1904249078 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ac4100ea0 0x7f6ac4073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:07.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.247+0000 7f6ac2ffd700 1 -- 172.21.15.67:0/1904249078 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab40095f0 con 0x7f6ac4101850 2024-04-22T23:19:07.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.247+0000 7f6ac37fe700 1 --2- 172.21.15.67:0/1904249078 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ac4100ea0 0x7f6ac4073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:19:07.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.247+0000 7f6ac2ffd700 1 --2- 172.21.15.67:0/1904249078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ac4101850 0x7f6ac40735d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6ab800c960 tx=0x7f6ab800cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:07.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.247+0000 7f6ac0ff9700 1 -- 172.21.15.67:0/1904249078 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ab8007a70 con 0x7f6ac4101850 2024-04-22T23:19:07.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.248+0000 7f6ac9e2a700 1 -- 172.21.15.67:0/1904249078 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ac41a2af0 con 0x7f6ac4101850 2024-04-22T23:19:07.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.248+0000 7f6ac0ff9700 1 -- 172.21.15.67:0/1904249078 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6ab8007bd0 con 0x7f6ac4101850 2024-04-22T23:19:07.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.248+0000 7f6ac0ff9700 1 -- 172.21.15.67:0/1904249078 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ab8018630 con 0x7f6ac4101850 2024-04-22T23:19:07.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.248+0000 7f6ac9e2a700 1 -- 172.21.15.67:0/1904249078 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ac41a2e50 con 0x7f6ac4101850 2024-04-22T23:19:07.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.248+0000 7f6ac9e2a700 1 -- 172.21.15.67:0/1904249078 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ac4109240 con 0x7f6ac4101850 2024-04-22T23:19:07.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.249+0000 7f6ac0ff9700 1 -- 172.21.15.67:0/1904249078 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f6ab800fa90 con 0x7f6ac4101850 2024-04-22T23:19:07.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.249+0000 7f6ac0ff9700 1 --2- 172.21.15.67:0/1904249078 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f6aac06c4e0 0x7f6aac06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:07.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.250+0000 7f6ac0ff9700 1 -- 172.21.15.67:0/1904249078 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f6ab808b510 con 0x7f6ac4101850 2024-04-22T23:19:07.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.250+0000 7f6ac37fe700 1 --2- 172.21.15.67:0/1904249078 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f6aac06c4e0 0x7f6aac06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:07.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.250+0000 7f6ac37fe700 1 --2- 172.21.15.67:0/1904249078 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f6aac06c4e0 0x7f6aac06e9a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6ab400b5c0 tx=0x7f6ab4005480 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:07.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.254+0000 7f6ac0ff9700 1 -- 172.21.15.67:0/1904249078 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6ab8059ac0 con 0x7f6ac4101850 2024-04-22T23:19:07.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.456+0000 7f6ac9e2a700 1 -- 172.21.15.67:0/1904249078 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ac404f2e0 con 0x7f6ac4101850 2024-04-22T23:19:07.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.457+0000 7f6ac0ff9700 1 -- 172.21.15.67:0/1904249078 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11325 (secure 0 0 0) 0x7f6ab8059650 con 0x7f6ac4101850 2024-04-22T23:19:07.458 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:19:07.459 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":40,"fsid":"d46c346e-00fd-11ef-bc93-c7b262605968","created":"2024-04-22T23:13:18.687917+0000","modified":"2024-04-22T23:19:01.899867+0000","last_up_change":"2024-04-22T23:19:00.895098+0000","last_in_change":"2024-04-22T23:18:41.696935+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T23:17:47.912850+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"25c9acd1-5743-4dc0-aee0-c7bb8884c0a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":268472413},{"type":"v1","addr":"172.21.15.67:6803","nonce":268472413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":268472413},{"type":"v1","addr":"172.21.15.67:6805","nonce":268472413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":268472413},{"type":"v1","addr":"172.21.15.67:6809","nonce":268472413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":268472413},{"type":"v1","addr":"172.21.15.67:6807","nonce":268472413}]},"public_addr":"172.21.15.67:6803/268472413","cluster_addr":"172.21.15.67:6805/268472413","heartbeat_back_addr":"172.21.15.67:6809/268472413","heartbeat_front_addr":"172.21.15.67:6807/268472413","state":["exists","up"]},{"osd":1,"uuid":"8be83014-01af-44d3-ab2f-b27840506e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3382820294},{"type":"v1","addr":"172.21.15.67:6811","nonce":3382820294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3382820294},{"type":"v1","addr":"172.21.15.67:6813","nonce":3382820294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3382820294},{"type":"v1","addr":"172.21.15.67:6817","nonce":3382820294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3382820294},{"type":"v1","addr":"172.21.15.67:6815","nonce":3382820294}]},"public_addr":"172.21.15.67:6811/3382820294","cluster_addr":"172.21.15.67:6813/3382820294","heartbeat_back_addr":"172.21.15.67:6817/3382820294","heartbeat_front_addr":"172.21.15.67:6815/3382820294","state":["exists","up"]},{"osd":2,"uuid":"ab1cbcda-14dd-4c24-876b-25ad5d4d8563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3835340581},{"type":"v1","addr":"172.21.15.67:6819","nonce":3835340581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3835340581},{"type":"v1","addr":"172.21.15.67:6821","nonce":3835340581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3835340581},{"type":"v1","addr":"172.21.15.67:6825","nonce":3835340581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3835340581},{"type":"v1","addr":"172.21.15.67:6823","nonce":3835340581}]},"public_addr":"172.21.15.67:6819/3835340581","cluster_addr":"172.21.15.67:6821/3835340581","heartbeat_back_addr":"172.21.15.67:6825/3835340581","heartbeat_front_addr":"172.21.15.67:6823/3835340581","state":["exists","up"]},{"osd":3,"uuid":"1a30b6c3-95bf-4b19-883b-1271ff359e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":2235786001},{"type":"v1","addr":"172.21.15.134:6801","nonce":2235786001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":2235786001},{"type":"v1","addr":"172.21.15.134:6803","nonce":2235786001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":2235786001},{"type":"v1","addr":"172.21.15.134:6807","nonce":2235786001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":2235786001},{"type":"v1","addr":"172.21.15.134:6805","nonce":2235786001}]},"public_addr":"172.21.15.134:6801/2235786001","cluster_addr":"172.21.15.134:6803/2235786001","heartbeat_back_addr":"172.21.15.134:6807/2235786001","heartbeat_front_addr":"172.21.15.134:6805/2235786001","state":["exists","up"]},{"osd":4,"uuid":"53368fc1-28fd-4622-bc9c-1d8901b06bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":668644810},{"type":"v1","addr":"172.21.15.134:6809","nonce":668644810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":668644810},{"type":"v1","addr":"172.21.15.134:6811","nonce":668644810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":668644810},{"type":"v1","addr":"172.21.15.134:6815","nonce":668644810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":668644810},{"type":"v1","addr":"172.21.15.134:6813","nonce":668644810}]},"public_addr":"172.21.15.134:6809/668644810","cluster_addr":"172.21.15.134:6811/668644810","heartbeat_back_addr":"172.21.15.134:6815/668644810","heartbeat_front_addr":"172.21.15.134:6813/668644810","state":["exists","up"]},{"osd":5,"uuid":"31a885f7-8f91-4b6a-922a-065753f70a95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3846690247},{"type":"v1","addr":"172.21.15.134:6817","nonce":3846690247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":3846690247},{"type":"v1","addr":"172.21.15.134:6819","nonce":3846690247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":3846690247},{"type":"v1","addr":"172.21.15.134:6823","nonce":3846690247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":3846690247},{"type":"v1","addr":"172.21.15.134:6821","nonce":3846690247}]},"public_addr":"172.21.15.134:6817/3846690247","cluster_addr":"172.21.15.134:6819/3846690247","heartbeat_back_addr":"172.21.15.134:6823/3846690247","heartbeat_front_addr":"172.21.15.134:6821/3846690247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:16:51.179651+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:17:18.694683+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:17:45.368963+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:18:08.585046+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:18:33.412804+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:18:58.909758+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6800/2590792187":"2024-04-23T23:15:37.396154+0000","172.21.15.67:0/1125624760":"2024-04-23T23:15:37.396154+0000","172.21.15.67:0/1189311107":"2024-04-23T23:15:37.396154+0000","172.21.15.67:6801/2590792187":"2024-04-23T23:15:37.396154+0000","172.21.15.67:0/2137374599":"2024-04-23T23:15:37.396154+0000","172.21.15.67:0/4165654004":"2024-04-23T23:14:29.214977+0000","172.21.15.67:6801/3568049827":"2024-04-23T23:14:29.214977+0000","172.21.15.67:6801/2451437776":"2024-04-23T23:13:56.301645+0000","172.21.15.67:0/2052573085":"2024-04-23T23:14:29.214977+0000","172.21.15.67:0/853292259":"2024-04-23T23:13:56.301645+0000","172.21.15.67:0/895378312":"2024-04-23T23:13:56.301645+0000","172.21.15.67:0/3387250764":"2024-04-23T23:13:56.301645+0000","172.21.15.67:0/2665297739":"2024-04-23T23:14:29.214977+0000","172.21.15.67:6800/3568049827":"2024-04-23T23:14:29.214977+0000","172.21.15.67:6800/2451437776":"2024-04-23T23:13:56.301645+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T23:19:07.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.459+0000 7f6ac9e2a700 1 -- 172.21.15.67:0/1904249078 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f6aac06c4e0 msgr2=0x7f6aac06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:07.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.459+0000 7f6ac9e2a700 1 --2- 172.21.15.67:0/1904249078 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f6aac06c4e0 0x7f6aac06e9a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6ab400b5c0 tx=0x7f6ab4005480 comp rx=0 tx=0).stop 2024-04-22T23:19:07.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.459+0000 7f6ac9e2a700 1 -- 172.21.15.67:0/1904249078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ac4101850 msgr2=0x7f6ac40735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:07.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.459+0000 7f6ac9e2a700 1 --2- 172.21.15.67:0/1904249078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ac4101850 0x7f6ac40735d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6ab800c960 tx=0x7f6ab800cd20 comp rx=0 tx=0).stop 2024-04-22T23:19:07.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.459+0000 7f6ac9e2a700 1 -- 172.21.15.67:0/1904249078 shutdown_connections 2024-04-22T23:19:07.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.459+0000 7f6ac9e2a700 1 --2- 172.21.15.67:0/1904249078 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f6aac06c4e0 0x7f6aac06e9a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:07.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.459+0000 7f6ac9e2a700 1 --2- 172.21.15.67:0/1904249078 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ac4100ea0 0x7f6ac4073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:07.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.459+0000 7f6ac9e2a700 1 --2- 172.21.15.67:0/1904249078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ac4101850 0x7f6ac40735d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:07.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.459+0000 7f6ac9e2a700 1 -- 172.21.15.67:0/1904249078 >> 172.21.15.67:0/1904249078 conn(0x7f6ac40fc750 msgr2=0x7f6ac40fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:07.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.459+0000 7f6ac9e2a700 1 -- 172.21.15.67:0/1904249078 shutdown_connections 2024-04-22T23:19:07.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:07.459+0000 7f6ac9e2a700 1 -- 172.21.15.67:0/1904249078 wait complete. 2024-04-22T23:19:07.762 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:07 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:19:07.762 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:07 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/1904249078' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T23:19:07.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:07 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:19:07.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:07 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/1904249078' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T23:19:08.137 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T23:17:47.912850+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T23:19:08.137 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-22T23:19:08.350 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:19:08.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:08 smithi067 ceph-mon[26453]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:08.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:08 smithi134 ceph-mon[34609]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:10.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.099+0000 7f116f752700 1 -- 172.21.15.67:0/3339178317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11680ffda0 msgr2=0x7f1168100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:10.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.099+0000 7f116f752700 1 --2- 172.21.15.67:0/3339178317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11680ffda0 0x7f1168100180 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1164009910 tx=0x7f1164009c20 comp rx=0 tx=0).stop 2024-04-22T23:19:10.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.099+0000 7f116f752700 1 -- 172.21.15.67:0/3339178317 shutdown_connections 2024-04-22T23:19:10.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.099+0000 7f116f752700 1 --2- 172.21.15.67:0/3339178317 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1168100750 0x7f11681047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:10.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.099+0000 7f116f752700 1 --2- 172.21.15.67:0/3339178317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11680ffda0 0x7f1168100180 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:10.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.099+0000 7f116f752700 1 -- 172.21.15.67:0/3339178317 >> 172.21.15.67:0/3339178317 conn(0x7f11680fb650 msgr2=0x7f11680fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:10.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.099+0000 7f116f752700 1 -- 172.21.15.67:0/3339178317 shutdown_connections 2024-04-22T23:19:10.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.099+0000 7f116f752700 1 -- 172.21.15.67:0/3339178317 wait complete. 2024-04-22T23:19:10.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.100+0000 7f116f752700 1 Processor -- start 2024-04-22T23:19:10.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.100+0000 7f116f752700 1 -- start start 2024-04-22T23:19:10.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.100+0000 7f116f752700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11680ffda0 0x7f116819c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:10.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.100+0000 7f116f752700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1168100750 0x7f116819cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:10.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.100+0000 7f116f752700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116819d2c0 con 0x7f11680ffda0 2024-04-22T23:19:10.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.100+0000 7f116f752700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1168196770 con 0x7f1168100750 2024-04-22T23:19:10.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.100+0000 7f116d4ee700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11680ffda0 0x7f116819c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:10.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.100+0000 7f116d4ee700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11680ffda0 0x7f116819c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48374/0 (socket says 172.21.15.67:48374) 2024-04-22T23:19:10.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.100+0000 7f116d4ee700 1 -- 172.21.15.67:0/82916972 learned_addr learned my addr 172.21.15.67:0/82916972 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:19:10.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.100+0000 7f116d4ee700 1 -- 172.21.15.67:0/82916972 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1168100750 msgr2=0x7f116819cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:10.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.100+0000 7f116d4ee700 1 --2- 172.21.15.67:0/82916972 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1168100750 0x7f116819cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:10.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.100+0000 7f116d4ee700 1 -- 172.21.15.67:0/82916972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11640095f0 con 0x7f11680ffda0 2024-04-22T23:19:10.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.100+0000 7f116d4ee700 1 --2- 172.21.15.67:0/82916972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11680ffda0 0x7f116819c6f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1164006010 tx=0x7f116400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:10.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.101+0000 7f115e7fc700 1 -- 172.21.15.67:0/82916972 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f116401d070 con 0x7f11680ffda0 2024-04-22T23:19:10.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.101+0000 7f115e7fc700 1 -- 172.21.15.67:0/82916972 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f116400f460 con 0x7f11680ffda0 2024-04-22T23:19:10.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.101+0000 7f116f752700 1 -- 172.21.15.67:0/82916972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11681969f0 con 0x7f11680ffda0 2024-04-22T23:19:10.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.101+0000 7f115e7fc700 1 -- 172.21.15.67:0/82916972 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1164021640 con 0x7f11680ffda0 2024-04-22T23:19:10.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.101+0000 7f116f752700 1 -- 172.21.15.67:0/82916972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1168196ee0 con 0x7f11680ffda0 2024-04-22T23:19:10.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.102+0000 7f116f752700 1 -- 172.21.15.67:0/82916972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1168108140 con 0x7f11680ffda0 2024-04-22T23:19:10.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.102+0000 7f115e7fc700 1 -- 172.21.15.67:0/82916972 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f116400faf0 con 0x7f11680ffda0 2024-04-22T23:19:10.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.103+0000 7f115e7fc700 1 --2- 172.21.15.67:0/82916972 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f115406c4e0 0x7f115406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:10.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.103+0000 7f115e7fc700 1 -- 172.21.15.67:0/82916972 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f116408c870 con 0x7f11680ffda0 2024-04-22T23:19:10.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.103+0000 7f116cced700 1 --2- 172.21.15.67:0/82916972 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f115406c4e0 0x7f115406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:10.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.103+0000 7f116cced700 1 --2- 172.21.15.67:0/82916972 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f115406c4e0 0x7f115406e9a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1168197be0 tx=0x7f1158006d20 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:10.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.107+0000 7f115e7fc700 1 -- 172.21.15.67:0/82916972 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f116405ae20 con 0x7f11680ffda0 2024-04-22T23:19:10.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.302+0000 7f116f752700 1 -- 172.21.15.67:0/82916972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1168107b20 con 0x7f11680ffda0 2024-04-22T23:19:10.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.302+0000 7f115e7fc700 1 -- 172.21.15.67:0/82916972 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7f116405a9b0 con 0x7f11680ffda0 2024-04-22T23:19:10.304 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-22T23:19:10.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.304+0000 7f116f752700 1 -- 172.21.15.67:0/82916972 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f115406c4e0 msgr2=0x7f115406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:10.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.304+0000 7f116f752700 1 --2- 172.21.15.67:0/82916972 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f115406c4e0 0x7f115406e9a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1168197be0 tx=0x7f1158006d20 comp rx=0 tx=0).stop 2024-04-22T23:19:10.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.304+0000 7f116f752700 1 -- 172.21.15.67:0/82916972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11680ffda0 msgr2=0x7f116819c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:10.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.304+0000 7f116f752700 1 --2- 172.21.15.67:0/82916972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11680ffda0 0x7f116819c6f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1164006010 tx=0x7f116400be20 comp rx=0 tx=0).stop 2024-04-22T23:19:10.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.304+0000 7f116f752700 1 -- 172.21.15.67:0/82916972 shutdown_connections 2024-04-22T23:19:10.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.304+0000 7f116f752700 1 --2- 172.21.15.67:0/82916972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11680ffda0 0x7f116819c6f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:10.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.304+0000 7f116f752700 1 --2- 172.21.15.67:0/82916972 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f115406c4e0 0x7f115406e9a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:10.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.304+0000 7f116f752700 1 --2- 172.21.15.67:0/82916972 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1168100750 0x7f116819cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:10.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.304+0000 7f116f752700 1 -- 172.21.15.67:0/82916972 >> 172.21.15.67:0/82916972 conn(0x7f11680fb650 msgr2=0x7f11680fcdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:10.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.305+0000 7f116f752700 1 -- 172.21.15.67:0/82916972 shutdown_connections 2024-04-22T23:19:10.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:10.305+0000 7f116f752700 1 -- 172.21.15.67:0/82916972 wait complete. 2024-04-22T23:19:10.593 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:10 smithi067 ceph-mon[26453]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:10.593 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:10 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/82916972' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T23:19:10.797 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:10 smithi134 ceph-mon[34609]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:10.797 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:10 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/82916972' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T23:19:11.018 INFO:tasks.cephadm:Setting up client nodes... 2024-04-22T23:19:11.018 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-22T23:19:11.232 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:19:12.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:12.739+0000 7f72962ad700 1 -- 172.21.15.67:0/419063160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72901040d0 msgr2=0x7f72901064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:12.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:12.739+0000 7f72962ad700 1 --2- 172.21.15.67:0/419063160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72901040d0 0x7f72901064c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7278009910 tx=0x7f7278009c20 comp rx=0 tx=0).stop 2024-04-22T23:19:12.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:12.739+0000 7f72962ad700 1 -- 172.21.15.67:0/419063160 shutdown_connections 2024-04-22T23:19:12.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:12.739+0000 7f72962ad700 1 --2- 172.21.15.67:0/419063160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72901040d0 0x7f72901064c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:12.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:12.739+0000 7f72962ad700 1 --2- 172.21.15.67:0/419063160 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f72901017a0 0x7f7290103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:12.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:12.739+0000 7f72962ad700 1 -- 172.21.15.67:0/419063160 >> 172.21.15.67:0/419063160 conn(0x7f72900fb190 msgr2=0x7f72900fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:12.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:12.739+0000 7f72962ad700 1 -- 172.21.15.67:0/419063160 shutdown_connections 2024-04-22T23:19:12.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:12.739+0000 7f72962ad700 1 -- 172.21.15.67:0/419063160 wait complete. 2024-04-22T23:19:12.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:12.740+0000 7f72962ad700 1 Processor -- start 2024-04-22T23:19:12.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:12.740+0000 7f72962ad700 1 -- start start 2024-04-22T23:19:12.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:12.740+0000 7f72962ad700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72901017a0 0x7f7290101100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:12.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:12.740+0000 7f72962ad700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f72901040d0 0x7f72900ff750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:12.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:12.740+0000 7f72962ad700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72900ffc90 con 0x7f72901017a0 2024-04-22T23:19:12.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:12.740+0000 7f72962ad700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72900ffdd0 con 0x7f72901040d0 2024-04-22T23:19:12.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:12.740+0000 7f728ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72901017a0 0x7f7290101100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:12.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:12.740+0000 7f728ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72901017a0 0x7f7290101100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48390/0 (socket says 172.21.15.67:48390) 2024-04-22T23:19:12.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:12.740+0000 7f728ffff700 1 -- 172.21.15.67:0/3217439101 learned_addr learned my addr 172.21.15.67:0/3217439101 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:19:12.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:12.740+0000 7f7287fff700 1 --2- 172.21.15.67:0/3217439101 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f72901040d0 0x7f72900ff750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:12.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:12.740+0000 7f728ffff700 1 -- 172.21.15.67:0/3217439101 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f72901040d0 msgr2=0x7f72900ff750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:12.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:12.740+0000 7f728ffff700 1 --2- 172.21.15.67:0/3217439101 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f72901040d0 0x7f72900ff750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:12.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:12.740+0000 7f728ffff700 1 -- 172.21.15.67:0/3217439101 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72780095f0 con 0x7f72901017a0 2024-04-22T23:19:12.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:12.741+0000 7f728ffff700 1 --2- 172.21.15.67:0/3217439101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72901017a0 0x7f7290101100 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f728000e920 tx=0x7f728000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:12.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:12.741+0000 7f728dffb700 1 -- 172.21.15.67:0/3217439101 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f728000c9d0 con 0x7f72901017a0 2024-04-22T23:19:12.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:12.741+0000 7f728dffb700 1 -- 172.21.15.67:0/3217439101 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7280010430 con 0x7f72901017a0 2024-04-22T23:19:12.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:12.741+0000 7f72962ad700 1 -- 172.21.15.67:0/3217439101 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72901000b0 con 0x7f72901017a0 2024-04-22T23:19:12.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:12.741+0000 7f728dffb700 1 -- 172.21.15.67:0/3217439101 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7280018650 con 0x7f72901017a0 2024-04-22T23:19:12.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:12.741+0000 7f72962ad700 1 -- 172.21.15.67:0/3217439101 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f729019e4b0 con 0x7f72901017a0 2024-04-22T23:19:12.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:12.742+0000 7f72962ad700 1 -- 172.21.15.67:0/3217439101 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7290190610 con 0x7f72901017a0 2024-04-22T23:19:12.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:12.742+0000 7f728dffb700 1 -- 172.21.15.67:0/3217439101 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f72800105a0 con 0x7f72901017a0 2024-04-22T23:19:12.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:12.743+0000 7f728dffb700 1 --2- 172.21.15.67:0/3217439101 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f727c06c4e0 0x7f727c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:12.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:12.743+0000 7f728dffb700 1 -- 172.21.15.67:0/3217439101 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f7280014070 con 0x7f72901017a0 2024-04-22T23:19:12.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:12.743+0000 7f7287fff700 1 --2- 172.21.15.67:0/3217439101 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f727c06c4e0 0x7f727c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:12.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:12.743+0000 7f7287fff700 1 --2- 172.21.15.67:0/3217439101 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f727c06c4e0 0x7f727c06e9a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7278009fd0 tx=0x7f7278005940 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:12.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:12.747+0000 7f728dffb700 1 -- 172.21.15.67:0/3217439101 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f72800520e0 con 0x7f72901017a0 2024-04-22T23:19:12.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:12 smithi067 ceph-mon[26453]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:12.800 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:12 smithi134 ceph-mon[34609]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:13.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:13.027+0000 7f72962ad700 1 -- 172.21.15.67:0/3217439101 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f729002cc70 con 0x7f72901017a0 2024-04-22T23:19:13.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:13.031+0000 7f728dffb700 1 -- 172.21.15.67:0/3217439101 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) v1 ==== 170+0+59 (secure 0 0 0) 0x7f7280059b20 con 0x7f72901017a0 2024-04-22T23:19:13.032 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-04-22T23:19:13.032 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQBx8CZmd7m4ARAAoD9zW1I8Xpehng/EF7aSIQ== 2024-04-22T23:19:13.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:13.033+0000 7f72962ad700 1 -- 172.21.15.67:0/3217439101 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f727c06c4e0 msgr2=0x7f727c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:13.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:13.033+0000 7f72962ad700 1 --2- 172.21.15.67:0/3217439101 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f727c06c4e0 0x7f727c06e9a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7278009fd0 tx=0x7f7278005940 comp rx=0 tx=0).stop 2024-04-22T23:19:13.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:13.033+0000 7f72962ad700 1 -- 172.21.15.67:0/3217439101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72901017a0 msgr2=0x7f7290101100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:13.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:13.033+0000 7f72962ad700 1 --2- 172.21.15.67:0/3217439101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72901017a0 0x7f7290101100 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f728000e920 tx=0x7f728000ec30 comp rx=0 tx=0).stop 2024-04-22T23:19:13.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:13.033+0000 7f72962ad700 1 -- 172.21.15.67:0/3217439101 shutdown_connections 2024-04-22T23:19:13.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:13.033+0000 7f72962ad700 1 --2- 172.21.15.67:0/3217439101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72901017a0 0x7f7290101100 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:13.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:13.033+0000 7f72962ad700 1 --2- 172.21.15.67:0/3217439101 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f727c06c4e0 0x7f727c06e9a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:13.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:13.033+0000 7f72962ad700 1 --2- 172.21.15.67:0/3217439101 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f72901040d0 0x7f72900ff750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:13.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:13.033+0000 7f72962ad700 1 -- 172.21.15.67:0/3217439101 >> 172.21.15.67:0/3217439101 conn(0x7f72900fb190 msgr2=0x7f72900fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:13.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:13.033+0000 7f72962ad700 1 -- 172.21.15.67:0/3217439101 shutdown_connections 2024-04-22T23:19:13.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:13.033+0000 7f72962ad700 1 -- 172.21.15.67:0/3217439101 wait complete. 2024-04-22T23:19:13.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:13 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/3217439101' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T23:19:13.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:13 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/3217439101' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T23:19:13.839 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-22T23:19:13.839 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-22T23:19:13.839 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-22T23:19:13.876 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-22T23:19:13.904 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:13 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/3217439101' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T23:19:13.904 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:13 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/3217439101' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T23:19:14.093 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-22T23:19:14.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:14 smithi067 ceph-mon[26453]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:14.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:14 smithi134 ceph-mon[34609]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:15.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.674+0000 7f23d4289700 1 -- 172.21.15.134:0/925467013 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23cc101eb0 msgr2=0x7f23cc102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:15.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.674+0000 7f23d4289700 1 --2- 172.21.15.134:0/925467013 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23cc101eb0 0x7f23cc102310 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f23bc009910 tx=0x7f23bc009c20 comp rx=0 tx=0).stop 2024-04-22T23:19:15.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.674+0000 7f23d4289700 1 -- 172.21.15.134:0/925467013 shutdown_connections 2024-04-22T23:19:15.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.674+0000 7f23d4289700 1 --2- 172.21.15.134:0/925467013 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23cc101eb0 0x7f23cc102310 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:15.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.674+0000 7f23d4289700 1 --2- 172.21.15.134:0/925467013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23cc100cb0 0x7f23cc1010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:15.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.674+0000 7f23d4289700 1 -- 172.21.15.134:0/925467013 >> 172.21.15.134:0/925467013 conn(0x7f23cc0fc230 msgr2=0x7f23cc0fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:15.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.674+0000 7f23d4289700 1 -- 172.21.15.134:0/925467013 shutdown_connections 2024-04-22T23:19:15.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.674+0000 7f23d4289700 1 -- 172.21.15.134:0/925467013 wait complete. 2024-04-22T23:19:15.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.675+0000 7f23d4289700 1 Processor -- start 2024-04-22T23:19:15.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.675+0000 7f23d4289700 1 -- start start 2024-04-22T23:19:15.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.675+0000 7f23d4289700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23cc100cb0 0x7f23cc1964e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:15.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.675+0000 7f23d4289700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23cc101eb0 0x7f23cc196a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:15.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.675+0000 7f23d4289700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23cc197040 con 0x7f23cc100cb0 2024-04-22T23:19:15.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.675+0000 7f23d4289700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23cc197180 con 0x7f23cc101eb0 2024-04-22T23:19:15.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.675+0000 7f23d1824700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23cc101eb0 0x7f23cc196a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:15.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.675+0000 7f23d1824700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23cc101eb0 0x7f23cc196a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:44554/0 (socket says 172.21.15.134:44554) 2024-04-22T23:19:15.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.675+0000 7f23d1824700 1 -- 172.21.15.134:0/1083254585 learned_addr learned my addr 172.21.15.134:0/1083254585 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-04-22T23:19:15.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.675+0000 7f23d2025700 1 --2- 172.21.15.134:0/1083254585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23cc100cb0 0x7f23cc1964e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:15.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.675+0000 7f23d1824700 1 -- 172.21.15.134:0/1083254585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23cc100cb0 msgr2=0x7f23cc1964e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:15.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.675+0000 7f23d1824700 1 --2- 172.21.15.134:0/1083254585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23cc100cb0 0x7f23cc1964e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:15.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.675+0000 7f23d1824700 1 -- 172.21.15.134:0/1083254585 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23bc0095f0 con 0x7f23cc101eb0 2024-04-22T23:19:15.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.675+0000 7f23d1824700 1 --2- 172.21.15.134:0/1083254585 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23cc101eb0 0x7f23cc196a20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f23bc005b40 tx=0x7f23bc00bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:15.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.676+0000 7f23c37fe700 1 -- 172.21.15.134:0/1083254585 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23bc01d070 con 0x7f23cc101eb0 2024-04-22T23:19:15.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.676+0000 7f23c37fe700 1 -- 172.21.15.134:0/1083254585 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f23bc00f460 con 0x7f23cc101eb0 2024-04-22T23:19:15.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.676+0000 7f23d4289700 1 -- 172.21.15.134:0/1083254585 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23cc19bbd0 con 0x7f23cc101eb0 2024-04-22T23:19:15.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.676+0000 7f23c37fe700 1 -- 172.21.15.134:0/1083254585 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23bc021620 con 0x7f23cc101eb0 2024-04-22T23:19:15.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.676+0000 7f23d4289700 1 -- 172.21.15.134:0/1083254585 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23cc19c0c0 con 0x7f23cc101eb0 2024-04-22T23:19:15.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.677+0000 7f23d4289700 1 -- 172.21.15.134:0/1083254585 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23cc109810 con 0x7f23cc101eb0 2024-04-22T23:19:15.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.677+0000 7f23c37fe700 1 -- 172.21.15.134:0/1083254585 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f23bc00fa40 con 0x7f23cc101eb0 2024-04-22T23:19:15.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.678+0000 7f23c37fe700 1 --2- 172.21.15.134:0/1083254585 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f23b806c4e0 0x7f23b806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:15.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.678+0000 7f23c37fe700 1 -- 172.21.15.134:0/1083254585 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f23bc08c710 con 0x7f23cc101eb0 2024-04-22T23:19:15.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.678+0000 7f23d2025700 1 --2- 172.21.15.134:0/1083254585 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f23b806c4e0 0x7f23b806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:15.681 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.678+0000 7f23d2025700 1 --2- 172.21.15.134:0/1083254585 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f23b806c4e0 0x7f23b806e9a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f23cc101d10 tx=0x7f23c800b410 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:15.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.682+0000 7f23c37fe700 1 -- 172.21.15.134:0/1083254585 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f23bc05acc0 con 0x7f23cc101eb0 2024-04-22T23:19:15.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.945+0000 7f23d4289700 1 -- 172.21.15.134:0/1083254585 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f23cc109a00 con 0x7f23cc101eb0 2024-04-22T23:19:15.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.949+0000 7f23c37fe700 1 -- 172.21.15.134:0/1083254585 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (secure 0 0 0) 0x7f23bc05a850 con 0x7f23cc101eb0 2024-04-22T23:19:15.951 INFO:teuthology.orchestra.run.smithi134.stdout:[client.1] 2024-04-22T23:19:15.951 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQBz8CZmsJR0OBAAD3LGWcuUITN7wmvKi8UFkA== 2024-04-22T23:19:15.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.951+0000 7f23d4289700 1 -- 172.21.15.134:0/1083254585 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f23b806c4e0 msgr2=0x7f23b806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:15.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.951+0000 7f23d4289700 1 --2- 172.21.15.134:0/1083254585 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f23b806c4e0 0x7f23b806e9a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f23cc101d10 tx=0x7f23c800b410 comp rx=0 tx=0).stop 2024-04-22T23:19:15.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.951+0000 7f23d4289700 1 -- 172.21.15.134:0/1083254585 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23cc101eb0 msgr2=0x7f23cc196a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:15.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.951+0000 7f23d4289700 1 --2- 172.21.15.134:0/1083254585 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23cc101eb0 0x7f23cc196a20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f23bc005b40 tx=0x7f23bc00bd70 comp rx=0 tx=0).stop 2024-04-22T23:19:15.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.952+0000 7f23d4289700 1 -- 172.21.15.134:0/1083254585 shutdown_connections 2024-04-22T23:19:15.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.952+0000 7f23d4289700 1 --2- 172.21.15.134:0/1083254585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23cc100cb0 0x7f23cc1964e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:15.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.952+0000 7f23d4289700 1 --2- 172.21.15.134:0/1083254585 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f23b806c4e0 0x7f23b806e9a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:15.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.952+0000 7f23d4289700 1 --2- 172.21.15.134:0/1083254585 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23cc101eb0 0x7f23cc196a20 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:15.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.952+0000 7f23d4289700 1 -- 172.21.15.134:0/1083254585 >> 172.21.15.134:0/1083254585 conn(0x7f23cc0fc230 msgr2=0x7f23cc1050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:15.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.952+0000 7f23d4289700 1 -- 172.21.15.134:0/1083254585 shutdown_connections 2024-04-22T23:19:15.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:19:15.952+0000 7f23d4289700 1 -- 172.21.15.134:0/1083254585 wait complete. 2024-04-22T23:19:16.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:16 smithi134 ceph-mon[34609]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:16.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:16 smithi134 ceph-mon[34609]: from='client.? 172.21.15.134:0/1083254585' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T23:19:16.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:16 smithi134 ceph-mon[34609]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T23:19:16.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:16 smithi134 ceph-mon[34609]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T23:19:16.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:16 smithi067 ceph-mon[26453]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:16.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:16 smithi067 ceph-mon[26453]: from='client.? 172.21.15.134:0/1083254585' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T23:19:16.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:16 smithi067 ceph-mon[26453]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T23:19:16.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:16 smithi067 ceph-mon[26453]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T23:19:16.609 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-22T23:19:16.609 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-22T23:19:16.609 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-22T23:19:16.684 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-22T23:19:16.684 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-22T23:19:16.685 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-22T23:19:16.897 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:19:18.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.561+0000 7ff3ba913700 1 -- 172.21.15.67:0/4030074275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b4102700 msgr2=0x7ff3b410abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:18.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.561+0000 7ff3ba913700 1 --2- 172.21.15.67:0/4030074275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b4102700 0x7ff3b410abf0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8009910 tx=0x7ff3a8009c20 comp rx=0 tx=0).stop 2024-04-22T23:19:18.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.561+0000 7ff3ba913700 1 -- 172.21.15.67:0/4030074275 shutdown_connections 2024-04-22T23:19:18.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.561+0000 7ff3ba913700 1 --2- 172.21.15.67:0/4030074275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b4102700 0x7ff3b410abf0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:18.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.561+0000 7ff3ba913700 1 --2- 172.21.15.67:0/4030074275 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff3b4101de0 0x7ff3b41021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:18.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.561+0000 7ff3ba913700 1 -- 172.21.15.67:0/4030074275 >> 172.21.15.67:0/4030074275 conn(0x7ff3b40fb650 msgr2=0x7ff3b40fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:18.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.561+0000 7ff3ba913700 1 -- 172.21.15.67:0/4030074275 shutdown_connections 2024-04-22T23:19:18.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.561+0000 7ff3ba913700 1 -- 172.21.15.67:0/4030074275 wait complete. 2024-04-22T23:19:18.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.562+0000 7ff3ba913700 1 Processor -- start 2024-04-22T23:19:18.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.562+0000 7ff3ba913700 1 -- start start 2024-04-22T23:19:18.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.562+0000 7ff3ba913700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff3b4101de0 0x7ff3b419c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:18.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.562+0000 7ff3ba913700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b4102700 0x7ff3b419cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:18.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.562+0000 7ff3ba913700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b419d370 con 0x7ff3b4102700 2024-04-22T23:19:18.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.562+0000 7ff3ba913700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b4196820 con 0x7ff3b4101de0 2024-04-22T23:19:18.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.562+0000 7ff3b37fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b4102700 0x7ff3b419cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:18.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.562+0000 7ff3b37fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b4102700 0x7ff3b419cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48404/0 (socket says 172.21.15.67:48404) 2024-04-22T23:19:18.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.562+0000 7ff3b37fe700 1 -- 172.21.15.67:0/1641695363 learned_addr learned my addr 172.21.15.67:0/1641695363 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:19:18.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.563+0000 7ff3b3fff700 1 --2- 172.21.15.67:0/1641695363 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff3b4101de0 0x7ff3b419c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:18.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.563+0000 7ff3b37fe700 1 -- 172.21.15.67:0/1641695363 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff3b4101de0 msgr2=0x7ff3b419c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:18.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.563+0000 7ff3b37fe700 1 --2- 172.21.15.67:0/1641695363 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff3b4101de0 0x7ff3b419c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:18.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.563+0000 7ff3b37fe700 1 -- 172.21.15.67:0/1641695363 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a80095f0 con 0x7ff3b4102700 2024-04-22T23:19:18.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.563+0000 7ff3b37fe700 1 --2- 172.21.15.67:0/1641695363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b4102700 0x7ff3b419cce0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff3a80092e0 tx=0x7ff3a800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:18.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.563+0000 7ff3b17fa700 1 -- 172.21.15.67:0/1641695363 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3a801d070 con 0x7ff3b4102700 2024-04-22T23:19:18.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.563+0000 7ff3b17fa700 1 -- 172.21.15.67:0/1641695363 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff3a800f460 con 0x7ff3b4102700 2024-04-22T23:19:18.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.563+0000 7ff3ba913700 1 -- 172.21.15.67:0/1641695363 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3b4196aa0 con 0x7ff3b4102700 2024-04-22T23:19:18.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.563+0000 7ff3b17fa700 1 -- 172.21.15.67:0/1641695363 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3a8017630 con 0x7ff3b4102700 2024-04-22T23:19:18.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.563+0000 7ff3ba913700 1 -- 172.21.15.67:0/1641695363 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3b4196f90 con 0x7ff3b4102700 2024-04-22T23:19:18.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.564+0000 7ff3ba913700 1 -- 172.21.15.67:0/1641695363 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3b41082f0 con 0x7ff3b4102700 2024-04-22T23:19:18.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.565+0000 7ff3b17fa700 1 -- 172.21.15.67:0/1641695363 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7ff3a800f5d0 con 0x7ff3b4102700 2024-04-22T23:19:18.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.565+0000 7ff3b17fa700 1 --2- 172.21.15.67:0/1641695363 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ff39c06c4e0 0x7ff39c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:18.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.565+0000 7ff3b17fa700 1 -- 172.21.15.67:0/1641695363 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ff3a808ca90 con 0x7ff3b4102700 2024-04-22T23:19:18.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.565+0000 7ff3b3fff700 1 --2- 172.21.15.67:0/1641695363 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ff39c06c4e0 0x7ff39c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:18.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.565+0000 7ff3b3fff700 1 --2- 172.21.15.67:0/1641695363 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ff39c06c4e0 0x7ff39c06e9a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff3a4006fd0 tx=0x7ff3a4008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:18.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.570+0000 7ff3b17fa700 1 -- 172.21.15.67:0/1641695363 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff3a805b040 con 0x7ff3b4102700 2024-04-22T23:19:18.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:18 smithi134 ceph-mon[34609]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:18.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:18 smithi067 ceph-mon[26453]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:18.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.825+0000 7ff3ba913700 1 -- 172.21.15.67:0/1641695363 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7ff3b4197dd0 con 0x7ff3b4102700 2024-04-22T23:19:18.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.828+0000 7ff3b17fa700 1 -- 172.21.15.67:0/1641695363 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+173038 (secure 0 0 0) 0x7ff3a805abd0 con 0x7ff3b4102700 2024-04-22T23:19:18.832 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:19:18.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.831+0000 7ff3ba913700 1 -- 172.21.15.67:0/1641695363 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ff39c06c4e0 msgr2=0x7ff39c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:18.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.831+0000 7ff3ba913700 1 --2- 172.21.15.67:0/1641695363 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ff39c06c4e0 0x7ff39c06e9a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff3a4006fd0 tx=0x7ff3a4008040 comp rx=0 tx=0).stop 2024-04-22T23:19:18.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.831+0000 7ff3ba913700 1 -- 172.21.15.67:0/1641695363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b4102700 msgr2=0x7ff3b419cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:18.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.831+0000 7ff3ba913700 1 --2- 172.21.15.67:0/1641695363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b4102700 0x7ff3b419cce0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff3a80092e0 tx=0x7ff3a800be20 comp rx=0 tx=0).stop 2024-04-22T23:19:18.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.831+0000 7ff3ba913700 1 -- 172.21.15.67:0/1641695363 shutdown_connections 2024-04-22T23:19:18.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.831+0000 7ff3ba913700 1 --2- 172.21.15.67:0/1641695363 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ff39c06c4e0 0x7ff39c06e9a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:18.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.831+0000 7ff3ba913700 1 --2- 172.21.15.67:0/1641695363 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff3b4101de0 0x7ff3b419c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:18.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.831+0000 7ff3ba913700 1 --2- 172.21.15.67:0/1641695363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b4102700 0x7ff3b419cce0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:18.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.831+0000 7ff3ba913700 1 -- 172.21.15.67:0/1641695363 >> 172.21.15.67:0/1641695363 conn(0x7ff3b40fb650 msgr2=0x7ff3b4105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:18.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.831+0000 7ff3ba913700 1 -- 172.21.15.67:0/1641695363 shutdown_connections 2024-04-22T23:19:18.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:18.831+0000 7ff3ba913700 1 -- 172.21.15.67:0/1641695363 wait complete. 2024-04-22T23:19:19.496 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":19,"active_gid":14215,"active_name":"smithi067.whdcsr","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":2589747850},{"type":"v1","addr":"172.21.15.67:6801","nonce":2589747850}]},"active_addr":"172.21.15.67:6801/2589747850","active_change":"2024-04-22T23:15:37.396310+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":14238,"name":"smithi134.ccxkcj","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.67:8443/","prometheus":"http://172.21.15.67:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":233140160}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":3750652317}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":2204477458}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":1933271294}]}]} 2024-04-22T23:19:19.500 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-22T23:19:19.500 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-22T23:19:19.500 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T23:19:19.713 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:19:19.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:19 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/1641695363' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-22T23:19:19.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:19 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/1641695363' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-22T23:19:20.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:20 smithi067 ceph-mon[26453]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:20.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:20 smithi134 ceph-mon[34609]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:21.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.403+0000 7f84755e0700 1 -- 172.21.15.67:0/3191164217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84700ffc20 msgr2=0x7f8470100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:21.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.403+0000 7f84755e0700 1 --2- 172.21.15.67:0/3191164217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84700ffc20 0x7f8470100080 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8458009910 tx=0x7f8458009c20 comp rx=0 tx=0).stop 2024-04-22T23:19:21.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.403+0000 7f84755e0700 1 -- 172.21.15.67:0/3191164217 shutdown_connections 2024-04-22T23:19:21.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.403+0000 7f84755e0700 1 --2- 172.21.15.67:0/3191164217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84700ffc20 0x7f8470100080 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:21.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.403+0000 7f84755e0700 1 --2- 172.21.15.67:0/3191164217 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8470105bd0 0x7f8470105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:21.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.403+0000 7f84755e0700 1 -- 172.21.15.67:0/3191164217 >> 172.21.15.67:0/3191164217 conn(0x7f8470075670 msgr2=0x7f8470075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:21.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.403+0000 7f84755e0700 1 -- 172.21.15.67:0/3191164217 shutdown_connections 2024-04-22T23:19:21.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.403+0000 7f84755e0700 1 -- 172.21.15.67:0/3191164217 wait complete. 2024-04-22T23:19:21.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.404+0000 7f84755e0700 1 Processor -- start 2024-04-22T23:19:21.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.404+0000 7f84755e0700 1 -- start start 2024-04-22T23:19:21.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.404+0000 7f84755e0700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84700ffc20 0x7f8470197760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:21.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.404+0000 7f84755e0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8470105bd0 0x7f8470197ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:21.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.404+0000 7f84755e0700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84701982c0 con 0x7f8470105bd0 2024-04-22T23:19:21.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.404+0000 7f84755e0700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8470198400 con 0x7f84700ffc20 2024-04-22T23:19:21.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.404+0000 7f846e7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8470105bd0 0x7f8470197ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:21.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.404+0000 7f846e7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8470105bd0 0x7f8470197ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57220/0 (socket says 172.21.15.67:57220) 2024-04-22T23:19:21.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.404+0000 7f846e7fc700 1 -- 172.21.15.67:0/3352043249 learned_addr learned my addr 172.21.15.67:0/3352043249 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:19:21.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.404+0000 7f846effd700 1 --2- 172.21.15.67:0/3352043249 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84700ffc20 0x7f8470197760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:21.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.404+0000 7f846e7fc700 1 -- 172.21.15.67:0/3352043249 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84700ffc20 msgr2=0x7f8470197760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:21.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.404+0000 7f846e7fc700 1 --2- 172.21.15.67:0/3352043249 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84700ffc20 0x7f8470197760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:21.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.404+0000 7f846e7fc700 1 -- 172.21.15.67:0/3352043249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84580095f0 con 0x7f8470105bd0 2024-04-22T23:19:21.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.405+0000 7f846e7fc700 1 --2- 172.21.15.67:0/3352043249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8470105bd0 0x7f8470197ca0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8458006010 tx=0x7f845800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:21.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.405+0000 7f8467fff700 1 -- 172.21.15.67:0/3352043249 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f845801d070 con 0x7f8470105bd0 2024-04-22T23:19:21.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.405+0000 7f8467fff700 1 -- 172.21.15.67:0/3352043249 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8458004fa0 con 0x7f8470105bd0 2024-04-22T23:19:21.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.405+0000 7f8467fff700 1 -- 172.21.15.67:0/3352043249 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f845800f690 con 0x7f8470105bd0 2024-04-22T23:19:21.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.405+0000 7f84755e0700 1 -- 172.21.15.67:0/3352043249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84701a07d0 con 0x7f8470105bd0 2024-04-22T23:19:21.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.405+0000 7f84755e0700 1 -- 172.21.15.67:0/3352043249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84701a0b80 con 0x7f8470105bd0 2024-04-22T23:19:21.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.406+0000 7f84755e0700 1 -- 172.21.15.67:0/3352043249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84701080f0 con 0x7f8470105bd0 2024-04-22T23:19:21.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.406+0000 7f8467fff700 1 -- 172.21.15.67:0/3352043249 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f8458022470 con 0x7f8470105bd0 2024-04-22T23:19:21.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.407+0000 7f8467fff700 1 --2- 172.21.15.67:0/3352043249 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f845c06c4e0 0x7f845c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:21.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.407+0000 7f846effd700 1 --2- 172.21.15.67:0/3352043249 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f845c06c4e0 0x7f845c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:21.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.407+0000 7f8467fff700 1 -- 172.21.15.67:0/3352043249 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f845808cc70 con 0x7f8470105bd0 2024-04-22T23:19:21.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.407+0000 7f846effd700 1 --2- 172.21.15.67:0/3352043249 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f845c06c4e0 0x7f845c06e9a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f846800a400 tx=0x7f846800d040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:21.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.412+0000 7f8467fff700 1 -- 172.21.15.67:0/3352043249 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f8458057790 con 0x7f8470105bd0 2024-04-22T23:19:21.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.610+0000 7f84755e0700 1 -- 172.21.15.67:0/3352043249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8470068be0 con 0x7f8470105bd0 2024-04-22T23:19:21.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.611+0000 7f8467fff700 1 -- 172.21.15.67:0/3352043249 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11325 (secure 0 0 0) 0x7f845805adb0 con 0x7f8470105bd0 2024-04-22T23:19:21.613 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:19:21.613 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":40,"fsid":"d46c346e-00fd-11ef-bc93-c7b262605968","created":"2024-04-22T23:13:18.687917+0000","modified":"2024-04-22T23:19:01.899867+0000","last_up_change":"2024-04-22T23:19:00.895098+0000","last_in_change":"2024-04-22T23:18:41.696935+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T23:17:47.912850+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"25c9acd1-5743-4dc0-aee0-c7bb8884c0a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":268472413},{"type":"v1","addr":"172.21.15.67:6803","nonce":268472413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":268472413},{"type":"v1","addr":"172.21.15.67:6805","nonce":268472413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":268472413},{"type":"v1","addr":"172.21.15.67:6809","nonce":268472413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":268472413},{"type":"v1","addr":"172.21.15.67:6807","nonce":268472413}]},"public_addr":"172.21.15.67:6803/268472413","cluster_addr":"172.21.15.67:6805/268472413","heartbeat_back_addr":"172.21.15.67:6809/268472413","heartbeat_front_addr":"172.21.15.67:6807/268472413","state":["exists","up"]},{"osd":1,"uuid":"8be83014-01af-44d3-ab2f-b27840506e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3382820294},{"type":"v1","addr":"172.21.15.67:6811","nonce":3382820294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3382820294},{"type":"v1","addr":"172.21.15.67:6813","nonce":3382820294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3382820294},{"type":"v1","addr":"172.21.15.67:6817","nonce":3382820294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3382820294},{"type":"v1","addr":"172.21.15.67:6815","nonce":3382820294}]},"public_addr":"172.21.15.67:6811/3382820294","cluster_addr":"172.21.15.67:6813/3382820294","heartbeat_back_addr":"172.21.15.67:6817/3382820294","heartbeat_front_addr":"172.21.15.67:6815/3382820294","state":["exists","up"]},{"osd":2,"uuid":"ab1cbcda-14dd-4c24-876b-25ad5d4d8563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3835340581},{"type":"v1","addr":"172.21.15.67:6819","nonce":3835340581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3835340581},{"type":"v1","addr":"172.21.15.67:6821","nonce":3835340581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3835340581},{"type":"v1","addr":"172.21.15.67:6825","nonce":3835340581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3835340581},{"type":"v1","addr":"172.21.15.67:6823","nonce":3835340581}]},"public_addr":"172.21.15.67:6819/3835340581","cluster_addr":"172.21.15.67:6821/3835340581","heartbeat_back_addr":"172.21.15.67:6825/3835340581","heartbeat_front_addr":"172.21.15.67:6823/3835340581","state":["exists","up"]},{"osd":3,"uuid":"1a30b6c3-95bf-4b19-883b-1271ff359e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":2235786001},{"type":"v1","addr":"172.21.15.134:6801","nonce":2235786001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":2235786001},{"type":"v1","addr":"172.21.15.134:6803","nonce":2235786001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":2235786001},{"type":"v1","addr":"172.21.15.134:6807","nonce":2235786001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":2235786001},{"type":"v1","addr":"172.21.15.134:6805","nonce":2235786001}]},"public_addr":"172.21.15.134:6801/2235786001","cluster_addr":"172.21.15.134:6803/2235786001","heartbeat_back_addr":"172.21.15.134:6807/2235786001","heartbeat_front_addr":"172.21.15.134:6805/2235786001","state":["exists","up"]},{"osd":4,"uuid":"53368fc1-28fd-4622-bc9c-1d8901b06bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":668644810},{"type":"v1","addr":"172.21.15.134:6809","nonce":668644810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":668644810},{"type":"v1","addr":"172.21.15.134:6811","nonce":668644810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":668644810},{"type":"v1","addr":"172.21.15.134:6815","nonce":668644810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":668644810},{"type":"v1","addr":"172.21.15.134:6813","nonce":668644810}]},"public_addr":"172.21.15.134:6809/668644810","cluster_addr":"172.21.15.134:6811/668644810","heartbeat_back_addr":"172.21.15.134:6815/668644810","heartbeat_front_addr":"172.21.15.134:6813/668644810","state":["exists","up"]},{"osd":5,"uuid":"31a885f7-8f91-4b6a-922a-065753f70a95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3846690247},{"type":"v1","addr":"172.21.15.134:6817","nonce":3846690247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":3846690247},{"type":"v1","addr":"172.21.15.134:6819","nonce":3846690247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":3846690247},{"type":"v1","addr":"172.21.15.134:6823","nonce":3846690247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":3846690247},{"type":"v1","addr":"172.21.15.134:6821","nonce":3846690247}]},"public_addr":"172.21.15.134:6817/3846690247","cluster_addr":"172.21.15.134:6819/3846690247","heartbeat_back_addr":"172.21.15.134:6823/3846690247","heartbeat_front_addr":"172.21.15.134:6821/3846690247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:16:51.179651+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:17:18.694683+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:17:45.368963+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:18:08.585046+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:18:33.412804+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:18:58.909758+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6800/2590792187":"2024-04-23T23:15:37.396154+0000","172.21.15.67:0/1125624760":"2024-04-23T23:15:37.396154+0000","172.21.15.67:0/1189311107":"2024-04-23T23:15:37.396154+0000","172.21.15.67:6801/2590792187":"2024-04-23T23:15:37.396154+0000","172.21.15.67:0/2137374599":"2024-04-23T23:15:37.396154+0000","172.21.15.67:0/4165654004":"2024-04-23T23:14:29.214977+0000","172.21.15.67:6801/3568049827":"2024-04-23T23:14:29.214977+0000","172.21.15.67:6801/2451437776":"2024-04-23T23:13:56.301645+0000","172.21.15.67:0/2052573085":"2024-04-23T23:14:29.214977+0000","172.21.15.67:0/853292259":"2024-04-23T23:13:56.301645+0000","172.21.15.67:0/895378312":"2024-04-23T23:13:56.301645+0000","172.21.15.67:0/3387250764":"2024-04-23T23:13:56.301645+0000","172.21.15.67:0/2665297739":"2024-04-23T23:14:29.214977+0000","172.21.15.67:6800/3568049827":"2024-04-23T23:14:29.214977+0000","172.21.15.67:6800/2451437776":"2024-04-23T23:13:56.301645+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T23:19:21.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.613+0000 7f84755e0700 1 -- 172.21.15.67:0/3352043249 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f845c06c4e0 msgr2=0x7f845c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:21.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.613+0000 7f84755e0700 1 --2- 172.21.15.67:0/3352043249 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f845c06c4e0 0x7f845c06e9a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f846800a400 tx=0x7f846800d040 comp rx=0 tx=0).stop 2024-04-22T23:19:21.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.613+0000 7f84755e0700 1 -- 172.21.15.67:0/3352043249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8470105bd0 msgr2=0x7f8470197ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:21.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.613+0000 7f84755e0700 1 --2- 172.21.15.67:0/3352043249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8470105bd0 0x7f8470197ca0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8458006010 tx=0x7f845800bd70 comp rx=0 tx=0).stop 2024-04-22T23:19:21.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.613+0000 7f84755e0700 1 -- 172.21.15.67:0/3352043249 shutdown_connections 2024-04-22T23:19:21.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.613+0000 7f84755e0700 1 --2- 172.21.15.67:0/3352043249 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f845c06c4e0 0x7f845c06e9a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:21.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.613+0000 7f84755e0700 1 --2- 172.21.15.67:0/3352043249 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84700ffc20 0x7f8470197760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:21.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.613+0000 7f84755e0700 1 --2- 172.21.15.67:0/3352043249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8470105bd0 0x7f8470197ca0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:21.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.613+0000 7f84755e0700 1 -- 172.21.15.67:0/3352043249 >> 172.21.15.67:0/3352043249 conn(0x7f8470075670 msgr2=0x7f84700fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:21.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.613+0000 7f84755e0700 1 -- 172.21.15.67:0/3352043249 shutdown_connections 2024-04-22T23:19:21.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:21.613+0000 7f84755e0700 1 -- 172.21.15.67:0/3352043249 wait complete. 2024-04-22T23:19:22.419 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-22T23:19:22.420 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T23:19:22.635 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:19:22.757 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:22 smithi067 ceph-mon[26453]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:22.757 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:22 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/3352043249' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T23:19:22.757 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:22 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:19:22.875 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:22 smithi134 ceph-mon[34609]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:22.876 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:22 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/3352043249' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T23:19:22.876 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:22 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:19:24.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.313+0000 7f3f4fa96700 1 -- 172.21.15.67:0/2421544093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f48101850 msgr2=0x7f3f481058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:24.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.313+0000 7f3f4fa96700 1 --2- 172.21.15.67:0/2421544093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f48101850 0x7f3f481058a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c009910 tx=0x7f3f3c009c20 comp rx=0 tx=0).stop 2024-04-22T23:19:24.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.313+0000 7f3f4fa96700 1 -- 172.21.15.67:0/2421544093 shutdown_connections 2024-04-22T23:19:24.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.313+0000 7f3f4fa96700 1 --2- 172.21.15.67:0/2421544093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f48101850 0x7f3f481058a0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:24.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.313+0000 7f3f4fa96700 1 --2- 172.21.15.67:0/2421544093 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3f48100ea0 0x7f3f48101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:24.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.313+0000 7f3f4fa96700 1 -- 172.21.15.67:0/2421544093 >> 172.21.15.67:0/2421544093 conn(0x7f3f480fc750 msgr2=0x7f3f480feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:24.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.313+0000 7f3f4fa96700 1 -- 172.21.15.67:0/2421544093 shutdown_connections 2024-04-22T23:19:24.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.313+0000 7f3f4fa96700 1 -- 172.21.15.67:0/2421544093 wait complete. 2024-04-22T23:19:24.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.314+0000 7f3f4fa96700 1 Processor -- start 2024-04-22T23:19:24.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.314+0000 7f3f4fa96700 1 -- start start 2024-04-22T23:19:24.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.314+0000 7f3f4fa96700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f48100ea0 0x7f3f48073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:24.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.314+0000 7f3f4fa96700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3f48101850 0x7f3f480735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:24.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.314+0000 7f3f4fa96700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f48073b10 con 0x7f3f48100ea0 2024-04-22T23:19:24.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.314+0000 7f3f4fa96700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f48073c50 con 0x7f3f48101850 2024-04-22T23:19:24.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.314+0000 7f3f4d832700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f48100ea0 0x7f3f48073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:24.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.314+0000 7f3f4d832700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f48100ea0 0x7f3f48073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57236/0 (socket says 172.21.15.67:57236) 2024-04-22T23:19:24.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.314+0000 7f3f4d832700 1 -- 172.21.15.67:0/620807379 learned_addr learned my addr 172.21.15.67:0/620807379 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:19:24.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.314+0000 7f3f4d031700 1 --2- 172.21.15.67:0/620807379 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3f48101850 0x7f3f480735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:24.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.314+0000 7f3f4d832700 1 -- 172.21.15.67:0/620807379 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3f48101850 msgr2=0x7f3f480735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:24.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.315+0000 7f3f4d832700 1 --2- 172.21.15.67:0/620807379 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3f48101850 0x7f3f480735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:24.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.315+0000 7f3f4d832700 1 -- 172.21.15.67:0/620807379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f3c0095f0 con 0x7f3f48100ea0 2024-04-22T23:19:24.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.315+0000 7f3f4d832700 1 --2- 172.21.15.67:0/620807379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f48100ea0 0x7f3f48073090 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3f380060b0 tx=0x7f3f3800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:24.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.315+0000 7f3f46ffd700 1 -- 172.21.15.67:0/620807379 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f3800dda0 con 0x7f3f48100ea0 2024-04-22T23:19:24.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.315+0000 7f3f46ffd700 1 -- 172.21.15.67:0/620807379 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3f3800f040 con 0x7f3f48100ea0 2024-04-22T23:19:24.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.315+0000 7f3f4fa96700 1 -- 172.21.15.67:0/620807379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f481a2b00 con 0x7f3f48100ea0 2024-04-22T23:19:24.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.315+0000 7f3f46ffd700 1 -- 172.21.15.67:0/620807379 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f380146f0 con 0x7f3f48100ea0 2024-04-22T23:19:24.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.315+0000 7f3f4fa96700 1 -- 172.21.15.67:0/620807379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f481a2f10 con 0x7f3f48100ea0 2024-04-22T23:19:24.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.316+0000 7f3f4fa96700 1 -- 172.21.15.67:0/620807379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f481092a0 con 0x7f3f48100ea0 2024-04-22T23:19:24.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.317+0000 7f3f46ffd700 1 -- 172.21.15.67:0/620807379 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f3f38015400 con 0x7f3f48100ea0 2024-04-22T23:19:24.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.317+0000 7f3f46ffd700 1 --2- 172.21.15.67:0/620807379 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f3f3006c4e0 0x7f3f3006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:24.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.317+0000 7f3f46ffd700 1 -- 172.21.15.67:0/620807379 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f3f3808aee0 con 0x7f3f48100ea0 2024-04-22T23:19:24.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.317+0000 7f3f4d031700 1 --2- 172.21.15.67:0/620807379 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f3f3006c4e0 0x7f3f3006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:24.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.317+0000 7f3f4d031700 1 --2- 172.21.15.67:0/620807379 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f3f3006c4e0 0x7f3f3006e9a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3f48074880 tx=0x7f3f3c01a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:24.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.321+0000 7f3f46ffd700 1 -- 172.21.15.67:0/620807379 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f3f38055ab0 con 0x7f3f48100ea0 2024-04-22T23:19:24.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.521+0000 7f3f4fa96700 1 -- 172.21.15.67:0/620807379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f4804f2e0 con 0x7f3f48100ea0 2024-04-22T23:19:24.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.522+0000 7f3f46ffd700 1 -- 172.21.15.67:0/620807379 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11325 (secure 0 0 0) 0x7f3f380590d0 con 0x7f3f48100ea0 2024-04-22T23:19:24.523 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:19:24.523 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":40,"fsid":"d46c346e-00fd-11ef-bc93-c7b262605968","created":"2024-04-22T23:13:18.687917+0000","modified":"2024-04-22T23:19:01.899867+0000","last_up_change":"2024-04-22T23:19:00.895098+0000","last_in_change":"2024-04-22T23:18:41.696935+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T23:17:47.912850+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"25c9acd1-5743-4dc0-aee0-c7bb8884c0a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":268472413},{"type":"v1","addr":"172.21.15.67:6803","nonce":268472413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":268472413},{"type":"v1","addr":"172.21.15.67:6805","nonce":268472413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":268472413},{"type":"v1","addr":"172.21.15.67:6809","nonce":268472413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":268472413},{"type":"v1","addr":"172.21.15.67:6807","nonce":268472413}]},"public_addr":"172.21.15.67:6803/268472413","cluster_addr":"172.21.15.67:6805/268472413","heartbeat_back_addr":"172.21.15.67:6809/268472413","heartbeat_front_addr":"172.21.15.67:6807/268472413","state":["exists","up"]},{"osd":1,"uuid":"8be83014-01af-44d3-ab2f-b27840506e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3382820294},{"type":"v1","addr":"172.21.15.67:6811","nonce":3382820294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3382820294},{"type":"v1","addr":"172.21.15.67:6813","nonce":3382820294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3382820294},{"type":"v1","addr":"172.21.15.67:6817","nonce":3382820294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3382820294},{"type":"v1","addr":"172.21.15.67:6815","nonce":3382820294}]},"public_addr":"172.21.15.67:6811/3382820294","cluster_addr":"172.21.15.67:6813/3382820294","heartbeat_back_addr":"172.21.15.67:6817/3382820294","heartbeat_front_addr":"172.21.15.67:6815/3382820294","state":["exists","up"]},{"osd":2,"uuid":"ab1cbcda-14dd-4c24-876b-25ad5d4d8563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3835340581},{"type":"v1","addr":"172.21.15.67:6819","nonce":3835340581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3835340581},{"type":"v1","addr":"172.21.15.67:6821","nonce":3835340581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3835340581},{"type":"v1","addr":"172.21.15.67:6825","nonce":3835340581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3835340581},{"type":"v1","addr":"172.21.15.67:6823","nonce":3835340581}]},"public_addr":"172.21.15.67:6819/3835340581","cluster_addr":"172.21.15.67:6821/3835340581","heartbeat_back_addr":"172.21.15.67:6825/3835340581","heartbeat_front_addr":"172.21.15.67:6823/3835340581","state":["exists","up"]},{"osd":3,"uuid":"1a30b6c3-95bf-4b19-883b-1271ff359e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":2235786001},{"type":"v1","addr":"172.21.15.134:6801","nonce":2235786001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":2235786001},{"type":"v1","addr":"172.21.15.134:6803","nonce":2235786001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":2235786001},{"type":"v1","addr":"172.21.15.134:6807","nonce":2235786001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":2235786001},{"type":"v1","addr":"172.21.15.134:6805","nonce":2235786001}]},"public_addr":"172.21.15.134:6801/2235786001","cluster_addr":"172.21.15.134:6803/2235786001","heartbeat_back_addr":"172.21.15.134:6807/2235786001","heartbeat_front_addr":"172.21.15.134:6805/2235786001","state":["exists","up"]},{"osd":4,"uuid":"53368fc1-28fd-4622-bc9c-1d8901b06bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":668644810},{"type":"v1","addr":"172.21.15.134:6809","nonce":668644810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":668644810},{"type":"v1","addr":"172.21.15.134:6811","nonce":668644810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":668644810},{"type":"v1","addr":"172.21.15.134:6815","nonce":668644810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":668644810},{"type":"v1","addr":"172.21.15.134:6813","nonce":668644810}]},"public_addr":"172.21.15.134:6809/668644810","cluster_addr":"172.21.15.134:6811/668644810","heartbeat_back_addr":"172.21.15.134:6815/668644810","heartbeat_front_addr":"172.21.15.134:6813/668644810","state":["exists","up"]},{"osd":5,"uuid":"31a885f7-8f91-4b6a-922a-065753f70a95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3846690247},{"type":"v1","addr":"172.21.15.134:6817","nonce":3846690247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":3846690247},{"type":"v1","addr":"172.21.15.134:6819","nonce":3846690247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":3846690247},{"type":"v1","addr":"172.21.15.134:6823","nonce":3846690247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":3846690247},{"type":"v1","addr":"172.21.15.134:6821","nonce":3846690247}]},"public_addr":"172.21.15.134:6817/3846690247","cluster_addr":"172.21.15.134:6819/3846690247","heartbeat_back_addr":"172.21.15.134:6823/3846690247","heartbeat_front_addr":"172.21.15.134:6821/3846690247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:16:51.179651+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:17:18.694683+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:17:45.368963+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:18:08.585046+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:18:33.412804+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:18:58.909758+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6800/2590792187":"2024-04-23T23:15:37.396154+0000","172.21.15.67:0/1125624760":"2024-04-23T23:15:37.396154+0000","172.21.15.67:0/1189311107":"2024-04-23T23:15:37.396154+0000","172.21.15.67:6801/2590792187":"2024-04-23T23:15:37.396154+0000","172.21.15.67:0/2137374599":"2024-04-23T23:15:37.396154+0000","172.21.15.67:0/4165654004":"2024-04-23T23:14:29.214977+0000","172.21.15.67:6801/3568049827":"2024-04-23T23:14:29.214977+0000","172.21.15.67:6801/2451437776":"2024-04-23T23:13:56.301645+0000","172.21.15.67:0/2052573085":"2024-04-23T23:14:29.214977+0000","172.21.15.67:0/853292259":"2024-04-23T23:13:56.301645+0000","172.21.15.67:0/895378312":"2024-04-23T23:13:56.301645+0000","172.21.15.67:0/3387250764":"2024-04-23T23:13:56.301645+0000","172.21.15.67:0/2665297739":"2024-04-23T23:14:29.214977+0000","172.21.15.67:6800/3568049827":"2024-04-23T23:14:29.214977+0000","172.21.15.67:6800/2451437776":"2024-04-23T23:13:56.301645+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T23:19:24.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.524+0000 7f3f4fa96700 1 -- 172.21.15.67:0/620807379 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f3f3006c4e0 msgr2=0x7f3f3006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:24.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.524+0000 7f3f4fa96700 1 --2- 172.21.15.67:0/620807379 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f3f3006c4e0 0x7f3f3006e9a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3f48074880 tx=0x7f3f3c01a040 comp rx=0 tx=0).stop 2024-04-22T23:19:24.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.524+0000 7f3f4fa96700 1 -- 172.21.15.67:0/620807379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f48100ea0 msgr2=0x7f3f48073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:24.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.524+0000 7f3f4fa96700 1 --2- 172.21.15.67:0/620807379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f48100ea0 0x7f3f48073090 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3f380060b0 tx=0x7f3f3800d3b0 comp rx=0 tx=0).stop 2024-04-22T23:19:24.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.524+0000 7f3f4fa96700 1 -- 172.21.15.67:0/620807379 shutdown_connections 2024-04-22T23:19:24.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.524+0000 7f3f4fa96700 1 --2- 172.21.15.67:0/620807379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f48100ea0 0x7f3f48073090 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:24.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.524+0000 7f3f4fa96700 1 --2- 172.21.15.67:0/620807379 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f3f3006c4e0 0x7f3f3006e9a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:24.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.524+0000 7f3f4fa96700 1 --2- 172.21.15.67:0/620807379 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3f48101850 0x7f3f480735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:24.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.524+0000 7f3f4fa96700 1 -- 172.21.15.67:0/620807379 >> 172.21.15.67:0/620807379 conn(0x7f3f480fc750 msgr2=0x7f3f480fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:24.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.524+0000 7f3f4fa96700 1 -- 172.21.15.67:0/620807379 shutdown_connections 2024-04-22T23:19:24.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:24.524+0000 7f3f4fa96700 1 -- 172.21.15.67:0/620807379 wait complete. 2024-04-22T23:19:24.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:24 smithi067 ceph-mon[26453]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:24.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:24 smithi134 ceph-mon[34609]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:25.156 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-22T23:19:25.157 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-22T23:19:25.157 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-22T23:19:25.158 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-22T23:19:25.158 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-22T23:19:25.158 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-22T23:19:25.424 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:19:25.435 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:19:25.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:25 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/620807379' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T23:19:25.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:25 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/620807379' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T23:19:26.757 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:19:26.758 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:19:26.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:26 smithi067 ceph-mon[26453]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:26.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:26 smithi134 ceph-mon[34609]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:27.160 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:19:27.161 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:19:28.696 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:28 smithi067 ceph-mon[26453]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:28.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:28 smithi134 ceph-mon[34609]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:29.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.867+0000 7fe706a11700 1 -- 172.21.15.67:0/3250905430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe700100e90 msgr2=0x7fe700101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:29.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.867+0000 7fe706a11700 1 --2- 172.21.15.67:0/3250905430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe700100e90 0x7fe700101270 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe6f0009910 tx=0x7fe6f0009c20 comp rx=0 tx=0).stop 2024-04-22T23:19:29.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.867+0000 7fe706a11700 1 -- 172.21.15.67:0/3250905430 shutdown_connections 2024-04-22T23:19:29.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.867+0000 7fe706a11700 1 --2- 172.21.15.67:0/3250905430 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe700101840 0x7fe700105890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:29.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.867+0000 7fe706a11700 1 --2- 172.21.15.67:0/3250905430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe700100e90 0x7fe700101270 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:29.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.867+0000 7fe706a11700 1 -- 172.21.15.67:0/3250905430 >> 172.21.15.67:0/3250905430 conn(0x7fe7000fc740 msgr2=0x7fe7000feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:29.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.867+0000 7fe706a11700 1 -- 172.21.15.67:0/3250905430 shutdown_connections 2024-04-22T23:19:29.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.867+0000 7fe706a11700 1 -- 172.21.15.67:0/3250905430 wait complete. 2024-04-22T23:19:29.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.867+0000 7fe706a11700 1 Processor -- start 2024-04-22T23:19:29.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.868+0000 7fe706a11700 1 -- start start 2024-04-22T23:19:29.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.868+0000 7fe706a11700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe700100e90 0x7fe70019c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:29.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.868+0000 7fe706a11700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe700101840 0x7fe70019cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:29.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.868+0000 7fe706a11700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe70019d230 con 0x7fe700101840 2024-04-22T23:19:29.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.868+0000 7fe706a11700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7001966e0 con 0x7fe700100e90 2024-04-22T23:19:29.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.868+0000 7fe6ff7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe700101840 0x7fe70019cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:29.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.868+0000 7fe6ff7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe700101840 0x7fe70019cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37664/0 (socket says 172.21.15.67:37664) 2024-04-22T23:19:29.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.868+0000 7fe6ff7fe700 1 -- 172.21.15.67:0/1183746959 learned_addr learned my addr 172.21.15.67:0/1183746959 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:19:29.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.868+0000 7fe6fffff700 1 --2- 172.21.15.67:0/1183746959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe700100e90 0x7fe70019c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:29.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.868+0000 7fe6ff7fe700 1 -- 172.21.15.67:0/1183746959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe700100e90 msgr2=0x7fe70019c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:29.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.868+0000 7fe6ff7fe700 1 --2- 172.21.15.67:0/1183746959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe700100e90 0x7fe70019c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:29.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.868+0000 7fe6ff7fe700 1 -- 172.21.15.67:0/1183746959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f00095f0 con 0x7fe700101840 2024-04-22T23:19:29.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.869+0000 7fe6ff7fe700 1 --2- 172.21.15.67:0/1183746959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe700101840 0x7fe70019cba0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe6f400c8f0 tx=0x7fe6f400cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:29.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.869+0000 7fe6fd7fa700 1 -- 172.21.15.67:0/1183746959 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe6f40078a0 con 0x7fe700101840 2024-04-22T23:19:29.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.869+0000 7fe6fd7fa700 1 -- 172.21.15.67:0/1183746959 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe6f400f450 con 0x7fe700101840 2024-04-22T23:19:29.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.869+0000 7fe706a11700 1 -- 172.21.15.67:0/1183746959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7001969c0 con 0x7fe700101840 2024-04-22T23:19:29.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.869+0000 7fe6fd7fa700 1 -- 172.21.15.67:0/1183746959 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe6f4018610 con 0x7fe700101840 2024-04-22T23:19:29.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.869+0000 7fe706a11700 1 -- 172.21.15.67:0/1183746959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe700196f10 con 0x7fe700101840 2024-04-22T23:19:29.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.870+0000 7fe706a11700 1 -- 172.21.15.67:0/1183746959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe70010e0a0 con 0x7fe700101840 2024-04-22T23:19:29.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.870+0000 7fe6fd7fa700 1 -- 172.21.15.67:0/1183746959 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fe6f4007a00 con 0x7fe700101840 2024-04-22T23:19:29.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.871+0000 7fe6fd7fa700 1 --2- 172.21.15.67:0/1183746959 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fe6e806c4e0 0x7fe6e806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:29.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.871+0000 7fe6fd7fa700 1 -- 172.21.15.67:0/1183746959 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fe6f408b440 con 0x7fe700101840 2024-04-22T23:19:29.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.871+0000 7fe6fffff700 1 --2- 172.21.15.67:0/1183746959 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fe6e806c4e0 0x7fe6e806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:29.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.871+0000 7fe6fd7fa700 1 --2- 172.21.15.67:0/1183746959 >> [v2:172.21.15.134:6800/2235786001,v1:172.21.15.134:6801/2235786001] conn(0x7fe6e80721c0 0x7fe6e80745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:29.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.871+0000 7fe6fd7fa700 1 -- 172.21.15.67:0/1183746959 --> [v2:172.21.15.134:6800/2235786001,v1:172.21.15.134:6801/2235786001] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe6e8074c90 con 0x7fe6e80721c0 2024-04-22T23:19:29.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.871+0000 7fe6fd7fa700 1 -- 172.21.15.67:0/1183746959 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6f408db10 con 0x7fe700101840 2024-04-22T23:19:29.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.871+0000 7fe6fffff700 1 --2- 172.21.15.67:0/1183746959 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fe6e806c4e0 0x7fe6e806e9a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe6f0006010 tx=0x7fe6f001a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:29.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.871+0000 7fe704fae700 1 --2- 172.21.15.67:0/1183746959 >> [v2:172.21.15.134:6800/2235786001,v1:172.21.15.134:6801/2235786001] conn(0x7fe6e80721c0 0x7fe6e80745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:29.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.872+0000 7fe704fae700 1 --2- 172.21.15.67:0/1183746959 >> [v2:172.21.15.134:6800/2235786001,v1:172.21.15.134:6801/2235786001] conn(0x7fe6e80721c0 0x7fe6e80745e0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:29.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.873+0000 7fe6fd7fa700 1 -- 172.21.15.67:0/1183746959 <== osd.3 v2:172.21.15.134:6800/2235786001 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fe6e8074c90 con 0x7fe6e80721c0 2024-04-22T23:19:29.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.888+0000 7fe706a11700 1 -- 172.21.15.67:0/1183746959 --> [v2:172.21.15.134:6800/2235786001,v1:172.21.15.134:6801/2235786001] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe70004f2e0 con 0x7fe6e80721c0 2024-04-22T23:19:29.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.889+0000 7fe6fd7fa700 1 -- 172.21.15.67:0/1183746959 <== osd.3 v2:172.21.15.134:6800/2235786001 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fe70004f2e0 con 0x7fe6e80721c0 2024-04-22T23:19:29.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.889+0000 7fe706a11700 1 -- 172.21.15.67:0/1183746959 >> [v2:172.21.15.134:6800/2235786001,v1:172.21.15.134:6801/2235786001] conn(0x7fe6e80721c0 msgr2=0x7fe6e80745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:29.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.889+0000 7fe706a11700 1 --2- 172.21.15.67:0/1183746959 >> [v2:172.21.15.134:6800/2235786001,v1:172.21.15.134:6801/2235786001] conn(0x7fe6e80721c0 0x7fe6e80745e0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:29.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.889+0000 7fe706a11700 1 -- 172.21.15.67:0/1183746959 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fe6e806c4e0 msgr2=0x7fe6e806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:29.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.889+0000 7fe706a11700 1 --2- 172.21.15.67:0/1183746959 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fe6e806c4e0 0x7fe6e806e9a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe6f0006010 tx=0x7fe6f001a040 comp rx=0 tx=0).stop 2024-04-22T23:19:29.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.889+0000 7fe706a11700 1 -- 172.21.15.67:0/1183746959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe700101840 msgr2=0x7fe70019cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:29.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.889+0000 7fe706a11700 1 --2- 172.21.15.67:0/1183746959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe700101840 0x7fe70019cba0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe6f400c8f0 tx=0x7fe6f400cc00 comp rx=0 tx=0).stop 2024-04-22T23:19:29.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.889+0000 7fe706a11700 1 -- 172.21.15.67:0/1183746959 shutdown_connections 2024-04-22T23:19:29.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.889+0000 7fe706a11700 1 --2- 172.21.15.67:0/1183746959 >> [v2:172.21.15.134:6800/2235786001,v1:172.21.15.134:6801/2235786001] conn(0x7fe6e80721c0 0x7fe6e80745e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:29.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.889+0000 7fe706a11700 1 --2- 172.21.15.67:0/1183746959 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fe6e806c4e0 0x7fe6e806e9a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:29.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.889+0000 7fe706a11700 1 --2- 172.21.15.67:0/1183746959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe700100e90 0x7fe70019c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:29.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.889+0000 7fe706a11700 1 --2- 172.21.15.67:0/1183746959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe700101840 0x7fe70019cba0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:29.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.889+0000 7fe706a11700 1 -- 172.21.15.67:0/1183746959 >> 172.21.15.67:0/1183746959 conn(0x7fe7000fc740 msgr2=0x7fe7000fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:29.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.889+0000 7fe706a11700 1 -- 172.21.15.67:0/1183746959 shutdown_connections 2024-04-22T23:19:29.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:29.889+0000 7fe706a11700 1 -- 172.21.15.67:0/1183746959 wait complete. 2024-04-22T23:19:30.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:30 smithi067 ceph-mon[26453]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:30.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:30 smithi134 ceph-mon[34609]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:32.671 INFO:teuthology.orchestra.run.smithi067.stdout:111669149714 2024-04-22T23:19:32.671 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-22T23:19:32.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:32 smithi067 ceph-mon[26453]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:32.953 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:32 smithi134 ceph-mon[34609]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:33.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.288+0000 7fd25ef7d700 1 -- 172.21.15.67:0/669084067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd258100ea0 msgr2=0x7fd258101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:33.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.288+0000 7fd25ef7d700 1 --2- 172.21.15.67:0/669084067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd258100ea0 0x7fd258101280 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd24c009910 tx=0x7fd24c009c20 comp rx=0 tx=0).stop 2024-04-22T23:19:33.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.288+0000 7fd25ef7d700 1 -- 172.21.15.67:0/669084067 shutdown_connections 2024-04-22T23:19:33.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.288+0000 7fd25ef7d700 1 --2- 172.21.15.67:0/669084067 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd258101850 0x7fd2581058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:33.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.288+0000 7fd25ef7d700 1 --2- 172.21.15.67:0/669084067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd258100ea0 0x7fd258101280 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:33.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.288+0000 7fd25ef7d700 1 -- 172.21.15.67:0/669084067 >> 172.21.15.67:0/669084067 conn(0x7fd2580fc750 msgr2=0x7fd2580feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:33.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.288+0000 7fd25ef7d700 1 -- 172.21.15.67:0/669084067 shutdown_connections 2024-04-22T23:19:33.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.288+0000 7fd25ef7d700 1 -- 172.21.15.67:0/669084067 wait complete. 2024-04-22T23:19:33.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.289+0000 7fd25ef7d700 1 Processor -- start 2024-04-22T23:19:33.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.289+0000 7fd25ef7d700 1 -- start start 2024-04-22T23:19:33.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.289+0000 7fd25ef7d700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd258100ea0 0x7fd25819c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:33.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.289+0000 7fd25ef7d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd258101850 0x7fd25819cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:33.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.289+0000 7fd25ef7d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd25819d1c0 con 0x7fd258101850 2024-04-22T23:19:33.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.289+0000 7fd25ef7d700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd258196770 con 0x7fd258100ea0 2024-04-22T23:19:33.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.289+0000 7fd257fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd258101850 0x7fd25819cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:33.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.289+0000 7fd257fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd258101850 0x7fd25819cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37672/0 (socket says 172.21.15.67:37672) 2024-04-22T23:19:33.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.289+0000 7fd257fff700 1 -- 172.21.15.67:0/1094975199 learned_addr learned my addr 172.21.15.67:0/1094975199 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:19:33.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.289+0000 7fd25cd19700 1 --2- 172.21.15.67:0/1094975199 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd258100ea0 0x7fd25819c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:33.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.289+0000 7fd257fff700 1 -- 172.21.15.67:0/1094975199 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd258100ea0 msgr2=0x7fd25819c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:33.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.289+0000 7fd257fff700 1 --2- 172.21.15.67:0/1094975199 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd258100ea0 0x7fd25819c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:33.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.289+0000 7fd257fff700 1 -- 172.21.15.67:0/1094975199 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd24c0095f0 con 0x7fd258101850 2024-04-22T23:19:33.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.290+0000 7fd257fff700 1 --2- 172.21.15.67:0/1094975199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd258101850 0x7fd25819cbc0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd24400c960 tx=0x7fd24400cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:33.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.290+0000 7fd255ffb700 1 -- 172.21.15.67:0/1094975199 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd244007a70 con 0x7fd258101850 2024-04-22T23:19:33.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.290+0000 7fd25ef7d700 1 -- 172.21.15.67:0/1094975199 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd258196a50 con 0x7fd258101850 2024-04-22T23:19:33.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.290+0000 7fd255ffb700 1 -- 172.21.15.67:0/1094975199 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd244007bd0 con 0x7fd258101850 2024-04-22T23:19:33.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.290+0000 7fd255ffb700 1 -- 172.21.15.67:0/1094975199 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd244018630 con 0x7fd258101850 2024-04-22T23:19:33.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.290+0000 7fd25ef7d700 1 -- 172.21.15.67:0/1094975199 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd258196fa0 con 0x7fd258101850 2024-04-22T23:19:33.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.291+0000 7fd25ef7d700 1 -- 172.21.15.67:0/1094975199 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd25810e0b0 con 0x7fd258101850 2024-04-22T23:19:33.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.291+0000 7fd255ffb700 1 -- 172.21.15.67:0/1094975199 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fd24400fa90 con 0x7fd258101850 2024-04-22T23:19:33.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.292+0000 7fd255ffb700 1 --2- 172.21.15.67:0/1094975199 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fd24806c4e0 0x7fd24806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:33.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.292+0000 7fd255ffb700 1 -- 172.21.15.67:0/1094975199 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fd24408b510 con 0x7fd258101850 2024-04-22T23:19:33.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.292+0000 7fd25cd19700 1 --2- 172.21.15.67:0/1094975199 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fd24806c4e0 0x7fd24806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:33.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.292+0000 7fd255ffb700 1 --2- 172.21.15.67:0/1094975199 >> [v2:172.21.15.67:6810/3382820294,v1:172.21.15.67:6811/3382820294] conn(0x7fd2480721c0 0x7fd2480745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:33.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.292+0000 7fd255ffb700 1 -- 172.21.15.67:0/1094975199 --> [v2:172.21.15.67:6810/3382820294,v1:172.21.15.67:6811/3382820294] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd248074c90 con 0x7fd2480721c0 2024-04-22T23:19:33.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.292+0000 7fd255ffb700 1 -- 172.21.15.67:0/1094975199 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd24408dbe0 con 0x7fd258101850 2024-04-22T23:19:33.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.292+0000 7fd25d51a700 1 --2- 172.21.15.67:0/1094975199 >> [v2:172.21.15.67:6810/3382820294,v1:172.21.15.67:6811/3382820294] conn(0x7fd2480721c0 0x7fd2480745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:33.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.292+0000 7fd25cd19700 1 --2- 172.21.15.67:0/1094975199 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fd24806c4e0 0x7fd24806e9a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd24c006010 tx=0x7fd24c01a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:33.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.292+0000 7fd25d51a700 1 --2- 172.21.15.67:0/1094975199 >> [v2:172.21.15.67:6810/3382820294,v1:172.21.15.67:6811/3382820294] conn(0x7fd2480721c0 0x7fd2480745e0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:33.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.293+0000 7fd255ffb700 1 -- 172.21.15.67:0/1094975199 <== osd.1 v2:172.21.15.67:6810/3382820294 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fd248074c90 con 0x7fd2480721c0 2024-04-22T23:19:33.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.309+0000 7fd25ef7d700 1 -- 172.21.15.67:0/1094975199 --> [v2:172.21.15.67:6810/3382820294,v1:172.21.15.67:6811/3382820294] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd2581974a0 con 0x7fd2480721c0 2024-04-22T23:19:33.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.309+0000 7fd255ffb700 1 -- 172.21.15.67:0/1094975199 <== osd.1 v2:172.21.15.67:6810/3382820294 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd2581974a0 con 0x7fd2480721c0 2024-04-22T23:19:33.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.310+0000 7fd25ef7d700 1 -- 172.21.15.67:0/1094975199 >> [v2:172.21.15.67:6810/3382820294,v1:172.21.15.67:6811/3382820294] conn(0x7fd2480721c0 msgr2=0x7fd2480745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:33.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.310+0000 7fd25ef7d700 1 --2- 172.21.15.67:0/1094975199 >> [v2:172.21.15.67:6810/3382820294,v1:172.21.15.67:6811/3382820294] conn(0x7fd2480721c0 0x7fd2480745e0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:33.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.310+0000 7fd25ef7d700 1 -- 172.21.15.67:0/1094975199 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fd24806c4e0 msgr2=0x7fd24806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:33.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.310+0000 7fd25ef7d700 1 --2- 172.21.15.67:0/1094975199 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fd24806c4e0 0x7fd24806e9a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd24c006010 tx=0x7fd24c01a040 comp rx=0 tx=0).stop 2024-04-22T23:19:33.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.310+0000 7fd25ef7d700 1 -- 172.21.15.67:0/1094975199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd258101850 msgr2=0x7fd25819cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:33.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.310+0000 7fd25ef7d700 1 --2- 172.21.15.67:0/1094975199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd258101850 0x7fd25819cbc0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd24400c960 tx=0x7fd24400cd20 comp rx=0 tx=0).stop 2024-04-22T23:19:33.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.310+0000 7fd25ef7d700 1 -- 172.21.15.67:0/1094975199 shutdown_connections 2024-04-22T23:19:33.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.310+0000 7fd25ef7d700 1 --2- 172.21.15.67:0/1094975199 >> [v2:172.21.15.67:6810/3382820294,v1:172.21.15.67:6811/3382820294] conn(0x7fd2480721c0 0x7fd2480745e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:33.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.310+0000 7fd25ef7d700 1 --2- 172.21.15.67:0/1094975199 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fd24806c4e0 0x7fd24806e9a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:33.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.310+0000 7fd25ef7d700 1 --2- 172.21.15.67:0/1094975199 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd258100ea0 0x7fd25819c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:33.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.310+0000 7fd25ef7d700 1 --2- 172.21.15.67:0/1094975199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd258101850 0x7fd25819cbc0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:33.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.310+0000 7fd25ef7d700 1 -- 172.21.15.67:0/1094975199 >> 172.21.15.67:0/1094975199 conn(0x7fd2580fc750 msgr2=0x7fd2580fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:33.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.310+0000 7fd25ef7d700 1 -- 172.21.15.67:0/1094975199 shutdown_connections 2024-04-22T23:19:33.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.310+0000 7fd25ef7d700 1 -- 172.21.15.67:0/1094975199 wait complete. 2024-04-22T23:19:33.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.750+0000 7fb23bc1b700 1 -- 172.21.15.67:0/1427587876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2340ff5a0 msgr2=0x7fb2340ff980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:33.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.750+0000 7fb23bc1b700 1 --2- 172.21.15.67:0/1427587876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2340ff5a0 0x7fb2340ff980 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb230009910 tx=0x7fb230009c20 comp rx=0 tx=0).stop 2024-04-22T23:19:33.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.750+0000 7fb23bc1b700 1 -- 172.21.15.67:0/1427587876 shutdown_connections 2024-04-22T23:19:33.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.750+0000 7fb23bc1b700 1 --2- 172.21.15.67:0/1427587876 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2340fff50 0x7fb234105840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:33.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.750+0000 7fb23bc1b700 1 --2- 172.21.15.67:0/1427587876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2340ff5a0 0x7fb2340ff980 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:33.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.750+0000 7fb23bc1b700 1 -- 172.21.15.67:0/1427587876 >> 172.21.15.67:0/1427587876 conn(0x7fb234074e30 msgr2=0x7fb234075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:33.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.750+0000 7fb23bc1b700 1 -- 172.21.15.67:0/1427587876 shutdown_connections 2024-04-22T23:19:33.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.750+0000 7fb23bc1b700 1 -- 172.21.15.67:0/1427587876 wait complete. 2024-04-22T23:19:33.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.751+0000 7fb23bc1b700 1 Processor -- start 2024-04-22T23:19:33.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.751+0000 7fb23bc1b700 1 -- start start 2024-04-22T23:19:33.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.751+0000 7fb23bc1b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2340ff5a0 0x7fb234100520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:33.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.751+0000 7fb23bc1b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2340fff50 0x7fb234100a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:33.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.751+0000 7fb23bc1b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb234100fa0 con 0x7fb2340ff5a0 2024-04-22T23:19:33.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.751+0000 7fb23bc1b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2341010e0 con 0x7fb2340fff50 2024-04-22T23:19:33.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.751+0000 7fb2399b7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2340ff5a0 0x7fb234100520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:33.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.751+0000 7fb2399b7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2340ff5a0 0x7fb234100520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37684/0 (socket says 172.21.15.67:37684) 2024-04-22T23:19:33.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.751+0000 7fb2399b7700 1 -- 172.21.15.67:0/3571314636 learned_addr learned my addr 172.21.15.67:0/3571314636 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:19:33.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.752+0000 7fb2391b6700 1 --2- 172.21.15.67:0/3571314636 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2340fff50 0x7fb234100a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:33.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.752+0000 7fb2399b7700 1 -- 172.21.15.67:0/3571314636 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2340fff50 msgr2=0x7fb234100a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:33.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.752+0000 7fb2399b7700 1 --2- 172.21.15.67:0/3571314636 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2340fff50 0x7fb234100a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:33.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.752+0000 7fb2399b7700 1 -- 172.21.15.67:0/3571314636 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2300095f0 con 0x7fb2340ff5a0 2024-04-22T23:19:33.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.752+0000 7fb2399b7700 1 --2- 172.21.15.67:0/3571314636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2340ff5a0 0x7fb234100520 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb230000c00 tx=0x7fb23000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:33.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.752+0000 7fb22affd700 1 -- 172.21.15.67:0/3571314636 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb23001d070 con 0x7fb2340ff5a0 2024-04-22T23:19:33.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.752+0000 7fb23bc1b700 1 -- 172.21.15.67:0/3571314636 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb234101360 con 0x7fb2340ff5a0 2024-04-22T23:19:33.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.752+0000 7fb22affd700 1 -- 172.21.15.67:0/3571314636 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb230005dc0 con 0x7fb2340ff5a0 2024-04-22T23:19:33.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.752+0000 7fb22affd700 1 -- 172.21.15.67:0/3571314636 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb230017400 con 0x7fb2340ff5a0 2024-04-22T23:19:33.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.752+0000 7fb23bc1b700 1 -- 172.21.15.67:0/3571314636 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2341a2db0 con 0x7fb2340ff5a0 2024-04-22T23:19:33.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.753+0000 7fb23bc1b700 1 -- 172.21.15.67:0/3571314636 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb23410e0a0 con 0x7fb2340ff5a0 2024-04-22T23:19:33.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.754+0000 7fb22affd700 1 -- 172.21.15.67:0/3571314636 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fb23000f7f0 con 0x7fb2340ff5a0 2024-04-22T23:19:33.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.754+0000 7fb22affd700 1 --2- 172.21.15.67:0/3571314636 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fb22006c4e0 0x7fb22006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:33.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.754+0000 7fb22affd700 1 -- 172.21.15.67:0/3571314636 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fb23008ce40 con 0x7fb2340ff5a0 2024-04-22T23:19:33.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.754+0000 7fb2391b6700 1 --2- 172.21.15.67:0/3571314636 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fb22006c4e0 0x7fb22006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:33.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.754+0000 7fb22affd700 1 --2- 172.21.15.67:0/3571314636 >> [v2:172.21.15.67:6802/268472413,v1:172.21.15.67:6803/268472413] conn(0x7fb2200721c0 0x7fb2200745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:33.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.754+0000 7fb22affd700 1 -- 172.21.15.67:0/3571314636 --> [v2:172.21.15.67:6802/268472413,v1:172.21.15.67:6803/268472413] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb220074c90 con 0x7fb2200721c0 2024-04-22T23:19:33.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.754+0000 7fb22affd700 1 -- 172.21.15.67:0/3571314636 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb23008d2a0 con 0x7fb2340ff5a0 2024-04-22T23:19:33.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.755+0000 7fb23a1b8700 1 --2- 172.21.15.67:0/3571314636 >> [v2:172.21.15.67:6802/268472413,v1:172.21.15.67:6803/268472413] conn(0x7fb2200721c0 0x7fb2200745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:33.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.755+0000 7fb2391b6700 1 --2- 172.21.15.67:0/3571314636 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fb22006c4e0 0x7fb22006e9a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb224005950 tx=0x7fb2240058e0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:33.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.755+0000 7fb23a1b8700 1 --2- 172.21.15.67:0/3571314636 >> [v2:172.21.15.67:6802/268472413,v1:172.21.15.67:6803/268472413] conn(0x7fb2200721c0 0x7fb2200745e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:33.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.756+0000 7fb22affd700 1 -- 172.21.15.67:0/3571314636 <== osd.0 v2:172.21.15.67:6802/268472413 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fb220074c90 con 0x7fb2200721c0 2024-04-22T23:19:33.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.771+0000 7fb23bc1b700 1 -- 172.21.15.67:0/3571314636 --> [v2:172.21.15.67:6802/268472413,v1:172.21.15.67:6803/268472413] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb23404f2e0 con 0x7fb2200721c0 2024-04-22T23:19:33.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.771+0000 7fb22affd700 1 -- 172.21.15.67:0/3571314636 <== osd.0 v2:172.21.15.67:6802/268472413 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb23404f2e0 con 0x7fb2200721c0 2024-04-22T23:19:33.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.772+0000 7fb23bc1b700 1 -- 172.21.15.67:0/3571314636 >> [v2:172.21.15.67:6802/268472413,v1:172.21.15.67:6803/268472413] conn(0x7fb2200721c0 msgr2=0x7fb2200745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:33.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.772+0000 7fb23bc1b700 1 --2- 172.21.15.67:0/3571314636 >> [v2:172.21.15.67:6802/268472413,v1:172.21.15.67:6803/268472413] conn(0x7fb2200721c0 0x7fb2200745e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:33.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.772+0000 7fb23bc1b700 1 -- 172.21.15.67:0/3571314636 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fb22006c4e0 msgr2=0x7fb22006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:33.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.772+0000 7fb23bc1b700 1 --2- 172.21.15.67:0/3571314636 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fb22006c4e0 0x7fb22006e9a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb224005950 tx=0x7fb2240058e0 comp rx=0 tx=0).stop 2024-04-22T23:19:33.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.772+0000 7fb23bc1b700 1 -- 172.21.15.67:0/3571314636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2340ff5a0 msgr2=0x7fb234100520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:33.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.772+0000 7fb23bc1b700 1 --2- 172.21.15.67:0/3571314636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2340ff5a0 0x7fb234100520 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb230000c00 tx=0x7fb23000be20 comp rx=0 tx=0).stop 2024-04-22T23:19:33.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.772+0000 7fb23bc1b700 1 -- 172.21.15.67:0/3571314636 shutdown_connections 2024-04-22T23:19:33.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.772+0000 7fb23bc1b700 1 --2- 172.21.15.67:0/3571314636 >> [v2:172.21.15.67:6802/268472413,v1:172.21.15.67:6803/268472413] conn(0x7fb2200721c0 0x7fb2200745e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:33.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.772+0000 7fb23bc1b700 1 --2- 172.21.15.67:0/3571314636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2340ff5a0 0x7fb234100520 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:33.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.772+0000 7fb23bc1b700 1 --2- 172.21.15.67:0/3571314636 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fb22006c4e0 0x7fb22006e9a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:33.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.772+0000 7fb23bc1b700 1 --2- 172.21.15.67:0/3571314636 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2340fff50 0x7fb234100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:33.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.772+0000 7fb23bc1b700 1 -- 172.21.15.67:0/3571314636 >> 172.21.15.67:0/3571314636 conn(0x7fb234074e30 msgr2=0x7fb2340fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:33.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.772+0000 7fb23bc1b700 1 -- 172.21.15.67:0/3571314636 shutdown_connections 2024-04-22T23:19:33.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.772+0000 7fb23bc1b700 1 -- 172.21.15.67:0/3571314636 wait complete. 2024-04-22T23:19:33.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.941+0000 7f26367fb700 1 -- 172.21.15.67:0/3911944962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2630106cd0 msgr2=0x7f26301070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:33.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.941+0000 7f26367fb700 1 --2- 172.21.15.67:0/3911944962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2630106cd0 0x7f26301070b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2620009910 tx=0x7f2620009c20 comp rx=0 tx=0).stop 2024-04-22T23:19:33.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.941+0000 7f26367fb700 1 -- 172.21.15.67:0/3911944962 shutdown_connections 2024-04-22T23:19:33.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.941+0000 7f26367fb700 1 --2- 172.21.15.67:0/3911944962 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2630068580 0x7f26300689e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:33.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.941+0000 7f26367fb700 1 --2- 172.21.15.67:0/3911944962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2630106cd0 0x7f26301070b0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:33.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.941+0000 7f26367fb700 1 -- 172.21.15.67:0/3911944962 >> 172.21.15.67:0/3911944962 conn(0x7f2630075670 msgr2=0x7f2630075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:33.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.941+0000 7f26367fb700 1 -- 172.21.15.67:0/3911944962 shutdown_connections 2024-04-22T23:19:33.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.941+0000 7f26367fb700 1 -- 172.21.15.67:0/3911944962 wait complete. 2024-04-22T23:19:33.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.942+0000 7f26367fb700 1 Processor -- start 2024-04-22T23:19:33.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.942+0000 7f26367fb700 1 -- start start 2024-04-22T23:19:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.943+0000 7f26367fb700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2630068580 0x7f263019c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.943+0000 7f26367fb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2630106cd0 0x7f263019cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.943+0000 7f26367fb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f263019d1a0 con 0x7f2630106cd0 2024-04-22T23:19:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.943+0000 7f26367fb700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26301966e0 con 0x7f2630068580 2024-04-22T23:19:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.943+0000 7f262f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2630106cd0 0x7f263019cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.943+0000 7f262f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2630106cd0 0x7f263019cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37698/0 (socket says 172.21.15.67:37698) 2024-04-22T23:19:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.943+0000 7f262f7fe700 1 -- 172.21.15.67:0/856471725 learned_addr learned my addr 172.21.15.67:0/856471725 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:19:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.943+0000 7f262ffff700 1 --2- 172.21.15.67:0/856471725 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2630068580 0x7f263019c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.943+0000 7f262f7fe700 1 -- 172.21.15.67:0/856471725 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2630068580 msgr2=0x7f263019c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.943+0000 7f262f7fe700 1 --2- 172.21.15.67:0/856471725 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2630068580 0x7f263019c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.943+0000 7f262f7fe700 1 -- 172.21.15.67:0/856471725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26200095f0 con 0x7f2630106cd0 2024-04-22T23:19:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.943+0000 7f262f7fe700 1 --2- 172.21.15.67:0/856471725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2630106cd0 0x7f263019cba0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f262400c930 tx=0x7f262400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.944+0000 7f262d7fa700 1 -- 172.21.15.67:0/856471725 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2624007ad0 con 0x7f2630106cd0 2024-04-22T23:19:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.944+0000 7f26367fb700 1 -- 172.21.15.67:0/856471725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26301969c0 con 0x7f2630106cd0 2024-04-22T23:19:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.944+0000 7f262d7fa700 1 -- 172.21.15.67:0/856471725 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2624007c30 con 0x7f2630106cd0 2024-04-22T23:19:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.944+0000 7f262d7fa700 1 -- 172.21.15.67:0/856471725 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2624018630 con 0x7f2630106cd0 2024-04-22T23:19:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.944+0000 7f26367fb700 1 -- 172.21.15.67:0/856471725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2630196f10 con 0x7f2630106cd0 2024-04-22T23:19:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.944+0000 7f26367fb700 1 -- 172.21.15.67:0/856471725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f263010e070 con 0x7f2630106cd0 2024-04-22T23:19:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.945+0000 7f262d7fa700 1 -- 172.21.15.67:0/856471725 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f262400f450 con 0x7f2630106cd0 2024-04-22T23:19:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.946+0000 7f262d7fa700 1 --2- 172.21.15.67:0/856471725 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f261806c4e0 0x7f261806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.946+0000 7f262d7fa700 1 -- 172.21.15.67:0/856471725 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f262408bc60 con 0x7f2630106cd0 2024-04-22T23:19:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.946+0000 7f262ffff700 1 --2- 172.21.15.67:0/856471725 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f261806c4e0 0x7f261806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.946+0000 7f262d7fa700 1 --2- 172.21.15.67:0/856471725 >> [v2:172.21.15.134:6816/3846690247,v1:172.21.15.134:6817/3846690247] conn(0x7f26180721c0 0x7f26180745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.946+0000 7f262d7fa700 1 -- 172.21.15.67:0/856471725 --> [v2:172.21.15.134:6816/3846690247,v1:172.21.15.134:6817/3846690247] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2618074c90 con 0x7f26180721c0 2024-04-22T23:19:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.946+0000 7f262d7fa700 1 -- 172.21.15.67:0/856471725 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f262408c0c0 con 0x7f2630106cd0 2024-04-22T23:19:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.946+0000 7f262ffff700 1 --2- 172.21.15.67:0/856471725 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f261806c4e0 0x7f261806e9a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f262000b5c0 tx=0x7f262001a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.946+0000 7f2634d98700 1 --2- 172.21.15.67:0/856471725 >> [v2:172.21.15.134:6816/3846690247,v1:172.21.15.134:6817/3846690247] conn(0x7f26180721c0 0x7f26180745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.947+0000 7f2634d98700 1 --2- 172.21.15.67:0/856471725 >> [v2:172.21.15.134:6816/3846690247,v1:172.21.15.134:6817/3846690247] conn(0x7f26180721c0 0x7f26180745e0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:33.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.948+0000 7f262d7fa700 1 -- 172.21.15.67:0/856471725 <== osd.5 v2:172.21.15.134:6816/3846690247 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f2618074c90 con 0x7f26180721c0 2024-04-22T23:19:33.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.978+0000 7f26367fb700 1 -- 172.21.15.67:0/856471725 --> [v2:172.21.15.134:6816/3846690247,v1:172.21.15.134:6817/3846690247] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f263004f2e0 con 0x7f26180721c0 2024-04-22T23:19:33.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.979+0000 7f262d7fa700 1 -- 172.21.15.67:0/856471725 <== osd.5 v2:172.21.15.134:6816/3846690247 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f263004f2e0 con 0x7f26180721c0 2024-04-22T23:19:33.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.979+0000 7f26367fb700 1 -- 172.21.15.67:0/856471725 >> [v2:172.21.15.134:6816/3846690247,v1:172.21.15.134:6817/3846690247] conn(0x7f26180721c0 msgr2=0x7f26180745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:33.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.979+0000 7f26367fb700 1 --2- 172.21.15.67:0/856471725 >> [v2:172.21.15.134:6816/3846690247,v1:172.21.15.134:6817/3846690247] conn(0x7f26180721c0 0x7f26180745e0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:33.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.979+0000 7f26367fb700 1 -- 172.21.15.67:0/856471725 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f261806c4e0 msgr2=0x7f261806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:33.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.979+0000 7f26367fb700 1 --2- 172.21.15.67:0/856471725 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f261806c4e0 0x7f261806e9a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f262000b5c0 tx=0x7f262001a040 comp rx=0 tx=0).stop 2024-04-22T23:19:33.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.979+0000 7f26367fb700 1 -- 172.21.15.67:0/856471725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2630106cd0 msgr2=0x7f263019cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:33.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.979+0000 7f26367fb700 1 --2- 172.21.15.67:0/856471725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2630106cd0 0x7f263019cba0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f262400c930 tx=0x7f262400ccf0 comp rx=0 tx=0).stop 2024-04-22T23:19:33.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.980+0000 7f26367fb700 1 -- 172.21.15.67:0/856471725 shutdown_connections 2024-04-22T23:19:33.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.980+0000 7f26367fb700 1 --2- 172.21.15.67:0/856471725 >> [v2:172.21.15.134:6816/3846690247,v1:172.21.15.134:6817/3846690247] conn(0x7f26180721c0 0x7f26180745e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:33.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.980+0000 7f26367fb700 1 --2- 172.21.15.67:0/856471725 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f261806c4e0 0x7f261806e9a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:33.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.980+0000 7f26367fb700 1 --2- 172.21.15.67:0/856471725 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2630068580 0x7f263019c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:33.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.980+0000 7f26367fb700 1 --2- 172.21.15.67:0/856471725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2630106cd0 0x7f263019cba0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:33.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.980+0000 7f26367fb700 1 -- 172.21.15.67:0/856471725 >> 172.21.15.67:0/856471725 conn(0x7f2630075670 msgr2=0x7f26300fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:33.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.980+0000 7f26367fb700 1 -- 172.21.15.67:0/856471725 shutdown_connections 2024-04-22T23:19:33.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:33.980+0000 7f26367fb700 1 -- 172.21.15.67:0/856471725 wait complete. 2024-04-22T23:19:34.641 INFO:teuthology.orchestra.run.smithi067.stdout:60129542174 2024-04-22T23:19:34.642 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-22T23:19:34.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:34 smithi067 ceph-mon[26453]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:34.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:34 smithi134 ceph-mon[34609]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:36.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.208+0000 7fefa463e700 1 -- 172.21.15.67:0/2245472063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef9c105bd0 msgr2=0x7fef9c105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:36.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.208+0000 7fefa463e700 1 --2- 172.21.15.67:0/2245472063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef9c105bd0 0x7fef9c105fb0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fef9400b0a0 tx=0x7fef9400b3b0 comp rx=0 tx=0).stop 2024-04-22T23:19:36.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.208+0000 7fefa463e700 1 -- 172.21.15.67:0/2245472063 shutdown_connections 2024-04-22T23:19:36.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.208+0000 7fefa463e700 1 --2- 172.21.15.67:0/2245472063 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef9c0ffc20 0x7fef9c100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:36.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.208+0000 7fefa463e700 1 --2- 172.21.15.67:0/2245472063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef9c105bd0 0x7fef9c105fb0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:36.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.208+0000 7fefa463e700 1 -- 172.21.15.67:0/2245472063 >> 172.21.15.67:0/2245472063 conn(0x7fef9c075670 msgr2=0x7fef9c075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:36.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.208+0000 7fefa463e700 1 -- 172.21.15.67:0/2245472063 shutdown_connections 2024-04-22T23:19:36.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.208+0000 7fefa463e700 1 -- 172.21.15.67:0/2245472063 wait complete. 2024-04-22T23:19:36.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.209+0000 7fefa463e700 1 Processor -- start 2024-04-22T23:19:36.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.209+0000 7fefa463e700 1 -- start start 2024-04-22T23:19:36.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.209+0000 7fefa463e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef9c0ffc20 0x7fef9c19c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:36.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.209+0000 7fefa463e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef9c105bd0 0x7fef9c19cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:36.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.209+0000 7fefa463e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c19d2d0 con 0x7fef9c105bd0 2024-04-22T23:19:36.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.209+0000 7fefa463e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c196750 con 0x7fef9c0ffc20 2024-04-22T23:19:36.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.209+0000 7fefa1bd9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef9c105bd0 0x7fef9c19cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:36.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.209+0000 7fefa1bd9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef9c105bd0 0x7fef9c19cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37712/0 (socket says 172.21.15.67:37712) 2024-04-22T23:19:36.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.209+0000 7fefa1bd9700 1 -- 172.21.15.67:0/234997027 learned_addr learned my addr 172.21.15.67:0/234997027 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:19:36.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.209+0000 7fefa23da700 1 --2- 172.21.15.67:0/234997027 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef9c0ffc20 0x7fef9c19c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:36.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.209+0000 7fefa1bd9700 1 -- 172.21.15.67:0/234997027 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef9c0ffc20 msgr2=0x7fef9c19c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:36.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.209+0000 7fefa1bd9700 1 --2- 172.21.15.67:0/234997027 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef9c0ffc20 0x7fef9c19c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:36.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.209+0000 7fefa1bd9700 1 -- 172.21.15.67:0/234997027 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef94009d00 con 0x7fef9c105bd0 2024-04-22T23:19:36.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.210+0000 7fefa1bd9700 1 --2- 172.21.15.67:0/234997027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef9c105bd0 0x7fef9c19cbf0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fef9000c8a0 tx=0x7fef9000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:36.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.210+0000 7fef9b7fe700 1 -- 172.21.15.67:0/234997027 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef9000cea0 con 0x7fef9c105bd0 2024-04-22T23:19:36.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.210+0000 7fef9b7fe700 1 -- 172.21.15.67:0/234997027 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fef90004830 con 0x7fef9c105bd0 2024-04-22T23:19:36.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.210+0000 7fefa463e700 1 -- 172.21.15.67:0/234997027 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef9c196a30 con 0x7fef9c105bd0 2024-04-22T23:19:36.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.210+0000 7fef9b7fe700 1 -- 172.21.15.67:0/234997027 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef900055b0 con 0x7fef9c105bd0 2024-04-22T23:19:36.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.210+0000 7fefa463e700 1 -- 172.21.15.67:0/234997027 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef9c196f80 con 0x7fef9c105bd0 2024-04-22T23:19:36.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.211+0000 7fefa463e700 1 -- 172.21.15.67:0/234997027 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fef9c10cfa0 con 0x7fef9c105bd0 2024-04-22T23:19:36.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.212+0000 7fef9b7fe700 1 -- 172.21.15.67:0/234997027 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fef90007ad0 con 0x7fef9c105bd0 2024-04-22T23:19:36.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.212+0000 7fef9b7fe700 1 --2- 172.21.15.67:0/234997027 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fef8406c4e0 0x7fef8406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:36.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.212+0000 7fef9b7fe700 1 -- 172.21.15.67:0/234997027 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fef9008b540 con 0x7fef9c105bd0 2024-04-22T23:19:36.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.212+0000 7fefa23da700 1 --2- 172.21.15.67:0/234997027 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fef8406c4e0 0x7fef8406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:36.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.212+0000 7fef9b7fe700 1 --2- 172.21.15.67:0/234997027 >> [v2:172.21.15.67:6818/3835340581,v1:172.21.15.67:6819/3835340581] conn(0x7fef840721c0 0x7fef840745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:36.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.212+0000 7fef9b7fe700 1 -- 172.21.15.67:0/234997027 --> [v2:172.21.15.67:6818/3835340581,v1:172.21.15.67:6819/3835340581] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fef84074c90 con 0x7fef840721c0 2024-04-22T23:19:36.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.212+0000 7fef9b7fe700 1 -- 172.21.15.67:0/234997027 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fef9008b9c0 con 0x7fef9c105bd0 2024-04-22T23:19:36.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.212+0000 7fefa2bdb700 1 --2- 172.21.15.67:0/234997027 >> [v2:172.21.15.67:6818/3835340581,v1:172.21.15.67:6819/3835340581] conn(0x7fef840721c0 0x7fef840745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:36.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.212+0000 7fefa23da700 1 --2- 172.21.15.67:0/234997027 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fef8406c4e0 0x7fef8406e9a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fef94015040 tx=0x7fef94000f40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:36.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.213+0000 7fefa2bdb700 1 --2- 172.21.15.67:0/234997027 >> [v2:172.21.15.67:6818/3835340581,v1:172.21.15.67:6819/3835340581] conn(0x7fef840721c0 0x7fef840745e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:36.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.213+0000 7fef9b7fe700 1 -- 172.21.15.67:0/234997027 <== osd.2 v2:172.21.15.67:6818/3835340581 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fef84074c90 con 0x7fef840721c0 2024-04-22T23:19:36.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.229+0000 7fefa463e700 1 -- 172.21.15.67:0/234997027 --> [v2:172.21.15.67:6818/3835340581,v1:172.21.15.67:6819/3835340581] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fef9c04f2e0 con 0x7fef840721c0 2024-04-22T23:19:36.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.229+0000 7fef9b7fe700 1 -- 172.21.15.67:0/234997027 <== osd.2 v2:172.21.15.67:6818/3835340581 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fef9c04f2e0 con 0x7fef840721c0 2024-04-22T23:19:36.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.229+0000 7fefa463e700 1 -- 172.21.15.67:0/234997027 >> [v2:172.21.15.67:6818/3835340581,v1:172.21.15.67:6819/3835340581] conn(0x7fef840721c0 msgr2=0x7fef840745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:36.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.229+0000 7fefa463e700 1 --2- 172.21.15.67:0/234997027 >> [v2:172.21.15.67:6818/3835340581,v1:172.21.15.67:6819/3835340581] conn(0x7fef840721c0 0x7fef840745e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:36.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.230+0000 7fefa463e700 1 -- 172.21.15.67:0/234997027 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fef8406c4e0 msgr2=0x7fef8406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:36.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.230+0000 7fefa463e700 1 --2- 172.21.15.67:0/234997027 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fef8406c4e0 0x7fef8406e9a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fef94015040 tx=0x7fef94000f40 comp rx=0 tx=0).stop 2024-04-22T23:19:36.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.230+0000 7fefa463e700 1 -- 172.21.15.67:0/234997027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef9c105bd0 msgr2=0x7fef9c19cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:36.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.230+0000 7fefa463e700 1 --2- 172.21.15.67:0/234997027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef9c105bd0 0x7fef9c19cbf0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fef9000c8a0 tx=0x7fef9000cc60 comp rx=0 tx=0).stop 2024-04-22T23:19:36.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.230+0000 7fefa463e700 1 -- 172.21.15.67:0/234997027 shutdown_connections 2024-04-22T23:19:36.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.230+0000 7fefa463e700 1 --2- 172.21.15.67:0/234997027 >> [v2:172.21.15.67:6818/3835340581,v1:172.21.15.67:6819/3835340581] conn(0x7fef840721c0 0x7fef840745e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:36.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.230+0000 7fefa463e700 1 --2- 172.21.15.67:0/234997027 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fef8406c4e0 0x7fef8406e9a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:36.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.230+0000 7fefa463e700 1 --2- 172.21.15.67:0/234997027 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef9c0ffc20 0x7fef9c19c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:36.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.230+0000 7fefa463e700 1 --2- 172.21.15.67:0/234997027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef9c105bd0 0x7fef9c19cbf0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:36.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.230+0000 7fefa463e700 1 -- 172.21.15.67:0/234997027 >> 172.21.15.67:0/234997027 conn(0x7fef9c075670 msgr2=0x7fef9c068720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:36.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.230+0000 7fefa463e700 1 -- 172.21.15.67:0/234997027 shutdown_connections 2024-04-22T23:19:36.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.230+0000 7fefa463e700 1 -- 172.21.15.67:0/234997027 wait complete. 2024-04-22T23:19:36.241 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:36 smithi067 ceph-mon[26453]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:36.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.433+0000 7f94ef54f700 1 -- 172.21.15.67:0/3974322291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94e8100750 msgr2=0x7f94e81047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:36.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.433+0000 7f94ef54f700 1 --2- 172.21.15.67:0/3974322291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94e8100750 0x7f94e81047a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f94d8009910 tx=0x7f94d8009c20 comp rx=0 tx=0).stop 2024-04-22T23:19:36.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.433+0000 7f94ef54f700 1 -- 172.21.15.67:0/3974322291 shutdown_connections 2024-04-22T23:19:36.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.433+0000 7f94ef54f700 1 --2- 172.21.15.67:0/3974322291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94e8100750 0x7f94e81047a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:36.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.433+0000 7f94ef54f700 1 --2- 172.21.15.67:0/3974322291 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f94e80ffda0 0x7f94e8100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:36.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.433+0000 7f94ef54f700 1 -- 172.21.15.67:0/3974322291 >> 172.21.15.67:0/3974322291 conn(0x7f94e80fb650 msgr2=0x7f94e80fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:36.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.434+0000 7f94ef54f700 1 -- 172.21.15.67:0/3974322291 shutdown_connections 2024-04-22T23:19:36.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.434+0000 7f94ef54f700 1 -- 172.21.15.67:0/3974322291 wait complete. 2024-04-22T23:19:36.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.434+0000 7f94ef54f700 1 Processor -- start 2024-04-22T23:19:36.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.434+0000 7f94ef54f700 1 -- start start 2024-04-22T23:19:36.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.435+0000 7f94ef54f700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f94e80ffda0 0x7f94e81977c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:36.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.435+0000 7f94ef54f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94e8100750 0x7f94e8197d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:36.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.435+0000 7f94ef54f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e8198320 con 0x7f94e8100750 2024-04-22T23:19:36.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.435+0000 7f94ef54f700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e8198460 con 0x7f94e80ffda0 2024-04-22T23:19:36.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.435+0000 7f94ecaea700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94e8100750 0x7f94e8197d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:36.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.435+0000 7f94ecaea700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94e8100750 0x7f94e8197d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37730/0 (socket says 172.21.15.67:37730) 2024-04-22T23:19:36.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.435+0000 7f94ecaea700 1 -- 172.21.15.67:0/3660587281 learned_addr learned my addr 172.21.15.67:0/3660587281 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:19:36.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.435+0000 7f94ed2eb700 1 --2- 172.21.15.67:0/3660587281 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f94e80ffda0 0x7f94e81977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:36.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.435+0000 7f94ecaea700 1 -- 172.21.15.67:0/3660587281 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f94e80ffda0 msgr2=0x7f94e81977c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:36.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.435+0000 7f94ecaea700 1 --2- 172.21.15.67:0/3660587281 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f94e80ffda0 0x7f94e81977c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:36.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.435+0000 7f94ecaea700 1 -- 172.21.15.67:0/3660587281 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d80095f0 con 0x7f94e8100750 2024-04-22T23:19:36.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.435+0000 7f94ecaea700 1 --2- 172.21.15.67:0/3660587281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94e8100750 0x7f94e8197d00 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f94d800bda0 tx=0x7f94d800bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:36.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.435+0000 7f94de7fc700 1 -- 172.21.15.67:0/3660587281 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f94d801d070 con 0x7f94e8100750 2024-04-22T23:19:36.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.435+0000 7f94de7fc700 1 -- 172.21.15.67:0/3660587281 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f94d800f460 con 0x7f94e8100750 2024-04-22T23:19:36.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.435+0000 7f94de7fc700 1 -- 172.21.15.67:0/3660587281 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f94d8017630 con 0x7f94e8100750 2024-04-22T23:19:36.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.435+0000 7f94ef54f700 1 -- 172.21.15.67:0/3660587281 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94e81a0830 con 0x7f94e8100750 2024-04-22T23:19:36.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.435+0000 7f94ef54f700 1 -- 172.21.15.67:0/3660587281 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94e81a0be0 con 0x7f94e8100750 2024-04-22T23:19:36.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.436+0000 7f94ef54f700 1 -- 172.21.15.67:0/3660587281 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f94e810cfc0 con 0x7f94e8100750 2024-04-22T23:19:36.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.437+0000 7f94de7fc700 1 -- 172.21.15.67:0/3660587281 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f94d800fab0 con 0x7f94e8100750 2024-04-22T23:19:36.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.437+0000 7f94de7fc700 1 --2- 172.21.15.67:0/3660587281 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f94d406c4e0 0x7f94d406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:36.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.437+0000 7f94de7fc700 1 -- 172.21.15.67:0/3660587281 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f94d808cca0 con 0x7f94e8100750 2024-04-22T23:19:36.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.437+0000 7f94ed2eb700 1 --2- 172.21.15.67:0/3660587281 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f94d406c4e0 0x7f94d406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:36.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.438+0000 7f94de7fc700 1 --2- 172.21.15.67:0/3660587281 >> [v2:172.21.15.134:6808/668644810,v1:172.21.15.134:6809/668644810] conn(0x7f94d40721c0 0x7f94d40745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:36.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.438+0000 7f94de7fc700 1 -- 172.21.15.67:0/3660587281 --> [v2:172.21.15.134:6808/668644810,v1:172.21.15.134:6809/668644810] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f94d4074c90 con 0x7f94d40721c0 2024-04-22T23:19:36.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.438+0000 7f94de7fc700 1 -- 172.21.15.67:0/3660587281 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f94d808d100 con 0x7f94e8100750 2024-04-22T23:19:36.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.438+0000 7f94ed2eb700 1 --2- 172.21.15.67:0/3660587281 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f94d406c4e0 0x7f94d406e9a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f94e4005950 tx=0x7f94e400b410 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:36.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.438+0000 7f94edaec700 1 --2- 172.21.15.67:0/3660587281 >> [v2:172.21.15.134:6808/668644810,v1:172.21.15.134:6809/668644810] conn(0x7f94d40721c0 0x7f94d40745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:36.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.438+0000 7f94edaec700 1 --2- 172.21.15.67:0/3660587281 >> [v2:172.21.15.134:6808/668644810,v1:172.21.15.134:6809/668644810] conn(0x7f94d40721c0 0x7f94d40745e0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:36.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.439+0000 7f94de7fc700 1 -- 172.21.15.67:0/3660587281 <== osd.4 v2:172.21.15.134:6808/668644810 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f94d4074c90 con 0x7f94d40721c0 2024-04-22T23:19:36.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.455+0000 7f94ef54f700 1 -- 172.21.15.67:0/3660587281 --> [v2:172.21.15.134:6808/668644810,v1:172.21.15.134:6809/668644810] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f94e804f2e0 con 0x7f94d40721c0 2024-04-22T23:19:36.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.455+0000 7f94de7fc700 1 -- 172.21.15.67:0/3660587281 <== osd.4 v2:172.21.15.134:6808/668644810 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f94e804f2e0 con 0x7f94d40721c0 2024-04-22T23:19:36.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.456+0000 7f94ef54f700 1 -- 172.21.15.67:0/3660587281 >> [v2:172.21.15.134:6808/668644810,v1:172.21.15.134:6809/668644810] conn(0x7f94d40721c0 msgr2=0x7f94d40745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:36.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.456+0000 7f94ef54f700 1 --2- 172.21.15.67:0/3660587281 >> [v2:172.21.15.134:6808/668644810,v1:172.21.15.134:6809/668644810] conn(0x7f94d40721c0 0x7f94d40745e0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:36.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.456+0000 7f94ef54f700 1 -- 172.21.15.67:0/3660587281 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f94d406c4e0 msgr2=0x7f94d406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:36.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.456+0000 7f94ef54f700 1 --2- 172.21.15.67:0/3660587281 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f94d406c4e0 0x7f94d406e9a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f94e4005950 tx=0x7f94e400b410 comp rx=0 tx=0).stop 2024-04-22T23:19:36.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.456+0000 7f94ef54f700 1 -- 172.21.15.67:0/3660587281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94e8100750 msgr2=0x7f94e8197d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:36.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.456+0000 7f94ef54f700 1 --2- 172.21.15.67:0/3660587281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94e8100750 0x7f94e8197d00 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f94d800bda0 tx=0x7f94d800bdd0 comp rx=0 tx=0).stop 2024-04-22T23:19:36.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.456+0000 7f94ef54f700 1 -- 172.21.15.67:0/3660587281 shutdown_connections 2024-04-22T23:19:36.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.456+0000 7f94ef54f700 1 --2- 172.21.15.67:0/3660587281 >> [v2:172.21.15.134:6808/668644810,v1:172.21.15.134:6809/668644810] conn(0x7f94d40721c0 0x7f94d40745e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:36.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.456+0000 7f94ef54f700 1 --2- 172.21.15.67:0/3660587281 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f94d406c4e0 0x7f94d406e9a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:36.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.456+0000 7f94ef54f700 1 --2- 172.21.15.67:0/3660587281 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f94e80ffda0 0x7f94e81977c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:36.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.456+0000 7f94ef54f700 1 --2- 172.21.15.67:0/3660587281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94e8100750 0x7f94e8197d00 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:36.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.456+0000 7f94ef54f700 1 -- 172.21.15.67:0/3660587281 >> 172.21.15.67:0/3660587281 conn(0x7f94e80fb650 msgr2=0x7f94e80fccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:36.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.456+0000 7f94ef54f700 1 -- 172.21.15.67:0/3660587281 shutdown_connections 2024-04-22T23:19:36.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:36.456+0000 7f94ef54f700 1 -- 172.21.15.67:0/3660587281 wait complete. 2024-04-22T23:19:36.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:36 smithi134 ceph-mon[34609]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:37.170 INFO:teuthology.orchestra.run.smithi067.stdout:38654705699 2024-04-22T23:19:37.171 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-22T23:19:37.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:37 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:19:37.810 INFO:teuthology.orchestra.run.smithi067.stdout:81604378649 2024-04-22T23:19:37.810 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-22T23:19:37.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:37 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:19:38.324 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:19:38.346 INFO:teuthology.orchestra.run.smithi067.stdout:146028888079 2024-04-22T23:19:38.347 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-22T23:19:38.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:38 smithi067 ceph-mon[26453]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:38.752 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:19:38.777 INFO:teuthology.orchestra.run.smithi067.stdout:167503724553 2024-04-22T23:19:38.777 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-22T23:19:38.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:38 smithi134 ceph-mon[34609]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:39.132 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:19:40.118 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:19:40.118 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:19:40.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:40 smithi067 ceph-mon[26453]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:40.796 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:40 smithi134 ceph-mon[34609]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:42.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.422+0000 7f52a4e55700 1 -- 172.21.15.67:0/2595120836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52a0068580 msgr2=0x7f52a00689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:42.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.422+0000 7f52a4e55700 1 --2- 172.21.15.67:0/2595120836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52a0068580 0x7f52a00689e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5294009910 tx=0x7f5294009c20 comp rx=0 tx=0).stop 2024-04-22T23:19:42.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.422+0000 7f52a4e55700 1 -- 172.21.15.67:0/2595120836 shutdown_connections 2024-04-22T23:19:42.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.422+0000 7f52a4e55700 1 --2- 172.21.15.67:0/2595120836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52a0068580 0x7f52a00689e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:42.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.422+0000 7f52a4e55700 1 --2- 172.21.15.67:0/2595120836 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52a0106cd0 0x7f52a01070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:42.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.422+0000 7f52a4e55700 1 -- 172.21.15.67:0/2595120836 >> 172.21.15.67:0/2595120836 conn(0x7f52a0075670 msgr2=0x7f52a0075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:42.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.422+0000 7f52a4e55700 1 -- 172.21.15.67:0/2595120836 shutdown_connections 2024-04-22T23:19:42.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.422+0000 7f52a4e55700 1 -- 172.21.15.67:0/2595120836 wait complete. 2024-04-22T23:19:42.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.423+0000 7f52a4e55700 1 Processor -- start 2024-04-22T23:19:42.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.423+0000 7f52a4e55700 1 -- start start 2024-04-22T23:19:42.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.423+0000 7f52a4e55700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52a0068580 0x7f52a00ffcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:42.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.423+0000 7f52a4e55700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52a0106cd0 0x7f52a0100230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:42.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.423+0000 7f52a4e55700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a0100770 con 0x7f52a0068580 2024-04-22T23:19:42.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.423+0000 7f52a4e55700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a01008b0 con 0x7f52a0106cd0 2024-04-22T23:19:42.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.423+0000 7f529e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52a0068580 0x7f52a00ffcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:42.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.423+0000 7f529e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52a0068580 0x7f52a00ffcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33760/0 (socket says 172.21.15.67:33760) 2024-04-22T23:19:42.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.423+0000 7f529e59c700 1 -- 172.21.15.67:0/4052714078 learned_addr learned my addr 172.21.15.67:0/4052714078 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:19:42.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.423+0000 7f529dd9b700 1 --2- 172.21.15.67:0/4052714078 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52a0106cd0 0x7f52a0100230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:42.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.423+0000 7f529e59c700 1 -- 172.21.15.67:0/4052714078 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52a0106cd0 msgr2=0x7f52a0100230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:42.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.423+0000 7f529e59c700 1 --2- 172.21.15.67:0/4052714078 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52a0106cd0 0x7f52a0100230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:42.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.423+0000 7f529e59c700 1 -- 172.21.15.67:0/4052714078 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52940095f0 con 0x7f52a0068580 2024-04-22T23:19:42.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.423+0000 7f529dd9b700 1 --2- 172.21.15.67:0/4052714078 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52a0106cd0 0x7f52a0100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:19:42.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.424+0000 7f529e59c700 1 --2- 172.21.15.67:0/4052714078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52a0068580 0x7f52a00ffcf0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f529000d8d0 tx=0x7f529000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:42.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.424+0000 7f528f7fe700 1 -- 172.21.15.67:0/4052714078 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5290009940 con 0x7f52a0068580 2024-04-22T23:19:42.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.424+0000 7f528f7fe700 1 -- 172.21.15.67:0/4052714078 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5290010460 con 0x7f52a0068580 2024-04-22T23:19:42.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.424+0000 7f52a4e55700 1 -- 172.21.15.67:0/4052714078 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52a01a2a30 con 0x7f52a0068580 2024-04-22T23:19:42.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.424+0000 7f528f7fe700 1 -- 172.21.15.67:0/4052714078 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f529000f5f0 con 0x7f52a0068580 2024-04-22T23:19:42.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.424+0000 7f52a4e55700 1 -- 172.21.15.67:0/4052714078 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52a01a2e40 con 0x7f52a0068580 2024-04-22T23:19:42.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.425+0000 7f52a4e55700 1 -- 172.21.15.67:0/4052714078 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52a01091d0 con 0x7f52a0068580 2024-04-22T23:19:42.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.425+0000 7f528f7fe700 1 -- 172.21.15.67:0/4052714078 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f52900105d0 con 0x7f52a0068580 2024-04-22T23:19:42.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.426+0000 7f528f7fe700 1 --2- 172.21.15.67:0/4052714078 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f528806c4e0 0x7f528806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:42.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.426+0000 7f528f7fe700 1 -- 172.21.15.67:0/4052714078 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f529008b4f0 con 0x7f52a0068580 2024-04-22T23:19:42.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.426+0000 7f529dd9b700 1 --2- 172.21.15.67:0/4052714078 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f528806c4e0 0x7f528806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:42.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.426+0000 7f529dd9b700 1 --2- 172.21.15.67:0/4052714078 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f528806c4e0 0x7f528806e9a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f52a01014e0 tx=0x7f529401a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:42.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.430+0000 7f528f7fe700 1 -- 172.21.15.67:0/4052714078 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f5290056010 con 0x7f52a0068580 2024-04-22T23:19:42.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.629+0000 7f52a4e55700 1 -- 172.21.15.67:0/4052714078 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f52a0108bb0 con 0x7f52a0068580 2024-04-22T23:19:42.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.629+0000 7f528f7fe700 1 -- 172.21.15.67:0/4052714078 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f5290059630 con 0x7f52a0068580 2024-04-22T23:19:42.632 INFO:teuthology.orchestra.run.smithi067.stdout:111669149717 2024-04-22T23:19:42.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.631+0000 7f52a4e55700 1 -- 172.21.15.67:0/4052714078 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f528806c4e0 msgr2=0x7f528806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:42.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.631+0000 7f52a4e55700 1 --2- 172.21.15.67:0/4052714078 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f528806c4e0 0x7f528806e9a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f52a01014e0 tx=0x7f529401a040 comp rx=0 tx=0).stop 2024-04-22T23:19:42.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.631+0000 7f52a4e55700 1 -- 172.21.15.67:0/4052714078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52a0068580 msgr2=0x7f52a00ffcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:42.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.631+0000 7f52a4e55700 1 --2- 172.21.15.67:0/4052714078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52a0068580 0x7f52a00ffcf0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f529000d8d0 tx=0x7f529000dc90 comp rx=0 tx=0).stop 2024-04-22T23:19:42.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.631+0000 7f52a4e55700 1 -- 172.21.15.67:0/4052714078 shutdown_connections 2024-04-22T23:19:42.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.631+0000 7f52a4e55700 1 --2- 172.21.15.67:0/4052714078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52a0068580 0x7f52a00ffcf0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:42.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.631+0000 7f52a4e55700 1 --2- 172.21.15.67:0/4052714078 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f528806c4e0 0x7f528806e9a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:42.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.631+0000 7f52a4e55700 1 --2- 172.21.15.67:0/4052714078 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52a0106cd0 0x7f52a0100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:42.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.631+0000 7f52a4e55700 1 -- 172.21.15.67:0/4052714078 >> 172.21.15.67:0/4052714078 conn(0x7f52a0075670 msgr2=0x7f52a00feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:42.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.632+0000 7f52a4e55700 1 -- 172.21.15.67:0/4052714078 shutdown_connections 2024-04-22T23:19:42.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:42.632+0000 7f52a4e55700 1 -- 172.21.15.67:0/4052714078 wait complete. 2024-04-22T23:19:42.644 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:42 smithi067 ceph-mon[26453]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:42.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:42 smithi134 ceph-mon[34609]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:43.728 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:43 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/4052714078' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-22T23:19:43.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:43 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/4052714078' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-22T23:19:44.555 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:44 smithi067 ceph-mon[26453]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:44.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:44 smithi134 ceph-mon[34609]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:45.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.004+0000 7f635b060700 1 -- 172.21.15.67:0/1015406341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6354101850 msgr2=0x7f63541058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:45.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.004+0000 7f635b060700 1 --2- 172.21.15.67:0/1015406341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6354101850 0x7f63541058a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6344009910 tx=0x7f6344009c20 comp rx=0 tx=0).stop 2024-04-22T23:19:45.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.004+0000 7f635b060700 1 -- 172.21.15.67:0/1015406341 shutdown_connections 2024-04-22T23:19:45.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.004+0000 7f635b060700 1 --2- 172.21.15.67:0/1015406341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6354101850 0x7f63541058a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:45.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.004+0000 7f635b060700 1 --2- 172.21.15.67:0/1015406341 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6354100ea0 0x7f6354101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:45.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.004+0000 7f635b060700 1 -- 172.21.15.67:0/1015406341 >> 172.21.15.67:0/1015406341 conn(0x7f63540fc750 msgr2=0x7f63540feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:45.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.004+0000 7f635b060700 1 -- 172.21.15.67:0/1015406341 shutdown_connections 2024-04-22T23:19:45.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.004+0000 7f635b060700 1 -- 172.21.15.67:0/1015406341 wait complete. 2024-04-22T23:19:45.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.005+0000 7f635b060700 1 Processor -- start 2024-04-22T23:19:45.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.005+0000 7f635b060700 1 -- start start 2024-04-22T23:19:45.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.005+0000 7f635b060700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6354100ea0 0x7f6354196970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:45.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.005+0000 7f635b060700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6354101850 0x7f6354196eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:45.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.005+0000 7f635b060700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6354197540 con 0x7f6354101850 2024-04-22T23:19:45.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.005+0000 7f635b060700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635419b350 con 0x7f6354100ea0 2024-04-22T23:19:45.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.005+0000 7f6353fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6354101850 0x7f6354196eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:45.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.005+0000 7f6353fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6354101850 0x7f6354196eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33784/0 (socket says 172.21.15.67:33784) 2024-04-22T23:19:45.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.005+0000 7f6353fff700 1 -- 172.21.15.67:0/4163915250 learned_addr learned my addr 172.21.15.67:0/4163915250 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:19:45.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.005+0000 7f6358dfc700 1 --2- 172.21.15.67:0/4163915250 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6354100ea0 0x7f6354196970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:45.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.005+0000 7f6353fff700 1 -- 172.21.15.67:0/4163915250 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6354100ea0 msgr2=0x7f6354196970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:45.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.005+0000 7f6353fff700 1 --2- 172.21.15.67:0/4163915250 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6354100ea0 0x7f6354196970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:45.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.005+0000 7f6353fff700 1 -- 172.21.15.67:0/4163915250 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63440095f0 con 0x7f6354101850 2024-04-22T23:19:45.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.005+0000 7f6353fff700 1 --2- 172.21.15.67:0/4163915250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6354101850 0x7f6354196eb0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f63440092e0 tx=0x7f634400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:45.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.006+0000 7f6351ffb700 1 -- 172.21.15.67:0/4163915250 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f634401d070 con 0x7f6354101850 2024-04-22T23:19:45.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.006+0000 7f6351ffb700 1 -- 172.21.15.67:0/4163915250 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6344022470 con 0x7f6354101850 2024-04-22T23:19:45.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.006+0000 7f635b060700 1 -- 172.21.15.67:0/4163915250 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f635419b5d0 con 0x7f6354101850 2024-04-22T23:19:45.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.006+0000 7f6351ffb700 1 -- 172.21.15.67:0/4163915250 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f634400f690 con 0x7f6354101850 2024-04-22T23:19:45.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.006+0000 7f635b060700 1 -- 172.21.15.67:0/4163915250 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f635419bac0 con 0x7f6354101850 2024-04-22T23:19:45.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.007+0000 7f635b060700 1 -- 172.21.15.67:0/4163915250 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63541091f0 con 0x7f6354101850 2024-04-22T23:19:45.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.007+0000 7f6351ffb700 1 -- 172.21.15.67:0/4163915250 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f63440225e0 con 0x7f6354101850 2024-04-22T23:19:45.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.008+0000 7f6351ffb700 1 --2- 172.21.15.67:0/4163915250 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f634006c4e0 0x7f634006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:45.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.008+0000 7f6351ffb700 1 -- 172.21.15.67:0/4163915250 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f634408ca30 con 0x7f6354101850 2024-04-22T23:19:45.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.008+0000 7f6358dfc700 1 --2- 172.21.15.67:0/4163915250 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f634006c4e0 0x7f634006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:45.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.008+0000 7f6358dfc700 1 --2- 172.21.15.67:0/4163915250 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f634006c4e0 0x7f634006e9a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f633c006fd0 tx=0x7f633c008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:45.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.012+0000 7f6351ffb700 1 -- 172.21.15.67:0/4163915250 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f634405b110 con 0x7f6354101850 2024-04-22T23:19:45.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.219+0000 7f635b060700 1 -- 172.21.15.67:0/4163915250 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f6354197d20 con 0x7f6354101850 2024-04-22T23:19:45.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.220+0000 7f6351ffb700 1 -- 172.21.15.67:0/4163915250 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6344027020 con 0x7f6354101850 2024-04-22T23:19:45.222 INFO:teuthology.orchestra.run.smithi067.stdout:38654705701 2024-04-22T23:19:45.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.222+0000 7f635b060700 1 -- 172.21.15.67:0/4163915250 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f634006c4e0 msgr2=0x7f634006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:45.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.222+0000 7f635b060700 1 --2- 172.21.15.67:0/4163915250 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f634006c4e0 0x7f634006e9a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f633c006fd0 tx=0x7f633c008040 comp rx=0 tx=0).stop 2024-04-22T23:19:45.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.222+0000 7f635b060700 1 -- 172.21.15.67:0/4163915250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6354101850 msgr2=0x7f6354196eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:45.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.222+0000 7f635b060700 1 --2- 172.21.15.67:0/4163915250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6354101850 0x7f6354196eb0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f63440092e0 tx=0x7f634400be20 comp rx=0 tx=0).stop 2024-04-22T23:19:45.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.222+0000 7f635b060700 1 -- 172.21.15.67:0/4163915250 shutdown_connections 2024-04-22T23:19:45.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.222+0000 7f635b060700 1 --2- 172.21.15.67:0/4163915250 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f634006c4e0 0x7f634006e9a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:45.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.222+0000 7f635b060700 1 --2- 172.21.15.67:0/4163915250 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6354100ea0 0x7f6354196970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:45.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.222+0000 7f635b060700 1 --2- 172.21.15.67:0/4163915250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6354101850 0x7f6354196eb0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:45.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.222+0000 7f635b060700 1 -- 172.21.15.67:0/4163915250 >> 172.21.15.67:0/4163915250 conn(0x7f63540fc750 msgr2=0x7f63540fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:45.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.222+0000 7f635b060700 1 -- 172.21.15.67:0/4163915250 shutdown_connections 2024-04-22T23:19:45.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:45.222+0000 7f635b060700 1 -- 172.21.15.67:0/4163915250 wait complete. 2024-04-22T23:19:46.209 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:46 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/4163915250' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-22T23:19:46.209 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:46 smithi067 ceph-mon[26453]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:46.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:46 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/4163915250' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-22T23:19:46.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:46 smithi134 ceph-mon[34609]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:46.506 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705699 got 38654705701 for osd.0 2024-04-22T23:19:46.506 DEBUG:teuthology.parallel:result is None 2024-04-22T23:19:47.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.170+0000 7fd1fa10e700 1 -- 172.21.15.67:0/3568068357 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1f4102700 msgr2=0x7fd1f410abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:47.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.170+0000 7fd1fa10e700 1 --2- 172.21.15.67:0/3568068357 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1f4102700 0x7fd1f410abf0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd1e8009910 tx=0x7fd1e8009c20 comp rx=0 tx=0).stop 2024-04-22T23:19:47.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.170+0000 7fd1fa10e700 1 -- 172.21.15.67:0/3568068357 shutdown_connections 2024-04-22T23:19:47.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.170+0000 7fd1fa10e700 1 --2- 172.21.15.67:0/3568068357 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1f4102700 0x7fd1f410abf0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:47.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.170+0000 7fd1fa10e700 1 --2- 172.21.15.67:0/3568068357 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd1f4101de0 0x7fd1f41021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:47.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.170+0000 7fd1fa10e700 1 -- 172.21.15.67:0/3568068357 >> 172.21.15.67:0/3568068357 conn(0x7fd1f40fb650 msgr2=0x7fd1f40fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:47.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.170+0000 7fd1fa10e700 1 -- 172.21.15.67:0/3568068357 shutdown_connections 2024-04-22T23:19:47.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.170+0000 7fd1fa10e700 1 -- 172.21.15.67:0/3568068357 wait complete. 2024-04-22T23:19:47.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.171+0000 7fd1fa10e700 1 Processor -- start 2024-04-22T23:19:47.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.171+0000 7fd1fa10e700 1 -- start start 2024-04-22T23:19:47.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.171+0000 7fd1fa10e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd1f4101de0 0x7fd1f419c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:47.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.171+0000 7fd1fa10e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1f4102700 0x7fd1f419cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:47.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.171+0000 7fd1fa10e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f419d370 con 0x7fd1f4102700 2024-04-22T23:19:47.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.171+0000 7fd1fa10e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f4196820 con 0x7fd1f4101de0 2024-04-22T23:19:47.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.171+0000 7fd1f2ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1f4102700 0x7fd1f419cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:47.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.171+0000 7fd1f2ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1f4102700 0x7fd1f419cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33800/0 (socket says 172.21.15.67:33800) 2024-04-22T23:19:47.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.171+0000 7fd1f2ffd700 1 -- 172.21.15.67:0/1280389781 learned_addr learned my addr 172.21.15.67:0/1280389781 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:19:47.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.172+0000 7fd1f2ffd700 1 -- 172.21.15.67:0/1280389781 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd1f4101de0 msgr2=0x7fd1f419c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:47.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.172+0000 7fd1f2ffd700 1 --2- 172.21.15.67:0/1280389781 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd1f4101de0 0x7fd1f419c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:47.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.172+0000 7fd1f2ffd700 1 -- 172.21.15.67:0/1280389781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e80095f0 con 0x7fd1f4102700 2024-04-22T23:19:47.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.172+0000 7fd1f2ffd700 1 --2- 172.21.15.67:0/1280389781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1f4102700 0x7fd1f419cce0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd1e8005f50 tx=0x7fd1e800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:47.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.172+0000 7fd1f0ff9700 1 -- 172.21.15.67:0/1280389781 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1e801d070 con 0x7fd1f4102700 2024-04-22T23:19:47.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.172+0000 7fd1f0ff9700 1 -- 172.21.15.67:0/1280389781 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd1e800f460 con 0x7fd1f4102700 2024-04-22T23:19:47.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.172+0000 7fd1fa10e700 1 -- 172.21.15.67:0/1280389781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1f4196aa0 con 0x7fd1f4102700 2024-04-22T23:19:47.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.172+0000 7fd1f0ff9700 1 -- 172.21.15.67:0/1280389781 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1e8022650 con 0x7fd1f4102700 2024-04-22T23:19:47.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.172+0000 7fd1fa10e700 1 -- 172.21.15.67:0/1280389781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1f4196f90 con 0x7fd1f4102700 2024-04-22T23:19:47.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.173+0000 7fd1fa10e700 1 -- 172.21.15.67:0/1280389781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1f41082f0 con 0x7fd1f4102700 2024-04-22T23:19:47.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.173+0000 7fd1f0ff9700 1 -- 172.21.15.67:0/1280389781 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fd1e800f5d0 con 0x7fd1f4102700 2024-04-22T23:19:47.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.174+0000 7fd1f0ff9700 1 --2- 172.21.15.67:0/1280389781 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fd1dc06c4e0 0x7fd1dc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:47.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.174+0000 7fd1f37fe700 1 --2- 172.21.15.67:0/1280389781 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fd1dc06c4e0 0x7fd1dc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:47.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.174+0000 7fd1f0ff9700 1 -- 172.21.15.67:0/1280389781 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fd1e808c8a0 con 0x7fd1f4102700 2024-04-22T23:19:47.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.174+0000 7fd1f37fe700 1 --2- 172.21.15.67:0/1280389781 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fd1dc06c4e0 0x7fd1dc06e9a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd1e40097b0 tx=0x7fd1e4006d20 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:47.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.179+0000 7fd1f0ff9700 1 -- 172.21.15.67:0/1280389781 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd1e805aed0 con 0x7fd1f4102700 2024-04-22T23:19:47.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.379+0000 7fd1fa10e700 1 -- 172.21.15.67:0/1280389781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fd1f404f2e0 con 0x7fd1f4102700 2024-04-22T23:19:47.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.379+0000 7fd1f0ff9700 1 -- 172.21.15.67:0/1280389781 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd1e805aa60 con 0x7fd1f4102700 2024-04-22T23:19:47.382 INFO:teuthology.orchestra.run.smithi067.stdout:60129542176 2024-04-22T23:19:47.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.381+0000 7fd1fa10e700 1 -- 172.21.15.67:0/1280389781 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fd1dc06c4e0 msgr2=0x7fd1dc06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:47.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.381+0000 7fd1fa10e700 1 --2- 172.21.15.67:0/1280389781 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fd1dc06c4e0 0x7fd1dc06e9a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd1e40097b0 tx=0x7fd1e4006d20 comp rx=0 tx=0).stop 2024-04-22T23:19:47.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.381+0000 7fd1fa10e700 1 -- 172.21.15.67:0/1280389781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1f4102700 msgr2=0x7fd1f419cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:47.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.381+0000 7fd1fa10e700 1 --2- 172.21.15.67:0/1280389781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1f4102700 0x7fd1f419cce0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd1e8005f50 tx=0x7fd1e800be20 comp rx=0 tx=0).stop 2024-04-22T23:19:47.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.381+0000 7fd1fa10e700 1 -- 172.21.15.67:0/1280389781 shutdown_connections 2024-04-22T23:19:47.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.381+0000 7fd1fa10e700 1 --2- 172.21.15.67:0/1280389781 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fd1dc06c4e0 0x7fd1dc06e9a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:47.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.381+0000 7fd1fa10e700 1 --2- 172.21.15.67:0/1280389781 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd1f4101de0 0x7fd1f419c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:47.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.381+0000 7fd1fa10e700 1 --2- 172.21.15.67:0/1280389781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1f4102700 0x7fd1f419cce0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:47.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.381+0000 7fd1fa10e700 1 -- 172.21.15.67:0/1280389781 >> 172.21.15.67:0/1280389781 conn(0x7fd1f40fb650 msgr2=0x7fd1f4105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:47.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.382+0000 7fd1fa10e700 1 -- 172.21.15.67:0/1280389781 shutdown_connections 2024-04-22T23:19:47.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.382+0000 7fd1fa10e700 1 -- 172.21.15.67:0/1280389781 wait complete. 2024-04-22T23:19:47.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.386+0000 7f52f9e15700 1 -- 172.21.15.67:0/86751614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52f4105bd0 msgr2=0x7f52f4105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:47.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.386+0000 7f52f9e15700 1 --2- 172.21.15.67:0/86751614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52f4105bd0 0x7f52f4105fb0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f52e4009910 tx=0x7f52e4009c20 comp rx=0 tx=0).stop 2024-04-22T23:19:47.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.387+0000 7f52f9e15700 1 -- 172.21.15.67:0/86751614 shutdown_connections 2024-04-22T23:19:47.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.387+0000 7f52f9e15700 1 --2- 172.21.15.67:0/86751614 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52f40ffc20 0x7f52f4100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:47.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.387+0000 7f52f9e15700 1 --2- 172.21.15.67:0/86751614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52f4105bd0 0x7f52f4105fb0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:47.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.387+0000 7f52f9e15700 1 -- 172.21.15.67:0/86751614 >> 172.21.15.67:0/86751614 conn(0x7f52f4075670 msgr2=0x7f52f4075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:47.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.387+0000 7f52f9e15700 1 -- 172.21.15.67:0/86751614 shutdown_connections 2024-04-22T23:19:47.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.387+0000 7f52f9e15700 1 -- 172.21.15.67:0/86751614 wait complete. 2024-04-22T23:19:47.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.388+0000 7f52f9e15700 1 Processor -- start 2024-04-22T23:19:47.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.388+0000 7f52f9e15700 1 -- start start 2024-04-22T23:19:47.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.388+0000 7f52f9e15700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52f40ffc20 0x7f52f41976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:47.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.388+0000 7f52f9e15700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52f4105bd0 0x7f52f4197c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:47.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.388+0000 7f52f9e15700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f41982f0 con 0x7f52f4105bd0 2024-04-22T23:19:47.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.388+0000 7f52f9e15700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f4198460 con 0x7f52f40ffc20 2024-04-22T23:19:47.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.388+0000 7f52f2ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52f4105bd0 0x7f52f4197c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:47.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.388+0000 7f52f2ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52f4105bd0 0x7f52f4197c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33828/0 (socket says 172.21.15.67:33828) 2024-04-22T23:19:47.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.388+0000 7f52f2ffd700 1 -- 172.21.15.67:0/959841988 learned_addr learned my addr 172.21.15.67:0/959841988 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:19:47.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.388+0000 7f52f37fe700 1 --2- 172.21.15.67:0/959841988 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52f40ffc20 0x7f52f41976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:47.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.388+0000 7f52f2ffd700 1 -- 172.21.15.67:0/959841988 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52f40ffc20 msgr2=0x7f52f41976d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:47.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.388+0000 7f52f2ffd700 1 --2- 172.21.15.67:0/959841988 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52f40ffc20 0x7f52f41976d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:47.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.389+0000 7f52f2ffd700 1 -- 172.21.15.67:0/959841988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e40095f0 con 0x7f52f4105bd0 2024-04-22T23:19:47.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.389+0000 7f52f2ffd700 1 --2- 172.21.15.67:0/959841988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52f4105bd0 0x7f52f4197c10 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f52e800d8d0 tx=0x7f52e800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:47.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.389+0000 7f52f0ff9700 1 -- 172.21.15.67:0/959841988 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52e80098a0 con 0x7f52f4105bd0 2024-04-22T23:19:47.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.389+0000 7f52f9e15700 1 -- 172.21.15.67:0/959841988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52f41a0740 con 0x7f52f4105bd0 2024-04-22T23:19:47.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.389+0000 7f52f0ff9700 1 -- 172.21.15.67:0/959841988 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f52e8010460 con 0x7f52f4105bd0 2024-04-22T23:19:47.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.389+0000 7f52f0ff9700 1 -- 172.21.15.67:0/959841988 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52e800f5f0 con 0x7f52f4105bd0 2024-04-22T23:19:47.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.389+0000 7f52f9e15700 1 -- 172.21.15.67:0/959841988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52f41a0c90 con 0x7f52f4105bd0 2024-04-22T23:19:47.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.390+0000 7f52f9e15700 1 -- 172.21.15.67:0/959841988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52f41080d0 con 0x7f52f4105bd0 2024-04-22T23:19:47.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.391+0000 7f52f0ff9700 1 -- 172.21.15.67:0/959841988 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f52e800f820 con 0x7f52f4105bd0 2024-04-22T23:19:47.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.391+0000 7f52f0ff9700 1 --2- 172.21.15.67:0/959841988 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f52dc06c4e0 0x7f52dc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:47.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.391+0000 7f52f0ff9700 1 -- 172.21.15.67:0/959841988 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f52e808b3d0 con 0x7f52f4105bd0 2024-04-22T23:19:47.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.391+0000 7f52f37fe700 1 --2- 172.21.15.67:0/959841988 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f52dc06c4e0 0x7f52dc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:47.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.392+0000 7f52f37fe700 1 --2- 172.21.15.67:0/959841988 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f52dc06c4e0 0x7f52dc06e9a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f52e400b5c0 tx=0x7f52e401a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:47.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.396+0000 7f52f0ff9700 1 -- 172.21.15.67:0/959841988 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f52e8055f60 con 0x7f52f4105bd0 2024-04-22T23:19:47.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.556+0000 7fb4c007e700 1 -- 172.21.15.67:0/3764569408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4b8100750 msgr2=0x7fb4b81047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:47.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.556+0000 7fb4c007e700 1 --2- 172.21.15.67:0/3764569408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4b8100750 0x7fb4b81047a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb4a8009910 tx=0x7fb4a8009c20 comp rx=0 tx=0).stop 2024-04-22T23:19:47.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.556+0000 7fb4c007e700 1 -- 172.21.15.67:0/3764569408 shutdown_connections 2024-04-22T23:19:47.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.556+0000 7fb4c007e700 1 --2- 172.21.15.67:0/3764569408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4b8100750 0x7fb4b81047a0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:47.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.556+0000 7fb4c007e700 1 --2- 172.21.15.67:0/3764569408 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb4b80ffda0 0x7fb4b8100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:47.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.556+0000 7fb4c007e700 1 -- 172.21.15.67:0/3764569408 >> 172.21.15.67:0/3764569408 conn(0x7fb4b80fb650 msgr2=0x7fb4b80fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:47.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.556+0000 7fb4c007e700 1 -- 172.21.15.67:0/3764569408 shutdown_connections 2024-04-22T23:19:47.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.556+0000 7fb4c007e700 1 -- 172.21.15.67:0/3764569408 wait complete. 2024-04-22T23:19:47.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.556+0000 7fb4c007e700 1 Processor -- start 2024-04-22T23:19:47.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.556+0000 7fb4c007e700 1 -- start start 2024-04-22T23:19:47.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.557+0000 7fb4c007e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4b80ffda0 0x7fb4b81955b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:47.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.557+0000 7fb4c007e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb4b8100750 0x7fb4b8195af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:47.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.557+0000 7fb4c007e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b8196030 con 0x7fb4b80ffda0 2024-04-22T23:19:47.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.557+0000 7fb4c007e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b81961a0 con 0x7fb4b8100750 2024-04-22T23:19:47.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.557+0000 7fb4bde1a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4b80ffda0 0x7fb4b81955b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:47.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.557+0000 7fb4bde1a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4b80ffda0 0x7fb4b81955b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33838/0 (socket says 172.21.15.67:33838) 2024-04-22T23:19:47.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.557+0000 7fb4bde1a700 1 -- 172.21.15.67:0/409287905 learned_addr learned my addr 172.21.15.67:0/409287905 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:19:47.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.557+0000 7fb4bd619700 1 --2- 172.21.15.67:0/409287905 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb4b8100750 0x7fb4b8195af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:47.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.557+0000 7fb4bde1a700 1 -- 172.21.15.67:0/409287905 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb4b8100750 msgr2=0x7fb4b8195af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:47.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.557+0000 7fb4bde1a700 1 --2- 172.21.15.67:0/409287905 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb4b8100750 0x7fb4b8195af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:47.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.557+0000 7fb4bde1a700 1 -- 172.21.15.67:0/409287905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a80095f0 con 0x7fb4b80ffda0 2024-04-22T23:19:47.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.557+0000 7fb4bde1a700 1 --2- 172.21.15.67:0/409287905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4b80ffda0 0x7fb4b81955b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb4b400e920 tx=0x7fb4b400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:47.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.557+0000 7fb4aeffd700 1 -- 172.21.15.67:0/409287905 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4b4009790 con 0x7fb4b80ffda0 2024-04-22T23:19:47.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.557+0000 7fb4aeffd700 1 -- 172.21.15.67:0/409287905 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb4b400cba0 con 0x7fb4b80ffda0 2024-04-22T23:19:47.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.558+0000 7fb4c007e700 1 -- 172.21.15.67:0/409287905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4b81a2c20 con 0x7fb4b80ffda0 2024-04-22T23:19:47.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.558+0000 7fb4aeffd700 1 -- 172.21.15.67:0/409287905 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4b4018630 con 0x7fb4b80ffda0 2024-04-22T23:19:47.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.558+0000 7fb4c007e700 1 -- 172.21.15.67:0/409287905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4b81a2f80 con 0x7fb4b80ffda0 2024-04-22T23:19:47.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.558+0000 7fb4c007e700 1 -- 172.21.15.67:0/409287905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4b81081a0 con 0x7fb4b80ffda0 2024-04-22T23:19:47.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.559+0000 7fb4aeffd700 1 -- 172.21.15.67:0/409287905 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fb4b400cd10 con 0x7fb4b80ffda0 2024-04-22T23:19:47.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.559+0000 7fb4aeffd700 1 --2- 172.21.15.67:0/409287905 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fb4a406c4e0 0x7fb4a406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:47.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.560+0000 7fb4aeffd700 1 -- 172.21.15.67:0/409287905 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fb4b4014070 con 0x7fb4b80ffda0 2024-04-22T23:19:47.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.560+0000 7fb4bd619700 1 --2- 172.21.15.67:0/409287905 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fb4a406c4e0 0x7fb4a406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:47.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.560+0000 7fb4bd619700 1 --2- 172.21.15.67:0/409287905 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fb4a406c4e0 0x7fb4a406e9a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8069770 tx=0x7fb4a8005940 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:47.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.568+0000 7fb4aeffd700 1 -- 172.21.15.67:0/409287905 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb4b40520e0 con 0x7fb4b80ffda0 2024-04-22T23:19:47.571 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:19:47.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.611+0000 7f52f9e15700 1 -- 172.21.15.67:0/959841988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f52f4107ab0 con 0x7f52f4105bd0 2024-04-22T23:19:47.614 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:47 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/1280389781' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-22T23:19:47.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.612+0000 7f52f0ff9700 1 -- 172.21.15.67:0/959841988 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f52e8059580 con 0x7f52f4105bd0 2024-04-22T23:19:47.614 INFO:teuthology.orchestra.run.smithi067.stdout:146028888080 2024-04-22T23:19:47.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.614+0000 7f52f9e15700 1 -- 172.21.15.67:0/959841988 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f52dc06c4e0 msgr2=0x7f52dc06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:47.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.614+0000 7f52f9e15700 1 --2- 172.21.15.67:0/959841988 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f52dc06c4e0 0x7f52dc06e9a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f52e400b5c0 tx=0x7f52e401a040 comp rx=0 tx=0).stop 2024-04-22T23:19:47.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.614+0000 7f52f9e15700 1 -- 172.21.15.67:0/959841988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52f4105bd0 msgr2=0x7f52f4197c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:47.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.614+0000 7f52f9e15700 1 --2- 172.21.15.67:0/959841988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52f4105bd0 0x7f52f4197c10 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f52e800d8d0 tx=0x7f52e800dbe0 comp rx=0 tx=0).stop 2024-04-22T23:19:47.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.614+0000 7f52f9e15700 1 -- 172.21.15.67:0/959841988 shutdown_connections 2024-04-22T23:19:47.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.614+0000 7f52f9e15700 1 --2- 172.21.15.67:0/959841988 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f52dc06c4e0 0x7f52dc06e9a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:47.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.614+0000 7f52f9e15700 1 --2- 172.21.15.67:0/959841988 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f52f40ffc20 0x7f52f41976d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:47.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.614+0000 7f52f9e15700 1 --2- 172.21.15.67:0/959841988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52f4105bd0 0x7f52f4197c10 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:47.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.614+0000 7f52f9e15700 1 -- 172.21.15.67:0/959841988 >> 172.21.15.67:0/959841988 conn(0x7f52f4075670 msgr2=0x7f52f40feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:47.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.614+0000 7f52f9e15700 1 -- 172.21.15.67:0/959841988 shutdown_connections 2024-04-22T23:19:47.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.614+0000 7f52f9e15700 1 -- 172.21.15.67:0/959841988 wait complete. 2024-04-22T23:19:47.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:47 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/1280389781' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-22T23:19:47.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.781+0000 7fb4c007e700 1 -- 172.21.15.67:0/409287905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fb4b80694d0 con 0x7fb4b80ffda0 2024-04-22T23:19:47.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.781+0000 7fb4aeffd700 1 -- 172.21.15.67:0/409287905 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb4b4059ab0 con 0x7fb4b80ffda0 2024-04-22T23:19:47.785 INFO:teuthology.orchestra.run.smithi067.stdout:81604378651 2024-04-22T23:19:47.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.783+0000 7fb4c007e700 1 -- 172.21.15.67:0/409287905 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fb4a406c4e0 msgr2=0x7fb4a406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:47.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.783+0000 7fb4c007e700 1 --2- 172.21.15.67:0/409287905 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fb4a406c4e0 0x7fb4a406e9a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8069770 tx=0x7fb4a8005940 comp rx=0 tx=0).stop 2024-04-22T23:19:47.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.784+0000 7fb4c007e700 1 -- 172.21.15.67:0/409287905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4b80ffda0 msgr2=0x7fb4b81955b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:47.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.784+0000 7fb4c007e700 1 --2- 172.21.15.67:0/409287905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4b80ffda0 0x7fb4b81955b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb4b400e920 tx=0x7fb4b400ece0 comp rx=0 tx=0).stop 2024-04-22T23:19:47.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.784+0000 7fb4c007e700 1 -- 172.21.15.67:0/409287905 shutdown_connections 2024-04-22T23:19:47.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.784+0000 7fb4c007e700 1 --2- 172.21.15.67:0/409287905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4b80ffda0 0x7fb4b81955b0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:47.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.784+0000 7fb4c007e700 1 --2- 172.21.15.67:0/409287905 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fb4a406c4e0 0x7fb4a406e9a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:47.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.784+0000 7fb4c007e700 1 --2- 172.21.15.67:0/409287905 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb4b8100750 0x7fb4b8195af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:47.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.784+0000 7fb4c007e700 1 -- 172.21.15.67:0/409287905 >> 172.21.15.67:0/409287905 conn(0x7fb4b80fb650 msgr2=0x7fb4b80fccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:47.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.784+0000 7fb4c007e700 1 -- 172.21.15.67:0/409287905 shutdown_connections 2024-04-22T23:19:47.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:47.784+0000 7fb4c007e700 1 -- 172.21.15.67:0/409287905 wait complete. 2024-04-22T23:19:48.110 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149714 got 111669149717 for osd.3 2024-04-22T23:19:48.110 DEBUG:teuthology.parallel:result is None 2024-04-22T23:19:48.710 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:48 smithi067 ceph-mon[26453]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:48.710 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:48 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/959841988' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-22T23:19:48.710 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:48 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/409287905' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-22T23:19:48.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:48 smithi134 ceph-mon[34609]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:48.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:48 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/959841988' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-22T23:19:48.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:48 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/409287905' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-22T23:19:49.094 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542174 got 60129542176 for osd.1 2024-04-22T23:19:49.094 DEBUG:teuthology.parallel:result is None 2024-04-22T23:19:49.574 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378649 got 81604378651 for osd.2 2024-04-22T23:19:49.574 DEBUG:teuthology.parallel:result is None 2024-04-22T23:19:50.143 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888079 got 146028888080 for osd.4 2024-04-22T23:19:50.143 DEBUG:teuthology.parallel:result is None 2024-04-22T23:19:50.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:50 smithi067 ceph-mon[26453]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:50.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:50 smithi134 ceph-mon[34609]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:51.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:51.856+0000 7f063e989700 1 -- 172.21.15.67:0/632162571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0638105bf0 msgr2=0x7f0638105fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:51.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:51.856+0000 7f063e989700 1 --2- 172.21.15.67:0/632162571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0638105bf0 0x7f0638105fd0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0628009910 tx=0x7f0628009c20 comp rx=0 tx=0).stop 2024-04-22T23:19:51.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:51.856+0000 7f063e989700 1 -- 172.21.15.67:0/632162571 shutdown_connections 2024-04-22T23:19:51.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:51.856+0000 7f063e989700 1 --2- 172.21.15.67:0/632162571 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06380ffb40 0x7f06380fffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:51.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:51.856+0000 7f063e989700 1 --2- 172.21.15.67:0/632162571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0638105bf0 0x7f0638105fd0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:51.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:51.856+0000 7f063e989700 1 -- 172.21.15.67:0/632162571 >> 172.21.15.67:0/632162571 conn(0x7f06380fb610 msgr2=0x7f06380fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:51.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:51.856+0000 7f063e989700 1 -- 172.21.15.67:0/632162571 shutdown_connections 2024-04-22T23:19:51.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:51.856+0000 7f063e989700 1 -- 172.21.15.67:0/632162571 wait complete. 2024-04-22T23:19:51.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:51.856+0000 7f063e989700 1 Processor -- start 2024-04-22T23:19:51.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:51.857+0000 7f063e989700 1 -- start start 2024-04-22T23:19:51.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:51.857+0000 7f063e989700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06380ffb40 0x7f06381976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:51.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:51.857+0000 7f063e989700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0638105bf0 0x7f0638197c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:51.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:51.857+0000 7f063e989700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0638198220 con 0x7f06380ffb40 2024-04-22T23:19:51.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:51.857+0000 7f063e989700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0638198360 con 0x7f0638105bf0 2024-04-22T23:19:51.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:51.857+0000 7f0637fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06380ffb40 0x7f06381976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:51.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:51.857+0000 7f0637fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06380ffb40 0x7f06381976c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46124/0 (socket says 172.21.15.67:46124) 2024-04-22T23:19:51.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:51.857+0000 7f0637fff700 1 -- 172.21.15.67:0/3498003747 learned_addr learned my addr 172.21.15.67:0/3498003747 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:19:51.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:51.857+0000 7f06377fe700 1 --2- 172.21.15.67:0/3498003747 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0638105bf0 0x7f0638197c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:51.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:51.857+0000 7f0637fff700 1 -- 172.21.15.67:0/3498003747 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0638105bf0 msgr2=0x7f0638197c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:51.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:51.857+0000 7f0637fff700 1 --2- 172.21.15.67:0/3498003747 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0638105bf0 0x7f0638197c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:51.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:51.857+0000 7f0637fff700 1 -- 172.21.15.67:0/3498003747 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06280095f0 con 0x7f06380ffb40 2024-04-22T23:19:51.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:51.858+0000 7f0637fff700 1 --2- 172.21.15.67:0/3498003747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06380ffb40 0x7f06381976c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0628006010 tx=0x7f062800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:51.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:51.858+0000 7f06357fa700 1 -- 172.21.15.67:0/3498003747 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f062801d070 con 0x7f06380ffb40 2024-04-22T23:19:51.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:51.858+0000 7f063e989700 1 -- 172.21.15.67:0/3498003747 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06381a0730 con 0x7f06380ffb40 2024-04-22T23:19:51.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:51.858+0000 7f06357fa700 1 -- 172.21.15.67:0/3498003747 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f062800f460 con 0x7f06380ffb40 2024-04-22T23:19:51.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:51.858+0000 7f06357fa700 1 -- 172.21.15.67:0/3498003747 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0628021640 con 0x7f06380ffb40 2024-04-22T23:19:51.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:51.858+0000 7f063e989700 1 -- 172.21.15.67:0/3498003747 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06381a0b40 con 0x7f06380ffb40 2024-04-22T23:19:51.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:51.859+0000 7f063e989700 1 -- 172.21.15.67:0/3498003747 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0638108020 con 0x7f06380ffb40 2024-04-22T23:19:51.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:51.859+0000 7f06357fa700 1 -- 172.21.15.67:0/3498003747 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f062800faf0 con 0x7f06380ffb40 2024-04-22T23:19:51.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:51.860+0000 7f06357fa700 1 --2- 172.21.15.67:0/3498003747 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f062006c4e0 0x7f062006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:51.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:51.860+0000 7f06377fe700 1 --2- 172.21.15.67:0/3498003747 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f062006c4e0 0x7f062006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:51.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:51.860+0000 7f06357fa700 1 -- 172.21.15.67:0/3498003747 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f062808c790 con 0x7f06380ffb40 2024-04-22T23:19:51.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:51.860+0000 7f06377fe700 1 --2- 172.21.15.67:0/3498003747 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f062006c4e0 0x7f062006e9a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f06380696c0 tx=0x7f062c005d10 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:51.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:51.864+0000 7f06357fa700 1 -- 172.21.15.67:0/3498003747 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f06280572b0 con 0x7f06380ffb40 2024-04-22T23:19:52.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:52.064+0000 7f063e989700 1 -- 172.21.15.67:0/3498003747 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f0638107a00 con 0x7f06380ffb40 2024-04-22T23:19:52.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:52.065+0000 7f06357fa700 1 -- 172.21.15.67:0/3498003747 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f062805a8d0 con 0x7f06380ffb40 2024-04-22T23:19:52.066 INFO:teuthology.orchestra.run.smithi067.stdout:167503724557 2024-04-22T23:19:52.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:52.067+0000 7f063e989700 1 -- 172.21.15.67:0/3498003747 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f062006c4e0 msgr2=0x7f062006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:52.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:52.067+0000 7f063e989700 1 --2- 172.21.15.67:0/3498003747 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f062006c4e0 0x7f062006e9a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f06380696c0 tx=0x7f062c005d10 comp rx=0 tx=0).stop 2024-04-22T23:19:52.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:52.067+0000 7f063e989700 1 -- 172.21.15.67:0/3498003747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06380ffb40 msgr2=0x7f06381976c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:52.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:52.067+0000 7f063e989700 1 --2- 172.21.15.67:0/3498003747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06380ffb40 0x7f06381976c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0628006010 tx=0x7f062800be20 comp rx=0 tx=0).stop 2024-04-22T23:19:52.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:52.067+0000 7f063e989700 1 -- 172.21.15.67:0/3498003747 shutdown_connections 2024-04-22T23:19:52.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:52.067+0000 7f063e989700 1 --2- 172.21.15.67:0/3498003747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06380ffb40 0x7f06381976c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:52.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:52.067+0000 7f063e989700 1 --2- 172.21.15.67:0/3498003747 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f062006c4e0 0x7f062006e9a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:52.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:52.067+0000 7f063e989700 1 --2- 172.21.15.67:0/3498003747 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0638105bf0 0x7f0638197c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:52.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:52.067+0000 7f063e989700 1 -- 172.21.15.67:0/3498003747 >> 172.21.15.67:0/3498003747 conn(0x7f06380fb610 msgr2=0x7f06380fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:52.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:52.067+0000 7f063e989700 1 -- 172.21.15.67:0/3498003747 shutdown_connections 2024-04-22T23:19:52.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:52.067+0000 7f063e989700 1 -- 172.21.15.67:0/3498003747 wait complete. 2024-04-22T23:19:52.780 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:52 smithi067 ceph-mon[26453]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:52.781 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:52 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/3498003747' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-22T23:19:52.781 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:52 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:19:52.806 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724553 got 167503724557 for osd.5 2024-04-22T23:19:52.806 DEBUG:teuthology.parallel:result is None 2024-04-22T23:19:52.806 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-22T23:19:52.806 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-22T23:19:52.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:52 smithi134 ceph-mon[34609]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:52.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:52 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/3498003747' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-22T23:19:52.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:52 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:19:53.023 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:19:54.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.586+0000 7f53beef1700 1 -- 172.21.15.67:0/498994118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53b8101850 msgr2=0x7f53b81058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:54.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.586+0000 7f53beef1700 1 --2- 172.21.15.67:0/498994118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53b8101850 0x7f53b81058a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f53a4009910 tx=0x7f53a4009c20 comp rx=0 tx=0).stop 2024-04-22T23:19:54.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.586+0000 7f53beef1700 1 -- 172.21.15.67:0/498994118 shutdown_connections 2024-04-22T23:19:54.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.586+0000 7f53beef1700 1 --2- 172.21.15.67:0/498994118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53b8101850 0x7f53b81058a0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:54.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.586+0000 7f53beef1700 1 --2- 172.21.15.67:0/498994118 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f53b8100ea0 0x7f53b8101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:54.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.586+0000 7f53beef1700 1 -- 172.21.15.67:0/498994118 >> 172.21.15.67:0/498994118 conn(0x7f53b80fc750 msgr2=0x7f53b80feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:54.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.586+0000 7f53beef1700 1 -- 172.21.15.67:0/498994118 shutdown_connections 2024-04-22T23:19:54.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.586+0000 7f53beef1700 1 -- 172.21.15.67:0/498994118 wait complete. 2024-04-22T23:19:54.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.586+0000 7f53beef1700 1 Processor -- start 2024-04-22T23:19:54.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.587+0000 7f53beef1700 1 -- start start 2024-04-22T23:19:54.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.587+0000 7f53beef1700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f53b8100ea0 0x7f53b819c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:54.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.587+0000 7f53beef1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53b8101850 0x7f53b819cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:54.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.587+0000 7f53beef1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b819d2d0 con 0x7f53b8101850 2024-04-22T23:19:54.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.587+0000 7f53beef1700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b8196780 con 0x7f53b8100ea0 2024-04-22T23:19:54.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.587+0000 7f53b7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53b8101850 0x7f53b819cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:54.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.587+0000 7f53b7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53b8101850 0x7f53b819cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46154/0 (socket says 172.21.15.67:46154) 2024-04-22T23:19:54.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.587+0000 7f53b7fff700 1 -- 172.21.15.67:0/3803589753 learned_addr learned my addr 172.21.15.67:0/3803589753 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:19:54.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.587+0000 7f53bcc8d700 1 --2- 172.21.15.67:0/3803589753 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f53b8100ea0 0x7f53b819c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:54.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.587+0000 7f53b7fff700 1 -- 172.21.15.67:0/3803589753 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f53b8100ea0 msgr2=0x7f53b819c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:54.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.587+0000 7f53b7fff700 1 --2- 172.21.15.67:0/3803589753 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f53b8100ea0 0x7f53b819c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:54.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.587+0000 7f53b7fff700 1 -- 172.21.15.67:0/3803589753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a40095f0 con 0x7f53b8101850 2024-04-22T23:19:54.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.587+0000 7f53b7fff700 1 --2- 172.21.15.67:0/3803589753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53b8101850 0x7f53b819cc40 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f53a40092e0 tx=0x7f53a400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:54.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.587+0000 7f53b5ffb700 1 -- 172.21.15.67:0/3803589753 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53a401d070 con 0x7f53b8101850 2024-04-22T23:19:54.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.588+0000 7f53beef1700 1 -- 172.21.15.67:0/3803589753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53b8196a00 con 0x7f53b8101850 2024-04-22T23:19:54.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.588+0000 7f53b5ffb700 1 -- 172.21.15.67:0/3803589753 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f53a4005dc0 con 0x7f53b8101850 2024-04-22T23:19:54.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.588+0000 7f53b5ffb700 1 -- 172.21.15.67:0/3803589753 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53a4017400 con 0x7f53b8101850 2024-04-22T23:19:54.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.588+0000 7f53beef1700 1 -- 172.21.15.67:0/3803589753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53b8196ef0 con 0x7f53b8101850 2024-04-22T23:19:54.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.589+0000 7f53beef1700 1 -- 172.21.15.67:0/3803589753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53b81092a0 con 0x7f53b8101850 2024-04-22T23:19:54.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.589+0000 7f53b5ffb700 1 -- 172.21.15.67:0/3803589753 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f53a400f460 con 0x7f53b8101850 2024-04-22T23:19:54.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.590+0000 7f53b5ffb700 1 --2- 172.21.15.67:0/3803589753 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f53a806c4e0 0x7f53a806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:54.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.590+0000 7f53b5ffb700 1 -- 172.21.15.67:0/3803589753 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f53a408c9b0 con 0x7f53b8101850 2024-04-22T23:19:54.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.590+0000 7f53bcc8d700 1 --2- 172.21.15.67:0/3803589753 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f53a806c4e0 0x7f53a806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:54.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.590+0000 7f53bcc8d700 1 --2- 172.21.15.67:0/3803589753 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f53a806c4e0 0x7f53a806e9a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f53ac005950 tx=0x7f53ac00b410 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:54.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.594+0000 7f53b5ffb700 1 -- 172.21.15.67:0/3803589753 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f53a405af00 con 0x7f53b8101850 2024-04-22T23:19:54.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.788+0000 7f53beef1700 1 -- 172.21.15.67:0/3803589753 --> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f53b8197d00 con 0x7f53a806c4e0 2024-04-22T23:19:54.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:54 smithi067 ceph-mon[26453]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:54.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.789+0000 7f53b5ffb700 1 -- 172.21.15.67:0/3803589753 <== mgr.14215 v2:172.21.15.67:6800/2589747850 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27838 (secure 0 0 0) 0x7f53b8197d00 con 0x7f53a806c4e0 2024-04-22T23:19:54.791 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:19:54.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.792+0000 7f53beef1700 1 -- 172.21.15.67:0/3803589753 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f53a806c4e0 msgr2=0x7f53a806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:54.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.792+0000 7f53beef1700 1 --2- 172.21.15.67:0/3803589753 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f53a806c4e0 0x7f53a806e9a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f53ac005950 tx=0x7f53ac00b410 comp rx=0 tx=0).stop 2024-04-22T23:19:54.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.792+0000 7f53beef1700 1 -- 172.21.15.67:0/3803589753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53b8101850 msgr2=0x7f53b819cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:54.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.792+0000 7f53beef1700 1 --2- 172.21.15.67:0/3803589753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53b8101850 0x7f53b819cc40 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f53a40092e0 tx=0x7f53a400bd70 comp rx=0 tx=0).stop 2024-04-22T23:19:54.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.792+0000 7f53beef1700 1 -- 172.21.15.67:0/3803589753 shutdown_connections 2024-04-22T23:19:54.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.792+0000 7f53beef1700 1 --2- 172.21.15.67:0/3803589753 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f53a806c4e0 0x7f53a806e9a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:54.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.792+0000 7f53beef1700 1 --2- 172.21.15.67:0/3803589753 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f53b8100ea0 0x7f53b819c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:54.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.792+0000 7f53beef1700 1 --2- 172.21.15.67:0/3803589753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53b8101850 0x7f53b819cc40 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:54.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.792+0000 7f53beef1700 1 -- 172.21.15.67:0/3803589753 >> 172.21.15.67:0/3803589753 conn(0x7f53b80fc750 msgr2=0x7f53b80fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:54.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.792+0000 7f53beef1700 1 -- 172.21.15.67:0/3803589753 shutdown_connections 2024-04-22T23:19:54.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:54.792+0000 7f53beef1700 1 -- 172.21.15.67:0/3803589753 wait complete. 2024-04-22T23:19:54.794 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-22T23:19:54.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:54 smithi134 ceph-mon[34609]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:55.485 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2024-04-22T23:19:53.442899+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":165588,"kb_used_data":3876,"kb_used_omap":0,"kb_used_meta":161664,"kb_avail":562305324,"statfs":{"total":575970213888,"available":575800651776,"internally_reserved":0,"allocated":3969024,"data_stored":2609166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":165543936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002158"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":187,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T23:19:00.897897+0000","last_change":"2024-04-22T23:18:35.039326+0000","last_active":"2024-04-22T23:19:00.897897+0000","last_peered":"2024-04-22T23:19:00.897897+0000","last_clean":"2024-04-22T23:19:00.897897+0000","last_became_active":"2024-04-22T23:18:35.039026+0000","last_became_peered":"2024-04-22T23:18:35.039026+0000","last_unstale":"2024-04-22T23:19:00.897897+0000","last_undegraded":"2024-04-22T23:19:00.897897+0000","last_fullsized":"2024-04-22T23:19:00.897897+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T23:17:48.243793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T23:17:48.243793+0000","last_clean_scrub_stamp":"2024-04-22T23:17:48.243793+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T06:31:22.380474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]}]},{"osd":4,"up_from":34,"seq":146028888082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 23:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.42599999999999999}]},{"osd":1,"last update":"Mon Apr 22 23:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.377},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.44500000000000001}]},{"osd":2,"last update":"Mon Apr 22 23:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.47899999999999998}]},{"osd":3,"last update":"Mon Apr 22 23:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.46300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 23:19:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.52500000000000002}]},{"osd":1,"last update":"Mon Apr 22 23:19:13 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.48799999999999999}]},{"osd":2,"last update":"Mon Apr 22 23:19:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.50800000000000001}]},{"osd":4,"last update":"Mon Apr 22 23:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.40699999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 23:19:49 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.34499999999999997,"15min":0.34300000000000003},"min":{"1min":0.222,"5min":0.186,"15min":0.186},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.376,"5min":0.371,"15min":0.37},"min":{"1min":0.25600000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.36899999999999999}]},{"osd":1,"last update":"Mon Apr 22 23:19:49 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38600000000000001,"15min":0.38700000000000001},"min":{"1min":0.26900000000000002,"5min":0.22,"15min":0.22},"max":{"1min":0.66100000000000003,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.37,"5min":0.38400000000000001,"15min":0.38600000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.59199999999999997,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.33200000000000002}]},{"osd":3,"last update":"Mon Apr 22 23:19:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.65600000000000003}]},{"osd":4,"last update":"Mon Apr 22 23:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.58599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 23:19:27 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"min":{"1min":0.245,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.441,"5min":0.502,"15min":0.502},"last":0.318},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.32700000000000001,"15min":0.32400000000000001},"min":{"1min":0.27900000000000003,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.51000000000000001,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.27100000000000002}]},{"osd":2,"last update":"Mon Apr 22 23:18:48 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.252},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.28699999999999998}]},{"osd":3,"last update":"Mon Apr 22 23:19:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.47599999999999998}]},{"osd":4,"last update":"Mon Apr 22 23:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.40999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]}]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 23:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.31900000000000001,"15min":0.317},"min":{"1min":0.223,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.443,"5min":0.501,"15min":0.501},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.34100000000000003,"15min":0.33900000000000002},"min":{"1min":0.26900000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.46400000000000002,"5min":0.499,"15min":0.499},"last":0.36699999999999999}]},{"osd":2,"last update":"Mon Apr 22 23:18:48 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"last":0.40600000000000003}]},{"osd":3,"last update":"Mon Apr 22 23:19:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.67400000000000004}]},{"osd":4,"last update":"Mon Apr 22 23:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.61799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T23:19:55.487 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-22T23:19:55.701 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:19:56.089 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:56 smithi067 ceph-mon[26453]: from='client.14428 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T23:19:56.089 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:56 smithi067 ceph-mon[26453]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:56.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:56 smithi134 ceph-mon[34609]: from='client.14428 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T23:19:56.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:56 smithi134 ceph-mon[34609]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:57.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.331+0000 7f88da29d700 1 -- 172.21.15.67:0/3489712109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88d4100ea0 msgr2=0x7f88d4101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:57.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.331+0000 7f88da29d700 1 --2- 172.21.15.67:0/3489712109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88d4100ea0 0x7f88d4101280 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f88c4009910 tx=0x7f88c4009c20 comp rx=0 tx=0).stop 2024-04-22T23:19:57.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.331+0000 7f88da29d700 1 -- 172.21.15.67:0/3489712109 shutdown_connections 2024-04-22T23:19:57.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.331+0000 7f88da29d700 1 --2- 172.21.15.67:0/3489712109 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88d4101850 0x7f88d41058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:57.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.331+0000 7f88da29d700 1 --2- 172.21.15.67:0/3489712109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88d4100ea0 0x7f88d4101280 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:57.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.331+0000 7f88da29d700 1 -- 172.21.15.67:0/3489712109 >> 172.21.15.67:0/3489712109 conn(0x7f88d40fc750 msgr2=0x7f88d40feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:57.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.331+0000 7f88da29d700 1 -- 172.21.15.67:0/3489712109 shutdown_connections 2024-04-22T23:19:57.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.331+0000 7f88da29d700 1 -- 172.21.15.67:0/3489712109 wait complete. 2024-04-22T23:19:57.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.331+0000 7f88da29d700 1 Processor -- start 2024-04-22T23:19:57.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.331+0000 7f88da29d700 1 -- start start 2024-04-22T23:19:57.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.332+0000 7f88da29d700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88d4100ea0 0x7f88d419c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:57.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.332+0000 7f88da29d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88d4101850 0x7f88d419cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:57.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.332+0000 7f88da29d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d419d2c0 con 0x7f88d4101850 2024-04-22T23:19:57.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.332+0000 7f88da29d700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d4196770 con 0x7f88d4100ea0 2024-04-22T23:19:57.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.332+0000 7f88d37fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88d4101850 0x7f88d419cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:57.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.332+0000 7f88d37fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88d4101850 0x7f88d419cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46170/0 (socket says 172.21.15.67:46170) 2024-04-22T23:19:57.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.332+0000 7f88d37fe700 1 -- 172.21.15.67:0/2528007817 learned_addr learned my addr 172.21.15.67:0/2528007817 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:19:57.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.332+0000 7f88d37fe700 1 -- 172.21.15.67:0/2528007817 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88d4100ea0 msgr2=0x7f88d419c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:57.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.332+0000 7f88d37fe700 1 --2- 172.21.15.67:0/2528007817 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88d4100ea0 0x7f88d419c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:57.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.332+0000 7f88d37fe700 1 -- 172.21.15.67:0/2528007817 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c40095f0 con 0x7f88d4101850 2024-04-22T23:19:57.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.332+0000 7f88d37fe700 1 --2- 172.21.15.67:0/2528007817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88d4101850 0x7f88d419cc30 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f88c800c8f0 tx=0x7f88c800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:57.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.333+0000 7f88d17fa700 1 -- 172.21.15.67:0/2528007817 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88c8007a50 con 0x7f88d4101850 2024-04-22T23:19:57.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.333+0000 7f88d17fa700 1 -- 172.21.15.67:0/2528007817 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f88c8007bb0 con 0x7f88d4101850 2024-04-22T23:19:57.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.333+0000 7f88da29d700 1 -- 172.21.15.67:0/2528007817 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88d4196a50 con 0x7f88d4101850 2024-04-22T23:19:57.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.333+0000 7f88d17fa700 1 -- 172.21.15.67:0/2528007817 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88c8018630 con 0x7f88d4101850 2024-04-22T23:19:57.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.333+0000 7f88da29d700 1 -- 172.21.15.67:0/2528007817 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88d4196fa0 con 0x7f88d4101850 2024-04-22T23:19:57.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.334+0000 7f88da29d700 1 -- 172.21.15.67:0/2528007817 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88d4109240 con 0x7f88d4101850 2024-04-22T23:19:57.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.334+0000 7f88d17fa700 1 -- 172.21.15.67:0/2528007817 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f88c8007d20 con 0x7f88d4101850 2024-04-22T23:19:57.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.334+0000 7f88d17fa700 1 --2- 172.21.15.67:0/2528007817 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f88bc06c4e0 0x7f88bc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:19:57.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.335+0000 7f88d17fa700 1 -- 172.21.15.67:0/2528007817 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f88c808b420 con 0x7f88d4101850 2024-04-22T23:19:57.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.335+0000 7f88d3fff700 1 --2- 172.21.15.67:0/2528007817 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f88bc06c4e0 0x7f88bc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:19:57.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.335+0000 7f88d3fff700 1 --2- 172.21.15.67:0/2528007817 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f88bc06c4e0 0x7f88bc06e9a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f88c4009fd0 tx=0x7f88c400b560 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:19:57.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.339+0000 7f88d17fa700 1 -- 172.21.15.67:0/2528007817 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f88c8059a50 con 0x7f88d4101850 2024-04-22T23:19:57.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.534+0000 7f88da29d700 1 -- 172.21.15.67:0/2528007817 --> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f88d4197270 con 0x7f88bc06c4e0 2024-04-22T23:19:57.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.537+0000 7f88d17fa700 1 -- 172.21.15.67:0/2528007817 <== mgr.14215 v2:172.21.15.67:6800/2589747850 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27709 (secure 0 0 0) 0x7f88d4197270 con 0x7f88bc06c4e0 2024-04-22T23:19:57.538 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:19:57.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.539+0000 7f88da29d700 1 -- 172.21.15.67:0/2528007817 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f88bc06c4e0 msgr2=0x7f88bc06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:57.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.539+0000 7f88da29d700 1 --2- 172.21.15.67:0/2528007817 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f88bc06c4e0 0x7f88bc06e9a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f88c4009fd0 tx=0x7f88c400b560 comp rx=0 tx=0).stop 2024-04-22T23:19:57.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.539+0000 7f88da29d700 1 -- 172.21.15.67:0/2528007817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88d4101850 msgr2=0x7f88d419cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:19:57.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.539+0000 7f88da29d700 1 --2- 172.21.15.67:0/2528007817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88d4101850 0x7f88d419cc30 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f88c800c8f0 tx=0x7f88c800ccb0 comp rx=0 tx=0).stop 2024-04-22T23:19:57.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.539+0000 7f88da29d700 1 -- 172.21.15.67:0/2528007817 shutdown_connections 2024-04-22T23:19:57.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.539+0000 7f88da29d700 1 --2- 172.21.15.67:0/2528007817 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f88bc06c4e0 0x7f88bc06e9a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:57.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.539+0000 7f88da29d700 1 --2- 172.21.15.67:0/2528007817 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88d4100ea0 0x7f88d419c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:57.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.539+0000 7f88da29d700 1 --2- 172.21.15.67:0/2528007817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88d4101850 0x7f88d419cc30 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:19:57.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.539+0000 7f88da29d700 1 -- 172.21.15.67:0/2528007817 >> 172.21.15.67:0/2528007817 conn(0x7f88d40fc750 msgr2=0x7f88d40fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:19:57.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.539+0000 7f88da29d700 1 -- 172.21.15.67:0/2528007817 shutdown_connections 2024-04-22T23:19:57.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:19:57.539+0000 7f88da29d700 1 -- 172.21.15.67:0/2528007817 wait complete. 2024-04-22T23:19:57.541 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-22T23:19:58.106 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2024-04-22T23:19:57.443578+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":165588,"kb_used_data":3876,"kb_used_omap":0,"kb_used_meta":161664,"kb_avail":562305324,"statfs":{"total":575970213888,"available":575800651776,"internally_reserved":0,"allocated":3969024,"data_stored":2609166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":165543936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002250"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":187,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-22T23:19:00.897897+0000","last_change":"2024-04-22T23:18:35.039326+0000","last_active":"2024-04-22T23:19:00.897897+0000","last_peered":"2024-04-22T23:19:00.897897+0000","last_clean":"2024-04-22T23:19:00.897897+0000","last_became_active":"2024-04-22T23:18:35.039026+0000","last_became_peered":"2024-04-22T23:18:35.039026+0000","last_unstale":"2024-04-22T23:19:00.897897+0000","last_undegraded":"2024-04-22T23:19:00.897897+0000","last_fullsized":"2024-04-22T23:19:00.897897+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T23:17:48.243793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T23:17:48.243793+0000","last_clean_scrub_stamp":"2024-04-22T23:17:48.243793+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T06:31:22.380474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724558,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]}]},{"osd":4,"up_from":34,"seq":146028888083,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 23:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.54900000000000004}]},{"osd":1,"last update":"Mon Apr 22 23:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.47199999999999998}]},{"osd":2,"last update":"Mon Apr 22 23:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.503}]},{"osd":3,"last update":"Mon Apr 22 23:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.59199999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]}]},{"osd":3,"up_from":26,"seq":111669149720,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 23:19:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.379},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.433}]},{"osd":1,"last update":"Mon Apr 22 23:19:13 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.44800000000000001}]},{"osd":2,"last update":"Mon Apr 22 23:19:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.499}]},{"osd":4,"last update":"Mon Apr 22 23:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.46700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 23:19:49 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.34499999999999997,"15min":0.34300000000000003},"min":{"1min":0.222,"5min":0.186,"15min":0.186},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.376,"5min":0.371,"15min":0.37},"min":{"1min":0.25600000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.36899999999999999}]},{"osd":1,"last update":"Mon Apr 22 23:19:49 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38600000000000001,"15min":0.38700000000000001},"min":{"1min":0.26900000000000002,"5min":0.22,"15min":0.22},"max":{"1min":0.66100000000000003,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.37,"5min":0.38400000000000001,"15min":0.38600000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.59199999999999997,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.33200000000000002}]},{"osd":3,"last update":"Mon Apr 22 23:19:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.65600000000000003}]},{"osd":4,"last update":"Mon Apr 22 23:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.58599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 23:19:27 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"min":{"1min":0.245,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.441,"5min":0.502,"15min":0.502},"last":0.31},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.32700000000000001,"15min":0.32400000000000001},"min":{"1min":0.27900000000000003,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.51000000000000001,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.25900000000000001}]},{"osd":2,"last update":"Mon Apr 22 23:19:51 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.36799999999999999,"15min":0.374},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.496,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.38100000000000001,"15min":0.38400000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.27100000000000002}]},{"osd":3,"last update":"Mon Apr 22 23:19:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.57199999999999995}]},{"osd":4,"last update":"Mon Apr 22 23:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.52700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]}]},{"osd":1,"up_from":14,"seq":60129542179,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 23:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.31900000000000001,"15min":0.317},"min":{"1min":0.223,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.443,"5min":0.501,"15min":0.501},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.34100000000000003,"15min":0.33900000000000002},"min":{"1min":0.26900000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.46400000000000002,"5min":0.499,"15min":0.499},"last":0.35299999999999998}]},{"osd":2,"last update":"Mon Apr 22 23:19:52 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.36699999999999999,"15min":0.37},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.44400000000000001,"5min":0.45700000000000002,"15min":0.45700000000000002},"last":0.317},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.36799999999999999,"15min":0.37},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.443,"5min":0.47899999999999998,"15min":0.47899999999999998},"last":0.28299999999999997}]},{"osd":3,"last update":"Mon Apr 22 23:19:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.70499999999999996}]},{"osd":4,"last update":"Mon Apr 22 23:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.623},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.53900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T23:19:58.108 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-22T23:19:58.108 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-22T23:19:58.108 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-22T23:19:58.109 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-22T23:19:58.320 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:19:58.675 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:58 smithi067 ceph-mon[26453]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:58.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:58 smithi134 ceph-mon[34609]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:19:59.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:19:59 smithi067 ceph-mon[26453]: from='client.14432 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T23:19:59.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:19:59 smithi134 ceph-mon[34609]: from='client.14432 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T23:20:00.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.004+0000 7f01470ae700 1 -- 172.21.15.67:0/3942997623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0140102700 msgr2=0x7f014010abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:00.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.004+0000 7f01470ae700 1 --2- 172.21.15.67:0/3942997623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0140102700 0x7f014010abf0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f012c009910 tx=0x7f012c009c20 comp rx=0 tx=0).stop 2024-04-22T23:20:00.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.004+0000 7f01470ae700 1 -- 172.21.15.67:0/3942997623 shutdown_connections 2024-04-22T23:20:00.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.004+0000 7f01470ae700 1 --2- 172.21.15.67:0/3942997623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0140102700 0x7f014010abf0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:00.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.004+0000 7f01470ae700 1 --2- 172.21.15.67:0/3942997623 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0140101de0 0x7f01401021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:00.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.004+0000 7f01470ae700 1 -- 172.21.15.67:0/3942997623 >> 172.21.15.67:0/3942997623 conn(0x7f01400fb650 msgr2=0x7f01400fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:00.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.004+0000 7f01470ae700 1 -- 172.21.15.67:0/3942997623 shutdown_connections 2024-04-22T23:20:00.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.004+0000 7f01470ae700 1 -- 172.21.15.67:0/3942997623 wait complete. 2024-04-22T23:20:00.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.005+0000 7f01470ae700 1 Processor -- start 2024-04-22T23:20:00.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.005+0000 7f01470ae700 1 -- start start 2024-04-22T23:20:00.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.005+0000 7f01470ae700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0140101de0 0x7f014019e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:00.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.005+0000 7f01470ae700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0140102700 0x7f014019ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:00.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.005+0000 7f01470ae700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f014019f4c0 con 0x7f0140102700 2024-04-22T23:20:00.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.005+0000 7f01470ae700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0140198970 con 0x7f0140101de0 2024-04-22T23:20:00.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.005+0000 7f013ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0140102700 0x7f014019ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:00.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.005+0000 7f013ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0140102700 0x7f014019ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34654/0 (socket says 172.21.15.67:34654) 2024-04-22T23:20:00.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.005+0000 7f013ffff700 1 -- 172.21.15.67:0/435932335 learned_addr learned my addr 172.21.15.67:0/435932335 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:20:00.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.005+0000 7f0144e4a700 1 --2- 172.21.15.67:0/435932335 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0140101de0 0x7f014019e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:00.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.006+0000 7f013ffff700 1 -- 172.21.15.67:0/435932335 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0140101de0 msgr2=0x7f014019e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:00.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.006+0000 7f013ffff700 1 --2- 172.21.15.67:0/435932335 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0140101de0 0x7f014019e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:00.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.006+0000 7f013ffff700 1 -- 172.21.15.67:0/435932335 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f012c0095f0 con 0x7f0140102700 2024-04-22T23:20:00.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.006+0000 7f0144e4a700 1 --2- 172.21.15.67:0/435932335 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0140101de0 0x7f014019e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:20:00.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.006+0000 7f013ffff700 1 --2- 172.21.15.67:0/435932335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0140102700 0x7f014019ee30 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f012c005f50 tx=0x7f012c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:00.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.006+0000 7f013dffb700 1 -- 172.21.15.67:0/435932335 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f012c01d070 con 0x7f0140102700 2024-04-22T23:20:00.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.006+0000 7f013dffb700 1 -- 172.21.15.67:0/435932335 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f012c00f460 con 0x7f0140102700 2024-04-22T23:20:00.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.006+0000 7f01470ae700 1 -- 172.21.15.67:0/435932335 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0140198bf0 con 0x7f0140102700 2024-04-22T23:20:00.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.006+0000 7f013dffb700 1 -- 172.21.15.67:0/435932335 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f012c022650 con 0x7f0140102700 2024-04-22T23:20:00.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.006+0000 7f01470ae700 1 -- 172.21.15.67:0/435932335 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01401990e0 con 0x7f0140102700 2024-04-22T23:20:00.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.007+0000 7f01470ae700 1 -- 172.21.15.67:0/435932335 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f014010a750 con 0x7f0140102700 2024-04-22T23:20:00.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.008+0000 7f013dffb700 1 -- 172.21.15.67:0/435932335 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f012c00f5d0 con 0x7f0140102700 2024-04-22T23:20:00.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.008+0000 7f013dffb700 1 --2- 172.21.15.67:0/435932335 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f013006c4e0 0x7f013006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:00.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.008+0000 7f013dffb700 1 -- 172.21.15.67:0/435932335 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f012c08c930 con 0x7f0140102700 2024-04-22T23:20:00.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.008+0000 7f0144e4a700 1 --2- 172.21.15.67:0/435932335 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f013006c4e0 0x7f013006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:00.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.009+0000 7f0144e4a700 1 --2- 172.21.15.67:0/435932335 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f013006c4e0 0x7f013006e9a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f01340099d0 tx=0x7f0134008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:00.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.013+0000 7f013dffb700 1 -- 172.21.15.67:0/435932335 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f012c05aee0 con 0x7f0140102700 2024-04-22T23:20:00.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.255+0000 7f01470ae700 1 -- 172.21.15.67:0/435932335 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f014010a130 con 0x7f0140102700 2024-04-22T23:20:00.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.255+0000 7f013dffb700 1 -- 172.21.15.67:0/435932335 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f012c05aa70 con 0x7f0140102700 2024-04-22T23:20:00.256 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:20:00.256 INFO:teuthology.orchestra.run.smithi067.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-22T23:20:00.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.257+0000 7f01470ae700 1 -- 172.21.15.67:0/435932335 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f013006c4e0 msgr2=0x7f013006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:00.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.257+0000 7f01470ae700 1 --2- 172.21.15.67:0/435932335 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f013006c4e0 0x7f013006e9a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f01340099d0 tx=0x7f0134008040 comp rx=0 tx=0).stop 2024-04-22T23:20:00.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.257+0000 7f01470ae700 1 -- 172.21.15.67:0/435932335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0140102700 msgr2=0x7f014019ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:00.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.257+0000 7f01470ae700 1 --2- 172.21.15.67:0/435932335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0140102700 0x7f014019ee30 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f012c005f50 tx=0x7f012c00be20 comp rx=0 tx=0).stop 2024-04-22T23:20:00.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.257+0000 7f01470ae700 1 -- 172.21.15.67:0/435932335 shutdown_connections 2024-04-22T23:20:00.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.257+0000 7f01470ae700 1 --2- 172.21.15.67:0/435932335 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f013006c4e0 0x7f013006e9a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:00.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.257+0000 7f01470ae700 1 --2- 172.21.15.67:0/435932335 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0140101de0 0x7f014019e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:00.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.257+0000 7f01470ae700 1 --2- 172.21.15.67:0/435932335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0140102700 0x7f014019ee30 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:00.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.257+0000 7f01470ae700 1 -- 172.21.15.67:0/435932335 >> 172.21.15.67:0/435932335 conn(0x7f01400fb650 msgr2=0x7f0140100200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:00.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.257+0000 7f01470ae700 1 -- 172.21.15.67:0/435932335 shutdown_connections 2024-04-22T23:20:00.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:00.257+0000 7f01470ae700 1 -- 172.21.15.67:0/435932335 wait complete. 2024-04-22T23:20:00.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:00 smithi067 ceph-mon[26453]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:20:00.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:00 smithi067 ceph-mon[26453]: overall HEALTH_OK 2024-04-22T23:20:00.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:00 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/435932335' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T23:20:00.878 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-22T23:20:00.879 INFO:tasks.cephadm:Setup complete, yielding 2024-04-22T23:20:00.879 INFO:teuthology.run_tasks:Running task print... 2024-04-22T23:20:00.888 INFO:teuthology.task.print:**** done end installing v18.2.1 cephadm ... 2024-04-22T23:20:00.888 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T23:20:00.897 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi067.front.sepia.ceph.com 2024-04-22T23:20:00.897 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-04-22T23:20:00.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:00 smithi134 ceph-mon[34609]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:20:00.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:00 smithi134 ceph-mon[34609]: overall HEALTH_OK 2024-04-22T23:20:00.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:00 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/435932335' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T23:20:01.110 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:20:02.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.725+0000 7f82752c3700 1 -- 172.21.15.67:0/247070729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8270104080 msgr2=0x7f8270106470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:02.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.725+0000 7f82752c3700 1 --2- 172.21.15.67:0/247070729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8270104080 0x7f8270106470 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f8264009910 tx=0x7f8264009c20 comp rx=0 tx=0).stop 2024-04-22T23:20:02.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.725+0000 7f82752c3700 1 -- 172.21.15.67:0/247070729 shutdown_connections 2024-04-22T23:20:02.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.725+0000 7f82752c3700 1 --2- 172.21.15.67:0/247070729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8270104080 0x7f8270106470 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:02.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.725+0000 7f82752c3700 1 --2- 172.21.15.67:0/247070729 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8270101750 0x7f8270103b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:02.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.725+0000 7f82752c3700 1 -- 172.21.15.67:0/247070729 >> 172.21.15.67:0/247070729 conn(0x7f82700fb160 msgr2=0x7f82700fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:02.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.726+0000 7f82752c3700 1 -- 172.21.15.67:0/247070729 shutdown_connections 2024-04-22T23:20:02.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.726+0000 7f82752c3700 1 -- 172.21.15.67:0/247070729 wait complete. 2024-04-22T23:20:02.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.726+0000 7f82752c3700 1 Processor -- start 2024-04-22T23:20:02.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.726+0000 7f82752c3700 1 -- start start 2024-04-22T23:20:02.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.726+0000 7f82752c3700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8270101750 0x7f82701967a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:02.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.726+0000 7f82752c3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8270104080 0x7f8270196ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:02.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.727+0000 7f82752c3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8270197270 con 0x7f8270104080 2024-04-22T23:20:02.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.727+0000 7f82752c3700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82701973b0 con 0x7f8270101750 2024-04-22T23:20:02.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.727+0000 7f826e7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8270104080 0x7f8270196ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:02.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.727+0000 7f826e7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8270104080 0x7f8270196ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34666/0 (socket says 172.21.15.67:34666) 2024-04-22T23:20:02.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.727+0000 7f826e7fc700 1 -- 172.21.15.67:0/328977640 learned_addr learned my addr 172.21.15.67:0/328977640 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:20:02.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.727+0000 7f826effd700 1 --2- 172.21.15.67:0/328977640 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8270101750 0x7f82701967a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:02.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.727+0000 7f826e7fc700 1 -- 172.21.15.67:0/328977640 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8270101750 msgr2=0x7f82701967a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:02.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.727+0000 7f826e7fc700 1 --2- 172.21.15.67:0/328977640 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8270101750 0x7f82701967a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:02.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.727+0000 7f826e7fc700 1 -- 172.21.15.67:0/328977640 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82640095f0 con 0x7f8270104080 2024-04-22T23:20:02.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.727+0000 7f826effd700 1 --2- 172.21.15.67:0/328977640 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8270101750 0x7f82701967a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:20:02.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.727+0000 7f826e7fc700 1 --2- 172.21.15.67:0/328977640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8270104080 0x7f8270196ce0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f826400bda0 tx=0x7f826400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:02.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.727+0000 7f8257fff700 1 -- 172.21.15.67:0/328977640 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f826401d070 con 0x7f8270104080 2024-04-22T23:20:02.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.727+0000 7f8257fff700 1 -- 172.21.15.67:0/328977640 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8264004c50 con 0x7f8270104080 2024-04-22T23:20:02.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.727+0000 7f82752c3700 1 -- 172.21.15.67:0/328977640 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f827019be10 con 0x7f8270104080 2024-04-22T23:20:02.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.727+0000 7f8257fff700 1 -- 172.21.15.67:0/328977640 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f826400f690 con 0x7f8270104080 2024-04-22T23:20:02.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.727+0000 7f82752c3700 1 -- 172.21.15.67:0/328977640 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f827019c2d0 con 0x7f8270104080 2024-04-22T23:20:02.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.728+0000 7f82752c3700 1 -- 172.21.15.67:0/328977640 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8270190630 con 0x7f8270104080 2024-04-22T23:20:02.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.729+0000 7f8257fff700 1 -- 172.21.15.67:0/328977640 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f82640229b0 con 0x7f8270104080 2024-04-22T23:20:02.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.729+0000 7f8257fff700 1 --2- 172.21.15.67:0/328977640 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f825806c4e0 0x7f825806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:02.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.730+0000 7f8257fff700 1 -- 172.21.15.67:0/328977640 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f826408ccd0 con 0x7f8270104080 2024-04-22T23:20:02.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.730+0000 7f826effd700 1 --2- 172.21.15.67:0/328977640 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f825806c4e0 0x7f825806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:02.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.730+0000 7f826effd700 1 --2- 172.21.15.67:0/328977640 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f825806c4e0 0x7f825806e9a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8260005fd0 tx=0x7f8260005e30 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:02.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.734+0000 7f8257fff700 1 -- 172.21.15.67:0/328977640 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f826405b4c0 con 0x7f8270104080 2024-04-22T23:20:02.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:02 smithi067 ceph-mon[26453]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:20:02.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.934+0000 7f82752c3700 1 -- 172.21.15.67:0/328977640 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7f8270068af0 con 0x7f8270104080 2024-04-22T23:20:02.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.938+0000 7f8257fff700 1 -- 172.21.15.67:0/328977640 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7f82640270d0 con 0x7f8270104080 2024-04-22T23:20:02.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.940+0000 7f82752c3700 1 -- 172.21.15.67:0/328977640 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f825806c4e0 msgr2=0x7f825806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:02.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.940+0000 7f82752c3700 1 --2- 172.21.15.67:0/328977640 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f825806c4e0 0x7f825806e9a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8260005fd0 tx=0x7f8260005e30 comp rx=0 tx=0).stop 2024-04-22T23:20:02.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.940+0000 7f82752c3700 1 -- 172.21.15.67:0/328977640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8270104080 msgr2=0x7f8270196ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:02.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.940+0000 7f82752c3700 1 --2- 172.21.15.67:0/328977640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8270104080 0x7f8270196ce0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f826400bda0 tx=0x7f826400be80 comp rx=0 tx=0).stop 2024-04-22T23:20:02.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.941+0000 7f82752c3700 1 -- 172.21.15.67:0/328977640 shutdown_connections 2024-04-22T23:20:02.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.941+0000 7f82752c3700 1 --2- 172.21.15.67:0/328977640 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f825806c4e0 0x7f825806e9a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:02.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.941+0000 7f82752c3700 1 --2- 172.21.15.67:0/328977640 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8270101750 0x7f82701967a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:02.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.941+0000 7f82752c3700 1 --2- 172.21.15.67:0/328977640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8270104080 0x7f8270196ce0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:02.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.941+0000 7f82752c3700 1 -- 172.21.15.67:0/328977640 >> 172.21.15.67:0/328977640 conn(0x7f82700fb160 msgr2=0x7f82700fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:02.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.941+0000 7f82752c3700 1 -- 172.21.15.67:0/328977640 shutdown_connections 2024-04-22T23:20:02.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:02.941+0000 7f82752c3700 1 -- 172.21.15.67:0/328977640 wait complete. 2024-04-22T23:20:02.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:02 smithi134 ceph-mon[34609]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:20:03.732 INFO:teuthology.run_tasks:Running task print... 2024-04-22T23:20:03.743 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-04-22T23:20:03.743 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T23:20:03.760 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi067.front.sepia.ceph.com 2024-04-22T23:20:03.760 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-22T23:20:03.981 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:20:04.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:03 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/328977640' entity='client.admin' 2024-04-22T23:20:04.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:03 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:20:04.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:03 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:20:04.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:03 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:20:04.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:03 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:04.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:03 smithi067 ceph-mon[26453]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:20:04.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:03 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/328977640' entity='client.admin' 2024-04-22T23:20:04.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:03 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:20:04.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:03 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:20:04.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:03 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:20:04.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:03 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:04.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:03 smithi134 ceph-mon[34609]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:20:05.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.557+0000 7f675a358700 1 -- 172.21.15.67:0/591128109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6754101eb0 msgr2=0x7f6754102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:05.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.557+0000 7f675a358700 1 --2- 172.21.15.67:0/591128109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6754101eb0 0x7f6754102310 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f6748009910 tx=0x7f6748009c20 comp rx=0 tx=0).stop 2024-04-22T23:20:05.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.557+0000 7f675a358700 1 -- 172.21.15.67:0/591128109 shutdown_connections 2024-04-22T23:20:05.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.557+0000 7f675a358700 1 --2- 172.21.15.67:0/591128109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6754101eb0 0x7f6754102310 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:05.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.557+0000 7f675a358700 1 --2- 172.21.15.67:0/591128109 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6754100cb0 0x7f67541010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:05.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.557+0000 7f675a358700 1 -- 172.21.15.67:0/591128109 >> 172.21.15.67:0/591128109 conn(0x7f67540fc230 msgr2=0x7f67540fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:05.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.558+0000 7f675a358700 1 -- 172.21.15.67:0/591128109 shutdown_connections 2024-04-22T23:20:05.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.558+0000 7f675a358700 1 -- 172.21.15.67:0/591128109 wait complete. 2024-04-22T23:20:05.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.558+0000 7f675a358700 1 Processor -- start 2024-04-22T23:20:05.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.558+0000 7f675a358700 1 -- start start 2024-04-22T23:20:05.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.559+0000 7f675a358700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6754100cb0 0x7f67541964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:05.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.559+0000 7f675a358700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6754101eb0 0x7f67541969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:05.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.559+0000 7f675a358700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6754196f80 con 0x7f6754100cb0 2024-04-22T23:20:05.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.559+0000 7f675a358700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67541970c0 con 0x7f6754101eb0 2024-04-22T23:20:05.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.559+0000 7f6753fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6754100cb0 0x7f67541964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:05.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.559+0000 7f6753fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6754100cb0 0x7f67541964b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34686/0 (socket says 172.21.15.67:34686) 2024-04-22T23:20:05.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.559+0000 7f6753fff700 1 -- 172.21.15.67:0/2083592723 learned_addr learned my addr 172.21.15.67:0/2083592723 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:20:05.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.559+0000 7f67537fe700 1 --2- 172.21.15.67:0/2083592723 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6754101eb0 0x7f67541969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:05.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.559+0000 7f6753fff700 1 -- 172.21.15.67:0/2083592723 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6754101eb0 msgr2=0x7f67541969f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:05.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.559+0000 7f6753fff700 1 --2- 172.21.15.67:0/2083592723 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6754101eb0 0x7f67541969f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:05.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.559+0000 7f6753fff700 1 -- 172.21.15.67:0/2083592723 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67480095f0 con 0x7f6754100cb0 2024-04-22T23:20:05.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.559+0000 7f6753fff700 1 --2- 172.21.15.67:0/2083592723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6754100cb0 0x7f67541964b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f674400c910 tx=0x7f674400ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:05.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.559+0000 7f67517fa700 1 -- 172.21.15.67:0/2083592723 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67440041f0 con 0x7f6754100cb0 2024-04-22T23:20:05.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.559+0000 7f67517fa700 1 -- 172.21.15.67:0/2083592723 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6744007db0 con 0x7f6754100cb0 2024-04-22T23:20:05.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.559+0000 7f675a358700 1 -- 172.21.15.67:0/2083592723 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f675419bb80 con 0x7f6754100cb0 2024-04-22T23:20:05.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.559+0000 7f67517fa700 1 -- 172.21.15.67:0/2083592723 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6744003d60 con 0x7f6754100cb0 2024-04-22T23:20:05.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.559+0000 7f675a358700 1 -- 172.21.15.67:0/2083592723 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f675419c0a0 con 0x7f6754100cb0 2024-04-22T23:20:05.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.561+0000 7f675a358700 1 -- 172.21.15.67:0/2083592723 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6754109810 con 0x7f6754100cb0 2024-04-22T23:20:05.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.561+0000 7f67517fa700 1 -- 172.21.15.67:0/2083592723 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f6744007900 con 0x7f6754100cb0 2024-04-22T23:20:05.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.561+0000 7f67517fa700 1 --2- 172.21.15.67:0/2083592723 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f673c06c4e0 0x7f673c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:05.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.562+0000 7f67517fa700 1 -- 172.21.15.67:0/2083592723 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f674408b160 con 0x7f6754100cb0 2024-04-22T23:20:05.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.562+0000 7f67537fe700 1 --2- 172.21.15.67:0/2083592723 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f673c06c4e0 0x7f673c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:05.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.562+0000 7f67537fe700 1 --2- 172.21.15.67:0/2083592723 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f673c06c4e0 0x7f673c06e9a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6748006010 tx=0x7f674800b580 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:05.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.566+0000 7f67517fa700 1 -- 172.21.15.67:0/2083592723 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6744059790 con 0x7f6754100cb0 2024-04-22T23:20:05.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.782+0000 7f675a358700 1 -- 172.21.15.67:0/2083592723 --> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6754106800 con 0x7f673c06c4e0 2024-04-22T23:20:05.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.783+0000 7f67517fa700 1 -- 172.21.15.67:0/2083592723 <== mgr.14215 v2:172.21.15.67:6800/2589747850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f6754106800 con 0x7f673c06c4e0 2024-04-22T23:20:05.784 INFO:teuthology.orchestra.run.smithi067.stdout:Backend: cephadm 2024-04-22T23:20:05.784 INFO:teuthology.orchestra.run.smithi067.stdout:Available: Yes 2024-04-22T23:20:05.784 INFO:teuthology.orchestra.run.smithi067.stdout:Paused: No 2024-04-22T23:20:05.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.786+0000 7f675a358700 1 -- 172.21.15.67:0/2083592723 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f673c06c4e0 msgr2=0x7f673c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:05.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.786+0000 7f675a358700 1 --2- 172.21.15.67:0/2083592723 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f673c06c4e0 0x7f673c06e9a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6748006010 tx=0x7f674800b580 comp rx=0 tx=0).stop 2024-04-22T23:20:05.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.786+0000 7f675a358700 1 -- 172.21.15.67:0/2083592723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6754100cb0 msgr2=0x7f67541964b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:05.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.786+0000 7f675a358700 1 --2- 172.21.15.67:0/2083592723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6754100cb0 0x7f67541964b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f674400c910 tx=0x7f674400ccd0 comp rx=0 tx=0).stop 2024-04-22T23:20:05.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.786+0000 7f675a358700 1 -- 172.21.15.67:0/2083592723 shutdown_connections 2024-04-22T23:20:05.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.786+0000 7f675a358700 1 --2- 172.21.15.67:0/2083592723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6754100cb0 0x7f67541964b0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:05.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.786+0000 7f675a358700 1 --2- 172.21.15.67:0/2083592723 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f673c06c4e0 0x7f673c06e9a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:05.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.786+0000 7f675a358700 1 --2- 172.21.15.67:0/2083592723 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6754101eb0 0x7f67541969f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:05.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.786+0000 7f675a358700 1 -- 172.21.15.67:0/2083592723 >> 172.21.15.67:0/2083592723 conn(0x7f67540fc230 msgr2=0x7f67541050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:05.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.786+0000 7f675a358700 1 -- 172.21.15.67:0/2083592723 shutdown_connections 2024-04-22T23:20:05.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:05.786+0000 7f675a358700 1 -- 172.21.15.67:0/2083592723 wait complete. 2024-04-22T23:20:06.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:06 smithi067 ceph-mon[26453]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:20:06.306 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-22T23:20:06.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:06 smithi134 ceph-mon[34609]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:20:06.518 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:20:07.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:07 smithi067 ceph-mon[26453]: from='client.14444 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:20:07.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:07 smithi134 ceph-mon[34609]: from='client.14444 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:20:08.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.136+0000 7f742a909700 1 -- 172.21.15.67:0/1849445102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74241017b0 msgr2=0x7f7424103ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:08.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.136+0000 7f742a909700 1 --2- 172.21.15.67:0/1849445102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74241017b0 0x7f7424103ba0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7414009910 tx=0x7f7414009c20 comp rx=0 tx=0).stop 2024-04-22T23:20:08.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.136+0000 7f742a909700 1 -- 172.21.15.67:0/1849445102 shutdown_connections 2024-04-22T23:20:08.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.136+0000 7f742a909700 1 --2- 172.21.15.67:0/1849445102 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74241040e0 0x7f74241064d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:08.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.136+0000 7f742a909700 1 --2- 172.21.15.67:0/1849445102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74241017b0 0x7f7424103ba0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:08.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.136+0000 7f742a909700 1 -- 172.21.15.67:0/1849445102 >> 172.21.15.67:0/1849445102 conn(0x7f74240fb160 msgr2=0x7f74240fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:08.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.136+0000 7f742a909700 1 -- 172.21.15.67:0/1849445102 shutdown_connections 2024-04-22T23:20:08.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.136+0000 7f742a909700 1 -- 172.21.15.67:0/1849445102 wait complete. 2024-04-22T23:20:08.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.136+0000 7f742a909700 1 Processor -- start 2024-04-22T23:20:08.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.136+0000 7f742a909700 1 -- start start 2024-04-22T23:20:08.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.137+0000 7f742a909700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74241017b0 0x7f74241967a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:08.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.137+0000 7f742a909700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74241040e0 0x7f7424196ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:08.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.137+0000 7f742a909700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7424197300 con 0x7f74241040e0 2024-04-22T23:20:08.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.137+0000 7f742a909700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7424197440 con 0x7f74241017b0 2024-04-22T23:20:08.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.137+0000 7f74237fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74241040e0 0x7f7424196ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:08.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.137+0000 7f74237fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74241040e0 0x7f7424196ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34700/0 (socket says 172.21.15.67:34700) 2024-04-22T23:20:08.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.137+0000 7f74237fe700 1 -- 172.21.15.67:0/3484684939 learned_addr learned my addr 172.21.15.67:0/3484684939 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:20:08.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.137+0000 7f7423fff700 1 --2- 172.21.15.67:0/3484684939 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74241017b0 0x7f74241967a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:08.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.137+0000 7f74237fe700 1 -- 172.21.15.67:0/3484684939 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74241017b0 msgr2=0x7f74241967a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:08.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.137+0000 7f74237fe700 1 --2- 172.21.15.67:0/3484684939 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74241017b0 0x7f74241967a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:08.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.137+0000 7f74237fe700 1 -- 172.21.15.67:0/3484684939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74140095f0 con 0x7f74241040e0 2024-04-22T23:20:08.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.137+0000 7f74237fe700 1 --2- 172.21.15.67:0/3484684939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74241040e0 0x7f7424196ce0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f741800c8a0 tx=0x7f741800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:08.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.137+0000 7f74217fa700 1 -- 172.21.15.67:0/3484684939 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74180078e0 con 0x7f74241040e0 2024-04-22T23:20:08.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.137+0000 7f74217fa700 1 -- 172.21.15.67:0/3484684939 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f741800f450 con 0x7f74241040e0 2024-04-22T23:20:08.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.138+0000 7f742a909700 1 -- 172.21.15.67:0/3484684939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f742419bef0 con 0x7f74241040e0 2024-04-22T23:20:08.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.138+0000 7f74217fa700 1 -- 172.21.15.67:0/3484684939 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f741800e5e0 con 0x7f74241040e0 2024-04-22T23:20:08.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.138+0000 7f742a909700 1 -- 172.21.15.67:0/3484684939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f742419c410 con 0x7f74241040e0 2024-04-22T23:20:08.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.138+0000 7f742a909700 1 -- 172.21.15.67:0/3484684939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7424190630 con 0x7f74241040e0 2024-04-22T23:20:08.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.139+0000 7f74217fa700 1 -- 172.21.15.67:0/3484684939 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f741800f5c0 con 0x7f74241040e0 2024-04-22T23:20:08.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.139+0000 7f74217fa700 1 --2- 172.21.15.67:0/3484684939 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f740c06c4e0 0x7f740c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:08.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.139+0000 7f74217fa700 1 -- 172.21.15.67:0/3484684939 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f741808b470 con 0x7f74241040e0 2024-04-22T23:20:08.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.139+0000 7f7423fff700 1 --2- 172.21.15.67:0/3484684939 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f740c06c4e0 0x7f740c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:08.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.140+0000 7f7423fff700 1 --2- 172.21.15.67:0/3484684939 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f740c06c4e0 0x7f740c06e9a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7414006010 tx=0x7f741400b540 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:08.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.144+0000 7f74217fa700 1 -- 172.21.15.67:0/3484684939 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f7418059a20 con 0x7f74241040e0 2024-04-22T23:20:08.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:08 smithi067 ceph-mon[26453]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:20:08.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:08 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:20:08.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.350+0000 7f742a909700 1 -- 172.21.15.67:0/3484684939 --> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f742402d050 con 0x7f740c06c4e0 2024-04-22T23:20:08.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.359+0000 7f74217fa700 1 -- 172.21.15.67:0/3484684939 <== mgr.14215 v2:172.21.15.67:6800/2589747850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2840 (secure 0 0 0) 0x7f742402d050 con 0x7f740c06c4e0 2024-04-22T23:20:08.361 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:20:08.361 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (3m) 2m ago 5m 25.7M - 0.25.0 c8568f914cd2 8c7b7ec00657 2024-04-22T23:20:08.361 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (5m) 2m ago 5m 7017k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:20:08.361 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (4m) 69s ago 4m 6983k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:20:08.361 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (5m) 2m ago 5m 6660k - 18.2.1 5be31c24972a 14085634c5d3 2024-04-22T23:20:08.361 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (4m) 69s ago 4m 6665k - 18.2.1 5be31c24972a fa995d458353 2024-04-22T23:20:08.361 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (3m) 2m ago 4m 78.2M - 9.4.7 954c08fa6188 0d8950c9af0c 2024-04-22T23:20:08.361 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:9283,8765,8443 running (6m) 2m ago 6m 481M - 18.2.1 5be31c24972a 1c64cb9e36bb 2024-04-22T23:20:08.361 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (4m) 69s ago 4m 443M - 18.2.1 5be31c24972a 0a0b76e72171 2024-04-22T23:20:08.361 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (6m) 2m ago 6m 44.0M 2048M 18.2.1 5be31c24972a 6789283703e0 2024-04-22T23:20:08.361 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (4m) 69s ago 4m 40.5M 2048M 18.2.1 5be31c24972a 2fb23795f7e3 2024-04-22T23:20:08.361 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (5m) 2m ago 5m 15.5M - 1.5.0 0da6a335fe13 4aa4da7089d0 2024-04-22T23:20:08.362 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (4m) 69s ago 4m 16.0M - 1.5.0 0da6a335fe13 26e43c2e756c 2024-04-22T23:20:08.362 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (3m) 2m ago 3m 41.0M 3946M 18.2.1 5be31c24972a a033247ee8c2 2024-04-22T23:20:08.362 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (2m) 2m ago 2m 40.1M 3946M 18.2.1 5be31c24972a 915b07a8edc4 2024-04-22T23:20:08.362 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (2m) 2m ago 2m 40.1M 3946M 18.2.1 5be31c24972a e257ecb4efa1 2024-04-22T23:20:08.362 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (2m) 69s ago 2m 40.3M 4970M 18.2.1 5be31c24972a 9c2c91a967ba 2024-04-22T23:20:08.362 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (100s) 69s ago 99s 39.2M 4970M 18.2.1 5be31c24972a c7586562ffa6 2024-04-22T23:20:08.362 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (74s) 69s ago 74s 11.5M 4970M 18.2.1 5be31c24972a 9fd97f1c65cb 2024-04-22T23:20:08.362 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (3m) 2m ago 4m 36.7M - 2.43.0 a07b618ecd1d d87cb15b5e55 2024-04-22T23:20:08.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.361+0000 7f742a909700 1 -- 172.21.15.67:0/3484684939 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f740c06c4e0 msgr2=0x7f740c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:08.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.361+0000 7f742a909700 1 --2- 172.21.15.67:0/3484684939 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f740c06c4e0 0x7f740c06e9a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7414006010 tx=0x7f741400b540 comp rx=0 tx=0).stop 2024-04-22T23:20:08.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.361+0000 7f742a909700 1 -- 172.21.15.67:0/3484684939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74241040e0 msgr2=0x7f7424196ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:08.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.361+0000 7f742a909700 1 --2- 172.21.15.67:0/3484684939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74241040e0 0x7f7424196ce0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f741800c8a0 tx=0x7f741800cbb0 comp rx=0 tx=0).stop 2024-04-22T23:20:08.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.362+0000 7f742a909700 1 -- 172.21.15.67:0/3484684939 shutdown_connections 2024-04-22T23:20:08.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.362+0000 7f742a909700 1 --2- 172.21.15.67:0/3484684939 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f740c06c4e0 0x7f740c06e9a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:08.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.362+0000 7f742a909700 1 --2- 172.21.15.67:0/3484684939 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74241017b0 0x7f74241967a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:08.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.362+0000 7f742a909700 1 --2- 172.21.15.67:0/3484684939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74241040e0 0x7f7424196ce0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:08.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.362+0000 7f742a909700 1 -- 172.21.15.67:0/3484684939 >> 172.21.15.67:0/3484684939 conn(0x7f74240fb160 msgr2=0x7f74240fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:08.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.362+0000 7f742a909700 1 -- 172.21.15.67:0/3484684939 shutdown_connections 2024-04-22T23:20:08.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:08.362+0000 7f742a909700 1 -- 172.21.15.67:0/3484684939 wait complete. 2024-04-22T23:20:08.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:08 smithi134 ceph-mon[34609]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:20:08.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:08 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:20:09.011 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-22T23:20:09.222 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:20:09.260 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:09 smithi067 ceph-mon[26453]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:20:09.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:09 smithi134 ceph-mon[34609]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:20:10.360 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:10 smithi067 ceph-mon[26453]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:20:10.396 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:10 smithi134 ceph-mon[34609]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:20:10.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.710+0000 7ff5af6af700 1 -- 172.21.15.67:0/1556910137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5a8100520 msgr2=0x7ff5a8100980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:10.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.710+0000 7ff5af6af700 1 --2- 172.21.15.67:0/1556910137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5a8100520 0x7ff5a8100980 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff598009910 tx=0x7ff598009c20 comp rx=0 tx=0).stop 2024-04-22T23:20:10.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.710+0000 7ff5af6af700 1 -- 172.21.15.67:0/1556910137 shutdown_connections 2024-04-22T23:20:10.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.710+0000 7ff5af6af700 1 --2- 172.21.15.67:0/1556910137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5a8100520 0x7ff5a8100980 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:10.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.710+0000 7ff5af6af700 1 --2- 172.21.15.67:0/1556910137 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff5a80ff320 0x7ff5a80ff740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:10.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.710+0000 7ff5af6af700 1 -- 172.21.15.67:0/1556910137 >> 172.21.15.67:0/1556910137 conn(0x7ff5a80fa8c0 msgr2=0x7ff5a80fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:10.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.710+0000 7ff5af6af700 1 -- 172.21.15.67:0/1556910137 shutdown_connections 2024-04-22T23:20:10.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.710+0000 7ff5af6af700 1 -- 172.21.15.67:0/1556910137 wait complete. 2024-04-22T23:20:10.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.710+0000 7ff5af6af700 1 Processor -- start 2024-04-22T23:20:10.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.710+0000 7ff5af6af700 1 -- start start 2024-04-22T23:20:10.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.710+0000 7ff5af6af700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff5a80ff320 0x7ff5a81022a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:10.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.710+0000 7ff5af6af700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5a8100520 0x7ff5a819d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:10.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.710+0000 7ff5af6af700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a8197060 con 0x7ff5a8100520 2024-04-22T23:20:10.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.710+0000 7ff5af6af700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a819db30 con 0x7ff5a80ff320 2024-04-22T23:20:10.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.711+0000 7ff5acc4a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5a8100520 0x7ff5a819d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:10.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.711+0000 7ff5acc4a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5a8100520 0x7ff5a819d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54558/0 (socket says 172.21.15.67:54558) 2024-04-22T23:20:10.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.711+0000 7ff5acc4a700 1 -- 172.21.15.67:0/560105695 learned_addr learned my addr 172.21.15.67:0/560105695 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:20:10.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.711+0000 7ff5ad44b700 1 --2- 172.21.15.67:0/560105695 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff5a80ff320 0x7ff5a81022a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:10.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.711+0000 7ff5acc4a700 1 -- 172.21.15.67:0/560105695 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff5a80ff320 msgr2=0x7ff5a81022a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:10.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.711+0000 7ff5acc4a700 1 --2- 172.21.15.67:0/560105695 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff5a80ff320 0x7ff5a81022a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:10.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.711+0000 7ff5acc4a700 1 -- 172.21.15.67:0/560105695 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5980095f0 con 0x7ff5a8100520 2024-04-22T23:20:10.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.711+0000 7ff5acc4a700 1 --2- 172.21.15.67:0/560105695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5a8100520 0x7ff5a819d5f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff5980092e0 tx=0x7ff59800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:10.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.711+0000 7ff59e7fc700 1 -- 172.21.15.67:0/560105695 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff59801d070 con 0x7ff5a8100520 2024-04-22T23:20:10.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.711+0000 7ff59e7fc700 1 -- 172.21.15.67:0/560105695 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff59800f460 con 0x7ff5a8100520 2024-04-22T23:20:10.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.711+0000 7ff5af6af700 1 -- 172.21.15.67:0/560105695 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5a819dd80 con 0x7ff5a8100520 2024-04-22T23:20:10.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.711+0000 7ff59e7fc700 1 -- 172.21.15.67:0/560105695 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff598017630 con 0x7ff5a8100520 2024-04-22T23:20:10.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.711+0000 7ff5af6af700 1 -- 172.21.15.67:0/560105695 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5a819e270 con 0x7ff5a8100520 2024-04-22T23:20:10.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.713+0000 7ff5af6af700 1 -- 172.21.15.67:0/560105695 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5a8107e80 con 0x7ff5a8100520 2024-04-22T23:20:10.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.713+0000 7ff59e7fc700 1 -- 172.21.15.67:0/560105695 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7ff59800f5d0 con 0x7ff5a8100520 2024-04-22T23:20:10.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.713+0000 7ff59e7fc700 1 --2- 172.21.15.67:0/560105695 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ff59406c4e0 0x7ff59406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:10.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.714+0000 7ff59e7fc700 1 -- 172.21.15.67:0/560105695 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ff59808ca50 con 0x7ff5a8100520 2024-04-22T23:20:10.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.714+0000 7ff5ad44b700 1 --2- 172.21.15.67:0/560105695 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ff59406c4e0 0x7ff59406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:10.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.714+0000 7ff5ad44b700 1 --2- 172.21.15.67:0/560105695 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ff59406c4e0 0x7ff59406e9a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff5a8100380 tx=0x7ff5a400b410 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:10.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.718+0000 7ff59e7fc700 1 -- 172.21.15.67:0/560105695 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff59805b000 con 0x7ff5a8100520 2024-04-22T23:20:10.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.925+0000 7ff5af6af700 1 -- 172.21.15.67:0/560105695 --> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7ff5a8104860 con 0x7ff59406c4e0 2024-04-22T23:20:10.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.929+0000 7ff59e7fc700 1 -- 172.21.15.67:0/560105695 <== mgr.14215 v2:172.21.15.67:6800/2589747850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1300 (secure 0 0 0) 0x7ff5a8104860 con 0x7ff59406c4e0 2024-04-22T23:20:10.931 INFO:teuthology.orchestra.run.smithi067.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-22T23:20:10.931 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager ?:9093,9094 1/1 2m ago 5m count:1 2024-04-22T23:20:10.931 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter 2/2 2m ago 5m * 2024-04-22T23:20:10.931 INFO:teuthology.orchestra.run.smithi067.stdout:crash 2/2 2m ago 6m * 2024-04-22T23:20:10.931 INFO:teuthology.orchestra.run.smithi067.stdout:grafana ?:3000 1/1 2m ago 5m count:1 2024-04-22T23:20:10.931 INFO:teuthology.orchestra.run.smithi067.stdout:mgr 2/2 2m ago 6m count:2 2024-04-22T23:20:10.931 INFO:teuthology.orchestra.run.smithi067.stdout:mon 2/2 2m ago 5m smithi067:172.21.15.67=smithi067;smithi134:172.21.15.134=smithi134;count:2 2024-04-22T23:20:10.931 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter ?:9100 2/2 2m ago 5m * 2024-04-22T23:20:10.931 INFO:teuthology.orchestra.run.smithi067.stdout:osd 6 2m ago - 2024-04-22T23:20:10.931 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus ?:9095 1/1 2m ago 5m count:1 2024-04-22T23:20:10.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.932+0000 7ff5af6af700 1 -- 172.21.15.67:0/560105695 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ff59406c4e0 msgr2=0x7ff59406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:10.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.932+0000 7ff5af6af700 1 --2- 172.21.15.67:0/560105695 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ff59406c4e0 0x7ff59406e9a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff5a8100380 tx=0x7ff5a400b410 comp rx=0 tx=0).stop 2024-04-22T23:20:10.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.932+0000 7ff5af6af700 1 -- 172.21.15.67:0/560105695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5a8100520 msgr2=0x7ff5a819d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:10.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.932+0000 7ff5af6af700 1 --2- 172.21.15.67:0/560105695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5a8100520 0x7ff5a819d5f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff5980092e0 tx=0x7ff59800bd70 comp rx=0 tx=0).stop 2024-04-22T23:20:10.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.932+0000 7ff5af6af700 1 -- 172.21.15.67:0/560105695 shutdown_connections 2024-04-22T23:20:10.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.932+0000 7ff5af6af700 1 --2- 172.21.15.67:0/560105695 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ff59406c4e0 0x7ff59406e9a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:10.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.932+0000 7ff5af6af700 1 --2- 172.21.15.67:0/560105695 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff5a80ff320 0x7ff5a81022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:10.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.932+0000 7ff5af6af700 1 --2- 172.21.15.67:0/560105695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5a8100520 0x7ff5a819d5f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:10.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.932+0000 7ff5af6af700 1 -- 172.21.15.67:0/560105695 >> 172.21.15.67:0/560105695 conn(0x7ff5a80fa8c0 msgr2=0x7ff5a80fca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:10.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.932+0000 7ff5af6af700 1 -- 172.21.15.67:0/560105695 shutdown_connections 2024-04-22T23:20:10.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:10.932+0000 7ff5af6af700 1 -- 172.21.15.67:0/560105695 wait complete. 2024-04-22T23:20:11.533 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-22T23:20:11.744 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:20:12.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:12 smithi067 ceph-mon[26453]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:20:12.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:12 smithi067 ceph-mon[26453]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:20:12.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:12 smithi134 ceph-mon[34609]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:20:12.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:12 smithi134 ceph-mon[34609]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:20:13.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.407+0000 7f8e7724c700 1 -- 172.21.15.67:0/789537540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e70100530 msgr2=0x7f8e70100990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:13.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.407+0000 7f8e7724c700 1 --2- 172.21.15.67:0/789537540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e70100530 0x7f8e70100990 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f8e5800aa20 tx=0x7f8e5800ad30 comp rx=0 tx=0).stop 2024-04-22T23:20:13.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.407+0000 7f8e7724c700 1 -- 172.21.15.67:0/789537540 shutdown_connections 2024-04-22T23:20:13.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.407+0000 7f8e7724c700 1 --2- 172.21.15.67:0/789537540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e70100530 0x7f8e70100990 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:13.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.407+0000 7f8e7724c700 1 --2- 172.21.15.67:0/789537540 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8e700ff330 0x7f8e700ff750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:13.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.407+0000 7f8e7724c700 1 -- 172.21.15.67:0/789537540 >> 172.21.15.67:0/789537540 conn(0x7f8e700fa8b0 msgr2=0x7f8e700fcd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:13.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.408+0000 7f8e7724c700 1 -- 172.21.15.67:0/789537540 shutdown_connections 2024-04-22T23:20:13.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.408+0000 7f8e7724c700 1 -- 172.21.15.67:0/789537540 wait complete. 2024-04-22T23:20:13.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.408+0000 7f8e7724c700 1 Processor -- start 2024-04-22T23:20:13.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.408+0000 7f8e7724c700 1 -- start start 2024-04-22T23:20:13.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.408+0000 7f8e7724c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e700ff330 0x7f8e701942e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:13.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.408+0000 7f8e7724c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8e70100530 0x7f8e70194820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:13.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.408+0000 7f8e7724c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e70194e40 con 0x7f8e700ff330 2024-04-22T23:20:13.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.408+0000 7f8e7724c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e70194f80 con 0x7f8e70100530 2024-04-22T23:20:13.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.408+0000 7f8e74fe8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e700ff330 0x7f8e701942e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:13.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.408+0000 7f8e74fe8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e700ff330 0x7f8e701942e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54584/0 (socket says 172.21.15.67:54584) 2024-04-22T23:20:13.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.408+0000 7f8e74fe8700 1 -- 172.21.15.67:0/1162150750 learned_addr learned my addr 172.21.15.67:0/1162150750 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:20:13.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.409+0000 7f8e6ffff700 1 --2- 172.21.15.67:0/1162150750 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8e70100530 0x7f8e70194820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:13.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.409+0000 7f8e74fe8700 1 -- 172.21.15.67:0/1162150750 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8e70100530 msgr2=0x7f8e70194820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:13.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.409+0000 7f8e74fe8700 1 --2- 172.21.15.67:0/1162150750 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8e70100530 0x7f8e70194820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:13.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.409+0000 7f8e74fe8700 1 -- 172.21.15.67:0/1162150750 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e5800a700 con 0x7f8e700ff330 2024-04-22T23:20:13.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.409+0000 7f8e6ffff700 1 --2- 172.21.15.67:0/1162150750 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8e70100530 0x7f8e70194820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:20:13.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.409+0000 7f8e74fe8700 1 --2- 172.21.15.67:0/1162150750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e700ff330 0x7f8e701942e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f8e680099e0 tx=0x7f8e68009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:13.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.409+0000 7f8e6dffb700 1 -- 172.21.15.67:0/1162150750 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e68010070 con 0x7f8e700ff330 2024-04-22T23:20:13.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.409+0000 7f8e6dffb700 1 -- 172.21.15.67:0/1162150750 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8e68008d10 con 0x7f8e700ff330 2024-04-22T23:20:13.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.409+0000 7f8e7724c700 1 -- 172.21.15.67:0/1162150750 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e70199a30 con 0x7f8e700ff330 2024-04-22T23:20:13.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.409+0000 7f8e6dffb700 1 -- 172.21.15.67:0/1162150750 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e680046f0 con 0x7f8e700ff330 2024-04-22T23:20:13.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.409+0000 7f8e7724c700 1 -- 172.21.15.67:0/1162150750 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e70199f80 con 0x7f8e700ff330 2024-04-22T23:20:13.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.410+0000 7f8e7724c700 1 -- 172.21.15.67:0/1162150750 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e70107600 con 0x7f8e700ff330 2024-04-22T23:20:13.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.411+0000 7f8e6dffb700 1 -- 172.21.15.67:0/1162150750 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f8e68014030 con 0x7f8e700ff330 2024-04-22T23:20:13.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.411+0000 7f8e6dffb700 1 --2- 172.21.15.67:0/1162150750 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f8e5c06c4e0 0x7f8e5c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:13.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.411+0000 7f8e6ffff700 1 --2- 172.21.15.67:0/1162150750 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f8e5c06c4e0 0x7f8e5c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:13.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.411+0000 7f8e6dffb700 1 -- 172.21.15.67:0/1162150750 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f8e6808ad90 con 0x7f8e700ff330 2024-04-22T23:20:13.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.412+0000 7f8e6ffff700 1 --2- 172.21.15.67:0/1162150750 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f8e5c06c4e0 0x7f8e5c06e9a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8e58011bd0 tx=0x7f8e58005fb0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:13.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.416+0000 7f8e6dffb700 1 -- 172.21.15.67:0/1162150750 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f8e68059340 con 0x7f8e700ff330 2024-04-22T23:20:13.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.629+0000 7f8e7724c700 1 -- 172.21.15.67:0/1162150750 --> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e70104e80 con 0x7f8e5c06c4e0 2024-04-22T23:20:13.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.630+0000 7f8e6dffb700 1 -- 172.21.15.67:0/1162150750 <== mgr.14215 v2:172.21.15.67:6800/2589747850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7f8e70104e80 con 0x7f8e5c06c4e0 2024-04-22T23:20:13.631 INFO:teuthology.orchestra.run.smithi067.stdout:HOST ADDR LABELS STATUS 2024-04-22T23:20:13.631 INFO:teuthology.orchestra.run.smithi067.stdout:smithi067 172.21.15.67 2024-04-22T23:20:13.631 INFO:teuthology.orchestra.run.smithi067.stdout:smithi134 172.21.15.134 2024-04-22T23:20:13.631 INFO:teuthology.orchestra.run.smithi067.stdout:2 hosts in cluster 2024-04-22T23:20:13.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.632+0000 7f8e7724c700 1 -- 172.21.15.67:0/1162150750 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f8e5c06c4e0 msgr2=0x7f8e5c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:13.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.632+0000 7f8e7724c700 1 --2- 172.21.15.67:0/1162150750 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f8e5c06c4e0 0x7f8e5c06e9a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8e58011bd0 tx=0x7f8e58005fb0 comp rx=0 tx=0).stop 2024-04-22T23:20:13.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.632+0000 7f8e7724c700 1 -- 172.21.15.67:0/1162150750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e700ff330 msgr2=0x7f8e701942e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:13.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.632+0000 7f8e7724c700 1 --2- 172.21.15.67:0/1162150750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e700ff330 0x7f8e701942e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f8e680099e0 tx=0x7f8e68009da0 comp rx=0 tx=0).stop 2024-04-22T23:20:13.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.632+0000 7f8e7724c700 1 -- 172.21.15.67:0/1162150750 shutdown_connections 2024-04-22T23:20:13.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.632+0000 7f8e7724c700 1 --2- 172.21.15.67:0/1162150750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e700ff330 0x7f8e701942e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:13.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.632+0000 7f8e7724c700 1 --2- 172.21.15.67:0/1162150750 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f8e5c06c4e0 0x7f8e5c06e9a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:13.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.632+0000 7f8e7724c700 1 --2- 172.21.15.67:0/1162150750 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8e70100530 0x7f8e70194820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:13.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.632+0000 7f8e7724c700 1 -- 172.21.15.67:0/1162150750 >> 172.21.15.67:0/1162150750 conn(0x7f8e700fa8b0 msgr2=0x7f8e70103760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:13.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.632+0000 7f8e7724c700 1 -- 172.21.15.67:0/1162150750 shutdown_connections 2024-04-22T23:20:13.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:13.632+0000 7f8e7724c700 1 -- 172.21.15.67:0/1162150750 wait complete. 2024-04-22T23:20:14.338 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-22T23:20:14.549 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:20:14.672 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:14 smithi067 ceph-mon[26453]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:20:14.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:14 smithi134 ceph-mon[34609]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:20:15.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:15 smithi067 ceph-mon[26453]: from='client.14456 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:20:15.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:15 smithi134 ceph-mon[34609]: from='client.14456 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:20:16.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.199+0000 7fd555b72700 1 -- 172.21.15.67:0/3216547553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd550100ca0 msgr2=0x7fd5501010c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:16.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.199+0000 7fd555b72700 1 --2- 172.21.15.67:0/3216547553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd550100ca0 0x7fd5501010c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd544009910 tx=0x7fd544009c20 comp rx=0 tx=0).stop 2024-04-22T23:20:16.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.199+0000 7fd555b72700 1 -- 172.21.15.67:0/3216547553 shutdown_connections 2024-04-22T23:20:16.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.199+0000 7fd555b72700 1 --2- 172.21.15.67:0/3216547553 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd550101ea0 0x7fd550102300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:16.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.199+0000 7fd555b72700 1 --2- 172.21.15.67:0/3216547553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd550100ca0 0x7fd5501010c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:16.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.199+0000 7fd555b72700 1 -- 172.21.15.67:0/3216547553 >> 172.21.15.67:0/3216547553 conn(0x7fd5500fc260 msgr2=0x7fd5500fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:16.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.199+0000 7fd555b72700 1 -- 172.21.15.67:0/3216547553 shutdown_connections 2024-04-22T23:20:16.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.199+0000 7fd555b72700 1 -- 172.21.15.67:0/3216547553 wait complete. 2024-04-22T23:20:16.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.200+0000 7fd555b72700 1 Processor -- start 2024-04-22T23:20:16.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.200+0000 7fd555b72700 1 -- start start 2024-04-22T23:20:16.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.200+0000 7fd555b72700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd550100ca0 0x7fd550196550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:16.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.200+0000 7fd555b72700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd550101ea0 0x7fd550196a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:16.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.200+0000 7fd555b72700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5501970b0 con 0x7fd550100ca0 2024-04-22T23:20:16.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.200+0000 7fd555b72700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5501971f0 con 0x7fd550101ea0 2024-04-22T23:20:16.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.200+0000 7fd54f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd550100ca0 0x7fd550196550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:16.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.200+0000 7fd54f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd550100ca0 0x7fd550196550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54600/0 (socket says 172.21.15.67:54600) 2024-04-22T23:20:16.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.200+0000 7fd54f7fe700 1 -- 172.21.15.67:0/3919532176 learned_addr learned my addr 172.21.15.67:0/3919532176 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:20:16.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.201+0000 7fd54f7fe700 1 -- 172.21.15.67:0/3919532176 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd550101ea0 msgr2=0x7fd550196a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:16.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.201+0000 7fd54f7fe700 1 --2- 172.21.15.67:0/3919532176 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd550101ea0 0x7fd550196a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:16.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.201+0000 7fd54f7fe700 1 -- 172.21.15.67:0/3919532176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5440095f0 con 0x7fd550100ca0 2024-04-22T23:20:16.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.201+0000 7fd54f7fe700 1 --2- 172.21.15.67:0/3919532176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd550100ca0 0x7fd550196550 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd54400bda0 tx=0x7fd54400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:16.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.201+0000 7fd54cff9700 1 -- 172.21.15.67:0/3919532176 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd54401d070 con 0x7fd550100ca0 2024-04-22T23:20:16.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.201+0000 7fd555b72700 1 -- 172.21.15.67:0/3919532176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd55019bc40 con 0x7fd550100ca0 2024-04-22T23:20:16.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.201+0000 7fd54cff9700 1 -- 172.21.15.67:0/3919532176 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fd544004c50 con 0x7fd550100ca0 2024-04-22T23:20:16.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.201+0000 7fd54cff9700 1 -- 172.21.15.67:0/3919532176 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd54400f690 con 0x7fd550100ca0 2024-04-22T23:20:16.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.201+0000 7fd555b72700 1 -- 172.21.15.67:0/3919532176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd55019c130 con 0x7fd550100ca0 2024-04-22T23:20:16.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.202+0000 7fd555b72700 1 -- 172.21.15.67:0/3919532176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd550109880 con 0x7fd550100ca0 2024-04-22T23:20:16.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.203+0000 7fd54cff9700 1 -- 172.21.15.67:0/3919532176 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fd54400f810 con 0x7fd550100ca0 2024-04-22T23:20:16.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.203+0000 7fd54cff9700 1 --2- 172.21.15.67:0/3919532176 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fd53806c4e0 0x7fd53806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:16.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.203+0000 7fd54cff9700 1 -- 172.21.15.67:0/3919532176 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fd54408cbf0 con 0x7fd550100ca0 2024-04-22T23:20:16.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.203+0000 7fd54effd700 1 --2- 172.21.15.67:0/3919532176 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fd53806c4e0 0x7fd53806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:16.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.204+0000 7fd54effd700 1 --2- 172.21.15.67:0/3919532176 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fd53806c4e0 0x7fd53806e9a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd5400097b0 tx=0x7fd540006d20 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:16.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.208+0000 7fd54cff9700 1 -- 172.21.15.67:0/3919532176 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd54405b250 con 0x7fd550100ca0 2024-04-22T23:20:16.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.415+0000 7fd555b72700 1 -- 172.21.15.67:0/3919532176 --> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fd5501067f0 con 0x7fd53806c4e0 2024-04-22T23:20:16.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.416+0000 7fd54cff9700 1 -- 172.21.15.67:0/3919532176 <== mgr.14215 v2:172.21.15.67:6800/2589747850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7fd5501067f0 con 0x7fd53806c4e0 2024-04-22T23:20:16.417 INFO:teuthology.orchestra.run.smithi067.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-22T23:20:16.418 INFO:teuthology.orchestra.run.smithi067.stdout:smithi067 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT53440042400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-22T23:20:16.418 INFO:teuthology.orchestra.run.smithi067.stdout:smithi134 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT62420002400BGN 372G No 74s ago Has a FileSystem, LVM detected 2024-04-22T23:20:16.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.418+0000 7fd555b72700 1 -- 172.21.15.67:0/3919532176 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fd53806c4e0 msgr2=0x7fd53806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:16.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.418+0000 7fd555b72700 1 --2- 172.21.15.67:0/3919532176 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fd53806c4e0 0x7fd53806e9a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd5400097b0 tx=0x7fd540006d20 comp rx=0 tx=0).stop 2024-04-22T23:20:16.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.418+0000 7fd555b72700 1 -- 172.21.15.67:0/3919532176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd550100ca0 msgr2=0x7fd550196550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:16.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.418+0000 7fd555b72700 1 --2- 172.21.15.67:0/3919532176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd550100ca0 0x7fd550196550 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd54400bda0 tx=0x7fd54400be80 comp rx=0 tx=0).stop 2024-04-22T23:20:16.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.418+0000 7fd555b72700 1 -- 172.21.15.67:0/3919532176 shutdown_connections 2024-04-22T23:20:16.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.418+0000 7fd555b72700 1 --2- 172.21.15.67:0/3919532176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd550100ca0 0x7fd550196550 secure :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd54400bda0 tx=0x7fd54400be80 comp rx=0 tx=0).stop 2024-04-22T23:20:16.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.418+0000 7fd555b72700 1 --2- 172.21.15.67:0/3919532176 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fd53806c4e0 0x7fd53806e9a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:16.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.418+0000 7fd555b72700 1 --2- 172.21.15.67:0/3919532176 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd550101ea0 0x7fd550196a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:16.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.418+0000 7fd555b72700 1 -- 172.21.15.67:0/3919532176 >> 172.21.15.67:0/3919532176 conn(0x7fd5500fc260 msgr2=0x7fd5501050d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:16.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.419+0000 7fd555b72700 1 -- 172.21.15.67:0/3919532176 shutdown_connections 2024-04-22T23:20:16.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:16.419+0000 7fd555b72700 1 -- 172.21.15.67:0/3919532176 wait complete. 2024-04-22T23:20:16.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:16 smithi067 ceph-mon[26453]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:20:16.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:16 smithi134 ceph-mon[34609]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:20:17.160 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T23:20:17.170 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi067.front.sepia.ceph.com 2024-04-22T23:20:17.170 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-04-22T23:20:17.384 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:20:17.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:17 smithi067 ceph-mon[26453]: from='client.14460 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:20:17.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:17 smithi134 ceph-mon[34609]: from='client.14460 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:20:18.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:18 smithi134 ceph-mon[34609]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:20:18.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:18 smithi067 ceph-mon[26453]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:20:19.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:19.084+0000 7f859ffd5700 1 -- 172.21.15.67:0/668168633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8598100cb0 msgr2=0x7f85981010d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:19.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:19.084+0000 7f859ffd5700 1 --2- 172.21.15.67:0/668168633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8598100cb0 0x7f85981010d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8590009910 tx=0x7f8590009c20 comp rx=0 tx=0).stop 2024-04-22T23:20:19.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:19.084+0000 7f859ffd5700 1 -- 172.21.15.67:0/668168633 shutdown_connections 2024-04-22T23:20:19.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:19.084+0000 7f859ffd5700 1 --2- 172.21.15.67:0/668168633 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8598101eb0 0x7f8598102310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:19.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:19.084+0000 7f859ffd5700 1 --2- 172.21.15.67:0/668168633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8598100cb0 0x7f85981010d0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:19.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:19.084+0000 7f859ffd5700 1 -- 172.21.15.67:0/668168633 >> 172.21.15.67:0/668168633 conn(0x7f85980fc230 msgr2=0x7f85980fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:19.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:19.084+0000 7f859ffd5700 1 -- 172.21.15.67:0/668168633 shutdown_connections 2024-04-22T23:20:19.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:19.084+0000 7f859ffd5700 1 -- 172.21.15.67:0/668168633 wait complete. 2024-04-22T23:20:19.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:19.084+0000 7f859ffd5700 1 Processor -- start 2024-04-22T23:20:19.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:19.084+0000 7f859ffd5700 1 -- start start 2024-04-22T23:20:19.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:19.085+0000 7f859ffd5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8598100cb0 0x7f85981964c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:19.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:19.085+0000 7f859ffd5700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8598101eb0 0x7f8598196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:19.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:19.085+0000 7f859ffd5700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8598197020 con 0x7f8598100cb0 2024-04-22T23:20:19.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:19.085+0000 7f859ffd5700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8598197160 con 0x7f8598101eb0 2024-04-22T23:20:19.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:19.085+0000 7f859dd71700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8598100cb0 0x7f85981964c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:19.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:19.085+0000 7f859dd71700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8598100cb0 0x7f85981964c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51940/0 (socket says 172.21.15.67:51940) 2024-04-22T23:20:19.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:19.085+0000 7f859dd71700 1 -- 172.21.15.67:0/897931670 learned_addr learned my addr 172.21.15.67:0/897931670 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:20:19.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:19.085+0000 7f859d570700 1 --2- 172.21.15.67:0/897931670 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8598101eb0 0x7f8598196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:19.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:19.085+0000 7f859dd71700 1 -- 172.21.15.67:0/897931670 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8598101eb0 msgr2=0x7f8598196a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:19.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:19.085+0000 7f859dd71700 1 --2- 172.21.15.67:0/897931670 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8598101eb0 0x7f8598196a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:19.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:19.085+0000 7f859dd71700 1 -- 172.21.15.67:0/897931670 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85900095f0 con 0x7f8598100cb0 2024-04-22T23:20:19.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:19.085+0000 7f859dd71700 1 --2- 172.21.15.67:0/897931670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8598100cb0 0x7f85981964c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f859000bd70 tx=0x7f859000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:19.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:19.085+0000 7f8596ffd700 1 -- 172.21.15.67:0/897931670 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f859001d070 con 0x7f8598100cb0 2024-04-22T23:20:19.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:19.085+0000 7f8596ffd700 1 -- 172.21.15.67:0/897931670 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f859000f460 con 0x7f8598100cb0 2024-04-22T23:20:19.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:19.085+0000 7f8596ffd700 1 -- 172.21.15.67:0/897931670 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8590021640 con 0x7f8598100cb0 2024-04-22T23:20:19.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:19.085+0000 7f859ffd5700 1 -- 172.21.15.67:0/897931670 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f859819bbb0 con 0x7f8598100cb0 2024-04-22T23:20:19.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:19.086+0000 7f859ffd5700 1 -- 172.21.15.67:0/897931670 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f859819c050 con 0x7f8598100cb0 2024-04-22T23:20:19.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:19.087+0000 7f859ffd5700 1 -- 172.21.15.67:0/897931670 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8598109810 con 0x7f8598100cb0 2024-04-22T23:20:19.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:19.087+0000 7f8596ffd700 1 -- 172.21.15.67:0/897931670 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f859000fab0 con 0x7f8598100cb0 2024-04-22T23:20:19.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:19.087+0000 7f8596ffd700 1 --2- 172.21.15.67:0/897931670 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f858006c4e0 0x7f858006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:19.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:19.088+0000 7f859d570700 1 --2- 172.21.15.67:0/897931670 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f858006c4e0 0x7f858006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:19.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:19.088+0000 7f8596ffd700 1 -- 172.21.15.67:0/897931670 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f859008c8d0 con 0x7f8598100cb0 2024-04-22T23:20:19.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:19.088+0000 7f859d570700 1 --2- 172.21.15.67:0/897931670 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f858006c4e0 0x7f858006e9a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f858c005fd0 tx=0x7f858c00a560 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:19.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:19.092+0000 7f8596ffd700 1 -- 172.21.15.67:0/897931670 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f859005ae80 con 0x7f8598100cb0 2024-04-22T23:20:19.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:19.320+0000 7f859ffd5700 1 -- 172.21.15.67:0/897931670 --> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f8598106800 con 0x7f858006c4e0 2024-04-22T23:20:20.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:20 smithi067 ceph-mon[26453]: from='client.14464 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:20:20.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:20 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-04-22T23:20:20.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:20 smithi067 ceph-mon[26453]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:20:20.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:20 smithi134 ceph-mon[34609]: from='client.14464 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:20:20.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:20 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-04-22T23:20:20.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:20 smithi134 ceph-mon[34609]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:20:20.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:20.842+0000 7f8596ffd700 1 -- 172.21.15.67:0/897931670 <== mgr.14215 v2:172.21.15.67:6800/2589747850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8598106800 con 0x7f858006c4e0 2024-04-22T23:20:20.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:20.846+0000 7f859ffd5700 1 -- 172.21.15.67:0/897931670 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f858006c4e0 msgr2=0x7f858006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:20.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:20.846+0000 7f859ffd5700 1 --2- 172.21.15.67:0/897931670 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f858006c4e0 0x7f858006e9a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f858c005fd0 tx=0x7f858c00a560 comp rx=0 tx=0).stop 2024-04-22T23:20:20.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:20.846+0000 7f859ffd5700 1 -- 172.21.15.67:0/897931670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8598100cb0 msgr2=0x7f85981964c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:20.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:20.846+0000 7f859ffd5700 1 --2- 172.21.15.67:0/897931670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8598100cb0 0x7f85981964c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f859000bd70 tx=0x7f859000be50 comp rx=0 tx=0).stop 2024-04-22T23:20:20.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:20.846+0000 7f859ffd5700 1 -- 172.21.15.67:0/897931670 shutdown_connections 2024-04-22T23:20:20.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:20.846+0000 7f859ffd5700 1 --2- 172.21.15.67:0/897931670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8598100cb0 0x7f85981964c0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:20.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:20.846+0000 7f859ffd5700 1 --2- 172.21.15.67:0/897931670 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f858006c4e0 0x7f858006e9a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:20.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:20.846+0000 7f859ffd5700 1 --2- 172.21.15.67:0/897931670 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8598101eb0 0x7f8598196a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:20.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:20.846+0000 7f859ffd5700 1 -- 172.21.15.67:0/897931670 >> 172.21.15.67:0/897931670 conn(0x7f85980fc230 msgr2=0x7f85981050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:20.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:20.846+0000 7f859ffd5700 1 -- 172.21.15.67:0/897931670 shutdown_connections 2024-04-22T23:20:20.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:20.846+0000 7f859ffd5700 1 -- 172.21.15.67:0/897931670 wait complete. 2024-04-22T23:20:21.169 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:20 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[26449]: 2024-04-22T23:20:20.822+0000 7f21924c3700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T23:20:21.429 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:21 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-04-22T23:20:21.429 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:21 smithi067 ceph-mon[26453]: osdmap e41: 6 total, 6 up, 6 in 2024-04-22T23:20:21.429 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:21 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-04-22T23:20:21.429 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:21 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-04-22T23:20:21.430 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:21 smithi067 ceph-mon[26453]: osdmap e42: 6 total, 6 up, 6 in 2024-04-22T23:20:21.430 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:21 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-04-22T23:20:21.430 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:21 smithi067 ceph-mon[26453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T23:20:21.430 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:21 smithi067 ceph-mon[26453]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-22T23:20:21.430 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:21 smithi067 ceph-mon[26453]: osdmap e43: 6 total, 6 up, 6 in 2024-04-22T23:20:21.430 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:21 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-04-22T23:20:21.430 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:21 smithi067 ceph-mon[26453]: fsmap cephfs:0 2024-04-22T23:20:21.430 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:21 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:21.430 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:21 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:20:21.450 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-22T23:20:21.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:21 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-04-22T23:20:21.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:21 smithi134 ceph-mon[34609]: osdmap e41: 6 total, 6 up, 6 in 2024-04-22T23:20:21.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:21 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-04-22T23:20:21.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:21 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-04-22T23:20:21.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:21 smithi134 ceph-mon[34609]: osdmap e42: 6 total, 6 up, 6 in 2024-04-22T23:20:21.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:21 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-04-22T23:20:21.496 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:21 smithi134 ceph-mon[34609]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T23:20:21.496 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:21 smithi134 ceph-mon[34609]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-22T23:20:21.496 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:21 smithi134 ceph-mon[34609]: osdmap e43: 6 total, 6 up, 6 in 2024-04-22T23:20:21.496 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:21 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-04-22T23:20:21.496 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:21 smithi134 ceph-mon[34609]: fsmap cephfs:0 2024-04-22T23:20:21.496 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:21 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:21.496 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:21 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:20:21.677 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:20:22.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:22 smithi067 ceph-mon[26453]: Saving service mds.cephfs spec with placement count:4 2024-04-22T23:20:22.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:22 smithi067 ceph-mon[26453]: pgmap v173: 65 pgs: 14 creating+peering, 24 unknown, 27 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:20:22.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:22 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:20:22.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:22 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:20:22.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:22 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:22.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:22 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi067.aqlxpv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T23:20:22.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:22 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi067.aqlxpv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T23:20:22.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:22 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:20:22.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:22 smithi067 ceph-mon[26453]: osdmap e44: 6 total, 6 up, 6 in 2024-04-22T23:20:22.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:22 smithi134 ceph-mon[34609]: Saving service mds.cephfs spec with placement count:4 2024-04-22T23:20:22.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:22 smithi134 ceph-mon[34609]: pgmap v173: 65 pgs: 14 creating+peering, 24 unknown, 27 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:20:22.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:22 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:20:22.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:22 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:20:22.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:22 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:22.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:22 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi067.aqlxpv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T23:20:22.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:22 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi067.aqlxpv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T23:20:22.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:22 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:20:22.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:22 smithi134 ceph-mon[34609]: osdmap e44: 6 total, 6 up, 6 in 2024-04-22T23:20:23.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.300+0000 7efcf7688700 1 -- 172.21.15.67:0/1591619480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcf00ffc10 msgr2=0x7efcf0100030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:23.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.300+0000 7efcf7688700 1 --2- 172.21.15.67:0/1591619480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcf00ffc10 0x7efcf0100030 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7efce4009910 tx=0x7efce4009c20 comp rx=0 tx=0).stop 2024-04-22T23:20:23.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.300+0000 7efcf7688700 1 -- 172.21.15.67:0/1591619480 shutdown_connections 2024-04-22T23:20:23.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.300+0000 7efcf7688700 1 --2- 172.21.15.67:0/1591619480 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efcf0100e10 0x7efcf0101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:23.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.300+0000 7efcf7688700 1 --2- 172.21.15.67:0/1591619480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcf00ffc10 0x7efcf0100030 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:23.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.300+0000 7efcf7688700 1 -- 172.21.15.67:0/1591619480 >> 172.21.15.67:0/1591619480 conn(0x7efcf00fb190 msgr2=0x7efcf00fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:23.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.300+0000 7efcf7688700 1 -- 172.21.15.67:0/1591619480 shutdown_connections 2024-04-22T23:20:23.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.300+0000 7efcf7688700 1 -- 172.21.15.67:0/1591619480 wait complete. 2024-04-22T23:20:23.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.301+0000 7efcf7688700 1 Processor -- start 2024-04-22T23:20:23.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.301+0000 7efcf7688700 1 -- start start 2024-04-22T23:20:23.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.301+0000 7efcf7688700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efcf00ffc10 0x7efcf01964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:23.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.301+0000 7efcf7688700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcf0100e10 0x7efcf0196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:23.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.301+0000 7efcf7688700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf0197030 con 0x7efcf0100e10 2024-04-22T23:20:23.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.301+0000 7efcf7688700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf0197170 con 0x7efcf00ffc10 2024-04-22T23:20:23.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.301+0000 7efcf4c23700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcf0100e10 0x7efcf0196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:23.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.301+0000 7efcf4c23700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcf0100e10 0x7efcf0196a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51952/0 (socket says 172.21.15.67:51952) 2024-04-22T23:20:23.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.301+0000 7efcf4c23700 1 -- 172.21.15.67:0/1424235673 learned_addr learned my addr 172.21.15.67:0/1424235673 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:20:23.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.301+0000 7efcf5424700 1 --2- 172.21.15.67:0/1424235673 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efcf00ffc10 0x7efcf01964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:23.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.302+0000 7efcf4c23700 1 -- 172.21.15.67:0/1424235673 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efcf00ffc10 msgr2=0x7efcf01964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:23.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.302+0000 7efcf4c23700 1 --2- 172.21.15.67:0/1424235673 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efcf00ffc10 0x7efcf01964d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:23.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.302+0000 7efcf4c23700 1 -- 172.21.15.67:0/1424235673 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce40095f0 con 0x7efcf0100e10 2024-04-22T23:20:23.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.302+0000 7efcf4c23700 1 --2- 172.21.15.67:0/1424235673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcf0100e10 0x7efcf0196a10 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7efcec00c8a0 tx=0x7efcec00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:23.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.302+0000 7efce27fc700 1 -- 172.21.15.67:0/1424235673 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efcec0078e0 con 0x7efcf0100e10 2024-04-22T23:20:23.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.302+0000 7efce27fc700 1 -- 172.21.15.67:0/1424235673 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7efcec00f450 con 0x7efcf0100e10 2024-04-22T23:20:23.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.302+0000 7efce27fc700 1 -- 172.21.15.67:0/1424235673 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efcec00e5e0 con 0x7efcf0100e10 2024-04-22T23:20:23.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.302+0000 7efcf7688700 1 -- 172.21.15.67:0/1424235673 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcf019bc20 con 0x7efcf0100e10 2024-04-22T23:20:23.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.302+0000 7efcf7688700 1 -- 172.21.15.67:0/1424235673 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcf019c170 con 0x7efcf0100e10 2024-04-22T23:20:23.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.303+0000 7efcf7688700 1 -- 172.21.15.67:0/1424235673 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcf0108700 con 0x7efcf0100e10 2024-04-22T23:20:23.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.304+0000 7efce27fc700 1 -- 172.21.15.67:0/1424235673 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7efcec00f5c0 con 0x7efcf0100e10 2024-04-22T23:20:23.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.304+0000 7efce27fc700 1 --2- 172.21.15.67:0/1424235673 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7efcdc06c4e0 0x7efcdc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:23.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.304+0000 7efce27fc700 1 -- 172.21.15.67:0/1424235673 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7efcec08b6c0 con 0x7efcf0100e10 2024-04-22T23:20:23.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.304+0000 7efcf5424700 1 --2- 172.21.15.67:0/1424235673 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7efcdc06c4e0 0x7efcdc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:23.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.304+0000 7efcf5424700 1 --2- 172.21.15.67:0/1424235673 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7efcdc06c4e0 0x7efcdc06e9a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efce4006010 tx=0x7efce400b540 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:23.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.309+0000 7efce27fc700 1 -- 172.21.15.67:0/1424235673 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7efcec0599b0 con 0x7efcf0100e10 2024-04-22T23:20:23.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.556+0000 7efcf7688700 1 -- 172.21.15.67:0/1424235673 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7efcf01088f0 con 0x7efcf0100e10 2024-04-22T23:20:23.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.556+0000 7efce27fc700 1 -- 172.21.15.67:0/1424235673 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1093 (secure 0 0 0) 0x7efcec059540 con 0x7efcf0100e10 2024-04-22T23:20:23.558 INFO:teuthology.orchestra.run.smithi067.stdout:e2 2024-04-22T23:20:23.558 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:20:23.558 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:20:23.558 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:20:23.558 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:20:23.558 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:20:23.558 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:20:23.558 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 2 2024-04-22T23:20:23.558 INFO:teuthology.orchestra.run.smithi067.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-22T23:20:23.558 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:20:23.558 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:20:20.823333+0000 2024-04-22T23:20:23.558 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:20:23.559 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:20:23.559 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:20:23.559 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:20:23.559 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:20:23.559 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:20:23.559 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:20:23.559 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 0 2024-04-22T23:20:23.559 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:20:23.559 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 1 2024-04-22T23:20:23.559 INFO:teuthology.orchestra.run.smithi067.stdout:in 2024-04-22T23:20:23.559 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:20:23.559 INFO:teuthology.orchestra.run.smithi067.stdout:failed 2024-04-22T23:20:23.559 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:20:23.559 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:20:23.559 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:20:23.560 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:20:23.560 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data disabled 2024-04-22T23:20:23.560 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:20:23.560 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:20:23.560 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 0 2024-04-22T23:20:23.560 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:20:23.560 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:20:23.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.558+0000 7efcf7688700 1 -- 172.21.15.67:0/1424235673 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7efcdc06c4e0 msgr2=0x7efcdc06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:23.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.558+0000 7efcf7688700 1 --2- 172.21.15.67:0/1424235673 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7efcdc06c4e0 0x7efcdc06e9a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efce4006010 tx=0x7efce400b540 comp rx=0 tx=0).stop 2024-04-22T23:20:23.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.558+0000 7efcf7688700 1 -- 172.21.15.67:0/1424235673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcf0100e10 msgr2=0x7efcf0196a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:23.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.558+0000 7efcf7688700 1 --2- 172.21.15.67:0/1424235673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcf0100e10 0x7efcf0196a10 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7efcec00c8a0 tx=0x7efcec00cbb0 comp rx=0 tx=0).stop 2024-04-22T23:20:23.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.558+0000 7efcf7688700 1 -- 172.21.15.67:0/1424235673 shutdown_connections 2024-04-22T23:20:23.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.558+0000 7efcf7688700 1 --2- 172.21.15.67:0/1424235673 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7efcdc06c4e0 0x7efcdc06e9a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:23.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.558+0000 7efcf7688700 1 --2- 172.21.15.67:0/1424235673 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efcf00ffc10 0x7efcf01964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:23.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.558+0000 7efcf7688700 1 --2- 172.21.15.67:0/1424235673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcf0100e10 0x7efcf0196a10 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:23.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.558+0000 7efcf7688700 1 -- 172.21.15.67:0/1424235673 >> 172.21.15.67:0/1424235673 conn(0x7efcf00fb190 msgr2=0x7efcf0068580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:23.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.558+0000 7efcf7688700 1 -- 172.21.15.67:0/1424235673 shutdown_connections 2024-04-22T23:20:23.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:23.558+0000 7efcf7688700 1 -- 172.21.15.67:0/1424235673 wait complete. 2024-04-22T23:20:23.561 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 2 2024-04-22T23:20:23.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:23 smithi134 ceph-mon[34609]: Deploying daemon mds.cephfs.smithi067.aqlxpv on smithi067 2024-04-22T23:20:23.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:23 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:23.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:23 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:20:23.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:23 smithi134 ceph-mon[34609]: osdmap e45: 6 total, 6 up, 6 in 2024-04-22T23:20:23.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:23 smithi067 ceph-mon[26453]: Deploying daemon mds.cephfs.smithi067.aqlxpv on smithi067 2024-04-22T23:20:23.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:23 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:23.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:23 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:20:23.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:23 smithi067 ceph-mon[26453]: osdmap e45: 6 total, 6 up, 6 in 2024-04-22T23:20:24.564 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:24 smithi067 ceph-mon[26453]: pgmap v176: 65 pgs: 14 creating+peering, 15 unknown, 36 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:20:24.564 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:24 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/1424235673' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:20:24.589 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T23:20:24.599 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi067.front.sepia.ceph.com 2024-04-22T23:20:24.600 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- bash -c 'ceph fs set cephfs max_mds 2' 2024-04-22T23:20:24.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:24 smithi134 ceph-mon[34609]: pgmap v176: 65 pgs: 14 creating+peering, 15 unknown, 36 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:20:24.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:24 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/1424235673' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:20:25.058 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:20:26.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:26 smithi067 ceph-mon[26453]: pgmap v177: 65 pgs: 5 creating+peering, 60 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:20:26.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:26 smithi134 ceph-mon[34609]: pgmap v177: 65 pgs: 5 creating+peering, 60 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:20:26.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:26.854+0000 7fb6c881d700 1 -- 172.21.15.67:0/396073203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6c4100e10 msgr2=0x7fb6c4101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:26.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:26.854+0000 7fb6c881d700 1 --2- 172.21.15.67:0/396073203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6c4100e10 0x7fb6c4101270 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac009910 tx=0x7fb6ac009c20 comp rx=0 tx=0).stop 2024-04-22T23:20:26.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:26.854+0000 7fb6c881d700 1 -- 172.21.15.67:0/396073203 shutdown_connections 2024-04-22T23:20:26.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:26.854+0000 7fb6c881d700 1 --2- 172.21.15.67:0/396073203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6c4100e10 0x7fb6c4101270 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:26.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:26.854+0000 7fb6c881d700 1 --2- 172.21.15.67:0/396073203 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb6c40ffc10 0x7fb6c4100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:26.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:26.854+0000 7fb6c881d700 1 -- 172.21.15.67:0/396073203 >> 172.21.15.67:0/396073203 conn(0x7fb6c40fb190 msgr2=0x7fb6c40fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:26.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:26.854+0000 7fb6c881d700 1 -- 172.21.15.67:0/396073203 shutdown_connections 2024-04-22T23:20:26.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:26.854+0000 7fb6c881d700 1 -- 172.21.15.67:0/396073203 wait complete. 2024-04-22T23:20:26.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:26.855+0000 7fb6c881d700 1 Processor -- start 2024-04-22T23:20:26.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:26.855+0000 7fb6c881d700 1 -- start start 2024-04-22T23:20:26.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:26.855+0000 7fb6c881d700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb6c40ffc10 0x7fb6c4196480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:26.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:26.855+0000 7fb6c881d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6c4100e10 0x7fb6c41969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:26.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:26.855+0000 7fb6c881d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c4196fe0 con 0x7fb6c4100e10 2024-04-22T23:20:26.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:26.855+0000 7fb6c881d700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c4197120 con 0x7fb6c40ffc10 2024-04-22T23:20:26.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:26.855+0000 7fb6c1d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6c4100e10 0x7fb6c41969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:26.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:26.855+0000 7fb6c1d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6c4100e10 0x7fb6c41969c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51970/0 (socket says 172.21.15.67:51970) 2024-04-22T23:20:26.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:26.855+0000 7fb6c1d9b700 1 -- 172.21.15.67:0/4249217034 learned_addr learned my addr 172.21.15.67:0/4249217034 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:20:26.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:26.855+0000 7fb6c259c700 1 --2- 172.21.15.67:0/4249217034 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb6c40ffc10 0x7fb6c4196480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:26.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:26.855+0000 7fb6c1d9b700 1 -- 172.21.15.67:0/4249217034 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb6c40ffc10 msgr2=0x7fb6c4196480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:26.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:26.855+0000 7fb6c1d9b700 1 --2- 172.21.15.67:0/4249217034 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb6c40ffc10 0x7fb6c4196480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:26.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:26.855+0000 7fb6c1d9b700 1 -- 172.21.15.67:0/4249217034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6ac0095f0 con 0x7fb6c4100e10 2024-04-22T23:20:26.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:26.856+0000 7fb6c1d9b700 1 --2- 172.21.15.67:0/4249217034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6c4100e10 0x7fb6c41969c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac006010 tx=0x7fb6ac00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:26.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:26.856+0000 7fb6bb7fe700 1 -- 172.21.15.67:0/4249217034 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6ac01d070 con 0x7fb6c4100e10 2024-04-22T23:20:26.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:26.856+0000 7fb6c881d700 1 -- 172.21.15.67:0/4249217034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6c419bb70 con 0x7fb6c4100e10 2024-04-22T23:20:26.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:26.856+0000 7fb6bb7fe700 1 -- 172.21.15.67:0/4249217034 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb6ac00f460 con 0x7fb6c4100e10 2024-04-22T23:20:26.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:26.856+0000 7fb6bb7fe700 1 -- 172.21.15.67:0/4249217034 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6ac021640 con 0x7fb6c4100e10 2024-04-22T23:20:26.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:26.856+0000 7fb6c881d700 1 -- 172.21.15.67:0/4249217034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6c419c060 con 0x7fb6c4100e10 2024-04-22T23:20:26.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:26.857+0000 7fb6c881d700 1 -- 172.21.15.67:0/4249217034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6c4108700 con 0x7fb6c4100e10 2024-04-22T23:20:26.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:26.857+0000 7fb6bb7fe700 1 -- 172.21.15.67:0/4249217034 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fb6ac00fab0 con 0x7fb6c4100e10 2024-04-22T23:20:26.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:26.858+0000 7fb6bb7fe700 1 --2- 172.21.15.67:0/4249217034 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fb6b006c4e0 0x7fb6b006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:26.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:26.858+0000 7fb6bb7fe700 1 -- 172.21.15.67:0/4249217034 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7fb6ac08cc40 con 0x7fb6c4100e10 2024-04-22T23:20:26.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:26.858+0000 7fb6c259c700 1 --2- 172.21.15.67:0/4249217034 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fb6b006c4e0 0x7fb6b006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:26.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:26.858+0000 7fb6c259c700 1 --2- 172.21.15.67:0/4249217034 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fb6b006c4e0 0x7fb6b006e9a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4005fd0 tx=0x7fb6b4005dd0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:26.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:26.862+0000 7fb6bb7fe700 1 -- 172.21.15.67:0/4249217034 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb6ac05aeb0 con 0x7fb6c4100e10 2024-04-22T23:20:27.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:27.109+0000 7fb6c881d700 1 -- 172.21.15.67:0/4249217034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7fb6c419c670 con 0x7fb6c4100e10 2024-04-22T23:20:27.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:27.267+0000 7fb6bb7fe700 1 -- 172.21.15.67:0/4249217034 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v3) v1 ==== 105+0+0 (secure 0 0 0) 0x7fb6ac05aa40 con 0x7fb6c4100e10 2024-04-22T23:20:27.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:27.269+0000 7fb6c881d700 1 -- 172.21.15.67:0/4249217034 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fb6b006c4e0 msgr2=0x7fb6b006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:27.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:27.269+0000 7fb6c881d700 1 --2- 172.21.15.67:0/4249217034 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fb6b006c4e0 0x7fb6b006e9a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4005fd0 tx=0x7fb6b4005dd0 comp rx=0 tx=0).stop 2024-04-22T23:20:27.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:27.269+0000 7fb6c881d700 1 -- 172.21.15.67:0/4249217034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6c4100e10 msgr2=0x7fb6c41969c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:27.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:27.269+0000 7fb6c881d700 1 --2- 172.21.15.67:0/4249217034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6c4100e10 0x7fb6c41969c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac006010 tx=0x7fb6ac00be20 comp rx=0 tx=0).stop 2024-04-22T23:20:27.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:27.269+0000 7fb6c881d700 1 -- 172.21.15.67:0/4249217034 shutdown_connections 2024-04-22T23:20:27.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:27.269+0000 7fb6c881d700 1 --2- 172.21.15.67:0/4249217034 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fb6b006c4e0 0x7fb6b006e9a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:27.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:27.269+0000 7fb6c881d700 1 --2- 172.21.15.67:0/4249217034 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb6c40ffc10 0x7fb6c4196480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:27.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:27.269+0000 7fb6c881d700 1 --2- 172.21.15.67:0/4249217034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6c4100e10 0x7fb6c41969c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:27.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:27.269+0000 7fb6c881d700 1 -- 172.21.15.67:0/4249217034 >> 172.21.15.67:0/4249217034 conn(0x7fb6c40fb190 msgr2=0x7fb6c4068580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:27.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:27.270+0000 7fb6c881d700 1 -- 172.21.15.67:0/4249217034 shutdown_connections 2024-04-22T23:20:27.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:27.270+0000 7fb6c881d700 1 -- 172.21.15.67:0/4249217034 wait complete. 2024-04-22T23:20:27.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:27 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/4249217034' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-04-22T23:20:27.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:27 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/4249217034' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-04-22T23:20:28.169 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T23:20:28.183 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi067.front.sepia.ceph.com 2024-04-22T23:20:28.183 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- bash -c 'ceph fs set cephfs allow_standby_replay true' 2024-04-22T23:20:28.408 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:20:28.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:28 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/4249217034' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-04-22T23:20:28.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:28 smithi134 ceph-mon[34609]: mds.? [v2:172.21.15.67:6826/2381214817,v1:172.21.15.67:6827/2381214817] up:boot 2024-04-22T23:20:28.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:28 smithi134 ceph-mon[34609]: daemon mds.cephfs.smithi067.aqlxpv assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-04-22T23:20:28.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:28 smithi134 ceph-mon[34609]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-22T23:20:28.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:28 smithi134 ceph-mon[34609]: fsmap cephfs:0 1 up:standby 2024-04-22T23:20:28.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:28 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.aqlxpv"}]: dispatch 2024-04-22T23:20:28.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:28 smithi134 ceph-mon[34609]: fsmap cephfs:1 {0=cephfs.smithi067.aqlxpv=up:creating} 2024-04-22T23:20:28.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:28 smithi134 ceph-mon[34609]: daemon mds.cephfs.smithi067.aqlxpv is now active in filesystem cephfs as rank 0 2024-04-22T23:20:28.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:28 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:28.496 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:28 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:28.496 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:28 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:28.496 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:28 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi134.owipad", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T23:20:28.496 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:28 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi134.owipad", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T23:20:28.496 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:28 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:20:28.496 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:28 smithi134 ceph-mon[34609]: Deploying daemon mds.cephfs.smithi134.owipad on smithi134 2024-04-22T23:20:28.496 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:28 smithi134 ceph-mon[34609]: pgmap v178: 65 pgs: 65 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:20:28.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:28 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/4249217034' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-04-22T23:20:28.520 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:28 smithi067 ceph-mon[26453]: mds.? [v2:172.21.15.67:6826/2381214817,v1:172.21.15.67:6827/2381214817] up:boot 2024-04-22T23:20:28.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:28 smithi067 ceph-mon[26453]: daemon mds.cephfs.smithi067.aqlxpv assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-04-22T23:20:28.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:28 smithi067 ceph-mon[26453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-22T23:20:28.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:28 smithi067 ceph-mon[26453]: fsmap cephfs:0 1 up:standby 2024-04-22T23:20:28.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:28 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.aqlxpv"}]: dispatch 2024-04-22T23:20:28.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:28 smithi067 ceph-mon[26453]: fsmap cephfs:1 {0=cephfs.smithi067.aqlxpv=up:creating} 2024-04-22T23:20:28.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:28 smithi067 ceph-mon[26453]: daemon mds.cephfs.smithi067.aqlxpv is now active in filesystem cephfs as rank 0 2024-04-22T23:20:28.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:28 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:28.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:28 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:28.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:28 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:28.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:28 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi134.owipad", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T23:20:28.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:28 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi134.owipad", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T23:20:28.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:28 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:20:28.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:28 smithi067 ceph-mon[26453]: Deploying daemon mds.cephfs.smithi134.owipad on smithi134 2024-04-22T23:20:28.521 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:28 smithi067 ceph-mon[26453]: pgmap v178: 65 pgs: 65 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-22T23:20:29.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:29 smithi134 ceph-mon[34609]: mds.? [v2:172.21.15.67:6826/2381214817,v1:172.21.15.67:6827/2381214817] up:active 2024-04-22T23:20:29.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:29 smithi134 ceph-mon[34609]: fsmap cephfs:1 {0=cephfs.smithi067.aqlxpv=up:active} 2024-04-22T23:20:29.523 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:29 smithi067 ceph-mon[26453]: mds.? [v2:172.21.15.67:6826/2381214817,v1:172.21.15.67:6827/2381214817] up:active 2024-04-22T23:20:29.524 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:29 smithi067 ceph-mon[26453]: fsmap cephfs:1 {0=cephfs.smithi067.aqlxpv=up:active} 2024-04-22T23:20:30.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:30.004+0000 7f78a17eb700 1 -- 172.21.15.67:0/804455282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f789c100ca0 msgr2=0x7f789c1010c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:30.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:30.004+0000 7f78a17eb700 1 --2- 172.21.15.67:0/804455282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f789c100ca0 0x7f789c1010c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f788c009910 tx=0x7f788c009c20 comp rx=0 tx=0).stop 2024-04-22T23:20:30.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:30.004+0000 7f78a17eb700 1 -- 172.21.15.67:0/804455282 shutdown_connections 2024-04-22T23:20:30.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:30.004+0000 7f78a17eb700 1 --2- 172.21.15.67:0/804455282 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f789c101ea0 0x7f789c102300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:30.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:30.004+0000 7f78a17eb700 1 --2- 172.21.15.67:0/804455282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f789c100ca0 0x7f789c1010c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:30.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:30.004+0000 7f78a17eb700 1 -- 172.21.15.67:0/804455282 >> 172.21.15.67:0/804455282 conn(0x7f789c0fc260 msgr2=0x7f789c0fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:30.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:30.004+0000 7f78a17eb700 1 -- 172.21.15.67:0/804455282 shutdown_connections 2024-04-22T23:20:30.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:30.004+0000 7f78a17eb700 1 -- 172.21.15.67:0/804455282 wait complete. 2024-04-22T23:20:30.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:30.005+0000 7f78a17eb700 1 Processor -- start 2024-04-22T23:20:30.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:30.005+0000 7f78a17eb700 1 -- start start 2024-04-22T23:20:30.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:30.005+0000 7f78a17eb700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f789c100ca0 0x7f789c1986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:30.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:30.005+0000 7f78a17eb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f789c101ea0 0x7f789c198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:30.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:30.005+0000 7f78a17eb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789c199220 con 0x7f789c101ea0 2024-04-22T23:20:30.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:30.005+0000 7f78a17eb700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789c199360 con 0x7f789c100ca0 2024-04-22T23:20:30.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:30.006+0000 7f789a7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f789c101ea0 0x7f789c198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:30.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:30.006+0000 7f789affd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f789c100ca0 0x7f789c1986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:30.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:30.006+0000 7f789a7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f789c101ea0 0x7f789c198c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38050/0 (socket says 172.21.15.67:38050) 2024-04-22T23:20:30.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:30.006+0000 7f789a7fc700 1 -- 172.21.15.67:0/28936025 learned_addr learned my addr 172.21.15.67:0/28936025 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:20:30.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:30.006+0000 7f789a7fc700 1 -- 172.21.15.67:0/28936025 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f789c100ca0 msgr2=0x7f789c1986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:30.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:30.006+0000 7f789a7fc700 1 --2- 172.21.15.67:0/28936025 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f789c100ca0 0x7f789c1986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:30.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:30.006+0000 7f789a7fc700 1 -- 172.21.15.67:0/28936025 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f788c0095f0 con 0x7f789c101ea0 2024-04-22T23:20:30.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:30.006+0000 7f789a7fc700 1 --2- 172.21.15.67:0/28936025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f789c101ea0 0x7f789c198c00 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f789000e920 tx=0x7f789000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:30.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:30.006+0000 7f7883fff700 1 -- 172.21.15.67:0/28936025 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f789000c9d0 con 0x7f789c101ea0 2024-04-22T23:20:30.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:30.006+0000 7f78a17eb700 1 -- 172.21.15.67:0/28936025 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f789c19de10 con 0x7f789c101ea0 2024-04-22T23:20:30.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:30.006+0000 7f7883fff700 1 -- 172.21.15.67:0/28936025 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7890010430 con 0x7f789c101ea0 2024-04-22T23:20:30.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:30.006+0000 7f7883fff700 1 -- 172.21.15.67:0/28936025 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7890018650 con 0x7f789c101ea0 2024-04-22T23:20:30.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:30.006+0000 7f78a17eb700 1 -- 172.21.15.67:0/28936025 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f789c19e330 con 0x7f789c101ea0 2024-04-22T23:20:30.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:30.007+0000 7f78a17eb700 1 -- 172.21.15.67:0/28936025 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f789c105f80 con 0x7f789c101ea0 2024-04-22T23:20:30.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:30.008+0000 7f7883fff700 1 -- 172.21.15.67:0/28936025 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f789000cb30 con 0x7f789c101ea0 2024-04-22T23:20:30.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:30.008+0000 7f7883fff700 1 --2- 172.21.15.67:0/28936025 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f788406c4e0 0x7f788406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:30.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:30.008+0000 7f7883fff700 1 -- 172.21.15.67:0/28936025 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f7890014070 con 0x7f789c101ea0 2024-04-22T23:20:30.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:30.008+0000 7f789affd700 1 --2- 172.21.15.67:0/28936025 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f788406c4e0 0x7f788406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:30.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:30.009+0000 7f789affd700 1 --2- 172.21.15.67:0/28936025 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f788406c4e0 0x7f788406e9a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f788c00b6f0 tx=0x7f788c005940 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:30.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:30.013+0000 7f7883fff700 1 -- 172.21.15.67:0/28936025 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f78900520e0 con 0x7f789c101ea0 2024-04-22T23:20:30.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:30.261+0000 7f78a17eb700 1 -- 172.21.15.67:0/28936025 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f789c02cc70 con 0x7f789c101ea0 2024-04-22T23:20:30.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:30 smithi067 ceph-mon[26453]: pgmap v179: 65 pgs: 65 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1023 B/s wr, 2 op/s 2024-04-22T23:20:30.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:30 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:30.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:30 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:30.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:30 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:30.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:30 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi067.xyazmh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T23:20:30.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:30 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi067.xyazmh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T23:20:30.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:30 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:20:30.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:30 smithi067 ceph-mon[26453]: daemon mds.cephfs.smithi134.owipad assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-22T23:20:30.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:30 smithi067 ceph-mon[26453]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-22T23:20:30.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:30 smithi067 ceph-mon[26453]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-22T23:20:30.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:30 smithi067 ceph-mon[26453]: mds.? [v2:172.21.15.134:6824/535957142,v1:172.21.15.134:6825/535957142] up:boot 2024-04-22T23:20:30.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:30 smithi067 ceph-mon[26453]: fsmap cephfs:2 {0=cephfs.smithi067.aqlxpv=up:active,1=cephfs.smithi134.owipad=up:creating} 2024-04-22T23:20:30.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:30 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.owipad"}]: dispatch 2024-04-22T23:20:30.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:30 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/28936025' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-22T23:20:30.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:30 smithi067 ceph-mon[26453]: daemon mds.cephfs.smithi134.owipad is now active in filesystem cephfs as rank 1 2024-04-22T23:20:30.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:30 smithi134 ceph-mon[34609]: pgmap v179: 65 pgs: 65 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1023 B/s wr, 2 op/s 2024-04-22T23:20:30.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:30 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:30.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:30 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:30.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:30 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:30.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:30 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi067.xyazmh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T23:20:30.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:30 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi067.xyazmh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T23:20:30.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:30 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:20:30.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:30 smithi134 ceph-mon[34609]: daemon mds.cephfs.smithi134.owipad assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-22T23:20:30.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:30 smithi134 ceph-mon[34609]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-22T23:20:30.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:30 smithi134 ceph-mon[34609]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-22T23:20:30.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:30 smithi134 ceph-mon[34609]: mds.? [v2:172.21.15.134:6824/535957142,v1:172.21.15.134:6825/535957142] up:boot 2024-04-22T23:20:30.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:30 smithi134 ceph-mon[34609]: fsmap cephfs:2 {0=cephfs.smithi067.aqlxpv=up:active,1=cephfs.smithi134.owipad=up:creating} 2024-04-22T23:20:30.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:30 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.owipad"}]: dispatch 2024-04-22T23:20:30.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:30 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/28936025' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-22T23:20:30.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:30 smithi134 ceph-mon[34609]: daemon mds.cephfs.smithi134.owipad is now active in filesystem cephfs as rank 1 2024-04-22T23:20:31.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:31.265+0000 7f7883fff700 1 -- 172.21.15.67:0/28936025 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v7) v1 ==== 121+0+0 (secure 0 0 0) 0x7f7890059d60 con 0x7f789c101ea0 2024-04-22T23:20:31.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:31.268+0000 7f78a17eb700 1 -- 172.21.15.67:0/28936025 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f788406c4e0 msgr2=0x7f788406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:31.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:31.268+0000 7f78a17eb700 1 --2- 172.21.15.67:0/28936025 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f788406c4e0 0x7f788406e9a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f788c00b6f0 tx=0x7f788c005940 comp rx=0 tx=0).stop 2024-04-22T23:20:31.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:31.268+0000 7f78a17eb700 1 -- 172.21.15.67:0/28936025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f789c101ea0 msgr2=0x7f789c198c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:31.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:31.268+0000 7f78a17eb700 1 --2- 172.21.15.67:0/28936025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f789c101ea0 0x7f789c198c00 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f789000e920 tx=0x7f789000ec30 comp rx=0 tx=0).stop 2024-04-22T23:20:31.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:31.268+0000 7f78a17eb700 1 -- 172.21.15.67:0/28936025 shutdown_connections 2024-04-22T23:20:31.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:31.268+0000 7f78a17eb700 1 --2- 172.21.15.67:0/28936025 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f788406c4e0 0x7f788406e9a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:31.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:31.268+0000 7f78a17eb700 1 --2- 172.21.15.67:0/28936025 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f789c100ca0 0x7f789c1986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:31.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:31.268+0000 7f78a17eb700 1 --2- 172.21.15.67:0/28936025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f789c101ea0 0x7f789c198c00 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:31.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:31.268+0000 7f78a17eb700 1 -- 172.21.15.67:0/28936025 >> 172.21.15.67:0/28936025 conn(0x7f789c0fc260 msgr2=0x7f789c1030c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:31.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:31.268+0000 7f78a17eb700 1 -- 172.21.15.67:0/28936025 shutdown_connections 2024-04-22T23:20:31.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:31.268+0000 7f78a17eb700 1 -- 172.21.15.67:0/28936025 wait complete. 2024-04-22T23:20:31.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:31 smithi067 ceph-mon[26453]: Deploying daemon mds.cephfs.smithi067.xyazmh on smithi067 2024-04-22T23:20:31.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:31 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/28936025' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-22T23:20:31.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:31 smithi067 ceph-mon[26453]: mds.? [v2:172.21.15.134:6824/535957142,v1:172.21.15.134:6825/535957142] up:active 2024-04-22T23:20:31.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:31 smithi067 ceph-mon[26453]: fsmap cephfs:2 {0=cephfs.smithi067.aqlxpv=up:active,1=cephfs.smithi134.owipad=up:active} 2024-04-22T23:20:31.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:31 smithi134 ceph-mon[34609]: Deploying daemon mds.cephfs.smithi067.xyazmh on smithi067 2024-04-22T23:20:31.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:31 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/28936025' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-22T23:20:31.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:31 smithi134 ceph-mon[34609]: mds.? [v2:172.21.15.134:6824/535957142,v1:172.21.15.134:6825/535957142] up:active 2024-04-22T23:20:31.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:31 smithi134 ceph-mon[34609]: fsmap cephfs:2 {0=cephfs.smithi067.aqlxpv=up:active,1=cephfs.smithi134.owipad=up:active} 2024-04-22T23:20:32.226 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T23:20:32.240 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi067.front.sepia.ceph.com 2024-04-22T23:20:32.241 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2024-04-22T23:20:32.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:32 smithi067 ceph-mon[26453]: pgmap v180: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s wr, 8 op/s 2024-04-22T23:20:32.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:32 smithi134 ceph-mon[34609]: pgmap v180: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s wr, 8 op/s 2024-04-22T23:20:32.762 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:20:33.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:33 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:33.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:33 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:33.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:33 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:33.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:33 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:33.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:33 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi134.vuxbsj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T23:20:33.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:33 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi134.vuxbsj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T23:20:33.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:33 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:20:33.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:33 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:33.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:33 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:33.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:33 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:33.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:33 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:33.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:33 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi134.vuxbsj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T23:20:33.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:33 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi134.vuxbsj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T23:20:33.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:33 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:20:34.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:34.357+0000 7ffad3717700 1 -- 172.21.15.67:0/953650929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffacc101ea0 msgr2=0x7ffacc102300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:34.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:34.357+0000 7ffad3717700 1 --2- 172.21.15.67:0/953650929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffacc101ea0 0x7ffacc102300 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ffabc009910 tx=0x7ffabc009c20 comp rx=0 tx=0).stop 2024-04-22T23:20:34.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:34.357+0000 7ffad3717700 1 -- 172.21.15.67:0/953650929 shutdown_connections 2024-04-22T23:20:34.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:34.357+0000 7ffad3717700 1 --2- 172.21.15.67:0/953650929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffacc101ea0 0x7ffacc102300 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:34.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:34.357+0000 7ffad3717700 1 --2- 172.21.15.67:0/953650929 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffacc100ca0 0x7ffacc1010c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:34.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:34.357+0000 7ffad3717700 1 -- 172.21.15.67:0/953650929 >> 172.21.15.67:0/953650929 conn(0x7ffacc0fc260 msgr2=0x7ffacc0fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:34.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:34.357+0000 7ffad3717700 1 -- 172.21.15.67:0/953650929 shutdown_connections 2024-04-22T23:20:34.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:34.357+0000 7ffad3717700 1 -- 172.21.15.67:0/953650929 wait complete. 2024-04-22T23:20:34.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:34.357+0000 7ffad3717700 1 Processor -- start 2024-04-22T23:20:34.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:34.357+0000 7ffad3717700 1 -- start start 2024-04-22T23:20:34.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:34.358+0000 7ffad3717700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffacc100ca0 0x7ffacc074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:34.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:34.358+0000 7ffad3717700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffacc101ea0 0x7ffacc073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:34.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:34.358+0000 7ffad3717700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffacc0736c0 con 0x7ffacc101ea0 2024-04-22T23:20:34.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:34.358+0000 7ffad3717700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffacc073800 con 0x7ffacc100ca0 2024-04-22T23:20:34.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:34.358+0000 7ffad0cb2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffacc101ea0 0x7ffacc073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:34.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:34.358+0000 7ffad0cb2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffacc101ea0 0x7ffacc073180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38098/0 (socket says 172.21.15.67:38098) 2024-04-22T23:20:34.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:34.358+0000 7ffad0cb2700 1 -- 172.21.15.67:0/4059094670 learned_addr learned my addr 172.21.15.67:0/4059094670 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:20:34.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:34.358+0000 7ffad14b3700 1 --2- 172.21.15.67:0/4059094670 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffacc100ca0 0x7ffacc074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:34.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:34.358+0000 7ffad0cb2700 1 -- 172.21.15.67:0/4059094670 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffacc100ca0 msgr2=0x7ffacc074b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:34.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:34.358+0000 7ffad0cb2700 1 --2- 172.21.15.67:0/4059094670 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffacc100ca0 0x7ffacc074b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:34.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:34.358+0000 7ffad0cb2700 1 -- 172.21.15.67:0/4059094670 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffabc0095f0 con 0x7ffacc101ea0 2024-04-22T23:20:34.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:34.358+0000 7ffad0cb2700 1 --2- 172.21.15.67:0/4059094670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffacc101ea0 0x7ffacc073180 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ffabc006010 tx=0x7ffabc00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:34.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:34.358+0000 7ffac27fc700 1 -- 172.21.15.67:0/4059094670 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffabc01d070 con 0x7ffacc101ea0 2024-04-22T23:20:34.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:34.358+0000 7ffac27fc700 1 -- 172.21.15.67:0/4059094670 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ffabc00f460 con 0x7ffacc101ea0 2024-04-22T23:20:34.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:34.358+0000 7ffad3717700 1 -- 172.21.15.67:0/4059094670 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffacc073a80 con 0x7ffacc101ea0 2024-04-22T23:20:34.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:34.358+0000 7ffac27fc700 1 -- 172.21.15.67:0/4059094670 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffabc021640 con 0x7ffacc101ea0 2024-04-22T23:20:34.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:34.358+0000 7ffad3717700 1 -- 172.21.15.67:0/4059094670 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffacc073f70 con 0x7ffacc101ea0 2024-04-22T23:20:34.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:34.360+0000 7ffab7fff700 1 -- 172.21.15.67:0/4059094670 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffacc109800 con 0x7ffacc101ea0 2024-04-22T23:20:34.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:34.360+0000 7ffac27fc700 1 -- 172.21.15.67:0/4059094670 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7ffabc00fab0 con 0x7ffacc101ea0 2024-04-22T23:20:34.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:34.360+0000 7ffac27fc700 1 --2- 172.21.15.67:0/4059094670 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ffab806c4e0 0x7ffab806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:34.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:34.361+0000 7ffac27fc700 1 -- 172.21.15.67:0/4059094670 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7ffabc08cac0 con 0x7ffacc101ea0 2024-04-22T23:20:34.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:34.361+0000 7ffad14b3700 1 --2- 172.21.15.67:0/4059094670 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ffab806c4e0 0x7ffab806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:34.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:34.361+0000 7ffad14b3700 1 --2- 172.21.15.67:0/4059094670 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ffab806c4e0 0x7ffab806e9a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ffacc101d00 tx=0x7ffac800b410 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:34.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:34.365+0000 7ffac27fc700 1 -- 172.21.15.67:0/4059094670 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ffabc05ac90 con 0x7ffacc101ea0 2024-04-22T23:20:34.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:34.623+0000 7ffab7fff700 1 -- 172.21.15.67:0/4059094670 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ffacc109990 con 0x7ffacc101ea0 2024-04-22T23:20:34.626 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:34 smithi067 ceph-mon[26453]: Deploying daemon mds.cephfs.smithi134.vuxbsj on smithi134 2024-04-22T23:20:34.626 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:34 smithi067 ceph-mon[26453]: pgmap v181: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s wr, 7 op/s 2024-04-22T23:20:34.626 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:34 smithi067 ceph-mon[26453]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-22T23:20:34.626 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:34 smithi067 ceph-mon[26453]: Cluster is now healthy 2024-04-22T23:20:34.626 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:34 smithi067 ceph-mon[26453]: mds.? [v2:172.21.15.67:6828/3606151860,v1:172.21.15.67:6829/3606151860] up:boot 2024-04-22T23:20:34.626 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:34 smithi067 ceph-mon[26453]: fsmap cephfs:2 {0=cephfs.smithi067.aqlxpv=up:active,1=cephfs.smithi134.owipad=up:active} 1 up:standby 2024-04-22T23:20:34.626 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:34 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.xyazmh"}]: dispatch 2024-04-22T23:20:34.627 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:34 smithi067 ceph-mon[26453]: fsmap cephfs:2 {0=cephfs.smithi067.aqlxpv=up:active,1=cephfs.smithi134.owipad=up:active} 1 up:standby-replay 2024-04-22T23:20:34.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:34 smithi134 ceph-mon[34609]: Deploying daemon mds.cephfs.smithi134.vuxbsj on smithi134 2024-04-22T23:20:34.729 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:34 smithi134 ceph-mon[34609]: pgmap v181: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s wr, 7 op/s 2024-04-22T23:20:34.729 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:34 smithi134 ceph-mon[34609]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-22T23:20:34.729 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:34 smithi134 ceph-mon[34609]: Cluster is now healthy 2024-04-22T23:20:34.729 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:34 smithi134 ceph-mon[34609]: mds.? [v2:172.21.15.67:6828/3606151860,v1:172.21.15.67:6829/3606151860] up:boot 2024-04-22T23:20:34.729 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:34 smithi134 ceph-mon[34609]: fsmap cephfs:2 {0=cephfs.smithi067.aqlxpv=up:active,1=cephfs.smithi134.owipad=up:active} 1 up:standby 2024-04-22T23:20:34.729 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:34 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.xyazmh"}]: dispatch 2024-04-22T23:20:34.729 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:34 smithi134 ceph-mon[34609]: fsmap cephfs:2 {0=cephfs.smithi067.aqlxpv=up:active,1=cephfs.smithi134.owipad=up:active} 1 up:standby-replay 2024-04-22T23:20:35.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:35.491+0000 7ffac27fc700 1 -- 172.21.15.67:0/4059094670 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v10) v1 ==== 168+0+0 (secure 0 0 0) 0x7ffacc109990 con 0x7ffacc101ea0 2024-04-22T23:20:35.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:35.494+0000 7ffab7fff700 1 -- 172.21.15.67:0/4059094670 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ffab806c4e0 msgr2=0x7ffab806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:35.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:35.494+0000 7ffab7fff700 1 --2- 172.21.15.67:0/4059094670 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ffab806c4e0 0x7ffab806e9a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ffacc101d00 tx=0x7ffac800b410 comp rx=0 tx=0).stop 2024-04-22T23:20:35.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:35.494+0000 7ffab7fff700 1 -- 172.21.15.67:0/4059094670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffacc101ea0 msgr2=0x7ffacc073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:35.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:35.494+0000 7ffab7fff700 1 --2- 172.21.15.67:0/4059094670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffacc101ea0 0x7ffacc073180 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ffabc006010 tx=0x7ffabc00bd70 comp rx=0 tx=0).stop 2024-04-22T23:20:35.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:35.494+0000 7ffab7fff700 1 -- 172.21.15.67:0/4059094670 shutdown_connections 2024-04-22T23:20:35.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:35.494+0000 7ffab7fff700 1 --2- 172.21.15.67:0/4059094670 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ffab806c4e0 0x7ffab806e9a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:35.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:35.494+0000 7ffab7fff700 1 --2- 172.21.15.67:0/4059094670 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffacc100ca0 0x7ffacc074b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:35.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:35.494+0000 7ffab7fff700 1 --2- 172.21.15.67:0/4059094670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffacc101ea0 0x7ffacc073180 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:35.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:35.494+0000 7ffab7fff700 1 -- 172.21.15.67:0/4059094670 >> 172.21.15.67:0/4059094670 conn(0x7ffacc0fc260 msgr2=0x7ffacc1050d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:35.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:35.494+0000 7ffab7fff700 1 -- 172.21.15.67:0/4059094670 shutdown_connections 2024-04-22T23:20:35.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:35.494+0000 7ffab7fff700 1 -- 172.21.15.67:0/4059094670 wait complete. 2024-04-22T23:20:35.496 INFO:teuthology.orchestra.run.smithi067.stderr:inline data enabled 2024-04-22T23:20:35.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:35 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/4059094670' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T23:20:35.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:35 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/4059094670' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T23:20:36.177 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T23:20:36.188 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi067.front.sepia.ceph.com 2024-04-22T23:20:36.188 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-22T23:20:36.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:36 smithi134 ceph-mon[34609]: pgmap v182: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 682 B/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-22T23:20:36.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:36 smithi134 ceph-mon[34609]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-04-22T23:20:36.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:36 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/4059094670' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-04-22T23:20:36.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:36 smithi134 ceph-mon[34609]: mds.? [v2:172.21.15.134:6826/1954821676,v1:172.21.15.134:6827/1954821676] up:boot 2024-04-22T23:20:36.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:36 smithi134 ceph-mon[34609]: fsmap cephfs:2 {0=cephfs.smithi067.aqlxpv=up:active,1=cephfs.smithi134.owipad=up:active} 1 up:standby-replay 1 up:standby 2024-04-22T23:20:36.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:36 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.vuxbsj"}]: dispatch 2024-04-22T23:20:36.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:36 smithi134 ceph-mon[34609]: fsmap cephfs:2 {0=cephfs.smithi067.aqlxpv=up:active,1=cephfs.smithi134.owipad=up:active} 2 up:standby-replay 2024-04-22T23:20:36.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:36 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:36.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:36 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:36.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:36 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:36.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:36 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:36.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:36 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:36.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:36 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:20:36.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:36 smithi067 ceph-mon[26453]: pgmap v182: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 682 B/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-22T23:20:36.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:36 smithi067 ceph-mon[26453]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-04-22T23:20:36.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:36 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/4059094670' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-04-22T23:20:36.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:36 smithi067 ceph-mon[26453]: mds.? [v2:172.21.15.134:6826/1954821676,v1:172.21.15.134:6827/1954821676] up:boot 2024-04-22T23:20:36.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:36 smithi067 ceph-mon[26453]: fsmap cephfs:2 {0=cephfs.smithi067.aqlxpv=up:active,1=cephfs.smithi134.owipad=up:active} 1 up:standby-replay 1 up:standby 2024-04-22T23:20:36.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:36 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.vuxbsj"}]: dispatch 2024-04-22T23:20:36.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:36 smithi067 ceph-mon[26453]: fsmap cephfs:2 {0=cephfs.smithi067.aqlxpv=up:active,1=cephfs.smithi134.owipad=up:active} 2 up:standby-replay 2024-04-22T23:20:36.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:36 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:36.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:36 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:36.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:36 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:36.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:36 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:36.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:36 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:36.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:36 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:20:37.052 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:20:38.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:38 smithi134 ceph-mon[34609]: pgmap v183: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-04-22T23:20:38.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:38 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:38.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:38 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:20:38.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:38 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:38.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:38 smithi134 ceph-mon[34609]: mds.? [v2:172.21.15.67:6828/3606151860,v1:172.21.15.67:6829/3606151860] up:standby-replay 2024-04-22T23:20:38.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:38 smithi134 ceph-mon[34609]: Dropping low affinity active daemon mds.cephfs.smithi067.aqlxpv in favor of higher affinity standby. 2024-04-22T23:20:38.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:38 smithi134 ceph-mon[34609]: Replacing daemon mds.cephfs.smithi067.aqlxpv as rank 0 with standby daemon mds.cephfs.smithi067.xyazmh 2024-04-22T23:20:38.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:38 smithi134 ceph-mon[34609]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T23:20:38.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:38 smithi134 ceph-mon[34609]: fsmap cephfs:2 {0=cephfs.smithi067.aqlxpv=up:active,1=cephfs.smithi134.owipad=up:active} 2 up:standby-replay 2024-04-22T23:20:38.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:38 smithi134 ceph-mon[34609]: osdmap e46: 6 total, 6 up, 6 in 2024-04-22T23:20:38.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:38 smithi134 ceph-mon[34609]: fsmap cephfs:2/2 {0=cephfs.smithi067.xyazmh=up:replay,1=cephfs.smithi134.owipad=up:active} 1 up:standby-replay 2024-04-22T23:20:38.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:38 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:38.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:38 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:38.753 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:38 smithi067 ceph-mon[26453]: pgmap v183: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-04-22T23:20:38.753 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:38 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:38.753 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:38 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:20:38.753 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:38 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:38.753 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:38 smithi067 ceph-mon[26453]: mds.? [v2:172.21.15.67:6828/3606151860,v1:172.21.15.67:6829/3606151860] up:standby-replay 2024-04-22T23:20:38.753 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:38 smithi067 ceph-mon[26453]: Dropping low affinity active daemon mds.cephfs.smithi067.aqlxpv in favor of higher affinity standby. 2024-04-22T23:20:38.753 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:38 smithi067 ceph-mon[26453]: Replacing daemon mds.cephfs.smithi067.aqlxpv as rank 0 with standby daemon mds.cephfs.smithi067.xyazmh 2024-04-22T23:20:38.753 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:38 smithi067 ceph-mon[26453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T23:20:38.753 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:38 smithi067 ceph-mon[26453]: fsmap cephfs:2 {0=cephfs.smithi067.aqlxpv=up:active,1=cephfs.smithi134.owipad=up:active} 2 up:standby-replay 2024-04-22T23:20:38.753 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:38 smithi067 ceph-mon[26453]: osdmap e46: 6 total, 6 up, 6 in 2024-04-22T23:20:38.753 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:38 smithi067 ceph-mon[26453]: fsmap cephfs:2/2 {0=cephfs.smithi067.xyazmh=up:replay,1=cephfs.smithi134.owipad=up:active} 1 up:standby-replay 2024-04-22T23:20:38.754 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:38 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:38.754 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:38 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:39.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:38.953+0000 7f1346150700 1 -- 172.21.15.67:0/2764679713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1340100e10 msgr2=0x7f1340101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:39.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:38.953+0000 7f1346150700 1 --2- 172.21.15.67:0/2764679713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1340100e10 0x7f1340101270 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1328009910 tx=0x7f1328009c20 comp rx=0 tx=0).stop 2024-04-22T23:20:39.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:38.953+0000 7f1346150700 1 -- 172.21.15.67:0/2764679713 shutdown_connections 2024-04-22T23:20:39.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:38.953+0000 7f1346150700 1 --2- 172.21.15.67:0/2764679713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1340100e10 0x7f1340101270 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:39.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:38.953+0000 7f1346150700 1 --2- 172.21.15.67:0/2764679713 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13400ffc10 0x7f1340100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:39.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:38.953+0000 7f1346150700 1 -- 172.21.15.67:0/2764679713 >> 172.21.15.67:0/2764679713 conn(0x7f13400fb190 msgr2=0x7f13400fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:39.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:38.953+0000 7f1346150700 1 -- 172.21.15.67:0/2764679713 shutdown_connections 2024-04-22T23:20:39.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:38.953+0000 7f1346150700 1 -- 172.21.15.67:0/2764679713 wait complete. 2024-04-22T23:20:39.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:38.954+0000 7f1346150700 1 Processor -- start 2024-04-22T23:20:39.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:38.954+0000 7f1346150700 1 -- start start 2024-04-22T23:20:39.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:38.954+0000 7f1346150700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13400ffc10 0x7f1340069100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:39.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:38.954+0000 7f1346150700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1340100e10 0x7f134019b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:39.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:38.954+0000 7f1346150700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134019b7b0 con 0x7f13400ffc10 2024-04-22T23:20:39.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:38.954+0000 7f1346150700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134019b920 con 0x7f1340100e10 2024-04-22T23:20:39.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:38.954+0000 7f133f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13400ffc10 0x7f1340069100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:39.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:38.954+0000 7f133f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13400ffc10 0x7f1340069100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43350/0 (socket says 172.21.15.67:43350) 2024-04-22T23:20:39.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:38.954+0000 7f133f7fe700 1 -- 172.21.15.67:0/4066066843 learned_addr learned my addr 172.21.15.67:0/4066066843 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:20:39.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:38.954+0000 7f133effd700 1 --2- 172.21.15.67:0/4066066843 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1340100e10 0x7f134019b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:39.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:38.955+0000 7f133f7fe700 1 -- 172.21.15.67:0/4066066843 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1340100e10 msgr2=0x7f134019b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:39.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:38.955+0000 7f133f7fe700 1 --2- 172.21.15.67:0/4066066843 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1340100e10 0x7f134019b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:39.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:38.955+0000 7f133f7fe700 1 -- 172.21.15.67:0/4066066843 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13280095f0 con 0x7f13400ffc10 2024-04-22T23:20:39.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:38.955+0000 7f133f7fe700 1 --2- 172.21.15.67:0/4066066843 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13400ffc10 0x7f1340069100 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f1330007e60 tx=0x7f133000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:39.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:38.955+0000 7f133cff9700 1 -- 172.21.15.67:0/4066066843 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f133000dda0 con 0x7f13400ffc10 2024-04-22T23:20:39.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:38.955+0000 7f133cff9700 1 -- 172.21.15.67:0/4066066843 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f133000f040 con 0x7f13400ffc10 2024-04-22T23:20:39.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:38.955+0000 7f1346150700 1 -- 172.21.15.67:0/4066066843 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f134019bc00 con 0x7f13400ffc10 2024-04-22T23:20:39.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:38.955+0000 7f133cff9700 1 -- 172.21.15.67:0/4066066843 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13300146a0 con 0x7f13400ffc10 2024-04-22T23:20:39.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:38.955+0000 7f1346150700 1 -- 172.21.15.67:0/4066066843 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f134019c120 con 0x7f13400ffc10 2024-04-22T23:20:39.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:38.956+0000 7f1346150700 1 -- 172.21.15.67:0/4066066843 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1340108700 con 0x7f13400ffc10 2024-04-22T23:20:39.270 INFO:teuthology.orchestra.run.smithi067.stdout:e14 2024-04-22T23:20:39.270 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:20:39.270 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:20:39.270 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:20:39.270 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:20:39.270 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:20:39.270 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:20:39.270 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 14 2024-04-22T23:20:39.270 INFO:teuthology.orchestra.run.smithi067.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T23:20:39.270 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:20:39.271 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:20:38.504641+0000 2024-04-22T23:20:39.271 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:20:39.271 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:20:39.271 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:20:39.271 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:20:39.271 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:20:39.271 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:20:39.271 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:20:39.271 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 46 2024-04-22T23:20:39.271 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:20:39.271 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:20:39.271 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:20:39.271 INFO:teuthology.orchestra.run.smithi067.stdout:up {0=14484,1=24273} 2024-04-22T23:20:39.271 INFO:teuthology.orchestra.run.smithi067.stdout:failed 2024-04-22T23:20:39.271 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:20:39.272 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:20:39.272 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:20:39.272 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:20:39.272 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:20:39.272 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:20:39.272 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:20:39.272 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:20:39.272 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{0:14484} state up:resolve seq 3 join_fscid=1 addr [v2:172.21.15.67:6828/3606151860,v1:172.21.15.67:6829/3606151860] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:20:39.272 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{1:24273} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.134:6824/535957142,v1:172.21.15.134:6825/535957142] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:20:39.272 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{1:24289} state up:standby-replay seq 1 addr [v2:172.21.15.134:6826/1954821676,v1:172.21.15.134:6827/1954821676] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:20:39.272 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:20:39.272 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:20:39.272 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:20:39.272 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:20:39.272 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:14496} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/3305304988,v1:172.21.15.67:6827/3305304988] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:20:39.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:38.957+0000 7f133cff9700 1 -- 172.21.15.67:0/4066066843 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f1330015400 con 0x7f13400ffc10 2024-04-22T23:20:39.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:38.957+0000 7f133cff9700 1 --2- 172.21.15.67:0/4066066843 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f132c06c4e0 0x7f132c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:39.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:38.957+0000 7f133effd700 1 --2- 172.21.15.67:0/4066066843 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f132c06c4e0 0x7f132c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:39.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:38.957+0000 7f133cff9700 1 -- 172.21.15.67:0/4066066843 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f133008af10 con 0x7f13400ffc10 2024-04-22T23:20:39.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:38.957+0000 7f133effd700 1 --2- 172.21.15.67:0/4066066843 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f132c06c4e0 0x7f132c06e9a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f132800b5c0 tx=0x7f1328005ee0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:39.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:38.961+0000 7f133cff9700 1 -- 172.21.15.67:0/4066066843 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f1330059250 con 0x7f13400ffc10 2024-04-22T23:20:39.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:39.217+0000 7f1346150700 1 -- 172.21.15.67:0/4066066843 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f13401088c0 con 0x7f13400ffc10 2024-04-22T23:20:39.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:39.218+0000 7f133cff9700 1 -- 172.21.15.67:0/4066066843 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 14 v14) v1 ==== 76+0+1849 (secure 0 0 0) 0x7f133001a750 con 0x7f13400ffc10 2024-04-22T23:20:39.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:39.220+0000 7f1346150700 1 -- 172.21.15.67:0/4066066843 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f132c06c4e0 msgr2=0x7f132c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:39.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:39.220+0000 7f1346150700 1 --2- 172.21.15.67:0/4066066843 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f132c06c4e0 0x7f132c06e9a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f132800b5c0 tx=0x7f1328005ee0 comp rx=0 tx=0).stop 2024-04-22T23:20:39.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:39.220+0000 7f1346150700 1 -- 172.21.15.67:0/4066066843 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13400ffc10 msgr2=0x7f1340069100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:39.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:39.220+0000 7f1346150700 1 --2- 172.21.15.67:0/4066066843 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13400ffc10 0x7f1340069100 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f1330007e60 tx=0x7f133000d3b0 comp rx=0 tx=0).stop 2024-04-22T23:20:39.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:39.220+0000 7f1346150700 1 -- 172.21.15.67:0/4066066843 shutdown_connections 2024-04-22T23:20:39.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:39.220+0000 7f1346150700 1 --2- 172.21.15.67:0/4066066843 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13400ffc10 0x7f1340069100 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:39.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:39.220+0000 7f1346150700 1 --2- 172.21.15.67:0/4066066843 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f132c06c4e0 0x7f132c06e9a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:39.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:39.220+0000 7f1346150700 1 --2- 172.21.15.67:0/4066066843 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1340100e10 0x7f134019b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:39.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:39.220+0000 7f1346150700 1 -- 172.21.15.67:0/4066066843 >> 172.21.15.67:0/4066066843 conn(0x7f13400fb190 msgr2=0x7f1340104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:39.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:39.220+0000 7f1346150700 1 -- 172.21.15.67:0/4066066843 shutdown_connections 2024-04-22T23:20:39.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:39.220+0000 7f1346150700 1 -- 172.21.15.67:0/4066066843 wait complete. 2024-04-22T23:20:39.274 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 14 2024-04-22T23:20:39.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:39 smithi067 ceph-mon[26453]: mds.? [v2:172.21.15.67:6826/3305304988,v1:172.21.15.67:6827/3305304988] up:boot 2024-04-22T23:20:39.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:39 smithi067 ceph-mon[26453]: mds.? [v2:172.21.15.134:6824/535957142,v1:172.21.15.134:6825/535957142] up:active 2024-04-22T23:20:39.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:39 smithi067 ceph-mon[26453]: mds.? [v2:172.21.15.67:6828/3606151860,v1:172.21.15.67:6829/3606151860] up:resolve 2024-04-22T23:20:39.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:39 smithi067 ceph-mon[26453]: fsmap cephfs:2/2 {0=cephfs.smithi067.xyazmh=up:resolve,1=cephfs.smithi134.owipad=up:active} 1 up:standby-replay 1 up:standby 2024-04-22T23:20:39.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:39 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.aqlxpv"}]: dispatch 2024-04-22T23:20:39.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:39 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/4066066843' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:20:39.906 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-04-22T23:20:39.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:39 smithi134 ceph-mon[34609]: mds.? [v2:172.21.15.67:6826/3305304988,v1:172.21.15.67:6827/3305304988] up:boot 2024-04-22T23:20:39.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:39 smithi134 ceph-mon[34609]: mds.? [v2:172.21.15.134:6824/535957142,v1:172.21.15.134:6825/535957142] up:active 2024-04-22T23:20:39.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:39 smithi134 ceph-mon[34609]: mds.? [v2:172.21.15.67:6828/3606151860,v1:172.21.15.67:6829/3606151860] up:resolve 2024-04-22T23:20:39.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:39 smithi134 ceph-mon[34609]: fsmap cephfs:2/2 {0=cephfs.smithi067.xyazmh=up:resolve,1=cephfs.smithi134.owipad=up:active} 1 up:standby-replay 1 up:standby 2024-04-22T23:20:39.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:39 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.aqlxpv"}]: dispatch 2024-04-22T23:20:39.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:39 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/4066066843' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:20:40.643 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:20:40.767 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:40 smithi067 ceph-mon[26453]: pgmap v185: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2024-04-22T23:20:40.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:40 smithi067 ceph-mon[26453]: mds.? [v2:172.21.15.67:6828/3606151860,v1:172.21.15.67:6829/3606151860] up:reconnect 2024-04-22T23:20:40.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:40 smithi067 ceph-mon[26453]: mds.? [v2:172.21.15.134:6826/1954821676,v1:172.21.15.134:6827/1954821676] up:standby-replay 2024-04-22T23:20:40.768 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:40 smithi067 ceph-mon[26453]: fsmap cephfs:2/2 {0=cephfs.smithi067.xyazmh=up:reconnect,1=cephfs.smithi134.owipad=up:active} 1 up:standby-replay 1 up:standby 2024-04-22T23:20:40.896 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:40 smithi134 ceph-mon[34609]: pgmap v185: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2024-04-22T23:20:40.896 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:40 smithi134 ceph-mon[34609]: mds.? [v2:172.21.15.67:6828/3606151860,v1:172.21.15.67:6829/3606151860] up:reconnect 2024-04-22T23:20:40.896 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:40 smithi134 ceph-mon[34609]: mds.? [v2:172.21.15.134:6826/1954821676,v1:172.21.15.134:6827/1954821676] up:standby-replay 2024-04-22T23:20:40.896 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:40 smithi134 ceph-mon[34609]: fsmap cephfs:2/2 {0=cephfs.smithi067.xyazmh=up:reconnect,1=cephfs.smithi134.owipad=up:active} 1 up:standby-replay 1 up:standby 2024-04-22T23:20:41.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:41 smithi067 ceph-mon[26453]: mds.? [v2:172.21.15.67:6828/3606151860,v1:172.21.15.67:6829/3606151860] up:rejoin 2024-04-22T23:20:41.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:41 smithi067 ceph-mon[26453]: fsmap cephfs:2/2 {0=cephfs.smithi067.xyazmh=up:rejoin,1=cephfs.smithi134.owipad=up:active} 1 up:standby-replay 1 up:standby 2024-04-22T23:20:41.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:41 smithi067 ceph-mon[26453]: daemon mds.cephfs.smithi067.xyazmh is now active in filesystem cephfs as rank 0 2024-04-22T23:20:41.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:41 smithi134 ceph-mon[34609]: mds.? [v2:172.21.15.67:6828/3606151860,v1:172.21.15.67:6829/3606151860] up:rejoin 2024-04-22T23:20:41.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:41 smithi134 ceph-mon[34609]: fsmap cephfs:2/2 {0=cephfs.smithi067.xyazmh=up:rejoin,1=cephfs.smithi134.owipad=up:active} 1 up:standby-replay 1 up:standby 2024-04-22T23:20:41.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:41 smithi134 ceph-mon[34609]: daemon mds.cephfs.smithi067.xyazmh is now active in filesystem cephfs as rank 0 2024-04-22T23:20:42.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:42 smithi067 ceph-mon[26453]: pgmap v186: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T23:20:42.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:42 smithi067 ceph-mon[26453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T23:20:42.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:42 smithi067 ceph-mon[26453]: mds.? [v2:172.21.15.67:6828/3606151860,v1:172.21.15.67:6829/3606151860] up:active 2024-04-22T23:20:42.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:42 smithi067 ceph-mon[26453]: fsmap cephfs:2 {0=cephfs.smithi067.xyazmh=up:active,1=cephfs.smithi134.owipad=up:active} 1 up:standby-replay 1 up:standby 2024-04-22T23:20:42.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:42 smithi067 ceph-mon[26453]: fsmap cephfs:2 {0=cephfs.smithi067.xyazmh=up:active,1=cephfs.smithi134.owipad=up:active} 2 up:standby-replay 2024-04-22T23:20:42.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:42 smithi134 ceph-mon[34609]: pgmap v186: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-22T23:20:42.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:42 smithi134 ceph-mon[34609]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T23:20:42.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:42 smithi134 ceph-mon[34609]: mds.? [v2:172.21.15.67:6828/3606151860,v1:172.21.15.67:6829/3606151860] up:active 2024-04-22T23:20:42.997 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:42 smithi134 ceph-mon[34609]: fsmap cephfs:2 {0=cephfs.smithi067.xyazmh=up:active,1=cephfs.smithi134.owipad=up:active} 1 up:standby-replay 1 up:standby 2024-04-22T23:20:42.997 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:42 smithi134 ceph-mon[34609]: fsmap cephfs:2 {0=cephfs.smithi067.xyazmh=up:active,1=cephfs.smithi134.owipad=up:active} 2 up:standby-replay 2024-04-22T23:20:43.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.411+0000 7f734169a700 1 -- 172.21.15.67:0/2086572012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f733c101eb0 msgr2=0x7f733c102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:43.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.411+0000 7f734169a700 1 --2- 172.21.15.67:0/2086572012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f733c101eb0 0x7f733c102310 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7324009910 tx=0x7f7324009c20 comp rx=0 tx=0).stop 2024-04-22T23:20:43.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.411+0000 7f734169a700 1 -- 172.21.15.67:0/2086572012 shutdown_connections 2024-04-22T23:20:43.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.411+0000 7f734169a700 1 --2- 172.21.15.67:0/2086572012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f733c101eb0 0x7f733c102310 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:43.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.411+0000 7f734169a700 1 --2- 172.21.15.67:0/2086572012 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f733c100cb0 0x7f733c1010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:43.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.411+0000 7f734169a700 1 -- 172.21.15.67:0/2086572012 >> 172.21.15.67:0/2086572012 conn(0x7f733c0fc230 msgr2=0x7f733c0fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:43.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.411+0000 7f734169a700 1 -- 172.21.15.67:0/2086572012 shutdown_connections 2024-04-22T23:20:43.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.411+0000 7f734169a700 1 -- 172.21.15.67:0/2086572012 wait complete. 2024-04-22T23:20:43.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.412+0000 7f734169a700 1 Processor -- start 2024-04-22T23:20:43.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.412+0000 7f734169a700 1 -- start start 2024-04-22T23:20:43.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.412+0000 7f734169a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f733c100cb0 0x7f733c196550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:43.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.412+0000 7f734169a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f733c101eb0 0x7f733c196a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:43.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.412+0000 7f734169a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733c1970b0 con 0x7f733c101eb0 2024-04-22T23:20:43.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.412+0000 7f734169a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733c1971f0 con 0x7f733c100cb0 2024-04-22T23:20:43.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.412+0000 7f733a7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f733c101eb0 0x7f733c196a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:43.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.412+0000 7f733a7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f733c101eb0 0x7f733c196a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43370/0 (socket says 172.21.15.67:43370) 2024-04-22T23:20:43.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.412+0000 7f733a7fc700 1 -- 172.21.15.67:0/2455825301 learned_addr learned my addr 172.21.15.67:0/2455825301 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:20:43.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.413+0000 7f733affd700 1 --2- 172.21.15.67:0/2455825301 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f733c100cb0 0x7f733c196550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:43.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.413+0000 7f733a7fc700 1 -- 172.21.15.67:0/2455825301 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f733c100cb0 msgr2=0x7f733c196550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:43.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.413+0000 7f733a7fc700 1 --2- 172.21.15.67:0/2455825301 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f733c100cb0 0x7f733c196550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:43.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.413+0000 7f733a7fc700 1 -- 172.21.15.67:0/2455825301 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73240095f0 con 0x7f733c101eb0 2024-04-22T23:20:43.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.413+0000 7f733affd700 1 --2- 172.21.15.67:0/2455825301 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f733c100cb0 0x7f733c196550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:20:43.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.413+0000 7f733a7fc700 1 --2- 172.21.15.67:0/2455825301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f733c101eb0 0x7f733c196a90 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7324006010 tx=0x7f732400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:43.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.413+0000 7f7333fff700 1 -- 172.21.15.67:0/2455825301 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f732401d070 con 0x7f733c101eb0 2024-04-22T23:20:43.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.413+0000 7f7333fff700 1 -- 172.21.15.67:0/2455825301 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f732400f460 con 0x7f733c101eb0 2024-04-22T23:20:43.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.413+0000 7f734169a700 1 -- 172.21.15.67:0/2455825301 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f733c19bc40 con 0x7f733c101eb0 2024-04-22T23:20:43.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.413+0000 7f7333fff700 1 -- 172.21.15.67:0/2455825301 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7324021640 con 0x7f733c101eb0 2024-04-22T23:20:43.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.413+0000 7f734169a700 1 -- 172.21.15.67:0/2455825301 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f733c19c130 con 0x7f733c101eb0 2024-04-22T23:20:43.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.414+0000 7f734169a700 1 -- 172.21.15.67:0/2455825301 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f733c1098b0 con 0x7f733c101eb0 2024-04-22T23:20:43.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.414+0000 7f7333fff700 1 -- 172.21.15.67:0/2455825301 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f732400fab0 con 0x7f733c101eb0 2024-04-22T23:20:43.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.415+0000 7f7333fff700 1 --2- 172.21.15.67:0/2455825301 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f732806c4e0 0x7f732806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:43.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.415+0000 7f7333fff700 1 -- 172.21.15.67:0/2455825301 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f732408cca0 con 0x7f733c101eb0 2024-04-22T23:20:43.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.415+0000 7f733affd700 1 --2- 172.21.15.67:0/2455825301 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f732806c4e0 0x7f732806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:43.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.415+0000 7f733affd700 1 --2- 172.21.15.67:0/2455825301 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f732806c4e0 0x7f732806e9a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f733c101d10 tx=0x7f732c008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:43.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.420+0000 7f7333fff700 1 -- 172.21.15.67:0/2455825301 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f7324091070 con 0x7f733c101eb0 2024-04-22T23:20:43.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.667+0000 7f734169a700 1 -- 172.21.15.67:0/2455825301 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f733c19c740 con 0x7f733c101eb0 2024-04-22T23:20:43.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.667+0000 7f7333fff700 1 -- 172.21.15.67:0/2455825301 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4773 (secure 0 0 0) 0x7f732405af30 con 0x7f733c101eb0 2024-04-22T23:20:43.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.669+0000 7f734169a700 1 -- 172.21.15.67:0/2455825301 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f732806c4e0 msgr2=0x7f732806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:43.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.669+0000 7f734169a700 1 --2- 172.21.15.67:0/2455825301 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f732806c4e0 0x7f732806e9a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f733c101d10 tx=0x7f732c008040 comp rx=0 tx=0).stop 2024-04-22T23:20:43.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.669+0000 7f734169a700 1 -- 172.21.15.67:0/2455825301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f733c101eb0 msgr2=0x7f733c196a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:43.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.669+0000 7f734169a700 1 --2- 172.21.15.67:0/2455825301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f733c101eb0 0x7f733c196a90 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7324006010 tx=0x7f732400be20 comp rx=0 tx=0).stop 2024-04-22T23:20:43.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.669+0000 7f734169a700 1 -- 172.21.15.67:0/2455825301 shutdown_connections 2024-04-22T23:20:43.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.669+0000 7f734169a700 1 --2- 172.21.15.67:0/2455825301 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f732806c4e0 0x7f732806e9a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:43.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.669+0000 7f734169a700 1 --2- 172.21.15.67:0/2455825301 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f733c100cb0 0x7f733c196550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:43.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.669+0000 7f734169a700 1 --2- 172.21.15.67:0/2455825301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f733c101eb0 0x7f733c196a90 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:43.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.670+0000 7f734169a700 1 -- 172.21.15.67:0/2455825301 >> 172.21.15.67:0/2455825301 conn(0x7f733c0fc230 msgr2=0x7f733c1050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:43.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.670+0000 7f734169a700 1 -- 172.21.15.67:0/2455825301 shutdown_connections 2024-04-22T23:20:43.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:43.670+0000 7f734169a700 1 -- 172.21.15.67:0/2455825301 wait complete. 2024-04-22T23:20:43.672 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 18 2024-04-22T23:20:43.681 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:20:44.253 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-04-22T23:20:44.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:44 smithi134 ceph-mon[34609]: pgmap v187: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-22T23:20:44.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:44 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:44.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:44 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:44.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:44 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:20:44.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:44 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:20:44.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:44 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:44.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:44 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/2455825301' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T23:20:44.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:44 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:20:44.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:44 smithi067 ceph-mon[26453]: pgmap v187: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-22T23:20:44.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:44 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:44.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:44 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:44.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:44 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:20:44.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:44 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:20:44.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:44 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:44.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:44 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/2455825301' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T23:20:44.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:44 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:20:45.318 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:20:46.422 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:46 smithi067 ceph-mon[26453]: pgmap v188: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 9.0 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-22T23:20:46.423 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:46 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:46.423 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:46 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:46.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:46 smithi134 ceph-mon[34609]: pgmap v188: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 9.0 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-22T23:20:46.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:46 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:46.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:46 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:47.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.482+0000 7fa103b7b700 1 -- 172.21.15.67:0/1345940292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0fc101840 msgr2=0x7fa0fc103c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:47.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.482+0000 7fa103b7b700 1 --2- 172.21.15.67:0/1345940292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0fc101840 0x7fa0fc103c30 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8009910 tx=0x7fa0f8009c20 comp rx=0 tx=0).stop 2024-04-22T23:20:47.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.482+0000 7fa103b7b700 1 -- 172.21.15.67:0/1345940292 shutdown_connections 2024-04-22T23:20:47.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.482+0000 7fa103b7b700 1 --2- 172.21.15.67:0/1345940292 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0fc104170 0x7fa0fc106560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:47.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.482+0000 7fa103b7b700 1 --2- 172.21.15.67:0/1345940292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0fc101840 0x7fa0fc103c30 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:47.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.482+0000 7fa103b7b700 1 -- 172.21.15.67:0/1345940292 >> 172.21.15.67:0/1345940292 conn(0x7fa0fc0fb190 msgr2=0x7fa0fc0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:47.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.482+0000 7fa103b7b700 1 -- 172.21.15.67:0/1345940292 shutdown_connections 2024-04-22T23:20:47.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.482+0000 7fa103b7b700 1 -- 172.21.15.67:0/1345940292 wait complete. 2024-04-22T23:20:47.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.483+0000 7fa103b7b700 1 Processor -- start 2024-04-22T23:20:47.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.483+0000 7fa103b7b700 1 -- start start 2024-04-22T23:20:47.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.483+0000 7fa103b7b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0fc101840 0x7fa0fc196450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:47.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.483+0000 7fa103b7b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0fc104170 0x7fa0fc196990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:47.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.483+0000 7fa103b7b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0fc196f20 con 0x7fa0fc101840 2024-04-22T23:20:47.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.483+0000 7fa103b7b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0fc197060 con 0x7fa0fc104170 2024-04-22T23:20:47.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.483+0000 7fa101917700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0fc101840 0x7fa0fc196450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:47.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.483+0000 7fa101917700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0fc101840 0x7fa0fc196450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43388/0 (socket says 172.21.15.67:43388) 2024-04-22T23:20:47.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.483+0000 7fa101917700 1 -- 172.21.15.67:0/2360212959 learned_addr learned my addr 172.21.15.67:0/2360212959 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:20:47.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.483+0000 7fa101116700 1 --2- 172.21.15.67:0/2360212959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0fc104170 0x7fa0fc196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:47.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.483+0000 7fa101917700 1 -- 172.21.15.67:0/2360212959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0fc104170 msgr2=0x7fa0fc196990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:47.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.483+0000 7fa101917700 1 --2- 172.21.15.67:0/2360212959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0fc104170 0x7fa0fc196990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:47.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.483+0000 7fa101917700 1 -- 172.21.15.67:0/2360212959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f80095f0 con 0x7fa0fc101840 2024-04-22T23:20:47.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.483+0000 7fa101116700 1 --2- 172.21.15.67:0/2360212959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0fc104170 0x7fa0fc196990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:20:47.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.483+0000 7fa101917700 1 --2- 172.21.15.67:0/2360212959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0fc101840 0x7fa0fc196450 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa0f800b5c0 tx=0x7fa0f800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:47.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.484+0000 7fa0f2ffd700 1 -- 172.21.15.67:0/2360212959 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0f801d070 con 0x7fa0fc101840 2024-04-22T23:20:47.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.484+0000 7fa0f2ffd700 1 -- 172.21.15.67:0/2360212959 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa0f8022470 con 0x7fa0fc101840 2024-04-22T23:20:47.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.484+0000 7fa103b7b700 1 -- 172.21.15.67:0/2360212959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0fc19bac0 con 0x7fa0fc101840 2024-04-22T23:20:47.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.484+0000 7fa0f2ffd700 1 -- 172.21.15.67:0/2360212959 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0f800f690 con 0x7fa0fc101840 2024-04-22T23:20:47.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.484+0000 7fa103b7b700 1 -- 172.21.15.67:0/2360212959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0fc19bf80 con 0x7fa0fc101840 2024-04-22T23:20:47.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.485+0000 7fa103b7b700 1 -- 172.21.15.67:0/2360212959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0fc109810 con 0x7fa0fc101840 2024-04-22T23:20:47.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.485+0000 7fa0f2ffd700 1 -- 172.21.15.67:0/2360212959 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fa0f8004b70 con 0x7fa0fc101840 2024-04-22T23:20:47.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.486+0000 7fa0f2ffd700 1 --2- 172.21.15.67:0/2360212959 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa0e806c4e0 0x7fa0e806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:47.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.486+0000 7fa101116700 1 --2- 172.21.15.67:0/2360212959 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa0e806c4e0 0x7fa0e806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:47.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.486+0000 7fa0f2ffd700 1 -- 172.21.15.67:0/2360212959 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fa0f808ce30 con 0x7fa0fc101840 2024-04-22T23:20:47.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.486+0000 7fa101116700 1 --2- 172.21.15.67:0/2360212959 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa0e806c4e0 0x7fa0e806e9a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec005fd0 tx=0x7fa0ec005e20 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:47.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.490+0000 7fa0f2ffd700 1 -- 172.21.15.67:0/2360212959 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa0f805b170 con 0x7fa0fc101840 2024-04-22T23:20:47.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.778+0000 7fa103b7b700 1 -- 172.21.15.67:0/2360212959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7fa0fc1099d0 con 0x7fa0fc101840 2024-04-22T23:20:47.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.778+0000 7fa0f2ffd700 1 -- 172.21.15.67:0/2360212959 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v18) v1 ==== 78+0+83 (secure 0 0 0) 0x7fa0f805fe60 con 0x7fa0fc101840 2024-04-22T23:20:47.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.780+0000 7fa103b7b700 1 -- 172.21.15.67:0/2360212959 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa0e806c4e0 msgr2=0x7fa0e806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:47.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.780+0000 7fa103b7b700 1 --2- 172.21.15.67:0/2360212959 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa0e806c4e0 0x7fa0e806e9a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec005fd0 tx=0x7fa0ec005e20 comp rx=0 tx=0).stop 2024-04-22T23:20:47.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.780+0000 7fa103b7b700 1 -- 172.21.15.67:0/2360212959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0fc101840 msgr2=0x7fa0fc196450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:47.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.780+0000 7fa103b7b700 1 --2- 172.21.15.67:0/2360212959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0fc101840 0x7fa0fc196450 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa0f800b5c0 tx=0x7fa0f800be20 comp rx=0 tx=0).stop 2024-04-22T23:20:47.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.780+0000 7fa103b7b700 1 -- 172.21.15.67:0/2360212959 shutdown_connections 2024-04-22T23:20:47.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.780+0000 7fa103b7b700 1 --2- 172.21.15.67:0/2360212959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0fc101840 0x7fa0fc196450 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:47.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.780+0000 7fa103b7b700 1 --2- 172.21.15.67:0/2360212959 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa0e806c4e0 0x7fa0e806e9a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:47.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.780+0000 7fa103b7b700 1 --2- 172.21.15.67:0/2360212959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0fc104170 0x7fa0fc196990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:47.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.781+0000 7fa103b7b700 1 -- 172.21.15.67:0/2360212959 >> 172.21.15.67:0/2360212959 conn(0x7fa0fc0fb190 msgr2=0x7fa0fc0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:47.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.781+0000 7fa103b7b700 1 -- 172.21.15.67:0/2360212959 shutdown_connections 2024-04-22T23:20:47.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:47.781+0000 7fa103b7b700 1 -- 172.21.15.67:0/2360212959 wait complete. 2024-04-22T23:20:47.792 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:20:48.333 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-04-22T23:20:48.346 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T23:20:48.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.494+0000 7efd63836640 1 -- 172.21.15.67:0/526162765 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd5c1000a0 msgr2=0x7efd5c1004c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:48.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.494+0000 7efd63836640 1 --2- 172.21.15.67:0/526162765 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd5c1000a0 0x7efd5c1004c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7efd4c009880 tx=0x7efd4c02bce0 comp rx=0 tx=0).stop 2024-04-22T23:20:48.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.494+0000 7efd63836640 1 -- 172.21.15.67:0/526162765 shutdown_connections 2024-04-22T23:20:48.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.494+0000 7efd63836640 1 --2- 172.21.15.67:0/526162765 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd5c101290 0x7efd5c101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:48.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.494+0000 7efd63836640 1 --2- 172.21.15.67:0/526162765 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd5c1000a0 0x7efd5c1004c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:48.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.494+0000 7efd63836640 1 -- 172.21.15.67:0/526162765 >> 172.21.15.67:0/526162765 conn(0x7efd5c0fb610 msgr2=0x7efd5c0fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:48.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.494+0000 7efd63836640 1 -- 172.21.15.67:0/526162765 shutdown_connections 2024-04-22T23:20:48.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.494+0000 7efd63836640 1 -- 172.21.15.67:0/526162765 wait complete. 2024-04-22T23:20:48.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.495+0000 7efd63836640 1 Processor -- start 2024-04-22T23:20:48.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.495+0000 7efd63836640 1 -- start start 2024-04-22T23:20:48.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.495+0000 7efd63836640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd5c1000a0 0x7efd5c074050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:48.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.495+0000 7efd63836640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd5c101290 0x7efd5c072690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:48.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.495+0000 7efd63836640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c072bd0 con 0x7efd5c101290 2024-04-22T23:20:48.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.495+0000 7efd63836640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c072d40 con 0x7efd5c1000a0 2024-04-22T23:20:48.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.495+0000 7efd60dd1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd5c101290 0x7efd5c072690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:48.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.495+0000 7efd60dd1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd5c101290 0x7efd5c072690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43412/0 (socket says 172.21.15.67:43412) 2024-04-22T23:20:48.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.495+0000 7efd60dd1640 1 -- 172.21.15.67:0/4287193508 learned_addr learned my addr 172.21.15.67:0/4287193508 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:20:48.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.495+0000 7efd60dd1640 1 -- 172.21.15.67:0/4287193508 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd5c1000a0 msgr2=0x7efd5c074050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:48.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.495+0000 7efd60dd1640 1 --2- 172.21.15.67:0/4287193508 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd5c1000a0 0x7efd5c074050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:48.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.495+0000 7efd60dd1640 1 -- 172.21.15.67:0/4287193508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd4c009560 con 0x7efd5c101290 2024-04-22T23:20:48.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.495+0000 7efd60dd1640 1 --2- 172.21.15.67:0/4287193508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd5c101290 0x7efd5c072690 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7efd5800d8d0 tx=0x7efd5800dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:48.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.496+0000 7efd527fc640 1 -- 172.21.15.67:0/4287193508 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd5800bea0 con 0x7efd5c101290 2024-04-22T23:20:48.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.496+0000 7efd63836640 1 -- 172.21.15.67:0/4287193508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd5c073020 con 0x7efd5c101290 2024-04-22T23:20:48.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.496+0000 7efd527fc640 1 -- 172.21.15.67:0/4287193508 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7efd58009c00 con 0x7efd5c101290 2024-04-22T23:20:48.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.496+0000 7efd527fc640 1 -- 172.21.15.67:0/4287193508 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd5800f710 con 0x7efd5c101290 2024-04-22T23:20:48.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.496+0000 7efd63836640 1 -- 172.21.15.67:0/4287193508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd5c19d910 con 0x7efd5c101290 2024-04-22T23:20:48.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.497+0000 7efd63836640 1 -- 172.21.15.67:0/4287193508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd5c108c80 con 0x7efd5c101290 2024-04-22T23:20:48.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.497+0000 7efd527fc640 1 -- 172.21.15.67:0/4287193508 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7efd5800f990 con 0x7efd5c101290 2024-04-22T23:20:48.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.497+0000 7efd527fc640 1 --2- 172.21.15.67:0/4287193508 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7efd4806c510 0x7efd4806e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:48.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.497+0000 7efd527fc640 1 -- 172.21.15.67:0/4287193508 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7efd5808b630 con 0x7efd5c101290 2024-04-22T23:20:48.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.498+0000 7efd615d2640 1 --2- 172.21.15.67:0/4287193508 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7efd4806c510 0x7efd4806e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:48.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.498+0000 7efd615d2640 1 --2- 172.21.15.67:0/4287193508 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7efd4806c510 0x7efd4806e9d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efd4c02c7c0 tx=0x7efd4c000bc0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:48.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.502+0000 7efd527fc640 1 -- 172.21.15.67:0/4287193508 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7efd58059840 con 0x7efd5c101290 2024-04-22T23:20:48.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.684+0000 7efd63836640 1 -- 172.21.15.67:0/4287193508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efd5c19daa0 con 0x7efd5c101290 2024-04-22T23:20:48.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.685+0000 7efd527fc640 1 -- 172.21.15.67:0/4287193508 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+15128 (secure 0 0 0) 0x7efd580593d0 con 0x7efd5c101290 2024-04-22T23:20:48.686 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:20:48.686 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":46,"fsid":"d46c346e-00fd-11ef-bc93-c7b262605968","created":"2024-04-22T23:13:18.687917+0000","modified":"2024-04-22T23:20:37.496165+0000","last_up_change":"2024-04-22T23:19:00.895098+0000","last_in_change":"2024-04-22T23:18:41.696935+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T23:17:47.912850+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-22T23:20:19.759956+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-22T23:20:20.252379+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"25c9acd1-5743-4dc0-aee0-c7bb8884c0a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":268472413},{"type":"v1","addr":"172.21.15.67:6803","nonce":268472413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":268472413},{"type":"v1","addr":"172.21.15.67:6805","nonce":268472413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":268472413},{"type":"v1","addr":"172.21.15.67:6809","nonce":268472413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":268472413},{"type":"v1","addr":"172.21.15.67:6807","nonce":268472413}]},"public_addr":"172.21.15.67:6803/268472413","cluster_addr":"172.21.15.67:6805/268472413","heartbeat_back_addr":"172.21.15.67:6809/268472413","heartbeat_front_addr":"172.21.15.67:6807/268472413","state":["exists","up"]},{"osd":1,"uuid":"8be83014-01af-44d3-ab2f-b27840506e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3382820294},{"type":"v1","addr":"172.21.15.67:6811","nonce":3382820294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3382820294},{"type":"v1","addr":"172.21.15.67:6813","nonce":3382820294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3382820294},{"type":"v1","addr":"172.21.15.67:6817","nonce":3382820294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3382820294},{"type":"v1","addr":"172.21.15.67:6815","nonce":3382820294}]},"public_addr":"172.21.15.67:6811/3382820294","cluster_addr":"172.21.15.67:6813/3382820294","heartbeat_back_addr":"172.21.15.67:6817/3382820294","heartbeat_front_addr":"172.21.15.67:6815/3382820294","state":["exists","up"]},{"osd":2,"uuid":"ab1cbcda-14dd-4c24-876b-25ad5d4d8563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":3835340581},{"type":"v1","addr":"172.21.15.67:6819","nonce":3835340581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3835340581},{"type":"v1","addr":"172.21.15.67:6821","nonce":3835340581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3835340581},{"type":"v1","addr":"172.21.15.67:6825","nonce":3835340581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3835340581},{"type":"v1","addr":"172.21.15.67:6823","nonce":3835340581}]},"public_addr":"172.21.15.67:6819/3835340581","cluster_addr":"172.21.15.67:6821/3835340581","heartbeat_back_addr":"172.21.15.67:6825/3835340581","heartbeat_front_addr":"172.21.15.67:6823/3835340581","state":["exists","up"]},{"osd":3,"uuid":"1a30b6c3-95bf-4b19-883b-1271ff359e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":2235786001},{"type":"v1","addr":"172.21.15.134:6801","nonce":2235786001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":2235786001},{"type":"v1","addr":"172.21.15.134:6803","nonce":2235786001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":2235786001},{"type":"v1","addr":"172.21.15.134:6807","nonce":2235786001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":2235786001},{"type":"v1","addr":"172.21.15.134:6805","nonce":2235786001}]},"public_addr":"172.21.15.134:6801/2235786001","cluster_addr":"172.21.15.134:6803/2235786001","heartbeat_back_addr":"172.21.15.134:6807/2235786001","heartbeat_front_addr":"172.21.15.134:6805/2235786001","state":["exists","up"]},{"osd":4,"uuid":"53368fc1-28fd-4622-bc9c-1d8901b06bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":668644810},{"type":"v1","addr":"172.21.15.134:6809","nonce":668644810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":668644810},{"type":"v1","addr":"172.21.15.134:6811","nonce":668644810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":668644810},{"type":"v1","addr":"172.21.15.134:6815","nonce":668644810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":668644810},{"type":"v1","addr":"172.21.15.134:6813","nonce":668644810}]},"public_addr":"172.21.15.134:6809/668644810","cluster_addr":"172.21.15.134:6811/668644810","heartbeat_back_addr":"172.21.15.134:6815/668644810","heartbeat_front_addr":"172.21.15.134:6813/668644810","state":["exists","up"]},{"osd":5,"uuid":"31a885f7-8f91-4b6a-922a-065753f70a95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3846690247},{"type":"v1","addr":"172.21.15.134:6817","nonce":3846690247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":3846690247},{"type":"v1","addr":"172.21.15.134:6819","nonce":3846690247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":3846690247},{"type":"v1","addr":"172.21.15.134:6823","nonce":3846690247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":3846690247},{"type":"v1","addr":"172.21.15.134:6821","nonce":3846690247}]},"public_addr":"172.21.15.134:6817/3846690247","cluster_addr":"172.21.15.134:6819/3846690247","heartbeat_back_addr":"172.21.15.134:6823/3846690247","heartbeat_front_addr":"172.21.15.134:6821/3846690247","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:16:51.179651+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:17:18.694683+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:17:45.368963+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:18:08.585046+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:18:33.412804+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:18:58.909758+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6800/3568049827":"2024-04-23T23:14:29.214977+0000","172.21.15.67:0/2665297739":"2024-04-23T23:14:29.214977+0000","172.21.15.67:0/3387250764":"2024-04-23T23:13:56.301645+0000","172.21.15.67:0/853292259":"2024-04-23T23:13:56.301645+0000","172.21.15.67:0/2052573085":"2024-04-23T23:14:29.214977+0000","172.21.15.67:6801/2451437776":"2024-04-23T23:13:56.301645+0000","172.21.15.67:6801/3568049827":"2024-04-23T23:14:29.214977+0000","172.21.15.67:6800/2451437776":"2024-04-23T23:13:56.301645+0000","172.21.15.67:0/4165654004":"2024-04-23T23:14:29.214977+0000","172.21.15.67:0/2137374599":"2024-04-23T23:15:37.396154+0000","172.21.15.67:6827/2381214817":"2024-04-23T23:20:37.496135+0000","172.21.15.67:6801/2590792187":"2024-04-23T23:15:37.396154+0000","172.21.15.67:0/1189311107":"2024-04-23T23:15:37.396154+0000","172.21.15.67:0/895378312":"2024-04-23T23:13:56.301645+0000","172.21.15.67:0/1125624760":"2024-04-23T23:15:37.396154+0000","172.21.15.67:6800/2590792187":"2024-04-23T23:15:37.396154+0000","172.21.15.67:6826/2381214817":"2024-04-23T23:20:37.496135+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T23:20:48.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.687+0000 7efd63836640 1 -- 172.21.15.67:0/4287193508 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7efd4806c510 msgr2=0x7efd4806e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:48.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.687+0000 7efd63836640 1 --2- 172.21.15.67:0/4287193508 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7efd4806c510 0x7efd4806e9d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efd4c02c7c0 tx=0x7efd4c000bc0 comp rx=0 tx=0).stop 2024-04-22T23:20:48.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.687+0000 7efd63836640 1 -- 172.21.15.67:0/4287193508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd5c101290 msgr2=0x7efd5c072690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:48.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.687+0000 7efd63836640 1 --2- 172.21.15.67:0/4287193508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd5c101290 0x7efd5c072690 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7efd5800d8d0 tx=0x7efd5800dd90 comp rx=0 tx=0).stop 2024-04-22T23:20:48.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.687+0000 7efd63836640 1 -- 172.21.15.67:0/4287193508 shutdown_connections 2024-04-22T23:20:48.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.687+0000 7efd63836640 1 --2- 172.21.15.67:0/4287193508 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7efd4806c510 0x7efd4806e9d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:48.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.687+0000 7efd63836640 1 --2- 172.21.15.67:0/4287193508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd5c101290 0x7efd5c072690 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:48.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.687+0000 7efd63836640 1 --2- 172.21.15.67:0/4287193508 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd5c1000a0 0x7efd5c074050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:48.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.687+0000 7efd63836640 1 -- 172.21.15.67:0/4287193508 >> 172.21.15.67:0/4287193508 conn(0x7efd5c0fb610 msgr2=0x7efd5c1044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:48.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.687+0000 7efd63836640 1 -- 172.21.15.67:0/4287193508 shutdown_connections 2024-04-22T23:20:48.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.687+0000 7efd63836640 1 -- 172.21.15.67:0/4287193508 wait complete. 2024-04-22T23:20:48.700 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T23:17:47.912850+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-22T23:20:19.759956+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-22T23:20:20.252379+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T23:20:48.701 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T23:20:48.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:48 smithi067 ceph-mon[26453]: pgmap v189: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 10 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-22T23:20:48.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:48 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/2360212959' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-04-22T23:20:48.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.808+0000 7fb1e297a640 1 -- 172.21.15.67:0/231078209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1dc0fefd0 msgr2=0x7fb1dc0ff3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:48.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.808+0000 7fb1e297a640 1 --2- 172.21.15.67:0/231078209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1dc0fefd0 0x7fb1dc0ff3f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc009880 tx=0x7fb1cc02bce0 comp rx=0 tx=0).stop 2024-04-22T23:20:48.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.808+0000 7fb1e297a640 1 -- 172.21.15.67:0/231078209 shutdown_connections 2024-04-22T23:20:48.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.808+0000 7fb1e297a640 1 --2- 172.21.15.67:0/231078209 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1dc1001c0 0x7fb1dc100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:48.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.808+0000 7fb1e297a640 1 --2- 172.21.15.67:0/231078209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1dc0fefd0 0x7fb1dc0ff3f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:48.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.808+0000 7fb1e297a640 1 -- 172.21.15.67:0/231078209 >> 172.21.15.67:0/231078209 conn(0x7fb1dc0fa560 msgr2=0x7fb1dc0fc980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:48.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.808+0000 7fb1e297a640 1 -- 172.21.15.67:0/231078209 shutdown_connections 2024-04-22T23:20:48.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.808+0000 7fb1e297a640 1 -- 172.21.15.67:0/231078209 wait complete. 2024-04-22T23:20:48.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.809+0000 7fb1e297a640 1 Processor -- start 2024-04-22T23:20:48.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.809+0000 7fb1e297a640 1 -- start start 2024-04-22T23:20:48.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.809+0000 7fb1e297a640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1dc0fefd0 0x7fb1dc102f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:48.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.809+0000 7fb1e297a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1dc1001c0 0x7fb1dc1015a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:48.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.809+0000 7fb1e297a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc101ae0 con 0x7fb1dc1001c0 2024-04-22T23:20:48.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.809+0000 7fb1e297a640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc101c50 con 0x7fb1dc0fefd0 2024-04-22T23:20:48.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.809+0000 7fb1db7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1dc1001c0 0x7fb1dc1015a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:48.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.809+0000 7fb1db7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1dc1001c0 0x7fb1dc1015a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58614/0 (socket says 172.21.15.67:58614) 2024-04-22T23:20:48.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.809+0000 7fb1db7fe640 1 -- 172.21.15.67:0/1687212654 learned_addr learned my addr 172.21.15.67:0/1687212654 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:20:48.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.810+0000 7fb1db7fe640 1 -- 172.21.15.67:0/1687212654 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1dc0fefd0 msgr2=0x7fb1dc102f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:48.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.810+0000 7fb1db7fe640 1 --2- 172.21.15.67:0/1687212654 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1dc0fefd0 0x7fb1dc102f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:48.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.810+0000 7fb1db7fe640 1 -- 172.21.15.67:0/1687212654 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1cc009560 con 0x7fb1dc1001c0 2024-04-22T23:20:48.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.810+0000 7fb1db7fe640 1 --2- 172.21.15.67:0/1687212654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1dc1001c0 0x7fb1dc1015a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb1d000b420 tx=0x7fb1d000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:48.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.810+0000 7fb1d97fa640 1 -- 172.21.15.67:0/1687212654 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1d0011800 con 0x7fb1dc1001c0 2024-04-22T23:20:48.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.810+0000 7fb1d97fa640 1 -- 172.21.15.67:0/1687212654 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb1d0011e10 con 0x7fb1dc1001c0 2024-04-22T23:20:48.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.810+0000 7fb1e297a640 1 -- 172.21.15.67:0/1687212654 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1dc101f30 con 0x7fb1dc1001c0 2024-04-22T23:20:48.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.810+0000 7fb1d97fa640 1 -- 172.21.15.67:0/1687212654 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1d000f2e0 con 0x7fb1dc1001c0 2024-04-22T23:20:48.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.810+0000 7fb1e297a640 1 -- 172.21.15.67:0/1687212654 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1dc19da60 con 0x7fb1dc1001c0 2024-04-22T23:20:48.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.811+0000 7fb1e297a640 1 -- 172.21.15.67:0/1687212654 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1dc107d80 con 0x7fb1dc1001c0 2024-04-22T23:20:48.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.811+0000 7fb1d97fa640 1 -- 172.21.15.67:0/1687212654 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fb1d0010380 con 0x7fb1dc1001c0 2024-04-22T23:20:48.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.812+0000 7fb1d97fa640 1 --2- 172.21.15.67:0/1687212654 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fb1c406c450 0x7fb1c406e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:48.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.812+0000 7fb1d97fa640 1 -- 172.21.15.67:0/1687212654 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fb1d008b260 con 0x7fb1dc1001c0 2024-04-22T23:20:48.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.812+0000 7fb1dbfff640 1 --2- 172.21.15.67:0/1687212654 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fb1c406c450 0x7fb1c406e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:48.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.812+0000 7fb1dbfff640 1 --2- 172.21.15.67:0/1687212654 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fb1c406c450 0x7fb1c406e910 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc02c7c0 tx=0x7fb1cc000bc0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:48.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:48.816+0000 7fb1d97fa640 1 -- 172.21.15.67:0/1687212654 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb1d00559e0 con 0x7fb1dc1001c0 2024-04-22T23:20:48.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:48 smithi134 ceph-mon[34609]: pgmap v189: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 10 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-22T23:20:48.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:48 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/2360212959' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-04-22T23:20:49.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.001+0000 7fb1e297a640 1 -- 172.21.15.67:0/1687212654 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb1dc107f10 con 0x7fb1dc1001c0 2024-04-22T23:20:49.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.002+0000 7fb1d97fa640 1 -- 172.21.15.67:0/1687212654 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb1dc107f10 con 0x7fb1dc1001c0 2024-04-22T23:20:49.003 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-22T23:20:49.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.004+0000 7fb1e297a640 1 -- 172.21.15.67:0/1687212654 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fb1c406c450 msgr2=0x7fb1c406e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:49.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.004+0000 7fb1e297a640 1 --2- 172.21.15.67:0/1687212654 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fb1c406c450 0x7fb1c406e910 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc02c7c0 tx=0x7fb1cc000bc0 comp rx=0 tx=0).stop 2024-04-22T23:20:49.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.004+0000 7fb1e297a640 1 -- 172.21.15.67:0/1687212654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1dc1001c0 msgr2=0x7fb1dc1015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:49.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.004+0000 7fb1e297a640 1 --2- 172.21.15.67:0/1687212654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1dc1001c0 0x7fb1dc1015a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb1d000b420 tx=0x7fb1d000b8e0 comp rx=0 tx=0).stop 2024-04-22T23:20:49.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.004+0000 7fb1e297a640 1 -- 172.21.15.67:0/1687212654 shutdown_connections 2024-04-22T23:20:49.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.004+0000 7fb1e297a640 1 --2- 172.21.15.67:0/1687212654 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fb1c406c450 0x7fb1c406e910 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:49.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.004+0000 7fb1e297a640 1 --2- 172.21.15.67:0/1687212654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1dc1001c0 0x7fb1dc1015a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:49.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.004+0000 7fb1e297a640 1 --2- 172.21.15.67:0/1687212654 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1dc0fefd0 0x7fb1dc102f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:49.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.004+0000 7fb1e297a640 1 -- 172.21.15.67:0/1687212654 >> 172.21.15.67:0/1687212654 conn(0x7fb1dc0fa560 msgr2=0x7fb1dc068380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:49.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.004+0000 7fb1e297a640 1 -- 172.21.15.67:0/1687212654 shutdown_connections 2024-04-22T23:20:49.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.004+0000 7fb1e297a640 1 -- 172.21.15.67:0/1687212654 wait complete. 2024-04-22T23:20:49.016 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-22T23:20:49.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.124+0000 7f644033c640 1 -- 172.21.15.67:0/4058150084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6438100090 msgr2=0x7f64381004b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:49.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.124+0000 7f644033c640 1 --2- 172.21.15.67:0/4058150084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6438100090 0x7f64381004b0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6434009880 tx=0x7f643402bce0 comp rx=0 tx=0).stop 2024-04-22T23:20:49.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.125+0000 7f644033c640 1 -- 172.21.15.67:0/4058150084 shutdown_connections 2024-04-22T23:20:49.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.125+0000 7f644033c640 1 --2- 172.21.15.67:0/4058150084 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6438101280 0x7f6438101700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:49.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.125+0000 7f644033c640 1 --2- 172.21.15.67:0/4058150084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6438100090 0x7f64381004b0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:49.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.125+0000 7f644033c640 1 -- 172.21.15.67:0/4058150084 >> 172.21.15.67:0/4058150084 conn(0x7f64380fb5e0 msgr2=0x7f64380fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:49.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.125+0000 7f644033c640 1 -- 172.21.15.67:0/4058150084 shutdown_connections 2024-04-22T23:20:49.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.125+0000 7f644033c640 1 -- 172.21.15.67:0/4058150084 wait complete. 2024-04-22T23:20:49.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.125+0000 7f644033c640 1 Processor -- start 2024-04-22T23:20:49.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.125+0000 7f644033c640 1 -- start start 2024-04-22T23:20:49.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.125+0000 7f644033c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6438100090 0x7f6438195960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:49.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.125+0000 7f644033c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6438101280 0x7f6438195ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:49.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.125+0000 7f644033c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6438196470 con 0x7f6438101280 2024-04-22T23:20:49.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.125+0000 7f644033c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64381965e0 con 0x7f6438100090 2024-04-22T23:20:49.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.125+0000 7f643d8d7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6438101280 0x7f6438195ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:49.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.125+0000 7f643d8d7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6438101280 0x7f6438195ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58638/0 (socket says 172.21.15.67:58638) 2024-04-22T23:20:49.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.125+0000 7f643d8d7640 1 -- 172.21.15.67:0/3156382611 learned_addr learned my addr 172.21.15.67:0/3156382611 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:20:49.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.126+0000 7f643d8d7640 1 -- 172.21.15.67:0/3156382611 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6438100090 msgr2=0x7f6438195960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:49.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.126+0000 7f643d8d7640 1 --2- 172.21.15.67:0/3156382611 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6438100090 0x7f6438195960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:49.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.126+0000 7f643d8d7640 1 -- 172.21.15.67:0/3156382611 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6434009560 con 0x7f6438101280 2024-04-22T23:20:49.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.126+0000 7f643d8d7640 1 --2- 172.21.15.67:0/3156382611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6438101280 0x7f6438195ea0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f642800d6f0 tx=0x7f642800dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:49.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.126+0000 7f642f7fe640 1 -- 172.21.15.67:0/3156382611 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6428009a30 con 0x7f6438101280 2024-04-22T23:20:49.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.126+0000 7f644033c640 1 -- 172.21.15.67:0/3156382611 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f643819b070 con 0x7f6438101280 2024-04-22T23:20:49.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.126+0000 7f642f7fe640 1 -- 172.21.15.67:0/3156382611 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f64280054d0 con 0x7f6438101280 2024-04-22T23:20:49.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.126+0000 7f642f7fe640 1 -- 172.21.15.67:0/3156382611 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6428010460 con 0x7f6438101280 2024-04-22T23:20:49.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.126+0000 7f644033c640 1 -- 172.21.15.67:0/3156382611 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f643819b5d0 con 0x7f6438101280 2024-04-22T23:20:49.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.127+0000 7f644033c640 1 -- 172.21.15.67:0/3156382611 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6438108de0 con 0x7f6438101280 2024-04-22T23:20:49.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.128+0000 7f642f7fe640 1 -- 172.21.15.67:0/3156382611 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f6428009b90 con 0x7f6438101280 2024-04-22T23:20:49.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.128+0000 7f642f7fe640 1 --2- 172.21.15.67:0/3156382611 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f642406c450 0x7f642406e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:49.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.128+0000 7f642f7fe640 1 -- 172.21.15.67:0/3156382611 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f642808b420 con 0x7f6438101280 2024-04-22T23:20:49.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.128+0000 7f643e0d8640 1 --2- 172.21.15.67:0/3156382611 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f642406c450 0x7f642406e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:49.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.128+0000 7f643e0d8640 1 --2- 172.21.15.67:0/3156382611 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f642406c450 0x7f642406e910 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f643402c7c0 tx=0x7f6434000bc0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:49.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.132+0000 7f642f7fe640 1 -- 172.21.15.67:0/3156382611 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6428055ba0 con 0x7f6438101280 2024-04-22T23:20:49.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.308+0000 7f644033c640 1 -- 172.21.15.67:0/3156382611 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f643819ba20 con 0x7f6438101280 2024-04-22T23:20:49.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.309+0000 7f642f7fe640 1 -- 172.21.15.67:0/3156382611 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7f64280591c0 con 0x7f6438101280 2024-04-22T23:20:49.310 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-22T23:20:49.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.311+0000 7f644033c640 1 -- 172.21.15.67:0/3156382611 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f642406c450 msgr2=0x7f642406e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:49.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.311+0000 7f644033c640 1 --2- 172.21.15.67:0/3156382611 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f642406c450 0x7f642406e910 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f643402c7c0 tx=0x7f6434000bc0 comp rx=0 tx=0).stop 2024-04-22T23:20:49.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.311+0000 7f644033c640 1 -- 172.21.15.67:0/3156382611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6438101280 msgr2=0x7f6438195ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:49.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.311+0000 7f644033c640 1 --2- 172.21.15.67:0/3156382611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6438101280 0x7f6438195ea0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f642800d6f0 tx=0x7f642800dbb0 comp rx=0 tx=0).stop 2024-04-22T23:20:49.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.311+0000 7f644033c640 1 -- 172.21.15.67:0/3156382611 shutdown_connections 2024-04-22T23:20:49.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.311+0000 7f644033c640 1 --2- 172.21.15.67:0/3156382611 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f642406c450 0x7f642406e910 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:49.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.311+0000 7f644033c640 1 --2- 172.21.15.67:0/3156382611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6438101280 0x7f6438195ea0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:49.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.311+0000 7f644033c640 1 --2- 172.21.15.67:0/3156382611 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6438100090 0x7f6438195960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:49.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.311+0000 7f644033c640 1 -- 172.21.15.67:0/3156382611 >> 172.21.15.67:0/3156382611 conn(0x7f64380fb5e0 msgr2=0x7f64381044b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:49.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.311+0000 7f644033c640 1 -- 172.21.15.67:0/3156382611 shutdown_connections 2024-04-22T23:20:49.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.311+0000 7f644033c640 1 -- 172.21.15.67:0/3156382611 wait complete. 2024-04-22T23:20:49.323 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-22T23:20:49.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.437+0000 7f5ed05c3640 1 -- 172.21.15.67:0/719004622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ec8100180 msgr2=0x7f5ec8100600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:49.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.437+0000 7f5ed05c3640 1 --2- 172.21.15.67:0/719004622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ec8100180 0x7f5ec8100600 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5eb8009880 tx=0x7f5eb802bce0 comp rx=0 tx=0).stop 2024-04-22T23:20:49.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.437+0000 7f5ed05c3640 1 -- 172.21.15.67:0/719004622 shutdown_connections 2024-04-22T23:20:49.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.437+0000 7f5ed05c3640 1 --2- 172.21.15.67:0/719004622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ec8100180 0x7f5ec8100600 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:49.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.437+0000 7f5ed05c3640 1 --2- 172.21.15.67:0/719004622 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ec80fef90 0x7f5ec80ff3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:49.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.437+0000 7f5ed05c3640 1 -- 172.21.15.67:0/719004622 >> 172.21.15.67:0/719004622 conn(0x7f5ec80fa4e0 msgr2=0x7f5ec80fc940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:49.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.437+0000 7f5ed05c3640 1 -- 172.21.15.67:0/719004622 shutdown_connections 2024-04-22T23:20:49.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.437+0000 7f5ed05c3640 1 -- 172.21.15.67:0/719004622 wait complete. 2024-04-22T23:20:49.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.437+0000 7f5ed05c3640 1 Processor -- start 2024-04-22T23:20:49.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.437+0000 7f5ed05c3640 1 -- start start 2024-04-22T23:20:49.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.438+0000 7f5ed05c3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ec80fef90 0x7f5ec8067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:49.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.438+0000 7f5ed05c3640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ec8100180 0x7f5ec819caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:49.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.438+0000 7f5ed05c3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec8196510 con 0x7f5ec80fef90 2024-04-22T23:20:49.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.438+0000 7f5ed05c3640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec819cfe0 con 0x7f5ec8100180 2024-04-22T23:20:49.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.438+0000 7f5ece35f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ec80fef90 0x7f5ec8067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:49.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.438+0000 7f5ece35f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ec80fef90 0x7f5ec8067ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58650/0 (socket says 172.21.15.67:58650) 2024-04-22T23:20:49.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.438+0000 7f5ece35f640 1 -- 172.21.15.67:0/618478442 learned_addr learned my addr 172.21.15.67:0/618478442 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:20:49.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.438+0000 7f5ecdb5e640 1 --2- 172.21.15.67:0/618478442 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ec8100180 0x7f5ec819caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:49.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.438+0000 7f5ece35f640 1 -- 172.21.15.67:0/618478442 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ec8100180 msgr2=0x7f5ec819caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:49.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.438+0000 7f5ece35f640 1 --2- 172.21.15.67:0/618478442 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ec8100180 0x7f5ec819caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:49.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.438+0000 7f5ece35f640 1 -- 172.21.15.67:0/618478442 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb8009560 con 0x7f5ec80fef90 2024-04-22T23:20:49.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.438+0000 7f5ece35f640 1 --2- 172.21.15.67:0/618478442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ec80fef90 0x7f5ec8067ec0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5ec400e940 tx=0x7f5ec400ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:49.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.438+0000 7f5ebf7fe640 1 -- 172.21.15.67:0/618478442 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ec40096e0 con 0x7f5ec80fef90 2024-04-22T23:20:49.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.438+0000 7f5ed05c3640 1 -- 172.21.15.67:0/618478442 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ec819d270 con 0x7f5ec80fef90 2024-04-22T23:20:49.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.438+0000 7f5ebf7fe640 1 -- 172.21.15.67:0/618478442 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5ec40044e0 con 0x7f5ec80fef90 2024-04-22T23:20:49.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.438+0000 7f5ebf7fe640 1 -- 172.21.15.67:0/618478442 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ec4010430 con 0x7f5ec80fef90 2024-04-22T23:20:49.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.438+0000 7f5ed05c3640 1 -- 172.21.15.67:0/618478442 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ec819d7d0 con 0x7f5ec80fef90 2024-04-22T23:20:49.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.440+0000 7f5ed05c3640 1 -- 172.21.15.67:0/618478442 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ec8107ce0 con 0x7f5ec80fef90 2024-04-22T23:20:49.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.440+0000 7f5ebf7fe640 1 -- 172.21.15.67:0/618478442 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f5ec400cc80 con 0x7f5ec80fef90 2024-04-22T23:20:49.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.440+0000 7f5ebf7fe640 1 --2- 172.21.15.67:0/618478442 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f5eb406c450 0x7f5eb406e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:49.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.441+0000 7f5ebf7fe640 1 -- 172.21.15.67:0/618478442 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f5ec4014070 con 0x7f5ec80fef90 2024-04-22T23:20:49.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.441+0000 7f5ecdb5e640 1 --2- 172.21.15.67:0/618478442 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f5eb406c450 0x7f5eb406e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:49.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.441+0000 7f5ecdb5e640 1 --2- 172.21.15.67:0/618478442 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f5eb406c450 0x7f5eb406e910 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5ec8067c30 tx=0x7f5eb802cf90 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:49.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.445+0000 7f5ebf7fe640 1 -- 172.21.15.67:0/618478442 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f5ec4056520 con 0x7f5ec80fef90 2024-04-22T23:20:49.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.622+0000 7f5ed05c3640 1 -- 172.21.15.67:0/618478442 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f5ec819def0 con 0x7f5ec80fef90 2024-04-22T23:20:49.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.623+0000 7f5ebf7fe640 1 -- 172.21.15.67:0/618478442 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7f5ec4059b40 con 0x7f5ec80fef90 2024-04-22T23:20:49.624 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-22T23:20:49.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.624+0000 7f5ed05c3640 1 -- 172.21.15.67:0/618478442 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f5eb406c450 msgr2=0x7f5eb406e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:49.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.624+0000 7f5ed05c3640 1 --2- 172.21.15.67:0/618478442 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f5eb406c450 0x7f5eb406e910 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5ec8067c30 tx=0x7f5eb802cf90 comp rx=0 tx=0).stop 2024-04-22T23:20:49.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.624+0000 7f5ed05c3640 1 -- 172.21.15.67:0/618478442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ec80fef90 msgr2=0x7f5ec8067ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:49.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.624+0000 7f5ed05c3640 1 --2- 172.21.15.67:0/618478442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ec80fef90 0x7f5ec8067ec0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5ec400e940 tx=0x7f5ec400ee00 comp rx=0 tx=0).stop 2024-04-22T23:20:49.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.624+0000 7f5ed05c3640 1 -- 172.21.15.67:0/618478442 shutdown_connections 2024-04-22T23:20:49.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.624+0000 7f5ed05c3640 1 --2- 172.21.15.67:0/618478442 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f5eb406c450 0x7f5eb406e910 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:49.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.624+0000 7f5ed05c3640 1 --2- 172.21.15.67:0/618478442 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ec8100180 0x7f5ec819caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:49.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.624+0000 7f5ed05c3640 1 --2- 172.21.15.67:0/618478442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ec80fef90 0x7f5ec8067ec0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:49.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.624+0000 7f5ed05c3640 1 -- 172.21.15.67:0/618478442 >> 172.21.15.67:0/618478442 conn(0x7f5ec80fa4e0 msgr2=0x7f5ec81033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:49.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.625+0000 7f5ed05c3640 1 -- 172.21.15.67:0/618478442 shutdown_connections 2024-04-22T23:20:49.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.625+0000 7f5ed05c3640 1 -- 172.21.15.67:0/618478442 wait complete. 2024-04-22T23:20:49.636 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T23:20:49.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.744+0000 7f8b8d477640 1 -- 172.21.15.67:0/2452613890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b88100b30 msgr2=0x7f8b88102f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:49.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.744+0000 7f8b8d477640 1 --2- 172.21.15.67:0/2452613890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b88100b30 0x7f8b88102f20 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8b780098b0 tx=0x7f8b7802bcd0 comp rx=0 tx=0).stop 2024-04-22T23:20:49.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.745+0000 7f8b8d477640 1 -- 172.21.15.67:0/2452613890 shutdown_connections 2024-04-22T23:20:49.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.745+0000 7f8b8d477640 1 --2- 172.21.15.67:0/2452613890 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8b88103460 0x7f8b88105850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:49.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.745+0000 7f8b8d477640 1 --2- 172.21.15.67:0/2452613890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b88100b30 0x7f8b88102f20 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:49.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.745+0000 7f8b8d477640 1 -- 172.21.15.67:0/2452613890 >> 172.21.15.67:0/2452613890 conn(0x7f8b880fa4e0 msgr2=0x7f8b880fc960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:49.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.745+0000 7f8b8d477640 1 -- 172.21.15.67:0/2452613890 shutdown_connections 2024-04-22T23:20:49.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.745+0000 7f8b8d477640 1 -- 172.21.15.67:0/2452613890 wait complete. 2024-04-22T23:20:49.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.745+0000 7f8b8d477640 1 Processor -- start 2024-04-22T23:20:49.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.745+0000 7f8b8d477640 1 -- start start 2024-04-22T23:20:49.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.745+0000 7f8b8d477640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8b88100b30 0x7f8b881958e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:49.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.745+0000 7f8b8d477640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b88103460 0x7f8b88195e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:49.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.745+0000 7f8b8d477640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b881963f0 con 0x7f8b88103460 2024-04-22T23:20:49.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.745+0000 7f8b8d477640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b88196560 con 0x7f8b88100b30 2024-04-22T23:20:49.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.746+0000 7f8b867fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b88103460 0x7f8b88195e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:49.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.746+0000 7f8b867fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b88103460 0x7f8b88195e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58666/0 (socket says 172.21.15.67:58666) 2024-04-22T23:20:49.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.746+0000 7f8b867fc640 1 -- 172.21.15.67:0/2326323216 learned_addr learned my addr 172.21.15.67:0/2326323216 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:20:49.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.746+0000 7f8b86ffd640 1 --2- 172.21.15.67:0/2326323216 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8b88100b30 0x7f8b881958e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:49.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.746+0000 7f8b867fc640 1 -- 172.21.15.67:0/2326323216 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8b88100b30 msgr2=0x7f8b881958e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:49.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.746+0000 7f8b867fc640 1 --2- 172.21.15.67:0/2326323216 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8b88100b30 0x7f8b881958e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:49.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.746+0000 7f8b867fc640 1 -- 172.21.15.67:0/2326323216 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b78009560 con 0x7f8b88103460 2024-04-22T23:20:49.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.746+0000 7f8b867fc640 1 --2- 172.21.15.67:0/2326323216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b88103460 0x7f8b88195e20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c00d6f0 tx=0x7f8b7c00dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:49.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.746+0000 7f8b6ffff640 1 -- 172.21.15.67:0/2326323216 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b7c009a30 con 0x7f8b88103460 2024-04-22T23:20:49.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.746+0000 7f8b8d477640 1 -- 172.21.15.67:0/2326323216 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b8819aff0 con 0x7f8b88103460 2024-04-22T23:20:49.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.746+0000 7f8b6ffff640 1 -- 172.21.15.67:0/2326323216 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8b7c009b90 con 0x7f8b88103460 2024-04-22T23:20:49.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.746+0000 7f8b6ffff640 1 -- 172.21.15.67:0/2326323216 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b7c00f6c0 con 0x7f8b88103460 2024-04-22T23:20:49.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.746+0000 7f8b8d477640 1 -- 172.21.15.67:0/2326323216 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b8819b520 con 0x7f8b88103460 2024-04-22T23:20:49.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.747+0000 7f8b8d477640 1 -- 172.21.15.67:0/2326323216 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b8818fa30 con 0x7f8b88103460 2024-04-22T23:20:49.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.748+0000 7f8b6ffff640 1 -- 172.21.15.67:0/2326323216 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f8b7c009d00 con 0x7f8b88103460 2024-04-22T23:20:49.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.748+0000 7f8b6ffff640 1 --2- 172.21.15.67:0/2326323216 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f8b7006c510 0x7f8b7006e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:49.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.748+0000 7f8b6ffff640 1 -- 172.21.15.67:0/2326323216 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f8b7c08b680 con 0x7f8b88103460 2024-04-22T23:20:49.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.748+0000 7f8b86ffd640 1 --2- 172.21.15.67:0/2326323216 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f8b7006c510 0x7f8b7006e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:49.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.749+0000 7f8b86ffd640 1 --2- 172.21.15.67:0/2326323216 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f8b7006c510 0x7f8b7006e9d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8b78000c00 tx=0x7f8b78005fd0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:49.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.753+0000 7f8b6ffff640 1 -- 172.21.15.67:0/2326323216 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f8b7c055e00 con 0x7f8b88103460 2024-04-22T23:20:49.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:49 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/4287193508' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T23:20:49.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:49 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/1687212654' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T23:20:49.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:49 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/3156382611' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-22T23:20:49.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.966+0000 7f8b8d477640 1 -- 172.21.15.67:0/2326323216 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b88108cb0 con 0x7f8b88103460 2024-04-22T23:20:49.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.967+0000 7f8b6ffff640 1 -- 172.21.15.67:0/2326323216 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4773 (secure 0 0 0) 0x7f8b7c059420 con 0x7f8b88103460 2024-04-22T23:20:49.968 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:20:49.968 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":18,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T23:20:20.823261+0000","modified":"2024-04-22T23:20:41.528764+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14484,"mds_1":24273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14484":{"gid":14484,"name":"cephfs.smithi067.xyazmh","rank":0,"incarnation":13,"state":"up:active","state_seq":6,"addr":"172.21.15.67:6829/3606151860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3606151860},{"type":"v1","addr":"172.21.15.67:6829","nonce":3606151860}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14496":{"gid":14496,"name":"cephfs.smithi067.aqlxpv","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6827/3305304988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3305304988},{"type":"v1","addr":"172.21.15.67:6827","nonce":3305304988}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24273":{"gid":24273,"name":"cephfs.smithi134.owipad","rank":1,"incarnation":6,"state":"up:active","state_seq":4,"addr":"172.21.15.134:6825/535957142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":535957142},{"type":"v1","addr":"172.21.15.134:6825","nonce":535957142}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24289":{"gid":24289,"name":"cephfs.smithi134.vuxbsj","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.134:6827/1954821676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":1954821676},{"type":"v1","addr":"172.21.15.134:6827","nonce":1954821676}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-22T23:20:49.969 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 18 2024-04-22T23:20:49.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.969+0000 7f8b8d477640 1 -- 172.21.15.67:0/2326323216 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f8b7006c510 msgr2=0x7f8b7006e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:49.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.969+0000 7f8b8d477640 1 --2- 172.21.15.67:0/2326323216 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f8b7006c510 0x7f8b7006e9d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8b78000c00 tx=0x7f8b78005fd0 comp rx=0 tx=0).stop 2024-04-22T23:20:49.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.969+0000 7f8b8d477640 1 -- 172.21.15.67:0/2326323216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b88103460 msgr2=0x7f8b88195e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:49.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.969+0000 7f8b8d477640 1 --2- 172.21.15.67:0/2326323216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b88103460 0x7f8b88195e20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c00d6f0 tx=0x7f8b7c00dbb0 comp rx=0 tx=0).stop 2024-04-22T23:20:49.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.969+0000 7f8b8d477640 1 -- 172.21.15.67:0/2326323216 shutdown_connections 2024-04-22T23:20:49.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.969+0000 7f8b8d477640 1 --2- 172.21.15.67:0/2326323216 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f8b7006c510 0x7f8b7006e9d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:49.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.969+0000 7f8b8d477640 1 --2- 172.21.15.67:0/2326323216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b88103460 0x7f8b88195e20 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:49.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.969+0000 7f8b8d477640 1 --2- 172.21.15.67:0/2326323216 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8b88100b30 0x7f8b881958e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:49.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.969+0000 7f8b8d477640 1 -- 172.21.15.67:0/2326323216 >> 172.21.15.67:0/2326323216 conn(0x7f8b880fa4e0 msgr2=0x7f8b880fc960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:49.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.969+0000 7f8b8d477640 1 -- 172.21.15.67:0/2326323216 shutdown_connections 2024-04-22T23:20:49.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:49.969+0000 7f8b8d477640 1 -- 172.21.15.67:0/2326323216 wait complete. 2024-04-22T23:20:49.981 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 18, 'max_mds': 2, 'flags': 50} 2024-04-22T23:20:49.981 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-04-22T23:20:50.008 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-04-22T23:20:50.008 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.1': {}} 2024-04-22T23:20:50.008 INFO:tasks.ceph_fuse:client.0 config is {} 2024-04-22T23:20:50.008 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-22T23:20:50.008 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-22T23:20:50.008 INFO:tasks.ceph_fuse:client.1 config is {} 2024-04-22T23:20:50.009 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-22T23:20:50.009 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-22T23:20:50.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T23:20:50.009 DEBUG:teuthology.orchestra.run.smithi134:> ip netns list 2024-04-22T23:20:50.010 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:49 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/4287193508' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T23:20:50.010 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:49 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/1687212654' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T23:20:50.010 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:49 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/3156382611' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-22T23:20:50.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T23:20:50.029 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip link delete ceph-brx 2024-04-22T23:20:50.101 INFO:teuthology.orchestra.run.smithi134.stderr:Cannot find device "ceph-brx" 2024-04-22T23:20:50.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T23:20:50.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T23:20:50.103 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-22T23:20:50.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T23:20:50.119 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link delete ceph-brx 2024-04-22T23:20:50.187 INFO:teuthology.orchestra.run.smithi067.stderr:Cannot find device "ceph-brx" 2024-04-22T23:20:50.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T23:20:50.189 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-04-22T23:20:50.189 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-22T23:20:50.467 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:20:50.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:50 smithi067 ceph-mon[26453]: pgmap v190: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-22T23:20:50.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:50 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/618478442' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-22T23:20:50.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:50 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/2326323216' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T23:20:50.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:50 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:50.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:50 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:50.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:50 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:20:50.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:50 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:20:50.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:50 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:50.953 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:50 smithi134 ceph-mon[34609]: pgmap v190: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-22T23:20:50.953 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:50 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/618478442' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-22T23:20:50.953 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:50 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/2326323216' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T23:20:50.953 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:50 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:50.953 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:50 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:50.953 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:50 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:20:50.953 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:50 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:20:50.953 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:50 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:20:52.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.089+0000 7f11728d0700 1 -- 172.21.15.67:0/1174973770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f116c101850 msgr2=0x7f116c1058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:52.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.089+0000 7f11728d0700 1 --2- 172.21.15.67:0/1174973770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f116c101850 0x7f116c1058a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1160009910 tx=0x7f1160009c20 comp rx=0 tx=0).stop 2024-04-22T23:20:52.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.090+0000 7f11728d0700 1 -- 172.21.15.67:0/1174973770 shutdown_connections 2024-04-22T23:20:52.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.090+0000 7f11728d0700 1 --2- 172.21.15.67:0/1174973770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f116c101850 0x7f116c1058a0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:52.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.090+0000 7f11728d0700 1 --2- 172.21.15.67:0/1174973770 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f116c100ea0 0x7f116c101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:52.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.090+0000 7f11728d0700 1 -- 172.21.15.67:0/1174973770 >> 172.21.15.67:0/1174973770 conn(0x7f116c0fc750 msgr2=0x7f116c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:52.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.090+0000 7f11728d0700 1 -- 172.21.15.67:0/1174973770 shutdown_connections 2024-04-22T23:20:52.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.090+0000 7f11728d0700 1 -- 172.21.15.67:0/1174973770 wait complete. 2024-04-22T23:20:52.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.090+0000 7f11728d0700 1 Processor -- start 2024-04-22T23:20:52.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.090+0000 7f11728d0700 1 -- start start 2024-04-22T23:20:52.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.091+0000 7f11728d0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f116c100ea0 0x7f116c073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:52.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.091+0000 7f11728d0700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f116c101850 0x7f116c0735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:52.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.091+0000 7f11728d0700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116c073b10 con 0x7f116c100ea0 2024-04-22T23:20:52.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.091+0000 7f11728d0700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116c073c50 con 0x7f116c101850 2024-04-22T23:20:52.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.091+0000 7f116bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f116c100ea0 0x7f116c073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:52.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.091+0000 7f116bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f116c100ea0 0x7f116c073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58680/0 (socket says 172.21.15.67:58680) 2024-04-22T23:20:52.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.091+0000 7f116bfff700 1 -- 172.21.15.67:0/2638599595 learned_addr learned my addr 172.21.15.67:0/2638599595 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:20:52.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.091+0000 7f116b7fe700 1 --2- 172.21.15.67:0/2638599595 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f116c101850 0x7f116c0735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:52.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.091+0000 7f116bfff700 1 -- 172.21.15.67:0/2638599595 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f116c101850 msgr2=0x7f116c0735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:52.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.091+0000 7f116bfff700 1 --2- 172.21.15.67:0/2638599595 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f116c101850 0x7f116c0735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:52.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.091+0000 7f116bfff700 1 -- 172.21.15.67:0/2638599595 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11600095f0 con 0x7f116c100ea0 2024-04-22T23:20:52.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.091+0000 7f116bfff700 1 --2- 172.21.15.67:0/2638599595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f116c100ea0 0x7f116c073090 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f115c00d8d0 tx=0x7f115c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:52.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.091+0000 7f11697fa700 1 -- 172.21.15.67:0/2638599595 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f115c0098a0 con 0x7f116c100ea0 2024-04-22T23:20:52.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.091+0000 7f11697fa700 1 -- 172.21.15.67:0/2638599595 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f115c010460 con 0x7f116c100ea0 2024-04-22T23:20:52.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.091+0000 7f11728d0700 1 -- 172.21.15.67:0/2638599595 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f116c1a2b00 con 0x7f116c100ea0 2024-04-22T23:20:52.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.091+0000 7f11697fa700 1 -- 172.21.15.67:0/2638599595 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f115c00f5f0 con 0x7f116c100ea0 2024-04-22T23:20:52.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.091+0000 7f11728d0700 1 -- 172.21.15.67:0/2638599595 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f116c1a2f10 con 0x7f116c100ea0 2024-04-22T23:20:52.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.092+0000 7f11728d0700 1 -- 172.21.15.67:0/2638599595 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f116c1092a0 con 0x7f116c100ea0 2024-04-22T23:20:52.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.093+0000 7f11697fa700 1 -- 172.21.15.67:0/2638599595 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f115c009a00 con 0x7f116c100ea0 2024-04-22T23:20:52.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.093+0000 7f11697fa700 1 --2- 172.21.15.67:0/2638599595 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f115406c4e0 0x7f115406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:52.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.093+0000 7f11697fa700 1 -- 172.21.15.67:0/2638599595 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f115c08b770 con 0x7f116c100ea0 2024-04-22T23:20:52.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.094+0000 7f116b7fe700 1 --2- 172.21.15.67:0/2638599595 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f115406c4e0 0x7f115406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:52.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.094+0000 7f116b7fe700 1 --2- 172.21.15.67:0/2638599595 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f115406c4e0 0x7f115406e9a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f116c074880 tx=0x7f116001a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:52.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.098+0000 7f11697fa700 1 -- 172.21.15.67:0/2638599595 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f115c055ef0 con 0x7f116c100ea0 2024-04-22T23:20:52.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.342+0000 7f11728d0700 1 -- 172.21.15.67:0/2638599595 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f116c108c80 con 0x7f116c100ea0 2024-04-22T23:20:52.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.342+0000 7f11697fa700 1 -- 172.21.15.67:0/2638599595 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) v1 ==== 53+0+83 (secure 0 0 0) 0x7f115c059510 con 0x7f116c100ea0 2024-04-22T23:20:52.344 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-04-22T23:20:52.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.344+0000 7f11728d0700 1 -- 172.21.15.67:0/2638599595 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f115406c4e0 msgr2=0x7f115406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:52.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.344+0000 7f11728d0700 1 --2- 172.21.15.67:0/2638599595 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f115406c4e0 0x7f115406e9a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f116c074880 tx=0x7f116001a040 comp rx=0 tx=0).stop 2024-04-22T23:20:52.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.344+0000 7f11728d0700 1 -- 172.21.15.67:0/2638599595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f116c100ea0 msgr2=0x7f116c073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:52.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.344+0000 7f11728d0700 1 --2- 172.21.15.67:0/2638599595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f116c100ea0 0x7f116c073090 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f115c00d8d0 tx=0x7f115c00dbe0 comp rx=0 tx=0).stop 2024-04-22T23:20:52.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.344+0000 7f11728d0700 1 -- 172.21.15.67:0/2638599595 shutdown_connections 2024-04-22T23:20:52.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.344+0000 7f11728d0700 1 --2- 172.21.15.67:0/2638599595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f116c100ea0 0x7f116c073090 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:52.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.344+0000 7f11728d0700 1 --2- 172.21.15.67:0/2638599595 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f115406c4e0 0x7f115406e9a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:52.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.344+0000 7f11728d0700 1 --2- 172.21.15.67:0/2638599595 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f116c101850 0x7f116c0735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:52.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.344+0000 7f11728d0700 1 -- 172.21.15.67:0/2638599595 >> 172.21.15.67:0/2638599595 conn(0x7f116c0fc750 msgr2=0x7f116c0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:52.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.345+0000 7f11728d0700 1 -- 172.21.15.67:0/2638599595 shutdown_connections 2024-04-22T23:20:52.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:52.345+0000 7f11728d0700 1 -- 172.21.15.67:0/2638599595 wait complete. 2024-04-22T23:20:52.745 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:52 smithi067 ceph-mon[26453]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-22T23:20:52.746 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:52 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/2638599595' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-22T23:20:52.746 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:52 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:20:52.980 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-22T23:20:52.980 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-22T23:20:52.980 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-22T23:20:52.980 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-22T23:20:52.980 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-22T23:20:52.980 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-22T23:20:52.980 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-22T23:20:52.980 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-22T23:20:52.980 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-22T23:20:52.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T23:20:52.981 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-22T23:20:52.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:52 smithi134 ceph-mon[34609]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-22T23:20:52.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:52 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/2638599595' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-22T23:20:52.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:52 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:20:52.997 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-22T23:20:52.997 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-22T23:20:52.997 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-22T23:20:52.997 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-22T23:20:52.997 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-22T23:20:52.997 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-22T23:20:52.997 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T23:20:52.997 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-22T23:20:52.997 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-22T23:20:52.997 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T23:20:52.997 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-22T23:20:52.997 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-22T23:20:52.997 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T23:20:52.998 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-22T23:20:52.998 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-22T23:20:52.998 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-22T23:20:52.998 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-22T23:20:52.998 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42296sec preferred_lft 42296sec 2024-04-22T23:20:52.998 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::b0a1:74d2:2efb:1383/64 scope link noprefixroute 2024-04-22T23:20:52.998 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-22T23:20:52.998 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-22T23:20:52.999 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T23:20:52.999 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-22T23:20:52.999 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link add name ceph-brx type bridge 2024-04-22T23:20:52.999 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip addr flush dev ceph-brx 2024-04-22T23:20:52.999 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set ceph-brx up 2024-04-22T23:20:52.999 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-22T23:20:52.999 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-22T23:20:53.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T23:20:53.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T23:20:53.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T23:20:53.186 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-22T23:20:53.210 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-22T23:20:53.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T23:20:53.212 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-22T23:20:53.277 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-22T23:20:53.277 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-22T23:20:53.277 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-22T23:20:53.278 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-22T23:20:53.278 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-22T23:20:53.278 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T23:20:53.278 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-22T23:20:53.278 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-22T23:20:53.278 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-22T23:20:53.278 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-22T23:20:53.278 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-22T23:20:53.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T23:20:53.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T23:20:53.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T23:20:53.561 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-22T23:20:53.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T23:20:53.576 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list-id 2024-04-22T23:20:53.631 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T23:20:53.631 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-22T23:20:53.631 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T23:20:53.632 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-22T23:20:53.632 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-22T23:20:53.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T23:20:53.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T23:20:53.748 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-22T23:20:53.748 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T23:20:53.748 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-22T23:20:53.748 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-22T23:20:53.748 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-22T23:20:53.748 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-22T23:20:53.748 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-22T23:20:53.749 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-22T23:20:53.749 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-22T23:20:53.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T23:20:53.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T23:20:53.875 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T23:20:53.875 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-22T23:20:53.875 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-04-22T23:20:53.875 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set dev brx.0 master ceph-brx 2024-04-22T23:20:53.876 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-22T23:20:53.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T23:20:53.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T23:20:53.957 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-22T23:20:53.957 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T23:20:53.957 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-22T23:20:53.972 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-22T23:20:53.973 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T23:20:53.973 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-22T23:20:54.028 DEBUG:teuthology.orchestra.run.smithi067:> sudo modprobe fuse 2024-04-22T23:20:54.094 DEBUG:teuthology.orchestra.run.smithi067:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T23:20:54.150 INFO:teuthology.orchestra.run.smithi067.stdout:/proc 2024-04-22T23:20:54.150 INFO:teuthology.orchestra.run.smithi067.stdout:/sys 2024-04-22T23:20:54.150 INFO:teuthology.orchestra.run.smithi067.stdout:/dev 2024-04-22T23:20:54.150 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/security 2024-04-22T23:20:54.150 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/shm 2024-04-22T23:20:54.151 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/pts 2024-04-22T23:20:54.151 INFO:teuthology.orchestra.run.smithi067.stdout:/run 2024-04-22T23:20:54.151 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/cgroup 2024-04-22T23:20:54.151 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/pstore 2024-04-22T23:20:54.151 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/bpf 2024-04-22T23:20:54.151 INFO:teuthology.orchestra.run.smithi067.stdout:/ 2024-04-22T23:20:54.151 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/selinux 2024-04-22T23:20:54.151 INFO:teuthology.orchestra.run.smithi067.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T23:20:54.151 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/debug 2024-04-22T23:20:54.151 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/mqueue 2024-04-22T23:20:54.151 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/tracing 2024-04-22T23:20:54.151 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/hugepages 2024-04-22T23:20:54.151 INFO:teuthology.orchestra.run.smithi067.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T23:20:54.151 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/config 2024-04-22T23:20:54.152 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/fuse/connections 2024-04-22T23:20:54.152 INFO:teuthology.orchestra.run.smithi067.stdout:/run/credentials/systemd-sysctl.service 2024-04-22T23:20:54.152 INFO:teuthology.orchestra.run.smithi067.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-22T23:20:54.152 INFO:teuthology.orchestra.run.smithi067.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-22T23:20:54.152 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/nfs/rpc_pipefs 2024-04-22T23:20:54.152 INFO:teuthology.orchestra.run.smithi067.stdout:/run/user/1000 2024-04-22T23:20:54.152 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph 2024-04-22T23:20:54.152 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay 2024-04-22T23:20:54.152 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/dd063c941570b8893624bf2756f6167b20a0f7923a89e255625cec043d4c0ce8/merged 2024-04-22T23:20:54.152 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/0ed497479cfa3d5f2203eab6e274dd7a51156d589338b592816eb98c6112f53d/merged 2024-04-22T23:20:54.152 INFO:teuthology.orchestra.run.smithi067.stdout:/run/user/0 2024-04-22T23:20:54.152 INFO:teuthology.orchestra.run.smithi067.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T23:20:54.152 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/b55db4df0945f1997724f7d4e395c9769990949aae48edc3810fffa697cbd2c4/merged 2024-04-22T23:20:54.152 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/03959875da731f295a985adf361704682a4db861dcfa72407a465811b6482015/merged 2024-04-22T23:20:54.152 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/ddefdc05b9323d657acfa30f12e3fbbb12c799dbc34a408c35dc1c8a0452a70d/merged 2024-04-22T23:20:54.152 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/a2b2d7ef59ad8b102ad044c94759f6937647bd5ab8e09f42727ff8a8041f30f6/merged 2024-04-22T23:20:54.153 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/9d822c830791db31903fd880a40cb32609063a8b5b3495949baf6a5930172780/merged 2024-04-22T23:20:54.153 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/544845e3044b4a3d7addfe05369d23eee3b31f8f66ac5a07c626419009537d73/merged 2024-04-22T23:20:54.153 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/89689e2a4a6fc1e4778dbe5aceaf979da79426b3e87e255c06f1687509f2609e/merged 2024-04-22T23:20:54.153 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/699f4c7ebbf55b8d6b1399f2214620367287a9f6cb5126116a53ae8cb16bec78/merged 2024-04-22T23:20:54.153 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/a19f31f5dfb16cfcd7e3ce2cafa2e29fdd44bc1fef6a2a9184d6ac17531126fe/merged 2024-04-22T23:20:54.153 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/74d969c0250bdde6dd5bac0b5710f5c8f27314f4163a71996e6e64ae6c20d178/merged 2024-04-22T23:20:54.153 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/cdcab95b3abf4e5d7e01d6110902d9c37f254db8c1a9e673ca41ebc6c52c466e/merged 2024-04-22T23:20:54.153 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns 2024-04-22T23:20:54.153 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T23:20:54.153 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T23:20:54.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T23:20:54.153 DEBUG:teuthology.orchestra.run.smithi067:> ls /sys/fs/fuse/connections 2024-04-22T23:20:54.206 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-22T23:20:54.206 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-22T23:20:54.249 DEBUG:teuthology.orchestra.run.smithi067:> sudo modprobe fuse 2024-04-22T23:20:54.273 DEBUG:teuthology.orchestra.run.smithi067:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T23:20:54.330 INFO:teuthology.orchestra.run.smithi067.stdout:/proc 2024-04-22T23:20:54.330 INFO:teuthology.orchestra.run.smithi067.stdout:/sys 2024-04-22T23:20:54.330 INFO:teuthology.orchestra.run.smithi067.stdout:/dev 2024-04-22T23:20:54.330 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/security 2024-04-22T23:20:54.330 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/shm 2024-04-22T23:20:54.330 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/pts 2024-04-22T23:20:54.330 INFO:teuthology.orchestra.run.smithi067.stdout:/run 2024-04-22T23:20:54.330 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/cgroup 2024-04-22T23:20:54.330 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/pstore 2024-04-22T23:20:54.330 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/bpf 2024-04-22T23:20:54.331 INFO:teuthology.orchestra.run.smithi067.stdout:/ 2024-04-22T23:20:54.331 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/selinux 2024-04-22T23:20:54.331 INFO:teuthology.orchestra.run.smithi067.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T23:20:54.331 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/debug 2024-04-22T23:20:54.331 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/mqueue 2024-04-22T23:20:54.331 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/tracing 2024-04-22T23:20:54.331 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/hugepages 2024-04-22T23:20:54.331 INFO:teuthology.orchestra.run.smithi067.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T23:20:54.331 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/config 2024-04-22T23:20:54.331 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/fuse/connections 2024-04-22T23:20:54.331 INFO:teuthology.orchestra.run.smithi067.stdout:/run/credentials/systemd-sysctl.service 2024-04-22T23:20:54.331 INFO:teuthology.orchestra.run.smithi067.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-22T23:20:54.331 INFO:teuthology.orchestra.run.smithi067.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-22T23:20:54.331 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/nfs/rpc_pipefs 2024-04-22T23:20:54.331 INFO:teuthology.orchestra.run.smithi067.stdout:/run/user/1000 2024-04-22T23:20:54.332 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph 2024-04-22T23:20:54.332 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay 2024-04-22T23:20:54.332 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/dd063c941570b8893624bf2756f6167b20a0f7923a89e255625cec043d4c0ce8/merged 2024-04-22T23:20:54.332 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/0ed497479cfa3d5f2203eab6e274dd7a51156d589338b592816eb98c6112f53d/merged 2024-04-22T23:20:54.332 INFO:teuthology.orchestra.run.smithi067.stdout:/run/user/0 2024-04-22T23:20:54.332 INFO:teuthology.orchestra.run.smithi067.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T23:20:54.332 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/b55db4df0945f1997724f7d4e395c9769990949aae48edc3810fffa697cbd2c4/merged 2024-04-22T23:20:54.332 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/03959875da731f295a985adf361704682a4db861dcfa72407a465811b6482015/merged 2024-04-22T23:20:54.332 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/ddefdc05b9323d657acfa30f12e3fbbb12c799dbc34a408c35dc1c8a0452a70d/merged 2024-04-22T23:20:54.332 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/a2b2d7ef59ad8b102ad044c94759f6937647bd5ab8e09f42727ff8a8041f30f6/merged 2024-04-22T23:20:54.332 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/9d822c830791db31903fd880a40cb32609063a8b5b3495949baf6a5930172780/merged 2024-04-22T23:20:54.332 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/544845e3044b4a3d7addfe05369d23eee3b31f8f66ac5a07c626419009537d73/merged 2024-04-22T23:20:54.332 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/89689e2a4a6fc1e4778dbe5aceaf979da79426b3e87e255c06f1687509f2609e/merged 2024-04-22T23:20:54.332 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/699f4c7ebbf55b8d6b1399f2214620367287a9f6cb5126116a53ae8cb16bec78/merged 2024-04-22T23:20:54.332 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/a19f31f5dfb16cfcd7e3ce2cafa2e29fdd44bc1fef6a2a9184d6ac17531126fe/merged 2024-04-22T23:20:54.333 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/74d969c0250bdde6dd5bac0b5710f5c8f27314f4163a71996e6e64ae6c20d178/merged 2024-04-22T23:20:54.333 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/containers/storage/overlay/cdcab95b3abf4e5d7e01d6110902d9c37f254db8c1a9e673ca41ebc6c52c466e/merged 2024-04-22T23:20:54.333 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns 2024-04-22T23:20:54.333 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T23:20:54.333 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T23:20:54.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T23:20:54.333 DEBUG:teuthology.orchestra.run.smithi067:> ls /sys/fs/fuse/connections 2024-04-22T23:20:54.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi067.stderr:2024-04-22T23:20:54.336+0000 7f788b836480 -1 init, newargv = 0x556c4365d490 newargc=15 2024-04-22T23:20:54.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi067.stderr:ceph-fuse[75366]: starting ceph client 2024-04-22T23:20:54.345 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi067.stderr:ceph-fuse[75366]: starting fuse 2024-04-22T23:20:54.353 INFO:teuthology.orchestra.run.smithi067.stdout:156 2024-04-22T23:20:54.353 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [156] 2024-04-22T23:20:54.353 DEBUG:teuthology.orchestra.run.smithi067:> sudo stdin-killer -- python3 -c ' 2024-04-22T23:20:54.353 DEBUG:teuthology.orchestra.run.smithi067:> import glob 2024-04-22T23:20:54.353 DEBUG:teuthology.orchestra.run.smithi067:> import re 2024-04-22T23:20:54.353 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-04-22T23:20:54.353 DEBUG:teuthology.orchestra.run.smithi067:> import subprocess 2024-04-22T23:20:54.353 DEBUG:teuthology.orchestra.run.smithi067:> 2024-04-22T23:20:54.353 DEBUG:teuthology.orchestra.run.smithi067:> def _find_admin_socket(client_name): 2024-04-22T23:20:54.353 DEBUG:teuthology.orchestra.run.smithi067:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-22T23:20:54.353 DEBUG:teuthology.orchestra.run.smithi067:> files = glob.glob(asok_path) 2024-04-22T23:20:54.354 DEBUG:teuthology.orchestra.run.smithi067:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-22T23:20:54.354 DEBUG:teuthology.orchestra.run.smithi067:> 2024-04-22T23:20:54.354 DEBUG:teuthology.orchestra.run.smithi067:> # Given a non-glob path, it better be there 2024-04-22T23:20:54.354 DEBUG:teuthology.orchestra.run.smithi067:> if "*" not in asok_path: 2024-04-22T23:20:54.354 DEBUG:teuthology.orchestra.run.smithi067:> assert(len(files) == 1) 2024-04-22T23:20:54.354 DEBUG:teuthology.orchestra.run.smithi067:> return files[0] 2024-04-22T23:20:54.354 DEBUG:teuthology.orchestra.run.smithi067:> 2024-04-22T23:20:54.354 DEBUG:teuthology.orchestra.run.smithi067:> for f in files: 2024-04-22T23:20:54.354 DEBUG:teuthology.orchestra.run.smithi067:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-22T23:20:54.354 DEBUG:teuthology.orchestra.run.smithi067:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-22T23:20:54.354 DEBUG:teuthology.orchestra.run.smithi067:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-22T23:20:54.354 DEBUG:teuthology.orchestra.run.smithi067:> contents = proc_f.read() 2024-04-22T23:20:54.354 DEBUG:teuthology.orchestra.run.smithi067:> if mountpoint in contents: 2024-04-22T23:20:54.354 DEBUG:teuthology.orchestra.run.smithi067:> return f 2024-04-22T23:20:54.354 DEBUG:teuthology.orchestra.run.smithi067:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-22T23:20:54.355 DEBUG:teuthology.orchestra.run.smithi067:> 2024-04-22T23:20:54.355 DEBUG:teuthology.orchestra.run.smithi067:> print(_find_admin_socket("client.0")) 2024-04-22T23:20:54.355 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-04-22T23:20:54.473 INFO:teuthology.orchestra.run.smithi067.stdout:/var/run/ceph/ceph-client.0.75366.asok 2024-04-22T23:20:54.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T23:20:54.483 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.75366.asok 2024-04-22T23:20:54.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T23:20:54.484 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.75366.asok status 2024-04-22T23:20:54.621 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:20:54.621 INFO:teuthology.orchestra.run.smithi067.stdout: "metadata": { 2024-04-22T23:20:54.621 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_sha1": "7fe91d5d5842e04be3b4f514d6dd990c54b29c76", 2024-04-22T23:20:54.621 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_version": "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)", 2024-04-22T23:20:54.621 INFO:teuthology.orchestra.run.smithi067.stdout: "entity_id": "0", 2024-04-22T23:20:54.621 INFO:teuthology.orchestra.run.smithi067.stdout: "hostname": "smithi067", 2024-04-22T23:20:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-22T23:20:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: "pid": "75366", 2024-04-22T23:20:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: "root": "/" 2024-04-22T23:20:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:20:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: "dentry_count": 0, 2024-04-22T23:20:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: "dentry_pinned_count": 0, 2024-04-22T23:20:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 14536, 2024-04-22T23:20:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: "inst": { 2024-04-22T23:20:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: "name": { 2024-04-22T23:20:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "client", 2024-04-22T23:20:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: "num": 14536 2024-04-22T23:20:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:20:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": { 2024-04-22T23:20:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-22T23:20:54.622 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "192.168.0.1:0", 2024-04-22T23:20:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 523873030 2024-04-22T23:20:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:20:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:20:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": { 2024-04-22T23:20:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-22T23:20:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "192.168.0.1:0", 2024-04-22T23:20:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 523873030 2024-04-22T23:20:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:20:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: "inst_str": "client.14536 192.168.0.1:0/523873030", 2024-04-22T23:20:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: "addr_str": "192.168.0.1:0/523873030", 2024-04-22T23:20:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: "inode_count": 1, 2024-04-22T23:20:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_epoch": 18, 2024-04-22T23:20:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_epoch": 46, 2024-04-22T23:20:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_epoch_barrier": 0, 2024-04-22T23:20:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: "blocklisted": false, 2024-04-22T23:20:54.624 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs" 2024-04-22T23:20:54.624 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:20:54.630 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-22T23:20:54.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:54 smithi067 ceph-mon[26453]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.1 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-22T23:20:54.842 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:20:54.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:54 smithi134 ceph-mon[34609]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.1 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-22T23:20:56.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:56 smithi067 ceph-mon[26453]: pgmap v193: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.1 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-22T23:20:56.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.362+0000 7fb2b35fc700 1 -- 172.21.15.67:0/874734413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2ac106cd0 msgr2=0x7fb2ac1070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:56.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.362+0000 7fb2b35fc700 1 --2- 172.21.15.67:0/874734413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2ac106cd0 0x7fb2ac1070b0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb29c00b0a0 tx=0x7fb29c00b3b0 comp rx=0 tx=0).stop 2024-04-22T23:20:56.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.362+0000 7fb2b35fc700 1 -- 172.21.15.67:0/874734413 shutdown_connections 2024-04-22T23:20:56.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.362+0000 7fb2b35fc700 1 --2- 172.21.15.67:0/874734413 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2ac068580 0x7fb2ac0689e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:56.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.362+0000 7fb2b35fc700 1 --2- 172.21.15.67:0/874734413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2ac106cd0 0x7fb2ac1070b0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:56.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.362+0000 7fb2b35fc700 1 -- 172.21.15.67:0/874734413 >> 172.21.15.67:0/874734413 conn(0x7fb2ac075670 msgr2=0x7fb2ac075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:56.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.362+0000 7fb2b35fc700 1 -- 172.21.15.67:0/874734413 shutdown_connections 2024-04-22T23:20:56.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.362+0000 7fb2b35fc700 1 -- 172.21.15.67:0/874734413 wait complete. 2024-04-22T23:20:56.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.363+0000 7fb2b35fc700 1 Processor -- start 2024-04-22T23:20:56.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.363+0000 7fb2b35fc700 1 -- start start 2024-04-22T23:20:56.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.363+0000 7fb2b35fc700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2ac068580 0x7fb2ac0ffcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:56.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.363+0000 7fb2b35fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2ac106cd0 0x7fb2ac100230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:56.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.363+0000 7fb2b35fc700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ac100770 con 0x7fb2ac106cd0 2024-04-22T23:20:56.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.363+0000 7fb2b35fc700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ac1008b0 con 0x7fb2ac068580 2024-04-22T23:20:56.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.363+0000 7fb2b0b97700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2ac106cd0 0x7fb2ac100230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:56.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.363+0000 7fb2b0b97700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2ac106cd0 0x7fb2ac100230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58706/0 (socket says 172.21.15.67:58706) 2024-04-22T23:20:56.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.363+0000 7fb2b0b97700 1 -- 172.21.15.67:0/2511482913 learned_addr learned my addr 172.21.15.67:0/2511482913 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:20:56.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.363+0000 7fb2b1398700 1 --2- 172.21.15.67:0/2511482913 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2ac068580 0x7fb2ac0ffcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:56.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.363+0000 7fb2b0b97700 1 -- 172.21.15.67:0/2511482913 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2ac068580 msgr2=0x7fb2ac0ffcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:56.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.363+0000 7fb2b0b97700 1 --2- 172.21.15.67:0/2511482913 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2ac068580 0x7fb2ac0ffcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:56.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.363+0000 7fb2b0b97700 1 -- 172.21.15.67:0/2511482913 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb29c009d00 con 0x7fb2ac106cd0 2024-04-22T23:20:56.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.364+0000 7fb2b0b97700 1 --2- 172.21.15.67:0/2511482913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2ac106cd0 0x7fb2ac100230 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb2a000b4b0 tx=0x7fb2a000b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:56.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.364+0000 7fb2aa7fc700 1 -- 172.21.15.67:0/2511482913 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2a0011800 con 0x7fb2ac106cd0 2024-04-22T23:20:56.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.364+0000 7fb2aa7fc700 1 -- 172.21.15.67:0/2511482913 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fb2a0011e10 con 0x7fb2ac106cd0 2024-04-22T23:20:56.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.364+0000 7fb2b35fc700 1 -- 172.21.15.67:0/2511482913 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2ac1a2b50 con 0x7fb2ac106cd0 2024-04-22T23:20:56.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.364+0000 7fb2aa7fc700 1 -- 172.21.15.67:0/2511482913 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2a000f2e0 con 0x7fb2ac106cd0 2024-04-22T23:20:56.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.364+0000 7fb2b35fc700 1 -- 172.21.15.67:0/2511482913 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2ac1a2f60 con 0x7fb2ac106cd0 2024-04-22T23:20:56.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.365+0000 7fb2b35fc700 1 -- 172.21.15.67:0/2511482913 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2ac1091d0 con 0x7fb2ac106cd0 2024-04-22T23:20:56.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.365+0000 7fb2aa7fc700 1 -- 172.21.15.67:0/2511482913 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fb2a0010380 con 0x7fb2ac106cd0 2024-04-22T23:20:56.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.366+0000 7fb2aa7fc700 1 --2- 172.21.15.67:0/2511482913 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fb29406c4e0 0x7fb29406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:20:56.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.366+0000 7fb2aa7fc700 1 -- 172.21.15.67:0/2511482913 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fb2a008b340 con 0x7fb2ac106cd0 2024-04-22T23:20:56.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.366+0000 7fb2b1398700 1 --2- 172.21.15.67:0/2511482913 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fb29406c4e0 0x7fb29406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:20:56.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.366+0000 7fb2b1398700 1 --2- 172.21.15.67:0/2511482913 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fb29406c4e0 0x7fb29406e9a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb29c000f80 tx=0x7fb29c0060c0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:20:56.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.371+0000 7fb2aa7fc700 1 -- 172.21.15.67:0/2511482913 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb2a0055b40 con 0x7fb2ac106cd0 2024-04-22T23:20:56.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.614+0000 7fb2b35fc700 1 -- 172.21.15.67:0/2511482913 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb2ac04f2e0 con 0x7fb2ac106cd0 2024-04-22T23:20:56.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.614+0000 7fb2aa7fc700 1 -- 172.21.15.67:0/2511482913 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) v1 ==== 53+0+83 (secure 0 0 0) 0x7fb2a0059160 con 0x7fb2ac106cd0 2024-04-22T23:20:56.616 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-04-22T23:20:56.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.616+0000 7fb2b35fc700 1 -- 172.21.15.67:0/2511482913 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fb29406c4e0 msgr2=0x7fb29406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:56.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.616+0000 7fb2b35fc700 1 --2- 172.21.15.67:0/2511482913 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fb29406c4e0 0x7fb29406e9a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb29c000f80 tx=0x7fb29c0060c0 comp rx=0 tx=0).stop 2024-04-22T23:20:56.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.616+0000 7fb2b35fc700 1 -- 172.21.15.67:0/2511482913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2ac106cd0 msgr2=0x7fb2ac100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:20:56.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.616+0000 7fb2b35fc700 1 --2- 172.21.15.67:0/2511482913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2ac106cd0 0x7fb2ac100230 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb2a000b4b0 tx=0x7fb2a000b7c0 comp rx=0 tx=0).stop 2024-04-22T23:20:56.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.616+0000 7fb2b35fc700 1 -- 172.21.15.67:0/2511482913 shutdown_connections 2024-04-22T23:20:56.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.616+0000 7fb2b35fc700 1 --2- 172.21.15.67:0/2511482913 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fb29406c4e0 0x7fb29406e9a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:56.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.616+0000 7fb2b35fc700 1 --2- 172.21.15.67:0/2511482913 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2ac068580 0x7fb2ac0ffcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:56.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.616+0000 7fb2b35fc700 1 --2- 172.21.15.67:0/2511482913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2ac106cd0 0x7fb2ac100230 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:20:56.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.616+0000 7fb2b35fc700 1 -- 172.21.15.67:0/2511482913 >> 172.21.15.67:0/2511482913 conn(0x7fb2ac075670 msgr2=0x7fb2ac0fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:20:56.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.616+0000 7fb2b35fc700 1 -- 172.21.15.67:0/2511482913 shutdown_connections 2024-04-22T23:20:56.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:20:56.616+0000 7fb2b35fc700 1 -- 172.21.15.67:0/2511482913 wait complete. 2024-04-22T23:20:56.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:56 smithi134 ceph-mon[34609]: pgmap v193: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.1 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-22T23:20:57.185 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-22T23:20:57.185 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-22T23:20:57.185 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi134.front.sepia.ceph.com 2024-04-22T23:20:57.185 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-22T23:20:57.185 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-22T23:20:57.185 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-22T23:20:57.185 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-22T23:20:57.185 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-22T23:20:57.185 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-22T23:20:57.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T23:20:57.185 DEBUG:teuthology.orchestra.run.smithi134:> ip addr 2024-04-22T23:20:57.204 INFO:teuthology.orchestra.run.smithi134.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-22T23:20:57.204 INFO:teuthology.orchestra.run.smithi134.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-22T23:20:57.204 INFO:teuthology.orchestra.run.smithi134.stdout: inet 127.0.0.1/8 scope host lo 2024-04-22T23:20:57.204 INFO:teuthology.orchestra.run.smithi134.stdout: valid_lft forever preferred_lft forever 2024-04-22T23:20:57.204 INFO:teuthology.orchestra.run.smithi134.stdout: inet6 ::1/128 scope host 2024-04-22T23:20:57.204 INFO:teuthology.orchestra.run.smithi134.stdout: valid_lft forever preferred_lft forever 2024-04-22T23:20:57.205 INFO:teuthology.orchestra.run.smithi134.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T23:20:57.205 INFO:teuthology.orchestra.run.smithi134.stdout: link/ether 0c:c4:7a:d9:38:e6 brd ff:ff:ff:ff:ff:ff 2024-04-22T23:20:57.205 INFO:teuthology.orchestra.run.smithi134.stdout: altname enp7s0f0 2024-04-22T23:20:57.205 INFO:teuthology.orchestra.run.smithi134.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T23:20:57.205 INFO:teuthology.orchestra.run.smithi134.stdout: link/ether 0c:c4:7a:d9:38:e7 brd ff:ff:ff:ff:ff:ff 2024-04-22T23:20:57.205 INFO:teuthology.orchestra.run.smithi134.stdout: altname enp7s0f1 2024-04-22T23:20:57.205 INFO:teuthology.orchestra.run.smithi134.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T23:20:57.205 INFO:teuthology.orchestra.run.smithi134.stdout: link/ether 0c:c4:7a:88:6d:e8 brd ff:ff:ff:ff:ff:ff 2024-04-22T23:20:57.205 INFO:teuthology.orchestra.run.smithi134.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-22T23:20:57.205 INFO:teuthology.orchestra.run.smithi134.stdout: link/ether 0c:c4:7a:88:6d:e9 brd ff:ff:ff:ff:ff:ff 2024-04-22T23:20:57.205 INFO:teuthology.orchestra.run.smithi134.stdout: inet 172.21.15.134/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-22T23:20:57.205 INFO:teuthology.orchestra.run.smithi134.stdout: valid_lft 42282sec preferred_lft 42282sec 2024-04-22T23:20:57.205 INFO:teuthology.orchestra.run.smithi134.stdout: inet6 fe80::7a11:ede5:a5a6:982e/64 scope link noprefixroute 2024-04-22T23:20:57.205 INFO:teuthology.orchestra.run.smithi134.stdout: valid_lft forever preferred_lft forever 2024-04-22T23:20:57.206 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-22T23:20:57.206 DEBUG:teuthology.orchestra.run.smithi134:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T23:20:57.206 DEBUG:teuthology.orchestra.run.smithi134:> set -e 2024-04-22T23:20:57.206 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip link add name ceph-brx type bridge 2024-04-22T23:20:57.206 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip addr flush dev ceph-brx 2024-04-22T23:20:57.206 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip link set ceph-brx up 2024-04-22T23:20:57.206 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-22T23:20:57.206 DEBUG:teuthology.orchestra.run.smithi134:> ') 2024-04-22T23:20:57.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:20:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T23:20:57.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:20:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T23:20:57.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T23:20:57.401 DEBUG:teuthology.orchestra.run.smithi134:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-22T23:20:57.424 INFO:teuthology.orchestra.run.smithi134.stdout:1 2024-04-22T23:20:57.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T23:20:57.426 DEBUG:teuthology.orchestra.run.smithi134:> route 2024-04-22T23:20:57.488 INFO:teuthology.orchestra.run.smithi134.stdout:Kernel IP routing table 2024-04-22T23:20:57.488 INFO:teuthology.orchestra.run.smithi134.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-22T23:20:57.489 INFO:teuthology.orchestra.run.smithi134.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-22T23:20:57.489 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-22T23:20:57.489 INFO:teuthology.orchestra.run.smithi134.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-22T23:20:57.489 DEBUG:teuthology.orchestra.run.smithi134:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T23:20:57.489 DEBUG:teuthology.orchestra.run.smithi134:> set -e 2024-04-22T23:20:57.489 DEBUG:teuthology.orchestra.run.smithi134:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-22T23:20:57.489 DEBUG:teuthology.orchestra.run.smithi134:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-22T23:20:57.489 DEBUG:teuthology.orchestra.run.smithi134:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-22T23:20:57.489 DEBUG:teuthology.orchestra.run.smithi134:> ') 2024-04-22T23:20:57.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:57 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/2511482913' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-22T23:20:57.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:20:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T23:20:57.583 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:57 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/2511482913' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-22T23:20:57.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:20:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T23:20:57.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T23:20:57.754 DEBUG:teuthology.orchestra.run.smithi134:> ip netns list 2024-04-22T23:20:57.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T23:20:57.772 DEBUG:teuthology.orchestra.run.smithi134:> ip netns list-id 2024-04-22T23:20:57.826 DEBUG:teuthology.orchestra.run.smithi134:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T23:20:57.827 DEBUG:teuthology.orchestra.run.smithi134:> set -e 2024-04-22T23:20:57.827 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T23:20:57.827 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-22T23:20:57.827 DEBUG:teuthology.orchestra.run.smithi134:> ') 2024-04-22T23:20:57.914 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:20:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T23:20:57.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:20:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T23:20:57.942 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-22T23:20:57.942 DEBUG:teuthology.orchestra.run.smithi134:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T23:20:57.942 DEBUG:teuthology.orchestra.run.smithi134:> set -e 2024-04-22T23:20:57.942 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-22T23:20:57.942 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-22T23:20:57.942 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-22T23:20:57.942 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-22T23:20:57.942 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-22T23:20:57.943 DEBUG:teuthology.orchestra.run.smithi134:> ') 2024-04-22T23:20:57.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:20:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T23:20:58.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:20:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T23:20:58.064 DEBUG:teuthology.orchestra.run.smithi134:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T23:20:58.064 DEBUG:teuthology.orchestra.run.smithi134:> set -e 2024-04-22T23:20:58.064 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip link set brx.0 up 2024-04-22T23:20:58.064 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip link set dev brx.0 master ceph-brx 2024-04-22T23:20:58.064 DEBUG:teuthology.orchestra.run.smithi134:> ') 2024-04-22T23:20:58.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:20:58 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T23:20:58.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:20:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T23:20:58.138 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-22T23:20:58.138 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T23:20:58.138 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-22T23:20:58.162 INFO:teuthology.orchestra.run.smithi134.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-22T23:20:58.163 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T23:20:58.163 DEBUG:teuthology.orchestra.run.smithi134:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-22T23:20:58.217 DEBUG:teuthology.orchestra.run.smithi134:> sudo modprobe fuse 2024-04-22T23:20:58.281 DEBUG:teuthology.orchestra.run.smithi134:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T23:20:58.338 INFO:teuthology.orchestra.run.smithi134.stdout:/proc 2024-04-22T23:20:58.338 INFO:teuthology.orchestra.run.smithi134.stdout:/sys 2024-04-22T23:20:58.338 INFO:teuthology.orchestra.run.smithi134.stdout:/dev 2024-04-22T23:20:58.338 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/kernel/security 2024-04-22T23:20:58.338 INFO:teuthology.orchestra.run.smithi134.stdout:/dev/shm 2024-04-22T23:20:58.338 INFO:teuthology.orchestra.run.smithi134.stdout:/dev/pts 2024-04-22T23:20:58.338 INFO:teuthology.orchestra.run.smithi134.stdout:/run 2024-04-22T23:20:58.338 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/fs/cgroup 2024-04-22T23:20:58.339 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/fs/pstore 2024-04-22T23:20:58.339 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/fs/bpf 2024-04-22T23:20:58.339 INFO:teuthology.orchestra.run.smithi134.stdout:/ 2024-04-22T23:20:58.339 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/fs/selinux 2024-04-22T23:20:58.339 INFO:teuthology.orchestra.run.smithi134.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T23:20:58.339 INFO:teuthology.orchestra.run.smithi134.stdout:/dev/hugepages 2024-04-22T23:20:58.339 INFO:teuthology.orchestra.run.smithi134.stdout:/dev/mqueue 2024-04-22T23:20:58.339 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/kernel/debug 2024-04-22T23:20:58.339 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/kernel/tracing 2024-04-22T23:20:58.339 INFO:teuthology.orchestra.run.smithi134.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T23:20:58.339 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/kernel/config 2024-04-22T23:20:58.339 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/fs/fuse/connections 2024-04-22T23:20:58.339 INFO:teuthology.orchestra.run.smithi134.stdout:/run/credentials/systemd-sysctl.service 2024-04-22T23:20:58.339 INFO:teuthology.orchestra.run.smithi134.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-22T23:20:58.339 INFO:teuthology.orchestra.run.smithi134.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-22T23:20:58.340 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/nfs/rpc_pipefs 2024-04-22T23:20:58.340 INFO:teuthology.orchestra.run.smithi134.stdout:/run/user/1000 2024-04-22T23:20:58.340 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/ceph 2024-04-22T23:20:58.340 INFO:teuthology.orchestra.run.smithi134.stdout:/run/user/0 2024-04-22T23:20:58.340 INFO:teuthology.orchestra.run.smithi134.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T23:20:58.340 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay 2024-04-22T23:20:58.340 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/3d7b3019808cf94d2469087bdec56bd3d1722d077d6a8d8fa687d72d860860e6/merged 2024-04-22T23:20:58.340 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/ec198691bbe1426483bdfe784cee4c5df1e67546e7f3c06684a346c0671a74d0/merged 2024-04-22T23:20:58.340 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/00887d84abe7bbdfe4e8a4071121b19d51070340a8f87fd942c5bcb4510085fe/merged 2024-04-22T23:20:58.340 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/b67efc626ac78c21a8c521a838137d57e32b454a322b00dc1a36255e25ab2d90/merged 2024-04-22T23:20:58.340 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/017d865a5effd7e0d0accac9bb0f0215cd126b99557e225bbbce25041fcba841/merged 2024-04-22T23:20:58.340 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/08deba94c9d0e6af395825f37176a79883e45749b66f2e0ed191110e0b46758e/merged 2024-04-22T23:20:58.340 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/a34651e397dc186eb2b4928859be7e822b4aeae44a587c835f4b212a3767211a/merged 2024-04-22T23:20:58.340 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/2b45c0bc09e76f64c6b8a104006da4b83bab48228e0230d562dfb4dd5ffb5ede/merged 2024-04-22T23:20:58.341 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/63e2dd59cb90edb84c1daaa26aad54157447cafdecde89e0ceedf2a44982874c/merged 2024-04-22T23:20:58.341 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/ec31229227fe1fe93c34c07ecf8feae88bd54a2c783f04fa8cbfb7de68a9291c/merged 2024-04-22T23:20:58.341 INFO:teuthology.orchestra.run.smithi134.stdout:/run/netns 2024-04-22T23:20:58.341 INFO:teuthology.orchestra.run.smithi134.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T23:20:58.341 INFO:teuthology.orchestra.run.smithi134.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T23:20:58.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T23:20:58.341 DEBUG:teuthology.orchestra.run.smithi134:> ls /sys/fs/fuse/connections 2024-04-22T23:20:58.393 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-22T23:20:58.394 DEBUG:teuthology.orchestra.run.smithi134:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-04-22T23:20:58.437 DEBUG:teuthology.orchestra.run.smithi134:> sudo modprobe fuse 2024-04-22T23:20:58.460 DEBUG:teuthology.orchestra.run.smithi134:> cat /proc/self/mounts | awk '{print $2}' 2024-04-22T23:20:58.516 INFO:teuthology.orchestra.run.smithi134.stdout:/proc 2024-04-22T23:20:58.516 INFO:teuthology.orchestra.run.smithi134.stdout:/sys 2024-04-22T23:20:58.516 INFO:teuthology.orchestra.run.smithi134.stdout:/dev 2024-04-22T23:20:58.516 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/kernel/security 2024-04-22T23:20:58.516 INFO:teuthology.orchestra.run.smithi134.stdout:/dev/shm 2024-04-22T23:20:58.517 INFO:teuthology.orchestra.run.smithi134.stdout:/dev/pts 2024-04-22T23:20:58.517 INFO:teuthology.orchestra.run.smithi134.stdout:/run 2024-04-22T23:20:58.517 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/fs/cgroup 2024-04-22T23:20:58.517 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/fs/pstore 2024-04-22T23:20:58.517 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/fs/bpf 2024-04-22T23:20:58.517 INFO:teuthology.orchestra.run.smithi134.stdout:/ 2024-04-22T23:20:58.517 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/fs/selinux 2024-04-22T23:20:58.517 INFO:teuthology.orchestra.run.smithi134.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T23:20:58.517 INFO:teuthology.orchestra.run.smithi134.stdout:/dev/hugepages 2024-04-22T23:20:58.517 INFO:teuthology.orchestra.run.smithi134.stdout:/dev/mqueue 2024-04-22T23:20:58.517 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/kernel/debug 2024-04-22T23:20:58.517 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/kernel/tracing 2024-04-22T23:20:58.517 INFO:teuthology.orchestra.run.smithi134.stdout:/run/credentials/systemd-sysusers.service 2024-04-22T23:20:58.517 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/kernel/config 2024-04-22T23:20:58.517 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/fs/fuse/connections 2024-04-22T23:20:58.517 INFO:teuthology.orchestra.run.smithi134.stdout:/run/credentials/systemd-sysctl.service 2024-04-22T23:20:58.518 INFO:teuthology.orchestra.run.smithi134.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-22T23:20:58.518 INFO:teuthology.orchestra.run.smithi134.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-22T23:20:58.518 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/nfs/rpc_pipefs 2024-04-22T23:20:58.518 INFO:teuthology.orchestra.run.smithi134.stdout:/run/user/1000 2024-04-22T23:20:58.518 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/ceph 2024-04-22T23:20:58.518 INFO:teuthology.orchestra.run.smithi134.stdout:/run/user/0 2024-04-22T23:20:58.518 INFO:teuthology.orchestra.run.smithi134.stdout:/proc/sys/fs/binfmt_misc 2024-04-22T23:20:58.518 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay 2024-04-22T23:20:58.518 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/3d7b3019808cf94d2469087bdec56bd3d1722d077d6a8d8fa687d72d860860e6/merged 2024-04-22T23:20:58.518 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/ec198691bbe1426483bdfe784cee4c5df1e67546e7f3c06684a346c0671a74d0/merged 2024-04-22T23:20:58.518 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/00887d84abe7bbdfe4e8a4071121b19d51070340a8f87fd942c5bcb4510085fe/merged 2024-04-22T23:20:58.518 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/b67efc626ac78c21a8c521a838137d57e32b454a322b00dc1a36255e25ab2d90/merged 2024-04-22T23:20:58.518 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/017d865a5effd7e0d0accac9bb0f0215cd126b99557e225bbbce25041fcba841/merged 2024-04-22T23:20:58.518 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/08deba94c9d0e6af395825f37176a79883e45749b66f2e0ed191110e0b46758e/merged 2024-04-22T23:20:58.519 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/a34651e397dc186eb2b4928859be7e822b4aeae44a587c835f4b212a3767211a/merged 2024-04-22T23:20:58.519 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/2b45c0bc09e76f64c6b8a104006da4b83bab48228e0230d562dfb4dd5ffb5ede/merged 2024-04-22T23:20:58.519 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/63e2dd59cb90edb84c1daaa26aad54157447cafdecde89e0ceedf2a44982874c/merged 2024-04-22T23:20:58.519 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/ec31229227fe1fe93c34c07ecf8feae88bd54a2c783f04fa8cbfb7de68a9291c/merged 2024-04-22T23:20:58.519 INFO:teuthology.orchestra.run.smithi134.stdout:/run/netns 2024-04-22T23:20:58.519 INFO:teuthology.orchestra.run.smithi134.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T23:20:58.519 INFO:teuthology.orchestra.run.smithi134.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-22T23:20:58.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T23:20:58.519 DEBUG:teuthology.orchestra.run.smithi134:> ls /sys/fs/fuse/connections 2024-04-22T23:20:58.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:20:58 smithi067 ceph-mon[26453]: pgmap v194: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 3.4 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-22T23:20:58.546 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi134.stderr:2024-04-22T23:20:58.544+0000 7f3c1b8b2480 -1 init, newargv = 0x55f622831d10 newargc=15 2024-04-22T23:20:58.546 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi134.stderr:ceph-fuse[60729]: starting ceph client 2024-04-22T23:20:58.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi134.stderr:ceph-fuse[60729]: starting fuse 2024-04-22T23:20:58.560 INFO:teuthology.orchestra.run.smithi134.stdout:123 2024-04-22T23:20:58.560 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [123] 2024-04-22T23:20:58.561 DEBUG:teuthology.orchestra.run.smithi134:> sudo stdin-killer -- python3 -c ' 2024-04-22T23:20:58.561 DEBUG:teuthology.orchestra.run.smithi134:> import glob 2024-04-22T23:20:58.561 DEBUG:teuthology.orchestra.run.smithi134:> import re 2024-04-22T23:20:58.561 DEBUG:teuthology.orchestra.run.smithi134:> import os 2024-04-22T23:20:58.561 DEBUG:teuthology.orchestra.run.smithi134:> import subprocess 2024-04-22T23:20:58.561 DEBUG:teuthology.orchestra.run.smithi134:> 2024-04-22T23:20:58.561 DEBUG:teuthology.orchestra.run.smithi134:> def _find_admin_socket(client_name): 2024-04-22T23:20:58.561 DEBUG:teuthology.orchestra.run.smithi134:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-22T23:20:58.561 DEBUG:teuthology.orchestra.run.smithi134:> files = glob.glob(asok_path) 2024-04-22T23:20:58.561 DEBUG:teuthology.orchestra.run.smithi134:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-22T23:20:58.562 DEBUG:teuthology.orchestra.run.smithi134:> 2024-04-22T23:20:58.562 DEBUG:teuthology.orchestra.run.smithi134:> # Given a non-glob path, it better be there 2024-04-22T23:20:58.562 DEBUG:teuthology.orchestra.run.smithi134:> if "*" not in asok_path: 2024-04-22T23:20:58.562 DEBUG:teuthology.orchestra.run.smithi134:> assert(len(files) == 1) 2024-04-22T23:20:58.562 DEBUG:teuthology.orchestra.run.smithi134:> return files[0] 2024-04-22T23:20:58.562 DEBUG:teuthology.orchestra.run.smithi134:> 2024-04-22T23:20:58.562 DEBUG:teuthology.orchestra.run.smithi134:> for f in files: 2024-04-22T23:20:58.562 DEBUG:teuthology.orchestra.run.smithi134:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-22T23:20:58.562 DEBUG:teuthology.orchestra.run.smithi134:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-22T23:20:58.562 DEBUG:teuthology.orchestra.run.smithi134:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-22T23:20:58.562 DEBUG:teuthology.orchestra.run.smithi134:> contents = proc_f.read() 2024-04-22T23:20:58.562 DEBUG:teuthology.orchestra.run.smithi134:> if mountpoint in contents: 2024-04-22T23:20:58.562 DEBUG:teuthology.orchestra.run.smithi134:> return f 2024-04-22T23:20:58.563 DEBUG:teuthology.orchestra.run.smithi134:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-22T23:20:58.563 DEBUG:teuthology.orchestra.run.smithi134:> 2024-04-22T23:20:58.563 DEBUG:teuthology.orchestra.run.smithi134:> print(_find_admin_socket("client.1")) 2024-04-22T23:20:58.563 DEBUG:teuthology.orchestra.run.smithi134:> ' 2024-04-22T23:20:58.621 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:20:58 smithi134 ceph-mon[34609]: pgmap v194: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 3.4 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-22T23:20:58.677 INFO:teuthology.orchestra.run.smithi134.stdout:/var/run/ceph/ceph-client.1.60729.asok 2024-04-22T23:20:58.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-22T23:20:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T23:20:58.688 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.60729.asok 2024-04-22T23:20:58.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T23:20:58.688 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.60729.asok status 2024-04-22T23:20:58.819 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-04-22T23:20:58.819 INFO:teuthology.orchestra.run.smithi134.stdout: "metadata": { 2024-04-22T23:20:58.819 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph_sha1": "7fe91d5d5842e04be3b4f514d6dd990c54b29c76", 2024-04-22T23:20:58.819 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph_version": "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)", 2024-04-22T23:20:58.819 INFO:teuthology.orchestra.run.smithi134.stdout: "entity_id": "1", 2024-04-22T23:20:58.819 INFO:teuthology.orchestra.run.smithi134.stdout: "hostname": "smithi134", 2024-04-22T23:20:58.819 INFO:teuthology.orchestra.run.smithi134.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-22T23:20:58.819 INFO:teuthology.orchestra.run.smithi134.stdout: "pid": "60729", 2024-04-22T23:20:58.819 INFO:teuthology.orchestra.run.smithi134.stdout: "root": "/" 2024-04-22T23:20:58.820 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-22T23:20:58.820 INFO:teuthology.orchestra.run.smithi134.stdout: "dentry_count": 0, 2024-04-22T23:20:58.820 INFO:teuthology.orchestra.run.smithi134.stdout: "dentry_pinned_count": 0, 2024-04-22T23:20:58.820 INFO:teuthology.orchestra.run.smithi134.stdout: "id": 24321, 2024-04-22T23:20:58.820 INFO:teuthology.orchestra.run.smithi134.stdout: "inst": { 2024-04-22T23:20:58.820 INFO:teuthology.orchestra.run.smithi134.stdout: "name": { 2024-04-22T23:20:58.820 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "client", 2024-04-22T23:20:58.820 INFO:teuthology.orchestra.run.smithi134.stdout: "num": 24321 2024-04-22T23:20:58.820 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-22T23:20:58.820 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": { 2024-04-22T23:20:58.820 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-04-22T23:20:58.820 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "192.168.0.1:0", 2024-04-22T23:20:58.820 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 919997813 2024-04-22T23:20:58.820 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-04-22T23:20:58.820 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-22T23:20:58.821 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": { 2024-04-22T23:20:58.821 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-04-22T23:20:58.821 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "192.168.0.1:0", 2024-04-22T23:20:58.821 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 919997813 2024-04-22T23:20:58.821 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-04-22T23:20:58.821 INFO:teuthology.orchestra.run.smithi134.stdout: "inst_str": "client.24321 192.168.0.1:0/919997813", 2024-04-22T23:20:58.821 INFO:teuthology.orchestra.run.smithi134.stdout: "addr_str": "192.168.0.1:0/919997813", 2024-04-22T23:20:58.821 INFO:teuthology.orchestra.run.smithi134.stdout: "inode_count": 1, 2024-04-22T23:20:58.821 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_epoch": 18, 2024-04-22T23:20:58.821 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_epoch": 46, 2024-04-22T23:20:58.821 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_epoch_barrier": 0, 2024-04-22T23:20:58.821 INFO:teuthology.orchestra.run.smithi134.stdout: "blocklisted": false, 2024-04-22T23:20:58.821 INFO:teuthology.orchestra.run.smithi134.stdout: "fs_name": "cephfs" 2024-04-22T23:20:58.821 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-04-22T23:20:58.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T23:20:58.827 DEBUG:teuthology.orchestra.run.smithi067:> stat --file-system '--printf=%T 2024-04-22T23:20:58.827 DEBUG:teuthology.orchestra.run.smithi067:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-22T23:20:58.849 INFO:teuthology.orchestra.run.smithi067.stdout:fuseblk 2024-04-22T23:20:58.849 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-22T23:20:58.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T23:20:58.849 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-22T23:20:58.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T23:20:58.922 DEBUG:teuthology.orchestra.run.smithi134:> stat --file-system '--printf=%T 2024-04-22T23:20:58.922 DEBUG:teuthology.orchestra.run.smithi134:> ' -- /home/ubuntu/cephtest/mnt.1 2024-04-22T23:20:58.938 INFO:teuthology.orchestra.run.smithi134.stdout:fuseblk 2024-04-22T23:20:58.938 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-04-22T23:20:58.938 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T23:20:58.938 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-22T23:21:00.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:00 smithi067 ceph-mon[26453]: pgmap v195: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-04-22T23:21:00.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:00 smithi134 ceph-mon[34609]: pgmap v195: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-04-22T23:21:02.679 INFO:teuthology.run_tasks:Running task print... 2024-04-22T23:21:02.689 INFO:teuthology.task.print:**** done client 2024-04-22T23:21:02.690 INFO:teuthology.run_tasks:Running task parallel... 2024-04-22T23:21:02.699 INFO:teuthology.task.parallel:starting parallel... 2024-04-22T23:21:02.700 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-22T23:21:02.700 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-22T23:21:02.700 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi067.front.sepia.ceph.com 2024-04-22T23:21:02.701 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -e sha1=43be020184947e53516056c9931e1ac5bdbbb1a5 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-22T23:21:02.701 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-22T23:21:02.701 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-04-22T23:21:02.704 INFO:tasks.workunit:Pulling workunits from ref 43be020184947e53516056c9931e1ac5bdbbb1a5 2024-04-22T23:21:02.705 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-22T23:21:02.705 INFO:tasks.workunit:timeout=3h 2024-04-22T23:21:02.705 INFO:tasks.workunit:cleanup=True 2024-04-22T23:21:02.705 DEBUG:teuthology.orchestra.run.smithi067:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-22T23:21:02.726 INFO:teuthology.orchestra.run.smithi067.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-22T23:21:02.726 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-04-22T23:21:02.726 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 9ch/156d Inode: 1 Links: 2 2024-04-22T23:21:02.726 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T23:21:02.726 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:fusefs_t:s0 2024-04-22T23:21:02.726 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-04-22T23:21:02.726 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-04-22 23:20:27.273048027 +0000 2024-04-22T23:21:02.726 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-04-22 23:20:58.918681529 +0000 2024-04-22T23:21:02.727 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2024-04-22T23:21:02.727 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-22T23:21:02.727 DEBUG:teuthology.orchestra.run.smithi067:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-22T23:21:02.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:02 smithi067 ceph-mon[26453]: pgmap v196: 65 pgs: 65 active+clean; 582 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-22T23:21:02.799 DEBUG:teuthology.orchestra.run.smithi134:> stat -- /home/ubuntu/cephtest/mnt.1 2024-04-22T23:21:02.819 INFO:teuthology.orchestra.run.smithi134.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-04-22T23:21:02.819 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-04-22T23:21:02.819 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 7bh/123d Inode: 1 Links: 3 2024-04-22T23:21:02.819 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T23:21:02.819 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:fusefs_t:s0 2024-04-22T23:21:02.819 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-04-22T23:21:02.820 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-04-22 23:21:02.792371010 +0000 2024-04-22T23:21:02.820 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-04-22 23:21:02.792371010 +0000 2024-04-22T23:21:02.820 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: - 2024-04-22T23:21:02.820 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-04-22T23:21:02.820 DEBUG:teuthology.orchestra.run.smithi134:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-04-22T23:21:02.883 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:02 smithi134 ceph-mon[34609]: pgmap v196: 65 pgs: 65 active+clean; 582 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-22T23:21:02.893 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 43be020184947e53516056c9931e1ac5bdbbb1a5 2024-04-22T23:21:02.894 DEBUG:teuthology.orchestra.run.smithi134:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 43be020184947e53516056c9931e1ac5bdbbb1a5 2024-04-22T23:21:02.909 INFO:tasks.workunit.client.0.smithi067.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-22T23:21:02.934 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:21:02.951 INFO:tasks.workunit.client.1.smithi134.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-04-22T23:21:04.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.449+0000 7f48cbeec700 1 -- 172.21.15.67:0/479431555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48c4101750 msgr2=0x7f48c4103b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:21:04.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.449+0000 7f48cbeec700 1 --2- 172.21.15.67:0/479431555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48c4101750 0x7f48c4103b40 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f48c0009910 tx=0x7f48c0009c20 comp rx=0 tx=0).stop 2024-04-22T23:21:04.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.449+0000 7f48cbeec700 1 -- 172.21.15.67:0/479431555 shutdown_connections 2024-04-22T23:21:04.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.449+0000 7f48cbeec700 1 --2- 172.21.15.67:0/479431555 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f48c4104080 0x7f48c4106470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:04.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.449+0000 7f48cbeec700 1 --2- 172.21.15.67:0/479431555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48c4101750 0x7f48c4103b40 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:04.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.449+0000 7f48cbeec700 1 -- 172.21.15.67:0/479431555 >> 172.21.15.67:0/479431555 conn(0x7f48c40fb160 msgr2=0x7f48c40fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:21:04.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.449+0000 7f48cbeec700 1 -- 172.21.15.67:0/479431555 shutdown_connections 2024-04-22T23:21:04.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.449+0000 7f48cbeec700 1 -- 172.21.15.67:0/479431555 wait complete. 2024-04-22T23:21:04.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.450+0000 7f48cbeec700 1 Processor -- start 2024-04-22T23:21:04.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.450+0000 7f48cbeec700 1 -- start start 2024-04-22T23:21:04.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.450+0000 7f48cbeec700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f48c4101750 0x7f48c41968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:21:04.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.450+0000 7f48cbeec700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48c4104080 0x7f48c4196df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:21:04.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.450+0000 7f48cbeec700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c4197410 con 0x7f48c4104080 2024-04-22T23:21:04.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.450+0000 7f48cbeec700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c4197550 con 0x7f48c4101750 2024-04-22T23:21:04.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.450+0000 7f48c9487700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48c4104080 0x7f48c4196df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:21:04.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.450+0000 7f48c9487700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48c4104080 0x7f48c4196df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50408/0 (socket says 172.21.15.67:50408) 2024-04-22T23:21:04.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.450+0000 7f48c9487700 1 -- 172.21.15.67:0/2164161619 learned_addr learned my addr 172.21.15.67:0/2164161619 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:21:04.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.451+0000 7f48c9c88700 1 --2- 172.21.15.67:0/2164161619 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f48c4101750 0x7f48c41968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:21:04.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.451+0000 7f48c9487700 1 -- 172.21.15.67:0/2164161619 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f48c4101750 msgr2=0x7f48c41968b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:21:04.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.451+0000 7f48c9487700 1 --2- 172.21.15.67:0/2164161619 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f48c4101750 0x7f48c41968b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:04.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.451+0000 7f48c9487700 1 -- 172.21.15.67:0/2164161619 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c00095f0 con 0x7f48c4104080 2024-04-22T23:21:04.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.451+0000 7f48c9487700 1 --2- 172.21.15.67:0/2164161619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48c4104080 0x7f48c4196df0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f48b400b4b0 tx=0x7f48b400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:21:04.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.451+0000 7f48baffd700 1 -- 172.21.15.67:0/2164161619 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48b4011800 con 0x7f48c4104080 2024-04-22T23:21:04.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.451+0000 7f48baffd700 1 -- 172.21.15.67:0/2164161619 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f48b4011e10 con 0x7f48c4104080 2024-04-22T23:21:04.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.451+0000 7f48cbeec700 1 -- 172.21.15.67:0/2164161619 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48c419c000 con 0x7f48c4104080 2024-04-22T23:21:04.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.451+0000 7f48baffd700 1 -- 172.21.15.67:0/2164161619 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48b400f2e0 con 0x7f48c4104080 2024-04-22T23:21:04.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.451+0000 7f48cbeec700 1 -- 172.21.15.67:0/2164161619 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48c419c520 con 0x7f48c4104080 2024-04-22T23:21:04.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.452+0000 7f48cbeec700 1 -- 172.21.15.67:0/2164161619 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48c41906b0 con 0x7f48c4104080 2024-04-22T23:21:04.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.453+0000 7f48baffd700 1 -- 172.21.15.67:0/2164161619 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f48b4010380 con 0x7f48c4104080 2024-04-22T23:21:04.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.453+0000 7f48baffd700 1 --2- 172.21.15.67:0/2164161619 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f48b006c4e0 0x7f48b006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:21:04.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.453+0000 7f48baffd700 1 -- 172.21.15.67:0/2164161619 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f48b408b4d0 con 0x7f48c4104080 2024-04-22T23:21:04.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.453+0000 7f48c9c88700 1 --2- 172.21.15.67:0/2164161619 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f48b006c4e0 0x7f48b006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:21:04.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.454+0000 7f48c9c88700 1 --2- 172.21.15.67:0/2164161619 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f48b006c4e0 0x7f48b006e9a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f48c000b5c0 tx=0x7f48c00058e0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:21:04.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.458+0000 7f48baffd700 1 -- 172.21.15.67:0/2164161619 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f48b40596e0 con 0x7f48c4104080 2024-04-22T23:21:04.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.658+0000 7f48cbeec700 1 -- 172.21.15.67:0/2164161619 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7f48c4068af0 con 0x7f48c4104080 2024-04-22T23:21:04.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.658+0000 7f48baffd700 1 -- 172.21.15.67:0/2164161619 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7f48b4059270 con 0x7f48c4104080 2024-04-22T23:21:04.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.660+0000 7f48cbeec700 1 -- 172.21.15.67:0/2164161619 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f48b006c4e0 msgr2=0x7f48b006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:21:04.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.660+0000 7f48cbeec700 1 --2- 172.21.15.67:0/2164161619 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f48b006c4e0 0x7f48b006e9a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f48c000b5c0 tx=0x7f48c00058e0 comp rx=0 tx=0).stop 2024-04-22T23:21:04.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.660+0000 7f48cbeec700 1 -- 172.21.15.67:0/2164161619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48c4104080 msgr2=0x7f48c4196df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:21:04.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.660+0000 7f48cbeec700 1 --2- 172.21.15.67:0/2164161619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48c4104080 0x7f48c4196df0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f48b400b4b0 tx=0x7f48b400b870 comp rx=0 tx=0).stop 2024-04-22T23:21:04.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.660+0000 7f48cbeec700 1 -- 172.21.15.67:0/2164161619 shutdown_connections 2024-04-22T23:21:04.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.660+0000 7f48cbeec700 1 --2- 172.21.15.67:0/2164161619 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f48b006c4e0 0x7f48b006e9a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:04.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.660+0000 7f48cbeec700 1 --2- 172.21.15.67:0/2164161619 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f48c4101750 0x7f48c41968b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:04.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.660+0000 7f48cbeec700 1 --2- 172.21.15.67:0/2164161619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48c4104080 0x7f48c4196df0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:04.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.660+0000 7f48cbeec700 1 -- 172.21.15.67:0/2164161619 >> 172.21.15.67:0/2164161619 conn(0x7f48c40fb160 msgr2=0x7f48c40fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:21:04.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.660+0000 7f48cbeec700 1 -- 172.21.15.67:0/2164161619 shutdown_connections 2024-04-22T23:21:04.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:04.660+0000 7f48cbeec700 1 -- 172.21.15.67:0/2164161619 wait complete. 2024-04-22T23:21:04.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:04 smithi067 ceph-mon[26453]: pgmap v197: 65 pgs: 65 active+clean; 582 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 597 B/s wr, 4 op/s 2024-04-22T23:21:04.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:04 smithi134 ceph-mon[34609]: pgmap v197: 65 pgs: 65 active+clean; 582 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 597 B/s wr, 4 op/s 2024-04-22T23:21:05.280 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -e sha1=43be020184947e53516056c9931e1ac5bdbbb1a5 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-22T23:21:05.544 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:21:06.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:06 smithi067 ceph-mon[26453]: pgmap v198: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-22T23:21:06.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:06 smithi134 ceph-mon[34609]: pgmap v198: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-22T23:21:07.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.111+0000 7f352d453700 1 -- 172.21.15.67:0/3833412272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3528101eb0 msgr2=0x7f3528102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:21:07.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.111+0000 7f352d453700 1 --2- 172.21.15.67:0/3833412272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3528101eb0 0x7f3528102310 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f351c009910 tx=0x7f351c009c20 comp rx=0 tx=0).stop 2024-04-22T23:21:07.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.111+0000 7f352d453700 1 -- 172.21.15.67:0/3833412272 shutdown_connections 2024-04-22T23:21:07.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.111+0000 7f352d453700 1 --2- 172.21.15.67:0/3833412272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3528101eb0 0x7f3528102310 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:07.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.111+0000 7f352d453700 1 --2- 172.21.15.67:0/3833412272 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3528100cb0 0x7f35281010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:07.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.111+0000 7f352d453700 1 -- 172.21.15.67:0/3833412272 >> 172.21.15.67:0/3833412272 conn(0x7f35280fc230 msgr2=0x7f35280fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:21:07.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.111+0000 7f352d453700 1 -- 172.21.15.67:0/3833412272 shutdown_connections 2024-04-22T23:21:07.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.111+0000 7f352d453700 1 -- 172.21.15.67:0/3833412272 wait complete. 2024-04-22T23:21:07.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.111+0000 7f352d453700 1 Processor -- start 2024-04-22T23:21:07.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.112+0000 7f352d453700 1 -- start start 2024-04-22T23:21:07.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.112+0000 7f352d453700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3528100cb0 0x7f3528074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:21:07.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.112+0000 7f352d453700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3528101eb0 0x7f3528073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:21:07.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.112+0000 7f352d453700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35280736c0 con 0x7f3528101eb0 2024-04-22T23:21:07.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.112+0000 7f352d453700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3528073800 con 0x7f3528100cb0 2024-04-22T23:21:07.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.112+0000 7f35267fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3528101eb0 0x7f3528073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:21:07.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.112+0000 7f35267fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3528101eb0 0x7f3528073180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50414/0 (socket says 172.21.15.67:50414) 2024-04-22T23:21:07.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.112+0000 7f35267fc700 1 -- 172.21.15.67:0/1817011198 learned_addr learned my addr 172.21.15.67:0/1817011198 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:21:07.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.112+0000 7f3526ffd700 1 --2- 172.21.15.67:0/1817011198 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3528100cb0 0x7f3528074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:21:07.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.112+0000 7f35267fc700 1 -- 172.21.15.67:0/1817011198 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3528100cb0 msgr2=0x7f3528074b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:21:07.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.112+0000 7f35267fc700 1 --2- 172.21.15.67:0/1817011198 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3528100cb0 0x7f3528074b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:07.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.112+0000 7f35267fc700 1 -- 172.21.15.67:0/1817011198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f351c0095f0 con 0x7f3528101eb0 2024-04-22T23:21:07.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.112+0000 7f3526ffd700 1 --2- 172.21.15.67:0/1817011198 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3528100cb0 0x7f3528074b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:21:07.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.112+0000 7f35267fc700 1 --2- 172.21.15.67:0/1817011198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3528101eb0 0x7f3528073180 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f351c00bd70 tx=0x7f351c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:21:07.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.113+0000 7f350ffff700 1 -- 172.21.15.67:0/1817011198 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f351c01d070 con 0x7f3528101eb0 2024-04-22T23:21:07.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.113+0000 7f350ffff700 1 -- 172.21.15.67:0/1817011198 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f351c022470 con 0x7f3528101eb0 2024-04-22T23:21:07.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.113+0000 7f350ffff700 1 -- 172.21.15.67:0/1817011198 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f351c00f670 con 0x7f3528101eb0 2024-04-22T23:21:07.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.113+0000 7f352d453700 1 -- 172.21.15.67:0/1817011198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3528073a80 con 0x7f3528101eb0 2024-04-22T23:21:07.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.113+0000 7f352d453700 1 -- 172.21.15.67:0/1817011198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3528073f70 con 0x7f3528101eb0 2024-04-22T23:21:07.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.114+0000 7f352d453700 1 -- 172.21.15.67:0/1817011198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3528109810 con 0x7f3528101eb0 2024-04-22T23:21:07.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.114+0000 7f350ffff700 1 -- 172.21.15.67:0/1817011198 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f351c022a50 con 0x7f3528101eb0 2024-04-22T23:21:07.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.115+0000 7f350ffff700 1 --2- 172.21.15.67:0/1817011198 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f351006c4e0 0x7f351006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:21:07.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.115+0000 7f350ffff700 1 -- 172.21.15.67:0/1817011198 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f351c08ceb0 con 0x7f3528101eb0 2024-04-22T23:21:07.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.115+0000 7f3526ffd700 1 --2- 172.21.15.67:0/1817011198 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f351006c4e0 0x7f351006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:21:07.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.115+0000 7f3526ffd700 1 --2- 172.21.15.67:0/1817011198 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f351006c4e0 0x7f351006e9a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3518005fd0 tx=0x7f3518005e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:21:07.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.119+0000 7f350ffff700 1 -- 172.21.15.67:0/1817011198 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f351c05b170 con 0x7f3528101eb0 2024-04-22T23:21:07.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.315+0000 7f352d453700 1 -- 172.21.15.67:0/1817011198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f352819e930 con 0x7f3528101eb0 2024-04-22T23:21:07.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.315+0000 7f350ffff700 1 -- 172.21.15.67:0/1817011198 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7f351c05fe60 con 0x7f3528101eb0 2024-04-22T23:21:07.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.317+0000 7f352d453700 1 -- 172.21.15.67:0/1817011198 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f351006c4e0 msgr2=0x7f351006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:21:07.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.317+0000 7f352d453700 1 --2- 172.21.15.67:0/1817011198 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f351006c4e0 0x7f351006e9a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3518005fd0 tx=0x7f3518005e40 comp rx=0 tx=0).stop 2024-04-22T23:21:07.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.317+0000 7f352d453700 1 -- 172.21.15.67:0/1817011198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3528101eb0 msgr2=0x7f3528073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:21:07.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.317+0000 7f352d453700 1 --2- 172.21.15.67:0/1817011198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3528101eb0 0x7f3528073180 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f351c00bd70 tx=0x7f351c00be50 comp rx=0 tx=0).stop 2024-04-22T23:21:07.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.317+0000 7f352d453700 1 -- 172.21.15.67:0/1817011198 shutdown_connections 2024-04-22T23:21:07.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.317+0000 7f352d453700 1 --2- 172.21.15.67:0/1817011198 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f351006c4e0 0x7f351006e9a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:07.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.317+0000 7f352d453700 1 --2- 172.21.15.67:0/1817011198 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3528100cb0 0x7f3528074b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:07.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.317+0000 7f352d453700 1 --2- 172.21.15.67:0/1817011198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3528101eb0 0x7f3528073180 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:07.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.317+0000 7f352d453700 1 -- 172.21.15.67:0/1817011198 >> 172.21.15.67:0/1817011198 conn(0x7f35280fc230 msgr2=0x7f35281050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:21:07.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.317+0000 7f352d453700 1 -- 172.21.15.67:0/1817011198 shutdown_connections 2024-04-22T23:21:07.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:07.318+0000 7f352d453700 1 -- 172.21.15.67:0/1817011198 wait complete. 2024-04-22T23:21:07.697 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:07 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:21:07.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:07 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:21:07.986 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -e sha1=43be020184947e53516056c9931e1ac5bdbbb1a5 -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-22T23:21:08.202 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:21:08.642 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:08 smithi067 ceph-mon[26453]: pgmap v199: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 938 B/s wr, 5 op/s 2024-04-22T23:21:08.783 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:08 smithi134 ceph-mon[34609]: pgmap v199: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 938 B/s wr, 5 op/s 2024-04-22T23:21:09.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.706+0000 7f1d2b01d700 1 -- 172.21.15.67:0/2276382877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d24102280 msgr2=0x7f1d241026a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:21:09.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.706+0000 7f1d2b01d700 1 --2- 172.21.15.67:0/2276382877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d24102280 0x7f1d241026a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1d18009910 tx=0x7f1d18009c20 comp rx=0 tx=0).stop 2024-04-22T23:21:09.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.706+0000 7f1d2b01d700 1 -- 172.21.15.67:0/2276382877 shutdown_connections 2024-04-22T23:21:09.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.706+0000 7f1d2b01d700 1 --2- 172.21.15.67:0/2276382877 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1d240fed00 0x7f1d240ff160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:09.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.706+0000 7f1d2b01d700 1 --2- 172.21.15.67:0/2276382877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d24102280 0x7f1d241026a0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:09.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.706+0000 7f1d2b01d700 1 -- 172.21.15.67:0/2276382877 >> 172.21.15.67:0/2276382877 conn(0x7f1d240fa8c0 msgr2=0x7f1d240fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:21:09.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.706+0000 7f1d2b01d700 1 -- 172.21.15.67:0/2276382877 shutdown_connections 2024-04-22T23:21:09.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.706+0000 7f1d2b01d700 1 -- 172.21.15.67:0/2276382877 wait complete. 2024-04-22T23:21:09.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.707+0000 7f1d2b01d700 1 Processor -- start 2024-04-22T23:21:09.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.707+0000 7f1d2b01d700 1 -- start start 2024-04-22T23:21:09.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.707+0000 7f1d2b01d700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1d240fed00 0x7f1d24196660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:21:09.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.707+0000 7f1d2b01d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d24102280 0x7f1d24196ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:21:09.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.707+0000 7f1d2b01d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d241971c0 con 0x7f1d24102280 2024-04-22T23:21:09.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.707+0000 7f1d2b01d700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d24197300 con 0x7f1d240fed00 2024-04-22T23:21:09.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.707+0000 7f1d23fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d24102280 0x7f1d24196ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:21:09.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.707+0000 7f1d23fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d24102280 0x7f1d24196ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35636/0 (socket says 172.21.15.67:35636) 2024-04-22T23:21:09.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.707+0000 7f1d23fff700 1 -- 172.21.15.67:0/1511036180 learned_addr learned my addr 172.21.15.67:0/1511036180 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:21:09.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.707+0000 7f1d28db9700 1 --2- 172.21.15.67:0/1511036180 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1d240fed00 0x7f1d24196660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:21:09.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.707+0000 7f1d23fff700 1 -- 172.21.15.67:0/1511036180 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1d240fed00 msgr2=0x7f1d24196660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:21:09.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.707+0000 7f1d23fff700 1 --2- 172.21.15.67:0/1511036180 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1d240fed00 0x7f1d24196660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:09.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.707+0000 7f1d23fff700 1 -- 172.21.15.67:0/1511036180 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d180095f0 con 0x7f1d24102280 2024-04-22T23:21:09.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.708+0000 7f1d23fff700 1 --2- 172.21.15.67:0/1511036180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d24102280 0x7f1d24196ba0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1d1000b4b0 tx=0x7f1d1000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:21:09.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.708+0000 7f1d21ffb700 1 -- 172.21.15.67:0/1511036180 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d10011800 con 0x7f1d24102280 2024-04-22T23:21:09.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.708+0000 7f1d2b01d700 1 -- 172.21.15.67:0/1511036180 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d2419bdb0 con 0x7f1d24102280 2024-04-22T23:21:09.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.708+0000 7f1d21ffb700 1 -- 172.21.15.67:0/1511036180 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1d10011e10 con 0x7f1d24102280 2024-04-22T23:21:09.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.708+0000 7f1d21ffb700 1 -- 172.21.15.67:0/1511036180 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d1000f2e0 con 0x7f1d24102280 2024-04-22T23:21:09.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.708+0000 7f1d2b01d700 1 -- 172.21.15.67:0/1511036180 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d2419c2d0 con 0x7f1d24102280 2024-04-22T23:21:09.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.709+0000 7f1d2b01d700 1 -- 172.21.15.67:0/1511036180 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d24109890 con 0x7f1d24102280 2024-04-22T23:21:09.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.709+0000 7f1d21ffb700 1 -- 172.21.15.67:0/1511036180 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f1d10010380 con 0x7f1d24102280 2024-04-22T23:21:09.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.710+0000 7f1d21ffb700 1 --2- 172.21.15.67:0/1511036180 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f1d1406c4e0 0x7f1d1406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:21:09.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.710+0000 7f1d21ffb700 1 -- 172.21.15.67:0/1511036180 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f1d1008b4f0 con 0x7f1d24102280 2024-04-22T23:21:09.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.710+0000 7f1d28db9700 1 --2- 172.21.15.67:0/1511036180 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f1d1406c4e0 0x7f1d1406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:21:09.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.711+0000 7f1d28db9700 1 --2- 172.21.15.67:0/1511036180 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f1d1406c4e0 0x7f1d1406e9a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1d18006010 tx=0x7f1d1800b560 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:21:09.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.715+0000 7f1d21ffb700 1 -- 172.21.15.67:0/1511036180 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f1d10059700 con 0x7f1d24102280 2024-04-22T23:21:09.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.915+0000 7f1d2b01d700 1 -- 172.21.15.67:0/1511036180 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7f1d2419c970 con 0x7f1d24102280 2024-04-22T23:21:09.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.915+0000 7f1d21ffb700 1 -- 172.21.15.67:0/1511036180 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7f1d10059290 con 0x7f1d24102280 2024-04-22T23:21:09.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.917+0000 7f1d2b01d700 1 -- 172.21.15.67:0/1511036180 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f1d1406c4e0 msgr2=0x7f1d1406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:21:09.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.917+0000 7f1d2b01d700 1 --2- 172.21.15.67:0/1511036180 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f1d1406c4e0 0x7f1d1406e9a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1d18006010 tx=0x7f1d1800b560 comp rx=0 tx=0).stop 2024-04-22T23:21:09.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.917+0000 7f1d2b01d700 1 -- 172.21.15.67:0/1511036180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d24102280 msgr2=0x7f1d24196ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:21:09.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.917+0000 7f1d2b01d700 1 --2- 172.21.15.67:0/1511036180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d24102280 0x7f1d24196ba0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1d1000b4b0 tx=0x7f1d1000b870 comp rx=0 tx=0).stop 2024-04-22T23:21:09.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.917+0000 7f1d2b01d700 1 -- 172.21.15.67:0/1511036180 shutdown_connections 2024-04-22T23:21:09.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.917+0000 7f1d2b01d700 1 --2- 172.21.15.67:0/1511036180 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f1d1406c4e0 0x7f1d1406e9a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:09.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.917+0000 7f1d2b01d700 1 --2- 172.21.15.67:0/1511036180 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1d240fed00 0x7f1d24196660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:09.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.917+0000 7f1d2b01d700 1 --2- 172.21.15.67:0/1511036180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d24102280 0x7f1d24196ba0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:09.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.917+0000 7f1d2b01d700 1 -- 172.21.15.67:0/1511036180 >> 172.21.15.67:0/1511036180 conn(0x7f1d240fa8c0 msgr2=0x7f1d24105250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:21:09.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.917+0000 7f1d2b01d700 1 -- 172.21.15.67:0/1511036180 shutdown_connections 2024-04-22T23:21:09.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:09.917+0000 7f1d2b01d700 1 -- 172.21.15.67:0/1511036180 wait complete. 2024-04-22T23:21:10.600 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:10 smithi067 ceph-mon[26453]: pgmap v200: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-04-22T23:21:10.601 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -e sha1=43be020184947e53516056c9931e1ac5bdbbb1a5 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-04-22T23:21:10.815 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:21:10.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:10 smithi134 ceph-mon[34609]: pgmap v200: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-04-22T23:21:12.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:12.542+0000 7fc88acad700 1 -- 172.21.15.67:0/3206524246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8840ffbc0 msgr2=0x7fc8840fffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:21:12.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:12.542+0000 7fc88acad700 1 --2- 172.21.15.67:0/3206524246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8840ffbc0 0x7fc8840fffe0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fc878009910 tx=0x7fc878009c20 comp rx=0 tx=0).stop 2024-04-22T23:21:12.544 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:12 smithi067 ceph-mon[26453]: pgmap v201: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 3.1 KiB/s rd, 853 B/s wr, 6 op/s 2024-04-22T23:21:12.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:12.542+0000 7fc88acad700 1 -- 172.21.15.67:0/3206524246 shutdown_connections 2024-04-22T23:21:12.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:12.542+0000 7fc88acad700 1 --2- 172.21.15.67:0/3206524246 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc884100dc0 0x7fc884101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:12.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:12.542+0000 7fc88acad700 1 --2- 172.21.15.67:0/3206524246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8840ffbc0 0x7fc8840fffe0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:12.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:12.542+0000 7fc88acad700 1 -- 172.21.15.67:0/3206524246 >> 172.21.15.67:0/3206524246 conn(0x7fc8840fb160 msgr2=0x7fc8840fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:21:12.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:12.542+0000 7fc88acad700 1 -- 172.21.15.67:0/3206524246 shutdown_connections 2024-04-22T23:21:12.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:12.542+0000 7fc88acad700 1 -- 172.21.15.67:0/3206524246 wait complete. 2024-04-22T23:21:12.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:12.542+0000 7fc88acad700 1 Processor -- start 2024-04-22T23:21:12.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:12.543+0000 7fc88acad700 1 -- start start 2024-04-22T23:21:12.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:12.543+0000 7fc88acad700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8840ffbc0 0x7fc8840699d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:21:12.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:12.543+0000 7fc88acad700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc884100dc0 0x7fc88419b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:21:12.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:12.543+0000 7fc88acad700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88419ba70 con 0x7fc8840ffbc0 2024-04-22T23:21:12.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:12.543+0000 7fc88acad700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88419bbb0 con 0x7fc884100dc0 2024-04-22T23:21:12.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:12.543+0000 7fc888a49700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8840ffbc0 0x7fc8840699d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:21:12.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:12.543+0000 7fc888a49700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8840ffbc0 0x7fc8840699d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35654/0 (socket says 172.21.15.67:35654) 2024-04-22T23:21:12.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:12.543+0000 7fc888a49700 1 -- 172.21.15.67:0/1089178464 learned_addr learned my addr 172.21.15.67:0/1089178464 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:21:12.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:12.543+0000 7fc883fff700 1 --2- 172.21.15.67:0/1089178464 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc884100dc0 0x7fc88419b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:21:12.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:12.543+0000 7fc888a49700 1 -- 172.21.15.67:0/1089178464 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc884100dc0 msgr2=0x7fc88419b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:21:12.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:12.543+0000 7fc888a49700 1 --2- 172.21.15.67:0/1089178464 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc884100dc0 0x7fc88419b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:12.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:12.543+0000 7fc888a49700 1 -- 172.21.15.67:0/1089178464 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8780095f0 con 0x7fc8840ffbc0 2024-04-22T23:21:12.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:12.543+0000 7fc883fff700 1 --2- 172.21.15.67:0/1089178464 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc884100dc0 0x7fc88419b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:21:12.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:12.543+0000 7fc888a49700 1 --2- 172.21.15.67:0/1089178464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8840ffbc0 0x7fc8840699d0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fc87800bda0 tx=0x7fc87800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:21:12.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:12.544+0000 7fc881ffb700 1 -- 172.21.15.67:0/1089178464 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc87801d070 con 0x7fc8840ffbc0 2024-04-22T23:21:12.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:12.544+0000 7fc88acad700 1 -- 172.21.15.67:0/1089178464 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc88419be30 con 0x7fc8840ffbc0 2024-04-22T23:21:12.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:12.544+0000 7fc881ffb700 1 -- 172.21.15.67:0/1089178464 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fc878004c50 con 0x7fc8840ffbc0 2024-04-22T23:21:12.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:12.544+0000 7fc881ffb700 1 -- 172.21.15.67:0/1089178464 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc87800f690 con 0x7fc8840ffbc0 2024-04-22T23:21:12.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:12.544+0000 7fc88acad700 1 -- 172.21.15.67:0/1089178464 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc88419c2a0 con 0x7fc8840ffbc0 2024-04-22T23:21:12.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:12.545+0000 7fc88acad700 1 -- 172.21.15.67:0/1089178464 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc88418f530 con 0x7fc8840ffbc0 2024-04-22T23:21:12.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:12.545+0000 7fc881ffb700 1 -- 172.21.15.67:0/1089178464 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fc8780229b0 con 0x7fc8840ffbc0 2024-04-22T23:21:12.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:12.546+0000 7fc881ffb700 1 --2- 172.21.15.67:0/1089178464 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fc87406c4e0 0x7fc87406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:21:12.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:12.546+0000 7fc881ffb700 1 -- 172.21.15.67:0/1089178464 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fc87808ceb0 con 0x7fc8840ffbc0 2024-04-22T23:21:12.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:12.546+0000 7fc883fff700 1 --2- 172.21.15.67:0/1089178464 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fc87406c4e0 0x7fc87406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:21:12.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:12.546+0000 7fc883fff700 1 --2- 172.21.15.67:0/1089178464 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fc87406c4e0 0x7fc87406e9a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc870005fd0 tx=0x7fc870005e30 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:21:12.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:12.551+0000 7fc881ffb700 1 -- 172.21.15.67:0/1089178464 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc87805b170 con 0x7fc8840ffbc0 2024-04-22T23:21:12.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:12.768+0000 7fc88acad700 1 -- 172.21.15.67:0/1089178464 --> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:43be020184947e53516056c9931e1ac5bdbbb1a5", "daemon_types": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fc8840611d0 con 0x7fc87406c4e0 2024-04-22T23:21:12.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:12 smithi134 ceph-mon[34609]: pgmap v201: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 3.1 KiB/s rd, 853 B/s wr, 6 op/s 2024-04-22T23:21:14.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:14 smithi067 ceph-mon[26453]: from='client.14558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:43be020184947e53516056c9931e1ac5bdbbb1a5", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:21:14.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:14 smithi067 ceph-mon[26453]: pgmap v202: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 597 B/s wr, 4 op/s 2024-04-22T23:21:14.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:14 smithi134 ceph-mon[34609]: from='client.14558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:43be020184947e53516056c9931e1ac5bdbbb1a5", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:21:14.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:14 smithi134 ceph-mon[34609]: pgmap v202: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 597 B/s wr, 4 op/s 2024-04-22T23:21:16.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:16 smithi067 ceph-mon[26453]: pgmap v203: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-22T23:21:16.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:16 smithi134 ceph-mon[34609]: pgmap v203: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-22T23:21:18.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:18 smithi067 ceph-mon[26453]: pgmap v204: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 682 B/s wr, 5 op/s 2024-04-22T23:21:18.893 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:18 smithi134 ceph-mon[34609]: pgmap v204: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 682 B/s wr, 5 op/s 2024-04-22T23:21:20.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:20 smithi067 ceph-mon[26453]: pgmap v205: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 597 B/s wr, 3 op/s 2024-04-22T23:21:20.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:20 smithi134 ceph-mon[34609]: pgmap v205: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 597 B/s wr, 3 op/s 2024-04-22T23:21:22.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:22 smithi067 ceph-mon[26453]: pgmap v206: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-22T23:21:22.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:22 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:21:22.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:22 smithi134 ceph-mon[34609]: pgmap v206: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-22T23:21:22.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:22 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:21:24.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:24 smithi067 ceph-mon[26453]: pgmap v207: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-22T23:21:24.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:24 smithi134 ceph-mon[34609]: pgmap v207: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-22T23:21:26.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:26 smithi067 ceph-mon[26453]: pgmap v208: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-22T23:21:26.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:26 smithi134 ceph-mon[34609]: pgmap v208: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-22T23:21:28.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:28 smithi067 ceph-mon[26453]: pgmap v209: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 4 op/s 2024-04-22T23:21:28.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:28 smithi134 ceph-mon[34609]: pgmap v209: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 4 op/s 2024-04-22T23:21:30.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:30 smithi067 ceph-mon[26453]: pgmap v210: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:21:30.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:30 smithi134 ceph-mon[34609]: pgmap v210: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:21:32.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:32 smithi067 ceph-mon[26453]: pgmap v211: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:21:32.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:32 smithi134 ceph-mon[34609]: pgmap v211: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:21:34.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:34 smithi067 ceph-mon[26453]: pgmap v212: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:21:34.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:34 smithi134 ceph-mon[34609]: pgmap v212: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:21:36.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:36 smithi067 ceph-mon[26453]: pgmap v213: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T23:21:36.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:36 smithi134 ceph-mon[34609]: pgmap v213: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T23:21:37.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:37 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:21:37.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:37 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:21:38.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:38 smithi067 ceph-mon[26453]: pgmap v214: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T23:21:38.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:38 smithi134 ceph-mon[34609]: pgmap v214: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T23:21:40.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:40 smithi067 ceph-mon[26453]: pgmap v215: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:21:40.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:40 smithi134 ceph-mon[34609]: pgmap v215: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:21:42.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:42 smithi067 ceph-mon[26453]: pgmap v216: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:21:42.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:42.983+0000 7fc881ffb700 1 -- 172.21.15.67:0/1089178464 <== mgr.14215 v2:172.21.15.67:6800/2589747850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7fc8840611d0 con 0x7fc87406c4e0 2024-04-22T23:21:42.985 INFO:teuthology.orchestra.run.smithi067.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:43be020184947e53516056c9931e1ac5bdbbb1a5 2024-04-22T23:21:42.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:42.986+0000 7fc88acad700 1 -- 172.21.15.67:0/1089178464 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fc87406c4e0 msgr2=0x7fc87406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:21:42.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:42.986+0000 7fc88acad700 1 --2- 172.21.15.67:0/1089178464 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fc87406c4e0 0x7fc87406e9a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc870005fd0 tx=0x7fc870005e30 comp rx=0 tx=0).stop 2024-04-22T23:21:42.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:42.986+0000 7fc88acad700 1 -- 172.21.15.67:0/1089178464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8840ffbc0 msgr2=0x7fc8840699d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:21:42.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:42.986+0000 7fc88acad700 1 --2- 172.21.15.67:0/1089178464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8840ffbc0 0x7fc8840699d0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fc87800bda0 tx=0x7fc87800be80 comp rx=0 tx=0).stop 2024-04-22T23:21:42.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:42.986+0000 7fc88acad700 1 -- 172.21.15.67:0/1089178464 shutdown_connections 2024-04-22T23:21:42.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:42.986+0000 7fc88acad700 1 --2- 172.21.15.67:0/1089178464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8840ffbc0 0x7fc8840699d0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:42.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:42.986+0000 7fc88acad700 1 --2- 172.21.15.67:0/1089178464 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fc87406c4e0 0x7fc87406e9a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:42.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:42.986+0000 7fc88acad700 1 --2- 172.21.15.67:0/1089178464 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc884100dc0 0x7fc88419b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:42.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:42.986+0000 7fc88acad700 1 -- 172.21.15.67:0/1089178464 >> 172.21.15.67:0/1089178464 conn(0x7fc8840fb160 msgr2=0x7fc8841040f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:21:42.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:42.986+0000 7fc88acad700 1 -- 172.21.15.67:0/1089178464 shutdown_connections 2024-04-22T23:21:42.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:42.986+0000 7fc88acad700 1 -- 172.21.15.67:0/1089178464 wait complete. 2024-04-22T23:21:42.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:42 smithi134 ceph-mon[34609]: pgmap v216: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:21:43.731 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -e sha1=43be020184947e53516056c9931e1ac5bdbbb1a5 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-04-22T23:21:43.952 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:21:44.240 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:43 smithi067 ceph-mon[26453]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:43be020184947e53516056c9931e1ac5bdbbb1a5 2024-04-22T23:21:44.241 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:43 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:21:44.241 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:43 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:21:44.241 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:43 smithi067 ceph-mon[26453]: pgmap v217: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:21:44.241 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:43 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:21:44.241 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:43 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:21:44.241 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:43 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:21:44.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:43 smithi134 ceph-mon[34609]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:43be020184947e53516056c9931e1ac5bdbbb1a5 2024-04-22T23:21:44.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:43 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:21:44.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:43 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:21:44.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:43 smithi134 ceph-mon[34609]: pgmap v217: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:21:44.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:43 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:21:44.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:43 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:21:44.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:43 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:21:45.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:44 smithi134 ceph-mon[34609]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:43be020184947e53516056c9931e1ac5bdbbb1a5 2024-04-22T23:21:45.255 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:44 smithi067 ceph-mon[26453]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:43be020184947e53516056c9931e1ac5bdbbb1a5 2024-04-22T23:21:45.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.483+0000 7f270fc4a700 1 -- 172.21.15.67:0/1890954025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2708100ca0 msgr2=0x7f27081010c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:21:45.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.483+0000 7f270fc4a700 1 --2- 172.21.15.67:0/1890954025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2708100ca0 0x7f27081010c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f26f8009910 tx=0x7f26f8009c20 comp rx=0 tx=0).stop 2024-04-22T23:21:45.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.483+0000 7f270fc4a700 1 -- 172.21.15.67:0/1890954025 shutdown_connections 2024-04-22T23:21:45.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.483+0000 7f270fc4a700 1 --2- 172.21.15.67:0/1890954025 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2708101ea0 0x7f2708102300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:45.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.483+0000 7f270fc4a700 1 --2- 172.21.15.67:0/1890954025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2708100ca0 0x7f27081010c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:45.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.483+0000 7f270fc4a700 1 -- 172.21.15.67:0/1890954025 >> 172.21.15.67:0/1890954025 conn(0x7f27080fc260 msgr2=0x7f27080fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:21:45.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.483+0000 7f270fc4a700 1 -- 172.21.15.67:0/1890954025 shutdown_connections 2024-04-22T23:21:45.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.483+0000 7f270fc4a700 1 -- 172.21.15.67:0/1890954025 wait complete. 2024-04-22T23:21:45.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.484+0000 7f270fc4a700 1 Processor -- start 2024-04-22T23:21:45.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.484+0000 7f270fc4a700 1 -- start start 2024-04-22T23:21:45.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.484+0000 7f270fc4a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2708100ca0 0x7f270819cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:21:45.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.484+0000 7f270fc4a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2708101ea0 0x7f270819d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:21:45.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.484+0000 7f270fc4a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270819d5b0 con 0x7f2708100ca0 2024-04-22T23:21:45.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.484+0000 7f270fc4a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270819d6f0 con 0x7f2708101ea0 2024-04-22T23:21:45.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.484+0000 7f270d9e6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2708100ca0 0x7f270819cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:21:45.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.484+0000 7f270d9e6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2708100ca0 0x7f270819cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48542/0 (socket says 172.21.15.67:48542) 2024-04-22T23:21:45.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.484+0000 7f270d9e6700 1 -- 172.21.15.67:0/3827863802 learned_addr learned my addr 172.21.15.67:0/3827863802 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:21:45.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.484+0000 7f270d1e5700 1 --2- 172.21.15.67:0/3827863802 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2708101ea0 0x7f270819d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:21:45.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.484+0000 7f270d9e6700 1 -- 172.21.15.67:0/3827863802 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2708101ea0 msgr2=0x7f270819d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:21:45.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.484+0000 7f270d9e6700 1 --2- 172.21.15.67:0/3827863802 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2708101ea0 0x7f270819d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:45.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.484+0000 7f270d9e6700 1 -- 172.21.15.67:0/3827863802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f80095f0 con 0x7f2708100ca0 2024-04-22T23:21:45.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.485+0000 7f270d9e6700 1 --2- 172.21.15.67:0/3827863802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2708100ca0 0x7f270819cae0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f26f800bda0 tx=0x7f26f800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:21:45.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.485+0000 7f26feffd700 1 -- 172.21.15.67:0/3827863802 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26f801d070 con 0x7f2708100ca0 2024-04-22T23:21:45.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.485+0000 7f26feffd700 1 -- 172.21.15.67:0/3827863802 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f26f8004c50 con 0x7f2708100ca0 2024-04-22T23:21:45.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.485+0000 7f270fc4a700 1 -- 172.21.15.67:0/3827863802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27081a2150 con 0x7f2708100ca0 2024-04-22T23:21:45.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.485+0000 7f26feffd700 1 -- 172.21.15.67:0/3827863802 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26f800f690 con 0x7f2708100ca0 2024-04-22T23:21:45.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.485+0000 7f270fc4a700 1 -- 172.21.15.67:0/3827863802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27081a2610 con 0x7f2708100ca0 2024-04-22T23:21:45.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.486+0000 7f270fc4a700 1 -- 172.21.15.67:0/3827863802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f270810fe10 con 0x7f2708100ca0 2024-04-22T23:21:45.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.487+0000 7f26feffd700 1 -- 172.21.15.67:0/3827863802 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f26f8022a20 con 0x7f2708100ca0 2024-04-22T23:21:45.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.487+0000 7f26feffd700 1 --2- 172.21.15.67:0/3827863802 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f26f406c4e0 0x7f26f406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:21:45.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.487+0000 7f26feffd700 1 -- 172.21.15.67:0/3827863802 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f26f808cf20 con 0x7f2708100ca0 2024-04-22T23:21:45.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.487+0000 7f270d1e5700 1 --2- 172.21.15.67:0/3827863802 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f26f406c4e0 0x7f26f406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:21:45.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.488+0000 7f270d1e5700 1 --2- 172.21.15.67:0/3827863802 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f26f406c4e0 0x7f26f406e9a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2704005ea0 tx=0x7f2704005e30 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:21:45.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.492+0000 7f26feffd700 1 -- 172.21.15.67:0/3827863802 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f26f80c00e0 con 0x7f2708100ca0 2024-04-22T23:21:45.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.715+0000 7f270fc4a700 1 -- 172.21.15.67:0/3827863802 --> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f27081a2c50 con 0x7f26f406c4e0 2024-04-22T23:21:45.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.717+0000 7f26feffd700 1 -- 172.21.15.67:0/3827863802 <== mgr.14215 v2:172.21.15.67:6800/2589747850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7f27081a2c50 con 0x7f26f406c4e0 2024-04-22T23:21:45.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.719+0000 7f270fc4a700 1 -- 172.21.15.67:0/3827863802 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f26f406c4e0 msgr2=0x7f26f406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:21:45.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.719+0000 7f270fc4a700 1 --2- 172.21.15.67:0/3827863802 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f26f406c4e0 0x7f26f406e9a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2704005ea0 tx=0x7f2704005e30 comp rx=0 tx=0).stop 2024-04-22T23:21:45.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.719+0000 7f270fc4a700 1 -- 172.21.15.67:0/3827863802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2708100ca0 msgr2=0x7f270819cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:21:45.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.719+0000 7f270fc4a700 1 --2- 172.21.15.67:0/3827863802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2708100ca0 0x7f270819cae0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f26f800bda0 tx=0x7f26f800be80 comp rx=0 tx=0).stop 2024-04-22T23:21:45.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.719+0000 7f270fc4a700 1 -- 172.21.15.67:0/3827863802 shutdown_connections 2024-04-22T23:21:45.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.719+0000 7f270fc4a700 1 --2- 172.21.15.67:0/3827863802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2708100ca0 0x7f270819cae0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:45.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.719+0000 7f270fc4a700 1 --2- 172.21.15.67:0/3827863802 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f26f406c4e0 0x7f26f406e9a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:45.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.719+0000 7f270fc4a700 1 --2- 172.21.15.67:0/3827863802 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2708101ea0 0x7f270819d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:45.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.719+0000 7f270fc4a700 1 -- 172.21.15.67:0/3827863802 >> 172.21.15.67:0/3827863802 conn(0x7f27080fc260 msgr2=0x7f27081050d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:21:45.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.719+0000 7f270fc4a700 1 -- 172.21.15.67:0/3827863802 shutdown_connections 2024-04-22T23:21:45.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.719+0000 7f270fc4a700 1 -- 172.21.15.67:0/3827863802 wait complete. 2024-04-22T23:21:45.732 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:21:45.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.831+0000 7fdbb7eb2700 1 -- 172.21.15.67:0/406069325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbb01040d0 msgr2=0x7fdbb01064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:21:45.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.831+0000 7fdbb7eb2700 1 --2- 172.21.15.67:0/406069325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbb01040d0 0x7fdbb01064c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fdba0009910 tx=0x7fdba0009c20 comp rx=0 tx=0).stop 2024-04-22T23:21:45.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.832+0000 7fdbb7eb2700 1 -- 172.21.15.67:0/406069325 shutdown_connections 2024-04-22T23:21:45.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.832+0000 7fdbb7eb2700 1 --2- 172.21.15.67:0/406069325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbb01040d0 0x7fdbb01064c0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:45.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.832+0000 7fdbb7eb2700 1 --2- 172.21.15.67:0/406069325 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdbb01017a0 0x7fdbb0103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:45.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.832+0000 7fdbb7eb2700 1 -- 172.21.15.67:0/406069325 >> 172.21.15.67:0/406069325 conn(0x7fdbb00fb150 msgr2=0x7fdbb00fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:21:45.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.832+0000 7fdbb7eb2700 1 -- 172.21.15.67:0/406069325 shutdown_connections 2024-04-22T23:21:45.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.832+0000 7fdbb7eb2700 1 -- 172.21.15.67:0/406069325 wait complete. 2024-04-22T23:21:45.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.832+0000 7fdbb7eb2700 1 Processor -- start 2024-04-22T23:21:45.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.833+0000 7fdbb7eb2700 1 -- start start 2024-04-22T23:21:45.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.833+0000 7fdbb7eb2700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdbb01017a0 0x7fdbb0101100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:21:45.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.833+0000 7fdbb7eb2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbb01040d0 0x7fdbb00ff750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:21:45.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.833+0000 7fdbb7eb2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb00ffc90 con 0x7fdbb01040d0 2024-04-22T23:21:45.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.833+0000 7fdbb7eb2700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb00ffdd0 con 0x7fdbb01017a0 2024-04-22T23:21:45.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.833+0000 7fdbb544d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbb01040d0 0x7fdbb00ff750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:21:45.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.833+0000 7fdbb544d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbb01040d0 0x7fdbb00ff750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48560/0 (socket says 172.21.15.67:48560) 2024-04-22T23:21:45.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.833+0000 7fdbb544d700 1 -- 172.21.15.67:0/2898451739 learned_addr learned my addr 172.21.15.67:0/2898451739 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:21:45.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.833+0000 7fdbb5c4e700 1 --2- 172.21.15.67:0/2898451739 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdbb01017a0 0x7fdbb0101100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:21:45.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.833+0000 7fdbb544d700 1 -- 172.21.15.67:0/2898451739 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdbb01017a0 msgr2=0x7fdbb0101100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:21:45.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.833+0000 7fdbb544d700 1 --2- 172.21.15.67:0/2898451739 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdbb01017a0 0x7fdbb0101100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:45.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.833+0000 7fdbb544d700 1 -- 172.21.15.67:0/2898451739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba00095f0 con 0x7fdbb01040d0 2024-04-22T23:21:45.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.834+0000 7fdbb544d700 1 --2- 172.21.15.67:0/2898451739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbb01040d0 0x7fdbb00ff750 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fdba0006010 tx=0x7fdba000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:21:45.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.834+0000 7fdba6ffd700 1 -- 172.21.15.67:0/2898451739 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdba001d070 con 0x7fdbb01040d0 2024-04-22T23:21:45.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.834+0000 7fdba6ffd700 1 -- 172.21.15.67:0/2898451739 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdba000f460 con 0x7fdbb01040d0 2024-04-22T23:21:45.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.834+0000 7fdbb7eb2700 1 -- 172.21.15.67:0/2898451739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbb0100050 con 0x7fdbb01040d0 2024-04-22T23:21:45.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.834+0000 7fdba6ffd700 1 -- 172.21.15.67:0/2898451739 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdba0021640 con 0x7fdbb01040d0 2024-04-22T23:21:45.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.834+0000 7fdbb7eb2700 1 -- 172.21.15.67:0/2898451739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbb0100540 con 0x7fdbb01040d0 2024-04-22T23:21:45.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.835+0000 7fdbb7eb2700 1 -- 172.21.15.67:0/2898451739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbb0190620 con 0x7fdbb01040d0 2024-04-22T23:21:45.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.836+0000 7fdba6ffd700 1 -- 172.21.15.67:0/2898451739 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fdba000fab0 con 0x7fdbb01040d0 2024-04-22T23:21:45.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.836+0000 7fdba6ffd700 1 --2- 172.21.15.67:0/2898451739 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fdb9c06c4e0 0x7fdb9c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:21:45.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.836+0000 7fdba6ffd700 1 -- 172.21.15.67:0/2898451739 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fdba008cbc0 con 0x7fdbb01040d0 2024-04-22T23:21:45.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.837+0000 7fdbb5c4e700 1 --2- 172.21.15.67:0/2898451739 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fdb9c06c4e0 0x7fdb9c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:21:45.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.837+0000 7fdbb5c4e700 1 --2- 172.21.15.67:0/2898451739 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fdb9c06c4e0 0x7fdb9c06e9a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdba8009550 tx=0x7fdba80091b0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:21:45.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:45.841+0000 7fdba6ffd700 1 -- 172.21.15.67:0/2898451739 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fdba005ad30 con 0x7fdbb01040d0 2024-04-22T23:21:46.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.061+0000 7fdbb7eb2700 1 -- 172.21.15.67:0/2898451739 --> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdbb00611d0 con 0x7fdb9c06c4e0 2024-04-22T23:21:46.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.062+0000 7fdba6ffd700 1 -- 172.21.15.67:0/2898451739 <== mgr.14215 v2:172.21.15.67:6800/2589747850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7fdbb00611d0 con 0x7fdb9c06c4e0 2024-04-22T23:21:46.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.064+0000 7fdbb7eb2700 1 -- 172.21.15.67:0/2898451739 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fdb9c06c4e0 msgr2=0x7fdb9c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:21:46.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.064+0000 7fdbb7eb2700 1 --2- 172.21.15.67:0/2898451739 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fdb9c06c4e0 0x7fdb9c06e9a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdba8009550 tx=0x7fdba80091b0 comp rx=0 tx=0).stop 2024-04-22T23:21:46.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.064+0000 7fdbb7eb2700 1 -- 172.21.15.67:0/2898451739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbb01040d0 msgr2=0x7fdbb00ff750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:21:46.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.064+0000 7fdbb7eb2700 1 --2- 172.21.15.67:0/2898451739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbb01040d0 0x7fdbb00ff750 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fdba0006010 tx=0x7fdba000bd70 comp rx=0 tx=0).stop 2024-04-22T23:21:46.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.064+0000 7fdbb7eb2700 1 -- 172.21.15.67:0/2898451739 shutdown_connections 2024-04-22T23:21:46.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.064+0000 7fdbb7eb2700 1 --2- 172.21.15.67:0/2898451739 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fdb9c06c4e0 0x7fdb9c06e9a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:46.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.064+0000 7fdbb7eb2700 1 --2- 172.21.15.67:0/2898451739 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdbb01017a0 0x7fdbb0101100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:46.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.064+0000 7fdbb7eb2700 1 --2- 172.21.15.67:0/2898451739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbb01040d0 0x7fdbb00ff750 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:46.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.064+0000 7fdbb7eb2700 1 -- 172.21.15.67:0/2898451739 >> 172.21.15.67:0/2898451739 conn(0x7fdbb00fb150 msgr2=0x7fdbb00fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:21:46.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.064+0000 7fdbb7eb2700 1 -- 172.21.15.67:0/2898451739 shutdown_connections 2024-04-22T23:21:46.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.064+0000 7fdbb7eb2700 1 -- 172.21.15.67:0/2898451739 wait complete. 2024-04-22T23:21:46.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.173+0000 7f0535992700 1 -- 172.21.15.67:0/761627440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0530101740 msgr2=0x7f0530103b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:21:46.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.173+0000 7f0535992700 1 --2- 172.21.15.67:0/761627440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0530101740 0x7f0530103b30 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0520009910 tx=0x7f0520009c20 comp rx=0 tx=0).stop 2024-04-22T23:21:46.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.173+0000 7f0535992700 1 -- 172.21.15.67:0/761627440 shutdown_connections 2024-04-22T23:21:46.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.173+0000 7f0535992700 1 --2- 172.21.15.67:0/761627440 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0530104070 0x7f0530106460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:46.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.173+0000 7f0535992700 1 --2- 172.21.15.67:0/761627440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0530101740 0x7f0530103b30 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:46.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.173+0000 7f0535992700 1 -- 172.21.15.67:0/761627440 >> 172.21.15.67:0/761627440 conn(0x7f05300fb150 msgr2=0x7f05300fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:21:46.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.173+0000 7f0535992700 1 -- 172.21.15.67:0/761627440 shutdown_connections 2024-04-22T23:21:46.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.173+0000 7f0535992700 1 -- 172.21.15.67:0/761627440 wait complete. 2024-04-22T23:21:46.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.174+0000 7f0535992700 1 Processor -- start 2024-04-22T23:21:46.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.174+0000 7f0535992700 1 -- start start 2024-04-22T23:21:46.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.174+0000 7f0535992700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0530101740 0x7f05301964a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:21:46.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.174+0000 7f0535992700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0530104070 0x7f05301969e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:21:46.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.174+0000 7f0535992700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0530196f70 con 0x7f0530101740 2024-04-22T23:21:46.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.174+0000 7f0535992700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05301970b0 con 0x7f0530104070 2024-04-22T23:21:46.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.175+0000 7f052e7fc700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0530104070 0x7f05301969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:21:46.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.175+0000 7f052e7fc700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0530104070 0x7f05301969e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.67:55594/0 (socket says 172.21.15.67:55594) 2024-04-22T23:21:46.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.175+0000 7f052effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0530101740 0x7f05301964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:21:46.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.175+0000 7f052e7fc700 1 -- 172.21.15.67:0/2992154344 learned_addr learned my addr 172.21.15.67:0/2992154344 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:21:46.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.175+0000 7f052e7fc700 1 -- 172.21.15.67:0/2992154344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0530101740 msgr2=0x7f05301964a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:21:46.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.175+0000 7f052e7fc700 1 --2- 172.21.15.67:0/2992154344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0530101740 0x7f05301964a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:46.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.175+0000 7f052e7fc700 1 -- 172.21.15.67:0/2992154344 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05200095f0 con 0x7f0530104070 2024-04-22T23:21:46.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.175+0000 7f052effd700 1 --2- 172.21.15.67:0/2992154344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0530101740 0x7f05301964a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T23:21:46.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.175+0000 7f052e7fc700 1 --2- 172.21.15.67:0/2992154344 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0530104070 0x7f05301969e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f051800c930 tx=0x7f051800cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:21:46.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.175+0000 7f0534990700 1 -- 172.21.15.67:0/2992154344 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0518007ad0 con 0x7f0530104070 2024-04-22T23:21:46.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.175+0000 7f0535992700 1 -- 172.21.15.67:0/2992154344 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f053019bb70 con 0x7f0530104070 2024-04-22T23:21:46.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.175+0000 7f0534990700 1 -- 172.21.15.67:0/2992154344 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f051800ce80 con 0x7f0530104070 2024-04-22T23:21:46.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.175+0000 7f0534990700 1 -- 172.21.15.67:0/2992154344 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0518018630 con 0x7f0530104070 2024-04-22T23:21:46.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.175+0000 7f0535992700 1 -- 172.21.15.67:0/2992154344 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f053019c090 con 0x7f0530104070 2024-04-22T23:21:46.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.176+0000 7f0535992700 1 -- 172.21.15.67:0/2992154344 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0530190620 con 0x7f0530104070 2024-04-22T23:21:46.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.177+0000 7f0534990700 1 -- 172.21.15.67:0/2992154344 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f0518007c30 con 0x7f0530104070 2024-04-22T23:21:46.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.178+0000 7f0534990700 1 --2- 172.21.15.67:0/2992154344 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f051c06c4e0 0x7f051c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:21:46.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.178+0000 7f0534990700 1 -- 172.21.15.67:0/2992154344 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f051803a020 con 0x7f0530104070 2024-04-22T23:21:46.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.178+0000 7f052effd700 1 --2- 172.21.15.67:0/2992154344 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f051c06c4e0 0x7f051c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:21:46.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.178+0000 7f052effd700 1 --2- 172.21.15.67:0/2992154344 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f051c06c4e0 0x7f051c06e9a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f052000b5c0 tx=0x7f0520005690 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:21:46.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.182+0000 7f0534990700 1 -- 172.21.15.67:0/2992154344 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f0518059e20 con 0x7f0530104070 2024-04-22T23:21:46.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.394+0000 7f0535992700 1 -- 172.21.15.67:0/2992154344 --> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f05300611d0 con 0x7f051c06c4e0 2024-04-22T23:21:46.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.404+0000 7f0534990700 1 -- 172.21.15.67:0/2992154344 <== mgr.14215 v2:172.21.15.67:6800/2589747850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3480 (secure 0 0 0) 0x7f05300611d0 con 0x7f051c06c4e0 2024-04-22T23:21:46.405 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:21:46.406 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (5m) 56s ago 6m 26.7M - 0.25.0 c8568f914cd2 8c7b7ec00657 2024-04-22T23:21:46.406 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (7m) 56s ago 6m 7490k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:21:46.406 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (5m) 60s ago 5m 7453k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:21:46.406 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (6m) 56s ago 6m 6660k - 18.2.1 5be31c24972a 14085634c5d3 2024-04-22T23:21:46.406 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (5m) 60s ago 5m 6665k - 18.2.1 5be31c24972a fa995d458353 2024-04-22T23:21:46.406 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (5m) 56s ago 6m 85.0M - 9.4.7 954c08fa6188 0d8950c9af0c 2024-04-22T23:21:46.406 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (79s) 56s ago 79s 12.8M - 18.2.1 5be31c24972a e24a471f0a8c 2024-04-22T23:21:46.406 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (74s) 56s ago 73s 16.7M - 18.2.1 5be31c24972a 0730f9fc8e61 2024-04-22T23:21:46.407 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (76s) 60s ago 76s 17.6M - 18.2.1 5be31c24972a b6a169586433 2024-04-22T23:21:46.407 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (71s) 60s ago 70s 14.6M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:21:46.407 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:9283,8765,8443 running (8m) 56s ago 8m 490M - 18.2.1 5be31c24972a 1c64cb9e36bb 2024-04-22T23:21:46.407 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (5m) 60s ago 5m 443M - 18.2.1 5be31c24972a 0a0b76e72171 2024-04-22T23:21:46.407 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (8m) 56s ago 8m 46.1M 2048M 18.2.1 5be31c24972a 6789283703e0 2024-04-22T23:21:46.407 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (5m) 60s ago 5m 35.9M 2048M 18.2.1 5be31c24972a 2fb23795f7e3 2024-04-22T23:21:46.407 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (6m) 56s ago 6m 16.3M - 1.5.0 0da6a335fe13 4aa4da7089d0 2024-04-22T23:21:46.407 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (5m) 60s ago 5m 15.6M - 1.5.0 0da6a335fe13 26e43c2e756c 2024-04-22T23:21:46.407 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (5m) 56s ago 5m 48.5M 3946M 18.2.1 5be31c24972a a033247ee8c2 2024-04-22T23:21:46.407 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (4m) 56s ago 4m 47.1M 3946M 18.2.1 5be31c24972a 915b07a8edc4 2024-04-22T23:21:46.407 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (4m) 56s ago 4m 46.8M 3946M 18.2.1 5be31c24972a e257ecb4efa1 2024-04-22T23:21:46.407 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (3m) 60s ago 3m 45.9M 4970M 18.2.1 5be31c24972a 9c2c91a967ba 2024-04-22T23:21:46.407 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (3m) 60s ago 3m 44.0M 4970M 18.2.1 5be31c24972a c7586562ffa6 2024-04-22T23:21:46.407 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (2m) 60s ago 2m 44.0M 4970M 18.2.1 5be31c24972a 9fd97f1c65cb 2024-04-22T23:21:46.408 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (5m) 56s ago 6m 55.5M - 2.43.0 a07b618ecd1d d87cb15b5e55 2024-04-22T23:21:46.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.407+0000 7f0535992700 1 -- 172.21.15.67:0/2992154344 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f051c06c4e0 msgr2=0x7f051c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:21:46.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.407+0000 7f0535992700 1 --2- 172.21.15.67:0/2992154344 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f051c06c4e0 0x7f051c06e9a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f052000b5c0 tx=0x7f0520005690 comp rx=0 tx=0).stop 2024-04-22T23:21:46.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.407+0000 7f0535992700 1 -- 172.21.15.67:0/2992154344 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0530104070 msgr2=0x7f05301969e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:21:46.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.407+0000 7f0535992700 1 --2- 172.21.15.67:0/2992154344 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0530104070 0x7f05301969e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f051800c930 tx=0x7f051800cc40 comp rx=0 tx=0).stop 2024-04-22T23:21:46.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.407+0000 7f0535992700 1 -- 172.21.15.67:0/2992154344 shutdown_connections 2024-04-22T23:21:46.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.407+0000 7f0535992700 1 --2- 172.21.15.67:0/2992154344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0530101740 0x7f05301964a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:46.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.407+0000 7f0535992700 1 --2- 172.21.15.67:0/2992154344 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f051c06c4e0 0x7f051c06e9a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:46.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.407+0000 7f0535992700 1 --2- 172.21.15.67:0/2992154344 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0530104070 0x7f05301969e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:46.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.407+0000 7f0535992700 1 -- 172.21.15.67:0/2992154344 >> 172.21.15.67:0/2992154344 conn(0x7f05300fb150 msgr2=0x7f05300fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:21:46.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.407+0000 7f0535992700 1 -- 172.21.15.67:0/2992154344 shutdown_connections 2024-04-22T23:21:46.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.407+0000 7f0535992700 1 -- 172.21.15.67:0/2992154344 wait complete. 2024-04-22T23:21:46.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.514+0000 7ff5dd7a0700 1 -- 172.21.15.67:0/3238052305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5d81017a0 msgr2=0x7ff5d8103b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:21:46.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.514+0000 7ff5dd7a0700 1 --2- 172.21.15.67:0/3238052305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5d81017a0 0x7ff5d8103b90 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff5d000b0a0 tx=0x7ff5d000b3b0 comp rx=0 tx=0).stop 2024-04-22T23:21:46.515 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:46 smithi067 ceph-mon[26453]: pgmap v218: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T23:21:46.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.514+0000 7ff5dd7a0700 1 -- 172.21.15.67:0/3238052305 shutdown_connections 2024-04-22T23:21:46.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.514+0000 7ff5dd7a0700 1 --2- 172.21.15.67:0/3238052305 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff5d81040d0 0x7ff5d81064c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:46.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.514+0000 7ff5dd7a0700 1 --2- 172.21.15.67:0/3238052305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5d81017a0 0x7ff5d8103b90 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:46.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.514+0000 7ff5dd7a0700 1 -- 172.21.15.67:0/3238052305 >> 172.21.15.67:0/3238052305 conn(0x7ff5d80fb190 msgr2=0x7ff5d80fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:21:46.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.514+0000 7ff5dd7a0700 1 -- 172.21.15.67:0/3238052305 shutdown_connections 2024-04-22T23:21:46.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.514+0000 7ff5dd7a0700 1 -- 172.21.15.67:0/3238052305 wait complete. 2024-04-22T23:21:46.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.514+0000 7ff5dd7a0700 1 Processor -- start 2024-04-22T23:21:46.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.515+0000 7ff5dd7a0700 1 -- start start 2024-04-22T23:21:46.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.515+0000 7ff5dd7a0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5d81017a0 0x7ff5d8196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:21:46.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.515+0000 7ff5dd7a0700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff5d81040d0 0x7ff5d8196ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:21:46.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.515+0000 7ff5dd7a0700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d81970e0 con 0x7ff5d81017a0 2024-04-22T23:21:46.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.515+0000 7ff5dd7a0700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d8197220 con 0x7ff5d81040d0 2024-04-22T23:21:46.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.515+0000 7ff5d6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5d81017a0 0x7ff5d8196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:21:46.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.515+0000 7ff5d6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5d81017a0 0x7ff5d8196580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48604/0 (socket says 172.21.15.67:48604) 2024-04-22T23:21:46.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.515+0000 7ff5d6ffd700 1 -- 172.21.15.67:0/1143868050 learned_addr learned my addr 172.21.15.67:0/1143868050 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:21:46.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.515+0000 7ff5d67fc700 1 --2- 172.21.15.67:0/1143868050 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff5d81040d0 0x7ff5d8196ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:21:46.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.515+0000 7ff5d6ffd700 1 -- 172.21.15.67:0/1143868050 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff5d81040d0 msgr2=0x7ff5d8196ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:21:46.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.515+0000 7ff5d6ffd700 1 --2- 172.21.15.67:0/1143868050 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff5d81040d0 0x7ff5d8196ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:46.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.516+0000 7ff5d6ffd700 1 -- 172.21.15.67:0/1143868050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d0009d00 con 0x7ff5d81017a0 2024-04-22T23:21:46.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.516+0000 7ff5d67fc700 1 --2- 172.21.15.67:0/1143868050 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff5d81040d0 0x7ff5d8196ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:21:46.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.516+0000 7ff5d6ffd700 1 --2- 172.21.15.67:0/1143868050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5d81017a0 0x7ff5d8196580 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff5d00077b0 tx=0x7ff5d0003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:21:46.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.516+0000 7ff5cffff700 1 -- 172.21.15.67:0/1143868050 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5d000e040 con 0x7ff5d81017a0 2024-04-22T23:21:46.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.516+0000 7ff5dd7a0700 1 -- 172.21.15.67:0/1143868050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5d819bc70 con 0x7ff5d81017a0 2024-04-22T23:21:46.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.516+0000 7ff5cffff700 1 -- 172.21.15.67:0/1143868050 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff5d000ce60 con 0x7ff5d81017a0 2024-04-22T23:21:46.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.516+0000 7ff5cffff700 1 -- 172.21.15.67:0/1143868050 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5d001daf0 con 0x7ff5d81017a0 2024-04-22T23:21:46.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.516+0000 7ff5dd7a0700 1 -- 172.21.15.67:0/1143868050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5d819c160 con 0x7ff5d81017a0 2024-04-22T23:21:46.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.517+0000 7ff5dd7a0700 1 -- 172.21.15.67:0/1143868050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5d8190690 con 0x7ff5d81017a0 2024-04-22T23:21:46.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.518+0000 7ff5cffff700 1 -- 172.21.15.67:0/1143868050 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7ff5d0004250 con 0x7ff5d81017a0 2024-04-22T23:21:46.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.518+0000 7ff5cffff700 1 --2- 172.21.15.67:0/1143868050 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ff5c406c4e0 0x7ff5c406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:21:46.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.518+0000 7ff5cffff700 1 -- 172.21.15.67:0/1143868050 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7ff5d008d380 con 0x7ff5d81017a0 2024-04-22T23:21:46.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.518+0000 7ff5d67fc700 1 --2- 172.21.15.67:0/1143868050 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ff5c406c4e0 0x7ff5c406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:21:46.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.519+0000 7ff5d67fc700 1 --2- 172.21.15.67:0/1143868050 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ff5c406c4e0 0x7ff5c406e9a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff5c0007900 tx=0x7ff5c0008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:21:46.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.523+0000 7ff5cffff700 1 -- 172.21.15.67:0/1143868050 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff5d005b590 con 0x7ff5d81017a0 2024-04-22T23:21:46.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:46 smithi134 ceph-mon[34609]: pgmap v218: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T23:21:46.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.806+0000 7ff5dd7a0700 1 -- 172.21.15.67:0/1143868050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff5d802cc70 con 0x7ff5d81017a0 2024-04-22T23:21:46.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.806+0000 7ff5cffff700 1 -- 172.21.15.67:0/1143868050 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+558 (secure 0 0 0) 0x7ff5d005b120 con 0x7ff5d81017a0 2024-04-22T23:21:46.808 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:21:46.808 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:21:46.808 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-22T23:21:46.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:21:46.808 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:21:46.808 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-22T23:21:46.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:21:46.808 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:21:46.808 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-22T23:21:46.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:21:46.808 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:21:46.808 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-22T23:21:46.809 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:21:46.809 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:21:46.809 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 14 2024-04-22T23:21:46.809 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:21:46.809 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:21:46.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.808+0000 7ff5dd7a0700 1 -- 172.21.15.67:0/1143868050 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ff5c406c4e0 msgr2=0x7ff5c406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:21:46.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.808+0000 7ff5dd7a0700 1 --2- 172.21.15.67:0/1143868050 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ff5c406c4e0 0x7ff5c406e9a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff5c0007900 tx=0x7ff5c0008040 comp rx=0 tx=0).stop 2024-04-22T23:21:46.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.808+0000 7ff5dd7a0700 1 -- 172.21.15.67:0/1143868050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5d81017a0 msgr2=0x7ff5d8196580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:21:46.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.808+0000 7ff5dd7a0700 1 --2- 172.21.15.67:0/1143868050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5d81017a0 0x7ff5d8196580 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff5d00077b0 tx=0x7ff5d0003ce0 comp rx=0 tx=0).stop 2024-04-22T23:21:46.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.808+0000 7ff5dd7a0700 1 -- 172.21.15.67:0/1143868050 shutdown_connections 2024-04-22T23:21:46.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.808+0000 7ff5dd7a0700 1 --2- 172.21.15.67:0/1143868050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5d81017a0 0x7ff5d8196580 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:46.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.808+0000 7ff5dd7a0700 1 --2- 172.21.15.67:0/1143868050 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ff5c406c4e0 0x7ff5c406e9a0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:46.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.808+0000 7ff5dd7a0700 1 --2- 172.21.15.67:0/1143868050 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff5d81040d0 0x7ff5d8196ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:46.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.808+0000 7ff5dd7a0700 1 -- 172.21.15.67:0/1143868050 >> 172.21.15.67:0/1143868050 conn(0x7ff5d80fb190 msgr2=0x7ff5d80fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:21:46.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.809+0000 7ff5dd7a0700 1 -- 172.21.15.67:0/1143868050 shutdown_connections 2024-04-22T23:21:46.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.809+0000 7ff5dd7a0700 1 -- 172.21.15.67:0/1143868050 wait complete. 2024-04-22T23:21:46.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.917+0000 7f13b0e91700 1 -- 172.21.15.67:0/440675891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13ac0ffc10 msgr2=0x7f13ac100030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:21:46.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.917+0000 7f13b0e91700 1 --2- 172.21.15.67:0/440675891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13ac0ffc10 0x7f13ac100030 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f139c009910 tx=0x7f139c009c20 comp rx=0 tx=0).stop 2024-04-22T23:21:46.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.917+0000 7f13b0e91700 1 -- 172.21.15.67:0/440675891 shutdown_connections 2024-04-22T23:21:46.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.917+0000 7f13b0e91700 1 --2- 172.21.15.67:0/440675891 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13ac100e10 0x7f13ac101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:46.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.917+0000 7f13b0e91700 1 --2- 172.21.15.67:0/440675891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13ac0ffc10 0x7f13ac100030 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:46.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.917+0000 7f13b0e91700 1 -- 172.21.15.67:0/440675891 >> 172.21.15.67:0/440675891 conn(0x7f13ac0fb190 msgr2=0x7f13ac0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:21:46.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.917+0000 7f13b0e91700 1 -- 172.21.15.67:0/440675891 shutdown_connections 2024-04-22T23:21:46.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.917+0000 7f13b0e91700 1 -- 172.21.15.67:0/440675891 wait complete. 2024-04-22T23:21:46.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.918+0000 7f13b0e91700 1 Processor -- start 2024-04-22T23:21:46.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.918+0000 7f13b0e91700 1 -- start start 2024-04-22T23:21:46.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.918+0000 7f13b0e91700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13ac0ffc10 0x7f13ac0690e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:21:46.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.918+0000 7f13b0e91700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13ac100e10 0x7f13ac19b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:21:46.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.918+0000 7f13b0e91700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ac19b810 con 0x7f13ac100e10 2024-04-22T23:21:46.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.918+0000 7f13b0e91700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ac19b950 con 0x7f13ac0ffc10 2024-04-22T23:21:46.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.918+0000 7f13a9d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13ac100e10 0x7f13ac19b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:21:46.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.918+0000 7f13a9d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13ac100e10 0x7f13ac19b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48624/0 (socket says 172.21.15.67:48624) 2024-04-22T23:21:46.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.918+0000 7f13a9d9b700 1 -- 172.21.15.67:0/1815939633 learned_addr learned my addr 172.21.15.67:0/1815939633 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:21:46.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.918+0000 7f13aa59c700 1 --2- 172.21.15.67:0/1815939633 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13ac0ffc10 0x7f13ac0690e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:21:46.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.919+0000 7f13a9d9b700 1 -- 172.21.15.67:0/1815939633 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13ac0ffc10 msgr2=0x7f13ac0690e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:21:46.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.919+0000 7f13a9d9b700 1 --2- 172.21.15.67:0/1815939633 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13ac0ffc10 0x7f13ac0690e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:46.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.919+0000 7f13a9d9b700 1 -- 172.21.15.67:0/1815939633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f139c0095f0 con 0x7f13ac100e10 2024-04-22T23:21:46.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.919+0000 7f13a9d9b700 1 --2- 172.21.15.67:0/1815939633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13ac100e10 0x7f13ac19b240 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f13a000d900 tx=0x7f13a000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:21:46.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.919+0000 7f139b7fe700 1 -- 172.21.15.67:0/1815939633 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13a0004a00 con 0x7f13ac100e10 2024-04-22T23:21:46.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.919+0000 7f139b7fe700 1 -- 172.21.15.67:0/1815939633 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f13a00054d0 con 0x7f13ac100e10 2024-04-22T23:21:46.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.919+0000 7f13b0e91700 1 -- 172.21.15.67:0/1815939633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13ac19bc30 con 0x7f13ac100e10 2024-04-22T23:21:46.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.919+0000 7f139b7fe700 1 -- 172.21.15.67:0/1815939633 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13a0009d00 con 0x7f13ac100e10 2024-04-22T23:21:46.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.919+0000 7f13b0e91700 1 -- 172.21.15.67:0/1815939633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13ac19c180 con 0x7f13ac100e10 2024-04-22T23:21:46.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.920+0000 7f13b0e91700 1 -- 172.21.15.67:0/1815939633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13ac108720 con 0x7f13ac100e10 2024-04-22T23:21:46.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.921+0000 7f139b7fe700 1 -- 172.21.15.67:0/1815939633 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f13a0005020 con 0x7f13ac100e10 2024-04-22T23:21:46.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.921+0000 7f139b7fe700 1 --2- 172.21.15.67:0/1815939633 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f139406c4e0 0x7f139406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:21:46.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.921+0000 7f13aa59c700 1 --2- 172.21.15.67:0/1815939633 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f139406c4e0 0x7f139406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:21:46.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.921+0000 7f139b7fe700 1 -- 172.21.15.67:0/1815939633 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f13a008b750 con 0x7f13ac100e10 2024-04-22T23:21:46.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.922+0000 7f13aa59c700 1 --2- 172.21.15.67:0/1815939633 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f139406c4e0 0x7f139406e9a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f139c006010 tx=0x7f139c00b540 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:21:46.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:46.926+0000 7f139b7fe700 1 -- 172.21.15.67:0/1815939633 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f13a00599e0 con 0x7f13ac100e10 2024-04-22T23:21:47.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:47.165+0000 7f13b0e91700 1 -- 172.21.15.67:0/1815939633 --> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f13ac105760 con 0x7f139406c4e0 2024-04-22T23:21:47.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:47.166+0000 7f139b7fe700 1 -- 172.21.15.67:0/1815939633 <== mgr.14215 v2:172.21.15.67:6800/2589747850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f13ac105760 con 0x7f139406c4e0 2024-04-22T23:21:47.167 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:21:47.167 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:21:47.167 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:21:47.167 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-22T23:21:47.167 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [], 2024-04-22T23:21:47.167 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "0/2 daemons upgraded", 2024-04-22T23:21:47.167 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c image on host smithi134", 2024-04-22T23:21:47.167 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:21:47.168 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:21:47.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:47.168+0000 7f13b0e91700 1 -- 172.21.15.67:0/1815939633 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f139406c4e0 msgr2=0x7f139406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:21:47.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:47.168+0000 7f13b0e91700 1 --2- 172.21.15.67:0/1815939633 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f139406c4e0 0x7f139406e9a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f139c006010 tx=0x7f139c00b540 comp rx=0 tx=0).stop 2024-04-22T23:21:47.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:47.168+0000 7f13b0e91700 1 -- 172.21.15.67:0/1815939633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13ac100e10 msgr2=0x7f13ac19b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:21:47.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:47.168+0000 7f13b0e91700 1 --2- 172.21.15.67:0/1815939633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13ac100e10 0x7f13ac19b240 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f13a000d900 tx=0x7f13a000dc10 comp rx=0 tx=0).stop 2024-04-22T23:21:47.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:47.168+0000 7f13b0e91700 1 -- 172.21.15.67:0/1815939633 shutdown_connections 2024-04-22T23:21:47.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:47.168+0000 7f13b0e91700 1 --2- 172.21.15.67:0/1815939633 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f139406c4e0 0x7f139406e9a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:47.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:47.168+0000 7f13b0e91700 1 --2- 172.21.15.67:0/1815939633 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13ac0ffc10 0x7f13ac0690e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:47.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:47.168+0000 7f13b0e91700 1 --2- 172.21.15.67:0/1815939633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13ac100e10 0x7f13ac19b240 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:21:47.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:47.168+0000 7f13b0e91700 1 -- 172.21.15.67:0/1815939633 >> 172.21.15.67:0/1815939633 conn(0x7f13ac0fb190 msgr2=0x7f13ac104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:21:47.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:47.168+0000 7f13b0e91700 1 -- 172.21.15.67:0/1815939633 shutdown_connections 2024-04-22T23:21:47.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:21:47.168+0000 7f13b0e91700 1 -- 172.21.15.67:0/1815939633 wait complete. 2024-04-22T23:21:47.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:47 smithi067 ceph-mon[26453]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:21:47.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:47 smithi067 ceph-mon[26453]: from='client.14566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:21:47.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:47 smithi067 ceph-mon[26453]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:21:47.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:47 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:21:47.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:47 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:21:47.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:47 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:21:47.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:47 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/1143868050' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:21:47.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:47 smithi134 ceph-mon[34609]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:21:47.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:47 smithi134 ceph-mon[34609]: from='client.14566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:21:47.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:47 smithi134 ceph-mon[34609]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:21:47.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:47 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:21:47.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:47 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:21:47.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:47 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:21:47.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:47 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/1143868050' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:21:48.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:48 smithi067 ceph-mon[26453]: Upgrade: Target is version 19.0.0-3252-g43be0201 (unknown) 2024-04-22T23:21:48.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:48 smithi067 ceph-mon[26453]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, digests ['quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c'] 2024-04-22T23:21:48.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:48 smithi067 ceph-mon[26453]: Upgrade: Need to upgrade myself (mgr.smithi067.whdcsr) 2024-04-22T23:21:48.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:48 smithi067 ceph-mon[26453]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c on smithi134 2024-04-22T23:21:48.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:48 smithi067 ceph-mon[26453]: from='client.14578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:21:48.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:48 smithi067 ceph-mon[26453]: pgmap v219: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T23:21:48.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:48 smithi134 ceph-mon[34609]: Upgrade: Target is version 19.0.0-3252-g43be0201 (unknown) 2024-04-22T23:21:48.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:48 smithi134 ceph-mon[34609]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, digests ['quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c'] 2024-04-22T23:21:48.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:48 smithi134 ceph-mon[34609]: Upgrade: Need to upgrade myself (mgr.smithi067.whdcsr) 2024-04-22T23:21:48.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:48 smithi134 ceph-mon[34609]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c on smithi134 2024-04-22T23:21:48.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:48 smithi134 ceph-mon[34609]: from='client.14578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:21:48.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:48 smithi134 ceph-mon[34609]: pgmap v219: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T23:21:50.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:50 smithi067 ceph-mon[26453]: pgmap v220: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:21:50.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:50 smithi134 ceph-mon[34609]: pgmap v220: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:21:52.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:52 smithi067 ceph-mon[26453]: pgmap v221: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:21:52.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:52 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:21:52.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:52 smithi134 ceph-mon[34609]: pgmap v221: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:21:52.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:52 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:21:54.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:54 smithi067 ceph-mon[26453]: pgmap v222: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:21:54.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:54 smithi134 ceph-mon[34609]: pgmap v222: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:21:56.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:56 smithi067 ceph-mon[26453]: pgmap v223: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T23:21:56.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:56 smithi134 ceph-mon[34609]: pgmap v223: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T23:21:58.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:21:58 smithi067 ceph-mon[26453]: pgmap v224: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T23:21:58.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:21:58 smithi134 ceph-mon[34609]: pgmap v224: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T23:22:00.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:00 smithi067 ceph-mon[26453]: pgmap v225: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:22:00.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:00 smithi134 ceph-mon[34609]: pgmap v225: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:22:02.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:02 smithi067 ceph-mon[26453]: pgmap v226: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:22:02.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:02 smithi134 ceph-mon[34609]: pgmap v226: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:22:04.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:04 smithi134 ceph-mon[34609]: pgmap v227: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:22:04.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:04 smithi067 ceph-mon[26453]: pgmap v227: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:22:06.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:06 smithi067 ceph-mon[26453]: pgmap v228: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T23:22:06.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:06 smithi134 ceph-mon[34609]: pgmap v228: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T23:22:07.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:07 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:22:07.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:07 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:22:08.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:08 smithi067 ceph-mon[26453]: pgmap v229: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T23:22:08.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:08 smithi134 ceph-mon[34609]: pgmap v229: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T23:22:10.798 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:10 smithi067 ceph-mon[26453]: pgmap v230: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:22:10.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:10 smithi134 ceph-mon[34609]: pgmap v230: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:22:12.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:12 smithi067 ceph-mon[26453]: pgmap v231: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:22:12.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:12 smithi134 ceph-mon[34609]: pgmap v231: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:22:14.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:14 smithi067 ceph-mon[26453]: pgmap v232: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:22:14.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:14 smithi134 ceph-mon[34609]: pgmap v232: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:22:16.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:16 smithi067 ceph-mon[26453]: pgmap v233: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T23:22:16.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:16 smithi134 ceph-mon[34609]: pgmap v233: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T23:22:17.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.337+0000 7fa6c50d2700 1 -- 172.21.15.67:0/2035958200 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa6c0100de0 msgr2=0x7fa6c0101260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:17.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.337+0000 7fa6c50d2700 1 --2- 172.21.15.67:0/2035958200 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa6c0100de0 0x7fa6c0101260 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4009910 tx=0x7fa6b4009c20 comp rx=0 tx=0).stop 2024-04-22T23:22:17.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.337+0000 7fa6c50d2700 1 -- 172.21.15.67:0/2035958200 shutdown_connections 2024-04-22T23:22:17.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.337+0000 7fa6c50d2700 1 --2- 172.21.15.67:0/2035958200 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa6c0100de0 0x7fa6c0101260 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:17.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.337+0000 7fa6c50d2700 1 --2- 172.21.15.67:0/2035958200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6c01000c0 0x7fa6c01004e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:17.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.337+0000 7fa6c50d2700 1 -- 172.21.15.67:0/2035958200 >> 172.21.15.67:0/2035958200 conn(0x7fa6c00fa8c0 msgr2=0x7fa6c00fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:22:17.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.337+0000 7fa6c50d2700 1 -- 172.21.15.67:0/2035958200 shutdown_connections 2024-04-22T23:22:17.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.337+0000 7fa6c50d2700 1 -- 172.21.15.67:0/2035958200 wait complete. 2024-04-22T23:22:17.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.338+0000 7fa6c50d2700 1 Processor -- start 2024-04-22T23:22:17.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.339+0000 7fa6c50d2700 1 -- start start 2024-04-22T23:22:17.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.339+0000 7fa6c50d2700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa6c01000c0 0x7fa6c0198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:22:17.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.339+0000 7fa6c50d2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6c0100de0 0x7fa6c0198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:22:17.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.339+0000 7fa6c50d2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c0199260 con 0x7fa6c0100de0 2024-04-22T23:22:17.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.339+0000 7fa6c50d2700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c01993a0 con 0x7fa6c01000c0 2024-04-22T23:22:17.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.339+0000 7fa6bed9d700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa6c01000c0 0x7fa6c0198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:22:17.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.339+0000 7fa6bed9d700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa6c01000c0 0x7fa6c0198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.67:38516/0 (socket says 172.21.15.67:38516) 2024-04-22T23:22:17.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.339+0000 7fa6bed9d700 1 -- 172.21.15.67:0/2524172166 learned_addr learned my addr 172.21.15.67:0/2524172166 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:22:17.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.340+0000 7fa6be59c700 1 --2- 172.21.15.67:0/2524172166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6c0100de0 0x7fa6c0198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:22:17.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.340+0000 7fa6bed9d700 1 -- 172.21.15.67:0/2524172166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6c0100de0 msgr2=0x7fa6c0198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:17.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.340+0000 7fa6bed9d700 1 --2- 172.21.15.67:0/2524172166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6c0100de0 0x7fa6c0198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:17.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.340+0000 7fa6bed9d700 1 -- 172.21.15.67:0/2524172166 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b40095f0 con 0x7fa6c01000c0 2024-04-22T23:22:17.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.340+0000 7fa6be59c700 1 --2- 172.21.15.67:0/2524172166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6c0100de0 0x7fa6c0198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:22:17.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.340+0000 7fa6bed9d700 1 --2- 172.21.15.67:0/2524172166 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa6c01000c0 0x7fa6c0198700 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa6b000b3c0 tx=0x7fa6b0009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:22:17.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.340+0000 7fa6a7fff700 1 -- 172.21.15.67:0/2524172166 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6b0010040 con 0x7fa6c01000c0 2024-04-22T23:22:17.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.340+0000 7fa6a7fff700 1 -- 172.21.15.67:0/2524172166 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa6b00044d0 con 0x7fa6c01000c0 2024-04-22T23:22:17.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.341+0000 7fa6a7fff700 1 -- 172.21.15.67:0/2524172166 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6b00148d0 con 0x7fa6c01000c0 2024-04-22T23:22:17.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.341+0000 7fa6c50d2700 1 -- 172.21.15.67:0/2524172166 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c019de50 con 0x7fa6c01000c0 2024-04-22T23:22:17.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.341+0000 7fa6c50d2700 1 -- 172.21.15.67:0/2524172166 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c0193f20 con 0x7fa6c01000c0 2024-04-22T23:22:17.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.342+0000 7fa6c50d2700 1 -- 172.21.15.67:0/2524172166 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6c0107620 con 0x7fa6c01000c0 2024-04-22T23:22:17.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.343+0000 7fa6a7fff700 1 -- 172.21.15.67:0/2524172166 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fa6b0003680 con 0x7fa6c01000c0 2024-04-22T23:22:17.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.344+0000 7fa6a7fff700 1 --2- 172.21.15.67:0/2524172166 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa6a8074cf0 0x7fa6a80771b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:22:17.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.344+0000 7fa6a7fff700 1 -- 172.21.15.67:0/2524172166 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fa6b008b160 con 0x7fa6c01000c0 2024-04-22T23:22:17.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.344+0000 7fa6be59c700 1 --2- 172.21.15.67:0/2524172166 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa6a8074cf0 0x7fa6a80771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:22:17.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.345+0000 7fa6be59c700 1 --2- 172.21.15.67:0/2524172166 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa6a8074cf0 0x7fa6a80771b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4005b40 tx=0x7fa6b4005a90 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:22:17.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.352+0000 7fa6a7fff700 1 -- 172.21.15.67:0/2524172166 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa6b0059350 con 0x7fa6c01000c0 2024-04-22T23:22:17.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.679+0000 7fa6c50d2700 1 -- 172.21.15.67:0/2524172166 --> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa6c0105730 con 0x7fa6a8074cf0 2024-04-22T23:22:17.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.681+0000 7fa6a7fff700 1 -- 172.21.15.67:0/2524172166 <== mgr.14215 v2:172.21.15.67:6800/2589747850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7fa6c0105730 con 0x7fa6a8074cf0 2024-04-22T23:22:17.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.683+0000 7fa6c50d2700 1 -- 172.21.15.67:0/2524172166 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa6a8074cf0 msgr2=0x7fa6a80771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:17.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.683+0000 7fa6c50d2700 1 --2- 172.21.15.67:0/2524172166 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa6a8074cf0 0x7fa6a80771b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4005b40 tx=0x7fa6b4005a90 comp rx=0 tx=0).stop 2024-04-22T23:22:17.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.683+0000 7fa6c50d2700 1 -- 172.21.15.67:0/2524172166 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa6c01000c0 msgr2=0x7fa6c0198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:17.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.683+0000 7fa6c50d2700 1 --2- 172.21.15.67:0/2524172166 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa6c01000c0 0x7fa6c0198700 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa6b000b3c0 tx=0x7fa6b0009b60 comp rx=0 tx=0).stop 2024-04-22T23:22:17.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.684+0000 7fa6c50d2700 1 -- 172.21.15.67:0/2524172166 shutdown_connections 2024-04-22T23:22:17.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.684+0000 7fa6c50d2700 1 --2- 172.21.15.67:0/2524172166 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa6a8074cf0 0x7fa6a80771b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:17.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.684+0000 7fa6c50d2700 1 --2- 172.21.15.67:0/2524172166 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa6c01000c0 0x7fa6c0198700 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:17.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.684+0000 7fa6c50d2700 1 --2- 172.21.15.67:0/2524172166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6c0100de0 0x7fa6c0198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:17.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.684+0000 7fa6c50d2700 1 -- 172.21.15.67:0/2524172166 >> 172.21.15.67:0/2524172166 conn(0x7fa6c00fa8c0 msgr2=0x7fa6c0104010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:22:17.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.684+0000 7fa6c50d2700 1 -- 172.21.15.67:0/2524172166 shutdown_connections 2024-04-22T23:22:17.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.684+0000 7fa6c50d2700 1 -- 172.21.15.67:0/2524172166 wait complete. 2024-04-22T23:22:17.699 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:22:17.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.799+0000 7fa860223700 1 -- 172.21.15.67:0/154406048 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa858100e10 msgr2=0x7fa858101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:17.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.799+0000 7fa860223700 1 --2- 172.21.15.67:0/154406048 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa858100e10 0x7fa858101270 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa848009910 tx=0x7fa848009c20 comp rx=0 tx=0).stop 2024-04-22T23:22:17.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.799+0000 7fa860223700 1 -- 172.21.15.67:0/154406048 shutdown_connections 2024-04-22T23:22:17.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.799+0000 7fa860223700 1 --2- 172.21.15.67:0/154406048 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa858100e10 0x7fa858101270 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:17.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.799+0000 7fa860223700 1 --2- 172.21.15.67:0/154406048 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa8580ffc10 0x7fa858100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:17.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.799+0000 7fa860223700 1 -- 172.21.15.67:0/154406048 >> 172.21.15.67:0/154406048 conn(0x7fa8580fb190 msgr2=0x7fa8580fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:22:17.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.799+0000 7fa860223700 1 -- 172.21.15.67:0/154406048 shutdown_connections 2024-04-22T23:22:17.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.799+0000 7fa860223700 1 -- 172.21.15.67:0/154406048 wait complete. 2024-04-22T23:22:17.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.800+0000 7fa860223700 1 Processor -- start 2024-04-22T23:22:17.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.800+0000 7fa860223700 1 -- start start 2024-04-22T23:22:17.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.800+0000 7fa860223700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8580ffc10 0x7fa8581916e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:22:17.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.800+0000 7fa860223700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa858100e10 0x7fa858191c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:22:17.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.800+0000 7fa860223700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa85818e1d0 con 0x7fa8580ffc10 2024-04-22T23:22:17.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.800+0000 7fa860223700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa85818e340 con 0x7fa858100e10 2024-04-22T23:22:17.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.800+0000 7fa85dfbf700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8580ffc10 0x7fa8581916e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:22:17.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.800+0000 7fa85dfbf700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8580ffc10 0x7fa8581916e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36880/0 (socket says 172.21.15.67:36880) 2024-04-22T23:22:17.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.800+0000 7fa85dfbf700 1 -- 172.21.15.67:0/1799131006 learned_addr learned my addr 172.21.15.67:0/1799131006 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:22:17.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.800+0000 7fa85d7be700 1 --2- 172.21.15.67:0/1799131006 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa858100e10 0x7fa858191c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:22:17.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.800+0000 7fa85dfbf700 1 -- 172.21.15.67:0/1799131006 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa858100e10 msgr2=0x7fa858191c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:17.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.800+0000 7fa85dfbf700 1 --2- 172.21.15.67:0/1799131006 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa858100e10 0x7fa858191c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:17.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.800+0000 7fa85dfbf700 1 -- 172.21.15.67:0/1799131006 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8480095f0 con 0x7fa8580ffc10 2024-04-22T23:22:17.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.801+0000 7fa85dfbf700 1 --2- 172.21.15.67:0/1799131006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8580ffc10 0x7fa8581916e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa85400e8f0 tx=0x7fa85400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:22:17.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.801+0000 7fa84effd700 1 -- 172.21.15.67:0/1799131006 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa854009790 con 0x7fa8580ffc10 2024-04-22T23:22:17.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.801+0000 7fa84effd700 1 -- 172.21.15.67:0/1799131006 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa85400c810 con 0x7fa8580ffc10 2024-04-22T23:22:17.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.801+0000 7fa84effd700 1 -- 172.21.15.67:0/1799131006 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa854003d20 con 0x7fa8580ffc10 2024-04-22T23:22:17.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.801+0000 7fa860223700 1 -- 172.21.15.67:0/1799131006 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa85818e620 con 0x7fa8580ffc10 2024-04-22T23:22:17.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.801+0000 7fa860223700 1 -- 172.21.15.67:0/1799131006 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa85818eb70 con 0x7fa8580ffc10 2024-04-22T23:22:17.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.802+0000 7fa860223700 1 -- 172.21.15.67:0/1799131006 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa858108700 con 0x7fa8580ffc10 2024-04-22T23:22:17.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.803+0000 7fa84effd700 1 -- 172.21.15.67:0/1799131006 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fa8540041f0 con 0x7fa8580ffc10 2024-04-22T23:22:17.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.803+0000 7fa84effd700 1 --2- 172.21.15.67:0/1799131006 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa84406c4e0 0x7fa84406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:22:17.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.803+0000 7fa84effd700 1 -- 172.21.15.67:0/1799131006 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fa854014070 con 0x7fa8580ffc10 2024-04-22T23:22:17.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.803+0000 7fa85d7be700 1 --2- 172.21.15.67:0/1799131006 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa84406c4e0 0x7fa84406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:22:17.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.804+0000 7fa85d7be700 1 --2- 172.21.15.67:0/1799131006 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa84406c4e0 0x7fa84406e9a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa848009fd0 tx=0x7fa848005940 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:22:17.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:17.812+0000 7fa84effd700 1 -- 172.21.15.67:0/1799131006 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa854059d20 con 0x7fa8580ffc10 2024-04-22T23:22:18.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.029+0000 7fa860223700 1 -- 172.21.15.67:0/1799131006 --> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa858068c90 con 0x7fa84406c4e0 2024-04-22T23:22:18.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.032+0000 7fa84effd700 1 -- 172.21.15.67:0/1799131006 <== mgr.14215 v2:172.21.15.67:6800/2589747850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7fa858068c90 con 0x7fa84406c4e0 2024-04-22T23:22:18.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.034+0000 7fa860223700 1 -- 172.21.15.67:0/1799131006 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa84406c4e0 msgr2=0x7fa84406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:18.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.034+0000 7fa860223700 1 --2- 172.21.15.67:0/1799131006 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa84406c4e0 0x7fa84406e9a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa848009fd0 tx=0x7fa848005940 comp rx=0 tx=0).stop 2024-04-22T23:22:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.034+0000 7fa860223700 1 -- 172.21.15.67:0/1799131006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8580ffc10 msgr2=0x7fa8581916e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.034+0000 7fa860223700 1 --2- 172.21.15.67:0/1799131006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8580ffc10 0x7fa8581916e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa85400e8f0 tx=0x7fa85400ec00 comp rx=0 tx=0).stop 2024-04-22T23:22:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.034+0000 7fa860223700 1 -- 172.21.15.67:0/1799131006 shutdown_connections 2024-04-22T23:22:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.034+0000 7fa860223700 1 --2- 172.21.15.67:0/1799131006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8580ffc10 0x7fa8581916e0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.034+0000 7fa860223700 1 --2- 172.21.15.67:0/1799131006 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa84406c4e0 0x7fa84406e9a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.034+0000 7fa860223700 1 --2- 172.21.15.67:0/1799131006 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa858100e10 0x7fa858191c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.034+0000 7fa860223700 1 -- 172.21.15.67:0/1799131006 >> 172.21.15.67:0/1799131006 conn(0x7fa8580fb190 msgr2=0x7fa858068580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:22:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.035+0000 7fa860223700 1 -- 172.21.15.67:0/1799131006 shutdown_connections 2024-04-22T23:22:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.035+0000 7fa860223700 1 -- 172.21.15.67:0/1799131006 wait complete. 2024-04-22T23:22:18.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.144+0000 7f040258c700 1 -- 172.21.15.67:0/4288663013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03fc0ff5f0 msgr2=0x7f03fc0ffa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:18.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.144+0000 7f040258c700 1 --2- 172.21.15.67:0/4288663013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03fc0ff5f0 0x7f03fc0ffa10 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f03ec009910 tx=0x7f03ec009c20 comp rx=0 tx=0).stop 2024-04-22T23:22:18.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.144+0000 7f040258c700 1 -- 172.21.15.67:0/4288663013 shutdown_connections 2024-04-22T23:22:18.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.144+0000 7f040258c700 1 --2- 172.21.15.67:0/4288663013 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03fc0fff50 0x7f03fc1003d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:18.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.144+0000 7f040258c700 1 --2- 172.21.15.67:0/4288663013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03fc0ff5f0 0x7f03fc0ffa10 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:18.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.144+0000 7f040258c700 1 -- 172.21.15.67:0/4288663013 >> 172.21.15.67:0/4288663013 conn(0x7f03fc0fb190 msgr2=0x7f03fc0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:22:18.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.144+0000 7f040258c700 1 -- 172.21.15.67:0/4288663013 shutdown_connections 2024-04-22T23:22:18.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.144+0000 7f040258c700 1 -- 172.21.15.67:0/4288663013 wait complete. 2024-04-22T23:22:18.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.145+0000 7f040258c700 1 Processor -- start 2024-04-22T23:22:18.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.145+0000 7f040258c700 1 -- start start 2024-04-22T23:22:18.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.145+0000 7f040258c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03fc0ff5f0 0x7f03fc198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:22:18.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.145+0000 7f040258c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03fc0fff50 0x7f03fc198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:22:18.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.145+0000 7f040258c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03fc1991f0 con 0x7f03fc0fff50 2024-04-22T23:22:18.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.145+0000 7f040258c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03fc199330 con 0x7f03fc0ff5f0 2024-04-22T23:22:18.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.145+0000 7f03fb7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03fc0fff50 0x7f03fc198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:22:18.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.145+0000 7f03fb7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03fc0fff50 0x7f03fc198bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36896/0 (socket says 172.21.15.67:36896) 2024-04-22T23:22:18.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.145+0000 7f03fb7fe700 1 -- 172.21.15.67:0/2932651073 learned_addr learned my addr 172.21.15.67:0/2932651073 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:22:18.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.145+0000 7f03fbfff700 1 --2- 172.21.15.67:0/2932651073 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03fc0ff5f0 0x7f03fc198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:22:18.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.146+0000 7f03fb7fe700 1 -- 172.21.15.67:0/2932651073 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03fc0ff5f0 msgr2=0x7f03fc198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:18.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.146+0000 7f03fb7fe700 1 --2- 172.21.15.67:0/2932651073 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03fc0ff5f0 0x7f03fc198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:18.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.146+0000 7f03fb7fe700 1 -- 172.21.15.67:0/2932651073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03ec0095f0 con 0x7f03fc0fff50 2024-04-22T23:22:18.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.146+0000 7f03fb7fe700 1 --2- 172.21.15.67:0/2932651073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03fc0fff50 0x7f03fc198bd0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f03f000e920 tx=0x7f03f000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:22:18.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.146+0000 7f03f97fa700 1 -- 172.21.15.67:0/2932651073 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03f000c9d0 con 0x7f03fc0fff50 2024-04-22T23:22:18.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.146+0000 7f040258c700 1 -- 172.21.15.67:0/2932651073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03fc19dde0 con 0x7f03fc0fff50 2024-04-22T23:22:18.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.146+0000 7f03f97fa700 1 -- 172.21.15.67:0/2932651073 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f03f0010430 con 0x7f03fc0fff50 2024-04-22T23:22:18.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.146+0000 7f03f97fa700 1 -- 172.21.15.67:0/2932651073 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03f0018650 con 0x7f03fc0fff50 2024-04-22T23:22:18.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.146+0000 7f040258c700 1 -- 172.21.15.67:0/2932651073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03fc19e300 con 0x7f03fc0fff50 2024-04-22T23:22:18.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.147+0000 7f040258c700 1 -- 172.21.15.67:0/2932651073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03fc10b9f0 con 0x7f03fc0fff50 2024-04-22T23:22:18.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.148+0000 7f03f97fa700 1 -- 172.21.15.67:0/2932651073 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f03f00105a0 con 0x7f03fc0fff50 2024-04-22T23:22:18.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.149+0000 7f03f97fa700 1 --2- 172.21.15.67:0/2932651073 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f03e406c4e0 0x7f03e406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:22:18.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.149+0000 7f03f97fa700 1 -- 172.21.15.67:0/2932651073 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f03f0014070 con 0x7f03fc0fff50 2024-04-22T23:22:18.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.149+0000 7f03fbfff700 1 --2- 172.21.15.67:0/2932651073 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f03e406c4e0 0x7f03e406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:22:18.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.150+0000 7f03fbfff700 1 --2- 172.21.15.67:0/2932651073 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f03e406c4e0 0x7f03e406e9a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f03ec009fd0 tx=0x7f03ec005940 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:22:18.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.156+0000 7f03f97fa700 1 -- 172.21.15.67:0/2932651073 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f03f005a1e0 con 0x7f03fc0fff50 2024-04-22T23:22:18.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.369+0000 7f040258c700 1 -- 172.21.15.67:0/2932651073 --> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f03fc109340 con 0x7f03e406c4e0 2024-04-22T23:22:18.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.380+0000 7f03f97fa700 1 -- 172.21.15.67:0/2932651073 <== mgr.14215 v2:172.21.15.67:6800/2589747850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3528 (secure 0 0 0) 0x7f03fc109340 con 0x7f03e406c4e0 2024-04-22T23:22:18.382 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:22:18.382 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (5m) 88s ago 7m 26.7M - 0.25.0 c8568f914cd2 8c7b7ec00657 2024-04-22T23:22:18.382 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (7m) 88s ago 7m 7490k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:22:18.382 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (6m) 92s ago 6m 7453k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:22:18.382 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (7m) 88s ago 7m 6660k - 18.2.1 5be31c24972a 14085634c5d3 2024-04-22T23:22:18.382 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (6m) 92s ago 6m 6665k - 18.2.1 5be31c24972a fa995d458353 2024-04-22T23:22:18.382 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (5m) 88s ago 6m 85.0M - 9.4.7 954c08fa6188 0d8950c9af0c 2024-04-22T23:22:18.382 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (111s) 88s ago 111s 12.8M - 18.2.1 5be31c24972a e24a471f0a8c 2024-04-22T23:22:18.382 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (106s) 88s ago 105s 16.7M - 18.2.1 5be31c24972a 0730f9fc8e61 2024-04-22T23:22:18.382 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (108s) 92s ago 108s 17.6M - 18.2.1 5be31c24972a b6a169586433 2024-04-22T23:22:18.382 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (103s) 92s ago 102s 14.6M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:22:18.382 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:9283,8765,8443 running (8m) 88s ago 8m 490M - 18.2.1 5be31c24972a 1c64cb9e36bb 2024-04-22T23:22:18.382 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (6m) 92s ago 6m 443M - 18.2.1 5be31c24972a 0a0b76e72171 2024-04-22T23:22:18.383 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (8m) 88s ago 8m 46.1M 2048M 18.2.1 5be31c24972a 6789283703e0 2024-04-22T23:22:18.383 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (6m) 92s ago 6m 35.9M 2048M 18.2.1 5be31c24972a 2fb23795f7e3 2024-04-22T23:22:18.383 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (7m) 88s ago 7m 16.3M - 1.5.0 0da6a335fe13 4aa4da7089d0 2024-04-22T23:22:18.383 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (6m) 92s ago 6m 15.6M - 1.5.0 0da6a335fe13 26e43c2e756c 2024-04-22T23:22:18.383 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (5m) 88s ago 5m 48.5M 3946M 18.2.1 5be31c24972a a033247ee8c2 2024-04-22T23:22:18.383 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (5m) 88s ago 5m 47.1M 3946M 18.2.1 5be31c24972a 915b07a8edc4 2024-04-22T23:22:18.383 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (4m) 88s ago 4m 46.8M 3946M 18.2.1 5be31c24972a e257ecb4efa1 2024-04-22T23:22:18.383 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (4m) 92s ago 4m 45.9M 4970M 18.2.1 5be31c24972a 9c2c91a967ba 2024-04-22T23:22:18.383 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (3m) 92s ago 3m 44.0M 4970M 18.2.1 5be31c24972a c7586562ffa6 2024-04-22T23:22:18.383 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (3m) 92s ago 3m 44.0M 4970M 18.2.1 5be31c24972a 9fd97f1c65cb 2024-04-22T23:22:18.383 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (5m) 88s ago 6m 55.5M - 2.43.0 a07b618ecd1d d87cb15b5e55 2024-04-22T23:22:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.382+0000 7f040258c700 1 -- 172.21.15.67:0/2932651073 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f03e406c4e0 msgr2=0x7f03e406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.382+0000 7f040258c700 1 --2- 172.21.15.67:0/2932651073 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f03e406c4e0 0x7f03e406e9a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f03ec009fd0 tx=0x7f03ec005940 comp rx=0 tx=0).stop 2024-04-22T23:22:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.382+0000 7f040258c700 1 -- 172.21.15.67:0/2932651073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03fc0fff50 msgr2=0x7f03fc198bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.382+0000 7f040258c700 1 --2- 172.21.15.67:0/2932651073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03fc0fff50 0x7f03fc198bd0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f03f000e920 tx=0x7f03f000ec30 comp rx=0 tx=0).stop 2024-04-22T23:22:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.383+0000 7f040258c700 1 -- 172.21.15.67:0/2932651073 shutdown_connections 2024-04-22T23:22:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.383+0000 7f040258c700 1 --2- 172.21.15.67:0/2932651073 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f03e406c4e0 0x7f03e406e9a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.383+0000 7f040258c700 1 --2- 172.21.15.67:0/2932651073 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03fc0ff5f0 0x7f03fc198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.383+0000 7f040258c700 1 --2- 172.21.15.67:0/2932651073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03fc0fff50 0x7f03fc198bd0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:18.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.383+0000 7f040258c700 1 -- 172.21.15.67:0/2932651073 >> 172.21.15.67:0/2932651073 conn(0x7f03fc0fb190 msgr2=0x7f03fc107c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:22:18.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.383+0000 7f040258c700 1 -- 172.21.15.67:0/2932651073 shutdown_connections 2024-04-22T23:22:18.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.383+0000 7f040258c700 1 -- 172.21.15.67:0/2932651073 wait complete. 2024-04-22T23:22:18.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.490+0000 7f35e5433700 1 -- 172.21.15.67:0/102423685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35e01040d0 msgr2=0x7f35e01064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:18.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.490+0000 7f35e5433700 1 --2- 172.21.15.67:0/102423685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35e01040d0 0x7f35e01064c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f35d4009910 tx=0x7f35d4009c20 comp rx=0 tx=0).stop 2024-04-22T23:22:18.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.491+0000 7f35e5433700 1 -- 172.21.15.67:0/102423685 shutdown_connections 2024-04-22T23:22:18.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.491+0000 7f35e5433700 1 --2- 172.21.15.67:0/102423685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35e01040d0 0x7f35e01064c0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:18.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.491+0000 7f35e5433700 1 --2- 172.21.15.67:0/102423685 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f35e01017a0 0x7f35e0103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:18.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.491+0000 7f35e5433700 1 -- 172.21.15.67:0/102423685 >> 172.21.15.67:0/102423685 conn(0x7f35e00fb190 msgr2=0x7f35e00fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:22:18.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.491+0000 7f35e5433700 1 -- 172.21.15.67:0/102423685 shutdown_connections 2024-04-22T23:22:18.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.491+0000 7f35e5433700 1 -- 172.21.15.67:0/102423685 wait complete. 2024-04-22T23:22:18.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.491+0000 7f35e5433700 1 Processor -- start 2024-04-22T23:22:18.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.492+0000 7f35e5433700 1 -- start start 2024-04-22T23:22:18.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.492+0000 7f35e5433700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f35e01017a0 0x7f35e0196450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:22:18.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.492+0000 7f35e5433700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35e01040d0 0x7f35e0196990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:22:18.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.492+0000 7f35e5433700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e0196fb0 con 0x7f35e01040d0 2024-04-22T23:22:18.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.492+0000 7f35e5433700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e01970f0 con 0x7f35e01017a0 2024-04-22T23:22:18.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.492+0000 7f35de7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35e01040d0 0x7f35e0196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:22:18.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.492+0000 7f35de7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35e01040d0 0x7f35e0196990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36914/0 (socket says 172.21.15.67:36914) 2024-04-22T23:22:18.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.492+0000 7f35de7fc700 1 -- 172.21.15.67:0/1560793223 learned_addr learned my addr 172.21.15.67:0/1560793223 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:22:18.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.492+0000 7f35de7fc700 1 -- 172.21.15.67:0/1560793223 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f35e01017a0 msgr2=0x7f35e0196450 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T23:22:18.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.492+0000 7f35de7fc700 1 --2- 172.21.15.67:0/1560793223 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f35e01017a0 0x7f35e0196450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:18.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.492+0000 7f35de7fc700 1 -- 172.21.15.67:0/1560793223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d40095f0 con 0x7f35e01040d0 2024-04-22T23:22:18.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.492+0000 7f35de7fc700 1 --2- 172.21.15.67:0/1560793223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35e01040d0 0x7f35e0196990 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f35d4000c00 tx=0x7f35d401e740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:22:18.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.493+0000 7f35c7fff700 1 -- 172.21.15.67:0/1560793223 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35d401edf0 con 0x7f35e01040d0 2024-04-22T23:22:18.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.493+0000 7f35c7fff700 1 -- 172.21.15.67:0/1560793223 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f35d4005dc0 con 0x7f35e01040d0 2024-04-22T23:22:18.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.493+0000 7f35e5433700 1 -- 172.21.15.67:0/1560793223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35e00ff590 con 0x7f35e01040d0 2024-04-22T23:22:18.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.493+0000 7f35e5433700 1 -- 172.21.15.67:0/1560793223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35e00ffa80 con 0x7f35e01040d0 2024-04-22T23:22:18.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.493+0000 7f35c7fff700 1 -- 172.21.15.67:0/1560793223 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35d400f770 con 0x7f35e01040d0 2024-04-22T23:22:18.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.494+0000 7f35e5433700 1 -- 172.21.15.67:0/1560793223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35e0190610 con 0x7f35e01040d0 2024-04-22T23:22:18.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.495+0000 7f35c7fff700 1 -- 172.21.15.67:0/1560793223 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f35d40050d0 con 0x7f35e01040d0 2024-04-22T23:22:18.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.495+0000 7f35c7fff700 1 --2- 172.21.15.67:0/1560793223 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f35c806c4e0 0x7f35c806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:22:18.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.495+0000 7f35c7fff700 1 -- 172.21.15.67:0/1560793223 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f35d4090910 con 0x7f35e01040d0 2024-04-22T23:22:18.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.496+0000 7f35deffd700 1 --2- 172.21.15.67:0/1560793223 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f35c806c4e0 0x7f35c806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:22:18.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.496+0000 7f35deffd700 1 --2- 172.21.15.67:0/1560793223 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f35c806c4e0 0x7f35c806e9a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f35d0009b40 tx=0x7f35d0009260 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:22:18.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.500+0000 7f35c7fff700 1 -- 172.21.15.67:0/1560793223 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f35d405ebd0 con 0x7f35e01040d0 2024-04-22T23:22:18.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.788+0000 7f35e5433700 1 -- 172.21.15.67:0/1560793223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f35e0100f30 con 0x7f35e01040d0 2024-04-22T23:22:18.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:18 smithi067 ceph-mon[26453]: pgmap v234: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T23:22:18.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.789+0000 7f35c7fff700 1 -- 172.21.15.67:0/1560793223 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+558 (secure 0 0 0) 0x7f35d405e760 con 0x7f35e01040d0 2024-04-22T23:22:18.790 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:22:18.791 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:22:18.791 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-22T23:22:18.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:22:18.791 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:22:18.791 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-22T23:22:18.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:22:18.791 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:22:18.791 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-22T23:22:18.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:22:18.791 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:22:18.791 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-22T23:22:18.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:22:18.792 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:22:18.792 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 14 2024-04-22T23:22:18.792 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:22:18.792 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:22:18.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.792+0000 7f35e5433700 1 -- 172.21.15.67:0/1560793223 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f35c806c4e0 msgr2=0x7f35c806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:18.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.792+0000 7f35e5433700 1 --2- 172.21.15.67:0/1560793223 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f35c806c4e0 0x7f35c806e9a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f35d0009b40 tx=0x7f35d0009260 comp rx=0 tx=0).stop 2024-04-22T23:22:18.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.792+0000 7f35e5433700 1 -- 172.21.15.67:0/1560793223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35e01040d0 msgr2=0x7f35e0196990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:18.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.792+0000 7f35e5433700 1 --2- 172.21.15.67:0/1560793223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35e01040d0 0x7f35e0196990 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f35d4000c00 tx=0x7f35d401e740 comp rx=0 tx=0).stop 2024-04-22T23:22:18.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.792+0000 7f35e5433700 1 -- 172.21.15.67:0/1560793223 shutdown_connections 2024-04-22T23:22:18.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.792+0000 7f35e5433700 1 --2- 172.21.15.67:0/1560793223 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f35c806c4e0 0x7f35c806e9a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:18.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.792+0000 7f35e5433700 1 --2- 172.21.15.67:0/1560793223 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f35e01017a0 0x7f35e0196450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:18.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.792+0000 7f35e5433700 1 --2- 172.21.15.67:0/1560793223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35e01040d0 0x7f35e0196990 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:18.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.792+0000 7f35e5433700 1 -- 172.21.15.67:0/1560793223 >> 172.21.15.67:0/1560793223 conn(0x7f35e00fb190 msgr2=0x7f35e00fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:22:18.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.792+0000 7f35e5433700 1 -- 172.21.15.67:0/1560793223 shutdown_connections 2024-04-22T23:22:18.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.792+0000 7f35e5433700 1 -- 172.21.15.67:0/1560793223 wait complete. 2024-04-22T23:22:18.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.902+0000 7f5fbb8f6700 1 -- 172.21.15.67:0/1371248713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fb4102290 msgr2=0x7f5fb41026b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:18.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.902+0000 7f5fbb8f6700 1 --2- 172.21.15.67:0/1371248713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fb4102290 0x7f5fb41026b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f5fb0009910 tx=0x7f5fb0009c20 comp rx=0 tx=0).stop 2024-04-22T23:22:18.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.902+0000 7f5fbb8f6700 1 -- 172.21.15.67:0/1371248713 shutdown_connections 2024-04-22T23:22:18.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.902+0000 7f5fbb8f6700 1 --2- 172.21.15.67:0/1371248713 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5fb40fed10 0x7f5fb40ff170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:18.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.902+0000 7f5fbb8f6700 1 --2- 172.21.15.67:0/1371248713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fb4102290 0x7f5fb41026b0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:18.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.902+0000 7f5fbb8f6700 1 -- 172.21.15.67:0/1371248713 >> 172.21.15.67:0/1371248713 conn(0x7f5fb40fa8b0 msgr2=0x7f5fb40fcd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:22:18.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.902+0000 7f5fbb8f6700 1 -- 172.21.15.67:0/1371248713 shutdown_connections 2024-04-22T23:22:18.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.902+0000 7f5fbb8f6700 1 -- 172.21.15.67:0/1371248713 wait complete. 2024-04-22T23:22:18.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.903+0000 7f5fbb8f6700 1 Processor -- start 2024-04-22T23:22:18.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.903+0000 7f5fbb8f6700 1 -- start start 2024-04-22T23:22:18.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.903+0000 7f5fbb8f6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fb40fed10 0x7f5fb4196530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:22:18.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.903+0000 7f5fbb8f6700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5fb4102290 0x7f5fb4196a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:22:18.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.903+0000 7f5fbb8f6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb4197090 con 0x7f5fb40fed10 2024-04-22T23:22:18.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.903+0000 7f5fbb8f6700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb41971d0 con 0x7f5fb4102290 2024-04-22T23:22:18.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.903+0000 7f5fb8e91700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5fb4102290 0x7f5fb4196a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:22:18.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.903+0000 7f5fb9692700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fb40fed10 0x7f5fb4196530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:22:18.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.903+0000 7f5fb8e91700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5fb4102290 0x7f5fb4196a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.67:38718/0 (socket says 172.21.15.67:38718) 2024-04-22T23:22:18.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.904+0000 7f5fb8e91700 1 -- 172.21.15.67:0/706680351 learned_addr learned my addr 172.21.15.67:0/706680351 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:22:18.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.904+0000 7f5fb9692700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fb40fed10 0x7f5fb4196530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47412/0 (socket says 172.21.15.67:47412) 2024-04-22T23:22:18.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.904+0000 7f5fb8e91700 1 -- 172.21.15.67:0/706680351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fb40fed10 msgr2=0x7f5fb4196530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:18.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.904+0000 7f5fb8e91700 1 --2- 172.21.15.67:0/706680351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fb40fed10 0x7f5fb4196530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:18.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.904+0000 7f5fb8e91700 1 -- 172.21.15.67:0/706680351 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb00095f0 con 0x7f5fb4102290 2024-04-22T23:22:18.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.904+0000 7f5fb8e91700 1 --2- 172.21.15.67:0/706680351 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5fb4102290 0x7f5fb4196a70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5fa400d8d0 tx=0x7f5fa400dbe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:22:18.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.904+0000 7f5faa7fc700 1 -- 172.21.15.67:0/706680351 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5fa40098a0 con 0x7f5fb4102290 2024-04-22T23:22:18.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.904+0000 7f5fbb8f6700 1 -- 172.21.15.67:0/706680351 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fb40ffe80 con 0x7f5fb4102290 2024-04-22T23:22:18.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.905+0000 7f5fbb8f6700 1 -- 172.21.15.67:0/706680351 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fb41003d0 con 0x7f5fb4102290 2024-04-22T23:22:18.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.904+0000 7f5faa7fc700 1 -- 172.21.15.67:0/706680351 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5fa4010460 con 0x7f5fb4102290 2024-04-22T23:22:18.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.905+0000 7f5faa7fc700 1 -- 172.21.15.67:0/706680351 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5fa400f5f0 con 0x7f5fb4102290 2024-04-22T23:22:18.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.906+0000 7f5fbb8f6700 1 -- 172.21.15.67:0/706680351 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f98005320 con 0x7f5fb4102290 2024-04-22T23:22:18.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.907+0000 7f5faa7fc700 1 -- 172.21.15.67:0/706680351 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f5fa4009a00 con 0x7f5fb4102290 2024-04-22T23:22:18.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.907+0000 7f5faa7fc700 1 --2- 172.21.15.67:0/706680351 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f5fa006c4e0 0x7f5fa006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:22:18.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.907+0000 7f5faa7fc700 1 -- 172.21.15.67:0/706680351 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f5fa408b6b0 con 0x7f5fb4102290 2024-04-22T23:22:18.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.908+0000 7f5fb9692700 1 --2- 172.21.15.67:0/706680351 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f5fa006c4e0 0x7f5fa006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:22:18.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.908+0000 7f5fb9692700 1 --2- 172.21.15.67:0/706680351 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f5fa006c4e0 0x7f5fa006e9a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5fb0006010 tx=0x7f5fb000b580 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:22:18.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:18.916+0000 7f5faa7fc700 1 -- 172.21.15.67:0/706680351 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f5fa40598e0 con 0x7f5fb4102290 2024-04-22T23:22:18.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:18 smithi134 ceph-mon[34609]: pgmap v234: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T23:22:19.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:19.353+0000 7f5fbb8f6700 1 -- 172.21.15.67:0/706680351 --> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5f98000bf0 con 0x7f5fa006c4e0 2024-04-22T23:22:19.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:19.355+0000 7f5faa7fc700 1 -- 172.21.15.67:0/706680351 <== mgr.14215 v2:172.21.15.67:6800/2589747850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f5f98000bf0 con 0x7f5fa006c4e0 2024-04-22T23:22:19.356 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:22:19.356 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:22:19.356 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:22:19.356 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-22T23:22:19.357 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [], 2024-04-22T23:22:19.357 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "0/2 daemons upgraded", 2024-04-22T23:22:19.357 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c image on host smithi134", 2024-04-22T23:22:19.357 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:22:19.357 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:22:19.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:19.358+0000 7f5fbb8f6700 1 -- 172.21.15.67:0/706680351 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f5fa006c4e0 msgr2=0x7f5fa006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:19.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:19.358+0000 7f5fbb8f6700 1 --2- 172.21.15.67:0/706680351 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f5fa006c4e0 0x7f5fa006e9a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5fb0006010 tx=0x7f5fb000b580 comp rx=0 tx=0).stop 2024-04-22T23:22:19.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:19.358+0000 7f5fbb8f6700 1 -- 172.21.15.67:0/706680351 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5fb4102290 msgr2=0x7f5fb4196a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:19.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:19.358+0000 7f5fbb8f6700 1 --2- 172.21.15.67:0/706680351 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5fb4102290 0x7f5fb4196a70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5fa400d8d0 tx=0x7f5fa400dbe0 comp rx=0 tx=0).stop 2024-04-22T23:22:19.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:19.358+0000 7f5fbb8f6700 1 -- 172.21.15.67:0/706680351 shutdown_connections 2024-04-22T23:22:19.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:19.358+0000 7f5fbb8f6700 1 --2- 172.21.15.67:0/706680351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fb40fed10 0x7f5fb4196530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:19.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:19.358+0000 7f5fbb8f6700 1 --2- 172.21.15.67:0/706680351 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f5fa006c4e0 0x7f5fa006e9a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:19.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:19.358+0000 7f5fbb8f6700 1 --2- 172.21.15.67:0/706680351 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5fb4102290 0x7f5fb4196a70 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:19.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:19.358+0000 7f5fbb8f6700 1 -- 172.21.15.67:0/706680351 >> 172.21.15.67:0/706680351 conn(0x7f5fb40fa8b0 msgr2=0x7f5fb4105260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:22:19.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:19.358+0000 7f5fbb8f6700 1 -- 172.21.15.67:0/706680351 shutdown_connections 2024-04-22T23:22:19.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:19.358+0000 7f5fbb8f6700 1 -- 172.21.15.67:0/706680351 wait complete. 2024-04-22T23:22:19.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:19 smithi067 ceph-mon[26453]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:22:19.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:19 smithi067 ceph-mon[26453]: from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:22:19.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:19 smithi067 ceph-mon[26453]: from='client.14590 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:22:19.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:19 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/1560793223' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:22:19.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:19 smithi134 ceph-mon[34609]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:22:19.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:19 smithi134 ceph-mon[34609]: from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:22:19.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:19 smithi134 ceph-mon[34609]: from='client.14590 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:22:19.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:19 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/1560793223' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:22:20.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:20 smithi067 ceph-mon[26453]: from='client.24357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:22:20.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:20 smithi067 ceph-mon[26453]: pgmap v235: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:22:20.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:20 smithi134 ceph-mon[34609]: from='client.24357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:22:20.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:20 smithi134 ceph-mon[34609]: pgmap v235: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:22:22.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:22 smithi067 ceph-mon[26453]: pgmap v236: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:22:22.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:22 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:22:22.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:22 smithi134 ceph-mon[34609]: pgmap v236: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:22:22.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:22 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:22:24.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:24 smithi067 ceph-mon[26453]: pgmap v237: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:22:24.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:24 smithi134 ceph-mon[34609]: pgmap v237: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:22:26.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:26 smithi067 ceph-mon[26453]: pgmap v238: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T23:22:26.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:26 smithi134 ceph-mon[34609]: pgmap v238: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T23:22:28.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:28 smithi067 ceph-mon[26453]: pgmap v239: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T23:22:28.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:28 smithi134 ceph-mon[34609]: pgmap v239: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-22T23:22:30.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:30 smithi067 ceph-mon[26453]: pgmap v240: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:22:30.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:30 smithi134 ceph-mon[34609]: pgmap v240: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:22:32.302 INFO:tasks.workunit.client.0.smithi067.stderr:Updating files: 93% (11438/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-04-22T23:22:32.688 INFO:tasks.workunit.client.1.smithi134.stderr:Updating files: 98% (12050/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-04-22T23:22:32.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:32 smithi067 ceph-mon[26453]: pgmap v241: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:22:32.843 INFO:tasks.workunit.client.0.smithi067.stderr:Note: switching to '43be020184947e53516056c9931e1ac5bdbbb1a5'. 2024-04-22T23:22:32.844 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-04-22T23:22:32.844 INFO:tasks.workunit.client.0.smithi067.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-22T23:22:32.844 INFO:tasks.workunit.client.0.smithi067.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-22T23:22:32.844 INFO:tasks.workunit.client.0.smithi067.stderr:state without impacting any branches by switching back to a branch. 2024-04-22T23:22:32.844 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-04-22T23:22:32.844 INFO:tasks.workunit.client.0.smithi067.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-22T23:22:32.844 INFO:tasks.workunit.client.0.smithi067.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-22T23:22:32.844 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-04-22T23:22:32.844 INFO:tasks.workunit.client.0.smithi067.stderr: git switch -c 2024-04-22T23:22:32.844 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-04-22T23:22:32.844 INFO:tasks.workunit.client.0.smithi067.stderr:Or undo this operation with: 2024-04-22T23:22:32.844 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-04-22T23:22:32.845 INFO:tasks.workunit.client.0.smithi067.stderr: git switch - 2024-04-22T23:22:32.845 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-04-22T23:22:32.845 INFO:tasks.workunit.client.0.smithi067.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-22T23:22:32.845 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-04-22T23:22:32.845 INFO:tasks.workunit.client.0.smithi067.stderr:HEAD is now at 43be0201849 Merge branch 'host-drain-test-ignore-stray-host' of https://github.com/adk3798/ceph into wip-adk-testing-2024-04-22-1618 2024-04-22T23:22:32.850 DEBUG:teuthology.orchestra.run.smithi067:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-22T23:22:32.881 INFO:tasks.workunit.client.0.smithi067.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-22T23:22:32.883 INFO:tasks.workunit.client.0.smithi067.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-22T23:22:32.883 INFO:tasks.workunit.client.0.smithi067.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-22T23:22:32.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:32 smithi134 ceph-mon[34609]: pgmap v241: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:22:33.446 INFO:tasks.workunit.client.1.smithi134.stderr:Note: switching to '43be020184947e53516056c9931e1ac5bdbbb1a5'. 2024-04-22T23:22:33.447 INFO:tasks.workunit.client.1.smithi134.stderr: 2024-04-22T23:22:33.447 INFO:tasks.workunit.client.1.smithi134.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-22T23:22:33.447 INFO:tasks.workunit.client.1.smithi134.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-22T23:22:33.447 INFO:tasks.workunit.client.1.smithi134.stderr:state without impacting any branches by switching back to a branch. 2024-04-22T23:22:33.447 INFO:tasks.workunit.client.1.smithi134.stderr: 2024-04-22T23:22:33.447 INFO:tasks.workunit.client.1.smithi134.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-22T23:22:33.447 INFO:tasks.workunit.client.1.smithi134.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-22T23:22:33.447 INFO:tasks.workunit.client.1.smithi134.stderr: 2024-04-22T23:22:33.447 INFO:tasks.workunit.client.1.smithi134.stderr: git switch -c 2024-04-22T23:22:33.447 INFO:tasks.workunit.client.1.smithi134.stderr: 2024-04-22T23:22:33.447 INFO:tasks.workunit.client.1.smithi134.stderr:Or undo this operation with: 2024-04-22T23:22:33.447 INFO:tasks.workunit.client.1.smithi134.stderr: 2024-04-22T23:22:33.447 INFO:tasks.workunit.client.1.smithi134.stderr: git switch - 2024-04-22T23:22:33.448 INFO:tasks.workunit.client.1.smithi134.stderr: 2024-04-22T23:22:33.448 INFO:tasks.workunit.client.1.smithi134.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-22T23:22:33.448 INFO:tasks.workunit.client.1.smithi134.stderr: 2024-04-22T23:22:33.448 INFO:tasks.workunit.client.1.smithi134.stderr:HEAD is now at 43be0201849 Merge branch 'host-drain-test-ignore-stray-host' of https://github.com/adk3798/ceph into wip-adk-testing-2024-04-22-1618 2024-04-22T23:22:33.452 DEBUG:teuthology.orchestra.run.smithi134:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-04-22T23:22:33.492 INFO:tasks.workunit.client.0.smithi067.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-22T23:22:33.549 INFO:tasks.workunit.client.0.smithi067.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-22T23:22:33.590 INFO:tasks.workunit.client.0.smithi067.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-22T23:22:33.592 INFO:tasks.workunit.client.0.smithi067.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-22T23:22:33.592 INFO:tasks.workunit.client.0.smithi067.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-22T23:22:33.625 INFO:tasks.workunit.client.1.smithi134.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-22T23:22:33.628 INFO:tasks.workunit.client.1.smithi134.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-04-22T23:22:33.628 INFO:tasks.workunit.client.1.smithi134.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-22T23:22:33.631 INFO:tasks.workunit.client.0.smithi067.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-22T23:22:33.635 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-22T23:22:33.635 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-22T23:22:33.691 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-04-22T23:22:33.693 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-04-22T23:22:33.693 DEBUG:teuthology.orchestra.run.smithi067:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=43be020184947e53516056c9931e1ac5bdbbb1a5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-04-22T23:22:33.758 INFO:tasks.workunit.client.0.smithi067.stderr:+ mkdir -p fsstress 2024-04-22T23:22:33.761 INFO:tasks.workunit.client.0.smithi067.stderr:+ pushd fsstress 2024-04-22T23:22:33.762 INFO:tasks.workunit.client.0.smithi067.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-04-22T23:22:33.762 INFO:tasks.workunit.client.0.smithi067.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-04-22T23:22:34.216 INFO:tasks.workunit.client.1.smithi134.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-22T23:22:34.274 INFO:tasks.workunit.client.1.smithi134.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-22T23:22:34.310 INFO:tasks.workunit.client.0.smithi067.stderr:+ tar xzf ltp-full.tgz 2024-04-22T23:22:34.313 INFO:tasks.workunit.client.1.smithi134.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-04-22T23:22:34.315 INFO:tasks.workunit.client.1.smithi134.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-04-22T23:22:34.315 INFO:tasks.workunit.client.1.smithi134.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-22T23:22:34.357 INFO:tasks.workunit.client.1.smithi134.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-04-22T23:22:34.361 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-22T23:22:34.361 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-04-22T23:22:34.377 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-04-22T23:22:34.378 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-04-22T23:22:34.378 DEBUG:teuthology.orchestra.run.smithi134:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=43be020184947e53516056c9931e1ac5bdbbb1a5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-04-22T23:22:34.441 INFO:tasks.workunit.client.1.smithi134.stderr:+ mkdir -p fsstress 2024-04-22T23:22:34.443 INFO:tasks.workunit.client.1.smithi134.stderr:+ pushd fsstress 2024-04-22T23:22:34.444 INFO:tasks.workunit.client.1.smithi134.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-22T23:22:34.444 INFO:tasks.workunit.client.1.smithi134.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-04-22T23:22:34.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:34 smithi067 ceph-mon[26453]: pgmap v242: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:22:34.825 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:34 smithi134 ceph-mon[34609]: pgmap v242: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:22:35.023 INFO:tasks.workunit.client.1.smithi134.stderr:+ tar xzf ltp-full.tgz 2024-04-22T23:22:36.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:36 smithi067 ceph-mon[26453]: pgmap v243: 65 pgs: 65 active+clean; 3.9 MiB data, 170 MiB used, 536 GiB / 536 GiB avail; 3.3 KiB/s rd, 287 KiB/s wr, 8 op/s 2024-04-22T23:22:36.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:36 smithi134 ceph-mon[34609]: pgmap v243: 65 pgs: 65 active+clean; 3.9 MiB data, 170 MiB used, 536 GiB / 536 GiB avail; 3.3 KiB/s rd, 287 KiB/s wr, 8 op/s 2024-04-22T23:22:37.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:37 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:22:37.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:37 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:22:38.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:38 smithi067 ceph-mon[26453]: pgmap v244: 65 pgs: 65 active+clean; 20 MiB data, 210 MiB used, 536 GiB / 536 GiB avail; 3.3 KiB/s rd, 1.6 MiB/s wr, 26 op/s 2024-04-22T23:22:38.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:38 smithi134 ceph-mon[34609]: pgmap v244: 65 pgs: 65 active+clean; 20 MiB data, 210 MiB used, 536 GiB / 536 GiB avail; 3.3 KiB/s rd, 1.6 MiB/s wr, 26 op/s 2024-04-22T23:22:40.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:40 smithi067 ceph-mon[26453]: pgmap v245: 65 pgs: 65 active+clean; 25 MiB data, 233 MiB used, 536 GiB / 536 GiB avail; 2.9 KiB/s rd, 2.1 MiB/s wr, 50 op/s 2024-04-22T23:22:40.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:40 smithi134 ceph-mon[34609]: pgmap v245: 65 pgs: 65 active+clean; 25 MiB data, 233 MiB used, 536 GiB / 536 GiB avail; 2.9 KiB/s rd, 2.1 MiB/s wr, 50 op/s 2024-04-22T23:22:42.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:42 smithi067 ceph-mon[26453]: pgmap v246: 65 pgs: 65 active+clean; 35 MiB data, 269 MiB used, 536 GiB / 536 GiB avail; 589 KiB/s rd, 2.9 MiB/s wr, 110 op/s 2024-04-22T23:22:42.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:42 smithi134 ceph-mon[34609]: pgmap v246: 65 pgs: 65 active+clean; 35 MiB data, 269 MiB used, 536 GiB / 536 GiB avail; 589 KiB/s rd, 2.9 MiB/s wr, 110 op/s 2024-04-22T23:22:44.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:44 smithi067 ceph-mon[26453]: pgmap v247: 65 pgs: 65 active+clean; 36 MiB data, 275 MiB used, 536 GiB / 536 GiB avail; 588 KiB/s rd, 3.0 MiB/s wr, 119 op/s 2024-04-22T23:22:44.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:44 smithi067 ceph-mon[26453]: fsmap cephfs:2 {0=cephfs.smithi067.xyazmh=up:active,1=cephfs.smithi134.owipad=up:active} 2 up:standby-replay 2024-04-22T23:22:44.947 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:44 smithi134 ceph-mon[34609]: pgmap v247: 65 pgs: 65 active+clean; 36 MiB data, 275 MiB used, 536 GiB / 536 GiB avail; 588 KiB/s rd, 3.0 MiB/s wr, 119 op/s 2024-04-22T23:22:44.948 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:44 smithi134 ceph-mon[34609]: fsmap cephfs:2 {0=cephfs.smithi067.xyazmh=up:active,1=cephfs.smithi134.owipad=up:active} 2 up:standby-replay 2024-04-22T23:22:46.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:46 smithi067 ceph-mon[26453]: pgmap v248: 65 pgs: 65 active+clean; 50 MiB data, 307 MiB used, 536 GiB / 536 GiB avail; 1.4 MiB/s rd, 4.1 MiB/s wr, 149 op/s 2024-04-22T23:22:46.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:46 smithi134 ceph-mon[34609]: pgmap v248: 65 pgs: 65 active+clean; 50 MiB data, 307 MiB used, 536 GiB / 536 GiB avail; 1.4 MiB/s rd, 4.1 MiB/s wr, 149 op/s 2024-04-22T23:22:48.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:48 smithi067 ceph-mon[26453]: pgmap v249: 65 pgs: 65 active+clean; 57 MiB data, 353 MiB used, 536 GiB / 536 GiB avail; 2.0 MiB/s rd, 4.5 MiB/s wr, 161 op/s 2024-04-22T23:22:48.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:48 smithi134 ceph-mon[34609]: pgmap v249: 65 pgs: 65 active+clean; 57 MiB data, 353 MiB used, 536 GiB / 536 GiB avail; 2.0 MiB/s rd, 4.5 MiB/s wr, 161 op/s 2024-04-22T23:22:49.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.493+0000 7fa2cf1cb700 1 -- 172.21.15.67:0/2429145332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2c80ffbb0 msgr2=0x7fa2c80fffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:49.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.493+0000 7fa2cf1cb700 1 --2- 172.21.15.67:0/2429145332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2c80ffbb0 0x7fa2c80fffd0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc009910 tx=0x7fa2bc009c20 comp rx=0 tx=0).stop 2024-04-22T23:22:49.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.493+0000 7fa2cf1cb700 1 -- 172.21.15.67:0/2429145332 shutdown_connections 2024-04-22T23:22:49.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.493+0000 7fa2cf1cb700 1 --2- 172.21.15.67:0/2429145332 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa2c8100db0 0x7fa2c8101210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:49.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.493+0000 7fa2cf1cb700 1 --2- 172.21.15.67:0/2429145332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2c80ffbb0 0x7fa2c80fffd0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:49.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.493+0000 7fa2cf1cb700 1 -- 172.21.15.67:0/2429145332 >> 172.21.15.67:0/2429145332 conn(0x7fa2c80fb150 msgr2=0x7fa2c80fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:22:49.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.493+0000 7fa2cf1cb700 1 -- 172.21.15.67:0/2429145332 shutdown_connections 2024-04-22T23:22:49.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.493+0000 7fa2cf1cb700 1 -- 172.21.15.67:0/2429145332 wait complete. 2024-04-22T23:22:49.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.494+0000 7fa2cf1cb700 1 Processor -- start 2024-04-22T23:22:49.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.494+0000 7fa2cf1cb700 1 -- start start 2024-04-22T23:22:49.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.494+0000 7fa2cf1cb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2c80ffbb0 0x7fa2c81964c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:22:49.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.494+0000 7fa2cf1cb700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa2c8100db0 0x7fa2c8196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:22:49.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.494+0000 7fa2cf1cb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c8197020 con 0x7fa2c80ffbb0 2024-04-22T23:22:49.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.494+0000 7fa2cf1cb700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c8197160 con 0x7fa2c8100db0 2024-04-22T23:22:49.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.494+0000 7fa2ccf67700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2c80ffbb0 0x7fa2c81964c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:22:49.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.494+0000 7fa2ccf67700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2c80ffbb0 0x7fa2c81964c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37790/0 (socket says 172.21.15.67:37790) 2024-04-22T23:22:49.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.494+0000 7fa2ccf67700 1 -- 172.21.15.67:0/3722040819 learned_addr learned my addr 172.21.15.67:0/3722040819 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:22:49.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.494+0000 7fa2c7fff700 1 --2- 172.21.15.67:0/3722040819 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa2c8100db0 0x7fa2c8196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:22:49.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.495+0000 7fa2ccf67700 1 -- 172.21.15.67:0/3722040819 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa2c8100db0 msgr2=0x7fa2c8196a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:49.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.495+0000 7fa2ccf67700 1 --2- 172.21.15.67:0/3722040819 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa2c8100db0 0x7fa2c8196a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:49.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.495+0000 7fa2ccf67700 1 -- 172.21.15.67:0/3722040819 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2bc0095f0 con 0x7fa2c80ffbb0 2024-04-22T23:22:49.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.495+0000 7fa2ccf67700 1 --2- 172.21.15.67:0/3722040819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2c80ffbb0 0x7fa2c81964c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc000c00 tx=0x7fa2bc00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:22:49.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.495+0000 7fa2c5ffb700 1 -- 172.21.15.67:0/3722040819 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2bc01d070 con 0x7fa2c80ffbb0 2024-04-22T23:22:49.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.495+0000 7fa2c5ffb700 1 -- 172.21.15.67:0/3722040819 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa2bc005dc0 con 0x7fa2c80ffbb0 2024-04-22T23:22:49.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.495+0000 7fa2cf1cb700 1 -- 172.21.15.67:0/3722040819 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2c819bbb0 con 0x7fa2c80ffbb0 2024-04-22T23:22:49.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.495+0000 7fa2c5ffb700 1 -- 172.21.15.67:0/3722040819 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2bc022470 con 0x7fa2c80ffbb0 2024-04-22T23:22:49.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.496+0000 7fa2cf1cb700 1 -- 172.21.15.67:0/3722040819 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2c819c0a0 con 0x7fa2c80ffbb0 2024-04-22T23:22:49.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.497+0000 7fa2cf1cb700 1 -- 172.21.15.67:0/3722040819 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2c8108710 con 0x7fa2c80ffbb0 2024-04-22T23:22:49.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.498+0000 7fa2c5ffb700 1 -- 172.21.15.67:0/3722040819 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fa2bc003810 con 0x7fa2c80ffbb0 2024-04-22T23:22:49.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.498+0000 7fa2c5ffb700 1 --2- 172.21.15.67:0/3722040819 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa2b806c530 0x7fa2b806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:22:49.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.498+0000 7fa2c5ffb700 1 -- 172.21.15.67:0/3722040819 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fa2bc08cbc0 con 0x7fa2c80ffbb0 2024-04-22T23:22:49.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.499+0000 7fa2c7fff700 1 --2- 172.21.15.67:0/3722040819 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa2b806c530 0x7fa2b806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:22:49.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.499+0000 7fa2c7fff700 1 --2- 172.21.15.67:0/3722040819 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa2b806c530 0x7fa2b806e9f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa2b4009c60 tx=0x7fa2b4009260 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:22:49.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.505+0000 7fa2c5ffb700 1 -- 172.21.15.67:0/3722040819 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa2bc05afa0 con 0x7fa2c80ffbb0 2024-04-22T23:22:49.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.805+0000 7fa2cf1cb700 1 -- 172.21.15.67:0/3722040819 --> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa2c8069520 con 0x7fa2b806c530 2024-04-22T23:22:49.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.807+0000 7fa2c5ffb700 1 -- 172.21.15.67:0/3722040819 <== mgr.14215 v2:172.21.15.67:6800/2589747850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7fa2c8069520 con 0x7fa2b806c530 2024-04-22T23:22:49.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.809+0000 7fa2cf1cb700 1 -- 172.21.15.67:0/3722040819 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa2b806c530 msgr2=0x7fa2b806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:49.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.809+0000 7fa2cf1cb700 1 --2- 172.21.15.67:0/3722040819 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa2b806c530 0x7fa2b806e9f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa2b4009c60 tx=0x7fa2b4009260 comp rx=0 tx=0).stop 2024-04-22T23:22:49.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.809+0000 7fa2cf1cb700 1 -- 172.21.15.67:0/3722040819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2c80ffbb0 msgr2=0x7fa2c81964c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:49.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.809+0000 7fa2cf1cb700 1 --2- 172.21.15.67:0/3722040819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2c80ffbb0 0x7fa2c81964c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc000c00 tx=0x7fa2bc00f690 comp rx=0 tx=0).stop 2024-04-22T23:22:49.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.809+0000 7fa2cf1cb700 1 -- 172.21.15.67:0/3722040819 shutdown_connections 2024-04-22T23:22:49.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.809+0000 7fa2cf1cb700 1 --2- 172.21.15.67:0/3722040819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2c80ffbb0 0x7fa2c81964c0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:49.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.809+0000 7fa2cf1cb700 1 --2- 172.21.15.67:0/3722040819 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa2b806c530 0x7fa2b806e9f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:49.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.810+0000 7fa2cf1cb700 1 --2- 172.21.15.67:0/3722040819 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa2c8100db0 0x7fa2c8196a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:49.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.810+0000 7fa2cf1cb700 1 -- 172.21.15.67:0/3722040819 >> 172.21.15.67:0/3722040819 conn(0x7fa2c80fb150 msgr2=0x7fa2c8068e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:22:49.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.810+0000 7fa2cf1cb700 1 -- 172.21.15.67:0/3722040819 shutdown_connections 2024-04-22T23:22:49.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.810+0000 7fa2cf1cb700 1 -- 172.21.15.67:0/3722040819 wait complete. 2024-04-22T23:22:49.825 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:22:49.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.956+0000 7fea489c4700 1 -- 172.21.15.67:0/1573752637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea440ff5f0 msgr2=0x7fea440ffa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:49.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.956+0000 7fea489c4700 1 --2- 172.21.15.67:0/1573752637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea440ff5f0 0x7fea440ffa10 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fea34009910 tx=0x7fea34009c20 comp rx=0 tx=0).stop 2024-04-22T23:22:49.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.956+0000 7fea489c4700 1 -- 172.21.15.67:0/1573752637 shutdown_connections 2024-04-22T23:22:49.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.956+0000 7fea489c4700 1 --2- 172.21.15.67:0/1573752637 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fea440fff50 0x7fea441003d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:49.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.956+0000 7fea489c4700 1 --2- 172.21.15.67:0/1573752637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea440ff5f0 0x7fea440ffa10 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:49.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.956+0000 7fea489c4700 1 -- 172.21.15.67:0/1573752637 >> 172.21.15.67:0/1573752637 conn(0x7fea440fb190 msgr2=0x7fea440fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:22:49.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.956+0000 7fea489c4700 1 -- 172.21.15.67:0/1573752637 shutdown_connections 2024-04-22T23:22:49.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.956+0000 7fea489c4700 1 -- 172.21.15.67:0/1573752637 wait complete. 2024-04-22T23:22:49.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.957+0000 7fea489c4700 1 Processor -- start 2024-04-22T23:22:49.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.957+0000 7fea489c4700 1 -- start start 2024-04-22T23:22:49.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.957+0000 7fea489c4700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fea440ff5f0 0x7fea441986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:22:49.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.957+0000 7fea489c4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea440fff50 0x7fea44198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:22:49.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.957+0000 7fea489c4700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea44199220 con 0x7fea440fff50 2024-04-22T23:22:49.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.957+0000 7fea489c4700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea44199360 con 0x7fea440ff5f0 2024-04-22T23:22:49.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.957+0000 7fea41d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea440fff50 0x7fea44198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:22:49.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.957+0000 7fea41d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea440fff50 0x7fea44198c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37806/0 (socket says 172.21.15.67:37806) 2024-04-22T23:22:49.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.957+0000 7fea41d9b700 1 -- 172.21.15.67:0/3410928 learned_addr learned my addr 172.21.15.67:0/3410928 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:22:49.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.957+0000 7fea4259c700 1 --2- 172.21.15.67:0/3410928 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fea440ff5f0 0x7fea441986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:22:49.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.958+0000 7fea41d9b700 1 -- 172.21.15.67:0/3410928 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fea440ff5f0 msgr2=0x7fea441986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:49.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.958+0000 7fea41d9b700 1 --2- 172.21.15.67:0/3410928 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fea440ff5f0 0x7fea441986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:49.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.958+0000 7fea41d9b700 1 -- 172.21.15.67:0/3410928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea340095f0 con 0x7fea440fff50 2024-04-22T23:22:49.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.958+0000 7fea41d9b700 1 --2- 172.21.15.67:0/3410928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea440fff50 0x7fea44198c00 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fea3800e920 tx=0x7fea3800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:22:49.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.958+0000 7fea337fe700 1 -- 172.21.15.67:0/3410928 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea3800c9d0 con 0x7fea440fff50 2024-04-22T23:22:49.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.958+0000 7fea489c4700 1 -- 172.21.15.67:0/3410928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea4419de10 con 0x7fea440fff50 2024-04-22T23:22:49.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.959+0000 7fea337fe700 1 -- 172.21.15.67:0/3410928 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fea38010430 con 0x7fea440fff50 2024-04-22T23:22:49.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.959+0000 7fea337fe700 1 -- 172.21.15.67:0/3410928 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea38018650 con 0x7fea440fff50 2024-04-22T23:22:49.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.959+0000 7fea489c4700 1 -- 172.21.15.67:0/3410928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea4419e270 con 0x7fea440fff50 2024-04-22T23:22:49.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.960+0000 7fea489c4700 1 -- 172.21.15.67:0/3410928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea4410b9f0 con 0x7fea440fff50 2024-04-22T23:22:49.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.960+0000 7fea337fe700 1 -- 172.21.15.67:0/3410928 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fea380187b0 con 0x7fea440fff50 2024-04-22T23:22:49.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.961+0000 7fea337fe700 1 --2- 172.21.15.67:0/3410928 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fea2c06c4e0 0x7fea2c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:22:49.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.961+0000 7fea337fe700 1 -- 172.21.15.67:0/3410928 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fea38014070 con 0x7fea440fff50 2024-04-22T23:22:49.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.961+0000 7fea4259c700 1 --2- 172.21.15.67:0/3410928 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fea2c06c4e0 0x7fea2c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:22:49.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.962+0000 7fea4259c700 1 --2- 172.21.15.67:0/3410928 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fea2c06c4e0 0x7fea2c06e9a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fea34009fd0 tx=0x7fea34005fd0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:22:49.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:49.968+0000 7fea337fe700 1 -- 172.21.15.67:0/3410928 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fea380520e0 con 0x7fea440fff50 2024-04-22T23:22:50.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.378+0000 7fea489c4700 1 -- 172.21.15.67:0/3410928 --> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fea44109340 con 0x7fea2c06c4e0 2024-04-22T23:22:50.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.379+0000 7fea337fe700 1 -- 172.21.15.67:0/3410928 <== mgr.14215 v2:172.21.15.67:6800/2589747850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7fea44109340 con 0x7fea2c06c4e0 2024-04-22T23:22:50.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.383+0000 7fea489c4700 1 -- 172.21.15.67:0/3410928 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fea2c06c4e0 msgr2=0x7fea2c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:50.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.383+0000 7fea489c4700 1 --2- 172.21.15.67:0/3410928 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fea2c06c4e0 0x7fea2c06e9a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fea34009fd0 tx=0x7fea34005fd0 comp rx=0 tx=0).stop 2024-04-22T23:22:50.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.383+0000 7fea489c4700 1 -- 172.21.15.67:0/3410928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea440fff50 msgr2=0x7fea44198c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:50.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.383+0000 7fea489c4700 1 --2- 172.21.15.67:0/3410928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea440fff50 0x7fea44198c00 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fea3800e920 tx=0x7fea3800ec30 comp rx=0 tx=0).stop 2024-04-22T23:22:50.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.383+0000 7fea489c4700 1 -- 172.21.15.67:0/3410928 shutdown_connections 2024-04-22T23:22:50.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.383+0000 7fea489c4700 1 --2- 172.21.15.67:0/3410928 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fea2c06c4e0 0x7fea2c06e9a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:50.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.383+0000 7fea489c4700 1 --2- 172.21.15.67:0/3410928 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fea440ff5f0 0x7fea441986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:50.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.383+0000 7fea489c4700 1 --2- 172.21.15.67:0/3410928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea440fff50 0x7fea44198c00 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:50.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.383+0000 7fea489c4700 1 -- 172.21.15.67:0/3410928 >> 172.21.15.67:0/3410928 conn(0x7fea440fb190 msgr2=0x7fea44107c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:22:50.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.383+0000 7fea489c4700 1 -- 172.21.15.67:0/3410928 shutdown_connections 2024-04-22T23:22:50.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.383+0000 7fea489c4700 1 -- 172.21.15.67:0/3410928 wait complete. 2024-04-22T23:22:50.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.520+0000 7ff987534700 1 -- 172.21.15.67:0/3158846734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9801017a0 msgr2=0x7ff980103b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:50.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.520+0000 7ff987534700 1 --2- 172.21.15.67:0/3158846734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9801017a0 0x7ff980103b90 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff97c009910 tx=0x7ff97c009c20 comp rx=0 tx=0).stop 2024-04-22T23:22:50.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.520+0000 7ff987534700 1 -- 172.21.15.67:0/3158846734 shutdown_connections 2024-04-22T23:22:50.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.520+0000 7ff987534700 1 --2- 172.21.15.67:0/3158846734 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9801040d0 0x7ff9801064c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:50.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.520+0000 7ff987534700 1 --2- 172.21.15.67:0/3158846734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9801017a0 0x7ff980103b90 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:50.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.520+0000 7ff987534700 1 -- 172.21.15.67:0/3158846734 >> 172.21.15.67:0/3158846734 conn(0x7ff9800fb190 msgr2=0x7ff9800fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:22:50.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.521+0000 7ff987534700 1 -- 172.21.15.67:0/3158846734 shutdown_connections 2024-04-22T23:22:50.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.521+0000 7ff987534700 1 -- 172.21.15.67:0/3158846734 wait complete. 2024-04-22T23:22:50.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.521+0000 7ff987534700 1 Processor -- start 2024-04-22T23:22:50.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.521+0000 7ff987534700 1 -- start start 2024-04-22T23:22:50.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.522+0000 7ff987534700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9801017a0 0x7ff980196460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:22:50.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.522+0000 7ff987534700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9801040d0 0x7ff9801969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:22:50.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.522+0000 7ff987534700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980196fc0 con 0x7ff9801040d0 2024-04-22T23:22:50.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.522+0000 7ff987534700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980197100 con 0x7ff9801017a0 2024-04-22T23:22:50.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.522+0000 7ff984acf700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9801040d0 0x7ff9801969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:22:50.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.522+0000 7ff9852d0700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9801017a0 0x7ff980196460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:22:50.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.522+0000 7ff9852d0700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9801017a0 0x7ff980196460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.67:54210/0 (socket says 172.21.15.67:54210) 2024-04-22T23:22:50.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.522+0000 7ff984acf700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9801040d0 0x7ff9801969a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37826/0 (socket says 172.21.15.67:37826) 2024-04-22T23:22:50.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.522+0000 7ff9852d0700 1 -- 172.21.15.67:0/4074076876 learned_addr learned my addr 172.21.15.67:0/4074076876 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:22:50.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.522+0000 7ff984acf700 1 -- 172.21.15.67:0/4074076876 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9801017a0 msgr2=0x7ff980196460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:50.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.522+0000 7ff984acf700 1 --2- 172.21.15.67:0/4074076876 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9801017a0 0x7ff980196460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:50.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.522+0000 7ff984acf700 1 -- 172.21.15.67:0/4074076876 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff97c0095f0 con 0x7ff9801040d0 2024-04-22T23:22:50.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.522+0000 7ff9852d0700 1 --2- 172.21.15.67:0/4074076876 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9801017a0 0x7ff980196460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T23:22:50.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.522+0000 7ff984acf700 1 --2- 172.21.15.67:0/4074076876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9801040d0 0x7ff9801969a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff97000e970 tx=0x7ff97000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:22:50.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.523+0000 7ff9767fc700 1 -- 172.21.15.67:0/4074076876 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff97000c9d0 con 0x7ff9801040d0 2024-04-22T23:22:50.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.523+0000 7ff987534700 1 -- 172.21.15.67:0/4074076876 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff98019bbb0 con 0x7ff9801040d0 2024-04-22T23:22:50.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.523+0000 7ff987534700 1 -- 172.21.15.67:0/4074076876 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff98019c100 con 0x7ff9801040d0 2024-04-22T23:22:50.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.523+0000 7ff9767fc700 1 -- 172.21.15.67:0/4074076876 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff970010430 con 0x7ff9801040d0 2024-04-22T23:22:50.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.523+0000 7ff9767fc700 1 -- 172.21.15.67:0/4074076876 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff970018650 con 0x7ff9801040d0 2024-04-22T23:22:50.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.524+0000 7ff987534700 1 -- 172.21.15.67:0/4074076876 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff964005320 con 0x7ff9801040d0 2024-04-22T23:22:50.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.525+0000 7ff9767fc700 1 -- 172.21.15.67:0/4074076876 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7ff97000cb30 con 0x7ff9801040d0 2024-04-22T23:22:50.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.525+0000 7ff9767fc700 1 --2- 172.21.15.67:0/4074076876 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ff96c06c4e0 0x7ff96c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:22:50.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.526+0000 7ff9767fc700 1 -- 172.21.15.67:0/4074076876 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7ff970014070 con 0x7ff9801040d0 2024-04-22T23:22:50.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.526+0000 7ff9852d0700 1 --2- 172.21.15.67:0/4074076876 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ff96c06c4e0 0x7ff96c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:22:50.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.526+0000 7ff9852d0700 1 --2- 172.21.15.67:0/4074076876 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ff96c06c4e0 0x7ff96c06e9a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff97c006010 tx=0x7ff97c005a90 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:22:50.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.531+0000 7ff9767fc700 1 -- 172.21.15.67:0/4074076876 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff9700520e0 con 0x7ff9801040d0 2024-04-22T23:22:50.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:50 smithi067 ceph-mon[26453]: pgmap v250: 65 pgs: 65 active+clean; 65 MiB data, 382 MiB used, 536 GiB / 536 GiB avail; 2.1 MiB/s rd, 3.9 MiB/s wr, 166 op/s 2024-04-22T23:22:50.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.815+0000 7ff987534700 1 -- 172.21.15.67:0/4074076876 --> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff964000bf0 con 0x7ff96c06c4e0 2024-04-22T23:22:50.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.833+0000 7ff9767fc700 1 -- 172.21.15.67:0/4074076876 <== mgr.14215 v2:172.21.15.67:6800/2589747850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3456 (secure 0 0 0) 0x7ff964000bf0 con 0x7ff96c06c4e0 2024-04-22T23:22:50.836 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:22:50.836 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (6m) 2m ago 7m 26.7M - 0.25.0 c8568f914cd2 8c7b7ec00657 2024-04-22T23:22:50.836 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (8m) 2m ago 8m 7490k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:22:50.836 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (7m) 2m ago 7m 7453k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:22:50.836 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (8m) 2m ago 8m 6660k - 18.2.1 5be31c24972a 14085634c5d3 2024-04-22T23:22:50.836 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (7m) 2m ago 6m 6665k - 18.2.1 5be31c24972a fa995d458353 2024-04-22T23:22:50.836 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (6m) 2m ago 7m 85.0M - 9.4.7 954c08fa6188 0d8950c9af0c 2024-04-22T23:22:50.836 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (2m) 2m ago 2m 12.8M - 18.2.1 5be31c24972a e24a471f0a8c 2024-04-22T23:22:50.836 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (2m) 2m ago 2m 16.7M - 18.2.1 5be31c24972a 0730f9fc8e61 2024-04-22T23:22:50.836 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (2m) 2m ago 2m 17.6M - 18.2.1 5be31c24972a b6a169586433 2024-04-22T23:22:50.836 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (2m) 2m ago 2m 14.6M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:22:50.837 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:9283,8765,8443 running (9m) 2m ago 9m 490M - 18.2.1 5be31c24972a 1c64cb9e36bb 2024-04-22T23:22:50.837 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (6m) 2m ago 6m 443M - 18.2.1 5be31c24972a 0a0b76e72171 2024-04-22T23:22:50.837 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (9m) 2m ago 9m 46.1M 2048M 18.2.1 5be31c24972a 6789283703e0 2024-04-22T23:22:50.837 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (6m) 2m ago 6m 35.9M 2048M 18.2.1 5be31c24972a 2fb23795f7e3 2024-04-22T23:22:50.837 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (7m) 2m ago 7m 16.3M - 1.5.0 0da6a335fe13 4aa4da7089d0 2024-04-22T23:22:50.837 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (6m) 2m ago 6m 15.6M - 1.5.0 0da6a335fe13 26e43c2e756c 2024-04-22T23:22:50.837 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (6m) 2m ago 6m 48.5M 3946M 18.2.1 5be31c24972a a033247ee8c2 2024-04-22T23:22:50.837 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (5m) 2m ago 5m 47.1M 3946M 18.2.1 5be31c24972a 915b07a8edc4 2024-04-22T23:22:50.837 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (5m) 2m ago 5m 46.8M 3946M 18.2.1 5be31c24972a e257ecb4efa1 2024-04-22T23:22:50.837 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (4m) 2m ago 4m 45.9M 4970M 18.2.1 5be31c24972a 9c2c91a967ba 2024-04-22T23:22:50.837 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (4m) 2m ago 4m 44.0M 4970M 18.2.1 5be31c24972a c7586562ffa6 2024-04-22T23:22:50.837 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (3m) 2m ago 3m 44.0M 4970M 18.2.1 5be31c24972a 9fd97f1c65cb 2024-04-22T23:22:50.837 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (6m) 2m ago 7m 55.5M - 2.43.0 a07b618ecd1d d87cb15b5e55 2024-04-22T23:22:50.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.837+0000 7ff987534700 1 -- 172.21.15.67:0/4074076876 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ff96c06c4e0 msgr2=0x7ff96c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:50.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.837+0000 7ff987534700 1 --2- 172.21.15.67:0/4074076876 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ff96c06c4e0 0x7ff96c06e9a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff97c006010 tx=0x7ff97c005a90 comp rx=0 tx=0).stop 2024-04-22T23:22:50.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.837+0000 7ff987534700 1 -- 172.21.15.67:0/4074076876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9801040d0 msgr2=0x7ff9801969a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:50.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.837+0000 7ff987534700 1 --2- 172.21.15.67:0/4074076876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9801040d0 0x7ff9801969a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff97000e970 tx=0x7ff97000ec80 comp rx=0 tx=0).stop 2024-04-22T23:22:50.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.837+0000 7ff987534700 1 -- 172.21.15.67:0/4074076876 shutdown_connections 2024-04-22T23:22:50.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.837+0000 7ff987534700 1 --2- 172.21.15.67:0/4074076876 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ff96c06c4e0 0x7ff96c06e9a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:50.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.837+0000 7ff987534700 1 --2- 172.21.15.67:0/4074076876 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9801017a0 0x7ff980196460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:50.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.837+0000 7ff987534700 1 --2- 172.21.15.67:0/4074076876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9801040d0 0x7ff9801969a0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:50.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.837+0000 7ff987534700 1 -- 172.21.15.67:0/4074076876 >> 172.21.15.67:0/4074076876 conn(0x7ff9800fb190 msgr2=0x7ff9800fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:22:50.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.837+0000 7ff987534700 1 -- 172.21.15.67:0/4074076876 shutdown_connections 2024-04-22T23:22:50.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.837+0000 7ff987534700 1 -- 172.21.15.67:0/4074076876 wait complete. 2024-04-22T23:22:50.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.985+0000 7f47c080b700 1 -- 172.21.15.67:0/3531928801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47bc104070 msgr2=0x7f47bc106460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:50.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.985+0000 7f47c080b700 1 --2- 172.21.15.67:0/3531928801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47bc104070 0x7f47bc106460 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f47b0009910 tx=0x7f47b0009c20 comp rx=0 tx=0).stop 2024-04-22T23:22:50.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.985+0000 7f47c080b700 1 -- 172.21.15.67:0/3531928801 shutdown_connections 2024-04-22T23:22:50.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.985+0000 7f47c080b700 1 --2- 172.21.15.67:0/3531928801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47bc104070 0x7f47bc106460 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:50.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.985+0000 7f47c080b700 1 --2- 172.21.15.67:0/3531928801 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f47bc101740 0x7f47bc103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:50.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.985+0000 7f47c080b700 1 -- 172.21.15.67:0/3531928801 >> 172.21.15.67:0/3531928801 conn(0x7f47bc0fb150 msgr2=0x7f47bc0fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:22:50.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.985+0000 7f47c080b700 1 -- 172.21.15.67:0/3531928801 shutdown_connections 2024-04-22T23:22:50.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.985+0000 7f47c080b700 1 -- 172.21.15.67:0/3531928801 wait complete. 2024-04-22T23:22:50.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.986+0000 7f47c080b700 1 Processor -- start 2024-04-22T23:22:50.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.986+0000 7f47c080b700 1 -- start start 2024-04-22T23:22:50.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.986+0000 7f47c080b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f47bc101740 0x7f47bc1986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:22:50.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.986+0000 7f47c080b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47bc104070 0x7f47bc198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:22:50.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.986+0000 7f47c080b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47bc199200 con 0x7f47bc104070 2024-04-22T23:22:50.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.986+0000 7f47c080b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47bc199340 con 0x7f47bc101740 2024-04-22T23:22:50.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.986+0000 7f47b9d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47bc104070 0x7f47bc198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:22:50.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.986+0000 7f47b9d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47bc104070 0x7f47bc198be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37844/0 (socket says 172.21.15.67:37844) 2024-04-22T23:22:50.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.986+0000 7f47b9d9b700 1 -- 172.21.15.67:0/1250390677 learned_addr learned my addr 172.21.15.67:0/1250390677 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:22:50.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.986+0000 7f47ba59c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f47bc101740 0x7f47bc1986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:22:50.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.987+0000 7f47b9d9b700 1 -- 172.21.15.67:0/1250390677 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f47bc101740 msgr2=0x7f47bc1986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:50.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.987+0000 7f47b9d9b700 1 --2- 172.21.15.67:0/1250390677 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f47bc101740 0x7f47bc1986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:50.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.987+0000 7f47b9d9b700 1 -- 172.21.15.67:0/1250390677 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b00095f0 con 0x7f47bc104070 2024-04-22T23:22:50.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.987+0000 7f47b9d9b700 1 --2- 172.21.15.67:0/1250390677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47bc104070 0x7f47bc198be0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f47b0000c00 tx=0x7f47b000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:22:50.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.987+0000 7f47ab7fe700 1 -- 172.21.15.67:0/1250390677 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47b001d070 con 0x7f47bc104070 2024-04-22T23:22:50.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.987+0000 7f47c080b700 1 -- 172.21.15.67:0/1250390677 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47bc19dd90 con 0x7f47bc104070 2024-04-22T23:22:50.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.987+0000 7f47c080b700 1 -- 172.21.15.67:0/1250390677 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47bc19e280 con 0x7f47bc104070 2024-04-22T23:22:50.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.987+0000 7f47ab7fe700 1 -- 172.21.15.67:0/1250390677 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f47b0005dc0 con 0x7f47bc104070 2024-04-22T23:22:50.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.987+0000 7f47ab7fe700 1 -- 172.21.15.67:0/1250390677 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47b00226a0 con 0x7f47bc104070 2024-04-22T23:22:50.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.988+0000 7f47c080b700 1 -- 172.21.15.67:0/1250390677 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47bc0fcd20 con 0x7f47bc104070 2024-04-22T23:22:50.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.989+0000 7f47ab7fe700 1 -- 172.21.15.67:0/1250390677 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f47b00038f0 con 0x7f47bc104070 2024-04-22T23:22:50.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.990+0000 7f47ab7fe700 1 --2- 172.21.15.67:0/1250390677 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f47a406c4e0 0x7f47a406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:22:50.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.990+0000 7f47ab7fe700 1 -- 172.21.15.67:0/1250390677 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f47b008cc30 con 0x7f47bc104070 2024-04-22T23:22:50.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.990+0000 7f47ba59c700 1 --2- 172.21.15.67:0/1250390677 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f47a406c4e0 0x7f47a406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:22:50.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.990+0000 7f47ba59c700 1 --2- 172.21.15.67:0/1250390677 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f47a406c4e0 0x7f47a406e9a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f47ac009c60 tx=0x7f47ac009260 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:22:50.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:50 smithi134 ceph-mon[34609]: pgmap v250: 65 pgs: 65 active+clean; 65 MiB data, 382 MiB used, 536 GiB / 536 GiB avail; 2.1 MiB/s rd, 3.9 MiB/s wr, 166 op/s 2024-04-22T23:22:50.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:50.997+0000 7f47ab7fe700 1 -- 172.21.15.67:0/1250390677 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f47b005aec0 con 0x7f47bc104070 2024-04-22T23:22:51.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.387+0000 7f47c080b700 1 -- 172.21.15.67:0/1250390677 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f47bc02cc70 con 0x7f47bc104070 2024-04-22T23:22:51.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.387+0000 7f47ab7fe700 1 -- 172.21.15.67:0/1250390677 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+558 (secure 0 0 0) 0x7f47b005aa50 con 0x7f47bc104070 2024-04-22T23:22:51.390 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:22:51.390 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:22:51.390 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-22T23:22:51.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:22:51.390 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:22:51.390 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-22T23:22:51.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:22:51.390 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:22:51.390 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-22T23:22:51.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:22:51.390 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:22:51.390 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-22T23:22:51.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:22:51.391 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:22:51.391 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 14 2024-04-22T23:22:51.391 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:22:51.391 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:22:51.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.390+0000 7f47c080b700 1 -- 172.21.15.67:0/1250390677 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f47a406c4e0 msgr2=0x7f47a406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:51.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.390+0000 7f47c080b700 1 --2- 172.21.15.67:0/1250390677 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f47a406c4e0 0x7f47a406e9a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f47ac009c60 tx=0x7f47ac009260 comp rx=0 tx=0).stop 2024-04-22T23:22:51.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.390+0000 7f47c080b700 1 -- 172.21.15.67:0/1250390677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47bc104070 msgr2=0x7f47bc198be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:51.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.390+0000 7f47c080b700 1 --2- 172.21.15.67:0/1250390677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47bc104070 0x7f47bc198be0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f47b0000c00 tx=0x7f47b000f690 comp rx=0 tx=0).stop 2024-04-22T23:22:51.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.390+0000 7f47c080b700 1 -- 172.21.15.67:0/1250390677 shutdown_connections 2024-04-22T23:22:51.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.390+0000 7f47c080b700 1 --2- 172.21.15.67:0/1250390677 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f47a406c4e0 0x7f47a406e9a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:51.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.390+0000 7f47c080b700 1 --2- 172.21.15.67:0/1250390677 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f47bc101740 0x7f47bc1986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:51.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.390+0000 7f47c080b700 1 --2- 172.21.15.67:0/1250390677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47bc104070 0x7f47bc198be0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:51.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.390+0000 7f47c080b700 1 -- 172.21.15.67:0/1250390677 >> 172.21.15.67:0/1250390677 conn(0x7f47bc0fb150 msgr2=0x7f47bc100fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:22:51.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.390+0000 7f47c080b700 1 -- 172.21.15.67:0/1250390677 shutdown_connections 2024-04-22T23:22:51.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.390+0000 7f47c080b700 1 -- 172.21.15.67:0/1250390677 wait complete. 2024-04-22T23:22:51.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.525+0000 7f4b70afa700 1 -- 172.21.15.67:0/2501176691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b6c0ff610 msgr2=0x7f4b6c0ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:51.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.525+0000 7f4b70afa700 1 --2- 172.21.15.67:0/2501176691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b6c0ff610 0x7f4b6c0ffa30 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c009910 tx=0x7f4b5c009c20 comp rx=0 tx=0).stop 2024-04-22T23:22:51.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.525+0000 7f4b70afa700 1 -- 172.21.15.67:0/2501176691 shutdown_connections 2024-04-22T23:22:51.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.525+0000 7f4b70afa700 1 --2- 172.21.15.67:0/2501176691 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b6c0fff70 0x7f4b6c1003f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:51.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.525+0000 7f4b70afa700 1 --2- 172.21.15.67:0/2501176691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b6c0ff610 0x7f4b6c0ffa30 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:51.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.525+0000 7f4b70afa700 1 -- 172.21.15.67:0/2501176691 >> 172.21.15.67:0/2501176691 conn(0x7f4b6c0fb190 msgr2=0x7f4b6c0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:22:51.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.526+0000 7f4b70afa700 1 -- 172.21.15.67:0/2501176691 shutdown_connections 2024-04-22T23:22:51.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.526+0000 7f4b70afa700 1 -- 172.21.15.67:0/2501176691 wait complete. 2024-04-22T23:22:51.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.527+0000 7f4b70afa700 1 Processor -- start 2024-04-22T23:22:51.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.527+0000 7f4b70afa700 1 -- start start 2024-04-22T23:22:51.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.527+0000 7f4b70afa700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b6c0ff610 0x7f4b6c19caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:22:51.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.527+0000 7f4b70afa700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b6c0fff70 0x7f4b6c19cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:22:51.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.527+0000 7f4b70afa700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6c19d600 con 0x7f4b6c0ff610 2024-04-22T23:22:51.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.527+0000 7f4b70afa700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6c19d740 con 0x7f4b6c0fff70 2024-04-22T23:22:51.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.527+0000 7f4b6a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b6c0ff610 0x7f4b6c19caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:22:51.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.527+0000 7f4b6a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b6c0ff610 0x7f4b6c19caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37870/0 (socket says 172.21.15.67:37870) 2024-04-22T23:22:51.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.527+0000 7f4b6a59c700 1 -- 172.21.15.67:0/2438859022 learned_addr learned my addr 172.21.15.67:0/2438859022 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:22:51.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.527+0000 7f4b69d9b700 1 --2- 172.21.15.67:0/2438859022 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b6c0fff70 0x7f4b6c19cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:22:51.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.528+0000 7f4b6a59c700 1 -- 172.21.15.67:0/2438859022 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b6c0fff70 msgr2=0x7f4b6c19cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:51.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.528+0000 7f4b6a59c700 1 --2- 172.21.15.67:0/2438859022 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b6c0fff70 0x7f4b6c19cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:51.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.528+0000 7f4b6a59c700 1 -- 172.21.15.67:0/2438859022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b5c0095f0 con 0x7f4b6c0ff610 2024-04-22T23:22:51.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.528+0000 7f4b6a59c700 1 --2- 172.21.15.67:0/2438859022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b6c0ff610 0x7f4b6c19caa0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c000c00 tx=0x7f4b5c00f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:22:51.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.528+0000 7f4b5b7fe700 1 -- 172.21.15.67:0/2438859022 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b5c01d070 con 0x7f4b6c0ff610 2024-04-22T23:22:51.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.528+0000 7f4b70afa700 1 -- 172.21.15.67:0/2438859022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b6c1a2190 con 0x7f4b6c0ff610 2024-04-22T23:22:51.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.528+0000 7f4b5b7fe700 1 -- 172.21.15.67:0/2438859022 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4b5c005dc0 con 0x7f4b6c0ff610 2024-04-22T23:22:51.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.528+0000 7f4b5b7fe700 1 -- 172.21.15.67:0/2438859022 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b5c0226a0 con 0x7f4b6c0ff610 2024-04-22T23:22:51.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.528+0000 7f4b70afa700 1 -- 172.21.15.67:0/2438859022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b6c1a2680 con 0x7f4b6c0ff610 2024-04-22T23:22:51.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.529+0000 7f4b70afa700 1 -- 172.21.15.67:0/2438859022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b6c1066f0 con 0x7f4b6c0ff610 2024-04-22T23:22:51.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.530+0000 7f4b5b7fe700 1 -- 172.21.15.67:0/2438859022 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f4b5c017400 con 0x7f4b6c0ff610 2024-04-22T23:22:51.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.530+0000 7f4b5b7fe700 1 --2- 172.21.15.67:0/2438859022 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f4b5406c4e0 0x7f4b5406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:22:51.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.531+0000 7f4b5b7fe700 1 -- 172.21.15.67:0/2438859022 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f4b5c08cd70 con 0x7f4b6c0ff610 2024-04-22T23:22:51.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.532+0000 7f4b69d9b700 1 --2- 172.21.15.67:0/2438859022 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f4b5406c4e0 0x7f4b5406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:22:51.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.532+0000 7f4b69d9b700 1 --2- 172.21.15.67:0/2438859022 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f4b5406c4e0 0x7f4b5406e9a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4b60009bf0 tx=0x7f4b60009260 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:22:51.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.537+0000 7f4b5b7fe700 1 -- 172.21.15.67:0/2438859022 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f4b5c05b1c0 con 0x7f4b6c0ff610 2024-04-22T23:22:51.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.789+0000 7f4b70afa700 1 -- 172.21.15.67:0/2438859022 --> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b6c0611d0 con 0x7f4b5406c4e0 2024-04-22T23:22:51.792 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:51 smithi067 ceph-mon[26453]: from='client.14602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:22:51.792 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:51 smithi067 ceph-mon[26453]: from='client.14606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:22:51.792 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:51 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/1250390677' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:22:51.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.790+0000 7f4b5b7fe700 1 -- 172.21.15.67:0/2438859022 <== mgr.14215 v2:172.21.15.67:6800/2589747850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f4b6c0611d0 con 0x7f4b5406c4e0 2024-04-22T23:22:51.793 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:22:51.793 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:22:51.793 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:22:51.793 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-22T23:22:51.793 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [], 2024-04-22T23:22:51.793 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "0/2 daemons upgraded", 2024-04-22T23:22:51.793 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c image on host smithi134", 2024-04-22T23:22:51.793 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:22:51.793 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:22:51.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.793+0000 7f4b70afa700 1 -- 172.21.15.67:0/2438859022 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f4b5406c4e0 msgr2=0x7f4b5406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:51.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.793+0000 7f4b70afa700 1 --2- 172.21.15.67:0/2438859022 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f4b5406c4e0 0x7f4b5406e9a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4b60009bf0 tx=0x7f4b60009260 comp rx=0 tx=0).stop 2024-04-22T23:22:51.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.793+0000 7f4b70afa700 1 -- 172.21.15.67:0/2438859022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b6c0ff610 msgr2=0x7f4b6c19caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:22:51.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.793+0000 7f4b70afa700 1 --2- 172.21.15.67:0/2438859022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b6c0ff610 0x7f4b6c19caa0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c000c00 tx=0x7f4b5c00f740 comp rx=0 tx=0).stop 2024-04-22T23:22:51.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.793+0000 7f4b70afa700 1 -- 172.21.15.67:0/2438859022 shutdown_connections 2024-04-22T23:22:51.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.793+0000 7f4b70afa700 1 --2- 172.21.15.67:0/2438859022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b6c0ff610 0x7f4b6c19caa0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:51.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.793+0000 7f4b70afa700 1 --2- 172.21.15.67:0/2438859022 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f4b5406c4e0 0x7f4b5406e9a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:51.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.793+0000 7f4b70afa700 1 --2- 172.21.15.67:0/2438859022 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b6c0fff70 0x7f4b6c19cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:22:51.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.793+0000 7f4b70afa700 1 -- 172.21.15.67:0/2438859022 >> 172.21.15.67:0/2438859022 conn(0x7f4b6c0fb190 msgr2=0x7f4b6c103830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:22:51.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.793+0000 7f4b70afa700 1 -- 172.21.15.67:0/2438859022 shutdown_connections 2024-04-22T23:22:51.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:22:51.793+0000 7f4b70afa700 1 -- 172.21.15.67:0/2438859022 wait complete. 2024-04-22T23:22:51.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:51 smithi134 ceph-mon[34609]: from='client.14602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:22:51.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:51 smithi134 ceph-mon[34609]: from='client.14606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:22:51.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:51 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/1250390677' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:22:52.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:52 smithi067 ceph-mon[26453]: from='client.14610 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:22:52.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:52 smithi067 ceph-mon[26453]: pgmap v251: 65 pgs: 65 active+clean; 87 MiB data, 444 MiB used, 536 GiB / 536 GiB avail; 3.0 MiB/s rd, 5.2 MiB/s wr, 226 op/s 2024-04-22T23:22:52.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:52 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:22:52.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:52 smithi134 ceph-mon[34609]: from='client.14610 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:22:52.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:52 smithi134 ceph-mon[34609]: pgmap v251: 65 pgs: 65 active+clean; 87 MiB data, 444 MiB used, 536 GiB / 536 GiB avail; 3.0 MiB/s rd, 5.2 MiB/s wr, 226 op/s 2024-04-22T23:22:52.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:52 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:22:53.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:53 smithi134 ceph-mon[34609]: from='client.14618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:22:54.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:53 smithi067 ceph-mon[26453]: from='client.14618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:22:54.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:54 smithi134 ceph-mon[34609]: pgmap v252: 65 pgs: 65 active+clean; 90 MiB data, 452 MiB used, 536 GiB / 536 GiB avail; 2.6 MiB/s rd, 4.6 MiB/s wr, 182 op/s 2024-04-22T23:22:55.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:54 smithi067 ceph-mon[26453]: pgmap v252: 65 pgs: 65 active+clean; 90 MiB data, 452 MiB used, 536 GiB / 536 GiB avail; 2.6 MiB/s rd, 4.6 MiB/s wr, 182 op/s 2024-04-22T23:22:56.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:56 smithi067 ceph-mon[26453]: pgmap v253: 65 pgs: 65 active+clean; 103 MiB data, 501 MiB used, 536 GiB / 536 GiB avail; 3.1 MiB/s rd, 5.7 MiB/s wr, 211 op/s 2024-04-22T23:22:56.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:56 smithi134 ceph-mon[34609]: pgmap v253: 65 pgs: 65 active+clean; 103 MiB data, 501 MiB used, 536 GiB / 536 GiB avail; 3.1 MiB/s rd, 5.7 MiB/s wr, 211 op/s 2024-04-22T23:22:58.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:22:58 smithi067 ceph-mon[26453]: pgmap v254: 65 pgs: 65 active+clean; 107 MiB data, 535 MiB used, 536 GiB / 536 GiB avail; 2.5 MiB/s rd, 4.9 MiB/s wr, 225 op/s 2024-04-22T23:22:58.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:22:58 smithi134 ceph-mon[34609]: pgmap v254: 65 pgs: 65 active+clean; 107 MiB data, 535 MiB used, 536 GiB / 536 GiB avail; 2.5 MiB/s rd, 4.9 MiB/s wr, 225 op/s 2024-04-22T23:23:00.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:00 smithi067 ceph-mon[26453]: pgmap v255: 65 pgs: 65 active+clean; 115 MiB data, 559 MiB used, 536 GiB / 536 GiB avail; 2.6 MiB/s rd, 4.9 MiB/s wr, 233 op/s 2024-04-22T23:23:00.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:00 smithi134 ceph-mon[34609]: pgmap v255: 65 pgs: 65 active+clean; 115 MiB data, 559 MiB used, 536 GiB / 536 GiB avail; 2.6 MiB/s rd, 4.9 MiB/s wr, 233 op/s 2024-04-22T23:23:02.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:02 smithi067 ceph-mon[26453]: pgmap v256: 65 pgs: 65 active+clean; 133 MiB data, 619 MiB used, 536 GiB / 536 GiB avail; 3.7 MiB/s rd, 5.8 MiB/s wr, 293 op/s 2024-04-22T23:23:02.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:02 smithi134 ceph-mon[34609]: pgmap v256: 65 pgs: 65 active+clean; 133 MiB data, 619 MiB used, 536 GiB / 536 GiB avail; 3.7 MiB/s rd, 5.8 MiB/s wr, 293 op/s 2024-04-22T23:23:04.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:04 smithi067 ceph-mon[26453]: pgmap v257: 65 pgs: 65 active+clean; 134 MiB data, 626 MiB used, 536 GiB / 536 GiB avail; 2.9 MiB/s rd, 4.1 MiB/s wr, 223 op/s 2024-04-22T23:23:04.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:04 smithi134 ceph-mon[34609]: pgmap v257: 65 pgs: 65 active+clean; 134 MiB data, 626 MiB used, 536 GiB / 536 GiB avail; 2.9 MiB/s rd, 4.1 MiB/s wr, 223 op/s 2024-04-22T23:23:06.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:06 smithi067 ceph-mon[26453]: pgmap v258: 65 pgs: 65 active+clean; 147 MiB data, 666 MiB used, 536 GiB / 536 GiB avail; 3.4 MiB/s rd, 4.9 MiB/s wr, 258 op/s 2024-04-22T23:23:06.555 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:06 smithi134 ceph-mon[34609]: pgmap v258: 65 pgs: 65 active+clean; 147 MiB data, 666 MiB used, 536 GiB / 536 GiB avail; 3.4 MiB/s rd, 4.9 MiB/s wr, 258 op/s 2024-04-22T23:23:07.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:07 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:23:07.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:07 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:23:08.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:08 smithi134 ceph-mon[34609]: pgmap v259: 65 pgs: 65 active+clean; 154 MiB data, 711 MiB used, 536 GiB / 536 GiB avail; 3.0 MiB/s rd, 4.4 MiB/s wr, 310 op/s 2024-04-22T23:23:08.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:08 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:23:08.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:08 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi134.ccxkcj", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T23:23:08.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:08 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T23:23:08.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:08 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:23:08.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:08 smithi067 ceph-mon[26453]: pgmap v259: 65 pgs: 65 active+clean; 154 MiB data, 711 MiB used, 536 GiB / 536 GiB avail; 3.0 MiB/s rd, 4.4 MiB/s wr, 310 op/s 2024-04-22T23:23:08.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:08 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:23:08.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:08 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi134.ccxkcj", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T23:23:08.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:08 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T23:23:08.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:08 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:23:09.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:09 smithi067 ceph-mon[26453]: Upgrade: Updating mgr.smithi134.ccxkcj 2024-04-22T23:23:09.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:09 smithi067 ceph-mon[26453]: Deploying daemon mgr.smithi134.ccxkcj on smithi134 2024-04-22T23:23:09.987 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:09 smithi134 ceph-mon[34609]: Upgrade: Updating mgr.smithi134.ccxkcj 2024-04-22T23:23:09.987 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:09 smithi134 ceph-mon[34609]: Deploying daemon mgr.smithi134.ccxkcj on smithi134 2024-04-22T23:23:10.798 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:10 smithi134 ceph-mon[34609]: pgmap v260: 65 pgs: 65 active+clean; 161 MiB data, 736 MiB used, 536 GiB / 536 GiB avail; 3.1 MiB/s rd, 4.6 MiB/s wr, 322 op/s 2024-04-22T23:23:11.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:10 smithi067 ceph-mon[26453]: pgmap v260: 65 pgs: 65 active+clean; 161 MiB data, 736 MiB used, 536 GiB / 536 GiB avail; 3.1 MiB/s rd, 4.6 MiB/s wr, 322 op/s 2024-04-22T23:23:12.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:12 smithi134 ceph-mon[34609]: pgmap v261: 65 pgs: 65 active+clean; 186 MiB data, 770 MiB used, 536 GiB / 536 GiB avail; 3.4 MiB/s rd, 6.1 MiB/s wr, 455 op/s 2024-04-22T23:23:13.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:12 smithi067 ceph-mon[26453]: pgmap v261: 65 pgs: 65 active+clean; 186 MiB data, 770 MiB used, 536 GiB / 536 GiB avail; 3.4 MiB/s rd, 6.1 MiB/s wr, 455 op/s 2024-04-22T23:23:13.485 INFO:tasks.workunit.client.0.smithi067.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-04-22T23:23:13.493 INFO:tasks.workunit.client.0.smithi067.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-04-22T23:23:13.493 INFO:tasks.workunit.client.0.smithi067.stderr:+ make 2024-04-22T23:23:13.628 INFO:tasks.workunit.client.0.smithi067.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-04-22T23:23:14.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:14 smithi134 ceph-mon[34609]: pgmap v262: 65 pgs: 65 active+clean; 187 MiB data, 783 MiB used, 536 GiB / 536 GiB avail; 2.1 MiB/s rd, 4.6 MiB/s wr, 392 op/s 2024-04-22T23:23:14.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:14 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:23:14.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:14 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:23:14.847 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:14 smithi067 ceph-mon[26453]: pgmap v262: 65 pgs: 65 active+clean; 187 MiB data, 783 MiB used, 536 GiB / 536 GiB avail; 2.1 MiB/s rd, 4.6 MiB/s wr, 392 op/s 2024-04-22T23:23:14.847 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:14 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:23:14.847 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:14 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:23:15.398 INFO:tasks.workunit.client.0.smithi067.stderr:++ readlink -f fsstress 2024-04-22T23:23:15.399 INFO:tasks.workunit.client.0.smithi067.stderr:+ BIN=/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-04-22T23:23:15.400 INFO:tasks.workunit.client.0.smithi067.stderr:+ popd 2024-04-22T23:23:15.400 INFO:tasks.workunit.client.0.smithi067.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-04-22T23:23:15.401 INFO:tasks.workunit.client.0.smithi067.stderr:+ popd 2024-04-22T23:23:15.401 INFO:tasks.workunit.client.0.smithi067.stdout:~/cephtest/mnt.0/client.0/tmp 2024-04-22T23:23:15.401 INFO:tasks.workunit.client.0.smithi067.stderr:++ mktemp -d -p . 2024-04-22T23:23:15.403 INFO:tasks.workunit.client.0.smithi067.stderr:+ T=./tmp.wHgf4sBEUR 2024-04-22T23:23:15.403 INFO:tasks.workunit.client.0.smithi067.stderr:+ /home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.wHgf4sBEUR -l 1 -n 1000 -p 10 -v 2024-04-22T23:23:15.407 INFO:tasks.workunit.client.0.smithi067.stdout:seed = 1713423993 2024-04-22T23:23:15.413 INFO:tasks.workunit.client.0.smithi067.stdout:0/0: creat f0 x:0 0 0 2024-04-22T23:23:15.414 INFO:tasks.workunit.client.0.smithi067.stdout:0/1: mknod c1 0 2024-04-22T23:23:15.415 INFO:tasks.workunit.client.0.smithi067.stdout:1/0: creat f0 x:0 0 0 2024-04-22T23:23:15.416 INFO:tasks.workunit.client.0.smithi067.stdout:3/0: rmdir - no directory 2024-04-22T23:23:15.416 INFO:tasks.workunit.client.0.smithi067.stdout:3/1: write - no filename 2024-04-22T23:23:15.417 INFO:tasks.workunit.client.0.smithi067.stdout:0/2: creat f2 x:0 0 0 2024-04-22T23:23:15.418 INFO:tasks.workunit.client.0.smithi067.stdout:2/0: creat f0 x:0 0 0 2024-04-22T23:23:15.419 INFO:tasks.workunit.client.0.smithi067.stdout:0/3: dread - f0 zero size 2024-04-22T23:23:15.420 INFO:tasks.workunit.client.0.smithi067.stdout:3/2: getdents . 0 2024-04-22T23:23:15.421 INFO:tasks.workunit.client.0.smithi067.stdout:4/0: truncate - no filename 2024-04-22T23:23:15.421 INFO:tasks.workunit.client.0.smithi067.stdout:4/1: write - no filename 2024-04-22T23:23:15.421 INFO:tasks.workunit.client.0.smithi067.stdout:4/2: dread - no filename 2024-04-22T23:23:15.421 INFO:tasks.workunit.client.0.smithi067.stdout:4/3: dwrite - no filename 2024-04-22T23:23:15.421 INFO:tasks.workunit.client.0.smithi067.stdout:4/4: link - no file 2024-04-22T23:23:15.421 INFO:tasks.workunit.client.0.smithi067.stdout:4/5: rmdir - no directory 2024-04-22T23:23:15.421 INFO:tasks.workunit.client.0.smithi067.stdout:4/6: dread - no filename 2024-04-22T23:23:15.421 INFO:tasks.workunit.client.0.smithi067.stdout:4/7: write - no filename 2024-04-22T23:23:15.421 INFO:tasks.workunit.client.0.smithi067.stdout:4/8: dread - no filename 2024-04-22T23:23:15.421 INFO:tasks.workunit.client.0.smithi067.stdout:4/9: dwrite - no filename 2024-04-22T23:23:15.421 INFO:tasks.workunit.client.0.smithi067.stdout:4/10: truncate - no filename 2024-04-22T23:23:15.421 INFO:tasks.workunit.client.0.smithi067.stdout:4/11: fdatasync - no filename 2024-04-22T23:23:15.421 INFO:tasks.workunit.client.0.smithi067.stdout:3/3: chown . 9119290 1 2024-04-22T23:23:15.422 INFO:tasks.workunit.client.0.smithi067.stdout:2/1: write f0 [311643,28782] 0 2024-04-22T23:23:15.422 INFO:tasks.workunit.client.0.smithi067.stdout:4/12: chown . 2161 1 2024-04-22T23:23:15.422 INFO:tasks.workunit.client.0.smithi067.stdout:0/4: mknod c3 0 2024-04-22T23:23:15.422 INFO:tasks.workunit.client.0.smithi067.stdout:0/5: rmdir - no directory 2024-04-22T23:23:15.423 INFO:tasks.workunit.client.0.smithi067.stdout:5/0: stat - no entries 2024-04-22T23:23:15.423 INFO:tasks.workunit.client.0.smithi067.stdout:5/1: fdatasync - no filename 2024-04-22T23:23:15.425 INFO:tasks.workunit.client.0.smithi067.stdout:2/2: mkdir d1 0 2024-04-22T23:23:15.425 INFO:tasks.workunit.client.0.smithi067.stdout:6/0: readlink - no filename 2024-04-22T23:23:15.425 INFO:tasks.workunit.client.0.smithi067.stdout:6/1: truncate - no filename 2024-04-22T23:23:15.427 INFO:tasks.workunit.client.0.smithi067.stdout:3/4: creat f0 x:0 0 0 2024-04-22T23:23:15.429 INFO:tasks.workunit.client.0.smithi067.stdout:1/1: dwrite f0 [0,4194304] 0 2024-04-22T23:23:15.429 INFO:tasks.workunit.client.0.smithi067.stdout:7/0: unlink - no file 2024-04-22T23:23:15.429 INFO:tasks.workunit.client.0.smithi067.stdout:7/1: rename - no filename 2024-04-22T23:23:15.430 INFO:tasks.workunit.client.0.smithi067.stdout:7/2: rename - no filename 2024-04-22T23:23:15.430 INFO:tasks.workunit.client.0.smithi067.stdout:4/13: creat f0 x:0 0 0 2024-04-22T23:23:15.430 INFO:tasks.workunit.client.0.smithi067.stdout:5/2: getdents . 0 2024-04-22T23:23:15.430 INFO:tasks.workunit.client.0.smithi067.stdout:5/3: fsync - no filename 2024-04-22T23:23:15.430 INFO:tasks.workunit.client.0.smithi067.stdout:5/4: fdatasync - no filename 2024-04-22T23:23:15.436 INFO:tasks.workunit.client.0.smithi067.stdout:8/0: fsync - no filename 2024-04-22T23:23:15.436 INFO:tasks.workunit.client.0.smithi067.stdout:8/1: dread - no filename 2024-04-22T23:23:15.437 INFO:tasks.workunit.client.0.smithi067.stdout:1/2: rename f0 to f1 0 2024-04-22T23:23:15.439 INFO:tasks.workunit.client.0.smithi067.stdout:6/2: creat f0 x:0 0 0 2024-04-22T23:23:15.440 INFO:tasks.workunit.client.0.smithi067.stdout:6/3: write f0 [111094,49726] 0 2024-04-22T23:23:15.441 INFO:tasks.workunit.client.0.smithi067.stdout:2/3: creat d1/f2 x:0 0 0 2024-04-22T23:23:15.441 INFO:tasks.workunit.client.0.smithi067.stdout:0/6: link c1 c4 0 2024-04-22T23:23:15.446 INFO:tasks.workunit.client.0.smithi067.stdout:6/4: dread f0 [0,4194304] 0 2024-04-22T23:23:15.448 INFO:tasks.workunit.client.0.smithi067.stdout:7/3: creat f0 x:0 0 0 2024-04-22T23:23:15.448 INFO:tasks.workunit.client.0.smithi067.stdout:9/0: link - no file 2024-04-22T23:23:15.449 INFO:tasks.workunit.client.0.smithi067.stdout:6/5: write f0 [1168456,13659] 0 2024-04-22T23:23:15.451 INFO:tasks.workunit.client.0.smithi067.stdout:7/4: write f0 [637286,4591] 0 2024-04-22T23:23:15.451 INFO:tasks.workunit.client.0.smithi067.stdout:1/3: symlink l2 0 2024-04-22T23:23:15.453 INFO:tasks.workunit.client.0.smithi067.stdout:1/4: stat l2 0 2024-04-22T23:23:15.453 INFO:tasks.workunit.client.0.smithi067.stdout:5/5: creat f0 x:0 0 0 2024-04-22T23:23:15.455 INFO:tasks.workunit.client.0.smithi067.stdout:5/6: dread - f0 zero size 2024-04-22T23:23:15.455 INFO:tasks.workunit.client.0.smithi067.stdout:1/5: chown f1 5 1 2024-04-22T23:23:15.456 INFO:tasks.workunit.client.0.smithi067.stdout:2/4: rename f0 to d1/f3 0 2024-04-22T23:23:15.457 INFO:tasks.workunit.client.0.smithi067.stdout:0/7: dwrite f2 [0,4194304] 0 2024-04-22T23:23:15.458 INFO:tasks.workunit.client.0.smithi067.stdout:6/6: read f0 [530615,21931] 0 2024-04-22T23:23:15.458 INFO:tasks.workunit.client.0.smithi067.stdout:5/7: truncate f0 395642 0 2024-04-22T23:23:15.459 INFO:tasks.workunit.client.0.smithi067.stdout:5/8: read f0 [231164,65339] 0 2024-04-22T23:23:15.460 INFO:tasks.workunit.client.0.smithi067.stdout:8/2: creat f0 x:0 0 0 2024-04-22T23:23:15.460 INFO:tasks.workunit.client.0.smithi067.stdout:5/9: fdatasync f0 0 2024-04-22T23:23:15.462 INFO:tasks.workunit.client.0.smithi067.stdout:5/10: chown f0 0 1 2024-04-22T23:23:15.467 INFO:tasks.workunit.client.0.smithi067.stdout:1/6: mknod c3 0 2024-04-22T23:23:15.468 INFO:tasks.workunit.client.0.smithi067.stdout:0/8: dwrite f0 [0,4194304] 0 2024-04-22T23:23:15.470 INFO:tasks.workunit.client.0.smithi067.stdout:0/9: write f2 [288877,35738] 0 2024-04-22T23:23:15.470 INFO:tasks.workunit.client.0.smithi067.stdout:2/5: creat d1/f4 x:0 0 0 2024-04-22T23:23:15.472 INFO:tasks.workunit.client.0.smithi067.stdout:2/6: write d1/f2 [189468,58691] 0 2024-04-22T23:23:15.474 INFO:tasks.workunit.client.0.smithi067.stdout:6/7: mknod c1 0 2024-04-22T23:23:15.476 INFO:tasks.workunit.client.0.smithi067.stdout:8/3: mkdir d1 0 2024-04-22T23:23:15.477 INFO:tasks.workunit.client.0.smithi067.stdout:8/4: truncate f0 287955 0 2024-04-22T23:23:15.478 INFO:tasks.workunit.client.0.smithi067.stdout:9/1: mkdir d0 0 2024-04-22T23:23:15.478 INFO:tasks.workunit.client.0.smithi067.stdout:9/2: dwrite - no filename 2024-04-22T23:23:15.478 INFO:tasks.workunit.client.0.smithi067.stdout:3/5: getdents . 0 2024-04-22T23:23:15.480 INFO:tasks.workunit.client.0.smithi067.stdout:5/11: symlink l1 0 2024-04-22T23:23:15.482 INFO:tasks.workunit.client.0.smithi067.stdout:1/7: creat f4 x:0 0 0 2024-04-22T23:23:15.483 INFO:tasks.workunit.client.0.smithi067.stdout:1/8: dread - f4 zero size 2024-04-22T23:23:15.485 INFO:tasks.workunit.client.0.smithi067.stdout:0/10: creat f5 x:0 0 0 2024-04-22T23:23:15.487 INFO:tasks.workunit.client.0.smithi067.stdout:0/11: write f5 [939351,57568] 0 2024-04-22T23:23:15.487 INFO:tasks.workunit.client.0.smithi067.stdout:2/7: symlink d1/l5 0 2024-04-22T23:23:15.490 INFO:tasks.workunit.client.0.smithi067.stdout:3/6: dwrite f0 [0,4194304] 0 2024-04-22T23:23:15.496 INFO:tasks.workunit.client.0.smithi067.stdout:1/9: write f1 [930613,12018] 0 2024-04-22T23:23:15.498 INFO:tasks.workunit.client.0.smithi067.stdout:8/5: creat d1/f2 x:0 0 0 2024-04-22T23:23:15.499 INFO:tasks.workunit.client.0.smithi067.stdout:1/10: dread - f4 zero size 2024-04-22T23:23:15.504 INFO:tasks.workunit.client.0.smithi067.stdout:8/6: truncate f0 467323 0 2024-04-22T23:23:15.505 INFO:tasks.workunit.client.0.smithi067.stdout:1/11: write f1 [3551568,85038] 0 2024-04-22T23:23:15.506 INFO:tasks.workunit.client.0.smithi067.stdout:9/3: rmdir d0 0 2024-04-22T23:23:15.506 INFO:tasks.workunit.client.0.smithi067.stdout:9/4: readlink - no filename 2024-04-22T23:23:15.507 INFO:tasks.workunit.client.0.smithi067.stdout:3/7: dread f0 [0,4194304] 0 2024-04-22T23:23:15.507 INFO:tasks.workunit.client.0.smithi067.stdout:0/12: dwrite f0 [0,4194304] 0 2024-04-22T23:23:15.509 INFO:tasks.workunit.client.0.smithi067.stdout:5/12: dwrite f0 [0,4194304] 0 2024-04-22T23:23:15.513 INFO:tasks.workunit.client.0.smithi067.stdout:5/13: write f0 [2854646,22422] 0 2024-04-22T23:23:15.514 INFO:tasks.workunit.client.0.smithi067.stdout:8/7: mkdir d1/d3 0 2024-04-22T23:23:15.515 INFO:tasks.workunit.client.0.smithi067.stdout:8/8: rename d1 to d1/d3/d4 22 2024-04-22T23:23:15.515 INFO:tasks.workunit.client.0.smithi067.stdout:9/5: mknod c1 0 2024-04-22T23:23:15.515 INFO:tasks.workunit.client.0.smithi067.stdout:9/6: dwrite - no filename 2024-04-22T23:23:15.516 INFO:tasks.workunit.client.0.smithi067.stdout:9/7: chown c1 4895 1 2024-04-22T23:23:15.516 INFO:tasks.workunit.client.0.smithi067.stdout:9/8: dread - no filename 2024-04-22T23:23:15.516 INFO:tasks.workunit.client.0.smithi067.stdout:9/9: write - no filename 2024-04-22T23:23:15.516 INFO:tasks.workunit.client.0.smithi067.stdout:9/10: dwrite - no filename 2024-04-22T23:23:15.517 INFO:tasks.workunit.client.0.smithi067.stdout:9/11: dread - no filename 2024-04-22T23:23:15.517 INFO:tasks.workunit.client.0.smithi067.stdout:0/13: mknod c6 0 2024-04-22T23:23:15.518 INFO:tasks.workunit.client.0.smithi067.stdout:0/14: write f2 [805241,78108] 0 2024-04-22T23:23:15.523 INFO:tasks.workunit.client.0.smithi067.stdout:5/14: rename l1 to l2 0 2024-04-22T23:23:15.525 INFO:tasks.workunit.client.0.smithi067.stdout:8/9: creat d1/f5 x:0 0 0 2024-04-22T23:23:15.527 INFO:tasks.workunit.client.0.smithi067.stdout:9/12: creat f2 x:0 0 0 2024-04-22T23:23:15.527 INFO:tasks.workunit.client.0.smithi067.stdout:3/8: dwrite f0 [0,4194304] 0 2024-04-22T23:23:15.529 INFO:tasks.workunit.client.0.smithi067.stdout:3/9: write f0 [1311188,96922] 0 2024-04-22T23:23:15.530 INFO:tasks.workunit.client.0.smithi067.stdout:1/12: link f4 f5 0 2024-04-22T23:23:15.530 INFO:tasks.workunit.client.0.smithi067.stdout:3/10: chown f0 30546949 1 2024-04-22T23:23:15.532 INFO:tasks.workunit.client.0.smithi067.stdout:5/15: dread f0 [0,4194304] 0 2024-04-22T23:23:15.538 INFO:tasks.workunit.client.0.smithi067.stdout:2/8: truncate d1/f3 44249 0 2024-04-22T23:23:15.539 INFO:tasks.workunit.client.0.smithi067.stdout:2/9: dread - d1/f4 zero size 2024-04-22T23:23:15.541 INFO:tasks.workunit.client.0.smithi067.stdout:1/13: symlink l6 0 2024-04-22T23:23:15.542 INFO:tasks.workunit.client.0.smithi067.stdout:1/14: chown f1 1933765 1 2024-04-22T23:23:15.543 INFO:tasks.workunit.client.0.smithi067.stdout:5/16: fsync f0 0 2024-04-22T23:23:15.546 INFO:tasks.workunit.client.0.smithi067.stdout:3/11: unlink f0 0 2024-04-22T23:23:15.546 INFO:tasks.workunit.client.0.smithi067.stdout:3/12: truncate - no filename 2024-04-22T23:23:15.546 INFO:tasks.workunit.client.0.smithi067.stdout:3/13: stat - no entries 2024-04-22T23:23:15.548 INFO:tasks.workunit.client.0.smithi067.stdout:2/10: rename d1/f2 to d1/f6 0 2024-04-22T23:23:15.548 INFO:tasks.workunit.client.0.smithi067.stdout:8/10: fsync d1/f5 0 2024-04-22T23:23:15.550 INFO:tasks.workunit.client.0.smithi067.stdout:8/11: write f0 [1511827,7510] 0 2024-04-22T23:23:15.552 INFO:tasks.workunit.client.0.smithi067.stdout:5/17: rename f0 to f3 0 2024-04-22T23:23:15.553 INFO:tasks.workunit.client.0.smithi067.stdout:3/14: creat f1 x:0 0 0 2024-04-22T23:23:15.554 INFO:tasks.workunit.client.0.smithi067.stdout:1/15: fsync f4 0 2024-04-22T23:23:15.557 INFO:tasks.workunit.client.0.smithi067.stdout:2/11: rename d1/l5 to d1/l7 0 2024-04-22T23:23:15.557 INFO:tasks.workunit.client.0.smithi067.stdout:2/12: dread - d1/f4 zero size 2024-04-22T23:23:15.558 INFO:tasks.workunit.client.0.smithi067.stdout:0/15: truncate f0 329144 0 2024-04-22T23:23:15.559 INFO:tasks.workunit.client.0.smithi067.stdout:2/13: truncate d1/f6 803142 0 2024-04-22T23:23:15.561 INFO:tasks.workunit.client.0.smithi067.stdout:8/12: creat d1/f6 x:0 0 0 2024-04-22T23:23:15.562 INFO:tasks.workunit.client.0.smithi067.stdout:8/13: dread - d1/f5 zero size 2024-04-22T23:23:15.563 INFO:tasks.workunit.client.0.smithi067.stdout:5/18: symlink l4 0 2024-04-22T23:23:15.563 INFO:tasks.workunit.client.0.smithi067.stdout:2/14: dread d1/f6 [0,4194304] 0 2024-04-22T23:23:15.564 INFO:tasks.workunit.client.0.smithi067.stdout:3/15: symlink l2 0 2024-04-22T23:23:15.569 INFO:tasks.workunit.client.0.smithi067.stdout:1/16: write f5 [683121,4270] 0 2024-04-22T23:23:15.573 INFO:tasks.workunit.client.0.smithi067.stdout:8/14: creat d1/f7 x:0 0 0 2024-04-22T23:23:15.574 INFO:tasks.workunit.client.0.smithi067.stdout:5/19: write f3 [1553961,43580] 0 2024-04-22T23:23:15.574 INFO:tasks.workunit.client.0.smithi067.stdout:1/17: dread f4 [0,4194304] 0 2024-04-22T23:23:15.576 INFO:tasks.workunit.client.0.smithi067.stdout:3/16: mkdir d3 0 2024-04-22T23:23:15.578 INFO:tasks.workunit.client.0.smithi067.stdout:0/16: write f0 [1179964,109671] 0 2024-04-22T23:23:15.580 INFO:tasks.workunit.client.0.smithi067.stdout:8/15: symlink d1/l8 0 2024-04-22T23:23:15.582 INFO:tasks.workunit.client.0.smithi067.stdout:8/16: dread - d1/f5 zero size 2024-04-22T23:23:15.583 INFO:tasks.workunit.client.0.smithi067.stdout:5/20: mkdir d5 0 2024-04-22T23:23:15.584 INFO:tasks.workunit.client.0.smithi067.stdout:0/17: dread f2 [0,4194304] 0 2024-04-22T23:23:15.584 INFO:tasks.workunit.client.0.smithi067.stdout:0/18: fdatasync f2 0 2024-04-22T23:23:15.585 INFO:tasks.workunit.client.0.smithi067.stdout:1/18: mknod c7 0 2024-04-22T23:23:15.592 INFO:tasks.workunit.client.0.smithi067.stdout:1/19: mkdir d8 0 2024-04-22T23:23:15.592 INFO:tasks.workunit.client.0.smithi067.stdout:1/20: read f5 [404596,82953] 0 2024-04-22T23:23:15.593 INFO:tasks.workunit.client.0.smithi067.stdout:3/17: mkdir d3/d4 0 2024-04-22T23:23:15.595 INFO:tasks.workunit.client.0.smithi067.stdout:8/17: mkdir d1/d3/d9 0 2024-04-22T23:23:15.596 INFO:tasks.workunit.client.0.smithi067.stdout:8/18: fdatasync f0 0 2024-04-22T23:23:15.597 INFO:tasks.workunit.client.0.smithi067.stdout:5/21: creat d5/f6 x:0 0 0 2024-04-22T23:23:15.600 INFO:tasks.workunit.client.0.smithi067.stdout:3/18: dwrite f1 [0,4194304] 0 2024-04-22T23:23:15.600 INFO:tasks.workunit.client.0.smithi067.stdout:8/19: truncate d1/f2 652909 0 2024-04-22T23:23:15.600 INFO:tasks.workunit.client.0.smithi067.stdout:5/22: write d5/f6 [300933,79086] 0 2024-04-22T23:23:15.601 INFO:tasks.workunit.client.0.smithi067.stdout:2/15: chown d1/f3 0 1 2024-04-22T23:23:15.602 INFO:tasks.workunit.client.0.smithi067.stdout:5/23: write d5/f6 [1093908,65542] 0 2024-04-22T23:23:15.604 INFO:tasks.workunit.client.0.smithi067.stdout:1/21: mkdir d8/d9 0 2024-04-22T23:23:15.606 INFO:tasks.workunit.client.0.smithi067.stdout:5/24: truncate d5/f6 1398573 0 2024-04-22T23:23:15.607 INFO:tasks.workunit.client.0.smithi067.stdout:0/19: getdents . 0 2024-04-22T23:23:15.608 INFO:tasks.workunit.client.0.smithi067.stdout:2/16: creat d1/f8 x:0 0 0 2024-04-22T23:23:15.609 INFO:tasks.workunit.client.0.smithi067.stdout:8/20: symlink d1/d3/la 0 2024-04-22T23:23:15.610 INFO:tasks.workunit.client.0.smithi067.stdout:0/20: chown f5 1059830 1 2024-04-22T23:23:15.610 INFO:tasks.workunit.client.0.smithi067.stdout:2/17: dread - d1/f4 zero size 2024-04-22T23:23:15.610 INFO:tasks.workunit.client.0.smithi067.stdout:1/22: mknod d8/ca 0 2024-04-22T23:23:15.612 INFO:tasks.workunit.client.0.smithi067.stdout:0/21: stat c6 0 2024-04-22T23:23:15.612 INFO:tasks.workunit.client.0.smithi067.stdout:5/25: creat d5/f7 x:0 0 0 2024-04-22T23:23:15.613 INFO:tasks.workunit.client.0.smithi067.stdout:3/19: dwrite f1 [0,4194304] 0 2024-04-22T23:23:15.614 INFO:tasks.workunit.client.0.smithi067.stdout:8/21: unlink d1/f2 0 2024-04-22T23:23:15.615 INFO:tasks.workunit.client.0.smithi067.stdout:8/22: stat d1/f7 0 2024-04-22T23:23:15.617 INFO:tasks.workunit.client.0.smithi067.stdout:1/23: dread f1 [0,4194304] 0 2024-04-22T23:23:15.622 INFO:tasks.workunit.client.0.smithi067.stdout:3/20: mknod d3/c5 0 2024-04-22T23:23:15.624 INFO:tasks.workunit.client.0.smithi067.stdout:8/23: unlink d1/d3/la 0 2024-04-22T23:23:15.626 INFO:tasks.workunit.client.0.smithi067.stdout:2/18: unlink d1/l7 0 2024-04-22T23:23:15.629 INFO:tasks.workunit.client.0.smithi067.stdout:5/26: link f3 d5/f8 0 2024-04-22T23:23:15.631 INFO:tasks.workunit.client.0.smithi067.stdout:3/21: rename f1 to d3/d4/f6 0 2024-04-22T23:23:15.633 INFO:tasks.workunit.client.0.smithi067.stdout:2/19: creat d1/f9 x:0 0 0 2024-04-22T23:23:15.634 INFO:tasks.workunit.client.0.smithi067.stdout:1/24: creat d8/d9/fb x:0 0 0 2024-04-22T23:23:15.636 INFO:tasks.workunit.client.0.smithi067.stdout:1/25: truncate d8/d9/fb 113310 0 2024-04-22T23:23:15.636 INFO:tasks.workunit.client.0.smithi067.stdout:5/27: write f3 [1385687,22197] 0 2024-04-22T23:23:15.636 INFO:tasks.workunit.client.0.smithi067.stdout:1/26: read f4 [203379,26216] 0 2024-04-22T23:23:15.639 INFO:tasks.workunit.client.0.smithi067.stdout:2/20: mkdir d1/da 0 2024-04-22T23:23:15.641 INFO:tasks.workunit.client.0.smithi067.stdout:5/28: symlink d5/l9 0 2024-04-22T23:23:15.641 INFO:tasks.workunit.client.0.smithi067.stdout:1/27: dread f1 [0,4194304] 0 2024-04-22T23:23:15.642 INFO:tasks.workunit.client.0.smithi067.stdout:1/28: chown d8/ca 15666885 1 2024-04-22T23:23:15.642 INFO:tasks.workunit.client.0.smithi067.stdout:3/22: link d3/d4/f6 d3/d4/f7 0 2024-04-22T23:23:15.643 INFO:tasks.workunit.client.0.smithi067.stdout:1/29: dread f4 [0,4194304] 0 2024-04-22T23:23:15.644 INFO:tasks.workunit.client.0.smithi067.stdout:1/30: truncate f4 1224464 0 2024-04-22T23:23:15.644 INFO:tasks.workunit.client.0.smithi067.stdout:8/24: rmdir d1/d3/d9 0 2024-04-22T23:23:15.646 INFO:tasks.workunit.client.0.smithi067.stdout:8/25: write f0 [1182126,49591] 0 2024-04-22T23:23:15.646 INFO:tasks.workunit.client.0.smithi067.stdout:5/29: dread f3 [0,4194304] 0 2024-04-22T23:23:15.647 INFO:tasks.workunit.client.0.smithi067.stdout:5/30: readlink d5/l9 0 2024-04-22T23:23:15.652 INFO:tasks.workunit.client.0.smithi067.stdout:8/26: creat d1/d3/fb x:0 0 0 2024-04-22T23:23:15.653 INFO:tasks.workunit.client.0.smithi067.stdout:5/31: creat d5/fa x:0 0 0 2024-04-22T23:23:15.655 INFO:tasks.workunit.client.0.smithi067.stdout:3/23: dwrite d3/d4/f6 [0,4194304] 0 2024-04-22T23:23:15.658 INFO:tasks.workunit.client.0.smithi067.stdout:1/31: link f4 d8/fc 0 2024-04-22T23:23:15.661 INFO:tasks.workunit.client.0.smithi067.stdout:8/27: creat d1/d3/fc x:0 0 0 2024-04-22T23:23:15.666 INFO:tasks.workunit.client.0.smithi067.stdout:2/21: getdents d1 0 2024-04-22T23:23:15.666 INFO:tasks.workunit.client.0.smithi067.stdout:3/24: write d3/d4/f7 [3430416,110709] 0 2024-04-22T23:23:15.666 INFO:tasks.workunit.client.0.smithi067.stdout:1/32: creat d8/d9/fd x:0 0 0 2024-04-22T23:23:15.666 INFO:tasks.workunit.client.0.smithi067.stdout:2/22: fdatasync d1/f6 0 2024-04-22T23:23:15.669 INFO:tasks.workunit.client.0.smithi067.stdout:2/23: write d1/f4 [994032,106973] 0 2024-04-22T23:23:15.669 INFO:tasks.workunit.client.0.smithi067.stdout:1/33: mkdir d8/de 0 2024-04-22T23:23:15.670 INFO:tasks.workunit.client.0.smithi067.stdout:8/28: dwrite d1/d3/fc [0,4194304] 0 2024-04-22T23:23:15.673 INFO:tasks.workunit.client.0.smithi067.stdout:8/29: mkdir d1/dd 0 2024-04-22T23:23:15.673 INFO:tasks.workunit.client.0.smithi067.stdout:1/34: dread f4 [0,4194304] 0 2024-04-22T23:23:15.673 INFO:tasks.workunit.client.0.smithi067.stdout:2/24: truncate d1/f3 1027270 0 2024-04-22T23:23:15.674 INFO:tasks.workunit.client.0.smithi067.stdout:1/35: stat f5 0 2024-04-22T23:23:15.675 INFO:tasks.workunit.client.0.smithi067.stdout:2/25: write d1/f6 [1171926,74886] 0 2024-04-22T23:23:15.676 INFO:tasks.workunit.client.0.smithi067.stdout:2/26: chown d1/da 38 1 2024-04-22T23:23:15.676 INFO:tasks.workunit.client.0.smithi067.stdout:8/30: mknod d1/d3/ce 0 2024-04-22T23:23:15.678 INFO:tasks.workunit.client.0.smithi067.stdout:2/27: rmdir d1 39 2024-04-22T23:23:15.686 INFO:tasks.workunit.client.0.smithi067.stdout:3/25: truncate d3/d4/f6 2648893 0 2024-04-22T23:23:15.686 INFO:tasks.workunit.client.0.smithi067.stdout:1/36: write f5 [1574139,10620] 0 2024-04-22T23:23:15.687 INFO:tasks.workunit.client.0.smithi067.stdout:8/31: getdents d1 0 2024-04-22T23:23:15.687 INFO:tasks.workunit.client.0.smithi067.stdout:1/37: chown d8 1 1 2024-04-22T23:23:15.689 INFO:tasks.workunit.client.0.smithi067.stdout:8/32: stat d1/d3/fb 0 2024-04-22T23:23:15.690 INFO:tasks.workunit.client.0.smithi067.stdout:2/28: dwrite d1/f4 [0,4194304] 0 2024-04-22T23:23:15.690 INFO:tasks.workunit.client.0.smithi067.stdout:3/26: rename d3/c5 to d3/c8 0 2024-04-22T23:23:15.691 INFO:tasks.workunit.client.0.smithi067.stdout:3/27: readlink l2 0 2024-04-22T23:23:15.693 INFO:tasks.workunit.client.0.smithi067.stdout:2/29: write d1/f6 [969106,99881] 0 2024-04-22T23:23:15.693 INFO:tasks.workunit.client.0.smithi067.stdout:8/33: truncate d1/f6 337038 0 2024-04-22T23:23:15.694 INFO:tasks.workunit.client.0.smithi067.stdout:8/34: write d1/f6 [526912,65827] 0 2024-04-22T23:23:15.696 INFO:tasks.workunit.client.0.smithi067.stdout:8/35: fdatasync d1/f5 0 2024-04-22T23:23:15.701 INFO:tasks.workunit.client.0.smithi067.stdout:8/36: mknod d1/dd/cf 0 2024-04-22T23:23:15.706 INFO:tasks.workunit.client.0.smithi067.stdout:8/37: mkdir d1/d3/d10 0 2024-04-22T23:23:15.706 INFO:tasks.workunit.client.0.smithi067.stdout:2/30: dwrite d1/f4 [0,4194304] 0 2024-04-22T23:23:15.708 INFO:tasks.workunit.client.0.smithi067.stdout:1/38: truncate f5 591759 0 2024-04-22T23:23:15.710 INFO:tasks.workunit.client.0.smithi067.stdout:3/28: dwrite d3/d4/f6 [0,4194304] 0 2024-04-22T23:23:15.711 INFO:tasks.workunit.client.0.smithi067.stdout:3/29: readlink l2 0 2024-04-22T23:23:15.715 INFO:tasks.workunit.client.0.smithi067.stdout:3/30: write d3/d4/f7 [2632119,1403] 0 2024-04-22T23:23:15.717 INFO:tasks.workunit.client.0.smithi067.stdout:8/38: dread d1/d3/fc [0,4194304] 0 2024-04-22T23:23:15.719 INFO:tasks.workunit.client.0.smithi067.stdout:2/31: dread d1/f4 [0,4194304] 0 2024-04-22T23:23:15.719 INFO:tasks.workunit.client.0.smithi067.stdout:2/32: readlink - no filename 2024-04-22T23:23:15.719 INFO:tasks.workunit.client.0.smithi067.stdout:3/31: chown d3/d4 6 1 2024-04-22T23:23:15.721 INFO:tasks.workunit.client.0.smithi067.stdout:8/39: creat d1/d3/f11 x:0 0 0 2024-04-22T23:23:15.723 INFO:tasks.workunit.client.0.smithi067.stdout:2/33: creat d1/fb x:0 0 0 2024-04-22T23:23:15.724 INFO:tasks.workunit.client.0.smithi067.stdout:3/32: mknod d3/d4/c9 0 2024-04-22T23:23:15.726 INFO:tasks.workunit.client.0.smithi067.stdout:2/34: truncate d1/f9 1023517 0 2024-04-22T23:23:15.728 INFO:tasks.workunit.client.0.smithi067.stdout:1/39: dwrite d8/fc [0,4194304] 0 2024-04-22T23:23:15.730 INFO:tasks.workunit.client.0.smithi067.stdout:3/33: creat d3/fa x:0 0 0 2024-04-22T23:23:15.730 INFO:tasks.workunit.client.0.smithi067.stdout:8/40: link d1/d3/ce d1/d3/d10/c12 0 2024-04-22T23:23:15.731 INFO:tasks.workunit.client.0.smithi067.stdout:3/34: write d3/fa [61539,92706] 0 2024-04-22T23:23:15.735 INFO:tasks.workunit.client.0.smithi067.stdout:1/40: dread f1 [0,4194304] 0 2024-04-22T23:23:15.736 INFO:tasks.workunit.client.0.smithi067.stdout:8/41: stat d1/d3/ce 0 2024-04-22T23:23:15.738 INFO:tasks.workunit.client.0.smithi067.stdout:8/42: dread - d1/d3/fb zero size 2024-04-22T23:23:15.738 INFO:tasks.workunit.client.0.smithi067.stdout:1/41: rmdir d8/d9 39 2024-04-22T23:23:15.742 INFO:tasks.workunit.client.0.smithi067.stdout:3/35: dwrite d3/d4/f7 [4194304,4194304] 0 2024-04-22T23:23:15.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:15 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:23:15.746 INFO:tasks.workunit.client.0.smithi067.stdout:1/42: rename d8/de to d8/d9/df 0 2024-04-22T23:23:15.750 INFO:tasks.workunit.client.0.smithi067.stdout:3/36: symlink d3/d4/lb 0 2024-04-22T23:23:15.752 INFO:tasks.workunit.client.0.smithi067.stdout:8/43: dwrite d1/d3/fc [0,4194304] 0 2024-04-22T23:23:15.755 INFO:tasks.workunit.client.0.smithi067.stdout:1/43: getdents d8/d9/df 0 2024-04-22T23:23:15.756 INFO:tasks.workunit.client.0.smithi067.stdout:8/44: creat d1/d3/f13 x:0 0 0 2024-04-22T23:23:15.758 INFO:tasks.workunit.client.0.smithi067.stdout:1/44: fsync f4 0 2024-04-22T23:23:15.758 INFO:tasks.workunit.client.0.smithi067.stdout:1/45: read f1 [3590414,90353] 0 2024-04-22T23:23:15.759 INFO:tasks.workunit.client.0.smithi067.stdout:8/45: mkdir d1/d3/d14 0 2024-04-22T23:23:15.759 INFO:tasks.workunit.client.0.smithi067.stdout:2/35: dwrite d1/f3 [0,4194304] 0 2024-04-22T23:23:15.760 INFO:tasks.workunit.client.0.smithi067.stdout:1/46: write f4 [2591366,49694] 0 2024-04-22T23:23:15.762 INFO:tasks.workunit.client.0.smithi067.stdout:1/47: read f1 [583697,114523] 0 2024-04-22T23:23:15.765 INFO:tasks.workunit.client.0.smithi067.stdout:3/37: rmdir d3 39 2024-04-22T23:23:15.767 INFO:tasks.workunit.client.0.smithi067.stdout:8/46: mkdir d1/d3/d14/d15 0 2024-04-22T23:23:15.767 INFO:tasks.workunit.client.0.smithi067.stdout:2/36: creat d1/fc x:0 0 0 2024-04-22T23:23:15.768 INFO:tasks.workunit.client.0.smithi067.stdout:2/37: write d1/f3 [3122591,22693] 0 2024-04-22T23:23:15.769 INFO:tasks.workunit.client.0.smithi067.stdout:2/38: fdatasync d1/f6 0 2024-04-22T23:23:15.771 INFO:tasks.workunit.client.0.smithi067.stdout:8/47: unlink d1/d3/fc 0 2024-04-22T23:23:15.772 INFO:tasks.workunit.client.0.smithi067.stdout:8/48: fsync d1/f5 0 2024-04-22T23:23:15.773 INFO:tasks.workunit.client.0.smithi067.stdout:3/38: creat d3/d4/fc x:0 0 0 2024-04-22T23:23:15.776 INFO:tasks.workunit.client.0.smithi067.stdout:2/39: symlink d1/da/ld 0 2024-04-22T23:23:15.783 INFO:tasks.workunit.client.0.smithi067.stdout:8/49: dwrite d1/d3/f11 [0,4194304] 0 2024-04-22T23:23:15.787 INFO:tasks.workunit.client.0.smithi067.stdout:3/39: dread d3/d4/f7 [0,4194304] 0 2024-04-22T23:23:15.789 INFO:tasks.workunit.client.0.smithi067.stdout:8/50: mkdir d1/d16 0 2024-04-22T23:23:15.790 INFO:tasks.workunit.client.0.smithi067.stdout:8/51: readlink d1/l8 0 2024-04-22T23:23:15.791 INFO:tasks.workunit.client.0.smithi067.stdout:3/40: dread d3/fa [0,4194304] 0 2024-04-22T23:23:15.791 INFO:tasks.workunit.client.0.smithi067.stdout:2/40: dwrite d1/f8 [0,4194304] 0 2024-04-22T23:23:15.792 INFO:tasks.workunit.client.0.smithi067.stdout:8/52: rename d1/dd to d1/dd/d17 22 2024-04-22T23:23:15.799 INFO:tasks.workunit.client.0.smithi067.stdout:3/41: dread d3/d4/f6 [4194304,4194304] 0 2024-04-22T23:23:15.799 INFO:tasks.workunit.client.0.smithi067.stdout:3/42: stat l2 0 2024-04-22T23:23:15.801 INFO:tasks.workunit.client.0.smithi067.stdout:8/53: dwrite f0 [0,4194304] 0 2024-04-22T23:23:15.801 INFO:tasks.workunit.client.0.smithi067.stdout:3/43: chown d3 961160 1 2024-04-22T23:23:15.803 INFO:tasks.workunit.client.0.smithi067.stdout:3/44: symlink d3/d4/ld 0 2024-04-22T23:23:15.804 INFO:tasks.workunit.client.0.smithi067.stdout:3/45: dread - d3/d4/fc zero size 2024-04-22T23:23:15.805 INFO:tasks.workunit.client.0.smithi067.stdout:3/46: read d3/fa [92906,117180] 0 2024-04-22T23:23:15.805 INFO:tasks.workunit.client.0.smithi067.stdout:8/54: creat d1/f18 x:0 0 0 2024-04-22T23:23:15.808 INFO:tasks.workunit.client.0.smithi067.stdout:8/55: symlink d1/dd/l19 0 2024-04-22T23:23:15.809 INFO:tasks.workunit.client.0.smithi067.stdout:3/47: creat d3/d4/fe x:0 0 0 2024-04-22T23:23:15.810 INFO:tasks.workunit.client.0.smithi067.stdout:8/56: write d1/d3/f13 [323990,22315] 0 2024-04-22T23:23:15.812 INFO:tasks.workunit.client.0.smithi067.stdout:3/48: dread d3/fa [0,4194304] 0 2024-04-22T23:23:15.814 INFO:tasks.workunit.client.0.smithi067.stdout:3/49: write d3/d4/fe [333876,89835] 0 2024-04-22T23:23:15.815 INFO:tasks.workunit.client.0.smithi067.stdout:3/50: chown d3/d4 23124 1 2024-04-22T23:23:15.818 INFO:tasks.workunit.client.0.smithi067.stdout:8/57: dwrite d1/f6 [0,4194304] 0 2024-04-22T23:23:15.820 INFO:tasks.workunit.client.0.smithi067.stdout:8/58: readlink d1/dd/l19 0 2024-04-22T23:23:15.821 INFO:tasks.workunit.client.0.smithi067.stdout:8/59: chown d1/dd/l19 34854759 1 2024-04-22T23:23:15.821 INFO:tasks.workunit.client.0.smithi067.stdout:3/51: dread d3/d4/f6 [0,4194304] 0 2024-04-22T23:23:15.824 INFO:tasks.workunit.client.0.smithi067.stdout:8/60: symlink d1/dd/l1a 0 2024-04-22T23:23:15.826 INFO:tasks.workunit.client.0.smithi067.stdout:3/52: dread d3/d4/f6 [0,4194304] 0 2024-04-22T23:23:15.826 INFO:tasks.workunit.client.0.smithi067.stdout:8/61: rename d1/f5 to d1/d3/f1b 0 2024-04-22T23:23:15.836 INFO:tasks.workunit.client.0.smithi067.stdout:3/53: dwrite d3/fa [0,4194304] 0 2024-04-22T23:23:15.837 INFO:tasks.workunit.client.0.smithi067.stdout:3/54: chown d3/d4/fe 83131455 1 2024-04-22T23:23:15.841 INFO:tasks.workunit.client.0.smithi067.stdout:3/55: unlink d3/d4/f6 0 2024-04-22T23:23:15.842 INFO:tasks.workunit.client.0.smithi067.stdout:8/62: truncate d1/f6 1038314 0 2024-04-22T23:23:15.843 INFO:tasks.workunit.client.0.smithi067.stdout:8/63: read d1/d3/f11 [838177,122228] 0 2024-04-22T23:23:15.843 INFO:tasks.workunit.client.0.smithi067.stdout:3/56: symlink d3/lf 0 2024-04-22T23:23:15.846 INFO:tasks.workunit.client.0.smithi067.stdout:3/57: mkdir d3/d4/d10 0 2024-04-22T23:23:15.847 INFO:tasks.workunit.client.0.smithi067.stdout:8/64: symlink d1/d3/d10/l1c 0 2024-04-22T23:23:15.848 INFO:tasks.workunit.client.0.smithi067.stdout:8/65: chown d1/d3/d10/l1c 1638 1 2024-04-22T23:23:15.849 INFO:tasks.workunit.client.0.smithi067.stdout:3/58: getdents d3/d4/d10 0 2024-04-22T23:23:15.855 INFO:tasks.workunit.client.0.smithi067.stdout:8/66: rename d1/dd/l1a to d1/d3/l1d 0 2024-04-22T23:23:15.855 INFO:tasks.workunit.client.0.smithi067.stdout:8/67: chown d1/f6 1044180722 1 2024-04-22T23:23:15.863 INFO:tasks.workunit.client.0.smithi067.stdout:8/68: dwrite d1/d3/fb [0,4194304] 0 2024-04-22T23:23:15.875 INFO:tasks.workunit.client.0.smithi067.stdout:8/69: dwrite d1/d3/f13 [0,4194304] 0 2024-04-22T23:23:15.875 INFO:tasks.workunit.client.0.smithi067.stdout:3/59: truncate d3/d4/f7 2534665 0 2024-04-22T23:23:15.875 INFO:tasks.workunit.client.0.smithi067.stdout:8/70: chown d1 15 1 2024-04-22T23:23:15.881 INFO:tasks.workunit.client.0.smithi067.stdout:8/71: dread f0 [0,4194304] 0 2024-04-22T23:23:15.884 INFO:tasks.workunit.client.0.smithi067.stdout:8/72: creat d1/d3/f1e x:0 0 0 2024-04-22T23:23:15.884 INFO:tasks.workunit.client.0.smithi067.stdout:3/60: dread d3/d4/fe [0,4194304] 0 2024-04-22T23:23:15.891 INFO:tasks.workunit.client.0.smithi067.stdout:3/61: rename d3/d4/ld to d3/d4/l11 0 2024-04-22T23:23:15.892 INFO:tasks.workunit.client.0.smithi067.stdout:8/73: dwrite d1/f6 [0,4194304] 0 2024-04-22T23:23:15.893 INFO:tasks.workunit.client.0.smithi067.stdout:3/62: dread d3/d4/fe [0,4194304] 0 2024-04-22T23:23:15.896 INFO:tasks.workunit.client.0.smithi067.stdout:3/63: mkdir d3/d4/d12 0 2024-04-22T23:23:15.897 INFO:tasks.workunit.client.0.smithi067.stdout:3/64: symlink d3/d4/d12/l13 0 2024-04-22T23:23:15.912 INFO:tasks.workunit.client.0.smithi067.stdout:3/65: dwrite d3/d4/f7 [0,4194304] 0 2024-04-22T23:23:15.917 INFO:tasks.workunit.client.0.smithi067.stdout:3/66: dread d3/d4/f7 [0,4194304] 0 2024-04-22T23:23:15.922 INFO:tasks.workunit.client.0.smithi067.stdout:3/67: link d3/d4/d12/l13 d3/d4/d10/l14 0 2024-04-22T23:23:15.925 INFO:tasks.workunit.client.0.smithi067.stdout:3/68: rename l2 to d3/d4/l15 0 2024-04-22T23:23:15.927 INFO:tasks.workunit.client.0.smithi067.stdout:3/69: truncate d3/d4/f7 510250 0 2024-04-22T23:23:15.928 INFO:tasks.workunit.client.0.smithi067.stdout:3/70: rmdir d3 39 2024-04-22T23:23:15.935 INFO:tasks.workunit.client.0.smithi067.stdout:3/71: dwrite d3/d4/fe [0,4194304] 0 2024-04-22T23:23:15.942 INFO:tasks.workunit.client.0.smithi067.stdout:3/72: dwrite d3/d4/fe [0,4194304] 0 2024-04-22T23:23:15.945 INFO:tasks.workunit.client.0.smithi067.stdout:3/73: creat d3/d4/f16 x:0 0 0 2024-04-22T23:23:15.946 INFO:tasks.workunit.client.0.smithi067.stdout:3/74: read d3/d4/fe [2193876,106309] 0 2024-04-22T23:23:15.957 INFO:tasks.workunit.client.0.smithi067.stdout:3/75: getdents d3/d4 0 2024-04-22T23:23:15.960 INFO:tasks.workunit.client.0.smithi067.stdout:3/76: dread d3/d4/fe [0,4194304] 0 2024-04-22T23:23:15.962 INFO:tasks.workunit.client.0.smithi067.stdout:3/77: dread d3/d4/f7 [0,4194304] 0 2024-04-22T23:23:15.963 INFO:tasks.workunit.client.0.smithi067.stdout:3/78: mkdir d3/d4/d17 0 2024-04-22T23:23:15.964 INFO:tasks.workunit.client.0.smithi067.stdout:3/79: dread - d3/d4/fc zero size 2024-04-22T23:23:15.965 INFO:tasks.workunit.client.0.smithi067.stdout:3/80: write d3/fa [132444,57919] 0 2024-04-22T23:23:15.968 INFO:tasks.workunit.client.0.smithi067.stdout:3/81: creat d3/d4/d12/f18 x:0 0 0 2024-04-22T23:23:15.970 INFO:tasks.workunit.client.0.smithi067.stdout:3/82: getdents d3/d4/d17 0 2024-04-22T23:23:15.971 INFO:tasks.workunit.client.0.smithi067.stdout:3/83: fdatasync d3/d4/f16 0 2024-04-22T23:23:15.971 INFO:tasks.workunit.client.0.smithi067.stdout:3/84: write d3/d4/d12/f18 [402335,94541] 0 2024-04-22T23:23:15.974 INFO:tasks.workunit.client.0.smithi067.stdout:3/85: mkdir d3/d4/d10/d19 0 2024-04-22T23:23:15.976 INFO:tasks.workunit.client.0.smithi067.stdout:3/86: creat d3/d4/d10/f1a x:0 0 0 2024-04-22T23:23:15.977 INFO:tasks.workunit.client.0.smithi067.stdout:3/87: write d3/d4/d12/f18 [1008477,113656] 0 2024-04-22T23:23:15.984 INFO:tasks.workunit.client.0.smithi067.stdout:3/88: dwrite d3/d4/fe [0,4194304] 0 2024-04-22T23:23:15.990 INFO:tasks.workunit.client.0.smithi067.stdout:3/89: dwrite d3/d4/fe [4194304,4194304] 0 2024-04-22T23:23:15.991 INFO:tasks.workunit.client.0.smithi067.stdout:3/90: chown d3/d4/f16 278217 1 2024-04-22T23:23:15.997 INFO:tasks.workunit.client.0.smithi067.stdout:3/91: unlink d3/d4/lb 0 2024-04-22T23:23:15.998 INFO:tasks.workunit.client.0.smithi067.stdout:3/92: dread - d3/d4/fc zero size 2024-04-22T23:23:16.001 INFO:tasks.workunit.client.0.smithi067.stdout:3/93: mkdir d3/d4/d1b 0 2024-04-22T23:23:16.001 INFO:tasks.workunit.client.0.smithi067.stdout:3/94: dread - d3/d4/fc zero size 2024-04-22T23:23:16.003 INFO:tasks.workunit.client.0.smithi067.stdout:3/95: mknod d3/d4/d10/d19/c1c 0 2024-04-22T23:23:16.012 INFO:tasks.workunit.client.0.smithi067.stdout:3/96: rmdir d3/d4 39 2024-04-22T23:23:16.014 INFO:tasks.workunit.client.0.smithi067.stdout:3/97: getdents d3/d4/d17 0 2024-04-22T23:23:16.017 INFO:tasks.workunit.client.0.smithi067.stdout:3/98: creat d3/d4/d12/f1d x:0 0 0 2024-04-22T23:23:16.019 INFO:tasks.workunit.client.0.smithi067.stdout:3/99: write d3/d4/f7 [1244723,114408] 0 2024-04-22T23:23:16.021 INFO:tasks.workunit.client.0.smithi067.stdout:3/100: truncate d3/d4/d12/f18 1592312 0 2024-04-22T23:23:16.021 INFO:tasks.workunit.client.0.smithi067.stdout:3/101: read d3/d4/fe [5663426,52490] 0 2024-04-22T23:23:16.022 INFO:tasks.workunit.client.0.smithi067.stdout:3/102: dread - d3/d4/f16 zero size 2024-04-22T23:23:16.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:15 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:23:16.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:16 smithi134 ceph-mon[34609]: pgmap v263: 65 pgs: 65 active+clean; 197 MiB data, 810 MiB used, 536 GiB / 536 GiB avail; 2.8 MiB/s rd, 5.4 MiB/s wr, 444 op/s 2024-04-22T23:23:17.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:16 smithi067 ceph-mon[26453]: pgmap v263: 65 pgs: 65 active+clean; 197 MiB data, 810 MiB used, 536 GiB / 536 GiB avail; 2.8 MiB/s rd, 5.4 MiB/s wr, 444 op/s 2024-04-22T23:23:17.594 INFO:tasks.workunit.client.0.smithi067.stdout:4/14: sync 2024-04-22T23:23:17.599 INFO:tasks.workunit.client.0.smithi067.stdout:4/15: dwrite f0 [0,4194304] 0 2024-04-22T23:23:17.600 INFO:tasks.workunit.client.0.smithi067.stdout:4/16: chown f0 204876117 1 2024-04-22T23:23:17.601 INFO:tasks.workunit.client.0.smithi067.stdout:4/17: write f0 [547350,3248] 0 2024-04-22T23:23:17.603 INFO:tasks.workunit.client.0.smithi067.stdout:4/18: symlink l1 0 2024-04-22T23:23:17.605 INFO:tasks.workunit.client.0.smithi067.stdout:4/19: creat f2 x:0 0 0 2024-04-22T23:23:17.605 INFO:tasks.workunit.client.0.smithi067.stdout:4/20: read - f2 zero size 2024-04-22T23:23:17.607 INFO:tasks.workunit.client.0.smithi067.stdout:4/21: creat f3 x:0 0 0 2024-04-22T23:23:17.609 INFO:tasks.workunit.client.0.smithi067.stdout:4/22: link f2 f4 0 2024-04-22T23:23:17.618 INFO:tasks.workunit.client.0.smithi067.stdout:4/23: dwrite f2 [0,4194304] 0 2024-04-22T23:23:17.621 INFO:tasks.workunit.client.0.smithi067.stdout:4/24: creat f5 x:0 0 0 2024-04-22T23:23:17.629 INFO:tasks.workunit.client.0.smithi067.stdout:4/25: dwrite f0 [0,4194304] 0 2024-04-22T23:23:17.629 INFO:tasks.workunit.client.0.smithi067.stdout:4/26: readlink l1 0 2024-04-22T23:23:17.632 INFO:tasks.workunit.client.0.smithi067.stdout:4/27: rename f2 to f6 0 2024-04-22T23:23:17.636 INFO:tasks.workunit.client.0.smithi067.stdout:4/28: unlink f0 0 2024-04-22T23:23:17.636 INFO:tasks.workunit.client.0.smithi067.stdout:4/29: read - f5 zero size 2024-04-22T23:23:17.637 INFO:tasks.workunit.client.0.smithi067.stdout:4/30: read f4 [2637921,126707] 0 2024-04-22T23:23:17.638 INFO:tasks.workunit.client.0.smithi067.stdout:4/31: write f3 [1015401,54412] 0 2024-04-22T23:23:17.710 INFO:tasks.workunit.client.0.smithi067.stdout:0/22: sync 2024-04-22T23:23:17.711 INFO:tasks.workunit.client.0.smithi067.stdout:1/48: sync 2024-04-22T23:23:17.711 INFO:tasks.workunit.client.0.smithi067.stdout:7/5: sync 2024-04-22T23:23:17.711 INFO:tasks.workunit.client.0.smithi067.stdout:5/32: sync 2024-04-22T23:23:17.711 INFO:tasks.workunit.client.0.smithi067.stdout:6/8: sync 2024-04-22T23:23:17.711 INFO:tasks.workunit.client.0.smithi067.stdout:8/74: sync 2024-04-22T23:23:17.711 INFO:tasks.workunit.client.0.smithi067.stdout:3/103: sync 2024-04-22T23:23:17.711 INFO:tasks.workunit.client.0.smithi067.stdout:9/13: sync 2024-04-22T23:23:17.711 INFO:tasks.workunit.client.0.smithi067.stdout:2/41: sync 2024-04-22T23:23:17.713 INFO:tasks.workunit.client.0.smithi067.stdout:0/23: mknod c7 0 2024-04-22T23:23:17.721 INFO:tasks.workunit.client.0.smithi067.stdout:1/49: read f5 [1626117,99834] 0 2024-04-22T23:23:17.721 INFO:tasks.workunit.client.0.smithi067.stdout:7/6: chown f0 223067 1 2024-04-22T23:23:17.722 INFO:tasks.workunit.client.0.smithi067.stdout:3/104: rename d3 to d3/d4/d12/d1e 22 2024-04-22T23:23:17.722 INFO:tasks.workunit.client.0.smithi067.stdout:5/33: write d5/fa [234839,55289] 0 2024-04-22T23:23:17.722 INFO:tasks.workunit.client.0.smithi067.stdout:0/24: write f0 [642008,53196] 0 2024-04-22T23:23:17.724 INFO:tasks.workunit.client.0.smithi067.stdout:7/7: truncate f0 903648 0 2024-04-22T23:23:17.724 INFO:tasks.workunit.client.0.smithi067.stdout:2/42: unlink d1/da/ld 0 2024-04-22T23:23:17.726 INFO:tasks.workunit.client.0.smithi067.stdout:9/14: rename c1 to c3 0 2024-04-22T23:23:17.726 INFO:tasks.workunit.client.0.smithi067.stdout:0/25: chown c7 314484649 1 2024-04-22T23:23:17.727 INFO:tasks.workunit.client.0.smithi067.stdout:6/9: dread f0 [0,4194304] 0 2024-04-22T23:23:17.727 INFO:tasks.workunit.client.0.smithi067.stdout:7/8: chown f0 101447068 1 2024-04-22T23:23:17.727 INFO:tasks.workunit.client.0.smithi067.stdout:3/105: read - d3/d4/f16 zero size 2024-04-22T23:23:17.731 INFO:tasks.workunit.client.0.smithi067.stdout:5/34: rmdir d5 39 2024-04-22T23:23:17.734 INFO:tasks.workunit.client.0.smithi067.stdout:0/26: chown c4 101 1 2024-04-22T23:23:17.734 INFO:tasks.workunit.client.0.smithi067.stdout:8/75: dread f0 [0,4194304] 0 2024-04-22T23:23:17.734 INFO:tasks.workunit.client.0.smithi067.stdout:7/9: rename f0 to f1 0 2024-04-22T23:23:17.735 INFO:tasks.workunit.client.0.smithi067.stdout:7/10: stat f1 0 2024-04-22T23:23:17.736 INFO:tasks.workunit.client.0.smithi067.stdout:3/106: mkdir d3/d4/d10/d1f 0 2024-04-22T23:23:17.738 INFO:tasks.workunit.client.0.smithi067.stdout:2/43: dwrite d1/f3 [0,4194304] 0 2024-04-22T23:23:17.742 INFO:tasks.workunit.client.0.smithi067.stdout:1/50: dwrite d8/fc [0,4194304] 0 2024-04-22T23:23:17.744 INFO:tasks.workunit.client.0.smithi067.stdout:3/107: write d3/fa [859218,3976] 0 2024-04-22T23:23:17.745 INFO:tasks.workunit.client.0.smithi067.stdout:0/27: dread f0 [0,4194304] 0 2024-04-22T23:23:17.745 INFO:tasks.workunit.client.0.smithi067.stdout:2/44: write d1/fb [66224,64317] 0 2024-04-22T23:23:17.748 INFO:tasks.workunit.client.0.smithi067.stdout:5/35: rmdir d5 39 2024-04-22T23:23:17.748 INFO:tasks.workunit.client.0.smithi067.stdout:3/108: dread - d3/d4/f16 zero size 2024-04-22T23:23:17.750 INFO:tasks.workunit.client.0.smithi067.stdout:7/11: dread f1 [0,4194304] 0 2024-04-22T23:23:17.750 INFO:tasks.workunit.client.0.smithi067.stdout:8/76: dread d1/d3/f13 [0,4194304] 0 2024-04-22T23:23:17.751 INFO:tasks.workunit.client.0.smithi067.stdout:7/12: chown f1 15045308 1 2024-04-22T23:23:17.751 INFO:tasks.workunit.client.0.smithi067.stdout:8/77: read - d1/d3/f1e zero size 2024-04-22T23:23:17.752 INFO:tasks.workunit.client.0.smithi067.stdout:8/78: write d1/f6 [2103845,115620] 0 2024-04-22T23:23:17.754 INFO:tasks.workunit.client.0.smithi067.stdout:2/45: write d1/f4 [5188674,47338] 0 2024-04-22T23:23:17.754 INFO:tasks.workunit.client.0.smithi067.stdout:2/46: stat d1/fc 0 2024-04-22T23:23:17.756 INFO:tasks.workunit.client.0.smithi067.stdout:2/47: write d1/f6 [810577,13867] 0 2024-04-22T23:23:17.756 INFO:tasks.workunit.client.0.smithi067.stdout:0/28: unlink c7 0 2024-04-22T23:23:17.759 INFO:tasks.workunit.client.0.smithi067.stdout:0/29: dread f0 [0,4194304] 0 2024-04-22T23:23:17.760 INFO:tasks.workunit.client.0.smithi067.stdout:0/30: write f5 [1518691,87840] 0 2024-04-22T23:23:17.765 INFO:tasks.workunit.client.0.smithi067.stdout:2/48: symlink d1/le 0 2024-04-22T23:23:17.767 INFO:tasks.workunit.client.0.smithi067.stdout:5/36: mkdir d5/db 0 2024-04-22T23:23:17.769 INFO:tasks.workunit.client.0.smithi067.stdout:1/51: fsync f5 0 2024-04-22T23:23:17.771 INFO:tasks.workunit.client.0.smithi067.stdout:0/31: fdatasync f0 0 2024-04-22T23:23:17.777 INFO:tasks.workunit.client.0.smithi067.stdout:3/109: creat d3/d4/f20 x:0 0 0 2024-04-22T23:23:17.777 INFO:tasks.workunit.client.0.smithi067.stdout:5/37: dwrite d5/f7 [0,4194304] 0 2024-04-22T23:23:17.781 INFO:tasks.workunit.client.0.smithi067.stdout:0/32: creat f8 x:0 0 0 2024-04-22T23:23:17.781 INFO:tasks.workunit.client.0.smithi067.stdout:0/33: readlink - no filename 2024-04-22T23:23:17.782 INFO:tasks.workunit.client.0.smithi067.stdout:8/79: rmdir d1/d16 0 2024-04-22T23:23:17.783 INFO:tasks.workunit.client.0.smithi067.stdout:1/52: dwrite d8/d9/fb [0,4194304] 0 2024-04-22T23:23:17.783 INFO:tasks.workunit.client.0.smithi067.stdout:1/53: readlink l6 0 2024-04-22T23:23:17.786 INFO:tasks.workunit.client.0.smithi067.stdout:3/110: read d3/d4/fe [552235,90981] 0 2024-04-22T23:23:17.787 INFO:tasks.workunit.client.0.smithi067.stdout:0/34: creat f9 x:0 0 0 2024-04-22T23:23:17.789 INFO:tasks.workunit.client.0.smithi067.stdout:1/54: symlink d8/d9/df/l10 0 2024-04-22T23:23:17.790 INFO:tasks.workunit.client.0.smithi067.stdout:5/38: dwrite d5/f6 [0,4194304] 0 2024-04-22T23:23:17.791 INFO:tasks.workunit.client.0.smithi067.stdout:5/39: rename d5 to d5/dc 22 2024-04-22T23:23:17.792 INFO:tasks.workunit.client.0.smithi067.stdout:3/111: creat d3/d4/d10/d19/f21 x:0 0 0 2024-04-22T23:23:17.801 INFO:tasks.workunit.client.0.smithi067.stdout:3/112: unlink d3/d4/f7 0 2024-04-22T23:23:17.804 INFO:tasks.workunit.client.0.smithi067.stdout:0/35: dwrite f8 [0,4194304] 0 2024-04-22T23:23:17.809 INFO:tasks.workunit.client.0.smithi067.stdout:1/55: dwrite d8/fc [0,4194304] 0 2024-04-22T23:23:17.811 INFO:tasks.workunit.client.0.smithi067.stdout:5/40: dwrite d5/fa [0,4194304] 0 2024-04-22T23:23:17.811 INFO:tasks.workunit.client.0.smithi067.stdout:3/113: unlink d3/d4/d12/f18 0 2024-04-22T23:23:17.813 INFO:tasks.workunit.client.0.smithi067.stdout:0/36: symlink la 0 2024-04-22T23:23:17.819 INFO:tasks.workunit.client.0.smithi067.stdout:0/37: mkdir db 0 2024-04-22T23:23:17.821 INFO:tasks.workunit.client.0.smithi067.stdout:5/41: creat d5/db/fd x:0 0 0 2024-04-22T23:23:17.822 INFO:tasks.workunit.client.0.smithi067.stdout:3/114: dwrite d3/d4/f16 [0,4194304] 0 2024-04-22T23:23:17.824 INFO:tasks.workunit.client.0.smithi067.stdout:1/56: creat d8/f11 x:0 0 0 2024-04-22T23:23:17.826 INFO:tasks.workunit.client.0.smithi067.stdout:5/42: mknod d5/db/ce 0 2024-04-22T23:23:17.826 INFO:tasks.workunit.client.0.smithi067.stdout:3/115: dread d3/d4/fe [4194304,4194304] 0 2024-04-22T23:23:17.828 INFO:tasks.workunit.client.0.smithi067.stdout:0/38: getdents db 0 2024-04-22T23:23:17.829 INFO:tasks.workunit.client.0.smithi067.stdout:5/43: creat d5/db/ff x:0 0 0 2024-04-22T23:23:17.831 INFO:tasks.workunit.client.0.smithi067.stdout:1/57: mkdir d8/d12 0 2024-04-22T23:23:17.833 INFO:tasks.workunit.client.0.smithi067.stdout:3/116: chown d3/d4/d12/l13 503 1 2024-04-22T23:23:17.835 INFO:tasks.workunit.client.0.smithi067.stdout:5/44: creat d5/db/f10 x:0 0 0 2024-04-22T23:23:17.837 INFO:tasks.workunit.client.0.smithi067.stdout:1/58: creat d8/d9/f13 x:0 0 0 2024-04-22T23:23:17.837 INFO:tasks.workunit.client.0.smithi067.stdout:0/39: dwrite f5 [0,4194304] 0 2024-04-22T23:23:17.837 INFO:tasks.workunit.client.0.smithi067.stdout:3/117: truncate d3/d4/d10/f1a 332211 0 2024-04-22T23:23:17.838 INFO:tasks.workunit.client.0.smithi067.stdout:1/59: chown d8/fc 582 1 2024-04-22T23:23:17.844 INFO:tasks.workunit.client.0.smithi067.stdout:0/40: dread f5 [0,4194304] 0 2024-04-22T23:23:17.845 INFO:tasks.workunit.client.0.smithi067.stdout:1/60: symlink d8/d9/df/l14 0 2024-04-22T23:23:17.847 INFO:tasks.workunit.client.0.smithi067.stdout:3/118: mknod d3/d4/d10/d1f/c22 0 2024-04-22T23:23:17.849 INFO:tasks.workunit.client.0.smithi067.stdout:1/61: rmdir d8/d9/df 39 2024-04-22T23:23:17.851 INFO:tasks.workunit.client.0.smithi067.stdout:0/41: getdents db 0 2024-04-22T23:23:17.851 INFO:tasks.workunit.client.0.smithi067.stdout:1/62: stat d8/d9/fd 0 2024-04-22T23:23:17.852 INFO:tasks.workunit.client.0.smithi067.stdout:1/63: chown d8/d9 2116 1 2024-04-22T23:23:17.852 INFO:tasks.workunit.client.0.smithi067.stdout:0/42: chown c3 0 1 2024-04-22T23:23:17.854 INFO:tasks.workunit.client.0.smithi067.stdout:0/43: read - f9 zero size 2024-04-22T23:23:17.854 INFO:tasks.workunit.client.0.smithi067.stdout:1/64: dread - d8/d9/f13 zero size 2024-04-22T23:23:17.855 INFO:tasks.workunit.client.0.smithi067.stdout:0/44: rename db to db/dc 22 2024-04-22T23:23:17.856 INFO:tasks.workunit.client.0.smithi067.stdout:0/45: stat f8 0 2024-04-22T23:23:17.856 INFO:tasks.workunit.client.0.smithi067.stdout:3/119: dwrite d3/d4/f20 [0,4194304] 0 2024-04-22T23:23:17.862 INFO:tasks.workunit.client.0.smithi067.stdout:3/120: mknod d3/d4/d10/c23 0 2024-04-22T23:23:17.867 INFO:tasks.workunit.client.0.smithi067.stdout:1/65: dwrite f5 [4194304,4194304] 0 2024-04-22T23:23:17.868 INFO:tasks.workunit.client.0.smithi067.stdout:3/121: creat d3/d4/f24 x:0 0 0 2024-04-22T23:23:17.868 INFO:tasks.workunit.client.0.smithi067.stdout:0/46: dread f2 [0,4194304] 0 2024-04-22T23:23:17.869 INFO:tasks.workunit.client.0.smithi067.stdout:1/66: write d8/d9/fb [2713473,46305] 0 2024-04-22T23:23:17.870 INFO:tasks.workunit.client.0.smithi067.stdout:1/67: dread - d8/d9/f13 zero size 2024-04-22T23:23:17.874 INFO:tasks.workunit.client.0.smithi067.stdout:0/47: link f2 db/fd 0 2024-04-22T23:23:17.876 INFO:tasks.workunit.client.0.smithi067.stdout:3/122: creat d3/d4/d17/f25 x:0 0 0 2024-04-22T23:23:17.878 INFO:tasks.workunit.client.0.smithi067.stdout:1/68: creat d8/f15 x:0 0 0 2024-04-22T23:23:17.880 INFO:tasks.workunit.client.0.smithi067.stdout:0/48: mkdir db/de 0 2024-04-22T23:23:17.881 INFO:tasks.workunit.client.0.smithi067.stdout:3/123: symlink d3/d4/d1b/l26 0 2024-04-22T23:23:17.883 INFO:tasks.workunit.client.0.smithi067.stdout:3/124: chown d3/d4/d10/d19/f21 2842 1 2024-04-22T23:23:17.883 INFO:tasks.workunit.client.0.smithi067.stdout:1/69: write f1 [1937297,46680] 0 2024-04-22T23:23:17.885 INFO:tasks.workunit.client.0.smithi067.stdout:3/125: write d3/d4/f20 [1641173,98692] 0 2024-04-22T23:23:17.885 INFO:tasks.workunit.client.0.smithi067.stdout:0/49: creat db/de/ff x:0 0 0 2024-04-22T23:23:17.889 INFO:tasks.workunit.client.0.smithi067.stdout:1/70: unlink f5 0 2024-04-22T23:23:17.890 INFO:tasks.workunit.client.0.smithi067.stdout:3/126: creat d3/d4/d12/f27 x:0 0 0 2024-04-22T23:23:17.891 INFO:tasks.workunit.client.0.smithi067.stdout:0/50: symlink db/de/l10 0 2024-04-22T23:23:17.895 INFO:tasks.workunit.client.0.smithi067.stdout:0/51: mkdir db/de/d11 0 2024-04-22T23:23:17.898 INFO:tasks.workunit.client.0.smithi067.stdout:3/127: mkdir d3/d4/d28 0 2024-04-22T23:23:17.899 INFO:tasks.workunit.client.0.smithi067.stdout:0/52: creat db/de/f12 x:0 0 0 2024-04-22T23:23:17.902 INFO:tasks.workunit.client.0.smithi067.stdout:3/128: symlink d3/d4/d12/l29 0 2024-04-22T23:23:17.904 INFO:tasks.workunit.client.0.smithi067.stdout:1/71: dwrite d8/fc [4194304,4194304] 0 2024-04-22T23:23:17.910 INFO:tasks.workunit.client.0.smithi067.stdout:0/53: dwrite f9 [0,4194304] 0 2024-04-22T23:23:17.915 INFO:tasks.workunit.client.0.smithi067.stdout:0/54: dread db/fd [0,4194304] 0 2024-04-22T23:23:17.917 INFO:tasks.workunit.client.0.smithi067.stdout:0/55: rmdir db 39 2024-04-22T23:23:17.976 INFO:tasks.workunit.client.0.smithi067.stdout:4/32: rename f6 to f7 0 2024-04-22T23:23:17.980 INFO:tasks.workunit.client.0.smithi067.stdout:4/33: dread f3 [0,4194304] 0 2024-04-22T23:23:17.984 INFO:tasks.workunit.client.0.smithi067.stdout:4/34: link f3 f8 0 2024-04-22T23:23:17.985 INFO:tasks.workunit.client.0.smithi067.stdout:4/35: creat f9 x:0 0 0 2024-04-22T23:23:17.988 INFO:tasks.workunit.client.0.smithi067.stdout:4/36: rename f9 to fa 0 2024-04-22T23:23:17.990 INFO:tasks.workunit.client.0.smithi067.stdout:4/37: link f3 fb 0 2024-04-22T23:23:17.992 INFO:tasks.workunit.client.0.smithi067.stdout:4/38: symlink lc 0 2024-04-22T23:23:17.994 INFO:tasks.workunit.client.0.smithi067.stdout:4/39: unlink lc 0 2024-04-22T23:23:17.996 INFO:tasks.workunit.client.0.smithi067.stdout:4/40: rename f3 to fd 0 2024-04-22T23:23:17.996 INFO:tasks.workunit.client.0.smithi067.stdout:4/41: read - fa zero size 2024-04-22T23:23:17.997 INFO:tasks.workunit.client.0.smithi067.stdout:4/42: readlink l1 0 2024-04-22T23:23:18.002 INFO:tasks.workunit.client.0.smithi067.stdout:9/15: unlink c3 0 2024-04-22T23:23:18.002 INFO:tasks.workunit.client.0.smithi067.stdout:9/16: chown f2 16 1 2024-04-22T23:23:18.002 INFO:tasks.workunit.client.0.smithi067.stdout:9/17: stat f2 0 2024-04-22T23:23:18.003 INFO:tasks.workunit.client.0.smithi067.stdout:9/18: write f2 [1041515,30865] 0 2024-04-22T23:23:18.004 INFO:tasks.workunit.client.0.smithi067.stdout:2/49: rmdir d1/da 0 2024-04-22T23:23:18.005 INFO:tasks.workunit.client.0.smithi067.stdout:9/19: write f2 [1416425,33154] 0 2024-04-22T23:23:18.005 INFO:tasks.workunit.client.0.smithi067.stdout:2/50: read d1/fb [116392,29042] 0 2024-04-22T23:23:18.006 INFO:tasks.workunit.client.0.smithi067.stdout:9/20: truncate f2 1666482 0 2024-04-22T23:23:18.007 INFO:tasks.workunit.client.0.smithi067.stdout:9/21: write f2 [756569,122204] 0 2024-04-22T23:23:18.008 INFO:tasks.workunit.client.0.smithi067.stdout:9/22: chown f2 680154502 1 2024-04-22T23:23:18.009 INFO:tasks.workunit.client.0.smithi067.stdout:6/10: truncate f0 1038925 0 2024-04-22T23:23:18.011 INFO:tasks.workunit.client.0.smithi067.stdout:2/51: rename d1/le to d1/lf 0 2024-04-22T23:23:18.011 INFO:tasks.workunit.client.0.smithi067.stdout:2/52: read - d1/fc zero size 2024-04-22T23:23:18.012 INFO:tasks.workunit.client.0.smithi067.stdout:9/23: mknod c4 0 2024-04-22T23:23:18.018 INFO:tasks.workunit.client.0.smithi067.stdout:9/24: mkdir d5 0 2024-04-22T23:23:18.023 INFO:tasks.workunit.client.0.smithi067.stdout:7/13: truncate f1 139113 0 2024-04-22T23:23:18.025 INFO:tasks.workunit.client.0.smithi067.stdout:2/53: dwrite d1/fb [0,4194304] 0 2024-04-22T23:23:18.031 INFO:tasks.workunit.client.0.smithi067.stdout:9/25: dwrite f2 [0,4194304] 0 2024-04-22T23:23:18.034 INFO:tasks.workunit.client.0.smithi067.stdout:8/80: truncate f0 4045969 0 2024-04-22T23:23:18.035 INFO:tasks.workunit.client.0.smithi067.stdout:8/81: dread - d1/d3/f1e zero size 2024-04-22T23:23:18.037 INFO:tasks.workunit.client.0.smithi067.stdout:9/26: creat d5/f6 x:0 0 0 2024-04-22T23:23:18.038 INFO:tasks.workunit.client.0.smithi067.stdout:2/54: dwrite d1/f4 [0,4194304] 0 2024-04-22T23:23:18.042 INFO:tasks.workunit.client.0.smithi067.stdout:9/27: rename f2 to d5/f7 0 2024-04-22T23:23:18.048 INFO:tasks.workunit.client.0.smithi067.stdout:0/56: readlink db/de/l10 0 2024-04-22T23:23:18.051 INFO:tasks.workunit.client.0.smithi067.stdout:9/28: dwrite d5/f6 [0,4194304] 0 2024-04-22T23:23:18.052 INFO:tasks.workunit.client.0.smithi067.stdout:0/57: write db/de/ff [31922,54990] 0 2024-04-22T23:23:18.055 INFO:tasks.workunit.client.0.smithi067.stdout:5/45: truncate f3 3075375 0 2024-04-22T23:23:18.056 INFO:tasks.workunit.client.0.smithi067.stdout:1/72: getdents d8/d9/df 0 2024-04-22T23:23:18.057 INFO:tasks.workunit.client.0.smithi067.stdout:1/73: write f1 [3640294,107684] 0 2024-04-22T23:23:18.059 INFO:tasks.workunit.client.0.smithi067.stdout:1/74: chown d8/d12 1574 1 2024-04-22T23:23:18.059 INFO:tasks.workunit.client.0.smithi067.stdout:5/46: rmdir d5/db 39 2024-04-22T23:23:18.061 INFO:tasks.workunit.client.0.smithi067.stdout:1/75: chown d8/d9/fb 112145008 1 2024-04-22T23:23:18.061 INFO:tasks.workunit.client.0.smithi067.stdout:9/29: dwrite d5/f6 [0,4194304] 0 2024-04-22T23:23:18.062 INFO:tasks.workunit.client.0.smithi067.stdout:1/76: read f1 [2096498,97016] 0 2024-04-22T23:23:18.067 INFO:tasks.workunit.client.0.smithi067.stdout:0/58: link c1 db/de/c13 0 2024-04-22T23:23:18.067 INFO:tasks.workunit.client.0.smithi067.stdout:9/30: dread d5/f6 [0,4194304] 0 2024-04-22T23:23:18.068 INFO:tasks.workunit.client.0.smithi067.stdout:9/31: chown d5/f6 8498029 1 2024-04-22T23:23:18.075 INFO:tasks.workunit.client.0.smithi067.stdout:0/59: rename f5 to db/de/f14 0 2024-04-22T23:23:18.076 INFO:tasks.workunit.client.0.smithi067.stdout:0/60: chown db/fd 52 1 2024-04-22T23:23:18.077 INFO:tasks.workunit.client.0.smithi067.stdout:0/61: dread db/de/ff [0,4194304] 0 2024-04-22T23:23:18.078 INFO:tasks.workunit.client.0.smithi067.stdout:1/77: symlink d8/d9/l16 0 2024-04-22T23:23:18.083 INFO:tasks.workunit.client.0.smithi067.stdout:9/32: dwrite d5/f7 [0,4194304] 0 2024-04-22T23:23:18.087 INFO:tasks.workunit.client.0.smithi067.stdout:0/62: rename db/de/c13 to db/de/d11/c15 0 2024-04-22T23:23:18.087 INFO:tasks.workunit.client.0.smithi067.stdout:0/63: readlink la 0 2024-04-22T23:23:18.088 INFO:tasks.workunit.client.0.smithi067.stdout:0/64: chown c6 213 1 2024-04-22T23:23:18.089 INFO:tasks.workunit.client.0.smithi067.stdout:1/78: creat d8/d12/f17 x:0 0 0 2024-04-22T23:23:18.101 INFO:tasks.workunit.client.0.smithi067.stdout:0/65: dwrite db/de/f12 [0,4194304] 0 2024-04-22T23:23:18.107 INFO:tasks.workunit.client.0.smithi067.stdout:1/79: dwrite d8/d9/f13 [0,4194304] 0 2024-04-22T23:23:18.107 INFO:tasks.workunit.client.0.smithi067.stdout:0/66: rmdir db 39 2024-04-22T23:23:18.129 INFO:tasks.workunit.client.0.smithi067.stdout:4/43: fsync fb 0 2024-04-22T23:23:18.129 INFO:tasks.workunit.client.0.smithi067.stdout:3/129: dwrite d3/d4/fe [0,4194304] 0 2024-04-22T23:23:18.131 INFO:tasks.workunit.client.0.smithi067.stdout:4/44: dread fd [0,4194304] 0 2024-04-22T23:23:18.140 INFO:tasks.workunit.client.0.smithi067.stdout:7/14: truncate f1 249575 0 2024-04-22T23:23:18.140 INFO:tasks.workunit.client.0.smithi067.stdout:7/15: readlink - no filename 2024-04-22T23:23:18.152 INFO:tasks.workunit.client.0.smithi067.stdout:8/82: dwrite d1/d3/f11 [4194304,4194304] 0 2024-04-22T23:23:18.155 INFO:tasks.workunit.client.0.smithi067.stdout:5/47: dread f3 [0,4194304] 0 2024-04-22T23:23:18.168 INFO:tasks.workunit.client.0.smithi067.stdout:9/33: truncate d5/f6 2206562 0 2024-04-22T23:23:18.169 INFO:tasks.workunit.client.0.smithi067.stdout:9/34: read d5/f7 [1926890,130970] 0 2024-04-22T23:23:18.282 INFO:tasks.workunit.client.0.smithi067.stdout:0/67: mknod db/c16 0 2024-04-22T23:23:18.282 INFO:tasks.workunit.client.0.smithi067.stdout:6/11: truncate f0 777865 0 2024-04-22T23:23:18.282 INFO:tasks.workunit.client.0.smithi067.stdout:0/68: chown db/fd 40165 1 2024-04-22T23:23:18.283 INFO:tasks.workunit.client.0.smithi067.stdout:4/45: unlink fb 0 2024-04-22T23:23:18.285 INFO:tasks.workunit.client.0.smithi067.stdout:8/83: symlink d1/dd/l1f 0 2024-04-22T23:23:18.286 INFO:tasks.workunit.client.0.smithi067.stdout:5/48: chown d5/db 0 1 2024-04-22T23:23:18.288 INFO:tasks.workunit.client.0.smithi067.stdout:9/35: mknod d5/c8 0 2024-04-22T23:23:18.292 INFO:tasks.workunit.client.0.smithi067.stdout:5/49: dwrite d5/f7 [0,4194304] 0 2024-04-22T23:23:18.294 INFO:tasks.workunit.client.0.smithi067.stdout:3/130: symlink d3/d4/l2a 0 2024-04-22T23:23:18.304 INFO:tasks.workunit.client.0.smithi067.stdout:1/80: creat d8/f18 x:0 0 0 2024-04-22T23:23:18.307 INFO:tasks.workunit.client.0.smithi067.stdout:8/84: dwrite d1/d3/f1b [0,4194304] 0 2024-04-22T23:23:18.314 INFO:tasks.workunit.client.0.smithi067.stdout:1/81: mkdir d8/d9/d19 0 2024-04-22T23:23:18.315 INFO:tasks.workunit.client.0.smithi067.stdout:1/82: write d8/d9/fb [811679,35975] 0 2024-04-22T23:23:18.316 INFO:tasks.workunit.client.0.smithi067.stdout:5/50: rename f3 to d5/f11 0 2024-04-22T23:23:18.321 INFO:tasks.workunit.client.0.smithi067.stdout:5/51: mkdir d5/db/d12 0 2024-04-22T23:23:18.322 INFO:tasks.workunit.client.0.smithi067.stdout:1/83: mknod d8/d9/c1a 0 2024-04-22T23:23:18.324 INFO:tasks.workunit.client.0.smithi067.stdout:5/52: rmdir d5/db 39 2024-04-22T23:23:18.324 INFO:tasks.workunit.client.0.smithi067.stdout:1/84: write d8/d12/f17 [494054,53858] 0 2024-04-22T23:23:18.324 INFO:tasks.workunit.client.0.smithi067.stdout:1/85: fsync d8/fc 0 2024-04-22T23:23:18.330 INFO:tasks.workunit.client.0.smithi067.stdout:5/53: creat d5/db/d12/f13 x:0 0 0 2024-04-22T23:23:18.330 INFO:tasks.workunit.client.0.smithi067.stdout:5/54: dread - d5/db/ff zero size 2024-04-22T23:23:18.331 INFO:tasks.workunit.client.0.smithi067.stdout:5/55: chown d5/db/f10 164 1 2024-04-22T23:23:18.333 INFO:tasks.workunit.client.0.smithi067.stdout:1/86: rename d8/d9/fb to d8/d9/f1b 0 2024-04-22T23:23:18.334 INFO:tasks.workunit.client.0.smithi067.stdout:1/87: dread - d8/f11 zero size 2024-04-22T23:23:18.335 INFO:tasks.workunit.client.0.smithi067.stdout:1/88: write d8/d9/fd [223599,58675] 0 2024-04-22T23:23:18.336 INFO:tasks.workunit.client.0.smithi067.stdout:5/56: mkdir d5/d14 0 2024-04-22T23:23:18.337 INFO:tasks.workunit.client.0.smithi067.stdout:5/57: chown d5/f8 15 1 2024-04-22T23:23:18.337 INFO:tasks.workunit.client.0.smithi067.stdout:5/58: write d5/fa [1902995,91500] 0 2024-04-22T23:23:18.338 INFO:tasks.workunit.client.0.smithi067.stdout:1/89: mkdir d8/d9/df/d1c 0 2024-04-22T23:23:18.338 INFO:tasks.workunit.client.0.smithi067.stdout:5/59: fsync d5/f6 0 2024-04-22T23:23:18.349 INFO:tasks.workunit.client.0.smithi067.stdout:1/90: dwrite d8/d9/f13 [0,4194304] 0 2024-04-22T23:23:18.349 INFO:tasks.workunit.client.0.smithi067.stdout:5/60: dwrite d5/db/fd [0,4194304] 0 2024-04-22T23:23:18.356 INFO:tasks.workunit.client.0.smithi067.stdout:1/91: rmdir d8/d9/d19 0 2024-04-22T23:23:18.360 INFO:tasks.workunit.client.0.smithi067.stdout:1/92: dread f1 [0,4194304] 0 2024-04-22T23:23:18.367 INFO:tasks.workunit.client.0.smithi067.stdout:1/93: dwrite d8/d12/f17 [0,4194304] 0 2024-04-22T23:23:18.370 INFO:tasks.workunit.client.0.smithi067.stdout:2/55: sync 2024-04-22T23:23:18.376 INFO:tasks.workunit.client.0.smithi067.stdout:9/36: getdents d5 0 2024-04-22T23:23:18.377 INFO:tasks.workunit.client.0.smithi067.stdout:7/16: chown f1 5 1 2024-04-22T23:23:18.378 INFO:tasks.workunit.client.0.smithi067.stdout:7/17: chown f1 4010113 1 2024-04-22T23:23:18.379 INFO:tasks.workunit.client.0.smithi067.stdout:2/56: dwrite d1/f4 [4194304,4194304] 0 2024-04-22T23:23:18.380 INFO:tasks.workunit.client.0.smithi067.stdout:7/18: chown f1 5587 1 2024-04-22T23:23:18.381 INFO:tasks.workunit.client.0.smithi067.stdout:9/37: creat d5/f9 x:0 0 0 2024-04-22T23:23:18.381 INFO:tasks.workunit.client.0.smithi067.stdout:0/69: write db/fd [1393643,13772] 0 2024-04-22T23:23:18.383 INFO:tasks.workunit.client.0.smithi067.stdout:0/70: fdatasync f8 0 2024-04-22T23:23:18.384 INFO:tasks.workunit.client.0.smithi067.stdout:8/85: rename f0 to d1/dd/f20 0 2024-04-22T23:23:18.390 INFO:tasks.workunit.client.0.smithi067.stdout:3/131: truncate d3/d4/f20 1874362 0 2024-04-22T23:23:18.391 INFO:tasks.workunit.client.0.smithi067.stdout:3/132: chown d3/d4/d10/d1f 20253131 1 2024-04-22T23:23:18.392 INFO:tasks.workunit.client.0.smithi067.stdout:8/86: dwrite d1/d3/f11 [0,4194304] 0 2024-04-22T23:23:18.392 INFO:tasks.workunit.client.0.smithi067.stdout:5/61: fsync d5/f11 0 2024-04-22T23:23:18.393 INFO:tasks.workunit.client.0.smithi067.stdout:5/62: chown d5/db/d12 1 1 2024-04-22T23:23:18.394 INFO:tasks.workunit.client.0.smithi067.stdout:9/38: truncate d5/f6 1930673 0 2024-04-22T23:23:18.395 INFO:tasks.workunit.client.0.smithi067.stdout:5/63: chown d5/fa 175060 1 2024-04-22T23:23:18.398 INFO:tasks.workunit.client.0.smithi067.stdout:0/71: getdents db 0 2024-04-22T23:23:18.399 INFO:tasks.workunit.client.0.smithi067.stdout:0/72: chown db/de/f12 1 1 2024-04-22T23:23:18.401 INFO:tasks.workunit.client.0.smithi067.stdout:5/64: creat d5/f15 x:0 0 0 2024-04-22T23:23:18.402 INFO:tasks.workunit.client.0.smithi067.stdout:5/65: truncate d5/db/f10 395691 0 2024-04-22T23:23:18.402 INFO:tasks.workunit.client.0.smithi067.stdout:5/66: fsync d5/f8 0 2024-04-22T23:23:18.403 INFO:tasks.workunit.client.0.smithi067.stdout:0/73: rename f2 to db/de/f17 0 2024-04-22T23:23:18.405 INFO:tasks.workunit.client.0.smithi067.stdout:5/67: rename d5/f6 to d5/d14/f16 0 2024-04-22T23:23:18.505 INFO:tasks.workunit.client.0.smithi067.stdout:1/94: truncate d8/fc 6122266 0 2024-04-22T23:23:18.506 INFO:tasks.workunit.client.0.smithi067.stdout:1/95: chown d8/d12/f17 12 1 2024-04-22T23:23:18.506 INFO:tasks.workunit.client.0.smithi067.stdout:5/68: getdents d5/db/d12 0 2024-04-22T23:23:18.512 INFO:tasks.workunit.client.0.smithi067.stdout:1/96: creat d8/f1d x:0 0 0 2024-04-22T23:23:18.512 INFO:tasks.workunit.client.0.smithi067.stdout:1/97: stat d8/ca 0 2024-04-22T23:23:18.513 INFO:tasks.workunit.client.0.smithi067.stdout:5/69: link d5/l9 d5/d14/l17 0 2024-04-22T23:23:18.515 INFO:tasks.workunit.client.0.smithi067.stdout:1/98: mknod d8/c1e 0 2024-04-22T23:23:18.516 INFO:tasks.workunit.client.0.smithi067.stdout:6/12: write f0 [145352,6911] 0 2024-04-22T23:23:18.517 INFO:tasks.workunit.client.0.smithi067.stdout:6/13: chown f0 435 1 2024-04-22T23:23:18.519 INFO:tasks.workunit.client.0.smithi067.stdout:1/99: rename d8/f18 to d8/d9/df/f1f 0 2024-04-22T23:23:18.524 INFO:tasks.workunit.client.0.smithi067.stdout:8/87: read d1/dd/f20 [3481350,103882] 0 2024-04-22T23:23:18.524 INFO:tasks.workunit.client.0.smithi067.stdout:4/46: write fd [947657,40945] 0 2024-04-22T23:23:18.526 INFO:tasks.workunit.client.0.smithi067.stdout:6/14: dwrite f0 [0,4194304] 0 2024-04-22T23:23:18.529 INFO:tasks.workunit.client.0.smithi067.stdout:6/15: read f0 [159489,39056] 0 2024-04-22T23:23:18.530 INFO:tasks.workunit.client.0.smithi067.stdout:1/100: dread d8/d12/f17 [0,4194304] 0 2024-04-22T23:23:18.531 INFO:tasks.workunit.client.0.smithi067.stdout:4/47: creat fe x:0 0 0 2024-04-22T23:23:18.533 INFO:tasks.workunit.client.0.smithi067.stdout:9/39: write d5/f7 [277994,30789] 0 2024-04-22T23:23:18.533 INFO:tasks.workunit.client.0.smithi067.stdout:3/133: dread d3/d4/f20 [0,4194304] 0 2024-04-22T23:23:18.537 INFO:tasks.workunit.client.0.smithi067.stdout:1/101: read d8/d9/fd [184725,95109] 0 2024-04-22T23:23:18.537 INFO:tasks.workunit.client.0.smithi067.stdout:7/19: dwrite f1 [0,4194304] 0 2024-04-22T23:23:18.543 INFO:tasks.workunit.client.0.smithi067.stdout:7/20: dread f1 [0,4194304] 0 2024-04-22T23:23:18.553 INFO:tasks.workunit.client.0.smithi067.stdout:7/21: symlink l2 0 2024-04-22T23:23:18.554 INFO:tasks.workunit.client.0.smithi067.stdout:8/88: link d1/dd/l1f d1/l21 0 2024-04-22T23:23:18.555 INFO:tasks.workunit.client.0.smithi067.stdout:1/102: dwrite d8/d12/f17 [0,4194304] 0 2024-04-22T23:23:18.556 INFO:tasks.workunit.client.0.smithi067.stdout:3/134: mknod d3/d4/d28/c2b 0 2024-04-22T23:23:18.557 INFO:tasks.workunit.client.0.smithi067.stdout:3/135: read - d3/d4/f24 zero size 2024-04-22T23:23:18.558 INFO:tasks.workunit.client.0.smithi067.stdout:3/136: read d3/fa [437544,121585] 0 2024-04-22T23:23:18.562 INFO:tasks.workunit.client.0.smithi067.stdout:7/22: rename l2 to l3 0 2024-04-22T23:23:18.563 INFO:tasks.workunit.client.0.smithi067.stdout:8/89: fsync d1/dd/f20 0 2024-04-22T23:23:18.566 INFO:tasks.workunit.client.0.smithi067.stdout:4/48: getdents . 0 2024-04-22T23:23:18.566 INFO:tasks.workunit.client.0.smithi067.stdout:6/16: truncate f0 368037 0 2024-04-22T23:23:18.567 INFO:tasks.workunit.client.0.smithi067.stdout:4/49: write fe [267739,44546] 0 2024-04-22T23:23:18.567 INFO:tasks.workunit.client.0.smithi067.stdout:6/17: chown c1 649 1 2024-04-22T23:23:18.567 INFO:tasks.workunit.client.0.smithi067.stdout:4/50: dread - fa zero size 2024-04-22T23:23:18.568 INFO:tasks.workunit.client.0.smithi067.stdout:3/137: mknod d3/d4/d17/c2c 0 2024-04-22T23:23:18.571 INFO:tasks.workunit.client.0.smithi067.stdout:9/40: getdents d5 0 2024-04-22T23:23:18.573 INFO:tasks.workunit.client.0.smithi067.stdout:6/18: mknod c2 0 2024-04-22T23:23:18.575 INFO:tasks.workunit.client.0.smithi067.stdout:3/138: mknod d3/d4/d10/c2d 0 2024-04-22T23:23:18.576 INFO:tasks.workunit.client.0.smithi067.stdout:9/41: creat d5/fa x:0 0 0 2024-04-22T23:23:18.577 INFO:tasks.workunit.client.0.smithi067.stdout:3/139: write d3/d4/fc [763300,63732] 0 2024-04-22T23:23:18.577 INFO:tasks.workunit.client.0.smithi067.stdout:8/90: creat d1/d3/d14/d15/f22 x:0 0 0 2024-04-22T23:23:18.578 INFO:tasks.workunit.client.0.smithi067.stdout:1/103: rename c3 to d8/c20 0 2024-04-22T23:23:18.580 INFO:tasks.workunit.client.0.smithi067.stdout:6/19: symlink l3 0 2024-04-22T23:23:18.580 INFO:tasks.workunit.client.0.smithi067.stdout:1/104: stat d8/d9/f1b 0 2024-04-22T23:23:18.581 INFO:tasks.workunit.client.0.smithi067.stdout:6/20: stat c1 0 2024-04-22T23:23:18.582 INFO:tasks.workunit.client.0.smithi067.stdout:9/42: creat d5/fb x:0 0 0 2024-04-22T23:23:18.584 INFO:tasks.workunit.client.0.smithi067.stdout:4/51: dwrite f7 [0,4194304] 0 2024-04-22T23:23:18.586 INFO:tasks.workunit.client.0.smithi067.stdout:7/23: getdents . 0 2024-04-22T23:23:18.587 INFO:tasks.workunit.client.0.smithi067.stdout:6/21: creat f4 x:0 0 0 2024-04-22T23:23:18.588 INFO:tasks.workunit.client.0.smithi067.stdout:8/91: dread d1/d3/f1b [0,4194304] 0 2024-04-22T23:23:18.590 INFO:tasks.workunit.client.0.smithi067.stdout:9/43: mkdir d5/dc 0 2024-04-22T23:23:18.596 INFO:tasks.workunit.client.0.smithi067.stdout:8/92: mknod d1/dd/c23 0 2024-04-22T23:23:18.598 INFO:tasks.workunit.client.0.smithi067.stdout:4/52: getdents . 0 2024-04-22T23:23:18.598 INFO:tasks.workunit.client.0.smithi067.stdout:8/93: fdatasync d1/d3/f11 0 2024-04-22T23:23:18.599 INFO:tasks.workunit.client.0.smithi067.stdout:9/44: dwrite d5/fb [0,4194304] 0 2024-04-22T23:23:18.599 INFO:tasks.workunit.client.0.smithi067.stdout:4/53: stat l1 0 2024-04-22T23:23:18.600 INFO:tasks.workunit.client.0.smithi067.stdout:8/94: fdatasync d1/d3/f1e 0 2024-04-22T23:23:18.600 INFO:tasks.workunit.client.0.smithi067.stdout:9/45: fsync d5/f9 0 2024-04-22T23:23:18.601 INFO:tasks.workunit.client.0.smithi067.stdout:3/140: link d3/d4/d28/c2b d3/d4/d10/d19/c2e 0 2024-04-22T23:23:18.607 INFO:tasks.workunit.client.0.smithi067.stdout:4/54: creat ff x:0 0 0 2024-04-22T23:23:18.608 INFO:tasks.workunit.client.0.smithi067.stdout:4/55: write f5 [254049,22413] 0 2024-04-22T23:23:18.608 INFO:tasks.workunit.client.0.smithi067.stdout:8/95: mknod d1/d3/c24 0 2024-04-22T23:23:18.610 INFO:tasks.workunit.client.0.smithi067.stdout:9/46: mkdir d5/dd 0 2024-04-22T23:23:18.612 INFO:tasks.workunit.client.0.smithi067.stdout:3/141: creat d3/d4/d12/f2f x:0 0 0 2024-04-22T23:23:18.614 INFO:tasks.workunit.client.0.smithi067.stdout:4/56: creat f10 x:0 0 0 2024-04-22T23:23:18.616 INFO:tasks.workunit.client.0.smithi067.stdout:8/96: rename d1/dd/cf to d1/d3/d14/d15/c25 0 2024-04-22T23:23:18.617 INFO:tasks.workunit.client.0.smithi067.stdout:3/142: rename d3/d4/d10 to d3/d4/d10/d19/d30 22 2024-04-22T23:23:18.619 INFO:tasks.workunit.client.0.smithi067.stdout:0/74: sync 2024-04-22T23:23:18.619 INFO:tasks.workunit.client.0.smithi067.stdout:2/57: sync 2024-04-22T23:23:18.622 INFO:tasks.workunit.client.0.smithi067.stdout:3/143: write d3/d4/d10/d19/f21 [778967,1011] 0 2024-04-22T23:23:18.626 INFO:tasks.workunit.client.0.smithi067.stdout:8/97: rename d1/d3/d14/d15 to d1/dd/d26 0 2024-04-22T23:23:18.627 INFO:tasks.workunit.client.0.smithi067.stdout:4/57: dwrite ff [0,4194304] 0 2024-04-22T23:23:18.627 INFO:tasks.workunit.client.0.smithi067.stdout:9/47: write d5/f6 [34491,49472] 0 2024-04-22T23:23:18.629 INFO:tasks.workunit.client.0.smithi067.stdout:8/98: dread - d1/f7 zero size 2024-04-22T23:23:18.631 INFO:tasks.workunit.client.0.smithi067.stdout:3/144: readlink d3/d4/l11 0 2024-04-22T23:23:18.631 INFO:tasks.workunit.client.0.smithi067.stdout:6/22: dwrite f0 [0,4194304] 0 2024-04-22T23:23:18.631 INFO:tasks.workunit.client.0.smithi067.stdout:9/48: fsync d5/f6 0 2024-04-22T23:23:18.632 INFO:tasks.workunit.client.0.smithi067.stdout:2/58: getdents d1 0 2024-04-22T23:23:18.635 INFO:tasks.workunit.client.0.smithi067.stdout:8/99: write d1/d3/fb [3237415,15292] 0 2024-04-22T23:23:18.637 INFO:tasks.workunit.client.0.smithi067.stdout:8/100: read - d1/f18 zero size 2024-04-22T23:23:18.638 INFO:tasks.workunit.client.0.smithi067.stdout:2/59: truncate d1/f9 1937890 0 2024-04-22T23:23:18.638 INFO:tasks.workunit.client.0.smithi067.stdout:9/49: dread - d5/fa zero size 2024-04-22T23:23:18.640 INFO:tasks.workunit.client.0.smithi067.stdout:8/101: truncate d1/f18 8212 0 2024-04-22T23:23:18.640 INFO:tasks.workunit.client.0.smithi067.stdout:3/145: symlink d3/d4/d17/l31 0 2024-04-22T23:23:18.640 INFO:tasks.workunit.client.0.smithi067.stdout:6/23: write f4 [878985,9824] 0 2024-04-22T23:23:18.642 INFO:tasks.workunit.client.0.smithi067.stdout:2/60: mkdir d1/d10 0 2024-04-22T23:23:18.642 INFO:tasks.workunit.client.0.smithi067.stdout:2/61: stat d1/fb 0 2024-04-22T23:23:18.644 INFO:tasks.workunit.client.0.smithi067.stdout:8/102: mkdir d1/d3/d27 0 2024-04-22T23:23:18.645 INFO:tasks.workunit.client.0.smithi067.stdout:6/24: mknod c5 0 2024-04-22T23:23:18.646 INFO:tasks.workunit.client.0.smithi067.stdout:3/146: unlink d3/d4/l11 0 2024-04-22T23:23:18.647 INFO:tasks.workunit.client.0.smithi067.stdout:4/58: getdents . 0 2024-04-22T23:23:18.647 INFO:tasks.workunit.client.0.smithi067.stdout:3/147: read d3/d4/f20 [1663038,97798] 0 2024-04-22T23:23:18.649 INFO:tasks.workunit.client.0.smithi067.stdout:2/62: rename d1/fb to d1/f11 0 2024-04-22T23:23:18.650 INFO:tasks.workunit.client.0.smithi067.stdout:2/63: truncate d1/f11 4758406 0 2024-04-22T23:23:18.651 INFO:tasks.workunit.client.0.smithi067.stdout:6/25: creat f6 x:0 0 0 2024-04-22T23:23:18.654 INFO:tasks.workunit.client.0.smithi067.stdout:4/59: creat f11 x:0 0 0 2024-04-22T23:23:18.655 INFO:tasks.workunit.client.0.smithi067.stdout:3/148: truncate d3/fa 4205871 0 2024-04-22T23:23:18.657 INFO:tasks.workunit.client.0.smithi067.stdout:9/50: rmdir d5/dc 0 2024-04-22T23:23:18.658 INFO:tasks.workunit.client.0.smithi067.stdout:4/60: dread fe [0,4194304] 0 2024-04-22T23:23:18.658 INFO:tasks.workunit.client.0.smithi067.stdout:2/64: mkdir d1/d12 0 2024-04-22T23:23:18.662 INFO:tasks.workunit.client.0.smithi067.stdout:3/149: symlink d3/d4/d10/d19/l32 0 2024-04-22T23:23:18.663 INFO:tasks.workunit.client.0.smithi067.stdout:3/150: chown d3/d4/d1b/l26 171118 1 2024-04-22T23:23:18.664 INFO:tasks.workunit.client.0.smithi067.stdout:3/151: chown d3/d4/d12 7257238 1 2024-04-22T23:23:18.665 INFO:tasks.workunit.client.0.smithi067.stdout:4/61: mkdir d12 0 2024-04-22T23:23:18.667 INFO:tasks.workunit.client.0.smithi067.stdout:6/26: link f6 f7 0 2024-04-22T23:23:18.669 INFO:tasks.workunit.client.0.smithi067.stdout:9/51: link d5/f9 d5/fe 0 2024-04-22T23:23:18.669 INFO:tasks.workunit.client.0.smithi067.stdout:2/65: symlink d1/d12/l13 0 2024-04-22T23:23:18.674 INFO:tasks.workunit.client.0.smithi067.stdout:2/66: mknod d1/c14 0 2024-04-22T23:23:18.674 INFO:tasks.workunit.client.0.smithi067.stdout:3/152: dwrite d3/d4/d10/d19/f21 [0,4194304] 0 2024-04-22T23:23:18.675 INFO:tasks.workunit.client.0.smithi067.stdout:9/52: rename d5/f7 to d5/ff 0 2024-04-22T23:23:18.679 INFO:tasks.workunit.client.0.smithi067.stdout:4/62: dwrite f7 [0,4194304] 0 2024-04-22T23:23:18.684 INFO:tasks.workunit.client.0.smithi067.stdout:2/67: dread d1/f6 [0,4194304] 0 2024-04-22T23:23:18.684 INFO:tasks.workunit.client.0.smithi067.stdout:9/53: write d5/ff [322178,102950] 0 2024-04-22T23:23:18.687 INFO:tasks.workunit.client.0.smithi067.stdout:4/63: dread fe [0,4194304] 0 2024-04-22T23:23:18.688 INFO:tasks.workunit.client.0.smithi067.stdout:6/27: dwrite f6 [0,4194304] 0 2024-04-22T23:23:18.689 INFO:tasks.workunit.client.0.smithi067.stdout:3/153: creat d3/d4/d1b/f33 x:0 0 0 2024-04-22T23:23:18.689 INFO:tasks.workunit.client.0.smithi067.stdout:4/64: readlink l1 0 2024-04-22T23:23:18.691 INFO:tasks.workunit.client.0.smithi067.stdout:2/68: unlink d1/f11 0 2024-04-22T23:23:18.691 INFO:tasks.workunit.client.0.smithi067.stdout:2/69: stat d1/f3 0 2024-04-22T23:23:18.693 INFO:tasks.workunit.client.0.smithi067.stdout:2/70: write d1/f3 [850467,58578] 0 2024-04-22T23:23:18.694 INFO:tasks.workunit.client.0.smithi067.stdout:9/54: creat d5/f10 x:0 0 0 2024-04-22T23:23:18.696 INFO:tasks.workunit.client.0.smithi067.stdout:3/154: dread d3/d4/fe [0,4194304] 0 2024-04-22T23:23:18.697 INFO:tasks.workunit.client.0.smithi067.stdout:2/71: dread d1/f9 [0,4194304] 0 2024-04-22T23:23:18.698 INFO:tasks.workunit.client.0.smithi067.stdout:3/155: write d3/d4/d10/f1a [1263069,19661] 0 2024-04-22T23:23:18.699 INFO:tasks.workunit.client.0.smithi067.stdout:2/72: write d1/f9 [455564,113307] 0 2024-04-22T23:23:18.703 INFO:tasks.workunit.client.0.smithi067.stdout:9/55: symlink d5/l11 0 2024-04-22T23:23:18.704 INFO:tasks.workunit.client.0.smithi067.stdout:9/56: write d5/f6 [1008512,61339] 0 2024-04-22T23:23:18.704 INFO:tasks.workunit.client.0.smithi067.stdout:4/65: mknod d12/c13 0 2024-04-22T23:23:18.704 INFO:tasks.workunit.client.0.smithi067.stdout:6/28: read f4 [800090,4004] 0 2024-04-22T23:23:18.705 INFO:tasks.workunit.client.0.smithi067.stdout:9/57: dread - d5/fe zero size 2024-04-22T23:23:18.707 INFO:tasks.workunit.client.0.smithi067.stdout:3/156: creat d3/d4/d10/d19/f34 x:0 0 0 2024-04-22T23:23:18.708 INFO:tasks.workunit.client.0.smithi067.stdout:4/66: symlink d12/l14 0 2024-04-22T23:23:18.710 INFO:tasks.workunit.client.0.smithi067.stdout:3/157: truncate d3/d4/f24 889325 0 2024-04-22T23:23:18.714 INFO:tasks.workunit.client.0.smithi067.stdout:9/58: dwrite d5/fb [0,4194304] 0 2024-04-22T23:23:18.720 INFO:tasks.workunit.client.0.smithi067.stdout:4/67: write fe [1224862,73283] 0 2024-04-22T23:23:18.724 INFO:tasks.workunit.client.0.smithi067.stdout:3/158: dwrite d3/d4/d10/f1a [0,4194304] 0 2024-04-22T23:23:18.724 INFO:tasks.workunit.client.0.smithi067.stdout:2/73: getdents d1/d12 0 2024-04-22T23:23:18.726 INFO:tasks.workunit.client.0.smithi067.stdout:4/68: creat d12/f15 x:0 0 0 2024-04-22T23:23:18.728 INFO:tasks.workunit.client.0.smithi067.stdout:9/59: creat d5/dd/f12 x:0 0 0 2024-04-22T23:23:18.731 INFO:tasks.workunit.client.0.smithi067.stdout:4/69: creat d12/f16 x:0 0 0 2024-04-22T23:23:18.732 INFO:tasks.workunit.client.0.smithi067.stdout:4/70: chown d12/c13 44 1 2024-04-22T23:23:18.733 INFO:tasks.workunit.client.0.smithi067.stdout:9/60: rename d5/l11 to d5/dd/l13 0 2024-04-22T23:23:18.736 INFO:tasks.workunit.client.0.smithi067.stdout:4/71: mkdir d12/d17 0 2024-04-22T23:23:18.741 INFO:tasks.workunit.client.0.smithi067.stdout:3/159: creat d3/f35 x:0 0 0 2024-04-22T23:23:18.742 INFO:tasks.workunit.client.0.smithi067.stdout:4/72: symlink d12/l18 0 2024-04-22T23:23:18.742 INFO:tasks.workunit.client.0.smithi067.stdout:6/29: truncate f6 2239223 0 2024-04-22T23:23:18.742 INFO:tasks.workunit.client.0.smithi067.stdout:3/160: rmdir d3/d4/d1b 39 2024-04-22T23:23:18.744 INFO:tasks.workunit.client.0.smithi067.stdout:6/30: creat f8 x:0 0 0 2024-04-22T23:23:18.744 INFO:tasks.workunit.client.0.smithi067.stdout:6/31: rmdir - no directory 2024-04-22T23:23:18.744 INFO:tasks.workunit.client.0.smithi067.stdout:9/61: dwrite d5/f6 [0,4194304] 0 2024-04-22T23:23:18.746 INFO:tasks.workunit.client.0.smithi067.stdout:6/32: dread - f8 zero size 2024-04-22T23:23:18.746 INFO:tasks.workunit.client.0.smithi067.stdout:3/161: fsync d3/d4/d17/f25 0 2024-04-22T23:23:18.746 INFO:tasks.workunit.client.0.smithi067.stdout:6/33: readlink l3 0 2024-04-22T23:23:18.747 INFO:tasks.workunit.client.0.smithi067.stdout:3/162: chown d3 21840948 1 2024-04-22T23:23:18.749 INFO:tasks.workunit.client.0.smithi067.stdout:9/62: write d5/dd/f12 [76049,32041] 0 2024-04-22T23:23:18.752 INFO:tasks.workunit.client.0.smithi067.stdout:9/63: read d5/fb [2846293,61982] 0 2024-04-22T23:23:18.754 INFO:tasks.workunit.client.0.smithi067.stdout:6/34: write f4 [588391,119009] 0 2024-04-22T23:23:18.754 INFO:tasks.workunit.client.0.smithi067.stdout:4/73: dwrite d12/f15 [0,4194304] 0 2024-04-22T23:23:18.756 INFO:tasks.workunit.client.0.smithi067.stdout:3/163: unlink d3/d4/d1b/f33 0 2024-04-22T23:23:18.761 INFO:tasks.workunit.client.0.smithi067.stdout:9/64: rmdir d5 39 2024-04-22T23:23:18.761 INFO:tasks.workunit.client.0.smithi067.stdout:6/35: dread f0 [0,4194304] 0 2024-04-22T23:23:18.761 INFO:tasks.workunit.client.0.smithi067.stdout:3/164: dread d3/d4/f16 [0,4194304] 0 2024-04-22T23:23:18.764 INFO:tasks.workunit.client.0.smithi067.stdout:4/74: rename f11 to d12/f19 0 2024-04-22T23:23:18.764 INFO:tasks.workunit.client.0.smithi067.stdout:4/75: write f7 [3205872,71232] 0 2024-04-22T23:23:18.766 INFO:tasks.workunit.client.0.smithi067.stdout:4/76: write fd [2085538,96345] 0 2024-04-22T23:23:18.766 INFO:tasks.workunit.client.0.smithi067.stdout:9/65: dread - d5/f9 zero size 2024-04-22T23:23:18.766 INFO:tasks.workunit.client.0.smithi067.stdout:4/77: stat fa 0 2024-04-22T23:23:18.767 INFO:tasks.workunit.client.0.smithi067.stdout:4/78: fsync f5 0 2024-04-22T23:23:18.768 INFO:tasks.workunit.client.0.smithi067.stdout:6/36: rename c5 to c9 0 2024-04-22T23:23:18.772 INFO:tasks.workunit.client.0.smithi067.stdout:6/37: dread f6 [0,4194304] 0 2024-04-22T23:23:18.773 INFO:tasks.workunit.client.0.smithi067.stdout:9/66: unlink d5/dd/f12 0 2024-04-22T23:23:18.773 INFO:tasks.workunit.client.0.smithi067.stdout:4/79: dread fe [0,4194304] 0 2024-04-22T23:23:18.775 INFO:tasks.workunit.client.0.smithi067.stdout:6/38: dread f7 [0,4194304] 0 2024-04-22T23:23:18.780 INFO:tasks.workunit.client.0.smithi067.stdout:6/39: symlink la 0 2024-04-22T23:23:18.783 INFO:tasks.workunit.client.0.smithi067.stdout:6/40: creat fb x:0 0 0 2024-04-22T23:23:18.788 INFO:tasks.workunit.client.0.smithi067.stdout:4/80: truncate f4 1183913 0 2024-04-22T23:23:18.790 INFO:tasks.workunit.client.0.smithi067.stdout:6/41: truncate f7 1836557 0 2024-04-22T23:23:18.796 INFO:tasks.workunit.client.0.smithi067.stdout:9/67: dwrite d5/fe [0,4194304] 0 2024-04-22T23:23:18.798 INFO:tasks.workunit.client.0.smithi067.stdout:6/42: rename f7 to fc 0 2024-04-22T23:23:18.800 INFO:tasks.workunit.client.0.smithi067.stdout:4/81: dwrite f10 [0,4194304] 0 2024-04-22T23:23:18.805 INFO:tasks.workunit.client.0.smithi067.stdout:4/82: symlink d12/d17/l1a 0 2024-04-22T23:23:18.805 INFO:tasks.workunit.client.0.smithi067.stdout:4/83: stat d12/d17 0 2024-04-22T23:23:18.807 INFO:tasks.workunit.client.0.smithi067.stdout:4/84: write fd [108475,78501] 0 2024-04-22T23:23:18.810 INFO:tasks.workunit.client.0.smithi067.stdout:6/43: dwrite fb [0,4194304] 0 2024-04-22T23:23:18.816 INFO:tasks.workunit.client.0.smithi067.stdout:6/44: mknod cd 0 2024-04-22T23:23:18.819 INFO:tasks.workunit.client.0.smithi067.stdout:4/85: dwrite f8 [0,4194304] 0 2024-04-22T23:23:18.820 INFO:tasks.workunit.client.0.smithi067.stdout:6/45: rename f4 to fe 0 2024-04-22T23:23:18.830 INFO:tasks.workunit.client.0.smithi067.stdout:5/70: sync 2024-04-22T23:23:18.833 INFO:tasks.workunit.client.0.smithi067.stdout:4/86: symlink d12/d17/l1b 0 2024-04-22T23:23:18.834 INFO:tasks.workunit.client.0.smithi067.stdout:5/71: symlink d5/db/d12/l18 0 2024-04-22T23:23:18.835 INFO:tasks.workunit.client.0.smithi067.stdout:4/87: symlink d12/l1c 0 2024-04-22T23:23:18.835 INFO:tasks.workunit.client.0.smithi067.stdout:4/88: dread - d12/f19 zero size 2024-04-22T23:23:18.837 INFO:tasks.workunit.client.0.smithi067.stdout:5/72: unlink d5/db/f10 0 2024-04-22T23:23:18.839 INFO:tasks.workunit.client.0.smithi067.stdout:4/89: read f7 [944524,111308] 0 2024-04-22T23:23:18.840 INFO:tasks.workunit.client.0.smithi067.stdout:5/73: dread d5/f7 [0,4194304] 0 2024-04-22T23:23:18.841 INFO:tasks.workunit.client.0.smithi067.stdout:4/90: mkdir d12/d17/d1d 0 2024-04-22T23:23:18.843 INFO:tasks.workunit.client.0.smithi067.stdout:4/91: unlink f10 0 2024-04-22T23:23:18.844 INFO:tasks.workunit.client.0.smithi067.stdout:5/74: write d5/d14/f16 [5226517,36477] 0 2024-04-22T23:23:18.845 INFO:tasks.workunit.client.0.smithi067.stdout:5/75: truncate d5/f15 330484 0 2024-04-22T23:23:18.845 INFO:tasks.workunit.client.0.smithi067.stdout:4/92: mkdir d12/d17/d1e 0 2024-04-22T23:23:18.846 INFO:tasks.workunit.client.0.smithi067.stdout:4/93: chown d12/f15 5 1 2024-04-22T23:23:18.852 INFO:tasks.workunit.client.0.smithi067.stdout:5/76: dread d5/db/fd [0,4194304] 0 2024-04-22T23:23:18.859 INFO:tasks.workunit.client.0.smithi067.stdout:5/77: dwrite d5/db/fd [0,4194304] 0 2024-04-22T23:23:18.863 INFO:tasks.workunit.client.0.smithi067.stdout:5/78: unlink d5/f8 0 2024-04-22T23:23:18.865 INFO:tasks.workunit.client.0.smithi067.stdout:5/79: mknod d5/db/c19 0 2024-04-22T23:23:18.874 INFO:tasks.workunit.client.0.smithi067.stdout:5/80: dread d5/f11 [0,4194304] 0 2024-04-22T23:23:18.876 INFO:tasks.workunit.client.0.smithi067.stdout:5/81: mkdir d5/d1a 0 2024-04-22T23:23:18.877 INFO:tasks.workunit.client.0.smithi067.stdout:5/82: chown d5/db/d12/f13 1748 1 2024-04-22T23:23:18.878 INFO:tasks.workunit.client.0.smithi067.stdout:5/83: rmdir d5/d14 39 2024-04-22T23:23:18.880 INFO:tasks.workunit.client.0.smithi067.stdout:5/84: creat d5/f1b x:0 0 0 2024-04-22T23:23:18.881 INFO:tasks.workunit.client.0.smithi067.stdout:5/85: symlink d5/d1a/l1c 0 2024-04-22T23:23:18.887 INFO:tasks.workunit.client.0.smithi067.stdout:5/86: truncate d5/db/fd 3019310 0 2024-04-22T23:23:18.890 INFO:tasks.workunit.client.0.smithi067.stdout:5/87: link d5/d1a/l1c d5/db/l1d 0 2024-04-22T23:23:18.902 INFO:tasks.workunit.client.0.smithi067.stdout:5/88: dwrite d5/db/fd [0,4194304] 0 2024-04-22T23:23:18.905 INFO:tasks.workunit.client.0.smithi067.stdout:5/89: mknod d5/d14/c1e 0 2024-04-22T23:23:18.906 INFO:tasks.workunit.client.0.smithi067.stdout:5/90: fsync d5/f7 0 2024-04-22T23:23:18.907 INFO:tasks.workunit.client.0.smithi067.stdout:5/91: fdatasync d5/d14/f16 0 2024-04-22T23:23:18.909 INFO:tasks.workunit.client.0.smithi067.stdout:5/92: mknod d5/d14/c1f 0 2024-04-22T23:23:18.910 INFO:tasks.workunit.client.0.smithi067.stdout:5/93: creat d5/d14/f20 x:0 0 0 2024-04-22T23:23:18.911 INFO:tasks.workunit.client.0.smithi067.stdout:5/94: mknod d5/db/c21 0 2024-04-22T23:23:18.914 INFO:tasks.workunit.client.0.smithi067.stdout:5/95: creat d5/db/d12/f22 x:0 0 0 2024-04-22T23:23:18.915 INFO:tasks.workunit.client.0.smithi067.stdout:5/96: write d5/f7 [3056945,32787] 0 2024-04-22T23:23:18.917 INFO:tasks.workunit.client.0.smithi067.stdout:5/97: creat d5/db/f23 x:0 0 0 2024-04-22T23:23:18.968 INFO:tasks.workunit.client.0.smithi067.stdout:1/105: sync 2024-04-22T23:23:18.969 INFO:tasks.workunit.client.0.smithi067.stdout:5/98: link d5/db/d12/l18 d5/db/l24 0 2024-04-22T23:23:18.969 INFO:tasks.workunit.client.0.smithi067.stdout:5/99: truncate d5/db/f23 797109 0 2024-04-22T23:23:18.970 INFO:tasks.workunit.client.0.smithi067.stdout:5/100: chown d5/db/d12 5801150 1 2024-04-22T23:23:18.971 INFO:tasks.workunit.client.0.smithi067.stdout:1/106: link d8/f11 d8/d9/df/f21 0 2024-04-22T23:23:18.973 INFO:tasks.workunit.client.0.smithi067.stdout:1/107: symlink d8/d9/l22 0 2024-04-22T23:23:18.974 INFO:tasks.workunit.client.0.smithi067.stdout:1/108: fdatasync f1 0 2024-04-22T23:23:18.977 INFO:tasks.workunit.client.0.smithi067.stdout:5/101: dwrite d5/d14/f20 [0,4194304] 0 2024-04-22T23:23:18.978 INFO:tasks.workunit.client.0.smithi067.stdout:1/109: write d8/d9/df/f1f [697681,40335] 0 2024-04-22T23:23:18.979 INFO:tasks.workunit.client.0.smithi067.stdout:5/102: write d5/f7 [1513267,110754] 0 2024-04-22T23:23:18.979 INFO:tasks.workunit.client.0.smithi067.stdout:5/103: chown d5/d1a 18160 1 2024-04-22T23:23:18.981 INFO:tasks.workunit.client.0.smithi067.stdout:1/110: creat d8/d9/df/f23 x:0 0 0 2024-04-22T23:23:18.983 INFO:tasks.workunit.client.0.smithi067.stdout:5/104: rename d5/db/fd to d5/db/d12/f25 0 2024-04-22T23:23:18.987 INFO:tasks.workunit.client.0.smithi067.stdout:1/111: fsync d8/fc 0 2024-04-22T23:23:18.988 INFO:tasks.workunit.client.0.smithi067.stdout:1/112: read - d8/f1d zero size 2024-04-22T23:23:18.990 INFO:tasks.workunit.client.0.smithi067.stdout:1/113: mkdir d8/d24 0 2024-04-22T23:23:18.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:18 smithi134 ceph-mon[34609]: pgmap v264: 65 pgs: 65 active+clean; 252 MiB data, 849 MiB used, 536 GiB / 536 GiB avail; 2.6 MiB/s rd, 10 MiB/s wr, 489 op/s 2024-04-22T23:23:19.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:18 smithi067 ceph-mon[26453]: pgmap v264: 65 pgs: 65 active+clean; 252 MiB data, 849 MiB used, 536 GiB / 536 GiB avail; 2.6 MiB/s rd, 10 MiB/s wr, 489 op/s 2024-04-22T23:23:19.112 INFO:tasks.workunit.client.0.smithi067.stdout:7/24: sync 2024-04-22T23:23:19.115 INFO:tasks.workunit.client.0.smithi067.stdout:7/25: dread f1 [0,4194304] 0 2024-04-22T23:23:19.115 INFO:tasks.workunit.client.0.smithi067.stdout:7/26: readlink l3 0 2024-04-22T23:23:19.116 INFO:tasks.workunit.client.0.smithi067.stdout:7/27: read f1 [720596,30343] 0 2024-04-22T23:23:19.117 INFO:tasks.workunit.client.0.smithi067.stdout:7/28: fsync f1 0 2024-04-22T23:23:19.119 INFO:tasks.workunit.client.0.smithi067.stdout:7/29: truncate f1 2632888 0 2024-04-22T23:23:19.119 INFO:tasks.workunit.client.0.smithi067.stdout:7/30: chown f1 1 1 2024-04-22T23:23:19.124 INFO:tasks.workunit.client.0.smithi067.stdout:7/31: dread f1 [0,4194304] 0 2024-04-22T23:23:19.126 INFO:tasks.workunit.client.0.smithi067.stdout:7/32: creat f4 x:0 0 0 2024-04-22T23:23:19.134 INFO:tasks.workunit.client.0.smithi067.stdout:7/33: dread f1 [0,4194304] 0 2024-04-22T23:23:19.171 INFO:tasks.workunit.client.0.smithi067.stdout:0/75: sync 2024-04-22T23:23:19.174 INFO:tasks.workunit.client.0.smithi067.stdout:0/76: dread db/de/f14 [0,4194304] 0 2024-04-22T23:23:19.176 INFO:tasks.workunit.client.0.smithi067.stdout:0/77: write db/de/f12 [2984781,10168] 0 2024-04-22T23:23:19.180 INFO:tasks.workunit.client.0.smithi067.stdout:0/78: dread db/de/f12 [0,4194304] 0 2024-04-22T23:23:19.180 INFO:tasks.workunit.client.0.smithi067.stdout:0/79: chown f9 1344 1 2024-04-22T23:23:19.198 INFO:tasks.workunit.client.0.smithi067.stdout:7/34: creat f5 x:0 0 0 2024-04-22T23:23:19.199 INFO:tasks.workunit.client.0.smithi067.stdout:7/35: write f4 [1025765,67847] 0 2024-04-22T23:23:19.200 INFO:tasks.workunit.client.0.smithi067.stdout:0/80: rmdir db/de/d11 39 2024-04-22T23:23:19.202 INFO:tasks.workunit.client.0.smithi067.stdout:7/36: creat f6 x:0 0 0 2024-04-22T23:23:19.203 INFO:tasks.workunit.client.0.smithi067.stdout:0/81: rmdir db/de 39 2024-04-22T23:23:19.206 INFO:tasks.workunit.client.0.smithi067.stdout:7/37: symlink l7 0 2024-04-22T23:23:19.209 INFO:tasks.workunit.client.0.smithi067.stdout:0/82: symlink db/l18 0 2024-04-22T23:23:19.213 INFO:tasks.workunit.client.0.smithi067.stdout:7/38: dwrite f4 [0,4194304] 0 2024-04-22T23:23:19.217 INFO:tasks.workunit.client.0.smithi067.stdout:7/39: symlink l8 0 2024-04-22T23:23:19.217 INFO:tasks.workunit.client.0.smithi067.stdout:7/40: chown f6 16024 1 2024-04-22T23:23:19.217 INFO:tasks.workunit.client.0.smithi067.stdout:0/83: dread db/fd [0,4194304] 0 2024-04-22T23:23:19.217 INFO:tasks.workunit.client.0.smithi067.stdout:7/41: rmdir - no directory 2024-04-22T23:23:19.220 INFO:tasks.workunit.client.0.smithi067.stdout:8/103: sync 2024-04-22T23:23:19.220 INFO:tasks.workunit.client.0.smithi067.stdout:5/105: sync 2024-04-22T23:23:19.220 INFO:tasks.workunit.client.0.smithi067.stdout:6/46: sync 2024-04-22T23:23:19.220 INFO:tasks.workunit.client.0.smithi067.stdout:9/68: sync 2024-04-22T23:23:19.220 INFO:tasks.workunit.client.0.smithi067.stdout:4/94: sync 2024-04-22T23:23:19.220 INFO:tasks.workunit.client.0.smithi067.stdout:3/165: sync 2024-04-22T23:23:19.220 INFO:tasks.workunit.client.0.smithi067.stdout:2/74: sync 2024-04-22T23:23:19.220 INFO:tasks.workunit.client.0.smithi067.stdout:1/114: sync 2024-04-22T23:23:19.221 INFO:tasks.workunit.client.0.smithi067.stdout:0/84: rename db to db/d19 22 2024-04-22T23:23:19.223 INFO:tasks.workunit.client.0.smithi067.stdout:7/42: link f1 f9 0 2024-04-22T23:23:19.225 INFO:tasks.workunit.client.0.smithi067.stdout:6/47: creat ff x:0 0 0 2024-04-22T23:23:19.226 INFO:tasks.workunit.client.0.smithi067.stdout:7/43: write f5 [357703,79440] 0 2024-04-22T23:23:19.226 INFO:tasks.workunit.client.0.smithi067.stdout:9/69: readlink d5/dd/l13 0 2024-04-22T23:23:19.228 INFO:tasks.workunit.client.0.smithi067.stdout:7/44: readlink l8 0 2024-04-22T23:23:19.230 INFO:tasks.workunit.client.0.smithi067.stdout:3/166: dread d3/d4/f24 [0,4194304] 0 2024-04-22T23:23:19.232 INFO:tasks.workunit.client.0.smithi067.stdout:2/75: dread d1/f8 [0,4194304] 0 2024-04-22T23:23:19.232 INFO:tasks.workunit.client.0.smithi067.stdout:5/106: dread d5/fa [0,4194304] 0 2024-04-22T23:23:19.236 INFO:tasks.workunit.client.0.smithi067.stdout:4/95: symlink d12/d17/d1e/l1f 0 2024-04-22T23:23:19.238 INFO:tasks.workunit.client.0.smithi067.stdout:7/45: dwrite f4 [0,4194304] 0 2024-04-22T23:23:19.239 INFO:tasks.workunit.client.0.smithi067.stdout:7/46: dread - f6 zero size 2024-04-22T23:23:19.239 INFO:tasks.workunit.client.0.smithi067.stdout:9/70: symlink d5/dd/l14 0 2024-04-22T23:23:19.246 INFO:tasks.workunit.client.0.smithi067.stdout:8/104: creat d1/d3/d27/f28 x:0 0 0 2024-04-22T23:23:19.248 INFO:tasks.workunit.client.0.smithi067.stdout:5/107: creat d5/d14/f26 x:0 0 0 2024-04-22T23:23:19.251 INFO:tasks.workunit.client.0.smithi067.stdout:8/105: dread d1/d3/f1b [0,4194304] 0 2024-04-22T23:23:19.252 INFO:tasks.workunit.client.0.smithi067.stdout:4/96: creat d12/d17/d1e/f20 x:0 0 0 2024-04-22T23:23:19.252 INFO:tasks.workunit.client.0.smithi067.stdout:8/106: stat d1/l8 0 2024-04-22T23:23:19.253 INFO:tasks.workunit.client.0.smithi067.stdout:7/47: creat fa x:0 0 0 2024-04-22T23:23:19.254 INFO:tasks.workunit.client.0.smithi067.stdout:9/71: symlink d5/l15 0 2024-04-22T23:23:19.256 INFO:tasks.workunit.client.0.smithi067.stdout:9/72: fdatasync d5/f9 0 2024-04-22T23:23:19.256 INFO:tasks.workunit.client.0.smithi067.stdout:1/115: creat d8/d9/f25 x:0 0 0 2024-04-22T23:23:19.259 INFO:tasks.workunit.client.0.smithi067.stdout:2/76: link d1/fc d1/d12/f15 0 2024-04-22T23:23:19.261 INFO:tasks.workunit.client.0.smithi067.stdout:2/77: chown d1/f8 30928 1 2024-04-22T23:23:19.262 INFO:tasks.workunit.client.0.smithi067.stdout:8/107: creat d1/d3/d10/f29 x:0 0 0 2024-04-22T23:23:19.262 INFO:tasks.workunit.client.0.smithi067.stdout:4/97: dwrite d12/f16 [0,4194304] 0 2024-04-22T23:23:19.263 INFO:tasks.workunit.client.0.smithi067.stdout:7/48: rename fa to fb 0 2024-04-22T23:23:19.265 INFO:tasks.workunit.client.0.smithi067.stdout:9/73: dread d5/f6 [0,4194304] 0 2024-04-22T23:23:19.267 INFO:tasks.workunit.client.0.smithi067.stdout:8/108: dread d1/f6 [0,4194304] 0 2024-04-22T23:23:19.272 INFO:tasks.workunit.client.0.smithi067.stdout:2/78: creat d1/d12/f16 x:0 0 0 2024-04-22T23:23:19.275 INFO:tasks.workunit.client.0.smithi067.stdout:7/49: creat fc x:0 0 0 2024-04-22T23:23:19.276 INFO:tasks.workunit.client.0.smithi067.stdout:7/50: chown f6 26094 1 2024-04-22T23:23:19.277 INFO:tasks.workunit.client.0.smithi067.stdout:2/79: dread d1/f4 [0,4194304] 0 2024-04-22T23:23:19.283 INFO:tasks.workunit.client.0.smithi067.stdout:7/51: creat fd x:0 0 0 2024-04-22T23:23:19.284 INFO:tasks.workunit.client.0.smithi067.stdout:2/80: mknod d1/c17 0 2024-04-22T23:23:19.288 INFO:tasks.workunit.client.0.smithi067.stdout:1/116: rename d8/d9/df/f21 to d8/f26 0 2024-04-22T23:23:19.291 INFO:tasks.workunit.client.0.smithi067.stdout:2/81: dwrite d1/f3 [4194304,4194304] 0 2024-04-22T23:23:19.293 INFO:tasks.workunit.client.0.smithi067.stdout:8/109: link d1/dd/f20 d1/d3/f2a 0 2024-04-22T23:23:19.295 INFO:tasks.workunit.client.0.smithi067.stdout:1/117: symlink d8/d12/l27 0 2024-04-22T23:23:19.299 INFO:tasks.workunit.client.0.smithi067.stdout:2/82: write d1/d12/f15 [141173,76642] 0 2024-04-22T23:23:19.299 INFO:tasks.workunit.client.0.smithi067.stdout:7/52: dwrite f1 [0,4194304] 0 2024-04-22T23:23:19.300 INFO:tasks.workunit.client.0.smithi067.stdout:8/110: read d1/f6 [2036728,2193] 0 2024-04-22T23:23:19.302 INFO:tasks.workunit.client.0.smithi067.stdout:7/53: stat f6 0 2024-04-22T23:23:19.303 INFO:tasks.workunit.client.0.smithi067.stdout:8/111: dread - d1/d3/d10/f29 zero size 2024-04-22T23:23:19.303 INFO:tasks.workunit.client.0.smithi067.stdout:7/54: write fd [373861,11068] 0 2024-04-22T23:23:19.304 INFO:tasks.workunit.client.0.smithi067.stdout:8/112: stat d1/f6 0 2024-04-22T23:23:19.305 INFO:tasks.workunit.client.0.smithi067.stdout:7/55: write f1 [3094526,52311] 0 2024-04-22T23:23:19.306 INFO:tasks.workunit.client.0.smithi067.stdout:1/118: dread d8/d9/f1b [0,4194304] 0 2024-04-22T23:23:19.307 INFO:tasks.workunit.client.0.smithi067.stdout:8/113: truncate d1/d3/d27/f28 785063 0 2024-04-22T23:23:19.308 INFO:tasks.workunit.client.0.smithi067.stdout:7/56: rename f9 to fe 0 2024-04-22T23:23:19.309 INFO:tasks.workunit.client.0.smithi067.stdout:7/57: chown f5 704093 1 2024-04-22T23:23:19.309 INFO:tasks.workunit.client.0.smithi067.stdout:7/58: stat fd 0 2024-04-22T23:23:19.313 INFO:tasks.workunit.client.0.smithi067.stdout:2/83: dread d1/f9 [0,4194304] 0 2024-04-22T23:23:19.313 INFO:tasks.workunit.client.0.smithi067.stdout:1/119: rename l6 to d8/d9/df/d1c/l28 0 2024-04-22T23:23:19.313 INFO:tasks.workunit.client.0.smithi067.stdout:2/84: readlink d1/d12/l13 0 2024-04-22T23:23:19.314 INFO:tasks.workunit.client.0.smithi067.stdout:8/114: mknod d1/dd/c2b 0 2024-04-22T23:23:19.315 INFO:tasks.workunit.client.0.smithi067.stdout:7/59: symlink lf 0 2024-04-22T23:23:19.319 INFO:tasks.workunit.client.0.smithi067.stdout:8/115: symlink d1/dd/d26/l2c 0 2024-04-22T23:23:19.319 INFO:tasks.workunit.client.0.smithi067.stdout:8/116: chown d1/d3 5159371 1 2024-04-22T23:23:19.320 INFO:tasks.workunit.client.0.smithi067.stdout:8/117: read - d1/d3/d10/f29 zero size 2024-04-22T23:23:19.321 INFO:tasks.workunit.client.0.smithi067.stdout:7/60: unlink lf 0 2024-04-22T23:23:19.323 INFO:tasks.workunit.client.0.smithi067.stdout:6/48: getdents . 0 2024-04-22T23:23:19.326 INFO:tasks.workunit.client.0.smithi067.stdout:8/118: dread d1/d3/f13 [0,4194304] 0 2024-04-22T23:23:19.326 INFO:tasks.workunit.client.0.smithi067.stdout:7/61: dread f4 [0,4194304] 0 2024-04-22T23:23:19.328 INFO:tasks.workunit.client.0.smithi067.stdout:7/62: mkdir d10 0 2024-04-22T23:23:19.330 INFO:tasks.workunit.client.0.smithi067.stdout:7/63: truncate fc 789044 0 2024-04-22T23:23:19.330 INFO:tasks.workunit.client.0.smithi067.stdout:8/119: chown d1/d3/d10/c12 7 1 2024-04-22T23:23:19.331 INFO:tasks.workunit.client.0.smithi067.stdout:6/49: link f6 f10 0 2024-04-22T23:23:19.331 INFO:tasks.workunit.client.0.smithi067.stdout:4/98: rmdir d12/d17/d1e 39 2024-04-22T23:23:19.331 INFO:tasks.workunit.client.0.smithi067.stdout:7/64: chown fe 2684 1 2024-04-22T23:23:19.333 INFO:tasks.workunit.client.0.smithi067.stdout:4/99: readlink d12/l1c 0 2024-04-22T23:23:19.333 INFO:tasks.workunit.client.0.smithi067.stdout:0/85: dwrite db/de/ff [0,4194304] 0 2024-04-22T23:23:19.334 INFO:tasks.workunit.client.0.smithi067.stdout:8/120: dread d1/f18 [0,4194304] 0 2024-04-22T23:23:19.335 INFO:tasks.workunit.client.0.smithi067.stdout:8/121: dread - d1/dd/d26/f22 zero size 2024-04-22T23:23:19.338 INFO:tasks.workunit.client.0.smithi067.stdout:4/100: rmdir d12 39 2024-04-22T23:23:19.344 INFO:tasks.workunit.client.0.smithi067.stdout:8/122: mknod d1/c2d 0 2024-04-22T23:23:19.345 INFO:tasks.workunit.client.0.smithi067.stdout:7/65: mkdir d10/d11 0 2024-04-22T23:23:19.347 INFO:tasks.workunit.client.0.smithi067.stdout:0/86: mkdir db/de/d11/d1a 0 2024-04-22T23:23:19.350 INFO:tasks.workunit.client.0.smithi067.stdout:7/66: creat d10/f12 x:0 0 0 2024-04-22T23:23:19.351 INFO:tasks.workunit.client.0.smithi067.stdout:6/50: dread f6 [0,4194304] 0 2024-04-22T23:23:19.352 INFO:tasks.workunit.client.0.smithi067.stdout:4/101: getdents d12/d17/d1d 0 2024-04-22T23:23:19.353 INFO:tasks.workunit.client.0.smithi067.stdout:7/67: truncate fe 4309913 0 2024-04-22T23:23:19.353 INFO:tasks.workunit.client.0.smithi067.stdout:6/51: readlink la 0 2024-04-22T23:23:19.354 INFO:tasks.workunit.client.0.smithi067.stdout:6/52: readlink l3 0 2024-04-22T23:23:19.355 INFO:tasks.workunit.client.0.smithi067.stdout:6/53: dread - f8 zero size 2024-04-22T23:23:19.355 INFO:tasks.workunit.client.0.smithi067.stdout:9/74: dwrite d5/fb [4194304,4194304] 0 2024-04-22T23:23:19.358 INFO:tasks.workunit.client.0.smithi067.stdout:6/54: symlink l11 0 2024-04-22T23:23:19.360 INFO:tasks.workunit.client.0.smithi067.stdout:4/102: symlink d12/d17/l21 0 2024-04-22T23:23:19.451 INFO:tasks.workunit.client.0.smithi067.stdout:7/68: rmdir d10/d11 0 2024-04-22T23:23:19.461 INFO:tasks.workunit.client.0.smithi067.stdout:1/120: dwrite d8/d9/fd [0,4194304] 0 2024-04-22T23:23:19.462 INFO:tasks.workunit.client.0.smithi067.stdout:4/103: dwrite d12/d17/d1e/f20 [0,4194304] 0 2024-04-22T23:23:19.463 INFO:tasks.workunit.client.0.smithi067.stdout:0/87: write db/de/f12 [15479,44800] 0 2024-04-22T23:23:19.463 INFO:tasks.workunit.client.0.smithi067.stdout:1/121: dread - d8/f15 zero size 2024-04-22T23:23:19.464 INFO:tasks.workunit.client.0.smithi067.stdout:9/75: write d5/f6 [3296371,72903] 0 2024-04-22T23:23:19.465 INFO:tasks.workunit.client.0.smithi067.stdout:1/122: chown d8/d9/l16 1 1 2024-04-22T23:23:19.465 INFO:tasks.workunit.client.0.smithi067.stdout:7/69: rmdir d10 39 2024-04-22T23:23:19.467 INFO:tasks.workunit.client.0.smithi067.stdout:1/123: write d8/f1d [122349,49828] 0 2024-04-22T23:23:19.468 INFO:tasks.workunit.client.0.smithi067.stdout:6/55: write f10 [1688207,112681] 0 2024-04-22T23:23:19.471 INFO:tasks.workunit.client.0.smithi067.stdout:9/76: unlink d5/f6 0 2024-04-22T23:23:19.472 INFO:tasks.workunit.client.0.smithi067.stdout:4/104: symlink d12/d17/d1d/l22 0 2024-04-22T23:23:19.474 INFO:tasks.workunit.client.0.smithi067.stdout:7/70: creat d10/f13 x:0 0 0 2024-04-22T23:23:19.475 INFO:tasks.workunit.client.0.smithi067.stdout:6/56: rename fb to f12 0 2024-04-22T23:23:19.476 INFO:tasks.workunit.client.0.smithi067.stdout:9/77: mknod d5/dd/c16 0 2024-04-22T23:23:19.478 INFO:tasks.workunit.client.0.smithi067.stdout:0/88: creat db/de/f1b x:0 0 0 2024-04-22T23:23:19.478 INFO:tasks.workunit.client.0.smithi067.stdout:4/105: mknod d12/d17/d1e/c23 0 2024-04-22T23:23:19.479 INFO:tasks.workunit.client.0.smithi067.stdout:3/167: sync 2024-04-22T23:23:19.480 INFO:tasks.workunit.client.0.smithi067.stdout:4/106: write fd [1866943,77462] 0 2024-04-22T23:23:19.481 INFO:tasks.workunit.client.0.smithi067.stdout:3/168: stat d3/d4/d10/d1f/c22 0 2024-04-22T23:23:19.482 INFO:tasks.workunit.client.0.smithi067.stdout:1/124: creat d8/d9/f29 x:0 0 0 2024-04-22T23:23:19.488 INFO:tasks.workunit.client.0.smithi067.stdout:4/107: symlink d12/d17/l24 0 2024-04-22T23:23:19.489 INFO:tasks.workunit.client.0.smithi067.stdout:1/125: mkdir d8/d9/d2a 0 2024-04-22T23:23:19.491 INFO:tasks.workunit.client.0.smithi067.stdout:0/89: dwrite f8 [0,4194304] 0 2024-04-22T23:23:19.493 INFO:tasks.workunit.client.0.smithi067.stdout:1/126: dread - d8/f15 zero size 2024-04-22T23:23:19.494 INFO:tasks.workunit.client.0.smithi067.stdout:6/57: getdents . 0 2024-04-22T23:23:19.496 INFO:tasks.workunit.client.0.smithi067.stdout:4/108: unlink d12/d17/d1d/l22 0 2024-04-22T23:23:19.497 INFO:tasks.workunit.client.0.smithi067.stdout:7/71: dwrite fb [0,4194304] 0 2024-04-22T23:23:19.500 INFO:tasks.workunit.client.0.smithi067.stdout:3/169: dwrite d3/f35 [0,4194304] 0 2024-04-22T23:23:19.503 INFO:tasks.workunit.client.0.smithi067.stdout:6/58: unlink f0 0 2024-04-22T23:23:19.506 INFO:tasks.workunit.client.0.smithi067.stdout:7/72: dread fb [0,4194304] 0 2024-04-22T23:23:19.508 INFO:tasks.workunit.client.0.smithi067.stdout:4/109: creat d12/d17/d1e/f25 x:0 0 0 2024-04-22T23:23:19.511 INFO:tasks.workunit.client.0.smithi067.stdout:6/59: creat f13 x:0 0 0 2024-04-22T23:23:19.512 INFO:tasks.workunit.client.0.smithi067.stdout:9/78: truncate d5/fb 7201343 0 2024-04-22T23:23:19.513 INFO:tasks.workunit.client.0.smithi067.stdout:3/170: symlink d3/d4/d10/d19/l36 0 2024-04-22T23:23:19.513 INFO:tasks.workunit.client.0.smithi067.stdout:3/171: readlink d3/d4/d12/l13 0 2024-04-22T23:23:19.514 INFO:tasks.workunit.client.0.smithi067.stdout:3/172: fdatasync d3/d4/d12/f2f 0 2024-04-22T23:23:19.515 INFO:tasks.workunit.client.0.smithi067.stdout:7/73: mkdir d10/d14 0 2024-04-22T23:23:19.517 INFO:tasks.workunit.client.0.smithi067.stdout:0/90: rename f8 to db/de/f1c 0 2024-04-22T23:23:19.518 INFO:tasks.workunit.client.0.smithi067.stdout:0/91: write db/de/f1b [406389,58871] 0 2024-04-22T23:23:19.521 INFO:tasks.workunit.client.0.smithi067.stdout:5/108: sync 2024-04-22T23:23:19.521 INFO:tasks.workunit.client.0.smithi067.stdout:2/85: sync 2024-04-22T23:23:19.523 INFO:tasks.workunit.client.0.smithi067.stdout:7/74: creat d10/f15 x:0 0 0 2024-04-22T23:23:19.528 INFO:tasks.workunit.client.0.smithi067.stdout:0/92: dwrite db/de/f1b [0,4194304] 0 2024-04-22T23:23:19.528 INFO:tasks.workunit.client.0.smithi067.stdout:2/86: mknod d1/c18 0 2024-04-22T23:23:19.530 INFO:tasks.workunit.client.0.smithi067.stdout:5/109: dread d5/fa [0,4194304] 0 2024-04-22T23:23:19.532 INFO:tasks.workunit.client.0.smithi067.stdout:1/127: rename d8/d9/df/d1c/l28 to d8/d24/l2b 0 2024-04-22T23:23:19.533 INFO:tasks.workunit.client.0.smithi067.stdout:6/60: link l11 l14 0 2024-04-22T23:23:19.533 INFO:tasks.workunit.client.0.smithi067.stdout:5/110: rename d5 to d5/db/d27 22 2024-04-22T23:23:19.533 INFO:tasks.workunit.client.0.smithi067.stdout:6/61: fsync f6 0 2024-04-22T23:23:19.535 INFO:tasks.workunit.client.0.smithi067.stdout:2/87: mkdir d1/d19 0 2024-04-22T23:23:19.539 INFO:tasks.workunit.client.0.smithi067.stdout:7/75: symlink d10/d14/l16 0 2024-04-22T23:23:19.541 INFO:tasks.workunit.client.0.smithi067.stdout:7/76: write f6 [812761,100822] 0 2024-04-22T23:23:19.543 INFO:tasks.workunit.client.0.smithi067.stdout:5/111: dwrite d5/f15 [0,4194304] 0 2024-04-22T23:23:19.543 INFO:tasks.workunit.client.0.smithi067.stdout:4/110: getdents d12/d17/d1d 0 2024-04-22T23:23:19.547 INFO:tasks.workunit.client.0.smithi067.stdout:7/77: dread fe [0,4194304] 0 2024-04-22T23:23:19.548 INFO:tasks.workunit.client.0.smithi067.stdout:6/62: write fe [457731,94513] 0 2024-04-22T23:23:19.549 INFO:tasks.workunit.client.0.smithi067.stdout:6/63: readlink la 0 2024-04-22T23:23:19.549 INFO:tasks.workunit.client.0.smithi067.stdout:6/64: chown f6 87523 1 2024-04-22T23:23:19.550 INFO:tasks.workunit.client.0.smithi067.stdout:2/88: rename d1/lf to d1/d12/l1a 0 2024-04-22T23:23:19.552 INFO:tasks.workunit.client.0.smithi067.stdout:3/173: getdents d3/d4/d10/d1f 0 2024-04-22T23:23:19.558 INFO:tasks.workunit.client.0.smithi067.stdout:0/93: symlink db/de/d11/d1a/l1d 0 2024-04-22T23:23:19.558 INFO:tasks.workunit.client.0.smithi067.stdout:5/112: rmdir d5/db/d12 39 2024-04-22T23:23:19.560 INFO:tasks.workunit.client.0.smithi067.stdout:6/65: symlink l15 0 2024-04-22T23:23:19.561 INFO:tasks.workunit.client.0.smithi067.stdout:8/123: sync 2024-04-22T23:23:19.562 INFO:tasks.workunit.client.0.smithi067.stdout:2/89: dwrite d1/f3 [8388608,4194304] 0 2024-04-22T23:23:19.564 INFO:tasks.workunit.client.0.smithi067.stdout:1/128: creat d8/d9/f2c x:0 0 0 2024-04-22T23:23:19.565 INFO:tasks.workunit.client.0.smithi067.stdout:3/174: dwrite d3/d4/d10/d19/f21 [0,4194304] 0 2024-04-22T23:23:19.567 INFO:tasks.workunit.client.0.smithi067.stdout:0/94: symlink db/de/d11/l1e 0 2024-04-22T23:23:19.570 INFO:tasks.workunit.client.0.smithi067.stdout:6/66: symlink l16 0 2024-04-22T23:23:19.570 INFO:tasks.workunit.client.0.smithi067.stdout:5/113: dread d5/fa [0,4194304] 0 2024-04-22T23:23:19.572 INFO:tasks.workunit.client.0.smithi067.stdout:8/124: stat d1/l21 0 2024-04-22T23:23:19.572 INFO:tasks.workunit.client.0.smithi067.stdout:6/67: fsync ff 0 2024-04-22T23:23:19.572 INFO:tasks.workunit.client.0.smithi067.stdout:8/125: chown d1/d3/ce 14 1 2024-04-22T23:23:19.576 INFO:tasks.workunit.client.0.smithi067.stdout:6/68: dread f12 [0,4194304] 0 2024-04-22T23:23:19.576 INFO:tasks.workunit.client.0.smithi067.stdout:4/111: rename d12/d17/l24 to d12/d17/l26 0 2024-04-22T23:23:19.577 INFO:tasks.workunit.client.0.smithi067.stdout:6/69: write fc [2031413,73122] 0 2024-04-22T23:23:19.578 INFO:tasks.workunit.client.0.smithi067.stdout:4/112: chown d12/d17/d1e/f25 55 1 2024-04-22T23:23:19.581 INFO:tasks.workunit.client.0.smithi067.stdout:1/129: mkdir d8/d9/d2d 0 2024-04-22T23:23:19.584 INFO:tasks.workunit.client.0.smithi067.stdout:9/79: truncate d5/fb 3803411 0 2024-04-22T23:23:19.585 INFO:tasks.workunit.client.0.smithi067.stdout:9/80: write d5/f10 [143371,107123] 0 2024-04-22T23:23:19.586 INFO:tasks.workunit.client.0.smithi067.stdout:9/81: chown d5/f9 2 1 2024-04-22T23:23:19.586 INFO:tasks.workunit.client.0.smithi067.stdout:0/95: symlink db/l1f 0 2024-04-22T23:23:19.587 INFO:tasks.workunit.client.0.smithi067.stdout:5/114: mkdir d5/d28 0 2024-04-22T23:23:19.588 INFO:tasks.workunit.client.0.smithi067.stdout:7/78: truncate f4 2898732 0 2024-04-22T23:23:19.590 INFO:tasks.workunit.client.0.smithi067.stdout:6/70: mkdir d17 0 2024-04-22T23:23:19.591 INFO:tasks.workunit.client.0.smithi067.stdout:5/115: dread d5/f11 [0,4194304] 0 2024-04-22T23:23:19.594 INFO:tasks.workunit.client.0.smithi067.stdout:1/130: rename l2 to d8/d9/df/l2e 0 2024-04-22T23:23:19.595 INFO:tasks.workunit.client.0.smithi067.stdout:3/175: creat d3/d4/f37 x:0 0 0 2024-04-22T23:23:19.596 INFO:tasks.workunit.client.0.smithi067.stdout:8/126: write d1/f18 [772485,23796] 0 2024-04-22T23:23:19.599 INFO:tasks.workunit.client.0.smithi067.stdout:9/82: symlink d5/l17 0 2024-04-22T23:23:19.599 INFO:tasks.workunit.client.0.smithi067.stdout:3/176: dread d3/d4/f20 [0,4194304] 0 2024-04-22T23:23:19.599 INFO:tasks.workunit.client.0.smithi067.stdout:7/79: write fb [2347888,73950] 0 2024-04-22T23:23:19.599 INFO:tasks.workunit.client.0.smithi067.stdout:2/90: truncate d1/f3 12347743 0 2024-04-22T23:23:19.601 INFO:tasks.workunit.client.0.smithi067.stdout:3/177: dread - d3/d4/d12/f27 zero size 2024-04-22T23:23:19.602 INFO:tasks.workunit.client.0.smithi067.stdout:5/116: rmdir d5/db 39 2024-04-22T23:23:19.603 INFO:tasks.workunit.client.0.smithi067.stdout:8/127: readlink d1/d3/l1d 0 2024-04-22T23:23:19.604 INFO:tasks.workunit.client.0.smithi067.stdout:9/83: mkdir d5/dd/d18 0 2024-04-22T23:23:19.605 INFO:tasks.workunit.client.0.smithi067.stdout:3/178: dread d3/f35 [0,4194304] 0 2024-04-22T23:23:19.607 INFO:tasks.workunit.client.0.smithi067.stdout:6/71: symlink d17/l18 0 2024-04-22T23:23:19.611 INFO:tasks.workunit.client.0.smithi067.stdout:4/113: rmdir d12/d17/d1d 0 2024-04-22T23:23:19.612 INFO:tasks.workunit.client.0.smithi067.stdout:4/114: chown f5 52 1 2024-04-22T23:23:19.614 INFO:tasks.workunit.client.0.smithi067.stdout:4/115: dread - d12/f19 zero size 2024-04-22T23:23:19.614 INFO:tasks.workunit.client.0.smithi067.stdout:6/72: dwrite ff [0,4194304] 0 2024-04-22T23:23:19.615 INFO:tasks.workunit.client.0.smithi067.stdout:5/117: truncate d5/db/ff 915481 0 2024-04-22T23:23:19.618 INFO:tasks.workunit.client.0.smithi067.stdout:3/179: truncate d3/d4/f24 1317089 0 2024-04-22T23:23:19.621 INFO:tasks.workunit.client.0.smithi067.stdout:9/84: rename c4 to d5/dd/d18/c19 0 2024-04-22T23:23:19.622 INFO:tasks.workunit.client.0.smithi067.stdout:9/85: write d5/f10 [639905,129881] 0 2024-04-22T23:23:19.622 INFO:tasks.workunit.client.0.smithi067.stdout:8/128: dread d1/dd/f20 [0,4194304] 0 2024-04-22T23:23:19.624 INFO:tasks.workunit.client.0.smithi067.stdout:8/129: write d1/f7 [606171,28352] 0 2024-04-22T23:23:19.625 INFO:tasks.workunit.client.0.smithi067.stdout:8/130: stat d1/dd/d26/l2c 0 2024-04-22T23:23:19.626 INFO:tasks.workunit.client.0.smithi067.stdout:0/96: getdents db 0 2024-04-22T23:23:19.628 INFO:tasks.workunit.client.0.smithi067.stdout:4/116: creat d12/d17/d1e/f27 x:0 0 0 2024-04-22T23:23:19.629 INFO:tasks.workunit.client.0.smithi067.stdout:6/73: mkdir d17/d19 0 2024-04-22T23:23:19.631 INFO:tasks.workunit.client.0.smithi067.stdout:5/118: truncate d5/db/d12/f22 573681 0 2024-04-22T23:23:19.632 INFO:tasks.workunit.client.0.smithi067.stdout:4/117: dread fd [0,4194304] 0 2024-04-22T23:23:19.632 INFO:tasks.workunit.client.0.smithi067.stdout:5/119: chown d5/db/d12 5 1 2024-04-22T23:23:19.633 INFO:tasks.workunit.client.0.smithi067.stdout:3/180: mkdir d3/d4/d10/d38 0 2024-04-22T23:23:19.634 INFO:tasks.workunit.client.0.smithi067.stdout:2/91: write d1/f9 [1372995,13275] 0 2024-04-22T23:23:19.635 INFO:tasks.workunit.client.0.smithi067.stdout:4/118: write d12/d17/d1e/f25 [902995,103760] 0 2024-04-22T23:23:19.635 INFO:tasks.workunit.client.0.smithi067.stdout:2/92: chown d1/d10 0 1 2024-04-22T23:23:19.637 INFO:tasks.workunit.client.0.smithi067.stdout:8/131: fdatasync d1/d3/f2a 0 2024-04-22T23:23:19.639 INFO:tasks.workunit.client.0.smithi067.stdout:5/120: dread d5/d14/f20 [0,4194304] 0 2024-04-22T23:23:19.640 INFO:tasks.workunit.client.0.smithi067.stdout:8/132: dread d1/d3/d27/f28 [0,4194304] 0 2024-04-22T23:23:19.643 INFO:tasks.workunit.client.0.smithi067.stdout:6/74: creat d17/f1a x:0 0 0 2024-04-22T23:23:19.643 INFO:tasks.workunit.client.0.smithi067.stdout:5/121: dread d5/db/ff [0,4194304] 0 2024-04-22T23:23:19.644 INFO:tasks.workunit.client.0.smithi067.stdout:9/86: rmdir d5/dd 39 2024-04-22T23:23:19.646 INFO:tasks.workunit.client.0.smithi067.stdout:9/87: chown d5/l15 1522 1 2024-04-22T23:23:19.647 INFO:tasks.workunit.client.0.smithi067.stdout:0/97: dwrite db/de/f1c [0,4194304] 0 2024-04-22T23:23:19.648 INFO:tasks.workunit.client.0.smithi067.stdout:9/88: write d5/f10 [637559,87457] 0 2024-04-22T23:23:19.649 INFO:tasks.workunit.client.0.smithi067.stdout:2/93: creat d1/f1b x:0 0 0 2024-04-22T23:23:19.649 INFO:tasks.workunit.client.0.smithi067.stdout:9/89: stat d5/fa 0 2024-04-22T23:23:19.651 INFO:tasks.workunit.client.0.smithi067.stdout:5/122: dread d5/f15 [0,4194304] 0 2024-04-22T23:23:19.651 INFO:tasks.workunit.client.0.smithi067.stdout:4/119: rename d12/d17/d1e/f25 to d12/d17/d1e/f28 0 2024-04-22T23:23:19.655 INFO:tasks.workunit.client.0.smithi067.stdout:6/75: creat d17/f1b x:0 0 0 2024-04-22T23:23:19.659 INFO:tasks.workunit.client.0.smithi067.stdout:9/90: rmdir d5 39 2024-04-22T23:23:19.660 INFO:tasks.workunit.client.0.smithi067.stdout:5/123: chown l2 3096462 1 2024-04-22T23:23:19.661 INFO:tasks.workunit.client.0.smithi067.stdout:4/120: unlink d12/f15 0 2024-04-22T23:23:19.662 INFO:tasks.workunit.client.0.smithi067.stdout:4/121: dread - d12/f19 zero size 2024-04-22T23:23:19.663 INFO:tasks.workunit.client.0.smithi067.stdout:6/76: symlink d17/l1c 0 2024-04-22T23:23:19.665 INFO:tasks.workunit.client.0.smithi067.stdout:2/94: mkdir d1/d10/d1c 0 2024-04-22T23:23:19.666 INFO:tasks.workunit.client.0.smithi067.stdout:6/77: dread f10 [0,4194304] 0 2024-04-22T23:23:19.667 INFO:tasks.workunit.client.0.smithi067.stdout:6/78: chown fe 389938 1 2024-04-22T23:23:19.668 INFO:tasks.workunit.client.0.smithi067.stdout:2/95: dread d1/f6 [0,4194304] 0 2024-04-22T23:23:19.668 INFO:tasks.workunit.client.0.smithi067.stdout:5/124: rename d5/db/ce to d5/db/d12/c29 0 2024-04-22T23:23:19.669 INFO:tasks.workunit.client.0.smithi067.stdout:2/96: write d1/d12/f15 [461457,32852] 0 2024-04-22T23:23:19.672 INFO:tasks.workunit.client.0.smithi067.stdout:8/133: getdents d1/d3/d10 0 2024-04-22T23:23:19.676 INFO:tasks.workunit.client.0.smithi067.stdout:9/91: creat d5/dd/f1a x:0 0 0 2024-04-22T23:23:19.677 INFO:tasks.workunit.client.0.smithi067.stdout:8/134: rename d1/d3/d10/c12 to d1/d3/d10/c2e 0 2024-04-22T23:23:19.679 INFO:tasks.workunit.client.0.smithi067.stdout:6/79: rmdir d17/d19 0 2024-04-22T23:23:19.679 INFO:tasks.workunit.client.0.smithi067.stdout:8/135: readlink d1/dd/d26/l2c 0 2024-04-22T23:23:19.679 INFO:tasks.workunit.client.0.smithi067.stdout:5/125: dwrite d5/f15 [0,4194304] 0 2024-04-22T23:23:19.680 INFO:tasks.workunit.client.0.smithi067.stdout:2/97: dread d1/f4 [0,4194304] 0 2024-04-22T23:23:19.681 INFO:tasks.workunit.client.0.smithi067.stdout:2/98: stat d1/d19 0 2024-04-22T23:23:19.683 INFO:tasks.workunit.client.0.smithi067.stdout:8/136: creat d1/dd/f2f x:0 0 0 2024-04-22T23:23:19.683 INFO:tasks.workunit.client.0.smithi067.stdout:8/137: readlink d1/l21 0 2024-04-22T23:23:19.684 INFO:tasks.workunit.client.0.smithi067.stdout:5/126: mkdir d5/db/d12/d2a 0 2024-04-22T23:23:19.684 INFO:tasks.workunit.client.0.smithi067.stdout:6/80: dread fc [0,4194304] 0 2024-04-22T23:23:19.684 INFO:tasks.workunit.client.0.smithi067.stdout:8/138: dread - d1/dd/f2f zero size 2024-04-22T23:23:19.685 INFO:tasks.workunit.client.0.smithi067.stdout:2/99: unlink d1/f6 0 2024-04-22T23:23:19.687 INFO:tasks.workunit.client.0.smithi067.stdout:6/81: rename l3 to d17/l1d 0 2024-04-22T23:23:19.689 INFO:tasks.workunit.client.0.smithi067.stdout:5/127: creat d5/db/d12/f2b x:0 0 0 2024-04-22T23:23:19.691 INFO:tasks.workunit.client.0.smithi067.stdout:6/82: creat d17/f1e x:0 0 0 2024-04-22T23:23:19.692 INFO:tasks.workunit.client.0.smithi067.stdout:8/139: rename d1/d3/fb to d1/d3/d14/f30 0 2024-04-22T23:23:19.693 INFO:tasks.workunit.client.0.smithi067.stdout:8/140: chown d1/d3/f13 1296344 1 2024-04-22T23:23:19.694 INFO:tasks.workunit.client.0.smithi067.stdout:5/128: creat d5/db/d12/f2c x:0 0 0 2024-04-22T23:23:19.697 INFO:tasks.workunit.client.0.smithi067.stdout:8/141: mknod d1/c31 0 2024-04-22T23:23:19.698 INFO:tasks.workunit.client.0.smithi067.stdout:6/83: dread fe [0,4194304] 0 2024-04-22T23:23:19.698 INFO:tasks.workunit.client.0.smithi067.stdout:5/129: chown d5/l9 2 1 2024-04-22T23:23:19.699 INFO:tasks.workunit.client.0.smithi067.stdout:4/122: dread d12/d17/d1e/f28 [0,4194304] 0 2024-04-22T23:23:19.700 INFO:tasks.workunit.client.0.smithi067.stdout:2/100: getdents d1/d12 0 2024-04-22T23:23:19.701 INFO:tasks.workunit.client.0.smithi067.stdout:8/142: creat d1/d3/d14/f32 x:0 0 0 2024-04-22T23:23:19.703 INFO:tasks.workunit.client.0.smithi067.stdout:4/123: write d12/d17/d1e/f28 [1323920,81334] 0 2024-04-22T23:23:19.703 INFO:tasks.workunit.client.0.smithi067.stdout:5/130: mknod d5/db/d12/c2d 0 2024-04-22T23:23:19.705 INFO:tasks.workunit.client.0.smithi067.stdout:4/124: dread - d12/d17/d1e/f27 zero size 2024-04-22T23:23:19.708 INFO:tasks.workunit.client.0.smithi067.stdout:4/125: dread - d12/d17/d1e/f27 zero size 2024-04-22T23:23:19.708 INFO:tasks.workunit.client.0.smithi067.stdout:6/84: dwrite d17/f1e [0,4194304] 0 2024-04-22T23:23:19.709 INFO:tasks.workunit.client.0.smithi067.stdout:8/143: truncate d1/d3/f11 4129250 0 2024-04-22T23:23:19.709 INFO:tasks.workunit.client.0.smithi067.stdout:6/85: read f12 [1871120,2539] 0 2024-04-22T23:23:19.716 INFO:tasks.workunit.client.0.smithi067.stdout:6/86: rename ff to d17/f1f 0 2024-04-22T23:23:19.720 INFO:tasks.workunit.client.0.smithi067.stdout:2/101: dread d1/f3 [0,4194304] 0 2024-04-22T23:23:19.722 INFO:tasks.workunit.client.0.smithi067.stdout:2/102: getdents d1/d19 0 2024-04-22T23:23:19.725 INFO:tasks.workunit.client.0.smithi067.stdout:3/181: write d3/fa [1334256,112524] 0 2024-04-22T23:23:19.729 INFO:tasks.workunit.client.0.smithi067.stdout:3/182: creat d3/d4/d10/f39 x:0 0 0 2024-04-22T23:23:19.730 INFO:tasks.workunit.client.0.smithi067.stdout:0/98: truncate db/de/f1c 3141188 0 2024-04-22T23:23:19.730 INFO:tasks.workunit.client.0.smithi067.stdout:2/103: dwrite d1/f3 [0,4194304] 0 2024-04-22T23:23:19.731 INFO:tasks.workunit.client.0.smithi067.stdout:2/104: readlink d1/d12/l1a 0 2024-04-22T23:23:19.733 INFO:tasks.workunit.client.0.smithi067.stdout:2/105: write d1/fc [1343635,45492] 0 2024-04-22T23:23:19.733 INFO:tasks.workunit.client.0.smithi067.stdout:3/183: symlink d3/d4/d17/l3a 0 2024-04-22T23:23:19.734 INFO:tasks.workunit.client.0.smithi067.stdout:2/106: read - d1/d12/f16 zero size 2024-04-22T23:23:19.737 INFO:tasks.workunit.client.0.smithi067.stdout:2/107: truncate d1/f9 2154708 0 2024-04-22T23:23:19.738 INFO:tasks.workunit.client.0.smithi067.stdout:0/99: dread db/de/f12 [0,4194304] 0 2024-04-22T23:23:19.739 INFO:tasks.workunit.client.0.smithi067.stdout:3/184: creat d3/d4/d10/d19/f3b x:0 0 0 2024-04-22T23:23:19.741 INFO:tasks.workunit.client.0.smithi067.stdout:3/185: write d3/d4/d10/d19/f34 [393201,101361] 0 2024-04-22T23:23:19.741 INFO:tasks.workunit.client.0.smithi067.stdout:2/108: rename d1/f9 to d1/d12/f1d 0 2024-04-22T23:23:19.742 INFO:tasks.workunit.client.0.smithi067.stdout:2/109: chown d1/c14 251559 1 2024-04-22T23:23:19.745 INFO:tasks.workunit.client.0.smithi067.stdout:9/92: truncate d5/ff 2065801 0 2024-04-22T23:23:19.749 INFO:tasks.workunit.client.0.smithi067.stdout:9/93: chown d5/dd/d18/c19 69770533 1 2024-04-22T23:23:19.751 INFO:tasks.workunit.client.0.smithi067.stdout:0/100: rename db/de/f17 to db/de/f20 0 2024-04-22T23:23:19.752 INFO:tasks.workunit.client.0.smithi067.stdout:2/110: link d1/c14 d1/d10/d1c/c1e 0 2024-04-22T23:23:19.752 INFO:tasks.workunit.client.0.smithi067.stdout:2/111: chown d1 530979 1 2024-04-22T23:23:19.753 INFO:tasks.workunit.client.0.smithi067.stdout:2/112: chown d1/fc 68 1 2024-04-22T23:23:19.753 INFO:tasks.workunit.client.0.smithi067.stdout:2/113: chown d1/d19 14 1 2024-04-22T23:23:19.755 INFO:tasks.workunit.client.0.smithi067.stdout:0/101: mkdir db/d21 0 2024-04-22T23:23:19.757 INFO:tasks.workunit.client.0.smithi067.stdout:2/114: creat d1/f1f x:0 0 0 2024-04-22T23:23:19.760 INFO:tasks.workunit.client.0.smithi067.stdout:5/131: rmdir d5/db/d12 39 2024-04-22T23:23:19.761 INFO:tasks.workunit.client.0.smithi067.stdout:9/94: link d5/dd/d18/c19 d5/c1b 0 2024-04-22T23:23:19.763 INFO:tasks.workunit.client.0.smithi067.stdout:9/95: rename d5 to d5/dd/d1c 22 2024-04-22T23:23:19.764 INFO:tasks.workunit.client.0.smithi067.stdout:2/115: dwrite d1/d12/f15 [0,4194304] 0 2024-04-22T23:23:19.765 INFO:tasks.workunit.client.0.smithi067.stdout:5/132: creat d5/d1a/f2e x:0 0 0 2024-04-22T23:23:19.767 INFO:tasks.workunit.client.0.smithi067.stdout:9/96: creat d5/dd/d18/f1d x:0 0 0 2024-04-22T23:23:19.769 INFO:tasks.workunit.client.0.smithi067.stdout:9/97: chown d5/l17 194709945 1 2024-04-22T23:23:19.770 INFO:tasks.workunit.client.0.smithi067.stdout:9/98: read - d5/dd/d18/f1d zero size 2024-04-22T23:23:19.771 INFO:tasks.workunit.client.0.smithi067.stdout:4/126: truncate d12/f16 1134576 0 2024-04-22T23:23:19.771 INFO:tasks.workunit.client.0.smithi067.stdout:9/99: rename d5/dd/d18 to d5/dd/d18/d1e 22 2024-04-22T23:23:19.772 INFO:tasks.workunit.client.0.smithi067.stdout:2/116: symlink d1/l20 0 2024-04-22T23:23:19.772 INFO:tasks.workunit.client.0.smithi067.stdout:4/127: chown d12/d17 22305186 1 2024-04-22T23:23:19.774 INFO:tasks.workunit.client.0.smithi067.stdout:2/117: rmdir d1/d12 39 2024-04-22T23:23:19.777 INFO:tasks.workunit.client.0.smithi067.stdout:9/100: creat d5/dd/d18/f1f x:0 0 0 2024-04-22T23:23:19.780 INFO:tasks.workunit.client.0.smithi067.stdout:4/128: symlink d12/d17/l29 0 2024-04-22T23:23:19.787 INFO:tasks.workunit.client.0.smithi067.stdout:2/118: dwrite d1/d12/f16 [0,4194304] 0 2024-04-22T23:23:19.792 INFO:tasks.workunit.client.0.smithi067.stdout:2/119: dread d1/fc [0,4194304] 0 2024-04-22T23:23:19.796 INFO:tasks.workunit.client.0.smithi067.stdout:8/144: dread d1/d3/f11 [0,4194304] 0 2024-04-22T23:23:19.797 INFO:tasks.workunit.client.0.smithi067.stdout:8/145: dread - d1/d3/d14/f32 zero size 2024-04-22T23:23:19.878 INFO:tasks.workunit.client.0.smithi067.stdout:9/101: mkdir d5/d20 0 2024-04-22T23:23:19.878 INFO:tasks.workunit.client.0.smithi067.stdout:9/102: readlink d5/l17 0 2024-04-22T23:23:19.883 INFO:tasks.workunit.client.0.smithi067.stdout:8/146: creat d1/d3/d27/f33 x:0 0 0 2024-04-22T23:23:19.887 INFO:tasks.workunit.client.0.smithi067.stdout:5/133: write d5/d14/f20 [4733239,49293] 0 2024-04-22T23:23:19.887 INFO:tasks.workunit.client.0.smithi067.stdout:2/120: creat d1/d19/f21 x:0 0 0 2024-04-22T23:23:19.889 INFO:tasks.workunit.client.0.smithi067.stdout:8/147: creat d1/d3/d14/f34 x:0 0 0 2024-04-22T23:23:19.889 INFO:tasks.workunit.client.0.smithi067.stdout:8/148: fdatasync d1/f7 0 2024-04-22T23:23:19.896 INFO:tasks.workunit.client.0.smithi067.stdout:4/129: dwrite f8 [0,4194304] 0 2024-04-22T23:23:19.899 INFO:tasks.workunit.client.0.smithi067.stdout:9/103: rmdir d5/d20 0 2024-04-22T23:23:19.901 INFO:tasks.workunit.client.0.smithi067.stdout:9/104: write d5/f10 [1236651,117899] 0 2024-04-22T23:23:19.901 INFO:tasks.workunit.client.0.smithi067.stdout:0/102: creat db/de/f22 x:0 0 0 2024-04-22T23:23:19.901 INFO:tasks.workunit.client.0.smithi067.stdout:0/103: readlink la 0 2024-04-22T23:23:19.903 INFO:tasks.workunit.client.0.smithi067.stdout:4/130: creat d12/f2a x:0 0 0 2024-04-22T23:23:19.907 INFO:tasks.workunit.client.0.smithi067.stdout:5/134: creat d5/db/f2f x:0 0 0 2024-04-22T23:23:19.907 INFO:tasks.workunit.client.0.smithi067.stdout:8/149: rmdir d1/d3/d14 39 2024-04-22T23:23:19.909 INFO:tasks.workunit.client.0.smithi067.stdout:0/104: symlink db/de/d11/l23 0 2024-04-22T23:23:19.914 INFO:tasks.workunit.client.0.smithi067.stdout:2/121: dwrite d1/f4 [4194304,4194304] 0 2024-04-22T23:23:19.914 INFO:tasks.workunit.client.0.smithi067.stdout:8/150: write d1/d3/f13 [3779829,12405] 0 2024-04-22T23:23:19.916 INFO:tasks.workunit.client.0.smithi067.stdout:9/105: mkdir d5/d21 0 2024-04-22T23:23:19.917 INFO:tasks.workunit.client.0.smithi067.stdout:0/105: dread db/de/ff [0,4194304] 0 2024-04-22T23:23:19.921 INFO:tasks.workunit.client.0.smithi067.stdout:5/135: creat d5/db/d12/f30 x:0 0 0 2024-04-22T23:23:19.921 INFO:tasks.workunit.client.0.smithi067.stdout:2/122: dread d1/f8 [0,4194304] 0 2024-04-22T23:23:19.922 INFO:tasks.workunit.client.0.smithi067.stdout:0/106: mknod db/de/d11/d1a/c24 0 2024-04-22T23:23:19.923 INFO:tasks.workunit.client.0.smithi067.stdout:9/106: dread d5/f9 [0,4194304] 0 2024-04-22T23:23:19.924 INFO:tasks.workunit.client.0.smithi067.stdout:8/151: dread d1/f6 [0,4194304] 0 2024-04-22T23:23:19.925 INFO:tasks.workunit.client.0.smithi067.stdout:0/107: unlink f9 0 2024-04-22T23:23:19.925 INFO:tasks.workunit.client.0.smithi067.stdout:8/152: truncate d1/f18 1156659 0 2024-04-22T23:23:19.927 INFO:tasks.workunit.client.0.smithi067.stdout:5/136: dread d5/f11 [0,4194304] 0 2024-04-22T23:23:19.928 INFO:tasks.workunit.client.0.smithi067.stdout:9/107: symlink d5/dd/d18/l22 0 2024-04-22T23:23:19.929 INFO:tasks.workunit.client.0.smithi067.stdout:0/108: write db/de/f1c [992579,36402] 0 2024-04-22T23:23:19.929 INFO:tasks.workunit.client.0.smithi067.stdout:9/108: fdatasync d5/f9 0 2024-04-22T23:23:19.930 INFO:tasks.workunit.client.0.smithi067.stdout:8/153: rmdir d1/d3 39 2024-04-22T23:23:19.930 INFO:tasks.workunit.client.0.smithi067.stdout:2/123: dread d1/d12/f15 [0,4194304] 0 2024-04-22T23:23:19.931 INFO:tasks.workunit.client.0.smithi067.stdout:0/109: rename db to db/de/d11/d1a/d25 22 2024-04-22T23:23:19.932 INFO:tasks.workunit.client.0.smithi067.stdout:2/124: rmdir d1 39 2024-04-22T23:23:19.933 INFO:tasks.workunit.client.0.smithi067.stdout:8/154: truncate d1/f6 4111286 0 2024-04-22T23:23:19.934 INFO:tasks.workunit.client.0.smithi067.stdout:9/109: mknod d5/c23 0 2024-04-22T23:23:19.934 INFO:tasks.workunit.client.0.smithi067.stdout:9/110: fdatasync d5/f9 0 2024-04-22T23:23:19.936 INFO:tasks.workunit.client.0.smithi067.stdout:5/137: creat d5/f31 x:0 0 0 2024-04-22T23:23:19.937 INFO:tasks.workunit.client.0.smithi067.stdout:8/155: read - d1/d3/d27/f33 zero size 2024-04-22T23:23:19.938 INFO:tasks.workunit.client.0.smithi067.stdout:9/111: creat d5/d21/f24 x:0 0 0 2024-04-22T23:23:19.940 INFO:tasks.workunit.client.0.smithi067.stdout:8/156: symlink d1/dd/l35 0 2024-04-22T23:23:19.940 INFO:tasks.workunit.client.0.smithi067.stdout:0/110: dwrite db/de/f12 [0,4194304] 0 2024-04-22T23:23:19.941 INFO:tasks.workunit.client.0.smithi067.stdout:2/125: link d1/l20 d1/d10/l22 0 2024-04-22T23:23:19.941 INFO:tasks.workunit.client.0.smithi067.stdout:9/112: chown d5/fb 1621 1 2024-04-22T23:23:19.945 INFO:tasks.workunit.client.0.smithi067.stdout:1/131: sync 2024-04-22T23:23:19.954 INFO:tasks.workunit.client.0.smithi067.stdout:8/157: mknod d1/d3/d14/c36 0 2024-04-22T23:23:19.955 INFO:tasks.workunit.client.0.smithi067.stdout:2/126: write d1/d12/f16 [1567649,30264] 0 2024-04-22T23:23:19.957 INFO:tasks.workunit.client.0.smithi067.stdout:8/158: chown d1/d3/l1d 2 1 2024-04-22T23:23:19.957 INFO:tasks.workunit.client.0.smithi067.stdout:8/159: readlink d1/l8 0 2024-04-22T23:23:19.959 INFO:tasks.workunit.client.0.smithi067.stdout:1/132: mkdir d8/d9/d2f 0 2024-04-22T23:23:19.960 INFO:tasks.workunit.client.0.smithi067.stdout:2/127: chown d1/d10/l22 4042 1 2024-04-22T23:23:19.961 INFO:tasks.workunit.client.0.smithi067.stdout:8/160: rmdir d1/d3/d14 39 2024-04-22T23:23:19.962 INFO:tasks.workunit.client.0.smithi067.stdout:9/113: dwrite d5/d21/f24 [0,4194304] 0 2024-04-22T23:23:19.963 INFO:tasks.workunit.client.0.smithi067.stdout:1/133: mknod d8/d9/df/c30 0 2024-04-22T23:23:19.963 INFO:tasks.workunit.client.0.smithi067.stdout:0/111: dread f0 [0,4194304] 0 2024-04-22T23:23:19.964 INFO:tasks.workunit.client.0.smithi067.stdout:1/134: dread - d8/f15 zero size 2024-04-22T23:23:19.965 INFO:tasks.workunit.client.0.smithi067.stdout:8/161: chown d1/d3/d14/f34 1922397 1 2024-04-22T23:23:19.966 INFO:tasks.workunit.client.0.smithi067.stdout:0/112: chown db/l1f 56998 1 2024-04-22T23:23:19.967 INFO:tasks.workunit.client.0.smithi067.stdout:2/128: creat d1/d10/d1c/f23 x:0 0 0 2024-04-22T23:23:19.970 INFO:tasks.workunit.client.0.smithi067.stdout:1/135: getdents d8/d9/d2f 0 2024-04-22T23:23:19.974 INFO:tasks.workunit.client.0.smithi067.stdout:0/113: creat db/de/d11/f26 x:0 0 0 2024-04-22T23:23:19.974 INFO:tasks.workunit.client.0.smithi067.stdout:9/114: dwrite d5/d21/f24 [0,4194304] 0 2024-04-22T23:23:19.975 INFO:tasks.workunit.client.0.smithi067.stdout:1/136: write d8/d9/f2c [117536,18675] 0 2024-04-22T23:23:19.977 INFO:tasks.workunit.client.0.smithi067.stdout:2/129: stat d1/l20 0 2024-04-22T23:23:19.979 INFO:tasks.workunit.client.0.smithi067.stdout:9/115: readlink d5/dd/d18/l22 0 2024-04-22T23:23:19.981 INFO:tasks.workunit.client.0.smithi067.stdout:8/162: dwrite d1/d3/d27/f33 [0,4194304] 0 2024-04-22T23:23:19.983 INFO:tasks.workunit.client.0.smithi067.stdout:1/137: write d8/d9/f1b [4052471,23851] 0 2024-04-22T23:23:19.985 INFO:tasks.workunit.client.0.smithi067.stdout:8/163: dread - d1/d3/d10/f29 zero size 2024-04-22T23:23:19.988 INFO:tasks.workunit.client.0.smithi067.stdout:9/116: link d5/dd/d18/f1d d5/d21/f25 0 2024-04-22T23:23:19.988 INFO:tasks.workunit.client.0.smithi067.stdout:0/114: dwrite f0 [0,4194304] 0 2024-04-22T23:23:19.990 INFO:tasks.workunit.client.0.smithi067.stdout:1/138: mkdir d8/d9/d2f/d31 0 2024-04-22T23:23:19.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:19 smithi134 ceph-mon[34609]: pgmap v265: 65 pgs: 65 active+clean; 302 MiB data, 938 MiB used, 535 GiB / 536 GiB avail; 2.6 MiB/s rd, 17 MiB/s wr, 425 op/s 2024-04-22T23:23:19.994 INFO:tasks.workunit.client.0.smithi067.stdout:8/164: dread d1/d3/f1b [0,4194304] 0 2024-04-22T23:23:19.995 INFO:tasks.workunit.client.0.smithi067.stdout:9/117: rename d5/d21/f25 to d5/dd/f26 0 2024-04-22T23:23:19.997 INFO:tasks.workunit.client.0.smithi067.stdout:8/165: chown d1/d3/d14/f32 6104777 1 2024-04-22T23:23:20.003 INFO:tasks.workunit.client.0.smithi067.stdout:9/118: unlink d5/c1b 0 2024-04-22T23:23:20.005 INFO:tasks.workunit.client.0.smithi067.stdout:1/139: creat d8/f32 x:0 0 0 2024-04-22T23:23:20.005 INFO:tasks.workunit.client.0.smithi067.stdout:9/119: write d5/f9 [1846940,74007] 0 2024-04-22T23:23:20.007 INFO:tasks.workunit.client.0.smithi067.stdout:9/120: chown d5/dd/l13 4728364 1 2024-04-22T23:23:20.008 INFO:tasks.workunit.client.0.smithi067.stdout:8/166: link d1/dd/d26/l2c d1/dd/d26/l37 0 2024-04-22T23:23:20.008 INFO:tasks.workunit.client.0.smithi067.stdout:4/131: dwrite d12/f16 [0,4194304] 0 2024-04-22T23:23:20.012 INFO:tasks.workunit.client.0.smithi067.stdout:9/121: creat d5/d21/f27 x:0 0 0 2024-04-22T23:23:20.012 INFO:tasks.workunit.client.0.smithi067.stdout:8/167: dread d1/f7 [0,4194304] 0 2024-04-22T23:23:20.013 INFO:tasks.workunit.client.0.smithi067.stdout:9/122: stat d5/dd/d18/f1f 0 2024-04-22T23:23:20.013 INFO:tasks.workunit.client.0.smithi067.stdout:4/132: mkdir d12/d17/d1e/d2b 0 2024-04-22T23:23:20.014 INFO:tasks.workunit.client.0.smithi067.stdout:9/123: dread - d5/dd/f1a zero size 2024-04-22T23:23:20.016 INFO:tasks.workunit.client.0.smithi067.stdout:1/140: rmdir d8/d9/df/d1c 0 2024-04-22T23:23:20.016 INFO:tasks.workunit.client.0.smithi067.stdout:1/141: stat f4 0 2024-04-22T23:23:20.017 INFO:tasks.workunit.client.0.smithi067.stdout:4/133: dread fd [0,4194304] 0 2024-04-22T23:23:20.021 INFO:tasks.workunit.client.0.smithi067.stdout:9/124: symlink d5/d21/l28 0 2024-04-22T23:23:20.022 INFO:tasks.workunit.client.0.smithi067.stdout:0/115: truncate db/de/f14 3642941 0 2024-04-22T23:23:20.025 INFO:tasks.workunit.client.0.smithi067.stdout:1/142: rename d8/d9/d2a to d8/d9/d2f/d33 0 2024-04-22T23:23:20.026 INFO:tasks.workunit.client.0.smithi067.stdout:0/116: dread db/de/f12 [0,4194304] 0 2024-04-22T23:23:20.029 INFO:tasks.workunit.client.0.smithi067.stdout:8/168: creat d1/d3/f38 x:0 0 0 2024-04-22T23:23:20.030 INFO:tasks.workunit.client.0.smithi067.stdout:8/169: fsync d1/d3/f1e 0 2024-04-22T23:23:20.032 INFO:tasks.workunit.client.0.smithi067.stdout:4/134: rename d12/d17/d1e/l1f to d12/l2c 0 2024-04-22T23:23:20.033 INFO:tasks.workunit.client.0.smithi067.stdout:0/117: creat db/de/d11/f27 x:0 0 0 2024-04-22T23:23:20.035 INFO:tasks.workunit.client.0.smithi067.stdout:0/118: chown db/de/d11/l1e 174648245 1 2024-04-22T23:23:20.037 INFO:tasks.workunit.client.0.smithi067.stdout:9/125: creat d5/f29 x:0 0 0 2024-04-22T23:23:20.039 INFO:tasks.workunit.client.0.smithi067.stdout:1/143: dread d8/d9/f13 [0,4194304] 0 2024-04-22T23:23:20.041 INFO:tasks.workunit.client.0.smithi067.stdout:8/170: rename d1/d3/d10/c2e to d1/d3/c39 0 2024-04-22T23:23:20.043 INFO:tasks.workunit.client.0.smithi067.stdout:8/171: chown d1/l21 126640493 1 2024-04-22T23:23:20.044 INFO:tasks.workunit.client.0.smithi067.stdout:1/144: unlink d8/d9/f13 0 2024-04-22T23:23:20.044 INFO:tasks.workunit.client.0.smithi067.stdout:9/126: dwrite d5/fe [0,4194304] 0 2024-04-22T23:23:20.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:19 smithi067 ceph-mon[26453]: pgmap v265: 65 pgs: 65 active+clean; 302 MiB data, 938 MiB used, 535 GiB / 536 GiB avail; 2.6 MiB/s rd, 17 MiB/s wr, 425 op/s 2024-04-22T23:23:20.047 INFO:tasks.workunit.client.0.smithi067.stdout:0/119: rename db/de/f20 to db/de/d11/d1a/f28 0 2024-04-22T23:23:20.052 INFO:tasks.workunit.client.0.smithi067.stdout:4/135: truncate fd 4870057 0 2024-04-22T23:23:20.055 INFO:tasks.workunit.client.0.smithi067.stdout:9/127: dwrite d5/dd/f1a [0,4194304] 0 2024-04-22T23:23:20.056 INFO:tasks.workunit.client.0.smithi067.stdout:0/120: mkdir db/d21/d29 0 2024-04-22T23:23:20.056 INFO:tasks.workunit.client.0.smithi067.stdout:1/145: dwrite d8/d9/f1b [0,4194304] 0 2024-04-22T23:23:20.058 INFO:tasks.workunit.client.0.smithi067.stdout:1/146: fdatasync d8/d9/f2c 0 2024-04-22T23:23:20.060 INFO:tasks.workunit.client.0.smithi067.stdout:4/136: creat d12/d17/d1e/f2d x:0 0 0 2024-04-22T23:23:20.060 INFO:tasks.workunit.client.0.smithi067.stdout:9/128: fsync d5/ff 0 2024-04-22T23:23:20.062 INFO:tasks.workunit.client.0.smithi067.stdout:9/129: write d5/d21/f24 [4022087,29138] 0 2024-04-22T23:23:20.065 INFO:tasks.workunit.client.0.smithi067.stdout:1/147: mkdir d8/d24/d34 0 2024-04-22T23:23:20.066 INFO:tasks.workunit.client.0.smithi067.stdout:1/148: write d8/d12/f17 [3100514,30474] 0 2024-04-22T23:23:20.067 INFO:tasks.workunit.client.0.smithi067.stdout:4/137: mkdir d12/d2e 0 2024-04-22T23:23:20.068 INFO:tasks.workunit.client.0.smithi067.stdout:4/138: truncate d12/f19 530260 0 2024-04-22T23:23:20.069 INFO:tasks.workunit.client.0.smithi067.stdout:4/139: stat d12/f2a 0 2024-04-22T23:23:20.070 INFO:tasks.workunit.client.0.smithi067.stdout:4/140: write d12/d17/d1e/f27 [459917,112377] 0 2024-04-22T23:23:20.077 INFO:tasks.workunit.client.0.smithi067.stdout:8/172: dwrite d1/dd/f20 [0,4194304] 0 2024-04-22T23:23:20.079 INFO:tasks.workunit.client.0.smithi067.stdout:9/130: write d5/ff [938000,105999] 0 2024-04-22T23:23:20.082 INFO:tasks.workunit.client.0.smithi067.stdout:1/149: symlink d8/d9/d2d/l35 0 2024-04-22T23:23:20.083 INFO:tasks.workunit.client.0.smithi067.stdout:9/131: symlink d5/dd/d18/l2a 0 2024-04-22T23:23:20.084 INFO:tasks.workunit.client.0.smithi067.stdout:9/132: fsync d5/f29 0 2024-04-22T23:23:20.085 INFO:tasks.workunit.client.0.smithi067.stdout:4/141: creat d12/d17/f2f x:0 0 0 2024-04-22T23:23:20.085 INFO:tasks.workunit.client.0.smithi067.stdout:4/142: chown f4 127 1 2024-04-22T23:23:20.086 INFO:tasks.workunit.client.0.smithi067.stdout:8/173: mknod d1/d3/c3a 0 2024-04-22T23:23:20.086 INFO:tasks.workunit.client.0.smithi067.stdout:8/174: readlink d1/dd/l19 0 2024-04-22T23:23:20.087 INFO:tasks.workunit.client.0.smithi067.stdout:1/150: mkdir d8/d12/d36 0 2024-04-22T23:23:20.088 INFO:tasks.workunit.client.0.smithi067.stdout:8/175: dread - d1/d3/d14/f34 zero size 2024-04-22T23:23:20.088 INFO:tasks.workunit.client.0.smithi067.stdout:1/151: dread - d8/f15 zero size 2024-04-22T23:23:20.089 INFO:tasks.workunit.client.0.smithi067.stdout:9/133: fdatasync d5/fb 0 2024-04-22T23:23:20.091 INFO:tasks.workunit.client.0.smithi067.stdout:8/176: mkdir d1/d3/d10/d3b 0 2024-04-22T23:23:20.094 INFO:tasks.workunit.client.0.smithi067.stdout:9/134: creat d5/d21/f2b x:0 0 0 2024-04-22T23:23:20.095 INFO:tasks.workunit.client.0.smithi067.stdout:0/121: truncate db/de/ff 3182989 0 2024-04-22T23:23:20.096 INFO:tasks.workunit.client.0.smithi067.stdout:4/143: dwrite fa [0,4194304] 0 2024-04-22T23:23:20.097 INFO:tasks.workunit.client.0.smithi067.stdout:1/152: creat d8/d24/d34/f37 x:0 0 0 2024-04-22T23:23:20.098 INFO:tasks.workunit.client.0.smithi067.stdout:4/144: chown d12/d17/d1e/f28 63861 1 2024-04-22T23:23:20.098 INFO:tasks.workunit.client.0.smithi067.stdout:4/145: chown d12/f2a 0 1 2024-04-22T23:23:20.104 INFO:tasks.workunit.client.0.smithi067.stdout:4/146: readlink d12/l2c 0 2024-04-22T23:23:20.106 INFO:tasks.workunit.client.0.smithi067.stdout:8/177: getdents d1/d3 0 2024-04-22T23:23:20.107 INFO:tasks.workunit.client.0.smithi067.stdout:4/147: symlink d12/d17/l30 0 2024-04-22T23:23:20.109 INFO:tasks.workunit.client.0.smithi067.stdout:8/178: read d1/d3/d14/f30 [3678489,7773] 0 2024-04-22T23:23:20.109 INFO:tasks.workunit.client.0.smithi067.stdout:4/148: mkdir d12/d17/d31 0 2024-04-22T23:23:20.110 INFO:tasks.workunit.client.0.smithi067.stdout:8/179: stat d1/dd/d26/l37 0 2024-04-22T23:23:20.147 INFO:tasks.workunit.client.0.smithi067.stdout:6/87: sync 2024-04-22T23:23:20.147 INFO:tasks.workunit.client.0.smithi067.stdout:7/80: sync 2024-04-22T23:23:20.154 INFO:tasks.workunit.client.0.smithi067.stdout:8/180: write d1/d3/d27/f28 [633134,74890] 0 2024-04-22T23:23:20.154 INFO:tasks.workunit.client.0.smithi067.stdout:6/88: creat d17/f20 x:0 0 0 2024-04-22T23:23:20.154 INFO:tasks.workunit.client.0.smithi067.stdout:7/81: truncate d10/f15 116636 0 2024-04-22T23:23:20.156 INFO:tasks.workunit.client.0.smithi067.stdout:6/89: chown l15 3 1 2024-04-22T23:23:20.158 INFO:tasks.workunit.client.0.smithi067.stdout:8/181: read d1/d3/f13 [3525062,98709] 0 2024-04-22T23:23:20.159 INFO:tasks.workunit.client.0.smithi067.stdout:7/82: write f1 [1145728,8415] 0 2024-04-22T23:23:20.160 INFO:tasks.workunit.client.0.smithi067.stdout:6/90: write fe [993945,12221] 0 2024-04-22T23:23:20.164 INFO:tasks.workunit.client.0.smithi067.stdout:1/153: dwrite f4 [0,4194304] 0 2024-04-22T23:23:20.165 INFO:tasks.workunit.client.0.smithi067.stdout:7/83: write d10/f13 [1040891,80402] 0 2024-04-22T23:23:20.166 INFO:tasks.workunit.client.0.smithi067.stdout:0/122: dwrite db/de/f14 [0,4194304] 0 2024-04-22T23:23:20.169 INFO:tasks.workunit.client.0.smithi067.stdout:6/91: readlink d17/l1d 0 2024-04-22T23:23:20.169 INFO:tasks.workunit.client.0.smithi067.stdout:4/149: dwrite f4 [0,4194304] 0 2024-04-22T23:23:20.170 INFO:tasks.workunit.client.0.smithi067.stdout:7/84: fdatasync fe 0 2024-04-22T23:23:20.177 INFO:tasks.workunit.client.0.smithi067.stdout:1/154: creat d8/d9/d2f/f38 x:0 0 0 2024-04-22T23:23:20.177 INFO:tasks.workunit.client.0.smithi067.stdout:0/123: mkdir db/de/d2a 0 2024-04-22T23:23:20.179 INFO:tasks.workunit.client.0.smithi067.stdout:8/182: dwrite d1/f7 [0,4194304] 0 2024-04-22T23:23:20.181 INFO:tasks.workunit.client.0.smithi067.stdout:4/150: dwrite f5 [0,4194304] 0 2024-04-22T23:23:20.182 INFO:tasks.workunit.client.0.smithi067.stdout:1/155: rmdir d8/d24/d34 39 2024-04-22T23:23:20.184 INFO:tasks.workunit.client.0.smithi067.stdout:8/183: chown d1/d3/d10/f29 2255223 1 2024-04-22T23:23:20.186 INFO:tasks.workunit.client.0.smithi067.stdout:1/156: chown d8/fc 1152 1 2024-04-22T23:23:20.186 INFO:tasks.workunit.client.0.smithi067.stdout:4/151: rmdir d12/d17 39 2024-04-22T23:23:20.188 INFO:tasks.workunit.client.0.smithi067.stdout:8/184: fdatasync d1/d3/d27/f28 0 2024-04-22T23:23:20.191 INFO:tasks.workunit.client.0.smithi067.stdout:7/85: dwrite f1 [0,4194304] 0 2024-04-22T23:23:20.193 INFO:tasks.workunit.client.0.smithi067.stdout:4/152: dread - d12/d17/f2f zero size 2024-04-22T23:23:20.193 INFO:tasks.workunit.client.0.smithi067.stdout:0/124: dwrite db/de/f22 [0,4194304] 0 2024-04-22T23:23:20.194 INFO:tasks.workunit.client.0.smithi067.stdout:1/157: creat d8/d24/d34/f39 x:0 0 0 2024-04-22T23:23:20.196 INFO:tasks.workunit.client.0.smithi067.stdout:1/158: stat d8/d9/d2f 0 2024-04-22T23:23:20.198 INFO:tasks.workunit.client.0.smithi067.stdout:7/86: unlink f6 0 2024-04-22T23:23:20.200 INFO:tasks.workunit.client.0.smithi067.stdout:1/159: read - d8/d9/f29 zero size 2024-04-22T23:23:20.204 INFO:tasks.workunit.client.0.smithi067.stdout:1/160: creat d8/d12/f3a x:0 0 0 2024-04-22T23:23:20.206 INFO:tasks.workunit.client.0.smithi067.stdout:1/161: fsync d8/d9/fd 0 2024-04-22T23:23:20.206 INFO:tasks.workunit.client.0.smithi067.stdout:8/185: dwrite d1/d3/f11 [0,4194304] 0 2024-04-22T23:23:20.207 INFO:tasks.workunit.client.0.smithi067.stdout:7/87: dread fb [0,4194304] 0 2024-04-22T23:23:20.207 INFO:tasks.workunit.client.0.smithi067.stdout:1/162: dread - d8/f15 zero size 2024-04-22T23:23:20.208 INFO:tasks.workunit.client.0.smithi067.stdout:1/163: fsync d8/d24/d34/f37 0 2024-04-22T23:23:20.210 INFO:tasks.workunit.client.0.smithi067.stdout:8/186: symlink d1/dd/d26/l3c 0 2024-04-22T23:23:20.210 INFO:tasks.workunit.client.0.smithi067.stdout:7/88: mknod d10/c17 0 2024-04-22T23:23:20.211 INFO:tasks.workunit.client.0.smithi067.stdout:7/89: chown d10/d14 1887 1 2024-04-22T23:23:20.214 INFO:tasks.workunit.client.0.smithi067.stdout:7/90: mkdir d10/d18 0 2024-04-22T23:23:20.214 INFO:tasks.workunit.client.0.smithi067.stdout:8/187: dread d1/d3/f1b [0,4194304] 0 2024-04-22T23:23:20.215 INFO:tasks.workunit.client.0.smithi067.stdout:1/164: creat d8/d9/f3b x:0 0 0 2024-04-22T23:23:20.215 INFO:tasks.workunit.client.0.smithi067.stdout:8/188: chown d1/d3/f13 621 1 2024-04-22T23:23:20.216 INFO:tasks.workunit.client.0.smithi067.stdout:7/91: unlink f1 0 2024-04-22T23:23:20.216 INFO:tasks.workunit.client.0.smithi067.stdout:8/189: chown d1/c2d 5951 1 2024-04-22T23:23:20.217 INFO:tasks.workunit.client.0.smithi067.stdout:8/190: dread - d1/d3/d14/f34 zero size 2024-04-22T23:23:20.218 INFO:tasks.workunit.client.0.smithi067.stdout:8/191: stat d1/dd/d26/l37 0 2024-04-22T23:23:20.221 INFO:tasks.workunit.client.0.smithi067.stdout:7/92: unlink f4 0 2024-04-22T23:23:20.221 INFO:tasks.workunit.client.0.smithi067.stdout:7/93: stat d10/f13 0 2024-04-22T23:23:20.223 INFO:tasks.workunit.client.0.smithi067.stdout:8/192: creat d1/d3/d10/f3d x:0 0 0 2024-04-22T23:23:20.223 INFO:tasks.workunit.client.0.smithi067.stdout:8/193: readlink d1/dd/d26/l3c 0 2024-04-22T23:23:20.224 INFO:tasks.workunit.client.0.smithi067.stdout:7/94: mkdir d10/d18/d19 0 2024-04-22T23:23:20.224 INFO:tasks.workunit.client.0.smithi067.stdout:7/95: truncate fc 1099631 0 2024-04-22T23:23:20.225 INFO:tasks.workunit.client.0.smithi067.stdout:8/194: creat d1/d3/d10/f3e x:0 0 0 2024-04-22T23:23:20.227 INFO:tasks.workunit.client.0.smithi067.stdout:8/195: symlink d1/d3/l3f 0 2024-04-22T23:23:20.242 INFO:tasks.workunit.client.0.smithi067.stdout:6/92: truncate f12 3386425 0 2024-04-22T23:23:20.242 INFO:tasks.workunit.client.0.smithi067.stdout:6/93: dread - f13 zero size 2024-04-22T23:23:20.254 INFO:tasks.workunit.client.0.smithi067.stdout:7/96: fsync fe 0 2024-04-22T23:23:20.256 INFO:tasks.workunit.client.0.smithi067.stdout:7/97: creat d10/d18/f1a x:0 0 0 2024-04-22T23:23:20.259 INFO:tasks.workunit.client.0.smithi067.stdout:1/165: truncate d8/d9/fd 2576631 0 2024-04-22T23:23:20.262 INFO:tasks.workunit.client.0.smithi067.stdout:1/166: write d8/d9/f3b [370349,74716] 0 2024-04-22T23:23:20.265 INFO:tasks.workunit.client.0.smithi067.stdout:0/125: dwrite db/de/f12 [0,4194304] 0 2024-04-22T23:23:20.265 INFO:tasks.workunit.client.0.smithi067.stdout:1/167: chown d8/d9/fd 1144062 1 2024-04-22T23:23:20.265 INFO:tasks.workunit.client.0.smithi067.stdout:4/153: dwrite fd [0,4194304] 0 2024-04-22T23:23:20.270 INFO:tasks.workunit.client.0.smithi067.stdout:1/168: truncate d8/d24/d34/f37 632653 0 2024-04-22T23:23:20.273 INFO:tasks.workunit.client.0.smithi067.stdout:4/154: unlink d12/d17/d1e/f2d 0 2024-04-22T23:23:20.274 INFO:tasks.workunit.client.0.smithi067.stdout:4/155: fdatasync d12/f16 0 2024-04-22T23:23:20.277 INFO:tasks.workunit.client.0.smithi067.stdout:5/138: sync 2024-04-22T23:23:20.277 INFO:tasks.workunit.client.0.smithi067.stdout:3/186: sync 2024-04-22T23:23:20.278 INFO:tasks.workunit.client.0.smithi067.stdout:1/169: dwrite f4 [0,4194304] 0 2024-04-22T23:23:20.280 INFO:tasks.workunit.client.0.smithi067.stdout:0/126: mknod db/d21/d29/c2b 0 2024-04-22T23:23:20.282 INFO:tasks.workunit.client.0.smithi067.stdout:6/94: write fc [1087194,65008] 0 2024-04-22T23:23:20.288 INFO:tasks.workunit.client.0.smithi067.stdout:6/95: unlink l16 0 2024-04-22T23:23:20.293 INFO:tasks.workunit.client.0.smithi067.stdout:3/187: dwrite d3/d4/d17/f25 [0,4194304] 0 2024-04-22T23:23:20.295 INFO:tasks.workunit.client.0.smithi067.stdout:5/139: rename d5/d14 to d5/d32 0 2024-04-22T23:23:20.297 INFO:tasks.workunit.client.0.smithi067.stdout:6/96: creat d17/f21 x:0 0 0 2024-04-22T23:23:20.297 INFO:tasks.workunit.client.0.smithi067.stdout:5/140: stat d5/l9 0 2024-04-22T23:23:20.300 INFO:tasks.workunit.client.0.smithi067.stdout:1/170: dwrite d8/f15 [0,4194304] 0 2024-04-22T23:23:20.303 INFO:tasks.workunit.client.0.smithi067.stdout:0/127: dread db/de/f1b [0,4194304] 0 2024-04-22T23:23:20.311 INFO:tasks.workunit.client.0.smithi067.stdout:8/196: dwrite d1/d3/d14/f32 [0,4194304] 0 2024-04-22T23:23:20.312 INFO:tasks.workunit.client.0.smithi067.stdout:6/97: rename d17/l18 to d17/l22 0 2024-04-22T23:23:20.312 INFO:tasks.workunit.client.0.smithi067.stdout:5/141: truncate d5/db/d12/f13 533877 0 2024-04-22T23:23:20.314 INFO:tasks.workunit.client.0.smithi067.stdout:0/128: mkdir db/d21/d29/d2c 0 2024-04-22T23:23:20.314 INFO:tasks.workunit.client.0.smithi067.stdout:3/188: dwrite d3/d4/d12/f2f [0,4194304] 0 2024-04-22T23:23:20.317 INFO:tasks.workunit.client.0.smithi067.stdout:3/189: write d3/d4/d10/f39 [556736,72226] 0 2024-04-22T23:23:20.317 INFO:tasks.workunit.client.0.smithi067.stdout:6/98: symlink d17/l23 0 2024-04-22T23:23:20.326 INFO:tasks.workunit.client.0.smithi067.stdout:0/129: mkdir db/d21/d29/d2d 0 2024-04-22T23:23:20.329 INFO:tasks.workunit.client.0.smithi067.stdout:8/197: dwrite d1/d3/d27/f33 [4194304,4194304] 0 2024-04-22T23:23:20.329 INFO:tasks.workunit.client.0.smithi067.stdout:3/190: readlink d3/d4/l15 0 2024-04-22T23:23:20.330 INFO:tasks.workunit.client.0.smithi067.stdout:6/99: creat d17/f24 x:0 0 0 2024-04-22T23:23:20.332 INFO:tasks.workunit.client.0.smithi067.stdout:6/100: dread - d17/f1a zero size 2024-04-22T23:23:20.338 INFO:tasks.workunit.client.0.smithi067.stdout:8/198: mknod d1/dd/c40 0 2024-04-22T23:23:20.338 INFO:tasks.workunit.client.0.smithi067.stdout:3/191: chown d3/d4/d10/d19/f3b 10442 1 2024-04-22T23:23:20.339 INFO:tasks.workunit.client.0.smithi067.stdout:8/199: chown d1/d3/d14 4 1 2024-04-22T23:23:20.339 INFO:tasks.workunit.client.0.smithi067.stdout:1/171: truncate d8/f15 3370112 0 2024-04-22T23:23:20.340 INFO:tasks.workunit.client.0.smithi067.stdout:6/101: creat d17/f25 x:0 0 0 2024-04-22T23:23:20.343 INFO:tasks.workunit.client.0.smithi067.stdout:0/130: mkdir db/d2e 0 2024-04-22T23:23:20.343 INFO:tasks.workunit.client.0.smithi067.stdout:3/192: chown d3/d4/c9 155 1 2024-04-22T23:23:20.345 INFO:tasks.workunit.client.0.smithi067.stdout:1/172: mknod d8/d12/c3c 0 2024-04-22T23:23:20.345 INFO:tasks.workunit.client.0.smithi067.stdout:1/173: chown d8/d9/f1b 2014440 1 2024-04-22T23:23:20.346 INFO:tasks.workunit.client.0.smithi067.stdout:8/200: write d1/d3/d14/f30 [168767,66280] 0 2024-04-22T23:23:20.348 INFO:tasks.workunit.client.0.smithi067.stdout:5/142: getdents d5/db/d12 0 2024-04-22T23:23:20.348 INFO:tasks.workunit.client.0.smithi067.stdout:6/102: rename d17/f1a to d17/f26 0 2024-04-22T23:23:20.349 INFO:tasks.workunit.client.0.smithi067.stdout:6/103: chown d17/f1e 3395 1 2024-04-22T23:23:20.351 INFO:tasks.workunit.client.0.smithi067.stdout:3/193: read d3/d4/f16 [709384,35312] 0 2024-04-22T23:23:20.352 INFO:tasks.workunit.client.0.smithi067.stdout:1/174: mknod d8/d9/d2f/d33/c3d 0 2024-04-22T23:23:20.352 INFO:tasks.workunit.client.0.smithi067.stdout:6/104: truncate d17/f1b 531084 0 2024-04-22T23:23:20.354 INFO:tasks.workunit.client.0.smithi067.stdout:5/143: dread d5/d32/f16 [4194304,4194304] 0 2024-04-22T23:23:20.357 INFO:tasks.workunit.client.0.smithi067.stdout:3/194: rename d3/d4/d17 to d3/d4/d10/d1f/d3c 0 2024-04-22T23:23:20.359 INFO:tasks.workunit.client.0.smithi067.stdout:5/144: mknod d5/d32/c33 0 2024-04-22T23:23:20.361 INFO:tasks.workunit.client.0.smithi067.stdout:0/131: dwrite db/de/d11/d1a/f28 [4194304,4194304] 0 2024-04-22T23:23:20.361 INFO:tasks.workunit.client.0.smithi067.stdout:6/105: dwrite f6 [0,4194304] 0 2024-04-22T23:23:20.362 INFO:tasks.workunit.client.0.smithi067.stdout:1/175: rename d8/f15 to d8/d9/d2f/d33/f3e 0 2024-04-22T23:23:20.367 INFO:tasks.workunit.client.0.smithi067.stdout:3/195: dread d3/d4/f24 [0,4194304] 0 2024-04-22T23:23:20.372 INFO:tasks.workunit.client.0.smithi067.stdout:6/106: rename c9 to d17/c27 0 2024-04-22T23:23:20.372 INFO:tasks.workunit.client.0.smithi067.stdout:3/196: mknod d3/d4/d10/d19/c3d 0 2024-04-22T23:23:20.374 INFO:tasks.workunit.client.0.smithi067.stdout:1/176: dwrite d8/d9/f1b [0,4194304] 0 2024-04-22T23:23:20.375 INFO:tasks.workunit.client.0.smithi067.stdout:3/197: fdatasync d3/d4/d10/d19/f3b 0 2024-04-22T23:23:20.375 INFO:tasks.workunit.client.0.smithi067.stdout:0/132: getdents db/de/d2a 0 2024-04-22T23:23:20.377 INFO:tasks.workunit.client.0.smithi067.stdout:0/133: rename db to db/de/d11/d1a/d2f 22 2024-04-22T23:23:20.378 INFO:tasks.workunit.client.0.smithi067.stdout:5/145: creat d5/db/f34 x:0 0 0 2024-04-22T23:23:20.382 INFO:tasks.workunit.client.0.smithi067.stdout:3/198: creat d3/d4/d10/d1f/f3e x:0 0 0 2024-04-22T23:23:20.385 INFO:tasks.workunit.client.0.smithi067.stdout:6/107: dwrite d17/f21 [0,4194304] 0 2024-04-22T23:23:20.388 INFO:tasks.workunit.client.0.smithi067.stdout:1/177: link d8/d9/fd d8/d24/d34/f3f 0 2024-04-22T23:23:20.391 INFO:tasks.workunit.client.0.smithi067.stdout:3/199: creat d3/d4/d10/d1f/f3f x:0 0 0 2024-04-22T23:23:20.391 INFO:tasks.workunit.client.0.smithi067.stdout:5/146: dwrite d5/f1b [0,4194304] 0 2024-04-22T23:23:20.392 INFO:tasks.workunit.client.0.smithi067.stdout:3/200: fdatasync d3/d4/d10/d19/f3b 0 2024-04-22T23:23:20.394 INFO:tasks.workunit.client.0.smithi067.stdout:1/178: mkdir d8/d12/d40 0 2024-04-22T23:23:20.395 INFO:tasks.workunit.client.0.smithi067.stdout:3/201: mkdir d3/d4/d28/d40 0 2024-04-22T23:23:20.396 INFO:tasks.workunit.client.0.smithi067.stdout:1/179: creat d8/d9/d2f/d33/f41 x:0 0 0 2024-04-22T23:23:20.398 INFO:tasks.workunit.client.0.smithi067.stdout:6/108: getdents d17 0 2024-04-22T23:23:20.398 INFO:tasks.workunit.client.0.smithi067.stdout:3/202: truncate d3/d4/fe 1270754 0 2024-04-22T23:23:20.398 INFO:tasks.workunit.client.0.smithi067.stdout:1/180: write d8/d9/f2c [113320,108223] 0 2024-04-22T23:23:20.400 INFO:tasks.workunit.client.0.smithi067.stdout:1/181: dread - d8/d12/f3a zero size 2024-04-22T23:23:20.402 INFO:tasks.workunit.client.0.smithi067.stdout:6/109: fdatasync f12 0 2024-04-22T23:23:20.404 INFO:tasks.workunit.client.0.smithi067.stdout:0/134: truncate db/de/d11/d1a/f28 4016461 0 2024-04-22T23:23:20.407 INFO:tasks.workunit.client.0.smithi067.stdout:6/110: mkdir d17/d28 0 2024-04-22T23:23:20.407 INFO:tasks.workunit.client.0.smithi067.stdout:6/111: chown l15 451836306 1 2024-04-22T23:23:20.410 INFO:tasks.workunit.client.0.smithi067.stdout:0/135: symlink db/de/d2a/l30 0 2024-04-22T23:23:20.411 INFO:tasks.workunit.client.0.smithi067.stdout:1/182: getdents d8/d9/d2d 0 2024-04-22T23:23:20.411 INFO:tasks.workunit.client.0.smithi067.stdout:6/112: write d17/f1f [925819,50506] 0 2024-04-22T23:23:20.412 INFO:tasks.workunit.client.0.smithi067.stdout:0/136: mkdir db/d2e/d31 0 2024-04-22T23:23:20.417 INFO:tasks.workunit.client.0.smithi067.stdout:6/113: getdents d17/d28 0 2024-04-22T23:23:20.418 INFO:tasks.workunit.client.0.smithi067.stdout:6/114: readlink d17/l23 0 2024-04-22T23:23:20.419 INFO:tasks.workunit.client.0.smithi067.stdout:6/115: stat f13 0 2024-04-22T23:23:20.420 INFO:tasks.workunit.client.0.smithi067.stdout:0/137: dwrite db/de/f1c [0,4194304] 0 2024-04-22T23:23:20.423 INFO:tasks.workunit.client.0.smithi067.stdout:1/183: creat d8/d9/f42 x:0 0 0 2024-04-22T23:23:20.426 INFO:tasks.workunit.client.0.smithi067.stdout:1/184: dread - d8/d9/df/f23 zero size 2024-04-22T23:23:20.426 INFO:tasks.workunit.client.0.smithi067.stdout:0/138: rmdir db/d2e 39 2024-04-22T23:23:20.429 INFO:tasks.workunit.client.0.smithi067.stdout:1/185: dread - d8/d9/df/f23 zero size 2024-04-22T23:23:20.430 INFO:tasks.workunit.client.0.smithi067.stdout:3/203: dwrite d3/d4/f20 [0,4194304] 0 2024-04-22T23:23:20.431 INFO:tasks.workunit.client.0.smithi067.stdout:3/204: chown d3/d4/l15 238082 1 2024-04-22T23:23:20.431 INFO:tasks.workunit.client.0.smithi067.stdout:6/116: dread f6 [0,4194304] 0 2024-04-22T23:23:20.437 INFO:tasks.workunit.client.0.smithi067.stdout:3/205: mknod d3/d4/d10/d19/c41 0 2024-04-22T23:23:20.439 INFO:tasks.workunit.client.0.smithi067.stdout:1/186: creat d8/d9/f43 x:0 0 0 2024-04-22T23:23:20.440 INFO:tasks.workunit.client.0.smithi067.stdout:6/117: rmdir d17/d28 0 2024-04-22T23:23:20.440 INFO:tasks.workunit.client.0.smithi067.stdout:3/206: dread - d3/d4/d10/d19/f3b zero size 2024-04-22T23:23:20.442 INFO:tasks.workunit.client.0.smithi067.stdout:1/187: chown d8/d9/df/l14 1019812 1 2024-04-22T23:23:20.442 INFO:tasks.workunit.client.0.smithi067.stdout:0/139: dwrite db/fd [4194304,4194304] 0 2024-04-22T23:23:20.442 INFO:tasks.workunit.client.0.smithi067.stdout:1/188: dread - d8/f32 zero size 2024-04-22T23:23:20.443 INFO:tasks.workunit.client.0.smithi067.stdout:6/118: unlink l15 0 2024-04-22T23:23:20.444 INFO:tasks.workunit.client.0.smithi067.stdout:6/119: dread - f13 zero size 2024-04-22T23:23:20.446 INFO:tasks.workunit.client.0.smithi067.stdout:6/120: write fe [1513666,51971] 0 2024-04-22T23:23:20.447 INFO:tasks.workunit.client.0.smithi067.stdout:1/189: read d8/d12/f17 [496453,56830] 0 2024-04-22T23:23:20.448 INFO:tasks.workunit.client.0.smithi067.stdout:0/140: rename db/de/f14 to db/de/d11/f32 0 2024-04-22T23:23:20.449 INFO:tasks.workunit.client.0.smithi067.stdout:0/141: rename db to db/d21/d29/d33 22 2024-04-22T23:23:20.450 INFO:tasks.workunit.client.0.smithi067.stdout:0/142: truncate db/de/d11/f27 881890 0 2024-04-22T23:23:20.451 INFO:tasks.workunit.client.0.smithi067.stdout:6/121: creat d17/f29 x:0 0 0 2024-04-22T23:23:20.457 INFO:tasks.workunit.client.0.smithi067.stdout:1/190: creat d8/d12/d40/f44 x:0 0 0 2024-04-22T23:23:20.457 INFO:tasks.workunit.client.0.smithi067.stdout:0/143: dwrite db/de/f12 [4194304,4194304] 0 2024-04-22T23:23:20.461 INFO:tasks.workunit.client.0.smithi067.stdout:1/191: dread d8/d9/f3b [0,4194304] 0 2024-04-22T23:23:20.462 INFO:tasks.workunit.client.0.smithi067.stdout:1/192: dread - d8/f32 zero size 2024-04-22T23:23:20.465 INFO:tasks.workunit.client.0.smithi067.stdout:3/207: truncate d3/d4/d10/d1f/d3c/f25 1955474 0 2024-04-22T23:23:20.466 INFO:tasks.workunit.client.0.smithi067.stdout:1/193: write d8/d9/f3b [712187,40010] 0 2024-04-22T23:23:20.468 INFO:tasks.workunit.client.0.smithi067.stdout:1/194: mkdir d8/d9/d2f/d45 0 2024-04-22T23:23:20.469 INFO:tasks.workunit.client.0.smithi067.stdout:0/144: truncate db/de/d11/d1a/f28 129102 0 2024-04-22T23:23:20.470 INFO:tasks.workunit.client.0.smithi067.stdout:1/195: truncate d8/d12/d40/f44 79128 0 2024-04-22T23:23:20.470 INFO:tasks.workunit.client.0.smithi067.stdout:1/196: stat d8/f26 0 2024-04-22T23:23:20.471 INFO:tasks.workunit.client.0.smithi067.stdout:1/197: fdatasync d8/d9/f43 0 2024-04-22T23:23:20.472 INFO:tasks.workunit.client.0.smithi067.stdout:3/208: link d3/d4/d10/d19/c2e d3/c42 0 2024-04-22T23:23:20.472 INFO:tasks.workunit.client.0.smithi067.stdout:1/198: stat d8/d12/l27 0 2024-04-22T23:23:20.473 INFO:tasks.workunit.client.0.smithi067.stdout:1/199: write d8/d12/f3a [897330,35874] 0 2024-04-22T23:23:20.480 INFO:tasks.workunit.client.0.smithi067.stdout:1/200: rename d8/d12/f3a to d8/d9/f46 0 2024-04-22T23:23:20.481 INFO:tasks.workunit.client.0.smithi067.stdout:1/201: fsync d8/d9/f25 0 2024-04-22T23:23:20.482 INFO:tasks.workunit.client.0.smithi067.stdout:1/202: stat d8/d9/f2c 0 2024-04-22T23:23:20.486 INFO:tasks.workunit.client.0.smithi067.stdout:1/203: dread d8/d9/f1b [0,4194304] 0 2024-04-22T23:23:20.486 INFO:tasks.workunit.client.0.smithi067.stdout:1/204: dread - d8/f26 zero size 2024-04-22T23:23:20.489 INFO:tasks.workunit.client.0.smithi067.stdout:3/209: dread d3/d4/d10/f1a [0,4194304] 0 2024-04-22T23:23:20.490 INFO:tasks.workunit.client.0.smithi067.stdout:3/210: write d3/d4/d10/d1f/f3f [536795,28632] 0 2024-04-22T23:23:20.490 INFO:tasks.workunit.client.0.smithi067.stdout:1/205: creat d8/d24/d34/f47 x:0 0 0 2024-04-22T23:23:20.492 INFO:tasks.workunit.client.0.smithi067.stdout:3/211: chown d3/d4/d10/d1f 4310 1 2024-04-22T23:23:20.494 INFO:tasks.workunit.client.0.smithi067.stdout:1/206: dread d8/d9/df/f1f [0,4194304] 0 2024-04-22T23:23:20.496 INFO:tasks.workunit.client.0.smithi067.stdout:1/207: mkdir d8/d9/df/d48 0 2024-04-22T23:23:20.498 INFO:tasks.workunit.client.0.smithi067.stdout:1/208: mknod d8/d9/df/d48/c49 0 2024-04-22T23:23:20.501 INFO:tasks.workunit.client.0.smithi067.stdout:1/209: dread d8/d9/f1b [0,4194304] 0 2024-04-22T23:23:20.507 INFO:tasks.workunit.client.0.smithi067.stdout:3/212: dread d3/fa [0,4194304] 0 2024-04-22T23:23:20.507 INFO:tasks.workunit.client.0.smithi067.stdout:1/210: write d8/d12/f17 [3770234,9488] 0 2024-04-22T23:23:20.508 INFO:tasks.workunit.client.0.smithi067.stdout:3/213: write d3/d4/d12/f27 [82435,56185] 0 2024-04-22T23:23:20.511 INFO:tasks.workunit.client.0.smithi067.stdout:1/211: mkdir d8/d4a 0 2024-04-22T23:23:20.511 INFO:tasks.workunit.client.0.smithi067.stdout:2/130: sync 2024-04-22T23:23:20.511 INFO:tasks.workunit.client.0.smithi067.stdout:9/135: sync 2024-04-22T23:23:20.511 INFO:tasks.workunit.client.0.smithi067.stdout:3/214: truncate d3/d4/fe 1213729 0 2024-04-22T23:23:20.511 INFO:tasks.workunit.client.0.smithi067.stdout:9/136: dread - d5/dd/f26 zero size 2024-04-22T23:23:20.512 INFO:tasks.workunit.client.0.smithi067.stdout:3/215: stat d3/d4/d10/c23 0 2024-04-22T23:23:20.512 INFO:tasks.workunit.client.0.smithi067.stdout:9/137: dread - d5/fa zero size 2024-04-22T23:23:20.514 INFO:tasks.workunit.client.0.smithi067.stdout:9/138: write d5/f29 [387077,116694] 0 2024-04-22T23:23:20.554 INFO:tasks.workunit.client.0.smithi067.stdout:3/216: mknod d3/d4/d1b/c43 0 2024-04-22T23:23:20.556 INFO:tasks.workunit.client.0.smithi067.stdout:3/217: stat d3/d4/d10/d19 0 2024-04-22T23:23:20.557 INFO:tasks.workunit.client.0.smithi067.stdout:2/131: write d1/fc [3597798,99507] 0 2024-04-22T23:23:20.559 INFO:tasks.workunit.client.0.smithi067.stdout:1/212: dwrite d8/d9/f25 [0,4194304] 0 2024-04-22T23:23:20.561 INFO:tasks.workunit.client.0.smithi067.stdout:1/213: read - d8/f26 zero size 2024-04-22T23:23:20.562 INFO:tasks.workunit.client.0.smithi067.stdout:3/218: mknod d3/d4/d10/d1f/d3c/c44 0 2024-04-22T23:23:20.566 INFO:tasks.workunit.client.0.smithi067.stdout:2/132: rename d1/d12/f1d to d1/d10/f24 0 2024-04-22T23:23:20.567 INFO:tasks.workunit.client.0.smithi067.stdout:2/133: readlink d1/d12/l1a 0 2024-04-22T23:23:20.568 INFO:tasks.workunit.client.0.smithi067.stdout:2/134: truncate d1/d10/d1c/f23 219213 0 2024-04-22T23:23:20.569 INFO:tasks.workunit.client.0.smithi067.stdout:2/135: rmdir d1/d10 39 2024-04-22T23:23:20.570 INFO:tasks.workunit.client.0.smithi067.stdout:2/136: chown d1/c18 255929 1 2024-04-22T23:23:20.570 INFO:tasks.workunit.client.0.smithi067.stdout:9/139: dread d5/f10 [0,4194304] 0 2024-04-22T23:23:20.573 INFO:tasks.workunit.client.0.smithi067.stdout:9/140: mkdir d5/d21/d2c 0 2024-04-22T23:23:20.575 INFO:tasks.workunit.client.0.smithi067.stdout:2/137: symlink d1/d10/l25 0 2024-04-22T23:23:20.578 INFO:tasks.workunit.client.0.smithi067.stdout:9/141: link d5/l17 d5/dd/l2d 0 2024-04-22T23:23:20.585 INFO:tasks.workunit.client.0.smithi067.stdout:9/142: dwrite d5/ff [0,4194304] 0 2024-04-22T23:23:20.587 INFO:tasks.workunit.client.0.smithi067.stdout:2/138: truncate d1/f3 7459991 0 2024-04-22T23:23:20.587 INFO:tasks.workunit.client.0.smithi067.stdout:9/143: write d5/dd/f1a [3544685,48078] 0 2024-04-22T23:23:20.587 INFO:tasks.workunit.client.0.smithi067.stdout:9/144: stat d5/f9 0 2024-04-22T23:23:20.588 INFO:tasks.workunit.client.0.smithi067.stdout:2/139: truncate d1/f1f 443301 0 2024-04-22T23:23:20.589 INFO:tasks.workunit.client.0.smithi067.stdout:2/140: truncate d1/d19/f21 377436 0 2024-04-22T23:23:20.591 INFO:tasks.workunit.client.0.smithi067.stdout:9/145: mkdir d5/d2e 0 2024-04-22T23:23:20.592 INFO:tasks.workunit.client.0.smithi067.stdout:2/141: rmdir d1/d10 39 2024-04-22T23:23:20.593 INFO:tasks.workunit.client.0.smithi067.stdout:2/142: chown d1/d12/f15 8249 1 2024-04-22T23:23:20.593 INFO:tasks.workunit.client.0.smithi067.stdout:2/143: chown d1/d12/l1a 8 1 2024-04-22T23:23:20.594 INFO:tasks.workunit.client.0.smithi067.stdout:9/146: unlink d5/dd/d18/l22 0 2024-04-22T23:23:20.594 INFO:tasks.workunit.client.0.smithi067.stdout:2/144: rmdir d1 39 2024-04-22T23:23:20.596 INFO:tasks.workunit.client.0.smithi067.stdout:9/147: readlink d5/dd/l2d 0 2024-04-22T23:23:20.596 INFO:tasks.workunit.client.0.smithi067.stdout:9/148: read d5/d21/f24 [2228327,12064] 0 2024-04-22T23:23:20.599 INFO:tasks.workunit.client.0.smithi067.stdout:9/149: write d5/f10 [1433076,92030] 0 2024-04-22T23:23:20.599 INFO:tasks.workunit.client.0.smithi067.stdout:2/145: mknod d1/d19/c26 0 2024-04-22T23:23:20.600 INFO:tasks.workunit.client.0.smithi067.stdout:2/146: write d1/d12/f16 [5119782,23141] 0 2024-04-22T23:23:20.605 INFO:tasks.workunit.client.0.smithi067.stdout:2/147: rmdir d1/d10/d1c 39 2024-04-22T23:23:20.611 INFO:tasks.workunit.client.0.smithi067.stdout:2/148: dwrite d1/d12/f15 [0,4194304] 0 2024-04-22T23:23:20.612 INFO:tasks.workunit.client.0.smithi067.stdout:2/149: write d1/f1b [502533,120798] 0 2024-04-22T23:23:20.620 INFO:tasks.workunit.client.0.smithi067.stdout:2/150: dread d1/f3 [4194304,4194304] 0 2024-04-22T23:23:20.623 INFO:tasks.workunit.client.0.smithi067.stdout:2/151: rename d1/d10/l25 to d1/l27 0 2024-04-22T23:23:20.623 INFO:tasks.workunit.client.0.smithi067.stdout:2/152: stat d1/f1f 0 2024-04-22T23:23:20.625 INFO:tasks.workunit.client.0.smithi067.stdout:2/153: chown d1/d10/d1c/c1e 220 1 2024-04-22T23:23:20.627 INFO:tasks.workunit.client.0.smithi067.stdout:2/154: mkdir d1/d12/d28 0 2024-04-22T23:23:20.645 INFO:tasks.workunit.client.1.smithi134.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-04-22T23:23:20.657 INFO:tasks.workunit.client.1.smithi134.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-22T23:23:20.658 INFO:tasks.workunit.client.1.smithi134.stderr:+ make 2024-04-22T23:23:20.688 INFO:tasks.workunit.client.1.smithi134.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-04-22T23:23:20.739 INFO:tasks.workunit.client.0.smithi067.stdout:7/98: sync 2024-04-22T23:23:20.740 INFO:tasks.workunit.client.0.smithi067.stdout:7/99: fsync d10/f12 0 2024-04-22T23:23:20.740 INFO:tasks.workunit.client.0.smithi067.stdout:7/100: chown d10/d14/l16 11734510 1 2024-04-22T23:23:20.746 INFO:tasks.workunit.client.0.smithi067.stdout:7/101: truncate d10/f13 952643 0 2024-04-22T23:23:20.755 INFO:tasks.workunit.client.0.smithi067.stdout:7/102: dwrite fe [0,4194304] 0 2024-04-22T23:23:20.757 INFO:tasks.workunit.client.0.smithi067.stdout:7/103: mknod d10/d18/d19/c1b 0 2024-04-22T23:23:20.760 INFO:tasks.workunit.client.0.smithi067.stdout:7/104: dread fb [0,4194304] 0 2024-04-22T23:23:20.762 INFO:tasks.workunit.client.0.smithi067.stdout:7/105: mknod d10/d14/c1c 0 2024-04-22T23:23:20.764 INFO:tasks.workunit.client.0.smithi067.stdout:7/106: mkdir d10/d14/d1d 0 2024-04-22T23:23:20.766 INFO:tasks.workunit.client.0.smithi067.stdout:7/107: creat d10/d18/f1e x:0 0 0 2024-04-22T23:23:20.775 INFO:tasks.workunit.client.0.smithi067.stdout:7/108: write d10/f13 [1200865,45453] 0 2024-04-22T23:23:20.777 INFO:tasks.workunit.client.0.smithi067.stdout:7/109: creat d10/d14/d1d/f1f x:0 0 0 2024-04-22T23:23:20.778 INFO:tasks.workunit.client.0.smithi067.stdout:7/110: symlink d10/d14/l20 0 2024-04-22T23:23:20.782 INFO:tasks.workunit.client.0.smithi067.stdout:7/111: mknod d10/d14/c21 0 2024-04-22T23:23:20.782 INFO:tasks.workunit.client.0.smithi067.stdout:7/112: write fd [1013876,4601] 0 2024-04-22T23:23:20.782 INFO:tasks.workunit.client.0.smithi067.stdout:7/113: write f5 [1480636,83826] 0 2024-04-22T23:23:20.783 INFO:tasks.workunit.client.0.smithi067.stdout:7/114: chown d10/d14/l20 0 1 2024-04-22T23:23:20.784 INFO:tasks.workunit.client.0.smithi067.stdout:7/115: truncate d10/f12 1003557 0 2024-04-22T23:23:20.850 INFO:tasks.workunit.client.0.smithi067.stdout:4/156: sync 2024-04-22T23:23:20.853 INFO:tasks.workunit.client.0.smithi067.stdout:4/157: rename d12/d17/d1e/f20 to d12/f32 0 2024-04-22T23:23:20.853 INFO:tasks.workunit.client.0.smithi067.stdout:4/158: chown d12/d17/l29 1667127 1 2024-04-22T23:23:20.855 INFO:tasks.workunit.client.0.smithi067.stdout:4/159: creat d12/d17/f33 x:0 0 0 2024-04-22T23:23:20.856 INFO:tasks.workunit.client.0.smithi067.stdout:4/160: chown d12/f19 16761194 1 2024-04-22T23:23:20.857 INFO:tasks.workunit.client.0.smithi067.stdout:4/161: write fd [256717,105860] 0 2024-04-22T23:23:20.863 INFO:tasks.workunit.client.0.smithi067.stdout:4/162: dwrite f8 [0,4194304] 0 2024-04-22T23:23:20.865 INFO:tasks.workunit.client.0.smithi067.stdout:4/163: mknod d12/d17/c34 0 2024-04-22T23:23:20.866 INFO:tasks.workunit.client.0.smithi067.stdout:4/164: write d12/d17/f33 [210904,19871] 0 2024-04-22T23:23:20.868 INFO:tasks.workunit.client.0.smithi067.stdout:4/165: creat d12/d17/d1e/f35 x:0 0 0 2024-04-22T23:23:20.871 INFO:tasks.workunit.client.0.smithi067.stdout:4/166: rename d12/d17/l30 to d12/d17/d1e/d2b/l36 0 2024-04-22T23:23:20.879 INFO:tasks.workunit.client.0.smithi067.stdout:4/167: fsync fd 0 2024-04-22T23:23:20.880 INFO:tasks.workunit.client.0.smithi067.stdout:4/168: write fa [2946539,49710] 0 2024-04-22T23:23:20.883 INFO:tasks.workunit.client.0.smithi067.stdout:4/169: symlink d12/d17/l37 0 2024-04-22T23:23:20.884 INFO:tasks.workunit.client.0.smithi067.stdout:4/170: stat d12/d17/l26 0 2024-04-22T23:23:20.888 INFO:tasks.workunit.client.0.smithi067.stdout:4/171: write fe [2156580,20946] 0 2024-04-22T23:23:20.889 INFO:tasks.workunit.client.0.smithi067.stdout:4/172: write d12/d17/f2f [502234,18435] 0 2024-04-22T23:23:20.892 INFO:tasks.workunit.client.0.smithi067.stdout:4/173: creat d12/f38 x:0 0 0 2024-04-22T23:23:20.897 INFO:tasks.workunit.client.0.smithi067.stdout:4/174: dwrite fe [0,4194304] 0 2024-04-22T23:23:20.898 INFO:tasks.workunit.client.0.smithi067.stdout:8/201: sync 2024-04-22T23:23:20.899 INFO:tasks.workunit.client.0.smithi067.stdout:5/147: sync 2024-04-22T23:23:20.900 INFO:tasks.workunit.client.0.smithi067.stdout:5/148: chown d5/f1b 1334 1 2024-04-22T23:23:20.900 INFO:tasks.workunit.client.0.smithi067.stdout:4/175: creat d12/d17/d31/f39 x:0 0 0 2024-04-22T23:23:20.901 INFO:tasks.workunit.client.0.smithi067.stdout:8/202: mkdir d1/d41 0 2024-04-22T23:23:20.903 INFO:tasks.workunit.client.0.smithi067.stdout:8/203: mknod d1/d3/c42 0 2024-04-22T23:23:20.905 INFO:tasks.workunit.client.0.smithi067.stdout:4/176: dwrite fe [0,4194304] 0 2024-04-22T23:23:20.908 INFO:tasks.workunit.client.0.smithi067.stdout:4/177: symlink d12/d17/d1e/l3a 0 2024-04-22T23:23:20.909 INFO:tasks.workunit.client.0.smithi067.stdout:4/178: dread - d12/d17/d1e/f35 zero size 2024-04-22T23:23:20.909 INFO:tasks.workunit.client.0.smithi067.stdout:4/179: chown d12/l1c 0 1 2024-04-22T23:23:20.910 INFO:tasks.workunit.client.0.smithi067.stdout:4/180: read ff [150324,118580] 0 2024-04-22T23:23:20.940 INFO:tasks.workunit.client.0.smithi067.stdout:4/181: symlink d12/d17/d1e/l3b 0 2024-04-22T23:23:20.942 INFO:tasks.workunit.client.0.smithi067.stdout:4/182: rename l1 to d12/d17/l3c 0 2024-04-22T23:23:20.953 INFO:tasks.workunit.client.0.smithi067.stdout:4/183: dread d12/f32 [0,4194304] 0 2024-04-22T23:23:20.955 INFO:tasks.workunit.client.0.smithi067.stdout:4/184: dread fd [0,4194304] 0 2024-04-22T23:23:20.958 INFO:tasks.workunit.client.0.smithi067.stdout:4/185: link d12/d17/d1e/l3b d12/l3d 0 2024-04-22T23:23:20.960 INFO:tasks.workunit.client.0.smithi067.stdout:4/186: creat d12/d2e/f3e x:0 0 0 2024-04-22T23:23:20.966 INFO:tasks.workunit.client.0.smithi067.stdout:4/187: dwrite d12/d17/f2f [0,4194304] 0 2024-04-22T23:23:20.972 INFO:tasks.workunit.client.0.smithi067.stdout:4/188: getdents d12/d17 0 2024-04-22T23:23:20.975 INFO:tasks.workunit.client.0.smithi067.stdout:4/189: getdents d12 0 2024-04-22T23:23:20.977 INFO:tasks.workunit.client.0.smithi067.stdout:4/190: rmdir d12/d17/d31 39 2024-04-22T23:23:20.978 INFO:tasks.workunit.client.0.smithi067.stdout:4/191: readlink d12/d17/l1b 0 2024-04-22T23:23:20.980 INFO:tasks.workunit.client.0.smithi067.stdout:4/192: unlink d12/l14 0 2024-04-22T23:23:20.982 INFO:tasks.workunit.client.0.smithi067.stdout:4/193: mkdir d12/d17/d3f 0 2024-04-22T23:23:20.986 INFO:tasks.workunit.client.0.smithi067.stdout:4/194: mknod d12/d17/d3f/c40 0 2024-04-22T23:23:20.986 INFO:tasks.workunit.client.0.smithi067.stdout:4/195: rename d12/f19 to d12/d17/d1e/f41 0 2024-04-22T23:23:20.987 INFO:tasks.workunit.client.0.smithi067.stdout:4/196: mkdir d12/d42 0 2024-04-22T23:23:20.988 INFO:tasks.workunit.client.0.smithi067.stdout:4/197: mknod d12/d2e/c43 0 2024-04-22T23:23:20.989 INFO:tasks.workunit.client.0.smithi067.stdout:4/198: chown d12/d17/d31 2877 1 2024-04-22T23:23:20.991 INFO:tasks.workunit.client.0.smithi067.stdout:4/199: creat d12/d42/f44 x:0 0 0 2024-04-22T23:23:20.993 INFO:tasks.workunit.client.0.smithi067.stdout:4/200: dread d12/d17/d1e/f27 [0,4194304] 0 2024-04-22T23:23:20.995 INFO:tasks.workunit.client.0.smithi067.stdout:4/201: mknod d12/d2e/c45 0 2024-04-22T23:23:20.996 INFO:tasks.workunit.client.0.smithi067.stdout:4/202: creat d12/f46 x:0 0 0 2024-04-22T23:23:20.998 INFO:tasks.workunit.client.0.smithi067.stdout:4/203: unlink d12/d2e/c43 0 2024-04-22T23:23:20.999 INFO:tasks.workunit.client.0.smithi067.stdout:4/204: write d12/d17/d1e/f28 [1777362,48254] 0 2024-04-22T23:23:20.999 INFO:tasks.workunit.client.0.smithi067.stdout:0/145: sync 2024-04-22T23:23:20.999 INFO:tasks.workunit.client.0.smithi067.stdout:2/155: sync 2024-04-22T23:23:20.999 INFO:tasks.workunit.client.0.smithi067.stdout:3/219: sync 2024-04-22T23:23:20.999 INFO:tasks.workunit.client.0.smithi067.stdout:1/214: sync 2024-04-22T23:23:20.999 INFO:tasks.workunit.client.0.smithi067.stdout:9/150: sync 2024-04-22T23:23:21.000 INFO:tasks.workunit.client.0.smithi067.stdout:6/122: sync 2024-04-22T23:23:21.000 INFO:tasks.workunit.client.0.smithi067.stdout:7/116: sync 2024-04-22T23:23:21.001 INFO:tasks.workunit.client.0.smithi067.stdout:4/205: write fd [3845198,3046] 0 2024-04-22T23:23:21.002 INFO:tasks.workunit.client.0.smithi067.stdout:6/123: rmdir d17 39 2024-04-22T23:23:21.002 INFO:tasks.workunit.client.0.smithi067.stdout:3/220: dread - d3/d4/d12/f1d zero size 2024-04-22T23:23:21.004 INFO:tasks.workunit.client.0.smithi067.stdout:4/206: write fd [1517456,50923] 0 2024-04-22T23:23:21.007 INFO:tasks.workunit.client.0.smithi067.stdout:4/207: chown d12/c13 28 1 2024-04-22T23:23:21.010 INFO:tasks.workunit.client.0.smithi067.stdout:3/221: rmdir d3/d4/d1b 39 2024-04-22T23:23:21.013 INFO:tasks.workunit.client.0.smithi067.stdout:0/146: dwrite f0 [0,4194304] 0 2024-04-22T23:23:21.017 INFO:tasks.workunit.client.0.smithi067.stdout:7/117: dwrite d10/f15 [0,4194304] 0 2024-04-22T23:23:21.018 INFO:tasks.workunit.client.0.smithi067.stdout:2/156: dwrite d1/d10/d1c/f23 [0,4194304] 0 2024-04-22T23:23:21.018 INFO:tasks.workunit.client.0.smithi067.stdout:1/215: creat d8/f4b x:0 0 0 2024-04-22T23:23:21.018 INFO:tasks.workunit.client.0.smithi067.stdout:3/222: chown d3/d4/d10/d19/f34 7362 1 2024-04-22T23:23:21.021 INFO:tasks.workunit.client.0.smithi067.stdout:7/118: write fe [805015,67640] 0 2024-04-22T23:23:21.025 INFO:tasks.workunit.client.0.smithi067.stdout:1/216: dread d8/d9/f3b [0,4194304] 0 2024-04-22T23:23:21.026 INFO:tasks.workunit.client.0.smithi067.stdout:4/208: dwrite fa [0,4194304] 0 2024-04-22T23:23:21.028 INFO:tasks.workunit.client.0.smithi067.stdout:2/157: dread d1/f1b [0,4194304] 0 2024-04-22T23:23:21.033 INFO:tasks.workunit.client.0.smithi067.stdout:3/223: dread d3/d4/d10/f39 [0,4194304] 0 2024-04-22T23:23:21.037 INFO:tasks.workunit.client.0.smithi067.stdout:2/158: dwrite d1/f1f [0,4194304] 0 2024-04-22T23:23:21.039 INFO:tasks.workunit.client.0.smithi067.stdout:2/159: write d1/d19/f21 [107836,47955] 0 2024-04-22T23:23:21.040 INFO:tasks.workunit.client.0.smithi067.stdout:2/160: write d1/f1f [2340666,96097] 0 2024-04-22T23:23:21.041 INFO:tasks.workunit.client.0.smithi067.stdout:2/161: chown d1/d12/f16 52732277 1 2024-04-22T23:23:21.067 INFO:tasks.workunit.client.0.smithi067.stdout:6/124: rename l11 to d17/l2a 0 2024-04-22T23:23:21.069 INFO:tasks.workunit.client.0.smithi067.stdout:0/147: mknod db/c34 0 2024-04-22T23:23:21.074 INFO:tasks.workunit.client.0.smithi067.stdout:2/162: rmdir d1/d10/d1c 39 2024-04-22T23:23:21.076 INFO:tasks.workunit.client.0.smithi067.stdout:2/163: write d1/d12/f15 [3925647,130093] 0 2024-04-22T23:23:21.076 INFO:tasks.workunit.client.0.smithi067.stdout:6/125: creat d17/f2b x:0 0 0 2024-04-22T23:23:21.077 INFO:tasks.workunit.client.0.smithi067.stdout:6/126: dread - d17/f2b zero size 2024-04-22T23:23:21.078 INFO:tasks.workunit.client.0.smithi067.stdout:6/127: truncate fe 1731087 0 2024-04-22T23:23:21.080 INFO:tasks.workunit.client.0.smithi067.stdout:0/148: dread db/de/d11/d1a/f28 [0,4194304] 0 2024-04-22T23:23:21.082 INFO:tasks.workunit.client.0.smithi067.stdout:4/209: mkdir d12/d17/d47 0 2024-04-22T23:23:21.082 INFO:tasks.workunit.client.0.smithi067.stdout:5/149: sync 2024-04-22T23:23:21.082 INFO:tasks.workunit.client.0.smithi067.stdout:4/210: readlink d12/l1c 0 2024-04-22T23:23:21.083 INFO:tasks.workunit.client.0.smithi067.stdout:3/224: creat d3/d4/d28/d40/f45 x:0 0 0 2024-04-22T23:23:21.088 INFO:tasks.workunit.client.0.smithi067.stdout:5/150: dwrite d5/db/d12/f13 [0,4194304] 0 2024-04-22T23:23:21.089 INFO:tasks.workunit.client.0.smithi067.stdout:5/151: read - d5/f31 zero size 2024-04-22T23:23:21.091 INFO:tasks.workunit.client.0.smithi067.stdout:4/211: mkdir d12/d17/d1e/d48 0 2024-04-22T23:23:21.095 INFO:tasks.workunit.client.0.smithi067.stdout:7/119: getdents d10/d18/d19 0 2024-04-22T23:23:21.095 INFO:tasks.workunit.client.0.smithi067.stdout:7/120: stat d10/d18/f1e 0 2024-04-22T23:23:21.096 INFO:tasks.workunit.client.0.smithi067.stdout:0/149: creat db/d21/d29/d2d/f35 x:0 0 0 2024-04-22T23:23:21.096 INFO:tasks.workunit.client.0.smithi067.stdout:2/164: write d1/f3 [7510352,76088] 0 2024-04-22T23:23:21.097 INFO:tasks.workunit.client.0.smithi067.stdout:1/217: getdents d8/d9/d2f 0 2024-04-22T23:23:21.098 INFO:tasks.workunit.client.0.smithi067.stdout:2/165: stat d1/d12 0 2024-04-22T23:23:21.099 INFO:tasks.workunit.client.0.smithi067.stdout:2/166: chown d1/f4 99 1 2024-04-22T23:23:21.100 INFO:tasks.workunit.client.0.smithi067.stdout:4/212: chown d12/l3d 38 1 2024-04-22T23:23:21.101 INFO:tasks.workunit.client.0.smithi067.stdout:3/225: symlink d3/d4/d1b/l46 0 2024-04-22T23:23:21.102 INFO:tasks.workunit.client.0.smithi067.stdout:7/121: symlink d10/l22 0 2024-04-22T23:23:21.102 INFO:tasks.workunit.client.0.smithi067.stdout:0/150: rmdir db/d21/d29 39 2024-04-22T23:23:21.103 INFO:tasks.workunit.client.0.smithi067.stdout:6/128: dwrite f12 [0,4194304] 0 2024-04-22T23:23:21.104 INFO:tasks.workunit.client.0.smithi067.stdout:5/152: getdents d5/db/d12/d2a 0 2024-04-22T23:23:21.105 INFO:tasks.workunit.client.0.smithi067.stdout:7/122: readlink d10/d14/l20 0 2024-04-22T23:23:21.106 INFO:tasks.workunit.client.0.smithi067.stdout:4/213: rename f7 to d12/d17/d47/f49 0 2024-04-22T23:23:21.106 INFO:tasks.workunit.client.0.smithi067.stdout:6/129: write f12 [4859104,36786] 0 2024-04-22T23:23:21.107 INFO:tasks.workunit.client.0.smithi067.stdout:5/153: truncate d5/db/f2f 548573 0 2024-04-22T23:23:21.108 INFO:tasks.workunit.client.0.smithi067.stdout:1/218: creat d8/d12/d36/f4c x:0 0 0 2024-04-22T23:23:21.108 INFO:tasks.workunit.client.0.smithi067.stdout:3/226: chown d3/d4/d10/d1f/f3f 320574604 1 2024-04-22T23:23:21.110 INFO:tasks.workunit.client.0.smithi067.stdout:3/227: write d3/d4/fc [1304419,9413] 0 2024-04-22T23:23:21.112 INFO:tasks.workunit.client.0.smithi067.stdout:7/123: mknod d10/c23 0 2024-04-22T23:23:21.114 INFO:tasks.workunit.client.0.smithi067.stdout:2/167: dread d1/d10/f24 [0,4194304] 0 2024-04-22T23:23:21.117 INFO:tasks.workunit.client.0.smithi067.stdout:1/219: rename d8/d9/df to d8/d9/d2d/d4d 0 2024-04-22T23:23:21.118 INFO:tasks.workunit.client.0.smithi067.stdout:3/228: rename d3 to d3/d4/d10/d1f/d47 22 2024-04-22T23:23:21.119 INFO:tasks.workunit.client.0.smithi067.stdout:0/151: symlink db/d21/l36 0 2024-04-22T23:23:21.122 INFO:tasks.workunit.client.0.smithi067.stdout:2/168: symlink d1/d12/l29 0 2024-04-22T23:23:21.125 INFO:tasks.workunit.client.0.smithi067.stdout:1/220: creat d8/d12/d36/f4e x:0 0 0 2024-04-22T23:23:21.128 INFO:tasks.workunit.client.0.smithi067.stdout:6/130: dwrite f6 [4194304,4194304] 0 2024-04-22T23:23:21.131 INFO:tasks.workunit.client.0.smithi067.stdout:5/154: dwrite d5/fa [0,4194304] 0 2024-04-22T23:23:21.131 INFO:tasks.workunit.client.0.smithi067.stdout:0/152: mknod db/de/d11/d1a/c37 0 2024-04-22T23:23:21.131 INFO:tasks.workunit.client.0.smithi067.stdout:7/124: dwrite d10/d14/d1d/f1f [0,4194304] 0 2024-04-22T23:23:21.136 INFO:tasks.workunit.client.0.smithi067.stdout:3/229: dread d3/d4/d10/d19/f34 [0,4194304] 0 2024-04-22T23:23:21.138 INFO:tasks.workunit.client.0.smithi067.stdout:4/214: link d12/d17/l21 d12/d17/d31/l4a 0 2024-04-22T23:23:21.138 INFO:tasks.workunit.client.0.smithi067.stdout:0/153: chown c4 123783577 1 2024-04-22T23:23:21.139 INFO:tasks.workunit.client.0.smithi067.stdout:6/131: mknod d17/c2c 0 2024-04-22T23:23:21.139 INFO:tasks.workunit.client.0.smithi067.stdout:6/132: truncate d17/f24 961255 0 2024-04-22T23:23:21.140 INFO:tasks.workunit.client.0.smithi067.stdout:0/154: fsync db/de/d11/f27 0 2024-04-22T23:23:21.142 INFO:tasks.workunit.client.0.smithi067.stdout:5/155: readlink d5/db/d12/l18 0 2024-04-22T23:23:21.143 INFO:tasks.workunit.client.0.smithi067.stdout:2/169: getdents d1/d12/d28 0 2024-04-22T23:23:21.143 INFO:tasks.workunit.client.0.smithi067.stdout:2/170: chown d1/d19 4277 1 2024-04-22T23:23:21.144 INFO:tasks.workunit.client.0.smithi067.stdout:4/215: mkdir d12/d42/d4b 0 2024-04-22T23:23:21.145 INFO:tasks.workunit.client.0.smithi067.stdout:8/204: sync 2024-04-22T23:23:21.156 INFO:tasks.workunit.client.0.smithi067.stdout:6/133: unlink d17/l1c 0 2024-04-22T23:23:21.157 INFO:tasks.workunit.client.0.smithi067.stdout:3/230: creat d3/d4/d10/d38/f48 x:0 0 0 2024-04-22T23:23:21.160 INFO:tasks.workunit.client.0.smithi067.stdout:4/216: rmdir d12/d17/d1e 39 2024-04-22T23:23:21.162 INFO:tasks.workunit.client.0.smithi067.stdout:8/205: write d1/d3/d10/f29 [777392,99603] 0 2024-04-22T23:23:21.163 INFO:tasks.workunit.client.0.smithi067.stdout:6/134: dwrite f13 [0,4194304] 0 2024-04-22T23:23:21.164 INFO:tasks.workunit.client.0.smithi067.stdout:7/125: truncate f5 1373494 0 2024-04-22T23:23:21.164 INFO:tasks.workunit.client.0.smithi067.stdout:1/221: link d8/f11 d8/d9/d2d/f4f 0 2024-04-22T23:23:21.164 INFO:tasks.workunit.client.0.smithi067.stdout:6/135: write d17/f25 [995814,103749] 0 2024-04-22T23:23:21.166 INFO:tasks.workunit.client.0.smithi067.stdout:6/136: chown la 1 1 2024-04-22T23:23:21.167 INFO:tasks.workunit.client.0.smithi067.stdout:1/222: truncate d8/f4b 735868 0 2024-04-22T23:23:21.167 INFO:tasks.workunit.client.0.smithi067.stdout:5/156: mkdir d5/db/d35 0 2024-04-22T23:23:21.169 INFO:tasks.workunit.client.0.smithi067.stdout:2/171: creat d1/d12/d28/f2a x:0 0 0 2024-04-22T23:23:21.170 INFO:tasks.workunit.client.0.smithi067.stdout:3/231: rename d3/d4/d10/d1f/d3c/l3a to d3/d4/d10/d1f/l49 0 2024-04-22T23:23:21.171 INFO:tasks.workunit.client.0.smithi067.stdout:4/217: chown d12/d17/l26 32 1 2024-04-22T23:23:21.172 INFO:tasks.workunit.client.0.smithi067.stdout:8/206: chown d1/d3/f2a 6 1 2024-04-22T23:23:21.172 INFO:tasks.workunit.client.0.smithi067.stdout:7/126: fsync fc 0 2024-04-22T23:23:21.173 INFO:tasks.workunit.client.0.smithi067.stdout:8/207: stat d1/d3 0 2024-04-22T23:23:21.173 INFO:tasks.workunit.client.0.smithi067.stdout:8/208: stat d1/d3/d10/f3d 0 2024-04-22T23:23:21.174 INFO:tasks.workunit.client.0.smithi067.stdout:6/137: stat l14 0 2024-04-22T23:23:21.175 INFO:tasks.workunit.client.0.smithi067.stdout:6/138: stat la 0 2024-04-22T23:23:21.176 INFO:tasks.workunit.client.0.smithi067.stdout:1/223: mknod d8/d9/d2f/d33/c50 0 2024-04-22T23:23:21.181 INFO:tasks.workunit.client.0.smithi067.stdout:4/218: symlink d12/d17/d47/l4c 0 2024-04-22T23:23:21.184 INFO:tasks.workunit.client.0.smithi067.stdout:7/127: mkdir d10/d14/d1d/d24 0 2024-04-22T23:23:21.186 INFO:tasks.workunit.client.0.smithi067.stdout:6/139: rmdir d17 39 2024-04-22T23:23:21.186 INFO:tasks.workunit.client.0.smithi067.stdout:5/157: mkdir d5/d28/d36 0 2024-04-22T23:23:21.187 INFO:tasks.workunit.client.0.smithi067.stdout:1/224: mknod d8/d9/d2d/d4d/d48/c51 0 2024-04-22T23:23:21.190 INFO:tasks.workunit.client.0.smithi067.stdout:2/172: dwrite d1/f1b [0,4194304] 0 2024-04-22T23:23:21.191 INFO:tasks.workunit.client.0.smithi067.stdout:5/158: truncate d5/db/f2f 1325344 0 2024-04-22T23:23:21.193 INFO:tasks.workunit.client.0.smithi067.stdout:7/128: rename fd to d10/d18/f25 0 2024-04-22T23:23:21.196 INFO:tasks.workunit.client.0.smithi067.stdout:8/209: dwrite d1/d3/d14/f30 [0,4194304] 0 2024-04-22T23:23:21.197 INFO:tasks.workunit.client.0.smithi067.stdout:4/219: mkdir d12/d17/d1e/d2b/d4d 0 2024-04-22T23:23:21.199 INFO:tasks.workunit.client.0.smithi067.stdout:3/232: dread d3/f35 [0,4194304] 0 2024-04-22T23:23:21.201 INFO:tasks.workunit.client.0.smithi067.stdout:5/159: dread d5/db/ff [0,4194304] 0 2024-04-22T23:23:21.202 INFO:tasks.workunit.client.0.smithi067.stdout:2/173: rename d1/c17 to d1/d12/c2b 0 2024-04-22T23:23:21.204 INFO:tasks.workunit.client.0.smithi067.stdout:3/233: write d3/d4/d12/f2f [3770453,68309] 0 2024-04-22T23:23:21.206 INFO:tasks.workunit.client.0.smithi067.stdout:7/129: dread d10/d14/d1d/f1f [0,4194304] 0 2024-04-22T23:23:21.207 INFO:tasks.workunit.client.0.smithi067.stdout:6/140: rename d17/c2c to d17/c2d 0 2024-04-22T23:23:21.209 INFO:tasks.workunit.client.0.smithi067.stdout:4/220: write ff [2766608,71193] 0 2024-04-22T23:23:21.209 INFO:tasks.workunit.client.0.smithi067.stdout:6/141: write d17/f20 [645260,130227] 0 2024-04-22T23:23:21.210 INFO:tasks.workunit.client.0.smithi067.stdout:4/221: fdatasync d12/f38 0 2024-04-22T23:23:21.211 INFO:tasks.workunit.client.0.smithi067.stdout:5/160: creat d5/db/d12/f37 x:0 0 0 2024-04-22T23:23:21.212 INFO:tasks.workunit.client.0.smithi067.stdout:3/234: mkdir d3/d4/d28/d4a 0 2024-04-22T23:23:21.213 INFO:tasks.workunit.client.0.smithi067.stdout:5/161: dread d5/d32/f16 [4194304,4194304] 0 2024-04-22T23:23:21.218 INFO:tasks.workunit.client.0.smithi067.stdout:7/130: rename d10/d18/d19 to d10/d14/d1d/d26 0 2024-04-22T23:23:21.219 INFO:tasks.workunit.client.0.smithi067.stdout:7/131: fsync d10/f15 0 2024-04-22T23:23:21.220 INFO:tasks.workunit.client.0.smithi067.stdout:6/142: mknod d17/c2e 0 2024-04-22T23:23:21.221 INFO:tasks.workunit.client.0.smithi067.stdout:4/222: write d12/d17/d1e/f27 [640393,78953] 0 2024-04-22T23:23:21.221 INFO:tasks.workunit.client.0.smithi067.stdout:1/225: getdents d8/d12/d36 0 2024-04-22T23:23:21.222 INFO:tasks.workunit.client.0.smithi067.stdout:3/235: mkdir d3/d4/d12/d4b 0 2024-04-22T23:23:21.227 INFO:tasks.workunit.client.0.smithi067.stdout:6/143: unlink d17/f1f 0 2024-04-22T23:23:21.228 INFO:tasks.workunit.client.0.smithi067.stdout:4/223: mknod d12/d17/d31/c4e 0 2024-04-22T23:23:21.230 INFO:tasks.workunit.client.0.smithi067.stdout:5/162: symlink d5/db/d35/l38 0 2024-04-22T23:23:21.230 INFO:tasks.workunit.client.0.smithi067.stdout:6/144: dread d17/f21 [0,4194304] 0 2024-04-22T23:23:21.230 INFO:tasks.workunit.client.0.smithi067.stdout:6/145: chown f6 938094 1 2024-04-22T23:23:21.231 INFO:tasks.workunit.client.0.smithi067.stdout:1/226: mknod d8/d12/d36/c52 0 2024-04-22T23:23:21.231 INFO:tasks.workunit.client.0.smithi067.stdout:6/146: truncate d17/f29 372499 0 2024-04-22T23:23:21.232 INFO:tasks.workunit.client.0.smithi067.stdout:2/174: truncate d1/d12/f16 1093455 0 2024-04-22T23:23:21.233 INFO:tasks.workunit.client.0.smithi067.stdout:1/227: write d8/d24/d34/f37 [1493344,77126] 0 2024-04-22T23:23:21.233 INFO:tasks.workunit.client.0.smithi067.stdout:3/236: mkdir d3/d4/d28/d4c 0 2024-04-22T23:23:21.233 INFO:tasks.workunit.client.0.smithi067.stdout:1/228: chown d8/d12 6131740 1 2024-04-22T23:23:21.233 INFO:tasks.workunit.client.0.smithi067.stdout:8/210: truncate d1/d3/f11 817949 0 2024-04-22T23:23:21.235 INFO:tasks.workunit.client.0.smithi067.stdout:7/132: write fb [2172135,126526] 0 2024-04-22T23:23:21.238 INFO:tasks.workunit.client.0.smithi067.stdout:5/163: creat d5/db/d35/f39 x:0 0 0 2024-04-22T23:23:21.240 INFO:tasks.workunit.client.0.smithi067.stdout:6/147: rmdir d17 39 2024-04-22T23:23:21.240 INFO:tasks.workunit.client.0.smithi067.stdout:6/148: write f13 [710440,54047] 0 2024-04-22T23:23:21.241 INFO:tasks.workunit.client.0.smithi067.stdout:2/175: mknod d1/d19/c2c 0 2024-04-22T23:23:21.242 INFO:tasks.workunit.client.0.smithi067.stdout:2/176: read d1/fc [725021,25281] 0 2024-04-22T23:23:21.243 INFO:tasks.workunit.client.0.smithi067.stdout:2/177: stat d1/d12/f15 0 2024-04-22T23:23:21.243 INFO:tasks.workunit.client.0.smithi067.stdout:3/237: mknod d3/d4/d10/d38/c4d 0 2024-04-22T23:23:21.243 INFO:tasks.workunit.client.0.smithi067.stdout:2/178: chown d1/d12/f15 1891893960 1 2024-04-22T23:23:21.244 INFO:tasks.workunit.client.0.smithi067.stdout:1/229: mknod d8/d9/d2d/c53 0 2024-04-22T23:23:21.244 INFO:tasks.workunit.client.0.smithi067.stdout:2/179: write d1/f1f [589287,105073] 0 2024-04-22T23:23:21.248 INFO:tasks.workunit.client.0.smithi067.stdout:8/211: rename d1/dd/d26/l3c to d1/d41/l43 0 2024-04-22T23:23:21.250 INFO:tasks.workunit.client.0.smithi067.stdout:1/230: dread d8/fc [0,4194304] 0 2024-04-22T23:23:21.251 INFO:tasks.workunit.client.0.smithi067.stdout:4/224: symlink d12/d17/l4f 0 2024-04-22T23:23:21.251 INFO:tasks.workunit.client.0.smithi067.stdout:1/231: stat d8/d9/d2d/d4d/d48/c51 0 2024-04-22T23:23:21.252 INFO:tasks.workunit.client.0.smithi067.stdout:4/225: dread - d12/d17/d31/f39 zero size 2024-04-22T23:23:21.252 INFO:tasks.workunit.client.0.smithi067.stdout:1/232: stat d8/d12/d40/f44 0 2024-04-22T23:23:21.255 INFO:tasks.workunit.client.0.smithi067.stdout:9/151: sync 2024-04-22T23:23:21.292 INFO:tasks.workunit.client.0.smithi067.stdout:3/238: dread d3/d4/d10/d19/f34 [0,4194304] 0 2024-04-22T23:23:21.294 INFO:tasks.workunit.client.0.smithi067.stdout:7/133: unlink l8 0 2024-04-22T23:23:21.296 INFO:tasks.workunit.client.0.smithi067.stdout:8/212: unlink d1/dd/c23 0 2024-04-22T23:23:21.297 INFO:tasks.workunit.client.0.smithi067.stdout:5/164: mkdir d5/d3a 0 2024-04-22T23:23:21.298 INFO:tasks.workunit.client.0.smithi067.stdout:1/233: creat d8/d12/f54 x:0 0 0 2024-04-22T23:23:21.300 INFO:tasks.workunit.client.0.smithi067.stdout:4/226: rename f8 to d12/d17/d31/f50 0 2024-04-22T23:23:21.302 INFO:tasks.workunit.client.0.smithi067.stdout:3/239: chown d3/d4/d10/d1f/l49 524 1 2024-04-22T23:23:21.303 INFO:tasks.workunit.client.0.smithi067.stdout:7/134: creat d10/d18/f27 x:0 0 0 2024-04-22T23:23:21.304 INFO:tasks.workunit.client.0.smithi067.stdout:7/135: write d10/f15 [835502,31794] 0 2024-04-22T23:23:21.305 INFO:tasks.workunit.client.0.smithi067.stdout:7/136: fsync d10/f12 0 2024-04-22T23:23:21.305 INFO:tasks.workunit.client.0.smithi067.stdout:8/213: chown d1/d3/c24 14937045 1 2024-04-22T23:23:21.307 INFO:tasks.workunit.client.0.smithi067.stdout:1/234: fsync d8/d9/d2d/f4f 0 2024-04-22T23:23:21.308 INFO:tasks.workunit.client.0.smithi067.stdout:6/149: rename f8 to d17/f2f 0 2024-04-22T23:23:21.309 INFO:tasks.workunit.client.0.smithi067.stdout:9/152: symlink d5/d21/d2c/l2f 0 2024-04-22T23:23:21.309 INFO:tasks.workunit.client.0.smithi067.stdout:6/150: chown f13 212139582 1 2024-04-22T23:23:21.310 INFO:tasks.workunit.client.0.smithi067.stdout:6/151: write d17/f1b [817005,85308] 0 2024-04-22T23:23:21.310 INFO:tasks.workunit.client.0.smithi067.stdout:6/152: chown d17/l1d 28 1 2024-04-22T23:23:21.314 INFO:tasks.workunit.client.0.smithi067.stdout:7/137: mkdir d10/d28 0 2024-04-22T23:23:21.318 INFO:tasks.workunit.client.0.smithi067.stdout:1/235: write d8/d9/d2d/d4d/f1f [384671,122133] 0 2024-04-22T23:23:21.322 INFO:tasks.workunit.client.0.smithi067.stdout:6/153: creat d17/f30 x:0 0 0 2024-04-22T23:23:21.324 INFO:tasks.workunit.client.0.smithi067.stdout:9/153: dread d5/d21/f24 [0,4194304] 0 2024-04-22T23:23:21.325 INFO:tasks.workunit.client.0.smithi067.stdout:8/214: write d1/d3/f13 [4997626,74084] 0 2024-04-22T23:23:21.326 INFO:tasks.workunit.client.0.smithi067.stdout:6/154: creat d17/f31 x:0 0 0 2024-04-22T23:23:21.327 INFO:tasks.workunit.client.0.smithi067.stdout:6/155: chown f10 189551 1 2024-04-22T23:23:21.329 INFO:tasks.workunit.client.0.smithi067.stdout:4/227: rename d12/d42/f44 to d12/d17/f51 0 2024-04-22T23:23:21.329 INFO:tasks.workunit.client.0.smithi067.stdout:8/215: dread d1/d3/d10/f29 [0,4194304] 0 2024-04-22T23:23:21.331 INFO:tasks.workunit.client.0.smithi067.stdout:5/165: truncate d5/d32/f20 2853161 0 2024-04-22T23:23:21.333 INFO:tasks.workunit.client.0.smithi067.stdout:4/228: dread fe [0,4194304] 0 2024-04-22T23:23:21.333 INFO:tasks.workunit.client.0.smithi067.stdout:7/138: link fe d10/d28/f29 0 2024-04-22T23:23:21.334 INFO:tasks.workunit.client.0.smithi067.stdout:6/156: creat d17/f32 x:0 0 0 2024-04-22T23:23:21.335 INFO:tasks.workunit.client.0.smithi067.stdout:9/154: symlink d5/l30 0 2024-04-22T23:23:21.336 INFO:tasks.workunit.client.0.smithi067.stdout:9/155: chown d5/d2e 174198 1 2024-04-22T23:23:21.338 INFO:tasks.workunit.client.0.smithi067.stdout:1/236: rename d8/d12/d36/f4c to d8/d9/d2d/d4d/d48/f55 0 2024-04-22T23:23:21.339 INFO:tasks.workunit.client.0.smithi067.stdout:7/139: rename d10/d14 to d10/d14/d1d/d24/d2a 22 2024-04-22T23:23:21.340 INFO:tasks.workunit.client.0.smithi067.stdout:5/166: dwrite d5/db/d12/f22 [0,4194304] 0 2024-04-22T23:23:21.341 INFO:tasks.workunit.client.0.smithi067.stdout:1/237: write d8/d12/f54 [299801,1300] 0 2024-04-22T23:23:21.342 INFO:tasks.workunit.client.0.smithi067.stdout:7/140: chown d10/d18/f1a 226312 1 2024-04-22T23:23:21.344 INFO:tasks.workunit.client.0.smithi067.stdout:8/216: mknod d1/c44 0 2024-04-22T23:23:21.345 INFO:tasks.workunit.client.0.smithi067.stdout:5/167: write d5/db/f2f [1872627,14141] 0 2024-04-22T23:23:21.346 INFO:tasks.workunit.client.0.smithi067.stdout:6/157: dwrite f6 [4194304,4194304] 0 2024-04-22T23:23:21.350 INFO:tasks.workunit.client.0.smithi067.stdout:8/217: chown d1/dd/d26/l2c 1957 1 2024-04-22T23:23:21.353 INFO:tasks.workunit.client.0.smithi067.stdout:8/218: write d1/dd/d26/f22 [27729,12720] 0 2024-04-22T23:23:21.354 INFO:tasks.workunit.client.0.smithi067.stdout:6/158: creat d17/f33 x:0 0 0 2024-04-22T23:23:21.354 INFO:tasks.workunit.client.0.smithi067.stdout:4/229: dwrite d12/f32 [0,4194304] 0 2024-04-22T23:23:21.356 INFO:tasks.workunit.client.0.smithi067.stdout:6/159: read - d17/f31 zero size 2024-04-22T23:23:21.364 INFO:tasks.workunit.client.0.smithi067.stdout:4/230: creat d12/f52 x:0 0 0 2024-04-22T23:23:21.366 INFO:tasks.workunit.client.0.smithi067.stdout:8/219: creat d1/d3/d10/f45 x:0 0 0 2024-04-22T23:23:21.367 INFO:tasks.workunit.client.0.smithi067.stdout:6/160: dwrite f10 [0,4194304] 0 2024-04-22T23:23:21.369 INFO:tasks.workunit.client.0.smithi067.stdout:6/161: write d17/f31 [1007719,8346] 0 2024-04-22T23:23:21.375 INFO:tasks.workunit.client.0.smithi067.stdout:4/231: dwrite d12/f16 [0,4194304] 0 2024-04-22T23:23:21.375 INFO:tasks.workunit.client.0.smithi067.stdout:4/232: stat d12/d2e/c45 0 2024-04-22T23:23:21.376 INFO:tasks.workunit.client.0.smithi067.stdout:4/233: read fa [2410848,87350] 0 2024-04-22T23:23:21.377 INFO:tasks.workunit.client.0.smithi067.stdout:1/238: unlink d8/d9/d2d/d4d/l2e 0 2024-04-22T23:23:21.377 INFO:tasks.workunit.client.0.smithi067.stdout:4/234: truncate d12/f46 665382 0 2024-04-22T23:23:21.379 INFO:tasks.workunit.client.0.smithi067.stdout:8/220: stat d1/d3/d27/f33 0 2024-04-22T23:23:21.380 INFO:tasks.workunit.client.0.smithi067.stdout:1/239: write d8/d9/f2c [318887,7050] 0 2024-04-22T23:23:21.380 INFO:tasks.workunit.client.0.smithi067.stdout:8/221: chown d1/dd/d26/f22 424624 1 2024-04-22T23:23:21.383 INFO:tasks.workunit.client.0.smithi067.stdout:8/222: write d1/d3/d14/f34 [963482,116942] 0 2024-04-22T23:23:21.385 INFO:tasks.workunit.client.0.smithi067.stdout:7/141: getdents d10/d14/d1d 0 2024-04-22T23:23:21.385 INFO:tasks.workunit.client.0.smithi067.stdout:7/142: fdatasync d10/f12 0 2024-04-22T23:23:21.391 INFO:tasks.workunit.client.0.smithi067.stdout:5/168: getdents d5/db 0 2024-04-22T23:23:21.394 INFO:tasks.workunit.client.0.smithi067.stdout:1/240: rename d8/d9/d2d/c53 to d8/d12/d36/c56 0 2024-04-22T23:23:21.395 INFO:tasks.workunit.client.0.smithi067.stdout:9/156: dwrite d5/dd/f26 [0,4194304] 0 2024-04-22T23:23:21.395 INFO:tasks.workunit.client.0.smithi067.stdout:8/223: mkdir d1/d41/d46 0 2024-04-22T23:23:21.396 INFO:tasks.workunit.client.0.smithi067.stdout:4/235: mknod d12/d42/d4b/c53 0 2024-04-22T23:23:21.396 INFO:tasks.workunit.client.0.smithi067.stdout:8/224: fdatasync d1/d3/d14/f30 0 2024-04-22T23:23:21.399 INFO:tasks.workunit.client.0.smithi067.stdout:8/225: chown d1/d3/d10/f45 370090 1 2024-04-22T23:23:21.399 INFO:tasks.workunit.client.0.smithi067.stdout:6/162: link d17/f21 d17/f34 0 2024-04-22T23:23:21.401 INFO:tasks.workunit.client.0.smithi067.stdout:0/155: sync 2024-04-22T23:23:21.401 INFO:tasks.workunit.client.0.smithi067.stdout:5/169: rmdir d5/d28 39 2024-04-22T23:23:21.403 INFO:tasks.workunit.client.0.smithi067.stdout:1/241: fsync d8/f26 0 2024-04-22T23:23:21.404 INFO:tasks.workunit.client.0.smithi067.stdout:1/242: dread - d8/d9/d2f/d33/f41 zero size 2024-04-22T23:23:21.405 INFO:tasks.workunit.client.0.smithi067.stdout:1/243: chown d8/fc 291342380 1 2024-04-22T23:23:21.407 INFO:tasks.workunit.client.0.smithi067.stdout:5/170: dwrite d5/f15 [0,4194304] 0 2024-04-22T23:23:21.407 INFO:tasks.workunit.client.0.smithi067.stdout:8/226: mknod d1/d3/d14/c47 0 2024-04-22T23:23:21.408 INFO:tasks.workunit.client.0.smithi067.stdout:9/157: fsync d5/dd/d18/f1d 0 2024-04-22T23:23:21.409 INFO:tasks.workunit.client.0.smithi067.stdout:6/163: rmdir d17 39 2024-04-22T23:23:21.411 INFO:tasks.workunit.client.0.smithi067.stdout:0/156: creat db/de/d11/d1a/f38 x:0 0 0 2024-04-22T23:23:21.417 INFO:tasks.workunit.client.0.smithi067.stdout:1/244: creat d8/d12/d40/f57 x:0 0 0 2024-04-22T23:23:21.418 INFO:tasks.workunit.client.0.smithi067.stdout:0/157: dwrite f0 [0,4194304] 0 2024-04-22T23:23:21.419 INFO:tasks.workunit.client.0.smithi067.stdout:8/227: dread - d1/d3/f38 zero size 2024-04-22T23:23:21.481 INFO:tasks.workunit.client.0.smithi067.stdout:4/236: link fe d12/f54 0 2024-04-22T23:23:21.481 INFO:tasks.workunit.client.0.smithi067.stdout:7/143: getdents d10/d18 0 2024-04-22T23:23:21.481 INFO:tasks.workunit.client.0.smithi067.stdout:4/237: dread - d12/f2a zero size 2024-04-22T23:23:21.482 INFO:tasks.workunit.client.0.smithi067.stdout:6/164: unlink d17/f2f 0 2024-04-22T23:23:21.482 INFO:tasks.workunit.client.0.smithi067.stdout:4/238: fsync d12/f2a 0 2024-04-22T23:23:21.483 INFO:tasks.workunit.client.0.smithi067.stdout:6/165: dread - d17/f32 zero size 2024-04-22T23:23:21.483 INFO:tasks.workunit.client.0.smithi067.stdout:1/245: creat d8/d9/d2f/d31/f58 x:0 0 0 2024-04-22T23:23:21.485 INFO:tasks.workunit.client.0.smithi067.stdout:7/144: mknod d10/d14/c2b 0 2024-04-22T23:23:21.488 INFO:tasks.workunit.client.0.smithi067.stdout:5/171: dread d5/f11 [0,4194304] 0 2024-04-22T23:23:21.488 INFO:tasks.workunit.client.0.smithi067.stdout:1/246: dread d8/d12/f54 [0,4194304] 0 2024-04-22T23:23:21.489 INFO:tasks.workunit.client.0.smithi067.stdout:6/166: creat d17/f35 x:0 0 0 2024-04-22T23:23:21.490 INFO:tasks.workunit.client.0.smithi067.stdout:1/247: readlink d8/d9/l22 0 2024-04-22T23:23:21.490 INFO:tasks.workunit.client.0.smithi067.stdout:5/172: dread - d5/db/d12/f37 zero size 2024-04-22T23:23:21.492 INFO:tasks.workunit.client.0.smithi067.stdout:0/158: getdents db/de/d11/d1a 0 2024-04-22T23:23:21.493 INFO:tasks.workunit.client.0.smithi067.stdout:0/159: stat db/de/d11/f27 0 2024-04-22T23:23:21.495 INFO:tasks.workunit.client.0.smithi067.stdout:7/145: mkdir d10/d14/d1d/d26/d2c 0 2024-04-22T23:23:21.496 INFO:tasks.workunit.client.0.smithi067.stdout:5/173: dread d5/db/f2f [0,4194304] 0 2024-04-22T23:23:21.496 INFO:tasks.workunit.client.0.smithi067.stdout:8/228: getdents d1 0 2024-04-22T23:23:21.497 INFO:tasks.workunit.client.0.smithi067.stdout:4/239: dread ff [0,4194304] 0 2024-04-22T23:23:21.498 INFO:tasks.workunit.client.0.smithi067.stdout:6/167: mkdir d17/d36 0 2024-04-22T23:23:21.498 INFO:tasks.workunit.client.0.smithi067.stdout:5/174: chown d5/l9 16115377 1 2024-04-22T23:23:21.498 INFO:tasks.workunit.client.0.smithi067.stdout:6/168: readlink d17/l2a 0 2024-04-22T23:23:21.498 INFO:tasks.workunit.client.0.smithi067.stdout:5/175: stat d5/db/ff 0 2024-04-22T23:23:21.499 INFO:tasks.workunit.client.0.smithi067.stdout:6/169: chown fc 5 1 2024-04-22T23:23:21.500 INFO:tasks.workunit.client.0.smithi067.stdout:5/176: read d5/f1b [2162601,87156] 0 2024-04-22T23:23:21.500 INFO:tasks.workunit.client.0.smithi067.stdout:1/248: mkdir d8/d9/d2f/d59 0 2024-04-22T23:23:21.502 INFO:tasks.workunit.client.0.smithi067.stdout:0/160: dread - db/d21/d29/d2d/f35 zero size 2024-04-22T23:23:21.503 INFO:tasks.workunit.client.0.smithi067.stdout:7/146: chown fc 3059 1 2024-04-22T23:23:21.506 INFO:tasks.workunit.client.0.smithi067.stdout:0/161: write db/de/d11/d1a/f38 [950112,115145] 0 2024-04-22T23:23:21.507 INFO:tasks.workunit.client.0.smithi067.stdout:5/177: unlink d5/d32/f26 0 2024-04-22T23:23:21.507 INFO:tasks.workunit.client.0.smithi067.stdout:8/229: dwrite d1/d3/f38 [0,4194304] 0 2024-04-22T23:23:21.509 INFO:tasks.workunit.client.0.smithi067.stdout:0/162: chown db/de/d11/f26 6 1 2024-04-22T23:23:21.514 INFO:tasks.workunit.client.0.smithi067.stdout:5/178: symlink d5/d32/l3b 0 2024-04-22T23:23:21.515 INFO:tasks.workunit.client.0.smithi067.stdout:8/230: mknod d1/dd/d26/c48 0 2024-04-22T23:23:21.516 INFO:tasks.workunit.client.0.smithi067.stdout:8/231: chown d1/d3/d10/f45 453 1 2024-04-22T23:23:21.517 INFO:tasks.workunit.client.0.smithi067.stdout:1/249: link d8/d9/f43 d8/d4a/f5a 0 2024-04-22T23:23:21.518 INFO:tasks.workunit.client.0.smithi067.stdout:0/163: creat db/de/d2a/f39 x:0 0 0 2024-04-22T23:23:21.519 INFO:tasks.workunit.client.0.smithi067.stdout:1/250: readlink d8/d9/d2d/d4d/l14 0 2024-04-22T23:23:21.519 INFO:tasks.workunit.client.0.smithi067.stdout:6/170: rmdir d17/d36 0 2024-04-22T23:23:21.520 INFO:tasks.workunit.client.0.smithi067.stdout:4/240: getdents d12/d42/d4b 0 2024-04-22T23:23:21.521 INFO:tasks.workunit.client.0.smithi067.stdout:1/251: dread - d8/d9/f29 zero size 2024-04-22T23:23:21.521 INFO:tasks.workunit.client.0.smithi067.stdout:0/164: write db/de/f22 [3831490,54561] 0 2024-04-22T23:23:21.522 INFO:tasks.workunit.client.0.smithi067.stdout:7/147: link d10/f12 d10/d28/f2d 0 2024-04-22T23:23:21.522 INFO:tasks.workunit.client.0.smithi067.stdout:6/171: write f12 [432489,50732] 0 2024-04-22T23:23:21.523 INFO:tasks.workunit.client.0.smithi067.stdout:5/179: dwrite d5/db/d12/f37 [0,4194304] 0 2024-04-22T23:23:21.524 INFO:tasks.workunit.client.0.smithi067.stdout:8/232: mknod d1/d3/d10/c49 0 2024-04-22T23:23:21.525 INFO:tasks.workunit.client.0.smithi067.stdout:4/241: read - d12/d17/d31/f39 zero size 2024-04-22T23:23:21.528 INFO:tasks.workunit.client.0.smithi067.stdout:1/252: write d8/d9/d2f/d31/f58 [970251,1265] 0 2024-04-22T23:23:21.528 INFO:tasks.workunit.client.0.smithi067.stdout:7/148: read f5 [256171,29749] 0 2024-04-22T23:23:21.528 INFO:tasks.workunit.client.0.smithi067.stdout:6/172: symlink d17/l37 0 2024-04-22T23:23:21.529 INFO:tasks.workunit.client.0.smithi067.stdout:0/165: write db/de/f1b [957260,51413] 0 2024-04-22T23:23:21.531 INFO:tasks.workunit.client.0.smithi067.stdout:4/242: creat d12/d17/d1e/f55 x:0 0 0 2024-04-22T23:23:21.532 INFO:tasks.workunit.client.0.smithi067.stdout:8/233: symlink d1/dd/d26/l4a 0 2024-04-22T23:23:21.532 INFO:tasks.workunit.client.0.smithi067.stdout:0/166: chown db/de/d2a 77 1 2024-04-22T23:23:21.533 INFO:tasks.workunit.client.0.smithi067.stdout:7/149: write d10/d14/d1d/f1f [1393802,43649] 0 2024-04-22T23:23:21.536 INFO:tasks.workunit.client.0.smithi067.stdout:8/234: write d1/d3/d14/f34 [1469524,71259] 0 2024-04-22T23:23:21.536 INFO:tasks.workunit.client.0.smithi067.stdout:1/253: symlink d8/d9/l5b 0 2024-04-22T23:23:21.537 INFO:tasks.workunit.client.0.smithi067.stdout:5/180: dwrite d5/f7 [0,4194304] 0 2024-04-22T23:23:21.539 INFO:tasks.workunit.client.0.smithi067.stdout:8/235: chown d1/d3/d10/d3b 3077 1 2024-04-22T23:23:21.539 INFO:tasks.workunit.client.0.smithi067.stdout:6/173: link d17/l2a d17/l38 0 2024-04-22T23:23:21.541 INFO:tasks.workunit.client.0.smithi067.stdout:1/254: read d8/d9/d2f/d31/f58 [870302,1029] 0 2024-04-22T23:23:21.543 INFO:tasks.workunit.client.0.smithi067.stdout:6/174: mkdir d17/d39 0 2024-04-22T23:23:21.544 INFO:tasks.workunit.client.0.smithi067.stdout:7/150: creat d10/d14/d1d/d26/d2c/f2e x:0 0 0 2024-04-22T23:23:21.544 INFO:tasks.workunit.client.0.smithi067.stdout:4/243: dwrite ff [0,4194304] 0 2024-04-22T23:23:21.546 INFO:tasks.workunit.client.0.smithi067.stdout:7/151: chown d10/d18/f1a 251803587 1 2024-04-22T23:23:21.546 INFO:tasks.workunit.client.0.smithi067.stdout:8/236: symlink d1/d41/d46/l4b 0 2024-04-22T23:23:21.547 INFO:tasks.workunit.client.0.smithi067.stdout:7/152: chown d10/d14/c1c 17042682 1 2024-04-22T23:23:21.549 INFO:tasks.workunit.client.0.smithi067.stdout:6/175: mkdir d17/d39/d3a 0 2024-04-22T23:23:21.549 INFO:tasks.workunit.client.0.smithi067.stdout:2/180: sync 2024-04-22T23:23:21.550 INFO:tasks.workunit.client.0.smithi067.stdout:8/237: chown d1/d3/d27/f33 49 1 2024-04-22T23:23:21.550 INFO:tasks.workunit.client.0.smithi067.stdout:4/244: write f4 [3362868,103341] 0 2024-04-22T23:23:21.550 INFO:tasks.workunit.client.0.smithi067.stdout:7/153: creat d10/d14/d1d/d26/f2f x:0 0 0 2024-04-22T23:23:21.551 INFO:tasks.workunit.client.0.smithi067.stdout:1/255: getdents d8/d12 0 2024-04-22T23:23:21.553 INFO:tasks.workunit.client.0.smithi067.stdout:1/256: dread d8/d12/f54 [0,4194304] 0 2024-04-22T23:23:21.554 INFO:tasks.workunit.client.0.smithi067.stdout:8/238: chown d1/d3/l1d 8722182 1 2024-04-22T23:23:21.554 INFO:tasks.workunit.client.0.smithi067.stdout:6/176: creat d17/d39/d3a/f3b x:0 0 0 2024-04-22T23:23:21.557 INFO:tasks.workunit.client.0.smithi067.stdout:4/245: dread d12/d17/f33 [0,4194304] 0 2024-04-22T23:23:21.562 INFO:tasks.workunit.client.0.smithi067.stdout:6/177: mknod d17/d39/d3a/c3c 0 2024-04-22T23:23:21.564 INFO:tasks.workunit.client.0.smithi067.stdout:4/246: dwrite d12/d17/d1e/f28 [0,4194304] 0 2024-04-22T23:23:21.565 INFO:tasks.workunit.client.0.smithi067.stdout:8/239: truncate d1/d3/f11 807328 0 2024-04-22T23:23:21.566 INFO:tasks.workunit.client.0.smithi067.stdout:4/247: write d12/f2a [257068,85715] 0 2024-04-22T23:23:21.568 INFO:tasks.workunit.client.0.smithi067.stdout:6/178: creat d17/f3d x:0 0 0 2024-04-22T23:23:21.569 INFO:tasks.workunit.client.0.smithi067.stdout:8/240: chown d1/d3/c39 1045 1 2024-04-22T23:23:21.571 INFO:tasks.workunit.client.0.smithi067.stdout:6/179: mkdir d17/d39/d3e 0 2024-04-22T23:23:21.573 INFO:tasks.workunit.client.0.smithi067.stdout:8/241: truncate d1/d3/d10/f45 90541 0 2024-04-22T23:23:21.573 INFO:tasks.workunit.client.0.smithi067.stdout:4/248: dwrite d12/d17/d31/f39 [0,4194304] 0 2024-04-22T23:23:21.580 INFO:tasks.workunit.client.0.smithi067.stdout:8/242: mknod d1/c4c 0 2024-04-22T23:23:21.586 INFO:tasks.workunit.client.0.smithi067.stdout:6/180: dread fe [0,4194304] 0 2024-04-22T23:23:21.587 INFO:tasks.workunit.client.0.smithi067.stdout:6/181: readlink d17/l23 0 2024-04-22T23:23:21.588 INFO:tasks.workunit.client.0.smithi067.stdout:4/249: dwrite d12/d17/f51 [0,4194304] 0 2024-04-22T23:23:21.588 INFO:tasks.workunit.client.0.smithi067.stdout:7/154: write d10/d28/f29 [1654821,58880] 0 2024-04-22T23:23:21.595 INFO:tasks.workunit.client.0.smithi067.stdout:2/181: truncate d1/d10/f24 83379 0 2024-04-22T23:23:21.595 INFO:tasks.workunit.client.0.smithi067.stdout:4/250: rmdir d12/d17/d47 39 2024-04-22T23:23:21.597 INFO:tasks.workunit.client.0.smithi067.stdout:5/181: dwrite d5/db/ff [0,4194304] 0 2024-04-22T23:23:21.597 INFO:tasks.workunit.client.0.smithi067.stdout:2/182: chown d1/f3 501822 1 2024-04-22T23:23:21.602 INFO:tasks.workunit.client.0.smithi067.stdout:1/257: dwrite f4 [0,4194304] 0 2024-04-22T23:23:21.602 INFO:tasks.workunit.client.0.smithi067.stdout:7/155: creat d10/d28/f30 x:0 0 0 2024-04-22T23:23:21.604 INFO:tasks.workunit.client.0.smithi067.stdout:5/182: write d5/db/d12/f25 [2230103,75821] 0 2024-04-22T23:23:21.605 INFO:tasks.workunit.client.0.smithi067.stdout:7/156: write fe [3140651,31089] 0 2024-04-22T23:23:21.605 INFO:tasks.workunit.client.0.smithi067.stdout:5/183: stat d5/d1a 0 2024-04-22T23:23:21.606 INFO:tasks.workunit.client.0.smithi067.stdout:4/251: mkdir d12/d17/d1e/d48/d56 0 2024-04-22T23:23:21.610 INFO:tasks.workunit.client.0.smithi067.stdout:7/157: symlink d10/l31 0 2024-04-22T23:23:21.611 INFO:tasks.workunit.client.0.smithi067.stdout:8/243: rename d1 to d1/d4d 22 2024-04-22T23:23:21.612 INFO:tasks.workunit.client.0.smithi067.stdout:7/158: write d10/d18/f1e [438520,91365] 0 2024-04-22T23:23:21.613 INFO:tasks.workunit.client.0.smithi067.stdout:2/183: creat d1/f2d x:0 0 0 2024-04-22T23:23:21.614 INFO:tasks.workunit.client.0.smithi067.stdout:5/184: creat d5/d3a/f3c x:0 0 0 2024-04-22T23:23:21.617 INFO:tasks.workunit.client.0.smithi067.stdout:7/159: symlink d10/d14/l32 0 2024-04-22T23:23:21.618 INFO:tasks.workunit.client.0.smithi067.stdout:4/252: dread d12/f54 [0,4194304] 0 2024-04-22T23:23:21.619 INFO:tasks.workunit.client.0.smithi067.stdout:5/185: mknod d5/d32/c3d 0 2024-04-22T23:23:21.619 INFO:tasks.workunit.client.0.smithi067.stdout:5/186: chown d5 49490 1 2024-04-22T23:23:21.621 INFO:tasks.workunit.client.0.smithi067.stdout:7/160: rmdir d10/d14/d1d/d26 39 2024-04-22T23:23:21.622 INFO:tasks.workunit.client.0.smithi067.stdout:8/244: mknod d1/d3/d10/d3b/c4e 0 2024-04-22T23:23:21.627 INFO:tasks.workunit.client.0.smithi067.stdout:5/187: dwrite d5/db/d12/f2b [0,4194304] 0 2024-04-22T23:23:21.627 INFO:tasks.workunit.client.0.smithi067.stdout:6/182: dread d17/f1e [0,4194304] 0 2024-04-22T23:23:21.631 INFO:tasks.workunit.client.0.smithi067.stdout:4/253: rmdir d12/d17/d1e/d2b/d4d 0 2024-04-22T23:23:21.635 INFO:tasks.workunit.client.0.smithi067.stdout:7/161: rename l7 to d10/d14/d1d/d26/d2c/l33 0 2024-04-22T23:23:21.636 INFO:tasks.workunit.client.0.smithi067.stdout:8/245: getdents d1 0 2024-04-22T23:23:21.637 INFO:tasks.workunit.client.0.smithi067.stdout:8/246: read - d1/d3/d10/f3d zero size 2024-04-22T23:23:21.638 INFO:tasks.workunit.client.0.smithi067.stdout:8/247: chown d1/dd/f20 74441 1 2024-04-22T23:23:21.638 INFO:tasks.workunit.client.0.smithi067.stdout:2/184: fsync d1/f2d 0 2024-04-22T23:23:21.639 INFO:tasks.workunit.client.0.smithi067.stdout:2/185: stat d1/d19/f21 0 2024-04-22T23:23:21.640 INFO:tasks.workunit.client.0.smithi067.stdout:7/162: creat d10/d14/f34 x:0 0 0 2024-04-22T23:23:21.642 INFO:tasks.workunit.client.0.smithi067.stdout:8/248: symlink d1/d41/l4f 0 2024-04-22T23:23:21.646 INFO:tasks.workunit.client.0.smithi067.stdout:7/163: mkdir d10/d18/d35 0 2024-04-22T23:23:21.649 INFO:tasks.workunit.client.0.smithi067.stdout:7/164: symlink d10/d18/l36 0 2024-04-22T23:23:21.650 INFO:tasks.workunit.client.0.smithi067.stdout:8/249: dwrite d1/dd/f20 [0,4194304] 0 2024-04-22T23:23:21.650 INFO:tasks.workunit.client.0.smithi067.stdout:7/165: readlink d10/d18/l36 0 2024-04-22T23:23:21.651 INFO:tasks.workunit.client.0.smithi067.stdout:4/254: dread f4 [0,4194304] 0 2024-04-22T23:23:21.652 INFO:tasks.workunit.client.0.smithi067.stdout:2/186: creat d1/d10/f2e x:0 0 0 2024-04-22T23:23:21.654 INFO:tasks.workunit.client.0.smithi067.stdout:2/187: chown d1/d12/d28 966557 1 2024-04-22T23:23:21.656 INFO:tasks.workunit.client.0.smithi067.stdout:7/166: symlink d10/l37 0 2024-04-22T23:23:21.657 INFO:tasks.workunit.client.0.smithi067.stdout:5/188: dread d5/db/d12/f25 [0,4194304] 0 2024-04-22T23:23:21.660 INFO:tasks.workunit.client.0.smithi067.stdout:2/188: creat d1/d19/f2f x:0 0 0 2024-04-22T23:23:21.660 INFO:tasks.workunit.client.0.smithi067.stdout:5/189: fsync d5/db/d35/f39 0 2024-04-22T23:23:21.662 INFO:tasks.workunit.client.0.smithi067.stdout:7/167: rename d10/d14/c21 to d10/d18/c38 0 2024-04-22T23:23:21.663 INFO:tasks.workunit.client.0.smithi067.stdout:8/250: dwrite d1/d3/d27/f28 [0,4194304] 0 2024-04-22T23:23:21.668 INFO:tasks.workunit.client.0.smithi067.stdout:8/251: dread d1/d3/d27/f28 [0,4194304] 0 2024-04-22T23:23:21.671 INFO:tasks.workunit.client.0.smithi067.stdout:2/189: getdents d1/d12 0 2024-04-22T23:23:21.673 INFO:tasks.workunit.client.0.smithi067.stdout:8/252: link d1/d3/d10/f3d d1/d3/d14/f50 0 2024-04-22T23:23:21.675 INFO:tasks.workunit.client.0.smithi067.stdout:2/190: link d1/d19/f2f d1/d10/f30 0 2024-04-22T23:23:21.676 INFO:tasks.workunit.client.0.smithi067.stdout:8/253: symlink d1/d3/l51 0 2024-04-22T23:23:21.677 INFO:tasks.workunit.client.0.smithi067.stdout:8/254: truncate d1/d3/d14/f30 4297386 0 2024-04-22T23:23:21.678 INFO:tasks.workunit.client.0.smithi067.stdout:2/191: truncate d1/d12/f15 3391039 0 2024-04-22T23:23:21.680 INFO:tasks.workunit.client.0.smithi067.stdout:8/255: unlink d1/d3/d10/f3e 0 2024-04-22T23:23:21.680 INFO:tasks.workunit.client.0.smithi067.stdout:8/256: stat d1/d41 0 2024-04-22T23:23:21.683 INFO:tasks.workunit.client.0.smithi067.stdout:8/257: write d1/d3/d10/f3d [180616,111523] 0 2024-04-22T23:23:21.684 INFO:tasks.workunit.client.0.smithi067.stdout:8/258: chown d1/d3/d10/c49 417358 1 2024-04-22T23:23:21.684 INFO:tasks.workunit.client.0.smithi067.stdout:2/192: rmdir d1/d10/d1c 39 2024-04-22T23:23:21.685 INFO:tasks.workunit.client.0.smithi067.stdout:4/255: write d12/f54 [1425862,25881] 0 2024-04-22T23:23:21.686 INFO:tasks.workunit.client.0.smithi067.stdout:2/193: write d1/f4 [7828585,127119] 0 2024-04-22T23:23:21.688 INFO:tasks.workunit.client.0.smithi067.stdout:7/168: getdents d10/d18 0 2024-04-22T23:23:21.690 INFO:tasks.workunit.client.0.smithi067.stdout:4/256: creat d12/d2e/f57 x:0 0 0 2024-04-22T23:23:21.691 INFO:tasks.workunit.client.0.smithi067.stdout:5/190: write d5/f1b [898556,103698] 0 2024-04-22T23:23:21.692 INFO:tasks.workunit.client.0.smithi067.stdout:2/194: mkdir d1/d19/d31 0 2024-04-22T23:23:21.692 INFO:tasks.workunit.client.0.smithi067.stdout:4/257: readlink d12/l18 0 2024-04-22T23:23:21.695 INFO:tasks.workunit.client.0.smithi067.stdout:7/169: dread f5 [0,4194304] 0 2024-04-22T23:23:21.696 INFO:tasks.workunit.client.0.smithi067.stdout:8/259: dwrite d1/d3/d14/f32 [4194304,4194304] 0 2024-04-22T23:23:21.696 INFO:tasks.workunit.client.0.smithi067.stdout:2/195: read d1/f4 [600321,66763] 0 2024-04-22T23:23:21.697 INFO:tasks.workunit.client.0.smithi067.stdout:4/258: write d12/d17/d1e/f27 [186619,602] 0 2024-04-22T23:23:21.697 INFO:tasks.workunit.client.0.smithi067.stdout:8/260: readlink d1/l21 0 2024-04-22T23:23:21.698 INFO:tasks.workunit.client.0.smithi067.stdout:2/196: truncate d1/f2d 565223 0 2024-04-22T23:23:21.705 INFO:tasks.workunit.client.0.smithi067.stdout:7/170: creat d10/d14/d1d/d26/f39 x:0 0 0 2024-04-22T23:23:21.707 INFO:tasks.workunit.client.0.smithi067.stdout:7/171: truncate fc 1865663 0 2024-04-22T23:23:21.709 INFO:tasks.workunit.client.0.smithi067.stdout:8/261: dwrite d1/d3/d14/f50 [0,4194304] 0 2024-04-22T23:23:21.710 INFO:tasks.workunit.client.0.smithi067.stdout:2/197: rename d1/c14 to d1/d19/d31/c32 0 2024-04-22T23:23:21.711 INFO:tasks.workunit.client.0.smithi067.stdout:5/191: rmdir d5/db/d12/d2a 0 2024-04-22T23:23:21.712 INFO:tasks.workunit.client.0.smithi067.stdout:4/259: dread fa [0,4194304] 0 2024-04-22T23:23:21.712 INFO:tasks.workunit.client.0.smithi067.stdout:2/198: write d1/d10/f2e [682885,109252] 0 2024-04-22T23:23:21.713 INFO:tasks.workunit.client.0.smithi067.stdout:7/172: creat d10/d14/d1d/d26/f3a x:0 0 0 2024-04-22T23:23:21.714 INFO:tasks.workunit.client.0.smithi067.stdout:8/262: symlink d1/d41/d46/l52 0 2024-04-22T23:23:21.715 INFO:tasks.workunit.client.0.smithi067.stdout:8/263: write d1/d3/f2a [488415,96996] 0 2024-04-22T23:23:21.716 INFO:tasks.workunit.client.0.smithi067.stdout:4/260: creat d12/f58 x:0 0 0 2024-04-22T23:23:21.718 INFO:tasks.workunit.client.0.smithi067.stdout:7/173: mknod d10/d18/d35/c3b 0 2024-04-22T23:23:21.720 INFO:tasks.workunit.client.0.smithi067.stdout:2/199: mkdir d1/d10/d33 0 2024-04-22T23:23:21.721 INFO:tasks.workunit.client.0.smithi067.stdout:8/264: rmdir d1/dd/d26 39 2024-04-22T23:23:21.722 INFO:tasks.workunit.client.0.smithi067.stdout:7/174: mknod d10/d28/c3c 0 2024-04-22T23:23:21.726 INFO:tasks.workunit.client.0.smithi067.stdout:4/261: dwrite d12/d17/d1e/f55 [0,4194304] 0 2024-04-22T23:23:21.727 INFO:tasks.workunit.client.0.smithi067.stdout:2/200: symlink d1/d10/d1c/l34 0 2024-04-22T23:23:21.729 INFO:tasks.workunit.client.0.smithi067.stdout:8/265: creat d1/dd/f53 x:0 0 0 2024-04-22T23:23:21.730 INFO:tasks.workunit.client.0.smithi067.stdout:7/175: dread d10/d28/f29 [0,4194304] 0 2024-04-22T23:23:21.732 INFO:tasks.workunit.client.0.smithi067.stdout:4/262: rename d12/d17/d31/f39 to d12/d17/d31/f59 0 2024-04-22T23:23:21.736 INFO:tasks.workunit.client.0.smithi067.stdout:5/192: getdents d5/db/d12 0 2024-04-22T23:23:21.739 INFO:tasks.workunit.client.0.smithi067.stdout:8/266: dwrite d1/d3/d14/f34 [0,4194304] 0 2024-04-22T23:23:21.740 INFO:tasks.workunit.client.0.smithi067.stdout:5/193: truncate d5/d1a/f2e 320033 0 2024-04-22T23:23:21.740 INFO:tasks.workunit.client.0.smithi067.stdout:7/176: mknod d10/d18/c3d 0 2024-04-22T23:23:21.741 INFO:tasks.workunit.client.0.smithi067.stdout:5/194: chown d5/db/d35 1 1 2024-04-22T23:23:21.742 INFO:tasks.workunit.client.0.smithi067.stdout:7/177: write fb [3282665,110151] 0 2024-04-22T23:23:21.743 INFO:tasks.workunit.client.0.smithi067.stdout:4/263: dwrite d12/d17/d1e/f55 [0,4194304] 0 2024-04-22T23:23:21.745 INFO:tasks.workunit.client.0.smithi067.stdout:2/201: link d1/d19/f21 d1/f35 0 2024-04-22T23:23:21.747 INFO:tasks.workunit.client.0.smithi067.stdout:4/264: creat d12/d17/d1e/f5a x:0 0 0 2024-04-22T23:23:21.749 INFO:tasks.workunit.client.0.smithi067.stdout:2/202: creat d1/d19/d31/f36 x:0 0 0 2024-04-22T23:23:21.750 INFO:tasks.workunit.client.0.smithi067.stdout:5/195: mkdir d5/d3e 0 2024-04-22T23:23:21.752 INFO:tasks.workunit.client.0.smithi067.stdout:3/240: sync 2024-04-22T23:23:21.754 INFO:tasks.workunit.client.0.smithi067.stdout:5/196: unlink d5/d32/c33 0 2024-04-22T23:23:21.755 INFO:tasks.workunit.client.0.smithi067.stdout:4/265: dwrite d12/d17/d1e/f28 [0,4194304] 0 2024-04-22T23:23:21.757 INFO:tasks.workunit.client.0.smithi067.stdout:2/203: creat d1/d10/d33/f37 x:0 0 0 2024-04-22T23:23:21.758 INFO:tasks.workunit.client.0.smithi067.stdout:4/266: truncate d12/d2e/f3e 29392 0 2024-04-22T23:23:21.758 INFO:tasks.workunit.client.0.smithi067.stdout:5/197: creat d5/d3a/f3f x:0 0 0 2024-04-22T23:23:21.759 INFO:tasks.workunit.client.0.smithi067.stdout:2/204: write d1/f3 [6484469,56833] 0 2024-04-22T23:23:21.762 INFO:tasks.workunit.client.0.smithi067.stdout:4/267: mkdir d12/d17/d1e/d2b/d5b 0 2024-04-22T23:23:21.764 INFO:tasks.workunit.client.0.smithi067.stdout:7/178: dread d10/d18/f25 [0,4194304] 0 2024-04-22T23:23:21.766 INFO:tasks.workunit.client.0.smithi067.stdout:2/205: symlink d1/l38 0 2024-04-22T23:23:21.768 INFO:tasks.workunit.client.0.smithi067.stdout:7/179: mknod d10/c3e 0 2024-04-22T23:23:21.769 INFO:tasks.workunit.client.0.smithi067.stdout:7/180: write fc [304937,116441] 0 2024-04-22T23:23:21.769 INFO:tasks.workunit.client.0.smithi067.stdout:4/268: creat d12/d17/d1e/d48/d56/f5c x:0 0 0 2024-04-22T23:23:21.770 INFO:tasks.workunit.client.0.smithi067.stdout:7/181: chown d10/d18/l36 3200701 1 2024-04-22T23:23:21.773 INFO:tasks.workunit.client.0.smithi067.stdout:4/269: mknod d12/d17/d1e/d48/c5d 0 2024-04-22T23:23:21.775 INFO:tasks.workunit.client.0.smithi067.stdout:2/206: symlink d1/d10/l39 0 2024-04-22T23:23:21.777 INFO:tasks.workunit.client.0.smithi067.stdout:4/270: mkdir d12/d42/d4b/d5e 0 2024-04-22T23:23:21.779 INFO:tasks.workunit.client.0.smithi067.stdout:2/207: creat d1/d10/d1c/f3a x:0 0 0 2024-04-22T23:23:21.780 INFO:tasks.workunit.client.0.smithi067.stdout:5/198: rename d5/d32 to d5/d28/d36/d40 0 2024-04-22T23:23:21.781 INFO:tasks.workunit.client.0.smithi067.stdout:5/199: truncate d5/f7 4702932 0 2024-04-22T23:23:21.783 INFO:tasks.workunit.client.0.smithi067.stdout:5/200: truncate d5/db/d12/f30 619022 0 2024-04-22T23:23:21.783 INFO:tasks.workunit.client.0.smithi067.stdout:5/201: fdatasync d5/d1a/f2e 0 2024-04-22T23:23:21.783 INFO:tasks.workunit.client.0.smithi067.stdout:2/208: creat d1/d12/f3b x:0 0 0 2024-04-22T23:23:21.784 INFO:tasks.workunit.client.0.smithi067.stdout:5/202: write d5/f15 [4040836,60332] 0 2024-04-22T23:23:21.784 INFO:tasks.workunit.client.0.smithi067.stdout:2/209: dread - d1/d12/f3b zero size 2024-04-22T23:23:21.786 INFO:tasks.workunit.client.0.smithi067.stdout:3/241: truncate d3/d4/d10/d1f/f3f 344995 0 2024-04-22T23:23:21.786 INFO:tasks.workunit.client.0.smithi067.stdout:7/182: link d10/d28/f2d d10/d14/d1d/d26/d2c/f3f 0 2024-04-22T23:23:21.789 INFO:tasks.workunit.client.0.smithi067.stdout:2/210: dread d1/f1f [0,4194304] 0 2024-04-22T23:23:21.790 INFO:tasks.workunit.client.0.smithi067.stdout:5/203: rename d5/db/d12/l18 to d5/db/d12/l41 0 2024-04-22T23:23:21.793 INFO:tasks.workunit.client.0.smithi067.stdout:3/242: rmdir d3/d4/d10/d19 39 2024-04-22T23:23:21.795 INFO:tasks.workunit.client.0.smithi067.stdout:4/271: rename d12/d17/d1e to d12/d5f 0 2024-04-22T23:23:21.796 INFO:tasks.workunit.client.0.smithi067.stdout:7/183: rmdir d10/d28 39 2024-04-22T23:23:21.797 INFO:tasks.workunit.client.0.smithi067.stdout:7/184: read - d10/d14/d1d/d26/f39 zero size 2024-04-22T23:23:21.806 INFO:tasks.workunit.client.0.smithi067.stdout:7/185: chown d10/d28/f30 189 1 2024-04-22T23:23:21.810 INFO:tasks.workunit.client.0.smithi067.stdout:2/211: dread d1/f4 [4194304,4194304] 0 2024-04-22T23:23:21.812 INFO:tasks.workunit.client.0.smithi067.stdout:4/272: rmdir d12/d5f/d2b/d5b 0 2024-04-22T23:23:21.813 INFO:tasks.workunit.client.0.smithi067.stdout:4/273: truncate d12/d17/f33 989115 0 2024-04-22T23:23:21.815 INFO:tasks.workunit.client.0.smithi067.stdout:5/204: getdents d5/db 0 2024-04-22T23:23:21.815 INFO:tasks.workunit.client.0.smithi067.stdout:3/243: getdents d3/d4/d10/d1f 0 2024-04-22T23:23:21.817 INFO:tasks.workunit.client.0.smithi067.stdout:7/186: write d10/f12 [1079084,49260] 0 2024-04-22T23:23:21.818 INFO:tasks.workunit.client.0.smithi067.stdout:2/212: write d1/f1f [945200,81050] 0 2024-04-22T23:23:21.819 INFO:tasks.workunit.client.0.smithi067.stdout:5/205: readlink d5/db/l1d 0 2024-04-22T23:23:21.821 INFO:tasks.workunit.client.0.smithi067.stdout:7/187: write d10/d14/d1d/d26/d2c/f2e [873105,77161] 0 2024-04-22T23:23:21.821 INFO:tasks.workunit.client.0.smithi067.stdout:3/244: creat d3/d4/d12/d4b/f4e x:0 0 0 2024-04-22T23:23:21.822 INFO:tasks.workunit.client.0.smithi067.stdout:2/213: mknod d1/d10/d33/c3c 0 2024-04-22T23:23:21.825 INFO:tasks.workunit.client.0.smithi067.stdout:2/214: chown d1/d10/d1c/c1e 2928 1 2024-04-22T23:23:21.826 INFO:tasks.workunit.client.0.smithi067.stdout:7/188: creat d10/d14/d1d/d26/f40 x:0 0 0 2024-04-22T23:23:21.827 INFO:tasks.workunit.client.0.smithi067.stdout:4/274: dread d12/d17/d47/f49 [0,4194304] 0 2024-04-22T23:23:21.830 INFO:tasks.workunit.client.0.smithi067.stdout:2/215: creat d1/d12/d28/f3d x:0 0 0 2024-04-22T23:23:21.832 INFO:tasks.workunit.client.0.smithi067.stdout:7/189: rmdir d10/d14/d1d/d26/d2c 39 2024-04-22T23:23:21.832 INFO:tasks.workunit.client.0.smithi067.stdout:5/206: dwrite d5/db/d35/f39 [0,4194304] 0 2024-04-22T23:23:21.835 INFO:tasks.workunit.client.0.smithi067.stdout:3/245: dwrite d3/d4/d10/d38/f48 [0,4194304] 0 2024-04-22T23:23:21.836 INFO:tasks.workunit.client.0.smithi067.stdout:2/216: creat d1/d10/d33/f3e x:0 0 0 2024-04-22T23:23:21.840 INFO:tasks.workunit.client.0.smithi067.stdout:4/275: dread f4 [0,4194304] 0 2024-04-22T23:23:21.840 INFO:tasks.workunit.client.0.smithi067.stdout:5/207: rename d5/db/d12/f30 to d5/d3e/f42 0 2024-04-22T23:23:21.842 INFO:tasks.workunit.client.0.smithi067.stdout:3/246: fdatasync d3/d4/d10/f39 0 2024-04-22T23:23:21.843 INFO:tasks.workunit.client.0.smithi067.stdout:5/208: dread d5/db/f23 [0,4194304] 0 2024-04-22T23:23:21.847 INFO:tasks.workunit.client.0.smithi067.stdout:5/209: creat d5/db/d35/f43 x:0 0 0 2024-04-22T23:23:21.850 INFO:tasks.workunit.client.0.smithi067.stdout:3/247: mknod d3/c4f 0 2024-04-22T23:23:21.852 INFO:tasks.workunit.client.0.smithi067.stdout:3/248: dread d3/d4/f24 [0,4194304] 0 2024-04-22T23:23:21.852 INFO:tasks.workunit.client.0.smithi067.stdout:4/276: creat d12/d17/f60 x:0 0 0 2024-04-22T23:23:21.853 INFO:tasks.workunit.client.0.smithi067.stdout:4/277: dread d12/f46 [0,4194304] 0 2024-04-22T23:23:21.854 INFO:tasks.workunit.client.0.smithi067.stdout:5/210: mkdir d5/db/d12/d44 0 2024-04-22T23:23:21.855 INFO:tasks.workunit.client.0.smithi067.stdout:5/211: write d5/db/d12/f22 [1159854,66573] 0 2024-04-22T23:23:21.857 INFO:tasks.workunit.client.0.smithi067.stdout:5/212: write d5/f15 [5228939,27686] 0 2024-04-22T23:23:21.858 INFO:tasks.workunit.client.0.smithi067.stdout:5/213: chown l4 68247 1 2024-04-22T23:23:21.860 INFO:tasks.workunit.client.0.smithi067.stdout:4/278: creat d12/d5f/d2b/f61 x:0 0 0 2024-04-22T23:23:21.861 INFO:tasks.workunit.client.0.smithi067.stdout:2/217: write d1/d12/f15 [3715228,77255] 0 2024-04-22T23:23:21.865 INFO:tasks.workunit.client.0.smithi067.stdout:3/249: dwrite d3/d4/d10/d19/f3b [0,4194304] 0 2024-04-22T23:23:21.866 INFO:tasks.workunit.client.0.smithi067.stdout:3/250: readlink d3/d4/d10/d1f/d3c/l31 0 2024-04-22T23:23:21.867 INFO:tasks.workunit.client.0.smithi067.stdout:2/218: mknod d1/d19/d31/c3f 0 2024-04-22T23:23:21.869 INFO:tasks.workunit.client.0.smithi067.stdout:4/279: mknod d12/d42/d4b/d5e/c62 0 2024-04-22T23:23:21.871 INFO:tasks.workunit.client.0.smithi067.stdout:4/280: chown d12/d42/d4b/d5e 15379272 1 2024-04-22T23:23:21.872 INFO:tasks.workunit.client.0.smithi067.stdout:4/281: readlink d12/l18 0 2024-04-22T23:23:21.874 INFO:tasks.workunit.client.0.smithi067.stdout:5/214: write d5/db/f2f [1764939,117425] 0 2024-04-22T23:23:21.875 INFO:tasks.workunit.client.0.smithi067.stdout:3/251: getdents d3/d4/d10/d1f 0 2024-04-22T23:23:21.877 INFO:tasks.workunit.client.0.smithi067.stdout:3/252: chown d3/d4/d10/c2d 424264442 1 2024-04-22T23:23:21.879 INFO:tasks.workunit.client.0.smithi067.stdout:4/282: dwrite ff [0,4194304] 0 2024-04-22T23:23:21.879 INFO:tasks.workunit.client.0.smithi067.stdout:5/215: unlink d5/f7 0 2024-04-22T23:23:21.881 INFO:tasks.workunit.client.0.smithi067.stdout:2/219: write d1/d10/f30 [496568,6039] 0 2024-04-22T23:23:21.881 INFO:tasks.workunit.client.0.smithi067.stdout:5/216: write d5/f15 [4842893,71125] 0 2024-04-22T23:23:21.882 INFO:tasks.workunit.client.0.smithi067.stdout:3/253: truncate d3/d4/d10/d1f/f3e 468495 0 2024-04-22T23:23:21.884 INFO:tasks.workunit.client.0.smithi067.stdout:4/283: write d12/d5f/f41 [240541,92219] 0 2024-04-22T23:23:21.887 INFO:tasks.workunit.client.0.smithi067.stdout:5/217: mknod d5/d3e/c45 0 2024-04-22T23:23:21.890 INFO:tasks.workunit.client.0.smithi067.stdout:4/284: dread d12/f46 [0,4194304] 0 2024-04-22T23:23:21.890 INFO:tasks.workunit.client.0.smithi067.stdout:3/254: dread d3/d4/f24 [0,4194304] 0 2024-04-22T23:23:21.890 INFO:tasks.workunit.client.0.smithi067.stdout:5/218: fdatasync d5/f11 0 2024-04-22T23:23:21.891 INFO:tasks.workunit.client.0.smithi067.stdout:2/220: dwrite d1/f8 [0,4194304] 0 2024-04-22T23:23:21.893 INFO:tasks.workunit.client.0.smithi067.stdout:3/255: write d3/d4/d12/d4b/f4e [700797,121802] 0 2024-04-22T23:23:21.894 INFO:tasks.workunit.client.0.smithi067.stdout:4/285: readlink d12/d17/d31/l4a 0 2024-04-22T23:23:21.895 INFO:tasks.workunit.client.0.smithi067.stdout:2/221: creat d1/d10/d1c/f40 x:0 0 0 2024-04-22T23:23:21.897 INFO:tasks.workunit.client.0.smithi067.stdout:4/286: creat d12/d17/d3f/f63 x:0 0 0 2024-04-22T23:23:21.899 INFO:tasks.workunit.client.0.smithi067.stdout:3/256: mknod d3/c50 0 2024-04-22T23:23:21.900 INFO:tasks.workunit.client.0.smithi067.stdout:3/257: chown d3/d4/d1b/c43 10698 1 2024-04-22T23:23:21.901 INFO:tasks.workunit.client.0.smithi067.stdout:2/222: mkdir d1/d10/d41 0 2024-04-22T23:23:21.904 INFO:tasks.workunit.client.0.smithi067.stdout:3/258: fdatasync d3/d4/f24 0 2024-04-22T23:23:21.906 INFO:tasks.workunit.client.0.smithi067.stdout:4/287: dwrite d12/d17/f2f [0,4194304] 0 2024-04-22T23:23:21.907 INFO:tasks.workunit.client.0.smithi067.stdout:2/223: mkdir d1/d42 0 2024-04-22T23:23:21.909 INFO:tasks.workunit.client.0.smithi067.stdout:4/288: creat d12/d2e/f64 x:0 0 0 2024-04-22T23:23:21.909 INFO:tasks.workunit.client.0.smithi067.stdout:2/224: readlink d1/d12/l13 0 2024-04-22T23:23:21.911 INFO:tasks.workunit.client.0.smithi067.stdout:4/289: write d12/d17/f33 [568971,26910] 0 2024-04-22T23:23:21.913 INFO:tasks.workunit.client.0.smithi067.stdout:4/290: read d12/d5f/f28 [95807,248] 0 2024-04-22T23:23:21.913 INFO:tasks.workunit.client.0.smithi067.stdout:2/225: creat d1/d12/f43 x:0 0 0 2024-04-22T23:23:21.916 INFO:tasks.workunit.client.0.smithi067.stdout:4/291: mknod d12/d17/d3f/c65 0 2024-04-22T23:23:21.918 INFO:tasks.workunit.client.0.smithi067.stdout:2/226: rename d1/d12/l1a to d1/d19/d31/l44 0 2024-04-22T23:23:21.921 INFO:tasks.workunit.client.0.smithi067.stdout:4/292: mknod d12/c66 0 2024-04-22T23:23:21.921 INFO:tasks.workunit.client.0.smithi067.stdout:4/293: chown d12/d5f/f41 110715481 1 2024-04-22T23:23:21.922 INFO:tasks.workunit.client.0.smithi067.stdout:2/227: truncate d1/d12/f16 447409 0 2024-04-22T23:23:21.924 INFO:tasks.workunit.client.0.smithi067.stdout:3/259: write d3/d4/d10/d1f/f3f [1252895,63922] 0 2024-04-22T23:23:21.926 INFO:tasks.workunit.client.0.smithi067.stdout:2/228: unlink d1/d12/f3b 0 2024-04-22T23:23:21.928 INFO:tasks.workunit.client.0.smithi067.stdout:3/260: mkdir d3/d4/d28/d4c/d51 0 2024-04-22T23:23:21.929 INFO:tasks.workunit.client.0.smithi067.stdout:2/229: rmdir d1/d10/d33 39 2024-04-22T23:23:21.929 INFO:tasks.workunit.client.0.smithi067.stdout:4/294: dwrite d12/f32 [0,4194304] 0 2024-04-22T23:23:21.935 INFO:tasks.workunit.client.0.smithi067.stdout:4/295: creat d12/d17/d31/f67 x:0 0 0 2024-04-22T23:23:21.936 INFO:tasks.workunit.client.0.smithi067.stdout:4/296: write d12/f54 [3150315,1969] 0 2024-04-22T23:23:21.941 INFO:tasks.workunit.client.0.smithi067.stdout:4/297: creat d12/d5f/d48/f68 x:0 0 0 2024-04-22T23:23:21.945 INFO:tasks.workunit.client.0.smithi067.stdout:4/298: mkdir d12/d5f/d69 0 2024-04-22T23:23:21.945 INFO:tasks.workunit.client.0.smithi067.stdout:2/230: rmdir d1/d12 39 2024-04-22T23:23:21.946 INFO:tasks.workunit.client.0.smithi067.stdout:4/299: truncate d12/d2e/f64 12426 0 2024-04-22T23:23:21.949 INFO:tasks.workunit.client.0.smithi067.stdout:4/300: rename d12/d17/l26 to d12/d42/l6a 0 2024-04-22T23:23:21.949 INFO:tasks.workunit.client.0.smithi067.stdout:3/261: write d3/d4/d10/d1f/d3c/f25 [2024001,82204] 0 2024-04-22T23:23:21.951 INFO:tasks.workunit.client.0.smithi067.stdout:4/301: dread - d12/d5f/d48/d56/f5c zero size 2024-04-22T23:23:21.951 INFO:tasks.workunit.client.0.smithi067.stdout:3/262: stat d3/d4/f37 0 2024-04-22T23:23:21.952 INFO:tasks.workunit.client.0.smithi067.stdout:2/231: getdents d1/d10/d33 0 2024-04-22T23:23:21.954 INFO:tasks.workunit.client.0.smithi067.stdout:3/263: creat d3/d4/d28/d4c/f52 x:0 0 0 2024-04-22T23:23:21.957 INFO:tasks.workunit.client.0.smithi067.stdout:4/302: rename d12/d17/l3c to d12/d42/d4b/d5e/l6b 0 2024-04-22T23:23:21.960 INFO:tasks.workunit.client.0.smithi067.stdout:2/232: getdents d1/d10 0 2024-04-22T23:23:21.961 INFO:tasks.workunit.client.0.smithi067.stdout:4/303: creat d12/d17/d47/f6c x:0 0 0 2024-04-22T23:23:21.962 INFO:tasks.workunit.client.0.smithi067.stdout:3/264: dwrite d3/d4/d10/d38/f48 [0,4194304] 0 2024-04-22T23:23:21.965 INFO:tasks.workunit.client.0.smithi067.stdout:2/233: rename d1/c18 to d1/d10/d33/c45 0 2024-04-22T23:23:21.969 INFO:tasks.workunit.client.0.smithi067.stdout:3/265: dread d3/d4/d12/d4b/f4e [0,4194304] 0 2024-04-22T23:23:21.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:21.971+0000 7f4349ea3700 1 -- 172.21.15.67:0/2837900848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43441072a0 msgr2=0x7f43441076c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:21.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:21.971+0000 7f4349ea3700 1 --2- 172.21.15.67:0/2837900848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43441072a0 0x7f43441076c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4338009910 tx=0x7f4338009c20 comp rx=0 tx=0).stop 2024-04-22T23:23:21.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:21.971+0000 7f4349ea3700 1 -- 172.21.15.67:0/2837900848 shutdown_connections 2024-04-22T23:23:21.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:21.971+0000 7f4349ea3700 1 --2- 172.21.15.67:0/2837900848 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f43441084a0 0x7f4344108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:21.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:21.971+0000 7f4349ea3700 1 --2- 172.21.15.67:0/2837900848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43441072a0 0x7f43441076c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:21.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:21.971+0000 7f4349ea3700 1 -- 172.21.15.67:0/2837900848 >> 172.21.15.67:0/2837900848 conn(0x7f434406a230 msgr2=0x7f43440787e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:23:21.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:21.971+0000 7f4349ea3700 1 -- 172.21.15.67:0/2837900848 shutdown_connections 2024-04-22T23:23:21.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:21.971+0000 7f4349ea3700 1 -- 172.21.15.67:0/2837900848 wait complete. 2024-04-22T23:23:21.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:21.972+0000 7f4349ea3700 1 Processor -- start 2024-04-22T23:23:21.974 INFO:tasks.workunit.client.0.smithi067.stdout:3/266: mkdir d3/d4/d10/d38/d53 0 2024-04-22T23:23:21.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:21.972+0000 7f4349ea3700 1 -- start start 2024-04-22T23:23:21.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:21.972+0000 7f4349ea3700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f43441072a0 0x7f434419cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:23:21.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:21.972+0000 7f4349ea3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43441084a0 0x7f434419d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:23:21.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:21.972+0000 7f4349ea3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434419d6e0 con 0x7f43441084a0 2024-04-22T23:23:21.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:21.972+0000 7f4349ea3700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434419d820 con 0x7f43441072a0 2024-04-22T23:23:21.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:21.972+0000 7f43437fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f43441072a0 0x7f434419cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:23:21.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:21.972+0000 7f4342ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43441084a0 0x7f434419d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:23:21.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:21.972+0000 7f43437fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f43441072a0 0x7f434419cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.67:41154/0 (socket says 172.21.15.67:41154) 2024-04-22T23:23:21.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:21.972+0000 7f43437fe700 1 -- 172.21.15.67:0/2264617709 learned_addr learned my addr 172.21.15.67:0/2264617709 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:23:21.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:21.973+0000 7f43437fe700 1 -- 172.21.15.67:0/2264617709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43441084a0 msgr2=0x7f434419d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:21.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:21.973+0000 7f43437fe700 1 --2- 172.21.15.67:0/2264617709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43441084a0 0x7f434419d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:21.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:21.973+0000 7f43437fe700 1 -- 172.21.15.67:0/2264617709 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43380095f0 con 0x7f43441072a0 2024-04-22T23:23:21.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:21.973+0000 7f4342ffd700 1 --2- 172.21.15.67:0/2264617709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43441084a0 0x7f434419d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T23:23:21.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:21.973+0000 7f43437fe700 1 --2- 172.21.15.67:0/2264617709 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f43441072a0 0x7f434419cb80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4338000c00 tx=0x7f433800bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:23:21.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:21.973+0000 7f4340ff9700 1 -- 172.21.15.67:0/2264617709 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f433801d070 con 0x7f43441072a0 2024-04-22T23:23:21.976 INFO:tasks.workunit.client.0.smithi067.stdout:9/158: sync 2024-04-22T23:23:21.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:21.974+0000 7f4340ff9700 1 -- 172.21.15.67:0/2264617709 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f433800f460 con 0x7f43441072a0 2024-04-22T23:23:21.977 INFO:tasks.workunit.client.0.smithi067.stdout:2/234: dwrite d1/d10/d33/f37 [0,4194304] 0 2024-04-22T23:23:21.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:21.974+0000 7f4340ff9700 1 -- 172.21.15.67:0/2264617709 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4338021620 con 0x7f43441072a0 2024-04-22T23:23:21.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:21.974+0000 7f4349ea3700 1 -- 172.21.15.67:0/2264617709 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f434406a8f0 con 0x7f43441072a0 2024-04-22T23:23:21.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:21.974+0000 7f4349ea3700 1 -- 172.21.15.67:0/2264617709 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f434406ade0 con 0x7f43441072a0 2024-04-22T23:23:21.977 INFO:tasks.workunit.client.0.smithi067.stdout:2/235: chown d1/d42 2282683 1 2024-04-22T23:23:21.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:21.975+0000 7f4349ea3700 1 -- 172.21.15.67:0/2264617709 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f434410fe80 con 0x7f43441072a0 2024-04-22T23:23:21.978 INFO:tasks.workunit.client.0.smithi067.stdout:0/167: sync 2024-04-22T23:23:21.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:21.977+0000 7f4340ff9700 1 -- 172.21.15.67:0/2264617709 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7f433800fa40 con 0x7f43441072a0 2024-04-22T23:23:21.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:21.977+0000 7f4340ff9700 1 --2- 172.21.15.67:0/2264617709 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f432c06c530 0x7f432c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:23:21.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:21.978+0000 7f4340ff9700 1 -- 172.21.15.67:0/2264617709 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f433808cb20 con 0x7f43441072a0 2024-04-22T23:23:21.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:21.978+0000 7f4342ffd700 1 --2- 172.21.15.67:0/2264617709 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f432c06c530 0x7f432c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:23:21.980 INFO:tasks.workunit.client.0.smithi067.stdout:3/267: creat d3/d4/d10/d38/d53/f54 x:0 0 0 2024-04-22T23:23:21.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:21.979+0000 7f4342ffd700 1 --2- 172.21.15.67:0/2264617709 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f432c06c530 0x7f432c06e9f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4334009de0 tx=0x7f4334009190 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:23:21.981 INFO:tasks.workunit.client.0.smithi067.stdout:2/236: mkdir d1/d10/d33/d46 0 2024-04-22T23:23:21.983 INFO:tasks.workunit.client.0.smithi067.stdout:4/304: truncate f4 2783086 0 2024-04-22T23:23:21.983 INFO:tasks.workunit.client.0.smithi067.stdout:4/305: chown d12/d5f/d2b 14 1 2024-04-22T23:23:21.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:21.982+0000 7f4340ff9700 1 -- 172.21.15.67:0/2264617709 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f433805adb0 con 0x7f43441072a0 2024-04-22T23:23:21.985 INFO:tasks.workunit.client.0.smithi067.stdout:9/159: rename d5/l30 to d5/dd/d18/l31 0 2024-04-22T23:23:21.987 INFO:tasks.workunit.client.0.smithi067.stdout:3/268: mkdir d3/d4/d10/d38/d53/d55 0 2024-04-22T23:23:21.989 INFO:tasks.workunit.client.0.smithi067.stdout:2/237: dread - d1/d12/d28/f2a zero size 2024-04-22T23:23:21.989 INFO:tasks.workunit.client.0.smithi067.stdout:4/306: symlink d12/l6d 0 2024-04-22T23:23:21.992 INFO:tasks.workunit.client.0.smithi067.stdout:9/160: dread d5/dd/d18/f1d [0,4194304] 0 2024-04-22T23:23:21.995 INFO:tasks.workunit.client.0.smithi067.stdout:4/307: creat d12/d42/d4b/d5e/f6e x:0 0 0 2024-04-22T23:23:21.996 INFO:tasks.workunit.client.0.smithi067.stdout:2/238: dread d1/d19/f2f [0,4194304] 0 2024-04-22T23:23:21.998 INFO:tasks.workunit.client.0.smithi067.stdout:0/168: dread db/de/ff [0,4194304] 0 2024-04-22T23:23:21.999 INFO:tasks.workunit.client.0.smithi067.stdout:2/239: readlink d1/l38 0 2024-04-22T23:23:22.000 INFO:tasks.workunit.client.0.smithi067.stdout:3/269: dwrite d3/d4/f20 [0,4194304] 0 2024-04-22T23:23:22.001 INFO:tasks.workunit.client.0.smithi067.stdout:4/308: dread - d12/d17/d3f/f63 zero size 2024-04-22T23:23:22.001 INFO:tasks.workunit.client.0.smithi067.stdout:2/240: readlink d1/d10/d1c/l34 0 2024-04-22T23:23:22.006 INFO:tasks.workunit.client.0.smithi067.stdout:9/161: dwrite d5/f9 [0,4194304] 0 2024-04-22T23:23:22.006 INFO:tasks.workunit.client.0.smithi067.stdout:4/309: write d12/d5f/d2b/f61 [571566,79094] 0 2024-04-22T23:23:22.010 INFO:tasks.workunit.client.0.smithi067.stdout:9/162: fsync d5/dd/d18/f1f 0 2024-04-22T23:23:22.010 INFO:tasks.workunit.client.0.smithi067.stdout:0/169: rename db/de/f22 to db/d2e/d31/f3a 0 2024-04-22T23:23:22.010 INFO:tasks.workunit.client.0.smithi067.stdout:9/163: write d5/dd/f1a [3610385,23953] 0 2024-04-22T23:23:22.011 INFO:tasks.workunit.client.0.smithi067.stdout:9/164: fdatasync d5/ff 0 2024-04-22T23:23:22.012 INFO:tasks.workunit.client.0.smithi067.stdout:9/165: chown d5/dd/f1a 1052819354 1 2024-04-22T23:23:22.013 INFO:tasks.workunit.client.0.smithi067.stdout:4/310: symlink d12/d17/d3f/l6f 0 2024-04-22T23:23:22.014 INFO:tasks.workunit.client.0.smithi067.stdout:9/166: fsync d5/dd/d18/f1f 0 2024-04-22T23:23:22.016 INFO:tasks.workunit.client.0.smithi067.stdout:2/241: getdents d1 0 2024-04-22T23:23:22.017 INFO:tasks.workunit.client.0.smithi067.stdout:9/167: fdatasync d5/fa 0 2024-04-22T23:23:22.017 INFO:tasks.workunit.client.0.smithi067.stdout:4/311: truncate d12/d5f/f35 856727 0 2024-04-22T23:23:22.019 INFO:tasks.workunit.client.0.smithi067.stdout:4/312: dread - d12/f52 zero size 2024-04-22T23:23:22.020 INFO:tasks.workunit.client.0.smithi067.stdout:0/170: dwrite db/de/f1b [0,4194304] 0 2024-04-22T23:23:22.021 INFO:tasks.workunit.client.0.smithi067.stdout:9/168: creat d5/dd/f32 x:0 0 0 2024-04-22T23:23:22.023 INFO:tasks.workunit.client.0.smithi067.stdout:9/169: write d5/f10 [1903761,20332] 0 2024-04-22T23:23:22.024 INFO:tasks.workunit.client.0.smithi067.stdout:4/313: rename d12/d42/d4b/d5e to d12/d42/d4b/d70 0 2024-04-22T23:23:22.027 INFO:tasks.workunit.client.0.smithi067.stdout:0/171: symlink db/d21/d29/d2c/l3b 0 2024-04-22T23:23:22.031 INFO:tasks.workunit.client.0.smithi067.stdout:9/170: rmdir d5/dd 39 2024-04-22T23:23:22.033 INFO:tasks.workunit.client.0.smithi067.stdout:0/172: rename db/d21/d29/d2d/f35 to db/de/d11/f3c 0 2024-04-22T23:23:22.035 INFO:tasks.workunit.client.0.smithi067.stdout:0/173: write db/de/f1b [340190,100029] 0 2024-04-22T23:23:22.035 INFO:tasks.workunit.client.0.smithi067.stdout:4/314: truncate d12/f54 106306 0 2024-04-22T23:23:22.036 INFO:tasks.workunit.client.0.smithi067.stdout:4/315: fsync d12/d17/f51 0 2024-04-22T23:23:22.038 INFO:tasks.workunit.client.0.smithi067.stdout:9/171: symlink d5/l33 0 2024-04-22T23:23:22.042 INFO:tasks.workunit.client.0.smithi067.stdout:0/174: rename db/de/d11/f27 to db/d21/d29/d2c/f3d 0 2024-04-22T23:23:22.044 INFO:tasks.workunit.client.0.smithi067.stdout:4/316: symlink d12/d42/l71 0 2024-04-22T23:23:22.044 INFO:tasks.workunit.client.0.smithi067.stdout:4/317: readlink d12/d17/d3f/l6f 0 2024-04-22T23:23:22.048 INFO:tasks.workunit.client.0.smithi067.stdout:2/242: dread d1/f4 [0,4194304] 0 2024-04-22T23:23:22.048 INFO:tasks.workunit.client.0.smithi067.stdout:0/175: symlink db/d21/d29/d2c/l3e 0 2024-04-22T23:23:22.051 INFO:tasks.workunit.client.0.smithi067.stdout:9/172: dread d5/dd/d18/f1d [0,4194304] 0 2024-04-22T23:23:22.053 INFO:tasks.workunit.client.0.smithi067.stdout:0/176: creat db/d21/d29/f3f x:0 0 0 2024-04-22T23:23:22.055 INFO:tasks.workunit.client.0.smithi067.stdout:9/173: creat d5/d21/f34 x:0 0 0 2024-04-22T23:23:22.055 INFO:tasks.workunit.client.0.smithi067.stdout:0/177: stat db/de/d11/d1a/c24 0 2024-04-22T23:23:22.056 INFO:tasks.workunit.client.0.smithi067.stdout:2/243: dwrite d1/fc [0,4194304] 0 2024-04-22T23:23:22.059 INFO:tasks.workunit.client.0.smithi067.stdout:0/178: creat db/d2e/f40 x:0 0 0 2024-04-22T23:23:22.063 INFO:tasks.workunit.client.0.smithi067.stdout:2/244: creat d1/d10/d33/d46/f47 x:0 0 0 2024-04-22T23:23:22.063 INFO:tasks.workunit.client.0.smithi067.stdout:2/245: chown d1/d12/l29 3165466 1 2024-04-22T23:23:22.064 INFO:tasks.workunit.client.0.smithi067.stdout:0/179: creat db/d21/d29/d2c/f41 x:0 0 0 2024-04-22T23:23:22.066 INFO:tasks.workunit.client.0.smithi067.stdout:9/174: dwrite d5/dd/d18/f1d [0,4194304] 0 2024-04-22T23:23:22.066 INFO:tasks.workunit.client.0.smithi067.stdout:4/318: chown fe 37210 1 2024-04-22T23:23:22.067 INFO:tasks.workunit.client.0.smithi067.stdout:2/246: creat d1/d12/f48 x:0 0 0 2024-04-22T23:23:22.068 INFO:tasks.workunit.client.0.smithi067.stdout:9/175: dread - d5/dd/d18/f1f zero size 2024-04-22T23:23:22.070 INFO:tasks.workunit.client.0.smithi067.stdout:4/319: creat d12/d5f/d2b/f72 x:0 0 0 2024-04-22T23:23:22.072 INFO:tasks.workunit.client.0.smithi067.stdout:9/176: mknod d5/dd/d18/c35 0 2024-04-22T23:23:22.073 INFO:tasks.workunit.client.0.smithi067.stdout:4/320: symlink d12/d5f/l73 0 2024-04-22T23:23:22.074 INFO:tasks.workunit.client.0.smithi067.stdout:4/321: dread - d12/d5f/d48/d56/f5c zero size 2024-04-22T23:23:22.077 INFO:tasks.workunit.client.0.smithi067.stdout:4/322: symlink d12/d17/l74 0 2024-04-22T23:23:22.077 INFO:tasks.workunit.client.0.smithi067.stdout:4/323: chown d12/d42/d4b/d70/c62 77282 1 2024-04-22T23:23:22.078 INFO:tasks.workunit.client.0.smithi067.stdout:4/324: write d12/f58 [327438,40465] 0 2024-04-22T23:23:22.080 INFO:tasks.workunit.client.0.smithi067.stdout:4/325: truncate d12/d5f/d48/f68 410476 0 2024-04-22T23:23:22.080 INFO:tasks.workunit.client.0.smithi067.stdout:4/326: read d12/f54 [85542,67009] 0 2024-04-22T23:23:22.082 INFO:tasks.workunit.client.0.smithi067.stdout:4/327: mkdir d12/d75 0 2024-04-22T23:23:22.084 INFO:tasks.workunit.client.0.smithi067.stdout:4/328: read d12/d5f/f28 [2004368,32931] 0 2024-04-22T23:23:22.086 INFO:tasks.workunit.client.0.smithi067.stdout:4/329: symlink d12/d5f/d48/d56/l76 0 2024-04-22T23:23:22.092 INFO:tasks.workunit.client.0.smithi067.stdout:4/330: dwrite fd [0,4194304] 0 2024-04-22T23:23:22.096 INFO:tasks.workunit.client.0.smithi067.stdout:0/180: dread db/de/f1c [0,4194304] 0 2024-04-22T23:23:22.097 INFO:tasks.workunit.client.0.smithi067.stdout:9/177: dread d5/fb [0,4194304] 0 2024-04-22T23:23:22.100 INFO:tasks.workunit.client.0.smithi067.stdout:0/181: creat db/d21/d29/d2d/f42 x:0 0 0 2024-04-22T23:23:22.100 INFO:tasks.workunit.client.0.smithi067.stdout:4/331: dwrite d12/d2e/f57 [0,4194304] 0 2024-04-22T23:23:22.106 INFO:tasks.workunit.client.0.smithi067.stdout:0/182: write db/de/d11/d1a/f38 [783979,72215] 0 2024-04-22T23:23:22.109 INFO:tasks.workunit.client.0.smithi067.stdout:9/178: dread d5/f9 [0,4194304] 0 2024-04-22T23:23:22.110 INFO:tasks.workunit.client.0.smithi067.stdout:4/332: dwrite f5 [4194304,4194304] 0 2024-04-22T23:23:22.110 INFO:tasks.workunit.client.0.smithi067.stdout:9/179: readlink d5/l15 0 2024-04-22T23:23:22.119 INFO:tasks.workunit.client.0.smithi067.stdout:0/183: dread db/de/ff [0,4194304] 0 2024-04-22T23:23:22.119 INFO:tasks.workunit.client.0.smithi067.stdout:4/333: dwrite d12/d5f/d48/f68 [0,4194304] 0 2024-04-22T23:23:22.122 INFO:tasks.workunit.client.0.smithi067.stdout:9/180: creat d5/f36 x:0 0 0 2024-04-22T23:23:22.124 INFO:tasks.workunit.client.0.smithi067.stdout:4/334: mkdir d12/d42/d4b/d77 0 2024-04-22T23:23:22.124 INFO:tasks.workunit.client.0.smithi067.stdout:9/181: dread - d5/dd/d18/f1f zero size 2024-04-22T23:23:22.127 INFO:tasks.workunit.client.0.smithi067.stdout:4/335: creat d12/d42/f78 x:0 0 0 2024-04-22T23:23:22.129 INFO:tasks.workunit.client.0.smithi067.stdout:9/182: chown d5/dd/d18/l31 43 1 2024-04-22T23:23:22.130 INFO:tasks.workunit.client.0.smithi067.stdout:4/336: mknod d12/d5f/d69/c79 0 2024-04-22T23:23:22.132 INFO:tasks.workunit.client.0.smithi067.stdout:0/184: dwrite db/d21/d29/d2c/f3d [0,4194304] 0 2024-04-22T23:23:22.132 INFO:tasks.workunit.client.0.smithi067.stdout:4/337: write d12/d42/f78 [825802,50966] 0 2024-04-22T23:23:22.137 INFO:tasks.workunit.client.0.smithi067.stdout:4/338: mkdir d12/d17/d3f/d7a 0 2024-04-22T23:23:22.139 INFO:tasks.workunit.client.0.smithi067.stdout:0/185: write db/de/d11/f32 [4608182,113776] 0 2024-04-22T23:23:22.140 INFO:tasks.workunit.client.0.smithi067.stdout:9/183: dwrite d5/ff [0,4194304] 0 2024-04-22T23:23:22.141 INFO:tasks.workunit.client.0.smithi067.stdout:4/339: symlink d12/d17/l7b 0 2024-04-22T23:23:22.159 INFO:tasks.workunit.client.0.smithi067.stdout:0/186: creat db/d21/d29/f43 x:0 0 0 2024-04-22T23:23:22.160 INFO:tasks.workunit.client.0.smithi067.stdout:0/187: chown db/de/d11/f3c 173 1 2024-04-22T23:23:22.160 INFO:tasks.workunit.client.0.smithi067.stdout:4/340: creat d12/d5f/d69/f7c x:0 0 0 2024-04-22T23:23:22.160 INFO:tasks.workunit.client.0.smithi067.stdout:0/188: chown c3 1690745 1 2024-04-22T23:23:22.162 INFO:tasks.workunit.client.0.smithi067.stdout:0/189: dread - db/de/d11/f3c zero size 2024-04-22T23:23:22.162 INFO:tasks.workunit.client.0.smithi067.stdout:9/184: rename d5/dd/l2d to d5/d21/l37 0 2024-04-22T23:23:22.163 INFO:tasks.workunit.client.0.smithi067.stdout:4/341: symlink d12/d17/d3f/l7d 0 2024-04-22T23:23:22.165 INFO:tasks.workunit.client.0.smithi067.stdout:9/185: rename d5/c23 to d5/d2e/c38 0 2024-04-22T23:23:22.166 INFO:tasks.workunit.client.0.smithi067.stdout:4/342: truncate d12/f46 252574 0 2024-04-22T23:23:22.166 INFO:tasks.workunit.client.0.smithi067.stdout:9/186: truncate d5/d21/f2b 181262 0 2024-04-22T23:23:22.169 INFO:tasks.workunit.client.0.smithi067.stdout:9/187: mkdir d5/dd/d18/d39 0 2024-04-22T23:23:22.171 INFO:tasks.workunit.client.0.smithi067.stdout:9/188: symlink d5/d21/d2c/l3a 0 2024-04-22T23:23:22.175 INFO:tasks.workunit.client.0.smithi067.stdout:0/190: dread db/d2e/d31/f3a [0,4194304] 0 2024-04-22T23:23:22.176 INFO:tasks.workunit.client.0.smithi067.stdout:0/191: chown db/de/d11/f26 323 1 2024-04-22T23:23:22.182 INFO:tasks.workunit.client.0.smithi067.stdout:4/343: dread f5 [0,4194304] 0 2024-04-22T23:23:22.184 INFO:tasks.workunit.client.0.smithi067.stdout:4/344: readlink d12/d17/l29 0 2024-04-22T23:23:22.186 INFO:tasks.workunit.client.0.smithi067.stdout:4/345: chown d12/d42/d4b 799 1 2024-04-22T23:23:22.187 INFO:tasks.workunit.client.0.smithi067.stdout:4/346: dread d12/f54 [0,4194304] 0 2024-04-22T23:23:22.188 INFO:tasks.workunit.client.0.smithi067.stdout:0/192: dwrite db/de/d11/d1a/f28 [0,4194304] 0 2024-04-22T23:23:22.190 INFO:tasks.workunit.client.0.smithi067.stdout:4/347: mkdir d12/d5f/d7e 0 2024-04-22T23:23:22.195 INFO:tasks.workunit.client.0.smithi067.stdout:4/348: dread d12/d17/d47/f49 [0,4194304] 0 2024-04-22T23:23:22.196 INFO:tasks.workunit.client.0.smithi067.stdout:0/193: link db/d21/l36 db/d21/d29/l44 0 2024-04-22T23:23:22.198 INFO:tasks.workunit.client.0.smithi067.stdout:0/194: write db/d21/d29/f43 [1007627,62919] 0 2024-04-22T23:23:22.198 INFO:tasks.workunit.client.0.smithi067.stdout:4/349: unlink d12/f32 0 2024-04-22T23:23:22.201 INFO:tasks.workunit.client.0.smithi067.stdout:0/195: dread db/de/ff [0,4194304] 0 2024-04-22T23:23:22.202 INFO:tasks.workunit.client.0.smithi067.stdout:4/350: mknod d12/d17/d3f/d7a/c7f 0 2024-04-22T23:23:22.203 INFO:tasks.workunit.client.0.smithi067.stdout:4/351: write d12/d17/d31/f50 [3463048,65392] 0 2024-04-22T23:23:22.209 INFO:tasks.workunit.client.0.smithi067.stdout:0/196: dwrite db/de/d11/f3c [0,4194304] 0 2024-04-22T23:23:22.214 INFO:tasks.workunit.client.0.smithi067.stdout:4/352: dwrite d12/d17/d31/f67 [0,4194304] 0 2024-04-22T23:23:22.214 INFO:tasks.workunit.client.1.smithi134.stderr:++ readlink -f fsstress 2024-04-22T23:23:22.214 INFO:tasks.workunit.client.1.smithi134.stderr:+ BIN=/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-04-22T23:23:22.214 INFO:tasks.workunit.client.1.smithi134.stderr:+ popd 2024-04-22T23:23:22.215 INFO:tasks.workunit.client.1.smithi134.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-22T23:23:22.215 INFO:tasks.workunit.client.1.smithi134.stderr:+ popd 2024-04-22T23:23:22.216 INFO:tasks.workunit.client.0.smithi067.stdout:0/197: creat db/de/d2a/f45 x:0 0 0 2024-04-22T23:23:22.216 INFO:tasks.workunit.client.1.smithi134.stdout:~/cephtest/mnt.1/client.1/tmp 2024-04-22T23:23:22.216 INFO:tasks.workunit.client.1.smithi134.stderr:++ mktemp -d -p . 2024-04-22T23:23:22.221 INFO:tasks.workunit.client.0.smithi067.stdout:0/198: mkdir db/d2e/d46 0 2024-04-22T23:23:22.222 INFO:tasks.workunit.client.0.smithi067.stdout:4/353: dwrite d12/d5f/f5a [0,4194304] 0 2024-04-22T23:23:22.222 INFO:tasks.workunit.client.0.smithi067.stdout:1/258: sync 2024-04-22T23:23:22.224 INFO:tasks.workunit.client.1.smithi134.stderr:+ T=./tmp.FX8jKn5bsq 2024-04-22T23:23:22.224 INFO:tasks.workunit.client.1.smithi134.stderr:+ /home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.FX8jKn5bsq -l 1 -n 1000 -p 10 -v 2024-04-22T23:23:22.224 INFO:tasks.workunit.client.0.smithi067.stdout:1/259: fdatasync d8/d12/d40/f57 0 2024-04-22T23:23:22.228 INFO:tasks.workunit.client.0.smithi067.stdout:0/199: rename c3 to db/d21/c47 0 2024-04-22T23:23:22.230 INFO:tasks.workunit.client.0.smithi067.stdout:1/260: dread d8/f1d [0,4194304] 0 2024-04-22T23:23:22.231 INFO:tasks.workunit.client.1.smithi134.stdout:seed = 1713731901 2024-04-22T23:23:22.232 INFO:tasks.workunit.client.0.smithi067.stdout:4/354: rename d12/d17/l29 to d12/d2e/l80 0 2024-04-22T23:23:22.232 INFO:tasks.workunit.client.0.smithi067.stdout:1/261: stat d8/d9/d2f/d59 0 2024-04-22T23:23:22.234 INFO:tasks.workunit.client.0.smithi067.stdout:1/262: chown d8/f11 625183 1 2024-04-22T23:23:22.235 INFO:tasks.workunit.client.0.smithi067.stdout:1/263: stat d8/fc 0 2024-04-22T23:23:22.236 INFO:tasks.workunit.client.0.smithi067.stdout:1/264: chown d8/d9/f43 12461 1 2024-04-22T23:23:22.237 INFO:tasks.workunit.client.0.smithi067.stdout:0/200: dwrite db/d21/d29/d2c/f41 [0,4194304] 0 2024-04-22T23:23:22.237 INFO:tasks.workunit.client.1.smithi134.stdout:0/0: chown . 81650 1 2024-04-22T23:23:22.237 INFO:tasks.workunit.client.0.smithi067.stdout:1/265: chown f4 1242 1 2024-04-22T23:23:22.238 INFO:tasks.workunit.client.1.smithi134.stdout:0/1: chown . 19 1 2024-04-22T23:23:22.238 INFO:tasks.workunit.client.1.smithi134.stdout:0/2: truncate - no filename 2024-04-22T23:23:22.238 INFO:tasks.workunit.client.1.smithi134.stdout:0/3: read - no filename 2024-04-22T23:23:22.239 INFO:tasks.workunit.client.0.smithi067.stdout:1/266: write d8/d24/d34/f39 [316477,72932] 0 2024-04-22T23:23:22.240 INFO:tasks.workunit.client.0.smithi067.stdout:0/201: mkdir db/de/d2a/d48 0 2024-04-22T23:23:22.241 INFO:tasks.workunit.client.1.smithi134.stdout:0/4: creat f0 x:0 0 0 2024-04-22T23:23:22.243 INFO:tasks.workunit.client.0.smithi067.stdout:0/202: creat db/d21/d29/d2d/f49 x:0 0 0 2024-04-22T23:23:22.243 INFO:tasks.workunit.client.1.smithi134.stdout:0/5: write f0 [1048130,815] 0 2024-04-22T23:23:22.244 INFO:tasks.workunit.client.1.smithi134.stdout:2/0: creat f0 x:0 0 0 2024-04-22T23:23:22.244 INFO:tasks.workunit.client.0.smithi067.stdout:0/203: write db/de/f12 [1473550,51987] 0 2024-04-22T23:23:22.245 INFO:tasks.workunit.client.1.smithi134.stdout:2/1: fdatasync f0 0 2024-04-22T23:23:22.246 INFO:tasks.workunit.client.1.smithi134.stdout:0/6: mkdir d1 0 2024-04-22T23:23:22.246 INFO:tasks.workunit.client.1.smithi134.stdout:2/2: dread - f0 zero size 2024-04-22T23:23:22.249 INFO:tasks.workunit.client.0.smithi067.stdout:0/204: rename c1 to db/d2e/c4a 0 2024-04-22T23:23:22.249 INFO:tasks.workunit.client.1.smithi134.stdout:4/0: rename - no filename 2024-04-22T23:23:22.249 INFO:tasks.workunit.client.1.smithi134.stdout:4/1: link - no file 2024-04-22T23:23:22.250 INFO:tasks.workunit.client.1.smithi134.stdout:4/2: write - no filename 2024-04-22T23:23:22.250 INFO:tasks.workunit.client.1.smithi134.stdout:2/3: symlink l1 0 2024-04-22T23:23:22.250 INFO:tasks.workunit.client.0.smithi067.stdout:0/205: write db/d21/d29/f3f [763764,27049] 0 2024-04-22T23:23:22.250 INFO:tasks.workunit.client.1.smithi134.stdout:3/0: creat f0 x:0 0 0 2024-04-22T23:23:22.251 INFO:tasks.workunit.client.1.smithi134.stdout:2/4: truncate f0 746978 0 2024-04-22T23:23:22.251 INFO:tasks.workunit.client.0.smithi067.stdout:0/206: chown db/de/d11/d1a/c24 241118256 1 2024-04-22T23:23:22.251 INFO:tasks.workunit.client.1.smithi134.stdout:2/5: chown l1 1 1 2024-04-22T23:23:22.252 INFO:tasks.workunit.client.0.smithi067.stdout:4/355: truncate d12/d17/f2f 325397 0 2024-04-22T23:23:22.253 INFO:tasks.workunit.client.1.smithi134.stdout:0/7: mkdir d1/d2 0 2024-04-22T23:23:22.253 INFO:tasks.workunit.client.0.smithi067.stdout:4/356: dread - d12/d17/d3f/f63 zero size 2024-04-22T23:23:22.254 INFO:tasks.workunit.client.0.smithi067.stdout:0/207: symlink db/de/d2a/d48/l4b 0 2024-04-22T23:23:22.254 INFO:tasks.workunit.client.1.smithi134.stdout:3/1: mkdir d1 0 2024-04-22T23:23:22.255 INFO:tasks.workunit.client.1.smithi134.stdout:2/6: mkdir d2 0 2024-04-22T23:23:22.257 INFO:tasks.workunit.client.1.smithi134.stdout:6/0: dwrite - no filename 2024-04-22T23:23:22.258 INFO:tasks.workunit.client.0.smithi067.stdout:4/357: dread d12/d5f/d48/f68 [0,4194304] 0 2024-04-22T23:23:22.258 INFO:tasks.workunit.client.1.smithi134.stdout:4/3: creat f0 x:0 0 0 2024-04-22T23:23:22.260 INFO:tasks.workunit.client.1.smithi134.stdout:7/0: write - no filename 2024-04-22T23:23:22.260 INFO:tasks.workunit.client.1.smithi134.stdout:7/1: dwrite - no filename 2024-04-22T23:23:22.261 INFO:tasks.workunit.client.1.smithi134.stdout:7/2: dwrite - no filename 2024-04-22T23:23:22.261 INFO:tasks.workunit.client.1.smithi134.stdout:7/3: dread - no filename 2024-04-22T23:23:22.261 INFO:tasks.workunit.client.1.smithi134.stdout:7/4: unlink - no file 2024-04-22T23:23:22.261 INFO:tasks.workunit.client.1.smithi134.stdout:7/5: chown . 1 1 2024-04-22T23:23:22.261 INFO:tasks.workunit.client.1.smithi134.stdout:7/6: rmdir - no directory 2024-04-22T23:23:22.261 INFO:tasks.workunit.client.1.smithi134.stdout:7/7: rmdir - no directory 2024-04-22T23:23:22.261 INFO:tasks.workunit.client.1.smithi134.stdout:7/8: truncate - no filename 2024-04-22T23:23:22.262 INFO:tasks.workunit.client.0.smithi067.stdout:0/208: dwrite db/d21/d29/f3f [0,4194304] 0 2024-04-22T23:23:22.262 INFO:tasks.workunit.client.1.smithi134.stdout:3/2: creat d1/f2 x:0 0 0 2024-04-22T23:23:22.262 INFO:tasks.workunit.client.1.smithi134.stdout:6/1: getdents . 0 2024-04-22T23:23:22.263 INFO:tasks.workunit.client.1.smithi134.stdout:6/2: dwrite - no filename 2024-04-22T23:23:22.263 INFO:tasks.workunit.client.1.smithi134.stdout:6/3: chown . 236500 1 2024-04-22T23:23:22.263 INFO:tasks.workunit.client.1.smithi134.stdout:6/4: rmdir - no directory 2024-04-22T23:23:22.263 INFO:tasks.workunit.client.1.smithi134.stdout:6/5: dread - no filename 2024-04-22T23:23:22.264 INFO:tasks.workunit.client.0.smithi067.stdout:4/358: link fd d12/d5f/d69/f81 0 2024-04-22T23:23:22.265 INFO:tasks.workunit.client.1.smithi134.stdout:3/3: unlink f0 0 2024-04-22T23:23:22.265 INFO:tasks.workunit.client.0.smithi067.stdout:4/359: chown d12/d42/d4b/d70/c62 5 1 2024-04-22T23:23:22.266 INFO:tasks.workunit.client.1.smithi134.stdout:7/9: creat f0 x:0 0 0 2024-04-22T23:23:22.269 INFO:tasks.workunit.client.0.smithi067.stdout:4/360: symlink d12/d42/d4b/l82 0 2024-04-22T23:23:22.271 INFO:tasks.workunit.client.0.smithi067.stdout:4/361: rmdir d12/d17/d3f/d7a 39 2024-04-22T23:23:22.271 INFO:tasks.workunit.client.0.smithi067.stdout:4/362: dread - d12/d17/d3f/f63 zero size 2024-04-22T23:23:22.277 INFO:tasks.workunit.client.0.smithi067.stdout:4/363: dwrite d12/f52 [0,4194304] 0 2024-04-22T23:23:22.282 INFO:tasks.workunit.client.0.smithi067.stdout:4/364: dread fd [0,4194304] 0 2024-04-22T23:23:22.283 INFO:tasks.workunit.client.0.smithi067.stdout:4/365: dread - d12/d17/d47/f6c zero size 2024-04-22T23:23:22.284 INFO:tasks.workunit.client.0.smithi067.stdout:4/366: stat d12/d17/f33 0 2024-04-22T23:23:22.286 INFO:tasks.workunit.client.0.smithi067.stdout:0/209: dread db/de/f12 [4194304,4194304] 0 2024-04-22T23:23:22.287 INFO:tasks.workunit.client.0.smithi067.stdout:4/367: dread f5 [0,4194304] 0 2024-04-22T23:23:22.287 INFO:tasks.workunit.client.0.smithi067.stdout:0/210: dread - db/d21/d29/d2d/f49 zero size 2024-04-22T23:23:22.291 INFO:tasks.workunit.client.0.smithi067.stdout:4/368: link d12/f16 d12/d42/f83 0 2024-04-22T23:23:22.292 INFO:tasks.workunit.client.0.smithi067.stdout:4/369: readlink d12/d17/l74 0 2024-04-22T23:23:22.294 INFO:tasks.workunit.client.0.smithi067.stdout:0/211: dwrite db/d21/d29/f43 [0,4194304] 0 2024-04-22T23:23:22.295 INFO:tasks.workunit.client.0.smithi067.stdout:0/212: stat db/d21/d29/f3f 0 2024-04-22T23:23:22.296 INFO:tasks.workunit.client.0.smithi067.stdout:0/213: readlink db/de/d11/d1a/l1d 0 2024-04-22T23:23:22.297 INFO:tasks.workunit.client.0.smithi067.stdout:4/370: dread d12/d5f/f27 [0,4194304] 0 2024-04-22T23:23:22.298 INFO:tasks.workunit.client.0.smithi067.stdout:0/214: creat db/d21/f4c x:0 0 0 2024-04-22T23:23:22.300 INFO:tasks.workunit.client.0.smithi067.stdout:4/371: mkdir d12/d17/d3f/d84 0 2024-04-22T23:23:22.304 INFO:tasks.workunit.client.0.smithi067.stdout:0/215: dwrite db/d21/d29/d2d/f49 [0,4194304] 0 2024-04-22T23:23:22.307 INFO:tasks.workunit.client.0.smithi067.stdout:0/216: dread db/de/ff [0,4194304] 0 2024-04-22T23:23:22.311 INFO:tasks.workunit.client.0.smithi067.stdout:0/217: dread db/d21/d29/f3f [0,4194304] 0 2024-04-22T23:23:22.315 INFO:tasks.workunit.client.0.smithi067.stdout:0/218: getdents db/de/d2a 0 2024-04-22T23:23:22.317 INFO:tasks.workunit.client.0.smithi067.stdout:0/219: creat db/de/f4d x:0 0 0 2024-04-22T23:23:22.324 INFO:tasks.workunit.client.0.smithi067.stdout:0/220: dwrite db/d21/d29/d2c/f41 [4194304,4194304] 0 2024-04-22T23:23:22.330 INFO:tasks.workunit.client.0.smithi067.stdout:0/221: dwrite db/d21/d29/f43 [0,4194304] 0 2024-04-22T23:23:22.331 INFO:tasks.workunit.client.0.smithi067.stdout:0/222: readlink db/de/d11/d1a/l1d 0 2024-04-22T23:23:22.332 INFO:tasks.workunit.client.0.smithi067.stdout:0/223: readlink db/de/d11/l23 0 2024-04-22T23:23:22.334 INFO:tasks.workunit.client.0.smithi067.stdout:0/224: unlink db/de/d11/d1a/f28 0 2024-04-22T23:23:22.334 INFO:tasks.workunit.client.0.smithi067.stdout:0/225: chown db/de/d11/f26 18783 1 2024-04-22T23:23:22.335 INFO:tasks.workunit.client.0.smithi067.stdout:0/226: chown db/de/d11/d1a/c37 652425 1 2024-04-22T23:23:22.336 INFO:tasks.workunit.client.0.smithi067.stdout:0/227: chown db/d2e/c4a 39445841 1 2024-04-22T23:23:22.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.352+0000 7f4349ea3700 1 -- 172.21.15.67:0/2264617709 --> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f434410cdf0 con 0x7f432c06c530 2024-04-22T23:23:22.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.353+0000 7f4340ff9700 1 -- 172.21.15.67:0/2264617709 <== mgr.14215 v2:172.21.15.67:6800/2589747850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+321 (secure 0 0 0) 0x7f434410cdf0 con 0x7f432c06c530 2024-04-22T23:23:22.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.355+0000 7f4349ea3700 1 -- 172.21.15.67:0/2264617709 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f432c06c530 msgr2=0x7f432c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:22.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.355+0000 7f4349ea3700 1 --2- 172.21.15.67:0/2264617709 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f432c06c530 0x7f432c06e9f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4334009de0 tx=0x7f4334009190 comp rx=0 tx=0).stop 2024-04-22T23:23:22.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.355+0000 7f4349ea3700 1 -- 172.21.15.67:0/2264617709 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f43441072a0 msgr2=0x7f434419cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:22.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.355+0000 7f4349ea3700 1 --2- 172.21.15.67:0/2264617709 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f43441072a0 0x7f434419cb80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4338000c00 tx=0x7f433800bd70 comp rx=0 tx=0).stop 2024-04-22T23:23:22.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.355+0000 7f4349ea3700 1 -- 172.21.15.67:0/2264617709 shutdown_connections 2024-04-22T23:23:22.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.355+0000 7f4349ea3700 1 --2- 172.21.15.67:0/2264617709 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f432c06c530 0x7f432c06e9f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:22.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.355+0000 7f4349ea3700 1 --2- 172.21.15.67:0/2264617709 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f43441072a0 0x7f434419cb80 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:22.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.355+0000 7f4349ea3700 1 --2- 172.21.15.67:0/2264617709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43441084a0 0x7f434419d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:22.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.355+0000 7f4349ea3700 1 -- 172.21.15.67:0/2264617709 >> 172.21.15.67:0/2264617709 conn(0x7f434406a230 msgr2=0x7f434410b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:23:22.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.355+0000 7f4349ea3700 1 -- 172.21.15.67:0/2264617709 shutdown_connections 2024-04-22T23:23:22.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.355+0000 7f4349ea3700 1 -- 172.21.15.67:0/2264617709 wait complete. 2024-04-22T23:23:22.367 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:23:22.393 INFO:tasks.workunit.client.0.smithi067.stdout:6/183: sync 2024-04-22T23:23:22.394 INFO:tasks.workunit.client.0.smithi067.stdout:6/184: chown d17/c2d 386234002 1 2024-04-22T23:23:22.395 INFO:tasks.workunit.client.0.smithi067.stdout:6/185: write d17/f29 [1137118,119561] 0 2024-04-22T23:23:22.399 INFO:tasks.workunit.client.0.smithi067.stdout:6/186: rename f10 to d17/d39/f3f 0 2024-04-22T23:23:22.400 INFO:tasks.workunit.client.0.smithi067.stdout:6/187: write f12 [640951,49634] 0 2024-04-22T23:23:22.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.465+0000 7fdf1691a700 1 -- 172.21.15.67:0/1542947601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf100ffbb0 msgr2=0x7fdf100fffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:22.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.465+0000 7fdf1691a700 1 --2- 172.21.15.67:0/1542947601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf100ffbb0 0x7fdf100fffd0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fdf00009910 tx=0x7fdf00009c20 comp rx=0 tx=0).stop 2024-04-22T23:23:22.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.465+0000 7fdf1691a700 1 -- 172.21.15.67:0/1542947601 shutdown_connections 2024-04-22T23:23:22.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.465+0000 7fdf1691a700 1 --2- 172.21.15.67:0/1542947601 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf10100db0 0x7fdf10101210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:22.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.465+0000 7fdf1691a700 1 --2- 172.21.15.67:0/1542947601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf100ffbb0 0x7fdf100fffd0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:22.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.465+0000 7fdf1691a700 1 -- 172.21.15.67:0/1542947601 >> 172.21.15.67:0/1542947601 conn(0x7fdf100fb150 msgr2=0x7fdf100fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:23:22.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.466+0000 7fdf1691a700 1 -- 172.21.15.67:0/1542947601 shutdown_connections 2024-04-22T23:23:22.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.466+0000 7fdf1691a700 1 -- 172.21.15.67:0/1542947601 wait complete. 2024-04-22T23:23:22.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.466+0000 7fdf1691a700 1 Processor -- start 2024-04-22T23:23:22.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.466+0000 7fdf1691a700 1 -- start start 2024-04-22T23:23:22.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.466+0000 7fdf1691a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf100ffbb0 0x7fdf101964c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:23:22.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.467+0000 7fdf1691a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf10100db0 0x7fdf10196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:23:22.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.467+0000 7fdf1691a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf10197020 con 0x7fdf100ffbb0 2024-04-22T23:23:22.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.467+0000 7fdf1691a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf10197160 con 0x7fdf10100db0 2024-04-22T23:23:22.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.467+0000 7fdf0ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf100ffbb0 0x7fdf101964c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:23:22.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.467+0000 7fdf0ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf100ffbb0 0x7fdf101964c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40724/0 (socket says 172.21.15.67:40724) 2024-04-22T23:23:22.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.467+0000 7fdf0ffff700 1 -- 172.21.15.67:0/3447106534 learned_addr learned my addr 172.21.15.67:0/3447106534 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:23:22.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.467+0000 7fdf0f7fe700 1 --2- 172.21.15.67:0/3447106534 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf10100db0 0x7fdf10196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:23:22.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.467+0000 7fdf0ffff700 1 -- 172.21.15.67:0/3447106534 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf10100db0 msgr2=0x7fdf10196a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:22.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.467+0000 7fdf0ffff700 1 --2- 172.21.15.67:0/3447106534 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf10100db0 0x7fdf10196a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:22.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.467+0000 7fdf0ffff700 1 -- 172.21.15.67:0/3447106534 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf000095f0 con 0x7fdf100ffbb0 2024-04-22T23:23:22.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.467+0000 7fdf0ffff700 1 --2- 172.21.15.67:0/3447106534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf100ffbb0 0x7fdf101964c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fdf0000bda0 tx=0x7fdf0000bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:23:22.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.467+0000 7fdf0d7fa700 1 -- 172.21.15.67:0/3447106534 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf0001d070 con 0x7fdf100ffbb0 2024-04-22T23:23:22.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.467+0000 7fdf1691a700 1 -- 172.21.15.67:0/3447106534 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf1019bbb0 con 0x7fdf100ffbb0 2024-04-22T23:23:22.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.467+0000 7fdf0d7fa700 1 -- 172.21.15.67:0/3447106534 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fdf0000f460 con 0x7fdf100ffbb0 2024-04-22T23:23:22.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.467+0000 7fdf0d7fa700 1 -- 172.21.15.67:0/3447106534 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf00021640 con 0x7fdf100ffbb0 2024-04-22T23:23:22.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.468+0000 7fdf1691a700 1 -- 172.21.15.67:0/3447106534 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf1019c0a0 con 0x7fdf100ffbb0 2024-04-22T23:23:22.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.469+0000 7fdf1691a700 1 -- 172.21.15.67:0/3447106534 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf10108710 con 0x7fdf100ffbb0 2024-04-22T23:23:22.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.469+0000 7fdf0d7fa700 1 -- 172.21.15.67:0/3447106534 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fdf0000f5d0 con 0x7fdf100ffbb0 2024-04-22T23:23:22.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.469+0000 7fdf0d7fa700 1 --2- 172.21.15.67:0/3447106534 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fdef806c4e0 0x7fdef806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:23:22.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.469+0000 7fdf0d7fa700 1 -- 172.21.15.67:0/3447106534 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fdf0008cb50 con 0x7fdf100ffbb0 2024-04-22T23:23:22.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.469+0000 7fdf0f7fe700 1 --2- 172.21.15.67:0/3447106534 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fdef806c4e0 0x7fdef806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:23:22.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.470+0000 7fdf0f7fe700 1 --2- 172.21.15.67:0/3447106534 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fdef806c4e0 0x7fdef806e9a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdf04007be0 tx=0x7fdf040073d0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:23:22.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.478+0000 7fdf0d7fa700 1 -- 172.21.15.67:0/3447106534 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fdf0005ad60 con 0x7fdf100ffbb0 2024-04-22T23:23:22.492 INFO:tasks.workunit.client.0.smithi067.stdout:6/188: read d17/f25 [590802,77841] 0 2024-04-22T23:23:22.493 INFO:tasks.workunit.client.0.smithi067.stdout:6/189: write d17/f31 [1082439,82499] 0 2024-04-22T23:23:22.498 INFO:tasks.workunit.client.0.smithi067.stdout:6/190: dwrite f12 [0,4194304] 0 2024-04-22T23:23:22.500 INFO:tasks.workunit.client.0.smithi067.stdout:6/191: creat d17/d39/d3a/f40 x:0 0 0 2024-04-22T23:23:22.503 INFO:tasks.workunit.client.0.smithi067.stdout:6/192: mknod d17/d39/d3e/c41 0 2024-04-22T23:23:22.510 INFO:tasks.workunit.client.0.smithi067.stdout:6/193: dwrite d17/f26 [0,4194304] 0 2024-04-22T23:23:22.511 INFO:tasks.workunit.client.0.smithi067.stdout:8/267: sync 2024-04-22T23:23:22.513 INFO:tasks.workunit.client.0.smithi067.stdout:6/194: creat d17/f42 x:0 0 0 2024-04-22T23:23:22.515 INFO:tasks.workunit.client.1.smithi134.stdout:6/6: mkdir d0 0 2024-04-22T23:23:22.515 INFO:tasks.workunit.client.1.smithi134.stdout:6/7: dread - no filename 2024-04-22T23:23:22.515 INFO:tasks.workunit.client.1.smithi134.stdout:6/8: fdatasync - no filename 2024-04-22T23:23:22.515 INFO:tasks.workunit.client.1.smithi134.stdout:6/9: truncate - no filename 2024-04-22T23:23:22.515 INFO:tasks.workunit.client.1.smithi134.stdout:6/10: truncate - no filename 2024-04-22T23:23:22.515 INFO:tasks.workunit.client.1.smithi134.stdout:6/11: dread - no filename 2024-04-22T23:23:22.515 INFO:tasks.workunit.client.1.smithi134.stdout:6/12: dread - no filename 2024-04-22T23:23:22.517 INFO:tasks.workunit.client.0.smithi067.stdout:8/268: getdents d1/d3 0 2024-04-22T23:23:22.517 INFO:tasks.workunit.client.1.smithi134.stdout:3/4: unlink d1/f2 0 2024-04-22T23:23:22.523 INFO:tasks.workunit.client.0.smithi067.stdout:8/269: dwrite d1/d3/d27/f33 [0,4194304] 0 2024-04-22T23:23:22.525 INFO:tasks.workunit.client.0.smithi067.stdout:4/372: fsync d12/d17/f2f 0 2024-04-22T23:23:22.556 INFO:tasks.workunit.client.1.smithi134.stdout:0/8: truncate f0 975994 0 2024-04-22T23:23:22.564 INFO:tasks.workunit.client.0.smithi067.stdout:8/270: truncate d1/f6 2194118 0 2024-04-22T23:23:22.567 INFO:tasks.workunit.client.0.smithi067.stdout:4/373: rename d12/d2e/c45 to d12/d5f/d48/c85 0 2024-04-22T23:23:22.569 INFO:tasks.workunit.client.0.smithi067.stdout:4/374: truncate d12/d42/f83 4206264 0 2024-04-22T23:23:22.569 INFO:tasks.workunit.client.0.smithi067.stdout:8/271: creat d1/d3/d10/d3b/f54 x:0 0 0 2024-04-22T23:23:22.571 INFO:tasks.workunit.client.0.smithi067.stdout:0/228: dread db/fd [0,4194304] 0 2024-04-22T23:23:22.573 INFO:tasks.workunit.client.0.smithi067.stdout:4/375: dread fa [0,4194304] 0 2024-04-22T23:23:22.577 INFO:tasks.workunit.client.0.smithi067.stdout:4/376: creat d12/d17/d47/f86 x:0 0 0 2024-04-22T23:23:22.581 INFO:tasks.workunit.client.0.smithi067.stdout:0/229: write db/d21/d29/d2c/f41 [9199332,25136] 0 2024-04-22T23:23:22.584 INFO:tasks.workunit.client.0.smithi067.stdout:4/377: creat d12/d5f/f87 x:0 0 0 2024-04-22T23:23:22.586 INFO:tasks.workunit.client.0.smithi067.stdout:8/272: write d1/d3/f11 [973795,33341] 0 2024-04-22T23:23:22.596 INFO:tasks.workunit.client.0.smithi067.stdout:4/378: mkdir d12/d17/d88 0 2024-04-22T23:23:22.597 INFO:tasks.workunit.client.0.smithi067.stdout:4/379: write d12/d2e/f64 [232659,57337] 0 2024-04-22T23:23:22.600 INFO:tasks.workunit.client.0.smithi067.stdout:8/273: creat d1/dd/d26/f55 x:0 0 0 2024-04-22T23:23:22.600 INFO:tasks.workunit.client.0.smithi067.stdout:0/230: getdents db/d2e/d46 0 2024-04-22T23:23:22.605 INFO:tasks.workunit.client.0.smithi067.stdout:4/380: dwrite ff [0,4194304] 0 2024-04-22T23:23:22.607 INFO:tasks.workunit.client.0.smithi067.stdout:8/274: unlink d1/d3/d10/d3b/c4e 0 2024-04-22T23:23:22.610 INFO:tasks.workunit.client.0.smithi067.stdout:8/275: write d1/d3/d14/f30 [861400,59890] 0 2024-04-22T23:23:22.610 INFO:tasks.workunit.client.0.smithi067.stdout:3/270: sync 2024-04-22T23:23:22.610 INFO:tasks.workunit.client.0.smithi067.stdout:1/267: sync 2024-04-22T23:23:22.611 INFO:tasks.workunit.client.0.smithi067.stdout:5/219: sync 2024-04-22T23:23:22.611 INFO:tasks.workunit.client.0.smithi067.stdout:9/189: sync 2024-04-22T23:23:22.611 INFO:tasks.workunit.client.0.smithi067.stdout:2/247: sync 2024-04-22T23:23:22.611 INFO:tasks.workunit.client.0.smithi067.stdout:7/190: sync 2024-04-22T23:23:22.623 INFO:tasks.workunit.client.0.smithi067.stdout:2/248: write d1/d12/f43 [560315,99165] 0 2024-04-22T23:23:22.623 INFO:tasks.workunit.client.0.smithi067.stdout:1/268: fdatasync d8/d4a/f5a 0 2024-04-22T23:23:22.624 INFO:tasks.workunit.client.0.smithi067.stdout:0/231: dwrite db/d21/d29/f43 [4194304,4194304] 0 2024-04-22T23:23:22.624 INFO:tasks.workunit.client.0.smithi067.stdout:5/220: chown d5/d28/d36/d40/c1f 236 1 2024-04-22T23:23:22.624 INFO:tasks.workunit.client.0.smithi067.stdout:9/190: write d5/dd/f26 [4165516,93297] 0 2024-04-22T23:23:22.627 INFO:tasks.workunit.client.0.smithi067.stdout:3/271: mknod d3/d4/d28/d4c/c56 0 2024-04-22T23:23:22.628 INFO:tasks.workunit.client.0.smithi067.stdout:2/249: chown d1/d10 19184604 1 2024-04-22T23:23:22.629 INFO:tasks.workunit.client.0.smithi067.stdout:7/191: mkdir d10/d14/d1d/d26/d2c/d41 0 2024-04-22T23:23:22.631 INFO:tasks.workunit.client.0.smithi067.stdout:5/221: creat d5/db/d35/f46 x:0 0 0 2024-04-22T23:23:22.633 INFO:tasks.workunit.client.0.smithi067.stdout:9/191: write d5/f36 [165813,123395] 0 2024-04-22T23:23:22.635 INFO:tasks.workunit.client.0.smithi067.stdout:1/269: write d8/d9/d2f/d31/f58 [1352053,29691] 0 2024-04-22T23:23:22.635 INFO:tasks.workunit.client.0.smithi067.stdout:2/250: creat d1/d10/d33/f49 x:0 0 0 2024-04-22T23:23:22.638 INFO:tasks.workunit.client.0.smithi067.stdout:6/195: sync 2024-04-22T23:23:22.639 INFO:tasks.workunit.client.0.smithi067.stdout:4/381: dwrite f4 [0,4194304] 0 2024-04-22T23:23:22.639 INFO:tasks.workunit.client.0.smithi067.stdout:9/192: truncate d5/dd/d18/f1f 979154 0 2024-04-22T23:23:22.640 INFO:tasks.workunit.client.0.smithi067.stdout:1/270: read - d8/f26 zero size 2024-04-22T23:23:22.641 INFO:tasks.workunit.client.0.smithi067.stdout:8/276: dwrite d1/d3/f13 [0,4194304] 0 2024-04-22T23:23:22.642 INFO:tasks.workunit.client.0.smithi067.stdout:3/272: link d3/fa d3/d4/d10/d38/f57 0 2024-04-22T23:23:22.642 INFO:tasks.workunit.client.0.smithi067.stdout:2/251: chown d1/d10/d1c/l34 4859 1 2024-04-22T23:23:22.643 INFO:tasks.workunit.client.0.smithi067.stdout:8/277: readlink d1/dd/l35 0 2024-04-22T23:23:22.647 INFO:tasks.workunit.client.0.smithi067.stdout:0/232: dwrite db/d21/d29/d2c/f41 [8388608,4194304] 0 2024-04-22T23:23:22.651 INFO:tasks.workunit.client.0.smithi067.stdout:7/192: creat d10/d14/d1d/f42 x:0 0 0 2024-04-22T23:23:22.654 INFO:tasks.workunit.client.0.smithi067.stdout:5/222: mkdir d5/db/d12/d44/d47 0 2024-04-22T23:23:22.655 INFO:tasks.workunit.client.0.smithi067.stdout:4/382: write d12/d17/d31/f67 [4873950,55569] 0 2024-04-22T23:23:22.656 INFO:tasks.workunit.client.0.smithi067.stdout:8/278: creat d1/d3/f56 x:0 0 0 2024-04-22T23:23:22.658 INFO:tasks.workunit.client.0.smithi067.stdout:1/271: mkdir d8/d9/d2f/d33/d5c 0 2024-04-22T23:23:22.659 INFO:tasks.workunit.client.0.smithi067.stdout:9/193: creat d5/d21/d2c/f3b x:0 0 0 2024-04-22T23:23:22.659 INFO:tasks.workunit.client.0.smithi067.stdout:3/273: creat d3/d4/d10/d38/d53/f58 x:0 0 0 2024-04-22T23:23:22.661 INFO:tasks.workunit.client.0.smithi067.stdout:6/196: rename d17/l2a to d17/l43 0 2024-04-22T23:23:22.663 INFO:tasks.workunit.client.0.smithi067.stdout:0/233: unlink db/d21/l36 0 2024-04-22T23:23:22.664 INFO:tasks.workunit.client.0.smithi067.stdout:2/252: dwrite d1/d10/f2e [0,4194304] 0 2024-04-22T23:23:22.666 INFO:tasks.workunit.client.0.smithi067.stdout:5/223: creat d5/db/d35/f48 x:0 0 0 2024-04-22T23:23:22.669 INFO:tasks.workunit.client.0.smithi067.stdout:9/194: mkdir d5/dd/d18/d3c 0 2024-04-22T23:23:22.670 INFO:tasks.workunit.client.0.smithi067.stdout:6/197: dread d17/f25 [0,4194304] 0 2024-04-22T23:23:22.670 INFO:tasks.workunit.client.0.smithi067.stdout:2/253: dread d1/d12/f43 [0,4194304] 0 2024-04-22T23:23:22.672 INFO:tasks.workunit.client.0.smithi067.stdout:3/274: rename d3/d4/d10/l14 to d3/d4/d10/d38/l59 0 2024-04-22T23:23:22.679 INFO:tasks.workunit.client.0.smithi067.stdout:6/198: dwrite f13 [4194304,4194304] 0 2024-04-22T23:23:22.686 INFO:tasks.workunit.client.0.smithi067.stdout:6/199: dwrite d17/d39/d3a/f3b [0,4194304] 0 2024-04-22T23:23:22.688 INFO:tasks.workunit.client.0.smithi067.stdout:6/200: truncate d17/f32 964382 0 2024-04-22T23:23:22.690 INFO:tasks.workunit.client.0.smithi067.stdout:1/272: dread d8/d9/f25 [0,4194304] 0 2024-04-22T23:23:22.691 INFO:tasks.workunit.client.0.smithi067.stdout:1/273: fdatasync d8/d24/d34/f37 0 2024-04-22T23:23:22.692 INFO:tasks.workunit.client.0.smithi067.stdout:1/274: truncate d8/d9/d2f/d33/f41 419748 0 2024-04-22T23:23:22.693 INFO:tasks.workunit.client.0.smithi067.stdout:1/275: dread d8/d12/d40/f44 [0,4194304] 0 2024-04-22T23:23:22.694 INFO:tasks.workunit.client.0.smithi067.stdout:1/276: write f1 [4244532,5042] 0 2024-04-22T23:23:22.698 INFO:tasks.workunit.client.0.smithi067.stdout:9/195: creat d5/d2e/f3d x:0 0 0 2024-04-22T23:23:22.703 INFO:tasks.workunit.client.0.smithi067.stdout:9/196: dwrite d5/ff [0,4194304] 0 2024-04-22T23:23:22.704 INFO:tasks.workunit.client.0.smithi067.stdout:9/197: dread - d5/d21/f34 zero size 2024-04-22T23:23:22.705 INFO:tasks.workunit.client.0.smithi067.stdout:3/275: mknod d3/d4/d10/d19/c5a 0 2024-04-22T23:23:22.707 INFO:tasks.workunit.client.0.smithi067.stdout:0/234: mknod db/d21/c4e 0 2024-04-22T23:23:22.712 INFO:tasks.workunit.client.0.smithi067.stdout:4/383: creat d12/d17/d3f/f89 x:0 0 0 2024-04-22T23:23:22.712 INFO:tasks.workunit.client.0.smithi067.stdout:7/193: truncate d10/d18/f1e 364861 0 2024-04-22T23:23:22.714 INFO:tasks.workunit.client.0.smithi067.stdout:7/194: write d10/d14/d1d/f1f [2244564,105644] 0 2024-04-22T23:23:22.715 INFO:tasks.workunit.client.0.smithi067.stdout:7/195: write fb [2906416,28690] 0 2024-04-22T23:23:22.716 INFO:tasks.workunit.client.0.smithi067.stdout:7/196: readlink d10/d14/l16 0 2024-04-22T23:23:22.717 INFO:tasks.workunit.client.0.smithi067.stdout:7/197: write fb [832808,85200] 0 2024-04-22T23:23:22.719 INFO:tasks.workunit.client.0.smithi067.stdout:6/201: readlink d17/l38 0 2024-04-22T23:23:22.725 INFO:tasks.workunit.client.0.smithi067.stdout:6/202: dwrite d17/f20 [0,4194304] 0 2024-04-22T23:23:22.730 INFO:tasks.workunit.client.0.smithi067.stdout:2/254: mkdir d1/d42/d4a 0 2024-04-22T23:23:22.734 INFO:tasks.workunit.client.0.smithi067.stdout:8/279: dwrite d1/d3/f2a [0,4194304] 0 2024-04-22T23:23:22.736 INFO:tasks.workunit.client.1.smithi134.stdout:1/0: creat f0 x:0 0 0 2024-04-22T23:23:22.737 INFO:tasks.workunit.client.1.smithi134.stdout:1/1: chown f0 29576 1 2024-04-22T23:23:22.738 INFO:tasks.workunit.client.1.smithi134.stdout:1/2: chown f0 425517 1 2024-04-22T23:23:22.738 INFO:tasks.workunit.client.1.smithi134.stdout:1/3: stat f0 0 2024-04-22T23:23:22.739 INFO:tasks.workunit.client.1.smithi134.stdout:1/4: write f0 [912140,40062] 0 2024-04-22T23:23:22.740 INFO:tasks.workunit.client.1.smithi134.stdout:7/10: link f0 f1 0 2024-04-22T23:23:22.742 INFO:tasks.workunit.client.1.smithi134.stdout:1/5: fdatasync f0 0 2024-04-22T23:23:22.742 INFO:tasks.workunit.client.1.smithi134.stdout:1/6: rmdir - no directory 2024-04-22T23:23:22.745 INFO:tasks.workunit.client.1.smithi134.stdout:6/13: creat d0/f1 x:0 0 0 2024-04-22T23:23:22.745 INFO:tasks.workunit.client.1.smithi134.stdout:6/14: readlink - no filename 2024-04-22T23:23:22.745 INFO:tasks.workunit.client.0.smithi067.stdout:0/235: dwrite db/de/f4d [0,4194304] 0 2024-04-22T23:23:22.746 INFO:tasks.workunit.client.0.smithi067.stdout:7/198: rename f5 to d10/d28/f43 0 2024-04-22T23:23:22.747 INFO:tasks.workunit.client.1.smithi134.stdout:7/11: symlink l2 0 2024-04-22T23:23:22.747 INFO:tasks.workunit.client.1.smithi134.stdout:6/15: write d0/f1 [596843,75892] 0 2024-04-22T23:23:22.748 INFO:tasks.workunit.client.1.smithi134.stdout:6/16: readlink - no filename 2024-04-22T23:23:22.748 INFO:tasks.workunit.client.1.smithi134.stdout:8/0: creat f0 x:0 0 0 2024-04-22T23:23:22.749 INFO:tasks.workunit.client.1.smithi134.stdout:6/17: chown d0 11 1 2024-04-22T23:23:22.750 INFO:tasks.workunit.client.1.smithi134.stdout:9/0: mkdir d0 0 2024-04-22T23:23:22.750 INFO:tasks.workunit.client.1.smithi134.stdout:9/1: dwrite - no filename 2024-04-22T23:23:22.750 INFO:tasks.workunit.client.1.smithi134.stdout:8/1: dread - f0 zero size 2024-04-22T23:23:22.752 INFO:tasks.workunit.client.0.smithi067.stdout:5/224: link d5/db/c19 d5/d28/d36/d40/c49 0 2024-04-22T23:23:22.752 INFO:tasks.workunit.client.1.smithi134.stdout:7/12: rename f1 to f3 0 2024-04-22T23:23:22.752 INFO:tasks.workunit.client.1.smithi134.stdout:8/2: dread - f0 zero size 2024-04-22T23:23:22.754 INFO:tasks.workunit.client.1.smithi134.stdout:6/18: write d0/f1 [862187,104029] 0 2024-04-22T23:23:22.754 INFO:tasks.workunit.client.1.smithi134.stdout:1/7: dwrite f0 [0,4194304] 0 2024-04-22T23:23:22.754 INFO:tasks.workunit.client.1.smithi134.stdout:1/8: rmdir - no directory 2024-04-22T23:23:22.757 INFO:tasks.workunit.client.0.smithi067.stdout:9/198: mkdir d5/dd/d18/d39/d3e 0 2024-04-22T23:23:22.758 INFO:tasks.workunit.client.1.smithi134.stdout:9/2: creat d0/f1 x:0 0 0 2024-04-22T23:23:22.760 INFO:tasks.workunit.client.0.smithi067.stdout:3/276: creat d3/d4/d28/d4c/d51/f5b x:0 0 0 2024-04-22T23:23:22.763 INFO:tasks.workunit.client.1.smithi134.stdout:9/3: write d0/f1 [305427,35544] 0 2024-04-22T23:23:22.764 INFO:tasks.workunit.client.0.smithi067.stdout:4/384: mknod d12/d5f/d7e/c8a 0 2024-04-22T23:23:22.765 INFO:tasks.workunit.client.1.smithi134.stdout:9/4: rename d0 to d0/d2 22 2024-04-22T23:23:22.767 INFO:tasks.workunit.client.0.smithi067.stdout:0/236: symlink db/de/d2a/d48/l4f 0 2024-04-22T23:23:22.769 INFO:tasks.workunit.client.0.smithi067.stdout:7/199: creat d10/d18/d35/f44 x:0 0 0 2024-04-22T23:23:22.769 INFO:tasks.workunit.client.0.smithi067.stdout:0/237: stat db/de/d11/d1a/c37 0 2024-04-22T23:23:22.769 INFO:tasks.workunit.client.1.smithi134.stdout:9/5: chown d0/f1 1291 1 2024-04-22T23:23:22.769 INFO:tasks.workunit.client.1.smithi134.stdout:8/3: dwrite f0 [0,4194304] 0 2024-04-22T23:23:22.771 INFO:tasks.workunit.client.0.smithi067.stdout:5/225: creat d5/db/d12/f4a x:0 0 0 2024-04-22T23:23:22.772 INFO:tasks.workunit.client.0.smithi067.stdout:5/226: rename d5 to d5/d3a/d4b 22 2024-04-22T23:23:22.772 INFO:tasks.workunit.client.0.smithi067.stdout:4/385: dwrite d12/f2a [0,4194304] 0 2024-04-22T23:23:22.772 INFO:tasks.workunit.client.1.smithi134.stdout:7/13: dwrite f0 [0,4194304] 0 2024-04-22T23:23:22.774 INFO:tasks.workunit.client.1.smithi134.stdout:7/14: write f0 [3696483,5435] 0 2024-04-22T23:23:22.778 INFO:tasks.workunit.client.1.smithi134.stdout:3/5: rmdir d1 0 2024-04-22T23:23:22.778 INFO:tasks.workunit.client.1.smithi134.stdout:3/6: chown . 9423 1 2024-04-22T23:23:22.778 INFO:tasks.workunit.client.1.smithi134.stdout:3/7: write - no filename 2024-04-22T23:23:22.778 INFO:tasks.workunit.client.1.smithi134.stdout:3/8: readlink - no filename 2024-04-22T23:23:22.778 INFO:tasks.workunit.client.1.smithi134.stdout:3/9: dwrite - no filename 2024-04-22T23:23:22.778 INFO:tasks.workunit.client.1.smithi134.stdout:3/10: write - no filename 2024-04-22T23:23:22.779 INFO:tasks.workunit.client.1.smithi134.stdout:9/6: dread d0/f1 [0,4194304] 0 2024-04-22T23:23:22.779 INFO:tasks.workunit.client.1.smithi134.stdout:9/7: chown d0 7817 1 2024-04-22T23:23:22.781 INFO:tasks.workunit.client.1.smithi134.stdout:8/4: mknod c1 0 2024-04-22T23:23:22.782 INFO:tasks.workunit.client.1.smithi134.stdout:8/5: rmdir - no directory 2024-04-22T23:23:22.782 INFO:tasks.workunit.client.0.smithi067.stdout:5/227: dwrite d5/db/d35/f39 [0,4194304] 0 2024-04-22T23:23:22.784 INFO:tasks.workunit.client.1.smithi134.stdout:7/15: mkdir d4 0 2024-04-22T23:23:22.785 INFO:tasks.workunit.client.1.smithi134.stdout:7/16: chown f0 1613962624 1 2024-04-22T23:23:22.786 INFO:tasks.workunit.client.1.smithi134.stdout:3/11: creat f3 x:0 0 0 2024-04-22T23:23:22.787 INFO:tasks.workunit.client.1.smithi134.stdout:3/12: chown f3 353317 1 2024-04-22T23:23:22.788 INFO:tasks.workunit.client.1.smithi134.stdout:9/8: creat d0/f3 x:0 0 0 2024-04-22T23:23:22.791 INFO:tasks.workunit.client.1.smithi134.stdout:8/6: rename c1 to c2 0 2024-04-22T23:23:22.792 INFO:tasks.workunit.client.0.smithi067.stdout:4/386: mkdir d12/d5f/d48/d56/d8b 0 2024-04-22T23:23:22.792 INFO:tasks.workunit.client.0.smithi067.stdout:5/228: dwrite d5/db/d12/f22 [4194304,4194304] 0 2024-04-22T23:23:22.793 INFO:tasks.workunit.client.0.smithi067.stdout:9/199: mkdir d5/dd/d18/d39/d3e/d3f 0 2024-04-22T23:23:22.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:22 smithi067 ceph-mon[26453]: pgmap v266: 65 pgs: 65 active+clean; 487 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 4.8 MiB/s rd, 48 MiB/s wr, 462 op/s 2024-04-22T23:23:22.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:22 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:23:22.795 INFO:tasks.workunit.client.0.smithi067.stdout:9/200: write d5/dd/f1a [528373,7227] 0 2024-04-22T23:23:22.796 INFO:tasks.workunit.client.1.smithi134.stdout:8/7: mknod c3 0 2024-04-22T23:23:22.798 INFO:tasks.workunit.client.1.smithi134.stdout:7/17: creat d4/f5 x:0 0 0 2024-04-22T23:23:22.799 INFO:tasks.workunit.client.1.smithi134.stdout:7/18: read - d4/f5 zero size 2024-04-22T23:23:22.800 INFO:tasks.workunit.client.1.smithi134.stdout:7/19: rename d4 to d4/d6 22 2024-04-22T23:23:22.800 INFO:tasks.workunit.client.1.smithi134.stdout:9/9: link d0/f1 d0/f4 0 2024-04-22T23:23:22.801 INFO:tasks.workunit.client.1.smithi134.stdout:7/20: write d4/f5 [185146,88355] 0 2024-04-22T23:23:22.802 INFO:tasks.workunit.client.1.smithi134.stdout:7/21: chown f0 304691 1 2024-04-22T23:23:22.803 INFO:tasks.workunit.client.0.smithi067.stdout:4/387: unlink d12/d5f/l3b 0 2024-04-22T23:23:22.803 INFO:tasks.workunit.client.1.smithi134.stdout:8/8: unlink f0 0 2024-04-22T23:23:22.803 INFO:tasks.workunit.client.1.smithi134.stdout:8/9: truncate - no filename 2024-04-22T23:23:22.804 INFO:tasks.workunit.client.1.smithi134.stdout:8/10: chown c3 0 1 2024-04-22T23:23:22.804 INFO:tasks.workunit.client.0.smithi067.stdout:5/229: symlink d5/d1a/l4c 0 2024-04-22T23:23:22.805 INFO:tasks.workunit.client.1.smithi134.stdout:9/10: creat d0/f5 x:0 0 0 2024-04-22T23:23:22.807 INFO:tasks.workunit.client.0.smithi067.stdout:0/238: fsync db/d21/d29/d2c/f41 0 2024-04-22T23:23:22.809 INFO:tasks.workunit.client.1.smithi134.stdout:7/22: rename f0 to d4/f7 0 2024-04-22T23:23:22.810 INFO:tasks.workunit.client.0.smithi067.stdout:2/255: getdents d1/d19 0 2024-04-22T23:23:22.811 INFO:tasks.workunit.client.1.smithi134.stdout:8/11: creat f4 x:0 0 0 2024-04-22T23:23:22.811 INFO:tasks.workunit.client.1.smithi134.stdout:8/12: dread - f4 zero size 2024-04-22T23:23:22.812 INFO:tasks.workunit.client.1.smithi134.stdout:9/11: creat d0/f6 x:0 0 0 2024-04-22T23:23:22.813 INFO:tasks.workunit.client.1.smithi134.stdout:9/12: fdatasync d0/f3 0 2024-04-22T23:23:22.813 INFO:tasks.workunit.client.0.smithi067.stdout:5/230: mkdir d5/d3a/d4d 0 2024-04-22T23:23:22.813 INFO:tasks.workunit.client.1.smithi134.stdout:9/13: dread - d0/f5 zero size 2024-04-22T23:23:22.815 INFO:tasks.workunit.client.0.smithi067.stdout:1/277: dwrite d8/d9/fd [0,4194304] 0 2024-04-22T23:23:22.816 INFO:tasks.workunit.client.0.smithi067.stdout:9/201: mkdir d5/dd/d18/d3c/d40 0 2024-04-22T23:23:22.817 INFO:tasks.workunit.client.1.smithi134.stdout:9/14: mkdir d0/d7 0 2024-04-22T23:23:22.817 INFO:tasks.workunit.client.0.smithi067.stdout:0/239: symlink db/de/d11/l50 0 2024-04-22T23:23:22.818 INFO:tasks.workunit.client.0.smithi067.stdout:4/388: mkdir d12/d42/d4b/d77/d8c 0 2024-04-22T23:23:22.819 INFO:tasks.workunit.client.1.smithi134.stdout:9/15: write d0/f3 [11638,19517] 0 2024-04-22T23:23:22.821 INFO:tasks.workunit.client.0.smithi067.stdout:0/240: dread db/de/ff [0,4194304] 0 2024-04-22T23:23:22.826 INFO:tasks.workunit.client.1.smithi134.stdout:7/23: dwrite d4/f7 [0,4194304] 0 2024-04-22T23:23:22.826 INFO:tasks.workunit.client.1.smithi134.stdout:7/24: chown d4/f7 1 1 2024-04-22T23:23:22.826 INFO:tasks.workunit.client.0.smithi067.stdout:1/278: mkdir d8/d24/d34/d5d 0 2024-04-22T23:23:22.826 INFO:tasks.workunit.client.0.smithi067.stdout:0/241: write db/de/d11/d1a/f38 [659980,55163] 0 2024-04-22T23:23:22.827 INFO:tasks.workunit.client.0.smithi067.stdout:7/200: rmdir d10 39 2024-04-22T23:23:22.828 INFO:tasks.workunit.client.0.smithi067.stdout:4/389: dwrite d12/d17/d47/f49 [4194304,4194304] 0 2024-04-22T23:23:22.829 INFO:tasks.workunit.client.1.smithi134.stdout:7/25: mkdir d4/d8 0 2024-04-22T23:23:22.835 INFO:tasks.workunit.client.0.smithi067.stdout:6/203: truncate f13 7609214 0 2024-04-22T23:23:22.836 INFO:tasks.workunit.client.0.smithi067.stdout:9/202: mknod d5/dd/d18/d3c/d40/c41 0 2024-04-22T23:23:22.836 INFO:tasks.workunit.client.0.smithi067.stdout:3/277: write d3/d4/d10/d19/f34 [37760,13260] 0 2024-04-22T23:23:22.838 INFO:tasks.workunit.client.0.smithi067.stdout:1/279: creat d8/d9/d2f/f5e x:0 0 0 2024-04-22T23:23:22.839 INFO:tasks.workunit.client.0.smithi067.stdout:0/242: mknod db/de/d2a/c51 0 2024-04-22T23:23:22.840 INFO:tasks.workunit.client.0.smithi067.stdout:2/256: getdents d1/d19 0 2024-04-22T23:23:22.842 INFO:tasks.workunit.client.0.smithi067.stdout:7/201: truncate d10/d18/f25 1713261 0 2024-04-22T23:23:22.843 INFO:tasks.workunit.client.0.smithi067.stdout:8/280: truncate d1/d3/d27/f28 2408536 0 2024-04-22T23:23:22.847 INFO:tasks.workunit.client.0.smithi067.stdout:1/280: readlink d8/d24/l2b 0 2024-04-22T23:23:22.848 INFO:tasks.workunit.client.0.smithi067.stdout:5/231: rmdir d5/d1a 39 2024-04-22T23:23:22.853 INFO:tasks.workunit.client.0.smithi067.stdout:2/257: creat d1/d42/f4b x:0 0 0 2024-04-22T23:23:22.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.853+0000 7fdf1691a700 1 -- 172.21.15.67:0/3447106534 --> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdf10069520 con 0x7fdef806c4e0 2024-04-22T23:23:22.855 INFO:tasks.workunit.client.0.smithi067.stdout:5/232: dwrite d5/db/d12/f4a [0,4194304] 0 2024-04-22T23:23:22.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.854+0000 7fdf0d7fa700 1 -- 172.21.15.67:0/3447106534 <== mgr.14215 v2:172.21.15.67:6800/2589747850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+321 (secure 0 0 0) 0x7fdf10069520 con 0x7fdef806c4e0 2024-04-22T23:23:22.855 INFO:tasks.workunit.client.0.smithi067.stdout:4/390: link d12/d17/f60 d12/f8d 0 2024-04-22T23:23:22.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.857+0000 7fdf1691a700 1 -- 172.21.15.67:0/3447106534 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fdef806c4e0 msgr2=0x7fdef806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:22.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.857+0000 7fdf1691a700 1 --2- 172.21.15.67:0/3447106534 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fdef806c4e0 0x7fdef806e9a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdf04007be0 tx=0x7fdf040073d0 comp rx=0 tx=0).stop 2024-04-22T23:23:22.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.857+0000 7fdf1691a700 1 -- 172.21.15.67:0/3447106534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf100ffbb0 msgr2=0x7fdf101964c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:22.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.857+0000 7fdf1691a700 1 --2- 172.21.15.67:0/3447106534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf100ffbb0 0x7fdf101964c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fdf0000bda0 tx=0x7fdf0000bdd0 comp rx=0 tx=0).stop 2024-04-22T23:23:22.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.857+0000 7fdf1691a700 1 -- 172.21.15.67:0/3447106534 shutdown_connections 2024-04-22T23:23:22.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.857+0000 7fdf1691a700 1 --2- 172.21.15.67:0/3447106534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf100ffbb0 0x7fdf101964c0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:22.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.857+0000 7fdf1691a700 1 --2- 172.21.15.67:0/3447106534 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fdef806c4e0 0x7fdef806e9a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:22.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.857+0000 7fdf1691a700 1 --2- 172.21.15.67:0/3447106534 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf10100db0 0x7fdf10196a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:22.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.857+0000 7fdf1691a700 1 -- 172.21.15.67:0/3447106534 >> 172.21.15.67:0/3447106534 conn(0x7fdf100fb150 msgr2=0x7fdf10068e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:23:22.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.857+0000 7fdf1691a700 1 -- 172.21.15.67:0/3447106534 shutdown_connections 2024-04-22T23:23:22.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:22.857+0000 7fdf1691a700 1 -- 172.21.15.67:0/3447106534 wait complete. 2024-04-22T23:23:22.863 INFO:tasks.workunit.client.0.smithi067.stdout:3/278: mkdir d3/d4/d10/d38/d53/d55/d5c 0 2024-04-22T23:23:22.865 INFO:tasks.workunit.client.0.smithi067.stdout:1/281: mknod d8/d9/d2d/c5f 0 2024-04-22T23:23:22.865 INFO:tasks.workunit.client.0.smithi067.stdout:3/279: write d3/d4/d28/d40/f45 [405080,3685] 0 2024-04-22T23:23:22.868 INFO:tasks.workunit.client.0.smithi067.stdout:3/280: dread d3/d4/d10/d1f/f3e [0,4194304] 0 2024-04-22T23:23:22.874 INFO:tasks.workunit.client.0.smithi067.stdout:8/281: creat d1/d3/d14/f57 x:0 0 0 2024-04-22T23:23:22.877 INFO:tasks.workunit.client.0.smithi067.stdout:6/204: write d17/f34 [76435,102184] 0 2024-04-22T23:23:22.877 INFO:tasks.workunit.client.0.smithi067.stdout:9/203: link d5/l15 d5/d21/d2c/l42 0 2024-04-22T23:23:22.877 INFO:tasks.workunit.client.0.smithi067.stdout:6/205: chown f6 0 1 2024-04-22T23:23:22.884 INFO:tasks.workunit.client.0.smithi067.stdout:5/233: creat d5/d3a/d4d/f4e x:0 0 0 2024-04-22T23:23:22.885 INFO:tasks.workunit.client.0.smithi067.stdout:1/282: dwrite d8/d9/d2d/d4d/d48/f55 [0,4194304] 0 2024-04-22T23:23:22.885 INFO:tasks.workunit.client.0.smithi067.stdout:7/202: creat d10/f45 x:0 0 0 2024-04-22T23:23:22.890 INFO:tasks.workunit.client.0.smithi067.stdout:9/204: unlink d5/l33 0 2024-04-22T23:23:22.893 INFO:tasks.workunit.client.0.smithi067.stdout:7/203: dwrite fc [0,4194304] 0 2024-04-22T23:23:22.895 INFO:tasks.workunit.client.0.smithi067.stdout:7/204: write d10/d14/d1d/d26/f39 [899610,124134] 0 2024-04-22T23:23:22.896 INFO:tasks.workunit.client.0.smithi067.stdout:3/281: link d3/d4/d10/d38/d53/f58 d3/d4/d10/d19/f5d 0 2024-04-22T23:23:22.901 INFO:tasks.workunit.client.0.smithi067.stdout:1/283: unlink d8/f32 0 2024-04-22T23:23:22.903 INFO:tasks.workunit.client.0.smithi067.stdout:1/284: dread - d8/d9/f29 zero size 2024-04-22T23:23:22.904 INFO:tasks.workunit.client.0.smithi067.stdout:1/285: fdatasync f1 0 2024-04-22T23:23:22.904 INFO:tasks.workunit.client.0.smithi067.stdout:7/205: dwrite fb [0,4194304] 0 2024-04-22T23:23:22.905 INFO:tasks.workunit.client.0.smithi067.stdout:8/282: link d1/f7 d1/d3/d27/f58 0 2024-04-22T23:23:22.906 INFO:tasks.workunit.client.0.smithi067.stdout:0/243: truncate db/d21/d29/f3f 4083309 0 2024-04-22T23:23:22.907 INFO:tasks.workunit.client.0.smithi067.stdout:0/244: stat db/d21/f4c 0 2024-04-22T23:23:22.910 INFO:tasks.workunit.client.1.smithi134.stdout:6/19: rmdir d0 39 2024-04-22T23:23:22.910 INFO:tasks.workunit.client.0.smithi067.stdout:1/286: dread d8/d12/f54 [0,4194304] 0 2024-04-22T23:23:22.913 INFO:tasks.workunit.client.1.smithi134.stdout:6/20: mknod d0/c2 0 2024-04-22T23:23:22.913 INFO:tasks.workunit.client.0.smithi067.stdout:9/205: creat d5/dd/d18/d39/d3e/f43 x:0 0 0 2024-04-22T23:23:22.915 INFO:tasks.workunit.client.1.smithi134.stdout:6/21: creat d0/f3 x:0 0 0 2024-04-22T23:23:22.916 INFO:tasks.workunit.client.0.smithi067.stdout:2/258: getdents d1/d12/d28 0 2024-04-22T23:23:22.917 INFO:tasks.workunit.client.0.smithi067.stdout:5/234: creat d5/db/d12/d44/d47/f4f x:0 0 0 2024-04-22T23:23:22.917 INFO:tasks.workunit.client.1.smithi134.stdout:6/22: symlink d0/l4 0 2024-04-22T23:23:22.918 INFO:tasks.workunit.client.1.smithi134.stdout:6/23: write d0/f1 [1384216,65202] 0 2024-04-22T23:23:22.919 INFO:tasks.workunit.client.1.smithi134.stdout:1/9: truncate f0 2683044 0 2024-04-22T23:23:22.919 INFO:tasks.workunit.client.1.smithi134.stdout:1/10: readlink - no filename 2024-04-22T23:23:22.920 INFO:tasks.workunit.client.0.smithi067.stdout:8/283: chown d1/d3/l51 1451 1 2024-04-22T23:23:22.920 INFO:tasks.workunit.client.1.smithi134.stdout:6/24: creat d0/f5 x:0 0 0 2024-04-22T23:23:22.922 INFO:tasks.workunit.client.0.smithi067.stdout:1/287: dwrite d8/d9/f29 [0,4194304] 0 2024-04-22T23:23:22.923 INFO:tasks.workunit.client.0.smithi067.stdout:6/206: link d17/l22 d17/l44 0 2024-04-22T23:23:22.924 INFO:tasks.workunit.client.0.smithi067.stdout:4/391: write d12/d5f/f27 [270407,43373] 0 2024-04-22T23:23:22.925 INFO:tasks.workunit.client.0.smithi067.stdout:3/282: symlink d3/d4/l5e 0 2024-04-22T23:23:22.926 INFO:tasks.workunit.client.1.smithi134.stdout:9/16: truncate d0/f4 237532 0 2024-04-22T23:23:22.927 INFO:tasks.workunit.client.0.smithi067.stdout:9/206: dread d5/f10 [0,4194304] 0 2024-04-22T23:23:22.928 INFO:tasks.workunit.client.1.smithi134.stdout:6/25: dwrite d0/f1 [0,4194304] 0 2024-04-22T23:23:22.930 INFO:tasks.workunit.client.1.smithi134.stdout:6/26: mkdir d0/d6 0 2024-04-22T23:23:22.931 INFO:tasks.workunit.client.0.smithi067.stdout:5/235: symlink d5/d3a/l50 0 2024-04-22T23:23:22.933 INFO:tasks.workunit.client.0.smithi067.stdout:9/207: chown d5/d21/d2c/f3b 669772842 1 2024-04-22T23:23:22.934 INFO:tasks.workunit.client.1.smithi134.stdout:0/9: write f0 [1341029,122003] 0 2024-04-22T23:23:22.934 INFO:tasks.workunit.client.1.smithi134.stdout:6/27: getdents d0/d6 0 2024-04-22T23:23:22.934 INFO:tasks.workunit.client.1.smithi134.stdout:9/17: fsync d0/f6 0 2024-04-22T23:23:22.934 INFO:tasks.workunit.client.1.smithi134.stdout:0/10: chown f0 13369091 1 2024-04-22T23:23:22.935 INFO:tasks.workunit.client.1.smithi134.stdout:6/28: write d0/f1 [2031635,38362] 0 2024-04-22T23:23:22.935 INFO:tasks.workunit.client.0.smithi067.stdout:6/207: symlink d17/d39/l45 0 2024-04-22T23:23:22.937 INFO:tasks.workunit.client.1.smithi134.stdout:7/26: truncate d4/f7 2428064 0 2024-04-22T23:23:22.938 INFO:tasks.workunit.client.0.smithi067.stdout:6/208: dread - d17/f33 zero size 2024-04-22T23:23:22.938 INFO:tasks.workunit.client.0.smithi067.stdout:3/283: unlink d3/d4/d28/d4c/d51/f5b 0 2024-04-22T23:23:22.939 INFO:tasks.workunit.client.0.smithi067.stdout:3/284: chown d3/d4/d10 1843890956 1 2024-04-22T23:23:22.940 INFO:tasks.workunit.client.0.smithi067.stdout:8/284: dread d1/dd/f20 [0,4194304] 0 2024-04-22T23:23:22.940 INFO:tasks.workunit.client.0.smithi067.stdout:6/209: chown c2 0 1 2024-04-22T23:23:22.940 INFO:tasks.workunit.client.1.smithi134.stdout:7/27: creat d4/f9 x:0 0 0 2024-04-22T23:23:22.942 INFO:tasks.workunit.client.1.smithi134.stdout:6/29: creat d0/d6/f7 x:0 0 0 2024-04-22T23:23:22.942 INFO:tasks.workunit.client.1.smithi134.stdout:1/11: read f0 [929481,61704] 0 2024-04-22T23:23:22.943 INFO:tasks.workunit.client.0.smithi067.stdout:6/210: write d17/d39/d3a/f3b [4061972,66849] 0 2024-04-22T23:23:22.945 INFO:tasks.workunit.client.1.smithi134.stdout:1/12: mknod c1 0 2024-04-22T23:23:22.945 INFO:tasks.workunit.client.1.smithi134.stdout:6/30: creat d0/f8 x:0 0 0 2024-04-22T23:23:22.948 INFO:tasks.workunit.client.1.smithi134.stdout:6/31: creat d0/d6/f9 x:0 0 0 2024-04-22T23:23:22.948 INFO:tasks.workunit.client.0.smithi067.stdout:3/285: mkdir d3/d4/d28/d5f 0 2024-04-22T23:23:22.952 INFO:tasks.workunit.client.1.smithi134.stdout:6/32: creat d0/d6/fa x:0 0 0 2024-04-22T23:23:22.953 INFO:tasks.workunit.client.0.smithi067.stdout:4/392: rename d12/d5f/d48 to d12/d42/d4b/d70/d8e 0 2024-04-22T23:23:22.953 INFO:tasks.workunit.client.1.smithi134.stdout:1/13: link c1 c2 0 2024-04-22T23:23:22.953 INFO:tasks.workunit.client.1.smithi134.stdout:6/33: write d0/f3 [491316,14586] 0 2024-04-22T23:23:22.954 INFO:tasks.workunit.client.1.smithi134.stdout:7/28: write d4/f7 [1514707,63541] 0 2024-04-22T23:23:22.954 INFO:tasks.workunit.client.0.smithi067.stdout:6/211: mkdir d17/d39/d3e/d46 0 2024-04-22T23:23:22.955 INFO:tasks.workunit.client.1.smithi134.stdout:7/29: write d4/f7 [3092010,129962] 0 2024-04-22T23:23:22.955 INFO:tasks.workunit.client.1.smithi134.stdout:1/14: symlink l3 0 2024-04-22T23:23:22.956 INFO:tasks.workunit.client.0.smithi067.stdout:9/208: link d5/dd/d18/d39/d3e/f43 d5/dd/d18/d3c/d40/f44 0 2024-04-22T23:23:22.956 INFO:tasks.workunit.client.1.smithi134.stdout:7/30: fdatasync d4/f9 0 2024-04-22T23:23:22.957 INFO:tasks.workunit.client.1.smithi134.stdout:0/11: truncate f0 1201060 0 2024-04-22T23:23:22.957 INFO:tasks.workunit.client.1.smithi134.stdout:7/31: read f3 [2102489,26188] 0 2024-04-22T23:23:22.957 INFO:tasks.workunit.client.1.smithi134.stdout:6/34: mkdir d0/d6/db 0 2024-04-22T23:23:22.957 INFO:tasks.workunit.client.0.smithi067.stdout:3/286: symlink d3/d4/d10/d19/l60 0 2024-04-22T23:23:22.958 INFO:tasks.workunit.client.1.smithi134.stdout:6/35: truncate d0/f5 538892 0 2024-04-22T23:23:22.963 INFO:tasks.workunit.client.0.smithi067.stdout:7/206: truncate d10/d14/d1d/f1f 4107872 0 2024-04-22T23:23:22.963 INFO:tasks.workunit.client.1.smithi134.stdout:7/32: unlink d4/f5 0 2024-04-22T23:23:22.963 INFO:tasks.workunit.client.0.smithi067.stdout:4/393: symlink d12/d42/d4b/d70/d8e/d56/d8b/l8f 0 2024-04-22T23:23:22.964 INFO:tasks.workunit.client.1.smithi134.stdout:9/18: write d0/f1 [617354,118896] 0 2024-04-22T23:23:22.966 INFO:tasks.workunit.client.1.smithi134.stdout:9/19: chown d0/f3 12984 1 2024-04-22T23:23:22.966 INFO:tasks.workunit.client.1.smithi134.stdout:1/15: dwrite f0 [0,4194304] 0 2024-04-22T23:23:22.966 INFO:tasks.workunit.client.0.smithi067.stdout:3/287: unlink d3/d4/d1b/l26 0 2024-04-22T23:23:22.967 INFO:tasks.workunit.client.0.smithi067.stdout:6/212: mknod d17/d39/d3e/d46/c47 0 2024-04-22T23:23:22.967 INFO:tasks.workunit.client.1.smithi134.stdout:6/36: symlink d0/d6/db/lc 0 2024-04-22T23:23:22.967 INFO:tasks.workunit.client.1.smithi134.stdout:1/16: write f0 [3425459,109336] 0 2024-04-22T23:23:22.968 INFO:tasks.workunit.client.1.smithi134.stdout:6/37: dread - d0/f8 zero size 2024-04-22T23:23:22.969 INFO:tasks.workunit.client.1.smithi134.stdout:6/38: fdatasync d0/f3 0 2024-04-22T23:23:22.969 INFO:tasks.workunit.client.1.smithi134.stdout:7/33: creat d4/fa x:0 0 0 2024-04-22T23:23:22.970 INFO:tasks.workunit.client.1.smithi134.stdout:6/39: write d0/f3 [1497791,36110] 0 2024-04-22T23:23:22.970 INFO:tasks.workunit.client.1.smithi134.stdout:6/40: dread - d0/d6/f9 zero size 2024-04-22T23:23:22.971 INFO:tasks.workunit.client.0.smithi067.stdout:0/245: truncate db/d21/d29/f3f 513505 0 2024-04-22T23:23:22.971 INFO:tasks.workunit.client.1.smithi134.stdout:6/41: read d0/f5 [459365,9350] 0 2024-04-22T23:23:22.972 INFO:tasks.workunit.client.1.smithi134.stdout:1/17: symlink l4 0 2024-04-22T23:23:22.973 INFO:tasks.workunit.client.0.smithi067.stdout:3/288: symlink d3/d4/d10/d1f/d3c/l61 0 2024-04-22T23:23:22.973 INFO:tasks.workunit.client.1.smithi134.stdout:0/12: write f0 [1685847,11291] 0 2024-04-22T23:23:22.973 INFO:tasks.workunit.client.1.smithi134.stdout:0/13: stat d1 0 2024-04-22T23:23:22.975 INFO:tasks.workunit.client.0.smithi067.stdout:6/213: creat d17/f48 x:0 0 0 2024-04-22T23:23:22.977 INFO:tasks.workunit.client.0.smithi067.stdout:6/214: truncate d17/d39/d3a/f40 378489 0 2024-04-22T23:23:22.978 INFO:tasks.workunit.client.1.smithi134.stdout:7/34: creat d4/d8/fb x:0 0 0 2024-04-22T23:23:22.978 INFO:tasks.workunit.client.0.smithi067.stdout:5/236: truncate d5/db/d12/f25 4127719 0 2024-04-22T23:23:22.978 INFO:tasks.workunit.client.1.smithi134.stdout:6/42: dread d0/f3 [0,4194304] 0 2024-04-22T23:23:22.979 INFO:tasks.workunit.client.0.smithi067.stdout:6/215: write d17/f3d [810798,2112] 0 2024-04-22T23:23:22.980 INFO:tasks.workunit.client.1.smithi134.stdout:0/14: rename f0 to d1/f3 0 2024-04-22T23:23:22.980 INFO:tasks.workunit.client.0.smithi067.stdout:7/207: mkdir d10/d14/d1d/d46 0 2024-04-22T23:23:22.981 INFO:tasks.workunit.client.0.smithi067.stdout:5/237: chown d5/d28/d36/d40/c1f 69 1 2024-04-22T23:23:22.981 INFO:tasks.workunit.client.0.smithi067.stdout:3/289: mkdir d3/d4/d28/d4c/d62 0 2024-04-22T23:23:22.982 INFO:tasks.workunit.client.0.smithi067.stdout:7/208: readlink d10/l37 0 2024-04-22T23:23:22.983 INFO:tasks.workunit.client.0.smithi067.stdout:5/238: write d5/db/d12/f4a [2881902,57752] 0 2024-04-22T23:23:22.983 INFO:tasks.workunit.client.0.smithi067.stdout:9/209: link d5/dd/d18/c19 d5/dd/d18/c45 0 2024-04-22T23:23:22.984 INFO:tasks.workunit.client.0.smithi067.stdout:0/246: symlink db/l52 0 2024-04-22T23:23:22.985 INFO:tasks.workunit.client.0.smithi067.stdout:4/394: mknod d12/d17/d3f/c90 0 2024-04-22T23:23:22.985 INFO:tasks.workunit.client.1.smithi134.stdout:6/43: dread d0/f5 [0,4194304] 0 2024-04-22T23:23:22.986 INFO:tasks.workunit.client.0.smithi067.stdout:4/395: write d12/d5f/f55 [209760,59668] 0 2024-04-22T23:23:22.986 INFO:tasks.workunit.client.0.smithi067.stdout:6/216: rename d17/f3d to d17/d39/f49 0 2024-04-22T23:23:22.987 INFO:tasks.workunit.client.1.smithi134.stdout:0/15: dread d1/f3 [0,4194304] 0 2024-04-22T23:23:22.989 INFO:tasks.workunit.client.1.smithi134.stdout:0/16: dread d1/f3 [0,4194304] 0 2024-04-22T23:23:22.989 INFO:tasks.workunit.client.1.smithi134.stdout:0/17: readlink - no filename 2024-04-22T23:23:22.990 INFO:tasks.workunit.client.0.smithi067.stdout:9/210: creat d5/d21/d2c/f46 x:0 0 0 2024-04-22T23:23:22.990 INFO:tasks.workunit.client.1.smithi134.stdout:6/44: creat d0/d6/db/fd x:0 0 0 2024-04-22T23:23:22.992 INFO:tasks.workunit.client.1.smithi134.stdout:0/18: dread d1/f3 [0,4194304] 0 2024-04-22T23:23:22.993 INFO:tasks.workunit.client.0.smithi067.stdout:0/247: unlink db/de/d11/l23 0 2024-04-22T23:23:22.993 INFO:tasks.workunit.client.1.smithi134.stdout:6/45: mkdir d0/d6/de 0 2024-04-22T23:23:22.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:22 smithi134 ceph-mon[34609]: pgmap v266: 65 pgs: 65 active+clean; 487 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 4.8 MiB/s rd, 48 MiB/s wr, 462 op/s 2024-04-22T23:23:22.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:22 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:23:22.995 INFO:tasks.workunit.client.0.smithi067.stdout:5/239: mknod d5/db/d35/c51 0 2024-04-22T23:23:22.996 INFO:tasks.workunit.client.1.smithi134.stdout:6/46: mknod d0/d6/de/cf 0 2024-04-22T23:23:22.996 INFO:tasks.workunit.client.1.smithi134.stdout:6/47: dread - d0/d6/f7 zero size 2024-04-22T23:23:22.996 INFO:tasks.workunit.client.1.smithi134.stdout:7/35: getdents d4/d8 0 2024-04-22T23:23:22.997 INFO:tasks.workunit.client.1.smithi134.stdout:6/48: read - d0/d6/db/fd zero size 2024-04-22T23:23:22.997 INFO:tasks.workunit.client.0.smithi067.stdout:5/240: read d5/db/d12/f37 [1319582,23643] 0 2024-04-22T23:23:22.998 INFO:tasks.workunit.client.1.smithi134.stdout:6/49: readlink d0/d6/db/lc 0 2024-04-22T23:23:22.998 INFO:tasks.workunit.client.1.smithi134.stdout:0/19: rename d1/f3 to d1/d2/f4 0 2024-04-22T23:23:22.998 INFO:tasks.workunit.client.0.smithi067.stdout:5/241: write d5/db/d35/f46 [857270,6084] 0 2024-04-22T23:23:22.998 INFO:tasks.workunit.client.1.smithi134.stdout:0/20: chown d1/d2 14400665 1 2024-04-22T23:23:22.999 INFO:tasks.workunit.client.0.smithi067.stdout:6/217: rename d17/l43 to d17/l4a 0 2024-04-22T23:23:22.999 INFO:tasks.workunit.client.1.smithi134.stdout:7/36: symlink d4/lc 0 2024-04-22T23:23:23.000 INFO:tasks.workunit.client.1.smithi134.stdout:7/37: dread - d4/fa zero size 2024-04-22T23:23:23.001 INFO:tasks.workunit.client.1.smithi134.stdout:6/50: creat d0/f10 x:0 0 0 2024-04-22T23:23:23.001 INFO:tasks.workunit.client.1.smithi134.stdout:7/38: chown d4/f7 1540837067 1 2024-04-22T23:23:23.001 INFO:tasks.workunit.client.0.smithi067.stdout:9/211: mkdir d5/d21/d47 0 2024-04-22T23:23:23.002 INFO:tasks.workunit.client.1.smithi134.stdout:1/18: truncate f0 4148211 0 2024-04-22T23:23:23.002 INFO:tasks.workunit.client.1.smithi134.stdout:7/39: fdatasync d4/d8/fb 0 2024-04-22T23:23:23.002 INFO:tasks.workunit.client.0.smithi067.stdout:9/212: truncate d5/d21/f34 559527 0 2024-04-22T23:23:23.002 INFO:tasks.workunit.client.1.smithi134.stdout:6/51: write d0/d6/f7 [361788,31784] 0 2024-04-22T23:23:23.006 INFO:tasks.workunit.client.1.smithi134.stdout:7/40: rmdir d4/d8 39 2024-04-22T23:23:23.008 INFO:tasks.workunit.client.0.smithi067.stdout:5/242: unlink d5/d28/d36/d40/l3b 0 2024-04-22T23:23:23.008 INFO:tasks.workunit.client.1.smithi134.stdout:7/41: write d4/f9 [158210,114015] 0 2024-04-22T23:23:23.011 INFO:tasks.workunit.client.0.smithi067.stdout:0/248: creat db/d2e/d46/f53 x:0 0 0 2024-04-22T23:23:23.012 INFO:tasks.workunit.client.1.smithi134.stdout:6/52: write d0/f5 [1494717,24981] 0 2024-04-22T23:23:23.013 INFO:tasks.workunit.client.1.smithi134.stdout:0/21: dwrite d1/d2/f4 [0,4194304] 0 2024-04-22T23:23:23.013 INFO:tasks.workunit.client.0.smithi067.stdout:0/249: truncate db/de/d2a/f45 768807 0 2024-04-22T23:23:23.015 INFO:tasks.workunit.client.0.smithi067.stdout:6/218: link d17/f2b d17/d39/f4b 0 2024-04-22T23:23:23.016 INFO:tasks.workunit.client.1.smithi134.stdout:7/42: creat d4/d8/fd x:0 0 0 2024-04-22T23:23:23.016 INFO:tasks.workunit.client.1.smithi134.stdout:1/19: dread f0 [0,4194304] 0 2024-04-22T23:23:23.017 INFO:tasks.workunit.client.1.smithi134.stdout:6/53: creat d0/d6/de/f11 x:0 0 0 2024-04-22T23:23:23.017 INFO:tasks.workunit.client.0.smithi067.stdout:6/219: write d17/f42 [450860,102707] 0 2024-04-22T23:23:23.020 INFO:tasks.workunit.client.1.smithi134.stdout:6/54: mkdir d0/d6/de/d12 0 2024-04-22T23:23:23.022 INFO:tasks.workunit.client.1.smithi134.stdout:0/22: getdents d1 0 2024-04-22T23:23:23.023 INFO:tasks.workunit.client.1.smithi134.stdout:6/55: creat d0/d6/de/f13 x:0 0 0 2024-04-22T23:23:23.024 INFO:tasks.workunit.client.1.smithi134.stdout:0/23: write d1/d2/f4 [3461602,61918] 0 2024-04-22T23:23:23.026 INFO:tasks.workunit.client.0.smithi067.stdout:0/250: dwrite db/de/d11/d1a/f38 [0,4194304] 0 2024-04-22T23:23:23.030 INFO:tasks.workunit.client.0.smithi067.stdout:6/220: dwrite d17/d39/d3a/f40 [0,4194304] 0 2024-04-22T23:23:23.032 INFO:tasks.workunit.client.0.smithi067.stdout:6/221: dread - d17/f30 zero size 2024-04-22T23:23:23.032 INFO:tasks.workunit.client.0.smithi067.stdout:5/243: creat d5/f52 x:0 0 0 2024-04-22T23:23:23.034 INFO:tasks.workunit.client.0.smithi067.stdout:0/251: truncate db/fd 411638 0 2024-04-22T23:23:23.034 INFO:tasks.workunit.client.0.smithi067.stdout:3/290: write d3/f35 [1269873,57143] 0 2024-04-22T23:23:23.036 INFO:tasks.workunit.client.0.smithi067.stdout:3/291: stat d3/d4/d28/d4c/d62 0 2024-04-22T23:23:23.036 INFO:tasks.workunit.client.0.smithi067.stdout:3/292: readlink d3/lf 0 2024-04-22T23:23:23.037 INFO:tasks.workunit.client.0.smithi067.stdout:0/252: write db/de/f1b [5103538,127137] 0 2024-04-22T23:23:23.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.039+0000 7fbd2369a700 1 -- 172.21.15.67:0/1191844282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd1c102e60 msgr2=0x7fbd1c103280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:23.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.039+0000 7fbd2369a700 1 --2- 172.21.15.67:0/1191844282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd1c102e60 0x7fbd1c103280 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fbd1400b0a0 tx=0x7fbd1400b3b0 comp rx=0 tx=0).stop 2024-04-22T23:23:23.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.039+0000 7fbd2369a700 1 -- 172.21.15.67:0/1191844282 shutdown_connections 2024-04-22T23:23:23.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.039+0000 7fbd2369a700 1 --2- 172.21.15.67:0/1191844282 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd1c104050 0x7fbd1c1044d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:23.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.039+0000 7fbd2369a700 1 --2- 172.21.15.67:0/1191844282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd1c102e60 0x7fbd1c103280 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:23.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.039+0000 7fbd2369a700 1 -- 172.21.15.67:0/1191844282 >> 172.21.15.67:0/1191844282 conn(0x7fbd1c0fe470 msgr2=0x7fbd1c100890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:23:23.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.039+0000 7fbd2369a700 1 -- 172.21.15.67:0/1191844282 shutdown_connections 2024-04-22T23:23:23.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.040+0000 7fbd2369a700 1 -- 172.21.15.67:0/1191844282 wait complete. 2024-04-22T23:23:23.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.040+0000 7fbd2369a700 1 Processor -- start 2024-04-22T23:23:23.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.040+0000 7fbd2369a700 1 -- start start 2024-04-22T23:23:23.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.041+0000 7fbd2369a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd1c102e60 0x7fbd1c078b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:23:23.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.041+0000 7fbd2369a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd1c104050 0x7fbd1c079080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:23:23.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.041+0000 7fbd2369a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd1c075630 con 0x7fbd1c102e60 2024-04-22T23:23:23.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.041+0000 7fbd2369a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd1c0757a0 con 0x7fbd1c104050 2024-04-22T23:23:23.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.041+0000 7fbd21436700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd1c102e60 0x7fbd1c078b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:23:23.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.041+0000 7fbd21436700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd1c102e60 0x7fbd1c078b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40746/0 (socket says 172.21.15.67:40746) 2024-04-22T23:23:23.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.041+0000 7fbd21436700 1 -- 172.21.15.67:0/4045964108 learned_addr learned my addr 172.21.15.67:0/4045964108 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:23:23.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.041+0000 7fbd21436700 1 -- 172.21.15.67:0/4045964108 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd1c104050 msgr2=0x7fbd1c079080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:23.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.041+0000 7fbd21436700 1 --2- 172.21.15.67:0/4045964108 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd1c104050 0x7fbd1c079080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:23.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.041+0000 7fbd21436700 1 -- 172.21.15.67:0/4045964108 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd14009d00 con 0x7fbd1c102e60 2024-04-22T23:23:23.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.042+0000 7fbd21436700 1 --2- 172.21.15.67:0/4045964108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd1c102e60 0x7fbd1c078b40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fbd1400b070 tx=0x7fbd14003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:23:23.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.042+0000 7fbd127fc700 1 -- 172.21.15.67:0/4045964108 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd1400e040 con 0x7fbd1c102e60 2024-04-22T23:23:23.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.042+0000 7fbd127fc700 1 -- 172.21.15.67:0/4045964108 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fbd14007e90 con 0x7fbd1c102e60 2024-04-22T23:23:23.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.042+0000 7fbd2369a700 1 -- 172.21.15.67:0/4045964108 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd1c075a20 con 0x7fbd1c102e60 2024-04-22T23:23:23.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.042+0000 7fbd2369a700 1 -- 172.21.15.67:0/4045964108 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd1c075f70 con 0x7fbd1c102e60 2024-04-22T23:23:23.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.043+0000 7fbd127fc700 1 -- 172.21.15.67:0/4045964108 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd14012b40 con 0x7fbd1c102e60 2024-04-22T23:23:23.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.044+0000 7fbd2369a700 1 -- 172.21.15.67:0/4045964108 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd00005320 con 0x7fbd1c102e60 2024-04-22T23:23:23.045 INFO:tasks.workunit.client.0.smithi067.stdout:5/244: rename d5/d1a/f2e to d5/db/d12/d44/f53 0 2024-04-22T23:23:23.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.045+0000 7fbd127fc700 1 -- 172.21.15.67:0/4045964108 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7fbd14019070 con 0x7fbd1c102e60 2024-04-22T23:23:23.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.046+0000 7fbd127fc700 1 --2- 172.21.15.67:0/4045964108 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fbd0806c650 0x7fbd0806eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:23:23.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.046+0000 7fbd127fc700 1 -- 172.21.15.67:0/4045964108 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fbd1408d450 con 0x7fbd1c102e60 2024-04-22T23:23:23.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.047+0000 7fbd20c35700 1 --2- 172.21.15.67:0/4045964108 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fbd0806c650 0x7fbd0806eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:23:23.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.047+0000 7fbd20c35700 1 --2- 172.21.15.67:0/4045964108 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fbd0806c650 0x7fbd0806eb10 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c102b90 tx=0x7fbd0c006cd0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:23:23.048 INFO:tasks.workunit.client.0.smithi067.stdout:3/293: getdents d3/d4/d28/d4c/d62 0 2024-04-22T23:23:23.050 INFO:tasks.workunit.client.0.smithi067.stdout:0/253: dwrite db/d21/d29/d2d/f42 [0,4194304] 0 2024-04-22T23:23:23.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.053+0000 7fbd127fc700 1 -- 172.21.15.67:0/4045964108 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fbd1405b6c0 con 0x7fbd1c102e60 2024-04-22T23:23:23.055 INFO:tasks.workunit.client.0.smithi067.stdout:6/222: getdents d17 0 2024-04-22T23:23:23.057 INFO:tasks.workunit.client.0.smithi067.stdout:0/254: mknod db/d21/d29/d2d/c54 0 2024-04-22T23:23:23.058 INFO:tasks.workunit.client.0.smithi067.stdout:6/223: write d17/f42 [1481633,64706] 0 2024-04-22T23:23:23.060 INFO:tasks.workunit.client.0.smithi067.stdout:5/245: dwrite d5/db/ff [4194304,4194304] 0 2024-04-22T23:23:23.061 INFO:tasks.workunit.client.0.smithi067.stdout:3/294: getdents d3/d4/d10/d38/d53 0 2024-04-22T23:23:23.061 INFO:tasks.workunit.client.0.smithi067.stdout:6/224: dread - d17/d39/f4b zero size 2024-04-22T23:23:23.061 INFO:tasks.workunit.client.0.smithi067.stdout:0/255: chown db/de/d2a/d48/l4f 91775132 1 2024-04-22T23:23:23.062 INFO:tasks.workunit.client.0.smithi067.stdout:9/213: dwrite d5/fb [0,4194304] 0 2024-04-22T23:23:23.065 INFO:tasks.workunit.client.0.smithi067.stdout:5/246: symlink d5/d28/l54 0 2024-04-22T23:23:23.066 INFO:tasks.workunit.client.0.smithi067.stdout:3/295: write d3/d4/f20 [2916260,43796] 0 2024-04-22T23:23:23.067 INFO:tasks.workunit.client.0.smithi067.stdout:9/214: truncate d5/d21/f27 999875 0 2024-04-22T23:23:23.068 INFO:tasks.workunit.client.0.smithi067.stdout:5/247: fdatasync d5/d3e/f42 0 2024-04-22T23:23:23.071 INFO:tasks.workunit.client.0.smithi067.stdout:5/248: write d5/d28/d36/d40/f16 [5367113,10476] 0 2024-04-22T23:23:23.072 INFO:tasks.workunit.client.0.smithi067.stdout:9/215: mknod d5/c48 0 2024-04-22T23:23:23.076 INFO:tasks.workunit.client.0.smithi067.stdout:3/296: rename d3/d4/f20 to d3/d4/d12/d4b/f63 0 2024-04-22T23:23:23.078 INFO:tasks.workunit.client.0.smithi067.stdout:5/249: creat d5/d3e/f55 x:0 0 0 2024-04-22T23:23:23.078 INFO:tasks.workunit.client.0.smithi067.stdout:0/256: link db/de/d2a/d48/l4f db/de/l55 0 2024-04-22T23:23:23.079 INFO:tasks.workunit.client.0.smithi067.stdout:5/250: truncate d5/db/f34 710821 0 2024-04-22T23:23:23.081 INFO:tasks.workunit.client.0.smithi067.stdout:9/216: mkdir d5/dd/d18/d39/d49 0 2024-04-22T23:23:23.082 INFO:tasks.workunit.client.0.smithi067.stdout:6/225: getdents d17/d39/d3e 0 2024-04-22T23:23:23.083 INFO:tasks.workunit.client.0.smithi067.stdout:3/297: fsync d3/d4/d10/f39 0 2024-04-22T23:23:23.084 INFO:tasks.workunit.client.0.smithi067.stdout:6/226: dread d17/f24 [0,4194304] 0 2024-04-22T23:23:23.085 INFO:tasks.workunit.client.0.smithi067.stdout:5/251: creat d5/db/d12/d44/f56 x:0 0 0 2024-04-22T23:23:23.090 INFO:tasks.workunit.client.0.smithi067.stdout:6/227: creat d17/d39/f4c x:0 0 0 2024-04-22T23:23:23.091 INFO:tasks.workunit.client.0.smithi067.stdout:9/217: mknod d5/dd/d18/d39/d49/c4a 0 2024-04-22T23:23:23.093 INFO:tasks.workunit.client.0.smithi067.stdout:5/252: dwrite d5/d3e/f42 [0,4194304] 0 2024-04-22T23:23:23.095 INFO:tasks.workunit.client.0.smithi067.stdout:9/218: mknod d5/dd/d18/d39/d49/c4b 0 2024-04-22T23:23:23.097 INFO:tasks.workunit.client.0.smithi067.stdout:3/298: rename d3/d4/d10/c23 to d3/d4/d10/c64 0 2024-04-22T23:23:23.098 INFO:tasks.workunit.client.0.smithi067.stdout:3/299: fsync d3/d4/d10/d38/f48 0 2024-04-22T23:23:23.100 INFO:tasks.workunit.client.0.smithi067.stdout:3/300: dread - d3/d4/d10/d38/d53/f58 zero size 2024-04-22T23:23:23.106 INFO:tasks.workunit.client.0.smithi067.stdout:5/253: getdents d5/db/d12/d44/d47 0 2024-04-22T23:23:23.106 INFO:tasks.workunit.client.0.smithi067.stdout:5/254: chown d5/db/d12 49990529 1 2024-04-22T23:23:23.108 INFO:tasks.workunit.client.0.smithi067.stdout:9/219: getdents d5/dd/d18/d3c/d40 0 2024-04-22T23:23:23.109 INFO:tasks.workunit.client.0.smithi067.stdout:9/220: write d5/d2e/f3d [78045,67319] 0 2024-04-22T23:23:23.120 INFO:tasks.workunit.client.0.smithi067.stdout:9/221: dread d5/dd/d18/f1d [0,4194304] 0 2024-04-22T23:23:23.122 INFO:tasks.workunit.client.0.smithi067.stdout:9/222: stat d5/l17 0 2024-04-22T23:23:23.220 INFO:tasks.workunit.client.1.smithi134.stdout:1/20: dread f0 [0,4194304] 0 2024-04-22T23:23:23.221 INFO:tasks.workunit.client.1.smithi134.stdout:7/43: dwrite d4/f9 [0,4194304] 0 2024-04-22T23:23:23.222 INFO:tasks.workunit.client.1.smithi134.stdout:6/56: mknod d0/c14 0 2024-04-22T23:23:23.230 INFO:tasks.workunit.client.1.smithi134.stdout:0/24: mkdir d1/d5 0 2024-04-22T23:23:23.230 INFO:tasks.workunit.client.1.smithi134.stdout:0/25: chown d1/d2 1965885 1 2024-04-22T23:23:23.232 INFO:tasks.workunit.client.1.smithi134.stdout:7/44: creat d4/d8/fe x:0 0 0 2024-04-22T23:23:23.235 INFO:tasks.workunit.client.1.smithi134.stdout:0/26: mkdir d1/d2/d6 0 2024-04-22T23:23:23.235 INFO:tasks.workunit.client.1.smithi134.stdout:6/57: mknod d0/d6/de/d12/c15 0 2024-04-22T23:23:23.262 INFO:tasks.workunit.client.0.smithi067.stdout:9/223: dread d5/dd/f1a [0,4194304] 0 2024-04-22T23:23:23.263 INFO:tasks.workunit.client.0.smithi067.stdout:9/224: write d5/dd/f32 [779290,119913] 0 2024-04-22T23:23:23.268 INFO:tasks.workunit.client.0.smithi067.stdout:9/225: dwrite d5/d21/d2c/f3b [0,4194304] 0 2024-04-22T23:23:23.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.285+0000 7fbd2369a700 1 -- 172.21.15.67:0/4045964108 --> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fbd00000bf0 con 0x7fbd0806c650 2024-04-22T23:23:23.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.295+0000 7fbd127fc700 1 -- 172.21.15.67:0/4045964108 <== mgr.14215 v2:172.21.15.67:6800/2589747850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3528 (secure 0 0 0) 0x7fbd00000bf0 con 0x7fbd0806c650 2024-04-22T23:23:23.297 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:23:23.297 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (6m) 2m ago 8m 26.7M - 0.25.0 c8568f914cd2 8c7b7ec00657 2024-04-22T23:23:23.297 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (8m) 2m ago 8m 7490k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:23:23.297 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (7m) 2m ago 7m 7453k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:23:23.297 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (8m) 2m ago 8m 6660k - 18.2.1 5be31c24972a 14085634c5d3 2024-04-22T23:23:23.297 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (7m) 2m ago 7m 6665k - 18.2.1 5be31c24972a fa995d458353 2024-04-22T23:23:23.297 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (6m) 2m ago 8m 85.0M - 9.4.7 954c08fa6188 0d8950c9af0c 2024-04-22T23:23:23.297 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (2m) 2m ago 2m 12.8M - 18.2.1 5be31c24972a e24a471f0a8c 2024-04-22T23:23:23.297 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (2m) 2m ago 2m 16.7M - 18.2.1 5be31c24972a 0730f9fc8e61 2024-04-22T23:23:23.297 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (2m) 2m ago 2m 17.6M - 18.2.1 5be31c24972a b6a169586433 2024-04-22T23:23:23.298 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (2m) 2m ago 2m 14.6M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:23:23.298 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:9283,8765,8443 running (9m) 2m ago 9m 490M - 18.2.1 5be31c24972a 1c64cb9e36bb 2024-04-22T23:23:23.298 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 starting - - - - 2024-04-22T23:23:23.298 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (10m) 2m ago 10m 46.1M 2048M 18.2.1 5be31c24972a 6789283703e0 2024-04-22T23:23:23.298 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (7m) 2m ago 7m 35.9M 2048M 18.2.1 5be31c24972a 2fb23795f7e3 2024-04-22T23:23:23.298 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (8m) 2m ago 8m 16.3M - 1.5.0 0da6a335fe13 4aa4da7089d0 2024-04-22T23:23:23.298 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (7m) 2m ago 7m 15.6M - 1.5.0 0da6a335fe13 26e43c2e756c 2024-04-22T23:23:23.298 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (6m) 2m ago 6m 48.5M 3946M 18.2.1 5be31c24972a a033247ee8c2 2024-04-22T23:23:23.298 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (6m) 2m ago 6m 47.1M 3946M 18.2.1 5be31c24972a 915b07a8edc4 2024-04-22T23:23:23.298 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (5m) 2m ago 5m 46.8M 3946M 18.2.1 5be31c24972a e257ecb4efa1 2024-04-22T23:23:23.298 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (5m) 2m ago 5m 45.9M 4970M 18.2.1 5be31c24972a 9c2c91a967ba 2024-04-22T23:23:23.298 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (4m) 2m ago 4m 44.0M 4970M 18.2.1 5be31c24972a c7586562ffa6 2024-04-22T23:23:23.298 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (4m) 2m ago 4m 44.0M 4970M 18.2.1 5be31c24972a 9fd97f1c65cb 2024-04-22T23:23:23.298 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (6m) 2m ago 7m 55.5M - 2.43.0 a07b618ecd1d d87cb15b5e55 2024-04-22T23:23:23.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.297+0000 7fbd2369a700 1 -- 172.21.15.67:0/4045964108 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fbd0806c650 msgr2=0x7fbd0806eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:23.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.297+0000 7fbd2369a700 1 --2- 172.21.15.67:0/4045964108 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fbd0806c650 0x7fbd0806eb10 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c102b90 tx=0x7fbd0c006cd0 comp rx=0 tx=0).stop 2024-04-22T23:23:23.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.297+0000 7fbd2369a700 1 -- 172.21.15.67:0/4045964108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd1c102e60 msgr2=0x7fbd1c078b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:23.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.297+0000 7fbd2369a700 1 --2- 172.21.15.67:0/4045964108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd1c102e60 0x7fbd1c078b40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fbd1400b070 tx=0x7fbd14003ce0 comp rx=0 tx=0).stop 2024-04-22T23:23:23.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.297+0000 7fbd2369a700 1 -- 172.21.15.67:0/4045964108 shutdown_connections 2024-04-22T23:23:23.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.297+0000 7fbd2369a700 1 --2- 172.21.15.67:0/4045964108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd1c102e60 0x7fbd1c078b40 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:23.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.297+0000 7fbd2369a700 1 --2- 172.21.15.67:0/4045964108 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fbd0806c650 0x7fbd0806eb10 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:23.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.297+0000 7fbd2369a700 1 --2- 172.21.15.67:0/4045964108 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd1c104050 0x7fbd1c079080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:23.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.297+0000 7fbd2369a700 1 -- 172.21.15.67:0/4045964108 >> 172.21.15.67:0/4045964108 conn(0x7fbd1c0fe470 msgr2=0x7fbd1c107310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:23:23.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.298+0000 7fbd2369a700 1 -- 172.21.15.67:0/4045964108 shutdown_connections 2024-04-22T23:23:23.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.298+0000 7fbd2369a700 1 -- 172.21.15.67:0/4045964108 wait complete. 2024-04-22T23:23:23.310 INFO:tasks.workunit.client.1.smithi134.stdout:7/45: link d4/f7 d4/d8/ff 0 2024-04-22T23:23:23.312 INFO:tasks.workunit.client.1.smithi134.stdout:6/58: creat d0/f16 x:0 0 0 2024-04-22T23:23:23.312 INFO:tasks.workunit.client.1.smithi134.stdout:1/21: dwrite f0 [4194304,4194304] 0 2024-04-22T23:23:23.313 INFO:tasks.workunit.client.0.smithi067.stdout:2/259: sync 2024-04-22T23:23:23.314 INFO:tasks.workunit.client.1.smithi134.stdout:0/27: rmdir d1/d2/d6 0 2024-04-22T23:23:23.315 INFO:tasks.workunit.client.0.smithi067.stdout:2/260: rmdir d1/d19/d31 39 2024-04-22T23:23:23.315 INFO:tasks.workunit.client.1.smithi134.stdout:7/46: mkdir d4/d10 0 2024-04-22T23:23:23.316 INFO:tasks.workunit.client.0.smithi067.stdout:2/261: rmdir d1/d12/d28 39 2024-04-22T23:23:23.317 INFO:tasks.workunit.client.1.smithi134.stdout:6/59: mknod d0/d6/db/c17 0 2024-04-22T23:23:23.317 INFO:tasks.workunit.client.0.smithi067.stdout:2/262: chown d1/d19/f2f 27 1 2024-04-22T23:23:23.318 INFO:tasks.workunit.client.1.smithi134.stdout:6/60: dread - d0/f10 zero size 2024-04-22T23:23:23.318 INFO:tasks.workunit.client.0.smithi067.stdout:2/263: write d1/d10/d33/d46/f47 [450636,5545] 0 2024-04-22T23:23:23.320 INFO:tasks.workunit.client.1.smithi134.stdout:0/28: creat d1/d5/f7 x:0 0 0 2024-04-22T23:23:23.320 INFO:tasks.workunit.client.0.smithi067.stdout:2/264: creat d1/d42/d4a/f4c x:0 0 0 2024-04-22T23:23:23.320 INFO:tasks.workunit.client.1.smithi134.stdout:0/29: chown d1/d2 410186 1 2024-04-22T23:23:23.321 INFO:tasks.workunit.client.1.smithi134.stdout:1/22: dwrite f0 [0,4194304] 0 2024-04-22T23:23:23.321 INFO:tasks.workunit.client.1.smithi134.stdout:7/47: creat d4/d10/f11 x:0 0 0 2024-04-22T23:23:23.322 INFO:tasks.workunit.client.0.smithi067.stdout:2/265: rmdir d1/d10/d1c 39 2024-04-22T23:23:23.322 INFO:tasks.workunit.client.0.smithi067.stdout:2/266: truncate d1/d12/f48 690617 0 2024-04-22T23:23:23.323 INFO:tasks.workunit.client.1.smithi134.stdout:0/30: write d1/d5/f7 [429863,124824] 0 2024-04-22T23:23:23.323 INFO:tasks.workunit.client.1.smithi134.stdout:6/61: creat d0/d6/de/f18 x:0 0 0 2024-04-22T23:23:23.324 INFO:tasks.workunit.client.0.smithi067.stdout:2/267: mkdir d1/d10/d1c/d4d 0 2024-04-22T23:23:23.326 INFO:tasks.workunit.client.1.smithi134.stdout:7/48: symlink d4/d10/l12 0 2024-04-22T23:23:23.328 INFO:tasks.workunit.client.1.smithi134.stdout:1/23: dread f0 [0,4194304] 0 2024-04-22T23:23:23.329 INFO:tasks.workunit.client.1.smithi134.stdout:0/31: symlink d1/l8 0 2024-04-22T23:23:23.331 INFO:tasks.workunit.client.1.smithi134.stdout:1/24: symlink l5 0 2024-04-22T23:23:23.334 INFO:tasks.workunit.client.1.smithi134.stdout:1/25: mknod c6 0 2024-04-22T23:23:23.336 INFO:tasks.workunit.client.1.smithi134.stdout:7/49: rename d4/d8/fe to d4/f13 0 2024-04-22T23:23:23.337 INFO:tasks.workunit.client.1.smithi134.stdout:1/26: creat f7 x:0 0 0 2024-04-22T23:23:23.338 INFO:tasks.workunit.client.1.smithi134.stdout:7/50: symlink d4/d8/l14 0 2024-04-22T23:23:23.339 INFO:tasks.workunit.client.1.smithi134.stdout:7/51: fsync d4/d8/fb 0 2024-04-22T23:23:23.339 INFO:tasks.workunit.client.1.smithi134.stdout:1/27: symlink l8 0 2024-04-22T23:23:23.340 INFO:tasks.workunit.client.1.smithi134.stdout:7/52: dread - d4/d8/fd zero size 2024-04-22T23:23:23.341 INFO:tasks.workunit.client.1.smithi134.stdout:6/62: truncate d0/f1 1536155 0 2024-04-22T23:23:23.342 INFO:tasks.workunit.client.1.smithi134.stdout:7/53: mknod d4/c15 0 2024-04-22T23:23:23.343 INFO:tasks.workunit.client.1.smithi134.stdout:6/63: write d0/d6/de/f13 [794608,15178] 0 2024-04-22T23:23:23.343 INFO:tasks.workunit.client.1.smithi134.stdout:7/54: fdatasync d4/fa 0 2024-04-22T23:23:23.344 INFO:tasks.workunit.client.1.smithi134.stdout:7/55: read f3 [2443609,87392] 0 2024-04-22T23:23:23.345 INFO:tasks.workunit.client.1.smithi134.stdout:7/56: read d4/f9 [593847,113166] 0 2024-04-22T23:23:23.347 INFO:tasks.workunit.client.1.smithi134.stdout:6/64: rename d0/d6 to d0/d19 0 2024-04-22T23:23:23.348 INFO:tasks.workunit.client.1.smithi134.stdout:6/65: write d0/d19/fa [414606,83724] 0 2024-04-22T23:23:23.348 INFO:tasks.workunit.client.1.smithi134.stdout:6/66: dread - d0/f8 zero size 2024-04-22T23:23:23.351 INFO:tasks.workunit.client.1.smithi134.stdout:6/67: creat d0/f1a x:0 0 0 2024-04-22T23:23:23.352 INFO:tasks.workunit.client.1.smithi134.stdout:6/68: chown d0/d19/de/cf 34 1 2024-04-22T23:23:23.352 INFO:tasks.workunit.client.1.smithi134.stdout:7/57: mknod d4/c16 0 2024-04-22T23:23:23.353 INFO:tasks.workunit.client.1.smithi134.stdout:7/58: truncate d4/f13 302081 0 2024-04-22T23:23:23.356 INFO:tasks.workunit.client.1.smithi134.stdout:6/69: creat d0/d19/f1b x:0 0 0 2024-04-22T23:23:23.361 INFO:tasks.workunit.client.1.smithi134.stdout:7/59: dwrite d4/d10/f11 [0,4194304] 0 2024-04-22T23:23:23.362 INFO:tasks.workunit.client.1.smithi134.stdout:7/60: fdatasync d4/f13 0 2024-04-22T23:23:23.364 INFO:tasks.workunit.client.1.smithi134.stdout:6/70: stat d0/f1 0 2024-04-22T23:23:23.365 INFO:tasks.workunit.client.1.smithi134.stdout:6/71: dread - d0/d19/f1b zero size 2024-04-22T23:23:23.366 INFO:tasks.workunit.client.1.smithi134.stdout:6/72: truncate d0/f10 26584 0 2024-04-22T23:23:23.368 INFO:tasks.workunit.client.1.smithi134.stdout:6/73: truncate d0/d19/de/f13 1412491 0 2024-04-22T23:23:23.370 INFO:tasks.workunit.client.1.smithi134.stdout:7/61: dwrite d4/f7 [0,4194304] 0 2024-04-22T23:23:23.372 INFO:tasks.workunit.client.1.smithi134.stdout:6/74: rename d0/f5 to d0/d19/f1c 0 2024-04-22T23:23:23.373 INFO:tasks.workunit.client.1.smithi134.stdout:7/62: mkdir d4/d10/d17 0 2024-04-22T23:23:23.374 INFO:tasks.workunit.client.1.smithi134.stdout:7/63: chown d4/d10/f11 584694311 1 2024-04-22T23:23:23.374 INFO:tasks.workunit.client.1.smithi134.stdout:7/64: readlink d4/d8/l14 0 2024-04-22T23:23:23.375 INFO:tasks.workunit.client.1.smithi134.stdout:6/75: dread d0/d19/fa [0,4194304] 0 2024-04-22T23:23:23.376 INFO:tasks.workunit.client.1.smithi134.stdout:6/76: write d0/f8 [238378,54247] 0 2024-04-22T23:23:23.378 INFO:tasks.workunit.client.1.smithi134.stdout:6/77: creat d0/d19/de/f1d x:0 0 0 2024-04-22T23:23:23.380 INFO:tasks.workunit.client.1.smithi134.stdout:6/78: creat d0/f1e x:0 0 0 2024-04-22T23:23:23.380 INFO:tasks.workunit.client.0.smithi067.stdout:7/209: sync 2024-04-22T23:23:23.381 INFO:tasks.workunit.client.0.smithi067.stdout:1/288: sync 2024-04-22T23:23:23.381 INFO:tasks.workunit.client.0.smithi067.stdout:4/396: sync 2024-04-22T23:23:23.381 INFO:tasks.workunit.client.0.smithi067.stdout:8/285: sync 2024-04-22T23:23:23.381 INFO:tasks.workunit.client.0.smithi067.stdout:0/257: sync 2024-04-22T23:23:23.381 INFO:tasks.workunit.client.0.smithi067.stdout:5/255: sync 2024-04-22T23:23:23.382 INFO:tasks.workunit.client.1.smithi134.stdout:7/65: truncate f3 2961149 0 2024-04-22T23:23:23.384 INFO:tasks.workunit.client.0.smithi067.stdout:5/256: write d5/db/d12/f4a [2720304,77643] 0 2024-04-22T23:23:23.384 INFO:tasks.workunit.client.1.smithi134.stdout:7/66: symlink d4/d10/d17/l18 0 2024-04-22T23:23:23.386 INFO:tasks.workunit.client.0.smithi067.stdout:1/289: mkdir d8/d24/d34/d60 0 2024-04-22T23:23:23.388 INFO:tasks.workunit.client.0.smithi067.stdout:7/210: mkdir d10/d47 0 2024-04-22T23:23:23.389 INFO:tasks.workunit.client.0.smithi067.stdout:7/211: fsync d10/f45 0 2024-04-22T23:23:23.390 INFO:tasks.workunit.client.0.smithi067.stdout:8/286: creat d1/d3/f59 x:0 0 0 2024-04-22T23:23:23.391 INFO:tasks.workunit.client.0.smithi067.stdout:4/397: dread d12/d5f/f5a [0,4194304] 0 2024-04-22T23:23:23.391 INFO:tasks.workunit.client.0.smithi067.stdout:0/258: dwrite db/d21/d29/d2d/f49 [0,4194304] 0 2024-04-22T23:23:23.391 INFO:tasks.workunit.client.0.smithi067.stdout:5/257: mkdir d5/d28/d36/d40/d57 0 2024-04-22T23:23:23.391 INFO:tasks.workunit.client.1.smithi134.stdout:7/67: dwrite d4/f13 [0,4194304] 0 2024-04-22T23:23:23.392 INFO:tasks.workunit.client.0.smithi067.stdout:1/290: rmdir d8/d24 39 2024-04-22T23:23:23.397 INFO:tasks.workunit.client.0.smithi067.stdout:0/259: mkdir db/d2e/d31/d56 0 2024-04-22T23:23:23.397 INFO:tasks.workunit.client.0.smithi067.stdout:7/212: symlink d10/d28/l48 0 2024-04-22T23:23:23.398 INFO:tasks.workunit.client.0.smithi067.stdout:8/287: write d1/d3/f1e [222616,8205] 0 2024-04-22T23:23:23.398 INFO:tasks.workunit.client.0.smithi067.stdout:5/258: creat d5/d3a/f58 x:0 0 0 2024-04-22T23:23:23.399 INFO:tasks.workunit.client.1.smithi134.stdout:7/68: dwrite d4/f7 [0,4194304] 0 2024-04-22T23:23:23.400 INFO:tasks.workunit.client.0.smithi067.stdout:5/259: chown d5/d28/d36 1008 1 2024-04-22T23:23:23.402 INFO:tasks.workunit.client.0.smithi067.stdout:5/260: write d5/fa [306558,89807] 0 2024-04-22T23:23:23.403 INFO:tasks.workunit.client.0.smithi067.stdout:1/291: creat d8/d9/f61 x:0 0 0 2024-04-22T23:23:23.403 INFO:tasks.workunit.client.1.smithi134.stdout:7/69: getdents d4 0 2024-04-22T23:23:23.404 INFO:tasks.workunit.client.0.smithi067.stdout:0/260: fsync db/d21/d29/f43 0 2024-04-22T23:23:23.404 INFO:tasks.workunit.client.1.smithi134.stdout:7/70: rmdir d4/d10/d17 39 2024-04-22T23:23:23.405 INFO:tasks.workunit.client.0.smithi067.stdout:8/288: rename d1/d3/l3f to d1/d41/l5a 0 2024-04-22T23:23:23.406 INFO:tasks.workunit.client.0.smithi067.stdout:7/213: mkdir d10/d47/d49 0 2024-04-22T23:23:23.407 INFO:tasks.workunit.client.1.smithi134.stdout:7/71: creat d4/d10/f19 x:0 0 0 2024-04-22T23:23:23.409 INFO:tasks.workunit.client.1.smithi134.stdout:7/72: rename d4/d10/d17/l18 to d4/l1a 0 2024-04-22T23:23:23.410 INFO:tasks.workunit.client.0.smithi067.stdout:4/398: dwrite d12/d5f/d2b/f72 [0,4194304] 0 2024-04-22T23:23:23.412 INFO:tasks.workunit.client.0.smithi067.stdout:8/289: chown d1/d3/d14/f32 49727 1 2024-04-22T23:23:23.413 INFO:tasks.workunit.client.0.smithi067.stdout:7/214: symlink d10/d28/l4a 0 2024-04-22T23:23:23.415 INFO:tasks.workunit.client.0.smithi067.stdout:7/215: truncate d10/d14/d1d/d26/f40 627450 0 2024-04-22T23:23:23.416 INFO:tasks.workunit.client.0.smithi067.stdout:4/399: creat d12/d42/d4b/f91 x:0 0 0 2024-04-22T23:23:23.419 INFO:tasks.workunit.client.1.smithi134.stdout:7/73: link d4/f9 d4/d10/f1b 0 2024-04-22T23:23:23.419 INFO:tasks.workunit.client.0.smithi067.stdout:8/290: rename d1/c4c to d1/c5b 0 2024-04-22T23:23:23.421 INFO:tasks.workunit.client.0.smithi067.stdout:4/400: mknod d12/d17/d47/c92 0 2024-04-22T23:23:23.422 INFO:tasks.workunit.client.0.smithi067.stdout:7/216: symlink d10/d14/d1d/d26/d2c/l4b 0 2024-04-22T23:23:23.422 INFO:tasks.workunit.client.0.smithi067.stdout:4/401: write d12/d5f/f87 [763467,94964] 0 2024-04-22T23:23:23.422 INFO:tasks.workunit.client.0.smithi067.stdout:5/261: link d5/db/l24 d5/d3a/d4d/l59 0 2024-04-22T23:23:23.424 INFO:tasks.workunit.client.0.smithi067.stdout:0/261: getdents db/de/d2a 0 2024-04-22T23:23:23.425 INFO:tasks.workunit.client.0.smithi067.stdout:5/262: readlink d5/db/l24 0 2024-04-22T23:23:23.425 INFO:tasks.workunit.client.0.smithi067.stdout:0/262: chown db/d2e/d31/d56 48 1 2024-04-22T23:23:23.426 INFO:tasks.workunit.client.0.smithi067.stdout:5/263: readlink d5/d28/l54 0 2024-04-22T23:23:23.427 INFO:tasks.workunit.client.0.smithi067.stdout:7/217: rename d10/l37 to d10/d47/l4c 0 2024-04-22T23:23:23.428 INFO:tasks.workunit.client.0.smithi067.stdout:7/218: dread - d10/d18/d35/f44 zero size 2024-04-22T23:23:23.429 INFO:tasks.workunit.client.0.smithi067.stdout:5/264: mknod d5/d1a/c5a 0 2024-04-22T23:23:23.430 INFO:tasks.workunit.client.0.smithi067.stdout:5/265: stat d5/db/d35 0 2024-04-22T23:23:23.433 INFO:tasks.workunit.client.0.smithi067.stdout:0/263: dwrite db/de/f12 [4194304,4194304] 0 2024-04-22T23:23:23.435 INFO:tasks.workunit.client.0.smithi067.stdout:5/266: unlink d5/d3a/f3f 0 2024-04-22T23:23:23.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.434+0000 7ffbe920e700 1 -- 172.21.15.67:0/1029430886 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffbe4100e10 msgr2=0x7ffbe4101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:23.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.434+0000 7ffbe920e700 1 --2- 172.21.15.67:0/1029430886 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffbe4100e10 0x7ffbe4101270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8009870 tx=0x7ffbd8009b80 comp rx=0 tx=0).stop 2024-04-22T23:23:23.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.434+0000 7ffbe920e700 1 -- 172.21.15.67:0/1029430886 shutdown_connections 2024-04-22T23:23:23.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.434+0000 7ffbe920e700 1 --2- 172.21.15.67:0/1029430886 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffbe4100e10 0x7ffbe4101270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:23.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.434+0000 7ffbe920e700 1 --2- 172.21.15.67:0/1029430886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbe40ffc10 0x7ffbe4100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:23.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.434+0000 7ffbe920e700 1 -- 172.21.15.67:0/1029430886 >> 172.21.15.67:0/1029430886 conn(0x7ffbe40fb190 msgr2=0x7ffbe40fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:23:23.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.434+0000 7ffbe920e700 1 -- 172.21.15.67:0/1029430886 shutdown_connections 2024-04-22T23:23:23.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.434+0000 7ffbe920e700 1 -- 172.21.15.67:0/1029430886 wait complete. 2024-04-22T23:23:23.437 INFO:tasks.workunit.client.0.smithi067.stdout:5/267: chown d5/f52 7 1 2024-04-22T23:23:23.437 INFO:tasks.workunit.client.0.smithi067.stdout:7/219: link d10/d18/c38 d10/d18/d35/c4d 0 2024-04-22T23:23:23.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.435+0000 7ffbe920e700 1 Processor -- start 2024-04-22T23:23:23.437 INFO:tasks.workunit.client.0.smithi067.stdout:5/268: fdatasync d5/d3a/f58 0 2024-04-22T23:23:23.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.436+0000 7ffbe920e700 1 -- start start 2024-04-22T23:23:23.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.436+0000 7ffbe920e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbe40ffc10 0x7ffbe4198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:23:23.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.436+0000 7ffbe920e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffbe4100e10 0x7ffbe4198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:23:23.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.436+0000 7ffbe920e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe41991d0 con 0x7ffbe40ffc10 2024-04-22T23:23:23.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.436+0000 7ffbe920e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe4199310 con 0x7ffbe4100e10 2024-04-22T23:23:23.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.436+0000 7ffbe259c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffbe4100e10 0x7ffbe4198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:23:23.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.436+0000 7ffbe259c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffbe4100e10 0x7ffbe4198bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.67:41188/0 (socket says 172.21.15.67:41188) 2024-04-22T23:23:23.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.436+0000 7ffbe259c700 1 -- 172.21.15.67:0/1010776181 learned_addr learned my addr 172.21.15.67:0/1010776181 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:23:23.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.436+0000 7ffbe2d9d700 1 --2- 172.21.15.67:0/1010776181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbe40ffc10 0x7ffbe4198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:23:23.442 INFO:tasks.workunit.client.0.smithi067.stdout:5/269: chown d5/db/d12/f4a 28702323 1 2024-04-22T23:23:23.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.437+0000 7ffbe2d9d700 1 -- 172.21.15.67:0/1010776181 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffbe4100e10 msgr2=0x7ffbe4198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:23.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.437+0000 7ffbe2d9d700 1 --2- 172.21.15.67:0/1010776181 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffbe4100e10 0x7ffbe4198bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:23.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.437+0000 7ffbe2d9d700 1 -- 172.21.15.67:0/1010776181 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd40095f0 con 0x7ffbe40ffc10 2024-04-22T23:23:23.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.437+0000 7ffbe259c700 1 --2- 172.21.15.67:0/1010776181 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffbe4100e10 0x7ffbe4198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T23:23:23.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.437+0000 7ffbe2d9d700 1 --2- 172.21.15.67:0/1010776181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbe40ffc10 0x7ffbe4198670 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ffbd4005fd0 tx=0x7ffbd400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:23:23.443 INFO:tasks.workunit.client.0.smithi067.stdout:5/270: write d5/d3e/f55 [776278,113000] 0 2024-04-22T23:23:23.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.438+0000 7ffbcbfff700 1 -- 172.21.15.67:0/1010776181 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffbd400cc60 con 0x7ffbe40ffc10 2024-04-22T23:23:23.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.438+0000 7ffbcbfff700 1 -- 172.21.15.67:0/1010776181 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ffbd400cdc0 con 0x7ffbe40ffc10 2024-04-22T23:23:23.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.438+0000 7ffbe920e700 1 -- 172.21.15.67:0/1010776181 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbd8009520 con 0x7ffbe40ffc10 2024-04-22T23:23:23.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.438+0000 7ffbe920e700 1 -- 172.21.15.67:0/1010776181 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbe419e180 con 0x7ffbe40ffc10 2024-04-22T23:23:23.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.439+0000 7ffbcbfff700 1 -- 172.21.15.67:0/1010776181 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffbd400cc60 con 0x7ffbe40ffc10 2024-04-22T23:23:23.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.440+0000 7ffbe920e700 1 -- 172.21.15.67:0/1010776181 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbe4108700 con 0x7ffbe40ffc10 2024-04-22T23:23:23.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.441+0000 7ffbcbfff700 1 -- 172.21.15.67:0/1010776181 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 19) v1 ==== 90312+0+0 (secure 0 0 0) 0x7ffbd4004750 con 0x7ffbe40ffc10 2024-04-22T23:23:23.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.441+0000 7ffbcbfff700 1 --2- 172.21.15.67:0/1010776181 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ffbcc06c5a0 0x7ffbcc06ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:23:23.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.442+0000 7ffbcbfff700 1 -- 172.21.15.67:0/1010776181 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7ffbd4014070 con 0x7ffbe40ffc10 2024-04-22T23:23:23.445 INFO:tasks.workunit.client.0.smithi067.stdout:5/271: unlink d5/db/d35/c51 0 2024-04-22T23:23:23.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.442+0000 7ffbe259c700 1 --2- 172.21.15.67:0/1010776181 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ffbcc06c5a0 0x7ffbcc06ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:23:23.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.442+0000 7ffbe259c700 1 --2- 172.21.15.67:0/1010776181 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ffbcc06c5a0 0x7ffbcc06ea60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ffbd800b6f0 tx=0x7ffbd8019040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:23:23.446 INFO:tasks.workunit.client.0.smithi067.stdout:5/272: getdents d5/d28/d36 0 2024-04-22T23:23:23.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.448+0000 7ffbcbfff700 1 -- 172.21.15.67:0/1010776181 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ffbd4059c10 con 0x7ffbe40ffc10 2024-04-22T23:23:23.451 INFO:tasks.workunit.client.0.smithi067.stdout:5/273: dwrite d5/db/ff [4194304,4194304] 0 2024-04-22T23:23:23.453 INFO:tasks.workunit.client.0.smithi067.stdout:5/274: rmdir d5/d1a 39 2024-04-22T23:23:23.459 INFO:tasks.workunit.client.0.smithi067.stdout:5/275: dwrite d5/db/d35/f48 [0,4194304] 0 2024-04-22T23:23:23.461 INFO:tasks.workunit.client.0.smithi067.stdout:5/276: write d5/db/d35/f43 [4783,122202] 0 2024-04-22T23:23:23.540 INFO:tasks.workunit.client.0.smithi067.stdout:2/268: dread d1/f35 [0,4194304] 0 2024-04-22T23:23:23.544 INFO:tasks.workunit.client.0.smithi067.stdout:2/269: creat d1/d12/f4e x:0 0 0 2024-04-22T23:23:23.547 INFO:tasks.workunit.client.0.smithi067.stdout:4/402: sync 2024-04-22T23:23:23.550 INFO:tasks.workunit.client.0.smithi067.stdout:2/270: rename d1/d12/d28 to d1/d4f 0 2024-04-22T23:23:23.553 INFO:tasks.workunit.client.0.smithi067.stdout:4/403: dread d12/d5f/f5a [0,4194304] 0 2024-04-22T23:23:23.556 INFO:tasks.workunit.client.0.smithi067.stdout:4/404: mkdir d12/d42/d4b/d70/d8e/d56/d93 0 2024-04-22T23:23:23.562 INFO:tasks.workunit.client.0.smithi067.stdout:4/405: symlink d12/d75/l94 0 2024-04-22T23:23:23.564 INFO:tasks.workunit.client.0.smithi067.stdout:5/277: unlink d5/db/d12/d44/f53 0 2024-04-22T23:23:23.568 INFO:tasks.workunit.client.0.smithi067.stdout:5/278: truncate d5/d28/d36/d40/f20 1602943 0 2024-04-22T23:23:23.571 INFO:tasks.workunit.client.0.smithi067.stdout:5/279: creat d5/f5b x:0 0 0 2024-04-22T23:23:23.577 INFO:tasks.workunit.client.0.smithi067.stdout:5/280: dwrite d5/db/f2f [0,4194304] 0 2024-04-22T23:23:23.585 INFO:tasks.workunit.client.0.smithi067.stdout:5/281: dwrite d5/d3e/f55 [0,4194304] 0 2024-04-22T23:23:23.589 INFO:tasks.workunit.client.1.smithi134.stdout:0/32: dread d1/d5/f7 [0,4194304] 0 2024-04-22T23:23:23.595 INFO:tasks.workunit.client.0.smithi067.stdout:0/264: sync 2024-04-22T23:23:23.596 INFO:tasks.workunit.client.0.smithi067.stdout:7/220: sync 2024-04-22T23:23:23.597 INFO:tasks.workunit.client.0.smithi067.stdout:7/221: rmdir d10/d14/d1d 39 2024-04-22T23:23:23.599 INFO:tasks.workunit.client.0.smithi067.stdout:7/222: read d10/f12 [673331,113458] 0 2024-04-22T23:23:23.600 INFO:tasks.workunit.client.0.smithi067.stdout:0/265: symlink db/d21/d29/d2d/l57 0 2024-04-22T23:23:23.604 INFO:tasks.workunit.client.0.smithi067.stdout:7/223: creat d10/d18/d35/f4e x:0 0 0 2024-04-22T23:23:23.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.608+0000 7ffbcbfff700 1 -- 172.21.15.67:0/1010776181 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 20) v1 ==== 91455+0+0 (secure 0 0 0) 0x7ffbd4056550 con 0x7ffbe40ffc10 2024-04-22T23:23:23.615 INFO:tasks.workunit.client.0.smithi067.stdout:7/224: creat d10/d14/d1d/d26/d2c/d41/f4f x:0 0 0 2024-04-22T23:23:23.621 INFO:tasks.workunit.client.0.smithi067.stdout:6/228: truncate d17/d39/f49 783840 0 2024-04-22T23:23:23.622 INFO:tasks.workunit.client.0.smithi067.stdout:3/301: truncate d3/f35 2255952 0 2024-04-22T23:23:23.623 INFO:tasks.workunit.client.0.smithi067.stdout:6/229: rmdir d17/d39/d3e 39 2024-04-22T23:23:23.625 INFO:tasks.workunit.client.0.smithi067.stdout:3/302: symlink d3/d4/d1b/l65 0 2024-04-22T23:23:23.626 INFO:tasks.workunit.client.0.smithi067.stdout:6/230: fsync d17/f1e 0 2024-04-22T23:23:23.627 INFO:tasks.workunit.client.0.smithi067.stdout:6/231: fsync d17/f31 0 2024-04-22T23:23:23.627 INFO:tasks.workunit.client.0.smithi067.stdout:9/226: truncate d5/dd/d18/f1f 508639 0 2024-04-22T23:23:23.629 INFO:tasks.workunit.client.0.smithi067.stdout:3/303: creat d3/d4/d10/d38/d53/d55/f66 x:0 0 0 2024-04-22T23:23:23.632 INFO:tasks.workunit.client.0.smithi067.stdout:6/232: creat d17/d39/d3a/f4d x:0 0 0 2024-04-22T23:23:23.632 INFO:tasks.workunit.client.0.smithi067.stdout:6/233: read - d17/f48 zero size 2024-04-22T23:23:23.634 INFO:tasks.workunit.client.0.smithi067.stdout:9/227: write d5/dd/d18/f1d [3733156,115060] 0 2024-04-22T23:23:23.635 INFO:tasks.workunit.client.0.smithi067.stdout:3/304: creat d3/d4/d10/d1f/d3c/f67 x:0 0 0 2024-04-22T23:23:23.639 INFO:tasks.workunit.client.0.smithi067.stdout:9/228: write d5/fe [131363,105845] 0 2024-04-22T23:23:23.641 INFO:tasks.workunit.client.0.smithi067.stdout:3/305: symlink d3/d4/d28/d4c/d51/l68 0 2024-04-22T23:23:23.643 INFO:tasks.workunit.client.0.smithi067.stdout:9/229: dread - d5/dd/d18/d39/d3e/f43 zero size 2024-04-22T23:23:23.644 INFO:tasks.workunit.client.0.smithi067.stdout:9/230: dread - d5/fa zero size 2024-04-22T23:23:23.647 INFO:tasks.workunit.client.0.smithi067.stdout:9/231: chown d5/dd/d18/c45 1535 1 2024-04-22T23:23:23.648 INFO:tasks.workunit.client.0.smithi067.stdout:6/234: dread d17/f34 [0,4194304] 0 2024-04-22T23:23:23.649 INFO:tasks.workunit.client.0.smithi067.stdout:9/232: chown d5/dd/d18/l31 5647781 1 2024-04-22T23:23:23.650 INFO:tasks.workunit.client.0.smithi067.stdout:9/233: chown d5/dd/d18/d39/d49/c4a 84941 1 2024-04-22T23:23:23.650 INFO:tasks.workunit.client.0.smithi067.stdout:3/306: link d3/d4/d10/d19/f34 d3/d4/d12/f69 0 2024-04-22T23:23:23.652 INFO:tasks.workunit.client.0.smithi067.stdout:9/234: write d5/d21/f27 [1766611,88736] 0 2024-04-22T23:23:23.655 INFO:tasks.workunit.client.0.smithi067.stdout:6/235: symlink d17/d39/d3e/l4e 0 2024-04-22T23:23:23.656 INFO:tasks.workunit.client.0.smithi067.stdout:6/236: write d17/f30 [678844,57678] 0 2024-04-22T23:23:23.657 INFO:tasks.workunit.client.0.smithi067.stdout:3/307: mknod d3/d4/d10/d38/d53/d55/d5c/c6a 0 2024-04-22T23:23:23.658 INFO:tasks.workunit.client.0.smithi067.stdout:7/225: rename d10/d28 to d10/d14/d1d/d46/d50 0 2024-04-22T23:23:23.660 INFO:tasks.workunit.client.0.smithi067.stdout:6/237: unlink d17/f31 0 2024-04-22T23:23:23.661 INFO:tasks.workunit.client.0.smithi067.stdout:6/238: chown d17/d39/f4c 27800 1 2024-04-22T23:23:23.663 INFO:tasks.workunit.client.0.smithi067.stdout:6/239: write d17/f33 [945273,47014] 0 2024-04-22T23:23:23.663 INFO:tasks.workunit.client.0.smithi067.stdout:1/292: truncate d8/d9/f29 2328523 0 2024-04-22T23:23:23.664 INFO:tasks.workunit.client.0.smithi067.stdout:3/308: truncate d3/d4/d10/d19/f5d 428565 0 2024-04-22T23:23:23.665 INFO:tasks.workunit.client.0.smithi067.stdout:7/226: dread d10/d18/f25 [0,4194304] 0 2024-04-22T23:23:23.665 INFO:tasks.workunit.client.0.smithi067.stdout:5/282: rmdir d5/d3a/d4d 39 2024-04-22T23:23:23.670 INFO:tasks.workunit.client.0.smithi067.stdout:9/235: rename d5/dd/d18/d3c/d40/c41 to d5/d21/d2c/c4c 0 2024-04-22T23:23:23.673 INFO:tasks.workunit.client.0.smithi067.stdout:6/240: creat d17/d39/d3a/f4f x:0 0 0 2024-04-22T23:23:23.674 INFO:tasks.workunit.client.0.smithi067.stdout:6/241: chown d17/f34 4092 1 2024-04-22T23:23:23.676 INFO:tasks.workunit.client.0.smithi067.stdout:5/283: symlink d5/db/l5c 0 2024-04-22T23:23:23.676 INFO:tasks.workunit.client.0.smithi067.stdout:9/236: dread d5/d21/f27 [0,4194304] 0 2024-04-22T23:23:23.677 INFO:tasks.workunit.client.0.smithi067.stdout:8/291: truncate d1/d3/f2a 408100 0 2024-04-22T23:23:23.678 INFO:tasks.workunit.client.0.smithi067.stdout:5/284: fsync d5/d28/d36/d40/f16 0 2024-04-22T23:23:23.679 INFO:tasks.workunit.client.0.smithi067.stdout:9/237: write d5/ff [2559571,74529] 0 2024-04-22T23:23:23.679 INFO:tasks.workunit.client.0.smithi067.stdout:8/292: readlink d1/dd/d26/l2c 0 2024-04-22T23:23:23.681 INFO:tasks.workunit.client.0.smithi067.stdout:9/238: chown d5/dd/d18/d39/d3e 45443405 1 2024-04-22T23:23:23.683 INFO:tasks.workunit.client.0.smithi067.stdout:3/309: symlink d3/d4/l6b 0 2024-04-22T23:23:23.684 INFO:tasks.workunit.client.0.smithi067.stdout:7/227: dread d10/d14/d1d/d26/d2c/f3f [0,4194304] 0 2024-04-22T23:23:23.688 INFO:tasks.workunit.client.0.smithi067.stdout:1/293: mkdir d8/d24/d34/d60/d62 0 2024-04-22T23:23:23.691 INFO:tasks.workunit.client.0.smithi067.stdout:3/310: creat d3/d4/d12/d4b/f6c x:0 0 0 2024-04-22T23:23:23.695 INFO:tasks.workunit.client.0.smithi067.stdout:8/293: mkdir d1/d3/d5c 0 2024-04-22T23:23:23.697 INFO:tasks.workunit.client.0.smithi067.stdout:4/406: write d12/f16 [2492404,72081] 0 2024-04-22T23:23:23.698 INFO:tasks.workunit.client.0.smithi067.stdout:6/242: link d17/f24 d17/d39/d3a/f50 0 2024-04-22T23:23:23.699 INFO:tasks.workunit.client.0.smithi067.stdout:2/271: dwrite d1/d12/f16 [0,4194304] 0 2024-04-22T23:23:23.700 INFO:tasks.workunit.client.0.smithi067.stdout:9/239: dwrite d5/dd/f1a [4194304,4194304] 0 2024-04-22T23:23:23.702 INFO:tasks.workunit.client.0.smithi067.stdout:7/228: symlink d10/d47/d49/l51 0 2024-04-22T23:23:23.703 INFO:tasks.workunit.client.0.smithi067.stdout:7/229: fsync fc 0 2024-04-22T23:23:23.706 INFO:tasks.workunit.client.0.smithi067.stdout:3/311: symlink d3/d4/d28/d4c/d51/l6d 0 2024-04-22T23:23:23.709 INFO:tasks.workunit.client.0.smithi067.stdout:3/312: read - d3/d4/d12/f1d zero size 2024-04-22T23:23:23.711 INFO:tasks.workunit.client.0.smithi067.stdout:2/272: chown d1/d10/f2e 239685805 1 2024-04-22T23:23:23.711 INFO:tasks.workunit.client.0.smithi067.stdout:1/294: symlink d8/d9/d2f/d59/l63 0 2024-04-22T23:23:23.713 INFO:tasks.workunit.client.0.smithi067.stdout:6/243: creat d17/d39/d3a/f51 x:0 0 0 2024-04-22T23:23:23.714 INFO:tasks.workunit.client.0.smithi067.stdout:2/273: chown d1/d12/f16 8033 1 2024-04-22T23:23:23.715 INFO:tasks.workunit.client.0.smithi067.stdout:9/240: mkdir d5/dd/d18/d39/d4d 0 2024-04-22T23:23:23.715 INFO:tasks.workunit.client.0.smithi067.stdout:4/407: write d12/d2e/f64 [989872,60796] 0 2024-04-22T23:23:23.717 INFO:tasks.workunit.client.0.smithi067.stdout:7/230: dwrite d10/d18/d35/f4e [0,4194304] 0 2024-04-22T23:23:23.719 INFO:tasks.workunit.client.0.smithi067.stdout:2/274: write d1/d12/f48 [119971,54402] 0 2024-04-22T23:23:23.721 INFO:tasks.workunit.client.0.smithi067.stdout:8/294: symlink d1/dd/l5d 0 2024-04-22T23:23:23.726 INFO:tasks.workunit.client.0.smithi067.stdout:4/408: creat d12/d75/f95 x:0 0 0 2024-04-22T23:23:23.728 INFO:tasks.workunit.client.0.smithi067.stdout:9/241: creat d5/d2e/f4e x:0 0 0 2024-04-22T23:23:23.728 INFO:tasks.workunit.client.0.smithi067.stdout:0/266: write db/d21/d29/f3f [578630,27168] 0 2024-04-22T23:23:23.733 INFO:tasks.workunit.client.0.smithi067.stdout:7/231: read d10/d14/d1d/d46/d50/f43 [1356406,41537] 0 2024-04-22T23:23:23.734 INFO:tasks.workunit.client.0.smithi067.stdout:7/232: readlink d10/d14/d1d/d46/d50/l4a 0 2024-04-22T23:23:23.737 INFO:tasks.workunit.client.0.smithi067.stdout:2/275: unlink d1/d10/d1c/f23 0 2024-04-22T23:23:23.741 INFO:tasks.workunit.client.0.smithi067.stdout:4/409: readlink d12/l1c 0 2024-04-22T23:23:23.745 INFO:tasks.workunit.client.0.smithi067.stdout:0/267: rename db/de/l10 to db/de/d2a/l58 0 2024-04-22T23:23:23.745 INFO:tasks.workunit.client.0.smithi067.stdout:2/276: rename d1 to d1/d10/d33/d46/d50 22 2024-04-22T23:23:23.746 INFO:tasks.workunit.client.0.smithi067.stdout:2/277: stat d1/d10/d33/f49 0 2024-04-22T23:23:23.749 INFO:tasks.workunit.client.0.smithi067.stdout:8/295: write d1/d3/f1b [4279172,57393] 0 2024-04-22T23:23:23.751 INFO:tasks.workunit.client.0.smithi067.stdout:4/410: creat d12/d42/d4b/d70/d8e/d56/f96 x:0 0 0 2024-04-22T23:23:23.756 INFO:tasks.workunit.client.0.smithi067.stdout:2/278: symlink d1/d10/d33/l51 0 2024-04-22T23:23:23.757 INFO:tasks.workunit.client.0.smithi067.stdout:6/244: getdents d17/d39 0 2024-04-22T23:23:23.761 INFO:tasks.workunit.client.0.smithi067.stdout:4/411: mkdir d12/d5f/d2b/d97 0 2024-04-22T23:23:23.762 INFO:tasks.workunit.client.0.smithi067.stdout:9/242: link d5/dd/l13 d5/d21/d47/l4f 0 2024-04-22T23:23:23.767 INFO:tasks.workunit.client.1.smithi134.stdout:0/33: truncate d1/d5/f7 422204 0 2024-04-22T23:23:23.767 INFO:tasks.workunit.client.0.smithi067.stdout:8/296: creat d1/d3/d10/d3b/f5e x:0 0 0 2024-04-22T23:23:23.768 INFO:tasks.workunit.client.1.smithi134.stdout:0/34: creat d1/f9 x:0 0 0 2024-04-22T23:23:23.769 INFO:tasks.workunit.client.0.smithi067.stdout:4/412: dwrite d12/d17/d47/f86 [0,4194304] 0 2024-04-22T23:23:23.772 INFO:tasks.workunit.client.0.smithi067.stdout:9/243: creat d5/dd/d18/d3c/d40/f50 x:0 0 0 2024-04-22T23:23:23.774 INFO:tasks.workunit.client.1.smithi134.stdout:0/35: dwrite d1/d2/f4 [0,4194304] 0 2024-04-22T23:23:23.776 INFO:tasks.workunit.client.0.smithi067.stdout:6/245: dread d17/f1e [0,4194304] 0 2024-04-22T23:23:23.779 INFO:tasks.workunit.client.0.smithi067.stdout:0/268: link db/c16 db/de/d2a/c59 0 2024-04-22T23:23:23.779 INFO:tasks.workunit.client.0.smithi067.stdout:4/413: dread ff [0,4194304] 0 2024-04-22T23:23:23.779 INFO:tasks.workunit.client.0.smithi067.stdout:6/246: chown d17/d39/d3a/f3b 89 1 2024-04-22T23:23:23.781 INFO:tasks.workunit.client.0.smithi067.stdout:2/279: creat d1/d10/f52 x:0 0 0 2024-04-22T23:23:23.785 INFO:tasks.workunit.client.1.smithi134.stdout:0/36: rmdir d1 39 2024-04-22T23:23:23.786 INFO:tasks.workunit.client.0.smithi067.stdout:0/269: write db/de/f1c [3302777,46038] 0 2024-04-22T23:23:23.788 INFO:tasks.workunit.client.0.smithi067.stdout:4/414: mknod d12/d17/d47/c98 0 2024-04-22T23:23:23.790 INFO:tasks.workunit.client.0.smithi067.stdout:4/415: dread fd [4194304,4194304] 0 2024-04-22T23:23:23.797 INFO:tasks.workunit.client.0.smithi067.stdout:2/280: mkdir d1/d19/d31/d53 0 2024-04-22T23:23:23.798 INFO:tasks.workunit.client.1.smithi134.stdout:0/37: fdatasync d1/d5/f7 0 2024-04-22T23:23:23.798 INFO:tasks.workunit.client.0.smithi067.stdout:8/297: link d1/c2d d1/dd/c5f 0 2024-04-22T23:23:23.800 INFO:tasks.workunit.client.0.smithi067.stdout:1/295: write d8/f11 [685130,71827] 0 2024-04-22T23:23:23.802 INFO:tasks.workunit.client.0.smithi067.stdout:8/298: creat d1/d41/d46/f60 x:0 0 0 2024-04-22T23:23:23.803 INFO:tasks.workunit.client.0.smithi067.stdout:0/270: creat db/d21/f5a x:0 0 0 2024-04-22T23:23:23.803 INFO:tasks.workunit.client.0.smithi067.stdout:8/299: fdatasync d1/d3/d14/f34 0 2024-04-22T23:23:23.804 INFO:tasks.workunit.client.0.smithi067.stdout:2/281: symlink d1/d10/d41/l54 0 2024-04-22T23:23:23.805 INFO:tasks.workunit.client.0.smithi067.stdout:1/296: mknod d8/d12/d36/c64 0 2024-04-22T23:23:23.807 INFO:tasks.workunit.client.0.smithi067.stdout:3/313: write d3/d4/fe [156925,605] 0 2024-04-22T23:23:23.808 INFO:tasks.workunit.client.0.smithi067.stdout:0/271: mknod db/de/d11/d1a/c5b 0 2024-04-22T23:23:23.810 INFO:tasks.workunit.client.0.smithi067.stdout:3/314: dread d3/d4/d12/f69 [0,4194304] 0 2024-04-22T23:23:23.815 INFO:tasks.workunit.client.0.smithi067.stdout:1/297: creat d8/d9/d2f/d33/d5c/f65 x:0 0 0 2024-04-22T23:23:23.817 INFO:tasks.workunit.client.0.smithi067.stdout:1/298: write d8/d9/f61 [503935,57548] 0 2024-04-22T23:23:23.820 INFO:tasks.workunit.client.0.smithi067.stdout:3/315: creat d3/d4/d12/f6e x:0 0 0 2024-04-22T23:23:23.820 INFO:tasks.workunit.client.0.smithi067.stdout:2/282: dread d1/f4 [4194304,4194304] 0 2024-04-22T23:23:23.823 INFO:tasks.workunit.client.0.smithi067.stdout:8/300: getdents d1/dd/d26 0 2024-04-22T23:23:23.826 INFO:tasks.workunit.client.0.smithi067.stdout:1/299: dwrite d8/d9/d2f/d33/d5c/f65 [0,4194304] 0 2024-04-22T23:23:23.830 INFO:tasks.workunit.client.0.smithi067.stdout:9/244: write d5/d21/f27 [1646086,125419] 0 2024-04-22T23:23:23.832 INFO:tasks.workunit.client.0.smithi067.stdout:1/300: dread d8/f11 [0,4194304] 0 2024-04-22T23:23:23.832 INFO:tasks.workunit.client.0.smithi067.stdout:8/301: unlink d1/dd/d26/l4a 0 2024-04-22T23:23:23.833 INFO:tasks.workunit.client.0.smithi067.stdout:2/283: mknod d1/d10/d1c/d4d/c55 0 2024-04-22T23:23:23.834 INFO:tasks.workunit.client.0.smithi067.stdout:1/301: write d8/d24/d34/f37 [287731,130980] 0 2024-04-22T23:23:23.836 INFO:tasks.workunit.client.0.smithi067.stdout:2/284: write d1/d12/f16 [1505097,85643] 0 2024-04-22T23:23:23.836 INFO:tasks.workunit.client.0.smithi067.stdout:3/316: creat d3/f6f x:0 0 0 2024-04-22T23:23:23.838 INFO:tasks.workunit.client.0.smithi067.stdout:2/285: stat d1/d42/d4a 0 2024-04-22T23:23:23.839 INFO:tasks.workunit.client.0.smithi067.stdout:9/245: dread d5/fb [0,4194304] 0 2024-04-22T23:23:23.849 INFO:tasks.workunit.client.0.smithi067.stdout:1/302: dwrite d8/d9/f61 [0,4194304] 0 2024-04-22T23:23:23.852 INFO:tasks.workunit.client.0.smithi067.stdout:3/317: mkdir d3/d4/d10/d38/d53/d55/d5c/d70 0 2024-04-22T23:23:23.853 INFO:tasks.workunit.client.0.smithi067.stdout:6/247: dwrite f13 [4194304,4194304] 0 2024-04-22T23:23:23.855 INFO:tasks.workunit.client.0.smithi067.stdout:9/246: rename d5/f29 to d5/dd/d18/d39/f51 0 2024-04-22T23:23:23.860 INFO:tasks.workunit.client.0.smithi067.stdout:2/286: dwrite d1/d10/d33/d46/f47 [0,4194304] 0 2024-04-22T23:23:23.860 INFO:tasks.workunit.client.0.smithi067.stdout:6/248: chown f13 1818 1 2024-04-22T23:23:23.860 INFO:tasks.workunit.client.0.smithi067.stdout:8/302: dwrite d1/d3/d27/f33 [4194304,4194304] 0 2024-04-22T23:23:23.866 INFO:tasks.workunit.client.0.smithi067.stdout:2/287: chown d1/d12/f4e 78 1 2024-04-22T23:23:23.867 INFO:tasks.workunit.client.0.smithi067.stdout:2/288: chown d1/d19/f21 27 1 2024-04-22T23:23:23.871 INFO:tasks.workunit.client.0.smithi067.stdout:9/247: rename d5/dd/d18/d3c/d40/f44 to d5/dd/d18/d39/d3e/d3f/f52 0 2024-04-22T23:23:23.874 INFO:tasks.workunit.client.0.smithi067.stdout:9/248: write d5/d2e/f4e [512745,4360] 0 2024-04-22T23:23:23.878 INFO:tasks.workunit.client.0.smithi067.stdout:2/289: rename d1/d10/d33/f37 to d1/d42/f56 0 2024-04-22T23:23:23.879 INFO:tasks.workunit.client.0.smithi067.stdout:2/290: chown d1/d10/d1c/l34 1045937 1 2024-04-22T23:23:23.881 INFO:tasks.workunit.client.0.smithi067.stdout:8/303: dread d1/d3/d10/f29 [0,4194304] 0 2024-04-22T23:23:23.883 INFO:tasks.workunit.client.0.smithi067.stdout:3/318: getdents d3/d4/d28 0 2024-04-22T23:23:23.885 INFO:tasks.workunit.client.0.smithi067.stdout:2/291: creat d1/d42/f57 x:0 0 0 2024-04-22T23:23:23.886 INFO:tasks.workunit.client.0.smithi067.stdout:6/249: getdents d17/d39/d3e 0 2024-04-22T23:23:23.887 INFO:tasks.workunit.client.0.smithi067.stdout:8/304: creat d1/d41/d46/f61 x:0 0 0 2024-04-22T23:23:23.888 INFO:tasks.workunit.client.0.smithi067.stdout:1/303: dread d8/d12/f17 [0,4194304] 0 2024-04-22T23:23:23.890 INFO:tasks.workunit.client.0.smithi067.stdout:6/250: unlink d17/d39/f4c 0 2024-04-22T23:23:23.890 INFO:tasks.workunit.client.0.smithi067.stdout:3/319: creat d3/d4/d28/d5f/f71 x:0 0 0 2024-04-22T23:23:23.893 INFO:tasks.workunit.client.0.smithi067.stdout:8/305: mkdir d1/dd/d26/d62 0 2024-04-22T23:23:23.894 INFO:tasks.workunit.client.0.smithi067.stdout:8/306: chown d1/dd/c40 9 1 2024-04-22T23:23:23.895 INFO:tasks.workunit.client.0.smithi067.stdout:1/304: mkdir d8/d9/d2f/d33/d66 0 2024-04-22T23:23:23.896 INFO:tasks.workunit.client.0.smithi067.stdout:6/251: symlink d17/l52 0 2024-04-22T23:23:23.899 INFO:tasks.workunit.client.0.smithi067.stdout:3/320: symlink d3/d4/d28/d4c/l72 0 2024-04-22T23:23:23.902 INFO:tasks.workunit.client.0.smithi067.stdout:2/292: getdents d1/d10/d41 0 2024-04-22T23:23:23.903 INFO:tasks.workunit.client.0.smithi067.stdout:8/307: dwrite d1/dd/d26/f22 [0,4194304] 0 2024-04-22T23:23:23.905 INFO:tasks.workunit.client.0.smithi067.stdout:2/293: fsync d1/f1f 0 2024-04-22T23:23:23.907 INFO:tasks.workunit.client.0.smithi067.stdout:6/252: dwrite f13 [4194304,4194304] 0 2024-04-22T23:23:23.908 INFO:tasks.workunit.client.0.smithi067.stdout:6/253: read f13 [7809964,11519] 0 2024-04-22T23:23:23.909 INFO:tasks.workunit.client.0.smithi067.stdout:5/285: sync 2024-04-22T23:23:23.914 INFO:tasks.workunit.client.0.smithi067.stdout:8/308: creat d1/d3/d27/f63 x:0 0 0 2024-04-22T23:23:23.916 INFO:tasks.workunit.client.0.smithi067.stdout:6/254: unlink fc 0 2024-04-22T23:23:23.917 INFO:tasks.workunit.client.0.smithi067.stdout:6/255: write d17/d39/d3a/f3b [4793349,7720] 0 2024-04-22T23:23:23.919 INFO:tasks.workunit.client.0.smithi067.stdout:6/256: write d17/d39/d3a/f3b [3868803,128117] 0 2024-04-22T23:23:23.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.918+0000 7ffbe920e700 1 -- 172.21.15.67:0/1010776181 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ffbe419ebd0 con 0x7ffbe40ffc10 2024-04-22T23:23:23.920 INFO:tasks.workunit.client.0.smithi067.stdout:2/294: creat d1/d19/d31/d53/f58 x:0 0 0 2024-04-22T23:23:23.920 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:23 smithi067 ceph-mon[26453]: from='client.24383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:23:23.920 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:23 smithi067 ceph-mon[26453]: Standby manager daemon smithi134.ccxkcj restarted 2024-04-22T23:23:23.920 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:23 smithi067 ceph-mon[26453]: Standby manager daemon smithi134.ccxkcj started 2024-04-22T23:23:23.921 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:23 smithi067 ceph-mon[26453]: from='mgr.? 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi134.ccxkcj/crt"}]: dispatch 2024-04-22T23:23:23.921 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:23 smithi067 ceph-mon[26453]: from='mgr.? 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T23:23:23.921 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:23 smithi067 ceph-mon[26453]: from='mgr.? 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi134.ccxkcj/key"}]: dispatch 2024-04-22T23:23:23.921 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:23 smithi067 ceph-mon[26453]: from='mgr.? 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T23:23:23.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.919+0000 7ffbcbfff700 1 -- 172.21.15.67:0/1010776181 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+766 (secure 0 0 0) 0x7ffbd4066650 con 0x7ffbe40ffc10 2024-04-22T23:23:23.921 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:23:23.921 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:23:23.921 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-22T23:23:23.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:23:23.922 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:23:23.922 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:23:23.922 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 1 2024-04-22T23:23:23.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:23:23.922 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:23:23.922 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-22T23:23:23.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:23:23.922 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:23:23.922 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-22T23:23:23.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:23:23.922 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:23:23.922 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 13, 2024-04-22T23:23:23.922 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 1 2024-04-22T23:23:23.922 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:23:23.923 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:23:23.923 INFO:tasks.workunit.client.0.smithi067.stdout:3/321: getdents d3/d4/d28/d40 0 2024-04-22T23:23:23.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.922+0000 7ffbe920e700 1 -- 172.21.15.67:0/1010776181 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ffbcc06c5a0 msgr2=0x7ffbcc06ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:23.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.922+0000 7ffbe920e700 1 --2- 172.21.15.67:0/1010776181 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ffbcc06c5a0 0x7ffbcc06ea60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ffbd800b6f0 tx=0x7ffbd8019040 comp rx=0 tx=0).stop 2024-04-22T23:23:23.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.922+0000 7ffbe920e700 1 -- 172.21.15.67:0/1010776181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbe40ffc10 msgr2=0x7ffbe4198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:23.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.922+0000 7ffbe920e700 1 --2- 172.21.15.67:0/1010776181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbe40ffc10 0x7ffbe4198670 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ffbd4005fd0 tx=0x7ffbd400edf0 comp rx=0 tx=0).stop 2024-04-22T23:23:23.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.922+0000 7ffbe920e700 1 -- 172.21.15.67:0/1010776181 shutdown_connections 2024-04-22T23:23:23.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.923+0000 7ffbe920e700 1 --2- 172.21.15.67:0/1010776181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbe40ffc10 0x7ffbe4198670 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:23.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.923+0000 7ffbe920e700 1 --2- 172.21.15.67:0/1010776181 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7ffbcc06c5a0 0x7ffbcc06ea60 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:23.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.923+0000 7ffbe920e700 1 --2- 172.21.15.67:0/1010776181 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffbe4100e10 0x7ffbe4198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:23.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.923+0000 7ffbe920e700 1 -- 172.21.15.67:0/1010776181 >> 172.21.15.67:0/1010776181 conn(0x7ffbe40fb190 msgr2=0x7ffbe4068580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:23:23.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.923+0000 7ffbe920e700 1 -- 172.21.15.67:0/1010776181 shutdown_connections 2024-04-22T23:23:23.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:23.923+0000 7ffbe920e700 1 -- 172.21.15.67:0/1010776181 wait complete. 2024-04-22T23:23:23.926 INFO:tasks.workunit.client.0.smithi067.stdout:6/257: unlink d17/d39/d3a/f4d 0 2024-04-22T23:23:23.927 INFO:tasks.workunit.client.0.smithi067.stdout:2/295: symlink d1/d19/d31/l59 0 2024-04-22T23:23:23.928 INFO:tasks.workunit.client.0.smithi067.stdout:5/286: creat d5/d3a/f5d x:0 0 0 2024-04-22T23:23:23.929 INFO:tasks.workunit.client.0.smithi067.stdout:2/296: truncate d1/d42/d4a/f4c 327639 0 2024-04-22T23:23:23.930 INFO:tasks.workunit.client.0.smithi067.stdout:5/287: rename d5/db to d5/db/d12/d44/d47/d5e 22 2024-04-22T23:23:23.932 INFO:tasks.workunit.client.0.smithi067.stdout:6/258: mkdir d17/d39/d53 0 2024-04-22T23:23:23.933 INFO:tasks.workunit.client.0.smithi067.stdout:1/305: truncate d8/d9/f61 2745754 0 2024-04-22T23:23:23.933 INFO:tasks.workunit.client.0.smithi067.stdout:2/297: dread d1/d19/f2f [0,4194304] 0 2024-04-22T23:23:23.934 INFO:tasks.workunit.client.0.smithi067.stdout:2/298: chown d1/d10/d33 16338 1 2024-04-22T23:23:23.934 INFO:tasks.workunit.client.0.smithi067.stdout:3/322: creat d3/d4/d28/d4a/f73 x:0 0 0 2024-04-22T23:23:23.937 INFO:tasks.workunit.client.0.smithi067.stdout:1/306: rmdir d8/d9/d2f/d59 39 2024-04-22T23:23:23.939 INFO:tasks.workunit.client.0.smithi067.stdout:1/307: dread - d8/d9/d2f/f5e zero size 2024-04-22T23:23:23.939 INFO:tasks.workunit.client.0.smithi067.stdout:5/288: mkdir d5/d28/d36/d40/d57/d5f 0 2024-04-22T23:23:23.940 INFO:tasks.workunit.client.0.smithi067.stdout:6/259: fsync d17/d39/f3f 0 2024-04-22T23:23:23.942 INFO:tasks.workunit.client.0.smithi067.stdout:2/299: dwrite d1/d10/f52 [0,4194304] 0 2024-04-22T23:23:23.942 INFO:tasks.workunit.client.0.smithi067.stdout:1/308: dread - d8/d12/d36/f4e zero size 2024-04-22T23:23:23.947 INFO:tasks.workunit.client.0.smithi067.stdout:1/309: creat d8/d9/d2d/d4d/d48/f67 x:0 0 0 2024-04-22T23:23:23.953 INFO:tasks.workunit.client.0.smithi067.stdout:2/300: rename d1/d10/d1c/d4d/c55 to d1/d4f/c5a 0 2024-04-22T23:23:23.956 INFO:tasks.workunit.client.0.smithi067.stdout:2/301: creat d1/d10/d33/f5b x:0 0 0 2024-04-22T23:23:23.957 INFO:tasks.workunit.client.0.smithi067.stdout:2/302: truncate d1/f8 4471538 0 2024-04-22T23:23:23.960 INFO:tasks.workunit.client.0.smithi067.stdout:2/303: unlink d1/d4f/f3d 0 2024-04-22T23:23:23.963 INFO:tasks.workunit.client.0.smithi067.stdout:6/260: dwrite fe [0,4194304] 0 2024-04-22T23:23:23.964 INFO:tasks.workunit.client.0.smithi067.stdout:5/289: truncate d5/db/d12/f22 357789 0 2024-04-22T23:23:23.965 INFO:tasks.workunit.client.0.smithi067.stdout:6/261: fsync d17/f48 0 2024-04-22T23:23:23.967 INFO:tasks.workunit.client.0.smithi067.stdout:5/290: fdatasync d5/db/ff 0 2024-04-22T23:23:23.967 INFO:tasks.workunit.client.0.smithi067.stdout:6/262: fsync d17/f42 0 2024-04-22T23:23:23.971 INFO:tasks.workunit.client.0.smithi067.stdout:2/304: dwrite d1/d10/d33/f3e [0,4194304] 0 2024-04-22T23:23:23.975 INFO:tasks.workunit.client.0.smithi067.stdout:6/263: creat d17/d39/d3e/d46/f54 x:0 0 0 2024-04-22T23:23:23.976 INFO:tasks.workunit.client.0.smithi067.stdout:6/264: readlink d17/l37 0 2024-04-22T23:23:23.977 INFO:tasks.workunit.client.0.smithi067.stdout:5/291: creat d5/d3a/f60 x:0 0 0 2024-04-22T23:23:23.978 INFO:tasks.workunit.client.0.smithi067.stdout:5/292: write d5/f5b [659684,58325] 0 2024-04-22T23:23:23.982 INFO:tasks.workunit.client.0.smithi067.stdout:2/305: getdents d1/d19/d31 0 2024-04-22T23:23:23.982 INFO:tasks.workunit.client.0.smithi067.stdout:2/306: stat d1/d10/d33/l51 0 2024-04-22T23:23:23.983 INFO:tasks.workunit.client.0.smithi067.stdout:2/307: chown d1/d12/l13 61 1 2024-04-22T23:23:23.987 INFO:tasks.workunit.client.0.smithi067.stdout:2/308: read d1/d10/f2e [1845886,71039] 0 2024-04-22T23:23:23.992 INFO:tasks.workunit.client.0.smithi067.stdout:2/309: creat d1/d19/f5c x:0 0 0 2024-04-22T23:23:23.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:23 smithi134 ceph-mon[34609]: from='client.24383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:23:23.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:23 smithi134 ceph-mon[34609]: Standby manager daemon smithi134.ccxkcj restarted 2024-04-22T23:23:23.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:23 smithi134 ceph-mon[34609]: Standby manager daemon smithi134.ccxkcj started 2024-04-22T23:23:23.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:23 smithi134 ceph-mon[34609]: from='mgr.? 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi134.ccxkcj/crt"}]: dispatch 2024-04-22T23:23:23.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:23 smithi134 ceph-mon[34609]: from='mgr.? 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T23:23:23.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:23 smithi134 ceph-mon[34609]: from='mgr.? 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi134.ccxkcj/key"}]: dispatch 2024-04-22T23:23:23.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:23 smithi134 ceph-mon[34609]: from='mgr.? 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T23:23:23.995 INFO:tasks.workunit.client.0.smithi067.stdout:2/310: creat d1/d10/d1c/f5d x:0 0 0 2024-04-22T23:23:23.995 INFO:tasks.workunit.client.0.smithi067.stdout:6/265: truncate d17/f20 3654568 0 2024-04-22T23:23:23.996 INFO:tasks.workunit.client.0.smithi067.stdout:2/311: write d1/d10/d33/f3e [2163349,96378] 0 2024-04-22T23:23:24.001 INFO:tasks.workunit.client.0.smithi067.stdout:6/266: rename d17/f1b to d17/d39/d53/f55 0 2024-04-22T23:23:24.001 INFO:tasks.workunit.client.0.smithi067.stdout:5/293: dwrite d5/f11 [0,4194304] 0 2024-04-22T23:23:24.005 INFO:tasks.workunit.client.0.smithi067.stdout:5/294: write d5/db/d35/f46 [1669574,60900] 0 2024-04-22T23:23:24.006 INFO:tasks.workunit.client.0.smithi067.stdout:2/312: chown d1/d4f/c5a 233 1 2024-04-22T23:23:24.008 INFO:tasks.workunit.client.0.smithi067.stdout:5/295: readlink d5/db/d35/l38 0 2024-04-22T23:23:24.012 INFO:tasks.workunit.client.0.smithi067.stdout:5/296: truncate d5/db/d12/f2b 4961206 0 2024-04-22T23:23:24.014 INFO:tasks.workunit.client.0.smithi067.stdout:6/267: dwrite d17/d39/d3a/f51 [0,4194304] 0 2024-04-22T23:23:24.018 INFO:tasks.workunit.client.0.smithi067.stdout:5/297: mkdir d5/d28/d36/d40/d57/d61 0 2024-04-22T23:23:24.018 INFO:tasks.workunit.client.0.smithi067.stdout:2/313: dwrite d1/d10/d1c/f3a [0,4194304] 0 2024-04-22T23:23:24.019 INFO:tasks.workunit.client.0.smithi067.stdout:6/268: creat d17/d39/d3a/f56 x:0 0 0 2024-04-22T23:23:24.020 INFO:tasks.workunit.client.0.smithi067.stdout:5/298: write d5/d3a/f3c [312917,63144] 0 2024-04-22T23:23:24.022 INFO:tasks.workunit.client.0.smithi067.stdout:2/314: creat d1/d10/d1c/f5e x:0 0 0 2024-04-22T23:23:24.043 INFO:tasks.workunit.client.0.smithi067.stdout:7/233: sync 2024-04-22T23:23:24.043 INFO:tasks.workunit.client.0.smithi067.stdout:7/234: dread - d10/d18/f27 zero size 2024-04-22T23:23:24.045 INFO:tasks.workunit.client.0.smithi067.stdout:7/235: write d10/d18/d35/f4e [1958526,9690] 0 2024-04-22T23:23:24.061 INFO:tasks.workunit.client.0.smithi067.stdout:6/269: rmdir d17/d39/d3e/d46 39 2024-04-22T23:23:24.063 INFO:tasks.workunit.client.0.smithi067.stdout:5/299: truncate d5/d3a/d4d/f4e 21258 0 2024-04-22T23:23:24.067 INFO:tasks.workunit.client.0.smithi067.stdout:5/300: mknod d5/d3e/c62 0 2024-04-22T23:23:24.070 INFO:tasks.workunit.client.0.smithi067.stdout:7/236: mknod d10/d14/d1d/d24/c52 0 2024-04-22T23:23:24.070 INFO:tasks.workunit.client.0.smithi067.stdout:6/270: dread d17/d39/f49 [0,4194304] 0 2024-04-22T23:23:24.073 INFO:tasks.workunit.client.0.smithi067.stdout:5/301: symlink d5/d28/d36/d40/d57/d61/l63 0 2024-04-22T23:23:24.073 INFO:tasks.workunit.client.0.smithi067.stdout:6/271: symlink d17/l57 0 2024-04-22T23:23:24.074 INFO:tasks.workunit.client.0.smithi067.stdout:7/237: truncate d10/d18/f1e 446185 0 2024-04-22T23:23:24.076 INFO:tasks.workunit.client.0.smithi067.stdout:6/272: fsync d17/d39/d3a/f4f 0 2024-04-22T23:23:24.076 INFO:tasks.workunit.client.0.smithi067.stdout:2/315: dwrite d1/f4 [4194304,4194304] 0 2024-04-22T23:23:24.078 INFO:tasks.workunit.client.0.smithi067.stdout:2/316: readlink d1/d19/d31/l59 0 2024-04-22T23:23:24.082 INFO:tasks.workunit.client.0.smithi067.stdout:6/273: read d17/f20 [3644408,108630] 0 2024-04-22T23:23:24.083 INFO:tasks.workunit.client.0.smithi067.stdout:5/302: dwrite d5/d28/d36/d40/f16 [4194304,4194304] 0 2024-04-22T23:23:24.084 INFO:tasks.workunit.client.0.smithi067.stdout:2/317: write d1/d10/d1c/f3a [3707552,74240] 0 2024-04-22T23:23:24.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.084+0000 7f1227f79700 1 -- 172.21.15.67:0/1165290570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12200fef60 msgr2=0x7f12200ff380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:24.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.084+0000 7f1227f79700 1 --2- 172.21.15.67:0/1165290570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12200fef60 0x7f12200ff380 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f121c009910 tx=0x7f121c009c20 comp rx=0 tx=0).stop 2024-04-22T23:23:24.086 INFO:tasks.workunit.client.0.smithi067.stdout:2/318: read - d1/d19/f5c zero size 2024-04-22T23:23:24.086 INFO:tasks.workunit.client.0.smithi067.stdout:6/274: dread - d17/d39/d3e/d46/f54 zero size 2024-04-22T23:23:24.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.085+0000 7f1227f79700 1 -- 172.21.15.67:0/1165290570 shutdown_connections 2024-04-22T23:23:24.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.085+0000 7f1227f79700 1 --2- 172.21.15.67:0/1165290570 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12200ffcd0 0x7f1220100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:24.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.085+0000 7f1227f79700 1 --2- 172.21.15.67:0/1165290570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12200fef60 0x7f12200ff380 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:24.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.085+0000 7f1227f79700 1 -- 172.21.15.67:0/1165290570 >> 172.21.15.67:0/1165290570 conn(0x7f12200fa8c0 msgr2=0x7f12200fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:23:24.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.086+0000 7f1227f79700 1 -- 172.21.15.67:0/1165290570 shutdown_connections 2024-04-22T23:23:24.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.086+0000 7f1227f79700 1 -- 172.21.15.67:0/1165290570 wait complete. 2024-04-22T23:23:24.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.087+0000 7f1227f79700 1 Processor -- start 2024-04-22T23:23:24.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.087+0000 7f1227f79700 1 -- start start 2024-04-22T23:23:24.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.087+0000 7f1227f79700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12200fef60 0x7f122019cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:23:24.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.087+0000 7f1227f79700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12200ffcd0 0x7f122019d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:23:24.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.088+0000 7f1225514700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12200ffcd0 0x7f122019d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:23:24.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.088+0000 7f1225514700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12200ffcd0 0x7f122019d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40774/0 (socket says 172.21.15.67:40774) 2024-04-22T23:23:24.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.088+0000 7f1225d15700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12200fef60 0x7f122019cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:23:24.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.088+0000 7f1225514700 1 -- 172.21.15.67:0/3903389359 learned_addr learned my addr 172.21.15.67:0/3903389359 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:23:24.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.088+0000 7f1227f79700 1 -- 172.21.15.67:0/3903389359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f122019d620 con 0x7f12200ffcd0 2024-04-22T23:23:24.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.088+0000 7f1227f79700 1 -- 172.21.15.67:0/3903389359 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f122019d760 con 0x7f12200fef60 2024-04-22T23:23:24.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.089+0000 7f1225514700 1 -- 172.21.15.67:0/3903389359 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12200fef60 msgr2=0x7f122019cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:24.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.089+0000 7f1225514700 1 --2- 172.21.15.67:0/3903389359 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12200fef60 0x7f122019cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:24.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.089+0000 7f1225514700 1 -- 172.21.15.67:0/3903389359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f121c0095f0 con 0x7f12200ffcd0 2024-04-22T23:23:24.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.089+0000 7f1225514700 1 --2- 172.21.15.67:0/3903389359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12200ffcd0 0x7f122019d000 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f121000d8d0 tx=0x7f121000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:23:24.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.090+0000 7f1216ffd700 1 -- 172.21.15.67:0/3903389359 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1210009940 con 0x7f12200ffcd0 2024-04-22T23:23:24.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.090+0000 7f1216ffd700 1 -- 172.21.15.67:0/3903389359 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1210010460 con 0x7f12200ffcd0 2024-04-22T23:23:24.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.090+0000 7f1216ffd700 1 -- 172.21.15.67:0/3903389359 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f121000f5f0 con 0x7f12200ffcd0 2024-04-22T23:23:24.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.090+0000 7f1227f79700 1 -- 172.21.15.67:0/3903389359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12201a2210 con 0x7f12200ffcd0 2024-04-22T23:23:24.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.090+0000 7f1227f79700 1 -- 172.21.15.67:0/3903389359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12201a2760 con 0x7f12200ffcd0 2024-04-22T23:23:24.093 INFO:tasks.workunit.client.0.smithi067.stdout:5/303: rename d5/f5b to d5/d28/f64 0 2024-04-22T23:23:24.093 INFO:tasks.workunit.client.0.smithi067.stdout:6/275: dread d17/d39/d3a/f51 [0,4194304] 0 2024-04-22T23:23:24.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.092+0000 7f1227f79700 1 -- 172.21.15.67:0/3903389359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1204005320 con 0x7f12200ffcd0 2024-04-22T23:23:24.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.093+0000 7f1216ffd700 1 -- 172.21.15.67:0/3903389359 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 20) v1 ==== 91455+0+0 (secure 0 0 0) 0x7f1210009aa0 con 0x7f12200ffcd0 2024-04-22T23:23:24.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.094+0000 7f1216ffd700 1 --2- 172.21.15.67:0/3903389359 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f120c06d710 0x7f120c06fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:23:24.095 INFO:tasks.workunit.client.0.smithi067.stdout:6/276: fdatasync d17/d39/d3a/f56 0 2024-04-22T23:23:24.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.094+0000 7f1216ffd700 1 -- 172.21.15.67:0/3903389359 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f121008ce00 con 0x7f12200ffcd0 2024-04-22T23:23:24.095 INFO:tasks.workunit.client.0.smithi067.stdout:2/319: getdents d1/d19/d31 0 2024-04-22T23:23:24.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.095+0000 7f1225d15700 1 --2- 172.21.15.67:0/3903389359 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f120c06d710 0x7f120c06fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:23:24.097 INFO:tasks.workunit.client.0.smithi067.stdout:2/320: dread - d1/d4f/f2a zero size 2024-04-22T23:23:24.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.096+0000 7f1225d15700 1 --2- 172.21.15.67:0/3903389359 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f120c06d710 0x7f120c06fbd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f121c006010 tx=0x7f121c00b540 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:23:24.097 INFO:tasks.workunit.client.0.smithi067.stdout:7/238: getdents d10/d14/d1d/d24 0 2024-04-22T23:23:24.100 INFO:tasks.workunit.client.0.smithi067.stdout:2/321: truncate d1/d10/f30 752798 0 2024-04-22T23:23:24.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.102+0000 7f1216ffd700 1 -- 172.21.15.67:0/3903389359 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f121005ae00 con 0x7f12200ffcd0 2024-04-22T23:23:24.103 INFO:tasks.workunit.client.0.smithi067.stdout:7/239: rmdir d10/d14/d1d/d46 39 2024-04-22T23:23:24.104 INFO:tasks.workunit.client.0.smithi067.stdout:5/304: dwrite d5/d3a/f58 [0,4194304] 0 2024-04-22T23:23:24.106 INFO:tasks.workunit.client.0.smithi067.stdout:2/322: mkdir d1/d42/d5f 0 2024-04-22T23:23:24.109 INFO:tasks.workunit.client.0.smithi067.stdout:2/323: dread - d1/d10/d1c/f5e zero size 2024-04-22T23:23:24.109 INFO:tasks.workunit.client.0.smithi067.stdout:2/324: chown d1/d10/d41/l54 6233 1 2024-04-22T23:23:24.111 INFO:tasks.workunit.client.0.smithi067.stdout:2/325: truncate d1/d4f/f2a 952461 0 2024-04-22T23:23:24.115 INFO:tasks.workunit.client.0.smithi067.stdout:2/326: unlink d1/d12/f16 0 2024-04-22T23:23:24.120 INFO:tasks.workunit.client.0.smithi067.stdout:7/240: link d10/d14/d1d/d26/d2c/l33 d10/d14/d1d/d24/l53 0 2024-04-22T23:23:24.120 INFO:tasks.workunit.client.0.smithi067.stdout:5/305: dread d5/db/d12/f22 [0,4194304] 0 2024-04-22T23:23:24.121 INFO:tasks.workunit.client.0.smithi067.stdout:6/277: dwrite d17/f24 [0,4194304] 0 2024-04-22T23:23:24.125 INFO:tasks.workunit.client.0.smithi067.stdout:2/327: dwrite d1/f4 [0,4194304] 0 2024-04-22T23:23:24.127 INFO:tasks.workunit.client.0.smithi067.stdout:5/306: unlink d5/db/d12/c2d 0 2024-04-22T23:23:24.128 INFO:tasks.workunit.client.0.smithi067.stdout:6/278: symlink d17/d39/d3e/d46/l58 0 2024-04-22T23:23:24.131 INFO:tasks.workunit.client.0.smithi067.stdout:6/279: chown l14 82810 1 2024-04-22T23:23:24.132 INFO:tasks.workunit.client.0.smithi067.stdout:2/328: symlink d1/d10/d33/l60 0 2024-04-22T23:23:24.132 INFO:tasks.workunit.client.0.smithi067.stdout:7/241: dwrite d10/d14/d1d/d26/f39 [0,4194304] 0 2024-04-22T23:23:24.134 INFO:tasks.workunit.client.0.smithi067.stdout:2/329: chown d1/f2d 7 1 2024-04-22T23:23:24.150 INFO:tasks.workunit.client.0.smithi067.stdout:6/280: dread d17/f26 [0,4194304] 0 2024-04-22T23:23:24.151 INFO:tasks.workunit.client.0.smithi067.stdout:6/281: dread - d17/d39/d3a/f56 zero size 2024-04-22T23:23:24.153 INFO:tasks.workunit.client.0.smithi067.stdout:6/282: creat d17/d39/d53/f59 x:0 0 0 2024-04-22T23:23:24.155 INFO:tasks.workunit.client.0.smithi067.stdout:6/283: creat d17/d39/d3a/f5a x:0 0 0 2024-04-22T23:23:24.158 INFO:tasks.workunit.client.0.smithi067.stdout:6/284: truncate d17/d39/f4b 193111 0 2024-04-22T23:23:24.163 INFO:tasks.workunit.client.0.smithi067.stdout:6/285: symlink d17/d39/d3a/l5b 0 2024-04-22T23:23:24.165 INFO:tasks.workunit.client.0.smithi067.stdout:7/242: dread fe [0,4194304] 0 2024-04-22T23:23:24.165 INFO:tasks.workunit.client.0.smithi067.stdout:6/286: mknod d17/d39/c5c 0 2024-04-22T23:23:24.170 INFO:tasks.workunit.client.0.smithi067.stdout:6/287: dread d17/f24 [0,4194304] 0 2024-04-22T23:23:24.174 INFO:tasks.workunit.client.0.smithi067.stdout:5/307: dread d5/f15 [4194304,4194304] 0 2024-04-22T23:23:24.175 INFO:tasks.workunit.client.0.smithi067.stdout:2/330: rmdir d1/d12 39 2024-04-22T23:23:24.177 INFO:tasks.workunit.client.0.smithi067.stdout:2/331: truncate d1/d19/d31/f36 823850 0 2024-04-22T23:23:24.177 INFO:tasks.workunit.client.0.smithi067.stdout:2/332: chown d1/d42/d5f 2192 1 2024-04-22T23:23:24.178 INFO:tasks.workunit.client.0.smithi067.stdout:5/308: creat d5/d28/d36/d40/f65 x:0 0 0 2024-04-22T23:23:24.179 INFO:tasks.workunit.client.0.smithi067.stdout:2/333: mkdir d1/d42/d61 0 2024-04-22T23:23:24.179 INFO:tasks.workunit.client.0.smithi067.stdout:2/334: read d1/f2d [411085,116145] 0 2024-04-22T23:23:24.181 INFO:tasks.workunit.client.0.smithi067.stdout:5/309: symlink d5/d28/d36/d40/l66 0 2024-04-22T23:23:24.184 INFO:tasks.workunit.client.0.smithi067.stdout:5/310: symlink d5/d3a/d4d/l67 0 2024-04-22T23:23:24.186 INFO:tasks.workunit.client.0.smithi067.stdout:2/335: dwrite d1/d10/d33/f5b [0,4194304] 0 2024-04-22T23:23:24.187 INFO:tasks.workunit.client.0.smithi067.stdout:5/311: chown d5/d28/d36/d40/d57/d61/l63 32746 1 2024-04-22T23:23:24.189 INFO:tasks.workunit.client.0.smithi067.stdout:2/336: write d1/d10/f30 [1751825,57914] 0 2024-04-22T23:23:24.195 INFO:tasks.workunit.client.0.smithi067.stdout:5/312: creat d5/f68 x:0 0 0 2024-04-22T23:23:24.199 INFO:tasks.workunit.client.0.smithi067.stdout:5/313: rmdir d5/d28/d36/d40 39 2024-04-22T23:23:24.202 INFO:tasks.workunit.client.0.smithi067.stdout:6/288: write d17/d39/f4b [671671,36949] 0 2024-04-22T23:23:24.204 INFO:tasks.workunit.client.0.smithi067.stdout:5/314: rename d5/d28/d36/d40/f65 to d5/d28/f69 0 2024-04-22T23:23:24.207 INFO:tasks.workunit.client.0.smithi067.stdout:6/289: dread d17/d39/d3a/f51 [0,4194304] 0 2024-04-22T23:23:24.208 INFO:tasks.workunit.client.0.smithi067.stdout:5/315: unlink d5/d3e/f55 0 2024-04-22T23:23:24.209 INFO:tasks.workunit.client.0.smithi067.stdout:2/337: dread d1/fc [0,4194304] 0 2024-04-22T23:23:24.210 INFO:tasks.workunit.client.0.smithi067.stdout:6/290: symlink d17/l5d 0 2024-04-22T23:23:24.211 INFO:tasks.workunit.client.0.smithi067.stdout:2/338: write d1/d12/f4e [90169,32019] 0 2024-04-22T23:23:24.212 INFO:tasks.workunit.client.0.smithi067.stdout:5/316: rename d5/db/d35/f48 to d5/db/f6a 0 2024-04-22T23:23:24.212 INFO:tasks.workunit.client.0.smithi067.stdout:2/339: write d1/d42/d4a/f4c [1247840,13831] 0 2024-04-22T23:23:24.215 INFO:tasks.workunit.client.0.smithi067.stdout:6/291: mknod d17/d39/d3e/c5e 0 2024-04-22T23:23:24.216 INFO:tasks.workunit.client.0.smithi067.stdout:5/317: mknod d5/d3e/c6b 0 2024-04-22T23:23:24.216 INFO:tasks.workunit.client.0.smithi067.stdout:6/292: dread - d17/d39/d53/f59 zero size 2024-04-22T23:23:24.218 INFO:tasks.workunit.client.0.smithi067.stdout:5/318: creat d5/db/d35/f6c x:0 0 0 2024-04-22T23:23:24.219 INFO:tasks.workunit.client.0.smithi067.stdout:6/293: creat d17/d39/d3e/d46/f5f x:0 0 0 2024-04-22T23:23:24.220 INFO:tasks.workunit.client.0.smithi067.stdout:2/340: symlink d1/d42/d61/l62 0 2024-04-22T23:23:24.222 INFO:tasks.workunit.client.0.smithi067.stdout:2/341: dread d1/d10/f24 [0,4194304] 0 2024-04-22T23:23:24.223 INFO:tasks.workunit.client.0.smithi067.stdout:5/319: dread d5/db/f23 [0,4194304] 0 2024-04-22T23:23:24.223 INFO:tasks.workunit.client.0.smithi067.stdout:6/294: creat d17/d39/d53/f60 x:0 0 0 2024-04-22T23:23:24.224 INFO:tasks.workunit.client.0.smithi067.stdout:2/342: write d1/d10/d1c/f5d [124856,17498] 0 2024-04-22T23:23:24.226 INFO:tasks.workunit.client.0.smithi067.stdout:5/320: creat d5/db/d12/d44/d47/f6d x:0 0 0 2024-04-22T23:23:24.227 INFO:tasks.workunit.client.0.smithi067.stdout:6/295: creat d17/d39/d3a/f61 x:0 0 0 2024-04-22T23:23:24.228 INFO:tasks.workunit.client.0.smithi067.stdout:2/343: mknod d1/d10/d33/c63 0 2024-04-22T23:23:24.229 INFO:tasks.workunit.client.0.smithi067.stdout:2/344: fsync d1/d19/d31/d53/f58 0 2024-04-22T23:23:24.230 INFO:tasks.workunit.client.0.smithi067.stdout:5/321: mkdir d5/d28/d36/d40/d57/d61/d6e 0 2024-04-22T23:23:24.231 INFO:tasks.workunit.client.0.smithi067.stdout:5/322: write d5/d3a/f5d [183780,122838] 0 2024-04-22T23:23:24.232 INFO:tasks.workunit.client.0.smithi067.stdout:2/345: symlink d1/d12/l64 0 2024-04-22T23:23:24.232 INFO:tasks.workunit.client.0.smithi067.stdout:2/346: chown d1/d10 0 1 2024-04-22T23:23:24.233 INFO:tasks.workunit.client.0.smithi067.stdout:2/347: fsync d1/d42/f57 0 2024-04-22T23:23:24.233 INFO:tasks.workunit.client.0.smithi067.stdout:5/323: creat d5/db/d35/f6f x:0 0 0 2024-04-22T23:23:24.234 INFO:tasks.workunit.client.0.smithi067.stdout:2/348: readlink d1/d10/l39 0 2024-04-22T23:23:24.236 INFO:tasks.workunit.client.0.smithi067.stdout:5/324: creat d5/d28/d36/d40/f70 x:0 0 0 2024-04-22T23:23:24.239 INFO:tasks.workunit.client.0.smithi067.stdout:4/416: sync 2024-04-22T23:23:24.240 INFO:tasks.workunit.client.0.smithi067.stdout:2/349: dwrite d1/f1b [0,4194304] 0 2024-04-22T23:23:24.243 INFO:tasks.workunit.client.0.smithi067.stdout:2/350: symlink d1/d42/d61/l65 0 2024-04-22T23:23:24.246 INFO:tasks.workunit.client.0.smithi067.stdout:2/351: mkdir d1/d42/d5f/d66 0 2024-04-22T23:23:24.248 INFO:tasks.workunit.client.0.smithi067.stdout:5/325: truncate d5/f11 4074996 0 2024-04-22T23:23:24.251 INFO:tasks.workunit.client.0.smithi067.stdout:5/326: creat d5/f71 x:0 0 0 2024-04-22T23:23:24.254 INFO:tasks.workunit.client.0.smithi067.stdout:2/352: dwrite d1/d10/d33/f49 [0,4194304] 0 2024-04-22T23:23:24.256 INFO:tasks.workunit.client.0.smithi067.stdout:2/353: unlink d1/d10/l39 0 2024-04-22T23:23:24.258 INFO:tasks.workunit.client.0.smithi067.stdout:5/327: link d5/db/d12/d44/f56 d5/d28/d36/d40/d57/f72 0 2024-04-22T23:23:24.261 INFO:tasks.workunit.client.0.smithi067.stdout:5/328: stat d5/db/d12/d44/d47/f4f 0 2024-04-22T23:23:24.262 INFO:tasks.workunit.client.0.smithi067.stdout:2/354: truncate d1/f1f 1702748 0 2024-04-22T23:23:24.264 INFO:tasks.workunit.client.0.smithi067.stdout:2/355: mknod d1/d42/d4a/c67 0 2024-04-22T23:23:24.267 INFO:tasks.workunit.client.0.smithi067.stdout:5/329: getdents d5/db/d12/d44/d47 0 2024-04-22T23:23:24.269 INFO:tasks.workunit.client.0.smithi067.stdout:2/356: rmdir d1/d42/d5f/d66 0 2024-04-22T23:23:24.269 INFO:tasks.workunit.client.0.smithi067.stdout:2/357: chown d1/d12/f4e 0 1 2024-04-22T23:23:24.270 INFO:tasks.workunit.client.0.smithi067.stdout:2/358: chown d1/d10/f52 1023008061 1 2024-04-22T23:23:24.270 INFO:tasks.workunit.client.0.smithi067.stdout:5/330: rename d5/db/d12/c29 to d5/db/c73 0 2024-04-22T23:23:24.272 INFO:tasks.workunit.client.0.smithi067.stdout:2/359: mkdir d1/d19/d31/d68 0 2024-04-22T23:23:24.274 INFO:tasks.workunit.client.0.smithi067.stdout:5/331: rename d5/d28/d36/d40/c1f to d5/db/d12/d44/c74 0 2024-04-22T23:23:24.279 INFO:tasks.workunit.client.0.smithi067.stdout:2/360: dwrite d1/d19/d31/d53/f58 [0,4194304] 0 2024-04-22T23:23:24.279 INFO:tasks.workunit.client.0.smithi067.stdout:5/332: link d5/d28/d36/d40/f70 d5/d28/d36/f75 0 2024-04-22T23:23:24.283 INFO:tasks.workunit.client.0.smithi067.stdout:2/361: unlink d1/d12/f48 0 2024-04-22T23:23:24.284 INFO:tasks.workunit.client.0.smithi067.stdout:2/362: readlink d1/d12/l29 0 2024-04-22T23:23:24.284 INFO:tasks.workunit.client.0.smithi067.stdout:5/333: creat d5/d3e/f76 x:0 0 0 2024-04-22T23:23:24.288 INFO:tasks.workunit.client.0.smithi067.stdout:5/334: creat d5/d3e/f77 x:0 0 0 2024-04-22T23:23:24.291 INFO:tasks.workunit.client.0.smithi067.stdout:2/363: creat d1/f69 x:0 0 0 2024-04-22T23:23:24.296 INFO:tasks.workunit.client.0.smithi067.stdout:2/364: unlink d1/d10/d1c/l34 0 2024-04-22T23:23:24.300 INFO:tasks.workunit.client.0.smithi067.stdout:2/365: truncate d1/d10/f24 615393 0 2024-04-22T23:23:24.302 INFO:tasks.workunit.client.0.smithi067.stdout:5/335: getdents d5/d28 0 2024-04-22T23:23:24.305 INFO:tasks.workunit.client.0.smithi067.stdout:2/366: getdents d1/d10/d33/d46 0 2024-04-22T23:23:24.311 INFO:tasks.workunit.client.0.smithi067.stdout:2/367: unlink d1/d19/f2f 0 2024-04-22T23:23:24.314 INFO:tasks.workunit.client.0.smithi067.stdout:2/368: mknod d1/d10/d1c/c6a 0 2024-04-22T23:23:24.316 INFO:tasks.workunit.client.0.smithi067.stdout:2/369: mkdir d1/d42/d6b 0 2024-04-22T23:23:24.317 INFO:tasks.workunit.client.0.smithi067.stdout:2/370: dread d1/d19/d31/f36 [0,4194304] 0 2024-04-22T23:23:24.319 INFO:tasks.workunit.client.0.smithi067.stdout:5/336: truncate d5/db/d35/f39 3198121 0 2024-04-22T23:23:24.320 INFO:tasks.workunit.client.0.smithi067.stdout:2/371: mknod d1/d42/d5f/c6c 0 2024-04-22T23:23:24.323 INFO:tasks.workunit.client.0.smithi067.stdout:5/337: creat d5/d28/d36/d40/d57/d61/f78 x:0 0 0 2024-04-22T23:23:24.325 INFO:tasks.workunit.client.0.smithi067.stdout:5/338: mkdir d5/d28/d36/d40/d57/d5f/d79 0 2024-04-22T23:23:24.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.413+0000 7f1227f79700 1 -- 172.21.15.67:0/3903389359 --> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1204000bf0 con 0x7f120c06d710 2024-04-22T23:23:24.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.414+0000 7f1216ffd700 1 -- 172.21.15.67:0/3903389359 <== mgr.14215 v2:172.21.15.67:6800/2589747850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+321 (secure 0 0 0) 0x7f1204000bf0 con 0x7f120c06d710 2024-04-22T23:23:24.415 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:23:24.416 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:23:24.416 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:23:24.416 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-22T23:23:24.416 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [], 2024-04-22T23:23:24.416 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "", 2024-04-22T23:23:24.416 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mgr daemons", 2024-04-22T23:23:24.416 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:23:24.416 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:23:24.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.416+0000 7f1227f79700 1 -- 172.21.15.67:0/3903389359 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f120c06d710 msgr2=0x7f120c06fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:24.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.416+0000 7f1227f79700 1 --2- 172.21.15.67:0/3903389359 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f120c06d710 0x7f120c06fbd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f121c006010 tx=0x7f121c00b540 comp rx=0 tx=0).stop 2024-04-22T23:23:24.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.416+0000 7f1227f79700 1 -- 172.21.15.67:0/3903389359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12200ffcd0 msgr2=0x7f122019d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:24.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.416+0000 7f1227f79700 1 --2- 172.21.15.67:0/3903389359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12200ffcd0 0x7f122019d000 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f121000d8d0 tx=0x7f121000dc90 comp rx=0 tx=0).stop 2024-04-22T23:23:24.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.416+0000 7f1227f79700 1 -- 172.21.15.67:0/3903389359 shutdown_connections 2024-04-22T23:23:24.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.416+0000 7f1227f79700 1 --2- 172.21.15.67:0/3903389359 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f120c06d710 0x7f120c06fbd0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:24.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.416+0000 7f1227f79700 1 --2- 172.21.15.67:0/3903389359 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12200fef60 0x7f122019cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:24.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.416+0000 7f1227f79700 1 --2- 172.21.15.67:0/3903389359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12200ffcd0 0x7f122019d000 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:24.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.416+0000 7f1227f79700 1 -- 172.21.15.67:0/3903389359 >> 172.21.15.67:0/3903389359 conn(0x7f12200fa8c0 msgr2=0x7f1220102f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:23:24.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.416+0000 7f1227f79700 1 -- 172.21.15.67:0/3903389359 shutdown_connections 2024-04-22T23:23:24.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:24.416+0000 7f1227f79700 1 -- 172.21.15.67:0/3903389359 wait complete. 2024-04-22T23:23:24.430 INFO:tasks.workunit.client.0.smithi067.stdout:0/272: sync 2024-04-22T23:23:24.430 INFO:tasks.workunit.client.0.smithi067.stdout:9/249: sync 2024-04-22T23:23:24.432 INFO:tasks.workunit.client.0.smithi067.stdout:9/250: creat d5/dd/d18/d39/d3e/f53 x:0 0 0 2024-04-22T23:23:24.433 INFO:tasks.workunit.client.0.smithi067.stdout:0/273: rename db/d21/c4e to db/d2e/d31/c5c 0 2024-04-22T23:23:24.438 INFO:tasks.workunit.client.0.smithi067.stdout:9/251: dread d5/f10 [0,4194304] 0 2024-04-22T23:23:24.439 INFO:tasks.workunit.client.0.smithi067.stdout:0/274: dwrite db/de/f4d [0,4194304] 0 2024-04-22T23:23:24.439 INFO:tasks.workunit.client.0.smithi067.stdout:0/275: stat db 0 2024-04-22T23:23:24.440 INFO:tasks.workunit.client.0.smithi067.stdout:0/276: chown db/d21/d29/f3f 18 1 2024-04-22T23:23:24.443 INFO:tasks.workunit.client.0.smithi067.stdout:9/252: rmdir d5/d2e 39 2024-04-22T23:23:24.448 INFO:tasks.workunit.client.0.smithi067.stdout:9/253: fdatasync d5/dd/d18/d39/d3e/d3f/f52 0 2024-04-22T23:23:24.448 INFO:tasks.workunit.client.0.smithi067.stdout:9/254: readlink d5/d21/d2c/l3a 0 2024-04-22T23:23:24.450 INFO:tasks.workunit.client.0.smithi067.stdout:9/255: write d5/f10 [1744507,30724] 0 2024-04-22T23:23:24.458 INFO:tasks.workunit.client.0.smithi067.stdout:9/256: dwrite d5/d21/f27 [0,4194304] 0 2024-04-22T23:23:24.459 INFO:tasks.workunit.client.0.smithi067.stdout:9/257: fsync d5/d21/d2c/f3b 0 2024-04-22T23:23:24.463 INFO:tasks.workunit.client.0.smithi067.stdout:0/277: dread db/d21/d29/d2c/f41 [0,4194304] 0 2024-04-22T23:23:24.467 INFO:tasks.workunit.client.0.smithi067.stdout:9/258: rename d5/d2e to d5/dd/d18/d39/d49/d54 0 2024-04-22T23:23:24.468 INFO:tasks.workunit.client.0.smithi067.stdout:0/278: creat db/d2e/f5d x:0 0 0 2024-04-22T23:23:24.468 INFO:tasks.workunit.client.0.smithi067.stdout:9/259: write d5/ff [1650062,121908] 0 2024-04-22T23:23:24.469 INFO:tasks.workunit.client.0.smithi067.stdout:0/279: write f0 [3664976,22839] 0 2024-04-22T23:23:24.474 INFO:tasks.workunit.client.0.smithi067.stdout:9/260: dread d5/dd/f32 [0,4194304] 0 2024-04-22T23:23:24.474 INFO:tasks.workunit.client.0.smithi067.stdout:0/280: getdents db/d2e/d46 0 2024-04-22T23:23:24.475 INFO:tasks.workunit.client.0.smithi067.stdout:0/281: chown db/d2e/c4a 858082 1 2024-04-22T23:23:24.476 INFO:tasks.workunit.client.0.smithi067.stdout:0/282: chown db/d2e/f5d 1802043 1 2024-04-22T23:23:24.478 INFO:tasks.workunit.client.0.smithi067.stdout:9/261: creat d5/dd/d18/d39/d49/f55 x:0 0 0 2024-04-22T23:23:24.479 INFO:tasks.workunit.client.0.smithi067.stdout:0/283: creat db/de/d2a/f5e x:0 0 0 2024-04-22T23:23:24.485 INFO:tasks.workunit.client.0.smithi067.stdout:9/262: dwrite d5/dd/d18/d39/d49/d54/f4e [0,4194304] 0 2024-04-22T23:23:24.489 INFO:tasks.workunit.client.0.smithi067.stdout:9/263: mkdir d5/d21/d47/d56 0 2024-04-22T23:23:24.492 INFO:tasks.workunit.client.0.smithi067.stdout:0/284: dread db/de/f1c [0,4194304] 0 2024-04-22T23:23:24.495 INFO:tasks.workunit.client.0.smithi067.stdout:0/285: link la db/d21/d29/d2d/l5f 0 2024-04-22T23:23:24.497 INFO:tasks.workunit.client.0.smithi067.stdout:0/286: mkdir db/d60 0 2024-04-22T23:23:24.498 INFO:tasks.workunit.client.0.smithi067.stdout:0/287: write db/de/d11/d1a/f38 [832789,91722] 0 2024-04-22T23:23:24.502 INFO:tasks.workunit.client.0.smithi067.stdout:0/288: unlink db/d21/d29/c2b 0 2024-04-22T23:23:24.503 INFO:tasks.workunit.client.0.smithi067.stdout:0/289: truncate db/d21/f5a 661885 0 2024-04-22T23:23:24.504 INFO:tasks.workunit.client.0.smithi067.stdout:0/290: rmdir db/de/d2a 39 2024-04-22T23:23:24.511 INFO:tasks.workunit.client.0.smithi067.stdout:9/264: dread d5/f9 [0,4194304] 0 2024-04-22T23:23:24.517 INFO:tasks.workunit.client.0.smithi067.stdout:9/265: dwrite d5/d21/f2b [0,4194304] 0 2024-04-22T23:23:24.523 INFO:tasks.workunit.client.0.smithi067.stdout:9/266: dwrite d5/ff [4194304,4194304] 0 2024-04-22T23:23:24.591 INFO:tasks.workunit.client.0.smithi067.stdout:8/309: sync 2024-04-22T23:23:24.597 INFO:tasks.workunit.client.0.smithi067.stdout:8/310: dwrite d1/d3/d14/f57 [0,4194304] 0 2024-04-22T23:23:24.603 INFO:tasks.workunit.client.0.smithi067.stdout:8/311: dread d1/f18 [0,4194304] 0 2024-04-22T23:23:24.605 INFO:tasks.workunit.client.0.smithi067.stdout:8/312: mkdir d1/d41/d46/d64 0 2024-04-22T23:23:24.620 INFO:tasks.workunit.client.0.smithi067.stdout:8/313: dread d1/f7 [0,4194304] 0 2024-04-22T23:23:24.625 INFO:tasks.workunit.client.0.smithi067.stdout:8/314: dwrite d1/d3/f59 [0,4194304] 0 2024-04-22T23:23:24.627 INFO:tasks.workunit.client.0.smithi067.stdout:8/315: symlink d1/dd/d26/d62/l65 0 2024-04-22T23:23:24.634 INFO:tasks.workunit.client.0.smithi067.stdout:8/316: dwrite d1/d3/d14/f57 [4194304,4194304] 0 2024-04-22T23:23:24.635 INFO:tasks.workunit.client.0.smithi067.stdout:8/317: readlink d1/dd/d26/l2c 0 2024-04-22T23:23:24.637 INFO:tasks.workunit.client.0.smithi067.stdout:8/318: creat d1/d41/d46/f66 x:0 0 0 2024-04-22T23:23:24.645 INFO:tasks.workunit.client.0.smithi067.stdout:8/319: dwrite d1/dd/f2f [0,4194304] 0 2024-04-22T23:23:24.652 INFO:tasks.workunit.client.0.smithi067.stdout:8/320: write d1/f6 [2849817,101794] 0 2024-04-22T23:23:24.654 INFO:tasks.workunit.client.0.smithi067.stdout:8/321: creat d1/d41/f67 x:0 0 0 2024-04-22T23:23:24.659 INFO:tasks.workunit.client.0.smithi067.stdout:8/322: dwrite d1/d3/d14/f34 [0,4194304] 0 2024-04-22T23:23:24.664 INFO:tasks.workunit.client.0.smithi067.stdout:8/323: getdents d1/d41/d46 0 2024-04-22T23:23:24.665 INFO:tasks.workunit.client.0.smithi067.stdout:8/324: mknod d1/d3/d14/c68 0 2024-04-22T23:23:24.669 INFO:tasks.workunit.client.0.smithi067.stdout:8/325: link d1/dd/d26/c25 d1/d3/d10/d3b/c69 0 2024-04-22T23:23:24.670 INFO:tasks.workunit.client.0.smithi067.stdout:8/326: creat d1/d41/f6a x:0 0 0 2024-04-22T23:23:24.675 INFO:tasks.workunit.client.0.smithi067.stdout:8/327: dwrite d1/f6 [0,4194304] 0 2024-04-22T23:23:24.678 INFO:tasks.workunit.client.0.smithi067.stdout:8/328: creat d1/dd/d26/f6b x:0 0 0 2024-04-22T23:23:24.679 INFO:tasks.workunit.client.0.smithi067.stdout:8/329: write d1/d3/d10/f45 [251400,75877] 0 2024-04-22T23:23:24.682 INFO:tasks.workunit.client.0.smithi067.stdout:8/330: symlink d1/d41/d46/l6c 0 2024-04-22T23:23:24.684 INFO:tasks.workunit.client.0.smithi067.stdout:3/323: sync 2024-04-22T23:23:24.684 INFO:tasks.workunit.client.0.smithi067.stdout:1/310: sync 2024-04-22T23:23:24.685 INFO:tasks.workunit.client.0.smithi067.stdout:8/331: unlink d1/dd/l1f 0 2024-04-22T23:23:24.687 INFO:tasks.workunit.client.0.smithi067.stdout:3/324: mkdir d3/d4/d10/d38/d53/d55/d74 0 2024-04-22T23:23:24.689 INFO:tasks.workunit.client.0.smithi067.stdout:8/332: creat d1/dd/d26/d62/f6d x:0 0 0 2024-04-22T23:23:24.691 INFO:tasks.workunit.client.0.smithi067.stdout:3/325: creat d3/d4/d10/d38/d53/d55/f75 x:0 0 0 2024-04-22T23:23:24.692 INFO:tasks.workunit.client.0.smithi067.stdout:8/333: stat d1/d3/l1d 0 2024-04-22T23:23:24.696 INFO:tasks.workunit.client.0.smithi067.stdout:8/334: link d1/d3/d14/f34 d1/d3/d27/f6e 0 2024-04-22T23:23:24.698 INFO:tasks.workunit.client.0.smithi067.stdout:8/335: chown d1/d3/d10/d3b/f54 255081056 1 2024-04-22T23:23:24.699 INFO:tasks.workunit.client.0.smithi067.stdout:8/336: write d1/dd/d26/f55 [473083,29318] 0 2024-04-22T23:23:24.702 INFO:tasks.workunit.client.0.smithi067.stdout:3/326: dread d3/fa [0,4194304] 0 2024-04-22T23:23:24.710 INFO:tasks.workunit.client.0.smithi067.stdout:8/337: fsync d1/d3/d27/f6e 0 2024-04-22T23:23:24.730 INFO:tasks.workunit.client.0.smithi067.stdout:8/338: dread d1/d3/f38 [0,4194304] 0 2024-04-22T23:23:24.731 INFO:tasks.workunit.client.0.smithi067.stdout:8/339: fdatasync d1/d3/d14/f57 0 2024-04-22T23:23:24.733 INFO:tasks.workunit.client.0.smithi067.stdout:8/340: write d1/f6 [549785,111422] 0 2024-04-22T23:23:24.737 INFO:tasks.workunit.client.0.smithi067.stdout:8/341: dread d1/dd/d26/f55 [0,4194304] 0 2024-04-22T23:23:24.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:24 smithi134 ceph-mon[34609]: from='client.14630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:23:24.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:24 smithi134 ceph-mon[34609]: from='client.14634 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:23:24.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:24 smithi134 ceph-mon[34609]: pgmap v267: 65 pgs: 65 active+clean; 578 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.7 MiB/s rd, 62 MiB/s wr, 325 op/s 2024-04-22T23:23:24.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:24 smithi134 ceph-mon[34609]: mgrmap e20: smithi067.whdcsr(active, since 7m), standbys: smithi134.ccxkcj 2024-04-22T23:23:24.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:24 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/1010776181' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:23:24.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:24 smithi134 ceph-mon[34609]: from='client.14642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:23:24.751 INFO:tasks.workunit.client.0.smithi067.stdout:8/342: truncate d1/d3/f59 1768843 0 2024-04-22T23:23:24.752 INFO:tasks.workunit.client.0.smithi067.stdout:8/343: read d1/d3/d14/f32 [7386812,38118] 0 2024-04-22T23:23:24.757 INFO:tasks.workunit.client.0.smithi067.stdout:8/344: rename d1/d3/d14 to d1/d41/d46/d64/d6f 0 2024-04-22T23:23:24.759 INFO:tasks.workunit.client.0.smithi067.stdout:8/345: creat d1/d41/d46/d64/d6f/f70 x:0 0 0 2024-04-22T23:23:24.760 INFO:tasks.workunit.client.0.smithi067.stdout:8/346: readlink d1/d41/d46/l52 0 2024-04-22T23:23:24.764 INFO:tasks.workunit.client.0.smithi067.stdout:8/347: rmdir d1/d41 39 2024-04-22T23:23:24.767 INFO:tasks.workunit.client.0.smithi067.stdout:8/348: dread d1/f6 [0,4194304] 0 2024-04-22T23:23:24.767 INFO:tasks.workunit.client.0.smithi067.stdout:8/349: fsync d1/dd/d26/d62/f6d 0 2024-04-22T23:23:24.771 INFO:tasks.workunit.client.0.smithi067.stdout:8/350: rename d1/c31 to d1/d41/d46/c71 0 2024-04-22T23:23:24.771 INFO:tasks.workunit.client.0.smithi067.stdout:8/351: chown d1/d41/d46/d64/d6f/f57 31859860 1 2024-04-22T23:23:24.773 INFO:tasks.workunit.client.0.smithi067.stdout:8/352: mknod d1/dd/d26/c72 0 2024-04-22T23:23:24.774 INFO:tasks.workunit.client.0.smithi067.stdout:8/353: mknod d1/dd/d26/c73 0 2024-04-22T23:23:24.776 INFO:tasks.workunit.client.0.smithi067.stdout:8/354: creat d1/d41/f74 x:0 0 0 2024-04-22T23:23:24.781 INFO:tasks.workunit.client.0.smithi067.stdout:8/355: dwrite d1/f7 [0,4194304] 0 2024-04-22T23:23:24.784 INFO:tasks.workunit.client.0.smithi067.stdout:8/356: chown d1/dd/c5f 6887463 1 2024-04-22T23:23:24.785 INFO:tasks.workunit.client.0.smithi067.stdout:8/357: dread - d1/d3/f56 zero size 2024-04-22T23:23:24.785 INFO:tasks.workunit.client.0.smithi067.stdout:8/358: readlink d1/d41/d46/l6c 0 2024-04-22T23:23:24.789 INFO:tasks.workunit.client.0.smithi067.stdout:8/359: dread d1/f7 [0,4194304] 0 2024-04-22T23:23:24.790 INFO:tasks.workunit.client.0.smithi067.stdout:8/360: write d1/d3/f56 [557397,71368] 0 2024-04-22T23:23:24.792 INFO:tasks.workunit.client.0.smithi067.stdout:8/361: mkdir d1/dd/d26/d62/d75 0 2024-04-22T23:23:24.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:24 smithi067 ceph-mon[26453]: from='client.14630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:23:24.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:24 smithi067 ceph-mon[26453]: from='client.14634 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:23:24.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:24 smithi067 ceph-mon[26453]: pgmap v267: 65 pgs: 65 active+clean; 578 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.7 MiB/s rd, 62 MiB/s wr, 325 op/s 2024-04-22T23:23:24.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:24 smithi067 ceph-mon[26453]: mgrmap e20: smithi067.whdcsr(active, since 7m), standbys: smithi134.ccxkcj 2024-04-22T23:23:24.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:24 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/1010776181' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:23:24.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:24 smithi067 ceph-mon[26453]: from='client.14642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:23:24.797 INFO:tasks.workunit.client.0.smithi067.stdout:8/362: dwrite d1/d3/f13 [0,4194304] 0 2024-04-22T23:23:24.799 INFO:tasks.workunit.client.0.smithi067.stdout:8/363: fsync d1/d41/d46/d64/d6f/f34 0 2024-04-22T23:23:24.800 INFO:tasks.workunit.client.0.smithi067.stdout:8/364: chown d1/d3/f13 1430036 1 2024-04-22T23:23:24.802 INFO:tasks.workunit.client.0.smithi067.stdout:8/365: fsync d1/dd/d26/f55 0 2024-04-22T23:23:24.804 INFO:tasks.workunit.client.0.smithi067.stdout:8/366: write d1/dd/d26/f6b [691909,97243] 0 2024-04-22T23:23:24.808 INFO:tasks.workunit.client.0.smithi067.stdout:8/367: creat d1/f76 x:0 0 0 2024-04-22T23:23:24.813 INFO:tasks.workunit.client.0.smithi067.stdout:8/368: dwrite d1/d3/d27/f33 [0,4194304] 0 2024-04-22T23:23:24.819 INFO:tasks.workunit.client.0.smithi067.stdout:5/339: sync 2024-04-22T23:23:24.819 INFO:tasks.workunit.client.0.smithi067.stdout:7/243: sync 2024-04-22T23:23:24.819 INFO:tasks.workunit.client.0.smithi067.stdout:6/296: sync 2024-04-22T23:23:24.819 INFO:tasks.workunit.client.0.smithi067.stdout:0/291: sync 2024-04-22T23:23:24.819 INFO:tasks.workunit.client.0.smithi067.stdout:4/417: sync 2024-04-22T23:23:24.819 INFO:tasks.workunit.client.0.smithi067.stdout:2/372: sync 2024-04-22T23:23:24.819 INFO:tasks.workunit.client.0.smithi067.stdout:9/267: sync 2024-04-22T23:23:24.822 INFO:tasks.workunit.client.0.smithi067.stdout:8/369: getdents d1 0 2024-04-22T23:23:24.823 INFO:tasks.workunit.client.0.smithi067.stdout:5/340: write d5/d3a/f3c [532000,107938] 0 2024-04-22T23:23:24.824 INFO:tasks.workunit.client.0.smithi067.stdout:4/418: mknod d12/d42/c99 0 2024-04-22T23:23:24.826 INFO:tasks.workunit.client.0.smithi067.stdout:7/244: write d10/d18/d35/f44 [684561,9117] 0 2024-04-22T23:23:24.826 INFO:tasks.workunit.client.0.smithi067.stdout:8/370: stat d1/d3/d10/f3d 0 2024-04-22T23:23:24.826 INFO:tasks.workunit.client.0.smithi067.stdout:9/268: mkdir d5/d21/d47/d57 0 2024-04-22T23:23:24.828 INFO:tasks.workunit.client.0.smithi067.stdout:0/292: mknod db/de/c61 0 2024-04-22T23:23:24.829 INFO:tasks.workunit.client.0.smithi067.stdout:5/341: mknod d5/db/d35/c7a 0 2024-04-22T23:23:24.830 INFO:tasks.workunit.client.0.smithi067.stdout:9/269: chown d5/c48 27913 1 2024-04-22T23:23:24.832 INFO:tasks.workunit.client.0.smithi067.stdout:6/297: dwrite d17/f30 [0,4194304] 0 2024-04-22T23:23:24.833 INFO:tasks.workunit.client.0.smithi067.stdout:2/373: rename d1/d42/d4a to d1/d10/d1c/d6d 0 2024-04-22T23:23:24.834 INFO:tasks.workunit.client.0.smithi067.stdout:6/298: fdatasync d17/f2b 0 2024-04-22T23:23:24.836 INFO:tasks.workunit.client.0.smithi067.stdout:2/374: readlink d1/d12/l64 0 2024-04-22T23:23:24.837 INFO:tasks.workunit.client.0.smithi067.stdout:2/375: write d1/d4f/f2a [851355,119981] 0 2024-04-22T23:23:24.837 INFO:tasks.workunit.client.0.smithi067.stdout:7/245: creat d10/d47/d49/f54 x:0 0 0 2024-04-22T23:23:24.841 INFO:tasks.workunit.client.0.smithi067.stdout:4/419: creat d12/d42/d4b/d77/d8c/f9a x:0 0 0 2024-04-22T23:23:24.843 INFO:tasks.workunit.client.0.smithi067.stdout:5/342: mknod d5/d28/d36/d40/d57/d5f/c7b 0 2024-04-22T23:23:24.845 INFO:tasks.workunit.client.0.smithi067.stdout:9/270: symlink d5/dd/d18/d39/d49/l58 0 2024-04-22T23:23:24.845 INFO:tasks.workunit.client.0.smithi067.stdout:6/299: fsync d17/d39/d3a/f51 0 2024-04-22T23:23:24.845 INFO:tasks.workunit.client.0.smithi067.stdout:8/371: dwrite d1/d41/d46/d64/d6f/f32 [0,4194304] 0 2024-04-22T23:23:24.847 INFO:tasks.workunit.client.0.smithi067.stdout:8/372: chown d1/d3/d27 3495 1 2024-04-22T23:23:24.848 INFO:tasks.workunit.client.0.smithi067.stdout:8/373: write d1/d3/d10/f45 [1365487,100591] 0 2024-04-22T23:23:24.850 INFO:tasks.workunit.client.0.smithi067.stdout:7/246: unlink d10/f13 0 2024-04-22T23:23:24.851 INFO:tasks.workunit.client.0.smithi067.stdout:8/374: write d1/d3/f1b [3629571,100991] 0 2024-04-22T23:23:24.852 INFO:tasks.workunit.client.0.smithi067.stdout:4/420: mkdir d12/d2e/d9b 0 2024-04-22T23:23:24.854 INFO:tasks.workunit.client.0.smithi067.stdout:5/343: symlink d5/db/d12/d44/d47/l7c 0 2024-04-22T23:23:24.855 INFO:tasks.workunit.client.0.smithi067.stdout:4/421: readlink d12/d42/l71 0 2024-04-22T23:23:24.855 INFO:tasks.workunit.client.0.smithi067.stdout:0/293: creat db/d21/d29/f62 x:0 0 0 2024-04-22T23:23:24.856 INFO:tasks.workunit.client.0.smithi067.stdout:7/247: fdatasync d10/d14/d1d/d26/d2c/d41/f4f 0 2024-04-22T23:23:24.856 INFO:tasks.workunit.client.0.smithi067.stdout:6/300: mkdir d17/d39/d3e/d46/d62 0 2024-04-22T23:23:24.859 INFO:tasks.workunit.client.0.smithi067.stdout:9/271: dwrite d5/dd/d18/d39/d3e/f53 [0,4194304] 0 2024-04-22T23:23:24.859 INFO:tasks.workunit.client.0.smithi067.stdout:6/301: chown l14 410880349 1 2024-04-22T23:23:24.860 INFO:tasks.workunit.client.0.smithi067.stdout:2/376: creat d1/d10/d33/d46/f6e x:0 0 0 2024-04-22T23:23:24.864 INFO:tasks.workunit.client.0.smithi067.stdout:4/422: symlink d12/d5f/l9c 0 2024-04-22T23:23:24.864 INFO:tasks.workunit.client.0.smithi067.stdout:6/302: write d17/d39/d3a/f3b [3969147,76816] 0 2024-04-22T23:23:24.866 INFO:tasks.workunit.client.0.smithi067.stdout:0/294: creat db/d2e/d31/f63 x:0 0 0 2024-04-22T23:23:24.867 INFO:tasks.workunit.client.0.smithi067.stdout:7/248: dread d10/f12 [0,4194304] 0 2024-04-22T23:23:24.869 INFO:tasks.workunit.client.0.smithi067.stdout:5/344: dread d5/d28/d36/d40/f16 [4194304,4194304] 0 2024-04-22T23:23:24.869 INFO:tasks.workunit.client.0.smithi067.stdout:6/303: write d17/d39/d3a/f50 [3674893,25698] 0 2024-04-22T23:23:24.870 INFO:tasks.workunit.client.0.smithi067.stdout:7/249: fdatasync d10/d47/d49/f54 0 2024-04-22T23:23:24.874 INFO:tasks.workunit.client.0.smithi067.stdout:7/250: chown d10/d14/l16 692 1 2024-04-22T23:23:24.874 INFO:tasks.workunit.client.0.smithi067.stdout:2/377: write d1/d10/f2e [3750482,130673] 0 2024-04-22T23:23:24.874 INFO:tasks.workunit.client.0.smithi067.stdout:9/272: rename d5/d21/d2c/c4c to d5/dd/d18/d3c/c59 0 2024-04-22T23:23:24.875 INFO:tasks.workunit.client.0.smithi067.stdout:0/295: creat db/de/d11/d1a/f64 x:0 0 0 2024-04-22T23:23:24.876 INFO:tasks.workunit.client.0.smithi067.stdout:0/296: write db/d2e/d46/f53 [1011215,108550] 0 2024-04-22T23:23:24.877 INFO:tasks.workunit.client.0.smithi067.stdout:6/304: symlink d17/d39/d53/l63 0 2024-04-22T23:23:24.881 INFO:tasks.workunit.client.0.smithi067.stdout:6/305: creat d17/f64 x:0 0 0 2024-04-22T23:23:24.883 INFO:tasks.workunit.client.0.smithi067.stdout:0/297: read db/d21/d29/d2c/f3d [1256754,99164] 0 2024-04-22T23:23:24.883 INFO:tasks.workunit.client.0.smithi067.stdout:2/378: mkdir d1/d19/d6f 0 2024-04-22T23:23:24.888 INFO:tasks.workunit.client.0.smithi067.stdout:5/345: getdents d5/d28/d36/d40 0 2024-04-22T23:23:24.888 INFO:tasks.workunit.client.0.smithi067.stdout:5/346: fsync d5/f31 0 2024-04-22T23:23:24.889 INFO:tasks.workunit.client.0.smithi067.stdout:5/347: chown d5/db/d35 4 1 2024-04-22T23:23:24.891 INFO:tasks.workunit.client.0.smithi067.stdout:0/298: symlink db/d21/d29/d2c/l65 0 2024-04-22T23:23:24.896 INFO:tasks.workunit.client.0.smithi067.stdout:6/306: mknod d17/d39/d3e/d46/d62/c65 0 2024-04-22T23:23:24.898 INFO:tasks.workunit.client.0.smithi067.stdout:6/307: chown d17/l38 30035696 1 2024-04-22T23:23:24.899 INFO:tasks.workunit.client.0.smithi067.stdout:0/299: dwrite db/de/d11/f3c [0,4194304] 0 2024-04-22T23:23:24.901 INFO:tasks.workunit.client.0.smithi067.stdout:6/308: write d17/f24 [4055298,5836] 0 2024-04-22T23:23:24.902 INFO:tasks.workunit.client.0.smithi067.stdout:6/309: write d17/f30 [24979,5827] 0 2024-04-22T23:23:24.903 INFO:tasks.workunit.client.0.smithi067.stdout:9/273: link d5/l15 d5/l5a 0 2024-04-22T23:23:24.904 INFO:tasks.workunit.client.0.smithi067.stdout:6/310: dread - d17/d39/d53/f59 zero size 2024-04-22T23:23:24.906 INFO:tasks.workunit.client.0.smithi067.stdout:6/311: write d17/f35 [907907,61012] 0 2024-04-22T23:23:24.913 INFO:tasks.workunit.client.0.smithi067.stdout:2/379: link d1/f2d d1/d10/d41/f70 0 2024-04-22T23:23:24.915 INFO:tasks.workunit.client.0.smithi067.stdout:0/300: mknod db/de/d2a/d48/c66 0 2024-04-22T23:23:24.915 INFO:tasks.workunit.client.0.smithi067.stdout:6/312: dwrite d17/d39/d3a/f4f [0,4194304] 0 2024-04-22T23:23:24.916 INFO:tasks.workunit.client.0.smithi067.stdout:5/348: write d5/d28/d36/d40/d57/f72 [448236,23601] 0 2024-04-22T23:23:24.917 INFO:tasks.workunit.client.0.smithi067.stdout:6/313: stat c1 0 2024-04-22T23:23:24.919 INFO:tasks.workunit.client.0.smithi067.stdout:2/380: getdents d1/d42/d6b 0 2024-04-22T23:23:24.920 INFO:tasks.workunit.client.0.smithi067.stdout:5/349: mknod d5/d28/c7d 0 2024-04-22T23:23:24.925 INFO:tasks.workunit.client.0.smithi067.stdout:0/301: dwrite db/de/d2a/f45 [0,4194304] 0 2024-04-22T23:23:24.926 INFO:tasks.workunit.client.0.smithi067.stdout:2/381: symlink d1/d10/d1c/d4d/l71 0 2024-04-22T23:23:24.928 INFO:tasks.workunit.client.0.smithi067.stdout:5/350: dread d5/db/d12/d44/f56 [0,4194304] 0 2024-04-22T23:23:24.932 INFO:tasks.workunit.client.0.smithi067.stdout:5/351: chown d5/d28/d36/d40/d57 2 1 2024-04-22T23:23:24.934 INFO:tasks.workunit.client.0.smithi067.stdout:5/352: unlink d5/f71 0 2024-04-22T23:23:24.935 INFO:tasks.workunit.client.0.smithi067.stdout:6/314: dread d17/f20 [0,4194304] 0 2024-04-22T23:23:24.939 INFO:tasks.workunit.client.0.smithi067.stdout:6/315: unlink d17/d39/d3e/c41 0 2024-04-22T23:23:24.941 INFO:tasks.workunit.client.0.smithi067.stdout:6/316: truncate d17/d39/d3e/d46/f5f 273145 0 2024-04-22T23:23:24.942 INFO:tasks.workunit.client.0.smithi067.stdout:5/353: link d5/db/c73 d5/d28/d36/d40/d57/d5f/c7e 0 2024-04-22T23:23:24.943 INFO:tasks.workunit.client.0.smithi067.stdout:6/317: fdatasync d17/f20 0 2024-04-22T23:23:24.944 INFO:tasks.workunit.client.0.smithi067.stdout:6/318: write d17/d39/d3a/f51 [1684607,58625] 0 2024-04-22T23:23:24.946 INFO:tasks.workunit.client.0.smithi067.stdout:6/319: read d17/d39/d3a/f4f [4115787,13230] 0 2024-04-22T23:23:24.947 INFO:tasks.workunit.client.0.smithi067.stdout:5/354: mknod d5/d1a/c7f 0 2024-04-22T23:23:24.952 INFO:tasks.workunit.client.0.smithi067.stdout:0/302: write db/fd [1285270,12598] 0 2024-04-22T23:23:24.953 INFO:tasks.workunit.client.0.smithi067.stdout:6/320: mknod d17/d39/d3e/c66 0 2024-04-22T23:23:24.953 INFO:tasks.workunit.client.0.smithi067.stdout:2/382: dwrite d1/d19/f21 [0,4194304] 0 2024-04-22T23:23:24.954 INFO:tasks.workunit.client.0.smithi067.stdout:6/321: chown d17/f32 189 1 2024-04-22T23:23:24.958 INFO:tasks.workunit.client.0.smithi067.stdout:0/303: symlink db/de/d2a/l67 0 2024-04-22T23:23:24.962 INFO:tasks.workunit.client.0.smithi067.stdout:5/355: rename d5/db/f34 to d5/d28/d36/f80 0 2024-04-22T23:23:24.963 INFO:tasks.workunit.client.0.smithi067.stdout:5/356: chown d5/db/d35/f43 45864405 1 2024-04-22T23:23:24.968 INFO:tasks.workunit.client.0.smithi067.stdout:0/304: symlink db/d2e/d31/d56/l68 0 2024-04-22T23:23:24.968 INFO:tasks.workunit.client.0.smithi067.stdout:2/383: dread d1/f3 [4194304,4194304] 0 2024-04-22T23:23:24.969 INFO:tasks.workunit.client.0.smithi067.stdout:2/384: stat d1/f8 0 2024-04-22T23:23:24.969 INFO:tasks.workunit.client.0.smithi067.stdout:5/357: fsync d5/db/d12/f37 0 2024-04-22T23:23:24.978 INFO:tasks.workunit.client.0.smithi067.stdout:0/305: dwrite db/de/f1b [0,4194304] 0 2024-04-22T23:23:24.978 INFO:tasks.workunit.client.0.smithi067.stdout:2/385: dread d1/d10/d33/f3e [0,4194304] 0 2024-04-22T23:23:24.978 INFO:tasks.workunit.client.0.smithi067.stdout:6/322: truncate d17/f26 2044851 0 2024-04-22T23:23:24.979 INFO:tasks.workunit.client.0.smithi067.stdout:6/323: write d17/f2b [45372,20816] 0 2024-04-22T23:23:24.986 INFO:tasks.workunit.client.0.smithi067.stdout:2/386: rename d1/d19/d31 to d1/d10/d72 0 2024-04-22T23:23:24.988 INFO:tasks.workunit.client.0.smithi067.stdout:6/324: dread d17/f42 [0,4194304] 0 2024-04-22T23:23:24.988 INFO:tasks.workunit.client.0.smithi067.stdout:2/387: read - d1/d42/f57 zero size 2024-04-22T23:23:24.989 INFO:tasks.workunit.client.0.smithi067.stdout:0/306: dwrite db/d21/d29/d2d/f49 [0,4194304] 0 2024-04-22T23:23:24.991 INFO:tasks.workunit.client.0.smithi067.stdout:1/311: sync 2024-04-22T23:23:24.991 INFO:tasks.workunit.client.0.smithi067.stdout:0/307: chown db/d2e 405665457 1 2024-04-22T23:23:24.992 INFO:tasks.workunit.client.0.smithi067.stdout:5/358: dwrite d5/d28/d36/d40/f70 [0,4194304] 0 2024-04-22T23:23:24.994 INFO:tasks.workunit.client.0.smithi067.stdout:0/308: chown db/d2e 84026 1 2024-04-22T23:23:24.995 INFO:tasks.workunit.client.0.smithi067.stdout:6/325: rename d17/l57 to d17/d39/l67 0 2024-04-22T23:23:24.995 INFO:tasks.workunit.client.0.smithi067.stdout:0/309: write db/d21/f4c [183360,77673] 0 2024-04-22T23:23:24.996 INFO:tasks.workunit.client.0.smithi067.stdout:6/326: chown d17/l52 114 1 2024-04-22T23:23:24.996 INFO:tasks.workunit.client.0.smithi067.stdout:2/388: symlink d1/d10/d33/l73 0 2024-04-22T23:23:24.998 INFO:tasks.workunit.client.0.smithi067.stdout:6/327: dread - d17/f64 zero size 2024-04-22T23:23:24.998 INFO:tasks.workunit.client.0.smithi067.stdout:2/389: fsync d1/d42/f4b 0 2024-04-22T23:23:24.998 INFO:tasks.workunit.client.0.smithi067.stdout:5/359: creat d5/db/d12/d44/f81 x:0 0 0 2024-04-22T23:23:25.000 INFO:tasks.workunit.client.0.smithi067.stdout:5/360: read - d5/d28/f69 zero size 2024-04-22T23:23:25.003 INFO:tasks.workunit.client.0.smithi067.stdout:2/390: creat d1/d10/d72/d53/f74 x:0 0 0 2024-04-22T23:23:25.006 INFO:tasks.workunit.client.0.smithi067.stdout:6/328: rename d17/f1e to d17/d39/d3e/d46/f68 0 2024-04-22T23:23:25.006 INFO:tasks.workunit.client.0.smithi067.stdout:1/312: link d8/d9/l16 d8/d4a/l68 0 2024-04-22T23:23:25.007 INFO:tasks.workunit.client.0.smithi067.stdout:0/310: dwrite db/d2e/d31/f3a [0,4194304] 0 2024-04-22T23:23:25.008 INFO:tasks.workunit.client.0.smithi067.stdout:5/361: creat d5/d28/d36/d40/d57/d61/f82 x:0 0 0 2024-04-22T23:23:25.014 INFO:tasks.workunit.client.0.smithi067.stdout:1/313: truncate d8/f11 728885 0 2024-04-22T23:23:25.017 INFO:tasks.workunit.client.0.smithi067.stdout:1/314: write d8/d9/fd [47461,76725] 0 2024-04-22T23:23:25.017 INFO:tasks.workunit.client.0.smithi067.stdout:2/391: getdents d1/d10/d33 0 2024-04-22T23:23:25.018 INFO:tasks.workunit.client.0.smithi067.stdout:6/329: dwrite d17/d39/d3a/f51 [4194304,4194304] 0 2024-04-22T23:23:25.019 INFO:tasks.workunit.client.0.smithi067.stdout:5/362: creat d5/db/d12/d44/f83 x:0 0 0 2024-04-22T23:23:25.020 INFO:tasks.workunit.client.0.smithi067.stdout:0/311: mknod db/d21/d29/c69 0 2024-04-22T23:23:25.022 INFO:tasks.workunit.client.0.smithi067.stdout:0/312: stat db/de/f12 0 2024-04-22T23:23:25.024 INFO:tasks.workunit.client.0.smithi067.stdout:6/330: truncate d17/d39/d53/f59 810829 0 2024-04-22T23:23:25.024 INFO:tasks.workunit.client.0.smithi067.stdout:1/315: write d8/f1d [916827,27655] 0 2024-04-22T23:23:25.025 INFO:tasks.workunit.client.0.smithi067.stdout:1/316: chown d8/d24 54851 1 2024-04-22T23:23:25.026 INFO:tasks.workunit.client.0.smithi067.stdout:5/363: mknod d5/db/c84 0 2024-04-22T23:23:25.026 INFO:tasks.workunit.client.0.smithi067.stdout:1/317: readlink d8/d9/d2d/d4d/l14 0 2024-04-22T23:23:25.027 INFO:tasks.workunit.client.0.smithi067.stdout:2/392: dread d1/d10/d1c/d6d/f4c [0,4194304] 0 2024-04-22T23:23:25.031 INFO:tasks.workunit.client.0.smithi067.stdout:5/364: rmdir d5/d3a/d4d 39 2024-04-22T23:23:25.034 INFO:tasks.workunit.client.0.smithi067.stdout:2/393: creat d1/d10/d72/d53/f75 x:0 0 0 2024-04-22T23:23:25.035 INFO:tasks.workunit.client.0.smithi067.stdout:2/394: readlink d1/d10/d1c/d4d/l71 0 2024-04-22T23:23:25.035 INFO:tasks.workunit.client.0.smithi067.stdout:0/313: rename db/de/d2a/f45 to db/f6a 0 2024-04-22T23:23:25.036 INFO:tasks.workunit.client.0.smithi067.stdout:5/365: creat d5/d28/d36/d40/f85 x:0 0 0 2024-04-22T23:23:25.040 INFO:tasks.workunit.client.0.smithi067.stdout:0/314: fdatasync db/d21/d29/d2c/f3d 0 2024-04-22T23:23:25.042 INFO:tasks.workunit.client.0.smithi067.stdout:6/331: truncate d17/f33 414736 0 2024-04-22T23:23:25.043 INFO:tasks.workunit.client.0.smithi067.stdout:6/332: read d17/f30 [2949922,11970] 0 2024-04-22T23:23:25.043 INFO:tasks.workunit.client.0.smithi067.stdout:6/333: stat d17/f20 0 2024-04-22T23:23:25.044 INFO:tasks.workunit.client.0.smithi067.stdout:1/318: rename d8/d9/f61 to d8/d12/d40/f69 0 2024-04-22T23:23:25.046 INFO:tasks.workunit.client.0.smithi067.stdout:0/315: mkdir db/d2e/d31/d56/d6b 0 2024-04-22T23:23:25.048 INFO:tasks.workunit.client.0.smithi067.stdout:0/316: write db/de/d11/d1a/f64 [304050,55853] 0 2024-04-22T23:23:25.048 INFO:tasks.workunit.client.0.smithi067.stdout:2/395: creat d1/d19/f76 x:0 0 0 2024-04-22T23:23:25.050 INFO:tasks.workunit.client.0.smithi067.stdout:2/396: truncate d1/d42/f57 629381 0 2024-04-22T23:23:25.050 INFO:tasks.workunit.client.0.smithi067.stdout:6/334: write d17/f25 [202308,25316] 0 2024-04-22T23:23:25.051 INFO:tasks.workunit.client.0.smithi067.stdout:1/319: symlink d8/d24/d34/l6a 0 2024-04-22T23:23:25.053 INFO:tasks.workunit.client.0.smithi067.stdout:5/366: creat d5/d28/d36/f86 x:0 0 0 2024-04-22T23:23:25.053 INFO:tasks.workunit.client.0.smithi067.stdout:1/320: read d8/d12/f54 [143841,128841] 0 2024-04-22T23:23:25.054 INFO:tasks.workunit.client.0.smithi067.stdout:2/397: symlink d1/d10/d72/l77 0 2024-04-22T23:23:25.054 INFO:tasks.workunit.client.0.smithi067.stdout:2/398: chown d1/d10 1 1 2024-04-22T23:23:25.055 INFO:tasks.workunit.client.0.smithi067.stdout:6/335: creat d17/d39/f69 x:0 0 0 2024-04-22T23:23:25.056 INFO:tasks.workunit.client.0.smithi067.stdout:0/317: mknod db/de/c6c 0 2024-04-22T23:23:25.057 INFO:tasks.workunit.client.0.smithi067.stdout:5/367: mkdir d5/d3e/d87 0 2024-04-22T23:23:25.058 INFO:tasks.workunit.client.0.smithi067.stdout:0/318: write db/de/f1b [1060197,18501] 0 2024-04-22T23:23:25.059 INFO:tasks.workunit.client.0.smithi067.stdout:6/336: dread d17/d39/f49 [0,4194304] 0 2024-04-22T23:23:25.059 INFO:tasks.workunit.client.0.smithi067.stdout:1/321: creat d8/d9/d2f/d33/d66/f6b x:0 0 0 2024-04-22T23:23:25.062 INFO:tasks.workunit.client.0.smithi067.stdout:1/322: read - d8/d9/d2d/d4d/f23 zero size 2024-04-22T23:23:25.064 INFO:tasks.workunit.client.0.smithi067.stdout:6/337: creat d17/d39/d53/f6a x:0 0 0 2024-04-22T23:23:25.065 INFO:tasks.workunit.client.0.smithi067.stdout:2/399: dwrite d1/d42/f4b [0,4194304] 0 2024-04-22T23:23:25.069 INFO:tasks.workunit.client.0.smithi067.stdout:6/338: mknod d17/d39/c6b 0 2024-04-22T23:23:25.071 INFO:tasks.workunit.client.0.smithi067.stdout:1/323: creat d8/d9/d2f/d45/f6c x:0 0 0 2024-04-22T23:23:25.074 INFO:tasks.workunit.client.0.smithi067.stdout:2/400: link d1/d10/d41/l54 d1/d42/d5f/l78 0 2024-04-22T23:23:25.079 INFO:tasks.workunit.client.0.smithi067.stdout:2/401: chown d1/d10/f52 15349 1 2024-04-22T23:23:25.079 INFO:tasks.workunit.client.0.smithi067.stdout:6/339: link d17/l44 d17/d39/d3a/l6c 0 2024-04-22T23:23:25.079 INFO:tasks.workunit.client.0.smithi067.stdout:2/402: chown d1/d10/d1c/f5e 0 1 2024-04-22T23:23:25.079 INFO:tasks.workunit.client.0.smithi067.stdout:6/340: creat d17/d39/d3a/f6d x:0 0 0 2024-04-22T23:23:25.081 INFO:tasks.workunit.client.0.smithi067.stdout:2/403: mknod d1/c79 0 2024-04-22T23:23:25.084 INFO:tasks.workunit.client.0.smithi067.stdout:2/404: write d1/d10/d1c/f5d [957868,88178] 0 2024-04-22T23:23:25.088 INFO:tasks.workunit.client.0.smithi067.stdout:6/341: dwrite d17/d39/d3e/d46/f54 [0,4194304] 0 2024-04-22T23:23:25.090 INFO:tasks.workunit.client.0.smithi067.stdout:2/405: mkdir d1/d19/d6f/d7a 0 2024-04-22T23:23:25.090 INFO:tasks.workunit.client.0.smithi067.stdout:5/368: dwrite d5/f11 [0,4194304] 0 2024-04-22T23:23:25.091 INFO:tasks.workunit.client.0.smithi067.stdout:2/406: stat d1/d10/d1c/f3a 0 2024-04-22T23:23:25.097 INFO:tasks.workunit.client.0.smithi067.stdout:2/407: mkdir d1/d10/d72/d68/d7b 0 2024-04-22T23:23:25.100 INFO:tasks.workunit.client.0.smithi067.stdout:5/369: dwrite d5/d3a/f3c [0,4194304] 0 2024-04-22T23:23:25.101 INFO:tasks.workunit.client.0.smithi067.stdout:6/342: dread d17/d39/d53/f55 [0,4194304] 0 2024-04-22T23:23:25.103 INFO:tasks.workunit.client.0.smithi067.stdout:5/370: fsync d5/d28/f69 0 2024-04-22T23:23:25.106 INFO:tasks.workunit.client.0.smithi067.stdout:6/343: mknod d17/d39/d3e/d46/d62/c6e 0 2024-04-22T23:23:25.109 INFO:tasks.workunit.client.0.smithi067.stdout:2/408: dwrite d1/f35 [4194304,4194304] 0 2024-04-22T23:23:25.109 INFO:tasks.workunit.client.0.smithi067.stdout:5/371: write d5/d3e/f77 [453616,13717] 0 2024-04-22T23:23:25.111 INFO:tasks.workunit.client.0.smithi067.stdout:2/409: write d1/d12/f4e [966309,122459] 0 2024-04-22T23:23:25.113 INFO:tasks.workunit.client.0.smithi067.stdout:6/344: creat d17/d39/d3e/f6f x:0 0 0 2024-04-22T23:23:25.115 INFO:tasks.workunit.client.0.smithi067.stdout:6/345: chown d17/f42 143 1 2024-04-22T23:23:25.115 INFO:tasks.workunit.client.0.smithi067.stdout:5/372: chown d5/d3a/d4d/l67 1 1 2024-04-22T23:23:25.119 INFO:tasks.workunit.client.0.smithi067.stdout:5/373: rename d5/d28/d36 to d5/d28/d36/d88 22 2024-04-22T23:23:25.120 INFO:tasks.workunit.client.0.smithi067.stdout:2/410: dwrite d1/d10/d72/d53/f75 [0,4194304] 0 2024-04-22T23:23:25.122 INFO:tasks.workunit.client.0.smithi067.stdout:5/374: mkdir d5/d28/d36/d40/d57/d61/d89 0 2024-04-22T23:23:25.124 INFO:tasks.workunit.client.0.smithi067.stdout:2/411: symlink d1/d10/l7c 0 2024-04-22T23:23:25.127 INFO:tasks.workunit.client.0.smithi067.stdout:5/375: unlink d5/d28/d36/d40/f16 0 2024-04-22T23:23:25.129 INFO:tasks.workunit.client.0.smithi067.stdout:5/376: mkdir d5/db/d12/d8a 0 2024-04-22T23:23:25.132 INFO:tasks.workunit.client.0.smithi067.stdout:5/377: mkdir d5/d1a/d8b 0 2024-04-22T23:23:25.132 INFO:tasks.workunit.client.0.smithi067.stdout:2/412: dwrite d1/d10/d72/d53/f74 [0,4194304] 0 2024-04-22T23:23:25.133 INFO:tasks.workunit.client.0.smithi067.stdout:6/346: write d17/f33 [1161802,71523] 0 2024-04-22T23:23:25.135 INFO:tasks.workunit.client.0.smithi067.stdout:6/347: chown d17/d39/d53/f60 0 1 2024-04-22T23:23:25.138 INFO:tasks.workunit.client.0.smithi067.stdout:2/413: creat d1/d10/d33/d46/f7d x:0 0 0 2024-04-22T23:23:25.140 INFO:tasks.workunit.client.0.smithi067.stdout:5/378: truncate d5/db/d12/f22 1293693 0 2024-04-22T23:23:25.141 INFO:tasks.workunit.client.0.smithi067.stdout:6/348: dread d17/f33 [0,4194304] 0 2024-04-22T23:23:25.143 INFO:tasks.workunit.client.0.smithi067.stdout:2/414: creat d1/d42/d61/f7e x:0 0 0 2024-04-22T23:23:25.144 INFO:tasks.workunit.client.0.smithi067.stdout:5/379: creat d5/db/d12/d44/d47/f8c x:0 0 0 2024-04-22T23:23:25.149 INFO:tasks.workunit.client.0.smithi067.stdout:6/349: dwrite d17/f64 [0,4194304] 0 2024-04-22T23:23:25.149 INFO:tasks.workunit.client.0.smithi067.stdout:2/415: rmdir d1/d19/d6f/d7a 0 2024-04-22T23:23:25.150 INFO:tasks.workunit.client.0.smithi067.stdout:2/416: stat d1/d12/f15 0 2024-04-22T23:23:25.151 INFO:tasks.workunit.client.0.smithi067.stdout:6/350: fdatasync d17/f34 0 2024-04-22T23:23:25.159 INFO:tasks.workunit.client.0.smithi067.stdout:6/351: link c1 d17/d39/d3a/c70 0 2024-04-22T23:23:25.161 INFO:tasks.workunit.client.0.smithi067.stdout:6/352: truncate d17/f21 4727495 0 2024-04-22T23:23:25.161 INFO:tasks.workunit.client.0.smithi067.stdout:2/417: write d1/d10/d41/f70 [1143268,98782] 0 2024-04-22T23:23:25.163 INFO:tasks.workunit.client.0.smithi067.stdout:6/353: read - d17/d39/d3e/f6f zero size 2024-04-22T23:23:25.163 INFO:tasks.workunit.client.0.smithi067.stdout:5/380: dread d5/d3a/f58 [0,4194304] 0 2024-04-22T23:23:25.169 INFO:tasks.workunit.client.0.smithi067.stdout:6/354: mknod d17/d39/c71 0 2024-04-22T23:23:25.172 INFO:tasks.workunit.client.0.smithi067.stdout:6/355: rename d17/d39/d3e/d46/f54 to d17/d39/d3e/d46/d62/f72 0 2024-04-22T23:23:25.177 INFO:tasks.workunit.client.0.smithi067.stdout:6/356: link d17/d39/d3a/f5a d17/d39/d3e/f73 0 2024-04-22T23:23:25.179 INFO:tasks.workunit.client.0.smithi067.stdout:6/357: fsync d17/f26 0 2024-04-22T23:23:25.180 INFO:tasks.workunit.client.0.smithi067.stdout:2/418: truncate d1/d42/f56 263332 0 2024-04-22T23:23:25.186 INFO:tasks.workunit.client.0.smithi067.stdout:6/358: rmdir d17/d39/d3a 39 2024-04-22T23:23:25.189 INFO:tasks.workunit.client.0.smithi067.stdout:6/359: creat d17/d39/d53/f74 x:0 0 0 2024-04-22T23:23:25.196 INFO:tasks.workunit.client.0.smithi067.stdout:2/419: dwrite d1/d10/f24 [0,4194304] 0 2024-04-22T23:23:25.198 INFO:tasks.workunit.client.0.smithi067.stdout:6/360: dwrite d17/f64 [4194304,4194304] 0 2024-04-22T23:23:25.201 INFO:tasks.workunit.client.0.smithi067.stdout:2/420: symlink d1/d10/d72/l7f 0 2024-04-22T23:23:25.203 INFO:tasks.workunit.client.0.smithi067.stdout:6/361: rename d17/d39/d3e/f73 to d17/d39/d3e/d46/f75 0 2024-04-22T23:23:25.203 INFO:tasks.workunit.client.0.smithi067.stdout:2/421: rmdir d1/d42/d5f 39 2024-04-22T23:23:25.204 INFO:tasks.workunit.client.0.smithi067.stdout:6/362: truncate d17/f42 2083780 0 2024-04-22T23:23:25.204 INFO:tasks.workunit.client.0.smithi067.stdout:2/422: stat d1/d10/l7c 0 2024-04-22T23:23:25.208 INFO:tasks.workunit.client.0.smithi067.stdout:6/363: mknod d17/d39/d3e/c76 0 2024-04-22T23:23:25.210 INFO:tasks.workunit.client.0.smithi067.stdout:6/364: write d17/f35 [403208,46092] 0 2024-04-22T23:23:25.214 INFO:tasks.workunit.client.0.smithi067.stdout:2/423: dwrite d1/d10/d1c/d6d/f4c [0,4194304] 0 2024-04-22T23:23:25.218 INFO:tasks.workunit.client.0.smithi067.stdout:2/424: truncate d1/d10/d1c/f5e 176131 0 2024-04-22T23:23:25.219 INFO:tasks.workunit.client.0.smithi067.stdout:2/425: fsync d1/d4f/f2a 0 2024-04-22T23:23:25.219 INFO:tasks.workunit.client.0.smithi067.stdout:6/365: dwrite d17/d39/d3a/f40 [0,4194304] 0 2024-04-22T23:23:25.224 INFO:tasks.workunit.client.0.smithi067.stdout:2/426: link d1/l27 d1/d4f/l80 0 2024-04-22T23:23:25.225 INFO:tasks.workunit.client.0.smithi067.stdout:2/427: chown d1/d10/d72/d53/f58 30623285 1 2024-04-22T23:23:25.226 INFO:tasks.workunit.client.0.smithi067.stdout:6/366: rename d17/d39/c5c to d17/d39/c77 0 2024-04-22T23:23:25.228 INFO:tasks.workunit.client.0.smithi067.stdout:2/428: read d1/d42/f56 [142372,100629] 0 2024-04-22T23:23:25.230 INFO:tasks.workunit.client.0.smithi067.stdout:2/429: fdatasync d1/d12/f15 0 2024-04-22T23:23:25.247 INFO:tasks.workunit.client.0.smithi067.stdout:6/367: fdatasync d17/f64 0 2024-04-22T23:23:25.247 INFO:tasks.workunit.client.0.smithi067.stdout:6/368: chown d17/f32 12275 1 2024-04-22T23:23:25.248 INFO:tasks.workunit.client.0.smithi067.stdout:6/369: fdatasync f13 0 2024-04-22T23:23:25.249 INFO:tasks.workunit.client.0.smithi067.stdout:6/370: write d17/f24 [3441335,74350] 0 2024-04-22T23:23:25.254 INFO:tasks.workunit.client.0.smithi067.stdout:3/327: sync 2024-04-22T23:23:25.256 INFO:tasks.workunit.client.0.smithi067.stdout:6/371: dwrite d17/f35 [0,4194304] 0 2024-04-22T23:23:25.256 INFO:tasks.workunit.client.0.smithi067.stdout:6/372: read d17/f35 [3154177,59726] 0 2024-04-22T23:23:25.260 INFO:tasks.workunit.client.0.smithi067.stdout:2/430: truncate d1/f3 574704 0 2024-04-22T23:23:25.262 INFO:tasks.workunit.client.0.smithi067.stdout:3/328: write d3/d4/f16 [3516167,2341] 0 2024-04-22T23:23:25.265 INFO:tasks.workunit.client.0.smithi067.stdout:6/373: getdents d17/d39/d53 0 2024-04-22T23:23:25.265 INFO:tasks.workunit.client.0.smithi067.stdout:6/374: chown d17/d39/d3e/d46/l58 3540 1 2024-04-22T23:23:25.270 INFO:tasks.workunit.client.0.smithi067.stdout:6/375: rename d17/f34 to d17/d39/d3e/d46/f78 0 2024-04-22T23:23:25.273 INFO:tasks.workunit.client.0.smithi067.stdout:6/376: rename d17/l4a to d17/d39/l79 0 2024-04-22T23:23:25.273 INFO:tasks.workunit.client.0.smithi067.stdout:6/377: stat fe 0 2024-04-22T23:23:25.274 INFO:tasks.workunit.client.0.smithi067.stdout:6/378: fsync f12 0 2024-04-22T23:23:25.274 INFO:tasks.workunit.client.0.smithi067.stdout:6/379: chown d17/d39/d53 0 1 2024-04-22T23:23:25.276 INFO:tasks.workunit.client.0.smithi067.stdout:6/380: symlink d17/d39/l7a 0 2024-04-22T23:23:25.364 INFO:tasks.workunit.client.0.smithi067.stdout:2/431: truncate d1/d10/d1c/f3a 3871577 0 2024-04-22T23:23:25.365 INFO:tasks.workunit.client.0.smithi067.stdout:3/329: truncate d3/d4/fe 294146 0 2024-04-22T23:23:25.365 INFO:tasks.workunit.client.0.smithi067.stdout:2/432: chown d1/d12/l29 48534 1 2024-04-22T23:23:25.367 INFO:tasks.workunit.client.0.smithi067.stdout:6/381: rmdir d17/d39 39 2024-04-22T23:23:25.369 INFO:tasks.workunit.client.0.smithi067.stdout:2/433: rename d1/d19/c2c to d1/d19/c81 0 2024-04-22T23:23:25.402 INFO:tasks.workunit.client.0.smithi067.stdout:8/375: sync 2024-04-22T23:23:25.418 INFO:tasks.workunit.client.0.smithi067.stdout:6/382: mknod d17/c7b 0 2024-04-22T23:23:25.419 INFO:tasks.workunit.client.0.smithi067.stdout:2/434: symlink d1/d10/d1c/d4d/l82 0 2024-04-22T23:23:25.419 INFO:tasks.workunit.client.0.smithi067.stdout:8/376: symlink d1/d41/l77 0 2024-04-22T23:23:25.420 INFO:tasks.workunit.client.0.smithi067.stdout:3/330: link d3/d4/d10/c2d d3/d4/c76 0 2024-04-22T23:23:25.421 INFO:tasks.workunit.client.0.smithi067.stdout:6/383: chown d17/d39/d3a/f50 338632903 1 2024-04-22T23:23:25.421 INFO:tasks.workunit.client.0.smithi067.stdout:2/435: truncate d1/d10/d72/f36 1484359 0 2024-04-22T23:23:25.423 INFO:tasks.workunit.client.0.smithi067.stdout:3/331: truncate d3/d4/f24 13445 0 2024-04-22T23:23:25.423 INFO:tasks.workunit.client.0.smithi067.stdout:3/332: chown d3/d4/d10/d38/d53 442097 1 2024-04-22T23:23:25.423 INFO:tasks.workunit.client.0.smithi067.stdout:2/436: mknod d1/d10/d1c/d6d/c83 0 2024-04-22T23:23:25.434 INFO:tasks.workunit.client.0.smithi067.stdout:8/377: dread d1/d3/f1b [0,4194304] 0 2024-04-22T23:23:25.436 INFO:tasks.workunit.client.0.smithi067.stdout:8/378: mknod d1/d41/d46/d64/d6f/c78 0 2024-04-22T23:23:25.437 INFO:tasks.workunit.client.0.smithi067.stdout:3/333: dread d3/d4/f24 [0,4194304] 0 2024-04-22T23:23:25.439 INFO:tasks.workunit.client.0.smithi067.stdout:3/334: chown d3/d4/l6b 424843909 1 2024-04-22T23:23:25.442 INFO:tasks.workunit.client.0.smithi067.stdout:3/335: symlink d3/d4/d28/l77 0 2024-04-22T23:23:25.445 INFO:tasks.workunit.client.0.smithi067.stdout:3/336: fdatasync d3/d4/d10/f39 0 2024-04-22T23:23:25.448 INFO:tasks.workunit.client.0.smithi067.stdout:3/337: write d3/d4/d12/f2f [1254339,63530] 0 2024-04-22T23:23:25.449 INFO:tasks.workunit.client.0.smithi067.stdout:2/437: dwrite d1/d10/d1c/f3a [0,4194304] 0 2024-04-22T23:23:25.450 INFO:tasks.workunit.client.0.smithi067.stdout:3/338: chown d3/d4/d10/d19/l60 67218894 1 2024-04-22T23:23:25.452 INFO:tasks.workunit.client.0.smithi067.stdout:2/438: chown d1/d42/f4b 14403153 1 2024-04-22T23:23:25.452 INFO:tasks.workunit.client.0.smithi067.stdout:3/339: write d3/d4/d12/f27 [548410,16648] 0 2024-04-22T23:23:25.453 INFO:tasks.workunit.client.0.smithi067.stdout:3/340: fdatasync d3/d4/d12/f1d 0 2024-04-22T23:23:25.453 INFO:tasks.workunit.client.0.smithi067.stdout:8/379: dread d1/d41/d46/d64/d6f/f30 [0,4194304] 0 2024-04-22T23:23:25.454 INFO:tasks.workunit.client.0.smithi067.stdout:2/439: symlink d1/d10/d72/d53/l84 0 2024-04-22T23:23:25.455 INFO:tasks.workunit.client.0.smithi067.stdout:8/380: write d1/d41/d46/f60 [999645,41894] 0 2024-04-22T23:23:25.459 INFO:tasks.workunit.client.0.smithi067.stdout:3/341: dread d3/fa [0,4194304] 0 2024-04-22T23:23:25.461 INFO:tasks.workunit.client.0.smithi067.stdout:8/381: creat d1/dd/f79 x:0 0 0 2024-04-22T23:23:25.462 INFO:tasks.workunit.client.0.smithi067.stdout:8/382: chown d1/dd/c40 167494397 1 2024-04-22T23:23:25.464 INFO:tasks.workunit.client.0.smithi067.stdout:8/383: write d1/d3/f56 [992630,12189] 0 2024-04-22T23:23:25.466 INFO:tasks.workunit.client.0.smithi067.stdout:3/342: dwrite d3/d4/d10/d19/f3b [0,4194304] 0 2024-04-22T23:23:25.472 INFO:tasks.workunit.client.0.smithi067.stdout:3/343: fdatasync d3/d4/d10/d19/f5d 0 2024-04-22T23:23:25.474 INFO:tasks.workunit.client.0.smithi067.stdout:3/344: rmdir d3/d4/d28/d40 39 2024-04-22T23:23:25.477 INFO:tasks.workunit.client.0.smithi067.stdout:3/345: creat d3/d4/d12/f78 x:0 0 0 2024-04-22T23:23:25.478 INFO:tasks.workunit.client.0.smithi067.stdout:2/440: dread d1/d10/d33/f5b [0,4194304] 0 2024-04-22T23:23:25.486 INFO:tasks.workunit.client.0.smithi067.stdout:3/346: dread d3/d4/d10/d1f/f3f [0,4194304] 0 2024-04-22T23:23:25.490 INFO:tasks.workunit.client.0.smithi067.stdout:3/347: dwrite d3/d4/d10/d19/f3b [0,4194304] 0 2024-04-22T23:23:25.491 INFO:tasks.workunit.client.0.smithi067.stdout:3/348: chown d3/d4/d1b/l46 101 1 2024-04-22T23:23:25.491 INFO:tasks.workunit.client.0.smithi067.stdout:8/384: dread d1/dd/d26/f22 [0,4194304] 0 2024-04-22T23:23:25.514 INFO:tasks.workunit.client.0.smithi067.stdout:4/423: sync 2024-04-22T23:23:25.514 INFO:tasks.workunit.client.0.smithi067.stdout:4/424: fdatasync d12/d5f/f41 0 2024-04-22T23:23:25.515 INFO:tasks.workunit.client.0.smithi067.stdout:4/425: truncate d12/d5f/f41 668452 0 2024-04-22T23:23:25.520 INFO:tasks.workunit.client.0.smithi067.stdout:4/426: dwrite d12/d42/f83 [0,4194304] 0 2024-04-22T23:23:25.523 INFO:tasks.workunit.client.0.smithi067.stdout:4/427: dread d12/d5f/d2b/f61 [0,4194304] 0 2024-04-22T23:23:25.530 INFO:tasks.workunit.client.0.smithi067.stdout:2/441: mknod d1/d10/d41/c85 0 2024-04-22T23:23:25.535 INFO:tasks.workunit.client.0.smithi067.stdout:4/428: mkdir d12/d42/d4b/d77/d9d 0 2024-04-22T23:23:25.535 INFO:tasks.workunit.client.0.smithi067.stdout:2/442: dread d1/d10/d1c/d6d/f4c [0,4194304] 0 2024-04-22T23:23:25.535 INFO:tasks.workunit.client.0.smithi067.stdout:4/429: truncate f4 9178303 0 2024-04-22T23:23:25.540 INFO:tasks.workunit.client.0.smithi067.stdout:4/430: readlink d12/d2e/l80 0 2024-04-22T23:23:25.543 INFO:tasks.workunit.client.0.smithi067.stdout:4/431: write d12/f52 [1471365,111484] 0 2024-04-22T23:23:25.544 INFO:tasks.workunit.client.0.smithi067.stdout:2/443: creat d1/d10/f86 x:0 0 0 2024-04-22T23:23:25.545 INFO:tasks.workunit.client.0.smithi067.stdout:2/444: write d1/d4f/f2a [1886444,128086] 0 2024-04-22T23:23:25.546 INFO:tasks.workunit.client.0.smithi067.stdout:2/445: readlink d1/d42/d61/l65 0 2024-04-22T23:23:25.549 INFO:tasks.workunit.client.0.smithi067.stdout:4/432: link d12/d5f/f5a d12/d75/f9e 0 2024-04-22T23:23:25.550 INFO:tasks.workunit.client.0.smithi067.stdout:2/446: mkdir d1/d10/d1c/d6d/d87 0 2024-04-22T23:23:25.553 INFO:tasks.workunit.client.0.smithi067.stdout:4/433: creat d12/d17/d3f/d84/f9f x:0 0 0 2024-04-22T23:23:25.556 INFO:tasks.workunit.client.0.smithi067.stdout:4/434: write d12/d42/d4b/d70/d8e/d56/f96 [641577,98014] 0 2024-04-22T23:23:25.557 INFO:tasks.workunit.client.0.smithi067.stdout:2/447: link d1/d42/f56 d1/d10/d33/d46/f88 0 2024-04-22T23:23:25.560 INFO:tasks.workunit.client.0.smithi067.stdout:3/349: dwrite d3/d4/fe [0,4194304] 0 2024-04-22T23:23:25.561 INFO:tasks.workunit.client.0.smithi067.stdout:2/448: rmdir d1/d10/d72 39 2024-04-22T23:23:25.562 INFO:tasks.workunit.client.0.smithi067.stdout:2/449: chown d1/d10/d33/d46 1504933 1 2024-04-22T23:23:25.566 INFO:tasks.workunit.client.0.smithi067.stdout:2/450: creat d1/d10/d1c/d6d/f89 x:0 0 0 2024-04-22T23:23:25.567 INFO:tasks.workunit.client.0.smithi067.stdout:4/435: getdents d12/d2e 0 2024-04-22T23:23:25.570 INFO:tasks.workunit.client.0.smithi067.stdout:3/350: write d3/d4/d10/d38/f57 [2184493,3701] 0 2024-04-22T23:23:25.572 INFO:tasks.workunit.client.0.smithi067.stdout:3/351: fsync d3/d4/d12/f2f 0 2024-04-22T23:23:25.576 INFO:tasks.workunit.client.0.smithi067.stdout:2/451: dread d1/d12/f43 [0,4194304] 0 2024-04-22T23:23:25.577 INFO:tasks.workunit.client.0.smithi067.stdout:2/452: truncate d1/d10/f86 648044 0 2024-04-22T23:23:25.580 INFO:tasks.workunit.client.0.smithi067.stdout:2/453: symlink d1/d10/d1c/d6d/d87/l8a 0 2024-04-22T23:23:25.581 INFO:tasks.workunit.client.0.smithi067.stdout:2/454: stat d1/d10/d41 0 2024-04-22T23:23:25.582 INFO:tasks.workunit.client.0.smithi067.stdout:4/436: dwrite d12/d5f/f28 [4194304,4194304] 0 2024-04-22T23:23:25.584 INFO:tasks.workunit.client.0.smithi067.stdout:2/455: unlink d1/d12/l64 0 2024-04-22T23:23:25.590 INFO:tasks.workunit.client.0.smithi067.stdout:4/437: mknod d12/d2e/d9b/ca0 0 2024-04-22T23:23:25.592 INFO:tasks.workunit.client.0.smithi067.stdout:4/438: symlink d12/d5f/d69/la1 0 2024-04-22T23:23:25.595 INFO:tasks.workunit.client.0.smithi067.stdout:2/456: dread d1/d42/f56 [0,4194304] 0 2024-04-22T23:23:25.599 INFO:tasks.workunit.client.0.smithi067.stdout:4/439: dwrite d12/d17/d3f/d84/f9f [0,4194304] 0 2024-04-22T23:23:25.600 INFO:tasks.workunit.client.0.smithi067.stdout:2/457: creat d1/d10/d72/d68/d7b/f8b x:0 0 0 2024-04-22T23:23:25.602 INFO:tasks.workunit.client.0.smithi067.stdout:4/440: truncate d12/d5f/f5a 3948693 0 2024-04-22T23:23:25.608 INFO:tasks.workunit.client.0.smithi067.stdout:2/458: truncate d1/d10/d1c/f5d 892366 0 2024-04-22T23:23:25.609 INFO:tasks.workunit.client.0.smithi067.stdout:2/459: write d1/d12/f4e [1020567,96354] 0 2024-04-22T23:23:25.611 INFO:tasks.workunit.client.0.smithi067.stdout:2/460: creat d1/d42/d6b/f8c x:0 0 0 2024-04-22T23:23:25.615 INFO:tasks.workunit.client.0.smithi067.stdout:2/461: rename d1/d12/l13 to d1/d42/d5f/l8d 0 2024-04-22T23:23:25.620 INFO:tasks.workunit.client.0.smithi067.stdout:2/462: dread d1/d12/f4e [0,4194304] 0 2024-04-22T23:23:25.620 INFO:tasks.workunit.client.0.smithi067.stdout:2/463: chown d1/d10/d1c/d4d 2543869 1 2024-04-22T23:23:25.621 INFO:tasks.workunit.client.0.smithi067.stdout:2/464: dread d1/d10/f86 [0,4194304] 0 2024-04-22T23:23:25.622 INFO:tasks.workunit.client.0.smithi067.stdout:2/465: stat d1/f35 0 2024-04-22T23:23:25.626 INFO:tasks.workunit.client.0.smithi067.stdout:2/466: link d1/d19/c81 d1/d42/c8e 0 2024-04-22T23:23:25.628 INFO:tasks.workunit.client.0.smithi067.stdout:2/467: getdents d1/d10/d1c/d4d 0 2024-04-22T23:23:25.630 INFO:tasks.workunit.client.0.smithi067.stdout:2/468: dread d1/d42/f57 [0,4194304] 0 2024-04-22T23:23:25.630 INFO:tasks.workunit.client.0.smithi067.stdout:2/469: truncate d1/d12/f4e 1239170 0 2024-04-22T23:23:25.632 INFO:tasks.workunit.client.0.smithi067.stdout:2/470: symlink d1/d10/d33/l8f 0 2024-04-22T23:23:25.634 INFO:tasks.workunit.client.0.smithi067.stdout:2/471: creat d1/d42/d5f/f90 x:0 0 0 2024-04-22T23:23:25.634 INFO:tasks.workunit.client.0.smithi067.stdout:2/472: chown d1/d10/f52 223923207 1 2024-04-22T23:23:25.636 INFO:tasks.workunit.client.0.smithi067.stdout:2/473: symlink d1/d10/d41/l91 0 2024-04-22T23:23:25.639 INFO:tasks.workunit.client.0.smithi067.stdout:2/474: creat d1/d12/f92 x:0 0 0 2024-04-22T23:23:25.645 INFO:tasks.workunit.client.0.smithi067.stdout:2/475: dwrite d1/d42/d6b/f8c [0,4194304] 0 2024-04-22T23:23:25.646 INFO:tasks.workunit.client.0.smithi067.stdout:2/476: chown d1/d42/d5f/c6c 10 1 2024-04-22T23:23:25.649 INFO:tasks.workunit.client.0.smithi067.stdout:7/251: sync 2024-04-22T23:23:25.649 INFO:tasks.workunit.client.0.smithi067.stdout:2/477: creat d1/d10/d41/f93 x:0 0 0 2024-04-22T23:23:25.650 INFO:tasks.workunit.client.0.smithi067.stdout:7/252: chown d10/d18/f1a 0 1 2024-04-22T23:23:25.652 INFO:tasks.workunit.client.0.smithi067.stdout:2/478: mknod d1/d10/c94 0 2024-04-22T23:23:25.653 INFO:tasks.workunit.client.0.smithi067.stdout:7/253: mknod d10/d14/d1d/d46/c55 0 2024-04-22T23:23:25.663 INFO:tasks.workunit.client.0.smithi067.stdout:2/479: mkdir d1/d10/d72/d68/d7b/d95 0 2024-04-22T23:23:25.664 INFO:tasks.workunit.client.0.smithi067.stdout:7/254: creat d10/d14/d1d/d26/d2c/f56 x:0 0 0 2024-04-22T23:23:25.667 INFO:tasks.workunit.client.0.smithi067.stdout:2/480: symlink d1/d42/d5f/l96 0 2024-04-22T23:23:25.669 INFO:tasks.workunit.client.0.smithi067.stdout:2/481: creat d1/d10/d72/d53/f97 x:0 0 0 2024-04-22T23:23:25.669 INFO:tasks.workunit.client.0.smithi067.stdout:2/482: fsync d1/d4f/f2a 0 2024-04-22T23:23:25.670 INFO:tasks.workunit.client.0.smithi067.stdout:7/255: rename d10/d47/l4c to d10/d14/d1d/d24/l57 0 2024-04-22T23:23:25.672 INFO:tasks.workunit.client.0.smithi067.stdout:7/256: creat d10/d47/d49/f58 x:0 0 0 2024-04-22T23:23:25.673 INFO:tasks.workunit.client.0.smithi067.stdout:7/257: dread - d10/d14/d1d/f42 zero size 2024-04-22T23:23:25.675 INFO:tasks.workunit.client.0.smithi067.stdout:7/258: link d10/f45 d10/d18/d35/f59 0 2024-04-22T23:23:25.677 INFO:tasks.workunit.client.0.smithi067.stdout:7/259: rmdir d10/d47 39 2024-04-22T23:23:25.679 INFO:tasks.workunit.client.0.smithi067.stdout:7/260: creat d10/d14/f5a x:0 0 0 2024-04-22T23:23:25.680 INFO:tasks.workunit.client.0.smithi067.stdout:7/261: creat d10/d18/d35/f5b x:0 0 0 2024-04-22T23:23:25.681 INFO:tasks.workunit.client.0.smithi067.stdout:7/262: dread - d10/d18/f27 zero size 2024-04-22T23:23:25.683 INFO:tasks.workunit.client.0.smithi067.stdout:7/263: rename d10/d18/d35/f59 to d10/d14/d1d/d46/d50/f5c 0 2024-04-22T23:23:25.686 INFO:tasks.workunit.client.0.smithi067.stdout:7/264: creat d10/d14/d1d/d26/f5d x:0 0 0 2024-04-22T23:23:25.686 INFO:tasks.workunit.client.0.smithi067.stdout:7/265: chown fb 335830 1 2024-04-22T23:23:25.689 INFO:tasks.workunit.client.0.smithi067.stdout:7/266: link d10/d14/d1d/d26/f2f d10/d18/d35/f5e 0 2024-04-22T23:23:25.697 INFO:tasks.workunit.client.0.smithi067.stdout:7/267: dwrite d10/f45 [0,4194304] 0 2024-04-22T23:23:25.700 INFO:tasks.workunit.client.0.smithi067.stdout:7/268: fsync d10/d18/f1e 0 2024-04-22T23:23:25.710 INFO:tasks.workunit.client.0.smithi067.stdout:7/269: truncate d10/f45 1518852 0 2024-04-22T23:23:25.711 INFO:tasks.workunit.client.0.smithi067.stdout:7/270: chown d10/d18/c3d 12771238 1 2024-04-22T23:23:25.712 INFO:tasks.workunit.client.0.smithi067.stdout:7/271: rmdir d10/d18 39 2024-04-22T23:23:25.714 INFO:tasks.workunit.client.0.smithi067.stdout:7/272: read - d10/d47/d49/f58 zero size 2024-04-22T23:23:25.715 INFO:tasks.workunit.client.0.smithi067.stdout:7/273: creat d10/d14/d1d/d24/f5f x:0 0 0 2024-04-22T23:23:25.717 INFO:tasks.workunit.client.0.smithi067.stdout:7/274: mknod d10/d14/d1d/d26/d2c/d41/c60 0 2024-04-22T23:23:25.718 INFO:tasks.workunit.client.0.smithi067.stdout:7/275: read - d10/d14/d1d/d24/f5f zero size 2024-04-22T23:23:25.719 INFO:tasks.workunit.client.0.smithi067.stdout:7/276: creat d10/d14/d1d/d26/d2c/f61 x:0 0 0 2024-04-22T23:23:25.720 INFO:tasks.workunit.client.0.smithi067.stdout:7/277: chown d10/l22 839 1 2024-04-22T23:23:25.721 INFO:tasks.workunit.client.0.smithi067.stdout:7/278: stat d10/d18/d35/c3b 0 2024-04-22T23:23:25.722 INFO:tasks.workunit.client.0.smithi067.stdout:7/279: mkdir d10/d18/d35/d62 0 2024-04-22T23:23:25.724 INFO:tasks.workunit.client.0.smithi067.stdout:7/280: truncate d10/d14/d1d/d46/d50/f2d 1474258 0 2024-04-22T23:23:25.725 INFO:tasks.workunit.client.0.smithi067.stdout:7/281: dread - d10/d14/d1d/d26/f3a zero size 2024-04-22T23:23:25.726 INFO:tasks.workunit.client.0.smithi067.stdout:7/282: creat d10/d18/f63 x:0 0 0 2024-04-22T23:23:25.727 INFO:tasks.workunit.client.0.smithi067.stdout:7/283: write d10/d18/d35/f4e [262161,89051] 0 2024-04-22T23:23:25.728 INFO:tasks.workunit.client.0.smithi067.stdout:7/284: chown d10/c23 214051 1 2024-04-22T23:23:25.730 INFO:tasks.workunit.client.0.smithi067.stdout:7/285: mknod d10/d14/d1d/d26/c64 0 2024-04-22T23:23:25.732 INFO:tasks.workunit.client.0.smithi067.stdout:7/286: fsync d10/d18/f1e 0 2024-04-22T23:23:25.733 INFO:tasks.workunit.client.0.smithi067.stdout:7/287: dread - d10/d14/d1d/d26/f3a zero size 2024-04-22T23:23:25.734 INFO:tasks.workunit.client.0.smithi067.stdout:7/288: truncate d10/d14/f34 605201 0 2024-04-22T23:23:25.736 INFO:tasks.workunit.client.0.smithi067.stdout:7/289: write d10/d14/d1d/d46/d50/f5c [2497227,61500] 0 2024-04-22T23:23:25.743 INFO:tasks.workunit.client.0.smithi067.stdout:7/290: dwrite fc [0,4194304] 0 2024-04-22T23:23:25.748 INFO:tasks.workunit.client.0.smithi067.stdout:7/291: getdents d10/d18/d35 0 2024-04-22T23:23:25.768 INFO:tasks.workunit.client.0.smithi067.stdout:7/292: dread d10/d14/d1d/f1f [0,4194304] 0 2024-04-22T23:23:25.773 INFO:tasks.workunit.client.0.smithi067.stdout:7/293: link d10/d14/d1d/f42 d10/d14/d1d/d26/d2c/d41/f65 0 2024-04-22T23:23:25.777 INFO:tasks.workunit.client.0.smithi067.stdout:7/294: dwrite fb [0,4194304] 0 2024-04-22T23:23:25.780 INFO:tasks.workunit.client.0.smithi067.stdout:7/295: creat d10/d18/d35/f66 x:0 0 0 2024-04-22T23:23:25.782 INFO:tasks.workunit.client.0.smithi067.stdout:7/296: truncate d10/d14/d1d/d26/f2f 679798 0 2024-04-22T23:23:25.784 INFO:tasks.workunit.client.0.smithi067.stdout:7/297: creat d10/d14/d1d/d24/f67 x:0 0 0 2024-04-22T23:23:25.785 INFO:tasks.workunit.client.0.smithi067.stdout:7/298: chown d10/d18/c38 241723 1 2024-04-22T23:23:25.786 INFO:tasks.workunit.client.0.smithi067.stdout:7/299: stat d10/d18/d35/c4d 0 2024-04-22T23:23:25.787 INFO:tasks.workunit.client.0.smithi067.stdout:7/300: creat d10/d47/f68 x:0 0 0 2024-04-22T23:23:25.789 INFO:tasks.workunit.client.0.smithi067.stdout:7/301: write d10/d14/d1d/d46/d50/f29 [2969659,57] 0 2024-04-22T23:23:25.790 INFO:tasks.workunit.client.0.smithi067.stdout:7/302: rmdir d10/d14/d1d 39 2024-04-22T23:23:25.798 INFO:tasks.workunit.client.0.smithi067.stdout:9/274: sync 2024-04-22T23:23:25.800 INFO:tasks.workunit.client.0.smithi067.stdout:7/303: dwrite d10/f12 [0,4194304] 0 2024-04-22T23:23:25.804 INFO:tasks.workunit.client.0.smithi067.stdout:7/304: mkdir d10/d14/d1d/d69 0 2024-04-22T23:23:25.806 INFO:tasks.workunit.client.0.smithi067.stdout:7/305: mkdir d10/d14/d1d/d46/d6a 0 2024-04-22T23:23:25.944 INFO:tasks.workunit.client.0.smithi067.stdout:1/324: sync 2024-04-22T23:23:25.944 INFO:tasks.workunit.client.0.smithi067.stdout:0/319: sync 2024-04-22T23:23:25.947 INFO:tasks.workunit.client.0.smithi067.stdout:0/320: creat db/d2e/d31/d56/f6d x:0 0 0 2024-04-22T23:23:25.947 INFO:tasks.workunit.client.0.smithi067.stdout:1/325: creat d8/d9/d2f/d33/d5c/f6d x:0 0 0 2024-04-22T23:23:25.948 INFO:tasks.workunit.client.0.smithi067.stdout:0/321: write db/de/d2a/f39 [1023153,51190] 0 2024-04-22T23:23:25.950 INFO:tasks.workunit.client.0.smithi067.stdout:1/326: unlink f4 0 2024-04-22T23:23:25.952 INFO:tasks.workunit.client.0.smithi067.stdout:1/327: unlink d8/d12/d36/c56 0 2024-04-22T23:23:25.955 INFO:tasks.workunit.client.0.smithi067.stdout:1/328: creat d8/d9/d2f/d59/f6e x:0 0 0 2024-04-22T23:23:25.957 INFO:tasks.workunit.client.0.smithi067.stdout:1/329: rename d8/d9/d2d/l35 to d8/d4a/l6f 0 2024-04-22T23:23:25.960 INFO:tasks.workunit.client.0.smithi067.stdout:1/330: rename d8/d24 to d8/d9/d2d/d4d/d70 0 2024-04-22T23:23:25.962 INFO:tasks.workunit.client.0.smithi067.stdout:1/331: mknod d8/d12/d40/c71 0 2024-04-22T23:23:25.963 INFO:tasks.workunit.client.0.smithi067.stdout:1/332: chown d8/d9/f2c 0 1 2024-04-22T23:23:25.965 INFO:tasks.workunit.client.0.smithi067.stdout:1/333: creat d8/d12/d36/f72 x:0 0 0 2024-04-22T23:23:26.065 INFO:tasks.workunit.client.0.smithi067.stdout:5/381: sync 2024-04-22T23:23:26.065 INFO:tasks.workunit.client.0.smithi067.stdout:5/382: chown d5/db/d12/d44 118 1 2024-04-22T23:23:26.066 INFO:tasks.workunit.client.0.smithi067.stdout:5/383: creat d5/d3a/d4d/f8d x:0 0 0 2024-04-22T23:23:26.068 INFO:tasks.workunit.client.0.smithi067.stdout:5/384: read d5/db/ff [4000990,36969] 0 2024-04-22T23:23:26.068 INFO:tasks.workunit.client.0.smithi067.stdout:5/385: dread - d5/db/d35/f6f zero size 2024-04-22T23:23:26.069 INFO:tasks.workunit.client.0.smithi067.stdout:5/386: chown d5/db/d12/d8a 715 1 2024-04-22T23:23:26.069 INFO:tasks.workunit.client.0.smithi067.stdout:5/387: chown d5 25958490 1 2024-04-22T23:23:26.071 INFO:tasks.workunit.client.0.smithi067.stdout:5/388: creat d5/db/d35/f8e x:0 0 0 2024-04-22T23:23:26.071 INFO:tasks.workunit.client.0.smithi067.stdout:5/389: chown d5/l9 264 1 2024-04-22T23:23:26.074 INFO:tasks.workunit.client.0.smithi067.stdout:5/390: link d5/d28/d36/f75 d5/d1a/d8b/f8f 0 2024-04-22T23:23:26.075 INFO:tasks.workunit.client.0.smithi067.stdout:5/391: chown d5/d28/d36/d40/f85 6 1 2024-04-22T23:23:26.076 INFO:tasks.workunit.client.0.smithi067.stdout:5/392: write d5/d28/f64 [1215562,37040] 0 2024-04-22T23:23:26.085 INFO:tasks.workunit.client.0.smithi067.stdout:5/393: rmdir d5 39 2024-04-22T23:23:26.100 INFO:tasks.workunit.client.0.smithi067.stdout:5/394: dread d5/db/ff [4194304,4194304] 0 2024-04-22T23:23:26.101 INFO:tasks.workunit.client.0.smithi067.stdout:5/395: mkdir d5/d3e/d90 0 2024-04-22T23:23:26.101 INFO:tasks.workunit.client.0.smithi067.stdout:5/396: fsync d5/f1b 0 2024-04-22T23:23:26.109 INFO:tasks.workunit.client.0.smithi067.stdout:5/397: dwrite d5/d1a/d8b/f8f [0,4194304] 0 2024-04-22T23:23:26.112 INFO:tasks.workunit.client.0.smithi067.stdout:5/398: mknod d5/c91 0 2024-04-22T23:23:26.115 INFO:tasks.workunit.client.0.smithi067.stdout:5/399: dread d5/db/d12/f2b [4194304,4194304] 0 2024-04-22T23:23:26.118 INFO:tasks.workunit.client.0.smithi067.stdout:5/400: creat d5/d28/d36/d40/d57/d61/d89/f92 x:0 0 0 2024-04-22T23:23:26.120 INFO:tasks.workunit.client.0.smithi067.stdout:5/401: read d5/d3e/f77 [99955,26706] 0 2024-04-22T23:23:26.121 INFO:tasks.workunit.client.0.smithi067.stdout:5/402: read - d5/d3a/d4d/f8d zero size 2024-04-22T23:23:26.124 INFO:tasks.workunit.client.0.smithi067.stdout:5/403: rename d5/db/d35/f46 to d5/db/f93 0 2024-04-22T23:23:26.127 INFO:tasks.workunit.client.0.smithi067.stdout:5/404: rename d5/db/d12/d44/f83 to d5/d3e/d90/f94 0 2024-04-22T23:23:26.128 INFO:tasks.workunit.client.0.smithi067.stdout:5/405: write d5/f1b [5049574,24615] 0 2024-04-22T23:23:26.137 INFO:tasks.workunit.client.0.smithi067.stdout:6/384: sync 2024-04-22T23:23:26.139 INFO:tasks.workunit.client.0.smithi067.stdout:6/385: unlink la 0 2024-04-22T23:23:26.139 INFO:tasks.workunit.client.0.smithi067.stdout:6/386: stat d17/d39/d3a/f61 0 2024-04-22T23:23:26.142 INFO:tasks.workunit.client.0.smithi067.stdout:5/406: dread d5/db/d12/f25 [0,4194304] 0 2024-04-22T23:23:26.143 INFO:tasks.workunit.client.0.smithi067.stdout:6/387: link d17/l22 d17/d39/d3e/d46/l7c 0 2024-04-22T23:23:26.144 INFO:tasks.workunit.client.0.smithi067.stdout:6/388: truncate d17/f2b 1018954 0 2024-04-22T23:23:26.145 INFO:tasks.workunit.client.0.smithi067.stdout:5/407: symlink d5/d1a/d8b/l95 0 2024-04-22T23:23:26.148 INFO:tasks.workunit.client.0.smithi067.stdout:5/408: rename d5/d3e/c6b to d5/d28/d36/d40/d57/d61/c96 0 2024-04-22T23:23:26.150 INFO:tasks.workunit.client.0.smithi067.stdout:6/389: dwrite d17/f48 [0,4194304] 0 2024-04-22T23:23:26.151 INFO:tasks.workunit.client.0.smithi067.stdout:6/390: rmdir d17/d39/d53 39 2024-04-22T23:23:26.171 INFO:tasks.workunit.client.0.smithi067.stdout:6/391: dread d17/d39/f3f [0,4194304] 0 2024-04-22T23:23:26.176 INFO:tasks.workunit.client.0.smithi067.stdout:6/392: getdents d17/d39/d3e 0 2024-04-22T23:23:26.177 INFO:tasks.workunit.client.0.smithi067.stdout:6/393: readlink d17/d39/l79 0 2024-04-22T23:23:26.177 INFO:tasks.workunit.client.0.smithi067.stdout:6/394: fsync d17/f48 0 2024-04-22T23:23:26.178 INFO:tasks.workunit.client.0.smithi067.stdout:6/395: read - d17/d39/f69 zero size 2024-04-22T23:23:26.181 INFO:tasks.workunit.client.0.smithi067.stdout:6/396: creat d17/f7d x:0 0 0 2024-04-22T23:23:26.181 INFO:tasks.workunit.client.0.smithi067.stdout:6/397: chown d17/f25 56243 1 2024-04-22T23:23:26.192 INFO:tasks.workunit.client.0.smithi067.stdout:6/398: dread f13 [0,4194304] 0 2024-04-22T23:23:26.195 INFO:tasks.workunit.client.0.smithi067.stdout:6/399: dread d17/f20 [0,4194304] 0 2024-04-22T23:23:26.196 INFO:tasks.workunit.client.0.smithi067.stdout:6/400: fdatasync d17/d39/f3f 0 2024-04-22T23:23:26.197 INFO:tasks.workunit.client.0.smithi067.stdout:6/401: write d17/f25 [831351,119409] 0 2024-04-22T23:23:26.198 INFO:tasks.workunit.client.0.smithi067.stdout:6/402: read d17/d39/d3a/f40 [324310,95237] 0 2024-04-22T23:23:26.198 INFO:tasks.workunit.client.0.smithi067.stdout:6/403: stat d17/f26 0 2024-04-22T23:23:26.202 INFO:tasks.workunit.client.0.smithi067.stdout:6/404: dread d17/d39/d3e/d46/d62/f72 [0,4194304] 0 2024-04-22T23:23:26.202 INFO:tasks.workunit.client.0.smithi067.stdout:6/405: fsync f13 0 2024-04-22T23:23:26.205 INFO:tasks.workunit.client.0.smithi067.stdout:6/406: unlink d17/d39/d53/l63 0 2024-04-22T23:23:26.206 INFO:tasks.workunit.client.0.smithi067.stdout:6/407: mknod d17/c7e 0 2024-04-22T23:23:26.209 INFO:tasks.workunit.client.0.smithi067.stdout:6/408: link fe d17/f7f 0 2024-04-22T23:23:26.212 INFO:tasks.workunit.client.0.smithi067.stdout:6/409: getdents d17/d39 0 2024-04-22T23:23:26.212 INFO:tasks.workunit.client.0.smithi067.stdout:6/410: readlink d17/d39/d3a/l5b 0 2024-04-22T23:23:26.214 INFO:tasks.workunit.client.0.smithi067.stdout:6/411: mknod d17/d39/d3e/d46/d62/c80 0 2024-04-22T23:23:26.234 INFO:tasks.workunit.client.0.smithi067.stdout:0/322: sync 2024-04-22T23:23:26.234 INFO:tasks.workunit.client.0.smithi067.stdout:7/306: sync 2024-04-22T23:23:26.234 INFO:tasks.workunit.client.0.smithi067.stdout:2/483: sync 2024-04-22T23:23:26.234 INFO:tasks.workunit.client.0.smithi067.stdout:8/385: sync 2024-04-22T23:23:26.234 INFO:tasks.workunit.client.0.smithi067.stdout:1/334: sync 2024-04-22T23:23:26.234 INFO:tasks.workunit.client.0.smithi067.stdout:4/441: sync 2024-04-22T23:23:26.234 INFO:tasks.workunit.client.0.smithi067.stdout:3/352: sync 2024-04-22T23:23:26.234 INFO:tasks.workunit.client.0.smithi067.stdout:9/275: sync 2024-04-22T23:23:26.235 INFO:tasks.workunit.client.0.smithi067.stdout:7/307: stat d10/d18/c38 0 2024-04-22T23:23:26.236 INFO:tasks.workunit.client.0.smithi067.stdout:0/323: write db/de/f12 [464701,20495] 0 2024-04-22T23:23:26.237 INFO:tasks.workunit.client.0.smithi067.stdout:2/484: read d1/d10/d72/d53/f75 [2709425,69421] 0 2024-04-22T23:23:26.237 INFO:tasks.workunit.client.0.smithi067.stdout:8/386: creat d1/f7a x:0 0 0 2024-04-22T23:23:26.241 INFO:tasks.workunit.client.0.smithi067.stdout:4/442: unlink d12/d5f/d2b/f61 0 2024-04-22T23:23:26.243 INFO:tasks.workunit.client.0.smithi067.stdout:7/308: dwrite d10/d47/f68 [0,4194304] 0 2024-04-22T23:23:26.244 INFO:tasks.workunit.client.0.smithi067.stdout:9/276: mknod d5/dd/d18/d39/d3e/c5b 0 2024-04-22T23:23:26.249 INFO:tasks.workunit.client.0.smithi067.stdout:2/485: creat d1/d42/d5f/f98 x:0 0 0 2024-04-22T23:23:26.249 INFO:tasks.workunit.client.0.smithi067.stdout:2/486: stat d1/d10/f24 0 2024-04-22T23:23:26.250 INFO:tasks.workunit.client.0.smithi067.stdout:2/487: read d1/d42/f4b [2556411,21851] 0 2024-04-22T23:23:26.252 INFO:tasks.workunit.client.0.smithi067.stdout:2/488: chown d1/d19/f76 104 1 2024-04-22T23:23:26.253 INFO:tasks.workunit.client.0.smithi067.stdout:2/489: dread - d1/f69 zero size 2024-04-22T23:23:26.254 INFO:tasks.workunit.client.0.smithi067.stdout:2/490: dread - d1/d12/f92 zero size 2024-04-22T23:23:26.255 INFO:tasks.workunit.client.0.smithi067.stdout:9/277: unlink d5/ff 0 2024-04-22T23:23:26.258 INFO:tasks.workunit.client.0.smithi067.stdout:0/324: rename db/c34 to db/d21/d29/d2d/c6e 0 2024-04-22T23:23:26.260 INFO:tasks.workunit.client.0.smithi067.stdout:0/325: dread db/d21/f4c [0,4194304] 0 2024-04-22T23:23:26.263 INFO:tasks.workunit.client.0.smithi067.stdout:1/335: truncate d8/d9/f29 394785 0 2024-04-22T23:23:26.264 INFO:tasks.workunit.client.0.smithi067.stdout:7/309: mknod d10/d14/d1d/c6b 0 2024-04-22T23:23:26.269 INFO:tasks.workunit.client.0.smithi067.stdout:9/278: mkdir d5/dd/d18/d5c 0 2024-04-22T23:23:26.271 INFO:tasks.workunit.client.0.smithi067.stdout:3/353: link d3/d4/d10/d38/f57 d3/d4/f79 0 2024-04-22T23:23:26.273 INFO:tasks.workunit.client.0.smithi067.stdout:8/387: symlink d1/dd/d26/d62/d75/l7b 0 2024-04-22T23:23:26.273 INFO:tasks.workunit.client.0.smithi067.stdout:8/388: readlink d1/dd/l5d 0 2024-04-22T23:23:26.277 INFO:tasks.workunit.client.0.smithi067.stdout:0/326: mknod db/de/d2a/c6f 0 2024-04-22T23:23:26.278 INFO:tasks.workunit.client.0.smithi067.stdout:2/491: write d1/d10/d33/d46/f88 [1127299,87465] 0 2024-04-22T23:23:26.280 INFO:tasks.workunit.client.0.smithi067.stdout:7/310: fdatasync d10/d18/d35/f5e 0 2024-04-22T23:23:26.282 INFO:tasks.workunit.client.0.smithi067.stdout:7/311: write d10/d14/d1d/d24/f5f [590637,108765] 0 2024-04-22T23:23:26.284 INFO:tasks.workunit.client.0.smithi067.stdout:7/312: dread - d10/d18/f63 zero size 2024-04-22T23:23:26.284 INFO:tasks.workunit.client.0.smithi067.stdout:9/279: dread d5/d21/f2b [0,4194304] 0 2024-04-22T23:23:26.286 INFO:tasks.workunit.client.0.smithi067.stdout:9/280: write d5/fa [856017,28113] 0 2024-04-22T23:23:26.287 INFO:tasks.workunit.client.0.smithi067.stdout:3/354: rename d3/d4/f24 to d3/d4/d28/d4a/f7a 0 2024-04-22T23:23:26.288 INFO:tasks.workunit.client.0.smithi067.stdout:3/355: write d3/d4/d28/d4a/f73 [837073,27810] 0 2024-04-22T23:23:26.290 INFO:tasks.workunit.client.0.smithi067.stdout:4/443: getdents d12/d17/d31 0 2024-04-22T23:23:26.292 INFO:tasks.workunit.client.0.smithi067.stdout:1/336: write d8/d12/f17 [4004405,2270] 0 2024-04-22T23:23:26.294 INFO:tasks.workunit.client.0.smithi067.stdout:2/492: creat d1/d10/d33/d46/f99 x:0 0 0 2024-04-22T23:23:26.295 INFO:tasks.workunit.client.0.smithi067.stdout:2/493: chown d1/d10/d1c/d6d 2571 1 2024-04-22T23:23:26.297 INFO:tasks.workunit.client.0.smithi067.stdout:0/327: dread db/de/d11/f3c [0,4194304] 0 2024-04-22T23:23:26.299 INFO:tasks.workunit.client.0.smithi067.stdout:4/444: creat d12/d42/d4b/d70/d8e/d56/d8b/fa2 x:0 0 0 2024-04-22T23:23:26.300 INFO:tasks.workunit.client.0.smithi067.stdout:4/445: stat d12/d17/d47/c92 0 2024-04-22T23:23:26.300 INFO:tasks.workunit.client.0.smithi067.stdout:1/337: creat d8/d9/d2f/d33/d66/f73 x:0 0 0 2024-04-22T23:23:26.301 INFO:tasks.workunit.client.0.smithi067.stdout:7/313: mknod d10/d14/d1d/d69/c6c 0 2024-04-22T23:23:26.302 INFO:tasks.workunit.client.0.smithi067.stdout:2/494: mkdir d1/d10/d72/d53/d9a 0 2024-04-22T23:23:26.303 INFO:tasks.workunit.client.0.smithi067.stdout:9/281: symlink d5/dd/d18/d39/d4d/l5d 0 2024-04-22T23:23:26.304 INFO:tasks.workunit.client.0.smithi067.stdout:0/328: mkdir db/d2e/d31/d56/d70 0 2024-04-22T23:23:26.306 INFO:tasks.workunit.client.0.smithi067.stdout:1/338: symlink d8/d9/d2f/d59/l74 0 2024-04-22T23:23:26.310 INFO:tasks.workunit.client.0.smithi067.stdout:4/446: dwrite d12/f52 [4194304,4194304] 0 2024-04-22T23:23:26.311 INFO:tasks.workunit.client.0.smithi067.stdout:7/314: mkdir d10/d14/d6d 0 2024-04-22T23:23:26.312 INFO:tasks.workunit.client.0.smithi067.stdout:3/356: getdents d3/d4/d12 0 2024-04-22T23:23:26.313 INFO:tasks.workunit.client.0.smithi067.stdout:2/495: fdatasync d1/d10/f30 0 2024-04-22T23:23:26.316 INFO:tasks.workunit.client.0.smithi067.stdout:3/357: stat d3/d4/d10/d19/f21 0 2024-04-22T23:23:26.318 INFO:tasks.workunit.client.0.smithi067.stdout:0/329: mkdir db/d71 0 2024-04-22T23:23:26.320 INFO:tasks.workunit.client.0.smithi067.stdout:9/282: dwrite d5/f9 [0,4194304] 0 2024-04-22T23:23:26.321 INFO:tasks.workunit.client.0.smithi067.stdout:7/315: dread fc [0,4194304] 0 2024-04-22T23:23:26.322 INFO:tasks.workunit.client.0.smithi067.stdout:7/316: readlink d10/d14/l20 0 2024-04-22T23:23:26.323 INFO:tasks.workunit.client.0.smithi067.stdout:7/317: chown d10/d14/d1d/d46/d50/l4a 3291 1 2024-04-22T23:23:26.329 INFO:tasks.workunit.client.0.smithi067.stdout:7/318: symlink d10/d47/l6e 0 2024-04-22T23:23:26.329 INFO:tasks.workunit.client.0.smithi067.stdout:7/319: read d10/d14/f34 [21631,43374] 0 2024-04-22T23:23:26.330 INFO:tasks.workunit.client.0.smithi067.stdout:2/496: fdatasync d1/d10/d72/f36 0 2024-04-22T23:23:26.331 INFO:tasks.workunit.client.0.smithi067.stdout:7/320: dread - d10/d14/d1d/d46/d50/f30 zero size 2024-04-22T23:23:26.332 INFO:tasks.workunit.client.0.smithi067.stdout:2/497: chown d1/d19/f5c 63050519 1 2024-04-22T23:23:26.333 INFO:tasks.workunit.client.0.smithi067.stdout:2/498: readlink d1/d10/d33/l73 0 2024-04-22T23:23:26.334 INFO:tasks.workunit.client.0.smithi067.stdout:7/321: read d10/d14/d1d/d46/d50/f43 [853714,98753] 0 2024-04-22T23:23:26.335 INFO:tasks.workunit.client.0.smithi067.stdout:3/358: link d3/d4/fe d3/d4/d10/d38/d53/f7b 0 2024-04-22T23:23:26.336 INFO:tasks.workunit.client.0.smithi067.stdout:7/322: chown d10/d14/d1d/d24/f67 10382 1 2024-04-22T23:23:26.337 INFO:tasks.workunit.client.0.smithi067.stdout:7/323: dread - d10/d14/d1d/d24/f67 zero size 2024-04-22T23:23:26.337 INFO:tasks.workunit.client.0.smithi067.stdout:5/409: sync 2024-04-22T23:23:26.338 INFO:tasks.workunit.client.0.smithi067.stdout:2/499: dread d1/d19/f21 [4194304,4194304] 0 2024-04-22T23:23:26.338 INFO:tasks.workunit.client.0.smithi067.stdout:1/339: truncate d8/d9/d2d/d4d/d70/d34/f39 122045 0 2024-04-22T23:23:26.339 INFO:tasks.workunit.client.0.smithi067.stdout:4/447: link d12/l2c d12/d5f/d69/la3 0 2024-04-22T23:23:26.339 INFO:tasks.workunit.client.0.smithi067.stdout:7/324: chown d10/d14/d1d/d46/d50/l4a 983251 1 2024-04-22T23:23:26.341 INFO:tasks.workunit.client.0.smithi067.stdout:2/500: write d1/d10/f2e [4396778,42555] 0 2024-04-22T23:23:26.342 INFO:tasks.workunit.client.0.smithi067.stdout:3/359: creat d3/d4/d1b/f7c x:0 0 0 2024-04-22T23:23:26.342 INFO:tasks.workunit.client.0.smithi067.stdout:7/325: write d10/d14/d1d/d46/d50/f30 [261100,112080] 0 2024-04-22T23:23:26.345 INFO:tasks.workunit.client.0.smithi067.stdout:1/340: creat d8/d9/d2d/d4d/d48/f75 x:0 0 0 2024-04-22T23:23:26.346 INFO:tasks.workunit.client.0.smithi067.stdout:1/341: chown d8/d9/d2d/d4d/d48/c51 9 1 2024-04-22T23:23:26.349 INFO:tasks.workunit.client.0.smithi067.stdout:5/410: rename d5/d28/d36/d40/d57/d61/d6e to d5/d28/d36/d40/d97 0 2024-04-22T23:23:26.349 INFO:tasks.workunit.client.0.smithi067.stdout:4/448: symlink d12/d42/d4b/d70/d8e/d56/d8b/la4 0 2024-04-22T23:23:26.351 INFO:tasks.workunit.client.0.smithi067.stdout:0/330: dread db/d21/d29/f43 [4194304,4194304] 0 2024-04-22T23:23:26.356 INFO:tasks.workunit.client.0.smithi067.stdout:5/411: mkdir d5/d28/d36/d40/d97/d98 0 2024-04-22T23:23:26.357 INFO:tasks.workunit.client.0.smithi067.stdout:5/412: dread - d5/f31 zero size 2024-04-22T23:23:26.358 INFO:tasks.workunit.client.0.smithi067.stdout:0/331: creat db/d21/d29/d2c/f72 x:0 0 0 2024-04-22T23:23:26.359 INFO:tasks.workunit.client.0.smithi067.stdout:2/501: dwrite d1/d10/d33/f5b [0,4194304] 0 2024-04-22T23:23:26.361 INFO:tasks.workunit.client.0.smithi067.stdout:2/502: write d1/d10/d1c/f40 [660286,87247] 0 2024-04-22T23:23:26.363 INFO:tasks.workunit.client.0.smithi067.stdout:7/326: creat d10/f6f x:0 0 0 2024-04-22T23:23:26.363 INFO:tasks.workunit.client.0.smithi067.stdout:3/360: write d3/d4/fc [2202747,15094] 0 2024-04-22T23:23:26.368 INFO:tasks.workunit.client.0.smithi067.stdout:5/413: unlink d5/db/d12/f4a 0 2024-04-22T23:23:26.370 INFO:tasks.workunit.client.0.smithi067.stdout:5/414: chown d5/d28/d36/d40/f20 1106133371 1 2024-04-22T23:23:26.370 INFO:tasks.workunit.client.0.smithi067.stdout:7/327: dwrite d10/d14/d1d/d46/d50/f5c [0,4194304] 0 2024-04-22T23:23:26.372 INFO:tasks.workunit.client.0.smithi067.stdout:5/415: dread - d5/db/d12/d44/f81 zero size 2024-04-22T23:23:26.372 INFO:tasks.workunit.client.0.smithi067.stdout:0/332: chown db/de/d2a/d48/l4f 887245 1 2024-04-22T23:23:26.373 INFO:tasks.workunit.client.0.smithi067.stdout:2/503: mkdir d1/d42/d5f/d9b 0 2024-04-22T23:23:26.374 INFO:tasks.workunit.client.0.smithi067.stdout:7/328: read d10/d47/f68 [3276717,118606] 0 2024-04-22T23:23:26.374 INFO:tasks.workunit.client.0.smithi067.stdout:4/449: truncate d12/d17/d47/f49 8291595 0 2024-04-22T23:23:26.375 INFO:tasks.workunit.client.0.smithi067.stdout:4/450: dread - d12/d17/f60 zero size 2024-04-22T23:23:26.376 INFO:tasks.workunit.client.0.smithi067.stdout:3/361: mknod d3/d4/d28/d4c/d51/c7d 0 2024-04-22T23:23:26.380 INFO:tasks.workunit.client.0.smithi067.stdout:2/504: mknod d1/d10/d1c/d4d/c9c 0 2024-04-22T23:23:26.381 INFO:tasks.workunit.client.0.smithi067.stdout:2/505: dread - d1/d42/d61/f7e zero size 2024-04-22T23:23:26.385 INFO:tasks.workunit.client.0.smithi067.stdout:7/329: truncate d10/d14/d1d/f1f 1241555 0 2024-04-22T23:23:26.386 INFO:tasks.workunit.client.0.smithi067.stdout:4/451: creat d12/d42/d4b/d77/d8c/fa5 x:0 0 0 2024-04-22T23:23:26.390 INFO:tasks.workunit.client.0.smithi067.stdout:5/416: mkdir d5/d3e/d87/d99 0 2024-04-22T23:23:26.391 INFO:tasks.workunit.client.0.smithi067.stdout:1/342: chown d8/d9/d2d/d4d/d70/d34/f39 95032550 1 2024-04-22T23:23:26.393 INFO:tasks.workunit.client.0.smithi067.stdout:2/506: mknod d1/d10/d1c/d6d/c9d 0 2024-04-22T23:23:26.393 INFO:tasks.workunit.client.0.smithi067.stdout:7/330: readlink d10/d14/d1d/d24/l57 0 2024-04-22T23:23:26.396 INFO:tasks.workunit.client.0.smithi067.stdout:5/417: read d5/db/d12/f13 [1613493,24997] 0 2024-04-22T23:23:26.399 INFO:tasks.workunit.client.0.smithi067.stdout:4/452: mknod d12/d42/d4b/d77/d9d/ca6 0 2024-04-22T23:23:26.402 INFO:tasks.workunit.client.0.smithi067.stdout:5/418: unlink d5/db/d12/d44/d47/f8c 0 2024-04-22T23:23:26.403 INFO:tasks.workunit.client.0.smithi067.stdout:2/507: truncate d1/f1f 59826 0 2024-04-22T23:23:26.403 INFO:tasks.workunit.client.0.smithi067.stdout:0/333: truncate db/de/f1c 2212667 0 2024-04-22T23:23:26.404 INFO:tasks.workunit.client.0.smithi067.stdout:7/331: creat d10/d14/d1d/d46/d6a/f70 x:0 0 0 2024-04-22T23:23:26.407 INFO:tasks.workunit.client.0.smithi067.stdout:4/453: creat d12/d42/d4b/d70/d8e/d56/d8b/fa7 x:0 0 0 2024-04-22T23:23:26.408 INFO:tasks.workunit.client.0.smithi067.stdout:1/343: write d8/d9/f3b [765159,96691] 0 2024-04-22T23:23:26.409 INFO:tasks.workunit.client.0.smithi067.stdout:5/419: mkdir d5/d1a/d8b/d9a 0 2024-04-22T23:23:26.410 INFO:tasks.workunit.client.0.smithi067.stdout:5/420: stat d5/f31 0 2024-04-22T23:23:26.411 INFO:tasks.workunit.client.0.smithi067.stdout:2/508: truncate d1/d10/d1c/d6d/f4c 3322974 0 2024-04-22T23:23:26.411 INFO:tasks.workunit.client.0.smithi067.stdout:5/421: readlink d5/d28/d36/d40/l17 0 2024-04-22T23:23:26.411 INFO:tasks.workunit.client.0.smithi067.stdout:2/509: readlink d1/d42/d5f/l96 0 2024-04-22T23:23:26.412 INFO:tasks.workunit.client.0.smithi067.stdout:5/422: rename d5/db to d5/db/d12/d44/d47/d9b 22 2024-04-22T23:23:26.413 INFO:tasks.workunit.client.0.smithi067.stdout:1/344: symlink d8/d9/d2d/l76 0 2024-04-22T23:23:26.413 INFO:tasks.workunit.client.0.smithi067.stdout:5/423: chown d5/d3a/f5d 2992 1 2024-04-22T23:23:26.416 INFO:tasks.workunit.client.0.smithi067.stdout:0/334: link db/d21/d29/d2c/l3b db/d2e/d31/l73 0 2024-04-22T23:23:26.418 INFO:tasks.workunit.client.0.smithi067.stdout:7/332: creat d10/d14/d1d/d26/f71 x:0 0 0 2024-04-22T23:23:26.418 INFO:tasks.workunit.client.0.smithi067.stdout:5/424: chown d5/d28/d36/d40/d57/d61/d89 199934 1 2024-04-22T23:23:26.422 INFO:tasks.workunit.client.0.smithi067.stdout:5/425: write d5/d1a/d8b/f8f [1710805,121281] 0 2024-04-22T23:23:26.423 INFO:tasks.workunit.client.0.smithi067.stdout:2/510: dwrite d1/d10/d41/f93 [0,4194304] 0 2024-04-22T23:23:26.423 INFO:tasks.workunit.client.0.smithi067.stdout:7/333: write d10/d14/d1d/d26/f40 [423432,123573] 0 2024-04-22T23:23:26.424 INFO:tasks.workunit.client.0.smithi067.stdout:4/454: dread ff [0,4194304] 0 2024-04-22T23:23:26.427 INFO:tasks.workunit.client.0.smithi067.stdout:7/334: chown fb 1197072033 1 2024-04-22T23:23:26.430 INFO:tasks.workunit.client.0.smithi067.stdout:0/335: getdents db/d2e/d31/d56 0 2024-04-22T23:23:26.433 INFO:tasks.workunit.client.0.smithi067.stdout:4/455: unlink d12/d17/l21 0 2024-04-22T23:23:26.435 INFO:tasks.workunit.client.0.smithi067.stdout:2/511: rename d1/d10/d33/l51 to d1/d10/d72/d53/l9e 0 2024-04-22T23:23:26.436 INFO:tasks.workunit.client.0.smithi067.stdout:5/426: dwrite d5/d3a/f58 [0,4194304] 0 2024-04-22T23:23:26.438 INFO:tasks.workunit.client.0.smithi067.stdout:5/427: chown d5/d3a/f60 544 1 2024-04-22T23:23:26.440 INFO:tasks.workunit.client.0.smithi067.stdout:4/456: unlink d12/d17/d31/f59 0 2024-04-22T23:23:26.442 INFO:tasks.workunit.client.0.smithi067.stdout:0/336: unlink db/l1f 0 2024-04-22T23:23:26.447 INFO:tasks.workunit.client.0.smithi067.stdout:4/457: creat d12/d42/d4b/fa8 x:0 0 0 2024-04-22T23:23:26.447 INFO:tasks.workunit.client.0.smithi067.stdout:5/428: dwrite d5/d3e/f76 [0,4194304] 0 2024-04-22T23:23:26.451 INFO:tasks.workunit.client.0.smithi067.stdout:2/512: rename d1/l20 to d1/d10/d1c/d6d/d87/l9f 0 2024-04-22T23:23:26.453 INFO:tasks.workunit.client.0.smithi067.stdout:2/513: fdatasync d1/d10/d1c/f5e 0 2024-04-22T23:23:26.453 INFO:tasks.workunit.client.0.smithi067.stdout:5/429: symlink d5/d3e/l9c 0 2024-04-22T23:23:26.453 INFO:tasks.workunit.client.0.smithi067.stdout:4/458: read d12/f58 [200086,27015] 0 2024-04-22T23:23:26.454 INFO:tasks.workunit.client.0.smithi067.stdout:7/335: getdents d10/d14/d1d/d26/d2c 0 2024-04-22T23:23:26.455 INFO:tasks.workunit.client.0.smithi067.stdout:5/430: fdatasync d5/d3a/f60 0 2024-04-22T23:23:26.455 INFO:tasks.workunit.client.0.smithi067.stdout:2/514: chown d1/d10/d1c/c1e 350577762 1 2024-04-22T23:23:26.456 INFO:tasks.workunit.client.0.smithi067.stdout:5/431: write d5/f11 [1919581,58892] 0 2024-04-22T23:23:26.456 INFO:tasks.workunit.client.0.smithi067.stdout:4/459: dread - d12/f38 zero size 2024-04-22T23:23:26.457 INFO:tasks.workunit.client.0.smithi067.stdout:5/432: stat d5/db/d35/c7a 0 2024-04-22T23:23:26.460 INFO:tasks.workunit.client.0.smithi067.stdout:2/515: symlink d1/la0 0 2024-04-22T23:23:26.464 INFO:tasks.workunit.client.0.smithi067.stdout:4/460: dwrite d12/d42/f78 [0,4194304] 0 2024-04-22T23:23:26.464 INFO:tasks.workunit.client.0.smithi067.stdout:5/433: truncate d5/db/d35/f39 398417 0 2024-04-22T23:23:26.467 INFO:tasks.workunit.client.0.smithi067.stdout:4/461: creat d12/d5f/d69/fa9 x:0 0 0 2024-04-22T23:23:26.467 INFO:tasks.workunit.client.0.smithi067.stdout:0/337: dread db/de/d11/f32 [0,4194304] 0 2024-04-22T23:23:26.468 INFO:tasks.workunit.client.0.smithi067.stdout:5/434: getdents d5/d3a 0 2024-04-22T23:23:26.469 INFO:tasks.workunit.client.0.smithi067.stdout:4/462: creat d12/d5f/faa x:0 0 0 2024-04-22T23:23:26.469 INFO:tasks.workunit.client.0.smithi067.stdout:0/338: write db/de/d11/d1a/f38 [4806937,11657] 0 2024-04-22T23:23:26.470 INFO:tasks.workunit.client.0.smithi067.stdout:5/435: write d5/db/d35/f8e [53654,29459] 0 2024-04-22T23:23:26.470 INFO:tasks.workunit.client.0.smithi067.stdout:4/463: write d12/d42/f78 [1510189,86237] 0 2024-04-22T23:23:26.475 INFO:tasks.workunit.client.0.smithi067.stdout:4/464: unlink d12/d42/d4b/d70/l6b 0 2024-04-22T23:23:26.478 INFO:tasks.workunit.client.0.smithi067.stdout:4/465: mknod d12/d5f/d7e/cab 0 2024-04-22T23:23:26.478 INFO:tasks.workunit.client.0.smithi067.stdout:5/436: dwrite d5/db/d35/f6c [0,4194304] 0 2024-04-22T23:23:26.480 INFO:tasks.workunit.client.0.smithi067.stdout:4/466: symlink d12/d2e/lac 0 2024-04-22T23:23:26.481 INFO:tasks.workunit.client.0.smithi067.stdout:4/467: fsync d12/d5f/f87 0 2024-04-22T23:23:26.483 INFO:tasks.workunit.client.0.smithi067.stdout:5/437: creat d5/d28/d36/f9d x:0 0 0 2024-04-22T23:23:26.484 INFO:tasks.workunit.client.0.smithi067.stdout:0/339: dread db/f6a [0,4194304] 0 2024-04-22T23:23:26.484 INFO:tasks.workunit.client.0.smithi067.stdout:4/468: mknod d12/d42/d4b/d77/cad 0 2024-04-22T23:23:26.485 INFO:tasks.workunit.client.0.smithi067.stdout:5/438: write d5/d3a/f58 [1112698,28298] 0 2024-04-22T23:23:26.486 INFO:tasks.workunit.client.0.smithi067.stdout:4/469: dread - d12/d5f/d69/f7c zero size 2024-04-22T23:23:26.486 INFO:tasks.workunit.client.0.smithi067.stdout:5/439: dread - d5/f68 zero size 2024-04-22T23:23:26.486 INFO:tasks.workunit.client.0.smithi067.stdout:0/340: write db/de/d2a/f5e [92439,47431] 0 2024-04-22T23:23:26.488 INFO:tasks.workunit.client.0.smithi067.stdout:4/470: stat d12/d5f/d69/fa9 0 2024-04-22T23:23:26.489 INFO:tasks.workunit.client.0.smithi067.stdout:4/471: dread d12/d2e/f3e [0,4194304] 0 2024-04-22T23:23:26.491 INFO:tasks.workunit.client.0.smithi067.stdout:5/440: link d5/d28/d36/d40/l66 d5/d3e/d87/l9e 0 2024-04-22T23:23:26.492 INFO:tasks.workunit.client.0.smithi067.stdout:6/412: sync 2024-04-22T23:23:26.492 INFO:tasks.workunit.client.0.smithi067.stdout:8/389: sync 2024-04-22T23:23:26.492 INFO:tasks.workunit.client.0.smithi067.stdout:0/341: link db/de/d2a/c51 db/d2e/d31/d56/d70/c74 0 2024-04-22T23:23:26.493 INFO:tasks.workunit.client.0.smithi067.stdout:5/441: chown d5/d28/d36/d40/d57/d5f/c7b 71600461 1 2024-04-22T23:23:26.493 INFO:tasks.workunit.client.0.smithi067.stdout:0/342: chown db/d2e/d31/d56 12 1 2024-04-22T23:23:26.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:26 smithi134 ceph-mon[34609]: pgmap v268: 65 pgs: 65 active+clean; 749 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 8.9 MiB/s rd, 84 MiB/s wr, 333 op/s 2024-04-22T23:23:26.494 INFO:tasks.workunit.client.0.smithi067.stdout:6/413: creat d17/d39/f81 x:0 0 0 2024-04-22T23:23:26.495 INFO:tasks.workunit.client.0.smithi067.stdout:0/343: readlink db/l18 0 2024-04-22T23:23:26.496 INFO:tasks.workunit.client.0.smithi067.stdout:5/442: dread - d5/db/d12/d44/d47/f6d zero size 2024-04-22T23:23:26.498 INFO:tasks.workunit.client.0.smithi067.stdout:0/344: chown db/d21/d29/d2d/f42 98 1 2024-04-22T23:23:26.498 INFO:tasks.workunit.client.0.smithi067.stdout:8/390: dwrite d1/d3/f56 [0,4194304] 0 2024-04-22T23:23:26.502 INFO:tasks.workunit.client.0.smithi067.stdout:8/391: creat d1/d3/d27/f7c x:0 0 0 2024-04-22T23:23:26.503 INFO:tasks.workunit.client.0.smithi067.stdout:5/443: dread d5/d28/d36/d40/f70 [0,4194304] 0 2024-04-22T23:23:26.505 INFO:tasks.workunit.client.0.smithi067.stdout:0/345: mkdir db/d21/d29/d75 0 2024-04-22T23:23:26.505 INFO:tasks.workunit.client.0.smithi067.stdout:5/444: read d5/d3e/f42 [1237868,58434] 0 2024-04-22T23:23:26.508 INFO:tasks.workunit.client.0.smithi067.stdout:5/445: truncate d5/db/d35/f6f 96479 0 2024-04-22T23:23:26.509 INFO:tasks.workunit.client.0.smithi067.stdout:8/392: creat d1/d41/f7d x:0 0 0 2024-04-22T23:23:26.509 INFO:tasks.workunit.client.0.smithi067.stdout:4/472: read fd [3615293,42225] 0 2024-04-22T23:23:26.514 INFO:tasks.workunit.client.0.smithi067.stdout:4/473: write d12/d2e/f57 [2419467,56498] 0 2024-04-22T23:23:26.516 INFO:tasks.workunit.client.0.smithi067.stdout:7/336: dread d10/d14/d1d/d26/d2c/f2e [0,4194304] 0 2024-04-22T23:23:26.519 INFO:tasks.workunit.client.0.smithi067.stdout:5/446: dwrite d5/db/d35/f43 [0,4194304] 0 2024-04-22T23:23:26.521 INFO:tasks.workunit.client.0.smithi067.stdout:5/447: read - d5/d28/f69 zero size 2024-04-22T23:23:26.522 INFO:tasks.workunit.client.0.smithi067.stdout:4/474: mkdir d12/d5f/dae 0 2024-04-22T23:23:26.524 INFO:tasks.workunit.client.0.smithi067.stdout:8/393: dread d1/d41/d46/f60 [0,4194304] 0 2024-04-22T23:23:26.528 INFO:tasks.workunit.client.0.smithi067.stdout:7/337: mknod d10/d14/d1d/d46/d50/c72 0 2024-04-22T23:23:26.540 INFO:tasks.workunit.client.0.smithi067.stdout:4/475: creat d12/d17/d3f/d7a/faf x:0 0 0 2024-04-22T23:23:26.541 INFO:tasks.workunit.client.0.smithi067.stdout:8/394: mkdir d1/d3/d7e 0 2024-04-22T23:23:26.544 INFO:tasks.workunit.client.0.smithi067.stdout:4/476: symlink d12/d42/d4b/d77/d9d/lb0 0 2024-04-22T23:23:26.547 INFO:tasks.workunit.client.0.smithi067.stdout:4/477: symlink d12/d2e/d9b/lb1 0 2024-04-22T23:23:26.548 INFO:tasks.workunit.client.0.smithi067.stdout:4/478: chown d12/d42/d4b/d77/d9d/ca6 0 1 2024-04-22T23:23:26.549 INFO:tasks.workunit.client.0.smithi067.stdout:7/338: getdents d10/d14/d1d/d26 0 2024-04-22T23:23:26.550 INFO:tasks.workunit.client.0.smithi067.stdout:8/395: mknod d1/d3/d7e/c7f 0 2024-04-22T23:23:26.551 INFO:tasks.workunit.client.0.smithi067.stdout:4/479: dread d12/f46 [0,4194304] 0 2024-04-22T23:23:26.554 INFO:tasks.workunit.client.0.smithi067.stdout:7/339: rmdir d10/d14/d1d/d24 39 2024-04-22T23:23:26.555 INFO:tasks.workunit.client.0.smithi067.stdout:4/480: stat d12/f54 0 2024-04-22T23:23:26.556 INFO:tasks.workunit.client.0.smithi067.stdout:1/345: dread d8/fc [0,4194304] 0 2024-04-22T23:23:26.557 INFO:tasks.workunit.client.0.smithi067.stdout:1/346: write d8/d9/f2c [1370487,125761] 0 2024-04-22T23:23:26.558 INFO:tasks.workunit.client.0.smithi067.stdout:7/340: mknod d10/d14/d1d/d26/d2c/d41/c73 0 2024-04-22T23:23:26.559 INFO:tasks.workunit.client.0.smithi067.stdout:7/341: readlink d10/d14/d1d/d26/d2c/l4b 0 2024-04-22T23:23:26.560 INFO:tasks.workunit.client.0.smithi067.stdout:8/396: rename d1/d41/d46/d64 to d1/dd/d80 0 2024-04-22T23:23:26.562 INFO:tasks.workunit.client.0.smithi067.stdout:4/481: symlink d12/d42/d4b/lb2 0 2024-04-22T23:23:26.563 INFO:tasks.workunit.client.0.smithi067.stdout:0/346: rmdir db/d2e 39 2024-04-22T23:23:26.564 INFO:tasks.workunit.client.0.smithi067.stdout:1/347: symlink d8/d9/d2d/d4d/d70/d34/d60/l77 0 2024-04-22T23:23:26.565 INFO:tasks.workunit.client.0.smithi067.stdout:7/342: symlink d10/d14/d1d/d46/d6a/l74 0 2024-04-22T23:23:26.568 INFO:tasks.workunit.client.0.smithi067.stdout:8/397: creat d1/d3/d27/f81 x:0 0 0 2024-04-22T23:23:26.570 INFO:tasks.workunit.client.0.smithi067.stdout:1/348: write d8/d9/f25 [1243213,57165] 0 2024-04-22T23:23:26.571 INFO:tasks.workunit.client.0.smithi067.stdout:7/343: truncate d10/d47/f68 3061650 0 2024-04-22T23:23:26.571 INFO:tasks.workunit.client.0.smithi067.stdout:7/344: chown d10/d18/f63 39495 1 2024-04-22T23:23:26.572 INFO:tasks.workunit.client.0.smithi067.stdout:7/345: dread - d10/d18/f63 zero size 2024-04-22T23:23:26.576 INFO:tasks.workunit.client.0.smithi067.stdout:0/347: rename db/d2e/d31/d56/l68 to db/de/d11/l76 0 2024-04-22T23:23:26.578 INFO:tasks.workunit.client.0.smithi067.stdout:6/414: dwrite d17/d39/f3f [4194304,4194304] 0 2024-04-22T23:23:26.579 INFO:tasks.workunit.client.0.smithi067.stdout:1/349: rename d8/d9/d2f/d33/d66/f6b to d8/d9/d2f/d59/f78 0 2024-04-22T23:23:26.581 INFO:tasks.workunit.client.0.smithi067.stdout:0/348: chown db/d2e/d31/c5c 934 1 2024-04-22T23:23:26.593 INFO:tasks.workunit.client.0.smithi067.stdout:5/448: write d5/db/d12/f22 [519769,121907] 0 2024-04-22T23:23:26.595 INFO:tasks.workunit.client.0.smithi067.stdout:0/349: link db/d21/d29/d2c/l3e db/d2e/d46/l77 0 2024-04-22T23:23:26.595 INFO:tasks.workunit.client.0.smithi067.stdout:0/350: readlink db/d21/d29/d2c/l65 0 2024-04-22T23:23:26.597 INFO:tasks.workunit.client.0.smithi067.stdout:6/415: rename d17/d39/c6b to d17/c82 0 2024-04-22T23:23:26.602 INFO:tasks.workunit.client.0.smithi067.stdout:1/350: dread d8/d12/d40/f69 [0,4194304] 0 2024-04-22T23:23:26.603 INFO:tasks.workunit.client.0.smithi067.stdout:1/351: chown d8/d9/c1a 1560640 1 2024-04-22T23:23:26.604 INFO:tasks.workunit.client.0.smithi067.stdout:1/352: chown d8/d9/f42 1 1 2024-04-22T23:23:26.610 INFO:tasks.workunit.client.0.smithi067.stdout:5/449: link d5/db/c73 d5/d3e/d90/c9f 0 2024-04-22T23:23:26.614 INFO:tasks.workunit.client.0.smithi067.stdout:0/351: getdents db/de/d2a 0 2024-04-22T23:23:26.615 INFO:tasks.workunit.client.0.smithi067.stdout:1/353: getdents d8/d9/d2d/d4d/d70/d34/d5d 0 2024-04-22T23:23:26.625 INFO:tasks.workunit.client.0.smithi067.stdout:5/450: symlink d5/db/d12/d8a/la0 0 2024-04-22T23:23:26.625 INFO:tasks.workunit.client.0.smithi067.stdout:0/352: dread db/fd [0,4194304] 0 2024-04-22T23:23:26.625 INFO:tasks.workunit.client.0.smithi067.stdout:6/416: dread d17/f24 [0,4194304] 0 2024-04-22T23:23:26.625 INFO:tasks.workunit.client.0.smithi067.stdout:8/398: dwrite d1/d3/d10/f29 [0,4194304] 0 2024-04-22T23:23:26.629 INFO:tasks.workunit.client.0.smithi067.stdout:8/399: truncate d1/dd/f53 769949 0 2024-04-22T23:23:26.631 INFO:tasks.workunit.client.0.smithi067.stdout:5/451: unlink d5/f1b 0 2024-04-22T23:23:26.633 INFO:tasks.workunit.client.0.smithi067.stdout:5/452: write d5/fa [4943762,45212] 0 2024-04-22T23:23:26.635 INFO:tasks.workunit.client.0.smithi067.stdout:8/400: creat d1/d3/d27/f82 x:0 0 0 2024-04-22T23:23:26.636 INFO:tasks.workunit.client.0.smithi067.stdout:6/417: mkdir d17/d83 0 2024-04-22T23:23:26.637 INFO:tasks.workunit.client.0.smithi067.stdout:8/401: chown d1/d41/l4f 149617 1 2024-04-22T23:23:26.639 INFO:tasks.workunit.client.0.smithi067.stdout:5/453: mknod d5/d1a/d8b/ca1 0 2024-04-22T23:23:26.641 INFO:tasks.workunit.client.0.smithi067.stdout:5/454: write d5/db/d35/f8e [16994,53804] 0 2024-04-22T23:23:26.642 INFO:tasks.workunit.client.0.smithi067.stdout:8/402: creat d1/d41/f83 x:0 0 0 2024-04-22T23:23:26.642 INFO:tasks.workunit.client.0.smithi067.stdout:1/354: dread d8/d9/d2f/d33/d5c/f65 [0,4194304] 0 2024-04-22T23:23:26.645 INFO:tasks.workunit.client.0.smithi067.stdout:0/353: dread db/d21/d29/d2c/f3d [0,4194304] 0 2024-04-22T23:23:26.645 INFO:tasks.workunit.client.0.smithi067.stdout:6/418: write d17/d39/d3e/d46/d62/f72 [2444532,28293] 0 2024-04-22T23:23:26.646 INFO:tasks.workunit.client.0.smithi067.stdout:6/419: fdatasync d17/f42 0 2024-04-22T23:23:26.649 INFO:tasks.workunit.client.0.smithi067.stdout:0/354: chown db/d21/c47 298795 1 2024-04-22T23:23:26.651 INFO:tasks.workunit.client.0.smithi067.stdout:8/403: dread d1/dd/f20 [0,4194304] 0 2024-04-22T23:23:26.652 INFO:tasks.workunit.client.0.smithi067.stdout:0/355: symlink db/de/d11/l78 0 2024-04-22T23:23:26.654 INFO:tasks.workunit.client.0.smithi067.stdout:6/420: truncate d17/d39/f49 1527988 0 2024-04-22T23:23:26.654 INFO:tasks.workunit.client.0.smithi067.stdout:1/355: getdents d8/d12 0 2024-04-22T23:23:26.654 INFO:tasks.workunit.client.0.smithi067.stdout:7/346: dwrite d10/d47/f68 [0,4194304] 0 2024-04-22T23:23:26.655 INFO:tasks.workunit.client.0.smithi067.stdout:5/455: dread d5/d28/d36/d40/f20 [0,4194304] 0 2024-04-22T23:23:26.656 INFO:tasks.workunit.client.0.smithi067.stdout:1/356: fdatasync d8/d9/f2c 0 2024-04-22T23:23:26.658 INFO:tasks.workunit.client.0.smithi067.stdout:7/347: dread - d10/d14/d1d/f42 zero size 2024-04-22T23:23:26.658 INFO:tasks.workunit.client.0.smithi067.stdout:1/357: chown d8/d9/d2f/d33/f3e 18272 1 2024-04-22T23:23:26.659 INFO:tasks.workunit.client.0.smithi067.stdout:0/356: creat db/f79 x:0 0 0 2024-04-22T23:23:26.660 INFO:tasks.workunit.client.0.smithi067.stdout:6/421: mknod d17/d39/d3e/d46/d62/c84 0 2024-04-22T23:23:26.661 INFO:tasks.workunit.client.0.smithi067.stdout:1/358: write d8/d12/d36/f72 [282273,11400] 0 2024-04-22T23:23:26.662 INFO:tasks.workunit.client.0.smithi067.stdout:7/348: dread - d10/d14/d1d/d26/d2c/d41/f65 zero size 2024-04-22T23:23:26.662 INFO:tasks.workunit.client.0.smithi067.stdout:7/349: stat fb 0 2024-04-22T23:23:26.664 INFO:tasks.workunit.client.0.smithi067.stdout:6/422: creat d17/d39/d3e/d46/d62/f85 x:0 0 0 2024-04-22T23:23:26.665 INFO:tasks.workunit.client.0.smithi067.stdout:1/359: chown d8/d4a 16332653 1 2024-04-22T23:23:26.665 INFO:tasks.workunit.client.0.smithi067.stdout:6/423: readlink d17/l37 0 2024-04-22T23:23:26.666 INFO:tasks.workunit.client.0.smithi067.stdout:1/360: chown d8/d12/f54 1107025043 1 2024-04-22T23:23:26.668 INFO:tasks.workunit.client.0.smithi067.stdout:0/357: mkdir db/d21/d7a 0 2024-04-22T23:23:26.669 INFO:tasks.workunit.client.0.smithi067.stdout:8/404: dread d1/dd/d80/d6f/f34 [0,4194304] 0 2024-04-22T23:23:26.670 INFO:tasks.workunit.client.0.smithi067.stdout:6/424: rename f6 to d17/d83/f86 0 2024-04-22T23:23:26.671 INFO:tasks.workunit.client.0.smithi067.stdout:8/405: chown d1/d3/d10/d3b/f5e 201054675 1 2024-04-22T23:23:26.672 INFO:tasks.workunit.client.0.smithi067.stdout:7/350: mknod d10/d14/c75 0 2024-04-22T23:23:26.677 INFO:tasks.workunit.client.0.smithi067.stdout:6/425: mknod d17/d39/d3e/d46/c87 0 2024-04-22T23:23:26.680 INFO:tasks.workunit.client.0.smithi067.stdout:7/351: dwrite d10/d18/d35/f66 [0,4194304] 0 2024-04-22T23:23:26.680 INFO:tasks.workunit.client.0.smithi067.stdout:1/361: dwrite d8/d9/d2f/d33/d5c/f65 [0,4194304] 0 2024-04-22T23:23:26.684 INFO:tasks.workunit.client.0.smithi067.stdout:8/406: link d1/d3/d27/f81 d1/d3/d10/d3b/f84 0 2024-04-22T23:23:26.685 INFO:tasks.workunit.client.0.smithi067.stdout:5/456: write d5/db/d12/f25 [685055,74168] 0 2024-04-22T23:23:26.691 INFO:tasks.workunit.client.0.smithi067.stdout:1/362: creat d8/d9/d2f/f79 x:0 0 0 2024-04-22T23:23:26.692 INFO:tasks.workunit.client.0.smithi067.stdout:1/363: chown d8/d9/d2f/d33/d5c/f6d 540 1 2024-04-22T23:23:26.692 INFO:tasks.workunit.client.0.smithi067.stdout:7/352: link d10/d14/d1d/d26/f5d d10/d14/d1d/f76 0 2024-04-22T23:23:26.694 INFO:tasks.workunit.client.0.smithi067.stdout:7/353: chown d10/d14/d1d/d26/f3a 8 1 2024-04-22T23:23:26.694 INFO:tasks.workunit.client.0.smithi067.stdout:1/364: symlink d8/d12/d36/l7a 0 2024-04-22T23:23:26.695 INFO:tasks.workunit.client.0.smithi067.stdout:7/354: mknod d10/d18/d35/d62/c77 0 2024-04-22T23:23:26.700 INFO:tasks.workunit.client.0.smithi067.stdout:7/355: dwrite fb [0,4194304] 0 2024-04-22T23:23:26.704 INFO:tasks.workunit.client.0.smithi067.stdout:7/356: mknod d10/d14/c78 0 2024-04-22T23:23:26.706 INFO:tasks.workunit.client.0.smithi067.stdout:5/457: dread d5/db/d12/f22 [0,4194304] 0 2024-04-22T23:23:26.706 INFO:tasks.workunit.client.0.smithi067.stdout:7/357: chown d10/d14/d1d/c6b 467 1 2024-04-22T23:23:26.707 INFO:tasks.workunit.client.0.smithi067.stdout:6/426: dread f12 [0,4194304] 0 2024-04-22T23:23:26.708 INFO:tasks.workunit.client.0.smithi067.stdout:6/427: dread - d17/d39/f81 zero size 2024-04-22T23:23:26.709 INFO:tasks.workunit.client.0.smithi067.stdout:5/458: symlink d5/d1a/la2 0 2024-04-22T23:23:26.710 INFO:tasks.workunit.client.0.smithi067.stdout:7/358: symlink d10/d18/l79 0 2024-04-22T23:23:26.710 INFO:tasks.workunit.client.0.smithi067.stdout:5/459: chown d5/db/d12/d44/d47/l7c 40365613 1 2024-04-22T23:23:26.711 INFO:tasks.workunit.client.0.smithi067.stdout:6/428: mkdir d17/d39/d3e/d46/d62/d88 0 2024-04-22T23:23:26.712 INFO:tasks.workunit.client.0.smithi067.stdout:5/460: creat d5/d3a/d4d/fa3 x:0 0 0 2024-04-22T23:23:26.714 INFO:tasks.workunit.client.0.smithi067.stdout:6/429: creat d17/d39/f89 x:0 0 0 2024-04-22T23:23:26.714 INFO:tasks.workunit.client.0.smithi067.stdout:5/461: dread - d5/db/d12/f2c zero size 2024-04-22T23:23:26.716 INFO:tasks.workunit.client.0.smithi067.stdout:5/462: write d5/db/d12/d44/d47/f4f [707577,18966] 0 2024-04-22T23:23:26.718 INFO:tasks.workunit.client.0.smithi067.stdout:6/430: creat d17/d39/d53/f8a x:0 0 0 2024-04-22T23:23:26.721 INFO:tasks.workunit.client.0.smithi067.stdout:6/431: creat d17/d39/d3e/f8b x:0 0 0 2024-04-22T23:23:26.722 INFO:tasks.workunit.client.0.smithi067.stdout:6/432: creat d17/d39/f8c x:0 0 0 2024-04-22T23:23:26.723 INFO:tasks.workunit.client.0.smithi067.stdout:6/433: dread - d17/d39/d3e/f8b zero size 2024-04-22T23:23:26.728 INFO:tasks.workunit.client.0.smithi067.stdout:6/434: dwrite d17/d39/d53/f74 [0,4194304] 0 2024-04-22T23:23:26.729 INFO:tasks.workunit.client.0.smithi067.stdout:6/435: fsync d17/f25 0 2024-04-22T23:23:26.729 INFO:tasks.workunit.client.0.smithi067.stdout:9/283: sync 2024-04-22T23:23:26.729 INFO:tasks.workunit.client.0.smithi067.stdout:3/362: sync 2024-04-22T23:23:26.730 INFO:tasks.workunit.client.0.smithi067.stdout:3/363: chown d3/d4/d1b/l65 5 1 2024-04-22T23:23:26.734 INFO:tasks.workunit.client.0.smithi067.stdout:6/436: mkdir d17/d39/d3a/d8d 0 2024-04-22T23:23:26.735 INFO:tasks.workunit.client.0.smithi067.stdout:3/364: mkdir d3/d4/d12/d7e 0 2024-04-22T23:23:26.736 INFO:tasks.workunit.client.0.smithi067.stdout:6/437: read d17/d39/d53/f55 [497180,38755] 0 2024-04-22T23:23:26.737 INFO:tasks.workunit.client.0.smithi067.stdout:9/284: dwrite d5/fe [4194304,4194304] 0 2024-04-22T23:23:26.739 INFO:tasks.workunit.client.0.smithi067.stdout:9/285: mknod d5/dd/d18/d39/d4d/c5e 0 2024-04-22T23:23:26.741 INFO:tasks.workunit.client.0.smithi067.stdout:3/365: mkdir d3/d4/d10/d38/d53/d55/d74/d7f 0 2024-04-22T23:23:26.742 INFO:tasks.workunit.client.0.smithi067.stdout:9/286: rmdir d5/d21/d2c 39 2024-04-22T23:23:26.744 INFO:tasks.workunit.client.0.smithi067.stdout:9/287: dread - d5/dd/d18/d39/d3e/d3f/f52 zero size 2024-04-22T23:23:26.745 INFO:tasks.workunit.client.0.smithi067.stdout:6/438: dwrite d17/f35 [0,4194304] 0 2024-04-22T23:23:26.746 INFO:tasks.workunit.client.0.smithi067.stdout:9/288: fdatasync d5/dd/d18/d39/d49/d54/f4e 0 2024-04-22T23:23:26.753 INFO:tasks.workunit.client.0.smithi067.stdout:6/439: rename d17/d39/d3a/l6c to d17/l8e 0 2024-04-22T23:23:26.755 INFO:tasks.workunit.client.0.smithi067.stdout:6/440: write d17/f24 [3624080,89526] 0 2024-04-22T23:23:26.757 INFO:tasks.workunit.client.0.smithi067.stdout:6/441: symlink d17/d39/d3e/l8f 0 2024-04-22T23:23:26.780 INFO:tasks.workunit.client.0.smithi067.stdout:0/358: write db/de/f1c [831766,125533] 0 2024-04-22T23:23:26.781 INFO:tasks.workunit.client.0.smithi067.stdout:0/359: stat db/d2e/c4a 0 2024-04-22T23:23:26.784 INFO:tasks.workunit.client.0.smithi067.stdout:0/360: dread db/f6a [0,4194304] 0 2024-04-22T23:23:26.786 INFO:tasks.workunit.client.0.smithi067.stdout:0/361: rmdir db/de/d2a/d48 39 2024-04-22T23:23:26.790 INFO:tasks.workunit.client.0.smithi067.stdout:0/362: creat db/d21/f7b x:0 0 0 2024-04-22T23:23:26.792 INFO:tasks.workunit.client.0.smithi067.stdout:0/363: write db/d2e/f5d [843296,82571] 0 2024-04-22T23:23:26.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:26 smithi067 ceph-mon[26453]: pgmap v268: 65 pgs: 65 active+clean; 749 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 8.9 MiB/s rd, 84 MiB/s wr, 333 op/s 2024-04-22T23:23:26.795 INFO:tasks.workunit.client.0.smithi067.stdout:0/364: getdents db/d21/d7a 0 2024-04-22T23:23:26.795 INFO:tasks.workunit.client.0.smithi067.stdout:5/463: write d5/db/d12/f2c [574602,103299] 0 2024-04-22T23:23:26.804 INFO:tasks.workunit.client.0.smithi067.stdout:9/289: truncate d5/dd/d18/d39/d49/d54/f4e 300706 0 2024-04-22T23:23:26.805 INFO:tasks.workunit.client.0.smithi067.stdout:9/290: chown d5/d21/d47/l4f 115 1 2024-04-22T23:23:26.807 INFO:tasks.workunit.client.0.smithi067.stdout:3/366: dwrite d3/d4/d10/d19/f34 [0,4194304] 0 2024-04-22T23:23:26.809 INFO:tasks.workunit.client.0.smithi067.stdout:9/291: getdents d5/dd/d18/d5c 0 2024-04-22T23:23:26.811 INFO:tasks.workunit.client.0.smithi067.stdout:9/292: symlink d5/dd/d18/d39/d49/d54/l5f 0 2024-04-22T23:23:26.820 INFO:tasks.workunit.client.0.smithi067.stdout:9/293: dwrite d5/dd/d18/d39/d49/d54/f3d [0,4194304] 0 2024-04-22T23:23:26.823 INFO:tasks.workunit.client.0.smithi067.stdout:6/442: rename d17/d39/d3e to d17/d39/d53/d90 0 2024-04-22T23:23:26.823 INFO:tasks.workunit.client.0.smithi067.stdout:3/367: dread d3/d4/f79 [0,4194304] 0 2024-04-22T23:23:26.824 INFO:tasks.workunit.client.0.smithi067.stdout:0/365: truncate db/de/d11/f3c 1152252 0 2024-04-22T23:23:26.824 INFO:tasks.workunit.client.0.smithi067.stdout:5/464: truncate d5/d1a/d8b/f8f 129303 0 2024-04-22T23:23:26.827 INFO:tasks.workunit.client.0.smithi067.stdout:0/366: chown db/d21/c47 23322141 1 2024-04-22T23:23:26.833 INFO:tasks.workunit.client.0.smithi067.stdout:6/443: symlink d17/l91 0 2024-04-22T23:23:26.834 INFO:tasks.workunit.client.0.smithi067.stdout:0/367: write db/de/d11/d1a/f64 [48563,1346] 0 2024-04-22T23:23:26.835 INFO:tasks.workunit.client.0.smithi067.stdout:9/294: dwrite d5/f36 [0,4194304] 0 2024-04-22T23:23:26.838 INFO:tasks.workunit.client.0.smithi067.stdout:9/295: write d5/dd/f1a [4646140,106054] 0 2024-04-22T23:23:26.841 INFO:tasks.workunit.client.0.smithi067.stdout:5/465: dwrite d5/d3a/f60 [0,4194304] 0 2024-04-22T23:23:26.842 INFO:tasks.workunit.client.0.smithi067.stdout:9/296: dread - d5/dd/d18/d3c/d40/f50 zero size 2024-04-22T23:23:26.844 INFO:tasks.workunit.client.0.smithi067.stdout:0/368: creat db/d2e/f7c x:0 0 0 2024-04-22T23:23:26.850 INFO:tasks.workunit.client.0.smithi067.stdout:9/297: dread d5/f36 [0,4194304] 0 2024-04-22T23:23:26.852 INFO:tasks.workunit.client.0.smithi067.stdout:6/444: creat d17/d39/d53/d90/d46/d62/d88/f92 x:0 0 0 2024-04-22T23:23:26.854 INFO:tasks.workunit.client.0.smithi067.stdout:0/369: creat db/d21/d29/d2c/f7d x:0 0 0 2024-04-22T23:23:26.859 INFO:tasks.workunit.client.0.smithi067.stdout:5/466: creat d5/fa4 x:0 0 0 2024-04-22T23:23:26.859 INFO:tasks.workunit.client.0.smithi067.stdout:0/370: chown db/de/d11/d1a/l1d 123524201 1 2024-04-22T23:23:26.859 INFO:tasks.workunit.client.0.smithi067.stdout:3/368: dwrite d3/d4/d10/d1f/f3f [0,4194304] 0 2024-04-22T23:23:26.863 INFO:tasks.workunit.client.0.smithi067.stdout:0/371: write db/de/f1b [3860028,9739] 0 2024-04-22T23:23:26.867 INFO:tasks.workunit.client.0.smithi067.stdout:0/372: fdatasync db/d2e/d31/f3a 0 2024-04-22T23:23:26.871 INFO:tasks.workunit.client.0.smithi067.stdout:6/445: dwrite d17/d39/d53/f6a [0,4194304] 0 2024-04-22T23:23:26.871 INFO:tasks.workunit.client.0.smithi067.stdout:5/467: rename d5/d3e/l9c to d5/d3e/d90/la5 0 2024-04-22T23:23:26.875 INFO:tasks.workunit.client.0.smithi067.stdout:9/298: dwrite d5/dd/f32 [0,4194304] 0 2024-04-22T23:23:26.880 INFO:tasks.workunit.client.0.smithi067.stdout:3/369: rename d3/d4/fe to d3/d4/d10/d19/f80 0 2024-04-22T23:23:26.881 INFO:tasks.workunit.client.0.smithi067.stdout:3/370: chown d3/d4/d10/d38/d53/d55 914605 1 2024-04-22T23:23:26.884 INFO:tasks.workunit.client.0.smithi067.stdout:6/446: read fe [2560705,46196] 0 2024-04-22T23:23:26.886 INFO:tasks.workunit.client.0.smithi067.stdout:3/371: dread d3/fa [0,4194304] 0 2024-04-22T23:23:26.889 INFO:tasks.workunit.client.0.smithi067.stdout:5/468: link d5/db/f2f d5/d3a/fa6 0 2024-04-22T23:23:26.891 INFO:tasks.workunit.client.0.smithi067.stdout:6/447: stat d17/d39/f49 0 2024-04-22T23:23:26.894 INFO:tasks.workunit.client.0.smithi067.stdout:5/469: mknod d5/d28/ca7 0 2024-04-22T23:23:26.896 INFO:tasks.workunit.client.0.smithi067.stdout:0/373: dwrite db/d21/f4c [0,4194304] 0 2024-04-22T23:23:26.897 INFO:tasks.workunit.client.0.smithi067.stdout:6/448: creat d17/d39/d53/f93 x:0 0 0 2024-04-22T23:23:26.900 INFO:tasks.workunit.client.0.smithi067.stdout:0/374: creat db/de/d11/f7e x:0 0 0 2024-04-22T23:23:26.902 INFO:tasks.workunit.client.0.smithi067.stdout:6/449: rmdir d17/d39/d53/d90/d46/d62/d88 39 2024-04-22T23:23:26.905 INFO:tasks.workunit.client.0.smithi067.stdout:5/470: rmdir d5/d28/d36/d40/d97/d98 0 2024-04-22T23:23:26.907 INFO:tasks.workunit.client.0.smithi067.stdout:9/299: dwrite d5/dd/d18/d39/d3e/f43 [0,4194304] 0 2024-04-22T23:23:26.907 INFO:tasks.workunit.client.0.smithi067.stdout:0/375: write db/f6a [4200867,47803] 0 2024-04-22T23:23:26.909 INFO:tasks.workunit.client.0.smithi067.stdout:3/372: write d3/d4/d10/d38/d53/f58 [1425835,50130] 0 2024-04-22T23:23:26.910 INFO:tasks.workunit.client.0.smithi067.stdout:9/300: truncate d5/dd/d18/d3c/d40/f50 798642 0 2024-04-22T23:23:26.911 INFO:tasks.workunit.client.0.smithi067.stdout:5/471: creat d5/d1a/fa8 x:0 0 0 2024-04-22T23:23:26.911 INFO:tasks.workunit.client.0.smithi067.stdout:5/472: dread - d5/fa4 zero size 2024-04-22T23:23:26.913 INFO:tasks.workunit.client.0.smithi067.stdout:6/450: link fe d17/d39/d53/d90/f94 0 2024-04-22T23:23:26.917 INFO:tasks.workunit.client.0.smithi067.stdout:9/301: unlink d5/dd/d18/d39/d4d/c5e 0 2024-04-22T23:23:26.922 INFO:tasks.workunit.client.0.smithi067.stdout:6/451: write d17/d39/f4b [433294,90947] 0 2024-04-22T23:23:26.923 INFO:tasks.workunit.client.0.smithi067.stdout:0/376: rename db/de/d11/d1a/c5b to db/de/d2a/c7f 0 2024-04-22T23:23:26.924 INFO:tasks.workunit.client.0.smithi067.stdout:9/302: dread - d5/d21/d2c/f46 zero size 2024-04-22T23:23:26.924 INFO:tasks.workunit.client.0.smithi067.stdout:5/473: dwrite d5/db/d12/d44/d47/f4f [0,4194304] 0 2024-04-22T23:23:26.926 INFO:tasks.workunit.client.0.smithi067.stdout:3/373: getdents d3/d4/d10/d38/d53/d55 0 2024-04-22T23:23:26.929 INFO:tasks.workunit.client.0.smithi067.stdout:0/377: mknod db/de/c80 0 2024-04-22T23:23:26.930 INFO:tasks.workunit.client.0.smithi067.stdout:3/374: fsync d3/d4/f16 0 2024-04-22T23:23:26.933 INFO:tasks.workunit.client.0.smithi067.stdout:6/452: dwrite d17/f42 [0,4194304] 0 2024-04-22T23:23:26.935 INFO:tasks.workunit.client.0.smithi067.stdout:0/378: truncate db/de/d2a/f5e 297917 0 2024-04-22T23:23:26.936 INFO:tasks.workunit.client.0.smithi067.stdout:6/453: fsync d17/d39/f69 0 2024-04-22T23:23:26.936 INFO:tasks.workunit.client.0.smithi067.stdout:9/303: dread d5/dd/d18/d39/d3e/f43 [0,4194304] 0 2024-04-22T23:23:26.939 INFO:tasks.workunit.client.0.smithi067.stdout:0/379: symlink db/de/d2a/l81 0 2024-04-22T23:23:26.940 INFO:tasks.workunit.client.0.smithi067.stdout:3/375: getdents d3/d4/d28/d40 0 2024-04-22T23:23:26.942 INFO:tasks.workunit.client.0.smithi067.stdout:6/454: mknod d17/d39/c95 0 2024-04-22T23:23:26.943 INFO:tasks.workunit.client.0.smithi067.stdout:6/455: chown d17/d39/f4b 1019 1 2024-04-22T23:23:26.943 INFO:tasks.workunit.client.0.smithi067.stdout:4/482: sync 2024-04-22T23:23:26.943 INFO:tasks.workunit.client.0.smithi067.stdout:2/516: sync 2024-04-22T23:23:26.944 INFO:tasks.workunit.client.0.smithi067.stdout:9/304: creat d5/d21/d47/d57/f60 x:0 0 0 2024-04-22T23:23:26.945 INFO:tasks.workunit.client.0.smithi067.stdout:9/305: truncate d5/f9 9366978 0 2024-04-22T23:23:26.947 INFO:tasks.workunit.client.0.smithi067.stdout:0/380: rename db/de/f4d to db/d2e/d46/f82 0 2024-04-22T23:23:26.949 INFO:tasks.workunit.client.0.smithi067.stdout:5/474: write d5/d3e/f77 [240048,91957] 0 2024-04-22T23:23:26.953 INFO:tasks.workunit.client.0.smithi067.stdout:2/517: mkdir d1/d10/d72/d68/da1 0 2024-04-22T23:23:26.955 INFO:tasks.workunit.client.0.smithi067.stdout:4/483: dwrite d12/d5f/f87 [0,4194304] 0 2024-04-22T23:23:26.959 INFO:tasks.workunit.client.0.smithi067.stdout:2/518: fsync d1/d12/f15 0 2024-04-22T23:23:26.959 INFO:tasks.workunit.client.0.smithi067.stdout:0/381: dwrite db/de/d11/d1a/f64 [0,4194304] 0 2024-04-22T23:23:26.965 INFO:tasks.workunit.client.0.smithi067.stdout:3/376: write d3/d4/d10/d38/f57 [3793214,31491] 0 2024-04-22T23:23:26.965 INFO:tasks.workunit.client.0.smithi067.stdout:2/519: read d1/d10/f2e [2964640,88403] 0 2024-04-22T23:23:26.965 INFO:tasks.workunit.client.0.smithi067.stdout:6/456: write d17/d39/f49 [341716,49137] 0 2024-04-22T23:23:26.966 INFO:tasks.workunit.client.0.smithi067.stdout:4/484: mknod d12/d42/d4b/d70/cb3 0 2024-04-22T23:23:26.970 INFO:tasks.workunit.client.0.smithi067.stdout:0/382: unlink db/d21/d29/d2c/f41 0 2024-04-22T23:23:26.972 INFO:tasks.workunit.client.0.smithi067.stdout:6/457: fdatasync d17/f33 0 2024-04-22T23:23:26.974 INFO:tasks.workunit.client.0.smithi067.stdout:2/520: mkdir d1/d10/d41/da2 0 2024-04-22T23:23:26.975 INFO:tasks.workunit.client.0.smithi067.stdout:5/475: mkdir d5/d3a/d4d/da9 0 2024-04-22T23:23:26.976 INFO:tasks.workunit.client.0.smithi067.stdout:4/485: dwrite d12/d42/f83 [0,4194304] 0 2024-04-22T23:23:26.979 INFO:tasks.workunit.client.0.smithi067.stdout:3/377: rename d3/d4/d10/d19/l36 to d3/d4/d10/d38/d53/d55/d5c/d70/l81 0 2024-04-22T23:23:26.979 INFO:tasks.workunit.client.0.smithi067.stdout:4/486: chown d12/d42/d4b/l82 171 1 2024-04-22T23:23:26.980 INFO:tasks.workunit.client.0.smithi067.stdout:9/306: dread d5/dd/d18/d39/d3e/f53 [0,4194304] 0 2024-04-22T23:23:26.981 INFO:tasks.workunit.client.0.smithi067.stdout:4/487: chown d12/d42/d4b/d70/d8e/d56 11776489 1 2024-04-22T23:23:26.984 INFO:tasks.workunit.client.0.smithi067.stdout:5/476: mknod d5/db/d12/caa 0 2024-04-22T23:23:26.985 INFO:tasks.workunit.client.0.smithi067.stdout:0/383: creat db/d21/d29/d75/f83 x:0 0 0 2024-04-22T23:23:26.986 INFO:tasks.workunit.client.0.smithi067.stdout:0/384: readlink db/l18 0 2024-04-22T23:23:26.987 INFO:tasks.workunit.client.0.smithi067.stdout:4/488: dread d12/d17/f2f [0,4194304] 0 2024-04-22T23:23:26.988 INFO:tasks.workunit.client.0.smithi067.stdout:9/307: unlink d5/dd/d18/c35 0 2024-04-22T23:23:26.989 INFO:tasks.workunit.client.0.smithi067.stdout:6/458: unlink d17/d39/c71 0 2024-04-22T23:23:26.990 INFO:tasks.workunit.client.0.smithi067.stdout:5/477: symlink d5/db/d12/lab 0 2024-04-22T23:23:26.992 INFO:tasks.workunit.client.0.smithi067.stdout:4/489: creat d12/d42/d4b/d77/d9d/fb4 x:0 0 0 2024-04-22T23:23:26.993 INFO:tasks.workunit.client.0.smithi067.stdout:0/385: symlink db/de/l84 0 2024-04-22T23:23:26.994 INFO:tasks.workunit.client.0.smithi067.stdout:2/521: dread d1/fc [0,4194304] 0 2024-04-22T23:23:26.995 INFO:tasks.workunit.client.0.smithi067.stdout:6/459: write d17/d39/d53/d90/d46/d62/d88/f92 [778257,85341] 0 2024-04-22T23:23:26.996 INFO:tasks.workunit.client.0.smithi067.stdout:9/308: mknod d5/dd/d18/d39/d3e/c61 0 2024-04-22T23:23:26.997 INFO:tasks.workunit.client.0.smithi067.stdout:6/460: readlink d17/l91 0 2024-04-22T23:23:26.997 INFO:tasks.workunit.client.0.smithi067.stdout:5/478: creat d5/d1a/d8b/d9a/fac x:0 0 0 2024-04-22T23:23:26.998 INFO:tasks.workunit.client.0.smithi067.stdout:4/490: creat d12/d5f/fb5 x:0 0 0 2024-04-22T23:23:27.000 INFO:tasks.workunit.client.0.smithi067.stdout:6/461: read d17/f64 [7178404,77841] 0 2024-04-22T23:23:27.001 INFO:tasks.workunit.client.0.smithi067.stdout:2/522: rename d1/f4 to d1/d4f/fa3 0 2024-04-22T23:23:27.003 INFO:tasks.workunit.client.0.smithi067.stdout:0/386: dwrite db/d21/d29/d75/f83 [0,4194304] 0 2024-04-22T23:23:27.004 INFO:tasks.workunit.client.0.smithi067.stdout:9/309: creat d5/dd/d18/d3c/d40/f62 x:0 0 0 2024-04-22T23:23:27.005 INFO:tasks.workunit.client.0.smithi067.stdout:9/310: chown d5/f36 279613 1 2024-04-22T23:23:27.006 INFO:tasks.workunit.client.0.smithi067.stdout:8/407: sync 2024-04-22T23:23:27.007 INFO:tasks.workunit.client.0.smithi067.stdout:8/408: write d1/d3/d10/f45 [275577,104441] 0 2024-04-22T23:23:27.007 INFO:tasks.workunit.client.0.smithi067.stdout:5/479: mknod d5/db/d12/d44/d47/cad 0 2024-04-22T23:23:27.011 INFO:tasks.workunit.client.0.smithi067.stdout:4/491: rename d12/d75 to d12/d5f/d2b/db6 0 2024-04-22T23:23:27.014 INFO:tasks.workunit.client.0.smithi067.stdout:6/462: rmdir d17/d39/d53/d90/d46 39 2024-04-22T23:23:27.015 INFO:tasks.workunit.client.0.smithi067.stdout:9/311: mkdir d5/dd/d18/d39/d3e/d3f/d63 0 2024-04-22T23:23:27.016 INFO:tasks.workunit.client.0.smithi067.stdout:3/378: dwrite d3/d4/d10/d19/f80 [4194304,4194304] 0 2024-04-22T23:23:27.018 INFO:tasks.workunit.client.0.smithi067.stdout:5/480: stat d5/db/l24 0 2024-04-22T23:23:27.020 INFO:tasks.workunit.client.0.smithi067.stdout:2/523: dread d1/d10/f30 [0,4194304] 0 2024-04-22T23:23:27.022 INFO:tasks.workunit.client.0.smithi067.stdout:4/492: dread d12/d42/f78 [0,4194304] 0 2024-04-22T23:23:27.025 INFO:tasks.workunit.client.0.smithi067.stdout:5/481: write d5/d28/d36/d40/d57/d61/f82 [670767,34425] 0 2024-04-22T23:23:27.026 INFO:tasks.workunit.client.0.smithi067.stdout:5/482: stat d5/d1a/d8b 0 2024-04-22T23:23:27.028 INFO:tasks.workunit.client.0.smithi067.stdout:9/312: unlink d5/fb 0 2024-04-22T23:23:27.029 INFO:tasks.workunit.client.0.smithi067.stdout:3/379: symlink d3/d4/d12/l82 0 2024-04-22T23:23:27.031 INFO:tasks.workunit.client.0.smithi067.stdout:3/380: write d3/d4/d10/d1f/d3c/f67 [243174,44029] 0 2024-04-22T23:23:27.034 INFO:tasks.workunit.client.0.smithi067.stdout:0/387: rename db/d2e/d46/l77 to db/d2e/d31/d56/d70/l85 0 2024-04-22T23:23:27.037 INFO:tasks.workunit.client.0.smithi067.stdout:5/483: mkdir d5/d28/dae 0 2024-04-22T23:23:27.038 INFO:tasks.workunit.client.0.smithi067.stdout:8/409: write d1/dd/d80/d6f/f34 [1682318,82399] 0 2024-04-22T23:23:27.039 INFO:tasks.workunit.client.0.smithi067.stdout:3/381: dwrite d3/d4/d10/d38/d53/f54 [0,4194304] 0 2024-04-22T23:23:27.040 INFO:tasks.workunit.client.0.smithi067.stdout:8/410: write d1/d41/f7d [1025818,114406] 0 2024-04-22T23:23:27.041 INFO:tasks.workunit.client.0.smithi067.stdout:5/484: write d5/d3a/d4d/f8d [632934,73636] 0 2024-04-22T23:23:27.044 INFO:tasks.workunit.client.0.smithi067.stdout:3/382: chown d3/d4/d10/d19/l32 65048633 1 2024-04-22T23:23:27.044 INFO:tasks.workunit.client.0.smithi067.stdout:2/524: dread d1/d10/f24 [0,4194304] 0 2024-04-22T23:23:27.045 INFO:tasks.workunit.client.0.smithi067.stdout:3/383: chown d3/d4/f37 435875 1 2024-04-22T23:23:27.045 INFO:tasks.workunit.client.0.smithi067.stdout:0/388: creat db/d21/d29/d75/f86 x:0 0 0 2024-04-22T23:23:27.050 INFO:tasks.workunit.client.0.smithi067.stdout:9/313: link d5/dd/f26 d5/dd/d18/d39/f64 0 2024-04-22T23:23:27.051 INFO:tasks.workunit.client.0.smithi067.stdout:8/411: fdatasync d1/d3/f38 0 2024-04-22T23:23:27.053 INFO:tasks.workunit.client.0.smithi067.stdout:2/525: mkdir d1/d10/d33/d46/da4 0 2024-04-22T23:23:27.054 INFO:tasks.workunit.client.0.smithi067.stdout:0/389: dwrite db/d21/d29/f3f [0,4194304] 0 2024-04-22T23:23:27.055 INFO:tasks.workunit.client.0.smithi067.stdout:4/493: truncate d12/d42/d4b/d70/d8e/f68 1660073 0 2024-04-22T23:23:27.055 INFO:tasks.workunit.client.0.smithi067.stdout:2/526: fsync d1/d10/d33/d46/f6e 0 2024-04-22T23:23:27.056 INFO:tasks.workunit.client.0.smithi067.stdout:0/390: write db/de/f1b [4864162,37965] 0 2024-04-22T23:23:27.058 INFO:tasks.workunit.client.0.smithi067.stdout:5/485: read d5/d3a/f5d [157603,96384] 0 2024-04-22T23:23:27.058 INFO:tasks.workunit.client.0.smithi067.stdout:2/527: chown d1/d10/d33/d46/da4 388272955 1 2024-04-22T23:23:27.059 INFO:tasks.workunit.client.0.smithi067.stdout:0/391: fsync db/d21/d29/f62 0 2024-04-22T23:23:27.060 INFO:tasks.workunit.client.0.smithi067.stdout:2/528: chown d1/d10/d33/c45 0 1 2024-04-22T23:23:27.060 INFO:tasks.workunit.client.0.smithi067.stdout:3/384: write d3/d4/d12/d4b/f4e [1177484,72490] 0 2024-04-22T23:23:27.062 INFO:tasks.workunit.client.0.smithi067.stdout:4/494: getdents d12/d5f/dae 0 2024-04-22T23:23:27.063 INFO:tasks.workunit.client.0.smithi067.stdout:9/314: creat d5/dd/d18/d39/d49/f65 x:0 0 0 2024-04-22T23:23:27.063 INFO:tasks.workunit.client.0.smithi067.stdout:5/486: mkdir d5/d28/daf 0 2024-04-22T23:23:27.064 INFO:tasks.workunit.client.0.smithi067.stdout:2/529: fdatasync d1/d10/d33/d46/f99 0 2024-04-22T23:23:27.065 INFO:tasks.workunit.client.0.smithi067.stdout:8/412: creat d1/dd/f85 x:0 0 0 2024-04-22T23:23:27.066 INFO:tasks.workunit.client.0.smithi067.stdout:0/392: symlink db/d2e/d31/l87 0 2024-04-22T23:23:27.066 INFO:tasks.workunit.client.0.smithi067.stdout:8/413: read - d1/d41/f74 zero size 2024-04-22T23:23:27.066 INFO:tasks.workunit.client.0.smithi067.stdout:5/487: write d5/d28/d36/d40/d57/d61/f82 [945574,114516] 0 2024-04-22T23:23:27.067 INFO:tasks.workunit.client.0.smithi067.stdout:3/385: creat d3/d4/d12/f83 x:0 0 0 2024-04-22T23:23:27.068 INFO:tasks.workunit.client.0.smithi067.stdout:8/414: chown d1/d41 113 1 2024-04-22T23:23:27.068 INFO:tasks.workunit.client.0.smithi067.stdout:0/393: truncate db/fd 2101363 0 2024-04-22T23:23:27.070 INFO:tasks.workunit.client.0.smithi067.stdout:2/530: creat d1/d10/d33/fa5 x:0 0 0 2024-04-22T23:23:27.073 INFO:tasks.workunit.client.0.smithi067.stdout:9/315: unlink d5/dd/d18/d39/d49/d54/f4e 0 2024-04-22T23:23:27.076 INFO:tasks.workunit.client.0.smithi067.stdout:3/386: mknod d3/d4/c84 0 2024-04-22T23:23:27.077 INFO:tasks.workunit.client.0.smithi067.stdout:4/495: unlink d12/d17/d31/c4e 0 2024-04-22T23:23:27.078 INFO:tasks.workunit.client.0.smithi067.stdout:5/488: dwrite d5/d28/d36/d40/f85 [0,4194304] 0 2024-04-22T23:23:27.079 INFO:tasks.workunit.client.0.smithi067.stdout:4/496: chown d12/f8d 7166 1 2024-04-22T23:23:27.079 INFO:tasks.workunit.client.0.smithi067.stdout:2/531: mkdir d1/d10/d1c/d6d/da6 0 2024-04-22T23:23:27.080 INFO:tasks.workunit.client.0.smithi067.stdout:3/387: dread - d3/d4/d10/d38/d53/d55/f75 zero size 2024-04-22T23:23:27.080 INFO:tasks.workunit.client.0.smithi067.stdout:4/497: chown d12/f8d 155 1 2024-04-22T23:23:27.084 INFO:tasks.workunit.client.0.smithi067.stdout:9/316: rename d5/c8 to d5/dd/d18/d39/d3e/d3f/c66 0 2024-04-22T23:23:27.084 INFO:tasks.workunit.client.0.smithi067.stdout:0/394: dread db/d2e/d46/f82 [0,4194304] 0 2024-04-22T23:23:27.085 INFO:tasks.workunit.client.0.smithi067.stdout:0/395: dread - db/d21/d29/f62 zero size 2024-04-22T23:23:27.086 INFO:tasks.workunit.client.0.smithi067.stdout:5/489: fdatasync d5/db/f93 0 2024-04-22T23:23:27.090 INFO:tasks.workunit.client.0.smithi067.stdout:4/498: creat d12/d2e/fb7 x:0 0 0 2024-04-22T23:23:27.092 INFO:tasks.workunit.client.0.smithi067.stdout:9/317: mknod d5/d21/d2c/c67 0 2024-04-22T23:23:27.093 INFO:tasks.workunit.client.0.smithi067.stdout:0/396: mknod db/d2e/d46/c88 0 2024-04-22T23:23:27.095 INFO:tasks.workunit.client.0.smithi067.stdout:2/532: symlink d1/la7 0 2024-04-22T23:23:27.095 INFO:tasks.workunit.client.0.smithi067.stdout:5/490: dwrite d5/d3e/f77 [0,4194304] 0 2024-04-22T23:23:27.097 INFO:tasks.workunit.client.0.smithi067.stdout:5/491: write d5/db/f93 [1476401,23957] 0 2024-04-22T23:23:27.100 INFO:tasks.workunit.client.0.smithi067.stdout:9/318: chown d5/dd/d18/d3c/c59 168832 1 2024-04-22T23:23:27.101 INFO:tasks.workunit.client.0.smithi067.stdout:8/415: write d1/d3/d27/f81 [897676,84511] 0 2024-04-22T23:23:27.102 INFO:tasks.workunit.client.0.smithi067.stdout:0/397: mknod db/de/d11/c89 0 2024-04-22T23:23:27.103 INFO:tasks.workunit.client.0.smithi067.stdout:9/319: stat d5/dd/d18/d39/d3e/d3f/f52 0 2024-04-22T23:23:27.109 INFO:tasks.workunit.client.0.smithi067.stdout:5/492: dread d5/db/d12/d44/f56 [0,4194304] 0 2024-04-22T23:23:27.109 INFO:tasks.workunit.client.0.smithi067.stdout:9/320: creat d5/dd/d18/d39/d3e/f68 x:0 0 0 2024-04-22T23:23:27.111 INFO:tasks.workunit.client.0.smithi067.stdout:8/416: dwrite d1/d3/d10/d3b/f54 [0,4194304] 0 2024-04-22T23:23:27.112 INFO:tasks.workunit.client.0.smithi067.stdout:2/533: rmdir d1/d10/d1c/d6d/da6 0 2024-04-22T23:23:27.116 INFO:tasks.workunit.client.0.smithi067.stdout:9/321: symlink d5/dd/d18/d39/d3e/d3f/l69 0 2024-04-22T23:23:27.121 INFO:tasks.workunit.client.0.smithi067.stdout:5/493: dwrite d5/db/f6a [0,4194304] 0 2024-04-22T23:23:27.121 INFO:tasks.workunit.client.0.smithi067.stdout:3/388: truncate d3/d4/d10/d1f/f3f 233721 0 2024-04-22T23:23:27.122 INFO:tasks.workunit.client.0.smithi067.stdout:3/389: fdatasync d3/d4/d12/f1d 0 2024-04-22T23:23:27.126 INFO:tasks.workunit.client.0.smithi067.stdout:2/534: creat d1/d42/d61/fa8 x:0 0 0 2024-04-22T23:23:27.126 INFO:tasks.workunit.client.0.smithi067.stdout:9/322: symlink d5/d21/d2c/l6a 0 2024-04-22T23:23:27.128 INFO:tasks.workunit.client.0.smithi067.stdout:5/494: dread d5/d1a/d8b/f8f [0,4194304] 0 2024-04-22T23:23:27.128 INFO:tasks.workunit.client.0.smithi067.stdout:0/398: link db/de/d2a/d48/l4b db/de/d2a/l8a 0 2024-04-22T23:23:27.129 INFO:tasks.workunit.client.0.smithi067.stdout:0/399: fdatasync db/de/f12 0 2024-04-22T23:23:27.130 INFO:tasks.workunit.client.0.smithi067.stdout:9/323: creat d5/dd/d18/d3c/d40/f6b x:0 0 0 2024-04-22T23:23:27.130 INFO:tasks.workunit.client.0.smithi067.stdout:0/400: chown db/d2e 1078104 1 2024-04-22T23:23:27.131 INFO:tasks.workunit.client.0.smithi067.stdout:3/390: symlink d3/d4/d10/l85 0 2024-04-22T23:23:27.133 INFO:tasks.workunit.client.0.smithi067.stdout:2/535: link d1/d10/d72/d68/d7b/f8b d1/d10/d1c/d4d/fa9 0 2024-04-22T23:23:27.138 INFO:tasks.workunit.client.0.smithi067.stdout:0/401: dwrite db/d2e/d31/f3a [4194304,4194304] 0 2024-04-22T23:23:27.138 INFO:tasks.workunit.client.0.smithi067.stdout:9/324: chown d5/l5a 10111 1 2024-04-22T23:23:27.138 INFO:tasks.workunit.client.0.smithi067.stdout:7/359: sync 2024-04-22T23:23:27.138 INFO:tasks.workunit.client.0.smithi067.stdout:1/365: sync 2024-04-22T23:23:27.139 INFO:tasks.workunit.client.0.smithi067.stdout:3/391: mknod d3/d4/d10/d1f/d3c/c86 0 2024-04-22T23:23:27.141 INFO:tasks.workunit.client.0.smithi067.stdout:3/392: chown d3/d4/d10/d1f/d3c 59 1 2024-04-22T23:23:27.141 INFO:tasks.workunit.client.0.smithi067.stdout:9/325: truncate d5/dd/d18/d39/d3e/f68 688598 0 2024-04-22T23:23:27.142 INFO:tasks.workunit.client.0.smithi067.stdout:2/536: dread d1/f2d [0,4194304] 0 2024-04-22T23:23:27.143 INFO:tasks.workunit.client.0.smithi067.stdout:8/417: truncate d1/dd/d26/f22 3382327 0 2024-04-22T23:23:27.145 INFO:tasks.workunit.client.0.smithi067.stdout:0/402: creat db/d2e/f8b x:0 0 0 2024-04-22T23:23:27.147 INFO:tasks.workunit.client.0.smithi067.stdout:7/360: read - d10/d14/d1d/d24/f67 zero size 2024-04-22T23:23:27.148 INFO:tasks.workunit.client.0.smithi067.stdout:1/366: creat d8/d9/d2f/d59/f7b x:0 0 0 2024-04-22T23:23:27.148 INFO:tasks.workunit.client.0.smithi067.stdout:7/361: read - d10/d18/f63 zero size 2024-04-22T23:23:27.149 INFO:tasks.workunit.client.0.smithi067.stdout:1/367: readlink d8/d12/d36/l7a 0 2024-04-22T23:23:27.151 INFO:tasks.workunit.client.0.smithi067.stdout:2/537: mkdir d1/d42/d61/daa 0 2024-04-22T23:23:27.154 INFO:tasks.workunit.client.0.smithi067.stdout:8/418: dwrite d1/dd/d80/d6f/f32 [0,4194304] 0 2024-04-22T23:23:27.156 INFO:tasks.workunit.client.0.smithi067.stdout:0/403: mkdir db/d2e/d31/d56/d70/d8c 0 2024-04-22T23:23:27.158 INFO:tasks.workunit.client.0.smithi067.stdout:7/362: mkdir d10/d14/d1d/d26/d2c/d7a 0 2024-04-22T23:23:27.163 INFO:tasks.workunit.client.0.smithi067.stdout:2/538: rename d1/d10/d33/f49 to d1/d10/d72/d68/fab 0 2024-04-22T23:23:27.166 INFO:tasks.workunit.client.0.smithi067.stdout:8/419: creat d1/dd/d80/f86 x:0 0 0 2024-04-22T23:23:27.167 INFO:tasks.workunit.client.0.smithi067.stdout:9/326: creat d5/d21/f6c x:0 0 0 2024-04-22T23:23:27.175 INFO:tasks.workunit.client.0.smithi067.stdout:7/363: mkdir d10/d14/d1d/d7b 0 2024-04-22T23:23:27.177 INFO:tasks.workunit.client.0.smithi067.stdout:3/393: truncate d3/d4/d10/d38/d53/f54 3878364 0 2024-04-22T23:23:27.178 INFO:tasks.workunit.client.0.smithi067.stdout:1/368: dread d8/d9/d2d/d4d/d70/d34/f3f [0,4194304] 0 2024-04-22T23:23:27.179 INFO:tasks.workunit.client.0.smithi067.stdout:2/539: dwrite d1/d10/d1c/d6d/f89 [0,4194304] 0 2024-04-22T23:23:27.189 INFO:tasks.workunit.client.0.smithi067.stdout:9/327: symlink d5/d21/d47/d56/l6d 0 2024-04-22T23:23:27.190 INFO:tasks.workunit.client.0.smithi067.stdout:7/364: creat d10/d14/d1d/d26/d2c/d41/f7c x:0 0 0 2024-04-22T23:23:27.191 INFO:tasks.workunit.client.0.smithi067.stdout:7/365: truncate fe 5001559 0 2024-04-22T23:23:27.194 INFO:tasks.workunit.client.0.smithi067.stdout:0/404: rename db/d21/d29/d2c/f3d to db/d21/f8d 0 2024-04-22T23:23:27.197 INFO:tasks.workunit.client.0.smithi067.stdout:7/366: mknod d10/d18/d35/c7d 0 2024-04-22T23:23:27.197 INFO:tasks.workunit.client.0.smithi067.stdout:1/369: dread d8/f26 [0,4194304] 0 2024-04-22T23:23:27.199 INFO:tasks.workunit.client.0.smithi067.stdout:9/328: dwrite d5/dd/d18/d3c/d40/f6b [0,4194304] 0 2024-04-22T23:23:27.200 INFO:tasks.workunit.client.0.smithi067.stdout:0/405: getdents db/d2e/d46 0 2024-04-22T23:23:27.201 INFO:tasks.workunit.client.0.smithi067.stdout:9/329: write d5/dd/d18/d39/d49/f55 [909214,57242] 0 2024-04-22T23:23:27.202 INFO:tasks.workunit.client.0.smithi067.stdout:7/367: mkdir d10/d14/d1d/d26/d7e 0 2024-04-22T23:23:27.203 INFO:tasks.workunit.client.0.smithi067.stdout:9/330: chown d5/dd/f32 15094818 1 2024-04-22T23:23:27.204 INFO:tasks.workunit.client.0.smithi067.stdout:9/331: write d5/fa [267190,110936] 0 2024-04-22T23:23:27.205 INFO:tasks.workunit.client.0.smithi067.stdout:0/406: unlink db/de/d11/f26 0 2024-04-22T23:23:27.208 INFO:tasks.workunit.client.0.smithi067.stdout:9/332: creat d5/dd/d18/d3c/f6e x:0 0 0 2024-04-22T23:23:27.209 INFO:tasks.workunit.client.0.smithi067.stdout:9/333: write d5/dd/d18/d39/d3e/f68 [887295,111955] 0 2024-04-22T23:23:27.212 INFO:tasks.workunit.client.0.smithi067.stdout:1/370: getdents d8/d9/d2f/d59 0 2024-04-22T23:23:27.242 INFO:tasks.workunit.client.0.smithi067.stdout:3/394: write d3/d4/d10/f39 [1077771,20586] 0 2024-04-22T23:23:27.242 INFO:tasks.workunit.client.0.smithi067.stdout:0/407: rmdir db/d21 39 2024-04-22T23:23:27.244 INFO:tasks.workunit.client.0.smithi067.stdout:8/420: truncate d1/d3/d10/f29 1658266 0 2024-04-22T23:23:27.245 INFO:tasks.workunit.client.0.smithi067.stdout:8/421: dread - d1/dd/f79 zero size 2024-04-22T23:23:27.245 INFO:tasks.workunit.client.0.smithi067.stdout:0/408: unlink db/de/f12 0 2024-04-22T23:23:27.247 INFO:tasks.workunit.client.0.smithi067.stdout:3/395: creat d3/d4/d10/d38/d53/d55/d5c/f87 x:0 0 0 2024-04-22T23:23:27.249 INFO:tasks.workunit.client.0.smithi067.stdout:8/422: symlink d1/d3/d7e/l87 0 2024-04-22T23:23:27.251 INFO:tasks.workunit.client.0.smithi067.stdout:3/396: mknod d3/d4/d1b/c88 0 2024-04-22T23:23:27.254 INFO:tasks.workunit.client.0.smithi067.stdout:9/334: write d5/d21/f24 [2162866,64678] 0 2024-04-22T23:23:27.254 INFO:tasks.workunit.client.0.smithi067.stdout:3/397: rmdir d3/d4/d28/d40 39 2024-04-22T23:23:27.257 INFO:tasks.workunit.client.0.smithi067.stdout:9/335: read d5/dd/d18/d39/d3e/f68 [160993,128807] 0 2024-04-22T23:23:27.259 INFO:tasks.workunit.client.0.smithi067.stdout:9/336: rename d5 to d5/d21/d47/d56/d6f 22 2024-04-22T23:23:27.259 INFO:tasks.workunit.client.0.smithi067.stdout:8/423: link d1/d3/d7e/c7f d1/d3/d27/c88 0 2024-04-22T23:23:27.261 INFO:tasks.workunit.client.0.smithi067.stdout:3/398: rmdir d3/d4/d10/d38/d53 39 2024-04-22T23:23:27.261 INFO:tasks.workunit.client.0.smithi067.stdout:9/337: fsync d5/dd/d18/d39/d49/f65 0 2024-04-22T23:23:27.262 INFO:tasks.workunit.client.0.smithi067.stdout:1/371: dwrite d8/d9/d2f/f38 [0,4194304] 0 2024-04-22T23:23:27.263 INFO:tasks.workunit.client.0.smithi067.stdout:8/424: dread d1/d3/f2a [0,4194304] 0 2024-04-22T23:23:27.268 INFO:tasks.workunit.client.0.smithi067.stdout:3/399: chown d3/d4/d28/d4a/f7a 349528 1 2024-04-22T23:23:27.271 INFO:tasks.workunit.client.0.smithi067.stdout:1/372: write d8/f1d [184023,4346] 0 2024-04-22T23:23:27.273 INFO:tasks.workunit.client.0.smithi067.stdout:3/400: chown d3/d4/d10/d19/c3d 38 1 2024-04-22T23:23:27.274 INFO:tasks.workunit.client.0.smithi067.stdout:9/338: dwrite d5/dd/d18/d39/d49/d54/f3d [0,4194304] 0 2024-04-22T23:23:27.281 INFO:tasks.workunit.client.0.smithi067.stdout:0/409: dwrite db/de/d11/f3c [0,4194304] 0 2024-04-22T23:23:27.282 INFO:tasks.workunit.client.0.smithi067.stdout:1/373: truncate d8/d9/d2d/d4d/d70/d34/f47 571856 0 2024-04-22T23:23:27.284 INFO:tasks.workunit.client.0.smithi067.stdout:1/374: stat d8/d12 0 2024-04-22T23:23:27.288 INFO:tasks.workunit.client.0.smithi067.stdout:0/410: fdatasync db/d21/d29/d2d/f49 0 2024-04-22T23:23:27.288 INFO:tasks.workunit.client.0.smithi067.stdout:1/375: chown d8/d9/d2d/c5f 20185562 1 2024-04-22T23:23:27.288 INFO:tasks.workunit.client.0.smithi067.stdout:8/425: dwrite d1/f7 [0,4194304] 0 2024-04-22T23:23:27.291 INFO:tasks.workunit.client.0.smithi067.stdout:8/426: dread - d1/d3/d10/d3b/f5e zero size 2024-04-22T23:23:27.291 INFO:tasks.workunit.client.0.smithi067.stdout:9/339: link d5/dd/d18/d39/d49/d54/l5f d5/dd/d18/d39/d4d/l70 0 2024-04-22T23:23:27.295 INFO:tasks.workunit.client.0.smithi067.stdout:8/427: truncate d1/dd/f85 727616 0 2024-04-22T23:23:27.295 INFO:tasks.workunit.client.0.smithi067.stdout:0/411: creat db/de/d2a/d48/f8e x:0 0 0 2024-04-22T23:23:27.299 INFO:tasks.workunit.client.0.smithi067.stdout:9/340: creat d5/f71 x:0 0 0 2024-04-22T23:23:27.299 INFO:tasks.workunit.client.0.smithi067.stdout:0/412: readlink db/de/d2a/l67 0 2024-04-22T23:23:27.299 INFO:tasks.workunit.client.0.smithi067.stdout:8/428: readlink d1/dd/d26/d62/d75/l7b 0 2024-04-22T23:23:27.299 INFO:tasks.workunit.client.0.smithi067.stdout:3/401: dwrite d3/d4/d28/d4c/f52 [0,4194304] 0 2024-04-22T23:23:27.303 INFO:tasks.workunit.client.0.smithi067.stdout:0/413: mknod db/d2e/d31/d56/d70/c8f 0 2024-04-22T23:23:27.304 INFO:tasks.workunit.client.0.smithi067.stdout:0/414: chown db/d2e/c4a 65272 1 2024-04-22T23:23:27.305 INFO:tasks.workunit.client.0.smithi067.stdout:9/341: dread d5/f36 [0,4194304] 0 2024-04-22T23:23:27.306 INFO:tasks.workunit.client.0.smithi067.stdout:0/415: read db/d21/f5a [650027,62090] 0 2024-04-22T23:23:27.310 INFO:tasks.workunit.client.0.smithi067.stdout:9/342: unlink d5/fe 0 2024-04-22T23:23:27.310 INFO:tasks.workunit.client.0.smithi067.stdout:0/416: readlink db/d2e/d31/l73 0 2024-04-22T23:23:27.315 INFO:tasks.workunit.client.0.smithi067.stdout:9/343: creat d5/d21/d47/f72 x:0 0 0 2024-04-22T23:23:27.317 INFO:tasks.workunit.client.0.smithi067.stdout:3/402: rename d3/d4/d10/d38/d53/d55/d5c/d70/l81 to d3/d4/d28/d40/l89 0 2024-04-22T23:23:27.317 INFO:tasks.workunit.client.0.smithi067.stdout:0/417: dread db/d21/f8d [0,4194304] 0 2024-04-22T23:23:27.319 INFO:tasks.workunit.client.0.smithi067.stdout:9/344: creat d5/dd/d18/d39/d3e/d3f/f73 x:0 0 0 2024-04-22T23:23:27.320 INFO:tasks.workunit.client.0.smithi067.stdout:0/418: chown db/d21/d29/f62 191 1 2024-04-22T23:23:27.321 INFO:tasks.workunit.client.0.smithi067.stdout:3/403: truncate d3/d4/d28/d40/f45 555567 0 2024-04-22T23:23:27.322 INFO:tasks.workunit.client.0.smithi067.stdout:9/345: symlink d5/dd/d18/d39/d3e/l74 0 2024-04-22T23:23:27.323 INFO:tasks.workunit.client.0.smithi067.stdout:0/419: symlink db/d2e/d31/d56/l90 0 2024-04-22T23:23:27.325 INFO:tasks.workunit.client.0.smithi067.stdout:8/429: dread d1/d3/d27/f33 [4194304,4194304] 0 2024-04-22T23:23:27.325 INFO:tasks.workunit.client.0.smithi067.stdout:1/376: dread d8/d9/f1b [0,4194304] 0 2024-04-22T23:23:27.326 INFO:tasks.workunit.client.0.smithi067.stdout:8/430: chown d1/d3/d10/f3d 108553015 1 2024-04-22T23:23:27.331 INFO:tasks.workunit.client.0.smithi067.stdout:8/431: dwrite d1/d3/d10/d3b/f5e [0,4194304] 0 2024-04-22T23:23:27.336 INFO:tasks.workunit.client.0.smithi067.stdout:8/432: dread d1/d3/d10/f45 [0,4194304] 0 2024-04-22T23:23:27.355 INFO:tasks.workunit.client.0.smithi067.stdout:3/404: rename d3/d4/d12/d7e to d3/d4/d10/d8a 0 2024-04-22T23:23:27.358 INFO:tasks.workunit.client.0.smithi067.stdout:1/377: rmdir d8/d9/d2f 39 2024-04-22T23:23:27.364 INFO:tasks.workunit.client.0.smithi067.stdout:9/346: rename d5/dd/d18/d39/d3e/l74 to d5/dd/d18/d5c/l75 0 2024-04-22T23:23:27.367 INFO:tasks.workunit.client.0.smithi067.stdout:1/378: dread - d8/d9/f42 zero size 2024-04-22T23:23:27.370 INFO:tasks.workunit.client.0.smithi067.stdout:0/420: truncate db/de/d11/f3c 1842821 0 2024-04-22T23:23:27.372 INFO:tasks.workunit.client.0.smithi067.stdout:9/347: mknod d5/dd/d18/d3c/c76 0 2024-04-22T23:23:27.373 INFO:tasks.workunit.client.0.smithi067.stdout:8/433: dwrite d1/dd/f20 [0,4194304] 0 2024-04-22T23:23:27.376 INFO:tasks.workunit.client.0.smithi067.stdout:0/421: creat db/d21/d29/d2c/f91 x:0 0 0 2024-04-22T23:23:27.377 INFO:tasks.workunit.client.0.smithi067.stdout:9/348: mknod d5/dd/d18/d39/c77 0 2024-04-22T23:23:27.378 INFO:tasks.workunit.client.0.smithi067.stdout:3/405: link d3/d4/c84 d3/d4/d10/d38/d53/c8b 0 2024-04-22T23:23:27.378 INFO:tasks.workunit.client.0.smithi067.stdout:0/422: write db/d21/d29/d75/f86 [41402,103867] 0 2024-04-22T23:23:27.383 INFO:tasks.workunit.client.0.smithi067.stdout:9/349: dread d5/fa [0,4194304] 0 2024-04-22T23:23:27.384 INFO:tasks.workunit.client.0.smithi067.stdout:1/379: rename d8/d9/d2d/d4d/d70/d34/f3f to d8/d9/d2f/d33/d5c/f7c 0 2024-04-22T23:23:27.386 INFO:tasks.workunit.client.0.smithi067.stdout:3/406: truncate d3/d4/d10/d1f/f3e 729553 0 2024-04-22T23:23:27.388 INFO:tasks.workunit.client.0.smithi067.stdout:8/434: symlink d1/d3/d5c/l89 0 2024-04-22T23:23:27.388 INFO:tasks.workunit.client.0.smithi067.stdout:9/350: mkdir d5/dd/d18/d3c/d78 0 2024-04-22T23:23:27.389 INFO:tasks.workunit.client.0.smithi067.stdout:1/380: dread d8/f1d [0,4194304] 0 2024-04-22T23:23:27.390 INFO:tasks.workunit.client.0.smithi067.stdout:9/351: write d5/d21/f34 [1217794,82763] 0 2024-04-22T23:23:27.391 INFO:tasks.workunit.client.0.smithi067.stdout:0/423: rename db/d21/d7a to db/de/d11/d92 0 2024-04-22T23:23:27.392 INFO:tasks.workunit.client.0.smithi067.stdout:9/352: read d5/dd/f1a [673093,90806] 0 2024-04-22T23:23:27.393 INFO:tasks.workunit.client.0.smithi067.stdout:0/424: write db/de/d11/d1a/f64 [2971253,108273] 0 2024-04-22T23:23:27.396 INFO:tasks.workunit.client.0.smithi067.stdout:3/407: rmdir d3/d4/d12 39 2024-04-22T23:23:27.397 INFO:tasks.workunit.client.0.smithi067.stdout:0/425: dread db/d21/d29/f3f [0,4194304] 0 2024-04-22T23:23:27.404 INFO:tasks.workunit.client.0.smithi067.stdout:9/353: creat d5/dd/d18/d39/d49/d54/f79 x:0 0 0 2024-04-22T23:23:27.406 INFO:tasks.workunit.client.0.smithi067.stdout:3/408: rmdir d3/d4/d10/d1f 39 2024-04-22T23:23:27.410 INFO:tasks.workunit.client.0.smithi067.stdout:9/354: creat d5/d21/d2c/f7a x:0 0 0 2024-04-22T23:23:27.410 INFO:tasks.workunit.client.0.smithi067.stdout:9/355: fdatasync d5/f10 0 2024-04-22T23:23:27.411 INFO:tasks.workunit.client.0.smithi067.stdout:3/409: mkdir d3/d4/d28/d4c/d8c 0 2024-04-22T23:23:27.412 INFO:tasks.workunit.client.0.smithi067.stdout:0/426: creat db/d71/f93 x:0 0 0 2024-04-22T23:23:27.413 INFO:tasks.workunit.client.0.smithi067.stdout:1/381: creat d8/d9/d2f/f7d x:0 0 0 2024-04-22T23:23:27.415 INFO:tasks.workunit.client.0.smithi067.stdout:3/410: rmdir d3/d4/d10/d19 39 2024-04-22T23:23:27.416 INFO:tasks.workunit.client.0.smithi067.stdout:8/435: truncate d1/f6 3518503 0 2024-04-22T23:23:27.417 INFO:tasks.workunit.client.0.smithi067.stdout:0/427: symlink db/d2e/d31/d56/d70/l94 0 2024-04-22T23:23:27.419 INFO:tasks.workunit.client.0.smithi067.stdout:3/411: mknod d3/d4/d1b/c8d 0 2024-04-22T23:23:27.421 INFO:tasks.workunit.client.0.smithi067.stdout:1/382: dwrite d8/d9/d2f/d59/f7b [0,4194304] 0 2024-04-22T23:23:27.421 INFO:tasks.workunit.client.0.smithi067.stdout:0/428: rename db/de to db/de/d11/d1a/d95 22 2024-04-22T23:23:27.425 INFO:tasks.workunit.client.0.smithi067.stdout:3/412: read d3/d4/d10/f1a [269529,39487] 0 2024-04-22T23:23:27.427 INFO:tasks.workunit.client.0.smithi067.stdout:0/429: unlink db/de/l84 0 2024-04-22T23:23:27.428 INFO:tasks.workunit.client.0.smithi067.stdout:0/430: write db/d2e/d31/f63 [601367,8078] 0 2024-04-22T23:23:27.430 INFO:tasks.workunit.client.0.smithi067.stdout:0/431: fsync db/de/f1b 0 2024-04-22T23:23:27.430 INFO:tasks.workunit.client.0.smithi067.stdout:1/383: write d8/d9/d2f/d33/d5c/f7c [5067544,91221] 0 2024-04-22T23:23:27.433 INFO:tasks.workunit.client.0.smithi067.stdout:3/413: creat d3/d4/d10/d38/d53/d55/f8e x:0 0 0 2024-04-22T23:23:27.435 INFO:tasks.workunit.client.0.smithi067.stdout:1/384: unlink d8/d9/d2d/d4d/l10 0 2024-04-22T23:23:27.437 INFO:tasks.workunit.client.0.smithi067.stdout:0/432: dread db/d21/d29/d75/f86 [0,4194304] 0 2024-04-22T23:23:27.437 INFO:tasks.workunit.client.0.smithi067.stdout:9/356: dwrite d5/dd/d18/f1f [0,4194304] 0 2024-04-22T23:23:27.444 INFO:tasks.workunit.client.0.smithi067.stdout:1/385: creat d8/d9/d2d/d4d/d70/d34/f7e x:0 0 0 2024-04-22T23:23:27.445 INFO:tasks.workunit.client.0.smithi067.stdout:8/436: dwrite d1/f6 [0,4194304] 0 2024-04-22T23:23:27.449 INFO:tasks.workunit.client.0.smithi067.stdout:9/357: creat d5/dd/d18/d39/d49/f7b x:0 0 0 2024-04-22T23:23:27.450 INFO:tasks.workunit.client.0.smithi067.stdout:0/433: dwrite db/d71/f93 [0,4194304] 0 2024-04-22T23:23:27.452 INFO:tasks.workunit.client.0.smithi067.stdout:0/434: dread - db/d2e/d31/d56/f6d zero size 2024-04-22T23:23:27.452 INFO:tasks.workunit.client.0.smithi067.stdout:3/414: rename d3/d4/d10/d19/f21 to d3/d4/d12/d4b/f8f 0 2024-04-22T23:23:27.453 INFO:tasks.workunit.client.0.smithi067.stdout:0/435: stat db/de/c61 0 2024-04-22T23:23:27.454 INFO:tasks.workunit.client.0.smithi067.stdout:9/358: creat d5/d21/d47/d56/f7c x:0 0 0 2024-04-22T23:23:27.458 INFO:tasks.workunit.client.0.smithi067.stdout:1/386: rename d8/d9/d2f/d33/d5c/f7c to d8/d9/d2d/d4d/d48/f7f 0 2024-04-22T23:23:27.460 INFO:tasks.workunit.client.0.smithi067.stdout:0/436: unlink db/de/d11/l50 0 2024-04-22T23:23:27.461 INFO:tasks.workunit.client.0.smithi067.stdout:8/437: dread d1/d3/d27/f81 [0,4194304] 0 2024-04-22T23:23:27.464 INFO:tasks.workunit.client.0.smithi067.stdout:0/437: truncate db/d21/d29/d2c/f72 706325 0 2024-04-22T23:23:27.466 INFO:tasks.workunit.client.0.smithi067.stdout:1/387: chown d8/d9/f29 1583524284 1 2024-04-22T23:23:27.467 INFO:tasks.workunit.client.0.smithi067.stdout:8/438: symlink d1/dd/d26/d62/d75/l8a 0 2024-04-22T23:23:27.467 INFO:tasks.workunit.client.0.smithi067.stdout:1/388: chown d8/d12/d40/c71 7 1 2024-04-22T23:23:27.468 INFO:tasks.workunit.client.0.smithi067.stdout:8/439: write d1/f7a [865081,697] 0 2024-04-22T23:23:27.468 INFO:tasks.workunit.client.0.smithi067.stdout:0/438: symlink db/de/l96 0 2024-04-22T23:23:27.470 INFO:tasks.workunit.client.0.smithi067.stdout:1/389: rmdir d8/d4a 39 2024-04-22T23:23:27.471 INFO:tasks.workunit.client.0.smithi067.stdout:8/440: mknod d1/dd/d26/d62/d75/c8b 0 2024-04-22T23:23:27.473 INFO:tasks.workunit.client.0.smithi067.stdout:8/441: read d1/d3/f1b [2172092,88716] 0 2024-04-22T23:23:27.473 INFO:tasks.workunit.client.0.smithi067.stdout:6/463: sync 2024-04-22T23:23:27.475 INFO:tasks.workunit.client.0.smithi067.stdout:6/464: write f13 [7036095,117382] 0 2024-04-22T23:23:27.477 INFO:tasks.workunit.client.0.smithi067.stdout:0/439: rename db/de/d2a/l8a to db/d21/l97 0 2024-04-22T23:23:27.478 INFO:tasks.workunit.client.0.smithi067.stdout:8/442: write d1/d3/f11 [2038108,7265] 0 2024-04-22T23:23:27.483 INFO:tasks.workunit.client.0.smithi067.stdout:0/440: dwrite db/f79 [0,4194304] 0 2024-04-22T23:23:27.485 INFO:tasks.workunit.client.0.smithi067.stdout:8/443: truncate d1/dd/d26/f22 1540799 0 2024-04-22T23:23:27.488 INFO:tasks.workunit.client.0.smithi067.stdout:0/441: chown db/de/d2a/d48/f8e 1 1 2024-04-22T23:23:27.489 INFO:tasks.workunit.client.0.smithi067.stdout:1/390: write d8/f11 [723994,51770] 0 2024-04-22T23:23:27.494 INFO:tasks.workunit.client.0.smithi067.stdout:3/415: dwrite d3/d4/d10/d1f/d3c/f25 [0,4194304] 0 2024-04-22T23:23:27.494 INFO:tasks.workunit.client.0.smithi067.stdout:9/359: dwrite d5/dd/d18/f1d [0,4194304] 0 2024-04-22T23:23:27.497 INFO:tasks.workunit.client.0.smithi067.stdout:1/391: write d8/d9/d2f/d33/d5c/f65 [2558241,27966] 0 2024-04-22T23:23:27.502 INFO:tasks.workunit.client.0.smithi067.stdout:3/416: mkdir d3/d4/d28/d4c/d90 0 2024-04-22T23:23:27.506 INFO:tasks.workunit.client.0.smithi067.stdout:0/442: link db/de/d2a/c7f db/d60/c98 0 2024-04-22T23:23:27.507 INFO:tasks.workunit.client.0.smithi067.stdout:9/360: dwrite d5/dd/d18/d39/d49/f55 [0,4194304] 0 2024-04-22T23:23:27.509 INFO:tasks.workunit.client.0.smithi067.stdout:8/444: truncate d1/d3/d27/f6e 2284347 0 2024-04-22T23:23:27.509 INFO:tasks.workunit.client.0.smithi067.stdout:1/392: symlink d8/d4a/l80 0 2024-04-22T23:23:27.509 INFO:tasks.workunit.client.0.smithi067.stdout:9/361: readlink d5/d21/d2c/l3a 0 2024-04-22T23:23:27.510 INFO:tasks.workunit.client.0.smithi067.stdout:1/393: readlink d8/d12/l27 0 2024-04-22T23:23:27.511 INFO:tasks.workunit.client.0.smithi067.stdout:9/362: fsync d5/d21/f34 0 2024-04-22T23:23:27.512 INFO:tasks.workunit.client.0.smithi067.stdout:3/417: creat d3/d4/d12/d4b/f91 x:0 0 0 2024-04-22T23:23:27.512 INFO:tasks.workunit.client.0.smithi067.stdout:1/394: dread - d8/d9/d2f/d59/f6e zero size 2024-04-22T23:23:27.513 INFO:tasks.workunit.client.0.smithi067.stdout:8/445: unlink d1/d41/l77 0 2024-04-22T23:23:27.514 INFO:tasks.workunit.client.0.smithi067.stdout:1/395: dread d8/f11 [0,4194304] 0 2024-04-22T23:23:27.515 INFO:tasks.workunit.client.0.smithi067.stdout:9/363: creat d5/dd/d18/d39/d4d/f7d x:0 0 0 2024-04-22T23:23:27.517 INFO:tasks.workunit.client.0.smithi067.stdout:8/446: creat d1/dd/d80/f8c x:0 0 0 2024-04-22T23:23:27.519 INFO:tasks.workunit.client.0.smithi067.stdout:0/443: creat db/de/d2a/f99 x:0 0 0 2024-04-22T23:23:27.520 INFO:tasks.workunit.client.0.smithi067.stdout:3/418: mknod d3/d4/d10/d38/c92 0 2024-04-22T23:23:27.521 INFO:tasks.workunit.client.0.smithi067.stdout:9/364: creat d5/dd/d18/d39/d3e/d3f/f7e x:0 0 0 2024-04-22T23:23:27.524 INFO:tasks.workunit.client.0.smithi067.stdout:8/447: rmdir d1/d3/d10 39 2024-04-22T23:23:27.526 INFO:tasks.workunit.client.0.smithi067.stdout:8/448: write d1/d3/f13 [1423975,130659] 0 2024-04-22T23:23:27.530 INFO:tasks.workunit.client.0.smithi067.stdout:3/419: dwrite d3/d4/d12/d4b/f6c [0,4194304] 0 2024-04-22T23:23:27.537 INFO:tasks.workunit.client.0.smithi067.stdout:0/444: rename db/d2e/d31 to db/d9a 0 2024-04-22T23:23:27.538 INFO:tasks.workunit.client.0.smithi067.stdout:8/449: creat d1/dd/d80/f8d x:0 0 0 2024-04-22T23:23:27.543 INFO:tasks.workunit.client.0.smithi067.stdout:8/450: symlink d1/dd/d80/d6f/l8e 0 2024-04-22T23:23:27.544 INFO:tasks.workunit.client.0.smithi067.stdout:8/451: read d1/f7a [711776,60291] 0 2024-04-22T23:23:27.555 INFO:tasks.workunit.client.0.smithi067.stdout:9/365: dwrite d5/dd/d18/d39/d3e/f68 [0,4194304] 0 2024-04-22T23:23:27.555 INFO:tasks.workunit.client.0.smithi067.stdout:3/420: dread d3/d4/d10/d19/f34 [0,4194304] 0 2024-04-22T23:23:27.556 INFO:tasks.workunit.client.0.smithi067.stdout:8/452: dread d1/dd/d80/d6f/f30 [0,4194304] 0 2024-04-22T23:23:27.560 INFO:tasks.workunit.client.0.smithi067.stdout:8/453: creat d1/dd/d26/f8f x:0 0 0 2024-04-22T23:23:27.561 INFO:tasks.workunit.client.0.smithi067.stdout:9/366: readlink d5/dd/d18/d39/d49/d54/l5f 0 2024-04-22T23:23:27.563 INFO:tasks.workunit.client.0.smithi067.stdout:0/445: dwrite db/de/ff [0,4194304] 0 2024-04-22T23:23:27.563 INFO:tasks.workunit.client.0.smithi067.stdout:3/421: unlink d3/d4/c84 0 2024-04-22T23:23:27.565 INFO:tasks.workunit.client.0.smithi067.stdout:8/454: chown d1/d3/d10/l1c 12256459 1 2024-04-22T23:23:27.567 INFO:tasks.workunit.client.0.smithi067.stdout:3/422: mkdir d3/d4/d1b/d93 0 2024-04-22T23:23:27.570 INFO:tasks.workunit.client.0.smithi067.stdout:0/446: chown db/de/d2a/d48/l4b 0 1 2024-04-22T23:23:27.571 INFO:tasks.workunit.client.0.smithi067.stdout:8/455: fdatasync d1/f7a 0 2024-04-22T23:23:27.573 INFO:tasks.workunit.client.0.smithi067.stdout:9/367: getdents d5/dd/d18/d39/d49/d54 0 2024-04-22T23:23:27.575 INFO:tasks.workunit.client.0.smithi067.stdout:8/456: symlink d1/d41/l90 0 2024-04-22T23:23:27.577 INFO:tasks.workunit.client.0.smithi067.stdout:8/457: stat d1/d3/d10/d3b/f84 0 2024-04-22T23:23:27.580 INFO:tasks.workunit.client.0.smithi067.stdout:9/368: dwrite d5/d21/d2c/f3b [4194304,4194304] 0 2024-04-22T23:23:27.582 INFO:tasks.workunit.client.0.smithi067.stdout:3/423: truncate d3/d4/d12/d4b/f63 1997128 0 2024-04-22T23:23:27.583 INFO:tasks.workunit.client.0.smithi067.stdout:0/447: getdents db/d9a/d56 0 2024-04-22T23:23:27.590 INFO:tasks.workunit.client.0.smithi067.stdout:0/448: dread db/d21/f8d [0,4194304] 0 2024-04-22T23:23:27.592 INFO:tasks.workunit.client.0.smithi067.stdout:9/369: readlink d5/dd/d18/d5c/l75 0 2024-04-22T23:23:27.593 INFO:tasks.workunit.client.0.smithi067.stdout:8/458: link d1/d3/d27/f33 d1/dd/d26/d62/f91 0 2024-04-22T23:23:27.594 INFO:tasks.workunit.client.0.smithi067.stdout:0/449: dread - db/d2e/f40 zero size 2024-04-22T23:23:27.595 INFO:tasks.workunit.client.0.smithi067.stdout:4/499: sync 2024-04-22T23:23:27.595 INFO:tasks.workunit.client.0.smithi067.stdout:5/495: sync 2024-04-22T23:23:27.595 INFO:tasks.workunit.client.0.smithi067.stdout:7/368: sync 2024-04-22T23:23:27.595 INFO:tasks.workunit.client.0.smithi067.stdout:3/424: dwrite d3/d4/d10/d1f/d3c/f25 [4194304,4194304] 0 2024-04-22T23:23:27.595 INFO:tasks.workunit.client.0.smithi067.stdout:2/540: sync 2024-04-22T23:23:27.596 INFO:tasks.workunit.client.0.smithi067.stdout:9/370: mkdir d5/d21/d47/d7f 0 2024-04-22T23:23:27.601 INFO:tasks.workunit.client.0.smithi067.stdout:9/371: stat d5/dd/d18/d39/d49 0 2024-04-22T23:23:27.602 INFO:tasks.workunit.client.0.smithi067.stdout:5/496: rmdir d5 39 2024-04-22T23:23:27.603 INFO:tasks.workunit.client.0.smithi067.stdout:7/369: write d10/d18/f63 [310744,9019] 0 2024-04-22T23:23:27.607 INFO:tasks.workunit.client.0.smithi067.stdout:4/500: symlink d12/d5f/d2b/lb8 0 2024-04-22T23:23:27.613 INFO:tasks.workunit.client.0.smithi067.stdout:0/450: symlink db/d21/d29/l9b 0 2024-04-22T23:23:27.614 INFO:tasks.workunit.client.0.smithi067.stdout:4/501: fdatasync d12/d5f/d2b/db6/f95 0 2024-04-22T23:23:27.615 INFO:tasks.workunit.client.0.smithi067.stdout:9/372: creat d5/d21/d47/d56/f80 x:0 0 0 2024-04-22T23:23:27.617 INFO:tasks.workunit.client.0.smithi067.stdout:3/425: dwrite d3/d4/fc [0,4194304] 0 2024-04-22T23:23:27.618 INFO:tasks.workunit.client.0.smithi067.stdout:5/497: write d5/d28/f69 [837904,123252] 0 2024-04-22T23:23:27.619 INFO:tasks.workunit.client.0.smithi067.stdout:8/459: dwrite d1/dd/f20 [4194304,4194304] 0 2024-04-22T23:23:27.620 INFO:tasks.workunit.client.0.smithi067.stdout:7/370: dwrite d10/f12 [0,4194304] 0 2024-04-22T23:23:27.627 INFO:tasks.workunit.client.0.smithi067.stdout:8/460: readlink d1/d3/d7e/l87 0 2024-04-22T23:23:27.627 INFO:tasks.workunit.client.0.smithi067.stdout:2/541: truncate d1/d10/d1c/d6d/f89 436223 0 2024-04-22T23:23:27.627 INFO:tasks.workunit.client.0.smithi067.stdout:4/502: write d12/d17/d3f/f63 [434540,24836] 0 2024-04-22T23:23:27.627 INFO:tasks.workunit.client.0.smithi067.stdout:0/451: write db/d21/d29/d2c/f72 [1009321,35499] 0 2024-04-22T23:23:27.629 INFO:tasks.workunit.client.0.smithi067.stdout:0/452: chown db 263650018 1 2024-04-22T23:23:27.629 INFO:tasks.workunit.client.0.smithi067.stdout:3/426: write d3/d4/d10/d19/f5d [287575,61408] 0 2024-04-22T23:23:27.631 INFO:tasks.workunit.client.0.smithi067.stdout:9/373: mknod d5/dd/d18/d5c/c81 0 2024-04-22T23:23:27.631 INFO:tasks.workunit.client.0.smithi067.stdout:5/498: fsync d5/d3a/f3c 0 2024-04-22T23:23:27.635 INFO:tasks.workunit.client.0.smithi067.stdout:2/542: creat d1/d10/d72/d68/d7b/d95/fac x:0 0 0 2024-04-22T23:23:27.636 INFO:tasks.workunit.client.0.smithi067.stdout:9/374: creat d5/dd/d18/d39/d3e/f82 x:0 0 0 2024-04-22T23:23:27.637 INFO:tasks.workunit.client.0.smithi067.stdout:7/371: rename d10/d14/f34 to d10/f7f 0 2024-04-22T23:23:27.638 INFO:tasks.workunit.client.0.smithi067.stdout:3/427: creat d3/d4/d12/d4b/f94 x:0 0 0 2024-04-22T23:23:27.639 INFO:tasks.workunit.client.0.smithi067.stdout:5/499: symlink d5/db/d12/d8a/lb0 0 2024-04-22T23:23:27.639 INFO:tasks.workunit.client.0.smithi067.stdout:8/461: dread d1/d3/f1e [0,4194304] 0 2024-04-22T23:23:27.640 INFO:tasks.workunit.client.0.smithi067.stdout:3/428: write d3/d4/d1b/f7c [807536,40652] 0 2024-04-22T23:23:27.641 INFO:tasks.workunit.client.0.smithi067.stdout:0/453: mknod db/d60/c9c 0 2024-04-22T23:23:27.643 INFO:tasks.workunit.client.0.smithi067.stdout:2/543: dread d1/d10/f86 [0,4194304] 0 2024-04-22T23:23:27.643 INFO:tasks.workunit.client.0.smithi067.stdout:3/429: fdatasync d3/d4/d28/d4c/f52 0 2024-04-22T23:23:27.647 INFO:tasks.workunit.client.0.smithi067.stdout:8/462: dread d1/dd/d26/f6b [0,4194304] 0 2024-04-22T23:23:27.647 INFO:tasks.workunit.client.0.smithi067.stdout:0/454: mkdir db/d21/d29/d75/d9d 0 2024-04-22T23:23:27.648 INFO:tasks.workunit.client.0.smithi067.stdout:0/455: stat db/de/ff 0 2024-04-22T23:23:27.649 INFO:tasks.workunit.client.0.smithi067.stdout:9/375: symlink d5/l83 0 2024-04-22T23:23:27.650 INFO:tasks.workunit.client.0.smithi067.stdout:3/430: readlink d3/d4/d12/l13 0 2024-04-22T23:23:27.650 INFO:tasks.workunit.client.0.smithi067.stdout:0/456: write db/de/d2a/f39 [2000012,89245] 0 2024-04-22T23:23:27.651 INFO:tasks.workunit.client.0.smithi067.stdout:2/544: mkdir d1/d42/d61/dad 0 2024-04-22T23:23:27.654 INFO:tasks.workunit.client.0.smithi067.stdout:5/500: getdents d5/d3e/d87/d99 0 2024-04-22T23:23:27.656 INFO:tasks.workunit.client.0.smithi067.stdout:3/431: rmdir d3/d4/d1b 39 2024-04-22T23:23:27.657 INFO:tasks.workunit.client.0.smithi067.stdout:0/457: mkdir db/d60/d9e 0 2024-04-22T23:23:27.658 INFO:tasks.workunit.client.0.smithi067.stdout:0/458: write db/d21/d29/d2c/f72 [1354326,111502] 0 2024-04-22T23:23:27.659 INFO:tasks.workunit.client.0.smithi067.stdout:4/503: rename d12/d42/d4b/f91 to d12/d17/fb9 0 2024-04-22T23:23:27.661 INFO:tasks.workunit.client.0.smithi067.stdout:5/501: creat d5/d3a/d4d/fb1 x:0 0 0 2024-04-22T23:23:27.663 INFO:tasks.workunit.client.0.smithi067.stdout:4/504: truncate d12/d42/d4b/d77/d8c/f9a 829581 0 2024-04-22T23:23:27.664 INFO:tasks.workunit.client.0.smithi067.stdout:4/505: readlink d12/d42/d4b/lb2 0 2024-04-22T23:23:27.665 INFO:tasks.workunit.client.0.smithi067.stdout:0/459: unlink db/de/d2a/l30 0 2024-04-22T23:23:27.666 INFO:tasks.workunit.client.0.smithi067.stdout:4/506: dread - d12/d17/d47/f6c zero size 2024-04-22T23:23:27.666 INFO:tasks.workunit.client.0.smithi067.stdout:2/545: dwrite d1/d10/d72/d68/fab [0,4194304] 0 2024-04-22T23:23:27.667 INFO:tasks.workunit.client.0.smithi067.stdout:9/376: rename d5/d21/d2c/f3b to d5/dd/d18/d39/d3e/d3f/d63/f84 0 2024-04-22T23:23:27.667 INFO:tasks.workunit.client.0.smithi067.stdout:3/432: mkdir d3/d4/d28/d4c/d62/d95 0 2024-04-22T23:23:27.668 INFO:tasks.workunit.client.0.smithi067.stdout:8/463: link d1/dd/c2b d1/dd/d80/d6f/c92 0 2024-04-22T23:23:27.669 INFO:tasks.workunit.client.0.smithi067.stdout:5/502: mkdir d5/d1a/d8b/db2 0 2024-04-22T23:23:27.671 INFO:tasks.workunit.client.0.smithi067.stdout:4/507: rmdir d12/d17/d47 39 2024-04-22T23:23:27.672 INFO:tasks.workunit.client.0.smithi067.stdout:5/503: truncate d5/d28/d36/f9d 309314 0 2024-04-22T23:23:27.674 INFO:tasks.workunit.client.0.smithi067.stdout:2/546: creat d1/d10/d72/d53/fae x:0 0 0 2024-04-22T23:23:27.674 INFO:tasks.workunit.client.0.smithi067.stdout:9/377: dwrite d5/d21/f6c [0,4194304] 0 2024-04-22T23:23:27.675 INFO:tasks.workunit.client.0.smithi067.stdout:8/464: creat d1/dd/d26/f93 x:0 0 0 2024-04-22T23:23:27.675 INFO:tasks.workunit.client.0.smithi067.stdout:0/460: mkdir db/d21/d9f 0 2024-04-22T23:23:27.678 INFO:tasks.workunit.client.0.smithi067.stdout:8/465: dread d1/dd/f53 [0,4194304] 0 2024-04-22T23:23:27.679 INFO:tasks.workunit.client.0.smithi067.stdout:3/433: creat d3/d4/d10/f96 x:0 0 0 2024-04-22T23:23:27.679 INFO:tasks.workunit.client.0.smithi067.stdout:8/466: readlink d1/dd/d26/d62/d75/l7b 0 2024-04-22T23:23:27.680 INFO:tasks.workunit.client.0.smithi067.stdout:8/467: write d1/d3/f2a [2782734,87918] 0 2024-04-22T23:23:27.681 INFO:tasks.workunit.client.0.smithi067.stdout:5/504: rename d5/db/f93 to d5/d28/d36/d40/d57/d61/d89/fb3 0 2024-04-22T23:23:27.681 INFO:tasks.workunit.client.0.smithi067.stdout:8/468: fdatasync d1/dd/d80/f86 0 2024-04-22T23:23:27.713 INFO:tasks.workunit.client.0.smithi067.stdout:9/378: creat d5/d21/d47/d57/f85 x:0 0 0 2024-04-22T23:23:27.715 INFO:tasks.workunit.client.0.smithi067.stdout:8/469: mkdir d1/dd/d26/d62/d94 0 2024-04-22T23:23:27.716 INFO:tasks.workunit.client.0.smithi067.stdout:9/379: mknod d5/d21/d47/d56/c86 0 2024-04-22T23:23:27.716 INFO:tasks.workunit.client.0.smithi067.stdout:5/505: mkdir d5/d28/daf/db4 0 2024-04-22T23:23:27.717 INFO:tasks.workunit.client.0.smithi067.stdout:5/506: fdatasync d5/db/d35/f6f 0 2024-04-22T23:23:27.718 INFO:tasks.workunit.client.0.smithi067.stdout:9/380: write d5/d21/f2b [253998,75430] 0 2024-04-22T23:23:27.719 INFO:tasks.workunit.client.0.smithi067.stdout:5/507: chown d5/d3e/f77 16197027 1 2024-04-22T23:23:27.724 INFO:tasks.workunit.client.0.smithi067.stdout:9/381: dwrite d5/d21/f6c [0,4194304] 0 2024-04-22T23:23:27.725 INFO:tasks.workunit.client.0.smithi067.stdout:9/382: dread - d5/d21/d47/d56/f80 zero size 2024-04-22T23:23:27.727 INFO:tasks.workunit.client.0.smithi067.stdout:9/383: truncate d5/dd/d18/d39/d3e/d3f/f73 11035 0 2024-04-22T23:23:27.729 INFO:tasks.workunit.client.0.smithi067.stdout:9/384: unlink d5/d21/d47/d56/l6d 0 2024-04-22T23:23:27.730 INFO:tasks.workunit.client.0.smithi067.stdout:9/385: creat d5/dd/d18/d5c/f87 x:0 0 0 2024-04-22T23:23:27.732 INFO:tasks.workunit.client.0.smithi067.stdout:9/386: mknod d5/dd/d18/d39/d49/c88 0 2024-04-22T23:23:27.734 INFO:tasks.workunit.client.0.smithi067.stdout:9/387: mknod d5/d21/d47/c89 0 2024-04-22T23:23:27.735 INFO:tasks.workunit.client.0.smithi067.stdout:9/388: mknod d5/dd/d18/d39/d49/d54/c8a 0 2024-04-22T23:23:27.736 INFO:tasks.workunit.client.0.smithi067.stdout:9/389: chown d5/dd/d18/d3c/d40 6992636 1 2024-04-22T23:23:27.737 INFO:tasks.workunit.client.0.smithi067.stdout:9/390: mknod d5/dd/d18/d39/d4d/c8b 0 2024-04-22T23:23:27.743 INFO:tasks.workunit.client.0.smithi067.stdout:9/391: dwrite d5/dd/d18/d3c/d40/f62 [0,4194304] 0 2024-04-22T23:23:27.746 INFO:tasks.workunit.client.0.smithi067.stdout:9/392: mkdir d5/d21/d47/d56/d8c 0 2024-04-22T23:23:27.748 INFO:tasks.workunit.client.0.smithi067.stdout:9/393: dread - d5/dd/d18/d39/d3e/d3f/f7e zero size 2024-04-22T23:23:27.775 INFO:tasks.workunit.client.0.smithi067.stdout:7/372: dwrite d10/d14/d1d/f1f [0,4194304] 0 2024-04-22T23:23:27.801 INFO:tasks.workunit.client.0.smithi067.stdout:2/547: dread d1/d4f/fa3 [0,4194304] 0 2024-04-22T23:23:27.802 INFO:tasks.workunit.client.0.smithi067.stdout:2/548: readlink d1/d10/d72/l59 0 2024-04-22T23:23:27.811 INFO:tasks.workunit.client.0.smithi067.stdout:2/549: dwrite d1/d12/f4e [0,4194304] 0 2024-04-22T23:23:27.817 INFO:tasks.workunit.client.0.smithi067.stdout:0/461: link db/de/c80 db/d2e/ca0 0 2024-04-22T23:23:27.818 INFO:tasks.workunit.client.0.smithi067.stdout:2/550: dread d1/d10/d1c/f5d [0,4194304] 0 2024-04-22T23:23:27.819 INFO:tasks.workunit.client.0.smithi067.stdout:2/551: readlink d1/d42/d61/l65 0 2024-04-22T23:23:27.820 INFO:tasks.workunit.client.0.smithi067.stdout:0/462: creat db/d21/d29/d75/fa1 x:0 0 0 2024-04-22T23:23:27.823 INFO:tasks.workunit.client.0.smithi067.stdout:5/508: write d5/db/f2f [4580677,52964] 0 2024-04-22T23:23:27.824 INFO:tasks.workunit.client.0.smithi067.stdout:5/509: stat d5/db/d12/f22 0 2024-04-22T23:23:27.825 INFO:tasks.workunit.client.0.smithi067.stdout:5/510: truncate d5/d28/f64 1728139 0 2024-04-22T23:23:27.826 INFO:tasks.workunit.client.0.smithi067.stdout:5/511: write d5/d28/f69 [1396051,125515] 0 2024-04-22T23:23:27.828 INFO:tasks.workunit.client.0.smithi067.stdout:5/512: fdatasync d5/db/d12/d44/f56 0 2024-04-22T23:23:27.831 INFO:tasks.workunit.client.0.smithi067.stdout:5/513: creat d5/d28/d36/d40/d57/d61/d89/fb5 x:0 0 0 2024-04-22T23:23:27.832 INFO:tasks.workunit.client.0.smithi067.stdout:5/514: write d5/db/f2f [4935180,54796] 0 2024-04-22T23:23:27.835 INFO:tasks.workunit.client.0.smithi067.stdout:3/434: rename d3/d4/d1b/l46 to d3/d4/d12/l97 0 2024-04-22T23:23:27.839 INFO:tasks.workunit.client.0.smithi067.stdout:9/394: rename d5/d21/d2c/f46 to d5/dd/d18/d39/d4d/f8d 0 2024-04-22T23:23:27.840 INFO:tasks.workunit.client.0.smithi067.stdout:5/515: dwrite d5/d28/d36/d40/d57/d61/f82 [0,4194304] 0 2024-04-22T23:23:27.840 INFO:tasks.workunit.client.0.smithi067.stdout:9/395: chown d5/l5a 7312 1 2024-04-22T23:23:27.843 INFO:tasks.workunit.client.0.smithi067.stdout:3/435: symlink d3/d4/d1b/d93/l98 0 2024-04-22T23:23:27.847 INFO:tasks.workunit.client.0.smithi067.stdout:9/396: rmdir d5/dd/d18/d5c 39 2024-04-22T23:23:27.850 INFO:tasks.workunit.client.0.smithi067.stdout:5/516: creat d5/d28/daf/db4/fb6 x:0 0 0 2024-04-22T23:23:27.851 INFO:tasks.workunit.client.0.smithi067.stdout:1/396: sync 2024-04-22T23:23:27.851 INFO:tasks.workunit.client.0.smithi067.stdout:6/465: sync 2024-04-22T23:23:27.855 INFO:tasks.workunit.client.0.smithi067.stdout:5/517: rename d5/d1a/d8b to d5/d3e/db7 0 2024-04-22T23:23:27.856 INFO:tasks.workunit.client.0.smithi067.stdout:9/397: creat d5/d21/d47/d7f/f8e x:0 0 0 2024-04-22T23:23:27.857 INFO:tasks.workunit.client.0.smithi067.stdout:9/398: chown d5/dd/d18/d39/c77 9266 1 2024-04-22T23:23:27.858 INFO:tasks.workunit.client.0.smithi067.stdout:6/466: write d17/f21 [729624,103718] 0 2024-04-22T23:23:27.860 INFO:tasks.workunit.client.0.smithi067.stdout:1/397: creat d8/d9/d2f/d59/f81 x:0 0 0 2024-04-22T23:23:27.860 INFO:tasks.workunit.client.0.smithi067.stdout:7/373: write d10/d14/d1d/d26/f39 [4170130,130849] 0 2024-04-22T23:23:27.861 INFO:tasks.workunit.client.0.smithi067.stdout:3/436: creat d3/f99 x:0 0 0 2024-04-22T23:23:27.864 INFO:tasks.workunit.client.0.smithi067.stdout:6/467: dread d17/f2b [0,4194304] 0 2024-04-22T23:23:27.866 INFO:tasks.workunit.client.0.smithi067.stdout:5/518: symlink d5/d3e/d87/d99/lb8 0 2024-04-22T23:23:27.867 INFO:tasks.workunit.client.0.smithi067.stdout:6/468: read d17/f64 [3685351,55974] 0 2024-04-22T23:23:27.867 INFO:tasks.workunit.client.0.smithi067.stdout:9/399: dwrite d5/d21/f34 [0,4194304] 0 2024-04-22T23:23:27.872 INFO:tasks.workunit.client.0.smithi067.stdout:1/398: symlink d8/d9/d2f/d45/l82 0 2024-04-22T23:23:27.876 INFO:tasks.workunit.client.0.smithi067.stdout:5/519: dwrite d5/d28/d36/d40/f85 [4194304,4194304] 0 2024-04-22T23:23:27.877 INFO:tasks.workunit.client.0.smithi067.stdout:6/469: mknod d17/d83/c96 0 2024-04-22T23:23:27.883 INFO:tasks.workunit.client.0.smithi067.stdout:7/374: mkdir d10/d14/d1d/d26/d80 0 2024-04-22T23:23:27.886 INFO:tasks.workunit.client.0.smithi067.stdout:6/470: dwrite d17/d39/d3a/f61 [0,4194304] 0 2024-04-22T23:23:27.888 INFO:tasks.workunit.client.0.smithi067.stdout:5/520: mknod d5/d3e/d90/cb9 0 2024-04-22T23:23:27.892 INFO:tasks.workunit.client.0.smithi067.stdout:3/437: creat d3/f9a x:0 0 0 2024-04-22T23:23:27.893 INFO:tasks.workunit.client.0.smithi067.stdout:1/399: mkdir d8/d9/d2d/d4d/d83 0 2024-04-22T23:23:27.896 INFO:tasks.workunit.client.0.smithi067.stdout:7/375: write d10/d14/d1d/d26/f40 [786545,117002] 0 2024-04-22T23:23:27.897 INFO:tasks.workunit.client.0.smithi067.stdout:6/471: dread d17/f29 [0,4194304] 0 2024-04-22T23:23:27.898 INFO:tasks.workunit.client.0.smithi067.stdout:4/508: sync 2024-04-22T23:23:27.901 INFO:tasks.workunit.client.0.smithi067.stdout:3/438: rmdir d3/d4/d10/d1f 39 2024-04-22T23:23:27.902 INFO:tasks.workunit.client.0.smithi067.stdout:1/400: mkdir d8/d9/d2f/d33/d66/d84 0 2024-04-22T23:23:27.904 INFO:tasks.workunit.client.0.smithi067.stdout:1/401: chown d8/d9/f25 38769 1 2024-04-22T23:23:27.905 INFO:tasks.workunit.client.0.smithi067.stdout:2/552: truncate d1/d4f/fa3 7073959 0 2024-04-22T23:23:27.908 INFO:tasks.workunit.client.0.smithi067.stdout:7/376: symlink d10/d18/d35/l81 0 2024-04-22T23:23:27.910 INFO:tasks.workunit.client.0.smithi067.stdout:9/400: getdents d5/dd 0 2024-04-22T23:23:27.911 INFO:tasks.workunit.client.0.smithi067.stdout:1/402: truncate d8/d9/d2d/d4d/d48/f67 209987 0 2024-04-22T23:23:27.911 INFO:tasks.workunit.client.0.smithi067.stdout:0/463: dwrite db/d21/f5a [0,4194304] 0 2024-04-22T23:23:27.912 INFO:tasks.workunit.client.0.smithi067.stdout:5/521: mkdir d5/d28/d36/d40/d57/d5f/d79/dba 0 2024-04-22T23:23:27.914 INFO:tasks.workunit.client.0.smithi067.stdout:0/464: chown db/de/f1b 24255 1 2024-04-22T23:23:27.923 INFO:tasks.workunit.client.0.smithi067.stdout:1/403: mknod d8/d9/d2d/d4d/d48/c85 0 2024-04-22T23:23:27.925 INFO:tasks.workunit.client.0.smithi067.stdout:1/404: read d8/d12/d40/f69 [1664621,104704] 0 2024-04-22T23:23:27.926 INFO:tasks.workunit.client.0.smithi067.stdout:1/405: truncate d8/d4a/f5a 458970 0 2024-04-22T23:23:27.927 INFO:tasks.workunit.client.0.smithi067.stdout:1/406: fdatasync d8/d9/d2f/f38 0 2024-04-22T23:23:27.931 INFO:tasks.workunit.client.0.smithi067.stdout:2/553: creat d1/d42/d61/daa/faf x:0 0 0 2024-04-22T23:23:27.935 INFO:tasks.workunit.client.0.smithi067.stdout:0/465: creat db/d9a/d56/d6b/fa2 x:0 0 0 2024-04-22T23:23:27.937 INFO:tasks.workunit.client.0.smithi067.stdout:0/466: truncate db/d2e/f8b 315042 0 2024-04-22T23:23:27.940 INFO:tasks.workunit.client.0.smithi067.stdout:3/439: creat d3/d4/d10/d38/d53/d55/d5c/d70/f9b x:0 0 0 2024-04-22T23:23:27.946 INFO:tasks.workunit.client.0.smithi067.stdout:7/377: getdents d10/d14/d1d/d69 0 2024-04-22T23:23:27.947 INFO:tasks.workunit.client.0.smithi067.stdout:8/470: creat d1/dd/f95 x:0 0 0 2024-04-22T23:23:27.948 INFO:tasks.workunit.client.0.smithi067.stdout:3/440: mknod d3/d4/d28/d5f/c9c 0 2024-04-22T23:23:27.950 INFO:tasks.workunit.client.0.smithi067.stdout:9/401: getdents d5/dd/d18/d39/d49/d54 0 2024-04-22T23:23:27.952 INFO:tasks.workunit.client.0.smithi067.stdout:9/402: stat d5/dd/d18/d39/d49/d54 0 2024-04-22T23:23:27.953 INFO:tasks.workunit.client.0.smithi067.stdout:0/467: mknod db/d9a/d56/d70/d8c/ca3 0 2024-04-22T23:23:27.956 INFO:tasks.workunit.client.0.smithi067.stdout:7/378: unlink d10/d18/l79 0 2024-04-22T23:23:27.961 INFO:tasks.workunit.client.0.smithi067.stdout:9/403: mkdir d5/dd/d18/d39/d8f 0 2024-04-22T23:23:27.962 INFO:tasks.workunit.client.0.smithi067.stdout:3/441: dwrite d3/d4/d10/d38/d53/d55/f75 [0,4194304] 0 2024-04-22T23:23:27.965 INFO:tasks.workunit.client.0.smithi067.stdout:8/471: dwrite d1/dd/d80/d6f/f32 [4194304,4194304] 0 2024-04-22T23:23:27.966 INFO:tasks.workunit.client.0.smithi067.stdout:9/404: read d5/f10 [291495,14135] 0 2024-04-22T23:23:27.966 INFO:tasks.workunit.client.0.smithi067.stdout:1/407: getdents d8/d9/d2f/d33/d66 0 2024-04-22T23:23:27.970 INFO:tasks.workunit.client.0.smithi067.stdout:9/405: stat d5/d21/f6c 0 2024-04-22T23:23:27.972 INFO:tasks.workunit.client.0.smithi067.stdout:0/468: dwrite db/d9a/d56/f6d [0,4194304] 0 2024-04-22T23:23:27.975 INFO:tasks.workunit.client.0.smithi067.stdout:7/379: mkdir d10/d14/d1d/d26/d2c/d41/d82 0 2024-04-22T23:23:27.975 INFO:tasks.workunit.client.0.smithi067.stdout:0/469: readlink db/d21/d29/d2c/l3b 0 2024-04-22T23:23:27.979 INFO:tasks.workunit.client.0.smithi067.stdout:5/522: rename d5/d28/d36/d40/d57 to d5/db/d12/dbb 0 2024-04-22T23:23:27.980 INFO:tasks.workunit.client.0.smithi067.stdout:9/406: unlink d5/dd/d18/d39/d3e/d3f/l69 0 2024-04-22T23:23:27.982 INFO:tasks.workunit.client.0.smithi067.stdout:8/472: mknod d1/dd/d26/d62/d75/c96 0 2024-04-22T23:23:27.983 INFO:tasks.workunit.client.0.smithi067.stdout:6/472: write d17/f7f [1544455,106935] 0 2024-04-22T23:23:27.983 INFO:tasks.workunit.client.0.smithi067.stdout:1/408: mknod d8/d9/d2d/d4d/d70/d34/d5d/c86 0 2024-04-22T23:23:27.985 INFO:tasks.workunit.client.0.smithi067.stdout:5/523: read d5/f15 [2416607,45798] 0 2024-04-22T23:23:27.985 INFO:tasks.workunit.client.0.smithi067.stdout:8/473: fdatasync d1/d41/f74 0 2024-04-22T23:23:27.987 INFO:tasks.workunit.client.0.smithi067.stdout:0/470: unlink db/de/d11/d1a/l1d 0 2024-04-22T23:23:27.988 INFO:tasks.workunit.client.0.smithi067.stdout:6/473: write d17/f7f [2978083,97568] 0 2024-04-22T23:23:27.988 INFO:tasks.workunit.client.0.smithi067.stdout:9/407: mknod d5/dd/d18/d39/c90 0 2024-04-22T23:23:27.990 INFO:tasks.workunit.client.0.smithi067.stdout:2/554: getdents d1/d42/d61/daa 0 2024-04-22T23:23:27.992 INFO:tasks.workunit.client.0.smithi067.stdout:5/524: chown d5/d28/d36/f80 20 1 2024-04-22T23:23:27.992 INFO:tasks.workunit.client.0.smithi067.stdout:4/509: dwrite d12/d2e/f3e [0,4194304] 0 2024-04-22T23:23:27.993 INFO:tasks.workunit.client.0.smithi067.stdout:0/471: dread - db/d21/d29/d75/fa1 zero size 2024-04-22T23:23:27.995 INFO:tasks.workunit.client.0.smithi067.stdout:1/409: mkdir d8/d12/d36/d87 0 2024-04-22T23:23:27.996 INFO:tasks.workunit.client.0.smithi067.stdout:4/510: fdatasync d12/d42/f83 0 2024-04-22T23:23:27.997 INFO:tasks.workunit.client.0.smithi067.stdout:0/472: dread - db/d9a/d56/d6b/fa2 zero size 2024-04-22T23:23:27.999 INFO:tasks.workunit.client.0.smithi067.stdout:9/408: creat d5/dd/d18/d39/d3e/d3f/d63/f91 x:0 0 0 2024-04-22T23:23:28.001 INFO:tasks.workunit.client.0.smithi067.stdout:7/380: dwrite d10/d14/d1d/d26/f3a [0,4194304] 0 2024-04-22T23:23:28.004 INFO:tasks.workunit.client.0.smithi067.stdout:2/555: symlink d1/lb0 0 2024-04-22T23:23:28.007 INFO:tasks.workunit.client.0.smithi067.stdout:5/525: truncate d5/db/f23 523329 0 2024-04-22T23:23:28.009 INFO:tasks.workunit.client.0.smithi067.stdout:0/473: dwrite db/d21/d29/d2d/f42 [0,4194304] 0 2024-04-22T23:23:28.014 INFO:tasks.workunit.client.0.smithi067.stdout:6/474: creat d17/d39/d3a/d8d/f97 x:0 0 0 2024-04-22T23:23:28.016 INFO:tasks.workunit.client.0.smithi067.stdout:6/475: truncate d17/d39/f81 139146 0 2024-04-22T23:23:28.017 INFO:tasks.workunit.client.0.smithi067.stdout:4/511: rename d12/d42/d4b/d77/d8c/f9a to d12/d42/d4b/d77/fba 0 2024-04-22T23:23:28.019 INFO:tasks.workunit.client.0.smithi067.stdout:9/409: truncate d5/fa 1652662 0 2024-04-22T23:23:28.020 INFO:tasks.workunit.client.0.smithi067.stdout:9/410: fsync d5/dd/f26 0 2024-04-22T23:23:28.029 INFO:tasks.workunit.client.0.smithi067.stdout:4/512: rmdir d12/d42/d4b 39 2024-04-22T23:23:28.030 INFO:tasks.workunit.client.0.smithi067.stdout:7/381: symlink d10/d14/d1d/d26/d7e/l83 0 2024-04-22T23:23:28.031 INFO:tasks.workunit.client.0.smithi067.stdout:7/382: fsync d10/d47/d49/f54 0 2024-04-22T23:23:28.032 INFO:tasks.workunit.client.0.smithi067.stdout:7/383: readlink d10/d47/d49/l51 0 2024-04-22T23:23:28.034 INFO:tasks.workunit.client.0.smithi067.stdout:8/474: truncate d1/dd/f2f 122006 0 2024-04-22T23:23:28.036 INFO:tasks.workunit.client.0.smithi067.stdout:1/410: getdents d8/d9/d2f/d33/d66 0 2024-04-22T23:23:28.039 INFO:tasks.workunit.client.0.smithi067.stdout:4/513: truncate d12/f54 126909 0 2024-04-22T23:23:28.041 INFO:tasks.workunit.client.0.smithi067.stdout:9/411: creat d5/dd/d18/d3c/d78/f92 x:0 0 0 2024-04-22T23:23:28.042 INFO:tasks.workunit.client.0.smithi067.stdout:8/475: symlink d1/dd/d80/d6f/l97 0 2024-04-22T23:23:28.043 INFO:tasks.workunit.client.0.smithi067.stdout:2/556: link d1/d10/f86 d1/d10/d72/d53/d9a/fb1 0 2024-04-22T23:23:28.046 INFO:tasks.workunit.client.0.smithi067.stdout:1/411: rename d8/d9/d2d/c5f to d8/d12/c88 0 2024-04-22T23:23:28.047 INFO:tasks.workunit.client.0.smithi067.stdout:6/476: dwrite d17/d39/f3f [4194304,4194304] 0 2024-04-22T23:23:28.048 INFO:tasks.workunit.client.0.smithi067.stdout:5/526: dwrite d5/db/d12/f2b [0,4194304] 0 2024-04-22T23:23:28.049 INFO:tasks.workunit.client.0.smithi067.stdout:0/474: getdents db/d2e 0 2024-04-22T23:23:28.052 INFO:tasks.workunit.client.0.smithi067.stdout:9/412: dread d5/dd/d18/d3c/d40/f62 [0,4194304] 0 2024-04-22T23:23:28.057 INFO:tasks.workunit.client.0.smithi067.stdout:8/476: unlink d1/dd/d80/d6f/f70 0 2024-04-22T23:23:28.061 INFO:tasks.workunit.client.0.smithi067.stdout:6/477: dwrite d17/d39/d3a/f3b [4194304,4194304] 0 2024-04-22T23:23:28.063 INFO:tasks.workunit.client.0.smithi067.stdout:1/412: dread d8/f4b [0,4194304] 0 2024-04-22T23:23:28.064 INFO:tasks.workunit.client.0.smithi067.stdout:5/527: stat d5/d28/d36/d40/l66 0 2024-04-22T23:23:28.070 INFO:tasks.workunit.client.0.smithi067.stdout:2/557: write d1/d10/f30 [525003,37775] 0 2024-04-22T23:23:28.070 INFO:tasks.workunit.client.0.smithi067.stdout:1/413: dwrite d8/d12/f17 [4194304,4194304] 0 2024-04-22T23:23:28.072 INFO:tasks.workunit.client.0.smithi067.stdout:1/414: write f1 [4650387,124280] 0 2024-04-22T23:23:28.072 INFO:tasks.workunit.client.0.smithi067.stdout:8/477: unlink d1/d3/f1b 0 2024-04-22T23:23:28.076 INFO:tasks.workunit.client.0.smithi067.stdout:4/514: rename d12/c66 to d12/d17/d3f/d7a/cbb 0 2024-04-22T23:23:28.078 INFO:tasks.workunit.client.0.smithi067.stdout:7/384: getdents d10/d14 0 2024-04-22T23:23:28.079 INFO:tasks.workunit.client.0.smithi067.stdout:4/515: dread - d12/d2e/fb7 zero size 2024-04-22T23:23:28.080 INFO:tasks.workunit.client.0.smithi067.stdout:6/478: creat d17/d83/f98 x:0 0 0 2024-04-22T23:23:28.082 INFO:tasks.workunit.client.0.smithi067.stdout:9/413: mknod d5/dd/d18/d5c/c93 0 2024-04-22T23:23:28.084 INFO:tasks.workunit.client.0.smithi067.stdout:8/478: mkdir d1/d3/d5c/d98 0 2024-04-22T23:23:28.086 INFO:tasks.workunit.client.0.smithi067.stdout:2/558: dwrite d1/d10/d1c/f3a [0,4194304] 0 2024-04-22T23:23:28.086 INFO:tasks.workunit.client.0.smithi067.stdout:0/475: rename db/de/d2a/d48 to db/de/d11/d1a/da4 0 2024-04-22T23:23:28.092 INFO:tasks.workunit.client.0.smithi067.stdout:1/415: dwrite d8/d9/d2f/f79 [0,4194304] 0 2024-04-22T23:23:28.092 INFO:tasks.workunit.client.0.smithi067.stdout:5/528: getdents d5/d3a/d4d/da9 0 2024-04-22T23:23:28.092 INFO:tasks.workunit.client.0.smithi067.stdout:8/479: creat d1/d3/d5c/f99 x:0 0 0 2024-04-22T23:23:28.095 INFO:tasks.workunit.client.0.smithi067.stdout:5/529: truncate d5/fa4 585609 0 2024-04-22T23:23:28.100 INFO:tasks.workunit.client.0.smithi067.stdout:2/559: rmdir d1/d42 39 2024-04-22T23:23:28.100 INFO:tasks.workunit.client.0.smithi067.stdout:6/479: dwrite f12 [0,4194304] 0 2024-04-22T23:23:28.100 INFO:tasks.workunit.client.0.smithi067.stdout:7/385: mknod d10/d14/d1d/c84 0 2024-04-22T23:23:28.100 INFO:tasks.workunit.client.0.smithi067.stdout:1/416: truncate d8/d9/d2f/d33/d66/f73 639065 0 2024-04-22T23:23:28.101 INFO:tasks.workunit.client.0.smithi067.stdout:5/530: dread d5/db/d35/f6f [0,4194304] 0 2024-04-22T23:23:28.102 INFO:tasks.workunit.client.0.smithi067.stdout:9/414: mkdir d5/dd/d18/d94 0 2024-04-22T23:23:28.105 INFO:tasks.workunit.client.0.smithi067.stdout:1/417: write d8/d9/d2f/d31/f58 [128714,88348] 0 2024-04-22T23:23:28.106 INFO:tasks.workunit.client.0.smithi067.stdout:8/480: readlink d1/d3/l51 0 2024-04-22T23:23:28.108 INFO:tasks.workunit.client.0.smithi067.stdout:2/560: write d1/d42/d5f/f90 [513751,130793] 0 2024-04-22T23:23:28.114 INFO:tasks.workunit.client.0.smithi067.stdout:5/531: creat d5/d3e/db7/fbc x:0 0 0 2024-04-22T23:23:28.115 INFO:tasks.workunit.client.0.smithi067.stdout:9/415: creat d5/dd/d18/d39/d4d/f95 x:0 0 0 2024-04-22T23:23:28.115 INFO:tasks.workunit.client.0.smithi067.stdout:6/480: truncate d17/d39/d53/d90/d46/d62/f72 4206940 0 2024-04-22T23:23:28.117 INFO:tasks.workunit.client.0.smithi067.stdout:8/481: dwrite d1/dd/d80/d6f/f57 [8388608,4194304] 0 2024-04-22T23:23:28.121 INFO:tasks.workunit.client.0.smithi067.stdout:2/561: fsync d1/d42/f4b 0 2024-04-22T23:23:28.123 INFO:tasks.workunit.client.0.smithi067.stdout:7/386: symlink d10/d14/d1d/d26/l85 0 2024-04-22T23:23:28.124 INFO:tasks.workunit.client.0.smithi067.stdout:8/482: chown d1/dd/d26/d62/d75/c96 30 1 2024-04-22T23:23:28.128 INFO:tasks.workunit.client.0.smithi067.stdout:0/476: getdents db/d9a/d56/d70/d8c 0 2024-04-22T23:23:28.129 INFO:tasks.workunit.client.0.smithi067.stdout:4/516: dread d12/f2a [0,4194304] 0 2024-04-22T23:23:28.130 INFO:tasks.workunit.client.0.smithi067.stdout:2/562: mkdir d1/d12/db2 0 2024-04-22T23:23:28.132 INFO:tasks.workunit.client.0.smithi067.stdout:7/387: write d10/d14/d1d/d46/d50/f30 [904230,116941] 0 2024-04-22T23:23:28.134 INFO:tasks.workunit.client.0.smithi067.stdout:5/532: dwrite d5/db/d12/dbb/d61/d89/fb5 [0,4194304] 0 2024-04-22T23:23:28.134 INFO:tasks.workunit.client.0.smithi067.stdout:1/418: dwrite d8/d9/fd [0,4194304] 0 2024-04-22T23:23:28.135 INFO:tasks.workunit.client.0.smithi067.stdout:4/517: dread - d12/d17/d47/f6c zero size 2024-04-22T23:23:28.137 INFO:tasks.workunit.client.0.smithi067.stdout:5/533: dread - d5/db/d12/dbb/d61/f78 zero size 2024-04-22T23:23:28.138 INFO:tasks.workunit.client.0.smithi067.stdout:0/477: symlink db/la5 0 2024-04-22T23:23:28.140 INFO:tasks.workunit.client.0.smithi067.stdout:2/563: symlink d1/d10/d72/lb3 0 2024-04-22T23:23:28.143 INFO:tasks.workunit.client.0.smithi067.stdout:7/388: creat d10/d14/d1d/d46/d6a/f86 x:0 0 0 2024-04-22T23:23:28.143 INFO:tasks.workunit.client.0.smithi067.stdout:4/518: mknod d12/d17/d31/cbc 0 2024-04-22T23:23:28.145 INFO:tasks.workunit.client.0.smithi067.stdout:9/416: dread d5/dd/f32 [0,4194304] 0 2024-04-22T23:23:28.150 INFO:tasks.workunit.client.0.smithi067.stdout:1/419: mknod d8/d9/d2d/d4d/d83/c89 0 2024-04-22T23:23:28.150 INFO:tasks.workunit.client.0.smithi067.stdout:2/564: creat d1/d42/d6b/fb4 x:0 0 0 2024-04-22T23:23:28.150 INFO:tasks.workunit.client.0.smithi067.stdout:0/478: dread db/de/d2a/f5e [0,4194304] 0 2024-04-22T23:23:28.151 INFO:tasks.workunit.client.0.smithi067.stdout:5/534: symlink d5/d3a/d4d/da9/lbd 0 2024-04-22T23:23:28.151 INFO:tasks.workunit.client.0.smithi067.stdout:0/479: stat db/d21/l97 0 2024-04-22T23:23:28.151 INFO:tasks.workunit.client.0.smithi067.stdout:9/417: mkdir d5/dd/d18/d39/d49/d96 0 2024-04-22T23:23:28.152 INFO:tasks.workunit.client.0.smithi067.stdout:5/535: dread - d5/db/d12/dbb/d61/f78 zero size 2024-04-22T23:23:28.153 INFO:tasks.workunit.client.0.smithi067.stdout:4/519: fdatasync d12/f54 0 2024-04-22T23:23:28.154 INFO:tasks.workunit.client.0.smithi067.stdout:0/480: write db/de/d11/f32 [334849,99818] 0 2024-04-22T23:23:28.155 INFO:tasks.workunit.client.0.smithi067.stdout:0/481: fsync db/de/f1b 0 2024-04-22T23:23:28.214 INFO:tasks.workunit.client.0.smithi067.stdout:2/565: creat d1/d10/d72/d68/da1/fb5 x:0 0 0 2024-04-22T23:23:28.214 INFO:tasks.workunit.client.0.smithi067.stdout:1/420: link d8/d9/f29 d8/d9/d2f/d45/f8a 0 2024-04-22T23:23:28.214 INFO:tasks.workunit.client.0.smithi067.stdout:2/566: chown d1/d42/d6b/f8c 1814 1 2024-04-22T23:23:28.218 INFO:tasks.workunit.client.0.smithi067.stdout:0/482: dread db/d21/d29/d75/f86 [0,4194304] 0 2024-04-22T23:23:28.219 INFO:tasks.workunit.client.0.smithi067.stdout:9/418: rmdir d5/dd/d18/d94 0 2024-04-22T23:23:28.219 INFO:tasks.workunit.client.0.smithi067.stdout:9/419: stat d5/d21/d47/d56/f80 0 2024-04-22T23:23:28.220 INFO:tasks.workunit.client.0.smithi067.stdout:0/483: mknod db/d60/ca6 0 2024-04-22T23:23:28.222 INFO:tasks.workunit.client.0.smithi067.stdout:0/484: truncate db/de/f1c 2850608 0 2024-04-22T23:23:28.222 INFO:tasks.workunit.client.0.smithi067.stdout:1/421: getdents d8/d9 0 2024-04-22T23:23:28.223 INFO:tasks.workunit.client.0.smithi067.stdout:1/422: dread - d8/d9/d2d/d4d/f23 zero size 2024-04-22T23:23:28.224 INFO:tasks.workunit.client.0.smithi067.stdout:0/485: write db/d21/d29/d2d/f42 [206533,9786] 0 2024-04-22T23:23:28.224 INFO:tasks.workunit.client.0.smithi067.stdout:1/423: fdatasync d8/d9/f43 0 2024-04-22T23:23:28.225 INFO:tasks.workunit.client.0.smithi067.stdout:9/420: dread d5/dd/f26 [0,4194304] 0 2024-04-22T23:23:28.225 INFO:tasks.workunit.client.0.smithi067.stdout:0/486: dread - db/de/d2a/f99 zero size 2024-04-22T23:23:28.226 INFO:tasks.workunit.client.0.smithi067.stdout:0/487: write db/d9a/f63 [1414925,121908] 0 2024-04-22T23:23:28.227 INFO:tasks.workunit.client.0.smithi067.stdout:9/421: creat d5/d21/d47/d57/f97 x:0 0 0 2024-04-22T23:23:28.228 INFO:tasks.workunit.client.0.smithi067.stdout:1/424: creat d8/d9/d2d/d4d/d70/d34/d5d/f8b x:0 0 0 2024-04-22T23:23:28.230 INFO:tasks.workunit.client.0.smithi067.stdout:1/425: symlink d8/d9/d2d/d4d/d70/l8c 0 2024-04-22T23:23:28.233 INFO:tasks.workunit.client.0.smithi067.stdout:9/422: dwrite d5/f36 [0,4194304] 0 2024-04-22T23:23:28.235 INFO:tasks.workunit.client.0.smithi067.stdout:9/423: write d5/dd/d18/d39/d3e/d3f/d63/f91 [1009469,28721] 0 2024-04-22T23:23:28.236 INFO:tasks.workunit.client.0.smithi067.stdout:9/424: stat d5/dd/d18/d39/f51 0 2024-04-22T23:23:28.236 INFO:tasks.workunit.client.0.smithi067.stdout:9/425: chown d5/dd/d18/d39 894444 1 2024-04-22T23:23:28.237 INFO:tasks.workunit.client.0.smithi067.stdout:9/426: write d5/dd/d18/d39/d49/f7b [1044025,29565] 0 2024-04-22T23:23:28.239 INFO:tasks.workunit.client.0.smithi067.stdout:9/427: mknod d5/dd/d18/d3c/c98 0 2024-04-22T23:23:28.240 INFO:tasks.workunit.client.0.smithi067.stdout:9/428: chown d5/dd/d18/d39/d49/d96 0 1 2024-04-22T23:23:28.241 INFO:tasks.workunit.client.0.smithi067.stdout:9/429: mkdir d5/dd/d18/d39/d49/d54/d99 0 2024-04-22T23:23:28.289 INFO:tasks.workunit.client.0.smithi067.stdout:0/488: dread db/d2e/f5d [0,4194304] 0 2024-04-22T23:23:28.289 INFO:tasks.workunit.client.0.smithi067.stdout:8/483: dread d1/dd/d80/d6f/f57 [0,4194304] 0 2024-04-22T23:23:28.290 INFO:tasks.workunit.client.0.smithi067.stdout:0/489: dread - db/de/d2a/f99 zero size 2024-04-22T23:23:28.291 INFO:tasks.workunit.client.0.smithi067.stdout:0/490: write db/d21/d29/d2d/f49 [3812909,130693] 0 2024-04-22T23:23:28.292 INFO:tasks.workunit.client.0.smithi067.stdout:8/484: rename d1/dd/d80/d6f/c36 to d1/d3/d5c/d98/c9a 0 2024-04-22T23:23:28.294 INFO:tasks.workunit.client.0.smithi067.stdout:0/491: creat db/d21/d29/fa7 x:0 0 0 2024-04-22T23:23:28.296 INFO:tasks.workunit.client.0.smithi067.stdout:8/485: rename d1/dd/d80/d6f/l8e to d1/dd/d26/d62/d75/l9b 0 2024-04-22T23:23:28.297 INFO:tasks.workunit.client.0.smithi067.stdout:0/492: mknod db/d60/ca8 0 2024-04-22T23:23:28.297 INFO:tasks.workunit.client.0.smithi067.stdout:8/486: write d1/f6 [3873633,41736] 0 2024-04-22T23:23:28.301 INFO:tasks.workunit.client.0.smithi067.stdout:8/487: read d1/d3/f59 [343446,68420] 0 2024-04-22T23:23:28.302 INFO:tasks.workunit.client.0.smithi067.stdout:0/493: rename db/d9a/d56/l90 to db/d2e/d46/la9 0 2024-04-22T23:23:28.314 INFO:tasks.workunit.client.0.smithi067.stdout:5/536: dwrite d5/d3e/f42 [0,4194304] 0 2024-04-22T23:23:28.314 INFO:tasks.workunit.client.0.smithi067.stdout:4/520: dwrite d12/d5f/f35 [0,4194304] 0 2024-04-22T23:23:28.316 INFO:tasks.workunit.client.0.smithi067.stdout:9/430: truncate d5/dd/d18/d3c/d40/f6b 2785106 0 2024-04-22T23:23:28.323 INFO:tasks.workunit.client.0.smithi067.stdout:1/426: dwrite d8/d9/d2d/d4d/f1f [0,4194304] 0 2024-04-22T23:23:28.324 INFO:tasks.workunit.client.0.smithi067.stdout:9/431: truncate d5/dd/d18/d39/d3e/d3f/f73 727340 0 2024-04-22T23:23:28.326 INFO:tasks.workunit.client.0.smithi067.stdout:5/537: rename d5/d28/d36/d40/c49 to d5/d3e/cbe 0 2024-04-22T23:23:28.332 INFO:tasks.workunit.client.0.smithi067.stdout:5/538: mkdir d5/d28/daf/dbf 0 2024-04-22T23:23:28.334 INFO:tasks.workunit.client.0.smithi067.stdout:4/521: rename d12/d42/d4b/d77/cad to d12/d2e/d9b/cbd 0 2024-04-22T23:23:28.338 INFO:tasks.workunit.client.0.smithi067.stdout:5/539: write d5/d3a/f58 [4527083,28424] 0 2024-04-22T23:23:28.339 INFO:tasks.workunit.client.0.smithi067.stdout:0/494: dwrite db/d21/d29/d75/f86 [0,4194304] 0 2024-04-22T23:23:28.339 INFO:tasks.workunit.client.0.smithi067.stdout:4/522: write d12/d17/d3f/d84/f9f [2250493,74328] 0 2024-04-22T23:23:28.339 INFO:tasks.workunit.client.0.smithi067.stdout:9/432: rename d5/dd/d18/d39/d3e/f68 to d5/dd/d18/d39/d49/d54/f9a 0 2024-04-22T23:23:28.341 INFO:tasks.workunit.client.0.smithi067.stdout:5/540: write d5/fa4 [370206,39975] 0 2024-04-22T23:23:28.342 INFO:tasks.workunit.client.0.smithi067.stdout:4/523: truncate d12/d42/f83 5045515 0 2024-04-22T23:23:28.343 INFO:tasks.workunit.client.0.smithi067.stdout:4/524: chown d12/d5f/d2b/f72 72872393 1 2024-04-22T23:23:28.345 INFO:tasks.workunit.client.0.smithi067.stdout:9/433: mknod d5/c9b 0 2024-04-22T23:23:28.346 INFO:tasks.workunit.client.0.smithi067.stdout:0/495: creat db/faa x:0 0 0 2024-04-22T23:23:28.350 INFO:tasks.workunit.client.0.smithi067.stdout:9/434: mknod d5/d21/d47/d57/c9c 0 2024-04-22T23:23:28.353 INFO:tasks.workunit.client.0.smithi067.stdout:9/435: write d5/dd/d18/d39/d3e/f82 [465479,39349] 0 2024-04-22T23:23:28.358 INFO:tasks.workunit.client.0.smithi067.stdout:0/496: dread db/d2e/d46/f82 [0,4194304] 0 2024-04-22T23:23:28.359 INFO:tasks.workunit.client.0.smithi067.stdout:0/497: dread - db/d2e/f7c zero size 2024-04-22T23:23:28.360 INFO:tasks.workunit.client.0.smithi067.stdout:9/436: rename d5/l15 to d5/dd/d18/d39/d3e/d3f/l9d 0 2024-04-22T23:23:28.360 INFO:tasks.workunit.client.0.smithi067.stdout:1/427: dread d8/d9/d2d/d4d/d48/f55 [0,4194304] 0 2024-04-22T23:23:28.362 INFO:tasks.workunit.client.0.smithi067.stdout:9/437: write d5/d21/d47/f72 [251023,114256] 0 2024-04-22T23:23:28.362 INFO:tasks.workunit.client.0.smithi067.stdout:1/428: fdatasync d8/d12/d36/f4e 0 2024-04-22T23:23:28.363 INFO:tasks.workunit.client.0.smithi067.stdout:1/429: readlink d8/d9/d2f/d59/l63 0 2024-04-22T23:23:28.364 INFO:tasks.workunit.client.0.smithi067.stdout:0/498: creat db/d21/fab x:0 0 0 2024-04-22T23:23:28.365 INFO:tasks.workunit.client.0.smithi067.stdout:0/499: chown db/d21/d29/f3f 104969 1 2024-04-22T23:23:28.365 INFO:tasks.workunit.client.0.smithi067.stdout:9/438: creat d5/dd/d18/d39/d4d/f9e x:0 0 0 2024-04-22T23:23:28.369 INFO:tasks.workunit.client.0.smithi067.stdout:9/439: unlink d5/d21/f6c 0 2024-04-22T23:23:28.371 INFO:tasks.workunit.client.0.smithi067.stdout:0/500: dwrite db/d9a/f3a [4194304,4194304] 0 2024-04-22T23:23:28.372 INFO:tasks.workunit.client.0.smithi067.stdout:9/440: chown d5/dd/d18/d39/d49/d54/d99 8269955 1 2024-04-22T23:23:28.374 INFO:tasks.workunit.client.0.smithi067.stdout:5/541: truncate d5/fa4 466843 0 2024-04-22T23:23:28.380 INFO:tasks.workunit.client.0.smithi067.stdout:1/430: write d8/d9/d2f/d45/f8a [1007905,71682] 0 2024-04-22T23:23:28.382 INFO:tasks.workunit.client.0.smithi067.stdout:5/542: chown d5/db/d12/dbb/d61/f78 8073017 1 2024-04-22T23:23:28.385 INFO:tasks.workunit.client.0.smithi067.stdout:5/543: dread - d5/d3e/db7/d9a/fac zero size 2024-04-22T23:23:28.391 INFO:tasks.workunit.client.0.smithi067.stdout:4/525: dwrite d12/d5f/f27 [0,4194304] 0 2024-04-22T23:23:28.391 INFO:tasks.workunit.client.0.smithi067.stdout:9/441: dwrite d5/d21/d47/f72 [0,4194304] 0 2024-04-22T23:23:28.391 INFO:tasks.workunit.client.0.smithi067.stdout:1/431: chown d8/d12/d40 72150826 1 2024-04-22T23:23:28.393 INFO:tasks.workunit.client.0.smithi067.stdout:0/501: dwrite db/de/d11/f7e [0,4194304] 0 2024-04-22T23:23:28.403 INFO:tasks.workunit.client.0.smithi067.stdout:9/442: dwrite d5/f10 [0,4194304] 0 2024-04-22T23:23:28.404 INFO:tasks.workunit.client.0.smithi067.stdout:4/526: chown d12/l2c 12750 1 2024-04-22T23:23:28.406 INFO:tasks.workunit.client.0.smithi067.stdout:0/502: mknod db/d21/d29/d2c/cac 0 2024-04-22T23:23:28.406 INFO:tasks.workunit.client.0.smithi067.stdout:5/544: chown d5/d3e/d90/c9f 4047572 1 2024-04-22T23:23:28.407 INFO:tasks.workunit.client.0.smithi067.stdout:0/503: stat db/de/d11/l78 0 2024-04-22T23:23:28.408 INFO:tasks.workunit.client.0.smithi067.stdout:9/443: creat d5/d21/d47/d7f/f9f x:0 0 0 2024-04-22T23:23:28.409 INFO:tasks.workunit.client.0.smithi067.stdout:0/504: write db/d9a/d56/d6b/fa2 [411607,27909] 0 2024-04-22T23:23:28.413 INFO:tasks.workunit.client.0.smithi067.stdout:9/444: unlink d5/dd/d18/l31 0 2024-04-22T23:23:28.415 INFO:tasks.workunit.client.0.smithi067.stdout:4/527: rename d12/d5f/f27 to d12/d17/d3f/fbe 0 2024-04-22T23:23:28.416 INFO:tasks.workunit.client.0.smithi067.stdout:0/505: symlink db/de/lad 0 2024-04-22T23:23:28.417 INFO:tasks.workunit.client.0.smithi067.stdout:4/528: write d12/f38 [658312,3607] 0 2024-04-22T23:23:28.424 INFO:tasks.workunit.client.0.smithi067.stdout:0/506: rename db/de/d11/d1a/f38 to db/fae 0 2024-04-22T23:23:28.425 INFO:tasks.workunit.client.0.smithi067.stdout:0/507: truncate db/f6a 4824624 0 2024-04-22T23:23:28.427 INFO:tasks.workunit.client.0.smithi067.stdout:4/529: link d12/d17/d31/f67 d12/d42/d4b/fbf 0 2024-04-22T23:23:28.427 INFO:tasks.workunit.client.0.smithi067.stdout:3/442: sync 2024-04-22T23:23:28.430 INFO:tasks.workunit.client.0.smithi067.stdout:0/508: creat db/faf x:0 0 0 2024-04-22T23:23:28.431 INFO:tasks.workunit.client.0.smithi067.stdout:3/443: read d3/d4/d28/d4a/f73 [111378,6351] 0 2024-04-22T23:23:28.431 INFO:tasks.workunit.client.0.smithi067.stdout:4/530: symlink d12/d42/lc0 0 2024-04-22T23:23:28.433 INFO:tasks.workunit.client.0.smithi067.stdout:1/432: dwrite d8/f1d [0,4194304] 0 2024-04-22T23:23:28.438 INFO:tasks.workunit.client.0.smithi067.stdout:0/509: symlink db/d9a/lb0 0 2024-04-22T23:23:28.441 INFO:tasks.workunit.client.0.smithi067.stdout:5/545: dwrite d5/d28/d36/f75 [0,4194304] 0 2024-04-22T23:23:28.445 INFO:tasks.workunit.client.0.smithi067.stdout:4/531: dwrite d12/f52 [4194304,4194304] 0 2024-04-22T23:23:28.445 INFO:tasks.workunit.client.0.smithi067.stdout:3/444: symlink d3/l9d 0 2024-04-22T23:23:28.448 INFO:tasks.workunit.client.0.smithi067.stdout:5/546: truncate d5/d28/d36/f75 4663532 0 2024-04-22T23:23:28.449 INFO:tasks.workunit.client.0.smithi067.stdout:9/445: dwrite d5/f9 [4194304,4194304] 0 2024-04-22T23:23:28.451 INFO:tasks.workunit.client.0.smithi067.stdout:4/532: mkdir d12/d42/d4b/d77/d9d/dc1 0 2024-04-22T23:23:28.451 INFO:tasks.workunit.client.0.smithi067.stdout:9/446: stat d5/d21/d47/d56/f7c 0 2024-04-22T23:23:28.452 INFO:tasks.workunit.client.0.smithi067.stdout:3/445: creat d3/d4/d10/d38/d53/d55/d5c/f9e x:0 0 0 2024-04-22T23:23:28.457 INFO:tasks.workunit.client.0.smithi067.stdout:5/547: rename d5/d3a/l50 to d5/db/d12/dbb/d61/d89/lc0 0 2024-04-22T23:23:28.461 INFO:tasks.workunit.client.0.smithi067.stdout:9/447: rename d5/dd/d18/d39/d49/d54/l5f to d5/d21/d47/d7f/la0 0 2024-04-22T23:23:28.461 INFO:tasks.workunit.client.0.smithi067.stdout:5/548: read d5/d28/f69 [378286,103739] 0 2024-04-22T23:23:28.464 INFO:tasks.workunit.client.0.smithi067.stdout:3/446: creat d3/d4/d28/d4c/d90/f9f x:0 0 0 2024-04-22T23:23:28.464 INFO:tasks.workunit.client.0.smithi067.stdout:0/510: dread db/d21/f8d [0,4194304] 0 2024-04-22T23:23:28.469 INFO:tasks.workunit.client.0.smithi067.stdout:3/447: creat d3/d4/d1b/d93/fa0 x:0 0 0 2024-04-22T23:23:28.470 INFO:tasks.workunit.client.0.smithi067.stdout:5/549: creat d5/d3e/db7/db2/fc1 x:0 0 0 2024-04-22T23:23:28.474 INFO:tasks.workunit.client.0.smithi067.stdout:0/511: dwrite db/d9a/f3a [0,4194304] 0 2024-04-22T23:23:28.475 INFO:tasks.workunit.client.0.smithi067.stdout:9/448: dwrite d5/dd/d18/d39/f51 [0,4194304] 0 2024-04-22T23:23:28.481 INFO:tasks.workunit.client.0.smithi067.stdout:0/512: mknod db/d21/d29/cb1 0 2024-04-22T23:23:28.485 INFO:tasks.workunit.client.0.smithi067.stdout:0/513: symlink db/d21/lb2 0 2024-04-22T23:23:28.487 INFO:tasks.workunit.client.0.smithi067.stdout:0/514: chown db/de/d2a/c51 190022 1 2024-04-22T23:23:28.491 INFO:tasks.workunit.client.0.smithi067.stdout:3/448: rmdir d3 39 2024-04-22T23:23:28.492 INFO:tasks.workunit.client.0.smithi067.stdout:4/533: dwrite d12/d5f/f5a [0,4194304] 0 2024-04-22T23:23:28.495 INFO:tasks.workunit.client.0.smithi067.stdout:3/449: chown d3/d4/d28/d4a 23414 1 2024-04-22T23:23:28.501 INFO:tasks.workunit.client.0.smithi067.stdout:0/515: dwrite db/d2e/d46/f82 [0,4194304] 0 2024-04-22T23:23:28.501 INFO:tasks.workunit.client.0.smithi067.stdout:3/450: truncate d3/f99 341480 0 2024-04-22T23:23:28.508 INFO:tasks.workunit.client.0.smithi067.stdout:3/451: dread - d3/d4/d12/f6e zero size 2024-04-22T23:23:28.509 INFO:tasks.workunit.client.0.smithi067.stdout:9/449: dwrite d5/fa [0,4194304] 0 2024-04-22T23:23:28.511 INFO:tasks.workunit.client.0.smithi067.stdout:5/550: dwrite d5/db/d35/f39 [0,4194304] 0 2024-04-22T23:23:28.512 INFO:tasks.workunit.client.0.smithi067.stdout:4/534: dwrite d12/f52 [4194304,4194304] 0 2024-04-22T23:23:28.512 INFO:tasks.workunit.client.0.smithi067.stdout:5/551: readlink d5/d1a/l4c 0 2024-04-22T23:23:28.513 INFO:tasks.workunit.client.0.smithi067.stdout:9/450: fsync d5/dd/f32 0 2024-04-22T23:23:28.516 INFO:tasks.workunit.client.0.smithi067.stdout:4/535: rename ff to d12/d5f/d7e/fc2 0 2024-04-22T23:23:28.517 INFO:tasks.workunit.client.0.smithi067.stdout:3/452: creat d3/d4/d10/d38/fa1 x:0 0 0 2024-04-22T23:23:28.519 INFO:tasks.workunit.client.0.smithi067.stdout:3/453: fdatasync d3/d4/d12/d4b/f6c 0 2024-04-22T23:23:28.523 INFO:tasks.workunit.client.0.smithi067.stdout:4/536: dwrite d12/d5f/f41 [0,4194304] 0 2024-04-22T23:23:28.524 INFO:tasks.workunit.client.0.smithi067.stdout:3/454: mkdir d3/d4/d28/d4c/d90/da2 0 2024-04-22T23:23:28.527 INFO:tasks.workunit.client.0.smithi067.stdout:4/537: unlink d12/d5f/f87 0 2024-04-22T23:23:28.528 INFO:tasks.workunit.client.0.smithi067.stdout:9/451: link d5/dd/d18/d3c/d40/f6b d5/dd/d18/d39/d49/d54/d99/fa1 0 2024-04-22T23:23:28.531 INFO:tasks.workunit.client.0.smithi067.stdout:9/452: creat d5/d21/d47/d57/fa2 x:0 0 0 2024-04-22T23:23:28.532 INFO:tasks.workunit.client.0.smithi067.stdout:3/455: rename d3/d4/d28/c2b to d3/d4/d28/d4c/d62/d95/ca3 0 2024-04-22T23:23:28.536 INFO:tasks.workunit.client.0.smithi067.stdout:3/456: mknod d3/d4/d10/d38/d53/d55/d74/ca4 0 2024-04-22T23:23:28.536 INFO:tasks.workunit.client.0.smithi067.stdout:4/538: dwrite d12/d5f/d7e/fc2 [0,4194304] 0 2024-04-22T23:23:28.541 INFO:tasks.workunit.client.0.smithi067.stdout:9/453: dwrite d5/dd/d18/d39/d49/d54/f79 [0,4194304] 0 2024-04-22T23:23:28.543 INFO:tasks.workunit.client.0.smithi067.stdout:4/539: mkdir d12/d42/d4b/d70/d8e/d56/d8b/dc3 0 2024-04-22T23:23:28.545 INFO:tasks.workunit.client.0.smithi067.stdout:9/454: symlink d5/dd/d18/d3c/la3 0 2024-04-22T23:23:28.548 INFO:tasks.workunit.client.0.smithi067.stdout:4/540: write d12/d42/d4b/d77/d8c/fa5 [650878,72481] 0 2024-04-22T23:23:28.549 INFO:tasks.workunit.client.0.smithi067.stdout:3/457: dwrite d3/d4/d28/d4c/f52 [0,4194304] 0 2024-04-22T23:23:28.555 INFO:tasks.workunit.client.0.smithi067.stdout:4/541: creat d12/d17/d3f/fc4 x:0 0 0 2024-04-22T23:23:28.557 INFO:tasks.workunit.client.0.smithi067.stdout:3/458: link d3/d4/d12/l97 d3/d4/d10/d1f/d3c/la5 0 2024-04-22T23:23:28.558 INFO:tasks.workunit.client.0.smithi067.stdout:4/542: getdents d12/d5f/d2b/d97 0 2024-04-22T23:23:28.559 INFO:tasks.workunit.client.0.smithi067.stdout:3/459: unlink d3/c4f 0 2024-04-22T23:23:28.562 INFO:tasks.workunit.client.0.smithi067.stdout:4/543: dread d12/d5f/d2b/db6/f9e [0,4194304] 0 2024-04-22T23:23:28.564 INFO:tasks.workunit.client.0.smithi067.stdout:3/460: link d3/d4/d12/f1d d3/d4/d10/d8a/fa6 0 2024-04-22T23:23:28.566 INFO:tasks.workunit.client.0.smithi067.stdout:3/461: write d3/d4/d28/d40/f45 [835213,129102] 0 2024-04-22T23:23:28.568 INFO:tasks.workunit.client.0.smithi067.stdout:3/462: symlink d3/d4/d10/d38/d53/d55/la7 0 2024-04-22T23:23:28.568 INFO:tasks.workunit.client.0.smithi067.stdout:3/463: dread - d3/d4/d12/f1d zero size 2024-04-22T23:23:28.571 INFO:tasks.workunit.client.0.smithi067.stdout:3/464: symlink d3/d4/la8 0 2024-04-22T23:23:28.572 INFO:tasks.workunit.client.0.smithi067.stdout:3/465: creat d3/d4/d28/d5f/fa9 x:0 0 0 2024-04-22T23:23:28.574 INFO:tasks.workunit.client.0.smithi067.stdout:3/466: creat d3/d4/d1b/faa x:0 0 0 2024-04-22T23:23:28.577 INFO:tasks.workunit.client.0.smithi067.stdout:3/467: link d3/d4/d12/d4b/f94 d3/d4/d28/d4c/d90/da2/fab 0 2024-04-22T23:23:28.579 INFO:tasks.workunit.client.0.smithi067.stdout:3/468: chown d3/c42 962 1 2024-04-22T23:23:28.581 INFO:tasks.workunit.client.0.smithi067.stdout:3/469: getdents d3/d4/d10/d38/d53/d55/d5c/d70 0 2024-04-22T23:23:28.583 INFO:tasks.workunit.client.0.smithi067.stdout:3/470: creat d3/d4/d10/d1f/d3c/fac x:0 0 0 2024-04-22T23:23:28.584 INFO:tasks.workunit.client.0.smithi067.stdout:3/471: stat d3/d4/d10/d1f/d3c 0 2024-04-22T23:23:28.590 INFO:tasks.workunit.client.0.smithi067.stdout:3/472: dwrite d3/d4/d10/d38/d53/d55/f8e [0,4194304] 0 2024-04-22T23:23:28.592 INFO:tasks.workunit.client.0.smithi067.stdout:3/473: write d3/d4/d10/d38/d53/d55/f66 [981624,48828] 0 2024-04-22T23:23:28.594 INFO:tasks.workunit.client.0.smithi067.stdout:7/389: sync 2024-04-22T23:23:28.595 INFO:tasks.workunit.client.0.smithi067.stdout:6/481: sync 2024-04-22T23:23:28.596 INFO:tasks.workunit.client.0.smithi067.stdout:3/474: write d3/d4/fc [4864842,64069] 0 2024-04-22T23:23:28.596 INFO:tasks.workunit.client.0.smithi067.stdout:7/390: fdatasync d10/d14/d1d/f1f 0 2024-04-22T23:23:28.597 INFO:tasks.workunit.client.0.smithi067.stdout:7/391: truncate fb 4316656 0 2024-04-22T23:23:28.599 INFO:tasks.workunit.client.0.smithi067.stdout:7/392: write d10/f12 [1598583,87707] 0 2024-04-22T23:23:28.600 INFO:tasks.workunit.client.0.smithi067.stdout:6/482: creat d17/d39/d3a/f99 x:0 0 0 2024-04-22T23:23:28.603 INFO:tasks.workunit.client.0.smithi067.stdout:7/393: dread d10/f7f [0,4194304] 0 2024-04-22T23:23:28.604 INFO:tasks.workunit.client.0.smithi067.stdout:6/483: rename d17/f32 to d17/f9a 0 2024-04-22T23:23:28.605 INFO:tasks.workunit.client.0.smithi067.stdout:6/484: stat d17/l5d 0 2024-04-22T23:23:28.607 INFO:tasks.workunit.client.0.smithi067.stdout:3/475: dwrite d3/d4/d1b/d93/fa0 [0,4194304] 0 2024-04-22T23:23:28.607 INFO:tasks.workunit.client.0.smithi067.stdout:7/394: dread d10/d18/d35/f5e [0,4194304] 0 2024-04-22T23:23:28.607 INFO:tasks.workunit.client.0.smithi067.stdout:6/485: write d17/f48 [516150,118001] 0 2024-04-22T23:23:28.608 INFO:tasks.workunit.client.0.smithi067.stdout:3/476: readlink d3/d4/d12/l82 0 2024-04-22T23:23:28.612 INFO:tasks.workunit.client.0.smithi067.stdout:7/395: fdatasync d10/d14/d1d/f76 0 2024-04-22T23:23:28.612 INFO:tasks.workunit.client.0.smithi067.stdout:6/486: rmdir d17/d39/d53/d90/d46/d62 39 2024-04-22T23:23:28.613 INFO:tasks.workunit.client.0.smithi067.stdout:6/487: readlink d17/l37 0 2024-04-22T23:23:28.615 INFO:tasks.workunit.client.0.smithi067.stdout:6/488: chown d17/d39/d53/d90/l8f 44742 1 2024-04-22T23:23:28.615 INFO:tasks.workunit.client.0.smithi067.stdout:7/396: unlink d10/f7f 0 2024-04-22T23:23:28.617 INFO:tasks.workunit.client.0.smithi067.stdout:6/489: readlink d17/l8e 0 2024-04-22T23:23:28.618 INFO:tasks.workunit.client.0.smithi067.stdout:3/477: dwrite d3/d4/d28/d5f/f71 [0,4194304] 0 2024-04-22T23:23:28.619 INFO:tasks.workunit.client.0.smithi067.stdout:3/478: readlink d3/d4/l2a 0 2024-04-22T23:23:28.622 INFO:tasks.workunit.client.0.smithi067.stdout:7/397: rename d10/d14/d1d/d46/d50/f5c to d10/d14/d1d/d26/d2c/d7a/f87 0 2024-04-22T23:23:28.624 INFO:tasks.workunit.client.0.smithi067.stdout:3/479: creat d3/d4/d10/d38/d53/d55/fad x:0 0 0 2024-04-22T23:23:28.626 INFO:tasks.workunit.client.0.smithi067.stdout:7/398: mkdir d10/d18/d88 0 2024-04-22T23:23:28.626 INFO:tasks.workunit.client.0.smithi067.stdout:6/490: dwrite d17/d39/d53/d90/f8b [0,4194304] 0 2024-04-22T23:23:28.628 INFO:tasks.workunit.client.0.smithi067.stdout:7/399: symlink d10/d47/l89 0 2024-04-22T23:23:28.632 INFO:tasks.workunit.client.0.smithi067.stdout:6/491: getdents d17/d39/d3a/d8d 0 2024-04-22T23:23:28.636 INFO:tasks.workunit.client.0.smithi067.stdout:6/492: dread d17/d39/d53/f59 [0,4194304] 0 2024-04-22T23:23:28.637 INFO:tasks.workunit.client.0.smithi067.stdout:7/400: dwrite d10/f6f [0,4194304] 0 2024-04-22T23:23:28.641 INFO:tasks.workunit.client.0.smithi067.stdout:7/401: creat d10/d14/d1d/d26/d80/f8a x:0 0 0 2024-04-22T23:23:28.657 INFO:tasks.workunit.client.0.smithi067.stdout:3/480: dread d3/d4/d10/d38/f48 [0,4194304] 0 2024-04-22T23:23:28.664 INFO:tasks.workunit.client.0.smithi067.stdout:3/481: dwrite d3/d4/d12/f83 [0,4194304] 0 2024-04-22T23:23:28.666 INFO:tasks.workunit.client.0.smithi067.stdout:3/482: write d3/d4/d10/d1f/d3c/fac [764427,32558] 0 2024-04-22T23:23:28.670 INFO:tasks.workunit.client.0.smithi067.stdout:3/483: creat d3/d4/d12/fae x:0 0 0 2024-04-22T23:23:28.675 INFO:tasks.workunit.client.0.smithi067.stdout:3/484: mkdir d3/d4/d28/d4c/d90/da2/daf 0 2024-04-22T23:23:28.678 INFO:tasks.workunit.client.0.smithi067.stdout:3/485: unlink d3/d4/l2a 0 2024-04-22T23:23:28.680 INFO:tasks.workunit.client.0.smithi067.stdout:3/486: write d3/d4/d10/d38/d53/f7b [2602716,29624] 0 2024-04-22T23:23:28.687 INFO:tasks.workunit.client.0.smithi067.stdout:3/487: rename d3/d4/l6b to d3/d4/d28/lb0 0 2024-04-22T23:23:28.697 INFO:tasks.workunit.client.0.smithi067.stdout:7/402: dread d10/d14/d1d/d24/f5f [0,4194304] 0 2024-04-22T23:23:28.699 INFO:tasks.workunit.client.0.smithi067.stdout:7/403: creat d10/d47/d49/f8b x:0 0 0 2024-04-22T23:23:28.701 INFO:tasks.workunit.client.0.smithi067.stdout:7/404: mknod d10/d14/d1d/d46/d6a/c8c 0 2024-04-22T23:23:28.701 INFO:tasks.workunit.client.0.smithi067.stdout:7/405: read - d10/d14/d1d/d26/d2c/d41/f7c zero size 2024-04-22T23:23:28.745 INFO:tasks.workunit.client.0.smithi067.stdout:0/516: dwrite db/de/d11/f3c [0,4194304] 0 2024-04-22T23:23:28.749 INFO:tasks.workunit.client.0.smithi067.stdout:9/455: rmdir d5/d21/d47 39 2024-04-22T23:23:28.752 INFO:tasks.workunit.client.0.smithi067.stdout:9/456: write d5/dd/d18/d39/d49/f55 [2916866,105960] 0 2024-04-22T23:23:28.753 INFO:tasks.workunit.client.0.smithi067.stdout:9/457: fsync d5/f10 0 2024-04-22T23:23:28.754 INFO:tasks.workunit.client.0.smithi067.stdout:0/517: dwrite db/d21/d29/d2d/f49 [0,4194304] 0 2024-04-22T23:23:28.755 INFO:tasks.workunit.client.0.smithi067.stdout:0/518: write db/faf [680440,90792] 0 2024-04-22T23:23:28.756 INFO:tasks.workunit.client.0.smithi067.stdout:9/458: read - d5/d21/d47/d7f/f9f zero size 2024-04-22T23:23:28.757 INFO:tasks.workunit.client.0.smithi067.stdout:0/519: write db/d21/d29/d2d/f42 [1761766,47463] 0 2024-04-22T23:23:28.759 INFO:tasks.workunit.client.0.smithi067.stdout:0/520: write db/faa [850383,115858] 0 2024-04-22T23:23:28.768 INFO:tasks.workunit.client.0.smithi067.stdout:2/567: sync 2024-04-22T23:23:28.769 INFO:tasks.workunit.client.0.smithi067.stdout:2/568: write d1/d12/f4e [3600919,74601] 0 2024-04-22T23:23:28.771 INFO:tasks.workunit.client.0.smithi067.stdout:9/459: rename d5/dd/d18/d39/d3e/d3f/f52 to d5/d21/d47/fa4 0 2024-04-22T23:23:28.774 INFO:tasks.workunit.client.0.smithi067.stdout:0/521: mkdir db/d2e/db3 0 2024-04-22T23:23:28.774 INFO:tasks.workunit.client.0.smithi067.stdout:4/544: write d12/d42/f78 [2365841,128543] 0 2024-04-22T23:23:28.775 INFO:tasks.workunit.client.0.smithi067.stdout:4/545: chown d12/d5f/d69 404572 1 2024-04-22T23:23:28.776 INFO:tasks.workunit.client.0.smithi067.stdout:9/460: dread d5/dd/d18/f1d [0,4194304] 0 2024-04-22T23:23:28.778 INFO:tasks.workunit.client.0.smithi067.stdout:0/522: getdents db/d21/d29/d75/d9d 0 2024-04-22T23:23:28.779 INFO:tasks.workunit.client.0.smithi067.stdout:0/523: fsync db/f79 0 2024-04-22T23:23:28.779 INFO:tasks.workunit.client.0.smithi067.stdout:9/461: creat d5/d21/d2c/fa5 x:0 0 0 2024-04-22T23:23:28.780 INFO:tasks.workunit.client.0.smithi067.stdout:0/524: fsync db/de/d11/d1a/da4/f8e 0 2024-04-22T23:23:28.780 INFO:tasks.workunit.client.0.smithi067.stdout:9/462: stat d5/dd/d18/d39/d49/c88 0 2024-04-22T23:23:28.782 INFO:tasks.workunit.client.0.smithi067.stdout:0/525: read db/de/d2a/f39 [1470658,108741] 0 2024-04-22T23:23:28.795 INFO:tasks.workunit.client.0.smithi067.stdout:0/526: dread db/d9a/d56/f6d [0,4194304] 0 2024-04-22T23:23:28.797 INFO:tasks.workunit.client.0.smithi067.stdout:9/463: dread d5/d21/d47/fa4 [0,4194304] 0 2024-04-22T23:23:28.800 INFO:tasks.workunit.client.0.smithi067.stdout:9/464: rmdir d5/dd/d18/d3c/d40 39 2024-04-22T23:23:28.802 INFO:tasks.workunit.client.0.smithi067.stdout:0/527: getdents db/d21/d29 0 2024-04-22T23:23:28.803 INFO:tasks.workunit.client.0.smithi067.stdout:0/528: fdatasync db/d21/f5a 0 2024-04-22T23:23:28.803 INFO:tasks.workunit.client.0.smithi067.stdout:7/406: truncate d10/d14/d1d/d26/f40 325753 0 2024-04-22T23:23:28.805 INFO:tasks.workunit.client.0.smithi067.stdout:9/465: rename d5/dd/d18/d39/d3e/f43 to d5/dd/d18/d39/d49/d96/fa6 0 2024-04-22T23:23:28.807 INFO:tasks.workunit.client.0.smithi067.stdout:0/529: mknod db/de/d11/d92/cb4 0 2024-04-22T23:23:28.810 INFO:tasks.workunit.client.0.smithi067.stdout:7/407: rename d10/d18/d35/l81 to d10/d14/d1d/d26/d2c/d7a/l8d 0 2024-04-22T23:23:28.812 INFO:tasks.workunit.client.0.smithi067.stdout:9/466: dwrite d5/dd/d18/d39/d49/f55 [4194304,4194304] 0 2024-04-22T23:23:28.812 INFO:tasks.workunit.client.0.smithi067.stdout:7/408: fsync d10/d14/d1d/d46/d6a/f86 0 2024-04-22T23:23:28.813 INFO:tasks.workunit.client.0.smithi067.stdout:9/467: dread - d5/d21/d47/d57/f60 zero size 2024-04-22T23:23:28.814 INFO:tasks.workunit.client.0.smithi067.stdout:0/530: rename db/de/d2a/c51 to db/d21/d29/d2d/cb5 0 2024-04-22T23:23:28.817 INFO:tasks.workunit.client.0.smithi067.stdout:7/409: unlink d10/d18/d35/f66 0 2024-04-22T23:23:28.818 INFO:tasks.workunit.client.0.smithi067.stdout:7/410: chown d10/d14/d1d/d26/d2c/d41/d82 6 1 2024-04-22T23:23:28.825 INFO:tasks.workunit.client.0.smithi067.stdout:4/546: dwrite f4 [0,4194304] 0 2024-04-22T23:23:28.826 INFO:tasks.workunit.client.0.smithi067.stdout:4/547: fdatasync d12/d17/d3f/d7a/faf 0 2024-04-22T23:23:28.828 INFO:tasks.workunit.client.0.smithi067.stdout:4/548: read d12/f38 [31027,116889] 0 2024-04-22T23:23:28.833 INFO:tasks.workunit.client.0.smithi067.stdout:4/549: fdatasync d12/d17/d3f/f89 0 2024-04-22T23:23:28.837 INFO:tasks.workunit.client.0.smithi067.stdout:0/531: write db/d21/f8d [1189989,130146] 0 2024-04-22T23:23:28.839 INFO:tasks.workunit.client.0.smithi067.stdout:0/532: mknod db/d21/d29/cb6 0 2024-04-22T23:23:28.844 INFO:tasks.workunit.client.0.smithi067.stdout:7/411: dread d10/d47/f68 [0,4194304] 0 2024-04-22T23:23:28.846 INFO:tasks.workunit.client.0.smithi067.stdout:0/533: dwrite db/d2e/d46/f82 [4194304,4194304] 0 2024-04-22T23:23:28.848 INFO:tasks.workunit.client.0.smithi067.stdout:9/468: dread d5/dd/d18/d3c/d40/f62 [0,4194304] 0 2024-04-22T23:23:28.851 INFO:tasks.workunit.client.0.smithi067.stdout:4/550: dwrite fd [0,4194304] 0 2024-04-22T23:23:28.883 INFO:tasks.workunit.client.0.smithi067.stdout:7/412: symlink d10/d14/d6d/l8e 0 2024-04-22T23:23:28.886 INFO:tasks.workunit.client.0.smithi067.stdout:0/534: creat db/fb7 x:0 0 0 2024-04-22T23:23:28.889 INFO:tasks.workunit.client.0.smithi067.stdout:9/469: dwrite d5/dd/d18/d39/d49/d96/fa6 [0,4194304] 0 2024-04-22T23:23:28.891 INFO:tasks.workunit.client.0.smithi067.stdout:9/470: write d5/dd/d18/d39/d49/d54/f79 [14745,65571] 0 2024-04-22T23:23:28.894 INFO:tasks.workunit.client.0.smithi067.stdout:9/471: dread d5/dd/d18/d39/d49/f7b [0,4194304] 0 2024-04-22T23:23:28.899 INFO:tasks.workunit.client.0.smithi067.stdout:8/488: sync 2024-04-22T23:23:28.900 INFO:tasks.workunit.client.0.smithi067.stdout:8/489: fdatasync d1/d3/d27/f7c 0 2024-04-22T23:23:28.902 INFO:tasks.workunit.client.0.smithi067.stdout:7/413: creat d10/f8f x:0 0 0 2024-04-22T23:23:28.909 INFO:tasks.workunit.client.0.smithi067.stdout:7/414: creat d10/d14/d1d/d26/d2c/d41/f90 x:0 0 0 2024-04-22T23:23:28.909 INFO:tasks.workunit.client.0.smithi067.stdout:0/535: rmdir db/d21/d29 39 2024-04-22T23:23:28.912 INFO:tasks.workunit.client.0.smithi067.stdout:9/472: mkdir d5/dd/d18/da7 0 2024-04-22T23:23:28.915 INFO:tasks.workunit.client.0.smithi067.stdout:4/551: dwrite d12/f8d [0,4194304] 0 2024-04-22T23:23:28.916 INFO:tasks.workunit.client.0.smithi067.stdout:0/536: write db/d2e/f5d [1260002,55498] 0 2024-04-22T23:23:28.917 INFO:tasks.workunit.client.0.smithi067.stdout:8/490: rename d1/d3/l1d to d1/dd/l9c 0 2024-04-22T23:23:28.918 INFO:tasks.workunit.client.0.smithi067.stdout:7/415: dwrite fe [0,4194304] 0 2024-04-22T23:23:28.921 INFO:tasks.workunit.client.0.smithi067.stdout:9/473: creat d5/dd/d18/d39/d3e/d3f/d63/fa8 x:0 0 0 2024-04-22T23:23:28.924 INFO:tasks.workunit.client.0.smithi067.stdout:9/474: write d5/dd/d18/d39/d4d/f7d [768052,122493] 0 2024-04-22T23:23:28.924 INFO:tasks.workunit.client.0.smithi067.stdout:4/552: fsync d12/d17/d31/f50 0 2024-04-22T23:23:28.925 INFO:tasks.workunit.client.0.smithi067.stdout:9/475: chown d5/d21/d2c/c67 31 1 2024-04-22T23:23:28.925 INFO:tasks.workunit.client.0.smithi067.stdout:8/491: unlink d1/d3/d10/d3b/f84 0 2024-04-22T23:23:28.926 INFO:tasks.workunit.client.0.smithi067.stdout:9/476: fsync d5/dd/d18/d3c/f6e 0 2024-04-22T23:23:28.927 INFO:tasks.workunit.client.0.smithi067.stdout:7/416: mkdir d10/d14/d1d/d46/d6a/d91 0 2024-04-22T23:23:28.927 INFO:tasks.workunit.client.0.smithi067.stdout:9/477: read - d5/dd/d18/d39/d4d/f8d zero size 2024-04-22T23:23:28.931 INFO:tasks.workunit.client.0.smithi067.stdout:8/492: mknod d1/d3/d5c/c9d 0 2024-04-22T23:23:28.934 INFO:tasks.workunit.client.0.smithi067.stdout:0/537: creat db/d21/d29/d75/d9d/fb8 x:0 0 0 2024-04-22T23:23:28.935 INFO:tasks.workunit.client.0.smithi067.stdout:8/493: mknod d1/d3/d7e/c9e 0 2024-04-22T23:23:28.936 INFO:tasks.workunit.client.0.smithi067.stdout:9/478: getdents d5/d21/d47/d56/d8c 0 2024-04-22T23:23:28.938 INFO:tasks.workunit.client.0.smithi067.stdout:7/417: rmdir d10/d14/d1d/d7b 0 2024-04-22T23:23:28.938 INFO:tasks.workunit.client.0.smithi067.stdout:7/418: read d10/f6f [2249951,90463] 0 2024-04-22T23:23:28.939 INFO:tasks.workunit.client.0.smithi067.stdout:7/419: dread - d10/d47/d49/f54 zero size 2024-04-22T23:23:28.940 INFO:tasks.workunit.client.0.smithi067.stdout:0/538: mkdir db/d9a/d56/d6b/db9 0 2024-04-22T23:23:28.941 INFO:tasks.workunit.client.0.smithi067.stdout:7/420: chown d10/d14/d1d/d24/l57 142667 1 2024-04-22T23:23:28.945 INFO:tasks.workunit.client.0.smithi067.stdout:0/539: mkdir db/d9a/d56/d70/dba 0 2024-04-22T23:23:28.947 INFO:tasks.workunit.client.0.smithi067.stdout:0/540: readlink db/d21/d29/d2c/l65 0 2024-04-22T23:23:28.947 INFO:tasks.workunit.client.0.smithi067.stdout:4/553: dwrite d12/d42/d4b/d70/f6e [0,4194304] 0 2024-04-22T23:23:28.948 INFO:tasks.workunit.client.0.smithi067.stdout:0/541: fdatasync db/d2e/f8b 0 2024-04-22T23:23:28.952 INFO:tasks.workunit.client.0.smithi067.stdout:4/554: symlink d12/d42/d4b/d70/d8e/lc5 0 2024-04-22T23:23:28.954 INFO:tasks.workunit.client.0.smithi067.stdout:0/542: mknod db/d60/d9e/cbb 0 2024-04-22T23:23:28.957 INFO:tasks.workunit.client.0.smithi067.stdout:7/421: getdents d10/d18 0 2024-04-22T23:23:28.959 INFO:tasks.workunit.client.0.smithi067.stdout:4/555: dread d12/f58 [0,4194304] 0 2024-04-22T23:23:28.960 INFO:tasks.workunit.client.0.smithi067.stdout:7/422: chown d10/d14/d1d/d26/d2c/f56 38 1 2024-04-22T23:23:28.961 INFO:tasks.workunit.client.0.smithi067.stdout:9/479: dwrite d5/dd/d18/d3c/d40/f6b [0,4194304] 0 2024-04-22T23:23:28.963 INFO:tasks.workunit.client.0.smithi067.stdout:4/556: chown d12/l3d 4519811 1 2024-04-22T23:23:28.963 INFO:tasks.workunit.client.0.smithi067.stdout:9/480: write d5/fa [1939359,15756] 0 2024-04-22T23:23:28.965 INFO:tasks.workunit.client.0.smithi067.stdout:7/423: mknod d10/d14/d1d/d26/d2c/d7a/c92 0 2024-04-22T23:23:28.970 INFO:tasks.workunit.client.0.smithi067.stdout:9/481: mkdir d5/dd/da9 0 2024-04-22T23:23:28.970 INFO:tasks.workunit.client.0.smithi067.stdout:4/557: rename d12/d5f/d7e/fc2 to d12/d5f/dae/fc6 0 2024-04-22T23:23:28.971 INFO:tasks.workunit.client.0.smithi067.stdout:0/543: truncate db/d2e/d46/f82 7899467 0 2024-04-22T23:23:28.974 INFO:tasks.workunit.client.0.smithi067.stdout:8/494: dread d1/d3/d10/d3b/f54 [0,4194304] 0 2024-04-22T23:23:28.975 INFO:tasks.workunit.client.0.smithi067.stdout:0/544: creat db/d71/fbc x:0 0 0 2024-04-22T23:23:28.976 INFO:tasks.workunit.client.0.smithi067.stdout:9/482: dwrite d5/dd/d18/d39/d4d/f9e [0,4194304] 0 2024-04-22T23:23:28.979 INFO:tasks.workunit.client.0.smithi067.stdout:8/495: creat d1/d41/d46/f9f x:0 0 0 2024-04-22T23:23:28.980 INFO:tasks.workunit.client.0.smithi067.stdout:9/483: write d5/dd/d18/d39/d3e/d3f/f73 [205665,62249] 0 2024-04-22T23:23:28.981 INFO:tasks.workunit.client.0.smithi067.stdout:4/558: dread d12/d42/d4b/d77/d8c/fa5 [0,4194304] 0 2024-04-22T23:23:28.983 INFO:tasks.workunit.client.0.smithi067.stdout:0/545: creat db/d9a/d56/d70/dba/fbd x:0 0 0 2024-04-22T23:23:28.985 INFO:tasks.workunit.client.0.smithi067.stdout:8/496: fsync d1/d3/f1e 0 2024-04-22T23:23:28.985 INFO:tasks.workunit.client.0.smithi067.stdout:0/546: stat db/d21/lb2 0 2024-04-22T23:23:28.987 INFO:tasks.workunit.client.0.smithi067.stdout:9/484: creat d5/dd/d18/d39/d4d/faa x:0 0 0 2024-04-22T23:23:28.993 INFO:tasks.workunit.client.0.smithi067.stdout:7/424: dread d10/d18/d35/f4e [0,4194304] 0 2024-04-22T23:23:28.993 INFO:tasks.workunit.client.0.smithi067.stdout:0/547: creat db/de/d11/d1a/fbe x:0 0 0 2024-04-22T23:23:28.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:28 smithi134 ceph-mon[34609]: pgmap v269: 65 pgs: 65 active+clean; 950 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 18 MiB/s rd, 112 MiB/s wr, 343 op/s 2024-04-22T23:23:28.997 INFO:tasks.workunit.client.0.smithi067.stdout:9/485: dwrite d5/dd/d18/d39/d49/d96/fa6 [4194304,4194304] 0 2024-04-22T23:23:29.000 INFO:tasks.workunit.client.0.smithi067.stdout:9/486: fdatasync d5/dd/d18/d39/d3e/d3f/f73 0 2024-04-22T23:23:29.001 INFO:tasks.workunit.client.0.smithi067.stdout:7/425: write d10/d14/d1d/d26/d2c/f2e [303895,112589] 0 2024-04-22T23:23:29.003 INFO:tasks.workunit.client.0.smithi067.stdout:4/559: link d12/d42/d4b/d70/d8e/lc5 d12/d5f/d2b/lc7 0 2024-04-22T23:23:29.008 INFO:tasks.workunit.client.0.smithi067.stdout:9/487: creat d5/dd/d18/d5c/fab x:0 0 0 2024-04-22T23:23:29.009 INFO:tasks.workunit.client.0.smithi067.stdout:1/433: sync 2024-04-22T23:23:29.011 INFO:tasks.workunit.client.0.smithi067.stdout:9/488: rmdir d5/d21/d47/d57 39 2024-04-22T23:23:29.015 INFO:tasks.workunit.client.0.smithi067.stdout:4/560: creat d12/d17/d88/fc8 x:0 0 0 2024-04-22T23:23:29.016 INFO:tasks.workunit.client.0.smithi067.stdout:7/426: creat d10/d14/d1d/d26/f93 x:0 0 0 2024-04-22T23:23:29.017 INFO:tasks.workunit.client.0.smithi067.stdout:9/489: symlink d5/dd/lac 0 2024-04-22T23:23:29.019 INFO:tasks.workunit.client.0.smithi067.stdout:4/561: fdatasync d12/d17/d3f/f63 0 2024-04-22T23:23:29.019 INFO:tasks.workunit.client.0.smithi067.stdout:8/497: dwrite d1/d3/d27/f6e [0,4194304] 0 2024-04-22T23:23:29.020 INFO:tasks.workunit.client.0.smithi067.stdout:0/548: dwrite db/de/d2a/f5e [0,4194304] 0 2024-04-22T23:23:29.023 INFO:tasks.workunit.client.0.smithi067.stdout:7/427: creat d10/d18/d35/d62/f94 x:0 0 0 2024-04-22T23:23:29.025 INFO:tasks.workunit.client.0.smithi067.stdout:4/562: fdatasync d12/f2a 0 2024-04-22T23:23:29.026 INFO:tasks.workunit.client.0.smithi067.stdout:8/498: unlink d1/d3/d27/f33 0 2024-04-22T23:23:29.032 INFO:tasks.workunit.client.0.smithi067.stdout:7/428: rename d10/d14/d1d/c84 to d10/d14/d1d/d26/d2c/d7a/c95 0 2024-04-22T23:23:29.033 INFO:tasks.workunit.client.0.smithi067.stdout:9/490: dwrite d5/dd/d18/d39/d4d/f95 [0,4194304] 0 2024-04-22T23:23:29.034 INFO:tasks.workunit.client.0.smithi067.stdout:7/429: truncate d10/d18/d35/f5b 292205 0 2024-04-22T23:23:29.035 INFO:tasks.workunit.client.0.smithi067.stdout:6/493: sync 2024-04-22T23:23:29.035 INFO:tasks.workunit.client.0.smithi067.stdout:2/569: sync 2024-04-22T23:23:29.035 INFO:tasks.workunit.client.0.smithi067.stdout:3/488: sync 2024-04-22T23:23:29.035 INFO:tasks.workunit.client.0.smithi067.stdout:5/552: sync 2024-04-22T23:23:29.036 INFO:tasks.workunit.client.0.smithi067.stdout:5/553: readlink d5/d28/l54 0 2024-04-22T23:23:29.036 INFO:tasks.workunit.client.0.smithi067.stdout:4/563: dread d12/d17/d3f/f63 [0,4194304] 0 2024-04-22T23:23:29.039 INFO:tasks.workunit.client.0.smithi067.stdout:9/491: creat d5/fad x:0 0 0 2024-04-22T23:23:29.040 INFO:tasks.workunit.client.0.smithi067.stdout:7/430: write d10/d47/d49/f54 [319723,80949] 0 2024-04-22T23:23:29.042 INFO:tasks.workunit.client.0.smithi067.stdout:9/492: dread - d5/d21/d47/d56/f80 zero size 2024-04-22T23:23:29.042 INFO:tasks.workunit.client.0.smithi067.stdout:8/499: dread d1/d3/f59 [0,4194304] 0 2024-04-22T23:23:29.044 INFO:tasks.workunit.client.0.smithi067.stdout:4/564: unlink d12/d42/d4b/d77/d9d/fb4 0 2024-04-22T23:23:29.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:28 smithi067 ceph-mon[26453]: pgmap v269: 65 pgs: 65 active+clean; 950 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 18 MiB/s rd, 112 MiB/s wr, 343 op/s 2024-04-22T23:23:29.045 INFO:tasks.workunit.client.0.smithi067.stdout:6/494: chown d17/d39/d53/d90/d46/d62 402 1 2024-04-22T23:23:29.046 INFO:tasks.workunit.client.0.smithi067.stdout:6/495: chown d17/d39/d53/d90 700 1 2024-04-22T23:23:29.047 INFO:tasks.workunit.client.0.smithi067.stdout:2/570: rename d1/f3 to d1/d10/d1c/d6d/d87/fb6 0 2024-04-22T23:23:29.048 INFO:tasks.workunit.client.0.smithi067.stdout:7/431: symlink d10/d14/d1d/d46/l96 0 2024-04-22T23:23:29.049 INFO:tasks.workunit.client.0.smithi067.stdout:7/432: chown d10/d14 136 1 2024-04-22T23:23:29.050 INFO:tasks.workunit.client.0.smithi067.stdout:5/554: dwrite d5/d3a/d4d/f8d [0,4194304] 0 2024-04-22T23:23:29.050 INFO:tasks.workunit.client.0.smithi067.stdout:8/500: unlink d1/dd/l5d 0 2024-04-22T23:23:29.051 INFO:tasks.workunit.client.0.smithi067.stdout:7/433: stat d10/d14/d1d/c6b 0 2024-04-22T23:23:29.051 INFO:tasks.workunit.client.0.smithi067.stdout:5/555: readlink d5/db/d12/l41 0 2024-04-22T23:23:29.052 INFO:tasks.workunit.client.0.smithi067.stdout:7/434: write d10/d18/f27 [220920,46899] 0 2024-04-22T23:23:29.053 INFO:tasks.workunit.client.0.smithi067.stdout:4/565: rename d12/d42/d4b/d77/fba to d12/d5f/d7e/fc9 0 2024-04-22T23:23:29.053 INFO:tasks.workunit.client.0.smithi067.stdout:7/435: write d10/d14/d1d/d26/d2c/f2e [1313930,114246] 0 2024-04-22T23:23:29.054 INFO:tasks.workunit.client.0.smithi067.stdout:6/496: mkdir d17/d83/d9b 0 2024-04-22T23:23:29.056 INFO:tasks.workunit.client.0.smithi067.stdout:6/497: chown d17/d39/d3a/f51 564941 1 2024-04-22T23:23:29.056 INFO:tasks.workunit.client.0.smithi067.stdout:7/436: truncate d10/d14/d1d/d46/d50/f30 1137644 0 2024-04-22T23:23:29.057 INFO:tasks.workunit.client.0.smithi067.stdout:7/437: fdatasync d10/d18/d35/f44 0 2024-04-22T23:23:29.057 INFO:tasks.workunit.client.0.smithi067.stdout:7/438: fdatasync fb 0 2024-04-22T23:23:29.057 INFO:tasks.workunit.client.0.smithi067.stdout:2/571: creat d1/d10/d72/d68/d7b/fb7 x:0 0 0 2024-04-22T23:23:29.059 INFO:tasks.workunit.client.0.smithi067.stdout:8/501: chown d1/d41/l5a 63579626 1 2024-04-22T23:23:29.059 INFO:tasks.workunit.client.0.smithi067.stdout:5/556: symlink d5/db/d12/d44/lc2 0 2024-04-22T23:23:29.061 INFO:tasks.workunit.client.0.smithi067.stdout:4/566: rmdir d12 39 2024-04-22T23:23:29.064 INFO:tasks.workunit.client.0.smithi067.stdout:2/572: creat d1/d42/d61/fb8 x:0 0 0 2024-04-22T23:23:29.066 INFO:tasks.workunit.client.0.smithi067.stdout:7/439: symlink d10/d14/d1d/d26/d2c/d41/l97 0 2024-04-22T23:23:29.067 INFO:tasks.workunit.client.0.smithi067.stdout:8/502: dwrite d1/d3/d27/f6e [4194304,4194304] 0 2024-04-22T23:23:29.068 INFO:tasks.workunit.client.0.smithi067.stdout:2/573: write d1/d10/d1c/f5e [570144,128475] 0 2024-04-22T23:23:29.069 INFO:tasks.workunit.client.0.smithi067.stdout:9/493: getdents d5/dd/d18/d39/d49/d54/d99 0 2024-04-22T23:23:29.075 INFO:tasks.workunit.client.0.smithi067.stdout:7/440: creat d10/d14/f98 x:0 0 0 2024-04-22T23:23:29.076 INFO:tasks.workunit.client.0.smithi067.stdout:6/498: creat d17/f9c x:0 0 0 2024-04-22T23:23:29.077 INFO:tasks.workunit.client.0.smithi067.stdout:4/567: rename d12/d42/d4b/d77/d8c/fa5 to d12/d5f/d2b/d97/fca 0 2024-04-22T23:23:29.079 INFO:tasks.workunit.client.0.smithi067.stdout:5/557: getdents d5/d28 0 2024-04-22T23:23:29.080 INFO:tasks.workunit.client.0.smithi067.stdout:5/558: read - d5/d1a/fa8 zero size 2024-04-22T23:23:29.080 INFO:tasks.workunit.client.0.smithi067.stdout:9/494: dwrite d5/dd/d18/d39/d4d/f9e [4194304,4194304] 0 2024-04-22T23:23:29.082 INFO:tasks.workunit.client.0.smithi067.stdout:8/503: rename d1/dd/d26/d62/d75/c96 to d1/dd/d80/d6f/ca0 0 2024-04-22T23:23:29.083 INFO:tasks.workunit.client.0.smithi067.stdout:4/568: creat d12/d2e/fcb x:0 0 0 2024-04-22T23:23:29.085 INFO:tasks.workunit.client.0.smithi067.stdout:5/559: truncate d5/d28/f69 2042196 0 2024-04-22T23:23:29.086 INFO:tasks.workunit.client.0.smithi067.stdout:8/504: write d1/d3/d10/d3b/f5e [902497,19544] 0 2024-04-22T23:23:29.087 INFO:tasks.workunit.client.0.smithi067.stdout:6/499: rename cd to d17/d83/c9d 0 2024-04-22T23:23:29.088 INFO:tasks.workunit.client.0.smithi067.stdout:5/560: write d5/f11 [2243471,99630] 0 2024-04-22T23:23:29.089 INFO:tasks.workunit.client.0.smithi067.stdout:1/434: write d8/d12/d40/f69 [512792,25210] 0 2024-04-22T23:23:29.090 INFO:tasks.workunit.client.0.smithi067.stdout:4/569: truncate d12/d5f/d69/f7c 806495 0 2024-04-22T23:23:29.093 INFO:tasks.workunit.client.0.smithi067.stdout:6/500: write d17/f42 [4953780,125367] 0 2024-04-22T23:23:29.095 INFO:tasks.workunit.client.0.smithi067.stdout:1/435: readlink d8/d9/d2f/d45/l82 0 2024-04-22T23:23:29.095 INFO:tasks.workunit.client.0.smithi067.stdout:9/495: dwrite d5/d21/d47/d7f/f8e [0,4194304] 0 2024-04-22T23:23:29.098 INFO:tasks.workunit.client.0.smithi067.stdout:5/561: creat d5/d3e/d87/d99/fc3 x:0 0 0 2024-04-22T23:23:29.099 INFO:tasks.workunit.client.0.smithi067.stdout:0/549: truncate db/d21/d29/d75/f83 3634045 0 2024-04-22T23:23:29.100 INFO:tasks.workunit.client.0.smithi067.stdout:5/562: stat d5/db/d12/d44/d47 0 2024-04-22T23:23:29.102 INFO:tasks.workunit.client.0.smithi067.stdout:9/496: dread d5/d21/d47/f72 [0,4194304] 0 2024-04-22T23:23:29.103 INFO:tasks.workunit.client.0.smithi067.stdout:9/497: stat d5/dd/d18/d39/d4d/faa 0 2024-04-22T23:23:29.105 INFO:tasks.workunit.client.0.smithi067.stdout:1/436: creat d8/d9/d2d/d4d/d48/f8d x:0 0 0 2024-04-22T23:23:29.106 INFO:tasks.workunit.client.0.smithi067.stdout:1/437: chown d8/d9/d2d/d4d/d70/l2b 0 1 2024-04-22T23:23:29.108 INFO:tasks.workunit.client.0.smithi067.stdout:1/438: read d8/d9/d2d/d4d/d48/f67 [43057,1853] 0 2024-04-22T23:23:29.112 INFO:tasks.workunit.client.0.smithi067.stdout:5/563: rename d5/d28/l54 to d5/db/d12/d44/d47/lc4 0 2024-04-22T23:23:29.112 INFO:tasks.workunit.client.0.smithi067.stdout:5/564: chown d5/fa 12 1 2024-04-22T23:23:29.113 INFO:tasks.workunit.client.0.smithi067.stdout:5/565: stat d5/db/d12/lab 0 2024-04-22T23:23:29.118 INFO:tasks.workunit.client.0.smithi067.stdout:6/501: truncate d17/f64 330021 0 2024-04-22T23:23:29.119 INFO:tasks.workunit.client.0.smithi067.stdout:6/502: stat d17/f29 0 2024-04-22T23:23:29.119 INFO:tasks.workunit.client.0.smithi067.stdout:8/505: dread d1/d3/f56 [0,4194304] 0 2024-04-22T23:23:29.120 INFO:tasks.workunit.client.0.smithi067.stdout:9/498: write d5/dd/d18/d39/d4d/f8d [850615,8547] 0 2024-04-22T23:23:29.122 INFO:tasks.workunit.client.0.smithi067.stdout:4/570: link d12/d5f/f41 d12/d17/d3f/d7a/fcc 0 2024-04-22T23:23:29.124 INFO:tasks.workunit.client.0.smithi067.stdout:1/439: unlink d8/d9/d2d/d4d/c30 0 2024-04-22T23:23:29.125 INFO:tasks.workunit.client.0.smithi067.stdout:5/566: mknod d5/d3e/db7/db2/cc5 0 2024-04-22T23:23:29.129 INFO:tasks.workunit.client.0.smithi067.stdout:9/499: creat d5/dd/d18/d39/d3e/fae x:0 0 0 2024-04-22T23:23:29.131 INFO:tasks.workunit.client.0.smithi067.stdout:1/440: dwrite d8/d9/d2f/f38 [4194304,4194304] 0 2024-04-22T23:23:29.134 INFO:tasks.workunit.client.0.smithi067.stdout:5/567: mkdir d5/d3a/d4d/dc6 0 2024-04-22T23:23:29.137 INFO:tasks.workunit.client.0.smithi067.stdout:9/500: dwrite d5/dd/f32 [0,4194304] 0 2024-04-22T23:23:29.139 INFO:tasks.workunit.client.0.smithi067.stdout:9/501: fdatasync d5/dd/d18/d39/d3e/d3f/f73 0 2024-04-22T23:23:29.140 INFO:tasks.workunit.client.0.smithi067.stdout:2/574: write d1/d4f/fa3 [2418958,98521] 0 2024-04-22T23:23:29.141 INFO:tasks.workunit.client.0.smithi067.stdout:9/502: write d5/dd/d18/d39/f64 [3017771,106972] 0 2024-04-22T23:23:29.143 INFO:tasks.workunit.client.0.smithi067.stdout:2/575: dread - d1/d42/d5f/f98 zero size 2024-04-22T23:23:29.145 INFO:tasks.workunit.client.0.smithi067.stdout:2/576: dread - d1/d10/d72/d68/d7b/fb7 zero size 2024-04-22T23:23:29.147 INFO:tasks.workunit.client.0.smithi067.stdout:1/441: creat d8/d9/d2f/d31/f8e x:0 0 0 2024-04-22T23:23:29.147 INFO:tasks.workunit.client.0.smithi067.stdout:2/577: write d1/d42/f56 [106606,57607] 0 2024-04-22T23:23:29.148 INFO:tasks.workunit.client.0.smithi067.stdout:6/503: link d17/f33 d17/d39/d53/f9e 0 2024-04-22T23:23:29.150 INFO:tasks.workunit.client.0.smithi067.stdout:5/568: write d5/db/d12/dbb/d61/d89/fb3 [653093,126187] 0 2024-04-22T23:23:29.152 INFO:tasks.workunit.client.0.smithi067.stdout:1/442: fdatasync d8/d9/d2d/d4d/d48/f75 0 2024-04-22T23:23:29.156 INFO:tasks.workunit.client.0.smithi067.stdout:7/441: rmdir d10/d14/d1d/d46/d50 39 2024-04-22T23:23:29.156 INFO:tasks.workunit.client.0.smithi067.stdout:1/443: write d8/d9/d2d/d4d/d70/d34/f37 [2031193,78176] 0 2024-04-22T23:23:29.159 INFO:tasks.workunit.client.0.smithi067.stdout:9/503: truncate d5/d21/d47/d57/f60 570568 0 2024-04-22T23:23:29.159 INFO:tasks.workunit.client.0.smithi067.stdout:4/571: rename d12/d17/d3f/d7a/cbb to d12/d17/d3f/ccd 0 2024-04-22T23:23:29.159 INFO:tasks.workunit.client.0.smithi067.stdout:5/569: dwrite d5/d3e/db7/f8f [0,4194304] 0 2024-04-22T23:23:29.160 INFO:tasks.workunit.client.0.smithi067.stdout:4/572: chown d12/d42/d4b/d77/d9d 231 1 2024-04-22T23:23:29.161 INFO:tasks.workunit.client.0.smithi067.stdout:9/504: stat d5/d21/d47/d7f/f8e 0 2024-04-22T23:23:29.168 INFO:tasks.workunit.client.0.smithi067.stdout:7/442: dwrite d10/d14/d1d/d26/d2c/d41/f4f [0,4194304] 0 2024-04-22T23:23:29.174 INFO:tasks.workunit.client.0.smithi067.stdout:9/505: dwrite d5/d21/d47/d57/f85 [0,4194304] 0 2024-04-22T23:23:29.179 INFO:tasks.workunit.client.0.smithi067.stdout:4/573: symlink d12/lce 0 2024-04-22T23:23:29.181 INFO:tasks.workunit.client.0.smithi067.stdout:6/504: creat d17/d39/d53/d90/d46/f9f x:0 0 0 2024-04-22T23:23:29.182 INFO:tasks.workunit.client.0.smithi067.stdout:6/505: fsync d17/d39/d53/d90/f6f 0 2024-04-22T23:23:29.182 INFO:tasks.workunit.client.0.smithi067.stdout:7/443: creat d10/d14/d1d/d26/d2c/d41/f99 x:0 0 0 2024-04-22T23:23:29.185 INFO:tasks.workunit.client.0.smithi067.stdout:1/444: link d8/f4b d8/d9/d2f/d31/f8f 0 2024-04-22T23:23:29.187 INFO:tasks.workunit.client.0.smithi067.stdout:4/574: dread - d12/d42/d4b/d70/d8e/d56/f5c zero size 2024-04-22T23:23:29.191 INFO:tasks.workunit.client.0.smithi067.stdout:7/444: creat d10/d18/d35/d62/f9a x:0 0 0 2024-04-22T23:23:29.192 INFO:tasks.workunit.client.0.smithi067.stdout:8/506: dwrite d1/dd/f2f [0,4194304] 0 2024-04-22T23:23:29.195 INFO:tasks.workunit.client.0.smithi067.stdout:1/445: mknod d8/d9/d2d/d4d/d70/d34/d5d/c90 0 2024-04-22T23:23:29.198 INFO:tasks.workunit.client.0.smithi067.stdout:9/506: creat d5/d21/faf x:0 0 0 2024-04-22T23:23:29.200 INFO:tasks.workunit.client.0.smithi067.stdout:9/507: write d5/d21/d47/d57/f60 [1344192,54047] 0 2024-04-22T23:23:29.200 INFO:tasks.workunit.client.0.smithi067.stdout:4/575: creat d12/d42/d4b/d70/d8e/fcf x:0 0 0 2024-04-22T23:23:29.201 INFO:tasks.workunit.client.0.smithi067.stdout:7/445: creat d10/d14/d1d/d26/d2c/d7a/f9b x:0 0 0 2024-04-22T23:23:29.208 INFO:tasks.workunit.client.0.smithi067.stdout:6/506: creat d17/fa0 x:0 0 0 2024-04-22T23:23:29.210 INFO:tasks.workunit.client.0.smithi067.stdout:6/507: write d17/d83/f86 [6676159,5607] 0 2024-04-22T23:23:29.212 INFO:tasks.workunit.client.0.smithi067.stdout:9/508: dwrite d5/dd/d18/d3c/f6e [0,4194304] 0 2024-04-22T23:23:29.212 INFO:tasks.workunit.client.0.smithi067.stdout:4/576: dread d12/d5f/d69/f81 [0,4194304] 0 2024-04-22T23:23:29.221 INFO:tasks.workunit.client.0.smithi067.stdout:9/509: chown d5/d21/d47/fa4 26822354 1 2024-04-22T23:23:29.223 INFO:tasks.workunit.client.0.smithi067.stdout:7/446: dread d10/f6f [0,4194304] 0 2024-04-22T23:23:29.227 INFO:tasks.workunit.client.0.smithi067.stdout:7/447: stat d10/d18/d35/d62/f9a 0 2024-04-22T23:23:29.228 INFO:tasks.workunit.client.0.smithi067.stdout:8/507: dwrite d1/d3/d10/f3d [0,4194304] 0 2024-04-22T23:23:29.230 INFO:tasks.workunit.client.0.smithi067.stdout:1/446: dwrite d8/d9/d2f/d33/f41 [0,4194304] 0 2024-04-22T23:23:29.230 INFO:tasks.workunit.client.0.smithi067.stdout:4/577: mkdir d12/d5f/dae/dd0 0 2024-04-22T23:23:29.233 INFO:tasks.workunit.client.0.smithi067.stdout:6/508: creat d17/d39/d53/d90/d46/fa1 x:0 0 0 2024-04-22T23:23:29.233 INFO:tasks.workunit.client.0.smithi067.stdout:8/508: chown d1/d3/d27/f82 0 1 2024-04-22T23:23:29.239 INFO:tasks.workunit.client.0.smithi067.stdout:9/510: dwrite d5/dd/d18/d39/d3e/fae [0,4194304] 0 2024-04-22T23:23:29.241 INFO:tasks.workunit.client.0.smithi067.stdout:1/447: rename d8/d9/d2f/d59/f6e to d8/d9/d2f/d33/d66/f91 0 2024-04-22T23:23:29.243 INFO:tasks.workunit.client.0.smithi067.stdout:9/511: chown d5/dd/d18/d3c/f6e 121 1 2024-04-22T23:23:29.244 INFO:tasks.workunit.client.0.smithi067.stdout:4/578: write d12/d5f/d2b/db6/f9e [5156288,109605] 0 2024-04-22T23:23:29.244 INFO:tasks.workunit.client.0.smithi067.stdout:9/512: chown d5 2676 1 2024-04-22T23:23:29.249 INFO:tasks.workunit.client.0.smithi067.stdout:8/509: dwrite d1/dd/f79 [0,4194304] 0 2024-04-22T23:23:29.251 INFO:tasks.workunit.client.0.smithi067.stdout:1/448: unlink d8/d9/d2f/d31/f8f 0 2024-04-22T23:23:29.252 INFO:tasks.workunit.client.0.smithi067.stdout:1/449: fdatasync d8/d9/d2f/f79 0 2024-04-22T23:23:29.255 INFO:tasks.workunit.client.0.smithi067.stdout:7/448: write d10/d14/d1d/d26/d2c/d41/f65 [470129,34809] 0 2024-04-22T23:23:29.255 INFO:tasks.workunit.client.0.smithi067.stdout:8/510: write d1/d3/f56 [54000,39705] 0 2024-04-22T23:23:29.261 INFO:tasks.workunit.client.0.smithi067.stdout:1/450: creat d8/d9/d2d/d4d/d70/d34/d60/f92 x:0 0 0 2024-04-22T23:23:29.261 INFO:tasks.workunit.client.0.smithi067.stdout:7/449: creat d10/d47/d49/f9c x:0 0 0 2024-04-22T23:23:29.262 INFO:tasks.workunit.client.0.smithi067.stdout:1/451: dread - d8/d12/d36/f4e zero size 2024-04-22T23:23:29.263 INFO:tasks.workunit.client.0.smithi067.stdout:8/511: dread d1/dd/f53 [0,4194304] 0 2024-04-22T23:23:29.266 INFO:tasks.workunit.client.0.smithi067.stdout:7/450: creat d10/d14/d1d/d26/d80/f9d x:0 0 0 2024-04-22T23:23:29.266 INFO:tasks.workunit.client.0.smithi067.stdout:8/512: dread - d1/d3/d5c/f99 zero size 2024-04-22T23:23:29.268 INFO:tasks.workunit.client.0.smithi067.stdout:7/451: chown d10/c3e 2279 1 2024-04-22T23:23:29.269 INFO:tasks.workunit.client.0.smithi067.stdout:1/452: symlink d8/d9/d2d/d4d/d70/l93 0 2024-04-22T23:23:29.270 INFO:tasks.workunit.client.0.smithi067.stdout:4/579: dread f5 [0,4194304] 0 2024-04-22T23:23:29.271 INFO:tasks.workunit.client.0.smithi067.stdout:6/509: dread fe [0,4194304] 0 2024-04-22T23:23:29.272 INFO:tasks.workunit.client.0.smithi067.stdout:8/513: rename d1/d3/d10/d3b/f54 to d1/d3/d5c/d98/fa1 0 2024-04-22T23:23:29.273 INFO:tasks.workunit.client.0.smithi067.stdout:9/513: truncate d5/dd/d18/d39/d49/d54/f79 3350354 0 2024-04-22T23:23:29.303 INFO:tasks.workunit.client.0.smithi067.stdout:3/489: sync 2024-04-22T23:23:29.304 INFO:tasks.workunit.client.0.smithi067.stdout:1/453: creat d8/d12/f94 x:0 0 0 2024-04-22T23:23:29.304 INFO:tasks.workunit.client.0.smithi067.stdout:9/514: fsync d5/dd/d18/d5c/fab 0 2024-04-22T23:23:29.305 INFO:tasks.workunit.client.0.smithi067.stdout:7/452: mkdir d10/d14/d6d/d9e 0 2024-04-22T23:23:29.307 INFO:tasks.workunit.client.0.smithi067.stdout:3/490: stat d3/d4/d12/f2f 0 2024-04-22T23:23:29.307 INFO:tasks.workunit.client.0.smithi067.stdout:8/514: creat d1/d41/d46/fa2 x:0 0 0 2024-04-22T23:23:29.308 INFO:tasks.workunit.client.0.smithi067.stdout:4/580: dwrite d12/f2a [0,4194304] 0 2024-04-22T23:23:29.309 INFO:tasks.workunit.client.0.smithi067.stdout:3/491: chown d3/d4/d28/l77 39435 1 2024-04-22T23:23:29.312 INFO:tasks.workunit.client.0.smithi067.stdout:3/492: read d3/d4/f79 [3575092,22425] 0 2024-04-22T23:23:29.312 INFO:tasks.workunit.client.0.smithi067.stdout:4/581: truncate f5 8371028 0 2024-04-22T23:23:29.321 INFO:tasks.workunit.client.0.smithi067.stdout:4/582: dwrite d12/d17/d47/f49 [8388608,4194304] 0 2024-04-22T23:23:29.322 INFO:tasks.workunit.client.0.smithi067.stdout:4/583: readlink d12/d5f/d69/la3 0 2024-04-22T23:23:29.323 INFO:tasks.workunit.client.0.smithi067.stdout:6/510: truncate f12 2164654 0 2024-04-22T23:23:29.326 INFO:tasks.workunit.client.0.smithi067.stdout:7/453: rename d10/d14/d1d/d26/d80/f9d to d10/d14/d1d/d26/d2c/d41/f9f 0 2024-04-22T23:23:29.327 INFO:tasks.workunit.client.0.smithi067.stdout:8/515: creat d1/dd/d26/d62/d94/fa3 x:0 0 0 2024-04-22T23:23:29.328 INFO:tasks.workunit.client.0.smithi067.stdout:4/584: symlink d12/d5f/ld1 0 2024-04-22T23:23:29.329 INFO:tasks.workunit.client.0.smithi067.stdout:9/515: getdents d5/dd/d18/d39/d49/d96 0 2024-04-22T23:23:29.331 INFO:tasks.workunit.client.0.smithi067.stdout:6/511: dwrite d17/d39/d3a/f50 [0,4194304] 0 2024-04-22T23:23:29.331 INFO:tasks.workunit.client.0.smithi067.stdout:1/454: rename d8/d9/d2f/f5e to d8/d9/d2d/d4d/d70/d34/d60/d62/f95 0 2024-04-22T23:23:29.332 INFO:tasks.workunit.client.0.smithi067.stdout:8/516: readlink d1/dd/d26/d62/d75/l7b 0 2024-04-22T23:23:29.336 INFO:tasks.workunit.client.0.smithi067.stdout:9/516: creat d5/dd/d18/d5c/fb0 x:0 0 0 2024-04-22T23:23:29.336 INFO:tasks.workunit.client.0.smithi067.stdout:3/493: rename d3/d4 to d3/d4/d10/db1 22 2024-04-22T23:23:29.339 INFO:tasks.workunit.client.0.smithi067.stdout:6/512: creat d17/d39/d53/d90/d46/fa2 x:0 0 0 2024-04-22T23:23:29.341 INFO:tasks.workunit.client.0.smithi067.stdout:8/517: truncate d1/d41/d46/f61 22252 0 2024-04-22T23:23:29.342 INFO:tasks.workunit.client.0.smithi067.stdout:3/494: mkdir d3/d4/d28/d4c/db2 0 2024-04-22T23:23:29.343 INFO:tasks.workunit.client.0.smithi067.stdout:9/517: creat d5/dd/d18/d39/d8f/fb1 x:0 0 0 2024-04-22T23:23:29.343 INFO:tasks.workunit.client.0.smithi067.stdout:8/518: stat d1/d3/d27/f6e 0 2024-04-22T23:23:29.343 INFO:tasks.workunit.client.0.smithi067.stdout:1/455: dwrite d8/d9/f29 [0,4194304] 0 2024-04-22T23:23:29.346 INFO:tasks.workunit.client.0.smithi067.stdout:6/513: link d17/d39/d53/f55 d17/d83/d9b/fa3 0 2024-04-22T23:23:29.349 INFO:tasks.workunit.client.0.smithi067.stdout:8/519: read - d1/d41/d46/f9f zero size 2024-04-22T23:23:29.350 INFO:tasks.workunit.client.0.smithi067.stdout:9/518: write d5/dd/d18/d39/d49/d54/d99/fa1 [4950656,33072] 0 2024-04-22T23:23:29.352 INFO:tasks.workunit.client.0.smithi067.stdout:3/495: rename d3/d4/d10/d19/c5a to d3/d4/d10/d1f/d3c/cb3 0 2024-04-22T23:23:29.353 INFO:tasks.workunit.client.0.smithi067.stdout:6/514: creat d17/d39/d3a/d8d/fa4 x:0 0 0 2024-04-22T23:23:29.354 INFO:tasks.workunit.client.0.smithi067.stdout:3/496: fdatasync d3/d4/d12/f78 0 2024-04-22T23:23:29.355 INFO:tasks.workunit.client.0.smithi067.stdout:8/520: truncate d1/f76 216724 0 2024-04-22T23:23:29.361 INFO:tasks.workunit.client.0.smithi067.stdout:6/515: dread d17/d39/d53/f59 [0,4194304] 0 2024-04-22T23:23:29.361 INFO:tasks.workunit.client.0.smithi067.stdout:3/497: creat d3/d4/d28/d4c/d90/da2/fb4 x:0 0 0 2024-04-22T23:23:29.361 INFO:tasks.workunit.client.0.smithi067.stdout:4/585: dread d12/d42/d4b/fbf [0,4194304] 0 2024-04-22T23:23:29.363 INFO:tasks.workunit.client.0.smithi067.stdout:9/519: dwrite d5/dd/d18/d3c/d40/f50 [0,4194304] 0 2024-04-22T23:23:29.366 INFO:tasks.workunit.client.0.smithi067.stdout:9/520: chown d5/dd/d18/d3c/d40/f6b 16652 1 2024-04-22T23:23:29.368 INFO:tasks.workunit.client.0.smithi067.stdout:8/521: link d1/d3/d10/d3b/f5e d1/d3/d10/d3b/fa4 0 2024-04-22T23:23:29.369 INFO:tasks.workunit.client.0.smithi067.stdout:4/586: unlink d12/d5f/d69/f7c 0 2024-04-22T23:23:29.370 INFO:tasks.workunit.client.0.smithi067.stdout:4/587: truncate d12/d5f/f35 4991216 0 2024-04-22T23:23:29.370 INFO:tasks.workunit.client.0.smithi067.stdout:4/588: readlink d12/d17/l4f 0 2024-04-22T23:23:29.371 INFO:tasks.workunit.client.0.smithi067.stdout:6/516: unlink d17/f64 0 2024-04-22T23:23:29.371 INFO:tasks.workunit.client.0.smithi067.stdout:4/589: fsync d12/f8d 0 2024-04-22T23:23:29.378 INFO:tasks.workunit.client.0.smithi067.stdout:1/456: dread d8/d9/d2d/d4d/d48/f7f [0,4194304] 0 2024-04-22T23:23:29.379 INFO:tasks.workunit.client.0.smithi067.stdout:6/517: creat d17/d83/d9b/fa5 x:0 0 0 2024-04-22T23:23:29.380 INFO:tasks.workunit.client.0.smithi067.stdout:6/518: readlink d17/l44 0 2024-04-22T23:23:29.380 INFO:tasks.workunit.client.0.smithi067.stdout:4/590: dwrite d12/f52 [0,4194304] 0 2024-04-22T23:23:29.382 INFO:tasks.workunit.client.0.smithi067.stdout:3/498: link d3/d4/d10/d1f/d3c/c86 d3/d4/d28/d5f/cb5 0 2024-04-22T23:23:29.382 INFO:tasks.workunit.client.0.smithi067.stdout:3/499: chown d3/d4/d28/d4a 14180658 1 2024-04-22T23:23:29.388 INFO:tasks.workunit.client.0.smithi067.stdout:1/457: creat d8/d9/d2f/d45/f96 x:0 0 0 2024-04-22T23:23:29.391 INFO:tasks.workunit.client.0.smithi067.stdout:8/522: link d1/dd/d26/d62/f6d d1/dd/d26/d62/fa5 0 2024-04-22T23:23:29.395 INFO:tasks.workunit.client.0.smithi067.stdout:6/519: dread d17/f33 [0,4194304] 0 2024-04-22T23:23:29.396 INFO:tasks.workunit.client.0.smithi067.stdout:1/458: symlink d8/d9/d2d/d4d/d48/l97 0 2024-04-22T23:23:29.398 INFO:tasks.workunit.client.0.smithi067.stdout:3/500: creat d3/d4/d28/d4c/d8c/fb6 x:0 0 0 2024-04-22T23:23:29.398 INFO:tasks.workunit.client.0.smithi067.stdout:1/459: readlink d8/d9/d2d/d4d/d70/l2b 0 2024-04-22T23:23:29.400 INFO:tasks.workunit.client.0.smithi067.stdout:8/523: dwrite d1/d3/f13 [4194304,4194304] 0 2024-04-22T23:23:29.403 INFO:tasks.workunit.client.0.smithi067.stdout:4/591: getdents d12/d5f/d2b/d97 0 2024-04-22T23:23:29.405 INFO:tasks.workunit.client.0.smithi067.stdout:3/501: read d3/d4/f79 [1136878,81525] 0 2024-04-22T23:23:29.407 INFO:tasks.workunit.client.0.smithi067.stdout:1/460: creat d8/d9/d2f/d33/d5c/f98 x:0 0 0 2024-04-22T23:23:29.407 INFO:tasks.workunit.client.0.smithi067.stdout:3/502: fdatasync d3/d4/d10/d38/d53/d55/f75 0 2024-04-22T23:23:29.408 INFO:tasks.workunit.client.0.smithi067.stdout:3/503: write d3/d4/d12/f6e [539361,45932] 0 2024-04-22T23:23:29.409 INFO:tasks.workunit.client.0.smithi067.stdout:4/592: creat d12/d42/d4b/d70/d8e/fd2 x:0 0 0 2024-04-22T23:23:29.411 INFO:tasks.workunit.client.0.smithi067.stdout:4/593: chown d12/d2e/fb7 37877 1 2024-04-22T23:23:29.411 INFO:tasks.workunit.client.0.smithi067.stdout:3/504: read d3/d4/d10/d38/d53/d55/f75 [1240753,90202] 0 2024-04-22T23:23:29.412 INFO:tasks.workunit.client.0.smithi067.stdout:1/461: rename d8/d12/d36/l7a to d8/d12/d36/l99 0 2024-04-22T23:23:29.413 INFO:tasks.workunit.client.0.smithi067.stdout:6/520: dread d17/f35 [0,4194304] 0 2024-04-22T23:23:29.415 INFO:tasks.workunit.client.0.smithi067.stdout:4/594: mkdir d12/d42/dd3 0 2024-04-22T23:23:29.416 INFO:tasks.workunit.client.0.smithi067.stdout:1/462: read f1 [1579423,90776] 0 2024-04-22T23:23:29.417 INFO:tasks.workunit.client.0.smithi067.stdout:3/505: symlink d3/d4/d28/d4c/d90/da2/lb7 0 2024-04-22T23:23:29.421 INFO:tasks.workunit.client.0.smithi067.stdout:4/595: creat d12/d5f/d7e/fd4 x:0 0 0 2024-04-22T23:23:29.422 INFO:tasks.workunit.client.0.smithi067.stdout:1/463: mkdir d8/d9/d2d/d4d/d48/d9a 0 2024-04-22T23:23:29.422 INFO:tasks.workunit.client.0.smithi067.stdout:4/596: write d12/d42/d4b/d70/d8e/d56/d8b/fa2 [674818,79440] 0 2024-04-22T23:23:29.424 INFO:tasks.workunit.client.0.smithi067.stdout:3/506: rename d3/d4/d10/l85 to d3/d4/d12/lb8 0 2024-04-22T23:23:29.428 INFO:tasks.workunit.client.0.smithi067.stdout:4/597: creat d12/d42/d4b/fd5 x:0 0 0 2024-04-22T23:23:29.429 INFO:tasks.workunit.client.0.smithi067.stdout:1/464: dwrite d8/d9/f29 [0,4194304] 0 2024-04-22T23:23:29.430 INFO:tasks.workunit.client.0.smithi067.stdout:3/507: creat d3/d4/d28/d4a/fb9 x:0 0 0 2024-04-22T23:23:29.435 INFO:tasks.workunit.client.0.smithi067.stdout:4/598: creat d12/d42/d4b/d70/d8e/d56/d8b/dc3/fd6 x:0 0 0 2024-04-22T23:23:29.438 INFO:tasks.workunit.client.0.smithi067.stdout:1/465: rename d8/d9/d2d/d4d/d48/f67 to d8/d9/d2f/d33/d66/f9b 0 2024-04-22T23:23:29.444 INFO:tasks.workunit.client.0.smithi067.stdout:1/466: rename d8/d9/d2f/d45 to d8/d9/d2d/d4d/d9c 0 2024-04-22T23:23:29.444 INFO:tasks.workunit.client.0.smithi067.stdout:1/467: stat d8/d9/d2d/d4d/d70/d34/d60/l77 0 2024-04-22T23:23:29.450 INFO:tasks.workunit.client.0.smithi067.stdout:6/521: write d17/d39/d53/f55 [1503291,118294] 0 2024-04-22T23:23:29.455 INFO:tasks.workunit.client.0.smithi067.stdout:6/522: creat d17/d39/d53/d90/d46/fa6 x:0 0 0 2024-04-22T23:23:29.458 INFO:tasks.workunit.client.0.smithi067.stdout:6/523: creat d17/d39/fa7 x:0 0 0 2024-04-22T23:23:29.458 INFO:tasks.workunit.client.0.smithi067.stdout:3/508: dread d3/d4/d10/d38/d53/f7b [4194304,4194304] 0 2024-04-22T23:23:29.461 INFO:tasks.workunit.client.0.smithi067.stdout:3/509: fdatasync d3/d4/d10/d38/d53/f7b 0 2024-04-22T23:23:29.465 INFO:tasks.workunit.client.0.smithi067.stdout:3/510: creat d3/d4/d12/d4b/fba x:0 0 0 2024-04-22T23:23:29.468 INFO:tasks.workunit.client.0.smithi067.stdout:3/511: stat d3/d4/d1b/d93 0 2024-04-22T23:23:29.469 INFO:tasks.workunit.client.0.smithi067.stdout:6/524: dwrite d17/d39/d53/d90/d46/d62/d88/f92 [0,4194304] 0 2024-04-22T23:23:29.470 INFO:tasks.workunit.client.0.smithi067.stdout:4/599: dwrite d12/d42/d4b/d70/d8e/f68 [0,4194304] 0 2024-04-22T23:23:29.471 INFO:tasks.workunit.client.0.smithi067.stdout:1/468: dwrite d8/d9/d2d/d4d/d70/d34/f39 [0,4194304] 0 2024-04-22T23:23:29.472 INFO:tasks.workunit.client.0.smithi067.stdout:3/512: unlink d3/l9d 0 2024-04-22T23:23:29.473 INFO:tasks.workunit.client.0.smithi067.stdout:0/550: sync 2024-04-22T23:23:29.477 INFO:tasks.workunit.client.0.smithi067.stdout:6/525: mknod d17/d39/d3a/d8d/ca8 0 2024-04-22T23:23:29.479 INFO:tasks.workunit.client.0.smithi067.stdout:4/600: dread d12/f16 [4194304,4194304] 0 2024-04-22T23:23:29.482 INFO:tasks.workunit.client.0.smithi067.stdout:4/601: fsync d12/d42/d4b/d70/d8e/d56/d8b/fa7 0 2024-04-22T23:23:29.484 INFO:tasks.workunit.client.0.smithi067.stdout:3/513: dread d3/d4/d12/f27 [0,4194304] 0 2024-04-22T23:23:29.484 INFO:tasks.workunit.client.0.smithi067.stdout:0/551: mkdir db/d9a/d56/d70/dbf 0 2024-04-22T23:23:29.486 INFO:tasks.workunit.client.0.smithi067.stdout:0/552: truncate db/de/d2a/f5e 4346006 0 2024-04-22T23:23:29.486 INFO:tasks.workunit.client.0.smithi067.stdout:3/514: truncate d3/d4/d28/d4a/fb9 468618 0 2024-04-22T23:23:29.487 INFO:tasks.workunit.client.0.smithi067.stdout:4/602: creat d12/d42/d4b/d70/d8e/d56/d8b/fd7 x:0 0 0 2024-04-22T23:23:29.487 INFO:tasks.workunit.client.0.smithi067.stdout:3/515: write d3/d4/d10/d38/d53/d55/f75 [1707969,52779] 0 2024-04-22T23:23:29.489 INFO:tasks.workunit.client.0.smithi067.stdout:3/516: write d3/d4/d28/d4c/d90/f9f [220772,69677] 0 2024-04-22T23:23:29.490 INFO:tasks.workunit.client.0.smithi067.stdout:0/553: rename db/d9a/f3a to db/d9a/d56/d70/dba/fc0 0 2024-04-22T23:23:29.491 INFO:tasks.workunit.client.0.smithi067.stdout:3/517: chown d3/d4/d28/d4c/d90/da2/lb7 337247 1 2024-04-22T23:23:29.491 INFO:tasks.workunit.client.0.smithi067.stdout:0/554: readlink db/d21/lb2 0 2024-04-22T23:23:29.492 INFO:tasks.workunit.client.0.smithi067.stdout:4/603: mkdir d12/d42/dd3/dd8 0 2024-04-22T23:23:29.494 INFO:tasks.workunit.client.0.smithi067.stdout:0/555: mknod db/de/cc1 0 2024-04-22T23:23:29.495 INFO:tasks.workunit.client.0.smithi067.stdout:4/604: unlink d12/d17/d3f/l7d 0 2024-04-22T23:23:29.496 INFO:tasks.workunit.client.0.smithi067.stdout:0/556: chown db/d9a/c5c 2 1 2024-04-22T23:23:29.498 INFO:tasks.workunit.client.0.smithi067.stdout:6/526: rmdir d17/d39/d3a 39 2024-04-22T23:23:29.502 INFO:tasks.workunit.client.0.smithi067.stdout:6/527: dread - d17/d39/d53/d90/d46/f9f zero size 2024-04-22T23:23:29.502 INFO:tasks.workunit.client.0.smithi067.stdout:3/518: dwrite d3/d4/d12/f78 [0,4194304] 0 2024-04-22T23:23:29.508 INFO:tasks.workunit.client.0.smithi067.stdout:6/528: mkdir d17/d39/d3a/da9 0 2024-04-22T23:23:29.513 INFO:tasks.workunit.client.0.smithi067.stdout:6/529: fsync d17/d39/f69 0 2024-04-22T23:23:29.513 INFO:tasks.workunit.client.0.smithi067.stdout:1/469: dwrite d8/d9/f1b [0,4194304] 0 2024-04-22T23:23:29.514 INFO:tasks.workunit.client.0.smithi067.stdout:0/557: dwrite db/d21/f7b [0,4194304] 0 2024-04-22T23:23:29.514 INFO:tasks.workunit.client.0.smithi067.stdout:3/519: mknod d3/d4/d10/d38/d53/d55/d74/d7f/cbb 0 2024-04-22T23:23:29.515 INFO:tasks.workunit.client.0.smithi067.stdout:4/605: dwrite d12/d17/d88/fc8 [0,4194304] 0 2024-04-22T23:23:29.516 INFO:tasks.workunit.client.0.smithi067.stdout:1/470: stat d8/d9/f42 0 2024-04-22T23:23:29.517 INFO:tasks.workunit.client.0.smithi067.stdout:3/520: fsync d3/d4/d28/d4c/d90/da2/fb4 0 2024-04-22T23:23:29.518 INFO:tasks.workunit.client.0.smithi067.stdout:1/471: write d8/d9/f29 [3352089,40524] 0 2024-04-22T23:23:29.520 INFO:tasks.workunit.client.0.smithi067.stdout:3/521: stat d3/d4/d28/d5f/f71 0 2024-04-22T23:23:29.522 INFO:tasks.workunit.client.0.smithi067.stdout:3/522: stat d3/c50 0 2024-04-22T23:23:29.523 INFO:tasks.workunit.client.0.smithi067.stdout:1/472: readlink d8/d9/d2d/d4d/d70/d34/d60/l77 0 2024-04-22T23:23:29.523 INFO:tasks.workunit.client.0.smithi067.stdout:6/530: symlink d17/d39/d3a/da9/laa 0 2024-04-22T23:23:29.525 INFO:tasks.workunit.client.0.smithi067.stdout:6/531: chown d17/d39/d3a/da9/laa 1777 1 2024-04-22T23:23:29.525 INFO:tasks.workunit.client.0.smithi067.stdout:4/606: read d12/d5f/f55 [3208646,8816] 0 2024-04-22T23:23:29.528 INFO:tasks.workunit.client.0.smithi067.stdout:0/558: dread db/d21/d29/d2c/f72 [0,4194304] 0 2024-04-22T23:23:29.530 INFO:tasks.workunit.client.0.smithi067.stdout:1/473: rename d8/d9/d2d/d4d/d70/d34/l6a to d8/d9/d2d/d4d/d70/d34/d60/d62/l9d 0 2024-04-22T23:23:29.531 INFO:tasks.workunit.client.0.smithi067.stdout:1/474: chown d8/d4a/l80 3463065 1 2024-04-22T23:23:29.532 INFO:tasks.workunit.client.0.smithi067.stdout:3/523: truncate d3/d4/d12/d4b/f6c 2410372 0 2024-04-22T23:23:29.532 INFO:tasks.workunit.client.0.smithi067.stdout:1/475: write d8/d9/f3b [1264070,78046] 0 2024-04-22T23:23:29.533 INFO:tasks.workunit.client.0.smithi067.stdout:3/524: fsync d3/d4/d28/d4a/fb9 0 2024-04-22T23:23:29.535 INFO:tasks.workunit.client.0.smithi067.stdout:3/525: readlink d3/d4/d10/d1f/l49 0 2024-04-22T23:23:29.535 INFO:tasks.workunit.client.0.smithi067.stdout:3/526: chown d3/d4/d10/d1f/l49 0 1 2024-04-22T23:23:29.536 INFO:tasks.workunit.client.0.smithi067.stdout:3/527: fsync d3/d4/d12/d4b/f4e 0 2024-04-22T23:23:29.536 INFO:tasks.workunit.client.0.smithi067.stdout:0/559: unlink db/d2e/d46/la9 0 2024-04-22T23:23:29.540 INFO:tasks.workunit.client.0.smithi067.stdout:6/532: rename d17/d39/d53/d90/d46/d62/c65 to d17/d39/cab 0 2024-04-22T23:23:29.542 INFO:tasks.workunit.client.0.smithi067.stdout:3/528: creat d3/d4/d10/fbc x:0 0 0 2024-04-22T23:23:29.546 INFO:tasks.workunit.client.0.smithi067.stdout:3/529: rmdir d3/d4/d28/d4c/db2 0 2024-04-22T23:23:29.547 INFO:tasks.workunit.client.0.smithi067.stdout:3/530: write d3/d4/d10/d19/f5d [500764,68897] 0 2024-04-22T23:23:29.549 INFO:tasks.workunit.client.0.smithi067.stdout:1/476: getdents d8/d9/d2d/d4d/d70/d34/d60/d62 0 2024-04-22T23:23:29.551 INFO:tasks.workunit.client.0.smithi067.stdout:4/607: write d12/f46 [517450,101758] 0 2024-04-22T23:23:29.554 INFO:tasks.workunit.client.0.smithi067.stdout:3/531: rename d3/c8 to d3/d4/d10/d38/d53/d55/d74/cbd 0 2024-04-22T23:23:29.556 INFO:tasks.workunit.client.0.smithi067.stdout:6/533: getdents d17/d39/d53/d90/d46/d62 0 2024-04-22T23:23:29.560 INFO:tasks.workunit.client.0.smithi067.stdout:1/477: rename d8/d9/d2d/d4d/d70/d34/f39 to d8/d9/d2d/d4d/f9e 0 2024-04-22T23:23:29.562 INFO:tasks.workunit.client.0.smithi067.stdout:6/534: rmdir d17/d39/d3a/da9 39 2024-04-22T23:23:29.565 INFO:tasks.workunit.client.0.smithi067.stdout:1/478: rename d8/d9/d2d/d4d/d48/c85 to d8/d9/d2f/d33/c9f 0 2024-04-22T23:23:29.567 INFO:tasks.workunit.client.0.smithi067.stdout:3/532: truncate d3/d4/d12/d4b/f63 622158 0 2024-04-22T23:23:29.568 INFO:tasks.workunit.client.0.smithi067.stdout:6/535: symlink d17/d39/d53/d90/d46/d62/lac 0 2024-04-22T23:23:29.570 INFO:tasks.workunit.client.0.smithi067.stdout:6/536: write d17/f25 [1140609,15647] 0 2024-04-22T23:23:29.573 INFO:tasks.workunit.client.0.smithi067.stdout:1/479: dwrite d8/d9/d2d/d4d/f1f [0,4194304] 0 2024-04-22T23:23:29.575 INFO:tasks.workunit.client.0.smithi067.stdout:1/480: read d8/d9/f25 [3678853,91674] 0 2024-04-22T23:23:29.577 INFO:tasks.workunit.client.0.smithi067.stdout:3/533: rename d3/d4/d10/c2d to d3/d4/d28/d4c/d8c/cbe 0 2024-04-22T23:23:29.584 INFO:tasks.workunit.client.0.smithi067.stdout:3/534: mkdir d3/d4/d10/d38/dbf 0 2024-04-22T23:23:29.589 INFO:tasks.workunit.client.0.smithi067.stdout:6/537: dwrite d17/d39/d53/f9e [0,4194304] 0 2024-04-22T23:23:29.590 INFO:tasks.workunit.client.0.smithi067.stdout:1/481: dwrite d8/d9/f46 [0,4194304] 0 2024-04-22T23:23:29.592 INFO:tasks.workunit.client.0.smithi067.stdout:1/482: readlink d8/d9/d2d/d4d/d70/l93 0 2024-04-22T23:23:29.592 INFO:tasks.workunit.client.0.smithi067.stdout:3/535: mkdir d3/d4/d28/dc0 0 2024-04-22T23:23:29.596 INFO:tasks.workunit.client.0.smithi067.stdout:6/538: creat d17/d83/fad x:0 0 0 2024-04-22T23:23:29.601 INFO:tasks.workunit.client.0.smithi067.stdout:3/536: creat d3/fc1 x:0 0 0 2024-04-22T23:23:29.602 INFO:tasks.workunit.client.0.smithi067.stdout:6/539: dread d17/f2b [0,4194304] 0 2024-04-22T23:23:29.602 INFO:tasks.workunit.client.0.smithi067.stdout:3/537: readlink d3/d4/l15 0 2024-04-22T23:23:29.604 INFO:tasks.workunit.client.0.smithi067.stdout:1/483: dwrite d8/d9/d2f/d33/d5c/f6d [0,4194304] 0 2024-04-22T23:23:29.608 INFO:tasks.workunit.client.0.smithi067.stdout:1/484: read d8/d9/d2d/d4d/d9c/f8a [791063,84177] 0 2024-04-22T23:23:29.609 INFO:tasks.workunit.client.0.smithi067.stdout:6/540: mkdir d17/dae 0 2024-04-22T23:23:29.611 INFO:tasks.workunit.client.0.smithi067.stdout:3/538: rename d3/d4/d28/d4c/d90/da2/lb7 to d3/d4/d10/d38/dbf/lc2 0 2024-04-22T23:23:29.613 INFO:tasks.workunit.client.0.smithi067.stdout:3/539: write d3/d4/d12/f2f [3421907,7275] 0 2024-04-22T23:23:29.694 INFO:tasks.workunit.client.0.smithi067.stdout:3/540: dread d3/d4/d28/d5f/f71 [0,4194304] 0 2024-04-22T23:23:29.695 INFO:tasks.workunit.client.0.smithi067.stdout:3/541: write d3/d4/d10/d38/fa1 [302492,119253] 0 2024-04-22T23:23:29.696 INFO:tasks.workunit.client.0.smithi067.stdout:1/485: symlink d8/d9/d2d/d4d/d70/la0 0 2024-04-22T23:23:29.699 INFO:tasks.workunit.client.0.smithi067.stdout:6/541: getdents d17/d39/d53/d90/d46/d62 0 2024-04-22T23:23:29.700 INFO:tasks.workunit.client.0.smithi067.stdout:6/542: fdatasync d17/d39/f3f 0 2024-04-22T23:23:29.701 INFO:tasks.workunit.client.0.smithi067.stdout:1/486: creat d8/d9/d2d/d4d/d70/d34/d60/fa1 x:0 0 0 2024-04-22T23:23:29.703 INFO:tasks.workunit.client.0.smithi067.stdout:3/542: dwrite d3/d4/d12/f78 [0,4194304] 0 2024-04-22T23:23:29.704 INFO:tasks.workunit.client.0.smithi067.stdout:6/543: creat d17/d39/d53/d90/d46/d62/faf x:0 0 0 2024-04-22T23:23:29.706 INFO:tasks.workunit.client.0.smithi067.stdout:1/487: unlink d8/d9/d2d/d4d/d48/f55 0 2024-04-22T23:23:29.708 INFO:tasks.workunit.client.0.smithi067.stdout:1/488: truncate d8/d9/d2d/d4d/d48/f8d 151562 0 2024-04-22T23:23:29.710 INFO:tasks.workunit.client.0.smithi067.stdout:3/543: mknod d3/d4/d10/cc3 0 2024-04-22T23:23:29.712 INFO:tasks.workunit.client.0.smithi067.stdout:6/544: mkdir d17/d39/d53/d90/d46/db0 0 2024-04-22T23:23:29.713 INFO:tasks.workunit.client.0.smithi067.stdout:1/489: mknod d8/d9/d2d/d4d/d70/d34/ca2 0 2024-04-22T23:23:29.717 INFO:tasks.workunit.client.0.smithi067.stdout:1/490: rmdir d8/d9/d2d/d4d/d70/d34/d60/d62 39 2024-04-22T23:23:29.719 INFO:tasks.workunit.client.0.smithi067.stdout:3/544: getdents d3/d4/d28/d40 0 2024-04-22T23:23:29.721 INFO:tasks.workunit.client.0.smithi067.stdout:6/545: rename d17/d39/cab to d17/d39/d53/d90/cb1 0 2024-04-22T23:23:29.723 INFO:tasks.workunit.client.0.smithi067.stdout:1/491: mknod d8/d9/d2d/d4d/d70/ca3 0 2024-04-22T23:23:29.724 INFO:tasks.workunit.client.0.smithi067.stdout:6/546: mknod d17/d39/d53/cb2 0 2024-04-22T23:23:29.725 INFO:tasks.workunit.client.0.smithi067.stdout:1/492: creat d8/d12/d40/fa4 x:0 0 0 2024-04-22T23:23:29.726 INFO:tasks.workunit.client.0.smithi067.stdout:3/545: rename d3/d4/d10/f1a to d3/d4/fc4 0 2024-04-22T23:23:29.730 INFO:tasks.workunit.client.0.smithi067.stdout:3/546: creat d3/d4/d28/d4c/d8c/fc5 x:0 0 0 2024-04-22T23:23:29.730 INFO:tasks.workunit.client.0.smithi067.stdout:6/547: dwrite d17/d39/f3f [4194304,4194304] 0 2024-04-22T23:23:29.732 INFO:tasks.workunit.client.0.smithi067.stdout:3/547: dread - d3/d4/d10/d38/d53/d55/d5c/f87 zero size 2024-04-22T23:23:29.734 INFO:tasks.workunit.client.0.smithi067.stdout:1/493: dwrite d8/d9/d2d/d4d/d48/f7f [0,4194304] 0 2024-04-22T23:23:29.735 INFO:tasks.workunit.client.0.smithi067.stdout:3/548: chown d3/d4/d10/d38/d53/d55/d5c/d70 43 1 2024-04-22T23:23:29.736 INFO:tasks.workunit.client.0.smithi067.stdout:1/494: chown d8/d9/d2f/f38 35 1 2024-04-22T23:23:29.737 INFO:tasks.workunit.client.0.smithi067.stdout:3/549: stat d3/d4/d12/d4b 0 2024-04-22T23:23:29.738 INFO:tasks.workunit.client.0.smithi067.stdout:6/548: dwrite d17/f21 [0,4194304] 0 2024-04-22T23:23:29.739 INFO:tasks.workunit.client.0.smithi067.stdout:3/550: write d3/d4/d10/d38/d53/f7b [8632228,38332] 0 2024-04-22T23:23:29.742 INFO:tasks.workunit.client.0.smithi067.stdout:6/549: dread d17/d39/f4b [0,4194304] 0 2024-04-22T23:23:29.742 INFO:tasks.workunit.client.0.smithi067.stdout:3/551: chown d3/d4/d10/d38/d53/d55/f75 855 1 2024-04-22T23:23:29.743 INFO:tasks.workunit.client.0.smithi067.stdout:1/495: creat d8/d9/d2d/d4d/d70/d34/d60/d62/fa5 x:0 0 0 2024-04-22T23:23:29.746 INFO:tasks.workunit.client.0.smithi067.stdout:3/552: symlink d3/d4/d10/d1f/d3c/lc6 0 2024-04-22T23:23:29.750 INFO:tasks.workunit.client.0.smithi067.stdout:3/553: link d3/d4/d10/d1f/d3c/l31 d3/d4/d28/d5f/lc7 0 2024-04-22T23:23:29.774 INFO:tasks.workunit.client.0.smithi067.stdout:6/550: symlink d17/d39/d53/d90/d46/d62/d88/lb3 0 2024-04-22T23:23:29.775 INFO:tasks.workunit.client.0.smithi067.stdout:6/551: write d17/d39/d3a/f6d [125165,96910] 0 2024-04-22T23:23:29.784 INFO:tasks.workunit.client.0.smithi067.stdout:3/554: write d3/d4/fc4 [5193025,75759] 0 2024-04-22T23:23:29.785 INFO:tasks.workunit.client.0.smithi067.stdout:3/555: fsync d3/d4/d10/d38/d53/d55/d5c/d70/f9b 0 2024-04-22T23:23:29.788 INFO:tasks.workunit.client.0.smithi067.stdout:3/556: mkdir d3/d4/d10/d8a/dc8 0 2024-04-22T23:23:29.790 INFO:tasks.workunit.client.0.smithi067.stdout:3/557: creat d3/d4/d10/d38/d53/d55/d74/fc9 x:0 0 0 2024-04-22T23:23:29.790 INFO:tasks.workunit.client.0.smithi067.stdout:6/552: dread d17/d83/f86 [0,4194304] 0 2024-04-22T23:23:29.792 INFO:tasks.workunit.client.0.smithi067.stdout:6/553: mknod d17/d39/d3a/d8d/cb4 0 2024-04-22T23:23:29.793 INFO:tasks.workunit.client.0.smithi067.stdout:3/558: mknod d3/d4/d10/d1f/d3c/cca 0 2024-04-22T23:23:29.793 INFO:tasks.workunit.client.0.smithi067.stdout:3/559: stat d3/d4/d12/l29 0 2024-04-22T23:23:29.796 INFO:tasks.workunit.client.0.smithi067.stdout:3/560: mkdir d3/dcb 0 2024-04-22T23:23:29.797 INFO:tasks.workunit.client.0.smithi067.stdout:6/554: dwrite d17/f24 [0,4194304] 0 2024-04-22T23:23:29.799 INFO:tasks.workunit.client.0.smithi067.stdout:3/561: creat d3/d4/d28/dc0/fcc x:0 0 0 2024-04-22T23:23:29.800 INFO:tasks.workunit.client.0.smithi067.stdout:6/555: mkdir d17/d83/db5 0 2024-04-22T23:23:29.801 INFO:tasks.workunit.client.0.smithi067.stdout:3/562: rmdir d3/d4/d28 39 2024-04-22T23:23:29.802 INFO:tasks.workunit.client.0.smithi067.stdout:6/556: chown d17/d39/d3a/da9 0 1 2024-04-22T23:23:29.818 INFO:tasks.workunit.client.0.smithi067.stdout:6/557: dwrite d17/f26 [0,4194304] 0 2024-04-22T23:23:29.819 INFO:tasks.workunit.client.0.smithi067.stdout:3/563: dwrite d3/d4/d10/d38/d53/f54 [0,4194304] 0 2024-04-22T23:23:29.824 INFO:tasks.workunit.client.0.smithi067.stdout:3/564: mknod d3/d4/d10/d1f/ccd 0 2024-04-22T23:23:29.826 INFO:tasks.workunit.client.0.smithi067.stdout:3/565: chown d3/d4/d10/cc3 203 1 2024-04-22T23:23:29.827 INFO:tasks.workunit.client.0.smithi067.stdout:3/566: stat d3/d4/d12/f69 0 2024-04-22T23:23:29.829 INFO:tasks.workunit.client.0.smithi067.stdout:6/558: dwrite d17/d39/d53/d90/d46/f68 [0,4194304] 0 2024-04-22T23:23:29.830 INFO:tasks.workunit.client.0.smithi067.stdout:3/567: creat d3/dcb/fce x:0 0 0 2024-04-22T23:23:29.831 INFO:tasks.workunit.client.0.smithi067.stdout:6/559: write d17/d39/d3a/d8d/f97 [353614,75087] 0 2024-04-22T23:23:29.832 INFO:tasks.workunit.client.0.smithi067.stdout:6/560: dread - d17/f7d zero size 2024-04-22T23:23:29.834 INFO:tasks.workunit.client.0.smithi067.stdout:6/561: truncate d17/d83/d9b/fa5 850381 0 2024-04-22T23:23:29.838 INFO:tasks.workunit.client.0.smithi067.stdout:5/570: sync 2024-04-22T23:23:29.839 INFO:tasks.workunit.client.0.smithi067.stdout:5/571: fsync d5/db/d12/d44/d47/f6d 0 2024-04-22T23:23:29.839 INFO:tasks.workunit.client.0.smithi067.stdout:6/562: link d17/d39/d53/f55 d17/d39/d3a/fb6 0 2024-04-22T23:23:29.841 INFO:tasks.workunit.client.0.smithi067.stdout:5/572: write d5/d3e/d90/f94 [300539,79593] 0 2024-04-22T23:23:29.842 INFO:tasks.workunit.client.0.smithi067.stdout:5/573: fsync d5/d28/daf/db4/fb6 0 2024-04-22T23:23:29.844 INFO:tasks.workunit.client.0.smithi067.stdout:5/574: creat d5/d3e/db7/fc7 x:0 0 0 2024-04-22T23:23:29.845 INFO:tasks.workunit.client.0.smithi067.stdout:2/578: sync 2024-04-22T23:23:29.846 INFO:tasks.workunit.client.0.smithi067.stdout:5/575: mkdir d5/d28/d36/d40/d97/dc8 0 2024-04-22T23:23:29.847 INFO:tasks.workunit.client.0.smithi067.stdout:2/579: fsync d1/f69 0 2024-04-22T23:23:29.847 INFO:tasks.workunit.client.0.smithi067.stdout:5/576: write d5/db/d12/dbb/d61/d89/f92 [670576,39736] 0 2024-04-22T23:23:29.848 INFO:tasks.workunit.client.0.smithi067.stdout:5/577: dread - d5/d28/d36/f86 zero size 2024-04-22T23:23:29.848 INFO:tasks.workunit.client.0.smithi067.stdout:2/580: fdatasync d1/d10/d72/d53/d9a/fb1 0 2024-04-22T23:23:29.850 INFO:tasks.workunit.client.0.smithi067.stdout:5/578: symlink d5/d3e/d87/d99/lc9 0 2024-04-22T23:23:29.852 INFO:tasks.workunit.client.0.smithi067.stdout:5/579: truncate d5/db/d12/f2c 1131491 0 2024-04-22T23:23:29.853 INFO:tasks.workunit.client.0.smithi067.stdout:5/580: chown d5/d28/d36/f86 86860 1 2024-04-22T23:23:29.854 INFO:tasks.workunit.client.0.smithi067.stdout:3/568: truncate d3/d4/d12/f78 3641532 0 2024-04-22T23:23:29.855 INFO:tasks.workunit.client.0.smithi067.stdout:2/581: dread d1/d10/d1c/f40 [0,4194304] 0 2024-04-22T23:23:29.859 INFO:tasks.workunit.client.0.smithi067.stdout:2/582: mkdir d1/d42/d61/db9 0 2024-04-22T23:23:29.860 INFO:tasks.workunit.client.0.smithi067.stdout:5/581: link l2 d5/d28/daf/db4/lca 0 2024-04-22T23:23:29.862 INFO:tasks.workunit.client.0.smithi067.stdout:6/563: dwrite d17/d39/d3a/f4f [0,4194304] 0 2024-04-22T23:23:29.863 INFO:tasks.workunit.client.0.smithi067.stdout:2/583: write d1/f2d [2708,20318] 0 2024-04-22T23:23:29.864 INFO:tasks.workunit.client.0.smithi067.stdout:3/569: dwrite d3/d4/d10/d38/d53/d55/d74/fc9 [0,4194304] 0 2024-04-22T23:23:29.865 INFO:tasks.workunit.client.0.smithi067.stdout:2/584: truncate d1/d4f/f2a 2577803 0 2024-04-22T23:23:29.869 INFO:tasks.workunit.client.0.smithi067.stdout:2/585: mkdir d1/d10/d72/d53/dba 0 2024-04-22T23:23:29.873 INFO:tasks.workunit.client.0.smithi067.stdout:3/570: mkdir d3/d4/d10/d38/d53/dcf 0 2024-04-22T23:23:29.874 INFO:tasks.workunit.client.0.smithi067.stdout:5/582: link d5/db/d12/f13 d5/d3e/db7/fcb 0 2024-04-22T23:23:29.877 INFO:tasks.workunit.client.0.smithi067.stdout:2/586: dwrite d1/d42/d61/f7e [0,4194304] 0 2024-04-22T23:23:29.878 INFO:tasks.workunit.client.0.smithi067.stdout:5/583: truncate d5/db/d12/dbb/d61/d89/fb5 4724837 0 2024-04-22T23:23:29.881 INFO:tasks.workunit.client.0.smithi067.stdout:2/587: creat d1/d10/d1c/d6d/d87/fbb x:0 0 0 2024-04-22T23:23:29.882 INFO:tasks.workunit.client.0.smithi067.stdout:5/584: symlink d5/d3e/db7/lcc 0 2024-04-22T23:23:29.883 INFO:tasks.workunit.client.0.smithi067.stdout:2/588: symlink d1/d42/d6b/lbc 0 2024-04-22T23:23:29.885 INFO:tasks.workunit.client.0.smithi067.stdout:3/571: creat d3/d4/d28/fd0 x:0 0 0 2024-04-22T23:23:29.885 INFO:tasks.workunit.client.0.smithi067.stdout:5/585: rmdir d5/db/d12/d44 39 2024-04-22T23:23:29.887 INFO:tasks.workunit.client.0.smithi067.stdout:2/589: creat d1/d42/d5f/fbd x:0 0 0 2024-04-22T23:23:29.892 INFO:tasks.workunit.client.0.smithi067.stdout:2/590: creat d1/d10/d72/d68/fbe x:0 0 0 2024-04-22T23:23:29.893 INFO:tasks.workunit.client.0.smithi067.stdout:3/572: dwrite d3/f9a [0,4194304] 0 2024-04-22T23:23:29.895 INFO:tasks.workunit.client.0.smithi067.stdout:3/573: dread d3/f99 [0,4194304] 0 2024-04-22T23:23:29.900 INFO:tasks.workunit.client.0.smithi067.stdout:3/574: mknod d3/d4/cd1 0 2024-04-22T23:23:29.900 INFO:tasks.workunit.client.0.smithi067.stdout:2/591: dwrite d1/d10/d33/d46/f88 [0,4194304] 0 2024-04-22T23:23:29.905 INFO:tasks.workunit.client.0.smithi067.stdout:3/575: mknod d3/cd2 0 2024-04-22T23:23:29.906 INFO:tasks.workunit.client.0.smithi067.stdout:3/576: truncate d3/d4/d10/d1f/d3c/fac 1066566 0 2024-04-22T23:23:29.909 INFO:tasks.workunit.client.0.smithi067.stdout:3/577: unlink d3/d4/d28/d4c/d90/da2/fab 0 2024-04-22T23:23:29.911 INFO:tasks.workunit.client.0.smithi067.stdout:3/578: rmdir d3/d4/d10/d8a/dc8 0 2024-04-22T23:23:29.913 INFO:tasks.workunit.client.0.smithi067.stdout:3/579: mkdir d3/d4/d28/d4c/d90/da2/dd3 0 2024-04-22T23:23:29.914 INFO:tasks.workunit.client.0.smithi067.stdout:3/580: dread d3/d4/d12/f27 [0,4194304] 0 2024-04-22T23:23:29.921 INFO:tasks.workunit.client.0.smithi067.stdout:2/592: rmdir d1/d10/d1c/d6d 39 2024-04-22T23:23:29.921 INFO:tasks.workunit.client.0.smithi067.stdout:3/581: dwrite d3/d4/d12/f6e [0,4194304] 0 2024-04-22T23:23:29.929 INFO:tasks.workunit.client.0.smithi067.stdout:5/586: dwrite d5/db/f23 [0,4194304] 0 2024-04-22T23:23:29.929 INFO:tasks.workunit.client.0.smithi067.stdout:3/582: link d3/d4/d10/d1f/d3c/l61 d3/d4/d10/d38/d53/ld4 0 2024-04-22T23:23:29.930 INFO:tasks.workunit.client.0.smithi067.stdout:2/593: dwrite d1/d10/d72/d53/f58 [0,4194304] 0 2024-04-22T23:23:29.934 INFO:tasks.workunit.client.0.smithi067.stdout:5/587: creat d5/d28/d36/d40/d97/fcd x:0 0 0 2024-04-22T23:23:29.936 INFO:tasks.workunit.client.0.smithi067.stdout:2/594: mkdir d1/dbf 0 2024-04-22T23:23:29.937 INFO:tasks.workunit.client.0.smithi067.stdout:5/588: creat d5/d28/d36/d40/d97/dc8/fce x:0 0 0 2024-04-22T23:23:29.938 INFO:tasks.workunit.client.0.smithi067.stdout:5/589: dread - d5/f31 zero size 2024-04-22T23:23:29.939 INFO:tasks.workunit.client.0.smithi067.stdout:2/595: getdents d1/dbf 0 2024-04-22T23:23:29.940 INFO:tasks.workunit.client.0.smithi067.stdout:5/590: rmdir d5/db/d35 39 2024-04-22T23:23:29.986 INFO:tasks.workunit.client.0.smithi067.stdout:7/454: sync 2024-04-22T23:23:29.987 INFO:tasks.workunit.client.0.smithi067.stdout:7/455: fdatasync d10/d18/d35/f5e 0 2024-04-22T23:23:29.989 INFO:tasks.workunit.client.0.smithi067.stdout:7/456: symlink d10/d47/la0 0 2024-04-22T23:23:30.062 INFO:tasks.workunit.client.0.smithi067.stdout:3/583: dread d3/d4/d28/d4a/f73 [0,4194304] 0 2024-04-22T23:23:30.065 INFO:tasks.workunit.client.0.smithi067.stdout:3/584: readlink d3/d4/d10/d38/d53/ld4 0 2024-04-22T23:23:30.068 INFO:tasks.workunit.client.0.smithi067.stdout:3/585: mkdir d3/d4/d10/d38/d53/d55/d74/dd5 0 2024-04-22T23:23:30.072 INFO:tasks.workunit.client.0.smithi067.stdout:2/596: dread d1/d10/d1c/d6d/f89 [0,4194304] 0 2024-04-22T23:23:30.073 INFO:tasks.workunit.client.0.smithi067.stdout:2/597: write d1/d4f/f2a [467378,16446] 0 2024-04-22T23:23:30.077 INFO:tasks.workunit.client.0.smithi067.stdout:2/598: getdents d1/d19 0 2024-04-22T23:23:30.079 INFO:tasks.workunit.client.0.smithi067.stdout:2/599: mkdir d1/d42/d5f/d9b/dc0 0 2024-04-22T23:23:30.084 INFO:tasks.workunit.client.0.smithi067.stdout:4/608: sync 2024-04-22T23:23:30.084 INFO:tasks.workunit.client.0.smithi067.stdout:1/496: sync 2024-04-22T23:23:30.084 INFO:tasks.workunit.client.0.smithi067.stdout:8/524: sync 2024-04-22T23:23:30.084 INFO:tasks.workunit.client.0.smithi067.stdout:0/560: sync 2024-04-22T23:23:30.084 INFO:tasks.workunit.client.0.smithi067.stdout:9/521: sync 2024-04-22T23:23:30.086 INFO:tasks.workunit.client.0.smithi067.stdout:2/600: link d1/d10/d72/d68/d7b/d95/fac d1/fc1 0 2024-04-22T23:23:30.092 INFO:tasks.workunit.client.0.smithi067.stdout:9/522: symlink d5/dd/lb2 0 2024-04-22T23:23:30.094 INFO:tasks.workunit.client.0.smithi067.stdout:4/609: dwrite d12/d5f/dae/fc6 [0,4194304] 0 2024-04-22T23:23:30.096 INFO:tasks.workunit.client.0.smithi067.stdout:4/610: truncate d12/d17/fb9 156053 0 2024-04-22T23:23:30.103 INFO:tasks.workunit.client.0.smithi067.stdout:0/561: dread db/d21/d29/d75/f86 [0,4194304] 0 2024-04-22T23:23:30.105 INFO:tasks.workunit.client.0.smithi067.stdout:9/523: dwrite d5/dd/d18/d39/d3e/f53 [0,4194304] 0 2024-04-22T23:23:30.108 INFO:tasks.workunit.client.0.smithi067.stdout:9/524: dread - d5/d21/faf zero size 2024-04-22T23:23:30.113 INFO:tasks.workunit.client.0.smithi067.stdout:3/586: dwrite d3/f6f [0,4194304] 0 2024-04-22T23:23:30.113 INFO:tasks.workunit.client.0.smithi067.stdout:0/562: mkdir db/d9a/d56/d70/d8c/dc2 0 2024-04-22T23:23:30.116 INFO:tasks.workunit.client.0.smithi067.stdout:4/611: creat d12/fd9 x:0 0 0 2024-04-22T23:23:30.119 INFO:tasks.workunit.client.0.smithi067.stdout:2/601: truncate d1/d10/f2e 3950527 0 2024-04-22T23:23:30.119 INFO:tasks.workunit.client.0.smithi067.stdout:0/563: chown db/de/d11/d1a/c37 2099404 1 2024-04-22T23:23:30.119 INFO:tasks.workunit.client.0.smithi067.stdout:3/587: write d3/d4/d10/d38/d53/d55/f8e [4332573,90581] 0 2024-04-22T23:23:30.123 INFO:tasks.workunit.client.0.smithi067.stdout:9/525: truncate d5/dd/d18/d39/d49/f7b 124860 0 2024-04-22T23:23:30.124 INFO:tasks.workunit.client.0.smithi067.stdout:4/612: symlink d12/d5f/dae/lda 0 2024-04-22T23:23:30.126 INFO:tasks.workunit.client.0.smithi067.stdout:3/588: fsync d3/d4/d12/fae 0 2024-04-22T23:23:30.128 INFO:tasks.workunit.client.0.smithi067.stdout:8/525: dwrite d1/dd/f53 [0,4194304] 0 2024-04-22T23:23:30.134 INFO:tasks.workunit.client.0.smithi067.stdout:7/457: dwrite d10/d14/d1d/d26/f2f [0,4194304] 0 2024-04-22T23:23:30.134 INFO:tasks.workunit.client.0.smithi067.stdout:9/526: write d5/d21/d47/fa4 [5737357,1303] 0 2024-04-22T23:23:30.156 INFO:tasks.workunit.client.0.smithi067.stdout:5/591: sync 2024-04-22T23:23:30.156 INFO:tasks.workunit.client.0.smithi067.stdout:6/564: sync 2024-04-22T23:23:30.179 INFO:tasks.workunit.client.0.smithi067.stdout:3/589: mkdir d3/d4/d28/d4c/d62/d95/dd6 0 2024-04-22T23:23:30.180 INFO:tasks.workunit.client.0.smithi067.stdout:5/592: read - d5/f68 zero size 2024-04-22T23:23:30.181 INFO:tasks.workunit.client.0.smithi067.stdout:9/527: write d5/d21/d2c/f7a [124376,72895] 0 2024-04-22T23:23:30.182 INFO:tasks.workunit.client.0.smithi067.stdout:6/565: chown d17/l52 1139 1 2024-04-22T23:23:30.184 INFO:tasks.workunit.client.0.smithi067.stdout:3/590: write d3/d4/d1b/d93/fa0 [4545726,735] 0 2024-04-22T23:23:30.186 INFO:tasks.workunit.client.0.smithi067.stdout:2/602: link d1/d10/d1c/d6d/c67 d1/d10/d1c/d6d/cc2 0 2024-04-22T23:23:30.187 INFO:tasks.workunit.client.0.smithi067.stdout:6/566: mknod d17/d39/d3a/d8d/cb7 0 2024-04-22T23:23:30.187 INFO:tasks.workunit.client.0.smithi067.stdout:7/458: creat d10/d14/d1d/d26/d2c/d41/fa1 x:0 0 0 2024-04-22T23:23:30.188 INFO:tasks.workunit.client.0.smithi067.stdout:8/526: truncate d1/d3/d10/f29 910000 0 2024-04-22T23:23:30.188 INFO:tasks.workunit.client.0.smithi067.stdout:9/528: symlink d5/dd/d18/d3c/d40/lb3 0 2024-04-22T23:23:30.189 INFO:tasks.workunit.client.0.smithi067.stdout:6/567: fsync d17/f24 0 2024-04-22T23:23:30.191 INFO:tasks.workunit.client.0.smithi067.stdout:0/564: dread db/d21/f5a [0,4194304] 0 2024-04-22T23:23:30.192 INFO:tasks.workunit.client.0.smithi067.stdout:6/568: dread - d17/d39/fa7 zero size 2024-04-22T23:23:30.192 INFO:tasks.workunit.client.0.smithi067.stdout:8/527: chown d1/d3/d10/f3d 9420 1 2024-04-22T23:23:30.193 INFO:tasks.workunit.client.0.smithi067.stdout:9/529: fdatasync d5/dd/d18/d39/d3e/d3f/f7e 0 2024-04-22T23:23:30.196 INFO:tasks.workunit.client.0.smithi067.stdout:0/565: stat db/d9a/d56/d70/c8f 0 2024-04-22T23:23:30.196 INFO:tasks.workunit.client.0.smithi067.stdout:5/593: dwrite d5/d3a/d4d/fb1 [0,4194304] 0 2024-04-22T23:23:30.197 INFO:tasks.workunit.client.0.smithi067.stdout:7/459: creat d10/d18/d88/fa2 x:0 0 0 2024-04-22T23:23:30.197 INFO:tasks.workunit.client.0.smithi067.stdout:2/603: creat d1/d10/d72/d53/dba/fc3 x:0 0 0 2024-04-22T23:23:30.197 INFO:tasks.workunit.client.0.smithi067.stdout:8/528: chown d1/d3/d5c/d98 119 1 2024-04-22T23:23:30.204 INFO:tasks.workunit.client.0.smithi067.stdout:6/569: symlink d17/d39/d3a/da9/lb8 0 2024-04-22T23:23:30.208 INFO:tasks.workunit.client.0.smithi067.stdout:0/566: creat db/d9a/d56/d70/fc3 x:0 0 0 2024-04-22T23:23:30.208 INFO:tasks.workunit.client.0.smithi067.stdout:9/530: dwrite d5/d21/f27 [4194304,4194304] 0 2024-04-22T23:23:30.213 INFO:tasks.workunit.client.0.smithi067.stdout:2/604: creat d1/d10/d33/d46/da4/fc4 x:0 0 0 2024-04-22T23:23:30.218 INFO:tasks.workunit.client.0.smithi067.stdout:5/594: dwrite d5/d28/d36/d40/d97/dc8/fce [0,4194304] 0 2024-04-22T23:23:30.219 INFO:tasks.workunit.client.0.smithi067.stdout:0/567: unlink c4 0 2024-04-22T23:23:30.229 INFO:tasks.workunit.client.0.smithi067.stdout:0/568: write db/d21/f5a [2495003,15112] 0 2024-04-22T23:23:30.230 INFO:tasks.workunit.client.0.smithi067.stdout:2/605: mkdir d1/d42/d6b/dc5 0 2024-04-22T23:23:30.230 INFO:tasks.workunit.client.0.smithi067.stdout:6/570: dwrite d17/d83/d9b/fa5 [0,4194304] 0 2024-04-22T23:23:30.230 INFO:tasks.workunit.client.0.smithi067.stdout:3/591: dwrite d3/d4/d12/d4b/f63 [0,4194304] 0 2024-04-22T23:23:30.234 INFO:tasks.workunit.client.0.smithi067.stdout:0/569: read - db/d21/d29/d75/fa1 zero size 2024-04-22T23:23:30.243 INFO:tasks.workunit.client.0.smithi067.stdout:7/460: rename d10/d14/d1d/d46/d50/l48 to d10/d14/la3 0 2024-04-22T23:23:30.244 INFO:tasks.workunit.client.0.smithi067.stdout:8/529: getdents d1/d3/d5c 0 2024-04-22T23:23:30.247 INFO:tasks.workunit.client.0.smithi067.stdout:0/570: chown db/de/c80 992 1 2024-04-22T23:23:30.248 INFO:tasks.workunit.client.0.smithi067.stdout:9/531: creat d5/dd/d18/d3c/fb4 x:0 0 0 2024-04-22T23:23:30.250 INFO:tasks.workunit.client.0.smithi067.stdout:5/595: rename d5/d3e/db7 to d5/d28/daf/dbf/dcf 0 2024-04-22T23:23:30.253 INFO:tasks.workunit.client.0.smithi067.stdout:8/530: write d1/d41/d46/f66 [210716,88959] 0 2024-04-22T23:23:30.255 INFO:tasks.workunit.client.0.smithi067.stdout:6/571: truncate d17/d39/d3a/f50 72637 0 2024-04-22T23:23:30.256 INFO:tasks.workunit.client.0.smithi067.stdout:3/592: truncate d3/d4/d10/d1f/f3e 573962 0 2024-04-22T23:23:30.258 INFO:tasks.workunit.client.0.smithi067.stdout:0/571: dwrite db/d21/d29/d2c/f7d [0,4194304] 0 2024-04-22T23:23:30.258 INFO:tasks.workunit.client.0.smithi067.stdout:9/532: symlink d5/dd/d18/d39/d8f/lb5 0 2024-04-22T23:23:30.260 INFO:tasks.workunit.client.0.smithi067.stdout:2/606: creat d1/d10/d1c/fc6 x:0 0 0 2024-04-22T23:23:30.264 INFO:tasks.workunit.client.0.smithi067.stdout:9/533: write d5/dd/d18/d39/d4d/f95 [3826836,29412] 0 2024-04-22T23:23:30.265 INFO:tasks.workunit.client.0.smithi067.stdout:3/593: rmdir d3/d4/d28/d4c/d62 39 2024-04-22T23:23:30.266 INFO:tasks.workunit.client.0.smithi067.stdout:7/461: mknod d10/d14/d1d/d26/ca4 0 2024-04-22T23:23:30.267 INFO:tasks.workunit.client.0.smithi067.stdout:8/531: dwrite d1/d41/d46/f9f [0,4194304] 0 2024-04-22T23:23:30.270 INFO:tasks.workunit.client.0.smithi067.stdout:0/572: write db/d21/d29/d2d/f49 [5199287,62360] 0 2024-04-22T23:23:30.273 INFO:tasks.workunit.client.0.smithi067.stdout:6/572: mkdir d17/d39/d53/d90/d46/db0/db9 0 2024-04-22T23:23:30.278 INFO:tasks.workunit.client.0.smithi067.stdout:2/607: symlink d1/d10/d41/lc7 0 2024-04-22T23:23:30.278 INFO:tasks.workunit.client.0.smithi067.stdout:9/534: dread d5/dd/d18/d39/d4d/f8d [0,4194304] 0 2024-04-22T23:23:30.279 INFO:tasks.workunit.client.0.smithi067.stdout:1/497: sync 2024-04-22T23:23:30.283 INFO:tasks.workunit.client.0.smithi067.stdout:0/573: dwrite db/d9a/d56/d6b/fa2 [0,4194304] 0 2024-04-22T23:23:30.286 INFO:tasks.workunit.client.0.smithi067.stdout:2/608: write d1/d42/d61/fa8 [1045910,45703] 0 2024-04-22T23:23:30.288 INFO:tasks.workunit.client.0.smithi067.stdout:3/594: rmdir d3/d4/d28/d4c/d51 39 2024-04-22T23:23:30.291 INFO:tasks.workunit.client.0.smithi067.stdout:7/462: creat d10/d14/d1d/d26/d2c/d7a/fa5 x:0 0 0 2024-04-22T23:23:30.292 INFO:tasks.workunit.client.0.smithi067.stdout:6/573: dwrite d17/d39/d53/d90/f8b [0,4194304] 0 2024-04-22T23:23:30.295 INFO:tasks.workunit.client.0.smithi067.stdout:3/595: write d3/d4/d1b/faa [498832,46921] 0 2024-04-22T23:23:30.295 INFO:tasks.workunit.client.0.smithi067.stdout:4/613: sync 2024-04-22T23:23:30.296 INFO:tasks.workunit.client.0.smithi067.stdout:6/574: dread - d17/d39/f89 zero size 2024-04-22T23:23:30.298 INFO:tasks.workunit.client.0.smithi067.stdout:0/574: write db/fd [652398,14222] 0 2024-04-22T23:23:30.298 INFO:tasks.workunit.client.0.smithi067.stdout:4/614: chown fe 528 1 2024-04-22T23:23:30.299 INFO:tasks.workunit.client.0.smithi067.stdout:5/596: truncate d5/db/d12/dbb/d61/d89/fb3 8356 0 2024-04-22T23:23:30.302 INFO:tasks.workunit.client.0.smithi067.stdout:8/532: dread d1/d3/f56 [0,4194304] 0 2024-04-22T23:23:30.303 INFO:tasks.workunit.client.0.smithi067.stdout:0/575: write db/faa [1695681,73127] 0 2024-04-22T23:23:30.303 INFO:tasks.workunit.client.0.smithi067.stdout:6/575: dread - d17/d39/d53/d90/d46/fa2 zero size 2024-04-22T23:23:30.305 INFO:tasks.workunit.client.0.smithi067.stdout:3/596: symlink d3/d4/d10/d19/ld7 0 2024-04-22T23:23:30.305 INFO:tasks.workunit.client.0.smithi067.stdout:9/535: dwrite d5/dd/d18/d39/d3e/fae [4194304,4194304] 0 2024-04-22T23:23:30.306 INFO:tasks.workunit.client.0.smithi067.stdout:1/498: symlink d8/d9/d2d/d4d/d48/d9a/la6 0 2024-04-22T23:23:30.306 INFO:tasks.workunit.client.0.smithi067.stdout:0/576: dread - db/de/d2a/f99 zero size 2024-04-22T23:23:30.308 INFO:tasks.workunit.client.0.smithi067.stdout:6/576: dread d17/d39/f81 [0,4194304] 0 2024-04-22T23:23:30.309 INFO:tasks.workunit.client.0.smithi067.stdout:5/597: creat d5/d3e/d90/fd0 x:0 0 0 2024-04-22T23:23:30.309 INFO:tasks.workunit.client.0.smithi067.stdout:8/533: write d1/dd/d26/f8f [305391,107635] 0 2024-04-22T23:23:30.317 INFO:tasks.workunit.client.0.smithi067.stdout:9/536: creat d5/dd/d18/d39/d3e/d3f/d63/fb6 x:0 0 0 2024-04-22T23:23:30.319 INFO:tasks.workunit.client.0.smithi067.stdout:0/577: creat db/d60/d9e/fc4 x:0 0 0 2024-04-22T23:23:30.320 INFO:tasks.workunit.client.0.smithi067.stdout:5/598: dwrite d5/d28/d36/f75 [4194304,4194304] 0 2024-04-22T23:23:30.321 INFO:tasks.workunit.client.0.smithi067.stdout:9/537: write d5/dd/d18/d39/d4d/f95 [1079692,52627] 0 2024-04-22T23:23:30.327 INFO:tasks.workunit.client.0.smithi067.stdout:1/499: mkdir d8/da7 0 2024-04-22T23:23:30.328 INFO:tasks.workunit.client.0.smithi067.stdout:2/609: getdents d1/d42 0 2024-04-22T23:23:30.333 INFO:tasks.workunit.client.0.smithi067.stdout:3/597: mkdir d3/d4/d28/d4c/d90/da2/daf/dd8 0 2024-04-22T23:23:30.334 INFO:tasks.workunit.client.0.smithi067.stdout:5/599: truncate d5/d3a/d4d/f4e 273402 0 2024-04-22T23:23:30.336 INFO:tasks.workunit.client.0.smithi067.stdout:0/578: dread db/faa [0,4194304] 0 2024-04-22T23:23:30.337 INFO:tasks.workunit.client.0.smithi067.stdout:9/538: rename d5/dd/d18/d39/d4d/c8b to d5/dd/d18/d39/d4d/cb7 0 2024-04-22T23:23:30.339 INFO:tasks.workunit.client.0.smithi067.stdout:2/610: dwrite d1/d10/d72/d68/fab [4194304,4194304] 0 2024-04-22T23:23:30.342 INFO:tasks.workunit.client.0.smithi067.stdout:4/615: getdents d12/d5f/d7e 0 2024-04-22T23:23:30.342 INFO:tasks.workunit.client.0.smithi067.stdout:5/600: write d5/d3e/d87/d99/fc3 [555945,66113] 0 2024-04-22T23:23:30.344 INFO:tasks.workunit.client.0.smithi067.stdout:6/577: dwrite d17/d39/d53/d90/d46/f5f [0,4194304] 0 2024-04-22T23:23:30.347 INFO:tasks.workunit.client.0.smithi067.stdout:2/611: chown d1/d10/d72/d53/dba/fc3 84789234 1 2024-04-22T23:23:30.347 INFO:tasks.workunit.client.0.smithi067.stdout:0/579: mkdir db/de/d11/d1a/da4/dc5 0 2024-04-22T23:23:30.347 INFO:tasks.workunit.client.0.smithi067.stdout:9/539: truncate d5/d21/d2c/fa5 85179 0 2024-04-22T23:23:30.350 INFO:tasks.workunit.client.0.smithi067.stdout:5/601: fsync d5/d28/d36/d40/f70 0 2024-04-22T23:23:30.351 INFO:tasks.workunit.client.0.smithi067.stdout:2/612: chown d1/f1b 1877 1 2024-04-22T23:23:30.352 INFO:tasks.workunit.client.0.smithi067.stdout:9/540: fdatasync d5/dd/f1a 0 2024-04-22T23:23:30.355 INFO:tasks.workunit.client.0.smithi067.stdout:3/598: rename d3/d4/d28/d40/l89 to d3/d4/d28/ld9 0 2024-04-22T23:23:30.356 INFO:tasks.workunit.client.0.smithi067.stdout:5/602: chown d5/c91 2020 1 2024-04-22T23:23:30.358 INFO:tasks.workunit.client.0.smithi067.stdout:6/578: creat d17/d39/d3a/da9/fba x:0 0 0 2024-04-22T23:23:30.360 INFO:tasks.workunit.client.0.smithi067.stdout:2/613: rmdir d1/d10/d33 39 2024-04-22T23:23:30.360 INFO:tasks.workunit.client.0.smithi067.stdout:8/534: dwrite d1/d3/d27/f28 [0,4194304] 0 2024-04-22T23:23:30.361 INFO:tasks.workunit.client.0.smithi067.stdout:0/580: symlink db/d2e/db3/lc6 0 2024-04-22T23:23:30.364 INFO:tasks.workunit.client.0.smithi067.stdout:5/603: creat d5/d28/d36/d40/fd1 x:0 0 0 2024-04-22T23:23:30.364 INFO:tasks.workunit.client.0.smithi067.stdout:4/616: dwrite d12/d17/d88/fc8 [0,4194304] 0 2024-04-22T23:23:30.371 INFO:tasks.workunit.client.0.smithi067.stdout:2/614: rename d1/d42/d6b to d1/d10/d1c/d6d/d87/dc8 0 2024-04-22T23:23:30.380 INFO:tasks.workunit.client.0.smithi067.stdout:9/541: dwrite d5/fad [0,4194304] 0 2024-04-22T23:23:30.381 INFO:tasks.workunit.client.0.smithi067.stdout:8/535: rename d1/l8 to d1/la6 0 2024-04-22T23:23:30.384 INFO:tasks.workunit.client.0.smithi067.stdout:2/615: write d1/d10/d41/f70 [2081338,34250] 0 2024-04-22T23:23:30.387 INFO:tasks.workunit.client.0.smithi067.stdout:0/581: dwrite db/faf [0,4194304] 0 2024-04-22T23:23:30.391 INFO:tasks.workunit.client.0.smithi067.stdout:5/604: creat d5/d3a/fd2 x:0 0 0 2024-04-22T23:23:30.393 INFO:tasks.workunit.client.0.smithi067.stdout:3/599: dwrite d3/d4/d10/d38/d53/d55/d74/fc9 [4194304,4194304] 0 2024-04-22T23:23:30.394 INFO:tasks.workunit.client.0.smithi067.stdout:6/579: dwrite d17/d39/d53/f74 [0,4194304] 0 2024-04-22T23:23:30.394 INFO:tasks.workunit.client.0.smithi067.stdout:9/542: symlink d5/d21/lb8 0 2024-04-22T23:23:30.395 INFO:tasks.workunit.client.0.smithi067.stdout:0/582: dread - db/d2e/f40 zero size 2024-04-22T23:23:30.397 INFO:tasks.workunit.client.0.smithi067.stdout:2/616: dread d1/f1f [0,4194304] 0 2024-04-22T23:23:30.401 INFO:tasks.workunit.client.0.smithi067.stdout:4/617: dwrite d12/d42/d4b/d70/d8e/fcf [0,4194304] 0 2024-04-22T23:23:30.406 INFO:tasks.workunit.client.0.smithi067.stdout:6/580: mkdir d17/d83/db5/dbb 0 2024-04-22T23:23:30.406 INFO:tasks.workunit.client.0.smithi067.stdout:0/583: stat db/d9a/d56/d70/l85 0 2024-04-22T23:23:30.409 INFO:tasks.workunit.client.0.smithi067.stdout:6/581: dread - d17/d39/fa7 zero size 2024-04-22T23:23:30.409 INFO:tasks.workunit.client.0.smithi067.stdout:3/600: creat d3/d4/d28/fda x:0 0 0 2024-04-22T23:23:30.411 INFO:tasks.workunit.client.0.smithi067.stdout:4/618: unlink d12/d17/d47/c98 0 2024-04-22T23:23:30.412 INFO:tasks.workunit.client.0.smithi067.stdout:9/543: dwrite d5/dd/d18/d39/d3e/fae [0,4194304] 0 2024-04-22T23:23:30.412 INFO:tasks.workunit.client.0.smithi067.stdout:2/617: creat d1/d10/d33/d46/da4/fc9 x:0 0 0 2024-04-22T23:23:30.413 INFO:tasks.workunit.client.0.smithi067.stdout:5/605: dwrite d5/d28/daf/dbf/dcf/f8f [4194304,4194304] 0 2024-04-22T23:23:30.417 INFO:tasks.workunit.client.0.smithi067.stdout:6/582: mkdir d17/d83/db5/dbc 0 2024-04-22T23:23:30.419 INFO:tasks.workunit.client.0.smithi067.stdout:4/619: chown d12/d17/l1a 669186582 1 2024-04-22T23:23:30.422 INFO:tasks.workunit.client.0.smithi067.stdout:8/536: dwrite d1/d3/d27/f63 [0,4194304] 0 2024-04-22T23:23:30.428 INFO:tasks.workunit.client.0.smithi067.stdout:2/618: truncate d1/d10/d33/d46/f47 2181271 0 2024-04-22T23:23:30.430 INFO:tasks.workunit.client.0.smithi067.stdout:4/620: fsync d12/d42/f83 0 2024-04-22T23:23:30.430 INFO:tasks.workunit.client.0.smithi067.stdout:3/601: write d3/d4/f37 [125959,45831] 0 2024-04-22T23:23:30.436 INFO:tasks.workunit.client.0.smithi067.stdout:5/606: truncate d5/db/d12/dbb/d61/d89/fb3 837341 0 2024-04-22T23:23:30.436 INFO:tasks.workunit.client.0.smithi067.stdout:8/537: mknod d1/d3/d5c/d98/ca7 0 2024-04-22T23:23:30.442 INFO:tasks.workunit.client.0.smithi067.stdout:4/621: dwrite d12/d17/d3f/d84/f9f [0,4194304] 0 2024-04-22T23:23:30.442 INFO:tasks.workunit.client.0.smithi067.stdout:8/538: mkdir d1/dd/d80/d6f/da8 0 2024-04-22T23:23:30.442 INFO:tasks.workunit.client.0.smithi067.stdout:6/583: link d17/d39/l7a d17/d39/lbd 0 2024-04-22T23:23:30.444 INFO:tasks.workunit.client.0.smithi067.stdout:8/539: stat d1/dd/d26/f6b 0 2024-04-22T23:23:30.446 INFO:tasks.workunit.client.0.smithi067.stdout:9/544: dwrite d5/dd/d18/d39/d49/d54/f79 [0,4194304] 0 2024-04-22T23:23:30.448 INFO:tasks.workunit.client.0.smithi067.stdout:3/602: rename d3/d4/d28/d4c to d3/d4/d10/d38/d53/d55/d74/dd5/ddb 0 2024-04-22T23:23:30.449 INFO:tasks.workunit.client.0.smithi067.stdout:5/607: creat d5/d28/dae/fd3 x:0 0 0 2024-04-22T23:23:30.454 INFO:tasks.workunit.client.0.smithi067.stdout:4/622: mkdir d12/d42/ddb 0 2024-04-22T23:23:30.455 INFO:tasks.workunit.client.0.smithi067.stdout:9/545: mkdir d5/dd/da9/db9 0 2024-04-22T23:23:30.456 INFO:tasks.workunit.client.0.smithi067.stdout:9/546: chown d5/dd/d18/d39/d3e/d3f/d63 5 1 2024-04-22T23:23:30.457 INFO:tasks.workunit.client.0.smithi067.stdout:8/540: symlink d1/dd/la9 0 2024-04-22T23:23:30.462 INFO:tasks.workunit.client.0.smithi067.stdout:2/619: rename d1/d42/f4b to d1/d42/d61/daa/fca 0 2024-04-22T23:23:30.464 INFO:tasks.workunit.client.0.smithi067.stdout:2/620: fsync d1/d42/d61/daa/faf 0 2024-04-22T23:23:30.465 INFO:tasks.workunit.client.0.smithi067.stdout:4/623: creat d12/d17/d88/fdc x:0 0 0 2024-04-22T23:23:30.466 INFO:tasks.workunit.client.0.smithi067.stdout:9/547: dwrite d5/dd/d18/d39/d3e/d3f/f7e [0,4194304] 0 2024-04-22T23:23:30.471 INFO:tasks.workunit.client.0.smithi067.stdout:6/584: rename d17/d83/d9b/fa5 to d17/d39/d3a/da9/fbe 0 2024-04-22T23:23:30.474 INFO:tasks.workunit.client.0.smithi067.stdout:4/624: fsync d12/f38 0 2024-04-22T23:23:30.475 INFO:tasks.workunit.client.0.smithi067.stdout:3/603: write d3/d4/d12/f1d [346924,37618] 0 2024-04-22T23:23:30.476 INFO:tasks.workunit.client.0.smithi067.stdout:4/625: dread - d12/d5f/fb5 zero size 2024-04-22T23:23:30.476 INFO:tasks.workunit.client.0.smithi067.stdout:9/548: mkdir d5/dd/d18/d39/d8f/dba 0 2024-04-22T23:23:30.481 INFO:tasks.workunit.client.0.smithi067.stdout:6/585: write d17/d39/f4b [1982761,56963] 0 2024-04-22T23:23:30.483 INFO:tasks.workunit.client.0.smithi067.stdout:2/621: mknod d1/d10/d41/da2/ccb 0 2024-04-22T23:23:30.483 INFO:tasks.workunit.client.0.smithi067.stdout:5/608: write d5/db/d12/dbb/d61/d89/fb3 [1215518,105180] 0 2024-04-22T23:23:30.486 INFO:tasks.workunit.client.0.smithi067.stdout:8/541: truncate d1/dd/d80/d6f/f32 5634604 0 2024-04-22T23:23:30.489 INFO:tasks.workunit.client.0.smithi067.stdout:2/622: creat d1/d10/d33/d46/da4/fcc x:0 0 0 2024-04-22T23:23:30.489 INFO:tasks.workunit.client.0.smithi067.stdout:5/609: read d5/db/d12/f13 [1115499,128400] 0 2024-04-22T23:23:30.490 INFO:tasks.workunit.client.0.smithi067.stdout:9/549: symlink d5/dd/d18/lbb 0 2024-04-22T23:23:30.491 INFO:tasks.workunit.client.0.smithi067.stdout:3/604: dwrite d3/d4/d12/f27 [0,4194304] 0 2024-04-22T23:23:30.492 INFO:tasks.workunit.client.0.smithi067.stdout:2/623: chown d1/d10/d72/d68/da1/fb5 21 1 2024-04-22T23:23:30.492 INFO:tasks.workunit.client.0.smithi067.stdout:8/542: creat d1/d3/d5c/faa x:0 0 0 2024-04-22T23:23:30.495 INFO:tasks.workunit.client.0.smithi067.stdout:2/624: read - d1/d12/f92 zero size 2024-04-22T23:23:30.495 INFO:tasks.workunit.client.0.smithi067.stdout:8/543: chown d1/d41/d46/f60 18340686 1 2024-04-22T23:23:30.497 INFO:tasks.workunit.client.0.smithi067.stdout:2/625: chown d1/d10/d72/c3f 16 1 2024-04-22T23:23:30.497 INFO:tasks.workunit.client.0.smithi067.stdout:9/550: mknod d5/dd/d18/d39/d3e/d3f/d63/cbc 0 2024-04-22T23:23:30.500 INFO:tasks.workunit.client.0.smithi067.stdout:6/586: truncate d17/f48 1546646 0 2024-04-22T23:23:30.503 INFO:tasks.workunit.client.0.smithi067.stdout:2/626: rename d1/d4f/c5a to d1/d10/d1c/d6d/d87/dc8/ccd 0 2024-04-22T23:23:30.507 INFO:tasks.workunit.client.0.smithi067.stdout:6/587: rename d17/d83/c96 to d17/d83/d9b/cbf 0 2024-04-22T23:23:30.508 INFO:tasks.workunit.client.0.smithi067.stdout:8/544: creat d1/dd/fab x:0 0 0 2024-04-22T23:23:30.510 INFO:tasks.workunit.client.0.smithi067.stdout:3/605: getdents d3/d4/d10/d38/d53/d55/d74 0 2024-04-22T23:23:30.510 INFO:tasks.workunit.client.0.smithi067.stdout:5/610: dread d5/db/ff [0,4194304] 0 2024-04-22T23:23:30.512 INFO:tasks.workunit.client.0.smithi067.stdout:6/588: symlink d17/d39/d3a/da9/lc0 0 2024-04-22T23:23:30.515 INFO:tasks.workunit.client.0.smithi067.stdout:8/545: creat d1/d3/d10/d3b/fac x:0 0 0 2024-04-22T23:23:30.517 INFO:tasks.workunit.client.0.smithi067.stdout:5/611: creat d5/d28/daf/dbf/dcf/d9a/fd4 x:0 0 0 2024-04-22T23:23:30.519 INFO:tasks.workunit.client.0.smithi067.stdout:6/589: unlink d17/fa0 0 2024-04-22T23:23:30.519 INFO:tasks.workunit.client.0.smithi067.stdout:2/627: write d1/d10/d33/d46/f7d [860230,41834] 0 2024-04-22T23:23:30.521 INFO:tasks.workunit.client.0.smithi067.stdout:3/606: mkdir d3/d4/d10/d38/d53/ddc 0 2024-04-22T23:23:30.521 INFO:tasks.workunit.client.0.smithi067.stdout:2/628: chown d1/d10/d33/d46/da4/fc9 17261542 1 2024-04-22T23:23:30.522 INFO:tasks.workunit.client.0.smithi067.stdout:6/590: fsync d17/d39/d3a/d8d/fa4 0 2024-04-22T23:23:30.523 INFO:tasks.workunit.client.0.smithi067.stdout:6/591: dread - d17/d83/f98 zero size 2024-04-22T23:23:30.525 INFO:tasks.workunit.client.0.smithi067.stdout:8/546: dread d1/dd/d80/d6f/f57 [0,4194304] 0 2024-04-22T23:23:30.525 INFO:tasks.workunit.client.0.smithi067.stdout:3/607: creat d3/d4/d10/d38/d53/d55/d5c/fdd x:0 0 0 2024-04-22T23:23:30.527 INFO:tasks.workunit.client.0.smithi067.stdout:2/629: mkdir d1/d10/d72/d68/d7b/dce 0 2024-04-22T23:23:30.527 INFO:tasks.workunit.client.0.smithi067.stdout:2/630: stat d1/f69 0 2024-04-22T23:23:30.527 INFO:tasks.workunit.client.0.smithi067.stdout:5/612: rmdir d5/db/d35 39 2024-04-22T23:23:30.530 INFO:tasks.workunit.client.0.smithi067.stdout:3/608: creat d3/dcb/fde x:0 0 0 2024-04-22T23:23:30.531 INFO:tasks.workunit.client.0.smithi067.stdout:6/592: creat d17/d83/db5/dbb/fc1 x:0 0 0 2024-04-22T23:23:30.533 INFO:tasks.workunit.client.0.smithi067.stdout:2/631: fsync d1/d19/f5c 0 2024-04-22T23:23:30.533 INFO:tasks.workunit.client.0.smithi067.stdout:8/547: dwrite d1/f76 [0,4194304] 0 2024-04-22T23:23:30.537 INFO:tasks.workunit.client.0.smithi067.stdout:5/613: mknod d5/db/d12/d44/cd5 0 2024-04-22T23:23:30.539 INFO:tasks.workunit.client.0.smithi067.stdout:2/632: truncate d1/d10/d1c/d4d/fa9 1576 0 2024-04-22T23:23:30.541 INFO:tasks.workunit.client.0.smithi067.stdout:8/548: creat d1/dd/d80/d6f/fad x:0 0 0 2024-04-22T23:23:30.544 INFO:tasks.workunit.client.0.smithi067.stdout:6/593: dwrite d17/d39/d53/f6a [0,4194304] 0 2024-04-22T23:23:30.544 INFO:tasks.workunit.client.0.smithi067.stdout:8/549: write d1/dd/d26/f93 [927956,58077] 0 2024-04-22T23:23:30.547 INFO:tasks.workunit.client.0.smithi067.stdout:5/614: rename d5/d3e/d90/c9f to d5/db/d12/dbb/d61/cd6 0 2024-04-22T23:23:30.548 INFO:tasks.workunit.client.0.smithi067.stdout:6/594: truncate d17/d39/f8c 542338 0 2024-04-22T23:23:30.549 INFO:tasks.workunit.client.0.smithi067.stdout:5/615: read d5/d28/daf/dbf/dcf/f8f [7227960,95584] 0 2024-04-22T23:23:30.550 INFO:tasks.workunit.client.0.smithi067.stdout:3/609: dread d3/d4/d28/d5f/f71 [0,4194304] 0 2024-04-22T23:23:30.553 INFO:tasks.workunit.client.0.smithi067.stdout:8/550: link d1/dd/d80/f86 d1/dd/d80/d6f/da8/fae 0 2024-04-22T23:23:30.558 INFO:tasks.workunit.client.0.smithi067.stdout:5/616: mknod d5/db/d12/dbb/d61/d89/cd7 0 2024-04-22T23:23:30.559 INFO:tasks.workunit.client.0.smithi067.stdout:3/610: dwrite d3/d4/d10/d1f/d3c/fac [0,4194304] 0 2024-04-22T23:23:30.561 INFO:tasks.workunit.client.0.smithi067.stdout:5/617: read - d5/db/d12/d44/d47/f6d zero size 2024-04-22T23:23:30.563 INFO:tasks.workunit.client.0.smithi067.stdout:6/595: dwrite d17/d39/d53/d90/d46/d62/f72 [0,4194304] 0 2024-04-22T23:23:30.563 INFO:tasks.workunit.client.0.smithi067.stdout:8/551: creat d1/faf x:0 0 0 2024-04-22T23:23:30.564 INFO:tasks.workunit.client.0.smithi067.stdout:2/633: write d1/f1f [658551,38274] 0 2024-04-22T23:23:30.566 INFO:tasks.workunit.client.0.smithi067.stdout:6/596: fdatasync d17/d39/d53/d90/d46/fa6 0 2024-04-22T23:23:30.566 INFO:tasks.workunit.client.0.smithi067.stdout:3/611: read d3/d4/d10/d38/f57 [1968814,81343] 0 2024-04-22T23:23:30.566 INFO:tasks.workunit.client.0.smithi067.stdout:2/634: chown d1/d10/d41/f70 7292818 1 2024-04-22T23:23:30.568 INFO:tasks.workunit.client.0.smithi067.stdout:8/552: truncate d1/d3/d5c/f99 489261 0 2024-04-22T23:23:30.569 INFO:tasks.workunit.client.0.smithi067.stdout:8/553: chown d1/dd/d80/d6f/c78 2 1 2024-04-22T23:23:30.571 INFO:tasks.workunit.client.0.smithi067.stdout:2/635: mknod d1/d4f/ccf 0 2024-04-22T23:23:30.571 INFO:tasks.workunit.client.0.smithi067.stdout:6/597: mknod d17/cc2 0 2024-04-22T23:23:30.572 INFO:tasks.workunit.client.0.smithi067.stdout:2/636: dread d1/d10/d1c/d4d/fa9 [0,4194304] 0 2024-04-22T23:23:30.573 INFO:tasks.workunit.client.0.smithi067.stdout:6/598: write d17/d39/d3a/f61 [4716218,82647] 0 2024-04-22T23:23:30.574 INFO:tasks.workunit.client.0.smithi067.stdout:8/554: creat d1/d41/d46/fb0 x:0 0 0 2024-04-22T23:23:30.574 INFO:tasks.workunit.client.0.smithi067.stdout:2/637: chown d1/d10/d1c/d4d/l71 5 1 2024-04-22T23:23:30.575 INFO:tasks.workunit.client.0.smithi067.stdout:8/555: fsync d1/d41/d46/f66 0 2024-04-22T23:23:30.576 INFO:tasks.workunit.client.0.smithi067.stdout:8/556: fdatasync d1/d3/d27/f28 0 2024-04-22T23:23:30.577 INFO:tasks.workunit.client.0.smithi067.stdout:5/618: link d5/d28/daf/db4/lca d5/db/ld8 0 2024-04-22T23:23:30.579 INFO:tasks.workunit.client.0.smithi067.stdout:2/638: symlink d1/d10/d72/ld0 0 2024-04-22T23:23:30.580 INFO:tasks.workunit.client.0.smithi067.stdout:8/557: creat d1/d3/d5c/fb1 x:0 0 0 2024-04-22T23:23:30.581 INFO:tasks.workunit.client.0.smithi067.stdout:6/599: creat d17/d39/d53/d90/d46/db0/db9/fc3 x:0 0 0 2024-04-22T23:23:30.581 INFO:tasks.workunit.client.0.smithi067.stdout:8/558: dread - d1/dd/d26/d62/d94/fa3 zero size 2024-04-22T23:23:30.582 INFO:tasks.workunit.client.0.smithi067.stdout:5/619: symlink d5/db/d12/dbb/d61/ld9 0 2024-04-22T23:23:30.585 INFO:tasks.workunit.client.0.smithi067.stdout:8/559: mknod d1/d3/d27/cb2 0 2024-04-22T23:23:30.591 INFO:tasks.workunit.client.0.smithi067.stdout:8/560: mknod d1/dd/d26/cb3 0 2024-04-22T23:23:30.595 INFO:tasks.workunit.client.0.smithi067.stdout:8/561: mkdir d1/d41/db4 0 2024-04-22T23:23:30.597 INFO:tasks.workunit.client.0.smithi067.stdout:2/639: dread d1/d10/d1c/d6d/f4c [0,4194304] 0 2024-04-22T23:23:30.598 INFO:tasks.workunit.client.0.smithi067.stdout:2/640: write d1/d42/d61/f7e [4642757,42114] 0 2024-04-22T23:23:30.617 INFO:tasks.workunit.client.0.smithi067.stdout:5/620: dwrite d5/d3a/d4d/f4e [0,4194304] 0 2024-04-22T23:23:30.619 INFO:tasks.workunit.client.0.smithi067.stdout:6/600: dwrite d17/f9a [0,4194304] 0 2024-04-22T23:23:30.622 INFO:tasks.workunit.client.0.smithi067.stdout:2/641: dwrite d1/d10/d72/d68/d7b/d95/fac [0,4194304] 0 2024-04-22T23:23:30.623 INFO:tasks.workunit.client.0.smithi067.stdout:8/562: dread d1/d3/d27/f58 [0,4194304] 0 2024-04-22T23:23:30.625 INFO:tasks.workunit.client.0.smithi067.stdout:6/601: rename d17/d39/d53 to d17/d39/d53/d90/d46/d62/d88/dc4 22 2024-04-22T23:23:30.625 INFO:tasks.workunit.client.0.smithi067.stdout:2/642: stat d1/d10/d33/c63 0 2024-04-22T23:23:30.626 INFO:tasks.workunit.client.0.smithi067.stdout:2/643: chown d1/lb0 0 1 2024-04-22T23:23:30.627 INFO:tasks.workunit.client.0.smithi067.stdout:8/563: mknod d1/d41/d46/cb5 0 2024-04-22T23:23:30.629 INFO:tasks.workunit.client.0.smithi067.stdout:2/644: symlink d1/d10/d72/d53/dba/ld1 0 2024-04-22T23:23:30.630 INFO:tasks.workunit.client.0.smithi067.stdout:8/564: mkdir d1/dd/d80/d6f/db6 0 2024-04-22T23:23:30.631 INFO:tasks.workunit.client.0.smithi067.stdout:8/565: chown d1/dd/f95 27970876 1 2024-04-22T23:23:30.636 INFO:tasks.workunit.client.0.smithi067.stdout:2/645: link d1/d42/d5f/l96 d1/dbf/ld2 0 2024-04-22T23:23:30.638 INFO:tasks.workunit.client.0.smithi067.stdout:8/566: rename d1/dd/fab to d1/dd/d26/d62/fb7 0 2024-04-22T23:23:30.640 INFO:tasks.workunit.client.0.smithi067.stdout:2/646: creat d1/d42/d61/dad/fd3 x:0 0 0 2024-04-22T23:23:30.642 INFO:tasks.workunit.client.0.smithi067.stdout:2/647: truncate d1/d19/f21 2588389 0 2024-04-22T23:23:30.643 INFO:tasks.workunit.client.0.smithi067.stdout:5/621: dread d5/db/d12/dbb/d61/d89/fb5 [0,4194304] 0 2024-04-22T23:23:30.646 INFO:tasks.workunit.client.0.smithi067.stdout:5/622: getdents d5/d28/daf/dbf/dcf/d9a 0 2024-04-22T23:23:30.652 INFO:tasks.workunit.client.0.smithi067.stdout:5/623: dwrite d5/d28/f64 [0,4194304] 0 2024-04-22T23:23:30.662 INFO:tasks.workunit.client.0.smithi067.stdout:5/624: dwrite d5/db/d12/dbb/f72 [0,4194304] 0 2024-04-22T23:23:30.666 INFO:tasks.workunit.client.0.smithi067.stdout:5/625: creat d5/db/d12/d44/d47/fda x:0 0 0 2024-04-22T23:23:30.680 INFO:tasks.workunit.client.0.smithi067.stdout:5/626: truncate d5/db/f23 3404787 0 2024-04-22T23:23:30.682 INFO:tasks.workunit.client.0.smithi067.stdout:1/500: sync 2024-04-22T23:23:30.682 INFO:tasks.workunit.client.0.smithi067.stdout:7/463: sync 2024-04-22T23:23:30.682 INFO:tasks.workunit.client.0.smithi067.stdout:5/627: creat d5/d28/daf/dbf/dcf/d9a/fdb x:0 0 0 2024-04-22T23:23:30.683 INFO:tasks.workunit.client.0.smithi067.stdout:7/464: chown d10/d14/d1d/d46/d6a 559 1 2024-04-22T23:23:30.684 INFO:tasks.workunit.client.0.smithi067.stdout:7/465: symlink d10/d18/d35/la6 0 2024-04-22T23:23:30.687 INFO:tasks.workunit.client.0.smithi067.stdout:1/501: link d8/f1d d8/d9/d2d/d4d/d70/d34/d60/d62/fa8 0 2024-04-22T23:23:30.710 INFO:tasks.workunit.client.0.smithi067.stdout:1/502: creat d8/d9/fa9 x:0 0 0 2024-04-22T23:23:30.710 INFO:tasks.workunit.client.0.smithi067.stdout:7/466: dwrite fb [0,4194304] 0 2024-04-22T23:23:30.711 INFO:tasks.workunit.client.0.smithi067.stdout:5/628: truncate d5/db/d12/f25 44346 0 2024-04-22T23:23:30.715 INFO:tasks.workunit.client.0.smithi067.stdout:5/629: rmdir d5/d28/daf 39 2024-04-22T23:23:30.719 INFO:tasks.workunit.client.0.smithi067.stdout:1/503: chown d8/d9/d2d/d4d/l14 10575 1 2024-04-22T23:23:30.720 INFO:tasks.workunit.client.0.smithi067.stdout:1/504: write d8/d12/f17 [409843,18678] 0 2024-04-22T23:23:30.721 INFO:tasks.workunit.client.0.smithi067.stdout:7/467: creat d10/d14/d1d/d46/d6a/d91/fa7 x:0 0 0 2024-04-22T23:23:30.724 INFO:tasks.workunit.client.0.smithi067.stdout:1/505: rmdir d8/d12/d40 39 2024-04-22T23:23:30.726 INFO:tasks.workunit.client.0.smithi067.stdout:7/468: rename d10/d14/d1d/d46/d50/l4a to d10/d14/d1d/d26/d7e/la8 0 2024-04-22T23:23:30.728 INFO:tasks.workunit.client.0.smithi067.stdout:1/506: write d8/d9/d2f/d59/f7b [5178706,33703] 0 2024-04-22T23:23:30.728 INFO:tasks.workunit.client.0.smithi067.stdout:5/630: truncate d5/d3a/d4d/fb1 36198 0 2024-04-22T23:23:30.731 INFO:tasks.workunit.client.0.smithi067.stdout:1/507: rmdir d8/d9/d2f/d33 39 2024-04-22T23:23:30.732 INFO:tasks.workunit.client.0.smithi067.stdout:1/508: truncate d8/d9/d2d/d4d/f9e 5010369 0 2024-04-22T23:23:30.736 INFO:tasks.workunit.client.0.smithi067.stdout:7/469: rename d10/f45 to d10/d14/d1d/fa9 0 2024-04-22T23:23:30.737 INFO:tasks.workunit.client.0.smithi067.stdout:7/470: stat d10/d14/d1d/d26/f93 0 2024-04-22T23:23:30.737 INFO:tasks.workunit.client.0.smithi067.stdout:1/509: rmdir d8 39 2024-04-22T23:23:30.738 INFO:tasks.workunit.client.0.smithi067.stdout:7/471: write d10/d14/d1d/d46/d6a/d91/fa7 [436076,22415] 0 2024-04-22T23:23:30.740 INFO:tasks.workunit.client.0.smithi067.stdout:7/472: write d10/d14/d1d/f1f [5110551,106731] 0 2024-04-22T23:23:30.742 INFO:tasks.workunit.client.0.smithi067.stdout:7/473: dread - d10/d14/d1d/d26/d2c/d41/f9f zero size 2024-04-22T23:23:30.744 INFO:tasks.workunit.client.0.smithi067.stdout:5/631: getdents d5/d28/d36/d40/d97 0 2024-04-22T23:23:30.746 INFO:tasks.workunit.client.0.smithi067.stdout:1/510: write d8/f1d [4132258,10498] 0 2024-04-22T23:23:30.749 INFO:tasks.workunit.client.0.smithi067.stdout:7/474: symlink d10/d14/d1d/d26/laa 0 2024-04-22T23:23:30.750 INFO:tasks.workunit.client.0.smithi067.stdout:5/632: symlink d5/db/d35/ldc 0 2024-04-22T23:23:30.751 INFO:tasks.workunit.client.0.smithi067.stdout:7/475: chown d10/d14/d1d/d26/f71 1 1 2024-04-22T23:23:30.753 INFO:tasks.workunit.client.0.smithi067.stdout:5/633: creat d5/d28/d36/fdd x:0 0 0 2024-04-22T23:23:30.753 INFO:tasks.workunit.client.0.smithi067.stdout:1/511: dwrite d8/d9/fd [0,4194304] 0 2024-04-22T23:23:30.754 INFO:tasks.workunit.client.0.smithi067.stdout:5/634: write d5/d3a/fd2 [696505,96621] 0 2024-04-22T23:23:30.755 INFO:tasks.workunit.client.0.smithi067.stdout:7/476: mkdir d10/d47/dab 0 2024-04-22T23:23:30.758 INFO:tasks.workunit.client.0.smithi067.stdout:1/512: creat d8/d9/d2d/d4d/d9c/faa x:0 0 0 2024-04-22T23:23:30.759 INFO:tasks.workunit.client.0.smithi067.stdout:5/635: creat d5/d3a/d4d/fde x:0 0 0 2024-04-22T23:23:30.761 INFO:tasks.workunit.client.0.smithi067.stdout:1/513: truncate d8/d9/d2d/d4d/d9c/faa 242325 0 2024-04-22T23:23:30.766 INFO:tasks.workunit.client.0.smithi067.stdout:1/514: mknod d8/d9/d2f/d33/d66/d84/cab 0 2024-04-22T23:23:30.767 INFO:tasks.workunit.client.0.smithi067.stdout:5/636: dwrite d5/d28/daf/db4/fb6 [0,4194304] 0 2024-04-22T23:23:30.772 INFO:tasks.workunit.client.0.smithi067.stdout:5/637: rename d5/db/d12/dbb/d61/l63 to d5/ldf 0 2024-04-22T23:23:30.776 INFO:tasks.workunit.client.0.smithi067.stdout:7/477: dread fe [0,4194304] 0 2024-04-22T23:23:30.779 INFO:tasks.workunit.client.0.smithi067.stdout:7/478: fdatasync d10/d14/d1d/d26/d2c/d41/f9f 0 2024-04-22T23:23:30.782 INFO:tasks.workunit.client.0.smithi067.stdout:7/479: symlink d10/d14/d1d/d26/d2c/lac 0 2024-04-22T23:23:30.787 INFO:tasks.workunit.client.0.smithi067.stdout:7/480: link d10/d14/d1d/d46/d50/c72 d10/d14/cad 0 2024-04-22T23:23:30.788 INFO:tasks.workunit.client.0.smithi067.stdout:7/481: fsync d10/d18/d35/d62/f9a 0 2024-04-22T23:23:30.790 INFO:tasks.workunit.client.0.smithi067.stdout:7/482: symlink d10/d14/d1d/d26/d2c/d41/d82/lae 0 2024-04-22T23:23:30.794 INFO:tasks.workunit.client.0.smithi067.stdout:1/515: dread d8/d9/d2f/f38 [4194304,4194304] 0 2024-04-22T23:23:30.796 INFO:tasks.workunit.client.0.smithi067.stdout:7/483: dwrite d10/d47/d49/f9c [0,4194304] 0 2024-04-22T23:23:30.797 INFO:tasks.workunit.client.0.smithi067.stdout:1/516: creat d8/d9/d2d/d4d/d48/d9a/fac x:0 0 0 2024-04-22T23:23:30.798 INFO:tasks.workunit.client.0.smithi067.stdout:1/517: chown d8/d12/f17 578 1 2024-04-22T23:23:30.801 INFO:tasks.workunit.client.0.smithi067.stdout:1/518: stat d8/d9/l16 0 2024-04-22T23:23:30.805 INFO:tasks.workunit.client.0.smithi067.stdout:7/484: dwrite d10/d14/d1d/d26/d2c/d41/fa1 [0,4194304] 0 2024-04-22T23:23:30.805 INFO:tasks.workunit.client.0.smithi067.stdout:1/519: creat d8/d12/d36/d87/fad x:0 0 0 2024-04-22T23:23:30.814 INFO:tasks.workunit.client.0.smithi067.stdout:1/520: getdents d8/d9/d2d/d4d/d70/d34/d60/d62 0 2024-04-22T23:23:30.814 INFO:tasks.workunit.client.0.smithi067.stdout:1/521: symlink d8/d9/d2f/lae 0 2024-04-22T23:23:30.816 INFO:tasks.workunit.client.0.smithi067.stdout:1/522: mknod d8/d9/d2d/d4d/d70/d34/d5d/caf 0 2024-04-22T23:23:30.821 INFO:tasks.workunit.client.0.smithi067.stdout:1/523: dread d8/d9/d2f/d59/f7b [4194304,4194304] 0 2024-04-22T23:23:30.823 INFO:tasks.workunit.client.0.smithi067.stdout:1/524: symlink d8/d9/d2f/lb0 0 2024-04-22T23:23:30.825 INFO:tasks.workunit.client.0.smithi067.stdout:1/525: fdatasync d8/f4b 0 2024-04-22T23:23:30.829 INFO:tasks.workunit.client.0.smithi067.stdout:1/526: truncate d8/d9/d2d/d4d/d70/d34/f47 684485 0 2024-04-22T23:23:30.833 INFO:tasks.workunit.client.0.smithi067.stdout:1/527: write d8/d12/d40/f44 [674524,80022] 0 2024-04-22T23:23:30.835 INFO:tasks.workunit.client.0.smithi067.stdout:1/528: mknod d8/d9/cb1 0 2024-04-22T23:23:30.839 INFO:tasks.workunit.client.0.smithi067.stdout:1/529: dwrite d8/d12/f17 [4194304,4194304] 0 2024-04-22T23:23:30.843 INFO:tasks.workunit.client.0.smithi067.stdout:1/530: fsync d8/d9/d2d/d4d/d70/d34/d60/d62/f95 0 2024-04-22T23:23:30.847 INFO:tasks.workunit.client.0.smithi067.stdout:1/531: truncate d8/d9/d2d/d4d/d9c/f8a 1916753 0 2024-04-22T23:23:30.850 INFO:tasks.workunit.client.0.smithi067.stdout:1/532: creat d8/d9/d2f/d33/fb2 x:0 0 0 2024-04-22T23:23:30.851 INFO:tasks.workunit.client.0.smithi067.stdout:1/533: write d8/d12/d40/f44 [1107028,75019] 0 2024-04-22T23:23:30.851 INFO:tasks.workunit.client.0.smithi067.stdout:1/534: chown d8 205 1 2024-04-22T23:23:30.853 INFO:tasks.workunit.client.0.smithi067.stdout:1/535: mknod d8/d9/d2d/d4d/d48/d9a/cb3 0 2024-04-22T23:23:30.855 INFO:tasks.workunit.client.0.smithi067.stdout:1/536: mknod d8/d9/d2f/d33/d5c/cb4 0 2024-04-22T23:23:30.856 INFO:tasks.workunit.client.0.smithi067.stdout:1/537: rmdir d8/d9/d2d 39 2024-04-22T23:23:30.859 INFO:tasks.workunit.client.0.smithi067.stdout:1/538: creat d8/d9/d2d/d4d/d70/d34/fb5 x:0 0 0 2024-04-22T23:23:30.859 INFO:tasks.workunit.client.0.smithi067.stdout:1/539: dread - d8/d9/d2f/d59/f81 zero size 2024-04-22T23:23:30.862 INFO:tasks.workunit.client.0.smithi067.stdout:1/540: rename d8/d9/d2f/f38 to d8/d9/d2f/d33/fb6 0 2024-04-22T23:23:30.865 INFO:tasks.workunit.client.0.smithi067.stdout:1/541: dread d8/d9/d2d/f4f [0,4194304] 0 2024-04-22T23:23:30.867 INFO:tasks.workunit.client.0.smithi067.stdout:1/542: mknod d8/d9/d2d/d4d/d70/cb7 0 2024-04-22T23:23:30.868 INFO:tasks.workunit.client.0.smithi067.stdout:1/543: dread - d8/d9/d2f/d59/f81 zero size 2024-04-22T23:23:30.871 INFO:tasks.workunit.client.0.smithi067.stdout:1/544: link d8/d9/d2f/lb0 d8/d9/d2d/d4d/d70/d34/d60/lb8 0 2024-04-22T23:23:30.873 INFO:tasks.workunit.client.0.smithi067.stdout:1/545: write d8/d9/d2f/d33/d5c/f98 [630652,105233] 0 2024-04-22T23:23:30.874 INFO:tasks.workunit.client.0.smithi067.stdout:1/546: truncate d8/f4b 407586 0 2024-04-22T23:23:30.876 INFO:tasks.workunit.client.0.smithi067.stdout:1/547: write d8/d9/d2f/d31/f58 [805221,57467] 0 2024-04-22T23:23:30.878 INFO:tasks.workunit.client.0.smithi067.stdout:1/548: creat d8/d9/d2d/d4d/d83/fb9 x:0 0 0 2024-04-22T23:23:30.879 INFO:tasks.workunit.client.0.smithi067.stdout:1/549: chown d8/f4b 1737009 1 2024-04-22T23:23:30.880 INFO:tasks.workunit.client.0.smithi067.stdout:1/550: creat d8/d12/d36/fba x:0 0 0 2024-04-22T23:23:30.882 INFO:tasks.workunit.client.0.smithi067.stdout:1/551: symlink d8/d9/d2f/d31/lbb 0 2024-04-22T23:23:30.887 INFO:tasks.workunit.client.0.smithi067.stdout:1/552: dwrite d8/d9/d2d/d4d/d70/d34/f7e [0,4194304] 0 2024-04-22T23:23:30.889 INFO:tasks.workunit.client.0.smithi067.stdout:1/553: dread - d8/d9/d2d/d4d/d48/d9a/fac zero size 2024-04-22T23:23:30.893 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:30 smithi067 ceph-mon[26453]: pgmap v270: 65 pgs: 65 active+clean; 1.1 GiB data, 3.1 GiB used, 533 GiB / 536 GiB avail; 27 MiB/s rd, 132 MiB/s wr, 310 op/s 2024-04-22T23:23:30.893 INFO:tasks.workunit.client.0.smithi067.stdout:4/626: sync 2024-04-22T23:23:30.893 INFO:tasks.workunit.client.0.smithi067.stdout:0/584: sync 2024-04-22T23:23:30.894 INFO:tasks.workunit.client.0.smithi067.stdout:4/627: stat d12/d42 0 2024-04-22T23:23:30.899 INFO:tasks.workunit.client.0.smithi067.stdout:0/585: fdatasync db/d9a/d56/d70/dba/fc0 0 2024-04-22T23:23:30.900 INFO:tasks.workunit.client.0.smithi067.stdout:0/586: fsync db/d2e/f7c 0 2024-04-22T23:23:30.901 INFO:tasks.workunit.client.0.smithi067.stdout:0/587: stat db/d2e/d46/c88 0 2024-04-22T23:23:30.903 INFO:tasks.workunit.client.0.smithi067.stdout:0/588: symlink db/d9a/d56/d6b/lc7 0 2024-04-22T23:23:30.904 INFO:tasks.workunit.client.0.smithi067.stdout:4/628: write d12/d42/d4b/fbf [5213563,46133] 0 2024-04-22T23:23:30.905 INFO:tasks.workunit.client.0.smithi067.stdout:4/629: fdatasync d12/d17/d47/f6c 0 2024-04-22T23:23:30.906 INFO:tasks.workunit.client.0.smithi067.stdout:4/630: dread - d12/d17/d88/fdc zero size 2024-04-22T23:23:30.908 INFO:tasks.workunit.client.0.smithi067.stdout:4/631: unlink fe 0 2024-04-22T23:23:30.909 INFO:tasks.workunit.client.0.smithi067.stdout:0/589: getdents db/de/d2a 0 2024-04-22T23:23:30.909 INFO:tasks.workunit.client.0.smithi067.stdout:4/632: readlink d12/d17/l37 0 2024-04-22T23:23:30.909 INFO:tasks.workunit.client.0.smithi067.stdout:0/590: readlink db/de/d2a/l81 0 2024-04-22T23:23:30.913 INFO:tasks.workunit.client.0.smithi067.stdout:4/633: getdents d12/d17/d3f/d7a 0 2024-04-22T23:23:30.915 INFO:tasks.workunit.client.0.smithi067.stdout:4/634: creat d12/d5f/dae/fdd x:0 0 0 2024-04-22T23:23:30.915 INFO:tasks.workunit.client.0.smithi067.stdout:4/635: fdatasync d12/d5f/f35 0 2024-04-22T23:23:30.941 INFO:tasks.workunit.client.0.smithi067.stdout:4/636: dwrite d12/d5f/d2b/f72 [0,4194304] 0 2024-04-22T23:23:30.945 INFO:tasks.workunit.client.0.smithi067.stdout:4/637: symlink d12/d2e/lde 0 2024-04-22T23:23:30.948 INFO:tasks.workunit.client.0.smithi067.stdout:4/638: mknod d12/d17/d31/cdf 0 2024-04-22T23:23:30.950 INFO:tasks.workunit.client.0.smithi067.stdout:4/639: read - d12/d42/d4b/d70/d8e/fd2 zero size 2024-04-22T23:23:30.950 INFO:tasks.workunit.client.0.smithi067.stdout:4/640: stat d12/d17/d3f/fbe 0 2024-04-22T23:23:30.963 INFO:tasks.workunit.client.0.smithi067.stdout:4/641: mknod d12/d5f/d69/ce0 0 2024-04-22T23:23:30.964 INFO:tasks.workunit.client.0.smithi067.stdout:4/642: chown d12/f8d 29 1 2024-04-22T23:23:30.966 INFO:tasks.workunit.client.0.smithi067.stdout:4/643: creat d12/d42/dd3/dd8/fe1 x:0 0 0 2024-04-22T23:23:30.967 INFO:tasks.workunit.client.0.smithi067.stdout:4/644: fsync d12/d42/d4b/d70/d8e/fcf 0 2024-04-22T23:23:30.969 INFO:tasks.workunit.client.0.smithi067.stdout:4/645: truncate d12/d17/f2f 1105611 0 2024-04-22T23:23:30.969 INFO:tasks.workunit.client.0.smithi067.stdout:4/646: stat d12/d42/d4b/d70/cb3 0 2024-04-22T23:23:30.971 INFO:tasks.workunit.client.0.smithi067.stdout:4/647: creat d12/d42/dd3/dd8/fe2 x:0 0 0 2024-04-22T23:23:30.976 INFO:tasks.workunit.client.0.smithi067.stdout:4/648: dwrite d12/f8d [0,4194304] 0 2024-04-22T23:23:30.977 INFO:tasks.workunit.client.0.smithi067.stdout:4/649: symlink d12/d5f/d7e/le3 0 2024-04-22T23:23:30.980 INFO:tasks.workunit.client.0.smithi067.stdout:4/650: mknod d12/d42/d4b/d70/ce4 0 2024-04-22T23:23:30.983 INFO:tasks.workunit.client.0.smithi067.stdout:4/651: creat d12/d17/d47/fe5 x:0 0 0 2024-04-22T23:23:30.984 INFO:tasks.workunit.client.0.smithi067.stdout:4/652: chown d12/f8d 56955160 1 2024-04-22T23:23:30.989 INFO:tasks.workunit.client.0.smithi067.stdout:4/653: dwrite d12/f2a [0,4194304] 0 2024-04-22T23:23:30.991 INFO:tasks.workunit.client.0.smithi067.stdout:4/654: creat d12/d5f/fe6 x:0 0 0 2024-04-22T23:23:30.994 INFO:tasks.workunit.client.0.smithi067.stdout:4/655: mkdir d12/d17/d47/de7 0 2024-04-22T23:23:30.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:30 smithi134 ceph-mon[34609]: pgmap v270: 65 pgs: 65 active+clean; 1.1 GiB data, 3.1 GiB used, 533 GiB / 536 GiB avail; 27 MiB/s rd, 132 MiB/s wr, 310 op/s 2024-04-22T23:23:31.002 INFO:tasks.workunit.client.0.smithi067.stdout:4/656: write d12/d2e/f64 [212536,1248] 0 2024-04-22T23:23:31.004 INFO:tasks.workunit.client.0.smithi067.stdout:4/657: dread - d12/d42/d4b/d70/d8e/d56/f5c zero size 2024-04-22T23:23:31.004 INFO:tasks.workunit.client.0.smithi067.stdout:4/658: fdatasync d12/d5f/dae/fc6 0 2024-04-22T23:23:31.006 INFO:tasks.workunit.client.0.smithi067.stdout:4/659: write d12/d5f/d2b/db6/f95 [931537,46811] 0 2024-04-22T23:23:31.006 INFO:tasks.workunit.client.0.smithi067.stdout:4/660: write d12/f8d [4492938,106170] 0 2024-04-22T23:23:31.009 INFO:tasks.workunit.client.0.smithi067.stdout:4/661: rmdir d12/d17/d47 39 2024-04-22T23:23:31.011 INFO:tasks.workunit.client.0.smithi067.stdout:4/662: creat d12/d42/d4b/d77/fe8 x:0 0 0 2024-04-22T23:23:31.018 INFO:tasks.workunit.client.0.smithi067.stdout:4/663: dwrite d12/d5f/dae/fdd [0,4194304] 0 2024-04-22T23:23:31.020 INFO:tasks.workunit.client.0.smithi067.stdout:4/664: rmdir d12/d17/d3f 39 2024-04-22T23:23:31.022 INFO:tasks.workunit.client.0.smithi067.stdout:4/665: symlink d12/d42/d4b/d77/le9 0 2024-04-22T23:23:31.024 INFO:tasks.workunit.client.0.smithi067.stdout:4/666: write d12/d42/d4b/d77/fe8 [786976,90700] 0 2024-04-22T23:23:31.024 INFO:tasks.workunit.client.0.smithi067.stdout:4/667: dread - d12/d5f/faa zero size 2024-04-22T23:23:31.026 INFO:tasks.workunit.client.0.smithi067.stdout:4/668: dread d12/d17/fb9 [0,4194304] 0 2024-04-22T23:23:31.026 INFO:tasks.workunit.client.0.smithi067.stdout:4/669: stat d12/d42/d4b/d77/d9d/dc1 0 2024-04-22T23:23:31.028 INFO:tasks.workunit.client.0.smithi067.stdout:4/670: write d12/d42/d4b/d70/d8e/d56/d8b/dc3/fd6 [974138,12987] 0 2024-04-22T23:23:31.028 INFO:tasks.workunit.client.0.smithi067.stdout:4/671: chown d12/d2e/fcb 7250 1 2024-04-22T23:23:31.029 INFO:tasks.workunit.client.0.smithi067.stdout:4/672: write d12/d5f/faa [257411,105619] 0 2024-04-22T23:23:31.031 INFO:tasks.workunit.client.0.smithi067.stdout:4/673: creat d12/d5f/d69/fea x:0 0 0 2024-04-22T23:23:31.039 INFO:tasks.workunit.client.0.smithi067.stdout:4/674: dwrite f5 [4194304,4194304] 0 2024-04-22T23:23:31.040 INFO:tasks.workunit.client.0.smithi067.stdout:4/675: write d12/d2e/fb7 [280347,118480] 0 2024-04-22T23:23:31.044 INFO:tasks.workunit.client.0.smithi067.stdout:4/676: creat d12/d42/d4b/d77/d8c/feb x:0 0 0 2024-04-22T23:23:31.046 INFO:tasks.workunit.client.0.smithi067.stdout:4/677: stat d12/d17/d47/c92 0 2024-04-22T23:23:31.046 INFO:tasks.workunit.client.0.smithi067.stdout:4/678: chown d12/d17/l1b 10 1 2024-04-22T23:23:31.047 INFO:tasks.workunit.client.0.smithi067.stdout:4/679: write d12/d2e/f3e [3378286,113303] 0 2024-04-22T23:23:31.047 INFO:tasks.workunit.client.0.smithi067.stdout:4/680: readlink d12/d5f/d69/la1 0 2024-04-22T23:23:31.048 INFO:tasks.workunit.client.0.smithi067.stdout:4/681: stat d12/d42/d4b/d70/d8e 0 2024-04-22T23:23:31.055 INFO:tasks.workunit.client.0.smithi067.stdout:4/682: dread d12/d2e/f64 [0,4194304] 0 2024-04-22T23:23:31.057 INFO:tasks.workunit.client.0.smithi067.stdout:4/683: rmdir d12/d17/d3f 39 2024-04-22T23:23:31.058 INFO:tasks.workunit.client.0.smithi067.stdout:4/684: symlink d12/d42/d4b/d77/d9d/dc1/lec 0 2024-04-22T23:23:31.059 INFO:tasks.workunit.client.0.smithi067.stdout:4/685: readlink d12/d5f/dae/lda 0 2024-04-22T23:23:31.062 INFO:tasks.workunit.client.0.smithi067.stdout:4/686: rename d12/d42/d4b/d70 to d12/d5f/d2b/ded 0 2024-04-22T23:23:31.064 INFO:tasks.workunit.client.0.smithi067.stdout:4/687: creat d12/d42/d4b/d77/fee x:0 0 0 2024-04-22T23:23:31.065 INFO:tasks.workunit.client.0.smithi067.stdout:4/688: chown d12/d5f/d69/fa9 7842 1 2024-04-22T23:23:31.067 INFO:tasks.workunit.client.0.smithi067.stdout:4/689: link d12/d2e/f3e d12/d42/d4b/d77/d9d/fef 0 2024-04-22T23:23:31.070 INFO:tasks.workunit.client.0.smithi067.stdout:4/690: creat d12/d17/d3f/ff0 x:0 0 0 2024-04-22T23:23:31.071 INFO:tasks.workunit.client.0.smithi067.stdout:4/691: readlink d12/d42/d4b/l82 0 2024-04-22T23:23:31.075 INFO:tasks.workunit.client.0.smithi067.stdout:4/692: rmdir d12/d42/d4b 39 2024-04-22T23:23:31.089 INFO:tasks.workunit.client.0.smithi067.stdout:4/693: dread d12/d5f/f55 [0,4194304] 0 2024-04-22T23:23:31.090 INFO:tasks.workunit.client.0.smithi067.stdout:4/694: creat d12/d5f/d7e/ff1 x:0 0 0 2024-04-22T23:23:31.091 INFO:tasks.workunit.client.0.smithi067.stdout:4/695: readlink d12/d2e/d9b/lb1 0 2024-04-22T23:23:31.092 INFO:tasks.workunit.client.0.smithi067.stdout:4/696: write d12/d5f/d7e/ff1 [346986,100116] 0 2024-04-22T23:23:31.092 INFO:tasks.workunit.client.0.smithi067.stdout:9/551: sync 2024-04-22T23:23:31.094 INFO:tasks.workunit.client.0.smithi067.stdout:9/552: rmdir d5/dd/d18/d3c/d40 39 2024-04-22T23:23:31.096 INFO:tasks.workunit.client.0.smithi067.stdout:4/697: rename d12/d42/d4b/d77/d9d/lb0 to d12/d5f/d7e/lf2 0 2024-04-22T23:23:31.097 INFO:tasks.workunit.client.0.smithi067.stdout:9/553: creat d5/d21/d47/fbd x:0 0 0 2024-04-22T23:23:31.101 INFO:tasks.workunit.client.0.smithi067.stdout:9/554: rename d5/d21/d47/d57/f85 to d5/d21/d47/d56/d8c/fbe 0 2024-04-22T23:23:31.102 INFO:tasks.workunit.client.0.smithi067.stdout:9/555: write d5/dd/d18/d39/f64 [1166806,117390] 0 2024-04-22T23:23:31.105 INFO:tasks.workunit.client.0.smithi067.stdout:4/698: truncate d12/f8d 1443207 0 2024-04-22T23:23:31.106 INFO:tasks.workunit.client.0.smithi067.stdout:9/556: dread d5/dd/d18/d39/d3e/d3f/f73 [0,4194304] 0 2024-04-22T23:23:31.107 INFO:tasks.workunit.client.0.smithi067.stdout:9/557: stat d5/dd/d18/d39/d49/d54/c38 0 2024-04-22T23:23:31.110 INFO:tasks.workunit.client.0.smithi067.stdout:9/558: rmdir d5/dd/d18/d39/d3e/d3f 39 2024-04-22T23:23:31.112 INFO:tasks.workunit.client.0.smithi067.stdout:9/559: mknod d5/dd/d18/d39/d8f/cbf 0 2024-04-22T23:23:31.113 INFO:tasks.workunit.client.0.smithi067.stdout:9/560: chown d5/d21/d2c/f7a 0 1 2024-04-22T23:23:31.118 INFO:tasks.workunit.client.0.smithi067.stdout:9/561: dwrite d5/d21/d47/fa4 [0,4194304] 0 2024-04-22T23:23:31.120 INFO:tasks.workunit.client.0.smithi067.stdout:4/699: dread d12/d5f/f28 [4194304,4194304] 0 2024-04-22T23:23:31.123 INFO:tasks.workunit.client.0.smithi067.stdout:4/700: write d12/d5f/d7e/fc9 [666415,112694] 0 2024-04-22T23:23:31.124 INFO:tasks.workunit.client.0.smithi067.stdout:9/562: getdents d5/dd/d18/d39/d3e/d3f/d63 0 2024-04-22T23:23:31.126 INFO:tasks.workunit.client.0.smithi067.stdout:9/563: write d5/d21/d47/fa4 [6041886,110367] 0 2024-04-22T23:23:31.128 INFO:tasks.workunit.client.0.smithi067.stdout:4/701: getdents d12/d42 0 2024-04-22T23:23:31.128 INFO:tasks.workunit.client.0.smithi067.stdout:4/702: chown f5 21342 1 2024-04-22T23:23:31.129 INFO:tasks.workunit.client.0.smithi067.stdout:9/564: mkdir d5/d21/d47/d57/dc0 0 2024-04-22T23:23:31.129 INFO:tasks.workunit.client.0.smithi067.stdout:4/703: chown d12/d17/d47/f86 7756 1 2024-04-22T23:23:31.133 INFO:tasks.workunit.client.0.smithi067.stdout:9/565: rename d5/dd/d18/d39/d49/d54/d99/fa1 to d5/dd/d18/d39/d8f/dba/fc1 0 2024-04-22T23:23:31.134 INFO:tasks.workunit.client.0.smithi067.stdout:9/566: dread - d5/dd/d18/d5c/fab zero size 2024-04-22T23:23:31.134 INFO:tasks.workunit.client.0.smithi067.stdout:4/704: link d12/d17/fb9 d12/d2e/ff3 0 2024-04-22T23:23:31.138 INFO:tasks.workunit.client.0.smithi067.stdout:9/567: creat d5/d21/d47/d57/dc0/fc2 x:0 0 0 2024-04-22T23:23:31.139 INFO:tasks.workunit.client.0.smithi067.stdout:9/568: truncate d5/d21/d47/fbd 280985 0 2024-04-22T23:23:31.145 INFO:tasks.workunit.client.0.smithi067.stdout:9/569: dwrite d5/d21/d47/d57/f97 [0,4194304] 0 2024-04-22T23:23:31.150 INFO:tasks.workunit.client.0.smithi067.stdout:4/705: dread d12/d17/d47/f86 [0,4194304] 0 2024-04-22T23:23:31.151 INFO:tasks.workunit.client.0.smithi067.stdout:9/570: link d5/dd/d18/d39/d4d/faa d5/d21/d47/d57/dc0/fc3 0 2024-04-22T23:23:31.153 INFO:tasks.workunit.client.0.smithi067.stdout:9/571: creat d5/dd/da9/fc4 x:0 0 0 2024-04-22T23:23:31.155 INFO:tasks.workunit.client.0.smithi067.stdout:4/706: truncate d12/d5f/dae/fc6 1151800 0 2024-04-22T23:23:31.160 INFO:tasks.workunit.client.0.smithi067.stdout:9/572: dwrite d5/fa [0,4194304] 0 2024-04-22T23:23:31.163 INFO:tasks.workunit.client.0.smithi067.stdout:4/707: dwrite d12/f2a [4194304,4194304] 0 2024-04-22T23:23:31.164 INFO:tasks.workunit.client.0.smithi067.stdout:9/573: rmdir d5/d21/d2c 39 2024-04-22T23:23:31.218 INFO:tasks.workunit.client.0.smithi067.stdout:4/708: truncate d12/d5f/d2b/ded/d8e/f68 2951243 0 2024-04-22T23:23:31.218 INFO:tasks.workunit.client.0.smithi067.stdout:9/574: truncate d5/dd/f32 923322 0 2024-04-22T23:23:31.222 INFO:tasks.workunit.client.0.smithi067.stdout:9/575: dread - d5/dd/d18/d39/d49/f65 zero size 2024-04-22T23:23:31.230 INFO:tasks.workunit.client.0.smithi067.stdout:9/576: link d5/d21/d47/d56/f80 d5/d21/d47/fc5 0 2024-04-22T23:23:31.232 INFO:tasks.workunit.client.0.smithi067.stdout:9/577: chown d5/d21/d47/l4f 68 1 2024-04-22T23:23:31.241 INFO:tasks.workunit.client.0.smithi067.stdout:9/578: mkdir d5/d21/d47/d56/dc6 0 2024-04-22T23:23:31.242 INFO:tasks.workunit.client.0.smithi067.stdout:4/709: dread d12/d5f/d2b/ded/d8e/f68 [0,4194304] 0 2024-04-22T23:23:31.243 INFO:tasks.workunit.client.0.smithi067.stdout:4/710: write d12/d5f/d2b/db6/f9e [1948218,33164] 0 2024-04-22T23:23:31.246 INFO:tasks.workunit.client.0.smithi067.stdout:9/579: rename d5/l83 to d5/dd/d18/d39/d49/d54/lc7 0 2024-04-22T23:23:31.249 INFO:tasks.workunit.client.0.smithi067.stdout:4/711: rename d12/d42/dd3/dd8/fe1 to d12/d5f/dae/dd0/ff4 0 2024-04-22T23:23:31.253 INFO:tasks.workunit.client.0.smithi067.stdout:9/580: creat d5/dd/d18/d3c/d40/fc8 x:0 0 0 2024-04-22T23:23:31.261 INFO:tasks.workunit.client.0.smithi067.stdout:4/712: write d12/d5f/dae/fc6 [656560,72981] 0 2024-04-22T23:23:31.264 INFO:tasks.workunit.client.0.smithi067.stdout:4/713: mknod d12/d42/d4b/d77/d9d/dc1/cf5 0 2024-04-22T23:23:31.265 INFO:tasks.workunit.client.0.smithi067.stdout:4/714: symlink d12/d5f/d2b/d97/lf6 0 2024-04-22T23:23:31.266 INFO:tasks.workunit.client.0.smithi067.stdout:4/715: fdatasync d12/d5f/d2b/ded/f6e 0 2024-04-22T23:23:31.272 INFO:tasks.workunit.client.0.smithi067.stdout:3/612: sync 2024-04-22T23:23:31.273 INFO:tasks.workunit.client.0.smithi067.stdout:3/613: dread - d3/dcb/fde zero size 2024-04-22T23:23:31.273 INFO:tasks.workunit.client.0.smithi067.stdout:4/716: dwrite d12/f8d [0,4194304] 0 2024-04-22T23:23:31.277 INFO:tasks.workunit.client.0.smithi067.stdout:3/614: symlink d3/d4/d10/d38/d53/d55/d74/ldf 0 2024-04-22T23:23:31.281 INFO:tasks.workunit.client.0.smithi067.stdout:3/615: creat d3/fe0 x:0 0 0 2024-04-22T23:23:31.281 INFO:tasks.workunit.client.0.smithi067.stdout:4/717: dwrite d12/d5f/d2b/ded/d8e/fcf [4194304,4194304] 0 2024-04-22T23:23:31.284 INFO:tasks.workunit.client.0.smithi067.stdout:3/616: unlink d3/d4/f16 0 2024-04-22T23:23:31.286 INFO:tasks.workunit.client.0.smithi067.stdout:4/718: rename d12/d5f/dae/fdd to d12/d42/ff7 0 2024-04-22T23:23:31.290 INFO:tasks.workunit.client.0.smithi067.stdout:4/719: unlink d12/d2e/lde 0 2024-04-22T23:23:31.291 INFO:tasks.workunit.client.0.smithi067.stdout:4/720: chown d12/d5f/fb5 317 1 2024-04-22T23:23:31.291 INFO:tasks.workunit.client.0.smithi067.stdout:3/617: truncate d3/d4/d12/d4b/f8f 1941697 0 2024-04-22T23:23:31.292 INFO:tasks.workunit.client.0.smithi067.stdout:4/721: dread - d12/d17/d3f/ff0 zero size 2024-04-22T23:23:31.295 INFO:tasks.workunit.client.0.smithi067.stdout:3/618: read d3/d4/fc [1729266,125011] 0 2024-04-22T23:23:31.302 INFO:tasks.workunit.client.0.smithi067.stdout:4/722: write d12/d42/dd3/dd8/fe2 [154850,37973] 0 2024-04-22T23:23:31.303 INFO:tasks.workunit.client.0.smithi067.stdout:4/723: chown d12/d5f/d69/ce0 21513345 1 2024-04-22T23:23:31.306 INFO:tasks.workunit.client.0.smithi067.stdout:4/724: chown d12/d42/d4b/d77/d8c/feb 52825 1 2024-04-22T23:23:31.312 INFO:tasks.workunit.client.0.smithi067.stdout:3/619: dread d3/d4/d12/f2f [0,4194304] 0 2024-04-22T23:23:31.312 INFO:tasks.workunit.client.0.smithi067.stdout:4/725: dwrite d12/d42/f78 [0,4194304] 0 2024-04-22T23:23:31.314 INFO:tasks.workunit.client.0.smithi067.stdout:3/620: read d3/d4/fc4 [712287,86184] 0 2024-04-22T23:23:31.315 INFO:tasks.workunit.client.0.smithi067.stdout:3/621: write d3/f6f [3672353,29488] 0 2024-04-22T23:23:31.316 INFO:tasks.workunit.client.0.smithi067.stdout:4/726: rmdir d12/d5f/d7e 39 2024-04-22T23:23:31.317 INFO:tasks.workunit.client.0.smithi067.stdout:4/727: read d12/d2e/f64 [662018,47986] 0 2024-04-22T23:23:31.318 INFO:tasks.workunit.client.0.smithi067.stdout:3/622: mknod d3/d4/d10/d38/d53/d55/ce1 0 2024-04-22T23:23:31.320 INFO:tasks.workunit.client.0.smithi067.stdout:4/728: unlink d12/d17/d3f/ff0 0 2024-04-22T23:23:31.322 INFO:tasks.workunit.client.0.smithi067.stdout:3/623: rename d3/fa to d3/d4/d10/d38/d53/d55/fe2 0 2024-04-22T23:23:31.325 INFO:tasks.workunit.client.0.smithi067.stdout:4/729: fsync d12/f54 0 2024-04-22T23:23:31.327 INFO:tasks.workunit.client.0.smithi067.stdout:3/624: mkdir d3/d4/d10/de3 0 2024-04-22T23:23:31.329 INFO:tasks.workunit.client.0.smithi067.stdout:3/625: write d3/d4/d10/d38/d53/f58 [1446133,55466] 0 2024-04-22T23:23:31.330 INFO:tasks.workunit.client.0.smithi067.stdout:4/730: rename d12/f16 to d12/d42/d4b/d77/ff8 0 2024-04-22T23:23:31.331 INFO:tasks.workunit.client.0.smithi067.stdout:3/626: read d3/d4/d1b/d93/fa0 [424535,5136] 0 2024-04-22T23:23:31.334 INFO:tasks.workunit.client.0.smithi067.stdout:3/627: truncate d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d90/da2/fb4 531675 0 2024-04-22T23:23:31.335 INFO:tasks.workunit.client.0.smithi067.stdout:4/731: symlink d12/d42/dd3/lf9 0 2024-04-22T23:23:31.337 INFO:tasks.workunit.client.0.smithi067.stdout:3/628: dread d3/d4/d10/d8a/fa6 [0,4194304] 0 2024-04-22T23:23:31.338 INFO:tasks.workunit.client.0.smithi067.stdout:4/732: fsync d12/d42/ff7 0 2024-04-22T23:23:31.340 INFO:tasks.workunit.client.0.smithi067.stdout:4/733: chown d12/d17/d47/f6c 62138 1 2024-04-22T23:23:31.340 INFO:tasks.workunit.client.0.smithi067.stdout:3/629: write d3/d4/d12/f1d [753064,123980] 0 2024-04-22T23:23:31.341 INFO:tasks.workunit.client.0.smithi067.stdout:4/734: write d12/d42/d4b/fa8 [220635,7038] 0 2024-04-22T23:23:31.342 INFO:tasks.workunit.client.0.smithi067.stdout:4/735: chown d12/d42/ff7 1 1 2024-04-22T23:23:31.342 INFO:tasks.workunit.client.0.smithi067.stdout:3/630: truncate d3/d4/d10/d38/d53/d55/f66 1540928 0 2024-04-22T23:23:31.347 INFO:tasks.workunit.client.0.smithi067.stdout:3/631: dwrite d3/d4/d10/d38/d53/d55/fad [0,4194304] 0 2024-04-22T23:23:31.351 INFO:tasks.workunit.client.0.smithi067.stdout:3/632: creat d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d90/da2/daf/fe4 x:0 0 0 2024-04-22T23:23:31.405 INFO:tasks.workunit.client.0.smithi067.stdout:3/633: write d3/d4/d10/d38/d53/d55/f75 [1383346,118638] 0 2024-04-22T23:23:31.407 INFO:tasks.workunit.client.0.smithi067.stdout:3/634: getdents d3/d4/d10/d38/d53/ddc 0 2024-04-22T23:23:31.411 INFO:tasks.workunit.client.0.smithi067.stdout:3/635: symlink d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d62/d95/dd6/le5 0 2024-04-22T23:23:31.413 INFO:tasks.workunit.client.0.smithi067.stdout:3/636: rename d3/d4/d10/d19/c1c to d3/ce6 0 2024-04-22T23:23:31.419 INFO:tasks.workunit.client.0.smithi067.stdout:3/637: dwrite d3/d4/d10/d38/d53/f7b [8388608,4194304] 0 2024-04-22T23:23:31.426 INFO:tasks.workunit.client.0.smithi067.stdout:3/638: dwrite d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d90/f9f [0,4194304] 0 2024-04-22T23:23:31.427 INFO:tasks.workunit.client.0.smithi067.stdout:3/639: read - d3/d4/d12/fae zero size 2024-04-22T23:23:31.429 INFO:tasks.workunit.client.0.smithi067.stdout:3/640: readlink d3/d4/d10/d1f/d3c/la5 0 2024-04-22T23:23:31.430 INFO:tasks.workunit.client.0.smithi067.stdout:3/641: rmdir d3/d4/d28/d4a 39 2024-04-22T23:23:31.431 INFO:tasks.workunit.client.0.smithi067.stdout:3/642: chown d3/d4/d10/d1f/d3c/c44 121845 1 2024-04-22T23:23:31.431 INFO:tasks.workunit.client.0.smithi067.stdout:3/643: fsync d3/d4/d12/d4b/fba 0 2024-04-22T23:23:31.433 INFO:tasks.workunit.client.0.smithi067.stdout:3/644: read d3/d4/d12/f1d [14449,7004] 0 2024-04-22T23:23:31.437 INFO:tasks.workunit.client.0.smithi067.stdout:3/645: mknod d3/d4/d10/d38/d53/ddc/ce7 0 2024-04-22T23:23:31.489 INFO:tasks.workunit.client.0.smithi067.stdout:6/602: sync 2024-04-22T23:23:31.489 INFO:tasks.workunit.client.0.smithi067.stdout:2/648: sync 2024-04-22T23:23:31.489 INFO:tasks.workunit.client.0.smithi067.stdout:5/638: sync 2024-04-22T23:23:31.489 INFO:tasks.workunit.client.0.smithi067.stdout:0/591: sync 2024-04-22T23:23:31.489 INFO:tasks.workunit.client.0.smithi067.stdout:1/554: sync 2024-04-22T23:23:31.489 INFO:tasks.workunit.client.0.smithi067.stdout:8/567: sync 2024-04-22T23:23:31.489 INFO:tasks.workunit.client.0.smithi067.stdout:7/485: sync 2024-04-22T23:23:31.491 INFO:tasks.workunit.client.0.smithi067.stdout:6/603: dread - d17/d39/d53/d90/d46/fa1 zero size 2024-04-22T23:23:31.492 INFO:tasks.workunit.client.0.smithi067.stdout:8/568: write d1/dd/d80/d6f/f34 [1928099,25104] 0 2024-04-22T23:23:31.492 INFO:tasks.workunit.client.0.smithi067.stdout:1/555: symlink d8/d9/d2d/lbc 0 2024-04-22T23:23:31.495 INFO:tasks.workunit.client.0.smithi067.stdout:5/639: mkdir d5/db/d12/dbb/d5f/d79/dba/de0 0 2024-04-22T23:23:31.497 INFO:tasks.workunit.client.0.smithi067.stdout:1/556: readlink d8/d9/d2f/d31/lbb 0 2024-04-22T23:23:31.499 INFO:tasks.workunit.client.0.smithi067.stdout:8/569: write d1/dd/d26/f6b [1542050,47249] 0 2024-04-22T23:23:31.499 INFO:tasks.workunit.client.0.smithi067.stdout:2/649: rename d1/d42/d5f/l96 to d1/d10/d72/d53/dba/ld4 0 2024-04-22T23:23:31.501 INFO:tasks.workunit.client.0.smithi067.stdout:1/557: fdatasync d8/d12/f94 0 2024-04-22T23:23:31.506 INFO:tasks.workunit.client.0.smithi067.stdout:2/650: creat d1/d19/d6f/fd5 x:0 0 0 2024-04-22T23:23:31.506 INFO:tasks.workunit.client.0.smithi067.stdout:7/486: dwrite d10/d14/d1d/d46/d6a/f70 [0,4194304] 0 2024-04-22T23:23:31.509 INFO:tasks.workunit.client.0.smithi067.stdout:0/592: dwrite db/d21/d29/d2c/f91 [0,4194304] 0 2024-04-22T23:23:31.512 INFO:tasks.workunit.client.0.smithi067.stdout:8/570: write d1/d3/d27/f58 [897566,56348] 0 2024-04-22T23:23:31.512 INFO:tasks.workunit.client.0.smithi067.stdout:1/558: symlink d8/d9/d2d/d4d/lbd 0 2024-04-22T23:23:31.514 INFO:tasks.workunit.client.0.smithi067.stdout:5/640: rename d5/db/d12/dbb/d61/d89/cd7 to d5/d28/daf/db4/ce1 0 2024-04-22T23:23:31.515 INFO:tasks.workunit.client.0.smithi067.stdout:6/604: dwrite d17/d39/d53/d90/d46/f68 [0,4194304] 0 2024-04-22T23:23:31.515 INFO:tasks.workunit.client.0.smithi067.stdout:8/571: readlink d1/d3/d10/l1c 0 2024-04-22T23:23:31.517 INFO:tasks.workunit.client.0.smithi067.stdout:7/487: mkdir d10/d18/d88/daf 0 2024-04-22T23:23:31.518 INFO:tasks.workunit.client.0.smithi067.stdout:1/559: dread - d8/d12/d40/f57 zero size 2024-04-22T23:23:31.519 INFO:tasks.workunit.client.0.smithi067.stdout:5/641: write d5/db/d12/d44/d47/f4f [2376565,72805] 0 2024-04-22T23:23:31.523 INFO:tasks.workunit.client.0.smithi067.stdout:8/572: mkdir d1/dd/d80/d6f/da8/db8 0 2024-04-22T23:23:31.526 INFO:tasks.workunit.client.0.smithi067.stdout:5/642: chown d5/db/d12/d44/d47/cad 3 1 2024-04-22T23:23:31.527 INFO:tasks.workunit.client.0.smithi067.stdout:0/593: dwrite db/f6a [4194304,4194304] 0 2024-04-22T23:23:31.529 INFO:tasks.workunit.client.0.smithi067.stdout:0/594: dread - db/de/d2a/f99 zero size 2024-04-22T23:23:31.531 INFO:tasks.workunit.client.0.smithi067.stdout:5/643: creat d5/d28/daf/fe2 x:0 0 0 2024-04-22T23:23:31.532 INFO:tasks.workunit.client.0.smithi067.stdout:7/488: dwrite d10/d14/d1d/d46/d50/f2d [0,4194304] 0 2024-04-22T23:23:31.532 INFO:tasks.workunit.client.0.smithi067.stdout:5/644: write d5/db/d35/f6c [2094890,104034] 0 2024-04-22T23:23:31.536 INFO:tasks.workunit.client.0.smithi067.stdout:5/645: creat d5/d28/daf/fe3 x:0 0 0 2024-04-22T23:23:31.537 INFO:tasks.workunit.client.0.smithi067.stdout:7/489: mkdir d10/d14/d1d/d26/d2c/d41/db0 0 2024-04-22T23:23:31.540 INFO:tasks.workunit.client.0.smithi067.stdout:7/490: write d10/d14/d1d/d26/d2c/d7a/fa5 [473365,12385] 0 2024-04-22T23:23:31.541 INFO:tasks.workunit.client.0.smithi067.stdout:8/573: dread d1/d3/d27/f81 [0,4194304] 0 2024-04-22T23:23:31.542 INFO:tasks.workunit.client.0.smithi067.stdout:7/491: fsync d10/d14/d1d/d26/f3a 0 2024-04-22T23:23:31.544 INFO:tasks.workunit.client.0.smithi067.stdout:7/492: dread - d10/d14/d1d/d46/d6a/f86 zero size 2024-04-22T23:23:31.545 INFO:tasks.workunit.client.0.smithi067.stdout:8/574: mkdir d1/dd/d80/db9 0 2024-04-22T23:23:31.545 INFO:tasks.workunit.client.0.smithi067.stdout:5/646: link d5/db/d12/d44/d47/cad d5/d28/d36/d40/ce4 0 2024-04-22T23:23:31.546 INFO:tasks.workunit.client.0.smithi067.stdout:7/493: write d10/d18/d35/d62/f94 [382774,105879] 0 2024-04-22T23:23:31.550 INFO:tasks.workunit.client.0.smithi067.stdout:5/647: unlink d5/d28/daf/fe2 0 2024-04-22T23:23:31.552 INFO:tasks.workunit.client.0.smithi067.stdout:7/494: mknod d10/d14/d1d/d46/cb1 0 2024-04-22T23:23:31.553 INFO:tasks.workunit.client.0.smithi067.stdout:8/575: dwrite d1/d3/f13 [0,4194304] 0 2024-04-22T23:23:31.558 INFO:tasks.workunit.client.0.smithi067.stdout:1/560: write d8/d9/fa9 [2051,24699] 0 2024-04-22T23:23:31.561 INFO:tasks.workunit.client.0.smithi067.stdout:8/576: creat d1/d3/d10/fba x:0 0 0 2024-04-22T23:23:31.562 INFO:tasks.workunit.client.0.smithi067.stdout:1/561: stat d8/d12/d36/fba 0 2024-04-22T23:23:31.564 INFO:tasks.workunit.client.0.smithi067.stdout:1/562: fdatasync d8/d12/f17 0 2024-04-22T23:23:31.564 INFO:tasks.workunit.client.0.smithi067.stdout:5/648: dread d5/db/d12/f2c [0,4194304] 0 2024-04-22T23:23:31.564 INFO:tasks.workunit.client.0.smithi067.stdout:6/605: write d17/d83/d9b/fa3 [1268154,92213] 0 2024-04-22T23:23:31.565 INFO:tasks.workunit.client.0.smithi067.stdout:7/495: dwrite d10/d18/d35/f5e [0,4194304] 0 2024-04-22T23:23:31.568 INFO:tasks.workunit.client.0.smithi067.stdout:5/649: rmdir d5/d3e 39 2024-04-22T23:23:31.570 INFO:tasks.workunit.client.0.smithi067.stdout:0/595: dread db/d21/d29/f43 [4194304,4194304] 0 2024-04-22T23:23:31.571 INFO:tasks.workunit.client.0.smithi067.stdout:1/563: mknod d8/d9/d2d/d4d/d70/d34/d5d/cbe 0 2024-04-22T23:23:31.573 INFO:tasks.workunit.client.0.smithi067.stdout:1/564: truncate d8/d9/d2f/d59/f78 897587 0 2024-04-22T23:23:31.573 INFO:tasks.workunit.client.0.smithi067.stdout:1/565: stat d8/d12/c3c 0 2024-04-22T23:23:31.574 INFO:tasks.workunit.client.0.smithi067.stdout:8/577: rename d1/d3/d5c/fb1 to d1/dd/fbb 0 2024-04-22T23:23:31.574 INFO:tasks.workunit.client.0.smithi067.stdout:5/650: rmdir d5/d28/d36/d40/d97 39 2024-04-22T23:23:31.575 INFO:tasks.workunit.client.0.smithi067.stdout:5/651: stat d5/db/d12/d8a/lb0 0 2024-04-22T23:23:31.576 INFO:tasks.workunit.client.0.smithi067.stdout:0/596: mknod db/d71/cc8 0 2024-04-22T23:23:31.578 INFO:tasks.workunit.client.0.smithi067.stdout:5/652: write d5/db/d35/f43 [695177,66279] 0 2024-04-22T23:23:31.584 INFO:tasks.workunit.client.0.smithi067.stdout:1/566: write d8/d9/f3b [1330349,124141] 0 2024-04-22T23:23:31.586 INFO:tasks.workunit.client.0.smithi067.stdout:8/578: rename d1/d3/d10/d3b/fa4 to d1/d3/d5c/d98/fbc 0 2024-04-22T23:23:31.586 INFO:tasks.workunit.client.0.smithi067.stdout:5/653: dwrite d5/d3a/f58 [0,4194304] 0 2024-04-22T23:23:31.595 INFO:tasks.workunit.client.0.smithi067.stdout:6/606: dread d17/d39/d53/f55 [0,4194304] 0 2024-04-22T23:23:31.597 INFO:tasks.workunit.client.0.smithi067.stdout:5/654: dread d5/db/d12/dbb/d61/d89/fb5 [0,4194304] 0 2024-04-22T23:23:31.597 INFO:tasks.workunit.client.0.smithi067.stdout:8/579: dwrite d1/dd/f95 [0,4194304] 0 2024-04-22T23:23:31.600 INFO:tasks.workunit.client.0.smithi067.stdout:8/580: write d1/dd/d26/d62/d94/fa3 [692212,83877] 0 2024-04-22T23:23:31.605 INFO:tasks.workunit.client.0.smithi067.stdout:7/496: truncate d10/d14/d1d/d26/f39 687612 0 2024-04-22T23:23:31.607 INFO:tasks.workunit.client.0.smithi067.stdout:7/497: chown d10/d18/l36 21218619 1 2024-04-22T23:23:31.607 INFO:tasks.workunit.client.0.smithi067.stdout:8/581: rename d1/dd/d26/c48 to d1/dd/d26/d62/d94/cbd 0 2024-04-22T23:23:31.609 INFO:tasks.workunit.client.0.smithi067.stdout:8/582: stat d1/d3/d27/f63 0 2024-04-22T23:23:31.611 INFO:tasks.workunit.client.0.smithi067.stdout:5/655: dwrite d5/f15 [0,4194304] 0 2024-04-22T23:23:31.611 INFO:tasks.workunit.client.0.smithi067.stdout:7/498: creat d10/d18/fb2 x:0 0 0 2024-04-22T23:23:31.611 INFO:tasks.workunit.client.0.smithi067.stdout:8/583: fsync d1/d3/f38 0 2024-04-22T23:23:31.612 INFO:tasks.workunit.client.0.smithi067.stdout:8/584: dread - d1/d41/f83 zero size 2024-04-22T23:23:31.614 INFO:tasks.workunit.client.0.smithi067.stdout:7/499: creat d10/d14/d1d/d26/d2c/d7a/fb3 x:0 0 0 2024-04-22T23:23:31.617 INFO:tasks.workunit.client.0.smithi067.stdout:7/500: chown d10/d14/d1d/d26/d2c/d7a/c95 555 1 2024-04-22T23:23:31.618 INFO:tasks.workunit.client.0.smithi067.stdout:5/656: link d5/d28/daf/dbf/dcf/lcc d5/d28/dae/le5 0 2024-04-22T23:23:31.620 INFO:tasks.workunit.client.0.smithi067.stdout:8/585: getdents d1/d3/d5c 0 2024-04-22T23:23:31.622 INFO:tasks.workunit.client.0.smithi067.stdout:5/657: unlink d5/d28/d36/f80 0 2024-04-22T23:23:31.625 INFO:tasks.workunit.client.0.smithi067.stdout:7/501: dwrite d10/d14/d1d/d26/d2c/f2e [0,4194304] 0 2024-04-22T23:23:31.626 INFO:tasks.workunit.client.0.smithi067.stdout:5/658: creat d5/d28/daf/dbf/dcf/db2/fe6 x:0 0 0 2024-04-22T23:23:31.627 INFO:tasks.workunit.client.0.smithi067.stdout:8/586: dwrite d1/dd/f53 [0,4194304] 0 2024-04-22T23:23:31.629 INFO:tasks.workunit.client.0.smithi067.stdout:7/502: mknod d10/d14/d1d/d46/cb4 0 2024-04-22T23:23:31.629 INFO:tasks.workunit.client.0.smithi067.stdout:5/659: readlink d5/d3e/d90/la5 0 2024-04-22T23:23:31.630 INFO:tasks.workunit.client.0.smithi067.stdout:7/503: stat d10/d14/d1d/d46/d50/f2d 0 2024-04-22T23:23:31.630 INFO:tasks.workunit.client.0.smithi067.stdout:5/660: chown d5/d3e/d90/la5 0 1 2024-04-22T23:23:31.633 INFO:tasks.workunit.client.0.smithi067.stdout:7/504: creat d10/d14/d1d/d26/d7e/fb5 x:0 0 0 2024-04-22T23:23:31.633 INFO:tasks.workunit.client.0.smithi067.stdout:7/505: stat fc 0 2024-04-22T23:23:31.634 INFO:tasks.workunit.client.0.smithi067.stdout:7/506: dread - d10/d14/d1d/d26/d2c/d41/f99 zero size 2024-04-22T23:23:31.635 INFO:tasks.workunit.client.0.smithi067.stdout:7/507: write d10/d14/f98 [804462,125794] 0 2024-04-22T23:23:31.636 INFO:tasks.workunit.client.0.smithi067.stdout:7/508: read d10/f6f [2218199,108768] 0 2024-04-22T23:23:31.641 INFO:tasks.workunit.client.0.smithi067.stdout:7/509: dwrite d10/d18/d35/f5e [0,4194304] 0 2024-04-22T23:23:31.643 INFO:tasks.workunit.client.0.smithi067.stdout:7/510: mknod d10/d18/d88/daf/cb6 0 2024-04-22T23:23:31.644 INFO:tasks.workunit.client.0.smithi067.stdout:9/581: sync 2024-04-22T23:23:31.645 INFO:tasks.workunit.client.0.smithi067.stdout:7/511: mknod d10/d14/d1d/d26/d2c/d7a/cb7 0 2024-04-22T23:23:31.646 INFO:tasks.workunit.client.0.smithi067.stdout:9/582: creat d5/dd/d18/d39/d49/d54/fc9 x:0 0 0 2024-04-22T23:23:31.647 INFO:tasks.workunit.client.0.smithi067.stdout:7/512: creat d10/d14/d1d/d69/fb8 x:0 0 0 2024-04-22T23:23:31.649 INFO:tasks.workunit.client.0.smithi067.stdout:7/513: chown d10/d14/d1d/d26/d2c/d41/f90 338452 1 2024-04-22T23:23:31.650 INFO:tasks.workunit.client.0.smithi067.stdout:9/583: creat d5/fca x:0 0 0 2024-04-22T23:23:31.650 INFO:tasks.workunit.client.0.smithi067.stdout:7/514: write d10/d14/d1d/d46/d50/f30 [347829,62789] 0 2024-04-22T23:23:31.651 INFO:tasks.workunit.client.0.smithi067.stdout:9/584: write d5/dd/d18/d39/d49/d54/f79 [4424451,17518] 0 2024-04-22T23:23:31.652 INFO:tasks.workunit.client.0.smithi067.stdout:9/585: chown d5/dd/d18/d39/d49 51413219 1 2024-04-22T23:23:31.654 INFO:tasks.workunit.client.0.smithi067.stdout:7/515: rename d10/d18/f25 to d10/d14/d1d/d26/d7e/fb9 0 2024-04-22T23:23:31.657 INFO:tasks.workunit.client.0.smithi067.stdout:7/516: rename d10/d14/d1d/d26/d2c/d41/f90 to d10/d14/d1d/d69/fba 0 2024-04-22T23:23:31.658 INFO:tasks.workunit.client.0.smithi067.stdout:7/517: write d10/d14/d1d/d26/d2c/f3f [4726415,51157] 0 2024-04-22T23:23:31.660 INFO:tasks.workunit.client.0.smithi067.stdout:7/518: creat d10/d14/d1d/d26/d80/fbb x:0 0 0 2024-04-22T23:23:31.661 INFO:tasks.workunit.client.0.smithi067.stdout:7/519: dread - d10/d14/d1d/d24/f67 zero size 2024-04-22T23:23:31.664 INFO:tasks.workunit.client.0.smithi067.stdout:7/520: rename d10/d14/d1d/d26/d2c/d7a/c95 to d10/d14/d1d/d26/d2c/d41/cbc 0 2024-04-22T23:23:31.665 INFO:tasks.workunit.client.0.smithi067.stdout:7/521: write d10/d14/d1d/d26/d2c/d7a/fa5 [184479,16568] 0 2024-04-22T23:23:31.670 INFO:tasks.workunit.client.0.smithi067.stdout:7/522: dwrite d10/d18/f27 [0,4194304] 0 2024-04-22T23:23:31.671 INFO:tasks.workunit.client.0.smithi067.stdout:7/523: chown d10/d14/d1d/d26/d2c/d7a/fb3 59 1 2024-04-22T23:23:31.672 INFO:tasks.workunit.client.0.smithi067.stdout:7/524: dread - d10/d47/d49/f8b zero size 2024-04-22T23:23:31.675 INFO:tasks.workunit.client.0.smithi067.stdout:7/525: creat d10/d14/d1d/d26/fbd x:0 0 0 2024-04-22T23:23:31.676 INFO:tasks.workunit.client.0.smithi067.stdout:7/526: mknod d10/d18/d88/cbe 0 2024-04-22T23:23:31.681 INFO:tasks.workunit.client.0.smithi067.stdout:7/527: dwrite d10/d14/d1d/d46/d6a/f86 [0,4194304] 0 2024-04-22T23:23:31.684 INFO:tasks.workunit.client.0.smithi067.stdout:7/528: mknod d10/d14/d1d/d26/d2c/cbf 0 2024-04-22T23:23:31.689 INFO:tasks.workunit.client.0.smithi067.stdout:7/529: dwrite d10/d14/d1d/d26/d2c/d41/fa1 [0,4194304] 0 2024-04-22T23:23:31.690 INFO:tasks.workunit.client.0.smithi067.stdout:7/530: chown d10/d14/d1d/fa9 133011770 1 2024-04-22T23:23:31.693 INFO:tasks.workunit.client.0.smithi067.stdout:7/531: mkdir d10/d14/d1d/d26/d7e/dc0 0 2024-04-22T23:23:31.699 INFO:tasks.workunit.client.0.smithi067.stdout:7/532: dwrite d10/d14/d1d/d26/d2c/d41/fa1 [0,4194304] 0 2024-04-22T23:23:31.700 INFO:tasks.workunit.client.0.smithi067.stdout:7/533: fdatasync d10/f8f 0 2024-04-22T23:23:31.707 INFO:tasks.workunit.client.0.smithi067.stdout:7/534: dwrite d10/d14/d1d/d26/d7e/fb5 [0,4194304] 0 2024-04-22T23:23:31.715 INFO:tasks.workunit.client.0.smithi067.stdout:7/535: dwrite d10/d14/d1d/d26/d2c/d7a/f87 [0,4194304] 0 2024-04-22T23:23:31.719 INFO:tasks.workunit.client.0.smithi067.stdout:7/536: link fb d10/d14/d1d/d26/fc1 0 2024-04-22T23:23:31.721 INFO:tasks.workunit.client.0.smithi067.stdout:7/537: mkdir d10/d14/d1d/d26/d2c/d7a/dc2 0 2024-04-22T23:23:31.721 INFO:tasks.workunit.client.0.smithi067.stdout:7/538: readlink d10/d14/d1d/d46/d6a/l74 0 2024-04-22T23:23:31.722 INFO:tasks.workunit.client.0.smithi067.stdout:7/539: readlink d10/d14/d1d/d26/d2c/lac 0 2024-04-22T23:23:31.726 INFO:tasks.workunit.client.0.smithi067.stdout:7/540: getdents d10/d47 0 2024-04-22T23:23:31.727 INFO:tasks.workunit.client.0.smithi067.stdout:7/541: symlink d10/d14/d1d/d26/d2c/d41/lc3 0 2024-04-22T23:23:31.729 INFO:tasks.workunit.client.0.smithi067.stdout:7/542: symlink d10/d14/d1d/d69/lc4 0 2024-04-22T23:23:31.731 INFO:tasks.workunit.client.0.smithi067.stdout:7/543: mknod d10/d14/d1d/d26/d2c/d41/db0/cc5 0 2024-04-22T23:23:31.732 INFO:tasks.workunit.client.0.smithi067.stdout:7/544: read d10/d14/d1d/d26/d2c/d41/fa1 [4111314,87117] 0 2024-04-22T23:23:31.746 INFO:tasks.workunit.client.0.smithi067.stdout:8/587: read d1/d3/d5c/d98/fa1 [1995583,80291] 0 2024-04-22T23:23:31.747 INFO:tasks.workunit.client.0.smithi067.stdout:8/588: dread - d1/dd/d80/d6f/fad zero size 2024-04-22T23:23:31.752 INFO:tasks.workunit.client.0.smithi067.stdout:8/589: rename d1/d3/d10/f29 to d1/d3/d10/fbe 0 2024-04-22T23:23:31.758 INFO:tasks.workunit.client.0.smithi067.stdout:3/646: sync 2024-04-22T23:23:31.758 INFO:tasks.workunit.client.0.smithi067.stdout:2/651: sync 2024-04-22T23:23:31.758 INFO:tasks.workunit.client.0.smithi067.stdout:4/736: sync 2024-04-22T23:23:31.759 INFO:tasks.workunit.client.0.smithi067.stdout:2/652: readlink d1/d42/d61/l65 0 2024-04-22T23:23:31.759 INFO:tasks.workunit.client.0.smithi067.stdout:3/647: chown d3/fc1 99280 1 2024-04-22T23:23:31.762 INFO:tasks.workunit.client.0.smithi067.stdout:4/737: mknod d12/d42/d4b/d77/d9d/dc1/cfa 0 2024-04-22T23:23:31.763 INFO:tasks.workunit.client.0.smithi067.stdout:4/738: dread - d12/d17/d88/fdc zero size 2024-04-22T23:23:31.765 INFO:tasks.workunit.client.0.smithi067.stdout:8/590: read d1/dd/d80/d6f/f32 [174456,108328] 0 2024-04-22T23:23:31.769 INFO:tasks.workunit.client.0.smithi067.stdout:3/648: rename d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d51/l6d to d3/d4/d12/d4b/le8 0 2024-04-22T23:23:31.771 INFO:tasks.workunit.client.0.smithi067.stdout:3/649: stat d3/d4/d10/d38/d53/d55/d74/dd5/ddb/l72 0 2024-04-22T23:23:31.775 INFO:tasks.workunit.client.0.smithi067.stdout:3/650: creat d3/d4/d10/d38/d53/d55/d5c/d70/fe9 x:0 0 0 2024-04-22T23:23:31.800 INFO:tasks.workunit.client.0.smithi067.stdout:6/607: sync 2024-04-22T23:23:31.800 INFO:tasks.workunit.client.0.smithi067.stdout:9/586: dread d5/dd/d18/d39/d4d/f9e [4194304,4194304] 0 2024-04-22T23:23:31.800 INFO:tasks.workunit.client.0.smithi067.stdout:6/608: write d17/d39/d3a/f61 [3403130,25777] 0 2024-04-22T23:23:31.802 INFO:tasks.workunit.client.0.smithi067.stdout:6/609: write d17/d39/d53/d90/d46/f5f [5010825,91251] 0 2024-04-22T23:23:31.804 INFO:tasks.workunit.client.0.smithi067.stdout:1/567: write d8/f11 [1649426,115730] 0 2024-04-22T23:23:31.805 INFO:tasks.workunit.client.0.smithi067.stdout:6/610: stat d17/d39/d53/d90/d46/db0 0 2024-04-22T23:23:31.805 INFO:tasks.workunit.client.0.smithi067.stdout:0/597: truncate db/fae 3715935 0 2024-04-22T23:23:31.814 INFO:tasks.workunit.client.0.smithi067.stdout:4/739: dread d12/d5f/d69/f81 [0,4194304] 0 2024-04-22T23:23:31.815 INFO:tasks.workunit.client.0.smithi067.stdout:0/598: dwrite db/de/d11/d1a/fbe [0,4194304] 0 2024-04-22T23:23:31.819 INFO:tasks.workunit.client.0.smithi067.stdout:6/611: getdents d17/d39/d53 0 2024-04-22T23:23:31.822 INFO:tasks.workunit.client.0.smithi067.stdout:0/599: unlink db/d71/f93 0 2024-04-22T23:23:31.825 INFO:tasks.workunit.client.0.smithi067.stdout:0/600: read db/d21/f8d [3727416,9858] 0 2024-04-22T23:23:31.828 INFO:tasks.workunit.client.0.smithi067.stdout:0/601: symlink db/d60/d9e/lc9 0 2024-04-22T23:23:31.829 INFO:tasks.workunit.client.0.smithi067.stdout:6/612: dwrite d17/d83/f86 [4194304,4194304] 0 2024-04-22T23:23:31.829 INFO:tasks.workunit.client.0.smithi067.stdout:0/602: stat db/d21/d29/d2d/l57 0 2024-04-22T23:23:31.830 INFO:tasks.workunit.client.0.smithi067.stdout:6/613: fsync d17/d39/d53/d90/d46/f9f 0 2024-04-22T23:23:31.833 INFO:tasks.workunit.client.0.smithi067.stdout:0/603: creat db/d9a/d56/fca x:0 0 0 2024-04-22T23:23:31.834 INFO:tasks.workunit.client.0.smithi067.stdout:0/604: dread - db/d21/d29/d75/fa1 zero size 2024-04-22T23:23:31.834 INFO:tasks.workunit.client.0.smithi067.stdout:6/614: creat d17/d39/d53/d90/d46/db0/fc5 x:0 0 0 2024-04-22T23:23:31.836 INFO:tasks.workunit.client.0.smithi067.stdout:5/661: write d5/fa4 [823852,126915] 0 2024-04-22T23:23:31.842 INFO:tasks.workunit.client.0.smithi067.stdout:5/662: unlink d5/d3e/d90/fd0 0 2024-04-22T23:23:31.844 INFO:tasks.workunit.client.0.smithi067.stdout:5/663: mknod d5/d1a/ce7 0 2024-04-22T23:23:31.846 INFO:tasks.workunit.client.0.smithi067.stdout:5/664: symlink d5/d1a/le8 0 2024-04-22T23:23:31.846 INFO:tasks.workunit.client.0.smithi067.stdout:5/665: chown d5/f52 65908854 1 2024-04-22T23:23:31.853 INFO:tasks.workunit.client.0.smithi067.stdout:5/666: dwrite d5/d28/d36/d40/fd1 [0,4194304] 0 2024-04-22T23:23:31.854 INFO:tasks.workunit.client.0.smithi067.stdout:7/545: getdents d10/d14/d1d/d26/d7e 0 2024-04-22T23:23:31.862 INFO:tasks.workunit.client.0.smithi067.stdout:6/615: dread d17/d39/d3a/f51 [4194304,4194304] 0 2024-04-22T23:23:31.864 INFO:tasks.workunit.client.0.smithi067.stdout:2/653: truncate d1/d12/f4e 2234030 0 2024-04-22T23:23:31.865 INFO:tasks.workunit.client.0.smithi067.stdout:5/667: dwrite d5/d3a/d4d/f4e [4194304,4194304] 0 2024-04-22T23:23:31.866 INFO:tasks.workunit.client.0.smithi067.stdout:7/546: dwrite d10/d14/d1d/d46/d50/f2d [4194304,4194304] 0 2024-04-22T23:23:31.869 INFO:tasks.workunit.client.0.smithi067.stdout:2/654: fsync d1/d10/d33/d46/f99 0 2024-04-22T23:23:31.872 INFO:tasks.workunit.client.0.smithi067.stdout:9/587: truncate d5/dd/d18/d39/d3e/f53 2825443 0 2024-04-22T23:23:31.874 INFO:tasks.workunit.client.0.smithi067.stdout:2/655: fdatasync d1/d10/d33/d46/f99 0 2024-04-22T23:23:31.877 INFO:tasks.workunit.client.0.smithi067.stdout:6/616: creat d17/d39/d53/d90/fc6 x:0 0 0 2024-04-22T23:23:31.879 INFO:tasks.workunit.client.0.smithi067.stdout:2/656: fsync d1/d10/d33/d46/f88 0 2024-04-22T23:23:31.879 INFO:tasks.workunit.client.0.smithi067.stdout:1/568: write d8/d9/f25 [3546492,78642] 0 2024-04-22T23:23:31.880 INFO:tasks.workunit.client.0.smithi067.stdout:7/547: write d10/d14/d1d/d26/d2c/d7a/fa5 [1398539,33381] 0 2024-04-22T23:23:31.882 INFO:tasks.workunit.client.0.smithi067.stdout:8/591: dwrite d1/d3/d10/f45 [0,4194304] 0 2024-04-22T23:23:31.886 INFO:tasks.workunit.client.0.smithi067.stdout:3/651: dwrite d3/d4/d10/d38/f48 [0,4194304] 0 2024-04-22T23:23:31.886 INFO:tasks.workunit.client.0.smithi067.stdout:9/588: write d5/dd/d18/d39/d3e/d3f/d63/f84 [5605106,5178] 0 2024-04-22T23:23:31.887 INFO:tasks.workunit.client.0.smithi067.stdout:3/652: readlink d3/lf 0 2024-04-22T23:23:31.889 INFO:tasks.workunit.client.0.smithi067.stdout:6/617: rmdir d17/d39/d53/d90/d46/db0/db9 39 2024-04-22T23:23:31.892 INFO:tasks.workunit.client.0.smithi067.stdout:2/657: symlink d1/d42/d5f/d9b/ld6 0 2024-04-22T23:23:31.893 INFO:tasks.workunit.client.0.smithi067.stdout:8/592: dread d1/d3/f1e [0,4194304] 0 2024-04-22T23:23:31.894 INFO:tasks.workunit.client.0.smithi067.stdout:7/548: mknod d10/d18/d35/d62/cc6 0 2024-04-22T23:23:31.898 INFO:tasks.workunit.client.0.smithi067.stdout:4/740: dwrite fa [0,4194304] 0 2024-04-22T23:23:31.901 INFO:tasks.workunit.client.0.smithi067.stdout:8/593: write d1/faf [958438,55459] 0 2024-04-22T23:23:31.903 INFO:tasks.workunit.client.0.smithi067.stdout:4/741: chown d12/d5f/d2b/db6 2785 1 2024-04-22T23:23:31.906 INFO:tasks.workunit.client.0.smithi067.stdout:1/569: mknod d8/d9/d2f/cbf 0 2024-04-22T23:23:31.911 INFO:tasks.workunit.client.0.smithi067.stdout:8/594: fsync d1/dd/d26/f93 0 2024-04-22T23:23:31.911 INFO:tasks.workunit.client.0.smithi067.stdout:5/668: dread d5/db/d35/f39 [0,4194304] 0 2024-04-22T23:23:31.911 INFO:tasks.workunit.client.0.smithi067.stdout:9/589: dwrite d5/dd/d18/d3c/d40/f50 [0,4194304] 0 2024-04-22T23:23:31.915 INFO:tasks.workunit.client.0.smithi067.stdout:5/669: readlink d5/db/d12/lab 0 2024-04-22T23:23:31.916 INFO:tasks.workunit.client.0.smithi067.stdout:2/658: rmdir d1/d12/db2 0 2024-04-22T23:23:31.918 INFO:tasks.workunit.client.0.smithi067.stdout:9/590: dread - d5/dd/d18/d39/d8f/fb1 zero size 2024-04-22T23:23:31.918 INFO:tasks.workunit.client.0.smithi067.stdout:3/653: getdents d3/d4/d10/d38/d53/d55/d5c 0 2024-04-22T23:23:31.922 INFO:tasks.workunit.client.0.smithi067.stdout:1/570: rename d8/d9/d2d/d4d/d70/d34/ca2 to d8/d9/d2f/d33/cc0 0 2024-04-22T23:23:31.922 INFO:tasks.workunit.client.0.smithi067.stdout:7/549: dwrite d10/d14/d1d/d26/d2c/d41/f99 [0,4194304] 0 2024-04-22T23:23:31.923 INFO:tasks.workunit.client.0.smithi067.stdout:0/605: dwrite db/fae [4194304,4194304] 0 2024-04-22T23:23:31.925 INFO:tasks.workunit.client.0.smithi067.stdout:5/670: symlink d5/d28/daf/le9 0 2024-04-22T23:23:31.925 INFO:tasks.workunit.client.0.smithi067.stdout:9/591: truncate d5/fca 125287 0 2024-04-22T23:23:31.927 INFO:tasks.workunit.client.0.smithi067.stdout:3/654: chown d3/d4/d10/d38/d53/d55/d74/dd5/ddb/l72 0 1 2024-04-22T23:23:31.927 INFO:tasks.workunit.client.0.smithi067.stdout:0/606: stat db/d21/d29/cb6 0 2024-04-22T23:23:31.928 INFO:tasks.workunit.client.0.smithi067.stdout:1/571: chown d8/d9/d2d/d4d/lbd 314394 1 2024-04-22T23:23:31.930 INFO:tasks.workunit.client.0.smithi067.stdout:8/595: rename d1/dd/d26/f8f to d1/dd/d80/d6f/da8/fbf 0 2024-04-22T23:23:31.931 INFO:tasks.workunit.client.0.smithi067.stdout:5/671: fdatasync d5/db/d12/d44/f81 0 2024-04-22T23:23:31.932 INFO:tasks.workunit.client.0.smithi067.stdout:6/618: truncate d17/f2b 1270840 0 2024-04-22T23:23:31.936 INFO:tasks.workunit.client.0.smithi067.stdout:0/607: mknod db/d21/d29/d2d/ccb 0 2024-04-22T23:23:31.937 INFO:tasks.workunit.client.0.smithi067.stdout:6/619: write d17/d39/d53/f9e [56091,52262] 0 2024-04-22T23:23:31.939 INFO:tasks.workunit.client.0.smithi067.stdout:3/655: mknod d3/d4/d10/d19/cea 0 2024-04-22T23:23:31.941 INFO:tasks.workunit.client.0.smithi067.stdout:9/592: dwrite d5/dd/d18/d39/d49/f55 [4194304,4194304] 0 2024-04-22T23:23:31.943 INFO:tasks.workunit.client.0.smithi067.stdout:9/593: fdatasync d5/fca 0 2024-04-22T23:23:31.944 INFO:tasks.workunit.client.0.smithi067.stdout:2/659: rename d1/fc to d1/d10/d72/fd7 0 2024-04-22T23:23:31.945 INFO:tasks.workunit.client.0.smithi067.stdout:8/596: dwrite d1/d3/d27/f7c [0,4194304] 0 2024-04-22T23:23:31.949 INFO:tasks.workunit.client.0.smithi067.stdout:1/572: symlink d8/d9/d2d/d4d/d70/d34/lc1 0 2024-04-22T23:23:31.950 INFO:tasks.workunit.client.0.smithi067.stdout:1/573: fsync d8/d9/d2d/f4f 0 2024-04-22T23:23:31.950 INFO:tasks.workunit.client.0.smithi067.stdout:1/574: dread - d8/d12/d36/fba zero size 2024-04-22T23:23:31.959 INFO:tasks.workunit.client.0.smithi067.stdout:7/550: link d10/d14/d1d/d26/d2c/d7a/f87 d10/d18/d88/fc7 0 2024-04-22T23:23:31.959 INFO:tasks.workunit.client.0.smithi067.stdout:2/660: write d1/d10/d1c/f40 [105958,7861] 0 2024-04-22T23:23:31.966 INFO:tasks.workunit.client.0.smithi067.stdout:1/575: fdatasync d8/d9/d2f/d59/f78 0 2024-04-22T23:23:31.966 INFO:tasks.workunit.client.0.smithi067.stdout:7/551: dwrite d10/d14/d1d/d26/d2c/d41/f99 [0,4194304] 0 2024-04-22T23:23:31.968 INFO:tasks.workunit.client.0.smithi067.stdout:1/576: write d8/d9/d2f/d59/f81 [601369,4303] 0 2024-04-22T23:23:31.968 INFO:tasks.workunit.client.0.smithi067.stdout:0/608: symlink db/d9a/d56/d70/dbf/lcc 0 2024-04-22T23:23:31.970 INFO:tasks.workunit.client.0.smithi067.stdout:6/620: symlink d17/d39/d53/d90/lc7 0 2024-04-22T23:23:31.974 INFO:tasks.workunit.client.0.smithi067.stdout:9/594: unlink d5/d21/d2c/fa5 0 2024-04-22T23:23:31.978 INFO:tasks.workunit.client.0.smithi067.stdout:3/656: dwrite d3/d4/d10/d19/f3b [0,4194304] 0 2024-04-22T23:23:31.979 INFO:tasks.workunit.client.0.smithi067.stdout:5/672: write d5/d28/f69 [1877933,10099] 0 2024-04-22T23:23:31.982 INFO:tasks.workunit.client.0.smithi067.stdout:8/597: creat d1/fc0 x:0 0 0 2024-04-22T23:23:31.985 INFO:tasks.workunit.client.0.smithi067.stdout:7/552: dread d10/d14/d1d/d46/d50/f29 [0,4194304] 0 2024-04-22T23:23:31.985 INFO:tasks.workunit.client.0.smithi067.stdout:8/598: readlink d1/dd/d80/d6f/l97 0 2024-04-22T23:23:31.988 INFO:tasks.workunit.client.0.smithi067.stdout:8/599: fdatasync d1/dd/f53 0 2024-04-22T23:23:31.988 INFO:tasks.workunit.client.0.smithi067.stdout:1/577: dwrite d8/d9/f2c [0,4194304] 0 2024-04-22T23:23:31.990 INFO:tasks.workunit.client.0.smithi067.stdout:0/609: mknod db/ccd 0 2024-04-22T23:23:31.990 INFO:tasks.workunit.client.0.smithi067.stdout:6/621: truncate f12 2951688 0 2024-04-22T23:23:31.991 INFO:tasks.workunit.client.0.smithi067.stdout:0/610: fsync db/d9a/f63 0 2024-04-22T23:23:31.992 INFO:tasks.workunit.client.0.smithi067.stdout:5/673: creat d5/db/d35/fea x:0 0 0 2024-04-22T23:23:31.996 INFO:tasks.workunit.client.0.smithi067.stdout:3/657: creat d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d8c/feb x:0 0 0 2024-04-22T23:23:31.998 INFO:tasks.workunit.client.0.smithi067.stdout:3/658: read d3/d4/d10/d1f/d3c/f67 [256932,11277] 0 2024-04-22T23:23:31.998 INFO:tasks.workunit.client.0.smithi067.stdout:1/578: symlink d8/d9/d2f/d33/lc2 0 2024-04-22T23:23:31.999 INFO:tasks.workunit.client.0.smithi067.stdout:8/600: creat d1/d3/d10/fc1 x:0 0 0 2024-04-22T23:23:32.001 INFO:tasks.workunit.client.0.smithi067.stdout:6/622: rename d17/f9a to d17/dae/fc8 0 2024-04-22T23:23:32.003 INFO:tasks.workunit.client.0.smithi067.stdout:6/623: read d17/f21 [2616703,3155] 0 2024-04-22T23:23:32.004 INFO:tasks.workunit.client.0.smithi067.stdout:6/624: stat d17/d39/c95 0 2024-04-22T23:23:32.005 INFO:tasks.workunit.client.0.smithi067.stdout:3/659: rmdir d3/d4/d10/d38/d53/d55/d74 39 2024-04-22T23:23:32.006 INFO:tasks.workunit.client.0.smithi067.stdout:1/579: chown d8/d9/d2d/d4d/d70/d34/d60/lb8 2044504 1 2024-04-22T23:23:32.007 INFO:tasks.workunit.client.0.smithi067.stdout:1/580: read d8/d9/fa9 [8709,81937] 0 2024-04-22T23:23:32.008 INFO:tasks.workunit.client.0.smithi067.stdout:1/581: write d8/d9/d2f/d31/f58 [1530444,96700] 0 2024-04-22T23:23:32.010 INFO:tasks.workunit.client.0.smithi067.stdout:8/601: creat d1/d41/d46/fc2 x:0 0 0 2024-04-22T23:23:32.011 INFO:tasks.workunit.client.0.smithi067.stdout:8/602: write d1/d41/d46/f66 [24709,123842] 0 2024-04-22T23:23:32.012 INFO:tasks.workunit.client.0.smithi067.stdout:8/603: read - d1/fc0 zero size 2024-04-22T23:23:32.013 INFO:tasks.workunit.client.0.smithi067.stdout:9/595: getdents d5/dd/d18/d39/d3e 0 2024-04-22T23:23:32.017 INFO:tasks.workunit.client.0.smithi067.stdout:6/625: mkdir d17/d39/d3a/d8d/dc9 0 2024-04-22T23:23:32.017 INFO:tasks.workunit.client.0.smithi067.stdout:0/611: write db/d21/f8d [3838018,126549] 0 2024-04-22T23:23:32.021 INFO:tasks.workunit.client.0.smithi067.stdout:0/612: chown db/de/d2a/l81 502328 1 2024-04-22T23:23:32.022 INFO:tasks.workunit.client.0.smithi067.stdout:9/596: dwrite d5/dd/d18/d39/f64 [4194304,4194304] 0 2024-04-22T23:23:32.025 INFO:tasks.workunit.client.0.smithi067.stdout:8/604: chown d1/d3/d10/d3b/c69 3956081 1 2024-04-22T23:23:32.025 INFO:tasks.workunit.client.0.smithi067.stdout:9/597: write d5/dd/d18/d3c/d40/f50 [4579222,42814] 0 2024-04-22T23:23:32.026 INFO:tasks.workunit.client.0.smithi067.stdout:7/553: getdents d10/d14/d1d/d26/d2c/d41 0 2024-04-22T23:23:32.027 INFO:tasks.workunit.client.0.smithi067.stdout:3/660: mkdir d3/d4/dec 0 2024-04-22T23:23:32.030 INFO:tasks.workunit.client.0.smithi067.stdout:0/613: chown db/d21/d29/d2d/cb5 45703 1 2024-04-22T23:23:32.031 INFO:tasks.workunit.client.0.smithi067.stdout:7/554: write d10/d14/d1d/d46/d6a/f86 [4820070,79432] 0 2024-04-22T23:23:32.031 INFO:tasks.workunit.client.0.smithi067.stdout:6/626: dwrite d17/d39/d53/d90/fc6 [0,4194304] 0 2024-04-22T23:23:32.034 INFO:tasks.workunit.client.0.smithi067.stdout:9/598: rmdir d5/dd/d18/d3c 39 2024-04-22T23:23:32.037 INFO:tasks.workunit.client.0.smithi067.stdout:8/605: write d1/dd/d80/d6f/f30 [4999776,67688] 0 2024-04-22T23:23:32.037 INFO:tasks.workunit.client.0.smithi067.stdout:5/674: getdents d5/d28/d36/d40/d97/dc8 0 2024-04-22T23:23:32.042 INFO:tasks.workunit.client.0.smithi067.stdout:7/555: creat d10/d14/d1d/d26/d80/fc8 x:0 0 0 2024-04-22T23:23:32.042 INFO:tasks.workunit.client.0.smithi067.stdout:1/582: rename d8/d9/d2f/d33/c9f to d8/d9/d2f/cc3 0 2024-04-22T23:23:32.046 INFO:tasks.workunit.client.0.smithi067.stdout:8/606: dwrite d1/dd/d80/d6f/f50 [4194304,4194304] 0 2024-04-22T23:23:32.046 INFO:tasks.workunit.client.0.smithi067.stdout:3/661: rename d3/d4/d10/d1f/ccd to d3/d4/d28/dc0/ced 0 2024-04-22T23:23:32.049 INFO:tasks.workunit.client.0.smithi067.stdout:1/583: creat d8/d9/d2d/d4d/d48/fc4 x:0 0 0 2024-04-22T23:23:32.050 INFO:tasks.workunit.client.0.smithi067.stdout:6/627: creat d17/fca x:0 0 0 2024-04-22T23:23:32.051 INFO:tasks.workunit.client.0.smithi067.stdout:0/614: getdents db/de/d11 0 2024-04-22T23:23:32.053 INFO:tasks.workunit.client.0.smithi067.stdout:0/615: write db/faf [4028858,100624] 0 2024-04-22T23:23:32.054 INFO:tasks.workunit.client.0.smithi067.stdout:9/599: rename d5/fca to d5/dd/d18/d39/d49/d54/d99/fcb 0 2024-04-22T23:23:32.056 INFO:tasks.workunit.client.0.smithi067.stdout:1/584: mkdir d8/d9/d2d/d4d/d83/dc5 0 2024-04-22T23:23:32.057 INFO:tasks.workunit.client.0.smithi067.stdout:6/628: creat d17/d83/fcb x:0 0 0 2024-04-22T23:23:32.057 INFO:tasks.workunit.client.0.smithi067.stdout:1/585: readlink d8/d9/d2d/l76 0 2024-04-22T23:23:32.061 INFO:tasks.workunit.client.0.smithi067.stdout:9/600: dwrite d5/f10 [0,4194304] 0 2024-04-22T23:23:32.061 INFO:tasks.workunit.client.0.smithi067.stdout:5/675: rename d5/l9 to d5/db/d12/d44/leb 0 2024-04-22T23:23:32.062 INFO:tasks.workunit.client.0.smithi067.stdout:5/676: chown d5/d3a/d4d 28988838 1 2024-04-22T23:23:32.064 INFO:tasks.workunit.client.0.smithi067.stdout:6/629: write d17/d39/d3a/fb6 [1215509,30772] 0 2024-04-22T23:23:32.065 INFO:tasks.workunit.client.0.smithi067.stdout:9/601: write d5/dd/d18/d5c/fab [569486,112921] 0 2024-04-22T23:23:32.067 INFO:tasks.workunit.client.0.smithi067.stdout:1/586: mknod d8/da7/cc6 0 2024-04-22T23:23:32.067 INFO:tasks.workunit.client.0.smithi067.stdout:9/602: write d5/d21/d47/d57/f97 [4553593,15278] 0 2024-04-22T23:23:32.069 INFO:tasks.workunit.client.0.smithi067.stdout:5/677: creat d5/db/d12/d8a/fec x:0 0 0 2024-04-22T23:23:32.071 INFO:tasks.workunit.client.0.smithi067.stdout:3/662: getdents d3/d4/d10/d1f/d3c 0 2024-04-22T23:23:32.072 INFO:tasks.workunit.client.0.smithi067.stdout:3/663: dread - d3/d4/d10/d38/d53/d55/d5c/d70/f9b zero size 2024-04-22T23:23:32.072 INFO:tasks.workunit.client.0.smithi067.stdout:6/630: rmdir d17/d83/db5/dbb 39 2024-04-22T23:23:32.075 INFO:tasks.workunit.client.0.smithi067.stdout:0/616: link la db/d9a/d56/d6b/lce 0 2024-04-22T23:23:32.075 INFO:tasks.workunit.client.0.smithi067.stdout:0/617: chown db/d2e/ca0 2 1 2024-04-22T23:23:32.077 INFO:tasks.workunit.client.0.smithi067.stdout:9/603: mkdir d5/dd/d18/d39/d49/dcc 0 2024-04-22T23:23:32.080 INFO:tasks.workunit.client.0.smithi067.stdout:1/587: rename d8/d9/d2d/d4d/d48/d9a/la6 to d8/d9/d2f/d33/lc7 0 2024-04-22T23:23:32.083 INFO:tasks.workunit.client.0.smithi067.stdout:0/618: mkdir db/d9a/d56/dcf 0 2024-04-22T23:23:32.084 INFO:tasks.workunit.client.0.smithi067.stdout:5/678: creat d5/d3a/d4d/dc6/fed x:0 0 0 2024-04-22T23:23:32.087 INFO:tasks.workunit.client.0.smithi067.stdout:6/631: rename d17/d39/d53/d90/f6f to d17/d83/d9b/fcc 0 2024-04-22T23:23:32.088 INFO:tasks.workunit.client.0.smithi067.stdout:1/588: fsync d8/d9/fa9 0 2024-04-22T23:23:32.096 INFO:tasks.workunit.client.0.smithi067.stdout:6/632: chown d17/d83/c9d 2256 1 2024-04-22T23:23:32.096 INFO:tasks.workunit.client.0.smithi067.stdout:6/633: readlink d17/l5d 0 2024-04-22T23:23:32.098 INFO:tasks.workunit.client.0.smithi067.stdout:1/589: creat d8/d9/d2d/d4d/d83/fc8 x:0 0 0 2024-04-22T23:23:32.100 INFO:tasks.workunit.client.0.smithi067.stdout:9/604: truncate d5/fad 3908973 0 2024-04-22T23:23:32.100 INFO:tasks.workunit.client.0.smithi067.stdout:3/664: dwrite d3/d4/d10/d8a/fa6 [0,4194304] 0 2024-04-22T23:23:32.102 INFO:tasks.workunit.client.0.smithi067.stdout:0/619: link db/d2e/db3/lc6 db/d9a/d56/d70/d8c/ld0 0 2024-04-22T23:23:32.103 INFO:tasks.workunit.client.0.smithi067.stdout:5/679: dread d5/db/d35/f6c [0,4194304] 0 2024-04-22T23:23:32.104 INFO:tasks.workunit.client.0.smithi067.stdout:0/620: dread - db/d71/fbc zero size 2024-04-22T23:23:32.105 INFO:tasks.workunit.client.0.smithi067.stdout:0/621: write db/faf [3510181,127356] 0 2024-04-22T23:23:32.107 INFO:tasks.workunit.client.0.smithi067.stdout:9/605: fdatasync d5/dd/d18/d39/d4d/faa 0 2024-04-22T23:23:32.109 INFO:tasks.workunit.client.0.smithi067.stdout:1/590: mknod d8/d9/d2d/d4d/d70/d34/d60/cc9 0 2024-04-22T23:23:32.112 INFO:tasks.workunit.client.0.smithi067.stdout:5/680: mknod d5/db/d12/dbb/cee 0 2024-04-22T23:23:32.115 INFO:tasks.workunit.client.0.smithi067.stdout:6/634: truncate fe 389285 0 2024-04-22T23:23:32.116 INFO:tasks.workunit.client.0.smithi067.stdout:1/591: creat d8/d9/d2d/d4d/d9c/fca x:0 0 0 2024-04-22T23:23:32.117 INFO:tasks.workunit.client.0.smithi067.stdout:1/592: stat d8/d12/d36/f4e 0 2024-04-22T23:23:32.117 INFO:tasks.workunit.client.0.smithi067.stdout:5/681: chown d5/db/f23 120 1 2024-04-22T23:23:32.118 INFO:tasks.workunit.client.0.smithi067.stdout:5/682: chown d5/d3a/f3c 3096883 1 2024-04-22T23:23:32.119 INFO:tasks.workunit.client.0.smithi067.stdout:1/593: dread - d8/d9/d2d/d4d/d70/d34/fb5 zero size 2024-04-22T23:23:32.119 INFO:tasks.workunit.client.0.smithi067.stdout:3/665: rename d3/d4/d10/c64 to d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d90/da2/daf/cee 0 2024-04-22T23:23:32.120 INFO:tasks.workunit.client.0.smithi067.stdout:1/594: chown d8/d12/d36/d87 1390963 1 2024-04-22T23:23:32.126 INFO:tasks.workunit.client.0.smithi067.stdout:1/595: write d8/d9/d2f/d33/f3e [2366261,25795] 0 2024-04-22T23:23:32.128 INFO:tasks.workunit.client.0.smithi067.stdout:9/606: dread d5/dd/d18/d39/d49/f55 [0,4194304] 0 2024-04-22T23:23:32.130 INFO:tasks.workunit.client.0.smithi067.stdout:3/666: rename d3/d4/d10/d38/d53/d55/d74/ca4 to d3/d4/d10/d38/d53/dcf/cef 0 2024-04-22T23:23:32.132 INFO:tasks.workunit.client.0.smithi067.stdout:5/683: dread d5/db/ff [4194304,4194304] 0 2024-04-22T23:23:32.133 INFO:tasks.workunit.client.0.smithi067.stdout:0/622: truncate db/d21/d29/d75/f86 2363099 0 2024-04-22T23:23:32.134 INFO:tasks.workunit.client.0.smithi067.stdout:9/607: symlink d5/dd/da9/lcd 0 2024-04-22T23:23:32.139 INFO:tasks.workunit.client.0.smithi067.stdout:0/623: dread - db/d21/d29/f62 zero size 2024-04-22T23:23:32.143 INFO:tasks.workunit.client.0.smithi067.stdout:9/608: chown d5/dd/d18/d3c/d78 65422 1 2024-04-22T23:23:32.145 INFO:tasks.workunit.client.0.smithi067.stdout:6/635: dwrite d17/f24 [0,4194304] 0 2024-04-22T23:23:32.150 INFO:tasks.workunit.client.0.smithi067.stdout:9/609: read d5/dd/d18/d39/d49/d54/d99/fcb [19061,55443] 0 2024-04-22T23:23:32.151 INFO:tasks.workunit.client.0.smithi067.stdout:3/667: dread d3/d4/d10/d38/d53/f54 [0,4194304] 0 2024-04-22T23:23:32.153 INFO:tasks.workunit.client.0.smithi067.stdout:6/636: readlink d17/d39/l79 0 2024-04-22T23:23:32.154 INFO:tasks.workunit.client.0.smithi067.stdout:5/684: getdents d5/db/d12/d8a 0 2024-04-22T23:23:32.155 INFO:tasks.workunit.client.0.smithi067.stdout:0/624: dwrite db/fae [0,4194304] 0 2024-04-22T23:23:32.156 INFO:tasks.workunit.client.0.smithi067.stdout:9/610: symlink d5/d21/d47/d56/lce 0 2024-04-22T23:23:32.160 INFO:tasks.workunit.client.0.smithi067.stdout:0/625: write db/fae [2188403,80621] 0 2024-04-22T23:23:32.160 INFO:tasks.workunit.client.0.smithi067.stdout:1/596: dwrite d8/d9/f29 [0,4194304] 0 2024-04-22T23:23:32.163 INFO:tasks.workunit.client.0.smithi067.stdout:3/668: write d3/d4/d10/d19/f34 [1215784,44579] 0 2024-04-22T23:23:32.163 INFO:tasks.workunit.client.0.smithi067.stdout:1/597: fdatasync d8/d9/f25 0 2024-04-22T23:23:32.164 INFO:tasks.workunit.client.0.smithi067.stdout:0/626: chown db/d21/d29/d75/d9d/fb8 287314 1 2024-04-22T23:23:32.166 INFO:tasks.workunit.client.0.smithi067.stdout:9/611: symlink d5/dd/d18/d39/d4d/lcf 0 2024-04-22T23:23:32.170 INFO:tasks.workunit.client.0.smithi067.stdout:9/612: write d5/d21/d47/d57/fa2 [227566,63371] 0 2024-04-22T23:23:32.170 INFO:tasks.workunit.client.0.smithi067.stdout:6/637: mknod d17/d39/d53/d90/ccd 0 2024-04-22T23:23:32.172 INFO:tasks.workunit.client.0.smithi067.stdout:5/685: link d5/d28/d36/d40/d97/dc8/fce d5/d28/dae/fef 0 2024-04-22T23:23:32.175 INFO:tasks.workunit.client.0.smithi067.stdout:3/669: rename d3/d4/d28/d5f/f71 to d3/d4/d10/d38/dbf/ff0 0 2024-04-22T23:23:32.177 INFO:tasks.workunit.client.0.smithi067.stdout:1/598: dwrite d8/d9/d2f/d59/f81 [0,4194304] 0 2024-04-22T23:23:32.177 INFO:tasks.workunit.client.0.smithi067.stdout:6/638: unlink d17/d39/d3a/f40 0 2024-04-22T23:23:32.179 INFO:tasks.workunit.client.0.smithi067.stdout:9/613: creat d5/dd/d18/d39/d4d/fd0 x:0 0 0 2024-04-22T23:23:32.182 INFO:tasks.workunit.client.0.smithi067.stdout:5/686: rename d5/d3e/f42 to d5/db/d12/dbb/d5f/d79/ff0 0 2024-04-22T23:23:32.185 INFO:tasks.workunit.client.0.smithi067.stdout:6/639: mknod d17/d39/d3a/da9/cce 0 2024-04-22T23:23:32.187 INFO:tasks.workunit.client.0.smithi067.stdout:1/599: rename d8/d9/d2d/d4d/d70/d34/d60/cc9 to d8/d9/d2d/d4d/d9c/ccb 0 2024-04-22T23:23:32.188 INFO:tasks.workunit.client.0.smithi067.stdout:3/670: dwrite d3/d4/d10/f96 [0,4194304] 0 2024-04-22T23:23:32.191 INFO:tasks.workunit.client.0.smithi067.stdout:9/614: mkdir d5/dd/d18/d39/d49/d54/dd1 0 2024-04-22T23:23:32.192 INFO:tasks.workunit.client.0.smithi067.stdout:5/687: dread d5/d28/d36/d40/fd1 [0,4194304] 0 2024-04-22T23:23:32.193 INFO:tasks.workunit.client.0.smithi067.stdout:5/688: dread - d5/d28/d36/f86 zero size 2024-04-22T23:23:32.195 INFO:tasks.workunit.client.0.smithi067.stdout:3/671: mknod d3/d4/d28/d5f/cf1 0 2024-04-22T23:23:32.197 INFO:tasks.workunit.client.0.smithi067.stdout:9/615: symlink d5/dd/d18/d39/d3e/d3f/d63/ld2 0 2024-04-22T23:23:32.200 INFO:tasks.workunit.client.0.smithi067.stdout:0/627: dread db/d21/d29/d2c/f7d [0,4194304] 0 2024-04-22T23:23:32.203 INFO:tasks.workunit.client.0.smithi067.stdout:3/672: creat d3/d4/d10/d38/d53/d55/d74/d7f/ff2 x:0 0 0 2024-04-22T23:23:32.216 INFO:tasks.workunit.client.0.smithi067.stdout:0/628: dread db/de/d11/d1a/f64 [0,4194304] 0 2024-04-22T23:23:32.255 INFO:tasks.workunit.client.0.smithi067.stdout:4/742: sync 2024-04-22T23:23:32.262 INFO:tasks.workunit.client.0.smithi067.stdout:9/616: rename d5/dd/f32 to d5/d21/d47/d7f/fd3 0 2024-04-22T23:23:32.263 INFO:tasks.workunit.client.0.smithi067.stdout:1/600: getdents d8/d9/d2d/d4d/d48/d9a 0 2024-04-22T23:23:32.263 INFO:tasks.workunit.client.0.smithi067.stdout:6/640: rmdir d17/d39 39 2024-04-22T23:23:32.264 INFO:tasks.workunit.client.0.smithi067.stdout:3/673: mknod d3/d4/d10/d38/d53/dcf/cf3 0 2024-04-22T23:23:32.264 INFO:tasks.workunit.client.0.smithi067.stdout:5/689: dread d5/d28/f64 [0,4194304] 0 2024-04-22T23:23:32.266 INFO:tasks.workunit.client.0.smithi067.stdout:3/674: write d3/fc1 [15291,125746] 0 2024-04-22T23:23:32.278 INFO:tasks.workunit.client.0.smithi067.stdout:6/641: dread d17/f42 [0,4194304] 0 2024-04-22T23:23:32.292 INFO:tasks.workunit.client.0.smithi067.stdout:4/743: rmdir d12/d17/d3f 39 2024-04-22T23:23:32.297 INFO:tasks.workunit.client.0.smithi067.stdout:5/690: creat d5/d3e/d90/ff1 x:0 0 0 2024-04-22T23:23:32.298 INFO:tasks.workunit.client.0.smithi067.stdout:3/675: mkdir d3/d4/d10/d1f/d3c/df4 0 2024-04-22T23:23:32.302 INFO:tasks.workunit.client.0.smithi067.stdout:9/617: mknod d5/d21/cd4 0 2024-04-22T23:23:32.302 INFO:tasks.workunit.client.0.smithi067.stdout:1/601: dwrite d8/d9/d2f/d33/d66/f91 [0,4194304] 0 2024-04-22T23:23:32.304 INFO:tasks.workunit.client.0.smithi067.stdout:4/744: truncate d12/fd9 386902 0 2024-04-22T23:23:32.306 INFO:tasks.workunit.client.0.smithi067.stdout:0/629: symlink db/de/d11/d1a/da4/dc5/ld1 0 2024-04-22T23:23:32.308 INFO:tasks.workunit.client.0.smithi067.stdout:5/691: mknod d5/d28/dae/cf2 0 2024-04-22T23:23:32.312 INFO:tasks.workunit.client.0.smithi067.stdout:6/642: symlink d17/d39/d53/lcf 0 2024-04-22T23:23:32.313 INFO:tasks.workunit.client.0.smithi067.stdout:3/676: mkdir d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d62/df5 0 2024-04-22T23:23:32.316 INFO:tasks.workunit.client.0.smithi067.stdout:9/618: rename d5/dd/d18/d39/d8f/dba to d5/d21/d47/d56/d8c/dd5 0 2024-04-22T23:23:32.318 INFO:tasks.workunit.client.0.smithi067.stdout:5/692: rmdir d5/db/d35 39 2024-04-22T23:23:32.320 INFO:tasks.workunit.client.0.smithi067.stdout:5/693: chown d5/d3e/d90/ff1 3354 1 2024-04-22T23:23:32.322 INFO:tasks.workunit.client.0.smithi067.stdout:1/602: mknod d8/d9/d2d/d4d/d83/dc5/ccc 0 2024-04-22T23:23:32.322 INFO:tasks.workunit.client.0.smithi067.stdout:6/643: read - d17/d83/db5/dbb/fc1 zero size 2024-04-22T23:23:32.324 INFO:tasks.workunit.client.0.smithi067.stdout:3/677: creat d3/d4/d10/d38/d53/d55/d74/ff6 x:0 0 0 2024-04-22T23:23:32.325 INFO:tasks.workunit.client.0.smithi067.stdout:9/619: dwrite d5/d21/faf [0,4194304] 0 2024-04-22T23:23:32.326 INFO:tasks.workunit.client.0.smithi067.stdout:4/745: rename d12/d5f/l3a to d12/d17/d47/lfb 0 2024-04-22T23:23:32.328 INFO:tasks.workunit.client.0.smithi067.stdout:1/603: write d8/d9/d2d/d4d/d83/fc8 [202194,87787] 0 2024-04-22T23:23:32.329 INFO:tasks.workunit.client.0.smithi067.stdout:0/630: link db/d9a/d56/d70/dba/fc0 db/d21/d29/d75/d9d/fd2 0 2024-04-22T23:23:32.333 INFO:tasks.workunit.client.0.smithi067.stdout:5/694: truncate d5/d28/d36/d40/f20 1226420 0 2024-04-22T23:23:32.338 INFO:tasks.workunit.client.0.smithi067.stdout:3/678: dwrite d3/dcb/fce [0,4194304] 0 2024-04-22T23:23:32.339 INFO:tasks.workunit.client.0.smithi067.stdout:1/604: fsync d8/d9/d2d/d4d/d70/d34/d60/d62/f95 0 2024-04-22T23:23:32.342 INFO:tasks.workunit.client.0.smithi067.stdout:9/620: symlink d5/d21/ld6 0 2024-04-22T23:23:32.343 INFO:tasks.workunit.client.0.smithi067.stdout:1/605: stat d8/d9/d2d/d4d/f23 0 2024-04-22T23:23:32.343 INFO:tasks.workunit.client.0.smithi067.stdout:5/695: truncate d5/db/d12/d44/d47/fda 564430 0 2024-04-22T23:23:32.343 INFO:tasks.workunit.client.0.smithi067.stdout:0/631: dwrite db/f6a [8388608,4194304] 0 2024-04-22T23:23:32.347 INFO:tasks.workunit.client.0.smithi067.stdout:4/746: link d12/d17/d47/c92 d12/d5f/d2b/cfc 0 2024-04-22T23:23:32.347 INFO:tasks.workunit.client.0.smithi067.stdout:9/621: dread - d5/dd/d18/d39/d3e/d3f/d63/fa8 zero size 2024-04-22T23:23:32.351 INFO:tasks.workunit.client.0.smithi067.stdout:6/644: rename d17/d39/d3a/fb6 to d17/d39/d53/d90/d46/d62/fd0 0 2024-04-22T23:23:32.353 INFO:tasks.workunit.client.0.smithi067.stdout:5/696: mknod d5/d28/d36/d40/d97/cf3 0 2024-04-22T23:23:32.355 INFO:tasks.workunit.client.0.smithi067.stdout:1/606: creat d8/d9/d2d/d4d/d83/dc5/fcd x:0 0 0 2024-04-22T23:23:32.356 INFO:tasks.workunit.client.0.smithi067.stdout:9/622: dwrite d5/d21/f2b [0,4194304] 0 2024-04-22T23:23:32.360 INFO:tasks.workunit.client.0.smithi067.stdout:4/747: truncate d12/d17/d3f/d7a/faf 624659 0 2024-04-22T23:23:32.363 INFO:tasks.workunit.client.0.smithi067.stdout:0/632: rename db/d21/fab to db/d9a/d56/d70/fd3 0 2024-04-22T23:23:32.369 INFO:tasks.workunit.client.0.smithi067.stdout:3/679: link d3/d4/d12/f27 d3/d4/d28/d4a/ff7 0 2024-04-22T23:23:32.375 INFO:tasks.workunit.client.0.smithi067.stdout:2/661: sync 2024-04-22T23:23:32.375 INFO:tasks.workunit.client.0.smithi067.stdout:4/748: mknod d12/d5f/d2b/ded/d8e/d56/d8b/cfd 0 2024-04-22T23:23:32.376 INFO:tasks.workunit.client.0.smithi067.stdout:2/662: truncate d1/d12/f92 336932 0 2024-04-22T23:23:32.376 INFO:tasks.workunit.client.0.smithi067.stdout:0/633: symlink db/d9a/d56/d6b/ld4 0 2024-04-22T23:23:32.379 INFO:tasks.workunit.client.0.smithi067.stdout:9/623: dread d5/dd/d18/d39/d3e/d3f/d63/f84 [0,4194304] 0 2024-04-22T23:23:32.380 INFO:tasks.workunit.client.0.smithi067.stdout:6/645: dread d17/d39/d53/d90/d46/d62/f72 [0,4194304] 0 2024-04-22T23:23:32.381 INFO:tasks.workunit.client.0.smithi067.stdout:6/646: chown d17/fca 30815 1 2024-04-22T23:23:32.383 INFO:tasks.workunit.client.0.smithi067.stdout:1/607: link d8/d9/d2d/d4d/d70/d34/d5d/f8b d8/d9/d2d/d4d/d48/fce 0 2024-04-22T23:23:32.388 INFO:tasks.workunit.client.0.smithi067.stdout:3/680: creat d3/d4/d10/de3/ff8 x:0 0 0 2024-04-22T23:23:32.388 INFO:tasks.workunit.client.0.smithi067.stdout:5/697: write d5/d3e/f77 [486519,110107] 0 2024-04-22T23:23:32.451 INFO:tasks.workunit.client.0.smithi067.stdout:4/749: rename d12/d5f/d2b/lc7 to d12/d17/d3f/d84/lfe 0 2024-04-22T23:23:32.452 INFO:tasks.workunit.client.0.smithi067.stdout:0/634: truncate db/d2e/d46/f82 2700253 0 2024-04-22T23:23:32.453 INFO:tasks.workunit.client.0.smithi067.stdout:3/681: creat d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d8c/ff9 x:0 0 0 2024-04-22T23:23:32.455 INFO:tasks.workunit.client.0.smithi067.stdout:4/750: symlink d12/d2e/d9b/lff 0 2024-04-22T23:23:32.483 INFO:tasks.workunit.client.0.smithi067.stdout:7/556: sync 2024-04-22T23:23:32.483 INFO:tasks.workunit.client.0.smithi067.stdout:8/607: sync 2024-04-22T23:23:32.523 INFO:tasks.workunit.client.0.smithi067.stdout:9/624: dwrite d5/dd/d18/d39/d4d/f9e [8388608,4194304] 0 2024-04-22T23:23:32.525 INFO:tasks.workunit.client.0.smithi067.stdout:2/663: getdents d1/d4f 0 2024-04-22T23:23:32.528 INFO:tasks.workunit.client.0.smithi067.stdout:5/698: link d5/d3e/d87/d99/fc3 d5/d28/d36/d40/ff4 0 2024-04-22T23:23:32.531 INFO:tasks.workunit.client.0.smithi067.stdout:6/647: dwrite d17/d39/f81 [0,4194304] 0 2024-04-22T23:23:32.534 INFO:tasks.workunit.client.0.smithi067.stdout:4/751: read d12/d17/d3f/d7a/fcc [806165,68251] 0 2024-04-22T23:23:32.536 INFO:tasks.workunit.client.0.smithi067.stdout:0/635: getdents db 0 2024-04-22T23:23:32.537 INFO:tasks.workunit.client.0.smithi067.stdout:0/636: chown db/d60/d9e/fc4 115353 1 2024-04-22T23:23:32.537 INFO:tasks.workunit.client.0.smithi067.stdout:7/557: mknod d10/d18/d88/daf/cc9 0 2024-04-22T23:23:32.539 INFO:tasks.workunit.client.0.smithi067.stdout:5/699: mknod d5/d3a/d4d/dc6/cf5 0 2024-04-22T23:23:32.543 INFO:tasks.workunit.client.0.smithi067.stdout:8/608: rename d1/d3/d27/c88 to d1/d41/db4/cc3 0 2024-04-22T23:23:32.544 INFO:tasks.workunit.client.0.smithi067.stdout:0/637: mkdir db/d9a/d56/dd5 0 2024-04-22T23:23:32.550 INFO:tasks.workunit.client.0.smithi067.stdout:5/700: creat d5/db/d12/ff6 x:0 0 0 2024-04-22T23:23:32.553 INFO:tasks.workunit.client.0.smithi067.stdout:4/752: getdents d12/d2e/d9b 0 2024-04-22T23:23:32.555 INFO:tasks.workunit.client.0.smithi067.stdout:3/682: truncate d3/d4/d10/d38/d53/f54 1328414 0 2024-04-22T23:23:32.556 INFO:tasks.workunit.client.0.smithi067.stdout:8/609: dwrite d1/f6 [4194304,4194304] 0 2024-04-22T23:23:32.558 INFO:tasks.workunit.client.0.smithi067.stdout:2/664: truncate d1/d12/f92 95817 0 2024-04-22T23:23:32.562 INFO:tasks.workunit.client.0.smithi067.stdout:0/638: write db/d21/d29/d75/f86 [784090,20167] 0 2024-04-22T23:23:32.563 INFO:tasks.workunit.client.0.smithi067.stdout:9/625: dwrite d5/d21/d47/d56/f80 [0,4194304] 0 2024-04-22T23:23:32.566 INFO:tasks.workunit.client.0.smithi067.stdout:2/665: rmdir d1/d42/d61 39 2024-04-22T23:23:32.568 INFO:tasks.workunit.client.0.smithi067.stdout:1/608: sync 2024-04-22T23:23:32.569 INFO:tasks.workunit.client.0.smithi067.stdout:7/558: dread d10/d14/d1d/d26/f3a [0,4194304] 0 2024-04-22T23:23:32.570 INFO:tasks.workunit.client.0.smithi067.stdout:0/639: chown db/d9a/l87 79683 1 2024-04-22T23:23:32.570 INFO:tasks.workunit.client.0.smithi067.stdout:6/648: dread d17/f30 [0,4194304] 0 2024-04-22T23:23:32.570 INFO:tasks.workunit.client.0.smithi067.stdout:3/683: creat d3/d4/d10/d38/d53/d55/d5c/ffa x:0 0 0 2024-04-22T23:23:32.573 INFO:tasks.workunit.client.0.smithi067.stdout:4/753: creat d12/d17/f100 x:0 0 0 2024-04-22T23:23:32.574 INFO:tasks.workunit.client.0.smithi067.stdout:8/610: rename d1/dd/d26/f93 to d1/dd/d80/d6f/db6/fc4 0 2024-04-22T23:23:32.576 INFO:tasks.workunit.client.0.smithi067.stdout:6/649: symlink d17/d83/ld1 0 2024-04-22T23:23:32.577 INFO:tasks.workunit.client.0.smithi067.stdout:0/640: truncate f0 2106932 0 2024-04-22T23:23:32.578 INFO:tasks.workunit.client.0.smithi067.stdout:6/650: chown d17/d39/d53/f55 0 1 2024-04-22T23:23:32.579 INFO:tasks.workunit.client.0.smithi067.stdout:1/609: creat d8/d9/d2d/d4d/d83/dc5/fcf x:0 0 0 2024-04-22T23:23:32.581 INFO:tasks.workunit.client.0.smithi067.stdout:2/666: creat d1/d10/d1c/d6d/d87/fd8 x:0 0 0 2024-04-22T23:23:32.583 INFO:tasks.workunit.client.0.smithi067.stdout:7/559: creat d10/d14/d6d/fca x:0 0 0 2024-04-22T23:23:32.585 INFO:tasks.workunit.client.0.smithi067.stdout:3/684: rename d3/d4/d10/d38/l59 to d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d62/d95/lfb 0 2024-04-22T23:23:32.586 INFO:tasks.workunit.client.0.smithi067.stdout:3/685: readlink d3/d4/d10/d19/l60 0 2024-04-22T23:23:32.588 INFO:tasks.workunit.client.0.smithi067.stdout:0/641: read db/d21/d29/d75/f83 [311125,69955] 0 2024-04-22T23:23:32.590 INFO:tasks.workunit.client.0.smithi067.stdout:1/610: creat d8/d9/d2d/d4d/d48/fd0 x:0 0 0 2024-04-22T23:23:32.590 INFO:tasks.workunit.client.0.smithi067.stdout:5/701: dread d5/d28/d36/d40/f70 [0,4194304] 0 2024-04-22T23:23:32.591 INFO:tasks.workunit.client.0.smithi067.stdout:2/667: dread - d1/d10/d72/d53/f97 zero size 2024-04-22T23:23:32.591 INFO:tasks.workunit.client.0.smithi067.stdout:4/754: truncate d12/d2e/f3e 2862578 0 2024-04-22T23:23:32.592 INFO:tasks.workunit.client.0.smithi067.stdout:1/611: chown d8/d9/d2f/d31/lbb 16609554 1 2024-04-22T23:23:32.592 INFO:tasks.workunit.client.0.smithi067.stdout:7/560: symlink d10/d18/d88/daf/lcb 0 2024-04-22T23:23:32.593 INFO:tasks.workunit.client.0.smithi067.stdout:4/755: dread - d12/d5f/fe6 zero size 2024-04-22T23:23:32.594 INFO:tasks.workunit.client.0.smithi067.stdout:6/651: creat d17/d39/d53/d90/d46/db0/db9/fd2 x:0 0 0 2024-04-22T23:23:32.595 INFO:tasks.workunit.client.0.smithi067.stdout:4/756: read - d12/d17/d3f/fc4 zero size 2024-04-22T23:23:32.596 INFO:tasks.workunit.client.0.smithi067.stdout:5/702: dread d5/d28/d36/d40/fd1 [0,4194304] 0 2024-04-22T23:23:32.597 INFO:tasks.workunit.client.0.smithi067.stdout:3/686: unlink d3/d4/d10/d38/d53/d55/f8e 0 2024-04-22T23:23:32.598 INFO:tasks.workunit.client.0.smithi067.stdout:0/642: creat db/d21/d29/d2c/fd6 x:0 0 0 2024-04-22T23:23:32.600 INFO:tasks.workunit.client.0.smithi067.stdout:0/643: write db/f6a [11957711,4428] 0 2024-04-22T23:23:32.602 INFO:tasks.workunit.client.0.smithi067.stdout:0/644: stat db/de/d11/f32 0 2024-04-22T23:23:32.605 INFO:tasks.workunit.client.0.smithi067.stdout:9/626: getdents d5/d21/d47/d56 0 2024-04-22T23:23:32.605 INFO:tasks.workunit.client.0.smithi067.stdout:9/627: chown d5/dd 22 1 2024-04-22T23:23:32.607 INFO:tasks.workunit.client.0.smithi067.stdout:2/668: mkdir d1/d10/d1c/dd9 0 2024-04-22T23:23:32.607 INFO:tasks.workunit.client.0.smithi067.stdout:3/687: dwrite d3/d4/d10/fbc [0,4194304] 0 2024-04-22T23:23:32.609 INFO:tasks.workunit.client.0.smithi067.stdout:5/703: mknod d5/d3a/d4d/dc6/cf7 0 2024-04-22T23:23:32.611 INFO:tasks.workunit.client.0.smithi067.stdout:1/612: symlink d8/d9/ld1 0 2024-04-22T23:23:32.614 INFO:tasks.workunit.client.0.smithi067.stdout:6/652: link d17/d39/f89 d17/d39/d3a/d8d/dc9/fd3 0 2024-04-22T23:23:32.621 INFO:tasks.workunit.client.0.smithi067.stdout:9/628: dread d5/dd/d18/d39/d49/f7b [0,4194304] 0 2024-04-22T23:23:32.627 INFO:tasks.workunit.client.0.smithi067.stdout:7/561: write d10/d18/f27 [4875504,67672] 0 2024-04-22T23:23:32.631 INFO:tasks.workunit.client.0.smithi067.stdout:3/688: link d3/d4/d28/d5f/fa9 d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d90/da2/daf/dd8/ffc 0 2024-04-22T23:23:32.631 INFO:tasks.workunit.client.0.smithi067.stdout:3/689: stat d3/d4/d10/d1f/d3c/f25 0 2024-04-22T23:23:32.632 INFO:tasks.workunit.client.0.smithi067.stdout:2/669: link d1/d19/c26 d1/d10/d41/cda 0 2024-04-22T23:23:32.635 INFO:tasks.workunit.client.0.smithi067.stdout:9/629: creat d5/d21/d47/d56/dc6/fd7 x:0 0 0 2024-04-22T23:23:32.636 INFO:tasks.workunit.client.0.smithi067.stdout:7/562: mkdir d10/d14/d1d/d46/d6a/dcc 0 2024-04-22T23:23:32.638 INFO:tasks.workunit.client.0.smithi067.stdout:0/645: getdents db/d9a/d56/d70/d8c 0 2024-04-22T23:23:32.639 INFO:tasks.workunit.client.0.smithi067.stdout:1/613: dread d8/d9/d2f/d59/f7b [0,4194304] 0 2024-04-22T23:23:32.639 INFO:tasks.workunit.client.0.smithi067.stdout:6/653: fsync d17/d39/f89 0 2024-04-22T23:23:32.640 INFO:tasks.workunit.client.0.smithi067.stdout:3/690: dwrite d3/d4/d1b/faa [0,4194304] 0 2024-04-22T23:23:32.641 INFO:tasks.workunit.client.0.smithi067.stdout:2/670: creat d1/d10/d33/d46/da4/fdb x:0 0 0 2024-04-22T23:23:32.641 INFO:tasks.workunit.client.0.smithi067.stdout:4/757: truncate d12/d5f/f5a 4889320 0 2024-04-22T23:23:32.642 INFO:tasks.workunit.client.0.smithi067.stdout:7/563: read d10/d14/d1d/d46/d6a/f70 [121708,18391] 0 2024-04-22T23:23:32.643 INFO:tasks.workunit.client.0.smithi067.stdout:5/704: getdents d5/db/d12/d44 0 2024-04-22T23:23:32.643 INFO:tasks.workunit.client.0.smithi067.stdout:3/691: chown d3/d4/d10/d1f/c22 5574 1 2024-04-22T23:23:32.700 INFO:tasks.workunit.client.0.smithi067.stdout:4/758: truncate d12/d5f/dae/dd0/ff4 5234 0 2024-04-22T23:23:32.700 INFO:tasks.workunit.client.0.smithi067.stdout:2/671: dread - d1/d10/d33/fa5 zero size 2024-04-22T23:23:32.700 INFO:tasks.workunit.client.0.smithi067.stdout:3/692: fsync d3/d4/fc4 0 2024-04-22T23:23:32.702 INFO:tasks.workunit.client.0.smithi067.stdout:3/693: write d3/d4/d10/d1f/d3c/f25 [7257227,22570] 0 2024-04-22T23:23:32.721 INFO:tasks.workunit.client.0.smithi067.stdout:7/564: mkdir d10/d14/d1d/d46/d50/dcd 0 2024-04-22T23:23:32.736 INFO:tasks.workunit.client.0.smithi067.stdout:3/694: unlink d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d90/da2/daf/fe4 0 2024-04-22T23:23:32.739 INFO:tasks.workunit.client.0.smithi067.stdout:0/646: getdents db/d9a/d56/d70/d8c 0 2024-04-22T23:23:32.739 INFO:tasks.workunit.client.0.smithi067.stdout:0/647: fdatasync db/d2e/f8b 0 2024-04-22T23:23:32.740 INFO:tasks.workunit.client.0.smithi067.stdout:0/648: chown db/ccd 116 1 2024-04-22T23:23:32.741 INFO:tasks.workunit.client.0.smithi067.stdout:0/649: fsync db/d21/d29/d2c/f91 0 2024-04-22T23:23:32.743 INFO:tasks.workunit.client.0.smithi067.stdout:4/759: mkdir d12/d17/d3f/d101 0 2024-04-22T23:23:32.747 INFO:tasks.workunit.client.0.smithi067.stdout:9/630: getdents d5/dd/d18/d39/d3e/d3f/d63 0 2024-04-22T23:23:32.749 INFO:tasks.workunit.client.0.smithi067.stdout:6/654: getdents d17/d83/db5 0 2024-04-22T23:23:32.749 INFO:tasks.workunit.client.0.smithi067.stdout:9/631: read d5/dd/d18/d3c/f6e [3010514,49803] 0 2024-04-22T23:23:32.749 INFO:tasks.workunit.client.0.smithi067.stdout:6/655: fsync d17/f25 0 2024-04-22T23:23:32.751 INFO:tasks.workunit.client.0.smithi067.stdout:0/650: creat db/d9a/d56/d70/dba/fd7 x:0 0 0 2024-04-22T23:23:32.752 INFO:tasks.workunit.client.0.smithi067.stdout:4/760: symlink d12/d2e/d9b/l102 0 2024-04-22T23:23:32.753 INFO:tasks.workunit.client.0.smithi067.stdout:4/761: write d12/d42/d4b/fa8 [1072367,6674] 0 2024-04-22T23:23:32.756 INFO:tasks.workunit.client.0.smithi067.stdout:6/656: creat d17/d83/db5/fd4 x:0 0 0 2024-04-22T23:23:32.757 INFO:tasks.workunit.client.0.smithi067.stdout:6/657: truncate d17/d83/fcb 574327 0 2024-04-22T23:23:32.758 INFO:tasks.workunit.client.0.smithi067.stdout:9/632: write d5/dd/d18/d3c/d40/f62 [2280908,108002] 0 2024-04-22T23:23:32.763 INFO:tasks.workunit.client.0.smithi067.stdout:4/762: rmdir d12/d5f/d2b/ded 39 2024-04-22T23:23:32.768 INFO:tasks.workunit.client.0.smithi067.stdout:3/695: getdents d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d62/d95 0 2024-04-22T23:23:32.772 INFO:tasks.workunit.client.0.smithi067.stdout:0/651: creat db/d21/fd8 x:0 0 0 2024-04-22T23:23:32.773 INFO:tasks.workunit.client.0.smithi067.stdout:3/696: dread d3/f99 [0,4194304] 0 2024-04-22T23:23:32.774 INFO:tasks.workunit.client.0.smithi067.stdout:4/763: link d12/d17/d3f/d7a/fcc d12/d17/d3f/d84/f103 0 2024-04-22T23:23:32.775 INFO:tasks.workunit.client.0.smithi067.stdout:8/611: sync 2024-04-22T23:23:32.775 INFO:tasks.workunit.client.0.smithi067.stdout:4/764: chown d12/d17/d3f/d84 80 1 2024-04-22T23:23:32.777 INFO:tasks.workunit.client.0.smithi067.stdout:0/652: mkdir db/d2e/dd9 0 2024-04-22T23:23:32.778 INFO:tasks.workunit.client.0.smithi067.stdout:4/765: chown d12/d42/d4b/d77/d9d/dc1/lec 943634 1 2024-04-22T23:23:32.779 INFO:tasks.workunit.client.0.smithi067.stdout:3/697: dwrite d3/d4/d1b/faa [0,4194304] 0 2024-04-22T23:23:32.781 INFO:tasks.workunit.client.0.smithi067.stdout:8/612: creat d1/d41/db4/fc5 x:0 0 0 2024-04-22T23:23:32.781 INFO:tasks.workunit.client.0.smithi067.stdout:3/698: write d3/d4/d28/fda [644091,88104] 0 2024-04-22T23:23:32.784 INFO:tasks.workunit.client.0.smithi067.stdout:8/613: read d1/dd/d80/d6f/f57 [1303811,91375] 0 2024-04-22T23:23:32.786 INFO:tasks.workunit.client.0.smithi067.stdout:4/766: creat d12/d42/d4b/d77/d9d/dc1/f104 x:0 0 0 2024-04-22T23:23:32.787 INFO:tasks.workunit.client.0.smithi067.stdout:4/767: fdatasync d12/d17/d47/fe5 0 2024-04-22T23:23:32.787 INFO:tasks.workunit.client.0.smithi067.stdout:3/699: mkdir d3/d4/d28/d40/dfd 0 2024-04-22T23:23:32.788 INFO:tasks.workunit.client.0.smithi067.stdout:8/614: creat d1/dd/d26/d62/d94/fc6 x:0 0 0 2024-04-22T23:23:32.792 INFO:tasks.workunit.client.0.smithi067.stdout:4/768: write d12/d17/d3f/d7a/fcc [3533222,107662] 0 2024-04-22T23:23:32.794 INFO:tasks.workunit.client.0.smithi067.stdout:0/653: getdents db/d71 0 2024-04-22T23:23:32.796 INFO:tasks.workunit.client.0.smithi067.stdout:8/615: dwrite d1/d41/d46/fb0 [0,4194304] 0 2024-04-22T23:23:32.797 INFO:tasks.workunit.client.0.smithi067.stdout:0/654: mkdir db/d71/dda 0 2024-04-22T23:23:32.798 INFO:tasks.workunit.client.0.smithi067.stdout:0/655: fsync db/d21/d29/d2c/f91 0 2024-04-22T23:23:32.800 INFO:tasks.workunit.client.0.smithi067.stdout:0/656: write db/d9a/d56/d70/dba/fbd [8463,85723] 0 2024-04-22T23:23:32.800 INFO:tasks.workunit.client.0.smithi067.stdout:8/616: fsync d1/f7a 0 2024-04-22T23:23:32.802 INFO:tasks.workunit.client.0.smithi067.stdout:0/657: dread - db/d21/fd8 zero size 2024-04-22T23:23:32.808 INFO:tasks.workunit.client.0.smithi067.stdout:0/658: dread db/fd [0,4194304] 0 2024-04-22T23:23:32.809 INFO:tasks.workunit.client.0.smithi067.stdout:7/565: write d10/d14/d1d/d26/d7e/fb9 [2414885,71423] 0 2024-04-22T23:23:32.810 INFO:tasks.workunit.client.0.smithi067.stdout:9/633: mknod d5/dd/d18/d39/d3e/cd8 0 2024-04-22T23:23:32.811 INFO:tasks.workunit.client.0.smithi067.stdout:0/659: symlink db/de/d11/ldb 0 2024-04-22T23:23:32.813 INFO:tasks.workunit.client.0.smithi067.stdout:7/566: creat d10/d14/d1d/d69/fce x:0 0 0 2024-04-22T23:23:32.814 INFO:tasks.workunit.client.0.smithi067.stdout:7/567: fdatasync d10/d14/d1d/d26/d2c/d7a/fa5 0 2024-04-22T23:23:32.818 INFO:tasks.workunit.client.0.smithi067.stdout:0/660: link db/d21/f8d db/de/fdc 0 2024-04-22T23:23:32.821 INFO:tasks.workunit.client.0.smithi067.stdout:7/568: dwrite d10/d14/d1d/d46/d6a/f86 [0,4194304] 0 2024-04-22T23:23:32.826 INFO:tasks.workunit.client.0.smithi067.stdout:7/569: creat d10/d14/d1d/d26/d2c/fcf x:0 0 0 2024-04-22T23:23:32.826 INFO:tasks.workunit.client.0.smithi067.stdout:9/634: dread d5/fad [0,4194304] 0 2024-04-22T23:23:32.827 INFO:tasks.workunit.client.0.smithi067.stdout:9/635: readlink d5/dd/d18/lbb 0 2024-04-22T23:23:32.829 INFO:tasks.workunit.client.0.smithi067.stdout:3/700: rename d3/d4/d12 to d3/d4/d28/d40/dfd/dfe 0 2024-04-22T23:23:32.829 INFO:tasks.workunit.client.0.smithi067.stdout:9/636: write d5/dd/d18/d3c/d40/f62 [4006813,42507] 0 2024-04-22T23:23:32.832 INFO:tasks.workunit.client.0.smithi067.stdout:2/672: mkdir d1/d10/d1c/ddc 0 2024-04-22T23:23:32.833 INFO:tasks.workunit.client.0.smithi067.stdout:7/570: mkdir d10/d14/d1d/d26/d2c/d7a/dc2/dd0 0 2024-04-22T23:23:32.835 INFO:tasks.workunit.client.0.smithi067.stdout:9/637: readlink d5/l5a 0 2024-04-22T23:23:32.836 INFO:tasks.workunit.client.0.smithi067.stdout:8/617: rename d1/dd/d26/d62/f6d to d1/dd/d80/fc7 0 2024-04-22T23:23:32.838 INFO:tasks.workunit.client.0.smithi067.stdout:3/701: symlink d3/d4/d28/d5f/lff 0 2024-04-22T23:23:32.838 INFO:tasks.workunit.client.0.smithi067.stdout:6/658: dwrite fe [0,4194304] 0 2024-04-22T23:23:32.840 INFO:tasks.workunit.client.0.smithi067.stdout:8/618: write d1/dd/d80/f8d [886977,12804] 0 2024-04-22T23:23:32.840 INFO:tasks.workunit.client.0.smithi067.stdout:3/702: chown d3/d4/d10/fbc 0 1 2024-04-22T23:23:32.844 INFO:tasks.workunit.client.0.smithi067.stdout:3/703: write d3/d4/d10/d38/d53/d55/d5c/d70/f9b [510729,22481] 0 2024-04-22T23:23:32.845 INFO:tasks.workunit.client.0.smithi067.stdout:7/571: symlink d10/d14/d1d/d46/d6a/d91/ld1 0 2024-04-22T23:23:32.847 INFO:tasks.workunit.client.0.smithi067.stdout:0/661: rename db/de/d11/d1a/f64 to db/de/d11/d92/fdd 0 2024-04-22T23:23:32.853 INFO:tasks.workunit.client.0.smithi067.stdout:7/572: fdatasync d10/d14/d1d/d26/d2c/d7a/f87 0 2024-04-22T23:23:32.853 INFO:tasks.workunit.client.0.smithi067.stdout:1/614: sync 2024-04-22T23:23:32.853 INFO:tasks.workunit.client.0.smithi067.stdout:5/705: sync 2024-04-22T23:23:32.854 INFO:tasks.workunit.client.0.smithi067.stdout:3/704: dwrite d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d8c/ff9 [0,4194304] 0 2024-04-22T23:23:32.859 INFO:tasks.workunit.client.0.smithi067.stdout:1/615: stat d8/d9/d2d/d4d/d83/fb9 0 2024-04-22T23:23:32.859 INFO:tasks.workunit.client.0.smithi067.stdout:7/573: dread - d10/d14/d1d/d26/d80/fbb zero size 2024-04-22T23:23:32.859 INFO:tasks.workunit.client.0.smithi067.stdout:3/705: fsync d3/d4/d10/d38/d53/d55/d5c/f9e 0 2024-04-22T23:23:32.859 INFO:tasks.workunit.client.0.smithi067.stdout:0/662: mkdir db/d21/d29/d75/d9d/dde 0 2024-04-22T23:23:32.862 INFO:tasks.workunit.client.0.smithi067.stdout:6/659: unlink d17/d39/d53/d90/d46/d62/c6e 0 2024-04-22T23:23:32.866 INFO:tasks.workunit.client.0.smithi067.stdout:4/769: truncate d12/d17/d3f/d84/f9f 1171156 0 2024-04-22T23:23:32.867 INFO:tasks.workunit.client.0.smithi067.stdout:8/619: creat d1/dd/d26/d62/fc8 x:0 0 0 2024-04-22T23:23:32.867 INFO:tasks.workunit.client.0.smithi067.stdout:0/663: dwrite db/d60/d9e/fc4 [0,4194304] 0 2024-04-22T23:23:32.869 INFO:tasks.workunit.client.0.smithi067.stdout:8/620: chown d1/d3/f1e 922 1 2024-04-22T23:23:32.870 INFO:tasks.workunit.client.0.smithi067.stdout:8/621: write d1/dd/d80/d6f/fad [944956,105402] 0 2024-04-22T23:23:32.870 INFO:tasks.workunit.client.0.smithi067.stdout:7/574: mkdir d10/d14/d1d/d26/d80/dd2 0 2024-04-22T23:23:32.872 INFO:tasks.workunit.client.0.smithi067.stdout:3/706: fsync d3/d4/d28/d40/dfd/dfe/f27 0 2024-04-22T23:23:32.872 INFO:tasks.workunit.client.0.smithi067.stdout:7/575: dread - d10/d14/d1d/f76 zero size 2024-04-22T23:23:32.874 INFO:tasks.workunit.client.0.smithi067.stdout:7/576: chown d10/d47/d49/f8b 33641 1 2024-04-22T23:23:32.876 INFO:tasks.workunit.client.0.smithi067.stdout:9/638: getdents d5/d21/d47/d56/d8c 0 2024-04-22T23:23:32.882 INFO:tasks.workunit.client.0.smithi067.stdout:0/664: rename db/d2e/ca0 to db/d71/dda/cdf 0 2024-04-22T23:23:32.883 INFO:tasks.workunit.client.0.smithi067.stdout:5/706: dwrite d5/db/d35/f39 [0,4194304] 0 2024-04-22T23:23:32.883 INFO:tasks.workunit.client.0.smithi067.stdout:0/665: read db/fd [894993,108324] 0 2024-04-22T23:23:32.886 INFO:tasks.workunit.client.0.smithi067.stdout:6/660: mknod d17/d39/d3a/cd5 0 2024-04-22T23:23:32.887 INFO:tasks.workunit.client.0.smithi067.stdout:4/770: dread d12/d17/d31/f50 [0,4194304] 0 2024-04-22T23:23:32.888 INFO:tasks.workunit.client.0.smithi067.stdout:2/673: write d1/d10/d72/f36 [24994,781] 0 2024-04-22T23:23:32.892 INFO:tasks.workunit.client.0.smithi067.stdout:6/661: dread d17/d39/d3a/d8d/f97 [0,4194304] 0 2024-04-22T23:23:32.898 INFO:tasks.workunit.client.0.smithi067.stdout:3/707: mknod d3/d4/d10/d38/d53/d55/d74/dd5/ddb/c100 0 2024-04-22T23:23:32.902 INFO:tasks.workunit.client.0.smithi067.stdout:1/616: getdents d8/d9/d2d 0 2024-04-22T23:23:32.904 INFO:tasks.workunit.client.0.smithi067.stdout:1/617: write d8/d9/d2f/f7d [561282,93462] 0 2024-04-22T23:23:32.904 INFO:tasks.workunit.client.0.smithi067.stdout:6/662: stat d17/d39/l67 0 2024-04-22T23:23:32.906 INFO:tasks.workunit.client.0.smithi067.stdout:0/666: symlink db/d9a/d56/d70/d8c/dc2/le0 0 2024-04-22T23:23:32.908 INFO:tasks.workunit.client.0.smithi067.stdout:3/708: fsync d3/d4/d28/d4a/f73 0 2024-04-22T23:23:32.909 INFO:tasks.workunit.client.0.smithi067.stdout:7/577: link d10/d14/d1d/d24/l57 d10/d18/d88/daf/ld3 0 2024-04-22T23:23:32.910 INFO:tasks.workunit.client.0.smithi067.stdout:3/709: chown d3/d4/d1b/c88 2 1 2024-04-22T23:23:32.912 INFO:tasks.workunit.client.0.smithi067.stdout:9/639: unlink d5/dd/d18/d39/d3e/c5b 0 2024-04-22T23:23:32.914 INFO:tasks.workunit.client.0.smithi067.stdout:2/674: unlink d1/d10/d1c/d6d/d87/l9f 0 2024-04-22T23:23:32.916 INFO:tasks.workunit.client.0.smithi067.stdout:1/618: rmdir d8/d9/d2f/d31 39 2024-04-22T23:23:32.916 INFO:tasks.workunit.client.0.smithi067.stdout:8/622: dwrite d1/f18 [0,4194304] 0 2024-04-22T23:23:32.918 INFO:tasks.workunit.client.0.smithi067.stdout:3/710: readlink d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d62/d95/dd6/le5 0 2024-04-22T23:23:32.919 INFO:tasks.workunit.client.0.smithi067.stdout:6/663: mknod d17/d39/d3a/da9/cd6 0 2024-04-22T23:23:32.921 INFO:tasks.workunit.client.0.smithi067.stdout:5/707: dwrite d5/d28/d36/d40/f70 [0,4194304] 0 2024-04-22T23:23:32.923 INFO:tasks.workunit.client.0.smithi067.stdout:6/664: write d17/d39/d3a/f6d [1106726,14965] 0 2024-04-22T23:23:32.924 INFO:tasks.workunit.client.0.smithi067.stdout:3/711: dread - d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d8c/fb6 zero size 2024-04-22T23:23:32.926 INFO:tasks.workunit.client.0.smithi067.stdout:7/578: dwrite d10/d18/d35/d62/f94 [0,4194304] 0 2024-04-22T23:23:32.931 INFO:tasks.workunit.client.0.smithi067.stdout:0/667: creat db/d2e/dd9/fe1 x:0 0 0 2024-04-22T23:23:32.933 INFO:tasks.workunit.client.0.smithi067.stdout:0/668: write db/de/d2a/f99 [67992,35410] 0 2024-04-22T23:23:32.936 INFO:tasks.workunit.client.0.smithi067.stdout:9/640: mknod d5/dd/d18/cd9 0 2024-04-22T23:23:32.937 INFO:tasks.workunit.client.0.smithi067.stdout:5/708: creat d5/db/d12/d44/ff8 x:0 0 0 2024-04-22T23:23:32.938 INFO:tasks.workunit.client.0.smithi067.stdout:2/675: truncate d1/f35 1017085 0 2024-04-22T23:23:32.939 INFO:tasks.workunit.client.0.smithi067.stdout:5/709: write d5/db/d12/dbb/d61/f82 [1285229,81326] 0 2024-04-22T23:23:32.940 INFO:tasks.workunit.client.0.smithi067.stdout:6/665: mknod d17/d39/d3a/da9/cd7 0 2024-04-22T23:23:32.945 INFO:tasks.workunit.client.0.smithi067.stdout:9/641: write d5/d21/d47/d56/d8c/dd5/fc1 [5809369,62798] 0 2024-04-22T23:23:32.947 INFO:tasks.workunit.client.0.smithi067.stdout:5/710: read - d5/f52 zero size 2024-04-22T23:23:32.949 INFO:tasks.workunit.client.0.smithi067.stdout:7/579: mkdir d10/d14/d1d/d26/d80/dd2/dd4 0 2024-04-22T23:23:32.950 INFO:tasks.workunit.client.0.smithi067.stdout:7/580: dread - d10/d14/d1d/d69/fce zero size 2024-04-22T23:23:32.951 INFO:tasks.workunit.client.0.smithi067.stdout:1/619: link d8/d9/d2d/d4d/d48/d9a/cb3 d8/d9/d2d/d4d/d9c/cd2 0 2024-04-22T23:23:32.952 INFO:tasks.workunit.client.0.smithi067.stdout:0/669: symlink db/d9a/d56/dcf/le2 0 2024-04-22T23:23:32.953 INFO:tasks.workunit.client.0.smithi067.stdout:1/620: dread - d8/d9/d2d/d4d/d48/fce zero size 2024-04-22T23:23:32.954 INFO:tasks.workunit.client.0.smithi067.stdout:0/670: fsync db/d21/d29/f62 0 2024-04-22T23:23:32.955 INFO:tasks.workunit.client.0.smithi067.stdout:9/642: rename d5/dd/d18/d39/d4d/lcf to d5/dd/da9/lda 0 2024-04-22T23:23:32.957 INFO:tasks.workunit.client.0.smithi067.stdout:5/711: dwrite d5/d3a/fd2 [0,4194304] 0 2024-04-22T23:23:32.964 INFO:tasks.workunit.client.0.smithi067.stdout:8/623: dwrite d1/dd/d26/d62/f91 [4194304,4194304] 0 2024-04-22T23:23:32.966 INFO:tasks.workunit.client.0.smithi067.stdout:8/624: stat d1/dd/c5f 0 2024-04-22T23:23:32.971 INFO:tasks.workunit.client.0.smithi067.stdout:8/625: chown d1/dd/f53 0 1 2024-04-22T23:23:32.972 INFO:tasks.workunit.client.0.smithi067.stdout:3/712: write d3/d4/d10/d38/f57 [163780,88181] 0 2024-04-22T23:23:32.973 INFO:tasks.workunit.client.0.smithi067.stdout:5/712: dwrite d5/db/d35/f43 [4194304,4194304] 0 2024-04-22T23:23:32.981 INFO:tasks.workunit.client.0.smithi067.stdout:8/626: dwrite d1/f6 [0,4194304] 0 2024-04-22T23:23:32.982 INFO:tasks.workunit.client.0.smithi067.stdout:0/671: unlink db/d2e/d46/f82 0 2024-04-22T23:23:32.982 INFO:tasks.workunit.client.0.smithi067.stdout:2/676: read d1/d10/f2e [1789438,27943] 0 2024-04-22T23:23:32.983 INFO:tasks.workunit.client.0.smithi067.stdout:6/666: mkdir d17/d83/db5/dbc/dd8 0 2024-04-22T23:23:32.983 INFO:tasks.workunit.client.0.smithi067.stdout:7/581: write fc [2625415,74079] 0 2024-04-22T23:23:32.985 INFO:tasks.workunit.client.0.smithi067.stdout:7/582: fdatasync d10/d18/d35/f5b 0 2024-04-22T23:23:32.985 INFO:tasks.workunit.client.0.smithi067.stdout:3/713: fsync d3/d4/d28/d40/dfd/dfe/f2f 0 2024-04-22T23:23:32.989 INFO:tasks.workunit.client.0.smithi067.stdout:8/627: creat d1/d3/d5c/d98/fc9 x:0 0 0 2024-04-22T23:23:32.990 INFO:tasks.workunit.client.0.smithi067.stdout:1/621: creat d8/d9/d2d/d4d/d70/d34/d60/fd3 x:0 0 0 2024-04-22T23:23:32.993 INFO:tasks.workunit.client.0.smithi067.stdout:0/672: rename db/d9a/d56/d70/d8c/ca3 to db/d9a/d56/d70/d8c/ce3 0 2024-04-22T23:23:32.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:32 smithi134 ceph-mon[34609]: pgmap v271: 65 pgs: 65 active+clean; 1.4 GiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 44 MiB/s rd, 177 MiB/s wr, 373 op/s 2024-04-22T23:23:32.994 INFO:tasks.workunit.client.0.smithi067.stdout:1/622: dread - d8/d12/d40/f57 zero size 2024-04-22T23:23:32.996 INFO:tasks.workunit.client.0.smithi067.stdout:0/673: fdatasync db/de/d11/d1a/da4/f8e 0 2024-04-22T23:23:32.996 INFO:tasks.workunit.client.0.smithi067.stdout:5/713: dwrite d5/d3a/f5d [0,4194304] 0 2024-04-22T23:23:32.996 INFO:tasks.workunit.client.0.smithi067.stdout:3/714: dread d3/d4/d28/d4a/f73 [0,4194304] 0 2024-04-22T23:23:32.999 INFO:tasks.workunit.client.0.smithi067.stdout:5/714: write d5/f11 [1166767,49444] 0 2024-04-22T23:23:33.000 INFO:tasks.workunit.client.0.smithi067.stdout:9/643: rename d5/dd/d18/d39/d49/c4b to d5/dd/d18/cdb 0 2024-04-22T23:23:33.003 INFO:tasks.workunit.client.0.smithi067.stdout:5/715: chown d5/fa 31741 1 2024-04-22T23:23:33.005 INFO:tasks.workunit.client.0.smithi067.stdout:1/623: mkdir d8/d9/d2d/d4d/d70/d34/d5d/dd4 0 2024-04-22T23:23:33.007 INFO:tasks.workunit.client.0.smithi067.stdout:7/583: creat d10/d47/dab/fd5 x:0 0 0 2024-04-22T23:23:33.009 INFO:tasks.workunit.client.0.smithi067.stdout:7/584: write d10/d18/d35/f5e [4155160,70728] 0 2024-04-22T23:23:33.009 INFO:tasks.workunit.client.0.smithi067.stdout:9/644: rename d5/dd/d18/lbb to d5/d21/d47/d56/ldc 0 2024-04-22T23:23:33.011 INFO:tasks.workunit.client.0.smithi067.stdout:6/667: creat d17/d39/fd9 x:0 0 0 2024-04-22T23:23:33.011 INFO:tasks.workunit.client.0.smithi067.stdout:9/645: chown d5/d21/d47/d57/f97 423631912 1 2024-04-22T23:23:33.011 INFO:tasks.workunit.client.0.smithi067.stdout:8/628: mknod d1/dd/d80/db9/cca 0 2024-04-22T23:23:33.012 INFO:tasks.workunit.client.0.smithi067.stdout:0/674: getdents db/d9a/d56/dd5 0 2024-04-22T23:23:33.018 INFO:tasks.workunit.client.0.smithi067.stdout:7/585: symlink d10/d14/d1d/d26/d2c/d7a/ld6 0 2024-04-22T23:23:33.024 INFO:tasks.workunit.client.0.smithi067.stdout:9/646: dwrite d5/f10 [0,4194304] 0 2024-04-22T23:23:33.026 INFO:tasks.workunit.client.0.smithi067.stdout:5/716: creat d5/d28/ff9 x:0 0 0 2024-04-22T23:23:33.029 INFO:tasks.workunit.client.0.smithi067.stdout:7/586: chown d10/d14/d1d/d26/d2c/d7a/c92 1064 1 2024-04-22T23:23:33.029 INFO:tasks.workunit.client.0.smithi067.stdout:6/668: dwrite d17/d39/d53/d90/d46/fa2 [0,4194304] 0 2024-04-22T23:23:33.029 INFO:tasks.workunit.client.0.smithi067.stdout:8/629: dwrite d1/d3/f2a [0,4194304] 0 2024-04-22T23:23:33.029 INFO:tasks.workunit.client.0.smithi067.stdout:2/677: rename d1/d19/f21 to d1/d10/d1c/d6d/d87/dc8/dc5/fdd 0 2024-04-22T23:23:33.032 INFO:tasks.workunit.client.0.smithi067.stdout:8/630: chown d1/dd/d26/d62/fc8 3229 1 2024-04-22T23:23:33.035 INFO:tasks.workunit.client.0.smithi067.stdout:1/624: dread d8/d9/d2f/d33/d5c/f6d [0,4194304] 0 2024-04-22T23:23:33.037 INFO:tasks.workunit.client.0.smithi067.stdout:0/675: write db/d21/d29/f3f [3203051,75598] 0 2024-04-22T23:23:33.044 INFO:tasks.workunit.client.0.smithi067.stdout:9/647: rename d5/dd/d18/d3c/fb4 to d5/d21/d47/d56/d8c/dd5/fdd 0 2024-04-22T23:23:33.044 INFO:tasks.workunit.client.0.smithi067.stdout:1/625: dwrite d8/f1d [0,4194304] 0 2024-04-22T23:23:33.045 INFO:tasks.workunit.client.0.smithi067.stdout:1/626: fdatasync d8/f26 0 2024-04-22T23:23:33.046 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:32 smithi067 ceph-mon[26453]: pgmap v271: 65 pgs: 65 active+clean; 1.4 GiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 44 MiB/s rd, 177 MiB/s wr, 373 op/s 2024-04-22T23:23:33.047 INFO:tasks.workunit.client.0.smithi067.stdout:3/715: truncate d3/d4/d10/fbc 1261492 0 2024-04-22T23:23:33.048 INFO:tasks.workunit.client.0.smithi067.stdout:7/587: mknod d10/d47/d49/cd7 0 2024-04-22T23:23:33.051 INFO:tasks.workunit.client.0.smithi067.stdout:3/716: rename d3/d4/d10/d38/d53 to d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d51/d101 22 2024-04-22T23:23:33.053 INFO:tasks.workunit.client.0.smithi067.stdout:0/676: chown db/de/d2a/c59 393 1 2024-04-22T23:23:33.055 INFO:tasks.workunit.client.0.smithi067.stdout:2/678: mkdir d1/d10/d33/d46/da4/dde 0 2024-04-22T23:23:33.057 INFO:tasks.workunit.client.0.smithi067.stdout:9/648: creat d5/dd/d18/d5c/fde x:0 0 0 2024-04-22T23:23:33.059 INFO:tasks.workunit.client.0.smithi067.stdout:7/588: write fe [3680288,59077] 0 2024-04-22T23:23:33.059 INFO:tasks.workunit.client.0.smithi067.stdout:8/631: getdents d1/dd/d80/d6f/da8/db8 0 2024-04-22T23:23:33.062 INFO:tasks.workunit.client.0.smithi067.stdout:9/649: chown d5/dd/d18/d39/d3e/c61 50 1 2024-04-22T23:23:33.064 INFO:tasks.workunit.client.0.smithi067.stdout:8/632: mknod d1/dd/d80/db9/ccb 0 2024-04-22T23:23:33.064 INFO:tasks.workunit.client.0.smithi067.stdout:0/677: symlink db/d9a/d56/d6b/db9/le4 0 2024-04-22T23:23:33.065 INFO:tasks.workunit.client.0.smithi067.stdout:8/633: chown d1/d3/d7e/l87 219559 1 2024-04-22T23:23:33.065 INFO:tasks.workunit.client.0.smithi067.stdout:2/679: creat d1/d10/d1c/ddc/fdf x:0 0 0 2024-04-22T23:23:33.067 INFO:tasks.workunit.client.0.smithi067.stdout:7/589: symlink d10/d14/d1d/d26/ld8 0 2024-04-22T23:23:33.074 INFO:tasks.workunit.client.0.smithi067.stdout:1/627: getdents d8/d9/d2f/d33 0 2024-04-22T23:23:33.075 INFO:tasks.workunit.client.0.smithi067.stdout:5/717: dread d5/d3e/f77 [0,4194304] 0 2024-04-22T23:23:33.075 INFO:tasks.workunit.client.0.smithi067.stdout:8/634: truncate d1/f7a 624524 0 2024-04-22T23:23:33.075 INFO:tasks.workunit.client.0.smithi067.stdout:1/628: dread d8/d9/fa9 [0,4194304] 0 2024-04-22T23:23:33.076 INFO:tasks.workunit.client.0.smithi067.stdout:7/590: dwrite d10/d18/d35/f5b [0,4194304] 0 2024-04-22T23:23:33.077 INFO:tasks.workunit.client.0.smithi067.stdout:9/650: mkdir d5/dd/d18/d3c/ddf 0 2024-04-22T23:23:33.078 INFO:tasks.workunit.client.0.smithi067.stdout:3/717: write d3/d4/d28/d4a/f73 [1490183,7249] 0 2024-04-22T23:23:33.078 INFO:tasks.workunit.client.0.smithi067.stdout:7/591: chown d10/c17 2567 1 2024-04-22T23:23:33.081 INFO:tasks.workunit.client.0.smithi067.stdout:2/680: symlink d1/d12/le0 0 2024-04-22T23:23:33.082 INFO:tasks.workunit.client.0.smithi067.stdout:3/718: dread - d3/d4/d10/d38/d53/d55/d5c/f87 zero size 2024-04-22T23:23:33.084 INFO:tasks.workunit.client.0.smithi067.stdout:5/718: rmdir d5/d3a/d4d 39 2024-04-22T23:23:33.085 INFO:tasks.workunit.client.0.smithi067.stdout:3/719: read d3/d4/d28/fda [88202,3149] 0 2024-04-22T23:23:33.086 INFO:tasks.workunit.client.0.smithi067.stdout:5/719: chown d5/db/d12/dbb/d61/f82 1805641 1 2024-04-22T23:23:33.089 INFO:tasks.workunit.client.0.smithi067.stdout:2/681: dread - d1/d42/d61/dad/fd3 zero size 2024-04-22T23:23:33.091 INFO:tasks.workunit.client.0.smithi067.stdout:0/678: link db/l18 db/d21/d29/d2c/le5 0 2024-04-22T23:23:33.092 INFO:tasks.workunit.client.0.smithi067.stdout:0/679: fdatasync db/d21/d29/d75/f86 0 2024-04-22T23:23:33.095 INFO:tasks.workunit.client.0.smithi067.stdout:0/680: dread db/fd [0,4194304] 0 2024-04-22T23:23:33.096 INFO:tasks.workunit.client.0.smithi067.stdout:8/635: dread d1/dd/d80/d6f/f32 [0,4194304] 0 2024-04-22T23:23:33.096 INFO:tasks.workunit.client.0.smithi067.stdout:0/681: dread - db/d9a/d56/fca zero size 2024-04-22T23:23:33.097 INFO:tasks.workunit.client.0.smithi067.stdout:9/651: symlink d5/dd/d18/d39/d3e/le0 0 2024-04-22T23:23:33.100 INFO:tasks.workunit.client.0.smithi067.stdout:7/592: truncate d10/d14/d1d/d26/f40 201583 0 2024-04-22T23:23:33.102 INFO:tasks.workunit.client.0.smithi067.stdout:9/652: dread d5/d21/d47/d56/f80 [0,4194304] 0 2024-04-22T23:23:33.102 INFO:tasks.workunit.client.0.smithi067.stdout:3/720: write d3/d4/d28/d4a/ff7 [2228616,43290] 0 2024-04-22T23:23:33.103 INFO:tasks.workunit.client.0.smithi067.stdout:7/593: write d10/d14/d1d/d26/d2c/d41/f65 [1009196,17991] 0 2024-04-22T23:23:33.105 INFO:tasks.workunit.client.0.smithi067.stdout:1/629: creat d8/d9/d2d/d4d/d70/fd5 x:0 0 0 2024-04-22T23:23:33.106 INFO:tasks.workunit.client.0.smithi067.stdout:7/594: read - d10/d14/d1d/d26/d80/f8a zero size 2024-04-22T23:23:33.108 INFO:tasks.workunit.client.0.smithi067.stdout:5/720: creat d5/db/d12/dbb/d5f/d79/dba/de0/ffa x:0 0 0 2024-04-22T23:23:33.111 INFO:tasks.workunit.client.0.smithi067.stdout:5/721: truncate d5/d28/daf/dbf/dcf/fc7 76155 0 2024-04-22T23:23:33.115 INFO:tasks.workunit.client.0.smithi067.stdout:8/636: dread d1/dd/d80/d6f/f30 [0,4194304] 0 2024-04-22T23:23:33.118 INFO:tasks.workunit.client.0.smithi067.stdout:7/595: fsync d10/f15 0 2024-04-22T23:23:33.121 INFO:tasks.workunit.client.0.smithi067.stdout:5/722: dwrite d5/d28/d36/d40/d97/fcd [0,4194304] 0 2024-04-22T23:23:33.124 INFO:tasks.workunit.client.0.smithi067.stdout:0/682: dwrite db/de/d2a/f39 [0,4194304] 0 2024-04-22T23:23:33.126 INFO:tasks.workunit.client.0.smithi067.stdout:3/721: symlink d3/d4/d10/d38/d53/d55/d74/dd5/ddb/l102 0 2024-04-22T23:23:33.132 INFO:tasks.workunit.client.0.smithi067.stdout:8/637: write d1/d3/d27/f82 [963475,93004] 0 2024-04-22T23:23:33.132 INFO:tasks.workunit.client.0.smithi067.stdout:7/596: symlink d10/d18/d88/daf/ld9 0 2024-04-22T23:23:33.135 INFO:tasks.workunit.client.0.smithi067.stdout:1/630: rename d8/d9/d2d/d4d/d9c/fca to d8/d12/fd6 0 2024-04-22T23:23:33.137 INFO:tasks.workunit.client.0.smithi067.stdout:9/653: getdents d5/dd/d18/d39/d49/d54/d99 0 2024-04-22T23:23:33.139 INFO:tasks.workunit.client.0.smithi067.stdout:2/682: dwrite d1/d10/d72/d53/d9a/fb1 [0,4194304] 0 2024-04-22T23:23:33.141 INFO:tasks.workunit.client.0.smithi067.stdout:4/771: sync 2024-04-22T23:23:33.144 INFO:tasks.workunit.client.0.smithi067.stdout:8/638: mknod d1/dd/d80/d6f/da8/db8/ccc 0 2024-04-22T23:23:33.145 INFO:tasks.workunit.client.0.smithi067.stdout:4/772: readlink d12/d5f/d2b/lb8 0 2024-04-22T23:23:33.145 INFO:tasks.workunit.client.0.smithi067.stdout:5/723: dwrite d5/d28/daf/dbf/dcf/db2/fc1 [0,4194304] 0 2024-04-22T23:23:33.149 INFO:tasks.workunit.client.0.smithi067.stdout:0/683: getdents db/d9a/d56/d70/d8c 0 2024-04-22T23:23:33.149 INFO:tasks.workunit.client.0.smithi067.stdout:3/722: dread d3/d4/d10/d38/d53/d55/f66 [0,4194304] 0 2024-04-22T23:23:33.153 INFO:tasks.workunit.client.0.smithi067.stdout:8/639: fdatasync d1/dd/d80/d6f/fad 0 2024-04-22T23:23:33.153 INFO:tasks.workunit.client.0.smithi067.stdout:9/654: symlink d5/d21/d47/le1 0 2024-04-22T23:23:33.154 INFO:tasks.workunit.client.0.smithi067.stdout:7/597: link d10/d14/d1d/d26/d2c/f2e d10/d14/d1d/d26/d7e/fda 0 2024-04-22T23:23:33.158 INFO:tasks.workunit.client.0.smithi067.stdout:1/631: dwrite d8/d9/d2d/d4d/d70/d34/d60/d62/fa5 [0,4194304] 0 2024-04-22T23:23:33.167 INFO:tasks.workunit.client.0.smithi067.stdout:9/655: creat d5/d21/d47/d56/d8c/dd5/fe2 x:0 0 0 2024-04-22T23:23:33.169 INFO:tasks.workunit.client.0.smithi067.stdout:0/684: dread db/d21/d29/d2c/f7d [0,4194304] 0 2024-04-22T23:23:33.172 INFO:tasks.workunit.client.0.smithi067.stdout:5/724: dread d5/d3a/d4d/fb1 [0,4194304] 0 2024-04-22T23:23:33.173 INFO:tasks.workunit.client.0.smithi067.stdout:2/683: write d1/d10/d1c/d6d/d87/dc8/dc5/fdd [1732187,45767] 0 2024-04-22T23:23:33.173 INFO:tasks.workunit.client.0.smithi067.stdout:5/725: chown d5/d28/d36/d40/d97/dc8/fce 8667 1 2024-04-22T23:23:33.175 INFO:tasks.workunit.client.0.smithi067.stdout:1/632: symlink d8/d9/d2f/d33/ld7 0 2024-04-22T23:23:33.178 INFO:tasks.workunit.client.0.smithi067.stdout:0/685: readlink la 0 2024-04-22T23:23:33.183 INFO:tasks.workunit.client.0.smithi067.stdout:5/726: rename d5/d3e/d87/d99/lb8 to d5/d3e/d90/lfb 0 2024-04-22T23:23:33.186 INFO:tasks.workunit.client.0.smithi067.stdout:9/656: creat d5/dd/d18/d39/d3e/d3f/d63/fe3 x:0 0 0 2024-04-22T23:23:33.187 INFO:tasks.workunit.client.0.smithi067.stdout:2/684: unlink d1/d10/f52 0 2024-04-22T23:23:33.190 INFO:tasks.workunit.client.0.smithi067.stdout:4/773: dwrite d12/f38 [0,4194304] 0 2024-04-22T23:23:33.191 INFO:tasks.workunit.client.0.smithi067.stdout:7/598: dread d10/d47/d49/f9c [0,4194304] 0 2024-04-22T23:23:33.194 INFO:tasks.workunit.client.0.smithi067.stdout:6/669: sync 2024-04-22T23:23:33.197 INFO:tasks.workunit.client.0.smithi067.stdout:3/723: dread d3/d4/d10/d38/d53/d55/f75 [0,4194304] 0 2024-04-22T23:23:33.199 INFO:tasks.workunit.client.0.smithi067.stdout:9/657: mknod d5/d21/d47/d7f/ce4 0 2024-04-22T23:23:33.199 INFO:tasks.workunit.client.0.smithi067.stdout:4/774: write d12/f38 [3623877,26687] 0 2024-04-22T23:23:33.205 INFO:tasks.workunit.client.0.smithi067.stdout:0/686: dwrite db/d2e/dd9/fe1 [0,4194304] 0 2024-04-22T23:23:33.206 INFO:tasks.workunit.client.0.smithi067.stdout:1/633: dwrite d8/d9/d2d/d4d/d9c/f8a [0,4194304] 0 2024-04-22T23:23:33.207 INFO:tasks.workunit.client.0.smithi067.stdout:8/640: dwrite d1/dd/d26/d62/fa5 [0,4194304] 0 2024-04-22T23:23:33.209 INFO:tasks.workunit.client.0.smithi067.stdout:7/599: symlink d10/d18/d88/daf/ldb 0 2024-04-22T23:23:33.211 INFO:tasks.workunit.client.0.smithi067.stdout:3/724: dread - d3/d4/d10/d38/d53/d55/d74/ff6 zero size 2024-04-22T23:23:33.213 INFO:tasks.workunit.client.0.smithi067.stdout:6/670: creat d17/d83/fda x:0 0 0 2024-04-22T23:23:33.214 INFO:tasks.workunit.client.0.smithi067.stdout:5/727: dwrite d5/d3e/f77 [0,4194304] 0 2024-04-22T23:23:33.217 INFO:tasks.workunit.client.0.smithi067.stdout:9/658: mkdir d5/d21/d47/d56/de5 0 2024-04-22T23:23:33.217 INFO:tasks.workunit.client.0.smithi067.stdout:0/687: dread - db/d2e/f40 zero size 2024-04-22T23:23:33.222 INFO:tasks.workunit.client.0.smithi067.stdout:8/641: creat d1/dd/d80/d6f/db6/fcd x:0 0 0 2024-04-22T23:23:33.222 INFO:tasks.workunit.client.0.smithi067.stdout:5/728: write d5/d28/dae/fd3 [171184,9411] 0 2024-04-22T23:23:33.222 INFO:tasks.workunit.client.0.smithi067.stdout:1/634: unlink d8/d9/d2d/d4d/d48/f75 0 2024-04-22T23:23:33.227 INFO:tasks.workunit.client.0.smithi067.stdout:9/659: rmdir d5/d21/d47 39 2024-04-22T23:23:33.227 INFO:tasks.workunit.client.0.smithi067.stdout:8/642: dread d1/dd/d80/d6f/f30 [4194304,4194304] 0 2024-04-22T23:23:33.228 INFO:tasks.workunit.client.0.smithi067.stdout:1/635: chown d8/d12/d36/c52 609059 1 2024-04-22T23:23:33.230 INFO:tasks.workunit.client.0.smithi067.stdout:8/643: creat d1/d41/d46/fce x:0 0 0 2024-04-22T23:23:33.230 INFO:tasks.workunit.client.0.smithi067.stdout:9/660: chown d5/dd/d18/d39/d49/d54/f79 220 1 2024-04-22T23:23:33.232 INFO:tasks.workunit.client.0.smithi067.stdout:8/644: fdatasync d1/f76 0 2024-04-22T23:23:33.233 INFO:tasks.workunit.client.0.smithi067.stdout:0/688: dwrite db/d9a/d56/d6b/fa2 [0,4194304] 0 2024-04-22T23:23:33.235 INFO:tasks.workunit.client.0.smithi067.stdout:5/729: rename d5/d3e/d87 to d5/d28/daf/dbf/dfc 0 2024-04-22T23:23:33.237 INFO:tasks.workunit.client.0.smithi067.stdout:3/725: creat d3/d4/d10/d38/d53/d55/f103 x:0 0 0 2024-04-22T23:23:33.240 INFO:tasks.workunit.client.0.smithi067.stdout:5/730: chown d5/d28/daf/dbf/dcf/fbc 394 1 2024-04-22T23:23:33.240 INFO:tasks.workunit.client.0.smithi067.stdout:9/661: mkdir d5/dd/de6 0 2024-04-22T23:23:33.242 INFO:tasks.workunit.client.0.smithi067.stdout:2/685: dwrite d1/d10/d72/d53/f74 [0,4194304] 0 2024-04-22T23:23:33.246 INFO:tasks.workunit.client.0.smithi067.stdout:5/731: creat d5/d3a/d4d/ffd x:0 0 0 2024-04-22T23:23:33.247 INFO:tasks.workunit.client.0.smithi067.stdout:9/662: write d5/dd/d18/f1d [2398375,4994] 0 2024-04-22T23:23:33.250 INFO:tasks.workunit.client.0.smithi067.stdout:5/732: readlink d5/db/d12/d8a/lb0 0 2024-04-22T23:23:33.251 INFO:tasks.workunit.client.0.smithi067.stdout:8/645: dwrite d1/d41/f83 [0,4194304] 0 2024-04-22T23:23:33.255 INFO:tasks.workunit.client.0.smithi067.stdout:0/689: rmdir db/d9a/d56/dd5 0 2024-04-22T23:23:33.256 INFO:tasks.workunit.client.0.smithi067.stdout:1/636: getdents d8/d9/d2d/d4d/d70 0 2024-04-22T23:23:33.256 INFO:tasks.workunit.client.0.smithi067.stdout:3/726: dwrite d3/dcb/fde [0,4194304] 0 2024-04-22T23:23:33.269 INFO:tasks.workunit.client.0.smithi067.stdout:7/600: truncate d10/d14/d1d/d26/f2f 2669719 0 2024-04-22T23:23:33.270 INFO:tasks.workunit.client.0.smithi067.stdout:9/663: rename d5/dd/c16 to d5/dd/d18/d3c/d78/ce7 0 2024-04-22T23:23:33.272 INFO:tasks.workunit.client.0.smithi067.stdout:0/690: dwrite db/f6a [8388608,4194304] 0 2024-04-22T23:23:33.273 INFO:tasks.workunit.client.0.smithi067.stdout:1/637: mkdir d8/d9/d2f/d59/dd8 0 2024-04-22T23:23:33.273 INFO:tasks.workunit.client.0.smithi067.stdout:8/646: mknod d1/d3/ccf 0 2024-04-22T23:23:33.274 INFO:tasks.workunit.client.0.smithi067.stdout:7/601: chown d10/d14/d1d/d26/d2c/l4b 14 1 2024-04-22T23:23:33.277 INFO:tasks.workunit.client.0.smithi067.stdout:0/691: write db/de/d2a/f5e [4110490,101423] 0 2024-04-22T23:23:33.283 INFO:tasks.workunit.client.0.smithi067.stdout:0/692: read db/f79 [1361455,47319] 0 2024-04-22T23:23:33.289 INFO:tasks.workunit.client.0.smithi067.stdout:2/686: dread d1/d10/d33/f5b [0,4194304] 0 2024-04-22T23:23:33.289 INFO:tasks.workunit.client.0.smithi067.stdout:2/687: chown d1/d10/d33/f3e 55690 1 2024-04-22T23:23:33.301 INFO:tasks.workunit.client.0.smithi067.stdout:8/647: dread d1/d41/d46/f61 [0,4194304] 0 2024-04-22T23:23:33.303 INFO:tasks.workunit.client.0.smithi067.stdout:7/602: rename fb to d10/d47/fdc 0 2024-04-22T23:23:33.307 INFO:tasks.workunit.client.0.smithi067.stdout:9/664: dwrite d5/dd/d18/d39/d3e/fae [4194304,4194304] 0 2024-04-22T23:23:33.308 INFO:tasks.workunit.client.0.smithi067.stdout:2/688: write d1/d10/d72/fd7 [1444837,58067] 0 2024-04-22T23:23:33.309 INFO:tasks.workunit.client.0.smithi067.stdout:3/727: creat d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d90/da2/daf/dd8/f104 x:0 0 0 2024-04-22T23:23:33.312 INFO:tasks.workunit.client.0.smithi067.stdout:8/648: dwrite d1/d3/f2a [4194304,4194304] 0 2024-04-22T23:23:33.314 INFO:tasks.workunit.client.0.smithi067.stdout:8/649: chown d1/d3/d27 216079 1 2024-04-22T23:23:33.314 INFO:tasks.workunit.client.0.smithi067.stdout:3/728: stat d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d8c/ff9 0 2024-04-22T23:23:33.318 INFO:tasks.workunit.client.0.smithi067.stdout:1/638: rename d8/da7 to d8/d9/d2f/d33/d5c/dd9 0 2024-04-22T23:23:33.319 INFO:tasks.workunit.client.0.smithi067.stdout:3/729: chown d3/d4/d10/d38/d53/d55/d74/fc9 60066621 1 2024-04-22T23:23:33.319 INFO:tasks.workunit.client.0.smithi067.stdout:5/733: getdents d5/d28/d36/d40/d97 0 2024-04-22T23:23:33.324 INFO:tasks.workunit.client.0.smithi067.stdout:8/650: dwrite d1/dd/d80/fc7 [4194304,4194304] 0 2024-04-22T23:23:33.331 INFO:tasks.workunit.client.0.smithi067.stdout:1/639: mkdir d8/d9/d2f/d33/d5c/dda 0 2024-04-22T23:23:33.332 INFO:tasks.workunit.client.0.smithi067.stdout:2/689: creat d1/d10/d41/fe1 x:0 0 0 2024-04-22T23:23:33.334 INFO:tasks.workunit.client.0.smithi067.stdout:9/665: dwrite d5/d21/d47/d56/d8c/dd5/fc1 [0,4194304] 0 2024-04-22T23:23:33.337 INFO:tasks.workunit.client.0.smithi067.stdout:8/651: unlink d1/d3/f1e 0 2024-04-22T23:23:33.340 INFO:tasks.workunit.client.0.smithi067.stdout:3/730: mknod d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d90/da2/dd3/c105 0 2024-04-22T23:23:33.342 INFO:tasks.workunit.client.0.smithi067.stdout:1/640: symlink d8/d9/d2f/d33/d66/d84/ldb 0 2024-04-22T23:23:33.344 INFO:tasks.workunit.client.0.smithi067.stdout:9/666: dwrite d5/dd/d18/d39/f51 [0,4194304] 0 2024-04-22T23:23:33.347 INFO:tasks.workunit.client.0.smithi067.stdout:3/731: creat d3/d4/d10/d8a/f106 x:0 0 0 2024-04-22T23:23:33.348 INFO:tasks.workunit.client.0.smithi067.stdout:9/667: chown d5/dd/d18/f1f 3765739 1 2024-04-22T23:23:33.349 INFO:tasks.workunit.client.0.smithi067.stdout:9/668: stat d5/dd/d18/d3c 0 2024-04-22T23:23:33.350 INFO:tasks.workunit.client.0.smithi067.stdout:5/734: getdents d5/d3e 0 2024-04-22T23:23:33.350 INFO:tasks.workunit.client.0.smithi067.stdout:8/652: dwrite d1/d3/d5c/d98/fc9 [0,4194304] 0 2024-04-22T23:23:33.352 INFO:tasks.workunit.client.0.smithi067.stdout:9/669: chown d5/dd/d18/d3c/d40/f50 1462676 1 2024-04-22T23:23:33.355 INFO:tasks.workunit.client.0.smithi067.stdout:8/653: chown d1/d3/d5c/f99 5476735 1 2024-04-22T23:23:33.355 INFO:tasks.workunit.client.0.smithi067.stdout:2/690: getdents d1/dbf 0 2024-04-22T23:23:33.356 INFO:tasks.workunit.client.0.smithi067.stdout:8/654: dread - d1/d3/d10/d3b/fac zero size 2024-04-22T23:23:33.356 INFO:tasks.workunit.client.0.smithi067.stdout:5/735: fsync d5/d28/daf/dbf/dcf/fbc 0 2024-04-22T23:23:33.360 INFO:tasks.workunit.client.0.smithi067.stdout:3/732: read d3/d4/d10/fbc [167436,94635] 0 2024-04-22T23:23:33.360 INFO:tasks.workunit.client.0.smithi067.stdout:8/655: fsync d1/d41/d46/f66 0 2024-04-22T23:23:33.367 INFO:tasks.workunit.client.0.smithi067.stdout:8/656: mkdir d1/dd/d80/db9/dd0 0 2024-04-22T23:23:33.368 INFO:tasks.workunit.client.0.smithi067.stdout:5/736: mkdir d5/d3a/dfe 0 2024-04-22T23:23:33.372 INFO:tasks.workunit.client.0.smithi067.stdout:1/641: truncate d8/d9/f43 1507153 0 2024-04-22T23:23:33.374 INFO:tasks.workunit.client.0.smithi067.stdout:8/657: stat d1/f7a 0 2024-04-22T23:23:33.375 INFO:tasks.workunit.client.0.smithi067.stdout:8/658: stat d1/f18 0 2024-04-22T23:23:33.376 INFO:tasks.workunit.client.0.smithi067.stdout:5/737: dwrite d5/d3e/d90/f94 [0,4194304] 0 2024-04-22T23:23:33.378 INFO:tasks.workunit.client.0.smithi067.stdout:5/738: write d5/f11 [5172283,104644] 0 2024-04-22T23:23:33.379 INFO:tasks.workunit.client.0.smithi067.stdout:0/693: truncate db/d21/d29/d2d/f42 3365293 0 2024-04-22T23:23:33.387 INFO:tasks.workunit.client.0.smithi067.stdout:0/694: mkdir db/d9a/d56/d70/dbf/de6 0 2024-04-22T23:23:33.389 INFO:tasks.workunit.client.0.smithi067.stdout:5/739: dwrite d5/db/d12/d44/ff8 [0,4194304] 0 2024-04-22T23:23:33.393 INFO:tasks.workunit.client.0.smithi067.stdout:1/642: link d8/d9/fd d8/d9/fdc 0 2024-04-22T23:23:33.394 INFO:tasks.workunit.client.0.smithi067.stdout:1/643: fsync d8/d9/d2d/d4d/f1f 0 2024-04-22T23:23:33.395 INFO:tasks.workunit.client.0.smithi067.stdout:0/695: creat db/d60/d9e/fe7 x:0 0 0 2024-04-22T23:23:33.398 INFO:tasks.workunit.client.0.smithi067.stdout:8/659: dread d1/d3/d10/fbe [0,4194304] 0 2024-04-22T23:23:33.399 INFO:tasks.workunit.client.0.smithi067.stdout:5/740: symlink d5/db/d12/lff 0 2024-04-22T23:23:33.401 INFO:tasks.workunit.client.0.smithi067.stdout:9/670: truncate d5/d21/d47/d57/f97 3956900 0 2024-04-22T23:23:33.405 INFO:tasks.workunit.client.0.smithi067.stdout:1/644: dwrite d8/d12/d36/fba [0,4194304] 0 2024-04-22T23:23:33.407 INFO:tasks.workunit.client.0.smithi067.stdout:5/741: creat d5/db/d12/d44/d47/f100 x:0 0 0 2024-04-22T23:23:33.407 INFO:tasks.workunit.client.0.smithi067.stdout:5/742: readlink d5/db/d12/d44/d47/l7c 0 2024-04-22T23:23:33.409 INFO:tasks.workunit.client.0.smithi067.stdout:0/696: link db/d9a/d56/d6b/lce db/d21/d29/d75/d9d/le8 0 2024-04-22T23:23:33.412 INFO:tasks.workunit.client.0.smithi067.stdout:5/743: fsync d5/db/d12/d44/ff8 0 2024-04-22T23:23:33.416 INFO:tasks.workunit.client.0.smithi067.stdout:9/671: rename d5/d21/d47/d7f/la0 to d5/dd/d18/da7/le8 0 2024-04-22T23:23:33.420 INFO:tasks.workunit.client.0.smithi067.stdout:1/645: unlink d8/ca 0 2024-04-22T23:23:33.420 INFO:tasks.workunit.client.0.smithi067.stdout:3/733: dwrite d3/d4/d10/d1f/f3e [0,4194304] 0 2024-04-22T23:23:33.421 INFO:tasks.workunit.client.0.smithi067.stdout:9/672: dread - d5/dd/d18/d39/d49/f65 zero size 2024-04-22T23:23:33.430 INFO:tasks.workunit.client.0.smithi067.stdout:8/660: dread d1/d3/f38 [0,4194304] 0 2024-04-22T23:23:33.430 INFO:tasks.workunit.client.0.smithi067.stdout:0/697: truncate db/d21/d29/d75/d9d/fd2 148122 0 2024-04-22T23:23:33.438 INFO:tasks.workunit.client.0.smithi067.stdout:7/603: write d10/d18/d35/f5e [343210,42607] 0 2024-04-22T23:23:33.442 INFO:tasks.workunit.client.0.smithi067.stdout:0/698: dwrite db/d60/d9e/fc4 [4194304,4194304] 0 2024-04-22T23:23:33.444 INFO:tasks.workunit.client.0.smithi067.stdout:0/699: dread - db/d21/d29/f62 zero size 2024-04-22T23:23:33.445 INFO:tasks.workunit.client.0.smithi067.stdout:7/604: mkdir d10/d14/d1d/d26/d2c/d41/db0/ddd 0 2024-04-22T23:23:33.446 INFO:tasks.workunit.client.0.smithi067.stdout:3/734: write d3/d4/d10/fbc [1946304,115027] 0 2024-04-22T23:23:33.448 INFO:tasks.workunit.client.0.smithi067.stdout:7/605: write d10/d14/d1d/d46/d50/f29 [3056062,47829] 0 2024-04-22T23:23:33.448 INFO:tasks.workunit.client.0.smithi067.stdout:5/744: dread d5/db/d35/f6c [0,4194304] 0 2024-04-22T23:23:33.450 INFO:tasks.workunit.client.0.smithi067.stdout:7/606: readlink d10/d47/d49/l51 0 2024-04-22T23:23:33.450 INFO:tasks.workunit.client.0.smithi067.stdout:0/700: creat db/d9a/d56/d6b/db9/fe9 x:0 0 0 2024-04-22T23:23:33.451 INFO:tasks.workunit.client.0.smithi067.stdout:8/661: write d1/dd/d80/f86 [886397,65351] 0 2024-04-22T23:23:33.453 INFO:tasks.workunit.client.0.smithi067.stdout:1/646: dread d8/d9/fdc [0,4194304] 0 2024-04-22T23:23:33.453 INFO:tasks.workunit.client.0.smithi067.stdout:3/735: chown d3/d4/c76 27781 1 2024-04-22T23:23:33.454 INFO:tasks.workunit.client.0.smithi067.stdout:9/673: dread d5/d21/f24 [0,4194304] 0 2024-04-22T23:23:33.455 INFO:tasks.workunit.client.0.smithi067.stdout:5/745: mkdir d5/d3a/d4d/d101 0 2024-04-22T23:23:33.457 INFO:tasks.workunit.client.0.smithi067.stdout:9/674: chown d5/dd/d18/d39/d4d/fd0 2494961 1 2024-04-22T23:23:33.457 INFO:tasks.workunit.client.0.smithi067.stdout:1/647: readlink d8/d9/l16 0 2024-04-22T23:23:33.459 INFO:tasks.workunit.client.0.smithi067.stdout:0/701: creat db/de/d11/d92/fea x:0 0 0 2024-04-22T23:23:33.460 INFO:tasks.workunit.client.0.smithi067.stdout:8/662: mknod d1/dd/d80/db9/cd1 0 2024-04-22T23:23:33.460 INFO:tasks.workunit.client.0.smithi067.stdout:3/736: write d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d90/f9f [3804827,96866] 0 2024-04-22T23:23:33.463 INFO:tasks.workunit.client.0.smithi067.stdout:7/607: mkdir d10/d14/d1d/d26/d2c/d7a/dc2/dd0/dde 0 2024-04-22T23:23:33.464 INFO:tasks.workunit.client.0.smithi067.stdout:9/675: truncate d5/d21/d47/d56/dc6/fd7 291019 0 2024-04-22T23:23:33.466 INFO:tasks.workunit.client.0.smithi067.stdout:3/737: read d3/f6f [2443712,15831] 0 2024-04-22T23:23:33.466 INFO:tasks.workunit.client.0.smithi067.stdout:1/648: dread d8/d9/d2d/f4f [0,4194304] 0 2024-04-22T23:23:33.468 INFO:tasks.workunit.client.0.smithi067.stdout:9/676: stat d5/d21/d2c/l42 0 2024-04-22T23:23:33.468 INFO:tasks.workunit.client.0.smithi067.stdout:8/663: mknod d1/dd/d80/d6f/da8/cd2 0 2024-04-22T23:23:33.472 INFO:tasks.workunit.client.0.smithi067.stdout:5/746: dwrite d5/db/d12/ff6 [0,4194304] 0 2024-04-22T23:23:33.473 INFO:tasks.workunit.client.0.smithi067.stdout:8/664: write d1/dd/d80/fc7 [1419754,15616] 0 2024-04-22T23:23:33.473 INFO:tasks.workunit.client.0.smithi067.stdout:9/677: fdatasync d5/dd/d18/d39/d3e/d3f/d63/fe3 0 2024-04-22T23:23:33.474 INFO:tasks.workunit.client.0.smithi067.stdout:3/738: fsync d3/d4/d10/d38/d53/d55/f66 0 2024-04-22T23:23:33.487 INFO:tasks.workunit.client.0.smithi067.stdout:7/608: dread d10/d14/d1d/d26/d2c/d7a/f87 [0,4194304] 0 2024-04-22T23:23:33.514 INFO:tasks.workunit.client.0.smithi067.stdout:6/671: sync 2024-04-22T23:23:33.514 INFO:tasks.workunit.client.0.smithi067.stdout:4/775: sync 2024-04-22T23:23:33.514 INFO:tasks.workunit.client.0.smithi067.stdout:0/702: dread db/de/d11/f3c [0,4194304] 0 2024-04-22T23:23:33.514 INFO:tasks.workunit.client.0.smithi067.stdout:9/678: mknod d5/d21/d47/d57/dc0/ce9 0 2024-04-22T23:23:33.516 INFO:tasks.workunit.client.0.smithi067.stdout:7/609: symlink d10/d47/dab/ldf 0 2024-04-22T23:23:33.517 INFO:tasks.workunit.client.0.smithi067.stdout:3/739: creat d3/d4/d10/d1f/d3c/df4/f107 x:0 0 0 2024-04-22T23:23:33.517 INFO:tasks.workunit.client.0.smithi067.stdout:0/703: read db/de/d2a/f99 [59954,87668] 0 2024-04-22T23:23:33.518 INFO:tasks.workunit.client.0.smithi067.stdout:8/665: truncate d1/d3/f38 3460871 0 2024-04-22T23:23:33.521 INFO:tasks.workunit.client.0.smithi067.stdout:6/672: unlink d17/d39/d53/d90/d46/l58 0 2024-04-22T23:23:33.523 INFO:tasks.workunit.client.0.smithi067.stdout:4/776: mknod d12/d42/c105 0 2024-04-22T23:23:33.524 INFO:tasks.workunit.client.0.smithi067.stdout:0/704: mknod db/de/d11/d1a/ceb 0 2024-04-22T23:23:33.525 INFO:tasks.workunit.client.0.smithi067.stdout:0/705: chown db/d9a/d56/d70/dbf/de6 7884 1 2024-04-22T23:23:33.526 INFO:tasks.workunit.client.0.smithi067.stdout:8/666: mknod d1/d3/d5c/d98/cd3 0 2024-04-22T23:23:33.527 INFO:tasks.workunit.client.0.smithi067.stdout:9/679: mknod d5/d21/d47/d56/de5/cea 0 2024-04-22T23:23:33.529 INFO:tasks.workunit.client.0.smithi067.stdout:7/610: mkdir d10/d14/de0 0 2024-04-22T23:23:33.529 INFO:tasks.workunit.client.0.smithi067.stdout:7/611: fdatasync d10/d47/d49/f8b 0 2024-04-22T23:23:33.530 INFO:tasks.workunit.client.0.smithi067.stdout:4/777: readlink d12/d42/d4b/lb2 0 2024-04-22T23:23:33.532 INFO:tasks.workunit.client.0.smithi067.stdout:8/667: unlink d1/dd/d26/d62/d75/l8a 0 2024-04-22T23:23:33.533 INFO:tasks.workunit.client.0.smithi067.stdout:9/680: symlink d5/dd/d18/d3c/d78/leb 0 2024-04-22T23:23:33.534 INFO:tasks.workunit.client.0.smithi067.stdout:6/673: rename d17/d39/d53/d90/d46/fa6 to d17/d39/d53/d90/d46/d62/d88/fdb 0 2024-04-22T23:23:33.536 INFO:tasks.workunit.client.0.smithi067.stdout:4/778: write d12/d5f/d7e/fc9 [426383,96949] 0 2024-04-22T23:23:33.536 INFO:tasks.workunit.client.0.smithi067.stdout:0/706: truncate db/d9a/d56/d70/dba/fc0 827233 0 2024-04-22T23:23:33.538 INFO:tasks.workunit.client.0.smithi067.stdout:8/668: symlink d1/d3/d5c/ld4 0 2024-04-22T23:23:33.539 INFO:tasks.workunit.client.0.smithi067.stdout:7/612: symlink d10/d14/d1d/d46/d50/dcd/le1 0 2024-04-22T23:23:33.539 INFO:tasks.workunit.client.0.smithi067.stdout:7/613: fdatasync fe 0 2024-04-22T23:23:33.544 INFO:tasks.workunit.client.0.smithi067.stdout:6/674: creat d17/d39/d3a/d8d/dc9/fdc x:0 0 0 2024-04-22T23:23:33.546 INFO:tasks.workunit.client.0.smithi067.stdout:7/614: rmdir d10/d14/d1d/d69 39 2024-04-22T23:23:33.550 INFO:tasks.workunit.client.0.smithi067.stdout:6/675: mkdir d17/d83/d9b/ddd 0 2024-04-22T23:23:33.551 INFO:tasks.workunit.client.0.smithi067.stdout:7/615: mkdir d10/d14/d1d/d46/d50/de2 0 2024-04-22T23:23:33.555 INFO:tasks.workunit.client.0.smithi067.stdout:7/616: truncate d10/d18/f1a 469112 0 2024-04-22T23:23:33.556 INFO:tasks.workunit.client.0.smithi067.stdout:7/617: write d10/d18/d35/d62/f9a [822033,17765] 0 2024-04-22T23:23:33.556 INFO:tasks.workunit.client.0.smithi067.stdout:6/676: truncate d17/d39/d53/d90/d46/d62/f85 346984 0 2024-04-22T23:23:33.559 INFO:tasks.workunit.client.0.smithi067.stdout:4/779: dread d12/d17/f51 [0,4194304] 0 2024-04-22T23:23:33.560 INFO:tasks.workunit.client.0.smithi067.stdout:7/618: mkdir d10/de3 0 2024-04-22T23:23:33.561 INFO:tasks.workunit.client.0.smithi067.stdout:6/677: truncate d17/d39/d3a/d8d/f97 80851 0 2024-04-22T23:23:33.564 INFO:tasks.workunit.client.0.smithi067.stdout:7/619: mkdir d10/d14/d1d/d26/d2c/d7a/dc2/dd0/de4 0 2024-04-22T23:23:33.565 INFO:tasks.workunit.client.0.smithi067.stdout:6/678: creat d17/d83/fde x:0 0 0 2024-04-22T23:23:33.567 INFO:tasks.workunit.client.0.smithi067.stdout:4/780: creat d12/d42/dd3/f106 x:0 0 0 2024-04-22T23:23:33.570 INFO:tasks.workunit.client.0.smithi067.stdout:0/707: dwrite db/de/f1b [0,4194304] 0 2024-04-22T23:23:33.571 INFO:tasks.workunit.client.0.smithi067.stdout:8/669: dwrite d1/d41/f74 [0,4194304] 0 2024-04-22T23:23:33.571 INFO:tasks.workunit.client.0.smithi067.stdout:4/781: write d12/d2e/fb7 [98711,111736] 0 2024-04-22T23:23:33.572 INFO:tasks.workunit.client.0.smithi067.stdout:0/708: dread - db/d60/d9e/fe7 zero size 2024-04-22T23:23:33.576 INFO:tasks.workunit.client.0.smithi067.stdout:6/679: creat d17/d39/fdf x:0 0 0 2024-04-22T23:23:33.576 INFO:tasks.workunit.client.0.smithi067.stdout:8/670: read d1/d3/d10/f45 [3931825,77026] 0 2024-04-22T23:23:33.577 INFO:tasks.workunit.client.0.smithi067.stdout:6/680: read - d17/d39/fdf zero size 2024-04-22T23:23:33.585 INFO:tasks.workunit.client.0.smithi067.stdout:8/671: creat d1/dd/d26/d62/d94/fd5 x:0 0 0 2024-04-22T23:23:33.587 INFO:tasks.workunit.client.0.smithi067.stdout:4/782: creat d12/d5f/d2b/ded/d8e/d56/d8b/f107 x:0 0 0 2024-04-22T23:23:33.589 INFO:tasks.workunit.client.0.smithi067.stdout:7/620: link d10/d14/d1d/d69/fce d10/d14/d1d/d26/d2c/d7a/dc2/dd0/de4/fe5 0 2024-04-22T23:23:33.590 INFO:tasks.workunit.client.0.smithi067.stdout:0/709: truncate f0 2131743 0 2024-04-22T23:23:33.592 INFO:tasks.workunit.client.0.smithi067.stdout:4/783: mknod d12/d5f/dae/c108 0 2024-04-22T23:23:33.593 INFO:tasks.workunit.client.0.smithi067.stdout:8/672: dwrite d1/dd/d26/f6b [0,4194304] 0 2024-04-22T23:23:33.595 INFO:tasks.workunit.client.0.smithi067.stdout:7/621: mknod d10/ce6 0 2024-04-22T23:23:33.597 INFO:tasks.workunit.client.0.smithi067.stdout:4/784: creat d12/d42/d4b/d77/d9d/dc1/f109 x:0 0 0 2024-04-22T23:23:33.597 INFO:tasks.workunit.client.0.smithi067.stdout:0/710: dread db/d2e/dd9/fe1 [0,4194304] 0 2024-04-22T23:23:33.598 INFO:tasks.workunit.client.0.smithi067.stdout:7/622: mkdir d10/d18/d35/d62/de7 0 2024-04-22T23:23:33.600 INFO:tasks.workunit.client.0.smithi067.stdout:0/711: stat db/d9a/d56/d6b/db9/le4 0 2024-04-22T23:23:33.603 INFO:tasks.workunit.client.0.smithi067.stdout:4/785: creat d12/d5f/d2b/d97/f10a x:0 0 0 2024-04-22T23:23:33.607 INFO:tasks.workunit.client.0.smithi067.stdout:7/623: truncate d10/d14/d1d/d26/f3a 1997045 0 2024-04-22T23:23:33.609 INFO:tasks.workunit.client.0.smithi067.stdout:0/712: dwrite db/de/d11/d92/fea [0,4194304] 0 2024-04-22T23:23:33.610 INFO:tasks.workunit.client.0.smithi067.stdout:4/786: chown d12/d42/ddb 2907865 1 2024-04-22T23:23:33.612 INFO:tasks.workunit.client.0.smithi067.stdout:0/713: chown db/c16 0 1 2024-04-22T23:23:33.615 INFO:tasks.workunit.client.0.smithi067.stdout:6/681: dwrite d17/d39/d53/f93 [0,4194304] 0 2024-04-22T23:23:33.616 INFO:tasks.workunit.client.0.smithi067.stdout:7/624: creat d10/d18/d35/d62/fe8 x:0 0 0 2024-04-22T23:23:33.618 INFO:tasks.workunit.client.0.smithi067.stdout:4/787: creat d12/d5f/d2b/d97/f10b x:0 0 0 2024-04-22T23:23:33.619 INFO:tasks.workunit.client.0.smithi067.stdout:0/714: mkdir db/de/d11/d1a/da4/dec 0 2024-04-22T23:23:33.627 INFO:tasks.workunit.client.0.smithi067.stdout:6/682: dwrite d17/d39/d53/d90/d46/f5f [4194304,4194304] 0 2024-04-22T23:23:33.631 INFO:tasks.workunit.client.0.smithi067.stdout:7/625: creat d10/d14/d1d/d46/d6a/dcc/fe9 x:0 0 0 2024-04-22T23:23:33.632 INFO:tasks.workunit.client.0.smithi067.stdout:0/715: dwrite db/de/f1b [4194304,4194304] 0 2024-04-22T23:23:33.638 INFO:tasks.workunit.client.0.smithi067.stdout:4/788: dread d12/d5f/d2b/d97/fca [0,4194304] 0 2024-04-22T23:23:33.638 INFO:tasks.workunit.client.0.smithi067.stdout:0/716: unlink db/d21/d29/d2c/l65 0 2024-04-22T23:23:33.640 INFO:tasks.workunit.client.0.smithi067.stdout:7/626: creat d10/d14/d1d/d26/d2c/d41/db0/fea x:0 0 0 2024-04-22T23:23:33.641 INFO:tasks.workunit.client.0.smithi067.stdout:4/789: write d12/d42/dd3/dd8/fe2 [962421,36175] 0 2024-04-22T23:23:33.643 INFO:tasks.workunit.client.0.smithi067.stdout:0/717: truncate db/d9a/d56/d6b/db9/fe9 565109 0 2024-04-22T23:23:33.644 INFO:tasks.workunit.client.0.smithi067.stdout:0/718: readlink db/de/d11/ldb 0 2024-04-22T23:23:33.644 INFO:tasks.workunit.client.0.smithi067.stdout:6/683: creat d17/d39/d3a/fe0 x:0 0 0 2024-04-22T23:23:33.646 INFO:tasks.workunit.client.0.smithi067.stdout:7/627: mkdir d10/d14/de0/deb 0 2024-04-22T23:23:33.647 INFO:tasks.workunit.client.0.smithi067.stdout:4/790: symlink d12/d17/d31/l10c 0 2024-04-22T23:23:33.649 INFO:tasks.workunit.client.0.smithi067.stdout:0/719: rename db/de/ff to db/d21/d29/d75/fed 0 2024-04-22T23:23:33.650 INFO:tasks.workunit.client.0.smithi067.stdout:6/684: creat d17/d39/d3a/d8d/dc9/fe1 x:0 0 0 2024-04-22T23:23:33.652 INFO:tasks.workunit.client.0.smithi067.stdout:0/720: creat db/de/d11/d92/fee x:0 0 0 2024-04-22T23:23:33.653 INFO:tasks.workunit.client.0.smithi067.stdout:6/685: creat d17/d39/d3a/d8d/dc9/fe2 x:0 0 0 2024-04-22T23:23:33.653 INFO:tasks.workunit.client.0.smithi067.stdout:0/721: truncate db/d2e/f5d 2187698 0 2024-04-22T23:23:33.656 INFO:tasks.workunit.client.0.smithi067.stdout:6/686: link d17/f7d d17/d83/db5/dbc/fe3 0 2024-04-22T23:23:33.657 INFO:tasks.workunit.client.0.smithi067.stdout:0/722: mknod db/d21/d29/d75/d9d/dde/cef 0 2024-04-22T23:23:33.659 INFO:tasks.workunit.client.0.smithi067.stdout:0/723: creat db/d9a/ff0 x:0 0 0 2024-04-22T23:23:33.661 INFO:tasks.workunit.client.0.smithi067.stdout:6/687: creat d17/d83/db5/fe4 x:0 0 0 2024-04-22T23:23:33.661 INFO:tasks.workunit.client.0.smithi067.stdout:2/691: sync 2024-04-22T23:23:33.662 INFO:tasks.workunit.client.0.smithi067.stdout:2/692: dread - d1/d42/d61/fb8 zero size 2024-04-22T23:23:33.664 INFO:tasks.workunit.client.0.smithi067.stdout:0/724: rename la to db/lf1 0 2024-04-22T23:23:33.671 INFO:tasks.workunit.client.0.smithi067.stdout:7/628: write d10/d47/fdc [870438,74172] 0 2024-04-22T23:23:33.672 INFO:tasks.workunit.client.0.smithi067.stdout:4/791: write d12/d5f/d2b/db6/f9e [492324,11241] 0 2024-04-22T23:23:33.674 INFO:tasks.workunit.client.0.smithi067.stdout:7/629: mknod d10/d18/d88/daf/cec 0 2024-04-22T23:23:33.676 INFO:tasks.workunit.client.0.smithi067.stdout:7/630: write d10/d14/d1d/d26/d2c/d7a/fb3 [83413,86212] 0 2024-04-22T23:23:33.677 INFO:tasks.workunit.client.0.smithi067.stdout:6/688: rmdir d17/d83/db5 39 2024-04-22T23:23:33.678 INFO:tasks.workunit.client.0.smithi067.stdout:2/693: write d1/d10/d1c/d6d/f4c [372378,60735] 0 2024-04-22T23:23:33.679 INFO:tasks.workunit.client.0.smithi067.stdout:0/725: write db/d21/d29/d2d/f42 [1786072,118831] 0 2024-04-22T23:23:33.680 INFO:tasks.workunit.client.0.smithi067.stdout:2/694: write d1/d42/d5f/fbd [740031,60533] 0 2024-04-22T23:23:33.682 INFO:tasks.workunit.client.0.smithi067.stdout:6/689: mkdir d17/d39/d53/d90/d46/db0/de5 0 2024-04-22T23:23:33.683 INFO:tasks.workunit.client.0.smithi067.stdout:6/690: readlink d17/d39/l67 0 2024-04-22T23:23:33.686 INFO:tasks.workunit.client.0.smithi067.stdout:0/726: link db/f6a db/d71/dda/ff2 0 2024-04-22T23:23:33.691 INFO:tasks.workunit.client.0.smithi067.stdout:7/631: getdents d10/d14/d1d/d26/d2c/d41/db0 0 2024-04-22T23:23:33.691 INFO:tasks.workunit.client.0.smithi067.stdout:2/695: truncate d1/d10/d72/d53/fae 322654 0 2024-04-22T23:23:33.691 INFO:tasks.workunit.client.0.smithi067.stdout:6/691: unlink d17/d83/ld1 0 2024-04-22T23:23:33.691 INFO:tasks.workunit.client.0.smithi067.stdout:2/696: stat d1/d12/c2b 0 2024-04-22T23:23:33.692 INFO:tasks.workunit.client.0.smithi067.stdout:4/792: dwrite d12/d5f/d69/fa9 [0,4194304] 0 2024-04-22T23:23:33.692 INFO:tasks.workunit.client.0.smithi067.stdout:7/632: read - d10/d14/d6d/fca zero size 2024-04-22T23:23:33.694 INFO:tasks.workunit.client.0.smithi067.stdout:6/692: truncate d17/d39/d3a/d8d/dc9/fe2 895321 0 2024-04-22T23:23:33.694 INFO:tasks.workunit.client.0.smithi067.stdout:7/633: write d10/d14/d1d/d26/d2c/d41/f99 [66147,13655] 0 2024-04-22T23:23:33.695 INFO:tasks.workunit.client.0.smithi067.stdout:6/693: chown d17/d83/fcb 43279027 1 2024-04-22T23:23:33.701 INFO:tasks.workunit.client.0.smithi067.stdout:0/727: link db/de/d11/d1a/da4/c66 db/d21/d29/d75/cf3 0 2024-04-22T23:23:33.702 INFO:tasks.workunit.client.0.smithi067.stdout:2/697: dwrite d1/d10/d1c/d6d/d87/dc8/dc5/fdd [0,4194304] 0 2024-04-22T23:23:33.702 INFO:tasks.workunit.client.0.smithi067.stdout:0/728: chown db/d2e/f7c 78 1 2024-04-22T23:23:33.704 INFO:tasks.workunit.client.0.smithi067.stdout:7/634: unlink d10/d14/d1d/d26/d2c/d7a/l8d 0 2024-04-22T23:23:33.704 INFO:tasks.workunit.client.0.smithi067.stdout:6/694: creat d17/d83/d9b/ddd/fe6 x:0 0 0 2024-04-22T23:23:33.707 INFO:tasks.workunit.client.0.smithi067.stdout:4/793: dread d12/d5f/f55 [0,4194304] 0 2024-04-22T23:23:33.708 INFO:tasks.workunit.client.0.smithi067.stdout:4/794: readlink d12/d17/d31/l10c 0 2024-04-22T23:23:33.710 INFO:tasks.workunit.client.0.smithi067.stdout:7/635: creat d10/fed x:0 0 0 2024-04-22T23:23:33.713 INFO:tasks.workunit.client.0.smithi067.stdout:7/636: mknod d10/d47/cee 0 2024-04-22T23:23:33.714 INFO:tasks.workunit.client.0.smithi067.stdout:0/729: dread db/d9a/d56/d70/dba/fc0 [0,4194304] 0 2024-04-22T23:23:33.716 INFO:tasks.workunit.client.0.smithi067.stdout:2/698: write d1/d10/d1c/d6d/d87/dc8/fb4 [749543,124933] 0 2024-04-22T23:23:33.718 INFO:tasks.workunit.client.0.smithi067.stdout:0/730: write db/de/d11/f3c [1256793,114790] 0 2024-04-22T23:23:33.719 INFO:tasks.workunit.client.0.smithi067.stdout:4/795: dwrite d12/d5f/d7e/fd4 [0,4194304] 0 2024-04-22T23:23:33.720 INFO:tasks.workunit.client.0.smithi067.stdout:0/731: fsync db/d21/d29/f62 0 2024-04-22T23:23:33.724 INFO:tasks.workunit.client.0.smithi067.stdout:4/796: mknod d12/d17/d3f/d84/c10d 0 2024-04-22T23:23:33.725 INFO:tasks.workunit.client.0.smithi067.stdout:4/797: write f5 [3032865,33465] 0 2024-04-22T23:23:33.727 INFO:tasks.workunit.client.0.smithi067.stdout:4/798: write d12/d17/d47/f49 [3754179,65944] 0 2024-04-22T23:23:33.728 INFO:tasks.workunit.client.0.smithi067.stdout:4/799: fsync d12/fd9 0 2024-04-22T23:23:33.730 INFO:tasks.workunit.client.0.smithi067.stdout:4/800: write d12/f2a [6156078,118352] 0 2024-04-22T23:23:33.732 INFO:tasks.workunit.client.0.smithi067.stdout:6/695: dread d17/d39/d53/d90/d46/f68 [0,4194304] 0 2024-04-22T23:23:33.734 INFO:tasks.workunit.client.0.smithi067.stdout:7/637: dwrite d10/d14/f5a [0,4194304] 0 2024-04-22T23:23:33.736 INFO:tasks.workunit.client.0.smithi067.stdout:4/801: mkdir d12/d5f/d2b/ded/d8e/d56/d10e 0 2024-04-22T23:23:33.739 INFO:tasks.workunit.client.0.smithi067.stdout:4/802: dread - d12/d42/d4b/fd5 zero size 2024-04-22T23:23:33.739 INFO:tasks.workunit.client.0.smithi067.stdout:0/732: write db/de/fdc [4213477,68859] 0 2024-04-22T23:23:33.743 INFO:tasks.workunit.client.0.smithi067.stdout:6/696: dwrite d17/d39/d53/d90/d46/fa2 [0,4194304] 0 2024-04-22T23:23:33.744 INFO:tasks.workunit.client.0.smithi067.stdout:7/638: dread d10/d14/d1d/d24/f5f [0,4194304] 0 2024-04-22T23:23:33.747 INFO:tasks.workunit.client.0.smithi067.stdout:0/733: rename db/d21/l97 to db/d21/d29/d2d/lf4 0 2024-04-22T23:23:33.748 INFO:tasks.workunit.client.0.smithi067.stdout:0/734: fdatasync db/d9a/f63 0 2024-04-22T23:23:33.750 INFO:tasks.workunit.client.0.smithi067.stdout:6/697: fdatasync d17/f30 0 2024-04-22T23:23:33.751 INFO:tasks.workunit.client.0.smithi067.stdout:7/639: chown d10/d14/la3 7110 1 2024-04-22T23:23:33.753 INFO:tasks.workunit.client.0.smithi067.stdout:4/803: rename d12/d5f/d2b/ded/d8e/d56/d8b/la4 to d12/d5f/d2b/ded/d8e/d56/d10e/l10f 0 2024-04-22T23:23:33.756 INFO:tasks.workunit.client.0.smithi067.stdout:7/640: rmdir d10/d14/d1d/d69 39 2024-04-22T23:23:33.759 INFO:tasks.workunit.client.0.smithi067.stdout:6/698: rename d17/d39/d3a/d8d/dc9 to d17/d39/d53/d90/d46/db0/db9/de7 0 2024-04-22T23:23:33.764 INFO:tasks.workunit.client.0.smithi067.stdout:0/735: write db/de/d2a/f99 [372557,97694] 0 2024-04-22T23:23:33.766 INFO:tasks.workunit.client.0.smithi067.stdout:7/641: getdents d10/d14/d1d/d46/d6a/d91 0 2024-04-22T23:23:33.767 INFO:tasks.workunit.client.0.smithi067.stdout:0/736: read db/d60/d9e/fc4 [2648686,100923] 0 2024-04-22T23:23:33.768 INFO:tasks.workunit.client.0.smithi067.stdout:4/804: write d12/d5f/d2b/ded/d8e/d56/f96 [1717537,92964] 0 2024-04-22T23:23:33.770 INFO:tasks.workunit.client.0.smithi067.stdout:0/737: read db/de/d11/f7e [1449218,82100] 0 2024-04-22T23:23:33.771 INFO:tasks.workunit.client.0.smithi067.stdout:7/642: dread - d10/d14/d1d/d69/fb8 zero size 2024-04-22T23:23:33.772 INFO:tasks.workunit.client.0.smithi067.stdout:0/738: readlink db/d21/d29/d2c/l3b 0 2024-04-22T23:23:33.773 INFO:tasks.workunit.client.0.smithi067.stdout:7/643: write d10/d14/f98 [1186289,37898] 0 2024-04-22T23:23:33.775 INFO:tasks.workunit.client.0.smithi067.stdout:0/739: read db/d21/d29/d2c/f91 [1210086,97564] 0 2024-04-22T23:23:33.777 INFO:tasks.workunit.client.0.smithi067.stdout:6/699: dread d17/d39/d53/d90/d46/d62/d88/f92 [0,4194304] 0 2024-04-22T23:23:33.778 INFO:tasks.workunit.client.0.smithi067.stdout:7/644: symlink d10/d14/d1d/d26/d2c/d41/db0/ddd/lef 0 2024-04-22T23:23:33.779 INFO:tasks.workunit.client.0.smithi067.stdout:0/740: mknod db/d9a/d56/d70/d8c/dc2/cf5 0 2024-04-22T23:23:33.780 INFO:tasks.workunit.client.0.smithi067.stdout:0/741: write db/d9a/d56/d6b/fa2 [1075604,14554] 0 2024-04-22T23:23:33.784 INFO:tasks.workunit.client.0.smithi067.stdout:0/742: mknod db/d2e/d46/cf6 0 2024-04-22T23:23:33.786 INFO:tasks.workunit.client.0.smithi067.stdout:6/700: mknod d17/d83/db5/ce8 0 2024-04-22T23:23:33.788 INFO:tasks.workunit.client.0.smithi067.stdout:4/805: dread f4 [8388608,4194304] 0 2024-04-22T23:23:33.788 INFO:tasks.workunit.client.0.smithi067.stdout:7/645: write d10/d14/d1d/f76 [294059,91947] 0 2024-04-22T23:23:33.789 INFO:tasks.workunit.client.0.smithi067.stdout:6/701: read - d17/d39/d53/f8a zero size 2024-04-22T23:23:33.791 INFO:tasks.workunit.client.0.smithi067.stdout:4/806: chown d12/f2a 5 1 2024-04-22T23:23:33.791 INFO:tasks.workunit.client.0.smithi067.stdout:6/702: stat d17/c7b 0 2024-04-22T23:23:33.793 INFO:tasks.workunit.client.0.smithi067.stdout:0/743: dwrite db/d9a/d56/fca [0,4194304] 0 2024-04-22T23:23:33.794 INFO:tasks.workunit.client.0.smithi067.stdout:4/807: fsync d12/d17/d88/fc8 0 2024-04-22T23:23:33.795 INFO:tasks.workunit.client.0.smithi067.stdout:7/646: creat d10/d14/d1d/d26/d2c/d41/db0/ff0 x:0 0 0 2024-04-22T23:23:33.798 INFO:tasks.workunit.client.0.smithi067.stdout:7/647: chown d10/d18/d35/d62/cc6 0 1 2024-04-22T23:23:33.800 INFO:tasks.workunit.client.0.smithi067.stdout:7/648: fsync d10/d14/d1d/d26/d80/fc8 0 2024-04-22T23:23:33.800 INFO:tasks.workunit.client.0.smithi067.stdout:4/808: creat d12/d5f/d2b/f110 x:0 0 0 2024-04-22T23:23:33.802 INFO:tasks.workunit.client.0.smithi067.stdout:6/703: rename d17/d83/c9d to d17/d39/d53/d90/d46/ce9 0 2024-04-22T23:23:33.802 INFO:tasks.workunit.client.0.smithi067.stdout:4/809: write d12/d42/d4b/fbf [5599520,73374] 0 2024-04-22T23:23:33.803 INFO:tasks.workunit.client.0.smithi067.stdout:7/649: mkdir d10/d14/d1d/d46/d6a/df1 0 2024-04-22T23:23:33.804 INFO:tasks.workunit.client.0.smithi067.stdout:4/810: readlink d12/d2e/d9b/l102 0 2024-04-22T23:23:33.806 INFO:tasks.workunit.client.0.smithi067.stdout:0/744: rename db/de/d11/d1a/fbe to db/d21/d29/d75/d9d/ff7 0 2024-04-22T23:23:33.814 INFO:tasks.workunit.client.0.smithi067.stdout:0/745: dwrite db/d21/d29/fa7 [0,4194304] 0 2024-04-22T23:23:33.819 INFO:tasks.workunit.client.0.smithi067.stdout:0/746: mknod db/de/d11/d1a/da4/dc5/cf8 0 2024-04-22T23:23:33.822 INFO:tasks.workunit.client.0.smithi067.stdout:0/747: rmdir db/de/d2a 39 2024-04-22T23:23:33.822 INFO:tasks.workunit.client.0.smithi067.stdout:7/650: dwrite d10/d14/d1d/d26/d7e/fda [0,4194304] 0 2024-04-22T23:23:33.824 INFO:tasks.workunit.client.0.smithi067.stdout:4/811: dwrite d12/d17/d3f/f89 [0,4194304] 0 2024-04-22T23:23:33.825 INFO:tasks.workunit.client.0.smithi067.stdout:7/651: chown d10/d14/d1d/d46/d6a/c8c 15377 1 2024-04-22T23:23:33.825 INFO:tasks.workunit.client.0.smithi067.stdout:4/812: dread - d12/d5f/d2b/d97/f10a zero size 2024-04-22T23:23:33.826 INFO:tasks.workunit.client.0.smithi067.stdout:0/748: truncate db/de/f1c 190464 0 2024-04-22T23:23:33.830 INFO:tasks.workunit.client.0.smithi067.stdout:7/652: chown d10/d18/f1a 0 1 2024-04-22T23:23:33.835 INFO:tasks.workunit.client.0.smithi067.stdout:4/813: dwrite d12/d5f/d7e/fc9 [0,4194304] 0 2024-04-22T23:23:33.836 INFO:tasks.workunit.client.0.smithi067.stdout:4/814: readlink d12/d5f/ld1 0 2024-04-22T23:23:33.838 INFO:tasks.workunit.client.0.smithi067.stdout:7/653: link d10/d14/d1d/d46/d50/f30 d10/d14/d1d/d26/d2c/ff2 0 2024-04-22T23:23:33.840 INFO:tasks.workunit.client.0.smithi067.stdout:4/815: mkdir d12/d42/d4b/d77/d9d/dc1/d111 0 2024-04-22T23:23:33.840 INFO:tasks.workunit.client.0.smithi067.stdout:7/654: creat d10/d14/d1d/d46/d6a/dcc/ff3 x:0 0 0 2024-04-22T23:23:33.846 INFO:tasks.workunit.client.0.smithi067.stdout:4/816: dread d12/f54 [0,4194304] 0 2024-04-22T23:23:33.846 INFO:tasks.workunit.client.0.smithi067.stdout:0/749: truncate db/d2e/dd9/fe1 608739 0 2024-04-22T23:23:33.847 INFO:tasks.workunit.client.0.smithi067.stdout:4/817: readlink d12/d17/d3f/l6f 0 2024-04-22T23:23:33.849 INFO:tasks.workunit.client.0.smithi067.stdout:4/818: rmdir d12/d5f/d2b/d97 39 2024-04-22T23:23:33.851 INFO:tasks.workunit.client.0.smithi067.stdout:0/750: truncate db/d71/dda/ff2 8904146 0 2024-04-22T23:23:33.853 INFO:tasks.workunit.client.0.smithi067.stdout:0/751: fsync db/de/d2a/f99 0 2024-04-22T23:23:33.854 INFO:tasks.workunit.client.0.smithi067.stdout:0/752: dread db/d2e/f8b [0,4194304] 0 2024-04-22T23:23:33.869 INFO:tasks.workunit.client.0.smithi067.stdout:4/819: dread d12/d17/d3f/fbe [0,4194304] 0 2024-04-22T23:23:33.872 INFO:tasks.workunit.client.0.smithi067.stdout:4/820: unlink d12/d5f/d7e/fd4 0 2024-04-22T23:23:33.874 INFO:tasks.workunit.client.0.smithi067.stdout:4/821: creat d12/d42/dd3/f112 x:0 0 0 2024-04-22T23:23:33.874 INFO:tasks.workunit.client.0.smithi067.stdout:0/753: dread db/d71/dda/ff2 [4194304,4194304] 0 2024-04-22T23:23:33.876 INFO:tasks.workunit.client.0.smithi067.stdout:4/822: creat d12/d5f/d69/f113 x:0 0 0 2024-04-22T23:23:33.877 INFO:tasks.workunit.client.0.smithi067.stdout:0/754: fdatasync db/de/f1c 0 2024-04-22T23:23:33.884 INFO:tasks.workunit.client.0.smithi067.stdout:0/755: dwrite db/d21/d29/d2d/f49 [4194304,4194304] 0 2024-04-22T23:23:33.887 INFO:tasks.workunit.client.0.smithi067.stdout:0/756: creat db/d71/dda/ff9 x:0 0 0 2024-04-22T23:23:33.889 INFO:tasks.workunit.client.0.smithi067.stdout:4/823: dread d12/d17/d3f/d7a/fcc [0,4194304] 0 2024-04-22T23:23:33.890 INFO:tasks.workunit.client.0.smithi067.stdout:4/824: stat d12/d42/d4b/l82 0 2024-04-22T23:23:33.894 INFO:tasks.workunit.client.0.smithi067.stdout:4/825: read d12/f38 [1715599,7133] 0 2024-04-22T23:23:33.894 INFO:tasks.workunit.client.0.smithi067.stdout:0/757: rename db/d21/fd8 to db/de/d11/d1a/da4/dec/ffa 0 2024-04-22T23:23:33.896 INFO:tasks.workunit.client.0.smithi067.stdout:4/826: truncate d12/d5f/d2b/ded/d8e/d56/f5c 327399 0 2024-04-22T23:23:33.898 INFO:tasks.workunit.client.0.smithi067.stdout:0/758: link db/d9a/d56/d70/d8c/dc2/le0 db/d60/lfb 0 2024-04-22T23:23:33.898 INFO:tasks.workunit.client.0.smithi067.stdout:0/759: chown db/faf 9848702 1 2024-04-22T23:23:33.900 INFO:tasks.workunit.client.0.smithi067.stdout:4/827: rename d12/d5f/d2b/ded/d8e/d56/d8b to d12/d5f/d2b/ded/d8e/d56/d114 0 2024-04-22T23:23:33.902 INFO:tasks.workunit.client.0.smithi067.stdout:0/760: rmdir db/d9a/d56/d6b/db9 39 2024-04-22T23:23:33.903 INFO:tasks.workunit.client.0.smithi067.stdout:4/828: write d12/d17/d47/f49 [9598009,59064] 0 2024-04-22T23:23:33.910 INFO:tasks.workunit.client.0.smithi067.stdout:0/761: truncate db/d9a/d56/f6d 3667269 0 2024-04-22T23:23:33.911 INFO:tasks.workunit.client.0.smithi067.stdout:4/829: truncate d12/d17/d3f/d84/f103 565974 0 2024-04-22T23:23:33.912 INFO:tasks.workunit.client.0.smithi067.stdout:0/762: mkdir db/de/dfc 0 2024-04-22T23:23:33.913 INFO:tasks.workunit.client.0.smithi067.stdout:0/763: write db/d9a/ff0 [395846,116090] 0 2024-04-22T23:23:33.914 INFO:tasks.workunit.client.0.smithi067.stdout:0/764: chown db/de/d2a/c6f 3278 1 2024-04-22T23:23:33.916 INFO:tasks.workunit.client.0.smithi067.stdout:0/765: chown db/l18 2437531 1 2024-04-22T23:23:33.917 INFO:tasks.workunit.client.0.smithi067.stdout:0/766: fsync db/d21/d29/d2d/f49 0 2024-04-22T23:23:33.919 INFO:tasks.workunit.client.0.smithi067.stdout:4/830: write d12/d2e/f57 [910491,92849] 0 2024-04-22T23:23:33.926 INFO:tasks.workunit.client.0.smithi067.stdout:4/831: dwrite d12/d17/d47/fe5 [0,4194304] 0 2024-04-22T23:23:33.931 INFO:tasks.workunit.client.0.smithi067.stdout:4/832: fdatasync d12/d17/d3f/d84/f9f 0 2024-04-22T23:23:33.934 INFO:tasks.workunit.client.0.smithi067.stdout:4/833: creat d12/d42/d4b/d77/d9d/f115 x:0 0 0 2024-04-22T23:23:33.938 INFO:tasks.workunit.client.0.smithi067.stdout:0/767: dread db/d21/d29/d2c/f91 [0,4194304] 0 2024-04-22T23:23:33.940 INFO:tasks.workunit.client.0.smithi067.stdout:0/768: write db/d21/d29/d75/d9d/ff7 [1709049,76006] 0 2024-04-22T23:23:33.940 INFO:tasks.workunit.client.0.smithi067.stdout:4/834: write d12/d17/d47/f86 [1439883,9587] 0 2024-04-22T23:23:33.943 INFO:tasks.workunit.client.0.smithi067.stdout:0/769: truncate db/d21/d29/d2d/f42 3948769 0 2024-04-22T23:23:33.945 INFO:tasks.workunit.client.0.smithi067.stdout:0/770: readlink db/d21/d29/d75/d9d/le8 0 2024-04-22T23:23:33.947 INFO:tasks.workunit.client.0.smithi067.stdout:4/835: dwrite d12/d17/d47/fe5 [0,4194304] 0 2024-04-22T23:23:33.949 INFO:tasks.workunit.client.0.smithi067.stdout:4/836: unlink d12/d5f/d69/la3 0 2024-04-22T23:23:33.957 INFO:tasks.workunit.client.0.smithi067.stdout:1/649: sync 2024-04-22T23:23:33.960 INFO:tasks.workunit.client.0.smithi067.stdout:1/650: mknod d8/d9/d2f/d33/d5c/dd9/cdd 0 2024-04-22T23:23:33.964 INFO:tasks.workunit.client.0.smithi067.stdout:1/651: rename d8/d12/d36/l99 to d8/d9/d2f/d31/lde 0 2024-04-22T23:23:33.966 INFO:tasks.workunit.client.0.smithi067.stdout:1/652: truncate d8/d12/d36/d87/fad 437684 0 2024-04-22T23:23:33.968 INFO:tasks.workunit.client.0.smithi067.stdout:4/837: dread d12/d17/d88/fc8 [0,4194304] 0 2024-04-22T23:23:33.970 INFO:tasks.workunit.client.0.smithi067.stdout:0/771: dread db/d21/f4c [0,4194304] 0 2024-04-22T23:23:33.971 INFO:tasks.workunit.client.0.smithi067.stdout:4/838: mknod d12/d17/d3f/c116 0 2024-04-22T23:23:33.972 INFO:tasks.workunit.client.0.smithi067.stdout:0/772: creat db/de/d2a/ffd x:0 0 0 2024-04-22T23:23:33.973 INFO:tasks.workunit.client.0.smithi067.stdout:0/773: truncate db/d9a/d56/d70/fc3 604570 0 2024-04-22T23:23:33.974 INFO:tasks.workunit.client.0.smithi067.stdout:4/839: creat d12/d5f/d2b/ded/d8e/d56/d114/dc3/f117 x:0 0 0 2024-04-22T23:23:33.975 INFO:tasks.workunit.client.0.smithi067.stdout:4/840: dread - d12/d42/d4b/d77/d9d/f115 zero size 2024-04-22T23:23:33.978 INFO:tasks.workunit.client.0.smithi067.stdout:4/841: mknod d12/d5f/d2b/db6/c118 0 2024-04-22T23:23:33.978 INFO:tasks.workunit.client.0.smithi067.stdout:1/653: truncate d8/d12/d36/fba 4090619 0 2024-04-22T23:23:33.980 INFO:tasks.workunit.client.0.smithi067.stdout:1/654: dread - d8/d9/d2d/d4d/d48/d9a/fac zero size 2024-04-22T23:23:33.981 INFO:tasks.workunit.client.0.smithi067.stdout:0/774: dwrite db/faa [0,4194304] 0 2024-04-22T23:23:33.986 INFO:tasks.workunit.client.0.smithi067.stdout:0/775: creat db/de/ffe x:0 0 0 2024-04-22T23:23:33.988 INFO:tasks.workunit.client.0.smithi067.stdout:1/655: dwrite d8/d12/f94 [0,4194304] 0 2024-04-22T23:23:33.990 INFO:tasks.workunit.client.0.smithi067.stdout:0/776: symlink db/d21/d29/d2d/lff 0 2024-04-22T23:23:33.991 INFO:tasks.workunit.client.0.smithi067.stdout:4/842: write d12/d17/d3f/d84/f103 [425478,96490] 0 2024-04-22T23:23:33.991 INFO:tasks.workunit.client.0.smithi067.stdout:1/656: dread d8/d9/d2d/d4d/d48/f8d [0,4194304] 0 2024-04-22T23:23:33.992 INFO:tasks.workunit.client.0.smithi067.stdout:0/777: fdatasync db/d60/d9e/fe7 0 2024-04-22T23:23:33.996 INFO:tasks.workunit.client.0.smithi067.stdout:1/657: creat d8/d12/d36/d87/fdf x:0 0 0 2024-04-22T23:23:33.996 INFO:tasks.workunit.client.0.smithi067.stdout:4/843: chown d12/d5f/d2b/ded/d8e/fd2 8052 1 2024-04-22T23:23:33.997 INFO:tasks.workunit.client.0.smithi067.stdout:0/778: mkdir db/d21/d29/d75/d9d/dde/d100 0 2024-04-22T23:23:34.002 INFO:tasks.workunit.client.0.smithi067.stdout:1/658: rename d8/d9/d2d/d4d/d70/d34 to d8/d9/de0 0 2024-04-22T23:23:34.003 INFO:tasks.workunit.client.0.smithi067.stdout:4/844: mknod d12/d5f/d2b/ded/d8e/d56/d93/c119 0 2024-04-22T23:23:34.004 INFO:tasks.workunit.client.0.smithi067.stdout:1/659: truncate d8/d9/d2f/d31/f8e 85419 0 2024-04-22T23:23:34.005 INFO:tasks.workunit.client.0.smithi067.stdout:0/779: dread db/d21/d29/d2c/f7d [0,4194304] 0 2024-04-22T23:23:34.006 INFO:tasks.workunit.client.0.smithi067.stdout:4/845: creat d12/d5f/f11a x:0 0 0 2024-04-22T23:23:34.006 INFO:tasks.workunit.client.0.smithi067.stdout:1/660: fdatasync d8/d12/fd6 0 2024-04-22T23:23:34.008 INFO:tasks.workunit.client.0.smithi067.stdout:4/846: write d12/d5f/f41 [204981,41666] 0 2024-04-22T23:23:34.008 INFO:tasks.workunit.client.0.smithi067.stdout:0/780: mkdir db/d60/d9e/d101 0 2024-04-22T23:23:34.016 INFO:tasks.workunit.client.0.smithi067.stdout:4/847: dwrite d12/d5f/d2b/ded/d8e/d56/d114/fd7 [0,4194304] 0 2024-04-22T23:23:34.021 INFO:tasks.workunit.client.0.smithi067.stdout:0/781: write db/de/f1c [768161,28742] 0 2024-04-22T23:23:34.023 INFO:tasks.workunit.client.0.smithi067.stdout:0/782: stat db/de/dfc 0 2024-04-22T23:23:34.026 INFO:tasks.workunit.client.0.smithi067.stdout:4/848: dwrite d12/d17/d3f/f89 [0,4194304] 0 2024-04-22T23:23:34.027 INFO:tasks.workunit.client.0.smithi067.stdout:1/661: dwrite d8/f4b [0,4194304] 0 2024-04-22T23:23:34.032 INFO:tasks.workunit.client.0.smithi067.stdout:1/662: rmdir d8/d9/d2f/d33/d5c/dd9 39 2024-04-22T23:23:34.036 INFO:tasks.workunit.client.0.smithi067.stdout:4/849: dwrite d12/d5f/d2b/ded/f6e [0,4194304] 0 2024-04-22T23:23:34.038 INFO:tasks.workunit.client.0.smithi067.stdout:4/850: write d12/d5f/d2b/ded/d8e/d56/d114/f107 [797793,119346] 0 2024-04-22T23:23:34.055 INFO:tasks.workunit.client.0.smithi067.stdout:1/663: write d8/d9/d2f/d33/d5c/f65 [2297484,5801] 0 2024-04-22T23:23:34.056 INFO:tasks.workunit.client.0.smithi067.stdout:4/851: write d12/d17/d3f/d84/f9f [1021557,35656] 0 2024-04-22T23:23:34.058 INFO:tasks.workunit.client.0.smithi067.stdout:1/664: mknod d8/d9/d2d/ce1 0 2024-04-22T23:23:34.059 INFO:tasks.workunit.client.0.smithi067.stdout:1/665: dread - d8/d12/d36/f4e zero size 2024-04-22T23:23:34.062 INFO:tasks.workunit.client.0.smithi067.stdout:4/852: dwrite d12/d17/d31/f50 [0,4194304] 0 2024-04-22T23:23:34.063 INFO:tasks.workunit.client.0.smithi067.stdout:4/853: dread - d12/d5f/d2b/ded/d8e/fd2 zero size 2024-04-22T23:23:34.064 INFO:tasks.workunit.client.0.smithi067.stdout:4/854: chown d12/d5f/l73 3962747 1 2024-04-22T23:23:34.065 INFO:tasks.workunit.client.0.smithi067.stdout:4/855: chown d12/d42/d4b/d77/d9d/dc1/cf5 5243106 1 2024-04-22T23:23:34.067 INFO:tasks.workunit.client.0.smithi067.stdout:4/856: mknod d12/d42/d4b/c11b 0 2024-04-22T23:23:34.073 INFO:tasks.workunit.client.0.smithi067.stdout:1/666: dread d8/d9/d2f/d59/f81 [0,4194304] 0 2024-04-22T23:23:34.076 INFO:tasks.workunit.client.0.smithi067.stdout:1/667: mkdir d8/d9/de0/d5d/de2 0 2024-04-22T23:23:34.077 INFO:tasks.workunit.client.0.smithi067.stdout:1/668: truncate d8/d9/d2d/d4d/d9c/f96 500028 0 2024-04-22T23:23:34.078 INFO:tasks.workunit.client.0.smithi067.stdout:1/669: write d8/d9/d2f/d31/f58 [830171,69233] 0 2024-04-22T23:23:34.082 INFO:tasks.workunit.client.0.smithi067.stdout:1/670: symlink d8/d9/d2f/d33/d5c/dd9/le3 0 2024-04-22T23:23:34.082 INFO:tasks.workunit.client.0.smithi067.stdout:4/857: dread fa [0,4194304] 0 2024-04-22T23:23:34.083 INFO:tasks.workunit.client.0.smithi067.stdout:1/671: chown d8/d9/de0/d5d/f8b 0 1 2024-04-22T23:23:34.083 INFO:tasks.workunit.client.0.smithi067.stdout:1/672: chown d8/d9/d2f 0 1 2024-04-22T23:23:34.084 INFO:tasks.workunit.client.0.smithi067.stdout:1/673: chown d8/d9/d2f 1183619 1 2024-04-22T23:23:34.084 INFO:tasks.workunit.client.0.smithi067.stdout:1/674: fsync d8/d9/d2f/d33/fb2 0 2024-04-22T23:23:34.088 INFO:tasks.workunit.client.0.smithi067.stdout:4/858: dread d12/d42/dd3/dd8/fe2 [0,4194304] 0 2024-04-22T23:23:34.089 INFO:tasks.workunit.client.0.smithi067.stdout:1/675: read d8/d9/de0/d60/d62/fa5 [1595040,128379] 0 2024-04-22T23:23:34.092 INFO:tasks.workunit.client.0.smithi067.stdout:1/676: mknod d8/d4a/ce4 0 2024-04-22T23:23:34.093 INFO:tasks.workunit.client.0.smithi067.stdout:4/859: link d12/d17/d3f/f63 d12/d5f/d2b/ded/d8e/d56/d93/f11c 0 2024-04-22T23:23:34.098 INFO:tasks.workunit.client.0.smithi067.stdout:4/860: creat d12/d42/d4b/d77/d9d/dc1/d111/f11d x:0 0 0 2024-04-22T23:23:34.098 INFO:tasks.workunit.client.0.smithi067.stdout:1/677: dwrite d8/d12/d40/fa4 [0,4194304] 0 2024-04-22T23:23:34.099 INFO:tasks.workunit.client.0.smithi067.stdout:4/861: symlink d12/d42/dd3/dd8/l11e 0 2024-04-22T23:23:34.104 INFO:tasks.workunit.client.0.smithi067.stdout:1/678: unlink d8/d4a/l68 0 2024-04-22T23:23:34.105 INFO:tasks.workunit.client.0.smithi067.stdout:4/862: unlink d12/f46 0 2024-04-22T23:23:34.107 INFO:tasks.workunit.client.0.smithi067.stdout:1/679: mknod d8/d9/d2d/d4d/d48/ce5 0 2024-04-22T23:23:34.114 INFO:tasks.workunit.client.0.smithi067.stdout:1/680: truncate d8/d9/d2d/d4d/d9c/f8a 3408924 0 2024-04-22T23:23:34.115 INFO:tasks.workunit.client.0.smithi067.stdout:1/681: dread - d8/d9/d2d/d4d/d48/fd0 zero size 2024-04-22T23:23:34.126 INFO:tasks.workunit.client.0.smithi067.stdout:1/682: dread d8/d9/de0/d60/d62/fa8 [0,4194304] 0 2024-04-22T23:23:34.128 INFO:tasks.workunit.client.0.smithi067.stdout:1/683: mkdir d8/de6 0 2024-04-22T23:23:34.130 INFO:tasks.workunit.client.0.smithi067.stdout:1/684: dread d8/d9/fd [0,4194304] 0 2024-04-22T23:23:34.133 INFO:tasks.workunit.client.0.smithi067.stdout:1/685: dread d8/f1d [0,4194304] 0 2024-04-22T23:23:34.135 INFO:tasks.workunit.client.0.smithi067.stdout:1/686: mknod d8/d9/d2d/ce7 0 2024-04-22T23:23:34.135 INFO:tasks.workunit.client.0.smithi067.stdout:1/687: chown d8/d12/f94 772 1 2024-04-22T23:23:34.136 INFO:tasks.workunit.client.0.smithi067.stdout:1/688: readlink d8/d9/de0/d60/d62/l9d 0 2024-04-22T23:23:34.137 INFO:tasks.workunit.client.0.smithi067.stdout:1/689: fdatasync d8/d9/d2f/d33/d5c/f98 0 2024-04-22T23:23:34.138 INFO:tasks.workunit.client.0.smithi067.stdout:1/690: creat d8/d9/d2f/d33/d66/fe8 x:0 0 0 2024-04-22T23:23:34.140 INFO:tasks.workunit.client.0.smithi067.stdout:1/691: unlink d8/d9/d2f/d33/d66/f73 0 2024-04-22T23:23:34.149 INFO:tasks.workunit.client.0.smithi067.stdout:3/740: sync 2024-04-22T23:23:34.150 INFO:tasks.workunit.client.0.smithi067.stdout:5/747: sync 2024-04-22T23:23:34.151 INFO:tasks.workunit.client.0.smithi067.stdout:3/741: symlink d3/d4/d10/d19/l108 0 2024-04-22T23:23:34.153 INFO:tasks.workunit.client.0.smithi067.stdout:1/692: dread d8/d9/f2c [0,4194304] 0 2024-04-22T23:23:34.155 INFO:tasks.workunit.client.0.smithi067.stdout:3/742: link d3/d4/d28/dc0/fcc d3/d4/dec/f109 0 2024-04-22T23:23:34.156 INFO:tasks.workunit.client.0.smithi067.stdout:3/743: fsync d3/d4/f79 0 2024-04-22T23:23:34.157 INFO:tasks.workunit.client.0.smithi067.stdout:3/744: chown d3/d4/f37 32699 1 2024-04-22T23:23:34.159 INFO:tasks.workunit.client.0.smithi067.stdout:1/693: dwrite d8/d12/d36/d87/fdf [0,4194304] 0 2024-04-22T23:23:34.164 INFO:tasks.workunit.client.0.smithi067.stdout:1/694: unlink d8/d9/de0/f37 0 2024-04-22T23:23:34.172 INFO:tasks.workunit.client.0.smithi067.stdout:3/745: dread d3/d4/d10/d38/d53/d55/d74/dd5/ddb/f52 [0,4194304] 0 2024-04-22T23:23:34.177 INFO:tasks.workunit.client.0.smithi067.stdout:3/746: dread d3/fc1 [0,4194304] 0 2024-04-22T23:23:34.180 INFO:tasks.workunit.client.0.smithi067.stdout:3/747: mkdir d3/d4/d10/d38/d53/d55/d5c/d70/d10a 0 2024-04-22T23:23:34.182 INFO:tasks.workunit.client.0.smithi067.stdout:3/748: chown d3/d4/d28/d40/dfd/dfe/lb8 687 1 2024-04-22T23:23:34.182 INFO:tasks.workunit.client.0.smithi067.stdout:1/695: dwrite d8/d12/d36/fba [0,4194304] 0 2024-04-22T23:23:34.187 INFO:tasks.workunit.client.0.smithi067.stdout:1/696: link d8/d9/d2d/d4d/d48/c51 d8/d9/de0/d5d/de2/ce9 0 2024-04-22T23:23:34.188 INFO:tasks.workunit.client.0.smithi067.stdout:3/749: rename d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d62 to d3/d4/d10/d10b 0 2024-04-22T23:23:34.189 INFO:tasks.workunit.client.0.smithi067.stdout:1/697: mknod d8/d9/d2d/d4d/d9c/cea 0 2024-04-22T23:23:34.190 INFO:tasks.workunit.client.0.smithi067.stdout:3/750: fsync d3/f6f 0 2024-04-22T23:23:34.195 INFO:tasks.workunit.client.0.smithi067.stdout:1/698: dwrite d8/d12/d40/f44 [0,4194304] 0 2024-04-22T23:23:34.196 INFO:tasks.workunit.client.0.smithi067.stdout:3/751: link d3/d4/d10/d8a/fa6 d3/d4/d1b/d93/f10c 0 2024-04-22T23:23:34.199 INFO:tasks.workunit.client.0.smithi067.stdout:3/752: symlink d3/d4/d28/d40/dfd/dfe/d4b/l10d 0 2024-04-22T23:23:34.203 INFO:tasks.workunit.client.0.smithi067.stdout:1/699: read d8/d9/de0/d60/d62/fa5 [1365500,83573] 0 2024-04-22T23:23:34.203 INFO:tasks.workunit.client.0.smithi067.stdout:3/753: mkdir d3/d4/d10/d38/d53/d55/d10e 0 2024-04-22T23:23:34.206 INFO:tasks.workunit.client.0.smithi067.stdout:3/754: symlink d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d90/da2/l10f 0 2024-04-22T23:23:34.208 INFO:tasks.workunit.client.0.smithi067.stdout:1/700: symlink d8/d9/d2d/d4d/d70/leb 0 2024-04-22T23:23:34.214 INFO:tasks.workunit.client.0.smithi067.stdout:1/701: dwrite d8/d9/fdc [0,4194304] 0 2024-04-22T23:23:34.215 INFO:tasks.workunit.client.0.smithi067.stdout:1/702: chown d8/d9/de0/d5d/c86 467887 1 2024-04-22T23:23:34.216 INFO:tasks.workunit.client.0.smithi067.stdout:2/699: sync 2024-04-22T23:23:34.216 INFO:tasks.workunit.client.0.smithi067.stdout:9/681: sync 2024-04-22T23:23:34.216 INFO:tasks.workunit.client.0.smithi067.stdout:7/655: sync 2024-04-22T23:23:34.216 INFO:tasks.workunit.client.0.smithi067.stdout:8/673: sync 2024-04-22T23:23:34.216 INFO:tasks.workunit.client.0.smithi067.stdout:6/704: sync 2024-04-22T23:23:34.219 INFO:tasks.workunit.client.0.smithi067.stdout:8/674: chown d1/dd/d80/f8c 125050069 1 2024-04-22T23:23:34.219 INFO:tasks.workunit.client.0.smithi067.stdout:7/656: fsync d10/d14/d1d/d69/fba 0 2024-04-22T23:23:34.220 INFO:tasks.workunit.client.0.smithi067.stdout:9/682: dread - d5/d21/d47/d56/f7c zero size 2024-04-22T23:23:34.222 INFO:tasks.workunit.client.0.smithi067.stdout:8/675: creat d1/dd/d26/fd6 x:0 0 0 2024-04-22T23:23:34.240 INFO:tasks.workunit.client.0.smithi067.stdout:7/657: rmdir d10/d14/d1d/d46/d50/dcd 39 2024-04-22T23:23:34.241 INFO:tasks.workunit.client.0.smithi067.stdout:7/658: fdatasync d10/d14/d1d/d26/d2c/d7a/fa5 0 2024-04-22T23:23:34.242 INFO:tasks.workunit.client.0.smithi067.stdout:9/683: creat d5/d21/d47/d56/fec x:0 0 0 2024-04-22T23:23:34.242 INFO:tasks.workunit.client.0.smithi067.stdout:7/659: write d10/d14/d1d/d46/d50/f29 [4429047,120873] 0 2024-04-22T23:23:34.245 INFO:tasks.workunit.client.0.smithi067.stdout:8/676: mkdir d1/dd/d80/d6f/da8/db8/dd7 0 2024-04-22T23:23:34.249 INFO:tasks.workunit.client.0.smithi067.stdout:6/705: rename d17/d39/l67 to d17/d39/d53/d90/lea 0 2024-04-22T23:23:34.250 INFO:tasks.workunit.client.0.smithi067.stdout:6/706: chown d17/d39/d3a/da9/lb8 985 1 2024-04-22T23:23:34.250 INFO:tasks.workunit.client.0.smithi067.stdout:9/684: creat d5/dd/d18/d39/d4d/fed x:0 0 0 2024-04-22T23:23:34.251 INFO:tasks.workunit.client.0.smithi067.stdout:9/685: chown d5/dd/d18/d3c/d40/f6b 4581225 1 2024-04-22T23:23:34.251 INFO:tasks.workunit.client.0.smithi067.stdout:8/677: creat d1/d3/d5c/d98/fd8 x:0 0 0 2024-04-22T23:23:34.252 INFO:tasks.workunit.client.0.smithi067.stdout:9/686: chown d5/d21/l37 393 1 2024-04-22T23:23:34.254 INFO:tasks.workunit.client.0.smithi067.stdout:1/703: write d8/d9/de0/d5d/f8b [1004309,112638] 0 2024-04-22T23:23:34.255 INFO:tasks.workunit.client.0.smithi067.stdout:6/707: creat d17/dae/feb x:0 0 0 2024-04-22T23:23:34.256 INFO:tasks.workunit.client.0.smithi067.stdout:9/687: write d5/d21/d47/d57/dc0/fc2 [29953,48509] 0 2024-04-22T23:23:34.257 INFO:tasks.workunit.client.0.smithi067.stdout:1/704: stat d8/d9/d2d/d4d/lbd 0 2024-04-22T23:23:34.259 INFO:tasks.workunit.client.0.smithi067.stdout:2/700: rename d1/d10/c94 to d1/d10/d72/d68/ce2 0 2024-04-22T23:23:34.264 INFO:tasks.workunit.client.0.smithi067.stdout:8/678: dwrite d1/d3/f13 [0,4194304] 0 2024-04-22T23:23:34.265 INFO:tasks.workunit.client.0.smithi067.stdout:9/688: rmdir d5/d21/d47/d56/dc6 39 2024-04-22T23:23:34.267 INFO:tasks.workunit.client.0.smithi067.stdout:9/689: write d5/d21/d47/d56/fec [934501,102001] 0 2024-04-22T23:23:34.268 INFO:tasks.workunit.client.0.smithi067.stdout:7/660: rename d10/d14/d1d/d46/d50/dcd to d10/d14/d1d/d26/d2c/df4 0 2024-04-22T23:23:34.285 INFO:tasks.workunit.client.0.smithi067.stdout:6/708: dread d17/d39/d3a/f4f [0,4194304] 0 2024-04-22T23:23:34.288 INFO:tasks.workunit.client.0.smithi067.stdout:2/701: dwrite d1/d10/d1c/f5d [0,4194304] 0 2024-04-22T23:23:34.292 INFO:tasks.workunit.client.0.smithi067.stdout:1/705: dread d8/d9/d2f/d33/d66/f91 [0,4194304] 0 2024-04-22T23:23:34.295 INFO:tasks.workunit.client.0.smithi067.stdout:6/709: dwrite d17/f24 [0,4194304] 0 2024-04-22T23:23:34.297 INFO:tasks.workunit.client.0.smithi067.stdout:6/710: dread - d17/d39/d3a/fe0 zero size 2024-04-22T23:23:34.304 INFO:tasks.workunit.client.0.smithi067.stdout:6/711: dwrite d17/d39/f69 [0,4194304] 0 2024-04-22T23:23:34.306 INFO:tasks.workunit.client.0.smithi067.stdout:8/679: symlink d1/d3/ld9 0 2024-04-22T23:23:34.306 INFO:tasks.workunit.client.0.smithi067.stdout:9/690: truncate d5/dd/d18/d39/d3e/f53 3134734 0 2024-04-22T23:23:34.309 INFO:tasks.workunit.client.0.smithi067.stdout:7/661: rename d10/d14/d1d/d26/fc1 to d10/d14/d1d/d26/d80/dd2/dd4/ff5 0 2024-04-22T23:23:34.309 INFO:tasks.workunit.client.0.smithi067.stdout:9/691: dread - d5/dd/d18/d39/d3e/d3f/d63/fa8 zero size 2024-04-22T23:23:34.310 INFO:tasks.workunit.client.0.smithi067.stdout:8/680: truncate d1/d3/d5c/d98/fbc 4939285 0 2024-04-22T23:23:34.313 INFO:tasks.workunit.client.0.smithi067.stdout:1/706: mkdir d8/d9/d2f/d59/dec 0 2024-04-22T23:23:34.314 INFO:tasks.workunit.client.0.smithi067.stdout:6/712: dwrite d17/d39/d3a/f6d [0,4194304] 0 2024-04-22T23:23:34.323 INFO:tasks.workunit.client.0.smithi067.stdout:9/692: getdents d5/dd/d18/d3c/d78 0 2024-04-22T23:23:34.326 INFO:tasks.workunit.client.0.smithi067.stdout:8/681: dwrite d1/d41/d46/f66 [0,4194304] 0 2024-04-22T23:23:34.327 INFO:tasks.workunit.client.0.smithi067.stdout:7/662: dwrite d10/d14/d1d/d69/fb8 [0,4194304] 0 2024-04-22T23:23:34.329 INFO:tasks.workunit.client.0.smithi067.stdout:1/707: rmdir d8/d9/d2f/d33/d66 39 2024-04-22T23:23:34.330 INFO:tasks.workunit.client.0.smithi067.stdout:1/708: chown d8/d9/d2d/d4d/f1f 3 1 2024-04-22T23:23:34.330 INFO:tasks.workunit.client.0.smithi067.stdout:9/693: symlink d5/d21/d47/d56/d8c/dd5/lee 0 2024-04-22T23:23:34.332 INFO:tasks.workunit.client.0.smithi067.stdout:8/682: creat d1/d3/d10/fda x:0 0 0 2024-04-22T23:23:34.333 INFO:tasks.workunit.client.0.smithi067.stdout:7/663: creat d10/d18/d35/d62/ff6 x:0 0 0 2024-04-22T23:23:34.336 INFO:tasks.workunit.client.0.smithi067.stdout:9/694: mkdir d5/d21/d47/d7f/def 0 2024-04-22T23:23:34.337 INFO:tasks.workunit.client.0.smithi067.stdout:1/709: dwrite d8/d9/d2d/d4d/d83/fb9 [0,4194304] 0 2024-04-22T23:23:34.339 INFO:tasks.workunit.client.0.smithi067.stdout:8/683: rmdir d1/d41/db4 39 2024-04-22T23:23:34.339 INFO:tasks.workunit.client.0.smithi067.stdout:8/684: fsync d1/d3/d10/f3d 0 2024-04-22T23:23:34.342 INFO:tasks.workunit.client.0.smithi067.stdout:9/695: mknod d5/dd/d18/d39/d49/d54/d99/cf0 0 2024-04-22T23:23:34.345 INFO:tasks.workunit.client.0.smithi067.stdout:7/664: dwrite d10/d14/d1d/d46/d6a/f70 [0,4194304] 0 2024-04-22T23:23:34.346 INFO:tasks.workunit.client.0.smithi067.stdout:0/783: sync 2024-04-22T23:23:34.347 INFO:tasks.workunit.client.0.smithi067.stdout:7/665: chown d10/d14/d1d/d26/d80/dd2/dd4/ff5 716 1 2024-04-22T23:23:34.347 INFO:tasks.workunit.client.0.smithi067.stdout:8/685: creat d1/dd/d80/fdb x:0 0 0 2024-04-22T23:23:34.349 INFO:tasks.workunit.client.0.smithi067.stdout:0/784: chown db/d21/d29/f43 366 1 2024-04-22T23:23:34.349 INFO:tasks.workunit.client.0.smithi067.stdout:8/686: write d1/f7 [4755220,99265] 0 2024-04-22T23:23:34.349 INFO:tasks.workunit.client.0.smithi067.stdout:7/666: read d10/d14/d1d/d46/d6a/f70 [646878,102113] 0 2024-04-22T23:23:34.350 INFO:tasks.workunit.client.0.smithi067.stdout:8/687: write d1/faf [1227274,65856] 0 2024-04-22T23:23:34.351 INFO:tasks.workunit.client.0.smithi067.stdout:0/785: truncate db/de/d11/d92/fee 297875 0 2024-04-22T23:23:34.353 INFO:tasks.workunit.client.0.smithi067.stdout:7/667: dread - d10/d14/d1d/d26/d2c/f56 zero size 2024-04-22T23:23:34.354 INFO:tasks.workunit.client.0.smithi067.stdout:7/668: read d10/d14/d1d/d69/fb8 [3118553,40357] 0 2024-04-22T23:23:34.355 INFO:tasks.workunit.client.0.smithi067.stdout:8/688: creat d1/dd/d80/d6f/fdc x:0 0 0 2024-04-22T23:23:34.355 INFO:tasks.workunit.client.0.smithi067.stdout:7/669: fdatasync d10/d14/f5a 0 2024-04-22T23:23:34.356 INFO:tasks.workunit.client.0.smithi067.stdout:7/670: fdatasync d10/d18/d88/fa2 0 2024-04-22T23:23:34.359 INFO:tasks.workunit.client.0.smithi067.stdout:8/689: creat d1/dd/d26/d62/d75/fdd x:0 0 0 2024-04-22T23:23:34.365 INFO:tasks.workunit.client.0.smithi067.stdout:6/713: write f12 [739390,95740] 0 2024-04-22T23:23:34.368 INFO:tasks.workunit.client.0.smithi067.stdout:6/714: fdatasync d17/f21 0 2024-04-22T23:23:34.369 INFO:tasks.workunit.client.0.smithi067.stdout:2/702: dwrite d1/d10/d72/d53/f97 [0,4194304] 0 2024-04-22T23:23:34.371 INFO:tasks.workunit.client.0.smithi067.stdout:2/703: symlink d1/d10/d33/le3 0 2024-04-22T23:23:34.372 INFO:tasks.workunit.client.0.smithi067.stdout:6/715: creat d17/d39/d53/d90/d46/fec x:0 0 0 2024-04-22T23:23:34.374 INFO:tasks.workunit.client.0.smithi067.stdout:6/716: unlink d17/d39/d3a/f51 0 2024-04-22T23:23:34.376 INFO:tasks.workunit.client.0.smithi067.stdout:6/717: chown d17/d39/d53/d90/l8f 114512575 1 2024-04-22T23:23:34.377 INFO:tasks.workunit.client.0.smithi067.stdout:2/704: dwrite d1/d10/d1c/d6d/d87/fd8 [0,4194304] 0 2024-04-22T23:23:34.377 INFO:tasks.workunit.client.0.smithi067.stdout:6/718: fsync d17/d39/f49 0 2024-04-22T23:23:34.382 INFO:tasks.workunit.client.0.smithi067.stdout:6/719: creat d17/d39/d53/d90/d46/db0/de5/fed x:0 0 0 2024-04-22T23:23:34.382 INFO:tasks.workunit.client.0.smithi067.stdout:2/705: link d1/d10/d1c/f3a d1/d4f/fe4 0 2024-04-22T23:23:34.384 INFO:tasks.workunit.client.0.smithi067.stdout:6/720: read - d17/d39/d53/d90/d46/db0/db9/de7/fe1 zero size 2024-04-22T23:23:34.389 INFO:tasks.workunit.client.0.smithi067.stdout:6/721: symlink d17/d39/d53/d90/d46/d62/lee 0 2024-04-22T23:23:34.389 INFO:tasks.workunit.client.0.smithi067.stdout:2/706: link d1/d10/d1c/d6d/c83 d1/d10/d72/d68/d7b/dce/ce5 0 2024-04-22T23:23:34.391 INFO:tasks.workunit.client.0.smithi067.stdout:6/722: mknod d17/d83/d9b/ddd/cef 0 2024-04-22T23:23:34.392 INFO:tasks.workunit.client.0.smithi067.stdout:6/723: read - d17/d39/d53/d90/d46/db0/db9/fd2 zero size 2024-04-22T23:23:34.393 INFO:tasks.workunit.client.0.smithi067.stdout:2/707: mkdir d1/d10/d33/d46/de6 0 2024-04-22T23:23:34.394 INFO:tasks.workunit.client.0.smithi067.stdout:6/724: write d17/d39/d53/d90/d46/db0/db9/fd2 [753743,67493] 0 2024-04-22T23:23:34.395 INFO:tasks.workunit.client.0.smithi067.stdout:2/708: mknod d1/d42/d5f/ce7 0 2024-04-22T23:23:34.395 INFO:tasks.workunit.client.0.smithi067.stdout:2/709: chown d1/d10/d72/d53 1093732 1 2024-04-22T23:23:34.396 INFO:tasks.workunit.client.0.smithi067.stdout:6/725: readlink d17/d39/d53/d90/d46/d62/lac 0 2024-04-22T23:23:34.398 INFO:tasks.workunit.client.0.smithi067.stdout:2/710: truncate d1/d10/d33/fa5 425266 0 2024-04-22T23:23:34.399 INFO:tasks.workunit.client.0.smithi067.stdout:6/726: symlink d17/d39/d53/d90/d46/d62/d88/lf0 0 2024-04-22T23:23:34.400 INFO:tasks.workunit.client.0.smithi067.stdout:6/727: mknod d17/d83/d9b/cf1 0 2024-04-22T23:23:34.420 INFO:tasks.workunit.client.0.smithi067.stdout:6/728: dread d17/d39/d53/f74 [0,4194304] 0 2024-04-22T23:23:34.421 INFO:tasks.workunit.client.0.smithi067.stdout:6/729: write d17/d39/fa7 [665170,104904] 0 2024-04-22T23:23:34.423 INFO:tasks.workunit.client.0.smithi067.stdout:6/730: dread d17/d39/d3a/d8d/f97 [0,4194304] 0 2024-04-22T23:23:34.424 INFO:tasks.workunit.client.0.smithi067.stdout:6/731: readlink d17/l91 0 2024-04-22T23:23:34.426 INFO:tasks.workunit.client.0.smithi067.stdout:6/732: creat d17/d39/d53/d90/d46/db0/db9/de7/ff2 x:0 0 0 2024-04-22T23:23:34.426 INFO:tasks.workunit.client.0.smithi067.stdout:6/733: stat d17/d39/d53/d90/f94 0 2024-04-22T23:23:34.427 INFO:tasks.workunit.client.0.smithi067.stdout:6/734: chown c1 0 1 2024-04-22T23:23:34.489 INFO:tasks.workunit.client.0.smithi067.stdout:4/863: sync 2024-04-22T23:23:34.517 INFO:tasks.workunit.client.0.smithi067.stdout:4/864: read d12/d5f/d2b/ded/d8e/fcf [6189965,102016] 0 2024-04-22T23:23:34.518 INFO:tasks.workunit.client.0.smithi067.stdout:4/865: stat d12/d42/dd3/dd8/fe2 0 2024-04-22T23:23:34.519 INFO:tasks.workunit.client.0.smithi067.stdout:4/866: write d12/d17/f100 [817768,124692] 0 2024-04-22T23:23:34.524 INFO:tasks.workunit.client.0.smithi067.stdout:4/867: dwrite d12/d5f/d2b/f72 [0,4194304] 0 2024-04-22T23:23:34.532 INFO:tasks.workunit.client.0.smithi067.stdout:4/868: dwrite d12/d17/d3f/d84/f103 [0,4194304] 0 2024-04-22T23:23:34.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:34 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:23:34.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:34 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:23:34.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:34 smithi067 ceph-mon[26453]: pgmap v272: 65 pgs: 65 active+clean; 1.5 GiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 45 MiB/s rd, 161 MiB/s wr, 309 op/s 2024-04-22T23:23:34.561 INFO:tasks.workunit.client.0.smithi067.stdout:8/690: dread d1/d3/d27/f63 [0,4194304] 0 2024-04-22T23:23:34.568 INFO:tasks.workunit.client.0.smithi067.stdout:4/869: dread d12/d5f/d2b/ded/d8e/fcf [0,4194304] 0 2024-04-22T23:23:34.570 INFO:tasks.workunit.client.0.smithi067.stdout:1/710: write d8/d9/d2f/d33/d66/f9b [941285,54209] 0 2024-04-22T23:23:34.572 INFO:tasks.workunit.client.0.smithi067.stdout:4/870: creat d12/d5f/d2b/ded/d8e/d56/d114/dc3/f11f x:0 0 0 2024-04-22T23:23:34.577 INFO:tasks.workunit.client.0.smithi067.stdout:1/711: dread d8/d9/d2f/f7d [0,4194304] 0 2024-04-22T23:23:34.577 INFO:tasks.workunit.client.0.smithi067.stdout:0/786: dwrite db/d60/d9e/fc4 [4194304,4194304] 0 2024-04-22T23:23:34.578 INFO:tasks.workunit.client.0.smithi067.stdout:1/712: readlink d8/d4a/l80 0 2024-04-22T23:23:34.579 INFO:tasks.workunit.client.0.smithi067.stdout:1/713: chown d8/d9/de0/d60/d62 211691423 1 2024-04-22T23:23:34.581 INFO:tasks.workunit.client.0.smithi067.stdout:0/787: creat db/d9a/d56/d70/dba/f102 x:0 0 0 2024-04-22T23:23:34.582 INFO:tasks.workunit.client.0.smithi067.stdout:0/788: fdatasync db/d2e/f7c 0 2024-04-22T23:23:34.583 INFO:tasks.workunit.client.0.smithi067.stdout:1/714: getdents d8/d9/de0/d60/d62 0 2024-04-22T23:23:34.585 INFO:tasks.workunit.client.0.smithi067.stdout:0/789: creat db/de/d11/d1a/da4/dec/f103 x:0 0 0 2024-04-22T23:23:34.587 INFO:tasks.workunit.client.0.smithi067.stdout:1/715: mkdir d8/d9/d2f/d31/ded 0 2024-04-22T23:23:34.587 INFO:tasks.workunit.client.0.smithi067.stdout:4/871: dread d12/d5f/f5a [4194304,4194304] 0 2024-04-22T23:23:34.590 INFO:tasks.workunit.client.0.smithi067.stdout:4/872: chown d12/d5f/d2b/ded/d8e/d56/d114/fa7 49923 1 2024-04-22T23:23:34.593 INFO:tasks.workunit.client.0.smithi067.stdout:1/716: rename d8/d9/d2d/lbc to d8/d9/d2d/d4d/d48/lee 0 2024-04-22T23:23:34.597 INFO:tasks.workunit.client.0.smithi067.stdout:1/717: fsync d8/d12/d40/f69 0 2024-04-22T23:23:34.598 INFO:tasks.workunit.client.0.smithi067.stdout:1/718: fdatasync d8/d12/d36/fba 0 2024-04-22T23:23:34.599 INFO:tasks.workunit.client.0.smithi067.stdout:2/711: write d1/d10/d33/d46/f6e [435874,104093] 0 2024-04-22T23:23:34.605 INFO:tasks.workunit.client.0.smithi067.stdout:1/719: rmdir d8/d9/de0/d5d 39 2024-04-22T23:23:34.606 INFO:tasks.workunit.client.0.smithi067.stdout:4/873: link d12/f52 d12/d42/d4b/f120 0 2024-04-22T23:23:34.607 INFO:tasks.workunit.client.0.smithi067.stdout:0/790: link db/d21/d29/d75/fed db/d9a/d56/d6b/db9/f104 0 2024-04-22T23:23:34.609 INFO:tasks.workunit.client.0.smithi067.stdout:8/691: truncate d1/dd/f95 644960 0 2024-04-22T23:23:34.611 INFO:tasks.workunit.client.0.smithi067.stdout:8/692: write d1/d3/d27/f82 [786959,65654] 0 2024-04-22T23:23:34.613 INFO:tasks.workunit.client.0.smithi067.stdout:8/693: write d1/d3/d10/fba [755966,14175] 0 2024-04-22T23:23:34.614 INFO:tasks.workunit.client.0.smithi067.stdout:4/874: mknod d12/d5f/d2b/d97/c121 0 2024-04-22T23:23:34.616 INFO:tasks.workunit.client.0.smithi067.stdout:0/791: dwrite db/d9a/d56/d6b/fa2 [0,4194304] 0 2024-04-22T23:23:34.618 INFO:tasks.workunit.client.0.smithi067.stdout:4/875: chown d12/l2c 79064 1 2024-04-22T23:23:34.618 INFO:tasks.workunit.client.0.smithi067.stdout:0/792: write db/de/d2a/f5e [4735798,55124] 0 2024-04-22T23:23:34.621 INFO:tasks.workunit.client.0.smithi067.stdout:0/793: write db/d21/d29/fa7 [263800,81596] 0 2024-04-22T23:23:34.629 INFO:tasks.workunit.client.0.smithi067.stdout:4/876: rename d12/d5f/d2b/ded to d12/d42/d4b/d122 0 2024-04-22T23:23:34.632 INFO:tasks.workunit.client.0.smithi067.stdout:4/877: symlink d12/d42/d4b/l123 0 2024-04-22T23:23:34.634 INFO:tasks.workunit.client.0.smithi067.stdout:4/878: write d12/d2e/fb7 [1378689,28509] 0 2024-04-22T23:23:34.637 INFO:tasks.workunit.client.0.smithi067.stdout:0/794: dwrite db/d71/dda/ff2 [0,4194304] 0 2024-04-22T23:23:34.637 INFO:tasks.workunit.client.0.smithi067.stdout:8/694: dread d1/d3/d10/d3b/f5e [0,4194304] 0 2024-04-22T23:23:34.638 INFO:tasks.workunit.client.0.smithi067.stdout:4/879: symlink d12/d42/d4b/l124 0 2024-04-22T23:23:34.644 INFO:tasks.workunit.client.0.smithi067.stdout:5/748: sync 2024-04-22T23:23:34.644 INFO:tasks.workunit.client.0.smithi067.stdout:8/695: chown d1/d3/d5c/d98/fc9 14427767 1 2024-04-22T23:23:34.648 INFO:tasks.workunit.client.0.smithi067.stdout:5/749: read d5/d28/daf/dbf/dcf/fc7 [65411,55076] 0 2024-04-22T23:23:34.649 INFO:tasks.workunit.client.0.smithi067.stdout:0/795: mkdir db/de/d11/d1a/da4/dec/d105 0 2024-04-22T23:23:34.650 INFO:tasks.workunit.client.0.smithi067.stdout:4/880: dwrite d12/d42/d4b/d122/d8e/d56/d114/fd7 [0,4194304] 0 2024-04-22T23:23:34.656 INFO:tasks.workunit.client.0.smithi067.stdout:0/796: symlink db/d21/d29/d2d/l106 0 2024-04-22T23:23:34.659 INFO:tasks.workunit.client.0.smithi067.stdout:4/881: dwrite d12/d17/d88/fdc [0,4194304] 0 2024-04-22T23:23:34.661 INFO:tasks.workunit.client.0.smithi067.stdout:4/882: fdatasync d12/d5f/d2b/d97/f10a 0 2024-04-22T23:23:34.661 INFO:tasks.workunit.client.0.smithi067.stdout:8/696: link d1/fc0 d1/fde 0 2024-04-22T23:23:34.662 INFO:tasks.workunit.client.0.smithi067.stdout:8/697: stat d1/d3/d5c/faa 0 2024-04-22T23:23:34.668 INFO:tasks.workunit.client.0.smithi067.stdout:5/750: creat d5/f102 x:0 0 0 2024-04-22T23:23:34.668 INFO:tasks.workunit.client.0.smithi067.stdout:5/751: dread - d5/f102 zero size 2024-04-22T23:23:34.671 INFO:tasks.workunit.client.0.smithi067.stdout:8/698: mkdir d1/dd/d26/d62/ddf 0 2024-04-22T23:23:34.675 INFO:tasks.workunit.client.0.smithi067.stdout:5/752: unlink d5/db/d12/d44/d47/f6d 0 2024-04-22T23:23:34.677 INFO:tasks.workunit.client.0.smithi067.stdout:5/753: write d5/f68 [148507,114731] 0 2024-04-22T23:23:34.678 INFO:tasks.workunit.client.0.smithi067.stdout:5/754: write d5/d28/d36/d40/f70 [1578418,57874] 0 2024-04-22T23:23:34.683 INFO:tasks.workunit.client.0.smithi067.stdout:5/755: rename d5/d3a to d5/d1a/d103 0 2024-04-22T23:23:34.685 INFO:tasks.workunit.client.0.smithi067.stdout:8/699: getdents d1/d3/d27 0 2024-04-22T23:23:34.685 INFO:tasks.workunit.client.0.smithi067.stdout:5/756: mkdir d5/d1a/d103/d4d/d104 0 2024-04-22T23:23:34.686 INFO:tasks.workunit.client.0.smithi067.stdout:0/797: dread db/d21/d29/d75/fed [0,4194304] 0 2024-04-22T23:23:34.690 INFO:tasks.workunit.client.0.smithi067.stdout:0/798: mknod db/d71/dda/c107 0 2024-04-22T23:23:34.692 INFO:tasks.workunit.client.0.smithi067.stdout:8/700: dwrite d1/d3/f56 [0,4194304] 0 2024-04-22T23:23:34.694 INFO:tasks.workunit.client.0.smithi067.stdout:8/701: dread - d1/d41/d46/fce zero size 2024-04-22T23:23:34.698 INFO:tasks.workunit.client.0.smithi067.stdout:8/702: creat d1/d3/d10/fe0 x:0 0 0 2024-04-22T23:23:34.702 INFO:tasks.workunit.client.0.smithi067.stdout:5/757: dwrite d5/d28/d36/d40/f85 [0,4194304] 0 2024-04-22T23:23:34.703 INFO:tasks.workunit.client.0.smithi067.stdout:8/703: creat d1/dd/d80/d6f/da8/db8/dd7/fe1 x:0 0 0 2024-04-22T23:23:34.706 INFO:tasks.workunit.client.0.smithi067.stdout:0/799: dwrite db/d9a/d56/f6d [0,4194304] 0 2024-04-22T23:23:34.706 INFO:tasks.workunit.client.0.smithi067.stdout:5/758: creat d5/d28/d36/d40/f105 x:0 0 0 2024-04-22T23:23:34.707 INFO:tasks.workunit.client.0.smithi067.stdout:5/759: fdatasync d5/d1a/d103/d4d/fde 0 2024-04-22T23:23:34.708 INFO:tasks.workunit.client.0.smithi067.stdout:8/704: fsync d1/d3/d27/f81 0 2024-04-22T23:23:34.710 INFO:tasks.workunit.client.0.smithi067.stdout:5/760: chown d5/db/d12/dbb/d61 115963387 1 2024-04-22T23:23:34.710 INFO:tasks.workunit.client.0.smithi067.stdout:0/800: mkdir db/d60/d9e/d101/d108 0 2024-04-22T23:23:34.711 INFO:tasks.workunit.client.0.smithi067.stdout:8/705: read d1/d3/d5c/d98/fbc [1444671,91932] 0 2024-04-22T23:23:34.712 INFO:tasks.workunit.client.0.smithi067.stdout:5/761: mkdir d5/d106 0 2024-04-22T23:23:34.714 INFO:tasks.workunit.client.0.smithi067.stdout:0/801: fdatasync db/d21/d29/d2c/f72 0 2024-04-22T23:23:34.715 INFO:tasks.workunit.client.0.smithi067.stdout:8/706: chown d1/l21 42438034 1 2024-04-22T23:23:34.718 INFO:tasks.workunit.client.0.smithi067.stdout:5/762: rename d5/d28/ff9 to d5/db/d12/dbb/d5f/f107 0 2024-04-22T23:23:34.719 INFO:tasks.workunit.client.0.smithi067.stdout:8/707: rmdir d1/d3/d5c 39 2024-04-22T23:23:34.721 INFO:tasks.workunit.client.0.smithi067.stdout:0/802: rename db/de/d11/f7e to db/d2e/d46/f109 0 2024-04-22T23:23:34.722 INFO:tasks.workunit.client.0.smithi067.stdout:5/763: symlink d5/d1a/d103/d4d/dc6/l108 0 2024-04-22T23:23:34.722 INFO:tasks.workunit.client.0.smithi067.stdout:5/764: stat d5/d28/dae/fef 0 2024-04-22T23:23:34.723 INFO:tasks.workunit.client.0.smithi067.stdout:0/803: unlink db/de/l96 0 2024-04-22T23:23:34.724 INFO:tasks.workunit.client.0.smithi067.stdout:8/708: creat d1/fe2 x:0 0 0 2024-04-22T23:23:34.724 INFO:tasks.workunit.client.0.smithi067.stdout:0/804: read db/d9a/d56/d6b/db9/fe9 [545706,19014] 0 2024-04-22T23:23:34.726 INFO:tasks.workunit.client.0.smithi067.stdout:5/765: creat d5/db/d12/dbb/d5f/d79/dba/de0/f109 x:0 0 0 2024-04-22T23:23:34.728 INFO:tasks.workunit.client.0.smithi067.stdout:0/805: rename db/de/dfc to db/d71/d10a 0 2024-04-22T23:23:34.730 INFO:tasks.workunit.client.0.smithi067.stdout:8/709: dwrite d1/d3/d27/f6e [4194304,4194304] 0 2024-04-22T23:23:34.735 INFO:tasks.workunit.client.0.smithi067.stdout:0/806: truncate db/d2e/dd9/fe1 498194 0 2024-04-22T23:23:34.741 INFO:tasks.workunit.client.0.smithi067.stdout:0/807: getdents db/d60/d9e/d101 0 2024-04-22T23:23:34.741 INFO:tasks.workunit.client.0.smithi067.stdout:0/808: chown db 1 1 2024-04-22T23:23:34.743 INFO:tasks.workunit.client.0.smithi067.stdout:0/809: mknod db/de/d11/d1a/c10b 0 2024-04-22T23:23:34.744 INFO:tasks.workunit.client.0.smithi067.stdout:0/810: chown db/d21/d29/d2c/f7d 5975445 1 2024-04-22T23:23:34.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:34 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:23:34.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:34 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:23:34.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:34 smithi134 ceph-mon[34609]: pgmap v272: 65 pgs: 65 active+clean; 1.5 GiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 45 MiB/s rd, 161 MiB/s wr, 309 op/s 2024-04-22T23:23:34.749 INFO:tasks.workunit.client.0.smithi067.stdout:8/710: truncate d1/d3/d5c/d98/fa1 948905 0 2024-04-22T23:23:34.751 INFO:tasks.workunit.client.0.smithi067.stdout:8/711: dread - d1/d41/f6a zero size 2024-04-22T23:23:34.753 INFO:tasks.workunit.client.0.smithi067.stdout:8/712: dread - d1/dd/d26/d62/d75/fdd zero size 2024-04-22T23:23:34.753 INFO:tasks.workunit.client.0.smithi067.stdout:5/766: dwrite d5/d28/d36/f9d [0,4194304] 0 2024-04-22T23:23:34.760 INFO:tasks.workunit.client.0.smithi067.stdout:8/713: rename d1/dd/d80/d6f/da8 to d1/dd/d80/db9/de3 0 2024-04-22T23:23:34.760 INFO:tasks.workunit.client.0.smithi067.stdout:0/811: dread db/d21/d29/f3f [0,4194304] 0 2024-04-22T23:23:34.761 INFO:tasks.workunit.client.0.smithi067.stdout:5/767: mknod d5/d28/c10a 0 2024-04-22T23:23:34.762 INFO:tasks.workunit.client.0.smithi067.stdout:0/812: write db/de/d11/d1a/da4/f8e [683462,70643] 0 2024-04-22T23:23:34.763 INFO:tasks.workunit.client.0.smithi067.stdout:8/714: symlink d1/d3/d7e/le4 0 2024-04-22T23:23:34.765 INFO:tasks.workunit.client.0.smithi067.stdout:5/768: chown d5/db/d12/d44/d47/cad 0 1 2024-04-22T23:23:34.767 INFO:tasks.workunit.client.0.smithi067.stdout:8/715: creat d1/dd/d26/d62/ddf/fe5 x:0 0 0 2024-04-22T23:23:34.767 INFO:tasks.workunit.client.0.smithi067.stdout:0/813: truncate db/d21/d29/d75/d9d/fb8 736039 0 2024-04-22T23:23:34.770 INFO:tasks.workunit.client.0.smithi067.stdout:0/814: mkdir db/d9a/d56/d70/d8c/dc2/d10c 0 2024-04-22T23:23:34.770 INFO:tasks.workunit.client.0.smithi067.stdout:5/769: creat d5/d28/d36/d40/f10b x:0 0 0 2024-04-22T23:23:34.772 INFO:tasks.workunit.client.0.smithi067.stdout:5/770: chown d5/db/d35/ldc 5410204 1 2024-04-22T23:23:34.775 INFO:tasks.workunit.client.0.smithi067.stdout:5/771: symlink d5/db/d12/d44/l10c 0 2024-04-22T23:23:34.775 INFO:tasks.workunit.client.0.smithi067.stdout:0/815: dwrite db/fae [4194304,4194304] 0 2024-04-22T23:23:34.776 INFO:tasks.workunit.client.0.smithi067.stdout:8/716: write d1/fde [283803,45246] 0 2024-04-22T23:23:34.779 INFO:tasks.workunit.client.0.smithi067.stdout:5/772: mkdir d5/d1a/d103/d10d 0 2024-04-22T23:23:34.782 INFO:tasks.workunit.client.0.smithi067.stdout:0/816: mknod db/d21/d29/c10d 0 2024-04-22T23:23:34.783 INFO:tasks.workunit.client.0.smithi067.stdout:8/717: fdatasync d1/d3/d5c/d98/fa1 0 2024-04-22T23:23:34.784 INFO:tasks.workunit.client.0.smithi067.stdout:5/773: mknod d5/d1a/d103/d10d/c10e 0 2024-04-22T23:23:34.784 INFO:tasks.workunit.client.0.smithi067.stdout:8/718: write d1/d41/d46/fc2 [568108,109286] 0 2024-04-22T23:23:34.786 INFO:tasks.workunit.client.0.smithi067.stdout:8/719: fsync d1/d41/d46/fa2 0 2024-04-22T23:23:34.787 INFO:tasks.workunit.client.0.smithi067.stdout:5/774: fsync d5/db/f2f 0 2024-04-22T23:23:34.788 INFO:tasks.workunit.client.0.smithi067.stdout:8/720: creat d1/d41/d46/fe6 x:0 0 0 2024-04-22T23:23:34.789 INFO:tasks.workunit.client.0.smithi067.stdout:8/721: fdatasync d1/d41/f83 0 2024-04-22T23:23:34.789 INFO:tasks.workunit.client.0.smithi067.stdout:5/775: mkdir d5/d1a/d10f 0 2024-04-22T23:23:34.792 INFO:tasks.workunit.client.0.smithi067.stdout:5/776: read d5/fa4 [411154,78820] 0 2024-04-22T23:23:34.794 INFO:tasks.workunit.client.0.smithi067.stdout:8/722: creat d1/fe7 x:0 0 0 2024-04-22T23:23:34.798 INFO:tasks.workunit.client.0.smithi067.stdout:8/723: mkdir d1/d41/db4/de8 0 2024-04-22T23:23:34.799 INFO:tasks.workunit.client.0.smithi067.stdout:5/777: dwrite d5/d28/d36/d40/f85 [4194304,4194304] 0 2024-04-22T23:23:34.800 INFO:tasks.workunit.client.0.smithi067.stdout:8/724: read d1/d41/d46/f61 [15392,1360] 0 2024-04-22T23:23:34.805 INFO:tasks.workunit.client.0.smithi067.stdout:0/817: dwrite db/d21/d29/d75/fed [4194304,4194304] 0 2024-04-22T23:23:34.807 INFO:tasks.workunit.client.0.smithi067.stdout:5/778: link d5/d1a/d103/fd2 d5/d28/d36/d40/f110 0 2024-04-22T23:23:34.812 INFO:tasks.workunit.client.0.smithi067.stdout:5/779: creat d5/d28/daf/f111 x:0 0 0 2024-04-22T23:23:34.812 INFO:tasks.workunit.client.0.smithi067.stdout:8/725: dwrite d1/d3/d10/d3b/fac [0,4194304] 0 2024-04-22T23:23:34.813 INFO:tasks.workunit.client.0.smithi067.stdout:5/780: dread - d5/db/d12/dbb/d5f/f107 zero size 2024-04-22T23:23:34.855 INFO:tasks.workunit.client.0.smithi067.stdout:3/755: sync 2024-04-22T23:23:34.855 INFO:tasks.workunit.client.0.smithi067.stdout:0/818: link db/d21/d29/d2d/lff db/d21/d9f/l10e 0 2024-04-22T23:23:34.858 INFO:tasks.workunit.client.0.smithi067.stdout:3/756: chown d3/d4/d28/d40/dfd/dfe/f27 247880846 1 2024-04-22T23:23:34.859 INFO:tasks.workunit.client.0.smithi067.stdout:3/757: fsync d3/d4/d28/d4a/fb9 0 2024-04-22T23:23:34.860 INFO:tasks.workunit.client.0.smithi067.stdout:5/781: creat d5/db/d12/dbb/d5f/d79/f112 x:0 0 0 2024-04-22T23:23:34.862 INFO:tasks.workunit.client.0.smithi067.stdout:0/819: rename db/d2e to db/d21/d29/d75/d9d/dde/d10f 0 2024-04-22T23:23:34.863 INFO:tasks.workunit.client.0.smithi067.stdout:5/782: truncate d5/db/d12/f37 896409 0 2024-04-22T23:23:34.863 INFO:tasks.workunit.client.0.smithi067.stdout:0/820: chown db/de/d11/f3c 104312942 1 2024-04-22T23:23:34.863 INFO:tasks.workunit.client.0.smithi067.stdout:5/783: chown d5/d28/d36 2156565 1 2024-04-22T23:23:34.865 INFO:tasks.workunit.client.0.smithi067.stdout:0/821: unlink db/d71/dda/ff9 0 2024-04-22T23:23:34.866 INFO:tasks.workunit.client.0.smithi067.stdout:5/784: mknod d5/d3e/c113 0 2024-04-22T23:23:34.868 INFO:tasks.workunit.client.0.smithi067.stdout:5/785: creat d5/db/d12/d44/d47/f114 x:0 0 0 2024-04-22T23:23:34.870 INFO:tasks.workunit.client.0.smithi067.stdout:5/786: mknod d5/d1a/d103/d4d/d104/c115 0 2024-04-22T23:23:34.874 INFO:tasks.workunit.client.0.smithi067.stdout:5/787: link d5/d1a/d103/d4d/fa3 d5/db/d12/f116 0 2024-04-22T23:23:34.877 INFO:tasks.workunit.client.0.smithi067.stdout:0/822: rmdir db/d21/d29/d75/d9d/dde/d10f 39 2024-04-22T23:23:34.881 INFO:tasks.workunit.client.0.smithi067.stdout:0/823: truncate db/d21/d29/d75/d9d/fb8 892096 0 2024-04-22T23:23:34.884 INFO:tasks.workunit.client.0.smithi067.stdout:0/824: creat db/d9a/d56/d70/d8c/dc2/f110 x:0 0 0 2024-04-22T23:23:34.884 INFO:tasks.workunit.client.0.smithi067.stdout:0/825: chown db/d21/d29/f3f 2380711 1 2024-04-22T23:23:34.887 INFO:tasks.workunit.client.0.smithi067.stdout:0/826: creat db/d9a/d56/d6b/f111 x:0 0 0 2024-04-22T23:23:34.891 INFO:tasks.workunit.client.0.smithi067.stdout:0/827: fsync db/d21/d29/d75/d9d/fb8 0 2024-04-22T23:23:34.891 INFO:tasks.workunit.client.0.smithi067.stdout:0/828: fsync db/de/d11/f3c 0 2024-04-22T23:23:34.894 INFO:tasks.workunit.client.0.smithi067.stdout:0/829: mkdir db/d60/d112 0 2024-04-22T23:23:34.895 INFO:tasks.workunit.client.0.smithi067.stdout:0/830: read db/d9a/d56/d70/fc3 [294073,46515] 0 2024-04-22T23:23:34.896 INFO:tasks.workunit.client.0.smithi067.stdout:0/831: write db/de/f1b [8951030,61932] 0 2024-04-22T23:23:34.898 INFO:tasks.workunit.client.0.smithi067.stdout:0/832: symlink db/de/d11/d92/l113 0 2024-04-22T23:23:34.903 INFO:tasks.workunit.client.0.smithi067.stdout:0/833: dwrite db/f79 [0,4194304] 0 2024-04-22T23:23:34.909 INFO:tasks.workunit.client.0.smithi067.stdout:0/834: truncate db/fae 2265326 0 2024-04-22T23:23:34.911 INFO:tasks.workunit.client.0.smithi067.stdout:0/835: unlink db/d21/d29/d75/f83 0 2024-04-22T23:23:34.917 INFO:tasks.workunit.client.0.smithi067.stdout:0/836: dwrite db/fd [0,4194304] 0 2024-04-22T23:23:34.922 INFO:tasks.workunit.client.0.smithi067.stdout:0/837: link db/de/d11/d92/fee db/d9a/d56/d6b/f114 0 2024-04-22T23:23:34.927 INFO:tasks.workunit.client.0.smithi067.stdout:0/838: dwrite db/d60/d9e/fe7 [0,4194304] 0 2024-04-22T23:23:34.935 INFO:tasks.workunit.client.0.smithi067.stdout:0/839: dwrite db/d9a/d56/d70/dba/f102 [0,4194304] 0 2024-04-22T23:23:34.941 INFO:tasks.workunit.client.0.smithi067.stdout:0/840: getdents db/d21/d29 0 2024-04-22T23:23:34.947 INFO:tasks.workunit.client.0.smithi067.stdout:0/841: dwrite db/faf [0,4194304] 0 2024-04-22T23:23:34.948 INFO:tasks.workunit.client.0.smithi067.stdout:0/842: chown db/d21/d29/l9b 6161 1 2024-04-22T23:23:34.955 INFO:tasks.workunit.client.0.smithi067.stdout:0/843: dwrite db/d21/d29/d2c/fd6 [0,4194304] 0 2024-04-22T23:23:34.956 INFO:tasks.workunit.client.0.smithi067.stdout:0/844: dread - db/de/ffe zero size 2024-04-22T23:23:34.956 INFO:tasks.workunit.client.0.smithi067.stdout:0/845: chown db/la5 2742669 1 2024-04-22T23:23:34.959 INFO:tasks.workunit.client.0.smithi067.stdout:0/846: creat db/d9a/d56/d70/dbf/de6/f115 x:0 0 0 2024-04-22T23:23:34.962 INFO:tasks.workunit.client.0.smithi067.stdout:0/847: creat db/de/d2a/f116 x:0 0 0 2024-04-22T23:23:34.967 INFO:tasks.workunit.client.0.smithi067.stdout:0/848: dwrite db/d21/f7b [0,4194304] 0 2024-04-22T23:23:35.024 INFO:tasks.workunit.client.0.smithi067.stdout:9/696: sync 2024-04-22T23:23:35.027 INFO:tasks.workunit.client.0.smithi067.stdout:9/697: fsync d5/dd/d18/d39/d3e/f53 0 2024-04-22T23:23:35.028 INFO:tasks.workunit.client.0.smithi067.stdout:9/698: read d5/fa [2863089,57884] 0 2024-04-22T23:23:35.030 INFO:tasks.workunit.client.0.smithi067.stdout:9/699: chown d5/dd/d18/d39/d4d/cb7 9 1 2024-04-22T23:23:35.032 INFO:tasks.workunit.client.0.smithi067.stdout:9/700: unlink d5/d21/d47/d56/c86 0 2024-04-22T23:23:35.033 INFO:tasks.workunit.client.0.smithi067.stdout:9/701: creat d5/dd/d18/ff1 x:0 0 0 2024-04-22T23:23:35.042 INFO:tasks.workunit.client.0.smithi067.stdout:9/702: dwrite d5/dd/d18/d3c/d78/f92 [0,4194304] 0 2024-04-22T23:23:35.043 INFO:tasks.workunit.client.0.smithi067.stdout:9/703: chown d5/dd/d18/d39/d4d/fed 58122 1 2024-04-22T23:23:35.044 INFO:tasks.workunit.client.0.smithi067.stdout:9/704: write d5/d21/d47/fbd [966357,76097] 0 2024-04-22T23:23:35.115 INFO:tasks.workunit.client.0.smithi067.stdout:9/705: mknod d5/dd/d18/d39/d3e/d3f/d63/cf2 0 2024-04-22T23:23:35.116 INFO:tasks.workunit.client.0.smithi067.stdout:9/706: mknod d5/d21/d47/d57/cf3 0 2024-04-22T23:23:35.117 INFO:tasks.workunit.client.0.smithi067.stdout:9/707: dread - d5/d21/d47/d56/d8c/dd5/fe2 zero size 2024-04-22T23:23:35.119 INFO:tasks.workunit.client.0.smithi067.stdout:9/708: rmdir d5/d21/d47/d56 39 2024-04-22T23:23:35.121 INFO:tasks.workunit.client.0.smithi067.stdout:9/709: dread d5/d21/d2c/f7a [0,4194304] 0 2024-04-22T23:23:35.124 INFO:tasks.workunit.client.0.smithi067.stdout:9/710: getdents d5/dd/d18 0 2024-04-22T23:23:35.134 INFO:tasks.workunit.client.0.smithi067.stdout:7/671: sync 2024-04-22T23:23:35.138 INFO:tasks.workunit.client.0.smithi067.stdout:9/711: dread d5/f10 [0,4194304] 0 2024-04-22T23:23:35.140 INFO:tasks.workunit.client.0.smithi067.stdout:9/712: symlink d5/dd/d18/d39/lf4 0 2024-04-22T23:23:35.142 INFO:tasks.workunit.client.0.smithi067.stdout:9/713: creat d5/dd/d18/d5c/ff5 x:0 0 0 2024-04-22T23:23:35.144 INFO:tasks.workunit.client.0.smithi067.stdout:9/714: rmdir d5/d21/d47/d57/dc0 39 2024-04-22T23:23:35.149 INFO:tasks.workunit.client.0.smithi067.stdout:9/715: dwrite d5/dd/d18/d39/d4d/f9e [0,4194304] 0 2024-04-22T23:23:35.151 INFO:tasks.workunit.client.0.smithi067.stdout:9/716: fdatasync d5/d21/d47/d7f/f8e 0 2024-04-22T23:23:35.154 INFO:tasks.workunit.client.0.smithi067.stdout:9/717: fsync d5/dd/d18/d39/d3e/f53 0 2024-04-22T23:23:35.156 INFO:tasks.workunit.client.0.smithi067.stdout:9/718: read d5/dd/d18/d3c/d40/f62 [468005,111736] 0 2024-04-22T23:23:35.171 INFO:tasks.workunit.client.0.smithi067.stdout:9/719: dread d5/dd/d18/d39/d49/d54/f9a [0,4194304] 0 2024-04-22T23:23:35.176 INFO:tasks.workunit.client.0.smithi067.stdout:9/720: dwrite d5/d21/f27 [4194304,4194304] 0 2024-04-22T23:23:35.177 INFO:tasks.workunit.client.0.smithi067.stdout:9/721: write d5/dd/d18/d3c/d40/f6b [65271,44385] 0 2024-04-22T23:23:35.187 INFO:tasks.workunit.client.0.smithi067.stdout:9/722: dwrite d5/dd/d18/d5c/f87 [0,4194304] 0 2024-04-22T23:23:35.191 INFO:tasks.workunit.client.0.smithi067.stdout:9/723: mkdir d5/df6 0 2024-04-22T23:23:35.191 INFO:tasks.workunit.client.0.smithi067.stdout:9/724: getdents d5/df6 0 2024-04-22T23:23:35.194 INFO:tasks.workunit.client.0.smithi067.stdout:9/725: dread d5/d21/d47/fbd [0,4194304] 0 2024-04-22T23:23:35.203 INFO:tasks.workunit.client.0.smithi067.stdout:9/726: dread d5/d21/d47/d57/fa2 [0,4194304] 0 2024-04-22T23:23:35.240 INFO:tasks.workunit.client.0.smithi067.stdout:6/735: sync 2024-04-22T23:23:35.241 INFO:tasks.workunit.client.0.smithi067.stdout:6/736: creat d17/d39/d53/d90/d46/db0/db9/ff3 x:0 0 0 2024-04-22T23:23:35.243 INFO:tasks.workunit.client.0.smithi067.stdout:6/737: truncate d17/d39/d3a/d8d/f97 173081 0 2024-04-22T23:23:35.244 INFO:tasks.workunit.client.0.smithi067.stdout:6/738: symlink d17/d39/d3a/lf4 0 2024-04-22T23:23:35.246 INFO:tasks.workunit.client.0.smithi067.stdout:6/739: stat d17/d39/lbd 0 2024-04-22T23:23:35.248 INFO:tasks.workunit.client.0.smithi067.stdout:6/740: link d17/d39/f49 d17/d39/d3a/da9/ff5 0 2024-04-22T23:23:35.249 INFO:tasks.workunit.client.0.smithi067.stdout:6/741: stat d17/d39/d53/d90/d46 0 2024-04-22T23:23:35.250 INFO:tasks.workunit.client.0.smithi067.stdout:6/742: dread - d17/d39/d3a/f5a zero size 2024-04-22T23:23:35.254 INFO:tasks.workunit.client.0.smithi067.stdout:6/743: rename d17/f21 to d17/ff6 0 2024-04-22T23:23:35.256 INFO:tasks.workunit.client.0.smithi067.stdout:6/744: mkdir d17/d39/d3a/da9/df7 0 2024-04-22T23:23:35.260 INFO:tasks.workunit.client.0.smithi067.stdout:6/745: rmdir d17/d39 39 2024-04-22T23:23:35.262 INFO:tasks.workunit.client.0.smithi067.stdout:6/746: truncate d17/f30 856604 0 2024-04-22T23:23:35.262 INFO:tasks.workunit.client.0.smithi067.stdout:6/747: readlink d17/l1d 0 2024-04-22T23:23:35.264 INFO:tasks.workunit.client.0.smithi067.stdout:6/748: symlink d17/d39/d53/d90/d46/db0/db9/lf8 0 2024-04-22T23:23:35.265 INFO:tasks.workunit.client.0.smithi067.stdout:6/749: mknod d17/d39/d3a/d8d/cf9 0 2024-04-22T23:23:35.268 INFO:tasks.workunit.client.0.smithi067.stdout:6/750: mknod d17/d83/db5/dbc/dd8/cfa 0 2024-04-22T23:23:35.271 INFO:tasks.workunit.client.0.smithi067.stdout:6/751: link d17/d83/d9b/cbf d17/d83/d9b/ddd/cfb 0 2024-04-22T23:23:35.272 INFO:tasks.workunit.client.0.smithi067.stdout:6/752: chown d17/d39/d3a/f99 2173368 1 2024-04-22T23:23:35.288 INFO:tasks.workunit.client.0.smithi067.stdout:6/753: dread d17/d39/d53/d90/d46/f78 [0,4194304] 0 2024-04-22T23:23:35.294 INFO:tasks.workunit.client.0.smithi067.stdout:6/754: getdents d17/d39/d53/d90/d46/db0/de5 0 2024-04-22T23:23:35.299 INFO:tasks.workunit.client.0.smithi067.stdout:6/755: write d17/d83/f98 [1043353,75707] 0 2024-04-22T23:23:35.299 INFO:tasks.workunit.client.0.smithi067.stdout:6/756: dread - d17/d39/d53/d90/d46/d62/d88/fdb zero size 2024-04-22T23:23:35.299 INFO:tasks.workunit.client.0.smithi067.stdout:6/757: rmdir d17/d39/d53/d90/d46 39 2024-04-22T23:23:35.300 INFO:tasks.workunit.client.0.smithi067.stdout:6/758: chown d17/d39/d3a/da9/lc0 117 1 2024-04-22T23:23:35.302 INFO:tasks.workunit.client.0.smithi067.stdout:3/758: sync 2024-04-22T23:23:35.303 INFO:tasks.workunit.client.0.smithi067.stdout:8/726: sync 2024-04-22T23:23:35.303 INFO:tasks.workunit.client.0.smithi067.stdout:1/720: sync 2024-04-22T23:23:35.303 INFO:tasks.workunit.client.0.smithi067.stdout:4/883: sync 2024-04-22T23:23:35.303 INFO:tasks.workunit.client.0.smithi067.stdout:5/788: sync 2024-04-22T23:23:35.303 INFO:tasks.workunit.client.0.smithi067.stdout:2/712: sync 2024-04-22T23:23:35.303 INFO:tasks.workunit.client.0.smithi067.stdout:0/849: sync 2024-04-22T23:23:35.303 INFO:tasks.workunit.client.0.smithi067.stdout:7/672: sync 2024-04-22T23:23:35.305 INFO:tasks.workunit.client.0.smithi067.stdout:5/789: fdatasync d5/f31 0 2024-04-22T23:23:35.305 INFO:tasks.workunit.client.0.smithi067.stdout:2/713: write d1/d10/d72/fd7 [4428774,96679] 0 2024-04-22T23:23:35.308 INFO:tasks.workunit.client.0.smithi067.stdout:3/759: chown d3/d4/d10/d38/d53/d55/d74/dd5/ddb/c56 9639795 1 2024-04-22T23:23:35.308 INFO:tasks.workunit.client.0.smithi067.stdout:2/714: dread - d1/d10/d41/fe1 zero size 2024-04-22T23:23:35.309 INFO:tasks.workunit.client.0.smithi067.stdout:0/850: mkdir db/d9a/d56/d70/d117 0 2024-04-22T23:23:35.310 INFO:tasks.workunit.client.0.smithi067.stdout:8/727: dread d1/d3/d27/f81 [0,4194304] 0 2024-04-22T23:23:35.312 INFO:tasks.workunit.client.0.smithi067.stdout:4/884: creat d12/d42/d4b/d77/d8c/f125 x:0 0 0 2024-04-22T23:23:35.313 INFO:tasks.workunit.client.0.smithi067.stdout:7/673: rmdir d10/d14/d1d/d26/d2c/d7a 39 2024-04-22T23:23:35.316 INFO:tasks.workunit.client.0.smithi067.stdout:3/760: rmdir d3/d4/d28/d40/dfd/dfe/d4b 39 2024-04-22T23:23:35.319 INFO:tasks.workunit.client.0.smithi067.stdout:8/728: truncate d1/dd/d26/d62/d94/fc6 581871 0 2024-04-22T23:23:35.320 INFO:tasks.workunit.client.0.smithi067.stdout:2/715: dwrite d1/d42/d61/fb8 [0,4194304] 0 2024-04-22T23:23:35.321 INFO:tasks.workunit.client.0.smithi067.stdout:7/674: creat d10/d14/de0/ff7 x:0 0 0 2024-04-22T23:23:35.323 INFO:tasks.workunit.client.0.smithi067.stdout:0/851: dwrite db/de/d2a/f99 [0,4194304] 0 2024-04-22T23:23:35.323 INFO:tasks.workunit.client.0.smithi067.stdout:5/790: dread d5/db/d12/f2c [0,4194304] 0 2024-04-22T23:23:35.327 INFO:tasks.workunit.client.0.smithi067.stdout:2/716: stat d1/d10/d72/d53/f97 0 2024-04-22T23:23:35.329 INFO:tasks.workunit.client.0.smithi067.stdout:8/729: creat d1/dd/d80/db9/fe9 x:0 0 0 2024-04-22T23:23:35.331 INFO:tasks.workunit.client.0.smithi067.stdout:3/761: read d3/d4/d28/d40/dfd/dfe/d4b/f4e [626765,121881] 0 2024-04-22T23:23:35.333 INFO:tasks.workunit.client.0.smithi067.stdout:0/852: rmdir db/de/d11 39 2024-04-22T23:23:35.333 INFO:tasks.workunit.client.1.smithi134.stdout:5/0: sync 2024-04-22T23:23:35.333 INFO:tasks.workunit.client.1.smithi134.stdout:5/1: truncate - no filename 2024-04-22T23:23:35.334 INFO:tasks.workunit.client.0.smithi067.stdout:3/762: readlink d3/d4/d10/d19/l108 0 2024-04-22T23:23:35.334 INFO:tasks.workunit.client.0.smithi067.stdout:7/675: mknod d10/d14/d1d/d46/d6a/d91/cf8 0 2024-04-22T23:23:35.336 INFO:tasks.workunit.client.1.smithi134.stdout:5/2: creat f0 x:0 0 0 2024-04-22T23:23:35.336 INFO:tasks.workunit.client.1.smithi134.stdout:5/3: rmdir - no directory 2024-04-22T23:23:35.336 INFO:tasks.workunit.client.0.smithi067.stdout:5/791: mknod d5/db/d12/dbb/d5f/d79/dba/de0/c117 0 2024-04-22T23:23:35.336 INFO:tasks.workunit.client.1.smithi134.stdout:5/4: dread - f0 zero size 2024-04-22T23:23:35.338 INFO:tasks.workunit.client.1.smithi134.stdout:5/5: write f0 [837362,33958] 0 2024-04-22T23:23:35.340 INFO:tasks.workunit.client.0.smithi067.stdout:7/676: write d10/d18/d35/d62/fe8 [64695,49897] 0 2024-04-22T23:23:35.341 INFO:tasks.workunit.client.0.smithi067.stdout:8/730: dwrite d1/d41/d46/fce [0,4194304] 0 2024-04-22T23:23:35.341 INFO:tasks.workunit.client.0.smithi067.stdout:9/727: sync 2024-04-22T23:23:35.346 INFO:tasks.workunit.client.0.smithi067.stdout:8/731: dread - d1/dd/d26/d62/d94/fd5 zero size 2024-04-22T23:23:35.346 INFO:tasks.workunit.client.0.smithi067.stdout:9/728: chown d5/d21/d47/f72 659431291 1 2024-04-22T23:23:35.350 INFO:tasks.workunit.client.0.smithi067.stdout:7/677: symlink d10/d14/d1d/d46/d6a/d91/lf9 0 2024-04-22T23:23:35.350 INFO:tasks.workunit.client.0.smithi067.stdout:0/853: write db/de/d11/d1a/da4/dec/ffa [403626,69025] 0 2024-04-22T23:23:35.352 INFO:tasks.workunit.client.0.smithi067.stdout:6/759: write d17/f2b [1218950,36076] 0 2024-04-22T23:23:35.353 INFO:tasks.workunit.client.0.smithi067.stdout:9/729: dread d5/d21/d47/d57/fa2 [0,4194304] 0 2024-04-22T23:23:35.354 INFO:tasks.workunit.client.0.smithi067.stdout:3/763: rename d3/d4/d10/d38/d53/d55/d74/dd5/ddb/c100 to d3/d4/d10/d38/d53/d55/d74/d7f/c110 0 2024-04-22T23:23:35.354 INFO:tasks.workunit.client.0.smithi067.stdout:5/792: dwrite d5/d1a/d103/d4d/f4e [0,4194304] 0 2024-04-22T23:23:35.355 INFO:tasks.workunit.client.0.smithi067.stdout:4/885: dread d12/d5f/f5a [0,4194304] 0 2024-04-22T23:23:35.355 INFO:tasks.workunit.client.1.smithi134.stdout:5/6: truncate f0 535843 0 2024-04-22T23:23:35.356 INFO:tasks.workunit.client.1.smithi134.stdout:5/7: chown f0 3955841 1 2024-04-22T23:23:35.357 INFO:tasks.workunit.client.0.smithi067.stdout:8/732: creat d1/d3/d27/fea x:0 0 0 2024-04-22T23:23:35.357 INFO:tasks.workunit.client.0.smithi067.stdout:6/760: truncate d17/d39/f8c 1329079 0 2024-04-22T23:23:35.357 INFO:tasks.workunit.client.1.smithi134.stdout:5/8: symlink l1 0 2024-04-22T23:23:35.358 INFO:tasks.workunit.client.0.smithi067.stdout:5/793: dread - d5/d28/daf/dbf/dcf/d9a/fdb zero size 2024-04-22T23:23:35.358 INFO:tasks.workunit.client.1.smithi134.stdout:5/9: creat f2 x:0 0 0 2024-04-22T23:23:35.363 INFO:tasks.workunit.client.0.smithi067.stdout:3/764: creat d3/d4/d10/d10b/f111 x:0 0 0 2024-04-22T23:23:35.363 INFO:tasks.workunit.client.0.smithi067.stdout:9/730: truncate d5/d21/d47/d56/d8c/dd5/fe2 78041 0 2024-04-22T23:23:35.364 INFO:tasks.workunit.client.0.smithi067.stdout:9/731: readlink d5/dd/d18/l2a 0 2024-04-22T23:23:35.365 INFO:tasks.workunit.client.1.smithi134.stdout:5/10: rename f0 to f3 0 2024-04-22T23:23:35.365 INFO:tasks.workunit.client.0.smithi067.stdout:4/886: creat d12/d17/d88/f126 x:0 0 0 2024-04-22T23:23:35.366 INFO:tasks.workunit.client.1.smithi134.stdout:5/11: dread - f2 zero size 2024-04-22T23:23:35.367 INFO:tasks.workunit.client.0.smithi067.stdout:1/721: write d8/d9/f1b [3472738,110623] 0 2024-04-22T23:23:35.368 INFO:tasks.workunit.client.0.smithi067.stdout:8/733: creat d1/d41/d46/feb x:0 0 0 2024-04-22T23:23:35.371 INFO:tasks.workunit.client.0.smithi067.stdout:5/794: rename d5/d1a/d10f to d5/db/d35/d118 0 2024-04-22T23:23:35.372 INFO:tasks.workunit.client.0.smithi067.stdout:2/717: write d1/d42/f57 [212353,36362] 0 2024-04-22T23:23:35.376 INFO:tasks.workunit.client.0.smithi067.stdout:9/732: chown d5/dd/d18/d39/d3e/f82 2243 1 2024-04-22T23:23:35.376 INFO:tasks.workunit.client.0.smithi067.stdout:4/887: unlink d12/d42/d4b/d122/d8e/d56/d114/fd7 0 2024-04-22T23:23:35.377 INFO:tasks.workunit.client.1.smithi134.stdout:5/12: link f3 f4 0 2024-04-22T23:23:35.379 INFO:tasks.workunit.client.0.smithi067.stdout:5/795: dwrite d5/db/d12/dbb/d61/f82 [0,4194304] 0 2024-04-22T23:23:35.383 INFO:tasks.workunit.client.1.smithi134.stdout:5/13: dread f4 [0,4194304] 0 2024-04-22T23:23:35.383 INFO:tasks.workunit.client.0.smithi067.stdout:5/796: write d5/db/d12/d44/d47/f114 [373050,21591] 0 2024-04-22T23:23:35.385 INFO:tasks.workunit.client.0.smithi067.stdout:8/734: mknod d1/d41/d46/cec 0 2024-04-22T23:23:35.386 INFO:tasks.workunit.client.0.smithi067.stdout:0/854: write db/fb7 [910200,21790] 0 2024-04-22T23:23:35.388 INFO:tasks.workunit.client.0.smithi067.stdout:3/765: dread d3/d4/d10/d1f/f3e [0,4194304] 0 2024-04-22T23:23:35.388 INFO:tasks.workunit.client.0.smithi067.stdout:1/722: dwrite d8/d9/de0/d60/d62/fa8 [0,4194304] 0 2024-04-22T23:23:35.389 INFO:tasks.workunit.client.1.smithi134.stdout:5/14: write f4 [1481008,27149] 0 2024-04-22T23:23:35.391 INFO:tasks.workunit.client.0.smithi067.stdout:6/761: write d17/f26 [3126669,60827] 0 2024-04-22T23:23:35.393 INFO:tasks.workunit.client.0.smithi067.stdout:1/723: chown d8/d9/d2d/d4d/d48 1 1 2024-04-22T23:23:35.397 INFO:tasks.workunit.client.0.smithi067.stdout:9/733: rename d5/d21/d47/d7f/def to d5/dd/da9/df7 0 2024-04-22T23:23:35.398 INFO:tasks.workunit.client.0.smithi067.stdout:7/678: link d10/d14/l32 d10/d14/d1d/d26/d80/dd2/dd4/lfa 0 2024-04-22T23:23:35.399 INFO:tasks.workunit.client.0.smithi067.stdout:5/797: creat d5/d28/dae/f119 x:0 0 0 2024-04-22T23:23:35.399 INFO:tasks.workunit.client.0.smithi067.stdout:9/734: stat d5/dd/d18/d5c 0 2024-04-22T23:23:35.400 INFO:tasks.workunit.client.0.smithi067.stdout:8/735: mkdir d1/dd/d26/d62/ddf/ded 0 2024-04-22T23:23:35.400 INFO:tasks.workunit.client.0.smithi067.stdout:3/766: dread - d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d90/da2/daf/dd8/ffc zero size 2024-04-22T23:23:35.401 INFO:tasks.workunit.client.1.smithi134.stdout:5/15: dread f3 [0,4194304] 0 2024-04-22T23:23:35.403 INFO:tasks.workunit.client.0.smithi067.stdout:0/855: dwrite db/d9a/d56/fca [0,4194304] 0 2024-04-22T23:23:35.404 INFO:tasks.workunit.client.1.smithi134.stdout:5/16: dread f4 [0,4194304] 0 2024-04-22T23:23:35.404 INFO:tasks.workunit.client.1.smithi134.stdout:2/7: sync 2024-04-22T23:23:35.406 INFO:tasks.workunit.client.1.smithi134.stdout:5/17: dread f3 [0,4194304] 0 2024-04-22T23:23:35.407 INFO:tasks.workunit.client.1.smithi134.stdout:2/8: symlink d2/l3 0 2024-04-22T23:23:35.407 INFO:tasks.workunit.client.0.smithi067.stdout:2/718: truncate d1/d10/d1c/f3a 4093986 0 2024-04-22T23:23:35.407 INFO:tasks.workunit.client.1.smithi134.stdout:5/18: chown f4 132915857 1 2024-04-22T23:23:35.407 INFO:tasks.workunit.client.1.smithi134.stdout:2/9: readlink d2/l3 0 2024-04-22T23:23:35.409 INFO:tasks.workunit.client.1.smithi134.stdout:5/19: unlink f3 0 2024-04-22T23:23:35.410 INFO:tasks.workunit.client.0.smithi067.stdout:4/888: dread d12/d5f/d7e/fc9 [0,4194304] 0 2024-04-22T23:23:35.411 INFO:tasks.workunit.client.1.smithi134.stdout:5/20: fdatasync f4 0 2024-04-22T23:23:35.415 INFO:tasks.workunit.client.0.smithi067.stdout:7/679: mknod d10/d14/d1d/d26/d2c/d41/db0/cfb 0 2024-04-22T23:23:35.415 INFO:tasks.workunit.client.1.smithi134.stdout:5/21: link l1 l5 0 2024-04-22T23:23:35.417 INFO:tasks.workunit.client.1.smithi134.stdout:5/22: creat f6 x:0 0 0 2024-04-22T23:23:35.418 INFO:tasks.workunit.client.0.smithi067.stdout:1/724: mknod d8/d9/de0/d5d/cef 0 2024-04-22T23:23:35.418 INFO:tasks.workunit.client.0.smithi067.stdout:3/767: read d3/d4/d10/d38/d53/f54 [647812,46543] 0 2024-04-22T23:23:35.419 INFO:tasks.workunit.client.0.smithi067.stdout:0/856: symlink db/d9a/d56/d70/l118 0 2024-04-22T23:23:35.421 INFO:tasks.workunit.client.0.smithi067.stdout:0/857: write db/d9a/f63 [933084,7559] 0 2024-04-22T23:23:35.422 INFO:tasks.workunit.client.0.smithi067.stdout:9/735: rename d5/dd/d18/d39/d49/d54/dd1 to d5/d21/d47/d56/d8c/df8 0 2024-04-22T23:23:35.423 INFO:tasks.workunit.client.0.smithi067.stdout:2/719: rmdir d1/d19 39 2024-04-22T23:23:35.423 INFO:tasks.workunit.client.1.smithi134.stdout:2/10: dwrite f0 [0,4194304] 0 2024-04-22T23:23:35.426 INFO:tasks.workunit.client.0.smithi067.stdout:4/889: rmdir d12/d17/d3f 39 2024-04-22T23:23:35.428 INFO:tasks.workunit.client.0.smithi067.stdout:4/890: stat d12/d42/d4b/d77/d8c 0 2024-04-22T23:23:35.428 INFO:tasks.workunit.client.0.smithi067.stdout:5/798: mknod d5/d28/c11a 0 2024-04-22T23:23:35.428 INFO:tasks.workunit.client.0.smithi067.stdout:3/768: dwrite d3/d4/d10/d19/f3b [0,4194304] 0 2024-04-22T23:23:35.429 INFO:tasks.workunit.client.0.smithi067.stdout:7/680: mknod d10/d14/d1d/d46/d6a/dcc/cfc 0 2024-04-22T23:23:35.432 INFO:tasks.workunit.client.0.smithi067.stdout:5/799: chown d5/f31 0 1 2024-04-22T23:23:35.432 INFO:tasks.workunit.client.1.smithi134.stdout:5/23: symlink l7 0 2024-04-22T23:23:35.447 INFO:tasks.workunit.client.1.smithi134.stdout:5/24: rename f4 to f8 0 2024-04-22T23:23:35.449 INFO:tasks.workunit.client.1.smithi134.stdout:5/25: unlink l5 0 2024-04-22T23:23:35.449 INFO:tasks.workunit.client.0.smithi067.stdout:0/858: readlink db/d21/d29/l44 0 2024-04-22T23:23:35.449 INFO:tasks.workunit.client.0.smithi067.stdout:2/720: creat d1/d10/d72/fe8 x:0 0 0 2024-04-22T23:23:35.449 INFO:tasks.workunit.client.0.smithi067.stdout:1/725: chown d8/d9/d2d/d4d/d48/d9a/cb3 36141 1 2024-04-22T23:23:35.449 INFO:tasks.workunit.client.0.smithi067.stdout:4/891: mkdir d12/d42/d4b/d77/d127 0 2024-04-22T23:23:35.450 INFO:tasks.workunit.client.0.smithi067.stdout:6/762: dwrite d17/d39/d53/d90/d46/d62/d88/f92 [0,4194304] 0 2024-04-22T23:23:35.450 INFO:tasks.workunit.client.0.smithi067.stdout:8/736: link d1/d3/d7e/l87 d1/d41/db4/de8/lee 0 2024-04-22T23:23:35.450 INFO:tasks.workunit.client.0.smithi067.stdout:3/769: readlink d3/d4/d28/ld9 0 2024-04-22T23:23:35.452 INFO:tasks.workunit.client.1.smithi134.stdout:2/11: fsync f0 0 2024-04-22T23:23:35.452 INFO:tasks.workunit.client.0.smithi067.stdout:7/681: mknod d10/de3/cfd 0 2024-04-22T23:23:35.453 INFO:tasks.workunit.client.1.smithi134.stdout:2/12: mkdir d2/d4 0 2024-04-22T23:23:35.454 INFO:tasks.workunit.client.1.smithi134.stdout:2/13: chown d2/d4 6380409 1 2024-04-22T23:23:35.454 INFO:tasks.workunit.client.0.smithi067.stdout:5/800: mknod d5/d28/daf/dbf/dcf/d9a/c11b 0 2024-04-22T23:23:35.455 INFO:tasks.workunit.client.0.smithi067.stdout:0/859: unlink db/d21/f7b 0 2024-04-22T23:23:35.456 INFO:tasks.workunit.client.1.smithi134.stdout:5/26: getdents . 0 2024-04-22T23:23:35.456 INFO:tasks.workunit.client.0.smithi067.stdout:8/737: dread d1/dd/d80/f8d [0,4194304] 0 2024-04-22T23:23:35.457 INFO:tasks.workunit.client.1.smithi134.stdout:5/27: chown f2 1 1 2024-04-22T23:23:35.457 INFO:tasks.workunit.client.0.smithi067.stdout:2/721: mknod d1/d10/d33/ce9 0 2024-04-22T23:23:35.457 INFO:tasks.workunit.client.0.smithi067.stdout:0/860: write db/d9a/d56/f6d [1682035,78160] 0 2024-04-22T23:23:35.458 INFO:tasks.workunit.client.1.smithi134.stdout:5/28: write f6 [171278,125866] 0 2024-04-22T23:23:35.459 INFO:tasks.workunit.client.0.smithi067.stdout:0/861: read - db/de/d2a/ffd zero size 2024-04-22T23:23:35.460 INFO:tasks.workunit.client.1.smithi134.stdout:5/29: mkdir d9 0 2024-04-22T23:23:35.461 INFO:tasks.workunit.client.0.smithi067.stdout:2/722: write d1/d10/d33/d46/da4/fcc [1042385,17632] 0 2024-04-22T23:23:35.461 INFO:tasks.workunit.client.0.smithi067.stdout:1/726: unlink d8/d9/d2f/d33/d5c/dd9/le3 0 2024-04-22T23:23:35.461 INFO:tasks.workunit.client.1.smithi134.stdout:2/14: dwrite f0 [0,4194304] 0 2024-04-22T23:23:35.462 INFO:tasks.workunit.client.1.smithi134.stdout:5/30: fsync f2 0 2024-04-22T23:23:35.463 INFO:tasks.workunit.client.0.smithi067.stdout:6/763: write d17/d39/d53/d90/d46/d62/fd0 [1948437,39983] 0 2024-04-22T23:23:35.464 INFO:tasks.workunit.client.0.smithi067.stdout:7/682: creat d10/d14/d1d/d46/d6a/dcc/ffe x:0 0 0 2024-04-22T23:23:35.464 INFO:tasks.workunit.client.0.smithi067.stdout:5/801: rmdir d5/d28/daf/dbf 39 2024-04-22T23:23:35.466 INFO:tasks.workunit.client.0.smithi067.stdout:4/892: getdents d12/d17/d47/de7 0 2024-04-22T23:23:35.466 INFO:tasks.workunit.client.0.smithi067.stdout:6/764: chown d17/d39/d3a/d8d/cf9 0 1 2024-04-22T23:23:35.466 INFO:tasks.workunit.client.0.smithi067.stdout:8/738: truncate d1/d41/f67 99555 0 2024-04-22T23:23:35.467 INFO:tasks.workunit.client.1.smithi134.stdout:2/15: rmdir d2/d4 0 2024-04-22T23:23:35.469 INFO:tasks.workunit.client.1.smithi134.stdout:5/31: write f8 [2279073,95375] 0 2024-04-22T23:23:35.471 INFO:tasks.workunit.client.0.smithi067.stdout:2/723: rename d1/d12/f15 to d1/d10/d33/d46/fea 0 2024-04-22T23:23:35.471 INFO:tasks.workunit.client.1.smithi134.stdout:2/16: dread f0 [0,4194304] 0 2024-04-22T23:23:35.471 INFO:tasks.workunit.client.1.smithi134.stdout:2/17: chown d2 19310725 1 2024-04-22T23:23:35.472 INFO:tasks.workunit.client.1.smithi134.stdout:5/32: creat d9/fa x:0 0 0 2024-04-22T23:23:35.473 INFO:tasks.workunit.client.1.smithi134.stdout:5/33: dread - f2 zero size 2024-04-22T23:23:35.473 INFO:tasks.workunit.client.1.smithi134.stdout:2/18: write f0 [421395,21189] 0 2024-04-22T23:23:35.474 INFO:tasks.workunit.client.0.smithi067.stdout:5/802: creat d5/d1a/d103/d4d/f11c x:0 0 0 2024-04-22T23:23:35.476 INFO:tasks.workunit.client.1.smithi134.stdout:2/19: mknod d2/c5 0 2024-04-22T23:23:35.477 INFO:tasks.workunit.client.0.smithi067.stdout:4/893: creat d12/d42/d4b/f128 x:0 0 0 2024-04-22T23:23:35.481 INFO:tasks.workunit.client.1.smithi134.stdout:5/34: dwrite f2 [0,4194304] 0 2024-04-22T23:23:35.482 INFO:tasks.workunit.client.0.smithi067.stdout:7/683: dread d10/d18/d35/f5e [0,4194304] 0 2024-04-22T23:23:35.486 INFO:tasks.workunit.client.0.smithi067.stdout:6/765: dread d17/f20 [0,4194304] 0 2024-04-22T23:23:35.488 INFO:tasks.workunit.client.0.smithi067.stdout:4/894: creat d12/d42/d4b/d77/f129 x:0 0 0 2024-04-22T23:23:35.490 INFO:tasks.workunit.client.1.smithi134.stdout:5/35: truncate f8 428061 0 2024-04-22T23:23:35.491 INFO:tasks.workunit.client.0.smithi067.stdout:7/684: rmdir d10/d14/d6d 39 2024-04-22T23:23:35.493 INFO:tasks.workunit.client.1.smithi134.stdout:5/36: dread f2 [0,4194304] 0 2024-04-22T23:23:35.496 INFO:tasks.workunit.client.1.smithi134.stdout:2/20: fdatasync f0 0 2024-04-22T23:23:35.497 INFO:tasks.workunit.client.1.smithi134.stdout:2/21: write f0 [2591174,68125] 0 2024-04-22T23:23:35.498 INFO:tasks.workunit.client.0.smithi067.stdout:1/727: getdents d8/d9/d2d/d4d/d9c 0 2024-04-22T23:23:35.499 INFO:tasks.workunit.client.1.smithi134.stdout:5/37: dread f8 [0,4194304] 0 2024-04-22T23:23:35.499 INFO:tasks.workunit.client.1.smithi134.stdout:2/22: rmdir d2 39 2024-04-22T23:23:35.500 INFO:tasks.workunit.client.1.smithi134.stdout:5/38: dread f8 [0,4194304] 0 2024-04-22T23:23:35.500 INFO:tasks.workunit.client.0.smithi067.stdout:1/728: write d8/d9/de0/d5d/f8b [1359008,73851] 0 2024-04-22T23:23:35.501 INFO:tasks.workunit.client.0.smithi067.stdout:2/724: rmdir d1/d10/d33/d46/de6 0 2024-04-22T23:23:35.502 INFO:tasks.workunit.client.1.smithi134.stdout:5/39: read f6 [48135,104556] 0 2024-04-22T23:23:35.503 INFO:tasks.workunit.client.1.smithi134.stdout:2/23: creat d2/f6 x:0 0 0 2024-04-22T23:23:35.504 INFO:tasks.workunit.client.0.smithi067.stdout:6/766: mknod d17/d39/d53/d90/cfc 0 2024-04-22T23:23:35.504 INFO:tasks.workunit.client.1.smithi134.stdout:5/40: creat d9/fb x:0 0 0 2024-04-22T23:23:35.505 INFO:tasks.workunit.client.0.smithi067.stdout:4/895: mkdir d12/d5f/d2b/d12a 0 2024-04-22T23:23:35.506 INFO:tasks.workunit.client.1.smithi134.stdout:2/24: rename d2/c5 to d2/c7 0 2024-04-22T23:23:35.506 INFO:tasks.workunit.client.0.smithi067.stdout:4/896: fsync d12/d17/f100 0 2024-04-22T23:23:35.506 INFO:tasks.workunit.client.0.smithi067.stdout:0/862: truncate db/de/fdc 1445545 0 2024-04-22T23:23:35.507 INFO:tasks.workunit.client.1.smithi134.stdout:5/41: creat d9/fc x:0 0 0 2024-04-22T23:23:35.507 INFO:tasks.workunit.client.1.smithi134.stdout:2/25: symlink d2/l8 0 2024-04-22T23:23:35.507 INFO:tasks.workunit.client.0.smithi067.stdout:1/729: dwrite d8/d12/f94 [0,4194304] 0 2024-04-22T23:23:35.508 INFO:tasks.workunit.client.1.smithi134.stdout:5/42: mkdir d9/dd 0 2024-04-22T23:23:35.508 INFO:tasks.workunit.client.1.smithi134.stdout:2/26: creat d2/f9 x:0 0 0 2024-04-22T23:23:35.509 INFO:tasks.workunit.client.0.smithi067.stdout:6/767: dread d17/d39/d53/d90/d46/db0/db9/de7/fe2 [0,4194304] 0 2024-04-22T23:23:35.510 INFO:tasks.workunit.client.1.smithi134.stdout:2/27: creat d2/fa x:0 0 0 2024-04-22T23:23:35.510 INFO:tasks.workunit.client.0.smithi067.stdout:2/725: mknod d1/d10/d41/da2/ceb 0 2024-04-22T23:23:35.511 INFO:tasks.workunit.client.1.smithi134.stdout:2/28: creat d2/fb x:0 0 0 2024-04-22T23:23:35.511 INFO:tasks.workunit.client.0.smithi067.stdout:8/739: truncate d1/dd/d26/d62/d94/fa3 484121 0 2024-04-22T23:23:35.514 INFO:tasks.workunit.client.0.smithi067.stdout:5/803: getdents d5/db 0 2024-04-22T23:23:35.515 INFO:tasks.workunit.client.1.smithi134.stdout:2/29: link d2/l8 d2/lc 0 2024-04-22T23:23:35.515 INFO:tasks.workunit.client.1.smithi134.stdout:7/74: sync 2024-04-22T23:23:35.515 INFO:tasks.workunit.client.1.smithi134.stdout:4/4: sync 2024-04-22T23:23:35.515 INFO:tasks.workunit.client.1.smithi134.stdout:8/13: sync 2024-04-22T23:23:35.515 INFO:tasks.workunit.client.1.smithi134.stdout:9/20: sync 2024-04-22T23:23:35.515 INFO:tasks.workunit.client.1.smithi134.stdout:6/79: sync 2024-04-22T23:23:35.515 INFO:tasks.workunit.client.1.smithi134.stdout:3/13: sync 2024-04-22T23:23:35.515 INFO:tasks.workunit.client.1.smithi134.stdout:1/28: sync 2024-04-22T23:23:35.515 INFO:tasks.workunit.client.1.smithi134.stdout:0/38: sync 2024-04-22T23:23:35.515 INFO:tasks.workunit.client.1.smithi134.stdout:3/14: stat f3 0 2024-04-22T23:23:35.516 INFO:tasks.workunit.client.0.smithi067.stdout:8/740: truncate d1/f76 4784371 0 2024-04-22T23:23:35.516 INFO:tasks.workunit.client.1.smithi134.stdout:8/14: chown c3 93099223 1 2024-04-22T23:23:35.516 INFO:tasks.workunit.client.0.smithi067.stdout:8/741: read - d1/fe7 zero size 2024-04-22T23:23:35.517 INFO:tasks.workunit.client.0.smithi067.stdout:5/804: write d5/d3e/d90/f94 [916973,113121] 0 2024-04-22T23:23:35.519 INFO:tasks.workunit.client.0.smithi067.stdout:8/742: dread - d1/fe2 zero size 2024-04-22T23:23:35.519 INFO:tasks.workunit.client.1.smithi134.stdout:4/5: rename f0 to f1 0 2024-04-22T23:23:35.520 INFO:tasks.workunit.client.0.smithi067.stdout:6/768: dwrite d17/d39/f3f [4194304,4194304] 0 2024-04-22T23:23:35.521 INFO:tasks.workunit.client.1.smithi134.stdout:2/30: dread f0 [0,4194304] 0 2024-04-22T23:23:35.521 INFO:tasks.workunit.client.1.smithi134.stdout:2/31: readlink l1 0 2024-04-22T23:23:35.522 INFO:tasks.workunit.client.0.smithi067.stdout:4/897: mknod d12/d2e/d9b/c12b 0 2024-04-22T23:23:35.522 INFO:tasks.workunit.client.1.smithi134.stdout:2/32: chown d2/f9 41926 1 2024-04-22T23:23:35.522 INFO:tasks.workunit.client.1.smithi134.stdout:1/29: rename f7 to f9 0 2024-04-22T23:23:35.522 INFO:tasks.workunit.client.0.smithi067.stdout:6/769: stat d17/d39/d53/d90/l4e 0 2024-04-22T23:23:35.523 INFO:tasks.workunit.client.1.smithi134.stdout:0/39: creat d1/d2/fa x:0 0 0 2024-04-22T23:23:35.524 INFO:tasks.workunit.client.0.smithi067.stdout:6/770: chown d17/d83/db5/dbc 85538957 1 2024-04-22T23:23:35.524 INFO:tasks.workunit.client.1.smithi134.stdout:8/15: symlink l5 0 2024-04-22T23:23:35.525 INFO:tasks.workunit.client.0.smithi067.stdout:7/685: getdents d10/d18/d88 0 2024-04-22T23:23:35.525 INFO:tasks.workunit.client.1.smithi134.stdout:3/15: symlink l4 0 2024-04-22T23:23:35.526 INFO:tasks.workunit.client.1.smithi134.stdout:9/21: truncate d0/f4 1030805 0 2024-04-22T23:23:35.528 INFO:tasks.workunit.client.1.smithi134.stdout:5/43: rmdir d9 39 2024-04-22T23:23:35.529 INFO:tasks.workunit.client.0.smithi067.stdout:5/805: rmdir d5/d28/d36/d40/d97 39 2024-04-22T23:23:35.530 INFO:tasks.workunit.client.1.smithi134.stdout:1/30: creat fa x:0 0 0 2024-04-22T23:23:35.531 INFO:tasks.workunit.client.0.smithi067.stdout:4/898: mkdir d12/d17/d88/d12c 0 2024-04-22T23:23:35.531 INFO:tasks.workunit.client.1.smithi134.stdout:2/33: mkdir d2/dd 0 2024-04-22T23:23:35.531 INFO:tasks.workunit.client.1.smithi134.stdout:2/34: stat l1 0 2024-04-22T23:23:35.532 INFO:tasks.workunit.client.0.smithi067.stdout:2/726: creat d1/d42/d5f/d9b/dc0/fec x:0 0 0 2024-04-22T23:23:35.532 INFO:tasks.workunit.client.1.smithi134.stdout:0/40: read d1/d5/f7 [281689,112868] 0 2024-04-22T23:23:35.532 INFO:tasks.workunit.client.0.smithi067.stdout:6/771: rmdir d17/d83/db5/dbc 39 2024-04-22T23:23:35.533 INFO:tasks.workunit.client.1.smithi134.stdout:8/16: creat f6 x:0 0 0 2024-04-22T23:23:35.533 INFO:tasks.workunit.client.1.smithi134.stdout:0/41: write d1/d2/fa [449302,52342] 0 2024-04-22T23:23:35.534 INFO:tasks.workunit.client.1.smithi134.stdout:8/17: dread - f6 zero size 2024-04-22T23:23:35.539 INFO:tasks.workunit.client.1.smithi134.stdout:9/22: symlink d0/l8 0 2024-04-22T23:23:35.539 INFO:tasks.workunit.client.1.smithi134.stdout:8/18: dread - f6 zero size 2024-04-22T23:23:35.539 INFO:tasks.workunit.client.1.smithi134.stdout:1/31: mkdir db 0 2024-04-22T23:23:35.540 INFO:tasks.workunit.client.0.smithi067.stdout:7/686: rename d10/d14/d1d/d26/d2c/d41/db0/cfb to d10/d18/d35/d62/de7/cff 0 2024-04-22T23:23:35.540 INFO:tasks.workunit.client.0.smithi067.stdout:0/863: creat db/d21/f119 x:0 0 0 2024-04-22T23:23:35.540 INFO:tasks.workunit.client.0.smithi067.stdout:8/743: mkdir d1/dd/d26/d62/ddf/ded/def 0 2024-04-22T23:23:35.540 INFO:tasks.workunit.client.0.smithi067.stdout:5/806: mkdir d5/db/d12/dbb/d5f/d79/d11d 0 2024-04-22T23:23:35.541 INFO:tasks.workunit.client.0.smithi067.stdout:2/727: dwrite d1/d10/d72/d53/f74 [0,4194304] 0 2024-04-22T23:23:35.541 INFO:tasks.workunit.client.0.smithi067.stdout:4/899: symlink d12/d5f/d2b/db6/l12d 0 2024-04-22T23:23:35.546 INFO:tasks.workunit.client.1.smithi134.stdout:5/44: dwrite f2 [0,4194304] 0 2024-04-22T23:23:35.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:35 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:23:35.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:35 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:23:35.547 INFO:tasks.workunit.client.0.smithi067.stdout:8/744: rmdir d1/d41 39 2024-04-22T23:23:35.549 INFO:tasks.workunit.client.1.smithi134.stdout:5/45: chown f8 279309909 1 2024-04-22T23:23:35.550 INFO:tasks.workunit.client.0.smithi067.stdout:0/864: dwrite db/d9a/d56/d70/d8c/dc2/f110 [0,4194304] 0 2024-04-22T23:23:35.550 INFO:tasks.workunit.client.0.smithi067.stdout:5/807: mkdir d5/d3e/d90/d11e 0 2024-04-22T23:23:35.553 INFO:tasks.workunit.client.0.smithi067.stdout:4/900: dwrite d12/d5f/d2b/f72 [0,4194304] 0 2024-04-22T23:23:35.553 INFO:tasks.workunit.client.1.smithi134.stdout:2/35: dwrite f0 [0,4194304] 0 2024-04-22T23:23:35.554 INFO:tasks.workunit.client.0.smithi067.stdout:0/865: chown db/d9a/d56/d6b/db9 11 1 2024-04-22T23:23:35.555 INFO:tasks.workunit.client.0.smithi067.stdout:0/866: dread - db/d71/fbc zero size 2024-04-22T23:23:35.558 INFO:tasks.workunit.client.1.smithi134.stdout:6/80: dwrite d0/f1a [0,4194304] 0 2024-04-22T23:23:35.559 INFO:tasks.workunit.client.1.smithi134.stdout:2/36: truncate d2/f9 607570 0 2024-04-22T23:23:35.559 INFO:tasks.workunit.client.1.smithi134.stdout:1/32: dwrite f0 [4194304,4194304] 0 2024-04-22T23:23:35.563 INFO:tasks.workunit.client.1.smithi134.stdout:6/81: chown d0/d19/db/lc 128786 1 2024-04-22T23:23:35.564 INFO:tasks.workunit.client.0.smithi067.stdout:2/728: mkdir d1/d19/ded 0 2024-04-22T23:23:35.564 INFO:tasks.workunit.client.1.smithi134.stdout:5/46: creat d9/fe x:0 0 0 2024-04-22T23:23:35.564 INFO:tasks.workunit.client.1.smithi134.stdout:6/82: stat d0/d19/de/d12 0 2024-04-22T23:23:35.565 INFO:tasks.workunit.client.0.smithi067.stdout:5/808: fdatasync d5/db/ff 0 2024-04-22T23:23:35.567 INFO:tasks.workunit.client.0.smithi067.stdout:8/745: dwrite d1/dd/d80/db9/de3/fbf [0,4194304] 0 2024-04-22T23:23:35.568 INFO:tasks.workunit.client.1.smithi134.stdout:6/83: fdatasync d0/d19/f1c 0 2024-04-22T23:23:35.569 INFO:tasks.workunit.client.1.smithi134.stdout:5/47: chown d9/dd 2606 1 2024-04-22T23:23:35.570 INFO:tasks.workunit.client.1.smithi134.stdout:1/33: creat db/fc x:0 0 0 2024-04-22T23:23:35.570 INFO:tasks.workunit.client.1.smithi134.stdout:1/34: stat l8 0 2024-04-22T23:23:35.572 INFO:tasks.workunit.client.1.smithi134.stdout:2/37: link f0 d2/dd/fe 0 2024-04-22T23:23:35.573 INFO:tasks.workunit.client.1.smithi134.stdout:1/35: write f0 [3661524,111633] 0 2024-04-22T23:23:35.574 INFO:tasks.workunit.client.0.smithi067.stdout:2/729: mknod d1/d42/d5f/cee 0 2024-04-22T23:23:35.575 INFO:tasks.workunit.client.0.smithi067.stdout:0/867: dread db/d9a/d56/d70/fc3 [0,4194304] 0 2024-04-22T23:23:35.575 INFO:tasks.workunit.client.0.smithi067.stdout:4/901: mknod d12/d17/c12e 0 2024-04-22T23:23:35.576 INFO:tasks.workunit.client.1.smithi134.stdout:5/48: mkdir d9/df 0 2024-04-22T23:23:35.577 INFO:tasks.workunit.client.0.smithi067.stdout:7/687: dwrite d10/d47/d49/f54 [0,4194304] 0 2024-04-22T23:23:35.577 INFO:tasks.workunit.client.0.smithi067.stdout:5/809: fdatasync d5/d28/d36/d40/ff4 0 2024-04-22T23:23:35.579 INFO:tasks.workunit.client.1.smithi134.stdout:1/36: creat db/fd x:0 0 0 2024-04-22T23:23:35.580 INFO:tasks.workunit.client.0.smithi067.stdout:0/868: chown db/d9a/d56/d70/dba/f102 3649104 1 2024-04-22T23:23:35.581 INFO:tasks.workunit.client.1.smithi134.stdout:2/38: mkdir d2/dd/df 0 2024-04-22T23:23:35.583 INFO:tasks.workunit.client.1.smithi134.stdout:5/49: creat d9/dd/f10 x:0 0 0 2024-04-22T23:23:35.586 INFO:tasks.workunit.client.1.smithi134.stdout:1/37: rename c2 to db/ce 0 2024-04-22T23:23:35.587 INFO:tasks.workunit.client.0.smithi067.stdout:7/688: rename d10/d18/d35 to d10/d18/d88/daf/d100 0 2024-04-22T23:23:35.589 INFO:tasks.workunit.client.1.smithi134.stdout:1/38: creat db/ff x:0 0 0 2024-04-22T23:23:35.591 INFO:tasks.workunit.client.0.smithi067.stdout:2/730: unlink d1/d42/d5f/l8d 0 2024-04-22T23:23:35.591 INFO:tasks.workunit.client.1.smithi134.stdout:1/39: symlink db/l10 0 2024-04-22T23:23:35.592 INFO:tasks.workunit.client.1.smithi134.stdout:2/39: dwrite d2/dd/fe [4194304,4194304] 0 2024-04-22T23:23:35.595 INFO:tasks.workunit.client.0.smithi067.stdout:5/810: unlink d5/d28/daf/dbf/dcf/lcc 0 2024-04-22T23:23:35.596 INFO:tasks.workunit.client.0.smithi067.stdout:6/772: dwrite d17/d83/db5/dbc/fe3 [0,4194304] 0 2024-04-22T23:23:35.598 INFO:tasks.workunit.client.0.smithi067.stdout:5/811: truncate d5/d28/daf/f111 120245 0 2024-04-22T23:23:35.598 INFO:tasks.workunit.client.0.smithi067.stdout:6/773: chown d17/f33 0 1 2024-04-22T23:23:35.599 INFO:tasks.workunit.client.0.smithi067.stdout:2/731: symlink d1/d10/d33/d46/da4/lef 0 2024-04-22T23:23:35.599 INFO:tasks.workunit.client.0.smithi067.stdout:5/812: read d5/db/d12/d44/d47/f114 [200874,48046] 0 2024-04-22T23:23:35.602 INFO:tasks.workunit.client.0.smithi067.stdout:2/732: write d1/d10/d72/d53/f74 [2232563,76941] 0 2024-04-22T23:23:35.603 INFO:tasks.workunit.client.0.smithi067.stdout:0/869: creat db/d21/d29/d75/d9d/dde/f11a x:0 0 0 2024-04-22T23:23:35.606 INFO:tasks.workunit.client.0.smithi067.stdout:5/813: mkdir d5/db/d11f 0 2024-04-22T23:23:35.607 INFO:tasks.workunit.client.0.smithi067.stdout:5/814: chown d5/d28/dae/fd3 932 1 2024-04-22T23:23:35.615 INFO:tasks.workunit.client.0.smithi067.stdout:8/746: dread d1/dd/f53 [0,4194304] 0 2024-04-22T23:23:35.615 INFO:tasks.workunit.client.0.smithi067.stdout:6/774: creat d17/d39/ffd x:0 0 0 2024-04-22T23:23:35.617 INFO:tasks.workunit.client.0.smithi067.stdout:5/815: dwrite d5/d1a/d103/d4d/dc6/fed [0,4194304] 0 2024-04-22T23:23:35.625 INFO:tasks.workunit.client.0.smithi067.stdout:5/816: creat d5/d1a/d103/d4d/da9/f120 x:0 0 0 2024-04-22T23:23:35.625 INFO:tasks.workunit.client.0.smithi067.stdout:8/747: dwrite d1/dd/d26/f6b [0,4194304] 0 2024-04-22T23:23:35.630 INFO:tasks.workunit.client.0.smithi067.stdout:0/870: getdents db/d9a/d56/d6b 0 2024-04-22T23:23:35.634 INFO:tasks.workunit.client.0.smithi067.stdout:0/871: dread db/de/d2a/f99 [0,4194304] 0 2024-04-22T23:23:35.635 INFO:tasks.workunit.client.0.smithi067.stdout:7/689: truncate d10/d14/d1d/d46/d50/f2d 2863487 0 2024-04-22T23:23:35.638 INFO:tasks.workunit.client.0.smithi067.stdout:8/748: link d1/dd/d80/db9/de3/fbf d1/d3/d5c/d98/ff0 0 2024-04-22T23:23:35.641 INFO:tasks.workunit.client.0.smithi067.stdout:0/872: creat db/d71/dda/f11b x:0 0 0 2024-04-22T23:23:35.645 INFO:tasks.workunit.client.0.smithi067.stdout:7/690: mknod d10/d14/de0/c101 0 2024-04-22T23:23:35.646 INFO:tasks.workunit.client.0.smithi067.stdout:2/733: write d1/d10/d33/f5b [4617413,88638] 0 2024-04-22T23:23:35.647 INFO:tasks.workunit.client.0.smithi067.stdout:2/734: chown d1/d10/d41/f70 14641 1 2024-04-22T23:23:35.649 INFO:tasks.workunit.client.0.smithi067.stdout:8/749: mknod d1/d3/d5c/cf1 0 2024-04-22T23:23:35.650 INFO:tasks.workunit.client.0.smithi067.stdout:0/873: symlink db/d71/d10a/l11c 0 2024-04-22T23:23:35.652 INFO:tasks.workunit.client.0.smithi067.stdout:2/735: fdatasync d1/d10/d1c/d6d/d87/fbb 0 2024-04-22T23:23:35.655 INFO:tasks.workunit.client.0.smithi067.stdout:5/817: rmdir d5/d1a/d103/d4d 39 2024-04-22T23:23:35.657 INFO:tasks.workunit.client.0.smithi067.stdout:7/691: creat d10/d14/d1d/d26/d2c/d7a/f102 x:0 0 0 2024-04-22T23:23:35.659 INFO:tasks.workunit.client.1.smithi134.stdout:4/6: dread - f1 zero size 2024-04-22T23:23:35.660 INFO:tasks.workunit.client.0.smithi067.stdout:0/874: mknod db/d21/d29/d75/d9d/dde/d10f/dd9/c11d 0 2024-04-22T23:23:35.661 INFO:tasks.workunit.client.1.smithi134.stdout:4/7: fsync f1 0 2024-04-22T23:23:35.662 INFO:tasks.workunit.client.0.smithi067.stdout:5/818: mkdir d5/d1a/d103/d10d/d121 0 2024-04-22T23:23:35.664 INFO:tasks.workunit.client.0.smithi067.stdout:6/775: dwrite d17/f30 [0,4194304] 0 2024-04-22T23:23:35.664 INFO:tasks.workunit.client.0.smithi067.stdout:0/875: write db/de/d11/f3c [754920,35742] 0 2024-04-22T23:23:35.665 INFO:tasks.workunit.client.0.smithi067.stdout:6/776: chown d17/d39/d3a/d8d 58 1 2024-04-22T23:23:35.665 INFO:tasks.workunit.client.0.smithi067.stdout:7/692: dread d10/d18/f27 [4194304,4194304] 0 2024-04-22T23:23:35.667 INFO:tasks.workunit.client.0.smithi067.stdout:7/693: fsync d10/d47/dab/fd5 0 2024-04-22T23:23:35.668 INFO:tasks.workunit.client.1.smithi134.stdout:1/40: fsync db/fc 0 2024-04-22T23:23:35.670 INFO:tasks.workunit.client.1.smithi134.stdout:2/40: fsync f0 0 2024-04-22T23:23:35.674 INFO:tasks.workunit.client.0.smithi067.stdout:7/694: mknod d10/d14/d1d/d26/d80/c103 0 2024-04-22T23:23:35.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:35 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:23:35.675 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:35 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:23:35.675 INFO:tasks.workunit.client.0.smithi067.stdout:7/695: chown d10/de3/cfd 46 1 2024-04-22T23:23:35.676 INFO:tasks.workunit.client.1.smithi134.stdout:6/84: write d0/d19/de/f11 [282753,103727] 0 2024-04-22T23:23:35.680 INFO:tasks.workunit.client.1.smithi134.stdout:8/19: dwrite f4 [0,4194304] 0 2024-04-22T23:23:35.681 INFO:tasks.workunit.client.0.smithi067.stdout:5/819: creat d5/d28/dae/f122 x:0 0 0 2024-04-22T23:23:35.682 INFO:tasks.workunit.client.1.smithi134.stdout:3/16: dwrite f3 [0,4194304] 0 2024-04-22T23:23:35.682 INFO:tasks.workunit.client.0.smithi067.stdout:2/736: write d1/d10/d72/d68/d7b/fb7 [813073,14412] 0 2024-04-22T23:23:35.685 INFO:tasks.workunit.client.0.smithi067.stdout:2/737: dread - d1/d19/d6f/fd5 zero size 2024-04-22T23:23:35.686 INFO:tasks.workunit.client.1.smithi134.stdout:4/8: truncate f1 713307 0 2024-04-22T23:23:35.688 INFO:tasks.workunit.client.1.smithi134.stdout:1/41: dread f0 [0,4194304] 0 2024-04-22T23:23:35.688 INFO:tasks.workunit.client.0.smithi067.stdout:5/820: chown d5/db/d12/dbb/d5f/d79/ff0 136186 1 2024-04-22T23:23:35.688 INFO:tasks.workunit.client.0.smithi067.stdout:8/750: dwrite d1/d3/f11 [0,4194304] 0 2024-04-22T23:23:35.689 INFO:tasks.workunit.client.1.smithi134.stdout:8/20: mkdir d7 0 2024-04-22T23:23:35.692 INFO:tasks.workunit.client.0.smithi067.stdout:2/738: chown d1/d10/d72/d53/dba/fc3 3599887 1 2024-04-22T23:23:35.692 INFO:tasks.workunit.client.0.smithi067.stdout:5/821: chown d5/db/d12/dbb/d61/f78 16676745 1 2024-04-22T23:23:35.693 INFO:tasks.workunit.client.1.smithi134.stdout:5/50: dwrite f8 [0,4194304] 0 2024-04-22T23:23:35.693 INFO:tasks.workunit.client.1.smithi134.stdout:3/17: creat f5 x:0 0 0 2024-04-22T23:23:35.696 INFO:tasks.workunit.client.0.smithi067.stdout:8/751: dread d1/dd/f53 [0,4194304] 0 2024-04-22T23:23:35.698 INFO:tasks.workunit.client.0.smithi067.stdout:9/736: sync 2024-04-22T23:23:35.699 INFO:tasks.workunit.client.1.smithi134.stdout:5/51: mkdir d9/d11 0 2024-04-22T23:23:35.699 INFO:tasks.workunit.client.1.smithi134.stdout:8/21: symlink d7/l8 0 2024-04-22T23:23:35.703 INFO:tasks.workunit.client.1.smithi134.stdout:5/52: fdatasync d9/fb 0 2024-04-22T23:23:35.703 INFO:tasks.workunit.client.0.smithi067.stdout:5/822: creat d5/d106/f123 x:0 0 0 2024-04-22T23:23:35.705 INFO:tasks.workunit.client.0.smithi067.stdout:0/876: truncate db/f79 3929938 0 2024-04-22T23:23:35.706 INFO:tasks.workunit.client.1.smithi134.stdout:8/22: creat d7/f9 x:0 0 0 2024-04-22T23:23:35.707 INFO:tasks.workunit.client.0.smithi067.stdout:0/877: chown db/d71/cc8 124 1 2024-04-22T23:23:35.708 INFO:tasks.workunit.client.1.smithi134.stdout:7/75: sync 2024-04-22T23:23:35.708 INFO:tasks.workunit.client.1.smithi134.stdout:0/42: sync 2024-04-22T23:23:35.708 INFO:tasks.workunit.client.1.smithi134.stdout:4/9: rename f1 to f2 0 2024-04-22T23:23:35.709 INFO:tasks.workunit.client.1.smithi134.stdout:8/23: stat c3 0 2024-04-22T23:23:35.709 INFO:tasks.workunit.client.0.smithi067.stdout:6/777: truncate d17/d39/d3a/f61 3185498 0 2024-04-22T23:23:35.710 INFO:tasks.workunit.client.1.smithi134.stdout:9/23: dwrite d0/f1 [0,4194304] 0 2024-04-22T23:23:35.711 INFO:tasks.workunit.client.0.smithi067.stdout:5/823: dread d5/db/d12/f2b [4194304,4194304] 0 2024-04-22T23:23:35.712 INFO:tasks.workunit.client.1.smithi134.stdout:2/41: truncate d2/f9 170089 0 2024-04-22T23:23:35.712 INFO:tasks.workunit.client.0.smithi067.stdout:0/878: creat db/de/d11/d1a/f11e x:0 0 0 2024-04-22T23:23:35.713 INFO:tasks.workunit.client.0.smithi067.stdout:7/696: chown d10/d14/d1d/d46/d50/f2d 668256 1 2024-04-22T23:23:35.715 INFO:tasks.workunit.client.0.smithi067.stdout:6/778: creat d17/dae/ffe x:0 0 0 2024-04-22T23:23:35.715 INFO:tasks.workunit.client.0.smithi067.stdout:5/824: fdatasync d5/d1a/d103/fd2 0 2024-04-22T23:23:35.719 INFO:tasks.workunit.client.0.smithi067.stdout:0/879: chown db/de/d11/d1a/c37 249 1 2024-04-22T23:23:35.719 INFO:tasks.workunit.client.1.smithi134.stdout:2/42: unlink f0 0 2024-04-22T23:23:35.721 INFO:tasks.workunit.client.0.smithi067.stdout:5/825: fdatasync d5/d1a/d103/d4d/dc6/fed 0 2024-04-22T23:23:35.721 INFO:tasks.workunit.client.1.smithi134.stdout:1/42: dwrite db/fd [0,4194304] 0 2024-04-22T23:23:35.722 INFO:tasks.workunit.client.0.smithi067.stdout:6/779: truncate d17/d39/d3a/fe0 121005 0 2024-04-22T23:23:35.723 INFO:tasks.workunit.client.1.smithi134.stdout:3/18: fdatasync f3 0 2024-04-22T23:23:35.727 INFO:tasks.workunit.client.0.smithi067.stdout:5/826: chown d5/d28/d36/d40/f110 0 1 2024-04-22T23:23:35.729 INFO:tasks.workunit.client.1.smithi134.stdout:3/19: stat f3 0 2024-04-22T23:23:35.729 INFO:tasks.workunit.client.0.smithi067.stdout:7/697: dread d10/d18/d88/daf/d100/d62/f9a [0,4194304] 0 2024-04-22T23:23:35.733 INFO:tasks.workunit.client.1.smithi134.stdout:6/85: dwrite d0/d19/f7 [0,4194304] 0 2024-04-22T23:23:35.733 INFO:tasks.workunit.client.0.smithi067.stdout:0/880: creat db/d21/d29/d75/d9d/dde/d10f/dd9/f11f x:0 0 0 2024-04-22T23:23:35.733 INFO:tasks.workunit.client.0.smithi067.stdout:6/780: chown d17/d39/d53/f55 92836627 1 2024-04-22T23:23:35.736 INFO:tasks.workunit.client.0.smithi067.stdout:3/770: sync 2024-04-22T23:23:35.736 INFO:tasks.workunit.client.0.smithi067.stdout:6/781: stat d17/d39/d3a/cd5 0 2024-04-22T23:23:35.737 INFO:tasks.workunit.client.1.smithi134.stdout:8/24: dwrite f4 [4194304,4194304] 0 2024-04-22T23:23:35.738 INFO:tasks.workunit.client.0.smithi067.stdout:5/827: unlink d5/db/d12/dbb/d61/ld9 0 2024-04-22T23:23:35.738 INFO:tasks.workunit.client.0.smithi067.stdout:8/752: dwrite d1/dd/d80/d6f/f32 [0,4194304] 0 2024-04-22T23:23:35.740 INFO:tasks.workunit.client.1.smithi134.stdout:5/53: dread f2 [0,4194304] 0 2024-04-22T23:23:35.741 INFO:tasks.workunit.client.1.smithi134.stdout:9/24: dread d0/f1 [0,4194304] 0 2024-04-22T23:23:35.742 INFO:tasks.workunit.client.0.smithi067.stdout:6/782: chown d17/d39/f69 95455 1 2024-04-22T23:23:35.742 INFO:tasks.workunit.client.0.smithi067.stdout:9/737: dread d5/dd/d18/d39/d3e/d3f/f7e [0,4194304] 0 2024-04-22T23:23:35.747 INFO:tasks.workunit.client.0.smithi067.stdout:2/739: dwrite d1/d10/d41/f93 [0,4194304] 0 2024-04-22T23:23:35.747 INFO:tasks.workunit.client.0.smithi067.stdout:7/698: rmdir d10/d14/d1d/d26/d80/dd2 39 2024-04-22T23:23:35.748 INFO:tasks.workunit.client.1.smithi134.stdout:0/43: dwrite d1/d2/fa [0,4194304] 0 2024-04-22T23:23:35.748 INFO:tasks.workunit.client.1.smithi134.stdout:1/43: symlink db/l11 0 2024-04-22T23:23:35.748 INFO:tasks.workunit.client.1.smithi134.stdout:5/54: chown d9/fa 885 1 2024-04-22T23:23:35.748 INFO:tasks.workunit.client.1.smithi134.stdout:1/44: dread - db/fc zero size 2024-04-22T23:23:35.749 INFO:tasks.workunit.client.1.smithi134.stdout:7/76: creat d4/d10/f1c x:0 0 0 2024-04-22T23:23:35.749 INFO:tasks.workunit.client.1.smithi134.stdout:3/20: rename l4 to l6 0 2024-04-22T23:23:35.750 INFO:tasks.workunit.client.1.smithi134.stdout:6/86: mkdir d0/d19/de/d1f 0 2024-04-22T23:23:35.751 INFO:tasks.workunit.client.1.smithi134.stdout:8/25: symlink d7/la 0 2024-04-22T23:23:35.751 INFO:tasks.workunit.client.0.smithi067.stdout:5/828: dread d5/db/d12/d44/d47/fda [0,4194304] 0 2024-04-22T23:23:35.752 INFO:tasks.workunit.client.1.smithi134.stdout:2/43: mknod d2/dd/df/c10 0 2024-04-22T23:23:35.752 INFO:tasks.workunit.client.1.smithi134.stdout:2/44: dread - d2/fb zero size 2024-04-22T23:23:35.753 INFO:tasks.workunit.client.0.smithi067.stdout:2/740: dread d1/d42/f57 [0,4194304] 0 2024-04-22T23:23:35.754 INFO:tasks.workunit.client.1.smithi134.stdout:4/10: chown f2 33 1 2024-04-22T23:23:35.761 INFO:tasks.workunit.client.1.smithi134.stdout:5/55: readlink l1 0 2024-04-22T23:23:35.761 INFO:tasks.workunit.client.0.smithi067.stdout:6/783: symlink d17/d39/d53/d90/d46/d62/d88/lff 0 2024-04-22T23:23:35.763 INFO:tasks.workunit.client.0.smithi067.stdout:5/829: readlink d5/db/ld8 0 2024-04-22T23:23:35.764 INFO:tasks.workunit.client.1.smithi134.stdout:5/56: chown f2 572 1 2024-04-22T23:23:35.764 INFO:tasks.workunit.client.1.smithi134.stdout:2/45: dread - d2/f6 zero size 2024-04-22T23:23:35.764 INFO:tasks.workunit.client.1.smithi134.stdout:5/57: fdatasync f8 0 2024-04-22T23:23:35.767 INFO:tasks.workunit.client.1.smithi134.stdout:9/25: dwrite d0/f1 [0,4194304] 0 2024-04-22T23:23:35.767 INFO:tasks.workunit.client.1.smithi134.stdout:0/44: mkdir d1/db 0 2024-04-22T23:23:35.767 INFO:tasks.workunit.client.0.smithi067.stdout:7/699: mknod d10/d14/d1d/c104 0 2024-04-22T23:23:35.768 INFO:tasks.workunit.client.1.smithi134.stdout:1/45: creat db/f12 x:0 0 0 2024-04-22T23:23:35.769 INFO:tasks.workunit.client.1.smithi134.stdout:6/87: symlink d0/l20 0 2024-04-22T23:23:35.769 INFO:tasks.workunit.client.1.smithi134.stdout:8/26: mkdir d7/db 0 2024-04-22T23:23:35.770 INFO:tasks.workunit.client.1.smithi134.stdout:4/11: rename f2 to f3 0 2024-04-22T23:23:35.770 INFO:tasks.workunit.client.1.smithi134.stdout:2/46: stat d2/lc 0 2024-04-22T23:23:35.771 INFO:tasks.workunit.client.1.smithi134.stdout:8/27: write f4 [7591146,11350] 0 2024-04-22T23:23:35.771 INFO:tasks.workunit.client.1.smithi134.stdout:9/26: dread - d0/f6 zero size 2024-04-22T23:23:35.771 INFO:tasks.workunit.client.0.smithi067.stdout:0/881: link db/de/d2a/c7f db/d21/d29/d75/d9d/dde/d10f/dd9/c120 0 2024-04-22T23:23:35.772 INFO:tasks.workunit.client.0.smithi067.stdout:8/753: link d1/d3/d5c/cf1 d1/dd/d26/d62/ddf/ded/def/cf2 0 2024-04-22T23:23:35.772 INFO:tasks.workunit.client.1.smithi134.stdout:5/58: creat d9/f12 x:0 0 0 2024-04-22T23:23:35.773 INFO:tasks.workunit.client.0.smithi067.stdout:6/784: mknod d17/d39/d3a/c100 0 2024-04-22T23:23:35.774 INFO:tasks.workunit.client.1.smithi134.stdout:8/28: write f6 [840935,5048] 0 2024-04-22T23:23:35.775 INFO:tasks.workunit.client.1.smithi134.stdout:6/88: fdatasync d0/f10 0 2024-04-22T23:23:35.775 INFO:tasks.workunit.client.1.smithi134.stdout:2/47: write d2/fa [150918,87489] 0 2024-04-22T23:23:35.776 INFO:tasks.workunit.client.1.smithi134.stdout:5/59: chown l1 164566828 1 2024-04-22T23:23:35.778 INFO:tasks.workunit.client.0.smithi067.stdout:8/754: fsync d1/d3/d5c/d98/ff0 0 2024-04-22T23:23:35.778 INFO:tasks.workunit.client.1.smithi134.stdout:9/27: dread - d0/f5 zero size 2024-04-22T23:23:35.779 INFO:tasks.workunit.client.0.smithi067.stdout:6/785: chown d17/d39/d53/d90/d46/f9f 14385 1 2024-04-22T23:23:35.781 INFO:tasks.workunit.client.0.smithi067.stdout:7/700: dwrite d10/d47/dab/fd5 [0,4194304] 0 2024-04-22T23:23:35.783 INFO:tasks.workunit.client.1.smithi134.stdout:5/60: unlink l1 0 2024-04-22T23:23:35.783 INFO:tasks.workunit.client.1.smithi134.stdout:1/46: link l4 db/l13 0 2024-04-22T23:23:35.784 INFO:tasks.workunit.client.1.smithi134.stdout:2/48: mknod d2/c11 0 2024-04-22T23:23:35.785 INFO:tasks.workunit.client.0.smithi067.stdout:8/755: symlink d1/dd/d26/d62/ddf/lf3 0 2024-04-22T23:23:35.786 INFO:tasks.workunit.client.1.smithi134.stdout:6/89: truncate d0/f1e 567786 0 2024-04-22T23:23:35.788 INFO:tasks.workunit.client.0.smithi067.stdout:7/701: chown d10/d14/d1d/d46/d50/f29 66 1 2024-04-22T23:23:35.788 INFO:tasks.workunit.client.0.smithi067.stdout:0/882: dwrite db/d21/d29/d75/d9d/dde/d10f/f5d [0,4194304] 0 2024-04-22T23:23:35.789 INFO:tasks.workunit.client.0.smithi067.stdout:8/756: stat d1/d41/d46/cb5 0 2024-04-22T23:23:35.789 INFO:tasks.workunit.client.0.smithi067.stdout:8/757: chown d1/d3 63497 1 2024-04-22T23:23:35.791 INFO:tasks.workunit.client.0.smithi067.stdout:7/702: rmdir d10/d14/d1d/d26/d7e 39 2024-04-22T23:23:35.794 INFO:tasks.workunit.client.0.smithi067.stdout:0/883: mknod db/de/d11/d1a/da4/c121 0 2024-04-22T23:23:35.795 INFO:tasks.workunit.client.1.smithi134.stdout:2/49: creat d2/f12 x:0 0 0 2024-04-22T23:23:35.797 INFO:tasks.workunit.client.0.smithi067.stdout:0/884: readlink db/d9a/d56/d6b/lc7 0 2024-04-22T23:23:35.797 INFO:tasks.workunit.client.0.smithi067.stdout:3/771: dread d3/d4/f79 [0,4194304] 0 2024-04-22T23:23:35.798 INFO:tasks.workunit.client.1.smithi134.stdout:6/90: fdatasync d0/f8 0 2024-04-22T23:23:35.798 INFO:tasks.workunit.client.0.smithi067.stdout:7/703: write d10/d18/f27 [5458117,68052] 0 2024-04-22T23:23:35.798 INFO:tasks.workunit.client.1.smithi134.stdout:3/21: stat l6 0 2024-04-22T23:23:35.799 INFO:tasks.workunit.client.1.smithi134.stdout:6/91: stat d0/d19/de/f13 0 2024-04-22T23:23:35.800 INFO:tasks.workunit.client.1.smithi134.stdout:1/47: truncate f9 94223 0 2024-04-22T23:23:35.805 INFO:tasks.workunit.client.1.smithi134.stdout:8/29: getdents d7 0 2024-04-22T23:23:35.806 INFO:tasks.workunit.client.1.smithi134.stdout:8/30: fdatasync d7/f9 0 2024-04-22T23:23:35.806 INFO:tasks.workunit.client.0.smithi067.stdout:8/758: link d1/dd/c2b d1/dd/d80/cf4 0 2024-04-22T23:23:35.807 INFO:tasks.workunit.client.1.smithi134.stdout:8/31: chown d7/l8 949 1 2024-04-22T23:23:35.807 INFO:tasks.workunit.client.0.smithi067.stdout:9/738: dread d5/d21/f27 [0,4194304] 0 2024-04-22T23:23:35.807 INFO:tasks.workunit.client.0.smithi067.stdout:5/830: write d5/d28/dae/fef [4485980,35422] 0 2024-04-22T23:23:35.808 INFO:tasks.workunit.client.1.smithi134.stdout:0/45: write d1/f9 [923448,101313] 0 2024-04-22T23:23:35.808 INFO:tasks.workunit.client.0.smithi067.stdout:9/739: dread - d5/dd/d18/ff1 zero size 2024-04-22T23:23:35.809 INFO:tasks.workunit.client.0.smithi067.stdout:5/831: fdatasync d5/d28/d36/f75 0 2024-04-22T23:23:35.809 INFO:tasks.workunit.client.1.smithi134.stdout:3/22: symlink l7 0 2024-04-22T23:23:35.809 INFO:tasks.workunit.client.1.smithi134.stdout:3/23: chown l6 43544 1 2024-04-22T23:23:35.810 INFO:tasks.workunit.client.0.smithi067.stdout:0/885: rename db/fae to db/de/d11/d1a/da4/dec/f122 0 2024-04-22T23:23:35.810 INFO:tasks.workunit.client.1.smithi134.stdout:1/48: chown c1 455 1 2024-04-22T23:23:35.811 INFO:tasks.workunit.client.0.smithi067.stdout:3/772: mknod d3/d4/d10/d38/d53/d55/d5c/d70/d10a/c112 0 2024-04-22T23:23:35.811 INFO:tasks.workunit.client.1.smithi134.stdout:5/61: link f6 d9/dd/f13 0 2024-04-22T23:23:35.812 INFO:tasks.workunit.client.0.smithi067.stdout:5/832: dread - d5/db/d12/dbb/d5f/d79/dba/de0/f109 zero size 2024-04-22T23:23:35.813 INFO:tasks.workunit.client.1.smithi134.stdout:4/12: chown f3 1251 1 2024-04-22T23:23:35.813 INFO:tasks.workunit.client.1.smithi134.stdout:4/13: readlink - no filename 2024-04-22T23:23:35.813 INFO:tasks.workunit.client.0.smithi067.stdout:0/886: dread db/d9a/d56/d70/fc3 [0,4194304] 0 2024-04-22T23:23:35.814 INFO:tasks.workunit.client.0.smithi067.stdout:5/833: write d5/db/d12/ff6 [3954881,68021] 0 2024-04-22T23:23:35.815 INFO:tasks.workunit.client.1.smithi134.stdout:4/14: dread f3 [0,4194304] 0 2024-04-22T23:23:35.816 INFO:tasks.workunit.client.0.smithi067.stdout:3/773: truncate d3/d4/d10/d8a/fa6 4227951 0 2024-04-22T23:23:35.816 INFO:tasks.workunit.client.0.smithi067.stdout:9/740: rmdir d5/d21 39 2024-04-22T23:23:35.817 INFO:tasks.workunit.client.1.smithi134.stdout:3/24: rename f5 to f8 0 2024-04-22T23:23:35.818 INFO:tasks.workunit.client.0.smithi067.stdout:8/759: symlink d1/d3/lf5 0 2024-04-22T23:23:35.820 INFO:tasks.workunit.client.1.smithi134.stdout:7/77: dwrite d4/d8/fd [0,4194304] 0 2024-04-22T23:23:35.822 INFO:tasks.workunit.client.0.smithi067.stdout:7/704: write d10/d14/d1d/d26/d2c/ff2 [1704463,95677] 0 2024-04-22T23:23:35.822 INFO:tasks.workunit.client.1.smithi134.stdout:9/28: write d0/f6 [828635,67747] 0 2024-04-22T23:23:35.824 INFO:tasks.workunit.client.1.smithi134.stdout:4/15: dread f3 [0,4194304] 0 2024-04-22T23:23:35.824 INFO:tasks.workunit.client.0.smithi067.stdout:6/786: dread d17/d39/d53/f9e [0,4194304] 0 2024-04-22T23:23:35.824 INFO:tasks.workunit.client.0.smithi067.stdout:9/741: mkdir d5/dd/d18/d39/d8f/df9 0 2024-04-22T23:23:35.825 INFO:tasks.workunit.client.0.smithi067.stdout:8/760: symlink d1/dd/d80/db9/lf6 0 2024-04-22T23:23:35.826 INFO:tasks.workunit.client.0.smithi067.stdout:5/834: creat d5/db/d35/f124 x:0 0 0 2024-04-22T23:23:35.826 INFO:tasks.workunit.client.0.smithi067.stdout:8/761: chown d1/d3/d27/f81 42 1 2024-04-22T23:23:35.827 INFO:tasks.workunit.client.0.smithi067.stdout:7/705: symlink d10/d18/d88/daf/d100/d62/l105 0 2024-04-22T23:23:35.827 INFO:tasks.workunit.client.1.smithi134.stdout:3/25: creat f9 x:0 0 0 2024-04-22T23:23:35.827 INFO:tasks.workunit.client.1.smithi134.stdout:7/78: mkdir d4/d8/d1d 0 2024-04-22T23:23:35.832 INFO:tasks.workunit.client.1.smithi134.stdout:1/49: link db/fc db/f14 0 2024-04-22T23:23:35.832 INFO:tasks.workunit.client.0.smithi067.stdout:6/787: creat d17/d39/d53/f101 x:0 0 0 2024-04-22T23:23:35.834 INFO:tasks.workunit.client.0.smithi067.stdout:5/835: mknod d5/d3e/d90/c125 0 2024-04-22T23:23:35.834 INFO:tasks.workunit.client.1.smithi134.stdout:0/46: dread d1/d2/f4 [0,4194304] 0 2024-04-22T23:23:35.836 INFO:tasks.workunit.client.1.smithi134.stdout:9/29: dwrite d0/f4 [0,4194304] 0 2024-04-22T23:23:35.836 INFO:tasks.workunit.client.0.smithi067.stdout:0/887: write db/d9a/d56/d6b/f114 [333879,49995] 0 2024-04-22T23:23:35.837 INFO:tasks.workunit.client.0.smithi067.stdout:9/742: truncate d5/d21/d47/d7f/f9f 503990 0 2024-04-22T23:23:35.838 INFO:tasks.workunit.client.1.smithi134.stdout:3/26: write f8 [108653,33708] 0 2024-04-22T23:23:35.839 INFO:tasks.workunit.client.0.smithi067.stdout:6/788: creat d17/d83/d9b/ddd/f102 x:0 0 0 2024-04-22T23:23:35.839 INFO:tasks.workunit.client.1.smithi134.stdout:2/50: getdents d2/dd 0 2024-04-22T23:23:35.840 INFO:tasks.workunit.client.1.smithi134.stdout:5/62: link f6 d9/df/f14 0 2024-04-22T23:23:35.841 INFO:tasks.workunit.client.0.smithi067.stdout:3/774: truncate d3/d4/d10/d1f/d3c/fac 1538995 0 2024-04-22T23:23:35.842 INFO:tasks.workunit.client.1.smithi134.stdout:0/47: mknod d1/db/cc 0 2024-04-22T23:23:35.843 INFO:tasks.workunit.client.0.smithi067.stdout:0/888: symlink db/l123 0 2024-04-22T23:23:35.844 INFO:tasks.workunit.client.1.smithi134.stdout:3/27: creat fa x:0 0 0 2024-04-22T23:23:35.846 INFO:tasks.workunit.client.0.smithi067.stdout:5/836: symlink d5/db/d11f/l126 0 2024-04-22T23:23:35.847 INFO:tasks.workunit.client.1.smithi134.stdout:5/63: fdatasync f2 0 2024-04-22T23:23:35.849 INFO:tasks.workunit.client.1.smithi134.stdout:4/16: write f3 [1537092,57568] 0 2024-04-22T23:23:35.850 INFO:tasks.workunit.client.0.smithi067.stdout:3/775: mkdir d3/d4/d10/d38/dbf/d113 0 2024-04-22T23:23:35.850 INFO:tasks.workunit.client.1.smithi134.stdout:9/30: dwrite d0/f6 [0,4194304] 0 2024-04-22T23:23:35.850 INFO:tasks.workunit.client.1.smithi134.stdout:7/79: dread d4/f13 [0,4194304] 0 2024-04-22T23:23:35.851 INFO:tasks.workunit.client.1.smithi134.stdout:0/48: unlink d1/l8 0 2024-04-22T23:23:35.853 INFO:tasks.workunit.client.0.smithi067.stdout:7/706: getdents d10/d14/d1d/d26/d2c 0 2024-04-22T23:23:35.853 INFO:tasks.workunit.client.1.smithi134.stdout:5/64: write f8 [3071034,20547] 0 2024-04-22T23:23:35.854 INFO:tasks.workunit.client.0.smithi067.stdout:8/762: dwrite d1/d3/d10/fbe [0,4194304] 0 2024-04-22T23:23:35.856 INFO:tasks.workunit.client.1.smithi134.stdout:4/17: write f3 [1033226,44277] 0 2024-04-22T23:23:35.858 INFO:tasks.workunit.client.0.smithi067.stdout:9/743: dwrite d5/dd/d18/d3c/d40/f62 [0,4194304] 0 2024-04-22T23:23:35.859 INFO:tasks.workunit.client.1.smithi134.stdout:1/50: getdents db 0 2024-04-22T23:23:35.860 INFO:tasks.workunit.client.0.smithi067.stdout:8/763: write d1/dd/d80/f86 [239096,101598] 0 2024-04-22T23:23:35.862 INFO:tasks.workunit.client.1.smithi134.stdout:0/49: mkdir d1/d5/dd 0 2024-04-22T23:23:35.862 INFO:tasks.workunit.client.0.smithi067.stdout:7/707: write d10/d14/d1d/d26/fbd [9055,99200] 0 2024-04-22T23:23:35.864 INFO:tasks.workunit.client.1.smithi134.stdout:3/28: dwrite fa [0,4194304] 0 2024-04-22T23:23:35.865 INFO:tasks.workunit.client.1.smithi134.stdout:5/65: creat d9/f15 x:0 0 0 2024-04-22T23:23:35.866 INFO:tasks.workunit.client.0.smithi067.stdout:6/789: write d17/d39/d53/d90/d46/db0/db9/de7/fe2 [549759,102389] 0 2024-04-22T23:23:35.868 INFO:tasks.workunit.client.0.smithi067.stdout:7/708: write d10/d14/d1d/d26/d2c/d7a/f102 [868897,101203] 0 2024-04-22T23:23:35.869 INFO:tasks.workunit.client.0.smithi067.stdout:3/776: mknod d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d90/c114 0 2024-04-22T23:23:35.870 INFO:tasks.workunit.client.1.smithi134.stdout:3/29: dread f8 [0,4194304] 0 2024-04-22T23:23:35.872 INFO:tasks.workunit.client.0.smithi067.stdout:7/709: chown d10/d14/d1d/d26/d2c/d41/c60 5325332 1 2024-04-22T23:23:35.873 INFO:tasks.workunit.client.1.smithi134.stdout:9/31: rename d0/f4 to d0/d7/f9 0 2024-04-22T23:23:35.874 INFO:tasks.workunit.client.1.smithi134.stdout:9/32: fsync d0/f6 0 2024-04-22T23:23:35.874 INFO:tasks.workunit.client.0.smithi067.stdout:3/777: readlink d3/d4/d28/lb0 0 2024-04-22T23:23:35.875 INFO:tasks.workunit.client.1.smithi134.stdout:4/18: dread f3 [0,4194304] 0 2024-04-22T23:23:35.876 INFO:tasks.workunit.client.1.smithi134.stdout:5/66: creat d9/f16 x:0 0 0 2024-04-22T23:23:35.878 INFO:tasks.workunit.client.0.smithi067.stdout:8/764: getdents d1/d41/db4 0 2024-04-22T23:23:35.880 INFO:tasks.workunit.client.0.smithi067.stdout:0/889: dread db/f6a [4194304,4194304] 0 2024-04-22T23:23:35.881 INFO:tasks.workunit.client.0.smithi067.stdout:6/790: rename d17/f30 to d17/d39/f103 0 2024-04-22T23:23:35.881 INFO:tasks.workunit.client.1.smithi134.stdout:4/19: dread f3 [0,4194304] 0 2024-04-22T23:23:35.882 INFO:tasks.workunit.client.1.smithi134.stdout:3/30: symlink lb 0 2024-04-22T23:23:35.882 INFO:tasks.workunit.client.0.smithi067.stdout:9/744: dwrite d5/d21/faf [0,4194304] 0 2024-04-22T23:23:35.883 INFO:tasks.workunit.client.0.smithi067.stdout:7/710: mkdir d10/d14/d1d/d46/d6a/dcc/d106 0 2024-04-22T23:23:35.883 INFO:tasks.workunit.client.1.smithi134.stdout:2/51: dwrite d2/dd/fe [8388608,4194304] 0 2024-04-22T23:23:35.885 INFO:tasks.workunit.client.0.smithi067.stdout:5/837: dwrite d5/db/d12/dbb/d61/d89/f92 [0,4194304] 0 2024-04-22T23:23:35.885 INFO:tasks.workunit.client.1.smithi134.stdout:2/52: write d2/fa [959084,94187] 0 2024-04-22T23:23:35.885 INFO:tasks.workunit.client.1.smithi134.stdout:0/50: rename d1/db/cc to d1/d2/ce 0 2024-04-22T23:23:35.888 INFO:tasks.workunit.client.1.smithi134.stdout:5/67: mkdir d9/dd/d17 0 2024-04-22T23:23:35.889 INFO:tasks.workunit.client.0.smithi067.stdout:6/791: readlink d17/d39/d53/d90/d46/d62/lee 0 2024-04-22T23:23:35.889 INFO:tasks.workunit.client.1.smithi134.stdout:4/20: mkdir d4 0 2024-04-22T23:23:35.891 INFO:tasks.workunit.client.1.smithi134.stdout:3/31: mkdir dc 0 2024-04-22T23:23:35.893 INFO:tasks.workunit.client.1.smithi134.stdout:7/80: link d4/lc d4/d8/l1e 0 2024-04-22T23:23:35.893 INFO:tasks.workunit.client.0.smithi067.stdout:6/792: dread d17/d83/fcb [0,4194304] 0 2024-04-22T23:23:35.893 INFO:tasks.workunit.client.0.smithi067.stdout:3/778: mkdir d3/d4/d10/d38/d53/d55/d5c/d115 0 2024-04-22T23:23:35.894 INFO:tasks.workunit.client.0.smithi067.stdout:5/838: chown d5/d1a/d103/d4d/l59 763386679 1 2024-04-22T23:23:35.895 INFO:tasks.workunit.client.0.smithi067.stdout:3/779: stat d3/d4/f79 0 2024-04-22T23:23:35.897 INFO:tasks.workunit.client.0.smithi067.stdout:7/711: rename d10/d18/d88/daf/d100/d62/f9a to d10/d14/d1d/d26/d2c/d7a/dc2/f107 0 2024-04-22T23:23:35.897 INFO:tasks.workunit.client.1.smithi134.stdout:2/53: creat d2/dd/f13 x:0 0 0 2024-04-22T23:23:35.897 INFO:tasks.workunit.client.1.smithi134.stdout:0/51: mknod d1/db/cf 0 2024-04-22T23:23:35.899 INFO:tasks.workunit.client.1.smithi134.stdout:0/52: chown d1/db/cf 19 1 2024-04-22T23:23:35.900 INFO:tasks.workunit.client.1.smithi134.stdout:5/68: stat d9/df/f14 0 2024-04-22T23:23:35.904 INFO:tasks.workunit.client.1.smithi134.stdout:0/53: rmdir d1/d5 39 2024-04-22T23:23:35.904 INFO:tasks.workunit.client.1.smithi134.stdout:4/21: symlink d4/l5 0 2024-04-22T23:23:35.904 INFO:tasks.workunit.client.0.smithi067.stdout:8/765: write d1/d3/d5c/d98/fbc [1901592,25655] 0 2024-04-22T23:23:35.905 INFO:tasks.workunit.client.1.smithi134.stdout:3/32: write f8 [311108,3456] 0 2024-04-22T23:23:35.906 INFO:tasks.workunit.client.0.smithi067.stdout:0/890: rename db/d21/d29/d75/d9d/dde/d10f/d46/c88 to db/d60/d112/c124 0 2024-04-22T23:23:35.907 INFO:tasks.workunit.client.1.smithi134.stdout:1/51: dwrite f9 [0,4194304] 0 2024-04-22T23:23:35.907 INFO:tasks.workunit.client.1.smithi134.stdout:7/81: creat d4/d10/f1f x:0 0 0 2024-04-22T23:23:35.908 INFO:tasks.workunit.client.1.smithi134.stdout:3/33: fdatasync f9 0 2024-04-22T23:23:35.908 INFO:tasks.workunit.client.0.smithi067.stdout:7/712: fsync d10/f12 0 2024-04-22T23:23:35.909 INFO:tasks.workunit.client.0.smithi067.stdout:0/891: write db/d9a/ff0 [1400787,97719] 0 2024-04-22T23:23:35.909 INFO:tasks.workunit.client.0.smithi067.stdout:6/793: mkdir d17/d39/d104 0 2024-04-22T23:23:35.911 INFO:tasks.workunit.client.0.smithi067.stdout:5/839: mkdir d5/d28/d36/d40/d97/dc8/d127 0 2024-04-22T23:23:35.912 INFO:tasks.workunit.client.0.smithi067.stdout:8/766: symlink d1/dd/d26/d62/ddf/ded/lf7 0 2024-04-22T23:23:35.912 INFO:tasks.workunit.client.1.smithi134.stdout:4/22: write f3 [2512031,121812] 0 2024-04-22T23:23:35.915 INFO:tasks.workunit.client.1.smithi134.stdout:2/54: dwrite d2/f12 [0,4194304] 0 2024-04-22T23:23:35.915 INFO:tasks.workunit.client.0.smithi067.stdout:5/840: chown d5/d28/c11a 4 1 2024-04-22T23:23:35.915 INFO:tasks.workunit.client.0.smithi067.stdout:0/892: dread - db/de/d11/d1a/da4/dec/f103 zero size 2024-04-22T23:23:35.916 INFO:tasks.workunit.client.1.smithi134.stdout:1/52: mkdir db/d15 0 2024-04-22T23:23:35.916 INFO:tasks.workunit.client.0.smithi067.stdout:7/713: readlink d10/d14/d1d/d26/d2c/d41/db0/ddd/lef 0 2024-04-22T23:23:35.916 INFO:tasks.workunit.client.1.smithi134.stdout:1/53: readlink db/l10 0 2024-04-22T23:23:35.917 INFO:tasks.workunit.client.1.smithi134.stdout:2/55: read - d2/f6 zero size 2024-04-22T23:23:35.917 INFO:tasks.workunit.client.1.smithi134.stdout:3/34: mkdir dc/dd 0 2024-04-22T23:23:35.917 INFO:tasks.workunit.client.0.smithi067.stdout:1/730: sync 2024-04-22T23:23:35.917 INFO:tasks.workunit.client.0.smithi067.stdout:4/902: sync 2024-04-22T23:23:35.918 INFO:tasks.workunit.client.0.smithi067.stdout:9/745: dwrite d5/f71 [0,4194304] 0 2024-04-22T23:23:35.918 INFO:tasks.workunit.client.1.smithi134.stdout:2/56: chown d2/fb 1969591684 1 2024-04-22T23:23:35.918 INFO:tasks.workunit.client.1.smithi134.stdout:2/57: stat d2/c11 0 2024-04-22T23:23:35.919 INFO:tasks.workunit.client.0.smithi067.stdout:7/714: readlink d10/d18/d88/daf/lcb 0 2024-04-22T23:23:35.920 INFO:tasks.workunit.client.1.smithi134.stdout:7/82: rmdir d4/d10/d17 0 2024-04-22T23:23:35.922 INFO:tasks.workunit.client.0.smithi067.stdout:1/731: write d8/d12/fd6 [5489,45332] 0 2024-04-22T23:23:35.922 INFO:tasks.workunit.client.0.smithi067.stdout:6/794: rename d17/d39/d53/d90/d46/f5f to d17/d39/d53/d90/d46/db0/db9/f105 0 2024-04-22T23:23:35.923 INFO:tasks.workunit.client.0.smithi067.stdout:5/841: symlink d5/db/d11f/l128 0 2024-04-22T23:23:35.925 INFO:tasks.workunit.client.1.smithi134.stdout:4/23: unlink f3 0 2024-04-22T23:23:35.926 INFO:tasks.workunit.client.1.smithi134.stdout:4/24: chown d4/l5 1014454 1 2024-04-22T23:23:35.926 INFO:tasks.workunit.client.1.smithi134.stdout:4/25: chown d4/l5 31393 1 2024-04-22T23:23:35.927 INFO:tasks.workunit.client.1.smithi134.stdout:4/26: dwrite - no filename 2024-04-22T23:23:35.927 INFO:tasks.workunit.client.1.smithi134.stdout:4/27: dread - no filename 2024-04-22T23:23:35.927 INFO:tasks.workunit.client.1.smithi134.stdout:4/28: write - no filename 2024-04-22T23:23:35.927 INFO:tasks.workunit.client.1.smithi134.stdout:1/54: mkdir db/d16 0 2024-04-22T23:23:35.927 INFO:tasks.workunit.client.1.smithi134.stdout:1/55: dread - db/f14 zero size 2024-04-22T23:23:35.929 INFO:tasks.workunit.client.1.smithi134.stdout:5/69: getdents d9/dd 0 2024-04-22T23:23:35.929 INFO:tasks.workunit.client.1.smithi134.stdout:1/56: dread - db/f14 zero size 2024-04-22T23:23:35.929 INFO:tasks.workunit.client.0.smithi067.stdout:0/893: symlink db/de/d2a/l125 0 2024-04-22T23:23:35.931 INFO:tasks.workunit.client.1.smithi134.stdout:3/35: mkdir dc/de 0 2024-04-22T23:23:35.931 INFO:tasks.workunit.client.0.smithi067.stdout:4/903: write d12/d5f/d7e/fc9 [9939,43600] 0 2024-04-22T23:23:35.931 INFO:tasks.workunit.client.0.smithi067.stdout:0/894: chown db/de/d11/d92/cb4 235408 1 2024-04-22T23:23:35.933 INFO:tasks.workunit.client.1.smithi134.stdout:2/58: rename d2/fa to d2/f14 0 2024-04-22T23:23:35.933 INFO:tasks.workunit.client.0.smithi067.stdout:9/746: chown d5/d21/d47/d57/dc0/fc3 3624921 1 2024-04-22T23:23:35.934 INFO:tasks.workunit.client.0.smithi067.stdout:4/904: write d12/d42/d4b/d77/d9d/dc1/f104 [375924,52803] 0 2024-04-22T23:23:35.935 INFO:tasks.workunit.client.0.smithi067.stdout:5/842: creat d5/db/d12/dbb/f129 x:0 0 0 2024-04-22T23:23:35.937 INFO:tasks.workunit.client.0.smithi067.stdout:1/732: mknod d8/d9/d2d/d4d/d48/d9a/cf0 0 2024-04-22T23:23:35.937 INFO:tasks.workunit.client.1.smithi134.stdout:4/29: rmdir d4 39 2024-04-22T23:23:35.938 INFO:tasks.workunit.client.1.smithi134.stdout:4/30: dread - no filename 2024-04-22T23:23:35.938 INFO:tasks.workunit.client.1.smithi134.stdout:4/31: write - no filename 2024-04-22T23:23:35.939 INFO:tasks.workunit.client.1.smithi134.stdout:5/70: dwrite f8 [0,4194304] 0 2024-04-22T23:23:35.939 INFO:tasks.workunit.client.0.smithi067.stdout:7/715: dread d10/d14/d1d/d46/d50/f30 [0,4194304] 0 2024-04-22T23:23:35.940 INFO:tasks.workunit.client.0.smithi067.stdout:0/895: mknod db/d9a/c126 0 2024-04-22T23:23:35.942 INFO:tasks.workunit.client.1.smithi134.stdout:0/54: write d1/d5/f7 [1269900,37218] 0 2024-04-22T23:23:35.942 INFO:tasks.workunit.client.0.smithi067.stdout:7/716: write d10/d14/d1d/d26/d80/fc8 [955287,98339] 0 2024-04-22T23:23:35.946 INFO:tasks.workunit.client.0.smithi067.stdout:9/747: unlink d5/d21/d47/d57/dc0/ce9 0 2024-04-22T23:23:35.946 INFO:tasks.workunit.client.0.smithi067.stdout:3/780: dread d3/d4/d28/d40/dfd/dfe/f6e [0,4194304] 0 2024-04-22T23:23:35.951 INFO:tasks.workunit.client.0.smithi067.stdout:5/843: stat d5/db/d12/f116 0 2024-04-22T23:23:35.952 INFO:tasks.workunit.client.0.smithi067.stdout:6/795: mkdir d17/d39/d104/d106 0 2024-04-22T23:23:35.952 INFO:tasks.workunit.client.1.smithi134.stdout:2/59: mkdir d2/d15 0 2024-04-22T23:23:35.952 INFO:tasks.workunit.client.1.smithi134.stdout:7/83: mknod d4/d10/c20 0 2024-04-22T23:23:35.952 INFO:tasks.workunit.client.1.smithi134.stdout:4/32: readlink d4/l5 0 2024-04-22T23:23:35.952 INFO:tasks.workunit.client.1.smithi134.stdout:4/33: dwrite - no filename 2024-04-22T23:23:35.952 INFO:tasks.workunit.client.1.smithi134.stdout:4/34: fdatasync - no filename 2024-04-22T23:23:35.952 INFO:tasks.workunit.client.1.smithi134.stdout:4/35: dread - no filename 2024-04-22T23:23:35.952 INFO:tasks.workunit.client.1.smithi134.stdout:1/57: mkdir db/d15/d17 0 2024-04-22T23:23:35.953 INFO:tasks.workunit.client.1.smithi134.stdout:3/36: mkdir dc/de/df 0 2024-04-22T23:23:35.956 INFO:tasks.workunit.client.1.smithi134.stdout:7/84: unlink f3 0 2024-04-22T23:23:35.956 INFO:tasks.workunit.client.1.smithi134.stdout:7/85: fdatasync d4/d10/f1c 0 2024-04-22T23:23:35.957 INFO:tasks.workunit.client.1.smithi134.stdout:4/36: creat d4/f6 x:0 0 0 2024-04-22T23:23:35.958 INFO:tasks.workunit.client.1.smithi134.stdout:4/37: fdatasync d4/f6 0 2024-04-22T23:23:35.958 INFO:tasks.workunit.client.1.smithi134.stdout:1/58: creat db/d15/f18 x:0 0 0 2024-04-22T23:23:35.959 INFO:tasks.workunit.client.1.smithi134.stdout:4/38: write d4/f6 [507550,78086] 0 2024-04-22T23:23:35.959 INFO:tasks.workunit.client.1.smithi134.stdout:3/37: symlink dc/de/l10 0 2024-04-22T23:23:35.964 INFO:tasks.workunit.client.1.smithi134.stdout:4/39: mkdir d4/d7 0 2024-04-22T23:23:35.964 INFO:tasks.workunit.client.1.smithi134.stdout:4/40: fdatasync d4/f6 0 2024-04-22T23:23:35.964 INFO:tasks.workunit.client.1.smithi134.stdout:1/59: mknod db/d15/c19 0 2024-04-22T23:23:35.965 INFO:tasks.workunit.client.1.smithi134.stdout:1/60: chown db/ce 133574608 1 2024-04-22T23:23:35.967 INFO:tasks.workunit.client.1.smithi134.stdout:5/71: link l7 d9/dd/d17/l18 0 2024-04-22T23:23:35.970 INFO:tasks.workunit.client.1.smithi134.stdout:2/60: getdents d2/dd 0 2024-04-22T23:23:35.972 INFO:tasks.workunit.client.1.smithi134.stdout:3/38: creat dc/dd/f11 x:0 0 0 2024-04-22T23:23:35.973 INFO:tasks.workunit.client.1.smithi134.stdout:5/72: write d9/fc [36061,101585] 0 2024-04-22T23:23:35.974 INFO:tasks.workunit.client.1.smithi134.stdout:7/86: dwrite d4/d10/f1b [4194304,4194304] 0 2024-04-22T23:23:35.979 INFO:tasks.workunit.client.1.smithi134.stdout:3/39: mknod dc/de/c12 0 2024-04-22T23:23:35.981 INFO:tasks.workunit.client.1.smithi134.stdout:3/40: chown fa 143 1 2024-04-22T23:23:35.981 INFO:tasks.workunit.client.1.smithi134.stdout:0/55: dwrite d1/d2/f4 [0,4194304] 0 2024-04-22T23:23:35.981 INFO:tasks.workunit.client.1.smithi134.stdout:4/41: dwrite d4/f6 [0,4194304] 0 2024-04-22T23:23:35.982 INFO:tasks.workunit.client.1.smithi134.stdout:1/61: dread db/fd [0,4194304] 0 2024-04-22T23:23:35.982 INFO:tasks.workunit.client.1.smithi134.stdout:0/56: chown d1/d2/fa 0 1 2024-04-22T23:23:35.983 INFO:tasks.workunit.client.1.smithi134.stdout:3/41: truncate dc/dd/f11 898006 0 2024-04-22T23:23:35.984 INFO:tasks.workunit.client.1.smithi134.stdout:1/62: write db/f12 [352246,83810] 0 2024-04-22T23:23:35.990 INFO:tasks.workunit.client.1.smithi134.stdout:0/57: dwrite d1/d2/f4 [4194304,4194304] 0 2024-04-22T23:23:35.997 INFO:tasks.workunit.client.1.smithi134.stdout:0/58: dwrite d1/d2/fa [0,4194304] 0 2024-04-22T23:23:36.011 INFO:tasks.workunit.client.1.smithi134.stdout:5/73: creat d9/f19 x:0 0 0 2024-04-22T23:23:36.011 INFO:tasks.workunit.client.1.smithi134.stdout:7/87: creat d4/d8/f21 x:0 0 0 2024-04-22T23:23:36.013 INFO:tasks.workunit.client.1.smithi134.stdout:2/61: rename d2/c7 to d2/c16 0 2024-04-22T23:23:36.014 INFO:tasks.workunit.client.1.smithi134.stdout:3/42: symlink dc/l13 0 2024-04-22T23:23:36.045 INFO:tasks.workunit.client.0.smithi067.stdout:1/733: unlink d8/d9/d2f/cc3 0 2024-04-22T23:23:36.046 INFO:tasks.workunit.client.0.smithi067.stdout:0/896: dwrite db/d9a/d56/d6b/f114 [0,4194304] 0 2024-04-22T23:23:36.047 INFO:tasks.workunit.client.0.smithi067.stdout:9/748: write d5/d21/d47/d56/dc6/fd7 [1223953,471] 0 2024-04-22T23:23:36.049 INFO:tasks.workunit.client.0.smithi067.stdout:4/905: getdents d12/d42/d4b/d77/d8c 0 2024-04-22T23:23:36.049 INFO:tasks.workunit.client.0.smithi067.stdout:0/897: chown db/d21/d29/d75/d9d/ff7 319909573 1 2024-04-22T23:23:36.050 INFO:tasks.workunit.client.0.smithi067.stdout:9/749: fdatasync d5/dd/d18/d39/d3e/d3f/d63/fa8 0 2024-04-22T23:23:36.052 INFO:tasks.workunit.client.0.smithi067.stdout:0/898: dread - db/de/d11/d1a/f11e zero size 2024-04-22T23:23:36.053 INFO:tasks.workunit.client.0.smithi067.stdout:0/899: read - db/d21/d29/d75/fa1 zero size 2024-04-22T23:23:36.057 INFO:tasks.workunit.client.0.smithi067.stdout:3/781: truncate d3/d4/d28/d40/dfd/dfe/d4b/f6c 36801 0 2024-04-22T23:23:36.060 INFO:tasks.workunit.client.0.smithi067.stdout:6/796: truncate d17/d39/d3a/da9/ff5 292132 0 2024-04-22T23:23:36.064 INFO:tasks.workunit.client.0.smithi067.stdout:5/844: link d5/db/d12/d44/f81 d5/d1a/d103/f12a 0 2024-04-22T23:23:36.065 INFO:tasks.workunit.client.0.smithi067.stdout:3/782: mknod d3/d4/d28/d5f/c116 0 2024-04-22T23:23:36.065 INFO:tasks.workunit.client.0.smithi067.stdout:5/845: dread - d5/db/d12/d44/f81 zero size 2024-04-22T23:23:36.067 INFO:tasks.workunit.client.0.smithi067.stdout:5/846: write d5/d28/f69 [1504090,113862] 0 2024-04-22T23:23:36.068 INFO:tasks.workunit.client.0.smithi067.stdout:6/797: creat d17/d83/d9b/ddd/f107 x:0 0 0 2024-04-22T23:23:36.069 INFO:tasks.workunit.client.0.smithi067.stdout:3/783: chown d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d90/da2/daf/dd8/ffc 169 1 2024-04-22T23:23:36.071 INFO:tasks.workunit.client.0.smithi067.stdout:4/906: mkdir d12/d17/d3f/d12f 0 2024-04-22T23:23:36.072 INFO:tasks.workunit.client.0.smithi067.stdout:0/900: write db/d9a/d56/d70/fc3 [1133240,23972] 0 2024-04-22T23:23:36.075 INFO:tasks.workunit.client.0.smithi067.stdout:1/734: dread d8/d9/d2d/d4d/d48/f7f [0,4194304] 0 2024-04-22T23:23:36.077 INFO:tasks.workunit.client.0.smithi067.stdout:9/750: dwrite d5/d21/d47/d7f/f8e [0,4194304] 0 2024-04-22T23:23:36.078 INFO:tasks.workunit.client.0.smithi067.stdout:6/798: unlink d17/d39/l79 0 2024-04-22T23:23:36.083 INFO:tasks.workunit.client.0.smithi067.stdout:5/847: rename d5/d28/daf/dbf/dcf/db2/fc1 to d5/d28/daf/dbf/dfc/d99/f12b 0 2024-04-22T23:23:36.084 INFO:tasks.workunit.client.0.smithi067.stdout:4/907: dwrite d12/d42/d4b/d77/d9d/dc1/f109 [0,4194304] 0 2024-04-22T23:23:36.086 INFO:tasks.workunit.client.0.smithi067.stdout:0/901: symlink db/d9a/d56/d70/dbf/l127 0 2024-04-22T23:23:36.086 INFO:tasks.workunit.client.0.smithi067.stdout:4/908: write d12/d17/d31/f50 [5386291,73443] 0 2024-04-22T23:23:36.088 INFO:tasks.workunit.client.0.smithi067.stdout:0/902: chown db/de/d11/d1a/c24 633 1 2024-04-22T23:23:36.089 INFO:tasks.workunit.client.0.smithi067.stdout:4/909: dread - d12/d5f/d69/f113 zero size 2024-04-22T23:23:36.091 INFO:tasks.workunit.client.0.smithi067.stdout:3/784: truncate d3/f6f 43201 0 2024-04-22T23:23:36.096 INFO:tasks.workunit.client.0.smithi067.stdout:0/903: symlink db/d21/d29/d75/d9d/l128 0 2024-04-22T23:23:36.098 INFO:tasks.workunit.client.0.smithi067.stdout:6/799: dread d17/d39/d53/d90/f8b [0,4194304] 0 2024-04-22T23:23:36.098 INFO:tasks.workunit.client.0.smithi067.stdout:4/910: creat d12/d5f/d7e/f130 x:0 0 0 2024-04-22T23:23:36.099 INFO:tasks.workunit.client.0.smithi067.stdout:6/800: readlink d17/d39/d53/d90/d46/d62/d88/lb3 0 2024-04-22T23:23:36.100 INFO:tasks.workunit.client.0.smithi067.stdout:1/735: link d8/d9/d2d/d4d/d48/fd0 d8/d9/d2f/d31/ff1 0 2024-04-22T23:23:36.101 INFO:tasks.workunit.client.0.smithi067.stdout:0/904: symlink db/de/d11/d92/l129 0 2024-04-22T23:23:36.106 INFO:tasks.workunit.client.0.smithi067.stdout:6/801: mknod d17/d39/d53/d90/d46/db0/db9/de7/c108 0 2024-04-22T23:23:36.108 INFO:tasks.workunit.client.1.smithi134.stdout:7/88: read - d4/fa zero size 2024-04-22T23:23:36.111 INFO:tasks.workunit.client.1.smithi134.stdout:2/62: unlink d2/f12 0 2024-04-22T23:23:36.113 INFO:tasks.workunit.client.1.smithi134.stdout:7/89: dread d4/f13 [0,4194304] 0 2024-04-22T23:23:36.113 INFO:tasks.workunit.client.0.smithi067.stdout:9/751: dwrite d5/f36 [0,4194304] 0 2024-04-22T23:23:36.113 INFO:tasks.workunit.client.1.smithi134.stdout:1/63: creat db/d15/d17/f1a x:0 0 0 2024-04-22T23:23:36.114 INFO:tasks.workunit.client.1.smithi134.stdout:1/64: write db/d15/f18 [808712,110743] 0 2024-04-22T23:23:36.115 INFO:tasks.workunit.client.1.smithi134.stdout:3/43: rename l7 to dc/dd/l14 0 2024-04-22T23:23:36.118 INFO:tasks.workunit.client.0.smithi067.stdout:4/911: creat d12/d17/d3f/f131 x:0 0 0 2024-04-22T23:23:36.119 INFO:tasks.workunit.client.1.smithi134.stdout:5/74: creat d9/d11/f1a x:0 0 0 2024-04-22T23:23:36.119 INFO:tasks.workunit.client.1.smithi134.stdout:5/75: read - d9/fe zero size 2024-04-22T23:23:36.119 INFO:tasks.workunit.client.0.smithi067.stdout:3/785: dread d3/d4/d28/d40/dfd/dfe/f2f [0,4194304] 0 2024-04-22T23:23:36.122 INFO:tasks.workunit.client.0.smithi067.stdout:9/752: dwrite d5/dd/d18/d39/d4d/f95 [4194304,4194304] 0 2024-04-22T23:23:36.122 INFO:tasks.workunit.client.1.smithi134.stdout:2/63: rename d2/f6 to d2/f17 0 2024-04-22T23:23:36.123 INFO:tasks.workunit.client.1.smithi134.stdout:4/42: truncate d4/f6 3202900 0 2024-04-22T23:23:36.124 INFO:tasks.workunit.client.1.smithi134.stdout:7/90: creat d4/d8/f22 x:0 0 0 2024-04-22T23:23:36.125 INFO:tasks.workunit.client.1.smithi134.stdout:7/91: fsync d4/d10/f1f 0 2024-04-22T23:23:36.125 INFO:tasks.workunit.client.1.smithi134.stdout:1/65: creat db/f1b x:0 0 0 2024-04-22T23:23:36.126 INFO:tasks.workunit.client.1.smithi134.stdout:1/66: chown db/d15/d17 0 1 2024-04-22T23:23:36.126 INFO:tasks.workunit.client.1.smithi134.stdout:3/44: mkdir dc/dd/d15 0 2024-04-22T23:23:36.128 INFO:tasks.workunit.client.0.smithi067.stdout:9/753: dread d5/f36 [0,4194304] 0 2024-04-22T23:23:36.128 INFO:tasks.workunit.client.0.smithi067.stdout:1/736: write d8/d9/d2f/f79 [2740578,95518] 0 2024-04-22T23:23:36.128 INFO:tasks.workunit.client.0.smithi067.stdout:4/912: mknod d12/d5f/c132 0 2024-04-22T23:23:36.129 INFO:tasks.workunit.client.1.smithi134.stdout:0/59: link d1/d5/f7 d1/f10 0 2024-04-22T23:23:36.129 INFO:tasks.workunit.client.1.smithi134.stdout:0/60: chown d1 0 1 2024-04-22T23:23:36.130 INFO:tasks.workunit.client.0.smithi067.stdout:2/741: sync 2024-04-22T23:23:36.130 INFO:tasks.workunit.client.0.smithi067.stdout:3/786: mkdir d3/d4/d28/d117 0 2024-04-22T23:23:36.130 INFO:tasks.workunit.client.0.smithi067.stdout:4/913: write d12/d17/f100 [1476337,69634] 0 2024-04-22T23:23:36.131 INFO:tasks.workunit.client.1.smithi134.stdout:5/76: rename f2 to d9/d11/f1b 0 2024-04-22T23:23:36.133 INFO:tasks.workunit.client.1.smithi134.stdout:2/64: creat d2/f18 x:0 0 0 2024-04-22T23:23:36.134 INFO:tasks.workunit.client.1.smithi134.stdout:2/65: dread - d2/dd/f13 zero size 2024-04-22T23:23:36.134 INFO:tasks.workunit.client.1.smithi134.stdout:2/66: readlink l1 0 2024-04-22T23:23:36.135 INFO:tasks.workunit.client.0.smithi067.stdout:9/754: mknod d5/d21/d47/d56/d8c/dd5/cfa 0 2024-04-22T23:23:36.137 INFO:tasks.workunit.client.1.smithi134.stdout:0/61: dread d1/f9 [0,4194304] 0 2024-04-22T23:23:36.137 INFO:tasks.workunit.client.0.smithi067.stdout:1/737: mknod d8/d9/d2f/d33/d66/d84/cf2 0 2024-04-22T23:23:36.137 INFO:tasks.workunit.client.1.smithi134.stdout:1/67: mknod db/d16/c1c 0 2024-04-22T23:23:36.138 INFO:tasks.workunit.client.0.smithi067.stdout:0/905: truncate db/de/f1b 6991986 0 2024-04-22T23:23:36.138 INFO:tasks.workunit.client.1.smithi134.stdout:0/62: truncate d1/f10 1635734 0 2024-04-22T23:23:36.138 INFO:tasks.workunit.client.1.smithi134.stdout:1/68: chown db/l11 28962627 1 2024-04-22T23:23:36.139 INFO:tasks.workunit.client.1.smithi134.stdout:3/45: creat dc/dd/d15/f16 x:0 0 0 2024-04-22T23:23:36.140 INFO:tasks.workunit.client.1.smithi134.stdout:2/67: dread d2/f14 [0,4194304] 0 2024-04-22T23:23:36.141 INFO:tasks.workunit.client.0.smithi067.stdout:2/742: rename d1/d12/le0 to d1/d10/d33/lf0 0 2024-04-22T23:23:36.141 INFO:tasks.workunit.client.1.smithi134.stdout:5/77: creat d9/df/f1c x:0 0 0 2024-04-22T23:23:36.144 INFO:tasks.workunit.client.0.smithi067.stdout:4/914: dwrite d12/d5f/fb5 [0,4194304] 0 2024-04-22T23:23:36.144 INFO:tasks.workunit.client.0.smithi067.stdout:1/738: write d8/d9/d2d/d4d/d83/dc5/fcd [505600,100536] 0 2024-04-22T23:23:36.150 INFO:tasks.workunit.client.0.smithi067.stdout:9/755: symlink d5/dd/d18/d5c/lfb 0 2024-04-22T23:23:36.150 INFO:tasks.workunit.client.0.smithi067.stdout:3/787: dwrite d3/d4/d10/d19/f34 [0,4194304] 0 2024-04-22T23:23:36.150 INFO:tasks.workunit.client.0.smithi067.stdout:1/739: write d8/d9/de0/d60/d62/fa8 [2826700,109332] 0 2024-04-22T23:23:36.151 INFO:tasks.workunit.client.1.smithi134.stdout:2/68: rename d2/dd/f13 to d2/d15/f19 0 2024-04-22T23:23:36.151 INFO:tasks.workunit.client.1.smithi134.stdout:5/78: unlink d9/df/f1c 0 2024-04-22T23:23:36.151 INFO:tasks.workunit.client.1.smithi134.stdout:7/92: getdents d4/d8 0 2024-04-22T23:23:36.152 INFO:tasks.workunit.client.0.smithi067.stdout:1/740: chown d8/d9/d2f/d33/d5c/dd9/cdd 171332337 1 2024-04-22T23:23:36.153 INFO:tasks.workunit.client.1.smithi134.stdout:4/43: dwrite d4/f6 [0,4194304] 0 2024-04-22T23:23:36.154 INFO:tasks.workunit.client.0.smithi067.stdout:2/743: creat d1/d19/d6f/ff1 x:0 0 0 2024-04-22T23:23:36.154 INFO:tasks.workunit.client.1.smithi134.stdout:7/93: dread - d4/d10/f1f zero size 2024-04-22T23:23:36.155 INFO:tasks.workunit.client.0.smithi067.stdout:4/915: creat d12/d5f/d7e/f133 x:0 0 0 2024-04-22T23:23:36.156 INFO:tasks.workunit.client.0.smithi067.stdout:4/916: readlink d12/l2c 0 2024-04-22T23:23:36.159 INFO:tasks.workunit.client.1.smithi134.stdout:0/63: mkdir d1/d5/dd/d11 0 2024-04-22T23:23:36.160 INFO:tasks.workunit.client.1.smithi134.stdout:3/46: creat dc/de/df/f17 x:0 0 0 2024-04-22T23:23:36.162 INFO:tasks.workunit.client.1.smithi134.stdout:4/44: chown d4/l5 2980 1 2024-04-22T23:23:36.165 INFO:tasks.workunit.client.0.smithi067.stdout:4/917: unlink d12/d5f/d2b/db6/f95 0 2024-04-22T23:23:36.167 INFO:tasks.workunit.client.1.smithi134.stdout:2/69: dwrite d2/dd/fe [8388608,4194304] 0 2024-04-22T23:23:36.168 INFO:tasks.workunit.client.0.smithi067.stdout:3/788: rename d3/d4/d28/d40/dfd/dfe/d4b/le8 to d3/dcb/l118 0 2024-04-22T23:23:36.171 INFO:tasks.workunit.client.0.smithi067.stdout:1/741: fdatasync d8/d4a/f5a 0 2024-04-22T23:23:36.172 INFO:tasks.workunit.client.1.smithi134.stdout:7/94: truncate d4/d10/f19 204538 0 2024-04-22T23:23:36.173 INFO:tasks.workunit.client.0.smithi067.stdout:9/756: dwrite d5/d21/d47/d57/fa2 [0,4194304] 0 2024-04-22T23:23:36.173 INFO:tasks.workunit.client.1.smithi134.stdout:5/79: dwrite d9/fa [0,4194304] 0 2024-04-22T23:23:36.173 INFO:tasks.workunit.client.1.smithi134.stdout:4/45: mkdir d4/d8 0 2024-04-22T23:23:36.174 INFO:tasks.workunit.client.1.smithi134.stdout:2/70: symlink d2/dd/l1a 0 2024-04-22T23:23:36.174 INFO:tasks.workunit.client.1.smithi134.stdout:7/95: stat d4/d8/d1d 0 2024-04-22T23:23:36.175 INFO:tasks.workunit.client.0.smithi067.stdout:0/906: write db/d9a/d56/d70/dba/fc0 [88732,118088] 0 2024-04-22T23:23:36.176 INFO:tasks.workunit.client.1.smithi134.stdout:2/71: chown d2/dd/df 796 1 2024-04-22T23:23:36.177 INFO:tasks.workunit.client.0.smithi067.stdout:9/757: stat d5/dd/d18/d39/d49/d54/f3d 0 2024-04-22T23:23:36.177 INFO:tasks.workunit.client.0.smithi067.stdout:2/744: dread d1/d10/d1c/d6d/d87/fd8 [0,4194304] 0 2024-04-22T23:23:36.178 INFO:tasks.workunit.client.1.smithi134.stdout:2/72: read - d2/d15/f19 zero size 2024-04-22T23:23:36.179 INFO:tasks.workunit.client.1.smithi134.stdout:3/47: dwrite fa [0,4194304] 0 2024-04-22T23:23:36.182 INFO:tasks.workunit.client.1.smithi134.stdout:3/48: write f3 [1421406,38286] 0 2024-04-22T23:23:36.183 INFO:tasks.workunit.client.1.smithi134.stdout:7/96: unlink d4/d10/f1b 0 2024-04-22T23:23:36.184 INFO:tasks.workunit.client.0.smithi067.stdout:6/802: truncate d17/d39/d53/d90/f94 1754935 0 2024-04-22T23:23:36.189 INFO:tasks.workunit.client.1.smithi134.stdout:4/46: dwrite d4/f6 [4194304,4194304] 0 2024-04-22T23:23:36.192 INFO:tasks.workunit.client.0.smithi067.stdout:3/789: symlink d3/d4/d10/d38/d53/l119 0 2024-04-22T23:23:36.194 INFO:tasks.workunit.client.0.smithi067.stdout:0/907: creat db/d71/f12a x:0 0 0 2024-04-22T23:23:36.195 INFO:tasks.workunit.client.0.smithi067.stdout:0/908: fdatasync db/d9a/d56/f6d 0 2024-04-22T23:23:36.197 INFO:tasks.workunit.client.0.smithi067.stdout:0/909: truncate db/d71/dda/f11b 1014111 0 2024-04-22T23:23:36.197 INFO:tasks.workunit.client.0.smithi067.stdout:4/918: rmdir d12/d17/d88/d12c 0 2024-04-22T23:23:36.198 INFO:tasks.workunit.client.1.smithi134.stdout:3/49: creat dc/f18 x:0 0 0 2024-04-22T23:23:36.199 INFO:tasks.workunit.client.1.smithi134.stdout:7/97: read d4/d10/f11 [1544455,55701] 0 2024-04-22T23:23:36.200 INFO:tasks.workunit.client.1.smithi134.stdout:0/64: write d1/f10 [1582772,117113] 0 2024-04-22T23:23:36.202 INFO:tasks.workunit.client.1.smithi134.stdout:4/47: symlink d4/d8/l9 0 2024-04-22T23:23:36.204 INFO:tasks.workunit.client.1.smithi134.stdout:4/48: write d4/f6 [5401399,70307] 0 2024-04-22T23:23:36.204 INFO:tasks.workunit.client.0.smithi067.stdout:4/919: symlink d12/d5f/d2b/d97/l134 0 2024-04-22T23:23:36.206 INFO:tasks.workunit.client.0.smithi067.stdout:1/742: getdents d8/d9/d2d/d4d/d48 0 2024-04-22T23:23:36.207 INFO:tasks.workunit.client.0.smithi067.stdout:6/803: mknod d17/d83/db5/c109 0 2024-04-22T23:23:36.208 INFO:tasks.workunit.client.1.smithi134.stdout:3/50: dwrite fa [0,4194304] 0 2024-04-22T23:23:36.208 INFO:tasks.workunit.client.1.smithi134.stdout:7/98: link d4/d8/fb d4/d10/f23 0 2024-04-22T23:23:36.209 INFO:tasks.workunit.client.0.smithi067.stdout:3/790: getdents d3/d4/d28/d4a 0 2024-04-22T23:23:36.210 INFO:tasks.workunit.client.1.smithi134.stdout:7/99: chown d4/d10/f11 28277760 1 2024-04-22T23:23:36.211 INFO:tasks.workunit.client.1.smithi134.stdout:3/51: mkdir dc/d19 0 2024-04-22T23:23:36.212 INFO:tasks.workunit.client.0.smithi067.stdout:4/920: getdents d12/d17 0 2024-04-22T23:23:36.212 INFO:tasks.workunit.client.0.smithi067.stdout:3/791: mkdir d3/d4/d10/d8a/d11a 0 2024-04-22T23:23:36.214 INFO:tasks.workunit.client.1.smithi134.stdout:4/49: dwrite d4/f6 [0,4194304] 0 2024-04-22T23:23:36.215 INFO:tasks.workunit.client.0.smithi067.stdout:6/804: creat d17/d39/d53/d90/d46/f10a x:0 0 0 2024-04-22T23:23:36.215 INFO:tasks.workunit.client.0.smithi067.stdout:3/792: chown d3/d4/d10/d38/d53/d55/ce1 600080 1 2024-04-22T23:23:36.216 INFO:tasks.workunit.client.1.smithi134.stdout:3/52: mknod dc/c1a 0 2024-04-22T23:23:36.216 INFO:tasks.workunit.client.1.smithi134.stdout:4/50: write d4/f6 [8215087,19610] 0 2024-04-22T23:23:36.217 INFO:tasks.workunit.client.0.smithi067.stdout:6/805: readlink d17/d39/d3a/da9/lc0 0 2024-04-22T23:23:36.217 INFO:tasks.workunit.client.1.smithi134.stdout:2/73: rmdir d2 39 2024-04-22T23:23:36.219 INFO:tasks.workunit.client.1.smithi134.stdout:4/51: rmdir d4/d8 39 2024-04-22T23:23:36.220 INFO:tasks.workunit.client.0.smithi067.stdout:2/745: write d1/d4f/f2a [1997227,53239] 0 2024-04-22T23:23:36.221 INFO:tasks.workunit.client.0.smithi067.stdout:3/793: mkdir d3/d4/d28/dc0/d11b 0 2024-04-22T23:23:36.222 INFO:tasks.workunit.client.1.smithi134.stdout:5/80: write d9/df/f14 [438121,116356] 0 2024-04-22T23:23:36.222 INFO:tasks.workunit.client.1.smithi134.stdout:2/74: dread - d2/d15/f19 zero size 2024-04-22T23:23:36.223 INFO:tasks.workunit.client.1.smithi134.stdout:5/81: write f6 [348603,48778] 0 2024-04-22T23:23:36.224 INFO:tasks.workunit.client.0.smithi067.stdout:1/743: link d8/d9/l5b d8/d9/de0/d5d/dd4/lf3 0 2024-04-22T23:23:36.224 INFO:tasks.workunit.client.1.smithi134.stdout:4/52: chown d4/d8/l9 19 1 2024-04-22T23:23:36.230 INFO:tasks.workunit.client.1.smithi134.stdout:3/53: getdents dc/dd/d15 0 2024-04-22T23:23:36.231 INFO:tasks.workunit.client.0.smithi067.stdout:4/921: dread d12/d5f/dae/fc6 [0,4194304] 0 2024-04-22T23:23:36.233 INFO:tasks.workunit.client.0.smithi067.stdout:1/744: rename d8/d9/de0/f7e to d8/d9/d2f/d33/d66/ff4 0 2024-04-22T23:23:36.233 INFO:tasks.workunit.client.1.smithi134.stdout:5/82: link f6 d9/dd/d17/f1d 0 2024-04-22T23:23:36.235 INFO:tasks.workunit.client.1.smithi134.stdout:4/53: dwrite d4/f6 [4194304,4194304] 0 2024-04-22T23:23:36.235 INFO:tasks.workunit.client.0.smithi067.stdout:6/806: dwrite d17/f7d [0,4194304] 0 2024-04-22T23:23:36.236 INFO:tasks.workunit.client.1.smithi134.stdout:5/83: dread - d9/dd/f10 zero size 2024-04-22T23:23:36.237 INFO:tasks.workunit.client.0.smithi067.stdout:2/746: creat d1/d12/ff2 x:0 0 0 2024-04-22T23:23:36.237 INFO:tasks.workunit.client.1.smithi134.stdout:4/54: readlink d4/l5 0 2024-04-22T23:23:36.238 INFO:tasks.workunit.client.1.smithi134.stdout:2/75: dwrite d2/d15/f19 [0,4194304] 0 2024-04-22T23:23:36.238 INFO:tasks.workunit.client.0.smithi067.stdout:4/922: mkdir d12/d42/d4b/d122/d8e/d135 0 2024-04-22T23:23:36.240 INFO:tasks.workunit.client.0.smithi067.stdout:3/794: getdents d3/d4/d28/d40/dfd/dfe/d4b 0 2024-04-22T23:23:36.244 INFO:tasks.workunit.client.1.smithi134.stdout:2/76: mkdir d2/dd/df/d1b 0 2024-04-22T23:23:36.245 INFO:tasks.workunit.client.0.smithi067.stdout:0/910: dwrite db/f79 [0,4194304] 0 2024-04-22T23:23:36.245 INFO:tasks.workunit.client.0.smithi067.stdout:3/795: read - d3/d4/d10/d8a/f106 zero size 2024-04-22T23:23:36.247 INFO:tasks.workunit.client.1.smithi134.stdout:2/77: chown d2/l3 371977550 1 2024-04-22T23:23:36.247 INFO:tasks.workunit.client.0.smithi067.stdout:4/923: chown d12/d17/d3f/d84/f103 848 1 2024-04-22T23:23:36.248 INFO:tasks.workunit.client.1.smithi134.stdout:4/55: rename d4/d8 to d4/d7/da 0 2024-04-22T23:23:36.251 INFO:tasks.workunit.client.1.smithi134.stdout:5/84: dwrite d9/d11/f1b [0,4194304] 0 2024-04-22T23:23:36.252 INFO:tasks.workunit.client.0.smithi067.stdout:6/807: rename d17/d39/d3a/d8d/ca8 to d17/d39/d53/d90/d46/db0/db9/de7/c10b 0 2024-04-22T23:23:36.252 INFO:tasks.workunit.client.1.smithi134.stdout:2/78: mkdir d2/dd/df/d1c 0 2024-04-22T23:23:36.253 INFO:tasks.workunit.client.0.smithi067.stdout:0/911: dread db/d71/dda/ff2 [4194304,4194304] 0 2024-04-22T23:23:36.256 INFO:tasks.workunit.client.0.smithi067.stdout:4/924: creat d12/d5f/d2b/d12a/f136 x:0 0 0 2024-04-22T23:23:36.256 INFO:tasks.workunit.client.1.smithi134.stdout:4/56: mknod d4/d7/da/cb 0 2024-04-22T23:23:36.256 INFO:tasks.workunit.client.1.smithi134.stdout:5/85: write f8 [2724163,30797] 0 2024-04-22T23:23:36.257 INFO:tasks.workunit.client.0.smithi067.stdout:6/808: mkdir d17/d83/db5/dbc/dd8/d10c 0 2024-04-22T23:23:36.258 INFO:tasks.workunit.client.1.smithi134.stdout:4/57: mknod d4/d7/da/cc 0 2024-04-22T23:23:36.259 INFO:tasks.workunit.client.1.smithi134.stdout:2/79: mkdir d2/dd/df/d1b/d1d 0 2024-04-22T23:23:36.259 INFO:tasks.workunit.client.1.smithi134.stdout:5/86: truncate d9/f12 740341 0 2024-04-22T23:23:36.259 INFO:tasks.workunit.client.0.smithi067.stdout:0/912: creat db/d21/d29/d75/f12b x:0 0 0 2024-04-22T23:23:36.260 INFO:tasks.workunit.client.0.smithi067.stdout:0/913: chown db/d60/ca6 14801749 1 2024-04-22T23:23:36.260 INFO:tasks.workunit.client.1.smithi134.stdout:6/92: sync 2024-04-22T23:23:36.261 INFO:tasks.workunit.client.0.smithi067.stdout:6/809: mknod d17/d39/d53/d90/d46/db0/db9/c10d 0 2024-04-22T23:23:36.261 INFO:tasks.workunit.client.0.smithi067.stdout:4/925: chown d12/d42/d4b/d122/d8e/d56/d114/fa7 8136 1 2024-04-22T23:23:36.262 INFO:tasks.workunit.client.1.smithi134.stdout:5/87: mkdir d9/d1e 0 2024-04-22T23:23:36.264 INFO:tasks.workunit.client.1.smithi134.stdout:2/80: write d2/f17 [472588,85094] 0 2024-04-22T23:23:36.265 INFO:tasks.workunit.client.1.smithi134.stdout:6/93: truncate d0/d19/de/f1d 192833 0 2024-04-22T23:23:36.267 INFO:tasks.workunit.client.1.smithi134.stdout:6/94: chown d0/d19/de/d1f 1446 1 2024-04-22T23:23:36.269 INFO:tasks.workunit.client.1.smithi134.stdout:6/95: mknod d0/c21 0 2024-04-22T23:23:36.269 INFO:tasks.workunit.client.1.smithi134.stdout:6/96: stat d0/d19/f9 0 2024-04-22T23:23:36.270 INFO:tasks.workunit.client.1.smithi134.stdout:2/81: link d2/d15/f19 d2/dd/df/d1b/f1e 0 2024-04-22T23:23:36.271 INFO:tasks.workunit.client.0.smithi067.stdout:1/745: dread d8/d9/d2f/d33/d5c/f65 [0,4194304] 0 2024-04-22T23:23:36.271 INFO:tasks.workunit.client.0.smithi067.stdout:4/926: link d12/d17/f33 d12/d17/d3f/d101/f137 0 2024-04-22T23:23:36.271 INFO:tasks.workunit.client.0.smithi067.stdout:0/914: dwrite db/d21/d29/fa7 [0,4194304] 0 2024-04-22T23:23:36.273 INFO:tasks.workunit.client.1.smithi134.stdout:6/97: rename d0/f1e to d0/d19/de/d1f/f22 0 2024-04-22T23:23:36.274 INFO:tasks.workunit.client.0.smithi067.stdout:4/927: creat d12/d17/d3f/d7a/f138 x:0 0 0 2024-04-22T23:23:36.274 INFO:tasks.workunit.client.1.smithi134.stdout:2/82: mknod d2/dd/df/d1b/d1d/c1f 0 2024-04-22T23:23:36.275 INFO:tasks.workunit.client.1.smithi134.stdout:2/83: fdatasync d2/dd/fe 0 2024-04-22T23:23:36.275 INFO:tasks.workunit.client.1.smithi134.stdout:6/98: fdatasync d0/d19/de/f18 0 2024-04-22T23:23:36.275 INFO:tasks.workunit.client.0.smithi067.stdout:4/928: dread - d12/d5f/d2b/d97/f10b zero size 2024-04-22T23:23:36.276 INFO:tasks.workunit.client.0.smithi067.stdout:1/746: unlink d8/d9/d2f/d33/d5c/f6d 0 2024-04-22T23:23:36.277 INFO:tasks.workunit.client.1.smithi134.stdout:2/84: unlink d2/l8 0 2024-04-22T23:23:36.278 INFO:tasks.workunit.client.0.smithi067.stdout:4/929: readlink d12/d42/d4b/d122/d8e/d56/d114/l8f 0 2024-04-22T23:23:36.280 INFO:tasks.workunit.client.1.smithi134.stdout:2/85: symlink d2/dd/df/d1b/d1d/l20 0 2024-04-22T23:23:36.281 INFO:tasks.workunit.client.1.smithi134.stdout:4/58: fsync d4/f6 0 2024-04-22T23:23:36.281 INFO:tasks.workunit.client.1.smithi134.stdout:2/86: write d2/f18 [427275,79553] 0 2024-04-22T23:23:36.282 INFO:tasks.workunit.client.1.smithi134.stdout:4/59: read d4/f6 [7324104,31591] 0 2024-04-22T23:23:36.283 INFO:tasks.workunit.client.1.smithi134.stdout:4/60: write d4/f6 [1159061,129011] 0 2024-04-22T23:23:36.284 INFO:tasks.workunit.client.1.smithi134.stdout:2/87: getdents d2/dd/df/d1c 0 2024-04-22T23:23:36.288 INFO:tasks.workunit.client.1.smithi134.stdout:2/88: dread d2/dd/df/d1b/f1e [0,4194304] 0 2024-04-22T23:23:36.289 INFO:tasks.workunit.client.0.smithi067.stdout:1/747: creat d8/d4a/ff5 x:0 0 0 2024-04-22T23:23:36.289 INFO:tasks.workunit.client.0.smithi067.stdout:0/915: getdents db/d71 0 2024-04-22T23:23:36.291 INFO:tasks.workunit.client.1.smithi134.stdout:2/89: creat d2/dd/df/d1c/f21 x:0 0 0 2024-04-22T23:23:36.292 INFO:tasks.workunit.client.0.smithi067.stdout:2/747: dread d1/d10/d1c/d6d/f4c [0,4194304] 0 2024-04-22T23:23:36.294 INFO:tasks.workunit.client.1.smithi134.stdout:2/90: creat d2/f22 x:0 0 0 2024-04-22T23:23:36.295 INFO:tasks.workunit.client.0.smithi067.stdout:8/767: sync 2024-04-22T23:23:36.296 INFO:tasks.workunit.client.1.smithi134.stdout:2/91: write d2/dd/df/d1c/f21 [430001,70720] 0 2024-04-22T23:23:36.297 INFO:tasks.workunit.client.1.smithi134.stdout:4/61: rename d4/d7/da to d4/dd 0 2024-04-22T23:23:36.300 INFO:tasks.workunit.client.1.smithi134.stdout:5/88: getdents d9 0 2024-04-22T23:23:36.301 INFO:tasks.workunit.client.1.smithi134.stdout:5/89: mknod d9/d11/c1f 0 2024-04-22T23:23:36.302 INFO:tasks.workunit.client.1.smithi134.stdout:5/90: truncate d9/f15 749176 0 2024-04-22T23:23:36.302 INFO:tasks.workunit.client.0.smithi067.stdout:4/930: dwrite d12/d42/d4b/d77/fee [0,4194304] 0 2024-04-22T23:23:36.303 INFO:tasks.workunit.client.0.smithi067.stdout:8/768: dread d1/fc0 [0,4194304] 0 2024-04-22T23:23:36.305 INFO:tasks.workunit.client.1.smithi134.stdout:5/91: dread d9/fa [0,4194304] 0 2024-04-22T23:23:36.306 INFO:tasks.workunit.client.1.smithi134.stdout:5/92: read - d9/dd/f10 zero size 2024-04-22T23:23:36.308 INFO:tasks.workunit.client.1.smithi134.stdout:5/93: mkdir d9/df/d20 0 2024-04-22T23:23:36.310 INFO:tasks.workunit.client.0.smithi067.stdout:2/748: creat d1/d10/d72/d68/da1/ff3 x:0 0 0 2024-04-22T23:23:36.310 INFO:tasks.workunit.client.0.smithi067.stdout:2/749: chown d1/d10/d72 1481523 1 2024-04-22T23:23:36.312 INFO:tasks.workunit.client.1.smithi134.stdout:5/94: link d9/dd/d17/f1d d9/d11/f21 0 2024-04-22T23:23:36.312 INFO:tasks.workunit.client.1.smithi134.stdout:5/95: fdatasync d9/f19 0 2024-04-22T23:23:36.312 INFO:tasks.workunit.client.0.smithi067.stdout:2/750: chown d1/d12/f43 28 1 2024-04-22T23:23:36.314 INFO:tasks.workunit.client.1.smithi134.stdout:5/96: creat d9/f22 x:0 0 0 2024-04-22T23:23:36.314 INFO:tasks.workunit.client.1.smithi134.stdout:5/97: fdatasync d9/f16 0 2024-04-22T23:23:36.314 INFO:tasks.workunit.client.1.smithi134.stdout:5/98: chown d9/d11 987801561 1 2024-04-22T23:23:36.315 INFO:tasks.workunit.client.0.smithi067.stdout:1/748: dread d8/d9/d2f/d33/f3e [0,4194304] 0 2024-04-22T23:23:36.316 INFO:tasks.workunit.client.1.smithi134.stdout:5/99: creat d9/d1e/f23 x:0 0 0 2024-04-22T23:23:36.316 INFO:tasks.workunit.client.1.smithi134.stdout:5/100: chown d9/d1e/f23 3041767 1 2024-04-22T23:23:36.317 INFO:tasks.workunit.client.0.smithi067.stdout:0/916: rename db/d60/d9e/d101/d108 to db/d21/d29/d75/d9d/dde/d12c 0 2024-04-22T23:23:36.317 INFO:tasks.workunit.client.1.smithi134.stdout:5/101: write d9/f12 [987539,105829] 0 2024-04-22T23:23:36.318 INFO:tasks.workunit.client.1.smithi134.stdout:5/102: chown d9/fc 25024 1 2024-04-22T23:23:36.319 INFO:tasks.workunit.client.0.smithi067.stdout:2/751: creat d1/d10/d1c/ff4 x:0 0 0 2024-04-22T23:23:36.321 INFO:tasks.workunit.client.0.smithi067.stdout:0/917: getdents db/d9a/d56/d6b 0 2024-04-22T23:23:36.321 INFO:tasks.workunit.client.0.smithi067.stdout:0/918: read - db/d21/f119 zero size 2024-04-22T23:23:36.322 INFO:tasks.workunit.client.0.smithi067.stdout:2/752: mknod d1/cf5 0 2024-04-22T23:23:36.323 INFO:tasks.workunit.client.0.smithi067.stdout:0/919: creat db/d9a/d56/d70/dbf/f12d x:0 0 0 2024-04-22T23:23:36.324 INFO:tasks.workunit.client.1.smithi134.stdout:5/103: dwrite d9/f22 [0,4194304] 0 2024-04-22T23:23:36.325 INFO:tasks.workunit.client.0.smithi067.stdout:0/920: write db/d21/d29/d75/fed [1977973,106991] 0 2024-04-22T23:23:36.330 INFO:tasks.workunit.client.1.smithi134.stdout:5/104: dwrite d9/d1e/f23 [0,4194304] 0 2024-04-22T23:23:36.336 INFO:tasks.workunit.client.1.smithi134.stdout:5/105: dread d9/d1e/f23 [0,4194304] 0 2024-04-22T23:23:36.347 INFO:tasks.workunit.client.1.smithi134.stdout:5/106: dread d9/dd/d17/f1d [0,4194304] 0 2024-04-22T23:23:36.349 INFO:tasks.workunit.client.1.smithi134.stdout:5/107: symlink d9/df/l24 0 2024-04-22T23:23:36.349 INFO:tasks.workunit.client.1.smithi134.stdout:5/108: chown d9/d11 33 1 2024-04-22T23:23:36.349 INFO:tasks.workunit.client.0.smithi067.stdout:3/796: dwrite d3/d4/d28/d5f/fa9 [0,4194304] 0 2024-04-22T23:23:36.351 INFO:tasks.workunit.client.1.smithi134.stdout:5/109: creat d9/d11/f25 x:0 0 0 2024-04-22T23:23:36.351 INFO:tasks.workunit.client.1.smithi134.stdout:5/110: chown d9 7505 1 2024-04-22T23:23:36.352 INFO:tasks.workunit.client.1.smithi134.stdout:5/111: creat d9/d1e/f26 x:0 0 0 2024-04-22T23:23:36.354 INFO:tasks.workunit.client.0.smithi067.stdout:3/797: creat d3/d4/f11c x:0 0 0 2024-04-22T23:23:36.355 INFO:tasks.workunit.client.0.smithi067.stdout:3/798: rename d3/d4/d28 to d3/d4/d28/d40/dfd/d11d 22 2024-04-22T23:23:36.356 INFO:tasks.workunit.client.1.smithi134.stdout:5/112: dread f6 [0,4194304] 0 2024-04-22T23:23:36.357 INFO:tasks.workunit.client.0.smithi067.stdout:3/799: read d3/d4/d28/d4a/f73 [420281,81620] 0 2024-04-22T23:23:36.358 INFO:tasks.workunit.client.1.smithi134.stdout:5/113: symlink d9/df/d20/l27 0 2024-04-22T23:23:36.364 INFO:tasks.workunit.client.1.smithi134.stdout:6/99: write d0/d19/f9 [919552,108149] 0 2024-04-22T23:23:36.370 INFO:tasks.workunit.client.1.smithi134.stdout:4/62: rmdir d4/dd 39 2024-04-22T23:23:36.371 INFO:tasks.workunit.client.1.smithi134.stdout:6/100: link d0/d19/db/c17 d0/d19/de/c23 0 2024-04-22T23:23:36.373 INFO:tasks.workunit.client.1.smithi134.stdout:6/101: truncate d0/d19/f9 1230327 0 2024-04-22T23:23:36.374 INFO:tasks.workunit.client.1.smithi134.stdout:4/63: dread d4/f6 [0,4194304] 0 2024-04-22T23:23:36.375 INFO:tasks.workunit.client.0.smithi067.stdout:1/749: dread d8/d9/f1b [0,4194304] 0 2024-04-22T23:23:36.376 INFO:tasks.workunit.client.0.smithi067.stdout:7/717: sync 2024-04-22T23:23:36.376 INFO:tasks.workunit.client.0.smithi067.stdout:5/848: sync 2024-04-22T23:23:36.377 INFO:tasks.workunit.client.0.smithi067.stdout:4/931: truncate d12/d5f/d69/fea 852029 0 2024-04-22T23:23:36.377 INFO:tasks.workunit.client.0.smithi067.stdout:3/800: dread d3/d4/d10/d19/f3b [0,4194304] 0 2024-04-22T23:23:36.377 INFO:tasks.workunit.client.1.smithi134.stdout:6/102: getdents d0/d19/de/d12 0 2024-04-22T23:23:36.379 INFO:tasks.workunit.client.1.smithi134.stdout:4/64: creat d4/fe x:0 0 0 2024-04-22T23:23:36.379 INFO:tasks.workunit.client.0.smithi067.stdout:1/750: mknod d8/d9/de0/cf6 0 2024-04-22T23:23:36.379 INFO:tasks.workunit.client.1.smithi134.stdout:6/103: mkdir d0/d19/de/d1f/d24 0 2024-04-22T23:23:36.381 INFO:tasks.workunit.client.0.smithi067.stdout:4/932: truncate d12/f38 3081145 0 2024-04-22T23:23:36.381 INFO:tasks.workunit.client.1.smithi134.stdout:6/104: mknod d0/d19/de/c25 0 2024-04-22T23:23:36.382 INFO:tasks.workunit.client.0.smithi067.stdout:4/933: write d12/d42/d4b/d77/d9d/dc1/f104 [840458,11037] 0 2024-04-22T23:23:36.382 INFO:tasks.workunit.client.1.smithi134.stdout:5/114: rmdir d9 39 2024-04-22T23:23:36.383 INFO:tasks.workunit.client.0.smithi067.stdout:5/849: rename d5/d28/d36/d40/f85 to d5/db/d11f/f12c 0 2024-04-22T23:23:36.384 INFO:tasks.workunit.client.0.smithi067.stdout:1/751: unlink d8/d12/d40/f69 0 2024-04-22T23:23:36.386 INFO:tasks.workunit.client.0.smithi067.stdout:1/752: write d8/d9/d2f/d33/d5c/f98 [294298,86387] 0 2024-04-22T23:23:36.387 INFO:tasks.workunit.client.0.smithi067.stdout:7/718: mknod d10/d14/d1d/d46/d6a/df1/c108 0 2024-04-22T23:23:36.388 INFO:tasks.workunit.client.0.smithi067.stdout:1/753: dread d8/d9/fa9 [0,4194304] 0 2024-04-22T23:23:36.389 INFO:tasks.workunit.client.1.smithi134.stdout:4/65: dwrite d4/f6 [4194304,4194304] 0 2024-04-22T23:23:36.389 INFO:tasks.workunit.client.1.smithi134.stdout:6/105: dread d0/d19/f1c [0,4194304] 0 2024-04-22T23:23:36.390 INFO:tasks.workunit.client.1.smithi134.stdout:5/115: unlink d9/df/f14 0 2024-04-22T23:23:36.391 INFO:tasks.workunit.client.1.smithi134.stdout:5/116: fsync d9/f22 0 2024-04-22T23:23:36.394 INFO:tasks.workunit.client.1.smithi134.stdout:6/106: creat d0/f26 x:0 0 0 2024-04-22T23:23:36.395 INFO:tasks.workunit.client.0.smithi067.stdout:4/934: dread d12/d5f/d7e/ff1 [0,4194304] 0 2024-04-22T23:23:36.396 INFO:tasks.workunit.client.1.smithi134.stdout:5/117: mknod d9/c28 0 2024-04-22T23:23:36.396 INFO:tasks.workunit.client.1.smithi134.stdout:5/118: stat d9/f16 0 2024-04-22T23:23:36.396 INFO:tasks.workunit.client.0.smithi067.stdout:1/754: unlink d8/d9/de0/d5d/cef 0 2024-04-22T23:23:36.397 INFO:tasks.workunit.client.1.smithi134.stdout:4/66: mknod d4/dd/cf 0 2024-04-22T23:23:36.398 INFO:tasks.workunit.client.1.smithi134.stdout:6/107: creat d0/f27 x:0 0 0 2024-04-22T23:23:36.398 INFO:tasks.workunit.client.0.smithi067.stdout:8/769: dwrite d1/d41/f67 [0,4194304] 0 2024-04-22T23:23:36.399 INFO:tasks.workunit.client.0.smithi067.stdout:5/850: creat d5/d1a/d103/d4d/d101/f12d x:0 0 0 2024-04-22T23:23:36.401 INFO:tasks.workunit.client.0.smithi067.stdout:8/770: fdatasync d1/dd/d26/d62/fc8 0 2024-04-22T23:23:36.404 INFO:tasks.workunit.client.0.smithi067.stdout:9/758: sync 2024-04-22T23:23:36.405 INFO:tasks.workunit.client.1.smithi134.stdout:5/119: dwrite d9/d11/f25 [0,4194304] 0 2024-04-22T23:23:36.405 INFO:tasks.workunit.client.0.smithi067.stdout:5/851: fdatasync d5/db/d12/dbb/d5f/d79/dba/de0/f109 0 2024-04-22T23:23:36.406 INFO:tasks.workunit.client.0.smithi067.stdout:4/935: mkdir d12/d42/d4b/d77/d8c/d139 0 2024-04-22T23:23:36.406 INFO:tasks.workunit.client.1.smithi134.stdout:4/67: dwrite d4/f6 [0,4194304] 0 2024-04-22T23:23:36.407 INFO:tasks.workunit.client.1.smithi134.stdout:5/120: write d9/d11/f1a [193114,35654] 0 2024-04-22T23:23:36.410 INFO:tasks.workunit.client.0.smithi067.stdout:3/801: getdents d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d90/da2/daf/dd8 0 2024-04-22T23:23:36.410 INFO:tasks.workunit.client.1.smithi134.stdout:5/121: mkdir d9/df/d20/d29 0 2024-04-22T23:23:36.411 INFO:tasks.workunit.client.0.smithi067.stdout:1/755: read d8/d12/d36/f72 [210510,23543] 0 2024-04-22T23:23:36.412 INFO:tasks.workunit.client.0.smithi067.stdout:7/719: creat d10/d14/f109 x:0 0 0 2024-04-22T23:23:36.415 INFO:tasks.workunit.client.1.smithi134.stdout:5/122: mkdir d9/dd/d17/d2a 0 2024-04-22T23:23:36.415 INFO:tasks.workunit.client.0.smithi067.stdout:2/753: dwrite d1/d10/d72/d53/fae [0,4194304] 0 2024-04-22T23:23:36.415 INFO:tasks.workunit.client.1.smithi134.stdout:4/68: dwrite d4/fe [0,4194304] 0 2024-04-22T23:23:36.418 INFO:tasks.workunit.client.0.smithi067.stdout:1/756: dread d8/d9/fa9 [0,4194304] 0 2024-04-22T23:23:36.419 INFO:tasks.workunit.client.1.smithi134.stdout:5/123: rename d9/dd/d17/f1d to d9/f2b 0 2024-04-22T23:23:36.420 INFO:tasks.workunit.client.1.smithi134.stdout:5/124: write d9/d11/f1a [885485,59553] 0 2024-04-22T23:23:36.422 INFO:tasks.workunit.client.0.smithi067.stdout:4/936: mkdir d12/d42/d4b/d77/d9d/d13a 0 2024-04-22T23:23:36.422 INFO:tasks.workunit.client.1.smithi134.stdout:5/125: mknod d9/d1e/c2c 0 2024-04-22T23:23:36.423 INFO:tasks.workunit.client.0.smithi067.stdout:8/771: truncate d1/dd/d26/fd6 633045 0 2024-04-22T23:23:36.424 INFO:tasks.workunit.client.1.smithi134.stdout:5/126: mknod d9/d1e/c2d 0 2024-04-22T23:23:36.425 INFO:tasks.workunit.client.1.smithi134.stdout:5/127: readlink d9/df/l24 0 2024-04-22T23:23:36.425 INFO:tasks.workunit.client.1.smithi134.stdout:6/108: write d0/d19/de/f18 [482704,30160] 0 2024-04-22T23:23:36.430 INFO:tasks.workunit.client.1.smithi134.stdout:1/69: sync 2024-04-22T23:23:36.430 INFO:tasks.workunit.client.1.smithi134.stdout:9/33: sync 2024-04-22T23:23:36.430 INFO:tasks.workunit.client.1.smithi134.stdout:8/32: sync 2024-04-22T23:23:36.430 INFO:tasks.workunit.client.0.smithi067.stdout:9/759: rename d5/dd/f1a to d5/dd/d18/d39/d8f/ffc 0 2024-04-22T23:23:36.431 INFO:tasks.workunit.client.1.smithi134.stdout:9/34: chown d0/f5 109612260 1 2024-04-22T23:23:36.433 INFO:tasks.workunit.client.0.smithi067.stdout:4/937: mkdir d12/d42/dd3/d13b 0 2024-04-22T23:23:36.434 INFO:tasks.workunit.client.0.smithi067.stdout:3/802: mknod d3/d4/d10/d1f/d3c/c11e 0 2024-04-22T23:23:36.434 INFO:tasks.workunit.client.0.smithi067.stdout:0/921: dwrite db/d21/d29/d75/d9d/dde/d10f/dd9/fe1 [0,4194304] 0 2024-04-22T23:23:36.434 INFO:tasks.workunit.client.1.smithi134.stdout:6/109: unlink d0/d19/de/f11 0 2024-04-22T23:23:36.436 INFO:tasks.workunit.client.0.smithi067.stdout:5/852: creat d5/d28/daf/dbf/dcf/db2/f12e x:0 0 0 2024-04-22T23:23:36.436 INFO:tasks.workunit.client.1.smithi134.stdout:9/35: creat d0/fa x:0 0 0 2024-04-22T23:23:36.437 INFO:tasks.workunit.client.0.smithi067.stdout:6/810: dwrite d17/d39/d53/d90/f94 [0,4194304] 0 2024-04-22T23:23:36.438 INFO:tasks.workunit.client.1.smithi134.stdout:6/110: chown d0/d19/de/f13 3 1 2024-04-22T23:23:36.443 INFO:tasks.workunit.client.0.smithi067.stdout:8/772: symlink d1/d41/d46/lf8 0 2024-04-22T23:23:36.444 INFO:tasks.workunit.client.1.smithi134.stdout:9/36: rename d0/f5 to d0/d7/fb 0 2024-04-22T23:23:36.444 INFO:tasks.workunit.client.0.smithi067.stdout:1/757: dread d8/d9/d2f/d33/d66/f9b [0,4194304] 0 2024-04-22T23:23:36.445 INFO:tasks.workunit.client.1.smithi134.stdout:6/111: rmdir d0/d19/de/d1f 39 2024-04-22T23:23:36.445 INFO:tasks.workunit.client.1.smithi134.stdout:5/128: dwrite d9/d11/f25 [0,4194304] 0 2024-04-22T23:23:36.446 INFO:tasks.workunit.client.1.smithi134.stdout:8/33: dwrite d7/f9 [0,4194304] 0 2024-04-22T23:23:36.446 INFO:tasks.workunit.client.0.smithi067.stdout:3/803: creat d3/d4/d10/d38/d53/d55/d5c/d70/f11f x:0 0 0 2024-04-22T23:23:36.447 INFO:tasks.workunit.client.0.smithi067.stdout:4/938: symlink d12/d42/d4b/d122/l13c 0 2024-04-22T23:23:36.448 INFO:tasks.workunit.client.0.smithi067.stdout:4/939: chown d12/d17/d31/f67 487 1 2024-04-22T23:23:36.449 INFO:tasks.workunit.client.1.smithi134.stdout:6/112: chown d0/l20 22633825 1 2024-04-22T23:23:36.451 INFO:tasks.workunit.client.0.smithi067.stdout:0/922: dread db/de/f1c [0,4194304] 0 2024-04-22T23:23:36.452 INFO:tasks.workunit.client.0.smithi067.stdout:2/754: link d1/d10/d41/c85 d1/d42/d61/db9/cf6 0 2024-04-22T23:23:36.455 INFO:tasks.workunit.client.0.smithi067.stdout:3/804: symlink d3/d4/d10/d38/d53/d55/d5c/l120 0 2024-04-22T23:23:36.456 INFO:tasks.workunit.client.1.smithi134.stdout:6/113: symlink d0/d19/de/d1f/l28 0 2024-04-22T23:23:36.458 INFO:tasks.workunit.client.0.smithi067.stdout:6/811: mknod d17/dae/c10e 0 2024-04-22T23:23:36.460 INFO:tasks.workunit.client.1.smithi134.stdout:9/37: dwrite d0/d7/f9 [4194304,4194304] 0 2024-04-22T23:23:36.463 INFO:tasks.workunit.client.0.smithi067.stdout:6/812: readlink d17/d39/d3a/l5b 0 2024-04-22T23:23:36.463 INFO:tasks.workunit.client.0.smithi067.stdout:7/720: link d10/de3/cfd d10/d14/d1d/d46/d6a/dcc/c10a 0 2024-04-22T23:23:36.463 INFO:tasks.workunit.client.1.smithi134.stdout:9/38: mkdir d0/d7/dc 0 2024-04-22T23:23:36.464 INFO:tasks.workunit.client.1.smithi134.stdout:9/39: chown d0/d7/dc 11031701 1 2024-04-22T23:23:36.465 INFO:tasks.workunit.client.1.smithi134.stdout:9/40: write d0/fa [634746,125717] 0 2024-04-22T23:23:36.466 INFO:tasks.workunit.client.1.smithi134.stdout:9/41: chown d0/f6 0 1 2024-04-22T23:23:36.467 INFO:tasks.workunit.client.0.smithi067.stdout:1/758: symlink d8/d9/lf7 0 2024-04-22T23:23:36.471 INFO:tasks.workunit.client.0.smithi067.stdout:3/805: dwrite d3/d4/d10/d38/d53/d55/d74/d7f/ff2 [0,4194304] 0 2024-04-22T23:23:36.471 INFO:tasks.workunit.client.0.smithi067.stdout:4/940: mkdir d12/d17/d3f/d13d 0 2024-04-22T23:23:36.472 INFO:tasks.workunit.client.1.smithi134.stdout:4/69: fsync d4/f6 0 2024-04-22T23:23:36.472 INFO:tasks.workunit.client.0.smithi067.stdout:8/773: dread d1/d3/d10/d3b/f5e [0,4194304] 0 2024-04-22T23:23:36.473 INFO:tasks.workunit.client.1.smithi134.stdout:4/70: write d4/f6 [8405225,79413] 0 2024-04-22T23:23:36.473 INFO:tasks.workunit.client.0.smithi067.stdout:0/923: creat db/de/d11/d1a/da4/dec/d105/f12e x:0 0 0 2024-04-22T23:23:36.475 INFO:tasks.workunit.client.0.smithi067.stdout:8/774: chown d1/dd/d26/d62/d94/fc6 2786453 1 2024-04-22T23:23:36.476 INFO:tasks.workunit.client.0.smithi067.stdout:1/759: dread d8/d12/fd6 [0,4194304] 0 2024-04-22T23:23:36.476 INFO:tasks.workunit.client.1.smithi134.stdout:4/71: creat d4/d7/f10 x:0 0 0 2024-04-22T23:23:36.477 INFO:tasks.workunit.client.1.smithi134.stdout:8/34: write d7/f9 [5157040,66902] 0 2024-04-22T23:23:36.478 INFO:tasks.workunit.client.1.smithi134.stdout:8/35: stat d7/l8 0 2024-04-22T23:23:36.482 INFO:tasks.workunit.client.1.smithi134.stdout:8/36: unlink c3 0 2024-04-22T23:23:36.483 INFO:tasks.workunit.client.0.smithi067.stdout:6/813: creat d17/d39/d53/d90/d46/db0/db9/de7/f10f x:0 0 0 2024-04-22T23:23:36.483 INFO:tasks.workunit.client.1.smithi134.stdout:4/72: getdents d4/dd 0 2024-04-22T23:23:36.485 INFO:tasks.workunit.client.1.smithi134.stdout:1/70: dwrite db/fc [0,4194304] 0 2024-04-22T23:23:36.486 INFO:tasks.workunit.client.1.smithi134.stdout:4/73: read d4/fe [582921,45946] 0 2024-04-22T23:23:36.487 INFO:tasks.workunit.client.1.smithi134.stdout:4/74: write d4/f6 [4324578,13787] 0 2024-04-22T23:23:36.488 INFO:tasks.workunit.client.1.smithi134.stdout:1/71: write db/f14 [1579053,61994] 0 2024-04-22T23:23:36.493 INFO:tasks.workunit.client.1.smithi134.stdout:4/75: mknod d4/dd/c11 0 2024-04-22T23:23:36.493 INFO:tasks.workunit.client.1.smithi134.stdout:4/76: stat d4/fe 0 2024-04-22T23:23:36.493 INFO:tasks.workunit.client.1.smithi134.stdout:8/37: dread f6 [0,4194304] 0 2024-04-22T23:23:36.495 INFO:tasks.workunit.client.1.smithi134.stdout:4/77: write d4/fe [4614318,47619] 0 2024-04-22T23:23:36.496 INFO:tasks.workunit.client.0.smithi067.stdout:8/775: mkdir d1/dd/d80/db9/dd0/df9 0 2024-04-22T23:23:36.497 INFO:tasks.workunit.client.1.smithi134.stdout:8/38: unlink d7/f9 0 2024-04-22T23:23:36.498 INFO:tasks.workunit.client.0.smithi067.stdout:1/760: dwrite d8/d9/f1b [0,4194304] 0 2024-04-22T23:23:36.498 INFO:tasks.workunit.client.0.smithi067.stdout:5/853: dwrite d5/d1a/d103/fd2 [0,4194304] 0 2024-04-22T23:23:36.499 INFO:tasks.workunit.client.0.smithi067.stdout:6/814: mknod d17/d39/d53/d90/d46/c110 0 2024-04-22T23:23:36.500 INFO:tasks.workunit.client.1.smithi134.stdout:4/78: symlink d4/dd/l12 0 2024-04-22T23:23:36.501 INFO:tasks.workunit.client.0.smithi067.stdout:7/721: getdents d10/d18/d88/daf 0 2024-04-22T23:23:36.502 INFO:tasks.workunit.client.1.smithi134.stdout:8/39: mkdir d7/dc 0 2024-04-22T23:23:36.502 INFO:tasks.workunit.client.0.smithi067.stdout:7/722: dread - d10/d18/d88/daf/d100/d62/ff6 zero size 2024-04-22T23:23:36.503 INFO:tasks.workunit.client.0.smithi067.stdout:4/941: rmdir d12/d42/d4b/d77/d9d/d13a 0 2024-04-22T23:23:36.506 INFO:tasks.workunit.client.1.smithi134.stdout:9/42: getdents d0/d7 0 2024-04-22T23:23:36.506 INFO:tasks.workunit.client.0.smithi067.stdout:8/776: unlink d1/d3/d10/f3d 0 2024-04-22T23:23:36.507 INFO:tasks.workunit.client.0.smithi067.stdout:6/815: creat d17/d83/d9b/f111 x:0 0 0 2024-04-22T23:23:36.511 INFO:tasks.workunit.client.1.smithi134.stdout:8/40: symlink d7/db/ld 0 2024-04-22T23:23:36.511 INFO:tasks.workunit.client.0.smithi067.stdout:2/755: dwrite d1/d10/f2e [4194304,4194304] 0 2024-04-22T23:23:36.514 INFO:tasks.workunit.client.0.smithi067.stdout:4/942: dread d12/f58 [0,4194304] 0 2024-04-22T23:23:36.515 INFO:tasks.workunit.client.1.smithi134.stdout:4/79: getdents d4 0 2024-04-22T23:23:36.517 INFO:tasks.workunit.client.1.smithi134.stdout:6/114: dwrite d0/d19/de/d1f/f22 [0,4194304] 0 2024-04-22T23:23:36.517 INFO:tasks.workunit.client.1.smithi134.stdout:8/41: creat d7/db/fe x:0 0 0 2024-04-22T23:23:36.519 INFO:tasks.workunit.client.1.smithi134.stdout:4/80: truncate d4/d7/f10 53405 0 2024-04-22T23:23:36.519 INFO:tasks.workunit.client.1.smithi134.stdout:4/81: stat d4 0 2024-04-22T23:23:36.520 INFO:tasks.workunit.client.1.smithi134.stdout:4/82: chown d4 3859469 1 2024-04-22T23:23:36.520 INFO:tasks.workunit.client.1.smithi134.stdout:9/43: getdents d0/d7 0 2024-04-22T23:23:36.520 INFO:tasks.workunit.client.0.smithi067.stdout:3/806: write d3/f6f [615149,13917] 0 2024-04-22T23:23:36.522 INFO:tasks.workunit.client.1.smithi134.stdout:9/44: dread - d0/d7/fb zero size 2024-04-22T23:23:36.522 INFO:tasks.workunit.client.1.smithi134.stdout:8/42: creat d7/dc/ff x:0 0 0 2024-04-22T23:23:36.523 INFO:tasks.workunit.client.0.smithi067.stdout:0/924: truncate db/de/d11/f3c 3794302 0 2024-04-22T23:23:36.524 INFO:tasks.workunit.client.1.smithi134.stdout:6/115: rename d0/d19/de/d1f/d24 to d0/d19/de/d12/d29 0 2024-04-22T23:23:36.526 INFO:tasks.workunit.client.1.smithi134.stdout:9/45: creat d0/d7/fd x:0 0 0 2024-04-22T23:23:36.527 INFO:tasks.workunit.client.0.smithi067.stdout:2/756: dwrite d1/d10/d1c/d6d/d87/fd8 [0,4194304] 0 2024-04-22T23:23:36.527 INFO:tasks.workunit.client.1.smithi134.stdout:4/83: dread d4/f6 [4194304,4194304] 0 2024-04-22T23:23:36.528 INFO:tasks.workunit.client.1.smithi134.stdout:4/84: write d4/fe [3368498,90941] 0 2024-04-22T23:23:36.529 INFO:tasks.workunit.client.1.smithi134.stdout:8/43: mknod d7/dc/c10 0 2024-04-22T23:23:36.529 INFO:tasks.workunit.client.1.smithi134.stdout:8/44: chown d7/db 4736068 1 2024-04-22T23:23:36.530 INFO:tasks.workunit.client.1.smithi134.stdout:6/116: mkdir d0/d19/de/d12/d2a 0 2024-04-22T23:23:36.531 INFO:tasks.workunit.client.0.smithi067.stdout:8/777: symlink d1/dd/d26/d62/ddf/ded/def/lfa 0 2024-04-22T23:23:36.532 INFO:tasks.workunit.client.0.smithi067.stdout:8/778: chown d1/dd/d26/d62/ddf/ded 46 1 2024-04-22T23:23:36.532 INFO:tasks.workunit.client.0.smithi067.stdout:6/816: mknod d17/d39/d53/d90/d46/d62/c112 0 2024-04-22T23:23:36.534 INFO:tasks.workunit.client.1.smithi134.stdout:9/46: rename d0/f1 to d0/d7/fe 0 2024-04-22T23:23:36.534 INFO:tasks.workunit.client.0.smithi067.stdout:8/779: write d1/d3/d27/f6e [5532408,49959] 0 2024-04-22T23:23:36.535 INFO:tasks.workunit.client.0.smithi067.stdout:6/817: read - d17/d39/d53/f60 zero size 2024-04-22T23:23:36.536 INFO:tasks.workunit.client.1.smithi134.stdout:9/47: write d0/d7/fd [536497,61934] 0 2024-04-22T23:23:36.536 INFO:tasks.workunit.client.1.smithi134.stdout:4/85: mkdir d4/d13 0 2024-04-22T23:23:36.537 INFO:tasks.workunit.client.0.smithi067.stdout:3/807: readlink d3/d4/d28/d40/dfd/dfe/l13 0 2024-04-22T23:23:36.537 INFO:tasks.workunit.client.1.smithi134.stdout:8/45: write f6 [413117,93660] 0 2024-04-22T23:23:36.537 INFO:tasks.workunit.client.0.smithi067.stdout:5/854: getdents d5/db/d12/dbb/d5f/d79/dba/de0 0 2024-04-22T23:23:36.538 INFO:tasks.workunit.client.0.smithi067.stdout:3/808: dread - d3/d4/d28/dc0/fcc zero size 2024-04-22T23:23:36.539 INFO:tasks.workunit.client.1.smithi134.stdout:8/46: readlink d7/db/ld 0 2024-04-22T23:23:36.539 INFO:tasks.workunit.client.1.smithi134.stdout:6/117: creat d0/d19/db/f2b x:0 0 0 2024-04-22T23:23:36.539 INFO:tasks.workunit.client.0.smithi067.stdout:7/723: write d10/d14/d1d/d26/d2c/d7a/f9b [387876,124186] 0 2024-04-22T23:23:36.541 INFO:tasks.workunit.client.1.smithi134.stdout:8/47: chown f6 447709559 1 2024-04-22T23:23:36.542 INFO:tasks.workunit.client.0.smithi067.stdout:4/943: creat d12/d17/d47/de7/f13e x:0 0 0 2024-04-22T23:23:36.543 INFO:tasks.workunit.client.1.smithi134.stdout:8/48: truncate f6 1532414 0 2024-04-22T23:23:36.544 INFO:tasks.workunit.client.1.smithi134.stdout:6/118: creat d0/d19/de/d1f/f2c x:0 0 0 2024-04-22T23:23:36.546 INFO:tasks.workunit.client.1.smithi134.stdout:9/48: dwrite d0/d7/fd [0,4194304] 0 2024-04-22T23:23:36.547 INFO:tasks.workunit.client.1.smithi134.stdout:8/49: read f4 [2111938,91490] 0 2024-04-22T23:23:36.547 INFO:tasks.workunit.client.1.smithi134.stdout:6/119: fdatasync d0/d19/f1c 0 2024-04-22T23:23:36.547 INFO:tasks.workunit.client.1.smithi134.stdout:6/120: read - d0/f27 zero size 2024-04-22T23:23:36.548 INFO:tasks.workunit.client.0.smithi067.stdout:0/925: unlink db/d21/d29/d75/d9d/dde/d10f/c4a 0 2024-04-22T23:23:36.549 INFO:tasks.workunit.client.1.smithi134.stdout:8/50: mknod d7/db/c11 0 2024-04-22T23:23:36.549 INFO:tasks.workunit.client.0.smithi067.stdout:3/809: rename d3/d4/fc to d3/d4/d10/d38/d53/ddc/f121 0 2024-04-22T23:23:36.550 INFO:tasks.workunit.client.1.smithi134.stdout:8/51: write d7/db/fe [284209,11970] 0 2024-04-22T23:23:36.551 INFO:tasks.workunit.client.1.smithi134.stdout:8/52: truncate f6 2465152 0 2024-04-22T23:23:36.553 INFO:tasks.workunit.client.0.smithi067.stdout:4/944: write d12/d42/f78 [2139450,17175] 0 2024-04-22T23:23:36.553 INFO:tasks.workunit.client.1.smithi134.stdout:6/121: getdents d0/d19/de 0 2024-04-22T23:23:36.554 INFO:tasks.workunit.client.1.smithi134.stdout:8/53: creat d7/dc/f12 x:0 0 0 2024-04-22T23:23:36.555 INFO:tasks.workunit.client.1.smithi134.stdout:6/122: chown d0/d19/de/f18 500 1 2024-04-22T23:23:36.555 INFO:tasks.workunit.client.0.smithi067.stdout:5/855: mkdir d5/d28/d36/d40/d12f 0 2024-04-22T23:23:36.556 INFO:tasks.workunit.client.0.smithi067.stdout:3/810: rename d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d90/da2/daf to d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d90/da2/daf/d122 22 2024-04-22T23:23:36.557 INFO:tasks.workunit.client.0.smithi067.stdout:0/926: readlink db/d9a/d56/d70/d8c/dc2/le0 0 2024-04-22T23:23:36.558 INFO:tasks.workunit.client.1.smithi134.stdout:6/123: mkdir d0/d19/de/d12/d2a/d2d 0 2024-04-22T23:23:36.559 INFO:tasks.workunit.client.0.smithi067.stdout:8/780: creat d1/dd/ffb x:0 0 0 2024-04-22T23:23:36.564 INFO:tasks.workunit.client.0.smithi067.stdout:3/811: dwrite d3/d4/d10/d8a/f106 [0,4194304] 0 2024-04-22T23:23:36.565 INFO:tasks.workunit.client.1.smithi134.stdout:4/86: write d4/fe [3598167,75299] 0 2024-04-22T23:23:36.568 INFO:tasks.workunit.client.0.smithi067.stdout:2/757: getdents d1/d10/d1c 0 2024-04-22T23:23:36.569 INFO:tasks.workunit.client.1.smithi134.stdout:4/87: symlink d4/dd/l14 0 2024-04-22T23:23:36.570 INFO:tasks.workunit.client.1.smithi134.stdout:6/124: dwrite d0/d19/de/d1f/f22 [0,4194304] 0 2024-04-22T23:23:36.570 INFO:tasks.workunit.client.1.smithi134.stdout:4/88: stat d4/dd 0 2024-04-22T23:23:36.570 INFO:tasks.workunit.client.0.smithi067.stdout:4/945: rename d12/d17/f100 to d12/d42/d4b/d122/d8e/d56/d93/f13f 0 2024-04-22T23:23:36.571 INFO:tasks.workunit.client.1.smithi134.stdout:6/125: dread - d0/f26 zero size 2024-04-22T23:23:36.571 INFO:tasks.workunit.client.0.smithi067.stdout:0/927: creat db/d71/dda/f12f x:0 0 0 2024-04-22T23:23:36.572 INFO:tasks.workunit.client.0.smithi067.stdout:3/812: truncate d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d90/da2/daf/dd8/ffc 4286208 0 2024-04-22T23:23:36.575 INFO:tasks.workunit.client.0.smithi067.stdout:6/818: truncate d17/dae/fc8 1639884 0 2024-04-22T23:23:36.581 INFO:tasks.workunit.client.0.smithi067.stdout:2/758: symlink d1/d10/d1c/d4d/lf7 0 2024-04-22T23:23:36.584 INFO:tasks.workunit.client.0.smithi067.stdout:4/946: unlink d12/d2e/fcb 0 2024-04-22T23:23:36.585 INFO:tasks.workunit.client.0.smithi067.stdout:6/819: dwrite d17/d39/d53/d90/d46/d62/fd0 [0,4194304] 0 2024-04-22T23:23:36.585 INFO:tasks.workunit.client.0.smithi067.stdout:7/724: truncate d10/d14/d1d/d46/d50/f29 1364316 0 2024-04-22T23:23:36.586 INFO:tasks.workunit.client.1.smithi134.stdout:6/126: dwrite d0/f16 [0,4194304] 0 2024-04-22T23:23:36.586 INFO:tasks.workunit.client.0.smithi067.stdout:6/820: stat d17/d83/db5/dbc/dd8/cfa 0 2024-04-22T23:23:36.586 INFO:tasks.workunit.client.0.smithi067.stdout:0/928: unlink db/d21/d29/d2c/f91 0 2024-04-22T23:23:36.587 INFO:tasks.workunit.client.1.smithi134.stdout:6/127: dread - d0/d19/f1b zero size 2024-04-22T23:23:36.588 INFO:tasks.workunit.client.0.smithi067.stdout:0/929: read db/d71/dda/ff2 [7833485,125786] 0 2024-04-22T23:23:36.589 INFO:tasks.workunit.client.1.smithi134.stdout:6/128: dread d0/d19/fa [0,4194304] 0 2024-04-22T23:23:36.590 INFO:tasks.workunit.client.0.smithi067.stdout:8/781: write d1/dd/d80/d6f/f34 [6529455,112628] 0 2024-04-22T23:23:36.591 INFO:tasks.workunit.client.0.smithi067.stdout:3/813: creat d3/d4/d28/d40/dfd/f123 x:0 0 0 2024-04-22T23:23:36.592 INFO:tasks.workunit.client.0.smithi067.stdout:6/821: truncate d17/d39/f4b 1638278 0 2024-04-22T23:23:36.593 INFO:tasks.workunit.client.0.smithi067.stdout:3/814: stat d3/d4/d28/fd0 0 2024-04-22T23:23:36.594 INFO:tasks.workunit.client.0.smithi067.stdout:6/822: read fe [1215238,102944] 0 2024-04-22T23:23:36.597 INFO:tasks.workunit.client.0.smithi067.stdout:5/856: dwrite d5/db/d12/d44/d47/f114 [0,4194304] 0 2024-04-22T23:23:36.597 INFO:tasks.workunit.client.0.smithi067.stdout:3/815: chown d3/d4/d10/d19/l108 15 1 2024-04-22T23:23:36.598 INFO:tasks.workunit.client.0.smithi067.stdout:2/759: mknod d1/d10/d72/d68/da1/cf8 0 2024-04-22T23:23:36.604 INFO:tasks.workunit.client.1.smithi134.stdout:0/65: sync 2024-04-22T23:23:36.604 INFO:tasks.workunit.client.0.smithi067.stdout:7/725: mknod d10/d18/d88/c10b 0 2024-04-22T23:23:36.604 INFO:tasks.workunit.client.0.smithi067.stdout:0/930: mknod db/d60/c130 0 2024-04-22T23:23:36.608 INFO:tasks.workunit.client.1.smithi134.stdout:0/66: mkdir d1/d12 0 2024-04-22T23:23:36.609 INFO:tasks.workunit.client.1.smithi134.stdout:0/67: write d1/f10 [594398,128229] 0 2024-04-22T23:23:36.609 INFO:tasks.workunit.client.1.smithi134.stdout:0/68: chown d1/d5/dd/d11 69205391 1 2024-04-22T23:23:36.612 INFO:tasks.workunit.client.1.smithi134.stdout:0/69: symlink d1/db/l13 0 2024-04-22T23:23:36.624 INFO:tasks.workunit.client.0.smithi067.stdout:7/726: read d10/d14/d1d/d26/d2c/d7a/f87 [4125073,65181] 0 2024-04-22T23:23:36.626 INFO:tasks.workunit.client.0.smithi067.stdout:6/823: truncate d17/d39/d53/d90/d46/db0/db9/fc3 79607 0 2024-04-22T23:23:36.627 INFO:tasks.workunit.client.1.smithi134.stdout:0/70: truncate d1/d2/f4 2776990 0 2024-04-22T23:23:36.627 INFO:tasks.workunit.client.0.smithi067.stdout:4/947: fsync d12/d17/d3f/d84/f9f 0 2024-04-22T23:23:36.628 INFO:tasks.workunit.client.0.smithi067.stdout:2/760: symlink d1/d10/d33/d46/da4/dde/lf9 0 2024-04-22T23:23:36.629 INFO:tasks.workunit.client.0.smithi067.stdout:8/782: rename d1/d41/db4 to d1/dd/d80/d6f/dfc 0 2024-04-22T23:23:36.630 INFO:tasks.workunit.client.0.smithi067.stdout:6/824: dread - d17/d39/d53/f60 zero size 2024-04-22T23:23:36.632 INFO:tasks.workunit.client.0.smithi067.stdout:0/931: dwrite db/d21/d29/d75/d9d/dde/d10f/f5d [0,4194304] 0 2024-04-22T23:23:36.634 INFO:tasks.workunit.client.1.smithi134.stdout:0/71: dwrite d1/f10 [0,4194304] 0 2024-04-22T23:23:36.636 INFO:tasks.workunit.client.0.smithi067.stdout:2/761: mknod d1/d10/d41/cfa 0 2024-04-22T23:23:36.638 INFO:tasks.workunit.client.0.smithi067.stdout:5/857: dread d5/db/d12/dbb/d61/f82 [0,4194304] 0 2024-04-22T23:23:36.638 INFO:tasks.workunit.client.0.smithi067.stdout:4/948: mkdir d12/d42/d4b/d122/d8e/d56/d114/d140 0 2024-04-22T23:23:36.639 INFO:tasks.workunit.client.1.smithi134.stdout:0/72: dread d1/f10 [0,4194304] 0 2024-04-22T23:23:36.641 INFO:tasks.workunit.client.0.smithi067.stdout:0/932: dwrite db/d9a/d56/d6b/f111 [0,4194304] 0 2024-04-22T23:23:36.645 INFO:tasks.workunit.client.1.smithi134.stdout:0/73: rename d1/d2/fa to d1/d5/dd/f14 0 2024-04-22T23:23:36.647 INFO:tasks.workunit.client.0.smithi067.stdout:7/727: mkdir d10/d14/d1d/d26/d7e/d10c 0 2024-04-22T23:23:36.649 INFO:tasks.workunit.client.0.smithi067.stdout:4/949: dread d12/d42/d4b/d122/d8e/d56/d93/f11c [0,4194304] 0 2024-04-22T23:23:36.651 INFO:tasks.workunit.client.0.smithi067.stdout:4/950: chown d12/d17 103524535 1 2024-04-22T23:23:36.654 INFO:tasks.workunit.client.0.smithi067.stdout:8/783: dwrite d1/d3/d10/d3b/f5e [0,4194304] 0 2024-04-22T23:23:36.656 INFO:tasks.workunit.client.0.smithi067.stdout:6/825: dwrite d17/d39/d53/d90/d46/f68 [4194304,4194304] 0 2024-04-22T23:23:36.657 INFO:tasks.workunit.client.0.smithi067.stdout:8/784: dread - d1/d3/d5c/d98/fd8 zero size 2024-04-22T23:23:36.657 INFO:tasks.workunit.client.0.smithi067.stdout:5/858: symlink d5/d3e/d90/d11e/l130 0 2024-04-22T23:23:36.659 INFO:tasks.workunit.client.0.smithi067.stdout:5/859: fsync d5/d28/daf/dbf/dcf/db2/f12e 0 2024-04-22T23:23:36.664 INFO:tasks.workunit.client.0.smithi067.stdout:4/951: write d12/d17/d47/f49 [3825671,63916] 0 2024-04-22T23:23:36.666 INFO:tasks.workunit.client.0.smithi067.stdout:6/826: mknod d17/d39/d53/c113 0 2024-04-22T23:23:36.667 INFO:tasks.workunit.client.0.smithi067.stdout:7/728: symlink d10/d14/d1d/d26/l10d 0 2024-04-22T23:23:36.669 INFO:tasks.workunit.client.1.smithi134.stdout:0/74: truncate d1/d5/dd/f14 3126821 0 2024-04-22T23:23:36.672 INFO:tasks.workunit.client.1.smithi134.stdout:0/75: dread d1/f10 [0,4194304] 0 2024-04-22T23:23:36.673 INFO:tasks.workunit.client.1.smithi134.stdout:0/76: readlink d1/db/l13 0 2024-04-22T23:23:36.673 INFO:tasks.workunit.client.1.smithi134.stdout:0/77: chown d1/db/l13 1 1 2024-04-22T23:23:36.674 INFO:tasks.workunit.client.0.smithi067.stdout:0/933: rename db/d9a/d56/d6b/fa2 to db/de/d11/d1a/da4/dc5/f131 0 2024-04-22T23:23:36.676 INFO:tasks.workunit.client.0.smithi067.stdout:4/952: creat d12/d5f/d2b/db6/f141 x:0 0 0 2024-04-22T23:23:36.679 INFO:tasks.workunit.client.0.smithi067.stdout:2/762: link d1/d10/d1c/d6d/d87/fb6 d1/d19/ded/ffb 0 2024-04-22T23:23:36.685 INFO:tasks.workunit.client.0.smithi067.stdout:8/785: rename d1/fe7 to d1/dd/d80/d6f/db6/ffd 0 2024-04-22T23:23:36.686 INFO:tasks.workunit.client.0.smithi067.stdout:0/934: write db/d9a/d56/d70/fd3 [561998,103543] 0 2024-04-22T23:23:36.688 INFO:tasks.workunit.client.0.smithi067.stdout:6/827: getdents d17/d39/d3a/da9/df7 0 2024-04-22T23:23:36.688 INFO:tasks.workunit.client.0.smithi067.stdout:0/935: stat db/d9a/d56/d70/fc3 0 2024-04-22T23:23:36.688 INFO:tasks.workunit.client.1.smithi134.stdout:0/78: dwrite d1/d2/f4 [0,4194304] 0 2024-04-22T23:23:36.689 INFO:tasks.workunit.client.1.smithi134.stdout:0/79: chown d1 124726124 1 2024-04-22T23:23:36.689 INFO:tasks.workunit.client.0.smithi067.stdout:7/729: mkdir d10/d14/d1d/d26/d7e/dc0/d10e 0 2024-04-22T23:23:36.690 INFO:tasks.workunit.client.1.smithi134.stdout:0/80: mkdir d1/d5/d15 0 2024-04-22T23:23:36.691 INFO:tasks.workunit.client.0.smithi067.stdout:7/730: stat d10/d14/d1d/d69/lc4 0 2024-04-22T23:23:36.691 INFO:tasks.workunit.client.0.smithi067.stdout:0/936: dread db/d21/d29/d75/d9d/fb8 [0,4194304] 0 2024-04-22T23:23:36.692 INFO:tasks.workunit.client.1.smithi134.stdout:0/81: creat d1/d5/dd/f16 x:0 0 0 2024-04-22T23:23:36.693 INFO:tasks.workunit.client.0.smithi067.stdout:4/953: symlink d12/d17/d88/l142 0 2024-04-22T23:23:36.694 INFO:tasks.workunit.client.1.smithi134.stdout:0/82: getdents d1/d12 0 2024-04-22T23:23:36.694 INFO:tasks.workunit.client.0.smithi067.stdout:8/786: truncate d1/d3/d5c/d98/fa1 1618548 0 2024-04-22T23:23:36.695 INFO:tasks.workunit.client.1.smithi134.stdout:0/83: mkdir d1/d12/d17 0 2024-04-22T23:23:36.697 INFO:tasks.workunit.client.0.smithi067.stdout:5/860: getdents d5/db/d11f 0 2024-04-22T23:23:36.698 INFO:tasks.workunit.client.0.smithi067.stdout:4/954: truncate d12/d42/dd3/f106 868528 0 2024-04-22T23:23:36.698 INFO:tasks.workunit.client.0.smithi067.stdout:2/763: dread d1/d10/d72/d53/d9a/fb1 [0,4194304] 0 2024-04-22T23:23:36.699 INFO:tasks.workunit.client.0.smithi067.stdout:5/861: readlink d5/d3e/d90/la5 0 2024-04-22T23:23:36.701 INFO:tasks.workunit.client.0.smithi067.stdout:8/787: truncate d1/dd/f85 402445 0 2024-04-22T23:23:36.703 INFO:tasks.workunit.client.1.smithi134.stdout:0/84: dwrite d1/f9 [0,4194304] 0 2024-04-22T23:23:36.703 INFO:tasks.workunit.client.0.smithi067.stdout:8/788: dread d1/dd/d26/d62/d94/fc6 [0,4194304] 0 2024-04-22T23:23:36.705 INFO:tasks.workunit.client.0.smithi067.stdout:0/937: rename db/d9a/d56/d6b/f111 to db/de/d11/d92/f132 0 2024-04-22T23:23:36.708 INFO:tasks.workunit.client.0.smithi067.stdout:5/862: truncate d5/db/d12/d44/d47/fda 1338938 0 2024-04-22T23:23:36.708 INFO:tasks.workunit.client.1.smithi134.stdout:0/85: write d1/d5/dd/f14 [1105877,44357] 0 2024-04-22T23:23:36.708 INFO:tasks.workunit.client.0.smithi067.stdout:5/863: dread - d5/f31 zero size 2024-04-22T23:23:36.710 INFO:tasks.workunit.client.0.smithi067.stdout:5/864: dread - d5/d28/d36/d40/f10b zero size 2024-04-22T23:23:36.711 INFO:tasks.workunit.client.1.smithi134.stdout:0/86: rename d1/d5/dd/f14 to d1/d5/dd/d11/f18 0 2024-04-22T23:23:36.714 INFO:tasks.workunit.client.0.smithi067.stdout:4/955: mkdir d12/d17/d143 0 2024-04-22T23:23:36.717 INFO:tasks.workunit.client.0.smithi067.stdout:7/731: link d10/d18/d88/cbe d10/d14/d1d/d26/d2c/d7a/dc2/dd0/c10f 0 2024-04-22T23:23:36.717 INFO:tasks.workunit.client.1.smithi134.stdout:0/87: dwrite d1/d5/f7 [0,4194304] 0 2024-04-22T23:23:36.719 INFO:tasks.workunit.client.0.smithi067.stdout:4/956: truncate d12/d17/d47/f6c 948261 0 2024-04-22T23:23:36.720 INFO:tasks.workunit.client.1.smithi134.stdout:0/88: getdents d1/d12/d17 0 2024-04-22T23:23:36.721 INFO:tasks.workunit.client.0.smithi067.stdout:4/957: truncate d12/d42/d4b/d122/d8e/fcf 8510800 0 2024-04-22T23:23:36.722 INFO:tasks.workunit.client.0.smithi067.stdout:4/958: readlink d12/d2e/l80 0 2024-04-22T23:23:36.722 INFO:tasks.workunit.client.0.smithi067.stdout:0/938: rename db/de/d11/d1a/da4/l4b to db/l133 0 2024-04-22T23:23:36.724 INFO:tasks.workunit.client.0.smithi067.stdout:7/732: unlink d10/d14/d1d/d26/d2c/d41/lc3 0 2024-04-22T23:23:36.724 INFO:tasks.workunit.client.1.smithi134.stdout:0/89: link d1/db/cf d1/d5/dd/d11/c19 0 2024-04-22T23:23:36.725 INFO:tasks.workunit.client.1.smithi134.stdout:0/90: write d1/f10 [2905147,42686] 0 2024-04-22T23:23:36.726 INFO:tasks.workunit.client.0.smithi067.stdout:7/733: write d10/d14/d1d/d26/d80/fbb [822838,74606] 0 2024-04-22T23:23:36.727 INFO:tasks.workunit.client.1.smithi134.stdout:0/91: truncate d1/d5/dd/f16 452588 0 2024-04-22T23:23:36.727 INFO:tasks.workunit.client.0.smithi067.stdout:2/764: dread d1/d10/d72/fd7 [0,4194304] 0 2024-04-22T23:23:36.727 INFO:tasks.workunit.client.0.smithi067.stdout:4/959: symlink d12/d42/d4b/l144 0 2024-04-22T23:23:36.728 INFO:tasks.workunit.client.1.smithi134.stdout:0/92: mknod d1/d12/d17/c1a 0 2024-04-22T23:23:36.731 INFO:tasks.workunit.client.0.smithi067.stdout:0/939: rename db/de/d2a/f5e to db/d60/d112/f134 0 2024-04-22T23:23:36.732 INFO:tasks.workunit.client.1.smithi134.stdout:0/93: dread d1/d5/dd/d11/f18 [0,4194304] 0 2024-04-22T23:23:36.733 INFO:tasks.workunit.client.1.smithi134.stdout:0/94: read d1/d5/dd/f16 [207062,52700] 0 2024-04-22T23:23:36.735 INFO:tasks.workunit.client.0.smithi067.stdout:7/734: dread d10/d14/d1d/d26/d2c/d7a/f102 [0,4194304] 0 2024-04-22T23:23:36.735 INFO:tasks.workunit.client.0.smithi067.stdout:2/765: dwrite d1/d10/d72/d53/f97 [0,4194304] 0 2024-04-22T23:23:36.735 INFO:tasks.workunit.client.1.smithi134.stdout:0/95: mknod d1/d5/dd/d11/c1b 0 2024-04-22T23:23:36.735 INFO:tasks.workunit.client.1.smithi134.stdout:0/96: stat d1/d5/dd/f16 0 2024-04-22T23:23:36.740 INFO:tasks.workunit.client.0.smithi067.stdout:5/865: write d5/d28/daf/dbf/dcf/fbc [294240,7334] 0 2024-04-22T23:23:36.740 INFO:tasks.workunit.client.0.smithi067.stdout:0/940: read db/de/d2a/f99 [3564359,114082] 0 2024-04-22T23:23:36.742 INFO:tasks.workunit.client.0.smithi067.stdout:0/941: chown db/d60/d9e/fc4 4379 1 2024-04-22T23:23:36.742 INFO:tasks.workunit.client.0.smithi067.stdout:5/866: truncate d5/d28/daf/f111 647969 0 2024-04-22T23:23:36.742 INFO:tasks.workunit.client.1.smithi134.stdout:0/97: dwrite d1/d5/f7 [0,4194304] 0 2024-04-22T23:23:36.743 INFO:tasks.workunit.client.0.smithi067.stdout:8/789: truncate d1/faf 755339 0 2024-04-22T23:23:36.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:36 smithi134 ceph-mon[34609]: pgmap v273: 65 pgs: 65 active+clean; 1.6 GiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 52 MiB/s rd, 169 MiB/s wr, 340 op/s 2024-04-22T23:23:36.746 INFO:tasks.workunit.client.0.smithi067.stdout:4/960: rename d12/d42/d4b/d122/d8e/lc5 to d12/d42/dd3/d13b/l145 0 2024-04-22T23:23:36.750 INFO:tasks.workunit.client.0.smithi067.stdout:5/867: mknod d5/d28/daf/db4/c131 0 2024-04-22T23:23:36.752 INFO:tasks.workunit.client.0.smithi067.stdout:5/868: chown d5/d28/daf/dbf/dfc/d99/fc3 13 1 2024-04-22T23:23:36.752 INFO:tasks.workunit.client.1.smithi134.stdout:0/98: dwrite d1/d5/dd/d11/f18 [0,4194304] 0 2024-04-22T23:23:36.753 INFO:tasks.workunit.client.0.smithi067.stdout:0/942: creat db/d21/d29/d75/d9d/dde/d10f/f135 x:0 0 0 2024-04-22T23:23:36.753 INFO:tasks.workunit.client.1.smithi134.stdout:0/99: write d1/f9 [2769232,70035] 0 2024-04-22T23:23:36.755 INFO:tasks.workunit.client.0.smithi067.stdout:2/766: creat d1/ffc x:0 0 0 2024-04-22T23:23:36.756 INFO:tasks.workunit.client.1.smithi134.stdout:0/100: write d1/d5/dd/f16 [364983,47582] 0 2024-04-22T23:23:36.757 INFO:tasks.workunit.client.0.smithi067.stdout:2/767: chown d1/d10/d72/d68/da1/cf8 48253558 1 2024-04-22T23:23:36.758 INFO:tasks.workunit.client.0.smithi067.stdout:7/735: rmdir d10/d14/d1d/d26/d7e/d10c 0 2024-04-22T23:23:36.759 INFO:tasks.workunit.client.1.smithi134.stdout:0/101: creat d1/f1c x:0 0 0 2024-04-22T23:23:36.760 INFO:tasks.workunit.client.0.smithi067.stdout:0/943: creat db/de/d11/d1a/f136 x:0 0 0 2024-04-22T23:23:36.762 INFO:tasks.workunit.client.0.smithi067.stdout:8/790: link d1/dd/d26/fd6 d1/d3/d7e/ffe 0 2024-04-22T23:23:36.763 INFO:tasks.workunit.client.0.smithi067.stdout:8/791: write d1/dd/d26/f6b [2839365,102478] 0 2024-04-22T23:23:36.764 INFO:tasks.workunit.client.0.smithi067.stdout:4/961: link d12/d5f/d2b/d97/fca d12/d17/d3f/d101/f146 0 2024-04-22T23:23:36.764 INFO:tasks.workunit.client.1.smithi134.stdout:0/102: dwrite d1/d5/f7 [0,4194304] 0 2024-04-22T23:23:36.765 INFO:tasks.workunit.client.0.smithi067.stdout:4/962: fsync d12/d2e/fb7 0 2024-04-22T23:23:36.769 INFO:tasks.workunit.client.0.smithi067.stdout:0/944: rename db/de/d11/d1a/da4/f8e to db/d21/d29/d75/d9d/dde/d100/f137 0 2024-04-22T23:23:36.770 INFO:tasks.workunit.client.0.smithi067.stdout:8/792: symlink d1/d41/d46/lff 0 2024-04-22T23:23:36.771 INFO:tasks.workunit.client.0.smithi067.stdout:2/768: dread d1/d4f/fa3 [4194304,4194304] 0 2024-04-22T23:23:36.773 INFO:tasks.workunit.client.0.smithi067.stdout:4/963: unlink d12/d42/d4b/d77/d9d/dc1/d111/f11d 0 2024-04-22T23:23:36.773 INFO:tasks.workunit.client.0.smithi067.stdout:0/945: mkdir db/d9a/d56/d70/d8c/d138 0 2024-04-22T23:23:36.774 INFO:tasks.workunit.client.0.smithi067.stdout:7/736: symlink d10/d14/d6d/d9e/l110 0 2024-04-22T23:23:36.777 INFO:tasks.workunit.client.0.smithi067.stdout:2/769: rename d1/f35 to d1/d42/d61/dad/ffd 0 2024-04-22T23:23:36.778 INFO:tasks.workunit.client.0.smithi067.stdout:0/946: unlink db/de/d2a/l58 0 2024-04-22T23:23:36.783 INFO:tasks.workunit.client.0.smithi067.stdout:7/737: creat d10/d14/d1d/d26/d2c/d7a/dc2/dd0/de4/f111 x:0 0 0 2024-04-22T23:23:36.784 INFO:tasks.workunit.client.0.smithi067.stdout:7/738: dread - d10/d14/d1d/d26/d2c/fcf zero size 2024-04-22T23:23:36.785 INFO:tasks.workunit.client.0.smithi067.stdout:0/947: dwrite db/d21/d29/d2c/f72 [0,4194304] 0 2024-04-22T23:23:36.785 INFO:tasks.workunit.client.0.smithi067.stdout:4/964: rename d12/c13 to d12/d42/d4b/d77/c147 0 2024-04-22T23:23:36.786 INFO:tasks.workunit.client.0.smithi067.stdout:7/739: dread - d10/d14/de0/ff7 zero size 2024-04-22T23:23:36.793 INFO:tasks.workunit.client.1.smithi134.stdout:7/100: sync 2024-04-22T23:23:36.793 INFO:tasks.workunit.client.1.smithi134.stdout:3/54: sync 2024-04-22T23:23:36.794 INFO:tasks.workunit.client.0.smithi067.stdout:7/740: symlink d10/d14/d1d/d26/d2c/d7a/dc2/dd0/l112 0 2024-04-22T23:23:36.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:36 smithi067 ceph-mon[26453]: pgmap v273: 65 pgs: 65 active+clean; 1.6 GiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 52 MiB/s rd, 169 MiB/s wr, 340 op/s 2024-04-22T23:23:36.796 INFO:tasks.workunit.client.0.smithi067.stdout:4/965: dwrite d12/f2a [8388608,4194304] 0 2024-04-22T23:23:36.797 INFO:tasks.workunit.client.0.smithi067.stdout:0/948: dwrite db/d9a/d56/d6b/db9/f104 [8388608,4194304] 0 2024-04-22T23:23:36.800 INFO:tasks.workunit.client.0.smithi067.stdout:4/966: mknod d12/d17/c148 0 2024-04-22T23:23:36.801 INFO:tasks.workunit.client.1.smithi134.stdout:7/101: symlink d4/d10/l24 0 2024-04-22T23:23:36.801 INFO:tasks.workunit.client.1.smithi134.stdout:0/103: dread d1/f10 [0,4194304] 0 2024-04-22T23:23:36.803 INFO:tasks.workunit.client.0.smithi067.stdout:0/949: mknod db/d9a/d56/d70/d8c/d138/c139 0 2024-04-22T23:23:36.804 INFO:tasks.workunit.client.0.smithi067.stdout:4/967: symlink d12/d2e/l149 0 2024-04-22T23:23:36.807 INFO:tasks.workunit.client.0.smithi067.stdout:7/741: link d10/d14/d1d/d26/d2c/d41/f7c d10/d14/d1d/d26/d2c/d41/d82/f113 0 2024-04-22T23:23:36.809 INFO:tasks.workunit.client.0.smithi067.stdout:7/742: chown d10/d14 686142 1 2024-04-22T23:23:36.809 INFO:tasks.workunit.client.1.smithi134.stdout:3/55: getdents dc/dd 0 2024-04-22T23:23:36.811 INFO:tasks.workunit.client.1.smithi134.stdout:0/104: mkdir d1/d2/d1d 0 2024-04-22T23:23:36.812 INFO:tasks.workunit.client.0.smithi067.stdout:5/869: dread d5/f11 [0,4194304] 0 2024-04-22T23:23:36.813 INFO:tasks.workunit.client.0.smithi067.stdout:4/968: getdents d12/d42/d4b/d77/d8c 0 2024-04-22T23:23:36.814 INFO:tasks.workunit.client.0.smithi067.stdout:0/950: dwrite db/d71/f12a [0,4194304] 0 2024-04-22T23:23:36.815 INFO:tasks.workunit.client.0.smithi067.stdout:7/743: creat d10/d14/d1d/d26/d2c/d41/f114 x:0 0 0 2024-04-22T23:23:36.815 INFO:tasks.workunit.client.1.smithi134.stdout:3/56: rename dc/dd/l14 to dc/l1b 0 2024-04-22T23:23:36.816 INFO:tasks.workunit.client.0.smithi067.stdout:4/969: write d12/d17/d47/fe5 [581982,126975] 0 2024-04-22T23:23:36.817 INFO:tasks.workunit.client.0.smithi067.stdout:5/870: dread d5/d28/daf/dbf/dcf/fc7 [0,4194304] 0 2024-04-22T23:23:36.817 INFO:tasks.workunit.client.1.smithi134.stdout:0/105: write d1/d5/dd/f16 [1132637,98419] 0 2024-04-22T23:23:36.819 INFO:tasks.workunit.client.1.smithi134.stdout:0/106: write d1/f1c [209708,121036] 0 2024-04-22T23:23:36.819 INFO:tasks.workunit.client.1.smithi134.stdout:3/57: rename f3 to dc/de/df/f1c 0 2024-04-22T23:23:36.821 INFO:tasks.workunit.client.0.smithi067.stdout:0/951: creat db/d71/dda/f13a x:0 0 0 2024-04-22T23:23:36.821 INFO:tasks.workunit.client.1.smithi134.stdout:3/58: dread - dc/de/df/f17 zero size 2024-04-22T23:23:36.822 INFO:tasks.workunit.client.1.smithi134.stdout:3/59: dread - dc/de/df/f17 zero size 2024-04-22T23:23:36.822 INFO:tasks.workunit.client.1.smithi134.stdout:0/107: creat d1/d12/d17/f1e x:0 0 0 2024-04-22T23:23:36.822 INFO:tasks.workunit.client.1.smithi134.stdout:3/60: chown dc 79026917 1 2024-04-22T23:23:36.822 INFO:tasks.workunit.client.1.smithi134.stdout:0/108: readlink d1/db/l13 0 2024-04-22T23:23:36.823 INFO:tasks.workunit.client.1.smithi134.stdout:3/61: chown dc 42448 1 2024-04-22T23:23:36.823 INFO:tasks.workunit.client.1.smithi134.stdout:3/62: stat l6 0 2024-04-22T23:23:36.824 INFO:tasks.workunit.client.0.smithi067.stdout:4/970: mknod d12/d42/dd3/c14a 0 2024-04-22T23:23:36.824 INFO:tasks.workunit.client.1.smithi134.stdout:0/109: rmdir d1/d12 39 2024-04-22T23:23:36.824 INFO:tasks.workunit.client.1.smithi134.stdout:3/63: truncate dc/dd/d15/f16 591710 0 2024-04-22T23:23:36.825 INFO:tasks.workunit.client.1.smithi134.stdout:3/64: write fa [4720120,90487] 0 2024-04-22T23:23:36.827 INFO:tasks.workunit.client.1.smithi134.stdout:0/110: dread - d1/d12/d17/f1e zero size 2024-04-22T23:23:36.828 INFO:tasks.workunit.client.1.smithi134.stdout:3/65: mkdir dc/de/d1d 0 2024-04-22T23:23:36.830 INFO:tasks.workunit.client.1.smithi134.stdout:0/111: creat d1/d5/f1f x:0 0 0 2024-04-22T23:23:36.830 INFO:tasks.workunit.client.1.smithi134.stdout:3/66: mknod dc/d19/c1e 0 2024-04-22T23:23:36.832 INFO:tasks.workunit.client.0.smithi067.stdout:4/971: dwrite d12/d5f/d2b/d97/f10b [0,4194304] 0 2024-04-22T23:23:36.832 INFO:tasks.workunit.client.1.smithi134.stdout:3/67: creat dc/f1f x:0 0 0 2024-04-22T23:23:36.833 INFO:tasks.workunit.client.0.smithi067.stdout:2/770: dread d1/d10/d1c/d6d/d87/dc8/dc5/fdd [0,4194304] 0 2024-04-22T23:23:36.835 INFO:tasks.workunit.client.1.smithi134.stdout:3/68: rename l6 to dc/dd/l20 0 2024-04-22T23:23:36.837 INFO:tasks.workunit.client.1.smithi134.stdout:3/69: getdents dc/dd/d15 0 2024-04-22T23:23:36.838 INFO:tasks.workunit.client.0.smithi067.stdout:4/972: rename d12/d17/d3f/d84/c10d to d12/d17/d47/c14b 0 2024-04-22T23:23:36.838 INFO:tasks.workunit.client.0.smithi067.stdout:4/973: read - d12/d5f/d69/f113 zero size 2024-04-22T23:23:36.839 INFO:tasks.workunit.client.0.smithi067.stdout:2/771: fsync d1/d10/f30 0 2024-04-22T23:23:36.841 INFO:tasks.workunit.client.1.smithi134.stdout:3/70: rmdir dc/de/d1d 0 2024-04-22T23:23:36.844 INFO:tasks.workunit.client.0.smithi067.stdout:2/772: creat d1/d10/d1c/dd9/ffe x:0 0 0 2024-04-22T23:23:36.845 INFO:tasks.workunit.client.1.smithi134.stdout:3/71: rename dc/dd/d15 to dc/de/d21 0 2024-04-22T23:23:36.848 INFO:tasks.workunit.client.0.smithi067.stdout:4/974: dwrite d12/d5f/d7e/f130 [0,4194304] 0 2024-04-22T23:23:36.848 INFO:tasks.workunit.client.1.smithi134.stdout:3/72: rename fa to dc/f22 0 2024-04-22T23:23:36.851 INFO:tasks.workunit.client.1.smithi134.stdout:3/73: rename dc/dd/f11 to dc/d19/f23 0 2024-04-22T23:23:36.852 INFO:tasks.workunit.client.1.smithi134.stdout:7/102: write d4/d8/ff [926106,87722] 0 2024-04-22T23:23:36.855 INFO:tasks.workunit.client.1.smithi134.stdout:3/74: mkdir dc/de/d21/d24 0 2024-04-22T23:23:36.858 INFO:tasks.workunit.client.1.smithi134.stdout:7/103: mknod d4/d8/d1d/c25 0 2024-04-22T23:23:36.859 INFO:tasks.workunit.client.1.smithi134.stdout:7/104: fdatasync d4/d10/f1c 0 2024-04-22T23:23:36.859 INFO:tasks.workunit.client.1.smithi134.stdout:0/112: rmdir d1/d5 39 2024-04-22T23:23:36.860 INFO:tasks.workunit.client.1.smithi134.stdout:0/113: chown d1/db 252817642 1 2024-04-22T23:23:36.863 INFO:tasks.workunit.client.0.smithi067.stdout:8/793: getdents d1/d41/d46 0 2024-04-22T23:23:36.863 INFO:tasks.workunit.client.1.smithi134.stdout:7/105: symlink d4/d8/d1d/l26 0 2024-04-22T23:23:36.864 INFO:tasks.workunit.client.0.smithi067.stdout:4/975: dread d12/d17/d3f/fbe [0,4194304] 0 2024-04-22T23:23:36.864 INFO:tasks.workunit.client.1.smithi134.stdout:3/75: dwrite f9 [0,4194304] 0 2024-04-22T23:23:36.866 INFO:tasks.workunit.client.0.smithi067.stdout:4/976: truncate d12/d5f/d2b/d12a/f136 897436 0 2024-04-22T23:23:36.867 INFO:tasks.workunit.client.1.smithi134.stdout:0/114: dread d1/f9 [0,4194304] 0 2024-04-22T23:23:36.867 INFO:tasks.workunit.client.0.smithi067.stdout:4/977: write d12/d5f/d7e/fc9 [207980,54143] 0 2024-04-22T23:23:36.868 INFO:tasks.workunit.client.0.smithi067.stdout:8/794: creat d1/dd/d80/d6f/dfc/de8/f100 x:0 0 0 2024-04-22T23:23:36.869 INFO:tasks.workunit.client.0.smithi067.stdout:8/795: chown d1/d3/d7e/le4 7964795 1 2024-04-22T23:23:36.872 INFO:tasks.workunit.client.1.smithi134.stdout:3/76: mknod dc/dd/c25 0 2024-04-22T23:23:36.874 INFO:tasks.workunit.client.1.smithi134.stdout:0/115: stat d1/db/cf 0 2024-04-22T23:23:36.875 INFO:tasks.workunit.client.0.smithi067.stdout:4/978: mknod d12/d17/d3f/c14c 0 2024-04-22T23:23:36.877 INFO:tasks.workunit.client.1.smithi134.stdout:0/116: read d1/f9 [682374,65929] 0 2024-04-22T23:23:36.880 INFO:tasks.workunit.client.1.smithi134.stdout:7/106: rename d4/d10/c20 to d4/c27 0 2024-04-22T23:23:36.882 INFO:tasks.workunit.client.1.smithi134.stdout:3/77: unlink dc/dd/l20 0 2024-04-22T23:23:36.882 INFO:tasks.workunit.client.0.smithi067.stdout:4/979: getdents d12/d42/d4b/d122/d8e/d56/d114/dc3 0 2024-04-22T23:23:36.882 INFO:tasks.workunit.client.0.smithi067.stdout:0/952: rmdir db/d9a 39 2024-04-22T23:23:36.883 INFO:tasks.workunit.client.0.smithi067.stdout:0/953: chown db/de/d2a 6938708 1 2024-04-22T23:23:36.884 INFO:tasks.workunit.client.1.smithi134.stdout:3/78: dread - dc/de/df/f17 zero size 2024-04-22T23:23:36.884 INFO:tasks.workunit.client.0.smithi067.stdout:0/954: readlink db/de/d11/d92/l113 0 2024-04-22T23:23:36.886 INFO:tasks.workunit.client.1.smithi134.stdout:0/117: dwrite d1/d2/f4 [0,4194304] 0 2024-04-22T23:23:36.888 INFO:tasks.workunit.client.0.smithi067.stdout:0/955: unlink db/d21/d29/d2c/f72 0 2024-04-22T23:23:36.888 INFO:tasks.workunit.client.0.smithi067.stdout:5/871: write d5/db/d12/f22 [987511,92467] 0 2024-04-22T23:23:36.889 INFO:tasks.workunit.client.0.smithi067.stdout:4/980: dwrite d12/d5f/d2b/d97/f10b [0,4194304] 0 2024-04-22T23:23:36.889 INFO:tasks.workunit.client.1.smithi134.stdout:7/107: truncate d4/d10/f23 511074 0 2024-04-22T23:23:36.889 INFO:tasks.workunit.client.1.smithi134.stdout:0/118: dread d1/f1c [0,4194304] 0 2024-04-22T23:23:36.891 INFO:tasks.workunit.client.0.smithi067.stdout:0/956: stat db/d21/d29/d75/d9d/dde 0 2024-04-22T23:23:36.891 INFO:tasks.workunit.client.1.smithi134.stdout:7/108: chown d4/d10/f1f 16 1 2024-04-22T23:23:36.892 INFO:tasks.workunit.client.0.smithi067.stdout:0/957: write db/faa [2722618,82280] 0 2024-04-22T23:23:36.896 INFO:tasks.workunit.client.1.smithi134.stdout:3/79: dwrite dc/de/df/f1c [4194304,4194304] 0 2024-04-22T23:23:36.900 INFO:tasks.workunit.client.1.smithi134.stdout:7/109: mknod d4/c28 0 2024-04-22T23:23:36.903 INFO:tasks.workunit.client.0.smithi067.stdout:8/796: write d1/dd/d26/f22 [1029590,57040] 0 2024-04-22T23:23:36.904 INFO:tasks.workunit.client.1.smithi134.stdout:0/119: rename d1/d12/d17 to d1/d5/dd/d20 0 2024-04-22T23:23:36.905 INFO:tasks.workunit.client.1.smithi134.stdout:0/120: chown d1/d5/d15 4522337 1 2024-04-22T23:23:36.905 INFO:tasks.workunit.client.0.smithi067.stdout:8/797: fdatasync d1/d41/d46/fb0 0 2024-04-22T23:23:36.907 INFO:tasks.workunit.client.1.smithi134.stdout:0/121: write d1/d5/dd/d11/f18 [416221,29681] 0 2024-04-22T23:23:36.908 INFO:tasks.workunit.client.0.smithi067.stdout:2/773: dwrite d1/d10/d1c/d6d/f89 [0,4194304] 0 2024-04-22T23:23:36.909 INFO:tasks.workunit.client.1.smithi134.stdout:3/80: unlink f8 0 2024-04-22T23:23:36.912 INFO:tasks.workunit.client.0.smithi067.stdout:2/774: read - d1/d10/d1c/fc6 zero size 2024-04-22T23:23:36.915 INFO:tasks.workunit.client.0.smithi067.stdout:8/798: unlink d1/d3/d5c/d98/fa1 0 2024-04-22T23:23:36.916 INFO:tasks.workunit.client.0.smithi067.stdout:2/775: creat d1/d10/d72/fff x:0 0 0 2024-04-22T23:23:36.919 INFO:tasks.workunit.client.0.smithi067.stdout:2/776: mknod d1/d10/d41/c100 0 2024-04-22T23:23:36.919 INFO:tasks.workunit.client.0.smithi067.stdout:5/872: dread d5/f15 [0,4194304] 0 2024-04-22T23:23:36.921 INFO:tasks.workunit.client.0.smithi067.stdout:5/873: dread - d5/d1a/d103/d4d/da9/f120 zero size 2024-04-22T23:23:36.921 INFO:tasks.workunit.client.0.smithi067.stdout:8/799: mkdir d1/d101 0 2024-04-22T23:23:36.922 INFO:tasks.workunit.client.0.smithi067.stdout:2/777: fsync d1/d10/d72/d53/d9a/fb1 0 2024-04-22T23:23:36.923 INFO:tasks.workunit.client.0.smithi067.stdout:5/874: write d5/d28/f64 [1986069,42493] 0 2024-04-22T23:23:36.923 INFO:tasks.workunit.client.1.smithi134.stdout:5/129: sync 2024-04-22T23:23:36.924 INFO:tasks.workunit.client.1.smithi134.stdout:2/92: sync 2024-04-22T23:23:36.924 INFO:tasks.workunit.client.0.smithi067.stdout:2/778: mknod d1/d4f/c101 0 2024-04-22T23:23:36.925 INFO:tasks.workunit.client.0.smithi067.stdout:4/981: dread d12/d17/d88/fdc [0,4194304] 0 2024-04-22T23:23:36.926 INFO:tasks.workunit.client.0.smithi067.stdout:8/800: unlink d1/d3/c24 0 2024-04-22T23:23:36.927 INFO:tasks.workunit.client.0.smithi067.stdout:5/875: mknod d5/d1a/d103/d10d/c132 0 2024-04-22T23:23:36.928 INFO:tasks.workunit.client.0.smithi067.stdout:4/982: write d12/d5f/d2b/d12a/f136 [310773,33390] 0 2024-04-22T23:23:36.929 INFO:tasks.workunit.client.0.smithi067.stdout:4/983: chown d12/f8d 107 1 2024-04-22T23:23:36.931 INFO:tasks.workunit.client.0.smithi067.stdout:4/984: write d12/d42/d4b/d77/fee [2423661,11453] 0 2024-04-22T23:23:36.932 INFO:tasks.workunit.client.0.smithi067.stdout:2/779: link d1/d10/d41/cfa d1/d10/d33/c102 0 2024-04-22T23:23:36.935 INFO:tasks.workunit.client.0.smithi067.stdout:2/780: fsync d1/f1b 0 2024-04-22T23:23:36.936 INFO:tasks.workunit.client.0.smithi067.stdout:8/801: dwrite d1/dd/d26/d62/f91 [8388608,4194304] 0 2024-04-22T23:23:36.938 INFO:tasks.workunit.client.0.smithi067.stdout:2/781: write d1/d10/d41/f93 [3971341,30615] 0 2024-04-22T23:23:36.938 INFO:tasks.workunit.client.0.smithi067.stdout:8/802: fsync d1/dd/d80/d6f/dfc/fc5 0 2024-04-22T23:23:36.939 INFO:tasks.workunit.client.0.smithi067.stdout:2/782: write d1/d19/d6f/fd5 [456696,70644] 0 2024-04-22T23:23:36.942 INFO:tasks.workunit.client.0.smithi067.stdout:8/803: chown d1/dd/d80/d6f/f57 1641 1 2024-04-22T23:23:36.943 INFO:tasks.workunit.client.0.smithi067.stdout:8/804: fdatasync d1/dd/d80/d6f/f34 0 2024-04-22T23:23:36.944 INFO:tasks.workunit.client.0.smithi067.stdout:4/985: link d12/d17/d88/f126 d12/d42/d4b/d77/d8c/f14d 0 2024-04-22T23:23:36.982 INFO:tasks.workunit.client.1.smithi134.stdout:0/122: dwrite d1/d5/f1f [0,4194304] 0 2024-04-22T23:23:36.982 INFO:tasks.workunit.client.1.smithi134.stdout:7/110: mkdir d4/d8/d1d/d29 0 2024-04-22T23:23:36.983 INFO:tasks.workunit.client.1.smithi134.stdout:3/81: dread f9 [0,4194304] 0 2024-04-22T23:23:36.984 INFO:tasks.workunit.client.1.smithi134.stdout:7/111: read d4/d8/fd [795212,20603] 0 2024-04-22T23:23:36.985 INFO:tasks.workunit.client.1.smithi134.stdout:3/82: write dc/f18 [167946,56320] 0 2024-04-22T23:23:36.985 INFO:tasks.workunit.client.1.smithi134.stdout:0/123: write d1/d5/dd/f16 [2131855,75186] 0 2024-04-22T23:23:36.985 INFO:tasks.workunit.client.1.smithi134.stdout:5/130: unlink d9/fb 0 2024-04-22T23:23:36.985 INFO:tasks.workunit.client.1.smithi134.stdout:3/83: readlink lb 0 2024-04-22T23:23:36.987 INFO:tasks.workunit.client.1.smithi134.stdout:3/84: fdatasync dc/de/df/f1c 0 2024-04-22T23:23:36.989 INFO:tasks.workunit.client.1.smithi134.stdout:3/85: fsync dc/de/df/f1c 0 2024-04-22T23:23:36.989 INFO:tasks.workunit.client.1.smithi134.stdout:7/112: stat d4/fa 0 2024-04-22T23:23:36.992 INFO:tasks.workunit.client.1.smithi134.stdout:0/124: rename d1/d5/f1f to d1/d5/f21 0 2024-04-22T23:23:36.993 INFO:tasks.workunit.client.0.smithi067.stdout:1/761: sync 2024-04-22T23:23:36.993 INFO:tasks.workunit.client.0.smithi067.stdout:9/760: sync 2024-04-22T23:23:36.993 INFO:tasks.workunit.client.1.smithi134.stdout:5/131: rename d9/df/d20/l27 to d9/d11/l2e 0 2024-04-22T23:23:36.994 INFO:tasks.workunit.client.1.smithi134.stdout:2/93: symlink d2/dd/df/d1b/l23 0 2024-04-22T23:23:36.995 INFO:tasks.workunit.client.0.smithi067.stdout:2/783: creat d1/d42/d5f/d9b/dc0/f103 x:0 0 0 2024-04-22T23:23:36.996 INFO:tasks.workunit.client.0.smithi067.stdout:2/784: fsync d1/d10/d72/fff 0 2024-04-22T23:23:36.997 INFO:tasks.workunit.client.1.smithi134.stdout:0/125: mkdir d1/d12/d22 0 2024-04-22T23:23:36.998 INFO:tasks.workunit.client.0.smithi067.stdout:4/986: stat d12/l18 0 2024-04-22T23:23:36.998 INFO:tasks.workunit.client.1.smithi134.stdout:7/113: fdatasync d4/d10/f23 0 2024-04-22T23:23:37.000 INFO:tasks.workunit.client.0.smithi067.stdout:9/761: rename d5/d21/d47/d56/dc6/fd7 to d5/dd/de6/ffd 0 2024-04-22T23:23:37.001 INFO:tasks.workunit.client.1.smithi134.stdout:3/86: rename dc/l1b to dc/de/df/l26 0 2024-04-22T23:23:37.002 INFO:tasks.workunit.client.0.smithi067.stdout:2/785: mknod d1/d10/d72/d68/da1/c104 0 2024-04-22T23:23:37.002 INFO:tasks.workunit.client.1.smithi134.stdout:3/87: read dc/d19/f23 [425190,39197] 0 2024-04-22T23:23:37.003 INFO:tasks.workunit.client.0.smithi067.stdout:8/805: mkdir d1/dd/d80/db9/dd0/df9/d102 0 2024-04-22T23:23:37.004 INFO:tasks.workunit.client.0.smithi067.stdout:5/876: dwrite d5/d3e/f76 [0,4194304] 0 2024-04-22T23:23:37.004 INFO:tasks.workunit.client.1.smithi134.stdout:7/114: mkdir d4/d8/d2a 0 2024-04-22T23:23:37.005 INFO:tasks.workunit.client.0.smithi067.stdout:4/987: mkdir d12/d42/d4b/d14e 0 2024-04-22T23:23:37.005 INFO:tasks.workunit.client.1.smithi134.stdout:0/126: symlink d1/d12/d22/l23 0 2024-04-22T23:23:37.008 INFO:tasks.workunit.client.0.smithi067.stdout:5/877: truncate d5/d28/dae/f122 903273 0 2024-04-22T23:23:37.009 INFO:tasks.workunit.client.0.smithi067.stdout:5/878: chown d5/d28/d36/d40/d97/cf3 4 1 2024-04-22T23:23:37.010 INFO:tasks.workunit.client.1.smithi134.stdout:2/94: rename d2/dd/df/d1b to d2/d15/d24 0 2024-04-22T23:23:37.010 INFO:tasks.workunit.client.0.smithi067.stdout:4/988: mkdir d12/d17/d14f 0 2024-04-22T23:23:37.012 INFO:tasks.workunit.client.1.smithi134.stdout:3/88: symlink dc/dd/l27 0 2024-04-22T23:23:37.016 INFO:tasks.workunit.client.0.smithi067.stdout:4/989: mkdir d12/d42/d4b/d77/d8c/d150 0 2024-04-22T23:23:37.016 INFO:tasks.workunit.client.1.smithi134.stdout:5/132: getdents d9/df/d20 0 2024-04-22T23:23:37.017 INFO:tasks.workunit.client.1.smithi134.stdout:5/133: fsync d9/f19 0 2024-04-22T23:23:37.017 INFO:tasks.workunit.client.0.smithi067.stdout:1/762: dread d8/d9/d2f/d33/d66/ff4 [0,4194304] 0 2024-04-22T23:23:37.018 INFO:tasks.workunit.client.1.smithi134.stdout:2/95: symlink d2/dd/df/d1c/l25 0 2024-04-22T23:23:37.018 INFO:tasks.workunit.client.0.smithi067.stdout:5/879: truncate d5/db/f2f 3767240 0 2024-04-22T23:23:37.018 INFO:tasks.workunit.client.0.smithi067.stdout:1/763: fdatasync d8/d12/d36/d87/fdf 0 2024-04-22T23:23:37.019 INFO:tasks.workunit.client.0.smithi067.stdout:8/806: getdents d1/dd/d26/d62/ddf/ded 0 2024-04-22T23:23:37.019 INFO:tasks.workunit.client.0.smithi067.stdout:5/880: dread - d5/d1a/d103/d4d/d101/f12d zero size 2024-04-22T23:23:37.021 INFO:tasks.workunit.client.1.smithi134.stdout:3/89: rename dc/de/df to dc/dd/d28 0 2024-04-22T23:23:37.022 INFO:tasks.workunit.client.0.smithi067.stdout:4/990: rename d12/d5f/dae/dd0 to d12/d42/d4b/d122/d8e/d56/d151 0 2024-04-22T23:23:37.023 INFO:tasks.workunit.client.1.smithi134.stdout:0/127: link d1/d5/f7 d1/d12/d22/f24 0 2024-04-22T23:23:37.024 INFO:tasks.workunit.client.0.smithi067.stdout:5/881: symlink d5/d3e/d90/d11e/l133 0 2024-04-22T23:23:37.024 INFO:tasks.workunit.client.1.smithi134.stdout:5/134: unlink d9/f22 0 2024-04-22T23:23:37.025 INFO:tasks.workunit.client.1.smithi134.stdout:5/135: stat d9/d11 0 2024-04-22T23:23:37.026 INFO:tasks.workunit.client.1.smithi134.stdout:2/96: fsync d2/fb 0 2024-04-22T23:23:37.026 INFO:tasks.workunit.client.0.smithi067.stdout:8/807: write d1/dd/d80/d6f/db6/fc4 [2342,22870] 0 2024-04-22T23:23:37.027 INFO:tasks.workunit.client.1.smithi134.stdout:5/136: chown d9/d1e/c2c 394 1 2024-04-22T23:23:37.027 INFO:tasks.workunit.client.1.smithi134.stdout:3/90: dread f9 [0,4194304] 0 2024-04-22T23:23:37.027 INFO:tasks.workunit.client.0.smithi067.stdout:1/764: truncate d8/d9/d2d/d4d/d9c/faa 929577 0 2024-04-22T23:23:37.027 INFO:tasks.workunit.client.0.smithi067.stdout:9/762: truncate d5/dd/d18/d39/f64 538949 0 2024-04-22T23:23:37.028 INFO:tasks.workunit.client.1.smithi134.stdout:0/128: creat d1/d5/dd/d20/f25 x:0 0 0 2024-04-22T23:23:37.028 INFO:tasks.workunit.client.1.smithi134.stdout:3/91: chown dc/d19/f23 4 1 2024-04-22T23:23:37.029 INFO:tasks.workunit.client.1.smithi134.stdout:5/137: write d9/f19 [507615,36350] 0 2024-04-22T23:23:37.029 INFO:tasks.workunit.client.1.smithi134.stdout:2/97: creat d2/d15/d24/d1d/f26 x:0 0 0 2024-04-22T23:23:37.029 INFO:tasks.workunit.client.0.smithi067.stdout:5/882: symlink d5/db/d12/d44/l134 0 2024-04-22T23:23:37.030 INFO:tasks.workunit.client.0.smithi067.stdout:2/786: write d1/d4f/fe4 [2427680,86802] 0 2024-04-22T23:23:37.031 INFO:tasks.workunit.client.0.smithi067.stdout:8/808: mknod d1/dd/d26/c103 0 2024-04-22T23:23:37.031 INFO:tasks.workunit.client.0.smithi067.stdout:5/883: write d5/d28/d36/d40/f70 [7176162,36868] 0 2024-04-22T23:23:37.032 INFO:tasks.workunit.client.1.smithi134.stdout:0/129: write d1/f10 [2875660,127098] 0 2024-04-22T23:23:37.032 INFO:tasks.workunit.client.0.smithi067.stdout:8/809: fsync d1/fe2 0 2024-04-22T23:23:37.032 INFO:tasks.workunit.client.0.smithi067.stdout:1/765: fsync d8/d9/fdc 0 2024-04-22T23:23:37.035 INFO:tasks.workunit.client.0.smithi067.stdout:1/766: truncate d8/d9/d2f/d33/f3e 4402738 0 2024-04-22T23:23:37.037 INFO:tasks.workunit.client.0.smithi067.stdout:9/763: rename d5/c48 to d5/d21/d2c/cfe 0 2024-04-22T23:23:37.038 INFO:tasks.workunit.client.0.smithi067.stdout:8/810: dread d1/d3/d10/fba [0,4194304] 0 2024-04-22T23:23:37.038 INFO:tasks.workunit.client.0.smithi067.stdout:1/767: truncate d8/d9/d2d/d4d/d48/d9a/fac 441889 0 2024-04-22T23:23:37.038 INFO:tasks.workunit.client.1.smithi134.stdout:5/138: truncate d9/f2b 285143 0 2024-04-22T23:23:37.040 INFO:tasks.workunit.client.0.smithi067.stdout:2/787: rmdir d1/d10/d72/d53/d9a 39 2024-04-22T23:23:37.040 INFO:tasks.workunit.client.1.smithi134.stdout:7/115: dwrite d4/f9 [0,4194304] 0 2024-04-22T23:23:37.042 INFO:tasks.workunit.client.0.smithi067.stdout:5/884: unlink d5/d3e/d90/d11e/l130 0 2024-04-22T23:23:37.042 INFO:tasks.workunit.client.1.smithi134.stdout:2/98: unlink d2/c16 0 2024-04-22T23:23:37.046 INFO:tasks.workunit.client.1.smithi134.stdout:2/99: read d2/d15/d24/f1e [3313316,61227] 0 2024-04-22T23:23:37.046 INFO:tasks.workunit.client.0.smithi067.stdout:1/768: dwrite d8/d9/f25 [4194304,4194304] 0 2024-04-22T23:23:37.047 INFO:tasks.workunit.client.0.smithi067.stdout:8/811: rmdir d1/dd/d80/db9/dd0/df9 39 2024-04-22T23:23:37.047 INFO:tasks.workunit.client.0.smithi067.stdout:4/991: write d12/d42/d4b/d77/fe8 [89100,34393] 0 2024-04-22T23:23:37.049 INFO:tasks.workunit.client.1.smithi134.stdout:7/116: creat d4/d8/d1d/f2b x:0 0 0 2024-04-22T23:23:37.049 INFO:tasks.workunit.client.1.smithi134.stdout:5/139: dwrite d9/d11/f1b [4194304,4194304] 0 2024-04-22T23:23:37.049 INFO:tasks.workunit.client.0.smithi067.stdout:8/812: write d1/d3/f11 [4883250,111874] 0 2024-04-22T23:23:37.050 INFO:tasks.workunit.client.1.smithi134.stdout:0/130: write d1/f9 [5026405,47448] 0 2024-04-22T23:23:37.051 INFO:tasks.workunit.client.1.smithi134.stdout:2/100: creat d2/d15/d24/f27 x:0 0 0 2024-04-22T23:23:37.051 INFO:tasks.workunit.client.1.smithi134.stdout:3/92: getdents dc/dd/d28 0 2024-04-22T23:23:37.051 INFO:tasks.workunit.client.1.smithi134.stdout:0/131: fsync d1/d5/dd/d20/f25 0 2024-04-22T23:23:37.053 INFO:tasks.workunit.client.1.smithi134.stdout:5/140: symlink d9/df/l2f 0 2024-04-22T23:23:37.053 INFO:tasks.workunit.client.0.smithi067.stdout:9/764: rename d5/dd/d18/d39/d3e/c61 to d5/dd/d18/d39/d49/cff 0 2024-04-22T23:23:37.054 INFO:tasks.workunit.client.1.smithi134.stdout:5/141: chown d9/fa 2906 1 2024-04-22T23:23:37.055 INFO:tasks.workunit.client.1.smithi134.stdout:2/101: creat d2/d15/d24/d1d/f28 x:0 0 0 2024-04-22T23:23:37.057 INFO:tasks.workunit.client.1.smithi134.stdout:7/117: creat d4/d8/d2a/f2c x:0 0 0 2024-04-22T23:23:37.057 INFO:tasks.workunit.client.1.smithi134.stdout:3/93: mkdir dc/de/d21/d24/d29 0 2024-04-22T23:23:37.058 INFO:tasks.workunit.client.1.smithi134.stdout:2/102: truncate d2/d15/d24/d1d/f28 28738 0 2024-04-22T23:23:37.059 INFO:tasks.workunit.client.1.smithi134.stdout:5/142: unlink d9/df/l24 0 2024-04-22T23:23:37.061 INFO:tasks.workunit.client.1.smithi134.stdout:0/132: rename d1/d2/ce to d1/d5/d15/c26 0 2024-04-22T23:23:37.061 INFO:tasks.workunit.client.0.smithi067.stdout:4/992: readlink d12/l1c 0 2024-04-22T23:23:37.062 INFO:tasks.workunit.client.1.smithi134.stdout:3/94: rename dc/d19 to dc/d2a 0 2024-04-22T23:23:37.064 INFO:tasks.workunit.client.0.smithi067.stdout:2/788: symlink d1/d42/l105 0 2024-04-22T23:23:37.065 INFO:tasks.workunit.client.0.smithi067.stdout:5/885: rename d5/db/d12/f2c to d5/db/d12/d44/f135 0 2024-04-22T23:23:37.066 INFO:tasks.workunit.client.1.smithi134.stdout:2/103: rename d2/fb to d2/d15/d24/f29 0 2024-04-22T23:23:37.066 INFO:tasks.workunit.client.0.smithi067.stdout:5/886: stat d5/db/d12/dbb/d5f/f107 0 2024-04-22T23:23:37.067 INFO:tasks.workunit.client.1.smithi134.stdout:5/143: creat d9/dd/d17/d2a/f30 x:0 0 0 2024-04-22T23:23:37.069 INFO:tasks.workunit.client.1.smithi134.stdout:0/133: dwrite d1/d5/dd/d11/f18 [4194304,4194304] 0 2024-04-22T23:23:37.069 INFO:tasks.workunit.client.1.smithi134.stdout:3/95: unlink dc/c1a 0 2024-04-22T23:23:37.070 INFO:tasks.workunit.client.1.smithi134.stdout:5/144: truncate d9/f16 160416 0 2024-04-22T23:23:37.070 INFO:tasks.workunit.client.1.smithi134.stdout:7/118: creat d4/f2d x:0 0 0 2024-04-22T23:23:37.071 INFO:tasks.workunit.client.0.smithi067.stdout:1/769: creat d8/d9/d2f/d33/ff8 x:0 0 0 2024-04-22T23:23:37.071 INFO:tasks.workunit.client.0.smithi067.stdout:4/993: mknod d12/d17/d143/c152 0 2024-04-22T23:23:37.071 INFO:tasks.workunit.client.1.smithi134.stdout:2/104: fsync d2/d15/f19 0 2024-04-22T23:23:37.074 INFO:tasks.workunit.client.0.smithi067.stdout:5/887: dwrite d5/d28/daf/fe3 [0,4194304] 0 2024-04-22T23:23:37.074 INFO:tasks.workunit.client.1.smithi134.stdout:3/96: write f9 [3558508,16774] 0 2024-04-22T23:23:37.075 INFO:tasks.workunit.client.0.smithi067.stdout:8/813: getdents d1/dd/d80 0 2024-04-22T23:23:37.075 INFO:tasks.workunit.client.0.smithi067.stdout:8/814: stat d1 0 2024-04-22T23:23:37.076 INFO:tasks.workunit.client.1.smithi134.stdout:5/145: write d9/d1e/f23 [2751241,78060] 0 2024-04-22T23:23:37.076 INFO:tasks.workunit.client.0.smithi067.stdout:1/770: creat d8/d9/d2d/d4d/d9c/ff9 x:0 0 0 2024-04-22T23:23:37.076 INFO:tasks.workunit.client.0.smithi067.stdout:5/888: readlink d5/db/l1d 0 2024-04-22T23:23:37.078 INFO:tasks.workunit.client.0.smithi067.stdout:1/771: write d8/d9/de0/d60/d62/fa8 [1222021,14410] 0 2024-04-22T23:23:37.082 INFO:tasks.workunit.client.1.smithi134.stdout:2/105: rename d2/d15/f19 to d2/d15/d24/f2a 0 2024-04-22T23:23:37.082 INFO:tasks.workunit.client.1.smithi134.stdout:5/146: chown d9/df/l2f 2344 1 2024-04-22T23:23:37.089 INFO:tasks.workunit.client.1.smithi134.stdout:5/147: symlink d9/dd/d17/d2a/l31 0 2024-04-22T23:23:37.090 INFO:tasks.workunit.client.1.smithi134.stdout:0/134: dwrite d1/d5/dd/f16 [0,4194304] 0 2024-04-22T23:23:37.090 INFO:tasks.workunit.client.1.smithi134.stdout:7/119: dread d4/d10/f11 [0,4194304] 0 2024-04-22T23:23:37.098 INFO:tasks.workunit.client.1.smithi134.stdout:3/97: dwrite dc/f18 [0,4194304] 0 2024-04-22T23:23:37.098 INFO:tasks.workunit.client.1.smithi134.stdout:0/135: creat d1/d5/d15/f27 x:0 0 0 2024-04-22T23:23:37.107 INFO:tasks.workunit.client.1.smithi134.stdout:3/98: dread - dc/dd/d28/f17 zero size 2024-04-22T23:23:37.108 INFO:tasks.workunit.client.1.smithi134.stdout:0/136: mknod d1/d5/dd/c28 0 2024-04-22T23:23:37.108 INFO:tasks.workunit.client.1.smithi134.stdout:3/99: read dc/f22 [3381024,14977] 0 2024-04-22T23:23:37.110 INFO:tasks.workunit.client.1.smithi134.stdout:2/106: dwrite d2/d15/d24/d1d/f26 [0,4194304] 0 2024-04-22T23:23:37.111 INFO:tasks.workunit.client.1.smithi134.stdout:0/137: creat d1/d12/f29 x:0 0 0 2024-04-22T23:23:37.112 INFO:tasks.workunit.client.1.smithi134.stdout:0/138: chown d1/f10 150158 1 2024-04-22T23:23:37.112 INFO:tasks.workunit.client.1.smithi134.stdout:2/107: write d2/d15/d24/f27 [245607,69731] 0 2024-04-22T23:23:37.117 INFO:tasks.workunit.client.1.smithi134.stdout:3/100: link dc/d2a/c1e dc/c2b 0 2024-04-22T23:23:37.121 INFO:tasks.workunit.client.1.smithi134.stdout:3/101: symlink dc/de/d21/l2c 0 2024-04-22T23:23:37.124 INFO:tasks.workunit.client.1.smithi134.stdout:3/102: symlink dc/dd/l2d 0 2024-04-22T23:23:37.125 INFO:tasks.workunit.client.1.smithi134.stdout:2/108: dwrite d2/d15/d24/f27 [0,4194304] 0 2024-04-22T23:23:37.127 INFO:tasks.workunit.client.1.smithi134.stdout:0/139: dwrite d1/d5/dd/d20/f25 [0,4194304] 0 2024-04-22T23:23:37.130 INFO:tasks.workunit.client.0.smithi067.stdout:5/889: symlink d5/d1a/d103/d10d/l136 0 2024-04-22T23:23:37.132 INFO:tasks.workunit.client.0.smithi067.stdout:2/789: write d1/d10/d33/fa5 [760338,127813] 0 2024-04-22T23:23:37.132 INFO:tasks.workunit.client.1.smithi134.stdout:0/140: creat d1/d5/dd/f2a x:0 0 0 2024-04-22T23:23:37.134 INFO:tasks.workunit.client.0.smithi067.stdout:4/994: getdents d12/d42/d4b/d122/d8e/d56/d10e 0 2024-04-22T23:23:37.134 INFO:tasks.workunit.client.1.smithi134.stdout:3/103: creat dc/de/d21/d24/d29/f2e x:0 0 0 2024-04-22T23:23:37.135 INFO:tasks.workunit.client.0.smithi067.stdout:5/890: unlink d5/db/d12/d44/f56 0 2024-04-22T23:23:37.136 INFO:tasks.workunit.client.1.smithi134.stdout:2/109: fdatasync d2/d15/d24/d1d/f26 0 2024-04-22T23:23:37.136 INFO:tasks.workunit.client.0.smithi067.stdout:8/815: dwrite d1/dd/d26/f6b [0,4194304] 0 2024-04-22T23:23:37.138 INFO:tasks.workunit.client.0.smithi067.stdout:2/790: chown d1/d10/d72/fff 128958 1 2024-04-22T23:23:37.138 INFO:tasks.workunit.client.1.smithi134.stdout:7/120: dwrite d4/d10/f19 [0,4194304] 0 2024-04-22T23:23:37.139 INFO:tasks.workunit.client.1.smithi134.stdout:9/49: sync 2024-04-22T23:23:37.139 INFO:tasks.workunit.client.1.smithi134.stdout:1/72: sync 2024-04-22T23:23:37.139 INFO:tasks.workunit.client.0.smithi067.stdout:1/772: unlink d8/d9/d2f/d33/ff8 0 2024-04-22T23:23:37.140 INFO:tasks.workunit.client.0.smithi067.stdout:2/791: chown d1/d10/d72/d68 297 1 2024-04-22T23:23:37.141 INFO:tasks.workunit.client.1.smithi134.stdout:3/104: fdatasync dc/dd/d28/f1c 0 2024-04-22T23:23:37.142 INFO:tasks.workunit.client.1.smithi134.stdout:9/50: chown d0/fa 105305623 1 2024-04-22T23:23:37.144 INFO:tasks.workunit.client.1.smithi134.stdout:8/54: sync 2024-04-22T23:23:37.145 INFO:tasks.workunit.client.1.smithi134.stdout:8/55: stat d7/db/ld 0 2024-04-22T23:23:37.146 INFO:tasks.workunit.client.1.smithi134.stdout:5/148: truncate d9/dd/f13 489551 0 2024-04-22T23:23:37.147 INFO:tasks.workunit.client.0.smithi067.stdout:9/765: dwrite d5/dd/d18/d39/f64 [0,4194304] 0 2024-04-22T23:23:37.149 INFO:tasks.workunit.client.1.smithi134.stdout:1/73: creat db/f1d x:0 0 0 2024-04-22T23:23:37.150 INFO:tasks.workunit.client.0.smithi067.stdout:9/766: readlink d5/dd/da9/lcd 0 2024-04-22T23:23:37.152 INFO:tasks.workunit.client.1.smithi134.stdout:8/56: mkdir d7/d13 0 2024-04-22T23:23:37.152 INFO:tasks.workunit.client.1.smithi134.stdout:1/74: write db/d15/f18 [1788536,111201] 0 2024-04-22T23:23:37.152 INFO:tasks.workunit.client.1.smithi134.stdout:2/110: dwrite d2/d15/d24/d1d/f28 [0,4194304] 0 2024-04-22T23:23:37.153 INFO:tasks.workunit.client.0.smithi067.stdout:1/773: symlink d8/d9/de0/d60/lfa 0 2024-04-22T23:23:37.154 INFO:tasks.workunit.client.1.smithi134.stdout:8/57: write d7/dc/ff [486805,54150] 0 2024-04-22T23:23:37.154 INFO:tasks.workunit.client.0.smithi067.stdout:9/767: chown d5/d21/d47/d56/d8c/dd5/fe2 54659 1 2024-04-22T23:23:37.154 INFO:tasks.workunit.client.0.smithi067.stdout:4/995: dread d12/d42/d4b/d122/d8e/d56/d114/dc3/fd6 [0,4194304] 0 2024-04-22T23:23:37.155 INFO:tasks.workunit.client.1.smithi134.stdout:3/105: rename dc/de to dc/dd/d28/d2f 0 2024-04-22T23:23:37.158 INFO:tasks.workunit.client.0.smithi067.stdout:5/891: dwrite d5/d1a/d103/d4d/fde [0,4194304] 0 2024-04-22T23:23:37.160 INFO:tasks.workunit.client.0.smithi067.stdout:1/774: mknod d8/d9/de0/d60/cfb 0 2024-04-22T23:23:37.160 INFO:tasks.workunit.client.0.smithi067.stdout:9/768: creat d5/dd/d18/d5c/f100 x:0 0 0 2024-04-22T23:23:37.162 INFO:tasks.workunit.client.0.smithi067.stdout:2/792: dwrite d1/d42/d61/daa/fca [0,4194304] 0 2024-04-22T23:23:37.163 INFO:tasks.workunit.client.0.smithi067.stdout:9/769: stat d5/dd/d18/d39/d4d/faa 0 2024-04-22T23:23:37.163 INFO:tasks.workunit.client.1.smithi134.stdout:5/149: creat d9/d1e/f32 x:0 0 0 2024-04-22T23:23:37.163 INFO:tasks.workunit.client.0.smithi067.stdout:5/892: unlink d5/d28/daf/le9 0 2024-04-22T23:23:37.164 INFO:tasks.workunit.client.0.smithi067.stdout:1/775: truncate d8/d12/f54 615804 0 2024-04-22T23:23:37.164 INFO:tasks.workunit.client.0.smithi067.stdout:9/770: write d5/dd/d18/d5c/f87 [784853,39313] 0 2024-04-22T23:23:37.165 INFO:tasks.workunit.client.0.smithi067.stdout:5/893: chown d5/d28/d36/d40/d97/cf3 2909 1 2024-04-22T23:23:37.171 INFO:tasks.workunit.client.1.smithi134.stdout:2/111: mkdir d2/dd/d2b 0 2024-04-22T23:23:37.171 INFO:tasks.workunit.client.1.smithi134.stdout:8/58: dread f6 [0,4194304] 0 2024-04-22T23:23:37.171 INFO:tasks.workunit.client.1.smithi134.stdout:8/59: stat f4 0 2024-04-22T23:23:37.171 INFO:tasks.workunit.client.1.smithi134.stdout:7/121: creat d4/f2e x:0 0 0 2024-04-22T23:23:37.171 INFO:tasks.workunit.client.1.smithi134.stdout:3/106: mkdir dc/d2a/d30 0 2024-04-22T23:23:37.171 INFO:tasks.workunit.client.1.smithi134.stdout:5/150: chown d9/f19 29221761 1 2024-04-22T23:23:37.171 INFO:tasks.workunit.client.1.smithi134.stdout:8/60: creat d7/dc/f14 x:0 0 0 2024-04-22T23:23:37.171 INFO:tasks.workunit.client.1.smithi134.stdout:2/112: stat d2/dd 0 2024-04-22T23:23:37.172 INFO:tasks.workunit.client.0.smithi067.stdout:6/828: sync 2024-04-22T23:23:37.172 INFO:tasks.workunit.client.0.smithi067.stdout:3/816: sync 2024-04-22T23:23:37.172 INFO:tasks.workunit.client.0.smithi067.stdout:1/776: creat d8/d9/d2d/d4d/d9c/ffc x:0 0 0 2024-04-22T23:23:37.172 INFO:tasks.workunit.client.0.smithi067.stdout:3/817: fdatasync d3/d4/d10/d38/d53/d55/d5c/d70/f11f 0 2024-04-22T23:23:37.173 INFO:tasks.workunit.client.1.smithi134.stdout:9/51: link d0/f3 d0/d7/dc/ff 0 2024-04-22T23:23:37.173 INFO:tasks.workunit.client.0.smithi067.stdout:3/818: chown d3/d4/d10/d1f/d3c/df4 1391853 1 2024-04-22T23:23:37.174 INFO:tasks.workunit.client.0.smithi067.stdout:5/894: dwrite d5/db/d12/f22 [0,4194304] 0 2024-04-22T23:23:37.176 INFO:tasks.workunit.client.0.smithi067.stdout:3/819: write d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d8c/ff9 [2344960,38782] 0 2024-04-22T23:23:37.178 INFO:tasks.workunit.client.0.smithi067.stdout:3/820: write d3/d4/f11c [122257,23188] 0 2024-04-22T23:23:37.178 INFO:tasks.workunit.client.1.smithi134.stdout:8/61: creat d7/f15 x:0 0 0 2024-04-22T23:23:37.181 INFO:tasks.workunit.client.1.smithi134.stdout:9/52: unlink d0/l8 0 2024-04-22T23:23:37.182 INFO:tasks.workunit.client.1.smithi134.stdout:3/107: dread dc/f22 [0,4194304] 0 2024-04-22T23:23:37.185 INFO:tasks.workunit.client.1.smithi134.stdout:9/53: chown d0/d7/f9 11410 1 2024-04-22T23:23:37.187 INFO:tasks.workunit.client.1.smithi134.stdout:0/141: truncate d1/d2/f4 2402028 0 2024-04-22T23:23:37.189 INFO:tasks.workunit.client.1.smithi134.stdout:5/151: dwrite d9/d11/f25 [0,4194304] 0 2024-04-22T23:23:37.190 INFO:tasks.workunit.client.1.smithi134.stdout:7/122: dwrite d4/d10/f1f [0,4194304] 0 2024-04-22T23:23:37.190 INFO:tasks.workunit.client.1.smithi134.stdout:8/62: rmdir d7/db 39 2024-04-22T23:23:37.192 INFO:tasks.workunit.client.1.smithi134.stdout:1/75: link l4 db/d15/l1e 0 2024-04-22T23:23:37.195 INFO:tasks.workunit.client.1.smithi134.stdout:4/89: sync 2024-04-22T23:23:37.195 INFO:tasks.workunit.client.1.smithi134.stdout:6/129: sync 2024-04-22T23:23:37.197 INFO:tasks.workunit.client.1.smithi134.stdout:1/76: write db/f12 [489520,74720] 0 2024-04-22T23:23:37.197 INFO:tasks.workunit.client.1.smithi134.stdout:3/108: rename dc/dd/d28/f17 to dc/dd/d28/d2f/d21/d24/f31 0 2024-04-22T23:23:37.200 INFO:tasks.workunit.client.1.smithi134.stdout:5/152: creat d9/d1e/f33 x:0 0 0 2024-04-22T23:23:37.201 INFO:tasks.workunit.client.1.smithi134.stdout:2/113: creat d2/f2c x:0 0 0 2024-04-22T23:23:37.204 INFO:tasks.workunit.client.1.smithi134.stdout:9/54: dwrite d0/fa [0,4194304] 0 2024-04-22T23:23:37.204 INFO:tasks.workunit.client.1.smithi134.stdout:3/109: write dc/f18 [3808703,1858] 0 2024-04-22T23:23:37.204 INFO:tasks.workunit.client.0.smithi067.stdout:4/996: link d12/d42/d4b/d122/d8e/d56/f5c d12/d17/d3f/d13d/f153 0 2024-04-22T23:23:37.206 INFO:tasks.workunit.client.1.smithi134.stdout:5/153: write d9/fe [613673,40152] 0 2024-04-22T23:23:37.208 INFO:tasks.workunit.client.0.smithi067.stdout:5/895: unlink d5/d1a/d103/d10d/l136 0 2024-04-22T23:23:37.209 INFO:tasks.workunit.client.1.smithi134.stdout:4/90: unlink d4/dd/cb 0 2024-04-22T23:23:37.210 INFO:tasks.workunit.client.1.smithi134.stdout:1/77: symlink db/d15/l1f 0 2024-04-22T23:23:37.213 INFO:tasks.workunit.client.0.smithi067.stdout:8/816: dwrite d1/d41/d46/f9f [0,4194304] 0 2024-04-22T23:23:37.216 INFO:tasks.workunit.client.0.smithi067.stdout:4/997: mkdir d12/d42/d4b/d122/d8e/d154 0 2024-04-22T23:23:37.218 INFO:tasks.workunit.client.0.smithi067.stdout:6/829: mknod d17/d39/d53/d90/d46/d62/c114 0 2024-04-22T23:23:37.219 INFO:tasks.workunit.client.0.smithi067.stdout:6/830: stat d17/d39/d53/d90/d46/db0/db9/fd2 0 2024-04-22T23:23:37.221 INFO:tasks.workunit.client.0.smithi067.stdout:5/896: write d5/fa4 [1870644,1273] 0 2024-04-22T23:23:37.224 INFO:tasks.workunit.client.0.smithi067.stdout:8/817: mknod d1/dd/d26/d62/c104 0 2024-04-22T23:23:37.225 INFO:tasks.workunit.client.0.smithi067.stdout:4/998: creat d12/d42/d4b/d77/d8c/d150/f155 x:0 0 0 2024-04-22T23:23:37.226 INFO:tasks.workunit.client.0.smithi067.stdout:4/999: write d12/d5f/d7e/f133 [364852,77320] 0 2024-04-22T23:23:37.226 INFO:tasks.workunit.client.0.smithi067.stdout:6/831: creat d17/d39/d104/d106/f115 x:0 0 0 2024-04-22T23:23:37.228 INFO:tasks.workunit.client.0.smithi067.stdout:8/818: rename d1/d41/d46/lf8 to d1/dd/d80/l105 0 2024-04-22T23:23:37.228 INFO:tasks.workunit.client.0.smithi067.stdout:8/819: readlink d1/d3/l51 0 2024-04-22T23:23:37.229 INFO:tasks.workunit.client.0.smithi067.stdout:6/832: symlink d17/d39/d53/d90/l116 0 2024-04-22T23:23:37.231 INFO:tasks.workunit.client.0.smithi067.stdout:8/820: truncate d1/dd/d80/f8c 705639 0 2024-04-22T23:23:37.236 INFO:tasks.workunit.client.1.smithi134.stdout:8/63: rename d7/l8 to d7/dc/l16 0 2024-04-22T23:23:37.236 INFO:tasks.workunit.client.1.smithi134.stdout:7/123: mkdir d4/d10/d2f 0 2024-04-22T23:23:37.237 INFO:tasks.workunit.client.0.smithi067.stdout:6/833: dwrite d17/d39/f8c [0,4194304] 0 2024-04-22T23:23:37.237 INFO:tasks.workunit.client.1.smithi134.stdout:8/64: chown d7/dc/f12 3446636 1 2024-04-22T23:23:37.238 INFO:tasks.workunit.client.1.smithi134.stdout:9/55: mkdir d0/d10 0 2024-04-22T23:23:37.238 INFO:tasks.workunit.client.1.smithi134.stdout:8/65: write d7/dc/ff [595121,3852] 0 2024-04-22T23:23:37.239 INFO:tasks.workunit.client.1.smithi134.stdout:5/154: mkdir d9/d11/d34 0 2024-04-22T23:23:37.240 INFO:tasks.workunit.client.1.smithi134.stdout:3/110: mknod dc/dd/d28/d2f/d21/c32 0 2024-04-22T23:23:37.242 INFO:tasks.workunit.client.0.smithi067.stdout:8/821: dwrite d1/dd/d80/d6f/fad [0,4194304] 0 2024-04-22T23:23:37.243 INFO:tasks.workunit.client.1.smithi134.stdout:1/78: mkdir db/d15/d17/d20 0 2024-04-22T23:23:37.247 INFO:tasks.workunit.client.1.smithi134.stdout:7/124: creat d4/d8/f30 x:0 0 0 2024-04-22T23:23:37.252 INFO:tasks.workunit.client.1.smithi134.stdout:5/155: unlink d9/d11/c1f 0 2024-04-22T23:23:37.254 INFO:tasks.workunit.client.1.smithi134.stdout:3/111: creat dc/dd/d28/d2f/d21/f33 x:0 0 0 2024-04-22T23:23:37.255 INFO:tasks.workunit.client.0.smithi067.stdout:9/771: dwrite d5/d21/d47/d56/f80 [0,4194304] 0 2024-04-22T23:23:37.257 INFO:tasks.workunit.client.1.smithi134.stdout:4/91: dwrite d4/d7/f10 [0,4194304] 0 2024-04-22T23:23:37.258 INFO:tasks.workunit.client.1.smithi134.stdout:9/56: mkdir d0/d10/d11 0 2024-04-22T23:23:37.258 INFO:tasks.workunit.client.1.smithi134.stdout:9/57: readlink - no filename 2024-04-22T23:23:37.260 INFO:tasks.workunit.client.0.smithi067.stdout:3/821: dwrite d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d8c/fc5 [0,4194304] 0 2024-04-22T23:23:37.260 INFO:tasks.workunit.client.0.smithi067.stdout:5/897: write d5/db/d12/dbb/d5f/f107 [69974,28808] 0 2024-04-22T23:23:37.260 INFO:tasks.workunit.client.1.smithi134.stdout:2/114: dwrite d2/f14 [0,4194304] 0 2024-04-22T23:23:37.260 INFO:tasks.workunit.client.1.smithi134.stdout:3/112: dread dc/d2a/f23 [0,4194304] 0 2024-04-22T23:23:37.261 INFO:tasks.workunit.client.0.smithi067.stdout:5/898: fdatasync d5/d3e/d90/f94 0 2024-04-22T23:23:37.262 INFO:tasks.workunit.client.0.smithi067.stdout:9/772: fdatasync d5/d21/d2c/f7a 0 2024-04-22T23:23:37.263 INFO:tasks.workunit.client.1.smithi134.stdout:5/156: creat d9/d1e/f35 x:0 0 0 2024-04-22T23:23:37.264 INFO:tasks.workunit.client.0.smithi067.stdout:3/822: dread - d3/d4/d28/fd0 zero size 2024-04-22T23:23:37.265 INFO:tasks.workunit.client.1.smithi134.stdout:1/79: symlink db/d15/d17/d20/l21 0 2024-04-22T23:23:37.266 INFO:tasks.workunit.client.1.smithi134.stdout:5/157: chown d9/c28 3826 1 2024-04-22T23:23:37.266 INFO:tasks.workunit.client.1.smithi134.stdout:3/113: rmdir dc/dd/d28 39 2024-04-22T23:23:37.268 INFO:tasks.workunit.client.1.smithi134.stdout:9/58: fdatasync d0/d7/fb 0 2024-04-22T23:23:37.270 INFO:tasks.workunit.client.0.smithi067.stdout:3/823: creat d3/d4/dec/f124 x:0 0 0 2024-04-22T23:23:37.271 INFO:tasks.workunit.client.1.smithi134.stdout:5/158: chown f8 0 1 2024-04-22T23:23:37.272 INFO:tasks.workunit.client.1.smithi134.stdout:2/115: symlink d2/d15/l2d 0 2024-04-22T23:23:37.272 INFO:tasks.workunit.client.0.smithi067.stdout:9/773: getdents d5/dd/d18/d39 0 2024-04-22T23:23:37.272 INFO:tasks.workunit.client.0.smithi067.stdout:8/822: write d1/fde [1144280,2332] 0 2024-04-22T23:23:37.275 INFO:tasks.workunit.client.0.smithi067.stdout:5/899: write d5/d28/daf/dbf/dfc/d99/fc3 [1305405,54024] 0 2024-04-22T23:23:37.277 INFO:tasks.workunit.client.0.smithi067.stdout:3/824: creat d3/d4/d10/d38/d53/d55/d5c/f125 x:0 0 0 2024-04-22T23:23:37.277 INFO:tasks.workunit.client.1.smithi134.stdout:2/116: fdatasync d2/dd/df/d1c/f21 0 2024-04-22T23:23:37.279 INFO:tasks.workunit.client.0.smithi067.stdout:8/823: symlink d1/dd/d80/d6f/l106 0 2024-04-22T23:23:37.279 INFO:tasks.workunit.client.1.smithi134.stdout:6/130: dwrite d0/f10 [0,4194304] 0 2024-04-22T23:23:37.279 INFO:tasks.workunit.client.1.smithi134.stdout:7/125: creat d4/f31 x:0 0 0 2024-04-22T23:23:37.280 INFO:tasks.workunit.client.0.smithi067.stdout:3/825: readlink d3/d4/l15 0 2024-04-22T23:23:37.280 INFO:tasks.workunit.client.0.smithi067.stdout:9/774: mkdir d5/dd/d18/d3c/ddf/d101 0 2024-04-22T23:23:37.280 INFO:tasks.workunit.client.1.smithi134.stdout:2/117: stat d2/d15 0 2024-04-22T23:23:37.281 INFO:tasks.workunit.client.0.smithi067.stdout:9/775: chown d5/d21/d2c/l42 174989498 1 2024-04-22T23:23:37.282 INFO:tasks.workunit.client.1.smithi134.stdout:9/59: rmdir d0/d7 39 2024-04-22T23:23:37.282 INFO:tasks.workunit.client.0.smithi067.stdout:5/900: truncate d5/d28/daf/dbf/dcf/fcb 2339912 0 2024-04-22T23:23:37.282 INFO:tasks.workunit.client.0.smithi067.stdout:3/826: fdatasync d3/d4/d10/d1f/d3c/f25 0 2024-04-22T23:23:37.284 INFO:tasks.workunit.client.0.smithi067.stdout:8/824: mkdir d1/d3/d27/d107 0 2024-04-22T23:23:37.284 INFO:tasks.workunit.client.0.smithi067.stdout:5/901: chown d5/d1a/d103/f58 18685620 1 2024-04-22T23:23:37.285 INFO:tasks.workunit.client.1.smithi134.stdout:5/159: dread d9/f12 [0,4194304] 0 2024-04-22T23:23:37.286 INFO:tasks.workunit.client.0.smithi067.stdout:9/776: unlink d5/dd/d18/d39/d8f/cbf 0 2024-04-22T23:23:37.288 INFO:tasks.workunit.client.0.smithi067.stdout:6/834: dread d17/d39/d3a/f61 [0,4194304] 0 2024-04-22T23:23:37.288 INFO:tasks.workunit.client.0.smithi067.stdout:5/902: rmdir d5/d1a/d103/d10d 39 2024-04-22T23:23:37.291 INFO:tasks.workunit.client.1.smithi134.stdout:7/126: rename d4/d10/f19 to d4/d10/d2f/f32 0 2024-04-22T23:23:37.293 INFO:tasks.workunit.client.1.smithi134.stdout:6/131: creat d0/d19/de/d12/d29/f2e x:0 0 0 2024-04-22T23:23:37.297 INFO:tasks.workunit.client.0.smithi067.stdout:8/825: rename d1/dd/d80/db9/de3/db8/dd7/fe1 to d1/dd/d80/db9/f108 0 2024-04-22T23:23:37.297 INFO:tasks.workunit.client.0.smithi067.stdout:3/827: creat d3/d4/d10/d10b/df5/f126 x:0 0 0 2024-04-22T23:23:37.298 INFO:tasks.workunit.client.0.smithi067.stdout:9/777: creat d5/dd/d18/d39/d49/d54/d99/f102 x:0 0 0 2024-04-22T23:23:37.298 INFO:tasks.workunit.client.0.smithi067.stdout:5/903: write d5/d1a/d103/f5d [1499139,68176] 0 2024-04-22T23:23:37.300 INFO:tasks.workunit.client.1.smithi134.stdout:3/114: truncate dc/dd/d28/f1c 2989978 0 2024-04-22T23:23:37.300 INFO:tasks.workunit.client.0.smithi067.stdout:6/835: chown d17/l5d 5 1 2024-04-22T23:23:37.301 INFO:tasks.workunit.client.0.smithi067.stdout:5/904: write d5/d28/daf/dbf/dfc/d99/fc3 [1912323,57701] 0 2024-04-22T23:23:37.302 INFO:tasks.workunit.client.0.smithi067.stdout:6/836: read d17/d39/d53/f55 [4186110,14658] 0 2024-04-22T23:23:37.303 INFO:tasks.workunit.client.0.smithi067.stdout:6/837: dread - d17/d39/d53/d90/d46/f75 zero size 2024-04-22T23:23:37.305 INFO:tasks.workunit.client.1.smithi134.stdout:6/132: mkdir d0/d19/de/d2f 0 2024-04-22T23:23:37.306 INFO:tasks.workunit.client.1.smithi134.stdout:9/60: mkdir d0/d7/d12 0 2024-04-22T23:23:37.307 INFO:tasks.workunit.client.1.smithi134.stdout:1/80: getdents db/d15 0 2024-04-22T23:23:37.308 INFO:tasks.workunit.client.0.smithi067.stdout:7/744: sync 2024-04-22T23:23:37.309 INFO:tasks.workunit.client.1.smithi134.stdout:5/160: link d9/fc d9/f36 0 2024-04-22T23:23:37.309 INFO:tasks.workunit.client.1.smithi134.stdout:4/92: truncate d4/f6 1328490 0 2024-04-22T23:23:37.309 INFO:tasks.workunit.client.1.smithi134.stdout:5/161: write d9/dd/f10 [252448,99026] 0 2024-04-22T23:23:37.314 INFO:tasks.workunit.client.1.smithi134.stdout:2/118: creat d2/f2e x:0 0 0 2024-04-22T23:23:37.314 INFO:tasks.workunit.client.1.smithi134.stdout:9/61: symlink d0/d7/l13 0 2024-04-22T23:23:37.315 INFO:tasks.workunit.client.1.smithi134.stdout:1/81: symlink db/d15/l22 0 2024-04-22T23:23:37.319 INFO:tasks.workunit.client.1.smithi134.stdout:3/115: unlink dc/d2a/c1e 0 2024-04-22T23:23:37.321 INFO:tasks.workunit.client.1.smithi134.stdout:5/162: mknod d9/d11/c37 0 2024-04-22T23:23:37.321 INFO:tasks.workunit.client.1.smithi134.stdout:1/82: dread db/d15/f18 [0,4194304] 0 2024-04-22T23:23:37.322 INFO:tasks.workunit.client.1.smithi134.stdout:9/62: dwrite d0/fa [0,4194304] 0 2024-04-22T23:23:37.325 INFO:tasks.workunit.client.1.smithi134.stdout:7/127: creat d4/f33 x:0 0 0 2024-04-22T23:23:37.330 INFO:tasks.workunit.client.1.smithi134.stdout:2/119: dread d2/f18 [0,4194304] 0 2024-04-22T23:23:37.331 INFO:tasks.workunit.client.1.smithi134.stdout:6/133: stat d0/d19/de/d1f/f22 0 2024-04-22T23:23:37.332 INFO:tasks.workunit.client.1.smithi134.stdout:6/134: chown d0/f1 439 1 2024-04-22T23:23:37.333 INFO:tasks.workunit.client.1.smithi134.stdout:3/116: mkdir dc/dd/d28/d34 0 2024-04-22T23:23:37.334 INFO:tasks.workunit.client.1.smithi134.stdout:3/117: read f9 [2185882,106128] 0 2024-04-22T23:23:37.335 INFO:tasks.workunit.client.0.smithi067.stdout:8/826: creat d1/dd/d26/d62/d75/f109 x:0 0 0 2024-04-22T23:23:37.335 INFO:tasks.workunit.client.1.smithi134.stdout:3/118: fdatasync dc/dd/d28/d2f/d21/d24/d29/f2e 0 2024-04-22T23:23:37.336 INFO:tasks.workunit.client.0.smithi067.stdout:3/828: symlink d3/d4/d10/d1f/d3c/l127 0 2024-04-22T23:23:37.337 INFO:tasks.workunit.client.0.smithi067.stdout:9/778: mkdir d5/d21/d47/d56/dc6/d103 0 2024-04-22T23:23:37.337 INFO:tasks.workunit.client.1.smithi134.stdout:1/83: mknod db/d16/c23 0 2024-04-22T23:23:37.338 INFO:tasks.workunit.client.1.smithi134.stdout:1/84: chown db/f12 715 1 2024-04-22T23:23:37.339 INFO:tasks.workunit.client.1.smithi134.stdout:1/85: write db/ff [339795,27214] 0 2024-04-22T23:23:37.341 INFO:tasks.workunit.client.1.smithi134.stdout:7/128: mkdir d4/d10/d2f/d34 0 2024-04-22T23:23:37.342 INFO:tasks.workunit.client.0.smithi067.stdout:8/827: mknod d1/d3/d10/d3b/c10a 0 2024-04-22T23:23:37.344 INFO:tasks.workunit.client.0.smithi067.stdout:3/829: dwrite d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d8c/fc5 [0,4194304] 0 2024-04-22T23:23:37.344 INFO:tasks.workunit.client.1.smithi134.stdout:2/120: symlink d2/d15/d24/d1d/l2f 0 2024-04-22T23:23:37.345 INFO:tasks.workunit.client.0.smithi067.stdout:6/838: dread fe [0,4194304] 0 2024-04-22T23:23:37.346 INFO:tasks.workunit.client.0.smithi067.stdout:9/779: rename d5/d21/faf to d5/dd/d18/d5c/f104 0 2024-04-22T23:23:37.349 INFO:tasks.workunit.client.1.smithi134.stdout:5/163: rename d9/d11/l2e to d9/df/d20/l38 0 2024-04-22T23:23:37.350 INFO:tasks.workunit.client.0.smithi067.stdout:7/745: mkdir d10/d14/d1d/d26/d80/dd2/dd4/d115 0 2024-04-22T23:23:37.351 INFO:tasks.workunit.client.1.smithi134.stdout:9/63: symlink d0/d7/d12/l14 0 2024-04-22T23:23:37.353 INFO:tasks.workunit.client.0.smithi067.stdout:7/746: chown d10/d14/d1d/d26/d2c/df4 1 1 2024-04-22T23:23:37.353 INFO:tasks.workunit.client.1.smithi134.stdout:4/93: dwrite d4/f6 [0,4194304] 0 2024-04-22T23:23:37.354 INFO:tasks.workunit.client.0.smithi067.stdout:8/828: symlink d1/dd/d26/d62/d75/l10b 0 2024-04-22T23:23:37.354 INFO:tasks.workunit.client.1.smithi134.stdout:5/164: write d9/d1e/f23 [4982093,65639] 0 2024-04-22T23:23:37.354 INFO:tasks.workunit.client.1.smithi134.stdout:7/129: dwrite d4/d8/ff [4194304,4194304] 0 2024-04-22T23:23:37.354 INFO:tasks.workunit.client.0.smithi067.stdout:0/958: sync 2024-04-22T23:23:37.355 INFO:tasks.workunit.client.0.smithi067.stdout:6/839: dwrite d17/d39/d53/d90/d46/f10a [0,4194304] 0 2024-04-22T23:23:37.356 INFO:tasks.workunit.client.1.smithi134.stdout:1/86: mknod db/d15/d17/d20/c24 0 2024-04-22T23:23:37.358 INFO:tasks.workunit.client.0.smithi067.stdout:3/830: dread d3/d4/d28/d40/dfd/dfe/d4b/f8f [0,4194304] 0 2024-04-22T23:23:37.360 INFO:tasks.workunit.client.1.smithi134.stdout:2/121: dread - d2/f22 zero size 2024-04-22T23:23:37.364 INFO:tasks.workunit.client.0.smithi067.stdout:9/780: fdatasync d5/dd/d18/d39/d8f/ffc 0 2024-04-22T23:23:37.364 INFO:tasks.workunit.client.1.smithi134.stdout:1/87: dwrite fa [0,4194304] 0 2024-04-22T23:23:37.365 INFO:tasks.workunit.client.0.smithi067.stdout:5/905: getdents d5/d1a/d103/d4d/d104 0 2024-04-22T23:23:37.366 INFO:tasks.workunit.client.1.smithi134.stdout:3/119: rename dc/dd/d28/d2f/d21/d24 to dc/dd/d28/d2f/d35 0 2024-04-22T23:23:37.367 INFO:tasks.workunit.client.0.smithi067.stdout:7/747: creat d10/d14/d1d/d26/d2c/d7a/dc2/dd0/de4/f116 x:0 0 0 2024-04-22T23:23:37.368 INFO:tasks.workunit.client.1.smithi134.stdout:2/122: dread d2/d15/d24/f27 [0,4194304] 0 2024-04-22T23:23:37.369 INFO:tasks.workunit.client.1.smithi134.stdout:2/123: dread - d2/f2c zero size 2024-04-22T23:23:37.369 INFO:tasks.workunit.client.0.smithi067.stdout:9/781: chown d5/dd/d18/d3c/d40/f50 111704 1 2024-04-22T23:23:37.373 INFO:tasks.workunit.client.0.smithi067.stdout:8/829: dwrite d1/dd/d26/f6b [0,4194304] 0 2024-04-22T23:23:37.373 INFO:tasks.workunit.client.0.smithi067.stdout:7/748: stat d10/d14/d1d/d26/d2c/d7a/cb7 0 2024-04-22T23:23:37.373 INFO:tasks.workunit.client.0.smithi067.stdout:0/959: mkdir db/d9a/d56/d70/dba/d13b 0 2024-04-22T23:23:37.373 INFO:tasks.workunit.client.0.smithi067.stdout:6/840: rename d17/d83/d9b/fa3 to d17/d39/d53/d90/d46/d62/d88/f117 0 2024-04-22T23:23:37.375 INFO:tasks.workunit.client.0.smithi067.stdout:9/782: write d5/dd/f26 [3847447,81100] 0 2024-04-22T23:23:37.377 INFO:tasks.workunit.client.0.smithi067.stdout:8/830: write d1/d41/f74 [2578293,69959] 0 2024-04-22T23:23:37.378 INFO:tasks.workunit.client.0.smithi067.stdout:3/831: creat d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d90/da2/daf/dd8/f128 x:0 0 0 2024-04-22T23:23:37.378 INFO:tasks.workunit.client.1.smithi134.stdout:6/135: truncate d0/f16 4148969 0 2024-04-22T23:23:37.380 INFO:tasks.workunit.client.1.smithi134.stdout:2/124: dwrite d2/d15/d24/f27 [0,4194304] 0 2024-04-22T23:23:37.381 INFO:tasks.workunit.client.1.smithi134.stdout:6/136: fsync d0/d19/de/d1f/f2c 0 2024-04-22T23:23:37.381 INFO:tasks.workunit.client.0.smithi067.stdout:8/831: chown d1/dd/d80/d6f 3229 1 2024-04-22T23:23:37.383 INFO:tasks.workunit.client.1.smithi134.stdout:2/125: write d2/d15/d24/d1d/f26 [3184044,11715] 0 2024-04-22T23:23:37.386 INFO:tasks.workunit.client.1.smithi134.stdout:4/94: mkdir d4/d15 0 2024-04-22T23:23:37.386 INFO:tasks.workunit.client.0.smithi067.stdout:3/832: stat d3/d4/dec/f124 0 2024-04-22T23:23:37.386 INFO:tasks.workunit.client.1.smithi134.stdout:1/88: rename c6 to db/d16/c25 0 2024-04-22T23:23:37.387 INFO:tasks.workunit.client.1.smithi134.stdout:7/130: creat d4/d10/d2f/d34/f35 x:0 0 0 2024-04-22T23:23:37.388 INFO:tasks.workunit.client.1.smithi134.stdout:7/131: fsync d4/d8/f22 0 2024-04-22T23:23:37.390 INFO:tasks.workunit.client.1.smithi134.stdout:3/120: stat dc/dd/d28/l26 0 2024-04-22T23:23:37.390 INFO:tasks.workunit.client.1.smithi134.stdout:7/132: dread - d4/d10/f1c zero size 2024-04-22T23:23:37.393 INFO:tasks.workunit.client.1.smithi134.stdout:2/126: stat d2/c11 0 2024-04-22T23:23:37.393 INFO:tasks.workunit.client.0.smithi067.stdout:7/749: symlink d10/d14/d1d/d46/l117 0 2024-04-22T23:23:37.394 INFO:tasks.workunit.client.1.smithi134.stdout:5/165: write d9/f36 [997253,110577] 0 2024-04-22T23:23:37.394 INFO:tasks.workunit.client.0.smithi067.stdout:0/960: dread db/d9a/f63 [0,4194304] 0 2024-04-22T23:23:37.395 INFO:tasks.workunit.client.1.smithi134.stdout:1/89: creat db/f26 x:0 0 0 2024-04-22T23:23:37.395 INFO:tasks.workunit.client.1.smithi134.stdout:1/90: chown db/f26 13 1 2024-04-22T23:23:37.396 INFO:tasks.workunit.client.0.smithi067.stdout:6/841: rename d17/d39/d3a/da9/cce to d17/d39/d53/d90/d46/d62/d88/c118 0 2024-04-22T23:23:37.396 INFO:tasks.workunit.client.1.smithi134.stdout:1/91: write db/f1d [668254,62065] 0 2024-04-22T23:23:37.397 INFO:tasks.workunit.client.1.smithi134.stdout:3/121: mkdir dc/dd/d28/d2f/d36 0 2024-04-22T23:23:37.398 INFO:tasks.workunit.client.0.smithi067.stdout:3/833: write d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d90/da2/daf/dd8/f128 [1024492,102148] 0 2024-04-22T23:23:37.398 INFO:tasks.workunit.client.0.smithi067.stdout:8/832: dwrite d1/d3/f56 [0,4194304] 0 2024-04-22T23:23:37.398 INFO:tasks.workunit.client.0.smithi067.stdout:7/750: stat d10/d14/d1d/d26/d2c/d7a/dc2/dd0/de4 0 2024-04-22T23:23:37.399 INFO:tasks.workunit.client.1.smithi134.stdout:7/133: creat d4/d10/d2f/f36 x:0 0 0 2024-04-22T23:23:37.399 INFO:tasks.workunit.client.1.smithi134.stdout:3/122: dread dc/dd/d28/d2f/d21/f16 [0,4194304] 0 2024-04-22T23:23:37.400 INFO:tasks.workunit.client.0.smithi067.stdout:5/906: link d5/db/d12/lab d5/d28/daf/dbf/l137 0 2024-04-22T23:23:37.400 INFO:tasks.workunit.client.1.smithi134.stdout:6/137: creat d0/d19/de/d12/d2a/f30 x:0 0 0 2024-04-22T23:23:37.402 INFO:tasks.workunit.client.1.smithi134.stdout:4/95: creat d4/dd/f16 x:0 0 0 2024-04-22T23:23:37.403 INFO:tasks.workunit.client.0.smithi067.stdout:3/834: creat d3/d4/d28/d4a/f129 x:0 0 0 2024-04-22T23:23:37.405 INFO:tasks.workunit.client.0.smithi067.stdout:7/751: rmdir d10/d14/d1d/d46/d6a/dcc 39 2024-04-22T23:23:37.407 INFO:tasks.workunit.client.0.smithi067.stdout:8/833: creat d1/dd/d80/d6f/dfc/de8/f10c x:0 0 0 2024-04-22T23:23:37.409 INFO:tasks.workunit.client.0.smithi067.stdout:5/907: creat d5/d1a/d103/d4d/da9/f138 x:0 0 0 2024-04-22T23:23:37.410 INFO:tasks.workunit.client.0.smithi067.stdout:7/752: creat d10/d47/dab/f118 x:0 0 0 2024-04-22T23:23:37.412 INFO:tasks.workunit.client.0.smithi067.stdout:2/793: sync 2024-04-22T23:23:37.412 INFO:tasks.workunit.client.0.smithi067.stdout:1/777: sync 2024-04-22T23:23:37.412 INFO:tasks.workunit.client.0.smithi067.stdout:6/842: creat d17/d39/f119 x:0 0 0 2024-04-22T23:23:37.412 INFO:tasks.workunit.client.1.smithi134.stdout:5/166: fsync d9/d11/f21 0 2024-04-22T23:23:37.415 INFO:tasks.workunit.client.1.smithi134.stdout:7/134: dwrite d4/d10/d2f/f32 [4194304,4194304] 0 2024-04-22T23:23:37.415 INFO:tasks.workunit.client.0.smithi067.stdout:8/834: rename d1/d41/d46/fce to d1/dd/d80/f10d 0 2024-04-22T23:23:37.416 INFO:tasks.workunit.client.1.smithi134.stdout:7/135: readlink l2 0 2024-04-22T23:23:37.416 INFO:tasks.workunit.client.1.smithi134.stdout:6/138: getdents d0/d19/de/d12/d2a/d2d 0 2024-04-22T23:23:37.416 INFO:tasks.workunit.client.0.smithi067.stdout:5/908: creat d5/d106/f139 x:0 0 0 2024-04-22T23:23:37.417 INFO:tasks.workunit.client.0.smithi067.stdout:8/835: truncate d1/d3/d10/fe0 224367 0 2024-04-22T23:23:37.419 INFO:tasks.workunit.client.0.smithi067.stdout:5/909: write d5/d1a/d103/d4d/da9/f138 [438190,67445] 0 2024-04-22T23:23:37.421 INFO:tasks.workunit.client.1.smithi134.stdout:5/167: creat d9/df/d20/f39 x:0 0 0 2024-04-22T23:23:37.423 INFO:tasks.workunit.client.1.smithi134.stdout:2/127: link d2/f18 d2/f30 0 2024-04-22T23:23:37.425 INFO:tasks.workunit.client.1.smithi134.stdout:7/136: unlink d4/f9 0 2024-04-22T23:23:37.425 INFO:tasks.workunit.client.1.smithi134.stdout:1/92: getdents db/d15/d17 0 2024-04-22T23:23:37.426 INFO:tasks.workunit.client.1.smithi134.stdout:7/137: write d4/d10/f1c [28299,118609] 0 2024-04-22T23:23:37.427 INFO:tasks.workunit.client.0.smithi067.stdout:2/794: unlink d1/d10/d72/d53/f97 0 2024-04-22T23:23:37.427 INFO:tasks.workunit.client.1.smithi134.stdout:1/93: read db/fc [3598141,34268] 0 2024-04-22T23:23:37.428 INFO:tasks.workunit.client.1.smithi134.stdout:1/94: stat db/d15/c19 0 2024-04-22T23:23:37.429 INFO:tasks.workunit.client.1.smithi134.stdout:3/123: link dc/dd/d28/d2f/l10 dc/dd/l37 0 2024-04-22T23:23:37.429 INFO:tasks.workunit.client.0.smithi067.stdout:7/753: dwrite d10/d18/d88/daf/d100/f5e [0,4194304] 0 2024-04-22T23:23:37.429 INFO:tasks.workunit.client.0.smithi067.stdout:8/836: rmdir d1/d3 39 2024-04-22T23:23:37.431 INFO:tasks.workunit.client.0.smithi067.stdout:8/837: truncate d1/fc0 1393395 0 2024-04-22T23:23:37.431 INFO:tasks.workunit.client.0.smithi067.stdout:0/961: dread db/de/d11/f32 [0,4194304] 0 2024-04-22T23:23:37.432 INFO:tasks.workunit.client.0.smithi067.stdout:7/754: fdatasync d10/d14/d1d/d26/d2c/f2e 0 2024-04-22T23:23:37.433 INFO:tasks.workunit.client.0.smithi067.stdout:8/838: chown d1/dd/d80/d6f/l106 15 1 2024-04-22T23:23:37.437 INFO:tasks.workunit.client.1.smithi134.stdout:3/124: mknod dc/dd/d28/c38 0 2024-04-22T23:23:37.437 INFO:tasks.workunit.client.1.smithi134.stdout:7/138: write d4/f7 [5425507,54608] 0 2024-04-22T23:23:37.439 INFO:tasks.workunit.client.0.smithi067.stdout:1/778: getdents d8/d9/d2d/d4d/d48 0 2024-04-22T23:23:37.439 INFO:tasks.workunit.client.1.smithi134.stdout:4/96: getdents d4/d7 0 2024-04-22T23:23:37.439 INFO:tasks.workunit.client.1.smithi134.stdout:3/125: dread - dc/dd/d28/d2f/d35/f31 zero size 2024-04-22T23:23:37.441 INFO:tasks.workunit.client.1.smithi134.stdout:6/139: rename d0/d19/de/d12/d29/f2e to d0/d19/de/f31 0 2024-04-22T23:23:37.445 INFO:tasks.workunit.client.1.smithi134.stdout:7/139: mkdir d4/d10/d2f/d37 0 2024-04-22T23:23:37.445 INFO:tasks.workunit.client.0.smithi067.stdout:8/839: mknod d1/dd/d26/c10e 0 2024-04-22T23:23:37.446 INFO:tasks.workunit.client.0.smithi067.stdout:5/910: dread d5/d28/daf/dbf/dfc/d99/f12b [0,4194304] 0 2024-04-22T23:23:37.448 INFO:tasks.workunit.client.0.smithi067.stdout:8/840: dread - d1/dd/d80/fdb zero size 2024-04-22T23:23:37.448 INFO:tasks.workunit.client.1.smithi134.stdout:1/95: rename f0 to db/f27 0 2024-04-22T23:23:37.449 INFO:tasks.workunit.client.1.smithi134.stdout:2/128: write d2/f9 [738401,100078] 0 2024-04-22T23:23:37.450 INFO:tasks.workunit.client.1.smithi134.stdout:5/168: dwrite d9/dd/f13 [0,4194304] 0 2024-04-22T23:23:37.450 INFO:tasks.workunit.client.1.smithi134.stdout:1/96: chown db/l11 21330 1 2024-04-22T23:23:37.450 INFO:tasks.workunit.client.0.smithi067.stdout:1/779: dread d8/d9/d2d/f4f [0,4194304] 0 2024-04-22T23:23:37.450 INFO:tasks.workunit.client.1.smithi134.stdout:4/97: mkdir d4/d17 0 2024-04-22T23:23:37.454 INFO:tasks.workunit.client.1.smithi134.stdout:3/126: dread f9 [0,4194304] 0 2024-04-22T23:23:37.456 INFO:tasks.workunit.client.1.smithi134.stdout:6/140: creat d0/d19/de/d2f/f32 x:0 0 0 2024-04-22T23:23:37.457 INFO:tasks.workunit.client.0.smithi067.stdout:5/911: dread d5/db/d35/f8e [0,4194304] 0 2024-04-22T23:23:37.457 INFO:tasks.workunit.client.1.smithi134.stdout:3/127: readlink dc/dd/d28/l26 0 2024-04-22T23:23:37.459 INFO:tasks.workunit.client.0.smithi067.stdout:8/841: fsync d1/dd/d26/d62/d94/fa3 0 2024-04-22T23:23:37.460 INFO:tasks.workunit.client.1.smithi134.stdout:4/98: write d4/fe [4418114,83600] 0 2024-04-22T23:23:37.461 INFO:tasks.workunit.client.1.smithi134.stdout:3/128: fsync dc/dd/d28/d2f/d35/d29/f2e 0 2024-04-22T23:23:37.461 INFO:tasks.workunit.client.0.smithi067.stdout:7/755: link d10/d18/d88/daf/lcb d10/d14/d1d/d46/d6a/dcc/d106/l119 0 2024-04-22T23:23:37.462 INFO:tasks.workunit.client.1.smithi134.stdout:5/169: dwrite f6 [0,4194304] 0 2024-04-22T23:23:37.462 INFO:tasks.workunit.client.1.smithi134.stdout:1/97: read db/d15/f18 [525056,89620] 0 2024-04-22T23:23:37.462 INFO:tasks.workunit.client.1.smithi134.stdout:3/129: chown dc/dd 4318 1 2024-04-22T23:23:37.464 INFO:tasks.workunit.client.1.smithi134.stdout:4/99: stat d4/dd/l12 0 2024-04-22T23:23:37.465 INFO:tasks.workunit.client.1.smithi134.stdout:5/170: stat d9/df/l2f 0 2024-04-22T23:23:37.466 INFO:tasks.workunit.client.0.smithi067.stdout:1/780: creat d8/d9/d2f/d31/ded/ffd x:0 0 0 2024-04-22T23:23:37.468 INFO:tasks.workunit.client.0.smithi067.stdout:0/962: dread db/d9a/d56/d6b/db9/f104 [4194304,4194304] 0 2024-04-22T23:23:37.470 INFO:tasks.workunit.client.0.smithi067.stdout:5/912: dread - d5/d28/dae/f119 zero size 2024-04-22T23:23:37.470 INFO:tasks.workunit.client.1.smithi134.stdout:1/98: link db/f14 db/f28 0 2024-04-22T23:23:37.473 INFO:tasks.workunit.client.1.smithi134.stdout:2/129: dwrite d2/d15/d24/f1e [0,4194304] 0 2024-04-22T23:23:37.473 INFO:tasks.workunit.client.0.smithi067.stdout:8/842: symlink d1/dd/d26/d62/ddf/ded/def/l10f 0 2024-04-22T23:23:37.473 INFO:tasks.workunit.client.1.smithi134.stdout:5/171: write d9/f12 [530732,74283] 0 2024-04-22T23:23:37.474 INFO:tasks.workunit.client.0.smithi067.stdout:9/783: dwrite d5/dd/d18/d39/d49/d54/fc9 [0,4194304] 0 2024-04-22T23:23:37.475 INFO:tasks.workunit.client.0.smithi067.stdout:7/756: dwrite fc [0,4194304] 0 2024-04-22T23:23:37.476 INFO:tasks.workunit.client.1.smithi134.stdout:1/99: read f9 [2562927,76081] 0 2024-04-22T23:23:37.479 INFO:tasks.workunit.client.1.smithi134.stdout:7/140: dwrite d4/d8/fb [0,4194304] 0 2024-04-22T23:23:37.479 INFO:tasks.workunit.client.0.smithi067.stdout:9/784: write d5/d21/f2b [475864,9768] 0 2024-04-22T23:23:37.483 INFO:tasks.workunit.client.0.smithi067.stdout:6/843: truncate d17/d83/f86 3891140 0 2024-04-22T23:23:37.485 INFO:tasks.workunit.client.1.smithi134.stdout:5/172: mknod d9/c3a 0 2024-04-22T23:23:37.486 INFO:tasks.workunit.client.0.smithi067.stdout:5/913: unlink d5/d28/dae/le5 0 2024-04-22T23:23:37.487 INFO:tasks.workunit.client.0.smithi067.stdout:2/795: write d1/f69 [714337,27436] 0 2024-04-22T23:23:37.489 INFO:tasks.workunit.client.1.smithi134.stdout:3/130: dread dc/f18 [0,4194304] 0 2024-04-22T23:23:37.489 INFO:tasks.workunit.client.1.smithi134.stdout:2/130: symlink d2/d15/d24/l31 0 2024-04-22T23:23:37.489 INFO:tasks.workunit.client.0.smithi067.stdout:7/757: symlink d10/d14/d1d/d26/d2c/l11a 0 2024-04-22T23:23:37.492 INFO:tasks.workunit.client.1.smithi134.stdout:4/100: dwrite d4/fe [0,4194304] 0 2024-04-22T23:23:37.492 INFO:tasks.workunit.client.1.smithi134.stdout:5/173: chown d9/d11/f25 0 1 2024-04-22T23:23:37.495 INFO:tasks.workunit.client.0.smithi067.stdout:9/785: mkdir d5/d21/d47/d57/dc0/d105 0 2024-04-22T23:23:37.498 INFO:tasks.workunit.client.1.smithi134.stdout:2/131: unlink d2/f9 0 2024-04-22T23:23:37.499 INFO:tasks.workunit.client.0.smithi067.stdout:6/844: dread d17/d39/d53/d90/d46/db0/db9/fc3 [0,4194304] 0 2024-04-22T23:23:37.500 INFO:tasks.workunit.client.1.smithi134.stdout:5/174: dread d9/dd/f13 [0,4194304] 0 2024-04-22T23:23:37.502 INFO:tasks.workunit.client.1.smithi134.stdout:3/131: fdatasync dc/dd/d28/d2f/d21/f16 0 2024-04-22T23:23:37.502 INFO:tasks.workunit.client.1.smithi134.stdout:4/101: symlink d4/d13/l18 0 2024-04-22T23:23:37.504 INFO:tasks.workunit.client.0.smithi067.stdout:9/786: creat d5/dd/d18/f106 x:0 0 0 2024-04-22T23:23:37.505 INFO:tasks.workunit.client.1.smithi134.stdout:7/141: link d4/d8/l1e d4/d8/d2a/l38 0 2024-04-22T23:23:37.505 INFO:tasks.workunit.client.0.smithi067.stdout:9/787: chown d5/dd/d18/d3c/d78/ce7 70 1 2024-04-22T23:23:37.506 INFO:tasks.workunit.client.0.smithi067.stdout:5/914: link d5/d28/d36/d40/f110 d5/db/d12/d44/f13a 0 2024-04-22T23:23:37.506 INFO:tasks.workunit.client.1.smithi134.stdout:3/132: write dc/dd/d28/d2f/d35/d29/f2e [487155,97629] 0 2024-04-22T23:23:37.507 INFO:tasks.workunit.client.0.smithi067.stdout:9/788: chown d5/dd/d18/d3c/ddf/d101 89679 1 2024-04-22T23:23:37.508 INFO:tasks.workunit.client.0.smithi067.stdout:9/789: write d5/dd/d18/d39/f64 [2905714,25504] 0 2024-04-22T23:23:37.509 INFO:tasks.workunit.client.0.smithi067.stdout:7/758: mknod d10/c11b 0 2024-04-22T23:23:37.510 INFO:tasks.workunit.client.0.smithi067.stdout:2/796: link d1/d10/d72/d53/f58 d1/d10/d72/d68/da1/f106 0 2024-04-22T23:23:37.510 INFO:tasks.workunit.client.1.smithi134.stdout:2/132: fsync d2/f18 0 2024-04-22T23:23:37.510 INFO:tasks.workunit.client.1.smithi134.stdout:4/102: symlink d4/d17/l19 0 2024-04-22T23:23:37.511 INFO:tasks.workunit.client.0.smithi067.stdout:9/790: write d5/fa [3232527,18425] 0 2024-04-22T23:23:37.512 INFO:tasks.workunit.client.1.smithi134.stdout:3/133: creat dc/dd/d28/f39 x:0 0 0 2024-04-22T23:23:37.512 INFO:tasks.workunit.client.1.smithi134.stdout:5/175: dwrite f8 [4194304,4194304] 0 2024-04-22T23:23:37.514 INFO:tasks.workunit.client.1.smithi134.stdout:2/133: write d2/f22 [816836,108054] 0 2024-04-22T23:23:37.515 INFO:tasks.workunit.client.1.smithi134.stdout:4/103: symlink d4/d17/l1a 0 2024-04-22T23:23:37.515 INFO:tasks.workunit.client.1.smithi134.stdout:3/134: fdatasync dc/dd/d28/f39 0 2024-04-22T23:23:37.518 INFO:tasks.workunit.client.0.smithi067.stdout:2/797: dwrite d1/d10/d72/d53/fae [0,4194304] 0 2024-04-22T23:23:37.518 INFO:tasks.workunit.client.1.smithi134.stdout:2/134: symlink d2/dd/df/l32 0 2024-04-22T23:23:37.520 INFO:tasks.workunit.client.1.smithi134.stdout:4/104: write d4/d7/f10 [3978184,94278] 0 2024-04-22T23:23:37.521 INFO:tasks.workunit.client.1.smithi134.stdout:7/142: dwrite d4/d8/f22 [0,4194304] 0 2024-04-22T23:23:37.521 INFO:tasks.workunit.client.0.smithi067.stdout:2/798: creat d1/d19/d6f/f107 x:0 0 0 2024-04-22T23:23:37.522 INFO:tasks.workunit.client.0.smithi067.stdout:2/799: stat d1/f69 0 2024-04-22T23:23:37.522 INFO:tasks.workunit.client.0.smithi067.stdout:2/800: readlink d1/d10/d72/lb3 0 2024-04-22T23:23:37.525 INFO:tasks.workunit.client.0.smithi067.stdout:8/843: rename d1/dd/d26 to d1/dd/d80/db9/d110 0 2024-04-22T23:23:37.525 INFO:tasks.workunit.client.1.smithi134.stdout:4/105: dread - d4/dd/f16 zero size 2024-04-22T23:23:37.527 INFO:tasks.workunit.client.0.smithi067.stdout:2/801: mkdir d1/d42/d61/dad/d108 0 2024-04-22T23:23:37.527 INFO:tasks.workunit.client.1.smithi134.stdout:3/135: truncate dc/dd/d28/d2f/d21/f33 310634 0 2024-04-22T23:23:37.529 INFO:tasks.workunit.client.1.smithi134.stdout:5/176: dwrite d9/dd/f13 [0,4194304] 0 2024-04-22T23:23:37.531 INFO:tasks.workunit.client.1.smithi134.stdout:7/143: symlink d4/d10/l39 0 2024-04-22T23:23:37.531 INFO:tasks.workunit.client.0.smithi067.stdout:5/915: rename d5/d28/d36/d40/f105 to d5/d106/f13b 0 2024-04-22T23:23:37.533 INFO:tasks.workunit.client.0.smithi067.stdout:8/844: mkdir d1/dd/d80/db9/de3/db8/d111 0 2024-04-22T23:23:37.535 INFO:tasks.workunit.client.0.smithi067.stdout:2/802: symlink d1/d10/d33/d46/l109 0 2024-04-22T23:23:37.535 INFO:tasks.workunit.client.1.smithi134.stdout:3/136: creat dc/dd/d28/f3a x:0 0 0 2024-04-22T23:23:37.535 INFO:tasks.workunit.client.1.smithi134.stdout:5/177: mkdir d9/d11/d34/d3b 0 2024-04-22T23:23:37.537 INFO:tasks.workunit.client.1.smithi134.stdout:7/144: mknod d4/d8/d1d/d29/c3a 0 2024-04-22T23:23:37.538 INFO:tasks.workunit.client.1.smithi134.stdout:2/135: dwrite d2/d15/d24/f27 [4194304,4194304] 0 2024-04-22T23:23:37.538 INFO:tasks.workunit.client.1.smithi134.stdout:4/106: dread d4/d7/f10 [0,4194304] 0 2024-04-22T23:23:37.538 INFO:tasks.workunit.client.0.smithi067.stdout:8/845: chown d1/dd/d80/db9/lf6 94275 1 2024-04-22T23:23:37.541 INFO:tasks.workunit.client.1.smithi134.stdout:4/107: rename d4/d17 to d4/d17/d1b 22 2024-04-22T23:23:37.546 INFO:tasks.workunit.client.0.smithi067.stdout:0/963: dwrite db/de/d11/d1a/da4/dec/f122 [0,4194304] 0 2024-04-22T23:23:37.548 INFO:tasks.workunit.client.0.smithi067.stdout:5/916: dwrite d5/db/d35/f124 [0,4194304] 0 2024-04-22T23:23:37.550 INFO:tasks.workunit.client.1.smithi134.stdout:5/178: dwrite d9/f36 [0,4194304] 0 2024-04-22T23:23:37.551 INFO:tasks.workunit.client.1.smithi134.stdout:2/136: dread d2/d15/d24/d1d/f28 [0,4194304] 0 2024-04-22T23:23:37.552 INFO:tasks.workunit.client.1.smithi134.stdout:4/108: creat d4/d15/f1c x:0 0 0 2024-04-22T23:23:37.554 INFO:tasks.workunit.client.1.smithi134.stdout:1/100: dwrite db/f14 [4194304,4194304] 0 2024-04-22T23:23:37.555 INFO:tasks.workunit.client.1.smithi134.stdout:5/179: symlink d9/dd/d17/l3c 0 2024-04-22T23:23:37.556 INFO:tasks.workunit.client.1.smithi134.stdout:5/180: chown d9/c3a 34039685 1 2024-04-22T23:23:37.558 INFO:tasks.workunit.client.1.smithi134.stdout:5/181: chown d9/f15 12 1 2024-04-22T23:23:37.559 INFO:tasks.workunit.client.0.smithi067.stdout:2/803: rename d1/d10/d1c/d6d/d87/dc8 to d1/d10/d72/d53/d9a/d10a 0 2024-04-22T23:23:37.560 INFO:tasks.workunit.client.1.smithi134.stdout:1/101: creat db/d15/f29 x:0 0 0 2024-04-22T23:23:37.560 INFO:tasks.workunit.client.0.smithi067.stdout:8/846: dwrite d1/dd/d80/f8d [0,4194304] 0 2024-04-22T23:23:37.561 INFO:tasks.workunit.client.0.smithi067.stdout:9/791: dread d5/d21/d47/f72 [0,4194304] 0 2024-04-22T23:23:37.563 INFO:tasks.workunit.client.0.smithi067.stdout:1/781: truncate d8/d9/d2d/d4d/d48/d9a/fac 152834 0 2024-04-22T23:23:37.563 INFO:tasks.workunit.client.0.smithi067.stdout:3/835: sync 2024-04-22T23:23:37.564 INFO:tasks.workunit.client.0.smithi067.stdout:5/917: creat d5/d28/daf/dbf/dfc/d99/f13c x:0 0 0 2024-04-22T23:23:37.568 INFO:tasks.workunit.client.1.smithi134.stdout:1/102: dwrite fa [0,4194304] 0 2024-04-22T23:23:37.569 INFO:tasks.workunit.client.1.smithi134.stdout:3/137: truncate dc/f22 417504 0 2024-04-22T23:23:37.570 INFO:tasks.workunit.client.0.smithi067.stdout:3/836: write d3/d4/d10/d38/d53/d55/d5c/f125 [383017,81349] 0 2024-04-22T23:23:37.572 INFO:tasks.workunit.client.1.smithi134.stdout:5/182: link d9/df/l2f d9/df/d20/d29/l3d 0 2024-04-22T23:23:37.573 INFO:tasks.workunit.client.1.smithi134.stdout:3/138: truncate dc/dd/d28/d2f/d21/f33 1073939 0 2024-04-22T23:23:37.575 INFO:tasks.workunit.client.1.smithi134.stdout:4/109: getdents d4/d15 0 2024-04-22T23:23:37.578 INFO:tasks.workunit.client.0.smithi067.stdout:3/837: dread - d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d90/da2/daf/dd8/f104 zero size 2024-04-22T23:23:37.579 INFO:tasks.workunit.client.0.smithi067.stdout:3/838: chown d3/d4/d28/d5f/lff 74364 1 2024-04-22T23:23:37.582 INFO:tasks.workunit.client.0.smithi067.stdout:9/792: fdatasync d5/d21/d47/fbd 0 2024-04-22T23:23:37.582 INFO:tasks.workunit.client.0.smithi067.stdout:0/964: fdatasync db/d21/d29/d75/d9d/dde/d10f/d46/f109 0 2024-04-22T23:23:37.584 INFO:tasks.workunit.client.1.smithi134.stdout:4/110: rmdir d4/d13 39 2024-04-22T23:23:37.585 INFO:tasks.workunit.client.1.smithi134.stdout:2/137: truncate d2/d15/d24/d1d/f26 3049638 0 2024-04-22T23:23:37.586 INFO:tasks.workunit.client.1.smithi134.stdout:1/103: getdents db/d15/d17/d20 0 2024-04-22T23:23:37.586 INFO:tasks.workunit.client.0.smithi067.stdout:8/847: write d1/d3/f2a [8803589,101497] 0 2024-04-22T23:23:37.587 INFO:tasks.workunit.client.0.smithi067.stdout:1/782: write d8/d9/d2f/d33/d5c/f65 [1393538,104760] 0 2024-04-22T23:23:37.590 INFO:tasks.workunit.client.1.smithi134.stdout:1/104: mkdir db/d15/d17/d20/d2a 0 2024-04-22T23:23:37.592 INFO:tasks.workunit.client.0.smithi067.stdout:6/845: dwrite d17/d39/d3a/f5a [0,4194304] 0 2024-04-22T23:23:37.593 INFO:tasks.workunit.client.1.smithi134.stdout:5/183: link l7 d9/d1e/l3e 0 2024-04-22T23:23:37.594 INFO:tasks.workunit.client.0.smithi067.stdout:3/839: dread d3/fc1 [0,4194304] 0 2024-04-22T23:23:37.595 INFO:tasks.workunit.client.1.smithi134.stdout:2/138: dwrite d2/d15/d24/f27 [0,4194304] 0 2024-04-22T23:23:37.595 INFO:tasks.workunit.client.1.smithi134.stdout:1/105: mknod db/d15/d17/c2b 0 2024-04-22T23:23:37.596 INFO:tasks.workunit.client.0.smithi067.stdout:3/840: chown d3/fe0 1085 1 2024-04-22T23:23:37.597 INFO:tasks.workunit.client.1.smithi134.stdout:1/106: chown db/d15/f18 1 1 2024-04-22T23:23:37.597 INFO:tasks.workunit.client.1.smithi134.stdout:4/111: dwrite d4/f6 [0,4194304] 0 2024-04-22T23:23:37.598 INFO:tasks.workunit.client.1.smithi134.stdout:8/66: sync 2024-04-22T23:23:37.598 INFO:tasks.workunit.client.1.smithi134.stdout:0/142: sync 2024-04-22T23:23:37.598 INFO:tasks.workunit.client.0.smithi067.stdout:2/804: getdents d1/d19/ded 0 2024-04-22T23:23:37.598 INFO:tasks.workunit.client.0.smithi067.stdout:0/965: mkdir db/d21/d29/d75/d9d/dde/d100/d13c 0 2024-04-22T23:23:37.599 INFO:tasks.workunit.client.0.smithi067.stdout:5/918: mkdir d5/d1a/d103/d10d/d121/d13d 0 2024-04-22T23:23:37.600 INFO:tasks.workunit.client.1.smithi134.stdout:1/107: unlink l5 0 2024-04-22T23:23:37.600 INFO:tasks.workunit.client.0.smithi067.stdout:7/759: dwrite d10/d47/d49/f58 [0,4194304] 0 2024-04-22T23:23:37.602 INFO:tasks.workunit.client.0.smithi067.stdout:1/783: creat d8/d9/d2f/d33/d66/ffe x:0 0 0 2024-04-22T23:23:37.602 INFO:tasks.workunit.client.1.smithi134.stdout:8/67: read - d7/dc/f12 zero size 2024-04-22T23:23:37.603 INFO:tasks.workunit.client.0.smithi067.stdout:9/793: dwrite d5/d21/f2b [0,4194304] 0 2024-04-22T23:23:37.604 INFO:tasks.workunit.client.1.smithi134.stdout:8/68: dread - d7/f15 zero size 2024-04-22T23:23:37.610 INFO:tasks.workunit.client.1.smithi134.stdout:1/108: dwrite fa [0,4194304] 0 2024-04-22T23:23:37.614 INFO:tasks.workunit.client.0.smithi067.stdout:8/848: rename d1/d3/d7e/l87 to d1/d3/d10/d3b/l112 0 2024-04-22T23:23:37.615 INFO:tasks.workunit.client.1.smithi134.stdout:4/112: creat d4/f1d x:0 0 0 2024-04-22T23:23:37.617 INFO:tasks.workunit.client.1.smithi134.stdout:2/139: dread d2/f17 [0,4194304] 0 2024-04-22T23:23:37.618 INFO:tasks.workunit.client.0.smithi067.stdout:0/966: mkdir db/de/d11/d1a/da4/dec/d105/d13d 0 2024-04-22T23:23:37.619 INFO:tasks.workunit.client.0.smithi067.stdout:9/794: symlink d5/dd/d18/d39/l107 0 2024-04-22T23:23:37.619 INFO:tasks.workunit.client.1.smithi134.stdout:3/139: dwrite dc/dd/d28/d2f/d21/f16 [0,4194304] 0 2024-04-22T23:23:37.620 INFO:tasks.workunit.client.0.smithi067.stdout:8/849: write d1/d41/f83 [5025297,14496] 0 2024-04-22T23:23:37.621 INFO:tasks.workunit.client.0.smithi067.stdout:1/784: dwrite d8/d9/d2d/d4d/d9c/f96 [0,4194304] 0 2024-04-22T23:23:37.622 INFO:tasks.workunit.client.1.smithi134.stdout:0/143: link d1/d5/dd/d20/f1e d1/d5/dd/f2b 0 2024-04-22T23:23:37.624 INFO:tasks.workunit.client.1.smithi134.stdout:1/109: fsync db/d15/f18 0 2024-04-22T23:23:37.624 INFO:tasks.workunit.client.0.smithi067.stdout:6/846: rename d17/d39/d53/cb2 to d17/d83/d9b/ddd/c11a 0 2024-04-22T23:23:37.627 INFO:tasks.workunit.client.1.smithi134.stdout:8/69: mknod d7/db/c17 0 2024-04-22T23:23:37.628 INFO:tasks.workunit.client.1.smithi134.stdout:4/113: rmdir d4/d15 39 2024-04-22T23:23:37.629 INFO:tasks.workunit.client.0.smithi067.stdout:6/847: dread - d17/d39/d53/d90/d46/d62/d88/fdb zero size 2024-04-22T23:23:37.629 INFO:tasks.workunit.client.1.smithi134.stdout:8/70: chown d7/db 75728 1 2024-04-22T23:23:37.631 INFO:tasks.workunit.client.0.smithi067.stdout:9/795: mknod d5/d21/d47/d56/dc6/c108 0 2024-04-22T23:23:37.631 INFO:tasks.workunit.client.0.smithi067.stdout:9/796: readlink d5/dd/d18/d5c/l75 0 2024-04-22T23:23:37.634 INFO:tasks.workunit.client.1.smithi134.stdout:2/140: rmdir d2/dd 39 2024-04-22T23:23:37.635 INFO:tasks.workunit.client.1.smithi134.stdout:0/144: mkdir d1/d5/dd/d20/d2c 0 2024-04-22T23:23:37.635 INFO:tasks.workunit.client.0.smithi067.stdout:7/760: fsync d10/d14/d1d/d26/d2c/d41/f7c 0 2024-04-22T23:23:37.635 INFO:tasks.workunit.client.1.smithi134.stdout:1/110: creat db/d15/d17/f2c x:0 0 0 2024-04-22T23:23:37.636 INFO:tasks.workunit.client.1.smithi134.stdout:3/140: rename dc/dd/d28/d2f/d35 to dc/dd/d28/d3b 0 2024-04-22T23:23:37.636 INFO:tasks.workunit.client.0.smithi067.stdout:7/761: chown d10/d47/la0 371721 1 2024-04-22T23:23:37.636 INFO:tasks.workunit.client.1.smithi134.stdout:3/141: readlink lb 0 2024-04-22T23:23:37.637 INFO:tasks.workunit.client.1.smithi134.stdout:8/71: mknod d7/dc/c18 0 2024-04-22T23:23:37.638 INFO:tasks.workunit.client.1.smithi134.stdout:4/114: dread - d4/d15/f1c zero size 2024-04-22T23:23:37.639 INFO:tasks.workunit.client.0.smithi067.stdout:8/850: dread - d1/dd/d80/d6f/fdc zero size 2024-04-22T23:23:37.639 INFO:tasks.workunit.client.1.smithi134.stdout:0/145: mknod d1/d2/c2d 0 2024-04-22T23:23:37.640 INFO:tasks.workunit.client.1.smithi134.stdout:8/72: chown d7/dc/c18 32270947 1 2024-04-22T23:23:37.640 INFO:tasks.workunit.client.0.smithi067.stdout:0/967: creat db/de/d11/d1a/da4/dec/d105/d13d/f13e x:0 0 0 2024-04-22T23:23:37.640 INFO:tasks.workunit.client.1.smithi134.stdout:3/142: write dc/dd/d28/d2f/d21/f33 [399192,108292] 0 2024-04-22T23:23:37.641 INFO:tasks.workunit.client.1.smithi134.stdout:1/111: unlink f9 0 2024-04-22T23:23:37.641 INFO:tasks.workunit.client.1.smithi134.stdout:8/73: read - d7/dc/f14 zero size 2024-04-22T23:23:37.642 INFO:tasks.workunit.client.0.smithi067.stdout:3/841: write d3/d4/d10/d38/d53/d55/f66 [19896,112351] 0 2024-04-22T23:23:37.643 INFO:tasks.workunit.client.1.smithi134.stdout:1/112: stat db/l11 0 2024-04-22T23:23:37.644 INFO:tasks.workunit.client.0.smithi067.stdout:8/851: rmdir d1/dd/d80/db9/d110/d62/ddf 39 2024-04-22T23:23:37.645 INFO:tasks.workunit.client.1.smithi134.stdout:1/113: dread - db/f26 zero size 2024-04-22T23:23:37.645 INFO:tasks.workunit.client.1.smithi134.stdout:2/141: dread d2/d15/d24/f2a [0,4194304] 0 2024-04-22T23:23:37.645 INFO:tasks.workunit.client.0.smithi067.stdout:6/848: dread d17/d39/d53/d90/f8b [0,4194304] 0 2024-04-22T23:23:37.646 INFO:tasks.workunit.client.0.smithi067.stdout:7/762: unlink d10/d14/d1d/d46/d6a/dcc/d106/l119 0 2024-04-22T23:23:37.651 INFO:tasks.workunit.client.1.smithi134.stdout:0/146: dwrite d1/d5/dd/f16 [0,4194304] 0 2024-04-22T23:23:37.651 INFO:tasks.workunit.client.1.smithi134.stdout:3/143: truncate f9 4239086 0 2024-04-22T23:23:37.651 INFO:tasks.workunit.client.0.smithi067.stdout:9/797: truncate d5/d21/d47/d7f/fd3 137675 0 2024-04-22T23:23:37.651 INFO:tasks.workunit.client.0.smithi067.stdout:8/852: creat d1/d3/d27/f113 x:0 0 0 2024-04-22T23:23:37.651 INFO:tasks.workunit.client.0.smithi067.stdout:3/842: creat d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d90/da2/f12a x:0 0 0 2024-04-22T23:23:37.652 INFO:tasks.workunit.client.1.smithi134.stdout:4/115: symlink d4/d13/l1e 0 2024-04-22T23:23:37.652 INFO:tasks.workunit.client.0.smithi067.stdout:8/853: write d1/d41/f83 [5451491,98739] 0 2024-04-22T23:23:37.652 INFO:tasks.workunit.client.0.smithi067.stdout:6/849: rename d17/d39/d53/f101 to d17/d83/d9b/f11b 0 2024-04-22T23:23:37.656 INFO:tasks.workunit.client.1.smithi134.stdout:0/147: dread d1/d5/dd/d11/f18 [4194304,4194304] 0 2024-04-22T23:23:37.657 INFO:tasks.workunit.client.1.smithi134.stdout:8/74: rmdir d7/d13 0 2024-04-22T23:23:37.658 INFO:tasks.workunit.client.0.smithi067.stdout:7/763: chown d10/d14/d1d/d26/d2c/d7a/f9b 6 1 2024-04-22T23:23:37.658 INFO:tasks.workunit.client.1.smithi134.stdout:2/142: mkdir d2/dd/d2b/d33 0 2024-04-22T23:23:37.660 INFO:tasks.workunit.client.1.smithi134.stdout:2/143: chown d2/dd/d2b 13 1 2024-04-22T23:23:37.664 INFO:tasks.workunit.client.0.smithi067.stdout:7/764: symlink d10/d14/d6d/l11c 0 2024-04-22T23:23:37.664 INFO:tasks.workunit.client.0.smithi067.stdout:2/805: write d1/d10/d33/d46/f88 [1000227,14596] 0 2024-04-22T23:23:37.666 INFO:tasks.workunit.client.0.smithi067.stdout:7/765: fdatasync d10/d47/dab/fd5 0 2024-04-22T23:23:37.667 INFO:tasks.workunit.client.1.smithi134.stdout:8/75: dwrite d7/db/fe [0,4194304] 0 2024-04-22T23:23:37.667 INFO:tasks.workunit.client.0.smithi067.stdout:8/854: creat d1/dd/d80/db9/de3/db8/d111/f114 x:0 0 0 2024-04-22T23:23:37.668 INFO:tasks.workunit.client.1.smithi134.stdout:2/144: write d2/d15/d24/f1e [1705770,29534] 0 2024-04-22T23:23:37.669 INFO:tasks.workunit.client.1.smithi134.stdout:0/148: dread d1/d5/dd/d11/f18 [0,4194304] 0 2024-04-22T23:23:37.669 INFO:tasks.workunit.client.0.smithi067.stdout:1/785: dread d8/d12/f17 [0,4194304] 0 2024-04-22T23:23:37.670 INFO:tasks.workunit.client.1.smithi134.stdout:0/149: readlink d1/d12/d22/l23 0 2024-04-22T23:23:37.670 INFO:tasks.workunit.client.1.smithi134.stdout:0/150: read d1/f10 [1663239,88207] 0 2024-04-22T23:23:37.672 INFO:tasks.workunit.client.1.smithi134.stdout:8/76: mkdir d7/dc/d19 0 2024-04-22T23:23:37.673 INFO:tasks.workunit.client.0.smithi067.stdout:9/798: creat d5/dd/d18/d39/d3e/f109 x:0 0 0 2024-04-22T23:23:37.676 INFO:tasks.workunit.client.1.smithi134.stdout:0/151: dread d1/d5/f7 [0,4194304] 0 2024-04-22T23:23:37.678 INFO:tasks.workunit.client.0.smithi067.stdout:5/919: dread d5/db/f6a [0,4194304] 0 2024-04-22T23:23:37.683 INFO:tasks.workunit.client.0.smithi067.stdout:8/855: dread d1/d3/d10/fba [0,4194304] 0 2024-04-22T23:23:37.685 INFO:tasks.workunit.client.1.smithi134.stdout:0/152: fdatasync d1/d2/f4 0 2024-04-22T23:23:37.686 INFO:tasks.workunit.client.0.smithi067.stdout:9/799: dwrite d5/dd/d18/d39/d49/d54/d99/f102 [0,4194304] 0 2024-04-22T23:23:37.687 INFO:tasks.workunit.client.1.smithi134.stdout:0/153: dread - d1/d5/d15/f27 zero size 2024-04-22T23:23:37.689 INFO:tasks.workunit.client.1.smithi134.stdout:0/154: write d1/d5/d15/f27 [702654,38835] 0 2024-04-22T23:23:37.694 INFO:tasks.workunit.client.0.smithi067.stdout:2/806: dread d1/d10/d1c/d6d/d87/fd8 [0,4194304] 0 2024-04-22T23:23:37.695 INFO:tasks.workunit.client.1.smithi134.stdout:0/155: rmdir d1/d5/dd/d20 39 2024-04-22T23:23:37.695 INFO:tasks.workunit.client.0.smithi067.stdout:2/807: fdatasync d1/d42/d61/fb8 0 2024-04-22T23:23:37.697 INFO:tasks.workunit.client.1.smithi134.stdout:4/116: truncate d4/fe 4113050 0 2024-04-22T23:23:37.698 INFO:tasks.workunit.client.1.smithi134.stdout:4/117: write d4/f1d [926741,104891] 0 2024-04-22T23:23:37.699 INFO:tasks.workunit.client.1.smithi134.stdout:0/156: creat d1/d12/f2e x:0 0 0 2024-04-22T23:23:37.701 INFO:tasks.workunit.client.1.smithi134.stdout:3/144: dread dc/f22 [0,4194304] 0 2024-04-22T23:23:37.702 INFO:tasks.workunit.client.1.smithi134.stdout:0/157: fdatasync d1/f10 0 2024-04-22T23:23:37.706 INFO:tasks.workunit.client.1.smithi134.stdout:3/145: chown dc/dd/d28/d2f 4493 1 2024-04-22T23:23:37.708 INFO:tasks.workunit.client.1.smithi134.stdout:3/146: read - dc/dd/d28/f3a zero size 2024-04-22T23:23:37.709 INFO:tasks.workunit.client.1.smithi134.stdout:8/77: dwrite f6 [0,4194304] 0 2024-04-22T23:23:37.709 INFO:tasks.workunit.client.1.smithi134.stdout:3/147: fdatasync dc/dd/d28/f3a 0 2024-04-22T23:23:37.714 INFO:tasks.workunit.client.1.smithi134.stdout:4/118: rename d4/fe to d4/d17/f1f 0 2024-04-22T23:23:37.717 INFO:tasks.workunit.client.1.smithi134.stdout:8/78: mknod d7/c1a 0 2024-04-22T23:23:37.722 INFO:tasks.workunit.client.0.smithi067.stdout:8/856: dread d1/d3/d10/d3b/f5e [0,4194304] 0 2024-04-22T23:23:37.722 INFO:tasks.workunit.client.1.smithi134.stdout:3/148: dwrite dc/dd/d28/d3b/d29/f2e [0,4194304] 0 2024-04-22T23:23:37.724 INFO:tasks.workunit.client.0.smithi067.stdout:8/857: write d1/d41/d46/f66 [4816511,106325] 0 2024-04-22T23:23:37.725 INFO:tasks.workunit.client.1.smithi134.stdout:3/149: mknod dc/dd/d28/d2f/d21/c3c 0 2024-04-22T23:23:37.726 INFO:tasks.workunit.client.1.smithi134.stdout:4/119: dwrite d4/d15/f1c [0,4194304] 0 2024-04-22T23:23:37.726 INFO:tasks.workunit.client.1.smithi134.stdout:3/150: read - dc/dd/d28/f39 zero size 2024-04-22T23:23:37.753 INFO:tasks.workunit.client.0.smithi067.stdout:5/920: write d5/db/d11f/f12c [3407303,78166] 0 2024-04-22T23:23:37.754 INFO:tasks.workunit.client.0.smithi067.stdout:0/968: truncate db/d9a/d56/d6b/db9/f104 435911 0 2024-04-22T23:23:37.760 INFO:tasks.workunit.client.1.smithi134.stdout:9/64: sync 2024-04-22T23:23:37.760 INFO:tasks.workunit.client.0.smithi067.stdout:6/850: truncate d17/d39/d53/d90/d46/f10a 3494280 0 2024-04-22T23:23:37.762 INFO:tasks.workunit.client.0.smithi067.stdout:9/800: unlink d5/dd/d18/d39/d49/f55 0 2024-04-22T23:23:37.768 INFO:tasks.workunit.client.0.smithi067.stdout:3/843: getdents d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d90/da2/daf 0 2024-04-22T23:23:37.769 INFO:tasks.workunit.client.1.smithi134.stdout:4/120: write d4/f1d [559602,127756] 0 2024-04-22T23:23:37.770 INFO:tasks.workunit.client.1.smithi134.stdout:9/65: fdatasync d0/d7/fd 0 2024-04-22T23:23:37.775 INFO:tasks.workunit.client.1.smithi134.stdout:9/66: unlink d0/d7/fe 0 2024-04-22T23:23:37.776 INFO:tasks.workunit.client.1.smithi134.stdout:4/121: creat d4/f20 x:0 0 0 2024-04-22T23:23:37.786 INFO:tasks.workunit.client.1.smithi134.stdout:8/79: write d7/db/fe [4595012,122683] 0 2024-04-22T23:23:37.786 INFO:tasks.workunit.client.1.smithi134.stdout:0/158: dwrite d1/d5/dd/d20/f1e [0,4194304] 0 2024-04-22T23:23:37.786 INFO:tasks.workunit.client.1.smithi134.stdout:3/151: rename dc/dd/d28/d2f/d21/l2c to dc/l3d 0 2024-04-22T23:23:37.788 INFO:tasks.workunit.client.1.smithi134.stdout:4/122: dread d4/d7/f10 [0,4194304] 0 2024-04-22T23:23:37.789 INFO:tasks.workunit.client.1.smithi134.stdout:4/123: chown d4/d17/l19 855987489 1 2024-04-22T23:23:37.789 INFO:tasks.workunit.client.1.smithi134.stdout:9/67: mknod d0/d10/d11/c15 0 2024-04-22T23:23:37.789 INFO:tasks.workunit.client.1.smithi134.stdout:4/124: stat d4/dd/c11 0 2024-04-22T23:23:37.791 INFO:tasks.workunit.client.0.smithi067.stdout:8/858: mkdir d1/d3/d27/d107/d115 0 2024-04-22T23:23:37.792 INFO:tasks.workunit.client.0.smithi067.stdout:8/859: write d1/dd/d80/f86 [1263795,3497] 0 2024-04-22T23:23:37.793 INFO:tasks.workunit.client.1.smithi134.stdout:8/80: rename d7/f15 to d7/db/f1b 0 2024-04-22T23:23:37.793 INFO:tasks.workunit.client.0.smithi067.stdout:8/860: fsync d1/d41/d46/fe6 0 2024-04-22T23:23:37.793 INFO:tasks.workunit.client.1.smithi134.stdout:3/152: fdatasync dc/dd/d28/d3b/f31 0 2024-04-22T23:23:37.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:37 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:23:37.796 INFO:tasks.workunit.client.0.smithi067.stdout:3/844: symlink d3/d4/d10/d8a/d11a/l12b 0 2024-04-22T23:23:37.798 INFO:tasks.workunit.client.0.smithi067.stdout:3/845: fdatasync d3/d4/d10/d38/d53/d55/d74/d7f/ff2 0 2024-04-22T23:23:37.799 INFO:tasks.workunit.client.0.smithi067.stdout:8/861: symlink d1/d3/d27/d107/d115/l116 0 2024-04-22T23:23:37.799 INFO:tasks.workunit.client.0.smithi067.stdout:3/846: dread - d3/d4/d10/d38/d53/d55/d74/ff6 zero size 2024-04-22T23:23:37.802 INFO:tasks.workunit.client.1.smithi134.stdout:9/68: mkdir d0/d7/dc/d16 0 2024-04-22T23:23:37.806 INFO:tasks.workunit.client.0.smithi067.stdout:8/862: link d1/dd/d80/db9/d110/cb3 d1/d3/d27/d107/c117 0 2024-04-22T23:23:37.809 INFO:tasks.workunit.client.1.smithi134.stdout:4/125: creat d4/f21 x:0 0 0 2024-04-22T23:23:37.809 INFO:tasks.workunit.client.0.smithi067.stdout:3/847: link d3/d4/d10/f39 d3/d4/d10/d38/d53/d55/d74/dd5/ddb/f12c 0 2024-04-22T23:23:37.809 INFO:tasks.workunit.client.1.smithi134.stdout:0/159: getdents d1/d5/dd/d20 0 2024-04-22T23:23:37.811 INFO:tasks.workunit.client.1.smithi134.stdout:8/81: creat d7/f1c x:0 0 0 2024-04-22T23:23:37.812 INFO:tasks.workunit.client.0.smithi067.stdout:3/848: creat d3/d4/d10/d8a/f12d x:0 0 0 2024-04-22T23:23:37.813 INFO:tasks.workunit.client.0.smithi067.stdout:1/786: write d8/d9/d2f/d33/fb6 [6224499,102106] 0 2024-04-22T23:23:37.814 INFO:tasks.workunit.client.1.smithi134.stdout:3/153: getdents dc/d2a/d30 0 2024-04-22T23:23:37.816 INFO:tasks.workunit.client.0.smithi067.stdout:3/849: symlink d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d8c/l12e 0 2024-04-22T23:23:37.817 INFO:tasks.workunit.client.1.smithi134.stdout:4/126: creat d4/d17/f22 x:0 0 0 2024-04-22T23:23:37.818 INFO:tasks.workunit.client.1.smithi134.stdout:0/160: dwrite d1/f9 [4194304,4194304] 0 2024-04-22T23:23:37.819 INFO:tasks.workunit.client.0.smithi067.stdout:9/801: dread d5/f9 [0,4194304] 0 2024-04-22T23:23:37.820 INFO:tasks.workunit.client.0.smithi067.stdout:9/802: readlink d5/dd/d18/d39/l107 0 2024-04-22T23:23:37.820 INFO:tasks.workunit.client.0.smithi067.stdout:3/850: mknod d3/dcb/c12f 0 2024-04-22T23:23:37.822 INFO:tasks.workunit.client.0.smithi067.stdout:3/851: truncate d3/d4/dec/f124 54943 0 2024-04-22T23:23:37.823 INFO:tasks.workunit.client.0.smithi067.stdout:3/852: readlink d3/d4/d28/ld9 0 2024-04-22T23:23:37.824 INFO:tasks.workunit.client.1.smithi134.stdout:4/127: mknod d4/dd/c23 0 2024-04-22T23:23:37.828 INFO:tasks.workunit.client.1.smithi134.stdout:9/69: read - d0/d7/fb zero size 2024-04-22T23:23:37.831 INFO:tasks.workunit.client.1.smithi134.stdout:4/128: chown d4/l5 364067 1 2024-04-22T23:23:37.832 INFO:tasks.workunit.client.1.smithi134.stdout:0/161: mknod d1/d2/d1d/c2f 0 2024-04-22T23:23:37.833 INFO:tasks.workunit.client.1.smithi134.stdout:8/82: dwrite d7/db/f1b [0,4194304] 0 2024-04-22T23:23:37.835 INFO:tasks.workunit.client.1.smithi134.stdout:0/162: mknod d1/db/c30 0 2024-04-22T23:23:37.836 INFO:tasks.workunit.client.1.smithi134.stdout:0/163: chown d1/d12 240 1 2024-04-22T23:23:37.836 INFO:tasks.workunit.client.1.smithi134.stdout:9/70: readlink d0/d7/d12/l14 0 2024-04-22T23:23:37.837 INFO:tasks.workunit.client.1.smithi134.stdout:4/129: mknod d4/c24 0 2024-04-22T23:23:37.841 INFO:tasks.workunit.client.1.smithi134.stdout:0/164: unlink d1/d5/dd/c28 0 2024-04-22T23:23:37.844 INFO:tasks.workunit.client.1.smithi134.stdout:3/154: truncate dc/dd/d28/d2f/d21/f16 1301361 0 2024-04-22T23:23:37.844 INFO:tasks.workunit.client.1.smithi134.stdout:8/83: dwrite d7/f1c [0,4194304] 0 2024-04-22T23:23:37.844 INFO:tasks.workunit.client.1.smithi134.stdout:9/71: write d0/d7/f9 [1331597,129451] 0 2024-04-22T23:23:37.849 INFO:tasks.workunit.client.1.smithi134.stdout:0/165: write d1/d5/dd/f16 [1823273,117786] 0 2024-04-22T23:23:37.849 INFO:tasks.workunit.client.1.smithi134.stdout:4/130: fsync d4/d7/f10 0 2024-04-22T23:23:37.852 INFO:tasks.workunit.client.0.smithi067.stdout:6/851: rmdir d17/d39/d53/d90/d46 39 2024-04-22T23:23:37.853 INFO:tasks.workunit.client.0.smithi067.stdout:6/852: chown d17/d39/d53/d90/l8f 258876633 1 2024-04-22T23:23:37.858 INFO:tasks.workunit.client.1.smithi134.stdout:0/166: write d1/f10 [3696273,12305] 0 2024-04-22T23:23:37.861 INFO:tasks.workunit.client.0.smithi067.stdout:2/808: dwrite d1/d10/d33/d46/f7d [0,4194304] 0 2024-04-22T23:23:37.863 INFO:tasks.workunit.client.1.smithi134.stdout:0/167: write d1/d5/dd/f16 [3958252,33914] 0 2024-04-22T23:23:37.864 INFO:tasks.workunit.client.1.smithi134.stdout:4/131: dwrite d4/d15/f1c [4194304,4194304] 0 2024-04-22T23:23:37.866 INFO:tasks.workunit.client.1.smithi134.stdout:9/72: link d0/fa d0/d7/f17 0 2024-04-22T23:23:37.866 INFO:tasks.workunit.client.0.smithi067.stdout:2/809: fdatasync d1/d10/d1c/d4d/fa9 0 2024-04-22T23:23:37.867 INFO:tasks.workunit.client.1.smithi134.stdout:8/84: dwrite f4 [4194304,4194304] 0 2024-04-22T23:23:37.868 INFO:tasks.workunit.client.1.smithi134.stdout:0/168: read d1/d2/f4 [2109806,96483] 0 2024-04-22T23:23:37.868 INFO:tasks.workunit.client.1.smithi134.stdout:3/155: rmdir dc/dd 39 2024-04-22T23:23:37.868 INFO:tasks.workunit.client.0.smithi067.stdout:5/921: dwrite d5/db/d12/d44/f13a [4194304,4194304] 0 2024-04-22T23:23:37.868 INFO:tasks.workunit.client.0.smithi067.stdout:2/810: chown d1/d42/d61/daa/fca 751590733 1 2024-04-22T23:23:37.872 INFO:tasks.workunit.client.1.smithi134.stdout:4/132: write d4/f20 [93643,104650] 0 2024-04-22T23:23:37.872 INFO:tasks.workunit.client.1.smithi134.stdout:8/85: write d7/db/fe [267709,67970] 0 2024-04-22T23:23:37.874 INFO:tasks.workunit.client.0.smithi067.stdout:5/922: creat d5/db/d12/d8a/f13e x:0 0 0 2024-04-22T23:23:37.877 INFO:tasks.workunit.client.0.smithi067.stdout:2/811: rename d1/d10/d41/f93 to d1/d42/d5f/d9b/dc0/f10b 0 2024-04-22T23:23:37.879 INFO:tasks.workunit.client.1.smithi134.stdout:9/73: creat d0/d7/d12/f18 x:0 0 0 2024-04-22T23:23:37.882 INFO:tasks.workunit.client.1.smithi134.stdout:0/169: unlink d1/db/cf 0 2024-04-22T23:23:37.883 INFO:tasks.workunit.client.1.smithi134.stdout:0/170: write d1/d5/d15/f27 [1008858,83003] 0 2024-04-22T23:23:37.884 INFO:tasks.workunit.client.0.smithi067.stdout:6/853: mkdir d17/d39/d3a/d11c 0 2024-04-22T23:23:37.885 INFO:tasks.workunit.client.1.smithi134.stdout:4/133: rmdir d4/d13 39 2024-04-22T23:23:37.887 INFO:tasks.workunit.client.0.smithi067.stdout:2/812: dwrite d1/d10/d33/d46/f88 [0,4194304] 0 2024-04-22T23:23:37.887 INFO:tasks.workunit.client.1.smithi134.stdout:3/156: mknod dc/c3e 0 2024-04-22T23:23:37.890 INFO:tasks.workunit.client.1.smithi134.stdout:8/86: symlink d7/dc/d19/l1d 0 2024-04-22T23:23:37.892 INFO:tasks.workunit.client.0.smithi067.stdout:6/854: mkdir d17/d83/db5/d11d 0 2024-04-22T23:23:37.893 INFO:tasks.workunit.client.0.smithi067.stdout:5/923: creat d5/db/d12/dbb/f13f x:0 0 0 2024-04-22T23:23:37.894 INFO:tasks.workunit.client.1.smithi134.stdout:4/134: dwrite d4/f1d [0,4194304] 0 2024-04-22T23:23:37.895 INFO:tasks.workunit.client.0.smithi067.stdout:2/813: truncate d1/d10/d33/d46/da4/fc4 145112 0 2024-04-22T23:23:37.896 INFO:tasks.workunit.client.1.smithi134.stdout:9/74: creat d0/d7/f19 x:0 0 0 2024-04-22T23:23:37.898 INFO:tasks.workunit.client.0.smithi067.stdout:6/855: rename d17/d83/d9b/f11b to d17/d39/d53/f11e 0 2024-04-22T23:23:37.900 INFO:tasks.workunit.client.0.smithi067.stdout:5/924: rmdir d5/d1a/d103/d10d 39 2024-04-22T23:23:37.901 INFO:tasks.workunit.client.1.smithi134.stdout:0/171: rename d1/d5/d15/f27 to d1/d5/dd/f31 0 2024-04-22T23:23:37.904 INFO:tasks.workunit.client.0.smithi067.stdout:6/856: truncate d17/d39/d3a/f56 886421 0 2024-04-22T23:23:37.904 INFO:tasks.workunit.client.1.smithi134.stdout:3/157: dwrite dc/dd/d28/d3b/f31 [0,4194304] 0 2024-04-22T23:23:37.904 INFO:tasks.workunit.client.0.smithi067.stdout:6/857: stat d17/d83/d9b/ddd/fe6 0 2024-04-22T23:23:37.906 INFO:tasks.workunit.client.1.smithi134.stdout:8/87: symlink d7/l1e 0 2024-04-22T23:23:37.906 INFO:tasks.workunit.client.0.smithi067.stdout:2/814: rename d1/d10/d41/da2 to d1/d10/d33/d10c 0 2024-04-22T23:23:37.908 INFO:tasks.workunit.client.0.smithi067.stdout:2/815: fsync d1/d10/d72/d53/d9a/fb1 0 2024-04-22T23:23:37.909 INFO:tasks.workunit.client.1.smithi134.stdout:4/135: mkdir d4/d17/d25 0 2024-04-22T23:23:37.909 INFO:tasks.workunit.client.0.smithi067.stdout:6/858: readlink d17/d39/d53/d90/d46/d62/lac 0 2024-04-22T23:23:37.910 INFO:tasks.workunit.client.1.smithi134.stdout:4/136: fdatasync d4/f20 0 2024-04-22T23:23:37.911 INFO:tasks.workunit.client.0.smithi067.stdout:6/859: dread - d17/d39/d53/d90/d46/db0/db9/de7/f10f zero size 2024-04-22T23:23:37.912 INFO:tasks.workunit.client.1.smithi134.stdout:4/137: write d4/f20 [735638,83181] 0 2024-04-22T23:23:37.912 INFO:tasks.workunit.client.1.smithi134.stdout:4/138: fsync d4/d7/f10 0 2024-04-22T23:23:37.915 INFO:tasks.workunit.client.1.smithi134.stdout:0/172: mknod d1/d2/c32 0 2024-04-22T23:23:37.917 INFO:tasks.workunit.client.1.smithi134.stdout:8/88: creat d7/db/f1f x:0 0 0 2024-04-22T23:23:37.920 INFO:tasks.workunit.client.0.smithi067.stdout:2/816: symlink d1/d10/l10d 0 2024-04-22T23:23:37.921 INFO:tasks.workunit.client.1.smithi134.stdout:9/75: truncate d0/f3 588104 0 2024-04-22T23:23:37.922 INFO:tasks.workunit.client.1.smithi134.stdout:9/76: write d0/d7/d12/f18 [273841,26419] 0 2024-04-22T23:23:37.923 INFO:tasks.workunit.client.1.smithi134.stdout:4/139: creat d4/d15/f26 x:0 0 0 2024-04-22T23:23:37.923 INFO:tasks.workunit.client.1.smithi134.stdout:9/77: chown d0/d10/d11/c15 773812313 1 2024-04-22T23:23:37.926 INFO:tasks.workunit.client.0.smithi067.stdout:1/787: dwrite d8/d9/d2d/d4d/d48/fd0 [0,4194304] 0 2024-04-22T23:23:37.926 INFO:tasks.workunit.client.1.smithi134.stdout:3/158: link dc/dd/d28/d3b/f31 dc/f3f 0 2024-04-22T23:23:37.927 INFO:tasks.workunit.client.1.smithi134.stdout:6/141: sync 2024-04-22T23:23:37.928 INFO:tasks.workunit.client.0.smithi067.stdout:2/817: symlink d1/d10/d1c/d6d/l10e 0 2024-04-22T23:23:37.928 INFO:tasks.workunit.client.0.smithi067.stdout:9/803: dwrite d5/d21/d47/d57/f60 [0,4194304] 0 2024-04-22T23:23:37.931 INFO:tasks.workunit.client.0.smithi067.stdout:3/853: dwrite d3/d4/d1b/d93/f10c [0,4194304] 0 2024-04-22T23:23:37.932 INFO:tasks.workunit.client.0.smithi067.stdout:7/766: mknod d10/c11d 0 2024-04-22T23:23:37.937 INFO:tasks.workunit.client.1.smithi134.stdout:6/142: mkdir d0/d19/db/d33 0 2024-04-22T23:23:37.941 INFO:tasks.workunit.client.1.smithi134.stdout:9/78: truncate d0/f6 2734517 0 2024-04-22T23:23:37.942 INFO:tasks.workunit.client.1.smithi134.stdout:4/140: symlink d4/l27 0 2024-04-22T23:23:37.942 INFO:tasks.workunit.client.1.smithi134.stdout:8/89: creat d7/f20 x:0 0 0 2024-04-22T23:23:37.944 INFO:tasks.workunit.client.1.smithi134.stdout:6/143: readlink d0/d19/de/d1f/l28 0 2024-04-22T23:23:37.945 INFO:tasks.workunit.client.0.smithi067.stdout:2/818: truncate d1/d10/d1c/d6d/d87/fbb 762729 0 2024-04-22T23:23:37.945 INFO:tasks.workunit.client.1.smithi134.stdout:9/79: rmdir d0/d7 39 2024-04-22T23:23:37.946 INFO:tasks.workunit.client.0.smithi067.stdout:5/925: creat d5/f140 x:0 0 0 2024-04-22T23:23:37.948 INFO:tasks.workunit.client.0.smithi067.stdout:1/788: dwrite d8/d9/d2f/d33/d66/f9b [0,4194304] 0 2024-04-22T23:23:37.950 INFO:tasks.workunit.client.1.smithi134.stdout:4/141: creat d4/d7/f28 x:0 0 0 2024-04-22T23:23:37.950 INFO:tasks.workunit.client.0.smithi067.stdout:9/804: truncate d5/dd/d18/d39/d4d/fd0 692863 0 2024-04-22T23:23:37.951 INFO:tasks.workunit.client.1.smithi134.stdout:8/90: dwrite d7/f1c [0,4194304] 0 2024-04-22T23:23:37.951 INFO:tasks.workunit.client.0.smithi067.stdout:3/854: mknod d3/d4/d10/d38/d53/c130 0 2024-04-22T23:23:37.953 INFO:tasks.workunit.client.0.smithi067.stdout:6/860: dread d17/f35 [0,4194304] 0 2024-04-22T23:23:37.954 INFO:tasks.workunit.client.0.smithi067.stdout:5/926: mkdir d5/db/d12/d8a/d141 0 2024-04-22T23:23:37.961 INFO:tasks.workunit.client.1.smithi134.stdout:6/144: mknod d0/d19/de/d12/d29/c34 0 2024-04-22T23:23:37.961 INFO:tasks.workunit.client.0.smithi067.stdout:7/767: rename d10/d14/l32 to d10/d14/d1d/d26/d2c/d7a/dc2/dd0/l11e 0 2024-04-22T23:23:37.962 INFO:tasks.workunit.client.1.smithi134.stdout:9/80: creat d0/d7/d12/f1a x:0 0 0 2024-04-22T23:23:37.963 INFO:tasks.workunit.client.0.smithi067.stdout:3/855: mknod d3/d4/d28/d4a/c131 0 2024-04-22T23:23:37.965 INFO:tasks.workunit.client.0.smithi067.stdout:9/805: symlink d5/dd/d18/d39/d3e/d3f/d63/l10a 0 2024-04-22T23:23:37.965 INFO:tasks.workunit.client.0.smithi067.stdout:5/927: dwrite d5/d1a/d103/d4d/ffd [0,4194304] 0 2024-04-22T23:23:37.966 INFO:tasks.workunit.client.1.smithi134.stdout:4/142: mkdir d4/d29 0 2024-04-22T23:23:37.967 INFO:tasks.workunit.client.1.smithi134.stdout:9/81: dread d0/fa [0,4194304] 0 2024-04-22T23:23:37.985 INFO:tasks.workunit.client.0.smithi067.stdout:1/789: mknod d8/d9/cff 0 2024-04-22T23:23:37.985 INFO:tasks.workunit.client.0.smithi067.stdout:1/790: stat d8/d9/d2f/d31/f8e 0 2024-04-22T23:23:37.986 INFO:tasks.workunit.client.0.smithi067.stdout:2/819: dread d1/d12/f4e [0,4194304] 0 2024-04-22T23:23:37.990 INFO:tasks.workunit.client.1.smithi134.stdout:4/143: mkdir d4/d15/d2a 0 2024-04-22T23:23:37.993 INFO:tasks.workunit.client.1.smithi134.stdout:6/145: write d0/f3 [876991,121444] 0 2024-04-22T23:23:37.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:37 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:23:37.995 INFO:tasks.workunit.client.0.smithi067.stdout:9/806: creat d5/d21/d47/d56/d8c/dd5/f10b x:0 0 0 2024-04-22T23:23:37.998 INFO:tasks.workunit.client.0.smithi067.stdout:5/928: creat d5/d28/daf/dbf/dcf/db2/f142 x:0 0 0 2024-04-22T23:23:37.999 INFO:tasks.workunit.client.0.smithi067.stdout:5/929: fdatasync d5/d1a/d103/d4d/ffd 0 2024-04-22T23:23:38.005 INFO:tasks.workunit.client.1.smithi134.stdout:8/91: getdents d7/dc 0 2024-04-22T23:23:38.007 INFO:tasks.workunit.client.1.smithi134.stdout:8/92: fsync d7/db/f1f 0 2024-04-22T23:23:38.011 INFO:tasks.workunit.client.1.smithi134.stdout:8/93: read d7/dc/ff [597151,70614] 0 2024-04-22T23:23:38.012 INFO:tasks.workunit.client.1.smithi134.stdout:4/144: creat d4/f2b x:0 0 0 2024-04-22T23:23:38.014 INFO:tasks.workunit.client.0.smithi067.stdout:5/930: creat d5/db/d35/d118/f143 x:0 0 0 2024-04-22T23:23:38.015 INFO:tasks.workunit.client.1.smithi134.stdout:6/146: dwrite d0/d19/de/f31 [0,4194304] 0 2024-04-22T23:23:38.016 INFO:tasks.workunit.client.0.smithi067.stdout:2/820: dwrite d1/d10/d72/d53/d9a/d10a/dc5/fdd [0,4194304] 0 2024-04-22T23:23:38.017 INFO:tasks.workunit.client.1.smithi134.stdout:6/147: stat d0/d19/de/f31 0 2024-04-22T23:23:38.017 INFO:tasks.workunit.client.0.smithi067.stdout:7/768: creat d10/d14/d1d/d46/d50/f11f x:0 0 0 2024-04-22T23:23:38.018 INFO:tasks.workunit.client.1.smithi134.stdout:8/94: chown c2 4 1 2024-04-22T23:23:38.020 INFO:tasks.workunit.client.0.smithi067.stdout:6/861: link d17/d39/d53/d90/lea d17/d39/d104/l11f 0 2024-04-22T23:23:38.021 INFO:tasks.workunit.client.1.smithi134.stdout:9/82: dwrite d0/d7/dc/ff [0,4194304] 0 2024-04-22T23:23:38.022 INFO:tasks.workunit.client.0.smithi067.stdout:5/931: fsync d5/d28/daf/dbf/dcf/db2/fe6 0 2024-04-22T23:23:38.025 INFO:tasks.workunit.client.0.smithi067.stdout:3/856: getdents d3/d4/d10/d38/d53/d55/d74 0 2024-04-22T23:23:38.026 INFO:tasks.workunit.client.1.smithi134.stdout:8/95: rmdir d7/dc/d19 39 2024-04-22T23:23:38.027 INFO:tasks.workunit.client.1.smithi134.stdout:8/96: chown f4 15 1 2024-04-22T23:23:38.027 INFO:tasks.workunit.client.0.smithi067.stdout:7/769: mknod d10/d14/d1d/d26/d7e/c120 0 2024-04-22T23:23:38.029 INFO:tasks.workunit.client.1.smithi134.stdout:6/148: mkdir d0/d19/db/d35 0 2024-04-22T23:23:38.029 INFO:tasks.workunit.client.0.smithi067.stdout:2/821: truncate d1/d10/d72/d68/d7b/f8b 993555 0 2024-04-22T23:23:38.031 INFO:tasks.workunit.client.0.smithi067.stdout:6/862: creat d17/d83/db5/dbc/dd8/f120 x:0 0 0 2024-04-22T23:23:38.032 INFO:tasks.workunit.client.1.smithi134.stdout:8/97: write d7/dc/f14 [18769,32829] 0 2024-04-22T23:23:38.033 INFO:tasks.workunit.client.1.smithi134.stdout:6/149: mknod d0/d19/de/d1f/c36 0 2024-04-22T23:23:38.033 INFO:tasks.workunit.client.1.smithi134.stdout:8/98: dread - d7/f20 zero size 2024-04-22T23:23:38.034 INFO:tasks.workunit.client.0.smithi067.stdout:1/791: write d8/d12/d36/f72 [1272901,111432] 0 2024-04-22T23:23:38.034 INFO:tasks.workunit.client.1.smithi134.stdout:4/145: dwrite d4/d17/f22 [0,4194304] 0 2024-04-22T23:23:38.035 INFO:tasks.workunit.client.1.smithi134.stdout:6/150: chown d0/d19/de/d12/d2a 47608562 1 2024-04-22T23:23:38.037 INFO:tasks.workunit.client.0.smithi067.stdout:7/770: mkdir d10/d14/d1d/d26/d2c/d7a/dc2/d121 0 2024-04-22T23:23:38.039 INFO:tasks.workunit.client.1.smithi134.stdout:4/146: unlink d4/dd/c11 0 2024-04-22T23:23:38.039 INFO:tasks.workunit.client.0.smithi067.stdout:9/807: truncate d5/f36 1156537 0 2024-04-22T23:23:38.039 INFO:tasks.workunit.client.0.smithi067.stdout:9/808: readlink d5/dd/d18/d39/d3e/le0 0 2024-04-22T23:23:38.041 INFO:tasks.workunit.client.1.smithi134.stdout:8/99: creat d7/dc/d19/f21 x:0 0 0 2024-04-22T23:23:38.043 INFO:tasks.workunit.client.0.smithi067.stdout:7/771: rmdir d10/d18/d88/daf/d100/d62/de7 39 2024-04-22T23:23:38.043 INFO:tasks.workunit.client.1.smithi134.stdout:4/147: creat d4/d29/f2c x:0 0 0 2024-04-22T23:23:38.045 INFO:tasks.workunit.client.1.smithi134.stdout:6/151: rename d0/d19/de/f18 to d0/d19/db/f37 0 2024-04-22T23:23:38.045 INFO:tasks.workunit.client.0.smithi067.stdout:6/863: truncate d17/f29 1457478 0 2024-04-22T23:23:38.045 INFO:tasks.workunit.client.1.smithi134.stdout:6/152: dread - d0/f27 zero size 2024-04-22T23:23:38.046 INFO:tasks.workunit.client.0.smithi067.stdout:9/809: rename d5/dd/d18/d5c/l75 to d5/d21/d47/d57/dc0/d105/l10c 0 2024-04-22T23:23:38.047 INFO:tasks.workunit.client.0.smithi067.stdout:3/857: getdents d3/d4/d10/de3 0 2024-04-22T23:23:38.048 INFO:tasks.workunit.client.1.smithi134.stdout:4/148: dread d4/d15/f1c [0,4194304] 0 2024-04-22T23:23:38.048 INFO:tasks.workunit.client.1.smithi134.stdout:6/153: creat d0/d19/f38 x:0 0 0 2024-04-22T23:23:38.049 INFO:tasks.workunit.client.0.smithi067.stdout:6/864: read d17/d39/d53/d90/d46/d62/fd0 [3908454,54307] 0 2024-04-22T23:23:38.049 INFO:tasks.workunit.client.0.smithi067.stdout:7/772: mkdir d10/d14/d1d/d26/d2c/d41/db0/ddd/d122 0 2024-04-22T23:23:38.049 INFO:tasks.workunit.client.1.smithi134.stdout:8/100: creat d7/dc/f22 x:0 0 0 2024-04-22T23:23:38.050 INFO:tasks.workunit.client.1.smithi134.stdout:8/101: fsync d7/f1c 0 2024-04-22T23:23:38.050 INFO:tasks.workunit.client.0.smithi067.stdout:1/792: dwrite d8/d9/d2f/f7d [0,4194304] 0 2024-04-22T23:23:38.050 INFO:tasks.workunit.client.1.smithi134.stdout:6/154: creat d0/d19/f39 x:0 0 0 2024-04-22T23:23:38.051 INFO:tasks.workunit.client.1.smithi134.stdout:6/155: readlink d0/d19/db/lc 0 2024-04-22T23:23:38.054 INFO:tasks.workunit.client.1.smithi134.stdout:4/149: creat d4/d15/d2a/f2d x:0 0 0 2024-04-22T23:23:38.057 INFO:tasks.workunit.client.1.smithi134.stdout:8/102: dread d7/db/f1b [0,4194304] 0 2024-04-22T23:23:38.058 INFO:tasks.workunit.client.0.smithi067.stdout:1/793: link d8/d12/c3c d8/d9/d2d/d4d/d83/c100 0 2024-04-22T23:23:38.059 INFO:tasks.workunit.client.1.smithi134.stdout:6/156: mkdir d0/d19/de/d12/d2a/d2d/d3a 0 2024-04-22T23:23:38.060 INFO:tasks.workunit.client.1.smithi134.stdout:6/157: stat d0/d19/f38 0 2024-04-22T23:23:38.060 INFO:tasks.workunit.client.0.smithi067.stdout:9/810: dwrite d5/dd/d18/d3c/d40/f62 [4194304,4194304] 0 2024-04-22T23:23:38.061 INFO:tasks.workunit.client.1.smithi134.stdout:6/158: stat d0 0 2024-04-22T23:23:38.062 INFO:tasks.workunit.client.1.smithi134.stdout:4/150: dread d4/d7/f10 [0,4194304] 0 2024-04-22T23:23:38.062 INFO:tasks.workunit.client.0.smithi067.stdout:3/858: dread d3/d4/d10/d38/d53/d55/d74/dd5/ddb/f52 [0,4194304] 0 2024-04-22T23:23:38.064 INFO:tasks.workunit.client.0.smithi067.stdout:7/773: rename d10/d14/d1d/d26/d2c/d41/d82 to d10/d14/d1d/d26/d2c/d41/db0/ddd/d122/d123 0 2024-04-22T23:23:38.065 INFO:tasks.workunit.client.1.smithi134.stdout:8/103: creat d7/f23 x:0 0 0 2024-04-22T23:23:38.067 INFO:tasks.workunit.client.0.smithi067.stdout:1/794: chown d8/d9/l5b 3884502 1 2024-04-22T23:23:38.068 INFO:tasks.workunit.client.0.smithi067.stdout:7/774: truncate d10/d14/d1d/f76 388795 0 2024-04-22T23:23:38.068 INFO:tasks.workunit.client.0.smithi067.stdout:2/822: write d1/d10/f30 [622613,77211] 0 2024-04-22T23:23:38.069 INFO:tasks.workunit.client.0.smithi067.stdout:9/811: read d5/fa [942867,56191] 0 2024-04-22T23:23:38.072 INFO:tasks.workunit.client.0.smithi067.stdout:1/795: readlink d8/d4a/l6f 0 2024-04-22T23:23:38.073 INFO:tasks.workunit.client.0.smithi067.stdout:5/932: dwrite d5/db/d12/d44/d47/f4f [4194304,4194304] 0 2024-04-22T23:23:38.073 INFO:tasks.workunit.client.0.smithi067.stdout:9/812: chown d5/d21/d2c/c67 39379779 1 2024-04-22T23:23:38.074 INFO:tasks.workunit.client.1.smithi134.stdout:8/104: write d7/db/f1b [2266725,68842] 0 2024-04-22T23:23:38.074 INFO:tasks.workunit.client.1.smithi134.stdout:4/151: link d4/dd/l14 d4/d15/l2e 0 2024-04-22T23:23:38.074 INFO:tasks.workunit.client.1.smithi134.stdout:8/105: dread - d7/dc/d19/f21 zero size 2024-04-22T23:23:38.075 INFO:tasks.workunit.client.0.smithi067.stdout:2/823: mkdir d1/d10/d1c/d10f 0 2024-04-22T23:23:38.077 INFO:tasks.workunit.client.1.smithi134.stdout:4/152: dread d4/f20 [0,4194304] 0 2024-04-22T23:23:38.078 INFO:tasks.workunit.client.1.smithi134.stdout:8/106: mkdir d7/dc/d19/d24 0 2024-04-22T23:23:38.081 INFO:tasks.workunit.client.0.smithi067.stdout:1/796: creat d8/d9/d2f/d31/ded/f101 x:0 0 0 2024-04-22T23:23:38.083 INFO:tasks.workunit.client.1.smithi134.stdout:8/107: rename d7/dc/d19/d24 to d7/dc/d19/d25 0 2024-04-22T23:23:38.083 INFO:tasks.workunit.client.0.smithi067.stdout:5/933: write d5/d28/dae/f119 [39774,88707] 0 2024-04-22T23:23:38.087 INFO:tasks.workunit.client.1.smithi134.stdout:4/153: dwrite d4/f21 [0,4194304] 0 2024-04-22T23:23:38.094 INFO:tasks.workunit.client.1.smithi134.stdout:8/108: dwrite f6 [4194304,4194304] 0 2024-04-22T23:23:38.096 INFO:tasks.workunit.client.1.smithi134.stdout:8/109: write d7/f23 [611282,87676] 0 2024-04-22T23:23:38.099 INFO:tasks.workunit.client.0.smithi067.stdout:3/859: dread d3/d4/d10/d38/d53/d55/d74/fc9 [0,4194304] 0 2024-04-22T23:23:38.103 INFO:tasks.workunit.client.0.smithi067.stdout:7/775: dread d10/d14/d1d/d26/d2c/d41/f4f [0,4194304] 0 2024-04-22T23:23:38.108 INFO:tasks.workunit.client.1.smithi134.stdout:8/110: rename d7/db/f1b to d7/dc/d19/d25/f26 0 2024-04-22T23:23:38.109 INFO:tasks.workunit.client.1.smithi134.stdout:6/159: dwrite d0/d19/de/f13 [0,4194304] 0 2024-04-22T23:23:38.114 INFO:tasks.workunit.client.1.smithi134.stdout:4/154: link d4/d7/f10 d4/f2f 0 2024-04-22T23:23:38.116 INFO:tasks.workunit.client.1.smithi134.stdout:8/111: dread d7/db/fe [0,4194304] 0 2024-04-22T23:23:38.117 INFO:tasks.workunit.client.1.smithi134.stdout:8/112: chown d7/dc/d19 8 1 2024-04-22T23:23:38.120 INFO:tasks.workunit.client.1.smithi134.stdout:6/160: dwrite d0/f1a [0,4194304] 0 2024-04-22T23:23:38.123 INFO:tasks.workunit.client.1.smithi134.stdout:8/113: dread d7/f1c [0,4194304] 0 2024-04-22T23:23:38.126 INFO:tasks.workunit.client.0.smithi067.stdout:1/797: dread d8/d9/f3b [0,4194304] 0 2024-04-22T23:23:38.127 INFO:tasks.workunit.client.0.smithi067.stdout:1/798: truncate d8/d12/d40/f44 4655749 0 2024-04-22T23:23:38.131 INFO:tasks.workunit.client.1.smithi134.stdout:6/161: creat d0/d19/de/d12/d2a/f3b x:0 0 0 2024-04-22T23:23:38.136 INFO:tasks.workunit.client.1.smithi134.stdout:5/184: sync 2024-04-22T23:23:38.136 INFO:tasks.workunit.client.1.smithi134.stdout:7/145: sync 2024-04-22T23:23:38.136 INFO:tasks.workunit.client.1.smithi134.stdout:1/114: sync 2024-04-22T23:23:38.136 INFO:tasks.workunit.client.1.smithi134.stdout:2/145: sync 2024-04-22T23:23:38.136 INFO:tasks.workunit.client.0.smithi067.stdout:7/776: read d10/d14/d1d/d26/d7e/fb5 [2656066,47462] 0 2024-04-22T23:23:38.137 INFO:tasks.workunit.client.0.smithi067.stdout:9/813: dwrite d5/dd/d18/d39/d3e/d3f/f73 [0,4194304] 0 2024-04-22T23:23:38.137 INFO:tasks.workunit.client.1.smithi134.stdout:5/185: dread - d9/d1e/f33 zero size 2024-04-22T23:23:38.139 INFO:tasks.workunit.client.1.smithi134.stdout:6/162: dwrite d0/d19/f7 [0,4194304] 0 2024-04-22T23:23:38.139 INFO:tasks.workunit.client.1.smithi134.stdout:5/186: chown d9/dd/d17/d2a 101373759 1 2024-04-22T23:23:38.140 INFO:tasks.workunit.client.1.smithi134.stdout:6/163: fsync d0/d19/de/f13 0 2024-04-22T23:23:38.142 INFO:tasks.workunit.client.0.smithi067.stdout:2/824: mkdir d1/d10/d110 0 2024-04-22T23:23:38.146 INFO:tasks.workunit.client.0.smithi067.stdout:3/860: dwrite d3/d4/dec/f109 [0,4194304] 0 2024-04-22T23:23:38.150 INFO:tasks.workunit.client.0.smithi067.stdout:3/861: chown d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d90/da2/f12a 14783 1 2024-04-22T23:23:38.150 INFO:tasks.workunit.client.1.smithi134.stdout:8/114: unlink d7/dc/l16 0 2024-04-22T23:23:38.151 INFO:tasks.workunit.client.1.smithi134.stdout:5/187: dwrite d9/f2b [4194304,4194304] 0 2024-04-22T23:23:38.151 INFO:tasks.workunit.client.1.smithi134.stdout:8/115: dread - d7/db/f1f zero size 2024-04-22T23:23:38.153 INFO:tasks.workunit.client.0.smithi067.stdout:3/862: read - d3/d4/d28/d40/dfd/dfe/fae zero size 2024-04-22T23:23:38.154 INFO:tasks.workunit.client.0.smithi067.stdout:3/863: fsync d3/d4/d10/fbc 0 2024-04-22T23:23:38.154 INFO:tasks.workunit.client.1.smithi134.stdout:8/116: dread - d7/dc/d19/f21 zero size 2024-04-22T23:23:38.154 INFO:tasks.workunit.client.1.smithi134.stdout:6/164: dwrite d0/d19/f9 [0,4194304] 0 2024-04-22T23:23:38.155 INFO:tasks.workunit.client.0.smithi067.stdout:2/825: mkdir d1/d10/d72/d53/d111 0 2024-04-22T23:23:38.155 INFO:tasks.workunit.client.0.smithi067.stdout:3/864: chown d3/d4/d10/d1f/d3c/cca 3375 1 2024-04-22T23:23:38.155 INFO:tasks.workunit.client.1.smithi134.stdout:5/188: write d9/f19 [417007,41660] 0 2024-04-22T23:23:38.157 INFO:tasks.workunit.client.1.smithi134.stdout:2/146: symlink d2/dd/l34 0 2024-04-22T23:23:38.157 INFO:tasks.workunit.client.0.smithi067.stdout:2/826: symlink d1/d10/d41/l112 0 2024-04-22T23:23:38.160 INFO:tasks.workunit.client.1.smithi134.stdout:7/146: mkdir d4/d8/d3b 0 2024-04-22T23:23:38.163 INFO:tasks.workunit.client.0.smithi067.stdout:2/827: dread d1/d42/f57 [0,4194304] 0 2024-04-22T23:23:38.166 INFO:tasks.workunit.client.1.smithi134.stdout:6/165: dread d0/d19/f7 [0,4194304] 0 2024-04-22T23:23:38.167 INFO:tasks.workunit.client.0.smithi067.stdout:2/828: chown d1/d10/d72/d53/dba/ld4 98 1 2024-04-22T23:23:38.168 INFO:tasks.workunit.client.1.smithi134.stdout:8/117: unlink d7/dc/d19/l1d 0 2024-04-22T23:23:38.169 INFO:tasks.workunit.client.1.smithi134.stdout:2/147: creat d2/d15/d24/f35 x:0 0 0 2024-04-22T23:23:38.171 INFO:tasks.workunit.client.1.smithi134.stdout:8/118: chown c2 47228327 1 2024-04-22T23:23:38.171 INFO:tasks.workunit.client.1.smithi134.stdout:2/148: chown d2/c11 27 1 2024-04-22T23:23:38.171 INFO:tasks.workunit.client.1.smithi134.stdout:4/155: dwrite d4/d17/f1f [4194304,4194304] 0 2024-04-22T23:23:38.172 INFO:tasks.workunit.client.0.smithi067.stdout:7/777: write d10/d18/f63 [1025569,91394] 0 2024-04-22T23:23:38.173 INFO:tasks.workunit.client.1.smithi134.stdout:6/166: truncate d0/d19/de/d2f/f32 644208 0 2024-04-22T23:23:38.173 INFO:tasks.workunit.client.1.smithi134.stdout:1/115: mkdir db/d15/d17/d20/d2a/d2d 0 2024-04-22T23:23:38.173 INFO:tasks.workunit.client.1.smithi134.stdout:8/119: dread - d7/dc/f22 zero size 2024-04-22T23:23:38.174 INFO:tasks.workunit.client.0.smithi067.stdout:7/778: fdatasync d10/d14/d1d/d26/d2c/d7a/f102 0 2024-04-22T23:23:38.176 INFO:tasks.workunit.client.0.smithi067.stdout:7/779: fsync d10/d18/f1e 0 2024-04-22T23:23:38.176 INFO:tasks.workunit.client.1.smithi134.stdout:4/156: write d4/f20 [485356,96836] 0 2024-04-22T23:23:38.178 INFO:tasks.workunit.client.0.smithi067.stdout:7/780: chown d10/d14/d1d/d26/d2c/d41/db0/ddd/d122 20200 1 2024-04-22T23:23:38.180 INFO:tasks.workunit.client.0.smithi067.stdout:7/781: write d10/d14/d1d/d26/d80/f8a [559965,85422] 0 2024-04-22T23:23:38.182 INFO:tasks.workunit.client.1.smithi134.stdout:8/120: unlink d7/dc/c18 0 2024-04-22T23:23:38.183 INFO:tasks.workunit.client.1.smithi134.stdout:6/167: symlink d0/d19/de/d12/d2a/d2d/l3c 0 2024-04-22T23:23:38.184 INFO:tasks.workunit.client.0.smithi067.stdout:1/799: truncate d8/d9/de0/d60/d62/fa8 718571 0 2024-04-22T23:23:38.185 INFO:tasks.workunit.client.0.smithi067.stdout:9/814: truncate d5/dd/de6/ffd 696203 0 2024-04-22T23:23:38.186 INFO:tasks.workunit.client.1.smithi134.stdout:4/157: write d4/d15/f1c [7497123,90638] 0 2024-04-22T23:23:38.187 INFO:tasks.workunit.client.1.smithi134.stdout:6/168: symlink d0/d19/de/d12/d2a/l3d 0 2024-04-22T23:23:38.187 INFO:tasks.workunit.client.0.smithi067.stdout:7/782: rename d10/d14/d1d/d46/d6a/dcc to d10/d14/d1d/d46/d124 0 2024-04-22T23:23:38.188 INFO:tasks.workunit.client.1.smithi134.stdout:1/116: mkdir db/d15/d17/d20/d2a/d2d/d2e 0 2024-04-22T23:23:38.190 INFO:tasks.workunit.client.0.smithi067.stdout:9/815: mkdir d5/dd/d10d 0 2024-04-22T23:23:38.190 INFO:tasks.workunit.client.1.smithi134.stdout:4/158: rmdir d4/d7 39 2024-04-22T23:23:38.191 INFO:tasks.workunit.client.0.smithi067.stdout:1/800: dwrite d8/d9/d2d/d4d/d83/fb9 [0,4194304] 0 2024-04-22T23:23:38.193 INFO:tasks.workunit.client.0.smithi067.stdout:7/783: unlink d10/d14/d1d/d26/d2c/l33 0 2024-04-22T23:23:38.194 INFO:tasks.workunit.client.1.smithi134.stdout:1/117: symlink db/d15/d17/d20/l2f 0 2024-04-22T23:23:38.195 INFO:tasks.workunit.client.0.smithi067.stdout:9/816: mknod d5/dd/d18/d39/d3e/c10e 0 2024-04-22T23:23:38.196 INFO:tasks.workunit.client.0.smithi067.stdout:1/801: mkdir d8/d9/d102 0 2024-04-22T23:23:38.196 INFO:tasks.workunit.client.1.smithi134.stdout:8/121: getdents d7/dc/d19/d25 0 2024-04-22T23:23:38.197 INFO:tasks.workunit.client.0.smithi067.stdout:9/817: unlink d5/d21/l28 0 2024-04-22T23:23:38.197 INFO:tasks.workunit.client.1.smithi134.stdout:4/159: mknod d4/c30 0 2024-04-22T23:23:38.198 INFO:tasks.workunit.client.0.smithi067.stdout:1/802: rename d8/d9/fd to d8/d9/de0/d5d/dd4/f103 0 2024-04-22T23:23:38.199 INFO:tasks.workunit.client.0.smithi067.stdout:1/803: rmdir d8/d9 39 2024-04-22T23:23:38.200 INFO:tasks.workunit.client.1.smithi134.stdout:6/169: dwrite d0/f3 [0,4194304] 0 2024-04-22T23:23:38.200 INFO:tasks.workunit.client.1.smithi134.stdout:1/118: mknod db/d15/d17/d20/c30 0 2024-04-22T23:23:38.200 INFO:tasks.workunit.client.0.smithi067.stdout:8/863: sync 2024-04-22T23:23:38.201 INFO:tasks.workunit.client.0.smithi067.stdout:1/804: chown d8/d9/de0/d60/f92 227637 1 2024-04-22T23:23:38.202 INFO:tasks.workunit.client.0.smithi067.stdout:8/864: symlink d1/d3/d27/l118 0 2024-04-22T23:23:38.202 INFO:tasks.workunit.client.1.smithi134.stdout:4/160: creat d4/dd/f31 x:0 0 0 2024-04-22T23:23:38.203 INFO:tasks.workunit.client.1.smithi134.stdout:1/119: write db/d15/d17/f1a [28417,54932] 0 2024-04-22T23:23:38.204 INFO:tasks.workunit.client.1.smithi134.stdout:4/161: chown d4/c24 14 1 2024-04-22T23:23:38.205 INFO:tasks.workunit.client.1.smithi134.stdout:1/120: readlink db/d15/d17/d20/l21 0 2024-04-22T23:23:38.205 INFO:tasks.workunit.client.1.smithi134.stdout:8/122: rename d7/c1a to d7/dc/d19/c27 0 2024-04-22T23:23:38.206 INFO:tasks.workunit.client.1.smithi134.stdout:1/121: chown db/d16/c23 0 1 2024-04-22T23:23:38.211 INFO:tasks.workunit.client.1.smithi134.stdout:4/162: mkdir d4/d7/d32 0 2024-04-22T23:23:38.215 INFO:tasks.workunit.client.1.smithi134.stdout:4/163: write d4/f20 [247363,29054] 0 2024-04-22T23:23:38.216 INFO:tasks.workunit.client.1.smithi134.stdout:4/164: dread - d4/d29/f2c zero size 2024-04-22T23:23:38.217 INFO:tasks.workunit.client.1.smithi134.stdout:2/149: dwrite d2/f18 [0,4194304] 0 2024-04-22T23:23:38.217 INFO:tasks.workunit.client.1.smithi134.stdout:8/123: dread d7/dc/ff [0,4194304] 0 2024-04-22T23:23:38.241 INFO:tasks.workunit.client.1.smithi134.stdout:2/150: dread d2/dd/fe [0,4194304] 0 2024-04-22T23:23:38.244 INFO:tasks.workunit.client.1.smithi134.stdout:6/170: write d0/f8 [548398,8037] 0 2024-04-22T23:23:38.246 INFO:tasks.workunit.client.1.smithi134.stdout:4/165: rename d4/dd/l12 to d4/d15/l33 0 2024-04-22T23:23:38.248 INFO:tasks.workunit.client.1.smithi134.stdout:2/151: creat d2/d15/d24/d1d/f36 x:0 0 0 2024-04-22T23:23:38.259 INFO:tasks.workunit.client.1.smithi134.stdout:4/166: creat d4/d15/f34 x:0 0 0 2024-04-22T23:23:38.263 INFO:tasks.workunit.client.0.smithi067.stdout:0/969: creat db/d21/d29/d75/d9d/dde/d12c/f13f x:0 0 0 2024-04-22T23:23:38.265 INFO:tasks.workunit.client.1.smithi134.stdout:8/124: dwrite d7/dc/ff [0,4194304] 0 2024-04-22T23:23:38.268 INFO:tasks.workunit.client.0.smithi067.stdout:2/829: dread d1/d4f/fe4 [0,4194304] 0 2024-04-22T23:23:38.271 INFO:tasks.workunit.client.1.smithi134.stdout:8/125: dwrite d7/dc/ff [0,4194304] 0 2024-04-22T23:23:38.271 INFO:tasks.workunit.client.0.smithi067.stdout:0/970: chown db/d9a/d56/d6b/db9/fe9 2089 1 2024-04-22T23:23:38.280 INFO:tasks.workunit.client.0.smithi067.stdout:3/865: dwrite d3/d4/d10/d38/dbf/ff0 [0,4194304] 0 2024-04-22T23:23:38.282 INFO:tasks.workunit.client.0.smithi067.stdout:8/865: dwrite d1/f76 [4194304,4194304] 0 2024-04-22T23:23:38.282 INFO:tasks.workunit.client.1.smithi134.stdout:4/167: mkdir d4/d35 0 2024-04-22T23:23:38.283 INFO:tasks.workunit.client.0.smithi067.stdout:1/805: dwrite d8/d9/d2f/d33/d5c/f65 [4194304,4194304] 0 2024-04-22T23:23:38.284 INFO:tasks.workunit.client.1.smithi134.stdout:4/168: truncate d4/d15/f26 1042996 0 2024-04-22T23:23:38.286 INFO:tasks.workunit.client.1.smithi134.stdout:8/126: symlink d7/l28 0 2024-04-22T23:23:38.288 INFO:tasks.workunit.client.0.smithi067.stdout:9/818: dread d5/fa [0,4194304] 0 2024-04-22T23:23:38.288 INFO:tasks.workunit.client.1.smithi134.stdout:4/169: symlink d4/d15/l36 0 2024-04-22T23:23:38.289 INFO:tasks.workunit.client.1.smithi134.stdout:8/127: symlink d7/dc/d19/l29 0 2024-04-22T23:23:38.290 INFO:tasks.workunit.client.1.smithi134.stdout:8/128: write d7/dc/d19/f21 [662769,10455] 0 2024-04-22T23:23:38.293 INFO:tasks.workunit.client.1.smithi134.stdout:8/129: write d7/dc/f22 [861288,69472] 0 2024-04-22T23:23:38.294 INFO:tasks.workunit.client.1.smithi134.stdout:8/130: stat d7/f23 0 2024-04-22T23:23:38.297 INFO:tasks.workunit.client.0.smithi067.stdout:8/866: read d1/d3/d27/f58 [2578429,8091] 0 2024-04-22T23:23:38.299 INFO:tasks.workunit.client.1.smithi134.stdout:4/170: dwrite d4/d15/f26 [0,4194304] 0 2024-04-22T23:23:38.303 INFO:tasks.workunit.client.1.smithi134.stdout:8/131: mknod d7/dc/c2a 0 2024-04-22T23:23:38.304 INFO:tasks.workunit.client.1.smithi134.stdout:8/132: write d7/f20 [776663,27195] 0 2024-04-22T23:23:38.305 INFO:tasks.workunit.client.0.smithi067.stdout:8/867: write d1/f76 [2485642,107063] 0 2024-04-22T23:23:38.306 INFO:tasks.workunit.client.1.smithi134.stdout:4/171: fsync d4/f2f 0 2024-04-22T23:23:38.309 INFO:tasks.workunit.client.1.smithi134.stdout:8/133: unlink d7/db/c11 0 2024-04-22T23:23:38.310 INFO:tasks.workunit.client.1.smithi134.stdout:4/172: dread d4/f2f [0,4194304] 0 2024-04-22T23:23:38.311 INFO:tasks.workunit.client.1.smithi134.stdout:8/134: unlink c2 0 2024-04-22T23:23:38.312 INFO:tasks.workunit.client.1.smithi134.stdout:8/135: dread - d7/db/f1f zero size 2024-04-22T23:23:38.313 INFO:tasks.workunit.client.1.smithi134.stdout:8/136: fdatasync f4 0 2024-04-22T23:23:38.313 INFO:tasks.workunit.client.1.smithi134.stdout:8/137: readlink d7/l1e 0 2024-04-22T23:23:38.318 INFO:tasks.workunit.client.1.smithi134.stdout:4/173: dwrite d4/d15/f26 [0,4194304] 0 2024-04-22T23:23:38.325 INFO:tasks.workunit.client.1.smithi134.stdout:8/138: chown d7/dc/d19/c27 1 1 2024-04-22T23:23:38.326 INFO:tasks.workunit.client.1.smithi134.stdout:4/174: dwrite d4/f2b [0,4194304] 0 2024-04-22T23:23:38.327 INFO:tasks.workunit.client.0.smithi067.stdout:3/866: mknod d3/d4/d10/d1f/c132 0 2024-04-22T23:23:38.331 INFO:tasks.workunit.client.1.smithi134.stdout:4/175: dread d4/f2b [0,4194304] 0 2024-04-22T23:23:38.332 INFO:tasks.workunit.client.1.smithi134.stdout:4/176: write d4/f6 [1468270,77211] 0 2024-04-22T23:23:38.339 INFO:tasks.workunit.client.1.smithi134.stdout:4/177: dwrite d4/d17/f22 [0,4194304] 0 2024-04-22T23:23:38.343 INFO:tasks.workunit.client.1.smithi134.stdout:8/139: rename d7/f20 to d7/dc/f2b 0 2024-04-22T23:23:38.343 INFO:tasks.workunit.client.0.smithi067.stdout:8/868: write d1/dd/d80/d6f/f32 [2085600,62783] 0 2024-04-22T23:23:38.348 INFO:tasks.workunit.client.0.smithi067.stdout:6/865: sync 2024-04-22T23:23:38.348 INFO:tasks.workunit.client.1.smithi134.stdout:4/178: dwrite d4/d15/f26 [0,4194304] 0 2024-04-22T23:23:38.349 INFO:tasks.workunit.client.0.smithi067.stdout:6/866: fdatasync d17/d83/d9b/ddd/fe6 0 2024-04-22T23:23:38.349 INFO:tasks.workunit.client.0.smithi067.stdout:9/819: link d5/d21/d2c/f7a d5/d21/d47/d57/dc0/f10f 0 2024-04-22T23:23:38.354 INFO:tasks.workunit.client.0.smithi067.stdout:5/934: symlink d5/d28/daf/l144 0 2024-04-22T23:23:38.354 INFO:tasks.workunit.client.0.smithi067.stdout:1/806: creat d8/d12/f104 x:0 0 0 2024-04-22T23:23:38.355 INFO:tasks.workunit.client.1.smithi134.stdout:8/140: dwrite d7/dc/f22 [0,4194304] 0 2024-04-22T23:23:38.356 INFO:tasks.workunit.client.1.smithi134.stdout:4/179: mknod d4/d29/c37 0 2024-04-22T23:23:38.357 INFO:tasks.workunit.client.0.smithi067.stdout:6/867: dread - d17/f9c zero size 2024-04-22T23:23:38.357 INFO:tasks.workunit.client.1.smithi134.stdout:4/180: write d4/f6 [3426646,21628] 0 2024-04-22T23:23:38.359 INFO:tasks.workunit.client.0.smithi067.stdout:3/867: creat d3/d4/d10/d38/dbf/d113/f133 x:0 0 0 2024-04-22T23:23:38.361 INFO:tasks.workunit.client.0.smithi067.stdout:9/820: dwrite d5/dd/d18/d39/d49/d54/fc9 [4194304,4194304] 0 2024-04-22T23:23:38.364 INFO:tasks.workunit.client.1.smithi134.stdout:0/173: sync 2024-04-22T23:23:38.365 INFO:tasks.workunit.client.1.smithi134.stdout:4/181: unlink d4/dd/c23 0 2024-04-22T23:23:38.366 INFO:tasks.workunit.client.0.smithi067.stdout:8/869: mkdir d1/dd/d119 0 2024-04-22T23:23:38.367 INFO:tasks.workunit.client.1.smithi134.stdout:8/141: mkdir d7/dc/d2c 0 2024-04-22T23:23:38.367 INFO:tasks.workunit.client.1.smithi134.stdout:8/142: readlink l5 0 2024-04-22T23:23:38.367 INFO:tasks.workunit.client.1.smithi134.stdout:8/143: dread - d7/db/f1f zero size 2024-04-22T23:23:38.368 INFO:tasks.workunit.client.1.smithi134.stdout:4/182: mknod d4/d7/c38 0 2024-04-22T23:23:38.368 INFO:tasks.workunit.client.1.smithi134.stdout:4/183: dread - d4/d7/f28 zero size 2024-04-22T23:23:38.370 INFO:tasks.workunit.client.1.smithi134.stdout:8/144: write d7/db/fe [3394579,57117] 0 2024-04-22T23:23:38.370 INFO:tasks.workunit.client.0.smithi067.stdout:0/971: getdents db/de/d11 0 2024-04-22T23:23:38.373 INFO:tasks.workunit.client.0.smithi067.stdout:8/870: rmdir d1/dd/d80 39 2024-04-22T23:23:38.373 INFO:tasks.workunit.client.1.smithi134.stdout:4/184: link d4/d29/c37 d4/d17/d25/c39 0 2024-04-22T23:23:38.374 INFO:tasks.workunit.client.0.smithi067.stdout:3/868: dwrite d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d8c/fc5 [4194304,4194304] 0 2024-04-22T23:23:38.375 INFO:tasks.workunit.client.0.smithi067.stdout:9/821: symlink d5/dd/d18/d3c/ddf/d101/l110 0 2024-04-22T23:23:38.377 INFO:tasks.workunit.client.1.smithi134.stdout:8/145: dwrite d7/f1c [0,4194304] 0 2024-04-22T23:23:38.378 INFO:tasks.workunit.client.0.smithi067.stdout:5/935: fdatasync d5/fa4 0 2024-04-22T23:23:38.380 INFO:tasks.workunit.client.1.smithi134.stdout:4/185: getdents d4 0 2024-04-22T23:23:38.381 INFO:tasks.workunit.client.0.smithi067.stdout:5/936: stat d5/db/d12/d8a/d141 0 2024-04-22T23:23:38.382 INFO:tasks.workunit.client.1.smithi134.stdout:8/146: mkdir d7/db/d2d 0 2024-04-22T23:23:38.383 INFO:tasks.workunit.client.0.smithi067.stdout:6/868: dwrite d17/d39/f103 [0,4194304] 0 2024-04-22T23:23:38.386 INFO:tasks.workunit.client.0.smithi067.stdout:0/972: getdents db/d21/d29/d75/d9d/dde/d10f/db3 0 2024-04-22T23:23:38.387 INFO:tasks.workunit.client.0.smithi067.stdout:9/822: truncate d5/dd/d18/d39/d8f/fb1 673040 0 2024-04-22T23:23:38.388 INFO:tasks.workunit.client.1.smithi134.stdout:8/147: creat d7/db/d2d/f2e x:0 0 0 2024-04-22T23:23:38.388 INFO:tasks.workunit.client.0.smithi067.stdout:7/784: sync 2024-04-22T23:23:38.390 INFO:tasks.workunit.client.1.smithi134.stdout:4/186: dwrite d4/dd/f16 [0,4194304] 0 2024-04-22T23:23:38.391 INFO:tasks.workunit.client.0.smithi067.stdout:6/869: symlink d17/d39/d3a/da9/l121 0 2024-04-22T23:23:38.393 INFO:tasks.workunit.client.0.smithi067.stdout:8/871: link d1/d41/f83 d1/dd/d80/db9/d110/d62/ddf/f11a 0 2024-04-22T23:23:38.396 INFO:tasks.workunit.client.1.smithi134.stdout:8/148: symlink d7/dc/d2c/l2f 0 2024-04-22T23:23:38.396 INFO:tasks.workunit.client.0.smithi067.stdout:7/785: mkdir d10/d14/d1d/d26/d2c/d7a/dc2/d125 0 2024-04-22T23:23:38.397 INFO:tasks.workunit.client.1.smithi134.stdout:8/149: fdatasync d7/dc/d19/f21 0 2024-04-22T23:23:38.397 INFO:tasks.workunit.client.0.smithi067.stdout:7/786: write d10/d47/d49/f8b [204871,108074] 0 2024-04-22T23:23:38.399 INFO:tasks.workunit.client.0.smithi067.stdout:8/872: symlink d1/dd/d80/db9/d110/d62/d75/l11b 0 2024-04-22T23:23:38.399 INFO:tasks.workunit.client.1.smithi134.stdout:4/187: mkdir d4/d7/d32/d3a 0 2024-04-22T23:23:38.400 INFO:tasks.workunit.client.0.smithi067.stdout:7/787: read - d10/d18/d88/fa2 zero size 2024-04-22T23:23:38.407 INFO:tasks.workunit.client.0.smithi067.stdout:8/873: creat d1/d41/f11c x:0 0 0 2024-04-22T23:23:38.407 INFO:tasks.workunit.client.1.smithi134.stdout:8/150: dwrite d7/dc/d19/f21 [0,4194304] 0 2024-04-22T23:23:38.407 INFO:tasks.workunit.client.1.smithi134.stdout:4/188: rename d4/d17/d25/c39 to d4/dd/c3b 0 2024-04-22T23:23:38.408 INFO:tasks.workunit.client.0.smithi067.stdout:2/830: write d1/d10/d72/d53/f58 [3143054,41511] 0 2024-04-22T23:23:38.409 INFO:tasks.workunit.client.1.smithi134.stdout:8/151: fsync d7/f1c 0 2024-04-22T23:23:38.414 INFO:tasks.workunit.client.1.smithi134.stdout:4/189: truncate d4/f2f 2876570 0 2024-04-22T23:23:38.416 INFO:tasks.workunit.client.0.smithi067.stdout:1/807: write d8/d12/d40/f57 [978656,3518] 0 2024-04-22T23:23:38.416 INFO:tasks.workunit.client.0.smithi067.stdout:5/937: dread d5/db/d12/d44/d47/f114 [0,4194304] 0 2024-04-22T23:23:38.417 INFO:tasks.workunit.client.1.smithi134.stdout:4/190: creat d4/d7/f3c x:0 0 0 2024-04-22T23:23:38.417 INFO:tasks.workunit.client.1.smithi134.stdout:8/152: mknod d7/dc/c30 0 2024-04-22T23:23:38.419 INFO:tasks.workunit.client.0.smithi067.stdout:2/831: creat d1/d10/d1c/d4d/f113 x:0 0 0 2024-04-22T23:23:38.420 INFO:tasks.workunit.client.0.smithi067.stdout:7/788: dread d10/d47/f68 [0,4194304] 0 2024-04-22T23:23:38.421 INFO:tasks.workunit.client.0.smithi067.stdout:0/973: rename db/d21/lb2 to db/d71/l140 0 2024-04-22T23:23:38.422 INFO:tasks.workunit.client.0.smithi067.stdout:1/808: dread d8/d9/d2f/d59/f78 [0,4194304] 0 2024-04-22T23:23:38.423 INFO:tasks.workunit.client.1.smithi134.stdout:8/153: creat d7/dc/d19/d25/f31 x:0 0 0 2024-04-22T23:23:38.423 INFO:tasks.workunit.client.1.smithi134.stdout:4/191: dwrite d4/d15/f34 [0,4194304] 0 2024-04-22T23:23:38.425 INFO:tasks.workunit.client.1.smithi134.stdout:8/154: read - d7/dc/f12 zero size 2024-04-22T23:23:38.425 INFO:tasks.workunit.client.0.smithi067.stdout:3/869: dwrite d3/d4/d28/d40/dfd/dfe/d4b/f4e [0,4194304] 0 2024-04-22T23:23:38.426 INFO:tasks.workunit.client.1.smithi134.stdout:8/155: dread - d7/dc/f12 zero size 2024-04-22T23:23:38.426 INFO:tasks.workunit.client.0.smithi067.stdout:7/789: chown d10/d14/d1d/d26/d2c/d41/db0/ddd/d122/d123/f113 9 1 2024-04-22T23:23:38.426 INFO:tasks.workunit.client.0.smithi067.stdout:1/809: write d8/d9/d2d/d4d/d9c/ffc [17055,34609] 0 2024-04-22T23:23:38.428 INFO:tasks.workunit.client.1.smithi134.stdout:4/192: symlink d4/d17/l3d 0 2024-04-22T23:23:38.430 INFO:tasks.workunit.client.1.smithi134.stdout:4/193: chown d4/d7/d32 7732 1 2024-04-22T23:23:38.431 INFO:tasks.workunit.client.0.smithi067.stdout:5/938: unlink d5/db/d12/dbb/d61/f82 0 2024-04-22T23:23:38.433 INFO:tasks.workunit.client.1.smithi134.stdout:8/156: dwrite d7/dc/ff [0,4194304] 0 2024-04-22T23:23:38.434 INFO:tasks.workunit.client.0.smithi067.stdout:5/939: readlink d5/d1a/l1c 0 2024-04-22T23:23:38.434 INFO:tasks.workunit.client.1.smithi134.stdout:8/157: readlink d7/l1e 0 2024-04-22T23:23:38.435 INFO:tasks.workunit.client.1.smithi134.stdout:4/194: mknod d4/d15/d2a/c3e 0 2024-04-22T23:23:38.436 INFO:tasks.workunit.client.0.smithi067.stdout:1/810: mkdir d8/d9/de0/d60/d105 0 2024-04-22T23:23:38.438 INFO:tasks.workunit.client.0.smithi067.stdout:9/823: dread d5/d21/d47/fa4 [0,4194304] 0 2024-04-22T23:23:38.438 INFO:tasks.workunit.client.1.smithi134.stdout:8/158: truncate d7/db/fe 4813809 0 2024-04-22T23:23:38.439 INFO:tasks.workunit.client.1.smithi134.stdout:8/159: write d7/dc/ff [4898159,108920] 0 2024-04-22T23:23:38.439 INFO:tasks.workunit.client.1.smithi134.stdout:4/195: symlink d4/d29/l3f 0 2024-04-22T23:23:38.439 INFO:tasks.workunit.client.0.smithi067.stdout:3/870: dwrite d3/d4/d28/d40/dfd/dfe/f69 [0,4194304] 0 2024-04-22T23:23:38.441 INFO:tasks.workunit.client.1.smithi134.stdout:8/160: truncate d7/db/fe 5178135 0 2024-04-22T23:23:38.442 INFO:tasks.workunit.client.1.smithi134.stdout:8/161: fsync d7/dc/f12 0 2024-04-22T23:23:38.444 INFO:tasks.workunit.client.0.smithi067.stdout:0/974: symlink db/d9a/d56/d70/l141 0 2024-04-22T23:23:38.450 INFO:tasks.workunit.client.1.smithi134.stdout:8/162: rename d7/dc/c10 to d7/db/d2d/c32 0 2024-04-22T23:23:38.451 INFO:tasks.workunit.client.0.smithi067.stdout:7/790: write fe [1886488,39174] 0 2024-04-22T23:23:38.453 INFO:tasks.workunit.client.0.smithi067.stdout:8/874: dwrite d1/dd/d80/d6f/db6/fcd [0,4194304] 0 2024-04-22T23:23:38.454 INFO:tasks.workunit.client.0.smithi067.stdout:7/791: chown d10/d14/d1d/d69/fb8 0 1 2024-04-22T23:23:38.457 INFO:tasks.workunit.client.1.smithi134.stdout:8/163: mkdir d7/db/d2d/d33 0 2024-04-22T23:23:38.460 INFO:tasks.workunit.client.0.smithi067.stdout:3/871: write d3/d4/d28/d40/dfd/dfe/d4b/f91 [190666,76205] 0 2024-04-22T23:23:38.462 INFO:tasks.workunit.client.1.smithi134.stdout:8/164: link d7/db/f1f d7/dc/d19/d25/f34 0 2024-04-22T23:23:38.463 INFO:tasks.workunit.client.0.smithi067.stdout:3/872: fsync d3/d4/d10/d38/d53/d55/fe2 0 2024-04-22T23:23:38.465 INFO:tasks.workunit.client.0.smithi067.stdout:0/975: getdents db/d9a/d56/d70/dba 0 2024-04-22T23:23:38.465 INFO:tasks.workunit.client.0.smithi067.stdout:7/792: getdents d10/d14/d1d/d46/d6a/df1 0 2024-04-22T23:23:38.466 INFO:tasks.workunit.client.1.smithi134.stdout:8/165: dread d7/f23 [0,4194304] 0 2024-04-22T23:23:38.468 INFO:tasks.workunit.client.0.smithi067.stdout:1/811: dwrite d8/d9/d2d/d4d/d48/f8d [0,4194304] 0 2024-04-22T23:23:38.468 INFO:tasks.workunit.client.0.smithi067.stdout:8/875: dread d1/dd/d80/d6f/db6/fc4 [0,4194304] 0 2024-04-22T23:23:38.469 INFO:tasks.workunit.client.0.smithi067.stdout:7/793: chown d10/d14/d1d/d46/d6a/f86 70314323 1 2024-04-22T23:23:38.469 INFO:tasks.workunit.client.0.smithi067.stdout:3/873: unlink d3/d4/d10/d19/l108 0 2024-04-22T23:23:38.469 INFO:tasks.workunit.client.1.smithi134.stdout:8/166: dread d7/db/fe [0,4194304] 0 2024-04-22T23:23:38.470 INFO:tasks.workunit.client.0.smithi067.stdout:0/976: unlink db/d71/fbc 0 2024-04-22T23:23:38.470 INFO:tasks.workunit.client.1.smithi134.stdout:8/167: chown d7 3014 1 2024-04-22T23:23:38.471 INFO:tasks.workunit.client.0.smithi067.stdout:8/876: fdatasync d1/d3/d27/f63 0 2024-04-22T23:23:38.471 INFO:tasks.workunit.client.1.smithi134.stdout:8/168: dread d7/dc/f14 [0,4194304] 0 2024-04-22T23:23:38.472 INFO:tasks.workunit.client.0.smithi067.stdout:8/877: fdatasync d1/fc0 0 2024-04-22T23:23:38.474 INFO:tasks.workunit.client.0.smithi067.stdout:1/812: creat d8/d9/de0/d5d/dd4/f106 x:0 0 0 2024-04-22T23:23:38.476 INFO:tasks.workunit.client.0.smithi067.stdout:5/940: truncate d5/db/d12/d44/d47/f4f 3818625 0 2024-04-22T23:23:38.480 INFO:tasks.workunit.client.0.smithi067.stdout:9/824: dread d5/dd/d18/d39/f64 [0,4194304] 0 2024-04-22T23:23:38.480 INFO:tasks.workunit.client.0.smithi067.stdout:0/977: dwrite db/d21/d29/d2d/f42 [0,4194304] 0 2024-04-22T23:23:38.482 INFO:tasks.workunit.client.0.smithi067.stdout:8/878: unlink d1/dd/d80/db9/d110/d62/d75/fdd 0 2024-04-22T23:23:38.484 INFO:tasks.workunit.client.0.smithi067.stdout:0/978: readlink db/d9a/d56/d70/l85 0 2024-04-22T23:23:38.485 INFO:tasks.workunit.client.0.smithi067.stdout:7/794: rename d10/ce6 to d10/d14/d1d/d26/d2c/df4/c126 0 2024-04-22T23:23:38.486 INFO:tasks.workunit.client.0.smithi067.stdout:1/813: mknod d8/c107 0 2024-04-22T23:23:38.487 INFO:tasks.workunit.client.0.smithi067.stdout:8/879: creat d1/d3/d10/f11d x:0 0 0 2024-04-22T23:23:38.488 INFO:tasks.workunit.client.0.smithi067.stdout:1/814: fsync d8/d9/d2f/d33/fb6 0 2024-04-22T23:23:38.488 INFO:tasks.workunit.client.0.smithi067.stdout:0/979: mkdir db/d71/d10a/d142 0 2024-04-22T23:23:38.492 INFO:tasks.workunit.client.0.smithi067.stdout:7/795: mknod d10/d14/d1d/d26/d2c/d7a/dc2/d121/c127 0 2024-04-22T23:23:38.494 INFO:tasks.workunit.client.0.smithi067.stdout:5/941: write d5/db/d12/d44/d47/f4f [1852237,6786] 0 2024-04-22T23:23:38.496 INFO:tasks.workunit.client.1.smithi134.stdout:4/196: dwrite d4/d7/f10 [0,4194304] 0 2024-04-22T23:23:38.496 INFO:tasks.workunit.client.0.smithi067.stdout:9/825: rename d5/dd/d18/d39/d49/d54/f9a to d5/dd/d18/d3c/f111 0 2024-04-22T23:23:38.497 INFO:tasks.workunit.client.1.smithi134.stdout:4/197: write d4/d7/f3c [35977,86817] 0 2024-04-22T23:23:38.497 INFO:tasks.workunit.client.0.smithi067.stdout:1/815: symlink d8/d9/d2d/d4d/d83/l108 0 2024-04-22T23:23:38.500 INFO:tasks.workunit.client.0.smithi067.stdout:0/980: dwrite db/d21/d29/d75/d9d/dde/d10f/f135 [0,4194304] 0 2024-04-22T23:23:38.501 INFO:tasks.workunit.client.0.smithi067.stdout:8/880: creat d1/dd/d80/d6f/f11e x:0 0 0 2024-04-22T23:23:38.502 INFO:tasks.workunit.client.1.smithi134.stdout:4/198: rename d4/dd/l14 to d4/l40 0 2024-04-22T23:23:38.504 INFO:tasks.workunit.client.0.smithi067.stdout:5/942: symlink d5/db/d12/dbb/d5f/d79/l145 0 2024-04-22T23:23:38.505 INFO:tasks.workunit.client.0.smithi067.stdout:5/943: fdatasync d5/d28/d36/d40/f70 0 2024-04-22T23:23:38.507 INFO:tasks.workunit.client.0.smithi067.stdout:9/826: write d5/d21/d47/d56/d8c/fbe [3601812,10048] 0 2024-04-22T23:23:38.508 INFO:tasks.workunit.client.0.smithi067.stdout:1/816: symlink d8/d9/d2d/d4d/d48/l109 0 2024-04-22T23:23:38.509 INFO:tasks.workunit.client.0.smithi067.stdout:3/874: dread d3/d4/d28/d40/dfd/dfe/f83 [0,4194304] 0 2024-04-22T23:23:38.509 INFO:tasks.workunit.client.0.smithi067.stdout:9/827: fdatasync d5/dd/d18/d39/d4d/fed 0 2024-04-22T23:23:38.511 INFO:tasks.workunit.client.0.smithi067.stdout:8/881: creat d1/dd/d80/d6f/db6/f11f x:0 0 0 2024-04-22T23:23:38.511 INFO:tasks.workunit.client.0.smithi067.stdout:0/981: rename db/d21/d29/d2c/f7d to db/d71/d10a/f143 0 2024-04-22T23:23:38.512 INFO:tasks.workunit.client.0.smithi067.stdout:7/796: link d10/d14/d1d/d26/d2c/d7a/f87 d10/d18/d88/daf/d100/d62/f128 0 2024-04-22T23:23:38.513 INFO:tasks.workunit.client.0.smithi067.stdout:9/828: mknod d5/dd/da9/c112 0 2024-04-22T23:23:38.514 INFO:tasks.workunit.client.0.smithi067.stdout:3/875: chown d3/d4/d10/d38/d53/d55/d74/dd5/ddb/c56 3 1 2024-04-22T23:23:38.515 INFO:tasks.workunit.client.0.smithi067.stdout:5/944: unlink d5/d1a/d103/d4d/d101/f12d 0 2024-04-22T23:23:38.515 INFO:tasks.workunit.client.0.smithi067.stdout:8/882: read d1/f6 [7604180,72940] 0 2024-04-22T23:23:38.516 INFO:tasks.workunit.client.0.smithi067.stdout:3/876: write d3/d4/f79 [420414,59522] 0 2024-04-22T23:23:38.517 INFO:tasks.workunit.client.0.smithi067.stdout:1/817: link d8/d9/d2d/d4d/f1f d8/d12/d36/d87/f10a 0 2024-04-22T23:23:38.517 INFO:tasks.workunit.client.1.smithi134.stdout:4/199: creat d4/dd/f41 x:0 0 0 2024-04-22T23:23:38.520 INFO:tasks.workunit.client.0.smithi067.stdout:0/982: truncate db/d21/d29/d75/d9d/dde/d10f/f7c 996716 0 2024-04-22T23:23:38.521 INFO:tasks.workunit.client.0.smithi067.stdout:8/883: unlink d1/d41/f6a 0 2024-04-22T23:23:38.522 INFO:tasks.workunit.client.0.smithi067.stdout:9/829: chown d5/dd/d18/d39/d4d/l70 4021 1 2024-04-22T23:23:38.523 INFO:tasks.workunit.client.1.smithi134.stdout:4/200: link d4/d7/f28 d4/f42 0 2024-04-22T23:23:38.525 INFO:tasks.workunit.client.0.smithi067.stdout:7/797: getdents d10/d14/d6d/d9e 0 2024-04-22T23:23:38.526 INFO:tasks.workunit.client.1.smithi134.stdout:4/201: readlink d4/dd/l9 0 2024-04-22T23:23:38.527 INFO:tasks.workunit.client.0.smithi067.stdout:9/830: mkdir d5/dd/d18/d3c/d40/d113 0 2024-04-22T23:23:38.527 INFO:tasks.workunit.client.0.smithi067.stdout:5/945: link d5/db/d12/d44/leb d5/d28/daf/db4/l146 0 2024-04-22T23:23:38.529 INFO:tasks.workunit.client.0.smithi067.stdout:5/946: chown d5/d28/daf/dbf/dcf/ca1 488 1 2024-04-22T23:23:38.530 INFO:tasks.workunit.client.0.smithi067.stdout:7/798: symlink d10/d14/d1d/d26/d2c/d41/db0/ddd/l129 0 2024-04-22T23:23:38.531 INFO:tasks.workunit.client.0.smithi067.stdout:9/831: read d5/dd/d18/d39/d49/d54/f79 [1107566,10708] 0 2024-04-22T23:23:38.531 INFO:tasks.workunit.client.1.smithi134.stdout:4/202: creat d4/d7/d32/d3a/f43 x:0 0 0 2024-04-22T23:23:38.532 INFO:tasks.workunit.client.0.smithi067.stdout:7/799: fsync d10/d14/d1d/f42 0 2024-04-22T23:23:38.535 INFO:tasks.workunit.client.1.smithi134.stdout:4/203: creat d4/d17/d25/f44 x:0 0 0 2024-04-22T23:23:38.536 INFO:tasks.workunit.client.0.smithi067.stdout:9/832: unlink d5/dd/d18/d39/d49/c4a 0 2024-04-22T23:23:38.538 INFO:tasks.workunit.client.1.smithi134.stdout:4/204: write d4/f2b [4472624,82299] 0 2024-04-22T23:23:38.539 INFO:tasks.workunit.client.1.smithi134.stdout:4/205: write d4/dd/f16 [3474705,52811] 0 2024-04-22T23:23:38.540 INFO:tasks.workunit.client.0.smithi067.stdout:0/983: symlink db/d21/d29/l144 0 2024-04-22T23:23:38.540 INFO:tasks.workunit.client.0.smithi067.stdout:7/800: rename d10/d14/d1d/d46/d124/cfc to d10/d14/d1d/d26/d2c/d7a/dc2/dd0/de4/c12a 0 2024-04-22T23:23:38.542 INFO:tasks.workunit.client.0.smithi067.stdout:1/818: dread d8/d9/fdc [0,4194304] 0 2024-04-22T23:23:38.543 INFO:tasks.workunit.client.1.smithi134.stdout:4/206: unlink d4/f20 0 2024-04-22T23:23:38.545 INFO:tasks.workunit.client.0.smithi067.stdout:9/833: dwrite d5/d21/d47/d56/f80 [0,4194304] 0 2024-04-22T23:23:38.545 INFO:tasks.workunit.client.1.smithi134.stdout:4/207: symlink d4/d7/d32/l45 0 2024-04-22T23:23:38.549 INFO:tasks.workunit.client.1.smithi134.stdout:4/208: dread d4/d7/f10 [0,4194304] 0 2024-04-22T23:23:38.550 INFO:tasks.workunit.client.0.smithi067.stdout:0/984: dwrite db/fb7 [0,4194304] 0 2024-04-22T23:23:38.551 INFO:tasks.workunit.client.1.smithi134.stdout:4/209: symlink d4/d7/l46 0 2024-04-22T23:23:38.551 INFO:tasks.workunit.client.0.smithi067.stdout:7/801: creat d10/d14/d1d/d46/f12b x:0 0 0 2024-04-22T23:23:38.551 INFO:tasks.workunit.client.1.smithi134.stdout:4/210: fdatasync d4/d17/f22 0 2024-04-22T23:23:38.554 INFO:tasks.workunit.client.0.smithi067.stdout:9/834: unlink d5/d21/d2c/l42 0 2024-04-22T23:23:38.554 INFO:tasks.workunit.client.0.smithi067.stdout:0/985: symlink db/d21/d29/d75/d9d/dde/l145 0 2024-04-22T23:23:38.556 INFO:tasks.workunit.client.0.smithi067.stdout:0/986: readlink db/d9a/l87 0 2024-04-22T23:23:38.558 INFO:tasks.workunit.client.1.smithi134.stdout:4/211: rename d4/d13/l1e to d4/l47 0 2024-04-22T23:23:38.558 INFO:tasks.workunit.client.0.smithi067.stdout:0/987: stat db/d9a/d56/d6b/db9/fe9 0 2024-04-22T23:23:38.561 INFO:tasks.workunit.client.0.smithi067.stdout:0/988: fsync db/d21/f4c 0 2024-04-22T23:23:38.562 INFO:tasks.workunit.client.0.smithi067.stdout:8/884: truncate d1/dd/d80/db9/d110/f22 406151 0 2024-04-22T23:23:38.564 INFO:tasks.workunit.client.0.smithi067.stdout:8/885: dread - d1/d3/d5c/d98/fd8 zero size 2024-04-22T23:23:38.566 INFO:tasks.workunit.client.1.smithi134.stdout:4/212: dwrite d4/d29/f2c [0,4194304] 0 2024-04-22T23:23:38.567 INFO:tasks.workunit.client.0.smithi067.stdout:3/877: dwrite d3/d4/d28/d40/dfd/dfe/f27 [0,4194304] 0 2024-04-22T23:23:38.568 INFO:tasks.workunit.client.0.smithi067.stdout:8/886: chown d1/d3/ld9 385738511 1 2024-04-22T23:23:38.568 INFO:tasks.workunit.client.0.smithi067.stdout:7/802: write d10/d14/d1d/d46/d6a/d91/fa7 [433353,73274] 0 2024-04-22T23:23:38.574 INFO:tasks.workunit.client.1.smithi134.stdout:4/213: dread d4/d15/f34 [0,4194304] 0 2024-04-22T23:23:38.574 INFO:tasks.workunit.client.0.smithi067.stdout:8/887: fdatasync d1/d3/d27/f58 0 2024-04-22T23:23:38.576 INFO:tasks.workunit.client.0.smithi067.stdout:7/803: mkdir d10/d14/d1d/d12c 0 2024-04-22T23:23:38.577 INFO:tasks.workunit.client.0.smithi067.stdout:1/819: dwrite d8/d9/de0/d60/d62/f95 [0,4194304] 0 2024-04-22T23:23:38.578 INFO:tasks.workunit.client.0.smithi067.stdout:3/878: fsync d3/d4/d10/d38/d53/d55/d5c/ffa 0 2024-04-22T23:23:38.580 INFO:tasks.workunit.client.1.smithi134.stdout:4/214: rename d4/d17/d25/f44 to d4/d15/d2a/f48 0 2024-04-22T23:23:38.581 INFO:tasks.workunit.client.1.smithi134.stdout:4/215: mkdir d4/d49 0 2024-04-22T23:23:38.583 INFO:tasks.workunit.client.1.smithi134.stdout:4/216: creat d4/d17/f4a x:0 0 0 2024-04-22T23:23:38.583 INFO:tasks.workunit.client.1.smithi134.stdout:4/217: read d4/d29/f2c [3508293,353] 0 2024-04-22T23:23:38.584 INFO:tasks.workunit.client.0.smithi067.stdout:9/835: dread d5/d21/d47/d56/d8c/dd5/fc1 [0,4194304] 0 2024-04-22T23:23:38.584 INFO:tasks.workunit.client.1.smithi134.stdout:4/218: chown d4/dd/f16 285 1 2024-04-22T23:23:38.586 INFO:tasks.workunit.client.0.smithi067.stdout:3/879: truncate d3/d4/d10/d38/d53/ddc/f121 2055670 0 2024-04-22T23:23:38.587 INFO:tasks.workunit.client.0.smithi067.stdout:0/989: truncate db/d21/d29/f3f 1692262 0 2024-04-22T23:23:38.588 INFO:tasks.workunit.client.0.smithi067.stdout:1/820: link d8/d9/de0/d5d/dd4/f106 d8/d9/d2d/d4d/d48/f10b 0 2024-04-22T23:23:38.588 INFO:tasks.workunit.client.1.smithi134.stdout:4/219: creat d4/d17/d25/f4b x:0 0 0 2024-04-22T23:23:38.588 INFO:tasks.workunit.client.1.smithi134.stdout:4/220: chown d4/d17/d25 155 1 2024-04-22T23:23:38.590 INFO:tasks.workunit.client.0.smithi067.stdout:0/990: creat db/d9a/d56/d6b/db9/f146 x:0 0 0 2024-04-22T23:23:38.592 INFO:tasks.workunit.client.0.smithi067.stdout:9/836: mkdir d5/d21/d47/d56/dc6/d103/d114 0 2024-04-22T23:23:38.593 INFO:tasks.workunit.client.0.smithi067.stdout:9/837: chown d5/dd/d18/d39/d49/c88 47444 1 2024-04-22T23:23:38.594 INFO:tasks.workunit.client.0.smithi067.stdout:7/804: creat d10/d14/d1d/d24/f12d x:0 0 0 2024-04-22T23:23:38.595 INFO:tasks.workunit.client.0.smithi067.stdout:0/991: rename db/d9a/d56/d70/d8c/dc2/le0 to db/de/d11/d1a/l147 0 2024-04-22T23:23:38.596 INFO:tasks.workunit.client.0.smithi067.stdout:7/805: fdatasync d10/d14/d1d/d46/d50/f29 0 2024-04-22T23:23:38.600 INFO:tasks.workunit.client.0.smithi067.stdout:7/806: read d10/d14/d1d/d46/d50/f43 [866011,66116] 0 2024-04-22T23:23:38.601 INFO:tasks.workunit.client.0.smithi067.stdout:1/821: getdents d8/d9/d2f 0 2024-04-22T23:23:38.601 INFO:tasks.workunit.client.1.smithi134.stdout:4/221: write d4/f42 [634545,84657] 0 2024-04-22T23:23:38.605 INFO:tasks.workunit.client.1.smithi134.stdout:4/222: unlink d4/d29/f2c 0 2024-04-22T23:23:38.607 INFO:tasks.workunit.client.1.smithi134.stdout:4/223: creat d4/d13/f4c x:0 0 0 2024-04-22T23:23:38.608 INFO:tasks.workunit.client.1.smithi134.stdout:4/224: creat d4/d29/f4d x:0 0 0 2024-04-22T23:23:38.608 INFO:tasks.workunit.client.1.smithi134.stdout:4/225: stat d4/d7/f3c 0 2024-04-22T23:23:38.609 INFO:tasks.workunit.client.0.smithi067.stdout:8/888: dwrite d1/d3/d5c/faa [0,4194304] 0 2024-04-22T23:23:38.609 INFO:tasks.workunit.client.0.smithi067.stdout:9/838: dwrite d5/dd/d18/d39/d49/f65 [0,4194304] 0 2024-04-22T23:23:38.610 INFO:tasks.workunit.client.1.smithi134.stdout:4/226: creat d4/d17/f4e x:0 0 0 2024-04-22T23:23:38.611 INFO:tasks.workunit.client.1.smithi134.stdout:4/227: mknod d4/d13/c4f 0 2024-04-22T23:23:38.614 INFO:tasks.workunit.client.0.smithi067.stdout:1/822: mknod d8/c10c 0 2024-04-22T23:23:38.614 INFO:tasks.workunit.client.1.smithi134.stdout:4/228: unlink d4/d7/f3c 0 2024-04-22T23:23:38.616 INFO:tasks.workunit.client.1.smithi134.stdout:4/229: rmdir d4/d17/d25 39 2024-04-22T23:23:38.616 INFO:tasks.workunit.client.0.smithi067.stdout:8/889: fsync d1/dd/d80/db9/d110/d62/d94/fc6 0 2024-04-22T23:23:38.620 INFO:tasks.workunit.client.0.smithi067.stdout:3/880: dwrite d3/d4/d10/d38/f48 [0,4194304] 0 2024-04-22T23:23:38.621 INFO:tasks.workunit.client.1.smithi134.stdout:4/230: symlink d4/d15/l50 0 2024-04-22T23:23:38.621 INFO:tasks.workunit.client.0.smithi067.stdout:1/823: mkdir d8/d9/de0/d5d/d10d 0 2024-04-22T23:23:38.621 INFO:tasks.workunit.client.1.smithi134.stdout:4/231: truncate d4/f42 1313356 0 2024-04-22T23:23:38.622 INFO:tasks.workunit.client.1.smithi134.stdout:4/232: write d4/d13/f4c [137841,100509] 0 2024-04-22T23:23:38.623 INFO:tasks.workunit.client.0.smithi067.stdout:0/992: dread db/d9a/d56/d70/dba/f102 [0,4194304] 0 2024-04-22T23:23:38.626 INFO:tasks.workunit.client.0.smithi067.stdout:9/839: write d5/dd/d18/d5c/fb0 [491006,91795] 0 2024-04-22T23:23:38.628 INFO:tasks.workunit.client.0.smithi067.stdout:7/807: dwrite d10/d14/d1d/d26/d2c/ff2 [0,4194304] 0 2024-04-22T23:23:38.631 INFO:tasks.workunit.client.1.smithi134.stdout:4/233: dwrite d4/d7/f10 [4194304,4194304] 0 2024-04-22T23:23:38.632 INFO:tasks.workunit.client.1.smithi134.stdout:4/234: fsync d4/d17/f1f 0 2024-04-22T23:23:38.633 INFO:tasks.workunit.client.0.smithi067.stdout:7/808: creat d10/d47/dab/f12e x:0 0 0 2024-04-22T23:23:38.636 INFO:tasks.workunit.client.0.smithi067.stdout:7/809: rmdir d10/d14/de0 39 2024-04-22T23:23:38.638 INFO:tasks.workunit.client.1.smithi134.stdout:4/235: creat d4/d7/d32/d3a/f51 x:0 0 0 2024-04-22T23:23:38.640 INFO:tasks.workunit.client.1.smithi134.stdout:4/236: symlink d4/d7/d32/l52 0 2024-04-22T23:23:38.641 INFO:tasks.workunit.client.0.smithi067.stdout:1/824: write d8/d9/d2d/d4d/f1f [887988,129681] 0 2024-04-22T23:23:38.648 INFO:tasks.workunit.client.1.smithi134.stdout:4/237: dwrite d4/dd/f41 [0,4194304] 0 2024-04-22T23:23:38.649 INFO:tasks.workunit.client.0.smithi067.stdout:3/881: truncate d3/d4/d10/d38/f48 2600317 0 2024-04-22T23:23:38.649 INFO:tasks.workunit.client.1.smithi134.stdout:4/238: readlink d4/d15/l50 0 2024-04-22T23:23:38.649 INFO:tasks.workunit.client.0.smithi067.stdout:1/825: dwrite d8/d9/fdc [0,4194304] 0 2024-04-22T23:23:38.650 INFO:tasks.workunit.client.1.smithi134.stdout:4/239: stat d4/d15/d2a/f48 0 2024-04-22T23:23:38.655 INFO:tasks.workunit.client.1.smithi134.stdout:4/240: mkdir d4/d35/d53 0 2024-04-22T23:23:38.657 INFO:tasks.workunit.client.0.smithi067.stdout:3/882: read d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d90/da2/daf/dd8/ffc [3624958,32641] 0 2024-04-22T23:23:38.658 INFO:tasks.workunit.client.0.smithi067.stdout:7/810: dwrite d10/d18/fb2 [0,4194304] 0 2024-04-22T23:23:38.662 INFO:tasks.workunit.client.0.smithi067.stdout:1/826: dwrite d8/d12/d36/f4e [0,4194304] 0 2024-04-22T23:23:38.665 INFO:tasks.workunit.client.0.smithi067.stdout:7/811: symlink d10/d14/d1d/d26/d80/l12f 0 2024-04-22T23:23:38.666 INFO:tasks.workunit.client.1.smithi134.stdout:4/241: dwrite d4/d17/d25/f4b [0,4194304] 0 2024-04-22T23:23:38.669 INFO:tasks.workunit.client.1.smithi134.stdout:4/242: creat d4/d15/f54 x:0 0 0 2024-04-22T23:23:38.670 INFO:tasks.workunit.client.0.smithi067.stdout:1/827: getdents d8/d9/d2f/d59 0 2024-04-22T23:23:38.671 INFO:tasks.workunit.client.1.smithi134.stdout:4/243: truncate d4/d7/d32/d3a/f43 771374 0 2024-04-22T23:23:38.672 INFO:tasks.workunit.client.0.smithi067.stdout:8/890: dread d1/dd/d80/d6f/fad [0,4194304] 0 2024-04-22T23:23:38.672 INFO:tasks.workunit.client.0.smithi067.stdout:7/812: truncate d10/d14/d1d/d26/d2c/f61 631031 0 2024-04-22T23:23:38.672 INFO:tasks.workunit.client.0.smithi067.stdout:1/828: read - d8/d9/d2d/d4d/f23 zero size 2024-04-22T23:23:38.673 INFO:tasks.workunit.client.0.smithi067.stdout:1/829: stat d8/d9/f1b 0 2024-04-22T23:23:38.674 INFO:tasks.workunit.client.1.smithi134.stdout:4/244: creat d4/d15/d2a/f55 x:0 0 0 2024-04-22T23:23:38.676 INFO:tasks.workunit.client.0.smithi067.stdout:8/891: creat d1/dd/d80/d6f/dfc/de8/f120 x:0 0 0 2024-04-22T23:23:38.677 INFO:tasks.workunit.client.0.smithi067.stdout:7/813: dread d10/d14/d1d/d26/d80/fbb [0,4194304] 0 2024-04-22T23:23:38.684 INFO:tasks.workunit.client.1.smithi134.stdout:4/245: dwrite d4/d15/d2a/f55 [0,4194304] 0 2024-04-22T23:23:38.687 INFO:tasks.workunit.client.0.smithi067.stdout:3/883: write d3/d4/d10/d38/d53/d55/d5c/fdd [207574,55470] 0 2024-04-22T23:23:38.689 INFO:tasks.workunit.client.1.smithi134.stdout:4/246: truncate d4/d17/f1f 1348017 0 2024-04-22T23:23:38.690 INFO:tasks.workunit.client.0.smithi067.stdout:3/884: truncate d3/f35 927708 0 2024-04-22T23:23:38.691 INFO:tasks.workunit.client.1.smithi134.stdout:4/247: rmdir d4/dd 39 2024-04-22T23:23:38.692 INFO:tasks.workunit.client.1.smithi134.stdout:4/248: readlink d4/d15/l33 0 2024-04-22T23:23:38.693 INFO:tasks.workunit.client.1.smithi134.stdout:4/249: chown d4/d15/f26 3089155 1 2024-04-22T23:23:38.693 INFO:tasks.workunit.client.0.smithi067.stdout:3/885: symlink d3/d4/d10/d38/d53/l134 0 2024-04-22T23:23:38.695 INFO:tasks.workunit.client.1.smithi134.stdout:4/250: rename d4/d15/f34 to d4/d35/f56 0 2024-04-22T23:23:38.696 INFO:tasks.workunit.client.1.smithi134.stdout:4/251: read d4/f2f [4535746,13116] 0 2024-04-22T23:23:38.697 INFO:tasks.workunit.client.1.smithi134.stdout:4/252: dread - d4/d29/f4d zero size 2024-04-22T23:23:38.698 INFO:tasks.workunit.client.0.smithi067.stdout:3/886: dwrite d3/d4/d10/d38/d53/d55/d5c/d70/f11f [0,4194304] 0 2024-04-22T23:23:38.699 INFO:tasks.workunit.client.1.smithi134.stdout:4/253: creat d4/d35/f57 x:0 0 0 2024-04-22T23:23:38.701 INFO:tasks.workunit.client.0.smithi067.stdout:3/887: read - d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d90/da2/daf/dd8/f104 zero size 2024-04-22T23:23:38.702 INFO:tasks.workunit.client.1.smithi134.stdout:4/254: rmdir d4/d49 0 2024-04-22T23:23:38.702 INFO:tasks.workunit.client.1.smithi134.stdout:4/255: fsync d4/d13/f4c 0 2024-04-22T23:23:38.704 INFO:tasks.workunit.client.0.smithi067.stdout:3/888: mkdir d3/d4/d1b/d93/d135 0 2024-04-22T23:23:38.708 INFO:tasks.workunit.client.0.smithi067.stdout:8/892: write d1/d3/d27/f58 [2220959,71597] 0 2024-04-22T23:23:38.710 INFO:tasks.workunit.client.1.smithi134.stdout:4/256: dwrite d4/dd/f41 [0,4194304] 0 2024-04-22T23:23:38.711 INFO:tasks.workunit.client.0.smithi067.stdout:3/889: truncate d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d90/da2/daf/dd8/ffc 2628651 0 2024-04-22T23:23:38.714 INFO:tasks.workunit.client.1.smithi134.stdout:4/257: mknod d4/d29/c58 0 2024-04-22T23:23:38.719 INFO:tasks.workunit.client.1.smithi134.stdout:4/258: dread d4/dd/f41 [0,4194304] 0 2024-04-22T23:23:38.719 INFO:tasks.workunit.client.0.smithi067.stdout:8/893: rename d1/d41/d46/c71 to d1/dd/d80/db9/de3/db8/c121 0 2024-04-22T23:23:38.719 INFO:tasks.workunit.client.0.smithi067.stdout:1/830: dwrite d8/d9/d2d/d4d/d9c/f6c [0,4194304] 0 2024-04-22T23:23:38.719 INFO:tasks.workunit.client.0.smithi067.stdout:3/890: chown d3/d4/d10/d38/d53/d55/d74/dd5/ddb/l72 8380 1 2024-04-22T23:23:38.720 INFO:tasks.workunit.client.0.smithi067.stdout:7/814: dwrite d10/d14/d1d/d46/d50/f43 [0,4194304] 0 2024-04-22T23:23:38.722 INFO:tasks.workunit.client.1.smithi134.stdout:4/259: link d4/d15/d2a/f55 d4/d13/f59 0 2024-04-22T23:23:38.724 INFO:tasks.workunit.client.1.smithi134.stdout:4/260: truncate d4/d7/d32/d3a/f51 801031 0 2024-04-22T23:23:38.727 INFO:tasks.workunit.client.1.smithi134.stdout:4/261: symlink d4/dd/l5a 0 2024-04-22T23:23:38.730 INFO:tasks.workunit.client.1.smithi134.stdout:4/262: rmdir d4/d7/d32/d3a 39 2024-04-22T23:23:38.731 INFO:tasks.workunit.client.0.smithi067.stdout:7/815: mknod d10/d14/d1d/d24/c130 0 2024-04-22T23:23:38.732 INFO:tasks.workunit.client.0.smithi067.stdout:3/891: truncate d3/d4/d28/d40/dfd/dfe/f78 3178491 0 2024-04-22T23:23:38.733 INFO:tasks.workunit.client.1.smithi134.stdout:4/263: dread d4/d7/d32/d3a/f51 [0,4194304] 0 2024-04-22T23:23:38.734 INFO:tasks.workunit.client.1.smithi134.stdout:4/264: truncate d4/d7/f28 2139798 0 2024-04-22T23:23:38.736 INFO:tasks.workunit.client.1.smithi134.stdout:4/265: symlink d4/d35/d53/l5b 0 2024-04-22T23:23:38.738 INFO:tasks.workunit.client.0.smithi067.stdout:1/831: write d8/d9/d2d/d4d/d83/dc5/fcf [716421,48487] 0 2024-04-22T23:23:38.740 INFO:tasks.workunit.client.0.smithi067.stdout:7/816: getdents d10/d14/de0 0 2024-04-22T23:23:38.742 INFO:tasks.workunit.client.0.smithi067.stdout:7/817: stat d10/d18/d88/daf/d100/d62/fe8 0 2024-04-22T23:23:38.742 INFO:tasks.workunit.client.1.smithi134.stdout:4/266: dwrite d4/d7/d32/d3a/f43 [0,4194304] 0 2024-04-22T23:23:38.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:38 smithi134 ceph-mon[34609]: pgmap v274: 65 pgs: 65 active+clean; 1.9 GiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 64 MiB/s rd, 187 MiB/s wr, 394 op/s 2024-04-22T23:23:38.744 INFO:tasks.workunit.client.0.smithi067.stdout:1/832: dwrite d8/d12/f104 [0,4194304] 0 2024-04-22T23:23:38.745 INFO:tasks.workunit.client.0.smithi067.stdout:7/818: truncate d10/f12 3752220 0 2024-04-22T23:23:38.746 INFO:tasks.workunit.client.1.smithi134.stdout:1/122: sync 2024-04-22T23:23:38.746 INFO:tasks.workunit.client.1.smithi134.stdout:3/159: sync 2024-04-22T23:23:38.746 INFO:tasks.workunit.client.1.smithi134.stdout:9/83: sync 2024-04-22T23:23:38.747 INFO:tasks.workunit.client.1.smithi134.stdout:4/267: write d4/d15/d2a/f48 [638385,39154] 0 2024-04-22T23:23:38.748 INFO:tasks.workunit.client.0.smithi067.stdout:3/892: dread d3/d4/d10/d38/d53/ddc/f121 [0,4194304] 0 2024-04-22T23:23:38.749 INFO:tasks.workunit.client.1.smithi134.stdout:4/268: chown d4/d35/f57 5 1 2024-04-22T23:23:38.750 INFO:tasks.workunit.client.1.smithi134.stdout:1/123: symlink db/d16/l31 0 2024-04-22T23:23:38.755 INFO:tasks.workunit.client.1.smithi134.stdout:3/160: creat dc/f40 x:0 0 0 2024-04-22T23:23:38.757 INFO:tasks.workunit.client.0.smithi067.stdout:1/833: link d8/d9/d2d/d4d/d83/dc5/fcd d8/d9/d2d/d4d/d83/f10e 0 2024-04-22T23:23:38.757 INFO:tasks.workunit.client.1.smithi134.stdout:9/84: rmdir d0/d7/dc/d16 0 2024-04-22T23:23:38.757 INFO:tasks.workunit.client.1.smithi134.stdout:1/124: dread db/f1d [0,4194304] 0 2024-04-22T23:23:38.757 INFO:tasks.workunit.client.1.smithi134.stdout:9/85: readlink d0/d7/d12/l14 0 2024-04-22T23:23:38.758 INFO:tasks.workunit.client.1.smithi134.stdout:4/269: creat d4/f5c x:0 0 0 2024-04-22T23:23:38.759 INFO:tasks.workunit.client.0.smithi067.stdout:7/819: write d10/d14/d1d/d26/d2c/d7a/f102 [1249911,458] 0 2024-04-22T23:23:38.759 INFO:tasks.workunit.client.1.smithi134.stdout:3/161: mknod dc/dd/d28/d34/c41 0 2024-04-22T23:23:38.760 INFO:tasks.workunit.client.1.smithi134.stdout:1/125: mknod db/d16/c32 0 2024-04-22T23:23:38.761 INFO:tasks.workunit.client.1.smithi134.stdout:9/86: dread d0/d7/d12/f18 [0,4194304] 0 2024-04-22T23:23:38.762 INFO:tasks.workunit.client.1.smithi134.stdout:4/270: mkdir d4/d15/d2a/d5d 0 2024-04-22T23:23:38.764 INFO:tasks.workunit.client.1.smithi134.stdout:5/189: sync 2024-04-22T23:23:38.764 INFO:tasks.workunit.client.1.smithi134.stdout:7/147: sync 2024-04-22T23:23:38.764 INFO:tasks.workunit.client.1.smithi134.stdout:1/126: mknod db/d15/c33 0 2024-04-22T23:23:38.765 INFO:tasks.workunit.client.1.smithi134.stdout:9/87: truncate d0/d7/d12/f1a 686799 0 2024-04-22T23:23:38.766 INFO:tasks.workunit.client.1.smithi134.stdout:7/148: readlink d4/d10/l39 0 2024-04-22T23:23:38.769 INFO:tasks.workunit.client.1.smithi134.stdout:5/190: mkdir d9/dd/d17/d2a/d3f 0 2024-04-22T23:23:38.770 INFO:tasks.workunit.client.1.smithi134.stdout:9/88: unlink d0/d7/dc/ff 0 2024-04-22T23:23:38.773 INFO:tasks.workunit.client.1.smithi134.stdout:5/191: fsync d9/d1e/f32 0 2024-04-22T23:23:38.773 INFO:tasks.workunit.client.0.smithi067.stdout:1/834: dwrite d8/d9/f43 [0,4194304] 0 2024-04-22T23:23:38.773 INFO:tasks.workunit.client.0.smithi067.stdout:7/820: dwrite d10/d47/f68 [0,4194304] 0 2024-04-22T23:23:38.774 INFO:tasks.workunit.client.0.smithi067.stdout:3/893: dread d3/d4/d10/d8a/fa6 [0,4194304] 0 2024-04-22T23:23:38.775 INFO:tasks.workunit.client.1.smithi134.stdout:3/162: dwrite dc/f18 [0,4194304] 0 2024-04-22T23:23:38.778 INFO:tasks.workunit.client.1.smithi134.stdout:9/89: readlink d0/d7/l13 0 2024-04-22T23:23:38.778 INFO:tasks.workunit.client.1.smithi134.stdout:4/271: dwrite d4/d35/f57 [0,4194304] 0 2024-04-22T23:23:38.780 INFO:tasks.workunit.client.1.smithi134.stdout:3/163: readlink dc/dd/d28/l26 0 2024-04-22T23:23:38.781 INFO:tasks.workunit.client.0.smithi067.stdout:1/835: creat d8/d9/d2f/d33/f10f x:0 0 0 2024-04-22T23:23:38.781 INFO:tasks.workunit.client.1.smithi134.stdout:5/192: chown d9/dd/d17/d2a/f30 59289777 1 2024-04-22T23:23:38.784 INFO:tasks.workunit.client.0.smithi067.stdout:1/836: write d8/d12/d36/fba [1919176,94141] 0 2024-04-22T23:23:38.785 INFO:tasks.workunit.client.1.smithi134.stdout:1/127: dwrite db/d15/d17/f2c [0,4194304] 0 2024-04-22T23:23:38.786 INFO:tasks.workunit.client.0.smithi067.stdout:3/894: dwrite d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d90/da2/daf/dd8/f104 [0,4194304] 0 2024-04-22T23:23:38.788 INFO:tasks.workunit.client.0.smithi067.stdout:7/821: dread fe [0,4194304] 0 2024-04-22T23:23:38.790 INFO:tasks.workunit.client.1.smithi134.stdout:3/164: mkdir dc/dd/d28/d2f/d21/d42 0 2024-04-22T23:23:38.792 INFO:tasks.workunit.client.0.smithi067.stdout:7/822: fdatasync d10/d18/d88/daf/d100/f5b 0 2024-04-22T23:23:38.794 INFO:tasks.workunit.client.0.smithi067.stdout:7/823: creat d10/d14/d1d/d26/d2c/d41/db0/f131 x:0 0 0 2024-04-22T23:23:38.799 INFO:tasks.workunit.client.1.smithi134.stdout:4/272: dwrite d4/f2b [0,4194304] 0 2024-04-22T23:23:38.805 INFO:tasks.workunit.client.1.smithi134.stdout:9/90: mkdir d0/d10/d1b 0 2024-04-22T23:23:38.805 INFO:tasks.workunit.client.1.smithi134.stdout:1/128: rmdir db/d15 39 2024-04-22T23:23:38.809 INFO:tasks.workunit.client.0.smithi067.stdout:6/870: sync 2024-04-22T23:23:38.811 INFO:tasks.workunit.client.1.smithi134.stdout:9/91: truncate d0/d7/d12/f1a 1220149 0 2024-04-22T23:23:38.811 INFO:tasks.workunit.client.1.smithi134.stdout:3/165: dread dc/f18 [0,4194304] 0 2024-04-22T23:23:38.813 INFO:tasks.workunit.client.1.smithi134.stdout:4/273: rename d4/d17 to d4/d35/d5e 0 2024-04-22T23:23:38.814 INFO:tasks.workunit.client.1.smithi134.stdout:7/149: dwrite d4/fa [0,4194304] 0 2024-04-22T23:23:38.815 INFO:tasks.workunit.client.0.smithi067.stdout:6/871: rename d17/d39/d3a/da9/fbe to d17/d83/d9b/ddd/f122 0 2024-04-22T23:23:38.816 INFO:tasks.workunit.client.1.smithi134.stdout:7/150: chown d4/d10/d2f/d34 531 1 2024-04-22T23:23:38.817 INFO:tasks.workunit.client.1.smithi134.stdout:4/274: symlink d4/dd/l5f 0 2024-04-22T23:23:38.817 INFO:tasks.workunit.client.1.smithi134.stdout:7/151: write d4/d10/f1c [1067787,119700] 0 2024-04-22T23:23:38.818 INFO:tasks.workunit.client.1.smithi134.stdout:9/92: mkdir d0/d1c 0 2024-04-22T23:23:38.819 INFO:tasks.workunit.client.1.smithi134.stdout:3/166: symlink dc/dd/l43 0 2024-04-22T23:23:38.820 INFO:tasks.workunit.client.1.smithi134.stdout:3/167: chown dc/d2a/d30 774569 1 2024-04-22T23:23:38.821 INFO:tasks.workunit.client.0.smithi067.stdout:6/872: link d17/d39/d3a/l5b d17/d39/d3a/da9/l123 0 2024-04-22T23:23:38.821 INFO:tasks.workunit.client.1.smithi134.stdout:7/152: creat d4/d8/d1d/f3c x:0 0 0 2024-04-22T23:23:38.822 INFO:tasks.workunit.client.1.smithi134.stdout:9/93: truncate d0/d7/f17 3775896 0 2024-04-22T23:23:38.823 INFO:tasks.workunit.client.1.smithi134.stdout:3/168: symlink dc/dd/d28/d3b/d29/l44 0 2024-04-22T23:23:38.827 INFO:tasks.workunit.client.1.smithi134.stdout:4/275: rename d4/d15/d2a/f55 to d4/d35/d5e/f60 0 2024-04-22T23:23:38.832 INFO:tasks.workunit.client.1.smithi134.stdout:7/153: mknod d4/c3d 0 2024-04-22T23:23:38.834 INFO:tasks.workunit.client.0.smithi067.stdout:1/837: write d8/d9/d2d/d4d/f23 [950892,102734] 0 2024-04-22T23:23:38.835 INFO:tasks.workunit.client.1.smithi134.stdout:9/94: dwrite d0/d7/fd [0,4194304] 0 2024-04-22T23:23:38.838 INFO:tasks.workunit.client.1.smithi134.stdout:4/276: link d4/d15/f1c d4/d15/f61 0 2024-04-22T23:23:38.839 INFO:tasks.workunit.client.1.smithi134.stdout:3/169: dwrite dc/dd/d28/d3b/f31 [0,4194304] 0 2024-04-22T23:23:38.841 INFO:tasks.workunit.client.1.smithi134.stdout:9/95: unlink d0/d10/d11/c15 0 2024-04-22T23:23:38.845 INFO:tasks.workunit.client.0.smithi067.stdout:7/824: write d10/f15 [1040150,129442] 0 2024-04-22T23:23:38.849 INFO:tasks.workunit.client.0.smithi067.stdout:7/825: chown d10/d14/d1d/d26/d80/dd2/dd4/lfa 1788074510 1 2024-04-22T23:23:38.850 INFO:tasks.workunit.client.1.smithi134.stdout:1/129: dread db/f28 [0,4194304] 0 2024-04-22T23:23:38.853 INFO:tasks.workunit.client.1.smithi134.stdout:4/277: dwrite d4/d13/f4c [0,4194304] 0 2024-04-22T23:23:38.853 INFO:tasks.workunit.client.1.smithi134.stdout:3/170: creat dc/f45 x:0 0 0 2024-04-22T23:23:38.854 INFO:tasks.workunit.client.1.smithi134.stdout:3/171: chown dc/dd/d28/d2f 1 1 2024-04-22T23:23:38.855 INFO:tasks.workunit.client.0.smithi067.stdout:6/873: dwrite d17/d39/f3f [0,4194304] 0 2024-04-22T23:23:38.855 INFO:tasks.workunit.client.0.smithi067.stdout:7/826: creat d10/d14/d1d/d26/f132 x:0 0 0 2024-04-22T23:23:38.855 INFO:tasks.workunit.client.1.smithi134.stdout:4/278: dread - d4/d35/d5e/f4a zero size 2024-04-22T23:23:38.856 INFO:tasks.workunit.client.1.smithi134.stdout:1/130: truncate db/d15/f29 337157 0 2024-04-22T23:23:38.858 INFO:tasks.workunit.client.1.smithi134.stdout:4/279: fsync d4/d35/d5e/f22 0 2024-04-22T23:23:38.860 INFO:tasks.workunit.client.0.smithi067.stdout:6/874: creat d17/d83/db5/dbc/dd8/d10c/f124 x:0 0 0 2024-04-22T23:23:38.863 INFO:tasks.workunit.client.0.smithi067.stdout:7/827: rename d10/d14/d6d/fca to d10/d14/d1d/d26/d2c/d41/db0/ddd/f133 0 2024-04-22T23:23:38.864 INFO:tasks.workunit.client.1.smithi134.stdout:3/172: rename dc/dd/l37 to dc/d2a/d30/l46 0 2024-04-22T23:23:38.864 INFO:tasks.workunit.client.1.smithi134.stdout:4/280: dread d4/d13/f4c [0,4194304] 0 2024-04-22T23:23:38.864 INFO:tasks.workunit.client.0.smithi067.stdout:1/838: dread d8/f4b [0,4194304] 0 2024-04-22T23:23:38.865 INFO:tasks.workunit.client.1.smithi134.stdout:3/173: write dc/f3f [3829721,28900] 0 2024-04-22T23:23:38.866 INFO:tasks.workunit.client.1.smithi134.stdout:4/281: write d4/d35/d5e/f4e [600902,126129] 0 2024-04-22T23:23:38.869 INFO:tasks.workunit.client.0.smithi067.stdout:6/875: rename d17/d39/f69 to d17/d83/f125 0 2024-04-22T23:23:38.869 INFO:tasks.workunit.client.0.smithi067.stdout:1/839: dread d8/d12/f104 [0,4194304] 0 2024-04-22T23:23:38.871 INFO:tasks.workunit.client.0.smithi067.stdout:1/840: chown d8/d9/d2f/d33/cc0 14242059 1 2024-04-22T23:23:38.872 INFO:tasks.workunit.client.1.smithi134.stdout:1/131: dread db/f28 [4194304,4194304] 0 2024-04-22T23:23:38.872 INFO:tasks.workunit.client.0.smithi067.stdout:6/876: write d17/d39/d3a/f4f [4133650,35325] 0 2024-04-22T23:23:38.872 INFO:tasks.workunit.client.1.smithi134.stdout:4/282: fdatasync d4/d15/f61 0 2024-04-22T23:23:38.874 INFO:tasks.workunit.client.0.smithi067.stdout:7/828: rename d10/d14/d1d/d26/c1b to d10/d18/d88/daf/d100/d62/c134 0 2024-04-22T23:23:38.875 INFO:tasks.workunit.client.0.smithi067.stdout:7/829: chown d10/d14/f98 189277859 1 2024-04-22T23:23:38.876 INFO:tasks.workunit.client.0.smithi067.stdout:7/830: write d10/f15 [2015338,93235] 0 2024-04-22T23:23:38.876 INFO:tasks.workunit.client.1.smithi134.stdout:1/132: creat db/d15/f34 x:0 0 0 2024-04-22T23:23:38.877 INFO:tasks.workunit.client.1.smithi134.stdout:1/133: chown db/d15/d17/d20 0 1 2024-04-22T23:23:38.878 INFO:tasks.workunit.client.1.smithi134.stdout:1/134: read db/f1d [337464,79907] 0 2024-04-22T23:23:38.878 INFO:tasks.workunit.client.1.smithi134.stdout:4/283: rmdir d4/d35/d53 39 2024-04-22T23:23:38.880 INFO:tasks.workunit.client.1.smithi134.stdout:3/174: getdents dc/d2a 0 2024-04-22T23:23:38.881 INFO:tasks.workunit.client.1.smithi134.stdout:7/154: dread d4/d10/d2f/f32 [0,4194304] 0 2024-04-22T23:23:38.882 INFO:tasks.workunit.client.1.smithi134.stdout:1/135: write db/f27 [8195433,42358] 0 2024-04-22T23:23:38.883 INFO:tasks.workunit.client.1.smithi134.stdout:1/136: fsync fa 0 2024-04-22T23:23:38.884 INFO:tasks.workunit.client.1.smithi134.stdout:7/155: readlink d4/d10/l39 0 2024-04-22T23:23:38.884 INFO:tasks.workunit.client.1.smithi134.stdout:4/284: chown d4/c24 386805999 1 2024-04-22T23:23:38.884 INFO:tasks.workunit.client.1.smithi134.stdout:1/137: dread - db/d15/f34 zero size 2024-04-22T23:23:38.885 INFO:tasks.workunit.client.0.smithi067.stdout:1/841: write d8/d9/de0/d60/d62/fa8 [904641,50112] 0 2024-04-22T23:23:38.885 INFO:tasks.workunit.client.1.smithi134.stdout:3/175: unlink dc/c3e 0 2024-04-22T23:23:38.886 INFO:tasks.workunit.client.1.smithi134.stdout:1/138: chown db/d15/c33 12159317 1 2024-04-22T23:23:38.886 INFO:tasks.workunit.client.0.smithi067.stdout:7/831: truncate d10/d14/d1d/d26/d80/dd2/dd4/ff5 1448026 0 2024-04-22T23:23:38.887 INFO:tasks.workunit.client.1.smithi134.stdout:3/176: read - dc/dd/d28/f39 zero size 2024-04-22T23:23:38.888 INFO:tasks.workunit.client.1.smithi134.stdout:1/139: rmdir db/d16 39 2024-04-22T23:23:38.889 INFO:tasks.workunit.client.1.smithi134.stdout:9/96: write d0/d7/f17 [1162938,128353] 0 2024-04-22T23:23:38.889 INFO:tasks.workunit.client.0.smithi067.stdout:7/832: fdatasync d10/d14/d1d/d26/d2c/d7a/fa5 0 2024-04-22T23:23:38.891 INFO:tasks.workunit.client.0.smithi067.stdout:6/877: dwrite d17/d39/d3a/f56 [0,4194304] 0 2024-04-22T23:23:38.891 INFO:tasks.workunit.client.1.smithi134.stdout:7/156: creat d4/d10/d2f/d37/f3e x:0 0 0 2024-04-22T23:23:38.893 INFO:tasks.workunit.client.0.smithi067.stdout:7/833: mknod d10/d14/d6d/d9e/c135 0 2024-04-22T23:23:38.893 INFO:tasks.workunit.client.0.smithi067.stdout:1/842: dread d8/d9/de0/d5d/f8b [0,4194304] 0 2024-04-22T23:23:38.894 INFO:tasks.workunit.client.1.smithi134.stdout:4/285: creat d4/d35/f62 x:0 0 0 2024-04-22T23:23:38.895 INFO:tasks.workunit.client.1.smithi134.stdout:1/140: creat db/d15/d17/d20/d2a/d2d/f35 x:0 0 0 2024-04-22T23:23:38.896 INFO:tasks.workunit.client.1.smithi134.stdout:9/97: creat d0/d7/dc/f1d x:0 0 0 2024-04-22T23:23:38.897 INFO:tasks.workunit.client.0.smithi067.stdout:1/843: write d8/d9/f25 [3103887,124500] 0 2024-04-22T23:23:38.897 INFO:tasks.workunit.client.1.smithi134.stdout:7/157: fsync d4/d10/d2f/f32 0 2024-04-22T23:23:38.899 INFO:tasks.workunit.client.1.smithi134.stdout:3/177: rename dc/f1f to dc/dd/d28/d2f/d21/d42/f47 0 2024-04-22T23:23:38.901 INFO:tasks.workunit.client.0.smithi067.stdout:7/834: creat d10/d14/d1d/d24/f136 x:0 0 0 2024-04-22T23:23:38.901 INFO:tasks.workunit.client.0.smithi067.stdout:1/844: mknod d8/d9/de0/d5d/d10d/c110 0 2024-04-22T23:23:38.901 INFO:tasks.workunit.client.0.smithi067.stdout:5/947: sync 2024-04-22T23:23:38.901 INFO:tasks.workunit.client.0.smithi067.stdout:2/832: sync 2024-04-22T23:23:38.902 INFO:tasks.workunit.client.1.smithi134.stdout:7/158: read d4/d10/f1f [2367542,103893] 0 2024-04-22T23:23:38.902 INFO:tasks.workunit.client.0.smithi067.stdout:7/835: chown d10/d14/d1d/d26/f93 967031339 1 2024-04-22T23:23:38.903 INFO:tasks.workunit.client.0.smithi067.stdout:6/878: getdents d17/d39/d53/d90/d46/d62 0 2024-04-22T23:23:38.904 INFO:tasks.workunit.client.0.smithi067.stdout:1/845: fsync d8/d9/d2d/d4d/d9c/ff9 0 2024-04-22T23:23:38.904 INFO:tasks.workunit.client.0.smithi067.stdout:2/833: dread - d1/d42/d5f/d9b/dc0/fec zero size 2024-04-22T23:23:38.904 INFO:tasks.workunit.client.1.smithi134.stdout:7/159: stat d4/f2d 0 2024-04-22T23:23:38.904 INFO:tasks.workunit.client.1.smithi134.stdout:3/178: creat dc/dd/d28/d34/f48 x:0 0 0 2024-04-22T23:23:38.905 INFO:tasks.workunit.client.1.smithi134.stdout:1/141: dwrite db/d15/d17/f1a [0,4194304] 0 2024-04-22T23:23:38.905 INFO:tasks.workunit.client.1.smithi134.stdout:9/98: mknod d0/c1e 0 2024-04-22T23:23:38.906 INFO:tasks.workunit.client.1.smithi134.stdout:9/99: stat d0/d10 0 2024-04-22T23:23:38.907 INFO:tasks.workunit.client.0.smithi067.stdout:5/948: mknod d5/d28/d36/c147 0 2024-04-22T23:23:38.907 INFO:tasks.workunit.client.0.smithi067.stdout:2/834: chown d1/d10/d41/l54 1557215 1 2024-04-22T23:23:38.908 INFO:tasks.workunit.client.0.smithi067.stdout:6/879: symlink d17/d83/db5/dbc/dd8/l126 0 2024-04-22T23:23:38.908 INFO:tasks.workunit.client.0.smithi067.stdout:5/949: readlink d5/d1a/d103/d4d/l59 0 2024-04-22T23:23:38.909 INFO:tasks.workunit.client.1.smithi134.stdout:9/100: read - d0/d7/dc/f1d zero size 2024-04-22T23:23:38.910 INFO:tasks.workunit.client.1.smithi134.stdout:7/160: readlink d4/lc 0 2024-04-22T23:23:38.910 INFO:tasks.workunit.client.0.smithi067.stdout:2/835: write d1/d19/d6f/f107 [680592,50320] 0 2024-04-22T23:23:38.910 INFO:tasks.workunit.client.0.smithi067.stdout:1/846: symlink d8/d9/d2f/d33/l111 0 2024-04-22T23:23:38.912 INFO:tasks.workunit.client.1.smithi134.stdout:3/179: rmdir dc/dd/d28/d2f/d21 39 2024-04-22T23:23:38.912 INFO:tasks.workunit.client.1.smithi134.stdout:7/161: readlink d4/d10/l12 0 2024-04-22T23:23:38.912 INFO:tasks.workunit.client.0.smithi067.stdout:6/880: fdatasync d17/f2b 0 2024-04-22T23:23:38.912 INFO:tasks.workunit.client.1.smithi134.stdout:4/286: dwrite d4/dd/f41 [0,4194304] 0 2024-04-22T23:23:38.913 INFO:tasks.workunit.client.0.smithi067.stdout:1/847: chown d8/d9/de0/d5d/cbe 431 1 2024-04-22T23:23:38.915 INFO:tasks.workunit.client.0.smithi067.stdout:2/836: unlink d1/d42/d5f/f98 0 2024-04-22T23:23:38.915 INFO:tasks.workunit.client.0.smithi067.stdout:2/837: readlink d1/la7 0 2024-04-22T23:23:38.916 INFO:tasks.workunit.client.1.smithi134.stdout:3/180: dread dc/f18 [0,4194304] 0 2024-04-22T23:23:38.918 INFO:tasks.workunit.client.0.smithi067.stdout:6/881: dread - d17/d39/d3a/da9/fba zero size 2024-04-22T23:23:38.918 INFO:tasks.workunit.client.1.smithi134.stdout:3/181: write dc/dd/d28/d3b/f31 [2395263,25194] 0 2024-04-22T23:23:38.919 INFO:tasks.workunit.client.1.smithi134.stdout:7/162: mknod d4/d8/d1d/c3f 0 2024-04-22T23:23:38.920 INFO:tasks.workunit.client.0.smithi067.stdout:1/848: creat d8/d9/d2d/f112 x:0 0 0 2024-04-22T23:23:38.920 INFO:tasks.workunit.client.0.smithi067.stdout:5/950: mkdir d5/d28/daf/d148 0 2024-04-22T23:23:38.920 INFO:tasks.workunit.client.1.smithi134.stdout:1/142: unlink db/d15/d17/d20/d2a/d2d/f35 0 2024-04-22T23:23:38.922 INFO:tasks.workunit.client.0.smithi067.stdout:2/838: mknod d1/d10/d72/c114 0 2024-04-22T23:23:38.922 INFO:tasks.workunit.client.0.smithi067.stdout:7/836: write d10/d14/d1d/d26/f40 [989412,110177] 0 2024-04-22T23:23:38.924 INFO:tasks.workunit.client.0.smithi067.stdout:1/849: rmdir d8/d9/d2d/d4d/d48 39 2024-04-22T23:23:38.924 INFO:tasks.workunit.client.1.smithi134.stdout:9/101: link d0/d7/f9 d0/d7/d12/f1f 0 2024-04-22T23:23:38.926 INFO:tasks.workunit.client.0.smithi067.stdout:6/882: rename d17/d39/d53/d90/d46/d62/d88/f92 to d17/d83/f127 0 2024-04-22T23:23:38.927 INFO:tasks.workunit.client.0.smithi067.stdout:5/951: symlink d5/d1a/d103/d10d/d121/l149 0 2024-04-22T23:23:38.929 INFO:tasks.workunit.client.0.smithi067.stdout:1/850: symlink d8/d9/de0/l113 0 2024-04-22T23:23:38.930 INFO:tasks.workunit.client.1.smithi134.stdout:7/163: symlink d4/d10/l40 0 2024-04-22T23:23:38.931 INFO:tasks.workunit.client.0.smithi067.stdout:2/839: rename d1/d12/f92 to d1/d10/d72/d68/d7b/d95/f115 0 2024-04-22T23:23:38.932 INFO:tasks.workunit.client.1.smithi134.stdout:7/164: write d4/d10/f1c [35917,43069] 0 2024-04-22T23:23:38.932 INFO:tasks.workunit.client.1.smithi134.stdout:8/169: sync 2024-04-22T23:23:38.932 INFO:tasks.workunit.client.1.smithi134.stdout:6/171: sync 2024-04-22T23:23:38.932 INFO:tasks.workunit.client.1.smithi134.stdout:0/174: sync 2024-04-22T23:23:38.933 INFO:tasks.workunit.client.1.smithi134.stdout:2/152: sync 2024-04-22T23:23:38.933 INFO:tasks.workunit.client.0.smithi067.stdout:6/883: read d17/f29 [1238406,92406] 0 2024-04-22T23:23:38.934 INFO:tasks.workunit.client.0.smithi067.stdout:1/851: dread - d8/d9/d2d/d4d/d70/fd5 zero size 2024-04-22T23:23:38.934 INFO:tasks.workunit.client.0.smithi067.stdout:5/952: symlink d5/d28/daf/l14a 0 2024-04-22T23:23:38.938 INFO:tasks.workunit.client.0.smithi067.stdout:7/837: rename d10/d14/d1d/d26/d2c/d7a/dc2/dd0/l11e to d10/d14/d1d/d26/d80/l137 0 2024-04-22T23:23:38.938 INFO:tasks.workunit.client.0.smithi067.stdout:5/953: mknod d5/d28/daf/dbf/dcf/c14b 0 2024-04-22T23:23:38.941 INFO:tasks.workunit.client.0.smithi067.stdout:6/884: dwrite d17/d39/f103 [0,4194304] 0 2024-04-22T23:23:38.941 INFO:tasks.workunit.client.0.smithi067.stdout:2/840: link d1/d10/d72/c32 d1/d10/d1c/d10f/c116 0 2024-04-22T23:23:38.948 INFO:tasks.workunit.client.0.smithi067.stdout:2/841: fsync d1/d42/d61/dad/fd3 0 2024-04-22T23:23:38.950 INFO:tasks.workunit.client.0.smithi067.stdout:2/842: chown d1/la7 26502741 1 2024-04-22T23:23:38.952 INFO:tasks.workunit.client.0.smithi067.stdout:5/954: getdents d5/db/d12/d44/d47 0 2024-04-22T23:23:38.953 INFO:tasks.workunit.client.0.smithi067.stdout:6/885: dwrite d17/d39/d53/f74 [4194304,4194304] 0 2024-04-22T23:23:38.953 INFO:tasks.workunit.client.0.smithi067.stdout:7/838: rmdir d10/d14/d1d/d26/d2c/d7a/dc2/d125 0 2024-04-22T23:23:38.954 INFO:tasks.workunit.client.0.smithi067.stdout:1/852: dread d8/d9/d2f/d33/d66/f9b [0,4194304] 0 2024-04-22T23:23:38.955 INFO:tasks.workunit.client.0.smithi067.stdout:5/955: write d5/d28/d36/d40/f10b [279444,111989] 0 2024-04-22T23:23:38.956 INFO:tasks.workunit.client.0.smithi067.stdout:2/843: dread - d1/d42/d61/daa/faf zero size 2024-04-22T23:23:38.960 INFO:tasks.workunit.client.0.smithi067.stdout:7/839: creat d10/d14/d1d/d26/d2c/d41/f138 x:0 0 0 2024-04-22T23:23:38.963 INFO:tasks.workunit.client.0.smithi067.stdout:6/886: rename d17/d39/d3a/f56 to d17/d83/d9b/ddd/f128 0 2024-04-22T23:23:38.965 INFO:tasks.workunit.client.0.smithi067.stdout:2/844: mkdir d1/d10/d72/d53/d9a/d10a/d117 0 2024-04-22T23:23:38.965 INFO:tasks.workunit.client.0.smithi067.stdout:5/956: dwrite d5/d28/daf/dbf/dcf/db2/f142 [0,4194304] 0 2024-04-22T23:23:38.966 INFO:tasks.workunit.client.0.smithi067.stdout:0/993: sync 2024-04-22T23:23:38.971 INFO:tasks.workunit.client.0.smithi067.stdout:7/840: creat d10/d14/d1d/d12c/f139 x:0 0 0 2024-04-22T23:23:38.973 INFO:tasks.workunit.client.0.smithi067.stdout:2/845: rename d1/d10/d72/d68/da1/fb5 to d1/d10/d72/d68/da1/f118 0 2024-04-22T23:23:38.978 INFO:tasks.workunit.client.0.smithi067.stdout:1/853: getdents d8/d9/d2f/d33/d5c 0 2024-04-22T23:23:38.979 INFO:tasks.workunit.client.1.smithi134.stdout:1/143: link db/d15/d17/f1a db/d15/d17/d20/d2a/f36 0 2024-04-22T23:23:38.979 INFO:tasks.workunit.client.0.smithi067.stdout:1/854: dread - d8/d9/d2f/d31/ded/f101 zero size 2024-04-22T23:23:38.980 INFO:tasks.workunit.client.0.smithi067.stdout:7/841: link d10/d14/d1d/d26/d80/fbb d10/d14/d1d/d26/d2c/d41/f13a 0 2024-04-22T23:23:38.980 INFO:tasks.workunit.client.1.smithi134.stdout:7/165: mkdir d4/d8/d41 0 2024-04-22T23:23:38.980 INFO:tasks.workunit.client.0.smithi067.stdout:1/855: stat d8/d9/d2d/d4d/lbd 0 2024-04-22T23:23:38.981 INFO:tasks.workunit.client.1.smithi134.stdout:9/102: dwrite d0/d7/f9 [0,4194304] 0 2024-04-22T23:23:38.981 INFO:tasks.workunit.client.1.smithi134.stdout:6/172: stat d0/d19/de/d2f/f32 0 2024-04-22T23:23:38.982 INFO:tasks.workunit.client.0.smithi067.stdout:2/846: link d1/d10/d33/d10c/ccb d1/d42/d61/c119 0 2024-04-22T23:23:38.984 INFO:tasks.workunit.client.0.smithi067.stdout:1/856: unlink d8/d9/d2f/d33/fb6 0 2024-04-22T23:23:38.985 INFO:tasks.workunit.client.1.smithi134.stdout:9/103: readlink d0/d7/d12/l14 0 2024-04-22T23:23:38.986 INFO:tasks.workunit.client.0.smithi067.stdout:2/847: mkdir d1/d10/d33/d46/d11a 0 2024-04-22T23:23:38.988 INFO:tasks.workunit.client.1.smithi134.stdout:7/166: rmdir d4/d8/d2a 39 2024-04-22T23:23:38.988 INFO:tasks.workunit.client.1.smithi134.stdout:0/175: creat d1/d12/d22/f33 x:0 0 0 2024-04-22T23:23:38.989 INFO:tasks.workunit.client.0.smithi067.stdout:5/957: write d5/db/d12/dbb/f72 [3685867,94667] 0 2024-04-22T23:23:38.990 INFO:tasks.workunit.client.0.smithi067.stdout:7/842: dwrite d10/d18/d88/fa2 [0,4194304] 0 2024-04-22T23:23:38.990 INFO:tasks.workunit.client.0.smithi067.stdout:6/887: write d17/f25 [932213,69901] 0 2024-04-22T23:23:38.991 INFO:tasks.workunit.client.0.smithi067.stdout:2/848: chown d1/d10/d33/c63 20482672 1 2024-04-22T23:23:38.993 INFO:tasks.workunit.client.0.smithi067.stdout:1/857: symlink d8/d9/d2f/d31/l114 0 2024-04-22T23:23:38.993 INFO:tasks.workunit.client.1.smithi134.stdout:9/104: symlink d0/d7/dc/l20 0 2024-04-22T23:23:38.995 INFO:tasks.workunit.client.0.smithi067.stdout:2/849: symlink d1/d10/d33/l11b 0 2024-04-22T23:23:38.997 INFO:tasks.workunit.client.0.smithi067.stdout:7/843: fdatasync d10/d14/d1d/d46/d6a/f86 0 2024-04-22T23:23:38.997 INFO:tasks.workunit.client.0.smithi067.stdout:0/994: dwrite db/d21/f5a [0,4194304] 0 2024-04-22T23:23:38.997 INFO:tasks.workunit.client.1.smithi134.stdout:7/167: dread - d4/d8/f21 zero size 2024-04-22T23:23:38.999 INFO:tasks.workunit.client.0.smithi067.stdout:5/958: creat d5/d106/f14c x:0 0 0 2024-04-22T23:23:38.999 INFO:tasks.workunit.client.1.smithi134.stdout:8/170: dwrite f6 [0,4194304] 0 2024-04-22T23:23:39.000 INFO:tasks.workunit.client.0.smithi067.stdout:0/995: write db/de/d2a/ffd [320847,91672] 0 2024-04-22T23:23:39.002 INFO:tasks.workunit.client.0.smithi067.stdout:2/850: creat d1/d10/d41/f11c x:0 0 0 2024-04-22T23:23:39.003 INFO:tasks.workunit.client.1.smithi134.stdout:6/173: rename d0/d19/de/d12/d2a/d2d to d0/d19/de/d12/d2a/d2d/d3e 22 2024-04-22T23:23:39.004 INFO:tasks.workunit.client.0.smithi067.stdout:7/844: rename d10/fed to d10/d14/d1d/d26/d7e/dc0/d10e/f13b 0 2024-04-22T23:23:39.006 INFO:tasks.workunit.client.1.smithi134.stdout:4/287: dwrite d4/d7/d32/d3a/f51 [0,4194304] 0 2024-04-22T23:23:39.006 INFO:tasks.workunit.client.0.smithi067.stdout:2/851: mkdir d1/d10/d72/d53/d9a/d11d 0 2024-04-22T23:23:39.009 INFO:tasks.workunit.client.0.smithi067.stdout:7/845: rmdir d10/d14/d1d/d26/d2c/df4 39 2024-04-22T23:23:39.010 INFO:tasks.workunit.client.0.smithi067.stdout:2/852: rmdir d1/d42/d61/daa 39 2024-04-22T23:23:39.011 INFO:tasks.workunit.client.0.smithi067.stdout:5/959: dwrite d5/d28/daf/dbf/dcf/fbc [0,4194304] 0 2024-04-22T23:23:39.012 INFO:tasks.workunit.client.0.smithi067.stdout:2/853: stat d1/d42/d5f/d9b/dc0/f103 0 2024-04-22T23:23:39.013 INFO:tasks.workunit.client.0.smithi067.stdout:5/960: read - d5/d106/f14c zero size 2024-04-22T23:23:39.014 INFO:tasks.workunit.client.0.smithi067.stdout:7/846: mknod d10/d14/de0/c13c 0 2024-04-22T23:23:39.014 INFO:tasks.workunit.client.0.smithi067.stdout:0/996: readlink db/de/d11/d1a/l147 0 2024-04-22T23:23:39.016 INFO:tasks.workunit.client.1.smithi134.stdout:7/168: truncate d4/d8/d1d/f2b 203132 0 2024-04-22T23:23:39.017 INFO:tasks.workunit.client.1.smithi134.stdout:4/288: creat d4/d29/f63 x:0 0 0 2024-04-22T23:23:39.017 INFO:tasks.workunit.client.0.smithi067.stdout:5/961: chown d5/db/d12/dbb/d61/d89/f92 129194 1 2024-04-22T23:23:39.018 INFO:tasks.workunit.client.1.smithi134.stdout:6/174: symlink d0/l3f 0 2024-04-22T23:23:39.018 INFO:tasks.workunit.client.1.smithi134.stdout:2/153: dwrite d2/d15/d24/d1d/f36 [0,4194304] 0 2024-04-22T23:23:39.019 INFO:tasks.workunit.client.0.smithi067.stdout:7/847: symlink d10/d14/d1d/d69/l13d 0 2024-04-22T23:23:39.021 INFO:tasks.workunit.client.1.smithi134.stdout:0/176: dwrite d1/f9 [0,4194304] 0 2024-04-22T23:23:39.022 INFO:tasks.workunit.client.0.smithi067.stdout:5/962: mkdir d5/d28/daf/db4/d14d 0 2024-04-22T23:23:39.023 INFO:tasks.workunit.client.0.smithi067.stdout:9/840: sync 2024-04-22T23:23:39.023 INFO:tasks.workunit.client.0.smithi067.stdout:8/894: sync 2024-04-22T23:23:39.023 INFO:tasks.workunit.client.0.smithi067.stdout:3/895: sync 2024-04-22T23:23:39.024 INFO:tasks.workunit.client.0.smithi067.stdout:7/848: fdatasync d10/d14/d1d/d26/f93 0 2024-04-22T23:23:39.026 INFO:tasks.workunit.client.1.smithi134.stdout:1/144: dwrite db/f28 [4194304,4194304] 0 2024-04-22T23:23:39.026 INFO:tasks.workunit.client.0.smithi067.stdout:3/896: chown d3/d4/d1b 0 1 2024-04-22T23:23:39.028 INFO:tasks.workunit.client.0.smithi067.stdout:2/854: dwrite d1/d42/f56 [0,4194304] 0 2024-04-22T23:23:39.031 INFO:tasks.workunit.client.0.smithi067.stdout:7/849: chown d10/c23 35 1 2024-04-22T23:23:39.032 INFO:tasks.workunit.client.0.smithi067.stdout:5/963: fsync d5/d28/daf/dbf/dcf/db2/fe6 0 2024-04-22T23:23:39.032 INFO:tasks.workunit.client.1.smithi134.stdout:1/145: write db/d15/d17/f1a [3203213,83981] 0 2024-04-22T23:23:39.034 INFO:tasks.workunit.client.1.smithi134.stdout:1/146: fdatasync db/d15/f29 0 2024-04-22T23:23:39.036 INFO:tasks.workunit.client.1.smithi134.stdout:2/154: link d2/f2c d2/dd/f37 0 2024-04-22T23:23:39.038 INFO:tasks.workunit.client.1.smithi134.stdout:7/169: mkdir d4/d8/d2a/d42 0 2024-04-22T23:23:39.039 INFO:tasks.workunit.client.0.smithi067.stdout:2/855: unlink d1/d10/d72/d53/dba/ld1 0 2024-04-22T23:23:39.044 INFO:tasks.workunit.client.1.smithi134.stdout:8/171: dwrite d7/db/fe [0,4194304] 0 2024-04-22T23:23:39.044 INFO:tasks.workunit.client.0.smithi067.stdout:1/858: dread d8/d9/d2d/d4d/d9c/f8a [0,4194304] 0 2024-04-22T23:23:39.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:38 smithi067 ceph-mon[26453]: pgmap v274: 65 pgs: 65 active+clean; 1.9 GiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 64 MiB/s rd, 187 MiB/s wr, 394 op/s 2024-04-22T23:23:39.047 INFO:tasks.workunit.client.1.smithi134.stdout:9/105: write d0/f3 [4340818,59094] 0 2024-04-22T23:23:39.048 INFO:tasks.workunit.client.0.smithi067.stdout:2/856: unlink d1/d10/d72/d53/f74 0 2024-04-22T23:23:39.048 INFO:tasks.workunit.client.1.smithi134.stdout:8/172: dread - d7/dc/f12 zero size 2024-04-22T23:23:39.048 INFO:tasks.workunit.client.0.smithi067.stdout:5/964: mknod d5/db/d12/d8a/c14e 0 2024-04-22T23:23:39.049 INFO:tasks.workunit.client.1.smithi134.stdout:2/155: dread d2/f18 [0,4194304] 0 2024-04-22T23:23:39.050 INFO:tasks.workunit.client.1.smithi134.stdout:2/156: read - d2/dd/f37 zero size 2024-04-22T23:23:39.050 INFO:tasks.workunit.client.0.smithi067.stdout:8/895: creat d1/dd/d80/db9/d110/d62/ddf/ded/def/f122 x:0 0 0 2024-04-22T23:23:39.050 INFO:tasks.workunit.client.0.smithi067.stdout:0/997: rmdir db/de/d11 39 2024-04-22T23:23:39.053 INFO:tasks.workunit.client.0.smithi067.stdout:7/850: dwrite d10/d14/d1d/d46/d50/f30 [0,4194304] 0 2024-04-22T23:23:39.053 INFO:tasks.workunit.client.0.smithi067.stdout:0/998: readlink db/d21/d29/l44 0 2024-04-22T23:23:39.053 INFO:tasks.workunit.client.1.smithi134.stdout:0/177: rename d1/d5/dd/d11/c19 to d1/d12/c34 0 2024-04-22T23:23:39.055 INFO:tasks.workunit.client.1.smithi134.stdout:6/175: link d0/d19/de/d1f/l28 d0/d19/de/d12/d2a/l40 0 2024-04-22T23:23:39.058 INFO:tasks.workunit.client.1.smithi134.stdout:8/173: symlink d7/dc/d2c/l35 0 2024-04-22T23:23:39.059 INFO:tasks.workunit.client.1.smithi134.stdout:9/106: creat d0/d10/f21 x:0 0 0 2024-04-22T23:23:39.060 INFO:tasks.workunit.client.0.smithi067.stdout:7/851: readlink d10/d14/d1d/d24/l57 0 2024-04-22T23:23:39.061 INFO:tasks.workunit.client.0.smithi067.stdout:7/852: chown d10/d14 22092371 1 2024-04-22T23:23:39.063 INFO:tasks.workunit.client.1.smithi134.stdout:7/170: creat d4/d8/d41/f43 x:0 0 0 2024-04-22T23:23:39.063 INFO:tasks.workunit.client.0.smithi067.stdout:1/859: dwrite d8/d9/d2d/d4d/d83/dc5/fcf [0,4194304] 0 2024-04-22T23:23:39.064 INFO:tasks.workunit.client.1.smithi134.stdout:5/193: sync 2024-04-22T23:23:39.065 INFO:tasks.workunit.client.1.smithi134.stdout:0/178: dwrite d1/d5/dd/d20/f1e [0,4194304] 0 2024-04-22T23:23:39.066 INFO:tasks.workunit.client.0.smithi067.stdout:8/896: dwrite d1/d3/f11 [4194304,4194304] 0 2024-04-22T23:23:39.066 INFO:tasks.workunit.client.0.smithi067.stdout:6/888: write d17/d39/d53/d90/d46/d62/d88/f117 [4584178,101963] 0 2024-04-22T23:23:39.066 INFO:tasks.workunit.client.1.smithi134.stdout:0/179: chown d1/d5/d15 0 1 2024-04-22T23:23:39.067 INFO:tasks.workunit.client.1.smithi134.stdout:7/171: write d4/f2e [244856,96699] 0 2024-04-22T23:23:39.067 INFO:tasks.workunit.client.0.smithi067.stdout:3/897: dread d3/d4/d28/d40/dfd/dfe/d4b/f8f [0,4194304] 0 2024-04-22T23:23:39.069 INFO:tasks.workunit.client.0.smithi067.stdout:0/999: creat db/d9a/d56/d70/dbf/f148 x:0 0 0 2024-04-22T23:23:39.069 INFO:tasks.workunit.client.0.smithi067.stdout:8/897: read d1/f7a [211116,12531] 0 2024-04-22T23:23:39.069 INFO:tasks.workunit.client.1.smithi134.stdout:7/172: read - d4/f33 zero size 2024-04-22T23:23:39.071 INFO:tasks.workunit.client.0.smithi067.stdout:8/898: readlink d1/dd/d80/db9/d110/l37 0 2024-04-22T23:23:39.073 INFO:tasks.workunit.client.0.smithi067.stdout:1/860: symlink d8/d9/d2f/d31/l115 0 2024-04-22T23:23:39.074 INFO:tasks.workunit.client.1.smithi134.stdout:8/174: symlink d7/db/d2d/d33/l36 0 2024-04-22T23:23:39.075 INFO:tasks.workunit.client.0.smithi067.stdout:1/861: write d8/d9/d2d/d4d/d9c/f6c [2345378,99102] 0 2024-04-22T23:23:39.076 INFO:tasks.workunit.client.1.smithi134.stdout:6/176: mknod d0/d19/de/c41 0 2024-04-22T23:23:39.076 INFO:tasks.workunit.client.0.smithi067.stdout:3/898: symlink d3/d4/d10/d38/d53/dcf/l136 0 2024-04-22T23:23:39.076 INFO:tasks.workunit.client.0.smithi067.stdout:2/857: read d1/f8 [3323440,31672] 0 2024-04-22T23:23:39.077 INFO:tasks.workunit.client.0.smithi067.stdout:8/899: creat d1/dd/d80/d6f/dfc/f123 x:0 0 0 2024-04-22T23:23:39.077 INFO:tasks.workunit.client.1.smithi134.stdout:8/175: read d7/dc/d19/d25/f26 [3162351,82288] 0 2024-04-22T23:23:39.078 INFO:tasks.workunit.client.1.smithi134.stdout:7/173: getdents d4/d8/d2a/d42 0 2024-04-22T23:23:39.078 INFO:tasks.workunit.client.0.smithi067.stdout:1/862: write d8/d9/d2d/f112 [666827,54133] 0 2024-04-22T23:23:39.080 INFO:tasks.workunit.client.1.smithi134.stdout:8/176: write d7/db/fe [2059711,115881] 0 2024-04-22T23:23:39.080 INFO:tasks.workunit.client.0.smithi067.stdout:3/899: chown d3/d4/d10/d1f/d3c/c44 151546 1 2024-04-22T23:23:39.080 INFO:tasks.workunit.client.0.smithi067.stdout:2/858: dread - d1/d10/d72/d68/da1/ff3 zero size 2024-04-22T23:23:39.081 INFO:tasks.workunit.client.0.smithi067.stdout:8/900: mknod d1/dd/d80/db9/d110/c124 0 2024-04-22T23:23:39.083 INFO:tasks.workunit.client.0.smithi067.stdout:3/900: readlink d3/d4/d10/d38/d53/d55/d74/dd5/ddb/l102 0 2024-04-22T23:23:39.083 INFO:tasks.workunit.client.0.smithi067.stdout:8/901: stat d1/d3/lf5 0 2024-04-22T23:23:39.084 INFO:tasks.workunit.client.1.smithi134.stdout:1/147: rename db/d15/l1e to db/l37 0 2024-04-22T23:23:39.087 INFO:tasks.workunit.client.1.smithi134.stdout:1/148: write db/d15/f29 [1258037,99525] 0 2024-04-22T23:23:39.088 INFO:tasks.workunit.client.1.smithi134.stdout:6/177: rmdir d0/d19/de/d12/d29 39 2024-04-22T23:23:39.089 INFO:tasks.workunit.client.0.smithi067.stdout:2/859: read d1/d10/f30 [1502510,99647] 0 2024-04-22T23:23:39.090 INFO:tasks.workunit.client.0.smithi067.stdout:1/863: dwrite d8/d9/d2f/d33/fb2 [0,4194304] 0 2024-04-22T23:23:39.091 INFO:tasks.workunit.client.1.smithi134.stdout:7/174: dwrite d4/d10/f23 [0,4194304] 0 2024-04-22T23:23:39.091 INFO:tasks.workunit.client.1.smithi134.stdout:1/149: chown db/d15/d17/d20/d2a/d2d/d2e 1223 1 2024-04-22T23:23:39.093 INFO:tasks.workunit.client.0.smithi067.stdout:3/901: write d3/d4/d10/d19/f3b [515010,56692] 0 2024-04-22T23:23:39.093 INFO:tasks.workunit.client.1.smithi134.stdout:8/177: dread d7/dc/d19/f21 [0,4194304] 0 2024-04-22T23:23:39.093 INFO:tasks.workunit.client.1.smithi134.stdout:1/150: truncate db/f12 1330224 0 2024-04-22T23:23:39.093 INFO:tasks.workunit.client.0.smithi067.stdout:8/902: creat d1/d3/f125 x:0 0 0 2024-04-22T23:23:39.095 INFO:tasks.workunit.client.0.smithi067.stdout:8/903: fdatasync d1/d3/d27/f113 0 2024-04-22T23:23:39.097 INFO:tasks.workunit.client.0.smithi067.stdout:3/902: fsync d3/dcb/fde 0 2024-04-22T23:23:39.098 INFO:tasks.workunit.client.1.smithi134.stdout:3/182: sync 2024-04-22T23:23:39.099 INFO:tasks.workunit.client.0.smithi067.stdout:2/860: mkdir d1/d10/d1c/d11e 0 2024-04-22T23:23:39.099 INFO:tasks.workunit.client.1.smithi134.stdout:4/289: truncate d4/f6 2950204 0 2024-04-22T23:23:39.101 INFO:tasks.workunit.client.0.smithi067.stdout:3/903: fsync d3/d4/d10/d38/d53/d55/d5c/ffa 0 2024-04-22T23:23:39.103 INFO:tasks.workunit.client.1.smithi134.stdout:1/151: dwrite fa [0,4194304] 0 2024-04-22T23:23:39.107 INFO:tasks.workunit.client.0.smithi067.stdout:9/841: dwrite d5/f9 [4194304,4194304] 0 2024-04-22T23:23:39.116 INFO:tasks.workunit.client.1.smithi134.stdout:4/290: dwrite d4/d35/f62 [0,4194304] 0 2024-04-22T23:23:39.116 INFO:tasks.workunit.client.1.smithi134.stdout:2/157: truncate d2/f30 2806085 0 2024-04-22T23:23:39.117 INFO:tasks.workunit.client.1.smithi134.stdout:9/107: dwrite d0/d7/fb [0,4194304] 0 2024-04-22T23:23:39.122 INFO:tasks.workunit.client.1.smithi134.stdout:6/178: rename d0/f3 to d0/f42 0 2024-04-22T23:23:39.123 INFO:tasks.workunit.client.0.smithi067.stdout:7/853: dread d10/d14/d1d/d46/d6a/f70 [0,4194304] 0 2024-04-22T23:23:39.124 INFO:tasks.workunit.client.1.smithi134.stdout:7/175: creat d4/d10/d2f/f44 x:0 0 0 2024-04-22T23:23:39.125 INFO:tasks.workunit.client.1.smithi134.stdout:8/178: write d7/f23 [241083,16233] 0 2024-04-22T23:23:39.125 INFO:tasks.workunit.client.1.smithi134.stdout:8/179: chown d7/dc/d2c 3 1 2024-04-22T23:23:39.127 INFO:tasks.workunit.client.1.smithi134.stdout:1/152: unlink db/d15/d17/f2c 0 2024-04-22T23:23:39.128 INFO:tasks.workunit.client.1.smithi134.stdout:6/179: chown d0/d19/de/d12/d2a/f3b 116 1 2024-04-22T23:23:39.128 INFO:tasks.workunit.client.1.smithi134.stdout:7/176: chown d4/f2d 3024607 1 2024-04-22T23:23:39.129 INFO:tasks.workunit.client.1.smithi134.stdout:1/153: dread - db/f1b zero size 2024-04-22T23:23:39.130 INFO:tasks.workunit.client.1.smithi134.stdout:1/154: dread - db/f26 zero size 2024-04-22T23:23:39.131 INFO:tasks.workunit.client.1.smithi134.stdout:1/155: stat db/ff 0 2024-04-22T23:23:39.132 INFO:tasks.workunit.client.1.smithi134.stdout:4/291: mkdir d4/d15/d64 0 2024-04-22T23:23:39.133 INFO:tasks.workunit.client.1.smithi134.stdout:2/158: dread d2/dd/df/d1c/f21 [0,4194304] 0 2024-04-22T23:23:39.133 INFO:tasks.workunit.client.1.smithi134.stdout:9/108: read d0/d7/d12/f18 [245319,25476] 0 2024-04-22T23:23:39.133 INFO:tasks.workunit.client.1.smithi134.stdout:4/292: stat d4/d35/d5e/l1a 0 2024-04-22T23:23:39.134 INFO:tasks.workunit.client.1.smithi134.stdout:4/293: read d4/d35/d5e/f22 [2408588,7298] 0 2024-04-22T23:23:39.136 INFO:tasks.workunit.client.1.smithi134.stdout:6/180: creat d0/d19/de/d2f/f43 x:0 0 0 2024-04-22T23:23:39.136 INFO:tasks.workunit.client.1.smithi134.stdout:7/177: creat d4/d8/d1d/f45 x:0 0 0 2024-04-22T23:23:39.138 INFO:tasks.workunit.client.1.smithi134.stdout:9/109: dread d0/d7/d12/f1f [0,4194304] 0 2024-04-22T23:23:39.138 INFO:tasks.workunit.client.1.smithi134.stdout:0/180: rename d1/d5/f21 to d1/d5/dd/f35 0 2024-04-22T23:23:39.139 INFO:tasks.workunit.client.1.smithi134.stdout:9/110: fsync d0/d7/fd 0 2024-04-22T23:23:39.139 INFO:tasks.workunit.client.1.smithi134.stdout:1/156: symlink db/d15/l38 0 2024-04-22T23:23:39.140 INFO:tasks.workunit.client.1.smithi134.stdout:2/159: write d2/f17 [858424,95458] 0 2024-04-22T23:23:39.141 INFO:tasks.workunit.client.1.smithi134.stdout:3/183: creat dc/dd/d28/d2f/f49 x:0 0 0 2024-04-22T23:23:39.142 INFO:tasks.workunit.client.1.smithi134.stdout:7/178: symlink d4/d8/d1d/l46 0 2024-04-22T23:23:39.144 INFO:tasks.workunit.client.1.smithi134.stdout:5/194: rename d9/f36 to d9/d11/d34/f40 0 2024-04-22T23:23:39.147 INFO:tasks.workunit.client.1.smithi134.stdout:7/179: dread d4/f2e [0,4194304] 0 2024-04-22T23:23:39.147 INFO:tasks.workunit.client.1.smithi134.stdout:7/180: chown d4/lc 5313155 1 2024-04-22T23:23:39.147 INFO:tasks.workunit.client.1.smithi134.stdout:0/181: dwrite d1/d12/d22/f33 [0,4194304] 0 2024-04-22T23:23:39.149 INFO:tasks.workunit.client.0.smithi067.stdout:8/904: rename d1/dd/d80/db9/d110/f22 to d1/d3/d27/d107/f126 0 2024-04-22T23:23:39.150 INFO:tasks.workunit.client.0.smithi067.stdout:8/905: readlink d1/dd/d80/db9/d110/d62/ddf/lf3 0 2024-04-22T23:23:39.151 INFO:tasks.workunit.client.1.smithi134.stdout:0/182: dread - d1/d5/dd/f2a zero size 2024-04-22T23:23:39.151 INFO:tasks.workunit.client.1.smithi134.stdout:1/157: readlink db/l37 0 2024-04-22T23:23:39.153 INFO:tasks.workunit.client.0.smithi067.stdout:2/861: truncate d1/d10/d72/d68/fbe 444637 0 2024-04-22T23:23:39.153 INFO:tasks.workunit.client.1.smithi134.stdout:2/160: symlink d2/dd/l38 0 2024-04-22T23:23:39.155 INFO:tasks.workunit.client.0.smithi067.stdout:7/854: truncate d10/d18/d88/daf/d100/d62/f94 4091113 0 2024-04-22T23:23:39.157 INFO:tasks.workunit.client.1.smithi134.stdout:6/181: creat d0/d19/db/d35/f44 x:0 0 0 2024-04-22T23:23:39.158 INFO:tasks.workunit.client.0.smithi067.stdout:7/855: truncate d10/d14/de0/ff7 114346 0 2024-04-22T23:23:39.158 INFO:tasks.workunit.client.0.smithi067.stdout:9/842: rename d5/dd/d18/d3c/d40/d113 to d5/dd/da9/db9/d115 0 2024-04-22T23:23:39.159 INFO:tasks.workunit.client.1.smithi134.stdout:3/184: readlink dc/d2a/d30/l46 0 2024-04-22T23:23:39.159 INFO:tasks.workunit.client.0.smithi067.stdout:8/906: symlink d1/dd/d80/d6f/dfc/l127 0 2024-04-22T23:23:39.160 INFO:tasks.workunit.client.1.smithi134.stdout:3/185: fdatasync dc/dd/d28/d34/f48 0 2024-04-22T23:23:39.161 INFO:tasks.workunit.client.0.smithi067.stdout:9/843: read d5/d21/f24 [4056873,115091] 0 2024-04-22T23:23:39.163 INFO:tasks.workunit.client.1.smithi134.stdout:3/186: write dc/dd/d28/f39 [47412,101357] 0 2024-04-22T23:23:39.166 INFO:tasks.workunit.client.0.smithi067.stdout:9/844: getdents d5/dd/d18/d39/d49/dcc 0 2024-04-22T23:23:39.167 INFO:tasks.workunit.client.0.smithi067.stdout:1/864: write d8/d9/fa9 [13480,107606] 0 2024-04-22T23:23:39.170 INFO:tasks.workunit.client.1.smithi134.stdout:0/183: mknod d1/d2/d1d/c36 0 2024-04-22T23:23:39.256 INFO:tasks.workunit.client.0.smithi067.stdout:9/845: fdatasync d5/dd/d18/d39/d3e/fae 0 2024-04-22T23:23:39.267 INFO:tasks.workunit.client.1.smithi134.stdout:0/184: truncate d1/d5/dd/d20/f25 4233177 0 2024-04-22T23:23:39.267 INFO:tasks.workunit.client.1.smithi134.stdout:8/180: getdents d7/dc 0 2024-04-22T23:23:39.267 INFO:tasks.workunit.client.1.smithi134.stdout:9/111: dread d0/f6 [0,4194304] 0 2024-04-22T23:23:39.268 INFO:tasks.workunit.client.1.smithi134.stdout:9/112: chown d0/d7/d12 70366144 1 2024-04-22T23:23:39.268 INFO:tasks.workunit.client.1.smithi134.stdout:6/182: symlink d0/d19/de/d12/d2a/l45 0 2024-04-22T23:23:39.268 INFO:tasks.workunit.client.1.smithi134.stdout:7/181: link d4/d8/ff d4/d8/d2a/f47 0 2024-04-22T23:23:39.268 INFO:tasks.workunit.client.1.smithi134.stdout:6/183: dwrite d0/d19/f1c [0,4194304] 0 2024-04-22T23:23:39.268 INFO:tasks.workunit.client.1.smithi134.stdout:7/182: write d4/d10/f23 [4845963,108838] 0 2024-04-22T23:23:39.268 INFO:tasks.workunit.client.1.smithi134.stdout:4/294: rename d4/d35/f56 to d4/d13/f65 0 2024-04-22T23:23:39.268 INFO:tasks.workunit.client.1.smithi134.stdout:6/184: write d0/d19/de/d2f/f43 [713945,108395] 0 2024-04-22T23:23:39.268 INFO:tasks.workunit.client.1.smithi134.stdout:4/295: stat d4/d7/l46 0 2024-04-22T23:23:39.268 INFO:tasks.workunit.client.1.smithi134.stdout:6/185: dread - d0/d19/db/d35/f44 zero size 2024-04-22T23:23:39.268 INFO:tasks.workunit.client.1.smithi134.stdout:1/158: creat db/d15/d17/d20/d2a/d2d/d2e/f39 x:0 0 0 2024-04-22T23:23:39.268 INFO:tasks.workunit.client.1.smithi134.stdout:6/186: chown d0/d19/de/d12/d2a 112910 1 2024-04-22T23:23:39.269 INFO:tasks.workunit.client.1.smithi134.stdout:0/185: symlink d1/d2/l37 0 2024-04-22T23:23:39.269 INFO:tasks.workunit.client.1.smithi134.stdout:6/187: chown d0/d19/f9 158557 1 2024-04-22T23:23:39.269 INFO:tasks.workunit.client.1.smithi134.stdout:8/181: mkdir d7/dc/d2c/d37 0 2024-04-22T23:23:39.269 INFO:tasks.workunit.client.1.smithi134.stdout:4/296: mkdir d4/d15/d66 0 2024-04-22T23:23:39.269 INFO:tasks.workunit.client.1.smithi134.stdout:1/159: symlink db/d15/d17/d20/d2a/l3a 0 2024-04-22T23:23:39.269 INFO:tasks.workunit.client.1.smithi134.stdout:0/186: creat d1/d2/f38 x:0 0 0 2024-04-22T23:23:39.269 INFO:tasks.workunit.client.1.smithi134.stdout:5/195: truncate d9/d11/f1a 293140 0 2024-04-22T23:23:39.269 INFO:tasks.workunit.client.1.smithi134.stdout:7/183: creat d4/d8/d3b/f48 x:0 0 0 2024-04-22T23:23:39.269 INFO:tasks.workunit.client.1.smithi134.stdout:4/297: stat d4/l40 0 2024-04-22T23:23:39.269 INFO:tasks.workunit.client.1.smithi134.stdout:1/160: mknod db/d15/d17/d20/c3b 0 2024-04-22T23:23:39.269 INFO:tasks.workunit.client.1.smithi134.stdout:6/188: rename d0/d19/de/cf to d0/d19/de/d12/d2a/d2d/c46 0 2024-04-22T23:23:39.269 INFO:tasks.workunit.client.1.smithi134.stdout:5/196: dread d9/f19 [0,4194304] 0 2024-04-22T23:23:39.269 INFO:tasks.workunit.client.1.smithi134.stdout:9/113: dwrite d0/f6 [0,4194304] 0 2024-04-22T23:23:39.270 INFO:tasks.workunit.client.1.smithi134.stdout:4/298: creat d4/d15/f67 x:0 0 0 2024-04-22T23:23:39.270 INFO:tasks.workunit.client.1.smithi134.stdout:3/187: dwrite dc/dd/d28/f1c [0,4194304] 0 2024-04-22T23:23:39.270 INFO:tasks.workunit.client.1.smithi134.stdout:9/114: write d0/d10/f21 [788493,36131] 0 2024-04-22T23:23:39.270 INFO:tasks.workunit.client.1.smithi134.stdout:9/115: chown d0/d7/f17 16095316 1 2024-04-22T23:23:39.270 INFO:tasks.workunit.client.1.smithi134.stdout:1/161: mknod db/d16/c3c 0 2024-04-22T23:23:39.270 INFO:tasks.workunit.client.1.smithi134.stdout:5/197: dwrite d9/d1e/f33 [0,4194304] 0 2024-04-22T23:23:39.270 INFO:tasks.workunit.client.1.smithi134.stdout:2/161: dread d2/f30 [0,4194304] 0 2024-04-22T23:23:39.270 INFO:tasks.workunit.client.1.smithi134.stdout:9/116: truncate d0/f6 4733959 0 2024-04-22T23:23:39.270 INFO:tasks.workunit.client.1.smithi134.stdout:1/162: write db/f12 [1502012,116627] 0 2024-04-22T23:23:39.270 INFO:tasks.workunit.client.1.smithi134.stdout:5/198: chown d9/fa 252 1 2024-04-22T23:23:39.270 INFO:tasks.workunit.client.1.smithi134.stdout:8/182: dwrite d7/dc/d19/d25/f34 [0,4194304] 0 2024-04-22T23:23:39.270 INFO:tasks.workunit.client.1.smithi134.stdout:3/188: mknod dc/d2a/c4a 0 2024-04-22T23:23:39.270 INFO:tasks.workunit.client.1.smithi134.stdout:0/187: rename d1/d5/dd/d11/c1b to d1/d5/dd/d11/c39 0 2024-04-22T23:23:39.271 INFO:tasks.workunit.client.1.smithi134.stdout:7/184: truncate d4/d8/f22 2027006 0 2024-04-22T23:23:39.271 INFO:tasks.workunit.client.1.smithi134.stdout:2/162: stat d2/lc 0 2024-04-22T23:23:39.271 INFO:tasks.workunit.client.1.smithi134.stdout:2/163: fsync d2/f2e 0 2024-04-22T23:23:39.271 INFO:tasks.workunit.client.1.smithi134.stdout:5/199: read d9/d11/f1b [68489,61046] 0 2024-04-22T23:23:39.271 INFO:tasks.workunit.client.1.smithi134.stdout:2/164: fdatasync d2/d15/d24/d1d/f36 0 2024-04-22T23:23:39.271 INFO:tasks.workunit.client.1.smithi134.stdout:9/117: rename d0/d10/d11 to d0/d7/d12/d22 0 2024-04-22T23:23:39.271 INFO:tasks.workunit.client.1.smithi134.stdout:2/165: chown d2/d15/d24/d1d/l20 4583331 1 2024-04-22T23:23:39.272 INFO:tasks.workunit.client.0.smithi067.stdout:8/907: dwrite d1/dd/d80/db9/f108 [0,4194304] 0 2024-04-22T23:23:39.272 INFO:tasks.workunit.client.0.smithi067.stdout:3/904: dwrite d3/d4/d1b/faa [0,4194304] 0 2024-04-22T23:23:39.272 INFO:tasks.workunit.client.0.smithi067.stdout:9/846: creat d5/dd/d18/d3c/f116 x:0 0 0 2024-04-22T23:23:39.273 INFO:tasks.workunit.client.0.smithi067.stdout:3/905: chown d3/d4/d28/d4a/f73 13560663 1 2024-04-22T23:23:39.273 INFO:tasks.workunit.client.0.smithi067.stdout:8/908: getdents d1/dd/d80/d6f/dfc 0 2024-04-22T23:23:39.273 INFO:tasks.workunit.client.0.smithi067.stdout:7/856: dread d10/d14/d1d/d46/d6a/f86 [0,4194304] 0 2024-04-22T23:23:39.273 INFO:tasks.workunit.client.0.smithi067.stdout:2/862: dwrite d1/d42/f57 [0,4194304] 0 2024-04-22T23:23:39.273 INFO:tasks.workunit.client.0.smithi067.stdout:9/847: creat d5/dd/d18/d39/d3e/d3f/d63/f117 x:0 0 0 2024-04-22T23:23:39.273 INFO:tasks.workunit.client.0.smithi067.stdout:3/906: unlink d3/d4/d10/d38/d53/d55/d74/ff6 0 2024-04-22T23:23:39.273 INFO:tasks.workunit.client.0.smithi067.stdout:7/857: rename d10/c11d to d10/d14/d1d/d26/d80/dd2/dd4/c13e 0 2024-04-22T23:23:39.273 INFO:tasks.workunit.client.0.smithi067.stdout:1/865: write d8/d9/d2f/d59/f7b [5768031,25488] 0 2024-04-22T23:23:39.273 INFO:tasks.workunit.client.0.smithi067.stdout:2/863: mkdir d1/d10/d1c/dd9/d11f 0 2024-04-22T23:23:39.273 INFO:tasks.workunit.client.0.smithi067.stdout:3/907: dread d3/d4/d28/d40/dfd/dfe/f83 [0,4194304] 0 2024-04-22T23:23:39.273 INFO:tasks.workunit.client.0.smithi067.stdout:1/866: fsync d8/d9/de0/d60/d62/f95 0 2024-04-22T23:23:39.273 INFO:tasks.workunit.client.0.smithi067.stdout:9/848: write d5/d21/d47/d56/d8c/dd5/fc1 [5417163,72494] 0 2024-04-22T23:23:39.273 INFO:tasks.workunit.client.0.smithi067.stdout:7/858: symlink d10/d14/d1d/d26/d2c/d7a/l13f 0 2024-04-22T23:23:39.273 INFO:tasks.workunit.client.0.smithi067.stdout:2/864: creat d1/d19/f120 x:0 0 0 2024-04-22T23:23:39.273 INFO:tasks.workunit.client.0.smithi067.stdout:1/867: dread - d8/d9/de0/d60/fa1 zero size 2024-04-22T23:23:39.274 INFO:tasks.workunit.client.0.smithi067.stdout:7/859: dwrite d10/d47/dab/fd5 [0,4194304] 0 2024-04-22T23:23:39.274 INFO:tasks.workunit.client.0.smithi067.stdout:9/849: link d5/dd/d18/d3c/d78/ce7 d5/dd/da9/df7/c118 0 2024-04-22T23:23:39.274 INFO:tasks.workunit.client.0.smithi067.stdout:9/850: chown d5/d21/d47/d57/fa2 0 1 2024-04-22T23:23:39.274 INFO:tasks.workunit.client.0.smithi067.stdout:1/868: mknod d8/d9/d2f/d59/dec/c116 0 2024-04-22T23:23:39.274 INFO:tasks.workunit.client.0.smithi067.stdout:7/860: symlink d10/d14/d1d/d26/d2c/d7a/l140 0 2024-04-22T23:23:39.274 INFO:tasks.workunit.client.0.smithi067.stdout:9/851: mkdir d5/dd/da9/db9/d115/d119 0 2024-04-22T23:23:39.274 INFO:tasks.workunit.client.0.smithi067.stdout:1/869: truncate d8/d9/d2f/d33/d66/ffe 380164 0 2024-04-22T23:23:39.274 INFO:tasks.workunit.client.0.smithi067.stdout:7/861: dread - d10/d14/d1d/d26/d2c/d41/f9f zero size 2024-04-22T23:23:39.274 INFO:tasks.workunit.client.0.smithi067.stdout:1/870: symlink d8/d9/d2f/d31/l117 0 2024-04-22T23:23:39.274 INFO:tasks.workunit.client.0.smithi067.stdout:7/862: rename d10/d18/d88/daf/ld3 to d10/d18/l141 0 2024-04-22T23:23:39.274 INFO:tasks.workunit.client.0.smithi067.stdout:1/871: symlink d8/d9/d2f/d59/dd8/l118 0 2024-04-22T23:23:39.274 INFO:tasks.workunit.client.0.smithi067.stdout:9/852: dwrite d5/d21/d47/d56/d8c/dd5/f10b [0,4194304] 0 2024-04-22T23:23:39.274 INFO:tasks.workunit.client.0.smithi067.stdout:7/863: rename d10/d14/d1d/d26/ca4 to d10/d14/d1d/d26/d2c/d41/db0/ddd/d122/c142 0 2024-04-22T23:23:39.274 INFO:tasks.workunit.client.0.smithi067.stdout:7/864: mkdir d10/d14/d1d/d46/d6a/df1/d143 0 2024-04-22T23:23:39.274 INFO:tasks.workunit.client.0.smithi067.stdout:3/908: write d3/d4/d10/de3/ff8 [134199,94364] 0 2024-04-22T23:23:39.275 INFO:tasks.workunit.client.0.smithi067.stdout:3/909: creat d3/d4/d10/d8a/f137 x:0 0 0 2024-04-22T23:23:39.275 INFO:tasks.workunit.client.0.smithi067.stdout:7/865: rename d10/d14/d1d/d26/d2c/d41/db0/fea to d10/d14/de0/f144 0 2024-04-22T23:23:39.275 INFO:tasks.workunit.client.0.smithi067.stdout:2/865: dwrite d1/d10/d72/d68/fab [0,4194304] 0 2024-04-22T23:23:39.275 INFO:tasks.workunit.client.0.smithi067.stdout:3/910: dread d3/d4/d10/d8a/fa6 [0,4194304] 0 2024-04-22T23:23:39.275 INFO:tasks.workunit.client.0.smithi067.stdout:9/853: dread d5/dd/d18/d39/d3e/d3f/f7e [0,4194304] 0 2024-04-22T23:23:39.275 INFO:tasks.workunit.client.0.smithi067.stdout:2/866: creat d1/d10/d72/d68/d7b/dce/f121 x:0 0 0 2024-04-22T23:23:39.275 INFO:tasks.workunit.client.0.smithi067.stdout:7/866: creat d10/d14/d1d/d26/d2c/d41/db0/ddd/d122/d123/f145 x:0 0 0 2024-04-22T23:23:39.275 INFO:tasks.workunit.client.0.smithi067.stdout:7/867: truncate d10/d47/dab/f12e 33371 0 2024-04-22T23:23:39.275 INFO:tasks.workunit.client.0.smithi067.stdout:7/868: write fc [4083023,42696] 0 2024-04-22T23:23:39.275 INFO:tasks.workunit.client.0.smithi067.stdout:3/911: rename d3/d4/d10/d10b/d95/ca3 to d3/d4/d10/de3/c138 0 2024-04-22T23:23:39.275 INFO:tasks.workunit.client.0.smithi067.stdout:3/912: stat d3/d4/d10/d1f/d3c/df4/f107 0 2024-04-22T23:23:39.275 INFO:tasks.workunit.client.0.smithi067.stdout:7/869: creat d10/f146 x:0 0 0 2024-04-22T23:23:39.275 INFO:tasks.workunit.client.0.smithi067.stdout:3/913: mkdir d3/d4/d28/dc0/d11b/d139 0 2024-04-22T23:23:39.275 INFO:tasks.workunit.client.0.smithi067.stdout:7/870: readlink d10/d14/d1d/d26/d2c/df4/le1 0 2024-04-22T23:23:39.275 INFO:tasks.workunit.client.0.smithi067.stdout:3/914: dread d3/fc1 [0,4194304] 0 2024-04-22T23:23:39.276 INFO:tasks.workunit.client.0.smithi067.stdout:7/871: chown d10/d14/d1d/f42 6815700 1 2024-04-22T23:23:39.276 INFO:tasks.workunit.client.0.smithi067.stdout:7/872: write d10/d14/d1d/d26/d80/f8a [1323560,47906] 0 2024-04-22T23:23:39.276 INFO:tasks.workunit.client.0.smithi067.stdout:7/873: fsync d10/d18/d88/daf/d100/d62/ff6 0 2024-04-22T23:23:39.276 INFO:tasks.workunit.client.0.smithi067.stdout:7/874: read - d10/f146 zero size 2024-04-22T23:23:39.276 INFO:tasks.workunit.client.0.smithi067.stdout:3/915: mkdir d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d51/d13a 0 2024-04-22T23:23:39.276 INFO:tasks.workunit.client.0.smithi067.stdout:3/916: truncate d3/d4/d10/d1f/f3e 4553107 0 2024-04-22T23:23:39.276 INFO:tasks.workunit.client.0.smithi067.stdout:7/875: mkdir d10/d18/d88/daf/d100/d62/d147 0 2024-04-22T23:23:39.277 INFO:tasks.workunit.client.1.smithi134.stdout:2/166: write d2/d15/d24/f27 [7700911,49863] 0 2024-04-22T23:23:39.277 INFO:tasks.workunit.client.1.smithi134.stdout:1/163: creat db/d16/f3d x:0 0 0 2024-04-22T23:23:39.277 INFO:tasks.workunit.client.1.smithi134.stdout:8/183: mknod d7/dc/c38 0 2024-04-22T23:23:39.277 INFO:tasks.workunit.client.1.smithi134.stdout:1/164: symlink db/d15/d17/d20/d2a/l3e 0 2024-04-22T23:23:39.277 INFO:tasks.workunit.client.1.smithi134.stdout:7/185: mknod d4/d8/d2a/d42/c49 0 2024-04-22T23:23:39.278 INFO:tasks.workunit.client.0.smithi067.stdout:7/876: dwrite d10/d14/d1d/d69/fb8 [0,4194304] 0 2024-04-22T23:23:39.278 INFO:tasks.workunit.client.1.smithi134.stdout:8/184: mkdir d7/db/d2d/d39 0 2024-04-22T23:23:39.279 INFO:tasks.workunit.client.1.smithi134.stdout:5/200: symlink d9/d11/d34/d3b/l41 0 2024-04-22T23:23:39.282 INFO:tasks.workunit.client.0.smithi067.stdout:7/877: creat d10/d14/de0/deb/f148 x:0 0 0 2024-04-22T23:23:39.282 INFO:tasks.workunit.client.1.smithi134.stdout:3/189: creat dc/dd/d28/d2f/f4b x:0 0 0 2024-04-22T23:23:39.284 INFO:tasks.workunit.client.0.smithi067.stdout:2/867: rename d1/d10/d72 to d1/d42/d5f/d122 0 2024-04-22T23:23:39.285 INFO:tasks.workunit.client.0.smithi067.stdout:7/878: chown d10/d18/d88/daf/d100/d62/c134 3 1 2024-04-22T23:23:39.289 INFO:tasks.workunit.client.0.smithi067.stdout:7/879: dread - d10/d14/d1d/d26/d2c/fcf zero size 2024-04-22T23:23:39.291 INFO:tasks.workunit.client.1.smithi134.stdout:7/186: dwrite d4/f31 [0,4194304] 0 2024-04-22T23:23:39.291 INFO:tasks.workunit.client.1.smithi134.stdout:5/201: mknod d9/d11/c42 0 2024-04-22T23:23:39.291 INFO:tasks.workunit.client.1.smithi134.stdout:6/189: truncate d0/d19/db/f37 180624 0 2024-04-22T23:23:39.292 INFO:tasks.workunit.client.0.smithi067.stdout:7/880: dread - d10/d14/d1d/d26/d2c/d41/f7c zero size 2024-04-22T23:23:39.292 INFO:tasks.workunit.client.0.smithi067.stdout:2/868: dwrite d1/d19/f120 [0,4194304] 0 2024-04-22T23:23:39.299 INFO:tasks.workunit.client.1.smithi134.stdout:5/202: dwrite d9/d1e/f35 [0,4194304] 0 2024-04-22T23:23:39.299 INFO:tasks.workunit.client.0.smithi067.stdout:7/881: fdatasync d10/d18/d88/daf/d100/f5b 0 2024-04-22T23:23:39.301 INFO:tasks.workunit.client.1.smithi134.stdout:9/118: rename d0/d10 to d0/d7/d23 0 2024-04-22T23:23:39.303 INFO:tasks.workunit.client.0.smithi067.stdout:7/882: symlink d10/d14/d1d/d26/d2c/df4/l149 0 2024-04-22T23:23:39.304 INFO:tasks.workunit.client.1.smithi134.stdout:1/165: link db/d15/d17/d20/c3b db/d15/d17/d20/d2a/d2d/c3f 0 2024-04-22T23:23:39.306 INFO:tasks.workunit.client.1.smithi134.stdout:4/299: truncate d4/dd/f41 334700 0 2024-04-22T23:23:39.306 INFO:tasks.workunit.client.1.smithi134.stdout:1/166: fdatasync db/d15/f29 0 2024-04-22T23:23:39.307 INFO:tasks.workunit.client.1.smithi134.stdout:3/190: symlink dc/dd/d28/d2f/d36/l4c 0 2024-04-22T23:23:39.307 INFO:tasks.workunit.client.1.smithi134.stdout:4/300: dread - d4/d35/d5e/f4a zero size 2024-04-22T23:23:39.310 INFO:tasks.workunit.client.1.smithi134.stdout:5/203: rename d9/dd/d17/d2a/l31 to d9/dd/d17/d2a/l43 0 2024-04-22T23:23:39.310 INFO:tasks.workunit.client.1.smithi134.stdout:4/301: readlink d4/d15/l50 0 2024-04-22T23:23:39.311 INFO:tasks.workunit.client.1.smithi134.stdout:3/191: read dc/dd/d28/f1c [3038869,14493] 0 2024-04-22T23:23:39.313 INFO:tasks.workunit.client.0.smithi067.stdout:2/869: creat d1/d10/d1c/f123 x:0 0 0 2024-04-22T23:23:39.313 INFO:tasks.workunit.client.1.smithi134.stdout:6/190: rename d0/l3f to d0/d19/de/d12/d29/l47 0 2024-04-22T23:23:39.314 INFO:tasks.workunit.client.1.smithi134.stdout:9/119: symlink d0/d7/dc/l24 0 2024-04-22T23:23:39.322 INFO:tasks.workunit.client.1.smithi134.stdout:7/187: dread d4/d8/d2a/f47 [0,4194304] 0 2024-04-22T23:23:39.323 INFO:tasks.workunit.client.1.smithi134.stdout:5/204: write d9/d11/d34/f40 [1446877,75141] 0 2024-04-22T23:23:39.325 INFO:tasks.workunit.client.1.smithi134.stdout:2/167: dwrite d2/d15/d24/d1d/f26 [0,4194304] 0 2024-04-22T23:23:39.328 INFO:tasks.workunit.client.1.smithi134.stdout:4/302: mkdir d4/d13/d68 0 2024-04-22T23:23:39.330 INFO:tasks.workunit.client.0.smithi067.stdout:7/883: fsync d10/d14/d1d/d26/d2c/d7a/dc2/f107 0 2024-04-22T23:23:39.331 INFO:tasks.workunit.client.1.smithi134.stdout:3/192: stat dc/dd/d28/d2f/d21 0 2024-04-22T23:23:39.331 INFO:tasks.workunit.client.0.smithi067.stdout:7/884: dread d10/d18/f1a [0,4194304] 0 2024-04-22T23:23:39.332 INFO:tasks.workunit.client.1.smithi134.stdout:4/303: write d4/f42 [1399086,35470] 0 2024-04-22T23:23:39.334 INFO:tasks.workunit.client.1.smithi134.stdout:5/205: dwrite d9/d1e/f32 [0,4194304] 0 2024-04-22T23:23:39.335 INFO:tasks.workunit.client.0.smithi067.stdout:7/885: rmdir d10/d14/d1d/d26/d2c/d7a 39 2024-04-22T23:23:39.336 INFO:tasks.workunit.client.0.smithi067.stdout:7/886: stat d10/d18/l36 0 2024-04-22T23:23:39.336 INFO:tasks.workunit.client.1.smithi134.stdout:9/120: symlink d0/d7/d23/l25 0 2024-04-22T23:23:39.340 INFO:tasks.workunit.client.1.smithi134.stdout:1/167: mknod db/c40 0 2024-04-22T23:23:39.342 INFO:tasks.workunit.client.1.smithi134.stdout:1/168: write db/d15/d17/d20/d2a/f36 [3158987,40130] 0 2024-04-22T23:23:39.344 INFO:tasks.workunit.client.0.smithi067.stdout:7/887: dwrite d10/d14/d1d/d26/d2c/d41/db0/ff0 [0,4194304] 0 2024-04-22T23:23:39.344 INFO:tasks.workunit.client.1.smithi134.stdout:1/169: write db/d15/d17/d20/d2a/d2d/d2e/f39 [449992,7406] 0 2024-04-22T23:23:39.350 INFO:tasks.workunit.client.0.smithi067.stdout:7/888: link d10/d18/f27 d10/f14a 0 2024-04-22T23:23:39.353 INFO:tasks.workunit.client.0.smithi067.stdout:7/889: creat d10/d14/d1d/d26/d80/dd2/dd4/d115/f14b x:0 0 0 2024-04-22T23:23:39.356 INFO:tasks.workunit.client.1.smithi134.stdout:2/168: creat d2/d15/d24/d1d/f39 x:0 0 0 2024-04-22T23:23:39.356 INFO:tasks.workunit.client.0.smithi067.stdout:7/890: mknod d10/d14/d1d/d46/d50/c14c 0 2024-04-22T23:23:39.356 INFO:tasks.workunit.client.1.smithi134.stdout:2/169: readlink d2/d15/d24/l31 0 2024-04-22T23:23:39.357 INFO:tasks.workunit.client.0.smithi067.stdout:7/891: dread - d10/d14/f109 zero size 2024-04-22T23:23:39.357 INFO:tasks.workunit.client.1.smithi134.stdout:2/170: read - d2/d15/d24/f35 zero size 2024-04-22T23:23:39.360 INFO:tasks.workunit.client.0.smithi067.stdout:7/892: rename d10/d18/d88/daf/d100/c4d to d10/d18/d88/daf/d100/c14d 0 2024-04-22T23:23:39.360 INFO:tasks.workunit.client.1.smithi134.stdout:6/191: mknod d0/d19/db/d33/c48 0 2024-04-22T23:23:39.362 INFO:tasks.workunit.client.0.smithi067.stdout:7/893: creat d10/d14/d1d/d26/d2c/d41/db0/f14e x:0 0 0 2024-04-22T23:23:39.364 INFO:tasks.workunit.client.0.smithi067.stdout:7/894: truncate d10/d14/d1d/d26/f40 1789217 0 2024-04-22T23:23:39.365 INFO:tasks.workunit.client.0.smithi067.stdout:5/965: sync 2024-04-22T23:23:39.365 INFO:tasks.workunit.client.0.smithi067.stdout:2/870: dwrite d1/d42/d5f/d122/d68/d7b/f8b [0,4194304] 0 2024-04-22T23:23:39.369 INFO:tasks.workunit.client.1.smithi134.stdout:5/206: rename d9/df/d20/d29 to d9/dd/d17/d44 0 2024-04-22T23:23:39.369 INFO:tasks.workunit.client.0.smithi067.stdout:2/871: write d1/d42/d5f/d9b/dc0/f10b [3863026,47680] 0 2024-04-22T23:23:39.371 INFO:tasks.workunit.client.1.smithi134.stdout:3/193: dwrite dc/dd/d28/d2f/d21/d42/f47 [0,4194304] 0 2024-04-22T23:23:39.371 INFO:tasks.workunit.client.0.smithi067.stdout:7/895: unlink d10/d14/d1d/d26/d2c/d7a/dc2/f107 0 2024-04-22T23:23:39.372 INFO:tasks.workunit.client.1.smithi134.stdout:3/194: chown dc/f3f 8350053 1 2024-04-22T23:23:39.373 INFO:tasks.workunit.client.0.smithi067.stdout:7/896: dread - d10/d14/d1d/d26/f93 zero size 2024-04-22T23:23:39.374 INFO:tasks.workunit.client.1.smithi134.stdout:9/121: unlink d0/d7/dc/f1d 0 2024-04-22T23:23:39.374 INFO:tasks.workunit.client.0.smithi067.stdout:2/872: dread d1/d19/d6f/f107 [0,4194304] 0 2024-04-22T23:23:39.374 INFO:tasks.workunit.client.1.smithi134.stdout:9/122: chown d0/d7/dc 5726043 1 2024-04-22T23:23:39.377 INFO:tasks.workunit.client.0.smithi067.stdout:7/897: mkdir d10/d14/d1d/d26/d2c/d41/db0/ddd/d14f 0 2024-04-22T23:23:39.378 INFO:tasks.workunit.client.1.smithi134.stdout:7/188: write d4/d8/fd [2725316,85327] 0 2024-04-22T23:23:39.379 INFO:tasks.workunit.client.1.smithi134.stdout:7/189: readlink d4/d10/l24 0 2024-04-22T23:23:39.379 INFO:tasks.workunit.client.1.smithi134.stdout:7/190: dread - d4/f33 zero size 2024-04-22T23:23:39.380 INFO:tasks.workunit.client.0.smithi067.stdout:7/898: mkdir d10/d14/d1d/d26/d80/dd2/dd4/d150 0 2024-04-22T23:23:39.380 INFO:tasks.workunit.client.1.smithi134.stdout:7/191: stat d4/d10/f23 0 2024-04-22T23:23:39.381 INFO:tasks.workunit.client.0.smithi067.stdout:7/899: chown d10/d18/d88/daf/ld9 195630892 1 2024-04-22T23:23:39.382 INFO:tasks.workunit.client.0.smithi067.stdout:2/873: dread d1/d10/d1c/f5e [0,4194304] 0 2024-04-22T23:23:39.382 INFO:tasks.workunit.client.1.smithi134.stdout:9/123: dwrite d0/f6 [0,4194304] 0 2024-04-22T23:23:39.389 INFO:tasks.workunit.client.0.smithi067.stdout:7/900: link d10/d14/d1d/d46/d124/ff3 d10/f151 0 2024-04-22T23:23:39.390 INFO:tasks.workunit.client.0.smithi067.stdout:7/901: chown d10/d14/d1d/d26/d2c/d41/db0/ddd/d122/d123 1322 1 2024-04-22T23:23:39.391 INFO:tasks.workunit.client.0.smithi067.stdout:7/902: rmdir d10/d18/d88 39 2024-04-22T23:23:39.429 INFO:tasks.workunit.client.1.smithi134.stdout:9/124: dwrite d0/fa [0,4194304] 0 2024-04-22T23:23:39.431 INFO:tasks.workunit.client.0.smithi067.stdout:2/874: write d1/d10/d1c/ddc/fdf [379379,68696] 0 2024-04-22T23:23:39.432 INFO:tasks.workunit.client.1.smithi134.stdout:6/192: creat d0/d19/de/d12/d2a/d2d/f49 x:0 0 0 2024-04-22T23:23:39.434 INFO:tasks.workunit.client.1.smithi134.stdout:6/193: write d0/f26 [569272,84986] 0 2024-04-22T23:23:39.434 INFO:tasks.workunit.client.0.smithi067.stdout:2/875: truncate d1/d10/d33/d46/f47 454578 0 2024-04-22T23:23:39.437 INFO:tasks.workunit.client.0.smithi067.stdout:2/876: chown d1/d10/d1c/d6d/cc2 109 1 2024-04-22T23:23:39.438 INFO:tasks.workunit.client.0.smithi067.stdout:2/877: write d1/d10/d41/f11c [997208,102318] 0 2024-04-22T23:23:39.442 INFO:tasks.workunit.client.1.smithi134.stdout:3/195: symlink dc/dd/d28/d3b/l4d 0 2024-04-22T23:23:39.442 INFO:tasks.workunit.client.0.smithi067.stdout:2/878: rmdir d1/d42/d5f/d122/d53/d111 0 2024-04-22T23:23:39.443 INFO:tasks.workunit.client.1.smithi134.stdout:2/171: creat d2/dd/d2b/d33/f3a x:0 0 0 2024-04-22T23:23:39.445 INFO:tasks.workunit.client.1.smithi134.stdout:7/192: unlink d4/d10/d2f/f44 0 2024-04-22T23:23:39.446 INFO:tasks.workunit.client.1.smithi134.stdout:4/304: dwrite d4/d13/f59 [0,4194304] 0 2024-04-22T23:23:39.447 INFO:tasks.workunit.client.0.smithi067.stdout:2/879: dwrite d1/d42/d61/dad/ffd [0,4194304] 0 2024-04-22T23:23:39.447 INFO:tasks.workunit.client.1.smithi134.stdout:9/125: mkdir d0/d7/d23/d26 0 2024-04-22T23:23:39.449 INFO:tasks.workunit.client.1.smithi134.stdout:3/196: symlink dc/d2a/d30/l4e 0 2024-04-22T23:23:39.449 INFO:tasks.workunit.client.1.smithi134.stdout:9/126: chown d0/d7/d12 205519 1 2024-04-22T23:23:39.450 INFO:tasks.workunit.client.0.smithi067.stdout:2/880: creat d1/d42/d5f/d122/d68/d7b/dce/f124 x:0 0 0 2024-04-22T23:23:39.452 INFO:tasks.workunit.client.1.smithi134.stdout:3/197: truncate dc/d2a/f23 1111823 0 2024-04-22T23:23:39.453 INFO:tasks.workunit.client.1.smithi134.stdout:6/194: creat d0/d19/de/d12/d2a/d2d/d3a/f4a x:0 0 0 2024-04-22T23:23:39.453 INFO:tasks.workunit.client.0.smithi067.stdout:2/881: stat d1/d42/d5f/d122/d53/l9e 0 2024-04-22T23:23:39.454 INFO:tasks.workunit.client.1.smithi134.stdout:2/172: dwrite d2/d15/d24/d1d/f26 [0,4194304] 0 2024-04-22T23:23:39.454 INFO:tasks.workunit.client.1.smithi134.stdout:7/193: mkdir d4/d4a 0 2024-04-22T23:23:39.455 INFO:tasks.workunit.client.0.smithi067.stdout:2/882: creat d1/d10/d41/f125 x:0 0 0 2024-04-22T23:23:39.459 INFO:tasks.workunit.client.1.smithi134.stdout:7/194: fdatasync d4/d10/d2f/d34/f35 0 2024-04-22T23:23:39.459 INFO:tasks.workunit.client.1.smithi134.stdout:4/305: creat d4/d15/d64/f69 x:0 0 0 2024-04-22T23:23:39.459 INFO:tasks.workunit.client.1.smithi134.stdout:9/127: mkdir d0/d7/d27 0 2024-04-22T23:23:39.461 INFO:tasks.workunit.client.1.smithi134.stdout:2/173: mkdir d2/d15/d24/d3b 0 2024-04-22T23:23:39.465 INFO:tasks.workunit.client.1.smithi134.stdout:6/195: creat d0/d19/db/d35/f4b x:0 0 0 2024-04-22T23:23:39.467 INFO:tasks.workunit.client.1.smithi134.stdout:4/306: creat d4/d15/f6a x:0 0 0 2024-04-22T23:23:39.468 INFO:tasks.workunit.client.1.smithi134.stdout:6/196: chown d0/d19/de/d12/d2a/d2d/f49 13 1 2024-04-22T23:23:39.468 INFO:tasks.workunit.client.1.smithi134.stdout:9/128: symlink d0/d7/dc/l28 0 2024-04-22T23:23:39.470 INFO:tasks.workunit.client.1.smithi134.stdout:2/174: symlink d2/dd/d2b/l3c 0 2024-04-22T23:23:39.470 INFO:tasks.workunit.client.1.smithi134.stdout:6/197: fdatasync d0/d19/de/d2f/f43 0 2024-04-22T23:23:39.472 INFO:tasks.workunit.client.1.smithi134.stdout:7/195: dwrite d4/f7 [0,4194304] 0 2024-04-22T23:23:39.472 INFO:tasks.workunit.client.1.smithi134.stdout:4/307: dread d4/d35/d5e/f4e [0,4194304] 0 2024-04-22T23:23:39.474 INFO:tasks.workunit.client.1.smithi134.stdout:6/198: truncate d0/d19/f39 608940 0 2024-04-22T23:23:39.475 INFO:tasks.workunit.client.1.smithi134.stdout:3/198: getdents dc/d2a 0 2024-04-22T23:23:39.476 INFO:tasks.workunit.client.1.smithi134.stdout:6/199: chown d0/d19/de/d12/c15 50 1 2024-04-22T23:23:39.477 INFO:tasks.workunit.client.1.smithi134.stdout:2/175: mkdir d2/d15/d24/d1d/d3d 0 2024-04-22T23:23:39.479 INFO:tasks.workunit.client.1.smithi134.stdout:5/207: write d9/f19 [470206,23219] 0 2024-04-22T23:23:39.483 INFO:tasks.workunit.client.1.smithi134.stdout:7/196: dwrite d4/d8/f30 [0,4194304] 0 2024-04-22T23:23:39.486 INFO:tasks.workunit.client.1.smithi134.stdout:6/200: creat d0/d19/de/d12/d2a/f4c x:0 0 0 2024-04-22T23:23:39.492 INFO:tasks.workunit.client.1.smithi134.stdout:7/197: dwrite d4/d8/f30 [0,4194304] 0 2024-04-22T23:23:39.496 INFO:tasks.workunit.client.1.smithi134.stdout:7/198: unlink d4/d8/d1d/f45 0 2024-04-22T23:23:39.497 INFO:tasks.workunit.client.1.smithi134.stdout:7/199: write d4/d8/fb [1607115,108002] 0 2024-04-22T23:23:39.498 INFO:tasks.workunit.client.1.smithi134.stdout:7/200: write d4/d8/fb [3147484,99112] 0 2024-04-22T23:23:39.501 INFO:tasks.workunit.client.1.smithi134.stdout:7/201: write d4/d8/fb [177785,7841] 0 2024-04-22T23:23:39.501 INFO:tasks.workunit.client.1.smithi134.stdout:9/129: dread d0/f3 [0,4194304] 0 2024-04-22T23:23:39.504 INFO:tasks.workunit.client.1.smithi134.stdout:7/202: rmdir d4/d8/d3b 39 2024-04-22T23:23:39.510 INFO:tasks.workunit.client.1.smithi134.stdout:7/203: symlink d4/d10/l4b 0 2024-04-22T23:23:39.510 INFO:tasks.workunit.client.0.smithi067.stdout:6/889: sync 2024-04-22T23:23:39.512 INFO:tasks.workunit.client.0.smithi067.stdout:5/966: write d5/d1a/d103/f58 [5059611,52719] 0 2024-04-22T23:23:39.513 INFO:tasks.workunit.client.1.smithi134.stdout:7/204: symlink d4/d8/d2a/l4c 0 2024-04-22T23:23:39.514 INFO:tasks.workunit.client.1.smithi134.stdout:4/308: truncate d4/d15/f26 911456 0 2024-04-22T23:23:39.515 INFO:tasks.workunit.client.0.smithi067.stdout:6/890: mkdir d17/d83/db5/dbc/dd8/d10c/d129 0 2024-04-22T23:23:39.515 INFO:tasks.workunit.client.1.smithi134.stdout:9/130: dwrite d0/f3 [0,4194304] 0 2024-04-22T23:23:39.515 INFO:tasks.workunit.client.1.smithi134.stdout:9/131: stat d0 0 2024-04-22T23:23:39.515 INFO:tasks.workunit.client.0.smithi067.stdout:2/883: symlink d1/d42/d5f/d122/d53/l126 0 2024-04-22T23:23:39.516 INFO:tasks.workunit.client.0.smithi067.stdout:5/967: mkdir d5/d1a/d14f 0 2024-04-22T23:23:39.516 INFO:tasks.workunit.client.1.smithi134.stdout:7/205: write d4/d8/d2a/f2c [249018,110576] 0 2024-04-22T23:23:39.520 INFO:tasks.workunit.client.1.smithi134.stdout:6/201: dread d0/f42 [0,4194304] 0 2024-04-22T23:23:39.526 INFO:tasks.workunit.client.1.smithi134.stdout:9/132: creat d0/d7/d23/d26/f29 x:0 0 0 2024-04-22T23:23:39.527 INFO:tasks.workunit.client.1.smithi134.stdout:9/133: stat d0/d7 0 2024-04-22T23:23:39.528 INFO:tasks.workunit.client.1.smithi134.stdout:2/176: dwrite d2/dd/f37 [0,4194304] 0 2024-04-22T23:23:39.532 INFO:tasks.workunit.client.1.smithi134.stdout:4/309: dwrite d4/f1d [0,4194304] 0 2024-04-22T23:23:39.532 INFO:tasks.workunit.client.1.smithi134.stdout:6/202: rename d0/d19/de/d12/d2a/d2d/l3c to d0/d19/de/d2f/l4d 0 2024-04-22T23:23:39.532 INFO:tasks.workunit.client.1.smithi134.stdout:5/208: dread d9/d11/f1b [4194304,4194304] 0 2024-04-22T23:23:39.532 INFO:tasks.workunit.client.1.smithi134.stdout:7/206: getdents d4/d10/d2f 0 2024-04-22T23:23:39.534 INFO:tasks.workunit.client.1.smithi134.stdout:5/209: chown d9/d1e/c2c 783534409 1 2024-04-22T23:23:39.535 INFO:tasks.workunit.client.1.smithi134.stdout:9/134: dread d0/d7/d12/f1a [0,4194304] 0 2024-04-22T23:23:39.536 INFO:tasks.workunit.client.1.smithi134.stdout:6/203: mkdir d0/d4e 0 2024-04-22T23:23:39.538 INFO:tasks.workunit.client.1.smithi134.stdout:4/310: read d4/f2f [6964621,113063] 0 2024-04-22T23:23:39.541 INFO:tasks.workunit.client.1.smithi134.stdout:9/135: creat d0/d7/d23/f2a x:0 0 0 2024-04-22T23:23:39.543 INFO:tasks.workunit.client.1.smithi134.stdout:6/204: mkdir d0/d4f 0 2024-04-22T23:23:39.545 INFO:tasks.workunit.client.1.smithi134.stdout:6/205: write d0/d19/db/d35/f44 [680982,39892] 0 2024-04-22T23:23:39.551 INFO:tasks.workunit.client.1.smithi134.stdout:5/210: getdents d9/dd/d17/d2a/d3f 0 2024-04-22T23:23:39.553 INFO:tasks.workunit.client.1.smithi134.stdout:9/136: mkdir d0/d7/d23/d2b 0 2024-04-22T23:23:39.554 INFO:tasks.workunit.client.1.smithi134.stdout:4/311: mkdir d4/d6b 0 2024-04-22T23:23:39.555 INFO:tasks.workunit.client.1.smithi134.stdout:2/177: getdents d2/d15/d24 0 2024-04-22T23:23:39.557 INFO:tasks.workunit.client.1.smithi134.stdout:9/137: chown d0/d7/dc/l28 1634 1 2024-04-22T23:23:39.557 INFO:tasks.workunit.client.1.smithi134.stdout:4/312: read - d4/d29/f4d zero size 2024-04-22T23:23:39.558 INFO:tasks.workunit.client.1.smithi134.stdout:9/138: stat d0/d7/f9 0 2024-04-22T23:23:39.560 INFO:tasks.workunit.client.1.smithi134.stdout:9/139: stat d0/d7/d23/f2a 0 2024-04-22T23:23:39.561 INFO:tasks.workunit.client.1.smithi134.stdout:4/313: unlink d4/d13/c4f 0 2024-04-22T23:23:39.562 INFO:tasks.workunit.client.1.smithi134.stdout:5/211: dwrite d9/d1e/f23 [4194304,4194304] 0 2024-04-22T23:23:39.562 INFO:tasks.workunit.client.1.smithi134.stdout:6/206: dread d0/d19/db/d35/f44 [0,4194304] 0 2024-04-22T23:23:39.563 INFO:tasks.workunit.client.1.smithi134.stdout:2/178: creat d2/f3e x:0 0 0 2024-04-22T23:23:39.563 INFO:tasks.workunit.client.1.smithi134.stdout:4/314: readlink d4/dd/l5f 0 2024-04-22T23:23:39.566 INFO:tasks.workunit.client.1.smithi134.stdout:2/179: chown d2/d15/d24/f35 3 1 2024-04-22T23:23:39.568 INFO:tasks.workunit.client.1.smithi134.stdout:9/140: creat d0/d7/f2c x:0 0 0 2024-04-22T23:23:39.569 INFO:tasks.workunit.client.1.smithi134.stdout:4/315: dread d4/d35/d5e/f4e [0,4194304] 0 2024-04-22T23:23:39.570 INFO:tasks.workunit.client.1.smithi134.stdout:5/212: rename d9/d11/f25 to d9/dd/d17/f45 0 2024-04-22T23:23:39.571 INFO:tasks.workunit.client.1.smithi134.stdout:6/207: creat d0/d19/db/d33/f50 x:0 0 0 2024-04-22T23:23:39.573 INFO:tasks.workunit.client.1.smithi134.stdout:2/180: symlink d2/dd/df/d1c/l3f 0 2024-04-22T23:23:39.574 INFO:tasks.workunit.client.1.smithi134.stdout:2/181: fsync d2/dd/fe 0 2024-04-22T23:23:39.575 INFO:tasks.workunit.client.1.smithi134.stdout:5/213: dread d9/d1e/f32 [0,4194304] 0 2024-04-22T23:23:39.576 INFO:tasks.workunit.client.1.smithi134.stdout:4/316: mknod d4/d15/d64/c6c 0 2024-04-22T23:23:39.577 INFO:tasks.workunit.client.0.smithi067.stdout:6/891: dwrite d17/d39/d53/d90/d46/db0/db9/fd2 [0,4194304] 0 2024-04-22T23:23:39.578 INFO:tasks.workunit.client.1.smithi134.stdout:7/207: dread d4/f13 [0,4194304] 0 2024-04-22T23:23:39.580 INFO:tasks.workunit.client.1.smithi134.stdout:7/208: write d4/d8/d2a/f2c [859197,13511] 0 2024-04-22T23:23:39.581 INFO:tasks.workunit.client.1.smithi134.stdout:9/141: mkdir d0/d2d 0 2024-04-22T23:23:39.582 INFO:tasks.workunit.client.1.smithi134.stdout:7/209: chown d4/d10/l39 10984742 1 2024-04-22T23:23:39.583 INFO:tasks.workunit.client.0.smithi067.stdout:2/884: dread d1/d4f/fe4 [0,4194304] 0 2024-04-22T23:23:39.585 INFO:tasks.workunit.client.0.smithi067.stdout:2/885: rmdir d1/d10/d1c/ddc 39 2024-04-22T23:23:39.585 INFO:tasks.workunit.client.1.smithi134.stdout:2/182: unlink d2/d15/d24/d1d/f39 0 2024-04-22T23:23:39.587 INFO:tasks.workunit.client.0.smithi067.stdout:2/886: write d1/d10/d33/d46/f88 [4262598,19138] 0 2024-04-22T23:23:39.587 INFO:tasks.workunit.client.1.smithi134.stdout:2/183: write d2/d15/d24/f27 [1058105,84797] 0 2024-04-22T23:23:39.589 INFO:tasks.workunit.client.1.smithi134.stdout:2/184: write d2/f2c [2607367,36903] 0 2024-04-22T23:23:39.590 INFO:tasks.workunit.client.0.smithi067.stdout:2/887: fsync d1/d10/d33/d46/fea 0 2024-04-22T23:23:39.591 INFO:tasks.workunit.client.0.smithi067.stdout:5/968: creat d5/f150 x:0 0 0 2024-04-22T23:23:39.592 INFO:tasks.workunit.client.1.smithi134.stdout:9/142: dwrite d0/d7/f19 [0,4194304] 0 2024-04-22T23:23:39.593 INFO:tasks.workunit.client.1.smithi134.stdout:9/143: chown d0/d7/d12/f1f 2 1 2024-04-22T23:23:39.594 INFO:tasks.workunit.client.0.smithi067.stdout:6/892: truncate d17/d83/db5/dbc/fe3 3534869 0 2024-04-22T23:23:39.595 INFO:tasks.workunit.client.0.smithi067.stdout:2/888: getdents d1/d4f 0 2024-04-22T23:23:39.596 INFO:tasks.workunit.client.0.smithi067.stdout:6/893: dread - d17/d39/d53/d90/d46/db0/db9/de7/ff2 zero size 2024-04-22T23:23:39.596 INFO:tasks.workunit.client.1.smithi134.stdout:9/144: write d0/f6 [3918802,72671] 0 2024-04-22T23:23:39.597 INFO:tasks.workunit.client.0.smithi067.stdout:2/889: dread - d1/d10/d33/d46/da4/fdb zero size 2024-04-22T23:23:39.599 INFO:tasks.workunit.client.1.smithi134.stdout:9/145: chown d0/d7/f9 284615 1 2024-04-22T23:23:39.600 INFO:tasks.workunit.client.0.smithi067.stdout:2/890: symlink d1/d10/d1c/d10f/l127 0 2024-04-22T23:23:39.602 INFO:tasks.workunit.client.0.smithi067.stdout:6/894: unlink d17/f20 0 2024-04-22T23:23:39.603 INFO:tasks.workunit.client.0.smithi067.stdout:2/891: dread - d1/d42/d5f/d122/d53/dba/fc3 zero size 2024-04-22T23:23:39.603 INFO:tasks.workunit.client.0.smithi067.stdout:6/895: stat d17/d39/d53/d90/d46/f75 0 2024-04-22T23:23:39.604 INFO:tasks.workunit.client.1.smithi134.stdout:6/208: dread d0/d19/f7 [0,4194304] 0 2024-04-22T23:23:39.606 INFO:tasks.workunit.client.1.smithi134.stdout:7/210: dwrite d4/d8/d3b/f48 [0,4194304] 0 2024-04-22T23:23:39.608 INFO:tasks.workunit.client.0.smithi067.stdout:5/969: creat d5/d1a/d103/d10d/f151 x:0 0 0 2024-04-22T23:23:39.610 INFO:tasks.workunit.client.0.smithi067.stdout:5/970: fsync d5/d1a/d103/d4d/da9/f120 0 2024-04-22T23:23:39.610 INFO:tasks.workunit.client.0.smithi067.stdout:6/896: dwrite d17/f26 [0,4194304] 0 2024-04-22T23:23:39.611 INFO:tasks.workunit.client.1.smithi134.stdout:2/185: creat d2/d15/d24/f40 x:0 0 0 2024-04-22T23:23:39.615 INFO:tasks.workunit.client.0.smithi067.stdout:6/897: mknod d17/d39/d3a/d8d/c12a 0 2024-04-22T23:23:39.616 INFO:tasks.workunit.client.1.smithi134.stdout:5/214: rmdir d9/dd/d17/d2a/d3f 0 2024-04-22T23:23:39.616 INFO:tasks.workunit.client.1.smithi134.stdout:5/215: fsync f8 0 2024-04-22T23:23:39.617 INFO:tasks.workunit.client.1.smithi134.stdout:6/209: chown d0/d19/de/c23 1313187268 1 2024-04-22T23:23:39.618 INFO:tasks.workunit.client.1.smithi134.stdout:6/210: fdatasync d0/d19/de/d2f/f32 0 2024-04-22T23:23:39.619 INFO:tasks.workunit.client.0.smithi067.stdout:6/898: mkdir d17/d39/d53/d90/d46/d62/d88/d12b 0 2024-04-22T23:23:39.620 INFO:tasks.workunit.client.1.smithi134.stdout:9/146: getdents d0/d7/d23/d2b 0 2024-04-22T23:23:39.620 INFO:tasks.workunit.client.0.smithi067.stdout:2/892: write d1/d42/d5f/d122/d53/dba/fc3 [792599,44990] 0 2024-04-22T23:23:39.623 INFO:tasks.workunit.client.0.smithi067.stdout:2/893: mknod d1/d42/d5f/d122/d53/d9a/d10a/c128 0 2024-04-22T23:23:39.625 INFO:tasks.workunit.client.0.smithi067.stdout:6/899: link d17/d83/d9b/ddd/c11a d17/d39/d3a/da9/c12c 0 2024-04-22T23:23:39.627 INFO:tasks.workunit.client.0.smithi067.stdout:8/909: sync 2024-04-22T23:23:39.637 INFO:tasks.workunit.client.0.smithi067.stdout:2/894: creat d1/d10/d1c/f129 x:0 0 0 2024-04-22T23:23:39.641 INFO:tasks.workunit.client.0.smithi067.stdout:8/910: rename d1/dd/d80/db9/d110/d62/ddf/ded/lf7 to d1/dd/d80/db9/de3/db8/dd7/l128 0 2024-04-22T23:23:39.642 INFO:tasks.workunit.client.0.smithi067.stdout:6/900: dwrite d17/d39/f4b [0,4194304] 0 2024-04-22T23:23:39.654 INFO:tasks.workunit.client.0.smithi067.stdout:6/901: truncate d17/d39/d53/f74 7678248 0 2024-04-22T23:23:39.659 INFO:tasks.workunit.client.0.smithi067.stdout:2/895: dwrite d1/d10/d1c/fc6 [0,4194304] 0 2024-04-22T23:23:39.660 INFO:tasks.workunit.client.0.smithi067.stdout:8/911: dwrite d1/d3/f38 [4194304,4194304] 0 2024-04-22T23:23:39.660 INFO:tasks.workunit.client.1.smithi134.stdout:2/186: link d2/d15/d24/f27 d2/dd/df/f41 0 2024-04-22T23:23:39.662 INFO:tasks.workunit.client.1.smithi134.stdout:9/147: write d0/f6 [897131,95590] 0 2024-04-22T23:23:39.662 INFO:tasks.workunit.client.0.smithi067.stdout:8/912: chown d1/dd/d80/d6f/c47 32247 1 2024-04-22T23:23:39.663 INFO:tasks.workunit.client.0.smithi067.stdout:8/913: fdatasync d1/d3/f2a 0 2024-04-22T23:23:39.664 INFO:tasks.workunit.client.1.smithi134.stdout:6/211: dread d0/d19/f7 [0,4194304] 0 2024-04-22T23:23:39.666 INFO:tasks.workunit.client.1.smithi134.stdout:2/187: mkdir d2/d42 0 2024-04-22T23:23:39.666 INFO:tasks.workunit.client.0.smithi067.stdout:2/896: rename d1/d42/c8e to d1/d4f/c12a 0 2024-04-22T23:23:39.667 INFO:tasks.workunit.client.1.smithi134.stdout:9/148: symlink d0/d7/d23/d26/l2e 0 2024-04-22T23:23:39.667 INFO:tasks.workunit.client.1.smithi134.stdout:5/216: getdents d9/d11/d34/d3b 0 2024-04-22T23:23:39.669 INFO:tasks.workunit.client.0.smithi067.stdout:2/897: creat d1/d10/d33/d46/da4/dde/f12b x:0 0 0 2024-04-22T23:23:39.671 INFO:tasks.workunit.client.0.smithi067.stdout:2/898: write d1/d10/d33/fa5 [666297,102166] 0 2024-04-22T23:23:39.673 INFO:tasks.workunit.client.1.smithi134.stdout:2/188: mknod d2/dd/c43 0 2024-04-22T23:23:39.674 INFO:tasks.workunit.client.0.smithi067.stdout:2/899: rmdir d1/d10/d33 39 2024-04-22T23:23:39.674 INFO:tasks.workunit.client.1.smithi134.stdout:2/189: chown d2/dd/l38 5657008 1 2024-04-22T23:23:39.678 INFO:tasks.workunit.client.0.smithi067.stdout:2/900: creat d1/d42/d61/db9/f12c x:0 0 0 2024-04-22T23:23:39.678 INFO:tasks.workunit.client.1.smithi134.stdout:2/190: write d2/d15/d24/f35 [520036,62063] 0 2024-04-22T23:23:39.678 INFO:tasks.workunit.client.1.smithi134.stdout:9/149: symlink d0/d7/d27/l2f 0 2024-04-22T23:23:39.680 INFO:tasks.workunit.client.0.smithi067.stdout:2/901: fdatasync d1/d10/f30 0 2024-04-22T23:23:39.680 INFO:tasks.workunit.client.1.smithi134.stdout:5/217: dwrite d9/d1e/f35 [0,4194304] 0 2024-04-22T23:23:39.681 INFO:tasks.workunit.client.0.smithi067.stdout:6/902: dread d17/d39/d3a/f50 [0,4194304] 0 2024-04-22T23:23:39.683 INFO:tasks.workunit.client.1.smithi134.stdout:2/191: stat d2/dd/df/c10 0 2024-04-22T23:23:39.683 INFO:tasks.workunit.client.0.smithi067.stdout:2/902: mkdir d1/d10/d110/d12d 0 2024-04-22T23:23:39.684 INFO:tasks.workunit.client.0.smithi067.stdout:2/903: dread - d1/d10/d1c/f123 zero size 2024-04-22T23:23:39.687 INFO:tasks.workunit.client.0.smithi067.stdout:2/904: creat d1/d10/d110/d12d/f12e x:0 0 0 2024-04-22T23:23:39.687 INFO:tasks.workunit.client.1.smithi134.stdout:7/211: dread d4/fa [0,4194304] 0 2024-04-22T23:23:39.690 INFO:tasks.workunit.client.1.smithi134.stdout:9/150: creat d0/d7/d12/f30 x:0 0 0 2024-04-22T23:23:39.691 INFO:tasks.workunit.client.1.smithi134.stdout:7/212: dread - d4/d8/d1d/f3c zero size 2024-04-22T23:23:39.692 INFO:tasks.workunit.client.1.smithi134.stdout:9/151: fdatasync d0/d7/fd 0 2024-04-22T23:23:39.693 INFO:tasks.workunit.client.1.smithi134.stdout:4/317: dwrite d4/d13/f65 [0,4194304] 0 2024-04-22T23:23:39.693 INFO:tasks.workunit.client.1.smithi134.stdout:5/218: dwrite f6 [4194304,4194304] 0 2024-04-22T23:23:39.695 INFO:tasks.workunit.client.1.smithi134.stdout:4/318: write d4/d35/f62 [3669787,59291] 0 2024-04-22T23:23:39.696 INFO:tasks.workunit.client.1.smithi134.stdout:7/213: creat d4/d8/d1d/d29/f4d x:0 0 0 2024-04-22T23:23:39.696 INFO:tasks.workunit.client.0.smithi067.stdout:2/905: dread d1/d42/d5f/d122/d68/d7b/fb7 [0,4194304] 0 2024-04-22T23:23:39.698 INFO:tasks.workunit.client.1.smithi134.stdout:9/152: creat d0/d7/d12/d22/f31 x:0 0 0 2024-04-22T23:23:39.699 INFO:tasks.workunit.client.0.smithi067.stdout:6/903: dread d17/d39/d3a/f61 [0,4194304] 0 2024-04-22T23:23:39.702 INFO:tasks.workunit.client.1.smithi134.stdout:4/319: rename d4/d15/f61 to d4/dd/f6d 0 2024-04-22T23:23:39.704 INFO:tasks.workunit.client.1.smithi134.stdout:4/320: fsync d4/d15/d2a/f2d 0 2024-04-22T23:23:39.704 INFO:tasks.workunit.client.1.smithi134.stdout:7/214: symlink d4/d8/d2a/d42/l4e 0 2024-04-22T23:23:39.705 INFO:tasks.workunit.client.0.smithi067.stdout:2/906: rename d1/d10/d1c/d6d/c83 to d1/d10/d1c/d11e/c12f 0 2024-04-22T23:23:39.705 INFO:tasks.workunit.client.1.smithi134.stdout:4/321: stat d4/d15/f6a 0 2024-04-22T23:23:39.705 INFO:tasks.workunit.client.0.smithi067.stdout:6/904: read d17/d39/d53/f93 [773343,22409] 0 2024-04-22T23:23:39.706 INFO:tasks.workunit.client.1.smithi134.stdout:4/322: fdatasync d4/d15/d64/f69 0 2024-04-22T23:23:39.707 INFO:tasks.workunit.client.1.smithi134.stdout:7/215: write d4/d10/d2f/f32 [1899577,66816] 0 2024-04-22T23:23:39.707 INFO:tasks.workunit.client.0.smithi067.stdout:2/907: fdatasync d1/d42/d61/daa/fca 0 2024-04-22T23:23:39.708 INFO:tasks.workunit.client.0.smithi067.stdout:6/905: write d17/d83/d9b/ddd/f122 [5008549,64616] 0 2024-04-22T23:23:39.708 INFO:tasks.workunit.client.1.smithi134.stdout:7/216: write d4/d8/fb [1385366,38867] 0 2024-04-22T23:23:39.710 INFO:tasks.workunit.client.1.smithi134.stdout:4/323: mkdir d4/d15/d2a/d6e 0 2024-04-22T23:23:39.711 INFO:tasks.workunit.client.1.smithi134.stdout:7/217: write d4/d8/f21 [163263,36867] 0 2024-04-22T23:23:39.712 INFO:tasks.workunit.client.1.smithi134.stdout:6/212: write d0/f16 [1499397,13292] 0 2024-04-22T23:23:39.713 INFO:tasks.workunit.client.1.smithi134.stdout:4/324: creat d4/d15/f6f x:0 0 0 2024-04-22T23:23:39.715 INFO:tasks.workunit.client.0.smithi067.stdout:2/908: dwrite d1/d10/f86 [4194304,4194304] 0 2024-04-22T23:23:39.716 INFO:tasks.workunit.client.1.smithi134.stdout:9/153: getdents d0/d7/d27 0 2024-04-22T23:23:39.719 INFO:tasks.workunit.client.1.smithi134.stdout:6/213: rename d0/d4e to d0/d4f/d51 0 2024-04-22T23:23:39.720 INFO:tasks.workunit.client.1.smithi134.stdout:6/214: write d0/f10 [2200834,44015] 0 2024-04-22T23:23:39.720 INFO:tasks.workunit.client.0.smithi067.stdout:2/909: rename d1/d10/d41/fe1 to d1/d10/d41/f130 0 2024-04-22T23:23:39.722 INFO:tasks.workunit.client.0.smithi067.stdout:2/910: chown d1/d10/d1c/d6d/f4c 1506417 1 2024-04-22T23:23:39.723 INFO:tasks.workunit.client.0.smithi067.stdout:2/911: write d1/d42/d5f/d122/fff [202922,36475] 0 2024-04-22T23:23:39.727 INFO:tasks.workunit.client.1.smithi134.stdout:9/154: rename d0/f6 to d0/d7/dc/f32 0 2024-04-22T23:23:39.728 INFO:tasks.workunit.client.0.smithi067.stdout:2/912: dread d1/d42/f56 [0,4194304] 0 2024-04-22T23:23:39.729 INFO:tasks.workunit.client.1.smithi134.stdout:7/218: link d4/d8/d41/f43 d4/d8/d2a/f4f 0 2024-04-22T23:23:39.729 INFO:tasks.workunit.client.0.smithi067.stdout:6/906: dread d17/d39/d3a/f6d [0,4194304] 0 2024-04-22T23:23:39.729 INFO:tasks.workunit.client.0.smithi067.stdout:2/913: write d1/d12/ff2 [329171,56572] 0 2024-04-22T23:23:39.732 INFO:tasks.workunit.client.0.smithi067.stdout:6/907: truncate d17/d39/d53/d90/d46/db0/db9/ff3 646524 0 2024-04-22T23:23:39.734 INFO:tasks.workunit.client.0.smithi067.stdout:6/908: creat d17/d39/d53/f12d x:0 0 0 2024-04-22T23:23:39.736 INFO:tasks.workunit.client.1.smithi134.stdout:4/325: truncate d4/d35/d5e/f60 3795371 0 2024-04-22T23:23:39.736 INFO:tasks.workunit.client.0.smithi067.stdout:6/909: mkdir d17/d39/d53/d90/d46/db0/db9/de7/d12e 0 2024-04-22T23:23:39.736 INFO:tasks.workunit.client.1.smithi134.stdout:9/155: dwrite d0/d7/d23/f2a [0,4194304] 0 2024-04-22T23:23:39.737 INFO:tasks.workunit.client.0.smithi067.stdout:6/910: chown d17/d83/fde 26304 1 2024-04-22T23:23:39.738 INFO:tasks.workunit.client.0.smithi067.stdout:6/911: symlink d17/d39/d53/d90/d46/db0/db9/de7/l12f 0 2024-04-22T23:23:39.739 INFO:tasks.workunit.client.1.smithi134.stdout:4/326: write d4/d7/d32/d3a/f51 [3463444,87630] 0 2024-04-22T23:23:39.739 INFO:tasks.workunit.client.1.smithi134.stdout:7/219: dwrite d4/f7 [4194304,4194304] 0 2024-04-22T23:23:39.741 INFO:tasks.workunit.client.1.smithi134.stdout:6/215: creat d0/d19/de/d12/f52 x:0 0 0 2024-04-22T23:23:39.748 INFO:tasks.workunit.client.1.smithi134.stdout:7/220: read d4/d8/f21 [153609,79487] 0 2024-04-22T23:23:39.751 INFO:tasks.workunit.client.1.smithi134.stdout:6/216: dwrite d0/d19/f39 [0,4194304] 0 2024-04-22T23:23:39.752 INFO:tasks.workunit.client.0.smithi067.stdout:2/914: dread d1/d10/d33/f5b [0,4194304] 0 2024-04-22T23:23:39.754 INFO:tasks.workunit.client.1.smithi134.stdout:6/217: fdatasync d0/f1a 0 2024-04-22T23:23:39.755 INFO:tasks.workunit.client.0.smithi067.stdout:2/915: unlink d1/fc1 0 2024-04-22T23:23:39.756 INFO:tasks.workunit.client.1.smithi134.stdout:7/221: rename d4/d10/d2f/d34/f35 to d4/d8/d3b/f50 0 2024-04-22T23:23:39.756 INFO:tasks.workunit.client.1.smithi134.stdout:4/327: write d4/d7/f10 [6912488,23287] 0 2024-04-22T23:23:39.758 INFO:tasks.workunit.client.0.smithi067.stdout:2/916: mknod d1/d42/d61/dad/c131 0 2024-04-22T23:23:39.758 INFO:tasks.workunit.client.1.smithi134.stdout:4/328: write d4/d35/d5e/f4a [97294,50673] 0 2024-04-22T23:23:39.760 INFO:tasks.workunit.client.1.smithi134.stdout:7/222: mknod d4/d10/d2f/c51 0 2024-04-22T23:23:39.761 INFO:tasks.workunit.client.0.smithi067.stdout:2/917: getdents d1/d19/d6f 0 2024-04-22T23:23:39.764 INFO:tasks.workunit.client.0.smithi067.stdout:2/918: creat d1/d42/d5f/d122/d68/d7b/f132 x:0 0 0 2024-04-22T23:23:39.765 INFO:tasks.workunit.client.0.smithi067.stdout:2/919: readlink d1/d12/l29 0 2024-04-22T23:23:39.765 INFO:tasks.workunit.client.0.smithi067.stdout:6/912: dread d17/d83/d9b/ddd/f122 [0,4194304] 0 2024-04-22T23:23:39.765 INFO:tasks.workunit.client.1.smithi134.stdout:4/329: link d4/dd/l5f d4/dd/l70 0 2024-04-22T23:23:39.766 INFO:tasks.workunit.client.1.smithi134.stdout:6/218: dwrite d0/d19/de/d2f/f32 [0,4194304] 0 2024-04-22T23:23:39.767 INFO:tasks.workunit.client.1.smithi134.stdout:6/219: read d0/f42 [3135503,114326] 0 2024-04-22T23:23:39.767 INFO:tasks.workunit.client.0.smithi067.stdout:2/920: creat d1/d10/d33/d46/da4/f133 x:0 0 0 2024-04-22T23:23:39.768 INFO:tasks.workunit.client.1.smithi134.stdout:7/223: dread d4/d10/f23 [0,4194304] 0 2024-04-22T23:23:39.768 INFO:tasks.workunit.client.0.smithi067.stdout:6/913: creat d17/d39/d3a/d8d/f130 x:0 0 0 2024-04-22T23:23:39.769 INFO:tasks.workunit.client.1.smithi134.stdout:6/220: creat d0/d4f/f53 x:0 0 0 2024-04-22T23:23:39.771 INFO:tasks.workunit.client.1.smithi134.stdout:7/224: mknod d4/d4a/c52 0 2024-04-22T23:23:39.771 INFO:tasks.workunit.client.0.smithi067.stdout:6/914: creat d17/d83/db5/dbc/dd8/f131 x:0 0 0 2024-04-22T23:23:39.771 INFO:tasks.workunit.client.1.smithi134.stdout:6/221: symlink d0/d19/de/d2f/l54 0 2024-04-22T23:23:39.773 INFO:tasks.workunit.client.1.smithi134.stdout:4/330: read d4/dd/f16 [3662293,106304] 0 2024-04-22T23:23:39.773 INFO:tasks.workunit.client.1.smithi134.stdout:6/222: creat d0/f55 x:0 0 0 2024-04-22T23:23:39.775 INFO:tasks.workunit.client.0.smithi067.stdout:6/915: mknod d17/d83/db5/dbb/c132 0 2024-04-22T23:23:39.777 INFO:tasks.workunit.client.1.smithi134.stdout:4/331: mknod d4/d35/d5e/d25/c71 0 2024-04-22T23:23:39.778 INFO:tasks.workunit.client.0.smithi067.stdout:6/916: getdents d17/d39/d3a 0 2024-04-22T23:23:39.779 INFO:tasks.workunit.client.1.smithi134.stdout:7/225: dwrite d4/f2d [0,4194304] 0 2024-04-22T23:23:39.779 INFO:tasks.workunit.client.0.smithi067.stdout:6/917: rmdir d17/d83/db5/dbc 39 2024-04-22T23:23:39.780 INFO:tasks.workunit.client.1.smithi134.stdout:7/226: chown d4/d10/d2f/d37 10 1 2024-04-22T23:23:39.781 INFO:tasks.workunit.client.1.smithi134.stdout:4/332: readlink d4/dd/l70 0 2024-04-22T23:23:39.783 INFO:tasks.workunit.client.0.smithi067.stdout:6/918: dread d17/f24 [0,4194304] 0 2024-04-22T23:23:39.784 INFO:tasks.workunit.client.1.smithi134.stdout:6/223: dwrite d0/d19/de/d12/d2a/d2d/d3a/f4a [0,4194304] 0 2024-04-22T23:23:39.784 INFO:tasks.workunit.client.1.smithi134.stdout:7/227: mknod d4/d8/d3b/c53 0 2024-04-22T23:23:39.785 INFO:tasks.workunit.client.0.smithi067.stdout:6/919: truncate d17/d83/fad 940583 0 2024-04-22T23:23:39.786 INFO:tasks.workunit.client.1.smithi134.stdout:6/224: write d0/d19/de/d2f/f32 [1363608,117020] 0 2024-04-22T23:23:39.786 INFO:tasks.workunit.client.1.smithi134.stdout:7/228: mkdir d4/d10/d2f/d34/d54 0 2024-04-22T23:23:39.788 INFO:tasks.workunit.client.1.smithi134.stdout:7/229: write d4/d10/f1c [1000802,47187] 0 2024-04-22T23:23:39.788 INFO:tasks.workunit.client.1.smithi134.stdout:6/225: dread - d0/d19/de/d12/d2a/f4c zero size 2024-04-22T23:23:39.790 INFO:tasks.workunit.client.1.smithi134.stdout:6/226: truncate d0/d19/db/d33/f50 226020 0 2024-04-22T23:23:39.792 INFO:tasks.workunit.client.1.smithi134.stdout:7/230: rename d4/d10/d2f/f32 to d4/d8/d1d/d29/f55 0 2024-04-22T23:23:39.795 INFO:tasks.workunit.client.1.smithi134.stdout:7/231: rename d4/d8/d1d/f3c to d4/d8/d1d/f56 0 2024-04-22T23:23:39.798 INFO:tasks.workunit.client.1.smithi134.stdout:7/232: unlink d4/d8/d1d/d29/f4d 0 2024-04-22T23:23:39.799 INFO:tasks.workunit.client.1.smithi134.stdout:6/227: link d0/d19/de/d12/d2a/l45 d0/d19/de/l56 0 2024-04-22T23:23:39.800 INFO:tasks.workunit.client.1.smithi134.stdout:6/228: chown d0/f8 108 1 2024-04-22T23:23:39.801 INFO:tasks.workunit.client.1.smithi134.stdout:4/333: dread d4/dd/f6d [0,4194304] 0 2024-04-22T23:23:39.808 INFO:tasks.workunit.client.0.smithi067.stdout:3/917: sync 2024-04-22T23:23:39.808 INFO:tasks.workunit.client.0.smithi067.stdout:9/854: sync 2024-04-22T23:23:39.808 INFO:tasks.workunit.client.0.smithi067.stdout:7/903: sync 2024-04-22T23:23:39.808 INFO:tasks.workunit.client.0.smithi067.stdout:1/872: sync 2024-04-22T23:23:39.809 INFO:tasks.workunit.client.1.smithi134.stdout:4/334: dwrite d4/d35/f57 [0,4194304] 0 2024-04-22T23:23:39.811 INFO:tasks.workunit.client.0.smithi067.stdout:3/918: unlink d3/d4/d10/d1f/d3c/f67 0 2024-04-22T23:23:39.811 INFO:tasks.workunit.client.1.smithi134.stdout:4/335: mknod d4/d35/d5e/c72 0 2024-04-22T23:23:39.813 INFO:tasks.workunit.client.0.smithi067.stdout:1/873: creat d8/d9/d102/f119 x:0 0 0 2024-04-22T23:23:39.817 INFO:tasks.workunit.client.0.smithi067.stdout:7/904: rename d10/d14/d1d/d26/d2c/l4b to d10/d14/d1d/d26/d2c/d41/db0/ddd/l152 0 2024-04-22T23:23:39.818 INFO:tasks.workunit.client.0.smithi067.stdout:3/919: dwrite d3/d4/d28/d40/dfd/dfe/f27 [0,4194304] 0 2024-04-22T23:23:39.820 INFO:tasks.workunit.client.0.smithi067.stdout:1/874: creat d8/d9/d2d/d4d/d48/f11a x:0 0 0 2024-04-22T23:23:39.826 INFO:tasks.workunit.client.1.smithi134.stdout:4/336: dread d4/f6 [0,4194304] 0 2024-04-22T23:23:39.826 INFO:tasks.workunit.client.0.smithi067.stdout:7/905: dwrite d10/d14/d1d/d26/d2c/d41/fa1 [0,4194304] 0 2024-04-22T23:23:39.826 INFO:tasks.workunit.client.1.smithi134.stdout:4/337: write d4/d29/f4d [1034651,128894] 0 2024-04-22T23:23:39.828 INFO:tasks.workunit.client.1.smithi134.stdout:4/338: creat d4/d15/d64/f73 x:0 0 0 2024-04-22T23:23:39.830 INFO:tasks.workunit.client.1.smithi134.stdout:4/339: write d4/d15/d64/f73 [1015139,121554] 0 2024-04-22T23:23:39.832 INFO:tasks.workunit.client.1.smithi134.stdout:4/340: creat d4/d13/f74 x:0 0 0 2024-04-22T23:23:39.832 INFO:tasks.workunit.client.1.smithi134.stdout:4/341: dread - d4/d15/d64/f69 zero size 2024-04-22T23:23:39.833 INFO:tasks.workunit.client.0.smithi067.stdout:7/906: dwrite d10/d14/d1d/d26/d2c/d41/db0/ff0 [0,4194304] 0 2024-04-22T23:23:39.834 INFO:tasks.workunit.client.1.smithi134.stdout:4/342: write d4/d15/d64/f73 [230622,104440] 0 2024-04-22T23:23:39.837 INFO:tasks.workunit.client.0.smithi067.stdout:7/907: read d10/d47/d49/f54 [2240990,53094] 0 2024-04-22T23:23:39.838 INFO:tasks.workunit.client.1.smithi134.stdout:4/343: creat d4/d15/d2a/f75 x:0 0 0 2024-04-22T23:23:39.841 INFO:tasks.workunit.client.0.smithi067.stdout:7/908: truncate d10/f151 451476 0 2024-04-22T23:23:39.841 INFO:tasks.workunit.client.1.smithi134.stdout:4/344: link d4/d35/d5e/c72 d4/dd/c76 0 2024-04-22T23:23:39.842 INFO:tasks.workunit.client.0.smithi067.stdout:5/971: sync 2024-04-22T23:23:39.842 INFO:tasks.workunit.client.0.smithi067.stdout:8/914: sync 2024-04-22T23:23:39.843 INFO:tasks.workunit.client.1.smithi134.stdout:4/345: getdents d4/d7/d32/d3a 0 2024-04-22T23:23:39.845 INFO:tasks.workunit.client.0.smithi067.stdout:8/915: creat d1/dd/d80/f129 x:0 0 0 2024-04-22T23:23:39.845 INFO:tasks.workunit.client.1.smithi134.stdout:4/346: creat d4/d29/f77 x:0 0 0 2024-04-22T23:23:39.845 INFO:tasks.workunit.client.0.smithi067.stdout:7/909: dread d10/d14/d1d/d26/d2c/d41/fa1 [0,4194304] 0 2024-04-22T23:23:39.846 INFO:tasks.workunit.client.0.smithi067.stdout:8/916: fsync d1/dd/d80/db9/d110/d62/fc8 0 2024-04-22T23:23:39.851 INFO:tasks.workunit.client.0.smithi067.stdout:8/917: dwrite d1/dd/f53 [0,4194304] 0 2024-04-22T23:23:39.852 INFO:tasks.workunit.client.0.smithi067.stdout:8/918: readlink d1/d41/d46/l52 0 2024-04-22T23:23:39.852 INFO:tasks.workunit.client.0.smithi067.stdout:8/919: fdatasync d1/fc0 0 2024-04-22T23:23:39.870 INFO:tasks.workunit.client.0.smithi067.stdout:8/920: dread d1/dd/d80/f8d [0,4194304] 0 2024-04-22T23:23:39.872 INFO:tasks.workunit.client.0.smithi067.stdout:8/921: mknod d1/dd/d80/d6f/dfc/de8/c12a 0 2024-04-22T23:23:39.875 INFO:tasks.workunit.client.0.smithi067.stdout:5/972: symlink d5/d3e/d90/l152 0 2024-04-22T23:23:39.876 INFO:tasks.workunit.client.1.smithi134.stdout:4/347: symlink d4/d15/d2a/d5d/l78 0 2024-04-22T23:23:39.885 INFO:tasks.workunit.client.1.smithi134.stdout:4/348: dwrite d4/d15/f67 [0,4194304] 0 2024-04-22T23:23:39.892 INFO:tasks.workunit.client.1.smithi134.stdout:4/349: mknod d4/d15/d64/c79 0 2024-04-22T23:23:39.903 INFO:tasks.workunit.client.1.smithi134.stdout:6/229: truncate d0/f1a 1393696 0 2024-04-22T23:23:39.904 INFO:tasks.workunit.client.1.smithi134.stdout:6/230: write d0/d19/de/d12/d2a/f4c [503103,75666] 0 2024-04-22T23:23:39.905 INFO:tasks.workunit.client.1.smithi134.stdout:6/231: dread - d0/d4f/f53 zero size 2024-04-22T23:23:39.906 INFO:tasks.workunit.client.0.smithi067.stdout:2/921: write d1/d42/d5f/d122/f36 [1611101,106824] 0 2024-04-22T23:23:39.908 INFO:tasks.workunit.client.0.smithi067.stdout:2/922: fdatasync d1/d19/d6f/ff1 0 2024-04-22T23:23:39.908 INFO:tasks.workunit.client.1.smithi134.stdout:6/232: mknod d0/d19/de/d12/d2a/d2d/d3a/c57 0 2024-04-22T23:23:39.909 INFO:tasks.workunit.client.0.smithi067.stdout:2/923: chown d1/d10/d1c/d6d 127236 1 2024-04-22T23:23:39.910 INFO:tasks.workunit.client.1.smithi134.stdout:6/233: chown d0/d19/de/d12/d29/c34 7817 1 2024-04-22T23:23:39.912 INFO:tasks.workunit.client.1.smithi134.stdout:6/234: read d0/d19/f9 [3177520,5094] 0 2024-04-22T23:23:39.912 INFO:tasks.workunit.client.0.smithi067.stdout:2/924: fsync d1/d4f/f2a 0 2024-04-22T23:23:39.915 INFO:tasks.workunit.client.0.smithi067.stdout:3/920: truncate d3/d4/dec/f109 3531767 0 2024-04-22T23:23:39.916 INFO:tasks.workunit.client.0.smithi067.stdout:2/925: rmdir d1/d10 39 2024-04-22T23:23:39.916 INFO:tasks.workunit.client.1.smithi134.stdout:4/350: dread d4/dd/f16 [0,4194304] 0 2024-04-22T23:23:39.916 INFO:tasks.workunit.client.0.smithi067.stdout:6/920: dwrite d17/d83/db5/fe4 [0,4194304] 0 2024-04-22T23:23:39.918 INFO:tasks.workunit.client.1.smithi134.stdout:6/235: rename d0/d19/db/f37 to d0/d19/db/d35/f58 0 2024-04-22T23:23:39.918 INFO:tasks.workunit.client.0.smithi067.stdout:2/926: chown d1/d12/f4e 12054 1 2024-04-22T23:23:39.919 INFO:tasks.workunit.client.1.smithi134.stdout:6/236: chown d0/f42 0 1 2024-04-22T23:23:39.919 INFO:tasks.workunit.client.0.smithi067.stdout:9/855: dwrite d5/dd/d18/d39/d3e/f82 [0,4194304] 0 2024-04-22T23:23:39.923 INFO:tasks.workunit.client.1.smithi134.stdout:7/233: truncate d4/d8/f21 125483 0 2024-04-22T23:23:39.925 INFO:tasks.workunit.client.1.smithi134.stdout:4/351: truncate d4/d15/f26 1436909 0 2024-04-22T23:23:39.927 INFO:tasks.workunit.client.1.smithi134.stdout:4/352: dread - d4/d29/f77 zero size 2024-04-22T23:23:39.927 INFO:tasks.workunit.client.0.smithi067.stdout:8/922: dread d1/d3/d10/fbe [0,4194304] 0 2024-04-22T23:23:39.928 INFO:tasks.workunit.client.0.smithi067.stdout:6/921: link d17/d83/fda d17/d39/d53/d90/d46/d62/d88/f133 0 2024-04-22T23:23:39.931 INFO:tasks.workunit.client.0.smithi067.stdout:3/921: dwrite d3/d4/d28/d40/dfd/dfe/f69 [0,4194304] 0 2024-04-22T23:23:39.933 INFO:tasks.workunit.client.1.smithi134.stdout:4/353: mkdir d4/d7a 0 2024-04-22T23:23:39.934 INFO:tasks.workunit.client.0.smithi067.stdout:1/875: truncate d8/d12/d36/fba 2500224 0 2024-04-22T23:23:39.934 INFO:tasks.workunit.client.0.smithi067.stdout:8/923: truncate d1/dd/d80/d6f/f11e 685449 0 2024-04-22T23:23:39.935 INFO:tasks.workunit.client.0.smithi067.stdout:9/856: link d5/dd/d18/d3c/d40/f6b d5/dd/d18/d39/d8f/f11a 0 2024-04-22T23:23:39.936 INFO:tasks.workunit.client.0.smithi067.stdout:1/876: truncate d8/f1d 1084491 0 2024-04-22T23:23:39.937 INFO:tasks.workunit.client.0.smithi067.stdout:2/927: dwrite d1/d42/d5f/d122/d68/d7b/fb7 [0,4194304] 0 2024-04-22T23:23:39.938 INFO:tasks.workunit.client.0.smithi067.stdout:8/924: write d1/d41/d46/fc2 [705615,58878] 0 2024-04-22T23:23:39.939 INFO:tasks.workunit.client.0.smithi067.stdout:1/877: chown d8/d9/de0/d60/cfb 0 1 2024-04-22T23:23:39.941 INFO:tasks.workunit.client.0.smithi067.stdout:7/910: unlink d10/f151 0 2024-04-22T23:23:39.941 INFO:tasks.workunit.client.0.smithi067.stdout:2/928: write d1/d42/d61/daa/fca [575016,36957] 0 2024-04-22T23:23:39.942 INFO:tasks.workunit.client.0.smithi067.stdout:1/878: stat d8/d12/d40/f44 0 2024-04-22T23:23:39.946 INFO:tasks.workunit.client.0.smithi067.stdout:9/857: creat d5/dd/d18/d39/d3e/d3f/d63/f11b x:0 0 0 2024-04-22T23:23:39.947 INFO:tasks.workunit.client.1.smithi134.stdout:4/354: getdents d4/d15/d64 0 2024-04-22T23:23:39.948 INFO:tasks.workunit.client.1.smithi134.stdout:4/355: rmdir d4/d7/d32 39 2024-04-22T23:23:39.950 INFO:tasks.workunit.client.1.smithi134.stdout:4/356: creat d4/d15/f7b x:0 0 0 2024-04-22T23:23:39.952 INFO:tasks.workunit.client.0.smithi067.stdout:8/925: creat d1/dd/d80/db9/de3/db8/f12b x:0 0 0 2024-04-22T23:23:39.955 INFO:tasks.workunit.client.0.smithi067.stdout:7/911: dwrite d10/d14/d1d/f76 [0,4194304] 0 2024-04-22T23:23:39.955 INFO:tasks.workunit.client.0.smithi067.stdout:3/922: mkdir d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d51/d13a/d13b 0 2024-04-22T23:23:39.959 INFO:tasks.workunit.client.1.smithi134.stdout:4/357: dread d4/f42 [0,4194304] 0 2024-04-22T23:23:39.962 INFO:tasks.workunit.client.1.smithi134.stdout:4/358: creat d4/d15/d2a/d6e/f7c x:0 0 0 2024-04-22T23:23:39.963 INFO:tasks.workunit.client.0.smithi067.stdout:2/929: symlink d1/d10/d1c/dd9/d11f/l134 0 2024-04-22T23:23:39.963 INFO:tasks.workunit.client.0.smithi067.stdout:5/973: dwrite d5/db/d35/fea [0,4194304] 0 2024-04-22T23:23:39.964 INFO:tasks.workunit.client.1.smithi134.stdout:6/237: truncate d0/f16 189834 0 2024-04-22T23:23:39.965 INFO:tasks.workunit.client.0.smithi067.stdout:3/923: mkdir d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d90/da2/d13c 0 2024-04-22T23:23:39.967 INFO:tasks.workunit.client.1.smithi134.stdout:7/234: truncate d4/d8/f21 278111 0 2024-04-22T23:23:39.970 INFO:tasks.workunit.client.0.smithi067.stdout:7/912: mkdir d10/d14/d1d/d26/d2c/d153 0 2024-04-22T23:23:39.971 INFO:tasks.workunit.client.0.smithi067.stdout:9/858: getdents d5/d21/d47/d57/dc0/d105 0 2024-04-22T23:23:39.971 INFO:tasks.workunit.client.1.smithi134.stdout:4/359: link d4/d15/d2a/c3e d4/d15/c7d 0 2024-04-22T23:23:39.974 INFO:tasks.workunit.client.0.smithi067.stdout:2/930: truncate d1/d10/d33/d46/da4/fc9 629799 0 2024-04-22T23:23:39.974 INFO:tasks.workunit.client.1.smithi134.stdout:4/360: symlink d4/d15/l7e 0 2024-04-22T23:23:39.975 INFO:tasks.workunit.client.0.smithi067.stdout:9/859: dread d5/d21/d47/d57/dc0/f10f [0,4194304] 0 2024-04-22T23:23:39.976 INFO:tasks.workunit.client.0.smithi067.stdout:6/922: write d17/fca [148823,2067] 0 2024-04-22T23:23:39.977 INFO:tasks.workunit.client.1.smithi134.stdout:4/361: dread d4/d35/d5e/f4a [0,4194304] 0 2024-04-22T23:23:39.978 INFO:tasks.workunit.client.0.smithi067.stdout:6/923: chown d17/f35 325821424 1 2024-04-22T23:23:39.979 INFO:tasks.workunit.client.0.smithi067.stdout:9/860: fsync d5/dd/d18/d5c/fb0 0 2024-04-22T23:23:39.981 INFO:tasks.workunit.client.0.smithi067.stdout:7/913: dwrite d10/d14/d1d/d26/d80/fc8 [0,4194304] 0 2024-04-22T23:23:39.982 INFO:tasks.workunit.client.0.smithi067.stdout:3/924: fdatasync d3/d4/d28/d40/dfd/dfe/d4b/f94 0 2024-04-22T23:23:39.983 INFO:tasks.workunit.client.1.smithi134.stdout:6/238: dread d0/f1a [0,4194304] 0 2024-04-22T23:23:39.983 INFO:tasks.workunit.client.0.smithi067.stdout:3/925: readlink d3/d4/d10/d1f/d3c/lc6 0 2024-04-22T23:23:39.985 INFO:tasks.workunit.client.1.smithi134.stdout:4/362: dwrite d4/d15/f67 [0,4194304] 0 2024-04-22T23:23:39.987 INFO:tasks.workunit.client.0.smithi067.stdout:1/879: write d8/d9/d2f/d33/d66/f91 [3861386,107356] 0 2024-04-22T23:23:39.987 INFO:tasks.workunit.client.1.smithi134.stdout:6/239: mknod d0/d19/c59 0 2024-04-22T23:23:39.987 INFO:tasks.workunit.client.0.smithi067.stdout:1/880: readlink d8/d9/de0/lc1 0 2024-04-22T23:23:39.990 INFO:tasks.workunit.client.0.smithi067.stdout:5/974: creat d5/d28/d36/d40/d97/dc8/d127/f153 x:0 0 0 2024-04-22T23:23:39.992 INFO:tasks.workunit.client.1.smithi134.stdout:4/363: rename d4/d15/d2a/f48 to d4/d15/f7f 0 2024-04-22T23:23:39.993 INFO:tasks.workunit.client.1.smithi134.stdout:6/240: truncate d0/f1 1261218 0 2024-04-22T23:23:39.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:39 smithi134 ceph-mon[34609]: pgmap v275: 65 pgs: 65 active+clean; 2.1 GiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 66 MiB/s rd, 203 MiB/s wr, 367 op/s 2024-04-22T23:23:39.996 INFO:tasks.workunit.client.0.smithi067.stdout:7/914: creat d10/d14/d1d/d26/d2c/d7a/f154 x:0 0 0 2024-04-22T23:23:39.996 INFO:tasks.workunit.client.0.smithi067.stdout:3/926: creat d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d51/f13d x:0 0 0 2024-04-22T23:23:39.996 INFO:tasks.workunit.client.0.smithi067.stdout:8/926: dwrite d1/dd/f85 [0,4194304] 0 2024-04-22T23:23:39.998 INFO:tasks.workunit.client.0.smithi067.stdout:2/931: creat d1/d10/f135 x:0 0 0 2024-04-22T23:23:39.999 INFO:tasks.workunit.client.0.smithi067.stdout:7/915: chown d10/d14/d1d/d26/d2c/ff2 13 1 2024-04-22T23:23:39.999 INFO:tasks.workunit.client.1.smithi134.stdout:4/364: dwrite d4/d35/f57 [4194304,4194304] 0 2024-04-22T23:23:40.000 INFO:tasks.workunit.client.0.smithi067.stdout:2/932: write d1/d10/d110/d12d/f12e [91799,59568] 0 2024-04-22T23:23:40.001 INFO:tasks.workunit.client.1.smithi134.stdout:6/241: symlink d0/d19/de/d12/d2a/d2d/l5a 0 2024-04-22T23:23:40.002 INFO:tasks.workunit.client.0.smithi067.stdout:2/933: dread - d1/d42/d61/db9/f12c zero size 2024-04-22T23:23:40.002 INFO:tasks.workunit.client.0.smithi067.stdout:1/881: rename d8/d9/d2d/d4d/d48/f11a to d8/d9/d2f/d33/d5c/dda/f11b 0 2024-04-22T23:23:40.009 INFO:tasks.workunit.client.1.smithi134.stdout:4/365: dwrite d4/d7/f28 [0,4194304] 0 2024-04-22T23:23:40.013 INFO:tasks.workunit.client.0.smithi067.stdout:3/927: creat d3/d4/d10/d10b/d95/f13e x:0 0 0 2024-04-22T23:23:40.014 INFO:tasks.workunit.client.0.smithi067.stdout:6/924: dwrite d17/f29 [0,4194304] 0 2024-04-22T23:23:40.014 INFO:tasks.workunit.client.1.smithi134.stdout:4/366: mkdir d4/d35/d80 0 2024-04-22T23:23:40.015 INFO:tasks.workunit.client.0.smithi067.stdout:2/934: creat d1/d10/d1c/d4d/f136 x:0 0 0 2024-04-22T23:23:40.018 INFO:tasks.workunit.client.0.smithi067.stdout:1/882: creat d8/d12/d36/f11c x:0 0 0 2024-04-22T23:23:40.018 INFO:tasks.workunit.client.0.smithi067.stdout:9/861: dwrite d5/d21/d47/d56/f7c [0,4194304] 0 2024-04-22T23:23:40.021 INFO:tasks.workunit.client.1.smithi134.stdout:6/242: truncate d0/f16 166103 0 2024-04-22T23:23:40.022 INFO:tasks.workunit.client.1.smithi134.stdout:6/243: truncate d0/f8 654613 0 2024-04-22T23:23:40.022 INFO:tasks.workunit.client.1.smithi134.stdout:4/367: truncate d4/d13/f65 439655 0 2024-04-22T23:23:40.023 INFO:tasks.workunit.client.1.smithi134.stdout:4/368: read d4/d7/f28 [2641263,25858] 0 2024-04-22T23:23:40.024 INFO:tasks.workunit.client.0.smithi067.stdout:6/925: dread - d17/d39/d53/d90/d46/fec zero size 2024-04-22T23:23:40.025 INFO:tasks.workunit.client.0.smithi067.stdout:7/916: truncate d10/d14/f5a 1808144 0 2024-04-22T23:23:40.026 INFO:tasks.workunit.client.1.smithi134.stdout:4/369: chown d4/d15/d2a/c3e 6 1 2024-04-22T23:23:40.027 INFO:tasks.workunit.client.0.smithi067.stdout:5/975: dwrite d5/db/d12/dbb/d61/d89/fb5 [0,4194304] 0 2024-04-22T23:23:40.027 INFO:tasks.workunit.client.0.smithi067.stdout:6/926: dread - d17/d83/d9b/fcc zero size 2024-04-22T23:23:40.028 INFO:tasks.workunit.client.0.smithi067.stdout:3/928: chown d3/d4/d10/d38/dbf/lc2 135 1 2024-04-22T23:23:40.029 INFO:tasks.workunit.client.1.smithi134.stdout:4/370: creat d4/d15/f81 x:0 0 0 2024-04-22T23:23:40.030 INFO:tasks.workunit.client.0.smithi067.stdout:5/976: readlink d5/d3e/d90/la5 0 2024-04-22T23:23:40.030 INFO:tasks.workunit.client.0.smithi067.stdout:2/935: read d1/d42/d5f/f90 [175312,43752] 0 2024-04-22T23:23:40.034 INFO:tasks.workunit.client.0.smithi067.stdout:1/883: write d8/d9/d2f/d33/d66/ff4 [4962832,68761] 0 2024-04-22T23:23:40.035 INFO:tasks.workunit.client.1.smithi134.stdout:4/371: mkdir d4/d15/d82 0 2024-04-22T23:23:40.035 INFO:tasks.workunit.client.0.smithi067.stdout:3/929: chown d3/d4/d10/d38/d53/d55/la7 164982514 1 2024-04-22T23:23:40.035 INFO:tasks.workunit.client.0.smithi067.stdout:6/927: chown d17/d39/d3a/d8d/cf9 61503 1 2024-04-22T23:23:40.036 INFO:tasks.workunit.client.0.smithi067.stdout:1/884: dread - d8/d9/d2f/d31/ded/ffd zero size 2024-04-22T23:23:40.039 INFO:tasks.workunit.client.0.smithi067.stdout:5/977: creat d5/d28/daf/dbf/dfc/f154 x:0 0 0 2024-04-22T23:23:40.042 INFO:tasks.workunit.client.0.smithi067.stdout:8/927: dwrite d1/d41/f83 [0,4194304] 0 2024-04-22T23:23:40.042 INFO:tasks.workunit.client.1.smithi134.stdout:4/372: dwrite d4/d15/f54 [0,4194304] 0 2024-04-22T23:23:40.044 INFO:tasks.workunit.client.0.smithi067.stdout:2/936: link d1/d42/d5f/d122/f36 d1/d4f/f137 0 2024-04-22T23:23:40.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:39 smithi067 ceph-mon[26453]: pgmap v275: 65 pgs: 65 active+clean; 2.1 GiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 66 MiB/s rd, 203 MiB/s wr, 367 op/s 2024-04-22T23:23:40.045 INFO:tasks.workunit.client.1.smithi134.stdout:4/373: write d4/f42 [4245544,2999] 0 2024-04-22T23:23:40.046 INFO:tasks.workunit.client.0.smithi067.stdout:2/937: fdatasync d1/d10/d1c/fc6 0 2024-04-22T23:23:40.046 INFO:tasks.workunit.client.0.smithi067.stdout:1/885: mkdir d8/d9/d2f/d33/d11d 0 2024-04-22T23:23:40.046 INFO:tasks.workunit.client.1.smithi134.stdout:4/374: fdatasync d4/d15/f6f 0 2024-04-22T23:23:40.048 INFO:tasks.workunit.client.0.smithi067.stdout:5/978: mknod d5/c155 0 2024-04-22T23:23:40.049 INFO:tasks.workunit.client.0.smithi067.stdout:8/928: dread d1/dd/d80/db9/d110/d62/ddf/f11a [0,4194304] 0 2024-04-22T23:23:40.052 INFO:tasks.workunit.client.0.smithi067.stdout:6/928: link d17/d39/d3a/c70 d17/d39/d3a/d11c/c134 0 2024-04-22T23:23:40.052 INFO:tasks.workunit.client.1.smithi134.stdout:4/375: dread d4/d15/f54 [0,4194304] 0 2024-04-22T23:23:40.053 INFO:tasks.workunit.client.1.smithi134.stdout:4/376: read - d4/d15/d2a/f75 zero size 2024-04-22T23:23:40.060 INFO:tasks.workunit.client.0.smithi067.stdout:6/929: mkdir d17/d39/d53/d90/d135 0 2024-04-22T23:23:40.061 INFO:tasks.workunit.client.1.smithi134.stdout:4/377: dwrite d4/d15/d2a/d6e/f7c [0,4194304] 0 2024-04-22T23:23:40.062 INFO:tasks.workunit.client.0.smithi067.stdout:9/862: dwrite d5/dd/d18/d39/d3e/d3f/d63/f91 [0,4194304] 0 2024-04-22T23:23:40.063 INFO:tasks.workunit.client.1.smithi134.stdout:4/378: creat d4/d15/f83 x:0 0 0 2024-04-22T23:23:40.068 INFO:tasks.workunit.client.0.smithi067.stdout:9/863: mkdir d5/d21/d47/d11c 0 2024-04-22T23:23:40.069 INFO:tasks.workunit.client.0.smithi067.stdout:6/930: link d17/f9c d17/d83/f136 0 2024-04-22T23:23:40.070 INFO:tasks.workunit.client.1.smithi134.stdout:4/379: mkdir d4/d7/d32/d84 0 2024-04-22T23:23:40.070 INFO:tasks.workunit.client.1.smithi134.stdout:4/380: stat d4/d15/f54 0 2024-04-22T23:23:40.071 INFO:tasks.workunit.client.1.smithi134.stdout:4/381: readlink d4/d15/d2a/d5d/l78 0 2024-04-22T23:23:40.072 INFO:tasks.workunit.client.0.smithi067.stdout:9/864: read - d5/dd/d18/d39/d4d/faa zero size 2024-04-22T23:23:40.075 INFO:tasks.workunit.client.0.smithi067.stdout:6/931: mknod d17/d39/d53/d90/d46/d62/d88/c137 0 2024-04-22T23:23:40.076 INFO:tasks.workunit.client.0.smithi067.stdout:9/865: write d5/d21/d47/d7f/f8e [1783702,98384] 0 2024-04-22T23:23:40.077 INFO:tasks.workunit.client.0.smithi067.stdout:7/917: dwrite d10/d18/f1a [0,4194304] 0 2024-04-22T23:23:40.080 INFO:tasks.workunit.client.0.smithi067.stdout:2/938: write d1/d10/d1c/f5d [3257124,12500] 0 2024-04-22T23:23:40.088 INFO:tasks.workunit.client.1.smithi134.stdout:1/170: sync 2024-04-22T23:23:40.088 INFO:tasks.workunit.client.1.smithi134.stdout:0/188: sync 2024-04-22T23:23:40.088 INFO:tasks.workunit.client.1.smithi134.stdout:8/185: sync 2024-04-22T23:23:40.091 INFO:tasks.workunit.client.0.smithi067.stdout:3/930: dread d3/d4/d10/d38/dbf/ff0 [0,4194304] 0 2024-04-22T23:23:40.091 INFO:tasks.workunit.client.0.smithi067.stdout:7/918: dread - d10/d14/d1d/d26/d2c/d41/db0/f14e zero size 2024-04-22T23:23:40.093 INFO:tasks.workunit.client.1.smithi134.stdout:1/171: mknod db/d15/d17/d20/d2a/c41 0 2024-04-22T23:23:40.094 INFO:tasks.workunit.client.1.smithi134.stdout:4/382: dwrite d4/d13/f59 [0,4194304] 0 2024-04-22T23:23:40.096 INFO:tasks.workunit.client.1.smithi134.stdout:0/189: getdents d1/d5/dd/d11 0 2024-04-22T23:23:40.098 INFO:tasks.workunit.client.1.smithi134.stdout:8/186: dwrite d7/f1c [0,4194304] 0 2024-04-22T23:23:40.099 INFO:tasks.workunit.client.0.smithi067.stdout:1/886: dwrite d8/d9/f29 [0,4194304] 0 2024-04-22T23:23:40.099 INFO:tasks.workunit.client.0.smithi067.stdout:8/929: dwrite d1/dd/f95 [0,4194304] 0 2024-04-22T23:23:40.101 INFO:tasks.workunit.client.0.smithi067.stdout:5/979: dwrite d5/d1a/d103/fa6 [0,4194304] 0 2024-04-22T23:23:40.102 INFO:tasks.workunit.client.1.smithi134.stdout:4/383: rename d4/d15/d2a/c3e to d4/d15/d64/c85 0 2024-04-22T23:23:40.102 INFO:tasks.workunit.client.0.smithi067.stdout:7/919: chown d10/d14/d1d/d26/d80/l12f 47539 1 2024-04-22T23:23:40.104 INFO:tasks.workunit.client.0.smithi067.stdout:9/866: getdents d5/dd/d18/d39/d3e/d3f/d63 0 2024-04-22T23:23:40.107 INFO:tasks.workunit.client.1.smithi134.stdout:1/172: dwrite db/fd [0,4194304] 0 2024-04-22T23:23:40.108 INFO:tasks.workunit.client.0.smithi067.stdout:6/932: dwrite d17/d39/d3a/fe0 [0,4194304] 0 2024-04-22T23:23:40.108 INFO:tasks.workunit.client.0.smithi067.stdout:1/887: dread - d8/d9/d102/f119 zero size 2024-04-22T23:23:40.110 INFO:tasks.workunit.client.0.smithi067.stdout:6/933: stat d17/d83/d9b/ddd/fe6 0 2024-04-22T23:23:40.111 INFO:tasks.workunit.client.0.smithi067.stdout:8/930: truncate d1/dd/d80/d6f/fdc 449024 0 2024-04-22T23:23:40.112 INFO:tasks.workunit.client.1.smithi134.stdout:1/173: mkdir db/d15/d17/d20/d2a/d2d/d2e/d42 0 2024-04-22T23:23:40.112 INFO:tasks.workunit.client.0.smithi067.stdout:6/934: chown d17/d39/d53/f60 1167179 1 2024-04-22T23:23:40.113 INFO:tasks.workunit.client.1.smithi134.stdout:0/190: dwrite d1/d5/dd/d20/f25 [4194304,4194304] 0 2024-04-22T23:23:40.113 INFO:tasks.workunit.client.1.smithi134.stdout:4/384: getdents d4/d7/d32/d84 0 2024-04-22T23:23:40.114 INFO:tasks.workunit.client.0.smithi067.stdout:6/935: readlink d17/d39/d53/d90/l116 0 2024-04-22T23:23:40.115 INFO:tasks.workunit.client.0.smithi067.stdout:9/867: mknod d5/dd/d18/d39/d3e/d3f/c11d 0 2024-04-22T23:23:40.115 INFO:tasks.workunit.client.0.smithi067.stdout:3/931: creat d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d8c/f13f x:0 0 0 2024-04-22T23:23:40.115 INFO:tasks.workunit.client.1.smithi134.stdout:0/191: dread - d1/d5/dd/f2a zero size 2024-04-22T23:23:40.117 INFO:tasks.workunit.client.1.smithi134.stdout:4/385: chown d4/d15/d2a/f75 105327 1 2024-04-22T23:23:40.119 INFO:tasks.workunit.client.1.smithi134.stdout:1/174: creat db/d15/d17/d20/d2a/d2d/f43 x:0 0 0 2024-04-22T23:23:40.119 INFO:tasks.workunit.client.1.smithi134.stdout:1/175: readlink db/d15/l38 0 2024-04-22T23:23:40.121 INFO:tasks.workunit.client.0.smithi067.stdout:6/936: mknod d17/d39/d104/d106/c138 0 2024-04-22T23:23:40.122 INFO:tasks.workunit.client.1.smithi134.stdout:0/192: unlink d1/d5/f7 0 2024-04-22T23:23:40.123 INFO:tasks.workunit.client.1.smithi134.stdout:4/386: readlink d4/l27 0 2024-04-22T23:23:40.125 INFO:tasks.workunit.client.0.smithi067.stdout:9/868: creat d5/d21/d47/d56/d8c/f11e x:0 0 0 2024-04-22T23:23:40.125 INFO:tasks.workunit.client.0.smithi067.stdout:5/980: mknod d5/db/d12/dbb/c156 0 2024-04-22T23:23:40.126 INFO:tasks.workunit.client.0.smithi067.stdout:9/869: chown d5/f9 89 1 2024-04-22T23:23:40.129 INFO:tasks.workunit.client.0.smithi067.stdout:9/870: dread d5/dd/d18/d39/d49/f7b [0,4194304] 0 2024-04-22T23:23:40.130 INFO:tasks.workunit.client.0.smithi067.stdout:8/931: creat d1/dd/d80/db9/dd0/f12c x:0 0 0 2024-04-22T23:23:40.130 INFO:tasks.workunit.client.0.smithi067.stdout:3/932: mkdir d3/d4/d10/d38/d53/d140 0 2024-04-22T23:23:40.131 INFO:tasks.workunit.client.0.smithi067.stdout:3/933: chown d3/f6f 3209 1 2024-04-22T23:23:40.131 INFO:tasks.workunit.client.1.smithi134.stdout:0/193: unlink d1/d2/c2d 0 2024-04-22T23:23:40.132 INFO:tasks.workunit.client.0.smithi067.stdout:9/871: mknod d5/dd/de6/c11f 0 2024-04-22T23:23:40.133 INFO:tasks.workunit.client.0.smithi067.stdout:5/981: mkdir d5/db/d12/dbb/d5f/d79/d11d/d157 0 2024-04-22T23:23:40.133 INFO:tasks.workunit.client.1.smithi134.stdout:0/194: chown d1/d2/f38 534058514 1 2024-04-22T23:23:40.134 INFO:tasks.workunit.client.0.smithi067.stdout:3/934: write d3/d4/d10/d1f/f3e [2666888,1106] 0 2024-04-22T23:23:40.135 INFO:tasks.workunit.client.1.smithi134.stdout:4/387: dwrite d4/d15/d64/f69 [0,4194304] 0 2024-04-22T23:23:40.136 INFO:tasks.workunit.client.0.smithi067.stdout:8/932: link d1/dd/d80/d6f/db6/fcd d1/dd/d80/d6f/f12d 0 2024-04-22T23:23:40.137 INFO:tasks.workunit.client.0.smithi067.stdout:5/982: mknod d5/d1a/d103/d4d/da9/c158 0 2024-04-22T23:23:40.138 INFO:tasks.workunit.client.0.smithi067.stdout:9/872: fsync d5/dd/d18/d39/d4d/fd0 0 2024-04-22T23:23:40.142 INFO:tasks.workunit.client.0.smithi067.stdout:8/933: read d1/dd/d80/db9/d110/f55 [291209,129778] 0 2024-04-22T23:23:40.143 INFO:tasks.workunit.client.0.smithi067.stdout:3/935: truncate d3/d4/d28/dc0/fcc 33021 0 2024-04-22T23:23:40.145 INFO:tasks.workunit.client.0.smithi067.stdout:5/983: creat d5/f159 x:0 0 0 2024-04-22T23:23:40.145 INFO:tasks.workunit.client.0.smithi067.stdout:9/873: mkdir d5/d21/d47/d56/d120 0 2024-04-22T23:23:40.146 INFO:tasks.workunit.client.0.smithi067.stdout:8/934: mkdir d1/d3/d5c/d12e 0 2024-04-22T23:23:40.148 INFO:tasks.workunit.client.0.smithi067.stdout:2/939: dread d1/f8 [0,4194304] 0 2024-04-22T23:23:40.148 INFO:tasks.workunit.client.0.smithi067.stdout:5/984: read - d5/d28/daf/dbf/dcf/d9a/fdb zero size 2024-04-22T23:23:40.150 INFO:tasks.workunit.client.0.smithi067.stdout:9/874: fsync d5/dd/d18/d39/d3e/f53 0 2024-04-22T23:23:40.151 INFO:tasks.workunit.client.0.smithi067.stdout:8/935: fdatasync d1/d41/d46/f61 0 2024-04-22T23:23:40.152 INFO:tasks.workunit.client.0.smithi067.stdout:5/985: mkdir d5/d28/daf/dbf/dcf/d9a/d15a 0 2024-04-22T23:23:40.153 INFO:tasks.workunit.client.0.smithi067.stdout:5/986: readlink d5/db/d12/d44/d47/l7c 0 2024-04-22T23:23:40.154 INFO:tasks.workunit.client.0.smithi067.stdout:2/940: symlink d1/d10/l138 0 2024-04-22T23:23:40.157 INFO:tasks.workunit.client.0.smithi067.stdout:5/987: mkdir d5/d28/d36/d40/d97/d15b 0 2024-04-22T23:23:40.157 INFO:tasks.workunit.client.0.smithi067.stdout:8/936: creat d1/dd/d80/db9/de3/db8/dd7/f12f x:0 0 0 2024-04-22T23:23:40.158 INFO:tasks.workunit.client.0.smithi067.stdout:2/941: fsync d1/d10/d1c/ff4 0 2024-04-22T23:23:40.160 INFO:tasks.workunit.client.0.smithi067.stdout:2/942: read d1/d10/f30 [1056085,17779] 0 2024-04-22T23:23:40.165 INFO:tasks.workunit.client.0.smithi067.stdout:8/937: link d1/d3/d5c/d98/fd8 d1/dd/d80/db9/dd0/df9/d102/f130 0 2024-04-22T23:23:40.166 INFO:tasks.workunit.client.0.smithi067.stdout:8/938: mkdir d1/d3/d27/d107/d115/d131 0 2024-04-22T23:23:40.167 INFO:tasks.workunit.client.1.smithi134.stdout:0/195: dread d1/d5/dd/f16 [0,4194304] 0 2024-04-22T23:23:40.169 INFO:tasks.workunit.client.0.smithi067.stdout:1/888: dread d8/d9/d2f/d33/d66/ff4 [0,4194304] 0 2024-04-22T23:23:40.170 INFO:tasks.workunit.client.1.smithi134.stdout:1/176: getdents db/d15/d17/d20/d2a/d2d/d2e 0 2024-04-22T23:23:40.170 INFO:tasks.workunit.client.0.smithi067.stdout:1/889: truncate d8/d9/fa9 1031134 0 2024-04-22T23:23:40.170 INFO:tasks.workunit.client.1.smithi134.stdout:1/177: stat db/d15/c33 0 2024-04-22T23:23:40.172 INFO:tasks.workunit.client.0.smithi067.stdout:6/937: dread fe [0,4194304] 0 2024-04-22T23:23:40.172 INFO:tasks.workunit.client.0.smithi067.stdout:1/890: chown d8/d9/c1a 410147665 1 2024-04-22T23:23:40.172 INFO:tasks.workunit.client.0.smithi067.stdout:7/920: write d10/d14/d1d/d26/d2c/d41/f4f [1685592,28034] 0 2024-04-22T23:23:40.173 INFO:tasks.workunit.client.1.smithi134.stdout:8/187: dread d7/dc/d19/f21 [0,4194304] 0 2024-04-22T23:23:40.173 INFO:tasks.workunit.client.0.smithi067.stdout:6/938: chown d17/d39/fdf 60 1 2024-04-22T23:23:40.174 INFO:tasks.workunit.client.0.smithi067.stdout:6/939: stat d17/d39/lbd 0 2024-04-22T23:23:40.174 INFO:tasks.workunit.client.1.smithi134.stdout:1/178: write db/f1d [27183,19764] 0 2024-04-22T23:23:40.177 INFO:tasks.workunit.client.0.smithi067.stdout:1/891: creat d8/d9/de0/d5d/f11e x:0 0 0 2024-04-22T23:23:40.178 INFO:tasks.workunit.client.0.smithi067.stdout:6/940: creat d17/d39/d3a/d8d/f139 x:0 0 0 2024-04-22T23:23:40.180 INFO:tasks.workunit.client.0.smithi067.stdout:8/939: dread d1/dd/d80/d6f/fad [0,4194304] 0 2024-04-22T23:23:40.180 INFO:tasks.workunit.client.1.smithi134.stdout:4/388: truncate d4/dd/f16 1884404 0 2024-04-22T23:23:40.182 INFO:tasks.workunit.client.0.smithi067.stdout:7/921: rename d10/d14/c78 to d10/d47/dab/c155 0 2024-04-22T23:23:40.182 INFO:tasks.workunit.client.1.smithi134.stdout:4/389: truncate d4/d13/f74 548670 0 2024-04-22T23:23:40.183 INFO:tasks.workunit.client.1.smithi134.stdout:4/390: write d4/d15/d64/f69 [3322810,79505] 0 2024-04-22T23:23:40.183 INFO:tasks.workunit.client.1.smithi134.stdout:1/179: creat db/d15/d17/d20/d2a/f44 x:0 0 0 2024-04-22T23:23:40.185 INFO:tasks.workunit.client.0.smithi067.stdout:6/941: fdatasync d17/d39/d3a/f3b 0 2024-04-22T23:23:40.185 INFO:tasks.workunit.client.0.smithi067.stdout:6/942: chown d17/d83/d9b/ddd 737 1 2024-04-22T23:23:40.189 INFO:tasks.workunit.client.1.smithi134.stdout:4/391: dread d4/d15/f54 [0,4194304] 0 2024-04-22T23:23:40.189 INFO:tasks.workunit.client.0.smithi067.stdout:9/875: dread d5/dd/d18/d39/d49/d54/fc9 [0,4194304] 0 2024-04-22T23:23:40.190 INFO:tasks.workunit.client.0.smithi067.stdout:5/988: truncate d5/db/d12/dbb/d61/d89/fb5 3598585 0 2024-04-22T23:23:40.190 INFO:tasks.workunit.client.0.smithi067.stdout:7/922: creat d10/d14/de0/deb/f156 x:0 0 0 2024-04-22T23:23:40.191 INFO:tasks.workunit.client.0.smithi067.stdout:5/989: read - d5/db/d12/f116 zero size 2024-04-22T23:23:40.191 INFO:tasks.workunit.client.1.smithi134.stdout:8/188: mkdir d7/dc/d2c/d37/d3a 0 2024-04-22T23:23:40.192 INFO:tasks.workunit.client.0.smithi067.stdout:1/892: creat d8/d9/de0/d5d/de2/f11f x:0 0 0 2024-04-22T23:23:40.192 INFO:tasks.workunit.client.0.smithi067.stdout:3/936: write d3/d4/d28/d4a/f7a [635098,84026] 0 2024-04-22T23:23:40.193 INFO:tasks.workunit.client.1.smithi134.stdout:4/392: truncate d4/d15/f6a 394076 0 2024-04-22T23:23:40.194 INFO:tasks.workunit.client.1.smithi134.stdout:1/180: dwrite db/d15/d17/d20/d2a/f36 [4194304,4194304] 0 2024-04-22T23:23:40.195 INFO:tasks.workunit.client.0.smithi067.stdout:6/943: mkdir d17/d83/db5/dbb/d13a 0 2024-04-22T23:23:40.196 INFO:tasks.workunit.client.1.smithi134.stdout:8/189: creat d7/db/f3b x:0 0 0 2024-04-22T23:23:40.198 INFO:tasks.workunit.client.1.smithi134.stdout:4/393: creat d4/d35/d5e/d25/f86 x:0 0 0 2024-04-22T23:23:40.198 INFO:tasks.workunit.client.0.smithi067.stdout:9/876: symlink d5/d21/d47/d56/dc6/d103/l121 0 2024-04-22T23:23:40.201 INFO:tasks.workunit.client.0.smithi067.stdout:3/937: creat d3/d4/d10/de3/f141 x:0 0 0 2024-04-22T23:23:40.201 INFO:tasks.workunit.client.1.smithi134.stdout:1/181: unlink db/d15/l22 0 2024-04-22T23:23:40.204 INFO:tasks.workunit.client.1.smithi134.stdout:1/182: read db/f1d [387508,101837] 0 2024-04-22T23:23:40.207 INFO:tasks.workunit.client.0.smithi067.stdout:1/893: dread d8/d9/f3b [0,4194304] 0 2024-04-22T23:23:40.208 INFO:tasks.workunit.client.0.smithi067.stdout:9/877: write d5/dd/d18/f1d [4681547,110620] 0 2024-04-22T23:23:40.208 INFO:tasks.workunit.client.0.smithi067.stdout:3/938: mkdir d3/d4/d1b/d93/d142 0 2024-04-22T23:23:40.213 INFO:tasks.workunit.client.1.smithi134.stdout:1/183: dwrite db/d15/d17/d20/d2a/f36 [4194304,4194304] 0 2024-04-22T23:23:40.213 INFO:tasks.workunit.client.1.smithi134.stdout:0/196: write d1/d12/d22/f24 [853456,1409] 0 2024-04-22T23:23:40.213 INFO:tasks.workunit.client.0.smithi067.stdout:7/923: unlink d10/d18/d88/daf/d100/d62/f94 0 2024-04-22T23:23:40.214 INFO:tasks.workunit.client.1.smithi134.stdout:8/190: write d7/dc/f12 [372,87043] 0 2024-04-22T23:23:40.216 INFO:tasks.workunit.client.0.smithi067.stdout:6/944: dwrite d17/d39/d53/d90/d46/db0/db9/f105 [4194304,4194304] 0 2024-04-22T23:23:40.218 INFO:tasks.workunit.client.0.smithi067.stdout:7/924: dread - d10/f146 zero size 2024-04-22T23:23:40.219 INFO:tasks.workunit.client.0.smithi067.stdout:2/943: dwrite d1/d12/f43 [0,4194304] 0 2024-04-22T23:23:40.221 INFO:tasks.workunit.client.0.smithi067.stdout:1/894: dread d8/d9/fa9 [0,4194304] 0 2024-04-22T23:23:40.225 INFO:tasks.workunit.client.1.smithi134.stdout:1/184: creat db/d15/d17/d20/f45 x:0 0 0 2024-04-22T23:23:40.225 INFO:tasks.workunit.client.1.smithi134.stdout:1/185: fsync db/fd 0 2024-04-22T23:23:40.225 INFO:tasks.workunit.client.0.smithi067.stdout:9/878: mknod d5/d21/d47/d56/d8c/c122 0 2024-04-22T23:23:40.226 INFO:tasks.workunit.client.1.smithi134.stdout:8/191: fsync d7/dc/f22 0 2024-04-22T23:23:40.230 INFO:tasks.workunit.client.0.smithi067.stdout:1/895: dread d8/d9/d2d/d4d/d9c/ffc [0,4194304] 0 2024-04-22T23:23:40.232 INFO:tasks.workunit.client.0.smithi067.stdout:7/925: fdatasync d10/d14/d1d/d26/d7e/fb5 0 2024-04-22T23:23:40.232 INFO:tasks.workunit.client.1.smithi134.stdout:4/394: write d4/dd/f41 [656668,11378] 0 2024-04-22T23:23:40.233 INFO:tasks.workunit.client.1.smithi134.stdout:4/395: dread - d4/f5c zero size 2024-04-22T23:23:40.237 INFO:tasks.workunit.client.0.smithi067.stdout:2/944: creat d1/d10/d33/f139 x:0 0 0 2024-04-22T23:23:40.238 INFO:tasks.workunit.client.1.smithi134.stdout:8/192: creat d7/db/d2d/f3c x:0 0 0 2024-04-22T23:23:40.239 INFO:tasks.workunit.client.1.smithi134.stdout:0/197: creat d1/d12/f3a x:0 0 0 2024-04-22T23:23:40.239 INFO:tasks.workunit.client.1.smithi134.stdout:4/396: rmdir d4/d35/d5e 39 2024-04-22T23:23:40.240 INFO:tasks.workunit.client.1.smithi134.stdout:4/397: chown d4/d13 119545 1 2024-04-22T23:23:40.241 INFO:tasks.workunit.client.0.smithi067.stdout:7/926: write d10/d47/d49/f58 [3651855,2479] 0 2024-04-22T23:23:40.242 INFO:tasks.workunit.client.1.smithi134.stdout:1/186: rename db/d15/d17/f1a to db/d15/d17/d20/f46 0 2024-04-22T23:23:40.247 INFO:tasks.workunit.client.0.smithi067.stdout:8/940: dwrite d1/dd/d80/fc7 [4194304,4194304] 0 2024-04-22T23:23:40.247 INFO:tasks.workunit.client.0.smithi067.stdout:1/896: creat d8/d9/d2f/d33/d66/f120 x:0 0 0 2024-04-22T23:23:40.248 INFO:tasks.workunit.client.1.smithi134.stdout:8/193: getdents d7/dc/d2c/d37 0 2024-04-22T23:23:40.249 INFO:tasks.workunit.client.0.smithi067.stdout:6/945: dwrite d17/d39/d53/d90/d46/f75 [0,4194304] 0 2024-04-22T23:23:40.250 INFO:tasks.workunit.client.1.smithi134.stdout:2/192: sync 2024-04-22T23:23:40.250 INFO:tasks.workunit.client.1.smithi134.stdout:5/219: sync 2024-04-22T23:23:40.250 INFO:tasks.workunit.client.1.smithi134.stdout:9/156: sync 2024-04-22T23:23:40.250 INFO:tasks.workunit.client.1.smithi134.stdout:3/199: sync 2024-04-22T23:23:40.252 INFO:tasks.workunit.client.0.smithi067.stdout:2/945: symlink d1/d42/d61/dad/l13a 0 2024-04-22T23:23:40.253 INFO:tasks.workunit.client.0.smithi067.stdout:5/990: rename d5/db/f2f to d5/f15c 0 2024-04-22T23:23:40.254 INFO:tasks.workunit.client.0.smithi067.stdout:9/879: dread d5/dd/d18/d39/d4d/f7d [0,4194304] 0 2024-04-22T23:23:40.254 INFO:tasks.workunit.client.0.smithi067.stdout:3/939: dwrite d3/d4/d10/d38/d53/d55/d5c/d70/f11f [0,4194304] 0 2024-04-22T23:23:40.254 INFO:tasks.workunit.client.1.smithi134.stdout:0/198: getdents d1/db 0 2024-04-22T23:23:40.255 INFO:tasks.workunit.client.1.smithi134.stdout:5/220: fdatasync d9/f19 0 2024-04-22T23:23:40.257 INFO:tasks.workunit.client.0.smithi067.stdout:8/941: write d1/dd/d80/db9/fe9 [508119,34853] 0 2024-04-22T23:23:40.258 INFO:tasks.workunit.client.1.smithi134.stdout:1/187: dwrite db/d15/f34 [0,4194304] 0 2024-04-22T23:23:40.259 INFO:tasks.workunit.client.1.smithi134.stdout:8/194: creat d7/db/d2d/d39/f3d x:0 0 0 2024-04-22T23:23:40.260 INFO:tasks.workunit.client.0.smithi067.stdout:6/946: readlink d17/d39/d53/d90/d46/db0/db9/lf8 0 2024-04-22T23:23:40.260 INFO:tasks.workunit.client.0.smithi067.stdout:1/897: write d8/d9/d2d/f112 [1330398,113002] 0 2024-04-22T23:23:40.262 INFO:tasks.workunit.client.0.smithi067.stdout:2/946: symlink d1/l13b 0 2024-04-22T23:23:40.264 INFO:tasks.workunit.client.0.smithi067.stdout:1/898: truncate d8/d9/de0/d5d/de2/f11f 449972 0 2024-04-22T23:23:40.264 INFO:tasks.workunit.client.1.smithi134.stdout:4/398: dwrite d4/f21 [0,4194304] 0 2024-04-22T23:23:40.264 INFO:tasks.workunit.client.0.smithi067.stdout:9/880: symlink d5/d21/d2c/l123 0 2024-04-22T23:23:40.265 INFO:tasks.workunit.client.0.smithi067.stdout:1/899: readlink d8/d12/l27 0 2024-04-22T23:23:40.266 INFO:tasks.workunit.client.0.smithi067.stdout:6/947: creat d17/d39/d53/d90/d46/d62/d88/f13b x:0 0 0 2024-04-22T23:23:40.268 INFO:tasks.workunit.client.1.smithi134.stdout:5/221: symlink d9/dd/d17/d2a/l46 0 2024-04-22T23:23:40.268 INFO:tasks.workunit.client.0.smithi067.stdout:8/942: mkdir d1/d3/d132 0 2024-04-22T23:23:40.269 INFO:tasks.workunit.client.0.smithi067.stdout:8/943: truncate d1/d3/d10/fda 217406 0 2024-04-22T23:23:40.270 INFO:tasks.workunit.client.1.smithi134.stdout:3/200: dwrite dc/f3f [0,4194304] 0 2024-04-22T23:23:40.271 INFO:tasks.workunit.client.0.smithi067.stdout:2/947: mknod d1/d10/d1c/d6d/d87/c13c 0 2024-04-22T23:23:40.272 INFO:tasks.workunit.client.0.smithi067.stdout:9/881: creat d5/d21/d47/d56/d8c/dd5/f124 x:0 0 0 2024-04-22T23:23:40.272 INFO:tasks.workunit.client.1.smithi134.stdout:2/193: dread d2/dd/df/d1c/f21 [0,4194304] 0 2024-04-22T23:23:40.272 INFO:tasks.workunit.client.1.smithi134.stdout:9/157: creat d0/d7/d27/f33 x:0 0 0 2024-04-22T23:23:40.274 INFO:tasks.workunit.client.1.smithi134.stdout:1/188: creat db/d15/d17/d20/d2a/d2d/d2e/f47 x:0 0 0 2024-04-22T23:23:40.274 INFO:tasks.workunit.client.0.smithi067.stdout:9/882: stat d5/dd/d18/d39/d3e/d3f/d63/f11b 0 2024-04-22T23:23:40.277 INFO:tasks.workunit.client.1.smithi134.stdout:1/189: write db/f28 [6078182,96665] 0 2024-04-22T23:23:40.277 INFO:tasks.workunit.client.0.smithi067.stdout:9/883: fsync d5/dd/d18/d39/d3e/f82 0 2024-04-22T23:23:40.278 INFO:tasks.workunit.client.1.smithi134.stdout:0/199: mknod d1/d5/dd/d11/c3b 0 2024-04-22T23:23:40.278 INFO:tasks.workunit.client.1.smithi134.stdout:5/222: creat d9/dd/d17/d2a/f47 x:0 0 0 2024-04-22T23:23:40.278 INFO:tasks.workunit.client.0.smithi067.stdout:3/940: rename d3/d4/d1b/f7c to d3/d4/d10/d38/d53/f143 0 2024-04-22T23:23:40.279 INFO:tasks.workunit.client.0.smithi067.stdout:1/900: symlink d8/d9/de0/d60/d105/l121 0 2024-04-22T23:23:40.281 INFO:tasks.workunit.client.1.smithi134.stdout:2/194: creat d2/d15/d24/d1d/f44 x:0 0 0 2024-04-22T23:23:40.282 INFO:tasks.workunit.client.0.smithi067.stdout:3/941: stat d3/d4/d10/d19/cea 0 2024-04-22T23:23:40.283 INFO:tasks.workunit.client.1.smithi134.stdout:4/399: creat d4/d35/d5e/d25/f87 x:0 0 0 2024-04-22T23:23:40.284 INFO:tasks.workunit.client.1.smithi134.stdout:7/235: sync 2024-04-22T23:23:40.284 INFO:tasks.workunit.client.1.smithi134.stdout:6/244: sync 2024-04-22T23:23:40.285 INFO:tasks.workunit.client.0.smithi067.stdout:9/884: chown d5/dd/d18/d5c/ff5 284 1 2024-04-22T23:23:40.285 INFO:tasks.workunit.client.0.smithi067.stdout:1/901: stat d8/d9/d2d 0 2024-04-22T23:23:40.286 INFO:tasks.workunit.client.0.smithi067.stdout:2/948: dwrite d1/d42/d61/fb8 [0,4194304] 0 2024-04-22T23:23:40.286 INFO:tasks.workunit.client.1.smithi134.stdout:7/236: chown d4/d8/d1d/d29/c3a 0 1 2024-04-22T23:23:40.288 INFO:tasks.workunit.client.1.smithi134.stdout:2/195: mkdir d2/dd/d45 0 2024-04-22T23:23:40.289 INFO:tasks.workunit.client.0.smithi067.stdout:6/948: getdents d17/d39/d3a/da9 0 2024-04-22T23:23:40.289 INFO:tasks.workunit.client.1.smithi134.stdout:5/223: write d9/dd/d17/f45 [4354457,70394] 0 2024-04-22T23:23:40.291 INFO:tasks.workunit.client.1.smithi134.stdout:9/158: dread d0/d7/dc/f32 [0,4194304] 0 2024-04-22T23:23:40.291 INFO:tasks.workunit.client.0.smithi067.stdout:2/949: symlink d1/d19/ded/l13d 0 2024-04-22T23:23:40.292 INFO:tasks.workunit.client.1.smithi134.stdout:8/195: getdents d7/dc 0 2024-04-22T23:23:40.293 INFO:tasks.workunit.client.1.smithi134.stdout:4/400: unlink d4/d15/d2a/d6e/f7c 0 2024-04-22T23:23:40.293 INFO:tasks.workunit.client.0.smithi067.stdout:5/991: chown d5/db/d35/f124 344 1 2024-04-22T23:23:40.294 INFO:tasks.workunit.client.0.smithi067.stdout:7/927: dread d10/d14/d1d/d46/d50/f30 [0,4194304] 0 2024-04-22T23:23:40.295 INFO:tasks.workunit.client.0.smithi067.stdout:6/949: write d17/d39/d53/f9e [2975158,64693] 0 2024-04-22T23:23:40.295 INFO:tasks.workunit.client.0.smithi067.stdout:9/885: getdents d5/dd/da9/db9/d115/d119 0 2024-04-22T23:23:40.296 INFO:tasks.workunit.client.1.smithi134.stdout:2/196: chown d2/f14 401216 1 2024-04-22T23:23:40.296 INFO:tasks.workunit.client.1.smithi134.stdout:3/201: getdents dc/dd/d28/d3b/d29 0 2024-04-22T23:23:40.298 INFO:tasks.workunit.client.0.smithi067.stdout:7/928: truncate d10/d47/dab/f118 659205 0 2024-04-22T23:23:40.298 INFO:tasks.workunit.client.1.smithi134.stdout:3/202: fdatasync dc/f45 0 2024-04-22T23:23:40.299 INFO:tasks.workunit.client.0.smithi067.stdout:1/902: link d8/d9/d2d/d4d/d83/l108 d8/d12/d40/l122 0 2024-04-22T23:23:40.300 INFO:tasks.workunit.client.0.smithi067.stdout:6/950: fdatasync d17/d39/d53/d90/f94 0 2024-04-22T23:23:40.301 INFO:tasks.workunit.client.1.smithi134.stdout:1/190: dwrite db/f1b [0,4194304] 0 2024-04-22T23:23:40.302 INFO:tasks.workunit.client.1.smithi134.stdout:6/245: chown d0/d19/db/d33/f50 11 1 2024-04-22T23:23:40.303 INFO:tasks.workunit.client.0.smithi067.stdout:7/929: mknod d10/d14/d1d/d26/d2c/d41/db0/c157 0 2024-04-22T23:23:40.304 INFO:tasks.workunit.client.1.smithi134.stdout:0/200: dwrite d1/d12/f3a [0,4194304] 0 2024-04-22T23:23:40.304 INFO:tasks.workunit.client.1.smithi134.stdout:8/196: unlink d7/la 0 2024-04-22T23:23:40.305 INFO:tasks.workunit.client.0.smithi067.stdout:2/950: dwrite d1/d10/d1c/d6d/f89 [4194304,4194304] 0 2024-04-22T23:23:40.305 INFO:tasks.workunit.client.0.smithi067.stdout:6/951: dread d17/fca [0,4194304] 0 2024-04-22T23:23:40.307 INFO:tasks.workunit.client.1.smithi134.stdout:8/197: readlink d7/dc/d19/l29 0 2024-04-22T23:23:40.310 INFO:tasks.workunit.client.0.smithi067.stdout:1/903: rename d8/d9/d2f/d31/ded/ffd to d8/d9/d2d/d4d/d70/f123 0 2024-04-22T23:23:40.312 INFO:tasks.workunit.client.0.smithi067.stdout:7/930: creat d10/d14/d1d/d46/d6a/df1/d143/f158 x:0 0 0 2024-04-22T23:23:40.318 INFO:tasks.workunit.client.1.smithi134.stdout:8/198: write d7/db/d2d/f3c [267998,72515] 0 2024-04-22T23:23:40.320 INFO:tasks.workunit.client.1.smithi134.stdout:7/237: dwrite d4/d8/d1d/f56 [0,4194304] 0 2024-04-22T23:23:40.321 INFO:tasks.workunit.client.1.smithi134.stdout:5/224: creat d9/df/d20/f48 x:0 0 0 2024-04-22T23:23:40.321 INFO:tasks.workunit.client.0.smithi067.stdout:1/904: readlink d8/d9/d2d/d4d/d83/l108 0 2024-04-22T23:23:40.321 INFO:tasks.workunit.client.1.smithi134.stdout:4/401: mknod d4/d35/c88 0 2024-04-22T23:23:40.323 INFO:tasks.workunit.client.1.smithi134.stdout:8/199: fdatasync d7/db/f3b 0 2024-04-22T23:23:40.324 INFO:tasks.workunit.client.1.smithi134.stdout:0/201: readlink d1/d12/d22/l23 0 2024-04-22T23:23:40.326 INFO:tasks.workunit.client.1.smithi134.stdout:8/200: write d7/f1c [1603744,71502] 0 2024-04-22T23:23:40.328 INFO:tasks.workunit.client.1.smithi134.stdout:9/159: creat d0/d7/d23/d2b/f34 x:0 0 0 2024-04-22T23:23:40.328 INFO:tasks.workunit.client.1.smithi134.stdout:2/197: dwrite d2/f2c [0,4194304] 0 2024-04-22T23:23:40.329 INFO:tasks.workunit.client.0.smithi067.stdout:8/944: truncate d1/dd/d80/f10d 59749 0 2024-04-22T23:23:40.330 INFO:tasks.workunit.client.0.smithi067.stdout:8/945: dread - d1/d3/d27/fea zero size 2024-04-22T23:23:40.331 INFO:tasks.workunit.client.0.smithi067.stdout:8/946: chown d1/dd/d80/db9/d110 12280 1 2024-04-22T23:23:40.335 INFO:tasks.workunit.client.0.smithi067.stdout:5/992: write d5/db/d12/dbb/d61/d89/fb3 [485499,30731] 0 2024-04-22T23:23:40.336 INFO:tasks.workunit.client.0.smithi067.stdout:5/993: write d5/db/d35/fea [4756654,47381] 0 2024-04-22T23:23:40.337 INFO:tasks.workunit.client.0.smithi067.stdout:9/886: write d5/dd/d18/d39/d4d/fd0 [397006,45047] 0 2024-04-22T23:23:40.338 INFO:tasks.workunit.client.1.smithi134.stdout:5/225: rmdir d9 39 2024-04-22T23:23:40.339 INFO:tasks.workunit.client.1.smithi134.stdout:9/160: dwrite d0/d7/f2c [0,4194304] 0 2024-04-22T23:23:40.341 INFO:tasks.workunit.client.1.smithi134.stdout:7/238: mknod d4/d8/d1d/d29/c57 0 2024-04-22T23:23:40.341 INFO:tasks.workunit.client.1.smithi134.stdout:9/161: stat d0/d7/d23/d26/l2e 0 2024-04-22T23:23:40.342 INFO:tasks.workunit.client.0.smithi067.stdout:1/905: dread d8/d9/d2f/d31/f8e [0,4194304] 0 2024-04-22T23:23:40.342 INFO:tasks.workunit.client.0.smithi067.stdout:9/887: rmdir d5/dd/d18/d3c/d78 39 2024-04-22T23:23:40.343 INFO:tasks.workunit.client.1.smithi134.stdout:9/162: fdatasync d0/d7/d23/f21 0 2024-04-22T23:23:40.345 INFO:tasks.workunit.client.1.smithi134.stdout:0/202: chown d1/f9 1012270606 1 2024-04-22T23:23:40.346 INFO:tasks.workunit.client.0.smithi067.stdout:6/952: rename d17/d39/d53/d90/d46/db0/db9/de7/fe2 to d17/d39/f13c 0 2024-04-22T23:23:40.347 INFO:tasks.workunit.client.1.smithi134.stdout:8/201: creat d7/db/d2d/f3e x:0 0 0 2024-04-22T23:23:40.348 INFO:tasks.workunit.client.1.smithi134.stdout:6/246: link d0/d19/de/d12/d2a/d2d/d3a/c57 d0/d19/db/d33/c5b 0 2024-04-22T23:23:40.349 INFO:tasks.workunit.client.1.smithi134.stdout:8/202: truncate d7/db/d2d/f3c 458481 0 2024-04-22T23:23:40.351 INFO:tasks.workunit.client.0.smithi067.stdout:8/947: dwrite d1/dd/d80/d6f/db6/fc4 [0,4194304] 0 2024-04-22T23:23:40.352 INFO:tasks.workunit.client.1.smithi134.stdout:2/198: truncate d2/f18 151170 0 2024-04-22T23:23:40.353 INFO:tasks.workunit.client.0.smithi067.stdout:6/953: stat d17/d39/d53/d90/d46/db0/db9/de7/d12e 0 2024-04-22T23:23:40.353 INFO:tasks.workunit.client.0.smithi067.stdout:2/951: dread d1/d42/d5f/d122/fd7 [0,4194304] 0 2024-04-22T23:23:40.354 INFO:tasks.workunit.client.0.smithi067.stdout:8/948: stat d1/dd/f85 0 2024-04-22T23:23:40.354 INFO:tasks.workunit.client.1.smithi134.stdout:8/203: dread d7/f1c [0,4194304] 0 2024-04-22T23:23:40.354 INFO:tasks.workunit.client.1.smithi134.stdout:4/402: truncate d4/d15/f26 2402154 0 2024-04-22T23:23:40.355 INFO:tasks.workunit.client.0.smithi067.stdout:1/906: creat d8/d9/de0/d60/f124 x:0 0 0 2024-04-22T23:23:40.355 INFO:tasks.workunit.client.0.smithi067.stdout:7/931: link d10/d47/dab/c155 d10/d14/d6d/c159 0 2024-04-22T23:23:40.359 INFO:tasks.workunit.client.0.smithi067.stdout:9/888: getdents d5 0 2024-04-22T23:23:40.362 INFO:tasks.workunit.client.0.smithi067.stdout:6/954: mkdir d17/d83/db5/dbb/d13a/d13d 0 2024-04-22T23:23:40.364 INFO:tasks.workunit.client.0.smithi067.stdout:2/952: truncate d1/d10/f24 2223556 0 2024-04-22T23:23:40.364 INFO:tasks.workunit.client.1.smithi134.stdout:8/204: unlink d7/dc/f14 0 2024-04-22T23:23:40.365 INFO:tasks.workunit.client.1.smithi134.stdout:4/403: rmdir d4/dd 39 2024-04-22T23:23:40.365 INFO:tasks.workunit.client.0.smithi067.stdout:8/949: rmdir d1/d3/d10/d3b 39 2024-04-22T23:23:40.366 INFO:tasks.workunit.client.1.smithi134.stdout:5/226: readlink d9/d1e/l3e 0 2024-04-22T23:23:40.367 INFO:tasks.workunit.client.1.smithi134.stdout:7/239: mkdir d4/d8/d58 0 2024-04-22T23:23:40.368 INFO:tasks.workunit.client.0.smithi067.stdout:7/932: rename d10/d18/d88/daf/d100/c7d to d10/d14/d6d/c15a 0 2024-04-22T23:23:40.368 INFO:tasks.workunit.client.1.smithi134.stdout:0/203: chown d1/d5/dd/d11/c39 1 1 2024-04-22T23:23:40.370 INFO:tasks.workunit.client.0.smithi067.stdout:2/953: dwrite d1/d10/d33/d46/fea [0,4194304] 0 2024-04-22T23:23:40.371 INFO:tasks.workunit.client.1.smithi134.stdout:2/199: link d2/dd/fe d2/d42/f46 0 2024-04-22T23:23:40.372 INFO:tasks.workunit.client.1.smithi134.stdout:2/200: fsync d2/dd/d2b/d33/f3a 0 2024-04-22T23:23:40.375 INFO:tasks.workunit.client.0.smithi067.stdout:5/994: getdents d5/d1a 0 2024-04-22T23:23:40.378 INFO:tasks.workunit.client.1.smithi134.stdout:5/227: mkdir d9/dd/d17/d2a/d49 0 2024-04-22T23:23:40.379 INFO:tasks.workunit.client.1.smithi134.stdout:9/163: creat d0/f35 x:0 0 0 2024-04-22T23:23:40.379 INFO:tasks.workunit.client.0.smithi067.stdout:2/954: mknod d1/d10/d1c/d11e/c13e 0 2024-04-22T23:23:40.382 INFO:tasks.workunit.client.0.smithi067.stdout:7/933: symlink d10/d14/d1d/l15b 0 2024-04-22T23:23:40.382 INFO:tasks.workunit.client.1.smithi134.stdout:2/201: dwrite d2/d15/d24/f40 [0,4194304] 0 2024-04-22T23:23:40.383 INFO:tasks.workunit.client.0.smithi067.stdout:6/955: link d17/d39/d53/d90/d46/d62/d88/lb3 d17/d39/d53/d90/d135/l13e 0 2024-04-22T23:23:40.385 INFO:tasks.workunit.client.0.smithi067.stdout:9/889: getdents d5/dd/d18/d3c/d40 0 2024-04-22T23:23:40.386 INFO:tasks.workunit.client.0.smithi067.stdout:7/934: rmdir d10/d14/d1d/d26/d7e 39 2024-04-22T23:23:40.388 INFO:tasks.workunit.client.0.smithi067.stdout:5/995: link d5/db/d12/ff6 d5/d28/daf/dbf/dcf/d9a/d15a/f15d 0 2024-04-22T23:23:40.390 INFO:tasks.workunit.client.0.smithi067.stdout:1/907: write d8/d9/d2f/d59/f78 [790468,17314] 0 2024-04-22T23:23:40.390 INFO:tasks.workunit.client.1.smithi134.stdout:0/204: creat d1/d5/f3c x:0 0 0 2024-04-22T23:23:40.390 INFO:tasks.workunit.client.1.smithi134.stdout:5/228: dread d9/f19 [0,4194304] 0 2024-04-22T23:23:40.391 INFO:tasks.workunit.client.0.smithi067.stdout:9/890: creat d5/dd/d18/d39/d49/f125 x:0 0 0 2024-04-22T23:23:40.391 INFO:tasks.workunit.client.1.smithi134.stdout:4/404: symlink d4/d35/d5e/l89 0 2024-04-22T23:23:40.392 INFO:tasks.workunit.client.0.smithi067.stdout:9/891: fsync d5/dd/d18/d3c/d40/f62 0 2024-04-22T23:23:40.393 INFO:tasks.workunit.client.0.smithi067.stdout:7/935: creat d10/d14/d1d/d26/d2c/df4/f15c x:0 0 0 2024-04-22T23:23:40.395 INFO:tasks.workunit.client.1.smithi134.stdout:2/202: rmdir d2/dd/df/d1c 39 2024-04-22T23:23:40.398 INFO:tasks.workunit.client.1.smithi134.stdout:6/247: rename d0/d19/de/d2f/l4d to d0/d19/de/l5c 0 2024-04-22T23:23:40.399 INFO:tasks.workunit.client.1.smithi134.stdout:7/240: dwrite d4/f2e [0,4194304] 0 2024-04-22T23:23:40.400 INFO:tasks.workunit.client.1.smithi134.stdout:7/241: chown d4/d8/d2a/d42/c49 3 1 2024-04-22T23:23:40.401 INFO:tasks.workunit.client.1.smithi134.stdout:7/242: fdatasync d4/d8/d2a/f2c 0 2024-04-22T23:23:40.401 INFO:tasks.workunit.client.1.smithi134.stdout:5/229: creat d9/f4a x:0 0 0 2024-04-22T23:23:40.402 INFO:tasks.workunit.client.0.smithi067.stdout:6/956: creat d17/d39/f13f x:0 0 0 2024-04-22T23:23:40.402 INFO:tasks.workunit.client.1.smithi134.stdout:7/243: stat d4/d10/d2f 0 2024-04-22T23:23:40.403 INFO:tasks.workunit.client.0.smithi067.stdout:8/950: dwrite d1/d41/f7d [0,4194304] 0 2024-04-22T23:23:40.403 INFO:tasks.workunit.client.0.smithi067.stdout:6/957: stat d17/d39/d3a/c3c 0 2024-04-22T23:23:40.404 INFO:tasks.workunit.client.0.smithi067.stdout:7/936: unlink d10/d14/d1d/d26/d2c/d41/c73 0 2024-04-22T23:23:40.405 INFO:tasks.workunit.client.1.smithi134.stdout:8/205: rmdir d7/dc/d19/d25 39 2024-04-22T23:23:40.406 INFO:tasks.workunit.client.0.smithi067.stdout:2/955: getdents d1/d42/d5f/d122/d68/d7b/d95 0 2024-04-22T23:23:40.406 INFO:tasks.workunit.client.0.smithi067.stdout:7/937: write d10/d14/d1d/f76 [1729960,10954] 0 2024-04-22T23:23:40.406 INFO:tasks.workunit.client.1.smithi134.stdout:9/164: rename d0/d7/d27/l2f to d0/d1c/l36 0 2024-04-22T23:23:40.408 INFO:tasks.workunit.client.0.smithi067.stdout:1/908: fsync d8/d12/d36/fba 0 2024-04-22T23:23:40.410 INFO:tasks.workunit.client.0.smithi067.stdout:1/909: truncate d8/d4a/ff5 89824 0 2024-04-22T23:23:40.411 INFO:tasks.workunit.client.1.smithi134.stdout:7/244: chown d4/l1a 918505 1 2024-04-22T23:23:40.411 INFO:tasks.workunit.client.1.smithi134.stdout:9/165: write d0/d7/f17 [3853863,36621] 0 2024-04-22T23:23:40.415 INFO:tasks.workunit.client.1.smithi134.stdout:4/405: link d4/d35/d5e/f4e d4/d15/d2a/f8a 0 2024-04-22T23:23:40.415 INFO:tasks.workunit.client.1.smithi134.stdout:9/166: dread - d0/f35 zero size 2024-04-22T23:23:40.416 INFO:tasks.workunit.client.0.smithi067.stdout:7/938: creat d10/d47/d49/f15d x:0 0 0 2024-04-22T23:23:40.416 INFO:tasks.workunit.client.0.smithi067.stdout:2/956: rmdir d1/d42/d5f/d122/d53/d9a/d10a/dc5 39 2024-04-22T23:23:40.417 INFO:tasks.workunit.client.1.smithi134.stdout:2/203: link d2/dd/f37 d2/dd/f47 0 2024-04-22T23:23:40.417 INFO:tasks.workunit.client.1.smithi134.stdout:8/206: dwrite d7/f23 [0,4194304] 0 2024-04-22T23:23:40.418 INFO:tasks.workunit.client.0.smithi067.stdout:9/892: creat d5/dd/d18/d3c/f126 x:0 0 0 2024-04-22T23:23:40.419 INFO:tasks.workunit.client.1.smithi134.stdout:5/230: rename d9/dd/d17/d2a/l46 to d9/dd/d17/d44/l4b 0 2024-04-22T23:23:40.419 INFO:tasks.workunit.client.1.smithi134.stdout:2/204: fdatasync d2/f14 0 2024-04-22T23:23:40.420 INFO:tasks.workunit.client.0.smithi067.stdout:6/958: creat d17/d39/d53/d90/d135/f140 x:0 0 0 2024-04-22T23:23:40.420 INFO:tasks.workunit.client.0.smithi067.stdout:8/951: getdents d1/d3/d27/d107/d115/d131 0 2024-04-22T23:23:40.422 INFO:tasks.workunit.client.0.smithi067.stdout:1/910: getdents d8/d9/d2f/d33/d11d 0 2024-04-22T23:23:40.423 INFO:tasks.workunit.client.1.smithi134.stdout:5/231: write d9/d11/d34/f40 [1311731,114004] 0 2024-04-22T23:23:40.424 INFO:tasks.workunit.client.1.smithi134.stdout:7/245: rename d4/d8/l14 to d4/d8/d3b/l59 0 2024-04-22T23:23:40.424 INFO:tasks.workunit.client.1.smithi134.stdout:8/207: chown d7/dc/f12 30 1 2024-04-22T23:23:40.426 INFO:tasks.workunit.client.1.smithi134.stdout:8/208: chown l5 17780533 1 2024-04-22T23:23:40.426 INFO:tasks.workunit.client.1.smithi134.stdout:0/205: dwrite d1/d5/dd/d11/f18 [4194304,4194304] 0 2024-04-22T23:23:40.427 INFO:tasks.workunit.client.0.smithi067.stdout:8/952: write d1/dd/d80/db9/dd0/f12c [63301,26731] 0 2024-04-22T23:23:40.427 INFO:tasks.workunit.client.0.smithi067.stdout:1/911: fsync d8/d9/de0/d5d/dd4/f103 0 2024-04-22T23:23:40.430 INFO:tasks.workunit.client.0.smithi067.stdout:2/957: dwrite d1/d10/d33/f139 [0,4194304] 0 2024-04-22T23:23:40.434 INFO:tasks.workunit.client.1.smithi134.stdout:6/248: write d0/d19/fa [1252746,59090] 0 2024-04-22T23:23:40.434 INFO:tasks.workunit.client.0.smithi067.stdout:6/959: creat d17/d39/f141 x:0 0 0 2024-04-22T23:23:40.434 INFO:tasks.workunit.client.1.smithi134.stdout:5/232: creat d9/dd/f4c x:0 0 0 2024-04-22T23:23:40.435 INFO:tasks.workunit.client.0.smithi067.stdout:1/912: creat d8/d9/de0/d5d/de2/f125 x:0 0 0 2024-04-22T23:23:40.437 INFO:tasks.workunit.client.1.smithi134.stdout:0/206: mknod d1/db/c3d 0 2024-04-22T23:23:40.439 INFO:tasks.workunit.client.0.smithi067.stdout:9/893: link d5/dd/d18/d39/f51 d5/d21/d47/d56/dc6/d103/d114/f127 0 2024-04-22T23:23:40.441 INFO:tasks.workunit.client.0.smithi067.stdout:2/958: write d1/d10/d1c/ddc/fdf [903771,96653] 0 2024-04-22T23:23:40.442 INFO:tasks.workunit.client.1.smithi134.stdout:6/249: write d0/d19/db/d35/f4b [518798,59686] 0 2024-04-22T23:23:40.442 INFO:tasks.workunit.client.1.smithi134.stdout:7/246: dread d4/d8/f21 [0,4194304] 0 2024-04-22T23:23:40.442 INFO:tasks.workunit.client.1.smithi134.stdout:4/406: dwrite d4/dd/f6d [4194304,4194304] 0 2024-04-22T23:23:40.442 INFO:tasks.workunit.client.1.smithi134.stdout:8/209: fdatasync d7/dc/d19/d25/f34 0 2024-04-22T23:23:40.443 INFO:tasks.workunit.client.0.smithi067.stdout:6/960: rmdir d17/d39/d53 39 2024-04-22T23:23:40.443 INFO:tasks.workunit.client.1.smithi134.stdout:8/210: chown d7/dc 9 1 2024-04-22T23:23:40.444 INFO:tasks.workunit.client.0.smithi067.stdout:7/939: dwrite d10/d14/d1d/f76 [4194304,4194304] 0 2024-04-22T23:23:40.444 INFO:tasks.workunit.client.0.smithi067.stdout:9/894: fdatasync d5/dd/d18/d39/d3e/d3f/d63/f11b 0 2024-04-22T23:23:40.445 INFO:tasks.workunit.client.1.smithi134.stdout:5/233: unlink d9/dd/d17/l3c 0 2024-04-22T23:23:40.446 INFO:tasks.workunit.client.0.smithi067.stdout:1/913: symlink d8/d9/d2f/d59/l126 0 2024-04-22T23:23:40.446 INFO:tasks.workunit.client.1.smithi134.stdout:5/234: stat f6 0 2024-04-22T23:23:40.449 INFO:tasks.workunit.client.1.smithi134.stdout:9/167: rename d0/d7/d23/d2b to d0/d37 0 2024-04-22T23:23:40.450 INFO:tasks.workunit.client.0.smithi067.stdout:6/961: chown d17/d39/d53/d90/d46/db0/db9/de7/fdc 0 1 2024-04-22T23:23:40.450 INFO:tasks.workunit.client.0.smithi067.stdout:7/940: creat d10/d47/dab/f15e x:0 0 0 2024-04-22T23:23:40.454 INFO:tasks.workunit.client.0.smithi067.stdout:9/895: rename d5/d21/d47/d56/d8c/dd5/lee to d5/d21/d47/d56/de5/l128 0 2024-04-22T23:23:40.454 INFO:tasks.workunit.client.1.smithi134.stdout:7/247: unlink d4/d8/d2a/d42/l4e 0 2024-04-22T23:23:40.455 INFO:tasks.workunit.client.1.smithi134.stdout:0/207: dwrite d1/d2/f38 [0,4194304] 0 2024-04-22T23:23:40.457 INFO:tasks.workunit.client.1.smithi134.stdout:8/211: write d7/dc/d19/d25/f26 [1860422,82695] 0 2024-04-22T23:23:40.457 INFO:tasks.workunit.client.1.smithi134.stdout:5/235: mkdir d9/d11/d34/d4d 0 2024-04-22T23:23:40.459 INFO:tasks.workunit.client.0.smithi067.stdout:6/962: read d17/d39/d53/d90/d46/d62/d88/f117 [631774,87483] 0 2024-04-22T23:23:40.459 INFO:tasks.workunit.client.1.smithi134.stdout:9/168: mknod d0/d7/d23/d26/c38 0 2024-04-22T23:23:40.459 INFO:tasks.workunit.client.0.smithi067.stdout:7/941: mkdir d10/d47/d49/d15f 0 2024-04-22T23:23:40.461 INFO:tasks.workunit.client.1.smithi134.stdout:6/250: rename d0/d4f/f53 to d0/d19/de/d12/d2a/f5d 0 2024-04-22T23:23:40.463 INFO:tasks.workunit.client.1.smithi134.stdout:4/407: creat d4/d15/d82/f8b x:0 0 0 2024-04-22T23:23:40.464 INFO:tasks.workunit.client.1.smithi134.stdout:4/408: dread - d4/dd/f31 zero size 2024-04-22T23:23:40.465 INFO:tasks.workunit.client.0.smithi067.stdout:7/942: symlink d10/d14/d1d/d46/d124/l160 0 2024-04-22T23:23:40.466 INFO:tasks.workunit.client.1.smithi134.stdout:8/212: dwrite d7/dc/d19/d25/f34 [0,4194304] 0 2024-04-22T23:23:40.466 INFO:tasks.workunit.client.0.smithi067.stdout:7/943: fsync d10/d47/d49/f15d 0 2024-04-22T23:23:40.466 INFO:tasks.workunit.client.0.smithi067.stdout:8/953: dread d1/d3/d27/f6e [4194304,4194304] 0 2024-04-22T23:23:40.468 INFO:tasks.workunit.client.1.smithi134.stdout:4/409: readlink d4/d35/d5e/l19 0 2024-04-22T23:23:40.468 INFO:tasks.workunit.client.1.smithi134.stdout:9/169: unlink d0/d7/d12/f18 0 2024-04-22T23:23:40.469 INFO:tasks.workunit.client.0.smithi067.stdout:2/959: link d1/d42/d5f/d122/l7f d1/d42/d5f/d122/d53/d9a/d10a/l13f 0 2024-04-22T23:23:40.471 INFO:tasks.workunit.client.1.smithi134.stdout:4/410: write d4/d35/d5e/f60 [3716887,3937] 0 2024-04-22T23:23:40.471 INFO:tasks.workunit.client.0.smithi067.stdout:2/960: chown d1/d42/d61/fa8 1757 1 2024-04-22T23:23:40.471 INFO:tasks.workunit.client.0.smithi067.stdout:9/896: getdents d5/d21/d47/d56/dc6/d103 0 2024-04-22T23:23:40.472 INFO:tasks.workunit.client.1.smithi134.stdout:7/248: rename d4/d10/d2f/d37/f3e to d4/d4a/f5a 0 2024-04-22T23:23:40.472 INFO:tasks.workunit.client.1.smithi134.stdout:2/205: dread d2/d15/d24/f2a [0,4194304] 0 2024-04-22T23:23:40.472 INFO:tasks.workunit.client.0.smithi067.stdout:6/963: truncate d17/d83/db5/dbc/fe3 2585669 0 2024-04-22T23:23:40.473 INFO:tasks.workunit.client.1.smithi134.stdout:0/208: mknod d1/d5/dd/d20/d2c/c3e 0 2024-04-22T23:23:40.478 INFO:tasks.workunit.client.0.smithi067.stdout:8/954: mknod d1/dd/d80/db9/dd0/c133 0 2024-04-22T23:23:40.478 INFO:tasks.workunit.client.1.smithi134.stdout:9/170: dwrite d0/d7/fb [0,4194304] 0 2024-04-22T23:23:40.479 INFO:tasks.workunit.client.1.smithi134.stdout:8/213: dread d7/db/fe [4194304,4194304] 0 2024-04-22T23:23:40.479 INFO:tasks.workunit.client.0.smithi067.stdout:6/964: rename d17/d39/d53/d90/l116 to d17/d83/db5/dbb/d13a/d13d/l142 0 2024-04-22T23:23:40.479 INFO:tasks.workunit.client.0.smithi067.stdout:8/955: chown d1/d3/d10/f11d 22755174 1 2024-04-22T23:23:40.483 INFO:tasks.workunit.client.0.smithi067.stdout:2/961: dwrite d1/d42/d5f/d9b/dc0/fec [0,4194304] 0 2024-04-22T23:23:40.483 INFO:tasks.workunit.client.0.smithi067.stdout:8/956: dread - d1/dd/d80/db9/de3/db8/f12b zero size 2024-04-22T23:23:40.483 INFO:tasks.workunit.client.1.smithi134.stdout:6/251: mkdir d0/d19/de/d12/d5e 0 2024-04-22T23:23:40.485 INFO:tasks.workunit.client.1.smithi134.stdout:5/236: link d9/d1e/c2c d9/dd/d17/d44/c4e 0 2024-04-22T23:23:40.485 INFO:tasks.workunit.client.1.smithi134.stdout:6/252: truncate d0/f27 611309 0 2024-04-22T23:23:40.485 INFO:tasks.workunit.client.0.smithi067.stdout:6/965: creat d17/d39/d53/d90/d46/db0/db9/f143 x:0 0 0 2024-04-22T23:23:40.488 INFO:tasks.workunit.client.1.smithi134.stdout:4/411: dwrite d4/d15/f7f [0,4194304] 0 2024-04-22T23:23:40.489 INFO:tasks.workunit.client.1.smithi134.stdout:4/412: dread - d4/d35/d5e/d25/f87 zero size 2024-04-22T23:23:40.489 INFO:tasks.workunit.client.1.smithi134.stdout:7/249: mkdir d4/d8/d41/d5b 0 2024-04-22T23:23:40.490 INFO:tasks.workunit.client.1.smithi134.stdout:9/171: symlink d0/d7/dc/l39 0 2024-04-22T23:23:40.491 INFO:tasks.workunit.client.1.smithi134.stdout:7/250: chown d4/d10/d2f/c51 5298 1 2024-04-22T23:23:40.493 INFO:tasks.workunit.client.0.smithi067.stdout:6/966: creat d17/d39/d53/d90/d46/db0/f144 x:0 0 0 2024-04-22T23:23:40.493 INFO:tasks.workunit.client.0.smithi067.stdout:8/957: dwrite d1/d3/d27/f6e [0,4194304] 0 2024-04-22T23:23:40.495 INFO:tasks.workunit.client.0.smithi067.stdout:8/958: dread d1/dd/d80/db9/d110/fd6 [0,4194304] 0 2024-04-22T23:23:40.496 INFO:tasks.workunit.client.1.smithi134.stdout:0/209: symlink d1/d5/dd/l3f 0 2024-04-22T23:23:40.498 INFO:tasks.workunit.client.1.smithi134.stdout:0/210: stat d1/d5/d15 0 2024-04-22T23:23:40.501 INFO:tasks.workunit.client.1.smithi134.stdout:7/251: dwrite d4/f33 [0,4194304] 0 2024-04-22T23:23:40.501 INFO:tasks.workunit.client.1.smithi134.stdout:6/253: chown d0/d19/db/d35/f58 42 1 2024-04-22T23:23:40.504 INFO:tasks.workunit.client.1.smithi134.stdout:6/254: dread - d0/d19/f38 zero size 2024-04-22T23:23:40.504 INFO:tasks.workunit.client.0.smithi067.stdout:6/967: rename d17/d39/d3a/d8d to d17/d83/db5/d145 0 2024-04-22T23:23:40.505 INFO:tasks.workunit.client.1.smithi134.stdout:7/252: truncate d4/d8/d41/f43 578671 0 2024-04-22T23:23:40.508 INFO:tasks.workunit.client.1.smithi134.stdout:8/214: creat d7/dc/d2c/d37/f3f x:0 0 0 2024-04-22T23:23:40.509 INFO:tasks.workunit.client.1.smithi134.stdout:5/237: chown d9/d1e/c2c 0 1 2024-04-22T23:23:40.510 INFO:tasks.workunit.client.1.smithi134.stdout:5/238: dread - d9/d1e/f26 zero size 2024-04-22T23:23:40.511 INFO:tasks.workunit.client.1.smithi134.stdout:2/206: rmdir d2/dd/d45 0 2024-04-22T23:23:40.511 INFO:tasks.workunit.client.1.smithi134.stdout:5/239: write d9/dd/d17/f45 [721080,11820] 0 2024-04-22T23:23:40.515 INFO:tasks.workunit.client.1.smithi134.stdout:8/215: symlink d7/dc/d2c/l40 0 2024-04-22T23:23:40.518 INFO:tasks.workunit.client.1.smithi134.stdout:7/253: mkdir d4/d5c 0 2024-04-22T23:23:40.518 INFO:tasks.workunit.client.0.smithi067.stdout:8/959: getdents d1/d3/d5c 0 2024-04-22T23:23:40.518 INFO:tasks.workunit.client.1.smithi134.stdout:9/172: link d0/d7/dc/l39 d0/d37/l3a 0 2024-04-22T23:23:40.519 INFO:tasks.workunit.client.0.smithi067.stdout:6/968: creat d17/d83/db5/dbc/dd8/d10c/f146 x:0 0 0 2024-04-22T23:23:40.520 INFO:tasks.workunit.client.1.smithi134.stdout:9/173: chown d0/d7/fb 31 1 2024-04-22T23:23:40.520 INFO:tasks.workunit.client.1.smithi134.stdout:2/207: dwrite d2/d15/d24/f35 [0,4194304] 0 2024-04-22T23:23:40.521 INFO:tasks.workunit.client.0.smithi067.stdout:9/897: dread d5/d21/f27 [4194304,4194304] 0 2024-04-22T23:23:40.522 INFO:tasks.workunit.client.0.smithi067.stdout:2/962: write d1/d42/d61/fa8 [1555306,76307] 0 2024-04-22T23:23:40.523 INFO:tasks.workunit.client.1.smithi134.stdout:6/255: creat d0/d19/de/f5f x:0 0 0 2024-04-22T23:23:40.523 INFO:tasks.workunit.client.0.smithi067.stdout:8/960: truncate d1/dd/f79 2232989 0 2024-04-22T23:23:40.525 INFO:tasks.workunit.client.1.smithi134.stdout:7/254: symlink d4/d8/d1d/l5d 0 2024-04-22T23:23:40.526 INFO:tasks.workunit.client.1.smithi134.stdout:9/174: creat d0/d1c/f3b x:0 0 0 2024-04-22T23:23:40.526 INFO:tasks.workunit.client.0.smithi067.stdout:6/969: write d17/d83/d9b/ddd/f122 [5847490,99101] 0 2024-04-22T23:23:40.526 INFO:tasks.workunit.client.0.smithi067.stdout:7/944: dwrite d10/d14/d1d/d26/f39 [0,4194304] 0 2024-04-22T23:23:40.527 INFO:tasks.workunit.client.1.smithi134.stdout:2/208: rename d2/d15/d24/l23 to d2/d42/l48 0 2024-04-22T23:23:40.528 INFO:tasks.workunit.client.0.smithi067.stdout:2/963: write d1/d10/d41/f125 [5175,11175] 0 2024-04-22T23:23:40.530 INFO:tasks.workunit.client.1.smithi134.stdout:5/240: link d9/d1e/c2c d9/df/d20/c4f 0 2024-04-22T23:23:40.532 INFO:tasks.workunit.client.1.smithi134.stdout:5/241: write d9/d1e/f33 [400535,61368] 0 2024-04-22T23:23:40.533 INFO:tasks.workunit.client.1.smithi134.stdout:2/209: symlink d2/d15/d24/l49 0 2024-04-22T23:23:40.534 INFO:tasks.workunit.client.0.smithi067.stdout:7/945: mknod d10/d14/d6d/d9e/c161 0 2024-04-22T23:23:40.535 INFO:tasks.workunit.client.0.smithi067.stdout:8/961: dread d1/dd/d80/db9/d110/d62/ddf/f11a [0,4194304] 0 2024-04-22T23:23:40.535 INFO:tasks.workunit.client.0.smithi067.stdout:7/946: read - d10/d47/d49/f15d zero size 2024-04-22T23:23:40.537 INFO:tasks.workunit.client.0.smithi067.stdout:6/970: creat d17/d39/d53/d90/d46/f147 x:0 0 0 2024-04-22T23:23:40.539 INFO:tasks.workunit.client.1.smithi134.stdout:0/211: getdents d1/d5/dd 0 2024-04-22T23:23:40.540 INFO:tasks.workunit.client.1.smithi134.stdout:5/242: chown f6 3688 1 2024-04-22T23:23:40.540 INFO:tasks.workunit.client.1.smithi134.stdout:7/255: getdents d4/d8/d3b 0 2024-04-22T23:23:40.542 INFO:tasks.workunit.client.0.smithi067.stdout:2/964: dread d1/d10/f30 [0,4194304] 0 2024-04-22T23:23:40.543 INFO:tasks.workunit.client.1.smithi134.stdout:9/175: dwrite d0/d7/d12/f30 [0,4194304] 0 2024-04-22T23:23:40.545 INFO:tasks.workunit.client.0.smithi067.stdout:6/971: mknod d17/d39/d53/d90/d46/c148 0 2024-04-22T23:23:40.548 INFO:tasks.workunit.client.1.smithi134.stdout:5/243: dread d9/f2b [4194304,4194304] 0 2024-04-22T23:23:40.549 INFO:tasks.workunit.client.1.smithi134.stdout:4/413: write d4/d35/d5e/f1f [156468,2582] 0 2024-04-22T23:23:40.549 INFO:tasks.workunit.client.0.smithi067.stdout:6/972: getdents d17/d39/d104/d106 0 2024-04-22T23:23:40.549 INFO:tasks.workunit.client.1.smithi134.stdout:5/244: stat d9/f16 0 2024-04-22T23:23:40.553 INFO:tasks.workunit.client.1.smithi134.stdout:0/212: creat d1/d5/dd/f40 x:0 0 0 2024-04-22T23:23:40.553 INFO:tasks.workunit.client.1.smithi134.stdout:3/203: sync 2024-04-22T23:23:40.557 INFO:tasks.workunit.client.1.smithi134.stdout:5/245: creat d9/dd/d17/d44/f50 x:0 0 0 2024-04-22T23:23:40.559 INFO:tasks.workunit.client.0.smithi067.stdout:8/962: truncate d1/dd/d80/d6f/f34 6218998 0 2024-04-22T23:23:40.559 INFO:tasks.workunit.client.1.smithi134.stdout:5/246: write d9/dd/f4c [300053,26414] 0 2024-04-22T23:23:40.560 INFO:tasks.workunit.client.0.smithi067.stdout:8/963: chown d1/dd/d80/db9/de3/fae 9051228 1 2024-04-22T23:23:40.561 INFO:tasks.workunit.client.1.smithi134.stdout:8/216: dwrite d7/dc/d19/f21 [0,4194304] 0 2024-04-22T23:23:40.563 INFO:tasks.workunit.client.0.smithi067.stdout:6/973: truncate d17/f2b 487802 0 2024-04-22T23:23:40.564 INFO:tasks.workunit.client.1.smithi134.stdout:4/414: mknod d4/d7/d32/c8c 0 2024-04-22T23:23:40.565 INFO:tasks.workunit.client.1.smithi134.stdout:8/217: write d7/db/d2d/d39/f3d [34666,45616] 0 2024-04-22T23:23:40.566 INFO:tasks.workunit.client.1.smithi134.stdout:6/256: write d0/f1 [1979959,20923] 0 2024-04-22T23:23:40.567 INFO:tasks.workunit.client.1.smithi134.stdout:0/213: write d1/d5/dd/f35 [4719633,44906] 0 2024-04-22T23:23:40.568 INFO:tasks.workunit.client.0.smithi067.stdout:2/965: dwrite d1/d10/d41/f70 [0,4194304] 0 2024-04-22T23:23:40.568 INFO:tasks.workunit.client.0.smithi067.stdout:6/974: truncate d17/dae/feb 252157 0 2024-04-22T23:23:40.568 INFO:tasks.workunit.client.1.smithi134.stdout:0/214: readlink d1/d12/d22/l23 0 2024-04-22T23:23:40.569 INFO:tasks.workunit.client.0.smithi067.stdout:2/966: chown d1/d10/d41/l112 636 1 2024-04-22T23:23:40.570 INFO:tasks.workunit.client.1.smithi134.stdout:6/257: write d0/d19/f38 [893940,59205] 0 2024-04-22T23:23:40.571 INFO:tasks.workunit.client.0.smithi067.stdout:2/967: write d1/d10/d33/f5b [4498984,117016] 0 2024-04-22T23:23:40.572 INFO:tasks.workunit.client.1.smithi134.stdout:3/204: creat dc/dd/f4f x:0 0 0 2024-04-22T23:23:40.573 INFO:tasks.workunit.client.0.smithi067.stdout:2/968: rmdir d1/d10/d1c/d11e 39 2024-04-22T23:23:40.575 INFO:tasks.workunit.client.0.smithi067.stdout:2/969: write d1/d10/d1c/ddc/fdf [893561,8231] 0 2024-04-22T23:23:40.576 INFO:tasks.workunit.client.1.smithi134.stdout:8/218: symlink d7/db/l41 0 2024-04-22T23:23:40.577 INFO:tasks.workunit.client.1.smithi134.stdout:4/415: symlink d4/d35/d5e/d25/l8d 0 2024-04-22T23:23:40.579 INFO:tasks.workunit.client.1.smithi134.stdout:2/210: write d2/dd/fe [3618324,118970] 0 2024-04-22T23:23:40.581 INFO:tasks.workunit.client.1.smithi134.stdout:7/256: write d4/d10/f11 [4265000,4875] 0 2024-04-22T23:23:40.584 INFO:tasks.workunit.client.1.smithi134.stdout:9/176: truncate d0/d7/f9 7166419 0 2024-04-22T23:23:40.584 INFO:tasks.workunit.client.1.smithi134.stdout:4/416: readlink d4/d29/l3f 0 2024-04-22T23:23:40.584 INFO:tasks.workunit.client.0.smithi067.stdout:2/970: dwrite d1/d42/d5f/d122/fd7 [0,4194304] 0 2024-04-22T23:23:40.585 INFO:tasks.workunit.client.1.smithi134.stdout:0/215: dwrite d1/d5/dd/f40 [0,4194304] 0 2024-04-22T23:23:40.586 INFO:tasks.workunit.client.1.smithi134.stdout:4/417: write d4/f1d [185630,123106] 0 2024-04-22T23:23:40.587 INFO:tasks.workunit.client.0.smithi067.stdout:8/964: dwrite d1/dd/d80/db9/d110/d62/ddf/fe5 [0,4194304] 0 2024-04-22T23:23:40.588 INFO:tasks.workunit.client.1.smithi134.stdout:7/257: readlink d4/d8/d3b/l59 0 2024-04-22T23:23:40.589 INFO:tasks.workunit.client.0.smithi067.stdout:2/971: truncate d1/d12/f4e 506257 0 2024-04-22T23:23:40.589 INFO:tasks.workunit.client.1.smithi134.stdout:3/205: truncate dc/dd/d28/d34/f48 572878 0 2024-04-22T23:23:40.591 INFO:tasks.workunit.client.0.smithi067.stdout:2/972: write d1/d42/d5f/d122/d53/dba/fc3 [976014,14797] 0 2024-04-22T23:23:40.594 INFO:tasks.workunit.client.1.smithi134.stdout:8/219: chown d7/dc/c2a 36597 1 2024-04-22T23:23:40.595 INFO:tasks.workunit.client.1.smithi134.stdout:4/418: truncate d4/d15/f83 379488 0 2024-04-22T23:23:40.595 INFO:tasks.workunit.client.1.smithi134.stdout:5/247: getdents d9/dd/d17 0 2024-04-22T23:23:40.597 INFO:tasks.workunit.client.1.smithi134.stdout:7/258: mknod d4/d10/d2f/d37/c5e 0 2024-04-22T23:23:40.599 INFO:tasks.workunit.client.1.smithi134.stdout:6/258: getdents d0/d19/de/d1f 0 2024-04-22T23:23:40.601 INFO:tasks.workunit.client.1.smithi134.stdout:8/220: mkdir d7/db/d42 0 2024-04-22T23:23:40.603 INFO:tasks.workunit.client.1.smithi134.stdout:3/206: truncate dc/dd/d28/d2f/d21/f16 1665504 0 2024-04-22T23:23:40.604 INFO:tasks.workunit.client.1.smithi134.stdout:0/216: mkdir d1/d5/dd/d41 0 2024-04-22T23:23:40.605 INFO:tasks.workunit.client.1.smithi134.stdout:7/259: creat d4/d8/d2a/f5f x:0 0 0 2024-04-22T23:23:40.606 INFO:tasks.workunit.client.1.smithi134.stdout:9/177: dwrite d0/d7/d12/d22/f31 [0,4194304] 0 2024-04-22T23:23:40.608 INFO:tasks.workunit.client.1.smithi134.stdout:7/260: chown d4/d10/l12 402908670 1 2024-04-22T23:23:40.609 INFO:tasks.workunit.client.1.smithi134.stdout:5/248: getdents d9/d1e 0 2024-04-22T23:23:40.612 INFO:tasks.workunit.client.1.smithi134.stdout:6/259: rename d0/d19/de/d12/d2a/d2d/d3a/f4a to d0/f60 0 2024-04-22T23:23:40.612 INFO:tasks.workunit.client.1.smithi134.stdout:0/217: readlink d1/d5/dd/l3f 0 2024-04-22T23:23:40.617 INFO:tasks.workunit.client.1.smithi134.stdout:8/221: dwrite d7/dc/d19/f21 [0,4194304] 0 2024-04-22T23:23:40.618 INFO:tasks.workunit.client.1.smithi134.stdout:0/218: write d1/d5/dd/d20/f25 [4420184,26578] 0 2024-04-22T23:23:40.619 INFO:tasks.workunit.client.1.smithi134.stdout:9/178: mkdir d0/d7/d3c 0 2024-04-22T23:23:40.620 INFO:tasks.workunit.client.1.smithi134.stdout:0/219: chown d1/d2/c32 126982 1 2024-04-22T23:23:40.622 INFO:tasks.workunit.client.1.smithi134.stdout:3/207: link dc/dd/d28/d2f/d21/f33 dc/dd/d28/d34/f50 0 2024-04-22T23:23:40.627 INFO:tasks.workunit.client.1.smithi134.stdout:7/261: mknod d4/d10/d2f/d34/d54/c60 0 2024-04-22T23:23:40.631 INFO:tasks.workunit.client.1.smithi134.stdout:0/220: read d1/f10 [1137253,101273] 0 2024-04-22T23:23:40.631 INFO:tasks.workunit.client.1.smithi134.stdout:3/208: truncate dc/f22 972256 0 2024-04-22T23:23:40.632 INFO:tasks.workunit.client.1.smithi134.stdout:8/222: dwrite d7/db/fe [0,4194304] 0 2024-04-22T23:23:40.635 INFO:tasks.workunit.client.1.smithi134.stdout:9/179: mkdir d0/d2d/d3d 0 2024-04-22T23:23:40.637 INFO:tasks.workunit.client.1.smithi134.stdout:6/260: rename d0/d19/f1c to d0/d19/de/d12/f61 0 2024-04-22T23:23:40.638 INFO:tasks.workunit.client.1.smithi134.stdout:7/262: dwrite d4/d8/ff [0,4194304] 0 2024-04-22T23:23:40.640 INFO:tasks.workunit.client.1.smithi134.stdout:5/249: link d9/d11/f1a d9/d11/d34/f51 0 2024-04-22T23:23:40.642 INFO:tasks.workunit.client.1.smithi134.stdout:3/209: unlink dc/dd/d28/d2f/d21/f33 0 2024-04-22T23:23:40.643 INFO:tasks.workunit.client.1.smithi134.stdout:0/221: mknod d1/d2/c42 0 2024-04-22T23:23:40.649 INFO:tasks.workunit.client.1.smithi134.stdout:9/180: creat d0/d7/d12/d22/f3e x:0 0 0 2024-04-22T23:23:40.650 INFO:tasks.workunit.client.1.smithi134.stdout:6/261: creat d0/d19/db/d33/f62 x:0 0 0 2024-04-22T23:23:40.653 INFO:tasks.workunit.client.1.smithi134.stdout:5/250: chown d9/df/d20/l38 5 1 2024-04-22T23:23:40.656 INFO:tasks.workunit.client.1.smithi134.stdout:6/262: mknod d0/d19/de/d2f/c63 0 2024-04-22T23:23:40.656 INFO:tasks.workunit.client.1.smithi134.stdout:7/263: symlink d4/d8/d41/d5b/l61 0 2024-04-22T23:23:40.658 INFO:tasks.workunit.client.1.smithi134.stdout:5/251: mkdir d9/dd/d52 0 2024-04-22T23:23:40.660 INFO:tasks.workunit.client.1.smithi134.stdout:6/263: creat d0/d19/db/d33/f64 x:0 0 0 2024-04-22T23:23:40.663 INFO:tasks.workunit.client.1.smithi134.stdout:7/264: rename d4/d8/d1d to d4/d10/d2f/d34/d62 0 2024-04-22T23:23:40.664 INFO:tasks.workunit.client.1.smithi134.stdout:7/265: write d4/d8/d2a/f5f [315283,112692] 0 2024-04-22T23:23:40.665 INFO:tasks.workunit.client.1.smithi134.stdout:3/210: creat dc/dd/d28/d2f/f51 x:0 0 0 2024-04-22T23:23:40.666 INFO:tasks.workunit.client.1.smithi134.stdout:5/252: unlink d9/d1e/f26 0 2024-04-22T23:23:40.668 INFO:tasks.workunit.client.1.smithi134.stdout:8/223: getdents d7/dc/d19/d25 0 2024-04-22T23:23:40.670 INFO:tasks.workunit.client.1.smithi134.stdout:6/264: symlink d0/d19/db/l65 0 2024-04-22T23:23:40.674 INFO:tasks.workunit.client.1.smithi134.stdout:7/266: stat d4/c16 0 2024-04-22T23:23:40.677 INFO:tasks.workunit.client.1.smithi134.stdout:0/222: dread d1/d5/dd/f2b [0,4194304] 0 2024-04-22T23:23:40.677 INFO:tasks.workunit.client.1.smithi134.stdout:5/253: rename d9/dd/d17/d44/f50 to d9/d11/d34/d3b/f53 0 2024-04-22T23:23:40.679 INFO:tasks.workunit.client.1.smithi134.stdout:7/267: dread d4/f2d [0,4194304] 0 2024-04-22T23:23:40.679 INFO:tasks.workunit.client.1.smithi134.stdout:6/265: write d0/d19/de/d12/f61 [3564583,96976] 0 2024-04-22T23:23:40.687 INFO:tasks.workunit.client.1.smithi134.stdout:7/268: dwrite d4/d10/d2f/d34/d62/f56 [4194304,4194304] 0 2024-04-22T23:23:40.687 INFO:tasks.workunit.client.1.smithi134.stdout:0/223: symlink d1/d5/dd/d20/l43 0 2024-04-22T23:23:40.688 INFO:tasks.workunit.client.0.smithi067.stdout:8/965: symlink d1/dd/d80/db9/d110/d62/d75/l134 0 2024-04-22T23:23:40.689 INFO:tasks.workunit.client.1.smithi134.stdout:0/224: readlink d1/d5/dd/d20/l43 0 2024-04-22T23:23:40.689 INFO:tasks.workunit.client.1.smithi134.stdout:5/254: creat d9/d11/d34/f54 x:0 0 0 2024-04-22T23:23:40.689 INFO:tasks.workunit.client.0.smithi067.stdout:8/966: write d1/d3/f2a [1697818,91450] 0 2024-04-22T23:23:40.689 INFO:tasks.workunit.client.1.smithi134.stdout:9/181: truncate d0/d7/f2c 1319139 0 2024-04-22T23:23:40.690 INFO:tasks.workunit.client.1.smithi134.stdout:3/211: creat dc/f52 x:0 0 0 2024-04-22T23:23:40.691 INFO:tasks.workunit.client.1.smithi134.stdout:9/182: dread - d0/d7/d27/f33 zero size 2024-04-22T23:23:40.692 INFO:tasks.workunit.client.1.smithi134.stdout:5/255: write d9/dd/d17/d2a/f47 [523546,56589] 0 2024-04-22T23:23:40.693 INFO:tasks.workunit.client.1.smithi134.stdout:3/212: dread - dc/dd/d28/d2f/f51 zero size 2024-04-22T23:23:40.694 INFO:tasks.workunit.client.0.smithi067.stdout:8/967: creat d1/d3/d7e/f135 x:0 0 0 2024-04-22T23:23:40.698 INFO:tasks.workunit.client.1.smithi134.stdout:0/225: truncate d1/d5/dd/f2a 574213 0 2024-04-22T23:23:40.698 INFO:tasks.workunit.client.1.smithi134.stdout:7/269: rename d4/d10/d2f/d34/d62/c25 to d4/d10/d2f/d34/d62/c63 0 2024-04-22T23:23:40.701 INFO:tasks.workunit.client.1.smithi134.stdout:6/266: creat d0/d19/de/d12/d5e/f66 x:0 0 0 2024-04-22T23:23:40.701 INFO:tasks.workunit.client.1.smithi134.stdout:9/183: dwrite d0/d7/d12/d22/f31 [0,4194304] 0 2024-04-22T23:23:40.708 INFO:tasks.workunit.client.0.smithi067.stdout:2/973: truncate d1/d42/d61/daa/fca 1468827 0 2024-04-22T23:23:40.709 INFO:tasks.workunit.client.0.smithi067.stdout:2/974: chown d1/d42/d5f/cee 0 1 2024-04-22T23:23:40.714 INFO:tasks.workunit.client.1.smithi134.stdout:9/184: mknod d0/d7/d12/d22/c3f 0 2024-04-22T23:23:40.715 INFO:tasks.workunit.client.0.smithi067.stdout:2/975: link d1/d10/d33/f139 d1/dbf/f140 0 2024-04-22T23:23:40.715 INFO:tasks.workunit.client.1.smithi134.stdout:5/256: symlink d9/dd/d52/l55 0 2024-04-22T23:23:40.715 INFO:tasks.workunit.client.1.smithi134.stdout:9/185: fdatasync d0/d7/d23/f21 0 2024-04-22T23:23:40.716 INFO:tasks.workunit.client.1.smithi134.stdout:3/213: symlink dc/l53 0 2024-04-22T23:23:40.716 INFO:tasks.workunit.client.1.smithi134.stdout:9/186: chown d0/d7/l13 111365340 1 2024-04-22T23:23:40.721 INFO:tasks.workunit.client.1.smithi134.stdout:5/257: mknod d9/dd/d17/d2a/c56 0 2024-04-22T23:23:40.721 INFO:tasks.workunit.client.0.smithi067.stdout:2/976: rename d1/d42/d5f/d122/d53/d9a/d10a/ccd to d1/d10/d1c/ddc/c141 0 2024-04-22T23:23:40.721 INFO:tasks.workunit.client.0.smithi067.stdout:8/968: write d1/d3/d27/f6e [3414961,37690] 0 2024-04-22T23:23:40.722 INFO:tasks.workunit.client.1.smithi134.stdout:0/226: creat d1/f44 x:0 0 0 2024-04-22T23:23:40.724 INFO:tasks.workunit.client.1.smithi134.stdout:7/270: write d4/d8/d2a/f4f [605056,98363] 0 2024-04-22T23:23:40.724 INFO:tasks.workunit.client.0.smithi067.stdout:2/977: creat d1/d42/d5f/d122/f142 x:0 0 0 2024-04-22T23:23:40.727 INFO:tasks.workunit.client.1.smithi134.stdout:3/214: mkdir dc/dd/d28/d2f/d54 0 2024-04-22T23:23:40.729 INFO:tasks.workunit.client.1.smithi134.stdout:5/258: dwrite d9/d1e/f35 [0,4194304] 0 2024-04-22T23:23:40.732 INFO:tasks.workunit.client.1.smithi134.stdout:3/215: unlink lb 0 2024-04-22T23:23:40.732 INFO:tasks.workunit.client.1.smithi134.stdout:0/227: creat d1/d5/dd/f45 x:0 0 0 2024-04-22T23:23:40.736 INFO:tasks.workunit.client.1.smithi134.stdout:6/267: dread d0/d19/de/d1f/f22 [0,4194304] 0 2024-04-22T23:23:40.736 INFO:tasks.workunit.client.1.smithi134.stdout:0/228: chown d1/d2/d1d/c36 3 1 2024-04-22T23:23:40.737 INFO:tasks.workunit.client.0.smithi067.stdout:2/978: dwrite d1/d19/d6f/f107 [0,4194304] 0 2024-04-22T23:23:40.738 INFO:tasks.workunit.client.1.smithi134.stdout:3/216: truncate dc/dd/d28/f39 874253 0 2024-04-22T23:23:40.738 INFO:tasks.workunit.client.1.smithi134.stdout:6/268: symlink d0/l67 0 2024-04-22T23:23:40.739 INFO:tasks.workunit.client.1.smithi134.stdout:0/229: creat d1/d12/f46 x:0 0 0 2024-04-22T23:23:40.740 INFO:tasks.workunit.client.1.smithi134.stdout:0/230: truncate d1/f44 1029045 0 2024-04-22T23:23:40.740 INFO:tasks.workunit.client.0.smithi067.stdout:3/942: sync 2024-04-22T23:23:40.742 INFO:tasks.workunit.client.1.smithi134.stdout:6/269: unlink d0/d19/de/d2f/c63 0 2024-04-22T23:23:40.743 INFO:tasks.workunit.client.1.smithi134.stdout:6/270: stat d0/d19/f1b 0 2024-04-22T23:23:40.743 INFO:tasks.workunit.client.1.smithi134.stdout:5/259: getdents d9/dd/d17 0 2024-04-22T23:23:40.743 INFO:tasks.workunit.client.1.smithi134.stdout:9/187: truncate d0/d7/d12/d22/f31 326541 0 2024-04-22T23:23:40.745 INFO:tasks.workunit.client.1.smithi134.stdout:6/271: creat d0/d19/db/f68 x:0 0 0 2024-04-22T23:23:40.746 INFO:tasks.workunit.client.1.smithi134.stdout:5/260: symlink d9/df/d20/l57 0 2024-04-22T23:23:40.746 INFO:tasks.workunit.client.0.smithi067.stdout:3/943: creat d3/d4/d28/d40/dfd/dfe/d4b/f144 x:0 0 0 2024-04-22T23:23:40.746 INFO:tasks.workunit.client.0.smithi067.stdout:2/979: dwrite d1/d42/d5f/d122/d53/d9a/d10a/dc5/fdd [0,4194304] 0 2024-04-22T23:23:40.746 INFO:tasks.workunit.client.1.smithi134.stdout:6/272: dread - d0/d19/de/d12/d2a/f5d zero size 2024-04-22T23:23:40.749 INFO:tasks.workunit.client.0.smithi067.stdout:2/980: mkdir d1/d10/d110/d143 0 2024-04-22T23:23:40.750 INFO:tasks.workunit.client.1.smithi134.stdout:9/188: mknod d0/d7/d23/d1b/c40 0 2024-04-22T23:23:40.750 INFO:tasks.workunit.client.0.smithi067.stdout:3/944: fdatasync d3/d4/d28/d40/dfd/dfe/d4b/f63 0 2024-04-22T23:23:40.751 INFO:tasks.workunit.client.0.smithi067.stdout:8/969: dread d1/d3/f11 [4194304,4194304] 0 2024-04-22T23:23:40.751 INFO:tasks.workunit.client.0.smithi067.stdout:2/981: write d1/d42/d5f/d122/d68/d7b/dce/f124 [510088,61575] 0 2024-04-22T23:23:40.753 INFO:tasks.workunit.client.0.smithi067.stdout:8/970: truncate d1/fc0 1764703 0 2024-04-22T23:23:40.754 INFO:tasks.workunit.client.0.smithi067.stdout:8/971: truncate d1/dd/ffb 918851 0 2024-04-22T23:23:40.754 INFO:tasks.workunit.client.0.smithi067.stdout:2/982: truncate d1/d42/d5f/d9b/dc0/fec 4791844 0 2024-04-22T23:23:40.756 INFO:tasks.workunit.client.0.smithi067.stdout:8/972: read - d1/d3/d5c/d98/fd8 zero size 2024-04-22T23:23:40.756 INFO:tasks.workunit.client.1.smithi134.stdout:5/261: dwrite d9/f15 [0,4194304] 0 2024-04-22T23:23:40.757 INFO:tasks.workunit.client.0.smithi067.stdout:8/973: fdatasync d1/dd/f95 0 2024-04-22T23:23:40.757 INFO:tasks.workunit.client.1.smithi134.stdout:6/273: dwrite d0/d19/db/f68 [0,4194304] 0 2024-04-22T23:23:40.757 INFO:tasks.workunit.client.0.smithi067.stdout:2/983: chown d1/d10/d41/cfa 584311947 1 2024-04-22T23:23:40.758 INFO:tasks.workunit.client.1.smithi134.stdout:5/262: readlink d9/df/l2f 0 2024-04-22T23:23:40.760 INFO:tasks.workunit.client.0.smithi067.stdout:8/974: chown d1/dd/d80/cf4 9096 1 2024-04-22T23:23:40.761 INFO:tasks.workunit.client.1.smithi134.stdout:5/263: creat d9/df/d20/f58 x:0 0 0 2024-04-22T23:23:40.762 INFO:tasks.workunit.client.1.smithi134.stdout:6/274: getdents d0/d19/de/d12/d2a 0 2024-04-22T23:23:40.764 INFO:tasks.workunit.client.1.smithi134.stdout:6/275: symlink d0/d19/de/d12/d5e/l69 0 2024-04-22T23:23:40.765 INFO:tasks.workunit.client.0.smithi067.stdout:8/975: mkdir d1/dd/d80/db9/d110/d62/ddf/d136 0 2024-04-22T23:23:40.770 INFO:tasks.workunit.client.1.smithi134.stdout:5/264: dwrite d9/dd/f10 [0,4194304] 0 2024-04-22T23:23:40.771 INFO:tasks.workunit.client.0.smithi067.stdout:2/984: dwrite d1/d10/d41/f130 [0,4194304] 0 2024-04-22T23:23:40.773 INFO:tasks.workunit.client.1.smithi134.stdout:5/265: chown d9/dd/d17/d44/l4b 16682134 1 2024-04-22T23:23:40.773 INFO:tasks.workunit.client.0.smithi067.stdout:3/945: dread d3/d4/d10/d1f/f3e [0,4194304] 0 2024-04-22T23:23:40.774 INFO:tasks.workunit.client.1.smithi134.stdout:5/266: chown d9/d1e/f32 0 1 2024-04-22T23:23:40.777 INFO:tasks.workunit.client.0.smithi067.stdout:2/985: symlink d1/d42/d5f/d122/d68/d7b/l144 0 2024-04-22T23:23:40.777 INFO:tasks.workunit.client.0.smithi067.stdout:8/976: dwrite d1/d3/d27/f58 [0,4194304] 0 2024-04-22T23:23:40.778 INFO:tasks.workunit.client.1.smithi134.stdout:1/191: sync 2024-04-22T23:23:40.778 INFO:tasks.workunit.client.1.smithi134.stdout:5/267: write d9/dd/f13 [7895691,14421] 0 2024-04-22T23:23:40.779 INFO:tasks.workunit.client.0.smithi067.stdout:2/986: chown d1/d42/d5f/d122/c114 477336 1 2024-04-22T23:23:40.781 INFO:tasks.workunit.client.1.smithi134.stdout:1/192: creat db/d15/d17/d20/d2a/d2d/f48 x:0 0 0 2024-04-22T23:23:40.782 INFO:tasks.workunit.client.1.smithi134.stdout:1/193: fsync db/f12 0 2024-04-22T23:23:40.786 INFO:tasks.workunit.client.0.smithi067.stdout:2/987: dread d1/d12/f43 [0,4194304] 0 2024-04-22T23:23:40.786 INFO:tasks.workunit.client.0.smithi067.stdout:8/977: rename d1/f76 to d1/d3/d10/f137 0 2024-04-22T23:23:40.786 INFO:tasks.workunit.client.1.smithi134.stdout:1/194: mknod db/d15/d17/d20/d2a/d2d/c49 0 2024-04-22T23:23:40.787 INFO:tasks.workunit.client.0.smithi067.stdout:8/978: read - d1/dd/d80/d6f/dfc/fc5 zero size 2024-04-22T23:23:40.788 INFO:tasks.workunit.client.1.smithi134.stdout:5/268: dwrite d9/df/d20/f39 [0,4194304] 0 2024-04-22T23:23:40.790 INFO:tasks.workunit.client.1.smithi134.stdout:1/195: symlink db/d15/d17/l4a 0 2024-04-22T23:23:40.791 INFO:tasks.workunit.client.0.smithi067.stdout:8/979: rename d1/dd/d80/db9/d110/f6b to d1/dd/d80/db9/d110/f138 0 2024-04-22T23:23:40.791 INFO:tasks.workunit.client.1.smithi134.stdout:5/269: creat d9/dd/d17/d44/f59 x:0 0 0 2024-04-22T23:23:40.793 INFO:tasks.workunit.client.0.smithi067.stdout:8/980: mkdir d1/d3/d5c/d98/d139 0 2024-04-22T23:23:40.796 INFO:tasks.workunit.client.0.smithi067.stdout:8/981: fsync d1/dd/d80/db9/d110/d62/d94/fa3 0 2024-04-22T23:23:40.797 INFO:tasks.workunit.client.0.smithi067.stdout:3/946: dread d3/f35 [0,4194304] 0 2024-04-22T23:23:40.800 INFO:tasks.workunit.client.0.smithi067.stdout:3/947: mknod d3/d4/d10/d38/d53/d55/d10e/c145 0 2024-04-22T23:23:40.804 INFO:tasks.workunit.client.0.smithi067.stdout:8/982: dwrite d1/dd/d80/db9/d110/d62/d75/f109 [0,4194304] 0 2024-04-22T23:23:40.820 INFO:tasks.workunit.client.0.smithi067.stdout:2/988: read d1/d10/d33/f3e [3327492,42697] 0 2024-04-22T23:23:40.821 INFO:tasks.workunit.client.0.smithi067.stdout:2/989: chown d1/d10/d33/d46/da4/dde/lf9 43 1 2024-04-22T23:23:40.823 INFO:tasks.workunit.client.0.smithi067.stdout:3/948: dread d3/d4/d28/d5f/fa9 [0,4194304] 0 2024-04-22T23:23:40.823 INFO:tasks.workunit.client.0.smithi067.stdout:2/990: symlink d1/d10/d1c/d6d/d87/l145 0 2024-04-22T23:23:40.827 INFO:tasks.workunit.client.1.smithi134.stdout:5/270: link d9/dd/f4c d9/d1e/f5a 0 2024-04-22T23:23:40.829 INFO:tasks.workunit.client.0.smithi067.stdout:2/991: dwrite d1/d42/d5f/d9b/dc0/f10b [0,4194304] 0 2024-04-22T23:23:40.830 INFO:tasks.workunit.client.0.smithi067.stdout:2/992: rmdir d1/d19/d6f 39 2024-04-22T23:23:40.838 INFO:tasks.workunit.client.1.smithi134.stdout:5/271: dwrite d9/d11/d34/d3b/f53 [0,4194304] 0 2024-04-22T23:23:40.840 INFO:tasks.workunit.client.1.smithi134.stdout:5/272: chown d9/d1e/c2c 99 1 2024-04-22T23:23:40.849 INFO:tasks.workunit.client.1.smithi134.stdout:5/273: dwrite d9/d11/d34/f54 [0,4194304] 0 2024-04-22T23:23:40.849 INFO:tasks.workunit.client.1.smithi134.stdout:5/274: chown d9/d11/d34 1529952 1 2024-04-22T23:23:40.852 INFO:tasks.workunit.client.1.smithi134.stdout:5/275: creat d9/d11/f5b x:0 0 0 2024-04-22T23:23:40.857 INFO:tasks.workunit.client.1.smithi134.stdout:5/276: link d9/d11/c42 d9/d11/d34/d4d/c5c 0 2024-04-22T23:23:40.860 INFO:tasks.workunit.client.0.smithi067.stdout:2/993: dread d1/d42/d5f/d122/d53/fae [0,4194304] 0 2024-04-22T23:23:40.861 INFO:tasks.workunit.client.1.smithi134.stdout:5/277: unlink d9/d1e/c2d 0 2024-04-22T23:23:40.864 INFO:tasks.workunit.client.0.smithi067.stdout:5/996: sync 2024-04-22T23:23:40.864 INFO:tasks.workunit.client.0.smithi067.stdout:1/914: sync 2024-04-22T23:23:40.868 INFO:tasks.workunit.client.0.smithi067.stdout:1/915: read d8/d9/d2d/d4d/d48/fce [939908,84435] 0 2024-04-22T23:23:40.871 INFO:tasks.workunit.client.1.smithi134.stdout:5/278: dwrite d9/dd/f13 [8388608,4194304] 0 2024-04-22T23:23:40.872 INFO:tasks.workunit.client.1.smithi134.stdout:9/189: write d0/d7/f2c [2252243,86877] 0 2024-04-22T23:23:40.873 INFO:tasks.workunit.client.0.smithi067.stdout:8/983: write d1/d3/d10/fc1 [321071,118553] 0 2024-04-22T23:23:40.874 INFO:tasks.workunit.client.0.smithi067.stdout:1/916: link d8/d9/l16 d8/d9/d2d/l127 0 2024-04-22T23:23:40.875 INFO:tasks.workunit.client.1.smithi134.stdout:1/196: dread db/f27 [4194304,4194304] 0 2024-04-22T23:23:40.876 INFO:tasks.workunit.client.0.smithi067.stdout:1/917: chown d8/d9/d2d/d4d/d48/d9a/cb3 48240153 1 2024-04-22T23:23:40.877 INFO:tasks.workunit.client.1.smithi134.stdout:0/231: dwrite d1/d12/d22/f24 [0,4194304] 0 2024-04-22T23:23:40.879 INFO:tasks.workunit.client.0.smithi067.stdout:2/994: write d1/d10/f24 [314553,52247] 0 2024-04-22T23:23:40.880 INFO:tasks.workunit.client.1.smithi134.stdout:1/197: chown db/d15/d17/c2b 6436947 1 2024-04-22T23:23:40.880 INFO:tasks.workunit.client.1.smithi134.stdout:5/279: unlink d9/dd/f13 0 2024-04-22T23:23:40.883 INFO:tasks.workunit.client.1.smithi134.stdout:0/232: chown d1/d5/dd/f40 1108644190 1 2024-04-22T23:23:40.884 INFO:tasks.workunit.client.0.smithi067.stdout:1/918: link d8/d9/d2d/d4d/d48/f8d d8/d9/d102/f128 0 2024-04-22T23:23:40.884 INFO:tasks.workunit.client.1.smithi134.stdout:6/276: dwrite d0/f16 [0,4194304] 0 2024-04-22T23:23:40.888 INFO:tasks.workunit.client.1.smithi134.stdout:1/198: creat db/d15/d17/d20/f4b x:0 0 0 2024-04-22T23:23:40.889 INFO:tasks.workunit.client.1.smithi134.stdout:9/190: creat d0/d7/f41 x:0 0 0 2024-04-22T23:23:40.890 INFO:tasks.workunit.client.0.smithi067.stdout:8/984: write d1/dd/f79 [3241075,33160] 0 2024-04-22T23:23:40.892 INFO:tasks.workunit.client.1.smithi134.stdout:9/191: write d0/d7/d23/f2a [1660780,46018] 0 2024-04-22T23:23:40.894 INFO:tasks.workunit.client.0.smithi067.stdout:8/985: unlink d1/d3/l51 0 2024-04-22T23:23:40.895 INFO:tasks.workunit.client.1.smithi134.stdout:9/192: write d0/d7/d23/f2a [4159072,98879] 0 2024-04-22T23:23:40.895 INFO:tasks.workunit.client.1.smithi134.stdout:9/193: stat d0/d7/fb 0 2024-04-22T23:23:40.897 INFO:tasks.workunit.client.1.smithi134.stdout:1/199: symlink db/d15/d17/d20/l4c 0 2024-04-22T23:23:40.897 INFO:tasks.workunit.client.0.smithi067.stdout:2/995: dwrite d1/d10/d33/d46/f6e [0,4194304] 0 2024-04-22T23:23:40.901 INFO:tasks.workunit.client.1.smithi134.stdout:6/277: link d0/d19/de/d2f/l54 d0/d19/de/d12/d2a/d2d/l6a 0 2024-04-22T23:23:40.902 INFO:tasks.workunit.client.1.smithi134.stdout:6/278: chown d0/d4f/d51 238 1 2024-04-22T23:23:40.903 INFO:tasks.workunit.client.1.smithi134.stdout:6/279: write d0/f10 [1900857,21574] 0 2024-04-22T23:23:40.905 INFO:tasks.workunit.client.1.smithi134.stdout:1/200: getdents db/d15/d17/d20/d2a/d2d/d2e/d42 0 2024-04-22T23:23:40.906 INFO:tasks.workunit.client.0.smithi067.stdout:2/996: dwrite d1/d10/d33/d46/fea [0,4194304] 0 2024-04-22T23:23:40.907 INFO:tasks.workunit.client.1.smithi134.stdout:1/201: fsync db/d15/f34 0 2024-04-22T23:23:40.907 INFO:tasks.workunit.client.0.smithi067.stdout:5/997: dread d5/d28/daf/dbf/dcf/db2/f142 [0,4194304] 0 2024-04-22T23:23:40.908 INFO:tasks.workunit.client.0.smithi067.stdout:2/997: chown d1/d42/d5f/d9b/ld6 39384 1 2024-04-22T23:23:40.909 INFO:tasks.workunit.client.1.smithi134.stdout:1/202: chown db/l37 9681 1 2024-04-22T23:23:40.912 INFO:tasks.workunit.client.1.smithi134.stdout:1/203: dread - db/d15/d17/d20/d2a/d2d/d2e/f47 zero size 2024-04-22T23:23:40.912 INFO:tasks.workunit.client.1.smithi134.stdout:6/280: dwrite d0/f55 [0,4194304] 0 2024-04-22T23:23:40.912 INFO:tasks.workunit.client.0.smithi067.stdout:5/998: mkdir d5/db/d12/d44/d47/d15e 0 2024-04-22T23:23:40.913 INFO:tasks.workunit.client.0.smithi067.stdout:8/986: write d1/d3/d10/d3b/fac [1621231,98646] 0 2024-04-22T23:23:40.914 INFO:tasks.workunit.client.1.smithi134.stdout:5/280: write d9/d11/f1b [8724797,69744] 0 2024-04-22T23:23:40.915 INFO:tasks.workunit.client.0.smithi067.stdout:1/919: dread d8/d12/f94 [0,4194304] 0 2024-04-22T23:23:40.915 INFO:tasks.workunit.client.0.smithi067.stdout:2/998: getdents d1/d10/d1c/dd9/d11f 0 2024-04-22T23:23:40.915 INFO:tasks.workunit.client.1.smithi134.stdout:1/204: truncate db/f27 9384335 0 2024-04-22T23:23:40.916 INFO:tasks.workunit.client.0.smithi067.stdout:1/920: write d8/d4a/f5a [436083,96600] 0 2024-04-22T23:23:40.918 INFO:tasks.workunit.client.0.smithi067.stdout:8/987: mkdir d1/d41/d46/d13a 0 2024-04-22T23:23:40.918 INFO:tasks.workunit.client.0.smithi067.stdout:1/921: fsync d8/d9/f29 0 2024-04-22T23:23:40.919 INFO:tasks.workunit.client.0.smithi067.stdout:2/999: symlink d1/d10/d110/d12d/l146 0 2024-04-22T23:23:40.920 INFO:tasks.workunit.client.1.smithi134.stdout:0/233: truncate d1/d5/dd/d20/f1e 2127728 0 2024-04-22T23:23:40.920 INFO:tasks.workunit.client.1.smithi134.stdout:1/205: mkdir db/d15/d17/d20/d2a/d4d 0 2024-04-22T23:23:40.922 INFO:tasks.workunit.client.1.smithi134.stdout:1/206: write db/d16/f3d [20929,89592] 0 2024-04-22T23:23:40.923 INFO:tasks.workunit.client.1.smithi134.stdout:5/281: unlink d9/d11/f21 0 2024-04-22T23:23:40.927 INFO:tasks.workunit.client.1.smithi134.stdout:9/194: truncate d0/d7/f17 2510334 0 2024-04-22T23:23:40.931 INFO:tasks.workunit.client.1.smithi134.stdout:9/195: creat d0/d7/d12/f42 x:0 0 0 2024-04-22T23:23:40.932 INFO:tasks.workunit.client.1.smithi134.stdout:1/207: getdents db/d15/d17/d20/d2a/d2d/d2e 0 2024-04-22T23:23:40.932 INFO:tasks.workunit.client.1.smithi134.stdout:9/196: fdatasync d0/d1c/f3b 0 2024-04-22T23:23:40.934 INFO:tasks.workunit.client.0.smithi067.stdout:1/922: write d8/d12/d36/fba [2125356,14750] 0 2024-04-22T23:23:40.937 INFO:tasks.workunit.client.0.smithi067.stdout:1/923: fsync d8/d12/d40/fa4 0 2024-04-22T23:23:40.937 INFO:tasks.workunit.client.1.smithi134.stdout:1/208: rename db/d15/l38 to db/d15/d17/d20/l4e 0 2024-04-22T23:23:40.938 INFO:tasks.workunit.client.1.smithi134.stdout:9/197: creat d0/d2d/d3d/f43 x:0 0 0 2024-04-22T23:23:40.939 INFO:tasks.workunit.client.0.smithi067.stdout:8/988: dwrite d1/dd/d80/d6f/f50 [4194304,4194304] 0 2024-04-22T23:23:40.940 INFO:tasks.workunit.client.0.smithi067.stdout:1/924: fdatasync d8/d12/f17 0 2024-04-22T23:23:40.940 INFO:tasks.workunit.client.1.smithi134.stdout:6/281: truncate d0/d19/f38 413321 0 2024-04-22T23:23:40.941 INFO:tasks.workunit.client.1.smithi134.stdout:9/198: creat d0/d2d/f44 x:0 0 0 2024-04-22T23:23:40.943 INFO:tasks.workunit.client.1.smithi134.stdout:1/209: symlink db/l4f 0 2024-04-22T23:23:40.943 INFO:tasks.workunit.client.0.smithi067.stdout:1/925: write d8/d9/d2d/d4d/d9c/ff9 [312675,112456] 0 2024-04-22T23:23:40.944 INFO:tasks.workunit.client.0.smithi067.stdout:1/926: chown d8/d12/d40/f57 3273481 1 2024-04-22T23:23:40.944 INFO:tasks.workunit.client.1.smithi134.stdout:9/199: truncate d0/d1c/f3b 400778 0 2024-04-22T23:23:40.946 INFO:tasks.workunit.client.1.smithi134.stdout:9/200: write d0/d7/f2c [1964364,45726] 0 2024-04-22T23:23:40.947 INFO:tasks.workunit.client.0.smithi067.stdout:1/927: mknod d8/d9/de0/d5d/de2/c129 0 2024-04-22T23:23:40.947 INFO:tasks.workunit.client.0.smithi067.stdout:5/999: truncate d5/db/d12/d44/d47/f4f 672851 0 2024-04-22T23:23:40.947 INFO:tasks.workunit.client.0.smithi067.stdout:8/989: link d1/dd/d80/db9/d110/f55 d1/dd/d80/db9/de3/db8/dd7/f13b 0 2024-04-22T23:23:40.948 INFO:tasks.workunit.client.0.smithi067.stdout:1/928: chown d8/d9/d2d/d4d/d9c/f6c 7 1 2024-04-22T23:23:40.949 INFO:tasks.workunit.client.0.smithi067.stdout:8/990: chown d1/d3/d27/d107/f126 10944 1 2024-04-22T23:23:40.949 INFO:tasks.workunit.client.1.smithi134.stdout:0/234: truncate d1/d5/dd/f16 1869881 0 2024-04-22T23:23:40.951 INFO:tasks.workunit.client.0.smithi067.stdout:8/991: fsync d1/d41/d46/fa2 0 2024-04-22T23:23:40.952 INFO:tasks.workunit.client.0.smithi067.stdout:1/929: rename d8/d9/d2d/d4d/d9c/ccb to d8/d9/de0/d5d/de2/c12a 0 2024-04-22T23:23:40.953 INFO:tasks.workunit.client.1.smithi134.stdout:9/201: dread d0/d7/d12/d22/f31 [0,4194304] 0 2024-04-22T23:23:40.954 INFO:tasks.workunit.client.1.smithi134.stdout:1/210: dwrite db/d15/d17/d20/f4b [0,4194304] 0 2024-04-22T23:23:40.954 INFO:tasks.workunit.client.0.smithi067.stdout:1/930: mkdir d8/d9/d12b 0 2024-04-22T23:23:40.956 INFO:tasks.workunit.client.1.smithi134.stdout:9/202: write d0/d7/fd [4664875,73622] 0 2024-04-22T23:23:40.956 INFO:tasks.workunit.client.0.smithi067.stdout:8/992: rename d1/d3/f59 to d1/d41/f13c 0 2024-04-22T23:23:40.959 INFO:tasks.workunit.client.1.smithi134.stdout:0/235: truncate d1/f1c 810660 0 2024-04-22T23:23:40.961 INFO:tasks.workunit.client.0.smithi067.stdout:1/931: creat d8/d12/f12c x:0 0 0 2024-04-22T23:23:40.962 INFO:tasks.workunit.client.1.smithi134.stdout:6/282: link d0/d19/de/d12/d2a/d2d/d3a/c57 d0/d19/de/d1f/c6b 0 2024-04-22T23:23:40.963 INFO:tasks.workunit.client.1.smithi134.stdout:9/203: dwrite d0/d7/fd [4194304,4194304] 0 2024-04-22T23:23:40.963 INFO:tasks.workunit.client.0.smithi067.stdout:8/993: dwrite d1/d41/d46/f9f [0,4194304] 0 2024-04-22T23:23:40.965 INFO:tasks.workunit.client.0.smithi067.stdout:8/994: write d1/dd/d80/db9/d110/d62/fa5 [2883240,32085] 0 2024-04-22T23:23:40.967 INFO:tasks.workunit.client.0.smithi067.stdout:1/932: read d8/d9/d2d/d4d/d9c/f6c [1723075,100052] 0 2024-04-22T23:23:40.968 INFO:tasks.workunit.client.1.smithi134.stdout:1/211: mknod db/c50 0 2024-04-22T23:23:40.970 INFO:tasks.workunit.client.0.smithi067.stdout:9/898: sync 2024-04-22T23:23:40.970 INFO:tasks.workunit.client.0.smithi067.stdout:7/947: sync 2024-04-22T23:23:40.972 INFO:tasks.workunit.client.1.smithi134.stdout:6/283: mknod d0/d19/de/d12/d5e/c6c 0 2024-04-22T23:23:40.973 INFO:tasks.workunit.client.0.smithi067.stdout:7/948: truncate d10/d14/d1d/d26/d2c/d41/db0/f14e 805874 0 2024-04-22T23:23:40.974 INFO:tasks.workunit.client.1.smithi134.stdout:1/212: creat db/d15/d17/d20/d2a/f51 x:0 0 0 2024-04-22T23:23:40.974 INFO:tasks.workunit.client.1.smithi134.stdout:0/236: write d1/d5/dd/f31 [1091659,24748] 0 2024-04-22T23:23:40.975 INFO:tasks.workunit.client.0.smithi067.stdout:9/899: rename d5/d21/d47/d7f/ce4 to d5/d21/d47/d56/d8c/df8/c129 0 2024-04-22T23:23:40.976 INFO:tasks.workunit.client.1.smithi134.stdout:1/213: read - db/d15/d17/d20/d2a/d2d/f48 zero size 2024-04-22T23:23:40.978 INFO:tasks.workunit.client.1.smithi134.stdout:0/237: creat d1/d5/f47 x:0 0 0 2024-04-22T23:23:40.980 INFO:tasks.workunit.client.0.smithi067.stdout:9/900: mkdir d5/dd/d18/d39/d3e/d3f/d63/d12a 0 2024-04-22T23:23:40.980 INFO:tasks.workunit.client.1.smithi134.stdout:1/214: symlink db/d15/d17/d20/d2a/d2d/d2e/l52 0 2024-04-22T23:23:40.981 INFO:tasks.workunit.client.0.smithi067.stdout:9/901: chown d5/dd/f26 2808 1 2024-04-22T23:23:40.982 INFO:tasks.workunit.client.1.smithi134.stdout:6/284: getdents d0/d19/db/d33 0 2024-04-22T23:23:40.983 INFO:tasks.workunit.client.1.smithi134.stdout:6/285: readlink d0/d19/de/d12/d5e/l69 0 2024-04-22T23:23:40.983 INFO:tasks.workunit.client.0.smithi067.stdout:7/949: dwrite d10/d14/d1d/d26/f5d [0,4194304] 0 2024-04-22T23:23:40.983 INFO:tasks.workunit.client.1.smithi134.stdout:6/286: write d0/d19/db/f68 [1357096,44721] 0 2024-04-22T23:23:40.984 INFO:tasks.workunit.client.1.smithi134.stdout:6/287: chown d0/d19/db/lc 0 1 2024-04-22T23:23:40.984 INFO:tasks.workunit.client.1.smithi134.stdout:1/215: symlink db/d15/d17/l53 0 2024-04-22T23:23:40.987 INFO:tasks.workunit.client.0.smithi067.stdout:7/950: mknod d10/d14/d1d/d26/d2c/d41/c162 0 2024-04-22T23:23:40.993 INFO:tasks.workunit.client.0.smithi067.stdout:7/951: dwrite d10/d14/d1d/d26/f39 [0,4194304] 0 2024-04-22T23:23:40.994 INFO:tasks.workunit.client.0.smithi067.stdout:3/949: sync 2024-04-22T23:23:40.994 INFO:tasks.workunit.client.0.smithi067.stdout:6/975: sync 2024-04-22T23:23:40.997 INFO:tasks.workunit.client.0.smithi067.stdout:3/950: symlink d3/d4/d10/d10b/d95/dd6/l146 0 2024-04-22T23:23:40.999 INFO:tasks.workunit.client.0.smithi067.stdout:7/952: fsync d10/d14/d1d/d26/d7e/fb9 0 2024-04-22T23:23:41.002 INFO:tasks.workunit.client.0.smithi067.stdout:7/953: symlink d10/d14/d1d/d26/d2c/d7a/dc2/dd0/l163 0 2024-04-22T23:23:41.003 INFO:tasks.workunit.client.1.smithi134.stdout:1/216: creat db/d15/f54 x:0 0 0 2024-04-22T23:23:41.004 INFO:tasks.workunit.client.0.smithi067.stdout:7/954: rmdir d10/d14/d1d/d26/d2c/d7a/dc2/d121 39 2024-04-22T23:23:41.005 INFO:tasks.workunit.client.0.smithi067.stdout:7/955: fsync d10/d47/d49/f9c 0 2024-04-22T23:23:41.006 INFO:tasks.workunit.client.1.smithi134.stdout:6/288: unlink d0/d19/de/d1f/l28 0 2024-04-22T23:23:41.006 INFO:tasks.workunit.client.0.smithi067.stdout:7/956: chown d10/d14/d1d/d12c/f139 1010940 1 2024-04-22T23:23:41.008 INFO:tasks.workunit.client.0.smithi067.stdout:7/957: symlink d10/d18/d88/daf/l164 0 2024-04-22T23:23:41.008 INFO:tasks.workunit.client.1.smithi134.stdout:0/238: getdents d1/d5/dd/d20/d2c 0 2024-04-22T23:23:41.010 INFO:tasks.workunit.client.0.smithi067.stdout:7/958: write d10/d14/d1d/d26/d80/dd2/dd4/d115/f14b [804114,59477] 0 2024-04-22T23:23:41.015 INFO:tasks.workunit.client.0.smithi067.stdout:7/959: truncate d10/d14/d1d/d26/d2c/d41/fa1 4553894 0 2024-04-22T23:23:41.017 INFO:tasks.workunit.client.1.smithi134.stdout:1/217: dwrite db/d16/f3d [0,4194304] 0 2024-04-22T23:23:41.017 INFO:tasks.workunit.client.0.smithi067.stdout:6/976: write d17/d39/d53/d90/d46/db0/db9/de7/fe1 [94379,97111] 0 2024-04-22T23:23:41.019 INFO:tasks.workunit.client.1.smithi134.stdout:6/289: dwrite d0/d19/db/d33/f64 [0,4194304] 0 2024-04-22T23:23:41.020 INFO:tasks.workunit.client.1.smithi134.stdout:0/239: dread d1/d5/dd/d20/f1e [0,4194304] 0 2024-04-22T23:23:41.020 INFO:tasks.workunit.client.1.smithi134.stdout:6/290: chown d0/l20 530 1 2024-04-22T23:23:41.020 INFO:tasks.workunit.client.0.smithi067.stdout:3/951: truncate d3/d4/d1b/d93/f10c 2890671 0 2024-04-22T23:23:41.021 INFO:tasks.workunit.client.1.smithi134.stdout:1/218: mknod db/c55 0 2024-04-22T23:23:41.022 INFO:tasks.workunit.client.1.smithi134.stdout:1/219: chown db/d15/f18 216906177 1 2024-04-22T23:23:41.022 INFO:tasks.workunit.client.1.smithi134.stdout:6/291: truncate d0/d19/de/d12/d2a/f4c 932740 0 2024-04-22T23:23:41.024 INFO:tasks.workunit.client.1.smithi134.stdout:6/292: chown d0/l67 18 1 2024-04-22T23:23:41.024 INFO:tasks.workunit.client.0.smithi067.stdout:8/995: dwrite d1/d3/d5c/d98/fbc [0,4194304] 0 2024-04-22T23:23:41.025 INFO:tasks.workunit.client.1.smithi134.stdout:0/240: symlink d1/d5/dd/d41/l48 0 2024-04-22T23:23:41.025 INFO:tasks.workunit.client.0.smithi067.stdout:6/977: read d17/d39/d3a/fe0 [2883935,41461] 0 2024-04-22T23:23:41.026 INFO:tasks.workunit.client.0.smithi067.stdout:1/933: dwrite d8/d9/de0/f47 [0,4194304] 0 2024-04-22T23:23:41.027 INFO:tasks.workunit.client.1.smithi134.stdout:1/220: mkdir db/d16/d56 0 2024-04-22T23:23:41.027 INFO:tasks.workunit.client.1.smithi134.stdout:6/293: fdatasync d0/d19/db/d35/f58 0 2024-04-22T23:23:41.031 INFO:tasks.workunit.client.1.smithi134.stdout:6/294: mkdir d0/d19/de/d12/d5e/d6d 0 2024-04-22T23:23:41.032 INFO:tasks.workunit.client.0.smithi067.stdout:9/902: dwrite d5/d21/d47/d56/dc6/d103/d114/f127 [4194304,4194304] 0 2024-04-22T23:23:41.034 INFO:tasks.workunit.client.0.smithi067.stdout:8/996: read d1/dd/d80/d6f/fad [2699747,93072] 0 2024-04-22T23:23:41.036 INFO:tasks.workunit.client.0.smithi067.stdout:9/903: fsync d5/d21/d47/d56/d8c/dd5/fe2 0 2024-04-22T23:23:41.036 INFO:tasks.workunit.client.0.smithi067.stdout:8/997: fdatasync d1/d41/f7d 0 2024-04-22T23:23:41.038 INFO:tasks.workunit.client.0.smithi067.stdout:9/904: write d5/d21/d47/d7f/f8e [1072685,40983] 0 2024-04-22T23:23:41.041 INFO:tasks.workunit.client.0.smithi067.stdout:1/934: rmdir d8/d9/de0/d5d/dd4 39 2024-04-22T23:23:41.042 INFO:tasks.workunit.client.0.smithi067.stdout:9/905: dread - d5/d21/d47/d56/d8c/dd5/fdd zero size 2024-04-22T23:23:41.049 INFO:tasks.workunit.client.0.smithi067.stdout:6/978: rename d17/d83/db5/dbc/dd8/d10c/d129 to d17/d39/d53/d90/d46/db0/db9/d149 0 2024-04-22T23:23:41.051 INFO:tasks.workunit.client.1.smithi134.stdout:0/241: truncate d1/d5/dd/f16 1851031 0 2024-04-22T23:23:41.052 INFO:tasks.workunit.client.1.smithi134.stdout:0/242: readlink d1/d2/l37 0 2024-04-22T23:23:41.052 INFO:tasks.workunit.client.0.smithi067.stdout:9/906: symlink d5/d21/d47/d56/d8c/l12b 0 2024-04-22T23:23:41.053 INFO:tasks.workunit.client.1.smithi134.stdout:0/243: fdatasync d1/f44 0 2024-04-22T23:23:41.053 INFO:tasks.workunit.client.0.smithi067.stdout:8/998: truncate d1/dd/d80/d6f/db6/fcd 3162504 0 2024-04-22T23:23:41.055 INFO:tasks.workunit.client.1.smithi134.stdout:0/244: mknod d1/db/c49 0 2024-04-22T23:23:41.055 INFO:tasks.workunit.client.1.smithi134.stdout:0/245: readlink d1/d5/dd/l3f 0 2024-04-22T23:23:41.056 INFO:tasks.workunit.client.0.smithi067.stdout:6/979: creat d17/d83/f14a x:0 0 0 2024-04-22T23:23:41.057 INFO:tasks.workunit.client.0.smithi067.stdout:9/907: readlink d5/dd/d18/d3c/d78/leb 0 2024-04-22T23:23:41.058 INFO:tasks.workunit.client.1.smithi134.stdout:0/246: dread d1/d5/dd/d20/f1e [0,4194304] 0 2024-04-22T23:23:41.058 INFO:tasks.workunit.client.1.smithi134.stdout:0/247: read - d1/d12/f46 zero size 2024-04-22T23:23:41.059 INFO:tasks.workunit.client.1.smithi134.stdout:0/248: fdatasync d1/d5/dd/f2a 0 2024-04-22T23:23:41.060 INFO:tasks.workunit.client.1.smithi134.stdout:0/249: chown d1/d2/f38 524 1 2024-04-22T23:23:41.062 INFO:tasks.workunit.client.0.smithi067.stdout:8/999: dwrite d1/dd/d80/db9/fe9 [0,4194304] 0 2024-04-22T23:23:41.062 INFO:tasks.workunit.client.1.smithi134.stdout:0/250: creat d1/d5/dd/d41/f4a x:0 0 0 2024-04-22T23:23:41.063 INFO:tasks.workunit.client.0.smithi067.stdout:9/908: unlink d5/dd/d18/d5c/ff5 0 2024-04-22T23:23:41.064 INFO:tasks.workunit.client.1.smithi134.stdout:1/221: dread fa [0,4194304] 0 2024-04-22T23:23:41.065 INFO:tasks.workunit.client.1.smithi134.stdout:6/295: dread d0/d19/f9 [0,4194304] 0 2024-04-22T23:23:41.067 INFO:tasks.workunit.client.0.smithi067.stdout:3/952: write d3/d4/d1b/d93/fa0 [478862,46710] 0 2024-04-22T23:23:41.068 INFO:tasks.workunit.client.1.smithi134.stdout:0/251: rename d1/d5/dd/d41/f4a to d1/d5/dd/d20/d2c/f4b 0 2024-04-22T23:23:41.069 INFO:tasks.workunit.client.1.smithi134.stdout:1/222: creat db/d16/d56/f57 x:0 0 0 2024-04-22T23:23:41.072 INFO:tasks.workunit.client.1.smithi134.stdout:6/296: mknod d0/d19/de/d12/c6e 0 2024-04-22T23:23:41.073 INFO:tasks.workunit.client.1.smithi134.stdout:1/223: symlink db/d15/d17/d20/d2a/d2d/l58 0 2024-04-22T23:23:41.074 INFO:tasks.workunit.client.0.smithi067.stdout:9/909: dwrite d5/d21/d47/d56/d8c/f11e [0,4194304] 0 2024-04-22T23:23:41.075 INFO:tasks.workunit.client.0.smithi067.stdout:7/960: dwrite d10/d47/fdc [0,4194304] 0 2024-04-22T23:23:41.075 INFO:tasks.workunit.client.0.smithi067.stdout:9/910: stat d5/dd/lb2 0 2024-04-22T23:23:41.076 INFO:tasks.workunit.client.1.smithi134.stdout:0/252: symlink d1/d5/dd/d11/l4c 0 2024-04-22T23:23:41.077 INFO:tasks.workunit.client.0.smithi067.stdout:7/961: chown d10/d14/d1d/d26/d7e/fb5 8388306 1 2024-04-22T23:23:41.078 INFO:tasks.workunit.client.1.smithi134.stdout:1/224: dread db/d15/d17/d20/f4b [0,4194304] 0 2024-04-22T23:23:41.078 INFO:tasks.workunit.client.1.smithi134.stdout:0/253: symlink d1/d5/dd/d41/l4d 0 2024-04-22T23:23:41.079 INFO:tasks.workunit.client.1.smithi134.stdout:1/225: chown db/f28 7449575 1 2024-04-22T23:23:41.080 INFO:tasks.workunit.client.0.smithi067.stdout:7/962: truncate d10/d18/d88/daf/d100/d62/fe8 255184 0 2024-04-22T23:23:41.080 INFO:tasks.workunit.client.1.smithi134.stdout:1/226: write db/d15/d17/d20/d2a/f51 [1044156,61059] 0 2024-04-22T23:23:41.081 INFO:tasks.workunit.client.0.smithi067.stdout:3/953: rmdir d3/d4/d1b/d93/d135 0 2024-04-22T23:23:41.082 INFO:tasks.workunit.client.1.smithi134.stdout:1/227: mknod db/d16/c59 0 2024-04-22T23:23:41.087 INFO:tasks.workunit.client.1.smithi134.stdout:1/228: dread db/f1b [0,4194304] 0 2024-04-22T23:23:41.088 INFO:tasks.workunit.client.0.smithi067.stdout:1/935: write d8/d9/d2d/d4d/d70/fd5 [863538,97798] 0 2024-04-22T23:23:41.089 INFO:tasks.workunit.client.1.smithi134.stdout:1/229: write db/d15/d17/d20/d2a/f44 [34121,15492] 0 2024-04-22T23:23:41.089 INFO:tasks.workunit.client.0.smithi067.stdout:9/911: creat d5/df6/f12c x:0 0 0 2024-04-22T23:23:41.093 INFO:tasks.workunit.client.0.smithi067.stdout:3/954: mknod d3/d4/d1b/d93/d142/c147 0 2024-04-22T23:23:41.095 INFO:tasks.workunit.client.0.smithi067.stdout:7/963: dwrite d10/d14/d1d/d26/d2c/f2e [0,4194304] 0 2024-04-22T23:23:41.095 INFO:tasks.workunit.client.0.smithi067.stdout:1/936: symlink d8/d9/d12b/l12d 0 2024-04-22T23:23:41.098 INFO:tasks.workunit.client.0.smithi067.stdout:7/964: readlink d10/d14/d1d/d26/d80/l12f 0 2024-04-22T23:23:41.101 INFO:tasks.workunit.client.1.smithi134.stdout:6/297: dread d0/f1 [0,4194304] 0 2024-04-22T23:23:41.101 INFO:tasks.workunit.client.1.smithi134.stdout:1/230: fsync db/d15/d17/d20/d2a/f44 0 2024-04-22T23:23:41.103 INFO:tasks.workunit.client.0.smithi067.stdout:3/955: fsync d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d8c/feb 0 2024-04-22T23:23:41.103 INFO:tasks.workunit.client.1.smithi134.stdout:6/298: mkdir d0/d6f 0 2024-04-22T23:23:41.103 INFO:tasks.workunit.client.0.smithi067.stdout:6/980: dwrite d17/d39/d53/d90/d46/fa1 [0,4194304] 0 2024-04-22T23:23:41.107 INFO:tasks.workunit.client.1.smithi134.stdout:6/299: creat d0/d19/de/d12/d29/f70 x:0 0 0 2024-04-22T23:23:41.109 INFO:tasks.workunit.client.0.smithi067.stdout:1/937: symlink d8/d9/de0/l12e 0 2024-04-22T23:23:41.110 INFO:tasks.workunit.client.0.smithi067.stdout:7/965: truncate d10/d14/d1d/f42 1427919 0 2024-04-22T23:23:41.112 INFO:tasks.workunit.client.1.smithi134.stdout:6/300: dread d0/d19/f38 [0,4194304] 0 2024-04-22T23:23:41.113 INFO:tasks.workunit.client.0.smithi067.stdout:3/956: creat d3/d4/d10/d38/d53/d55/d5c/d70/d10a/f148 x:0 0 0 2024-04-22T23:23:41.116 INFO:tasks.workunit.client.1.smithi134.stdout:6/301: readlink d0/d19/de/d12/d2a/l3d 0 2024-04-22T23:23:41.118 INFO:tasks.workunit.client.1.smithi134.stdout:6/302: creat d0/d19/de/d2f/f71 x:0 0 0 2024-04-22T23:23:41.119 INFO:tasks.workunit.client.0.smithi067.stdout:3/957: unlink d3/d4/cd1 0 2024-04-22T23:23:41.120 INFO:tasks.workunit.client.0.smithi067.stdout:3/958: write d3/d4/d28/d40/dfd/dfe/f69 [5111123,5366] 0 2024-04-22T23:23:41.120 INFO:tasks.workunit.client.0.smithi067.stdout:7/966: rename d10/d14/d1d/d26/f3a to d10/d14/d1d/d46/d50/de2/f165 0 2024-04-22T23:23:41.121 INFO:tasks.workunit.client.0.smithi067.stdout:3/959: chown d3/d4/d10/d19/c3d 311 1 2024-04-22T23:23:41.122 INFO:tasks.workunit.client.0.smithi067.stdout:7/967: dread - d10/d14/d1d/d24/f136 zero size 2024-04-22T23:23:41.122 INFO:tasks.workunit.client.0.smithi067.stdout:3/960: stat d3/d4/d1b/l65 0 2024-04-22T23:23:41.123 INFO:tasks.workunit.client.1.smithi134.stdout:1/231: dwrite db/f1d [0,4194304] 0 2024-04-22T23:23:41.123 INFO:tasks.workunit.client.1.smithi134.stdout:6/303: creat d0/d19/de/d12/d2a/d2d/f72 x:0 0 0 2024-04-22T23:23:41.123 INFO:tasks.workunit.client.0.smithi067.stdout:9/912: truncate d5/d21/d47/d56/dc6/d103/d114/f127 7975448 0 2024-04-22T23:23:41.124 INFO:tasks.workunit.client.1.smithi134.stdout:1/232: truncate db/f12 2481176 0 2024-04-22T23:23:41.125 INFO:tasks.workunit.client.1.smithi134.stdout:6/304: write d0/d19/de/d12/d29/f70 [554433,49374] 0 2024-04-22T23:23:41.126 INFO:tasks.workunit.client.0.smithi067.stdout:3/961: mkdir d3/d4/d28/dc0/d149 0 2024-04-22T23:23:41.127 INFO:tasks.workunit.client.1.smithi134.stdout:1/233: creat db/d15/d17/f5a x:0 0 0 2024-04-22T23:23:41.128 INFO:tasks.workunit.client.0.smithi067.stdout:3/962: dread - d3/d4/d10/d10b/d95/f13e zero size 2024-04-22T23:23:41.128 INFO:tasks.workunit.client.1.smithi134.stdout:6/305: creat d0/d19/de/d12/d5e/d6d/f73 x:0 0 0 2024-04-22T23:23:41.130 INFO:tasks.workunit.client.0.smithi067.stdout:3/963: readlink d3/d4/d10/d1f/d3c/l127 0 2024-04-22T23:23:41.130 INFO:tasks.workunit.client.1.smithi134.stdout:6/306: stat d0/d19/de/d12/d29/l47 0 2024-04-22T23:23:41.133 INFO:tasks.workunit.client.1.smithi134.stdout:6/307: readlink d0/d19/de/l56 0 2024-04-22T23:23:41.134 INFO:tasks.workunit.client.0.smithi067.stdout:7/968: rename d10/d14/d1d/d26/d2c/d41/cbc to d10/d14/d1d/d26/d2c/d7a/dc2/dd0/dde/c166 0 2024-04-22T23:23:41.134 INFO:tasks.workunit.client.0.smithi067.stdout:3/964: symlink d3/d4/d28/d4a/l14a 0 2024-04-22T23:23:41.136 INFO:tasks.workunit.client.1.smithi134.stdout:6/308: mkdir d0/d19/de/d12/d29/d74 0 2024-04-22T23:23:41.137 INFO:tasks.workunit.client.0.smithi067.stdout:6/981: dread d17/d39/d3a/f3b [0,4194304] 0 2024-04-22T23:23:41.137 INFO:tasks.workunit.client.0.smithi067.stdout:9/913: dread d5/d21/d47/d56/fec [0,4194304] 0 2024-04-22T23:23:41.137 INFO:tasks.workunit.client.0.smithi067.stdout:3/965: symlink d3/d4/d28/dc0/l14b 0 2024-04-22T23:23:41.137 INFO:tasks.workunit.client.1.smithi134.stdout:1/234: truncate db/d15/d17/d20/d2a/f36 1018316 0 2024-04-22T23:23:41.138 INFO:tasks.workunit.client.1.smithi134.stdout:1/235: chown db/d15/f18 12389903 1 2024-04-22T23:23:41.138 INFO:tasks.workunit.client.0.smithi067.stdout:9/914: chown d5/dd/d18/f1f 359 1 2024-04-22T23:23:41.139 INFO:tasks.workunit.client.1.smithi134.stdout:6/309: mkdir d0/d19/db/d75 0 2024-04-22T23:23:41.139 INFO:tasks.workunit.client.0.smithi067.stdout:1/938: dread d8/d9/d2f/f7d [0,4194304] 0 2024-04-22T23:23:41.140 INFO:tasks.workunit.client.0.smithi067.stdout:6/982: fsync d17/d83/db5/fd4 0 2024-04-22T23:23:41.142 INFO:tasks.workunit.client.0.smithi067.stdout:7/969: link d10/d14/d1d/d26/d2c/d41/f4f d10/d14/d1d/d46/d50/de2/f167 0 2024-04-22T23:23:41.143 INFO:tasks.workunit.client.0.smithi067.stdout:9/915: fdatasync d5/d21/d47/fa4 0 2024-04-22T23:23:41.144 INFO:tasks.workunit.client.1.smithi134.stdout:6/310: getdents d0/d19/de/d12/d2a/d2d/d3a 0 2024-04-22T23:23:41.146 INFO:tasks.workunit.client.0.smithi067.stdout:3/966: dwrite d3/d4/d10/fbc [0,4194304] 0 2024-04-22T23:23:41.147 INFO:tasks.workunit.client.1.smithi134.stdout:6/311: mknod d0/d19/db/c76 0 2024-04-22T23:23:41.148 INFO:tasks.workunit.client.0.smithi067.stdout:6/983: mknod d17/d83/d9b/ddd/c14b 0 2024-04-22T23:23:41.149 INFO:tasks.workunit.client.1.smithi134.stdout:6/312: write d0/d19/de/d12/d2a/d2d/f49 [1018789,17808] 0 2024-04-22T23:23:41.150 INFO:tasks.workunit.client.0.smithi067.stdout:3/967: chown d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d90/da2/daf 124628 1 2024-04-22T23:23:41.152 INFO:tasks.workunit.client.1.smithi134.stdout:1/236: dwrite db/d15/f18 [0,4194304] 0 2024-04-22T23:23:41.155 INFO:tasks.workunit.client.1.smithi134.stdout:1/237: creat db/d16/f5b x:0 0 0 2024-04-22T23:23:41.156 INFO:tasks.workunit.client.0.smithi067.stdout:9/916: unlink d5/dd/l13 0 2024-04-22T23:23:41.157 INFO:tasks.workunit.client.0.smithi067.stdout:6/984: readlink d17/l37 0 2024-04-22T23:23:41.158 INFO:tasks.workunit.client.0.smithi067.stdout:6/985: stat d17/d39/d53/f59 0 2024-04-22T23:23:41.159 INFO:tasks.workunit.client.1.smithi134.stdout:1/238: dread fa [0,4194304] 0 2024-04-22T23:23:41.160 INFO:tasks.workunit.client.0.smithi067.stdout:7/970: symlink d10/d14/d1d/d26/d2c/d41/db0/ddd/d14f/l168 0 2024-04-22T23:23:41.162 INFO:tasks.workunit.client.1.smithi134.stdout:1/239: mknod db/d16/d56/c5c 0 2024-04-22T23:23:41.162 INFO:tasks.workunit.client.0.smithi067.stdout:9/917: mkdir d5/d21/d47/d56/d8c/df8/d12d 0 2024-04-22T23:23:41.163 INFO:tasks.workunit.client.0.smithi067.stdout:7/971: chown d10/d14/d1d/d26/d2c/df4/f15c 15677981 1 2024-04-22T23:23:41.164 INFO:tasks.workunit.client.0.smithi067.stdout:3/968: mkdir d3/d4/d10/d38/d53/d55/d5c/d115/d14c 0 2024-04-22T23:23:41.167 INFO:tasks.workunit.client.0.smithi067.stdout:9/918: symlink d5/dd/d18/d39/d8f/l12e 0 2024-04-22T23:23:41.169 INFO:tasks.workunit.client.0.smithi067.stdout:1/939: write f1 [1737401,61698] 0 2024-04-22T23:23:41.170 INFO:tasks.workunit.client.0.smithi067.stdout:3/969: fsync d3/d4/d28/d40/dfd/dfe/d4b/fba 0 2024-04-22T23:23:41.171 INFO:tasks.workunit.client.1.smithi134.stdout:1/240: getdents db/d16/d56 0 2024-04-22T23:23:41.171 INFO:tasks.workunit.client.0.smithi067.stdout:1/940: chown d8/d9/d2f/d33/ld7 115505108 1 2024-04-22T23:23:41.172 INFO:tasks.workunit.client.1.smithi134.stdout:1/241: write db/d16/d56/f57 [505525,92324] 0 2024-04-22T23:23:41.174 INFO:tasks.workunit.client.0.smithi067.stdout:9/919: truncate d5/dd/d18/d39/d49/d54/d99/fcb 1099643 0 2024-04-22T23:23:41.175 INFO:tasks.workunit.client.0.smithi067.stdout:3/970: mkdir d3/d4/d10/d38/d53/d55/d74/d14d 0 2024-04-22T23:23:41.176 INFO:tasks.workunit.client.0.smithi067.stdout:1/941: mkdir d8/d9/d2d/d4d/d48/d9a/d12f 0 2024-04-22T23:23:41.177 INFO:tasks.workunit.client.1.smithi134.stdout:1/242: dwrite db/d15/d17/d20/d2a/f51 [0,4194304] 0 2024-04-22T23:23:41.178 INFO:tasks.workunit.client.1.smithi134.stdout:1/243: write db/d15/d17/d20/d2a/d2d/f48 [371747,91699] 0 2024-04-22T23:23:41.180 INFO:tasks.workunit.client.0.smithi067.stdout:9/920: symlink d5/d21/d47/d56/d8c/l12f 0 2024-04-22T23:23:41.183 INFO:tasks.workunit.client.1.smithi134.stdout:1/244: symlink db/d15/d17/d20/d2a/d2d/d2e/l5d 0 2024-04-22T23:23:41.185 INFO:tasks.workunit.client.1.smithi134.stdout:1/245: mknod db/c5e 0 2024-04-22T23:23:41.188 INFO:tasks.workunit.client.0.smithi067.stdout:6/986: write d17/f7d [2098980,17614] 0 2024-04-22T23:23:41.189 INFO:tasks.workunit.client.1.smithi134.stdout:1/246: dread db/f1d [0,4194304] 0 2024-04-22T23:23:41.193 INFO:tasks.workunit.client.0.smithi067.stdout:1/942: mkdir d8/d9/d2d/d4d/d48/d9a/d12f/d130 0 2024-04-22T23:23:41.194 INFO:tasks.workunit.client.0.smithi067.stdout:1/943: stat d8/d9/d2d/d4d 0 2024-04-22T23:23:41.195 INFO:tasks.workunit.client.0.smithi067.stdout:6/987: symlink d17/d39/d53/d90/d46/d62/l14c 0 2024-04-22T23:23:41.195 INFO:tasks.workunit.client.1.smithi134.stdout:1/247: truncate db/f14 4562556 0 2024-04-22T23:23:41.196 INFO:tasks.workunit.client.0.smithi067.stdout:7/972: dwrite d10/d14/d1d/d69/fba [0,4194304] 0 2024-04-22T23:23:41.197 INFO:tasks.workunit.client.0.smithi067.stdout:3/971: rmdir d3/d4 39 2024-04-22T23:23:41.198 INFO:tasks.workunit.client.1.smithi134.stdout:1/248: creat db/f5f x:0 0 0 2024-04-22T23:23:41.198 INFO:tasks.workunit.client.0.smithi067.stdout:9/921: truncate d5/dd/d18/d39/d8f/fb1 390872 0 2024-04-22T23:23:41.199 INFO:tasks.workunit.client.1.smithi134.stdout:1/249: chown l3 1 1 2024-04-22T23:23:41.199 INFO:tasks.workunit.client.1.smithi134.stdout:1/250: read - db/d15/d17/d20/d2a/d2d/f43 zero size 2024-04-22T23:23:41.200 INFO:tasks.workunit.client.1.smithi134.stdout:1/251: truncate db/d15/d17/d20/d2a/f51 5222806 0 2024-04-22T23:23:41.201 INFO:tasks.workunit.client.1.smithi134.stdout:1/252: read - db/f5f zero size 2024-04-22T23:23:41.201 INFO:tasks.workunit.client.0.smithi067.stdout:9/922: dread d5/dd/d18/d39/d49/f7b [0,4194304] 0 2024-04-22T23:23:41.205 INFO:tasks.workunit.client.1.smithi134.stdout:1/253: rmdir db/d15 39 2024-04-22T23:23:41.206 INFO:tasks.workunit.client.0.smithi067.stdout:7/973: symlink d10/d14/d6d/l169 0 2024-04-22T23:23:41.208 INFO:tasks.workunit.client.0.smithi067.stdout:7/974: truncate d10/d14/f98 1874678 0 2024-04-22T23:23:41.213 INFO:tasks.workunit.client.0.smithi067.stdout:6/988: getdents d17/d83/db5/dbc/dd8/d10c 0 2024-04-22T23:23:41.214 INFO:tasks.workunit.client.0.smithi067.stdout:6/989: stat d17/d39/d53/d90/d46/f75 0 2024-04-22T23:23:41.215 INFO:tasks.workunit.client.0.smithi067.stdout:3/972: creat d3/d4/d10/d38/d53/f14e x:0 0 0 2024-04-22T23:23:41.217 INFO:tasks.workunit.client.1.smithi134.stdout:1/254: dwrite db/d15/d17/d20/d2a/f36 [0,4194304] 0 2024-04-22T23:23:41.218 INFO:tasks.workunit.client.1.smithi134.stdout:1/255: stat db/d15/d17/d20/d2a/d2d 0 2024-04-22T23:23:41.219 INFO:tasks.workunit.client.0.smithi067.stdout:3/973: dread d3/d4/d10/de3/ff8 [0,4194304] 0 2024-04-22T23:23:41.219 INFO:tasks.workunit.client.0.smithi067.stdout:3/974: fsync d3/d4/d10/de3/f141 0 2024-04-22T23:23:41.220 INFO:tasks.workunit.client.0.smithi067.stdout:9/923: rename d5/dd/da9 to d5/d21/d47/d130 0 2024-04-22T23:23:41.222 INFO:tasks.workunit.client.1.smithi134.stdout:1/256: creat db/d16/d56/f60 x:0 0 0 2024-04-22T23:23:41.224 INFO:tasks.workunit.client.0.smithi067.stdout:1/944: dwrite d8/d9/d2d/d4d/d48/f8d [0,4194304] 0 2024-04-22T23:23:41.226 INFO:tasks.workunit.client.0.smithi067.stdout:3/975: creat d3/d4/d28/d5f/f14f x:0 0 0 2024-04-22T23:23:41.226 INFO:tasks.workunit.client.0.smithi067.stdout:1/945: fsync d8/d9/d2f/d33/d66/f91 0 2024-04-22T23:23:41.228 INFO:tasks.workunit.client.0.smithi067.stdout:3/976: write d3/d4/d10/d10b/d95/f13e [838866,22993] 0 2024-04-22T23:23:41.229 INFO:tasks.workunit.client.0.smithi067.stdout:6/990: mkdir d17/d39/d14d 0 2024-04-22T23:23:41.229 INFO:tasks.workunit.client.1.smithi134.stdout:1/257: truncate db/d15/d17/d20/f4b 1329188 0 2024-04-22T23:23:41.274 INFO:tasks.workunit.client.0.smithi067.stdout:6/991: chown d17/d83/db5/dbb/d13a/d13d 7775 1 2024-04-22T23:23:41.278 INFO:tasks.workunit.client.0.smithi067.stdout:7/975: dwrite d10/d14/d1d/d26/d2c/d41/f7c [0,4194304] 0 2024-04-22T23:23:41.278 INFO:tasks.workunit.client.0.smithi067.stdout:6/992: creat d17/d39/d14d/f14e x:0 0 0 2024-04-22T23:23:41.280 INFO:tasks.workunit.client.0.smithi067.stdout:7/976: write d10/d18/d88/daf/d100/f5e [4128226,82198] 0 2024-04-22T23:23:41.285 INFO:tasks.workunit.client.0.smithi067.stdout:6/993: fsync d17/f2b 0 2024-04-22T23:23:41.287 INFO:tasks.workunit.client.0.smithi067.stdout:7/977: mknod d10/d14/d1d/d26/d2c/d41/db0/ddd/d122/d123/c16a 0 2024-04-22T23:23:41.289 INFO:tasks.workunit.client.0.smithi067.stdout:6/994: unlink d17/d39/d3a/c100 0 2024-04-22T23:23:41.290 INFO:tasks.workunit.client.0.smithi067.stdout:7/978: chown d10/d14/d1d/d26/d2c/f3f 256 1 2024-04-22T23:23:41.292 INFO:tasks.workunit.client.0.smithi067.stdout:6/995: creat d17/d39/d53/d90/d46/db0/db9/de7/f14f x:0 0 0 2024-04-22T23:23:41.294 INFO:tasks.workunit.client.0.smithi067.stdout:7/979: rename d10/d14/d1d/d26/d2c/d7a/dc2/dd0/l112 to d10/d18/d88/daf/l16b 0 2024-04-22T23:23:41.294 INFO:tasks.workunit.client.0.smithi067.stdout:3/977: write d3/d4/d10/d38/d53/d55/f103 [784745,20704] 0 2024-04-22T23:23:41.296 INFO:tasks.workunit.client.0.smithi067.stdout:6/996: read - d17/d83/fde zero size 2024-04-22T23:23:41.299 INFO:tasks.workunit.client.0.smithi067.stdout:1/946: dwrite d8/d9/d2d/d4d/d83/dc5/fcd [0,4194304] 0 2024-04-22T23:23:41.299 INFO:tasks.workunit.client.0.smithi067.stdout:7/980: chown d10/d14/cad 57040 1 2024-04-22T23:23:41.301 INFO:tasks.workunit.client.0.smithi067.stdout:6/997: creat d17/d39/d3a/da9/f150 x:0 0 0 2024-04-22T23:23:41.305 INFO:tasks.workunit.client.0.smithi067.stdout:7/981: symlink d10/d14/d1d/d26/d80/dd2/dd4/d150/l16c 0 2024-04-22T23:23:41.309 INFO:tasks.workunit.client.0.smithi067.stdout:1/947: dwrite d8/d9/d2f/d33/d66/f120 [0,4194304] 0 2024-04-22T23:23:41.309 INFO:tasks.workunit.client.0.smithi067.stdout:6/998: mkdir d17/d83/db5/d151 0 2024-04-22T23:23:41.311 INFO:tasks.workunit.client.0.smithi067.stdout:7/982: creat d10/d14/de0/deb/f16d x:0 0 0 2024-04-22T23:23:41.313 INFO:tasks.workunit.client.0.smithi067.stdout:3/978: dread d3/d4/fc4 [0,4194304] 0 2024-04-22T23:23:41.314 INFO:tasks.workunit.client.0.smithi067.stdout:3/979: stat d3/d4/d28/d4a 0 2024-04-22T23:23:41.315 INFO:tasks.workunit.client.0.smithi067.stdout:1/948: unlink d8/d12/l27 0 2024-04-22T23:23:41.318 INFO:tasks.workunit.client.0.smithi067.stdout:7/983: mknod d10/d14/c16e 0 2024-04-22T23:23:41.319 INFO:tasks.workunit.client.0.smithi067.stdout:7/984: stat d10/d14/d1d/d24/f12d 0 2024-04-22T23:23:41.326 INFO:tasks.workunit.client.0.smithi067.stdout:1/949: creat d8/d9/d2d/f131 x:0 0 0 2024-04-22T23:23:41.326 INFO:tasks.workunit.client.0.smithi067.stdout:3/980: truncate d3/d4/d10/d38/d53/d55/d5c/f9e 9428 0 2024-04-22T23:23:41.327 INFO:tasks.workunit.client.0.smithi067.stdout:7/985: fdatasync d10/d18/d88/daf/d100/d62/fe8 0 2024-04-22T23:23:41.334 INFO:tasks.workunit.client.0.smithi067.stdout:7/986: dwrite d10/d18/fb2 [0,4194304] 0 2024-04-22T23:23:41.334 INFO:tasks.workunit.client.0.smithi067.stdout:1/950: mknod d8/d9/d2f/d59/c132 0 2024-04-22T23:23:41.335 INFO:tasks.workunit.client.0.smithi067.stdout:6/999: dread d17/d39/f103 [0,4194304] 0 2024-04-22T23:23:41.336 INFO:tasks.workunit.client.0.smithi067.stdout:1/951: chown d8/d9/d2d 217445717 1 2024-04-22T23:23:41.336 INFO:tasks.workunit.client.0.smithi067.stdout:7/987: write d10/d47/d49/f15d [723759,25291] 0 2024-04-22T23:23:41.340 INFO:tasks.workunit.client.0.smithi067.stdout:1/952: creat d8/d9/d2f/d31/f133 x:0 0 0 2024-04-22T23:23:41.341 INFO:tasks.workunit.client.0.smithi067.stdout:7/988: creat d10/d14/d1d/d46/d50/f16f x:0 0 0 2024-04-22T23:23:41.342 INFO:tasks.workunit.client.0.smithi067.stdout:1/953: creat d8/d9/d2d/d4d/d9c/f134 x:0 0 0 2024-04-22T23:23:41.342 INFO:tasks.workunit.client.0.smithi067.stdout:7/989: dread - d10/d14/d1d/d46/f12b zero size 2024-04-22T23:23:41.343 INFO:tasks.workunit.client.0.smithi067.stdout:7/990: dread - d10/d14/d1d/d26/d2c/d41/f138 zero size 2024-04-22T23:23:41.348 INFO:tasks.workunit.client.0.smithi067.stdout:1/954: dwrite d8/d9/de0/d5d/f11e [0,4194304] 0 2024-04-22T23:23:41.350 INFO:tasks.workunit.client.0.smithi067.stdout:7/991: link d10/d14/d1d/c6b d10/d14/d6d/d9e/c170 0 2024-04-22T23:23:41.352 INFO:tasks.workunit.client.0.smithi067.stdout:1/955: creat d8/d9/d2f/d59/f135 x:0 0 0 2024-04-22T23:23:41.353 INFO:tasks.workunit.client.0.smithi067.stdout:7/992: mkdir d10/d14/d1d/d26/d2c/df4/d171 0 2024-04-22T23:23:41.353 INFO:tasks.workunit.client.0.smithi067.stdout:1/956: truncate d8/d12/d36/f11c 429568 0 2024-04-22T23:23:41.354 INFO:tasks.workunit.client.0.smithi067.stdout:7/993: fdatasync d10/d14/d1d/d69/fba 0 2024-04-22T23:23:41.356 INFO:tasks.workunit.client.0.smithi067.stdout:7/994: write d10/d14/d1d/d24/f136 [337551,112280] 0 2024-04-22T23:23:41.356 INFO:tasks.workunit.client.0.smithi067.stdout:7/995: chown d10/d18/f1e 28479932 1 2024-04-22T23:23:41.357 INFO:tasks.workunit.client.0.smithi067.stdout:1/957: creat d8/d9/d12b/f136 x:0 0 0 2024-04-22T23:23:41.359 INFO:tasks.workunit.client.0.smithi067.stdout:1/958: chown d8/d9/de0/d60/l77 117 1 2024-04-22T23:23:41.360 INFO:tasks.workunit.client.0.smithi067.stdout:7/996: stat d10/d14/d1d/d46/d50/de2/f165 0 2024-04-22T23:23:41.361 INFO:tasks.workunit.client.0.smithi067.stdout:1/959: read d8/d12/d40/f57 [475955,94779] 0 2024-04-22T23:23:41.365 INFO:tasks.workunit.client.0.smithi067.stdout:1/960: creat d8/d9/de0/d5d/f137 x:0 0 0 2024-04-22T23:23:41.367 INFO:tasks.workunit.client.0.smithi067.stdout:1/961: write d8/d9/d2d/d4d/d70/f123 [22064,50186] 0 2024-04-22T23:23:41.367 INFO:tasks.workunit.client.0.smithi067.stdout:3/981: truncate d3/d4/d28/d4a/f7a 207309 0 2024-04-22T23:23:41.370 INFO:tasks.workunit.client.0.smithi067.stdout:7/997: dread d10/d14/d1d/f42 [0,4194304] 0 2024-04-22T23:23:41.370 INFO:tasks.workunit.client.0.smithi067.stdout:3/982: unlink d3/d4/f37 0 2024-04-22T23:23:41.372 INFO:tasks.workunit.client.0.smithi067.stdout:7/998: read - d10/d14/d1d/d46/d6a/df1/d143/f158 zero size 2024-04-22T23:23:41.372 INFO:tasks.workunit.client.0.smithi067.stdout:1/962: mknod d8/d9/d2d/d4d/d83/c138 0 2024-04-22T23:23:41.375 INFO:tasks.workunit.client.0.smithi067.stdout:3/983: truncate d3/d4/d28/d40/dfd/dfe/f6e 2183468 0 2024-04-22T23:23:41.376 INFO:tasks.workunit.client.0.smithi067.stdout:3/984: chown d3/d4/d10/d8a/f106 28320 1 2024-04-22T23:23:41.377 INFO:tasks.workunit.client.0.smithi067.stdout:1/963: creat d8/d9/de0/f139 x:0 0 0 2024-04-22T23:23:41.378 INFO:tasks.workunit.client.0.smithi067.stdout:7/999: getdents d10/d14/d1d/d26/d2c/d41/db0/ddd/d14f 0 2024-04-22T23:23:41.379 INFO:tasks.workunit.client.0.smithi067.stdout:1/964: truncate d8/d9/d2f/d33/d66/ffe 681699 0 2024-04-22T23:23:41.380 INFO:tasks.workunit.client.0.smithi067.stdout:3/985: rename d3/d4/dec/f124 to d3/d4/d1b/d93/f150 0 2024-04-22T23:23:41.384 INFO:tasks.workunit.client.0.smithi067.stdout:1/965: getdents d8/d12/d36 0 2024-04-22T23:23:41.392 INFO:tasks.workunit.client.0.smithi067.stdout:1/966: write d8/d12/d36/d87/fad [464829,130419] 0 2024-04-22T23:23:41.396 INFO:tasks.workunit.client.0.smithi067.stdout:3/986: dwrite d3/d4/d10/d1f/d3c/df4/f107 [0,4194304] 0 2024-04-22T23:23:41.400 INFO:tasks.workunit.client.0.smithi067.stdout:3/987: truncate d3/d4/d10/d8a/f106 4613368 0 2024-04-22T23:23:41.402 INFO:tasks.workunit.client.0.smithi067.stdout:1/967: dwrite d8/d9/d2f/d59/f135 [0,4194304] 0 2024-04-22T23:23:41.406 INFO:tasks.workunit.client.0.smithi067.stdout:1/968: creat d8/d9/d2f/d33/d5c/f13a x:0 0 0 2024-04-22T23:23:41.407 INFO:tasks.workunit.client.0.smithi067.stdout:3/988: creat d3/d4/d28/dc0/d11b/d139/f151 x:0 0 0 2024-04-22T23:23:41.408 INFO:tasks.workunit.client.0.smithi067.stdout:3/989: fdatasync d3/d4/d10/d38/d53/d55/d5c/d70/f11f 0 2024-04-22T23:23:41.411 INFO:tasks.workunit.client.0.smithi067.stdout:3/990: rmdir d3/d4/d10/d38 39 2024-04-22T23:23:41.421 INFO:tasks.workunit.client.0.smithi067.stdout:1/969: truncate d8/d9/d2f/d59/f7b 2613328 0 2024-04-22T23:23:41.423 INFO:tasks.workunit.client.0.smithi067.stdout:1/970: creat d8/d9/d2d/d4d/d83/dc5/f13b x:0 0 0 2024-04-22T23:23:41.425 INFO:tasks.workunit.client.0.smithi067.stdout:3/991: dwrite d3/d4/d10/d38/d53/d55/d74/dd5/ddb/f12c [0,4194304] 0 2024-04-22T23:23:41.427 INFO:tasks.workunit.client.0.smithi067.stdout:3/992: dread d3/d4/d1b/d93/f150 [0,4194304] 0 2024-04-22T23:23:41.429 INFO:tasks.workunit.client.0.smithi067.stdout:3/993: dread - d3/d4/d10/d38/d53/d55/d74/dd5/ddb/d90/da2/f12a zero size 2024-04-22T23:23:41.431 INFO:tasks.workunit.client.0.smithi067.stdout:1/971: rename d8/d9/de0/l113 to d8/d9/d2f/d33/d66/d84/l13c 0 2024-04-22T23:23:41.432 INFO:tasks.workunit.client.0.smithi067.stdout:3/994: fsync d3/d4/d28/d40/f45 0 2024-04-22T23:23:41.433 INFO:tasks.workunit.client.0.smithi067.stdout:1/972: write d8/d9/de0/d60/d62/f95 [287946,94153] 0 2024-04-22T23:23:41.436 INFO:tasks.workunit.client.0.smithi067.stdout:1/973: creat d8/d9/d12b/f13d x:0 0 0 2024-04-22T23:23:41.437 INFO:tasks.workunit.client.0.smithi067.stdout:3/995: rmdir d3/d4/d28/d40/dfd/dfe 39 2024-04-22T23:23:41.440 INFO:tasks.workunit.client.0.smithi067.stdout:1/974: creat d8/d9/d2f/d31/f13e x:0 0 0 2024-04-22T23:23:41.441 INFO:tasks.workunit.client.0.smithi067.stdout:1/975: readlink d8/d9/d2f/d33/lc7 0 2024-04-22T23:23:41.441 INFO:tasks.workunit.client.0.smithi067.stdout:3/996: creat d3/d4/d10/d38/dbf/f152 x:0 0 0 2024-04-22T23:23:41.446 INFO:tasks.workunit.client.0.smithi067.stdout:1/976: dread d8/d9/d2f/d59/f78 [0,4194304] 0 2024-04-22T23:23:41.448 INFO:tasks.workunit.client.0.smithi067.stdout:3/997: write d3/d4/d10/d38/d53/d55/d74/fc9 [4177273,7090] 0 2024-04-22T23:23:41.448 INFO:tasks.workunit.client.0.smithi067.stdout:1/977: mkdir d8/d9/de0/d13f 0 2024-04-22T23:23:41.450 INFO:tasks.workunit.client.0.smithi067.stdout:1/978: symlink d8/d12/d40/l140 0 2024-04-22T23:23:41.455 INFO:tasks.workunit.client.0.smithi067.stdout:1/979: rename d8/d9/d2f/d33/d66/fe8 to d8/d9/d2f/d33/d11d/f141 0 2024-04-22T23:23:41.455 INFO:tasks.workunit.client.0.smithi067.stdout:3/998: dwrite d3/d4/d10/d38/d53/d55/d74/dd5/ddb/f12c [4194304,4194304] 0 2024-04-22T23:23:41.459 INFO:tasks.workunit.client.0.smithi067.stdout:1/980: rename d8/d12/d36/c64 to d8/d9/d2f/d33/d11d/c142 0 2024-04-22T23:23:41.460 INFO:tasks.workunit.client.0.smithi067.stdout:1/981: fdatasync d8/d12/d40/f44 0 2024-04-22T23:23:41.461 INFO:tasks.workunit.client.0.smithi067.stdout:3/999: fdatasync d3/d4/d10/d38/d53/d55/d5c/d70/f9b 0 2024-04-22T23:23:41.464 INFO:tasks.workunit.client.0.smithi067.stdout:1/982: mkdir d8/d9/d143 0 2024-04-22T23:23:41.469 INFO:tasks.workunit.client.0.smithi067.stdout:1/983: dread d8/d12/fd6 [0,4194304] 0 2024-04-22T23:23:41.470 INFO:tasks.workunit.client.0.smithi067.stdout:1/984: stat d8/d9/d2d/d4d/d83/c100 0 2024-04-22T23:23:41.471 INFO:tasks.workunit.client.0.smithi067.stdout:1/985: stat d8/d9/d2d/d4d/d83/dc5/fcd 0 2024-04-22T23:23:41.504 INFO:tasks.workunit.client.1.smithi134.stdout:2/211: sync 2024-04-22T23:23:41.504 INFO:tasks.workunit.client.1.smithi134.stdout:2/212: dread - d2/f3e zero size 2024-04-22T23:23:41.505 INFO:tasks.workunit.client.1.smithi134.stdout:2/213: stat d2/f2e 0 2024-04-22T23:23:41.506 INFO:tasks.workunit.client.1.smithi134.stdout:2/214: fdatasync d2/dd/df/f41 0 2024-04-22T23:23:41.567 INFO:tasks.workunit.client.0.smithi067.stdout:9/924: sync 2024-04-22T23:23:41.575 INFO:tasks.workunit.client.1.smithi134.stdout:7/271: sync 2024-04-22T23:23:41.575 INFO:tasks.workunit.client.1.smithi134.stdout:4/419: sync 2024-04-22T23:23:41.575 INFO:tasks.workunit.client.1.smithi134.stdout:3/217: sync 2024-04-22T23:23:41.575 INFO:tasks.workunit.client.1.smithi134.stdout:9/204: sync 2024-04-22T23:23:41.575 INFO:tasks.workunit.client.1.smithi134.stdout:0/254: sync 2024-04-22T23:23:41.575 INFO:tasks.workunit.client.1.smithi134.stdout:8/224: sync 2024-04-22T23:23:41.575 INFO:tasks.workunit.client.1.smithi134.stdout:6/313: sync 2024-04-22T23:23:41.575 INFO:tasks.workunit.client.1.smithi134.stdout:5/282: sync 2024-04-22T23:23:41.575 INFO:tasks.workunit.client.1.smithi134.stdout:1/258: sync 2024-04-22T23:23:41.576 INFO:tasks.workunit.client.1.smithi134.stdout:7/272: stat d4/d10 0 2024-04-22T23:23:41.578 INFO:tasks.workunit.client.1.smithi134.stdout:3/218: read f9 [3050638,52984] 0 2024-04-22T23:23:41.578 INFO:tasks.workunit.client.1.smithi134.stdout:5/283: write d9/d1e/f35 [241362,35213] 0 2024-04-22T23:23:41.578 INFO:tasks.workunit.client.1.smithi134.stdout:8/225: mknod d7/dc/d2c/c43 0 2024-04-22T23:23:41.579 INFO:tasks.workunit.client.0.smithi067.stdout:9/925: dread d5/d21/d47/f72 [0,4194304] 0 2024-04-22T23:23:41.581 INFO:tasks.workunit.client.0.smithi067.stdout:9/926: read d5/d21/f27 [7406014,90630] 0 2024-04-22T23:23:41.583 INFO:tasks.workunit.client.1.smithi134.stdout:3/219: symlink dc/d2a/d30/l55 0 2024-04-22T23:23:41.584 INFO:tasks.workunit.client.0.smithi067.stdout:9/927: creat d5/d21/d47/d56/d8c/f131 x:0 0 0 2024-04-22T23:23:41.584 INFO:tasks.workunit.client.1.smithi134.stdout:6/314: dread d0/d19/de/d2f/f43 [0,4194304] 0 2024-04-22T23:23:41.584 INFO:tasks.workunit.client.1.smithi134.stdout:5/284: chown d9/dd/d17/d2a/l43 19010 1 2024-04-22T23:23:41.586 INFO:tasks.workunit.client.1.smithi134.stdout:0/255: mknod d1/db/c4e 0 2024-04-22T23:23:41.587 INFO:tasks.workunit.client.1.smithi134.stdout:3/220: write dc/dd/d28/d2f/d21/d42/f47 [2202439,4607] 0 2024-04-22T23:23:41.589 INFO:tasks.workunit.client.0.smithi067.stdout:9/928: dwrite d5/dd/d18/d39/d3e/d3f/f73 [0,4194304] 0 2024-04-22T23:23:41.589 INFO:tasks.workunit.client.1.smithi134.stdout:0/256: truncate d1/d5/dd/f35 4956373 0 2024-04-22T23:23:41.590 INFO:tasks.workunit.client.1.smithi134.stdout:4/420: rename d4/d15/d2a to d4/d35/d8e 0 2024-04-22T23:23:41.591 INFO:tasks.workunit.client.1.smithi134.stdout:4/421: chown d4/d35/f57 0 1 2024-04-22T23:23:41.591 INFO:tasks.workunit.client.1.smithi134.stdout:9/205: dread d0/d7/d12/f30 [0,4194304] 0 2024-04-22T23:23:41.592 INFO:tasks.workunit.client.1.smithi134.stdout:5/285: creat d9/d11/f5d x:0 0 0 2024-04-22T23:23:41.595 INFO:tasks.workunit.client.1.smithi134.stdout:7/273: dread d4/f2d [0,4194304] 0 2024-04-22T23:23:41.601 INFO:tasks.workunit.client.1.smithi134.stdout:4/422: fdatasync d4/d35/d8e/f8a 0 2024-04-22T23:23:41.604 INFO:tasks.workunit.client.1.smithi134.stdout:9/206: dwrite d0/d7/d23/d26/f29 [0,4194304] 0 2024-04-22T23:23:41.605 INFO:tasks.workunit.client.1.smithi134.stdout:0/257: mknod d1/c4f 0 2024-04-22T23:23:41.606 INFO:tasks.workunit.client.0.smithi067.stdout:9/929: dwrite d5/d21/d47/d56/dc6/d103/d114/f127 [4194304,4194304] 0 2024-04-22T23:23:41.607 INFO:tasks.workunit.client.1.smithi134.stdout:7/274: rename d4/d4a/f5a to d4/d8/d41/d5b/f64 0 2024-04-22T23:23:41.610 INFO:tasks.workunit.client.0.smithi067.stdout:9/930: symlink d5/d21/d47/d130/db9/d115/d119/l132 0 2024-04-22T23:23:41.610 INFO:tasks.workunit.client.0.smithi067.stdout:9/931: chown d5/dd/d18/d3c/f111 1414383 1 2024-04-22T23:23:41.613 INFO:tasks.workunit.client.0.smithi067.stdout:9/932: creat d5/dd/d18/d39/d8f/f133 x:0 0 0 2024-04-22T23:23:41.613 INFO:tasks.workunit.client.1.smithi134.stdout:8/226: dwrite d7/f23 [4194304,4194304] 0 2024-04-22T23:23:41.614 INFO:tasks.workunit.client.0.smithi067.stdout:9/933: chown d5/dd/d18/d3c/c59 109822 1 2024-04-22T23:23:41.615 INFO:tasks.workunit.client.1.smithi134.stdout:3/221: dread dc/dd/d28/d34/f50 [0,4194304] 0 2024-04-22T23:23:41.616 INFO:tasks.workunit.client.1.smithi134.stdout:8/227: truncate d7/db/d2d/f2e 82334 0 2024-04-22T23:23:41.616 INFO:tasks.workunit.client.1.smithi134.stdout:6/315: truncate d0/d19/de/f31 3940315 0 2024-04-22T23:23:41.616 INFO:tasks.workunit.client.0.smithi067.stdout:9/934: symlink d5/d21/d47/d56/d120/l134 0 2024-04-22T23:23:41.617 INFO:tasks.workunit.client.0.smithi067.stdout:9/935: chown d5/dd/d18/d39/d3e/d3f/d63/fa8 292 1 2024-04-22T23:23:41.619 INFO:tasks.workunit.client.1.smithi134.stdout:6/316: write d0/f26 [1684989,80516] 0 2024-04-22T23:23:41.622 INFO:tasks.workunit.client.1.smithi134.stdout:2/215: sync 2024-04-22T23:23:41.628 INFO:tasks.workunit.client.1.smithi134.stdout:3/222: chown dc/f22 98417 1 2024-04-22T23:23:41.629 INFO:tasks.workunit.client.1.smithi134.stdout:3/223: chown dc/f3f 610 1 2024-04-22T23:23:41.630 INFO:tasks.workunit.client.1.smithi134.stdout:6/317: dwrite d0/f8 [0,4194304] 0 2024-04-22T23:23:41.630 INFO:tasks.workunit.client.1.smithi134.stdout:8/228: rmdir d7/db 39 2024-04-22T23:23:41.631 INFO:tasks.workunit.client.0.smithi067.stdout:9/936: dread d5/dd/d18/d39/d49/d54/fc9 [0,4194304] 0 2024-04-22T23:23:41.631 INFO:tasks.workunit.client.1.smithi134.stdout:5/286: write d9/d11/f1a [983073,25705] 0 2024-04-22T23:23:41.632 INFO:tasks.workunit.client.1.smithi134.stdout:4/423: mknod d4/d13/d68/c8f 0 2024-04-22T23:23:41.632 INFO:tasks.workunit.client.0.smithi067.stdout:9/937: read d5/dd/d18/d39/f64 [2405399,21025] 0 2024-04-22T23:23:41.636 INFO:tasks.workunit.client.1.smithi134.stdout:4/424: chown d4/dd/cf 0 1 2024-04-22T23:23:41.636 INFO:tasks.workunit.client.0.smithi067.stdout:9/938: chown d5/d21/d47/d7f/f9f 73460 1 2024-04-22T23:23:41.636 INFO:tasks.workunit.client.1.smithi134.stdout:4/425: write d4/d15/f1c [1137357,38253] 0 2024-04-22T23:23:41.637 INFO:tasks.workunit.client.0.smithi067.stdout:1/986: sync 2024-04-22T23:23:41.637 INFO:tasks.workunit.client.1.smithi134.stdout:2/216: mkdir d2/dd/d4a 0 2024-04-22T23:23:41.639 INFO:tasks.workunit.client.0.smithi067.stdout:9/939: symlink d5/d21/d47/d11c/l135 0 2024-04-22T23:23:41.640 INFO:tasks.workunit.client.0.smithi067.stdout:9/940: read - d5/dd/d18/d3c/f126 zero size 2024-04-22T23:23:41.642 INFO:tasks.workunit.client.1.smithi134.stdout:1/259: dwrite db/d15/d17/d20/f4b [0,4194304] 0 2024-04-22T23:23:41.642 INFO:tasks.workunit.client.1.smithi134.stdout:8/229: stat d7/db/d2d/f3c 0 2024-04-22T23:23:41.643 INFO:tasks.workunit.client.1.smithi134.stdout:0/258: write d1/d5/dd/d20/f1e [3052366,125070] 0 2024-04-22T23:23:41.643 INFO:tasks.workunit.client.1.smithi134.stdout:0/259: chown d1/d2/f4 11 1 2024-04-22T23:23:41.645 INFO:tasks.workunit.client.1.smithi134.stdout:0/260: chown d1/d2/d1d/c36 4929 1 2024-04-22T23:23:41.645 INFO:tasks.workunit.client.1.smithi134.stdout:2/217: symlink d2/dd/d2b/d33/l4b 0 2024-04-22T23:23:41.648 INFO:tasks.workunit.client.1.smithi134.stdout:5/287: symlink d9/dd/d17/l5e 0 2024-04-22T23:23:41.648 INFO:tasks.workunit.client.0.smithi067.stdout:9/941: dwrite d5/dd/d18/d39/d3e/d3f/d63/f84 [0,4194304] 0 2024-04-22T23:23:41.649 INFO:tasks.workunit.client.1.smithi134.stdout:1/260: fdatasync fa 0 2024-04-22T23:23:41.649 INFO:tasks.workunit.client.1.smithi134.stdout:1/261: chown db/ff 785907 1 2024-04-22T23:23:41.650 INFO:tasks.workunit.client.0.smithi067.stdout:9/942: dread - d5/dd/d18/f106 zero size 2024-04-22T23:23:41.652 INFO:tasks.workunit.client.1.smithi134.stdout:6/318: link d0/d19/de/d12/f52 d0/d19/de/d12/d2a/d2d/d3a/f77 0 2024-04-22T23:23:41.653 INFO:tasks.workunit.client.1.smithi134.stdout:9/207: write d0/d7/d12/f1f [2585369,112960] 0 2024-04-22T23:23:41.653 INFO:tasks.workunit.client.1.smithi134.stdout:6/319: stat d0/f1 0 2024-04-22T23:23:41.653 INFO:tasks.workunit.client.1.smithi134.stdout:7/275: link d4/d10/d2f/d34/d62/l26 d4/d10/l65 0 2024-04-22T23:23:41.653 INFO:tasks.workunit.client.0.smithi067.stdout:1/987: dread d8/d9/d2d/f112 [0,4194304] 0 2024-04-22T23:23:41.654 INFO:tasks.workunit.client.1.smithi134.stdout:0/261: creat d1/d5/f50 x:0 0 0 2024-04-22T23:23:41.654 INFO:tasks.workunit.client.0.smithi067.stdout:9/943: write d5/d21/d47/fc5 [4810632,93360] 0 2024-04-22T23:23:41.655 INFO:tasks.workunit.client.1.smithi134.stdout:2/218: symlink d2/d15/d24/l4c 0 2024-04-22T23:23:41.656 INFO:tasks.workunit.client.1.smithi134.stdout:6/320: write d0/d19/de/d12/d2a/d2d/f49 [1334349,46406] 0 2024-04-22T23:23:41.657 INFO:tasks.workunit.client.1.smithi134.stdout:2/219: readlink d2/d15/d24/l49 0 2024-04-22T23:23:41.658 INFO:tasks.workunit.client.1.smithi134.stdout:0/262: dread - d1/d5/f47 zero size 2024-04-22T23:23:41.658 INFO:tasks.workunit.client.1.smithi134.stdout:9/208: chown d0/d7/d23/d26/f29 110802 1 2024-04-22T23:23:41.658 INFO:tasks.workunit.client.0.smithi067.stdout:9/944: mknod d5/dd/d18/d39/d3e/c136 0 2024-04-22T23:23:41.659 INFO:tasks.workunit.client.1.smithi134.stdout:4/426: creat d4/d35/d80/f90 x:0 0 0 2024-04-22T23:23:41.660 INFO:tasks.workunit.client.0.smithi067.stdout:1/988: creat d8/d9/d2f/d33/d5c/dda/f144 x:0 0 0 2024-04-22T23:23:41.660 INFO:tasks.workunit.client.0.smithi067.stdout:9/945: chown d5/dd/d18/d39/d49/d54/f3d 1 1 2024-04-22T23:23:41.661 INFO:tasks.workunit.client.1.smithi134.stdout:5/288: unlink d9/f12 0 2024-04-22T23:23:41.661 INFO:tasks.workunit.client.1.smithi134.stdout:1/262: dread db/d15/f18 [0,4194304] 0 2024-04-22T23:23:41.661 INFO:tasks.workunit.client.1.smithi134.stdout:4/427: chown d4/d7 0 1 2024-04-22T23:23:41.661 INFO:tasks.workunit.client.1.smithi134.stdout:6/321: read - d0/d19/de/d12/d5e/d6d/f73 zero size 2024-04-22T23:23:41.662 INFO:tasks.workunit.client.0.smithi067.stdout:9/946: chown d5/d21/d47/d56/d8c/dd5/fe2 16 1 2024-04-22T23:23:41.663 INFO:tasks.workunit.client.0.smithi067.stdout:1/989: symlink d8/d9/de0/l145 0 2024-04-22T23:23:41.664 INFO:tasks.workunit.client.1.smithi134.stdout:3/224: getdents dc/dd/d28 0 2024-04-22T23:23:41.664 INFO:tasks.workunit.client.1.smithi134.stdout:5/289: readlink d9/df/d20/l57 0 2024-04-22T23:23:41.664 INFO:tasks.workunit.client.0.smithi067.stdout:9/947: mkdir d5/dd/d18/d39/d49/d54/d99/d137 0 2024-04-22T23:23:41.665 INFO:tasks.workunit.client.0.smithi067.stdout:1/990: creat d8/d9/d2f/d59/f146 x:0 0 0 2024-04-22T23:23:41.665 INFO:tasks.workunit.client.1.smithi134.stdout:7/276: readlink d4/d10/d2f/d34/d62/l26 0 2024-04-22T23:23:41.665 INFO:tasks.workunit.client.1.smithi134.stdout:0/263: write d1/d12/f2e [606298,94385] 0 2024-04-22T23:23:41.666 INFO:tasks.workunit.client.1.smithi134.stdout:0/264: stat d1/d5/f50 0 2024-04-22T23:23:41.667 INFO:tasks.workunit.client.0.smithi067.stdout:1/991: mkdir d8/d9/d2f/d147 0 2024-04-22T23:23:41.667 INFO:tasks.workunit.client.1.smithi134.stdout:4/428: rmdir d4/d15/d82 39 2024-04-22T23:23:41.668 INFO:tasks.workunit.client.0.smithi067.stdout:9/948: creat d5/dd/d18/d39/d49/d54/d99/d137/f138 x:0 0 0 2024-04-22T23:23:41.669 INFO:tasks.workunit.client.0.smithi067.stdout:1/992: read d8/d9/f43 [197043,16345] 0 2024-04-22T23:23:41.670 INFO:tasks.workunit.client.1.smithi134.stdout:9/209: rename d0/f3 to d0/d7/d23/f45 0 2024-04-22T23:23:41.670 INFO:tasks.workunit.client.1.smithi134.stdout:4/429: rename d4 to d4/d13/d68/d91 22 2024-04-22T23:23:41.670 INFO:tasks.workunit.client.1.smithi134.stdout:1/263: rmdir db/d15/d17/d20/d2a 39 2024-04-22T23:23:41.671 INFO:tasks.workunit.client.0.smithi067.stdout:1/993: symlink d8/d9/d12b/l148 0 2024-04-22T23:23:41.672 INFO:tasks.workunit.client.1.smithi134.stdout:4/430: chown d4/d7/d32/d84 23840055 1 2024-04-22T23:23:41.675 INFO:tasks.workunit.client.0.smithi067.stdout:9/949: dwrite d5/dd/d18/d39/d49/d54/d99/f102 [0,4194304] 0 2024-04-22T23:23:41.676 INFO:tasks.workunit.client.1.smithi134.stdout:0/265: mknod d1/d2/d1d/c51 0 2024-04-22T23:23:41.677 INFO:tasks.workunit.client.0.smithi067.stdout:9/950: chown d5/dd/d18/c45 39 1 2024-04-22T23:23:41.678 INFO:tasks.workunit.client.1.smithi134.stdout:0/266: write d1/d5/dd/d11/f18 [735671,50187] 0 2024-04-22T23:23:41.679 INFO:tasks.workunit.client.0.smithi067.stdout:9/951: creat d5/d21/d47/d56/d8c/dd5/f139 x:0 0 0 2024-04-22T23:23:41.679 INFO:tasks.workunit.client.1.smithi134.stdout:1/264: creat db/d15/f61 x:0 0 0 2024-04-22T23:23:41.680 INFO:tasks.workunit.client.1.smithi134.stdout:7/277: dwrite d4/d8/fb [4194304,4194304] 0 2024-04-22T23:23:41.681 INFO:tasks.workunit.client.0.smithi067.stdout:9/952: fsync d5/dd/f26 0 2024-04-22T23:23:41.681 INFO:tasks.workunit.client.0.smithi067.stdout:9/953: chown d5/dd/de6 3565554 1 2024-04-22T23:23:41.683 INFO:tasks.workunit.client.0.smithi067.stdout:9/954: write d5/dd/d18/d39/d8f/f133 [40746,3199] 0 2024-04-22T23:23:41.683 INFO:tasks.workunit.client.1.smithi134.stdout:6/322: mknod d0/d19/db/d75/c78 0 2024-04-22T23:23:41.687 INFO:tasks.workunit.client.1.smithi134.stdout:1/265: write db/fd [3223038,109529] 0 2024-04-22T23:23:41.689 INFO:tasks.workunit.client.1.smithi134.stdout:1/266: chown db/d16/c59 18 1 2024-04-22T23:23:41.689 INFO:tasks.workunit.client.1.smithi134.stdout:2/220: dread d2/dd/fe [0,4194304] 0 2024-04-22T23:23:41.690 INFO:tasks.workunit.client.1.smithi134.stdout:0/267: rmdir d1/db 39 2024-04-22T23:23:41.694 INFO:tasks.workunit.client.1.smithi134.stdout:7/278: dwrite d4/d8/d3b/f48 [4194304,4194304] 0 2024-04-22T23:23:41.696 INFO:tasks.workunit.client.0.smithi067.stdout:9/955: write d5/d21/f34 [1279450,91925] 0 2024-04-22T23:23:41.697 INFO:tasks.workunit.client.0.smithi067.stdout:9/956: readlink d5/d21/d47/d56/dc6/d103/l121 0 2024-04-22T23:23:41.699 INFO:tasks.workunit.client.0.smithi067.stdout:1/994: dwrite d8/d9/d2f/d33/f41 [0,4194304] 0 2024-04-22T23:23:41.700 INFO:tasks.workunit.client.0.smithi067.stdout:9/957: creat d5/d21/d47/d130/db9/f13a x:0 0 0 2024-04-22T23:23:41.701 INFO:tasks.workunit.client.0.smithi067.stdout:1/995: write d8/d9/f1b [3789592,107739] 0 2024-04-22T23:23:41.701 INFO:tasks.workunit.client.1.smithi134.stdout:4/431: creat d4/d15/f92 x:0 0 0 2024-04-22T23:23:41.705 INFO:tasks.workunit.client.1.smithi134.stdout:4/432: mkdir d4/d7/d32/d3a/d93 0 2024-04-22T23:23:41.706 INFO:tasks.workunit.client.0.smithi067.stdout:9/958: mknod d5/d21/d47/d56/d120/c13b 0 2024-04-22T23:23:41.709 INFO:tasks.workunit.client.1.smithi134.stdout:3/225: write dc/dd/d28/f1c [3937701,126789] 0 2024-04-22T23:23:41.710 INFO:tasks.workunit.client.0.smithi067.stdout:9/959: mkdir d5/d21/d47/d56/d8c/df8/d12d/d13c 0 2024-04-22T23:23:41.714 INFO:tasks.workunit.client.1.smithi134.stdout:0/268: link d1/db/c49 d1/d2/c52 0 2024-04-22T23:23:41.715 INFO:tasks.workunit.client.1.smithi134.stdout:2/221: link d2/dd/df/d1c/l3f d2/d15/d24/l4d 0 2024-04-22T23:23:41.722 INFO:tasks.workunit.client.0.smithi067.stdout:9/960: write d5/dd/d18/d3c/d40/f50 [3756232,118652] 0 2024-04-22T23:23:41.722 INFO:tasks.workunit.client.1.smithi134.stdout:7/279: truncate d4/d10/f11 266198 0 2024-04-22T23:23:41.723 INFO:tasks.workunit.client.1.smithi134.stdout:3/226: dwrite dc/dd/d28/d34/f48 [0,4194304] 0 2024-04-22T23:23:41.724 INFO:tasks.workunit.client.0.smithi067.stdout:1/996: dwrite d8/d12/f17 [0,4194304] 0 2024-04-22T23:23:41.726 INFO:tasks.workunit.client.1.smithi134.stdout:1/267: dwrite db/d15/f18 [4194304,4194304] 0 2024-04-22T23:23:41.727 INFO:tasks.workunit.client.1.smithi134.stdout:7/280: chown d4/f2e 5213297 1 2024-04-22T23:23:41.729 INFO:tasks.workunit.client.0.smithi067.stdout:1/997: rename d8/d9/d2f/f79 to d8/d9/d2f/d31/ded/f149 0 2024-04-22T23:23:41.731 INFO:tasks.workunit.client.0.smithi067.stdout:1/998: write d8/d12/d36/d87/fdf [980777,61963] 0 2024-04-22T23:23:41.733 INFO:tasks.workunit.client.1.smithi134.stdout:1/268: write db/f5f [246890,38986] 0 2024-04-22T23:23:41.735 INFO:tasks.workunit.client.0.smithi067.stdout:1/999: symlink d8/d12/d40/l14a 0 2024-04-22T23:23:41.736 INFO:tasks.workunit.client.1.smithi134.stdout:2/222: dwrite d2/dd/df/d1c/f21 [0,4194304] 0 2024-04-22T23:23:41.737 INFO:tasks.workunit.client.0.smithi067.stdout:9/961: dwrite d5/d21/d47/d130/fc4 [0,4194304] 0 2024-04-22T23:23:41.739 INFO:tasks.workunit.client.1.smithi134.stdout:4/433: dread d4/d35/d5e/f22 [0,4194304] 0 2024-04-22T23:23:41.741 INFO:tasks.workunit.client.0.smithi067.stdout:9/962: stat d5/d21/d47/d57/dc0/d105/l10c 0 2024-04-22T23:23:41.743 INFO:tasks.workunit.client.1.smithi134.stdout:2/223: symlink d2/d15/d24/d1d/l4e 0 2024-04-22T23:23:41.744 INFO:tasks.workunit.client.1.smithi134.stdout:0/269: dread d1/d5/dd/f16 [0,4194304] 0 2024-04-22T23:23:41.745 INFO:tasks.workunit.client.1.smithi134.stdout:0/270: readlink d1/d5/dd/d11/l4c 0 2024-04-22T23:23:41.746 INFO:tasks.workunit.client.0.smithi067.stdout:9/963: creat d5/d21/d47/d56/f13d x:0 0 0 2024-04-22T23:23:41.751 INFO:tasks.workunit.client.1.smithi134.stdout:1/269: getdents db/d15 0 2024-04-22T23:23:41.754 INFO:tasks.workunit.client.1.smithi134.stdout:4/434: creat d4/d35/d5e/f94 x:0 0 0 2024-04-22T23:23:41.756 INFO:tasks.workunit.client.1.smithi134.stdout:0/271: dwrite d1/d12/f3a [0,4194304] 0 2024-04-22T23:23:41.757 INFO:tasks.workunit.client.1.smithi134.stdout:2/224: rename d2/f18 to d2/d15/d24/d3b/f4f 0 2024-04-22T23:23:41.758 INFO:tasks.workunit.client.1.smithi134.stdout:0/272: fsync d1/d12/d22/f24 0 2024-04-22T23:23:41.759 INFO:tasks.workunit.client.1.smithi134.stdout:4/435: truncate d4/f6 1502584 0 2024-04-22T23:23:41.760 INFO:tasks.workunit.client.1.smithi134.stdout:0/273: chown d1/d2/d1d 25 1 2024-04-22T23:23:41.761 INFO:tasks.workunit.client.1.smithi134.stdout:4/436: write d4/f21 [4616562,17704] 0 2024-04-22T23:23:41.762 INFO:tasks.workunit.client.1.smithi134.stdout:4/437: fdatasync d4/d7/d32/d3a/f51 0 2024-04-22T23:23:41.765 INFO:tasks.workunit.client.1.smithi134.stdout:2/225: link d2/dd/f37 d2/d15/f50 0 2024-04-22T23:23:41.770 INFO:tasks.workunit.client.1.smithi134.stdout:3/227: dread f9 [0,4194304] 0 2024-04-22T23:23:41.771 INFO:tasks.workunit.client.1.smithi134.stdout:2/226: creat d2/dd/d2b/d33/f51 x:0 0 0 2024-04-22T23:23:41.773 INFO:tasks.workunit.client.1.smithi134.stdout:3/228: symlink dc/d2a/d30/l56 0 2024-04-22T23:23:41.774 INFO:tasks.workunit.client.0.smithi067.stdout:9/964: write d5/dd/f26 [2504410,54037] 0 2024-04-22T23:23:41.775 INFO:tasks.workunit.client.1.smithi134.stdout:3/229: chown dc/dd/d28/d3b 15612651 1 2024-04-22T23:23:41.776 INFO:tasks.workunit.client.1.smithi134.stdout:2/227: rename d2/d15/d24/f40 to d2/dd/d4a/f52 0 2024-04-22T23:23:41.777 INFO:tasks.workunit.client.0.smithi067.stdout:9/965: rename d5/dd/d18/d39/d4d/l5d to d5/d21/d47/d56/d8c/dd5/l13e 0 2024-04-22T23:23:41.781 INFO:tasks.workunit.client.1.smithi134.stdout:2/228: dread d2/d15/d24/f35 [0,4194304] 0 2024-04-22T23:23:41.783 INFO:tasks.workunit.client.0.smithi067.stdout:9/966: dwrite d5/dd/d18/d39/d3e/d3f/d63/f11b [0,4194304] 0 2024-04-22T23:23:41.784 INFO:tasks.workunit.client.1.smithi134.stdout:3/230: symlink dc/dd/d28/d2f/d21/l57 0 2024-04-22T23:23:41.786 INFO:tasks.workunit.client.1.smithi134.stdout:1/270: write db/f14 [4767859,15849] 0 2024-04-22T23:23:41.787 INFO:tasks.workunit.client.0.smithi067.stdout:9/967: mknod d5/dd/d18/d39/d49/dcc/c13f 0 2024-04-22T23:23:41.788 INFO:tasks.workunit.client.1.smithi134.stdout:2/229: read d2/d15/d24/d3b/f4f [64156,65343] 0 2024-04-22T23:23:41.791 INFO:tasks.workunit.client.1.smithi134.stdout:4/438: truncate d4/d7/d32/d3a/f51 1999045 0 2024-04-22T23:23:41.795 INFO:tasks.workunit.client.1.smithi134.stdout:3/231: symlink dc/dd/d28/d2f/l58 0 2024-04-22T23:23:41.796 INFO:tasks.workunit.client.1.smithi134.stdout:2/230: write d2/d15/d24/d1d/f28 [2787702,9374] 0 2024-04-22T23:23:41.797 INFO:tasks.workunit.client.1.smithi134.stdout:2/231: stat d2/d15/d24/f1e 0 2024-04-22T23:23:41.798 INFO:tasks.workunit.client.1.smithi134.stdout:1/271: rmdir db/d15/d17/d20/d2a 39 2024-04-22T23:23:41.799 INFO:tasks.workunit.client.0.smithi067.stdout:9/968: dwrite d5/d21/d47/d57/dc0/f10f [0,4194304] 0 2024-04-22T23:23:41.803 INFO:tasks.workunit.client.1.smithi134.stdout:7/281: truncate d4/d10/f11 1167469 0 2024-04-22T23:23:41.804 INFO:tasks.workunit.client.0.smithi067.stdout:9/969: dread d5/dd/d18/d39/d4d/fd0 [0,4194304] 0 2024-04-22T23:23:41.805 INFO:tasks.workunit.client.1.smithi134.stdout:3/232: dwrite dc/dd/d28/d2f/f49 [0,4194304] 0 2024-04-22T23:23:41.805 INFO:tasks.workunit.client.1.smithi134.stdout:4/439: dread d4/d15/d64/f73 [0,4194304] 0 2024-04-22T23:23:41.807 INFO:tasks.workunit.client.0.smithi067.stdout:9/970: truncate d5/dd/d18/d39/d4d/faa 54121 0 2024-04-22T23:23:41.811 INFO:tasks.workunit.client.1.smithi134.stdout:4/440: dread - d4/d35/d5e/d25/f87 zero size 2024-04-22T23:23:41.812 INFO:tasks.workunit.client.1.smithi134.stdout:7/282: symlink d4/d10/d2f/l66 0 2024-04-22T23:23:41.813 INFO:tasks.workunit.client.1.smithi134.stdout:7/283: fsync d4/f33 0 2024-04-22T23:23:41.814 INFO:tasks.workunit.client.0.smithi067.stdout:9/971: dwrite d5/dd/d18/d39/d4d/fd0 [0,4194304] 0 2024-04-22T23:23:41.817 INFO:tasks.workunit.client.1.smithi134.stdout:4/441: rmdir d4/d35/d80 39 2024-04-22T23:23:41.817 INFO:tasks.workunit.client.0.smithi067.stdout:9/972: write d5/dd/d18/d5c/f104 [4183441,33991] 0 2024-04-22T23:23:41.818 INFO:tasks.workunit.client.1.smithi134.stdout:4/442: chown d4/dd/f41 592901099 1 2024-04-22T23:23:41.819 INFO:tasks.workunit.client.1.smithi134.stdout:1/272: dread db/fc [0,4194304] 0 2024-04-22T23:23:41.820 INFO:tasks.workunit.client.1.smithi134.stdout:4/443: fsync d4/d29/f77 0 2024-04-22T23:23:41.820 INFO:tasks.workunit.client.0.smithi067.stdout:9/973: dread d5/d21/d47/d56/fec [0,4194304] 0 2024-04-22T23:23:41.822 INFO:tasks.workunit.client.0.smithi067.stdout:9/974: chown d5/d21/d47/d130/lcd 20 1 2024-04-22T23:23:41.824 INFO:tasks.workunit.client.1.smithi134.stdout:7/284: rename d4/d8/f30 to d4/d10/d2f/d34/f67 0 2024-04-22T23:23:41.825 INFO:tasks.workunit.client.0.smithi067.stdout:9/975: getdents d5/d21/d47/d56/dc6/d103/d114 0 2024-04-22T23:23:41.825 INFO:tasks.workunit.client.0.smithi067.stdout:9/976: stat d5/d21/d2c/l6a 0 2024-04-22T23:23:41.825 INFO:tasks.workunit.client.1.smithi134.stdout:3/233: link dc/dd/d28/l26 dc/dd/d28/d2f/l59 0 2024-04-22T23:23:41.827 INFO:tasks.workunit.client.0.smithi067.stdout:9/977: rename d5/fad to d5/d21/d47/d7f/f140 0 2024-04-22T23:23:41.828 INFO:tasks.workunit.client.1.smithi134.stdout:4/444: unlink d4/d15/f7b 0 2024-04-22T23:23:41.829 INFO:tasks.workunit.client.1.smithi134.stdout:2/232: dwrite d2/dd/f47 [0,4194304] 0 2024-04-22T23:23:41.830 INFO:tasks.workunit.client.1.smithi134.stdout:4/445: dread - d4/dd/f31 zero size 2024-04-22T23:23:41.832 INFO:tasks.workunit.client.1.smithi134.stdout:4/446: chown d4/d13/d68/c8f 57762 1 2024-04-22T23:23:41.832 INFO:tasks.workunit.client.1.smithi134.stdout:4/447: fdatasync d4/f42 0 2024-04-22T23:23:41.834 INFO:tasks.workunit.client.1.smithi134.stdout:3/234: symlink dc/dd/d28/d2f/d21/d42/l5a 0 2024-04-22T23:23:41.835 INFO:tasks.workunit.client.1.smithi134.stdout:1/273: symlink db/d15/d17/d20/d2a/d2d/l62 0 2024-04-22T23:23:41.835 INFO:tasks.workunit.client.0.smithi067.stdout:9/978: readlink d5/d21/d2c/l2f 0 2024-04-22T23:23:41.836 INFO:tasks.workunit.client.1.smithi134.stdout:2/233: write d2/dd/fe [1737952,36500] 0 2024-04-22T23:23:41.837 INFO:tasks.workunit.client.1.smithi134.stdout:4/448: fsync d4/d35/d5e/f4e 0 2024-04-22T23:23:41.839 INFO:tasks.workunit.client.0.smithi067.stdout:9/979: write d5/dd/d18/d3c/d40/f50 [3756289,9609] 0 2024-04-22T23:23:41.839 INFO:tasks.workunit.client.1.smithi134.stdout:3/235: mknod dc/dd/d28/d2f/d36/c5b 0 2024-04-22T23:23:41.840 INFO:tasks.workunit.client.0.smithi067.stdout:9/980: write d5/dd/d18/d39/d3e/f109 [550122,119607] 0 2024-04-22T23:23:41.841 INFO:tasks.workunit.client.1.smithi134.stdout:3/236: chown dc/d2a/d30/l56 774351 1 2024-04-22T23:23:41.846 INFO:tasks.workunit.client.1.smithi134.stdout:4/449: unlink d4/f2f 0 2024-04-22T23:23:41.846 INFO:tasks.workunit.client.1.smithi134.stdout:7/285: getdents d4/d10/d2f 0 2024-04-22T23:23:41.846 INFO:tasks.workunit.client.0.smithi067.stdout:9/981: write d5/dd/d18/d39/d4d/f8d [1586118,92175] 0 2024-04-22T23:23:41.847 INFO:tasks.workunit.client.1.smithi134.stdout:7/286: chown d4/d10/d2f/d34/d62/f56 719464151 1 2024-04-22T23:23:41.853 INFO:tasks.workunit.client.1.smithi134.stdout:7/287: creat d4/d8/d3b/f68 x:0 0 0 2024-04-22T23:23:41.855 INFO:tasks.workunit.client.1.smithi134.stdout:2/234: creat d2/dd/df/f53 x:0 0 0 2024-04-22T23:23:41.855 INFO:tasks.workunit.client.0.smithi067.stdout:9/982: write d5/dd/d18/d39/d4d/faa [329495,29080] 0 2024-04-22T23:23:41.856 INFO:tasks.workunit.client.1.smithi134.stdout:3/237: getdents dc/dd/d28/d3b/d29 0 2024-04-22T23:23:41.857 INFO:tasks.workunit.client.1.smithi134.stdout:4/450: mknod d4/d7a/c95 0 2024-04-22T23:23:41.857 INFO:tasks.workunit.client.0.smithi067.stdout:9/983: symlink d5/dd/d18/d3c/ddf/l141 0 2024-04-22T23:23:41.858 INFO:tasks.workunit.client.1.smithi134.stdout:7/288: mkdir d4/d10/d2f/d69 0 2024-04-22T23:23:41.860 INFO:tasks.workunit.client.0.smithi067.stdout:9/984: dread d5/d21/d47/d130/fc4 [0,4194304] 0 2024-04-22T23:23:41.862 INFO:tasks.workunit.client.1.smithi134.stdout:2/235: rename d2/dd/d2b to d2/d42/d54 0 2024-04-22T23:23:41.864 INFO:tasks.workunit.client.1.smithi134.stdout:3/238: unlink dc/dd/d28/d3b/d29/l44 0 2024-04-22T23:23:41.864 INFO:tasks.workunit.client.1.smithi134.stdout:4/451: dwrite d4/d7/f28 [0,4194304] 0 2024-04-22T23:23:41.865 INFO:tasks.workunit.client.0.smithi067.stdout:9/985: rename d5/dd/d18/d39/d49/cff to d5/dd/c142 0 2024-04-22T23:23:41.866 INFO:tasks.workunit.client.1.smithi134.stdout:3/239: truncate dc/f45 983899 0 2024-04-22T23:23:41.866 INFO:tasks.workunit.client.1.smithi134.stdout:1/274: truncate db/fc 3336029 0 2024-04-22T23:23:41.868 INFO:tasks.workunit.client.1.smithi134.stdout:2/236: symlink d2/dd/d4a/l55 0 2024-04-22T23:23:41.870 INFO:tasks.workunit.client.1.smithi134.stdout:4/452: symlink d4/d35/d8e/l96 0 2024-04-22T23:23:41.872 INFO:tasks.workunit.client.0.smithi067.stdout:9/986: dwrite d5/d21/d47/d130/db9/f13a [0,4194304] 0 2024-04-22T23:23:41.872 INFO:tasks.workunit.client.1.smithi134.stdout:3/240: creat dc/dd/d28/f5c x:0 0 0 2024-04-22T23:23:41.873 INFO:tasks.workunit.client.1.smithi134.stdout:1/275: creat db/d16/d56/f63 x:0 0 0 2024-04-22T23:23:41.875 INFO:tasks.workunit.client.1.smithi134.stdout:2/237: symlink d2/d15/d24/l56 0 2024-04-22T23:23:41.876 INFO:tasks.workunit.client.0.smithi067.stdout:9/987: mkdir d5/dd/d18/d39/d49/d54/d99/d143 0 2024-04-22T23:23:41.877 INFO:tasks.workunit.client.0.smithi067.stdout:9/988: fdatasync d5/d21/d47/d56/d8c/dd5/f10b 0 2024-04-22T23:23:41.878 INFO:tasks.workunit.client.1.smithi134.stdout:3/241: creat dc/dd/d28/d2f/d36/f5d x:0 0 0 2024-04-22T23:23:41.879 INFO:tasks.workunit.client.0.smithi067.stdout:9/989: read d5/d21/d47/d57/f60 [4143906,19484] 0 2024-04-22T23:23:41.882 INFO:tasks.workunit.client.0.smithi067.stdout:9/990: symlink d5/dd/d18/d5c/l144 0 2024-04-22T23:23:41.883 INFO:tasks.workunit.client.1.smithi134.stdout:1/276: rename db/fd to db/d15/d17/d20/d2a/d2d/f64 0 2024-04-22T23:23:41.883 INFO:tasks.workunit.client.1.smithi134.stdout:1/277: readlink db/l37 0 2024-04-22T23:23:41.884 INFO:tasks.workunit.client.0.smithi067.stdout:9/991: fsync d5/d21/d47/d56/d8c/dd5/fdd 0 2024-04-22T23:23:41.884 INFO:tasks.workunit.client.1.smithi134.stdout:2/238: creat d2/dd/f57 x:0 0 0 2024-04-22T23:23:41.885 INFO:tasks.workunit.client.0.smithi067.stdout:9/992: stat d5/dd/d18/d39/d3e/d3f/d63/fe3 0 2024-04-22T23:23:41.890 INFO:tasks.workunit.client.1.smithi134.stdout:3/242: creat dc/dd/d28/d2f/d54/f5e x:0 0 0 2024-04-22T23:23:41.891 INFO:tasks.workunit.client.1.smithi134.stdout:2/239: dwrite d2/f17 [0,4194304] 0 2024-04-22T23:23:41.892 INFO:tasks.workunit.client.1.smithi134.stdout:7/289: dread d4/d10/f1f [0,4194304] 0 2024-04-22T23:23:41.894 INFO:tasks.workunit.client.1.smithi134.stdout:7/290: truncate d4/d8/d3b/f50 35041 0 2024-04-22T23:23:41.895 INFO:tasks.workunit.client.1.smithi134.stdout:2/240: rmdir d2/dd/df 39 2024-04-22T23:23:41.896 INFO:tasks.workunit.client.0.smithi067.stdout:9/993: truncate d5/d21/d47/d56/d8c/fbe 2831116 0 2024-04-22T23:23:41.899 INFO:tasks.workunit.client.0.smithi067.stdout:9/994: creat d5/d21/d47/d56/de5/f145 x:0 0 0 2024-04-22T23:23:41.902 INFO:tasks.workunit.client.0.smithi067.stdout:9/995: dread d5/dd/d18/d39/d8f/f133 [0,4194304] 0 2024-04-22T23:23:41.903 INFO:tasks.workunit.client.0.smithi067.stdout:9/996: write d5/dd/d18/d39/d3e/f109 [1473033,33611] 0 2024-04-22T23:23:41.903 INFO:tasks.workunit.client.1.smithi134.stdout:3/243: dread dc/f22 [0,4194304] 0 2024-04-22T23:23:41.905 INFO:tasks.workunit.client.0.smithi067.stdout:9/997: unlink d5/d21/d47/d57/dc0/fc2 0 2024-04-22T23:23:41.905 INFO:tasks.workunit.client.1.smithi134.stdout:1/278: rmdir db 39 2024-04-22T23:23:41.907 INFO:tasks.workunit.client.1.smithi134.stdout:3/244: mkdir dc/dd/d28/d2f/d36/d5f 0 2024-04-22T23:23:41.908 INFO:tasks.workunit.client.1.smithi134.stdout:4/453: write d4/dd/f16 [2826348,27685] 0 2024-04-22T23:23:41.909 INFO:tasks.workunit.client.1.smithi134.stdout:4/454: fdatasync d4/d29/f4d 0 2024-04-22T23:23:41.910 INFO:tasks.workunit.client.0.smithi067.stdout:9/998: dwrite d5/d21/f2b [0,4194304] 0 2024-04-22T23:23:41.912 INFO:tasks.workunit.client.0.smithi067.stdout:9/999: truncate d5/d21/d47/d56/d8c/dd5/f139 188018 0 2024-04-22T23:23:41.916 INFO:tasks.workunit.client.0.smithi067.stderr:+ rm -rf -- ./tmp.wHgf4sBEUR 2024-04-22T23:23:41.919 INFO:tasks.workunit.client.1.smithi134.stdout:7/291: dread d4/d10/f23 [0,4194304] 0 2024-04-22T23:23:41.920 INFO:tasks.workunit.client.1.smithi134.stdout:4/455: fsync d4/d13/f65 0 2024-04-22T23:23:41.923 INFO:tasks.workunit.client.1.smithi134.stdout:4/456: fsync d4/f21 0 2024-04-22T23:23:41.923 INFO:tasks.workunit.client.1.smithi134.stdout:1/279: dwrite db/d15/d17/d20/d2a/d2d/d2e/f47 [0,4194304] 0 2024-04-22T23:23:41.924 INFO:tasks.workunit.client.1.smithi134.stdout:7/292: creat d4/d8/d2a/d42/f6a x:0 0 0 2024-04-22T23:23:41.925 INFO:tasks.workunit.client.1.smithi134.stdout:7/293: dread - d4/d8/d2a/d42/f6a zero size 2024-04-22T23:23:41.928 INFO:tasks.workunit.client.1.smithi134.stdout:7/294: chown d4/d10/d2f/d34/d62/c63 1161 1 2024-04-22T23:23:41.930 INFO:tasks.workunit.client.1.smithi134.stdout:4/457: creat d4/d35/d5e/f97 x:0 0 0 2024-04-22T23:23:41.932 INFO:tasks.workunit.client.1.smithi134.stdout:3/245: dwrite f9 [4194304,4194304] 0 2024-04-22T23:23:41.934 INFO:tasks.workunit.client.1.smithi134.stdout:7/295: truncate d4/f13 2369537 0 2024-04-22T23:23:41.936 INFO:tasks.workunit.client.1.smithi134.stdout:7/296: stat d4/d10/d2f/d34/d62/d29/c57 0 2024-04-22T23:23:41.936 INFO:tasks.workunit.client.1.smithi134.stdout:4/458: write d4/d35/d80/f90 [5768,39876] 0 2024-04-22T23:23:41.938 INFO:tasks.workunit.client.1.smithi134.stdout:7/297: dread - d4/d8/d3b/f68 zero size 2024-04-22T23:23:41.939 INFO:tasks.workunit.client.1.smithi134.stdout:3/246: creat dc/d2a/d30/f60 x:0 0 0 2024-04-22T23:23:41.940 INFO:tasks.workunit.client.1.smithi134.stdout:4/459: creat d4/d35/d5e/d25/f98 x:0 0 0 2024-04-22T23:23:41.940 INFO:tasks.workunit.client.1.smithi134.stdout:3/247: read - dc/dd/f4f zero size 2024-04-22T23:23:41.944 INFO:tasks.workunit.client.1.smithi134.stdout:7/298: link d4/f31 d4/d10/f6b 0 2024-04-22T23:23:41.946 INFO:tasks.workunit.client.1.smithi134.stdout:7/299: write d4/d8/d2a/f4f [732727,84567] 0 2024-04-22T23:23:41.947 INFO:tasks.workunit.client.1.smithi134.stdout:4/460: symlink d4/d35/l99 0 2024-04-22T23:23:41.947 INFO:tasks.workunit.client.1.smithi134.stdout:7/300: chown d4/d10/d2f/d34 546806373 1 2024-04-22T23:23:41.950 INFO:tasks.workunit.client.1.smithi134.stdout:3/248: mkdir dc/dd/d28/d2f/d61 0 2024-04-22T23:23:41.953 INFO:tasks.workunit.client.1.smithi134.stdout:7/301: rename d4/d8/d2a/l38 to d4/d8/d2a/l6c 0 2024-04-22T23:23:41.953 INFO:tasks.workunit.client.1.smithi134.stdout:1/280: write db/f28 [500599,106660] 0 2024-04-22T23:23:41.957 INFO:tasks.workunit.client.1.smithi134.stdout:7/302: symlink d4/d8/l6d 0 2024-04-22T23:23:41.958 INFO:tasks.workunit.client.1.smithi134.stdout:7/303: fsync d4/f33 0 2024-04-22T23:23:41.961 INFO:tasks.workunit.client.1.smithi134.stdout:7/304: mkdir d4/d8/d2a/d42/d6e 0 2024-04-22T23:23:41.962 INFO:tasks.workunit.client.1.smithi134.stdout:7/305: write d4/d10/d2f/f36 [700026,11842] 0 2024-04-22T23:23:41.965 INFO:tasks.workunit.client.1.smithi134.stdout:3/249: write dc/dd/d28/d34/f50 [65593,91965] 0 2024-04-22T23:23:41.967 INFO:tasks.workunit.client.1.smithi134.stdout:7/306: mknod d4/d8/d58/c6f 0 2024-04-22T23:23:41.968 INFO:tasks.workunit.client.1.smithi134.stdout:1/281: truncate db/d15/d17/d20/d2a/d2d/d2e/f39 347761 0 2024-04-22T23:23:41.969 INFO:tasks.workunit.client.1.smithi134.stdout:3/250: mkdir dc/d2a/d62 0 2024-04-22T23:23:41.970 INFO:tasks.workunit.client.1.smithi134.stdout:7/307: rename d4/d5c to d4/d10/d2f/d70 0 2024-04-22T23:23:41.976 INFO:tasks.workunit.client.1.smithi134.stdout:1/282: creat db/d15/d17/d20/f65 x:0 0 0 2024-04-22T23:23:41.976 INFO:tasks.workunit.client.1.smithi134.stdout:1/283: chown db/d15/d17/d20/d2a/d2d 122441 1 2024-04-22T23:23:41.978 INFO:tasks.workunit.client.1.smithi134.stdout:1/284: fdatasync db/f27 0 2024-04-22T23:23:41.980 INFO:tasks.workunit.client.1.smithi134.stdout:1/285: write db/d15/d17/d20/d2a/f44 [656864,33643] 0 2024-04-22T23:23:41.981 INFO:tasks.workunit.client.1.smithi134.stdout:7/308: write d4/f31 [4711929,69237] 0 2024-04-22T23:23:41.981 INFO:tasks.workunit.client.1.smithi134.stdout:3/251: truncate dc/dd/d28/d3b/d29/f2e 4064214 0 2024-04-22T23:23:41.983 INFO:tasks.workunit.client.1.smithi134.stdout:7/309: read d4/d8/d2a/f2c [596195,51894] 0 2024-04-22T23:23:41.985 INFO:tasks.workunit.client.1.smithi134.stdout:1/286: dread db/f14 [0,4194304] 0 2024-04-22T23:23:41.988 INFO:tasks.workunit.client.1.smithi134.stdout:7/310: dread d4/d8/d2a/f5f [0,4194304] 0 2024-04-22T23:23:41.989 INFO:tasks.workunit.client.1.smithi134.stdout:1/287: rename db/d15/f61 to db/d15/d17/d20/d2a/d2d/d2e/d42/f66 0 2024-04-22T23:23:41.990 INFO:tasks.workunit.client.1.smithi134.stdout:7/311: write d4/d10/d2f/d34/f67 [2709566,91731] 0 2024-04-22T23:23:41.991 INFO:tasks.workunit.client.1.smithi134.stdout:1/288: mknod db/d15/d17/d20/d2a/d2d/c67 0 2024-04-22T23:23:41.991 INFO:tasks.workunit.client.1.smithi134.stdout:7/312: chown d4/d8/d2a/f2c 2 1 2024-04-22T23:23:41.994 INFO:tasks.workunit.client.1.smithi134.stdout:7/313: creat d4/d10/d2f/d34/d62/f71 x:0 0 0 2024-04-22T23:23:41.994 INFO:tasks.workunit.client.1.smithi134.stdout:1/289: symlink db/d15/d17/d20/l68 0 2024-04-22T23:23:41.995 INFO:tasks.workunit.client.1.smithi134.stdout:7/314: creat d4/d10/d2f/d70/f72 x:0 0 0 2024-04-22T23:23:41.996 INFO:tasks.workunit.client.1.smithi134.stdout:7/315: read d4/d10/f1f [252498,87381] 0 2024-04-22T23:23:41.996 INFO:tasks.workunit.client.1.smithi134.stdout:1/290: rmdir db/d15/d17/d20/d2a/d2d/d2e/d42 39 2024-04-22T23:23:42.000 INFO:tasks.workunit.client.1.smithi134.stdout:1/291: dread db/f14 [0,4194304] 0 2024-04-22T23:23:42.002 INFO:tasks.workunit.client.1.smithi134.stdout:7/316: dwrite d4/d10/f6b [0,4194304] 0 2024-04-22T23:23:42.006 INFO:tasks.workunit.client.1.smithi134.stdout:7/317: mknod d4/d8/d2a/d42/d6e/c73 0 2024-04-22T23:23:42.008 INFO:tasks.workunit.client.1.smithi134.stdout:7/318: rmdir d4/d10/d2f 39 2024-04-22T23:23:42.010 INFO:tasks.workunit.client.1.smithi134.stdout:1/292: dwrite db/d16/d56/f57 [0,4194304] 0 2024-04-22T23:23:42.013 INFO:tasks.workunit.client.1.smithi134.stdout:7/319: creat d4/d10/d2f/d70/f74 x:0 0 0 2024-04-22T23:23:42.014 INFO:tasks.workunit.client.1.smithi134.stdout:1/293: creat db/d15/d17/f69 x:0 0 0 2024-04-22T23:23:42.024 INFO:tasks.workunit.client.1.smithi134.stdout:7/320: dwrite d4/d8/d2a/f5f [0,4194304] 0 2024-04-22T23:23:42.026 INFO:tasks.workunit.client.1.smithi134.stdout:7/321: read d4/d10/f6b [2066939,21184] 0 2024-04-22T23:23:42.028 INFO:tasks.workunit.client.1.smithi134.stdout:7/322: mkdir d4/d8/d2a/d42/d75 0 2024-04-22T23:23:42.028 INFO:tasks.workunit.client.1.smithi134.stdout:7/323: readlink d4/d10/l24 0 2024-04-22T23:23:42.029 INFO:tasks.workunit.client.1.smithi134.stdout:7/324: chown d4/d10/d2f/d34/d62/f2b 903524411 1 2024-04-22T23:23:42.037 INFO:tasks.workunit.client.1.smithi134.stdout:1/294: truncate db/fc 3023914 0 2024-04-22T23:23:42.038 INFO:tasks.workunit.client.1.smithi134.stdout:1/295: write db/d15/f54 [1039867,72083] 0 2024-04-22T23:23:42.040 INFO:tasks.workunit.client.1.smithi134.stdout:1/296: readlink db/d15/d17/d20/d2a/d2d/d2e/l52 0 2024-04-22T23:23:42.040 INFO:tasks.workunit.client.1.smithi134.stdout:7/325: rmdir d4 39 2024-04-22T23:23:42.051 INFO:tasks.workunit.client.1.smithi134.stdout:7/326: truncate d4/f7 4085395 0 2024-04-22T23:23:42.054 INFO:tasks.workunit.client.1.smithi134.stdout:1/297: dread db/d15/d17/d20/d2a/d2d/f64 [0,4194304] 0 2024-04-22T23:23:42.059 INFO:tasks.workunit.client.1.smithi134.stdout:7/327: read d4/d8/d2a/f47 [1049152,81988] 0 2024-04-22T23:23:42.060 INFO:tasks.workunit.client.1.smithi134.stdout:1/298: truncate db/d15/d17/d20/d2a/d2d/f64 3868441 0 2024-04-22T23:23:42.061 INFO:tasks.workunit.client.1.smithi134.stdout:7/328: mkdir d4/d8/d41/d76 0 2024-04-22T23:23:42.062 INFO:tasks.workunit.client.1.smithi134.stdout:7/329: read d4/d10/f6b [3661565,126498] 0 2024-04-22T23:23:42.066 INFO:tasks.workunit.client.1.smithi134.stdout:7/330: creat d4/d8/d2a/d42/d75/f77 x:0 0 0 2024-04-22T23:23:42.067 INFO:tasks.workunit.client.1.smithi134.stdout:1/299: dwrite db/f1d [0,4194304] 0 2024-04-22T23:23:42.068 INFO:tasks.workunit.client.1.smithi134.stdout:1/300: chown db/d16/c1c 31792930 1 2024-04-22T23:23:42.072 INFO:tasks.workunit.client.1.smithi134.stdout:1/301: mknod db/d15/d17/d20/d2a/d2d/c6a 0 2024-04-22T23:23:42.077 INFO:tasks.workunit.client.1.smithi134.stdout:7/331: truncate d4/d10/d2f/d34/d62/d29/f55 2587948 0 2024-04-22T23:23:42.078 INFO:tasks.workunit.client.1.smithi134.stdout:7/332: chown d4/d10/d2f/d34/d62/f56 0 1 2024-04-22T23:23:42.078 INFO:tasks.workunit.client.1.smithi134.stdout:1/302: rmdir db/d15/d17/d20/d2a/d4d 0 2024-04-22T23:23:42.080 INFO:tasks.workunit.client.1.smithi134.stdout:7/333: write d4/d8/fb [6316558,126377] 0 2024-04-22T23:23:42.082 INFO:tasks.workunit.client.1.smithi134.stdout:1/303: rename l3 to db/d15/d17/d20/d2a/l6b 0 2024-04-22T23:23:42.083 INFO:tasks.workunit.client.1.smithi134.stdout:1/304: write db/f5f [1066408,84687] 0 2024-04-22T23:23:42.084 INFO:tasks.workunit.client.1.smithi134.stdout:7/334: creat d4/d10/d2f/d69/f78 x:0 0 0 2024-04-22T23:23:42.088 INFO:tasks.workunit.client.1.smithi134.stdout:1/305: creat db/d15/d17/d20/d2a/d2d/d2e/f6c x:0 0 0 2024-04-22T23:23:42.095 INFO:tasks.workunit.client.1.smithi134.stdout:1/306: unlink db/d15/d17/d20/c3b 0 2024-04-22T23:23:42.095 INFO:tasks.workunit.client.1.smithi134.stdout:1/307: chown db/d16/c1c 4 1 2024-04-22T23:23:42.096 INFO:tasks.workunit.client.1.smithi134.stdout:1/308: chown db/d15/d17/d20/f4b 117049991 1 2024-04-22T23:23:42.100 INFO:tasks.workunit.client.1.smithi134.stdout:1/309: rename db/d15/d17/d20/d2a/l6b to db/d16/d56/l6d 0 2024-04-22T23:23:42.103 INFO:tasks.workunit.client.1.smithi134.stdout:1/310: fsync db/d15/d17/d20/d2a/d2d/d2e/d42/f66 0 2024-04-22T23:23:42.104 INFO:tasks.workunit.client.1.smithi134.stdout:7/335: dread d4/d10/d2f/d34/d62/f56 [0,4194304] 0 2024-04-22T23:23:42.106 INFO:tasks.workunit.client.1.smithi134.stdout:7/336: mkdir d4/d8/d2a/d42/d79 0 2024-04-22T23:23:42.109 INFO:tasks.workunit.client.1.smithi134.stdout:1/311: rename db/fc to db/d15/d17/d20/d2a/d2d/f6e 0 2024-04-22T23:23:42.110 INFO:tasks.workunit.client.1.smithi134.stdout:7/337: dread d4/d10/f6b [0,4194304] 0 2024-04-22T23:23:42.113 INFO:tasks.workunit.client.1.smithi134.stdout:1/312: dread fa [0,4194304] 0 2024-04-22T23:23:42.115 INFO:tasks.workunit.client.1.smithi134.stdout:1/313: read db/ff [314385,75150] 0 2024-04-22T23:23:42.116 INFO:tasks.workunit.client.1.smithi134.stdout:1/314: dread - db/d16/d56/f63 zero size 2024-04-22T23:23:42.118 INFO:tasks.workunit.client.1.smithi134.stdout:1/315: mkdir db/d15/d6f 0 2024-04-22T23:23:42.120 INFO:tasks.workunit.client.1.smithi134.stdout:1/316: rmdir db/d15/d17/d20/d2a 39 2024-04-22T23:23:42.125 INFO:tasks.workunit.client.1.smithi134.stdout:1/317: mknod db/d15/d17/d20/d2a/d2d/d2e/c70 0 2024-04-22T23:23:42.126 INFO:tasks.workunit.client.1.smithi134.stdout:1/318: fsync db/d15/f34 0 2024-04-22T23:23:42.126 INFO:tasks.workunit.client.1.smithi134.stdout:7/338: dread d4/f7 [0,4194304] 0 2024-04-22T23:23:42.129 INFO:tasks.workunit.client.1.smithi134.stdout:1/319: creat db/f71 x:0 0 0 2024-04-22T23:23:42.132 INFO:tasks.workunit.client.1.smithi134.stdout:7/339: rename d4/f33 to d4/d8/f7a 0 2024-04-22T23:23:42.133 INFO:tasks.workunit.client.1.smithi134.stdout:1/320: rmdir db/d16/d56 39 2024-04-22T23:23:42.135 INFO:tasks.workunit.client.1.smithi134.stdout:1/321: mkdir db/d15/d72 0 2024-04-22T23:23:42.136 INFO:tasks.workunit.client.1.smithi134.stdout:7/340: rename d4/d10/d2f/d34/d62/f2b to d4/f7b 0 2024-04-22T23:23:42.137 INFO:tasks.workunit.client.1.smithi134.stdout:7/341: stat d4/d10/d2f/d37 0 2024-04-22T23:23:42.144 INFO:tasks.workunit.client.1.smithi134.stdout:7/342: dwrite d4/d10/d2f/f36 [0,4194304] 0 2024-04-22T23:23:42.151 INFO:tasks.workunit.client.1.smithi134.stdout:7/343: mknod d4/d8/c7c 0 2024-04-22T23:23:42.154 INFO:tasks.workunit.client.1.smithi134.stdout:7/344: mkdir d4/d8/d2a/d42/d7d 0 2024-04-22T23:23:42.156 INFO:tasks.workunit.client.1.smithi134.stdout:7/345: creat d4/d4a/f7e x:0 0 0 2024-04-22T23:23:42.157 INFO:tasks.workunit.client.1.smithi134.stdout:1/322: truncate db/d16/d56/f57 1335601 0 2024-04-22T23:23:42.158 INFO:tasks.workunit.client.1.smithi134.stdout:1/323: chown db/d16/c32 196661339 1 2024-04-22T23:23:42.163 INFO:tasks.workunit.client.1.smithi134.stdout:7/346: dwrite d4/d10/d2f/d34/f67 [0,4194304] 0 2024-04-22T23:23:42.167 INFO:tasks.workunit.client.1.smithi134.stdout:1/324: link db/d15/d17/d20/l21 db/d15/d17/d20/l73 0 2024-04-22T23:23:42.168 INFO:tasks.workunit.client.1.smithi134.stdout:7/347: mknod d4/d10/d2f/d34/c7f 0 2024-04-22T23:23:42.171 INFO:tasks.workunit.client.1.smithi134.stdout:1/325: write db/d15/d17/d20/d2a/d2d/d2e/f47 [1701283,103027] 0 2024-04-22T23:23:42.172 INFO:tasks.workunit.client.1.smithi134.stdout:7/348: creat d4/d10/d2f/d37/f80 x:0 0 0 2024-04-22T23:23:42.175 INFO:tasks.workunit.client.1.smithi134.stdout:1/326: creat db/d16/d56/f74 x:0 0 0 2024-04-22T23:23:42.178 INFO:tasks.workunit.client.1.smithi134.stdout:1/327: read db/d15/d17/d20/d2a/f36 [2668002,59030] 0 2024-04-22T23:23:42.178 INFO:tasks.workunit.client.1.smithi134.stdout:7/349: rename d4/d10/d2f/d34/d62/c63 to d4/d8/c81 0 2024-04-22T23:23:42.179 INFO:tasks.workunit.client.1.smithi134.stdout:7/350: chown d4/d8/d58 23039 1 2024-04-22T23:23:42.181 INFO:tasks.workunit.client.1.smithi134.stdout:1/328: symlink db/d16/l75 0 2024-04-22T23:23:42.188 INFO:tasks.workunit.client.1.smithi134.stdout:7/351: dwrite d4/d8/d2a/d42/f6a [0,4194304] 0 2024-04-22T23:23:42.190 INFO:tasks.workunit.client.1.smithi134.stdout:1/329: dwrite db/d15/f18 [0,4194304] 0 2024-04-22T23:23:42.192 INFO:tasks.workunit.client.1.smithi134.stdout:7/352: creat d4/d8/d2a/d42/f82 x:0 0 0 2024-04-22T23:23:42.195 INFO:tasks.workunit.client.1.smithi134.stdout:7/353: unlink d4/d10/d2f/f36 0 2024-04-22T23:23:42.200 INFO:tasks.workunit.client.1.smithi134.stdout:1/330: creat db/d15/d17/d20/d2a/d2d/f76 x:0 0 0 2024-04-22T23:23:42.203 INFO:tasks.workunit.client.1.smithi134.stdout:7/354: dread d4/d8/d41/f43 [0,4194304] 0 2024-04-22T23:23:42.205 INFO:tasks.workunit.client.1.smithi134.stdout:7/355: mkdir d4/d8/d3b/d83 0 2024-04-22T23:23:42.206 INFO:tasks.workunit.client.1.smithi134.stdout:1/331: link db/l13 db/d15/d17/d20/l77 0 2024-04-22T23:23:42.207 INFO:tasks.workunit.client.1.smithi134.stdout:7/356: write d4/f7 [1850109,52972] 0 2024-04-22T23:23:42.208 INFO:tasks.workunit.client.1.smithi134.stdout:1/332: chown db/d15/d17/d20/l21 11638 1 2024-04-22T23:23:42.209 INFO:tasks.workunit.client.1.smithi134.stdout:7/357: mkdir d4/d10/d2f/d34/d62/d84 0 2024-04-22T23:23:42.210 INFO:tasks.workunit.client.1.smithi134.stdout:7/358: readlink d4/d10/d2f/l66 0 2024-04-22T23:23:42.211 INFO:tasks.workunit.client.1.smithi134.stdout:7/359: truncate d4/d8/fd 4999074 0 2024-04-22T23:23:42.211 INFO:tasks.workunit.client.1.smithi134.stdout:1/333: creat db/d16/f78 x:0 0 0 2024-04-22T23:23:42.212 INFO:tasks.workunit.client.1.smithi134.stdout:1/334: dread - db/d15/d17/f5a zero size 2024-04-22T23:23:42.213 INFO:tasks.workunit.client.1.smithi134.stdout:7/360: write d4/d10/d2f/d69/f78 [825427,12956] 0 2024-04-22T23:23:42.216 INFO:tasks.workunit.client.1.smithi134.stdout:1/335: symlink db/d15/d17/l79 0 2024-04-22T23:23:42.216 INFO:tasks.workunit.client.1.smithi134.stdout:1/336: readlink db/l4f 0 2024-04-22T23:23:42.218 INFO:tasks.workunit.client.1.smithi134.stdout:1/337: mkdir db/d15/d7a 0 2024-04-22T23:23:42.221 INFO:tasks.workunit.client.1.smithi134.stdout:1/338: dread db/d15/d17/d20/d2a/f44 [0,4194304] 0 2024-04-22T23:23:42.223 INFO:tasks.workunit.client.1.smithi134.stdout:7/361: dread d4/d8/f22 [0,4194304] 0 2024-04-22T23:23:42.224 INFO:tasks.workunit.client.1.smithi134.stdout:1/339: mknod db/d16/c7b 0 2024-04-22T23:23:42.225 INFO:tasks.workunit.client.1.smithi134.stdout:7/362: mkdir d4/d4a/d85 0 2024-04-22T23:23:42.227 INFO:tasks.workunit.client.1.smithi134.stdout:7/363: mkdir d4/d10/d2f/d70/d86 0 2024-04-22T23:23:42.227 INFO:tasks.workunit.client.1.smithi134.stdout:7/364: dread - d4/d4a/f7e zero size 2024-04-22T23:23:42.228 INFO:tasks.workunit.client.1.smithi134.stdout:7/365: chown d4/d10/d2f/d34/d62/d84 127962 1 2024-04-22T23:23:42.231 INFO:tasks.workunit.client.1.smithi134.stdout:7/366: mknod d4/d10/d2f/d34/d54/c87 0 2024-04-22T23:23:42.231 INFO:tasks.workunit.client.1.smithi134.stdout:1/340: dwrite db/d15/d17/d20/d2a/f44 [0,4194304] 0 2024-04-22T23:23:42.235 INFO:tasks.workunit.client.1.smithi134.stdout:1/341: chown db/d15/d17/d20/d2a/d2d/f43 45860 1 2024-04-22T23:23:42.236 INFO:tasks.workunit.client.1.smithi134.stdout:7/367: creat d4/d8/d3b/d83/f88 x:0 0 0 2024-04-22T23:23:42.238 INFO:tasks.workunit.client.1.smithi134.stdout:1/342: creat db/d15/f7c x:0 0 0 2024-04-22T23:23:42.243 INFO:tasks.workunit.client.1.smithi134.stdout:7/368: getdents d4/d8 0 2024-04-22T23:23:42.244 INFO:tasks.workunit.client.1.smithi134.stdout:1/343: rmdir db/d15/d7a 0 2024-04-22T23:23:42.244 INFO:tasks.workunit.client.1.smithi134.stdout:7/369: write d4/d8/d2a/d42/f6a [709338,53233] 0 2024-04-22T23:23:42.246 INFO:tasks.workunit.client.1.smithi134.stdout:1/344: truncate db/d15/d17/d20/f65 701742 0 2024-04-22T23:23:42.258 INFO:tasks.workunit.client.1.smithi134.stdout:7/370: dread d4/f13 [0,4194304] 0 2024-04-22T23:23:42.262 INFO:tasks.workunit.client.1.smithi134.stdout:1/345: write db/d15/d17/d20/d2a/d2d/f6e [1347451,96233] 0 2024-04-22T23:23:42.265 INFO:tasks.workunit.client.1.smithi134.stdout:1/346: fdatasync db/d15/d17/d20/d2a/d2d/f48 0 2024-04-22T23:23:42.269 INFO:tasks.workunit.client.1.smithi134.stdout:1/347: link db/d15/d17/d20/c24 db/d15/d17/d20/d2a/d2d/c7d 0 2024-04-22T23:23:42.273 INFO:tasks.workunit.client.1.smithi134.stdout:1/348: dread db/d15/f18 [0,4194304] 0 2024-04-22T23:23:42.278 INFO:tasks.workunit.client.1.smithi134.stdout:1/349: write db/d15/d17/d20/f46 [2182492,34521] 0 2024-04-22T23:23:42.285 INFO:tasks.workunit.client.1.smithi134.stdout:1/350: dwrite db/d15/d17/f5a [0,4194304] 0 2024-04-22T23:23:42.289 INFO:tasks.workunit.client.1.smithi134.stdout:1/351: creat db/d15/d17/d20/d2a/d2d/f7e x:0 0 0 2024-04-22T23:23:42.290 INFO:tasks.workunit.client.1.smithi134.stdout:1/352: dread - db/d16/f78 zero size 2024-04-22T23:23:42.290 INFO:tasks.workunit.client.1.smithi134.stdout:1/353: stat db/d15/d17/l4a 0 2024-04-22T23:23:42.292 INFO:tasks.workunit.client.1.smithi134.stdout:1/354: write db/ff [7589,60910] 0 2024-04-22T23:23:42.295 INFO:tasks.workunit.client.1.smithi134.stdout:1/355: symlink db/d15/d17/d20/d2a/l7f 0 2024-04-22T23:23:42.296 INFO:tasks.workunit.client.1.smithi134.stdout:1/356: dread - db/d16/f5b zero size 2024-04-22T23:23:42.298 INFO:tasks.workunit.client.1.smithi134.stdout:1/357: rename db/d16/c7b to db/d15/c80 0 2024-04-22T23:23:42.302 INFO:tasks.workunit.client.1.smithi134.stdout:1/358: rename db/c40 to db/d15/d17/d20/d2a/d2d/c81 0 2024-04-22T23:23:42.304 INFO:tasks.workunit.client.1.smithi134.stdout:1/359: rmdir db/d15/d6f 0 2024-04-22T23:23:42.305 INFO:tasks.workunit.client.1.smithi134.stdout:1/360: stat db/d15/d17/d20/l4c 0 2024-04-22T23:23:42.305 INFO:tasks.workunit.client.1.smithi134.stdout:1/361: chown db/d15 16 1 2024-04-22T23:23:42.394 INFO:tasks.workunit.client.1.smithi134.stdout:8/230: sync 2024-04-22T23:23:42.397 INFO:tasks.workunit.client.1.smithi134.stdout:8/231: dread d7/dc/f12 [0,4194304] 0 2024-04-22T23:23:42.398 INFO:tasks.workunit.client.1.smithi134.stdout:8/232: chown d7/db/ld 65993 1 2024-04-22T23:23:42.399 INFO:tasks.workunit.client.1.smithi134.stdout:8/233: write d7/dc/d2c/d37/f3f [458764,126733] 0 2024-04-22T23:23:42.402 INFO:tasks.workunit.client.1.smithi134.stdout:8/234: symlink d7/db/d2d/l44 0 2024-04-22T23:23:42.402 INFO:tasks.workunit.client.1.smithi134.stdout:8/235: chown d7/dc/d19/d25 582 1 2024-04-22T23:23:42.403 INFO:tasks.workunit.client.1.smithi134.stdout:8/236: readlink d7/dc/d2c/l35 0 2024-04-22T23:23:42.403 INFO:tasks.workunit.client.1.smithi134.stdout:8/237: chown d7/dc/d19/c27 4524 1 2024-04-22T23:23:42.404 INFO:tasks.workunit.client.1.smithi134.stdout:8/238: dread d7/db/d2d/f2e [0,4194304] 0 2024-04-22T23:23:42.522 INFO:tasks.workunit.client.1.smithi134.stdout:6/323: sync 2024-04-22T23:23:42.522 INFO:tasks.workunit.client.1.smithi134.stdout:9/210: sync 2024-04-22T23:23:42.522 INFO:tasks.workunit.client.1.smithi134.stdout:5/290: sync 2024-04-22T23:23:42.523 INFO:tasks.workunit.client.1.smithi134.stdout:6/324: stat d0/d19/de/l56 0 2024-04-22T23:23:42.527 INFO:tasks.workunit.client.1.smithi134.stdout:9/211: rename d0/f35 to d0/d2d/d3d/f46 0 2024-04-22T23:23:42.528 INFO:tasks.workunit.client.1.smithi134.stdout:6/325: write d0/f60 [2771532,64709] 0 2024-04-22T23:23:42.529 INFO:tasks.workunit.client.1.smithi134.stdout:6/326: read - d0/d19/de/d1f/f2c zero size 2024-04-22T23:23:42.530 INFO:tasks.workunit.client.1.smithi134.stdout:6/327: fsync d0/d19/de/d2f/f43 0 2024-04-22T23:23:42.531 INFO:tasks.workunit.client.1.smithi134.stdout:9/212: mkdir d0/d7/d23/d47 0 2024-04-22T23:23:42.535 INFO:tasks.workunit.client.1.smithi134.stdout:9/213: mkdir d0/d7/d12/d48 0 2024-04-22T23:23:42.536 INFO:tasks.workunit.client.1.smithi134.stdout:9/214: dread - d0/d7/f41 zero size 2024-04-22T23:23:42.537 INFO:tasks.workunit.client.1.smithi134.stdout:9/215: write d0/d7/f2c [2396263,76151] 0 2024-04-22T23:23:42.539 INFO:tasks.workunit.client.1.smithi134.stdout:5/291: dread d9/d11/d34/d3b/f53 [0,4194304] 0 2024-04-22T23:23:42.540 INFO:tasks.workunit.client.1.smithi134.stdout:9/216: creat d0/d7/d23/d1b/f49 x:0 0 0 2024-04-22T23:23:42.547 INFO:tasks.workunit.client.1.smithi134.stdout:5/292: dwrite d9/dd/d17/d2a/f30 [0,4194304] 0 2024-04-22T23:23:42.551 INFO:tasks.workunit.client.1.smithi134.stdout:5/293: creat d9/d11/d34/d3b/f5f x:0 0 0 2024-04-22T23:23:42.555 INFO:tasks.workunit.client.1.smithi134.stdout:9/217: dread d0/fa [0,4194304] 0 2024-04-22T23:23:42.557 INFO:tasks.workunit.client.1.smithi134.stdout:5/294: getdents d9/d1e 0 2024-04-22T23:23:42.557 INFO:tasks.workunit.client.1.smithi134.stdout:5/295: dread - d9/df/d20/f48 zero size 2024-04-22T23:23:42.558 INFO:tasks.workunit.client.1.smithi134.stdout:5/296: dread - d9/f4a zero size 2024-04-22T23:23:42.558 INFO:tasks.workunit.client.1.smithi134.stdout:9/218: write d0/d7/dc/f32 [3610484,107883] 0 2024-04-22T23:23:42.560 INFO:tasks.workunit.client.1.smithi134.stdout:5/297: mkdir d9/df/d20/d60 0 2024-04-22T23:23:42.561 INFO:tasks.workunit.client.1.smithi134.stdout:9/219: mknod d0/d7/d12/c4a 0 2024-04-22T23:23:42.564 INFO:tasks.workunit.client.1.smithi134.stdout:9/220: symlink d0/d7/d3c/l4b 0 2024-04-22T23:23:42.569 INFO:tasks.workunit.client.1.smithi134.stdout:5/298: dwrite d9/df/d20/f58 [0,4194304] 0 2024-04-22T23:23:42.572 INFO:tasks.workunit.client.1.smithi134.stdout:5/299: mknod d9/d11/d34/c61 0 2024-04-22T23:23:42.574 INFO:tasks.workunit.client.1.smithi134.stdout:9/221: getdents d0/d2d 0 2024-04-22T23:23:42.576 INFO:tasks.workunit.client.1.smithi134.stdout:5/300: getdents d9/d1e 0 2024-04-22T23:23:42.578 INFO:tasks.workunit.client.1.smithi134.stdout:9/222: mkdir d0/d7/d3c/d4c 0 2024-04-22T23:23:42.579 INFO:tasks.workunit.client.1.smithi134.stdout:6/328: dwrite d0/d19/db/d35/f58 [0,4194304] 0 2024-04-22T23:23:42.581 INFO:tasks.workunit.client.1.smithi134.stdout:5/301: creat d9/f62 x:0 0 0 2024-04-22T23:23:42.585 INFO:tasks.workunit.client.1.smithi134.stdout:9/223: rename d0/d1c to d0/d7/d23/d26/d4d 0 2024-04-22T23:23:42.588 INFO:tasks.workunit.client.1.smithi134.stdout:5/302: mknod d9/d11/c63 0 2024-04-22T23:23:42.591 INFO:tasks.workunit.client.1.smithi134.stdout:9/224: rename d0/d7/dc to d0/d2d/d4e 0 2024-04-22T23:23:42.596 INFO:tasks.workunit.client.1.smithi134.stdout:5/303: dwrite d9/f15 [4194304,4194304] 0 2024-04-22T23:23:42.598 INFO:tasks.workunit.client.1.smithi134.stdout:5/304: chown d9/dd/d17/d44/c4e 44 1 2024-04-22T23:23:42.598 INFO:tasks.workunit.client.1.smithi134.stdout:2/241: sync 2024-04-22T23:23:42.598 INFO:tasks.workunit.client.1.smithi134.stdout:0/274: sync 2024-04-22T23:23:42.599 INFO:tasks.workunit.client.1.smithi134.stdout:0/275: write d1/f9 [5849456,116584] 0 2024-04-22T23:23:42.600 INFO:tasks.workunit.client.1.smithi134.stdout:5/305: rmdir d9/dd/d17/d44 39 2024-04-22T23:23:42.600 INFO:tasks.workunit.client.1.smithi134.stdout:2/242: write d2/d42/d54/d33/f51 [268703,111018] 0 2024-04-22T23:23:42.601 INFO:tasks.workunit.client.1.smithi134.stdout:5/306: rename d9/d11 to d9/d11/d64 22 2024-04-22T23:23:42.605 INFO:tasks.workunit.client.1.smithi134.stdout:9/225: write d0/fa [1287294,101358] 0 2024-04-22T23:23:42.607 INFO:tasks.workunit.client.1.smithi134.stdout:6/329: dread d0/f10 [0,4194304] 0 2024-04-22T23:23:42.607 INFO:tasks.workunit.client.1.smithi134.stdout:0/276: rename d1/d5/f47 to d1/d5/dd/d41/f53 0 2024-04-22T23:23:42.609 INFO:tasks.workunit.client.1.smithi134.stdout:2/243: creat d2/d15/d24/d1d/d3d/f58 x:0 0 0 2024-04-22T23:23:42.609 INFO:tasks.workunit.client.1.smithi134.stdout:6/330: read d0/f10 [2521155,104151] 0 2024-04-22T23:23:42.610 INFO:tasks.workunit.client.1.smithi134.stdout:0/277: truncate d1/d12/f29 686100 0 2024-04-22T23:23:42.613 INFO:tasks.workunit.client.1.smithi134.stdout:2/244: creat d2/d42/d54/f59 x:0 0 0 2024-04-22T23:23:42.617 INFO:tasks.workunit.client.1.smithi134.stdout:6/331: unlink d0/d19/de/d12/d2a/f5d 0 2024-04-22T23:23:42.619 INFO:tasks.workunit.client.1.smithi134.stdout:6/332: creat d0/d19/db/d35/f79 x:0 0 0 2024-04-22T23:23:42.619 INFO:tasks.workunit.client.1.smithi134.stdout:5/307: truncate d9/d11/d34/f51 149243 0 2024-04-22T23:23:42.620 INFO:tasks.workunit.client.1.smithi134.stdout:5/308: write d9/f4a [993123,31371] 0 2024-04-22T23:23:42.643 INFO:tasks.workunit.client.1.smithi134.stdout:5/309: symlink d9/d11/d34/d3b/l65 0 2024-04-22T23:23:42.643 INFO:tasks.workunit.client.1.smithi134.stdout:2/245: getdents d2/d15/d24/d1d/d3d 0 2024-04-22T23:23:42.643 INFO:tasks.workunit.client.1.smithi134.stdout:1/362: sync 2024-04-22T23:23:42.644 INFO:tasks.workunit.client.1.smithi134.stdout:7/371: sync 2024-04-22T23:23:42.644 INFO:tasks.workunit.client.1.smithi134.stdout:8/239: sync 2024-04-22T23:23:42.644 INFO:tasks.workunit.client.1.smithi134.stdout:4/461: sync 2024-04-22T23:23:42.644 INFO:tasks.workunit.client.1.smithi134.stdout:3/252: sync 2024-04-22T23:23:42.644 INFO:tasks.workunit.client.1.smithi134.stdout:6/333: write d0/d19/de/f1d [545566,90991] 0 2024-04-22T23:23:42.649 INFO:tasks.workunit.client.1.smithi134.stdout:8/240: creat d7/db/d2d/d39/f45 x:0 0 0 2024-04-22T23:23:42.649 INFO:tasks.workunit.client.1.smithi134.stdout:9/226: dread d0/d7/fb [0,4194304] 0 2024-04-22T23:23:42.651 INFO:tasks.workunit.client.1.smithi134.stdout:3/253: rmdir dc/dd/d28/d2f/d54 39 2024-04-22T23:23:42.652 INFO:tasks.workunit.client.1.smithi134.stdout:1/363: rename db/d16/c59 to db/d15/d17/c82 0 2024-04-22T23:23:42.653 INFO:tasks.workunit.client.1.smithi134.stdout:7/372: truncate d4/f31 398874 0 2024-04-22T23:23:42.654 INFO:tasks.workunit.client.1.smithi134.stdout:1/364: fdatasync db/d16/f78 0 2024-04-22T23:23:42.656 INFO:tasks.workunit.client.1.smithi134.stdout:4/462: rmdir d4/d35/d53 39 2024-04-22T23:23:42.656 INFO:tasks.workunit.client.1.smithi134.stdout:2/246: dwrite d2/dd/d4a/f52 [0,4194304] 0 2024-04-22T23:23:42.660 INFO:tasks.workunit.client.1.smithi134.stdout:9/227: creat d0/d37/f4f x:0 0 0 2024-04-22T23:23:42.660 INFO:tasks.workunit.client.1.smithi134.stdout:9/228: fsync d0/d7/d12/f42 0 2024-04-22T23:23:42.662 INFO:tasks.workunit.client.1.smithi134.stdout:3/254: mknod dc/d2a/d30/c63 0 2024-04-22T23:23:42.662 INFO:tasks.workunit.client.1.smithi134.stdout:7/373: mknod d4/d8/d3b/c89 0 2024-04-22T23:23:42.663 INFO:tasks.workunit.client.1.smithi134.stdout:5/310: creat d9/dd/d17/f66 x:0 0 0 2024-04-22T23:23:42.663 INFO:tasks.workunit.client.1.smithi134.stdout:4/463: rmdir d4/d7/d32 39 2024-04-22T23:23:42.664 INFO:tasks.workunit.client.1.smithi134.stdout:2/247: fdatasync d2/d15/d24/f2a 0 2024-04-22T23:23:42.665 INFO:tasks.workunit.client.1.smithi134.stdout:4/464: write d4/f21 [5522438,128305] 0 2024-04-22T23:23:42.667 INFO:tasks.workunit.client.1.smithi134.stdout:2/248: fsync d2/d15/d24/d1d/f36 0 2024-04-22T23:23:42.667 INFO:tasks.workunit.client.1.smithi134.stdout:6/334: rmdir d0/d4f/d51 0 2024-04-22T23:23:42.670 INFO:tasks.workunit.client.1.smithi134.stdout:7/374: mkdir d4/d10/d2f/d69/d8a 0 2024-04-22T23:23:42.674 INFO:tasks.workunit.client.1.smithi134.stdout:6/335: mkdir d0/d19/de/d12/d5e/d6d/d7a 0 2024-04-22T23:23:42.674 INFO:tasks.workunit.client.1.smithi134.stdout:9/229: rmdir d0/d7/d12/d48 0 2024-04-22T23:23:42.675 INFO:tasks.workunit.client.1.smithi134.stdout:4/465: mkdir d4/d15/d82/d9a 0 2024-04-22T23:23:42.676 INFO:tasks.workunit.client.1.smithi134.stdout:9/230: dread - d0/d2d/d3d/f46 zero size 2024-04-22T23:23:42.676 INFO:tasks.workunit.client.1.smithi134.stdout:8/241: dread d7/dc/d19/f21 [0,4194304] 0 2024-04-22T23:23:42.677 INFO:tasks.workunit.client.1.smithi134.stdout:7/375: fsync d4/d8/f21 0 2024-04-22T23:23:42.680 INFO:tasks.workunit.client.1.smithi134.stdout:2/249: link d2/dd/fe d2/d15/d24/d1d/f5a 0 2024-04-22T23:23:42.681 INFO:tasks.workunit.client.1.smithi134.stdout:7/376: chown d4/d10/d2f/d34/d62/f71 1725441287 1 2024-04-22T23:23:42.685 INFO:tasks.workunit.client.1.smithi134.stdout:8/242: mkdir d7/db/d2d/d33/d46 0 2024-04-22T23:23:42.691 INFO:tasks.workunit.client.1.smithi134.stdout:3/255: getdents dc/dd/d28/d2f/d54 0 2024-04-22T23:23:42.692 INFO:tasks.workunit.client.1.smithi134.stdout:2/250: write d2/dd/f57 [85524,97197] 0 2024-04-22T23:23:42.695 INFO:tasks.workunit.client.1.smithi134.stdout:6/336: dwrite d0/f8 [0,4194304] 0 2024-04-22T23:23:42.704 INFO:tasks.workunit.client.1.smithi134.stdout:8/243: dread - d7/db/f3b zero size 2024-04-22T23:23:42.704 INFO:tasks.workunit.client.1.smithi134.stdout:7/377: truncate d4/d10/d2f/d37/f80 253680 0 2024-04-22T23:23:42.705 INFO:tasks.workunit.client.1.smithi134.stdout:9/231: dwrite d0/d7/d12/d22/f3e [0,4194304] 0 2024-04-22T23:23:42.705 INFO:tasks.workunit.client.1.smithi134.stdout:6/337: chown d0/d19/db/d33/c48 5003 1 2024-04-22T23:23:42.706 INFO:tasks.workunit.client.1.smithi134.stdout:5/311: dread f6 [8388608,4194304] 0 2024-04-22T23:23:42.706 INFO:tasks.workunit.client.1.smithi134.stdout:4/466: dwrite d4/d15/d82/f8b [0,4194304] 0 2024-04-22T23:23:42.710 INFO:tasks.workunit.client.1.smithi134.stdout:4/467: fdatasync d4/dd/f6d 0 2024-04-22T23:23:42.713 INFO:tasks.workunit.client.1.smithi134.stdout:7/378: truncate d4/d8/d41/d5b/f64 292667 0 2024-04-22T23:23:42.715 INFO:tasks.workunit.client.1.smithi134.stdout:3/256: dread dc/dd/d28/d34/f50 [0,4194304] 0 2024-04-22T23:23:42.717 INFO:tasks.workunit.client.1.smithi134.stdout:5/312: fsync d9/d1e/f32 0 2024-04-22T23:23:42.718 INFO:tasks.workunit.client.1.smithi134.stdout:6/338: readlink d0/d19/de/d2f/l54 0 2024-04-22T23:23:42.722 INFO:tasks.workunit.client.1.smithi134.stdout:9/232: symlink d0/d7/d12/l50 0 2024-04-22T23:23:42.725 INFO:tasks.workunit.client.1.smithi134.stdout:9/233: dread - d0/d37/f34 zero size 2024-04-22T23:23:42.728 INFO:tasks.workunit.client.1.smithi134.stdout:8/244: rename d7/dc/d19/d25 to d7/dc/d2c/d37/d47 0 2024-04-22T23:23:42.729 INFO:tasks.workunit.client.1.smithi134.stdout:3/257: dwrite dc/dd/d28/f3a [0,4194304] 0 2024-04-22T23:23:42.731 INFO:tasks.workunit.client.1.smithi134.stdout:5/313: creat d9/f67 x:0 0 0 2024-04-22T23:23:42.736 INFO:tasks.workunit.client.1.smithi134.stdout:9/234: rename d0/d7/d3c to d0/d7/d23/d26/d51 0 2024-04-22T23:23:42.746 INFO:tasks.workunit.client.1.smithi134.stdout:7/379: link d4/d10/d2f/d34/d54/c87 d4/d8/d3b/c8b 0 2024-04-22T23:23:42.746 INFO:tasks.workunit.client.1.smithi134.stdout:9/235: dwrite d0/d7/d23/d26/f29 [0,4194304] 0 2024-04-22T23:23:42.752 INFO:tasks.workunit.client.1.smithi134.stdout:9/236: write d0/d7/d23/f21 [573058,18090] 0 2024-04-22T23:23:42.753 INFO:tasks.workunit.client.1.smithi134.stdout:4/468: write d4/d35/d8e/f8a [1710083,58850] 0 2024-04-22T23:23:42.754 INFO:tasks.workunit.client.1.smithi134.stdout:8/245: rename d7/dc/c30 to d7/db/d2d/d33/d46/c48 0 2024-04-22T23:23:42.757 INFO:tasks.workunit.client.1.smithi134.stdout:9/237: chown d0/d7/d27/f33 1941 1 2024-04-22T23:23:42.758 INFO:tasks.workunit.client.1.smithi134.stdout:9/238: chown d0/d7/d23/d26/d51/d4c 14541638 1 2024-04-22T23:23:42.762 INFO:tasks.workunit.client.1.smithi134.stdout:9/239: write d0/d7/d23/d26/d4d/f3b [1284102,31744] 0 2024-04-22T23:23:42.763 INFO:tasks.workunit.client.1.smithi134.stdout:4/469: rmdir d4/d13 39 2024-04-22T23:23:42.766 INFO:tasks.workunit.client.1.smithi134.stdout:5/314: truncate d9/d11/d34/f40 2647265 0 2024-04-22T23:23:42.768 INFO:tasks.workunit.client.1.smithi134.stdout:3/258: dwrite dc/f22 [0,4194304] 0 2024-04-22T23:23:42.771 INFO:tasks.workunit.client.1.smithi134.stdout:9/240: mknod d0/d37/c52 0 2024-04-22T23:23:42.773 INFO:tasks.workunit.client.1.smithi134.stdout:7/380: getdents d4/d8/d2a/d42/d7d 0 2024-04-22T23:23:42.779 INFO:tasks.workunit.client.1.smithi134.stdout:4/470: unlink d4/d29/c37 0 2024-04-22T23:23:42.782 INFO:tasks.workunit.client.1.smithi134.stdout:5/315: rename d9/df/d20/l38 to d9/df/d20/d60/l68 0 2024-04-22T23:23:42.783 INFO:tasks.workunit.client.1.smithi134.stdout:4/471: chown d4/d15/l2e 63 1 2024-04-22T23:23:42.783 INFO:tasks.workunit.client.1.smithi134.stdout:5/316: chown d9/d1e/f23 654 1 2024-04-22T23:23:42.785 INFO:tasks.workunit.client.1.smithi134.stdout:2/251: dread d2/d15/d24/d1d/f26 [0,4194304] 0 2024-04-22T23:23:42.787 INFO:tasks.workunit.client.1.smithi134.stdout:6/339: dread d0/d19/db/d33/f64 [0,4194304] 0 2024-04-22T23:23:42.790 INFO:tasks.workunit.client.1.smithi134.stdout:3/259: dwrite dc/f45 [0,4194304] 0 2024-04-22T23:23:42.792 INFO:tasks.workunit.client.1.smithi134.stdout:9/241: dwrite d0/d2d/d4e/f32 [0,4194304] 0 2024-04-22T23:23:42.794 INFO:tasks.workunit.client.1.smithi134.stdout:2/252: chown d2/dd/df/l32 201404132 1 2024-04-22T23:23:42.795 INFO:tasks.workunit.client.1.smithi134.stdout:2/253: truncate d2/f2e 677602 0 2024-04-22T23:23:42.798 INFO:tasks.workunit.client.1.smithi134.stdout:7/381: mkdir d4/d8c 0 2024-04-22T23:23:42.800 INFO:tasks.workunit.client.1.smithi134.stdout:7/382: chown d4/d8/d2a/f47 2060518263 1 2024-04-22T23:23:42.801 INFO:tasks.workunit.client.1.smithi134.stdout:8/246: creat d7/dc/d2c/f49 x:0 0 0 2024-04-22T23:23:42.803 INFO:tasks.workunit.client.1.smithi134.stdout:4/472: mkdir d4/d9b 0 2024-04-22T23:23:42.806 INFO:tasks.workunit.client.1.smithi134.stdout:9/242: mknod d0/d7/d23/d26/c53 0 2024-04-22T23:23:42.808 INFO:tasks.workunit.client.1.smithi134.stdout:8/247: dwrite f4 [0,4194304] 0 2024-04-22T23:23:42.808 INFO:tasks.workunit.client.1.smithi134.stdout:6/340: creat d0/d6f/f7b x:0 0 0 2024-04-22T23:23:42.809 INFO:tasks.workunit.client.1.smithi134.stdout:2/254: truncate d2/f30 90220 0 2024-04-22T23:23:42.811 INFO:tasks.workunit.client.1.smithi134.stdout:7/383: creat d4/d8/d3b/f8d x:0 0 0 2024-04-22T23:23:42.817 INFO:tasks.workunit.client.1.smithi134.stdout:7/384: chown d4/d8/d41/d5b/l61 1754839434 1 2024-04-22T23:23:42.817 INFO:tasks.workunit.client.1.smithi134.stdout:9/243: write d0/d7/d23/f45 [3507659,18079] 0 2024-04-22T23:23:42.820 INFO:tasks.workunit.client.1.smithi134.stdout:6/341: dwrite d0/f55 [4194304,4194304] 0 2024-04-22T23:23:42.824 INFO:tasks.workunit.client.1.smithi134.stdout:8/248: creat d7/f4a x:0 0 0 2024-04-22T23:23:42.824 INFO:tasks.workunit.client.1.smithi134.stdout:8/249: chown d7/db/d42 1305164 1 2024-04-22T23:23:42.824 INFO:tasks.workunit.client.1.smithi134.stdout:7/385: rmdir d4/d10/d2f/d34/d62/d29 39 2024-04-22T23:23:42.826 INFO:tasks.workunit.client.1.smithi134.stdout:6/342: fsync d0/d19/f1b 0 2024-04-22T23:23:42.830 INFO:tasks.workunit.client.1.smithi134.stdout:8/250: creat d7/db/d2d/d39/f4b x:0 0 0 2024-04-22T23:23:42.831 INFO:tasks.workunit.client.1.smithi134.stdout:7/386: mknod d4/d8/d41/d5b/c8e 0 2024-04-22T23:23:42.834 INFO:tasks.workunit.client.1.smithi134.stdout:8/251: read d7/db/d2d/d39/f3d [21560,45216] 0 2024-04-22T23:23:42.835 INFO:tasks.workunit.client.1.smithi134.stdout:6/343: symlink d0/d19/db/d75/l7c 0 2024-04-22T23:23:42.837 INFO:tasks.workunit.client.1.smithi134.stdout:7/387: dread d4/d10/f1f [0,4194304] 0 2024-04-22T23:23:42.837 INFO:tasks.workunit.client.1.smithi134.stdout:9/244: link d0/d7/d27/f33 d0/f54 0 2024-04-22T23:23:42.840 INFO:tasks.workunit.client.1.smithi134.stdout:6/344: mkdir d0/d19/de/d1f/d7d 0 2024-04-22T23:23:42.840 INFO:tasks.workunit.client.1.smithi134.stdout:6/345: readlink d0/d19/db/l65 0 2024-04-22T23:23:42.841 INFO:tasks.workunit.client.1.smithi134.stdout:6/346: chown d0/d19/de/f5f 15506 1 2024-04-22T23:23:42.842 INFO:tasks.workunit.client.1.smithi134.stdout:8/252: rename d7/db/fe to d7/db/d2d/d33/f4c 0 2024-04-22T23:23:42.842 INFO:tasks.workunit.client.1.smithi134.stdout:6/347: write d0/f8 [2834740,90676] 0 2024-04-22T23:23:42.845 INFO:tasks.workunit.client.1.smithi134.stdout:9/245: creat d0/d7/d23/f55 x:0 0 0 2024-04-22T23:23:42.849 INFO:tasks.workunit.client.1.smithi134.stdout:8/253: truncate d7/f1c 1634399 0 2024-04-22T23:23:42.852 INFO:tasks.workunit.client.1.smithi134.stdout:9/246: fsync d0/d7/fb 0 2024-04-22T23:23:42.853 INFO:tasks.workunit.client.1.smithi134.stdout:7/388: dread d4/f2e [0,4194304] 0 2024-04-22T23:23:42.855 INFO:tasks.workunit.client.1.smithi134.stdout:8/254: creat d7/dc/d2c/d37/f4d x:0 0 0 2024-04-22T23:23:42.859 INFO:tasks.workunit.client.1.smithi134.stdout:9/247: chown d0/d37/l3a 305 1 2024-04-22T23:23:42.859 INFO:tasks.workunit.client.1.smithi134.stdout:8/255: dread d7/dc/d2c/d37/f3f [0,4194304] 0 2024-04-22T23:23:42.860 INFO:tasks.workunit.client.1.smithi134.stdout:9/248: write d0/d7/d12/f1f [4634809,43429] 0 2024-04-22T23:23:42.861 INFO:tasks.workunit.client.1.smithi134.stdout:7/389: getdents d4/d8/d41/d76 0 2024-04-22T23:23:42.864 INFO:tasks.workunit.client.1.smithi134.stdout:8/256: mknod d7/db/c4e 0 2024-04-22T23:23:42.868 INFO:tasks.workunit.client.1.smithi134.stdout:9/249: dwrite d0/d37/f4f [0,4194304] 0 2024-04-22T23:23:42.869 INFO:tasks.workunit.client.1.smithi134.stdout:8/257: mkdir d7/dc/d2c/d4f 0 2024-04-22T23:23:42.872 INFO:tasks.workunit.client.1.smithi134.stdout:7/390: creat d4/d10/d2f/d34/d62/d84/f8f x:0 0 0 2024-04-22T23:23:42.876 INFO:tasks.workunit.client.1.smithi134.stdout:9/250: getdents d0/d7/d23/d47 0 2024-04-22T23:23:42.878 INFO:tasks.workunit.client.1.smithi134.stdout:6/348: dwrite d0/d19/de/d12/d2a/f30 [0,4194304] 0 2024-04-22T23:23:42.880 INFO:tasks.workunit.client.1.smithi134.stdout:7/391: rename d4/d8/d2a/d42/c49 to d4/d8/c90 0 2024-04-22T23:23:42.881 INFO:tasks.workunit.client.1.smithi134.stdout:9/251: mknod d0/d7/d23/c56 0 2024-04-22T23:23:42.888 INFO:tasks.workunit.client.1.smithi134.stdout:9/252: creat d0/f57 x:0 0 0 2024-04-22T23:23:42.890 INFO:tasks.workunit.client.1.smithi134.stdout:9/253: chown d0/d7/d12/c4a 54299 1 2024-04-22T23:23:42.892 INFO:tasks.workunit.client.1.smithi134.stdout:6/349: dwrite d0/d19/de/d12/d2a/f4c [0,4194304] 0 2024-04-22T23:23:42.895 INFO:tasks.workunit.client.1.smithi134.stdout:0/278: sync 2024-04-22T23:23:42.896 INFO:tasks.workunit.client.1.smithi134.stdout:1/365: sync 2024-04-22T23:23:42.897 INFO:tasks.workunit.client.1.smithi134.stdout:9/254: rmdir d0/d7 39 2024-04-22T23:23:42.897 INFO:tasks.workunit.client.1.smithi134.stdout:8/258: dread d7/db/f1f [0,4194304] 0 2024-04-22T23:23:42.900 INFO:tasks.workunit.client.1.smithi134.stdout:0/279: creat d1/d2/f54 x:0 0 0 2024-04-22T23:23:42.904 INFO:tasks.workunit.client.1.smithi134.stdout:0/280: creat d1/d2/d1d/f55 x:0 0 0 2024-04-22T23:23:42.904 INFO:tasks.workunit.client.1.smithi134.stdout:6/350: dread d0/f1a [0,4194304] 0 2024-04-22T23:23:42.905 INFO:tasks.workunit.client.1.smithi134.stdout:1/366: fsync db/d16/d56/f57 0 2024-04-22T23:23:42.907 INFO:tasks.workunit.client.1.smithi134.stdout:7/392: getdents d4/d8/d2a/d42/d75 0 2024-04-22T23:23:42.907 INFO:tasks.workunit.client.1.smithi134.stdout:0/281: fdatasync d1/d5/dd/f2b 0 2024-04-22T23:23:42.908 INFO:tasks.workunit.client.1.smithi134.stdout:9/255: rename d0/d7/d12/f1a to d0/d7/d23/d26/f58 0 2024-04-22T23:23:42.910 INFO:tasks.workunit.client.1.smithi134.stdout:0/282: fdatasync d1/d12/f2e 0 2024-04-22T23:23:42.911 INFO:tasks.workunit.client.1.smithi134.stdout:8/259: dwrite d7/dc/d2c/f49 [0,4194304] 0 2024-04-22T23:23:42.911 INFO:tasks.workunit.client.1.smithi134.stdout:0/283: dread - d1/d2/f54 zero size 2024-04-22T23:23:42.913 INFO:tasks.workunit.client.1.smithi134.stdout:8/260: truncate f4 8566152 0 2024-04-22T23:23:42.913 INFO:tasks.workunit.client.1.smithi134.stdout:0/284: write d1/d5/dd/d11/f18 [4878114,117137] 0 2024-04-22T23:23:42.915 INFO:tasks.workunit.client.1.smithi134.stdout:8/261: dread - d7/dc/d2c/d37/f4d zero size 2024-04-22T23:23:42.916 INFO:tasks.workunit.client.1.smithi134.stdout:9/256: chown d0/d7/d23/d26/d4d/l36 142315 1 2024-04-22T23:23:42.919 INFO:tasks.workunit.client.1.smithi134.stdout:9/257: write d0/d7/d12/f1f [8195617,62381] 0 2024-04-22T23:23:42.919 INFO:tasks.workunit.client.1.smithi134.stdout:7/393: symlink d4/d8/d58/l91 0 2024-04-22T23:23:42.921 INFO:tasks.workunit.client.1.smithi134.stdout:6/351: dread d0/f26 [0,4194304] 0 2024-04-22T23:23:42.921 INFO:tasks.workunit.client.1.smithi134.stdout:0/285: creat d1/d2/f56 x:0 0 0 2024-04-22T23:23:42.922 INFO:tasks.workunit.client.1.smithi134.stdout:8/262: symlink d7/dc/d2c/d37/d47/l50 0 2024-04-22T23:23:42.925 INFO:tasks.workunit.client.1.smithi134.stdout:7/394: creat d4/d4a/f92 x:0 0 0 2024-04-22T23:23:42.927 INFO:tasks.workunit.client.1.smithi134.stdout:8/263: creat d7/dc/d2c/d37/d47/f51 x:0 0 0 2024-04-22T23:23:42.928 INFO:tasks.workunit.client.1.smithi134.stdout:0/286: rename d1/db/c30 to d1/d5/dd/d20/c57 0 2024-04-22T23:23:42.930 INFO:tasks.workunit.client.1.smithi134.stdout:9/258: dwrite d0/d7/f19 [0,4194304] 0 2024-04-22T23:23:42.930 INFO:tasks.workunit.client.1.smithi134.stdout:1/367: getdents db/d15 0 2024-04-22T23:23:42.932 INFO:tasks.workunit.client.1.smithi134.stdout:9/259: fsync d0/d37/f34 0 2024-04-22T23:23:42.935 INFO:tasks.workunit.client.1.smithi134.stdout:0/287: stat d1/d5/dd/d11/f18 0 2024-04-22T23:23:42.935 INFO:tasks.workunit.client.1.smithi134.stdout:1/368: readlink db/d15/d17/d20/l68 0 2024-04-22T23:23:42.937 INFO:tasks.workunit.client.1.smithi134.stdout:6/352: dwrite d0/f26 [0,4194304] 0 2024-04-22T23:23:42.938 INFO:tasks.workunit.client.1.smithi134.stdout:9/260: truncate d0/d7/f17 2831689 0 2024-04-22T23:23:42.939 INFO:tasks.workunit.client.1.smithi134.stdout:8/264: mknod d7/db/d42/c52 0 2024-04-22T23:23:42.939 INFO:tasks.workunit.client.1.smithi134.stdout:0/288: readlink d1/db/l13 0 2024-04-22T23:23:42.940 INFO:tasks.workunit.client.1.smithi134.stdout:8/265: stat d7/dc 0 2024-04-22T23:23:42.940 INFO:tasks.workunit.client.1.smithi134.stdout:1/369: creat db/d16/d56/f83 x:0 0 0 2024-04-22T23:23:42.941 INFO:tasks.workunit.client.1.smithi134.stdout:9/261: fdatasync d0/d7/fb 0 2024-04-22T23:23:42.943 INFO:tasks.workunit.client.1.smithi134.stdout:1/370: read - db/d15/d17/d20/d2a/d2d/f7e zero size 2024-04-22T23:23:42.948 INFO:tasks.workunit.client.1.smithi134.stdout:6/353: creat d0/d19/de/d12/f7e x:0 0 0 2024-04-22T23:23:42.949 INFO:tasks.workunit.client.1.smithi134.stdout:1/371: creat db/d15/d17/f84 x:0 0 0 2024-04-22T23:23:42.953 INFO:tasks.workunit.client.1.smithi134.stdout:9/262: dwrite d0/d37/f4f [0,4194304] 0 2024-04-22T23:23:42.955 INFO:tasks.workunit.client.1.smithi134.stdout:1/372: readlink db/d16/d56/l6d 0 2024-04-22T23:23:42.956 INFO:tasks.workunit.client.1.smithi134.stdout:6/354: fdatasync d0/d19/db/fd 0 2024-04-22T23:23:42.957 INFO:tasks.workunit.client.1.smithi134.stdout:8/266: dwrite d7/dc/d2c/d37/d47/f51 [0,4194304] 0 2024-04-22T23:23:42.960 INFO:tasks.workunit.client.1.smithi134.stdout:9/263: chown d0/d2d/d4e/l28 2082 1 2024-04-22T23:23:42.962 INFO:tasks.workunit.client.1.smithi134.stdout:7/395: dwrite d4/d8/f22 [0,4194304] 0 2024-04-22T23:23:42.965 INFO:tasks.workunit.client.1.smithi134.stdout:9/264: stat d0/d7/f19 0 2024-04-22T23:23:42.965 INFO:tasks.workunit.client.1.smithi134.stdout:0/289: write d1/d5/dd/f16 [1568753,124142] 0 2024-04-22T23:23:42.969 INFO:tasks.workunit.client.1.smithi134.stdout:6/355: write d0/d19/db/fd [467190,38845] 0 2024-04-22T23:23:42.972 INFO:tasks.workunit.client.1.smithi134.stdout:1/373: rename db/d15/d17/d20/l73 to db/d16/l85 0 2024-04-22T23:23:42.978 INFO:tasks.workunit.client.1.smithi134.stdout:8/267: dwrite d7/dc/d2c/d37/f4d [0,4194304] 0 2024-04-22T23:23:42.980 INFO:tasks.workunit.client.1.smithi134.stdout:6/356: mkdir d0/d19/db/d35/d7f 0 2024-04-22T23:23:42.982 INFO:tasks.workunit.client.1.smithi134.stdout:7/396: dread d4/d8/d2a/f2c [0,4194304] 0 2024-04-22T23:23:42.984 INFO:tasks.workunit.client.1.smithi134.stdout:1/374: unlink db/f12 0 2024-04-22T23:23:42.984 INFO:tasks.workunit.client.1.smithi134.stdout:0/290: rename d1/d12 to d1/d5/d58 0 2024-04-22T23:23:42.987 INFO:tasks.workunit.client.1.smithi134.stdout:0/291: fdatasync d1/d2/d1d/f55 0 2024-04-22T23:23:42.988 INFO:tasks.workunit.client.1.smithi134.stdout:8/268: chown d7/dc/c2a 1671507112 1 2024-04-22T23:23:42.989 INFO:tasks.workunit.client.1.smithi134.stdout:0/292: write d1/f44 [261864,105860] 0 2024-04-22T23:23:42.990 INFO:tasks.workunit.client.1.smithi134.stdout:7/397: unlink d4/d8/d58/c6f 0 2024-04-22T23:23:42.990 INFO:tasks.workunit.client.1.smithi134.stdout:6/357: dread d0/f8 [0,4194304] 0 2024-04-22T23:23:42.993 INFO:tasks.workunit.client.1.smithi134.stdout:0/293: truncate d1/d5/d58/f2e 985504 0 2024-04-22T23:23:42.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:42 smithi134 ceph-mon[34609]: pgmap v276: 65 pgs: 65 active+clean; 2.6 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 84 MiB/s rd, 269 MiB/s wr, 500 op/s 2024-04-22T23:23:42.996 INFO:tasks.workunit.client.1.smithi134.stdout:6/358: rmdir d0/d19/de/d1f 39 2024-04-22T23:23:42.997 INFO:tasks.workunit.client.1.smithi134.stdout:6/359: dread - d0/d19/de/d12/d5e/f66 zero size 2024-04-22T23:23:42.998 INFO:tasks.workunit.client.1.smithi134.stdout:1/375: mkdir db/d15/d17/d20/d2a/d2d/d86 0 2024-04-22T23:23:43.001 INFO:tasks.workunit.client.1.smithi134.stdout:9/265: dread d0/d7/f9 [4194304,4194304] 0 2024-04-22T23:23:43.002 INFO:tasks.workunit.client.1.smithi134.stdout:0/294: dread d1/f44 [0,4194304] 0 2024-04-22T23:23:43.002 INFO:tasks.workunit.client.1.smithi134.stdout:7/398: mknod d4/d10/d2f/d69/d8a/c93 0 2024-04-22T23:23:43.003 INFO:tasks.workunit.client.1.smithi134.stdout:0/295: write d1/f10 [3284156,20456] 0 2024-04-22T23:23:43.006 INFO:tasks.workunit.client.1.smithi134.stdout:6/360: creat d0/d19/de/d12/d2a/d2d/f80 x:0 0 0 2024-04-22T23:23:43.007 INFO:tasks.workunit.client.1.smithi134.stdout:9/266: creat d0/d7/d27/f59 x:0 0 0 2024-04-22T23:23:43.009 INFO:tasks.workunit.client.1.smithi134.stdout:1/376: mkdir db/d16/d87 0 2024-04-22T23:23:43.009 INFO:tasks.workunit.client.1.smithi134.stdout:9/267: read d0/d7/fb [820248,27802] 0 2024-04-22T23:23:43.011 INFO:tasks.workunit.client.1.smithi134.stdout:7/399: truncate d4/d8/d3b/f68 39289 0 2024-04-22T23:23:43.011 INFO:tasks.workunit.client.1.smithi134.stdout:0/296: mknod d1/d2/d1d/c59 0 2024-04-22T23:23:43.012 INFO:tasks.workunit.client.1.smithi134.stdout:8/269: getdents d7/dc/d2c 0 2024-04-22T23:23:43.013 INFO:tasks.workunit.client.1.smithi134.stdout:1/377: readlink db/l37 0 2024-04-22T23:23:43.015 INFO:tasks.workunit.client.1.smithi134.stdout:9/268: mkdir d0/d7/d23/d1b/d5a 0 2024-04-22T23:23:43.017 INFO:tasks.workunit.client.1.smithi134.stdout:6/361: dwrite d0/d19/de/d12/f7e [0,4194304] 0 2024-04-22T23:23:43.019 INFO:tasks.workunit.client.1.smithi134.stdout:7/400: write d4/d8/f7a [3022111,63187] 0 2024-04-22T23:23:43.020 INFO:tasks.workunit.client.1.smithi134.stdout:1/378: creat db/d15/d17/d20/d2a/f88 x:0 0 0 2024-04-22T23:23:43.022 INFO:tasks.workunit.client.1.smithi134.stdout:8/270: mknod d7/dc/d2c/d37/d3a/c53 0 2024-04-22T23:23:43.024 INFO:tasks.workunit.client.1.smithi134.stdout:7/401: chown d4/d10/l4b 105254 1 2024-04-22T23:23:43.024 INFO:tasks.workunit.client.1.smithi134.stdout:6/362: fsync d0/d19/f9 0 2024-04-22T23:23:43.024 INFO:tasks.workunit.client.1.smithi134.stdout:0/297: dwrite d1/f10 [4194304,4194304] 0 2024-04-22T23:23:43.026 INFO:tasks.workunit.client.1.smithi134.stdout:6/363: chown d0/d19/de/c23 1 1 2024-04-22T23:23:43.026 INFO:tasks.workunit.client.1.smithi134.stdout:1/379: chown db/d15/d17/d20/f45 1 1 2024-04-22T23:23:43.029 INFO:tasks.workunit.client.1.smithi134.stdout:6/364: truncate d0/d19/db/d33/f64 4817635 0 2024-04-22T23:23:43.029 INFO:tasks.workunit.client.1.smithi134.stdout:7/402: rmdir d4/d4a 39 2024-04-22T23:23:43.031 INFO:tasks.workunit.client.1.smithi134.stdout:8/271: dread d7/dc/d2c/d37/d47/f34 [0,4194304] 0 2024-04-22T23:23:43.031 INFO:tasks.workunit.client.1.smithi134.stdout:9/269: creat d0/d7/f5b x:0 0 0 2024-04-22T23:23:43.032 INFO:tasks.workunit.client.1.smithi134.stdout:9/270: readlink d0/d37/l3a 0 2024-04-22T23:23:43.034 INFO:tasks.workunit.client.1.smithi134.stdout:6/365: fsync d0/f10 0 2024-04-22T23:23:43.036 INFO:tasks.workunit.client.1.smithi134.stdout:1/380: mkdir db/d16/d89 0 2024-04-22T23:23:43.037 INFO:tasks.workunit.client.1.smithi134.stdout:0/298: symlink d1/d5/dd/d11/l5a 0 2024-04-22T23:23:43.041 INFO:tasks.workunit.client.1.smithi134.stdout:7/403: dwrite d4/d8/fb [4194304,4194304] 0 2024-04-22T23:23:43.041 INFO:tasks.workunit.client.1.smithi134.stdout:8/272: fsync d7/dc/f12 0 2024-04-22T23:23:43.042 INFO:tasks.workunit.client.1.smithi134.stdout:7/404: fsync d4/d8/d3b/f68 0 2024-04-22T23:23:43.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:42 smithi067 ceph-mon[26453]: pgmap v276: 65 pgs: 65 active+clean; 2.6 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 84 MiB/s rd, 269 MiB/s wr, 500 op/s 2024-04-22T23:23:43.046 INFO:tasks.workunit.client.1.smithi134.stdout:9/271: creat d0/f5c x:0 0 0 2024-04-22T23:23:43.048 INFO:tasks.workunit.client.1.smithi134.stdout:0/299: creat d1/d5/dd/f5b x:0 0 0 2024-04-22T23:23:43.052 INFO:tasks.workunit.client.1.smithi134.stdout:6/366: truncate d0/d19/db/f68 3724526 0 2024-04-22T23:23:43.054 INFO:tasks.workunit.client.1.smithi134.stdout:9/272: mkdir d0/d7/d23/d26/d4d/d5d 0 2024-04-22T23:23:43.055 INFO:tasks.workunit.client.1.smithi134.stdout:1/381: dwrite db/d15/d17/d20/d2a/d2d/f7e [0,4194304] 0 2024-04-22T23:23:43.060 INFO:tasks.workunit.client.1.smithi134.stdout:8/273: dwrite d7/f4a [0,4194304] 0 2024-04-22T23:23:43.061 INFO:tasks.workunit.client.1.smithi134.stdout:0/300: creat d1/f5c x:0 0 0 2024-04-22T23:23:43.062 INFO:tasks.workunit.client.1.smithi134.stdout:1/382: fdatasync db/d15/d17/d20/d2a/d2d/d2e/d42/f66 0 2024-04-22T23:23:43.068 INFO:tasks.workunit.client.1.smithi134.stdout:7/405: dwrite d4/d10/d2f/d34/d62/f56 [4194304,4194304] 0 2024-04-22T23:23:43.069 INFO:tasks.workunit.client.1.smithi134.stdout:9/273: link d0/f5c d0/d7/f5e 0 2024-04-22T23:23:43.069 INFO:tasks.workunit.client.1.smithi134.stdout:0/301: creat d1/d5/dd/d41/f5d x:0 0 0 2024-04-22T23:23:43.075 INFO:tasks.workunit.client.1.smithi134.stdout:8/274: rename d7/dc/d2c/d37/d47/f26 to d7/db/d2d/f54 0 2024-04-22T23:23:43.077 INFO:tasks.workunit.client.1.smithi134.stdout:5/317: sync 2024-04-22T23:23:43.077 INFO:tasks.workunit.client.1.smithi134.stdout:0/302: unlink d1/f5c 0 2024-04-22T23:23:43.078 INFO:tasks.workunit.client.1.smithi134.stdout:9/274: mknod d0/d7/d23/c5f 0 2024-04-22T23:23:43.082 INFO:tasks.workunit.client.1.smithi134.stdout:6/367: dwrite d0/f8 [4194304,4194304] 0 2024-04-22T23:23:43.082 INFO:tasks.workunit.client.1.smithi134.stdout:7/406: chown d4/d10/d2f/d34/d54/c87 56 1 2024-04-22T23:23:43.082 INFO:tasks.workunit.client.1.smithi134.stdout:0/303: creat d1/d5/f5e x:0 0 0 2024-04-22T23:23:43.083 INFO:tasks.workunit.client.1.smithi134.stdout:9/275: write d0/d37/f34 [794077,97534] 0 2024-04-22T23:23:43.088 INFO:tasks.workunit.client.1.smithi134.stdout:1/383: dwrite db/ff [0,4194304] 0 2024-04-22T23:23:43.089 INFO:tasks.workunit.client.1.smithi134.stdout:8/275: dread d7/db/d2d/d39/f3d [0,4194304] 0 2024-04-22T23:23:43.098 INFO:tasks.workunit.client.1.smithi134.stdout:9/276: truncate d0/d7/f19 4637340 0 2024-04-22T23:23:43.100 INFO:tasks.workunit.client.1.smithi134.stdout:7/407: mknod d4/d10/d2f/d70/d86/c94 0 2024-04-22T23:23:43.101 INFO:tasks.workunit.client.1.smithi134.stdout:0/304: getdents d1/db 0 2024-04-22T23:23:43.103 INFO:tasks.workunit.client.1.smithi134.stdout:7/408: chown d4/d8/d58/l91 889 1 2024-04-22T23:23:43.103 INFO:tasks.workunit.client.1.smithi134.stdout:1/384: dread db/d15/d17/d20/d2a/d2d/f7e [0,4194304] 0 2024-04-22T23:23:43.103 INFO:tasks.workunit.client.1.smithi134.stdout:8/276: chown d7/dc/f2b 6269838 1 2024-04-22T23:23:43.104 INFO:tasks.workunit.client.1.smithi134.stdout:9/277: symlink d0/d7/d12/l60 0 2024-04-22T23:23:43.107 INFO:tasks.workunit.client.1.smithi134.stdout:6/368: dread d0/d19/f9 [0,4194304] 0 2024-04-22T23:23:43.108 INFO:tasks.workunit.client.1.smithi134.stdout:1/385: mknod db/d15/c8a 0 2024-04-22T23:23:43.109 INFO:tasks.workunit.client.1.smithi134.stdout:9/278: write d0/d7/d23/d26/d4d/f3b [1065385,117190] 0 2024-04-22T23:23:43.109 INFO:tasks.workunit.client.1.smithi134.stdout:1/386: chown db/d15/f34 132 1 2024-04-22T23:23:43.110 INFO:tasks.workunit.client.1.smithi134.stdout:0/305: rename d1/d5/dd/d20/f25 to d1/d2/d1d/f5f 0 2024-04-22T23:23:43.115 INFO:tasks.workunit.client.1.smithi134.stdout:6/369: unlink d0/f27 0 2024-04-22T23:23:43.118 INFO:tasks.workunit.client.1.smithi134.stdout:0/306: dwrite d1/d5/dd/d20/f1e [0,4194304] 0 2024-04-22T23:23:43.120 INFO:tasks.workunit.client.1.smithi134.stdout:8/277: rename d7/dc/c2a to d7/dc/d2c/d37/d47/c55 0 2024-04-22T23:23:43.128 INFO:tasks.workunit.client.1.smithi134.stdout:5/318: dread d9/fa [0,4194304] 0 2024-04-22T23:23:43.129 INFO:tasks.workunit.client.1.smithi134.stdout:6/370: dwrite d0/d19/de/f1d [0,4194304] 0 2024-04-22T23:23:43.131 INFO:tasks.workunit.client.1.smithi134.stdout:6/371: fsync d0/d19/de/d12/f7e 0 2024-04-22T23:23:43.131 INFO:tasks.workunit.client.1.smithi134.stdout:9/279: mknod d0/d7/d23/d26/d4d/d5d/c61 0 2024-04-22T23:23:43.133 INFO:tasks.workunit.client.1.smithi134.stdout:9/280: fdatasync d0/d37/f4f 0 2024-04-22T23:23:43.134 INFO:tasks.workunit.client.1.smithi134.stdout:9/281: readlink d0/d7/d23/d26/l2e 0 2024-04-22T23:23:43.135 INFO:tasks.workunit.client.1.smithi134.stdout:7/409: dread d4/d8/fd [0,4194304] 0 2024-04-22T23:23:43.135 INFO:tasks.workunit.client.1.smithi134.stdout:0/307: symlink d1/d5/d58/d22/l60 0 2024-04-22T23:23:43.136 INFO:tasks.workunit.client.1.smithi134.stdout:8/278: creat d7/db/f56 x:0 0 0 2024-04-22T23:23:43.144 INFO:tasks.workunit.client.1.smithi134.stdout:9/282: dwrite d0/d7/d23/d26/f29 [0,4194304] 0 2024-04-22T23:23:43.145 INFO:tasks.workunit.client.1.smithi134.stdout:0/308: symlink d1/d5/d58/l61 0 2024-04-22T23:23:43.146 INFO:tasks.workunit.client.1.smithi134.stdout:9/283: dread - d0/d7/d23/f55 zero size 2024-04-22T23:23:43.148 INFO:tasks.workunit.client.1.smithi134.stdout:0/309: chown d1/d5/dd/d20/l43 2 1 2024-04-22T23:23:43.149 INFO:tasks.workunit.client.1.smithi134.stdout:1/387: rename db/d16/d56/f74 to db/f8b 0 2024-04-22T23:23:43.156 INFO:tasks.workunit.client.1.smithi134.stdout:1/388: symlink db/d16/d56/l8c 0 2024-04-22T23:23:43.158 INFO:tasks.workunit.client.1.smithi134.stdout:1/389: chown db/d15/d17/d20/d2a/d2d/d2e/f6c 20896 1 2024-04-22T23:23:43.159 INFO:tasks.workunit.client.1.smithi134.stdout:5/319: rename d9/c28 to d9/dd/d52/c69 0 2024-04-22T23:23:43.162 INFO:tasks.workunit.client.1.smithi134.stdout:0/310: creat d1/d5/d58/d22/f62 x:0 0 0 2024-04-22T23:23:43.163 INFO:tasks.workunit.client.1.smithi134.stdout:0/311: stat d1/d2/f56 0 2024-04-22T23:23:43.165 INFO:tasks.workunit.client.1.smithi134.stdout:8/279: dwrite d7/dc/f22 [0,4194304] 0 2024-04-22T23:23:43.165 INFO:tasks.workunit.client.1.smithi134.stdout:9/284: rename d0/d7/d23/f21 to d0/d2d/d4e/f62 0 2024-04-22T23:23:43.166 INFO:tasks.workunit.client.1.smithi134.stdout:6/372: truncate d0/f26 4193363 0 2024-04-22T23:23:43.167 INFO:tasks.workunit.client.1.smithi134.stdout:5/320: creat d9/d11/d34/f6a x:0 0 0 2024-04-22T23:23:43.169 INFO:tasks.workunit.client.1.smithi134.stdout:1/390: dread fa [0,4194304] 0 2024-04-22T23:23:43.172 INFO:tasks.workunit.client.1.smithi134.stdout:5/321: creat d9/dd/d52/f6b x:0 0 0 2024-04-22T23:23:43.173 INFO:tasks.workunit.client.1.smithi134.stdout:5/322: read - d9/f67 zero size 2024-04-22T23:23:43.173 INFO:tasks.workunit.client.1.smithi134.stdout:0/312: dread d1/d5/dd/d20/f1e [0,4194304] 0 2024-04-22T23:23:43.174 INFO:tasks.workunit.client.1.smithi134.stdout:1/391: rmdir db/d15/d17 39 2024-04-22T23:23:43.177 INFO:tasks.workunit.client.1.smithi134.stdout:6/373: rename d0/d19/de/c41 to d0/d19/de/d12/d5e/d6d/c81 0 2024-04-22T23:23:43.182 INFO:tasks.workunit.client.1.smithi134.stdout:6/374: rename d0/d19/de/d12/d5e/l69 to d0/d19/de/d12/d29/d74/l82 0 2024-04-22T23:23:43.184 INFO:tasks.workunit.client.1.smithi134.stdout:5/323: dwrite d9/f2b [12582912,4194304] 0 2024-04-22T23:23:43.186 INFO:tasks.workunit.client.1.smithi134.stdout:5/324: write d9/f4a [797428,88414] 0 2024-04-22T23:23:43.186 INFO:tasks.workunit.client.1.smithi134.stdout:0/313: mknod d1/d5/d58/d22/c63 0 2024-04-22T23:23:43.190 INFO:tasks.workunit.client.1.smithi134.stdout:0/314: read d1/d5/d58/f29 [344219,89324] 0 2024-04-22T23:23:43.190 INFO:tasks.workunit.client.1.smithi134.stdout:5/325: write d9/f2b [17625325,108865] 0 2024-04-22T23:23:43.195 INFO:tasks.workunit.client.1.smithi134.stdout:6/375: dwrite d0/d19/de/d12/d2a/d2d/f49 [0,4194304] 0 2024-04-22T23:23:43.198 INFO:tasks.workunit.client.1.smithi134.stdout:1/392: getdents db/d15 0 2024-04-22T23:23:43.202 INFO:tasks.workunit.client.1.smithi134.stdout:6/376: fsync d0/d19/f9 0 2024-04-22T23:23:43.205 INFO:tasks.workunit.client.1.smithi134.stdout:0/315: dwrite d1/d2/d1d/f5f [0,4194304] 0 2024-04-22T23:23:43.206 INFO:tasks.workunit.client.1.smithi134.stdout:1/393: mknod db/d15/d17/d20/d2a/d2d/d2e/c8d 0 2024-04-22T23:23:43.206 INFO:tasks.workunit.client.1.smithi134.stdout:0/316: fdatasync d1/d2/f54 0 2024-04-22T23:23:43.207 INFO:tasks.workunit.client.1.smithi134.stdout:6/377: mknod d0/d19/db/c83 0 2024-04-22T23:23:43.209 INFO:tasks.workunit.client.1.smithi134.stdout:6/378: fdatasync d0/d19/de/d12/d2a/d2d/f72 0 2024-04-22T23:23:43.209 INFO:tasks.workunit.client.1.smithi134.stdout:5/326: truncate d9/f15 1937207 0 2024-04-22T23:23:43.212 INFO:tasks.workunit.client.1.smithi134.stdout:0/317: unlink d1/f1c 0 2024-04-22T23:23:43.215 INFO:tasks.workunit.client.1.smithi134.stdout:5/327: read d9/d11/f1a [103255,122364] 0 2024-04-22T23:23:43.217 INFO:tasks.workunit.client.1.smithi134.stdout:6/379: truncate d0/d19/de/d1f/f22 4967753 0 2024-04-22T23:23:43.217 INFO:tasks.workunit.client.1.smithi134.stdout:1/394: symlink db/d16/l8e 0 2024-04-22T23:23:43.219 INFO:tasks.workunit.client.1.smithi134.stdout:6/380: chown d0/d19/db/d33/f50 1029248 1 2024-04-22T23:23:43.223 INFO:tasks.workunit.client.1.smithi134.stdout:0/318: dwrite d1/d5/dd/f5b [0,4194304] 0 2024-04-22T23:23:43.225 INFO:tasks.workunit.client.1.smithi134.stdout:6/381: unlink d0/d19/de/l5c 0 2024-04-22T23:23:43.228 INFO:tasks.workunit.client.1.smithi134.stdout:5/328: dwrite d9/f4a [0,4194304] 0 2024-04-22T23:23:43.228 INFO:tasks.workunit.client.1.smithi134.stdout:6/382: dread - d0/d19/db/d35/f79 zero size 2024-04-22T23:23:43.230 INFO:tasks.workunit.client.1.smithi134.stdout:6/383: readlink d0/l4 0 2024-04-22T23:23:43.232 INFO:tasks.workunit.client.1.smithi134.stdout:1/395: dwrite db/d15/f34 [0,4194304] 0 2024-04-22T23:23:43.242 INFO:tasks.workunit.client.1.smithi134.stdout:0/319: creat d1/d5/dd/d11/f64 x:0 0 0 2024-04-22T23:23:43.242 INFO:tasks.workunit.client.1.smithi134.stdout:1/396: symlink db/d16/d56/l8f 0 2024-04-22T23:23:43.243 INFO:tasks.workunit.client.1.smithi134.stdout:0/320: readlink d1/d5/dd/d41/l48 0 2024-04-22T23:23:43.244 INFO:tasks.workunit.client.1.smithi134.stdout:6/384: link d0/d19/db/d35/f79 d0/d6f/f84 0 2024-04-22T23:23:43.248 INFO:tasks.workunit.client.1.smithi134.stdout:1/397: creat db/d15/d17/d20/d2a/d2d/d2e/f90 x:0 0 0 2024-04-22T23:23:43.248 INFO:tasks.workunit.client.1.smithi134.stdout:6/385: creat d0/d19/de/d12/d2a/d2d/f85 x:0 0 0 2024-04-22T23:23:43.251 INFO:tasks.workunit.client.1.smithi134.stdout:0/321: dwrite d1/d5/dd/f16 [0,4194304] 0 2024-04-22T23:23:43.254 INFO:tasks.workunit.client.1.smithi134.stdout:0/322: mknod d1/d2/c65 0 2024-04-22T23:23:43.254 INFO:tasks.workunit.client.1.smithi134.stdout:5/329: truncate f6 11550320 0 2024-04-22T23:23:43.255 INFO:tasks.workunit.client.1.smithi134.stdout:6/386: dread d0/d19/de/d12/d29/f70 [0,4194304] 0 2024-04-22T23:23:43.255 INFO:tasks.workunit.client.1.smithi134.stdout:0/323: chown d1/f44 947896 1 2024-04-22T23:23:43.257 INFO:tasks.workunit.client.1.smithi134.stdout:5/330: rmdir d9/dd/d17/d44 39 2024-04-22T23:23:43.258 INFO:tasks.workunit.client.1.smithi134.stdout:5/331: stat d9/f67 0 2024-04-22T23:23:43.258 INFO:tasks.workunit.client.1.smithi134.stdout:6/387: chown d0/d19/de/d12/d2a/d2d/d3a/c57 627 1 2024-04-22T23:23:43.259 INFO:tasks.workunit.client.1.smithi134.stdout:5/332: chown d9/d11/d34/f6a 6629 1 2024-04-22T23:23:43.266 INFO:tasks.workunit.client.1.smithi134.stdout:6/388: getdents d0/d19/de/d1f 0 2024-04-22T23:23:43.277 INFO:tasks.workunit.client.1.smithi134.stdout:0/324: truncate d1/d5/d58/d22/f33 1224013 0 2024-04-22T23:23:43.278 INFO:tasks.workunit.client.1.smithi134.stdout:5/333: dwrite d9/dd/f10 [4194304,4194304] 0 2024-04-22T23:23:43.278 INFO:tasks.workunit.client.1.smithi134.stdout:6/389: write d0/d19/de/d12/f52 [286661,9983] 0 2024-04-22T23:23:43.280 INFO:tasks.workunit.client.1.smithi134.stdout:5/334: write d9/d1e/f33 [1051791,120698] 0 2024-04-22T23:23:43.280 INFO:tasks.workunit.client.1.smithi134.stdout:6/390: stat d0/d19/de/l56 0 2024-04-22T23:23:43.281 INFO:tasks.workunit.client.1.smithi134.stdout:5/335: chown d9/df/d20 1 1 2024-04-22T23:23:43.284 INFO:tasks.workunit.client.1.smithi134.stdout:6/391: mkdir d0/d19/db/d33/d86 0 2024-04-22T23:23:43.291 INFO:tasks.workunit.client.1.smithi134.stdout:0/325: unlink d1/d5/d58/d22/f33 0 2024-04-22T23:23:43.293 INFO:tasks.workunit.client.1.smithi134.stdout:0/326: chown d1/d5/f50 11475278 1 2024-04-22T23:23:43.293 INFO:tasks.workunit.client.1.smithi134.stdout:6/392: dwrite d0/d19/de/d12/d2a/d2d/d3a/f77 [0,4194304] 0 2024-04-22T23:23:43.295 INFO:tasks.workunit.client.1.smithi134.stdout:0/327: truncate d1/d5/f50 321156 0 2024-04-22T23:23:43.295 INFO:tasks.workunit.client.1.smithi134.stdout:5/336: link d9/dd/d17/l5e d9/dd/d17/l6c 0 2024-04-22T23:23:43.297 INFO:tasks.workunit.client.1.smithi134.stdout:6/393: mknod d0/d19/de/d12/d5e/c87 0 2024-04-22T23:23:43.298 INFO:tasks.workunit.client.1.smithi134.stdout:5/337: mknod d9/c6d 0 2024-04-22T23:23:43.299 INFO:tasks.workunit.client.1.smithi134.stdout:6/394: chown d0/d19/de/l56 484 1 2024-04-22T23:23:43.299 INFO:tasks.workunit.client.1.smithi134.stdout:5/338: chown d9/d11/d34/f6a 42 1 2024-04-22T23:23:43.301 INFO:tasks.workunit.client.1.smithi134.stdout:6/395: write d0/f42 [3039993,124491] 0 2024-04-22T23:23:43.303 INFO:tasks.workunit.client.1.smithi134.stdout:3/260: sync 2024-04-22T23:23:43.303 INFO:tasks.workunit.client.1.smithi134.stdout:2/255: sync 2024-04-22T23:23:43.303 INFO:tasks.workunit.client.1.smithi134.stdout:4/473: sync 2024-04-22T23:23:43.306 INFO:tasks.workunit.client.1.smithi134.stdout:5/339: link d9/d11/f1b d9/d1e/f6e 0 2024-04-22T23:23:43.306 INFO:tasks.workunit.client.1.smithi134.stdout:6/396: mkdir d0/d6f/d88 0 2024-04-22T23:23:43.307 INFO:tasks.workunit.client.1.smithi134.stdout:0/328: dwrite d1/d5/dd/f35 [0,4194304] 0 2024-04-22T23:23:43.308 INFO:tasks.workunit.client.1.smithi134.stdout:2/256: unlink d2/dd/df/f41 0 2024-04-22T23:23:43.311 INFO:tasks.workunit.client.1.smithi134.stdout:0/329: write d1/d5/dd/d20/f1e [3940459,67283] 0 2024-04-22T23:23:43.312 INFO:tasks.workunit.client.1.smithi134.stdout:0/330: write d1/d5/dd/f45 [78667,121157] 0 2024-04-22T23:23:43.313 INFO:tasks.workunit.client.1.smithi134.stdout:6/397: creat d0/d19/de/d12/d29/d74/f89 x:0 0 0 2024-04-22T23:23:43.317 INFO:tasks.workunit.client.1.smithi134.stdout:4/474: symlink d4/d35/d5e/l9c 0 2024-04-22T23:23:43.317 INFO:tasks.workunit.client.1.smithi134.stdout:2/257: write d2/d15/d24/d1d/f26 [3704530,1786] 0 2024-04-22T23:23:43.318 INFO:tasks.workunit.client.1.smithi134.stdout:6/398: mkdir d0/d19/db/d35/d8a 0 2024-04-22T23:23:43.321 INFO:tasks.workunit.client.1.smithi134.stdout:2/258: chown d2/d42/f46 521486 1 2024-04-22T23:23:43.321 INFO:tasks.workunit.client.1.smithi134.stdout:2/259: write d2/d42/d54/d33/f3a [541630,81809] 0 2024-04-22T23:23:43.322 INFO:tasks.workunit.client.1.smithi134.stdout:2/260: write d2/d42/d54/d33/f51 [88189,127157] 0 2024-04-22T23:23:43.323 INFO:tasks.workunit.client.1.smithi134.stdout:2/261: fsync d2/d15/d24/d1d/f36 0 2024-04-22T23:23:43.325 INFO:tasks.workunit.client.1.smithi134.stdout:0/331: rename d1/d5/dd/d20/c57 to d1/d5/dd/d20/c66 0 2024-04-22T23:23:43.328 INFO:tasks.workunit.client.1.smithi134.stdout:3/261: link dc/d2a/d30/c63 dc/dd/d28/d34/c64 0 2024-04-22T23:23:43.329 INFO:tasks.workunit.client.1.smithi134.stdout:0/332: read d1/d5/dd/f40 [3460104,58620] 0 2024-04-22T23:23:43.332 INFO:tasks.workunit.client.1.smithi134.stdout:2/262: dwrite d2/f2c [4194304,4194304] 0 2024-04-22T23:23:43.335 INFO:tasks.workunit.client.1.smithi134.stdout:6/399: mkdir d0/d6f/d88/d8b 0 2024-04-22T23:23:43.336 INFO:tasks.workunit.client.1.smithi134.stdout:6/400: write d0/d19/de/d1f/f2c [155766,127799] 0 2024-04-22T23:23:43.338 INFO:tasks.workunit.client.1.smithi134.stdout:6/401: write d0/d19/de/d12/d2a/d2d/d3a/f77 [2667326,75482] 0 2024-04-22T23:23:43.340 INFO:tasks.workunit.client.1.smithi134.stdout:5/340: truncate d9/dd/f10 2051187 0 2024-04-22T23:23:43.347 INFO:tasks.workunit.client.1.smithi134.stdout:2/263: mknod d2/d15/d24/c5b 0 2024-04-22T23:23:43.353 INFO:tasks.workunit.client.1.smithi134.stdout:3/262: creat dc/d2a/d62/f65 x:0 0 0 2024-04-22T23:23:43.355 INFO:tasks.workunit.client.1.smithi134.stdout:6/402: chown d0/d19/de/d12/d29/d74/l82 43771 1 2024-04-22T23:23:43.358 INFO:tasks.workunit.client.1.smithi134.stdout:4/475: getdents d4/d15/d82 0 2024-04-22T23:23:43.362 INFO:tasks.workunit.client.1.smithi134.stdout:6/403: dwrite d0/f60 [0,4194304] 0 2024-04-22T23:23:43.374 INFO:tasks.workunit.client.1.smithi134.stdout:0/333: truncate d1/d5/dd/f5b 3174155 0 2024-04-22T23:23:43.376 INFO:tasks.workunit.client.1.smithi134.stdout:4/476: symlink d4/d7/d32/d3a/l9d 0 2024-04-22T23:23:43.377 INFO:tasks.workunit.client.1.smithi134.stdout:2/264: write d2/dd/fe [5467432,102360] 0 2024-04-22T23:23:43.380 INFO:tasks.workunit.client.1.smithi134.stdout:0/334: symlink d1/d5/d15/l67 0 2024-04-22T23:23:43.382 INFO:tasks.workunit.client.1.smithi134.stdout:2/265: getdents d2/d15/d24/d1d/d3d 0 2024-04-22T23:23:43.382 INFO:tasks.workunit.client.1.smithi134.stdout:0/335: write d1/d5/dd/f16 [1256620,8735] 0 2024-04-22T23:23:43.383 INFO:tasks.workunit.client.1.smithi134.stdout:4/477: creat d4/d35/f9e x:0 0 0 2024-04-22T23:23:43.383 INFO:tasks.workunit.client.1.smithi134.stdout:0/336: stat d1/db/c49 0 2024-04-22T23:23:43.386 INFO:tasks.workunit.client.1.smithi134.stdout:2/266: write d2/d15/d24/d1d/f44 [756703,81859] 0 2024-04-22T23:23:43.386 INFO:tasks.workunit.client.1.smithi134.stdout:0/337: read d1/d5/dd/f31 [854328,106787] 0 2024-04-22T23:23:43.388 INFO:tasks.workunit.client.1.smithi134.stdout:0/338: write d1/d5/dd/f35 [555596,3681] 0 2024-04-22T23:23:43.389 INFO:tasks.workunit.client.1.smithi134.stdout:0/339: fsync d1/d2/d1d/f5f 0 2024-04-22T23:23:43.391 INFO:tasks.workunit.client.1.smithi134.stdout:5/341: dread d9/dd/f10 [0,4194304] 0 2024-04-22T23:23:43.393 INFO:tasks.workunit.client.1.smithi134.stdout:5/342: write f8 [3610039,18988] 0 2024-04-22T23:23:43.395 INFO:tasks.workunit.client.1.smithi134.stdout:5/343: stat d9/dd/d17/d2a/l43 0 2024-04-22T23:23:43.396 INFO:tasks.workunit.client.1.smithi134.stdout:2/267: dwrite d2/d15/f50 [8388608,4194304] 0 2024-04-22T23:23:43.397 INFO:tasks.workunit.client.1.smithi134.stdout:5/344: stat d9/df/d20/c4f 0 2024-04-22T23:23:43.408 INFO:tasks.workunit.client.1.smithi134.stdout:2/268: getdents d2 0 2024-04-22T23:23:43.409 INFO:tasks.workunit.client.1.smithi134.stdout:5/345: dwrite d9/f67 [0,4194304] 0 2024-04-22T23:23:43.412 INFO:tasks.workunit.client.1.smithi134.stdout:2/269: mknod d2/d15/d24/d1d/c5c 0 2024-04-22T23:23:43.448 INFO:tasks.workunit.client.1.smithi134.stdout:5/346: creat d9/df/f6f x:0 0 0 2024-04-22T23:23:43.449 INFO:tasks.workunit.client.1.smithi134.stdout:0/340: dwrite d1/d2/f4 [0,4194304] 0 2024-04-22T23:23:43.449 INFO:tasks.workunit.client.1.smithi134.stdout:4/478: dread d4/f2b [0,4194304] 0 2024-04-22T23:23:43.453 INFO:tasks.workunit.client.1.smithi134.stdout:0/341: mkdir d1/d5/dd/d68 0 2024-04-22T23:23:43.454 INFO:tasks.workunit.client.1.smithi134.stdout:4/479: rename d4/d35/d5e/l1a to d4/d35/d53/l9f 0 2024-04-22T23:23:43.507 INFO:tasks.workunit.client.1.smithi134.stdout:0/342: creat d1/d2/d1d/f69 x:0 0 0 2024-04-22T23:23:43.508 INFO:tasks.workunit.client.1.smithi134.stdout:2/270: getdents d2/dd 0 2024-04-22T23:23:43.510 INFO:tasks.workunit.client.1.smithi134.stdout:4/480: rename d4/d35/f9e to d4/d7/d32/d3a/fa0 0 2024-04-22T23:23:43.511 INFO:tasks.workunit.client.1.smithi134.stdout:0/343: mknod d1/d5/d58/c6a 0 2024-04-22T23:23:43.514 INFO:tasks.workunit.client.1.smithi134.stdout:2/271: rename d2/dd/fe to d2/d15/d24/f5d 0 2024-04-22T23:23:43.517 INFO:tasks.workunit.client.1.smithi134.stdout:5/347: dread d9/d11/d34/f40 [0,4194304] 0 2024-04-22T23:23:43.518 INFO:tasks.workunit.client.1.smithi134.stdout:4/481: mknod d4/d6b/ca1 0 2024-04-22T23:23:43.519 INFO:tasks.workunit.client.1.smithi134.stdout:0/344: dwrite d1/d2/f4 [0,4194304] 0 2024-04-22T23:23:43.523 INFO:tasks.workunit.client.1.smithi134.stdout:5/348: creat d9/dd/d52/f70 x:0 0 0 2024-04-22T23:23:43.527 INFO:tasks.workunit.client.1.smithi134.stdout:4/482: rename d4/d35/d5e/l9c to d4/d35/d8e/d5d/la2 0 2024-04-22T23:23:43.528 INFO:tasks.workunit.client.1.smithi134.stdout:0/345: dwrite d1/d5/d58/f2e [0,4194304] 0 2024-04-22T23:23:43.530 INFO:tasks.workunit.client.1.smithi134.stdout:0/346: truncate d1/f44 1489639 0 2024-04-22T23:23:43.532 INFO:tasks.workunit.client.1.smithi134.stdout:4/483: chown d4/d13/d68 7442 1 2024-04-22T23:23:43.533 INFO:tasks.workunit.client.1.smithi134.stdout:2/272: getdents d2/d15/d24 0 2024-04-22T23:23:43.534 INFO:tasks.workunit.client.1.smithi134.stdout:0/347: mkdir d1/d5/d58/d6b 0 2024-04-22T23:23:43.536 INFO:tasks.workunit.client.1.smithi134.stdout:5/349: creat d9/dd/d17/f71 x:0 0 0 2024-04-22T23:23:43.539 INFO:tasks.workunit.client.1.smithi134.stdout:5/350: symlink d9/d1e/l72 0 2024-04-22T23:23:43.542 INFO:tasks.workunit.client.1.smithi134.stdout:5/351: mkdir d9/d11/d73 0 2024-04-22T23:23:43.543 INFO:tasks.workunit.client.1.smithi134.stdout:4/484: link d4/f6 d4/d35/d8e/fa3 0 2024-04-22T23:23:43.544 INFO:tasks.workunit.client.1.smithi134.stdout:4/485: stat d4/d35/d80 0 2024-04-22T23:23:43.544 INFO:tasks.workunit.client.1.smithi134.stdout:5/352: write d9/d11/d34/d3b/f5f [954726,120828] 0 2024-04-22T23:23:43.551 INFO:tasks.workunit.client.1.smithi134.stdout:4/486: dread d4/d13/f65 [0,4194304] 0 2024-04-22T23:23:43.553 INFO:tasks.workunit.client.1.smithi134.stdout:4/487: write d4/d7/d32/d3a/f43 [2917203,80586] 0 2024-04-22T23:23:43.556 INFO:tasks.workunit.client.1.smithi134.stdout:4/488: creat d4/d35/d53/fa4 x:0 0 0 2024-04-22T23:23:43.557 INFO:tasks.workunit.client.1.smithi134.stdout:2/273: dread d2/d15/d24/d1d/f28 [0,4194304] 0 2024-04-22T23:23:43.560 INFO:tasks.workunit.client.1.smithi134.stdout:4/489: fsync d4/f6 0 2024-04-22T23:23:43.561 INFO:tasks.workunit.client.1.smithi134.stdout:4/490: write d4/d35/d5e/d25/f86 [222629,54292] 0 2024-04-22T23:23:43.566 INFO:tasks.workunit.client.1.smithi134.stdout:5/353: dwrite d9/fa [0,4194304] 0 2024-04-22T23:23:43.568 INFO:tasks.workunit.client.1.smithi134.stdout:5/354: read d9/dd/f4c [88505,47352] 0 2024-04-22T23:23:43.570 INFO:tasks.workunit.client.1.smithi134.stdout:4/491: link d4/d7/d32/d3a/f51 d4/fa5 0 2024-04-22T23:23:43.571 INFO:tasks.workunit.client.1.smithi134.stdout:5/355: mknod d9/dd/c74 0 2024-04-22T23:23:43.572 INFO:tasks.workunit.client.1.smithi134.stdout:5/356: chown l7 756 1 2024-04-22T23:23:43.573 INFO:tasks.workunit.client.1.smithi134.stdout:4/492: dread d4/d35/d5e/d25/f86 [0,4194304] 0 2024-04-22T23:23:43.574 INFO:tasks.workunit.client.1.smithi134.stdout:5/357: stat d9/dd/d17/d44/l3d 0 2024-04-22T23:23:43.575 INFO:tasks.workunit.client.1.smithi134.stdout:4/493: write d4/f2b [992051,29402] 0 2024-04-22T23:23:43.576 INFO:tasks.workunit.client.1.smithi134.stdout:4/494: chown d4/d29 117496 1 2024-04-22T23:23:43.576 INFO:tasks.workunit.client.1.smithi134.stdout:5/358: symlink d9/d11/d34/d3b/l75 0 2024-04-22T23:23:43.577 INFO:tasks.workunit.client.1.smithi134.stdout:4/495: write d4/f5c [755591,97046] 0 2024-04-22T23:23:43.582 INFO:tasks.workunit.client.1.smithi134.stdout:4/496: truncate d4/dd/f41 1436008 0 2024-04-22T23:23:43.583 INFO:tasks.workunit.client.1.smithi134.stdout:4/497: write d4/dd/f31 [785804,69455] 0 2024-04-22T23:23:43.584 INFO:tasks.workunit.client.1.smithi134.stdout:4/498: chown d4/d35/d5e/d25/f86 534565992 1 2024-04-22T23:23:43.588 INFO:tasks.workunit.client.1.smithi134.stdout:4/499: rename d4/f5c to d4/d15/fa6 0 2024-04-22T23:23:43.590 INFO:tasks.workunit.client.1.smithi134.stdout:4/500: mknod d4/d7/d32/ca7 0 2024-04-22T23:23:43.590 INFO:tasks.workunit.client.1.smithi134.stdout:4/501: readlink d4/d15/l36 0 2024-04-22T23:23:43.591 INFO:tasks.workunit.client.1.smithi134.stdout:4/502: write d4/d29/f77 [638303,111102] 0 2024-04-22T23:23:43.592 INFO:tasks.workunit.client.1.smithi134.stdout:4/503: dread - d4/d15/f6f zero size 2024-04-22T23:23:43.592 INFO:tasks.workunit.client.1.smithi134.stdout:4/504: dread - d4/d15/f81 zero size 2024-04-22T23:23:43.593 INFO:tasks.workunit.client.1.smithi134.stdout:5/359: dread d9/df/d20/f39 [0,4194304] 0 2024-04-22T23:23:43.594 INFO:tasks.workunit.client.1.smithi134.stdout:4/505: write d4/d29/f63 [614377,81999] 0 2024-04-22T23:23:43.596 INFO:tasks.workunit.client.1.smithi134.stdout:4/506: dread d4/d15/f6a [0,4194304] 0 2024-04-22T23:23:43.599 INFO:tasks.workunit.client.1.smithi134.stdout:4/507: mkdir d4/dd/da8 0 2024-04-22T23:23:43.600 INFO:tasks.workunit.client.1.smithi134.stdout:4/508: chown d4/c24 239633 1 2024-04-22T23:23:43.605 INFO:tasks.workunit.client.1.smithi134.stdout:5/360: dwrite d9/d11/f1a [0,4194304] 0 2024-04-22T23:23:43.609 INFO:tasks.workunit.client.1.smithi134.stdout:5/361: rename d9/dd/d17/d2a/l43 to d9/df/d20/d60/l76 0 2024-04-22T23:23:43.614 INFO:tasks.workunit.client.1.smithi134.stdout:5/362: dwrite d9/df/f6f [0,4194304] 0 2024-04-22T23:23:43.618 INFO:tasks.workunit.client.1.smithi134.stdout:5/363: fdatasync d9/f15 0 2024-04-22T23:23:43.621 INFO:tasks.workunit.client.1.smithi134.stdout:5/364: rename d9/dd/d52/f70 to d9/dd/f77 0 2024-04-22T23:23:43.622 INFO:tasks.workunit.client.1.smithi134.stdout:5/365: fsync d9/d11/d34/f40 0 2024-04-22T23:23:43.626 INFO:tasks.workunit.client.1.smithi134.stdout:5/366: symlink d9/d11/d73/l78 0 2024-04-22T23:23:43.627 INFO:tasks.workunit.client.1.smithi134.stdout:5/367: dread - d9/df/d20/f48 zero size 2024-04-22T23:23:43.628 INFO:tasks.workunit.client.1.smithi134.stdout:5/368: readlink d9/dd/d17/l6c 0 2024-04-22T23:23:43.634 INFO:tasks.workunit.client.1.smithi134.stdout:5/369: truncate d9/fc 1326458 0 2024-04-22T23:23:43.636 INFO:tasks.workunit.client.1.smithi134.stdout:5/370: creat d9/dd/d52/f79 x:0 0 0 2024-04-22T23:23:43.636 INFO:tasks.workunit.client.1.smithi134.stdout:4/509: dread d4/f21 [0,4194304] 0 2024-04-22T23:23:43.643 INFO:tasks.workunit.client.1.smithi134.stdout:5/371: dwrite d9/f67 [4194304,4194304] 0 2024-04-22T23:23:43.645 INFO:tasks.workunit.client.1.smithi134.stdout:5/372: write d9/dd/f10 [2108653,115725] 0 2024-04-22T23:23:43.650 INFO:tasks.workunit.client.1.smithi134.stdout:4/510: dwrite d4/d15/f26 [0,4194304] 0 2024-04-22T23:23:43.651 INFO:tasks.workunit.client.1.smithi134.stdout:5/373: link d9/fe d9/d11/f7a 0 2024-04-22T23:23:43.652 INFO:tasks.workunit.client.1.smithi134.stdout:5/374: creat d9/dd/f7b x:0 0 0 2024-04-22T23:23:43.656 INFO:tasks.workunit.client.1.smithi134.stdout:7/410: sync 2024-04-22T23:23:43.661 INFO:tasks.workunit.client.1.smithi134.stdout:5/375: dread d9/fe [0,4194304] 0 2024-04-22T23:23:43.664 INFO:tasks.workunit.client.1.smithi134.stdout:7/411: unlink d4/f7 0 2024-04-22T23:23:43.667 INFO:tasks.workunit.client.1.smithi134.stdout:5/376: symlink d9/df/l7c 0 2024-04-22T23:23:43.668 INFO:tasks.workunit.client.1.smithi134.stdout:7/412: mkdir d4/d10/d2f/d37/d95 0 2024-04-22T23:23:43.670 INFO:tasks.workunit.client.1.smithi134.stdout:5/377: creat d9/df/d20/d60/f7d x:0 0 0 2024-04-22T23:23:43.670 INFO:tasks.workunit.client.1.smithi134.stdout:7/413: readlink d4/d8/d58/l91 0 2024-04-22T23:23:43.671 INFO:tasks.workunit.client.1.smithi134.stdout:4/511: dwrite d4/f6 [0,4194304] 0 2024-04-22T23:23:43.673 INFO:tasks.workunit.client.1.smithi134.stdout:4/512: read d4/d13/f4c [2714067,49419] 0 2024-04-22T23:23:43.674 INFO:tasks.workunit.client.1.smithi134.stdout:4/513: chown d4/l27 22953107 1 2024-04-22T23:23:43.675 INFO:tasks.workunit.client.1.smithi134.stdout:5/378: creat d9/dd/d17/d44/f7e x:0 0 0 2024-04-22T23:23:43.675 INFO:tasks.workunit.client.1.smithi134.stdout:5/379: stat d9/dd/d17/l6c 0 2024-04-22T23:23:43.676 INFO:tasks.workunit.client.1.smithi134.stdout:5/380: write d9/dd/d17/d44/f59 [733758,1937] 0 2024-04-22T23:23:43.678 INFO:tasks.workunit.client.1.smithi134.stdout:5/381: chown d9/dd/d17/d44/l4b 1176872145 1 2024-04-22T23:23:43.679 INFO:tasks.workunit.client.1.smithi134.stdout:7/414: mknod d4/d10/d2f/d37/d95/c96 0 2024-04-22T23:23:43.679 INFO:tasks.workunit.client.1.smithi134.stdout:5/382: truncate d9/d11/d34/f6a 56361 0 2024-04-22T23:23:43.680 INFO:tasks.workunit.client.1.smithi134.stdout:5/383: chown d9/d1e/f32 0 1 2024-04-22T23:23:43.681 INFO:tasks.workunit.client.1.smithi134.stdout:5/384: write d9/d1e/f33 [3969107,97982] 0 2024-04-22T23:23:43.682 INFO:tasks.workunit.client.1.smithi134.stdout:4/514: fdatasync d4/fa5 0 2024-04-22T23:23:43.685 INFO:tasks.workunit.client.1.smithi134.stdout:5/385: mknod d9/d1e/c7f 0 2024-04-22T23:23:43.687 INFO:tasks.workunit.client.1.smithi134.stdout:4/515: creat d4/d35/d8e/d5d/fa9 x:0 0 0 2024-04-22T23:23:43.687 INFO:tasks.workunit.client.1.smithi134.stdout:4/516: fdatasync d4/d15/f92 0 2024-04-22T23:23:43.688 INFO:tasks.workunit.client.1.smithi134.stdout:5/386: creat d9/d11/d73/f80 x:0 0 0 2024-04-22T23:23:43.689 INFO:tasks.workunit.client.1.smithi134.stdout:5/387: dread - d9/d11/f5b zero size 2024-04-22T23:23:43.691 INFO:tasks.workunit.client.1.smithi134.stdout:4/517: unlink d4/d35/f57 0 2024-04-22T23:23:43.692 INFO:tasks.workunit.client.1.smithi134.stdout:4/518: stat d4/d7/d32/d3a/f43 0 2024-04-22T23:23:43.696 INFO:tasks.workunit.client.1.smithi134.stdout:5/388: mknod d9/d1e/c81 0 2024-04-22T23:23:43.700 INFO:tasks.workunit.client.1.smithi134.stdout:7/415: write d4/d10/d2f/d34/d62/d29/f55 [2632155,116866] 0 2024-04-22T23:23:43.702 INFO:tasks.workunit.client.1.smithi134.stdout:5/389: write d9/fc [571249,114004] 0 2024-04-22T23:23:43.703 INFO:tasks.workunit.client.1.smithi134.stdout:7/416: mkdir d4/d10/d2f/d70/d86/d97 0 2024-04-22T23:23:43.705 INFO:tasks.workunit.client.1.smithi134.stdout:5/390: symlink d9/d11/d34/l82 0 2024-04-22T23:23:43.705 INFO:tasks.workunit.client.1.smithi134.stdout:4/519: dread d4/d13/f4c [0,4194304] 0 2024-04-22T23:23:43.706 INFO:tasks.workunit.client.1.smithi134.stdout:5/391: readlink d9/df/d20/l57 0 2024-04-22T23:23:43.706 INFO:tasks.workunit.client.1.smithi134.stdout:4/520: dread - d4/d35/d8e/f75 zero size 2024-04-22T23:23:43.708 INFO:tasks.workunit.client.1.smithi134.stdout:5/392: dread d9/d11/d34/f40 [0,4194304] 0 2024-04-22T23:23:43.709 INFO:tasks.workunit.client.1.smithi134.stdout:4/521: creat d4/d15/d82/d9a/faa x:0 0 0 2024-04-22T23:23:43.710 INFO:tasks.workunit.client.1.smithi134.stdout:7/417: dread d4/d10/f6b [0,4194304] 0 2024-04-22T23:23:43.710 INFO:tasks.workunit.client.1.smithi134.stdout:5/393: creat d9/df/d20/d60/f83 x:0 0 0 2024-04-22T23:23:43.712 INFO:tasks.workunit.client.1.smithi134.stdout:4/522: mknod d4/d7/d32/cab 0 2024-04-22T23:23:43.714 INFO:tasks.workunit.client.1.smithi134.stdout:5/394: link f8 d9/d1e/f84 0 2024-04-22T23:23:43.716 INFO:tasks.workunit.client.1.smithi134.stdout:7/418: rename d4/d8/d2a/l4c to d4/d8/d2a/d42/l98 0 2024-04-22T23:23:43.716 INFO:tasks.workunit.client.1.smithi134.stdout:4/523: symlink d4/d7/d32/lac 0 2024-04-22T23:23:43.717 INFO:tasks.workunit.client.1.smithi134.stdout:5/395: dread d9/dd/f10 [0,4194304] 0 2024-04-22T23:23:43.718 INFO:tasks.workunit.client.1.smithi134.stdout:5/396: read d9/fe [232783,61237] 0 2024-04-22T23:23:43.720 INFO:tasks.workunit.client.1.smithi134.stdout:7/419: creat d4/d10/d2f/d69/d8a/f99 x:0 0 0 2024-04-22T23:23:43.723 INFO:tasks.workunit.client.1.smithi134.stdout:5/397: mkdir d9/d11/d34/d85 0 2024-04-22T23:23:43.724 INFO:tasks.workunit.client.1.smithi134.stdout:7/420: creat d4/d10/d2f/d69/f9a x:0 0 0 2024-04-22T23:23:43.724 INFO:tasks.workunit.client.1.smithi134.stdout:4/524: dwrite d4/d35/d5e/f4e [0,4194304] 0 2024-04-22T23:23:43.726 INFO:tasks.workunit.client.1.smithi134.stdout:5/398: rmdir d9/dd/d52 39 2024-04-22T23:23:43.732 INFO:tasks.workunit.client.1.smithi134.stdout:4/525: dwrite d4/f1d [0,4194304] 0 2024-04-22T23:23:43.733 INFO:tasks.workunit.client.1.smithi134.stdout:7/421: mknod d4/d4a/d85/c9b 0 2024-04-22T23:23:43.734 INFO:tasks.workunit.client.1.smithi134.stdout:4/526: chown d4/d15/d82/d9a/faa 5 1 2024-04-22T23:23:43.736 INFO:tasks.workunit.client.1.smithi134.stdout:7/422: mkdir d4/d10/d2f/d34/d54/d9c 0 2024-04-22T23:23:43.739 INFO:tasks.workunit.client.1.smithi134.stdout:7/423: creat d4/d10/d2f/d69/d8a/f9d x:0 0 0 2024-04-22T23:23:43.739 INFO:tasks.workunit.client.1.smithi134.stdout:4/527: link d4/l27 d4/d7/d32/d84/lad 0 2024-04-22T23:23:43.740 INFO:tasks.workunit.client.1.smithi134.stdout:7/424: dread - d4/d4a/f7e zero size 2024-04-22T23:23:43.741 INFO:tasks.workunit.client.1.smithi134.stdout:4/528: rmdir d4/d13 39 2024-04-22T23:23:43.741 INFO:tasks.workunit.client.1.smithi134.stdout:7/425: chown d4/d10/d2f/d34/c7f 6466254 1 2024-04-22T23:23:43.742 INFO:tasks.workunit.client.1.smithi134.stdout:4/529: rmdir d4/d15/d82 39 2024-04-22T23:23:43.745 INFO:tasks.workunit.client.1.smithi134.stdout:7/426: read d4/d8/d3b/f48 [2742418,46705] 0 2024-04-22T23:23:43.747 INFO:tasks.workunit.client.1.smithi134.stdout:4/530: rename d4/d35/d5e/f4e to d4/d7/d32/d3a/d93/fae 0 2024-04-22T23:23:43.750 INFO:tasks.workunit.client.1.smithi134.stdout:4/531: symlink d4/d15/d64/laf 0 2024-04-22T23:23:43.752 INFO:tasks.workunit.client.1.smithi134.stdout:4/532: truncate d4/d35/d5e/d25/f87 237647 0 2024-04-22T23:23:43.754 INFO:tasks.workunit.client.1.smithi134.stdout:4/533: mknod d4/d35/d8e/d5d/cb0 0 2024-04-22T23:23:43.754 INFO:tasks.workunit.client.1.smithi134.stdout:7/427: dread d4/d8/d41/f43 [0,4194304] 0 2024-04-22T23:23:43.756 INFO:tasks.workunit.client.1.smithi134.stdout:5/399: dread d9/d1e/f84 [4194304,4194304] 0 2024-04-22T23:23:43.762 INFO:tasks.workunit.client.1.smithi134.stdout:7/428: dwrite d4/f7b [0,4194304] 0 2024-04-22T23:23:43.765 INFO:tasks.workunit.client.1.smithi134.stdout:5/400: creat d9/dd/f86 x:0 0 0 2024-04-22T23:23:43.766 INFO:tasks.workunit.client.1.smithi134.stdout:7/429: fdatasync d4/d10/f1f 0 2024-04-22T23:23:43.767 INFO:tasks.workunit.client.1.smithi134.stdout:4/534: link d4/d35/l99 d4/d13/lb1 0 2024-04-22T23:23:43.767 INFO:tasks.workunit.client.1.smithi134.stdout:7/430: dread - d4/d8/d2a/d42/d75/f77 zero size 2024-04-22T23:23:43.768 INFO:tasks.workunit.client.1.smithi134.stdout:5/401: mknod d9/c87 0 2024-04-22T23:23:43.768 INFO:tasks.workunit.client.1.smithi134.stdout:7/431: chown d4/d10/d2f/l66 0 1 2024-04-22T23:23:43.771 INFO:tasks.workunit.client.1.smithi134.stdout:7/432: dread d4/d8/d41/d5b/f64 [0,4194304] 0 2024-04-22T23:23:43.776 INFO:tasks.workunit.client.1.smithi134.stdout:5/402: mkdir d9/dd/d88 0 2024-04-22T23:23:43.777 INFO:tasks.workunit.client.1.smithi134.stdout:5/403: write d9/df/d20/f48 [955664,68353] 0 2024-04-22T23:23:43.780 INFO:tasks.workunit.client.1.smithi134.stdout:5/404: mknod d9/c89 0 2024-04-22T23:23:43.782 INFO:tasks.workunit.client.1.smithi134.stdout:5/405: write d9/d11/d34/f40 [681715,79177] 0 2024-04-22T23:23:43.785 INFO:tasks.workunit.client.1.smithi134.stdout:5/406: rename d9/df/d20/d60/f83 to d9/dd/d17/d2a/f8a 0 2024-04-22T23:23:43.793 INFO:tasks.workunit.client.1.smithi134.stdout:4/535: truncate d4/d35/d5e/d25/f4b 2629974 0 2024-04-22T23:23:43.797 INFO:tasks.workunit.client.1.smithi134.stdout:8/280: sync 2024-04-22T23:23:43.800 INFO:tasks.workunit.client.1.smithi134.stdout:7/433: dwrite d4/d8/d2a/f47 [0,4194304] 0 2024-04-22T23:23:43.802 INFO:tasks.workunit.client.1.smithi134.stdout:4/536: dwrite d4/d15/f83 [0,4194304] 0 2024-04-22T23:23:43.804 INFO:tasks.workunit.client.1.smithi134.stdout:5/407: truncate d9/d1e/f33 3896012 0 2024-04-22T23:23:43.808 INFO:tasks.workunit.client.1.smithi134.stdout:8/281: creat d7/dc/d2c/d4f/f57 x:0 0 0 2024-04-22T23:23:43.810 INFO:tasks.workunit.client.1.smithi134.stdout:7/434: unlink d4/d8/f7a 0 2024-04-22T23:23:43.820 INFO:tasks.workunit.client.1.smithi134.stdout:5/408: rename f6 to d9/dd/d17/d2a/d49/f8b 0 2024-04-22T23:23:43.821 INFO:tasks.workunit.client.1.smithi134.stdout:8/282: dread d7/dc/f12 [0,4194304] 0 2024-04-22T23:23:43.828 INFO:tasks.workunit.client.1.smithi134.stdout:5/409: creat d9/d11/d73/f8c x:0 0 0 2024-04-22T23:23:43.828 INFO:tasks.workunit.client.1.smithi134.stdout:8/283: rmdir d7/db/d2d/d33 39 2024-04-22T23:23:43.829 INFO:tasks.workunit.client.1.smithi134.stdout:5/410: write d9/df/d20/f48 [598026,16181] 0 2024-04-22T23:23:43.831 INFO:tasks.workunit.client.1.smithi134.stdout:7/435: getdents d4/d10/d2f/d70 0 2024-04-22T23:23:43.832 INFO:tasks.workunit.client.1.smithi134.stdout:5/411: fdatasync d9/d1e/f5a 0 2024-04-22T23:23:43.834 INFO:tasks.workunit.client.1.smithi134.stdout:7/436: mknod d4/d8/d41/c9e 0 2024-04-22T23:23:43.836 INFO:tasks.workunit.client.1.smithi134.stdout:7/437: dread d4/d10/d2f/d37/f80 [0,4194304] 0 2024-04-22T23:23:43.837 INFO:tasks.workunit.client.1.smithi134.stdout:4/537: dread d4/d15/f67 [0,4194304] 0 2024-04-22T23:23:43.839 INFO:tasks.workunit.client.1.smithi134.stdout:5/412: mknod d9/dd/d88/c8d 0 2024-04-22T23:23:43.841 INFO:tasks.workunit.client.1.smithi134.stdout:8/284: getdents d7/db/d2d/d33 0 2024-04-22T23:23:43.845 INFO:tasks.workunit.client.1.smithi134.stdout:4/538: write d4/d7/d32/d3a/d93/fae [4453217,91414] 0 2024-04-22T23:23:43.847 INFO:tasks.workunit.client.1.smithi134.stdout:5/413: creat d9/dd/d88/f8e x:0 0 0 2024-04-22T23:23:43.848 INFO:tasks.workunit.client.1.smithi134.stdout:8/285: mkdir d7/db/d2d/d39/d58 0 2024-04-22T23:23:43.850 INFO:tasks.workunit.client.1.smithi134.stdout:8/286: dread d7/db/d2d/d39/f3d [0,4194304] 0 2024-04-22T23:23:43.852 INFO:tasks.workunit.client.1.smithi134.stdout:5/414: dread d9/df/d20/f48 [0,4194304] 0 2024-04-22T23:23:43.856 INFO:tasks.workunit.client.1.smithi134.stdout:5/415: rename d9/d1e/l3e to d9/dd/d17/l8f 0 2024-04-22T23:23:43.857 INFO:tasks.workunit.client.1.smithi134.stdout:5/416: chown d9/d11/d34/d4d/c5c 4395146 1 2024-04-22T23:23:43.858 INFO:tasks.workunit.client.1.smithi134.stdout:5/417: truncate d9/d11/f7a 1655740 0 2024-04-22T23:23:43.859 INFO:tasks.workunit.client.1.smithi134.stdout:5/418: mkdir d9/d1e/d90 0 2024-04-22T23:23:43.861 INFO:tasks.workunit.client.1.smithi134.stdout:5/419: creat d9/df/d20/f91 x:0 0 0 2024-04-22T23:23:43.863 INFO:tasks.workunit.client.1.smithi134.stdout:5/420: mkdir d9/d11/d34/d4d/d92 0 2024-04-22T23:23:43.866 INFO:tasks.workunit.client.1.smithi134.stdout:5/421: link l7 d9/d11/d34/d4d/l93 0 2024-04-22T23:23:43.867 INFO:tasks.workunit.client.1.smithi134.stdout:5/422: mknod d9/dd/d17/d44/c94 0 2024-04-22T23:23:43.872 INFO:tasks.workunit.client.1.smithi134.stdout:5/423: dwrite d9/dd/f10 [0,4194304] 0 2024-04-22T23:23:43.874 INFO:tasks.workunit.client.1.smithi134.stdout:5/424: write d9/df/d20/d60/f7d [508522,28904] 0 2024-04-22T23:23:43.890 INFO:tasks.workunit.client.1.smithi134.stdout:5/425: rename d9/df to d9/d11/d73/d95 0 2024-04-22T23:23:43.895 INFO:tasks.workunit.client.1.smithi134.stdout:5/426: write d9/dd/d17/d2a/d49/f8b [5382647,15619] 0 2024-04-22T23:23:43.898 INFO:tasks.workunit.client.1.smithi134.stdout:5/427: symlink d9/dd/l96 0 2024-04-22T23:23:43.898 INFO:tasks.workunit.client.1.smithi134.stdout:5/428: chown d9/d11/d34/d3b/l65 237 1 2024-04-22T23:23:43.901 INFO:tasks.workunit.client.1.smithi134.stdout:5/429: rename d9/d11/d73/l78 to d9/dd/l97 0 2024-04-22T23:23:43.902 INFO:tasks.workunit.client.1.smithi134.stdout:5/430: fdatasync d9/f4a 0 2024-04-22T23:23:43.902 INFO:tasks.workunit.client.1.smithi134.stdout:5/431: stat d9/d1e/f35 0 2024-04-22T23:23:43.906 INFO:tasks.workunit.client.1.smithi134.stdout:5/432: write d9/fe [1004033,79616] 0 2024-04-22T23:23:43.907 INFO:tasks.workunit.client.1.smithi134.stdout:5/433: truncate d9/f62 25316 0 2024-04-22T23:23:43.908 INFO:tasks.workunit.client.1.smithi134.stdout:5/434: creat d9/dd/d17/d2a/f98 x:0 0 0 2024-04-22T23:23:43.911 INFO:tasks.workunit.client.1.smithi134.stdout:5/435: rmdir d9/dd 39 2024-04-22T23:23:43.913 INFO:tasks.workunit.client.1.smithi134.stdout:5/436: fsync d9/dd/d17/d2a/d49/f8b 0 2024-04-22T23:23:43.918 INFO:tasks.workunit.client.1.smithi134.stdout:5/437: dwrite d9/d11/d34/f40 [0,4194304] 0 2024-04-22T23:23:43.921 INFO:tasks.workunit.client.1.smithi134.stdout:5/438: mknod d9/dd/c99 0 2024-04-22T23:23:43.945 INFO:tasks.workunit.client.1.smithi134.stdout:5/439: dread d9/dd/d17/d2a/d49/f8b [0,4194304] 0 2024-04-22T23:23:43.953 INFO:tasks.workunit.client.1.smithi134.stdout:5/440: dwrite d9/d1e/f84 [4194304,4194304] 0 2024-04-22T23:23:43.958 INFO:tasks.workunit.client.1.smithi134.stdout:5/441: dread d9/d11/d73/d95/d20/d60/f7d [0,4194304] 0 2024-04-22T23:23:43.960 INFO:tasks.workunit.client.1.smithi134.stdout:5/442: rename d9/d11/d34/c61 to d9/dd/d88/c9a 0 2024-04-22T23:23:43.961 INFO:tasks.workunit.client.1.smithi134.stdout:5/443: mknod d9/dd/d88/c9b 0 2024-04-22T23:23:43.963 INFO:tasks.workunit.client.1.smithi134.stdout:5/444: symlink d9/d11/l9c 0 2024-04-22T23:23:43.971 INFO:tasks.workunit.client.1.smithi134.stdout:5/445: truncate d9/d1e/f23 461366 0 2024-04-22T23:23:43.973 INFO:tasks.workunit.client.1.smithi134.stdout:5/446: creat d9/f9d x:0 0 0 2024-04-22T23:23:43.975 INFO:tasks.workunit.client.1.smithi134.stdout:5/447: rename d9/f16 to d9/d11/d34/d85/f9e 0 2024-04-22T23:23:43.976 INFO:tasks.workunit.client.1.smithi134.stdout:5/448: creat d9/d11/d34/d4d/f9f x:0 0 0 2024-04-22T23:23:43.977 INFO:tasks.workunit.client.1.smithi134.stdout:5/449: write d9/d11/d73/d95/f6f [2385004,5036] 0 2024-04-22T23:23:43.983 INFO:tasks.workunit.client.1.smithi134.stdout:5/450: link d9/d11/d34/d4d/c5c d9/d11/ca0 0 2024-04-22T23:23:43.989 INFO:tasks.workunit.client.1.smithi134.stdout:5/451: dwrite d9/dd/f7b [0,4194304] 0 2024-04-22T23:23:43.992 INFO:tasks.workunit.client.1.smithi134.stdout:5/452: creat d9/d1e/d90/fa1 x:0 0 0 2024-04-22T23:23:43.997 INFO:tasks.workunit.client.1.smithi134.stdout:1/398: sync 2024-04-22T23:23:43.998 INFO:tasks.workunit.client.1.smithi134.stdout:9/285: sync 2024-04-22T23:23:43.998 INFO:tasks.workunit.client.1.smithi134.stdout:5/453: truncate d9/fe 225467 0 2024-04-22T23:23:43.999 INFO:tasks.workunit.client.1.smithi134.stdout:5/454: dread - d9/dd/d88/f8e zero size 2024-04-22T23:23:44.000 INFO:tasks.workunit.client.1.smithi134.stdout:9/286: symlink d0/d2d/d3d/l63 0 2024-04-22T23:23:44.001 INFO:tasks.workunit.client.1.smithi134.stdout:9/287: dread - d0/d7/d27/f33 zero size 2024-04-22T23:23:44.001 INFO:tasks.workunit.client.1.smithi134.stdout:5/455: mkdir d9/da2 0 2024-04-22T23:23:44.002 INFO:tasks.workunit.client.1.smithi134.stdout:9/288: stat d0/d37/c52 0 2024-04-22T23:23:44.006 INFO:tasks.workunit.client.1.smithi134.stdout:1/399: dwrite db/d15/f7c [0,4194304] 0 2024-04-22T23:23:44.013 INFO:tasks.workunit.client.1.smithi134.stdout:1/400: unlink db/d15/d17/d20/f4b 0 2024-04-22T23:23:44.013 INFO:tasks.workunit.client.1.smithi134.stdout:5/456: dwrite d9/d1e/d90/fa1 [0,4194304] 0 2024-04-22T23:23:44.014 INFO:tasks.workunit.client.1.smithi134.stdout:5/457: chown d9/c87 794 1 2024-04-22T23:23:44.019 INFO:tasks.workunit.client.1.smithi134.stdout:1/401: creat db/d15/d17/d20/d2a/f91 x:0 0 0 2024-04-22T23:23:44.020 INFO:tasks.workunit.client.1.smithi134.stdout:5/458: creat d9/d11/d34/fa3 x:0 0 0 2024-04-22T23:23:44.023 INFO:tasks.workunit.client.1.smithi134.stdout:5/459: rmdir d9/d1e/d90 39 2024-04-22T23:23:44.025 INFO:tasks.workunit.client.1.smithi134.stdout:9/289: truncate d0/d7/d12/f1f 524598 0 2024-04-22T23:23:44.026 INFO:tasks.workunit.client.1.smithi134.stdout:9/290: write d0/d7/f5b [95015,21490] 0 2024-04-22T23:23:44.026 INFO:tasks.workunit.client.1.smithi134.stdout:5/460: symlink d9/d11/d73/d95/d20/d60/la4 0 2024-04-22T23:23:44.029 INFO:tasks.workunit.client.1.smithi134.stdout:5/461: creat d9/d11/d34/d85/fa5 x:0 0 0 2024-04-22T23:23:44.033 INFO:tasks.workunit.client.1.smithi134.stdout:9/291: link d0/d7/d23/f2a d0/d7/d23/d1b/f64 0 2024-04-22T23:23:44.034 INFO:tasks.workunit.client.1.smithi134.stdout:5/462: mkdir d9/dd/da6 0 2024-04-22T23:23:44.035 INFO:tasks.workunit.client.1.smithi134.stdout:5/463: readlink l7 0 2024-04-22T23:23:44.036 INFO:tasks.workunit.client.1.smithi134.stdout:9/292: mknod d0/d7/d23/d47/c65 0 2024-04-22T23:23:44.037 INFO:tasks.workunit.client.1.smithi134.stdout:1/402: dread db/d15/d17/d20/d2a/f44 [0,4194304] 0 2024-04-22T23:23:44.037 INFO:tasks.workunit.client.1.smithi134.stdout:9/293: dread - d0/d7/d12/f42 zero size 2024-04-22T23:23:44.037 INFO:tasks.workunit.client.1.smithi134.stdout:1/403: chown db/d16/d56 1 1 2024-04-22T23:23:44.038 INFO:tasks.workunit.client.1.smithi134.stdout:1/404: write db/f26 [994203,14425] 0 2024-04-22T23:23:44.042 INFO:tasks.workunit.client.1.smithi134.stdout:9/294: getdents d0/d2d 0 2024-04-22T23:23:44.043 INFO:tasks.workunit.client.1.smithi134.stdout:1/405: getdents db/d15 0 2024-04-22T23:23:44.045 INFO:tasks.workunit.client.1.smithi134.stdout:1/406: chown db/d15/d17/d20/l77 442124718 1 2024-04-22T23:23:44.049 INFO:tasks.workunit.client.1.smithi134.stdout:5/464: truncate d9/dd/f7b 2502155 0 2024-04-22T23:23:44.056 INFO:tasks.workunit.client.1.smithi134.stdout:5/465: dwrite d9/d11/d73/d95/d20/f91 [0,4194304] 0 2024-04-22T23:23:44.062 INFO:tasks.workunit.client.1.smithi134.stdout:5/466: mknod d9/d11/d34/ca7 0 2024-04-22T23:23:44.062 INFO:tasks.workunit.client.1.smithi134.stdout:5/467: stat d9/f19 0 2024-04-22T23:23:44.063 INFO:tasks.workunit.client.1.smithi134.stdout:9/295: dread d0/d7/f19 [0,4194304] 0 2024-04-22T23:23:44.063 INFO:tasks.workunit.client.1.smithi134.stdout:5/468: dread - d9/d11/f5d zero size 2024-04-22T23:23:44.063 INFO:tasks.workunit.client.1.smithi134.stdout:1/407: dread db/d15/d17/d20/d2a/d2d/d2e/f47 [0,4194304] 0 2024-04-22T23:23:44.066 INFO:tasks.workunit.client.1.smithi134.stdout:5/469: symlink d9/d11/d34/la8 0 2024-04-22T23:23:44.066 INFO:tasks.workunit.client.1.smithi134.stdout:5/470: truncate d9/d11/f5d 1030505 0 2024-04-22T23:23:44.067 INFO:tasks.workunit.client.1.smithi134.stdout:1/408: symlink db/d15/d17/l92 0 2024-04-22T23:23:44.069 INFO:tasks.workunit.client.1.smithi134.stdout:5/471: dread d9/d11/f7a [0,4194304] 0 2024-04-22T23:23:44.070 INFO:tasks.workunit.client.1.smithi134.stdout:9/296: getdents d0/d7/d12 0 2024-04-22T23:23:44.071 INFO:tasks.workunit.client.1.smithi134.stdout:9/297: write d0/d2d/d4e/f32 [5518105,20568] 0 2024-04-22T23:23:44.074 INFO:tasks.workunit.client.1.smithi134.stdout:9/298: write d0/d7/d23/f55 [222570,43211] 0 2024-04-22T23:23:44.074 INFO:tasks.workunit.client.1.smithi134.stdout:1/409: creat db/d15/d17/d20/d2a/d2d/d2e/f93 x:0 0 0 2024-04-22T23:23:44.077 INFO:tasks.workunit.client.1.smithi134.stdout:5/472: dwrite d9/d1e/f84 [4194304,4194304] 0 2024-04-22T23:23:44.083 INFO:tasks.workunit.client.1.smithi134.stdout:9/299: link d0/d7/d23/d26/d4d/d5d/c61 d0/d2d/c66 0 2024-04-22T23:23:44.086 INFO:tasks.workunit.client.1.smithi134.stdout:9/300: dread d0/d7/f5b [0,4194304] 0 2024-04-22T23:23:44.086 INFO:tasks.workunit.client.1.smithi134.stdout:9/301: readlink d0/d7/d23/l25 0 2024-04-22T23:23:44.087 INFO:tasks.workunit.client.1.smithi134.stdout:1/410: truncate db/d15/f18 1640756 0 2024-04-22T23:23:44.087 INFO:tasks.workunit.client.1.smithi134.stdout:9/302: readlink d0/d7/d12/l50 0 2024-04-22T23:23:44.088 INFO:tasks.workunit.client.1.smithi134.stdout:9/303: write d0/d7/d12/d22/f3e [2469409,92851] 0 2024-04-22T23:23:44.089 INFO:tasks.workunit.client.1.smithi134.stdout:1/411: rmdir db/d15/d17 39 2024-04-22T23:23:44.089 INFO:tasks.workunit.client.1.smithi134.stdout:9/304: read - d0/f57 zero size 2024-04-22T23:23:44.090 INFO:tasks.workunit.client.1.smithi134.stdout:9/305: chown d0/d7/f5b 1588 1 2024-04-22T23:23:44.091 INFO:tasks.workunit.client.1.smithi134.stdout:5/473: truncate f8 6440980 0 2024-04-22T23:23:44.092 INFO:tasks.workunit.client.1.smithi134.stdout:5/474: write d9/dd/d17/f71 [618037,61236] 0 2024-04-22T23:23:44.092 INFO:tasks.workunit.client.1.smithi134.stdout:9/306: rmdir d0/d2d/d4e 39 2024-04-22T23:23:44.094 INFO:tasks.workunit.client.1.smithi134.stdout:5/475: dread - d9/f9d zero size 2024-04-22T23:23:44.097 INFO:tasks.workunit.client.1.smithi134.stdout:1/412: rename db/d15/d17/d20/d2a/l3a to db/d15/d17/l94 0 2024-04-22T23:23:44.099 INFO:tasks.workunit.client.1.smithi134.stdout:1/413: write db/d15/d17/d20/d2a/d2d/f6e [3544059,16162] 0 2024-04-22T23:23:44.106 INFO:tasks.workunit.client.1.smithi134.stdout:1/414: mknod db/d15/d17/d20/d2a/d2d/d86/c95 0 2024-04-22T23:23:44.110 INFO:tasks.workunit.client.1.smithi134.stdout:9/307: dwrite d0/d7/d27/f33 [0,4194304] 0 2024-04-22T23:23:44.111 INFO:tasks.workunit.client.1.smithi134.stdout:5/476: dread d9/d1e/f35 [0,4194304] 0 2024-04-22T23:23:44.114 INFO:tasks.workunit.client.1.smithi134.stdout:1/415: rename db/c50 to db/d15/d17/d20/d2a/d2d/d2e/c96 0 2024-04-22T23:23:44.115 INFO:tasks.workunit.client.1.smithi134.stdout:1/416: stat db/d15/d17/d20/d2a/d2d/f48 0 2024-04-22T23:23:44.115 INFO:tasks.workunit.client.1.smithi134.stdout:5/477: mkdir d9/d11/d73/da9 0 2024-04-22T23:23:44.116 INFO:tasks.workunit.client.1.smithi134.stdout:5/478: readlink d9/d11/d34/d3b/l65 0 2024-04-22T23:23:44.118 INFO:tasks.workunit.client.1.smithi134.stdout:1/417: rename db/d16/c32 to db/d16/d56/c97 0 2024-04-22T23:23:44.122 INFO:tasks.workunit.client.1.smithi134.stdout:1/418: rmdir db/d16/d89 0 2024-04-22T23:23:44.122 INFO:tasks.workunit.client.1.smithi134.stdout:5/479: dwrite d9/f67 [8388608,4194304] 0 2024-04-22T23:23:44.123 INFO:tasks.workunit.client.1.smithi134.stdout:5/480: write d9/dd/f86 [236229,40517] 0 2024-04-22T23:23:44.124 INFO:tasks.workunit.client.1.smithi134.stdout:1/419: fdatasync db/f1b 0 2024-04-22T23:23:44.126 INFO:tasks.workunit.client.1.smithi134.stdout:5/481: creat d9/da2/faa x:0 0 0 2024-04-22T23:23:44.128 INFO:tasks.workunit.client.1.smithi134.stdout:1/420: stat db/d15/d17/d20/d2a/d2d/c67 0 2024-04-22T23:23:44.128 INFO:tasks.workunit.client.1.smithi134.stdout:1/421: rename db to db/d16/d56/d98 22 2024-04-22T23:23:44.129 INFO:tasks.workunit.client.1.smithi134.stdout:1/422: fdatasync db/f27 0 2024-04-22T23:23:44.131 INFO:tasks.workunit.client.1.smithi134.stdout:1/423: write db/d15/d17/d20/d2a/d2d/f6e [4466,54613] 0 2024-04-22T23:23:44.141 INFO:tasks.workunit.client.1.smithi134.stdout:5/482: dwrite d9/d1e/f5a [0,4194304] 0 2024-04-22T23:23:44.144 INFO:tasks.workunit.client.1.smithi134.stdout:5/483: creat d9/d11/d34/d4d/d92/fab x:0 0 0 2024-04-22T23:23:44.149 INFO:tasks.workunit.client.1.smithi134.stdout:1/424: dread db/d15/d17/d20/f46 [0,4194304] 0 2024-04-22T23:23:44.151 INFO:tasks.workunit.client.1.smithi134.stdout:1/425: mknod db/d16/d56/c99 0 2024-04-22T23:23:44.152 INFO:tasks.workunit.client.1.smithi134.stdout:1/426: dread - db/d15/d17/f69 zero size 2024-04-22T23:23:44.152 INFO:tasks.workunit.client.1.smithi134.stdout:5/484: truncate d9/d11/d34/d3b/f5f 825995 0 2024-04-22T23:23:44.153 INFO:tasks.workunit.client.1.smithi134.stdout:1/427: write db/d15/d17/f69 [930470,36519] 0 2024-04-22T23:23:44.155 INFO:tasks.workunit.client.1.smithi134.stdout:5/485: symlink d9/dd/da6/lac 0 2024-04-22T23:23:44.155 INFO:tasks.workunit.client.1.smithi134.stdout:1/428: write db/d15/d17/d20/d2a/d2d/f48 [121962,91709] 0 2024-04-22T23:23:44.166 INFO:tasks.workunit.client.1.smithi134.stdout:5/486: truncate d9/d11/d34/d3b/f5f 957341 0 2024-04-22T23:23:44.168 INFO:tasks.workunit.client.1.smithi134.stdout:5/487: creat d9/dd/d17/d2a/d49/fad x:0 0 0 2024-04-22T23:23:44.170 INFO:tasks.workunit.client.1.smithi134.stdout:5/488: mkdir d9/d11/d73/d95/dae 0 2024-04-22T23:23:44.174 INFO:tasks.workunit.client.1.smithi134.stdout:1/429: dread db/f14 [0,4194304] 0 2024-04-22T23:23:44.175 INFO:tasks.workunit.client.1.smithi134.stdout:1/430: readlink db/d16/l75 0 2024-04-22T23:23:44.176 INFO:tasks.workunit.client.1.smithi134.stdout:5/489: dwrite d9/d1e/f35 [0,4194304] 0 2024-04-22T23:23:44.181 INFO:tasks.workunit.client.1.smithi134.stdout:1/431: creat db/d15/d17/d20/d2a/d2d/d2e/d42/f9a x:0 0 0 2024-04-22T23:23:44.183 INFO:tasks.workunit.client.1.smithi134.stdout:5/490: creat d9/d11/d34/d4d/faf x:0 0 0 2024-04-22T23:23:44.183 INFO:tasks.workunit.client.1.smithi134.stdout:1/432: read - db/d15/d17/d20/d2a/d2d/d2e/d42/f66 zero size 2024-04-22T23:23:44.184 INFO:tasks.workunit.client.1.smithi134.stdout:3/263: sync 2024-04-22T23:23:44.187 INFO:tasks.workunit.client.1.smithi134.stdout:5/491: rename d9/d11/c37 to d9/d11/d34/d4d/d92/cb0 0 2024-04-22T23:23:44.190 INFO:tasks.workunit.client.1.smithi134.stdout:1/433: dwrite db/d15/f7c [0,4194304] 0 2024-04-22T23:23:44.190 INFO:tasks.workunit.client.1.smithi134.stdout:3/264: creat dc/dd/d28/d2f/d21/d42/f66 x:0 0 0 2024-04-22T23:23:44.191 INFO:tasks.workunit.client.1.smithi134.stdout:5/492: symlink d9/dd/d17/d44/lb1 0 2024-04-22T23:23:44.197 INFO:tasks.workunit.client.1.smithi134.stdout:3/265: truncate dc/dd/d28/d2f/f4b 31455 0 2024-04-22T23:23:44.198 INFO:tasks.workunit.client.1.smithi134.stdout:3/266: dread - dc/d2a/d62/f65 zero size 2024-04-22T23:23:44.199 INFO:tasks.workunit.client.1.smithi134.stdout:3/267: chown dc/d2a/d30/f60 11679 1 2024-04-22T23:23:44.200 INFO:tasks.workunit.client.1.smithi134.stdout:3/268: chown dc/l53 209 1 2024-04-22T23:23:44.206 INFO:tasks.workunit.client.1.smithi134.stdout:5/493: dwrite d9/f15 [0,4194304] 0 2024-04-22T23:23:44.209 INFO:tasks.workunit.client.1.smithi134.stdout:5/494: mknod d9/da2/cb2 0 2024-04-22T23:23:44.210 INFO:tasks.workunit.client.1.smithi134.stdout:3/269: dwrite dc/dd/d28/d34/f50 [0,4194304] 0 2024-04-22T23:23:44.219 INFO:tasks.workunit.client.1.smithi134.stdout:3/270: dwrite dc/dd/d28/d3b/f31 [0,4194304] 0 2024-04-22T23:23:44.226 INFO:tasks.workunit.client.1.smithi134.stdout:3/271: rename dc/dd/d28/d34/c64 to dc/dd/c67 0 2024-04-22T23:23:44.227 INFO:tasks.workunit.client.1.smithi134.stdout:1/434: dread db/d15/d17/d20/d2a/d2d/f64 [0,4194304] 0 2024-04-22T23:23:44.228 INFO:tasks.workunit.client.1.smithi134.stdout:1/435: fdatasync db/f1d 0 2024-04-22T23:23:44.230 INFO:tasks.workunit.client.1.smithi134.stdout:3/272: creat dc/dd/d28/d2f/f68 x:0 0 0 2024-04-22T23:23:44.232 INFO:tasks.workunit.client.1.smithi134.stdout:3/273: chown dc/dd/d28/f5c 2061507399 1 2024-04-22T23:23:44.233 INFO:tasks.workunit.client.1.smithi134.stdout:1/436: getdents db/d15 0 2024-04-22T23:23:44.234 INFO:tasks.workunit.client.1.smithi134.stdout:3/274: creat dc/dd/d28/f69 x:0 0 0 2024-04-22T23:23:44.237 INFO:tasks.workunit.client.1.smithi134.stdout:3/275: mkdir dc/dd/d28/d2f/d61/d6a 0 2024-04-22T23:23:44.239 INFO:tasks.workunit.client.1.smithi134.stdout:3/276: mkdir dc/dd/d28/d34/d6b 0 2024-04-22T23:23:44.240 INFO:tasks.workunit.client.1.smithi134.stdout:3/277: write dc/f45 [4786675,112337] 0 2024-04-22T23:23:44.241 INFO:tasks.workunit.client.1.smithi134.stdout:3/278: fsync dc/dd/d28/d2f/f51 0 2024-04-22T23:23:44.251 INFO:tasks.workunit.client.1.smithi134.stdout:1/437: dread fa [0,4194304] 0 2024-04-22T23:23:44.252 INFO:tasks.workunit.client.1.smithi134.stdout:1/438: dread - db/d15/d17/d20/d2a/d2d/d2e/f93 zero size 2024-04-22T23:23:44.254 INFO:tasks.workunit.client.1.smithi134.stdout:1/439: symlink db/d15/d17/d20/l9b 0 2024-04-22T23:23:44.256 INFO:tasks.workunit.client.1.smithi134.stdout:1/440: write db/d15/d17/f84 [829131,18934] 0 2024-04-22T23:23:44.256 INFO:tasks.workunit.client.1.smithi134.stdout:3/279: dread dc/dd/d28/d2f/f49 [0,4194304] 0 2024-04-22T23:23:44.259 INFO:tasks.workunit.client.1.smithi134.stdout:1/441: rename db/d15/d17/d20/d2a/d2d/f7e to db/d15/d17/f9c 0 2024-04-22T23:23:44.260 INFO:tasks.workunit.client.1.smithi134.stdout:1/442: dread - db/d15/d17/d20/d2a/d2d/f76 zero size 2024-04-22T23:23:44.261 INFO:tasks.workunit.client.1.smithi134.stdout:1/443: creat db/d15/d17/f9d x:0 0 0 2024-04-22T23:23:44.263 INFO:tasks.workunit.client.1.smithi134.stdout:1/444: creat db/d16/f9e x:0 0 0 2024-04-22T23:23:44.264 INFO:tasks.workunit.client.1.smithi134.stdout:1/445: dread - db/d15/d17/d20/d2a/d2d/d2e/f90 zero size 2024-04-22T23:23:44.267 INFO:tasks.workunit.client.1.smithi134.stdout:1/446: unlink db/d15/d17/l53 0 2024-04-22T23:23:44.267 INFO:tasks.workunit.client.1.smithi134.stdout:1/447: chown db/f5f 298498 1 2024-04-22T23:23:44.269 INFO:tasks.workunit.client.1.smithi134.stdout:1/448: creat db/d15/d17/d20/d2a/f9f x:0 0 0 2024-04-22T23:23:44.271 INFO:tasks.workunit.client.1.smithi134.stdout:1/449: symlink db/d15/d17/d20/d2a/d2d/d86/la0 0 2024-04-22T23:23:44.275 INFO:tasks.workunit.client.1.smithi134.stdout:1/450: dread db/d15/d17/f69 [0,4194304] 0 2024-04-22T23:23:44.277 INFO:tasks.workunit.client.1.smithi134.stdout:1/451: creat db/d16/d56/fa1 x:0 0 0 2024-04-22T23:23:44.278 INFO:tasks.workunit.client.1.smithi134.stdout:1/452: creat db/d15/d17/fa2 x:0 0 0 2024-04-22T23:23:44.293 INFO:tasks.workunit.client.1.smithi134.stdout:1/453: dread db/d15/f34 [0,4194304] 0 2024-04-22T23:23:44.299 INFO:tasks.workunit.client.1.smithi134.stdout:1/454: dwrite db/d15/d17/d20/d2a/f9f [0,4194304] 0 2024-04-22T23:23:44.308 INFO:tasks.workunit.client.1.smithi134.stdout:1/455: dwrite db/d15/d17/d20/f45 [0,4194304] 0 2024-04-22T23:23:44.315 INFO:tasks.workunit.client.1.smithi134.stdout:1/456: dwrite db/d15/f29 [0,4194304] 0 2024-04-22T23:23:44.318 INFO:tasks.workunit.client.1.smithi134.stdout:1/457: symlink db/la3 0 2024-04-22T23:23:44.324 INFO:tasks.workunit.client.1.smithi134.stdout:1/458: link db/d15/d17/f5a db/d15/d17/d20/d2a/d2d/d86/fa4 0 2024-04-22T23:23:44.325 INFO:tasks.workunit.client.1.smithi134.stdout:1/459: truncate db/d15/d17/d20/d2a/d2d/d2e/d42/f66 49067 0 2024-04-22T23:23:44.327 INFO:tasks.workunit.client.1.smithi134.stdout:1/460: symlink db/d15/d17/d20/d2a/d2d/la5 0 2024-04-22T23:23:44.329 INFO:tasks.workunit.client.1.smithi134.stdout:1/461: write db/d15/d17/d20/d2a/d2d/f6e [1702097,15873] 0 2024-04-22T23:23:44.330 INFO:tasks.workunit.client.1.smithi134.stdout:1/462: dread - db/d16/f78 zero size 2024-04-22T23:23:44.333 INFO:tasks.workunit.client.1.smithi134.stdout:1/463: link db/d16/f3d db/fa6 0 2024-04-22T23:23:44.335 INFO:tasks.workunit.client.1.smithi134.stdout:1/464: symlink db/la7 0 2024-04-22T23:23:44.337 INFO:tasks.workunit.client.1.smithi134.stdout:1/465: truncate db/d15/f18 1745819 0 2024-04-22T23:23:44.337 INFO:tasks.workunit.client.1.smithi134.stdout:6/404: sync 2024-04-22T23:23:44.339 INFO:tasks.workunit.client.1.smithi134.stdout:1/466: mknod db/d16/d56/ca8 0 2024-04-22T23:23:44.340 INFO:tasks.workunit.client.1.smithi134.stdout:6/405: truncate d0/f26 2767811 0 2024-04-22T23:23:44.343 INFO:tasks.workunit.client.1.smithi134.stdout:6/406: symlink d0/d19/de/d1f/d7d/l8c 0 2024-04-22T23:23:44.345 INFO:tasks.workunit.client.1.smithi134.stdout:6/407: dread - d0/d19/de/d12/d2a/d2d/f80 zero size 2024-04-22T23:23:44.347 INFO:tasks.workunit.client.1.smithi134.stdout:1/467: dwrite db/d16/f78 [0,4194304] 0 2024-04-22T23:23:44.349 INFO:tasks.workunit.client.1.smithi134.stdout:6/408: fdatasync d0/d19/db/d35/f44 0 2024-04-22T23:23:44.363 INFO:tasks.workunit.client.1.smithi134.stdout:1/468: rename db/d16/c25 to db/d15/d17/d20/d2a/d2d/ca9 0 2024-04-22T23:23:44.364 INFO:tasks.workunit.client.1.smithi134.stdout:1/469: fsync db/d15/d17/d20/f65 0 2024-04-22T23:23:44.368 INFO:tasks.workunit.client.1.smithi134.stdout:6/409: rename d0/d19/db/d33/f50 to d0/d19/de/d12/d29/f8d 0 2024-04-22T23:23:44.371 INFO:tasks.workunit.client.1.smithi134.stdout:1/470: rmdir db/d15/d17/d20/d2a/d2d/d2e/d42 39 2024-04-22T23:23:44.374 INFO:tasks.workunit.client.1.smithi134.stdout:6/410: creat d0/d19/de/d12/d29/d74/f8e x:0 0 0 2024-04-22T23:23:44.376 INFO:tasks.workunit.client.1.smithi134.stdout:1/471: mknod db/d16/caa 0 2024-04-22T23:23:44.377 INFO:tasks.workunit.client.1.smithi134.stdout:1/472: fdatasync db/d15/f29 0 2024-04-22T23:23:44.384 INFO:tasks.workunit.client.1.smithi134.stdout:1/473: dwrite db/d15/f29 [0,4194304] 0 2024-04-22T23:23:44.389 INFO:tasks.workunit.client.1.smithi134.stdout:6/411: write d0/d19/db/f68 [1771973,41770] 0 2024-04-22T23:23:44.391 INFO:tasks.workunit.client.1.smithi134.stdout:6/412: dread - d0/d19/de/d12/d5e/d6d/f73 zero size 2024-04-22T23:23:44.391 INFO:tasks.workunit.client.1.smithi134.stdout:1/474: getdents db/d15/d17/d20/d2a/d2d/d86 0 2024-04-22T23:23:44.393 INFO:tasks.workunit.client.1.smithi134.stdout:6/413: truncate d0/d19/de/f13 3141358 0 2024-04-22T23:23:44.395 INFO:tasks.workunit.client.1.smithi134.stdout:1/475: rmdir db/d15/d17/d20 39 2024-04-22T23:23:44.396 INFO:tasks.workunit.client.1.smithi134.stdout:6/414: mknod d0/d19/db/d33/c8f 0 2024-04-22T23:23:44.399 INFO:tasks.workunit.client.1.smithi134.stdout:6/415: mknod d0/c90 0 2024-04-22T23:23:44.404 INFO:tasks.workunit.client.1.smithi134.stdout:6/416: fsync d0/f26 0 2024-04-22T23:23:44.406 INFO:tasks.workunit.client.1.smithi134.stdout:1/476: creat db/d15/d17/d20/fab x:0 0 0 2024-04-22T23:23:44.408 INFO:tasks.workunit.client.1.smithi134.stdout:1/477: write db/d15/d17/d20/d2a/d2d/f48 [984484,103265] 0 2024-04-22T23:23:44.415 INFO:tasks.workunit.client.1.smithi134.stdout:6/417: dwrite d0/d19/de/d12/d29/f70 [0,4194304] 0 2024-04-22T23:23:44.420 INFO:tasks.workunit.client.1.smithi134.stdout:6/418: creat d0/d6f/d88/d8b/f91 x:0 0 0 2024-04-22T23:23:44.423 INFO:tasks.workunit.client.1.smithi134.stdout:1/478: dwrite db/d15/d17/d20/d2a/f51 [4194304,4194304] 0 2024-04-22T23:23:44.431 INFO:tasks.workunit.client.1.smithi134.stdout:6/419: dwrite d0/d19/de/d12/d29/d74/f89 [0,4194304] 0 2024-04-22T23:23:44.434 INFO:tasks.workunit.client.1.smithi134.stdout:6/420: mkdir d0/d19/db/d75/d92 0 2024-04-22T23:23:44.437 INFO:tasks.workunit.client.1.smithi134.stdout:1/479: dwrite db/d15/d17/f69 [0,4194304] 0 2024-04-22T23:23:44.437 INFO:tasks.workunit.client.1.smithi134.stdout:6/421: write d0/d19/de/d2f/f71 [32579,24222] 0 2024-04-22T23:23:44.439 INFO:tasks.workunit.client.1.smithi134.stdout:6/422: symlink d0/d19/de/d2f/l93 0 2024-04-22T23:23:44.440 INFO:tasks.workunit.client.1.smithi134.stdout:6/423: stat d0/d19/db/d75/c78 0 2024-04-22T23:23:44.445 INFO:tasks.workunit.client.1.smithi134.stdout:1/480: dwrite db/d15/d17/f69 [0,4194304] 0 2024-04-22T23:23:44.446 INFO:tasks.workunit.client.1.smithi134.stdout:6/424: dread d0/f1a [0,4194304] 0 2024-04-22T23:23:44.449 INFO:tasks.workunit.client.1.smithi134.stdout:1/481: rename db/d15/d17/f9c to db/d16/d56/fac 0 2024-04-22T23:23:44.449 INFO:tasks.workunit.client.1.smithi134.stdout:6/425: mkdir d0/d19/de/d1f/d94 0 2024-04-22T23:23:44.450 INFO:tasks.workunit.client.1.smithi134.stdout:6/426: fsync d0/d19/de/d1f/f2c 0 2024-04-22T23:23:44.453 INFO:tasks.workunit.client.1.smithi134.stdout:6/427: creat d0/d19/db/d33/f95 x:0 0 0 2024-04-22T23:23:44.482 INFO:tasks.workunit.client.1.smithi134.stdout:1/482: dread db/f1d [0,4194304] 0 2024-04-22T23:23:44.484 INFO:tasks.workunit.client.1.smithi134.stdout:1/483: mkdir db/d15/d17/d20/d2a/d2d/d86/dad 0 2024-04-22T23:23:44.490 INFO:tasks.workunit.client.1.smithi134.stdout:1/484: dwrite db/d15/d17/d20/d2a/d2d/f43 [0,4194304] 0 2024-04-22T23:23:44.493 INFO:tasks.workunit.client.1.smithi134.stdout:1/485: getdents db/d16/d87 0 2024-04-22T23:23:44.495 INFO:tasks.workunit.client.1.smithi134.stdout:1/486: fdatasync db/d15/d17/d20/d2a/d2d/d2e/f6c 0 2024-04-22T23:23:44.501 INFO:tasks.workunit.client.1.smithi134.stdout:1/487: dwrite db/f14 [0,4194304] 0 2024-04-22T23:23:44.505 INFO:tasks.workunit.client.1.smithi134.stdout:1/488: symlink db/d15/d17/d20/d2a/d2d/d2e/lae 0 2024-04-22T23:23:44.505 INFO:tasks.workunit.client.1.smithi134.stdout:1/489: stat db/d15/l1f 0 2024-04-22T23:23:44.506 INFO:tasks.workunit.client.1.smithi134.stdout:1/490: write db/d15/d17/d20/fab [104134,70004] 0 2024-04-22T23:23:44.508 INFO:tasks.workunit.client.1.smithi134.stdout:1/491: creat db/d16/d87/faf x:0 0 0 2024-04-22T23:23:44.510 INFO:tasks.workunit.client.1.smithi134.stdout:1/492: dread - db/d15/d17/d20/d2a/d2d/d2e/d42/f9a zero size 2024-04-22T23:23:44.552 INFO:tasks.workunit.client.1.smithi134.stdout:0/348: sync 2024-04-22T23:23:44.556 INFO:tasks.workunit.client.1.smithi134.stdout:0/349: creat d1/d5/dd/d20/f6c x:0 0 0 2024-04-22T23:23:44.563 INFO:tasks.workunit.client.1.smithi134.stdout:0/350: dread d1/d5/dd/f45 [0,4194304] 0 2024-04-22T23:23:44.567 INFO:tasks.workunit.client.1.smithi134.stdout:0/351: creat d1/d5/dd/d11/f6d x:0 0 0 2024-04-22T23:23:44.570 INFO:tasks.workunit.client.1.smithi134.stdout:0/352: rename d1/d2/c65 to d1/d2/d1d/c6e 0 2024-04-22T23:23:44.574 INFO:tasks.workunit.client.1.smithi134.stdout:0/353: rename d1/d5/dd/f31 to d1/d5/dd/d20/d2c/f6f 0 2024-04-22T23:23:44.574 INFO:tasks.workunit.client.1.smithi134.stdout:0/354: readlink d1/d5/dd/d11/l4c 0 2024-04-22T23:23:44.575 INFO:tasks.workunit.client.1.smithi134.stdout:0/355: rename d1/d2/d1d/f55 to d1/db/f70 0 2024-04-22T23:23:44.578 INFO:tasks.workunit.client.1.smithi134.stdout:0/356: getdents d1/d5 0 2024-04-22T23:23:44.584 INFO:tasks.workunit.client.1.smithi134.stdout:0/357: dwrite d1/d2/d1d/f5f [0,4194304] 0 2024-04-22T23:23:44.585 INFO:tasks.workunit.client.1.smithi134.stdout:0/358: getdents d1/d5/dd/d68 0 2024-04-22T23:23:44.587 INFO:tasks.workunit.client.1.smithi134.stdout:0/359: write d1/f9 [7345380,29853] 0 2024-04-22T23:23:44.589 INFO:tasks.workunit.client.1.smithi134.stdout:0/360: readlink d1/d5/d58/d22/l23 0 2024-04-22T23:23:44.589 INFO:tasks.workunit.client.1.smithi134.stdout:0/361: chown d1/d5/dd 0 1 2024-04-22T23:23:44.591 INFO:tasks.workunit.client.1.smithi134.stdout:0/362: truncate d1/f10 8617957 0 2024-04-22T23:23:44.593 INFO:tasks.workunit.client.1.smithi134.stdout:0/363: symlink d1/d5/d58/d6b/l71 0 2024-04-22T23:23:44.594 INFO:tasks.workunit.client.1.smithi134.stdout:0/364: unlink d1/d5/f3c 0 2024-04-22T23:23:44.595 INFO:tasks.workunit.client.1.smithi134.stdout:0/365: mkdir d1/d2/d1d/d72 0 2024-04-22T23:23:44.596 INFO:tasks.workunit.client.1.smithi134.stdout:0/366: write d1/d5/d58/d22/f62 [528907,1774] 0 2024-04-22T23:23:44.598 INFO:tasks.workunit.client.1.smithi134.stdout:0/367: symlink d1/d2/l73 0 2024-04-22T23:23:44.606 INFO:tasks.workunit.client.1.smithi134.stdout:0/368: read d1/d5/d58/d22/f62 [200509,129676] 0 2024-04-22T23:23:44.607 INFO:tasks.workunit.client.1.smithi134.stdout:0/369: chown d1/db/f70 71006408 1 2024-04-22T23:23:44.607 INFO:tasks.workunit.client.1.smithi134.stdout:0/370: fdatasync d1/d5/f5e 0 2024-04-22T23:23:44.607 INFO:tasks.workunit.client.1.smithi134.stdout:0/371: chown d1/d5/d58/f2e 61 1 2024-04-22T23:23:44.610 INFO:tasks.workunit.client.1.smithi134.stdout:0/372: creat d1/d2/f74 x:0 0 0 2024-04-22T23:23:44.616 INFO:tasks.workunit.client.1.smithi134.stdout:0/373: write d1/d5/dd/d20/d2c/f6f [268584,30521] 0 2024-04-22T23:23:44.616 INFO:tasks.workunit.client.1.smithi134.stdout:0/374: chown d1/d5/d58/d22/l60 1310 1 2024-04-22T23:23:44.622 INFO:tasks.workunit.client.1.smithi134.stdout:0/375: dwrite d1/d5/dd/d20/f6c [0,4194304] 0 2024-04-22T23:23:44.625 INFO:tasks.workunit.client.1.smithi134.stdout:0/376: mkdir d1/d5/dd/d11/d75 0 2024-04-22T23:23:44.629 INFO:tasks.workunit.client.1.smithi134.stdout:0/377: getdents d1/d5/d15 0 2024-04-22T23:23:44.633 INFO:tasks.workunit.client.1.smithi134.stdout:0/378: rename d1/d2/c52 to d1/d5/d15/c76 0 2024-04-22T23:23:44.635 INFO:tasks.workunit.client.1.smithi134.stdout:0/379: unlink d1/d5/dd/f5b 0 2024-04-22T23:23:44.636 INFO:tasks.workunit.client.1.smithi134.stdout:0/380: fsync d1/d5/d58/f3a 0 2024-04-22T23:23:44.637 INFO:tasks.workunit.client.1.smithi134.stdout:0/381: chown d1/d5/dd/f45 494 1 2024-04-22T23:23:44.648 INFO:tasks.workunit.client.1.smithi134.stdout:0/382: dread d1/d5/dd/d11/f18 [0,4194304] 0 2024-04-22T23:23:44.650 INFO:tasks.workunit.client.1.smithi134.stdout:0/383: mkdir d1/d77 0 2024-04-22T23:23:44.652 INFO:tasks.workunit.client.1.smithi134.stdout:0/384: unlink d1/d5/dd/f2b 0 2024-04-22T23:23:44.652 INFO:tasks.workunit.client.1.smithi134.stdout:0/385: stat d1/d2/f54 0 2024-04-22T23:23:44.653 INFO:tasks.workunit.client.1.smithi134.stdout:0/386: dread - d1/d5/dd/d41/f5d zero size 2024-04-22T23:23:44.654 INFO:tasks.workunit.client.1.smithi134.stdout:0/387: write d1/d5/dd/d20/f6c [2424654,131018] 0 2024-04-22T23:23:44.660 INFO:tasks.workunit.client.1.smithi134.stdout:7/438: sync 2024-04-22T23:23:44.660 INFO:tasks.workunit.client.1.smithi134.stdout:2/274: sync 2024-04-22T23:23:44.660 INFO:tasks.workunit.client.1.smithi134.stdout:9/308: sync 2024-04-22T23:23:44.660 INFO:tasks.workunit.client.1.smithi134.stdout:3/280: sync 2024-04-22T23:23:44.660 INFO:tasks.workunit.client.1.smithi134.stdout:5/495: sync 2024-04-22T23:23:44.660 INFO:tasks.workunit.client.1.smithi134.stdout:8/287: sync 2024-04-22T23:23:44.660 INFO:tasks.workunit.client.1.smithi134.stdout:4/539: sync 2024-04-22T23:23:44.663 INFO:tasks.workunit.client.1.smithi134.stdout:0/388: dwrite d1/d5/d58/f46 [0,4194304] 0 2024-04-22T23:23:44.663 INFO:tasks.workunit.client.1.smithi134.stdout:9/309: stat d0/d7/d23/d1b/d5a 0 2024-04-22T23:23:44.663 INFO:tasks.workunit.client.1.smithi134.stdout:5/496: chown d9/dd/l96 0 1 2024-04-22T23:23:44.664 INFO:tasks.workunit.client.1.smithi134.stdout:8/288: chown d7/dc/d2c/l35 6472 1 2024-04-22T23:23:44.665 INFO:tasks.workunit.client.1.smithi134.stdout:3/281: mknod dc/dd/c6c 0 2024-04-22T23:23:44.667 INFO:tasks.workunit.client.1.smithi134.stdout:7/439: creat d4/d10/d2f/d34/d62/d84/f9f x:0 0 0 2024-04-22T23:23:44.670 INFO:tasks.workunit.client.1.smithi134.stdout:5/497: mknod d9/dd/d17/d44/cb3 0 2024-04-22T23:23:44.671 INFO:tasks.workunit.client.1.smithi134.stdout:8/289: read d7/dc/ff [1879387,82937] 0 2024-04-22T23:23:44.672 INFO:tasks.workunit.client.1.smithi134.stdout:0/389: symlink d1/d5/dd/d20/d2c/l78 0 2024-04-22T23:23:44.673 INFO:tasks.workunit.client.1.smithi134.stdout:7/440: mknod d4/d8/d41/ca0 0 2024-04-22T23:23:44.674 INFO:tasks.workunit.client.1.smithi134.stdout:4/540: rename d4/f2b to d4/d15/d82/fb2 0 2024-04-22T23:23:44.675 INFO:tasks.workunit.client.1.smithi134.stdout:5/498: write d9/dd/d17/d2a/f98 [775997,120072] 0 2024-04-22T23:23:44.677 INFO:tasks.workunit.client.1.smithi134.stdout:0/390: stat d1/d5/dd/d20/d2c/f4b 0 2024-04-22T23:23:44.678 INFO:tasks.workunit.client.1.smithi134.stdout:4/541: dread - d4/d15/d82/d9a/faa zero size 2024-04-22T23:23:44.680 INFO:tasks.workunit.client.1.smithi134.stdout:9/310: dwrite d0/d7/d27/f59 [0,4194304] 0 2024-04-22T23:23:44.681 INFO:tasks.workunit.client.1.smithi134.stdout:5/499: dread - d9/dd/d52/f6b zero size 2024-04-22T23:23:44.682 INFO:tasks.workunit.client.1.smithi134.stdout:2/275: dread d2/d15/d24/d1d/f28 [0,4194304] 0 2024-04-22T23:23:44.683 INFO:tasks.workunit.client.1.smithi134.stdout:9/311: chown d0/d37/l3a 31 1 2024-04-22T23:23:44.683 INFO:tasks.workunit.client.1.smithi134.stdout:7/441: mknod d4/d8/d58/ca1 0 2024-04-22T23:23:44.684 INFO:tasks.workunit.client.1.smithi134.stdout:2/276: write d2/f17 [2881311,22203] 0 2024-04-22T23:23:44.686 INFO:tasks.workunit.client.1.smithi134.stdout:7/442: truncate d4/d10/d2f/d34/d62/d84/f9f 589030 0 2024-04-22T23:23:44.690 INFO:tasks.workunit.client.1.smithi134.stdout:9/312: dread d0/d7/f19 [0,4194304] 0 2024-04-22T23:23:44.691 INFO:tasks.workunit.client.1.smithi134.stdout:2/277: stat d2/f30 0 2024-04-22T23:23:44.691 INFO:tasks.workunit.client.1.smithi134.stdout:4/542: symlink d4/d15/d66/lb3 0 2024-04-22T23:23:44.694 INFO:tasks.workunit.client.1.smithi134.stdout:5/500: mkdir d9/d1e/db4 0 2024-04-22T23:23:44.695 INFO:tasks.workunit.client.1.smithi134.stdout:4/543: chown d4/d15/d64/laf 123017127 1 2024-04-22T23:23:44.696 INFO:tasks.workunit.client.1.smithi134.stdout:0/391: dread d1/d5/dd/d20/d2c/f6f [0,4194304] 0 2024-04-22T23:23:44.696 INFO:tasks.workunit.client.1.smithi134.stdout:8/290: dwrite d7/dc/d2c/d37/f3f [0,4194304] 0 2024-04-22T23:23:44.696 INFO:tasks.workunit.client.1.smithi134.stdout:9/313: chown d0/d7/d12/d22/f31 49616142 1 2024-04-22T23:23:44.697 INFO:tasks.workunit.client.1.smithi134.stdout:7/443: rename d4/d8/d41/d5b/f64 to d4/d8/d2a/d42/d79/fa2 0 2024-04-22T23:23:44.699 INFO:tasks.workunit.client.1.smithi134.stdout:2/278: symlink d2/d15/d24/l5e 0 2024-04-22T23:23:44.700 INFO:tasks.workunit.client.1.smithi134.stdout:0/392: dread - d1/d2/f54 zero size 2024-04-22T23:23:44.701 INFO:tasks.workunit.client.1.smithi134.stdout:4/544: fdatasync d4/d13/f4c 0 2024-04-22T23:23:44.702 INFO:tasks.workunit.client.1.smithi134.stdout:2/279: chown d2/d15/d24/f5d 2 1 2024-04-22T23:23:44.703 INFO:tasks.workunit.client.1.smithi134.stdout:0/393: read - d1/d2/f74 zero size 2024-04-22T23:23:44.704 INFO:tasks.workunit.client.1.smithi134.stdout:7/444: write d4/d10/d2f/d34/d62/f71 [296462,51519] 0 2024-04-22T23:23:44.704 INFO:tasks.workunit.client.1.smithi134.stdout:8/291: creat d7/db/d42/f59 x:0 0 0 2024-04-22T23:23:44.704 INFO:tasks.workunit.client.1.smithi134.stdout:9/314: mknod d0/d7/c67 0 2024-04-22T23:23:44.706 INFO:tasks.workunit.client.1.smithi134.stdout:2/280: chown d2/dd 3291 1 2024-04-22T23:23:44.708 INFO:tasks.workunit.client.1.smithi134.stdout:5/501: dwrite d9/d11/d34/d4d/faf [0,4194304] 0 2024-04-22T23:23:44.708 INFO:tasks.workunit.client.1.smithi134.stdout:9/315: dread - d0/d2d/f44 zero size 2024-04-22T23:23:44.710 INFO:tasks.workunit.client.1.smithi134.stdout:8/292: dread d7/db/d2d/d39/f3d [0,4194304] 0 2024-04-22T23:23:44.710 INFO:tasks.workunit.client.1.smithi134.stdout:2/281: write d2/d15/f50 [3684120,17284] 0 2024-04-22T23:23:44.713 INFO:tasks.workunit.client.1.smithi134.stdout:0/394: mkdir d1/d79 0 2024-04-22T23:23:44.713 INFO:tasks.workunit.client.1.smithi134.stdout:7/445: rename d4/d10/f6b to d4/d10/d2f/d34/d54/fa3 0 2024-04-22T23:23:44.715 INFO:tasks.workunit.client.1.smithi134.stdout:9/316: symlink d0/d37/l68 0 2024-04-22T23:23:44.715 INFO:tasks.workunit.client.1.smithi134.stdout:7/446: dread - d4/d10/d2f/d34/d62/d84/f8f zero size 2024-04-22T23:23:44.719 INFO:tasks.workunit.client.1.smithi134.stdout:2/282: mknod d2/d42/d54/d33/c5f 0 2024-04-22T23:23:44.723 INFO:tasks.workunit.client.1.smithi134.stdout:2/283: readlink d2/d15/d24/l4d 0 2024-04-22T23:23:44.726 INFO:tasks.workunit.client.1.smithi134.stdout:7/447: creat d4/d8/d2a/d42/d7d/fa4 x:0 0 0 2024-04-22T23:23:44.726 INFO:tasks.workunit.client.1.smithi134.stdout:2/284: creat d2/f60 x:0 0 0 2024-04-22T23:23:44.727 INFO:tasks.workunit.client.1.smithi134.stdout:8/293: creat d7/db/d2d/f5a x:0 0 0 2024-04-22T23:23:44.728 INFO:tasks.workunit.client.1.smithi134.stdout:2/285: stat d2/d15/d24/d3b 0 2024-04-22T23:23:44.729 INFO:tasks.workunit.client.1.smithi134.stdout:0/395: rename d1/f10 to d1/d5/dd/f7a 0 2024-04-22T23:23:44.730 INFO:tasks.workunit.client.1.smithi134.stdout:0/396: readlink d1/d2/l37 0 2024-04-22T23:23:44.730 INFO:tasks.workunit.client.1.smithi134.stdout:5/502: dwrite d9/d11/f1b [4194304,4194304] 0 2024-04-22T23:23:44.731 INFO:tasks.workunit.client.1.smithi134.stdout:8/294: rmdir d7/dc/d2c/d4f 39 2024-04-22T23:23:44.732 INFO:tasks.workunit.client.1.smithi134.stdout:2/286: rmdir d2/d42 39 2024-04-22T23:23:44.732 INFO:tasks.workunit.client.1.smithi134.stdout:5/503: write d9/d11/f5b [424373,53468] 0 2024-04-22T23:23:44.738 INFO:tasks.workunit.client.1.smithi134.stdout:5/504: mknod d9/d11/d73/d95/d20/cb5 0 2024-04-22T23:23:44.740 INFO:tasks.workunit.client.1.smithi134.stdout:7/448: creat d4/d10/fa5 x:0 0 0 2024-04-22T23:23:44.742 INFO:tasks.workunit.client.1.smithi134.stdout:0/397: creat d1/d79/f7b x:0 0 0 2024-04-22T23:23:44.744 INFO:tasks.workunit.client.1.smithi134.stdout:9/317: rename d0/d7/d12/d22/f3e to d0/f69 0 2024-04-22T23:23:44.745 INFO:tasks.workunit.client.1.smithi134.stdout:2/287: symlink d2/l61 0 2024-04-22T23:23:44.747 INFO:tasks.workunit.client.1.smithi134.stdout:7/449: rename d4/d8c to d4/d10/d2f/da6 0 2024-04-22T23:23:44.749 INFO:tasks.workunit.client.1.smithi134.stdout:0/398: getdents d1/d2/d1d/d72 0 2024-04-22T23:23:44.749 INFO:tasks.workunit.client.1.smithi134.stdout:7/450: chown d4/d10/d2f/d34/d54/c60 3 1 2024-04-22T23:23:44.750 INFO:tasks.workunit.client.1.smithi134.stdout:9/318: fsync d0/d7/d12/d22/f31 0 2024-04-22T23:23:44.751 INFO:tasks.workunit.client.1.smithi134.stdout:0/399: write d1/d2/d1d/f69 [64051,18644] 0 2024-04-22T23:23:44.753 INFO:tasks.workunit.client.1.smithi134.stdout:8/295: truncate d7/dc/f22 294100 0 2024-04-22T23:23:44.754 INFO:tasks.workunit.client.1.smithi134.stdout:8/296: write d7/db/d2d/d39/f4b [409850,16701] 0 2024-04-22T23:23:44.755 INFO:tasks.workunit.client.1.smithi134.stdout:8/297: fsync d7/dc/d2c/d37/f3f 0 2024-04-22T23:23:44.756 INFO:tasks.workunit.client.1.smithi134.stdout:9/319: creat d0/d7/d12/f6a x:0 0 0 2024-04-22T23:23:44.756 INFO:tasks.workunit.client.1.smithi134.stdout:9/320: write d0/d37/f34 [920902,62346] 0 2024-04-22T23:23:44.757 INFO:tasks.workunit.client.1.smithi134.stdout:9/321: stat d0/d7/d12/l14 0 2024-04-22T23:23:44.758 INFO:tasks.workunit.client.1.smithi134.stdout:7/451: rename d4/d10/d2f/d34/d62/l5d to d4/d10/d2f/d69/la7 0 2024-04-22T23:23:44.759 INFO:tasks.workunit.client.1.smithi134.stdout:9/322: write d0/d7/d23/d1b/f49 [309060,70876] 0 2024-04-22T23:23:44.760 INFO:tasks.workunit.client.1.smithi134.stdout:2/288: link d2/d15/d24/d1d/l2f d2/d15/d24/d1d/d3d/l62 0 2024-04-22T23:23:44.763 INFO:tasks.workunit.client.1.smithi134.stdout:0/400: mknod d1/d2/d1d/d72/c7c 0 2024-04-22T23:23:44.765 INFO:tasks.workunit.client.1.smithi134.stdout:0/401: stat d1/d5/dd/d11 0 2024-04-22T23:23:44.767 INFO:tasks.workunit.client.1.smithi134.stdout:8/298: truncate d7/dc/d2c/d4f/f57 741835 0 2024-04-22T23:23:44.767 INFO:tasks.workunit.client.1.smithi134.stdout:7/452: mkdir d4/d10/d2f/d69/d8a/da8 0 2024-04-22T23:23:44.769 INFO:tasks.workunit.client.1.smithi134.stdout:7/453: write d4/f7b [4214545,61063] 0 2024-04-22T23:23:44.770 INFO:tasks.workunit.client.1.smithi134.stdout:2/289: dwrite d2/d15/d24/f1e [4194304,4194304] 0 2024-04-22T23:23:44.773 INFO:tasks.workunit.client.1.smithi134.stdout:7/454: truncate d4/d8/d3b/f68 427484 0 2024-04-22T23:23:44.774 INFO:tasks.workunit.client.1.smithi134.stdout:9/323: creat d0/d2d/f6b x:0 0 0 2024-04-22T23:23:44.776 INFO:tasks.workunit.client.1.smithi134.stdout:7/455: chown d4/d10/d2f/d37/d95/c96 131272086 1 2024-04-22T23:23:44.779 INFO:tasks.workunit.client.1.smithi134.stdout:7/456: symlink d4/d10/d2f/da6/la9 0 2024-04-22T23:23:44.784 INFO:tasks.workunit.client.1.smithi134.stdout:0/402: dwrite d1/d5/dd/f7a [4194304,4194304] 0 2024-04-22T23:23:44.784 INFO:tasks.workunit.client.1.smithi134.stdout:2/290: truncate d2/d15/d24/d1d/f36 571151 0 2024-04-22T23:23:44.792 INFO:tasks.workunit.client.1.smithi134.stdout:8/299: dread d7/dc/d2c/d37/d47/f34 [0,4194304] 0 2024-04-22T23:23:44.792 INFO:tasks.workunit.client.1.smithi134.stdout:0/403: rename d1/d5/d58/f3a to d1/d2/d1d/f7d 0 2024-04-22T23:23:44.793 INFO:tasks.workunit.client.1.smithi134.stdout:8/300: stat d7/dc/d2c/d37/d47/f34 0 2024-04-22T23:23:44.795 INFO:tasks.workunit.client.1.smithi134.stdout:7/457: creat d4/d10/faa x:0 0 0 2024-04-22T23:23:44.795 INFO:tasks.workunit.client.1.smithi134.stdout:9/324: truncate d0/d7/d23/d26/f29 1866759 0 2024-04-22T23:23:44.796 INFO:tasks.workunit.client.1.smithi134.stdout:9/325: chown d0/c1e 11264388 1 2024-04-22T23:23:44.846 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:44 smithi067 ceph-mon[26453]: pgmap v277: 65 pgs: 65 active+clean; 2.7 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 73 MiB/s rd, 230 MiB/s wr, 429 op/s 2024-04-22T23:23:44.851 INFO:tasks.workunit.client.1.smithi134.stdout:2/291: truncate d2/d15/f50 9048807 0 2024-04-22T23:23:44.854 INFO:tasks.workunit.client.1.smithi134.stdout:8/301: rmdir d7/db 39 2024-04-22T23:23:44.857 INFO:tasks.workunit.client.1.smithi134.stdout:7/458: rename d4/d10/d2f/d37 to d4/d10/d2f/d69/dab 0 2024-04-22T23:23:44.860 INFO:tasks.workunit.client.1.smithi134.stdout:0/404: mknod d1/d5/dd/d68/c7e 0 2024-04-22T23:23:44.866 INFO:tasks.workunit.client.1.smithi134.stdout:7/459: dwrite d4/d10/fa5 [0,4194304] 0 2024-04-22T23:23:44.874 INFO:tasks.workunit.client.1.smithi134.stdout:9/326: dwrite d0/d7/f19 [4194304,4194304] 0 2024-04-22T23:23:44.876 INFO:tasks.workunit.client.1.smithi134.stdout:9/327: readlink d0/d2d/d3d/l63 0 2024-04-22T23:23:44.878 INFO:tasks.workunit.client.1.smithi134.stdout:7/460: dwrite d4/d8/d3b/f8d [0,4194304] 0 2024-04-22T23:23:44.880 INFO:tasks.workunit.client.1.smithi134.stdout:7/461: write d4/d8/d3b/f68 [993503,108788] 0 2024-04-22T23:23:44.882 INFO:tasks.workunit.client.1.smithi134.stdout:7/462: write d4/f7b [4706095,5028] 0 2024-04-22T23:23:44.885 INFO:tasks.workunit.client.1.smithi134.stdout:8/302: chown d7/db/d2d/f3e 323564905 1 2024-04-22T23:23:44.958 INFO:tasks.workunit.client.1.smithi134.stdout:9/328: dread d0/d7/f9 [0,4194304] 0 2024-04-22T23:23:44.963 INFO:tasks.workunit.client.1.smithi134.stdout:6/428: sync 2024-04-22T23:23:44.978 INFO:tasks.workunit.client.1.smithi134.stdout:0/405: dread d1/d5/dd/d20/f1e [0,4194304] 0 2024-04-22T23:23:44.983 INFO:tasks.workunit.client.1.smithi134.stdout:2/292: symlink d2/dd/df/l63 0 2024-04-22T23:23:44.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:44 smithi134 ceph-mon[34609]: pgmap v277: 65 pgs: 65 active+clean; 2.7 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 73 MiB/s rd, 230 MiB/s wr, 429 op/s 2024-04-22T23:23:44.997 INFO:tasks.workunit.client.1.smithi134.stdout:0/406: rename d1/d5/d15/c26 to d1/db/c7f 0 2024-04-22T23:23:45.007 INFO:tasks.workunit.client.1.smithi134.stdout:2/293: readlink d2/d42/l48 0 2024-04-22T23:23:45.009 INFO:tasks.workunit.client.1.smithi134.stdout:2/294: chown d2/dd/l38 4 1 2024-04-22T23:23:45.039 INFO:tasks.workunit.client.1.smithi134.stdout:2/295: rmdir d2/dd/df 39 2024-04-22T23:23:45.042 INFO:tasks.workunit.client.1.smithi134.stdout:9/329: write d0/d7/d12/d22/f31 [679352,53648] 0 2024-04-22T23:23:45.044 INFO:tasks.workunit.client.1.smithi134.stdout:2/296: creat d2/dd/d4a/f64 x:0 0 0 2024-04-22T23:23:45.049 INFO:tasks.workunit.client.1.smithi134.stdout:2/297: symlink d2/d15/d24/d1d/d3d/l65 0 2024-04-22T23:23:45.052 INFO:tasks.workunit.client.1.smithi134.stdout:9/330: dwrite d0/d7/d12/d22/f31 [0,4194304] 0 2024-04-22T23:23:45.053 INFO:tasks.workunit.client.1.smithi134.stdout:8/303: truncate d7/db/d2d/d39/f4b 278996 0 2024-04-22T23:23:45.053 INFO:tasks.workunit.client.1.smithi134.stdout:2/298: chown d2/d15/d24/f5d 2954 1 2024-04-22T23:23:45.056 INFO:tasks.workunit.client.1.smithi134.stdout:6/429: dwrite d0/d19/db/f2b [0,4194304] 0 2024-04-22T23:23:45.057 INFO:tasks.workunit.client.1.smithi134.stdout:9/331: write d0/d7/f17 [916928,112507] 0 2024-04-22T23:23:45.058 INFO:tasks.workunit.client.1.smithi134.stdout:6/430: stat d0/d19/db/c17 0 2024-04-22T23:23:45.061 INFO:tasks.workunit.client.1.smithi134.stdout:9/332: truncate d0/d2d/d3d/f46 767571 0 2024-04-22T23:23:45.063 INFO:tasks.workunit.client.1.smithi134.stdout:9/333: dread - d0/d7/d12/f6a zero size 2024-04-22T23:23:45.064 INFO:tasks.workunit.client.1.smithi134.stdout:2/299: dwrite d2/f17 [0,4194304] 0 2024-04-22T23:23:45.068 INFO:tasks.workunit.client.1.smithi134.stdout:8/304: unlink d7/db/f56 0 2024-04-22T23:23:45.068 INFO:tasks.workunit.client.1.smithi134.stdout:8/305: readlink d7/db/l41 0 2024-04-22T23:23:45.069 INFO:tasks.workunit.client.1.smithi134.stdout:8/306: chown d7/dc/d2c/f49 1111655 1 2024-04-22T23:23:45.070 INFO:tasks.workunit.client.1.smithi134.stdout:9/334: mknod d0/d37/c6c 0 2024-04-22T23:23:45.074 INFO:tasks.workunit.client.1.smithi134.stdout:8/307: dread d7/db/d2d/f2e [0,4194304] 0 2024-04-22T23:23:45.078 INFO:tasks.workunit.client.1.smithi134.stdout:6/431: link d0/d19/db/l65 d0/d4f/l96 0 2024-04-22T23:23:45.079 INFO:tasks.workunit.client.1.smithi134.stdout:8/308: rmdir d7/db/d2d/d33/d46 39 2024-04-22T23:23:45.081 INFO:tasks.workunit.client.1.smithi134.stdout:9/335: mknod d0/d2d/d4e/c6d 0 2024-04-22T23:23:45.082 INFO:tasks.workunit.client.1.smithi134.stdout:9/336: stat d0/d7/d23/d47/c65 0 2024-04-22T23:23:45.083 INFO:tasks.workunit.client.1.smithi134.stdout:6/432: chown d0/d19/de/d12/d2a/f3b 3 1 2024-04-22T23:23:45.085 INFO:tasks.workunit.client.1.smithi134.stdout:9/337: stat d0/d7/d23/d26/f29 0 2024-04-22T23:23:45.086 INFO:tasks.workunit.client.1.smithi134.stdout:9/338: fdatasync d0/d7/d23/f45 0 2024-04-22T23:23:45.088 INFO:tasks.workunit.client.1.smithi134.stdout:6/433: rmdir d0/d19/de/d12/d29/d74 39 2024-04-22T23:23:45.089 INFO:tasks.workunit.client.1.smithi134.stdout:6/434: dread - d0/d19/db/d33/f95 zero size 2024-04-22T23:23:45.091 INFO:tasks.workunit.client.1.smithi134.stdout:6/435: truncate d0/d19/db/d35/f79 264892 0 2024-04-22T23:23:45.091 INFO:tasks.workunit.client.1.smithi134.stdout:8/309: link d7/db/d42/c52 d7/dc/d2c/d4f/c5b 0 2024-04-22T23:23:45.093 INFO:tasks.workunit.client.1.smithi134.stdout:6/436: chown d0/d19/de/d1f/c36 90 1 2024-04-22T23:23:45.093 INFO:tasks.workunit.client.1.smithi134.stdout:9/339: symlink d0/d7/d27/l6e 0 2024-04-22T23:23:45.094 INFO:tasks.workunit.client.1.smithi134.stdout:9/340: chown d0/d7/d12/l14 5 1 2024-04-22T23:23:45.096 INFO:tasks.workunit.client.1.smithi134.stdout:8/310: unlink d7/dc/d2c/d4f/f57 0 2024-04-22T23:23:45.104 INFO:tasks.workunit.client.1.smithi134.stdout:6/437: creat d0/d19/de/d12/f97 x:0 0 0 2024-04-22T23:23:45.105 INFO:tasks.workunit.client.1.smithi134.stdout:9/341: link d0/d2d/d4e/f62 d0/d7/f6f 0 2024-04-22T23:23:45.106 INFO:tasks.workunit.client.1.smithi134.stdout:9/342: fsync d0/d7/d23/f45 0 2024-04-22T23:23:45.107 INFO:tasks.workunit.client.1.smithi134.stdout:9/343: fdatasync d0/d37/f34 0 2024-04-22T23:23:45.108 INFO:tasks.workunit.client.1.smithi134.stdout:6/438: mkdir d0/d19/de/d12/d2a/d2d/d3a/d98 0 2024-04-22T23:23:45.109 INFO:tasks.workunit.client.1.smithi134.stdout:8/311: getdents d7/db/d2d/d33 0 2024-04-22T23:23:45.118 INFO:tasks.workunit.client.1.smithi134.stdout:7/463: creat d4/fac x:0 0 0 2024-04-22T23:23:45.118 INFO:tasks.workunit.client.1.smithi134.stdout:9/344: dwrite d0/d2d/d4e/f32 [4194304,4194304] 0 2024-04-22T23:23:45.119 INFO:tasks.workunit.client.1.smithi134.stdout:8/312: dwrite d7/dc/d2c/d37/f4d [0,4194304] 0 2024-04-22T23:23:45.127 INFO:tasks.workunit.client.1.smithi134.stdout:8/313: creat d7/dc/d2c/d37/d47/f5c x:0 0 0 2024-04-22T23:23:45.128 INFO:tasks.workunit.client.1.smithi134.stdout:1/493: sync 2024-04-22T23:23:45.128 INFO:tasks.workunit.client.1.smithi134.stdout:3/282: sync 2024-04-22T23:23:45.129 INFO:tasks.workunit.client.1.smithi134.stdout:1/494: dread - db/d15/d17/f9d zero size 2024-04-22T23:23:45.129 INFO:tasks.workunit.client.1.smithi134.stdout:3/283: write dc/f22 [3504935,82144] 0 2024-04-22T23:23:45.129 INFO:tasks.workunit.client.1.smithi134.stdout:7/464: mkdir d4/dad 0 2024-04-22T23:23:45.132 INFO:tasks.workunit.client.1.smithi134.stdout:1/495: write db/f28 [2921026,116142] 0 2024-04-22T23:23:45.135 INFO:tasks.workunit.client.1.smithi134.stdout:7/465: dread - d4/d10/d2f/d70/f72 zero size 2024-04-22T23:23:45.138 INFO:tasks.workunit.client.1.smithi134.stdout:3/284: rmdir dc/dd/d28 39 2024-04-22T23:23:45.144 INFO:tasks.workunit.client.1.smithi134.stdout:9/345: getdents d0/d7/d12 0 2024-04-22T23:23:45.146 INFO:tasks.workunit.client.1.smithi134.stdout:1/496: mkdir db/db0 0 2024-04-22T23:23:45.147 INFO:tasks.workunit.client.1.smithi134.stdout:9/346: read d0/d7/d27/f33 [3089280,14036] 0 2024-04-22T23:23:45.150 INFO:tasks.workunit.client.1.smithi134.stdout:3/285: dwrite dc/f22 [0,4194304] 0 2024-04-22T23:23:45.150 INFO:tasks.workunit.client.1.smithi134.stdout:2/300: dwrite d2/d15/d24/d1d/f36 [0,4194304] 0 2024-04-22T23:23:45.160 INFO:tasks.workunit.client.1.smithi134.stdout:3/286: rmdir dc/d2a 39 2024-04-22T23:23:45.163 INFO:tasks.workunit.client.1.smithi134.stdout:1/497: symlink db/d15/d17/d20/d2a/lb1 0 2024-04-22T23:23:45.166 INFO:tasks.workunit.client.1.smithi134.stdout:2/301: symlink d2/dd/d4a/l66 0 2024-04-22T23:23:45.170 INFO:tasks.workunit.client.1.smithi134.stdout:2/302: chown d2/d15/d24/d1d/d3d 24653 1 2024-04-22T23:23:45.170 INFO:tasks.workunit.client.1.smithi134.stdout:6/439: dwrite d0/f1a [0,4194304] 0 2024-04-22T23:23:45.172 INFO:tasks.workunit.client.1.smithi134.stdout:2/303: chown d2/d15/d24/d1d/l20 52 1 2024-04-22T23:23:45.174 INFO:tasks.workunit.client.1.smithi134.stdout:2/304: write d2/d15/d24/f1e [438269,110647] 0 2024-04-22T23:23:45.175 INFO:tasks.workunit.client.1.smithi134.stdout:3/287: mkdir dc/dd/d28/d2f/d36/d6d 0 2024-04-22T23:23:45.176 INFO:tasks.workunit.client.1.smithi134.stdout:3/288: write dc/f52 [543119,38833] 0 2024-04-22T23:23:45.178 INFO:tasks.workunit.client.1.smithi134.stdout:1/498: mknod db/d15/d17/d20/d2a/d2d/cb2 0 2024-04-22T23:23:45.179 INFO:tasks.workunit.client.1.smithi134.stdout:8/314: truncate d7/f23 6129163 0 2024-04-22T23:23:45.183 INFO:tasks.workunit.client.1.smithi134.stdout:6/440: mkdir d0/d19/de/d12/d5e/d6d/d7a/d99 0 2024-04-22T23:23:45.187 INFO:tasks.workunit.client.1.smithi134.stdout:2/305: symlink d2/dd/l67 0 2024-04-22T23:23:45.188 INFO:tasks.workunit.client.1.smithi134.stdout:1/499: dwrite db/d15/d17/d20/d2a/d2d/f76 [0,4194304] 0 2024-04-22T23:23:45.193 INFO:tasks.workunit.client.1.smithi134.stdout:1/500: fsync db/d15/f54 0 2024-04-22T23:23:45.197 INFO:tasks.workunit.client.1.smithi134.stdout:1/501: chown db/d15/f7c 215 1 2024-04-22T23:23:45.199 INFO:tasks.workunit.client.1.smithi134.stdout:6/441: creat d0/d19/de/d12/d2a/d2d/d3a/f9a x:0 0 0 2024-04-22T23:23:45.199 INFO:tasks.workunit.client.1.smithi134.stdout:9/347: dwrite d0/d7/d23/d26/f29 [0,4194304] 0 2024-04-22T23:23:45.199 INFO:tasks.workunit.client.1.smithi134.stdout:8/315: dread d7/db/d2d/d39/f4b [0,4194304] 0 2024-04-22T23:23:45.204 INFO:tasks.workunit.client.1.smithi134.stdout:9/348: truncate d0/d2d/d4e/f32 8562233 0 2024-04-22T23:23:45.206 INFO:tasks.workunit.client.1.smithi134.stdout:1/502: creat db/d15/d17/d20/d2a/fb3 x:0 0 0 2024-04-22T23:23:45.209 INFO:tasks.workunit.client.1.smithi134.stdout:9/349: symlink d0/d2d/d3d/l70 0 2024-04-22T23:23:45.210 INFO:tasks.workunit.client.1.smithi134.stdout:3/289: dread dc/f45 [0,4194304] 0 2024-04-22T23:23:45.210 INFO:tasks.workunit.client.1.smithi134.stdout:8/316: mknod d7/dc/d2c/c5d 0 2024-04-22T23:23:45.212 INFO:tasks.workunit.client.1.smithi134.stdout:1/503: mknod db/d15/d17/d20/d2a/d2d/d86/cb4 0 2024-04-22T23:23:45.212 INFO:tasks.workunit.client.1.smithi134.stdout:2/306: write d2/d15/d24/d3b/f4f [954955,22378] 0 2024-04-22T23:23:45.215 INFO:tasks.workunit.client.1.smithi134.stdout:1/504: write db/d15/d17/d20/d2a/f9f [408919,49978] 0 2024-04-22T23:23:45.216 INFO:tasks.workunit.client.1.smithi134.stdout:9/350: dread d0/d7/d12/d22/f31 [0,4194304] 0 2024-04-22T23:23:45.219 INFO:tasks.workunit.client.1.smithi134.stdout:3/290: symlink dc/dd/l6e 0 2024-04-22T23:23:45.219 INFO:tasks.workunit.client.1.smithi134.stdout:3/291: dread - dc/dd/d28/f69 zero size 2024-04-22T23:23:45.221 INFO:tasks.workunit.client.1.smithi134.stdout:8/317: write d7/dc/d19/f21 [3570947,97433] 0 2024-04-22T23:23:45.223 INFO:tasks.workunit.client.1.smithi134.stdout:9/351: unlink d0/d2d/d3d/f43 0 2024-04-22T23:23:45.224 INFO:tasks.workunit.client.1.smithi134.stdout:9/352: dread - d0/d7/f5e zero size 2024-04-22T23:23:45.225 INFO:tasks.workunit.client.1.smithi134.stdout:1/505: mkdir db/db5 0 2024-04-22T23:23:45.228 INFO:tasks.workunit.client.1.smithi134.stdout:9/353: dread d0/d7/d23/d1b/f49 [0,4194304] 0 2024-04-22T23:23:45.229 INFO:tasks.workunit.client.1.smithi134.stdout:6/442: dread d0/d19/de/d12/d2a/d2d/d3a/f77 [0,4194304] 0 2024-04-22T23:23:45.230 INFO:tasks.workunit.client.1.smithi134.stdout:3/292: mkdir dc/dd/d28/d2f/d36/d6d/d6f 0 2024-04-22T23:23:45.231 INFO:tasks.workunit.client.1.smithi134.stdout:6/443: write d0/f55 [5257477,107738] 0 2024-04-22T23:23:45.232 INFO:tasks.workunit.client.1.smithi134.stdout:8/318: getdents d7/db/d2d/d39/d58 0 2024-04-22T23:23:45.232 INFO:tasks.workunit.client.1.smithi134.stdout:2/307: dwrite d2/f22 [0,4194304] 0 2024-04-22T23:23:45.237 INFO:tasks.workunit.client.1.smithi134.stdout:8/319: symlink d7/dc/d2c/d4f/l5e 0 2024-04-22T23:23:45.241 INFO:tasks.workunit.client.1.smithi134.stdout:1/506: rename db/d16/d56/l8f to db/lb6 0 2024-04-22T23:23:45.241 INFO:tasks.workunit.client.1.smithi134.stdout:6/444: dwrite d0/d19/de/d12/f97 [0,4194304] 0 2024-04-22T23:23:45.244 INFO:tasks.workunit.client.1.smithi134.stdout:9/354: creat d0/d7/d23/d26/d51/d4c/f71 x:0 0 0 2024-04-22T23:23:45.248 INFO:tasks.workunit.client.1.smithi134.stdout:1/507: symlink db/d15/d17/d20/d2a/lb7 0 2024-04-22T23:23:45.250 INFO:tasks.workunit.client.1.smithi134.stdout:9/355: creat d0/d2d/d3d/f72 x:0 0 0 2024-04-22T23:23:45.251 INFO:tasks.workunit.client.1.smithi134.stdout:9/356: chown d0/d37/c6c 2492 1 2024-04-22T23:23:45.252 INFO:tasks.workunit.client.1.smithi134.stdout:8/320: getdents d7 0 2024-04-22T23:23:45.254 INFO:tasks.workunit.client.1.smithi134.stdout:6/445: link d0/d19/de/d12/c6e d0/d19/de/c9b 0 2024-04-22T23:23:45.257 INFO:tasks.workunit.client.1.smithi134.stdout:1/508: dwrite db/d15/d17/fa2 [0,4194304] 0 2024-04-22T23:23:45.259 INFO:tasks.workunit.client.1.smithi134.stdout:8/321: creat d7/dc/d2c/d37/f5f x:0 0 0 2024-04-22T23:23:45.266 INFO:tasks.workunit.client.1.smithi134.stdout:8/322: unlink d7/dc/d2c/d37/f5f 0 2024-04-22T23:23:45.267 INFO:tasks.workunit.client.1.smithi134.stdout:1/509: creat db/d16/fb8 x:0 0 0 2024-04-22T23:23:45.273 INFO:tasks.workunit.client.1.smithi134.stdout:1/510: rmdir db/d15/d17/d20/d2a/d2d/d2e/d42 39 2024-04-22T23:23:45.274 INFO:tasks.workunit.client.1.smithi134.stdout:3/293: dread dc/dd/d28/d3b/d29/f2e [0,4194304] 0 2024-04-22T23:23:45.274 INFO:tasks.workunit.client.1.smithi134.stdout:2/308: dread d2/dd/f37 [0,4194304] 0 2024-04-22T23:23:45.275 INFO:tasks.workunit.client.1.smithi134.stdout:9/357: dread d0/d7/fd [4194304,4194304] 0 2024-04-22T23:23:45.276 INFO:tasks.workunit.client.1.smithi134.stdout:6/446: dwrite d0/d19/f38 [0,4194304] 0 2024-04-22T23:23:45.277 INFO:tasks.workunit.client.1.smithi134.stdout:3/294: fsync dc/dd/d28/d34/f48 0 2024-04-22T23:23:45.277 INFO:tasks.workunit.client.1.smithi134.stdout:2/309: truncate d2/f3e 746369 0 2024-04-22T23:23:45.279 INFO:tasks.workunit.client.1.smithi134.stdout:3/295: truncate dc/dd/d28/f5c 638805 0 2024-04-22T23:23:45.280 INFO:tasks.workunit.client.1.smithi134.stdout:9/358: symlink d0/d2d/d4e/l73 0 2024-04-22T23:23:45.281 INFO:tasks.workunit.client.1.smithi134.stdout:8/323: write d7/f1c [146510,100641] 0 2024-04-22T23:23:45.283 INFO:tasks.workunit.client.1.smithi134.stdout:2/310: chown d2/dd/l1a 6709 1 2024-04-22T23:23:45.285 INFO:tasks.workunit.client.1.smithi134.stdout:2/311: chown d2/d15/d24/d3b 185414 1 2024-04-22T23:23:45.288 INFO:tasks.workunit.client.1.smithi134.stdout:8/324: unlink d7/dc/d2c/d37/d47/f51 0 2024-04-22T23:23:45.290 INFO:tasks.workunit.client.1.smithi134.stdout:3/296: mkdir dc/dd/d28/d34/d70 0 2024-04-22T23:23:45.291 INFO:tasks.workunit.client.1.smithi134.stdout:1/511: dwrite db/d15/d17/d20/d2a/d2d/f64 [0,4194304] 0 2024-04-22T23:23:45.293 INFO:tasks.workunit.client.1.smithi134.stdout:2/312: mkdir d2/dd/d68 0 2024-04-22T23:23:45.293 INFO:tasks.workunit.client.1.smithi134.stdout:4/545: sync 2024-04-22T23:23:45.293 INFO:tasks.workunit.client.1.smithi134.stdout:5/505: sync 2024-04-22T23:23:45.293 INFO:tasks.workunit.client.1.smithi134.stdout:3/297: readlink dc/dd/d28/d2f/l10 0 2024-04-22T23:23:45.295 INFO:tasks.workunit.client.1.smithi134.stdout:9/359: write d0/f54 [309684,59753] 0 2024-04-22T23:23:45.296 INFO:tasks.workunit.client.1.smithi134.stdout:1/512: rmdir db/d15 39 2024-04-22T23:23:45.299 INFO:tasks.workunit.client.1.smithi134.stdout:2/313: creat d2/d15/d24/f69 x:0 0 0 2024-04-22T23:23:45.301 INFO:tasks.workunit.client.1.smithi134.stdout:6/447: dwrite d0/d19/de/d12/d2a/f3b [0,4194304] 0 2024-04-22T23:23:45.303 INFO:tasks.workunit.client.1.smithi134.stdout:9/360: unlink d0/d2d/d4e/l73 0 2024-04-22T23:23:45.304 INFO:tasks.workunit.client.1.smithi134.stdout:2/314: write d2/f14 [1096709,82486] 0 2024-04-22T23:23:45.310 INFO:tasks.workunit.client.1.smithi134.stdout:8/325: dwrite d7/db/f1f [0,4194304] 0 2024-04-22T23:23:45.313 INFO:tasks.workunit.client.1.smithi134.stdout:2/315: dread - d2/d15/d24/f29 zero size 2024-04-22T23:23:45.315 INFO:tasks.workunit.client.1.smithi134.stdout:4/546: dwrite d4/d35/d5e/f22 [0,4194304] 0 2024-04-22T23:23:45.315 INFO:tasks.workunit.client.1.smithi134.stdout:6/448: creat d0/d19/de/d12/d2a/d2d/f9c x:0 0 0 2024-04-22T23:23:45.318 INFO:tasks.workunit.client.1.smithi134.stdout:5/506: dwrite d9/d11/f1a [0,4194304] 0 2024-04-22T23:23:45.320 INFO:tasks.workunit.client.1.smithi134.stdout:8/326: symlink d7/dc/l60 0 2024-04-22T23:23:45.325 INFO:tasks.workunit.client.1.smithi134.stdout:8/327: stat d7/l28 0 2024-04-22T23:23:45.325 INFO:tasks.workunit.client.1.smithi134.stdout:2/316: fsync d2/f2c 0 2024-04-22T23:23:45.325 INFO:tasks.workunit.client.1.smithi134.stdout:9/361: link d0/d7/c67 d0/d7/d23/d26/c74 0 2024-04-22T23:23:45.325 INFO:tasks.workunit.client.1.smithi134.stdout:4/547: mkdir d4/d35/db4 0 2024-04-22T23:23:45.326 INFO:tasks.workunit.client.1.smithi134.stdout:5/507: write d9/dd/d17/d2a/f8a [58981,116775] 0 2024-04-22T23:23:45.331 INFO:tasks.workunit.client.1.smithi134.stdout:2/317: symlink d2/d15/d24/d1d/d3d/l6a 0 2024-04-22T23:23:45.334 INFO:tasks.workunit.client.1.smithi134.stdout:1/513: dread db/d15/d17/d20/d2a/d2d/d2e/f39 [0,4194304] 0 2024-04-22T23:23:45.335 INFO:tasks.workunit.client.1.smithi134.stdout:5/508: creat d9/d1e/fb6 x:0 0 0 2024-04-22T23:23:45.336 INFO:tasks.workunit.client.1.smithi134.stdout:1/514: dread - db/d16/fb8 zero size 2024-04-22T23:23:45.336 INFO:tasks.workunit.client.1.smithi134.stdout:6/449: dwrite d0/d6f/f7b [0,4194304] 0 2024-04-22T23:23:45.336 INFO:tasks.workunit.client.1.smithi134.stdout:2/318: read d2/d15/d24/d1d/f44 [490017,88032] 0 2024-04-22T23:23:45.337 INFO:tasks.workunit.client.1.smithi134.stdout:4/548: read d4/d35/f62 [3335961,62218] 0 2024-04-22T23:23:45.338 INFO:tasks.workunit.client.1.smithi134.stdout:2/319: write d2/f3e [1475275,40548] 0 2024-04-22T23:23:45.339 INFO:tasks.workunit.client.1.smithi134.stdout:1/515: chown db/d15/d17/d20/d2a/d2d/f64 31 1 2024-04-22T23:23:45.340 INFO:tasks.workunit.client.1.smithi134.stdout:1/516: write db/d16/d56/fa1 [245681,44887] 0 2024-04-22T23:23:45.342 INFO:tasks.workunit.client.1.smithi134.stdout:5/509: unlink d9/d11/d73/d95/f6f 0 2024-04-22T23:23:45.343 INFO:tasks.workunit.client.1.smithi134.stdout:1/517: write db/f14 [2240266,67142] 0 2024-04-22T23:23:45.345 INFO:tasks.workunit.client.1.smithi134.stdout:2/320: symlink d2/dd/d4a/l6b 0 2024-04-22T23:23:45.346 INFO:tasks.workunit.client.1.smithi134.stdout:5/510: mkdir d9/dd/d88/db7 0 2024-04-22T23:23:45.348 INFO:tasks.workunit.client.1.smithi134.stdout:2/321: fsync d2/dd/df/f53 0 2024-04-22T23:23:45.349 INFO:tasks.workunit.client.1.smithi134.stdout:5/511: chown d9/d11/d73/d95/d20/f39 1 1 2024-04-22T23:23:45.349 INFO:tasks.workunit.client.1.smithi134.stdout:1/518: rmdir db/d16/d56 39 2024-04-22T23:23:45.353 INFO:tasks.workunit.client.1.smithi134.stdout:5/512: truncate d9/dd/d17/f66 168435 0 2024-04-22T23:23:45.354 INFO:tasks.workunit.client.1.smithi134.stdout:6/450: dwrite d0/d19/de/d1f/f22 [4194304,4194304] 0 2024-04-22T23:23:45.354 INFO:tasks.workunit.client.1.smithi134.stdout:2/322: symlink d2/dd/d4a/l6c 0 2024-04-22T23:23:45.356 INFO:tasks.workunit.client.1.smithi134.stdout:1/519: creat db/fb9 x:0 0 0 2024-04-22T23:23:45.359 INFO:tasks.workunit.client.1.smithi134.stdout:2/323: creat d2/d15/d24/f6d x:0 0 0 2024-04-22T23:23:45.361 INFO:tasks.workunit.client.1.smithi134.stdout:1/520: fsync db/d16/d56/f83 0 2024-04-22T23:23:45.362 INFO:tasks.workunit.client.1.smithi134.stdout:6/451: dwrite d0/d19/de/f1d [0,4194304] 0 2024-04-22T23:23:45.367 INFO:tasks.workunit.client.1.smithi134.stdout:6/452: rename d0/d19/db/f2b to d0/f9d 0 2024-04-22T23:23:45.368 INFO:tasks.workunit.client.1.smithi134.stdout:2/324: dwrite d2/f14 [0,4194304] 0 2024-04-22T23:23:45.369 INFO:tasks.workunit.client.1.smithi134.stdout:6/453: chown d0/d19/de/f5f 1584837043 1 2024-04-22T23:23:45.372 INFO:tasks.workunit.client.1.smithi134.stdout:1/521: link db/d15/d17/c2b db/d15/d72/cba 0 2024-04-22T23:23:45.373 INFO:tasks.workunit.client.1.smithi134.stdout:8/328: dread d7/dc/ff [0,4194304] 0 2024-04-22T23:23:45.374 INFO:tasks.workunit.client.1.smithi134.stdout:6/454: creat d0/d6f/f9e x:0 0 0 2024-04-22T23:23:45.385 INFO:tasks.workunit.client.1.smithi134.stdout:1/522: dwrite db/d16/d56/f63 [0,4194304] 0 2024-04-22T23:23:45.386 INFO:tasks.workunit.client.1.smithi134.stdout:9/362: dwrite d0/d7/d23/d1b/f64 [0,4194304] 0 2024-04-22T23:23:45.389 INFO:tasks.workunit.client.1.smithi134.stdout:1/523: dread - db/fb9 zero size 2024-04-22T23:23:45.390 INFO:tasks.workunit.client.1.smithi134.stdout:9/363: unlink d0/d7/d12/d22/f31 0 2024-04-22T23:23:45.393 INFO:tasks.workunit.client.1.smithi134.stdout:1/524: symlink db/d15/d17/d20/d2a/d2d/d2e/lbb 0 2024-04-22T23:23:45.397 INFO:tasks.workunit.client.1.smithi134.stdout:9/364: dread d0/d7/d23/d1b/f64 [0,4194304] 0 2024-04-22T23:23:45.398 INFO:tasks.workunit.client.1.smithi134.stdout:2/325: dread d2/dd/d4a/f52 [0,4194304] 0 2024-04-22T23:23:45.399 INFO:tasks.workunit.client.1.smithi134.stdout:1/525: mkdir db/d16/dbc 0 2024-04-22T23:23:45.399 INFO:tasks.workunit.client.1.smithi134.stdout:2/326: chown d2/d15/d24/f1e 5657955 1 2024-04-22T23:23:45.401 INFO:tasks.workunit.client.1.smithi134.stdout:2/327: mkdir d2/d42/d54/d33/d6e 0 2024-04-22T23:23:45.402 INFO:tasks.workunit.client.1.smithi134.stdout:6/455: write d0/d19/de/f13 [3748735,87714] 0 2024-04-22T23:23:45.403 INFO:tasks.workunit.client.1.smithi134.stdout:1/526: mkdir db/d16/d87/dbd 0 2024-04-22T23:23:45.405 INFO:tasks.workunit.client.1.smithi134.stdout:6/456: write d0/d19/de/d12/d2a/d2d/f85 [910989,15238] 0 2024-04-22T23:23:45.405 INFO:tasks.workunit.client.1.smithi134.stdout:9/365: creat d0/f75 x:0 0 0 2024-04-22T23:23:45.406 INFO:tasks.workunit.client.1.smithi134.stdout:6/457: readlink d0/l4 0 2024-04-22T23:23:45.406 INFO:tasks.workunit.client.1.smithi134.stdout:1/527: readlink db/d15/d17/d20/d2a/d2d/d2e/l52 0 2024-04-22T23:23:45.407 INFO:tasks.workunit.client.1.smithi134.stdout:6/458: read - d0/d19/db/d33/f62 zero size 2024-04-22T23:23:45.408 INFO:tasks.workunit.client.1.smithi134.stdout:2/328: getdents d2/d42 0 2024-04-22T23:23:45.410 INFO:tasks.workunit.client.1.smithi134.stdout:2/329: write d2/d15/d24/f69 [564030,65807] 0 2024-04-22T23:23:45.410 INFO:tasks.workunit.client.1.smithi134.stdout:6/459: creat d0/d19/de/d12/d5e/d6d/f9f x:0 0 0 2024-04-22T23:23:45.412 INFO:tasks.workunit.client.1.smithi134.stdout:6/460: chown d0/f16 103 1 2024-04-22T23:23:45.413 INFO:tasks.workunit.client.1.smithi134.stdout:2/330: write d2/dd/d4a/f64 [273294,42773] 0 2024-04-22T23:23:45.413 INFO:tasks.workunit.client.1.smithi134.stdout:1/528: getdents db/d15/d17/d20/d2a/d2d/d2e/d42 0 2024-04-22T23:23:45.416 INFO:tasks.workunit.client.1.smithi134.stdout:6/461: unlink d0/d19/db/c17 0 2024-04-22T23:23:45.417 INFO:tasks.workunit.client.1.smithi134.stdout:1/529: mknod db/d15/cbe 0 2024-04-22T23:23:45.418 INFO:tasks.workunit.client.1.smithi134.stdout:2/331: write d2/dd/d4a/f52 [1319930,24365] 0 2024-04-22T23:23:45.420 INFO:tasks.workunit.client.1.smithi134.stdout:1/530: truncate db/d16/f5b 31664 0 2024-04-22T23:23:45.421 INFO:tasks.workunit.client.1.smithi134.stdout:8/329: dread d7/db/d2d/d33/f4c [0,4194304] 0 2024-04-22T23:23:45.421 INFO:tasks.workunit.client.1.smithi134.stdout:6/462: mkdir d0/d19/de/d12/d29/d74/da0 0 2024-04-22T23:23:45.422 INFO:tasks.workunit.client.1.smithi134.stdout:2/332: write d2/d15/d24/f2a [4395049,52483] 0 2024-04-22T23:23:45.424 INFO:tasks.workunit.client.1.smithi134.stdout:2/333: readlink d2/dd/d4a/l66 0 2024-04-22T23:23:45.424 INFO:tasks.workunit.client.1.smithi134.stdout:9/366: dread d0/fa [0,4194304] 0 2024-04-22T23:23:45.426 INFO:tasks.workunit.client.1.smithi134.stdout:6/463: creat d0/d19/de/d12/d29/d74/fa1 x:0 0 0 2024-04-22T23:23:45.428 INFO:tasks.workunit.client.1.smithi134.stdout:9/367: fsync d0/d7/f6f 0 2024-04-22T23:23:45.430 INFO:tasks.workunit.client.1.smithi134.stdout:6/464: write d0/f9d [1458943,59789] 0 2024-04-22T23:23:45.431 INFO:tasks.workunit.client.1.smithi134.stdout:8/330: dwrite d7/f4a [0,4194304] 0 2024-04-22T23:23:45.435 INFO:tasks.workunit.client.1.smithi134.stdout:9/368: mkdir d0/d37/d76 0 2024-04-22T23:23:45.440 INFO:tasks.workunit.client.1.smithi134.stdout:8/331: dread - d7/dc/d2c/d37/d47/f5c zero size 2024-04-22T23:23:45.440 INFO:tasks.workunit.client.1.smithi134.stdout:9/369: mknod d0/d7/d23/d26/d4d/c77 0 2024-04-22T23:23:45.441 INFO:tasks.workunit.client.1.smithi134.stdout:2/334: dwrite d2/d15/d24/d1d/f5a [4194304,4194304] 0 2024-04-22T23:23:45.441 INFO:tasks.workunit.client.1.smithi134.stdout:8/332: chown d7/dc/d2c/d4f 132667273 1 2024-04-22T23:23:45.444 INFO:tasks.workunit.client.1.smithi134.stdout:0/407: sync 2024-04-22T23:23:45.447 INFO:tasks.workunit.client.1.smithi134.stdout:6/465: dwrite d0/d19/de/d12/d2a/d2d/f80 [0,4194304] 0 2024-04-22T23:23:45.456 INFO:tasks.workunit.client.1.smithi134.stdout:1/531: dread db/d15/d17/d20/d2a/d2d/d2e/f39 [0,4194304] 0 2024-04-22T23:23:45.457 INFO:tasks.workunit.client.1.smithi134.stdout:0/408: creat d1/d5/d58/f80 x:0 0 0 2024-04-22T23:23:45.458 INFO:tasks.workunit.client.1.smithi134.stdout:9/370: dwrite d0/d2d/d4e/f32 [4194304,4194304] 0 2024-04-22T23:23:45.461 INFO:tasks.workunit.client.1.smithi134.stdout:9/371: fsync d0/f57 0 2024-04-22T23:23:45.462 INFO:tasks.workunit.client.1.smithi134.stdout:8/333: rename d7/db/c4e to d7/db/d2d/d33/c61 0 2024-04-22T23:23:45.462 INFO:tasks.workunit.client.1.smithi134.stdout:8/334: readlink d7/dc/l60 0 2024-04-22T23:23:45.463 INFO:tasks.workunit.client.1.smithi134.stdout:9/372: chown d0/d7/d23/d26/d4d/d5d 254 1 2024-04-22T23:23:45.466 INFO:tasks.workunit.client.1.smithi134.stdout:0/409: mknod d1/d5/dd/d11/d75/c81 0 2024-04-22T23:23:45.467 INFO:tasks.workunit.client.1.smithi134.stdout:0/410: fdatasync d1/d5/dd/f16 0 2024-04-22T23:23:45.468 INFO:tasks.workunit.client.1.smithi134.stdout:8/335: fsync d7/f23 0 2024-04-22T23:23:45.473 INFO:tasks.workunit.client.1.smithi134.stdout:2/335: creat d2/dd/df/f6f x:0 0 0 2024-04-22T23:23:45.476 INFO:tasks.workunit.client.1.smithi134.stdout:0/411: fdatasync d1/d5/d58/f29 0 2024-04-22T23:23:45.477 INFO:tasks.workunit.client.1.smithi134.stdout:8/336: dread - d7/dc/d2c/d37/d47/f31 zero size 2024-04-22T23:23:45.480 INFO:tasks.workunit.client.1.smithi134.stdout:8/337: fsync d7/dc/d2c/d37/d47/f5c 0 2024-04-22T23:23:45.481 INFO:tasks.workunit.client.1.smithi134.stdout:6/466: truncate d0/d19/de/d12/d2a/d2d/d3a/f9a 738885 0 2024-04-22T23:23:45.481 INFO:tasks.workunit.client.1.smithi134.stdout:0/412: chown d1/d5/d58/f46 6604 1 2024-04-22T23:23:45.483 INFO:tasks.workunit.client.1.smithi134.stdout:2/336: dwrite d2/d15/d24/f2a [0,4194304] 0 2024-04-22T23:23:45.484 INFO:tasks.workunit.client.1.smithi134.stdout:8/338: symlink d7/db/l62 0 2024-04-22T23:23:45.486 INFO:tasks.workunit.client.1.smithi134.stdout:0/413: unlink d1/d5/d58/d6b/l71 0 2024-04-22T23:23:45.489 INFO:tasks.workunit.client.1.smithi134.stdout:2/337: truncate d2/d15/d24/d1d/d3d/f58 693777 0 2024-04-22T23:23:45.490 INFO:tasks.workunit.client.1.smithi134.stdout:6/467: rename d0/d19/f38 to d0/d6f/fa2 0 2024-04-22T23:23:45.490 INFO:tasks.workunit.client.1.smithi134.stdout:2/338: chown d2/d15/d24/f5d 1 1 2024-04-22T23:23:45.492 INFO:tasks.workunit.client.1.smithi134.stdout:0/414: mknod d1/d5/dd/d20/d2c/c82 0 2024-04-22T23:23:45.494 INFO:tasks.workunit.client.1.smithi134.stdout:8/339: mknod d7/db/d2d/d39/d58/c63 0 2024-04-22T23:23:45.496 INFO:tasks.workunit.client.1.smithi134.stdout:8/340: write d7/dc/d2c/d37/d47/f34 [4214114,61775] 0 2024-04-22T23:23:45.498 INFO:tasks.workunit.client.1.smithi134.stdout:2/339: dwrite d2/d15/d24/d1d/f5a [4194304,4194304] 0 2024-04-22T23:23:45.500 INFO:tasks.workunit.client.1.smithi134.stdout:2/340: symlink d2/d42/d54/l70 0 2024-04-22T23:23:45.501 INFO:tasks.workunit.client.1.smithi134.stdout:2/341: readlink d2/d15/d24/d1d/l4e 0 2024-04-22T23:23:45.501 INFO:tasks.workunit.client.1.smithi134.stdout:2/342: chown d2/dd/l1a 254 1 2024-04-22T23:23:45.501 INFO:tasks.workunit.client.1.smithi134.stdout:2/343: chown d2/d15/d24/f35 12 1 2024-04-22T23:23:45.503 INFO:tasks.workunit.client.1.smithi134.stdout:2/344: creat d2/d15/d24/d1d/d3d/f71 x:0 0 0 2024-04-22T23:23:45.506 INFO:tasks.workunit.client.1.smithi134.stdout:2/345: getdents d2/d15/d24 0 2024-04-22T23:23:45.507 INFO:tasks.workunit.client.1.smithi134.stdout:2/346: mkdir d2/dd/d72 0 2024-04-22T23:23:45.508 INFO:tasks.workunit.client.1.smithi134.stdout:2/347: truncate d2/d15/d24/f6d 947824 0 2024-04-22T23:23:45.509 INFO:tasks.workunit.client.1.smithi134.stdout:2/348: chown d2/f17 4791409 1 2024-04-22T23:23:45.509 INFO:tasks.workunit.client.1.smithi134.stdout:2/349: write d2/d42/d54/d33/f3a [1583191,58347] 0 2024-04-22T23:23:45.515 INFO:tasks.workunit.client.1.smithi134.stdout:2/350: dwrite d2/d15/d24/f6d [0,4194304] 0 2024-04-22T23:23:45.516 INFO:tasks.workunit.client.1.smithi134.stdout:2/351: mkdir d2/dd/d4a/d73 0 2024-04-22T23:23:45.517 INFO:tasks.workunit.client.1.smithi134.stdout:2/352: chown d2/d15/d24/f29 3235 1 2024-04-22T23:23:45.522 INFO:tasks.workunit.client.1.smithi134.stdout:2/353: dwrite d2/dd/df/f6f [0,4194304] 0 2024-04-22T23:23:45.535 INFO:tasks.workunit.client.1.smithi134.stdout:6/468: dread d0/d19/de/d12/d2a/d2d/f85 [0,4194304] 0 2024-04-22T23:23:45.536 INFO:tasks.workunit.client.1.smithi134.stdout:6/469: write d0/f16 [254276,36134] 0 2024-04-22T23:23:45.537 INFO:tasks.workunit.client.1.smithi134.stdout:6/470: read d0/d6f/f84 [210174,119702] 0 2024-04-22T23:23:45.543 INFO:tasks.workunit.client.1.smithi134.stdout:2/354: dread d2/f3e [0,4194304] 0 2024-04-22T23:23:45.544 INFO:tasks.workunit.client.1.smithi134.stdout:6/471: unlink d0/d19/de/d12/d2a/l45 0 2024-04-22T23:23:45.546 INFO:tasks.workunit.client.1.smithi134.stdout:8/341: mknod d7/db/d2d/d33/c64 0 2024-04-22T23:23:45.546 INFO:tasks.workunit.client.1.smithi134.stdout:2/355: mknod d2/dd/d72/c74 0 2024-04-22T23:23:45.554 INFO:tasks.workunit.client.1.smithi134.stdout:2/356: dwrite d2/d15/d24/f2a [4194304,4194304] 0 2024-04-22T23:23:45.554 INFO:tasks.workunit.client.1.smithi134.stdout:6/472: dwrite d0/d19/de/d1f/f2c [0,4194304] 0 2024-04-22T23:23:45.556 INFO:tasks.workunit.client.1.smithi134.stdout:6/473: chown d0/d19/de/d1f/d7d 15196777 1 2024-04-22T23:23:45.557 INFO:tasks.workunit.client.1.smithi134.stdout:2/357: write d2/d15/d24/d1d/d3d/f71 [968928,44210] 0 2024-04-22T23:23:45.560 INFO:tasks.workunit.client.1.smithi134.stdout:7/466: sync 2024-04-22T23:23:45.563 INFO:tasks.workunit.client.1.smithi134.stdout:1/532: truncate db/d15/d17/d20/f45 1504489 0 2024-04-22T23:23:45.566 INFO:tasks.workunit.client.1.smithi134.stdout:1/533: write db/f5f [1078990,22032] 0 2024-04-22T23:23:45.569 INFO:tasks.workunit.client.1.smithi134.stdout:7/467: write d4/d8/d2a/d42/d79/fa2 [284794,126887] 0 2024-04-22T23:23:45.574 INFO:tasks.workunit.client.1.smithi134.stdout:2/358: dwrite d2/d15/d24/d1d/d3d/f58 [0,4194304] 0 2024-04-22T23:23:45.575 INFO:tasks.workunit.client.1.smithi134.stdout:9/373: dwrite d0/f5c [0,4194304] 0 2024-04-22T23:23:45.579 INFO:tasks.workunit.client.1.smithi134.stdout:9/374: truncate d0/d7/d23/f45 4755129 0 2024-04-22T23:23:45.579 INFO:tasks.workunit.client.1.smithi134.stdout:7/468: creat d4/d4a/fae x:0 0 0 2024-04-22T23:23:45.581 INFO:tasks.workunit.client.1.smithi134.stdout:2/359: symlink d2/d42/d54/d33/l75 0 2024-04-22T23:23:45.582 INFO:tasks.workunit.client.1.smithi134.stdout:0/415: write d1/d5/dd/f40 [3427827,16569] 0 2024-04-22T23:23:45.586 INFO:tasks.workunit.client.1.smithi134.stdout:1/534: dwrite db/ff [0,4194304] 0 2024-04-22T23:23:45.587 INFO:tasks.workunit.client.1.smithi134.stdout:0/416: chown d1/d5/dd/d11/f6d 1929986767 1 2024-04-22T23:23:45.588 INFO:tasks.workunit.client.1.smithi134.stdout:0/417: dread - d1/d2/f54 zero size 2024-04-22T23:23:45.588 INFO:tasks.workunit.client.1.smithi134.stdout:7/469: symlink d4/d8/d2a/d42/d7d/laf 0 2024-04-22T23:23:45.592 INFO:tasks.workunit.client.1.smithi134.stdout:9/375: link d0/d7/d27/f59 d0/d2d/d4e/f78 0 2024-04-22T23:23:45.593 INFO:tasks.workunit.client.1.smithi134.stdout:2/360: creat d2/dd/df/d1c/f76 x:0 0 0 2024-04-22T23:23:45.595 INFO:tasks.workunit.client.1.smithi134.stdout:9/376: truncate d0/d7/f6f 1846186 0 2024-04-22T23:23:45.597 INFO:tasks.workunit.client.1.smithi134.stdout:0/418: mkdir d1/d5/dd/d11/d83 0 2024-04-22T23:23:45.601 INFO:tasks.workunit.client.1.smithi134.stdout:2/361: dwrite d2/dd/df/d1c/f21 [0,4194304] 0 2024-04-22T23:23:45.603 INFO:tasks.workunit.client.1.smithi134.stdout:8/342: rename d7/db/d2d/d33 to d7/db/d42/d65 0 2024-04-22T23:23:45.611 INFO:tasks.workunit.client.1.smithi134.stdout:8/343: creat d7/dc/d2c/d37/d47/f66 x:0 0 0 2024-04-22T23:23:45.613 INFO:tasks.workunit.client.1.smithi134.stdout:6/474: rename d0/d19/de/d12/d2a/l40 to d0/d19/de/d12/d5e/la3 0 2024-04-22T23:23:45.616 INFO:tasks.workunit.client.1.smithi134.stdout:8/344: fsync d7/db/d42/d65/f4c 0 2024-04-22T23:23:45.617 INFO:tasks.workunit.client.1.smithi134.stdout:6/475: symlink d0/d6f/la4 0 2024-04-22T23:23:45.618 INFO:tasks.workunit.client.1.smithi134.stdout:7/470: rename d4/d8/c7c to d4/d8/d3b/d83/cb0 0 2024-04-22T23:23:45.618 INFO:tasks.workunit.client.1.smithi134.stdout:6/476: write d0/d19/db/d35/f4b [717054,26265] 0 2024-04-22T23:23:45.619 INFO:tasks.workunit.client.1.smithi134.stdout:8/345: mknod d7/db/d2d/d39/c67 0 2024-04-22T23:23:45.622 INFO:tasks.workunit.client.1.smithi134.stdout:1/535: rename db/d15/d17/d20/d2a/d2d/cb2 to db/d16/d56/cbf 0 2024-04-22T23:23:45.625 INFO:tasks.workunit.client.1.smithi134.stdout:8/346: creat d7/dc/d2c/d37/f68 x:0 0 0 2024-04-22T23:23:45.626 INFO:tasks.workunit.client.1.smithi134.stdout:7/471: dwrite d4/d8/d2a/d42/d79/fa2 [0,4194304] 0 2024-04-22T23:23:45.626 INFO:tasks.workunit.client.1.smithi134.stdout:8/347: fsync f4 0 2024-04-22T23:23:45.628 INFO:tasks.workunit.client.1.smithi134.stdout:1/536: mknod db/d15/d17/d20/d2a/d2d/d86/dad/cc0 0 2024-04-22T23:23:45.631 INFO:tasks.workunit.client.1.smithi134.stdout:7/472: readlink d4/d8/l1e 0 2024-04-22T23:23:45.636 INFO:tasks.workunit.client.1.smithi134.stdout:8/348: dwrite d7/dc/d2c/d37/d47/f34 [0,4194304] 0 2024-04-22T23:23:45.636 INFO:tasks.workunit.client.1.smithi134.stdout:1/537: mkdir db/db0/dc1 0 2024-04-22T23:23:45.640 INFO:tasks.workunit.client.1.smithi134.stdout:7/473: dread d4/f13 [0,4194304] 0 2024-04-22T23:23:45.643 INFO:tasks.workunit.client.1.smithi134.stdout:8/349: mknod d7/c69 0 2024-04-22T23:23:45.644 INFO:tasks.workunit.client.1.smithi134.stdout:1/538: write db/d15/d17/d20/d2a/d2d/d2e/d42/f66 [904406,36560] 0 2024-04-22T23:23:45.645 INFO:tasks.workunit.client.1.smithi134.stdout:2/362: truncate d2/dd/d4a/f52 3549103 0 2024-04-22T23:23:45.646 INFO:tasks.workunit.client.1.smithi134.stdout:0/419: dwrite d1/d5/dd/d20/f1e [0,4194304] 0 2024-04-22T23:23:45.650 INFO:tasks.workunit.client.1.smithi134.stdout:7/474: creat d4/d8/d2a/fb1 x:0 0 0 2024-04-22T23:23:45.653 INFO:tasks.workunit.client.1.smithi134.stdout:6/477: dwrite d0/d19/de/f5f [0,4194304] 0 2024-04-22T23:23:45.669 INFO:tasks.workunit.client.1.smithi134.stdout:2/363: truncate d2/f14 4046947 0 2024-04-22T23:23:45.671 INFO:tasks.workunit.client.1.smithi134.stdout:8/350: dwrite f6 [4194304,4194304] 0 2024-04-22T23:23:45.679 INFO:tasks.workunit.client.1.smithi134.stdout:6/478: write d0/d19/de/f31 [4617046,91673] 0 2024-04-22T23:23:45.680 INFO:tasks.workunit.client.1.smithi134.stdout:8/351: symlink d7/db/d42/d65/l6a 0 2024-04-22T23:23:45.682 INFO:tasks.workunit.client.1.smithi134.stdout:1/539: dread db/fa6 [0,4194304] 0 2024-04-22T23:23:45.684 INFO:tasks.workunit.client.1.smithi134.stdout:7/475: dread d4/d8/ff [0,4194304] 0 2024-04-22T23:23:45.684 INFO:tasks.workunit.client.1.smithi134.stdout:1/540: mkdir db/db0/dc2 0 2024-04-22T23:23:45.685 INFO:tasks.workunit.client.1.smithi134.stdout:6/479: getdents d0/d19/de/d12 0 2024-04-22T23:23:45.686 INFO:tasks.workunit.client.1.smithi134.stdout:7/476: fdatasync d4/d10/d2f/d69/d8a/f99 0 2024-04-22T23:23:45.687 INFO:tasks.workunit.client.1.smithi134.stdout:8/352: rename d7/dc/d2c/f49 to d7/dc/d2c/d4f/f6b 0 2024-04-22T23:23:45.687 INFO:tasks.workunit.client.1.smithi134.stdout:6/480: truncate d0/d19/db/d35/f4b 1645284 0 2024-04-22T23:23:45.688 INFO:tasks.workunit.client.1.smithi134.stdout:1/541: creat db/d15/d17/d20/d2a/fc3 x:0 0 0 2024-04-22T23:23:45.690 INFO:tasks.workunit.client.1.smithi134.stdout:7/477: dread d4/d10/d2f/d34/d62/f71 [0,4194304] 0 2024-04-22T23:23:45.692 INFO:tasks.workunit.client.1.smithi134.stdout:7/478: readlink d4/d10/l40 0 2024-04-22T23:23:45.694 INFO:tasks.workunit.client.1.smithi134.stdout:8/353: chown d7/db/d42/d65/d46/c48 614349 1 2024-04-22T23:23:45.695 INFO:tasks.workunit.client.1.smithi134.stdout:6/481: dwrite d0/d19/de/d12/d29/f70 [0,4194304] 0 2024-04-22T23:23:45.695 INFO:tasks.workunit.client.1.smithi134.stdout:7/479: creat d4/d4a/d85/fb2 x:0 0 0 2024-04-22T23:23:45.697 INFO:tasks.workunit.client.1.smithi134.stdout:6/482: write d0/d19/de/d12/d5e/f66 [359666,100830] 0 2024-04-22T23:23:45.698 INFO:tasks.workunit.client.1.smithi134.stdout:1/542: mknod db/d15/d17/cc4 0 2024-04-22T23:23:45.700 INFO:tasks.workunit.client.1.smithi134.stdout:7/480: symlink d4/d4a/lb3 0 2024-04-22T23:23:45.701 INFO:tasks.workunit.client.1.smithi134.stdout:8/354: rmdir d7/dc/d19 39 2024-04-22T23:23:45.703 INFO:tasks.workunit.client.1.smithi134.stdout:1/543: symlink db/db0/lc5 0 2024-04-22T23:23:45.705 INFO:tasks.workunit.client.1.smithi134.stdout:7/481: rmdir d4/d10/d2f/d70 39 2024-04-22T23:23:45.706 INFO:tasks.workunit.client.1.smithi134.stdout:8/355: mkdir d7/dc/d2c/d4f/d6c 0 2024-04-22T23:23:45.707 INFO:tasks.workunit.client.1.smithi134.stdout:8/356: chown d7/dc/d2c/l2f 5 1 2024-04-22T23:23:45.708 INFO:tasks.workunit.client.1.smithi134.stdout:8/357: stat d7/db/d2d/d39/f45 0 2024-04-22T23:23:45.710 INFO:tasks.workunit.client.1.smithi134.stdout:1/544: truncate db/d15/d17/f5a 2792556 0 2024-04-22T23:23:45.711 INFO:tasks.workunit.client.1.smithi134.stdout:8/358: symlink d7/db/d2d/d39/d58/l6d 0 2024-04-22T23:23:45.713 INFO:tasks.workunit.client.1.smithi134.stdout:7/482: rename d4/d10/d2f/d34/d54/fa3 to d4/d10/d2f/d34/d54/fb4 0 2024-04-22T23:23:45.715 INFO:tasks.workunit.client.1.smithi134.stdout:8/359: fsync d7/f1c 0 2024-04-22T23:23:45.715 INFO:tasks.workunit.client.1.smithi134.stdout:7/483: truncate d4/f2d 4748941 0 2024-04-22T23:23:45.717 INFO:tasks.workunit.client.1.smithi134.stdout:8/360: write d7/dc/d2c/d37/d47/f34 [4920614,107462] 0 2024-04-22T23:23:45.718 INFO:tasks.workunit.client.1.smithi134.stdout:1/545: truncate db/d15/f54 838303 0 2024-04-22T23:23:45.719 INFO:tasks.workunit.client.1.smithi134.stdout:1/546: dread - db/fb9 zero size 2024-04-22T23:23:45.719 INFO:tasks.workunit.client.1.smithi134.stdout:7/484: symlink d4/d8/d41/d5b/lb5 0 2024-04-22T23:23:45.721 INFO:tasks.workunit.client.1.smithi134.stdout:7/485: chown d4/d10/d2f/d34/d62/d29 96791790 1 2024-04-22T23:23:45.722 INFO:tasks.workunit.client.1.smithi134.stdout:1/547: read db/d15/d17/d20/d2a/f36 [2759715,87883] 0 2024-04-22T23:23:45.723 INFO:tasks.workunit.client.1.smithi134.stdout:6/483: dread d0/d19/de/d12/d29/d74/f89 [0,4194304] 0 2024-04-22T23:23:45.725 INFO:tasks.workunit.client.1.smithi134.stdout:6/484: dread - d0/d19/de/d12/d29/d74/fa1 zero size 2024-04-22T23:23:45.727 INFO:tasks.workunit.client.1.smithi134.stdout:1/548: read db/d15/f34 [1400789,29585] 0 2024-04-22T23:23:45.728 INFO:tasks.workunit.client.1.smithi134.stdout:1/549: truncate db/f8b 530590 0 2024-04-22T23:23:45.728 INFO:tasks.workunit.client.1.smithi134.stdout:7/486: rename d4/d8/c81 to d4/d8/d3b/cb6 0 2024-04-22T23:23:45.729 INFO:tasks.workunit.client.1.smithi134.stdout:1/550: dread - db/d15/d17/d20/d2a/f88 zero size 2024-04-22T23:23:45.729 INFO:tasks.workunit.client.1.smithi134.stdout:6/485: symlink d0/d19/db/la5 0 2024-04-22T23:23:45.731 INFO:tasks.workunit.client.1.smithi134.stdout:7/487: write d4/d10/d2f/d34/d62/d84/f8f [847464,16427] 0 2024-04-22T23:23:45.731 INFO:tasks.workunit.client.1.smithi134.stdout:6/486: fsync d0/d19/db/d35/f4b 0 2024-04-22T23:23:45.733 INFO:tasks.workunit.client.1.smithi134.stdout:1/551: stat db/d15/d17/d20/d2a/d2d/l58 0 2024-04-22T23:23:45.733 INFO:tasks.workunit.client.1.smithi134.stdout:8/361: write d7/dc/f22 [1110468,15917] 0 2024-04-22T23:23:45.734 INFO:tasks.workunit.client.1.smithi134.stdout:8/362: chown d7/db/c17 420065774 1 2024-04-22T23:23:45.735 INFO:tasks.workunit.client.1.smithi134.stdout:8/363: fsync d7/db/f3b 0 2024-04-22T23:23:45.736 INFO:tasks.workunit.client.1.smithi134.stdout:7/488: creat d4/d10/d2f/d69/d8a/fb7 x:0 0 0 2024-04-22T23:23:45.738 INFO:tasks.workunit.client.1.smithi134.stdout:8/364: mkdir d7/dc/d2c/d4f/d6c/d6e 0 2024-04-22T23:23:45.739 INFO:tasks.workunit.client.1.smithi134.stdout:7/489: fdatasync d4/d10/d2f/d34/d62/f71 0 2024-04-22T23:23:45.740 INFO:tasks.workunit.client.1.smithi134.stdout:6/487: dread d0/d19/de/d1f/f22 [4194304,4194304] 0 2024-04-22T23:23:45.741 INFO:tasks.workunit.client.1.smithi134.stdout:1/552: creat db/d15/fc6 x:0 0 0 2024-04-22T23:23:45.741 INFO:tasks.workunit.client.1.smithi134.stdout:7/490: write d4/d10/d2f/d69/f9a [436564,54792] 0 2024-04-22T23:23:45.744 INFO:tasks.workunit.client.1.smithi134.stdout:8/365: creat d7/dc/d2c/d4f/d6c/d6e/f6f x:0 0 0 2024-04-22T23:23:45.746 INFO:tasks.workunit.client.1.smithi134.stdout:1/553: chown db/d15/d17/l79 3540916 1 2024-04-22T23:23:45.749 INFO:tasks.workunit.client.1.smithi134.stdout:8/366: creat d7/db/d2d/d39/d58/f70 x:0 0 0 2024-04-22T23:23:45.753 INFO:tasks.workunit.client.1.smithi134.stdout:8/367: creat d7/db/d2d/d39/d58/f71 x:0 0 0 2024-04-22T23:23:45.755 INFO:tasks.workunit.client.1.smithi134.stdout:7/491: dwrite d4/d8/d41/f43 [0,4194304] 0 2024-04-22T23:23:45.756 INFO:tasks.workunit.client.1.smithi134.stdout:1/554: dwrite db/f26 [0,4194304] 0 2024-04-22T23:23:45.757 INFO:tasks.workunit.client.1.smithi134.stdout:7/492: chown d4/d8/d3b/c8b 62973039 1 2024-04-22T23:23:45.757 INFO:tasks.workunit.client.1.smithi134.stdout:8/368: read d7/dc/f2b [388509,54798] 0 2024-04-22T23:23:45.759 INFO:tasks.workunit.client.1.smithi134.stdout:7/493: chown d4/d8/d41/d5b/c8e 0 1 2024-04-22T23:23:45.763 INFO:tasks.workunit.client.1.smithi134.stdout:8/369: dread d7/db/d2d/d39/f4b [0,4194304] 0 2024-04-22T23:23:45.763 INFO:tasks.workunit.client.1.smithi134.stdout:6/488: dread d0/d19/de/d12/d2a/f4c [0,4194304] 0 2024-04-22T23:23:45.766 INFO:tasks.workunit.client.1.smithi134.stdout:6/489: symlink d0/la6 0 2024-04-22T23:23:45.767 INFO:tasks.workunit.client.1.smithi134.stdout:8/370: creat d7/db/d2d/d39/f72 x:0 0 0 2024-04-22T23:23:45.768 INFO:tasks.workunit.client.1.smithi134.stdout:1/555: truncate db/d16/f78 1797406 0 2024-04-22T23:23:45.771 INFO:tasks.workunit.client.1.smithi134.stdout:1/556: write db/d16/f3d [4569565,79688] 0 2024-04-22T23:23:45.771 INFO:tasks.workunit.client.1.smithi134.stdout:8/371: mknod d7/dc/d2c/d4f/d6c/d6e/c73 0 2024-04-22T23:23:45.775 INFO:tasks.workunit.client.1.smithi134.stdout:7/494: dwrite d4/f31 [0,4194304] 0 2024-04-22T23:23:45.777 INFO:tasks.workunit.client.1.smithi134.stdout:8/372: unlink d7/dc/d2c/d37/f68 0 2024-04-22T23:23:45.778 INFO:tasks.workunit.client.1.smithi134.stdout:1/557: creat db/db5/fc7 x:0 0 0 2024-04-22T23:23:45.779 INFO:tasks.workunit.client.1.smithi134.stdout:8/373: write d7/dc/d2c/d37/f3f [5123597,72962] 0 2024-04-22T23:23:45.783 INFO:tasks.workunit.client.1.smithi134.stdout:7/495: creat d4/d10/fb8 x:0 0 0 2024-04-22T23:23:45.789 INFO:tasks.workunit.client.1.smithi134.stdout:1/558: dwrite db/d16/f9e [0,4194304] 0 2024-04-22T23:23:45.789 INFO:tasks.workunit.client.1.smithi134.stdout:7/496: rename d4/d10/d2f/d70/d86 to d4/d10/d2f/d70/d86/d97/db9 22 2024-04-22T23:23:45.790 INFO:tasks.workunit.client.1.smithi134.stdout:1/559: write db/d16/f5b [701188,44801] 0 2024-04-22T23:23:45.792 INFO:tasks.workunit.client.1.smithi134.stdout:1/560: fdatasync db/d16/f3d 0 2024-04-22T23:23:45.796 INFO:tasks.workunit.client.1.smithi134.stdout:7/497: mknod d4/d10/cba 0 2024-04-22T23:23:45.796 INFO:tasks.workunit.client.1.smithi134.stdout:6/490: dread d0/f8 [0,4194304] 0 2024-04-22T23:23:45.799 INFO:tasks.workunit.client.1.smithi134.stdout:1/561: unlink db/d16/d56/c5c 0 2024-04-22T23:23:45.801 INFO:tasks.workunit.client.1.smithi134.stdout:6/491: unlink d0/d19/de/d2f/f43 0 2024-04-22T23:23:45.801 INFO:tasks.workunit.client.1.smithi134.stdout:1/562: write db/ff [4893813,50625] 0 2024-04-22T23:23:45.802 INFO:tasks.workunit.client.1.smithi134.stdout:7/498: symlink d4/d8/d41/d76/lbb 0 2024-04-22T23:23:45.803 INFO:tasks.workunit.client.1.smithi134.stdout:6/492: write d0/d6f/f9e [407140,70313] 0 2024-04-22T23:23:45.803 INFO:tasks.workunit.client.1.smithi134.stdout:8/374: truncate d7/db/d2d/f54 411496 0 2024-04-22T23:23:45.805 INFO:tasks.workunit.client.1.smithi134.stdout:6/493: write d0/d19/de/d12/d29/d74/fa1 [318166,128228] 0 2024-04-22T23:23:45.806 INFO:tasks.workunit.client.1.smithi134.stdout:8/375: truncate d7/dc/d2c/d4f/f6b 4417999 0 2024-04-22T23:23:45.806 INFO:tasks.workunit.client.1.smithi134.stdout:6/494: chown d0/d19/f9 3 1 2024-04-22T23:23:45.808 INFO:tasks.workunit.client.1.smithi134.stdout:3/298: sync 2024-04-22T23:23:45.808 INFO:tasks.workunit.client.1.smithi134.stdout:8/376: dread - d7/dc/d2c/d37/d47/f5c zero size 2024-04-22T23:23:45.810 INFO:tasks.workunit.client.1.smithi134.stdout:3/299: write dc/dd/d28/f39 [662175,129599] 0 2024-04-22T23:23:45.813 INFO:tasks.workunit.client.1.smithi134.stdout:1/563: creat db/d15/d17/fc8 x:0 0 0 2024-04-22T23:23:45.814 INFO:tasks.workunit.client.1.smithi134.stdout:6/495: mkdir d0/d19/de/d12/d2a/d2d/d3a/da7 0 2024-04-22T23:23:45.814 INFO:tasks.workunit.client.1.smithi134.stdout:7/499: link l2 d4/d10/d2f/d69/d8a/da8/lbc 0 2024-04-22T23:23:45.816 INFO:tasks.workunit.client.1.smithi134.stdout:6/496: chown d0/d19/f9 1018341 1 2024-04-22T23:23:45.818 INFO:tasks.workunit.client.1.smithi134.stdout:6/497: stat d0/d19/de/d12/d2a/d2d/d3a 0 2024-04-22T23:23:45.819 INFO:tasks.workunit.client.1.smithi134.stdout:8/377: mknod d7/db/d2d/c74 0 2024-04-22T23:23:45.823 INFO:tasks.workunit.client.1.smithi134.stdout:3/300: link dc/dd/c25 dc/dd/d28/c71 0 2024-04-22T23:23:45.828 INFO:tasks.workunit.client.1.smithi134.stdout:8/378: dread d7/db/d2d/f2e [0,4194304] 0 2024-04-22T23:23:45.828 INFO:tasks.workunit.client.1.smithi134.stdout:7/500: mkdir d4/dbd 0 2024-04-22T23:23:45.828 INFO:tasks.workunit.client.1.smithi134.stdout:8/379: chown f4 1003 1 2024-04-22T23:23:45.830 INFO:tasks.workunit.client.1.smithi134.stdout:1/564: dwrite db/d15/f29 [4194304,4194304] 0 2024-04-22T23:23:45.832 INFO:tasks.workunit.client.1.smithi134.stdout:8/380: chown d7/db/d42/d65/d46 421373 1 2024-04-22T23:23:45.832 INFO:tasks.workunit.client.1.smithi134.stdout:8/381: stat d7/dc/f22 0 2024-04-22T23:23:45.838 INFO:tasks.workunit.client.1.smithi134.stdout:8/382: mknod d7/dc/d2c/d4f/d6c/c75 0 2024-04-22T23:23:45.838 INFO:tasks.workunit.client.1.smithi134.stdout:1/565: dwrite db/f1d [0,4194304] 0 2024-04-22T23:23:45.840 INFO:tasks.workunit.client.1.smithi134.stdout:1/566: dread - db/d16/d87/faf zero size 2024-04-22T23:23:45.842 INFO:tasks.workunit.client.1.smithi134.stdout:8/383: truncate d7/db/d2d/d39/f4b 916518 0 2024-04-22T23:23:45.844 INFO:tasks.workunit.client.1.smithi134.stdout:8/384: write d7/dc/d2c/d37/d47/f5c [413965,107018] 0 2024-04-22T23:23:45.845 INFO:tasks.workunit.client.1.smithi134.stdout:1/567: mkdir db/d15/d17/dc9 0 2024-04-22T23:23:45.847 INFO:tasks.workunit.client.1.smithi134.stdout:8/385: creat d7/db/d42/f76 x:0 0 0 2024-04-22T23:23:45.854 INFO:tasks.workunit.client.1.smithi134.stdout:1/568: dwrite db/d15/d17/d20/f46 [0,4194304] 0 2024-04-22T23:23:45.859 INFO:tasks.workunit.client.1.smithi134.stdout:1/569: stat db/d15/d17/d20/l4e 0 2024-04-22T23:23:45.860 INFO:tasks.workunit.client.1.smithi134.stdout:1/570: chown db/f27 31750 1 2024-04-22T23:23:45.862 INFO:tasks.workunit.client.1.smithi134.stdout:1/571: dread - db/d15/d17/d20/d2a/fc3 zero size 2024-04-22T23:23:45.869 INFO:tasks.workunit.client.1.smithi134.stdout:1/572: getdents db/d16 0 2024-04-22T23:23:45.874 INFO:tasks.workunit.client.1.smithi134.stdout:1/573: getdents db/d16/d87 0 2024-04-22T23:23:45.875 INFO:tasks.workunit.client.1.smithi134.stdout:6/498: truncate d0/f55 3549581 0 2024-04-22T23:23:45.877 INFO:tasks.workunit.client.1.smithi134.stdout:8/386: rmdir d7/db/d2d/d39 39 2024-04-22T23:23:45.882 INFO:tasks.workunit.client.1.smithi134.stdout:8/387: read d7/db/d2d/d39/f3d [76558,52197] 0 2024-04-22T23:23:45.885 INFO:tasks.workunit.client.1.smithi134.stdout:1/574: dwrite db/d15/d17/d20/d2a/fb3 [0,4194304] 0 2024-04-22T23:23:45.887 INFO:tasks.workunit.client.1.smithi134.stdout:1/575: write db/d15/d17/d20/f65 [1388682,58956] 0 2024-04-22T23:23:45.888 INFO:tasks.workunit.client.1.smithi134.stdout:8/388: mkdir d7/dc/d2c/d77 0 2024-04-22T23:23:45.892 INFO:tasks.workunit.client.1.smithi134.stdout:8/389: chown d7/dc/d2c/d37/d47/c55 79560 1 2024-04-22T23:23:45.894 INFO:tasks.workunit.client.1.smithi134.stdout:6/499: truncate d0/d6f/f9e 284392 0 2024-04-22T23:23:45.896 INFO:tasks.workunit.client.1.smithi134.stdout:1/576: rename db/d15/d17/d20/l4e to db/db0/lca 0 2024-04-22T23:23:45.899 INFO:tasks.workunit.client.1.smithi134.stdout:6/500: chown d0/d19/de/f1d 6063 1 2024-04-22T23:23:45.900 INFO:tasks.workunit.client.1.smithi134.stdout:8/390: unlink d7/dc/d2c/d4f/c5b 0 2024-04-22T23:23:45.906 INFO:tasks.workunit.client.1.smithi134.stdout:1/577: dwrite db/d15/d17/f9d [0,4194304] 0 2024-04-22T23:23:45.908 INFO:tasks.workunit.client.1.smithi134.stdout:8/391: symlink d7/dc/d19/l78 0 2024-04-22T23:23:45.912 INFO:tasks.workunit.client.1.smithi134.stdout:8/392: chown d7/db/d42/c52 63678220 1 2024-04-22T23:23:45.916 INFO:tasks.workunit.client.1.smithi134.stdout:1/578: rename db/ce to db/d15/d17/d20/d2a/d2d/d2e/d42/ccb 0 2024-04-22T23:23:45.923 INFO:tasks.workunit.client.1.smithi134.stdout:6/501: dread d0/f16 [0,4194304] 0 2024-04-22T23:23:45.925 INFO:tasks.workunit.client.1.smithi134.stdout:6/502: dread d0/d6f/f84 [0,4194304] 0 2024-04-22T23:23:45.926 INFO:tasks.workunit.client.1.smithi134.stdout:8/393: write d7/db/d2d/d39/f3d [990389,29122] 0 2024-04-22T23:23:45.928 INFO:tasks.workunit.client.1.smithi134.stdout:8/394: chown d7/dc/d19/l78 7926444 1 2024-04-22T23:23:45.929 INFO:tasks.workunit.client.1.smithi134.stdout:1/579: rename db/d15/d17/d20/d2a/d2d/d2e/d42/ccb to db/d15/ccc 0 2024-04-22T23:23:45.931 INFO:tasks.workunit.client.1.smithi134.stdout:8/395: mknod d7/db/d2d/d39/d58/c79 0 2024-04-22T23:23:45.932 INFO:tasks.workunit.client.1.smithi134.stdout:1/580: read db/d16/f78 [672399,92052] 0 2024-04-22T23:23:45.932 INFO:tasks.workunit.client.1.smithi134.stdout:6/503: dwrite d0/d6f/fa2 [0,4194304] 0 2024-04-22T23:23:45.933 INFO:tasks.workunit.client.1.smithi134.stdout:8/396: rmdir d7/dc/d19 39 2024-04-22T23:23:45.935 INFO:tasks.workunit.client.1.smithi134.stdout:6/504: symlink d0/d19/db/la8 0 2024-04-22T23:23:45.936 INFO:tasks.workunit.client.1.smithi134.stdout:6/505: fsync d0/d19/de/d12/d5e/d6d/f9f 0 2024-04-22T23:23:45.936 INFO:tasks.workunit.client.1.smithi134.stdout:8/397: rename d7/c69 to d7/dc/d2c/d37/d47/c7a 0 2024-04-22T23:23:45.937 INFO:tasks.workunit.client.1.smithi134.stdout:6/506: symlink d0/d19/la9 0 2024-04-22T23:23:45.940 INFO:tasks.workunit.client.1.smithi134.stdout:8/398: unlink d7/db/d2d/d39/f4b 0 2024-04-22T23:23:45.942 INFO:tasks.workunit.client.1.smithi134.stdout:8/399: dread d7/db/d42/d65/f4c [4194304,4194304] 0 2024-04-22T23:23:45.944 INFO:tasks.workunit.client.1.smithi134.stdout:8/400: truncate d7/dc/f22 1886549 0 2024-04-22T23:23:45.946 INFO:tasks.workunit.client.1.smithi134.stdout:6/507: dwrite d0/d19/de/f13 [0,4194304] 0 2024-04-22T23:23:45.949 INFO:tasks.workunit.client.1.smithi134.stdout:6/508: mknod d0/d19/de/d2f/caa 0 2024-04-22T23:23:45.952 INFO:tasks.workunit.client.1.smithi134.stdout:6/509: creat d0/d19/de/d12/d29/d74/da0/fab x:0 0 0 2024-04-22T23:23:45.959 INFO:tasks.workunit.client.1.smithi134.stdout:8/401: truncate d7/dc/d2c/d37/f3f 4757742 0 2024-04-22T23:23:45.959 INFO:tasks.workunit.client.1.smithi134.stdout:8/402: fsync d7/db/d2d/d39/f72 0 2024-04-22T23:23:45.962 INFO:tasks.workunit.client.1.smithi134.stdout:8/403: mkdir d7/db/d2d/d39/d58/d7b 0 2024-04-22T23:23:45.962 INFO:tasks.workunit.client.1.smithi134.stdout:6/510: truncate d0/f9d 623033 0 2024-04-22T23:23:45.968 INFO:tasks.workunit.client.1.smithi134.stdout:8/404: dwrite d7/dc/f12 [0,4194304] 0 2024-04-22T23:23:45.969 INFO:tasks.workunit.client.1.smithi134.stdout:6/511: dwrite d0/d19/de/d12/f7e [0,4194304] 0 2024-04-22T23:23:45.971 INFO:tasks.workunit.client.1.smithi134.stdout:6/512: write d0/d19/de/d12/d29/d74/f8e [772592,52779] 0 2024-04-22T23:23:45.975 INFO:tasks.workunit.client.1.smithi134.stdout:8/405: mknod d7/dc/d2c/d4f/c7c 0 2024-04-22T23:23:45.979 INFO:tasks.workunit.client.1.smithi134.stdout:8/406: symlink d7/db/l7d 0 2024-04-22T23:23:45.980 INFO:tasks.workunit.client.1.smithi134.stdout:8/407: chown d7/f1c 0 1 2024-04-22T23:23:45.994 INFO:tasks.workunit.client.1.smithi134.stdout:8/408: write d7/dc/d2c/d37/d47/f31 [305857,43921] 0 2024-04-22T23:23:45.994 INFO:tasks.workunit.client.1.smithi134.stdout:5/513: sync 2024-04-22T23:23:45.995 INFO:tasks.workunit.client.1.smithi134.stdout:4/549: sync 2024-04-22T23:23:45.995 INFO:tasks.workunit.client.1.smithi134.stdout:5/514: dread - d9/dd/d52/f79 zero size 2024-04-22T23:23:45.997 INFO:tasks.workunit.client.1.smithi134.stdout:4/550: dread d4/d15/f6a [0,4194304] 0 2024-04-22T23:23:45.998 INFO:tasks.workunit.client.1.smithi134.stdout:8/409: creat d7/db/d42/d65/f7e x:0 0 0 2024-04-22T23:23:45.998 INFO:tasks.workunit.client.1.smithi134.stdout:5/515: symlink d9/dd/d88/lb8 0 2024-04-22T23:23:46.000 INFO:tasks.workunit.client.1.smithi134.stdout:4/551: mkdir d4/d7/d32/d3a/d93/db5 0 2024-04-22T23:23:46.000 INFO:tasks.workunit.client.1.smithi134.stdout:5/516: truncate d9/dd/d17/d2a/d49/fad 914136 0 2024-04-22T23:23:46.002 INFO:tasks.workunit.client.1.smithi134.stdout:5/517: symlink d9/dd/da6/lb9 0 2024-04-22T23:23:46.004 INFO:tasks.workunit.client.1.smithi134.stdout:4/552: rename d4/d35/d8e/fa3 to d4/d35/d8e/d6e/fb6 0 2024-04-22T23:23:46.009 INFO:tasks.workunit.client.1.smithi134.stdout:5/518: rename d9/dd/da6/lb9 to d9/d1e/d90/lba 0 2024-04-22T23:23:46.010 INFO:tasks.workunit.client.1.smithi134.stdout:5/519: truncate d9/d11/f5d 1198843 0 2024-04-22T23:23:46.011 INFO:tasks.workunit.client.1.smithi134.stdout:8/410: dread f4 [4194304,4194304] 0 2024-04-22T23:23:46.011 INFO:tasks.workunit.client.1.smithi134.stdout:5/520: chown d9/d11/f1a 2 1 2024-04-22T23:23:46.012 INFO:tasks.workunit.client.1.smithi134.stdout:8/411: readlink d7/dc/d2c/d37/d47/l50 0 2024-04-22T23:23:46.012 INFO:tasks.workunit.client.1.smithi134.stdout:5/521: write d9/d11/d73/f8c [171757,7878] 0 2024-04-22T23:23:46.017 INFO:tasks.workunit.client.1.smithi134.stdout:4/553: dwrite d4/d15/d64/f69 [0,4194304] 0 2024-04-22T23:23:46.020 INFO:tasks.workunit.client.1.smithi134.stdout:4/554: symlink d4/d13/lb7 0 2024-04-22T23:23:46.025 INFO:tasks.workunit.client.1.smithi134.stdout:4/555: link d4/d35/d53/fa4 d4/d35/d53/fb8 0 2024-04-22T23:23:46.027 INFO:tasks.workunit.client.1.smithi134.stdout:4/556: rmdir d4/d35/d8e/d5d 39 2024-04-22T23:23:46.030 INFO:tasks.workunit.client.1.smithi134.stdout:5/522: dread d9/d11/f1b [0,4194304] 0 2024-04-22T23:23:46.032 INFO:tasks.workunit.client.1.smithi134.stdout:8/412: dread d7/dc/d2c/d37/f3f [0,4194304] 0 2024-04-22T23:23:46.033 INFO:tasks.workunit.client.1.smithi134.stdout:5/523: mkdir d9/dbb 0 2024-04-22T23:23:46.035 INFO:tasks.workunit.client.1.smithi134.stdout:4/557: dwrite d4/d35/d5e/f22 [0,4194304] 0 2024-04-22T23:23:46.038 INFO:tasks.workunit.client.1.smithi134.stdout:8/413: rename d7/f4a to d7/dc/f7f 0 2024-04-22T23:23:46.039 INFO:tasks.workunit.client.1.smithi134.stdout:8/414: dread - d7/db/d2d/d39/f72 zero size 2024-04-22T23:23:46.042 INFO:tasks.workunit.client.1.smithi134.stdout:8/415: write d7/db/d2d/d39/f45 [478442,98623] 0 2024-04-22T23:23:46.043 INFO:tasks.workunit.client.1.smithi134.stdout:4/558: mknod d4/d35/d8e/d6e/cb9 0 2024-04-22T23:23:46.047 INFO:tasks.workunit.client.1.smithi134.stdout:4/559: symlink d4/d6b/lba 0 2024-04-22T23:23:46.048 INFO:tasks.workunit.client.1.smithi134.stdout:8/416: mknod d7/dc/d2c/d77/c80 0 2024-04-22T23:23:46.049 INFO:tasks.workunit.client.1.smithi134.stdout:5/524: dwrite d9/d11/d73/d95/d20/d60/f7d [0,4194304] 0 2024-04-22T23:23:46.049 INFO:tasks.workunit.client.1.smithi134.stdout:4/560: write d4/d15/fa6 [893265,2516] 0 2024-04-22T23:23:46.050 INFO:tasks.workunit.client.1.smithi134.stdout:8/417: chown d7/dc/d2c/d77/c80 162518 1 2024-04-22T23:23:46.053 INFO:tasks.workunit.client.1.smithi134.stdout:8/418: rmdir d7 39 2024-04-22T23:23:46.059 INFO:tasks.workunit.client.1.smithi134.stdout:5/525: dwrite d9/d11/d34/f6a [0,4194304] 0 2024-04-22T23:23:46.062 INFO:tasks.workunit.client.1.smithi134.stdout:4/561: dwrite d4/d15/d64/f69 [0,4194304] 0 2024-04-22T23:23:46.064 INFO:tasks.workunit.client.1.smithi134.stdout:5/526: mknod d9/dbb/cbc 0 2024-04-22T23:23:46.068 INFO:tasks.workunit.client.1.smithi134.stdout:5/527: unlink d9/d11/d73/d95/d20/f39 0 2024-04-22T23:23:46.070 INFO:tasks.workunit.client.1.smithi134.stdout:5/528: creat d9/d11/d34/fbd x:0 0 0 2024-04-22T23:23:46.070 INFO:tasks.workunit.client.1.smithi134.stdout:8/419: write d7/db/d2d/f3e [358266,82415] 0 2024-04-22T23:23:46.073 INFO:tasks.workunit.client.1.smithi134.stdout:8/420: read d7/dc/d2c/d37/d47/f31 [73291,45484] 0 2024-04-22T23:23:46.075 INFO:tasks.workunit.client.1.smithi134.stdout:5/529: rename d9/fc to d9/dd/fbe 0 2024-04-22T23:23:46.076 INFO:tasks.workunit.client.1.smithi134.stdout:4/562: dwrite d4/d15/f6a [0,4194304] 0 2024-04-22T23:23:46.077 INFO:tasks.workunit.client.1.smithi134.stdout:4/563: fsync d4/d15/d82/f8b 0 2024-04-22T23:23:46.079 INFO:tasks.workunit.client.1.smithi134.stdout:8/421: creat d7/dc/d2c/d4f/d6c/f81 x:0 0 0 2024-04-22T23:23:46.080 INFO:tasks.workunit.client.1.smithi134.stdout:8/422: write d7/dc/f22 [2923553,826] 0 2024-04-22T23:23:46.081 INFO:tasks.workunit.client.1.smithi134.stdout:8/423: stat d7/db/d2d 0 2024-04-22T23:23:46.081 INFO:tasks.workunit.client.1.smithi134.stdout:4/564: dread d4/d13/f74 [0,4194304] 0 2024-04-22T23:23:46.082 INFO:tasks.workunit.client.1.smithi134.stdout:5/530: readlink d9/d11/d73/d95/d20/d60/l76 0 2024-04-22T23:23:46.087 INFO:tasks.workunit.client.1.smithi134.stdout:8/424: creat d7/dc/d2c/d37/d3a/f82 x:0 0 0 2024-04-22T23:23:46.090 INFO:tasks.workunit.client.1.smithi134.stdout:4/565: dwrite d4/d15/f92 [0,4194304] 0 2024-04-22T23:23:46.090 INFO:tasks.workunit.client.1.smithi134.stdout:4/566: stat d4/d6b 0 2024-04-22T23:23:46.092 INFO:tasks.workunit.client.1.smithi134.stdout:8/425: rmdir d7/dc/d19 39 2024-04-22T23:23:46.095 INFO:tasks.workunit.client.1.smithi134.stdout:8/426: creat d7/db/d42/f83 x:0 0 0 2024-04-22T23:23:46.097 INFO:tasks.workunit.client.1.smithi134.stdout:4/567: creat d4/d35/fbb x:0 0 0 2024-04-22T23:23:46.100 INFO:tasks.workunit.client.1.smithi134.stdout:4/568: fdatasync d4/d35/d53/fb8 0 2024-04-22T23:23:46.106 INFO:tasks.workunit.client.1.smithi134.stdout:5/531: dwrite d9/d11/f7a [0,4194304] 0 2024-04-22T23:23:46.106 INFO:tasks.workunit.client.1.smithi134.stdout:8/427: dwrite d7/db/d42/f83 [0,4194304] 0 2024-04-22T23:23:46.108 INFO:tasks.workunit.client.1.smithi134.stdout:5/532: fsync d9/dd/d17/d2a/f98 0 2024-04-22T23:23:46.117 INFO:tasks.workunit.client.1.smithi134.stdout:8/428: dwrite d7/db/d42/f59 [0,4194304] 0 2024-04-22T23:23:46.121 INFO:tasks.workunit.client.1.smithi134.stdout:5/533: dwrite d9/d11/d73/d95/d20/d60/f7d [0,4194304] 0 2024-04-22T23:23:46.124 INFO:tasks.workunit.client.1.smithi134.stdout:8/429: mkdir d7/db/d42/d65/d46/d84 0 2024-04-22T23:23:46.128 INFO:tasks.workunit.client.1.smithi134.stdout:8/430: mknod d7/dc/c85 0 2024-04-22T23:23:46.135 INFO:tasks.workunit.client.1.smithi134.stdout:8/431: dwrite d7/dc/d2c/d4f/d6c/d6e/f6f [0,4194304] 0 2024-04-22T23:23:46.137 INFO:tasks.workunit.client.1.smithi134.stdout:8/432: write d7/db/f1f [700969,56328] 0 2024-04-22T23:23:46.138 INFO:tasks.workunit.client.1.smithi134.stdout:4/569: dread d4/d15/f7f [0,4194304] 0 2024-04-22T23:23:46.139 INFO:tasks.workunit.client.1.smithi134.stdout:4/570: fdatasync d4/d35/d5e/f97 0 2024-04-22T23:23:46.140 INFO:tasks.workunit.client.1.smithi134.stdout:4/571: stat d4/d13/l18 0 2024-04-22T23:23:46.141 INFO:tasks.workunit.client.1.smithi134.stdout:4/572: stat d4/d35/d8e/d5d 0 2024-04-22T23:23:46.143 INFO:tasks.workunit.client.1.smithi134.stdout:8/433: getdents d7/dc/d19 0 2024-04-22T23:23:46.144 INFO:tasks.workunit.client.1.smithi134.stdout:4/573: read d4/dd/f16 [440802,40119] 0 2024-04-22T23:23:46.144 INFO:tasks.workunit.client.1.smithi134.stdout:8/434: chown d7/db/d42/f59 15 1 2024-04-22T23:23:46.145 INFO:tasks.workunit.client.1.smithi134.stdout:8/435: chown d7/dc/d2c/d37/d47/f66 249353 1 2024-04-22T23:23:46.148 INFO:tasks.workunit.client.1.smithi134.stdout:8/436: mkdir d7/dc/d2c/d4f/d6c/d86 0 2024-04-22T23:23:46.150 INFO:tasks.workunit.client.1.smithi134.stdout:5/534: truncate d9/d11/f7a 397420 0 2024-04-22T23:23:46.150 INFO:tasks.workunit.client.1.smithi134.stdout:8/437: rename d7/db/d42/d65 to d7/db/d42/d65/d87 22 2024-04-22T23:23:46.153 INFO:tasks.workunit.client.1.smithi134.stdout:5/535: link d9/d11/d34/f54 d9/dd/d88/db7/fbf 0 2024-04-22T23:23:46.154 INFO:tasks.workunit.client.1.smithi134.stdout:5/536: mkdir d9/d11/d34/d3b/dc0 0 2024-04-22T23:23:46.156 INFO:tasks.workunit.client.1.smithi134.stdout:5/537: mknod d9/dd/d17/d44/cc1 0 2024-04-22T23:23:46.156 INFO:tasks.workunit.client.1.smithi134.stdout:5/538: chown d9/dd/d17/d2a/d49 737 1 2024-04-22T23:23:46.159 INFO:tasks.workunit.client.1.smithi134.stdout:5/539: unlink d9/d11/d34/d3b/l75 0 2024-04-22T23:23:46.191 INFO:tasks.workunit.client.1.smithi134.stdout:9/377: sync 2024-04-22T23:23:46.191 INFO:tasks.workunit.client.1.smithi134.stdout:0/420: sync 2024-04-22T23:23:46.191 INFO:tasks.workunit.client.1.smithi134.stdout:2/364: sync 2024-04-22T23:23:46.191 INFO:tasks.workunit.client.1.smithi134.stdout:4/574: dwrite d4/fa5 [0,4194304] 0 2024-04-22T23:23:46.192 INFO:tasks.workunit.client.1.smithi134.stdout:5/540: chown d9/d1e/d90/fa1 58945 1 2024-04-22T23:23:46.193 INFO:tasks.workunit.client.1.smithi134.stdout:0/421: chown d1/d5/dd/d11 10 1 2024-04-22T23:23:46.193 INFO:tasks.workunit.client.1.smithi134.stdout:9/378: chown d0/d7/d27 1746522 1 2024-04-22T23:23:46.193 INFO:tasks.workunit.client.1.smithi134.stdout:2/365: stat d2/d15/d24/d1d/d3d/f71 0 2024-04-22T23:23:46.195 INFO:tasks.workunit.client.1.smithi134.stdout:4/575: mkdir d4/d7/d32/d3a/dbc 0 2024-04-22T23:23:46.196 INFO:tasks.workunit.client.1.smithi134.stdout:9/379: read d0/d37/f4f [1574117,49998] 0 2024-04-22T23:23:46.197 INFO:tasks.workunit.client.1.smithi134.stdout:5/541: dread d9/d11/f5d [0,4194304] 0 2024-04-22T23:23:46.197 INFO:tasks.workunit.client.1.smithi134.stdout:4/576: write d4/d15/f92 [2614960,124014] 0 2024-04-22T23:23:46.197 INFO:tasks.workunit.client.1.smithi134.stdout:0/422: symlink d1/d5/d15/l84 0 2024-04-22T23:23:46.199 INFO:tasks.workunit.client.1.smithi134.stdout:9/380: fsync d0/d7/d12/f6a 0 2024-04-22T23:23:46.200 INFO:tasks.workunit.client.1.smithi134.stdout:4/577: chown d4/d29/f4d 333445653 1 2024-04-22T23:23:46.200 INFO:tasks.workunit.client.1.smithi134.stdout:2/366: mkdir d2/dd/d77 0 2024-04-22T23:23:46.201 INFO:tasks.workunit.client.1.smithi134.stdout:5/542: mkdir d9/dd/d17/d2a/d49/dc2 0 2024-04-22T23:23:46.205 INFO:tasks.workunit.client.1.smithi134.stdout:5/543: write d9/d1e/f35 [3566126,104883] 0 2024-04-22T23:23:46.206 INFO:tasks.workunit.client.1.smithi134.stdout:2/367: truncate d2/d15/d24/d3b/f4f 1246375 0 2024-04-22T23:23:46.207 INFO:tasks.workunit.client.1.smithi134.stdout:0/423: rename d1/d5/dd/d20/d2c/f6f to d1/d5/d58/d22/f85 0 2024-04-22T23:23:46.209 INFO:tasks.workunit.client.1.smithi134.stdout:2/368: chown d2/dd/d4a 4438899 1 2024-04-22T23:23:46.210 INFO:tasks.workunit.client.1.smithi134.stdout:9/381: dwrite d0/d2d/f44 [0,4194304] 0 2024-04-22T23:23:46.212 INFO:tasks.workunit.client.1.smithi134.stdout:0/424: symlink d1/d2/d1d/l86 0 2024-04-22T23:23:46.214 INFO:tasks.workunit.client.1.smithi134.stdout:4/578: dwrite d4/d15/d64/f73 [0,4194304] 0 2024-04-22T23:23:46.216 INFO:tasks.workunit.client.1.smithi134.stdout:4/579: write d4/d15/f6a [3994677,102407] 0 2024-04-22T23:23:46.219 INFO:tasks.workunit.client.1.smithi134.stdout:2/369: creat d2/dd/d4a/d73/f78 x:0 0 0 2024-04-22T23:23:46.219 INFO:tasks.workunit.client.1.smithi134.stdout:0/425: mknod d1/d5/dd/d20/d2c/c87 0 2024-04-22T23:23:46.219 INFO:tasks.workunit.client.1.smithi134.stdout:0/426: fdatasync d1/d2/f74 0 2024-04-22T23:23:46.220 INFO:tasks.workunit.client.1.smithi134.stdout:2/370: read d2/f22 [4137833,34654] 0 2024-04-22T23:23:46.223 INFO:tasks.workunit.client.1.smithi134.stdout:4/580: rmdir d4/d7/d32/d84 39 2024-04-22T23:23:46.224 INFO:tasks.workunit.client.1.smithi134.stdout:4/581: chown d4/d35/d5e/f97 3763 1 2024-04-22T23:23:46.225 INFO:tasks.workunit.client.1.smithi134.stdout:0/427: mkdir d1/d79/d88 0 2024-04-22T23:23:46.226 INFO:tasks.workunit.client.1.smithi134.stdout:0/428: dread - d1/d2/f56 zero size 2024-04-22T23:23:46.227 INFO:tasks.workunit.client.1.smithi134.stdout:9/382: link d0/d7/f5e d0/d7/d23/d26/d51/d4c/f79 0 2024-04-22T23:23:46.228 INFO:tasks.workunit.client.1.smithi134.stdout:2/371: rename d2/d15/f50 to d2/dd/df/d1c/f79 0 2024-04-22T23:23:46.229 INFO:tasks.workunit.client.1.smithi134.stdout:2/372: write d2/d15/d24/f1e [5796851,63133] 0 2024-04-22T23:23:46.231 INFO:tasks.workunit.client.1.smithi134.stdout:9/383: rmdir d0/d7/d23/d47 39 2024-04-22T23:23:46.234 INFO:tasks.workunit.client.1.smithi134.stdout:0/429: unlink d1/db/c49 0 2024-04-22T23:23:46.237 INFO:tasks.workunit.client.1.smithi134.stdout:9/384: symlink d0/d7/d12/l7a 0 2024-04-22T23:23:46.239 INFO:tasks.workunit.client.1.smithi134.stdout:0/430: creat d1/d5/d58/f89 x:0 0 0 2024-04-22T23:23:46.240 INFO:tasks.workunit.client.1.smithi134.stdout:5/544: dread d9/d11/d34/f51 [0,4194304] 0 2024-04-22T23:23:46.240 INFO:tasks.workunit.client.1.smithi134.stdout:4/582: dwrite d4/dd/f6d [0,4194304] 0 2024-04-22T23:23:46.241 INFO:tasks.workunit.client.1.smithi134.stdout:2/373: dwrite d2/dd/df/d1c/f21 [4194304,4194304] 0 2024-04-22T23:23:46.242 INFO:tasks.workunit.client.1.smithi134.stdout:9/385: symlink d0/d7/d23/d26/l7b 0 2024-04-22T23:23:46.243 INFO:tasks.workunit.client.1.smithi134.stdout:2/374: dread - d2/f60 zero size 2024-04-22T23:23:46.243 INFO:tasks.workunit.client.1.smithi134.stdout:0/431: symlink d1/d5/l8a 0 2024-04-22T23:23:46.243 INFO:tasks.workunit.client.1.smithi134.stdout:4/583: write d4/d35/d8e/f8a [3193052,79173] 0 2024-04-22T23:23:46.245 INFO:tasks.workunit.client.1.smithi134.stdout:5/545: mknod d9/d11/d73/d95/d20/cc3 0 2024-04-22T23:23:46.250 INFO:tasks.workunit.client.1.smithi134.stdout:2/375: write d2/dd/f57 [107069,1542] 0 2024-04-22T23:23:46.252 INFO:tasks.workunit.client.1.smithi134.stdout:0/432: creat d1/d5/f8b x:0 0 0 2024-04-22T23:23:46.253 INFO:tasks.workunit.client.1.smithi134.stdout:5/546: creat d9/d1e/d90/fc4 x:0 0 0 2024-04-22T23:23:46.256 INFO:tasks.workunit.client.1.smithi134.stdout:0/433: dread d1/d5/dd/f45 [0,4194304] 0 2024-04-22T23:23:46.256 INFO:tasks.workunit.client.1.smithi134.stdout:9/386: rename d0/d7/d23/l25 to d0/d7/d23/d26/d51/d4c/l7c 0 2024-04-22T23:23:46.257 INFO:tasks.workunit.client.1.smithi134.stdout:0/434: truncate d1/d2/d1d/f7d 5068398 0 2024-04-22T23:23:46.258 INFO:tasks.workunit.client.1.smithi134.stdout:2/376: symlink d2/l7a 0 2024-04-22T23:23:46.258 INFO:tasks.workunit.client.1.smithi134.stdout:5/547: truncate d9/f19 1565723 0 2024-04-22T23:23:46.265 INFO:tasks.workunit.client.1.smithi134.stdout:5/548: link d9/d11/d34/f54 d9/d11/d34/d4d/fc5 0 2024-04-22T23:23:46.267 INFO:tasks.workunit.client.1.smithi134.stdout:2/377: dwrite d2/d15/d24/d1d/f26 [0,4194304] 0 2024-04-22T23:23:46.267 INFO:tasks.workunit.client.1.smithi134.stdout:5/549: chown d9/d11/f5d 54790 1 2024-04-22T23:23:46.270 INFO:tasks.workunit.client.1.smithi134.stdout:2/378: dread - d2/d42/d54/f59 zero size 2024-04-22T23:23:46.270 INFO:tasks.workunit.client.1.smithi134.stdout:9/387: dwrite d0/d7/d23/d26/d51/d4c/f79 [4194304,4194304] 0 2024-04-22T23:23:46.272 INFO:tasks.workunit.client.1.smithi134.stdout:9/388: stat d0/d2d/d4e/l39 0 2024-04-22T23:23:46.274 INFO:tasks.workunit.client.1.smithi134.stdout:5/550: creat d9/dd/da6/fc6 x:0 0 0 2024-04-22T23:23:46.275 INFO:tasks.workunit.client.1.smithi134.stdout:4/584: dread d4/d15/f1c [4194304,4194304] 0 2024-04-22T23:23:46.278 INFO:tasks.workunit.client.1.smithi134.stdout:2/379: mkdir d2/d15/d24/d1d/d3d/d7b 0 2024-04-22T23:23:46.278 INFO:tasks.workunit.client.1.smithi134.stdout:5/551: write d9/d11/d34/d4d/f9f [382531,125102] 0 2024-04-22T23:23:46.280 INFO:tasks.workunit.client.1.smithi134.stdout:9/389: creat d0/d7/d12/d22/f7d x:0 0 0 2024-04-22T23:23:46.281 INFO:tasks.workunit.client.1.smithi134.stdout:5/552: readlink d9/d11/d34/la8 0 2024-04-22T23:23:46.285 INFO:tasks.workunit.client.1.smithi134.stdout:4/585: dwrite d4/d35/d80/f90 [0,4194304] 0 2024-04-22T23:23:46.286 INFO:tasks.workunit.client.1.smithi134.stdout:9/390: read d0/d7/d23/d1b/f49 [283549,117507] 0 2024-04-22T23:23:46.292 INFO:tasks.workunit.client.1.smithi134.stdout:4/586: creat d4/d7/d32/d3a/dbc/fbd x:0 0 0 2024-04-22T23:23:46.294 INFO:tasks.workunit.client.1.smithi134.stdout:9/391: mknod d0/d2d/c7e 0 2024-04-22T23:23:46.295 INFO:tasks.workunit.client.1.smithi134.stdout:4/587: dread d4/d13/f74 [0,4194304] 0 2024-04-22T23:23:46.299 INFO:tasks.workunit.client.1.smithi134.stdout:0/435: truncate d1/d5/d58/d22/f24 8012583 0 2024-04-22T23:23:46.300 INFO:tasks.workunit.client.1.smithi134.stdout:9/392: creat d0/d7/d23/d47/f7f x:0 0 0 2024-04-22T23:23:46.301 INFO:tasks.workunit.client.1.smithi134.stdout:0/436: dread d1/d5/f50 [0,4194304] 0 2024-04-22T23:23:46.302 INFO:tasks.workunit.client.1.smithi134.stdout:4/588: creat d4/d35/db4/fbe x:0 0 0 2024-04-22T23:23:46.303 INFO:tasks.workunit.client.1.smithi134.stdout:0/437: write d1/d5/d58/f80 [9067,63741] 0 2024-04-22T23:23:46.307 INFO:tasks.workunit.client.1.smithi134.stdout:5/553: dread d9/d1e/f35 [0,4194304] 0 2024-04-22T23:23:46.308 INFO:tasks.workunit.client.1.smithi134.stdout:5/554: stat d9/dd/d88/db7 0 2024-04-22T23:23:46.310 INFO:tasks.workunit.client.1.smithi134.stdout:0/438: fsync d1/d5/dd/f45 0 2024-04-22T23:23:46.311 INFO:tasks.workunit.client.1.smithi134.stdout:2/380: dread d2/d15/d24/f2a [4194304,4194304] 0 2024-04-22T23:23:46.315 INFO:tasks.workunit.client.1.smithi134.stdout:9/393: dread d0/d7/f2c [0,4194304] 0 2024-04-22T23:23:46.316 INFO:tasks.workunit.client.1.smithi134.stdout:0/439: creat d1/d77/f8c x:0 0 0 2024-04-22T23:23:46.317 INFO:tasks.workunit.client.1.smithi134.stdout:4/589: write d4/d35/d5e/f1f [1201265,7704] 0 2024-04-22T23:23:46.321 INFO:tasks.workunit.client.1.smithi134.stdout:9/394: dwrite d0/d7/fb [0,4194304] 0 2024-04-22T23:23:46.324 INFO:tasks.workunit.client.1.smithi134.stdout:2/381: dwrite d2/f22 [0,4194304] 0 2024-04-22T23:23:46.329 INFO:tasks.workunit.client.1.smithi134.stdout:9/395: link d0/d7/d12/l60 d0/d7/d23/d26/d51/l80 0 2024-04-22T23:23:46.329 INFO:tasks.workunit.client.1.smithi134.stdout:2/382: write d2/f22 [1925634,41582] 0 2024-04-22T23:23:46.331 INFO:tasks.workunit.client.1.smithi134.stdout:2/383: truncate d2/dd/d4a/f64 969291 0 2024-04-22T23:23:46.336 INFO:tasks.workunit.client.1.smithi134.stdout:3/301: sync 2024-04-22T23:23:46.336 INFO:tasks.workunit.client.1.smithi134.stdout:2/384: dwrite d2/d42/f46 [8388608,4194304] 0 2024-04-22T23:23:46.339 INFO:tasks.workunit.client.1.smithi134.stdout:3/302: mkdir dc/dd/d28/d34/d72 0 2024-04-22T23:23:46.340 INFO:tasks.workunit.client.1.smithi134.stdout:4/590: dread d4/d15/f26 [0,4194304] 0 2024-04-22T23:23:46.341 INFO:tasks.workunit.client.1.smithi134.stdout:3/303: fsync dc/f18 0 2024-04-22T23:23:46.342 INFO:tasks.workunit.client.1.smithi134.stdout:2/385: link d2/dd/df/f6f d2/d15/d24/d1d/d3d/d7b/f7c 0 2024-04-22T23:23:46.343 INFO:tasks.workunit.client.1.smithi134.stdout:2/386: stat d2/d42/d54/d33/c5f 0 2024-04-22T23:23:46.344 INFO:tasks.workunit.client.1.smithi134.stdout:2/387: read d2/d15/d24/d1d/d3d/f71 [972129,118191] 0 2024-04-22T23:23:46.371 INFO:tasks.workunit.client.1.smithi134.stdout:3/304: chown dc/l13 0 1 2024-04-22T23:23:46.375 INFO:tasks.workunit.client.1.smithi134.stdout:2/388: symlink d2/dd/d4a/d73/l7d 0 2024-04-22T23:23:46.375 INFO:tasks.workunit.client.1.smithi134.stdout:2/389: chown d2/dd/d4a 13 1 2024-04-22T23:23:46.376 INFO:tasks.workunit.client.1.smithi134.stdout:3/305: write dc/dd/d28/d2f/f49 [4274224,91791] 0 2024-04-22T23:23:46.378 INFO:tasks.workunit.client.1.smithi134.stdout:3/306: write dc/dd/d28/d2f/f49 [2857484,76613] 0 2024-04-22T23:23:46.379 INFO:tasks.workunit.client.1.smithi134.stdout:2/390: unlink d2/dd/df/l32 0 2024-04-22T23:23:46.381 INFO:tasks.workunit.client.1.smithi134.stdout:4/591: write d4/d15/f7f [5199932,72468] 0 2024-04-22T23:23:46.382 INFO:tasks.workunit.client.1.smithi134.stdout:4/592: chown d4/d35/d8e/f8a 2 1 2024-04-22T23:23:46.383 INFO:tasks.workunit.client.1.smithi134.stdout:2/391: link d2/f17 d2/dd/d77/f7e 0 2024-04-22T23:23:46.385 INFO:tasks.workunit.client.1.smithi134.stdout:3/307: creat dc/dd/d28/d2f/f73 x:0 0 0 2024-04-22T23:23:46.386 INFO:tasks.workunit.client.1.smithi134.stdout:3/308: dread - dc/dd/f4f zero size 2024-04-22T23:23:46.389 INFO:tasks.workunit.client.1.smithi134.stdout:4/593: dwrite d4/d35/fbb [0,4194304] 0 2024-04-22T23:23:46.481 INFO:tasks.workunit.client.1.smithi134.stdout:3/309: symlink dc/dd/l74 0 2024-04-22T23:23:46.483 INFO:tasks.workunit.client.1.smithi134.stdout:4/594: creat d4/d7/d32/fbf x:0 0 0 2024-04-22T23:23:46.485 INFO:tasks.workunit.client.1.smithi134.stdout:3/310: dread dc/f52 [0,4194304] 0 2024-04-22T23:23:46.486 INFO:tasks.workunit.client.1.smithi134.stdout:3/311: write dc/dd/d28/f39 [1086835,117061] 0 2024-04-22T23:23:46.487 INFO:tasks.workunit.client.1.smithi134.stdout:4/595: creat d4/d7/d32/d84/fc0 x:0 0 0 2024-04-22T23:23:46.487 INFO:tasks.workunit.client.1.smithi134.stdout:4/596: readlink d4/d29/l3f 0 2024-04-22T23:23:46.491 INFO:tasks.workunit.client.1.smithi134.stdout:4/597: truncate d4/d35/d5e/d25/f98 159186 0 2024-04-22T23:23:46.491 INFO:tasks.workunit.client.1.smithi134.stdout:3/312: getdents dc/dd/d28/d2f/d21/d42 0 2024-04-22T23:23:46.493 INFO:tasks.workunit.client.1.smithi134.stdout:4/598: chown d4/d35/d8e/d5d/fa9 229 1 2024-04-22T23:23:46.494 INFO:tasks.workunit.client.1.smithi134.stdout:2/392: dwrite d2/dd/f37 [8388608,4194304] 0 2024-04-22T23:23:46.494 INFO:tasks.workunit.client.1.smithi134.stdout:2/393: readlink d2/lc 0 2024-04-22T23:23:46.495 INFO:tasks.workunit.client.1.smithi134.stdout:4/599: dread - d4/d7/d32/d84/fc0 zero size 2024-04-22T23:23:46.496 INFO:tasks.workunit.client.1.smithi134.stdout:3/313: mkdir dc/dd/d28/d34/d72/d75 0 2024-04-22T23:23:46.497 INFO:tasks.workunit.client.1.smithi134.stdout:4/600: write d4/d35/d5e/d25/f98 [747556,108288] 0 2024-04-22T23:23:46.499 INFO:tasks.workunit.client.1.smithi134.stdout:4/601: fsync d4/d7/d32/fbf 0 2024-04-22T23:23:46.500 INFO:tasks.workunit.client.1.smithi134.stdout:3/314: mknod dc/dd/d28/d3b/d29/c76 0 2024-04-22T23:23:46.503 INFO:tasks.workunit.client.1.smithi134.stdout:3/315: mkdir dc/dd/d28/d2f/d21/d42/d77 0 2024-04-22T23:23:46.504 INFO:tasks.workunit.client.1.smithi134.stdout:3/316: stat dc/dd/d28/f1c 0 2024-04-22T23:23:46.504 INFO:tasks.workunit.client.1.smithi134.stdout:4/602: creat d4/d7/d32/fc1 x:0 0 0 2024-04-22T23:23:46.506 INFO:tasks.workunit.client.1.smithi134.stdout:4/603: chown d4/d7/d32/d3a/f43 48 1 2024-04-22T23:23:46.507 INFO:tasks.workunit.client.1.smithi134.stdout:3/317: symlink dc/dd/d28/d2f/d36/d5f/l78 0 2024-04-22T23:23:46.507 INFO:tasks.workunit.client.1.smithi134.stdout:4/604: write d4/d35/d80/f90 [1732085,46464] 0 2024-04-22T23:23:46.508 INFO:tasks.workunit.client.1.smithi134.stdout:4/605: readlink d4/d35/d5e/l19 0 2024-04-22T23:23:46.509 INFO:tasks.workunit.client.1.smithi134.stdout:4/606: readlink d4/d7/d32/l45 0 2024-04-22T23:23:46.510 INFO:tasks.workunit.client.1.smithi134.stdout:4/607: write d4/f42 [3783634,94217] 0 2024-04-22T23:23:46.510 INFO:tasks.workunit.client.1.smithi134.stdout:3/318: creat dc/dd/d28/d3b/f79 x:0 0 0 2024-04-22T23:23:46.513 INFO:tasks.workunit.client.1.smithi134.stdout:4/608: truncate d4/d15/f54 698552 0 2024-04-22T23:23:46.514 INFO:tasks.workunit.client.1.smithi134.stdout:4/609: dread - d4/d35/d5e/f94 zero size 2024-04-22T23:23:46.514 INFO:tasks.workunit.client.1.smithi134.stdout:2/394: truncate d2/dd/df/d1c/f79 4061601 0 2024-04-22T23:23:46.518 INFO:tasks.workunit.client.1.smithi134.stdout:3/319: creat dc/dd/f7a x:0 0 0 2024-04-22T23:23:46.520 INFO:tasks.workunit.client.1.smithi134.stdout:3/320: chown dc/dd/d28/d2f/d21/d42/l5a 10 1 2024-04-22T23:23:46.521 INFO:tasks.workunit.client.1.smithi134.stdout:4/610: mkdir d4/dd/da8/dc2 0 2024-04-22T23:23:46.522 INFO:tasks.workunit.client.1.smithi134.stdout:2/395: dwrite d2/d15/d24/d1d/f26 [0,4194304] 0 2024-04-22T23:23:46.523 INFO:tasks.workunit.client.1.smithi134.stdout:4/611: dread - d4/d35/d5e/f94 zero size 2024-04-22T23:23:46.525 INFO:tasks.workunit.client.1.smithi134.stdout:3/321: creat dc/dd/d28/d2f/d54/f7b x:0 0 0 2024-04-22T23:23:46.530 INFO:tasks.workunit.client.1.smithi134.stdout:2/396: rename d2/d42/d54/d33/f51 to d2/dd/d4a/d73/f7f 0 2024-04-22T23:23:46.533 INFO:tasks.workunit.client.1.smithi134.stdout:3/322: creat dc/dd/d28/d34/d6b/f7c x:0 0 0 2024-04-22T23:23:46.534 INFO:tasks.workunit.client.1.smithi134.stdout:2/397: fsync d2/d15/d24/f2a 0 2024-04-22T23:23:46.536 INFO:tasks.workunit.client.1.smithi134.stdout:2/398: read d2/f30 [913352,51313] 0 2024-04-22T23:23:46.538 INFO:tasks.workunit.client.1.smithi134.stdout:1/581: sync 2024-04-22T23:23:46.539 INFO:tasks.workunit.client.1.smithi134.stdout:8/438: sync 2024-04-22T23:23:46.539 INFO:tasks.workunit.client.1.smithi134.stdout:6/513: sync 2024-04-22T23:23:46.539 INFO:tasks.workunit.client.1.smithi134.stdout:7/501: sync 2024-04-22T23:23:46.540 INFO:tasks.workunit.client.1.smithi134.stdout:3/323: dwrite f9 [0,4194304] 0 2024-04-22T23:23:46.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:46 smithi067 ceph-mon[26453]: pgmap v278: 65 pgs: 65 active+clean; 2.9 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 80 MiB/s rd, 253 MiB/s wr, 488 op/s 2024-04-22T23:23:46.554 INFO:tasks.workunit.client.1.smithi134.stdout:4/612: truncate d4/d15/f7f 213798 0 2024-04-22T23:23:46.555 INFO:tasks.workunit.client.1.smithi134.stdout:1/582: unlink db/d15/d17/d20/l68 0 2024-04-22T23:23:46.558 INFO:tasks.workunit.client.1.smithi134.stdout:2/399: write d2/d15/d24/f27 [4663825,113513] 0 2024-04-22T23:23:46.561 INFO:tasks.workunit.client.1.smithi134.stdout:7/502: dwrite d4/d10/fb8 [0,4194304] 0 2024-04-22T23:23:46.562 INFO:tasks.workunit.client.1.smithi134.stdout:8/439: dwrite d7/db/d42/f59 [0,4194304] 0 2024-04-22T23:23:46.567 INFO:tasks.workunit.client.1.smithi134.stdout:4/613: write d4/d15/d64/f69 [3255353,21225] 0 2024-04-22T23:23:46.570 INFO:tasks.workunit.client.1.smithi134.stdout:7/503: write d4/d10/fa5 [2502907,116267] 0 2024-04-22T23:23:46.570 INFO:tasks.workunit.client.1.smithi134.stdout:6/514: dwrite d0/d19/db/d35/f58 [4194304,4194304] 0 2024-04-22T23:23:46.572 INFO:tasks.workunit.client.1.smithi134.stdout:8/440: fdatasync d7/db/d42/f76 0 2024-04-22T23:23:46.573 INFO:tasks.workunit.client.1.smithi134.stdout:4/614: dread - d4/d35/d8e/d5d/fa9 zero size 2024-04-22T23:23:46.576 INFO:tasks.workunit.client.1.smithi134.stdout:4/615: fsync d4/d35/d8e/f8a 0 2024-04-22T23:23:46.576 INFO:tasks.workunit.client.1.smithi134.stdout:7/504: unlink d4/lc 0 2024-04-22T23:23:46.578 INFO:tasks.workunit.client.1.smithi134.stdout:2/400: dread d2/f22 [0,4194304] 0 2024-04-22T23:23:46.581 INFO:tasks.workunit.client.1.smithi134.stdout:8/441: mkdir d7/db/d2d/d39/d58/d7b/d88 0 2024-04-22T23:23:46.581 INFO:tasks.workunit.client.1.smithi134.stdout:3/324: write dc/dd/d28/d2f/d21/f16 [1908374,91652] 0 2024-04-22T23:23:46.582 INFO:tasks.workunit.client.1.smithi134.stdout:7/505: symlink d4/d8/d3b/d83/lbe 0 2024-04-22T23:23:46.584 INFO:tasks.workunit.client.1.smithi134.stdout:6/515: rename d0/d19/de/c25 to d0/d19/db/d35/cac 0 2024-04-22T23:23:46.587 INFO:tasks.workunit.client.1.smithi134.stdout:2/401: write d2/f3e [799205,98950] 0 2024-04-22T23:23:46.588 INFO:tasks.workunit.client.1.smithi134.stdout:8/442: fsync d7/db/d2d/f3c 0 2024-04-22T23:23:46.590 INFO:tasks.workunit.client.1.smithi134.stdout:7/506: creat d4/d8/d2a/d42/d75/fbf x:0 0 0 2024-04-22T23:23:46.591 INFO:tasks.workunit.client.1.smithi134.stdout:4/616: creat d4/d7/d32/fc3 x:0 0 0 2024-04-22T23:23:46.593 INFO:tasks.workunit.client.1.smithi134.stdout:4/617: chown d4/d15/d82 0 1 2024-04-22T23:23:46.594 INFO:tasks.workunit.client.1.smithi134.stdout:3/325: dwrite dc/dd/d28/d2f/d21/d42/f66 [0,4194304] 0 2024-04-22T23:23:46.598 INFO:tasks.workunit.client.1.smithi134.stdout:4/618: dread - d4/d7/d32/d84/fc0 zero size 2024-04-22T23:23:46.598 INFO:tasks.workunit.client.1.smithi134.stdout:1/583: dwrite db/d15/d17/f5a [0,4194304] 0 2024-04-22T23:23:46.598 INFO:tasks.workunit.client.1.smithi134.stdout:5/555: sync 2024-04-22T23:23:46.605 INFO:tasks.workunit.client.1.smithi134.stdout:6/516: creat d0/d19/de/d1f/d94/fad x:0 0 0 2024-04-22T23:23:46.608 INFO:tasks.workunit.client.1.smithi134.stdout:7/507: dwrite d4/d8/d2a/f4f [0,4194304] 0 2024-04-22T23:23:46.613 INFO:tasks.workunit.client.1.smithi134.stdout:5/556: rmdir d9 39 2024-04-22T23:23:46.613 INFO:tasks.workunit.client.1.smithi134.stdout:1/584: rmdir db/d15/d17/d20/d2a/d2d/d2e/d42 39 2024-04-22T23:23:46.613 INFO:tasks.workunit.client.1.smithi134.stdout:2/402: dwrite d2/d15/d24/d1d/f44 [0,4194304] 0 2024-04-22T23:23:46.616 INFO:tasks.workunit.client.1.smithi134.stdout:6/517: mkdir d0/d19/de/d1f/dae 0 2024-04-22T23:23:46.620 INFO:tasks.workunit.client.1.smithi134.stdout:8/443: creat d7/db/d42/d65/f89 x:0 0 0 2024-04-22T23:23:46.620 INFO:tasks.workunit.client.1.smithi134.stdout:6/518: stat d0/d19/de/d12/c15 0 2024-04-22T23:23:46.628 INFO:tasks.workunit.client.1.smithi134.stdout:1/585: rename db/d15/d17/d20/d2a/l7f to db/d15/d72/lcd 0 2024-04-22T23:23:46.629 INFO:tasks.workunit.client.1.smithi134.stdout:4/619: dwrite d4/f21 [4194304,4194304] 0 2024-04-22T23:23:46.633 INFO:tasks.workunit.client.1.smithi134.stdout:5/557: fsync d9/d1e/f23 0 2024-04-22T23:23:46.637 INFO:tasks.workunit.client.1.smithi134.stdout:4/620: dread - d4/d7/d32/d3a/fa0 zero size 2024-04-22T23:23:46.637 INFO:tasks.workunit.client.1.smithi134.stdout:8/444: dwrite d7/dc/f12 [0,4194304] 0 2024-04-22T23:23:46.639 INFO:tasks.workunit.client.1.smithi134.stdout:2/403: write d2/dd/d4a/f52 [4189392,114103] 0 2024-04-22T23:23:46.640 INFO:tasks.workunit.client.1.smithi134.stdout:3/326: dwrite dc/f18 [0,4194304] 0 2024-04-22T23:23:46.641 INFO:tasks.workunit.client.1.smithi134.stdout:7/508: write d4/d8/d2a/fb1 [182792,33842] 0 2024-04-22T23:23:46.644 INFO:tasks.workunit.client.1.smithi134.stdout:8/445: chown d7/dc/d2c/d4f 56871077 1 2024-04-22T23:23:46.649 INFO:tasks.workunit.client.1.smithi134.stdout:4/621: mkdir d4/d7/d32/d3a/dbc/dc4 0 2024-04-22T23:23:46.651 INFO:tasks.workunit.client.1.smithi134.stdout:7/509: chown d4/d10/d2f/d34/d62 71 1 2024-04-22T23:23:46.651 INFO:tasks.workunit.client.1.smithi134.stdout:4/622: stat d4/d15/f1c 0 2024-04-22T23:23:46.654 INFO:tasks.workunit.client.1.smithi134.stdout:4/623: chown d4/d35/d8e/d5d 8438 1 2024-04-22T23:23:46.656 INFO:tasks.workunit.client.1.smithi134.stdout:8/446: write d7/dc/d2c/d37/d3a/f82 [921890,760] 0 2024-04-22T23:23:46.656 INFO:tasks.workunit.client.1.smithi134.stdout:7/510: chown d4/d10/d2f/d34 155266704 1 2024-04-22T23:23:46.657 INFO:tasks.workunit.client.1.smithi134.stdout:4/624: dread - d4/d15/f6f zero size 2024-04-22T23:23:46.657 INFO:tasks.workunit.client.1.smithi134.stdout:6/519: dread d0/f26 [0,4194304] 0 2024-04-22T23:23:46.658 INFO:tasks.workunit.client.1.smithi134.stdout:1/586: dwrite db/d15/d17/d20/d2a/d2d/f48 [0,4194304] 0 2024-04-22T23:23:46.662 INFO:tasks.workunit.client.1.smithi134.stdout:6/520: stat d0/d19/db/d75/l7c 0 2024-04-22T23:23:46.663 INFO:tasks.workunit.client.1.smithi134.stdout:8/447: mkdir d7/db/d2d/d39/d58/d8a 0 2024-04-22T23:23:46.664 INFO:tasks.workunit.client.1.smithi134.stdout:4/625: mkdir d4/d7/d32/dc5 0 2024-04-22T23:23:46.665 INFO:tasks.workunit.client.1.smithi134.stdout:7/511: rmdir d4/d8/d41/d76 39 2024-04-22T23:23:46.665 INFO:tasks.workunit.client.1.smithi134.stdout:1/587: rmdir db/d15/d17/d20/d2a/d2d/d2e 39 2024-04-22T23:23:46.666 INFO:tasks.workunit.client.1.smithi134.stdout:2/404: dwrite d2/dd/f57 [0,4194304] 0 2024-04-22T23:23:46.666 INFO:tasks.workunit.client.1.smithi134.stdout:1/588: readlink l4 0 2024-04-22T23:23:46.666 INFO:tasks.workunit.client.1.smithi134.stdout:4/626: stat d4/d35/d8e/d5d/cb0 0 2024-04-22T23:23:46.667 INFO:tasks.workunit.client.1.smithi134.stdout:8/448: fdatasync d7/dc/d2c/d37/d47/f34 0 2024-04-22T23:23:46.672 INFO:tasks.workunit.client.1.smithi134.stdout:4/627: creat d4/d35/d8e/d5d/fc6 x:0 0 0 2024-04-22T23:23:46.674 INFO:tasks.workunit.client.1.smithi134.stdout:2/405: symlink d2/d15/d24/l80 0 2024-04-22T23:23:46.675 INFO:tasks.workunit.client.1.smithi134.stdout:1/589: read db/d15/d17/d20/d2a/d2d/d2e/f47 [2537807,90906] 0 2024-04-22T23:23:46.675 INFO:tasks.workunit.client.1.smithi134.stdout:8/449: mknod d7/db/d2d/d39/d58/d7b/c8b 0 2024-04-22T23:23:46.676 INFO:tasks.workunit.client.1.smithi134.stdout:6/521: dwrite d0/d19/de/d12/d29/f8d [0,4194304] 0 2024-04-22T23:23:46.677 INFO:tasks.workunit.client.1.smithi134.stdout:2/406: symlink d2/l81 0 2024-04-22T23:23:46.681 INFO:tasks.workunit.client.1.smithi134.stdout:4/628: truncate d4/d29/f4d 991886 0 2024-04-22T23:23:46.682 INFO:tasks.workunit.client.1.smithi134.stdout:2/407: rmdir d2/d15/d24 39 2024-04-22T23:23:46.682 INFO:tasks.workunit.client.1.smithi134.stdout:6/522: fdatasync d0/d19/de/d12/d29/d74/f89 0 2024-04-22T23:23:46.684 INFO:tasks.workunit.client.1.smithi134.stdout:2/408: readlink d2/dd/l38 0 2024-04-22T23:23:46.685 INFO:tasks.workunit.client.1.smithi134.stdout:1/590: rename db/d15/f18 to db/db5/fce 0 2024-04-22T23:23:46.688 INFO:tasks.workunit.client.1.smithi134.stdout:8/450: dwrite d7/db/d42/d65/f7e [0,4194304] 0 2024-04-22T23:23:46.690 INFO:tasks.workunit.client.1.smithi134.stdout:6/523: dread d0/d6f/f9e [0,4194304] 0 2024-04-22T23:23:46.690 INFO:tasks.workunit.client.1.smithi134.stdout:8/451: dread d7/db/d2d/f2e [0,4194304] 0 2024-04-22T23:23:46.692 INFO:tasks.workunit.client.1.smithi134.stdout:6/524: chown d0/d19/de/d12/d5e 2951 1 2024-04-22T23:23:46.697 INFO:tasks.workunit.client.1.smithi134.stdout:2/409: link d2/d15/d24/d1d/l20 d2/d42/d54/d33/d6e/l82 0 2024-04-22T23:23:46.698 INFO:tasks.workunit.client.1.smithi134.stdout:1/591: getdents db/d15/d17/d20/d2a/d2d/d86/dad 0 2024-04-22T23:23:46.699 INFO:tasks.workunit.client.1.smithi134.stdout:8/452: mknod d7/db/d2d/d39/d58/d7b/d88/c8c 0 2024-04-22T23:23:46.701 INFO:tasks.workunit.client.1.smithi134.stdout:8/453: mknod d7/dc/d2c/d4f/d6c/c8d 0 2024-04-22T23:23:46.702 INFO:tasks.workunit.client.1.smithi134.stdout:6/525: link d0/d19/de/d1f/c6b d0/d19/db/d35/caf 0 2024-04-22T23:23:46.703 INFO:tasks.workunit.client.1.smithi134.stdout:2/410: symlink d2/dd/d68/l83 0 2024-04-22T23:23:46.704 INFO:tasks.workunit.client.1.smithi134.stdout:6/526: chown d0/d19/de/d12/d2a/d2d/d3a/f77 11 1 2024-04-22T23:23:46.704 INFO:tasks.workunit.client.1.smithi134.stdout:1/592: creat db/db0/dc1/fcf x:0 0 0 2024-04-22T23:23:46.708 INFO:tasks.workunit.client.1.smithi134.stdout:8/454: getdents d7/db/d2d 0 2024-04-22T23:23:46.711 INFO:tasks.workunit.client.1.smithi134.stdout:1/593: mkdir db/d15/d17/d20/dd0 0 2024-04-22T23:23:46.714 INFO:tasks.workunit.client.1.smithi134.stdout:8/455: mknod d7/db/d42/d65/d46/d84/c8e 0 2024-04-22T23:23:46.718 INFO:tasks.workunit.client.1.smithi134.stdout:1/594: write db/db0/dc1/fcf [638587,87356] 0 2024-04-22T23:23:46.718 INFO:tasks.workunit.client.1.smithi134.stdout:7/512: write d4/d10/f1f [3466569,9187] 0 2024-04-22T23:23:46.718 INFO:tasks.workunit.client.1.smithi134.stdout:6/527: dwrite d0/d19/de/d12/d2a/d2d/f9c [0,4194304] 0 2024-04-22T23:23:46.720 INFO:tasks.workunit.client.1.smithi134.stdout:3/327: dwrite dc/d2a/f23 [0,4194304] 0 2024-04-22T23:23:46.722 INFO:tasks.workunit.client.1.smithi134.stdout:4/629: dread d4/d15/d82/f8b [0,4194304] 0 2024-04-22T23:23:46.723 INFO:tasks.workunit.client.1.smithi134.stdout:8/456: mknod d7/dc/c8f 0 2024-04-22T23:23:46.728 INFO:tasks.workunit.client.1.smithi134.stdout:3/328: write dc/dd/d28/d34/f50 [3094753,73666] 0 2024-04-22T23:23:46.728 INFO:tasks.workunit.client.1.smithi134.stdout:7/513: chown d4/d8/d3b/f8d 997708 1 2024-04-22T23:23:46.729 INFO:tasks.workunit.client.1.smithi134.stdout:2/411: truncate d2/d15/d24/d1d/f28 1913549 0 2024-04-22T23:23:46.735 INFO:tasks.workunit.client.1.smithi134.stdout:1/595: dwrite db/d15/d17/fa2 [0,4194304] 0 2024-04-22T23:23:46.736 INFO:tasks.workunit.client.1.smithi134.stdout:4/630: creat d4/d7/d32/d3a/fc7 x:0 0 0 2024-04-22T23:23:46.737 INFO:tasks.workunit.client.1.smithi134.stdout:1/596: stat db/d15/d17/cc4 0 2024-04-22T23:23:46.739 INFO:tasks.workunit.client.1.smithi134.stdout:3/329: dwrite dc/dd/d28/f1c [0,4194304] 0 2024-04-22T23:23:46.741 INFO:tasks.workunit.client.1.smithi134.stdout:6/528: getdents d0/d19/de/d12/d29/d74/da0 0 2024-04-22T23:23:46.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:46 smithi134 ceph-mon[34609]: pgmap v278: 65 pgs: 65 active+clean; 2.9 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 80 MiB/s rd, 253 MiB/s wr, 488 op/s 2024-04-22T23:23:46.749 INFO:tasks.workunit.client.1.smithi134.stdout:3/330: mkdir dc/d7d 0 2024-04-22T23:23:46.750 INFO:tasks.workunit.client.1.smithi134.stdout:4/631: truncate d4/d15/f54 1475407 0 2024-04-22T23:23:46.752 INFO:tasks.workunit.client.1.smithi134.stdout:1/597: creat db/d15/d17/d20/fd1 x:0 0 0 2024-04-22T23:23:46.753 INFO:tasks.workunit.client.1.smithi134.stdout:4/632: stat d4/d7/d32/d3a/d93 0 2024-04-22T23:23:46.756 INFO:tasks.workunit.client.1.smithi134.stdout:1/598: symlink db/d15/d72/ld2 0 2024-04-22T23:23:46.758 INFO:tasks.workunit.client.1.smithi134.stdout:6/529: dwrite d0/d19/db/fd [0,4194304] 0 2024-04-22T23:23:46.758 INFO:tasks.workunit.client.1.smithi134.stdout:3/331: symlink dc/dd/d28/d34/l7e 0 2024-04-22T23:23:46.759 INFO:tasks.workunit.client.1.smithi134.stdout:8/457: dwrite d7/db/d2d/f2e [0,4194304] 0 2024-04-22T23:23:46.763 INFO:tasks.workunit.client.1.smithi134.stdout:7/514: dread d4/d8/d3b/f8d [0,4194304] 0 2024-04-22T23:23:46.764 INFO:tasks.workunit.client.1.smithi134.stdout:8/458: stat d7/db/d2d/l44 0 2024-04-22T23:23:46.766 INFO:tasks.workunit.client.1.smithi134.stdout:9/396: sync 2024-04-22T23:23:46.766 INFO:tasks.workunit.client.1.smithi134.stdout:0/440: sync 2024-04-22T23:23:46.768 INFO:tasks.workunit.client.1.smithi134.stdout:3/332: creat dc/d2a/d62/f7f x:0 0 0 2024-04-22T23:23:46.773 INFO:tasks.workunit.client.1.smithi134.stdout:0/441: chown d1/d5/d58/f46 12339 1 2024-04-22T23:23:46.775 INFO:tasks.workunit.client.1.smithi134.stdout:2/412: dwrite d2/d15/d24/f35 [0,4194304] 0 2024-04-22T23:23:46.776 INFO:tasks.workunit.client.1.smithi134.stdout:3/333: write dc/dd/f4f [727342,95038] 0 2024-04-22T23:23:46.778 INFO:tasks.workunit.client.1.smithi134.stdout:2/413: chown d2/dd/df/c10 24 1 2024-04-22T23:23:46.779 INFO:tasks.workunit.client.1.smithi134.stdout:8/459: getdents d7/db/d42/d65 0 2024-04-22T23:23:46.781 INFO:tasks.workunit.client.1.smithi134.stdout:9/397: symlink d0/d7/d23/d1b/d5a/l81 0 2024-04-22T23:23:46.781 INFO:tasks.workunit.client.1.smithi134.stdout:4/633: dwrite d4/d35/d5e/f22 [0,4194304] 0 2024-04-22T23:23:46.787 INFO:tasks.workunit.client.1.smithi134.stdout:8/460: truncate d7/db/d42/f76 514045 0 2024-04-22T23:23:46.787 INFO:tasks.workunit.client.1.smithi134.stdout:1/599: dwrite db/f27 [8388608,4194304] 0 2024-04-22T23:23:46.787 INFO:tasks.workunit.client.1.smithi134.stdout:6/530: fsync d0/d19/db/fd 0 2024-04-22T23:23:46.789 INFO:tasks.workunit.client.1.smithi134.stdout:4/634: dread - d4/d15/f6f zero size 2024-04-22T23:23:46.789 INFO:tasks.workunit.client.1.smithi134.stdout:9/398: mkdir d0/d2d/d82 0 2024-04-22T23:23:46.792 INFO:tasks.workunit.client.1.smithi134.stdout:4/635: read - d4/d15/d82/d9a/faa zero size 2024-04-22T23:23:46.798 INFO:tasks.workunit.client.1.smithi134.stdout:4/636: fsync d4/d7/d32/d84/fc0 0 2024-04-22T23:23:46.807 INFO:tasks.workunit.client.1.smithi134.stdout:7/515: dread d4/d8/fd [0,4194304] 0 2024-04-22T23:23:46.808 INFO:tasks.workunit.client.1.smithi134.stdout:8/461: mknod d7/db/d2d/d39/d58/c90 0 2024-04-22T23:23:46.809 INFO:tasks.workunit.client.1.smithi134.stdout:0/442: dwrite d1/d5/dd/d11/f18 [0,4194304] 0 2024-04-22T23:23:46.816 INFO:tasks.workunit.client.1.smithi134.stdout:3/334: dwrite dc/dd/d28/d2f/f49 [0,4194304] 0 2024-04-22T23:23:46.816 INFO:tasks.workunit.client.1.smithi134.stdout:6/531: rename d0/d19/f1b to d0/d19/de/fb0 0 2024-04-22T23:23:46.816 INFO:tasks.workunit.client.1.smithi134.stdout:9/399: dwrite d0/f5c [4194304,4194304] 0 2024-04-22T23:23:46.818 INFO:tasks.workunit.client.1.smithi134.stdout:2/414: dwrite d2/d15/d24/d1d/f5a [0,4194304] 0 2024-04-22T23:23:46.821 INFO:tasks.workunit.client.1.smithi134.stdout:8/462: truncate d7/db/d42/d65/f4c 5419840 0 2024-04-22T23:23:46.822 INFO:tasks.workunit.client.1.smithi134.stdout:8/463: readlink d7/db/l7d 0 2024-04-22T23:23:46.827 INFO:tasks.workunit.client.1.smithi134.stdout:6/532: fsync d0/d6f/d88/d8b/f91 0 2024-04-22T23:23:46.827 INFO:tasks.workunit.client.1.smithi134.stdout:0/443: unlink d1/d2/l73 0 2024-04-22T23:23:46.827 INFO:tasks.workunit.client.1.smithi134.stdout:9/400: chown d0/d2d/d3d/f72 555575 1 2024-04-22T23:23:46.828 INFO:tasks.workunit.client.1.smithi134.stdout:1/600: dwrite db/d15/d17/d20/d2a/f9f [0,4194304] 0 2024-04-22T23:23:46.835 INFO:tasks.workunit.client.1.smithi134.stdout:9/401: fsync d0/d2d/f6b 0 2024-04-22T23:23:46.837 INFO:tasks.workunit.client.1.smithi134.stdout:4/637: link d4/d35/d80/f90 d4/d7/d32/d3a/dbc/dc4/fc8 0 2024-04-22T23:23:46.837 INFO:tasks.workunit.client.1.smithi134.stdout:8/464: unlink d7/db/d2d/d39/d58/d7b/c8b 0 2024-04-22T23:23:46.839 INFO:tasks.workunit.client.1.smithi134.stdout:4/638: chown d4/d7/d32/d3a/dbc 34372061 1 2024-04-22T23:23:46.840 INFO:tasks.workunit.client.1.smithi134.stdout:3/335: dwrite dc/d2a/d30/f60 [0,4194304] 0 2024-04-22T23:23:46.842 INFO:tasks.workunit.client.1.smithi134.stdout:1/601: mknod db/d15/d17/d20/d2a/d2d/cd3 0 2024-04-22T23:23:46.842 INFO:tasks.workunit.client.1.smithi134.stdout:0/444: dread d1/d5/dd/f2a [0,4194304] 0 2024-04-22T23:23:46.846 INFO:tasks.workunit.client.1.smithi134.stdout:6/533: rename d0/d19/de/d12/d5e/f66 to d0/d19/de/d12/d5e/d6d/d7a/d99/fb1 0 2024-04-22T23:23:46.849 INFO:tasks.workunit.client.1.smithi134.stdout:8/465: creat d7/dc/d2c/d4f/d6c/d6e/f91 x:0 0 0 2024-04-22T23:23:46.852 INFO:tasks.workunit.client.1.smithi134.stdout:6/534: mkdir d0/d19/de/d12/d5e/d6d/d7a/db2 0 2024-04-22T23:23:46.853 INFO:tasks.workunit.client.1.smithi134.stdout:6/535: chown d0/f42 616 1 2024-04-22T23:23:46.855 INFO:tasks.workunit.client.1.smithi134.stdout:0/445: rename d1/d5/dd/d20/l43 to d1/d79/l8d 0 2024-04-22T23:23:46.858 INFO:tasks.workunit.client.1.smithi134.stdout:4/639: link d4/d15/f67 d4/d7/d32/d3a/d93/fc9 0 2024-04-22T23:23:46.859 INFO:tasks.workunit.client.1.smithi134.stdout:4/640: dread d4/d13/f74 [0,4194304] 0 2024-04-22T23:23:46.860 INFO:tasks.workunit.client.1.smithi134.stdout:1/602: mknod db/d15/d17/d20/d2a/d2d/d2e/d42/cd4 0 2024-04-22T23:23:46.860 INFO:tasks.workunit.client.1.smithi134.stdout:6/536: creat d0/d4f/fb3 x:0 0 0 2024-04-22T23:23:46.864 INFO:tasks.workunit.client.1.smithi134.stdout:8/466: rename d7/db/d42/d65/f7e to d7/db/d2d/d39/d58/d7b/f92 0 2024-04-22T23:23:46.865 INFO:tasks.workunit.client.1.smithi134.stdout:9/402: link d0/d2d/d4e/f62 d0/d2d/d82/f83 0 2024-04-22T23:23:46.865 INFO:tasks.workunit.client.1.smithi134.stdout:8/467: chown d7/db 12466481 1 2024-04-22T23:23:46.867 INFO:tasks.workunit.client.1.smithi134.stdout:2/415: dread d2/d15/d24/f1e [0,4194304] 0 2024-04-22T23:23:46.871 INFO:tasks.workunit.client.1.smithi134.stdout:1/603: dwrite db/d15/d17/f5a [0,4194304] 0 2024-04-22T23:23:46.872 INFO:tasks.workunit.client.1.smithi134.stdout:6/537: symlink d0/d19/de/d12/d5e/lb4 0 2024-04-22T23:23:46.874 INFO:tasks.workunit.client.1.smithi134.stdout:9/403: symlink d0/d7/d12/d22/l84 0 2024-04-22T23:23:46.876 INFO:tasks.workunit.client.1.smithi134.stdout:3/336: truncate dc/dd/d28/d2f/d21/d42/f47 3963384 0 2024-04-22T23:23:46.883 INFO:tasks.workunit.client.1.smithi134.stdout:1/604: dread db/db0/dc1/fcf [0,4194304] 0 2024-04-22T23:23:46.886 INFO:tasks.workunit.client.1.smithi134.stdout:3/337: dwrite dc/dd/d28/d3b/f79 [0,4194304] 0 2024-04-22T23:23:46.891 INFO:tasks.workunit.client.1.smithi134.stdout:6/538: mkdir d0/d19/de/d12/d29/db5 0 2024-04-22T23:23:46.894 INFO:tasks.workunit.client.1.smithi134.stdout:3/338: dwrite f9 [0,4194304] 0 2024-04-22T23:23:46.902 INFO:tasks.workunit.client.1.smithi134.stdout:3/339: mkdir dc/dd/d28/d2f/d36/d6d/d80 0 2024-04-22T23:23:46.907 INFO:tasks.workunit.client.1.smithi134.stdout:4/641: dread d4/dd/f31 [0,4194304] 0 2024-04-22T23:23:46.909 INFO:tasks.workunit.client.1.smithi134.stdout:2/416: truncate d2/dd/d4a/f52 2517360 0 2024-04-22T23:23:46.910 INFO:tasks.workunit.client.1.smithi134.stdout:2/417: fdatasync d2/f3e 0 2024-04-22T23:23:46.910 INFO:tasks.workunit.client.1.smithi134.stdout:4/642: creat d4/d13/fca x:0 0 0 2024-04-22T23:23:46.910 INFO:tasks.workunit.client.1.smithi134.stdout:1/605: dread db/d16/f5b [0,4194304] 0 2024-04-22T23:23:46.912 INFO:tasks.workunit.client.1.smithi134.stdout:1/606: readlink db/d15/d17/d20/l77 0 2024-04-22T23:23:46.914 INFO:tasks.workunit.client.1.smithi134.stdout:3/340: dread dc/f18 [0,4194304] 0 2024-04-22T23:23:46.914 INFO:tasks.workunit.client.1.smithi134.stdout:4/643: creat d4/d7/d32/d84/fcb x:0 0 0 2024-04-22T23:23:46.915 INFO:tasks.workunit.client.1.smithi134.stdout:2/418: mknod d2/d15/d24/d1d/d3d/d7b/c84 0 2024-04-22T23:23:46.916 INFO:tasks.workunit.client.1.smithi134.stdout:4/644: chown d4/d7/d32/d3a/d93/fc9 0 1 2024-04-22T23:23:46.917 INFO:tasks.workunit.client.1.smithi134.stdout:1/607: creat db/d16/d87/fd5 x:0 0 0 2024-04-22T23:23:46.918 INFO:tasks.workunit.client.1.smithi134.stdout:1/608: fsync db/f26 0 2024-04-22T23:23:46.918 INFO:tasks.workunit.client.1.smithi134.stdout:3/341: symlink dc/dd/d28/d2f/d36/d6d/l81 0 2024-04-22T23:23:46.920 INFO:tasks.workunit.client.1.smithi134.stdout:4/645: truncate d4/d35/d8e/f75 861780 0 2024-04-22T23:23:46.921 INFO:tasks.workunit.client.1.smithi134.stdout:2/419: creat d2/dd/df/f85 x:0 0 0 2024-04-22T23:23:46.925 INFO:tasks.workunit.client.1.smithi134.stdout:4/646: rename d4/d15 to d4/d29/dcc 0 2024-04-22T23:23:46.926 INFO:tasks.workunit.client.1.smithi134.stdout:2/420: creat d2/d15/d24/d1d/f86 x:0 0 0 2024-04-22T23:23:46.927 INFO:tasks.workunit.client.1.smithi134.stdout:3/342: dwrite dc/dd/d28/f39 [0,4194304] 0 2024-04-22T23:23:46.929 INFO:tasks.workunit.client.1.smithi134.stdout:4/647: read - d4/d7/d32/d84/fc0 zero size 2024-04-22T23:23:46.930 INFO:tasks.workunit.client.1.smithi134.stdout:1/609: dread db/d15/d17/d20/f65 [0,4194304] 0 2024-04-22T23:23:46.931 INFO:tasks.workunit.client.1.smithi134.stdout:3/343: stat dc/dd/d28/d2f/l58 0 2024-04-22T23:23:46.934 INFO:tasks.workunit.client.1.smithi134.stdout:2/421: creat d2/f87 x:0 0 0 2024-04-22T23:23:46.934 INFO:tasks.workunit.client.1.smithi134.stdout:9/404: dread d0/d7/d23/f2a [0,4194304] 0 2024-04-22T23:23:46.935 INFO:tasks.workunit.client.1.smithi134.stdout:5/558: sync 2024-04-22T23:23:46.935 INFO:tasks.workunit.client.1.smithi134.stdout:4/648: creat d4/d29/dcc/d82/fcd x:0 0 0 2024-04-22T23:23:46.936 INFO:tasks.workunit.client.1.smithi134.stdout:0/446: dread d1/d2/f38 [0,4194304] 0 2024-04-22T23:23:46.939 INFO:tasks.workunit.client.1.smithi134.stdout:5/559: creat d9/d11/fc7 x:0 0 0 2024-04-22T23:23:46.940 INFO:tasks.workunit.client.1.smithi134.stdout:5/560: chown d9 2276 1 2024-04-22T23:23:46.940 INFO:tasks.workunit.client.1.smithi134.stdout:0/447: symlink d1/d5/dd/d68/l8e 0 2024-04-22T23:23:46.941 INFO:tasks.workunit.client.1.smithi134.stdout:6/539: dread d0/d19/de/d12/f7e [0,4194304] 0 2024-04-22T23:23:46.943 INFO:tasks.workunit.client.1.smithi134.stdout:4/649: unlink d4/d29/dcc/f6a 0 2024-04-22T23:23:46.944 INFO:tasks.workunit.client.1.smithi134.stdout:1/610: dwrite db/d15/d17/d20/d2a/d2d/d86/fa4 [0,4194304] 0 2024-04-22T23:23:46.947 INFO:tasks.workunit.client.1.smithi134.stdout:4/650: chown d4/d35/d8e/d5d/cb0 0 1 2024-04-22T23:23:46.947 INFO:tasks.workunit.client.1.smithi134.stdout:5/561: fsync d9/d11/d34/f51 0 2024-04-22T23:23:46.948 INFO:tasks.workunit.client.1.smithi134.stdout:0/448: write d1/db/f70 [375093,124363] 0 2024-04-22T23:23:46.949 INFO:tasks.workunit.client.1.smithi134.stdout:9/405: link d0/d7/l13 d0/d7/d23/d26/d4d/d5d/l85 0 2024-04-22T23:23:46.949 INFO:tasks.workunit.client.1.smithi134.stdout:1/611: chown db/d15/f54 2 1 2024-04-22T23:23:46.952 INFO:tasks.workunit.client.1.smithi134.stdout:6/540: rename d0/d19/de/d12/d2a/d2d/d3a/da7 to d0/d19/db/d33/d86/db6 0 2024-04-22T23:23:46.953 INFO:tasks.workunit.client.1.smithi134.stdout:4/651: mkdir d4/d35/d8e/d5d/dce 0 2024-04-22T23:23:46.954 INFO:tasks.workunit.client.1.smithi134.stdout:0/449: creat d1/d5/f8f x:0 0 0 2024-04-22T23:23:46.955 INFO:tasks.workunit.client.1.smithi134.stdout:0/450: readlink d1/db/l13 0 2024-04-22T23:23:46.958 INFO:tasks.workunit.client.1.smithi134.stdout:9/406: creat d0/d2d/d3d/f86 x:0 0 0 2024-04-22T23:23:46.961 INFO:tasks.workunit.client.1.smithi134.stdout:9/407: dread d0/d7/f2c [0,4194304] 0 2024-04-22T23:23:46.962 INFO:tasks.workunit.client.1.smithi134.stdout:4/652: mkdir d4/d7a/dcf 0 2024-04-22T23:23:46.965 INFO:tasks.workunit.client.1.smithi134.stdout:9/408: creat d0/d2d/d3d/f87 x:0 0 0 2024-04-22T23:23:46.967 INFO:tasks.workunit.client.1.smithi134.stdout:9/409: truncate d0/d2d/d4e/f32 8776976 0 2024-04-22T23:23:46.975 INFO:tasks.workunit.client.1.smithi134.stdout:6/541: creat d0/d19/de/d12/d2a/fb7 x:0 0 0 2024-04-22T23:23:46.977 INFO:tasks.workunit.client.1.smithi134.stdout:9/410: mkdir d0/d7/d23/d26/d51/d88 0 2024-04-22T23:23:46.981 INFO:tasks.workunit.client.1.smithi134.stdout:1/612: rename db/d15/d72/cba to db/d15/d17/cd6 0 2024-04-22T23:23:46.987 INFO:tasks.workunit.client.1.smithi134.stdout:6/542: link d0/d19/de/f13 d0/d19/de/d2f/fb8 0 2024-04-22T23:23:46.992 INFO:tasks.workunit.client.1.smithi134.stdout:6/543: unlink d0/d19/de/d12/d2a/f30 0 2024-04-22T23:23:46.993 INFO:tasks.workunit.client.1.smithi134.stdout:0/451: dread d1/d2/d1d/f5f [4194304,4194304] 0 2024-04-22T23:23:46.996 INFO:tasks.workunit.client.1.smithi134.stdout:6/544: symlink d0/d19/de/d12/d29/d74/lb9 0 2024-04-22T23:23:46.996 INFO:tasks.workunit.client.1.smithi134.stdout:2/422: truncate d2/dd/df/d1c/f79 3784254 0 2024-04-22T23:23:46.997 INFO:tasks.workunit.client.1.smithi134.stdout:6/545: dread - d0/d19/db/d33/f95 zero size 2024-04-22T23:23:46.997 INFO:tasks.workunit.client.1.smithi134.stdout:2/423: write d2/f60 [506075,94334] 0 2024-04-22T23:23:46.999 INFO:tasks.workunit.client.1.smithi134.stdout:0/452: mknod d1/d5/dd/d20/d2c/c90 0 2024-04-22T23:23:47.000 INFO:tasks.workunit.client.1.smithi134.stdout:2/424: fsync d2/d15/d24/d1d/f5a 0 2024-04-22T23:23:47.000 INFO:tasks.workunit.client.1.smithi134.stdout:2/425: fdatasync d2/d15/d24/f69 0 2024-04-22T23:23:47.001 INFO:tasks.workunit.client.1.smithi134.stdout:6/546: symlink d0/d19/lba 0 2024-04-22T23:23:47.003 INFO:tasks.workunit.client.1.smithi134.stdout:0/453: creat d1/d2/d1d/f91 x:0 0 0 2024-04-22T23:23:47.005 INFO:tasks.workunit.client.1.smithi134.stdout:5/562: write d9/dd/d88/db7/fbf [5104116,8613] 0 2024-04-22T23:23:47.009 INFO:tasks.workunit.client.1.smithi134.stdout:5/563: chown d9/d11/d34/d4d/l93 17 1 2024-04-22T23:23:47.010 INFO:tasks.workunit.client.1.smithi134.stdout:2/426: fdatasync d2/dd/f47 0 2024-04-22T23:23:47.010 INFO:tasks.workunit.client.1.smithi134.stdout:6/547: mkdir d0/d19/de/d12/d2a/dbb 0 2024-04-22T23:23:47.015 INFO:tasks.workunit.client.1.smithi134.stdout:2/427: fdatasync d2/d15/d24/d1d/f36 0 2024-04-22T23:23:47.021 INFO:tasks.workunit.client.1.smithi134.stdout:0/454: dwrite d1/d2/f74 [0,4194304] 0 2024-04-22T23:23:47.021 INFO:tasks.workunit.client.1.smithi134.stdout:9/411: truncate d0/fa 2590991 0 2024-04-22T23:23:47.026 INFO:tasks.workunit.client.1.smithi134.stdout:1/613: dread db/f14 [0,4194304] 0 2024-04-22T23:23:47.027 INFO:tasks.workunit.client.1.smithi134.stdout:4/653: dwrite d4/dd/f16 [0,4194304] 0 2024-04-22T23:23:47.027 INFO:tasks.workunit.client.1.smithi134.stdout:0/455: read - d1/d77/f8c zero size 2024-04-22T23:23:47.029 INFO:tasks.workunit.client.1.smithi134.stdout:2/428: creat d2/dd/d68/f88 x:0 0 0 2024-04-22T23:23:47.031 INFO:tasks.workunit.client.1.smithi134.stdout:0/456: write d1/d2/d1d/f69 [730651,112626] 0 2024-04-22T23:23:47.033 INFO:tasks.workunit.client.1.smithi134.stdout:2/429: fsync d2/dd/f57 0 2024-04-22T23:23:47.033 INFO:tasks.workunit.client.1.smithi134.stdout:6/548: dwrite d0/d19/db/f68 [0,4194304] 0 2024-04-22T23:23:47.035 INFO:tasks.workunit.client.1.smithi134.stdout:4/654: mknod d4/d13/cd0 0 2024-04-22T23:23:47.037 INFO:tasks.workunit.client.1.smithi134.stdout:9/412: creat d0/d37/d76/f89 x:0 0 0 2024-04-22T23:23:47.040 INFO:tasks.workunit.client.1.smithi134.stdout:4/655: creat d4/d7a/fd1 x:0 0 0 2024-04-22T23:23:47.040 INFO:tasks.workunit.client.1.smithi134.stdout:4/656: fsync d4/d29/f77 0 2024-04-22T23:23:47.044 INFO:tasks.workunit.client.1.smithi134.stdout:0/457: mknod d1/d79/d88/c92 0 2024-04-22T23:23:47.050 INFO:tasks.workunit.client.1.smithi134.stdout:2/430: mkdir d2/dd/df/d89 0 2024-04-22T23:23:47.053 INFO:tasks.workunit.client.1.smithi134.stdout:5/564: dread d9/d1e/d90/fa1 [0,4194304] 0 2024-04-22T23:23:47.053 INFO:tasks.workunit.client.1.smithi134.stdout:0/458: creat d1/d79/f93 x:0 0 0 2024-04-22T23:23:47.055 INFO:tasks.workunit.client.1.smithi134.stdout:2/431: mkdir d2/d15/d24/d3b/d8a 0 2024-04-22T23:23:47.056 INFO:tasks.workunit.client.1.smithi134.stdout:5/565: rmdir d9/d11/d34/d85 39 2024-04-22T23:23:47.057 INFO:tasks.workunit.client.1.smithi134.stdout:0/459: mkdir d1/d5/d58/d94 0 2024-04-22T23:23:47.058 INFO:tasks.workunit.client.1.smithi134.stdout:2/432: symlink d2/d42/d54/l8b 0 2024-04-22T23:23:47.058 INFO:tasks.workunit.client.1.smithi134.stdout:0/460: readlink d1/d2/l37 0 2024-04-22T23:23:47.058 INFO:tasks.workunit.client.1.smithi134.stdout:2/433: fsync d2/d15/d24/f69 0 2024-04-22T23:23:47.059 INFO:tasks.workunit.client.1.smithi134.stdout:5/566: creat d9/dbb/fc8 x:0 0 0 2024-04-22T23:23:47.060 INFO:tasks.workunit.client.1.smithi134.stdout:5/567: write d9/d11/f1a [1701863,92434] 0 2024-04-22T23:23:47.061 INFO:tasks.workunit.client.1.smithi134.stdout:5/568: dread - d9/d11/d34/fa3 zero size 2024-04-22T23:23:47.063 INFO:tasks.workunit.client.1.smithi134.stdout:0/461: dread d1/f44 [0,4194304] 0 2024-04-22T23:23:47.063 INFO:tasks.workunit.client.1.smithi134.stdout:5/569: write d9/d1e/f35 [5034789,28580] 0 2024-04-22T23:23:47.065 INFO:tasks.workunit.client.1.smithi134.stdout:5/570: stat d9/d1e/f33 0 2024-04-22T23:23:47.065 INFO:tasks.workunit.client.1.smithi134.stdout:5/571: read - d9/dbb/fc8 zero size 2024-04-22T23:23:47.066 INFO:tasks.workunit.client.1.smithi134.stdout:0/462: creat d1/f95 x:0 0 0 2024-04-22T23:23:47.068 INFO:tasks.workunit.client.1.smithi134.stdout:0/463: truncate d1/d5/dd/d41/f53 807734 0 2024-04-22T23:23:47.069 INFO:tasks.workunit.client.1.smithi134.stdout:5/572: creat d9/d11/d34/d85/fc9 x:0 0 0 2024-04-22T23:23:47.070 INFO:tasks.workunit.client.1.smithi134.stdout:0/464: write d1/f9 [2426056,21885] 0 2024-04-22T23:23:47.071 INFO:tasks.workunit.client.1.smithi134.stdout:2/434: dread d2/dd/d4a/f52 [0,4194304] 0 2024-04-22T23:23:47.071 INFO:tasks.workunit.client.1.smithi134.stdout:5/573: write d9/f9d [537353,110247] 0 2024-04-22T23:23:47.073 INFO:tasks.workunit.client.1.smithi134.stdout:6/549: truncate d0/f16 2583734 0 2024-04-22T23:23:47.074 INFO:tasks.workunit.client.1.smithi134.stdout:1/614: truncate db/d15/d17/d20/d2a/f9f 3618902 0 2024-04-22T23:23:47.075 INFO:tasks.workunit.client.1.smithi134.stdout:2/435: rmdir d2/dd/d4a/d73 39 2024-04-22T23:23:47.078 INFO:tasks.workunit.client.1.smithi134.stdout:5/574: symlink d9/dd/d17/d2a/lca 0 2024-04-22T23:23:47.078 INFO:tasks.workunit.client.1.smithi134.stdout:5/575: chown l7 348892204 1 2024-04-22T23:23:47.079 INFO:tasks.workunit.client.1.smithi134.stdout:6/550: truncate d0/d19/db/d35/f79 222495 0 2024-04-22T23:23:47.082 INFO:tasks.workunit.client.1.smithi134.stdout:9/413: truncate d0/d7/f19 1792551 0 2024-04-22T23:23:47.084 INFO:tasks.workunit.client.1.smithi134.stdout:6/551: dread - d0/d19/de/d12/d2a/d2d/f72 zero size 2024-04-22T23:23:47.084 INFO:tasks.workunit.client.1.smithi134.stdout:1/615: dwrite db/d15/d17/f5a [0,4194304] 0 2024-04-22T23:23:47.087 INFO:tasks.workunit.client.1.smithi134.stdout:5/576: rename d9/dd/f77 to d9/dd/d17/d2a/d49/dc2/fcb 0 2024-04-22T23:23:47.087 INFO:tasks.workunit.client.1.smithi134.stdout:1/616: fsync fa 0 2024-04-22T23:23:47.089 INFO:tasks.workunit.client.1.smithi134.stdout:4/657: dwrite d4/d35/d5e/f4a [0,4194304] 0 2024-04-22T23:23:47.090 INFO:tasks.workunit.client.1.smithi134.stdout:9/414: creat d0/d7/d23/d26/d4d/d5d/f8a x:0 0 0 2024-04-22T23:23:47.100 INFO:tasks.workunit.client.1.smithi134.stdout:2/436: dread d2/f17 [0,4194304] 0 2024-04-22T23:23:47.102 INFO:tasks.workunit.client.1.smithi134.stdout:5/577: symlink d9/d11/d73/d95/d20/d60/lcc 0 2024-04-22T23:23:47.104 INFO:tasks.workunit.client.1.smithi134.stdout:1/617: mknod db/d15/d17/d20/d2a/d2d/d86/dad/cd7 0 2024-04-22T23:23:47.105 INFO:tasks.workunit.client.1.smithi134.stdout:2/437: symlink d2/dd/d68/l8c 0 2024-04-22T23:23:47.106 INFO:tasks.workunit.client.1.smithi134.stdout:4/658: mkdir d4/d7/d32/d3a/dbc/dc4/dd2 0 2024-04-22T23:23:47.107 INFO:tasks.workunit.client.1.smithi134.stdout:5/578: symlink d9/d11/d34/d85/lcd 0 2024-04-22T23:23:47.107 INFO:tasks.workunit.client.1.smithi134.stdout:4/659: write d4/f42 [4831166,116453] 0 2024-04-22T23:23:47.109 INFO:tasks.workunit.client.1.smithi134.stdout:4/660: stat d4/d35/d8e/d6e/cb9 0 2024-04-22T23:23:47.110 INFO:tasks.workunit.client.1.smithi134.stdout:0/465: dread d1/d5/dd/d20/f6c [0,4194304] 0 2024-04-22T23:23:47.110 INFO:tasks.workunit.client.1.smithi134.stdout:2/438: creat d2/d15/d24/d1d/f8d x:0 0 0 2024-04-22T23:23:47.111 INFO:tasks.workunit.client.1.smithi134.stdout:0/466: write d1/d5/f8b [1027730,87347] 0 2024-04-22T23:23:47.113 INFO:tasks.workunit.client.1.smithi134.stdout:4/661: symlink d4/d35/db4/ld3 0 2024-04-22T23:23:47.113 INFO:tasks.workunit.client.1.smithi134.stdout:0/467: truncate d1/d5/f5e 132895 0 2024-04-22T23:23:47.114 INFO:tasks.workunit.client.1.smithi134.stdout:2/439: rename d2/dd/df/f53 to d2/d42/d54/d33/d6e/f8e 0 2024-04-22T23:23:47.115 INFO:tasks.workunit.client.1.smithi134.stdout:0/468: readlink d1/d5/dd/d41/l4d 0 2024-04-22T23:23:47.115 INFO:tasks.workunit.client.1.smithi134.stdout:2/440: chown d2/d15/d24/d1d/f8d 24 1 2024-04-22T23:23:47.116 INFO:tasks.workunit.client.1.smithi134.stdout:0/469: write d1/d5/dd/f16 [2922610,10970] 0 2024-04-22T23:23:47.116 INFO:tasks.workunit.client.1.smithi134.stdout:5/579: dread d9/d11/f7a [0,4194304] 0 2024-04-22T23:23:47.122 INFO:tasks.workunit.client.1.smithi134.stdout:4/662: dwrite d4/f6 [0,4194304] 0 2024-04-22T23:23:47.122 INFO:tasks.workunit.client.1.smithi134.stdout:5/580: mkdir d9/d11/d34/d4d/d92/dce 0 2024-04-22T23:23:47.123 INFO:tasks.workunit.client.1.smithi134.stdout:2/441: link d2/l61 d2/dd/d68/l8f 0 2024-04-22T23:23:47.126 INFO:tasks.workunit.client.1.smithi134.stdout:5/581: write d9/dd/d17/d2a/d49/fad [903522,122515] 0 2024-04-22T23:23:47.128 INFO:tasks.workunit.client.1.smithi134.stdout:5/582: chown d9/dd/d17/d2a/f98 101688 1 2024-04-22T23:23:47.128 INFO:tasks.workunit.client.1.smithi134.stdout:6/552: dread d0/d19/de/d1f/f2c [0,4194304] 0 2024-04-22T23:23:47.130 INFO:tasks.workunit.client.1.smithi134.stdout:5/583: fdatasync d9/d11/d73/d95/d20/d60/f7d 0 2024-04-22T23:23:47.130 INFO:tasks.workunit.client.1.smithi134.stdout:2/442: mknod d2/d42/d54/d33/c90 0 2024-04-22T23:23:47.131 INFO:tasks.workunit.client.1.smithi134.stdout:2/443: readlink d2/dd/d4a/l55 0 2024-04-22T23:23:47.132 INFO:tasks.workunit.client.1.smithi134.stdout:6/553: mkdir d0/d19/db/d35/dbc 0 2024-04-22T23:23:47.133 INFO:tasks.workunit.client.1.smithi134.stdout:6/554: chown d0/d19/de/f31 31 1 2024-04-22T23:23:47.134 INFO:tasks.workunit.client.1.smithi134.stdout:6/555: stat d0/f60 0 2024-04-22T23:23:47.136 INFO:tasks.workunit.client.1.smithi134.stdout:6/556: dread - d0/d19/de/d12/d29/d74/da0/fab zero size 2024-04-22T23:23:47.136 INFO:tasks.workunit.client.1.smithi134.stdout:4/663: creat d4/d35/d8e/fd4 x:0 0 0 2024-04-22T23:23:47.137 INFO:tasks.workunit.client.1.smithi134.stdout:9/415: write d0/d7/f9 [719164,126378] 0 2024-04-22T23:23:47.140 INFO:tasks.workunit.client.1.smithi134.stdout:5/584: mknod d9/dd/ccf 0 2024-04-22T23:23:47.140 INFO:tasks.workunit.client.1.smithi134.stdout:6/557: read d0/d19/de/d12/d2a/f3b [926892,102597] 0 2024-04-22T23:23:47.142 INFO:tasks.workunit.client.1.smithi134.stdout:1/618: dread db/d16/d56/f63 [0,4194304] 0 2024-04-22T23:23:47.143 INFO:tasks.workunit.client.1.smithi134.stdout:1/619: chown db/d16/f5b 65544 1 2024-04-22T23:23:47.144 INFO:tasks.workunit.client.1.smithi134.stdout:2/444: dread d2/dd/f47 [0,4194304] 0 2024-04-22T23:23:47.147 INFO:tasks.workunit.client.1.smithi134.stdout:9/416: rename d0/f57 to d0/d37/f8b 0 2024-04-22T23:23:47.150 INFO:tasks.workunit.client.1.smithi134.stdout:2/445: dread d2/f2c [0,4194304] 0 2024-04-22T23:23:47.151 INFO:tasks.workunit.client.1.smithi134.stdout:4/664: fdatasync d4/d29/dcc/f54 0 2024-04-22T23:23:47.152 INFO:tasks.workunit.client.1.smithi134.stdout:1/620: dwrite db/d15/d17/fc8 [0,4194304] 0 2024-04-22T23:23:47.152 INFO:tasks.workunit.client.1.smithi134.stdout:2/446: fsync d2/f3e 0 2024-04-22T23:23:47.154 INFO:tasks.workunit.client.1.smithi134.stdout:4/665: fdatasync d4/d35/d8e/d5d/fa9 0 2024-04-22T23:23:47.155 INFO:tasks.workunit.client.1.smithi134.stdout:4/666: readlink d4/d35/d5e/d25/l8d 0 2024-04-22T23:23:47.155 INFO:tasks.workunit.client.1.smithi134.stdout:5/585: rename d9/d11/f7a to d9/dd/d17/d44/fd0 0 2024-04-22T23:23:47.158 INFO:tasks.workunit.client.1.smithi134.stdout:0/470: write d1/d5/dd/f2a [324560,41118] 0 2024-04-22T23:23:47.159 INFO:tasks.workunit.client.1.smithi134.stdout:2/447: mkdir d2/d42/d54/d33/d91 0 2024-04-22T23:23:47.162 INFO:tasks.workunit.client.1.smithi134.stdout:5/586: mkdir d9/d11/d34/d3b/dd1 0 2024-04-22T23:23:47.163 INFO:tasks.workunit.client.1.smithi134.stdout:2/448: dread d2/dd/d4a/f52 [0,4194304] 0 2024-04-22T23:23:47.164 INFO:tasks.workunit.client.1.smithi134.stdout:4/667: rename d4/d7/d32/d3a/dbc/dc4/dd2 to d4/d29/dcc/d82/dd5 0 2024-04-22T23:23:47.210 INFO:tasks.workunit.client.1.smithi134.stdout:7/516: sync 2024-04-22T23:23:47.210 INFO:tasks.workunit.client.1.smithi134.stdout:7/517: chown d4/dad 61178811 1 2024-04-22T23:23:47.210 INFO:tasks.workunit.client.1.smithi134.stdout:9/417: creat d0/d7/f8c x:0 0 0 2024-04-22T23:23:47.214 INFO:tasks.workunit.client.1.smithi134.stdout:2/449: truncate d2/f30 2242097 0 2024-04-22T23:23:47.215 INFO:tasks.workunit.client.1.smithi134.stdout:4/668: creat d4/d13/fd6 x:0 0 0 2024-04-22T23:23:47.217 INFO:tasks.workunit.client.1.smithi134.stdout:7/518: mkdir d4/d10/d2f/d34/dc0 0 2024-04-22T23:23:47.217 INFO:tasks.workunit.client.1.smithi134.stdout:9/418: dwrite d0/d7/d12/f42 [0,4194304] 0 2024-04-22T23:23:47.219 INFO:tasks.workunit.client.1.smithi134.stdout:5/587: symlink d9/dd/d88/db7/ld2 0 2024-04-22T23:23:47.224 INFO:tasks.workunit.client.1.smithi134.stdout:2/450: creat d2/d15/d24/d1d/d3d/f92 x:0 0 0 2024-04-22T23:23:47.224 INFO:tasks.workunit.client.1.smithi134.stdout:5/588: dread - d9/d11/d34/d4d/d92/fab zero size 2024-04-22T23:23:47.225 INFO:tasks.workunit.client.1.smithi134.stdout:1/621: link db/d15/d17/d20/d2a/d2d/d86/dad/cd7 db/d15/d17/cd8 0 2024-04-22T23:23:47.227 INFO:tasks.workunit.client.1.smithi134.stdout:2/451: mkdir d2/d42/d93 0 2024-04-22T23:23:47.227 INFO:tasks.workunit.client.1.smithi134.stdout:4/669: symlink d4/d35/d5e/ld7 0 2024-04-22T23:23:47.229 INFO:tasks.workunit.client.1.smithi134.stdout:5/589: chown d9/d1e/f6e 14344 1 2024-04-22T23:23:47.231 INFO:tasks.workunit.client.1.smithi134.stdout:9/419: dread d0/d37/f34 [0,4194304] 0 2024-04-22T23:23:47.231 INFO:tasks.workunit.client.1.smithi134.stdout:7/519: dwrite d4/d8/d2a/f2c [0,4194304] 0 2024-04-22T23:23:47.232 INFO:tasks.workunit.client.1.smithi134.stdout:6/558: dwrite d0/d19/de/f13 [0,4194304] 0 2024-04-22T23:23:47.233 INFO:tasks.workunit.client.1.smithi134.stdout:2/452: mkdir d2/d94 0 2024-04-22T23:23:47.236 INFO:tasks.workunit.client.1.smithi134.stdout:1/622: dread db/d16/d56/fa1 [0,4194304] 0 2024-04-22T23:23:47.239 INFO:tasks.workunit.client.1.smithi134.stdout:1/623: write db/d16/f3d [1738545,99338] 0 2024-04-22T23:23:47.239 INFO:tasks.workunit.client.1.smithi134.stdout:1/624: chown db/d15/d17/f9d 8486 1 2024-04-22T23:23:47.240 INFO:tasks.workunit.client.1.smithi134.stdout:5/590: unlink d9/d1e/f84 0 2024-04-22T23:23:47.241 INFO:tasks.workunit.client.1.smithi134.stdout:8/468: sync 2024-04-22T23:23:47.241 INFO:tasks.workunit.client.1.smithi134.stdout:3/344: sync 2024-04-22T23:23:47.244 INFO:tasks.workunit.client.1.smithi134.stdout:9/420: rename d0/d7/d23/d26/f29 to d0/d7/d23/d47/f8d 0 2024-04-22T23:23:47.246 INFO:tasks.workunit.client.1.smithi134.stdout:8/469: readlink d7/l1e 0 2024-04-22T23:23:47.247 INFO:tasks.workunit.client.1.smithi134.stdout:3/345: creat dc/dd/d28/d2f/d21/d42/f82 x:0 0 0 2024-04-22T23:23:47.252 INFO:tasks.workunit.client.1.smithi134.stdout:4/670: getdents d4 0 2024-04-22T23:23:47.253 INFO:tasks.workunit.client.1.smithi134.stdout:9/421: mkdir d0/d7/d23/d26/d4d/d5d/d8e 0 2024-04-22T23:23:47.254 INFO:tasks.workunit.client.1.smithi134.stdout:9/422: dread - d0/d37/d76/f89 zero size 2024-04-22T23:23:47.255 INFO:tasks.workunit.client.1.smithi134.stdout:8/470: symlink d7/db/l93 0 2024-04-22T23:23:47.257 INFO:tasks.workunit.client.1.smithi134.stdout:3/346: creat dc/d2a/d30/f83 x:0 0 0 2024-04-22T23:23:47.257 INFO:tasks.workunit.client.1.smithi134.stdout:3/347: dread - dc/d2a/d62/f65 zero size 2024-04-22T23:23:47.258 INFO:tasks.workunit.client.1.smithi134.stdout:6/559: rename d0/d19/de/l56 to d0/d19/db/d33/d86/lbd 0 2024-04-22T23:23:47.262 INFO:tasks.workunit.client.1.smithi134.stdout:8/471: unlink d7/db/d2d/d39/d58/f70 0 2024-04-22T23:23:47.264 INFO:tasks.workunit.client.1.smithi134.stdout:3/348: symlink dc/dd/d28/d34/d72/l84 0 2024-04-22T23:23:47.267 INFO:tasks.workunit.client.1.smithi134.stdout:1/625: rename db/d15/d17/d20/dd0 to db/db0/dc1/dd9 0 2024-04-22T23:23:47.268 INFO:tasks.workunit.client.1.smithi134.stdout:4/671: mkdir d4/d29/dcc/dd8 0 2024-04-22T23:23:47.270 INFO:tasks.workunit.client.1.smithi134.stdout:3/349: dwrite dc/dd/d28/d34/f50 [0,4194304] 0 2024-04-22T23:23:47.276 INFO:tasks.workunit.client.1.smithi134.stdout:5/591: rename d9/dd/f10 to d9/d11/d34/d85/fd3 0 2024-04-22T23:23:47.277 INFO:tasks.workunit.client.1.smithi134.stdout:3/350: dread f9 [0,4194304] 0 2024-04-22T23:23:47.279 INFO:tasks.workunit.client.1.smithi134.stdout:5/592: write d9/d11/d34/d85/fa5 [914168,124822] 0 2024-04-22T23:23:47.279 INFO:tasks.workunit.client.1.smithi134.stdout:1/626: read db/d15/f7c [2340732,32419] 0 2024-04-22T23:23:47.279 INFO:tasks.workunit.client.1.smithi134.stdout:8/472: creat d7/db/d2d/d39/d58/d8a/f94 x:0 0 0 2024-04-22T23:23:47.281 INFO:tasks.workunit.client.1.smithi134.stdout:1/627: read fa [950492,21057] 0 2024-04-22T23:23:47.281 INFO:tasks.workunit.client.1.smithi134.stdout:5/593: chown d9/dd/d17/l8f 120870 1 2024-04-22T23:23:47.282 INFO:tasks.workunit.client.1.smithi134.stdout:1/628: write fa [4161439,130934] 0 2024-04-22T23:23:47.289 INFO:tasks.workunit.client.1.smithi134.stdout:5/594: dwrite d9/f62 [0,4194304] 0 2024-04-22T23:23:47.289 INFO:tasks.workunit.client.1.smithi134.stdout:7/520: rename d4/d10 to d4/d8/d41/d76/dc1 0 2024-04-22T23:23:47.292 INFO:tasks.workunit.client.1.smithi134.stdout:9/423: link d0/d2d/d82/f83 d0/d7/d23/d26/d4d/d5d/f8f 0 2024-04-22T23:23:47.293 INFO:tasks.workunit.client.1.smithi134.stdout:9/424: readlink d0/d2d/d4e/l24 0 2024-04-22T23:23:47.297 INFO:tasks.workunit.client.1.smithi134.stdout:8/473: unlink d7/db/d2d/d39/c67 0 2024-04-22T23:23:47.298 INFO:tasks.workunit.client.1.smithi134.stdout:6/560: dwrite d0/f9d [0,4194304] 0 2024-04-22T23:23:47.299 INFO:tasks.workunit.client.1.smithi134.stdout:8/474: write d7/db/f3b [491225,48863] 0 2024-04-22T23:23:47.301 INFO:tasks.workunit.client.1.smithi134.stdout:1/629: rmdir db/d16/d56 39 2024-04-22T23:23:47.305 INFO:tasks.workunit.client.1.smithi134.stdout:2/453: rename d2/dd/d4a/d73/f78 to d2/d42/d93/f95 0 2024-04-22T23:23:47.307 INFO:tasks.workunit.client.1.smithi134.stdout:6/561: dwrite d0/f60 [4194304,4194304] 0 2024-04-22T23:23:47.308 INFO:tasks.workunit.client.1.smithi134.stdout:2/454: write d2/d15/d24/d1d/d3d/f58 [3720856,130027] 0 2024-04-22T23:23:47.310 INFO:tasks.workunit.client.1.smithi134.stdout:4/672: truncate d4/d35/fbb 2557623 0 2024-04-22T23:23:47.310 INFO:tasks.workunit.client.1.smithi134.stdout:3/351: creat dc/dd/d28/d34/d72/d75/f85 x:0 0 0 2024-04-22T23:23:47.310 INFO:tasks.workunit.client.1.smithi134.stdout:9/425: readlink d0/d7/d23/d26/d51/l80 0 2024-04-22T23:23:47.311 INFO:tasks.workunit.client.1.smithi134.stdout:3/352: stat dc/d2a/d62/f65 0 2024-04-22T23:23:47.312 INFO:tasks.workunit.client.1.smithi134.stdout:3/353: chown dc/dd/d28/d34/f50 40 1 2024-04-22T23:23:47.313 INFO:tasks.workunit.client.1.smithi134.stdout:7/521: rename d4/d8/d41/d76/dc1/d2f/d34/d54/d9c to d4/d8/d41/d76/dc1/d2f/d69/dab/d95/dc2 0 2024-04-22T23:23:47.318 INFO:tasks.workunit.client.1.smithi134.stdout:3/354: mkdir dc/d2a/d62/d86 0 2024-04-22T23:23:47.321 INFO:tasks.workunit.client.1.smithi134.stdout:7/522: creat d4/d8/d2a/d42/fc3 x:0 0 0 2024-04-22T23:23:47.321 INFO:tasks.workunit.client.1.smithi134.stdout:2/455: mknod d2/d15/c96 0 2024-04-22T23:23:47.323 INFO:tasks.workunit.client.1.smithi134.stdout:1/630: creat db/d16/fda x:0 0 0 2024-04-22T23:23:47.323 INFO:tasks.workunit.client.1.smithi134.stdout:3/355: symlink dc/d2a/d30/l87 0 2024-04-22T23:23:47.325 INFO:tasks.workunit.client.1.smithi134.stdout:7/523: write d4/f2e [4157134,5320] 0 2024-04-22T23:23:47.326 INFO:tasks.workunit.client.1.smithi134.stdout:1/631: creat db/db5/fdb x:0 0 0 2024-04-22T23:23:47.328 INFO:tasks.workunit.client.1.smithi134.stdout:9/426: rename d0/d7/d23/d1b/f49 to d0/d7/f90 0 2024-04-22T23:23:47.329 INFO:tasks.workunit.client.1.smithi134.stdout:7/524: mkdir d4/d8/d41/d76/dc1/d2f/d70/d86/dc4 0 2024-04-22T23:23:47.333 INFO:tasks.workunit.client.1.smithi134.stdout:2/456: dread d2/d15/d24/d3b/f4f [0,4194304] 0 2024-04-22T23:23:47.337 INFO:tasks.workunit.client.1.smithi134.stdout:9/427: rename d0/d7/d23/d26/d51/d4c/f79 to d0/d37/d76/f91 0 2024-04-22T23:23:47.339 INFO:tasks.workunit.client.1.smithi134.stdout:7/525: chown d4/f2d 217 1 2024-04-22T23:23:47.342 INFO:tasks.workunit.client.1.smithi134.stdout:9/428: fsync d0/d37/d76/f89 0 2024-04-22T23:23:47.343 INFO:tasks.workunit.client.1.smithi134.stdout:5/595: truncate d9/dd/d17/f71 139519 0 2024-04-22T23:23:47.344 INFO:tasks.workunit.client.1.smithi134.stdout:2/457: dwrite d2/f3e [0,4194304] 0 2024-04-22T23:23:47.347 INFO:tasks.workunit.client.1.smithi134.stdout:8/475: dread d7/dc/f2b [0,4194304] 0 2024-04-22T23:23:47.347 INFO:tasks.workunit.client.1.smithi134.stdout:6/562: truncate d0/d19/de/d12/f7e 1972402 0 2024-04-22T23:23:47.353 INFO:tasks.workunit.client.1.smithi134.stdout:5/596: rename d9/d11/d34/d3b/dd1 to d9/d11/d34/d4d/dd4 0 2024-04-22T23:23:47.353 INFO:tasks.workunit.client.1.smithi134.stdout:2/458: write d2/d15/d24/f29 [62572,115327] 0 2024-04-22T23:23:47.354 INFO:tasks.workunit.client.1.smithi134.stdout:5/597: dread - d9/d11/fc7 zero size 2024-04-22T23:23:47.357 INFO:tasks.workunit.client.1.smithi134.stdout:4/673: dwrite d4/d35/d8e/f2d [0,4194304] 0 2024-04-22T23:23:47.358 INFO:tasks.workunit.client.1.smithi134.stdout:1/632: truncate db/d15/d17/f5a 577913 0 2024-04-22T23:23:47.361 INFO:tasks.workunit.client.1.smithi134.stdout:1/633: stat db/d15/f29 0 2024-04-22T23:23:47.361 INFO:tasks.workunit.client.1.smithi134.stdout:6/563: dread d0/d6f/f84 [0,4194304] 0 2024-04-22T23:23:47.361 INFO:tasks.workunit.client.1.smithi134.stdout:7/526: dread d4/d8/d2a/f47 [0,4194304] 0 2024-04-22T23:23:47.364 INFO:tasks.workunit.client.1.smithi134.stdout:6/564: write d0/f42 [2226960,109975] 0 2024-04-22T23:23:47.366 INFO:tasks.workunit.client.1.smithi134.stdout:5/598: chown d9/dd/d88/c9a 4189742 1 2024-04-22T23:23:47.366 INFO:tasks.workunit.client.1.smithi134.stdout:2/459: write d2/f22 [1514998,105517] 0 2024-04-22T23:23:47.373 INFO:tasks.workunit.client.1.smithi134.stdout:7/527: read d4/d8/f22 [2907449,58638] 0 2024-04-22T23:23:47.375 INFO:tasks.workunit.client.1.smithi134.stdout:9/429: dread d0/d7/d23/f45 [0,4194304] 0 2024-04-22T23:23:47.376 INFO:tasks.workunit.client.1.smithi134.stdout:1/634: mknod db/d15/d17/dc9/cdc 0 2024-04-22T23:23:47.379 INFO:tasks.workunit.client.1.smithi134.stdout:6/565: fdatasync d0/f16 0 2024-04-22T23:23:47.381 INFO:tasks.workunit.client.1.smithi134.stdout:6/566: write d0/d19/de/d12/d29/f8d [1323859,87291] 0 2024-04-22T23:23:47.385 INFO:tasks.workunit.client.1.smithi134.stdout:7/528: rmdir d4/d8/d41/d76/dc1/d2f/d34/d62/d29 39 2024-04-22T23:23:47.387 INFO:tasks.workunit.client.1.smithi134.stdout:6/567: dwrite d0/d6f/f7b [0,4194304] 0 2024-04-22T23:23:47.393 INFO:tasks.workunit.client.1.smithi134.stdout:8/476: link d7/dc/d2c/c5d d7/dc/c95 0 2024-04-22T23:23:47.398 INFO:tasks.workunit.client.1.smithi134.stdout:2/460: creat d2/dd/df/f97 x:0 0 0 2024-04-22T23:23:47.399 INFO:tasks.workunit.client.1.smithi134.stdout:8/477: creat d7/db/d2d/d39/d58/f96 x:0 0 0 2024-04-22T23:23:47.400 INFO:tasks.workunit.client.1.smithi134.stdout:5/599: dwrite d9/d11/d34/d3b/f5f [0,4194304] 0 2024-04-22T23:23:47.400 INFO:tasks.workunit.client.1.smithi134.stdout:9/430: creat d0/d7/f92 x:0 0 0 2024-04-22T23:23:47.402 INFO:tasks.workunit.client.1.smithi134.stdout:1/635: rename db/d15/d17/d20/d2a/fb3 to db/d15/d17/d20/d2a/d2d/fdd 0 2024-04-22T23:23:47.402 INFO:tasks.workunit.client.1.smithi134.stdout:5/600: chown d9/dd/d17/d44/cc1 171 1 2024-04-22T23:23:47.402 INFO:tasks.workunit.client.1.smithi134.stdout:8/478: read d7/dc/d2c/d4f/f6b [4310641,18685] 0 2024-04-22T23:23:47.404 INFO:tasks.workunit.client.1.smithi134.stdout:8/479: chown d7/dc/f7f 130 1 2024-04-22T23:23:47.406 INFO:tasks.workunit.client.1.smithi134.stdout:6/568: truncate d0/f10 2914413 0 2024-04-22T23:23:47.407 INFO:tasks.workunit.client.1.smithi134.stdout:6/569: chown d0/d19/db/d35/f44 174076 1 2024-04-22T23:23:47.408 INFO:tasks.workunit.client.1.smithi134.stdout:6/570: write d0/d19/de/d12/d5e/d6d/f9f [831956,116218] 0 2024-04-22T23:23:47.411 INFO:tasks.workunit.client.1.smithi134.stdout:5/601: symlink d9/d11/d34/d85/ld5 0 2024-04-22T23:23:47.414 INFO:tasks.workunit.client.1.smithi134.stdout:9/431: mknod d0/d2d/c93 0 2024-04-22T23:23:47.415 INFO:tasks.workunit.client.1.smithi134.stdout:6/571: dwrite d0/d19/de/f31 [0,4194304] 0 2024-04-22T23:23:47.422 INFO:tasks.workunit.client.1.smithi134.stdout:5/602: write d9/d11/d34/d85/fd3 [995159,82895] 0 2024-04-22T23:23:47.423 INFO:tasks.workunit.client.1.smithi134.stdout:9/432: dwrite d0/d7/d27/f33 [0,4194304] 0 2024-04-22T23:23:47.425 INFO:tasks.workunit.client.1.smithi134.stdout:9/433: fsync d0/d7/d23/f55 0 2024-04-22T23:23:47.426 INFO:tasks.workunit.client.1.smithi134.stdout:2/461: rename d2/d42/d54/d33/d6e/f8e to d2/f98 0 2024-04-22T23:23:47.429 INFO:tasks.workunit.client.1.smithi134.stdout:1/636: dwrite db/d16/d56/f63 [0,4194304] 0 2024-04-22T23:23:47.432 INFO:tasks.workunit.client.1.smithi134.stdout:8/480: write d7/db/d2d/f54 [90891,114418] 0 2024-04-22T23:23:47.434 INFO:tasks.workunit.client.1.smithi134.stdout:6/572: mkdir d0/d19/de/dbe 0 2024-04-22T23:23:47.437 INFO:tasks.workunit.client.1.smithi134.stdout:8/481: write d7/dc/d2c/d4f/d6c/f81 [44432,99258] 0 2024-04-22T23:23:47.438 INFO:tasks.workunit.client.1.smithi134.stdout:9/434: dread d0/d7/f90 [0,4194304] 0 2024-04-22T23:23:47.443 INFO:tasks.workunit.client.1.smithi134.stdout:6/573: getdents d0/d6f/d88 0 2024-04-22T23:23:47.444 INFO:tasks.workunit.client.1.smithi134.stdout:5/603: dwrite d9/d11/d34/d85/fd3 [0,4194304] 0 2024-04-22T23:23:47.444 INFO:tasks.workunit.client.1.smithi134.stdout:8/482: readlink d7/db/d42/d65/l36 0 2024-04-22T23:23:47.444 INFO:tasks.workunit.client.1.smithi134.stdout:9/435: mkdir d0/d2d/d94 0 2024-04-22T23:23:47.446 INFO:tasks.workunit.client.1.smithi134.stdout:1/637: dwrite db/d15/d17/d20/d2a/d2d/f64 [4194304,4194304] 0 2024-04-22T23:23:47.448 INFO:tasks.workunit.client.1.smithi134.stdout:5/604: mknod d9/dd/d17/d44/cd6 0 2024-04-22T23:23:47.450 INFO:tasks.workunit.client.1.smithi134.stdout:8/483: creat d7/db/d2d/d39/d58/f97 x:0 0 0 2024-04-22T23:23:47.452 INFO:tasks.workunit.client.1.smithi134.stdout:1/638: symlink db/d16/d87/lde 0 2024-04-22T23:23:47.454 INFO:tasks.workunit.client.1.smithi134.stdout:6/574: creat d0/d19/de/d12/d5e/d6d/d7a/db2/fbf x:0 0 0 2024-04-22T23:23:47.457 INFO:tasks.workunit.client.1.smithi134.stdout:1/639: mknod db/d16/dbc/cdf 0 2024-04-22T23:23:47.460 INFO:tasks.workunit.client.1.smithi134.stdout:8/484: getdents d7/db/d2d/d39/d58/d8a 0 2024-04-22T23:23:47.462 INFO:tasks.workunit.client.1.smithi134.stdout:5/605: dwrite d9/dd/fbe [0,4194304] 0 2024-04-22T23:23:47.464 INFO:tasks.workunit.client.1.smithi134.stdout:2/462: write d2/d15/d24/d1d/d3d/d7b/f7c [4783505,75526] 0 2024-04-22T23:23:47.467 INFO:tasks.workunit.client.1.smithi134.stdout:6/575: dwrite d0/d19/db/d33/f62 [0,4194304] 0 2024-04-22T23:23:47.467 INFO:tasks.workunit.client.1.smithi134.stdout:1/640: symlink db/d15/d17/d20/d2a/d2d/le0 0 2024-04-22T23:23:47.471 INFO:tasks.workunit.client.1.smithi134.stdout:2/463: fdatasync d2/d15/d24/d1d/d3d/f71 0 2024-04-22T23:23:47.475 INFO:tasks.workunit.client.1.smithi134.stdout:8/485: symlink d7/db/d42/d65/l98 0 2024-04-22T23:23:47.477 INFO:tasks.workunit.client.1.smithi134.stdout:5/606: dwrite d9/d11/d34/d85/fa5 [0,4194304] 0 2024-04-22T23:23:47.479 INFO:tasks.workunit.client.1.smithi134.stdout:1/641: truncate db/d15/d17/d20/d2a/d2d/d2e/d42/f9a 707617 0 2024-04-22T23:23:47.483 INFO:tasks.workunit.client.1.smithi134.stdout:1/642: read db/f1b [1100995,70459] 0 2024-04-22T23:23:47.483 INFO:tasks.workunit.client.1.smithi134.stdout:5/607: mkdir d9/d11/d34/d4d/dd4/dd7 0 2024-04-22T23:23:47.484 INFO:tasks.workunit.client.1.smithi134.stdout:1/643: dread - db/db5/fc7 zero size 2024-04-22T23:23:47.485 INFO:tasks.workunit.client.1.smithi134.stdout:8/486: dwrite d7/dc/d2c/d37/f4d [0,4194304] 0 2024-04-22T23:23:47.486 INFO:tasks.workunit.client.1.smithi134.stdout:1/644: write db/d16/fda [317742,3369] 0 2024-04-22T23:23:47.487 INFO:tasks.workunit.client.1.smithi134.stdout:5/608: stat d9/f19 0 2024-04-22T23:23:47.488 INFO:tasks.workunit.client.1.smithi134.stdout:8/487: stat d7/dc/d2c/d4f/d6c/d6e/c73 0 2024-04-22T23:23:47.492 INFO:tasks.workunit.client.1.smithi134.stdout:1/645: mkdir db/d16/dbc/de1 0 2024-04-22T23:23:47.494 INFO:tasks.workunit.client.1.smithi134.stdout:8/488: rmdir d7/db/d2d/d39/d58/d7b 39 2024-04-22T23:23:47.495 INFO:tasks.workunit.client.1.smithi134.stdout:1/646: write db/f8b [1059601,76248] 0 2024-04-22T23:23:47.495 INFO:tasks.workunit.client.1.smithi134.stdout:8/489: fsync d7/dc/f7f 0 2024-04-22T23:23:47.496 INFO:tasks.workunit.client.1.smithi134.stdout:1/647: truncate db/db5/fc7 302162 0 2024-04-22T23:23:47.497 INFO:tasks.workunit.client.1.smithi134.stdout:8/490: creat d7/db/d2d/f99 x:0 0 0 2024-04-22T23:23:47.498 INFO:tasks.workunit.client.1.smithi134.stdout:8/491: chown d7/db/d42/f83 9 1 2024-04-22T23:23:47.499 INFO:tasks.workunit.client.1.smithi134.stdout:8/492: truncate d7/db/f3b 736351 0 2024-04-22T23:23:47.527 INFO:tasks.workunit.client.1.smithi134.stdout:8/493: mkdir d7/db/d42/d65/d46/d9a 0 2024-04-22T23:23:47.528 INFO:tasks.workunit.client.1.smithi134.stdout:6/576: truncate d0/d19/de/d12/f97 1877072 0 2024-04-22T23:23:47.529 INFO:tasks.workunit.client.1.smithi134.stdout:2/464: dread d2/d15/d24/d1d/d3d/d7b/f7c [0,4194304] 0 2024-04-22T23:23:47.532 INFO:tasks.workunit.client.1.smithi134.stdout:8/494: symlink d7/db/d42/d65/d46/d84/l9b 0 2024-04-22T23:23:47.532 INFO:tasks.workunit.client.1.smithi134.stdout:2/465: rmdir d2/d42/d93 39 2024-04-22T23:23:47.534 INFO:tasks.workunit.client.1.smithi134.stdout:6/577: mknod d0/d19/de/d12/d5e/d6d/cc0 0 2024-04-22T23:23:47.534 INFO:tasks.workunit.client.1.smithi134.stdout:8/495: write d7/dc/d2c/d37/f4d [4615365,86415] 0 2024-04-22T23:23:47.537 INFO:tasks.workunit.client.1.smithi134.stdout:6/578: mkdir d0/d19/de/d12/d5e/dc1 0 2024-04-22T23:23:47.538 INFO:tasks.workunit.client.1.smithi134.stdout:1/648: dread fa [0,4194304] 0 2024-04-22T23:23:47.539 INFO:tasks.workunit.client.1.smithi134.stdout:8/496: rename d7/db/d42/f83 to d7/dc/f9c 0 2024-04-22T23:23:47.540 INFO:tasks.workunit.client.1.smithi134.stdout:6/579: readlink d0/d19/db/d33/d86/lbd 0 2024-04-22T23:23:47.542 INFO:tasks.workunit.client.1.smithi134.stdout:6/580: stat d0/d19/de/d12/d2a/d2d/d3a/f77 0 2024-04-22T23:23:47.545 INFO:tasks.workunit.client.1.smithi134.stdout:1/649: dwrite db/d15/d17/d20/d2a/fc3 [0,4194304] 0 2024-04-22T23:23:47.546 INFO:tasks.workunit.client.1.smithi134.stdout:1/650: fsync db/d15/d17/d20/fab 0 2024-04-22T23:23:47.547 INFO:tasks.workunit.client.1.smithi134.stdout:8/497: dwrite f4 [0,4194304] 0 2024-04-22T23:23:47.548 INFO:tasks.workunit.client.1.smithi134.stdout:8/498: write d7/f1c [75750,112218] 0 2024-04-22T23:23:47.548 INFO:tasks.workunit.client.1.smithi134.stdout:1/651: dread - db/d15/d17/d20/d2a/d2d/d2e/f90 zero size 2024-04-22T23:23:47.549 INFO:tasks.workunit.client.1.smithi134.stdout:6/581: rename d0/d19/de/d12/d29/d74/l82 to d0/d19/db/d35/d8a/lc2 0 2024-04-22T23:23:47.549 INFO:tasks.workunit.client.1.smithi134.stdout:8/499: rename d7/dc to d7/dc/d19/d9d 22 2024-04-22T23:23:47.551 INFO:tasks.workunit.client.1.smithi134.stdout:6/582: write d0/d19/de/d2f/fb8 [2001090,107013] 0 2024-04-22T23:23:47.552 INFO:tasks.workunit.client.1.smithi134.stdout:8/500: truncate d7/dc/d2c/d4f/d6c/f81 1090660 0 2024-04-22T23:23:47.552 INFO:tasks.workunit.client.1.smithi134.stdout:6/583: readlink d0/d19/db/lc 0 2024-04-22T23:23:47.555 INFO:tasks.workunit.client.1.smithi134.stdout:1/652: link db/d16/f3d db/d16/d87/dbd/fe2 0 2024-04-22T23:23:47.556 INFO:tasks.workunit.client.1.smithi134.stdout:1/653: chown db/d15/d17/d20/f46 9829 1 2024-04-22T23:23:47.556 INFO:tasks.workunit.client.1.smithi134.stdout:8/501: getdents d7/dc/d2c/d4f/d6c/d86 0 2024-04-22T23:23:47.559 INFO:tasks.workunit.client.1.smithi134.stdout:8/502: creat d7/dc/d2c/d37/d3a/f9e x:0 0 0 2024-04-22T23:23:47.565 INFO:tasks.workunit.client.1.smithi134.stdout:6/584: dread d0/d19/de/d12/d29/d74/fa1 [0,4194304] 0 2024-04-22T23:23:47.565 INFO:tasks.workunit.client.1.smithi134.stdout:6/585: readlink d0/la6 0 2024-04-22T23:23:47.567 INFO:tasks.workunit.client.1.smithi134.stdout:6/586: mknod d0/d19/db/d33/cc3 0 2024-04-22T23:23:47.569 INFO:tasks.workunit.client.1.smithi134.stdout:6/587: truncate d0/d19/de/fb0 855523 0 2024-04-22T23:23:47.574 INFO:tasks.workunit.client.1.smithi134.stdout:6/588: dwrite d0/d19/db/f68 [0,4194304] 0 2024-04-22T23:23:47.599 INFO:tasks.workunit.client.1.smithi134.stdout:1/654: dread db/d15/d17/f69 [0,4194304] 0 2024-04-22T23:23:47.601 INFO:tasks.workunit.client.1.smithi134.stdout:1/655: creat db/d15/fe3 x:0 0 0 2024-04-22T23:23:47.603 INFO:tasks.workunit.client.1.smithi134.stdout:1/656: dread db/d15/d17/d20/d2a/d2d/d2e/d42/f9a [0,4194304] 0 2024-04-22T23:23:47.623 INFO:tasks.workunit.client.1.smithi134.stdout:1/657: rmdir db/d15 39 2024-04-22T23:23:47.623 INFO:tasks.workunit.client.1.smithi134.stdout:6/589: dwrite d0/d19/f7 [0,4194304] 0 2024-04-22T23:23:47.625 INFO:tasks.workunit.client.1.smithi134.stdout:6/590: readlink d0/d19/de/d12/d2a/l3d 0 2024-04-22T23:23:47.625 INFO:tasks.workunit.client.1.smithi134.stdout:1/658: chown db/d15/fc6 3306892 1 2024-04-22T23:23:47.631 INFO:tasks.workunit.client.1.smithi134.stdout:1/659: rmdir db/db0/dc2 0 2024-04-22T23:23:47.633 INFO:tasks.workunit.client.1.smithi134.stdout:1/660: symlink db/db0/dc1/dd9/le4 0 2024-04-22T23:23:47.636 INFO:tasks.workunit.client.1.smithi134.stdout:1/661: truncate db/d15/f7c 3396487 0 2024-04-22T23:23:47.637 INFO:tasks.workunit.client.1.smithi134.stdout:1/662: fsync db/d15/d17/d20/d2a/f91 0 2024-04-22T23:23:47.640 INFO:tasks.workunit.client.1.smithi134.stdout:6/591: dread d0/f1a [0,4194304] 0 2024-04-22T23:23:47.641 INFO:tasks.workunit.client.1.smithi134.stdout:1/663: unlink db/d15/d17/d20/d2a/d2d/l62 0 2024-04-22T23:23:47.642 INFO:tasks.workunit.client.1.smithi134.stdout:6/592: symlink d0/d19/lc4 0 2024-04-22T23:23:47.648 INFO:tasks.workunit.client.1.smithi134.stdout:1/664: dwrite db/f8b [0,4194304] 0 2024-04-22T23:23:47.648 INFO:tasks.workunit.client.1.smithi134.stdout:6/593: dread d0/d19/de/d2f/f71 [0,4194304] 0 2024-04-22T23:23:47.650 INFO:tasks.workunit.client.1.smithi134.stdout:1/665: symlink db/db0/dc1/le5 0 2024-04-22T23:23:47.653 INFO:tasks.workunit.client.1.smithi134.stdout:6/594: write d0/d19/de/d12/d2a/d2d/d3a/f77 [3522111,68422] 0 2024-04-22T23:23:47.654 INFO:tasks.workunit.client.1.smithi134.stdout:6/595: chown d0/d19/de/f5f 1790781933 1 2024-04-22T23:23:47.656 INFO:tasks.workunit.client.1.smithi134.stdout:1/666: dread db/f8b [0,4194304] 0 2024-04-22T23:23:47.658 INFO:tasks.workunit.client.1.smithi134.stdout:1/667: dread - db/d16/d56/f60 zero size 2024-04-22T23:23:47.664 INFO:tasks.workunit.client.1.smithi134.stdout:6/596: dwrite d0/d19/de/d12/d29/d74/fa1 [0,4194304] 0 2024-04-22T23:23:47.669 INFO:tasks.workunit.client.1.smithi134.stdout:1/668: dwrite db/d16/d56/f63 [0,4194304] 0 2024-04-22T23:23:47.675 INFO:tasks.workunit.client.1.smithi134.stdout:6/597: dwrite d0/d19/de/f5f [0,4194304] 0 2024-04-22T23:23:47.685 INFO:tasks.workunit.client.1.smithi134.stdout:6/598: dwrite d0/d19/de/d1f/f2c [0,4194304] 0 2024-04-22T23:23:47.688 INFO:tasks.workunit.client.1.smithi134.stdout:6/599: mkdir d0/d6f/d88/d8b/dc5 0 2024-04-22T23:23:47.691 INFO:tasks.workunit.client.1.smithi134.stdout:6/600: read d0/d19/de/fb0 [772517,70122] 0 2024-04-22T23:23:47.694 INFO:tasks.workunit.client.1.smithi134.stdout:6/601: rmdir d0/d6f/d88 39 2024-04-22T23:23:47.700 INFO:tasks.workunit.client.1.smithi134.stdout:6/602: read d0/d19/de/d12/d29/d74/f89 [379894,55201] 0 2024-04-22T23:23:47.702 INFO:tasks.workunit.client.1.smithi134.stdout:6/603: write d0/d19/de/f31 [4066833,116819] 0 2024-04-22T23:23:47.709 INFO:tasks.workunit.client.1.smithi134.stdout:6/604: creat d0/d19/de/dbe/fc6 x:0 0 0 2024-04-22T23:23:47.714 INFO:tasks.workunit.client.1.smithi134.stdout:1/669: dread db/f1d [0,4194304] 0 2024-04-22T23:23:47.714 INFO:tasks.workunit.client.1.smithi134.stdout:1/670: readlink db/la3 0 2024-04-22T23:23:47.715 INFO:tasks.workunit.client.1.smithi134.stdout:6/605: rename d0/d4f/l96 to d0/d19/db/d75/d92/lc7 0 2024-04-22T23:23:47.717 INFO:tasks.workunit.client.1.smithi134.stdout:1/671: symlink db/d15/d17/d20/d2a/d2d/d2e/d42/le6 0 2024-04-22T23:23:47.719 INFO:tasks.workunit.client.1.smithi134.stdout:6/606: creat d0/d19/de/d12/d29/d74/da0/fc8 x:0 0 0 2024-04-22T23:23:47.722 INFO:tasks.workunit.client.1.smithi134.stdout:1/672: unlink db/d15/d17/d20/d2a/d2d/c7d 0 2024-04-22T23:23:47.723 INFO:tasks.workunit.client.1.smithi134.stdout:6/607: creat d0/d19/db/d35/dbc/fc9 x:0 0 0 2024-04-22T23:23:47.725 INFO:tasks.workunit.client.1.smithi134.stdout:6/608: truncate d0/f26 1386315 0 2024-04-22T23:23:47.729 INFO:tasks.workunit.client.1.smithi134.stdout:6/609: read d0/f42 [2464101,115348] 0 2024-04-22T23:23:47.731 INFO:tasks.workunit.client.1.smithi134.stdout:1/673: dwrite db/d15/d17/d20/d2a/f51 [8388608,4194304] 0 2024-04-22T23:23:47.732 INFO:tasks.workunit.client.1.smithi134.stdout:6/610: chown d0/d19/de/d12/d5e/c87 68218 1 2024-04-22T23:23:47.760 INFO:tasks.workunit.client.1.smithi134.stdout:6/611: mknod d0/d19/de/d12/d5e/cca 0 2024-04-22T23:23:47.761 INFO:tasks.workunit.client.1.smithi134.stdout:1/674: creat db/d15/d17/d20/d2a/d2d/d86/dad/fe7 x:0 0 0 2024-04-22T23:23:47.763 INFO:tasks.workunit.client.1.smithi134.stdout:6/612: rmdir d0/d6f/d88 39 2024-04-22T23:23:47.768 INFO:tasks.workunit.client.1.smithi134.stdout:1/675: creat db/d16/d87/dbd/fe8 x:0 0 0 2024-04-22T23:23:47.770 INFO:tasks.workunit.client.1.smithi134.stdout:1/676: write db/f5f [2111278,51200] 0 2024-04-22T23:23:47.770 INFO:tasks.workunit.client.1.smithi134.stdout:6/613: dwrite d0/d19/de/d12/d5e/d6d/f9f [0,4194304] 0 2024-04-22T23:23:47.777 INFO:tasks.workunit.client.1.smithi134.stdout:6/614: dread d0/d19/db/f68 [0,4194304] 0 2024-04-22T23:23:47.777 INFO:tasks.workunit.client.1.smithi134.stdout:1/677: dwrite db/f71 [0,4194304] 0 2024-04-22T23:23:47.781 INFO:tasks.workunit.client.1.smithi134.stdout:6/615: symlink d0/d19/de/d12/d2a/lcb 0 2024-04-22T23:23:47.784 INFO:tasks.workunit.client.1.smithi134.stdout:1/678: rename db/d16/caa to db/d15/d17/d20/ce9 0 2024-04-22T23:23:47.788 INFO:tasks.workunit.client.1.smithi134.stdout:6/616: link d0/d19/de/d12/d2a/f3b d0/d4f/fcc 0 2024-04-22T23:23:47.792 INFO:tasks.workunit.client.1.smithi134.stdout:1/679: getdents db/d15/d17/d20/d2a/d2d/d2e/d42 0 2024-04-22T23:23:47.793 INFO:tasks.workunit.client.1.smithi134.stdout:6/617: mkdir d0/d19/de/d12/d5e/dc1/dcd 0 2024-04-22T23:23:47.793 INFO:tasks.workunit.client.1.smithi134.stdout:1/680: dread - db/d16/d56/f60 zero size 2024-04-22T23:23:47.796 INFO:tasks.workunit.client.1.smithi134.stdout:1/681: rename db/d15/f7c to db/db0/fea 0 2024-04-22T23:23:47.799 INFO:tasks.workunit.client.1.smithi134.stdout:1/682: mkdir db/deb 0 2024-04-22T23:23:47.804 INFO:tasks.workunit.client.1.smithi134.stdout:1/683: dwrite db/d15/d17/d20/f46 [0,4194304] 0 2024-04-22T23:23:47.807 INFO:tasks.workunit.client.1.smithi134.stdout:6/618: truncate d0/d19/de/d2f/fb8 724893 0 2024-04-22T23:23:47.808 INFO:tasks.workunit.client.1.smithi134.stdout:1/684: rmdir db/db5 39 2024-04-22T23:23:47.813 INFO:tasks.workunit.client.1.smithi134.stdout:1/685: unlink db/d16/d56/l8c 0 2024-04-22T23:23:47.818 INFO:tasks.workunit.client.1.smithi134.stdout:1/686: rename db/f14 to db/d16/dbc/fec 0 2024-04-22T23:23:47.821 INFO:tasks.workunit.client.1.smithi134.stdout:1/687: mknod db/db0/ced 0 2024-04-22T23:23:47.824 INFO:tasks.workunit.client.1.smithi134.stdout:1/688: getdents db/d15/d17/d20/d2a 0 2024-04-22T23:23:47.850 INFO:tasks.workunit.client.1.smithi134.stdout:6/619: dread d0/f55 [0,4194304] 0 2024-04-22T23:23:47.855 INFO:tasks.workunit.client.1.smithi134.stdout:6/620: dread d0/d19/db/d33/f62 [0,4194304] 0 2024-04-22T23:23:47.859 INFO:tasks.workunit.client.1.smithi134.stdout:6/621: mknod d0/d6f/d88/cce 0 2024-04-22T23:23:47.872 INFO:tasks.workunit.client.1.smithi134.stdout:1/689: dread db/f26 [0,4194304] 0 2024-04-22T23:23:47.872 INFO:tasks.workunit.client.1.smithi134.stdout:6/622: dwrite d0/d19/de/d12/d2a/d2d/f85 [0,4194304] 0 2024-04-22T23:23:47.872 INFO:tasks.workunit.client.1.smithi134.stdout:6/623: chown d0/f1 2284 1 2024-04-22T23:23:47.873 INFO:tasks.workunit.client.1.smithi134.stdout:6/624: chown d0/d19/fa 2321851 1 2024-04-22T23:23:47.875 INFO:tasks.workunit.client.1.smithi134.stdout:6/625: write d0/f42 [2651197,13537] 0 2024-04-22T23:23:48.112 INFO:tasks.workunit.client.1.smithi134.stdout:7/529: sync 2024-04-22T23:23:48.113 INFO:tasks.workunit.client.1.smithi134.stdout:0/471: sync 2024-04-22T23:23:48.113 INFO:tasks.workunit.client.1.smithi134.stdout:3/356: sync 2024-04-22T23:23:48.113 INFO:tasks.workunit.client.1.smithi134.stdout:4/674: sync 2024-04-22T23:23:48.114 INFO:tasks.workunit.client.1.smithi134.stdout:0/472: write d1/d5/d58/f89 [883019,10638] 0 2024-04-22T23:23:48.115 INFO:tasks.workunit.client.1.smithi134.stdout:7/530: truncate d4/d8/d2a/f4f 5117715 0 2024-04-22T23:23:48.115 INFO:tasks.workunit.client.1.smithi134.stdout:4/675: mkdir d4/d6b/dd9 0 2024-04-22T23:23:48.116 INFO:tasks.workunit.client.1.smithi134.stdout:3/357: symlink dc/dd/d28/l88 0 2024-04-22T23:23:48.117 INFO:tasks.workunit.client.1.smithi134.stdout:4/676: stat d4/d29/dcc/d64/f73 0 2024-04-22T23:23:48.117 INFO:tasks.workunit.client.1.smithi134.stdout:7/531: write d4/d4a/d85/fb2 [562748,12646] 0 2024-04-22T23:23:48.121 INFO:tasks.workunit.client.1.smithi134.stdout:3/358: dread - dc/dd/d28/d2f/d21/d42/f82 zero size 2024-04-22T23:23:48.122 INFO:tasks.workunit.client.1.smithi134.stdout:4/677: mknod d4/d7/d32/d3a/cda 0 2024-04-22T23:23:48.123 INFO:tasks.workunit.client.1.smithi134.stdout:0/473: dwrite d1/d5/f8f [0,4194304] 0 2024-04-22T23:23:48.130 INFO:tasks.workunit.client.1.smithi134.stdout:4/678: dread d4/d35/d5e/d25/f86 [0,4194304] 0 2024-04-22T23:23:48.131 INFO:tasks.workunit.client.1.smithi134.stdout:0/474: rename d1/d5/dd/d11/l5a to d1/d2/l96 0 2024-04-22T23:23:48.133 INFO:tasks.workunit.client.1.smithi134.stdout:3/359: link dc/dd/d28/l88 dc/d2a/l89 0 2024-04-22T23:23:48.136 INFO:tasks.workunit.client.1.smithi134.stdout:3/360: fsync dc/dd/d28/d2f/d36/f5d 0 2024-04-22T23:23:48.136 INFO:tasks.workunit.client.1.smithi134.stdout:4/679: dread d4/d35/d8e/d6e/fb6 [0,4194304] 0 2024-04-22T23:23:48.138 INFO:tasks.workunit.client.1.smithi134.stdout:7/532: dread d4/fa [0,4194304] 0 2024-04-22T23:23:48.144 INFO:tasks.workunit.client.1.smithi134.stdout:4/680: getdents d4/d7/d32/dc5 0 2024-04-22T23:23:48.145 INFO:tasks.workunit.client.1.smithi134.stdout:3/361: write dc/dd/d28/f3a [146637,14348] 0 2024-04-22T23:23:48.145 INFO:tasks.workunit.client.1.smithi134.stdout:7/533: dread d4/d8/d3b/f68 [0,4194304] 0 2024-04-22T23:23:48.149 INFO:tasks.workunit.client.1.smithi134.stdout:4/681: link d4/d7a/c95 d4/d29/dcc/d82/d9a/cdb 0 2024-04-22T23:23:48.149 INFO:tasks.workunit.client.1.smithi134.stdout:7/534: dread d4/d8/d41/d76/dc1/d2f/d69/dab/f80 [0,4194304] 0 2024-04-22T23:23:48.151 INFO:tasks.workunit.client.1.smithi134.stdout:3/362: link dc/f18 dc/d2a/d62/d86/f8a 0 2024-04-22T23:23:48.153 INFO:tasks.workunit.client.1.smithi134.stdout:0/475: dread d1/d5/d58/f2e [0,4194304] 0 2024-04-22T23:23:48.153 INFO:tasks.workunit.client.1.smithi134.stdout:4/682: getdents d4/d9b 0 2024-04-22T23:23:48.155 INFO:tasks.workunit.client.1.smithi134.stdout:4/683: read d4/d29/dcc/f54 [1183495,71806] 0 2024-04-22T23:23:48.155 INFO:tasks.workunit.client.1.smithi134.stdout:0/476: dread d1/d5/d58/f29 [0,4194304] 0 2024-04-22T23:23:48.156 INFO:tasks.workunit.client.1.smithi134.stdout:7/535: rmdir d4/d8/d41/d76/dc1/d2f/d70/d86/dc4 0 2024-04-22T23:23:48.156 INFO:tasks.workunit.client.1.smithi134.stdout:0/477: readlink d1/d5/dd/d20/d2c/l78 0 2024-04-22T23:23:48.159 INFO:tasks.workunit.client.1.smithi134.stdout:0/478: write d1/f44 [2309391,10860] 0 2024-04-22T23:23:48.163 INFO:tasks.workunit.client.1.smithi134.stdout:0/479: creat d1/d5/dd/d68/f97 x:0 0 0 2024-04-22T23:23:48.164 INFO:tasks.workunit.client.1.smithi134.stdout:7/536: mkdir d4/d8/d41/d76/dc1/d2f/d34/d62/d29/dc5 0 2024-04-22T23:23:48.165 INFO:tasks.workunit.client.1.smithi134.stdout:7/537: readlink d4/d8/d41/d76/dc1/l4b 0 2024-04-22T23:23:48.166 INFO:tasks.workunit.client.1.smithi134.stdout:4/684: getdents d4/d35/d8e/d5d 0 2024-04-22T23:23:48.166 INFO:tasks.workunit.client.1.smithi134.stdout:0/480: mknod d1/db/c98 0 2024-04-22T23:23:48.170 INFO:tasks.workunit.client.1.smithi134.stdout:3/363: dread dc/dd/d28/d3b/f79 [0,4194304] 0 2024-04-22T23:23:48.170 INFO:tasks.workunit.client.1.smithi134.stdout:7/538: getdents d4/d8/d41/d76/dc1/d2f/d70/d86/d97 0 2024-04-22T23:23:48.171 INFO:tasks.workunit.client.1.smithi134.stdout:0/481: creat d1/d5/dd/d11/f99 x:0 0 0 2024-04-22T23:23:48.175 INFO:tasks.workunit.client.1.smithi134.stdout:4/685: truncate d4/d29/f63 559618 0 2024-04-22T23:23:48.175 INFO:tasks.workunit.client.1.smithi134.stdout:0/482: write d1/d2/d1d/f5f [4092394,50531] 0 2024-04-22T23:23:48.178 INFO:tasks.workunit.client.1.smithi134.stdout:4/686: readlink d4/dd/l5a 0 2024-04-22T23:23:48.178 INFO:tasks.workunit.client.1.smithi134.stdout:7/539: link d4/d8/d2a/d42/f6a d4/d4a/fc6 0 2024-04-22T23:23:48.178 INFO:tasks.workunit.client.1.smithi134.stdout:0/483: readlink d1/d5/dd/d20/d2c/l78 0 2024-04-22T23:23:48.178 INFO:tasks.workunit.client.1.smithi134.stdout:3/364: dwrite dc/dd/f4f [0,4194304] 0 2024-04-22T23:23:48.179 INFO:tasks.workunit.client.1.smithi134.stdout:0/484: stat d1/d5/dd 0 2024-04-22T23:23:48.182 INFO:tasks.workunit.client.1.smithi134.stdout:0/485: creat d1/d2/d1d/f9a x:0 0 0 2024-04-22T23:23:48.183 INFO:tasks.workunit.client.1.smithi134.stdout:3/365: symlink dc/dd/d28/d2f/d36/d5f/l8b 0 2024-04-22T23:23:48.185 INFO:tasks.workunit.client.1.smithi134.stdout:3/366: dread - dc/f40 zero size 2024-04-22T23:23:48.189 INFO:tasks.workunit.client.1.smithi134.stdout:3/367: dread dc/f52 [0,4194304] 0 2024-04-22T23:23:48.189 INFO:tasks.workunit.client.1.smithi134.stdout:7/540: getdents d4/d8/d41/d76 0 2024-04-22T23:23:48.190 INFO:tasks.workunit.client.1.smithi134.stdout:4/687: write d4/d29/dcc/f54 [971265,129314] 0 2024-04-22T23:23:48.192 INFO:tasks.workunit.client.1.smithi134.stdout:4/688: read d4/d35/d5e/f1f [1346951,99149] 0 2024-04-22T23:23:48.192 INFO:tasks.workunit.client.1.smithi134.stdout:3/368: creat dc/f8c x:0 0 0 2024-04-22T23:23:48.196 INFO:tasks.workunit.client.1.smithi134.stdout:4/689: unlink d4/d7/d32/d3a/fa0 0 2024-04-22T23:23:48.197 INFO:tasks.workunit.client.1.smithi134.stdout:3/369: symlink dc/d7d/l8d 0 2024-04-22T23:23:48.197 INFO:tasks.workunit.client.1.smithi134.stdout:7/541: link d4/d8/d41/ca0 d4/d8/d2a/d42/cc7 0 2024-04-22T23:23:48.200 INFO:tasks.workunit.client.1.smithi134.stdout:3/370: mknod dc/dd/d28/d34/c8e 0 2024-04-22T23:23:48.201 INFO:tasks.workunit.client.1.smithi134.stdout:9/436: sync 2024-04-22T23:23:48.205 INFO:tasks.workunit.client.1.smithi134.stdout:9/437: dread d0/d7/f90 [0,4194304] 0 2024-04-22T23:23:48.205 INFO:tasks.workunit.client.1.smithi134.stdout:3/371: truncate dc/d2a/d30/f83 167597 0 2024-04-22T23:23:48.206 INFO:tasks.workunit.client.1.smithi134.stdout:4/690: dwrite d4/d7/d32/d3a/d93/fae [0,4194304] 0 2024-04-22T23:23:48.207 INFO:tasks.workunit.client.1.smithi134.stdout:7/542: dwrite d4/d8/fb [4194304,4194304] 0 2024-04-22T23:23:48.210 INFO:tasks.workunit.client.1.smithi134.stdout:9/438: fdatasync d0/d7/d23/d26/d51/d4c/f71 0 2024-04-22T23:23:48.210 INFO:tasks.workunit.client.1.smithi134.stdout:0/486: dwrite d1/d5/d58/d22/f24 [0,4194304] 0 2024-04-22T23:23:48.217 INFO:tasks.workunit.client.1.smithi134.stdout:3/372: write dc/dd/d28/d2f/d21/d42/f66 [81328,35778] 0 2024-04-22T23:23:48.217 INFO:tasks.workunit.client.1.smithi134.stdout:7/543: mknod d4/d8/d2a/cc8 0 2024-04-22T23:23:48.217 INFO:tasks.workunit.client.1.smithi134.stdout:9/439: unlink d0/d37/f34 0 2024-04-22T23:23:48.217 INFO:tasks.workunit.client.1.smithi134.stdout:3/373: dread - dc/dd/d28/d2f/f51 zero size 2024-04-22T23:23:48.224 INFO:tasks.workunit.client.1.smithi134.stdout:9/440: rename d0/d7/f8c to d0/d37/f95 0 2024-04-22T23:23:48.225 INFO:tasks.workunit.client.1.smithi134.stdout:3/374: mkdir dc/dd/d28/d2f/d21/d42/d77/d8f 0 2024-04-22T23:23:48.227 INFO:tasks.workunit.client.1.smithi134.stdout:3/375: truncate dc/dd/d28/d2f/f4b 425349 0 2024-04-22T23:23:48.228 INFO:tasks.workunit.client.1.smithi134.stdout:9/441: rename d0/d37/c6c to d0/d7/d23/d26/d51/d88/c96 0 2024-04-22T23:23:48.229 INFO:tasks.workunit.client.1.smithi134.stdout:9/442: write d0/d2d/f44 [2112301,122673] 0 2024-04-22T23:23:48.238 INFO:tasks.workunit.client.1.smithi134.stdout:9/443: dwrite d0/d7/d23/f45 [0,4194304] 0 2024-04-22T23:23:48.244 INFO:tasks.workunit.client.1.smithi134.stdout:4/691: dwrite d4/d13/f59 [0,4194304] 0 2024-04-22T23:23:48.244 INFO:tasks.workunit.client.1.smithi134.stdout:0/487: truncate d1/d2/f4 2692649 0 2024-04-22T23:23:48.250 INFO:tasks.workunit.client.1.smithi134.stdout:9/444: mknod d0/d7/d23/d26/d4d/d5d/d8e/c97 0 2024-04-22T23:23:48.251 INFO:tasks.workunit.client.1.smithi134.stdout:7/544: dwrite d4/d8/f21 [0,4194304] 0 2024-04-22T23:23:48.252 INFO:tasks.workunit.client.1.smithi134.stdout:9/445: chown d0/d7/d23/d1b/f64 10 1 2024-04-22T23:23:48.254 INFO:tasks.workunit.client.1.smithi134.stdout:0/488: rename d1/db/c98 to d1/d2/c9b 0 2024-04-22T23:23:48.254 INFO:tasks.workunit.client.1.smithi134.stdout:4/692: mknod d4/d7/d32/d3a/d93/db5/cdc 0 2024-04-22T23:23:48.259 INFO:tasks.workunit.client.1.smithi134.stdout:9/446: fsync d0/d7/d27/f59 0 2024-04-22T23:23:48.259 INFO:tasks.workunit.client.1.smithi134.stdout:1/690: sync 2024-04-22T23:23:48.259 INFO:tasks.workunit.client.1.smithi134.stdout:5/609: sync 2024-04-22T23:23:48.259 INFO:tasks.workunit.client.1.smithi134.stdout:2/466: sync 2024-04-22T23:23:48.259 INFO:tasks.workunit.client.1.smithi134.stdout:6/626: sync 2024-04-22T23:23:48.260 INFO:tasks.workunit.client.1.smithi134.stdout:8/503: sync 2024-04-22T23:23:48.260 INFO:tasks.workunit.client.1.smithi134.stdout:4/693: rename d4/d13/cd0 to d4/d7/d32/d3a/d93/cdd 0 2024-04-22T23:23:48.261 INFO:tasks.workunit.client.1.smithi134.stdout:9/447: readlink d0/d7/d12/d22/l84 0 2024-04-22T23:23:48.262 INFO:tasks.workunit.client.1.smithi134.stdout:5/610: fsync d9/f62 0 2024-04-22T23:23:48.263 INFO:tasks.workunit.client.1.smithi134.stdout:7/545: rename d4/d8/d41/d5b/lb5 to d4/d8/d41/d5b/lc9 0 2024-04-22T23:23:48.265 INFO:tasks.workunit.client.1.smithi134.stdout:3/376: dread dc/f3f [0,4194304] 0 2024-04-22T23:23:48.266 INFO:tasks.workunit.client.1.smithi134.stdout:2/467: write d2/d15/d24/f29 [268565,29525] 0 2024-04-22T23:23:48.267 INFO:tasks.workunit.client.1.smithi134.stdout:0/489: rename d1/d5/dd/d11/c39 to d1/d5/d58/c9c 0 2024-04-22T23:23:48.269 INFO:tasks.workunit.client.1.smithi134.stdout:7/546: chown d4/d8/d41/d76/dc1/d2f/da6 26483 1 2024-04-22T23:23:48.269 INFO:tasks.workunit.client.1.smithi134.stdout:8/504: creat d7/dc/d2c/d4f/f9f x:0 0 0 2024-04-22T23:23:48.271 INFO:tasks.workunit.client.1.smithi134.stdout:6/627: readlink d0/d19/db/d35/d8a/lc2 0 2024-04-22T23:23:48.273 INFO:tasks.workunit.client.1.smithi134.stdout:6/628: write d0/d19/f7 [411432,111276] 0 2024-04-22T23:23:48.274 INFO:tasks.workunit.client.1.smithi134.stdout:1/691: creat db/d15/d17/d20/d2a/d2d/d2e/d42/fee x:0 0 0 2024-04-22T23:23:48.274 INFO:tasks.workunit.client.1.smithi134.stdout:6/629: readlink d0/d19/lc4 0 2024-04-22T23:23:48.274 INFO:tasks.workunit.client.1.smithi134.stdout:6/630: stat d0 0 2024-04-22T23:23:48.275 INFO:tasks.workunit.client.1.smithi134.stdout:3/377: symlink dc/d2a/d30/l90 0 2024-04-22T23:23:48.275 INFO:tasks.workunit.client.1.smithi134.stdout:4/694: dwrite d4/d35/db4/fbe [0,4194304] 0 2024-04-22T23:23:48.277 INFO:tasks.workunit.client.1.smithi134.stdout:1/692: read - db/d15/d17/d20/d2a/d2d/d2e/f6c zero size 2024-04-22T23:23:48.278 INFO:tasks.workunit.client.1.smithi134.stdout:3/378: dread dc/dd/d28/d2f/f4b [0,4194304] 0 2024-04-22T23:23:48.279 INFO:tasks.workunit.client.1.smithi134.stdout:2/468: unlink d2/f2e 0 2024-04-22T23:23:48.285 INFO:tasks.workunit.client.1.smithi134.stdout:8/505: chown d7/db/d42/d65/c64 46636748 1 2024-04-22T23:23:48.285 INFO:tasks.workunit.client.1.smithi134.stdout:5/611: mknod d9/d11/d34/d4d/dd4/dd7/cd8 0 2024-04-22T23:23:48.286 INFO:tasks.workunit.client.1.smithi134.stdout:8/506: stat d7/db/d42/d65/d46/d84/c8e 0 2024-04-22T23:23:48.287 INFO:tasks.workunit.client.1.smithi134.stdout:6/631: rmdir d0/d19/de/d12/d29/d74 39 2024-04-22T23:23:48.289 INFO:tasks.workunit.client.1.smithi134.stdout:1/693: stat db/d15/d17/f5a 0 2024-04-22T23:23:48.289 INFO:tasks.workunit.client.1.smithi134.stdout:8/507: fsync d7/dc/d2c/d4f/f9f 0 2024-04-22T23:23:48.291 INFO:tasks.workunit.client.1.smithi134.stdout:3/379: fsync dc/dd/d28/d2f/d54/f5e 0 2024-04-22T23:23:48.292 INFO:tasks.workunit.client.1.smithi134.stdout:3/380: fdatasync dc/dd/d28/d2f/d21/f16 0 2024-04-22T23:23:48.293 INFO:tasks.workunit.client.1.smithi134.stdout:0/490: mknod d1/d5/d58/d22/c9d 0 2024-04-22T23:23:48.294 INFO:tasks.workunit.client.1.smithi134.stdout:3/381: fsync dc/d2a/d30/f60 0 2024-04-22T23:23:48.297 INFO:tasks.workunit.client.1.smithi134.stdout:7/547: write d4/fa [1808646,13909] 0 2024-04-22T23:23:48.299 INFO:tasks.workunit.client.1.smithi134.stdout:5/612: rename d9/dd/d17/d2a/lca to d9/d11/d73/d95/dae/ld9 0 2024-04-22T23:23:48.299 INFO:tasks.workunit.client.1.smithi134.stdout:7/548: chown d4/d8/d41/d76/dc1/fb8 136439 1 2024-04-22T23:23:48.299 INFO:tasks.workunit.client.1.smithi134.stdout:5/613: fdatasync d9/dbb/fc8 0 2024-04-22T23:23:48.303 INFO:tasks.workunit.client.1.smithi134.stdout:1/694: chown db/d15/d17/cd8 190836188 1 2024-04-22T23:23:48.305 INFO:tasks.workunit.client.1.smithi134.stdout:5/614: write d9/d11/f5b [98631,10971] 0 2024-04-22T23:23:48.305 INFO:tasks.workunit.client.1.smithi134.stdout:4/695: link d4/d29/dcc/d64/f69 d4/d29/dcc/d82/fde 0 2024-04-22T23:23:48.305 INFO:tasks.workunit.client.1.smithi134.stdout:6/632: dread d0/d6f/fa2 [0,4194304] 0 2024-04-22T23:23:48.309 INFO:tasks.workunit.client.1.smithi134.stdout:2/469: mkdir d2/d15/d24/d3b/d8a/d99 0 2024-04-22T23:23:48.311 INFO:tasks.workunit.client.1.smithi134.stdout:7/549: dwrite d4/d4a/d85/fb2 [0,4194304] 0 2024-04-22T23:23:48.311 INFO:tasks.workunit.client.1.smithi134.stdout:4/696: write d4/d29/dcc/fa6 [1399128,8845] 0 2024-04-22T23:23:48.313 INFO:tasks.workunit.client.1.smithi134.stdout:0/491: symlink d1/d5/l9e 0 2024-04-22T23:23:48.316 INFO:tasks.workunit.client.1.smithi134.stdout:3/382: truncate dc/f40 844145 0 2024-04-22T23:23:48.322 INFO:tasks.workunit.client.1.smithi134.stdout:1/695: dwrite db/d15/f29 [0,4194304] 0 2024-04-22T23:23:48.325 INFO:tasks.workunit.client.1.smithi134.stdout:5/615: symlink d9/d11/d34/d4d/d92/lda 0 2024-04-22T23:23:48.325 INFO:tasks.workunit.client.1.smithi134.stdout:2/470: fdatasync d2/dd/d77/f7e 0 2024-04-22T23:23:48.327 INFO:tasks.workunit.client.1.smithi134.stdout:4/697: chown d4/d35/d53/l9f 22299920 1 2024-04-22T23:23:48.329 INFO:tasks.workunit.client.1.smithi134.stdout:6/633: dread d0/d19/de/d12/d5e/d6d/d7a/d99/fb1 [0,4194304] 0 2024-04-22T23:23:48.329 INFO:tasks.workunit.client.1.smithi134.stdout:7/550: dwrite d4/d8/d41/d76/dc1/d2f/d69/d8a/f9d [0,4194304] 0 2024-04-22T23:23:48.331 INFO:tasks.workunit.client.1.smithi134.stdout:3/383: unlink dc/dd/l43 0 2024-04-22T23:23:48.333 INFO:tasks.workunit.client.1.smithi134.stdout:0/492: rename d1/d79/d88 to d1/db/d9f 0 2024-04-22T23:23:48.338 INFO:tasks.workunit.client.1.smithi134.stdout:1/696: creat db/d16/d87/fef x:0 0 0 2024-04-22T23:23:48.339 INFO:tasks.workunit.client.1.smithi134.stdout:5/616: creat d9/da2/fdb x:0 0 0 2024-04-22T23:23:48.342 INFO:tasks.workunit.client.1.smithi134.stdout:4/698: dread d4/d35/d5e/d25/f98 [0,4194304] 0 2024-04-22T23:23:48.342 INFO:tasks.workunit.client.1.smithi134.stdout:1/697: truncate db/d16/d87/dbd/fe8 897588 0 2024-04-22T23:23:48.343 INFO:tasks.workunit.client.1.smithi134.stdout:3/384: write dc/f45 [4902726,103382] 0 2024-04-22T23:23:48.344 INFO:tasks.workunit.client.1.smithi134.stdout:4/699: fdatasync d4/fa5 0 2024-04-22T23:23:48.344 INFO:tasks.workunit.client.1.smithi134.stdout:1/698: fsync db/d15/fc6 0 2024-04-22T23:23:48.346 INFO:tasks.workunit.client.1.smithi134.stdout:2/471: dwrite d2/d15/d24/d1d/d3d/f58 [4194304,4194304] 0 2024-04-22T23:23:48.347 INFO:tasks.workunit.client.1.smithi134.stdout:6/634: creat d0/d19/de/d12/d29/db5/fcf x:0 0 0 2024-04-22T23:23:48.348 INFO:tasks.workunit.client.1.smithi134.stdout:5/617: creat d9/d1e/d90/fdc x:0 0 0 2024-04-22T23:23:48.349 INFO:tasks.workunit.client.1.smithi134.stdout:3/385: chown dc/dd/d28/f3a 2537505 1 2024-04-22T23:23:48.350 INFO:tasks.workunit.client.1.smithi134.stdout:5/618: write d9/d1e/f5a [245457,125426] 0 2024-04-22T23:23:48.351 INFO:tasks.workunit.client.1.smithi134.stdout:3/386: chown dc/dd/d28/f69 5578930 1 2024-04-22T23:23:48.351 INFO:tasks.workunit.client.1.smithi134.stdout:4/700: symlink d4/d29/ldf 0 2024-04-22T23:23:48.351 INFO:tasks.workunit.client.1.smithi134.stdout:2/472: write d2/d15/d24/d1d/d3d/f92 [927811,108867] 0 2024-04-22T23:23:48.358 INFO:tasks.workunit.client.1.smithi134.stdout:0/493: rename d1/d79 to d1/d5/d58/da0 0 2024-04-22T23:23:48.360 INFO:tasks.workunit.client.1.smithi134.stdout:0/494: stat d1/d5/d58/d22/l23 0 2024-04-22T23:23:48.360 INFO:tasks.workunit.client.1.smithi134.stdout:3/387: rmdir dc/d2a 39 2024-04-22T23:23:48.362 INFO:tasks.workunit.client.1.smithi134.stdout:5/619: mknod d9/dd/d88/db7/cdd 0 2024-04-22T23:23:48.363 INFO:tasks.workunit.client.1.smithi134.stdout:2/473: dwrite d2/dd/df/f85 [0,4194304] 0 2024-04-22T23:23:48.364 INFO:tasks.workunit.client.1.smithi134.stdout:5/620: read d9/d11/d34/d3b/f5f [1993644,16250] 0 2024-04-22T23:23:48.367 INFO:tasks.workunit.client.1.smithi134.stdout:2/474: write d2/d15/d24/f2a [2487806,34197] 0 2024-04-22T23:23:48.367 INFO:tasks.workunit.client.1.smithi134.stdout:7/551: write d4/d8/f22 [5173004,14349] 0 2024-04-22T23:23:48.367 INFO:tasks.workunit.client.1.smithi134.stdout:8/508: getdents d7/db/d2d/d39/d58/d7b 0 2024-04-22T23:23:48.371 INFO:tasks.workunit.client.1.smithi134.stdout:0/495: creat d1/d2/fa1 x:0 0 0 2024-04-22T23:23:48.376 INFO:tasks.workunit.client.1.smithi134.stdout:5/621: creat d9/d11/d34/d4d/fde x:0 0 0 2024-04-22T23:23:48.376 INFO:tasks.workunit.client.1.smithi134.stdout:5/622: chown d9/dd/d17/d44/f59 246174531 1 2024-04-22T23:23:48.378 INFO:tasks.workunit.client.1.smithi134.stdout:1/699: dwrite db/fa6 [0,4194304] 0 2024-04-22T23:23:48.379 INFO:tasks.workunit.client.1.smithi134.stdout:5/623: chown d9/d11/d34/d85/fc9 9 1 2024-04-22T23:23:48.381 INFO:tasks.workunit.client.1.smithi134.stdout:2/475: mkdir d2/d15/d24/d1d/d3d/d9a 0 2024-04-22T23:23:48.383 INFO:tasks.workunit.client.1.smithi134.stdout:5/624: chown d9/dd/d88/c9a 559 1 2024-04-22T23:23:48.385 INFO:tasks.workunit.client.1.smithi134.stdout:1/700: write db/d15/d17/d20/fd1 [443036,113525] 0 2024-04-22T23:23:48.386 INFO:tasks.workunit.client.1.smithi134.stdout:0/496: mkdir d1/d5/da2 0 2024-04-22T23:23:48.389 INFO:tasks.workunit.client.1.smithi134.stdout:5/625: write d9/f2b [8004369,95689] 0 2024-04-22T23:23:48.389 INFO:tasks.workunit.client.1.smithi134.stdout:6/635: getdents d0/d19/de/d12/d5e/dc1 0 2024-04-22T23:23:48.389 INFO:tasks.workunit.client.1.smithi134.stdout:0/497: fsync d1/d2/d1d/f9a 0 2024-04-22T23:23:48.389 INFO:tasks.workunit.client.1.smithi134.stdout:7/552: mknod d4/d8/d41/d76/dc1/cca 0 2024-04-22T23:23:48.390 INFO:tasks.workunit.client.1.smithi134.stdout:8/509: link f6 d7/dc/d2c/d4f/fa0 0 2024-04-22T23:23:48.394 INFO:tasks.workunit.client.1.smithi134.stdout:3/388: creat dc/dd/d28/d2f/f91 x:0 0 0 2024-04-22T23:23:48.394 INFO:tasks.workunit.client.1.smithi134.stdout:4/701: dwrite d4/d7/d32/d3a/d93/fc9 [0,4194304] 0 2024-04-22T23:23:48.395 INFO:tasks.workunit.client.1.smithi134.stdout:2/476: creat d2/d42/d93/f9b x:0 0 0 2024-04-22T23:23:48.396 INFO:tasks.workunit.client.1.smithi134.stdout:5/626: truncate d9/fe 1344657 0 2024-04-22T23:23:48.400 INFO:tasks.workunit.client.1.smithi134.stdout:3/389: chown dc/d2a 0 1 2024-04-22T23:23:48.400 INFO:tasks.workunit.client.1.smithi134.stdout:1/701: dread db/d15/d17/d20/fd1 [0,4194304] 0 2024-04-22T23:23:48.401 INFO:tasks.workunit.client.1.smithi134.stdout:3/390: read dc/f3f [502163,2472] 0 2024-04-22T23:23:48.402 INFO:tasks.workunit.client.1.smithi134.stdout:6/636: mknod d0/d19/de/d12/d2a/d2d/cd0 0 2024-04-22T23:23:48.403 INFO:tasks.workunit.client.1.smithi134.stdout:7/553: fdatasync d4/d8/d2a/f5f 0 2024-04-22T23:23:48.404 INFO:tasks.workunit.client.1.smithi134.stdout:4/702: mknod d4/d9b/ce0 0 2024-04-22T23:23:48.405 INFO:tasks.workunit.client.1.smithi134.stdout:4/703: readlink d4/l40 0 2024-04-22T23:23:48.406 INFO:tasks.workunit.client.1.smithi134.stdout:1/702: chown db/d15/d17/d20/d2a/d2d/f43 292 1 2024-04-22T23:23:48.406 INFO:tasks.workunit.client.1.smithi134.stdout:3/391: symlink dc/d2a/d30/l92 0 2024-04-22T23:23:48.408 INFO:tasks.workunit.client.1.smithi134.stdout:6/637: mkdir d0/d19/de/dbe/dd1 0 2024-04-22T23:23:48.409 INFO:tasks.workunit.client.1.smithi134.stdout:5/627: creat d9/d1e/fdf x:0 0 0 2024-04-22T23:23:48.410 INFO:tasks.workunit.client.1.smithi134.stdout:2/477: dwrite d2/d15/d24/f27 [4194304,4194304] 0 2024-04-22T23:23:48.411 INFO:tasks.workunit.client.1.smithi134.stdout:4/704: write d4/f21 [3364463,39944] 0 2024-04-22T23:23:48.416 INFO:tasks.workunit.client.1.smithi134.stdout:1/703: creat db/db0/dc1/dd9/ff0 x:0 0 0 2024-04-22T23:23:48.421 INFO:tasks.workunit.client.1.smithi134.stdout:7/554: dwrite d4/d8/fb [4194304,4194304] 0 2024-04-22T23:23:48.427 INFO:tasks.workunit.client.1.smithi134.stdout:3/392: rename dc/dd/d28/d34/f50 to dc/dd/d28/d2f/d36/d6d/d80/f93 0 2024-04-22T23:23:48.427 INFO:tasks.workunit.client.1.smithi134.stdout:4/705: dread d4/dd/f41 [0,4194304] 0 2024-04-22T23:23:48.427 INFO:tasks.workunit.client.1.smithi134.stdout:7/555: dread d4/d4a/d85/fb2 [0,4194304] 0 2024-04-22T23:23:48.430 INFO:tasks.workunit.client.1.smithi134.stdout:7/556: write d4/d8/f22 [2933460,128428] 0 2024-04-22T23:23:48.431 INFO:tasks.workunit.client.1.smithi134.stdout:6/638: mknod d0/d19/de/d1f/dae/cd2 0 2024-04-22T23:23:48.431 INFO:tasks.workunit.client.1.smithi134.stdout:8/510: truncate d7/db/d2d/d39/d58/d7b/f92 3570758 0 2024-04-22T23:23:48.433 INFO:tasks.workunit.client.1.smithi134.stdout:6/639: chown d0/d19/de/d12/d29/f8d 65470436 1 2024-04-22T23:23:48.434 INFO:tasks.workunit.client.1.smithi134.stdout:8/511: truncate d7/db/d42/f76 1530722 0 2024-04-22T23:23:48.435 INFO:tasks.workunit.client.1.smithi134.stdout:7/557: write d4/d8/d41/d76/dc1/d2f/d69/d8a/f99 [788011,100658] 0 2024-04-22T23:23:48.440 INFO:tasks.workunit.client.1.smithi134.stdout:4/706: rename d4/d35/d5e/d25/c71 to d4/d9b/ce1 0 2024-04-22T23:23:48.441 INFO:tasks.workunit.client.1.smithi134.stdout:1/704: creat db/d15/d17/d20/d2a/d2d/ff1 x:0 0 0 2024-04-22T23:23:48.444 INFO:tasks.workunit.client.1.smithi134.stdout:2/478: getdents d2/d15/d24 0 2024-04-22T23:23:48.446 INFO:tasks.workunit.client.1.smithi134.stdout:6/640: mkdir d0/d19/de/d12/d5e/d6d/dd3 0 2024-04-22T23:23:48.449 INFO:tasks.workunit.client.1.smithi134.stdout:4/707: write d4/dd/f31 [416471,107174] 0 2024-04-22T23:23:48.450 INFO:tasks.workunit.client.1.smithi134.stdout:8/512: creat d7/db/d42/d65/d46/d9a/fa1 x:0 0 0 2024-04-22T23:23:48.453 INFO:tasks.workunit.client.1.smithi134.stdout:1/705: mknod db/d15/d17/dc9/cf2 0 2024-04-22T23:23:48.454 INFO:tasks.workunit.client.1.smithi134.stdout:2/479: dwrite d2/dd/f47 [4194304,4194304] 0 2024-04-22T23:23:48.455 INFO:tasks.workunit.client.1.smithi134.stdout:5/628: dread d9/dd/f4c [0,4194304] 0 2024-04-22T23:23:48.455 INFO:tasks.workunit.client.1.smithi134.stdout:2/480: dread - d2/f87 zero size 2024-04-22T23:23:48.464 INFO:tasks.workunit.client.1.smithi134.stdout:4/708: write d4/d29/dcc/d82/fb2 [536301,106121] 0 2024-04-22T23:23:48.464 INFO:tasks.workunit.client.1.smithi134.stdout:3/393: dwrite dc/dd/d28/d2f/d54/f5e [0,4194304] 0 2024-04-22T23:23:48.466 INFO:tasks.workunit.client.1.smithi134.stdout:8/513: symlink d7/db/d2d/d39/d58/d7b/d88/la2 0 2024-04-22T23:23:48.467 INFO:tasks.workunit.client.1.smithi134.stdout:5/629: mkdir d9/d11/d73/d95/dae/de0 0 2024-04-22T23:23:48.471 INFO:tasks.workunit.client.1.smithi134.stdout:6/641: symlink d0/d19/db/d35/d7f/ld4 0 2024-04-22T23:23:48.471 INFO:tasks.workunit.client.1.smithi134.stdout:8/514: write d7/db/d2d/d39/d58/d8a/f94 [191541,34434] 0 2024-04-22T23:23:48.473 INFO:tasks.workunit.client.1.smithi134.stdout:7/558: truncate d4/f31 1209962 0 2024-04-22T23:23:48.474 INFO:tasks.workunit.client.1.smithi134.stdout:1/706: creat db/db5/ff3 x:0 0 0 2024-04-22T23:23:48.476 INFO:tasks.workunit.client.1.smithi134.stdout:6/642: dread d0/d6f/f9e [0,4194304] 0 2024-04-22T23:23:48.477 INFO:tasks.workunit.client.1.smithi134.stdout:4/709: chown d4/dd/cc 670722312 1 2024-04-22T23:23:48.477 INFO:tasks.workunit.client.1.smithi134.stdout:5/630: mkdir d9/d11/de1 0 2024-04-22T23:23:48.478 INFO:tasks.workunit.client.1.smithi134.stdout:5/631: read - d9/d1e/d90/fc4 zero size 2024-04-22T23:23:48.480 INFO:tasks.workunit.client.1.smithi134.stdout:3/394: truncate dc/f40 1288326 0 2024-04-22T23:23:48.481 INFO:tasks.workunit.client.1.smithi134.stdout:1/707: write db/d16/d56/fac [5168025,78400] 0 2024-04-22T23:23:48.481 INFO:tasks.workunit.client.1.smithi134.stdout:4/710: write d4/d29/dcc/f92 [2531672,101534] 0 2024-04-22T23:23:48.483 INFO:tasks.workunit.client.1.smithi134.stdout:7/559: dread - d4/d8/d41/d76/dc1/d2f/d70/f74 zero size 2024-04-22T23:23:48.486 INFO:tasks.workunit.client.1.smithi134.stdout:2/481: dread d2/d15/d24/d1d/f44 [0,4194304] 0 2024-04-22T23:23:48.486 INFO:tasks.workunit.client.1.smithi134.stdout:2/482: chown d2/f98 44019 1 2024-04-22T23:23:48.486 INFO:tasks.workunit.client.1.smithi134.stdout:5/632: rmdir d9/d11/d34 39 2024-04-22T23:23:48.494 INFO:tasks.workunit.client.1.smithi134.stdout:5/633: dwrite d9/f2b [8388608,4194304] 0 2024-04-22T23:23:48.502 INFO:tasks.workunit.client.1.smithi134.stdout:3/395: rename dc/dd/d28/l88 to dc/dd/d28/d2f/d61/l94 0 2024-04-22T23:23:48.504 INFO:tasks.workunit.client.1.smithi134.stdout:2/483: mkdir d2/dd/df/d9c 0 2024-04-22T23:23:48.505 INFO:tasks.workunit.client.1.smithi134.stdout:3/396: fdatasync dc/f45 0 2024-04-22T23:23:48.507 INFO:tasks.workunit.client.1.smithi134.stdout:1/708: dwrite db/d16/d56/fa1 [0,4194304] 0 2024-04-22T23:23:48.507 INFO:tasks.workunit.client.1.smithi134.stdout:3/397: symlink dc/d2a/l95 2 2024-04-22T23:23:48.511 INFO:tasks.workunit.client.1.smithi134.stdout:5/634: dwrite d9/d11/f1a [0,4194304] 0 2024-04-22T23:23:48.512 INFO:tasks.workunit.client.1.smithi134.stdout:6/643: dwrite d0/d19/f9 [4194304,4194304] 0 2024-04-22T23:23:48.512 INFO:tasks.workunit.client.1.smithi134.stdout:8/515: getdents d7 0 2024-04-22T23:23:48.513 INFO:tasks.workunit.client.1.smithi134.stdout:7/560: link d4/d8/d3b/d83/f88 d4/d8/d41/d76/dc1/d2f/d69/dab/d95/fcb 0 2024-04-22T23:23:48.514 INFO:tasks.workunit.client.1.smithi134.stdout:3/398: dread - dc/dd/d28/d2f/f51 zero size 2024-04-22T23:23:48.518 INFO:tasks.workunit.client.1.smithi134.stdout:4/711: getdents d4/d35/d5e 0 2024-04-22T23:23:48.522 INFO:tasks.workunit.client.1.smithi134.stdout:4/712: fdatasync d4/d35/db4/fbe 0 2024-04-22T23:23:48.523 INFO:tasks.workunit.client.1.smithi134.stdout:5/635: truncate d9/d11/d34/d4d/d92/fab 78680 0 2024-04-22T23:23:48.523 INFO:tasks.workunit.client.1.smithi134.stdout:2/484: symlink d2/d42/l9d 0 2024-04-22T23:23:48.524 INFO:tasks.workunit.client.1.smithi134.stdout:3/399: symlink dc/dd/d28/d3b/l96 0 2024-04-22T23:23:48.526 INFO:tasks.workunit.client.1.smithi134.stdout:3/400: read - dc/dd/f7a zero size 2024-04-22T23:23:48.526 INFO:tasks.workunit.client.1.smithi134.stdout:9/448: sync 2024-04-22T23:23:48.526 INFO:tasks.workunit.client.1.smithi134.stdout:8/516: rename d7/db/d2d/d39/d58/d7b to d7/db/d42/d65/d46/d9a/da3 0 2024-04-22T23:23:48.528 INFO:tasks.workunit.client.1.smithi134.stdout:6/644: dread d0/d19/de/d12/f7e [0,4194304] 0 2024-04-22T23:23:48.533 INFO:tasks.workunit.client.1.smithi134.stdout:6/645: write d0/d19/db/d33/f95 [152783,124721] 0 2024-04-22T23:23:48.534 INFO:tasks.workunit.client.1.smithi134.stdout:1/709: mkdir db/d16/dbc/de1/df4 0 2024-04-22T23:23:48.535 INFO:tasks.workunit.client.1.smithi134.stdout:2/485: dread d2/d15/d24/f69 [0,4194304] 0 2024-04-22T23:23:48.537 INFO:tasks.workunit.client.1.smithi134.stdout:2/486: readlink d2/dd/d4a/l66 0 2024-04-22T23:23:48.538 INFO:tasks.workunit.client.1.smithi134.stdout:7/561: mknod d4/d8/d41/d76/dc1/d2f/d34/d62/ccc 0 2024-04-22T23:23:48.541 INFO:tasks.workunit.client.1.smithi134.stdout:9/449: mkdir d0/d2d/d94/d98 0 2024-04-22T23:23:48.545 INFO:tasks.workunit.client.1.smithi134.stdout:7/562: fdatasync d4/d8/d3b/f68 0 2024-04-22T23:23:48.545 INFO:tasks.workunit.client.1.smithi134.stdout:2/487: creat d2/d42/d54/f9e x:0 0 0 2024-04-22T23:23:48.552 INFO:tasks.workunit.client.1.smithi134.stdout:8/517: dwrite d7/db/d2d/f54 [0,4194304] 0 2024-04-22T23:23:48.555 INFO:tasks.workunit.client.1.smithi134.stdout:6/646: getdents d0/d19/de/d12/d2a/d2d/d3a 0 2024-04-22T23:23:48.560 INFO:tasks.workunit.client.1.smithi134.stdout:3/401: dread dc/dd/d28/d2f/d21/f16 [0,4194304] 0 2024-04-22T23:23:48.561 INFO:tasks.workunit.client.1.smithi134.stdout:9/450: dread d0/d7/d23/f45 [0,4194304] 0 2024-04-22T23:23:48.561 INFO:tasks.workunit.client.1.smithi134.stdout:5/636: truncate d9/dd/d17/d44/fd0 209617 0 2024-04-22T23:23:48.566 INFO:tasks.workunit.client.1.smithi134.stdout:2/488: dread d2/dd/df/d1c/f79 [4194304,4194304] 0 2024-04-22T23:23:48.567 INFO:tasks.workunit.client.1.smithi134.stdout:6/647: readlink d0/d19/de/d2f/l93 0 2024-04-22T23:23:48.569 INFO:tasks.workunit.client.1.smithi134.stdout:4/713: dread d4/dd/f6d [0,4194304] 0 2024-04-22T23:23:48.571 INFO:tasks.workunit.client.1.smithi134.stdout:1/710: dread db/f27 [8388608,4194304] 0 2024-04-22T23:23:48.571 INFO:tasks.workunit.client.1.smithi134.stdout:8/518: symlink d7/db/d42/d65/la4 0 2024-04-22T23:23:48.572 INFO:tasks.workunit.client.1.smithi134.stdout:3/402: symlink dc/dd/d28/d3b/l97 0 2024-04-22T23:23:48.575 INFO:tasks.workunit.client.1.smithi134.stdout:7/563: rmdir d4/d8/d41/d76/dc1/d2f/d69/dab/d95/dc2 0 2024-04-22T23:23:48.577 INFO:tasks.workunit.client.1.smithi134.stdout:0/498: sync 2024-04-22T23:23:48.577 INFO:tasks.workunit.client.1.smithi134.stdout:3/403: read dc/dd/d28/f3a [2401323,20814] 0 2024-04-22T23:23:48.578 INFO:tasks.workunit.client.1.smithi134.stdout:5/637: symlink d9/d11/d34/d3b/dc0/le2 0 2024-04-22T23:23:48.580 INFO:tasks.workunit.client.1.smithi134.stdout:1/711: fdatasync db/db5/fce 0 2024-04-22T23:23:48.582 INFO:tasks.workunit.client.1.smithi134.stdout:2/489: dwrite d2/d15/d24/f35 [0,4194304] 0 2024-04-22T23:23:48.582 INFO:tasks.workunit.client.1.smithi134.stdout:4/714: mknod d4/d35/ce2 0 2024-04-22T23:23:48.584 INFO:tasks.workunit.client.1.smithi134.stdout:2/490: stat d2/d15/d24/f29 0 2024-04-22T23:23:48.588 INFO:tasks.workunit.client.1.smithi134.stdout:2/491: dread - d2/d42/d93/f95 zero size 2024-04-22T23:23:48.589 INFO:tasks.workunit.client.1.smithi134.stdout:7/564: mkdir d4/dad/dcd 0 2024-04-22T23:23:48.591 INFO:tasks.workunit.client.1.smithi134.stdout:8/519: dwrite d7/db/d2d/d39/d58/f97 [0,4194304] 0 2024-04-22T23:23:48.593 INFO:tasks.workunit.client.1.smithi134.stdout:0/499: dread d1/d5/f8b [0,4194304] 0 2024-04-22T23:23:48.595 INFO:tasks.workunit.client.1.smithi134.stdout:6/648: truncate d0/d19/de/d12/f97 1112845 0 2024-04-22T23:23:48.599 INFO:tasks.workunit.client.1.smithi134.stdout:3/404: rename dc/d2a/d30/l90 to dc/d7d/l98 0 2024-04-22T23:23:48.601 INFO:tasks.workunit.client.1.smithi134.stdout:2/492: symlink d2/d15/d24/d3b/d8a/l9f 0 2024-04-22T23:23:48.601 INFO:tasks.workunit.client.1.smithi134.stdout:8/520: mkdir d7/dc/d19/da5 0 2024-04-22T23:23:48.601 INFO:tasks.workunit.client.1.smithi134.stdout:9/451: link d0/d7/d12/l7a d0/l99 0 2024-04-22T23:23:48.603 INFO:tasks.workunit.client.1.smithi134.stdout:6/649: mkdir d0/d19/de/d1f/dd5 0 2024-04-22T23:23:48.603 INFO:tasks.workunit.client.1.smithi134.stdout:0/500: dread d1/d5/dd/f2a [0,4194304] 0 2024-04-22T23:23:48.605 INFO:tasks.workunit.client.1.smithi134.stdout:0/501: readlink d1/d5/l8a 0 2024-04-22T23:23:48.609 INFO:tasks.workunit.client.1.smithi134.stdout:1/712: rmdir db/deb 0 2024-04-22T23:23:48.609 INFO:tasks.workunit.client.1.smithi134.stdout:0/502: dread - d1/d5/dd/d11/f99 zero size 2024-04-22T23:23:48.611 INFO:tasks.workunit.client.1.smithi134.stdout:4/715: rename d4/d29/dcc/l36 to d4/d29/le3 0 2024-04-22T23:23:48.613 INFO:tasks.workunit.client.1.smithi134.stdout:3/405: creat dc/dd/d28/d34/d6b/f99 x:0 0 0 2024-04-22T23:23:48.615 INFO:tasks.workunit.client.1.smithi134.stdout:7/565: mknod d4/d8/d41/d76/dc1/d2f/d69/d8a/da8/cce 0 2024-04-22T23:23:48.615 INFO:tasks.workunit.client.1.smithi134.stdout:9/452: dread - d0/d7/f41 zero size 2024-04-22T23:23:48.617 INFO:tasks.workunit.client.1.smithi134.stdout:2/493: dwrite d2/d15/d24/f69 [0,4194304] 0 2024-04-22T23:23:48.619 INFO:tasks.workunit.client.1.smithi134.stdout:5/638: dread d9/f62 [0,4194304] 0 2024-04-22T23:23:48.621 INFO:tasks.workunit.client.1.smithi134.stdout:7/566: readlink d4/d8/d41/d76/dc1/d2f/l66 0 2024-04-22T23:23:48.624 INFO:tasks.workunit.client.1.smithi134.stdout:1/713: unlink db/d15/d17/f9d 0 2024-04-22T23:23:48.626 INFO:tasks.workunit.client.1.smithi134.stdout:8/521: dwrite d7/db/d2d/d39/d58/f96 [0,4194304] 0 2024-04-22T23:23:48.627 INFO:tasks.workunit.client.1.smithi134.stdout:7/567: dread - d4/d4a/fae zero size 2024-04-22T23:23:48.628 INFO:tasks.workunit.client.1.smithi134.stdout:4/716: creat d4/d7/d32/d3a/d93/db5/fe4 x:0 0 0 2024-04-22T23:23:48.640 INFO:tasks.workunit.client.1.smithi134.stdout:6/650: dread d0/d19/de/d12/d2a/d2d/f49 [0,4194304] 0 2024-04-22T23:23:48.641 INFO:tasks.workunit.client.1.smithi134.stdout:7/568: truncate d4/f13 3307030 0 2024-04-22T23:23:48.644 INFO:tasks.workunit.client.1.smithi134.stdout:5/639: dwrite d9/da2/faa [0,4194304] 0 2024-04-22T23:23:48.644 INFO:tasks.workunit.client.1.smithi134.stdout:2/494: creat d2/dd/df/fa0 x:0 0 0 2024-04-22T23:23:48.644 INFO:tasks.workunit.client.1.smithi134.stdout:4/717: creat d4/d7/d32/d3a/d93/db5/fe5 x:0 0 0 2024-04-22T23:23:48.646 INFO:tasks.workunit.client.1.smithi134.stdout:0/503: dwrite d1/d5/dd/d11/f6d [0,4194304] 0 2024-04-22T23:23:48.647 INFO:tasks.workunit.client.1.smithi134.stdout:8/522: mkdir d7/db/d42/d65/d46/d9a/da3/d88/da6 0 2024-04-22T23:23:48.651 INFO:tasks.workunit.client.1.smithi134.stdout:4/718: dread - d4/d13/fca zero size 2024-04-22T23:23:48.651 INFO:tasks.workunit.client.1.smithi134.stdout:5/640: chown d9/d11/d73/d95/d20/d60 32 1 2024-04-22T23:23:48.654 INFO:tasks.workunit.client.1.smithi134.stdout:7/569: mkdir d4/d8/d41/d76/dc1/d2f/da6/dcf 0 2024-04-22T23:23:48.661 INFO:tasks.workunit.client.1.smithi134.stdout:4/719: mknod d4/d7/d32/d84/ce6 0 2024-04-22T23:23:48.663 INFO:tasks.workunit.client.1.smithi134.stdout:5/641: symlink d9/d11/de1/le3 0 2024-04-22T23:23:48.666 INFO:tasks.workunit.client.1.smithi134.stdout:6/651: dwrite d0/d19/db/d35/f44 [0,4194304] 0 2024-04-22T23:23:48.669 INFO:tasks.workunit.client.1.smithi134.stdout:4/720: chown d4/d35/d5e/d25/f98 1 1 2024-04-22T23:23:48.674 INFO:tasks.workunit.client.1.smithi134.stdout:0/504: dwrite d1/d5/dd/f35 [0,4194304] 0 2024-04-22T23:23:48.674 INFO:tasks.workunit.client.1.smithi134.stdout:1/714: dwrite db/d15/d17/d20/d2a/d2d/d2e/d42/f9a [0,4194304] 0 2024-04-22T23:23:48.678 INFO:tasks.workunit.client.1.smithi134.stdout:3/406: dread dc/d2a/d30/f60 [0,4194304] 0 2024-04-22T23:23:48.678 INFO:tasks.workunit.client.1.smithi134.stdout:7/570: link l2 d4/dad/ld0 0 2024-04-22T23:23:48.678 INFO:tasks.workunit.client.1.smithi134.stdout:2/495: dwrite d2/dd/df/d1c/f79 [4194304,4194304] 0 2024-04-22T23:23:48.681 INFO:tasks.workunit.client.1.smithi134.stdout:8/523: rename d7/db/d42/d65/f4c to d7/db/d2d/fa7 0 2024-04-22T23:23:48.681 INFO:tasks.workunit.client.1.smithi134.stdout:6/652: mkdir d0/d19/de/d12/d5e/dd6 0 2024-04-22T23:23:48.682 INFO:tasks.workunit.client.1.smithi134.stdout:4/721: truncate d4/d7/d32/d84/fcb 59196 0 2024-04-22T23:23:48.685 INFO:tasks.workunit.client.1.smithi134.stdout:0/505: mknod d1/db/ca3 0 2024-04-22T23:23:48.686 INFO:tasks.workunit.client.1.smithi134.stdout:7/571: creat d4/d8/d3b/fd1 x:0 0 0 2024-04-22T23:23:48.689 INFO:tasks.workunit.client.1.smithi134.stdout:0/506: readlink d1/d5/d58/l61 0 2024-04-22T23:23:48.696 INFO:tasks.workunit.client.1.smithi134.stdout:4/722: write d4/d7/d32/d3a/d93/fc9 [3011643,60398] 0 2024-04-22T23:23:48.697 INFO:tasks.workunit.client.1.smithi134.stdout:8/524: getdents d7/db/d42 0 2024-04-22T23:23:48.700 INFO:tasks.workunit.client.1.smithi134.stdout:5/642: dwrite d9/d11/f1b [4194304,4194304] 0 2024-04-22T23:23:48.702 INFO:tasks.workunit.client.1.smithi134.stdout:1/715: dwrite db/d16/d87/fef [0,4194304] 0 2024-04-22T23:23:48.704 INFO:tasks.workunit.client.1.smithi134.stdout:2/496: dread d2/dd/df/f6f [0,4194304] 0 2024-04-22T23:23:48.707 INFO:tasks.workunit.client.1.smithi134.stdout:8/525: write d7/db/d2d/d39/f72 [759578,70864] 0 2024-04-22T23:23:48.714 INFO:tasks.workunit.client.1.smithi134.stdout:5/643: chown d9/dd/d17/d44/fd0 841040 1 2024-04-22T23:23:48.714 INFO:tasks.workunit.client.1.smithi134.stdout:1/716: rename db/d15/d17/f84 to db/d16/dbc/de1/ff5 0 2024-04-22T23:23:48.720 INFO:tasks.workunit.client.1.smithi134.stdout:5/644: read - d9/d11/d34/d85/fc9 zero size 2024-04-22T23:23:48.720 INFO:tasks.workunit.client.1.smithi134.stdout:2/497: dwrite d2/dd/df/d1c/f21 [8388608,4194304] 0 2024-04-22T23:23:48.721 INFO:tasks.workunit.client.1.smithi134.stdout:4/723: dwrite d4/d7/d32/d3a/d93/fc9 [0,4194304] 0 2024-04-22T23:23:48.722 INFO:tasks.workunit.client.1.smithi134.stdout:1/717: write db/f27 [11687749,129114] 0 2024-04-22T23:23:48.723 INFO:tasks.workunit.client.1.smithi134.stdout:8/526: creat d7/dc/fa8 x:0 0 0 2024-04-22T23:23:48.725 INFO:tasks.workunit.client.1.smithi134.stdout:4/724: read d4/d35/d5e/f60 [1264621,47494] 0 2024-04-22T23:23:48.735 INFO:tasks.workunit.client.1.smithi134.stdout:7/572: truncate d4/d8/f22 3621772 0 2024-04-22T23:23:48.738 INFO:tasks.workunit.client.1.smithi134.stdout:5/645: dwrite d9/d11/d34/d85/fa5 [0,4194304] 0 2024-04-22T23:23:48.738 INFO:tasks.workunit.client.1.smithi134.stdout:4/725: unlink d4/d7/d32/l52 0 2024-04-22T23:23:48.740 INFO:tasks.workunit.client.1.smithi134.stdout:5/646: fsync d9/dbb/fc8 0 2024-04-22T23:23:48.741 INFO:tasks.workunit.client.1.smithi134.stdout:6/653: dwrite d0/d6f/f84 [0,4194304] 0 2024-04-22T23:23:48.743 INFO:tasks.workunit.client.1.smithi134.stdout:8/527: fdatasync d7/dc/d19/f21 0 2024-04-22T23:23:48.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:48 smithi134 ceph-mon[34609]: pgmap v279: 65 pgs: 65 active+clean; 2.8 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail; 87 MiB/s rd, 267 MiB/s wr, 547 op/s 2024-04-22T23:23:48.748 INFO:tasks.workunit.client.1.smithi134.stdout:4/726: mkdir d4/d29/dcc/d82/de7 0 2024-04-22T23:23:48.750 INFO:tasks.workunit.client.1.smithi134.stdout:3/407: dread f9 [4194304,4194304] 0 2024-04-22T23:23:48.752 INFO:tasks.workunit.client.1.smithi134.stdout:1/718: dwrite db/f28 [4194304,4194304] 0 2024-04-22T23:23:48.752 INFO:tasks.workunit.client.1.smithi134.stdout:6/654: creat d0/d19/de/d12/d5e/d6d/d7a/db2/fd7 x:0 0 0 2024-04-22T23:23:48.754 INFO:tasks.workunit.client.1.smithi134.stdout:7/573: mkdir d4/d8/d41/d76/dc1/d2f/d34/d62/d29/dc5/dd2 0 2024-04-22T23:23:48.755 INFO:tasks.workunit.client.1.smithi134.stdout:5/647: dread d9/d11/d34/d85/fa5 [0,4194304] 0 2024-04-22T23:23:48.757 INFO:tasks.workunit.client.1.smithi134.stdout:4/727: truncate d4/d7/d32/d3a/d93/db5/fe5 596836 0 2024-04-22T23:23:48.758 INFO:tasks.workunit.client.1.smithi134.stdout:8/528: mknod d7/dc/ca9 0 2024-04-22T23:23:48.760 INFO:tasks.workunit.client.1.smithi134.stdout:7/574: readlink d4/d8/d41/d76/dc1/l39 0 2024-04-22T23:23:48.765 INFO:tasks.workunit.client.1.smithi134.stdout:8/529: write d7/dc/d2c/d4f/d6c/d6e/f6f [3974880,92010] 0 2024-04-22T23:23:48.766 INFO:tasks.workunit.client.1.smithi134.stdout:3/408: creat dc/dd/d28/d34/d6b/f9a x:0 0 0 2024-04-22T23:23:48.767 INFO:tasks.workunit.client.1.smithi134.stdout:5/648: dwrite d9/dd/da6/fc6 [0,4194304] 0 2024-04-22T23:23:48.770 INFO:tasks.workunit.client.1.smithi134.stdout:8/530: write d7/db/f3b [1454746,118767] 0 2024-04-22T23:23:48.771 INFO:tasks.workunit.client.1.smithi134.stdout:1/719: chown db/d15/l1f 38998 1 2024-04-22T23:23:48.771 INFO:tasks.workunit.client.1.smithi134.stdout:4/728: mknod d4/d7a/dcf/ce8 0 2024-04-22T23:23:48.771 INFO:tasks.workunit.client.1.smithi134.stdout:3/409: chown dc/dd/d28/d2f/d21/d42/d77 60 1 2024-04-22T23:23:48.775 INFO:tasks.workunit.client.1.smithi134.stdout:1/720: fdatasync db/d15/d17/d20/fd1 0 2024-04-22T23:23:48.779 INFO:tasks.workunit.client.1.smithi134.stdout:4/729: unlink d4/d35/d5e/d25/f87 0 2024-04-22T23:23:48.780 INFO:tasks.workunit.client.1.smithi134.stdout:1/721: write db/d15/d17/d20/d2a/fc3 [1717590,16897] 0 2024-04-22T23:23:48.780 INFO:tasks.workunit.client.1.smithi134.stdout:4/730: stat d4/d29/c58 0 2024-04-22T23:23:48.783 INFO:tasks.workunit.client.1.smithi134.stdout:3/410: dwrite dc/dd/d28/d2f/f73 [0,4194304] 0 2024-04-22T23:23:48.783 INFO:tasks.workunit.client.1.smithi134.stdout:7/575: link d4/d8/d41/d76/dc1/l4b d4/d8/d41/d76/dc1/ld3 0 2024-04-22T23:23:48.784 INFO:tasks.workunit.client.1.smithi134.stdout:4/731: dread - d4/d7/d32/d3a/dbc/fbd zero size 2024-04-22T23:23:48.787 INFO:tasks.workunit.client.1.smithi134.stdout:7/576: truncate d4/d4a/f92 504579 0 2024-04-22T23:23:48.787 INFO:tasks.workunit.client.1.smithi134.stdout:4/732: fsync d4/d7/d32/d84/fcb 0 2024-04-22T23:23:48.789 INFO:tasks.workunit.client.1.smithi134.stdout:7/577: chown d4/d8/d41/d76/dc1/d2f/d69/dab/d95 2501 1 2024-04-22T23:23:48.790 INFO:tasks.workunit.client.1.smithi134.stdout:7/578: read - d4/d8/d41/d76/dc1/d2f/d70/f74 zero size 2024-04-22T23:23:48.790 INFO:tasks.workunit.client.1.smithi134.stdout:7/579: stat d4/d4a/f7e 0 2024-04-22T23:23:48.797 INFO:tasks.workunit.client.1.smithi134.stdout:6/655: dwrite d0/f16 [0,4194304] 0 2024-04-22T23:23:48.799 INFO:tasks.workunit.client.1.smithi134.stdout:6/656: chown d0/d19/db/d33/d86/db6 204125 1 2024-04-22T23:23:48.799 INFO:tasks.workunit.client.1.smithi134.stdout:4/733: readlink d4/d29/le3 0 2024-04-22T23:23:48.800 INFO:tasks.workunit.client.1.smithi134.stdout:6/657: readlink d0/d19/db/la8 0 2024-04-22T23:23:48.801 INFO:tasks.workunit.client.1.smithi134.stdout:3/411: write dc/dd/d28/d2f/f4b [830204,86083] 0 2024-04-22T23:23:48.803 INFO:tasks.workunit.client.1.smithi134.stdout:1/722: write db/d15/d17/d20/d2a/d2d/d86/fa4 [503329,18544] 0 2024-04-22T23:23:48.806 INFO:tasks.workunit.client.1.smithi134.stdout:7/580: creat d4/d8/d41/d76/dc1/d2f/da6/dcf/fd4 x:0 0 0 2024-04-22T23:23:48.807 INFO:tasks.workunit.client.1.smithi134.stdout:1/723: read db/d16/fda [66335,130360] 0 2024-04-22T23:23:48.811 INFO:tasks.workunit.client.1.smithi134.stdout:7/581: rename d4/c3d to d4/d8/d41/d76/dc1/d2f/d69/dab/cd5 0 2024-04-22T23:23:48.814 INFO:tasks.workunit.client.1.smithi134.stdout:3/412: dwrite dc/f52 [0,4194304] 0 2024-04-22T23:23:48.816 INFO:tasks.workunit.client.1.smithi134.stdout:3/413: dread - dc/dd/f7a zero size 2024-04-22T23:23:48.819 INFO:tasks.workunit.client.1.smithi134.stdout:3/414: write dc/d2a/d30/f60 [3321504,70102] 0 2024-04-22T23:23:48.820 INFO:tasks.workunit.client.1.smithi134.stdout:7/582: fdatasync d4/d8/d2a/d42/f6a 0 2024-04-22T23:23:48.826 INFO:tasks.workunit.client.1.smithi134.stdout:7/583: read - d4/d8/d2a/d42/f82 zero size 2024-04-22T23:23:48.829 INFO:tasks.workunit.client.1.smithi134.stdout:7/584: fsync d4/d8/d41/d76/dc1/fa5 0 2024-04-22T23:23:48.831 INFO:tasks.workunit.client.1.smithi134.stdout:3/415: dwrite dc/dd/d28/f5c [0,4194304] 0 2024-04-22T23:23:48.834 INFO:tasks.workunit.client.1.smithi134.stdout:1/724: dwrite db/d15/d17/d20/f45 [0,4194304] 0 2024-04-22T23:23:48.836 INFO:tasks.workunit.client.1.smithi134.stdout:3/416: fsync dc/dd/d28/d2f/d54/f7b 0 2024-04-22T23:23:48.837 INFO:tasks.workunit.client.1.smithi134.stdout:4/734: dread d4/d29/dcc/d64/f73 [0,4194304] 0 2024-04-22T23:23:48.846 INFO:tasks.workunit.client.1.smithi134.stdout:4/735: dread - d4/d35/d53/fa4 zero size 2024-04-22T23:23:48.849 INFO:tasks.workunit.client.1.smithi134.stdout:7/585: dwrite d4/d8/fb [0,4194304] 0 2024-04-22T23:23:48.856 INFO:tasks.workunit.client.1.smithi134.stdout:7/586: rmdir d4/d8/d2a 39 2024-04-22T23:23:48.862 INFO:tasks.workunit.client.1.smithi134.stdout:3/417: creat dc/dd/d28/d34/d70/f9b x:0 0 0 2024-04-22T23:23:48.863 INFO:tasks.workunit.client.1.smithi134.stdout:1/725: dwrite db/d15/d17/d20/d2a/d2d/fdd [0,4194304] 0 2024-04-22T23:23:48.868 INFO:tasks.workunit.client.1.smithi134.stdout:3/418: rename dc/dd/d28/d34/d6b/f99 to dc/dd/d28/d2f/d36/d6d/f9c 0 2024-04-22T23:23:48.873 INFO:tasks.workunit.client.1.smithi134.stdout:1/726: creat db/ff6 x:0 0 0 2024-04-22T23:23:48.874 INFO:tasks.workunit.client.1.smithi134.stdout:7/587: getdents d4/d8/d2a/d42 0 2024-04-22T23:23:48.878 INFO:tasks.workunit.client.1.smithi134.stdout:3/419: dwrite dc/dd/f7a [0,4194304] 0 2024-04-22T23:23:48.884 INFO:tasks.workunit.client.1.smithi134.stdout:7/588: unlink d4/d8/d2a/d42/cc7 0 2024-04-22T23:23:48.889 INFO:tasks.workunit.client.1.smithi134.stdout:1/727: dwrite db/d15/d17/d20/d2a/d2d/f76 [0,4194304] 0 2024-04-22T23:23:48.889 INFO:tasks.workunit.client.1.smithi134.stdout:3/420: rename dc/dd/d28/d2f/d36/l4c to dc/dd/d28/d34/d70/l9d 0 2024-04-22T23:23:48.890 INFO:tasks.workunit.client.1.smithi134.stdout:7/589: fsync d4/d8/d41/d76/dc1/d2f/da6/dcf/fd4 0 2024-04-22T23:23:48.894 INFO:tasks.workunit.client.1.smithi134.stdout:4/736: dwrite d4/d29/f63 [0,4194304] 0 2024-04-22T23:23:48.894 INFO:tasks.workunit.client.1.smithi134.stdout:1/728: chown db/d15/l1f 1 1 2024-04-22T23:23:48.902 INFO:tasks.workunit.client.1.smithi134.stdout:7/590: mkdir d4/d8/d2a/d42/d6e/dd6 0 2024-04-22T23:23:48.906 INFO:tasks.workunit.client.1.smithi134.stdout:7/591: mkdir d4/d4a/d85/dd7 0 2024-04-22T23:23:48.906 INFO:tasks.workunit.client.1.smithi134.stdout:7/592: readlink d4/d8/d3b/l59 0 2024-04-22T23:23:48.907 INFO:tasks.workunit.client.1.smithi134.stdout:4/737: symlink d4/d35/d8e/d5d/dce/le9 0 2024-04-22T23:23:48.907 INFO:tasks.workunit.client.1.smithi134.stdout:4/738: readlink d4/l47 0 2024-04-22T23:23:48.908 INFO:tasks.workunit.client.1.smithi134.stdout:3/421: getdents dc/dd/d28/d2f/d21 0 2024-04-22T23:23:48.909 INFO:tasks.workunit.client.1.smithi134.stdout:7/593: chown d4/d8/d41/d76/dc1/d2f/d34/d62/d29/dc5 13641062 1 2024-04-22T23:23:48.910 INFO:tasks.workunit.client.1.smithi134.stdout:7/594: chown d4/d8/d41/d76/dc1 26 1 2024-04-22T23:23:48.913 INFO:tasks.workunit.client.1.smithi134.stdout:7/595: chown d4/d8/d3b/cb6 3776485 1 2024-04-22T23:23:48.920 INFO:tasks.workunit.client.1.smithi134.stdout:1/729: dread db/fa6 [0,4194304] 0 2024-04-22T23:23:48.922 INFO:tasks.workunit.client.1.smithi134.stdout:7/596: dwrite d4/f2e [4194304,4194304] 0 2024-04-22T23:23:48.923 INFO:tasks.workunit.client.1.smithi134.stdout:7/597: read d4/d8/fb [1466875,3935] 0 2024-04-22T23:23:48.926 INFO:tasks.workunit.client.1.smithi134.stdout:1/730: truncate db/d15/d17/d20/d2a/d2d/ff1 708851 0 2024-04-22T23:23:48.939 INFO:tasks.workunit.client.1.smithi134.stdout:9/453: sync 2024-04-22T23:23:48.958 INFO:tasks.workunit.client.1.smithi134.stdout:1/731: write db/f5f [193633,84266] 0 2024-04-22T23:23:48.960 INFO:tasks.workunit.client.1.smithi134.stdout:3/422: dread dc/dd/d28/d2f/d36/d6d/d80/f93 [0,4194304] 0 2024-04-22T23:23:48.960 INFO:tasks.workunit.client.1.smithi134.stdout:7/598: unlink d4/d8/d41/d76/dc1/d2f/da6/la9 0 2024-04-22T23:23:48.962 INFO:tasks.workunit.client.1.smithi134.stdout:4/739: dread d4/dd/f16 [0,4194304] 0 2024-04-22T23:23:48.964 INFO:tasks.workunit.client.1.smithi134.stdout:1/732: mknod db/d15/d17/cf7 0 2024-04-22T23:23:48.966 INFO:tasks.workunit.client.1.smithi134.stdout:4/740: chown d4/d35/d8e/f8a 15793 1 2024-04-22T23:23:48.967 INFO:tasks.workunit.client.1.smithi134.stdout:4/741: stat d4/d6b 0 2024-04-22T23:23:48.969 INFO:tasks.workunit.client.1.smithi134.stdout:7/599: mknod d4/d8/d41/d76/dc1/d2f/d69/dab/cd8 0 2024-04-22T23:23:48.970 INFO:tasks.workunit.client.1.smithi134.stdout:9/454: dwrite d0/d37/f8b [0,4194304] 0 2024-04-22T23:23:48.970 INFO:tasks.workunit.client.1.smithi134.stdout:1/733: mknod db/d16/dbc/cf8 0 2024-04-22T23:23:48.972 INFO:tasks.workunit.client.1.smithi134.stdout:3/423: dread dc/dd/f7a [0,4194304] 0 2024-04-22T23:23:48.974 INFO:tasks.workunit.client.1.smithi134.stdout:1/734: unlink db/d16/dbc/cf8 0 2024-04-22T23:23:48.976 INFO:tasks.workunit.client.1.smithi134.stdout:7/600: creat d4/d4a/d85/dd7/fd9 x:0 0 0 2024-04-22T23:23:48.977 INFO:tasks.workunit.client.1.smithi134.stdout:9/455: creat d0/d2d/d82/f9a x:0 0 0 2024-04-22T23:23:48.977 INFO:tasks.workunit.client.1.smithi134.stdout:4/742: dread d4/dd/f41 [0,4194304] 0 2024-04-22T23:23:48.978 INFO:tasks.workunit.client.1.smithi134.stdout:7/601: write d4/d8/d2a/d42/fc3 [285973,26885] 0 2024-04-22T23:23:48.981 INFO:tasks.workunit.client.1.smithi134.stdout:1/735: rename db/d15/d72/ld2 to db/d16/lf9 0 2024-04-22T23:23:48.983 INFO:tasks.workunit.client.1.smithi134.stdout:4/743: unlink d4/d7/f28 0 2024-04-22T23:23:48.985 INFO:tasks.workunit.client.1.smithi134.stdout:3/424: dwrite dc/dd/d28/d2f/d36/d6d/d80/f93 [0,4194304] 0 2024-04-22T23:23:48.992 INFO:tasks.workunit.client.1.smithi134.stdout:3/425: symlink dc/dd/d28/d2f/d36/d6d/d80/l9e 0 2024-04-22T23:23:48.994 INFO:tasks.workunit.client.1.smithi134.stdout:4/744: dwrite d4/d13/fca [0,4194304] 0 2024-04-22T23:23:48.996 INFO:tasks.workunit.client.1.smithi134.stdout:7/602: write d4/d8/d2a/d42/d75/f77 [873039,23848] 0 2024-04-22T23:23:48.999 INFO:tasks.workunit.client.1.smithi134.stdout:3/426: creat dc/dd/d28/d34/f9f x:0 0 0 2024-04-22T23:23:49.002 INFO:tasks.workunit.client.1.smithi134.stdout:4/745: mkdir d4/d6b/dd9/dea 0 2024-04-22T23:23:49.003 INFO:tasks.workunit.client.1.smithi134.stdout:7/603: symlink d4/d8/d41/d76/dc1/d2f/d69/d8a/lda 0 2024-04-22T23:23:49.004 INFO:tasks.workunit.client.1.smithi134.stdout:7/604: chown d4/d8/d41/d76/dc1/d2f/d70 18614162 1 2024-04-22T23:23:49.005 INFO:tasks.workunit.client.1.smithi134.stdout:7/605: write d4/fa [3295934,120668] 0 2024-04-22T23:23:49.006 INFO:tasks.workunit.client.1.smithi134.stdout:7/606: dread - d4/d8/d2a/d42/f82 zero size 2024-04-22T23:23:49.007 INFO:tasks.workunit.client.1.smithi134.stdout:3/427: getdents dc/dd/d28 0 2024-04-22T23:23:49.013 INFO:tasks.workunit.client.1.smithi134.stdout:3/428: dread dc/dd/d28/d2f/f73 [0,4194304] 0 2024-04-22T23:23:49.013 INFO:tasks.workunit.client.1.smithi134.stdout:7/607: rename d4/d8/d41/d76/dc1/d2f/d69/d8a/da8 to d4/d8/d41/d76/dc1/d2f/d69/ddb 0 2024-04-22T23:23:49.018 INFO:tasks.workunit.client.1.smithi134.stdout:3/429: dread dc/dd/d28/d2f/d36/d6d/d80/f93 [0,4194304] 0 2024-04-22T23:23:49.019 INFO:tasks.workunit.client.1.smithi134.stdout:7/608: dread d4/f7b [4194304,4194304] 0 2024-04-22T23:23:49.020 INFO:tasks.workunit.client.1.smithi134.stdout:3/430: fsync dc/dd/d28/d2f/d21/d42/f66 0 2024-04-22T23:23:49.020 INFO:tasks.workunit.client.1.smithi134.stdout:3/431: chown dc/d2a/d62/d86 224173992 1 2024-04-22T23:23:49.025 INFO:tasks.workunit.client.1.smithi134.stdout:7/609: write d4/d8/ff [5185159,59335] 0 2024-04-22T23:23:49.026 INFO:tasks.workunit.client.1.smithi134.stdout:4/746: mknod d4/d29/dcc/d82/de7/ceb 0 2024-04-22T23:23:49.027 INFO:tasks.workunit.client.1.smithi134.stdout:3/432: link dc/dd/d28/d34/d6b/f9a dc/d2a/d30/fa0 0 2024-04-22T23:23:49.028 INFO:tasks.workunit.client.1.smithi134.stdout:7/610: creat d4/d8/d2a/d42/fdc x:0 0 0 2024-04-22T23:23:49.031 INFO:tasks.workunit.client.1.smithi134.stdout:3/433: link dc/d2a/d30/f60 dc/dd/d28/d34/d72/fa1 0 2024-04-22T23:23:49.033 INFO:tasks.workunit.client.1.smithi134.stdout:3/434: creat dc/dd/d28/d2f/d36/d6d/d80/fa2 x:0 0 0 2024-04-22T23:23:49.034 INFO:tasks.workunit.client.1.smithi134.stdout:7/611: rmdir d4/d8/d41/d76/dc1/d2f/d70/d86/d97 0 2024-04-22T23:23:49.035 INFO:tasks.workunit.client.1.smithi134.stdout:7/612: write d4/d8/d41/d76/dc1/fa5 [5136608,28407] 0 2024-04-22T23:23:49.036 INFO:tasks.workunit.client.1.smithi134.stdout:3/435: unlink dc/f8c 0 2024-04-22T23:23:49.044 INFO:tasks.workunit.client.1.smithi134.stdout:3/436: dwrite dc/dd/d28/d2f/d36/d6d/d80/fa2 [0,4194304] 0 2024-04-22T23:23:49.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:48 smithi067 ceph-mon[26453]: pgmap v279: 65 pgs: 65 active+clean; 2.8 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail; 87 MiB/s rd, 267 MiB/s wr, 547 op/s 2024-04-22T23:23:49.046 INFO:tasks.workunit.client.1.smithi134.stdout:3/437: mknod dc/d2a/d30/ca3 0 2024-04-22T23:23:49.055 INFO:tasks.workunit.client.1.smithi134.stdout:3/438: dwrite dc/dd/d28/f39 [0,4194304] 0 2024-04-22T23:23:49.062 INFO:tasks.workunit.client.1.smithi134.stdout:3/439: dread dc/dd/d28/d2f/f4b [0,4194304] 0 2024-04-22T23:23:49.062 INFO:tasks.workunit.client.1.smithi134.stdout:7/613: dread d4/d8/d3b/f48 [4194304,4194304] 0 2024-04-22T23:23:49.064 INFO:tasks.workunit.client.1.smithi134.stdout:7/614: write d4/f2e [2161093,47691] 0 2024-04-22T23:23:49.064 INFO:tasks.workunit.client.1.smithi134.stdout:3/440: symlink dc/dd/d28/d2f/d21/d42/d77/la4 0 2024-04-22T23:23:49.066 INFO:tasks.workunit.client.1.smithi134.stdout:3/441: write dc/d2a/d30/f83 [1037526,77045] 0 2024-04-22T23:23:49.067 INFO:tasks.workunit.client.1.smithi134.stdout:7/615: write d4/d8/d41/d76/dc1/d2f/d69/d8a/fb7 [972280,112974] 0 2024-04-22T23:23:49.068 INFO:tasks.workunit.client.1.smithi134.stdout:3/442: creat dc/d2a/d62/fa5 x:0 0 0 2024-04-22T23:23:49.069 INFO:tasks.workunit.client.1.smithi134.stdout:3/443: write dc/f52 [2070386,32834] 0 2024-04-22T23:23:49.071 INFO:tasks.workunit.client.1.smithi134.stdout:3/444: readlink dc/dd/d28/d2f/l59 0 2024-04-22T23:23:49.077 INFO:tasks.workunit.client.1.smithi134.stdout:3/445: truncate f9 7709768 0 2024-04-22T23:23:49.078 INFO:tasks.workunit.client.1.smithi134.stdout:3/446: write dc/dd/d28/d2f/f68 [429026,20763] 0 2024-04-22T23:23:49.084 INFO:tasks.workunit.client.1.smithi134.stdout:3/447: dwrite dc/f22 [0,4194304] 0 2024-04-22T23:23:49.086 INFO:tasks.workunit.client.1.smithi134.stdout:3/448: creat dc/d2a/d62/fa6 x:0 0 0 2024-04-22T23:23:49.088 INFO:tasks.workunit.client.1.smithi134.stdout:7/616: dread d4/d4a/d85/fb2 [0,4194304] 0 2024-04-22T23:23:49.089 INFO:tasks.workunit.client.1.smithi134.stdout:3/449: symlink dc/dd/d28/d2f/d36/d6d/d80/la7 0 2024-04-22T23:23:49.093 INFO:tasks.workunit.client.1.smithi134.stdout:3/450: mknod dc/dd/d28/d34/d70/ca8 0 2024-04-22T23:23:49.095 INFO:tasks.workunit.client.1.smithi134.stdout:7/617: dwrite d4/d8/d41/f43 [0,4194304] 0 2024-04-22T23:23:49.095 INFO:tasks.workunit.client.1.smithi134.stdout:3/451: read dc/dd/d28/d2f/f73 [1657316,119572] 0 2024-04-22T23:23:49.099 INFO:tasks.workunit.client.1.smithi134.stdout:3/452: unlink dc/dd/d28/c38 0 2024-04-22T23:23:49.101 INFO:tasks.workunit.client.1.smithi134.stdout:3/453: write dc/dd/d28/d2f/d54/f7b [886745,40991] 0 2024-04-22T23:23:49.101 INFO:tasks.workunit.client.1.smithi134.stdout:7/618: rename d4/d8/d41/d76/dc1/d2f/d69/ddb to d4/d8/d2a/d42/d6e/ddd 0 2024-04-22T23:23:49.103 INFO:tasks.workunit.client.1.smithi134.stdout:7/619: chown d4/dad/ld0 33 1 2024-04-22T23:23:49.107 INFO:tasks.workunit.client.1.smithi134.stdout:7/620: write d4/d8/d41/d76/dc1/d2f/d34/d62/d84/f9f [1623668,50427] 0 2024-04-22T23:23:49.108 INFO:tasks.workunit.client.1.smithi134.stdout:7/621: chown d4/d8/d3b/c53 12828351 1 2024-04-22T23:23:49.109 INFO:tasks.workunit.client.1.smithi134.stdout:3/454: dwrite dc/d2a/d62/fa5 [0,4194304] 0 2024-04-22T23:23:49.112 INFO:tasks.workunit.client.1.smithi134.stdout:3/455: mknod dc/d2a/d30/ca9 0 2024-04-22T23:23:49.120 INFO:tasks.workunit.client.1.smithi134.stdout:7/622: dwrite d4/d8/d41/d76/dc1/d2f/da6/dcf/fd4 [0,4194304] 0 2024-04-22T23:23:49.124 INFO:tasks.workunit.client.1.smithi134.stdout:7/623: symlink d4/d4a/d85/lde 0 2024-04-22T23:23:49.124 INFO:tasks.workunit.client.1.smithi134.stdout:3/456: dwrite dc/dd/d28/f1c [4194304,4194304] 0 2024-04-22T23:23:49.125 INFO:tasks.workunit.client.1.smithi134.stdout:7/624: fdatasync d4/d8/d3b/f68 0 2024-04-22T23:23:49.126 INFO:tasks.workunit.client.1.smithi134.stdout:2/498: sync 2024-04-22T23:23:49.126 INFO:tasks.workunit.client.1.smithi134.stdout:0/507: sync 2024-04-22T23:23:49.127 INFO:tasks.workunit.client.1.smithi134.stdout:7/625: chown d4/d8/d41/d76 479559216 1 2024-04-22T23:23:49.128 INFO:tasks.workunit.client.1.smithi134.stdout:7/626: chown d4/d8/d3b/c8b 401 1 2024-04-22T23:23:49.130 INFO:tasks.workunit.client.1.smithi134.stdout:7/627: chown d4/d8/d41/d76/dc1/d2f/d69/d8a/lda 59943114 1 2024-04-22T23:23:49.131 INFO:tasks.workunit.client.1.smithi134.stdout:0/508: symlink d1/d5/dd/d41/la4 0 2024-04-22T23:23:49.131 INFO:tasks.workunit.client.1.smithi134.stdout:2/499: creat d2/dd/df/d1c/fa1 x:0 0 0 2024-04-22T23:23:49.132 INFO:tasks.workunit.client.1.smithi134.stdout:3/457: getdents dc/dd/d28/d2f/d61/d6a 0 2024-04-22T23:23:49.133 INFO:tasks.workunit.client.1.smithi134.stdout:0/509: mknod d1/d5/ca5 0 2024-04-22T23:23:49.134 INFO:tasks.workunit.client.1.smithi134.stdout:7/628: symlink d4/d8/d3b/ldf 0 2024-04-22T23:23:49.135 INFO:tasks.workunit.client.1.smithi134.stdout:2/500: rename d2/dd/df/d1c/l25 to d2/d42/d54/d33/la2 0 2024-04-22T23:23:49.136 INFO:tasks.workunit.client.1.smithi134.stdout:0/510: creat d1/d77/fa6 x:0 0 0 2024-04-22T23:23:49.136 INFO:tasks.workunit.client.1.smithi134.stdout:2/501: fsync d2/f17 0 2024-04-22T23:23:49.137 INFO:tasks.workunit.client.1.smithi134.stdout:3/458: symlink dc/d7d/laa 0 2024-04-22T23:23:49.139 INFO:tasks.workunit.client.1.smithi134.stdout:0/511: chown d1/d5/d58/da0/f93 7 1 2024-04-22T23:23:49.140 INFO:tasks.workunit.client.1.smithi134.stdout:2/502: getdents d2/dd/df/d89 0 2024-04-22T23:23:49.142 INFO:tasks.workunit.client.1.smithi134.stdout:7/629: dwrite d4/d8/d41/d76/dc1/f1f [0,4194304] 0 2024-04-22T23:23:49.144 INFO:tasks.workunit.client.1.smithi134.stdout:2/503: creat d2/dd/d77/fa3 x:0 0 0 2024-04-22T23:23:49.145 INFO:tasks.workunit.client.1.smithi134.stdout:0/512: creat d1/d5/d58/d22/fa7 x:0 0 0 2024-04-22T23:23:49.151 INFO:tasks.workunit.client.1.smithi134.stdout:0/513: mkdir d1/da8 0 2024-04-22T23:23:49.153 INFO:tasks.workunit.client.1.smithi134.stdout:2/504: link d2/d15/d24/d1d/f26 d2/d42/d54/d33/fa4 0 2024-04-22T23:23:49.156 INFO:tasks.workunit.client.1.smithi134.stdout:2/505: mknod d2/d15/d24/d1d/d3d/ca5 0 2024-04-22T23:23:49.158 INFO:tasks.workunit.client.1.smithi134.stdout:0/514: rmdir d1/da8 0 2024-04-22T23:23:49.160 INFO:tasks.workunit.client.1.smithi134.stdout:0/515: chown d1/d5/dd/d41/l4d 24 1 2024-04-22T23:23:49.161 INFO:tasks.workunit.client.1.smithi134.stdout:7/630: truncate d4/f7b 3567626 0 2024-04-22T23:23:49.163 INFO:tasks.workunit.client.1.smithi134.stdout:0/516: creat d1/db/fa9 x:0 0 0 2024-04-22T23:23:49.164 INFO:tasks.workunit.client.1.smithi134.stdout:3/459: dwrite dc/dd/d28/d2f/d21/d42/f47 [0,4194304] 0 2024-04-22T23:23:49.169 INFO:tasks.workunit.client.1.smithi134.stdout:7/631: creat d4/d8/d41/d76/dc1/d2f/d34/d62/d29/dc5/fe0 x:0 0 0 2024-04-22T23:23:49.171 INFO:tasks.workunit.client.1.smithi134.stdout:3/460: creat dc/dd/d28/d34/d72/d75/fab x:0 0 0 2024-04-22T23:23:49.174 INFO:tasks.workunit.client.1.smithi134.stdout:7/632: truncate d4/d8/d41/d76/dc1/f23 5135311 0 2024-04-22T23:23:49.176 INFO:tasks.workunit.client.1.smithi134.stdout:7/633: chown d4/c28 667 1 2024-04-22T23:23:49.176 INFO:tasks.workunit.client.1.smithi134.stdout:3/461: mkdir dc/dd/d28/d2f/d54/dac 0 2024-04-22T23:23:49.176 INFO:tasks.workunit.client.1.smithi134.stdout:0/517: rmdir d1 39 2024-04-22T23:23:49.180 INFO:tasks.workunit.client.1.smithi134.stdout:3/462: rmdir dc/dd/d28/d2f/d36/d5f 39 2024-04-22T23:23:49.181 INFO:tasks.workunit.client.1.smithi134.stdout:2/506: dwrite d2/d15/d24/d1d/d3d/f71 [0,4194304] 0 2024-04-22T23:23:49.182 INFO:tasks.workunit.client.1.smithi134.stdout:3/463: chown dc/dd/d28/d2f/d21/d42/l5a 12 1 2024-04-22T23:23:49.183 INFO:tasks.workunit.client.1.smithi134.stdout:7/634: mkdir d4/d8/d41/d76/dc1/d2f/d69/dab/de1 0 2024-04-22T23:23:49.185 INFO:tasks.workunit.client.1.smithi134.stdout:7/635: chown d4/d8/d41/d76/dc1/d2f/l66 8769 1 2024-04-22T23:23:49.186 INFO:tasks.workunit.client.1.smithi134.stdout:7/636: readlink d4/d8/d2a/d42/l98 0 2024-04-22T23:23:49.187 INFO:tasks.workunit.client.1.smithi134.stdout:2/507: creat d2/d15/d24/d1d/fa6 x:0 0 0 2024-04-22T23:23:49.190 INFO:tasks.workunit.client.1.smithi134.stdout:2/508: truncate d2/d15/d24/d1d/f8d 840969 0 2024-04-22T23:23:49.190 INFO:tasks.workunit.client.1.smithi134.stdout:7/637: truncate d4/d8/d2a/d42/d75/fbf 886561 0 2024-04-22T23:23:49.190 INFO:tasks.workunit.client.1.smithi134.stdout:0/518: creat d1/d5/d58/d94/faa x:0 0 0 2024-04-22T23:23:49.192 INFO:tasks.workunit.client.1.smithi134.stdout:7/638: stat d4/d8/d41/d76/dc1/d2f/d34 0 2024-04-22T23:23:49.192 INFO:tasks.workunit.client.1.smithi134.stdout:0/519: write d1/d5/dd/f40 [2799010,130692] 0 2024-04-22T23:23:49.193 INFO:tasks.workunit.client.1.smithi134.stdout:2/509: symlink d2/dd/d4a/la7 0 2024-04-22T23:23:49.194 INFO:tasks.workunit.client.1.smithi134.stdout:7/639: dread - d4/d8/d41/d76/dc1/d2f/d70/f72 zero size 2024-04-22T23:23:49.195 INFO:tasks.workunit.client.1.smithi134.stdout:2/510: mkdir d2/dd/d4a/d73/da8 0 2024-04-22T23:23:49.198 INFO:tasks.workunit.client.1.smithi134.stdout:2/511: dread - d2/f98 zero size 2024-04-22T23:23:49.198 INFO:tasks.workunit.client.1.smithi134.stdout:7/640: creat d4/d8/d2a/d42/d6e/ddd/fe2 x:0 0 0 2024-04-22T23:23:49.200 INFO:tasks.workunit.client.1.smithi134.stdout:2/512: write d2/dd/df/f97 [317272,103887] 0 2024-04-22T23:23:49.201 INFO:tasks.workunit.client.1.smithi134.stdout:0/520: dwrite d1/d5/dd/d11/f18 [8388608,4194304] 0 2024-04-22T23:23:49.202 INFO:tasks.workunit.client.1.smithi134.stdout:2/513: chown d2/d15/d24/d1d/c5c 397 1 2024-04-22T23:23:49.204 INFO:tasks.workunit.client.1.smithi134.stdout:0/521: creat d1/db/fab x:0 0 0 2024-04-22T23:23:49.207 INFO:tasks.workunit.client.1.smithi134.stdout:2/514: creat d2/dd/d68/fa9 x:0 0 0 2024-04-22T23:23:49.208 INFO:tasks.workunit.client.1.smithi134.stdout:3/464: write dc/dd/f4f [4381140,12356] 0 2024-04-22T23:23:49.208 INFO:tasks.workunit.client.1.smithi134.stdout:0/522: mknod d1/d77/cac 0 2024-04-22T23:23:49.209 INFO:tasks.workunit.client.1.smithi134.stdout:2/515: stat d2/d42/d54/d33/l75 0 2024-04-22T23:23:49.209 INFO:tasks.workunit.client.1.smithi134.stdout:7/641: dwrite d4/d8/d41/d76/dc1/fa5 [0,4194304] 0 2024-04-22T23:23:49.210 INFO:tasks.workunit.client.1.smithi134.stdout:0/523: fdatasync d1/d5/d58/d22/fa7 0 2024-04-22T23:23:49.212 INFO:tasks.workunit.client.1.smithi134.stdout:0/524: fdatasync d1/db/fab 0 2024-04-22T23:23:49.212 INFO:tasks.workunit.client.1.smithi134.stdout:7/642: write d4/d4a/d85/dd7/fd9 [830137,72827] 0 2024-04-22T23:23:49.213 INFO:tasks.workunit.client.1.smithi134.stdout:7/643: readlink d4/d8/d41/d76/dc1/d2f/l66 0 2024-04-22T23:23:49.215 INFO:tasks.workunit.client.1.smithi134.stdout:0/525: write d1/d2/f54 [353488,10082] 0 2024-04-22T23:23:49.216 INFO:tasks.workunit.client.1.smithi134.stdout:7/644: chown d4/d8/d41/d76/dc1/d2f/d69/dab/de1 4 1 2024-04-22T23:23:49.221 INFO:tasks.workunit.client.1.smithi134.stdout:2/516: rmdir d2/dd/d4a/d73/da8 0 2024-04-22T23:23:49.224 INFO:tasks.workunit.client.1.smithi134.stdout:7/645: rename d4/d8/d41/d76/dc1/d2f/d34/d62/c3f to d4/d8/d41/d76/dc1/d2f/d70/ce3 0 2024-04-22T23:23:49.227 INFO:tasks.workunit.client.1.smithi134.stdout:2/517: read d2/f14 [1118967,56767] 0 2024-04-22T23:23:49.227 INFO:tasks.workunit.client.1.smithi134.stdout:0/526: dwrite d1/d5/d58/d22/f62 [0,4194304] 0 2024-04-22T23:23:49.227 INFO:tasks.workunit.client.1.smithi134.stdout:7/646: chown d4/d8/d41/d76/dc1/faa 251 1 2024-04-22T23:23:49.230 INFO:tasks.workunit.client.1.smithi134.stdout:7/647: mknod d4/d4a/ce4 0 2024-04-22T23:23:49.234 INFO:tasks.workunit.client.1.smithi134.stdout:2/518: mkdir d2/dd/df/d9c/daa 0 2024-04-22T23:23:49.235 INFO:tasks.workunit.client.1.smithi134.stdout:7/648: creat d4/d8/d41/d76/dc1/d2f/d69/d8a/fe5 x:0 0 0 2024-04-22T23:23:49.238 INFO:tasks.workunit.client.1.smithi134.stdout:3/465: dwrite dc/dd/d28/d2f/d36/f5d [0,4194304] 0 2024-04-22T23:23:49.238 INFO:tasks.workunit.client.1.smithi134.stdout:0/527: dwrite d1/d2/f74 [0,4194304] 0 2024-04-22T23:23:49.243 INFO:tasks.workunit.client.1.smithi134.stdout:3/466: write dc/dd/d28/f1c [569176,33959] 0 2024-04-22T23:23:49.247 INFO:tasks.workunit.client.1.smithi134.stdout:3/467: creat dc/d7d/fad x:0 0 0 2024-04-22T23:23:49.247 INFO:tasks.workunit.client.1.smithi134.stdout:0/528: write d1/d5/dd/d20/d2c/f4b [458912,115338] 0 2024-04-22T23:23:49.247 INFO:tasks.workunit.client.1.smithi134.stdout:2/519: dwrite d2/f17 [0,4194304] 0 2024-04-22T23:23:49.249 INFO:tasks.workunit.client.1.smithi134.stdout:2/520: read - d2/d42/d54/f9e zero size 2024-04-22T23:23:49.250 INFO:tasks.workunit.client.1.smithi134.stdout:0/529: symlink d1/d5/dd/d68/lad 0 2024-04-22T23:23:49.251 INFO:tasks.workunit.client.1.smithi134.stdout:2/521: chown d2/d42/d54/d33/d91 57 1 2024-04-22T23:23:49.252 INFO:tasks.workunit.client.1.smithi134.stdout:2/522: read d2/dd/d4a/d73/f7f [358469,4398] 0 2024-04-22T23:23:49.252 INFO:tasks.workunit.client.1.smithi134.stdout:3/468: dread dc/d2a/d62/fa5 [0,4194304] 0 2024-04-22T23:23:49.254 INFO:tasks.workunit.client.1.smithi134.stdout:3/469: write dc/dd/d28/d34/d70/f9b [520703,98495] 0 2024-04-22T23:23:49.255 INFO:tasks.workunit.client.1.smithi134.stdout:0/530: unlink d1/d5/l8a 0 2024-04-22T23:23:49.256 INFO:tasks.workunit.client.1.smithi134.stdout:0/531: rename d1 to d1/d5/da2/dae 22 2024-04-22T23:23:49.257 INFO:tasks.workunit.client.1.smithi134.stdout:0/532: chown d1/d2/d1d/c2f 454365 1 2024-04-22T23:23:49.259 INFO:tasks.workunit.client.1.smithi134.stdout:3/470: write dc/dd/d28/d34/d72/d75/f85 [98228,123297] 0 2024-04-22T23:23:49.260 INFO:tasks.workunit.client.1.smithi134.stdout:2/523: getdents d2 0 2024-04-22T23:23:49.262 INFO:tasks.workunit.client.1.smithi134.stdout:3/471: chown dc/dd/d28/d2f/d36/d5f 4 1 2024-04-22T23:23:49.266 INFO:tasks.workunit.client.1.smithi134.stdout:0/533: truncate d1/d2/f4 1315249 0 2024-04-22T23:23:49.266 INFO:tasks.workunit.client.1.smithi134.stdout:0/534: chown d1/d5/dd/d41/f5d 7326509 1 2024-04-22T23:23:49.267 INFO:tasks.workunit.client.1.smithi134.stdout:2/524: creat d2/dd/d72/fab x:0 0 0 2024-04-22T23:23:49.267 INFO:tasks.workunit.client.1.smithi134.stdout:3/472: rename dc/dd/d28/d34/d72/d75 to dc/dd/d28/d34/d6b/dae 0 2024-04-22T23:23:49.268 INFO:tasks.workunit.client.1.smithi134.stdout:0/535: symlink d1/d5/dd/d11/d75/laf 0 2024-04-22T23:23:49.269 INFO:tasks.workunit.client.1.smithi134.stdout:0/536: chown d1/d5/d58/f2e 14751 1 2024-04-22T23:23:49.272 INFO:tasks.workunit.client.1.smithi134.stdout:3/473: rename dc/dd/d28/c71 to dc/dd/d28/d2f/d61/d6a/caf 0 2024-04-22T23:23:49.273 INFO:tasks.workunit.client.1.smithi134.stdout:0/537: symlink d1/d5/dd/d68/lb0 0 2024-04-22T23:23:49.275 INFO:tasks.workunit.client.1.smithi134.stdout:7/649: dwrite d4/d8/d41/d76/dc1/d2f/d34/d62/d29/f55 [0,4194304] 0 2024-04-22T23:23:49.276 INFO:tasks.workunit.client.1.smithi134.stdout:3/474: creat dc/dd/d28/d3b/fb0 x:0 0 0 2024-04-22T23:23:49.283 INFO:tasks.workunit.client.1.smithi134.stdout:0/538: symlink d1/d5/dd/lb1 0 2024-04-22T23:23:49.285 INFO:tasks.workunit.client.1.smithi134.stdout:3/475: chown dc/dd/c25 869037 1 2024-04-22T23:23:49.286 INFO:tasks.workunit.client.1.smithi134.stdout:0/539: mknod d1/db/cb2 0 2024-04-22T23:23:49.288 INFO:tasks.workunit.client.1.smithi134.stdout:3/476: symlink dc/d2a/d62/d86/lb1 0 2024-04-22T23:23:49.295 INFO:tasks.workunit.client.1.smithi134.stdout:3/477: dwrite dc/dd/d28/d3b/f31 [4194304,4194304] 0 2024-04-22T23:23:49.297 INFO:tasks.workunit.client.1.smithi134.stdout:2/525: dread d2/d42/d54/d33/fa4 [0,4194304] 0 2024-04-22T23:23:49.297 INFO:tasks.workunit.client.1.smithi134.stdout:7/650: dread d4/d8/ff [0,4194304] 0 2024-04-22T23:23:49.339 INFO:tasks.workunit.client.1.smithi134.stdout:5/649: sync 2024-04-22T23:23:49.339 INFO:tasks.workunit.client.1.smithi134.stdout:0/540: creat d1/d5/d58/d22/fb3 x:0 0 0 2024-04-22T23:23:49.344 INFO:tasks.workunit.client.1.smithi134.stdout:3/478: rename dc/dd/d28 to dc/d2a/d62/db2 0 2024-04-22T23:23:49.345 INFO:tasks.workunit.client.1.smithi134.stdout:2/526: symlink d2/dd/df/d1c/lac 0 2024-04-22T23:23:49.346 INFO:tasks.workunit.client.1.smithi134.stdout:7/651: rmdir d4/d8/d41/d76 39 2024-04-22T23:23:49.352 INFO:tasks.workunit.client.1.smithi134.stdout:3/479: truncate dc/d2a/d62/db2/d34/d72/fa1 1030896 0 2024-04-22T23:23:49.353 INFO:tasks.workunit.client.1.smithi134.stdout:2/527: creat d2/d42/d54/d33/d91/fad x:0 0 0 2024-04-22T23:23:49.353 INFO:tasks.workunit.client.1.smithi134.stdout:3/480: dread - dc/d2a/d62/db2/d2f/d36/d6d/f9c zero size 2024-04-22T23:23:49.355 INFO:tasks.workunit.client.1.smithi134.stdout:0/541: creat d1/d5/d15/fb4 x:0 0 0 2024-04-22T23:23:49.356 INFO:tasks.workunit.client.1.smithi134.stdout:2/528: rmdir d2/d15/d24 39 2024-04-22T23:23:49.356 INFO:tasks.workunit.client.1.smithi134.stdout:5/650: dread d9/dd/f7b [0,4194304] 0 2024-04-22T23:23:49.357 INFO:tasks.workunit.client.1.smithi134.stdout:0/542: write d1/d5/dd/f7a [8650995,50571] 0 2024-04-22T23:23:49.357 INFO:tasks.workunit.client.1.smithi134.stdout:2/529: chown d2/dd/df/f85 1 1 2024-04-22T23:23:49.359 INFO:tasks.workunit.client.1.smithi134.stdout:7/652: rename d4/d8/d41/d76/dc1/l65 to d4/d8/d2a/d42/d6e/dd6/le6 0 2024-04-22T23:23:49.359 INFO:tasks.workunit.client.1.smithi134.stdout:3/481: write dc/d2a/d62/fa5 [3596166,16056] 0 2024-04-22T23:23:49.359 INFO:tasks.workunit.client.1.smithi134.stdout:0/543: chown d1/d5/dd/f35 115122831 1 2024-04-22T23:23:49.360 INFO:tasks.workunit.client.1.smithi134.stdout:5/651: mkdir d9/dd/d88/de4 0 2024-04-22T23:23:49.361 INFO:tasks.workunit.client.1.smithi134.stdout:2/530: creat d2/dd/d72/fae x:0 0 0 2024-04-22T23:23:49.363 INFO:tasks.workunit.client.1.smithi134.stdout:5/652: mkdir d9/dd/d17/d2a/de5 0 2024-04-22T23:23:49.365 INFO:tasks.workunit.client.1.smithi134.stdout:5/653: chown d9/dd/d88/db7/cdd 11 1 2024-04-22T23:23:49.365 INFO:tasks.workunit.client.1.smithi134.stdout:2/531: write d2/d15/d24/f6d [3952444,22274] 0 2024-04-22T23:23:49.366 INFO:tasks.workunit.client.1.smithi134.stdout:0/544: dwrite d1/d5/d58/f46 [0,4194304] 0 2024-04-22T23:23:49.367 INFO:tasks.workunit.client.1.smithi134.stdout:0/545: stat d1/d5/dd/f45 0 2024-04-22T23:23:49.373 INFO:tasks.workunit.client.1.smithi134.stdout:2/532: dwrite d2/d15/d24/f69 [0,4194304] 0 2024-04-22T23:23:49.376 INFO:tasks.workunit.client.1.smithi134.stdout:0/546: dwrite d1/d5/d58/d22/fa7 [0,4194304] 0 2024-04-22T23:23:49.409 INFO:tasks.workunit.client.1.smithi134.stdout:0/547: dread d1/d2/d1d/f7d [0,4194304] 0 2024-04-22T23:23:49.464 INFO:tasks.workunit.client.1.smithi134.stdout:5/654: unlink d9/d1e/c7f 0 2024-04-22T23:23:49.469 INFO:tasks.workunit.client.1.smithi134.stdout:0/548: mknod d1/d5/dd/d20/d2c/cb5 0 2024-04-22T23:23:49.470 INFO:tasks.workunit.client.1.smithi134.stdout:5/655: rename d9/d11/d34/d3b to d9/da2/de6 0 2024-04-22T23:23:49.471 INFO:tasks.workunit.client.1.smithi134.stdout:2/533: symlink d2/d42/laf 0 2024-04-22T23:23:49.472 INFO:tasks.workunit.client.1.smithi134.stdout:2/534: stat d2/d15/d24/l4c 0 2024-04-22T23:23:49.474 INFO:tasks.workunit.client.1.smithi134.stdout:0/549: creat d1/d5/d58/da0/fb6 x:0 0 0 2024-04-22T23:23:49.475 INFO:tasks.workunit.client.1.smithi134.stdout:2/535: write d2/d15/d24/d1d/d3d/f58 [6304645,106556] 0 2024-04-22T23:23:49.477 INFO:tasks.workunit.client.1.smithi134.stdout:0/550: fdatasync d1/d5/d58/da0/f93 0 2024-04-22T23:23:49.479 INFO:tasks.workunit.client.1.smithi134.stdout:5/656: dwrite d9/dd/d17/d2a/f98 [0,4194304] 0 2024-04-22T23:23:49.483 INFO:tasks.workunit.client.1.smithi134.stdout:0/551: write d1/d2/d1d/f7d [4957712,49552] 0 2024-04-22T23:23:49.484 INFO:tasks.workunit.client.1.smithi134.stdout:5/657: symlink d9/dd/d88/le7 0 2024-04-22T23:23:49.487 INFO:tasks.workunit.client.1.smithi134.stdout:5/658: symlink d9/da2/le8 0 2024-04-22T23:23:49.488 INFO:tasks.workunit.client.1.smithi134.stdout:0/552: readlink d1/d5/d15/l67 0 2024-04-22T23:23:49.491 INFO:tasks.workunit.client.1.smithi134.stdout:0/553: creat d1/d5/dd/d41/fb7 x:0 0 0 2024-04-22T23:23:49.492 INFO:tasks.workunit.client.1.smithi134.stdout:2/536: write d2/f14 [3743915,8341] 0 2024-04-22T23:23:49.493 INFO:tasks.workunit.client.1.smithi134.stdout:0/554: creat d1/d5/dd/d11/d75/fb8 x:0 0 0 2024-04-22T23:23:49.496 INFO:tasks.workunit.client.1.smithi134.stdout:2/537: rename d2/d42/d93 to d2/dd/df/d9c/daa/db0 0 2024-04-22T23:23:49.499 INFO:tasks.workunit.client.1.smithi134.stdout:2/538: mknod d2/dd/d4a/d73/cb1 0 2024-04-22T23:23:49.501 INFO:tasks.workunit.client.1.smithi134.stdout:0/555: rename d1/d5/dd/d11 to d1/d5/d58/d6b/db9 0 2024-04-22T23:23:49.503 INFO:tasks.workunit.client.1.smithi134.stdout:5/659: dwrite d9/dd/d17/f71 [0,4194304] 0 2024-04-22T23:23:49.505 INFO:tasks.workunit.client.1.smithi134.stdout:2/539: symlink d2/d42/lb2 0 2024-04-22T23:23:49.507 INFO:tasks.workunit.client.1.smithi134.stdout:2/540: chown d2/dd/d4a/la7 28 1 2024-04-22T23:23:49.509 INFO:tasks.workunit.client.1.smithi134.stdout:2/541: dread d2/d15/d24/d1d/f8d [0,4194304] 0 2024-04-22T23:23:49.510 INFO:tasks.workunit.client.1.smithi134.stdout:0/556: getdents d1 0 2024-04-22T23:23:49.510 INFO:tasks.workunit.client.1.smithi134.stdout:2/542: write d2/dd/d68/fa9 [87023,353] 0 2024-04-22T23:23:49.512 INFO:tasks.workunit.client.1.smithi134.stdout:8/531: sync 2024-04-22T23:23:49.515 INFO:tasks.workunit.client.1.smithi134.stdout:2/543: mkdir d2/d15/d24/d1d/db3 0 2024-04-22T23:23:49.515 INFO:tasks.workunit.client.1.smithi134.stdout:8/532: write d7/db/d2d/d39/f3d [895969,129556] 0 2024-04-22T23:23:49.517 INFO:tasks.workunit.client.1.smithi134.stdout:2/544: readlink d2/d15/d24/l80 0 2024-04-22T23:23:49.518 INFO:tasks.workunit.client.1.smithi134.stdout:8/533: rmdir d7/dc/d2c 39 2024-04-22T23:23:49.519 INFO:tasks.workunit.client.1.smithi134.stdout:2/545: dread - d2/dd/d77/fa3 zero size 2024-04-22T23:23:49.519 INFO:tasks.workunit.client.1.smithi134.stdout:0/557: dwrite d1/d2/f56 [0,4194304] 0 2024-04-22T23:23:49.522 INFO:tasks.workunit.client.1.smithi134.stdout:5/660: truncate d9/d11/d34/f51 3224828 0 2024-04-22T23:23:49.522 INFO:tasks.workunit.client.1.smithi134.stdout:5/661: chown d9/d1e/f33 1800959 1 2024-04-22T23:23:49.523 INFO:tasks.workunit.client.1.smithi134.stdout:5/662: dread - d9/dbb/fc8 zero size 2024-04-22T23:23:49.524 INFO:tasks.workunit.client.1.smithi134.stdout:0/558: creat d1/d5/d15/fba x:0 0 0 2024-04-22T23:23:49.525 INFO:tasks.workunit.client.1.smithi134.stdout:5/663: symlink d9/dd/d17/d2a/le9 0 2024-04-22T23:23:49.527 INFO:tasks.workunit.client.1.smithi134.stdout:5/664: chown d9/dd/d17/d2a/d49/dc2 4935 1 2024-04-22T23:23:49.528 INFO:tasks.workunit.client.1.smithi134.stdout:0/559: dread d1/d5/dd/d20/d2c/f4b [0,4194304] 0 2024-04-22T23:23:49.530 INFO:tasks.workunit.client.1.smithi134.stdout:0/560: rmdir d1/d2 39 2024-04-22T23:23:49.536 INFO:tasks.workunit.client.1.smithi134.stdout:0/561: creat d1/d5/d58/d22/fbb x:0 0 0 2024-04-22T23:23:49.539 INFO:tasks.workunit.client.1.smithi134.stdout:0/562: creat d1/fbc x:0 0 0 2024-04-22T23:23:49.545 INFO:tasks.workunit.client.1.smithi134.stdout:5/665: dwrite d9/d1e/f33 [4194304,4194304] 0 2024-04-22T23:23:49.549 INFO:tasks.workunit.client.1.smithi134.stdout:0/563: dwrite d1/d5/d58/da0/f7b [0,4194304] 0 2024-04-22T23:23:49.549 INFO:tasks.workunit.client.1.smithi134.stdout:5/666: symlink d9/da2/lea 0 2024-04-22T23:23:49.550 INFO:tasks.workunit.client.1.smithi134.stdout:0/564: stat d1/d5/d58/d6b 0 2024-04-22T23:23:49.551 INFO:tasks.workunit.client.1.smithi134.stdout:0/565: truncate d1/d5/f5e 888234 0 2024-04-22T23:23:49.557 INFO:tasks.workunit.client.1.smithi134.stdout:0/566: mknod d1/db/d9f/cbd 0 2024-04-22T23:23:49.559 INFO:tasks.workunit.client.1.smithi134.stdout:5/667: dwrite d9/d1e/f33 [0,4194304] 0 2024-04-22T23:23:49.560 INFO:tasks.workunit.client.1.smithi134.stdout:0/567: creat d1/d5/fbe x:0 0 0 2024-04-22T23:23:49.564 INFO:tasks.workunit.client.1.smithi134.stdout:0/568: truncate d1/f44 2777435 0 2024-04-22T23:23:49.567 INFO:tasks.workunit.client.1.smithi134.stdout:0/569: mknod d1/d5/dd/d68/cbf 0 2024-04-22T23:23:49.568 INFO:tasks.workunit.client.1.smithi134.stdout:5/668: dwrite d9/d11/f5b [0,4194304] 0 2024-04-22T23:23:49.570 INFO:tasks.workunit.client.1.smithi134.stdout:0/570: write d1/d5/d58/da0/f93 [182346,61657] 0 2024-04-22T23:23:49.571 INFO:tasks.workunit.client.1.smithi134.stdout:0/571: chown d1/d77 2379 1 2024-04-22T23:23:49.574 INFO:tasks.workunit.client.1.smithi134.stdout:0/572: mkdir d1/d5/dd/d68/dc0 0 2024-04-22T23:23:49.577 INFO:tasks.workunit.client.1.smithi134.stdout:5/669: dwrite d9/dd/f4c [4194304,4194304] 0 2024-04-22T23:23:49.579 INFO:tasks.workunit.client.1.smithi134.stdout:5/670: write d9/da2/fdb [865650,125222] 0 2024-04-22T23:23:49.582 INFO:tasks.workunit.client.1.smithi134.stdout:0/573: rename d1/d5/d58/d6b/db9/f99 to d1/d5/fc1 0 2024-04-22T23:23:49.584 INFO:tasks.workunit.client.1.smithi134.stdout:0/574: chown d1/d5/d58/d6b/db9/d75/laf 3 1 2024-04-22T23:23:49.585 INFO:tasks.workunit.client.1.smithi134.stdout:0/575: write d1/fbc [916795,73189] 0 2024-04-22T23:23:49.587 INFO:tasks.workunit.client.1.smithi134.stdout:5/671: dwrite d9/d11/d73/d95/d20/f91 [0,4194304] 0 2024-04-22T23:23:49.592 INFO:tasks.workunit.client.1.smithi134.stdout:0/576: link d1/d5/dd/f7a d1/d2/d1d/fc2 0 2024-04-22T23:23:49.593 INFO:tasks.workunit.client.1.smithi134.stdout:0/577: chown d1/d5/fbe 6568 1 2024-04-22T23:23:49.596 INFO:tasks.workunit.client.1.smithi134.stdout:0/578: link d1/d2/d1d/f7d d1/d5/dd/d41/fc3 0 2024-04-22T23:23:49.598 INFO:tasks.workunit.client.1.smithi134.stdout:0/579: truncate d1/f95 707129 0 2024-04-22T23:23:49.604 INFO:tasks.workunit.client.1.smithi134.stdout:0/580: dwrite d1/d5/dd/f40 [0,4194304] 0 2024-04-22T23:23:49.607 INFO:tasks.workunit.client.1.smithi134.stdout:0/581: fdatasync d1/d2/d1d/fc2 0 2024-04-22T23:23:49.609 INFO:tasks.workunit.client.1.smithi134.stdout:0/582: truncate d1/d5/d58/d22/fb3 765106 0 2024-04-22T23:23:49.611 INFO:tasks.workunit.client.1.smithi134.stdout:0/583: symlink d1/d5/dd/d68/lc4 0 2024-04-22T23:23:49.616 INFO:tasks.workunit.client.1.smithi134.stdout:5/672: dread d9/d11/d34/d85/fa5 [0,4194304] 0 2024-04-22T23:23:49.618 INFO:tasks.workunit.client.1.smithi134.stdout:5/673: symlink d9/da2/de6/dc0/leb 0 2024-04-22T23:23:49.619 INFO:tasks.workunit.client.1.smithi134.stdout:5/674: getdents d9/dd/d88/de4 0 2024-04-22T23:23:49.621 INFO:tasks.workunit.client.1.smithi134.stdout:5/675: mknod d9/d11/d73/d95/dae/de0/cec 0 2024-04-22T23:23:49.622 INFO:tasks.workunit.client.1.smithi134.stdout:5/676: mknod d9/d1e/db4/ced 0 2024-04-22T23:23:49.625 INFO:tasks.workunit.client.1.smithi134.stdout:5/677: link d9/d11/d34/d85/ld5 d9/da2/de6/lee 0 2024-04-22T23:23:49.628 INFO:tasks.workunit.client.1.smithi134.stdout:5/678: link d9/f15 d9/dd/d17/fef 0 2024-04-22T23:23:49.630 INFO:tasks.workunit.client.1.smithi134.stdout:0/584: truncate d1/d5/dd/f16 3891633 0 2024-04-22T23:23:49.630 INFO:tasks.workunit.client.1.smithi134.stdout:5/679: symlink d9/d11/de1/lf0 0 2024-04-22T23:23:49.631 INFO:tasks.workunit.client.1.smithi134.stdout:0/585: write d1/d2/d1d/f69 [794484,105779] 0 2024-04-22T23:23:49.631 INFO:tasks.workunit.client.1.smithi134.stdout:0/586: read - d1/d5/d15/fba zero size 2024-04-22T23:23:49.632 INFO:tasks.workunit.client.1.smithi134.stdout:0/587: fdatasync d1/d5/f8f 0 2024-04-22T23:23:49.636 INFO:tasks.workunit.client.1.smithi134.stdout:6/658: sync 2024-04-22T23:23:49.636 INFO:tasks.workunit.client.1.smithi134.stdout:7/653: sync 2024-04-22T23:23:49.636 INFO:tasks.workunit.client.1.smithi134.stdout:1/736: sync 2024-04-22T23:23:49.636 INFO:tasks.workunit.client.1.smithi134.stdout:9/456: sync 2024-04-22T23:23:49.637 INFO:tasks.workunit.client.1.smithi134.stdout:4/747: sync 2024-04-22T23:23:49.638 INFO:tasks.workunit.client.1.smithi134.stdout:5/680: dwrite d9/d11/d34/d85/fd3 [0,4194304] 0 2024-04-22T23:23:49.644 INFO:tasks.workunit.client.1.smithi134.stdout:0/588: dwrite d1/d2/f54 [0,4194304] 0 2024-04-22T23:23:49.646 INFO:tasks.workunit.client.1.smithi134.stdout:1/737: truncate db/d15/d17/d20/d2a/d2d/ff1 917927 0 2024-04-22T23:23:49.646 INFO:tasks.workunit.client.1.smithi134.stdout:6/659: rename d0/d19/de/f5f to d0/d19/de/d1f/dae/fd8 0 2024-04-22T23:23:49.651 INFO:tasks.workunit.client.1.smithi134.stdout:7/654: mknod d4/d8/d41/d76/dc1/d2f/d69/dab/ce7 0 2024-04-22T23:23:49.652 INFO:tasks.workunit.client.1.smithi134.stdout:9/457: mknod d0/d2d/d94/d98/c9b 0 2024-04-22T23:23:49.654 INFO:tasks.workunit.client.1.smithi134.stdout:5/681: dwrite d9/d1e/fb6 [0,4194304] 0 2024-04-22T23:23:49.654 INFO:tasks.workunit.client.1.smithi134.stdout:4/748: truncate d4/d35/f62 4126652 0 2024-04-22T23:23:49.660 INFO:tasks.workunit.client.1.smithi134.stdout:1/738: dwrite db/d16/fb8 [0,4194304] 0 2024-04-22T23:23:49.661 INFO:tasks.workunit.client.1.smithi134.stdout:1/739: readlink db/d15/d17/d20/d2a/l3e 0 2024-04-22T23:23:49.663 INFO:tasks.workunit.client.1.smithi134.stdout:7/655: mknod d4/d8/d3b/ce8 0 2024-04-22T23:23:49.663 INFO:tasks.workunit.client.1.smithi134.stdout:0/589: mknod d1/cc5 0 2024-04-22T23:23:49.666 INFO:tasks.workunit.client.1.smithi134.stdout:6/660: creat d0/d19/de/d12/d29/d74/da0/fd9 x:0 0 0 2024-04-22T23:23:49.671 INFO:tasks.workunit.client.1.smithi134.stdout:0/590: creat d1/d2/d1d/fc6 x:0 0 0 2024-04-22T23:23:49.671 INFO:tasks.workunit.client.1.smithi134.stdout:9/458: rmdir d0 39 2024-04-22T23:23:49.674 INFO:tasks.workunit.client.1.smithi134.stdout:5/682: dread d9/d1e/f5a [0,4194304] 0 2024-04-22T23:23:49.675 INFO:tasks.workunit.client.1.smithi134.stdout:6/661: rename d0/d19/db/d35/f44 to d0/d19/de/d1f/dd5/fda 0 2024-04-22T23:23:49.679 INFO:tasks.workunit.client.1.smithi134.stdout:0/591: creat d1/d5/d58/fc7 x:0 0 0 2024-04-22T23:23:49.681 INFO:tasks.workunit.client.1.smithi134.stdout:5/683: chown d9/dd/d52/c69 3 1 2024-04-22T23:23:49.683 INFO:tasks.workunit.client.1.smithi134.stdout:7/656: dread d4/f31 [0,4194304] 0 2024-04-22T23:23:49.685 INFO:tasks.workunit.client.1.smithi134.stdout:6/662: symlink d0/d19/de/d12/d5e/dc1/ldb 0 2024-04-22T23:23:49.686 INFO:tasks.workunit.client.1.smithi134.stdout:0/592: creat d1/d5/dd/d41/fc8 x:0 0 0 2024-04-22T23:23:49.687 INFO:tasks.workunit.client.1.smithi134.stdout:1/740: getdents db/d15/d17/d20/d2a/d2d/d86/dad 0 2024-04-22T23:23:49.692 INFO:tasks.workunit.client.1.smithi134.stdout:7/657: dwrite d4/d4a/d85/dd7/fd9 [0,4194304] 0 2024-04-22T23:23:49.694 INFO:tasks.workunit.client.1.smithi134.stdout:6/663: creat d0/d19/db/d75/d92/fdc x:0 0 0 2024-04-22T23:23:49.695 INFO:tasks.workunit.client.1.smithi134.stdout:6/664: stat d0/d19/de/f1d 0 2024-04-22T23:23:49.699 INFO:tasks.workunit.client.1.smithi134.stdout:6/665: fsync d0/d19/de/d12/d29/d74/da0/fab 0 2024-04-22T23:23:49.699 INFO:tasks.workunit.client.1.smithi134.stdout:0/593: truncate d1/d5/dd/d41/fc3 1722185 0 2024-04-22T23:23:49.699 INFO:tasks.workunit.client.1.smithi134.stdout:5/684: write d9/f15 [4054559,78671] 0 2024-04-22T23:23:49.699 INFO:tasks.workunit.client.1.smithi134.stdout:9/459: dread d0/d37/d76/f91 [0,4194304] 0 2024-04-22T23:23:49.701 INFO:tasks.workunit.client.1.smithi134.stdout:9/460: write d0/d7/f9 [44535,51323] 0 2024-04-22T23:23:49.702 INFO:tasks.workunit.client.1.smithi134.stdout:9/461: stat d0/d2d/d94/d98 0 2024-04-22T23:23:49.703 INFO:tasks.workunit.client.1.smithi134.stdout:7/658: rename d4/d8/d2a/d42/f82 to d4/d8/d41/d76/dc1/d2f/d69/fe9 0 2024-04-22T23:23:49.705 INFO:tasks.workunit.client.1.smithi134.stdout:5/685: fsync d9/dd/d17/d44/f7e 0 2024-04-22T23:23:49.708 INFO:tasks.workunit.client.1.smithi134.stdout:6/666: creat d0/d19/de/d12/d29/d74/da0/fdd x:0 0 0 2024-04-22T23:23:49.709 INFO:tasks.workunit.client.1.smithi134.stdout:1/741: truncate db/d15/d17/d20/d2a/fc3 2839707 0 2024-04-22T23:23:49.711 INFO:tasks.workunit.client.1.smithi134.stdout:9/462: unlink d0/d2d/d4e/l39 0 2024-04-22T23:23:49.713 INFO:tasks.workunit.client.1.smithi134.stdout:7/659: symlink d4/d8/d41/d76/dc1/d2f/d34/d62/lea 0 2024-04-22T23:23:49.713 INFO:tasks.workunit.client.1.smithi134.stdout:1/742: write db/db5/fdb [610392,6496] 0 2024-04-22T23:23:49.715 INFO:tasks.workunit.client.1.smithi134.stdout:6/667: write d0/d19/de/d12/d5e/d6d/f73 [858427,117925] 0 2024-04-22T23:23:49.718 INFO:tasks.workunit.client.1.smithi134.stdout:6/668: fdatasync d0/f16 0 2024-04-22T23:23:49.720 INFO:tasks.workunit.client.1.smithi134.stdout:0/594: dwrite d1/d5/d58/d22/f85 [0,4194304] 0 2024-04-22T23:23:49.722 INFO:tasks.workunit.client.1.smithi134.stdout:7/660: rename d4/f2d to d4/d8/d2a/d42/d75/feb 0 2024-04-22T23:23:49.723 INFO:tasks.workunit.client.1.smithi134.stdout:5/686: dwrite d9/dd/f86 [0,4194304] 0 2024-04-22T23:23:49.725 INFO:tasks.workunit.client.1.smithi134.stdout:0/595: fsync d1/d5/dd/f2a 0 2024-04-22T23:23:49.726 INFO:tasks.workunit.client.1.smithi134.stdout:0/596: readlink d1/d5/d58/d6b/db9/d75/laf 0 2024-04-22T23:23:49.727 INFO:tasks.workunit.client.1.smithi134.stdout:6/669: creat d0/d19/de/d12/d5e/d6d/d7a/fde x:0 0 0 2024-04-22T23:23:49.729 INFO:tasks.workunit.client.1.smithi134.stdout:5/687: fdatasync d9/dd/d17/d2a/d49/dc2/fcb 0 2024-04-22T23:23:49.730 INFO:tasks.workunit.client.1.smithi134.stdout:5/688: dread - d9/d11/d34/fbd zero size 2024-04-22T23:23:49.730 INFO:tasks.workunit.client.1.smithi134.stdout:7/661: mkdir d4/d8/d41/d76/dc1/d2f/d69/dab/d95/dec 0 2024-04-22T23:23:49.732 INFO:tasks.workunit.client.1.smithi134.stdout:7/662: write d4/d8/d41/d76/dc1/d2f/d70/f72 [428658,59255] 0 2024-04-22T23:23:49.734 INFO:tasks.workunit.client.1.smithi134.stdout:9/463: write d0/d7/f17 [711364,1715] 0 2024-04-22T23:23:49.739 INFO:tasks.workunit.client.1.smithi134.stdout:0/597: link d1/d2/f74 d1/d5/dd/d20/fc9 0 2024-04-22T23:23:49.740 INFO:tasks.workunit.client.1.smithi134.stdout:1/743: rename db/d15/d72 to db/d15/d17/d20/dfa 0 2024-04-22T23:23:49.742 INFO:tasks.workunit.client.1.smithi134.stdout:7/663: symlink d4/d8/d41/d76/dc1/d2f/d70/led 0 2024-04-22T23:23:49.743 INFO:tasks.workunit.client.1.smithi134.stdout:6/670: creat d0/d19/de/d12/d2a/dbb/fdf x:0 0 0 2024-04-22T23:23:49.744 INFO:tasks.workunit.client.1.smithi134.stdout:5/689: mkdir d9/dd/df1 0 2024-04-22T23:23:49.749 INFO:tasks.workunit.client.1.smithi134.stdout:0/598: truncate d1/d2/d1d/f7d 2468543 0 2024-04-22T23:23:49.751 INFO:tasks.workunit.client.1.smithi134.stdout:0/599: chown d1/d5/dd/lb1 34880283 1 2024-04-22T23:23:49.752 INFO:tasks.workunit.client.1.smithi134.stdout:7/664: dwrite d4/d8/d41/d76/dc1/fa5 [0,4194304] 0 2024-04-22T23:23:49.754 INFO:tasks.workunit.client.1.smithi134.stdout:1/744: link db/d16/d56/c97 db/db0/cfb 0 2024-04-22T23:23:49.757 INFO:tasks.workunit.client.1.smithi134.stdout:6/671: dwrite d0/d19/de/d12/d29/d74/da0/fdd [0,4194304] 0 2024-04-22T23:23:49.757 INFO:tasks.workunit.client.1.smithi134.stdout:0/600: fdatasync d1/d5/d58/d22/fbb 0 2024-04-22T23:23:49.761 INFO:tasks.workunit.client.1.smithi134.stdout:6/672: write d0/d19/de/d12/d2a/dbb/fdf [839656,97016] 0 2024-04-22T23:23:49.768 INFO:tasks.workunit.client.1.smithi134.stdout:7/665: link d4/d8/d41/d76/dc1/fa5 d4/d8/d41/d76/dc1/d2f/d34/d62/d84/fee 0 2024-04-22T23:23:49.771 INFO:tasks.workunit.client.1.smithi134.stdout:6/673: creat d0/d19/de/d12/d5e/dd6/fe0 x:0 0 0 2024-04-22T23:23:49.772 INFO:tasks.workunit.client.1.smithi134.stdout:0/601: dwrite d1/d5/dd/f40 [0,4194304] 0 2024-04-22T23:23:49.775 INFO:tasks.workunit.client.1.smithi134.stdout:7/666: chown d4/d8/d3b/ldf 8723 1 2024-04-22T23:23:49.775 INFO:tasks.workunit.client.1.smithi134.stdout:9/464: dwrite d0/d7/f90 [0,4194304] 0 2024-04-22T23:23:49.780 INFO:tasks.workunit.client.1.smithi134.stdout:1/745: dwrite db/d15/d17/d20/d2a/d2d/d2e/d42/f9a [0,4194304] 0 2024-04-22T23:23:49.785 INFO:tasks.workunit.client.1.smithi134.stdout:5/690: dwrite d9/d11/f1a [0,4194304] 0 2024-04-22T23:23:49.785 INFO:tasks.workunit.client.1.smithi134.stdout:7/667: chown d4/d8/d41/d76/dc1/cba 25187302 1 2024-04-22T23:23:49.790 INFO:tasks.workunit.client.1.smithi134.stdout:0/602: link d1/d5/d58/d22/c63 d1/d5/dd/d20/cca 0 2024-04-22T23:23:49.791 INFO:tasks.workunit.client.1.smithi134.stdout:7/668: chown d4/d4a/d85/c9b 12142957 1 2024-04-22T23:23:49.792 INFO:tasks.workunit.client.1.smithi134.stdout:5/691: write d9/d11/fc7 [171826,53089] 0 2024-04-22T23:23:49.795 INFO:tasks.workunit.client.1.smithi134.stdout:9/465: dwrite d0/d7/d12/f6a [0,4194304] 0 2024-04-22T23:23:49.798 INFO:tasks.workunit.client.1.smithi134.stdout:1/746: dwrite db/fb9 [0,4194304] 0 2024-04-22T23:23:49.801 INFO:tasks.workunit.client.1.smithi134.stdout:0/603: mkdir d1/d5/d58/d22/dcb 0 2024-04-22T23:23:49.802 INFO:tasks.workunit.client.1.smithi134.stdout:1/747: readlink db/d15/d17/d20/d2a/d2d/d2e/d42/le6 0 2024-04-22T23:23:49.803 INFO:tasks.workunit.client.1.smithi134.stdout:5/692: creat d9/d11/d73/d95/dae/ff2 x:0 0 0 2024-04-22T23:23:49.803 INFO:tasks.workunit.client.1.smithi134.stdout:7/669: mknod d4/d8/d41/cef 0 2024-04-22T23:23:49.803 INFO:tasks.workunit.client.1.smithi134.stdout:0/604: stat d1/d5/dd/d41/la4 0 2024-04-22T23:23:49.804 INFO:tasks.workunit.client.1.smithi134.stdout:5/693: chown d9/dd/d17/d44/c4e 2525354 1 2024-04-22T23:23:49.805 INFO:tasks.workunit.client.1.smithi134.stdout:8/534: sync 2024-04-22T23:23:49.805 INFO:tasks.workunit.client.1.smithi134.stdout:3/482: sync 2024-04-22T23:23:49.806 INFO:tasks.workunit.client.1.smithi134.stdout:9/466: creat d0/d7/d23/d26/d4d/d5d/d8e/f9c x:0 0 0 2024-04-22T23:23:49.807 INFO:tasks.workunit.client.1.smithi134.stdout:0/605: write d1/d5/d58/f80 [944684,16369] 0 2024-04-22T23:23:49.808 INFO:tasks.workunit.client.1.smithi134.stdout:8/535: chown d7/dc/l60 205104992 1 2024-04-22T23:23:49.809 INFO:tasks.workunit.client.1.smithi134.stdout:5/694: chown d9/dd/d17/d44/cc1 14 1 2024-04-22T23:23:49.809 INFO:tasks.workunit.client.1.smithi134.stdout:9/467: fdatasync d0/d2d/f6b 0 2024-04-22T23:23:49.811 INFO:tasks.workunit.client.1.smithi134.stdout:3/483: mkdir dc/d2a/d62/d86/db3 0 2024-04-22T23:23:49.811 INFO:tasks.workunit.client.1.smithi134.stdout:0/606: truncate d1/d5/d58/d6b/db9/f18 12639772 0 2024-04-22T23:23:49.812 INFO:tasks.workunit.client.1.smithi134.stdout:9/468: write d0/d7/d23/d26/d51/d4c/f71 [800823,117754] 0 2024-04-22T23:23:49.813 INFO:tasks.workunit.client.1.smithi134.stdout:9/469: chown d0/d7/d12/l60 0 1 2024-04-22T23:23:49.815 INFO:tasks.workunit.client.1.smithi134.stdout:1/748: rename db/d15/d17/d20/d2a/d2d/d2e/f93 to db/d15/d17/dc9/ffc 0 2024-04-22T23:23:49.815 INFO:tasks.workunit.client.1.smithi134.stdout:6/674: truncate d0/d19/db/d35/f58 4652144 0 2024-04-22T23:23:49.816 INFO:tasks.workunit.client.1.smithi134.stdout:1/749: read db/d15/d17/d20/d2a/d2d/ff1 [423063,114628] 0 2024-04-22T23:23:49.817 INFO:tasks.workunit.client.1.smithi134.stdout:7/670: creat d4/d8/d41/d76/dc1/d2f/d70/d86/ff0 x:0 0 0 2024-04-22T23:23:49.819 INFO:tasks.workunit.client.1.smithi134.stdout:8/536: truncate d7/dc/d2c/d4f/d6c/d6e/f91 24044 0 2024-04-22T23:23:49.821 INFO:tasks.workunit.client.1.smithi134.stdout:8/537: stat d7/dc/d2c/d37/d47/c55 0 2024-04-22T23:23:49.821 INFO:tasks.workunit.client.1.smithi134.stdout:9/470: creat d0/d7/d23/d26/d51/f9d x:0 0 0 2024-04-22T23:23:49.821 INFO:tasks.workunit.client.1.smithi134.stdout:8/538: truncate f6 9126132 0 2024-04-22T23:23:49.823 INFO:tasks.workunit.client.1.smithi134.stdout:8/539: readlink d7/db/d2d/d39/d58/l6d 0 2024-04-22T23:23:49.826 INFO:tasks.workunit.client.1.smithi134.stdout:0/607: creat d1/d5/d58/d6b/db9/d83/fcc x:0 0 0 2024-04-22T23:23:49.828 INFO:tasks.workunit.client.1.smithi134.stdout:7/671: readlink d4/d8/d41/d5b/lc9 0 2024-04-22T23:23:49.829 INFO:tasks.workunit.client.1.smithi134.stdout:1/750: creat db/d16/dbc/de1/df4/ffd x:0 0 0 2024-04-22T23:23:49.831 INFO:tasks.workunit.client.1.smithi134.stdout:9/471: creat d0/d2d/d3d/f9e x:0 0 0 2024-04-22T23:23:49.833 INFO:tasks.workunit.client.1.smithi134.stdout:8/540: write d7/db/d2d/f3e [551286,73304] 0 2024-04-22T23:23:49.837 INFO:tasks.workunit.client.1.smithi134.stdout:3/484: getdents dc/d2a 0 2024-04-22T23:23:49.837 INFO:tasks.workunit.client.1.smithi134.stdout:8/541: fdatasync d7/dc/d2c/d37/f4d 0 2024-04-22T23:23:49.838 INFO:tasks.workunit.client.1.smithi134.stdout:1/751: symlink db/d15/d17/dc9/lfe 0 2024-04-22T23:23:49.838 INFO:tasks.workunit.client.1.smithi134.stdout:8/542: stat d7/db/d2d/fa7 0 2024-04-22T23:23:49.842 INFO:tasks.workunit.client.1.smithi134.stdout:7/672: dread d4/d8/d41/d76/dc1/d2f/d34/d62/d29/f55 [0,4194304] 0 2024-04-22T23:23:49.850 INFO:tasks.workunit.client.1.smithi134.stdout:3/485: creat dc/d2a/d62/db2/d2f/d36/d6d/fb4 x:0 0 0 2024-04-22T23:23:49.859 INFO:tasks.workunit.client.1.smithi134.stdout:9/472: dread d0/d7/f17 [0,4194304] 0 2024-04-22T23:23:49.878 INFO:tasks.workunit.client.1.smithi134.stdout:0/608: dwrite d1/d5/d58/d6b/db9/f64 [0,4194304] 0 2024-04-22T23:23:49.879 INFO:tasks.workunit.client.1.smithi134.stdout:8/543: getdents d7/db/d2d/d39/d58/d8a 0 2024-04-22T23:23:49.883 INFO:tasks.workunit.client.1.smithi134.stdout:0/609: truncate d1/d5/dd/d41/fc8 239121 0 2024-04-22T23:23:49.884 INFO:tasks.workunit.client.1.smithi134.stdout:0/610: readlink d1/d5/dd/d20/d2c/l78 0 2024-04-22T23:23:49.887 INFO:tasks.workunit.client.1.smithi134.stdout:0/611: chown d1/d5/dd/d41/l4d 814834055 1 2024-04-22T23:23:49.887 INFO:tasks.workunit.client.1.smithi134.stdout:1/752: getdents db/d16 0 2024-04-22T23:23:49.887 INFO:tasks.workunit.client.1.smithi134.stdout:7/673: getdents d4/d8/d41/d76/dc1/d2f/d69/dab/d95 0 2024-04-22T23:23:49.887 INFO:tasks.workunit.client.1.smithi134.stdout:7/674: readlink l2 0 2024-04-22T23:23:49.888 INFO:tasks.workunit.client.1.smithi134.stdout:0/612: write d1/d2/d1d/f5f [5098817,128501] 0 2024-04-22T23:23:49.889 INFO:tasks.workunit.client.1.smithi134.stdout:7/675: write d4/d8/d41/d76/dc1/d2f/da6/dcf/fd4 [925632,53585] 0 2024-04-22T23:23:49.929 INFO:tasks.workunit.client.1.smithi134.stdout:7/676: creat d4/d8/d41/d76/dc1/d2f/d34/d62/d29/ff1 x:0 0 0 2024-04-22T23:23:49.936 INFO:tasks.workunit.client.1.smithi134.stdout:7/677: dwrite d4/d8/d2a/d42/d75/f77 [0,4194304] 0 2024-04-22T23:23:49.936 INFO:tasks.workunit.client.1.smithi134.stdout:7/678: fsync d4/d8/d41/f43 0 2024-04-22T23:23:49.937 INFO:tasks.workunit.client.1.smithi134.stdout:1/753: creat db/fff x:0 0 0 2024-04-22T23:23:49.947 INFO:tasks.workunit.client.1.smithi134.stdout:1/754: dwrite db/d15/d17/d20/f65 [0,4194304] 0 2024-04-22T23:23:49.948 INFO:tasks.workunit.client.1.smithi134.stdout:1/755: truncate db/d16/d87/fd5 931414 0 2024-04-22T23:23:49.949 INFO:tasks.workunit.client.1.smithi134.stdout:9/473: rename d0/fa to d0/d2d/f9f 0 2024-04-22T23:23:49.953 INFO:tasks.workunit.client.1.smithi134.stdout:9/474: truncate d0/d7/f19 1355522 0 2024-04-22T23:23:49.955 INFO:tasks.workunit.client.1.smithi134.stdout:1/756: rename db/d15/d17/d20/d2a/d2d/c3f to db/d15/d17/d20/d2a/d2d/d86/c100 0 2024-04-22T23:23:49.956 INFO:tasks.workunit.client.1.smithi134.stdout:1/757: chown db/d15/d17/d20/d2a/d2d/d2e/f47 55 1 2024-04-22T23:23:49.958 INFO:tasks.workunit.client.1.smithi134.stdout:9/475: rename d0/d7/d23/d47 to d0/d2d/d82/da0 0 2024-04-22T23:23:49.960 INFO:tasks.workunit.client.1.smithi134.stdout:1/758: unlink db/d15/d17/d20/d2a/d2d/la5 0 2024-04-22T23:23:49.960 INFO:tasks.workunit.client.1.smithi134.stdout:9/476: chown d0/d7/d23/d26/c74 299 1 2024-04-22T23:23:49.962 INFO:tasks.workunit.client.1.smithi134.stdout:9/477: symlink d0/d2d/d82/da0/la1 0 2024-04-22T23:23:49.965 INFO:tasks.workunit.client.1.smithi134.stdout:7/679: unlink d4/d8/d3b/ldf 0 2024-04-22T23:23:49.966 INFO:tasks.workunit.client.1.smithi134.stdout:1/759: getdents db/d16/d87 0 2024-04-22T23:23:49.968 INFO:tasks.workunit.client.1.smithi134.stdout:1/760: write db/d15/d17/d20/d2a/d2d/f64 [4576915,34267] 0 2024-04-22T23:23:49.969 INFO:tasks.workunit.client.1.smithi134.stdout:7/680: unlink d4/d8/d41/d76/dc1/d2f/d34/d62/d29/c57 0 2024-04-22T23:23:49.972 INFO:tasks.workunit.client.1.smithi134.stdout:7/681: symlink d4/d8/d2a/d42/d6e/lf2 0 2024-04-22T23:23:49.974 INFO:tasks.workunit.client.1.smithi134.stdout:8/544: write d7/db/d2d/f3c [658129,99976] 0 2024-04-22T23:23:49.976 INFO:tasks.workunit.client.1.smithi134.stdout:7/682: truncate d4/d8/d41/d76/dc1/d2f/d34/d62/f71 817527 0 2024-04-22T23:23:49.978 INFO:tasks.workunit.client.1.smithi134.stdout:8/545: creat d7/db/d42/d65/d46/d9a/faa x:0 0 0 2024-04-22T23:23:49.981 INFO:tasks.workunit.client.1.smithi134.stdout:7/683: link d4/d8/d3b/f68 d4/d8/d41/d5b/ff3 0 2024-04-22T23:23:49.988 INFO:tasks.workunit.client.1.smithi134.stdout:9/478: dread d0/d37/f4f [0,4194304] 0 2024-04-22T23:23:49.992 INFO:tasks.workunit.client.1.smithi134.stdout:7/684: mkdir d4/d8/d41/d76/dc1/d2f/d69/dab/d95/dec/df4 0 2024-04-22T23:23:49.993 INFO:tasks.workunit.client.1.smithi134.stdout:9/479: mknod d0/d7/d12/ca2 0 2024-04-22T23:23:49.994 INFO:tasks.workunit.client.1.smithi134.stdout:7/685: read d4/d8/d41/d76/dc1/d2f/d69/dab/f80 [151901,73432] 0 2024-04-22T23:23:49.994 INFO:tasks.workunit.client.1.smithi134.stdout:8/546: read d7/dc/d2c/d37/d47/f5c [349854,129318] 0 2024-04-22T23:23:49.996 INFO:tasks.workunit.client.1.smithi134.stdout:9/480: unlink d0/d7/f2c 0 2024-04-22T23:23:49.997 INFO:tasks.workunit.client.1.smithi134.stdout:9/481: read d0/d7/f90 [1014665,66119] 0 2024-04-22T23:23:49.998 INFO:tasks.workunit.client.1.smithi134.stdout:8/547: fdatasync d7/dc/d2c/d4f/fa0 0 2024-04-22T23:23:50.005 INFO:tasks.workunit.client.1.smithi134.stdout:9/482: readlink d0/d37/l3a 0 2024-04-22T23:23:50.011 INFO:tasks.workunit.client.1.smithi134.stdout:8/548: creat d7/dc/d19/da5/fab x:0 0 0 2024-04-22T23:23:50.012 INFO:tasks.workunit.client.1.smithi134.stdout:9/483: truncate d0/d7/f6f 68271 0 2024-04-22T23:23:50.012 INFO:tasks.workunit.client.1.smithi134.stdout:8/549: dread - d7/dc/d2c/d4f/f9f zero size 2024-04-22T23:23:50.029 INFO:tasks.workunit.client.1.smithi134.stdout:4/749: sync 2024-04-22T23:23:50.029 INFO:tasks.workunit.client.1.smithi134.stdout:2/546: sync 2024-04-22T23:23:50.032 INFO:tasks.workunit.client.1.smithi134.stdout:7/686: write d4/d8/d41/d76/dc1/d2f/d34/f67 [5050354,21308] 0 2024-04-22T23:23:50.033 INFO:tasks.workunit.client.1.smithi134.stdout:9/484: symlink d0/d7/d23/d26/d4d/d5d/d8e/la3 0 2024-04-22T23:23:50.035 INFO:tasks.workunit.client.1.smithi134.stdout:4/750: symlink d4/d35/db4/lec 0 2024-04-22T23:23:50.037 INFO:tasks.workunit.client.1.smithi134.stdout:9/485: rename d0/d7/d23/d26 to d0/d7/d23/d26/d51/d4c/da4 22 2024-04-22T23:23:50.037 INFO:tasks.workunit.client.1.smithi134.stdout:2/547: fdatasync d2/d15/d24/d1d/f28 0 2024-04-22T23:23:50.038 INFO:tasks.workunit.client.1.smithi134.stdout:1/761: dwrite db/d16/d56/f57 [0,4194304] 0 2024-04-22T23:23:50.039 INFO:tasks.workunit.client.1.smithi134.stdout:7/687: creat d4/d8/d2a/d42/d6e/dd6/ff5 x:0 0 0 2024-04-22T23:23:50.042 INFO:tasks.workunit.client.1.smithi134.stdout:9/486: truncate d0/d7/d23/d26/d51/d4c/f71 1519068 0 2024-04-22T23:23:50.047 INFO:tasks.workunit.client.1.smithi134.stdout:4/751: dread d4/d13/f65 [0,4194304] 0 2024-04-22T23:23:50.049 INFO:tasks.workunit.client.1.smithi134.stdout:9/487: rename d0/d2d/d4e/f78 to d0/d7/d27/fa5 0 2024-04-22T23:23:50.050 INFO:tasks.workunit.client.1.smithi134.stdout:8/550: dwrite d7/db/d2d/f5a [0,4194304] 0 2024-04-22T23:23:50.053 INFO:tasks.workunit.client.1.smithi134.stdout:1/762: creat db/d15/d17/d20/f101 x:0 0 0 2024-04-22T23:23:50.055 INFO:tasks.workunit.client.1.smithi134.stdout:2/548: link d2/d15/d24/l56 d2/d42/d54/d33/d91/lb4 0 2024-04-22T23:23:50.056 INFO:tasks.workunit.client.1.smithi134.stdout:4/752: stat d4/d7/d32/d3a/d93/db5/cdc 0 2024-04-22T23:23:50.058 INFO:tasks.workunit.client.1.smithi134.stdout:7/688: link d4/d8/d2a/d42/d6e/dd6/ff5 d4/d8/d41/ff6 0 2024-04-22T23:23:50.059 INFO:tasks.workunit.client.1.smithi134.stdout:1/763: symlink db/d15/d17/dc9/l102 0 2024-04-22T23:23:50.059 INFO:tasks.workunit.client.1.smithi134.stdout:9/488: dwrite d0/d2d/d82/da0/f7f [0,4194304] 0 2024-04-22T23:23:50.060 INFO:tasks.workunit.client.1.smithi134.stdout:1/764: fdatasync db/d16/d87/faf 0 2024-04-22T23:23:50.062 INFO:tasks.workunit.client.1.smithi134.stdout:2/549: rmdir d2/d42/d54 39 2024-04-22T23:23:50.069 INFO:tasks.workunit.client.1.smithi134.stdout:1/765: creat db/d15/d17/d20/d2a/d2d/d86/dad/f103 x:0 0 0 2024-04-22T23:23:50.074 INFO:tasks.workunit.client.1.smithi134.stdout:9/489: rename d0/d7/d23/d26/d51/l80 to d0/d7/la6 0 2024-04-22T23:23:50.074 INFO:tasks.workunit.client.1.smithi134.stdout:2/550: dwrite d2/dd/d77/fa3 [0,4194304] 0 2024-04-22T23:23:50.074 INFO:tasks.workunit.client.1.smithi134.stdout:4/753: dread d4/dd/f41 [0,4194304] 0 2024-04-22T23:23:50.077 INFO:tasks.workunit.client.1.smithi134.stdout:2/551: dread - d2/dd/d68/f88 zero size 2024-04-22T23:23:50.077 INFO:tasks.workunit.client.1.smithi134.stdout:8/551: dread f6 [4194304,4194304] 0 2024-04-22T23:23:50.079 INFO:tasks.workunit.client.1.smithi134.stdout:9/490: read d0/d7/f9 [239858,119062] 0 2024-04-22T23:23:50.079 INFO:tasks.workunit.client.1.smithi134.stdout:1/766: truncate db/d15/d17/d20/d2a/d2d/f43 2879285 0 2024-04-22T23:23:50.081 INFO:tasks.workunit.client.1.smithi134.stdout:4/754: write d4/d29/dcc/f1c [4183140,68879] 0 2024-04-22T23:23:50.085 INFO:tasks.workunit.client.1.smithi134.stdout:2/552: fsync d2/d15/d24/d3b/f4f 0 2024-04-22T23:23:50.089 INFO:tasks.workunit.client.1.smithi134.stdout:4/755: symlink d4/d29/dcc/d82/led 0 2024-04-22T23:23:50.094 INFO:tasks.workunit.client.1.smithi134.stdout:9/491: dread d0/d7/f5e [0,4194304] 0 2024-04-22T23:23:50.094 INFO:tasks.workunit.client.1.smithi134.stdout:7/689: dwrite d4/d4a/d85/fb2 [0,4194304] 0 2024-04-22T23:23:50.095 INFO:tasks.workunit.client.1.smithi134.stdout:8/552: dwrite d7/dc/d2c/d4f/f6b [0,4194304] 0 2024-04-22T23:23:50.101 INFO:tasks.workunit.client.1.smithi134.stdout:2/553: link l1 d2/d15/d24/d1d/d3d/d7b/lb5 0 2024-04-22T23:23:50.101 INFO:tasks.workunit.client.1.smithi134.stdout:4/756: read d4/d29/dcc/f67 [2317737,112081] 0 2024-04-22T23:23:50.110 INFO:tasks.workunit.client.1.smithi134.stdout:1/767: dread db/d16/dbc/fec [4194304,4194304] 0 2024-04-22T23:23:50.111 INFO:tasks.workunit.client.1.smithi134.stdout:2/554: creat d2/dd/df/fb6 x:0 0 0 2024-04-22T23:23:50.111 INFO:tasks.workunit.client.1.smithi134.stdout:8/553: link d7/db/d2d/d39/d58/c90 d7/db/d2d/d39/d58/cac 0 2024-04-22T23:23:50.111 INFO:tasks.workunit.client.1.smithi134.stdout:4/757: dwrite d4/f21 [4194304,4194304] 0 2024-04-22T23:23:50.114 INFO:tasks.workunit.client.1.smithi134.stdout:4/758: fdatasync d4/d35/d5e/f4a 0 2024-04-22T23:23:50.114 INFO:tasks.workunit.client.1.smithi134.stdout:2/555: rename d2 to d2/dd/df/d9c/daa/db0/db7 22 2024-04-22T23:23:50.115 INFO:tasks.workunit.client.1.smithi134.stdout:4/759: stat d4/d35/d5e/f1f 0 2024-04-22T23:23:50.115 INFO:tasks.workunit.client.1.smithi134.stdout:1/768: read db/d15/d17/fa2 [3829072,2702] 0 2024-04-22T23:23:50.115 INFO:tasks.workunit.client.1.smithi134.stdout:2/556: chown d2/dd/d4a/d73/cb1 13473760 1 2024-04-22T23:23:50.116 INFO:tasks.workunit.client.1.smithi134.stdout:8/554: mknod d7/dc/d2c/d37/cad 0 2024-04-22T23:23:50.117 INFO:tasks.workunit.client.1.smithi134.stdout:4/760: write d4/d35/d8e/fd4 [285443,57056] 0 2024-04-22T23:23:50.121 INFO:tasks.workunit.client.1.smithi134.stdout:4/761: write d4/d35/d8e/f75 [166661,10876] 0 2024-04-22T23:23:50.122 INFO:tasks.workunit.client.1.smithi134.stdout:4/762: chown d4/d29/dcc/d64/c6c 1 1 2024-04-22T23:23:50.123 INFO:tasks.workunit.client.1.smithi134.stdout:1/769: getdents db/d15/d17/d20/d2a 0 2024-04-22T23:23:50.123 INFO:tasks.workunit.client.1.smithi134.stdout:4/763: stat d4/d7/d32/dc5 0 2024-04-22T23:23:50.123 INFO:tasks.workunit.client.1.smithi134.stdout:2/557: symlink d2/d42/d54/d33/lb8 0 2024-04-22T23:23:50.124 INFO:tasks.workunit.client.1.smithi134.stdout:1/770: dread - db/d15/fc6 zero size 2024-04-22T23:23:50.125 INFO:tasks.workunit.client.1.smithi134.stdout:9/492: write d0/d2d/d82/f83 [709826,71571] 0 2024-04-22T23:23:50.126 INFO:tasks.workunit.client.1.smithi134.stdout:4/764: write d4/d35/d8e/fd4 [532959,9039] 0 2024-04-22T23:23:50.127 INFO:tasks.workunit.client.1.smithi134.stdout:9/493: read - d0/d2d/f6b zero size 2024-04-22T23:23:50.130 INFO:tasks.workunit.client.1.smithi134.stdout:9/494: stat d0/d7/f6f 0 2024-04-22T23:23:50.130 INFO:tasks.workunit.client.1.smithi134.stdout:2/558: unlink d2/d15/d24/d1d/fa6 0 2024-04-22T23:23:50.131 INFO:tasks.workunit.client.1.smithi134.stdout:7/690: dread d4/d8/d41/f43 [0,4194304] 0 2024-04-22T23:23:50.133 INFO:tasks.workunit.client.1.smithi134.stdout:1/771: mkdir db/d16/d104 0 2024-04-22T23:23:50.133 INFO:tasks.workunit.client.1.smithi134.stdout:9/495: chown d0/d7/d12/d22/l84 8358 1 2024-04-22T23:23:50.136 INFO:tasks.workunit.client.1.smithi134.stdout:8/555: truncate d7/dc/f12 2227519 0 2024-04-22T23:23:50.136 INFO:tasks.workunit.client.1.smithi134.stdout:4/765: dwrite d4/d29/dcc/d82/f8b [0,4194304] 0 2024-04-22T23:23:50.141 INFO:tasks.workunit.client.1.smithi134.stdout:8/556: mknod d7/db/d42/cae 0 2024-04-22T23:23:50.142 INFO:tasks.workunit.client.1.smithi134.stdout:8/557: stat d7/dc/d2c/d77 0 2024-04-22T23:23:50.142 INFO:tasks.workunit.client.1.smithi134.stdout:8/558: readlink d7/dc/l60 0 2024-04-22T23:23:50.146 INFO:tasks.workunit.client.1.smithi134.stdout:1/772: rename db/d15/d17/d20/d2a/d2d/ca9 to db/d16/d56/c105 0 2024-04-22T23:23:50.147 INFO:tasks.workunit.client.1.smithi134.stdout:1/773: write db/d15/d17/d20/f65 [1254285,120863] 0 2024-04-22T23:23:50.147 INFO:tasks.workunit.client.1.smithi134.stdout:8/559: mkdir d7/dc/d2c/d37/d47/daf 0 2024-04-22T23:23:50.149 INFO:tasks.workunit.client.1.smithi134.stdout:4/766: link d4/d7/d32/l45 d4/d29/dcc/d82/dd5/lee 0 2024-04-22T23:23:50.150 INFO:tasks.workunit.client.1.smithi134.stdout:4/767: dread - d4/d7/d32/fc3 zero size 2024-04-22T23:23:50.151 INFO:tasks.workunit.client.1.smithi134.stdout:7/691: write d4/d8/d41/d76/dc1/f23 [3534052,41325] 0 2024-04-22T23:23:50.151 INFO:tasks.workunit.client.1.smithi134.stdout:8/560: read d7/dc/d19/f21 [1416497,105130] 0 2024-04-22T23:23:50.152 INFO:tasks.workunit.client.1.smithi134.stdout:1/774: fdatasync db/d16/d56/f60 0 2024-04-22T23:23:50.155 INFO:tasks.workunit.client.1.smithi134.stdout:4/768: creat d4/d29/dcc/d64/fef x:0 0 0 2024-04-22T23:23:50.159 INFO:tasks.workunit.client.1.smithi134.stdout:4/769: read d4/d29/f63 [323828,93006] 0 2024-04-22T23:23:50.162 INFO:tasks.workunit.client.1.smithi134.stdout:9/496: dread d0/d7/d27/f33 [0,4194304] 0 2024-04-22T23:23:50.162 INFO:tasks.workunit.client.1.smithi134.stdout:1/775: symlink db/d15/d17/d20/d2a/d2d/d2e/l106 0 2024-04-22T23:23:50.163 INFO:tasks.workunit.client.1.smithi134.stdout:8/561: dread d7/db/f3b [0,4194304] 0 2024-04-22T23:23:50.164 INFO:tasks.workunit.client.1.smithi134.stdout:1/776: write db/d15/d17/d20/d2a/d2d/f48 [581718,77979] 0 2024-04-22T23:23:50.165 INFO:tasks.workunit.client.1.smithi134.stdout:4/770: symlink d4/d7/d32/d3a/dbc/lf0 0 2024-04-22T23:23:50.169 INFO:tasks.workunit.client.1.smithi134.stdout:8/562: rmdir d7/dc 39 2024-04-22T23:23:50.173 INFO:tasks.workunit.client.1.smithi134.stdout:7/692: write d4/d8/d3b/f50 [506019,68326] 0 2024-04-22T23:23:50.176 INFO:tasks.workunit.client.1.smithi134.stdout:1/777: mkdir db/d107 0 2024-04-22T23:23:50.178 INFO:tasks.workunit.client.1.smithi134.stdout:4/771: symlink d4/dd/da8/dc2/lf1 0 2024-04-22T23:23:50.178 INFO:tasks.workunit.client.1.smithi134.stdout:9/497: dread d0/d2d/d4e/f32 [0,4194304] 0 2024-04-22T23:23:50.179 INFO:tasks.workunit.client.1.smithi134.stdout:8/563: chown d7/dc 75335021 1 2024-04-22T23:23:50.179 INFO:tasks.workunit.client.1.smithi134.stdout:4/772: write d4/d29/dcc/f92 [5067038,25562] 0 2024-04-22T23:23:50.180 INFO:tasks.workunit.client.1.smithi134.stdout:9/498: dread - d0/d2d/d82/f9a zero size 2024-04-22T23:23:50.181 INFO:tasks.workunit.client.1.smithi134.stdout:4/773: stat d4/d29/dcc/d64/f73 0 2024-04-22T23:23:50.186 INFO:tasks.workunit.client.1.smithi134.stdout:1/778: mkdir db/d15/d17/d20/d2a/d2d/d108 0 2024-04-22T23:23:50.186 INFO:tasks.workunit.client.1.smithi134.stdout:1/779: chown db 43113 1 2024-04-22T23:23:50.186 INFO:tasks.workunit.client.1.smithi134.stdout:4/774: symlink d4/d35/d8e/d5d/lf2 0 2024-04-22T23:23:50.186 INFO:tasks.workunit.client.1.smithi134.stdout:7/693: link d4/d8/d41/d5b/ff3 d4/d8/d41/d76/dc1/d2f/d69/dab/de1/ff7 0 2024-04-22T23:23:50.187 INFO:tasks.workunit.client.1.smithi134.stdout:1/780: stat db/d15/d17/d20/d2a/d2d/d2e/d42/cd4 0 2024-04-22T23:23:50.187 INFO:tasks.workunit.client.1.smithi134.stdout:9/499: mknod d0/d7/d23/ca7 0 2024-04-22T23:23:50.188 INFO:tasks.workunit.client.1.smithi134.stdout:4/775: rmdir d4/d35/d5e/d25 39 2024-04-22T23:23:50.189 INFO:tasks.workunit.client.1.smithi134.stdout:7/694: mkdir d4/d8/d41/d76/dc1/d2f/d69/df8 0 2024-04-22T23:23:50.189 INFO:tasks.workunit.client.1.smithi134.stdout:4/776: write d4/d7/d32/d84/fc0 [930155,113464] 0 2024-04-22T23:23:50.248 INFO:tasks.workunit.client.1.smithi134.stdout:9/500: mknod d0/d7/d23/ca8 0 2024-04-22T23:23:50.248 INFO:tasks.workunit.client.1.smithi134.stdout:7/695: dread - d4/d8/d41/d76/dc1/d2f/d70/d86/ff0 zero size 2024-04-22T23:23:50.249 INFO:tasks.workunit.client.1.smithi134.stdout:1/781: rename db/d16/fb8 to db/f109 0 2024-04-22T23:23:50.249 INFO:tasks.workunit.client.1.smithi134.stdout:4/777: symlink d4/dd/da8/dc2/lf3 0 2024-04-22T23:23:50.255 INFO:tasks.workunit.client.1.smithi134.stdout:7/696: dwrite d4/d8/d3b/f50 [0,4194304] 0 2024-04-22T23:23:50.344 INFO:tasks.workunit.client.1.smithi134.stdout:9/501: symlink d0/d7/d23/d26/la9 0 2024-04-22T23:23:50.348 INFO:tasks.workunit.client.1.smithi134.stdout:9/502: fsync d0/d2d/f9f 0 2024-04-22T23:23:50.360 INFO:tasks.workunit.client.1.smithi134.stdout:4/778: dwrite d4/d29/dcc/f81 [0,4194304] 0 2024-04-22T23:23:50.360 INFO:tasks.workunit.client.1.smithi134.stdout:1/782: dwrite db/d16/d56/f60 [0,4194304] 0 2024-04-22T23:23:50.360 INFO:tasks.workunit.client.1.smithi134.stdout:9/503: dwrite d0/f54 [0,4194304] 0 2024-04-22T23:23:50.366 INFO:tasks.workunit.client.1.smithi134.stdout:4/779: chown d4/l40 1562 1 2024-04-22T23:23:50.369 INFO:tasks.workunit.client.1.smithi134.stdout:4/780: write d4/f21 [276560,76957] 0 2024-04-22T23:23:50.371 INFO:tasks.workunit.client.1.smithi134.stdout:9/504: dwrite d0/d2d/d4e/f62 [0,4194304] 0 2024-04-22T23:23:50.374 INFO:tasks.workunit.client.1.smithi134.stdout:1/783: rename db/d15/d17/d20/f101 to db/d15/d17/d20/d2a/d2d/f10a 0 2024-04-22T23:23:50.377 INFO:tasks.workunit.client.1.smithi134.stdout:4/781: dread d4/d29/dcc/f81 [0,4194304] 0 2024-04-22T23:23:50.389 INFO:tasks.workunit.client.1.smithi134.stdout:5/695: sync 2024-04-22T23:23:50.390 INFO:tasks.workunit.client.1.smithi134.stdout:4/782: rmdir d4 39 2024-04-22T23:23:50.392 INFO:tasks.workunit.client.1.smithi134.stdout:1/784: symlink db/d15/d17/l10b 0 2024-04-22T23:23:50.393 INFO:tasks.workunit.client.1.smithi134.stdout:1/785: fsync db/d15/d17/d20/d2a/d2d/d2e/f90 0 2024-04-22T23:23:50.396 INFO:tasks.workunit.client.1.smithi134.stdout:5/696: link d9/d1e/fb6 d9/d11/d34/d4d/ff3 0 2024-04-22T23:23:50.397 INFO:tasks.workunit.client.1.smithi134.stdout:5/697: truncate d9/d11/d34/d85/fd3 4925429 0 2024-04-22T23:23:50.398 INFO:tasks.workunit.client.1.smithi134.stdout:5/698: chown d9/d11/d34/ca7 2246370 1 2024-04-22T23:23:50.399 INFO:tasks.workunit.client.1.smithi134.stdout:1/786: rename db/d15/d17/d20/d2a/d2d/d2e/d42/cd4 to db/db5/c10c 0 2024-04-22T23:23:50.401 INFO:tasks.workunit.client.1.smithi134.stdout:4/783: rename d4/d29/dcc/d66 to d4/d7/d32/d84/df4 0 2024-04-22T23:23:50.404 INFO:tasks.workunit.client.1.smithi134.stdout:9/505: dread d0/d7/d27/fa5 [0,4194304] 0 2024-04-22T23:23:50.406 INFO:tasks.workunit.client.1.smithi134.stdout:5/699: getdents d9/d11/d73/d95 0 2024-04-22T23:23:50.407 INFO:tasks.workunit.client.1.smithi134.stdout:4/784: truncate d4/f42 442746 0 2024-04-22T23:23:50.408 INFO:tasks.workunit.client.1.smithi134.stdout:1/787: link c1 db/d107/c10d 0 2024-04-22T23:23:50.409 INFO:tasks.workunit.client.1.smithi134.stdout:1/788: chown db/d15/d17/d20/l9b 441310 1 2024-04-22T23:23:50.410 INFO:tasks.workunit.client.1.smithi134.stdout:1/789: truncate db/d15/d17/d20/d2a/d2d/f10a 294059 0 2024-04-22T23:23:50.411 INFO:tasks.workunit.client.1.smithi134.stdout:9/506: rename d0/d2d/d4e/l24 to d0/d7/d23/d26/d4d/d5d/laa 0 2024-04-22T23:23:50.412 INFO:tasks.workunit.client.1.smithi134.stdout:9/507: dread - d0/d7/d23/d26/d4d/d5d/d8e/f9c zero size 2024-04-22T23:23:50.412 INFO:tasks.workunit.client.1.smithi134.stdout:5/700: mknod d9/d11/d34/d4d/dd4/cf4 0 2024-04-22T23:23:50.412 INFO:tasks.workunit.client.1.smithi134.stdout:9/508: write d0/d7/f17 [3339767,43855] 0 2024-04-22T23:23:50.414 INFO:tasks.workunit.client.1.smithi134.stdout:9/509: chown d0/d7/d12/c4a 55622 1 2024-04-22T23:23:50.415 INFO:tasks.workunit.client.1.smithi134.stdout:9/510: chown d0/d7/d23/d26/la9 14740 1 2024-04-22T23:23:50.416 INFO:tasks.workunit.client.1.smithi134.stdout:5/701: creat d9/d11/d73/d95/d20/d60/ff5 x:0 0 0 2024-04-22T23:23:50.416 INFO:tasks.workunit.client.1.smithi134.stdout:5/702: chown d9/dd 12908804 1 2024-04-22T23:23:50.417 INFO:tasks.workunit.client.1.smithi134.stdout:9/511: read d0/d7/d12/f1f [184715,63059] 0 2024-04-22T23:23:50.417 INFO:tasks.workunit.client.1.smithi134.stdout:4/785: creat d4/d7a/dcf/ff5 x:0 0 0 2024-04-22T23:23:50.418 INFO:tasks.workunit.client.1.smithi134.stdout:5/703: read d9/d11/d73/d95/d20/d60/f7d [3373756,44139] 0 2024-04-22T23:23:50.419 INFO:tasks.workunit.client.1.smithi134.stdout:1/790: getdents db/d16 0 2024-04-22T23:23:50.420 INFO:tasks.workunit.client.1.smithi134.stdout:9/512: write d0/d7/d23/d1b/f64 [3167092,61243] 0 2024-04-22T23:23:50.422 INFO:tasks.workunit.client.1.smithi134.stdout:5/704: mknod d9/dd/d17/d2a/d49/dc2/cf6 0 2024-04-22T23:23:50.423 INFO:tasks.workunit.client.1.smithi134.stdout:9/513: truncate d0/d2d/d3d/f9e 255388 0 2024-04-22T23:23:50.424 INFO:tasks.workunit.client.1.smithi134.stdout:5/705: dread - d9/dd/d17/d2a/d49/dc2/fcb zero size 2024-04-22T23:23:50.426 INFO:tasks.workunit.client.1.smithi134.stdout:4/786: truncate d4/d35/d5e/f4a 118729 0 2024-04-22T23:23:50.426 INFO:tasks.workunit.client.1.smithi134.stdout:9/514: fsync d0/d2d/d82/da0/f8d 0 2024-04-22T23:23:50.428 INFO:tasks.workunit.client.1.smithi134.stdout:4/787: stat d4/fa5 0 2024-04-22T23:23:50.431 INFO:tasks.workunit.client.1.smithi134.stdout:9/515: fsync d0/d7/fd 0 2024-04-22T23:23:50.434 INFO:tasks.workunit.client.1.smithi134.stdout:4/788: truncate d4/d35/d5e/f97 392332 0 2024-04-22T23:23:50.435 INFO:tasks.workunit.client.1.smithi134.stdout:5/706: dwrite d9/d1e/f5a [4194304,4194304] 0 2024-04-22T23:23:50.439 INFO:tasks.workunit.client.1.smithi134.stdout:4/789: read d4/d29/dcc/f92 [4698630,12345] 0 2024-04-22T23:23:50.440 INFO:tasks.workunit.client.1.smithi134.stdout:9/516: truncate d0/d7/d23/d26/f58 596626 0 2024-04-22T23:23:50.442 INFO:tasks.workunit.client.1.smithi134.stdout:1/791: dread db/d15/d17/d20/d2a/f36 [0,4194304] 0 2024-04-22T23:23:50.443 INFO:tasks.workunit.client.1.smithi134.stdout:4/790: mkdir d4/d13/d68/df6 0 2024-04-22T23:23:50.448 INFO:tasks.workunit.client.1.smithi134.stdout:9/517: creat d0/d7/fab x:0 0 0 2024-04-22T23:23:50.448 INFO:tasks.workunit.client.1.smithi134.stdout:5/707: link d9/dd/c99 d9/dd/d17/cf7 0 2024-04-22T23:23:50.449 INFO:tasks.workunit.client.1.smithi134.stdout:9/518: write d0/d2d/f44 [3921234,27298] 0 2024-04-22T23:23:50.450 INFO:tasks.workunit.client.1.smithi134.stdout:5/708: dread d9/dd/d17/f66 [0,4194304] 0 2024-04-22T23:23:50.451 INFO:tasks.workunit.client.1.smithi134.stdout:1/792: getdents db/db0/dc1/dd9 0 2024-04-22T23:23:50.455 INFO:tasks.workunit.client.1.smithi134.stdout:9/519: rename d0/d7/f90 to d0/d2d/d94/fac 0 2024-04-22T23:23:50.459 INFO:tasks.workunit.client.1.smithi134.stdout:9/520: chown d0/d7/d12/l60 332937 1 2024-04-22T23:23:50.461 INFO:tasks.workunit.client.1.smithi134.stdout:4/791: truncate d4/d29/dcc/d64/f73 1122629 0 2024-04-22T23:23:50.463 INFO:tasks.workunit.client.1.smithi134.stdout:1/793: getdents db/d107 0 2024-04-22T23:23:50.468 INFO:tasks.workunit.client.1.smithi134.stdout:5/709: truncate d9/dd/d17/d2a/f98 1701951 0 2024-04-22T23:23:50.471 INFO:tasks.workunit.client.1.smithi134.stdout:9/521: rename d0/d2d/d82/f9a to d0/d7/fad 0 2024-04-22T23:23:50.471 INFO:tasks.workunit.client.1.smithi134.stdout:1/794: dwrite db/d15/d17/d20/d2a/d2d/d86/fa4 [0,4194304] 0 2024-04-22T23:23:50.471 INFO:tasks.workunit.client.1.smithi134.stdout:5/710: read d9/d1e/d90/fa1 [1034063,61884] 0 2024-04-22T23:23:50.478 INFO:tasks.workunit.client.1.smithi134.stdout:1/795: rmdir db 39 2024-04-22T23:23:50.480 INFO:tasks.workunit.client.1.smithi134.stdout:4/792: dwrite d4/d35/d5e/f4a [0,4194304] 0 2024-04-22T23:23:50.480 INFO:tasks.workunit.client.1.smithi134.stdout:9/522: mkdir d0/d7/d23/d26/d4d/d5d/d8e/dae 0 2024-04-22T23:23:50.483 INFO:tasks.workunit.client.1.smithi134.stdout:9/523: fdatasync d0/d7/fb 0 2024-04-22T23:23:50.483 INFO:tasks.workunit.client.1.smithi134.stdout:9/524: write d0/d2d/d3d/f9e [474008,56185] 0 2024-04-22T23:23:50.485 INFO:tasks.workunit.client.1.smithi134.stdout:5/711: unlink d9/d11/f5d 0 2024-04-22T23:23:50.491 INFO:tasks.workunit.client.1.smithi134.stdout:9/525: fdatasync d0/f5c 0 2024-04-22T23:23:50.491 INFO:tasks.workunit.client.1.smithi134.stdout:4/793: read d4/d7/d32/d3a/f43 [4188683,62637] 0 2024-04-22T23:23:50.493 INFO:tasks.workunit.client.1.smithi134.stdout:5/712: dwrite d9/f2b [0,4194304] 0 2024-04-22T23:23:50.495 INFO:tasks.workunit.client.1.smithi134.stdout:5/713: write d9/d11/d73/d95/dae/ff2 [691437,76807] 0 2024-04-22T23:23:50.496 INFO:tasks.workunit.client.1.smithi134.stdout:9/526: read d0/d7/fd [1371111,57926] 0 2024-04-22T23:23:50.497 INFO:tasks.workunit.client.1.smithi134.stdout:1/796: mknod db/d15/d17/c10e 0 2024-04-22T23:23:50.499 INFO:tasks.workunit.client.1.smithi134.stdout:4/794: truncate d4/fa5 429939 0 2024-04-22T23:23:50.500 INFO:tasks.workunit.client.1.smithi134.stdout:1/797: truncate db/db5/ff3 699609 0 2024-04-22T23:23:50.502 INFO:tasks.workunit.client.1.smithi134.stdout:9/527: creat d0/d7/d23/d26/d51/d88/faf x:0 0 0 2024-04-22T23:23:50.505 INFO:tasks.workunit.client.1.smithi134.stdout:5/714: dwrite d9/d11/d34/f40 [0,4194304] 0 2024-04-22T23:23:50.508 INFO:tasks.workunit.client.1.smithi134.stdout:5/715: truncate d9/d11/d34/d4d/f9f 1036305 0 2024-04-22T23:23:50.508 INFO:tasks.workunit.client.1.smithi134.stdout:9/528: creat d0/d7/d23/d26/fb0 x:0 0 0 2024-04-22T23:23:50.511 INFO:tasks.workunit.client.1.smithi134.stdout:9/529: unlink d0/d2d/d4e/f32 0 2024-04-22T23:23:50.515 INFO:tasks.workunit.client.1.smithi134.stdout:9/530: mkdir d0/d7/d23/d26/d4d/d5d/d8e/dae/db1 0 2024-04-22T23:23:50.518 INFO:tasks.workunit.client.1.smithi134.stdout:9/531: mkdir d0/d2d/db2 0 2024-04-22T23:23:50.523 INFO:tasks.workunit.client.1.smithi134.stdout:9/532: read d0/d7/d23/f2a [1377835,71697] 0 2024-04-22T23:23:50.523 INFO:tasks.workunit.client.1.smithi134.stdout:4/795: dwrite d4/d35/d5e/f1f [0,4194304] 0 2024-04-22T23:23:50.526 INFO:tasks.workunit.client.1.smithi134.stdout:4/796: readlink d4/l47 0 2024-04-22T23:23:50.526 INFO:tasks.workunit.client.1.smithi134.stdout:5/716: dwrite d9/dd/d17/d44/fd0 [0,4194304] 0 2024-04-22T23:23:50.528 INFO:tasks.workunit.client.1.smithi134.stdout:9/533: creat d0/d7/d23/d1b/d5a/fb3 x:0 0 0 2024-04-22T23:23:50.530 INFO:tasks.workunit.client.1.smithi134.stdout:5/717: creat d9/dd/d17/d2a/ff8 x:0 0 0 2024-04-22T23:23:50.532 INFO:tasks.workunit.client.1.smithi134.stdout:9/534: readlink d0/d7/d23/d26/d4d/d5d/l85 0 2024-04-22T23:23:50.571 INFO:tasks.workunit.client.1.smithi134.stdout:9/535: chown d0/d7/d12/f6a 165 1 2024-04-22T23:23:50.571 INFO:tasks.workunit.client.1.smithi134.stdout:6/675: sync 2024-04-22T23:23:50.571 INFO:tasks.workunit.client.1.smithi134.stdout:5/718: mknod d9/d11/d73/d95/dae/cf9 0 2024-04-22T23:23:50.573 INFO:tasks.workunit.client.1.smithi134.stdout:4/797: dwrite d4/d7/d32/d3a/d93/db5/fe4 [0,4194304] 0 2024-04-22T23:23:50.576 INFO:tasks.workunit.client.1.smithi134.stdout:5/719: truncate d9/d11/d34/d4d/f9f 1506774 0 2024-04-22T23:23:50.576 INFO:tasks.workunit.client.1.smithi134.stdout:6/676: mknod d0/d19/db/d35/ce1 0 2024-04-22T23:23:50.580 INFO:tasks.workunit.client.1.smithi134.stdout:9/536: dwrite d0/d7/d12/f42 [0,4194304] 0 2024-04-22T23:23:50.581 INFO:tasks.workunit.client.1.smithi134.stdout:4/798: write d4/d29/dcc/f83 [4388633,94624] 0 2024-04-22T23:23:50.583 INFO:tasks.workunit.client.1.smithi134.stdout:5/720: unlink d9/d11/d73/d95/dae/ld9 0 2024-04-22T23:23:50.586 INFO:tasks.workunit.client.1.smithi134.stdout:5/721: truncate d9/dd/d17/d2a/d49/dc2/fcb 981325 0 2024-04-22T23:23:50.588 INFO:tasks.workunit.client.1.smithi134.stdout:4/799: creat d4/d35/d8e/d5d/ff7 x:0 0 0 2024-04-22T23:23:50.589 INFO:tasks.workunit.client.1.smithi134.stdout:9/537: mkdir d0/d7/d23/d26/d51/d88/db4 0 2024-04-22T23:23:50.589 INFO:tasks.workunit.client.1.smithi134.stdout:4/800: read - d4/d7/d32/fbf zero size 2024-04-22T23:23:50.589 INFO:tasks.workunit.client.1.smithi134.stdout:5/722: rmdir d9/d11/d73/d95/dae/de0 39 2024-04-22T23:23:50.590 INFO:tasks.workunit.client.1.smithi134.stdout:9/538: fsync d0/d2d/d3d/f9e 0 2024-04-22T23:23:50.593 INFO:tasks.workunit.client.1.smithi134.stdout:5/723: creat d9/dd/d88/db7/ffa x:0 0 0 2024-04-22T23:23:50.594 INFO:tasks.workunit.client.1.smithi134.stdout:6/677: getdents d0/d19/de/d12/d5e 0 2024-04-22T23:23:50.595 INFO:tasks.workunit.client.1.smithi134.stdout:9/539: rmdir d0/d7/d23/d1b/d5a 39 2024-04-22T23:23:50.598 INFO:tasks.workunit.client.1.smithi134.stdout:6/678: creat d0/d19/de/d1f/dd5/fe2 x:0 0 0 2024-04-22T23:23:50.598 INFO:tasks.workunit.client.1.smithi134.stdout:4/801: dwrite d4/d35/d5e/f4a [0,4194304] 0 2024-04-22T23:23:50.599 INFO:tasks.workunit.client.1.smithi134.stdout:6/679: readlink d0/d19/db/d75/l7c 0 2024-04-22T23:23:50.603 INFO:tasks.workunit.client.1.smithi134.stdout:6/680: write d0/f60 [3594707,109990] 0 2024-04-22T23:23:50.604 INFO:tasks.workunit.client.1.smithi134.stdout:5/724: dwrite d9/dd/d17/fef [4194304,4194304] 0 2024-04-22T23:23:50.606 INFO:tasks.workunit.client.1.smithi134.stdout:6/681: chown d0/d19/de/d12/d5e/cca 5708 1 2024-04-22T23:23:50.607 INFO:tasks.workunit.client.1.smithi134.stdout:6/682: readlink d0/d19/de/d2f/l54 0 2024-04-22T23:23:50.608 INFO:tasks.workunit.client.1.smithi134.stdout:4/802: write d4/d35/d5e/d25/f98 [1902009,120862] 0 2024-04-22T23:23:50.609 INFO:tasks.workunit.client.1.smithi134.stdout:5/725: creat d9/dd/d88/ffb x:0 0 0 2024-04-22T23:23:50.610 INFO:tasks.workunit.client.1.smithi134.stdout:5/726: chown d9/d11/d34/d4d/d92/lda 25 1 2024-04-22T23:23:50.611 INFO:tasks.workunit.client.1.smithi134.stdout:6/683: write d0/d19/de/d12/d2a/f4c [4207139,5936] 0 2024-04-22T23:23:50.613 INFO:tasks.workunit.client.1.smithi134.stdout:4/803: rename d4/d35/d5e to d4/dd/da8/dc2/df8 0 2024-04-22T23:23:50.614 INFO:tasks.workunit.client.1.smithi134.stdout:6/684: rmdir d0/d19/de/d12/d29/d74 39 2024-04-22T23:23:50.616 INFO:tasks.workunit.client.1.smithi134.stdout:4/804: chown d4/d6b/lba 14 1 2024-04-22T23:23:50.618 INFO:tasks.workunit.client.1.smithi134.stdout:5/727: dread d9/da2/fdb [0,4194304] 0 2024-04-22T23:23:50.622 INFO:tasks.workunit.client.1.smithi134.stdout:5/728: fdatasync d9/d1e/d90/fc4 0 2024-04-22T23:23:50.623 INFO:tasks.workunit.client.1.smithi134.stdout:6/685: dwrite d0/f42 [0,4194304] 0 2024-04-22T23:23:50.623 INFO:tasks.workunit.client.1.smithi134.stdout:4/805: dread d4/d29/dcc/f83 [4194304,4194304] 0 2024-04-22T23:23:50.625 INFO:tasks.workunit.client.1.smithi134.stdout:9/540: dread d0/d7/d12/f30 [0,4194304] 0 2024-04-22T23:23:50.626 INFO:tasks.workunit.client.1.smithi134.stdout:5/729: mknod d9/dd/d17/d44/cfc 0 2024-04-22T23:23:50.630 INFO:tasks.workunit.client.1.smithi134.stdout:5/730: truncate d9/d11/d34/d4d/fde 997355 0 2024-04-22T23:23:50.630 INFO:tasks.workunit.client.1.smithi134.stdout:9/541: getdents d0/d2d 0 2024-04-22T23:23:50.634 INFO:tasks.workunit.client.1.smithi134.stdout:5/731: mknod d9/d11/d73/d95/d20/cfd 0 2024-04-22T23:23:50.634 INFO:tasks.workunit.client.1.smithi134.stdout:4/806: dwrite d4/d35/db4/fbe [0,4194304] 0 2024-04-22T23:23:50.636 INFO:tasks.workunit.client.1.smithi134.stdout:9/542: creat d0/d2d/d94/fb5 x:0 0 0 2024-04-22T23:23:50.637 INFO:tasks.workunit.client.1.smithi134.stdout:5/732: chown f8 330 1 2024-04-22T23:23:50.638 INFO:tasks.workunit.client.1.smithi134.stdout:9/543: fdatasync d0/d7/d23/d26/d4d/d5d/f8f 0 2024-04-22T23:23:50.639 INFO:tasks.workunit.client.1.smithi134.stdout:6/686: dread d0/d19/fa [0,4194304] 0 2024-04-22T23:23:50.644 INFO:tasks.workunit.client.1.smithi134.stdout:5/733: link d9/d1e/f33 d9/d11/d34/d85/ffe 0 2024-04-22T23:23:50.647 INFO:tasks.workunit.client.1.smithi134.stdout:4/807: dwrite d4/d13/fd6 [0,4194304] 0 2024-04-22T23:23:50.656 INFO:tasks.workunit.client.1.smithi134.stdout:6/687: dread d0/d19/db/d35/f4b [0,4194304] 0 2024-04-22T23:23:50.659 INFO:tasks.workunit.client.1.smithi134.stdout:5/734: dwrite d9/dd/fbe [0,4194304] 0 2024-04-22T23:23:50.660 INFO:tasks.workunit.client.1.smithi134.stdout:9/544: dwrite d0/d37/d76/f89 [0,4194304] 0 2024-04-22T23:23:50.664 INFO:tasks.workunit.client.1.smithi134.stdout:4/808: dwrite d4/d7/d32/d3a/d93/db5/fe4 [0,4194304] 0 2024-04-22T23:23:50.667 INFO:tasks.workunit.client.1.smithi134.stdout:6/688: fdatasync d0/d19/de/d1f/dae/fd8 0 2024-04-22T23:23:50.669 INFO:tasks.workunit.client.1.smithi134.stdout:5/735: symlink d9/d11/d34/lff 0 2024-04-22T23:23:50.671 INFO:tasks.workunit.client.1.smithi134.stdout:9/545: rename d0/d7/d23/d26/d51/d88/faf to d0/d7/d23/d26/d4d/fb6 0 2024-04-22T23:23:50.673 INFO:tasks.workunit.client.1.smithi134.stdout:3/486: sync 2024-04-22T23:23:50.673 INFO:tasks.workunit.client.1.smithi134.stdout:0/613: sync 2024-04-22T23:23:50.673 INFO:tasks.workunit.client.1.smithi134.stdout:4/809: creat d4/d7/d32/d3a/dbc/dc4/ff9 x:0 0 0 2024-04-22T23:23:50.673 INFO:tasks.workunit.client.1.smithi134.stdout:9/546: write d0/d37/d76/f91 [1690766,9040] 0 2024-04-22T23:23:50.673 INFO:tasks.workunit.client.1.smithi134.stdout:6/689: symlink d0/d19/db/d35/dbc/le3 0 2024-04-22T23:23:50.674 INFO:tasks.workunit.client.1.smithi134.stdout:0/614: dread - d1/d5/dd/d41/fb7 zero size 2024-04-22T23:23:50.676 INFO:tasks.workunit.client.1.smithi134.stdout:9/547: chown d0/d7/f5e 1332325 1 2024-04-22T23:23:50.676 INFO:tasks.workunit.client.1.smithi134.stdout:4/810: readlink d4/dd/da8/dc2/df8/ld7 0 2024-04-22T23:23:50.677 INFO:tasks.workunit.client.1.smithi134.stdout:3/487: write dc/dd/f7a [5151218,41017] 0 2024-04-22T23:23:50.677 INFO:tasks.workunit.client.1.smithi134.stdout:6/690: chown d0/d19/db/d33 757502 1 2024-04-22T23:23:50.678 INFO:tasks.workunit.client.1.smithi134.stdout:0/615: chown d1/d2/d1d/c51 135 1 2024-04-22T23:23:50.678 INFO:tasks.workunit.client.1.smithi134.stdout:3/488: read - dc/d2a/d62/f65 zero size 2024-04-22T23:23:50.681 INFO:tasks.workunit.client.1.smithi134.stdout:9/548: creat d0/d2d/d82/fb7 x:0 0 0 2024-04-22T23:23:50.681 INFO:tasks.workunit.client.1.smithi134.stdout:9/549: fdatasync d0/d2d/f44 0 2024-04-22T23:23:50.682 INFO:tasks.workunit.client.1.smithi134.stdout:4/811: creat d4/d7/d32/d3a/d93/db5/ffa x:0 0 0 2024-04-22T23:23:50.682 INFO:tasks.workunit.client.1.smithi134.stdout:3/489: mknod dc/d2a/d62/cb5 0 2024-04-22T23:23:50.684 INFO:tasks.workunit.client.1.smithi134.stdout:6/691: rename d0/d19/de/d12/d29/f8d to d0/d19/db/d35/d7f/fe4 0 2024-04-22T23:23:50.687 INFO:tasks.workunit.client.1.smithi134.stdout:9/550: dread d0/d2d/d82/f83 [0,4194304] 0 2024-04-22T23:23:50.687 INFO:tasks.workunit.client.1.smithi134.stdout:4/812: symlink d4/d35/d53/lfb 0 2024-04-22T23:23:50.687 INFO:tasks.workunit.client.1.smithi134.stdout:9/551: dread - d0/d2d/d94/fb5 zero size 2024-04-22T23:23:50.690 INFO:tasks.workunit.client.1.smithi134.stdout:3/490: creat dc/d2a/d62/db2/d2f/d21/d42/d77/fb6 x:0 0 0 2024-04-22T23:23:50.693 INFO:tasks.workunit.client.1.smithi134.stdout:9/552: read d0/d7/d23/d26/d4d/f3b [709336,61293] 0 2024-04-22T23:23:50.694 INFO:tasks.workunit.client.1.smithi134.stdout:3/491: read dc/f22 [4043432,125792] 0 2024-04-22T23:23:50.694 INFO:tasks.workunit.client.1.smithi134.stdout:4/813: rename d4/d7/d32/d84/fcb to d4/d13/d68/df6/ffc 0 2024-04-22T23:23:50.698 INFO:tasks.workunit.client.1.smithi134.stdout:9/553: symlink d0/d7/d23/d26/d51/d4c/lb8 0 2024-04-22T23:23:50.700 INFO:tasks.workunit.client.1.smithi134.stdout:5/736: dwrite d9/d11/d34/d85/ffe [8388608,4194304] 0 2024-04-22T23:23:50.704 INFO:tasks.workunit.client.1.smithi134.stdout:9/554: dread d0/d7/d12/f42 [0,4194304] 0 2024-04-22T23:23:50.705 INFO:tasks.workunit.client.1.smithi134.stdout:8/564: sync 2024-04-22T23:23:50.705 INFO:tasks.workunit.client.1.smithi134.stdout:7/697: sync 2024-04-22T23:23:50.705 INFO:tasks.workunit.client.1.smithi134.stdout:1/798: sync 2024-04-22T23:23:50.705 INFO:tasks.workunit.client.1.smithi134.stdout:2/559: sync 2024-04-22T23:23:50.710 INFO:tasks.workunit.client.1.smithi134.stdout:9/555: dwrite d0/f54 [0,4194304] 0 2024-04-22T23:23:50.756 INFO:tasks.workunit.client.1.smithi134.stdout:4/814: mkdir d4/dd/da8/dc2/df8/d25/dfd 0 2024-04-22T23:23:50.760 INFO:tasks.workunit.client.1.smithi134.stdout:5/737: mknod d9/dbb/c100 0 2024-04-22T23:23:50.762 INFO:tasks.workunit.client.1.smithi134.stdout:0/616: dwrite d1/d2/f4 [0,4194304] 0 2024-04-22T23:23:50.764 INFO:tasks.workunit.client.1.smithi134.stdout:1/799: fdatasync db/f109 0 2024-04-22T23:23:50.765 INFO:tasks.workunit.client.1.smithi134.stdout:7/698: truncate d4/d8/d41/d76/dc1/d2f/d69/f78 104641 0 2024-04-22T23:23:50.766 INFO:tasks.workunit.client.1.smithi134.stdout:8/565: chown d7/db/d42/d65/d46/c48 3 1 2024-04-22T23:23:50.766 INFO:tasks.workunit.client.1.smithi134.stdout:9/556: unlink d0/d7/d23/d26/c38 0 2024-04-22T23:23:50.767 INFO:tasks.workunit.client.1.smithi134.stdout:4/815: symlink d4/dd/da8/lfe 0 2024-04-22T23:23:50.768 INFO:tasks.workunit.client.1.smithi134.stdout:9/557: chown d0/d2d/d82 1 1 2024-04-22T23:23:50.769 INFO:tasks.workunit.client.1.smithi134.stdout:5/738: fsync d9/d1e/f23 0 2024-04-22T23:23:50.773 INFO:tasks.workunit.client.1.smithi134.stdout:1/800: mkdir db/d16/d87/dbd/d10f 0 2024-04-22T23:23:50.773 INFO:tasks.workunit.client.1.smithi134.stdout:6/692: dwrite d0/d19/de/d2f/fb8 [0,4194304] 0 2024-04-22T23:23:50.773 INFO:tasks.workunit.client.1.smithi134.stdout:5/739: stat d9/d11/d34/d4d/dd4/cf4 0 2024-04-22T23:23:50.775 INFO:tasks.workunit.client.1.smithi134.stdout:8/566: dread d7/db/d2d/d39/d58/d8a/f94 [0,4194304] 0 2024-04-22T23:23:50.777 INFO:tasks.workunit.client.1.smithi134.stdout:7/699: creat d4/d8/d41/d76/dc1/d2f/d34/d54/ff9 x:0 0 0 2024-04-22T23:23:50.778 INFO:tasks.workunit.client.1.smithi134.stdout:6/693: write d0/f60 [6651437,118965] 0 2024-04-22T23:23:50.778 INFO:tasks.workunit.client.1.smithi134.stdout:8/567: readlink d7/db/l41 0 2024-04-22T23:23:50.781 INFO:tasks.workunit.client.1.smithi134.stdout:1/801: write db/d15/d17/d20/d2a/d2d/f48 [3610171,128087] 0 2024-04-22T23:23:50.781 INFO:tasks.workunit.client.1.smithi134.stdout:8/568: fsync d7/db/d2d/f99 0 2024-04-22T23:23:50.782 INFO:tasks.workunit.client.1.smithi134.stdout:9/558: chown d0/d7/d23/d1b/d5a 202179922 1 2024-04-22T23:23:50.783 INFO:tasks.workunit.client.1.smithi134.stdout:6/694: chown d0/d19/de/d12/f7e 1 1 2024-04-22T23:23:50.783 INFO:tasks.workunit.client.1.smithi134.stdout:2/560: creat d2/dd/df/fb9 x:0 0 0 2024-04-22T23:23:50.785 INFO:tasks.workunit.client.1.smithi134.stdout:4/816: mknod d4/d9b/cff 0 2024-04-22T23:23:50.785 INFO:tasks.workunit.client.1.smithi134.stdout:1/802: write db/db5/ff3 [1130259,56365] 0 2024-04-22T23:23:50.789 INFO:tasks.workunit.client.1.smithi134.stdout:5/740: dwrite d9/dd/f86 [0,4194304] 0 2024-04-22T23:23:50.793 INFO:tasks.workunit.client.1.smithi134.stdout:7/700: mkdir d4/d4a/d85/dd7/dfa 0 2024-04-22T23:23:50.795 INFO:tasks.workunit.client.1.smithi134.stdout:8/569: dread d7/dc/d2c/d4f/f6b [0,4194304] 0 2024-04-22T23:23:50.795 INFO:tasks.workunit.client.1.smithi134.stdout:9/559: dread d0/d7/f9 [0,4194304] 0 2024-04-22T23:23:50.797 INFO:tasks.workunit.client.1.smithi134.stdout:6/695: mkdir d0/d19/de/d12/de5 0 2024-04-22T23:23:50.798 INFO:tasks.workunit.client.1.smithi134.stdout:7/701: truncate d4/d8/d41/d76/dc1/d2f/d70/d86/ff0 56488 0 2024-04-22T23:23:50.798 INFO:tasks.workunit.client.1.smithi134.stdout:1/803: rmdir db/d16 39 2024-04-22T23:23:50.800 INFO:tasks.workunit.client.1.smithi134.stdout:6/696: write d0/d19/db/d35/f79 [2356141,109523] 0 2024-04-22T23:23:50.800 INFO:tasks.workunit.client.1.smithi134.stdout:2/561: mkdir d2/d15/d24/d1d/dba 0 2024-04-22T23:23:50.800 INFO:tasks.workunit.client.1.smithi134.stdout:7/702: write d4/d8/d41/d76/dc1/f1f [1653513,50024] 0 2024-04-22T23:23:50.801 INFO:tasks.workunit.client.1.smithi134.stdout:0/617: getdents d1/d77 0 2024-04-22T23:23:50.804 INFO:tasks.workunit.client.1.smithi134.stdout:4/817: symlink d4/d7/l100 0 2024-04-22T23:23:50.805 INFO:tasks.workunit.client.1.smithi134.stdout:9/560: write d0/d7/d23/f2a [76768,34724] 0 2024-04-22T23:23:50.806 INFO:tasks.workunit.client.1.smithi134.stdout:9/561: chown d0/d7/f6f 85561 1 2024-04-22T23:23:50.809 INFO:tasks.workunit.client.1.smithi134.stdout:6/697: unlink d0/d19/de/d12/d5e/d6d/cc0 0 2024-04-22T23:23:50.810 INFO:tasks.workunit.client.1.smithi134.stdout:2/562: dread d2/d15/d24/d1d/f8d [0,4194304] 0 2024-04-22T23:23:50.812 INFO:tasks.workunit.client.1.smithi134.stdout:8/570: truncate d7/db/f1f 1482142 0 2024-04-22T23:23:50.813 INFO:tasks.workunit.client.1.smithi134.stdout:9/562: mkdir d0/d7/d23/d26/d51/d88/db9 0 2024-04-22T23:23:50.817 INFO:tasks.workunit.client.1.smithi134.stdout:5/741: dread d9/d11/d73/d95/d20/f91 [0,4194304] 0 2024-04-22T23:23:50.819 INFO:tasks.workunit.client.1.smithi134.stdout:4/818: unlink d4/dd/da8/dc2/df8/c72 0 2024-04-22T23:23:50.819 INFO:tasks.workunit.client.1.smithi134.stdout:1/804: link db/d15/d17/d20/d2a/d2d/d2e/f39 db/d15/d17/d20/d2a/d2d/d86/f110 0 2024-04-22T23:23:50.820 INFO:tasks.workunit.client.1.smithi134.stdout:5/742: chown d9/dd/d17/d44/cb3 130795238 1 2024-04-22T23:23:50.821 INFO:tasks.workunit.client.1.smithi134.stdout:8/571: dwrite d7/db/d42/d65/d46/d9a/faa [0,4194304] 0 2024-04-22T23:23:50.823 INFO:tasks.workunit.client.1.smithi134.stdout:8/572: stat d7/db/d2d/d39/f45 0 2024-04-22T23:23:50.823 INFO:tasks.workunit.client.1.smithi134.stdout:9/563: dread d0/d7/f5b [0,4194304] 0 2024-04-22T23:23:50.827 INFO:tasks.workunit.client.1.smithi134.stdout:4/819: chown d4/d35/l99 270834 1 2024-04-22T23:23:50.829 INFO:tasks.workunit.client.1.smithi134.stdout:5/743: fdatasync d9/dd/d17/f45 0 2024-04-22T23:23:50.830 INFO:tasks.workunit.client.1.smithi134.stdout:0/618: link d1/d5/d58/f29 d1/d5/d58/d6b/db9/fcd 0 2024-04-22T23:23:50.833 INFO:tasks.workunit.client.1.smithi134.stdout:2/563: creat d2/dd/df/fbb x:0 0 0 2024-04-22T23:23:50.833 INFO:tasks.workunit.client.1.smithi134.stdout:2/564: chown d2/dd/d68 45584038 1 2024-04-22T23:23:50.835 INFO:tasks.workunit.client.1.smithi134.stdout:1/805: mkdir db/d15/d17/d20/d111 0 2024-04-22T23:23:50.838 INFO:tasks.workunit.client.1.smithi134.stdout:2/565: fsync d2/dd/df/d9c/daa/db0/f9b 0 2024-04-22T23:23:50.840 INFO:tasks.workunit.client.1.smithi134.stdout:5/744: dwrite d9/fe [0,4194304] 0 2024-04-22T23:23:50.848 INFO:tasks.workunit.client.1.smithi134.stdout:4/820: dwrite d4/d29/dcc/f26 [0,4194304] 0 2024-04-22T23:23:50.856 INFO:tasks.workunit.client.1.smithi134.stdout:0/619: rename d1/d5/dd/f16 to d1/d2/fce 0 2024-04-22T23:23:50.856 INFO:tasks.workunit.client.1.smithi134.stdout:8/573: creat d7/db/d2d/fb0 x:0 0 0 2024-04-22T23:23:50.859 INFO:tasks.workunit.client.1.smithi134.stdout:2/566: mknod d2/d42/d54/cbc 0 2024-04-22T23:23:50.859 INFO:tasks.workunit.client.1.smithi134.stdout:9/564: dread d0/d7/fb [0,4194304] 0 2024-04-22T23:23:50.864 INFO:tasks.workunit.client.1.smithi134.stdout:4/821: chown d4/d29/dcc/c7d 1158814206 1 2024-04-22T23:23:50.878 INFO:tasks.workunit.client.1.smithi134.stdout:1/806: truncate db/fa6 4013426 0 2024-04-22T23:23:50.880 INFO:tasks.workunit.client.1.smithi134.stdout:1/807: stat db/d15/d17/d20/d2a/lb7 0 2024-04-22T23:23:50.881 INFO:tasks.workunit.client.1.smithi134.stdout:1/808: chown db/f27 113348353 1 2024-04-22T23:23:50.882 INFO:tasks.workunit.client.1.smithi134.stdout:1/809: stat db/d15/d17/dc9/lfe 0 2024-04-22T23:23:50.892 INFO:tasks.workunit.client.1.smithi134.stdout:5/745: creat d9/dd/df1/f101 x:0 0 0 2024-04-22T23:23:50.895 INFO:tasks.workunit.client.1.smithi134.stdout:0/620: symlink d1/d5/d58/d6b/lcf 0 2024-04-22T23:23:50.897 INFO:tasks.workunit.client.1.smithi134.stdout:0/621: chown d1/d5/dd/f2a 32942 1 2024-04-22T23:23:50.897 INFO:tasks.workunit.client.1.smithi134.stdout:2/567: mknod d2/dd/d68/cbd 0 2024-04-22T23:23:50.900 INFO:tasks.workunit.client.1.smithi134.stdout:9/565: write d0/d7/d23/d26/d4d/fb6 [763966,117787] 0 2024-04-22T23:23:50.901 INFO:tasks.workunit.client.1.smithi134.stdout:5/746: mknod d9/da2/de6/c102 0 2024-04-22T23:23:50.904 INFO:tasks.workunit.client.1.smithi134.stdout:1/810: mkdir db/d15/d17/d112 0 2024-04-22T23:23:50.908 INFO:tasks.workunit.client.1.smithi134.stdout:2/568: fsync d2/dd/d4a/d73/f7f 0 2024-04-22T23:23:50.909 INFO:tasks.workunit.client.1.smithi134.stdout:8/574: rename d7/dc/d2c/d4f/fa0 to d7/dc/d2c/fb1 0 2024-04-22T23:23:50.910 INFO:tasks.workunit.client.1.smithi134.stdout:5/747: mknod d9/dd/df1/c103 0 2024-04-22T23:23:50.911 INFO:tasks.workunit.client.1.smithi134.stdout:8/575: chown d7/db/l93 1125325745 1 2024-04-22T23:23:50.913 INFO:tasks.workunit.client.1.smithi134.stdout:9/566: mknod d0/d7/d23/d26/d4d/d5d/d8e/dae/db1/cba 0 2024-04-22T23:23:50.913 INFO:tasks.workunit.client.1.smithi134.stdout:4/822: creat d4/dd/da8/dc2/df8/d25/dfd/f101 x:0 0 0 2024-04-22T23:23:50.920 INFO:tasks.workunit.client.1.smithi134.stdout:2/569: dwrite d2/dd/df/d9c/daa/db0/f9b [0,4194304] 0 2024-04-22T23:23:50.922 INFO:tasks.workunit.client.1.smithi134.stdout:8/576: rename d7/dc/d2c/d37/d3a/f82 to d7/dc/d19/da5/fb2 0 2024-04-22T23:23:50.928 INFO:tasks.workunit.client.1.smithi134.stdout:9/567: symlink d0/d7/d23/d26/d4d/d5d/d8e/dae/lbb 0 2024-04-22T23:23:50.929 INFO:tasks.workunit.client.1.smithi134.stdout:4/823: rmdir d4/d7/d32/d3a/dbc 39 2024-04-22T23:23:50.960 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:50 smithi134 ceph-mon[34609]: pgmap v280: 65 pgs: 65 active+clean; 2.8 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 85 MiB/s rd, 253 MiB/s wr, 511 op/s 2024-04-22T23:23:50.965 INFO:tasks.workunit.client.1.smithi134.stdout:2/570: rename d2/d15/d24/d3b/d8a/l9f to d2/d42/d54/d33/d91/lbe 0 2024-04-22T23:23:50.969 INFO:tasks.workunit.client.1.smithi134.stdout:9/568: mkdir d0/d7/d23/d26/dbc 0 2024-04-22T23:23:50.972 INFO:tasks.workunit.client.1.smithi134.stdout:9/569: chown d0/d7/d12/l7a 1289395423 1 2024-04-22T23:23:50.974 INFO:tasks.workunit.client.1.smithi134.stdout:4/824: truncate d4/dd/f41 1440809 0 2024-04-22T23:23:50.974 INFO:tasks.workunit.client.1.smithi134.stdout:4/825: stat d4/d29/dcc/d64/c79 0 2024-04-22T23:23:50.978 INFO:tasks.workunit.client.1.smithi134.stdout:9/570: creat d0/d7/d23/d26/d4d/d5d/d8e/dae/fbd x:0 0 0 2024-04-22T23:23:50.982 INFO:tasks.workunit.client.1.smithi134.stdout:9/571: write d0/d7/d27/f33 [4638689,117794] 0 2024-04-22T23:23:50.983 INFO:tasks.workunit.client.1.smithi134.stdout:5/748: dwrite d9/d11/d34/d85/f9e [0,4194304] 0 2024-04-22T23:23:50.988 INFO:tasks.workunit.client.1.smithi134.stdout:5/749: rmdir d9/dd/d17 39 2024-04-22T23:23:50.989 INFO:tasks.workunit.client.1.smithi134.stdout:8/577: dwrite d7/db/d2d/d39/d58/d8a/f94 [0,4194304] 0 2024-04-22T23:23:50.997 INFO:tasks.workunit.client.1.smithi134.stdout:9/572: rmdir d0/d7/d23/d26/d51/d88/db4 0 2024-04-22T23:23:51.000 INFO:tasks.workunit.client.1.smithi134.stdout:5/750: dwrite d9/d11/d34/f40 [4194304,4194304] 0 2024-04-22T23:23:51.004 INFO:tasks.workunit.client.1.smithi134.stdout:5/751: fsync d9/dd/d88/db7/ffa 0 2024-04-22T23:23:51.007 INFO:tasks.workunit.client.1.smithi134.stdout:9/573: getdents d0/d7/d23/d26/dbc 0 2024-04-22T23:23:51.009 INFO:tasks.workunit.client.1.smithi134.stdout:5/752: truncate d9/dd/d17/d44/f59 56384 0 2024-04-22T23:23:51.009 INFO:tasks.workunit.client.1.smithi134.stdout:8/578: link d7/dc/d2c/d4f/d6c/c8d d7/db/d42/d65/cb3 0 2024-04-22T23:23:51.015 INFO:tasks.workunit.client.1.smithi134.stdout:9/574: dread d0/d7/fb [0,4194304] 0 2024-04-22T23:23:51.018 INFO:tasks.workunit.client.1.smithi134.stdout:8/579: dwrite d7/db/d2d/d39/f72 [0,4194304] 0 2024-04-22T23:23:51.018 INFO:tasks.workunit.client.1.smithi134.stdout:9/575: truncate d0/d7/d12/f6a 5227629 0 2024-04-22T23:23:51.020 INFO:tasks.workunit.client.1.smithi134.stdout:8/580: write d7/dc/f22 [2969861,92590] 0 2024-04-22T23:23:51.022 INFO:tasks.workunit.client.1.smithi134.stdout:8/581: readlink d7/db/d42/d65/l6a 0 2024-04-22T23:23:51.023 INFO:tasks.workunit.client.1.smithi134.stdout:9/576: truncate d0/d7/d27/f59 3104820 0 2024-04-22T23:23:51.025 INFO:tasks.workunit.client.1.smithi134.stdout:8/582: symlink d7/db/d42/lb4 0 2024-04-22T23:23:51.028 INFO:tasks.workunit.client.1.smithi134.stdout:9/577: symlink d0/d7/d23/d26/d51/d88/db9/lbe 0 2024-04-22T23:23:51.028 INFO:tasks.workunit.client.1.smithi134.stdout:8/583: read d7/dc/f12 [1354503,49818] 0 2024-04-22T23:23:51.031 INFO:tasks.workunit.client.1.smithi134.stdout:9/578: mkdir d0/d2d/d82/dbf 0 2024-04-22T23:23:51.034 INFO:tasks.workunit.client.1.smithi134.stdout:8/584: symlink d7/db/d42/d65/d46/d9a/da3/d88/lb5 0 2024-04-22T23:23:51.034 INFO:tasks.workunit.client.1.smithi134.stdout:5/753: write d9/dd/d17/f66 [1015887,83836] 0 2024-04-22T23:23:51.037 INFO:tasks.workunit.client.1.smithi134.stdout:9/579: unlink d0/d37/d76/f89 0 2024-04-22T23:23:51.040 INFO:tasks.workunit.client.1.smithi134.stdout:8/585: symlink d7/db/lb6 0 2024-04-22T23:23:51.041 INFO:tasks.workunit.client.1.smithi134.stdout:9/580: dread d0/d7/fb [0,4194304] 0 2024-04-22T23:23:51.042 INFO:tasks.workunit.client.1.smithi134.stdout:5/754: fsync d9/d11/d73/f80 0 2024-04-22T23:23:51.044 INFO:tasks.workunit.client.1.smithi134.stdout:5/755: chown d9/dbb 50 1 2024-04-22T23:23:51.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:50 smithi067 ceph-mon[26453]: pgmap v280: 65 pgs: 65 active+clean; 2.8 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 85 MiB/s rd, 253 MiB/s wr, 511 op/s 2024-04-22T23:23:51.045 INFO:tasks.workunit.client.1.smithi134.stdout:5/756: fsync d9/d11/d34/d4d/d92/fab 0 2024-04-22T23:23:51.047 INFO:tasks.workunit.client.1.smithi134.stdout:8/586: rename d7/db/l62 to d7/db/lb7 0 2024-04-22T23:23:51.047 INFO:tasks.workunit.client.1.smithi134.stdout:9/581: fdatasync d0/d7/d23/f45 0 2024-04-22T23:23:51.049 INFO:tasks.workunit.client.1.smithi134.stdout:9/582: write d0/d37/d76/f91 [9233614,92111] 0 2024-04-22T23:23:51.051 INFO:tasks.workunit.client.1.smithi134.stdout:5/757: creat d9/d11/d34/d4d/dd4/f104 x:0 0 0 2024-04-22T23:23:51.053 INFO:tasks.workunit.client.1.smithi134.stdout:5/758: chown d9/dd/d17/d2a/c56 0 1 2024-04-22T23:23:51.054 INFO:tasks.workunit.client.1.smithi134.stdout:8/587: dwrite d7/dc/d2c/d4f/d6c/d6e/f6f [0,4194304] 0 2024-04-22T23:23:51.054 INFO:tasks.workunit.client.1.smithi134.stdout:5/759: readlink d9/d11/d73/d95/d20/l57 0 2024-04-22T23:23:51.056 INFO:tasks.workunit.client.1.smithi134.stdout:9/583: mknod d0/d7/d23/d26/d4d/d5d/d8e/cc0 0 2024-04-22T23:23:51.058 INFO:tasks.workunit.client.1.smithi134.stdout:5/760: creat d9/d11/d73/d95/f105 x:0 0 0 2024-04-22T23:23:51.059 INFO:tasks.workunit.client.1.smithi134.stdout:8/588: symlink d7/dc/d19/lb8 0 2024-04-22T23:23:51.059 INFO:tasks.workunit.client.1.smithi134.stdout:5/761: read - d9/dd/d52/f79 zero size 2024-04-22T23:23:51.059 INFO:tasks.workunit.client.1.smithi134.stdout:9/584: mkdir d0/d2d/dc1 0 2024-04-22T23:23:51.061 INFO:tasks.workunit.client.1.smithi134.stdout:9/585: dread - d0/d7/f92 zero size 2024-04-22T23:23:51.061 INFO:tasks.workunit.client.1.smithi134.stdout:5/762: rmdir d9/dd/d17 39 2024-04-22T23:23:51.063 INFO:tasks.workunit.client.1.smithi134.stdout:5/763: symlink d9/d11/d34/d4d/d92/l106 0 2024-04-22T23:23:51.064 INFO:tasks.workunit.client.1.smithi134.stdout:8/589: creat d7/db/d2d/d39/d58/fb9 x:0 0 0 2024-04-22T23:23:51.076 INFO:tasks.workunit.client.1.smithi134.stdout:9/586: symlink d0/d2d/d82/da0/lc2 0 2024-04-22T23:23:51.078 INFO:tasks.workunit.client.1.smithi134.stdout:5/764: write d9/dd/d17/d44/f59 [983217,76813] 0 2024-04-22T23:23:51.080 INFO:tasks.workunit.client.1.smithi134.stdout:3/492: sync 2024-04-22T23:23:51.081 INFO:tasks.workunit.client.1.smithi134.stdout:8/590: dwrite d7/f23 [0,4194304] 0 2024-04-22T23:23:51.081 INFO:tasks.workunit.client.1.smithi134.stdout:9/587: write d0/d7/f19 [1794947,45656] 0 2024-04-22T23:23:51.081 INFO:tasks.workunit.client.1.smithi134.stdout:5/765: truncate d9/d11/d34/d4d/dd4/f104 809246 0 2024-04-22T23:23:51.083 INFO:tasks.workunit.client.1.smithi134.stdout:9/588: stat d0/d7/d23/d26/d4d/d5d/l85 0 2024-04-22T23:23:51.084 INFO:tasks.workunit.client.1.smithi134.stdout:3/493: creat dc/d2a/d62/db2/d3b/d29/fb7 x:0 0 0 2024-04-22T23:23:51.085 INFO:tasks.workunit.client.1.smithi134.stdout:3/494: readlink dc/d2a/d62/db2/d2f/d21/d42/d77/la4 0 2024-04-22T23:23:51.085 INFO:tasks.workunit.client.1.smithi134.stdout:9/589: mknod d0/d7/d23/d1b/cc3 0 2024-04-22T23:23:51.086 INFO:tasks.workunit.client.1.smithi134.stdout:3/495: chown dc/dd/l27 2082474792 1 2024-04-22T23:23:51.103 INFO:tasks.workunit.client.1.smithi134.stdout:8/591: rename d7/dc/d19/l29 to d7/lba 0 2024-04-22T23:23:51.104 INFO:tasks.workunit.client.1.smithi134.stdout:3/496: creat dc/d2a/d62/db2/d3b/d29/fb8 x:0 0 0 2024-04-22T23:23:51.105 INFO:tasks.workunit.client.1.smithi134.stdout:9/590: mknod d0/d2d/dc1/cc4 0 2024-04-22T23:23:51.108 INFO:tasks.workunit.client.1.smithi134.stdout:8/592: rename d7/dc/d2c/d4f/d6c/d6e/c73 to d7/dc/d2c/cbb 0 2024-04-22T23:23:51.108 INFO:tasks.workunit.client.1.smithi134.stdout:9/591: mkdir d0/d7/d23/d1b/dc5 0 2024-04-22T23:23:51.112 INFO:tasks.workunit.client.1.smithi134.stdout:8/593: rename d7/db/d2d/d39/d58/cac to d7/dc/d2c/d37/d47/daf/cbc 0 2024-04-22T23:23:51.114 INFO:tasks.workunit.client.1.smithi134.stdout:8/594: write d7/dc/f9c [2231216,126773] 0 2024-04-22T23:23:51.115 INFO:tasks.workunit.client.1.smithi134.stdout:9/592: dwrite d0/d7/d23/d26/fb0 [0,4194304] 0 2024-04-22T23:23:51.117 INFO:tasks.workunit.client.1.smithi134.stdout:8/595: write d7/db/d2d/d39/d58/f96 [1223621,49773] 0 2024-04-22T23:23:51.120 INFO:tasks.workunit.client.1.smithi134.stdout:9/593: mknod d0/d2d/d82/da0/cc6 0 2024-04-22T23:23:51.121 INFO:tasks.workunit.client.1.smithi134.stdout:9/594: chown d0 0 1 2024-04-22T23:23:51.126 INFO:tasks.workunit.client.1.smithi134.stdout:9/595: rename d0/d7/d23/d26/d4d/d5d/d8e/dae/fbd to d0/d2d/dc1/fc7 0 2024-04-22T23:23:51.126 INFO:tasks.workunit.client.1.smithi134.stdout:8/596: dread f6 [4194304,4194304] 0 2024-04-22T23:23:51.130 INFO:tasks.workunit.client.1.smithi134.stdout:8/597: symlink d7/db/d42/d65/d46/d9a/da3/d88/da6/lbd 0 2024-04-22T23:23:51.131 INFO:tasks.workunit.client.1.smithi134.stdout:9/596: link d0/d7/d23/d26/d4d/c77 d0/d7/cc8 0 2024-04-22T23:23:51.133 INFO:tasks.workunit.client.1.smithi134.stdout:8/598: symlink d7/dc/lbe 0 2024-04-22T23:23:51.133 INFO:tasks.workunit.client.1.smithi134.stdout:8/599: dread - d7/dc/fa8 zero size 2024-04-22T23:23:51.133 INFO:tasks.workunit.client.1.smithi134.stdout:9/597: mknod d0/d7/d23/d26/d51/d4c/cc9 0 2024-04-22T23:23:51.134 INFO:tasks.workunit.client.1.smithi134.stdout:8/600: stat d7/dc/d2c 0 2024-04-22T23:23:51.134 INFO:tasks.workunit.client.1.smithi134.stdout:8/601: read - d7/dc/fa8 zero size 2024-04-22T23:23:51.135 INFO:tasks.workunit.client.1.smithi134.stdout:8/602: chown d7/dc/d2c/d37/f3f 938416309 1 2024-04-22T23:23:51.139 INFO:tasks.workunit.client.1.smithi134.stdout:9/598: dwrite d0/d7/f92 [0,4194304] 0 2024-04-22T23:23:51.140 INFO:tasks.workunit.client.1.smithi134.stdout:9/599: fdatasync d0/d7/d23/d26/d51/d4c/f71 0 2024-04-22T23:23:51.141 INFO:tasks.workunit.client.1.smithi134.stdout:8/603: rename d7/dc/d19/f21 to d7/dc/d2c/d37/d47/daf/fbf 0 2024-04-22T23:23:51.146 INFO:tasks.workunit.client.1.smithi134.stdout:9/600: chown d0/d2d/d3d/f72 57542530 1 2024-04-22T23:23:51.148 INFO:tasks.workunit.client.1.smithi134.stdout:8/604: dwrite d7/db/d2d/f3e [0,4194304] 0 2024-04-22T23:23:51.150 INFO:tasks.workunit.client.1.smithi134.stdout:9/601: symlink d0/d2d/d82/dbf/lca 0 2024-04-22T23:23:51.152 INFO:tasks.workunit.client.1.smithi134.stdout:8/605: creat d7/db/fc0 x:0 0 0 2024-04-22T23:23:51.154 INFO:tasks.workunit.client.1.smithi134.stdout:9/602: rename d0/d7/f41 to d0/d7/d23/d26/d51/d88/db9/fcb 0 2024-04-22T23:23:51.154 INFO:tasks.workunit.client.1.smithi134.stdout:8/606: symlink d7/dc/d2c/d37/lc1 0 2024-04-22T23:23:51.156 INFO:tasks.workunit.client.1.smithi134.stdout:8/607: rmdir d7/db/d42 39 2024-04-22T23:23:51.159 INFO:tasks.workunit.client.1.smithi134.stdout:8/608: rename d7/dc/d2c/d4f/d6c/c75 to d7/dc/d2c/d37/d47/daf/cc2 0 2024-04-22T23:23:51.161 INFO:tasks.workunit.client.1.smithi134.stdout:8/609: getdents d7/dc/d2c 0 2024-04-22T23:23:51.181 INFO:tasks.workunit.client.1.smithi134.stdout:9/603: dread d0/d37/f8b [0,4194304] 0 2024-04-22T23:23:51.184 INFO:tasks.workunit.client.1.smithi134.stdout:9/604: creat d0/d7/d12/fcc x:0 0 0 2024-04-22T23:23:51.185 INFO:tasks.workunit.client.1.smithi134.stdout:8/610: dread d7/db/d42/f59 [0,4194304] 0 2024-04-22T23:23:51.185 INFO:tasks.workunit.client.1.smithi134.stdout:8/611: dread - d7/db/d2d/fb0 zero size 2024-04-22T23:23:51.188 INFO:tasks.workunit.client.1.smithi134.stdout:8/612: symlink d7/db/d42/d65/d46/lc3 0 2024-04-22T23:23:51.190 INFO:tasks.workunit.client.1.smithi134.stdout:8/613: stat d7/db/d2d/c32 0 2024-04-22T23:23:51.190 INFO:tasks.workunit.client.1.smithi134.stdout:9/605: dwrite d0/d2d/d82/da0/f8d [4194304,4194304] 0 2024-04-22T23:23:51.193 INFO:tasks.workunit.client.1.smithi134.stdout:8/614: creat d7/dc/d2c/d4f/d6c/d6e/fc4 x:0 0 0 2024-04-22T23:23:51.196 INFO:tasks.workunit.client.1.smithi134.stdout:9/606: link d0/d7/f17 d0/d7/d23/d26/d51/d88/fcd 0 2024-04-22T23:23:51.197 INFO:tasks.workunit.client.1.smithi134.stdout:9/607: rmdir d0/d7/d12 39 2024-04-22T23:23:51.199 INFO:tasks.workunit.client.1.smithi134.stdout:9/608: dread - d0/d7/d23/d26/d4d/d5d/d8e/f9c zero size 2024-04-22T23:23:51.278 INFO:tasks.workunit.client.1.smithi134.stdout:7/703: sync 2024-04-22T23:23:51.279 INFO:tasks.workunit.client.1.smithi134.stdout:8/615: write f6 [7855561,26042] 0 2024-04-22T23:23:51.281 INFO:tasks.workunit.client.1.smithi134.stdout:7/704: creat d4/d8/d3b/ffb x:0 0 0 2024-04-22T23:23:51.282 INFO:tasks.workunit.client.1.smithi134.stdout:9/609: creat d0/d7/d23/d26/dbc/fce x:0 0 0 2024-04-22T23:23:51.283 INFO:tasks.workunit.client.1.smithi134.stdout:8/616: chown d7/dc/d19/l78 570750 1 2024-04-22T23:23:51.284 INFO:tasks.workunit.client.1.smithi134.stdout:9/610: fdatasync d0/d7/fad 0 2024-04-22T23:23:51.287 INFO:tasks.workunit.client.1.smithi134.stdout:7/705: dwrite d4/fa [0,4194304] 0 2024-04-22T23:23:51.288 INFO:tasks.workunit.client.1.smithi134.stdout:8/617: unlink d7/db/d42/d65/d46/d9a/da3/d88/da6/lbd 0 2024-04-22T23:23:51.292 INFO:tasks.workunit.client.1.smithi134.stdout:8/618: creat d7/fc5 x:0 0 0 2024-04-22T23:23:51.297 INFO:tasks.workunit.client.1.smithi134.stdout:8/619: link d7/db/d2d/l44 d7/dc/d2c/d4f/lc6 0 2024-04-22T23:23:51.302 INFO:tasks.workunit.client.1.smithi134.stdout:8/620: link d7/lba d7/dc/d2c/d4f/lc7 0 2024-04-22T23:23:51.305 INFO:tasks.workunit.client.1.smithi134.stdout:9/611: getdents d0/d7/d23/d26/dbc 0 2024-04-22T23:23:51.306 INFO:tasks.workunit.client.1.smithi134.stdout:8/621: creat d7/dc/d2c/d37/d3a/fc8 x:0 0 0 2024-04-22T23:23:51.307 INFO:tasks.workunit.client.1.smithi134.stdout:9/612: mknod d0/d7/d23/d26/dbc/ccf 0 2024-04-22T23:23:51.309 INFO:tasks.workunit.client.1.smithi134.stdout:7/706: dread d4/d8/f22 [0,4194304] 0 2024-04-22T23:23:51.309 INFO:tasks.workunit.client.1.smithi134.stdout:9/613: mknod d0/d7/d12/cd0 0 2024-04-22T23:23:51.311 INFO:tasks.workunit.client.1.smithi134.stdout:8/622: dread d7/dc/d2c/d4f/f6b [0,4194304] 0 2024-04-22T23:23:51.311 INFO:tasks.workunit.client.1.smithi134.stdout:9/614: read - d0/d7/d23/d26/d4d/d5d/d8e/f9c zero size 2024-04-22T23:23:51.314 INFO:tasks.workunit.client.1.smithi134.stdout:9/615: creat d0/d2d/d82/dbf/fd1 x:0 0 0 2024-04-22T23:23:51.317 INFO:tasks.workunit.client.1.smithi134.stdout:9/616: read - d0/d7/d12/d22/f7d zero size 2024-04-22T23:23:51.319 INFO:tasks.workunit.client.1.smithi134.stdout:8/623: dwrite d7/dc/fa8 [0,4194304] 0 2024-04-22T23:23:51.319 INFO:tasks.workunit.client.1.smithi134.stdout:7/707: write d4/d8/d41/d5b/ff3 [2041682,130467] 0 2024-04-22T23:23:51.323 INFO:tasks.workunit.client.1.smithi134.stdout:9/617: getdents d0/d7/d23/d26 0 2024-04-22T23:23:51.323 INFO:tasks.workunit.client.1.smithi134.stdout:7/708: mknod d4/cfc 0 2024-04-22T23:23:51.328 INFO:tasks.workunit.client.1.smithi134.stdout:8/624: dwrite d7/dc/d19/da5/fab [0,4194304] 0 2024-04-22T23:23:51.330 INFO:tasks.workunit.client.1.smithi134.stdout:8/625: chown d7/db/d2d/d39/d58 0 1 2024-04-22T23:23:51.333 INFO:tasks.workunit.client.1.smithi134.stdout:7/709: dwrite d4/d4a/d85/dd7/fd9 [0,4194304] 0 2024-04-22T23:23:51.339 INFO:tasks.workunit.client.1.smithi134.stdout:7/710: creat d4/d8/d41/d76/dc1/d2f/d34/d54/ffd x:0 0 0 2024-04-22T23:23:51.339 INFO:tasks.workunit.client.1.smithi134.stdout:9/618: write d0/d7/d23/d26/f58 [1129636,22257] 0 2024-04-22T23:23:51.340 INFO:tasks.workunit.client.1.smithi134.stdout:8/626: dwrite d7/dc/d2c/fb1 [4194304,4194304] 0 2024-04-22T23:23:51.344 INFO:tasks.workunit.client.1.smithi134.stdout:8/627: dread - d7/dc/d2c/d37/d47/f66 zero size 2024-04-22T23:23:51.352 INFO:tasks.workunit.client.1.smithi134.stdout:7/711: dwrite d4/d8/d2a/d42/d75/fbf [0,4194304] 0 2024-04-22T23:23:51.356 INFO:tasks.workunit.client.1.smithi134.stdout:9/619: dwrite d0/d37/f95 [0,4194304] 0 2024-04-22T23:23:51.356 INFO:tasks.workunit.client.1.smithi134.stdout:7/712: mknod d4/d8/d2a/d42/d6e/dd6/cfe 0 2024-04-22T23:23:51.358 INFO:tasks.workunit.client.1.smithi134.stdout:9/620: rmdir d0/d7/d12/d22 39 2024-04-22T23:23:51.359 INFO:tasks.workunit.client.1.smithi134.stdout:7/713: symlink d4/d8/d2a/d42/d7d/lff 0 2024-04-22T23:23:51.362 INFO:tasks.workunit.client.1.smithi134.stdout:9/621: truncate d0/d7/d23/d26/d4d/f3b 280959 0 2024-04-22T23:23:51.364 INFO:tasks.workunit.client.1.smithi134.stdout:9/622: mknod d0/d2d/d82/dbf/cd2 0 2024-04-22T23:23:51.366 INFO:tasks.workunit.client.1.smithi134.stdout:7/714: creat d4/d8/d41/d76/dc1/f100 x:0 0 0 2024-04-22T23:23:51.368 INFO:tasks.workunit.client.1.smithi134.stdout:7/715: stat d4/d8/d41/d76/dc1/d2f/d70/d86/c94 0 2024-04-22T23:23:51.368 INFO:tasks.workunit.client.1.smithi134.stdout:9/623: mknod d0/d7/d23/d1b/cd3 0 2024-04-22T23:23:51.372 INFO:tasks.workunit.client.1.smithi134.stdout:9/624: unlink d0/d7/d23/ca7 0 2024-04-22T23:23:51.374 INFO:tasks.workunit.client.1.smithi134.stdout:9/625: chown d0/d37/c52 65890458 1 2024-04-22T23:23:51.375 INFO:tasks.workunit.client.1.smithi134.stdout:7/716: dwrite d4/d8/d41/d76/dc1/fa5 [0,4194304] 0 2024-04-22T23:23:51.377 INFO:tasks.workunit.client.1.smithi134.stdout:8/628: dread d7/db/f1f [0,4194304] 0 2024-04-22T23:23:51.381 INFO:tasks.workunit.client.1.smithi134.stdout:9/626: creat d0/d7/d23/d26/d4d/d5d/d8e/dae/db1/fd4 x:0 0 0 2024-04-22T23:23:51.381 INFO:tasks.workunit.client.1.smithi134.stdout:7/717: mkdir d4/d8/d101 0 2024-04-22T23:23:51.387 INFO:tasks.workunit.client.1.smithi134.stdout:8/629: read d7/db/d2d/d39/d58/f96 [3227917,32029] 0 2024-04-22T23:23:51.390 INFO:tasks.workunit.client.1.smithi134.stdout:8/630: dread - d7/dc/d2c/d4f/f9f zero size 2024-04-22T23:23:51.390 INFO:tasks.workunit.client.1.smithi134.stdout:9/627: dwrite d0/d7/d23/d26/d4d/d5d/d8e/dae/db1/fd4 [0,4194304] 0 2024-04-22T23:23:51.392 INFO:tasks.workunit.client.1.smithi134.stdout:9/628: write d0/d37/f95 [2657547,18279] 0 2024-04-22T23:23:51.395 INFO:tasks.workunit.client.1.smithi134.stdout:9/629: truncate d0/d7/d23/f55 1060863 0 2024-04-22T23:23:51.397 INFO:tasks.workunit.client.1.smithi134.stdout:8/631: dread d7/db/d2d/d39/f72 [0,4194304] 0 2024-04-22T23:23:51.399 INFO:tasks.workunit.client.1.smithi134.stdout:9/630: creat d0/d2d/d82/da0/fd5 x:0 0 0 2024-04-22T23:23:51.401 INFO:tasks.workunit.client.1.smithi134.stdout:7/718: rmdir d4 39 2024-04-22T23:23:51.403 INFO:tasks.workunit.client.1.smithi134.stdout:6/698: sync 2024-04-22T23:23:51.404 INFO:tasks.workunit.client.1.smithi134.stdout:8/632: dwrite d7/dc/d2c/d37/f4d [0,4194304] 0 2024-04-22T23:23:51.407 INFO:tasks.workunit.client.1.smithi134.stdout:8/633: dread - d7/dc/d2c/d37/d3a/fc8 zero size 2024-04-22T23:23:51.408 INFO:tasks.workunit.client.1.smithi134.stdout:7/719: chown d4/d8/d41/d76/dc1/d2f/d69/dab/cd5 362549778 1 2024-04-22T23:23:51.409 INFO:tasks.workunit.client.1.smithi134.stdout:8/634: stat d7/dc/d2c/d4f/d6c/d86 0 2024-04-22T23:23:51.409 INFO:tasks.workunit.client.1.smithi134.stdout:9/631: dwrite d0/d37/f4f [0,4194304] 0 2024-04-22T23:23:51.411 INFO:tasks.workunit.client.1.smithi134.stdout:6/699: truncate d0/d19/de/d12/f97 347797 0 2024-04-22T23:23:51.412 INFO:tasks.workunit.client.1.smithi134.stdout:7/720: rmdir d4/d8/d41/d76/dc1/d2f/da6/dcf 39 2024-04-22T23:23:51.413 INFO:tasks.workunit.client.1.smithi134.stdout:6/700: write d0/d19/de/d1f/dd5/fe2 [1005013,61079] 0 2024-04-22T23:23:51.413 INFO:tasks.workunit.client.1.smithi134.stdout:9/632: creat d0/d2d/d4e/fd6 x:0 0 0 2024-04-22T23:23:51.415 INFO:tasks.workunit.client.1.smithi134.stdout:7/721: mknod d4/d8/d2a/d42/d79/c102 0 2024-04-22T23:23:51.417 INFO:tasks.workunit.client.1.smithi134.stdout:7/722: fsync d4/d8/d41/d76/dc1/d2f/d34/d62/d29/dc5/fe0 0 2024-04-22T23:23:51.417 INFO:tasks.workunit.client.1.smithi134.stdout:8/635: link d7/dc/d2c/d4f/d6c/d6e/f6f d7/db/d42/d65/fc9 0 2024-04-22T23:23:51.418 INFO:tasks.workunit.client.1.smithi134.stdout:9/633: symlink d0/d7/d23/d26/d4d/d5d/ld7 0 2024-04-22T23:23:51.419 INFO:tasks.workunit.client.1.smithi134.stdout:7/723: read d4/d8/d41/d76/dc1/fb8 [3569556,27264] 0 2024-04-22T23:23:51.420 INFO:tasks.workunit.client.1.smithi134.stdout:9/634: write d0/d2d/d82/da0/fd5 [499614,86694] 0 2024-04-22T23:23:51.420 INFO:tasks.workunit.client.1.smithi134.stdout:6/701: link d0/d19/lba d0/d19/db/d35/d8a/le6 0 2024-04-22T23:23:51.421 INFO:tasks.workunit.client.1.smithi134.stdout:6/702: chown d0/c2 56463 1 2024-04-22T23:23:51.425 INFO:tasks.workunit.client.1.smithi134.stdout:8/636: link d7/db/d2d/fb0 d7/dc/d2c/d4f/d6c/d6e/fca 0 2024-04-22T23:23:51.426 INFO:tasks.workunit.client.1.smithi134.stdout:6/703: dread d0/d19/de/f13 [0,4194304] 0 2024-04-22T23:23:51.427 INFO:tasks.workunit.client.1.smithi134.stdout:9/635: symlink d0/d2d/d94/d98/ld8 0 2024-04-22T23:23:51.429 INFO:tasks.workunit.client.1.smithi134.stdout:6/704: read d0/d19/de/d12/d29/d74/f89 [663238,56190] 0 2024-04-22T23:23:51.431 INFO:tasks.workunit.client.1.smithi134.stdout:9/636: link d0/d7/f92 d0/d7/d23/d26/d4d/d5d/d8e/dae/db1/fd9 0 2024-04-22T23:23:51.432 INFO:tasks.workunit.client.1.smithi134.stdout:9/637: chown d0/d7/cc8 269437828 1 2024-04-22T23:23:51.433 INFO:tasks.workunit.client.1.smithi134.stdout:9/638: fsync d0/d7/d23/f45 0 2024-04-22T23:23:51.435 INFO:tasks.workunit.client.1.smithi134.stdout:9/639: creat d0/d2d/d3d/fda x:0 0 0 2024-04-22T23:23:51.435 INFO:tasks.workunit.client.1.smithi134.stdout:6/705: dwrite d0/d19/db/d35/d7f/fe4 [0,4194304] 0 2024-04-22T23:23:51.436 INFO:tasks.workunit.client.1.smithi134.stdout:6/706: stat d0/d19/de/d12/c15 0 2024-04-22T23:23:51.441 INFO:tasks.workunit.client.1.smithi134.stdout:9/640: rename d0/f69 to d0/d7/d23/fdb 0 2024-04-22T23:23:51.444 INFO:tasks.workunit.client.1.smithi134.stdout:9/641: mknod d0/cdc 0 2024-04-22T23:23:51.446 INFO:tasks.workunit.client.1.smithi134.stdout:9/642: fdatasync d0/d7/d23/d26/d51/d88/db9/fcb 0 2024-04-22T23:23:51.505 INFO:tasks.workunit.client.1.smithi134.stdout:6/707: read d0/d6f/f7b [876810,110112] 0 2024-04-22T23:23:51.510 INFO:tasks.workunit.client.1.smithi134.stdout:6/708: getdents d0/d4f 0 2024-04-22T23:23:51.510 INFO:tasks.workunit.client.1.smithi134.stdout:6/709: chown d0/f8 3 1 2024-04-22T23:23:51.511 INFO:tasks.workunit.client.1.smithi134.stdout:6/710: stat d0/d19/db/d33/f95 0 2024-04-22T23:23:51.513 INFO:tasks.workunit.client.1.smithi134.stdout:6/711: mknod d0/d19/de/d12/d29/d74/ce7 0 2024-04-22T23:23:51.517 INFO:tasks.workunit.client.1.smithi134.stdout:6/712: rename d0/d19/db/d35/caf to d0/d19/de/d12/d5e/dc1/ce8 0 2024-04-22T23:23:51.521 INFO:tasks.workunit.client.1.smithi134.stdout:6/713: getdents d0/d19/de/d12/d5e/d6d 0 2024-04-22T23:23:51.527 INFO:tasks.workunit.client.1.smithi134.stdout:6/714: dread d0/d19/de/d12/d29/d74/f8e [0,4194304] 0 2024-04-22T23:23:51.531 INFO:tasks.workunit.client.1.smithi134.stdout:7/724: write d4/d8/d41/d76/dc1/faa [326264,57045] 0 2024-04-22T23:23:51.539 INFO:tasks.workunit.client.1.smithi134.stdout:9/643: write d0/d7/d23/d26/d51/d88/fcd [4301374,109139] 0 2024-04-22T23:23:51.539 INFO:tasks.workunit.client.1.smithi134.stdout:8/637: dwrite d7/db/d2d/d39/d58/f71 [0,4194304] 0 2024-04-22T23:23:51.542 INFO:tasks.workunit.client.1.smithi134.stdout:8/638: truncate d7/dc/d2c/d37/d47/f31 444574 0 2024-04-22T23:23:51.558 INFO:tasks.workunit.client.1.smithi134.stdout:1/811: sync 2024-04-22T23:23:51.558 INFO:tasks.workunit.client.1.smithi134.stdout:0/622: sync 2024-04-22T23:23:51.568 INFO:tasks.workunit.client.1.smithi134.stdout:9/644: mknod d0/d37/d76/cdd 0 2024-04-22T23:23:51.569 INFO:tasks.workunit.client.1.smithi134.stdout:8/639: stat d7/db 0 2024-04-22T23:23:51.569 INFO:tasks.workunit.client.1.smithi134.stdout:1/812: chown db/d15 174500 1 2024-04-22T23:23:51.570 INFO:tasks.workunit.client.1.smithi134.stdout:0/623: read d1/d5/d58/f89 [434745,108924] 0 2024-04-22T23:23:51.572 INFO:tasks.workunit.client.1.smithi134.stdout:9/645: symlink d0/d7/d23/d26/d51/lde 0 2024-04-22T23:23:51.575 INFO:tasks.workunit.client.1.smithi134.stdout:6/715: dwrite d0/d19/de/d12/f7e [0,4194304] 0 2024-04-22T23:23:51.577 INFO:tasks.workunit.client.1.smithi134.stdout:0/624: mkdir d1/d5/d58/d22/dd0 0 2024-04-22T23:23:51.581 INFO:tasks.workunit.client.1.smithi134.stdout:1/813: write db/d16/f3d [2855914,106261] 0 2024-04-22T23:23:51.583 INFO:tasks.workunit.client.1.smithi134.stdout:0/625: stat d1/d5/d58/da0/l8d 0 2024-04-22T23:23:51.583 INFO:tasks.workunit.client.1.smithi134.stdout:8/640: dwrite d7/db/d42/f59 [0,4194304] 0 2024-04-22T23:23:51.587 INFO:tasks.workunit.client.1.smithi134.stdout:1/814: creat db/d16/d87/dbd/f113 x:0 0 0 2024-04-22T23:23:51.588 INFO:tasks.workunit.client.1.smithi134.stdout:6/716: getdents d0/d19/de/d12/d5e 0 2024-04-22T23:23:51.594 INFO:tasks.workunit.client.1.smithi134.stdout:1/815: rename db/d15/d17/d20/d2a/d2d/d2e/d42 to db/d15/d17/d20/d2a/d2d/d108/d114 0 2024-04-22T23:23:51.596 INFO:tasks.workunit.client.1.smithi134.stdout:0/626: truncate d1/d5/d58/f80 276930 0 2024-04-22T23:23:51.598 INFO:tasks.workunit.client.1.smithi134.stdout:8/641: dwrite d7/db/f3b [0,4194304] 0 2024-04-22T23:23:51.600 INFO:tasks.workunit.client.1.smithi134.stdout:6/717: dwrite d0/d19/de/d1f/d94/fad [0,4194304] 0 2024-04-22T23:23:51.602 INFO:tasks.workunit.client.1.smithi134.stdout:1/816: read db/f1b [2777399,112455] 0 2024-04-22T23:23:51.603 INFO:tasks.workunit.client.1.smithi134.stdout:0/627: mknod d1/d5/dd/d41/cd1 0 2024-04-22T23:23:51.603 INFO:tasks.workunit.client.1.smithi134.stdout:8/642: read f4 [6389247,6977] 0 2024-04-22T23:23:51.607 INFO:tasks.workunit.client.1.smithi134.stdout:0/628: dread d1/d5/dd/d41/fc8 [0,4194304] 0 2024-04-22T23:23:51.608 INFO:tasks.workunit.client.1.smithi134.stdout:9/646: dread d0/d7/d12/f6a [0,4194304] 0 2024-04-22T23:23:51.613 INFO:tasks.workunit.client.1.smithi134.stdout:6/718: fsync d0/d4f/fcc 0 2024-04-22T23:23:51.615 INFO:tasks.workunit.client.1.smithi134.stdout:6/719: stat d0/d19/de/d12/d29/d74/lb9 0 2024-04-22T23:23:51.618 INFO:tasks.workunit.client.1.smithi134.stdout:9/647: rename d0/d7/d12/l60 to d0/d2d/d82/dbf/ldf 0 2024-04-22T23:23:51.619 INFO:tasks.workunit.client.1.smithi134.stdout:9/648: write d0/d7/d12/fcc [855034,70149] 0 2024-04-22T23:23:51.623 INFO:tasks.workunit.client.1.smithi134.stdout:8/643: creat d7/db/d42/d65/fcb x:0 0 0 2024-04-22T23:23:51.624 INFO:tasks.workunit.client.1.smithi134.stdout:9/649: creat d0/d37/d76/fe0 x:0 0 0 2024-04-22T23:23:51.626 INFO:tasks.workunit.client.1.smithi134.stdout:0/629: link d1/d5/ca5 d1/d5/d15/cd2 0 2024-04-22T23:23:51.628 INFO:tasks.workunit.client.1.smithi134.stdout:0/630: chown d1/d2/d1d 1 1 2024-04-22T23:23:51.629 INFO:tasks.workunit.client.1.smithi134.stdout:8/644: write d7/db/d42/d65/d46/d9a/fa1 [618150,66511] 0 2024-04-22T23:23:51.629 INFO:tasks.workunit.client.1.smithi134.stdout:6/720: dwrite d0/d19/de/d2f/f71 [0,4194304] 0 2024-04-22T23:23:51.633 INFO:tasks.workunit.client.1.smithi134.stdout:1/817: dread db/d16/d56/fa1 [0,4194304] 0 2024-04-22T23:23:51.635 INFO:tasks.workunit.client.1.smithi134.stdout:8/645: creat d7/dc/d19/da5/fcc x:0 0 0 2024-04-22T23:23:51.636 INFO:tasks.workunit.client.1.smithi134.stdout:1/818: write db/d15/d17/d20/d2a/d2d/d108/d114/f9a [2152895,42000] 0 2024-04-22T23:23:51.637 INFO:tasks.workunit.client.1.smithi134.stdout:0/631: mkdir d1/db/d9f/dd3 0 2024-04-22T23:23:51.640 INFO:tasks.workunit.client.1.smithi134.stdout:6/721: dwrite d0/d19/db/d35/f79 [0,4194304] 0 2024-04-22T23:23:51.640 INFO:tasks.workunit.client.1.smithi134.stdout:9/650: link d0/d37/f8b d0/d7/d23/d26/d51/d4c/fe1 0 2024-04-22T23:23:51.645 INFO:tasks.workunit.client.1.smithi134.stdout:8/646: rename d7/dc/d2c/d37/d3a/c53 to d7/dc/d19/da5/ccd 0 2024-04-22T23:23:51.651 INFO:tasks.workunit.client.1.smithi134.stdout:8/647: link d7/db/d2d/d39/d58/f71 d7/dc/d2c/d77/fce 0 2024-04-22T23:23:51.652 INFO:tasks.workunit.client.1.smithi134.stdout:1/819: dwrite db/d16/dbc/de1/df4/ffd [0,4194304] 0 2024-04-22T23:23:51.654 INFO:tasks.workunit.client.1.smithi134.stdout:8/648: chown d7/db/d42/d65/l6a 4608 1 2024-04-22T23:23:51.656 INFO:tasks.workunit.client.1.smithi134.stdout:1/820: fdatasync db/d16/d56/f57 0 2024-04-22T23:23:51.658 INFO:tasks.workunit.client.1.smithi134.stdout:0/632: dread d1/d2/d1d/f5f [4194304,4194304] 0 2024-04-22T23:23:51.659 INFO:tasks.workunit.client.1.smithi134.stdout:1/821: readlink db/d15/d17/d20/d2a/d2d/d2e/lbb 0 2024-04-22T23:23:51.661 INFO:tasks.workunit.client.1.smithi134.stdout:1/822: chown db/db0/dc1/le5 289 1 2024-04-22T23:23:51.662 INFO:tasks.workunit.client.1.smithi134.stdout:8/649: symlink d7/dc/d19/lcf 0 2024-04-22T23:23:51.663 INFO:tasks.workunit.client.1.smithi134.stdout:0/633: mknod d1/d5/d58/d6b/cd4 0 2024-04-22T23:23:51.668 INFO:tasks.workunit.client.1.smithi134.stdout:8/650: creat d7/db/d42/d65/d46/d9a/da3/d88/fd0 x:0 0 0 2024-04-22T23:23:51.668 INFO:tasks.workunit.client.1.smithi134.stdout:9/651: dread d0/d7/fd [0,4194304] 0 2024-04-22T23:23:51.670 INFO:tasks.workunit.client.1.smithi134.stdout:1/823: dwrite db/d16/d87/fef [0,4194304] 0 2024-04-22T23:23:51.671 INFO:tasks.workunit.client.1.smithi134.stdout:1/824: readlink db/l4f 0 2024-04-22T23:23:51.674 INFO:tasks.workunit.client.1.smithi134.stdout:8/651: mkdir d7/dc/d2c/d4f/d6c/dd1 0 2024-04-22T23:23:51.679 INFO:tasks.workunit.client.1.smithi134.stdout:8/652: dread - d7/db/d42/d65/f89 zero size 2024-04-22T23:23:51.679 INFO:tasks.workunit.client.1.smithi134.stdout:9/652: dread d0/d7/d23/d26/d4d/d5d/d8e/dae/db1/fd9 [0,4194304] 0 2024-04-22T23:23:51.679 INFO:tasks.workunit.client.1.smithi134.stdout:6/722: dread d0/d19/de/d12/d2a/d2d/f49 [0,4194304] 0 2024-04-22T23:23:51.680 INFO:tasks.workunit.client.1.smithi134.stdout:6/723: chown d0/d19/db/d35/ce1 105378 1 2024-04-22T23:23:51.680 INFO:tasks.workunit.client.1.smithi134.stdout:1/825: rename db/d16/d56/cbf to db/d15/d17/c115 0 2024-04-22T23:23:51.682 INFO:tasks.workunit.client.1.smithi134.stdout:6/724: write d0/d19/de/d12/d29/d74/da0/fc8 [950401,23644] 0 2024-04-22T23:23:51.683 INFO:tasks.workunit.client.1.smithi134.stdout:8/653: symlink d7/dc/d19/da5/ld2 0 2024-04-22T23:23:51.684 INFO:tasks.workunit.client.1.smithi134.stdout:6/725: write d0/d19/db/d35/d7f/fe4 [4144110,38871] 0 2024-04-22T23:23:51.684 INFO:tasks.workunit.client.1.smithi134.stdout:0/634: truncate d1/d5/d58/d6b/db9/f18 3816050 0 2024-04-22T23:23:51.686 INFO:tasks.workunit.client.1.smithi134.stdout:1/826: mknod db/d15/d17/d20/d2a/d2d/d86/dad/c116 0 2024-04-22T23:23:51.688 INFO:tasks.workunit.client.1.smithi134.stdout:8/654: mknod d7/db/d2d/d39/d58/d8a/cd3 0 2024-04-22T23:23:51.689 INFO:tasks.workunit.client.1.smithi134.stdout:8/655: readlink d7/db/d42/d65/l36 0 2024-04-22T23:23:51.690 INFO:tasks.workunit.client.1.smithi134.stdout:1/827: unlink db/d15/d17/d20/d2a/d2d/d2e/f47 0 2024-04-22T23:23:51.692 INFO:tasks.workunit.client.1.smithi134.stdout:0/635: rename d1/c4f to d1/d5/dd/d68/cd5 0 2024-04-22T23:23:51.693 INFO:tasks.workunit.client.1.smithi134.stdout:8/656: fsync f4 0 2024-04-22T23:23:51.693 INFO:tasks.workunit.client.1.smithi134.stdout:9/653: getdents d0/d7/d27 0 2024-04-22T23:23:51.694 INFO:tasks.workunit.client.1.smithi134.stdout:1/828: rename db to db/d15/d17/d20/d2a/d2d/d2e/d117 22 2024-04-22T23:23:51.695 INFO:tasks.workunit.client.1.smithi134.stdout:8/657: chown d7/db/d42/d65/d46/d9a/fa1 329 1 2024-04-22T23:23:51.696 INFO:tasks.workunit.client.1.smithi134.stdout:9/654: read d0/d2d/d3d/f9e [160632,29674] 0 2024-04-22T23:23:51.696 INFO:tasks.workunit.client.1.smithi134.stdout:0/636: truncate d1/d5/f50 165514 0 2024-04-22T23:23:51.697 INFO:tasks.workunit.client.1.smithi134.stdout:1/829: rmdir db/d16/dbc 39 2024-04-22T23:23:51.697 INFO:tasks.workunit.client.1.smithi134.stdout:9/655: write d0/d7/d23/f55 [1445361,122506] 0 2024-04-22T23:23:51.698 INFO:tasks.workunit.client.1.smithi134.stdout:0/637: readlink d1/d5/d58/d6b/db9/d75/laf 0 2024-04-22T23:23:51.701 INFO:tasks.workunit.client.1.smithi134.stdout:1/830: read - db/d15/d17/d20/d2a/d2d/d86/dad/fe7 zero size 2024-04-22T23:23:51.704 INFO:tasks.workunit.client.1.smithi134.stdout:8/658: dread d7/dc/d2c/d4f/d6c/f81 [0,4194304] 0 2024-04-22T23:23:51.705 INFO:tasks.workunit.client.1.smithi134.stdout:8/659: write d7/dc/f9c [2293570,115798] 0 2024-04-22T23:23:51.706 INFO:tasks.workunit.client.1.smithi134.stdout:1/831: creat db/db0/dc1/f118 x:0 0 0 2024-04-22T23:23:51.706 INFO:tasks.workunit.client.1.smithi134.stdout:2/571: sync 2024-04-22T23:23:51.706 INFO:tasks.workunit.client.1.smithi134.stdout:4/826: sync 2024-04-22T23:23:51.710 INFO:tasks.workunit.client.1.smithi134.stdout:6/726: dread d0/d19/de/d1f/f22 [4194304,4194304] 0 2024-04-22T23:23:51.710 INFO:tasks.workunit.client.1.smithi134.stdout:9/656: mkdir d0/d7/d23/d1b/de2 0 2024-04-22T23:23:51.711 INFO:tasks.workunit.client.1.smithi134.stdout:6/727: fsync d0/d19/de/d12/d2a/dbb/fdf 0 2024-04-22T23:23:51.711 INFO:tasks.workunit.client.1.smithi134.stdout:0/638: mknod d1/d5/dd/d68/dc0/cd6 0 2024-04-22T23:23:51.717 INFO:tasks.workunit.client.1.smithi134.stdout:2/572: creat d2/d42/d54/d33/d6e/fbf x:0 0 0 2024-04-22T23:23:51.719 INFO:tasks.workunit.client.1.smithi134.stdout:2/573: read d2/d42/f46 [10050994,121446] 0 2024-04-22T23:23:51.723 INFO:tasks.workunit.client.1.smithi134.stdout:8/660: write d7/dc/d2c/d37/d47/f66 [369687,54740] 0 2024-04-22T23:23:51.725 INFO:tasks.workunit.client.1.smithi134.stdout:1/832: dread db/d15/d17/f5a [0,4194304] 0 2024-04-22T23:23:51.725 INFO:tasks.workunit.client.1.smithi134.stdout:2/574: symlink d2/dd/d4a/d73/lc0 0 2024-04-22T23:23:51.726 INFO:tasks.workunit.client.1.smithi134.stdout:1/833: readlink db/l13 0 2024-04-22T23:23:51.727 INFO:tasks.workunit.client.1.smithi134.stdout:6/728: mkdir d0/d19/de/dbe/dd1/de9 0 2024-04-22T23:23:51.729 INFO:tasks.workunit.client.1.smithi134.stdout:6/729: readlink d0/d19/de/d12/d29/d74/lb9 0 2024-04-22T23:23:51.729 INFO:tasks.workunit.client.1.smithi134.stdout:4/827: dread d4/d29/dcc/d82/fb2 [0,4194304] 0 2024-04-22T23:23:51.730 INFO:tasks.workunit.client.1.smithi134.stdout:4/828: fsync d4/d35/d8e/f75 0 2024-04-22T23:23:51.730 INFO:tasks.workunit.client.1.smithi134.stdout:2/575: creat d2/fc1 x:0 0 0 2024-04-22T23:23:51.732 INFO:tasks.workunit.client.1.smithi134.stdout:0/639: write d1/d5/dd/d20/f1e [3094288,62807] 0 2024-04-22T23:23:51.733 INFO:tasks.workunit.client.1.smithi134.stdout:0/640: write d1/d5/d58/da0/f7b [4026012,43772] 0 2024-04-22T23:23:51.735 INFO:tasks.workunit.client.1.smithi134.stdout:0/641: readlink d1/d2/d1d/l86 0 2024-04-22T23:23:51.735 INFO:tasks.workunit.client.1.smithi134.stdout:0/642: chown d1/d5/dd/f40 61394 1 2024-04-22T23:23:51.737 INFO:tasks.workunit.client.1.smithi134.stdout:6/730: rename d0/d19/de/d12/d5e/dd6/fe0 to d0/d19/de/d12/d5e/d6d/d7a/d99/fea 0 2024-04-22T23:23:51.737 INFO:tasks.workunit.client.1.smithi134.stdout:6/731: fdatasync d0/d6f/f84 0 2024-04-22T23:23:51.739 INFO:tasks.workunit.client.1.smithi134.stdout:6/732: chown d0/d19/de/d12/d29/d74/f89 2432 1 2024-04-22T23:23:51.741 INFO:tasks.workunit.client.1.smithi134.stdout:6/733: mkdir d0/d19/de/d12/d29/d74/deb 0 2024-04-22T23:23:51.743 INFO:tasks.workunit.client.1.smithi134.stdout:1/834: write db/d15/f54 [806759,130171] 0 2024-04-22T23:23:51.746 INFO:tasks.workunit.client.1.smithi134.stdout:6/734: chown d0/d19/de/c9b 7034 1 2024-04-22T23:23:51.750 INFO:tasks.workunit.client.1.smithi134.stdout:6/735: creat d0/d19/db/fec x:0 0 0 2024-04-22T23:23:51.753 INFO:tasks.workunit.client.1.smithi134.stdout:0/643: truncate d1/d5/dd/d20/d2c/f4b 38541 0 2024-04-22T23:23:51.753 INFO:tasks.workunit.client.1.smithi134.stdout:4/829: dread d4/d13/f4c [0,4194304] 0 2024-04-22T23:23:51.754 INFO:tasks.workunit.client.1.smithi134.stdout:1/835: link db/d15/d17/d20/d2a/d2d/d86/dad/cc0 db/d15/d17/d20/d2a/d2d/d86/c119 0 2024-04-22T23:23:51.754 INFO:tasks.workunit.client.1.smithi134.stdout:2/576: dwrite d2/dd/d4a/d73/f7f [0,4194304] 0 2024-04-22T23:23:51.755 INFO:tasks.workunit.client.1.smithi134.stdout:6/736: symlink d0/d19/de/d12/d5e/d6d/dd3/led 0 2024-04-22T23:23:51.756 INFO:tasks.workunit.client.1.smithi134.stdout:6/737: stat d0/d6f/f9e 0 2024-04-22T23:23:51.756 INFO:tasks.workunit.client.1.smithi134.stdout:6/738: readlink d0/d6f/la4 0 2024-04-22T23:23:51.758 INFO:tasks.workunit.client.1.smithi134.stdout:0/644: mknod d1/db/d9f/cd7 0 2024-04-22T23:23:51.758 INFO:tasks.workunit.client.1.smithi134.stdout:6/739: chown d0/d19/de/d12/d5e/d6d/d7a/d99/fb1 350643733 1 2024-04-22T23:23:51.759 INFO:tasks.workunit.client.1.smithi134.stdout:1/836: creat db/d16/d104/f11a x:0 0 0 2024-04-22T23:23:51.764 INFO:tasks.workunit.client.1.smithi134.stdout:2/577: mkdir d2/dd/df/dc2 0 2024-04-22T23:23:51.764 INFO:tasks.workunit.client.1.smithi134.stdout:4/830: dwrite d4/d7/d32/d84/fc0 [0,4194304] 0 2024-04-22T23:23:51.768 INFO:tasks.workunit.client.1.smithi134.stdout:2/578: truncate d2/f87 216979 0 2024-04-22T23:23:51.770 INFO:tasks.workunit.client.1.smithi134.stdout:1/837: creat db/d15/d17/f11b x:0 0 0 2024-04-22T23:23:51.772 INFO:tasks.workunit.client.1.smithi134.stdout:1/838: write db/d16/d104/f11a [556133,82199] 0 2024-04-22T23:23:51.773 INFO:tasks.workunit.client.1.smithi134.stdout:2/579: symlink d2/dd/d4a/d73/lc3 0 2024-04-22T23:23:51.773 INFO:tasks.workunit.client.1.smithi134.stdout:6/740: link d0/d19/de/d12/d2a/d2d/d3a/f9a d0/d19/db/d35/d7f/fee 0 2024-04-22T23:23:51.778 INFO:tasks.workunit.client.1.smithi134.stdout:6/741: creat d0/d19/de/d1f/d7d/fef x:0 0 0 2024-04-22T23:23:51.779 INFO:tasks.workunit.client.1.smithi134.stdout:2/580: getdents d2/dd/df/dc2 0 2024-04-22T23:23:51.781 INFO:tasks.workunit.client.1.smithi134.stdout:2/581: read d2/dd/df/f85 [1721412,94976] 0 2024-04-22T23:23:51.789 INFO:tasks.workunit.client.1.smithi134.stdout:2/582: dwrite d2/fc1 [0,4194304] 0 2024-04-22T23:23:51.793 INFO:tasks.workunit.client.1.smithi134.stdout:0/645: dread d1/d5/d58/d6b/db9/f18 [0,4194304] 0 2024-04-22T23:23:51.794 INFO:tasks.workunit.client.1.smithi134.stdout:1/839: truncate db/f27 11808897 0 2024-04-22T23:23:51.796 INFO:tasks.workunit.client.1.smithi134.stdout:2/583: symlink d2/dd/d4a/d73/lc4 0 2024-04-22T23:23:51.798 INFO:tasks.workunit.client.1.smithi134.stdout:4/831: dread d4/f1d [0,4194304] 0 2024-04-22T23:23:51.798 INFO:tasks.workunit.client.1.smithi134.stdout:6/742: dread d0/d19/f9 [4194304,4194304] 0 2024-04-22T23:23:51.800 INFO:tasks.workunit.client.1.smithi134.stdout:1/840: creat db/d16/d56/f11c x:0 0 0 2024-04-22T23:23:51.801 INFO:tasks.workunit.client.1.smithi134.stdout:6/743: write d0/d19/de/d12/d29/d74/f89 [4409279,76436] 0 2024-04-22T23:23:51.802 INFO:tasks.workunit.client.1.smithi134.stdout:0/646: rename d1/d77/f8c to d1/d77/fd8 0 2024-04-22T23:23:51.803 INFO:tasks.workunit.client.1.smithi134.stdout:0/647: fsync d1/d5/d58/d22/fbb 0 2024-04-22T23:23:51.803 INFO:tasks.workunit.client.1.smithi134.stdout:2/584: mknod d2/d15/d24/d3b/d8a/cc5 0 2024-04-22T23:23:51.804 INFO:tasks.workunit.client.1.smithi134.stdout:1/841: chown db/d15/d17/d20/d2a/d2d/d86/dad/cc0 614167 1 2024-04-22T23:23:51.805 INFO:tasks.workunit.client.1.smithi134.stdout:1/842: stat db/d16/f78 0 2024-04-22T23:23:51.805 INFO:tasks.workunit.client.1.smithi134.stdout:6/744: creat d0/d19/de/dbe/dd1/ff0 x:0 0 0 2024-04-22T23:23:51.806 INFO:tasks.workunit.client.1.smithi134.stdout:1/843: readlink db/d15/d17/d20/d2a/d2d/d2e/l5d 0 2024-04-22T23:23:51.811 INFO:tasks.workunit.client.1.smithi134.stdout:2/585: rmdir d2/dd/d68 39 2024-04-22T23:23:51.812 INFO:tasks.workunit.client.1.smithi134.stdout:6/745: dwrite d0/d19/de/f31 [0,4194304] 0 2024-04-22T23:23:51.813 INFO:tasks.workunit.client.1.smithi134.stdout:4/832: creat d4/d7/d32/f102 x:0 0 0 2024-04-22T23:23:51.815 INFO:tasks.workunit.client.1.smithi134.stdout:6/746: chown d0/d19/de/d1f/dd5/fda 62708 1 2024-04-22T23:23:51.815 INFO:tasks.workunit.client.1.smithi134.stdout:1/844: mknod db/d15/d17/d20/d2a/d2d/d86/c11d 0 2024-04-22T23:23:51.816 INFO:tasks.workunit.client.1.smithi134.stdout:0/648: rmdir d1 39 2024-04-22T23:23:51.819 INFO:tasks.workunit.client.1.smithi134.stdout:2/586: dwrite d2/dd/df/fb6 [0,4194304] 0 2024-04-22T23:23:51.822 INFO:tasks.workunit.client.1.smithi134.stdout:4/833: symlink d4/d6b/dd9/dea/l103 0 2024-04-22T23:23:51.826 INFO:tasks.workunit.client.1.smithi134.stdout:4/834: dread d4/d13/f4c [0,4194304] 0 2024-04-22T23:23:51.826 INFO:tasks.workunit.client.1.smithi134.stdout:0/649: unlink d1/d5/dd/f2a 0 2024-04-22T23:23:51.827 INFO:tasks.workunit.client.1.smithi134.stdout:4/835: read - d4/d29/dcc/d82/d9a/faa zero size 2024-04-22T23:23:51.828 INFO:tasks.workunit.client.1.smithi134.stdout:6/747: creat d0/d19/de/d12/d29/d74/deb/ff1 x:0 0 0 2024-04-22T23:23:51.834 INFO:tasks.workunit.client.1.smithi134.stdout:0/650: dwrite d1/f44 [0,4194304] 0 2024-04-22T23:23:51.838 INFO:tasks.workunit.client.1.smithi134.stdout:2/587: truncate d2/d42/f46 10723905 0 2024-04-22T23:23:51.839 INFO:tasks.workunit.client.1.smithi134.stdout:2/588: read - d2/dd/df/fbb zero size 2024-04-22T23:23:51.839 INFO:tasks.workunit.client.1.smithi134.stdout:0/651: dread d1/d5/d58/d22/fb3 [0,4194304] 0 2024-04-22T23:23:51.840 INFO:tasks.workunit.client.1.smithi134.stdout:6/748: creat d0/d19/de/d12/d5e/dc1/ff2 x:0 0 0 2024-04-22T23:23:51.841 INFO:tasks.workunit.client.1.smithi134.stdout:2/589: write d2/dd/d77/fa3 [4458815,77361] 0 2024-04-22T23:23:51.841 INFO:tasks.workunit.client.1.smithi134.stdout:1/845: getdents db/d16/dbc 0 2024-04-22T23:23:51.844 INFO:tasks.workunit.client.1.smithi134.stdout:6/749: chown d0/d19/de/d12/d29/d74/f8e 51977 1 2024-04-22T23:23:51.851 INFO:tasks.workunit.client.1.smithi134.stdout:2/590: dread d2/dd/df/f97 [0,4194304] 0 2024-04-22T23:23:51.854 INFO:tasks.workunit.client.1.smithi134.stdout:1/846: chown db/d15/d17/d20/d2a/d2d/c49 14123 1 2024-04-22T23:23:51.861 INFO:tasks.workunit.client.1.smithi134.stdout:6/750: symlink d0/d19/de/d12/d5e/d6d/d7a/d99/lf3 0 2024-04-22T23:23:51.867 INFO:tasks.workunit.client.1.smithi134.stdout:2/591: dwrite d2/d15/d24/d1d/f36 [0,4194304] 0 2024-04-22T23:23:51.868 INFO:tasks.workunit.client.1.smithi134.stdout:4/836: truncate d4/dd/f6d 92233 0 2024-04-22T23:23:51.869 INFO:tasks.workunit.client.1.smithi134.stdout:6/751: write d0/d19/de/d12/d2a/d2d/f85 [5122036,79978] 0 2024-04-22T23:23:51.870 INFO:tasks.workunit.client.1.smithi134.stdout:4/837: stat d4/d7/d32/c8c 0 2024-04-22T23:23:51.882 INFO:tasks.workunit.client.1.smithi134.stdout:3/497: sync 2024-04-22T23:23:51.882 INFO:tasks.workunit.client.1.smithi134.stdout:7/725: sync 2024-04-22T23:23:51.882 INFO:tasks.workunit.client.1.smithi134.stdout:5/766: sync 2024-04-22T23:23:51.883 INFO:tasks.workunit.client.1.smithi134.stdout:3/498: chown dc/d2a/f23 13164835 1 2024-04-22T23:23:51.884 INFO:tasks.workunit.client.1.smithi134.stdout:5/767: readlink d9/dd/d17/l6c 0 2024-04-22T23:23:51.884 INFO:tasks.workunit.client.1.smithi134.stdout:4/838: mkdir d4/d7/d32/d3a/d93/d104 0 2024-04-22T23:23:51.887 INFO:tasks.workunit.client.1.smithi134.stdout:6/752: rename d0/d6f/d88/d8b/dc5 to d0/d19/db/d35/d8a/df4 0 2024-04-22T23:23:51.887 INFO:tasks.workunit.client.1.smithi134.stdout:3/499: chown dc/d2a/d62/db2/d2f/d21/d42/f82 7180 1 2024-04-22T23:23:51.889 INFO:tasks.workunit.client.1.smithi134.stdout:0/652: link d1/d5/f50 d1/d5/d58/d22/dcb/fd9 0 2024-04-22T23:23:51.890 INFO:tasks.workunit.client.1.smithi134.stdout:6/753: stat d0/d19/db/d35/dbc/fc9 0 2024-04-22T23:23:51.892 INFO:tasks.workunit.client.1.smithi134.stdout:3/500: stat dc/f22 0 2024-04-22T23:23:51.903 INFO:tasks.workunit.client.1.smithi134.stdout:5/768: dwrite d9/dd/fbe [8388608,4194304] 0 2024-04-22T23:23:51.904 INFO:tasks.workunit.client.1.smithi134.stdout:6/754: rmdir d0/d19/de/d1f 39 2024-04-22T23:23:51.905 INFO:tasks.workunit.client.1.smithi134.stdout:2/592: link d2/d42/d54/l3c d2/dd/df/d89/lc6 0 2024-04-22T23:23:51.907 INFO:tasks.workunit.client.1.smithi134.stdout:1/847: link db/d16/d56/f83 db/d15/d17/d20/d2a/d2d/d86/f11e 0 2024-04-22T23:23:51.910 INFO:tasks.workunit.client.1.smithi134.stdout:0/653: dwrite d1/d5/dd/f35 [0,4194304] 0 2024-04-22T23:23:51.913 INFO:tasks.workunit.client.1.smithi134.stdout:5/769: write d9/dd/d88/db7/fbf [956982,7688] 0 2024-04-22T23:23:51.913 INFO:tasks.workunit.client.1.smithi134.stdout:7/726: symlink d4/d8/d41/d76/dc1/d2f/d69/dab/d95/dec/df4/l103 0 2024-04-22T23:23:51.914 INFO:tasks.workunit.client.1.smithi134.stdout:1/848: truncate db/d15/d17/d20/d2a/d2d/d2e/f90 533643 0 2024-04-22T23:23:51.919 INFO:tasks.workunit.client.1.smithi134.stdout:3/501: mknod dc/d2a/d62/db2/d2f/d36/cb9 0 2024-04-22T23:23:51.923 INFO:tasks.workunit.client.1.smithi134.stdout:1/849: rmdir db/d16/dbc/de1 39 2024-04-22T23:23:51.929 INFO:tasks.workunit.client.1.smithi134.stdout:6/755: mkdir d0/d19/de/d12/d2a/d2d/d3a/d98/df5 0 2024-04-22T23:23:51.931 INFO:tasks.workunit.client.1.smithi134.stdout:1/850: mknod db/d16/d87/c11f 0 2024-04-22T23:23:51.935 INFO:tasks.workunit.client.1.smithi134.stdout:6/756: mknod d0/d19/de/d12/d5e/dd6/cf6 0 2024-04-22T23:23:51.935 INFO:tasks.workunit.client.1.smithi134.stdout:1/851: truncate db/d15/d17/d20/d2a/d2d/d2e/f90 1460388 0 2024-04-22T23:23:51.937 INFO:tasks.workunit.client.1.smithi134.stdout:1/852: stat db/d15/d17/d20/d111 0 2024-04-22T23:23:51.945 INFO:tasks.workunit.client.1.smithi134.stdout:2/593: dread d2/dd/d77/f7e [0,4194304] 0 2024-04-22T23:23:51.946 INFO:tasks.workunit.client.1.smithi134.stdout:5/770: getdents d9/dd/d17/d2a/d49 0 2024-04-22T23:23:51.948 INFO:tasks.workunit.client.1.smithi134.stdout:2/594: write d2/d15/d24/f1e [2156545,61813] 0 2024-04-22T23:23:51.955 INFO:tasks.workunit.client.1.smithi134.stdout:6/757: rmdir d0/d19/de/d12/d5e/dd6 39 2024-04-22T23:23:51.955 INFO:tasks.workunit.client.1.smithi134.stdout:4/839: dwrite d4/d29/dcc/f7f [0,4194304] 0 2024-04-22T23:23:51.957 INFO:tasks.workunit.client.1.smithi134.stdout:0/654: dwrite d1/d2/fce [0,4194304] 0 2024-04-22T23:23:51.958 INFO:tasks.workunit.client.1.smithi134.stdout:2/595: mkdir d2/d15/d24/d3b/d8a/dc7 0 2024-04-22T23:23:51.959 INFO:tasks.workunit.client.1.smithi134.stdout:3/502: write dc/d2a/d62/db2/d34/d72/fa1 [1249076,13799] 0 2024-04-22T23:23:51.959 INFO:tasks.workunit.client.1.smithi134.stdout:1/853: truncate db/fb9 2274806 0 2024-04-22T23:23:51.961 INFO:tasks.workunit.client.1.smithi134.stdout:2/596: write d2/d15/d24/f6d [79057,63546] 0 2024-04-22T23:23:51.962 INFO:tasks.workunit.client.1.smithi134.stdout:0/655: readlink d1/d5/dd/d68/l8e 0 2024-04-22T23:23:51.962 INFO:tasks.workunit.client.1.smithi134.stdout:4/840: readlink d4/dd/da8/dc2/df8/l19 0 2024-04-22T23:23:51.962 INFO:tasks.workunit.client.1.smithi134.stdout:6/758: mknod d0/d19/de/d2f/cf7 0 2024-04-22T23:23:51.964 INFO:tasks.workunit.client.1.smithi134.stdout:0/656: chown d1/db/d9f/dd3 127 1 2024-04-22T23:23:51.967 INFO:tasks.workunit.client.1.smithi134.stdout:1/854: creat db/d15/d17/d20/d2a/d2d/d86/f120 x:0 0 0 2024-04-22T23:23:51.968 INFO:tasks.workunit.client.1.smithi134.stdout:1/855: chown db/db5/fdb 0 1 2024-04-22T23:23:51.968 INFO:tasks.workunit.client.1.smithi134.stdout:2/597: creat d2/d15/d24/d1d/d3d/fc8 x:0 0 0 2024-04-22T23:23:51.971 INFO:tasks.workunit.client.1.smithi134.stdout:4/841: creat d4/d7/d32/d3a/d93/db5/f105 x:0 0 0 2024-04-22T23:23:51.973 INFO:tasks.workunit.client.1.smithi134.stdout:5/771: dread d9/fa [0,4194304] 0 2024-04-22T23:23:51.975 INFO:tasks.workunit.client.1.smithi134.stdout:2/598: mkdir d2/dd/dc9 0 2024-04-22T23:23:51.976 INFO:tasks.workunit.client.1.smithi134.stdout:0/657: rmdir d1/db/d9f/dd3 0 2024-04-22T23:23:51.977 INFO:tasks.workunit.client.1.smithi134.stdout:0/658: fsync d1/d5/dd/f35 0 2024-04-22T23:23:51.989 INFO:tasks.workunit.client.1.smithi134.stdout:0/659: link d1/d5/dd/d20/d2c/f4b d1/db/d9f/fda 0 2024-04-22T23:23:51.989 INFO:tasks.workunit.client.1.smithi134.stdout:3/503: dread dc/d2a/d62/db2/f3a [0,4194304] 0 2024-04-22T23:23:51.990 INFO:tasks.workunit.client.1.smithi134.stdout:0/660: stat d1/d2/d1d 0 2024-04-22T23:23:51.992 INFO:tasks.workunit.client.1.smithi134.stdout:3/504: creat dc/d2a/d62/d86/fba x:0 0 0 2024-04-22T23:23:51.996 INFO:tasks.workunit.client.1.smithi134.stdout:2/599: dread d2/dd/f37 [4194304,4194304] 0 2024-04-22T23:23:51.998 INFO:tasks.workunit.client.1.smithi134.stdout:3/505: creat dc/d2a/d62/db2/d2f/d36/d6d/d80/fbb x:0 0 0 2024-04-22T23:23:51.998 INFO:tasks.workunit.client.1.smithi134.stdout:1/856: truncate db/f5f 865060 0 2024-04-22T23:23:52.001 INFO:tasks.workunit.client.1.smithi134.stdout:2/600: fsync d2/d42/d54/d33/f3a 0 2024-04-22T23:23:52.001 INFO:tasks.workunit.client.1.smithi134.stdout:6/759: truncate d0/d19/de/d1f/f2c 1039571 0 2024-04-22T23:23:52.003 INFO:tasks.workunit.client.1.smithi134.stdout:5/772: dwrite d9/dd/d17/d2a/f47 [0,4194304] 0 2024-04-22T23:23:52.009 INFO:tasks.workunit.client.1.smithi134.stdout:5/773: chown d9/dd/df1/f101 202581523 1 2024-04-22T23:23:52.012 INFO:tasks.workunit.client.1.smithi134.stdout:4/842: dread d4/d29/dcc/f67 [0,4194304] 0 2024-04-22T23:23:52.012 INFO:tasks.workunit.client.1.smithi134.stdout:5/774: fsync d9/dd/d17/f71 0 2024-04-22T23:23:52.013 INFO:tasks.workunit.client.1.smithi134.stdout:2/601: mkdir d2/dd/df/d9c/dca 0 2024-04-22T23:23:52.015 INFO:tasks.workunit.client.1.smithi134.stdout:6/760: creat d0/d19/de/d12/d29/d74/deb/ff8 x:0 0 0 2024-04-22T23:23:52.020 INFO:tasks.workunit.client.1.smithi134.stdout:0/661: dwrite d1/d5/d58/d6b/db9/f18 [4194304,4194304] 0 2024-04-22T23:23:52.020 INFO:tasks.workunit.client.1.smithi134.stdout:4/843: unlink d4/d7/d32/fbf 0 2024-04-22T23:23:52.023 INFO:tasks.workunit.client.1.smithi134.stdout:2/602: dwrite d2/dd/d72/fae [0,4194304] 0 2024-04-22T23:23:52.025 INFO:tasks.workunit.client.1.smithi134.stdout:2/603: fsync d2/d15/d24/f29 0 2024-04-22T23:23:52.026 INFO:tasks.workunit.client.1.smithi134.stdout:2/604: write d2/d15/d24/f2a [7367608,125082] 0 2024-04-22T23:23:52.027 INFO:tasks.workunit.client.1.smithi134.stdout:2/605: write d2/dd/df/fb6 [1034161,100842] 0 2024-04-22T23:23:52.035 INFO:tasks.workunit.client.1.smithi134.stdout:3/506: truncate dc/d2a/d62/db2/d3b/f79 1274421 0 2024-04-22T23:23:52.036 INFO:tasks.workunit.client.1.smithi134.stdout:6/761: symlink d0/d19/de/d12/d5e/d6d/d7a/db2/lf9 0 2024-04-22T23:23:52.038 INFO:tasks.workunit.client.1.smithi134.stdout:1/857: truncate db/d15/d17/d20/d2a/d2d/f48 1507268 0 2024-04-22T23:23:52.039 INFO:tasks.workunit.client.1.smithi134.stdout:1/858: chown db/d16/fda 191833168 1 2024-04-22T23:23:52.039 INFO:tasks.workunit.client.1.smithi134.stdout:0/662: creat d1/d5/dd/d41/fdb x:0 0 0 2024-04-22T23:23:52.044 INFO:tasks.workunit.client.1.smithi134.stdout:5/775: link d9/d1e/d90/fdc d9/d11/d34/d85/f107 0 2024-04-22T23:23:52.045 INFO:tasks.workunit.client.1.smithi134.stdout:3/507: mknod dc/d2a/d62/db2/d34/d6b/cbc 0 2024-04-22T23:23:52.046 INFO:tasks.workunit.client.1.smithi134.stdout:6/762: mkdir d0/d19/db/dfa 0 2024-04-22T23:23:52.047 INFO:tasks.workunit.client.1.smithi134.stdout:6/763: write d0/d4f/fb3 [59953,1095] 0 2024-04-22T23:23:52.048 INFO:tasks.workunit.client.1.smithi134.stdout:4/844: rename d4/c24 to d4/d35/d8e/d5d/c106 0 2024-04-22T23:23:52.051 INFO:tasks.workunit.client.1.smithi134.stdout:0/663: creat d1/d2/d1d/d72/fdc x:0 0 0 2024-04-22T23:23:52.052 INFO:tasks.workunit.client.1.smithi134.stdout:0/664: chown d1/d5/d58/f46 3530366 1 2024-04-22T23:23:52.052 INFO:tasks.workunit.client.1.smithi134.stdout:2/606: mkdir d2/d15/d24/d1d/dba/dcb 0 2024-04-22T23:23:52.053 INFO:tasks.workunit.client.1.smithi134.stdout:5/776: mkdir d9/d11/d73/d95/d20/d108 0 2024-04-22T23:23:52.060 INFO:tasks.workunit.client.1.smithi134.stdout:5/777: dwrite d9/d11/d34/d4d/f9f [0,4194304] 0 2024-04-22T23:23:52.061 INFO:tasks.workunit.client.1.smithi134.stdout:3/508: unlink dc/d2a/d62/db2/d2f/d36/d6d/d80/f93 0 2024-04-22T23:23:52.064 INFO:tasks.workunit.client.1.smithi134.stdout:6/764: mknod d0/d19/de/dbe/dd1/cfb 0 2024-04-22T23:23:52.066 INFO:tasks.workunit.client.1.smithi134.stdout:4/845: mknod d4/dd/da8/dc2/df8/d25/dfd/c107 0 2024-04-22T23:23:52.067 INFO:tasks.workunit.client.1.smithi134.stdout:4/846: readlink d4/d35/d53/l5b 0 2024-04-22T23:23:52.067 INFO:tasks.workunit.client.1.smithi134.stdout:6/765: write d0/d19/de/d12/d5e/d6d/d7a/db2/fbf [42959,83721] 0 2024-04-22T23:23:52.069 INFO:tasks.workunit.client.1.smithi134.stdout:4/847: write d4/d29/dcc/f26 [2120437,25221] 0 2024-04-22T23:23:52.070 INFO:tasks.workunit.client.1.smithi134.stdout:6/766: fsync d0/d19/de/d12/d29/d74/da0/fd9 0 2024-04-22T23:23:52.071 INFO:tasks.workunit.client.1.smithi134.stdout:4/848: truncate d4/d35/d8e/d5d/ff7 89313 0 2024-04-22T23:23:52.072 INFO:tasks.workunit.client.1.smithi134.stdout:4/849: dread - d4/d35/d53/fb8 zero size 2024-04-22T23:23:52.073 INFO:tasks.workunit.client.1.smithi134.stdout:4/850: readlink d4/d7/d32/d84/df4/lb3 0 2024-04-22T23:23:52.074 INFO:tasks.workunit.client.1.smithi134.stdout:4/851: chown d4/d29/dcc/f83 61 1 2024-04-22T23:23:52.074 INFO:tasks.workunit.client.1.smithi134.stdout:5/778: mkdir d9/da2/de6/dc0/d109 0 2024-04-22T23:23:52.075 INFO:tasks.workunit.client.1.smithi134.stdout:4/852: chown d4/d7/d32/d84/df4/lb3 131932 1 2024-04-22T23:23:52.076 INFO:tasks.workunit.client.1.smithi134.stdout:4/853: chown d4/d29/dcc/f83 0 1 2024-04-22T23:23:52.078 INFO:tasks.workunit.client.1.smithi134.stdout:2/607: link d2/d15/d24/d1d/f86 d2/dd/df/dc2/fcc 0 2024-04-22T23:23:52.082 INFO:tasks.workunit.client.1.smithi134.stdout:3/509: symlink dc/d2a/d62/db2/d2f/d21/d42/d77/d8f/lbd 0 2024-04-22T23:23:52.084 INFO:tasks.workunit.client.1.smithi134.stdout:3/510: chown dc/d2a/d30/ca9 221173 1 2024-04-22T23:23:52.084 INFO:tasks.workunit.client.1.smithi134.stdout:1/859: dwrite db/d15/d17/d20/d2a/d2d/f48 [0,4194304] 0 2024-04-22T23:23:52.085 INFO:tasks.workunit.client.1.smithi134.stdout:3/511: write dc/d2a/d62/db2/d2f/f51 [752891,91770] 0 2024-04-22T23:23:52.088 INFO:tasks.workunit.client.1.smithi134.stdout:0/665: creat d1/d5/dd/fdd x:0 0 0 2024-04-22T23:23:52.090 INFO:tasks.workunit.client.1.smithi134.stdout:4/854: mknod d4/d29/dcc/d82/de7/c108 0 2024-04-22T23:23:52.090 INFO:tasks.workunit.client.1.smithi134.stdout:4/855: chown d4/dd/da8 0 1 2024-04-22T23:23:52.090 INFO:tasks.workunit.client.1.smithi134.stdout:0/666: truncate d1/d5/d58/da0/fb6 1021988 0 2024-04-22T23:23:52.091 INFO:tasks.workunit.client.1.smithi134.stdout:2/608: rename d2/d42/d54/d33/lb8 to d2/dd/d4a/d73/lcd 0 2024-04-22T23:23:52.092 INFO:tasks.workunit.client.1.smithi134.stdout:4/856: dread - d4/d29/dcc/d64/fef zero size 2024-04-22T23:23:52.095 INFO:tasks.workunit.client.1.smithi134.stdout:5/779: creat d9/d11/d73/d95/d20/d108/f10a x:0 0 0 2024-04-22T23:23:52.098 INFO:tasks.workunit.client.1.smithi134.stdout:1/860: mkdir db/d15/d121 0 2024-04-22T23:23:52.099 INFO:tasks.workunit.client.1.smithi134.stdout:0/667: dwrite d1/d2/d1d/d72/fdc [0,4194304] 0 2024-04-22T23:23:52.101 INFO:tasks.workunit.client.1.smithi134.stdout:3/512: link dc/d2a/d62/db2/d2f/f68 dc/d2a/d62/db2/d34/d6b/fbe 0 2024-04-22T23:23:52.103 INFO:tasks.workunit.client.1.smithi134.stdout:5/780: symlink d9/d11/d73/d95/d20/d60/l10b 0 2024-04-22T23:23:52.104 INFO:tasks.workunit.client.1.smithi134.stdout:5/781: chown d9/dd/fbe 7 1 2024-04-22T23:23:52.105 INFO:tasks.workunit.client.1.smithi134.stdout:1/861: fdatasync db/db0/dc1/fcf 0 2024-04-22T23:23:52.108 INFO:tasks.workunit.client.1.smithi134.stdout:2/609: rename d2/d15/d24/d1d/c5c to d2/d42/cce 0 2024-04-22T23:23:52.110 INFO:tasks.workunit.client.1.smithi134.stdout:0/668: creat d1/d5/da2/fde x:0 0 0 2024-04-22T23:23:52.111 INFO:tasks.workunit.client.1.smithi134.stdout:5/782: symlink d9/dd/d88/db7/l10c 0 2024-04-22T23:23:52.113 INFO:tasks.workunit.client.1.smithi134.stdout:1/862: mknod db/d15/d17/d20/d2a/d2d/d108/d114/c122 0 2024-04-22T23:23:52.114 INFO:tasks.workunit.client.1.smithi134.stdout:6/767: dwrite d0/d19/de/d12/d2a/d2d/d3a/f9a [0,4194304] 0 2024-04-22T23:23:52.114 INFO:tasks.workunit.client.1.smithi134.stdout:4/857: getdents d4/d35/d8e/d5d/dce 0 2024-04-22T23:23:52.114 INFO:tasks.workunit.client.1.smithi134.stdout:1/863: readlink db/d16/d87/lde 0 2024-04-22T23:23:52.115 INFO:tasks.workunit.client.1.smithi134.stdout:3/513: fdatasync dc/d2a/d62/db2/d3b/f79 0 2024-04-22T23:23:52.116 INFO:tasks.workunit.client.1.smithi134.stdout:4/858: readlink d4/d29/dcc/d82/led 0 2024-04-22T23:23:52.118 INFO:tasks.workunit.client.1.smithi134.stdout:3/514: write dc/d2a/d62/fa5 [3788803,49594] 0 2024-04-22T23:23:52.118 INFO:tasks.workunit.client.1.smithi134.stdout:0/669: rename d1/db/d9f to d1/d5/d58/d22/dd0/ddf 0 2024-04-22T23:23:52.122 INFO:tasks.workunit.client.1.smithi134.stdout:1/864: write db/d15/d17/f5a [2545813,15486] 0 2024-04-22T23:23:52.123 INFO:tasks.workunit.client.1.smithi134.stdout:6/768: creat d0/d19/de/d12/d29/d74/deb/ffc x:0 0 0 2024-04-22T23:23:52.124 INFO:tasks.workunit.client.1.smithi134.stdout:5/783: rmdir d9/d11/d73/d95/d20/d60 39 2024-04-22T23:23:52.127 INFO:tasks.workunit.client.1.smithi134.stdout:0/670: creat d1/d5/dd/d41/fe0 x:0 0 0 2024-04-22T23:23:52.130 INFO:tasks.workunit.client.1.smithi134.stdout:2/610: write d2/d42/d54/f59 [457254,19932] 0 2024-04-22T23:23:52.131 INFO:tasks.workunit.client.1.smithi134.stdout:1/865: creat db/d15/d17/d20/d2a/d2d/d86/f123 x:0 0 0 2024-04-22T23:23:52.136 INFO:tasks.workunit.client.1.smithi134.stdout:6/769: mknod d0/d19/de/d12/d29/cfd 0 2024-04-22T23:23:52.136 INFO:tasks.workunit.client.1.smithi134.stdout:6/770: fdatasync d0/d6f/f84 0 2024-04-22T23:23:52.138 INFO:tasks.workunit.client.1.smithi134.stdout:5/784: symlink d9/d11/d73/da9/l10d 0 2024-04-22T23:23:52.142 INFO:tasks.workunit.client.1.smithi134.stdout:4/859: creat d4/dd/da8/dc2/df8/f109 x:0 0 0 2024-04-22T23:23:52.145 INFO:tasks.workunit.client.1.smithi134.stdout:6/771: mknod d0/d19/de/d1f/cfe 0 2024-04-22T23:23:52.146 INFO:tasks.workunit.client.1.smithi134.stdout:5/785: dread d9/d1e/f35 [4194304,4194304] 0 2024-04-22T23:23:52.147 INFO:tasks.workunit.client.1.smithi134.stdout:4/860: write d4/d13/fd6 [4075158,101432] 0 2024-04-22T23:23:52.148 INFO:tasks.workunit.client.1.smithi134.stdout:1/866: write db/d15/d17/d20/d2a/f88 [552021,7940] 0 2024-04-22T23:23:52.150 INFO:tasks.workunit.client.1.smithi134.stdout:3/515: dwrite dc/d2a/d62/db2/d34/d6b/fbe [0,4194304] 0 2024-04-22T23:23:52.153 INFO:tasks.workunit.client.1.smithi134.stdout:2/611: dread d2/d15/d24/f6d [0,4194304] 0 2024-04-22T23:23:52.153 INFO:tasks.workunit.client.1.smithi134.stdout:4/861: dread - d4/dd/da8/dc2/df8/f109 zero size 2024-04-22T23:23:52.155 INFO:tasks.workunit.client.1.smithi134.stdout:5/786: creat d9/dd/d88/f10e x:0 0 0 2024-04-22T23:23:52.155 INFO:tasks.workunit.client.1.smithi134.stdout:1/867: write db/f1d [1138734,36970] 0 2024-04-22T23:23:52.156 INFO:tasks.workunit.client.1.smithi134.stdout:3/516: dread - dc/d2a/d62/f7f zero size 2024-04-22T23:23:52.158 INFO:tasks.workunit.client.1.smithi134.stdout:0/671: dwrite d1/d2/f74 [0,4194304] 0 2024-04-22T23:23:52.158 INFO:tasks.workunit.client.1.smithi134.stdout:2/612: creat d2/dd/df/dc2/fcf x:0 0 0 2024-04-22T23:23:52.161 INFO:tasks.workunit.client.1.smithi134.stdout:4/862: creat d4/d35/db4/f10a x:0 0 0 2024-04-22T23:23:52.163 INFO:tasks.workunit.client.1.smithi134.stdout:5/787: truncate d9/dd/d17/d2a/d49/dc2/fcb 1551242 0 2024-04-22T23:23:52.165 INFO:tasks.workunit.client.1.smithi134.stdout:4/863: chown d4/dd/f41 25 1 2024-04-22T23:23:52.167 INFO:tasks.workunit.client.1.smithi134.stdout:5/788: unlink d9/d1e/c81 0 2024-04-22T23:23:52.169 INFO:tasks.workunit.client.1.smithi134.stdout:5/789: stat d9/d11/d34/d4d/d92/l106 0 2024-04-22T23:23:52.169 INFO:tasks.workunit.client.1.smithi134.stdout:0/672: dwrite d1/d5/dd/d41/fe0 [0,4194304] 0 2024-04-22T23:23:52.174 INFO:tasks.workunit.client.1.smithi134.stdout:6/772: truncate d0/d19/de/d12/d5e/d6d/f73 863759 0 2024-04-22T23:23:52.174 INFO:tasks.workunit.client.1.smithi134.stdout:1/868: write db/db0/fea [2142554,123486] 0 2024-04-22T23:23:52.174 INFO:tasks.workunit.client.1.smithi134.stdout:2/613: link d2/dd/df/d1c/f79 d2/d15/fd0 0 2024-04-22T23:23:52.176 INFO:tasks.workunit.client.1.smithi134.stdout:5/790: rename d9/d11/de1 to d9/d11/d34/d4d/dd4/d10f 0 2024-04-22T23:23:52.179 INFO:tasks.workunit.client.1.smithi134.stdout:6/773: mkdir d0/d19/de/d12/d2a/d2d/d3a/dff 0 2024-04-22T23:23:52.180 INFO:tasks.workunit.client.1.smithi134.stdout:2/614: creat d2/d15/d24/d1d/d3d/fd1 x:0 0 0 2024-04-22T23:23:52.181 INFO:tasks.workunit.client.1.smithi134.stdout:1/869: unlink db/d15/c19 0 2024-04-22T23:23:52.182 INFO:tasks.workunit.client.1.smithi134.stdout:5/791: creat d9/dd/d17/d2a/f110 x:0 0 0 2024-04-22T23:23:52.182 INFO:tasks.workunit.client.1.smithi134.stdout:6/774: dread - d0/d19/de/d12/d29/d74/da0/fab zero size 2024-04-22T23:23:52.182 INFO:tasks.workunit.client.1.smithi134.stdout:2/615: read d2/f87 [133171,107252] 0 2024-04-22T23:23:52.183 INFO:tasks.workunit.client.1.smithi134.stdout:0/673: dwrite d1/d2/f74 [0,4194304] 0 2024-04-22T23:23:52.189 INFO:tasks.workunit.client.1.smithi134.stdout:1/870: dread db/d15/d17/d20/fd1 [0,4194304] 0 2024-04-22T23:23:52.194 INFO:tasks.workunit.client.1.smithi134.stdout:4/864: truncate d4/d29/f63 4098886 0 2024-04-22T23:23:52.198 INFO:tasks.workunit.client.1.smithi134.stdout:4/865: chown d4/d7/d32/d3a/f51 1735 1 2024-04-22T23:23:52.199 INFO:tasks.workunit.client.1.smithi134.stdout:2/616: link d2/dd/df/d89/lc6 d2/dd/df/d9c/daa/ld2 0 2024-04-22T23:23:52.201 INFO:tasks.workunit.client.1.smithi134.stdout:4/866: fdatasync d4/d7/d32/fc1 0 2024-04-22T23:23:52.202 INFO:tasks.workunit.client.1.smithi134.stdout:5/792: truncate d9/dd/f86 952977 0 2024-04-22T23:23:52.204 INFO:tasks.workunit.client.1.smithi134.stdout:0/674: dread d1/d5/f8f [0,4194304] 0 2024-04-22T23:23:52.204 INFO:tasks.workunit.client.1.smithi134.stdout:0/675: chown d1/d5/dd/d41 528552485 1 2024-04-22T23:23:52.208 INFO:tasks.workunit.client.1.smithi134.stdout:1/871: write db/db5/fce [2025017,51714] 0 2024-04-22T23:23:52.211 INFO:tasks.workunit.client.1.smithi134.stdout:2/617: creat d2/dd/df/d9c/dca/fd3 x:0 0 0 2024-04-22T23:23:52.211 INFO:tasks.workunit.client.1.smithi134.stdout:1/872: stat db/d15/d17/cc4 0 2024-04-22T23:23:52.212 INFO:tasks.workunit.client.1.smithi134.stdout:5/793: creat d9/dd/d17/d2a/f111 x:0 0 0 2024-04-22T23:23:52.213 INFO:tasks.workunit.client.1.smithi134.stdout:0/676: dwrite d1/d2/f4 [0,4194304] 0 2024-04-22T23:23:52.213 INFO:tasks.workunit.client.1.smithi134.stdout:1/873: write db/d16/d56/f60 [4780816,7518] 0 2024-04-22T23:23:52.215 INFO:tasks.workunit.client.1.smithi134.stdout:5/794: chown d9/dd/d17/d44/c94 1266 1 2024-04-22T23:23:52.216 INFO:tasks.workunit.client.1.smithi134.stdout:0/677: write d1/d5/d58/d6b/db9/f18 [8337699,125568] 0 2024-04-22T23:23:52.223 INFO:tasks.workunit.client.1.smithi134.stdout:2/618: dwrite d2/d42/d54/d33/d6e/fbf [0,4194304] 0 2024-04-22T23:23:52.223 INFO:tasks.workunit.client.1.smithi134.stdout:0/678: unlink d1/d2/d1d/c6e 0 2024-04-22T23:23:52.226 INFO:tasks.workunit.client.1.smithi134.stdout:4/867: dwrite d4/d35/d80/f90 [0,4194304] 0 2024-04-22T23:23:52.228 INFO:tasks.workunit.client.1.smithi134.stdout:5/795: link d9/d11/d34/fbd d9/d11/d73/da9/f112 0 2024-04-22T23:23:52.230 INFO:tasks.workunit.client.1.smithi134.stdout:0/679: mknod d1/d5/dd/d20/ce1 0 2024-04-22T23:23:52.241 INFO:tasks.workunit.client.1.smithi134.stdout:0/680: dwrite d1/db/f70 [0,4194304] 0 2024-04-22T23:23:52.242 INFO:tasks.workunit.client.1.smithi134.stdout:2/619: creat d2/d94/fd4 x:0 0 0 2024-04-22T23:23:52.243 INFO:tasks.workunit.client.1.smithi134.stdout:5/796: creat d9/da2/de6/dc0/d109/f113 x:0 0 0 2024-04-22T23:23:52.243 INFO:tasks.workunit.client.1.smithi134.stdout:0/681: stat d1/d5/dd/d68/c7e 0 2024-04-22T23:23:52.244 INFO:tasks.workunit.client.1.smithi134.stdout:2/620: fdatasync d2/d42/d54/d33/d91/fad 0 2024-04-22T23:23:52.245 INFO:tasks.workunit.client.1.smithi134.stdout:0/682: creat d1/d2/fe2 x:0 0 0 2024-04-22T23:23:52.248 INFO:tasks.workunit.client.1.smithi134.stdout:2/621: write d2/d42/d54/d33/fa4 [106789,70496] 0 2024-04-22T23:23:52.251 INFO:tasks.workunit.client.1.smithi134.stdout:4/868: dwrite d4/f42 [0,4194304] 0 2024-04-22T23:23:52.255 INFO:tasks.workunit.client.1.smithi134.stdout:0/683: rename d1/db/c7f to d1/d5/d58/d94/ce3 0 2024-04-22T23:23:52.257 INFO:tasks.workunit.client.1.smithi134.stdout:4/869: write d4/d7/d32/d3a/dbc/fbd [448082,85493] 0 2024-04-22T23:23:52.258 INFO:tasks.workunit.client.1.smithi134.stdout:0/684: symlink d1/d5/d58/d6b/db9/d83/le4 0 2024-04-22T23:23:52.259 INFO:tasks.workunit.client.1.smithi134.stdout:4/870: mknod d4/d6b/dd9/c10b 0 2024-04-22T23:23:52.261 INFO:tasks.workunit.client.1.smithi134.stdout:4/871: write d4/d29/dcc/f81 [1555919,117702] 0 2024-04-22T23:23:52.266 INFO:tasks.workunit.client.1.smithi134.stdout:2/622: dwrite d2/d15/fd0 [0,4194304] 0 2024-04-22T23:23:52.266 INFO:tasks.workunit.client.1.smithi134.stdout:4/872: mknod d4/d7/d32/d3a/d93/c10c 0 2024-04-22T23:23:52.267 INFO:tasks.workunit.client.1.smithi134.stdout:2/623: fsync d2/d15/d24/f35 0 2024-04-22T23:23:52.271 INFO:tasks.workunit.client.1.smithi134.stdout:4/873: creat d4/d6b/f10d x:0 0 0 2024-04-22T23:23:52.272 INFO:tasks.workunit.client.1.smithi134.stdout:2/624: getdents d2/d15/d24/d1d/dba/dcb 0 2024-04-22T23:23:52.274 INFO:tasks.workunit.client.1.smithi134.stdout:4/874: mknod d4/d13/d68/c10e 0 2024-04-22T23:23:52.276 INFO:tasks.workunit.client.1.smithi134.stdout:4/875: rmdir d4/d29/dcc/d82/dd5 39 2024-04-22T23:23:52.279 INFO:tasks.workunit.client.1.smithi134.stdout:2/625: link d2/f3e d2/d15/d24/d1d/d3d/d9a/fd5 0 2024-04-22T23:23:52.281 INFO:tasks.workunit.client.1.smithi134.stdout:4/876: mkdir d4/dd/da8/dc2/df8/d25/d10f 0 2024-04-22T23:23:52.284 INFO:tasks.workunit.client.1.smithi134.stdout:4/877: truncate d4/dd/da8/dc2/df8/f97 935678 0 2024-04-22T23:23:52.284 INFO:tasks.workunit.client.1.smithi134.stdout:2/626: rename d2/dd/f37 to d2/d42/d54/fd6 0 2024-04-22T23:23:52.285 INFO:tasks.workunit.client.1.smithi134.stdout:0/685: dwrite d1/d5/d58/d22/dcb/fd9 [0,4194304] 0 2024-04-22T23:23:52.286 INFO:tasks.workunit.client.1.smithi134.stdout:4/878: chown d4/d7/d32/d84/df4/lb3 91486 1 2024-04-22T23:23:52.290 INFO:tasks.workunit.client.1.smithi134.stdout:4/879: symlink d4/d35/d8e/d5d/dce/l110 0 2024-04-22T23:23:52.294 INFO:tasks.workunit.client.1.smithi134.stdout:0/686: dwrite d1/d5/dd/d20/fc9 [0,4194304] 0 2024-04-22T23:23:52.298 INFO:tasks.workunit.client.1.smithi134.stdout:0/687: fdatasync d1/d5/dd/f45 0 2024-04-22T23:23:52.299 INFO:tasks.workunit.client.1.smithi134.stdout:0/688: mkdir d1/d5/d58/da0/de5 0 2024-04-22T23:23:52.301 INFO:tasks.workunit.client.1.smithi134.stdout:0/689: mkdir d1/d5/d58/d6b/db9/d75/de6 0 2024-04-22T23:23:52.303 INFO:tasks.workunit.client.1.smithi134.stdout:0/690: mknod d1/d5/d58/d94/ce7 0 2024-04-22T23:23:52.307 INFO:tasks.workunit.client.1.smithi134.stdout:0/691: truncate d1/d5/f8f 2171266 0 2024-04-22T23:23:52.318 INFO:tasks.workunit.client.1.smithi134.stdout:4/880: dread d4/d29/dcc/f26 [0,4194304] 0 2024-04-22T23:23:52.336 INFO:tasks.workunit.client.1.smithi134.stdout:9/657: sync 2024-04-22T23:23:52.338 INFO:tasks.workunit.client.1.smithi134.stdout:0/692: dread d1/d5/d58/f80 [0,4194304] 0 2024-04-22T23:23:52.339 INFO:tasks.workunit.client.1.smithi134.stdout:0/693: dread - d1/d5/da2/fde zero size 2024-04-22T23:23:52.342 INFO:tasks.workunit.client.1.smithi134.stdout:9/658: link d0/d2d/d4e/f62 d0/d7/d23/d26/fe3 0 2024-04-22T23:23:52.343 INFO:tasks.workunit.client.1.smithi134.stdout:0/694: symlink d1/d5/d58/d6b/db9/d83/le8 0 2024-04-22T23:23:52.344 INFO:tasks.workunit.client.1.smithi134.stdout:4/881: dwrite d4/dd/f16 [0,4194304] 0 2024-04-22T23:23:52.345 INFO:tasks.workunit.client.1.smithi134.stdout:9/659: symlink d0/d7/d23/d26/d51/le4 0 2024-04-22T23:23:52.347 INFO:tasks.workunit.client.1.smithi134.stdout:4/882: mkdir d4/d13/d111 0 2024-04-22T23:23:52.349 INFO:tasks.workunit.client.1.smithi134.stdout:0/695: creat d1/d5/d15/fe9 x:0 0 0 2024-04-22T23:23:52.350 INFO:tasks.workunit.client.1.smithi134.stdout:9/660: getdents d0/d7/d23/d1b/dc5 0 2024-04-22T23:23:52.350 INFO:tasks.workunit.client.1.smithi134.stdout:0/696: chown d1/d5/d58/d6b/db9/d83/le4 187 1 2024-04-22T23:23:52.351 INFO:tasks.workunit.client.1.smithi134.stdout:4/883: creat d4/d7/d32/d84/df4/f112 x:0 0 0 2024-04-22T23:23:52.352 INFO:tasks.workunit.client.1.smithi134.stdout:4/884: readlink d4/d29/ldf 0 2024-04-22T23:23:52.354 INFO:tasks.workunit.client.1.smithi134.stdout:4/885: dread d4/dd/da8/dc2/df8/f97 [0,4194304] 0 2024-04-22T23:23:52.357 INFO:tasks.workunit.client.1.smithi134.stdout:4/886: mknod d4/d29/dcc/dd8/c113 0 2024-04-22T23:23:52.358 INFO:tasks.workunit.client.1.smithi134.stdout:0/697: dwrite d1/d5/f8b [0,4194304] 0 2024-04-22T23:23:52.363 INFO:tasks.workunit.client.1.smithi134.stdout:0/698: mkdir d1/d5/d58/d94/dea 0 2024-04-22T23:23:52.365 INFO:tasks.workunit.client.1.smithi134.stdout:4/887: link d4/d6b/f10d d4/d13/d111/f114 0 2024-04-22T23:23:52.370 INFO:tasks.workunit.client.1.smithi134.stdout:0/699: link d1/d5/d58/d6b/db9/d83/le4 d1/d5/d58/d6b/db9/d83/leb 0 2024-04-22T23:23:52.384 INFO:tasks.workunit.client.1.smithi134.stdout:0/700: dwrite d1/d5/dd/f45 [0,4194304] 0 2024-04-22T23:23:52.387 INFO:tasks.workunit.client.1.smithi134.stdout:0/701: fdatasync d1/d2/f38 0 2024-04-22T23:23:52.389 INFO:tasks.workunit.client.1.smithi134.stdout:0/702: truncate d1/d5/dd/d41/fb7 774854 0 2024-04-22T23:23:52.391 INFO:tasks.workunit.client.1.smithi134.stdout:4/888: dread d4/d29/dcc/f92 [0,4194304] 0 2024-04-22T23:23:52.392 INFO:tasks.workunit.client.1.smithi134.stdout:4/889: write d4/d7/d32/d3a/dbc/dc4/ff9 [25411,77085] 0 2024-04-22T23:23:52.394 INFO:tasks.workunit.client.1.smithi134.stdout:4/890: write d4/d7/d32/d84/df4/f112 [511234,19299] 0 2024-04-22T23:23:52.398 INFO:tasks.workunit.client.1.smithi134.stdout:0/703: truncate d1/d5/dd/d41/fb7 401561 0 2024-04-22T23:23:52.399 INFO:tasks.workunit.client.1.smithi134.stdout:0/704: chown d1/d5/d58/da0/l8d 7266693 1 2024-04-22T23:23:52.401 INFO:tasks.workunit.client.1.smithi134.stdout:0/705: symlink d1/db/lec 0 2024-04-22T23:23:52.404 INFO:tasks.workunit.client.1.smithi134.stdout:0/706: creat d1/d5/d58/d6b/db9/d75/de6/fed x:0 0 0 2024-04-22T23:23:52.407 INFO:tasks.workunit.client.1.smithi134.stdout:4/891: dread d4/d7/f10 [0,4194304] 0 2024-04-22T23:23:52.409 INFO:tasks.workunit.client.1.smithi134.stdout:4/892: mknod d4/d7/d32/c115 0 2024-04-22T23:23:52.414 INFO:tasks.workunit.client.1.smithi134.stdout:4/893: dwrite d4/d7/d32/d3a/d93/db5/f105 [0,4194304] 0 2024-04-22T23:23:52.420 INFO:tasks.workunit.client.1.smithi134.stdout:4/894: getdents d4/d13/d68/df6 0 2024-04-22T23:23:52.421 INFO:tasks.workunit.client.1.smithi134.stdout:4/895: read - d4/d7/d32/d3a/d93/db5/ffa zero size 2024-04-22T23:23:52.480 INFO:tasks.workunit.client.1.smithi134.stdout:8/661: sync 2024-04-22T23:23:52.480 INFO:tasks.workunit.client.1.smithi134.stdout:7/727: sync 2024-04-22T23:23:52.481 INFO:tasks.workunit.client.1.smithi134.stdout:7/728: write d4/d4a/d85/dd7/fd9 [1855899,20814] 0 2024-04-22T23:23:52.482 INFO:tasks.workunit.client.1.smithi134.stdout:8/662: creat d7/db/d42/d65/d46/d9a/fd4 x:0 0 0 2024-04-22T23:23:52.489 INFO:tasks.workunit.client.1.smithi134.stdout:8/663: dwrite d7/db/d42/f76 [0,4194304] 0 2024-04-22T23:23:52.494 INFO:tasks.workunit.client.1.smithi134.stdout:7/729: dwrite d4/d8/d41/d76/dc1/d2f/d70/f74 [0,4194304] 0 2024-04-22T23:23:52.496 INFO:tasks.workunit.client.1.smithi134.stdout:8/664: dread d7/db/f1f [0,4194304] 0 2024-04-22T23:23:52.509 INFO:tasks.workunit.client.1.smithi134.stdout:8/665: symlink d7/db/d42/d65/d46/ld5 0 2024-04-22T23:23:52.511 INFO:tasks.workunit.client.1.smithi134.stdout:8/666: mknod d7/db/d2d/d39/d58/d8a/cd6 0 2024-04-22T23:23:52.515 INFO:tasks.workunit.client.1.smithi134.stdout:7/730: dwrite d4/d8/d2a/d42/d79/fa2 [0,4194304] 0 2024-04-22T23:23:52.516 INFO:tasks.workunit.client.1.smithi134.stdout:8/667: write d7/db/d42/d65/fc9 [1478667,39017] 0 2024-04-22T23:23:52.518 INFO:tasks.workunit.client.1.smithi134.stdout:7/731: write d4/d8/d41/d76/dc1/faa [1215529,118865] 0 2024-04-22T23:23:52.522 INFO:tasks.workunit.client.1.smithi134.stdout:8/668: creat d7/db/d42/d65/d46/d9a/da3/d88/da6/fd7 x:0 0 0 2024-04-22T23:23:52.523 INFO:tasks.workunit.client.1.smithi134.stdout:7/732: mkdir d4/d8/d41/d5b/d104 0 2024-04-22T23:23:52.530 INFO:tasks.workunit.client.1.smithi134.stdout:7/733: rename d4/d8/d2a/d42/d6e/dd6 to d4/d8/d41/d76/dc1/d2f/d69/d8a/d105 0 2024-04-22T23:23:52.533 INFO:tasks.workunit.client.1.smithi134.stdout:8/669: truncate d7/db/d2d/f5a 1639634 0 2024-04-22T23:23:52.535 INFO:tasks.workunit.client.1.smithi134.stdout:7/734: rename d4/d8/d41/d76/dc1/d2f/d34/d62/ccc to d4/d8/d41/d76/dc1/d2f/d69/dab/d95/dec/c106 0 2024-04-22T23:23:52.538 INFO:tasks.workunit.client.1.smithi134.stdout:8/670: symlink d7/db/d42/d65/ld8 0 2024-04-22T23:23:52.539 INFO:tasks.workunit.client.1.smithi134.stdout:8/671: write d7/dc/f22 [643070,49620] 0 2024-04-22T23:23:52.539 INFO:tasks.workunit.client.1.smithi134.stdout:7/735: creat d4/d8/d41/d76/dc1/d2f/d34/dc0/f107 x:0 0 0 2024-04-22T23:23:52.544 INFO:tasks.workunit.client.1.smithi134.stdout:7/736: mknod d4/d8/d58/c108 0 2024-04-22T23:23:52.548 INFO:tasks.workunit.client.1.smithi134.stdout:8/672: dwrite d7/dc/d19/da5/fcc [0,4194304] 0 2024-04-22T23:23:52.549 INFO:tasks.workunit.client.1.smithi134.stdout:8/673: readlink d7/db/l41 0 2024-04-22T23:23:52.550 INFO:tasks.workunit.client.1.smithi134.stdout:7/737: symlink d4/d8/d41/d76/dc1/d2f/d69/df8/l109 0 2024-04-22T23:23:52.556 INFO:tasks.workunit.client.1.smithi134.stdout:8/674: mkdir d7/db/d42/d65/dd9 0 2024-04-22T23:23:52.559 INFO:tasks.workunit.client.1.smithi134.stdout:7/738: dwrite d4/d4a/f92 [0,4194304] 0 2024-04-22T23:23:52.565 INFO:tasks.workunit.client.1.smithi134.stdout:7/739: write d4/d8/d41/d76/dc1/faa [2296530,85058] 0 2024-04-22T23:23:52.568 INFO:tasks.workunit.client.1.smithi134.stdout:8/675: dwrite d7/db/d42/d65/d46/d9a/da3/d88/fd0 [0,4194304] 0 2024-04-22T23:23:52.572 INFO:tasks.workunit.client.1.smithi134.stdout:8/676: chown d7/db/d42/d65/d46/d9a/fa1 83626 1 2024-04-22T23:23:52.574 INFO:tasks.workunit.client.1.smithi134.stdout:8/677: dread - d7/db/d2d/d39/d58/fb9 zero size 2024-04-22T23:23:52.579 INFO:tasks.workunit.client.1.smithi134.stdout:8/678: rename d7/dc/d2c/d37/d47/daf to d7/db/d42/d65/dd9/dda 0 2024-04-22T23:23:52.585 INFO:tasks.workunit.client.1.smithi134.stdout:8/679: getdents d7 0 2024-04-22T23:23:52.588 INFO:tasks.workunit.client.1.smithi134.stdout:7/740: dwrite d4/f31 [0,4194304] 0 2024-04-22T23:23:52.590 INFO:tasks.workunit.client.1.smithi134.stdout:7/741: chown d4/d8/d41/d76/dc1/d2f/d69/dab/de1 388 1 2024-04-22T23:23:52.591 INFO:tasks.workunit.client.1.smithi134.stdout:7/742: read d4/d8/d41/d76/dc1/d2f/d34/d62/d84/f9f [1030326,8457] 0 2024-04-22T23:23:52.595 INFO:tasks.workunit.client.1.smithi134.stdout:8/680: getdents d7/db 0 2024-04-22T23:23:52.599 INFO:tasks.workunit.client.1.smithi134.stdout:7/743: truncate d4/d8/d41/d76/dc1/faa 458734 0 2024-04-22T23:23:52.602 INFO:tasks.workunit.client.1.smithi134.stdout:8/681: dwrite d7/f1c [0,4194304] 0 2024-04-22T23:23:52.606 INFO:tasks.workunit.client.1.smithi134.stdout:7/744: getdents d4 0 2024-04-22T23:23:52.607 INFO:tasks.workunit.client.1.smithi134.stdout:7/745: fdatasync d4/d8/d3b/fd1 0 2024-04-22T23:23:52.608 INFO:tasks.workunit.client.1.smithi134.stdout:8/682: creat d7/db/d2d/d39/d58/d8a/fdb x:0 0 0 2024-04-22T23:23:52.611 INFO:tasks.workunit.client.1.smithi134.stdout:7/746: dread d4/f31 [0,4194304] 0 2024-04-22T23:23:52.614 INFO:tasks.workunit.client.1.smithi134.stdout:7/747: creat d4/d8/d101/f10a x:0 0 0 2024-04-22T23:23:52.616 INFO:tasks.workunit.client.1.smithi134.stdout:7/748: creat d4/dad/f10b x:0 0 0 2024-04-22T23:23:52.616 INFO:tasks.workunit.client.1.smithi134.stdout:7/749: dread - d4/dad/f10b zero size 2024-04-22T23:23:52.619 INFO:tasks.workunit.client.1.smithi134.stdout:7/750: dread - d4/d8/d41/d76/dc1/d2f/d69/dab/d95/fcb zero size 2024-04-22T23:23:52.623 INFO:tasks.workunit.client.1.smithi134.stdout:8/683: write d7/dc/ff [1943564,13115] 0 2024-04-22T23:23:52.626 INFO:tasks.workunit.client.1.smithi134.stdout:7/751: symlink d4/d8/d41/d76/dc1/d2f/da6/dcf/l10c 0 2024-04-22T23:23:52.627 INFO:tasks.workunit.client.1.smithi134.stdout:7/752: write d4/d8/d2a/d42/d79/fa2 [2770714,69704] 0 2024-04-22T23:23:52.647 INFO:tasks.workunit.client.1.smithi134.stdout:8/684: creat d7/dc/d2c/d4f/d6c/dd1/fdc x:0 0 0 2024-04-22T23:23:52.653 INFO:tasks.workunit.client.1.smithi134.stdout:8/685: fsync d7/db/d2d/f2e 0 2024-04-22T23:23:52.660 INFO:tasks.workunit.client.1.smithi134.stdout:8/686: rename d7/dc/d2c/d77 to d7/db/d2d/d39/d58/d8a/ddd 0 2024-04-22T23:23:52.667 INFO:tasks.workunit.client.1.smithi134.stdout:8/687: dwrite d7/f23 [4194304,4194304] 0 2024-04-22T23:23:52.668 INFO:tasks.workunit.client.1.smithi134.stdout:7/753: dwrite d4/d8/d41/d76/dc1/f11 [0,4194304] 0 2024-04-22T23:23:52.673 INFO:tasks.workunit.client.1.smithi134.stdout:8/688: creat d7/db/d2d/d39/d58/d8a/fde x:0 0 0 2024-04-22T23:23:52.679 INFO:tasks.workunit.client.1.smithi134.stdout:7/754: link d4/c16 d4/d8/d2a/d42/d79/c10d 0 2024-04-22T23:23:52.681 INFO:tasks.workunit.client.1.smithi134.stdout:8/689: unlink d7/dc/d2c/d4f/f9f 0 2024-04-22T23:23:52.682 INFO:tasks.workunit.client.1.smithi134.stdout:7/755: mknod d4/d4a/c10e 0 2024-04-22T23:23:52.687 INFO:tasks.workunit.client.1.smithi134.stdout:7/756: rename d4/d8/d41/c9e to d4/d8/d41/d76/dc1/d2f/d69/dab/de1/c10f 0 2024-04-22T23:23:52.688 INFO:tasks.workunit.client.1.smithi134.stdout:7/757: write d4/fa [4620506,73651] 0 2024-04-22T23:23:52.688 INFO:tasks.workunit.client.1.smithi134.stdout:8/690: truncate d7/dc/d2c/d37/d47/f34 813106 0 2024-04-22T23:23:52.689 INFO:tasks.workunit.client.1.smithi134.stdout:7/758: chown d4/d8/f22 54692161 1 2024-04-22T23:23:52.695 INFO:tasks.workunit.client.1.smithi134.stdout:7/759: dread d4/d8/d41/d76/dc1/f11 [0,4194304] 0 2024-04-22T23:23:52.708 INFO:tasks.workunit.client.1.smithi134.stdout:8/691: truncate d7/db/d2d/d39/d58/f97 1145322 0 2024-04-22T23:23:52.711 INFO:tasks.workunit.client.1.smithi134.stdout:8/692: creat d7/db/d42/d65/d46/fdf x:0 0 0 2024-04-22T23:23:52.711 INFO:tasks.workunit.client.1.smithi134.stdout:7/760: dread d4/d8/d2a/f5f [0,4194304] 0 2024-04-22T23:23:52.719 INFO:tasks.workunit.client.1.smithi134.stdout:8/693: dwrite d7/dc/f2b [0,4194304] 0 2024-04-22T23:23:52.723 INFO:tasks.workunit.client.1.smithi134.stdout:7/761: link d4/d8/d41/d76/dc1/f1c d4/d8/d41/d76/dc1/d2f/d69/dab/d95/dec/f110 0 2024-04-22T23:23:52.727 INFO:tasks.workunit.client.1.smithi134.stdout:8/694: dwrite d7/db/d2d/d39/d58/fb9 [0,4194304] 0 2024-04-22T23:23:52.731 INFO:tasks.workunit.client.1.smithi134.stdout:7/762: creat d4/d8/d41/d5b/d104/f111 x:0 0 0 2024-04-22T23:23:52.732 INFO:tasks.workunit.client.1.smithi134.stdout:8/695: mkdir d7/dc/de0 0 2024-04-22T23:23:52.734 INFO:tasks.workunit.client.1.smithi134.stdout:8/696: write d7/dc/d19/da5/fab [4682305,2821] 0 2024-04-22T23:23:52.734 INFO:tasks.workunit.client.1.smithi134.stdout:7/763: rmdir d4/d8/d2a/d42/d79 39 2024-04-22T23:23:52.739 INFO:tasks.workunit.client.1.smithi134.stdout:8/697: symlink d7/db/d42/d65/d46/d9a/da3/d88/le1 0 2024-04-22T23:23:52.740 INFO:tasks.workunit.client.1.smithi134.stdout:8/698: chown d7/db/d2d/d39/d58/d8a/cd6 392 1 2024-04-22T23:23:52.741 INFO:tasks.workunit.client.1.smithi134.stdout:7/764: dwrite d4/d8/d2a/f2c [0,4194304] 0 2024-04-22T23:23:52.746 INFO:tasks.workunit.client.1.smithi134.stdout:7/765: truncate d4/d8/d41/d76/dc1/d2f/d34/d54/fb4 4500452 0 2024-04-22T23:23:52.750 INFO:tasks.workunit.client.1.smithi134.stdout:8/699: dwrite d7/db/d2d/fa7 [0,4194304] 0 2024-04-22T23:23:52.751 INFO:tasks.workunit.client.1.smithi134.stdout:7/766: mkdir d4/d8/d41/d5b/d112 0 2024-04-22T23:23:52.752 INFO:tasks.workunit.client.1.smithi134.stdout:8/700: chown d7/db/d2d/f99 416791 1 2024-04-22T23:23:52.753 INFO:tasks.workunit.client.1.smithi134.stdout:7/767: fsync d4/d8/d41/d76/dc1/d2f/d34/d62/d29/ff1 0 2024-04-22T23:23:52.754 INFO:tasks.workunit.client.1.smithi134.stdout:8/701: creat d7/dc/d2c/d37/d3a/fe2 x:0 0 0 2024-04-22T23:23:52.755 INFO:tasks.workunit.client.1.smithi134.stdout:8/702: readlink d7/db/d42/d65/l36 0 2024-04-22T23:23:52.756 INFO:tasks.workunit.client.1.smithi134.stdout:7/768: mkdir d4/d4a/d85/dd7/d113 0 2024-04-22T23:23:52.758 INFO:tasks.workunit.client.1.smithi134.stdout:8/703: rmdir d7/dc/d2c/d4f 39 2024-04-22T23:23:52.761 INFO:tasks.workunit.client.1.smithi134.stdout:8/704: creat d7/db/d42/d65/d46/d9a/da3/d88/da6/fe3 x:0 0 0 2024-04-22T23:23:52.766 INFO:tasks.workunit.client.1.smithi134.stdout:7/769: link d4/d4a/c52 d4/d8/d2a/d42/d6e/c114 0 2024-04-22T23:23:52.767 INFO:tasks.workunit.client.1.smithi134.stdout:7/770: read - d4/d8/d41/d76/dc1/d2f/d34/dc0/f107 zero size 2024-04-22T23:23:52.769 INFO:tasks.workunit.client.1.smithi134.stdout:8/705: dread d7/db/d2d/f3c [0,4194304] 0 2024-04-22T23:23:52.773 INFO:tasks.workunit.client.1.smithi134.stdout:8/706: unlink d7/db/fc0 0 2024-04-22T23:23:52.774 INFO:tasks.workunit.client.1.smithi134.stdout:8/707: write d7/dc/f9c [3730543,17395] 0 2024-04-22T23:23:52.778 INFO:tasks.workunit.client.1.smithi134.stdout:7/771: write d4/d8/d41/d76/dc1/d2f/d69/fe9 [23136,85751] 0 2024-04-22T23:23:52.780 INFO:tasks.workunit.client.1.smithi134.stdout:8/708: chown d7/dc/d2c/c43 2 1 2024-04-22T23:23:52.781 INFO:tasks.workunit.client.1.smithi134.stdout:7/772: unlink d4/d8/d41/d76/dc1/fa5 0 2024-04-22T23:23:52.783 INFO:tasks.workunit.client.1.smithi134.stdout:8/709: chown d7/dc/d2c/cbb 5428177 1 2024-04-22T23:23:52.784 INFO:tasks.workunit.client.1.smithi134.stdout:7/773: creat d4/d8/d41/d76/dc1/d2f/d69/df8/f115 x:0 0 0 2024-04-22T23:23:52.786 INFO:tasks.workunit.client.1.smithi134.stdout:7/774: mkdir d4/d4a/d116 0 2024-04-22T23:23:52.788 INFO:tasks.workunit.client.1.smithi134.stdout:8/710: chown d7/db/d42 1995 1 2024-04-22T23:23:52.790 INFO:tasks.workunit.client.1.smithi134.stdout:7/775: unlink d4/d8/d41/d76/dc1/d2f/d34/d62/f71 0 2024-04-22T23:23:52.791 INFO:tasks.workunit.client.1.smithi134.stdout:8/711: chown d7/dc/d2c/d4f/d6c/dd1 90058 1 2024-04-22T23:23:52.792 INFO:tasks.workunit.client.1.smithi134.stdout:8/712: fsync d7/dc/d2c/d4f/d6c/d6e/fc4 0 2024-04-22T23:23:52.796 INFO:tasks.workunit.client.1.smithi134.stdout:8/713: mkdir d7/db/d42/d65/dd9/dda/de4 0 2024-04-22T23:23:52.799 INFO:tasks.workunit.client.1.smithi134.stdout:3/517: sync 2024-04-22T23:23:52.799 INFO:tasks.workunit.client.1.smithi134.stdout:6/775: sync 2024-04-22T23:23:52.801 INFO:tasks.workunit.client.1.smithi134.stdout:8/714: dread d7/dc/d2c/d37/d47/f5c [0,4194304] 0 2024-04-22T23:23:52.802 INFO:tasks.workunit.client.1.smithi134.stdout:7/776: dwrite d4/d8/d41/d76/dc1/d2f/d34/d62/d29/f55 [4194304,4194304] 0 2024-04-22T23:23:52.803 INFO:tasks.workunit.client.1.smithi134.stdout:3/518: unlink dc/l13 0 2024-04-22T23:23:52.805 INFO:tasks.workunit.client.1.smithi134.stdout:7/777: chown d4/d8/d41/d76/dc1/d2f/d34/c7f 63 1 2024-04-22T23:23:52.808 INFO:tasks.workunit.client.1.smithi134.stdout:6/776: dwrite d0/d19/de/d12/d2a/f4c [0,4194304] 0 2024-04-22T23:23:52.812 INFO:tasks.workunit.client.1.smithi134.stdout:3/519: rename dc/d2a/d62/db2/d3b/f79 to dc/d2a/d62/db2/d2f/d21/d42/d77/d8f/fbf 0 2024-04-22T23:23:52.814 INFO:tasks.workunit.client.1.smithi134.stdout:7/778: unlink d4/d8/d41/d76/dc1/d2f/d34/d62/d29/f55 0 2024-04-22T23:23:52.819 INFO:tasks.workunit.client.1.smithi134.stdout:3/520: creat dc/d2a/d62/db2/d2f/d36/d6d/d80/fc0 x:0 0 0 2024-04-22T23:23:52.821 INFO:tasks.workunit.client.1.smithi134.stdout:7/779: symlink d4/d8/d41/d76/dc1/d2f/d69/d8a/d105/l117 0 2024-04-22T23:23:52.824 INFO:tasks.workunit.client.1.smithi134.stdout:8/715: creat d7/db/d2d/d39/d58/fe5 x:0 0 0 2024-04-22T23:23:52.827 INFO:tasks.workunit.client.1.smithi134.stdout:3/521: mkdir dc/d2a/d62/d86/dc1 0 2024-04-22T23:23:52.827 INFO:tasks.workunit.client.1.smithi134.stdout:6/777: write d0/d19/f39 [2569119,82184] 0 2024-04-22T23:23:52.829 INFO:tasks.workunit.client.1.smithi134.stdout:7/780: mknod d4/d8/d41/d76/dc1/d2f/d69/d8a/d105/c118 0 2024-04-22T23:23:52.830 INFO:tasks.workunit.client.1.smithi134.stdout:8/716: mknod d7/db/d42/d65/d46/d9a/da3/d88/ce6 0 2024-04-22T23:23:52.835 INFO:tasks.workunit.client.1.smithi134.stdout:8/717: rename d7/db/d2d/d39/d58/d8a to d7/db/d42/d65/dd9/dda/de7 0 2024-04-22T23:23:52.837 INFO:tasks.workunit.client.1.smithi134.stdout:6/778: getdents d0/d19/db/d33/d86 0 2024-04-22T23:23:52.838 INFO:tasks.workunit.client.1.smithi134.stdout:6/779: chown d0/f16 412252 1 2024-04-22T23:23:52.842 INFO:tasks.workunit.client.1.smithi134.stdout:6/780: dwrite d0/d6f/f84 [0,4194304] 0 2024-04-22T23:23:52.851 INFO:tasks.workunit.client.1.smithi134.stdout:8/718: link d7/dc/f9c d7/db/d42/d65/dd9/dda/de4/fe8 0 2024-04-22T23:23:52.858 INFO:tasks.workunit.client.1.smithi134.stdout:6/781: unlink d0/d19/de/d12/d2a/d2d/f72 0 2024-04-22T23:23:52.868 INFO:tasks.workunit.client.1.smithi134.stdout:3/522: dread dc/d2a/d62/db2/d2f/f73 [0,4194304] 0 2024-04-22T23:23:52.890 INFO:tasks.workunit.client.1.smithi134.stdout:8/719: dread d7/dc/ff [0,4194304] 0 2024-04-22T23:23:52.892 INFO:tasks.workunit.client.1.smithi134.stdout:6/782: write d0/d19/de/d1f/f22 [3968836,86201] 0 2024-04-22T23:23:52.921 INFO:tasks.workunit.client.1.smithi134.stdout:8/720: write d7/db/d2d/f54 [2559635,55441] 0 2024-04-22T23:23:52.934 INFO:tasks.workunit.client.1.smithi134.stdout:8/721: dread d7/dc/d19/da5/fab [0,4194304] 0 2024-04-22T23:23:52.936 INFO:tasks.workunit.client.1.smithi134.stdout:8/722: dread - d7/db/d42/d65/d46/d9a/da3/d88/da6/fe3 zero size 2024-04-22T23:23:52.939 INFO:tasks.workunit.client.1.smithi134.stdout:6/783: write d0/d19/db/d35/f58 [2746629,50343] 0 2024-04-22T23:23:52.939 INFO:tasks.workunit.client.1.smithi134.stdout:8/723: write d7/db/d2d/f3c [1599856,121359] 0 2024-04-22T23:23:52.941 INFO:tasks.workunit.client.1.smithi134.stdout:8/724: chown d7/dc/f12 3066832 1 2024-04-22T23:23:52.942 INFO:tasks.workunit.client.1.smithi134.stdout:3/523: dwrite dc/d2a/d62/db2/d3b/d29/f2e [4194304,4194304] 0 2024-04-22T23:23:52.943 INFO:tasks.workunit.client.1.smithi134.stdout:3/524: chown dc/d2a/d62/db2/d2f/d54/dac 302106813 1 2024-04-22T23:23:52.963 INFO:tasks.workunit.client.1.smithi134.stdout:8/725: dread d7/db/d2d/f54 [0,4194304] 0 2024-04-22T23:23:52.964 INFO:tasks.workunit.client.1.smithi134.stdout:8/726: chown d7/dc/d2c/d37/d47/c55 2 1 2024-04-22T23:23:52.969 INFO:tasks.workunit.client.1.smithi134.stdout:8/727: dwrite f6 [4194304,4194304] 0 2024-04-22T23:23:52.975 INFO:tasks.workunit.client.1.smithi134.stdout:8/728: read d7/db/d42/d65/d46/d9a/da3/f92 [2381492,70721] 0 2024-04-22T23:23:52.978 INFO:tasks.workunit.client.1.smithi134.stdout:8/729: unlink d7/dc/d2c/d4f/c7c 0 2024-04-22T23:23:52.979 INFO:tasks.workunit.client.1.smithi134.stdout:8/730: chown d7/db/d2d/d39/d58/f97 1 1 2024-04-22T23:23:52.980 INFO:tasks.workunit.client.1.smithi134.stdout:9/661: sync 2024-04-22T23:23:52.981 INFO:tasks.workunit.client.1.smithi134.stdout:4/896: sync 2024-04-22T23:23:52.981 INFO:tasks.workunit.client.1.smithi134.stdout:2/627: sync 2024-04-22T23:23:52.981 INFO:tasks.workunit.client.1.smithi134.stdout:0/707: sync 2024-04-22T23:23:52.981 INFO:tasks.workunit.client.1.smithi134.stdout:1/874: sync 2024-04-22T23:23:52.981 INFO:tasks.workunit.client.1.smithi134.stdout:5/797: sync 2024-04-22T23:23:52.984 INFO:tasks.workunit.client.1.smithi134.stdout:4/897: dread d4/d13/f74 [0,4194304] 0 2024-04-22T23:23:52.984 INFO:tasks.workunit.client.1.smithi134.stdout:9/662: chown d0/d7/d23/d26/d51/d88/db9/fcb 94 1 2024-04-22T23:23:52.986 INFO:tasks.workunit.client.1.smithi134.stdout:0/708: creat d1/d5/d58/da0/fee x:0 0 0 2024-04-22T23:23:52.986 INFO:tasks.workunit.client.1.smithi134.stdout:8/731: dread d7/dc/f2b [0,4194304] 0 2024-04-22T23:23:52.986 INFO:tasks.workunit.client.1.smithi134.stdout:9/663: chown d0/d2d/d82/dbf/fd1 1010 1 2024-04-22T23:23:52.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:52 smithi134 ceph-mon[34609]: pgmap v281: 65 pgs: 65 active+clean; 2.8 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 93 MiB/s rd, 268 MiB/s wr, 613 op/s 2024-04-22T23:23:52.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:52 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:23:52.997 INFO:tasks.workunit.client.1.smithi134.stdout:1/875: dread db/f26 [0,4194304] 0 2024-04-22T23:23:52.999 INFO:tasks.workunit.client.1.smithi134.stdout:2/628: unlink d2/d15/d24/d1d/d3d/l62 0 2024-04-22T23:23:53.000 INFO:tasks.workunit.client.1.smithi134.stdout:9/664: mknod d0/d7/d27/ce5 0 2024-04-22T23:23:53.004 INFO:tasks.workunit.client.1.smithi134.stdout:4/898: fdatasync d4/d29/dcc/d64/f73 0 2024-04-22T23:23:53.006 INFO:tasks.workunit.client.1.smithi134.stdout:5/798: dwrite d9/dd/d52/f6b [0,4194304] 0 2024-04-22T23:23:53.006 INFO:tasks.workunit.client.1.smithi134.stdout:0/709: mkdir d1/d5/d58/d6b/db9/def 0 2024-04-22T23:23:53.010 INFO:tasks.workunit.client.1.smithi134.stdout:4/899: dread d4/d35/d8e/d5d/ff7 [0,4194304] 0 2024-04-22T23:23:53.010 INFO:tasks.workunit.client.1.smithi134.stdout:8/732: dwrite d7/dc/f12 [0,4194304] 0 2024-04-22T23:23:53.012 INFO:tasks.workunit.client.1.smithi134.stdout:2/629: fsync d2/f87 0 2024-04-22T23:23:53.012 INFO:tasks.workunit.client.1.smithi134.stdout:5/799: chown d9/dd/d17/d2a/d49/dc2/cf6 1774093927 1 2024-04-22T23:23:53.014 INFO:tasks.workunit.client.1.smithi134.stdout:9/665: dwrite d0/d2d/d3d/f9e [0,4194304] 0 2024-04-22T23:23:53.016 INFO:tasks.workunit.client.1.smithi134.stdout:0/710: read d1/d2/d1d/fc2 [1830590,74197] 0 2024-04-22T23:23:53.020 INFO:tasks.workunit.client.1.smithi134.stdout:9/666: dread - d0/d7/d23/d26/d4d/d5d/d8e/f9c zero size 2024-04-22T23:23:53.020 INFO:tasks.workunit.client.1.smithi134.stdout:5/800: fdatasync d9/dd/da6/fc6 0 2024-04-22T23:23:53.023 INFO:tasks.workunit.client.1.smithi134.stdout:0/711: write d1/d5/dd/d41/fe0 [4262022,77473] 0 2024-04-22T23:23:53.028 INFO:tasks.workunit.client.1.smithi134.stdout:4/900: rename d4/d29/dcc/d64 to d4/d29/d116 0 2024-04-22T23:23:53.030 INFO:tasks.workunit.client.1.smithi134.stdout:1/876: link db/d15/d17/l10b db/d15/d17/d20/d2a/l124 0 2024-04-22T23:23:53.031 INFO:tasks.workunit.client.1.smithi134.stdout:1/877: chown db/f8b 448499 1 2024-04-22T23:23:53.031 INFO:tasks.workunit.client.1.smithi134.stdout:1/878: readlink db/d16/d87/lde 0 2024-04-22T23:23:53.033 INFO:tasks.workunit.client.1.smithi134.stdout:5/801: mkdir d9/dd/da6/d114 0 2024-04-22T23:23:53.033 INFO:tasks.workunit.client.1.smithi134.stdout:5/802: chown d9/f15 3103679 1 2024-04-22T23:23:53.034 INFO:tasks.workunit.client.1.smithi134.stdout:9/667: truncate d0/d7/f92 647712 0 2024-04-22T23:23:53.036 INFO:tasks.workunit.client.1.smithi134.stdout:9/668: chown d0/d7/d23/d26/d4d/d5d/d8e/dae/db1 2775621 1 2024-04-22T23:23:53.037 INFO:tasks.workunit.client.1.smithi134.stdout:4/901: rename d4/d29/dcc/d82 to d4/d7/d32/d3a/d93/d117 0 2024-04-22T23:23:53.039 INFO:tasks.workunit.client.1.smithi134.stdout:4/902: write d4/d7/d32/d3a/d93/db5/fe5 [725719,83477] 0 2024-04-22T23:23:53.040 INFO:tasks.workunit.client.1.smithi134.stdout:8/733: link d7/db/d42/d65/d46/d9a/faa d7/fe9 0 2024-04-22T23:23:53.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:52 smithi067 ceph-mon[26453]: pgmap v281: 65 pgs: 65 active+clean; 2.8 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 93 MiB/s rd, 268 MiB/s wr, 613 op/s 2024-04-22T23:23:53.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:52 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:23:53.046 INFO:tasks.workunit.client.1.smithi134.stdout:1/879: dread db/d15/f29 [4194304,4194304] 0 2024-04-22T23:23:53.133 INFO:tasks.workunit.client.1.smithi134.stdout:5/803: creat d9/d11/d73/d95/f115 x:0 0 0 2024-04-22T23:23:53.134 INFO:tasks.workunit.client.1.smithi134.stdout:5/804: write d9/d11/d73/d95/dae/ff2 [1946,23563] 0 2024-04-22T23:23:53.158 INFO:tasks.workunit.client.1.smithi134.stdout:4/903: symlink d4/d35/d53/l118 0 2024-04-22T23:23:53.159 INFO:tasks.workunit.client.1.smithi134.stdout:4/904: chown d4/d13/f59 120156 1 2024-04-22T23:23:53.160 INFO:tasks.workunit.client.1.smithi134.stdout:8/734: write d7/db/d2d/f54 [2806250,100600] 0 2024-04-22T23:23:53.161 INFO:tasks.workunit.client.1.smithi134.stdout:1/880: fsync db/d15/d17/d20/d2a/f44 0 2024-04-22T23:23:53.161 INFO:tasks.workunit.client.1.smithi134.stdout:2/630: link d2/d15/d24/l31 d2/d15/d24/d3b/d8a/dc7/ld7 0 2024-04-22T23:23:53.162 INFO:tasks.workunit.client.1.smithi134.stdout:1/881: chown db/d15/d17/d20/d2a/f9f 0 1 2024-04-22T23:23:53.163 INFO:tasks.workunit.client.1.smithi134.stdout:1/882: read db/fa6 [637447,13196] 0 2024-04-22T23:23:53.166 INFO:tasks.workunit.client.1.smithi134.stdout:5/805: rename d9/d11/d34/d85/fd3 to d9/d11/d73/d95/d20/d108/f116 0 2024-04-22T23:23:53.169 INFO:tasks.workunit.client.1.smithi134.stdout:9/669: link d0/d7/d23/d26/d51/d88/fcd d0/d2d/d94/fe6 0 2024-04-22T23:23:53.170 INFO:tasks.workunit.client.1.smithi134.stdout:9/670: chown d0/d7/d23/d26/d51/d88/db9/lbe 51141007 1 2024-04-22T23:23:53.171 INFO:tasks.workunit.client.1.smithi134.stdout:9/671: chown d0/d7/d23/d26/d4d/d5d/d8e 2090148580 1 2024-04-22T23:23:53.173 INFO:tasks.workunit.client.1.smithi134.stdout:9/672: write d0/d7/d23/d1b/f64 [4920395,20273] 0 2024-04-22T23:23:53.173 INFO:tasks.workunit.client.1.smithi134.stdout:8/735: symlink d7/dc/d19/da5/lea 0 2024-04-22T23:23:53.175 INFO:tasks.workunit.client.1.smithi134.stdout:8/736: write d7/dc/d2c/d4f/d6c/d6e/f91 [196427,65754] 0 2024-04-22T23:23:53.177 INFO:tasks.workunit.client.1.smithi134.stdout:0/712: link d1/d5/ca5 d1/d5/d58/d6b/cf0 0 2024-04-22T23:23:53.177 INFO:tasks.workunit.client.1.smithi134.stdout:8/737: dread - d7/db/d2d/d39/d58/fe5 zero size 2024-04-22T23:23:53.178 INFO:tasks.workunit.client.1.smithi134.stdout:1/883: dread db/d15/d17/d20/d2a/f44 [0,4194304] 0 2024-04-22T23:23:53.178 INFO:tasks.workunit.client.1.smithi134.stdout:0/713: fsync d1/d5/f50 0 2024-04-22T23:23:53.182 INFO:tasks.workunit.client.1.smithi134.stdout:1/884: truncate db/d15/d17/d20/d2a/d2d/d108/d114/f9a 4839081 0 2024-04-22T23:23:53.187 INFO:tasks.workunit.client.1.smithi134.stdout:0/714: dwrite d1/d5/d58/d22/fbb [0,4194304] 0 2024-04-22T23:23:53.189 INFO:tasks.workunit.client.1.smithi134.stdout:9/673: creat d0/d7/d23/d26/dbc/fe7 x:0 0 0 2024-04-22T23:23:53.193 INFO:tasks.workunit.client.1.smithi134.stdout:2/631: mknod d2/dd/d68/cd8 0 2024-04-22T23:23:53.194 INFO:tasks.workunit.client.1.smithi134.stdout:2/632: read - d2/d94/fd4 zero size 2024-04-22T23:23:53.198 INFO:tasks.workunit.client.1.smithi134.stdout:5/806: mkdir d9/d11/d73/d95/d20/d117 0 2024-04-22T23:23:53.199 INFO:tasks.workunit.client.1.smithi134.stdout:8/738: dread d7/dc/d2c/d4f/f6b [0,4194304] 0 2024-04-22T23:23:53.199 INFO:tasks.workunit.client.1.smithi134.stdout:1/885: mkdir db/d15/d17/d20/d2a/d125 0 2024-04-22T23:23:53.201 INFO:tasks.workunit.client.1.smithi134.stdout:4/905: truncate d4/d35/d8e/f8a 1043682 0 2024-04-22T23:23:53.202 INFO:tasks.workunit.client.1.smithi134.stdout:0/715: fsync d1/d5/d58/f2e 0 2024-04-22T23:23:53.203 INFO:tasks.workunit.client.1.smithi134.stdout:0/716: chown d1/db/f70 1798 1 2024-04-22T23:23:53.206 INFO:tasks.workunit.client.1.smithi134.stdout:2/633: mknod d2/d15/d24/d1d/dba/cd9 0 2024-04-22T23:23:53.208 INFO:tasks.workunit.client.1.smithi134.stdout:5/807: rename d9/da2/cb2 to d9/d1e/db4/c118 0 2024-04-22T23:23:53.210 INFO:tasks.workunit.client.1.smithi134.stdout:8/739: mkdir d7/db/d42/d65/d46/d9a/da3/d88/deb 0 2024-04-22T23:23:53.211 INFO:tasks.workunit.client.1.smithi134.stdout:8/740: readlink d7/db/d42/d65/d46/d84/l9b 0 2024-04-22T23:23:53.216 INFO:tasks.workunit.client.1.smithi134.stdout:9/674: mkdir d0/d7/d12/de8 0 2024-04-22T23:23:53.218 INFO:tasks.workunit.client.1.smithi134.stdout:2/634: mknod d2/d15/d24/d3b/cda 0 2024-04-22T23:23:53.218 INFO:tasks.workunit.client.1.smithi134.stdout:9/675: write d0/d7/d23/d26/f58 [1330856,77598] 0 2024-04-22T23:23:53.219 INFO:tasks.workunit.client.1.smithi134.stdout:9/676: stat d0/d2d/dc1 0 2024-04-22T23:23:53.220 INFO:tasks.workunit.client.1.smithi134.stdout:0/717: dwrite d1/d2/d1d/f5f [4194304,4194304] 0 2024-04-22T23:23:53.223 INFO:tasks.workunit.client.1.smithi134.stdout:4/906: write d4/d7/d32/d3a/f43 [486862,85241] 0 2024-04-22T23:23:53.230 INFO:tasks.workunit.client.1.smithi134.stdout:1/886: dwrite db/d16/dbc/de1/ff5 [0,4194304] 0 2024-04-22T23:23:53.232 INFO:tasks.workunit.client.1.smithi134.stdout:0/718: mknod d1/d5/dd/d20/d2c/cf1 0 2024-04-22T23:23:53.234 INFO:tasks.workunit.client.1.smithi134.stdout:5/808: write d9/dd/d17/f45 [2617797,15557] 0 2024-04-22T23:23:53.236 INFO:tasks.workunit.client.1.smithi134.stdout:2/635: write d2/f3e [4241578,120540] 0 2024-04-22T23:23:53.236 INFO:tasks.workunit.client.1.smithi134.stdout:5/809: read d9/d1e/d90/fa1 [4176878,66700] 0 2024-04-22T23:23:53.237 INFO:tasks.workunit.client.1.smithi134.stdout:2/636: fsync d2/d15/d24/f2a 0 2024-04-22T23:23:53.240 INFO:tasks.workunit.client.1.smithi134.stdout:9/677: symlink d0/d7/d23/d1b/le9 0 2024-04-22T23:23:53.245 INFO:tasks.workunit.client.1.smithi134.stdout:4/907: link d4/d35/d8e/d6e/fb6 d4/dd/da8/f119 0 2024-04-22T23:23:53.248 INFO:tasks.workunit.client.1.smithi134.stdout:5/810: unlink d9/dd/d17/f66 0 2024-04-22T23:23:53.249 INFO:tasks.workunit.client.1.smithi134.stdout:2/637: creat d2/dd/d4a/d73/fdb x:0 0 0 2024-04-22T23:23:53.249 INFO:tasks.workunit.client.1.smithi134.stdout:0/719: symlink d1/d5/d58/d6b/db9/def/lf2 0 2024-04-22T23:23:53.254 INFO:tasks.workunit.client.1.smithi134.stdout:4/908: creat d4/d7/d32/d3a/d93/d117/de7/f11a x:0 0 0 2024-04-22T23:23:53.258 INFO:tasks.workunit.client.1.smithi134.stdout:9/678: dread d0/d2d/d4e/f62 [0,4194304] 0 2024-04-22T23:23:53.259 INFO:tasks.workunit.client.1.smithi134.stdout:0/720: rmdir d1/d5/d58/d22/dd0 39 2024-04-22T23:23:53.260 INFO:tasks.workunit.client.1.smithi134.stdout:0/721: chown d1/d5/d15/fba 208 1 2024-04-22T23:23:53.263 INFO:tasks.workunit.client.1.smithi134.stdout:4/909: creat d4/dd/da8/dc2/f11b x:0 0 0 2024-04-22T23:23:53.267 INFO:tasks.workunit.client.1.smithi134.stdout:1/887: dwrite db/d15/d17/d20/d2a/d2d/f43 [0,4194304] 0 2024-04-22T23:23:53.271 INFO:tasks.workunit.client.1.smithi134.stdout:4/910: mkdir d4/d13/d11c 0 2024-04-22T23:23:53.273 INFO:tasks.workunit.client.1.smithi134.stdout:4/911: dread - d4/dd/da8/dc2/df8/f109 zero size 2024-04-22T23:23:53.275 INFO:tasks.workunit.client.1.smithi134.stdout:5/811: dread d9/dd/d17/d2a/f47 [0,4194304] 0 2024-04-22T23:23:53.275 INFO:tasks.workunit.client.1.smithi134.stdout:4/912: chown d4/f1d 15283 1 2024-04-22T23:23:53.278 INFO:tasks.workunit.client.1.smithi134.stdout:2/638: link d2/lc d2/dd/d77/ldc 0 2024-04-22T23:23:53.281 INFO:tasks.workunit.client.1.smithi134.stdout:4/913: creat d4/d7/d32/d84/f11d x:0 0 0 2024-04-22T23:23:53.283 INFO:tasks.workunit.client.1.smithi134.stdout:0/722: creat d1/d5/dd/ff3 x:0 0 0 2024-04-22T23:23:53.284 INFO:tasks.workunit.client.1.smithi134.stdout:2/639: mkdir d2/d42/d54/ddd 0 2024-04-22T23:23:53.285 INFO:tasks.workunit.client.1.smithi134.stdout:0/723: read d1/d5/d58/d22/fb3 [592063,39808] 0 2024-04-22T23:23:53.286 INFO:tasks.workunit.client.1.smithi134.stdout:4/914: mknod d4/d7/d32/d3a/d93/d117/dd5/c11e 0 2024-04-22T23:23:53.289 INFO:tasks.workunit.client.1.smithi134.stdout:2/640: creat d2/fde x:0 0 0 2024-04-22T23:23:53.290 INFO:tasks.workunit.client.1.smithi134.stdout:4/915: unlink d4/d35/d53/l9f 0 2024-04-22T23:23:53.294 INFO:tasks.workunit.client.1.smithi134.stdout:0/724: getdents d1 0 2024-04-22T23:23:53.295 INFO:tasks.workunit.client.1.smithi134.stdout:0/725: dread - d1/d5/d58/d6b/db9/d75/fb8 zero size 2024-04-22T23:23:53.295 INFO:tasks.workunit.client.1.smithi134.stdout:4/916: symlink d4/d7/d32/d3a/d93/db5/l11f 0 2024-04-22T23:23:53.296 INFO:tasks.workunit.client.1.smithi134.stdout:0/726: stat d1/d5/dd/l3f 0 2024-04-22T23:23:53.296 INFO:tasks.workunit.client.1.smithi134.stdout:2/641: write d2/dd/df/d9c/daa/db0/f95 [543369,13196] 0 2024-04-22T23:23:53.299 INFO:tasks.workunit.client.1.smithi134.stdout:0/727: creat d1/d5/ff4 x:0 0 0 2024-04-22T23:23:53.300 INFO:tasks.workunit.client.1.smithi134.stdout:0/728: fdatasync d1/d2/f74 0 2024-04-22T23:23:53.300 INFO:tasks.workunit.client.1.smithi134.stdout:4/917: mkdir d4/d13/d111/d120 0 2024-04-22T23:23:53.302 INFO:tasks.workunit.client.1.smithi134.stdout:5/812: write d9/d11/d34/fbd [299685,119454] 0 2024-04-22T23:23:53.305 INFO:tasks.workunit.client.1.smithi134.stdout:9/679: dwrite d0/d7/d27/fa5 [0,4194304] 0 2024-04-22T23:23:53.305 INFO:tasks.workunit.client.1.smithi134.stdout:2/642: write d2/d15/d24/d1d/d3d/d9a/fd5 [1502406,79278] 0 2024-04-22T23:23:53.306 INFO:tasks.workunit.client.1.smithi134.stdout:1/888: truncate db/d16/d56/f60 1626322 0 2024-04-22T23:23:53.306 INFO:tasks.workunit.client.1.smithi134.stdout:4/918: chown d4/d35/d8e/f8a 974061 1 2024-04-22T23:23:53.308 INFO:tasks.workunit.client.1.smithi134.stdout:2/643: chown d2/dd/d77/f7e 608174 1 2024-04-22T23:23:53.309 INFO:tasks.workunit.client.1.smithi134.stdout:1/889: chown db/d15/d17/d20/d2a/lb7 3 1 2024-04-22T23:23:53.310 INFO:tasks.workunit.client.1.smithi134.stdout:0/729: mknod d1/d5/d58/d22/dd0/cf5 0 2024-04-22T23:23:53.312 INFO:tasks.workunit.client.1.smithi134.stdout:2/644: fdatasync d2/d15/fd0 0 2024-04-22T23:23:53.314 INFO:tasks.workunit.client.1.smithi134.stdout:0/730: truncate d1/db/fa9 909486 0 2024-04-22T23:23:53.316 INFO:tasks.workunit.client.1.smithi134.stdout:5/813: dwrite d9/d11/d73/d95/d20/d108/f10a [0,4194304] 0 2024-04-22T23:23:53.323 INFO:tasks.workunit.client.1.smithi134.stdout:0/731: unlink d1/d5/dd/d20/fc9 0 2024-04-22T23:23:53.324 INFO:tasks.workunit.client.1.smithi134.stdout:0/732: chown d1/d2/d1d/f69 1363117 1 2024-04-22T23:23:53.328 INFO:tasks.workunit.client.1.smithi134.stdout:9/680: dread d0/d7/d12/f42 [0,4194304] 0 2024-04-22T23:23:53.329 INFO:tasks.workunit.client.1.smithi134.stdout:1/890: mkdir db/d15/d17/d20/d2a/d2d/d126 0 2024-04-22T23:23:53.333 INFO:tasks.workunit.client.1.smithi134.stdout:4/919: getdents d4/d13 0 2024-04-22T23:23:53.339 INFO:tasks.workunit.client.1.smithi134.stdout:5/814: write d9/d11/d34/f6a [4150797,81476] 0 2024-04-22T23:23:53.340 INFO:tasks.workunit.client.1.smithi134.stdout:5/815: fdatasync d9/d1e/fdf 0 2024-04-22T23:23:53.354 INFO:tasks.workunit.client.1.smithi134.stdout:7/781: sync 2024-04-22T23:23:53.355 INFO:tasks.workunit.client.1.smithi134.stdout:7/782: dread - d4/d4a/fae zero size 2024-04-22T23:23:53.355 INFO:tasks.workunit.client.1.smithi134.stdout:1/891: write db/d15/d17/d20/d2a/f91 [897051,101752] 0 2024-04-22T23:23:53.356 INFO:tasks.workunit.client.1.smithi134.stdout:9/681: creat d0/d7/d12/d22/fea x:0 0 0 2024-04-22T23:23:53.357 INFO:tasks.workunit.client.1.smithi134.stdout:9/682: stat d0/d37/l3a 0 2024-04-22T23:23:53.359 INFO:tasks.workunit.client.1.smithi134.stdout:5/816: creat d9/d11/d73/d95/f119 x:0 0 0 2024-04-22T23:23:53.361 INFO:tasks.workunit.client.1.smithi134.stdout:1/892: fdatasync db/fa6 0 2024-04-22T23:23:53.362 INFO:tasks.workunit.client.1.smithi134.stdout:1/893: write db/db0/fea [976240,38068] 0 2024-04-22T23:23:53.363 INFO:tasks.workunit.client.1.smithi134.stdout:7/783: rename d4/d8/d41/d76/dc1/f100 to d4/d8/d41/d76/dc1/d2f/d69/dab/de1/f119 0 2024-04-22T23:23:53.367 INFO:tasks.workunit.client.1.smithi134.stdout:9/683: unlink d0/d7/d27/f59 0 2024-04-22T23:23:53.369 INFO:tasks.workunit.client.1.smithi134.stdout:4/920: link d4/dd/da8/dc2/df8/ld7 d4/d13/d11c/l121 0 2024-04-22T23:23:53.369 INFO:tasks.workunit.client.1.smithi134.stdout:1/894: dwrite db/d15/d17/d20/d2a/d2d/d2e/f90 [0,4194304] 0 2024-04-22T23:23:53.375 INFO:tasks.workunit.client.1.smithi134.stdout:2/645: link d2/d42/d54/d33/d91/lb4 d2/d94/ldf 0 2024-04-22T23:23:53.376 INFO:tasks.workunit.client.1.smithi134.stdout:9/684: dwrite d0/d7/d23/d26/dbc/fce [0,4194304] 0 2024-04-22T23:23:53.383 INFO:tasks.workunit.client.1.smithi134.stdout:9/685: dread d0/d7/d12/f42 [0,4194304] 0 2024-04-22T23:23:53.383 INFO:tasks.workunit.client.1.smithi134.stdout:7/784: mknod d4/d8/d41/d5b/d112/c11a 0 2024-04-22T23:23:53.384 INFO:tasks.workunit.client.1.smithi134.stdout:9/686: chown d0/d2d/dc1/fc7 2001 1 2024-04-22T23:23:53.389 INFO:tasks.workunit.client.1.smithi134.stdout:1/895: rmdir db/d15/d17/d20/d2a/d125 0 2024-04-22T23:23:53.392 INFO:tasks.workunit.client.1.smithi134.stdout:9/687: getdents d0/d7/d12/de8 0 2024-04-22T23:23:53.395 INFO:tasks.workunit.client.1.smithi134.stdout:2/646: getdents d2/d42/d54/d33 0 2024-04-22T23:23:53.396 INFO:tasks.workunit.client.1.smithi134.stdout:1/896: dwrite db/d16/dbc/de1/df4/ffd [0,4194304] 0 2024-04-22T23:23:53.401 INFO:tasks.workunit.client.1.smithi134.stdout:9/688: creat d0/d7/d23/d26/d51/feb x:0 0 0 2024-04-22T23:23:53.406 INFO:tasks.workunit.client.1.smithi134.stdout:2/647: creat d2/d42/d54/d33/d6e/fe0 x:0 0 0 2024-04-22T23:23:53.408 INFO:tasks.workunit.client.1.smithi134.stdout:1/897: fdatasync db/d15/d17/d20/d2a/d2d/ff1 0 2024-04-22T23:23:53.412 INFO:tasks.workunit.client.1.smithi134.stdout:1/898: dread - db/db0/dc1/dd9/ff0 zero size 2024-04-22T23:23:53.412 INFO:tasks.workunit.client.1.smithi134.stdout:4/921: dwrite d4/d35/d53/fb8 [0,4194304] 0 2024-04-22T23:23:53.413 INFO:tasks.workunit.client.1.smithi134.stdout:2/648: readlink d2/d15/d24/l31 0 2024-04-22T23:23:53.415 INFO:tasks.workunit.client.1.smithi134.stdout:7/785: dwrite d4/d8/d41/d76/dc1/d2f/d69/f9a [0,4194304] 0 2024-04-22T23:23:53.419 INFO:tasks.workunit.client.1.smithi134.stdout:9/689: creat d0/d7/d23/d1b/de2/fec x:0 0 0 2024-04-22T23:23:53.420 INFO:tasks.workunit.client.1.smithi134.stdout:9/690: readlink d0/d7/d23/d26/d4d/l36 0 2024-04-22T23:23:53.421 INFO:tasks.workunit.client.1.smithi134.stdout:9/691: chown d0/d37/d76/cdd 15 1 2024-04-22T23:23:53.422 INFO:tasks.workunit.client.1.smithi134.stdout:9/692: chown d0/d37/d76 16245743 1 2024-04-22T23:23:53.422 INFO:tasks.workunit.client.1.smithi134.stdout:2/649: chown d2/l3 325797 1 2024-04-22T23:23:53.423 INFO:tasks.workunit.client.1.smithi134.stdout:1/899: symlink db/d15/d17/d20/d2a/l127 0 2024-04-22T23:23:53.425 INFO:tasks.workunit.client.1.smithi134.stdout:1/900: fsync db/d16/d87/fef 0 2024-04-22T23:23:53.425 INFO:tasks.workunit.client.1.smithi134.stdout:4/922: truncate d4/d7a/fd1 780977 0 2024-04-22T23:23:53.427 INFO:tasks.workunit.client.1.smithi134.stdout:4/923: fsync d4/d13/fd6 0 2024-04-22T23:23:53.428 INFO:tasks.workunit.client.1.smithi134.stdout:4/924: dread - d4/d7/d32/fc1 zero size 2024-04-22T23:23:53.429 INFO:tasks.workunit.client.1.smithi134.stdout:9/693: dread d0/d2d/d4e/f62 [0,4194304] 0 2024-04-22T23:23:53.429 INFO:tasks.workunit.client.1.smithi134.stdout:2/650: rmdir d2/dd/df/d9c 39 2024-04-22T23:23:53.432 INFO:tasks.workunit.client.1.smithi134.stdout:2/651: creat d2/d42/d54/d33/d6e/fe1 x:0 0 0 2024-04-22T23:23:53.435 INFO:tasks.workunit.client.1.smithi134.stdout:4/925: symlink d4/d35/l122 0 2024-04-22T23:23:53.437 INFO:tasks.workunit.client.1.smithi134.stdout:2/652: fsync d2/f2c 0 2024-04-22T23:23:53.437 INFO:tasks.workunit.client.1.smithi134.stdout:1/901: link db/db0/dc1/le5 db/d15/d17/d20/d2a/d2d/d108/d114/l128 0 2024-04-22T23:23:53.439 INFO:tasks.workunit.client.1.smithi134.stdout:9/694: dwrite d0/d7/d23/d26/d4d/fb6 [0,4194304] 0 2024-04-22T23:23:53.443 INFO:tasks.workunit.client.1.smithi134.stdout:1/902: mknod db/c129 0 2024-04-22T23:23:53.445 INFO:tasks.workunit.client.1.smithi134.stdout:9/695: write d0/d7/d23/d26/fe3 [2580173,18935] 0 2024-04-22T23:23:53.445 INFO:tasks.workunit.client.1.smithi134.stdout:7/786: dwrite d4/d8/d2a/f5f [0,4194304] 0 2024-04-22T23:23:53.446 INFO:tasks.workunit.client.1.smithi134.stdout:4/926: getdents d4/d35/d53 0 2024-04-22T23:23:53.447 INFO:tasks.workunit.client.1.smithi134.stdout:1/903: creat db/d16/d104/f12a x:0 0 0 2024-04-22T23:23:53.449 INFO:tasks.workunit.client.1.smithi134.stdout:7/787: creat d4/d8/d2a/f11b x:0 0 0 2024-04-22T23:23:53.450 INFO:tasks.workunit.client.1.smithi134.stdout:4/927: symlink d4/d13/l123 0 2024-04-22T23:23:53.453 INFO:tasks.workunit.client.1.smithi134.stdout:9/696: unlink d0/d7/d12/l7a 0 2024-04-22T23:23:53.457 INFO:tasks.workunit.client.1.smithi134.stdout:1/904: creat db/d107/f12b x:0 0 0 2024-04-22T23:23:53.459 INFO:tasks.workunit.client.1.smithi134.stdout:1/905: stat db/d15/d17/cd8 0 2024-04-22T23:23:53.459 INFO:tasks.workunit.client.1.smithi134.stdout:7/788: symlink d4/dad/l11c 0 2024-04-22T23:23:53.462 INFO:tasks.workunit.client.1.smithi134.stdout:9/697: dwrite d0/d7/d23/d26/d51/f9d [0,4194304] 0 2024-04-22T23:23:53.464 INFO:tasks.workunit.client.1.smithi134.stdout:4/928: mkdir d4/d7/d124 0 2024-04-22T23:23:53.467 INFO:tasks.workunit.client.1.smithi134.stdout:7/789: readlink d4/d8/d41/d76/dc1/d2f/d34/d62/l26 0 2024-04-22T23:23:53.467 INFO:tasks.workunit.client.1.smithi134.stdout:9/698: creat d0/d2d/d94/fed x:0 0 0 2024-04-22T23:23:53.469 INFO:tasks.workunit.client.1.smithi134.stdout:4/929: creat d4/d35/d8e/d5d/f125 x:0 0 0 2024-04-22T23:23:53.469 INFO:tasks.workunit.client.1.smithi134.stdout:9/699: dread - d0/d2d/d3d/f86 zero size 2024-04-22T23:23:53.474 INFO:tasks.workunit.client.1.smithi134.stdout:7/790: rename d4/d8/d41/d76/dc1/cba to d4/d8/d41/d76/dc1/d2f/d69/dab/d95/dec/c11d 0 2024-04-22T23:23:53.480 INFO:tasks.workunit.client.1.smithi134.stdout:7/791: dread - d4/d8/d2a/d42/d7d/fa4 zero size 2024-04-22T23:23:53.484 INFO:tasks.workunit.client.1.smithi134.stdout:7/792: symlink d4/l11e 0 2024-04-22T23:23:53.485 INFO:tasks.workunit.client.1.smithi134.stdout:1/906: dread db/ff [0,4194304] 0 2024-04-22T23:23:53.487 INFO:tasks.workunit.client.1.smithi134.stdout:4/930: rename d4/dd/da8/dc2/lf1 to d4/d7/d32/d3a/l126 0 2024-04-22T23:23:53.490 INFO:tasks.workunit.client.1.smithi134.stdout:1/907: stat db/d15/d17/d20/d2a/d2d/c81 0 2024-04-22T23:23:53.494 INFO:tasks.workunit.client.1.smithi134.stdout:4/931: truncate d4/d29/f77 391969 0 2024-04-22T23:23:53.495 INFO:tasks.workunit.client.1.smithi134.stdout:4/932: chown d4/dd/da8/dc2/df8/d25/f98 494 1 2024-04-22T23:23:53.495 INFO:tasks.workunit.client.1.smithi134.stdout:1/908: fsync db/f8b 0 2024-04-22T23:23:53.496 INFO:tasks.workunit.client.1.smithi134.stdout:4/933: truncate d4/d13/d68/df6/ffc 615459 0 2024-04-22T23:23:53.499 INFO:tasks.workunit.client.1.smithi134.stdout:9/700: rename d0/d2d/f6b to d0/d7/d12/de8/fee 0 2024-04-22T23:23:53.501 INFO:tasks.workunit.client.1.smithi134.stdout:7/793: truncate d4/d8/d41/d76/dc1/d2f/d70/d86/ff0 1012624 0 2024-04-22T23:23:53.501 INFO:tasks.workunit.client.1.smithi134.stdout:1/909: symlink db/d16/d104/l12c 0 2024-04-22T23:23:53.501 INFO:tasks.workunit.client.1.smithi134.stdout:7/794: fsync d4/d8/d101/f10a 0 2024-04-22T23:23:53.502 INFO:tasks.workunit.client.1.smithi134.stdout:4/934: mknod d4/d7/d32/d3a/dbc/dc4/c127 0 2024-04-22T23:23:53.507 INFO:tasks.workunit.client.1.smithi134.stdout:4/935: rmdir d4/d7/d32/d3a/d93/d117 39 2024-04-22T23:23:53.508 INFO:tasks.workunit.client.1.smithi134.stdout:4/936: chown d4/d7/d32/fc1 116 1 2024-04-22T23:23:53.509 INFO:tasks.workunit.client.1.smithi134.stdout:7/795: dwrite d4/d8/d41/d76/dc1/d2f/d70/f74 [0,4194304] 0 2024-04-22T23:23:53.521 INFO:tasks.workunit.client.1.smithi134.stdout:4/937: creat d4/dd/da8/dc2/f128 x:0 0 0 2024-04-22T23:23:53.524 INFO:tasks.workunit.client.1.smithi134.stdout:7/796: dwrite d4/d4a/f92 [0,4194304] 0 2024-04-22T23:23:53.524 INFO:tasks.workunit.client.1.smithi134.stdout:1/910: truncate db/d16/dbc/de1/df4/ffd 1282457 0 2024-04-22T23:23:53.528 INFO:tasks.workunit.client.1.smithi134.stdout:9/701: mknod d0/d2d/d94/d98/cef 0 2024-04-22T23:23:53.529 INFO:tasks.workunit.client.1.smithi134.stdout:4/938: fsync d4/d7/d32/d3a/d93/fae 0 2024-04-22T23:23:53.530 INFO:tasks.workunit.client.1.smithi134.stdout:1/911: stat db/d15/c80 0 2024-04-22T23:23:53.532 INFO:tasks.workunit.client.1.smithi134.stdout:7/797: symlink d4/d8/d41/d76/dc1/d2f/d69/dab/d95/dec/l11f 0 2024-04-22T23:23:53.537 INFO:tasks.workunit.client.1.smithi134.stdout:4/939: creat d4/d29/d116/f129 x:0 0 0 2024-04-22T23:23:53.537 INFO:tasks.workunit.client.1.smithi134.stdout:4/940: chown d4/d13/d68/c8f 3974917 1 2024-04-22T23:23:53.538 INFO:tasks.workunit.client.1.smithi134.stdout:4/941: chown d4/d6b/ca1 9381 1 2024-04-22T23:23:53.539 INFO:tasks.workunit.client.1.smithi134.stdout:7/798: fdatasync d4/d4a/f7e 0 2024-04-22T23:23:53.546 INFO:tasks.workunit.client.1.smithi134.stdout:1/912: link db/d15/d17/cf7 db/db0/dc1/c12d 0 2024-04-22T23:23:53.547 INFO:tasks.workunit.client.1.smithi134.stdout:9/702: read d0/d7/d27/f33 [2267015,30932] 0 2024-04-22T23:23:53.548 INFO:tasks.workunit.client.1.smithi134.stdout:9/703: read d0/d7/d23/fdb [2317751,27085] 0 2024-04-22T23:23:53.549 INFO:tasks.workunit.client.1.smithi134.stdout:9/704: write d0/d7/f5e [9733262,16439] 0 2024-04-22T23:23:53.576 INFO:tasks.workunit.client.1.smithi134.stdout:6/784: sync 2024-04-22T23:23:53.578 INFO:tasks.workunit.client.1.smithi134.stdout:9/705: chown d0/d2d/f9f 187835359 1 2024-04-22T23:23:53.579 INFO:tasks.workunit.client.1.smithi134.stdout:6/785: chown d0/d19/de/d12/d5e/d6d/d7a/d99/lf3 0 1 2024-04-22T23:23:53.581 INFO:tasks.workunit.client.1.smithi134.stdout:1/913: mkdir db/d16/d12e 0 2024-04-22T23:23:53.581 INFO:tasks.workunit.client.1.smithi134.stdout:6/786: fsync d0/d19/de/d12/d2a/d2d/d3a/f9a 0 2024-04-22T23:23:53.581 INFO:tasks.workunit.client.1.smithi134.stdout:4/942: getdents d4/dd/da8/dc2/df8/d25 0 2024-04-22T23:23:53.583 INFO:tasks.workunit.client.1.smithi134.stdout:7/799: rmdir d4/dad/dcd 0 2024-04-22T23:23:53.584 INFO:tasks.workunit.client.1.smithi134.stdout:9/706: creat d0/d7/d23/d26/d51/d4c/ff0 x:0 0 0 2024-04-22T23:23:53.590 INFO:tasks.workunit.client.1.smithi134.stdout:4/943: fsync d4/d29/dcc/f6f 0 2024-04-22T23:23:53.592 INFO:tasks.workunit.client.1.smithi134.stdout:7/800: chown d4/d8/d41/d76/dc1/d2f/d69/dab/de1/c10f 0 1 2024-04-22T23:23:53.597 INFO:tasks.workunit.client.1.smithi134.stdout:1/914: unlink db/d15/d17/d20/d2a/fc3 0 2024-04-22T23:23:53.599 INFO:tasks.workunit.client.1.smithi134.stdout:4/944: unlink d4/d13/d68/c10e 0 2024-04-22T23:23:53.600 INFO:tasks.workunit.client.1.smithi134.stdout:7/801: creat d4/d8/d2a/f120 x:0 0 0 2024-04-22T23:23:53.602 INFO:tasks.workunit.client.1.smithi134.stdout:1/915: write db/d15/d17/d20/d2a/d2d/f10a [746050,117589] 0 2024-04-22T23:23:53.609 INFO:tasks.workunit.client.1.smithi134.stdout:6/787: write d0/d19/de/d12/f61 [1587293,108817] 0 2024-04-22T23:23:53.612 INFO:tasks.workunit.client.1.smithi134.stdout:1/916: creat db/d15/d17/d20/d2a/d2d/d2e/f12f x:0 0 0 2024-04-22T23:23:53.613 INFO:tasks.workunit.client.1.smithi134.stdout:6/788: creat d0/d19/db/d75/d92/f100 x:0 0 0 2024-04-22T23:23:53.615 INFO:tasks.workunit.client.1.smithi134.stdout:4/945: dwrite d4/d7/d32/d3a/d93/d117/fb2 [0,4194304] 0 2024-04-22T23:23:53.617 INFO:tasks.workunit.client.1.smithi134.stdout:6/789: creat d0/d19/de/dbe/dd1/f101 x:0 0 0 2024-04-22T23:23:53.619 INFO:tasks.workunit.client.1.smithi134.stdout:9/707: truncate d0/d2d/d3d/f9e 1975088 0 2024-04-22T23:23:53.620 INFO:tasks.workunit.client.1.smithi134.stdout:7/802: write d4/d8/d3b/f48 [1007023,99028] 0 2024-04-22T23:23:53.621 INFO:tasks.workunit.client.1.smithi134.stdout:1/917: creat db/d15/d17/d20/d2a/d2d/d126/f130 x:0 0 0 2024-04-22T23:23:53.621 INFO:tasks.workunit.client.1.smithi134.stdout:7/803: chown d4/d4a/f7e 35616 1 2024-04-22T23:23:53.622 INFO:tasks.workunit.client.1.smithi134.stdout:6/790: symlink d0/d19/de/d12/d29/d74/deb/l102 0 2024-04-22T23:23:53.624 INFO:tasks.workunit.client.1.smithi134.stdout:1/918: rmdir db/db5 39 2024-04-22T23:23:53.624 INFO:tasks.workunit.client.1.smithi134.stdout:6/791: read - d0/d19/de/d12/d29/d74/da0/fab zero size 2024-04-22T23:23:53.625 INFO:tasks.workunit.client.1.smithi134.stdout:9/708: read d0/d2d/d82/da0/f8d [467391,118144] 0 2024-04-22T23:23:53.626 INFO:tasks.workunit.client.1.smithi134.stdout:7/804: creat d4/d8/d2a/f121 x:0 0 0 2024-04-22T23:23:53.626 INFO:tasks.workunit.client.1.smithi134.stdout:6/792: dread - d0/d19/de/d12/d5e/d6d/d7a/fde zero size 2024-04-22T23:23:53.628 INFO:tasks.workunit.client.1.smithi134.stdout:4/946: creat d4/d7/d32/d3a/d93/d117/f12a x:0 0 0 2024-04-22T23:23:53.628 INFO:tasks.workunit.client.1.smithi134.stdout:9/709: dread d0/d7/f5b [0,4194304] 0 2024-04-22T23:23:53.628 INFO:tasks.workunit.client.1.smithi134.stdout:4/947: chown d4/d35/ce2 4528 1 2024-04-22T23:23:53.631 INFO:tasks.workunit.client.1.smithi134.stdout:7/805: creat d4/d8/d41/d5b/d112/f122 x:0 0 0 2024-04-22T23:23:53.632 INFO:tasks.workunit.client.1.smithi134.stdout:4/948: symlink d4/d6b/dd9/dea/l12b 0 2024-04-22T23:23:53.633 INFO:tasks.workunit.client.1.smithi134.stdout:4/949: stat d4/d7/l100 0 2024-04-22T23:23:53.634 INFO:tasks.workunit.client.1.smithi134.stdout:1/919: creat db/d15/d121/f131 x:0 0 0 2024-04-22T23:23:53.635 INFO:tasks.workunit.client.1.smithi134.stdout:4/950: stat d4/dd/da8/dc2/df8/f94 0 2024-04-22T23:23:53.636 INFO:tasks.workunit.client.1.smithi134.stdout:7/806: mknod d4/d8/d41/d76/dc1/d2f/c123 0 2024-04-22T23:23:53.637 INFO:tasks.workunit.client.1.smithi134.stdout:4/951: creat d4/d7/d32/dc5/f12c x:0 0 0 2024-04-22T23:23:53.638 INFO:tasks.workunit.client.1.smithi134.stdout:7/807: mkdir d4/d8/d41/d76/dc1/d2f/d124 0 2024-04-22T23:23:53.638 INFO:tasks.workunit.client.1.smithi134.stdout:4/952: chown d4/d6b/ca1 11894469 1 2024-04-22T23:23:53.639 INFO:tasks.workunit.client.1.smithi134.stdout:7/808: fsync d4/d8/d41/d76/dc1/d2f/d34/f67 0 2024-04-22T23:23:53.641 INFO:tasks.workunit.client.1.smithi134.stdout:7/809: write d4/d8/d41/d76/dc1/d2f/d69/f9a [1038039,124374] 0 2024-04-22T23:23:53.646 INFO:tasks.workunit.client.1.smithi134.stdout:7/810: truncate d4/d8/d41/d76/dc1/d2f/d34/d62/f56 1446922 0 2024-04-22T23:23:53.660 INFO:tasks.workunit.client.1.smithi134.stdout:4/953: dwrite d4/fa5 [0,4194304] 0 2024-04-22T23:23:53.668 INFO:tasks.workunit.client.1.smithi134.stdout:4/954: dwrite d4/dd/da8/dc2/df8/d25/dfd/f101 [0,4194304] 0 2024-04-22T23:23:53.673 INFO:tasks.workunit.client.1.smithi134.stdout:4/955: mkdir d4/d7/d32/d3a/d93/d12d 0 2024-04-22T23:23:53.676 INFO:tasks.workunit.client.1.smithi134.stdout:4/956: rename d4/d7/d32/d3a/d93/d117/fde to d4/d6b/f12e 0 2024-04-22T23:23:53.682 INFO:tasks.workunit.client.1.smithi134.stdout:4/957: dwrite d4/d7/d32/d84/f11d [0,4194304] 0 2024-04-22T23:23:53.684 INFO:tasks.workunit.client.1.smithi134.stdout:4/958: mkdir d4/d35/d8e/d5d/d12f 0 2024-04-22T23:23:53.685 INFO:tasks.workunit.client.1.smithi134.stdout:4/959: read - d4/d13/d111/f114 zero size 2024-04-22T23:23:53.699 INFO:tasks.workunit.client.1.smithi134.stdout:4/960: write d4/d35/d8e/d5d/fa9 [84809,60765] 0 2024-04-22T23:23:53.700 INFO:tasks.workunit.client.1.smithi134.stdout:4/961: readlink d4/d13/lb7 0 2024-04-22T23:23:53.702 INFO:tasks.workunit.client.1.smithi134.stdout:4/962: mkdir d4/d13/d111/d130 0 2024-04-22T23:23:53.704 INFO:tasks.workunit.client.1.smithi134.stdout:4/963: unlink d4/d35/d53/fb8 0 2024-04-22T23:23:53.705 INFO:tasks.workunit.client.1.smithi134.stdout:4/964: chown d4/d7/d32/d84/df4/lb3 380128502 1 2024-04-22T23:23:53.706 INFO:tasks.workunit.client.1.smithi134.stdout:4/965: write d4/d29/d116/f129 [165603,70680] 0 2024-04-22T23:23:53.707 INFO:tasks.workunit.client.1.smithi134.stdout:4/966: chown d4/d29/dcc/l50 3911 1 2024-04-22T23:23:53.709 INFO:tasks.workunit.client.1.smithi134.stdout:4/967: dread d4/dd/da8/dc2/df8/f97 [0,4194304] 0 2024-04-22T23:23:53.716 INFO:tasks.workunit.client.1.smithi134.stdout:4/968: dwrite d4/d13/fca [4194304,4194304] 0 2024-04-22T23:23:53.717 INFO:tasks.workunit.client.1.smithi134.stdout:4/969: write d4/d35/db4/f10a [878282,108169] 0 2024-04-22T23:23:53.721 INFO:tasks.workunit.client.1.smithi134.stdout:4/970: creat d4/d7/d32/d3a/dbc/dc4/f131 x:0 0 0 2024-04-22T23:23:53.725 INFO:tasks.workunit.client.1.smithi134.stdout:4/971: rename d4/dd/da8/dc2/df8/f1f to d4/d35/f132 0 2024-04-22T23:23:53.728 INFO:tasks.workunit.client.1.smithi134.stdout:4/972: mkdir d4/d7/d32/d3a/d93/d12d/d133 0 2024-04-22T23:23:53.730 INFO:tasks.workunit.client.1.smithi134.stdout:4/973: mknod d4/d35/c134 0 2024-04-22T23:23:53.732 INFO:tasks.workunit.client.1.smithi134.stdout:4/974: rmdir d4/d29/d116 39 2024-04-22T23:23:53.733 INFO:tasks.workunit.client.1.smithi134.stdout:4/975: stat d4/d7/d32/d3a/dbc/dc4/c127 0 2024-04-22T23:23:53.734 INFO:tasks.workunit.client.1.smithi134.stdout:4/976: write d4/d7/d32/d3a/d93/d117/f12a [515766,24602] 0 2024-04-22T23:23:53.737 INFO:tasks.workunit.client.1.smithi134.stdout:4/977: truncate d4/d7/d32/d3a/d93/db5/ffa 634310 0 2024-04-22T23:23:53.743 INFO:tasks.workunit.client.1.smithi134.stdout:4/978: link d4/d29/d116/f69 d4/d7/d32/f135 0 2024-04-22T23:23:53.746 INFO:tasks.workunit.client.1.smithi134.stdout:4/979: link d4/d7/d32/ca7 d4/d13/d68/c136 0 2024-04-22T23:23:53.749 INFO:tasks.workunit.client.1.smithi134.stdout:4/980: creat d4/d7/d32/f137 x:0 0 0 2024-04-22T23:23:53.751 INFO:tasks.workunit.client.1.smithi134.stdout:3/525: sync 2024-04-22T23:23:53.752 INFO:tasks.workunit.client.1.smithi134.stdout:4/981: symlink d4/d7/d32/l138 0 2024-04-22T23:23:53.752 INFO:tasks.workunit.client.1.smithi134.stdout:4/982: fdatasync d4/d7/d32/fc3 0 2024-04-22T23:23:53.755 INFO:tasks.workunit.client.1.smithi134.stdout:3/526: unlink dc/d2a/d62/db2/d34/d70/f9b 0 2024-04-22T23:23:53.766 INFO:tasks.workunit.client.1.smithi134.stdout:3/527: symlink dc/d2a/d62/db2/d2f/d21/d42/d77/lc2 0 2024-04-22T23:23:53.769 INFO:tasks.workunit.client.1.smithi134.stdout:3/528: mkdir dc/d2a/d62/db2/d34/d6b/dae/dc3 0 2024-04-22T23:23:53.769 INFO:tasks.workunit.client.1.smithi134.stdout:3/529: chown dc/d2a/d62/db2/d3b 65555018 1 2024-04-22T23:23:53.771 INFO:tasks.workunit.client.1.smithi134.stdout:3/530: write dc/d2a/d62/db2/d3b/fb0 [646152,111156] 0 2024-04-22T23:23:53.772 INFO:tasks.workunit.client.1.smithi134.stdout:3/531: write dc/d2a/d62/db2/d34/d6b/dae/f85 [398750,98499] 0 2024-04-22T23:23:53.774 INFO:tasks.workunit.client.1.smithi134.stdout:3/532: write dc/d2a/d62/db2/d2f/d36/d6d/fb4 [647171,35896] 0 2024-04-22T23:23:53.777 INFO:tasks.workunit.client.1.smithi134.stdout:3/533: symlink dc/dd/lc4 0 2024-04-22T23:23:53.779 INFO:tasks.workunit.client.1.smithi134.stdout:3/534: mknod dc/d2a/d62/db2/d2f/d54/cc5 0 2024-04-22T23:23:53.788 INFO:tasks.workunit.client.1.smithi134.stdout:3/535: write dc/d2a/d62/f7f [69884,83424] 0 2024-04-22T23:23:53.791 INFO:tasks.workunit.client.1.smithi134.stdout:3/536: unlink dc/d2a/d62/db2/f39 0 2024-04-22T23:23:53.794 INFO:tasks.workunit.client.1.smithi134.stdout:3/537: mknod dc/d2a/d62/db2/d2f/d54/dac/cc6 0 2024-04-22T23:23:53.795 INFO:tasks.workunit.client.1.smithi134.stdout:3/538: chown dc/d2a/d62/db2/d2f/d54/f5e 18 1 2024-04-22T23:23:53.797 INFO:tasks.workunit.client.1.smithi134.stdout:3/539: mkdir dc/d2a/d62/d86/dc7 0 2024-04-22T23:23:53.798 INFO:tasks.workunit.client.1.smithi134.stdout:3/540: stat dc/d2a/d62/db2/d2f/d36/d6d/d80/l9e 0 2024-04-22T23:23:53.801 INFO:tasks.workunit.client.1.smithi134.stdout:3/541: rename dc/d2a/d62/db2/d34/f48 to dc/d2a/d62/d86/db3/fc8 0 2024-04-22T23:23:53.803 INFO:tasks.workunit.client.1.smithi134.stdout:3/542: creat dc/d2a/d62/db2/d34/d6b/fc9 x:0 0 0 2024-04-22T23:23:53.805 INFO:tasks.workunit.client.1.smithi134.stdout:3/543: unlink dc/d2a/d62/db2/d34/d6b/fc9 0 2024-04-22T23:23:53.806 INFO:tasks.workunit.client.1.smithi134.stdout:3/544: write dc/d2a/d62/db2/d2f/d21/d42/f82 [35800,13685] 0 2024-04-22T23:23:53.808 INFO:tasks.workunit.client.1.smithi134.stdout:3/545: rmdir dc/d2a/d62/db2/d2f/d21 39 2024-04-22T23:23:53.812 INFO:tasks.workunit.client.1.smithi134.stdout:3/546: rename dc/d2a/d62/db2/d2f/d54/dac/cc6 to dc/d2a/d62/db2/d2f/d61/d6a/cca 0 2024-04-22T23:23:53.817 INFO:tasks.workunit.client.1.smithi134.stdout:3/547: dwrite dc/dd/f7a [4194304,4194304] 0 2024-04-22T23:23:53.822 INFO:tasks.workunit.client.1.smithi134.stdout:3/548: rename dc/l53 to dc/d2a/d62/d86/db3/lcb 0 2024-04-22T23:23:53.942 INFO:tasks.workunit.client.1.smithi134.stdout:8/741: sync 2024-04-22T23:23:53.944 INFO:tasks.workunit.client.1.smithi134.stdout:8/742: creat d7/dc/d2c/d4f/d6c/fec x:0 0 0 2024-04-22T23:23:53.948 INFO:tasks.workunit.client.1.smithi134.stdout:8/743: getdents d7/db/d42 0 2024-04-22T23:23:53.949 INFO:tasks.workunit.client.1.smithi134.stdout:8/744: write d7/dc/d19/da5/fab [259208,68924] 0 2024-04-22T23:23:53.955 INFO:tasks.workunit.client.1.smithi134.stdout:8/745: truncate d7/dc/ff 2835047 0 2024-04-22T23:23:53.959 INFO:tasks.workunit.client.1.smithi134.stdout:8/746: write d7/dc/d2c/d4f/f6b [4882552,116134] 0 2024-04-22T23:23:53.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:53 smithi134 ceph-mon[34609]: pgmap v282: 65 pgs: 65 active+clean; 2.8 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 68 MiB/s rd, 191 MiB/s wr, 432 op/s 2024-04-22T23:23:54.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:53 smithi067 ceph-mon[26453]: pgmap v282: 65 pgs: 65 active+clean; 2.8 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 68 MiB/s rd, 191 MiB/s wr, 432 op/s 2024-04-22T23:23:54.120 INFO:tasks.workunit.client.1.smithi134.stdout:0/733: sync 2024-04-22T23:23:54.122 INFO:tasks.workunit.client.1.smithi134.stdout:0/734: creat d1/db/ff6 x:0 0 0 2024-04-22T23:23:54.124 INFO:tasks.workunit.client.1.smithi134.stdout:0/735: write d1/d5/fc1 [86879,52987] 0 2024-04-22T23:23:54.125 INFO:tasks.workunit.client.1.smithi134.stdout:0/736: write d1/d5/fbe [386908,58048] 0 2024-04-22T23:23:54.128 INFO:tasks.workunit.client.1.smithi134.stdout:0/737: dread - d1/d5/dd/d41/f5d zero size 2024-04-22T23:23:54.129 INFO:tasks.workunit.client.1.smithi134.stdout:0/738: fsync d1/d2/d1d/f5f 0 2024-04-22T23:23:54.136 INFO:tasks.workunit.client.1.smithi134.stdout:0/739: dwrite d1/d5/d15/fe9 [0,4194304] 0 2024-04-22T23:23:54.143 INFO:tasks.workunit.client.1.smithi134.stdout:0/740: dwrite d1/d5/d58/d22/f24 [8388608,4194304] 0 2024-04-22T23:23:54.146 INFO:tasks.workunit.client.1.smithi134.stdout:0/741: write d1/d5/d15/fba [438975,17170] 0 2024-04-22T23:23:54.147 INFO:tasks.workunit.client.1.smithi134.stdout:0/742: chown d1/d5/d58/l61 90903303 1 2024-04-22T23:23:54.151 INFO:tasks.workunit.client.1.smithi134.stdout:0/743: getdents d1/d5/dd/d68/dc0 0 2024-04-22T23:23:54.155 INFO:tasks.workunit.client.1.smithi134.stdout:0/744: dread d1/d2/d1d/f5f [4194304,4194304] 0 2024-04-22T23:23:54.307 INFO:tasks.workunit.client.1.smithi134.stdout:5/817: sync 2024-04-22T23:23:54.321 INFO:tasks.workunit.client.1.smithi134.stdout:5/818: dread f8 [0,4194304] 0 2024-04-22T23:23:54.322 INFO:tasks.workunit.client.1.smithi134.stdout:5/819: chown d9/da2/de6/dc0/leb 19701 1 2024-04-22T23:23:54.323 INFO:tasks.workunit.client.1.smithi134.stdout:5/820: read d9/da2/fdb [808072,46492] 0 2024-04-22T23:23:54.324 INFO:tasks.workunit.client.1.smithi134.stdout:5/821: creat d9/dd/df1/f11a x:0 0 0 2024-04-22T23:23:54.326 INFO:tasks.workunit.client.1.smithi134.stdout:5/822: read d9/dd/da6/fc6 [3371792,125586] 0 2024-04-22T23:23:54.336 INFO:tasks.workunit.client.1.smithi134.stdout:5/823: dwrite d9/d1e/fb6 [0,4194304] 0 2024-04-22T23:23:54.337 INFO:tasks.workunit.client.1.smithi134.stdout:5/824: write d9/dd/d17/d44/f59 [513144,125342] 0 2024-04-22T23:23:54.359 INFO:tasks.workunit.client.1.smithi134.stdout:5/825: dread d9/da2/faa [0,4194304] 0 2024-04-22T23:23:54.362 INFO:tasks.workunit.client.1.smithi134.stdout:5/826: rename d9/dd to d9/dbb/d11b 0 2024-04-22T23:23:54.370 INFO:tasks.workunit.client.1.smithi134.stdout:5/827: dwrite d9/d1e/f23 [0,4194304] 0 2024-04-22T23:23:54.371 INFO:tasks.workunit.client.1.smithi134.stdout:5/828: readlink d9/d11/d73/d95/l7c 0 2024-04-22T23:23:54.376 INFO:tasks.workunit.client.1.smithi134.stdout:5/829: dread d9/d11/d34/fbd [0,4194304] 0 2024-04-22T23:23:54.377 INFO:tasks.workunit.client.1.smithi134.stdout:5/830: write d9/d11/d34/f54 [4297127,3143] 0 2024-04-22T23:23:54.379 INFO:tasks.workunit.client.1.smithi134.stdout:5/831: write d9/d11/d34/d4d/fde [613251,41483] 0 2024-04-22T23:23:54.380 INFO:tasks.workunit.client.1.smithi134.stdout:5/832: readlink d9/d11/d73/d95/d20/l57 0 2024-04-22T23:23:54.382 INFO:tasks.workunit.client.1.smithi134.stdout:5/833: fdatasync d9/dbb/d11b/d17/d2a/f30 0 2024-04-22T23:23:54.394 INFO:tasks.workunit.client.1.smithi134.stdout:5/834: truncate d9/d11/d73/d95/d20/f91 4021836 0 2024-04-22T23:23:54.400 INFO:tasks.workunit.client.1.smithi134.stdout:5/835: write d9/d11/d34/d4d/faf [4789989,129724] 0 2024-04-22T23:23:54.403 INFO:tasks.workunit.client.1.smithi134.stdout:5/836: mkdir d9/dbb/d11b/da6/d114/d11c 0 2024-04-22T23:23:54.404 INFO:tasks.workunit.client.1.smithi134.stdout:5/837: stat d9/dbb/d11b/d88/ffb 0 2024-04-22T23:23:54.405 INFO:tasks.workunit.client.1.smithi134.stdout:5/838: chown d9/dbb/d11b/d17/f45 46 1 2024-04-22T23:23:54.411 INFO:tasks.workunit.client.1.smithi134.stdout:5/839: rmdir d9/dbb 39 2024-04-22T23:23:54.414 INFO:tasks.workunit.client.1.smithi134.stdout:5/840: mknod d9/dbb/d11b/d17/d2a/d49/c11d 0 2024-04-22T23:23:54.416 INFO:tasks.workunit.client.1.smithi134.stdout:5/841: creat d9/d1e/d90/f11e x:0 0 0 2024-04-22T23:23:54.418 INFO:tasks.workunit.client.1.smithi134.stdout:5/842: symlink d9/d11/d73/d95/l11f 0 2024-04-22T23:23:54.421 INFO:tasks.workunit.client.1.smithi134.stdout:5/843: getdents d9/dbb/d11b/d17/d2a/de5 0 2024-04-22T23:23:54.423 INFO:tasks.workunit.client.1.smithi134.stdout:5/844: creat d9/dbb/d11b/d88/db7/f120 x:0 0 0 2024-04-22T23:23:54.425 INFO:tasks.workunit.client.1.smithi134.stdout:5/845: mkdir d9/d11/d34/d4d/dd4/d121 0 2024-04-22T23:23:54.429 INFO:tasks.workunit.client.1.smithi134.stdout:5/846: link d9/d1e/l72 d9/dbb/d11b/l122 0 2024-04-22T23:23:54.430 INFO:tasks.workunit.client.1.smithi134.stdout:5/847: fdatasync d9/dbb/d11b/d17/d44/f7e 0 2024-04-22T23:23:54.438 INFO:tasks.workunit.client.1.smithi134.stdout:2/653: sync 2024-04-22T23:23:54.443 INFO:tasks.workunit.client.1.smithi134.stdout:5/848: dread d9/f62 [0,4194304] 0 2024-04-22T23:23:54.445 INFO:tasks.workunit.client.1.smithi134.stdout:5/849: chown d9/d11/d34/d85/ld5 16875 1 2024-04-22T23:23:54.452 INFO:tasks.workunit.client.1.smithi134.stdout:5/850: dwrite d9/d11/d73/d95/f119 [0,4194304] 0 2024-04-22T23:23:54.456 INFO:tasks.workunit.client.1.smithi134.stdout:5/851: creat d9/d11/d73/d95/d20/d60/f123 x:0 0 0 2024-04-22T23:23:54.458 INFO:tasks.workunit.client.1.smithi134.stdout:5/852: creat d9/dbb/d11b/d17/d44/f124 x:0 0 0 2024-04-22T23:23:54.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.547+0000 7fa8f8caa700 1 -- 172.21.15.67:0/3474225172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f4101ea0 msgr2=0x7fa8f4102300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:54.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.547+0000 7fa8f8caa700 1 --2- 172.21.15.67:0/3474225172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f4101ea0 0x7fa8f4102300 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa8e8009910 tx=0x7fa8e8009c20 comp rx=0 tx=0).stop 2024-04-22T23:23:54.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.547+0000 7fa8f8caa700 1 -- 172.21.15.67:0/3474225172 shutdown_connections 2024-04-22T23:23:54.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.547+0000 7fa8f8caa700 1 --2- 172.21.15.67:0/3474225172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f4101ea0 0x7fa8f4102300 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:54.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.547+0000 7fa8f8caa700 1 --2- 172.21.15.67:0/3474225172 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa8f4100ca0 0x7fa8f41010c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:54.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.547+0000 7fa8f8caa700 1 -- 172.21.15.67:0/3474225172 >> 172.21.15.67:0/3474225172 conn(0x7fa8f40fc260 msgr2=0x7fa8f40fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:23:54.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.547+0000 7fa8f8caa700 1 -- 172.21.15.67:0/3474225172 shutdown_connections 2024-04-22T23:23:54.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.547+0000 7fa8f8caa700 1 -- 172.21.15.67:0/3474225172 wait complete. 2024-04-22T23:23:54.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.548+0000 7fa8f8caa700 1 Processor -- start 2024-04-22T23:23:54.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.549+0000 7fa8f8caa700 1 -- start start 2024-04-22T23:23:54.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.549+0000 7fa8f8caa700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f4100ca0 0x7fa8f4074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:23:54.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.549+0000 7fa8f8caa700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa8f4101ea0 0x7fa8f4073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:23:54.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.549+0000 7fa8f8caa700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f419a130 con 0x7fa8f4100ca0 2024-04-22T23:23:54.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.549+0000 7fa8f8caa700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f419a270 con 0x7fa8f4101ea0 2024-04-22T23:23:54.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.549+0000 7fa8f259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f4100ca0 0x7fa8f4074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:23:54.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.549+0000 7fa8f259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f4100ca0 0x7fa8f4074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53452/0 (socket says 172.21.15.67:53452) 2024-04-22T23:23:54.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.549+0000 7fa8f259c700 1 -- 172.21.15.67:0/2529558521 learned_addr learned my addr 172.21.15.67:0/2529558521 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:23:54.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.549+0000 7fa8f1d9b700 1 --2- 172.21.15.67:0/2529558521 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa8f4101ea0 0x7fa8f4073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:23:54.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.550+0000 7fa8f259c700 1 -- 172.21.15.67:0/2529558521 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa8f4101ea0 msgr2=0x7fa8f4073180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:54.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.550+0000 7fa8f259c700 1 --2- 172.21.15.67:0/2529558521 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa8f4101ea0 0x7fa8f4073180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:54.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.550+0000 7fa8f259c700 1 -- 172.21.15.67:0/2529558521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e80095f0 con 0x7fa8f4100ca0 2024-04-22T23:23:54.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.550+0000 7fa8f259c700 1 --2- 172.21.15.67:0/2529558521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f4100ca0 0x7fa8f4074b30 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa8e400e8f0 tx=0x7fa8e400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:23:54.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.551+0000 7fa8e37fe700 1 -- 172.21.15.67:0/2529558521 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8e4009790 con 0x7fa8f4100ca0 2024-04-22T23:23:54.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.551+0000 7fa8f8caa700 1 -- 172.21.15.67:0/2529558521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8f40736c0 con 0x7fa8f4100ca0 2024-04-22T23:23:54.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.551+0000 7fa8e37fe700 1 -- 172.21.15.67:0/2529558521 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fa8e40044d0 con 0x7fa8f4100ca0 2024-04-22T23:23:54.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.551+0000 7fa8e37fe700 1 -- 172.21.15.67:0/2529558521 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8e4003da0 con 0x7fa8f4100ca0 2024-04-22T23:23:54.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.551+0000 7fa8f8caa700 1 -- 172.21.15.67:0/2529558521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8f4073be0 con 0x7fa8f4100ca0 2024-04-22T23:23:54.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.553+0000 7fa8f8caa700 1 -- 172.21.15.67:0/2529558521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8f4073ec0 con 0x7fa8f4100ca0 2024-04-22T23:23:54.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.553+0000 7fa8e37fe700 1 -- 172.21.15.67:0/2529558521 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 20) v1 ==== 91455+0+0 (secure 0 0 0) 0x7fa8e40104c0 con 0x7fa8f4100ca0 2024-04-22T23:23:54.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.554+0000 7fa8e37fe700 1 --2- 172.21.15.67:0/2529558521 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa8dc06d780 0x7fa8dc06fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:23:54.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.554+0000 7fa8e37fe700 1 -- 172.21.15.67:0/2529558521 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fa8e4014070 con 0x7fa8f4100ca0 2024-04-22T23:23:54.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.554+0000 7fa8f1d9b700 1 --2- 172.21.15.67:0/2529558521 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa8dc06d780 0x7fa8dc06fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:23:54.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.555+0000 7fa8f1d9b700 1 --2- 172.21.15.67:0/2529558521 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa8dc06d780 0x7fa8dc06fc40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa8e80059d0 tx=0x7fa8e8005940 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:23:54.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.563+0000 7fa8e37fe700 1 -- 172.21.15.67:0/2529558521 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa8e405b5e0 con 0x7fa8f4100ca0 2024-04-22T23:23:54.575 INFO:tasks.workunit.client.1.smithi134.stdout:3/549: sync 2024-04-22T23:23:54.575 INFO:tasks.workunit.client.1.smithi134.stdout:7/811: sync 2024-04-22T23:23:54.575 INFO:tasks.workunit.client.1.smithi134.stdout:9/710: sync 2024-04-22T23:23:54.575 INFO:tasks.workunit.client.1.smithi134.stdout:1/920: sync 2024-04-22T23:23:54.575 INFO:tasks.workunit.client.1.smithi134.stdout:0/745: sync 2024-04-22T23:23:54.575 INFO:tasks.workunit.client.1.smithi134.stdout:6/793: sync 2024-04-22T23:23:54.575 INFO:tasks.workunit.client.1.smithi134.stdout:8/747: sync 2024-04-22T23:23:54.576 INFO:tasks.workunit.client.1.smithi134.stdout:4/983: sync 2024-04-22T23:23:54.579 INFO:tasks.workunit.client.1.smithi134.stdout:1/921: readlink db/d16/l75 0 2024-04-22T23:23:54.582 INFO:tasks.workunit.client.1.smithi134.stdout:8/748: chown d7/db/d42/d65/d46/d84/c8e 289635452 1 2024-04-22T23:23:54.584 INFO:tasks.workunit.client.1.smithi134.stdout:3/550: stat dc/dd/c25 0 2024-04-22T23:23:54.586 INFO:tasks.workunit.client.1.smithi134.stdout:0/746: mkdir d1/d2/df7 0 2024-04-22T23:23:54.587 INFO:tasks.workunit.client.1.smithi134.stdout:4/984: symlink d4/dd/da8/dc2/df8/d25/dfd/l139 0 2024-04-22T23:23:54.587 INFO:tasks.workunit.client.1.smithi134.stdout:8/749: truncate d7/db/d42/d65/d46/d9a/fd4 116810 0 2024-04-22T23:23:54.589 INFO:tasks.workunit.client.1.smithi134.stdout:0/747: write d1/f44 [3815222,105568] 0 2024-04-22T23:23:54.594 INFO:tasks.workunit.client.1.smithi134.stdout:6/794: unlink d0/d6f/d88/d8b/f91 0 2024-04-22T23:23:54.597 INFO:tasks.workunit.client.1.smithi134.stdout:1/922: creat db/db0/dc1/dd9/f132 x:0 0 0 2024-04-22T23:23:54.599 INFO:tasks.workunit.client.1.smithi134.stdout:3/551: stat dc/d2a/d62/db2/d2f/d21/d42/d77/d8f/lbd 0 2024-04-22T23:23:54.605 INFO:tasks.workunit.client.1.smithi134.stdout:7/812: rmdir d4/d8/d41/d76/dc1/d2f/d124 0 2024-04-22T23:23:54.607 INFO:tasks.workunit.client.1.smithi134.stdout:7/813: truncate d4/d8/d41/d76/dc1/f11 4817674 0 2024-04-22T23:23:54.610 INFO:tasks.workunit.client.1.smithi134.stdout:0/748: creat d1/d5/dd/d20/d2c/ff8 x:0 0 0 2024-04-22T23:23:54.612 INFO:tasks.workunit.client.1.smithi134.stdout:1/923: mkdir db/d107/d133 0 2024-04-22T23:23:54.617 INFO:tasks.workunit.client.1.smithi134.stdout:3/552: fdatasync dc/d2a/d62/db2/d2f/d54/f5e 0 2024-04-22T23:23:54.618 INFO:tasks.workunit.client.1.smithi134.stdout:8/750: truncate d7/db/d2d/f5a 292846 0 2024-04-22T23:23:54.620 INFO:tasks.workunit.client.1.smithi134.stdout:0/749: dwrite d1/d5/d58/d94/faa [0,4194304] 0 2024-04-22T23:23:54.622 INFO:tasks.workunit.client.1.smithi134.stdout:4/985: creat d4/d35/f13a x:0 0 0 2024-04-22T23:23:54.625 INFO:tasks.workunit.client.1.smithi134.stdout:4/986: write d4/d35/d8e/d5d/fa9 [575808,88401] 0 2024-04-22T23:23:54.626 INFO:tasks.workunit.client.1.smithi134.stdout:4/987: stat d4/d35/d53/lfb 0 2024-04-22T23:23:54.628 INFO:tasks.workunit.client.1.smithi134.stdout:8/751: unlink d7/db/d2d/fb0 0 2024-04-22T23:23:54.629 INFO:tasks.workunit.client.1.smithi134.stdout:4/988: read d4/f1d [2800951,111601] 0 2024-04-22T23:23:54.629 INFO:tasks.workunit.client.1.smithi134.stdout:3/553: dread dc/f40 [0,4194304] 0 2024-04-22T23:23:54.633 INFO:tasks.workunit.client.1.smithi134.stdout:0/750: mknod d1/d5/dd/d20/d2c/cf9 0 2024-04-22T23:23:54.633 INFO:tasks.workunit.client.1.smithi134.stdout:7/814: link d4/d8/f22 d4/d4a/f125 0 2024-04-22T23:23:54.635 INFO:tasks.workunit.client.1.smithi134.stdout:0/751: chown d1/d5/d58/d6b/db9/d83/fcc 343 1 2024-04-22T23:23:54.635 INFO:tasks.workunit.client.1.smithi134.stdout:8/752: mknod d7/ced 0 2024-04-22T23:23:54.636 INFO:tasks.workunit.client.1.smithi134.stdout:7/815: truncate d4/d8/d2a/d42/d75/fbf 4430966 0 2024-04-22T23:23:54.637 INFO:tasks.workunit.client.1.smithi134.stdout:4/989: mkdir d4/d9b/d13b 0 2024-04-22T23:23:54.639 INFO:tasks.workunit.client.1.smithi134.stdout:3/554: creat dc/d2a/d62/db2/d3b/fcc x:0 0 0 2024-04-22T23:23:54.639 INFO:tasks.workunit.client.1.smithi134.stdout:8/753: readlink d7/db/l7d 0 2024-04-22T23:23:54.640 INFO:tasks.workunit.client.1.smithi134.stdout:0/752: mknod d1/d2/cfa 0 2024-04-22T23:23:54.641 INFO:tasks.workunit.client.1.smithi134.stdout:8/754: chown d7/l1e 8023 1 2024-04-22T23:23:54.646 INFO:tasks.workunit.client.1.smithi134.stdout:4/990: rename d4/d7/d32/d3a/d93/d117 to d4/d7/d32/dc5/d13c 0 2024-04-22T23:23:54.646 INFO:tasks.workunit.client.1.smithi134.stdout:0/753: read d1/d5/d58/d22/fa7 [3473320,63676] 0 2024-04-22T23:23:54.646 INFO:tasks.workunit.client.1.smithi134.stdout:3/555: symlink dc/d2a/d62/db2/d2f/d21/d42/lcd 0 2024-04-22T23:23:54.648 INFO:tasks.workunit.client.1.smithi134.stdout:0/754: write d1/d5/d58/d6b/db9/f18 [355844,12414] 0 2024-04-22T23:23:54.648 INFO:tasks.workunit.client.1.smithi134.stdout:7/816: truncate d4/d8/d41/d76/dc1/d2f/d34/d62/f56 780371 0 2024-04-22T23:23:54.650 INFO:tasks.workunit.client.1.smithi134.stdout:4/991: symlink d4/d29/d116/l13d 0 2024-04-22T23:23:54.650 INFO:tasks.workunit.client.1.smithi134.stdout:0/755: write d1/d5/d58/d22/f24 [4051478,26594] 0 2024-04-22T23:23:54.652 INFO:tasks.workunit.client.1.smithi134.stdout:0/756: write d1/d5/f50 [321097,71001] 0 2024-04-22T23:23:54.653 INFO:tasks.workunit.client.1.smithi134.stdout:1/924: dread db/d16/dbc/fec [0,4194304] 0 2024-04-22T23:23:54.656 INFO:tasks.workunit.client.1.smithi134.stdout:1/925: chown db/d15/d17/d20/d2a/d2d/c67 5601134 1 2024-04-22T23:23:54.657 INFO:tasks.workunit.client.1.smithi134.stdout:7/817: mkdir d4/d8/d41/d76/dc1/d2f/d69/d8a/d126 0 2024-04-22T23:23:54.657 INFO:tasks.workunit.client.1.smithi134.stdout:4/992: rmdir d4/d35/d8e/d6e 39 2024-04-22T23:23:54.665 INFO:tasks.workunit.client.1.smithi134.stdout:3/556: getdents dc/d2a/d62/db2/d2f/d61/d6a 0 2024-04-22T23:23:54.666 INFO:tasks.workunit.client.1.smithi134.stdout:1/926: dread db/d16/dbc/de1/ff5 [0,4194304] 0 2024-04-22T23:23:54.668 INFO:tasks.workunit.client.1.smithi134.stdout:0/757: link d1/d5/d58/d6b/db9/fcd d1/d5/d58/d6b/db9/def/ffb 0 2024-04-22T23:23:54.669 INFO:tasks.workunit.client.1.smithi134.stdout:7/818: dwrite d4/d8/d41/d76/dc1/d2f/d69/dab/de1/ff7 [0,4194304] 0 2024-04-22T23:23:54.670 INFO:tasks.workunit.client.1.smithi134.stdout:4/993: dwrite d4/d35/d8e/f75 [0,4194304] 0 2024-04-22T23:23:54.671 INFO:tasks.workunit.client.1.smithi134.stdout:0/758: chown d1/d5/d58/d6b/db9/f64 349241462 1 2024-04-22T23:23:54.673 INFO:tasks.workunit.client.1.smithi134.stdout:7/819: chown d4/d8/d2a/d42/d6e/ddd/cce 237863045 1 2024-04-22T23:23:54.676 INFO:tasks.workunit.client.1.smithi134.stdout:1/927: creat db/d15/d17/d20/d2a/d2d/d2e/f134 x:0 0 0 2024-04-22T23:23:54.677 INFO:tasks.workunit.client.1.smithi134.stdout:4/994: creat d4/d35/d8e/d5d/dce/f13e x:0 0 0 2024-04-22T23:23:54.679 INFO:tasks.workunit.client.1.smithi134.stdout:1/928: chown db/d15/d17/d20/dfa/lcd 3397 1 2024-04-22T23:23:54.680 INFO:tasks.workunit.client.1.smithi134.stdout:0/759: dwrite d1/d5/f5e [0,4194304] 0 2024-04-22T23:23:54.681 INFO:tasks.workunit.client.1.smithi134.stdout:4/995: dread d4/dd/da8/dc2/df8/f97 [0,4194304] 0 2024-04-22T23:23:54.682 INFO:tasks.workunit.client.1.smithi134.stdout:9/711: write d0/d2d/d82/da0/f8d [1339956,11736] 0 2024-04-22T23:23:54.693 INFO:tasks.workunit.client.1.smithi134.stdout:4/996: dwrite d4/d7/d32/d3a/f43 [4194304,4194304] 0 2024-04-22T23:23:54.694 INFO:tasks.workunit.client.1.smithi134.stdout:9/712: creat d0/d7/d23/d26/d51/d88/db9/ff1 x:0 0 0 2024-04-22T23:23:54.694 INFO:tasks.workunit.client.1.smithi134.stdout:1/929: dwrite db/d15/fc6 [0,4194304] 0 2024-04-22T23:23:54.700 INFO:tasks.workunit.client.1.smithi134.stdout:6/795: write d0/d19/de/fb0 [640052,34480] 0 2024-04-22T23:23:54.701 INFO:tasks.workunit.client.1.smithi134.stdout:9/713: stat d0/d37/d76/cdd 0 2024-04-22T23:23:54.707 INFO:tasks.workunit.client.1.smithi134.stdout:6/796: symlink d0/d19/db/d33/l103 0 2024-04-22T23:23:54.733 INFO:tasks.workunit.client.1.smithi134.stdout:6/797: dread d0/d19/db/d35/d7f/fee [0,4194304] 0 2024-04-22T23:23:54.740 INFO:tasks.workunit.client.1.smithi134.stdout:6/798: dwrite d0/d19/de/d12/d29/d74/da0/fc8 [0,4194304] 0 2024-04-22T23:23:54.741 INFO:tasks.workunit.client.1.smithi134.stdout:1/930: write db/d15/d17/d20/d2a/d2d/d86/f110 [337668,21740] 0 2024-04-22T23:23:54.743 INFO:tasks.workunit.client.1.smithi134.stdout:1/931: chown db/ff6 571878969 1 2024-04-22T23:23:54.747 INFO:tasks.workunit.client.1.smithi134.stdout:1/932: symlink db/db0/dc1/dd9/l135 0 2024-04-22T23:23:54.749 INFO:tasks.workunit.client.1.smithi134.stdout:6/799: dread d0/f1a [0,4194304] 0 2024-04-22T23:23:54.752 INFO:tasks.workunit.client.1.smithi134.stdout:1/933: rmdir db/d15/d17/dc9 39 2024-04-22T23:23:54.755 INFO:tasks.workunit.client.1.smithi134.stdout:6/800: creat d0/d19/de/f104 x:0 0 0 2024-04-22T23:23:54.756 INFO:tasks.workunit.client.1.smithi134.stdout:6/801: chown d0/d19/f9 173641992 1 2024-04-22T23:23:54.757 INFO:tasks.workunit.client.1.smithi134.stdout:1/934: link db/ff db/d16/d87/f136 0 2024-04-22T23:23:54.760 INFO:tasks.workunit.client.1.smithi134.stdout:6/802: rename d0/d19/db/c76 to d0/d6f/d88/c105 0 2024-04-22T23:23:54.761 INFO:tasks.workunit.client.1.smithi134.stdout:1/935: read - db/d16/d56/f83 zero size 2024-04-22T23:23:54.762 INFO:tasks.workunit.client.1.smithi134.stdout:1/936: write db/db0/dc1/f118 [520234,122289] 0 2024-04-22T23:23:54.766 INFO:tasks.workunit.client.1.smithi134.stdout:1/937: creat db/d15/d17/d20/d2a/d2d/d2e/f137 x:0 0 0 2024-04-22T23:23:54.771 INFO:tasks.workunit.client.1.smithi134.stdout:1/938: rename db/d16/dbc/de1/ff5 to db/d16/d87/f138 0 2024-04-22T23:23:54.774 INFO:tasks.workunit.client.1.smithi134.stdout:6/803: dread d0/d19/db/d33/f64 [0,4194304] 0 2024-04-22T23:23:54.777 INFO:tasks.workunit.client.1.smithi134.stdout:1/939: mkdir db/d15/d17/d20/d2a/d2d/d2e/d139 0 2024-04-22T23:23:54.778 INFO:tasks.workunit.client.1.smithi134.stdout:6/804: rename d0/d19/lc4 to d0/d19/de/d12/d5e/d6d/l106 0 2024-04-22T23:23:54.779 INFO:tasks.workunit.client.1.smithi134.stdout:6/805: chown d0/d19/de/d12 219 1 2024-04-22T23:23:54.779 INFO:tasks.workunit.client.1.smithi134.stdout:1/940: readlink db/d15/d17/d20/d2a/d2d/d108/d114/l128 0 2024-04-22T23:23:54.781 INFO:tasks.workunit.client.1.smithi134.stdout:1/941: creat db/db0/dc1/f13a x:0 0 0 2024-04-22T23:23:54.782 INFO:tasks.workunit.client.1.smithi134.stdout:6/806: mkdir d0/d19/db/dfa/d107 0 2024-04-22T23:23:54.783 INFO:tasks.workunit.client.1.smithi134.stdout:1/942: write db/db5/fdb [1613889,111920] 0 2024-04-22T23:23:54.786 INFO:tasks.workunit.client.1.smithi134.stdout:6/807: mkdir d0/d19/de/d108 0 2024-04-22T23:23:54.789 INFO:tasks.workunit.client.1.smithi134.stdout:6/808: rmdir d0/d19/de/d1f/dd5 39 2024-04-22T23:23:54.795 INFO:tasks.workunit.client.1.smithi134.stdout:1/943: symlink db/d15/d17/d20/d2a/d2d/l13b 0 2024-04-22T23:23:54.797 INFO:tasks.workunit.client.1.smithi134.stdout:1/944: dread - db/d15/d17/d20/d2a/d2d/d86/dad/fe7 zero size 2024-04-22T23:23:54.802 INFO:tasks.workunit.client.1.smithi134.stdout:1/945: truncate db/d15/d17/d20/d2a/d2d/d108/d114/f66 24920 0 2024-04-22T23:23:54.804 INFO:tasks.workunit.client.1.smithi134.stdout:6/809: getdents d0/d19/de/d12 0 2024-04-22T23:23:54.806 INFO:tasks.workunit.client.1.smithi134.stdout:1/946: creat db/d16/d104/f13c x:0 0 0 2024-04-22T23:23:54.806 INFO:tasks.workunit.client.1.smithi134.stdout:1/947: read - db/db0/dc1/f13a zero size 2024-04-22T23:23:54.807 INFO:tasks.workunit.client.1.smithi134.stdout:6/810: truncate d0/d19/de/d12/d5e/d6d/f9f 3387259 0 2024-04-22T23:23:54.808 INFO:tasks.workunit.client.1.smithi134.stdout:6/811: chown d0/d19/de/d12/d29/d74/deb/ffc 344195 1 2024-04-22T23:23:54.809 INFO:tasks.workunit.client.1.smithi134.stdout:6/812: readlink d0/l67 0 2024-04-22T23:23:54.809 INFO:tasks.workunit.client.1.smithi134.stdout:6/813: chown d0/f9d 40 1 2024-04-22T23:23:54.828 INFO:tasks.workunit.client.1.smithi134.stdout:6/814: write d0/d19/de/d12/d5e/d6d/f73 [620441,5896] 0 2024-04-22T23:23:54.831 INFO:tasks.workunit.client.1.smithi134.stdout:1/948: dread db/d15/d17/d20/d2a/d2d/fdd [0,4194304] 0 2024-04-22T23:23:54.833 INFO:tasks.workunit.client.1.smithi134.stdout:6/815: creat d0/d19/de/d1f/dd5/f109 x:0 0 0 2024-04-22T23:23:54.835 INFO:tasks.workunit.client.1.smithi134.stdout:6/816: write d0/d6f/f84 [1706422,17964] 0 2024-04-22T23:23:54.836 INFO:tasks.workunit.client.1.smithi134.stdout:1/949: mkdir db/d15/d17/d20/d2a/d2d/d108/d13d 0 2024-04-22T23:23:54.854 INFO:tasks.workunit.client.1.smithi134.stdout:1/950: dread db/f5f [0,4194304] 0 2024-04-22T23:23:54.854 INFO:tasks.workunit.client.1.smithi134.stdout:6/817: dread d0/d19/de/d12/f7e [0,4194304] 0 2024-04-22T23:23:54.857 INFO:tasks.workunit.client.1.smithi134.stdout:1/951: mknod db/d107/d133/c13e 0 2024-04-22T23:23:54.859 INFO:tasks.workunit.client.1.smithi134.stdout:6/818: rename d0/l4 to d0/d19/de/d12/d29/d74/l10a 0 2024-04-22T23:23:54.859 INFO:tasks.workunit.client.1.smithi134.stdout:1/952: truncate db/d15/d17/fc8 4685775 0 2024-04-22T23:23:54.861 INFO:tasks.workunit.client.1.smithi134.stdout:6/819: rmdir d0/d19/de/d12/d5e/d6d 39 2024-04-22T23:23:54.862 INFO:tasks.workunit.client.1.smithi134.stdout:6/820: stat d0/d19/de/f104 0 2024-04-22T23:23:54.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.863+0000 7fa8f8caa700 1 -- 172.21.15.67:0/2529558521 --> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa8f41067f0 con 0x7fa8dc06d780 2024-04-22T23:23:54.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.865+0000 7fa8e37fe700 1 -- 172.21.15.67:0/2529558521 <== mgr.14215 v2:172.21.15.67:6800/2589747850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7fa8f41067f0 con 0x7fa8dc06d780 2024-04-22T23:23:54.866 INFO:tasks.workunit.client.1.smithi134.stdout:1/953: dwrite db/d15/d17/d20/d2a/d2d/d86/f110 [0,4194304] 0 2024-04-22T23:23:54.868 INFO:tasks.workunit.client.1.smithi134.stdout:6/821: dread d0/d19/db/d33/f64 [0,4194304] 0 2024-04-22T23:23:54.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.867+0000 7fa8f8caa700 1 -- 172.21.15.67:0/2529558521 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa8dc06d780 msgr2=0x7fa8dc06fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:54.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.867+0000 7fa8f8caa700 1 --2- 172.21.15.67:0/2529558521 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa8dc06d780 0x7fa8dc06fc40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa8e80059d0 tx=0x7fa8e8005940 comp rx=0 tx=0).stop 2024-04-22T23:23:54.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.867+0000 7fa8f8caa700 1 -- 172.21.15.67:0/2529558521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f4100ca0 msgr2=0x7fa8f4074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:54.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.867+0000 7fa8f8caa700 1 --2- 172.21.15.67:0/2529558521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f4100ca0 0x7fa8f4074b30 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa8e400e8f0 tx=0x7fa8e400ecb0 comp rx=0 tx=0).stop 2024-04-22T23:23:54.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.867+0000 7fa8f8caa700 1 -- 172.21.15.67:0/2529558521 shutdown_connections 2024-04-22T23:23:54.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.867+0000 7fa8f8caa700 1 --2- 172.21.15.67:0/2529558521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f4100ca0 0x7fa8f4074b30 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:54.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.867+0000 7fa8f8caa700 1 --2- 172.21.15.67:0/2529558521 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7fa8dc06d780 0x7fa8dc06fc40 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:54.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.868+0000 7fa8f8caa700 1 --2- 172.21.15.67:0/2529558521 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa8f4101ea0 0x7fa8f4073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:54.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.868+0000 7fa8f8caa700 1 -- 172.21.15.67:0/2529558521 >> 172.21.15.67:0/2529558521 conn(0x7fa8f40fc260 msgr2=0x7fa8f41050d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:23:54.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.868+0000 7fa8f8caa700 1 -- 172.21.15.67:0/2529558521 shutdown_connections 2024-04-22T23:23:54.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.868+0000 7fa8f8caa700 1 -- 172.21.15.67:0/2529558521 wait complete. 2024-04-22T23:23:54.870 INFO:tasks.workunit.client.1.smithi134.stdout:6/822: readlink d0/d19/de/d12/d5e/dc1/ldb 0 2024-04-22T23:23:54.884 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:23:54.885 INFO:tasks.workunit.client.1.smithi134.stdout:6/823: dwrite d0/d19/de/dbe/fc6 [0,4194304] 0 2024-04-22T23:23:54.944 INFO:tasks.workunit.client.1.smithi134.stdout:5/853: sync 2024-04-22T23:23:54.944 INFO:tasks.workunit.client.1.smithi134.stdout:2/654: sync 2024-04-22T23:23:54.945 INFO:tasks.workunit.client.1.smithi134.stdout:5/854: chown d9/dbb/d11b 1266946 1 2024-04-22T23:23:54.946 INFO:tasks.workunit.client.1.smithi134.stdout:5/855: chown d9/d1e/f35 0 1 2024-04-22T23:23:54.947 INFO:tasks.workunit.client.1.smithi134.stdout:5/856: chown d9/d11/d34/d4d/f9f 526582110 1 2024-04-22T23:23:54.952 INFO:tasks.workunit.client.1.smithi134.stdout:2/655: dwrite d2/dd/d4a/d73/fdb [0,4194304] 0 2024-04-22T23:23:55.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.999+0000 7f2f5603f700 1 -- 172.21.15.67:0/771359304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f50104170 msgr2=0x7f2f50106560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:55.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.999+0000 7f2f5603f700 1 --2- 172.21.15.67:0/771359304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f50104170 0x7f2f50106560 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2f44009910 tx=0x7f2f44009c20 comp rx=0 tx=0).stop 2024-04-22T23:23:55.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.999+0000 7f2f5603f700 1 -- 172.21.15.67:0/771359304 shutdown_connections 2024-04-22T23:23:55.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.999+0000 7f2f5603f700 1 --2- 172.21.15.67:0/771359304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f50104170 0x7f2f50106560 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:55.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.999+0000 7f2f5603f700 1 --2- 172.21.15.67:0/771359304 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f50101840 0x7f2f50103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:55.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.999+0000 7f2f5603f700 1 -- 172.21.15.67:0/771359304 >> 172.21.15.67:0/771359304 conn(0x7f2f500fb190 msgr2=0x7f2f500fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:23:55.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.999+0000 7f2f5603f700 1 -- 172.21.15.67:0/771359304 shutdown_connections 2024-04-22T23:23:55.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:54.999+0000 7f2f5603f700 1 -- 172.21.15.67:0/771359304 wait complete. 2024-04-22T23:23:55.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.000+0000 7f2f5603f700 1 Processor -- start 2024-04-22T23:23:55.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.000+0000 7f2f5603f700 1 -- start start 2024-04-22T23:23:55.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.000+0000 7f2f5603f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f50101840 0x7f2f50198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:23:55.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.000+0000 7f2f5603f700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f50104170 0x7f2f50198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:23:55.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.000+0000 7f2f5603f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f501991e0 con 0x7f2f50101840 2024-04-22T23:23:55.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.000+0000 7f2f5603f700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f50199320 con 0x7f2f50104170 2024-04-22T23:23:55.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.001+0000 7f2f4effd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f50104170 0x7f2f50198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:23:55.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.001+0000 7f2f4effd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f50104170 0x7f2f50198bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.67:46312/0 (socket says 172.21.15.67:46312) 2024-04-22T23:23:55.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.001+0000 7f2f4effd700 1 -- 172.21.15.67:0/2679308385 learned_addr learned my addr 172.21.15.67:0/2679308385 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:23:55.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.001+0000 7f2f4f7fe700 1 --2- 172.21.15.67:0/2679308385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f50101840 0x7f2f50198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:23:55.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.001+0000 7f2f4effd700 1 -- 172.21.15.67:0/2679308385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f50101840 msgr2=0x7f2f50198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:55.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.001+0000 7f2f4effd700 1 --2- 172.21.15.67:0/2679308385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f50101840 0x7f2f50198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:55.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.001+0000 7f2f4effd700 1 -- 172.21.15.67:0/2679308385 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f440095f0 con 0x7f2f50104170 2024-04-22T23:23:55.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.002+0000 7f2f4effd700 1 --2- 172.21.15.67:0/2679308385 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f50104170 0x7f2f50198bc0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2f440092e0 tx=0x7f2f4400f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:23:55.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.002+0000 7f2f4cff9700 1 -- 172.21.15.67:0/2679308385 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f4401d070 con 0x7f2f50104170 2024-04-22T23:23:55.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.002+0000 7f2f4cff9700 1 -- 172.21.15.67:0/2679308385 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2f44005dc0 con 0x7f2f50104170 2024-04-22T23:23:55.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.002+0000 7f2f5603f700 1 -- 172.21.15.67:0/2679308385 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f5019dd70 con 0x7f2f50104170 2024-04-22T23:23:55.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.002+0000 7f2f5603f700 1 -- 172.21.15.67:0/2679308385 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f5019e1a0 con 0x7f2f50104170 2024-04-22T23:23:55.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.002+0000 7f2f4cff9700 1 -- 172.21.15.67:0/2679308385 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f44017630 con 0x7f2f50104170 2024-04-22T23:23:55.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.003+0000 7f2f5603f700 1 -- 172.21.15.67:0/2679308385 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f3c005320 con 0x7f2f50104170 2024-04-22T23:23:55.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.004+0000 7f2f4cff9700 1 -- 172.21.15.67:0/2679308385 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 20) v1 ==== 91455+0+0 (secure 0 0 0) 0x7f2f4400faf0 con 0x7f2f50104170 2024-04-22T23:23:55.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.005+0000 7f2f4cff9700 1 --2- 172.21.15.67:0/2679308385 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f2f3806d760 0x7f2f3806fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:23:55.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.005+0000 7f2f4cff9700 1 -- 172.21.15.67:0/2679308385 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f2f4408e360 con 0x7f2f50104170 2024-04-22T23:23:55.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.005+0000 7f2f4f7fe700 1 --2- 172.21.15.67:0/2679308385 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f2f3806d760 0x7f2f3806fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:23:55.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.006+0000 7f2f4f7fe700 1 --2- 172.21.15.67:0/2679308385 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f2f3806d760 0x7f2f3806fc20 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2f40009c60 tx=0x7f2f40009260 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:23:55.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.012+0000 7f2f4cff9700 1 -- 172.21.15.67:0/2679308385 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f2f4405c4d0 con 0x7f2f50104170 2024-04-22T23:23:55.057 INFO:tasks.workunit.client.1.smithi134.stdout:1/954: link db/db0/dc1/dd9/l135 db/db0/dc1/l13f 0 2024-04-22T23:23:55.061 INFO:tasks.workunit.client.1.smithi134.stdout:6/824: creat d0/d19/de/dbe/f10b x:0 0 0 2024-04-22T23:23:55.065 INFO:tasks.workunit.client.1.smithi134.stdout:2/656: symlink d2/dd/df/d1c/le2 0 2024-04-22T23:23:55.087 INFO:tasks.workunit.client.1.smithi134.stdout:6/825: dread d0/d19/de/d12/d2a/d2d/f80 [0,4194304] 0 2024-04-22T23:23:55.087 INFO:tasks.workunit.client.1.smithi134.stdout:6/826: chown d0/c21 9 1 2024-04-22T23:23:55.089 INFO:tasks.workunit.client.1.smithi134.stdout:6/827: truncate d0/d19/db/d75/d92/f100 915899 0 2024-04-22T23:23:55.089 INFO:tasks.workunit.client.1.smithi134.stdout:6/828: chown d0/d19/de/d12 997338057 1 2024-04-22T23:23:55.096 INFO:tasks.workunit.client.1.smithi134.stdout:6/829: creat d0/d19/de/d12/d5e/d6d/d7a/f10c x:0 0 0 2024-04-22T23:23:55.099 INFO:tasks.workunit.client.1.smithi134.stdout:6/830: creat d0/d19/db/d33/f10d x:0 0 0 2024-04-22T23:23:55.101 INFO:tasks.workunit.client.1.smithi134.stdout:6/831: symlink d0/d19/de/d12/d5e/d6d/d7a/l10e 0 2024-04-22T23:23:55.104 INFO:tasks.workunit.client.1.smithi134.stdout:6/832: symlink d0/d19/db/d75/d92/l10f 0 2024-04-22T23:23:55.113 INFO:tasks.workunit.client.1.smithi134.stdout:6/833: getdents d0/d19/db/d35/d7f 0 2024-04-22T23:23:55.115 INFO:tasks.workunit.client.1.smithi134.stdout:6/834: read d0/d19/fa [899133,129322] 0 2024-04-22T23:23:55.156 INFO:tasks.workunit.client.1.smithi134.stdout:5/857: write d9/dbb/d11b/d17/d2a/f98 [1438542,109337] 0 2024-04-22T23:23:55.156 INFO:tasks.workunit.client.1.smithi134.stdout:2/657: write d2/d15/d24/d1d/f8d [1455227,16998] 0 2024-04-22T23:23:55.157 INFO:tasks.workunit.client.1.smithi134.stdout:5/858: fdatasync d9/dbb/d11b/d17/d44/fd0 0 2024-04-22T23:23:55.161 INFO:tasks.workunit.client.1.smithi134.stdout:1/955: rename db/d15/d17/d20/d2a/d2d/le0 to db/d16/d87/dbd/d10f/l140 0 2024-04-22T23:23:55.163 INFO:tasks.workunit.client.1.smithi134.stdout:2/658: dwrite d2/d15/d24/d1d/f8d [0,4194304] 0 2024-04-22T23:23:55.164 INFO:tasks.workunit.client.1.smithi134.stdout:5/859: rmdir d9/d11/d73/d95/dae 39 2024-04-22T23:23:55.169 INFO:tasks.workunit.client.1.smithi134.stdout:1/956: write db/d15/d17/dc9/ffc [395693,37590] 0 2024-04-22T23:23:55.170 INFO:tasks.workunit.client.1.smithi134.stdout:1/957: stat db/d107 0 2024-04-22T23:23:55.171 INFO:tasks.workunit.client.1.smithi134.stdout:2/659: dwrite d2/fc1 [0,4194304] 0 2024-04-22T23:23:55.181 INFO:tasks.workunit.client.1.smithi134.stdout:2/660: dwrite d2/dd/d4a/d73/fdb [4194304,4194304] 0 2024-04-22T23:23:55.184 INFO:tasks.workunit.client.1.smithi134.stdout:2/661: stat d2/d42/d54 0 2024-04-22T23:23:55.188 INFO:tasks.workunit.client.1.smithi134.stdout:2/662: truncate d2/d42/f46 11300622 0 2024-04-22T23:23:55.191 INFO:tasks.workunit.client.1.smithi134.stdout:2/663: rmdir d2/dd/df/d9c/daa 39 2024-04-22T23:23:55.195 INFO:tasks.workunit.client.1.smithi134.stdout:6/835: dwrite d0/f55 [0,4194304] 0 2024-04-22T23:23:55.196 INFO:tasks.workunit.client.1.smithi134.stdout:2/664: mkdir d2/dd/df/d1c/de3 0 2024-04-22T23:23:55.200 INFO:tasks.workunit.client.1.smithi134.stdout:2/665: mkdir d2/dd/de4 0 2024-04-22T23:23:55.208 INFO:tasks.workunit.client.1.smithi134.stdout:5/860: write d9/f4a [4016814,84762] 0 2024-04-22T23:23:55.209 INFO:tasks.workunit.client.1.smithi134.stdout:5/861: readlink d9/da2/de6/l65 0 2024-04-22T23:23:55.210 INFO:tasks.workunit.client.1.smithi134.stdout:8/755: sync 2024-04-22T23:23:55.215 INFO:tasks.workunit.client.1.smithi134.stdout:8/756: read d7/db/f1f [257806,3976] 0 2024-04-22T23:23:55.215 INFO:tasks.workunit.client.1.smithi134.stdout:1/958: dwrite db/d15/d17/d20/fab [0,4194304] 0 2024-04-22T23:23:55.219 INFO:tasks.workunit.client.1.smithi134.stdout:1/959: fdatasync db/d15/d17/f69 0 2024-04-22T23:23:55.220 INFO:tasks.workunit.client.1.smithi134.stdout:1/960: dread - db/d16/d104/f12a zero size 2024-04-22T23:23:55.221 INFO:tasks.workunit.client.1.smithi134.stdout:5/862: mkdir d9/d11/d34/d4d/dd4/d121/d125 0 2024-04-22T23:23:55.222 INFO:tasks.workunit.client.1.smithi134.stdout:1/961: write db/d16/d87/faf [827679,30619] 0 2024-04-22T23:23:55.226 INFO:tasks.workunit.client.1.smithi134.stdout:8/757: mkdir d7/db/d42/d65/dd9/dda/de7/ddd/dee 0 2024-04-22T23:23:55.228 INFO:tasks.workunit.client.1.smithi134.stdout:2/666: dwrite d2/d15/d24/d1d/d3d/d7b/f7c [0,4194304] 0 2024-04-22T23:23:55.235 INFO:tasks.workunit.client.1.smithi134.stdout:5/863: creat d9/d11/d34/d4d/dd4/d121/f126 x:0 0 0 2024-04-22T23:23:55.236 INFO:tasks.workunit.client.1.smithi134.stdout:5/864: dread - d9/dbb/d11b/d88/db7/ffa zero size 2024-04-22T23:23:55.239 INFO:tasks.workunit.client.1.smithi134.stdout:5/865: truncate d9/da2/de6/dc0/d109/f113 404000 0 2024-04-22T23:23:55.248 INFO:tasks.workunit.client.1.smithi134.stdout:5/866: creat d9/d11/d73/d95/dae/f127 x:0 0 0 2024-04-22T23:23:55.251 INFO:tasks.workunit.client.1.smithi134.stdout:8/758: dread d7/dc/d2c/d37/f3f [0,4194304] 0 2024-04-22T23:23:55.253 INFO:tasks.workunit.client.1.smithi134.stdout:8/759: chown d7/db/d2d/d39/d58/l6d 1419264 1 2024-04-22T23:23:55.253 INFO:tasks.workunit.client.1.smithi134.stdout:5/867: mkdir d9/d1e/d128 0 2024-04-22T23:23:55.254 INFO:tasks.workunit.client.1.smithi134.stdout:8/760: write d7/db/d42/d65/fc9 [3653904,90174] 0 2024-04-22T23:23:55.256 INFO:tasks.workunit.client.1.smithi134.stdout:8/761: write d7/dc/f12 [2271614,124616] 0 2024-04-22T23:23:55.258 INFO:tasks.workunit.client.1.smithi134.stdout:5/868: write d9/da2/fdb [590927,79760] 0 2024-04-22T23:23:55.261 INFO:tasks.workunit.client.1.smithi134.stdout:8/762: write d7/db/f3b [963625,30735] 0 2024-04-22T23:23:55.263 INFO:tasks.workunit.client.1.smithi134.stdout:5/869: mkdir d9/dbb/d11b/d88/db7/d129 0 2024-04-22T23:23:55.264 INFO:tasks.workunit.client.1.smithi134.stdout:5/870: dread - d9/dbb/d11b/d52/f79 zero size 2024-04-22T23:23:55.264 INFO:tasks.workunit.client.1.smithi134.stdout:2/667: write d2/dd/d4a/f64 [1138896,87397] 0 2024-04-22T23:23:55.265 INFO:tasks.workunit.client.1.smithi134.stdout:1/962: write db/d16/d56/f60 [1938231,70931] 0 2024-04-22T23:23:55.267 INFO:tasks.workunit.client.1.smithi134.stdout:8/763: fdatasync d7/db/d42/d65/dd9/dda/de7/ddd/fce 0 2024-04-22T23:23:55.267 INFO:tasks.workunit.client.1.smithi134.stdout:1/963: chown db/d15/cbe 325 1 2024-04-22T23:23:55.270 INFO:tasks.workunit.client.1.smithi134.stdout:5/871: dread d9/d11/d73/d95/f119 [0,4194304] 0 2024-04-22T23:23:55.270 INFO:tasks.workunit.client.1.smithi134.stdout:2/668: read d2/dd/f57 [3747993,83883] 0 2024-04-22T23:23:55.279 INFO:tasks.workunit.client.1.smithi134.stdout:2/669: mknod d2/d15/d24/d1d/d3d/d9a/ce5 0 2024-04-22T23:23:55.280 INFO:tasks.workunit.client.1.smithi134.stdout:5/872: link f8 d9/d1e/f12a 0 2024-04-22T23:23:55.281 INFO:tasks.workunit.client.1.smithi134.stdout:5/873: stat d9/dbb/d11b/da6/fc6 0 2024-04-22T23:23:55.281 INFO:tasks.workunit.client.1.smithi134.stdout:5/874: chown d9/d11/d73/da9/l10d 256414187 1 2024-04-22T23:23:55.284 INFO:tasks.workunit.client.1.smithi134.stdout:2/670: symlink d2/d42/le6 0 2024-04-22T23:23:55.284 INFO:tasks.workunit.client.1.smithi134.stdout:2/671: dread - d2/fde zero size 2024-04-22T23:23:55.285 INFO:tasks.workunit.client.1.smithi134.stdout:2/672: chown d2/d42 42162203 1 2024-04-22T23:23:55.287 INFO:tasks.workunit.client.1.smithi134.stdout:5/875: mknod d9/dbb/d11b/d88/db7/d129/c12b 0 2024-04-22T23:23:55.288 INFO:tasks.workunit.client.1.smithi134.stdout:1/964: dwrite db/d16/f3d [0,4194304] 0 2024-04-22T23:23:55.290 INFO:tasks.workunit.client.1.smithi134.stdout:1/965: chown db/d16/dbc/de1 43 1 2024-04-22T23:23:55.291 INFO:tasks.workunit.client.1.smithi134.stdout:1/966: readlink db/d15/d17/d20/l77 0 2024-04-22T23:23:55.294 INFO:tasks.workunit.client.1.smithi134.stdout:5/876: dread d9/d11/d73/d95/dae/ff2 [0,4194304] 0 2024-04-22T23:23:55.294 INFO:tasks.workunit.client.1.smithi134.stdout:1/967: dread db/d15/d17/d20/d2a/d2d/d108/d114/f66 [0,4194304] 0 2024-04-22T23:23:55.295 INFO:tasks.workunit.client.1.smithi134.stdout:5/877: read - d9/dbb/d11b/d17/d44/f7e zero size 2024-04-22T23:23:55.296 INFO:tasks.workunit.client.1.smithi134.stdout:5/878: readlink d9/d11/d34/lff 0 2024-04-22T23:23:55.302 INFO:tasks.workunit.client.1.smithi134.stdout:1/968: dwrite db/d15/d121/f131 [0,4194304] 0 2024-04-22T23:23:55.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.301+0000 7f2f5603f700 1 -- 172.21.15.67:0/2679308385 --> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2f3c000bf0 con 0x7f2f3806d760 2024-04-22T23:23:55.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.302+0000 7f2f4cff9700 1 -- 172.21.15.67:0/2679308385 <== mgr.14215 v2:172.21.15.67:6800/2589747850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7f2f3c000bf0 con 0x7f2f3806d760 2024-04-22T23:23:55.304 INFO:tasks.workunit.client.1.smithi134.stdout:1/969: dread - db/db0/dc1/dd9/ff0 zero size 2024-04-22T23:23:55.305 INFO:tasks.workunit.client.1.smithi134.stdout:1/970: dread - db/d15/d17/f11b zero size 2024-04-22T23:23:55.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.305+0000 7f2f5603f700 1 -- 172.21.15.67:0/2679308385 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f2f3806d760 msgr2=0x7f2f3806fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:55.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.305+0000 7f2f5603f700 1 --2- 172.21.15.67:0/2679308385 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f2f3806d760 0x7f2f3806fc20 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2f40009c60 tx=0x7f2f40009260 comp rx=0 tx=0).stop 2024-04-22T23:23:55.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.305+0000 7f2f5603f700 1 -- 172.21.15.67:0/2679308385 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f50104170 msgr2=0x7f2f50198bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:55.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.305+0000 7f2f5603f700 1 --2- 172.21.15.67:0/2679308385 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f50104170 0x7f2f50198bc0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2f440092e0 tx=0x7f2f4400f690 comp rx=0 tx=0).stop 2024-04-22T23:23:55.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.305+0000 7f2f5603f700 1 -- 172.21.15.67:0/2679308385 shutdown_connections 2024-04-22T23:23:55.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.305+0000 7f2f5603f700 1 --2- 172.21.15.67:0/2679308385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f50101840 0x7f2f50198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:55.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.305+0000 7f2f5603f700 1 --2- 172.21.15.67:0/2679308385 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f2f3806d760 0x7f2f3806fc20 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:55.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.305+0000 7f2f5603f700 1 --2- 172.21.15.67:0/2679308385 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f50104170 0x7f2f50198bc0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:55.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.305+0000 7f2f5603f700 1 -- 172.21.15.67:0/2679308385 >> 172.21.15.67:0/2679308385 conn(0x7f2f500fb190 msgr2=0x7f2f50104de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:23:55.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.305+0000 7f2f5603f700 1 -- 172.21.15.67:0/2679308385 shutdown_connections 2024-04-22T23:23:55.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.305+0000 7f2f5603f700 1 -- 172.21.15.67:0/2679308385 wait complete. 2024-04-22T23:23:55.308 INFO:tasks.workunit.client.1.smithi134.stdout:1/971: creat db/d15/d17/d20/d2a/d2d/d126/f141 x:0 0 0 2024-04-22T23:23:55.315 INFO:tasks.workunit.client.1.smithi134.stdout:1/972: mkdir db/d15/d142 0 2024-04-22T23:23:55.316 INFO:tasks.workunit.client.1.smithi134.stdout:1/973: chown db/d107/f12b 3529 1 2024-04-22T23:23:55.317 INFO:tasks.workunit.client.1.smithi134.stdout:1/974: fsync db/d15/d17/d20/d2a/d2d/d126/f130 0 2024-04-22T23:23:55.323 INFO:tasks.workunit.client.1.smithi134.stdout:1/975: getdents db/d16/d87 0 2024-04-22T23:23:55.324 INFO:tasks.workunit.client.1.smithi134.stdout:5/879: truncate d9/d11/d34/f6a 2991531 0 2024-04-22T23:23:55.325 INFO:tasks.workunit.client.1.smithi134.stdout:5/880: chown d9/d11/d34/d4d/dd4/f104 2008654 1 2024-04-22T23:23:55.328 INFO:tasks.workunit.client.1.smithi134.stdout:1/976: link db/d16/l8e db/d15/d17/d112/l143 0 2024-04-22T23:23:55.334 INFO:tasks.workunit.client.1.smithi134.stdout:5/881: write d9/d11/d34/fa3 [777632,91289] 0 2024-04-22T23:23:55.336 INFO:tasks.workunit.client.1.smithi134.stdout:5/882: rmdir d9/d11/d34/d4d/dd4/dd7 39 2024-04-22T23:23:55.342 INFO:tasks.workunit.client.1.smithi134.stdout:5/883: dwrite d9/d11/d73/d95/f105 [0,4194304] 0 2024-04-22T23:23:55.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.431+0000 7f1052ddf700 1 -- 172.21.15.67:0/3949191602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f104c1040d0 msgr2=0x7f104c1064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:55.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.431+0000 7f1052ddf700 1 --2- 172.21.15.67:0/3949191602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f104c1040d0 0x7f104c1064c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f1038009910 tx=0x7f1038009c20 comp rx=0 tx=0).stop 2024-04-22T23:23:55.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.431+0000 7f1052ddf700 1 -- 172.21.15.67:0/3949191602 shutdown_connections 2024-04-22T23:23:55.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.431+0000 7f1052ddf700 1 --2- 172.21.15.67:0/3949191602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f104c1040d0 0x7f104c1064c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:55.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.431+0000 7f1052ddf700 1 --2- 172.21.15.67:0/3949191602 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f104c1017a0 0x7f104c103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:55.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.431+0000 7f1052ddf700 1 -- 172.21.15.67:0/3949191602 >> 172.21.15.67:0/3949191602 conn(0x7f104c0fb190 msgr2=0x7f104c0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:23:55.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.431+0000 7f1052ddf700 1 -- 172.21.15.67:0/3949191602 shutdown_connections 2024-04-22T23:23:55.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.431+0000 7f1052ddf700 1 -- 172.21.15.67:0/3949191602 wait complete. 2024-04-22T23:23:55.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.432+0000 7f1052ddf700 1 Processor -- start 2024-04-22T23:23:55.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.432+0000 7f1052ddf700 1 -- start start 2024-04-22T23:23:55.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.432+0000 7f1052ddf700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f104c1017a0 0x7f104c196450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:23:55.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.432+0000 7f1052ddf700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f104c1040d0 0x7f104c196990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:23:55.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.432+0000 7f1052ddf700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f104c196f20 con 0x7f104c1040d0 2024-04-22T23:23:55.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.432+0000 7f1052ddf700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f104c197060 con 0x7f104c1017a0 2024-04-22T23:23:55.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.432+0000 7f104bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f104c1040d0 0x7f104c196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:23:55.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.432+0000 7f104bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f104c1040d0 0x7f104c196990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53492/0 (socket says 172.21.15.67:53492) 2024-04-22T23:23:55.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.432+0000 7f104bfff700 1 -- 172.21.15.67:0/3853201670 learned_addr learned my addr 172.21.15.67:0/3853201670 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:23:55.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.432+0000 7f1050b7b700 1 --2- 172.21.15.67:0/3853201670 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f104c1017a0 0x7f104c196450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:23:55.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.433+0000 7f104bfff700 1 -- 172.21.15.67:0/3853201670 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f104c1017a0 msgr2=0x7f104c196450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:55.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.433+0000 7f104bfff700 1 --2- 172.21.15.67:0/3853201670 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f104c1017a0 0x7f104c196450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:55.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.433+0000 7f104bfff700 1 -- 172.21.15.67:0/3853201670 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10380095f0 con 0x7f104c1040d0 2024-04-22T23:23:55.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.433+0000 7f104bfff700 1 --2- 172.21.15.67:0/3853201670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f104c1040d0 0x7f104c196990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f103800bd70 tx=0x7f103800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:23:55.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.433+0000 7f1049ffb700 1 -- 172.21.15.67:0/3853201670 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f103801d070 con 0x7f104c1040d0 2024-04-22T23:23:55.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.433+0000 7f1049ffb700 1 -- 172.21.15.67:0/3853201670 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f1038005940 con 0x7f104c1040d0 2024-04-22T23:23:55.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.433+0000 7f1052ddf700 1 -- 172.21.15.67:0/3853201670 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f104c19bac0 con 0x7f104c1040d0 2024-04-22T23:23:55.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.433+0000 7f1049ffb700 1 -- 172.21.15.67:0/3853201670 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1038021430 con 0x7f104c1040d0 2024-04-22T23:23:55.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.433+0000 7f1052ddf700 1 -- 172.21.15.67:0/3853201670 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f104c19bf80 con 0x7f104c1040d0 2024-04-22T23:23:55.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.434+0000 7f1052ddf700 1 -- 172.21.15.67:0/3853201670 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f104c190610 con 0x7f104c1040d0 2024-04-22T23:23:55.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.435+0000 7f1049ffb700 1 -- 172.21.15.67:0/3853201670 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 20) v1 ==== 91455+0+0 (secure 0 0 0) 0x7f103800f860 con 0x7f104c1040d0 2024-04-22T23:23:55.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.435+0000 7f1049ffb700 1 --2- 172.21.15.67:0/3853201670 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f103c06d710 0x7f103c06fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:23:55.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.435+0000 7f1049ffb700 1 -- 172.21.15.67:0/3853201670 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f103808e290 con 0x7f104c1040d0 2024-04-22T23:23:55.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.435+0000 7f1050b7b700 1 --2- 172.21.15.67:0/3853201670 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f103c06d710 0x7f103c06fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:23:55.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.436+0000 7f1050b7b700 1 --2- 172.21.15.67:0/3853201670 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f103c06d710 0x7f103c06fbd0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f104000ac50 tx=0x7f104000a380 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:23:55.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.440+0000 7f1049ffb700 1 -- 172.21.15.67:0/3853201670 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f103805c4a0 con 0x7f104c1040d0 2024-04-22T23:23:55.453 INFO:tasks.workunit.client.1.smithi134.stdout:4/997: sync 2024-04-22T23:23:55.453 INFO:tasks.workunit.client.1.smithi134.stdout:9/714: sync 2024-04-22T23:23:55.454 INFO:tasks.workunit.client.1.smithi134.stdout:3/557: sync 2024-04-22T23:23:55.454 INFO:tasks.workunit.client.1.smithi134.stdout:7/820: sync 2024-04-22T23:23:55.454 INFO:tasks.workunit.client.1.smithi134.stdout:0/760: sync 2024-04-22T23:23:55.455 INFO:tasks.workunit.client.1.smithi134.stdout:3/558: chown dc/d2a/d62/db2/d3b/l96 49636771 1 2024-04-22T23:23:55.457 INFO:tasks.workunit.client.1.smithi134.stdout:9/715: write d0/d2d/dc1/fc7 [856243,56998] 0 2024-04-22T23:23:55.457 INFO:tasks.workunit.client.1.smithi134.stdout:4/998: read d4/d6b/f12e [2073867,111479] 0 2024-04-22T23:23:55.457 INFO:tasks.workunit.client.1.smithi134.stdout:3/559: fdatasync dc/d2a/d62/db2/d2f/d21/d42/f82 0 2024-04-22T23:23:55.458 INFO:tasks.workunit.client.1.smithi134.stdout:0/761: creat d1/d5/dd/d41/ffc x:0 0 0 2024-04-22T23:23:55.459 INFO:tasks.workunit.client.1.smithi134.stdout:7/821: creat d4/d8/d41/d76/dc1/d2f/d70/f127 x:0 0 0 2024-04-22T23:23:55.460 INFO:tasks.workunit.client.1.smithi134.stdout:7/822: chown d4/d8/d2a/f121 6 1 2024-04-22T23:23:55.461 INFO:tasks.workunit.client.1.smithi134.stdout:0/762: symlink d1/d2/d1d/lfd 0 2024-04-22T23:23:55.462 INFO:tasks.workunit.client.1.smithi134.stdout:0/763: chown d1/d5/d58/da0/de5 37101 1 2024-04-22T23:23:55.463 INFO:tasks.workunit.client.1.smithi134.stdout:3/560: mknod dc/d2a/d62/db2/d2f/d36/d6d/d6f/cce 0 2024-04-22T23:23:55.465 INFO:tasks.workunit.client.1.smithi134.stdout:7/823: rename d4/d4a/lb3 to d4/d8/d41/d76/dc1/d2f/d70/d86/l128 0 2024-04-22T23:23:55.468 INFO:tasks.workunit.client.1.smithi134.stdout:3/561: unlink dc/f3f 0 2024-04-22T23:23:55.471 INFO:tasks.workunit.client.1.smithi134.stdout:7/824: dwrite d4/d8/d2a/f11b [0,4194304] 0 2024-04-22T23:23:55.471 INFO:tasks.workunit.client.1.smithi134.stdout:0/764: symlink d1/d5/dd/lfe 0 2024-04-22T23:23:55.472 INFO:tasks.workunit.client.1.smithi134.stdout:4/999: getdents d4/d7a 0 2024-04-22T23:23:55.482 INFO:tasks.workunit.client.1.smithi134.stdout:0/765: rename d1/d5/dd/d20/c1a to d1/d5/d15/cff 0 2024-04-22T23:23:55.484 INFO:tasks.workunit.client.1.smithi134.stdout:7/825: rename d4/d8/d41/d5b/d112/c11a to d4/d8/d3b/c129 0 2024-04-22T23:23:55.488 INFO:tasks.workunit.client.1.smithi134.stdout:7/826: rmdir d4/d4a/d85/dd7 39 2024-04-22T23:23:55.489 INFO:tasks.workunit.client.1.smithi134.stdout:7/827: chown d4/d8/d41/d76/dc1/d2f/d70 87897 1 2024-04-22T23:23:55.491 INFO:tasks.workunit.client.1.smithi134.stdout:7/828: symlink d4/d4a/l12a 0 2024-04-22T23:23:55.501 INFO:tasks.workunit.client.1.smithi134.stdout:3/562: write f9 [1468438,26567] 0 2024-04-22T23:23:55.506 INFO:tasks.workunit.client.1.smithi134.stdout:3/563: mkdir dc/d2a/d62/d86/dc7/dcf 0 2024-04-22T23:23:55.513 INFO:tasks.workunit.client.1.smithi134.stdout:0/766: dwrite d1/d2/d1d/f9a [0,4194304] 0 2024-04-22T23:23:55.515 INFO:tasks.workunit.client.1.smithi134.stdout:7/829: dwrite d4/fac [0,4194304] 0 2024-04-22T23:23:55.520 INFO:tasks.workunit.client.1.smithi134.stdout:0/767: creat d1/f100 x:0 0 0 2024-04-22T23:23:55.524 INFO:tasks.workunit.client.1.smithi134.stdout:7/830: rmdir d4/d4a/d85 39 2024-04-22T23:23:55.526 INFO:tasks.workunit.client.1.smithi134.stdout:3/564: link dc/d2a/d62/db2/d3b/f31 dc/d2a/d62/db2/d2f/d54/fd0 0 2024-04-22T23:23:55.534 INFO:tasks.workunit.client.1.smithi134.stdout:3/565: dwrite dc/d2a/d62/db2/d2f/d36/d6d/d80/fc0 [0,4194304] 0 2024-04-22T23:23:55.539 INFO:tasks.workunit.client.1.smithi134.stdout:3/566: symlink dc/d2a/d62/db2/d3b/d29/ld1 0 2024-04-22T23:23:55.545 INFO:tasks.workunit.client.1.smithi134.stdout:3/567: mkdir dc/d2a/d62/db2/d2f/d36/d6d/d6f/dd2 0 2024-04-22T23:23:55.547 INFO:tasks.workunit.client.1.smithi134.stdout:7/831: dwrite d4/f7b [0,4194304] 0 2024-04-22T23:23:55.550 INFO:tasks.workunit.client.1.smithi134.stdout:3/568: creat dc/d2a/d62/db2/d2f/d36/d5f/fd3 x:0 0 0 2024-04-22T23:23:55.552 INFO:tasks.workunit.client.1.smithi134.stdout:7/832: symlink d4/d8/d41/d76/dc1/d2f/d70/d86/l12b 0 2024-04-22T23:23:55.552 INFO:tasks.workunit.client.1.smithi134.stdout:0/768: link d1/d5/dd/d41/cd1 d1/d5/dd/d20/d2c/c101 0 2024-04-22T23:23:55.559 INFO:tasks.workunit.client.1.smithi134.stdout:7/833: link d4/d8/d3b/l59 d4/d8/d41/d76/dc1/d2f/d70/d86/l12c 0 2024-04-22T23:23:55.560 INFO:tasks.workunit.client.1.smithi134.stdout:7/834: read - d4/d8/d2a/d42/d7d/fa4 zero size 2024-04-22T23:23:55.563 INFO:tasks.workunit.client.1.smithi134.stdout:7/835: creat d4/d8/d2a/d42/d6e/f12d x:0 0 0 2024-04-22T23:23:55.566 INFO:tasks.workunit.client.1.smithi134.stdout:7/836: mkdir d4/d8/d41/d76/dc1/d2f/d34/d62/d29/dc5/d12e 0 2024-04-22T23:23:55.568 INFO:tasks.workunit.client.1.smithi134.stdout:7/837: write d4/d8/d3b/fd1 [924211,8352] 0 2024-04-22T23:23:55.570 INFO:tasks.workunit.client.1.smithi134.stdout:0/769: dwrite d1/d5/d58/f29 [0,4194304] 0 2024-04-22T23:23:55.578 INFO:tasks.workunit.client.1.smithi134.stdout:0/770: dwrite d1/d5/d15/fe9 [0,4194304] 0 2024-04-22T23:23:55.582 INFO:tasks.workunit.client.1.smithi134.stdout:0/771: read - d1/d5/d15/fb4 zero size 2024-04-22T23:23:55.585 INFO:tasks.workunit.client.1.smithi134.stdout:7/838: truncate d4/d8/d41/d5b/ff3 3083030 0 2024-04-22T23:23:55.586 INFO:tasks.workunit.client.1.smithi134.stdout:0/772: truncate d1/d5/d58/d22/fa7 4836988 0 2024-04-22T23:23:55.588 INFO:tasks.workunit.client.1.smithi134.stdout:0/773: mknod d1/d5/d58/d6b/db9/c102 0 2024-04-22T23:23:55.589 INFO:tasks.workunit.client.1.smithi134.stdout:0/774: write d1/d5/dd/f7a [10428081,123448] 0 2024-04-22T23:23:55.613 INFO:tasks.workunit.client.1.smithi134.stdout:0/775: dread d1/d5/d58/d6b/db9/f6d [0,4194304] 0 2024-04-22T23:23:55.613 INFO:tasks.workunit.client.1.smithi134.stdout:0/776: chown d1/d5/d58/da0 12872 1 2024-04-22T23:23:55.616 INFO:tasks.workunit.client.1.smithi134.stdout:6/836: sync 2024-04-22T23:23:55.622 INFO:tasks.workunit.client.1.smithi134.stdout:6/837: dwrite d0/d19/de/d12/d29/d74/f89 [0,4194304] 0 2024-04-22T23:23:55.631 INFO:tasks.workunit.client.1.smithi134.stdout:6/838: link d0/d19/de/d1f/f2c d0/d19/de/d12/d2a/d2d/d3a/d98/f110 0 2024-04-22T23:23:55.633 INFO:tasks.workunit.client.1.smithi134.stdout:6/839: read d0/d19/de/d12/d5e/d6d/f73 [191084,26802] 0 2024-04-22T23:23:55.634 INFO:tasks.workunit.client.1.smithi134.stdout:0/777: write d1/d2/f74 [5225430,57083] 0 2024-04-22T23:23:55.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.679+0000 7f1052ddf700 1 -- 172.21.15.67:0/3853201670 --> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f104c0611d0 con 0x7f103c06d710 2024-04-22T23:23:55.692 INFO:tasks.workunit.client.1.smithi134.stdout:0/778: rename d1/d5/d58/d6b/db9/def/lf2 to d1/d5/d58/d6b/l103 0 2024-04-22T23:23:55.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.690+0000 7f1049ffb700 1 -- 172.21.15.67:0/3853201670 <== mgr.14215 v2:172.21.15.67:6800/2589747850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f104c0611d0 con 0x7f103c06d710 2024-04-22T23:23:55.692 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:23:55.692 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (7m) 3m ago 8m 26.7M - 0.25.0 c8568f914cd2 8c7b7ec00657 2024-04-22T23:23:55.692 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (9m) 3m ago 9m 7490k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:23:55.692 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (8m) 22s ago 8m 7738k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:23:55.692 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (9m) 3m ago 9m 6660k - 18.2.1 5be31c24972a 14085634c5d3 2024-04-22T23:23:55.692 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (8m) 22s ago 8m 6665k - 18.2.1 5be31c24972a fa995d458353 2024-04-22T23:23:55.692 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (7m) 3m ago 8m 85.0M - 9.4.7 954c08fa6188 0d8950c9af0c 2024-04-22T23:23:55.693 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (3m) 3m ago 3m 12.8M - 18.2.1 5be31c24972a e24a471f0a8c 2024-04-22T23:23:55.693 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (3m) 3m ago 3m 16.7M - 18.2.1 5be31c24972a 0730f9fc8e61 2024-04-22T23:23:55.693 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (3m) 22s ago 3m 131M - 18.2.1 5be31c24972a b6a169586433 2024-04-22T23:23:55.693 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (3m) 22s ago 3m 106M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:23:55.693 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:9283,8765,8443 running (10m) 3m ago 10m 490M - 18.2.1 5be31c24972a 1c64cb9e36bb 2024-04-22T23:23:55.693 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (43s) 22s ago 7m 93.0M - 19.0.0-3252-g43be0201 a0434699c1d7 a86aaa7b9552 2024-04-22T23:23:55.693 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (10m) 3m ago 10m 46.1M 2048M 18.2.1 5be31c24972a 6789283703e0 2024-04-22T23:23:55.693 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (7m) 22s ago 7m 36.5M 2048M 18.2.1 5be31c24972a 2fb23795f7e3 2024-04-22T23:23:55.693 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (9m) 3m ago 8m 16.3M - 1.5.0 0da6a335fe13 4aa4da7089d0 2024-04-22T23:23:55.693 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (8m) 22s ago 8m 15.4M - 1.5.0 0da6a335fe13 26e43c2e756c 2024-04-22T23:23:55.693 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (7m) 3m ago 7m 48.5M 3946M 18.2.1 5be31c24972a a033247ee8c2 2024-04-22T23:23:55.693 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (6m) 3m ago 6m 47.1M 3946M 18.2.1 5be31c24972a 915b07a8edc4 2024-04-22T23:23:55.693 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (6m) 3m ago 6m 46.8M 3946M 18.2.1 5be31c24972a e257ecb4efa1 2024-04-22T23:23:55.693 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (5m) 22s ago 5m 100M 4970M 18.2.1 5be31c24972a 9c2c91a967ba 2024-04-22T23:23:55.693 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (5m) 22s ago 5m 92.7M 4970M 18.2.1 5be31c24972a c7586562ffa6 2024-04-22T23:23:55.693 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (5m) 22s ago 5m 89.7M 4970M 18.2.1 5be31c24972a 9fd97f1c65cb 2024-04-22T23:23:55.694 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (7m) 3m ago 8m 55.5M - 2.43.0 a07b618ecd1d d87cb15b5e55 2024-04-22T23:23:55.694 INFO:tasks.workunit.client.1.smithi134.stdout:6/840: dwrite d0/d19/de/d12/d29/d74/da0/fdd [4194304,4194304] 0 2024-04-22T23:23:55.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.692+0000 7f1052ddf700 1 -- 172.21.15.67:0/3853201670 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f103c06d710 msgr2=0x7f103c06fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:55.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.692+0000 7f1052ddf700 1 --2- 172.21.15.67:0/3853201670 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f103c06d710 0x7f103c06fbd0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f104000ac50 tx=0x7f104000a380 comp rx=0 tx=0).stop 2024-04-22T23:23:55.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.692+0000 7f1052ddf700 1 -- 172.21.15.67:0/3853201670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f104c1040d0 msgr2=0x7f104c196990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:55.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.692+0000 7f1052ddf700 1 --2- 172.21.15.67:0/3853201670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f104c1040d0 0x7f104c196990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f103800bd70 tx=0x7f103800bda0 comp rx=0 tx=0).stop 2024-04-22T23:23:55.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.692+0000 7f1052ddf700 1 -- 172.21.15.67:0/3853201670 shutdown_connections 2024-04-22T23:23:55.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.692+0000 7f1052ddf700 1 --2- 172.21.15.67:0/3853201670 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f103c06d710 0x7f103c06fbd0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:55.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.692+0000 7f1052ddf700 1 --2- 172.21.15.67:0/3853201670 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f104c1017a0 0x7f104c196450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:55.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.692+0000 7f1052ddf700 1 --2- 172.21.15.67:0/3853201670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f104c1040d0 0x7f104c196990 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:55.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.692+0000 7f1052ddf700 1 -- 172.21.15.67:0/3853201670 >> 172.21.15.67:0/3853201670 conn(0x7f104c0fb190 msgr2=0x7f104c0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:23:55.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.693+0000 7f1052ddf700 1 -- 172.21.15.67:0/3853201670 shutdown_connections 2024-04-22T23:23:55.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.693+0000 7f1052ddf700 1 -- 172.21.15.67:0/3853201670 wait complete. 2024-04-22T23:23:55.700 INFO:tasks.workunit.client.1.smithi134.stdout:0/779: dwrite d1/d5/d58/d22/f24 [4194304,4194304] 0 2024-04-22T23:23:55.701 INFO:tasks.workunit.client.1.smithi134.stdout:0/780: chown d1/d5/dd/f7a 22382826 1 2024-04-22T23:23:55.705 INFO:tasks.workunit.client.1.smithi134.stdout:6/841: dread d0/d19/de/d1f/f2c [0,4194304] 0 2024-04-22T23:23:55.707 INFO:tasks.workunit.client.1.smithi134.stdout:0/781: creat d1/d5/d15/f104 x:0 0 0 2024-04-22T23:23:55.708 INFO:tasks.workunit.client.1.smithi134.stdout:6/842: creat d0/d19/de/d1f/dae/f111 x:0 0 0 2024-04-22T23:23:55.709 INFO:tasks.workunit.client.1.smithi134.stdout:6/843: fsync d0/f60 0 2024-04-22T23:23:55.709 INFO:tasks.workunit.client.1.smithi134.stdout:0/782: dread - d1/d2/d1d/f91 zero size 2024-04-22T23:23:55.710 INFO:tasks.workunit.client.1.smithi134.stdout:0/783: chown d1/d5/dd/d41/fe0 2293938 1 2024-04-22T23:23:55.711 INFO:tasks.workunit.client.1.smithi134.stdout:0/784: chown d1/d2/cfa 1376 1 2024-04-22T23:23:55.711 INFO:tasks.workunit.client.1.smithi134.stdout:6/844: symlink d0/d19/l112 0 2024-04-22T23:23:55.712 INFO:tasks.workunit.client.1.smithi134.stdout:0/785: truncate d1/f44 4736918 0 2024-04-22T23:23:55.713 INFO:tasks.workunit.client.1.smithi134.stdout:6/845: mkdir d0/d19/d113 0 2024-04-22T23:23:55.715 INFO:tasks.workunit.client.1.smithi134.stdout:6/846: truncate d0/d19/fa 831775 0 2024-04-22T23:23:55.718 INFO:tasks.workunit.client.1.smithi134.stdout:6/847: creat d0/d19/db/dfa/f114 x:0 0 0 2024-04-22T23:23:55.723 INFO:tasks.workunit.client.1.smithi134.stdout:6/848: dwrite d0/d19/db/d35/d7f/fe4 [0,4194304] 0 2024-04-22T23:23:55.727 INFO:tasks.workunit.client.1.smithi134.stdout:6/849: link d0/d19/de/d12/d2a/d2d/f80 d0/d19/de/f115 0 2024-04-22T23:23:55.729 INFO:tasks.workunit.client.1.smithi134.stdout:6/850: creat d0/d19/de/d12/d2a/d2d/d3a/d98/df5/f116 x:0 0 0 2024-04-22T23:23:55.734 INFO:tasks.workunit.client.1.smithi134.stdout:6/851: dwrite d0/d19/de/d12/d5e/d6d/d7a/d99/fea [0,4194304] 0 2024-04-22T23:23:55.781 INFO:tasks.workunit.client.1.smithi134.stdout:0/786: dwrite d1/d5/dd/d20/d2c/f4b [0,4194304] 0 2024-04-22T23:23:55.782 INFO:tasks.workunit.client.1.smithi134.stdout:0/787: write d1/d5/dd/d41/fe0 [690678,69995] 0 2024-04-22T23:23:55.789 INFO:tasks.workunit.client.1.smithi134.stdout:6/852: dwrite d0/d19/de/d12/d29/db5/fcf [0,4194304] 0 2024-04-22T23:23:55.794 INFO:tasks.workunit.client.1.smithi134.stdout:6/853: creat d0/d19/db/d35/f117 x:0 0 0 2024-04-22T23:23:55.810 INFO:tasks.workunit.client.1.smithi134.stdout:6/854: dwrite d0/f16 [0,4194304] 0 2024-04-22T23:23:55.811 INFO:tasks.workunit.client.1.smithi134.stdout:6/855: read - d0/d19/de/d12/d29/d74/deb/ffc zero size 2024-04-22T23:23:55.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.813+0000 7f813fbed700 1 -- 172.21.15.67:0/2313071937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81380ff5b0 msgr2=0x7f81380ff9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:55.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.813+0000 7f813fbed700 1 --2- 172.21.15.67:0/2313071937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81380ff5b0 0x7f81380ff9d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8134009910 tx=0x7f8134009c20 comp rx=0 tx=0).stop 2024-04-22T23:23:55.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.813+0000 7f813fbed700 1 -- 172.21.15.67:0/2313071937 shutdown_connections 2024-04-22T23:23:55.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.813+0000 7f813fbed700 1 --2- 172.21.15.67:0/2313071937 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f81380fff10 0x7f8138100390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:55.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.813+0000 7f813fbed700 1 --2- 172.21.15.67:0/2313071937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81380ff5b0 0x7f81380ff9d0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:55.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.813+0000 7f813fbed700 1 -- 172.21.15.67:0/2313071937 >> 172.21.15.67:0/2313071937 conn(0x7f81380fb190 msgr2=0x7f81380fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:23:55.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.813+0000 7f813fbed700 1 -- 172.21.15.67:0/2313071937 shutdown_connections 2024-04-22T23:23:55.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.813+0000 7f813fbed700 1 -- 172.21.15.67:0/2313071937 wait complete. 2024-04-22T23:23:55.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.814+0000 7f813fbed700 1 Processor -- start 2024-04-22T23:23:55.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.814+0000 7f813fbed700 1 -- start start 2024-04-22T23:23:55.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.814+0000 7f813fbed700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f81380ff5b0 0x7f8138198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:23:55.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.814+0000 7f813fbed700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81380fff10 0x7f8138198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:23:55.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.814+0000 7f813fbed700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8138199270 con 0x7f81380fff10 2024-04-22T23:23:55.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.814+0000 7f813fbed700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81381993b0 con 0x7f81380ff5b0 2024-04-22T23:23:55.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.814+0000 7f813d188700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81380fff10 0x7f8138198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:23:55.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.814+0000 7f813d188700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81380fff10 0x7f8138198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53506/0 (socket says 172.21.15.67:53506) 2024-04-22T23:23:55.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.814+0000 7f813d188700 1 -- 172.21.15.67:0/3748584440 learned_addr learned my addr 172.21.15.67:0/3748584440 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:23:55.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.814+0000 7f813d989700 1 --2- 172.21.15.67:0/3748584440 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f81380ff5b0 0x7f8138198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:23:55.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.815+0000 7f813d188700 1 -- 172.21.15.67:0/3748584440 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f81380ff5b0 msgr2=0x7f8138198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:55.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.815+0000 7f813d188700 1 --2- 172.21.15.67:0/3748584440 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f81380ff5b0 0x7f8138198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:55.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.815+0000 7f813d188700 1 -- 172.21.15.67:0/3748584440 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81340095f0 con 0x7f81380fff10 2024-04-22T23:23:55.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.815+0000 7f813d188700 1 --2- 172.21.15.67:0/3748584440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81380fff10 0x7f8138198c50 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f812800b4b0 tx=0x7f812800b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:23:55.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.815+0000 7f812effd700 1 -- 172.21.15.67:0/3748584440 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8128011800 con 0x7f81380fff10 2024-04-22T23:23:55.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.815+0000 7f812effd700 1 -- 172.21.15.67:0/3748584440 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8128011e10 con 0x7f81380fff10 2024-04-22T23:23:55.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.815+0000 7f812effd700 1 -- 172.21.15.67:0/3748584440 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f812800f2e0 con 0x7f81380fff10 2024-04-22T23:23:55.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.815+0000 7f813fbed700 1 -- 172.21.15.67:0/3748584440 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f813819de60 con 0x7f81380fff10 2024-04-22T23:23:55.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.815+0000 7f813fbed700 1 -- 172.21.15.67:0/3748584440 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f813819e380 con 0x7f81380fff10 2024-04-22T23:23:55.819 INFO:tasks.workunit.client.1.smithi134.stdout:6/856: rmdir d0/d19/de/d12/de5 0 2024-04-22T23:23:55.819 INFO:tasks.workunit.client.1.smithi134.stdout:8/764: sync 2024-04-22T23:23:55.819 INFO:tasks.workunit.client.1.smithi134.stdout:6/857: creat d0/d19/db/d75/d92/f118 x:0 0 0 2024-04-22T23:23:55.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.816+0000 7f813fbed700 1 -- 172.21.15.67:0/3748584440 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f813810ba70 con 0x7f81380fff10 2024-04-22T23:23:55.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.817+0000 7f812effd700 1 -- 172.21.15.67:0/3748584440 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 20) v1 ==== 91455+0+0 (secure 0 0 0) 0x7f812800f440 con 0x7f81380fff10 2024-04-22T23:23:55.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.818+0000 7f812effd700 1 --2- 172.21.15.67:0/3748584440 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f812406d710 0x7f812406fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:23:55.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.818+0000 7f812effd700 1 -- 172.21.15.67:0/3748584440 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f812808ca80 con 0x7f81380fff10 2024-04-22T23:23:55.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.818+0000 7f813d989700 1 --2- 172.21.15.67:0/3748584440 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f812406d710 0x7f812406fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:23:55.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.818+0000 7f813d989700 1 --2- 172.21.15.67:0/3748584440 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f812406d710 0x7f812406fbd0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8134006010 tx=0x7f813400b540 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:23:55.821 INFO:tasks.workunit.client.1.smithi134.stdout:8/765: creat d7/db/d42/d65/dd9/dda/de7/fef x:0 0 0 2024-04-22T23:23:55.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:55.823+0000 7f812effd700 1 -- 172.21.15.67:0/3748584440 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f812805ad10 con 0x7f81380fff10 2024-04-22T23:23:55.827 INFO:tasks.workunit.client.1.smithi134.stdout:6/858: getdents d0/d19/de/d12/d5e/dc1 0 2024-04-22T23:23:55.829 INFO:tasks.workunit.client.1.smithi134.stdout:6/859: mknod d0/d19/de/d12/d29/d74/c119 0 2024-04-22T23:23:55.830 INFO:tasks.workunit.client.1.smithi134.stdout:6/860: creat d0/d19/db/dfa/f11a x:0 0 0 2024-04-22T23:23:55.836 INFO:tasks.workunit.client.1.smithi134.stdout:8/766: dwrite d7/dc/d2c/d4f/d6c/f81 [0,4194304] 0 2024-04-22T23:23:55.839 INFO:tasks.workunit.client.1.smithi134.stdout:8/767: creat d7/dc/d2c/d4f/ff0 x:0 0 0 2024-04-22T23:23:55.839 INFO:tasks.workunit.client.1.smithi134.stdout:6/861: link d0/d19/de/d2f/f32 d0/d19/de/dbe/dd1/de9/f11b 0 2024-04-22T23:23:55.841 INFO:tasks.workunit.client.1.smithi134.stdout:6/862: mkdir d0/d19/de/d12/d29/db5/d11c 0 2024-04-22T23:23:55.844 INFO:tasks.workunit.client.1.smithi134.stdout:8/768: rename d7/db/l41 to d7/db/d42/d65/d46/d9a/da3/lf1 0 2024-04-22T23:23:55.848 INFO:tasks.workunit.client.1.smithi134.stdout:6/863: link d0/d19/de/d1f/dae/cd2 d0/d19/de/d12/d5e/dc1/dcd/c11d 0 2024-04-22T23:23:55.848 INFO:tasks.workunit.client.1.smithi134.stdout:6/864: chown d0/d19/db/d75/l7c 992556 1 2024-04-22T23:23:55.850 INFO:tasks.workunit.client.1.smithi134.stdout:6/865: write d0/d6f/f7b [2614458,4421] 0 2024-04-22T23:23:55.852 INFO:tasks.workunit.client.1.smithi134.stdout:6/866: write d0/d19/de/dbe/dd1/ff0 [532680,72975] 0 2024-04-22T23:23:55.853 INFO:tasks.workunit.client.1.smithi134.stdout:6/867: read d0/d19/de/d1f/dae/fd8 [3551016,31610] 0 2024-04-22T23:23:55.855 INFO:tasks.workunit.client.1.smithi134.stdout:6/868: chown d0/d19/de/d12/d5e/d6d/d7a/f10c 6433 1 2024-04-22T23:23:55.857 INFO:tasks.workunit.client.1.smithi134.stdout:6/869: mknod d0/d19/de/dbe/c11e 0 2024-04-22T23:23:55.862 INFO:tasks.workunit.client.1.smithi134.stdout:8/769: readlink d7/db/d2d/l44 0 2024-04-22T23:23:55.863 INFO:tasks.workunit.client.1.smithi134.stdout:8/770: readlink d7/dc/d19/l78 0 2024-04-22T23:23:55.863 INFO:tasks.workunit.client.1.smithi134.stdout:6/870: dwrite d0/d6f/f84 [4194304,4194304] 0 2024-04-22T23:23:55.866 INFO:tasks.workunit.client.1.smithi134.stdout:8/771: creat d7/dc/d2c/d37/ff2 x:0 0 0 2024-04-22T23:23:55.869 INFO:tasks.workunit.client.1.smithi134.stdout:6/871: rename d0/d19/de/d12/d2a/d2d/f9c to d0/d19/db/d33/d86/f11f 0 2024-04-22T23:23:55.873 INFO:tasks.workunit.client.1.smithi134.stdout:6/872: rename d0/d19/de/d12/c15 to d0/d6f/c120 0 2024-04-22T23:23:55.874 INFO:tasks.workunit.client.1.smithi134.stdout:6/873: rmdir d0/d19/de/d12/d5e/d6d/dd3 39 2024-04-22T23:23:55.884 INFO:tasks.workunit.client.1.smithi134.stdout:6/874: read d0/d19/db/fd [3311658,6201] 0 2024-04-22T23:23:55.928 INFO:tasks.workunit.client.1.smithi134.stdout:6/875: mkdir d0/d19/db/d121 0 2024-04-22T23:23:55.966 INFO:tasks.workunit.client.1.smithi134.stdout:6/876: dwrite d0/d6f/fa2 [4194304,4194304] 0 2024-04-22T23:23:55.976 INFO:tasks.workunit.client.1.smithi134.stdout:6/877: dwrite d0/d19/de/d12/d29/d74/deb/ffc [0,4194304] 0 2024-04-22T23:23:55.978 INFO:tasks.workunit.client.1.smithi134.stdout:6/878: dread d0/d19/de/d1f/f2c [0,4194304] 0 2024-04-22T23:23:55.982 INFO:tasks.workunit.client.1.smithi134.stdout:6/879: mkdir d0/d19/de/d1f/dd5/d122 0 2024-04-22T23:23:55.991 INFO:tasks.workunit.client.1.smithi134.stdout:2/673: sync 2024-04-22T23:23:55.999 INFO:tasks.workunit.client.1.smithi134.stdout:2/674: getdents d2/dd/d4a/d73 0 2024-04-22T23:23:56.001 INFO:tasks.workunit.client.1.smithi134.stdout:6/880: write d0/d19/de/dbe/dd1/de9/f11b [4592163,87064] 0 2024-04-22T23:23:56.003 INFO:tasks.workunit.client.1.smithi134.stdout:6/881: write d0/d6f/f7b [3790162,46946] 0 2024-04-22T23:23:56.011 INFO:tasks.workunit.client.1.smithi134.stdout:2/675: dread d2/d42/d54/fd6 [4194304,4194304] 0 2024-04-22T23:23:56.012 INFO:tasks.workunit.client.1.smithi134.stdout:2/676: readlink d2/d15/d24/l4d 0 2024-04-22T23:23:56.013 INFO:tasks.workunit.client.1.smithi134.stdout:6/882: dwrite d0/d19/de/d12/d29/d74/f8e [0,4194304] 0 2024-04-22T23:23:56.017 INFO:tasks.workunit.client.1.smithi134.stdout:2/677: creat d2/d42/d54/d33/d91/fe7 x:0 0 0 2024-04-22T23:23:56.018 INFO:tasks.workunit.client.1.smithi134.stdout:6/883: creat d0/d19/de/d12/d5e/d6d/d7a/db2/f123 x:0 0 0 2024-04-22T23:23:56.021 INFO:tasks.workunit.client.1.smithi134.stdout:2/678: creat d2/d42/fe8 x:0 0 0 2024-04-22T23:23:56.024 INFO:tasks.workunit.client.1.smithi134.stdout:2/679: creat d2/d15/d24/d3b/d8a/dc7/fe9 x:0 0 0 2024-04-22T23:23:56.026 INFO:tasks.workunit.client.1.smithi134.stdout:1/977: sync 2024-04-22T23:23:56.030 INFO:tasks.workunit.client.1.smithi134.stdout:1/978: creat db/d15/d17/d112/f144 x:0 0 0 2024-04-22T23:23:56.033 INFO:tasks.workunit.client.1.smithi134.stdout:1/979: creat db/d15/d17/d20/d2a/f145 x:0 0 0 2024-04-22T23:23:56.037 INFO:tasks.workunit.client.1.smithi134.stdout:2/680: dwrite d2/dd/d4a/f52 [0,4194304] 0 2024-04-22T23:23:56.039 INFO:tasks.workunit.client.1.smithi134.stdout:1/980: dread db/d15/fc6 [0,4194304] 0 2024-04-22T23:23:56.043 INFO:tasks.workunit.client.1.smithi134.stdout:1/981: symlink db/db5/l146 0 2024-04-22T23:23:56.045 INFO:tasks.workunit.client.1.smithi134.stdout:1/982: write db/d15/d17/d20/d2a/d2d/d2e/f137 [284293,86298] 0 2024-04-22T23:23:56.050 INFO:tasks.workunit.client.1.smithi134.stdout:2/681: getdents d2/d42/d54/d33/d91 0 2024-04-22T23:23:56.051 INFO:tasks.workunit.client.1.smithi134.stdout:2/682: chown d2/d15/d24/l49 648 1 2024-04-22T23:23:56.056 INFO:tasks.workunit.client.1.smithi134.stdout:2/683: dread d2/dd/d4a/f64 [0,4194304] 0 2024-04-22T23:23:56.057 INFO:tasks.workunit.client.1.smithi134.stdout:1/983: write db/d15/f29 [7815925,23233] 0 2024-04-22T23:23:56.059 INFO:tasks.workunit.client.1.smithi134.stdout:2/684: rmdir d2/d15/d24/d3b/d8a/dc7 39 2024-04-22T23:23:56.063 INFO:tasks.workunit.client.1.smithi134.stdout:1/984: chown db/d15/d17/d20/d2a/d2d/d2e/c96 150 1 2024-04-22T23:23:56.066 INFO:tasks.workunit.client.1.smithi134.stdout:2/685: dwrite d2/dd/d4a/d73/f7f [0,4194304] 0 2024-04-22T23:23:56.070 INFO:tasks.workunit.client.1.smithi134.stdout:2/686: mkdir d2/d15/d24/d1d/dba/dea 0 2024-04-22T23:23:56.071 INFO:tasks.workunit.client.1.smithi134.stdout:2/687: read d2/dd/d4a/d73/fdb [4704112,13640] 0 2024-04-22T23:23:56.071 INFO:tasks.workunit.client.1.smithi134.stdout:1/985: link db/db0/dc1/l13f db/d15/d17/d20/l147 0 2024-04-22T23:23:56.073 INFO:tasks.workunit.client.1.smithi134.stdout:1/986: chown db/d15/d17/d20/d2a/d2d/d2e/f6c 189 1 2024-04-22T23:23:56.076 INFO:tasks.workunit.client.1.smithi134.stdout:1/987: readlink db/db0/lca 0 2024-04-22T23:23:56.080 INFO:tasks.workunit.client.1.smithi134.stdout:2/688: dwrite d2/d15/d24/d1d/d3d/fc8 [0,4194304] 0 2024-04-22T23:23:56.083 INFO:tasks.workunit.client.1.smithi134.stdout:1/988: rename db/l37 to db/db0/l148 0 2024-04-22T23:23:56.083 INFO:tasks.workunit.client.1.smithi134.stdout:1/989: stat db/d16/f3d 0 2024-04-22T23:23:56.085 INFO:tasks.workunit.client.1.smithi134.stdout:1/990: chown db/d15/d17/d20/d2a/d2d/d108/d114/f9a 456 1 2024-04-22T23:23:56.087 INFO:tasks.workunit.client.1.smithi134.stdout:1/991: write db/d15/d17/d20/d2a/d2d/d86/fa4 [4813354,6035] 0 2024-04-22T23:23:56.088 INFO:tasks.workunit.client.1.smithi134.stdout:2/689: unlink d2/dd/df/f97 0 2024-04-22T23:23:56.104 INFO:tasks.workunit.client.1.smithi134.stdout:2/690: write d2/f60 [395307,111503] 0 2024-04-22T23:23:56.105 INFO:tasks.workunit.client.1.smithi134.stdout:1/992: truncate db/d15/d121/f131 217893 0 2024-04-22T23:23:56.106 INFO:tasks.workunit.client.1.smithi134.stdout:1/993: write db/d16/d104/f11a [16969,71565] 0 2024-04-22T23:23:56.109 INFO:tasks.workunit.client.1.smithi134.stdout:1/994: chown db/d16/l8e 37865595 1 2024-04-22T23:23:56.111 INFO:tasks.workunit.client.1.smithi134.stdout:2/691: link d2/dd/l1a d2/leb 0 2024-04-22T23:23:56.116 INFO:tasks.workunit.client.1.smithi134.stdout:1/995: write db/d16/f9e [3571492,43867] 0 2024-04-22T23:23:56.118 INFO:tasks.workunit.client.1.smithi134.stdout:1/996: readlink l4 0 2024-04-22T23:23:56.120 INFO:tasks.workunit.client.1.smithi134.stdout:1/997: write db/d15/d17/d20/d2a/f91 [2029356,49791] 0 2024-04-22T23:23:56.123 INFO:tasks.workunit.client.1.smithi134.stdout:1/998: creat db/d15/d17/d20/d2a/f149 x:0 0 0 2024-04-22T23:23:56.128 INFO:tasks.workunit.client.1.smithi134.stdout:2/692: dread d2/d15/d24/d1d/d3d/d9a/fd5 [0,4194304] 0 2024-04-22T23:23:56.134 INFO:tasks.workunit.client.1.smithi134.stdout:2/693: dwrite d2/fc1 [0,4194304] 0 2024-04-22T23:23:56.136 INFO:tasks.workunit.client.1.smithi134.stdout:1/999: truncate db/d15/d17/d20/d2a/d2d/f43 932983 0 2024-04-22T23:23:56.138 INFO:tasks.workunit.client.1.smithi134.stdout:2/694: symlink d2/d15/lec 0 2024-04-22T23:23:56.141 INFO:tasks.workunit.client.1.smithi134.stdout:2/695: link d2/d42/lb2 d2/d15/d24/d3b/led 0 2024-04-22T23:23:56.147 INFO:tasks.workunit.client.1.smithi134.stdout:2/696: link d2/dd/df/l63 d2/dd/df/d9c/daa/db0/lee 0 2024-04-22T23:23:56.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.149+0000 7f813fbed700 1 -- 172.21.15.67:0/3748584440 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f813819ea70 con 0x7f81380fff10 2024-04-22T23:23:56.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.149+0000 7f812effd700 1 -- 172.21.15.67:0/3748584440 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+766 (secure 0 0 0) 0x7f8128015310 con 0x7f81380fff10 2024-04-22T23:23:56.151 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:23:56.151 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:23:56.151 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-22T23:23:56.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:23:56.151 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:23:56.151 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:23:56.151 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 1 2024-04-22T23:23:56.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:23:56.151 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:23:56.151 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-22T23:23:56.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:23:56.151 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:23:56.151 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-22T23:23:56.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:23:56.151 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:23:56.152 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 13, 2024-04-22T23:23:56.152 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 1 2024-04-22T23:23:56.152 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:23:56.152 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:23:56.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.151+0000 7f813fbed700 1 -- 172.21.15.67:0/3748584440 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f812406d710 msgr2=0x7f812406fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:56.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.151+0000 7f813fbed700 1 --2- 172.21.15.67:0/3748584440 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f812406d710 0x7f812406fbd0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8134006010 tx=0x7f813400b540 comp rx=0 tx=0).stop 2024-04-22T23:23:56.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.151+0000 7f813fbed700 1 -- 172.21.15.67:0/3748584440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81380fff10 msgr2=0x7f8138198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:56.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.151+0000 7f813fbed700 1 --2- 172.21.15.67:0/3748584440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81380fff10 0x7f8138198c50 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f812800b4b0 tx=0x7f812800b7c0 comp rx=0 tx=0).stop 2024-04-22T23:23:56.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.151+0000 7f813fbed700 1 -- 172.21.15.67:0/3748584440 shutdown_connections 2024-04-22T23:23:56.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.151+0000 7f813fbed700 1 --2- 172.21.15.67:0/3748584440 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f812406d710 0x7f812406fbd0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:56.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.151+0000 7f813fbed700 1 --2- 172.21.15.67:0/3748584440 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f81380ff5b0 0x7f8138198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:56.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.151+0000 7f813fbed700 1 --2- 172.21.15.67:0/3748584440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81380fff10 0x7f8138198c50 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:56.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.151+0000 7f813fbed700 1 -- 172.21.15.67:0/3748584440 >> 172.21.15.67:0/3748584440 conn(0x7f81380fb190 msgr2=0x7f8138107be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:23:56.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.152+0000 7f813fbed700 1 -- 172.21.15.67:0/3748584440 shutdown_connections 2024-04-22T23:23:56.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.152+0000 7f813fbed700 1 -- 172.21.15.67:0/3748584440 wait complete. 2024-04-22T23:23:56.161 INFO:tasks.workunit.client.1.smithi134.stdout:2/697: dread d2/d15/d24/d1d/f26 [0,4194304] 0 2024-04-22T23:23:56.179 INFO:tasks.workunit.client.1.smithi134.stdout:2/698: dread d2/d15/d24/d1d/f44 [0,4194304] 0 2024-04-22T23:23:56.180 INFO:tasks.workunit.client.1.smithi134.stdout:2/699: dread - d2/d15/d24/d1d/f86 zero size 2024-04-22T23:23:56.181 INFO:tasks.workunit.client.1.smithi134.stdout:2/700: chown d2/d15/d24/d1d/f26 31883 1 2024-04-22T23:23:56.188 INFO:tasks.workunit.client.1.smithi134.stdout:2/701: write d2/d42/d54/f9e [341370,108100] 0 2024-04-22T23:23:56.191 INFO:tasks.workunit.client.1.smithi134.stdout:2/702: creat d2/d15/d24/d3b/d8a/d99/fef x:0 0 0 2024-04-22T23:23:56.193 INFO:tasks.workunit.client.1.smithi134.stdout:2/703: rmdir d2/d42/d54 39 2024-04-22T23:23:56.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:55 smithi134 ceph-mon[34609]: from='client.14646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:23:56.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:55 smithi134 ceph-mon[34609]: fsmap cephfs:2 {0=cephfs.smithi067.xyazmh=up:active,1=cephfs.smithi134.owipad=up:active} 2 up:standby-replay 2024-04-22T23:23:56.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:55 smithi134 ceph-mon[34609]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:23:56.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:55 smithi134 ceph-mon[34609]: pgmap v283: 65 pgs: 65 active+clean; 2.7 GiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 71 MiB/s rd, 199 MiB/s wr, 491 op/s 2024-04-22T23:23:56.245 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:55 smithi067 ceph-mon[26453]: from='client.14646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:23:56.245 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:55 smithi067 ceph-mon[26453]: fsmap cephfs:2 {0=cephfs.smithi067.xyazmh=up:active,1=cephfs.smithi134.owipad=up:active} 2 up:standby-replay 2024-04-22T23:23:56.245 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:55 smithi067 ceph-mon[26453]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:23:56.245 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:55 smithi067 ceph-mon[26453]: pgmap v283: 65 pgs: 65 active+clean; 2.7 GiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 71 MiB/s rd, 199 MiB/s wr, 491 op/s 2024-04-22T23:23:56.264 INFO:tasks.workunit.client.1.smithi134.stdout:3/569: sync 2024-04-22T23:23:56.264 INFO:tasks.workunit.client.1.smithi134.stdout:9/716: sync 2024-04-22T23:23:56.264 INFO:tasks.workunit.client.1.smithi134.stdout:8/772: sync 2024-04-22T23:23:56.264 INFO:tasks.workunit.client.1.smithi134.stdout:7/839: sync 2024-04-22T23:23:56.264 INFO:tasks.workunit.client.1.smithi134.stdout:5/884: sync 2024-04-22T23:23:56.264 INFO:tasks.workunit.client.1.smithi134.stdout:0/788: sync 2024-04-22T23:23:56.265 INFO:tasks.workunit.client.1.smithi134.stdout:7/840: readlink d4/d8/d58/l91 0 2024-04-22T23:23:56.271 INFO:tasks.workunit.client.1.smithi134.stdout:7/841: unlink d4/d8/d2a/d42/d7d/fa4 0 2024-04-22T23:23:56.272 INFO:tasks.workunit.client.1.smithi134.stdout:0/789: dwrite d1/d5/d58/d22/fbb [4194304,4194304] 0 2024-04-22T23:23:56.273 INFO:tasks.workunit.client.1.smithi134.stdout:7/842: dread - d4/d8/d3b/ffb zero size 2024-04-22T23:23:56.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.277+0000 7f420e89a700 1 -- 172.21.15.67:0/3496511544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4208101800 msgr2=0x7f4208103bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:56.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.277+0000 7f420e89a700 1 --2- 172.21.15.67:0/3496511544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4208101800 0x7f4208103bf0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f41f8009910 tx=0x7f41f8009c20 comp rx=0 tx=0).stop 2024-04-22T23:23:56.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.277+0000 7f420e89a700 1 -- 172.21.15.67:0/3496511544 shutdown_connections 2024-04-22T23:23:56.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.277+0000 7f420e89a700 1 --2- 172.21.15.67:0/3496511544 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4208104130 0x7f4208106520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:56.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.277+0000 7f420e89a700 1 --2- 172.21.15.67:0/3496511544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4208101800 0x7f4208103bf0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:56.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.277+0000 7f420e89a700 1 -- 172.21.15.67:0/3496511544 >> 172.21.15.67:0/3496511544 conn(0x7f42080fb190 msgr2=0x7f42080fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:23:56.279 INFO:tasks.workunit.client.1.smithi134.stdout:8/773: getdents d7/dc/d2c/d4f/d6c/d86 0 2024-04-22T23:23:56.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.278+0000 7f420e89a700 1 -- 172.21.15.67:0/3496511544 shutdown_connections 2024-04-22T23:23:56.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.278+0000 7f420e89a700 1 -- 172.21.15.67:0/3496511544 wait complete. 2024-04-22T23:23:56.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.279+0000 7f420e89a700 1 Processor -- start 2024-04-22T23:23:56.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.279+0000 7f420e89a700 1 -- start start 2024-04-22T23:23:56.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.280+0000 7f420e89a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4208101800 0x7f42081964e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:23:56.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.280+0000 7f420e89a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4208104130 0x7f4208196a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:23:56.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.280+0000 7f420e89a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4208197040 con 0x7f4208101800 2024-04-22T23:23:56.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.280+0000 7f420e89a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4208197180 con 0x7f4208104130 2024-04-22T23:23:56.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.280+0000 7f4207fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4208101800 0x7f42081964e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:23:56.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.280+0000 7f4207fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4208101800 0x7f42081964e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53536/0 (socket says 172.21.15.67:53536) 2024-04-22T23:23:56.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.280+0000 7f4207fff700 1 -- 172.21.15.67:0/2490027793 learned_addr learned my addr 172.21.15.67:0/2490027793 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:23:56.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.281+0000 7f4207fff700 1 -- 172.21.15.67:0/2490027793 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4208104130 msgr2=0x7f4208196a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:56.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.281+0000 7f4207fff700 1 --2- 172.21.15.67:0/2490027793 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4208104130 0x7f4208196a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:56.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.281+0000 7f4207fff700 1 -- 172.21.15.67:0/2490027793 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f80095f0 con 0x7f4208101800 2024-04-22T23:23:56.283 INFO:tasks.workunit.client.1.smithi134.stdout:0/790: fdatasync d1/d5/d58/d22/fb3 0 2024-04-22T23:23:56.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.281+0000 7f4207fff700 1 --2- 172.21.15.67:0/2490027793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4208101800 0x7f42081964e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f41f8005b40 tx=0x7f41f800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:23:56.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.281+0000 7f42057fa700 1 -- 172.21.15.67:0/2490027793 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f41f801d070 con 0x7f4208101800 2024-04-22T23:23:56.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.282+0000 7f42057fa700 1 -- 172.21.15.67:0/2490027793 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f41f8022470 con 0x7f4208101800 2024-04-22T23:23:56.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.282+0000 7f420e89a700 1 -- 172.21.15.67:0/2490027793 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f420819bbd0 con 0x7f4208101800 2024-04-22T23:23:56.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.282+0000 7f42057fa700 1 -- 172.21.15.67:0/2490027793 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f41f800f690 con 0x7f4208101800 2024-04-22T23:23:56.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.282+0000 7f420e89a700 1 -- 172.21.15.67:0/2490027793 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f420819c0c0 con 0x7f4208101800 2024-04-22T23:23:56.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.283+0000 7f420e89a700 1 -- 172.21.15.67:0/2490027793 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4208190690 con 0x7f4208101800 2024-04-22T23:23:56.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.283+0000 7f42057fa700 1 -- 172.21.15.67:0/2490027793 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 20) v1 ==== 91455+0+0 (secure 0 0 0) 0x7f41f800f7f0 con 0x7f4208101800 2024-04-22T23:23:56.285 INFO:tasks.workunit.client.1.smithi134.stdout:5/885: dwrite d9/d11/d73/f8c [0,4194304] 0 2024-04-22T23:23:56.286 INFO:tasks.workunit.client.1.smithi134.stdout:7/843: stat d4/d8/d41/d76/dc1/d2f/d69/dab/de1/ff7 0 2024-04-22T23:23:56.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.284+0000 7f42057fa700 1 --2- 172.21.15.67:0/2490027793 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f41f006d710 0x7f41f006fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:23:56.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.284+0000 7f42057fa700 1 -- 172.21.15.67:0/2490027793 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f41f808e790 con 0x7f4208101800 2024-04-22T23:23:56.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.284+0000 7f42077fe700 1 --2- 172.21.15.67:0/2490027793 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f41f006d710 0x7f41f006fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:23:56.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.285+0000 7f42077fe700 1 --2- 172.21.15.67:0/2490027793 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f41f006d710 0x7f41f006fbd0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f41fc0097b0 tx=0x7f41fc006d20 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:23:56.287 INFO:tasks.workunit.client.1.smithi134.stdout:3/570: getdents dc/d2a/d62/db2/d2f/d36/d6d/d6f/dd2 0 2024-04-22T23:23:56.288 INFO:tasks.workunit.client.1.smithi134.stdout:7/844: chown d4/d8/d58/c108 1059637 1 2024-04-22T23:23:56.288 INFO:tasks.workunit.client.1.smithi134.stdout:5/886: chown d9/da2/de6/c102 734 1 2024-04-22T23:23:56.289 INFO:tasks.workunit.client.1.smithi134.stdout:5/887: write d9/dbb/d11b/fbe [3065851,65651] 0 2024-04-22T23:23:56.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.290+0000 7f42057fa700 1 -- 172.21.15.67:0/2490027793 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f41f805cc60 con 0x7f4208101800 2024-04-22T23:23:56.292 INFO:tasks.workunit.client.1.smithi134.stdout:0/791: rmdir d1/d5/d58/d6b 39 2024-04-22T23:23:56.300 INFO:tasks.workunit.client.1.smithi134.stdout:9/717: link d0/d7/d23/d26/d51/d88/db9/lbe d0/d7/d27/lf2 0 2024-04-22T23:23:56.301 INFO:tasks.workunit.client.1.smithi134.stdout:9/718: chown d0/d7/d23/d26/d51/d88/db9/ff1 10 1 2024-04-22T23:23:56.305 INFO:tasks.workunit.client.1.smithi134.stdout:3/571: write dc/f18 [2440733,81118] 0 2024-04-22T23:23:56.306 INFO:tasks.workunit.client.1.smithi134.stdout:7/845: write d4/d8/d3b/d83/f88 [265411,105137] 0 2024-04-22T23:23:56.307 INFO:tasks.workunit.client.1.smithi134.stdout:8/774: creat d7/db/d2d/d39/d58/ff3 x:0 0 0 2024-04-22T23:23:56.308 INFO:tasks.workunit.client.1.smithi134.stdout:8/775: write d7/dc/fa8 [5099800,88346] 0 2024-04-22T23:23:56.311 INFO:tasks.workunit.client.1.smithi134.stdout:9/719: chown d0/d2d/d94/fac 6516122 1 2024-04-22T23:23:56.313 INFO:tasks.workunit.client.1.smithi134.stdout:3/572: creat dc/d2a/d62/db2/d2f/d21/d42/d77/fd4 x:0 0 0 2024-04-22T23:23:56.325 INFO:tasks.workunit.client.1.smithi134.stdout:9/720: mkdir d0/d7/d12/de8/df3 0 2024-04-22T23:23:56.327 INFO:tasks.workunit.client.1.smithi134.stdout:8/776: write d7/dc/d2c/d37/f3f [784405,94587] 0 2024-04-22T23:23:56.328 INFO:tasks.workunit.client.1.smithi134.stdout:5/888: link d9/da2/lea d9/d11/d73/d95/d20/l12c 0 2024-04-22T23:23:56.331 INFO:tasks.workunit.client.1.smithi134.stdout:0/792: getdents d1/d5/dd/d68 0 2024-04-22T23:23:56.333 INFO:tasks.workunit.client.1.smithi134.stdout:7/846: symlink d4/d4a/d85/dd7/l12f 0 2024-04-22T23:23:56.333 INFO:tasks.workunit.client.1.smithi134.stdout:9/721: dread d0/d7/d23/f55 [0,4194304] 0 2024-04-22T23:23:56.333 INFO:tasks.workunit.client.1.smithi134.stdout:3/573: creat dc/d2a/d62/d86/dc7/dcf/fd5 x:0 0 0 2024-04-22T23:23:56.335 INFO:tasks.workunit.client.1.smithi134.stdout:7/847: stat d4/d8/d41/d76/dc1/d2f/d34/d54/fb4 0 2024-04-22T23:23:56.336 INFO:tasks.workunit.client.1.smithi134.stdout:7/848: readlink d4/dad/l11c 0 2024-04-22T23:23:56.338 INFO:tasks.workunit.client.1.smithi134.stdout:7/849: write d4/d8/d41/d76/dc1/d2f/d70/f74 [4488505,33739] 0 2024-04-22T23:23:56.340 INFO:tasks.workunit.client.1.smithi134.stdout:7/850: dread - d4/d8/d41/d76/dc1/d2f/d34/dc0/f107 zero size 2024-04-22T23:23:56.341 INFO:tasks.workunit.client.1.smithi134.stdout:7/851: chown d4/d8/d41 191517 1 2024-04-22T23:23:56.346 INFO:tasks.workunit.client.1.smithi134.stdout:9/722: rename d0/d2d/d82/da0/c65 to d0/d7/d12/de8/cf4 0 2024-04-22T23:23:56.349 INFO:tasks.workunit.client.1.smithi134.stdout:5/889: creat d9/d11/d73/d95/dae/de0/f12d x:0 0 0 2024-04-22T23:23:56.350 INFO:tasks.workunit.client.1.smithi134.stdout:5/890: write d9/fe [698785,50795] 0 2024-04-22T23:23:56.351 INFO:tasks.workunit.client.1.smithi134.stdout:5/891: fdatasync d9/dbb/d11b/d88/db7/ffa 0 2024-04-22T23:23:56.353 INFO:tasks.workunit.client.1.smithi134.stdout:9/723: dread d0/d7/f19 [0,4194304] 0 2024-04-22T23:23:56.354 INFO:tasks.workunit.client.1.smithi134.stdout:7/852: rmdir d4 39 2024-04-22T23:23:56.355 INFO:tasks.workunit.client.1.smithi134.stdout:9/724: chown d0/d7/d23/d26/d4d/c77 9481 1 2024-04-22T23:23:56.362 INFO:tasks.workunit.client.1.smithi134.stdout:3/574: dwrite dc/d2a/d62/fa6 [0,4194304] 0 2024-04-22T23:23:56.363 INFO:tasks.workunit.client.1.smithi134.stdout:5/892: write d9/dbb/d11b/d17/d2a/d49/fad [698833,6507] 0 2024-04-22T23:23:56.365 INFO:tasks.workunit.client.1.smithi134.stdout:8/777: dwrite d7/db/d2d/f5a [0,4194304] 0 2024-04-22T23:23:56.368 INFO:tasks.workunit.client.1.smithi134.stdout:3/575: stat dc/d2a/d62/db2/d2f/d21/d42/d77 0 2024-04-22T23:23:56.371 INFO:tasks.workunit.client.1.smithi134.stdout:0/793: dwrite d1/d77/fa6 [0,4194304] 0 2024-04-22T23:23:56.371 INFO:tasks.workunit.client.1.smithi134.stdout:7/853: dread d4/d8/d41/d76/dc1/d2f/d70/d86/ff0 [0,4194304] 0 2024-04-22T23:23:56.371 INFO:tasks.workunit.client.1.smithi134.stdout:9/725: unlink d0/d7/d23/d26/d4d/d5d/d8e/f9c 0 2024-04-22T23:23:56.375 INFO:tasks.workunit.client.1.smithi134.stdout:5/893: mknod d9/d11/d34/d4d/d92/c12e 0 2024-04-22T23:23:56.375 INFO:tasks.workunit.client.1.smithi134.stdout:0/794: read d1/f44 [654898,118485] 0 2024-04-22T23:23:56.384 INFO:tasks.workunit.client.1.smithi134.stdout:6/884: sync 2024-04-22T23:23:56.384 INFO:tasks.workunit.client.1.smithi134.stdout:9/726: rename d0/d7/d23/d26/dbc to d0/d2d/d82/df5 0 2024-04-22T23:23:56.387 INFO:tasks.workunit.client.1.smithi134.stdout:3/576: mkdir dc/d2a/d62/db2/d34/dd6 0 2024-04-22T23:23:56.388 INFO:tasks.workunit.client.1.smithi134.stdout:3/577: readlink dc/d7d/laa 0 2024-04-22T23:23:56.390 INFO:tasks.workunit.client.1.smithi134.stdout:5/894: symlink d9/d11/d34/d4d/dd4/l12f 0 2024-04-22T23:23:56.391 INFO:tasks.workunit.client.1.smithi134.stdout:0/795: creat d1/d2/df7/f105 x:0 0 0 2024-04-22T23:23:56.395 INFO:tasks.workunit.client.1.smithi134.stdout:9/727: fsync d0/f75 0 2024-04-22T23:23:56.396 INFO:tasks.workunit.client.1.smithi134.stdout:5/895: symlink d9/dbb/l130 0 2024-04-22T23:23:56.396 INFO:tasks.workunit.client.1.smithi134.stdout:5/896: stat d9/d1e/d90 0 2024-04-22T23:23:56.401 INFO:tasks.workunit.client.1.smithi134.stdout:8/778: getdents d7/dc/d2c/d4f/d6c/dd1 0 2024-04-22T23:23:56.401 INFO:tasks.workunit.client.1.smithi134.stdout:5/897: dwrite d9/d11/d34/d4d/faf [0,4194304] 0 2024-04-22T23:23:56.402 INFO:tasks.workunit.client.1.smithi134.stdout:8/779: chown d7/dc/d2c/d4f/lc6 0 1 2024-04-22T23:23:56.409 INFO:tasks.workunit.client.1.smithi134.stdout:7/854: dwrite d4/d8/d3b/f50 [0,4194304] 0 2024-04-22T23:23:56.416 INFO:tasks.workunit.client.1.smithi134.stdout:0/796: write d1/d5/d15/fb4 [816483,20565] 0 2024-04-22T23:23:56.417 INFO:tasks.workunit.client.1.smithi134.stdout:5/898: truncate d9/d1e/f32 4281377 0 2024-04-22T23:23:56.419 INFO:tasks.workunit.client.1.smithi134.stdout:9/728: creat d0/d2d/d4e/ff6 x:0 0 0 2024-04-22T23:23:56.419 INFO:tasks.workunit.client.1.smithi134.stdout:0/797: read d1/d2/f4 [3914443,48793] 0 2024-04-22T23:23:56.420 INFO:tasks.workunit.client.1.smithi134.stdout:6/885: link d0/d19/db/d35/dbc/fc9 d0/d19/de/d12/d5e/dd6/f124 0 2024-04-22T23:23:56.422 INFO:tasks.workunit.client.1.smithi134.stdout:3/578: getdents dc/d2a/d62/db2 0 2024-04-22T23:23:56.425 INFO:tasks.workunit.client.1.smithi134.stdout:8/780: dwrite d7/db/d42/d65/dd9/dda/de7/fef [0,4194304] 0 2024-04-22T23:23:56.425 INFO:tasks.workunit.client.1.smithi134.stdout:5/899: unlink d9/d11/d73/d95/dae/de0/cec 0 2024-04-22T23:23:56.427 INFO:tasks.workunit.client.1.smithi134.stdout:5/900: chown d9/dbb/l130 0 1 2024-04-22T23:23:56.428 INFO:tasks.workunit.client.1.smithi134.stdout:7/855: dread d4/d8/d2a/d42/fc3 [0,4194304] 0 2024-04-22T23:23:56.432 INFO:tasks.workunit.client.1.smithi134.stdout:9/729: dread d0/d7/d23/d26/d51/d4c/f71 [0,4194304] 0 2024-04-22T23:23:56.436 INFO:tasks.workunit.client.1.smithi134.stdout:2/704: sync 2024-04-22T23:23:56.438 INFO:tasks.workunit.client.1.smithi134.stdout:0/798: truncate d1/d5/d58/d22/f85 893023 0 2024-04-22T23:23:56.439 INFO:tasks.workunit.client.1.smithi134.stdout:7/856: dwrite d4/d8/d41/d76/dc1/d2f/d34/d62/d29/ff1 [0,4194304] 0 2024-04-22T23:23:56.441 INFO:tasks.workunit.client.1.smithi134.stdout:7/857: fsync d4/d8/d2a/f2c 0 2024-04-22T23:23:56.443 INFO:tasks.workunit.client.1.smithi134.stdout:9/730: mkdir d0/d7/d12/de8/df7 0 2024-04-22T23:23:56.450 INFO:tasks.workunit.client.1.smithi134.stdout:8/781: dread d7/db/f1f [0,4194304] 0 2024-04-22T23:23:56.451 INFO:tasks.workunit.client.1.smithi134.stdout:6/886: creat d0/d6f/f125 x:0 0 0 2024-04-22T23:23:56.451 INFO:tasks.workunit.client.1.smithi134.stdout:0/799: fdatasync d1/d2/d1d/f69 0 2024-04-22T23:23:56.455 INFO:tasks.workunit.client.1.smithi134.stdout:3/579: dwrite dc/f52 [0,4194304] 0 2024-04-22T23:23:56.457 INFO:tasks.workunit.client.1.smithi134.stdout:2/705: dread d2/d15/d24/f2a [0,4194304] 0 2024-04-22T23:23:56.462 INFO:tasks.workunit.client.1.smithi134.stdout:5/901: dread d9/fe [0,4194304] 0 2024-04-22T23:23:56.462 INFO:tasks.workunit.client.1.smithi134.stdout:5/902: stat d9/d11/d34/ca7 0 2024-04-22T23:23:56.463 INFO:tasks.workunit.client.1.smithi134.stdout:9/731: rmdir d0/d7/d23/d26/d4d/d5d 39 2024-04-22T23:23:56.470 INFO:tasks.workunit.client.1.smithi134.stdout:8/782: rename d7/db/d42/d65/d46/d9a/da3/f92 to d7/dc/d2c/d37/d47/ff4 0 2024-04-22T23:23:56.473 INFO:tasks.workunit.client.1.smithi134.stdout:7/858: write d4/f31 [1553645,84107] 0 2024-04-22T23:23:56.473 INFO:tasks.workunit.client.1.smithi134.stdout:3/580: mknod dc/d2a/d62/db2/d34/d72/cd7 0 2024-04-22T23:23:56.474 INFO:tasks.workunit.client.1.smithi134.stdout:0/800: dread d1/d5/dd/d41/fc3 [0,4194304] 0 2024-04-22T23:23:56.480 INFO:tasks.workunit.client.1.smithi134.stdout:9/732: write d0/d2d/d82/fb7 [290031,66766] 0 2024-04-22T23:23:56.483 INFO:tasks.workunit.client.1.smithi134.stdout:5/903: mknod d9/d11/d73/d95/d20/d117/c131 0 2024-04-22T23:23:56.484 INFO:tasks.workunit.client.1.smithi134.stdout:7/859: creat d4/d8/d41/d76/dc1/d2f/d69/dab/f130 x:0 0 0 2024-04-22T23:23:56.486 INFO:tasks.workunit.client.1.smithi134.stdout:2/706: dwrite d2/d42/f46 [0,4194304] 0 2024-04-22T23:23:56.488 INFO:tasks.workunit.client.1.smithi134.stdout:6/887: dread d0/d19/db/d33/d86/f11f [0,4194304] 0 2024-04-22T23:23:56.490 INFO:tasks.workunit.client.1.smithi134.stdout:6/888: chown d0/f55 15 1 2024-04-22T23:23:56.491 INFO:tasks.workunit.client.1.smithi134.stdout:8/783: symlink d7/db/d2d/lf5 0 2024-04-22T23:23:56.492 INFO:tasks.workunit.client.1.smithi134.stdout:6/889: read d0/d19/de/d12/d29/d74/f89 [209359,58502] 0 2024-04-22T23:23:56.493 INFO:tasks.workunit.client.1.smithi134.stdout:3/581: dwrite dc/d2a/d62/f65 [0,4194304] 0 2024-04-22T23:23:56.493 INFO:tasks.workunit.client.1.smithi134.stdout:9/733: fsync d0/d7/f9 0 2024-04-22T23:23:56.496 INFO:tasks.workunit.client.1.smithi134.stdout:3/582: stat dc/d2a/d62/db2/d34/d72/fa1 0 2024-04-22T23:23:56.497 INFO:tasks.workunit.client.1.smithi134.stdout:5/904: symlink d9/dbb/l132 0 2024-04-22T23:23:56.500 INFO:tasks.workunit.client.1.smithi134.stdout:2/707: creat d2/dd/df/d1c/ff0 x:0 0 0 2024-04-22T23:23:56.502 INFO:tasks.workunit.client.1.smithi134.stdout:7/860: unlink d4/d8/d41/d76/dc1/d2f/da6/dcf/fd4 0 2024-04-22T23:23:56.508 INFO:tasks.workunit.client.1.smithi134.stdout:0/801: dwrite d1/d5/d58/d22/f62 [0,4194304] 0 2024-04-22T23:23:56.513 INFO:tasks.workunit.client.1.smithi134.stdout:5/905: dwrite d9/dbb/d11b/df1/f101 [0,4194304] 0 2024-04-22T23:23:56.516 INFO:tasks.workunit.client.1.smithi134.stdout:2/708: write d2/d42/d54/d33/d91/fe7 [135715,118848] 0 2024-04-22T23:23:56.517 INFO:tasks.workunit.client.1.smithi134.stdout:2/709: stat d2/d42 0 2024-04-22T23:23:56.519 INFO:tasks.workunit.client.1.smithi134.stdout:0/802: symlink d1/d5/d58/d94/l106 0 2024-04-22T23:23:56.525 INFO:tasks.workunit.client.1.smithi134.stdout:9/734: dread d0/d7/d12/f30 [0,4194304] 0 2024-04-22T23:23:56.525 INFO:tasks.workunit.client.1.smithi134.stdout:3/583: mkdir dc/d2a/d62/d86/dc1/dd8 0 2024-04-22T23:23:56.529 INFO:tasks.workunit.client.1.smithi134.stdout:9/735: truncate d0/d7/d23/d26/d51/d88/db9/ff1 854619 0 2024-04-22T23:23:56.532 INFO:tasks.workunit.client.1.smithi134.stdout:2/710: creat d2/dd/d4a/d73/ff1 x:0 0 0 2024-04-22T23:23:56.537 INFO:tasks.workunit.client.1.smithi134.stdout:9/736: dwrite d0/d2d/d82/f83 [0,4194304] 0 2024-04-22T23:23:56.538 INFO:tasks.workunit.client.1.smithi134.stdout:6/890: dwrite d0/d4f/fcc [0,4194304] 0 2024-04-22T23:23:56.547 INFO:tasks.workunit.client.1.smithi134.stdout:7/861: rename d4/d8/d41/d76/dc1/d2f/d34/d62/f56 to d4/d8/d41/d76/dc1/d2f/d69/dab/de1/f131 0 2024-04-22T23:23:56.549 INFO:tasks.workunit.client.1.smithi134.stdout:5/906: write d9/d11/d34/d85/f107 [258322,6416] 0 2024-04-22T23:23:56.558 INFO:tasks.workunit.client.1.smithi134.stdout:9/737: mkdir d0/d2d/dc1/df8 0 2024-04-22T23:23:56.559 INFO:tasks.workunit.client.1.smithi134.stdout:9/738: fdatasync d0/d7/d23/d1b/de2/fec 0 2024-04-22T23:23:56.559 INFO:tasks.workunit.client.1.smithi134.stdout:6/891: mknod d0/d19/de/d12/d29/d74/da0/c126 0 2024-04-22T23:23:56.560 INFO:tasks.workunit.client.1.smithi134.stdout:8/784: link d7/db/ld d7/db/d42/d65/d46/d9a/da3/d88/deb/lf6 0 2024-04-22T23:23:56.560 INFO:tasks.workunit.client.1.smithi134.stdout:8/785: dread - d7/db/d2d/d39/d58/fe5 zero size 2024-04-22T23:23:56.564 INFO:tasks.workunit.client.1.smithi134.stdout:0/803: rename d1/d5/f50 to d1/d5/d58/d6b/db9/def/f107 0 2024-04-22T23:23:56.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.566+0000 7f420e89a700 1 -- 172.21.15.67:0/2490027793 --> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f42080611d0 con 0x7f41f006d710 2024-04-22T23:23:56.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.567+0000 7f42057fa700 1 -- 172.21.15.67:0/2490027793 <== mgr.14215 v2:172.21.15.67:6800/2589747850 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7f42080611d0 con 0x7f41f006d710 2024-04-22T23:23:56.568 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:23:56.568 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:23:56.568 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:23:56.568 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-22T23:23:56.569 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [], 2024-04-22T23:23:56.569 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "1/2 daemons upgraded", 2024-04-22T23:23:56.569 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mgr daemons", 2024-04-22T23:23:56.569 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:23:56.569 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:23:56.569 INFO:tasks.workunit.client.1.smithi134.stdout:8/786: dwrite d7/fc5 [0,4194304] 0 2024-04-22T23:23:56.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.569+0000 7f420e89a700 1 -- 172.21.15.67:0/2490027793 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f41f006d710 msgr2=0x7f41f006fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:56.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.569+0000 7f420e89a700 1 --2- 172.21.15.67:0/2490027793 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f41f006d710 0x7f41f006fbd0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f41fc0097b0 tx=0x7f41fc006d20 comp rx=0 tx=0).stop 2024-04-22T23:23:56.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.569+0000 7f420e89a700 1 -- 172.21.15.67:0/2490027793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4208101800 msgr2=0x7f42081964e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:23:56.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.569+0000 7f420e89a700 1 --2- 172.21.15.67:0/2490027793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4208101800 0x7f42081964e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f41f8005b40 tx=0x7f41f800be20 comp rx=0 tx=0).stop 2024-04-22T23:23:56.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.569+0000 7f420e89a700 1 -- 172.21.15.67:0/2490027793 shutdown_connections 2024-04-22T23:23:56.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.569+0000 7f420e89a700 1 --2- 172.21.15.67:0/2490027793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4208101800 0x7f42081964e0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:56.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.569+0000 7f420e89a700 1 --2- 172.21.15.67:0/2490027793 >> [v2:172.21.15.67:6800/2589747850,v1:172.21.15.67:6801/2589747850] conn(0x7f41f006d710 0x7f41f006fbd0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:56.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.569+0000 7f420e89a700 1 --2- 172.21.15.67:0/2490027793 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4208104130 0x7f4208196a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:23:56.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.569+0000 7f420e89a700 1 -- 172.21.15.67:0/2490027793 >> 172.21.15.67:0/2490027793 conn(0x7f42080fb190 msgr2=0x7f42080fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:23:56.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.569+0000 7f420e89a700 1 -- 172.21.15.67:0/2490027793 shutdown_connections 2024-04-22T23:23:56.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:23:56.569+0000 7f420e89a700 1 -- 172.21.15.67:0/2490027793 wait complete. 2024-04-22T23:23:56.573 INFO:tasks.workunit.client.1.smithi134.stdout:9/739: mkdir d0/d2d/d82/df9 0 2024-04-22T23:23:56.573 INFO:tasks.workunit.client.1.smithi134.stdout:9/740: read - d0/d7/d12/de8/fee zero size 2024-04-22T23:23:56.575 INFO:tasks.workunit.client.1.smithi134.stdout:9/741: write d0/d2d/d3d/fda [374775,121592] 0 2024-04-22T23:23:56.576 INFO:tasks.workunit.client.1.smithi134.stdout:3/584: write dc/d2a/d62/db2/d2f/d21/d42/d77/fb6 [994361,50174] 0 2024-04-22T23:23:56.580 INFO:tasks.workunit.client.1.smithi134.stdout:0/804: stat d1/d5/ca5 0 2024-04-22T23:23:56.581 INFO:tasks.workunit.client.1.smithi134.stdout:0/805: dread - d1/d5/dd/d41/fdb zero size 2024-04-22T23:23:56.584 INFO:tasks.workunit.client.1.smithi134.stdout:2/711: write d2/dd/df/f85 [692830,67954] 0 2024-04-22T23:23:56.585 INFO:tasks.workunit.client.1.smithi134.stdout:8/787: mknod d7/db/d42/d65/dd9/dda/de7/ddd/cf7 0 2024-04-22T23:23:56.587 INFO:tasks.workunit.client.1.smithi134.stdout:3/585: mkdir dc/d2a/d62/db2/d2f/d36/d6d/d6f/dd9 0 2024-04-22T23:23:56.588 INFO:tasks.workunit.client.1.smithi134.stdout:0/806: dwrite d1/d5/ff4 [0,4194304] 0 2024-04-22T23:23:56.591 INFO:tasks.workunit.client.1.smithi134.stdout:2/712: unlink d2/d15/d24/d1d/f44 0 2024-04-22T23:23:56.593 INFO:tasks.workunit.client.1.smithi134.stdout:7/862: rename d4/d8/d41/d76/dc1/d2f/d69/d8a/d105/c118 to d4/d8/d41/d76/dc1/d2f/d34/d62/d84/c132 0 2024-04-22T23:23:56.595 INFO:tasks.workunit.client.1.smithi134.stdout:3/586: fdatasync dc/d2a/d62/db2/d34/d6b/f7c 0 2024-04-22T23:23:56.598 INFO:tasks.workunit.client.1.smithi134.stdout:2/713: dwrite d2/dd/d4a/f52 [0,4194304] 0 2024-04-22T23:23:56.600 INFO:tasks.workunit.client.1.smithi134.stdout:9/742: rename d0/d7/d12/f6a to d0/d7/d23/d26/d51/d88/ffa 0 2024-04-22T23:23:56.600 INFO:tasks.workunit.client.1.smithi134.stdout:2/714: chown d2/d42/d54/d33/d6e/fe1 5288005 1 2024-04-22T23:23:56.600 INFO:tasks.workunit.client.1.smithi134.stdout:8/788: rename d7/db/d42 to d7/db/d42/df8 22 2024-04-22T23:23:56.603 INFO:tasks.workunit.client.1.smithi134.stdout:7/863: read d4/d8/d41/d76/dc1/faa [175066,119347] 0 2024-04-22T23:23:56.604 INFO:tasks.workunit.client.1.smithi134.stdout:8/789: write d7/db/d42/d65/fc9 [4034641,118453] 0 2024-04-22T23:23:56.607 INFO:tasks.workunit.client.1.smithi134.stdout:8/790: write d7/db/d2d/d39/d58/fe5 [280682,54405] 0 2024-04-22T23:23:56.607 INFO:tasks.workunit.client.1.smithi134.stdout:7/864: read d4/d8/d41/d76/dc1/d2f/d69/dab/de1/ff7 [44067,15181] 0 2024-04-22T23:23:56.609 INFO:tasks.workunit.client.1.smithi134.stdout:8/791: write d7/db/d2d/d39/d58/ff3 [567274,7519] 0 2024-04-22T23:23:56.611 INFO:tasks.workunit.client.1.smithi134.stdout:3/587: rename dc/dd/lc4 to dc/d2a/d62/d86/dc7/lda 0 2024-04-22T23:23:56.611 INFO:tasks.workunit.client.1.smithi134.stdout:3/588: chown dc/d2a/d62/cb5 319438722 1 2024-04-22T23:23:56.616 INFO:tasks.workunit.client.1.smithi134.stdout:0/807: link d1/d2/f56 d1/d5/d58/d94/f108 0 2024-04-22T23:23:56.618 INFO:tasks.workunit.client.1.smithi134.stdout:0/808: fdatasync d1/d5/d15/f104 0 2024-04-22T23:23:56.618 INFO:tasks.workunit.client.1.smithi134.stdout:6/892: dwrite d0/f1a [0,4194304] 0 2024-04-22T23:23:56.619 INFO:tasks.workunit.client.1.smithi134.stdout:0/809: stat d1/d5/d58/d22/dd0/cf5 0 2024-04-22T23:23:56.622 INFO:tasks.workunit.client.1.smithi134.stdout:2/715: mknod d2/dd/df/d9c/cf2 0 2024-04-22T23:23:56.627 INFO:tasks.workunit.client.1.smithi134.stdout:3/589: mknod dc/d2a/d62/db2/d2f/d21/d42/cdb 0 2024-04-22T23:23:56.628 INFO:tasks.workunit.client.1.smithi134.stdout:0/810: stat d1/d5/dd/d41/fb7 0 2024-04-22T23:23:56.632 INFO:tasks.workunit.client.1.smithi134.stdout:2/716: rmdir d2/d15/d24/d1d/dba 39 2024-04-22T23:23:56.633 INFO:tasks.workunit.client.1.smithi134.stdout:5/907: dwrite d9/d1e/f32 [0,4194304] 0 2024-04-22T23:23:56.634 INFO:tasks.workunit.client.1.smithi134.stdout:5/908: stat d9/dbb/d11b/fbe 0 2024-04-22T23:23:56.644 INFO:tasks.workunit.client.1.smithi134.stdout:6/893: link d0/d19/de/d1f/d94/fad d0/d19/de/d2f/f127 0 2024-04-22T23:23:56.647 INFO:tasks.workunit.client.1.smithi134.stdout:8/792: dwrite d7/dc/d2c/d4f/d6c/d6e/f6f [0,4194304] 0 2024-04-22T23:23:56.655 INFO:tasks.workunit.client.1.smithi134.stdout:7/865: rename d4/d8/d41/d76/dc1/d2f/d69/f78 to d4/d8/f133 0 2024-04-22T23:23:56.658 INFO:tasks.workunit.client.1.smithi134.stdout:8/793: creat d7/dc/d2c/d4f/d6c/dd1/ff9 x:0 0 0 2024-04-22T23:23:56.665 INFO:tasks.workunit.client.1.smithi134.stdout:8/794: unlink d7/db/d2d/fa7 0 2024-04-22T23:23:56.666 INFO:tasks.workunit.client.1.smithi134.stdout:8/795: stat d7/db/d2d/d39/d58/ff3 0 2024-04-22T23:23:56.667 INFO:tasks.workunit.client.1.smithi134.stdout:8/796: dread - d7/db/d42/d65/d46/fdf zero size 2024-04-22T23:23:56.670 INFO:tasks.workunit.client.1.smithi134.stdout:9/743: write d0/d2d/f9f [2233431,103439] 0 2024-04-22T23:23:56.671 INFO:tasks.workunit.client.1.smithi134.stdout:9/744: chown d0/d7/d23/d1b/d5a/l81 81 1 2024-04-22T23:23:56.675 INFO:tasks.workunit.client.1.smithi134.stdout:7/866: creat d4/f134 x:0 0 0 2024-04-22T23:23:56.682 INFO:tasks.workunit.client.1.smithi134.stdout:7/867: dwrite d4/fac [0,4194304] 0 2024-04-22T23:23:56.682 INFO:tasks.workunit.client.1.smithi134.stdout:0/811: rename d1/d2/d1d/d72/c7c to d1/d5/d58/c109 0 2024-04-22T23:23:56.686 INFO:tasks.workunit.client.1.smithi134.stdout:8/797: mkdir d7/dc/d19/dfa 0 2024-04-22T23:23:56.691 INFO:tasks.workunit.client.1.smithi134.stdout:0/812: creat d1/d5/d58/d6b/db9/d75/f10a x:0 0 0 2024-04-22T23:23:56.693 INFO:tasks.workunit.client.1.smithi134.stdout:7/868: dwrite d4/d8/d2a/f11b [0,4194304] 0 2024-04-22T23:23:56.694 INFO:tasks.workunit.client.1.smithi134.stdout:8/798: write d7/dc/d2c/d4f/d6c/dd1/fdc [325876,4234] 0 2024-04-22T23:23:56.701 INFO:tasks.workunit.client.1.smithi134.stdout:0/813: creat d1/d5/d58/d22/dd0/ddf/f10b x:0 0 0 2024-04-22T23:23:56.704 INFO:tasks.workunit.client.1.smithi134.stdout:0/814: fdatasync d1/d2/d1d/fc2 0 2024-04-22T23:23:56.706 INFO:tasks.workunit.client.1.smithi134.stdout:9/745: link d0/d2d/c66 d0/d7/d23/d26/d4d/d5d/d8e/cfb 0 2024-04-22T23:23:56.706 INFO:tasks.workunit.client.1.smithi134.stdout:3/590: dwrite dc/dd/f4f [0,4194304] 0 2024-04-22T23:23:56.710 INFO:tasks.workunit.client.1.smithi134.stdout:0/815: write d1/d2/d1d/f9a [4178612,45972] 0 2024-04-22T23:23:56.711 INFO:tasks.workunit.client.1.smithi134.stdout:3/591: chown dc/dd/c6c 1218131 1 2024-04-22T23:23:56.713 INFO:tasks.workunit.client.1.smithi134.stdout:7/869: mkdir d4/dbd/d135 0 2024-04-22T23:23:56.713 INFO:tasks.workunit.client.1.smithi134.stdout:5/909: dwrite d9/dbb/d11b/f7b [0,4194304] 0 2024-04-22T23:23:56.716 INFO:tasks.workunit.client.1.smithi134.stdout:6/894: dwrite d0/d19/de/d12/d29/d74/fa1 [0,4194304] 0 2024-04-22T23:23:56.718 INFO:tasks.workunit.client.1.smithi134.stdout:3/592: write f9 [8715698,106151] 0 2024-04-22T23:23:56.725 INFO:tasks.workunit.client.1.smithi134.stdout:5/910: dwrite d9/dbb/d11b/d52/f6b [0,4194304] 0 2024-04-22T23:23:56.725 INFO:tasks.workunit.client.1.smithi134.stdout:7/870: rmdir d4/d8/d101 39 2024-04-22T23:23:56.731 INFO:tasks.workunit.client.1.smithi134.stdout:0/816: fsync d1/d5/d58/f46 0 2024-04-22T23:23:56.732 INFO:tasks.workunit.client.1.smithi134.stdout:8/799: getdents d7/db/d42/d65/dd9 0 2024-04-22T23:23:56.733 INFO:tasks.workunit.client.1.smithi134.stdout:8/800: stat d7/dc/d2c/c43 0 2024-04-22T23:23:56.734 INFO:tasks.workunit.client.1.smithi134.stdout:0/817: write d1/d5/d58/d22/dd0/ddf/fda [3553196,17309] 0 2024-04-22T23:23:56.735 INFO:tasks.workunit.client.1.smithi134.stdout:3/593: symlink dc/d2a/d62/db2/d2f/d61/d6a/ldc 0 2024-04-22T23:23:56.744 INFO:tasks.workunit.client.1.smithi134.stdout:5/911: getdents d9/dbb/d11b/d88/de4 0 2024-04-22T23:23:56.748 INFO:tasks.workunit.client.1.smithi134.stdout:3/594: creat dc/d2a/d62/db2/d34/fdd x:0 0 0 2024-04-22T23:23:56.749 INFO:tasks.workunit.client.1.smithi134.stdout:5/912: truncate d9/fa 137042 0 2024-04-22T23:23:56.750 INFO:tasks.workunit.client.1.smithi134.stdout:8/801: link d7/db/d42/d65/dd9/dda/de7/ddd/cf7 d7/db/d42/d65/dd9/dda/de7/ddd/dee/cfb 0 2024-04-22T23:23:56.751 INFO:tasks.workunit.client.1.smithi134.stdout:3/595: dread - dc/d2a/d62/db2/d34/f9f zero size 2024-04-22T23:23:56.753 INFO:tasks.workunit.client.1.smithi134.stdout:3/596: write dc/d2a/d62/db2/d2f/f68 [726330,38286] 0 2024-04-22T23:23:56.754 INFO:tasks.workunit.client.1.smithi134.stdout:5/913: mknod d9/dbb/d11b/d88/de4/c133 0 2024-04-22T23:23:56.757 INFO:tasks.workunit.client.1.smithi134.stdout:3/597: write dc/d2a/d62/db2/d2f/d21/d42/f82 [280156,1875] 0 2024-04-22T23:23:56.759 INFO:tasks.workunit.client.1.smithi134.stdout:3/598: stat dc/d2a/d30/ca3 0 2024-04-22T23:23:56.760 INFO:tasks.workunit.client.1.smithi134.stdout:0/818: dread d1/d5/dd/f7a [0,4194304] 0 2024-04-22T23:23:56.761 INFO:tasks.workunit.client.1.smithi134.stdout:8/802: rename d7/db/d42/d65/d46/d9a/da3/d88/ce6 to d7/db/d42/d65/cfc 0 2024-04-22T23:23:56.762 INFO:tasks.workunit.client.1.smithi134.stdout:3/599: readlink dc/d2a/d62/db2/d2f/d21/d42/d77/la4 0 2024-04-22T23:23:56.766 INFO:tasks.workunit.client.1.smithi134.stdout:7/871: dwrite d4/d8/d41/f43 [4194304,4194304] 0 2024-04-22T23:23:56.768 INFO:tasks.workunit.client.1.smithi134.stdout:0/819: read d1/f9 [4672376,27274] 0 2024-04-22T23:23:56.769 INFO:tasks.workunit.client.1.smithi134.stdout:5/914: link d9/dbb/d11b/d88/db7/d129/c12b d9/dbb/d11b/d17/d2a/d49/dc2/c134 0 2024-04-22T23:23:56.770 INFO:tasks.workunit.client.1.smithi134.stdout:5/915: read d9/d1e/f12a [3890624,39626] 0 2024-04-22T23:23:56.772 INFO:tasks.workunit.client.1.smithi134.stdout:3/600: mkdir dc/d2a/d62/db2/d34/d6b/dae/dc3/dde 0 2024-04-22T23:23:56.780 INFO:tasks.workunit.client.1.smithi134.stdout:5/916: stat d9/d11/ca0 0 2024-04-22T23:23:56.781 INFO:tasks.workunit.client.1.smithi134.stdout:3/601: unlink dc/dd/c25 0 2024-04-22T23:23:56.783 INFO:tasks.workunit.client.1.smithi134.stdout:7/872: mkdir d4/d8/d41/d76/dc1/d2f/d34/d62/d136 0 2024-04-22T23:23:56.785 INFO:tasks.workunit.client.1.smithi134.stdout:8/803: getdents d7/db/d42/d65/d46/d9a/da3 0 2024-04-22T23:23:56.785 INFO:tasks.workunit.client.1.smithi134.stdout:7/873: write d4/d8/d41/d76/dc1/d2f/d70/f127 [690180,61134] 0 2024-04-22T23:23:56.788 INFO:tasks.workunit.client.1.smithi134.stdout:0/820: creat d1/d5/dd/f10c x:0 0 0 2024-04-22T23:23:56.790 INFO:tasks.workunit.client.1.smithi134.stdout:0/821: write d1/d5/da2/fde [788494,42760] 0 2024-04-22T23:23:56.794 INFO:tasks.workunit.client.1.smithi134.stdout:5/917: symlink d9/d11/d73/d95/d20/l135 0 2024-04-22T23:23:56.802 INFO:tasks.workunit.client.1.smithi134.stdout:7/874: symlink d4/d8/d41/d76/dc1/d2f/d34/d62/d29/dc5/dd2/l137 0 2024-04-22T23:23:56.807 INFO:tasks.workunit.client.1.smithi134.stdout:5/918: symlink d9/d11/d34/l136 0 2024-04-22T23:23:56.809 INFO:tasks.workunit.client.1.smithi134.stdout:7/875: unlink d4/d8/d2a/d42/d75/feb 0 2024-04-22T23:23:56.814 INFO:tasks.workunit.client.1.smithi134.stdout:8/804: truncate d7/dc/f12 342056 0 2024-04-22T23:23:56.818 INFO:tasks.workunit.client.1.smithi134.stdout:0/822: write d1/d5/dd/d68/f97 [265402,10510] 0 2024-04-22T23:23:56.825 INFO:tasks.workunit.client.1.smithi134.stdout:8/805: symlink d7/dc/d2c/d4f/d6c/dd1/lfd 0 2024-04-22T23:23:56.828 INFO:tasks.workunit.client.1.smithi134.stdout:0/823: creat d1/d5/d58/d6b/db9/d83/f10d x:0 0 0 2024-04-22T23:23:56.830 INFO:tasks.workunit.client.1.smithi134.stdout:5/919: dwrite d9/d11/f1b [0,4194304] 0 2024-04-22T23:23:56.831 INFO:tasks.workunit.client.1.smithi134.stdout:0/824: fdatasync d1/d2/d1d/fc2 0 2024-04-22T23:23:56.831 INFO:tasks.workunit.client.1.smithi134.stdout:7/876: dread d4/d8/d2a/f5f [0,4194304] 0 2024-04-22T23:23:56.833 INFO:tasks.workunit.client.1.smithi134.stdout:5/920: chown d9/d11/d34/d85/ld5 8 1 2024-04-22T23:23:56.841 INFO:tasks.workunit.client.1.smithi134.stdout:0/825: truncate d1/d5/dd/f40 2252021 0 2024-04-22T23:23:56.846 INFO:tasks.workunit.client.1.smithi134.stdout:8/806: link d7/db/d42/d65/dd9/dda/de7/f94 d7/db/d42/d65/dd9/dda/de7/ddd/dee/ffe 0 2024-04-22T23:23:56.846 INFO:tasks.workunit.client.1.smithi134.stdout:0/826: dread d1/d5/da2/fde [0,4194304] 0 2024-04-22T23:23:56.847 INFO:tasks.workunit.client.1.smithi134.stdout:0/827: stat d1/d5/dd/d41/l4d 0 2024-04-22T23:23:56.852 INFO:tasks.workunit.client.1.smithi134.stdout:8/807: symlink d7/lff 0 2024-04-22T23:23:56.853 INFO:tasks.workunit.client.1.smithi134.stdout:0/828: creat d1/d5/dd/d68/dc0/f10e x:0 0 0 2024-04-22T23:23:56.856 INFO:tasks.workunit.client.1.smithi134.stdout:7/877: rename d4/d8/d41/d76/dc1/d2f/d70/d86/l12c to d4/d8/d2a/d42/d79/l138 0 2024-04-22T23:23:56.858 INFO:tasks.workunit.client.1.smithi134.stdout:8/808: rmdir d7/db/d42/d65/dd9/dda/de4 39 2024-04-22T23:23:56.863 INFO:tasks.workunit.client.1.smithi134.stdout:7/878: mknod d4/d8/d41/d76/dc1/d2f/d69/df8/c139 0 2024-04-22T23:23:56.871 INFO:tasks.workunit.client.1.smithi134.stdout:7/879: symlink d4/d8/d41/d76/dc1/d2f/d34/d62/d136/l13a 0 2024-04-22T23:23:56.878 INFO:tasks.workunit.client.1.smithi134.stdout:7/880: rename d4/d8/d41/d76/dc1/d2f/d69/dab/d95/fcb to d4/dbd/f13b 0 2024-04-22T23:23:56.879 INFO:tasks.workunit.client.1.smithi134.stdout:0/829: write d1/d5/d58/d6b/db9/f64 [4074415,127056] 0 2024-04-22T23:23:56.882 INFO:tasks.workunit.client.1.smithi134.stdout:7/881: truncate d4/d8/d41/d76/dc1/d2f/d34/dc0/f107 169912 0 2024-04-22T23:23:56.894 INFO:tasks.workunit.client.1.smithi134.stdout:0/830: dwrite d1/d2/d1d/f9a [4194304,4194304] 0 2024-04-22T23:23:56.960 INFO:tasks.workunit.client.1.smithi134.stdout:7/882: fdatasync d4/d8/d41/d76/dc1/d2f/d34/d62/d29/dc5/fe0 0 2024-04-22T23:23:56.961 INFO:tasks.workunit.client.1.smithi134.stdout:0/831: dread - d1/d2/d1d/fc6 zero size 2024-04-22T23:23:56.964 INFO:tasks.workunit.client.1.smithi134.stdout:7/883: mkdir d4/dbd/d13c 0 2024-04-22T23:23:56.965 INFO:tasks.workunit.client.1.smithi134.stdout:7/884: readlink d4/d8/d2a/d42/d6e/lf2 0 2024-04-22T23:23:56.966 INFO:tasks.workunit.client.1.smithi134.stdout:7/885: dread - d4/d4a/f7e zero size 2024-04-22T23:23:57.062 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:56 smithi134 ceph-mon[34609]: from='client.14654 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:23:57.063 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:56 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/3748584440' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:23:57.063 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:56 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:23:57.063 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:56 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:23:57.063 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:56 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:23:57.063 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:56 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:23:57.063 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:56 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:23:57.063 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:56 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:23:57.063 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:56 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mgr fail", "who": "smithi067.whdcsr"}]: dispatch 2024-04-22T23:23:57.063 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:56 smithi134 ceph-mon[34609]: osdmap e47: 6 total, 6 up, 6 in 2024-04-22T23:23:57.063 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:56 smithi134 ceph-mon[34609]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "mgr fail", "who": "smithi067.whdcsr"}]': finished 2024-04-22T23:23:57.063 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:56 smithi134 ceph-mon[34609]: mgrmap e21: smithi134.ccxkcj(active, starting, since 0.00783447s) 2024-04-22T23:23:57.063 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:56 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-22T23:23:57.063 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:56 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-04-22T23:23:57.064 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:56 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.xyazmh"}]: dispatch 2024-04-22T23:23:57.064 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:56 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.aqlxpv"}]: dispatch 2024-04-22T23:23:57.064 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:56 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.owipad"}]: dispatch 2024-04-22T23:23:57.064 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:56 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.vuxbsj"}]: dispatch 2024-04-22T23:23:57.064 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:56 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "mgr metadata", "who": "smithi134.ccxkcj", "id": "smithi134.ccxkcj"}]: dispatch 2024-04-22T23:23:57.064 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:56 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T23:23:57.064 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:56 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T23:23:57.064 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:56 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T23:23:57.064 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:56 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T23:23:57.064 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:56 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T23:23:57.065 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:56 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T23:23:57.065 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:56 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T23:23:57.065 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:56 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T23:23:57.065 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:56 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T23:23:57.065 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:56 smithi134 ceph-mon[34609]: Manager daemon smithi134.ccxkcj is now available 2024-04-22T23:23:57.065 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:56 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:23:57.065 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:56 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:23:57.065 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:56 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi134.ccxkcj/mirror_snapshot_schedule"}]: dispatch 2024-04-22T23:23:57.065 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:56 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi134.ccxkcj/mirror_snapshot_schedule"}]: dispatch 2024-04-22T23:23:57.065 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:56 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi134.ccxkcj/trash_purge_schedule"}]: dispatch 2024-04-22T23:23:57.065 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:56 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi134.ccxkcj/trash_purge_schedule"}]: dispatch 2024-04-22T23:23:57.136 INFO:tasks.workunit.client.1.smithi134.stdout:0/832: mknod d1/d5/dd/d20/c10f 0 2024-04-22T23:23:57.138 INFO:tasks.workunit.client.1.smithi134.stdout:0/833: write d1/d5/d58/d6b/db9/d75/f10a [873623,78173] 0 2024-04-22T23:23:57.139 INFO:tasks.workunit.client.1.smithi134.stdout:0/834: dread - d1/d5/d58/d6b/db9/d83/f10d zero size 2024-04-22T23:23:57.150 INFO:tasks.workunit.client.1.smithi134.stdout:7/886: creat d4/d8/d41/d76/dc1/d2f/d34/d62/f13d x:0 0 0 2024-04-22T23:23:57.155 INFO:tasks.workunit.client.1.smithi134.stdout:7/887: symlink d4/d8/d2a/d42/d7d/l13e 0 2024-04-22T23:23:57.156 INFO:tasks.workunit.client.1.smithi134.stdout:7/888: fsync d4/dad/f10b 0 2024-04-22T23:23:57.158 INFO:tasks.workunit.client.1.smithi134.stdout:0/835: getdents d1/d5/dd/d20/d2c 0 2024-04-22T23:23:57.158 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:56 smithi067 ceph-mon[26453]: from='client.14654 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:23:57.159 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:56 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/3748584440' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:23:57.159 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:56 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:23:57.159 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:56 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:23:57.159 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:56 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:23:57.159 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:56 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:23:57.159 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:56 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' 2024-04-22T23:23:57.159 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:56 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:23:57.159 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:56 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mgr fail", "who": "smithi067.whdcsr"}]: dispatch 2024-04-22T23:23:57.159 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:56 smithi067 ceph-mon[26453]: osdmap e47: 6 total, 6 up, 6 in 2024-04-22T23:23:57.159 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:56 smithi067 ceph-mon[26453]: from='mgr.14215 172.21.15.67:0/541350712' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "mgr fail", "who": "smithi067.whdcsr"}]': finished 2024-04-22T23:23:57.159 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:56 smithi067 ceph-mon[26453]: mgrmap e21: smithi134.ccxkcj(active, starting, since 0.00783447s) 2024-04-22T23:23:57.159 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:56 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-22T23:23:57.159 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:56 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-04-22T23:23:57.159 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:56 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.xyazmh"}]: dispatch 2024-04-22T23:23:57.159 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:56 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.aqlxpv"}]: dispatch 2024-04-22T23:23:57.160 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:56 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.owipad"}]: dispatch 2024-04-22T23:23:57.160 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:56 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.vuxbsj"}]: dispatch 2024-04-22T23:23:57.160 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:56 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "mgr metadata", "who": "smithi134.ccxkcj", "id": "smithi134.ccxkcj"}]: dispatch 2024-04-22T23:23:57.160 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:56 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T23:23:57.160 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:56 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T23:23:57.160 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:56 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T23:23:57.160 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:56 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T23:23:57.160 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:56 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T23:23:57.160 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:56 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T23:23:57.160 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:56 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T23:23:57.160 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:56 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T23:23:57.160 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:56 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T23:23:57.160 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:56 smithi067 ceph-mon[26453]: Manager daemon smithi134.ccxkcj is now available 2024-04-22T23:23:57.160 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:56 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:23:57.161 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:56 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:23:57.161 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:56 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi134.ccxkcj/mirror_snapshot_schedule"}]: dispatch 2024-04-22T23:23:57.161 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:56 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi134.ccxkcj/mirror_snapshot_schedule"}]: dispatch 2024-04-22T23:23:57.161 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:56 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi134.ccxkcj/trash_purge_schedule"}]: dispatch 2024-04-22T23:23:57.161 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:56 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi134.ccxkcj/trash_purge_schedule"}]: dispatch 2024-04-22T23:23:57.163 INFO:tasks.workunit.client.1.smithi134.stdout:7/889: mknod d4/d8/d41/d76/dc1/d2f/d69/dab/c13f 0 2024-04-22T23:23:57.165 INFO:tasks.workunit.client.1.smithi134.stdout:0/836: dwrite d1/d5/dd/d68/f97 [0,4194304] 0 2024-04-22T23:23:57.167 INFO:tasks.workunit.client.1.smithi134.stdout:0/837: readlink d1/d5/d58/d22/l23 0 2024-04-22T23:23:57.175 INFO:tasks.workunit.client.1.smithi134.stdout:0/838: dwrite d1/d5/dd/d20/d2c/f4b [0,4194304] 0 2024-04-22T23:23:57.177 INFO:tasks.workunit.client.1.smithi134.stdout:7/890: dread d4/d8/d3b/f68 [0,4194304] 0 2024-04-22T23:23:57.184 INFO:tasks.workunit.client.1.smithi134.stdout:0/839: creat d1/d5/d58/da0/de5/f110 x:0 0 0 2024-04-22T23:23:57.187 INFO:tasks.workunit.client.1.smithi134.stdout:0/840: creat d1/d5/dd/d68/f111 x:0 0 0 2024-04-22T23:23:57.190 INFO:tasks.workunit.client.1.smithi134.stdout:0/841: truncate d1/db/ff6 842650 0 2024-04-22T23:23:57.194 INFO:tasks.workunit.client.1.smithi134.stdout:0/842: dread d1/d5/d58/d22/f62 [0,4194304] 0 2024-04-22T23:23:57.194 INFO:tasks.workunit.client.1.smithi134.stdout:0/843: stat d1/d5/fbe 0 2024-04-22T23:23:57.195 INFO:tasks.workunit.client.1.smithi134.stdout:0/844: dread - d1/d5/dd/d20/d2c/ff8 zero size 2024-04-22T23:23:57.196 INFO:tasks.workunit.client.1.smithi134.stdout:0/845: chown d1/d5/d58/d22/dd0/ddf/cd7 97189 1 2024-04-22T23:23:57.199 INFO:tasks.workunit.client.1.smithi134.stdout:0/846: symlink d1/d5/d58/l112 0 2024-04-22T23:23:57.201 INFO:tasks.workunit.client.1.smithi134.stdout:2/717: sync 2024-04-22T23:23:57.202 INFO:tasks.workunit.client.1.smithi134.stdout:0/847: creat d1/d5/dd/d68/dc0/f113 x:0 0 0 2024-04-22T23:23:57.206 INFO:tasks.workunit.client.1.smithi134.stdout:2/718: rename d2/d15/d24/d1d/d3d/f71 to d2/d15/d24/d3b/d8a/d99/ff3 0 2024-04-22T23:23:57.210 INFO:tasks.workunit.client.1.smithi134.stdout:2/719: rmdir d2/dd/df 39 2024-04-22T23:23:57.214 INFO:tasks.workunit.client.1.smithi134.stdout:2/720: creat d2/dd/df/d9c/daa/ff4 x:0 0 0 2024-04-22T23:23:57.216 INFO:tasks.workunit.client.1.smithi134.stdout:2/721: truncate d2/dd/df/d9c/daa/db0/f95 647552 0 2024-04-22T23:23:57.218 INFO:tasks.workunit.client.1.smithi134.stdout:0/848: dread d1/d5/d15/fe9 [0,4194304] 0 2024-04-22T23:23:57.220 INFO:tasks.workunit.client.1.smithi134.stdout:2/722: symlink d2/dd/dc9/lf5 0 2024-04-22T23:23:57.228 INFO:tasks.workunit.client.1.smithi134.stdout:0/849: truncate d1/d5/f5e 1207361 0 2024-04-22T23:23:57.230 INFO:tasks.workunit.client.1.smithi134.stdout:0/850: read - d1/d2/d1d/f91 zero size 2024-04-22T23:23:57.234 INFO:tasks.workunit.client.1.smithi134.stdout:2/723: dwrite d2/dd/df/d1c/f76 [0,4194304] 0 2024-04-22T23:23:57.244 INFO:tasks.workunit.client.1.smithi134.stdout:0/851: creat d1/d5/d58/f114 x:0 0 0 2024-04-22T23:23:57.253 INFO:tasks.workunit.client.1.smithi134.stdout:2/724: dwrite d2/d15/d24/d3b/f4f [0,4194304] 0 2024-04-22T23:23:57.256 INFO:tasks.workunit.client.1.smithi134.stdout:0/852: write d1/d2/d1d/f69 [644916,92185] 0 2024-04-22T23:23:57.259 INFO:tasks.workunit.client.1.smithi134.stdout:2/725: creat d2/dd/dc9/ff6 x:0 0 0 2024-04-22T23:23:57.260 INFO:tasks.workunit.client.1.smithi134.stdout:0/853: rmdir d1 39 2024-04-22T23:23:57.260 INFO:tasks.workunit.client.1.smithi134.stdout:2/726: write d2/d15/d24/f5d [9373162,57938] 0 2024-04-22T23:23:57.266 INFO:tasks.workunit.client.1.smithi134.stdout:2/727: rename d2/dd/df/d1c/fa1 to d2/d15/d24/d1d/d3d/d7b/ff7 0 2024-04-22T23:23:57.268 INFO:tasks.workunit.client.1.smithi134.stdout:0/854: creat d1/d2/df7/f115 x:0 0 0 2024-04-22T23:23:57.268 INFO:tasks.workunit.client.1.smithi134.stdout:2/728: fdatasync d2/d15/d24/d1d/d3d/fd1 0 2024-04-22T23:23:57.272 INFO:tasks.workunit.client.1.smithi134.stdout:2/729: mknod d2/dd/de4/cf8 0 2024-04-22T23:23:57.278 INFO:tasks.workunit.client.1.smithi134.stdout:2/730: mkdir d2/dd/df9 0 2024-04-22T23:23:57.286 INFO:tasks.workunit.client.1.smithi134.stdout:2/731: dread d2/d15/d24/d1d/d3d/f92 [0,4194304] 0 2024-04-22T23:23:57.287 INFO:tasks.workunit.client.1.smithi134.stdout:2/732: readlink d2/d15/d24/l80 0 2024-04-22T23:23:57.288 INFO:tasks.workunit.client.1.smithi134.stdout:2/733: dread - d2/d15/d24/d1d/d3d/fd1 zero size 2024-04-22T23:23:57.293 INFO:tasks.workunit.client.1.smithi134.stdout:0/855: dwrite d1/d5/d58/d22/fa7 [0,4194304] 0 2024-04-22T23:23:57.297 INFO:tasks.workunit.client.1.smithi134.stdout:0/856: write d1/d5/dd/d68/f97 [3520121,75953] 0 2024-04-22T23:23:57.298 INFO:tasks.workunit.client.1.smithi134.stdout:0/857: write d1/d2/d1d/fc2 [1763617,111078] 0 2024-04-22T23:23:57.306 INFO:tasks.workunit.client.1.smithi134.stdout:0/858: rename d1/d5/d15/l84 to d1/db/l116 0 2024-04-22T23:23:57.307 INFO:tasks.workunit.client.1.smithi134.stdout:0/859: dread - d1/d2/d1d/f91 zero size 2024-04-22T23:23:57.310 INFO:tasks.workunit.client.1.smithi134.stdout:0/860: mkdir d1/d2/d117 0 2024-04-22T23:23:57.312 INFO:tasks.workunit.client.1.smithi134.stdout:2/734: write d2/f14 [844499,103323] 0 2024-04-22T23:23:57.315 INFO:tasks.workunit.client.1.smithi134.stdout:2/735: truncate d2/dd/d68/fa9 250650 0 2024-04-22T23:23:57.316 INFO:tasks.workunit.client.1.smithi134.stdout:2/736: fdatasync d2/d15/d24/f1e 0 2024-04-22T23:23:57.317 INFO:tasks.workunit.client.1.smithi134.stdout:2/737: write d2/dd/df/d1c/f79 [1950825,91402] 0 2024-04-22T23:23:57.323 INFO:tasks.workunit.client.1.smithi134.stdout:2/738: rename d2/d42/d54/d33/c90 to d2/dd/de4/cfa 0 2024-04-22T23:23:57.327 INFO:tasks.workunit.client.1.smithi134.stdout:0/861: dwrite d1/d2/f54 [4194304,4194304] 0 2024-04-22T23:23:57.332 INFO:tasks.workunit.client.1.smithi134.stdout:0/862: unlink d1/d5/dd/d41/fc3 0 2024-04-22T23:23:57.333 INFO:tasks.workunit.client.1.smithi134.stdout:2/739: dwrite d2/d15/d24/d1d/d3d/fd1 [0,4194304] 0 2024-04-22T23:23:57.343 INFO:tasks.workunit.client.1.smithi134.stdout:2/740: mkdir d2/d15/d24/d3b/d8a/d99/dfb 0 2024-04-22T23:23:57.348 INFO:tasks.workunit.client.1.smithi134.stdout:2/741: creat d2/dd/df/dc2/ffc x:0 0 0 2024-04-22T23:23:57.349 INFO:tasks.workunit.client.1.smithi134.stdout:0/863: dwrite d1/d5/d58/d6b/db9/def/f107 [4194304,4194304] 0 2024-04-22T23:23:57.357 INFO:tasks.workunit.client.1.smithi134.stdout:0/864: rename d1/d5/dd/d41/fc8 to d1/d2/f118 0 2024-04-22T23:23:57.373 INFO:tasks.workunit.client.1.smithi134.stdout:2/742: dwrite d2/dd/df/d9c/daa/db0/f9b [0,4194304] 0 2024-04-22T23:23:57.377 INFO:tasks.workunit.client.1.smithi134.stdout:2/743: mkdir d2/d42/d54/dfd 0 2024-04-22T23:23:57.384 INFO:tasks.workunit.client.1.smithi134.stdout:2/744: chown d2/d42/d54/d33/d6e/l82 7270195 1 2024-04-22T23:23:57.386 INFO:tasks.workunit.client.1.smithi134.stdout:0/865: write d1/d5/d58/d94/f108 [1925596,11349] 0 2024-04-22T23:23:57.388 INFO:tasks.workunit.client.1.smithi134.stdout:2/745: mkdir d2/d15/d24/d1d/d3d/dfe 0 2024-04-22T23:23:57.395 INFO:tasks.workunit.client.1.smithi134.stdout:2/746: dread d2/d15/d24/d1d/d3d/fd1 [0,4194304] 0 2024-04-22T23:23:57.399 INFO:tasks.workunit.client.1.smithi134.stdout:2/747: creat d2/dd/df/d9c/daa/fff x:0 0 0 2024-04-22T23:23:57.402 INFO:tasks.workunit.client.1.smithi134.stdout:2/748: dread - d2/dd/df/fb9 zero size 2024-04-22T23:23:57.410 INFO:tasks.workunit.client.1.smithi134.stdout:6/895: sync 2024-04-22T23:23:57.410 INFO:tasks.workunit.client.1.smithi134.stdout:3/602: sync 2024-04-22T23:23:57.410 INFO:tasks.workunit.client.1.smithi134.stdout:9/746: sync 2024-04-22T23:23:57.419 INFO:tasks.workunit.client.1.smithi134.stdout:2/749: dread d2/dd/f47 [0,4194304] 0 2024-04-22T23:23:57.424 INFO:tasks.workunit.client.1.smithi134.stdout:6/896: write d0/d19/de/f13 [4559344,101850] 0 2024-04-22T23:23:57.427 INFO:tasks.workunit.client.1.smithi134.stdout:9/747: unlink d0/d7/d23/d1b/le9 0 2024-04-22T23:23:57.433 INFO:tasks.workunit.client.1.smithi134.stdout:2/750: dread d2/dd/d68/fa9 [0,4194304] 0 2024-04-22T23:23:57.438 INFO:tasks.workunit.client.1.smithi134.stdout:2/751: read d2/fc1 [877267,48001] 0 2024-04-22T23:23:57.438 INFO:tasks.workunit.client.1.smithi134.stdout:6/897: dwrite d0/d19/de/d12/d29/d74/da0/fd9 [0,4194304] 0 2024-04-22T23:23:57.439 INFO:tasks.workunit.client.1.smithi134.stdout:2/752: fdatasync d2/d15/d24/f29 0 2024-04-22T23:23:57.446 INFO:tasks.workunit.client.1.smithi134.stdout:9/748: creat d0/d7/ffc x:0 0 0 2024-04-22T23:23:57.447 INFO:tasks.workunit.client.1.smithi134.stdout:3/603: link dc/d2a/d62/db2/d2f/d61/d6a/caf dc/cdf 0 2024-04-22T23:23:57.454 INFO:tasks.workunit.client.1.smithi134.stdout:2/753: dwrite d2/dd/df/d1c/f76 [0,4194304] 0 2024-04-22T23:23:57.454 INFO:tasks.workunit.client.1.smithi134.stdout:6/898: dwrite d0/d19/de/d12/d29/d74/f8e [0,4194304] 0 2024-04-22T23:23:57.456 INFO:tasks.workunit.client.1.smithi134.stdout:3/604: creat dc/d2a/d62/db2/d2f/d36/d6d/d6f/dd9/fe0 x:0 0 0 2024-04-22T23:23:57.464 INFO:tasks.workunit.client.1.smithi134.stdout:9/749: creat d0/ffd x:0 0 0 2024-04-22T23:23:57.472 INFO:tasks.workunit.client.1.smithi134.stdout:3/605: creat dc/d2a/d62/d86/dc7/dcf/fe1 x:0 0 0 2024-04-22T23:23:57.472 INFO:tasks.workunit.client.1.smithi134.stdout:2/754: link d2/dd/df/d1c/f79 d2/d94/f100 0 2024-04-22T23:23:57.475 INFO:tasks.workunit.client.1.smithi134.stdout:3/606: creat dc/d2a/d62/d86/dc1/fe2 x:0 0 0 2024-04-22T23:23:57.476 INFO:tasks.workunit.client.1.smithi134.stdout:2/755: rmdir d2/dd/d4a/d73 39 2024-04-22T23:23:57.477 INFO:tasks.workunit.client.1.smithi134.stdout:3/607: chown dc/d2a/d62/db2/d34/c8e 1259939762 1 2024-04-22T23:23:57.486 INFO:tasks.workunit.client.1.smithi134.stdout:3/608: dwrite dc/d2a/d62/d86/fba [0,4194304] 0 2024-04-22T23:23:57.489 INFO:tasks.workunit.client.1.smithi134.stdout:6/899: write d0/d19/de/d12/d2a/dbb/fdf [1969329,121183] 0 2024-04-22T23:23:57.495 INFO:tasks.workunit.client.1.smithi134.stdout:3/609: rename dc/dd/l74 to dc/d2a/d62/db2/d2f/d36/d5f/le3 0 2024-04-22T23:23:57.500 INFO:tasks.workunit.client.1.smithi134.stdout:6/900: mknod d0/d19/de/d1f/dd5/d122/c128 0 2024-04-22T23:23:57.500 INFO:tasks.workunit.client.1.smithi134.stdout:2/756: dwrite d2/dd/d77/fa3 [4194304,4194304] 0 2024-04-22T23:23:57.503 INFO:tasks.workunit.client.1.smithi134.stdout:3/610: creat dc/d2a/d62/d86/dc1/dd8/fe4 x:0 0 0 2024-04-22T23:23:57.506 INFO:tasks.workunit.client.1.smithi134.stdout:3/611: write dc/d2a/d62/fa6 [2471561,32851] 0 2024-04-22T23:23:57.510 INFO:tasks.workunit.client.1.smithi134.stdout:9/750: dread d0/d7/d23/d26/d51/d88/fcd [0,4194304] 0 2024-04-22T23:23:57.511 INFO:tasks.workunit.client.1.smithi134.stdout:9/751: chown d0/d2d/d94/fed 41064434 1 2024-04-22T23:23:57.512 INFO:tasks.workunit.client.1.smithi134.stdout:3/612: dread - dc/d2a/d62/db2/d2f/d36/d6d/f9c zero size 2024-04-22T23:23:57.515 INFO:tasks.workunit.client.1.smithi134.stdout:9/752: chown d0/d7/d23/d26/d4d/d5d/d8e/dae/db1/fd9 1691360895 1 2024-04-22T23:23:57.519 INFO:tasks.workunit.client.1.smithi134.stdout:3/613: rename dc/d2a/d30/f83 to dc/d2a/fe5 0 2024-04-22T23:23:57.523 INFO:tasks.workunit.client.1.smithi134.stdout:6/901: dwrite d0/d19/de/d12/d2a/d2d/d3a/f77 [0,4194304] 0 2024-04-22T23:23:57.529 INFO:tasks.workunit.client.1.smithi134.stdout:2/757: dwrite d2/f87 [0,4194304] 0 2024-04-22T23:23:57.532 INFO:tasks.workunit.client.1.smithi134.stdout:3/614: dwrite dc/d2a/d62/db2/d3b/d29/f2e [4194304,4194304] 0 2024-04-22T23:23:57.532 INFO:tasks.workunit.client.1.smithi134.stdout:6/902: rename d0/d19/de/fb0 to d0/d19/db/d75/d92/f129 0 2024-04-22T23:23:57.535 INFO:tasks.workunit.client.1.smithi134.stdout:9/753: link d0/d7/f19 d0/d7/d12/ffe 0 2024-04-22T23:23:57.535 INFO:tasks.workunit.client.1.smithi134.stdout:6/903: chown d0/d19/de/d12/f97 73 1 2024-04-22T23:23:57.543 INFO:tasks.workunit.client.1.smithi134.stdout:2/758: dread d2/d15/d24/f1e [4194304,4194304] 0 2024-04-22T23:23:57.553 INFO:tasks.workunit.client.1.smithi134.stdout:9/754: dread d0/d7/d23/d26/f58 [0,4194304] 0 2024-04-22T23:23:57.554 INFO:tasks.workunit.client.1.smithi134.stdout:3/615: creat dc/d2a/d62/db2/d2f/d21/d42/fe6 x:0 0 0 2024-04-22T23:23:57.555 INFO:tasks.workunit.client.1.smithi134.stdout:6/904: symlink d0/d19/db/d75/d92/l12a 0 2024-04-22T23:23:57.556 INFO:tasks.workunit.client.1.smithi134.stdout:5/921: sync 2024-04-22T23:23:57.557 INFO:tasks.workunit.client.1.smithi134.stdout:8/809: sync 2024-04-22T23:23:57.562 INFO:tasks.workunit.client.1.smithi134.stdout:5/922: symlink d9/dbb/d11b/df1/l137 0 2024-04-22T23:23:57.563 INFO:tasks.workunit.client.1.smithi134.stdout:3/616: unlink dc/d2a/d62/db2/d2f/d36/d6d/d80/fbb 0 2024-04-22T23:23:57.564 INFO:tasks.workunit.client.1.smithi134.stdout:5/923: read d9/d11/d34/f54 [2255862,19055] 0 2024-04-22T23:23:57.579 INFO:tasks.workunit.client.1.smithi134.stdout:3/617: symlink dc/d2a/d62/db2/d34/d70/le7 0 2024-04-22T23:23:57.581 INFO:tasks.workunit.client.1.smithi134.stdout:2/759: dwrite d2/d94/f100 [8388608,4194304] 0 2024-04-22T23:23:57.581 INFO:tasks.workunit.client.1.smithi134.stdout:6/905: dread d0/f8 [0,4194304] 0 2024-04-22T23:23:57.591 INFO:tasks.workunit.client.1.smithi134.stdout:2/760: dwrite d2/d42/d54/d33/d6e/fe1 [0,4194304] 0 2024-04-22T23:23:57.592 INFO:tasks.workunit.client.1.smithi134.stdout:9/755: truncate d0/d7/d23/d26/fe3 3088163 0 2024-04-22T23:23:57.594 INFO:tasks.workunit.client.1.smithi134.stdout:3/618: truncate dc/d2a/d62/db2/d2f/d21/d42/f47 4535705 0 2024-04-22T23:23:57.595 INFO:tasks.workunit.client.1.smithi134.stdout:8/810: dwrite d7/db/d42/d65/d46/d9a/fa1 [0,4194304] 0 2024-04-22T23:23:57.596 INFO:tasks.workunit.client.1.smithi134.stdout:8/811: chown d7/dc/d19 8153 1 2024-04-22T23:23:57.612 INFO:tasks.workunit.client.1.smithi134.stdout:8/812: mknod d7/db/d2d/d39/c100 0 2024-04-22T23:23:57.613 INFO:tasks.workunit.client.1.smithi134.stdout:5/924: rename d9/d11/d34/d85/f9e to d9/dbb/d11b/d17/f138 0 2024-04-22T23:23:57.626 INFO:tasks.workunit.client.1.smithi134.stdout:8/813: creat d7/db/d2d/d39/d58/f101 x:0 0 0 2024-04-22T23:23:57.627 INFO:tasks.workunit.client.1.smithi134.stdout:5/925: link d9/da2/de6/dc0/le2 d9/dbb/d11b/da6/d114/d11c/l139 0 2024-04-22T23:23:57.629 INFO:tasks.workunit.client.1.smithi134.stdout:8/814: dread - d7/db/d42/d65/dd9/dda/de7/fdb zero size 2024-04-22T23:23:57.633 INFO:tasks.workunit.client.1.smithi134.stdout:8/815: readlink d7/dc/d2c/l35 0 2024-04-22T23:23:57.633 INFO:tasks.workunit.client.1.smithi134.stdout:5/926: dread d9/d11/fc7 [0,4194304] 0 2024-04-22T23:23:57.643 INFO:tasks.workunit.client.1.smithi134.stdout:2/761: dwrite d2/d15/d24/d1d/d3d/f92 [0,4194304] 0 2024-04-22T23:23:57.648 INFO:tasks.workunit.client.1.smithi134.stdout:5/927: getdents d9/dbb/d11b/d17/d2a 0 2024-04-22T23:23:57.649 INFO:tasks.workunit.client.1.smithi134.stdout:9/756: write d0/d7/f9 [990551,68211] 0 2024-04-22T23:23:57.651 INFO:tasks.workunit.client.1.smithi134.stdout:2/762: mkdir d2/d15/d24/d3b/d8a/d101 0 2024-04-22T23:23:57.654 INFO:tasks.workunit.client.1.smithi134.stdout:5/928: unlink d9/d11/d34/f51 0 2024-04-22T23:23:57.655 INFO:tasks.workunit.client.1.smithi134.stdout:3/619: write dc/d2a/d62/db2/d3b/d29/fb7 [493345,113438] 0 2024-04-22T23:23:57.657 INFO:tasks.workunit.client.1.smithi134.stdout:3/620: fdatasync dc/d2a/d62/db2/d2f/d36/d6d/d80/fc0 0 2024-04-22T23:23:57.658 INFO:tasks.workunit.client.1.smithi134.stdout:9/757: mknod d0/d7/d23/d1b/cff 0 2024-04-22T23:23:57.660 INFO:tasks.workunit.client.1.smithi134.stdout:5/929: fsync d9/d11/d73/f80 0 2024-04-22T23:23:57.660 INFO:tasks.workunit.client.1.smithi134.stdout:9/758: read d0/d7/d23/f2a [3732786,4329] 0 2024-04-22T23:23:57.664 INFO:tasks.workunit.client.1.smithi134.stdout:2/763: symlink d2/l102 0 2024-04-22T23:23:57.672 INFO:tasks.workunit.client.1.smithi134.stdout:2/764: dwrite d2/d42/d54/d33/d91/fe7 [0,4194304] 0 2024-04-22T23:23:57.677 INFO:tasks.workunit.client.1.smithi134.stdout:3/621: dread dc/f45 [4194304,4194304] 0 2024-04-22T23:23:57.679 INFO:tasks.workunit.client.1.smithi134.stdout:5/930: creat d9/d11/d73/f13a x:0 0 0 2024-04-22T23:23:57.691 INFO:tasks.workunit.client.1.smithi134.stdout:5/931: dwrite d9/dbb/d11b/d52/f6b [0,4194304] 0 2024-04-22T23:23:57.691 INFO:tasks.workunit.client.1.smithi134.stdout:3/622: creat dc/d2a/d62/d86/dc1/dd8/fe8 x:0 0 0 2024-04-22T23:23:57.699 INFO:tasks.workunit.client.1.smithi134.stdout:8/816: dwrite d7/dc/d2c/d37/d47/f31 [0,4194304] 0 2024-04-22T23:23:57.701 INFO:tasks.workunit.client.1.smithi134.stdout:5/932: mknod d9/dbb/d11b/df1/c13b 0 2024-04-22T23:23:57.702 INFO:tasks.workunit.client.1.smithi134.stdout:3/623: creat dc/d2a/d62/db2/d34/d6b/fe9 x:0 0 0 2024-04-22T23:23:57.707 INFO:tasks.workunit.client.1.smithi134.stdout:8/817: unlink d7/db/d42/d65/dd9/dda/de7/ddd/fce 0 2024-04-22T23:23:57.714 INFO:tasks.workunit.client.1.smithi134.stdout:8/818: dwrite d7/dc/d19/da5/fab [0,4194304] 0 2024-04-22T23:23:57.720 INFO:tasks.workunit.client.1.smithi134.stdout:5/933: creat d9/d11/d34/d4d/dd4/f13c x:0 0 0 2024-04-22T23:23:57.720 INFO:tasks.workunit.client.1.smithi134.stdout:3/624: link dc/d2a/d62/db2/d3b/l96 dc/d2a/d62/db2/d2f/d36/d6d/d6f/dd9/lea 0 2024-04-22T23:23:57.725 INFO:tasks.workunit.client.1.smithi134.stdout:8/819: creat d7/db/d42/f102 x:0 0 0 2024-04-22T23:23:57.727 INFO:tasks.workunit.client.1.smithi134.stdout:8/820: fdatasync d7/dc/d2c/d37/d3a/fe2 0 2024-04-22T23:23:57.728 INFO:tasks.workunit.client.1.smithi134.stdout:5/934: dwrite d9/d11/d34/d4d/f9f [0,4194304] 0 2024-04-22T23:23:57.733 INFO:tasks.workunit.client.1.smithi134.stdout:3/625: mknod dc/d2a/d62/db2/d2f/d36/d6d/d6f/dd9/ceb 0 2024-04-22T23:23:57.741 INFO:tasks.workunit.client.1.smithi134.stdout:9/759: dwrite d0/d7/d23/f55 [0,4194304] 0 2024-04-22T23:23:57.742 INFO:tasks.workunit.client.1.smithi134.stdout:9/760: stat d0/d37/d76/f91 0 2024-04-22T23:23:57.745 INFO:tasks.workunit.client.1.smithi134.stdout:2/765: truncate d2/f2c 8054200 0 2024-04-22T23:23:57.746 INFO:tasks.workunit.client.1.smithi134.stdout:5/935: creat d9/d11/d73/d95/d20/d108/f13d x:0 0 0 2024-04-22T23:23:57.759 INFO:tasks.workunit.client.1.smithi134.stdout:0/866: sync 2024-04-22T23:23:57.759 INFO:tasks.workunit.client.1.smithi134.stdout:7/891: sync 2024-04-22T23:23:57.759 INFO:tasks.workunit.client.1.smithi134.stdout:7/892: readlink d4/d8/d41/d5b/l61 0 2024-04-22T23:23:57.762 INFO:tasks.workunit.client.1.smithi134.stdout:8/821: write d7/db/d42/f59 [661165,34523] 0 2024-04-22T23:23:57.764 INFO:tasks.workunit.client.1.smithi134.stdout:3/626: write dc/d2a/d62/db2/d3b/f31 [980675,40863] 0 2024-04-22T23:23:57.766 INFO:tasks.workunit.client.1.smithi134.stdout:3/627: chown dc/d2a/d62/db2/d2f/d36/d6d/d80 16872 1 2024-04-22T23:23:57.768 INFO:tasks.workunit.client.1.smithi134.stdout:9/761: link d0/d2d/d82/da0/cc6 d0/d37/d76/c100 0 2024-04-22T23:23:57.769 INFO:tasks.workunit.client.1.smithi134.stdout:9/762: stat d0/d2d/d82/da0/lc2 0 2024-04-22T23:23:57.779 INFO:tasks.workunit.client.1.smithi134.stdout:5/936: write d9/fa [1083439,11320] 0 2024-04-22T23:23:57.785 INFO:tasks.workunit.client.1.smithi134.stdout:0/867: dwrite d1/d5/d15/fe9 [0,4194304] 0 2024-04-22T23:23:57.785 INFO:tasks.workunit.client.1.smithi134.stdout:0/868: stat d1/d2/f74 0 2024-04-22T23:23:57.785 INFO:tasks.workunit.client.1.smithi134.stdout:7/893: mkdir d4/d8/d41/d5b/d104/d140 0 2024-04-22T23:23:57.786 INFO:tasks.workunit.client.1.smithi134.stdout:8/822: creat d7/db/d42/d65/d46/d9a/da3/d88/f103 x:0 0 0 2024-04-22T23:23:57.789 INFO:tasks.workunit.client.1.smithi134.stdout:3/628: rename dc/d2a/d62/d86/dc7 to dc/d2a/d62/db2/d2f/d21/dec 0 2024-04-22T23:23:57.790 INFO:tasks.workunit.client.1.smithi134.stdout:7/894: write d4/dad/f10b [211098,33388] 0 2024-04-22T23:23:57.800 INFO:tasks.workunit.client.1.smithi134.stdout:9/763: unlink d0/d7/d27/lf2 0 2024-04-22T23:23:57.816 INFO:tasks.workunit.client.1.smithi134.stdout:2/766: truncate d2/dd/d4a/d73/f7f 648939 0 2024-04-22T23:23:57.817 INFO:tasks.workunit.client.1.smithi134.stdout:2/767: write d2/d42/f46 [9027800,9989] 0 2024-04-22T23:23:57.821 INFO:tasks.workunit.client.1.smithi134.stdout:3/629: creat dc/d2a/d62/db2/d2f/d54/dac/fed x:0 0 0 2024-04-22T23:23:57.829 INFO:tasks.workunit.client.1.smithi134.stdout:2/768: write d2/d15/d24/f6d [2069275,115074] 0 2024-04-22T23:23:57.834 INFO:tasks.workunit.client.1.smithi134.stdout:8/823: symlink d7/dc/d19/l104 0 2024-04-22T23:23:57.836 INFO:tasks.workunit.client.1.smithi134.stdout:7/895: symlink d4/l141 0 2024-04-22T23:23:57.836 INFO:tasks.workunit.client.1.smithi134.stdout:5/937: truncate d9/d1e/d90/fa1 1112513 0 2024-04-22T23:23:57.840 INFO:tasks.workunit.client.1.smithi134.stdout:6/906: sync 2024-04-22T23:23:57.842 INFO:tasks.workunit.client.1.smithi134.stdout:0/869: link d1/d2/d1d/f91 d1/d5/d58/d6b/db9/d75/f119 0 2024-04-22T23:23:57.845 INFO:tasks.workunit.client.1.smithi134.stdout:3/630: dread dc/d2a/d62/db2/d3b/f31 [0,4194304] 0 2024-04-22T23:23:57.845 INFO:tasks.workunit.client.1.smithi134.stdout:2/769: rename d2/d15/d24/d1d/d3d/fc8 to d2/d42/d54/f103 0 2024-04-22T23:23:57.847 INFO:tasks.workunit.client.1.smithi134.stdout:8/824: mkdir d7/db/d2d/d39/d105 0 2024-04-22T23:23:57.848 INFO:tasks.workunit.client.1.smithi134.stdout:7/896: write d4/dbd/f13b [1357393,57673] 0 2024-04-22T23:23:57.851 INFO:tasks.workunit.client.1.smithi134.stdout:9/764: creat d0/d7/d23/d1b/f101 x:0 0 0 2024-04-22T23:23:57.852 INFO:tasks.workunit.client.1.smithi134.stdout:3/631: dwrite dc/f52 [0,4194304] 0 2024-04-22T23:23:57.854 INFO:tasks.workunit.client.1.smithi134.stdout:6/907: creat d0/d19/de/d12/d2a/dbb/f12b x:0 0 0 2024-04-22T23:23:57.864 INFO:tasks.workunit.client.1.smithi134.stdout:8/825: dread d7/dc/f22 [0,4194304] 0 2024-04-22T23:23:57.866 INFO:tasks.workunit.client.1.smithi134.stdout:3/632: mknod dc/d2a/d62/db2/d34/d70/cee 0 2024-04-22T23:23:57.872 INFO:tasks.workunit.client.1.smithi134.stdout:8/826: dwrite d7/db/d42/d65/d46/fdf [0,4194304] 0 2024-04-22T23:23:57.878 INFO:tasks.workunit.client.1.smithi134.stdout:2/770: unlink d2/dd/c43 0 2024-04-22T23:23:57.883 INFO:tasks.workunit.client.1.smithi134.stdout:3/633: unlink dc/d2a/d30/fa0 0 2024-04-22T23:23:57.885 INFO:tasks.workunit.client.1.smithi134.stdout:0/870: link d1/d5/d58/f89 d1/d5/d58/d6b/db9/d75/de6/f11a 0 2024-04-22T23:23:57.888 INFO:tasks.workunit.client.1.smithi134.stdout:2/771: rmdir d2/dd/df/d89 39 2024-04-22T23:23:57.896 INFO:tasks.workunit.client.1.smithi134.stdout:3/634: mkdir dc/d2a/d62/db2/d34/d6b/dae/def 0 2024-04-22T23:23:57.900 INFO:tasks.workunit.client.1.smithi134.stdout:0/871: rename d1/d5/dd/d20/f1e to d1/d5/dd/d41/f11b 0 2024-04-22T23:23:57.900 INFO:tasks.workunit.client.1.smithi134.stdout:9/765: write d0/d7/d23/d26/d4d/d5d/f8f [3060269,52979] 0 2024-04-22T23:23:57.904 INFO:tasks.workunit.client.1.smithi134.stdout:7/897: write d4/d8/d41/d76/dc1/d2f/d34/d62/d84/fee [3619072,41771] 0 2024-04-22T23:23:57.905 INFO:tasks.workunit.client.1.smithi134.stdout:6/908: dwrite d0/d19/db/d33/f64 [4194304,4194304] 0 2024-04-22T23:23:57.907 INFO:tasks.workunit.client.1.smithi134.stdout:8/827: truncate d7/f23 1502427 0 2024-04-22T23:23:57.909 INFO:tasks.workunit.client.1.smithi134.stdout:3/635: dwrite dc/f18 [4194304,4194304] 0 2024-04-22T23:23:57.987 INFO:tasks.workunit.client.1.smithi134.stdout:2/772: symlink d2/d42/d54/ddd/l104 0 2024-04-22T23:23:57.987 INFO:tasks.workunit.client.1.smithi134.stdout:0/872: rmdir d1/d5/dd/d68/dc0 39 2024-04-22T23:23:57.989 INFO:tasks.workunit.client.1.smithi134.stdout:0/873: write d1/d5/d15/f104 [517320,23720] 0 2024-04-22T23:23:57.995 INFO:tasks.workunit.client.1.smithi134.stdout:2/773: dread d2/d15/d24/d1d/f28 [0,4194304] 0 2024-04-22T23:23:57.998 INFO:tasks.workunit.client.1.smithi134.stdout:3/636: symlink dc/d7d/lf0 0 2024-04-22T23:23:58.002 INFO:tasks.workunit.client.1.smithi134.stdout:0/874: mknod d1/d5/d58/d94/c11c 0 2024-04-22T23:23:58.003 INFO:tasks.workunit.client.1.smithi134.stdout:7/898: dread d4/d8/f21 [0,4194304] 0 2024-04-22T23:23:58.003 INFO:tasks.workunit.client.1.smithi134.stdout:0/875: readlink d1/d5/d58/d22/l60 0 2024-04-22T23:23:58.008 INFO:tasks.workunit.client.1.smithi134.stdout:9/766: link d0/d7/d23/d26/d4d/d5d/f8f d0/d2d/d3d/f102 0 2024-04-22T23:23:58.010 INFO:tasks.workunit.client.1.smithi134.stdout:9/767: chown d0/d7/d23/d26/d51/d88/fcd 51411 1 2024-04-22T23:23:58.010 INFO:tasks.workunit.client.1.smithi134.stdout:6/909: fsync d0/d19/fa 0 2024-04-22T23:23:58.012 INFO:tasks.workunit.client.1.smithi134.stdout:2/774: fsync d2/d42/d54/f103 0 2024-04-22T23:23:58.014 INFO:tasks.workunit.client.1.smithi134.stdout:3/637: rename dc/d2a/d62/db2/f1c to dc/d2a/d62/db2/d2f/d21/d42/ff1 0 2024-04-22T23:23:58.016 INFO:tasks.workunit.client.1.smithi134.stdout:7/899: creat d4/d8/d41/d76/dc1/d2f/d34/dc0/f142 x:0 0 0 2024-04-22T23:23:58.017 INFO:tasks.workunit.client.1.smithi134.stdout:7/900: chown d4/d4a/d85 8899 1 2024-04-22T23:23:58.104 INFO:tasks.workunit.client.1.smithi134.stdout:3/638: mkdir dc/d2a/d62/db2/d34/d6b/dae/dc3/df2 0 2024-04-22T23:23:58.105 INFO:tasks.workunit.client.1.smithi134.stdout:6/910: fdatasync d0/f26 0 2024-04-22T23:23:58.113 INFO:tasks.workunit.client.1.smithi134.stdout:7/901: dread d4/d8/d41/d76/dc1/d2f/d69/d8a/fb7 [0,4194304] 0 2024-04-22T23:23:58.115 INFO:tasks.workunit.client.1.smithi134.stdout:2/775: link d2/dd/d4a/d73/l7d d2/d15/d24/d1d/d3d/d7b/l105 0 2024-04-22T23:23:58.117 INFO:tasks.workunit.client.1.smithi134.stdout:6/911: rename d0/d19/de/d12/f61 to d0/d19/db/d75/f12c 0 2024-04-22T23:23:58.119 INFO:tasks.workunit.client.1.smithi134.stdout:9/768: link d0/d2d/dc1/fc7 d0/f103 0 2024-04-22T23:23:58.122 INFO:tasks.workunit.client.1.smithi134.stdout:7/902: fdatasync d4/d8/d41/d76/dc1/d2f/d70/d86/ff0 0 2024-04-22T23:23:58.124 INFO:tasks.workunit.client.1.smithi134.stdout:2/776: mknod d2/dd/df/d9c/dca/c106 0 2024-04-22T23:23:58.125 INFO:tasks.workunit.client.1.smithi134.stdout:3/639: dwrite dc/d2a/d62/db2/d34/d6b/f7c [0,4194304] 0 2024-04-22T23:23:58.128 INFO:tasks.workunit.client.1.smithi134.stdout:3/640: write dc/f18 [2632360,34466] 0 2024-04-22T23:23:58.128 INFO:tasks.workunit.client.1.smithi134.stdout:3/641: stat dc/d2a/d62/db2/d3b/fb0 0 2024-04-22T23:23:58.132 INFO:tasks.workunit.client.1.smithi134.stdout:9/769: symlink d0/d7/d23/d26/d4d/d5d/d8e/dae/l104 0 2024-04-22T23:23:58.135 INFO:tasks.workunit.client.1.smithi134.stdout:2/777: dread d2/d15/d24/d1d/d3d/d9a/fd5 [0,4194304] 0 2024-04-22T23:23:58.139 INFO:tasks.workunit.client.1.smithi134.stdout:3/642: rmdir dc 39 2024-04-22T23:23:58.141 INFO:tasks.workunit.client.1.smithi134.stdout:6/912: dwrite d0/d19/db/d33/f95 [0,4194304] 0 2024-04-22T23:23:58.143 INFO:tasks.workunit.client.1.smithi134.stdout:9/770: mkdir d0/d7/d23/d26/d4d/d5d/d8e/d105 0 2024-04-22T23:23:58.151 INFO:tasks.workunit.client.1.smithi134.stdout:6/913: rmdir d0/d19/de/d12 39 2024-04-22T23:23:58.152 INFO:tasks.workunit.client.1.smithi134.stdout:7/903: dwrite d4/d8/d41/d76/dc1/d2f/d69/dab/de1/ff7 [0,4194304] 0 2024-04-22T23:23:58.153 INFO:tasks.workunit.client.1.smithi134.stdout:9/771: creat d0/d2d/d94/f106 x:0 0 0 2024-04-22T23:23:58.159 INFO:tasks.workunit.client.1.smithi134.stdout:2/778: creat d2/d15/d24/d1d/dba/dea/f107 x:0 0 0 2024-04-22T23:23:58.163 INFO:tasks.workunit.client.1.smithi134.stdout:6/914: write d0/d19/de/d12/d5e/d6d/d7a/db2/f123 [142089,101512] 0 2024-04-22T23:23:58.166 INFO:tasks.workunit.client.1.smithi134.stdout:2/779: dwrite d2/f87 [4194304,4194304] 0 2024-04-22T23:23:58.167 INFO:tasks.workunit.client.1.smithi134.stdout:9/772: unlink d0/d2d/d3d/l70 0 2024-04-22T23:23:58.169 INFO:tasks.workunit.client.1.smithi134.stdout:3/643: write dc/d2a/d62/db2/f5c [2189450,71592] 0 2024-04-22T23:23:58.169 INFO:tasks.workunit.client.1.smithi134.stdout:6/915: mknod d0/c12d 0 2024-04-22T23:23:58.169 INFO:tasks.workunit.client.1.smithi134.stdout:3/644: readlink dc/dd/l27 0 2024-04-22T23:23:58.171 INFO:tasks.workunit.client.1.smithi134.stdout:9/773: chown d0/d7/d27/fa5 3 1 2024-04-22T23:23:58.173 INFO:tasks.workunit.client.1.smithi134.stdout:2/780: creat d2/dd/df/d1c/de3/f108 x:0 0 0 2024-04-22T23:23:58.178 INFO:tasks.workunit.client.1.smithi134.stdout:6/916: creat d0/d19/de/d12/d5e/d6d/d7a/d99/f12e x:0 0 0 2024-04-22T23:23:58.183 INFO:tasks.workunit.client.1.smithi134.stdout:2/781: mkdir d2/d42/d54/d109 0 2024-04-22T23:23:58.184 INFO:tasks.workunit.client.1.smithi134.stdout:7/904: truncate d4/fac 1601048 0 2024-04-22T23:23:58.188 INFO:tasks.workunit.client.1.smithi134.stdout:9/774: write d0/d7/d23/d26/d4d/d5d/f8a [789176,59569] 0 2024-04-22T23:23:58.189 INFO:tasks.workunit.client.1.smithi134.stdout:2/782: dread - d2/dd/d72/fab zero size 2024-04-22T23:23:58.193 INFO:tasks.workunit.client.1.smithi134.stdout:6/917: creat d0/d19/db/d121/f12f x:0 0 0 2024-04-22T23:23:58.193 INFO:tasks.workunit.client.1.smithi134.stdout:3/645: dwrite dc/d2a/d62/db2/d34/d72/fa1 [0,4194304] 0 2024-04-22T23:23:58.195 INFO:tasks.workunit.client.1.smithi134.stdout:7/905: creat d4/d8/d101/f143 x:0 0 0 2024-04-22T23:23:58.197 INFO:tasks.workunit.client.1.smithi134.stdout:2/783: rename d2/d15/d24/d1d/f28 to d2/dd/d72/f10a 0 2024-04-22T23:23:58.198 INFO:tasks.workunit.client.1.smithi134.stdout:3/646: read - dc/d2a/d62/db2/d2f/d36/d6d/d6f/dd9/fe0 zero size 2024-04-22T23:23:58.200 INFO:tasks.workunit.client.1.smithi134.stdout:2/784: fsync d2/dd/df/d9c/daa/db0/f95 0 2024-04-22T23:23:58.201 INFO:tasks.workunit.client.1.smithi134.stdout:3/647: chown dc/d2a/d30/l92 113 1 2024-04-22T23:23:58.203 INFO:tasks.workunit.client.1.smithi134.stdout:6/918: unlink d0/d19/de/dbe/dd1/f101 0 2024-04-22T23:23:58.203 INFO:tasks.workunit.client.1.smithi134.stdout:3/648: readlink dc/d2a/d62/db2/d2f/l58 0 2024-04-22T23:23:58.204 INFO:tasks.workunit.client.1.smithi134.stdout:9/775: dwrite d0/d7/d12/d22/fea [0,4194304] 0 2024-04-22T23:23:58.205 INFO:tasks.workunit.client.1.smithi134.stdout:2/785: dread - d2/dd/df/d9c/daa/ff4 zero size 2024-04-22T23:23:58.207 INFO:tasks.workunit.client.1.smithi134.stdout:6/919: read d0/d19/db/d33/f95 [4115520,84479] 0 2024-04-22T23:23:58.207 INFO:tasks.workunit.client.1.smithi134.stdout:8/828: sync 2024-04-22T23:23:58.208 INFO:tasks.workunit.client.1.smithi134.stdout:0/876: sync 2024-04-22T23:23:58.208 INFO:tasks.workunit.client.1.smithi134.stdout:5/938: sync 2024-04-22T23:23:58.210 INFO:tasks.workunit.client.1.smithi134.stdout:5/939: chown d9/d11/d73/d95/f119 485 1 2024-04-22T23:23:58.225 INFO:tasks.workunit.client.1.smithi134.stdout:7/906: dwrite d4/f13 [0,4194304] 0 2024-04-22T23:23:58.225 INFO:tasks.workunit.client.1.smithi134.stdout:8/829: dread - d7/dc/d2c/d4f/d6c/d6e/fc4 zero size 2024-04-22T23:23:58.226 INFO:tasks.workunit.client.1.smithi134.stdout:7/907: stat d4/d8/d58 0 2024-04-22T23:23:58.228 INFO:tasks.workunit.client.1.smithi134.stdout:8/830: read d7/db/d42/d65/dd9/dda/fbf [2415031,18888] 0 2024-04-22T23:23:58.230 INFO:tasks.workunit.client.1.smithi134.stdout:2/786: mknod d2/d42/d54/dfd/c10b 0 2024-04-22T23:23:58.234 INFO:tasks.workunit.client.1.smithi134.stdout:5/940: mknod d9/d11/d34/d4d/dd4/d121/d125/c13e 0 2024-04-22T23:23:58.235 INFO:tasks.workunit.client.1.smithi134.stdout:6/920: dwrite d0/d19/db/d33/d86/f11f [0,4194304] 0 2024-04-22T23:23:58.236 INFO:tasks.workunit.client.1.smithi134.stdout:7/908: mknod d4/d8/d41/d5b/c144 0 2024-04-22T23:23:58.237 INFO:tasks.workunit.client.1.smithi134.stdout:5/941: chown d9/dbb/d11b/d17/d44/c94 65166 1 2024-04-22T23:23:58.242 INFO:tasks.workunit.client.1.smithi134.stdout:0/877: rename d1/d5/dd/d41/l4d to d1/d5/d15/l11d 0 2024-04-22T23:23:58.244 INFO:tasks.workunit.client.1.smithi134.stdout:8/831: dread d7/dc/d2c/d4f/d6c/d6e/f91 [0,4194304] 0 2024-04-22T23:23:58.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:57 smithi134 ceph-mon[34609]: Deploying cephadm binary to smithi067 2024-04-22T23:23:58.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:57 smithi134 ceph-mon[34609]: Deploying cephadm binary to smithi134 2024-04-22T23:23:58.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:57 smithi134 ceph-mon[34609]: mgrmap e22: smithi134.ccxkcj(active, since 1.01105s) 2024-04-22T23:23:58.245 INFO:tasks.workunit.client.1.smithi134.stdout:8/832: stat d7/dc/d19/l104 0 2024-04-22T23:23:58.246 INFO:tasks.workunit.client.1.smithi134.stdout:9/776: getdents d0/d2d/d82 0 2024-04-22T23:23:58.247 INFO:tasks.workunit.client.1.smithi134.stdout:6/921: dread - d0/d19/de/d12/d5e/d6d/d7a/db2/fd7 zero size 2024-04-22T23:23:58.248 INFO:tasks.workunit.client.1.smithi134.stdout:6/922: fdatasync d0/d19/de/d12/d5e/d6d/d7a/fde 0 2024-04-22T23:23:58.250 INFO:tasks.workunit.client.1.smithi134.stdout:3/649: getdents dc/d2a/d62/d86/db3 0 2024-04-22T23:23:58.251 INFO:tasks.workunit.client.1.smithi134.stdout:7/909: truncate d4/d8/d41/d76/dc1/d2f/d34/d62/d29/dc5/fe0 303000 0 2024-04-22T23:23:58.253 INFO:tasks.workunit.client.1.smithi134.stdout:5/942: mkdir d9/d11/d73/da9/d13f 0 2024-04-22T23:23:58.257 INFO:tasks.workunit.client.1.smithi134.stdout:9/777: fsync d0/d7/d12/f30 0 2024-04-22T23:23:58.260 INFO:tasks.workunit.client.1.smithi134.stdout:9/778: chown d0/d7/d23/d26/d51/d88/db9/ff1 3864 1 2024-04-22T23:23:58.262 INFO:tasks.workunit.client.1.smithi134.stdout:3/650: rename dc/d2a/d62/db2/d2f/f4b to dc/d2a/ff3 0 2024-04-22T23:23:58.267 INFO:tasks.workunit.client.1.smithi134.stdout:0/878: dread d1/d5/dd/d20/f6c [0,4194304] 0 2024-04-22T23:23:58.269 INFO:tasks.workunit.client.1.smithi134.stdout:6/923: symlink d0/d19/de/dbe/dd1/de9/l130 0 2024-04-22T23:23:58.273 INFO:tasks.workunit.client.1.smithi134.stdout:3/651: creat dc/d2a/d62/db2/d2f/d36/d6d/ff4 x:0 0 0 2024-04-22T23:23:58.275 INFO:tasks.workunit.client.1.smithi134.stdout:2/787: link d2/dd/d68/l8f d2/dd/df/d9c/daa/db0/l10c 0 2024-04-22T23:23:58.279 INFO:tasks.workunit.client.1.smithi134.stdout:0/879: creat d1/d2/d1d/d72/f11e x:0 0 0 2024-04-22T23:23:58.280 INFO:tasks.workunit.client.1.smithi134.stdout:8/833: write d7/dc/d2c/d37/d47/ff4 [2639773,109293] 0 2024-04-22T23:23:58.286 INFO:tasks.workunit.client.1.smithi134.stdout:9/779: dwrite d0/d2d/d82/da0/f7f [0,4194304] 0 2024-04-22T23:23:58.289 INFO:tasks.workunit.client.1.smithi134.stdout:8/834: dread - d7/db/d42/d65/d46/d9a/da3/d88/da6/fd7 zero size 2024-04-22T23:23:58.289 INFO:tasks.workunit.client.1.smithi134.stdout:7/910: getdents d4/d8/d41 0 2024-04-22T23:23:58.290 INFO:tasks.workunit.client.1.smithi134.stdout:5/943: getdents d9/dbb/d11b/da6 0 2024-04-22T23:23:58.292 INFO:tasks.workunit.client.1.smithi134.stdout:7/911: write d4/d8/d3b/fd1 [1236852,43450] 0 2024-04-22T23:23:58.292 INFO:tasks.workunit.client.1.smithi134.stdout:3/652: creat dc/d2a/d62/db2/d2f/d36/d6d/d6f/dd2/ff5 x:0 0 0 2024-04-22T23:23:58.294 INFO:tasks.workunit.client.1.smithi134.stdout:6/924: creat d0/d19/de/f131 x:0 0 0 2024-04-22T23:23:58.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:57 smithi067 ceph-mon[26453]: Deploying cephadm binary to smithi067 2024-04-22T23:23:58.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:57 smithi067 ceph-mon[26453]: Deploying cephadm binary to smithi134 2024-04-22T23:23:58.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:57 smithi067 ceph-mon[26453]: mgrmap e22: smithi134.ccxkcj(active, since 1.01105s) 2024-04-22T23:23:58.295 INFO:tasks.workunit.client.1.smithi134.stdout:7/912: chown d4/d8/d41/d76/dc1/d2f/d34/d62/d84/fee 63 1 2024-04-22T23:23:58.296 INFO:tasks.workunit.client.1.smithi134.stdout:2/788: creat d2/d15/f10d x:0 0 0 2024-04-22T23:23:58.298 INFO:tasks.workunit.client.1.smithi134.stdout:2/789: write d2/dd/df/d1c/ff0 [152824,77589] 0 2024-04-22T23:23:58.299 INFO:tasks.workunit.client.1.smithi134.stdout:2/790: stat d2/d42/d54/d33/d6e/fe1 0 2024-04-22T23:23:58.303 INFO:tasks.workunit.client.1.smithi134.stdout:6/925: creat d0/d6f/d88/d8b/f132 x:0 0 0 2024-04-22T23:23:58.305 INFO:tasks.workunit.client.1.smithi134.stdout:9/780: rename d0/d2d/d94/d98/c9b to d0/d2d/c107 0 2024-04-22T23:23:58.308 INFO:tasks.workunit.client.1.smithi134.stdout:2/791: rmdir d2/d15/d24/d1d/d3d/d9a 39 2024-04-22T23:23:58.309 INFO:tasks.workunit.client.1.smithi134.stdout:6/926: truncate d0/d19/de/d2f/f71 2963997 0 2024-04-22T23:23:58.313 INFO:tasks.workunit.client.1.smithi134.stdout:7/913: write d4/d8/d41/d76/dc1/d2f/d34/d54/ff9 [497810,2362] 0 2024-04-22T23:23:58.314 INFO:tasks.workunit.client.1.smithi134.stdout:7/914: truncate d4/d8/d101/f10a 140251 0 2024-04-22T23:23:58.316 INFO:tasks.workunit.client.1.smithi134.stdout:7/915: chown d4/d8/d41/d76/dc1/d2f/d69/df8/l109 38 1 2024-04-22T23:23:58.317 INFO:tasks.workunit.client.1.smithi134.stdout:3/653: rmdir dc/d2a/d62/db2/d34/dd6 0 2024-04-22T23:23:58.318 INFO:tasks.workunit.client.1.smithi134.stdout:3/654: write dc/d2a/d62/f7f [318703,101691] 0 2024-04-22T23:23:58.324 INFO:tasks.workunit.client.1.smithi134.stdout:8/835: rename d7/db/lb6 to d7/db/d42/d65/dd9/dda/de4/l106 0 2024-04-22T23:23:58.327 INFO:tasks.workunit.client.1.smithi134.stdout:2/792: dwrite d2/d15/d24/d1d/d3d/f58 [4194304,4194304] 0 2024-04-22T23:23:58.329 INFO:tasks.workunit.client.1.smithi134.stdout:9/781: creat d0/d7/d12/de8/df7/f108 x:0 0 0 2024-04-22T23:23:58.333 INFO:tasks.workunit.client.1.smithi134.stdout:7/916: creat d4/d8/d41/d76/dc1/d2f/d34/d62/d29/f145 x:0 0 0 2024-04-22T23:23:58.336 INFO:tasks.workunit.client.1.smithi134.stdout:3/655: unlink dc/d2a/d62/db2/d2f/d21/d42/f82 0 2024-04-22T23:23:58.337 INFO:tasks.workunit.client.1.smithi134.stdout:6/927: mknod d0/d19/de/d12/d5e/dc1/dcd/c133 0 2024-04-22T23:23:58.344 INFO:tasks.workunit.client.1.smithi134.stdout:2/793: truncate d2/d15/d24/f1e 4891599 0 2024-04-22T23:23:58.347 INFO:tasks.workunit.client.1.smithi134.stdout:8/836: getdents d7/dc/d2c/d37/d47 0 2024-04-22T23:23:58.349 INFO:tasks.workunit.client.1.smithi134.stdout:2/794: mknod d2/d42/d54/d33/c10e 0 2024-04-22T23:23:58.352 INFO:tasks.workunit.client.1.smithi134.stdout:2/795: rename d2/dd/df/d9c/dca/fd3 to d2/dd/d68/f10f 0 2024-04-22T23:23:58.353 INFO:tasks.workunit.client.1.smithi134.stdout:8/837: dwrite d7/dc/d2c/d4f/d6c/fec [0,4194304] 0 2024-04-22T23:23:58.365 INFO:tasks.workunit.client.1.smithi134.stdout:7/917: write d4/d8/d41/d76/dc1/d2f/d69/d8a/f99 [1600397,23205] 0 2024-04-22T23:23:58.367 INFO:tasks.workunit.client.1.smithi134.stdout:9/782: dwrite d0/d7/d27/fa5 [0,4194304] 0 2024-04-22T23:23:58.372 INFO:tasks.workunit.client.1.smithi134.stdout:2/796: dread d2/dd/d4a/f52 [0,4194304] 0 2024-04-22T23:23:58.374 INFO:tasks.workunit.client.1.smithi134.stdout:3/656: dwrite dc/d2a/d62/db2/d2f/d36/f5d [0,4194304] 0 2024-04-22T23:23:58.377 INFO:tasks.workunit.client.1.smithi134.stdout:9/783: chown d0/d2d/d3d/l63 38794153 1 2024-04-22T23:23:58.378 INFO:tasks.workunit.client.1.smithi134.stdout:7/918: creat d4/d8/d41/d76/dc1/d2f/d34/d62/d29/dc5/f146 x:0 0 0 2024-04-22T23:23:58.383 INFO:tasks.workunit.client.1.smithi134.stdout:7/919: write d4/d8/d41/d76/dc1/d2f/d34/f67 [989370,31340] 0 2024-04-22T23:23:58.384 INFO:tasks.workunit.client.1.smithi134.stdout:3/657: mkdir dc/d2a/d62/d86/df6 0 2024-04-22T23:23:58.385 INFO:tasks.workunit.client.1.smithi134.stdout:3/658: fdatasync dc/d2a/d62/db2/f5c 0 2024-04-22T23:23:58.387 INFO:tasks.workunit.client.1.smithi134.stdout:9/784: rename d0/d7/d23/d26/d4d/d5d/d8e/dae/l104 to d0/d7/d23/d1b/dc5/l109 0 2024-04-22T23:23:58.388 INFO:tasks.workunit.client.1.smithi134.stdout:8/838: creat d7/db/d42/d65/f107 x:0 0 0 2024-04-22T23:23:58.392 INFO:tasks.workunit.client.1.smithi134.stdout:7/920: read d4/d8/d41/d76/dc1/d2f/d69/fe9 [98794,103695] 0 2024-04-22T23:23:58.394 INFO:tasks.workunit.client.1.smithi134.stdout:8/839: mknod d7/dc/d2c/d4f/d6c/dd1/c108 0 2024-04-22T23:23:58.394 INFO:tasks.workunit.client.1.smithi134.stdout:3/659: dwrite dc/d2a/d62/db2/d3b/fb0 [0,4194304] 0 2024-04-22T23:23:58.398 INFO:tasks.workunit.client.1.smithi134.stdout:9/785: rename d0/f75 to d0/d37/f10a 0 2024-04-22T23:23:58.401 INFO:tasks.workunit.client.1.smithi134.stdout:9/786: chown d0/d7/d27/f33 186517 1 2024-04-22T23:23:58.404 INFO:tasks.workunit.client.1.smithi134.stdout:2/797: dread d2/d15/d24/d1d/f36 [0,4194304] 0 2024-04-22T23:23:58.409 INFO:tasks.workunit.client.1.smithi134.stdout:7/921: link d4/d8/d41/d76/dc1/d2f/d70/d86/ff0 d4/d8/d41/d76/dc1/d2f/d34/d62/d29/dc5/f147 0 2024-04-22T23:23:58.412 INFO:tasks.workunit.client.1.smithi134.stdout:7/922: read d4/d8/d41/f43 [382679,44459] 0 2024-04-22T23:23:58.416 INFO:tasks.workunit.client.1.smithi134.stdout:9/787: dread d0/d7/d12/f30 [0,4194304] 0 2024-04-22T23:23:58.423 INFO:tasks.workunit.client.1.smithi134.stdout:7/923: mknod d4/d8/d41/d5b/d104/d140/c148 0 2024-04-22T23:23:58.425 INFO:tasks.workunit.client.1.smithi134.stdout:9/788: dwrite d0/d7/ffc [0,4194304] 0 2024-04-22T23:23:58.427 INFO:tasks.workunit.client.1.smithi134.stdout:2/798: dread d2/dd/df/f6f [0,4194304] 0 2024-04-22T23:23:58.429 INFO:tasks.workunit.client.1.smithi134.stdout:2/799: chown d2/dd/df/d9c/dca 263568305 1 2024-04-22T23:23:58.430 INFO:tasks.workunit.client.1.smithi134.stdout:3/660: dwrite dc/d2a/fe5 [0,4194304] 0 2024-04-22T23:23:58.432 INFO:tasks.workunit.client.1.smithi134.stdout:7/924: creat d4/d8/d41/f149 x:0 0 0 2024-04-22T23:23:58.436 INFO:tasks.workunit.client.1.smithi134.stdout:2/800: mknod d2/dd/df/d9c/daa/c110 0 2024-04-22T23:23:58.437 INFO:tasks.workunit.client.1.smithi134.stdout:2/801: chown d2/d15/d24/f6d 32733229 1 2024-04-22T23:23:58.437 INFO:tasks.workunit.client.1.smithi134.stdout:3/661: mkdir dc/d2a/d62/d86/dc1/df7 0 2024-04-22T23:23:58.440 INFO:tasks.workunit.client.1.smithi134.stdout:2/802: write d2/dd/df/f85 [964884,26589] 0 2024-04-22T23:23:58.441 INFO:tasks.workunit.client.1.smithi134.stdout:2/803: chown d2/dd/dc9 244 1 2024-04-22T23:23:58.443 INFO:tasks.workunit.client.1.smithi134.stdout:3/662: symlink dc/d2a/d62/db2/d34/d72/lf8 0 2024-04-22T23:23:58.445 INFO:tasks.workunit.client.1.smithi134.stdout:9/789: creat d0/d7/f10b x:0 0 0 2024-04-22T23:23:58.450 INFO:tasks.workunit.client.1.smithi134.stdout:9/790: creat d0/d7/d23/d26/d51/d88/db9/f10c x:0 0 0 2024-04-22T23:23:58.452 INFO:tasks.workunit.client.1.smithi134.stdout:9/791: chown d0/d2d/c107 3511142 1 2024-04-22T23:23:58.455 INFO:tasks.workunit.client.1.smithi134.stdout:9/792: mkdir d0/d2d/d82/df5/d10d 0 2024-04-22T23:23:58.455 INFO:tasks.workunit.client.1.smithi134.stdout:7/925: write d4/d4a/d85/dd7/fd9 [3082353,12879] 0 2024-04-22T23:23:58.458 INFO:tasks.workunit.client.1.smithi134.stdout:2/804: write d2/dd/d72/fab [851044,123075] 0 2024-04-22T23:23:58.462 INFO:tasks.workunit.client.1.smithi134.stdout:2/805: symlink d2/d42/d54/dfd/l111 0 2024-04-22T23:23:58.464 INFO:tasks.workunit.client.1.smithi134.stdout:2/806: chown d2/d15/d24/d1d/f5a 6513036 1 2024-04-22T23:23:58.466 INFO:tasks.workunit.client.1.smithi134.stdout:3/663: dwrite dc/d2a/d62/db2/d2f/f91 [0,4194304] 0 2024-04-22T23:23:58.466 INFO:tasks.workunit.client.1.smithi134.stdout:7/926: mknod d4/d8/d41/d76/dc1/d2f/d69/d8a/d126/c14a 0 2024-04-22T23:23:58.475 INFO:tasks.workunit.client.1.smithi134.stdout:2/807: dwrite d2/dd/d72/fab [0,4194304] 0 2024-04-22T23:23:58.479 INFO:tasks.workunit.client.1.smithi134.stdout:2/808: read d2/d42/d54/f103 [3530077,6432] 0 2024-04-22T23:23:58.482 INFO:tasks.workunit.client.1.smithi134.stdout:2/809: creat d2/d42/d54/d109/f112 x:0 0 0 2024-04-22T23:23:58.484 INFO:tasks.workunit.client.1.smithi134.stdout:2/810: rmdir d2 39 2024-04-22T23:23:58.492 INFO:tasks.workunit.client.1.smithi134.stdout:3/664: write dc/d2a/d62/db2/d2f/d21/d42/f47 [575479,74903] 0 2024-04-22T23:23:58.497 INFO:tasks.workunit.client.1.smithi134.stdout:2/811: write d2/d42/d54/d33/f3a [1981995,68322] 0 2024-04-22T23:23:58.499 INFO:tasks.workunit.client.1.smithi134.stdout:7/927: dread d4/d8/d41/d76/dc1/d2f/d69/d8a/f9d [0,4194304] 0 2024-04-22T23:23:58.504 INFO:tasks.workunit.client.1.smithi134.stdout:3/665: rename dc/d2a/d62/db2/d2f/f68 to dc/d2a/d62/db2/d2f/d36/ff9 0 2024-04-22T23:23:58.505 INFO:tasks.workunit.client.1.smithi134.stdout:3/666: chown dc/d2a/d62/db2/d2f/d54/f5e 5543 1 2024-04-22T23:23:58.506 INFO:tasks.workunit.client.1.smithi134.stdout:7/928: mknod d4/d8/d41/d76/dc1/d2f/d70/c14b 0 2024-04-22T23:23:58.507 INFO:tasks.workunit.client.1.smithi134.stdout:2/812: write d2/fc1 [1183793,94216] 0 2024-04-22T23:23:58.509 INFO:tasks.workunit.client.1.smithi134.stdout:7/929: dread d4/f31 [4194304,4194304] 0 2024-04-22T23:23:58.517 INFO:tasks.workunit.client.1.smithi134.stdout:3/667: symlink dc/d2a/d62/db2/d34/d6b/dae/def/lfa 0 2024-04-22T23:23:58.527 INFO:tasks.workunit.client.1.smithi134.stdout:7/930: creat d4/d4a/f14c x:0 0 0 2024-04-22T23:23:58.528 INFO:tasks.workunit.client.1.smithi134.stdout:2/813: getdents d2/d42/d54/ddd 0 2024-04-22T23:23:58.531 INFO:tasks.workunit.client.1.smithi134.stdout:3/668: dread dc/d2a/d62/db2/d2f/f51 [0,4194304] 0 2024-04-22T23:23:58.533 INFO:tasks.workunit.client.1.smithi134.stdout:3/669: chown dc/d2a/d62/db2/d34/d6b/fe9 0 1 2024-04-22T23:23:58.534 INFO:tasks.workunit.client.1.smithi134.stdout:7/931: write d4/d8/d41/d76/dc1/d2f/d70/f72 [1036267,62096] 0 2024-04-22T23:23:58.534 INFO:tasks.workunit.client.1.smithi134.stdout:3/670: readlink dc/d2a/d62/db2/d3b/l4d 0 2024-04-22T23:23:58.536 INFO:tasks.workunit.client.1.smithi134.stdout:3/671: read - dc/d2a/d62/db2/d2f/d54/dac/fed zero size 2024-04-22T23:23:58.537 INFO:tasks.workunit.client.1.smithi134.stdout:7/932: readlink d4/d8/d41/d76/dc1/d2f/d34/d62/d136/l13a 0 2024-04-22T23:23:58.539 INFO:tasks.workunit.client.1.smithi134.stdout:3/672: mkdir dc/d2a/d62/db2/d2f/d21/dec/dcf/dfb 0 2024-04-22T23:23:58.545 INFO:tasks.workunit.client.1.smithi134.stdout:7/933: dwrite d4/d8/d41/d76/dc1/d2f/d34/dc0/f107 [0,4194304] 0 2024-04-22T23:23:58.559 INFO:tasks.workunit.client.1.smithi134.stdout:3/673: write dc/d2a/d62/db2/d2f/f51 [659541,28606] 0 2024-04-22T23:23:58.566 INFO:tasks.workunit.client.1.smithi134.stdout:3/674: getdents dc/d2a/d62/db2/d34 0 2024-04-22T23:23:58.569 INFO:tasks.workunit.client.1.smithi134.stdout:3/675: mkdir dc/d2a/d30/dfc 0 2024-04-22T23:23:58.570 INFO:tasks.workunit.client.1.smithi134.stdout:3/676: readlink dc/d2a/d62/db2/d3b/l97 0 2024-04-22T23:23:58.573 INFO:tasks.workunit.client.1.smithi134.stdout:3/677: mkdir dc/d2a/d62/db2/d3b/dfd 0 2024-04-22T23:23:58.576 INFO:tasks.workunit.client.1.smithi134.stdout:3/678: mkdir dc/d2a/d62/db2/d34/d6b/dfe 0 2024-04-22T23:23:58.579 INFO:tasks.workunit.client.1.smithi134.stdout:3/679: symlink dc/d2a/d62/d86/dc1/lff 0 2024-04-22T23:23:58.580 INFO:tasks.workunit.client.1.smithi134.stdout:3/680: write dc/d2a/d30/f60 [4396516,90784] 0 2024-04-22T23:23:58.584 INFO:tasks.workunit.client.1.smithi134.stdout:3/681: symlink dc/d2a/d62/db2/d2f/d61/d6a/l100 0 2024-04-22T23:23:58.586 INFO:tasks.workunit.client.1.smithi134.stdout:3/682: creat dc/d7d/f101 x:0 0 0 2024-04-22T23:23:58.587 INFO:tasks.workunit.client.1.smithi134.stdout:3/683: dread - dc/d2a/d62/db2/d2f/d54/dac/fed zero size 2024-04-22T23:23:58.588 INFO:tasks.workunit.client.1.smithi134.stdout:3/684: dread - dc/d2a/d62/db2/d34/d6b/fe9 zero size 2024-04-22T23:23:58.591 INFO:tasks.workunit.client.1.smithi134.stdout:3/685: mknod dc/d2a/d62/db2/d2f/d36/d5f/c102 0 2024-04-22T23:23:58.593 INFO:tasks.workunit.client.1.smithi134.stdout:3/686: unlink dc/d2a/d62/db2/d2f/d54/f5e 0 2024-04-22T23:23:58.595 INFO:tasks.workunit.client.1.smithi134.stdout:3/687: write dc/d2a/d62/db2/d2f/d36/d6d/d80/fc0 [1827770,59497] 0 2024-04-22T23:23:58.597 INFO:tasks.workunit.client.1.smithi134.stdout:3/688: dread - dc/d7d/fad zero size 2024-04-22T23:23:58.605 INFO:tasks.workunit.client.1.smithi134.stdout:3/689: dread dc/d2a/d62/db2/d2f/d21/d42/d77/d8f/fbf [0,4194304] 0 2024-04-22T23:23:58.608 INFO:tasks.workunit.client.1.smithi134.stdout:3/690: mkdir dc/d2a/d62/d103 0 2024-04-22T23:23:58.610 INFO:tasks.workunit.client.1.smithi134.stdout:3/691: symlink dc/d2a/d62/db2/d34/d6b/l104 0 2024-04-22T23:23:58.617 INFO:tasks.workunit.client.1.smithi134.stdout:3/692: dwrite dc/d2a/d62/db2/d3b/d29/f2e [4194304,4194304] 0 2024-04-22T23:23:58.619 INFO:tasks.workunit.client.1.smithi134.stdout:3/693: write dc/d2a/d62/db2/d2f/d36/f5d [2584995,41381] 0 2024-04-22T23:23:58.624 INFO:tasks.workunit.client.1.smithi134.stdout:3/694: creat dc/d2a/d62/db2/d34/d6b/dae/f105 x:0 0 0 2024-04-22T23:23:58.630 INFO:tasks.workunit.client.1.smithi134.stdout:0/880: sync 2024-04-22T23:23:58.636 INFO:tasks.workunit.client.1.smithi134.stdout:0/881: truncate d1/d5/d58/d6b/db9/f6d 3448244 0 2024-04-22T23:23:58.643 INFO:tasks.workunit.client.1.smithi134.stdout:0/882: dread d1/d5/fc1 [0,4194304] 0 2024-04-22T23:23:58.648 INFO:tasks.workunit.client.1.smithi134.stdout:0/883: creat d1/d5/d58/d22/dd0/f11f x:0 0 0 2024-04-22T23:23:58.652 INFO:tasks.workunit.client.1.smithi134.stdout:0/884: symlink d1/d5/dd/d20/d2c/l120 0 2024-04-22T23:23:58.657 INFO:tasks.workunit.client.1.smithi134.stdout:0/885: creat d1/d2/d1d/f121 x:0 0 0 2024-04-22T23:23:58.662 INFO:tasks.workunit.client.1.smithi134.stdout:0/886: unlink d1/d5/d58/d6b/lcf 0 2024-04-22T23:23:58.669 INFO:tasks.workunit.client.1.smithi134.stdout:3/695: write dc/d2a/d62/db2/d2f/d21/d42/f66 [4845882,128601] 0 2024-04-22T23:23:58.670 INFO:tasks.workunit.client.1.smithi134.stdout:0/887: dwrite d1/d5/d15/fba [0,4194304] 0 2024-04-22T23:23:58.672 INFO:tasks.workunit.client.1.smithi134.stdout:0/888: readlink d1/d5/dd/d20/d2c/l120 0 2024-04-22T23:23:58.680 INFO:tasks.workunit.client.1.smithi134.stdout:0/889: rename d1/d2/d1d/c2f to d1/d5/d58/d6b/db9/d75/c122 0 2024-04-22T23:23:58.681 INFO:tasks.workunit.client.1.smithi134.stdout:0/890: chown d1/d2/d117 3696421 1 2024-04-22T23:23:58.699 INFO:tasks.workunit.client.1.smithi134.stdout:3/696: write dc/d2a/d62/db2/d34/d6b/f9a [407507,44390] 0 2024-04-22T23:23:58.704 INFO:tasks.workunit.client.1.smithi134.stdout:3/697: creat dc/d2a/d62/db2/d2f/d21/d42/d77/f106 x:0 0 0 2024-04-22T23:23:58.707 INFO:tasks.workunit.client.1.smithi134.stdout:3/698: rmdir dc/d2a/d62/db2/d34 39 2024-04-22T23:23:58.710 INFO:tasks.workunit.client.1.smithi134.stdout:0/891: dwrite d1/d5/d58/da0/fb6 [0,4194304] 0 2024-04-22T23:23:58.716 INFO:tasks.workunit.client.1.smithi134.stdout:3/699: mkdir dc/d2a/d62/db2/d3b/dfd/d107 0 2024-04-22T23:23:58.723 INFO:tasks.workunit.client.1.smithi134.stdout:0/892: dwrite d1/d5/d58/f80 [0,4194304] 0 2024-04-22T23:23:58.724 INFO:tasks.workunit.client.1.smithi134.stdout:3/700: dread dc/d2a/d62/db2/d3b/fb0 [0,4194304] 0 2024-04-22T23:23:58.739 INFO:tasks.workunit.client.1.smithi134.stdout:0/893: dwrite d1/d5/d58/da0/f7b [0,4194304] 0 2024-04-22T23:23:58.742 INFO:tasks.workunit.client.1.smithi134.stdout:0/894: stat d1/db/l116 0 2024-04-22T23:23:58.743 INFO:tasks.workunit.client.1.smithi134.stdout:0/895: dread - d1/d2/d1d/fc6 zero size 2024-04-22T23:23:58.747 INFO:tasks.workunit.client.1.smithi134.stdout:3/701: dread dc/d2a/d62/db2/f5c [0,4194304] 0 2024-04-22T23:23:58.748 INFO:tasks.workunit.client.1.smithi134.stdout:0/896: rmdir d1/d5/d58/d22/dcb 39 2024-04-22T23:23:58.749 INFO:tasks.workunit.client.1.smithi134.stdout:3/702: chown dc/d2a/d62/db2/d2f/d21/d42/f47 1 1 2024-04-22T23:23:58.750 INFO:tasks.workunit.client.1.smithi134.stdout:3/703: write dc/d2a/d62/db2/d2f/d21/d42/d77/fb6 [1177562,78850] 0 2024-04-22T23:23:58.751 INFO:tasks.workunit.client.1.smithi134.stdout:5/944: sync 2024-04-22T23:23:58.751 INFO:tasks.workunit.client.1.smithi134.stdout:6/928: sync 2024-04-22T23:23:58.751 INFO:tasks.workunit.client.1.smithi134.stdout:8/840: sync 2024-04-22T23:23:58.752 INFO:tasks.workunit.client.1.smithi134.stdout:6/929: chown d0/d19/db/dfa/f114 3 1 2024-04-22T23:23:58.758 INFO:tasks.workunit.client.1.smithi134.stdout:5/945: dwrite d9/dbb/d11b/df1/f11a [0,4194304] 0 2024-04-22T23:23:58.759 INFO:tasks.workunit.client.1.smithi134.stdout:5/946: chown d9/da2/de6 8968806 1 2024-04-22T23:23:58.759 INFO:tasks.workunit.client.1.smithi134.stdout:3/704: fdatasync dc/d2a/d62/db2/d2f/d54/f7b 0 2024-04-22T23:23:58.763 INFO:tasks.workunit.client.1.smithi134.stdout:6/930: symlink d0/d6f/d88/l134 0 2024-04-22T23:23:58.768 INFO:tasks.workunit.client.1.smithi134.stdout:3/705: truncate dc/f40 276890 0 2024-04-22T23:23:58.772 INFO:tasks.workunit.client.1.smithi134.stdout:5/947: creat d9/d11/d73/d95/d20/d108/f140 x:0 0 0 2024-04-22T23:23:58.774 INFO:tasks.workunit.client.1.smithi134.stdout:0/897: getdents d1/d5/dd/d41 0 2024-04-22T23:23:58.779 INFO:tasks.workunit.client.1.smithi134.stdout:5/948: symlink d9/dbb/d11b/d52/l141 0 2024-04-22T23:23:58.782 INFO:tasks.workunit.client.1.smithi134.stdout:8/841: link d7/lff d7/db/d2d/d39/d105/l109 0 2024-04-22T23:23:58.787 INFO:tasks.workunit.client.1.smithi134.stdout:3/706: truncate dc/d2a/d30/f60 3463626 0 2024-04-22T23:23:58.791 INFO:tasks.workunit.client.1.smithi134.stdout:8/842: rename d7/db/lb7 to d7/db/d42/d65/d46/l10a 0 2024-04-22T23:23:58.792 INFO:tasks.workunit.client.1.smithi134.stdout:0/898: getdents d1/d5/d58/d22/dcb 0 2024-04-22T23:23:58.794 INFO:tasks.workunit.client.1.smithi134.stdout:3/707: creat dc/d2a/d30/dfc/f108 x:0 0 0 2024-04-22T23:23:58.795 INFO:tasks.workunit.client.1.smithi134.stdout:0/899: symlink d1/d5/d58/d94/l123 0 2024-04-22T23:23:58.796 INFO:tasks.workunit.client.1.smithi134.stdout:0/900: chown d1/d2/cfa 58123125 1 2024-04-22T23:23:58.796 INFO:tasks.workunit.client.1.smithi134.stdout:8/843: creat d7/db/d42/d65/d46/d9a/da3/d88/da6/f10b x:0 0 0 2024-04-22T23:23:58.800 INFO:tasks.workunit.client.1.smithi134.stdout:5/949: dread d9/d11/d73/d95/d20/d60/f7d [0,4194304] 0 2024-04-22T23:23:58.801 INFO:tasks.workunit.client.1.smithi134.stdout:3/708: truncate dc/d2a/d62/db2/d2f/d36/d6d/f9c 313418 0 2024-04-22T23:23:58.802 INFO:tasks.workunit.client.1.smithi134.stdout:0/901: dread d1/d5/d58/f80 [0,4194304] 0 2024-04-22T23:23:58.804 INFO:tasks.workunit.client.1.smithi134.stdout:3/709: truncate dc/d2a/d62/db2/d2f/d36/d6d/ff4 1016623 0 2024-04-22T23:23:58.807 INFO:tasks.workunit.client.1.smithi134.stdout:5/950: rmdir d9/d11/d34/d4d 39 2024-04-22T23:23:58.808 INFO:tasks.workunit.client.1.smithi134.stdout:5/951: stat d9/dbb/d11b/d17/d2a/f98 0 2024-04-22T23:23:58.811 INFO:tasks.workunit.client.1.smithi134.stdout:0/902: symlink d1/d5/d58/d6b/db9/d75/l124 0 2024-04-22T23:23:58.812 INFO:tasks.workunit.client.1.smithi134.stdout:3/710: dwrite dc/d2a/d62/db2/d2f/d54/dac/fed [0,4194304] 0 2024-04-22T23:23:58.815 INFO:tasks.workunit.client.1.smithi134.stdout:8/844: link d7/f23 d7/f10c 0 2024-04-22T23:23:58.819 INFO:tasks.workunit.client.1.smithi134.stdout:0/903: mknod d1/d77/c125 0 2024-04-22T23:23:58.819 INFO:tasks.workunit.client.1.smithi134.stdout:3/711: dwrite dc/f52 [0,4194304] 0 2024-04-22T23:23:58.826 INFO:tasks.workunit.client.1.smithi134.stdout:8/845: creat d7/db/d42/d65/dd9/dda/de7/f10d x:0 0 0 2024-04-22T23:23:58.828 INFO:tasks.workunit.client.1.smithi134.stdout:3/712: dread dc/d2a/d62/db2/d2f/d54/dac/fed [0,4194304] 0 2024-04-22T23:23:58.834 INFO:tasks.workunit.client.1.smithi134.stdout:3/713: creat dc/d2a/d62/d86/dc1/f109 x:0 0 0 2024-04-22T23:23:58.835 INFO:tasks.workunit.client.1.smithi134.stdout:5/952: dwrite d9/d11/d34/d85/ffe [0,4194304] 0 2024-04-22T23:23:58.836 INFO:tasks.workunit.client.1.smithi134.stdout:3/714: readlink dc/d2a/d62/db2/d2f/d36/d6d/l81 0 2024-04-22T23:23:58.838 INFO:tasks.workunit.client.1.smithi134.stdout:5/953: dread - d9/d11/d34/d4d/dd4/d121/f126 zero size 2024-04-22T23:23:58.841 INFO:tasks.workunit.client.1.smithi134.stdout:3/715: write dc/dd/f4f [3188277,27817] 0 2024-04-22T23:23:58.847 INFO:tasks.workunit.client.1.smithi134.stdout:5/954: rename d9/d11/d73/d95/dae/ff2 to d9/d11/d34/d4d/d92/dce/f142 0 2024-04-22T23:23:58.847 INFO:tasks.workunit.client.1.smithi134.stdout:0/904: dread d1/d2/f38 [0,4194304] 0 2024-04-22T23:23:58.852 INFO:tasks.workunit.client.1.smithi134.stdout:3/716: rename dc/d2a/d62/d86/lb1 to dc/d2a/l10a 0 2024-04-22T23:23:58.857 INFO:tasks.workunit.client.1.smithi134.stdout:0/905: chown d1/d5/d58/d6b/db9/d75/c122 13 1 2024-04-22T23:23:58.858 INFO:tasks.workunit.client.1.smithi134.stdout:8/846: dwrite d7/db/d42/d65/d46/d9a/faa [0,4194304] 0 2024-04-22T23:23:58.858 INFO:tasks.workunit.client.1.smithi134.stdout:3/717: mkdir dc/d2a/d62/d86/d10b 0 2024-04-22T23:23:58.860 INFO:tasks.workunit.client.1.smithi134.stdout:3/718: chown dc/d2a/d62/db2/d3b/d29/ld1 29 1 2024-04-22T23:23:58.862 INFO:tasks.workunit.client.1.smithi134.stdout:0/906: symlink d1/d2/d1d/d72/l126 0 2024-04-22T23:23:58.866 INFO:tasks.workunit.client.1.smithi134.stdout:0/907: write d1/d5/d58/da0/fb6 [240655,78631] 0 2024-04-22T23:23:58.866 INFO:tasks.workunit.client.1.smithi134.stdout:8/847: fsync d7/db/d42/d65/dd9/dda/de7/f94 0 2024-04-22T23:23:58.868 INFO:tasks.workunit.client.1.smithi134.stdout:3/719: rename dc/d2a/d30/l46 to dc/d2a/d62/db2/d3b/l10c 0 2024-04-22T23:23:58.872 INFO:tasks.workunit.client.1.smithi134.stdout:0/908: mkdir d1/d5/dd/d20/d127 0 2024-04-22T23:23:58.877 INFO:tasks.workunit.client.1.smithi134.stdout:8/848: creat d7/dc/d19/dfa/f10e x:0 0 0 2024-04-22T23:23:58.878 INFO:tasks.workunit.client.1.smithi134.stdout:5/955: dwrite d9/dbb/d11b/d17/d2a/f8a [0,4194304] 0 2024-04-22T23:23:58.879 INFO:tasks.workunit.client.1.smithi134.stdout:0/909: mkdir d1/d5/d58/da0/d128 0 2024-04-22T23:23:58.882 INFO:tasks.workunit.client.1.smithi134.stdout:3/720: write dc/d2a/d62/db2/d34/d6b/fbe [1919025,72604] 0 2024-04-22T23:23:58.883 INFO:tasks.workunit.client.1.smithi134.stdout:8/849: fdatasync d7/db/d2d/f3e 0 2024-04-22T23:23:58.886 INFO:tasks.workunit.client.1.smithi134.stdout:3/721: dread dc/d2a/d62/db2/d2f/d21/d42/d77/d8f/fbf [0,4194304] 0 2024-04-22T23:23:58.888 INFO:tasks.workunit.client.1.smithi134.stdout:0/910: mkdir d1/d5/d58/d22/dd0/ddf/d129 0 2024-04-22T23:23:58.895 INFO:tasks.workunit.client.1.smithi134.stdout:0/911: symlink d1/d5/d58/d22/dcb/l12a 0 2024-04-22T23:23:58.903 INFO:tasks.workunit.client.1.smithi134.stdout:3/722: dwrite dc/d2a/d62/db2/d2f/d21/dec/dcf/fe1 [0,4194304] 0 2024-04-22T23:23:58.911 INFO:tasks.workunit.client.1.smithi134.stdout:8/850: dwrite d7/db/d2d/d39/d58/f97 [0,4194304] 0 2024-04-22T23:23:58.918 INFO:tasks.workunit.client.1.smithi134.stdout:5/956: dread d9/fe [0,4194304] 0 2024-04-22T23:23:58.918 INFO:tasks.workunit.client.1.smithi134.stdout:3/723: dwrite dc/d2a/d62/d86/dc1/fe2 [0,4194304] 0 2024-04-22T23:23:58.971 INFO:tasks.workunit.client.1.smithi134.stdout:8/851: creat d7/db/d42/d65/d46/d9a/da3/f10f x:0 0 0 2024-04-22T23:23:58.974 INFO:tasks.workunit.client.1.smithi134.stdout:5/957: write d9/dbb/d11b/d17/d44/fd0 [2034368,12552] 0 2024-04-22T23:23:58.976 INFO:tasks.workunit.client.1.smithi134.stdout:0/912: mkdir d1/d5/dd/d12b 0 2024-04-22T23:23:58.983 INFO:tasks.workunit.client.1.smithi134.stdout:5/958: dwrite d9/d11/d34/d4d/fde [0,4194304] 0 2024-04-22T23:23:58.985 INFO:tasks.workunit.client.1.smithi134.stdout:8/852: symlink d7/dc/d2c/d4f/d6c/dd1/l110 0 2024-04-22T23:23:58.988 INFO:tasks.workunit.client.1.smithi134.stdout:8/853: fsync d7/dc/d2c/d4f/ff0 0 2024-04-22T23:23:58.989 INFO:tasks.workunit.client.1.smithi134.stdout:0/913: dwrite d1/d5/d58/d22/dd0/ddf/f10b [0,4194304] 0 2024-04-22T23:23:58.997 INFO:tasks.workunit.client.1.smithi134.stdout:5/959: mkdir d9/d11/d34/d85/d143 0 2024-04-22T23:23:58.999 INFO:tasks.workunit.client.1.smithi134.stdout:8/854: mkdir d7/db/d42/d65/dd9/dda/de4/d111 0 2024-04-22T23:23:59.001 INFO:tasks.workunit.client.1.smithi134.stdout:5/960: rmdir d9/d11/d34/d4d/dd4/d121/d125 39 2024-04-22T23:23:59.005 INFO:tasks.workunit.client.1.smithi134.stdout:8/855: rename d7/db/d42/d65/d46/d9a/da3/d88/deb to d7/dc/d2c/d37/d112 0 2024-04-22T23:23:59.010 INFO:tasks.workunit.client.1.smithi134.stdout:8/856: stat d7/db/ld 0 2024-04-22T23:23:59.014 INFO:tasks.workunit.client.1.smithi134.stdout:9/793: sync 2024-04-22T23:23:59.017 INFO:tasks.workunit.client.1.smithi134.stdout:0/914: dread d1/d5/d58/d22/dd0/ddf/fda [0,4194304] 0 2024-04-22T23:23:59.019 INFO:tasks.workunit.client.1.smithi134.stdout:8/857: mknod d7/db/d2d/d39/d105/c113 0 2024-04-22T23:23:59.022 INFO:tasks.workunit.client.1.smithi134.stdout:0/915: dread d1/db/fa9 [0,4194304] 0 2024-04-22T23:23:59.023 INFO:tasks.workunit.client.1.smithi134.stdout:0/916: chown d1/d5/d58/c6a 23 1 2024-04-22T23:23:59.029 INFO:tasks.workunit.client.1.smithi134.stdout:3/724: dwrite dc/d2a/d62/db2/f3a [4194304,4194304] 0 2024-04-22T23:23:59.031 INFO:tasks.workunit.client.1.smithi134.stdout:3/725: write dc/f18 [7045747,39019] 0 2024-04-22T23:23:59.033 INFO:tasks.workunit.client.1.smithi134.stdout:9/794: dread d0/d37/f4f [0,4194304] 0 2024-04-22T23:23:59.042 INFO:tasks.workunit.client.1.smithi134.stdout:7/934: sync 2024-04-22T23:23:59.042 INFO:tasks.workunit.client.1.smithi134.stdout:2/814: sync 2024-04-22T23:23:59.042 INFO:tasks.workunit.client.1.smithi134.stdout:6/931: sync 2024-04-22T23:23:59.044 INFO:tasks.workunit.client.1.smithi134.stdout:5/961: truncate d9/d11/d34/d4d/ff3 543632 0 2024-04-22T23:23:59.045 INFO:tasks.workunit.client.1.smithi134.stdout:5/962: chown d9/d11/d34/d4d/d92/l106 516 1 2024-04-22T23:23:59.047 INFO:tasks.workunit.client.1.smithi134.stdout:3/726: link dc/d2a/d62/db2/d2f/d36/ff9 dc/d2a/d62/db2/d34/d6b/dae/def/f10d 0 2024-04-22T23:23:59.050 INFO:tasks.workunit.client.1.smithi134.stdout:3/727: dread - dc/d7d/f101 zero size 2024-04-22T23:23:59.050 INFO:tasks.workunit.client.1.smithi134.stdout:8/858: write d7/db/d42/d65/dd9/dda/de7/ddd/dee/ffe [333148,16949] 0 2024-04-22T23:23:59.051 INFO:tasks.workunit.client.1.smithi134.stdout:9/795: dwrite d0/d7/d23/d26/f58 [0,4194304] 0 2024-04-22T23:23:59.055 INFO:tasks.workunit.client.1.smithi134.stdout:8/859: write d7/db/d42/d65/dd9/dda/de7/fdb [898346,77340] 0 2024-04-22T23:23:59.055 INFO:tasks.workunit.client.1.smithi134.stdout:7/935: stat d4/d8/d3b/c129 0 2024-04-22T23:23:59.064 INFO:tasks.workunit.client.1.smithi134.stdout:5/963: creat d9/d11/d73/d95/d20/d117/f144 x:0 0 0 2024-04-22T23:23:59.064 INFO:tasks.workunit.client.1.smithi134.stdout:0/917: dwrite d1/d2/d1d/fc6 [0,4194304] 0 2024-04-22T23:23:59.064 INFO:tasks.workunit.client.1.smithi134.stdout:6/932: write d0/d19/db/d35/f4b [1113254,42827] 0 2024-04-22T23:23:59.070 INFO:tasks.workunit.client.1.smithi134.stdout:2/815: dread d2/dd/d4a/d73/f7f [0,4194304] 0 2024-04-22T23:23:59.071 INFO:tasks.workunit.client.1.smithi134.stdout:7/936: readlink d4/d8/d3b/l59 0 2024-04-22T23:23:59.073 INFO:tasks.workunit.client.1.smithi134.stdout:5/964: chown d9/dbb/d11b/d17/cf7 1348 1 2024-04-22T23:23:59.075 INFO:tasks.workunit.client.1.smithi134.stdout:5/965: write d9/dbb/d11b/d17/d44/fd0 [4536394,67629] 0 2024-04-22T23:23:59.077 INFO:tasks.workunit.client.1.smithi134.stdout:0/918: mkdir d1/d5/d12c 0 2024-04-22T23:23:59.078 INFO:tasks.workunit.client.1.smithi134.stdout:0/919: write d1/d2/df7/f105 [345893,15365] 0 2024-04-22T23:23:59.083 INFO:tasks.workunit.client.1.smithi134.stdout:9/796: mkdir d0/d2d/d94/d10e 0 2024-04-22T23:23:59.089 INFO:tasks.workunit.client.1.smithi134.stdout:2/816: fsync d2/d94/f100 0 2024-04-22T23:23:59.092 INFO:tasks.workunit.client.1.smithi134.stdout:9/797: dread - d0/d7/d23/d26/d51/feb zero size 2024-04-22T23:23:59.094 INFO:tasks.workunit.client.1.smithi134.stdout:0/920: mkdir d1/d5/d58/d94/dea/d12d 0 2024-04-22T23:23:59.103 INFO:tasks.workunit.client.1.smithi134.stdout:3/728: dread dc/d2a/d62/db2/d2f/f73 [0,4194304] 0 2024-04-22T23:23:59.105 INFO:tasks.workunit.client.1.smithi134.stdout:8/860: getdents d7/db/d42/d65/dd9/dda/de7/ddd/dee 0 2024-04-22T23:23:59.106 INFO:tasks.workunit.client.1.smithi134.stdout:6/933: truncate d0/d19/db/d33/d86/f11f 3756862 0 2024-04-22T23:23:59.107 INFO:tasks.workunit.client.1.smithi134.stdout:2/817: write d2/f2c [4681771,69047] 0 2024-04-22T23:23:59.108 INFO:tasks.workunit.client.1.smithi134.stdout:7/937: getdents d4/d8/d41 0 2024-04-22T23:23:59.113 INFO:tasks.workunit.client.1.smithi134.stdout:9/798: write d0/d7/d12/de8/fee [417765,99808] 0 2024-04-22T23:23:59.123 INFO:tasks.workunit.client.1.smithi134.stdout:7/938: mknod d4/dad/c14d 0 2024-04-22T23:23:59.126 INFO:tasks.workunit.client.1.smithi134.stdout:6/934: dwrite d0/d19/de/d1f/dd5/fda [0,4194304] 0 2024-04-22T23:23:59.129 INFO:tasks.workunit.client.1.smithi134.stdout:9/799: creat d0/d7/d23/d26/d4d/d5d/f10f x:0 0 0 2024-04-22T23:23:59.132 INFO:tasks.workunit.client.1.smithi134.stdout:3/729: link dc/d2a/d62/db2/d2f/d21/d42/d77/f106 dc/d2a/f10e 0 2024-04-22T23:23:59.138 INFO:tasks.workunit.client.1.smithi134.stdout:0/921: link d1/db/lec d1/d5/d12c/l12e 0 2024-04-22T23:23:59.145 INFO:tasks.workunit.client.1.smithi134.stdout:3/730: unlink dc/d2a/d30/l55 0 2024-04-22T23:23:59.146 INFO:tasks.workunit.client.1.smithi134.stdout:3/731: write dc/d2a/d62/db2/d34/d6b/f9a [947172,47558] 0 2024-04-22T23:23:59.147 INFO:tasks.workunit.client.1.smithi134.stdout:8/861: creat d7/db/f114 x:0 0 0 2024-04-22T23:23:59.149 INFO:tasks.workunit.client.1.smithi134.stdout:2/818: truncate d2/d15/d24/d1d/d3d/f58 1304418 0 2024-04-22T23:23:59.152 INFO:tasks.workunit.client.1.smithi134.stdout:0/922: rmdir d1/d5/d58/d6b 39 2024-04-22T23:23:59.153 INFO:tasks.workunit.client.1.smithi134.stdout:6/935: write d0/f9d [2188732,45440] 0 2024-04-22T23:23:59.159 INFO:tasks.workunit.client.1.smithi134.stdout:9/800: dwrite d0/d37/d76/f91 [8388608,4194304] 0 2024-04-22T23:23:59.166 INFO:tasks.workunit.client.1.smithi134.stdout:3/732: dwrite dc/d2a/d62/db2/d3b/fb0 [0,4194304] 0 2024-04-22T23:23:59.171 INFO:tasks.workunit.client.1.smithi134.stdout:6/936: unlink d0/d19/de/d1f/f2c 0 2024-04-22T23:23:59.173 INFO:tasks.workunit.client.1.smithi134.stdout:0/923: dread - d1/d5/d58/d6b/db9/d75/fb8 zero size 2024-04-22T23:23:59.174 INFO:tasks.workunit.client.1.smithi134.stdout:8/862: rename d7/dc/d2c/d37/d47/f31 to d7/db/d2d/d39/d58/f115 0 2024-04-22T23:23:59.178 INFO:tasks.workunit.client.1.smithi134.stdout:2/819: creat d2/d15/f113 x:0 0 0 2024-04-22T23:23:59.182 INFO:tasks.workunit.client.1.smithi134.stdout:0/924: unlink d1/d5/d58/da0/fee 0 2024-04-22T23:23:59.183 INFO:tasks.workunit.client.1.smithi134.stdout:8/863: fdatasync d7/db/d2d/f54 0 2024-04-22T23:23:59.185 INFO:tasks.workunit.client.1.smithi134.stdout:2/820: write d2/d15/d24/d1d/d3d/d7b/f7c [336620,65805] 0 2024-04-22T23:23:59.188 INFO:tasks.workunit.client.1.smithi134.stdout:3/733: creat dc/dd/f10f x:0 0 0 2024-04-22T23:23:59.189 INFO:tasks.workunit.client.1.smithi134.stdout:8/864: creat d7/dc/d2c/d4f/d6c/dd1/f116 x:0 0 0 2024-04-22T23:23:59.193 INFO:tasks.workunit.client.1.smithi134.stdout:0/925: creat d1/d5/d58/da0/d128/f12f x:0 0 0 2024-04-22T23:23:59.197 INFO:tasks.workunit.client.1.smithi134.stdout:8/865: unlink d7/db/d42/d65/c61 0 2024-04-22T23:23:59.200 INFO:tasks.workunit.client.1.smithi134.stdout:3/734: dwrite dc/d2a/d62/db2/d34/d6b/f9a [0,4194304] 0 2024-04-22T23:23:59.204 INFO:tasks.workunit.client.1.smithi134.stdout:9/801: dwrite d0/d7/d23/d1b/f64 [4194304,4194304] 0 2024-04-22T23:23:59.206 INFO:tasks.workunit.client.1.smithi134.stdout:6/937: dread d0/d19/de/f31 [0,4194304] 0 2024-04-22T23:23:59.209 INFO:tasks.workunit.client.1.smithi134.stdout:8/866: symlink d7/db/d42/d65/d46/l117 0 2024-04-22T23:23:59.210 INFO:tasks.workunit.client.1.smithi134.stdout:0/926: dwrite d1/d5/d58/d6b/db9/d75/f10a [0,4194304] 0 2024-04-22T23:23:59.212 INFO:tasks.workunit.client.1.smithi134.stdout:9/802: chown d0/d7/d23/d26/d4d/d5d/d8e/la3 692748968 1 2024-04-22T23:23:59.216 INFO:tasks.workunit.client.1.smithi134.stdout:6/938: write d0/d19/de/d12/d29/db5/fcf [4966859,54019] 0 2024-04-22T23:23:59.220 INFO:tasks.workunit.client.1.smithi134.stdout:3/735: dread dc/d2a/d62/db2/d3b/d29/fb7 [0,4194304] 0 2024-04-22T23:23:59.221 INFO:tasks.workunit.client.1.smithi134.stdout:0/927: mkdir d1/db/d130 0 2024-04-22T23:23:59.224 INFO:tasks.workunit.client.1.smithi134.stdout:8/867: symlink d7/db/d2d/l118 0 2024-04-22T23:23:59.224 INFO:tasks.workunit.client.1.smithi134.stdout:0/928: fsync d1/d5/d58/d94/f108 0 2024-04-22T23:23:59.225 INFO:tasks.workunit.client.1.smithi134.stdout:8/868: chown d7/dc/f7f 688018217 1 2024-04-22T23:23:59.227 INFO:tasks.workunit.client.1.smithi134.stdout:2/821: dwrite d2/d42/d54/d33/d6e/fe0 [0,4194304] 0 2024-04-22T23:23:59.227 INFO:tasks.workunit.client.1.smithi134.stdout:6/939: rename d0/d19/de/d12/d5e/d6d/l106 to d0/d19/db/d35/d8a/l135 0 2024-04-22T23:23:59.229 INFO:tasks.workunit.client.1.smithi134.stdout:8/869: write d7/db/d42/d65/d46/fdf [5181724,18358] 0 2024-04-22T23:23:59.232 INFO:tasks.workunit.client.1.smithi134.stdout:8/870: chown d7/db/d2d/d39/f3d 4814288 1 2024-04-22T23:23:59.237 INFO:tasks.workunit.client.1.smithi134.stdout:2/822: dwrite d2/d15/d24/f29 [0,4194304] 0 2024-04-22T23:23:59.239 INFO:tasks.workunit.client.1.smithi134.stdout:6/940: mkdir d0/d19/de/d12/d5e/dc1/d136 0 2024-04-22T23:23:59.241 INFO:tasks.workunit.client.1.smithi134.stdout:2/823: write d2/dd/df/d9c/daa/ff4 [328051,10771] 0 2024-04-22T23:23:59.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:58 smithi134 ceph-mon[34609]: pgmap v3: 65 pgs: 65 active+clean; 2.4 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail 2024-04-22T23:23:59.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:58 smithi134 ceph-mon[34609]: [22/Apr/2024:23:23:58] ENGINE Bus STARTING 2024-04-22T23:23:59.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:58 smithi134 ceph-mon[34609]: [22/Apr/2024:23:23:58] ENGINE Serving on https://172.21.15.134:7150 2024-04-22T23:23:59.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:58 smithi134 ceph-mon[34609]: [22/Apr/2024:23:23:58] ENGINE Serving on http://172.21.15.134:8765 2024-04-22T23:23:59.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:58 smithi134 ceph-mon[34609]: [22/Apr/2024:23:23:58] ENGINE Bus STARTED 2024-04-22T23:23:59.251 INFO:tasks.workunit.client.1.smithi134.stdout:6/941: dwrite d0/d19/de/d12/d5e/d6d/d7a/d99/fea [0,4194304] 0 2024-04-22T23:23:59.252 INFO:tasks.workunit.client.1.smithi134.stdout:2/824: mknod d2/d42/d54/d33/d6e/c114 0 2024-04-22T23:23:59.256 INFO:tasks.workunit.client.1.smithi134.stdout:0/929: creat d1/d5/dd/d68/f131 x:0 0 0 2024-04-22T23:23:59.259 INFO:tasks.workunit.client.1.smithi134.stdout:6/942: rmdir d0/d19/de/d12/d5e/d6d/dd3 39 2024-04-22T23:23:59.260 INFO:tasks.workunit.client.1.smithi134.stdout:3/736: dwrite dc/d2a/d62/db2/d2f/d21/d42/ff1 [4194304,4194304] 0 2024-04-22T23:23:59.261 INFO:tasks.workunit.client.1.smithi134.stdout:6/943: write d0/d6f/f125 [412632,1799] 0 2024-04-22T23:23:59.263 INFO:tasks.workunit.client.1.smithi134.stdout:2/825: rename d2/l61 to d2/dd/df9/l115 0 2024-04-22T23:23:59.266 INFO:tasks.workunit.client.1.smithi134.stdout:0/930: symlink d1/d5/dd/d68/l132 0 2024-04-22T23:23:59.267 INFO:tasks.workunit.client.1.smithi134.stdout:0/931: readlink d1/d5/dd/d68/lb0 0 2024-04-22T23:23:59.268 INFO:tasks.workunit.client.1.smithi134.stdout:3/737: write dc/d2a/d62/db2/d3b/d29/fb7 [1256484,72831] 0 2024-04-22T23:23:59.273 INFO:tasks.workunit.client.1.smithi134.stdout:2/826: mkdir d2/dd/d116 0 2024-04-22T23:23:59.273 INFO:tasks.workunit.client.1.smithi134.stdout:0/932: dread - d1/d5/dd/ff3 zero size 2024-04-22T23:23:59.274 INFO:tasks.workunit.client.1.smithi134.stdout:0/933: stat d1/d5/d12c 0 2024-04-22T23:23:59.275 INFO:tasks.workunit.client.1.smithi134.stdout:6/944: rename d0/d19/de/d12/d2a to d0/d19/de/d12/d2a/d2d/d3a/d98/df5/d137 22 2024-04-22T23:23:59.279 INFO:tasks.workunit.client.1.smithi134.stdout:3/738: symlink dc/d2a/d62/d86/df6/l110 0 2024-04-22T23:23:59.280 INFO:tasks.workunit.client.1.smithi134.stdout:6/945: creat d0/d19/de/d2f/f138 x:0 0 0 2024-04-22T23:23:59.280 INFO:tasks.workunit.client.1.smithi134.stdout:3/739: readlink dc/d2a/d62/db2/d34/l7e 0 2024-04-22T23:23:59.283 INFO:tasks.workunit.client.1.smithi134.stdout:3/740: creat dc/d7d/f111 x:0 0 0 2024-04-22T23:23:59.289 INFO:tasks.workunit.client.1.smithi134.stdout:3/741: write dc/d2a/d62/db2/d2f/d54/dac/fed [2809878,66196] 0 2024-04-22T23:23:59.292 INFO:tasks.workunit.client.1.smithi134.stdout:0/934: dread d1/d5/d58/f29 [0,4194304] 0 2024-04-22T23:23:59.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:58 smithi067 ceph-mon[26453]: pgmap v3: 65 pgs: 65 active+clean; 2.4 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail 2024-04-22T23:23:59.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:58 smithi067 ceph-mon[26453]: [22/Apr/2024:23:23:58] ENGINE Bus STARTING 2024-04-22T23:23:59.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:58 smithi067 ceph-mon[26453]: [22/Apr/2024:23:23:58] ENGINE Serving on https://172.21.15.134:7150 2024-04-22T23:23:59.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:58 smithi067 ceph-mon[26453]: [22/Apr/2024:23:23:58] ENGINE Serving on http://172.21.15.134:8765 2024-04-22T23:23:59.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:58 smithi067 ceph-mon[26453]: [22/Apr/2024:23:23:58] ENGINE Bus STARTED 2024-04-22T23:23:59.295 INFO:tasks.workunit.client.1.smithi134.stdout:3/742: creat dc/d2a/d62/d86/dc1/df7/f112 x:0 0 0 2024-04-22T23:23:59.296 INFO:tasks.workunit.client.1.smithi134.stdout:2/827: write d2/d15/d24/f1e [3992430,72506] 0 2024-04-22T23:23:59.298 INFO:tasks.workunit.client.1.smithi134.stdout:2/828: write d2/dd/df/d1c/de3/f108 [330167,15592] 0 2024-04-22T23:23:59.298 INFO:tasks.workunit.client.1.smithi134.stdout:3/743: mknod dc/d2a/c113 0 2024-04-22T23:23:59.305 INFO:tasks.workunit.client.1.smithi134.stdout:2/829: write d2/d15/d24/d3b/d8a/dc7/fe9 [258846,60928] 0 2024-04-22T23:23:59.306 INFO:tasks.workunit.client.1.smithi134.stdout:6/946: dwrite d0/d19/de/d12/d2a/fb7 [0,4194304] 0 2024-04-22T23:23:59.306 INFO:tasks.workunit.client.1.smithi134.stdout:3/744: symlink dc/d2a/d62/db2/d2f/d36/l114 0 2024-04-22T23:23:59.313 INFO:tasks.workunit.client.1.smithi134.stdout:3/745: mkdir dc/d2a/d62/db2/d2f/d54/dac/d115 0 2024-04-22T23:23:59.318 INFO:tasks.workunit.client.1.smithi134.stdout:3/746: symlink dc/d2a/d62/l116 0 2024-04-22T23:23:59.320 INFO:tasks.workunit.client.1.smithi134.stdout:2/830: link d2/dd/df/d9c/dca/c106 d2/dd/df/d1c/c117 0 2024-04-22T23:23:59.321 INFO:tasks.workunit.client.1.smithi134.stdout:2/831: write d2/f2c [7084485,84169] 0 2024-04-22T23:23:59.327 INFO:tasks.workunit.client.1.smithi134.stdout:3/747: dwrite dc/d2a/d62/db2/d2f/d36/d6d/d6f/dd9/fe0 [0,4194304] 0 2024-04-22T23:23:59.333 INFO:tasks.workunit.client.1.smithi134.stdout:2/832: write d2/d94/fd4 [646104,62950] 0 2024-04-22T23:23:59.335 INFO:tasks.workunit.client.1.smithi134.stdout:6/947: dread d0/d19/de/d12/d2a/f4c [0,4194304] 0 2024-04-22T23:23:59.336 INFO:tasks.workunit.client.1.smithi134.stdout:6/948: fsync d0/d19/db/d33/f64 0 2024-04-22T23:23:59.337 INFO:tasks.workunit.client.1.smithi134.stdout:6/949: readlink d0/d19/de/d12/d5e/lb4 0 2024-04-22T23:23:59.338 INFO:tasks.workunit.client.1.smithi134.stdout:3/748: creat dc/d2a/d62/db2/d2f/d61/d6a/f117 x:0 0 0 2024-04-22T23:23:59.338 INFO:tasks.workunit.client.1.smithi134.stdout:3/749: read dc/d2a/d62/db2/d2f/f73 [1678536,57784] 0 2024-04-22T23:23:59.340 INFO:tasks.workunit.client.1.smithi134.stdout:6/950: fdatasync d0/d4f/fb3 0 2024-04-22T23:23:59.342 INFO:tasks.workunit.client.1.smithi134.stdout:2/833: creat d2/d15/f118 x:0 0 0 2024-04-22T23:23:59.346 INFO:tasks.workunit.client.1.smithi134.stdout:3/750: dread dc/d2a/d62/db2/d2f/d21/d42/d77/fb6 [0,4194304] 0 2024-04-22T23:23:59.349 INFO:tasks.workunit.client.1.smithi134.stdout:3/751: chown dc/d2a/d62/db2/d2f/d61/d6a/cca 1066 1 2024-04-22T23:23:59.352 INFO:tasks.workunit.client.1.smithi134.stdout:2/834: rename d2/dd/df/l63 to d2/d42/d54/d33/d6e/l119 0 2024-04-22T23:23:59.354 INFO:tasks.workunit.client.1.smithi134.stdout:6/951: link d0/d19/de/d1f/dd5/f109 d0/d19/de/f139 0 2024-04-22T23:23:59.356 INFO:tasks.workunit.client.1.smithi134.stdout:2/835: mknod d2/dd/df/d9c/dca/c11a 0 2024-04-22T23:23:59.356 INFO:tasks.workunit.client.1.smithi134.stdout:3/752: dwrite dc/d2a/d62/f7f [0,4194304] 0 2024-04-22T23:23:59.356 INFO:tasks.workunit.client.1.smithi134.stdout:6/952: write d0/d19/de/dbe/f10b [725865,19565] 0 2024-04-22T23:23:59.359 INFO:tasks.workunit.client.1.smithi134.stdout:2/836: fdatasync d2/d15/d24/f27 0 2024-04-22T23:23:59.360 INFO:tasks.workunit.client.1.smithi134.stdout:5/966: sync 2024-04-22T23:23:59.361 INFO:tasks.workunit.client.1.smithi134.stdout:3/753: mknod dc/d2a/d62/d86/df6/c118 0 2024-04-22T23:23:59.361 INFO:tasks.workunit.client.1.smithi134.stdout:6/953: creat d0/d19/de/d12/d5e/d6d/d7a/d99/f13a x:0 0 0 2024-04-22T23:23:59.362 INFO:tasks.workunit.client.1.smithi134.stdout:5/967: dread - d9/d11/d73/d95/dae/de0/f12d zero size 2024-04-22T23:23:59.362 INFO:tasks.workunit.client.1.smithi134.stdout:3/754: read - dc/dd/f10f zero size 2024-04-22T23:23:59.370 INFO:tasks.workunit.client.1.smithi134.stdout:6/954: truncate d0/d19/de/f1d 4852837 0 2024-04-22T23:23:59.372 INFO:tasks.workunit.client.1.smithi134.stdout:2/837: link d2/d42/d54/d33/d6e/fe1 d2/d15/d24/d1d/dba/dea/f11b 0 2024-04-22T23:23:59.373 INFO:tasks.workunit.client.1.smithi134.stdout:2/838: readlink d2/dd/l67 0 2024-04-22T23:23:59.374 INFO:tasks.workunit.client.1.smithi134.stdout:3/755: mknod dc/d2a/d62/db2/d2f/d21/dec/c119 0 2024-04-22T23:23:59.378 INFO:tasks.workunit.client.1.smithi134.stdout:3/756: mknod dc/d2a/d62/db2/d2f/d54/dac/c11a 0 2024-04-22T23:23:59.382 INFO:tasks.workunit.client.1.smithi134.stdout:5/968: dwrite d9/dbb/d11b/f4c [0,4194304] 0 2024-04-22T23:23:59.384 INFO:tasks.workunit.client.1.smithi134.stdout:3/757: stat dc/d2a/d62/db2/d2f/d61/l94 0 2024-04-22T23:23:59.384 INFO:tasks.workunit.client.1.smithi134.stdout:5/969: chown d9/dbb/d11b/d88/ffb 40778 1 2024-04-22T23:23:59.384 INFO:tasks.workunit.client.1.smithi134.stdout:2/839: write d2/dd/df/dc2/fcf [1021903,107773] 0 2024-04-22T23:23:59.385 INFO:tasks.workunit.client.1.smithi134.stdout:5/970: chown d9/d11/d73/d95/d20/d117/f144 97 1 2024-04-22T23:23:59.503 INFO:tasks.workunit.client.1.smithi134.stdout:3/758: creat dc/d2a/d62/db2/d2f/d61/d6a/f11b x:0 0 0 2024-04-22T23:23:59.504 INFO:tasks.workunit.client.1.smithi134.stdout:2/840: write d2/d42/d54/d33/d91/fe7 [4555286,22013] 0 2024-04-22T23:23:59.507 INFO:tasks.workunit.client.1.smithi134.stdout:5/971: write d9/dbb/d11b/d17/d2a/f110 [198577,59864] 0 2024-04-22T23:23:59.507 INFO:tasks.workunit.client.1.smithi134.stdout:3/759: dread dc/d2a/d62/db2/d34/d6b/f9a [0,4194304] 0 2024-04-22T23:23:59.509 INFO:tasks.workunit.client.1.smithi134.stdout:2/841: creat d2/dd/d4a/d73/f11c x:0 0 0 2024-04-22T23:23:59.510 INFO:tasks.workunit.client.1.smithi134.stdout:5/972: truncate d9/dbb/d11b/d17/d2a/d49/fad 1770281 0 2024-04-22T23:23:59.511 INFO:tasks.workunit.client.1.smithi134.stdout:5/973: read d9/d11/d73/d95/d20/d60/f7d [3156038,108532] 0 2024-04-22T23:23:59.513 INFO:tasks.workunit.client.1.smithi134.stdout:2/842: mkdir d2/d42/d54/d33/d91/d11d 0 2024-04-22T23:23:59.513 INFO:tasks.workunit.client.1.smithi134.stdout:5/974: read d9/f19 [697457,117414] 0 2024-04-22T23:23:59.514 INFO:tasks.workunit.client.1.smithi134.stdout:5/975: fsync d9/da2/de6/dc0/d109/f113 0 2024-04-22T23:23:59.514 INFO:tasks.workunit.client.1.smithi134.stdout:2/843: write d2/dd/df/d9c/daa/db0/f9b [327561,124138] 0 2024-04-22T23:23:59.515 INFO:tasks.workunit.client.1.smithi134.stdout:2/844: chown d2/dd/df/d9c/dca 6010781 1 2024-04-22T23:23:59.518 INFO:tasks.workunit.client.1.smithi134.stdout:3/760: creat dc/d2a/d62/db2/d2f/d36/f11c x:0 0 0 2024-04-22T23:23:59.519 INFO:tasks.workunit.client.1.smithi134.stdout:5/976: creat d9/d11/d73/f145 x:0 0 0 2024-04-22T23:23:59.521 INFO:tasks.workunit.client.1.smithi134.stdout:2/845: symlink d2/d15/d24/l11e 0 2024-04-22T23:23:59.522 INFO:tasks.workunit.client.1.smithi134.stdout:5/977: symlink d9/dbb/d11b/d17/l146 0 2024-04-22T23:23:59.525 INFO:tasks.workunit.client.1.smithi134.stdout:2/846: write d2/f30 [4689679,7238] 0 2024-04-22T23:23:59.526 INFO:tasks.workunit.client.1.smithi134.stdout:5/978: fdatasync d9/dbb/d11b/d88/f10e 0 2024-04-22T23:23:59.535 INFO:tasks.workunit.client.1.smithi134.stdout:3/761: getdents dc/d2a/d62/db2/d2f/d36/d6d/d6f/dd2 0 2024-04-22T23:23:59.537 INFO:tasks.workunit.client.1.smithi134.stdout:3/762: write dc/d2a/d30/dfc/f108 [215389,72593] 0 2024-04-22T23:23:59.540 INFO:tasks.workunit.client.1.smithi134.stdout:3/763: creat dc/d2a/d62/d86/dc1/f11d x:0 0 0 2024-04-22T23:23:59.542 INFO:tasks.workunit.client.1.smithi134.stdout:5/979: getdents d9/dbb/d11b 0 2024-04-22T23:23:59.543 INFO:tasks.workunit.client.1.smithi134.stdout:2/847: dread d2/dd/df/d9c/daa/db0/f9b [0,4194304] 0 2024-04-22T23:23:59.551 INFO:tasks.workunit.client.1.smithi134.stdout:7/939: sync 2024-04-22T23:23:59.551 INFO:tasks.workunit.client.1.smithi134.stdout:2/848: truncate d2/d15/d24/f29 1180372 0 2024-04-22T23:23:59.553 INFO:tasks.workunit.client.1.smithi134.stdout:2/849: chown d2/dd/df/d9c/daa/db0 130469298 1 2024-04-22T23:23:59.555 INFO:tasks.workunit.client.1.smithi134.stdout:5/980: dwrite d9/dbb/d11b/d17/d2a/d49/dc2/fcb [0,4194304] 0 2024-04-22T23:23:59.558 INFO:tasks.workunit.client.1.smithi134.stdout:3/764: dread dc/d2a/d30/f60 [0,4194304] 0 2024-04-22T23:23:59.564 INFO:tasks.workunit.client.1.smithi134.stdout:2/850: fsync d2/dd/d4a/f52 0 2024-04-22T23:23:59.566 INFO:tasks.workunit.client.1.smithi134.stdout:5/981: mkdir d9/da2/d147 0 2024-04-22T23:23:59.576 INFO:tasks.workunit.client.1.smithi134.stdout:2/851: creat d2/dd/d116/f11f x:0 0 0 2024-04-22T23:23:59.577 INFO:tasks.workunit.client.1.smithi134.stdout:7/940: dwrite d4/d8/d41/d76/dc1/d2f/d69/d8a/fb7 [0,4194304] 0 2024-04-22T23:23:59.579 INFO:tasks.workunit.client.1.smithi134.stdout:3/765: rmdir dc/d2a/d62/d86/d10b 0 2024-04-22T23:23:59.580 INFO:tasks.workunit.client.1.smithi134.stdout:5/982: mkdir d9/d11/d34/d85/d143/d148 0 2024-04-22T23:23:59.581 INFO:tasks.workunit.client.1.smithi134.stdout:2/852: dread d2/d15/d24/d3b/d8a/dc7/fe9 [0,4194304] 0 2024-04-22T23:23:59.581 INFO:tasks.workunit.client.1.smithi134.stdout:5/983: dread - d9/d11/d73/d95/d20/d108/f140 zero size 2024-04-22T23:23:59.584 INFO:tasks.workunit.client.1.smithi134.stdout:9/803: sync 2024-04-22T23:23:59.584 INFO:tasks.workunit.client.1.smithi134.stdout:5/984: read d9/d11/d34/d4d/fde [2565593,59538] 0 2024-04-22T23:23:59.584 INFO:tasks.workunit.client.1.smithi134.stdout:2/853: truncate d2/dd/df/d1c/de3/f108 906113 0 2024-04-22T23:23:59.591 INFO:tasks.workunit.client.1.smithi134.stdout:7/941: dwrite d4/d8/d2a/f5f [0,4194304] 0 2024-04-22T23:23:59.595 INFO:tasks.workunit.client.1.smithi134.stdout:9/804: symlink d0/d7/d23/d26/d51/d4c/l110 0 2024-04-22T23:23:59.602 INFO:tasks.workunit.client.1.smithi134.stdout:3/766: mkdir dc/d2a/d62/d103/d11e 0 2024-04-22T23:23:59.614 INFO:tasks.workunit.client.1.smithi134.stdout:7/942: symlink d4/d8/d41/d76/dc1/d2f/da6/dcf/l14e 0 2024-04-22T23:23:59.616 INFO:tasks.workunit.client.1.smithi134.stdout:9/805: mknod d0/d7/d12/de8/c111 0 2024-04-22T23:23:59.617 INFO:tasks.workunit.client.1.smithi134.stdout:9/806: stat d0/d7/d23/d26/d51/feb 0 2024-04-22T23:23:59.617 INFO:tasks.workunit.client.1.smithi134.stdout:3/767: mknod dc/d2a/d62/db2/d3b/c11f 0 2024-04-22T23:23:59.619 INFO:tasks.workunit.client.1.smithi134.stdout:9/807: readlink d0/d7/d23/d1b/d5a/l81 0 2024-04-22T23:23:59.624 INFO:tasks.workunit.client.1.smithi134.stdout:7/943: mknod d4/d4a/d85/c14f 0 2024-04-22T23:23:59.628 INFO:tasks.workunit.client.1.smithi134.stdout:7/944: write d4/d8/d41/d76/dc1/d2f/d69/dab/de1/f119 [613388,22700] 0 2024-04-22T23:23:59.629 INFO:tasks.workunit.client.1.smithi134.stdout:2/854: chown d2/d15/d24/f29 4 1 2024-04-22T23:23:59.632 INFO:tasks.workunit.client.1.smithi134.stdout:7/945: chown d4/d8/d41/d76/dc1/d2f/d69/la7 354412 1 2024-04-22T23:23:59.633 INFO:tasks.workunit.client.1.smithi134.stdout:7/946: write d4/d8/d41/f149 [12056,108791] 0 2024-04-22T23:23:59.634 INFO:tasks.workunit.client.1.smithi134.stdout:2/855: read d2/dd/df/fb6 [2909744,101213] 0 2024-04-22T23:23:59.637 INFO:tasks.workunit.client.1.smithi134.stdout:5/985: write d9/d11/f1a [119114,43329] 0 2024-04-22T23:23:59.642 INFO:tasks.workunit.client.1.smithi134.stdout:2/856: dwrite d2/dd/d4a/d73/f11c [0,4194304] 0 2024-04-22T23:23:59.647 INFO:tasks.workunit.client.1.smithi134.stdout:5/986: creat d9/dbb/d11b/da6/d114/f149 x:0 0 0 2024-04-22T23:23:59.647 INFO:tasks.workunit.client.1.smithi134.stdout:9/808: write d0/f54 [4203212,120043] 0 2024-04-22T23:23:59.651 INFO:tasks.workunit.client.1.smithi134.stdout:5/987: rmdir d9/d11/d34/d4d 39 2024-04-22T23:23:59.655 INFO:tasks.workunit.client.1.smithi134.stdout:7/947: dwrite d4/fac [0,4194304] 0 2024-04-22T23:23:59.656 INFO:tasks.workunit.client.1.smithi134.stdout:9/809: creat d0/d2d/d82/df5/d10d/f112 x:0 0 0 2024-04-22T23:23:59.662 INFO:tasks.workunit.client.1.smithi134.stdout:5/988: truncate d9/d11/d73/d95/d20/f48 1709357 0 2024-04-22T23:23:59.662 INFO:tasks.workunit.client.1.smithi134.stdout:5/989: stat d9/d11/d34 0 2024-04-22T23:23:59.663 INFO:tasks.workunit.client.1.smithi134.stdout:2/857: getdents d2/d42/d54/d33 0 2024-04-22T23:23:59.663 INFO:tasks.workunit.client.1.smithi134.stdout:9/810: fdatasync d0/d7/d23/d26/d51/d4c/ff0 0 2024-04-22T23:23:59.664 INFO:tasks.workunit.client.1.smithi134.stdout:7/948: truncate d4/d4a/f125 2909646 0 2024-04-22T23:23:59.665 INFO:tasks.workunit.client.1.smithi134.stdout:2/858: fsync d2/d15/d24/d3b/d8a/d99/fef 0 2024-04-22T23:23:59.666 INFO:tasks.workunit.client.1.smithi134.stdout:9/811: chown d0/d7/d23/d26/d51/d88/db9/ff1 1251 1 2024-04-22T23:23:59.666 INFO:tasks.workunit.client.1.smithi134.stdout:5/990: dread - d9/dbb/fc8 zero size 2024-04-22T23:23:59.670 INFO:tasks.workunit.client.1.smithi134.stdout:9/812: fdatasync d0/d2d/d82/df5/d10d/f112 0 2024-04-22T23:23:59.670 INFO:tasks.workunit.client.1.smithi134.stdout:5/991: fdatasync d9/dbb/d11b/d88/db7/ffa 0 2024-04-22T23:23:59.675 INFO:tasks.workunit.client.1.smithi134.stdout:7/949: creat d4/d8/d41/d76/dc1/d2f/da6/f150 x:0 0 0 2024-04-22T23:23:59.679 INFO:tasks.workunit.client.1.smithi134.stdout:2/859: creat d2/d15/d24/d1d/f120 x:0 0 0 2024-04-22T23:23:59.685 INFO:tasks.workunit.client.1.smithi134.stdout:2/860: read d2/d42/f46 [9757344,128127] 0 2024-04-22T23:23:59.686 INFO:tasks.workunit.client.1.smithi134.stdout:7/950: mknod d4/d4a/d85/c151 0 2024-04-22T23:23:59.689 INFO:tasks.workunit.client.1.smithi134.stdout:9/813: dread d0/d7/d23/d1b/f64 [0,4194304] 0 2024-04-22T23:23:59.690 INFO:tasks.workunit.client.1.smithi134.stdout:2/861: mknod d2/dd/df/d9c/daa/c121 0 2024-04-22T23:23:59.694 INFO:tasks.workunit.client.1.smithi134.stdout:5/992: dread d9/d1e/f5a [4194304,4194304] 0 2024-04-22T23:23:59.696 INFO:tasks.workunit.client.1.smithi134.stdout:7/951: rename d4/d8/f133 to d4/d8/d2a/d42/d6e/f152 0 2024-04-22T23:23:59.698 INFO:tasks.workunit.client.1.smithi134.stdout:2/862: rename d2/dd/df/d9c/cf2 to d2/d15/d24/c122 0 2024-04-22T23:23:59.700 INFO:tasks.workunit.client.1.smithi134.stdout:2/863: write d2/d15/d24/f1e [4406182,9023] 0 2024-04-22T23:23:59.700 INFO:tasks.workunit.client.1.smithi134.stdout:7/952: symlink d4/d4a/d85/dd7/l153 0 2024-04-22T23:23:59.707 INFO:tasks.workunit.client.1.smithi134.stdout:2/864: dwrite d2/dd/df/d1c/f76 [0,4194304] 0 2024-04-22T23:23:59.708 INFO:tasks.workunit.client.1.smithi134.stdout:9/814: dwrite d0/d7/fd [0,4194304] 0 2024-04-22T23:23:59.712 INFO:tasks.workunit.client.1.smithi134.stdout:7/953: truncate d4/d8/d41/d76/dc1/d2f/d69/dab/de1/f131 1751343 0 2024-04-22T23:23:59.714 INFO:tasks.workunit.client.1.smithi134.stdout:2/865: mkdir d2/d15/d24/d3b/d8a/dc7/d123 0 2024-04-22T23:23:59.715 INFO:tasks.workunit.client.1.smithi134.stdout:9/815: symlink d0/d37/l113 0 2024-04-22T23:23:59.732 INFO:tasks.workunit.client.1.smithi134.stdout:9/816: write d0/d2d/d4e/fd6 [1046856,17167] 0 2024-04-22T23:23:59.736 INFO:tasks.workunit.client.1.smithi134.stdout:7/954: dwrite d4/fa [0,4194304] 0 2024-04-22T23:23:59.737 INFO:tasks.workunit.client.1.smithi134.stdout:9/817: symlink d0/d7/d23/d26/d4d/d5d/d8e/dae/db1/l114 0 2024-04-22T23:23:59.741 INFO:tasks.workunit.client.1.smithi134.stdout:2/866: dread d2/d42/d54/fd6 [0,4194304] 0 2024-04-22T23:23:59.743 INFO:tasks.workunit.client.1.smithi134.stdout:2/867: read d2/d15/d24/f29 [602962,86313] 0 2024-04-22T23:23:59.744 INFO:tasks.workunit.client.1.smithi134.stdout:7/955: mkdir d4/d8/d41/d76/dc1/d2f/d34/d62/d29/dc5/d12e/d154 0 2024-04-22T23:23:59.747 INFO:tasks.workunit.client.1.smithi134.stdout:2/868: creat d2/d15/d24/d1d/d3d/d7b/f124 x:0 0 0 2024-04-22T23:23:59.749 INFO:tasks.workunit.client.1.smithi134.stdout:7/956: creat d4/d8/d41/d5b/d104/d140/f155 x:0 0 0 2024-04-22T23:23:59.753 INFO:tasks.workunit.client.1.smithi134.stdout:2/869: symlink d2/d15/d24/d3b/d8a/d99/l125 0 2024-04-22T23:23:59.753 INFO:tasks.workunit.client.1.smithi134.stdout:9/818: write d0/d7/d23/d26/d4d/d5d/f8f [750151,98972] 0 2024-04-22T23:23:59.754 INFO:tasks.workunit.client.1.smithi134.stdout:9/819: chown d0/d37 7 1 2024-04-22T23:23:59.763 INFO:tasks.workunit.client.1.smithi134.stdout:2/870: link d2/f98 d2/dd/dc9/f126 0 2024-04-22T23:23:59.763 INFO:tasks.workunit.client.1.smithi134.stdout:9/820: dwrite d0/d7/d23/d1b/f64 [0,4194304] 0 2024-04-22T23:23:59.767 INFO:tasks.workunit.client.1.smithi134.stdout:0/935: sync 2024-04-22T23:23:59.767 INFO:tasks.workunit.client.1.smithi134.stdout:6/955: sync 2024-04-22T23:23:59.767 INFO:tasks.workunit.client.1.smithi134.stdout:2/871: mknod d2/d42/d54/d33/d6e/c127 0 2024-04-22T23:23:59.767 INFO:tasks.workunit.client.1.smithi134.stdout:8/871: sync 2024-04-22T23:23:59.771 INFO:tasks.workunit.client.1.smithi134.stdout:6/956: unlink d0/d19/db/f68 0 2024-04-22T23:23:59.775 INFO:tasks.workunit.client.1.smithi134.stdout:9/821: dwrite d0/d7/d23/d1b/f64 [4194304,4194304] 0 2024-04-22T23:23:59.778 INFO:tasks.workunit.client.1.smithi134.stdout:7/957: dread d4/f2e [4194304,4194304] 0 2024-04-22T23:23:59.782 INFO:tasks.workunit.client.1.smithi134.stdout:7/958: chown d4/d8/d41/d76/dc1/d2f/d34/dc0 12894 1 2024-04-22T23:23:59.782 INFO:tasks.workunit.client.1.smithi134.stdout:9/822: symlink d0/d37/d76/l115 0 2024-04-22T23:23:59.786 INFO:tasks.workunit.client.1.smithi134.stdout:2/872: dread d2/d15/d24/d1d/d3d/d7b/f7c [0,4194304] 0 2024-04-22T23:23:59.788 INFO:tasks.workunit.client.1.smithi134.stdout:9/823: unlink d0/d7/fb 0 2024-04-22T23:23:59.789 INFO:tasks.workunit.client.1.smithi134.stdout:7/959: creat d4/d8/d41/d5b/d104/f156 x:0 0 0 2024-04-22T23:23:59.791 INFO:tasks.workunit.client.1.smithi134.stdout:7/960: fsync d4/d8/d3b/ffb 0 2024-04-22T23:23:59.799 INFO:tasks.workunit.client.1.smithi134.stdout:2/873: dwrite d2/d15/d24/d1d/d3d/d9a/fd5 [0,4194304] 0 2024-04-22T23:23:59.801 INFO:tasks.workunit.client.1.smithi134.stdout:8/872: dread d7/db/d2d/f54 [0,4194304] 0 2024-04-22T23:23:59.805 INFO:tasks.workunit.client.1.smithi134.stdout:0/936: dread d1/d5/d58/da0/fb6 [0,4194304] 0 2024-04-22T23:23:59.811 INFO:tasks.workunit.client.1.smithi134.stdout:8/873: dwrite d7/db/d2d/d39/d58/f97 [4194304,4194304] 0 2024-04-22T23:23:59.816 INFO:tasks.workunit.client.1.smithi134.stdout:6/957: dread d0/d19/db/d75/f12c [0,4194304] 0 2024-04-22T23:23:59.818 INFO:tasks.workunit.client.1.smithi134.stdout:0/937: creat d1/d5/dd/d20/d127/f133 x:0 0 0 2024-04-22T23:23:59.818 INFO:tasks.workunit.client.1.smithi134.stdout:7/961: dwrite d4/d8/d41/d76/dc1/d2f/d69/df8/f115 [0,4194304] 0 2024-04-22T23:23:59.820 INFO:tasks.workunit.client.1.smithi134.stdout:0/938: chown d1/d5/d58/d6b/db9/def/ffb 52 1 2024-04-22T23:23:59.821 INFO:tasks.workunit.client.1.smithi134.stdout:2/874: write d2/dd/d72/fae [300127,57544] 0 2024-04-22T23:23:59.825 INFO:tasks.workunit.client.1.smithi134.stdout:0/939: symlink d1/d5/d58/da0/l134 0 2024-04-22T23:23:59.827 INFO:tasks.workunit.client.1.smithi134.stdout:8/874: unlink d7/dc/f12 0 2024-04-22T23:23:59.829 INFO:tasks.workunit.client.1.smithi134.stdout:7/962: symlink d4/d4a/d116/l157 0 2024-04-22T23:23:59.830 INFO:tasks.workunit.client.1.smithi134.stdout:0/940: creat d1/d2/f135 x:0 0 0 2024-04-22T23:23:59.832 INFO:tasks.workunit.client.1.smithi134.stdout:2/875: mknod d2/c128 0 2024-04-22T23:23:59.836 INFO:tasks.workunit.client.1.smithi134.stdout:8/875: mknod d7/db/d42/d65/dd9/dda/de4/d111/c119 0 2024-04-22T23:23:59.837 INFO:tasks.workunit.client.1.smithi134.stdout:2/876: unlink d2/dd/dc9/ff6 0 2024-04-22T23:23:59.845 INFO:tasks.workunit.client.1.smithi134.stdout:8/876: mknod d7/db/d42/d65/dd9/c11a 0 2024-04-22T23:23:59.848 INFO:tasks.workunit.client.1.smithi134.stdout:2/877: symlink d2/dd/dc9/l129 0 2024-04-22T23:23:59.850 INFO:tasks.workunit.client.1.smithi134.stdout:8/877: fdatasync d7/db/d42/d65/d46/d9a/da3/d88/f103 0 2024-04-22T23:23:59.851 INFO:tasks.workunit.client.1.smithi134.stdout:7/963: dwrite d4/d8/d41/d76/dc1/faa [0,4194304] 0 2024-04-22T23:23:59.852 INFO:tasks.workunit.client.1.smithi134.stdout:0/941: dread d1/d5/dd/d41/f11b [0,4194304] 0 2024-04-22T23:23:59.856 INFO:tasks.workunit.client.1.smithi134.stdout:8/878: creat d7/db/d42/d65/dd9/dda/de4/f11b x:0 0 0 2024-04-22T23:23:59.859 INFO:tasks.workunit.client.1.smithi134.stdout:7/964: truncate d4/d8/d2a/d42/d79/fa2 931620 0 2024-04-22T23:23:59.863 INFO:tasks.workunit.client.1.smithi134.stdout:8/879: fdatasync d7/dc/d2c/d37/d47/f34 0 2024-04-22T23:23:59.864 INFO:tasks.workunit.client.1.smithi134.stdout:8/880: write d7/db/d42/d65/dd9/dda/de7/fdb [911995,57541] 0 2024-04-22T23:23:59.866 INFO:tasks.workunit.client.1.smithi134.stdout:8/881: chown d7/dc/d2c/d4f/d6c/fec 682184514 1 2024-04-22T23:23:59.866 INFO:tasks.workunit.client.1.smithi134.stdout:7/965: mkdir d4/d8/d41/d76/dc1/d2f/d34/dc0/d158 0 2024-04-22T23:23:59.872 INFO:tasks.workunit.client.1.smithi134.stdout:2/878: link d2/dd/df/d9c/daa/ld2 d2/dd/de4/l12a 0 2024-04-22T23:23:59.873 INFO:tasks.workunit.client.1.smithi134.stdout:0/942: getdents d1/d5/d58/da0/d128 0 2024-04-22T23:23:59.876 INFO:tasks.workunit.client.1.smithi134.stdout:7/966: rename d4/d8/d58/c108 to d4/d8/d41/d76/dc1/d2f/d34/d62/d29/dc5/d12e/c159 0 2024-04-22T23:23:59.878 INFO:tasks.workunit.client.1.smithi134.stdout:0/943: chown d1/d5/dd/d20/cca 567 1 2024-04-22T23:23:59.879 INFO:tasks.workunit.client.1.smithi134.stdout:0/944: fsync d1/d2/f74 0 2024-04-22T23:23:59.881 INFO:tasks.workunit.client.1.smithi134.stdout:8/882: link d7/dc/d2c/d4f/d6c/dd1/fdc d7/db/d42/d65/dd9/f11c 0 2024-04-22T23:23:59.886 INFO:tasks.workunit.client.1.smithi134.stdout:0/945: dwrite d1/d5/dd/d41/fdb [0,4194304] 0 2024-04-22T23:23:59.895 INFO:tasks.workunit.client.1.smithi134.stdout:0/946: dwrite d1/d5/fbe [0,4194304] 0 2024-04-22T23:23:59.910 INFO:tasks.workunit.client.1.smithi134.stdout:7/967: dread d4/d8/d41/d76/dc1/fb8 [0,4194304] 0 2024-04-22T23:23:59.994 INFO:tasks.workunit.client.1.smithi134.stdout:2/879: getdents d2/d15/d24/d3b/d8a/d99 0 2024-04-22T23:24:00.013 INFO:tasks.workunit.client.1.smithi134.stdout:3/768: sync 2024-04-22T23:24:00.019 INFO:tasks.workunit.client.1.smithi134.stdout:3/769: dwrite dc/d2a/d62/db2/d2f/d21/d42/ff1 [0,4194304] 0 2024-04-22T23:24:00.021 INFO:tasks.workunit.client.1.smithi134.stdout:2/880: creat d2/dd/de4/f12b x:0 0 0 2024-04-22T23:24:00.022 INFO:tasks.workunit.client.1.smithi134.stdout:0/947: creat d1/d5/d15/f136 x:0 0 0 2024-04-22T23:24:00.032 INFO:tasks.workunit.client.1.smithi134.stdout:0/948: mkdir d1/d5/dd/d12b/d137 0 2024-04-22T23:24:00.036 INFO:tasks.workunit.client.1.smithi134.stdout:2/881: link d2/dd/df/d9c/daa/ld2 d2/dd/d4a/d73/l12c 0 2024-04-22T23:24:00.038 INFO:tasks.workunit.client.1.smithi134.stdout:3/770: creat dc/d2a/d62/db2/d2f/d21/d42/d77/f120 x:0 0 0 2024-04-22T23:24:00.039 INFO:tasks.workunit.client.1.smithi134.stdout:2/882: symlink d2/dd/df/d1c/l12d 0 2024-04-22T23:24:00.042 INFO:tasks.workunit.client.1.smithi134.stdout:3/771: write dc/d2a/d62/db2/d3b/f31 [473352,52085] 0 2024-04-22T23:24:00.043 INFO:tasks.workunit.client.1.smithi134.stdout:0/949: getdents d1/d5/d58/da0/de5 0 2024-04-22T23:24:00.047 INFO:tasks.workunit.client.1.smithi134.stdout:3/772: creat dc/d2a/d62/db2/d2f/d36/d6d/d6f/dd9/f121 x:0 0 0 2024-04-22T23:24:00.048 INFO:tasks.workunit.client.1.smithi134.stdout:0/950: symlink d1/d5/d58/d6b/db9/def/l138 0 2024-04-22T23:24:00.049 INFO:tasks.workunit.client.1.smithi134.stdout:0/951: write d1/d2/d1d/f121 [953938,15529] 0 2024-04-22T23:24:00.051 INFO:tasks.workunit.client.1.smithi134.stdout:2/883: rmdir d2/d15/d24/d1d/db3 0 2024-04-22T23:24:00.054 INFO:tasks.workunit.client.1.smithi134.stdout:2/884: dread - d2/dd/dc9/f126 zero size 2024-04-22T23:24:00.057 INFO:tasks.workunit.client.1.smithi134.stdout:2/885: symlink d2/dd/d116/l12e 0 2024-04-22T23:24:00.058 INFO:tasks.workunit.client.1.smithi134.stdout:0/952: dread d1/d5/dd/d20/d2c/f4b [0,4194304] 0 2024-04-22T23:24:00.058 INFO:tasks.workunit.client.1.smithi134.stdout:3/773: link dc/dd/l27 dc/d2a/d62/db2/d2f/d54/dac/d115/l122 0 2024-04-22T23:24:00.060 INFO:tasks.workunit.client.1.smithi134.stdout:2/886: mkdir d2/d42/d54/d33/d91/d12f 0 2024-04-22T23:24:00.065 INFO:tasks.workunit.client.1.smithi134.stdout:2/887: rmdir d2/d15/d24/d1d/dba/dea 39 2024-04-22T23:24:00.066 INFO:tasks.workunit.client.1.smithi134.stdout:2/888: dread - d2/dd/dc9/f126 zero size 2024-04-22T23:24:00.073 INFO:tasks.workunit.client.1.smithi134.stdout:0/953: creat d1/d5/d58/d6b/db9/f139 x:0 0 0 2024-04-22T23:24:00.076 INFO:tasks.workunit.client.1.smithi134.stdout:3/774: rmdir dc/d2a/d62/db2/d3b/dfd/d107 0 2024-04-22T23:24:00.080 INFO:tasks.workunit.client.1.smithi134.stdout:0/954: rename d1/d5/d58/d6b/db9/d75 to d1/d2/d117/d13a 0 2024-04-22T23:24:00.081 INFO:tasks.workunit.client.1.smithi134.stdout:0/955: fsync d1/d5/d15/f104 0 2024-04-22T23:24:00.081 INFO:tasks.workunit.client.1.smithi134.stdout:3/775: mkdir dc/d2a/d62/d86/d123 0 2024-04-22T23:24:00.082 INFO:tasks.workunit.client.1.smithi134.stdout:2/889: link d2/d42/d54/l8b d2/d15/d24/d3b/d8a/dc7/l130 0 2024-04-22T23:24:00.082 INFO:tasks.workunit.client.1.smithi134.stdout:3/776: write dc/f18 [1876169,54483] 0 2024-04-22T23:24:00.085 INFO:tasks.workunit.client.1.smithi134.stdout:0/956: fdatasync d1/d5/dd/f45 0 2024-04-22T23:24:00.086 INFO:tasks.workunit.client.1.smithi134.stdout:0/957: chown d1/d5/dd/d41/ffc 3617 1 2024-04-22T23:24:00.087 INFO:tasks.workunit.client.1.smithi134.stdout:0/958: fsync d1/d5/dd/d68/f111 0 2024-04-22T23:24:00.089 INFO:tasks.workunit.client.1.smithi134.stdout:3/777: mknod dc/d2a/d62/db2/d34/c124 0 2024-04-22T23:24:00.090 INFO:tasks.workunit.client.1.smithi134.stdout:3/778: write dc/d2a/d62/d86/dc1/fe2 [4488733,70052] 0 2024-04-22T23:24:00.093 INFO:tasks.workunit.client.1.smithi134.stdout:0/959: symlink d1/d5/d58/d94/dea/d12d/l13b 0 2024-04-22T23:24:00.097 INFO:tasks.workunit.client.1.smithi134.stdout:3/779: unlink dc/f22 0 2024-04-22T23:24:00.097 INFO:tasks.workunit.client.1.smithi134.stdout:2/890: getdents d2/d15/d24/d1d/d3d 0 2024-04-22T23:24:00.098 INFO:tasks.workunit.client.1.smithi134.stdout:0/960: mkdir d1/d5/d58/da0/de5/d13c 0 2024-04-22T23:24:00.098 INFO:tasks.workunit.client.1.smithi134.stdout:2/891: chown d2/d15/d24/d3b/d8a/d99/l125 0 1 2024-04-22T23:24:00.099 INFO:tasks.workunit.client.1.smithi134.stdout:3/780: mkdir dc/d2a/d30/d125 0 2024-04-22T23:24:00.102 INFO:tasks.workunit.client.1.smithi134.stdout:3/781: creat dc/d2a/d62/db2/d2f/d54/f126 x:0 0 0 2024-04-22T23:24:00.104 INFO:tasks.workunit.client.1.smithi134.stdout:3/782: symlink dc/d2a/d62/d86/dc1/df7/l127 0 2024-04-22T23:24:00.105 INFO:tasks.workunit.client.1.smithi134.stdout:3/783: getdents dc/d2a/d30/d125 0 2024-04-22T23:24:00.106 INFO:tasks.workunit.client.1.smithi134.stdout:3/784: read - dc/d2a/d62/d86/dc1/f109 zero size 2024-04-22T23:24:00.112 INFO:tasks.workunit.client.1.smithi134.stdout:7/968: dwrite d4/d8/d41/d76/dc1/f23 [4194304,4194304] 0 2024-04-22T23:24:00.116 INFO:tasks.workunit.client.1.smithi134.stdout:8/883: mknod d7/db/d42/d65/c11d 0 2024-04-22T23:24:00.118 INFO:tasks.workunit.client.1.smithi134.stdout:8/884: write d7/db/d42/d65/dd9/dda/de7/f10d [825771,35567] 0 2024-04-22T23:24:00.126 INFO:tasks.workunit.client.1.smithi134.stdout:7/969: dread d4/f2e [0,4194304] 0 2024-04-22T23:24:00.127 INFO:tasks.workunit.client.1.smithi134.stdout:8/885: rename d7/db/d42/d65/d46/d9a/da3/d88/lb5 to d7/dc/d2c/l11e 0 2024-04-22T23:24:00.130 INFO:tasks.workunit.client.1.smithi134.stdout:5/993: sync 2024-04-22T23:24:00.130 INFO:tasks.workunit.client.1.smithi134.stdout:8/886: chown d7/db/d42/d65/dd9/dda/de7 52956867 1 2024-04-22T23:24:00.131 INFO:tasks.workunit.client.1.smithi134.stdout:8/887: chown d7/dc/lbe 350560004 1 2024-04-22T23:24:00.132 INFO:tasks.workunit.client.1.smithi134.stdout:7/970: creat d4/d8/d41/d76/dc1/d2f/d69/d8a/f15a x:0 0 0 2024-04-22T23:24:00.134 INFO:tasks.workunit.client.1.smithi134.stdout:7/971: read d4/d8/d3b/f50 [1410571,105342] 0 2024-04-22T23:24:00.135 INFO:tasks.workunit.client.1.smithi134.stdout:7/972: write d4/dbd/f13b [549907,89873] 0 2024-04-22T23:24:00.136 INFO:tasks.workunit.client.1.smithi134.stdout:8/888: creat d7/dc/d2c/d4f/d6c/dd1/f11f x:0 0 0 2024-04-22T23:24:00.137 INFO:tasks.workunit.client.1.smithi134.stdout:8/889: dread - d7/db/f114 zero size 2024-04-22T23:24:00.138 INFO:tasks.workunit.client.1.smithi134.stdout:8/890: stat d7/db/d42/d65/dd9/dda/de7/fde 0 2024-04-22T23:24:00.139 INFO:tasks.workunit.client.1.smithi134.stdout:8/891: write d7/db/d42/f102 [823437,120487] 0 2024-04-22T23:24:00.142 INFO:tasks.workunit.client.1.smithi134.stdout:5/994: mkdir d9/d11/d73/d95/d20/d14a 0 2024-04-22T23:24:00.146 INFO:tasks.workunit.client.1.smithi134.stdout:0/961: write d1/d5/d58/fc7 [169976,120567] 0 2024-04-22T23:24:00.147 INFO:tasks.workunit.client.1.smithi134.stdout:0/962: dread - d1/d5/dd/d41/ffc zero size 2024-04-22T23:24:00.148 INFO:tasks.workunit.client.1.smithi134.stdout:0/963: truncate d1/d2/d1d/f121 974491 0 2024-04-22T23:24:00.149 INFO:tasks.workunit.client.1.smithi134.stdout:0/964: chown d1/d5/dd/d68/lad 8 1 2024-04-22T23:24:00.153 INFO:tasks.workunit.client.1.smithi134.stdout:3/785: fsync dc/d2a/d62/db2/d34/d6b/dae/f85 0 2024-04-22T23:24:00.154 INFO:tasks.workunit.client.1.smithi134.stdout:3/786: chown dc/d2a/d62/db2/d2f/d21/d42/lcd 100417434 1 2024-04-22T23:24:00.160 INFO:tasks.workunit.client.1.smithi134.stdout:0/965: mkdir d1/d2/d117/d13a/de6/d13d 0 2024-04-22T23:24:00.160 INFO:tasks.workunit.client.1.smithi134.stdout:7/973: dwrite d4/d8/d2a/f4f [4194304,4194304] 0 2024-04-22T23:24:00.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:59 smithi134 ceph-mon[34609]: pgmap v4: 65 pgs: 65 active+clean; 2.4 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail 2024-04-22T23:24:00.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:23:59 smithi134 ceph-mon[34609]: mgrmap e23: smithi134.ccxkcj(active, since 2s) 2024-04-22T23:24:00.267 INFO:tasks.workunit.client.1.smithi134.stdout:8/892: truncate d7/dc/d19/da5/fab 1893763 0 2024-04-22T23:24:00.278 INFO:tasks.workunit.client.1.smithi134.stdout:0/966: rename d1/d5/d58/d6b/cf0 to d1/d5/dd/c13e 0 2024-04-22T23:24:00.281 INFO:tasks.workunit.client.1.smithi134.stdout:3/787: dread dc/d2a/d62/db2/d2f/d36/f5d [0,4194304] 0 2024-04-22T23:24:00.285 INFO:tasks.workunit.client.1.smithi134.stdout:3/788: mknod dc/d2a/d62/db2/d2f/d21/c128 0 2024-04-22T23:24:00.286 INFO:tasks.workunit.client.1.smithi134.stdout:8/893: write f4 [2710517,29967] 0 2024-04-22T23:24:00.292 INFO:tasks.workunit.client.1.smithi134.stdout:0/967: getdents d1/d5/d58/d22/dd0/ddf 0 2024-04-22T23:24:00.292 INFO:tasks.workunit.client.1.smithi134.stdout:7/974: dwrite d4/d8/d2a/d42/d75/fbf [0,4194304] 0 2024-04-22T23:24:00.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:59 smithi067 ceph-mon[26453]: pgmap v4: 65 pgs: 65 active+clean; 2.4 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail 2024-04-22T23:24:00.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:23:59 smithi067 ceph-mon[26453]: mgrmap e23: smithi134.ccxkcj(active, since 2s) 2024-04-22T23:24:00.296 INFO:tasks.workunit.client.1.smithi134.stdout:3/789: write dc/d2a/d62/fa5 [473149,77472] 0 2024-04-22T23:24:00.296 INFO:tasks.workunit.client.1.smithi134.stdout:0/968: mkdir d1/d5/d58/da0/d13f 0 2024-04-22T23:24:00.297 INFO:tasks.workunit.client.1.smithi134.stdout:3/790: readlink dc/d2a/d62/db2/d2f/d21/d42/l5a 0 2024-04-22T23:24:00.302 INFO:tasks.workunit.client.1.smithi134.stdout:7/975: unlink d4/d8/d41/d76/dc1/f11 0 2024-04-22T23:24:00.302 INFO:tasks.workunit.client.1.smithi134.stdout:0/969: mknod d1/d5/d58/da0/d13f/c140 0 2024-04-22T23:24:00.304 INFO:tasks.workunit.client.1.smithi134.stdout:0/970: creat d1/d5/d58/d22/dd0/f141 x:0 0 0 2024-04-22T23:24:00.309 INFO:tasks.workunit.client.1.smithi134.stdout:7/976: dwrite d4/d8/d2a/f47 [4194304,4194304] 0 2024-04-22T23:24:00.311 INFO:tasks.workunit.client.1.smithi134.stdout:7/977: read - d4/d8/d41/d5b/d104/f156 zero size 2024-04-22T23:24:00.312 INFO:tasks.workunit.client.1.smithi134.stdout:7/978: symlink d4/d8/d41/d5b/l15b 0 2024-04-22T23:24:00.316 INFO:tasks.workunit.client.1.smithi134.stdout:7/979: creat d4/d8/d41/d5b/d112/f15c x:0 0 0 2024-04-22T23:24:00.318 INFO:tasks.workunit.client.1.smithi134.stdout:7/980: mknod d4/d8/d2a/d42/d6e/c15d 0 2024-04-22T23:24:00.319 INFO:tasks.workunit.client.1.smithi134.stdout:0/971: truncate d1/d5/d58/d22/fa7 1945126 0 2024-04-22T23:24:00.321 INFO:tasks.workunit.client.1.smithi134.stdout:7/981: mkdir d4/d8/d2a/d15e 0 2024-04-22T23:24:00.321 INFO:tasks.workunit.client.1.smithi134.stdout:7/982: stat d4/d8/ff 0 2024-04-22T23:24:00.322 INFO:tasks.workunit.client.1.smithi134.stdout:0/972: mkdir d1/d2/d117/d13a/de6/d13d/d142 0 2024-04-22T23:24:00.323 INFO:tasks.workunit.client.1.smithi134.stdout:8/894: dread d7/dc/d2c/d4f/d6c/d6e/f6f [0,4194304] 0 2024-04-22T23:24:00.326 INFO:tasks.workunit.client.1.smithi134.stdout:7/983: getdents d4/d4a/d85/dd7/dfa 0 2024-04-22T23:24:00.326 INFO:tasks.workunit.client.1.smithi134.stdout:0/973: read d1/d5/da2/fde [230579,109875] 0 2024-04-22T23:24:00.326 INFO:tasks.workunit.client.1.smithi134.stdout:8/895: mkdir d7/db/d42/d65/d46/d120 0 2024-04-22T23:24:00.327 INFO:tasks.workunit.client.1.smithi134.stdout:8/896: write f4 [6282859,89932] 0 2024-04-22T23:24:00.330 INFO:tasks.workunit.client.1.smithi134.stdout:6/958: sync 2024-04-22T23:24:00.330 INFO:tasks.workunit.client.1.smithi134.stdout:9/824: sync 2024-04-22T23:24:00.340 INFO:tasks.workunit.client.1.smithi134.stdout:9/825: creat d0/d7/d23/d1b/d5a/f116 x:0 0 0 2024-04-22T23:24:00.340 INFO:tasks.workunit.client.1.smithi134.stdout:0/974: getdents d1/d5/d58/d22/dd0/ddf 0 2024-04-22T23:24:00.341 INFO:tasks.workunit.client.1.smithi134.stdout:0/975: chown d1/db/ff6 26 1 2024-04-22T23:24:00.342 INFO:tasks.workunit.client.1.smithi134.stdout:8/897: creat d7/dc/d19/f121 x:0 0 0 2024-04-22T23:24:00.344 INFO:tasks.workunit.client.1.smithi134.stdout:9/826: unlink d0/d2d/d3d/f87 0 2024-04-22T23:24:00.347 INFO:tasks.workunit.client.1.smithi134.stdout:7/984: link d4/d8/d2a/d42/d6e/c114 d4/d8/d41/d76/dc1/d2f/d34/d62/d29/dc5/dd2/c15f 0 2024-04-22T23:24:00.348 INFO:tasks.workunit.client.1.smithi134.stdout:6/959: creat d0/d19/de/d1f/f13b x:0 0 0 2024-04-22T23:24:00.352 INFO:tasks.workunit.client.1.smithi134.stdout:8/898: creat d7/dc/de0/f122 x:0 0 0 2024-04-22T23:24:00.354 INFO:tasks.workunit.client.1.smithi134.stdout:7/985: rmdir d4/d8/d2a/d42/d7d 39 2024-04-22T23:24:00.356 INFO:tasks.workunit.client.1.smithi134.stdout:7/986: write d4/d8/d41/d76/dc1/d2f/d70/f74 [409048,25388] 0 2024-04-22T23:24:00.356 INFO:tasks.workunit.client.1.smithi134.stdout:6/960: creat d0/d19/de/d12/d5e/dd6/f13c x:0 0 0 2024-04-22T23:24:00.360 INFO:tasks.workunit.client.1.smithi134.stdout:0/976: truncate d1/d5/ff4 3903327 0 2024-04-22T23:24:00.364 INFO:tasks.workunit.client.1.smithi134.stdout:6/961: dwrite d0/f16 [0,4194304] 0 2024-04-22T23:24:00.366 INFO:tasks.workunit.client.1.smithi134.stdout:9/827: rename d0/d7/d23/d26/d4d/d5d/f8a to d0/d7/d23/d1b/f117 0 2024-04-22T23:24:00.371 INFO:tasks.workunit.client.1.smithi134.stdout:0/977: symlink d1/d5/d58/da0/d13f/l143 0 2024-04-22T23:24:00.372 INFO:tasks.workunit.client.1.smithi134.stdout:0/978: write d1/d5/d58/f114 [438475,91542] 0 2024-04-22T23:24:00.374 INFO:tasks.workunit.client.1.smithi134.stdout:6/962: rmdir d0/d19/de/d12/d29/d74/deb 39 2024-04-22T23:24:00.376 INFO:tasks.workunit.client.1.smithi134.stdout:9/828: creat d0/d7/d23/d1b/dc5/f118 x:0 0 0 2024-04-22T23:24:00.377 INFO:tasks.workunit.client.1.smithi134.stdout:6/963: truncate d0/d19/db/d75/d92/f118 541704 0 2024-04-22T23:24:00.380 INFO:tasks.workunit.client.1.smithi134.stdout:6/964: creat d0/d6f/d88/d8b/f13d x:0 0 0 2024-04-22T23:24:00.382 INFO:tasks.workunit.client.1.smithi134.stdout:8/899: link d7/dc/ff d7/dc/d2c/d37/d47/f123 0 2024-04-22T23:24:00.384 INFO:tasks.workunit.client.1.smithi134.stdout:9/829: dwrite d0/d2d/d4e/ff6 [0,4194304] 0 2024-04-22T23:24:00.388 INFO:tasks.workunit.client.1.smithi134.stdout:6/965: creat d0/d19/db/d121/f13e x:0 0 0 2024-04-22T23:24:00.390 INFO:tasks.workunit.client.1.smithi134.stdout:9/830: dread d0/d7/d12/de8/fee [0,4194304] 0 2024-04-22T23:24:00.391 INFO:tasks.workunit.client.1.smithi134.stdout:8/900: unlink d7/db/d42/f59 0 2024-04-22T23:24:00.393 INFO:tasks.workunit.client.1.smithi134.stdout:8/901: readlink d7/db/d42/d65/d46/d9a/da3/d88/la2 0 2024-04-22T23:24:00.400 INFO:tasks.workunit.client.1.smithi134.stdout:8/902: dwrite d7/db/d42/d65/f107 [0,4194304] 0 2024-04-22T23:24:00.403 INFO:tasks.workunit.client.1.smithi134.stdout:7/987: truncate d4/d8/d41/d76/dc1/d2f/d34/dc0/f107 4188109 0 2024-04-22T23:24:00.416 INFO:tasks.workunit.client.1.smithi134.stdout:8/903: mkdir d7/dc/d124 0 2024-04-22T23:24:00.417 INFO:tasks.workunit.client.1.smithi134.stdout:9/831: link d0/d7/d23/d26/d4d/d5d/d8e/c97 d0/d7/d23/d1b/de2/c119 0 2024-04-22T23:24:00.419 INFO:tasks.workunit.client.1.smithi134.stdout:8/904: symlink d7/dc/d2c/d4f/d6c/l125 0 2024-04-22T23:24:00.422 INFO:tasks.workunit.client.1.smithi134.stdout:6/966: dwrite d0/d19/db/d35/d7f/fee [0,4194304] 0 2024-04-22T23:24:00.427 INFO:tasks.workunit.client.1.smithi134.stdout:8/905: dwrite d7/db/d42/d65/d46/fdf [4194304,4194304] 0 2024-04-22T23:24:00.428 INFO:tasks.workunit.client.1.smithi134.stdout:8/906: chown d7/f1c 29626 1 2024-04-22T23:24:00.429 INFO:tasks.workunit.client.1.smithi134.stdout:7/988: getdents d4/d8/d41/d5b 0 2024-04-22T23:24:00.435 INFO:tasks.workunit.client.1.smithi134.stdout:8/907: symlink d7/db/d42/d65/d46/d9a/da3/d88/l126 0 2024-04-22T23:24:00.442 INFO:tasks.workunit.client.1.smithi134.stdout:9/832: rename d0/d7/d23/d26/c74 to d0/c11a 0 2024-04-22T23:24:00.443 INFO:tasks.workunit.client.1.smithi134.stdout:9/833: write d0/d7/d12/de8/df7/f108 [267523,114283] 0 2024-04-22T23:24:00.444 INFO:tasks.workunit.client.1.smithi134.stdout:6/967: creat d0/d6f/f13f x:0 0 0 2024-04-22T23:24:00.445 INFO:tasks.workunit.client.1.smithi134.stdout:6/968: stat d0/d19/de/f139 0 2024-04-22T23:24:00.447 INFO:tasks.workunit.client.1.smithi134.stdout:8/908: truncate d7/dc/d2c/d4f/d6c/d6e/f91 1061015 0 2024-04-22T23:24:00.448 INFO:tasks.workunit.client.1.smithi134.stdout:8/909: stat d7/db/d42/d65/dd9/dda/de4/f11b 0 2024-04-22T23:24:00.453 INFO:tasks.workunit.client.1.smithi134.stdout:6/969: dwrite d0/d19/de/d12/d2a/d2d/d3a/f9a [0,4194304] 0 2024-04-22T23:24:00.455 INFO:tasks.workunit.client.1.smithi134.stdout:8/910: mknod d7/dc/d2c/d4f/d6c/c127 0 2024-04-22T23:24:00.456 INFO:tasks.workunit.client.1.smithi134.stdout:9/834: link d0/d37/f4f d0/d7/d23/d26/d4d/d5d/d8e/d105/f11b 0 2024-04-22T23:24:00.457 INFO:tasks.workunit.client.1.smithi134.stdout:9/835: chown d0/d7/d23/d26/d51/d4c 8008366 1 2024-04-22T23:24:00.461 INFO:tasks.workunit.client.1.smithi134.stdout:8/911: rmdir d7/dc/d2c/d4f/d6c/dd1 39 2024-04-22T23:24:00.466 INFO:tasks.workunit.client.1.smithi134.stdout:9/836: mknod d0/d2d/d82/dbf/c11c 0 2024-04-22T23:24:00.468 INFO:tasks.workunit.client.1.smithi134.stdout:7/989: write d4/d8/d2a/d42/d6e/f152 [1082410,48418] 0 2024-04-22T23:24:00.471 INFO:tasks.workunit.client.1.smithi134.stdout:8/912: dwrite d7/db/d42/d65/d46/d9a/da3/d88/f103 [0,4194304] 0 2024-04-22T23:24:00.472 INFO:tasks.workunit.client.1.smithi134.stdout:7/990: write d4/d8/d3b/fd1 [2282491,108525] 0 2024-04-22T23:24:00.473 INFO:tasks.workunit.client.1.smithi134.stdout:9/837: mknod d0/d2d/d82/da0/c11d 0 2024-04-22T23:24:00.479 INFO:tasks.workunit.client.1.smithi134.stdout:6/970: dread d0/d4f/fcc [0,4194304] 0 2024-04-22T23:24:00.480 INFO:tasks.workunit.client.1.smithi134.stdout:9/838: symlink d0/d2d/d82/dbf/l11e 0 2024-04-22T23:24:00.483 INFO:tasks.workunit.client.1.smithi134.stdout:8/913: truncate d7/dc/d2c/d37/d47/f5c 361442 0 2024-04-22T23:24:00.487 INFO:tasks.workunit.client.1.smithi134.stdout:6/971: mknod d0/d4f/c140 0 2024-04-22T23:24:00.491 INFO:tasks.workunit.client.1.smithi134.stdout:8/914: readlink d7/lff 0 2024-04-22T23:24:00.497 INFO:tasks.workunit.client.1.smithi134.stdout:7/991: truncate d4/fac 2142818 0 2024-04-22T23:24:00.497 INFO:tasks.workunit.client.1.smithi134.stdout:8/915: dwrite d7/dc/fa8 [0,4194304] 0 2024-04-22T23:24:00.498 INFO:tasks.workunit.client.1.smithi134.stdout:6/972: chown d0/f10 78 1 2024-04-22T23:24:00.508 INFO:tasks.workunit.client.1.smithi134.stdout:9/839: rename d0/d7/d23/f55 to d0/d2d/f11f 0 2024-04-22T23:24:00.511 INFO:tasks.workunit.client.1.smithi134.stdout:2/892: sync 2024-04-22T23:24:00.513 INFO:tasks.workunit.client.1.smithi134.stdout:6/973: dread d0/d19/db/fd [0,4194304] 0 2024-04-22T23:24:00.521 INFO:tasks.workunit.client.1.smithi134.stdout:6/974: rename d0/d19/de/dbe/fc6 to d0/d19/de/d12/d2a/d2d/f141 0 2024-04-22T23:24:00.523 INFO:tasks.workunit.client.1.smithi134.stdout:2/893: mknod d2/d42/d54/d33/d91/d11d/c131 0 2024-04-22T23:24:00.524 INFO:tasks.workunit.client.1.smithi134.stdout:7/992: truncate d4/d8/fb 7556171 0 2024-04-22T23:24:00.524 INFO:tasks.workunit.client.1.smithi134.stdout:7/993: fdatasync d4/d8/d2a/f2c 0 2024-04-22T23:24:00.527 INFO:tasks.workunit.client.1.smithi134.stdout:6/975: mkdir d0/d19/de/d12/d5e/dc1/d136/d142 0 2024-04-22T23:24:00.529 INFO:tasks.workunit.client.1.smithi134.stdout:8/916: truncate d7/db/d42/d65/f107 417759 0 2024-04-22T23:24:00.530 INFO:tasks.workunit.client.1.smithi134.stdout:7/994: dread - d4/d8/d41/d5b/d112/f122 zero size 2024-04-22T23:24:00.531 INFO:tasks.workunit.client.1.smithi134.stdout:7/995: chown d4/d8/d41/d5b/d104/f111 114517828 1 2024-04-22T23:24:00.533 INFO:tasks.workunit.client.1.smithi134.stdout:9/840: truncate d0/d2d/d82/fb7 85742 0 2024-04-22T23:24:00.535 INFO:tasks.workunit.client.1.smithi134.stdout:3/791: sync 2024-04-22T23:24:00.535 INFO:tasks.workunit.client.1.smithi134.stdout:5/995: sync 2024-04-22T23:24:00.536 INFO:tasks.workunit.client.1.smithi134.stdout:3/792: readlink dc/d2a/d62/db2/d2f/d36/d6d/l81 0 2024-04-22T23:24:00.539 INFO:tasks.workunit.client.1.smithi134.stdout:7/996: symlink d4/d8/d41/d5b/d104/l160 0 2024-04-22T23:24:00.546 INFO:tasks.workunit.client.1.smithi134.stdout:9/841: creat d0/d2d/d82/df5/d10d/f120 x:0 0 0 2024-04-22T23:24:00.547 INFO:tasks.workunit.client.1.smithi134.stdout:9/842: stat d0/d7/d23/d26/d4d/d5d/d8e/dae/db1/cba 0 2024-04-22T23:24:00.552 INFO:tasks.workunit.client.1.smithi134.stdout:3/793: creat dc/d2a/d62/db2/d2f/d21/f129 x:0 0 0 2024-04-22T23:24:00.553 INFO:tasks.workunit.client.1.smithi134.stdout:3/794: chown dc/d2a/d62/d86/dc1/lff 3794 1 2024-04-22T23:24:00.554 INFO:tasks.workunit.client.1.smithi134.stdout:8/917: mkdir d7/dc/d2c/d4f/d6c/d128 0 2024-04-22T23:24:00.557 INFO:tasks.workunit.client.1.smithi134.stdout:6/976: link d0/d19/db/fec d0/d19/f143 0 2024-04-22T23:24:00.558 INFO:tasks.workunit.client.1.smithi134.stdout:6/977: write d0/d19/de/dbe/f10b [907234,24714] 0 2024-04-22T23:24:00.565 INFO:tasks.workunit.client.1.smithi134.stdout:2/894: link d2/d15/d24/d1d/d3d/d7b/ff7 d2/d42/d54/d33/d6e/f132 0 2024-04-22T23:24:00.570 INFO:tasks.workunit.client.1.smithi134.stdout:5/996: mkdir d9/d11/d34/d4d/dd4/d10f/d14b 0 2024-04-22T23:24:00.573 INFO:tasks.workunit.client.1.smithi134.stdout:3/795: rename dc/f18 to dc/d2a/d30/f12a 0 2024-04-22T23:24:00.573 INFO:tasks.workunit.client.1.smithi134.stdout:5/997: truncate d9/dbb/d11b/d17/d2a/f98 1951739 0 2024-04-22T23:24:00.576 INFO:tasks.workunit.client.1.smithi134.stdout:9/843: dwrite d0/d7/fad [0,4194304] 0 2024-04-22T23:24:00.583 INFO:tasks.workunit.client.1.smithi134.stdout:6/978: creat d0/d19/db/d75/d92/f144 x:0 0 0 2024-04-22T23:24:00.590 INFO:tasks.workunit.client.1.smithi134.stdout:5/998: fsync d9/f62 0 2024-04-22T23:24:00.591 INFO:tasks.workunit.client.1.smithi134.stdout:7/997: write d4/d8/d41/d76/dc1/d2f/d70/d86/ff0 [701073,44759] 0 2024-04-22T23:24:00.593 INFO:tasks.workunit.client.1.smithi134.stdout:3/796: rmdir dc/d2a/d62/db2/d2f/d36/d6d/d6f/dd2 39 2024-04-22T23:24:00.595 INFO:tasks.workunit.client.1.smithi134.stdout:9/844: creat d0/d2d/d94/f121 x:0 0 0 2024-04-22T23:24:00.596 INFO:tasks.workunit.client.1.smithi134.stdout:5/999: dread d9/d1e/f5a [0,4194304] 0 2024-04-22T23:24:00.598 INFO:tasks.workunit.client.1.smithi134.stdout:8/918: dread d7/dc/d2c/d4f/f6b [0,4194304] 0 2024-04-22T23:24:00.599 INFO:tasks.workunit.client.1.smithi134.stdout:7/998: creat d4/d8/d41/d76/dc1/d2f/d69/dab/d95/f161 x:0 0 0 2024-04-22T23:24:00.601 INFO:tasks.workunit.client.1.smithi134.stdout:7/999: readlink d4/d8/d41/d76/dc1/d2f/d70/d86/l12b 0 2024-04-22T23:24:00.608 INFO:tasks.workunit.client.1.smithi134.stdout:6/979: dread d0/d19/de/d12/d2a/d2d/f49 [0,4194304] 0 2024-04-22T23:24:00.609 INFO:tasks.workunit.client.1.smithi134.stdout:8/919: link d7/dc/d2c/d37/f4d d7/db/d42/d65/d46/d9a/da3/f129 0 2024-04-22T23:24:00.609 INFO:tasks.workunit.client.1.smithi134.stdout:8/920: chown d7/dc/d124 1 1 2024-04-22T23:24:00.610 INFO:tasks.workunit.client.1.smithi134.stdout:3/797: getdents dc/d2a/d62/d86 0 2024-04-22T23:24:00.611 INFO:tasks.workunit.client.1.smithi134.stdout:3/798: readlink dc/d2a/d30/l92 0 2024-04-22T23:24:00.613 INFO:tasks.workunit.client.1.smithi134.stdout:6/980: rename d0/d6f/f13f to d0/d19/db/f145 0 2024-04-22T23:24:00.615 INFO:tasks.workunit.client.1.smithi134.stdout:3/799: mkdir dc/d2a/d62/db2/d2f/d21/d42/d12b 0 2024-04-22T23:24:00.621 INFO:tasks.workunit.client.1.smithi134.stdout:3/800: truncate dc/d2a/d62/db2/d2f/d36/ff9 3671413 0 2024-04-22T23:24:00.621 INFO:tasks.workunit.client.1.smithi134.stdout:3/801: stat dc/d2a/d62/db2/d3b 0 2024-04-22T23:24:00.623 INFO:tasks.workunit.client.1.smithi134.stdout:3/802: chown dc/d2a/d62/db2/f3a 9204 1 2024-04-22T23:24:00.626 INFO:tasks.workunit.client.1.smithi134.stdout:3/803: creat dc/d2a/d62/d103/f12c x:0 0 0 2024-04-22T23:24:00.628 INFO:tasks.workunit.client.1.smithi134.stdout:3/804: write dc/d2a/d62/db2/d2f/d36/f11c [373476,13347] 0 2024-04-22T23:24:00.635 INFO:tasks.workunit.client.1.smithi134.stdout:9/845: dwrite d0/d7/d12/f30 [0,4194304] 0 2024-04-22T23:24:00.637 INFO:tasks.workunit.client.1.smithi134.stdout:9/846: dread - d0/d2d/d82/df5/d10d/f120 zero size 2024-04-22T23:24:00.639 INFO:tasks.workunit.client.1.smithi134.stdout:3/805: dwrite dc/d2a/d62/db2/d2f/d36/d5f/fd3 [0,4194304] 0 2024-04-22T23:24:00.640 INFO:tasks.workunit.client.1.smithi134.stdout:8/921: write d7/dc/d19/da5/fab [982571,7368] 0 2024-04-22T23:24:00.642 INFO:tasks.workunit.client.1.smithi134.stdout:3/806: chown dc/d2a/d62/d86/dc1/dd8/fe4 14 1 2024-04-22T23:24:00.643 INFO:tasks.workunit.client.1.smithi134.stdout:6/981: write d0/d19/db/d75/d92/fdc [40458,42734] 0 2024-04-22T23:24:00.647 INFO:tasks.workunit.client.1.smithi134.stdout:8/922: symlink d7/db/d42/d65/dd9/dda/de4/l12a 0 2024-04-22T23:24:00.648 INFO:tasks.workunit.client.1.smithi134.stdout:3/807: write dc/d2a/d62/db2/d2f/d21/d42/d77/fd4 [873144,38844] 0 2024-04-22T23:24:00.653 INFO:tasks.workunit.client.1.smithi134.stdout:9/847: dwrite d0/d7/d23/d1b/d5a/f116 [0,4194304] 0 2024-04-22T23:24:00.653 INFO:tasks.workunit.client.1.smithi134.stdout:6/982: write d0/d19/de/d12/d29/d74/deb/ffc [850599,101330] 0 2024-04-22T23:24:00.653 INFO:tasks.workunit.client.1.smithi134.stdout:8/923: creat d7/dc/de0/f12b x:0 0 0 2024-04-22T23:24:00.660 INFO:tasks.workunit.client.1.smithi134.stdout:3/808: mknod dc/d2a/d62/d103/d11e/c12d 0 2024-04-22T23:24:00.662 INFO:tasks.workunit.client.1.smithi134.stdout:6/983: symlink d0/d19/de/d12/d29/d74/da0/l146 0 2024-04-22T23:24:00.663 INFO:tasks.workunit.client.1.smithi134.stdout:9/848: mknod d0/d7/d23/d26/d4d/d5d/d8e/dae/db1/c122 0 2024-04-22T23:24:00.664 INFO:tasks.workunit.client.1.smithi134.stdout:3/809: mknod dc/d2a/d62/db2/d2f/d21/c12e 0 2024-04-22T23:24:00.665 INFO:tasks.workunit.client.1.smithi134.stdout:8/924: read d7/db/d42/d65/d46/d9a/da3/d88/fd0 [1407448,73363] 0 2024-04-22T23:24:00.669 INFO:tasks.workunit.client.1.smithi134.stdout:8/925: read d7/dc/d2c/d37/d47/f66 [364796,20229] 0 2024-04-22T23:24:00.670 INFO:tasks.workunit.client.1.smithi134.stdout:8/926: write d7/dc/d2c/d37/d47/ff4 [3786333,74597] 0 2024-04-22T23:24:00.671 INFO:tasks.workunit.client.1.smithi134.stdout:8/927: write d7/fe9 [2417203,45043] 0 2024-04-22T23:24:00.677 INFO:tasks.workunit.client.1.smithi134.stdout:8/928: symlink d7/db/d42/d65/d46/d9a/da3/d88/l12c 0 2024-04-22T23:24:00.690 INFO:tasks.workunit.client.1.smithi134.stdout:3/810: dread dc/d2a/d62/db2/d2f/d36/d6d/d80/fa2 [0,4194304] 0 2024-04-22T23:24:00.696 INFO:tasks.workunit.client.1.smithi134.stdout:3/811: creat dc/d2a/d62/db2/f12f x:0 0 0 2024-04-22T23:24:00.697 INFO:tasks.workunit.client.1.smithi134.stdout:8/929: dread d7/dc/d19/da5/fcc [0,4194304] 0 2024-04-22T23:24:00.699 INFO:tasks.workunit.client.1.smithi134.stdout:6/984: write d0/d19/de/d1f/f22 [3126394,107331] 0 2024-04-22T23:24:00.700 INFO:tasks.workunit.client.1.smithi134.stdout:0/979: sync 2024-04-22T23:24:00.701 INFO:tasks.workunit.client.1.smithi134.stdout:3/812: symlink dc/d2a/d62/db2/d2f/d61/d6a/l130 0 2024-04-22T23:24:00.703 INFO:tasks.workunit.client.1.smithi134.stdout:3/813: truncate dc/d2a/d62/db2/d2f/d36/d6d/ff4 1535359 0 2024-04-22T23:24:00.706 INFO:tasks.workunit.client.1.smithi134.stdout:3/814: fsync dc/d2a/d62/db2/d3b/d29/fb7 0 2024-04-22T23:24:00.709 INFO:tasks.workunit.client.1.smithi134.stdout:9/849: dwrite d0/d7/f19 [0,4194304] 0 2024-04-22T23:24:00.712 INFO:tasks.workunit.client.1.smithi134.stdout:8/930: creat d7/db/d2d/f12d x:0 0 0 2024-04-22T23:24:00.719 INFO:tasks.workunit.client.1.smithi134.stdout:0/980: mkdir d1/d5/d58/d6b/db9/d144 0 2024-04-22T23:24:00.719 INFO:tasks.workunit.client.1.smithi134.stdout:2/895: sync 2024-04-22T23:24:00.725 INFO:tasks.workunit.client.1.smithi134.stdout:9/850: creat d0/d7/d12/de8/f123 x:0 0 0 2024-04-22T23:24:00.728 INFO:tasks.workunit.client.1.smithi134.stdout:3/815: unlink dc/d2a/d62/db2/d2f/d21/d42/d77/f120 0 2024-04-22T23:24:00.730 INFO:tasks.workunit.client.1.smithi134.stdout:8/931: mkdir d7/dc/de0/d12e 0 2024-04-22T23:24:00.731 INFO:tasks.workunit.client.1.smithi134.stdout:6/985: dwrite d0/f26 [0,4194304] 0 2024-04-22T23:24:00.733 INFO:tasks.workunit.client.1.smithi134.stdout:9/851: dread d0/d7/d23/d1b/f117 [0,4194304] 0 2024-04-22T23:24:00.735 INFO:tasks.workunit.client.1.smithi134.stdout:0/981: symlink d1/d2/d117/d13a/de6/d13d/d142/l145 0 2024-04-22T23:24:00.736 INFO:tasks.workunit.client.1.smithi134.stdout:8/932: fsync d7/db/d42/d65/d46/d9a/da3/d88/fd0 0 2024-04-22T23:24:00.737 INFO:tasks.workunit.client.1.smithi134.stdout:0/982: fsync d1/d5/dd/d20/d127/f133 0 2024-04-22T23:24:00.739 INFO:tasks.workunit.client.1.smithi134.stdout:6/986: mkdir d0/d19/de/d12/d2a/d2d/d147 0 2024-04-22T23:24:00.745 INFO:tasks.workunit.client.1.smithi134.stdout:6/987: chown d0/d19/de/d12/d5e/d6d/d7a/d99/fb1 252018137 1 2024-04-22T23:24:00.746 INFO:tasks.workunit.client.1.smithi134.stdout:3/816: dwrite dc/d2a/d62/db2/d2f/d36/d6d/d6f/dd9/f121 [0,4194304] 0 2024-04-22T23:24:00.749 INFO:tasks.workunit.client.1.smithi134.stdout:2/896: getdents d2/dd/df 0 2024-04-22T23:24:00.750 INFO:tasks.workunit.client.1.smithi134.stdout:9/852: dwrite d0/f54 [0,4194304] 0 2024-04-22T23:24:00.751 INFO:tasks.workunit.client.1.smithi134.stdout:8/933: write d7/db/d42/d65/dd9/dda/fbf [3877736,60203] 0 2024-04-22T23:24:00.751 INFO:tasks.workunit.client.1.smithi134.stdout:2/897: fdatasync d2/d15/d24/d3b/d8a/d99/fef 0 2024-04-22T23:24:00.753 INFO:tasks.workunit.client.1.smithi134.stdout:9/853: dread - d0/d2d/d3d/f86 zero size 2024-04-22T23:24:00.759 INFO:tasks.workunit.client.1.smithi134.stdout:2/898: write d2/d15/d24/f1e [3909731,18917] 0 2024-04-22T23:24:00.759 INFO:tasks.workunit.client.1.smithi134.stdout:0/983: fsync d1/d2/d117/d13a/de6/fed 0 2024-04-22T23:24:00.762 INFO:tasks.workunit.client.1.smithi134.stdout:3/817: unlink f9 0 2024-04-22T23:24:00.764 INFO:tasks.workunit.client.1.smithi134.stdout:6/988: creat d0/d6f/d88/d8b/f148 x:0 0 0 2024-04-22T23:24:00.764 INFO:tasks.workunit.client.1.smithi134.stdout:9/854: dwrite d0/d7/d23/d1b/de2/fec [0,4194304] 0 2024-04-22T23:24:00.768 INFO:tasks.workunit.client.1.smithi134.stdout:8/934: mknod d7/db/d42/d65/dd9/dda/de7/ddd/c12f 0 2024-04-22T23:24:00.774 INFO:tasks.workunit.client.1.smithi134.stdout:2/899: fsync d2/dd/d4a/d73/fdb 0 2024-04-22T23:24:00.782 INFO:tasks.workunit.client.1.smithi134.stdout:6/989: fsync d0/d19/de/d12/d5e/d6d/d7a/db2/fd7 0 2024-04-22T23:24:00.784 INFO:tasks.workunit.client.1.smithi134.stdout:3/818: rename dc/d2a/d62/d86/df6/c118 to dc/d2a/d62/db2/d2f/d36/d6d/d80/c131 0 2024-04-22T23:24:00.795 INFO:tasks.workunit.client.1.smithi134.stdout:0/984: dread d1/db/f70 [0,4194304] 0 2024-04-22T23:24:00.798 INFO:tasks.workunit.client.1.smithi134.stdout:3/819: dread dc/d2a/d62/d86/dc1/fe2 [0,4194304] 0 2024-04-22T23:24:00.799 INFO:tasks.workunit.client.1.smithi134.stdout:8/935: mknod d7/dc/de0/d12e/c130 0 2024-04-22T23:24:00.800 INFO:tasks.workunit.client.1.smithi134.stdout:3/820: write dc/d2a/d62/db2/d34/d6b/f7c [1191499,127985] 0 2024-04-22T23:24:00.802 INFO:tasks.workunit.client.1.smithi134.stdout:9/855: creat d0/d37/f124 x:0 0 0 2024-04-22T23:24:00.803 INFO:tasks.workunit.client.1.smithi134.stdout:9/856: readlink d0/d7/d23/d26/d51/l4b 0 2024-04-22T23:24:00.809 INFO:tasks.workunit.client.1.smithi134.stdout:9/857: dwrite d0/d7/d27/fa5 [4194304,4194304] 0 2024-04-22T23:24:00.813 INFO:tasks.workunit.client.1.smithi134.stdout:8/936: mknod d7/dc/d2c/d4f/c131 0 2024-04-22T23:24:00.818 INFO:tasks.workunit.client.1.smithi134.stdout:0/985: dwrite d1/d2/d1d/f5f [8388608,4194304] 0 2024-04-22T23:24:00.822 INFO:tasks.workunit.client.1.smithi134.stdout:6/990: creat d0/d19/de/f149 x:0 0 0 2024-04-22T23:24:00.831 INFO:tasks.workunit.client.1.smithi134.stdout:2/900: link d2/dd/d4a/d73/l7d d2/d15/d24/d1d/l133 0 2024-04-22T23:24:00.831 INFO:tasks.workunit.client.1.smithi134.stdout:3/821: link dc/d2a/d30/l92 dc/d2a/d62/d86/l132 0 2024-04-22T23:24:00.833 INFO:tasks.workunit.client.1.smithi134.stdout:0/986: symlink d1/d5/d58/d22/dd0/ddf/d129/l146 0 2024-04-22T23:24:00.834 INFO:tasks.workunit.client.1.smithi134.stdout:2/901: mkdir d2/d15/d24/d1d/d3d/d9a/d134 0 2024-04-22T23:24:00.836 INFO:tasks.workunit.client.1.smithi134.stdout:6/991: creat d0/d19/de/d12/d2a/d2d/d3a/d98/f14a x:0 0 0 2024-04-22T23:24:00.838 INFO:tasks.workunit.client.1.smithi134.stdout:0/987: symlink d1/d5/d15/l147 0 2024-04-22T23:24:00.843 INFO:tasks.workunit.client.1.smithi134.stdout:3/822: dwrite dc/d2a/d62/db2/d3b/f31 [4194304,4194304] 0 2024-04-22T23:24:00.843 INFO:tasks.workunit.client.1.smithi134.stdout:8/937: dread d7/db/d2d/d39/d58/f115 [0,4194304] 0 2024-04-22T23:24:00.849 INFO:tasks.workunit.client.1.smithi134.stdout:2/902: dwrite d2/d42/d54/fd6 [0,4194304] 0 2024-04-22T23:24:00.850 INFO:tasks.workunit.client.1.smithi134.stdout:9/858: truncate d0/d7/d23/f2a 5966371 0 2024-04-22T23:24:00.851 INFO:tasks.workunit.client.1.smithi134.stdout:2/903: fsync d2/dd/d72/fab 0 2024-04-22T23:24:00.854 INFO:tasks.workunit.client.1.smithi134.stdout:8/938: rmdir d7/db/d42/d65/d46/d84 39 2024-04-22T23:24:00.854 INFO:tasks.workunit.client.1.smithi134.stdout:3/823: truncate dc/d2a/d62/db2/d2f/d21/d42/d77/fd4 1517140 0 2024-04-22T23:24:00.856 INFO:tasks.workunit.client.1.smithi134.stdout:6/992: dwrite d0/d19/db/d75/f12c [0,4194304] 0 2024-04-22T23:24:00.860 INFO:tasks.workunit.client.1.smithi134.stdout:9/859: fsync d0/d2d/d3d/f9e 0 2024-04-22T23:24:00.862 INFO:tasks.workunit.client.1.smithi134.stdout:6/993: symlink d0/d19/de/d1f/dd5/l14b 0 2024-04-22T23:24:00.864 INFO:tasks.workunit.client.1.smithi134.stdout:8/939: creat d7/dc/d2c/d4f/d6c/d128/f132 x:0 0 0 2024-04-22T23:24:00.866 INFO:tasks.workunit.client.1.smithi134.stdout:9/860: symlink d0/d7/d12/de8/df7/l125 0 2024-04-22T23:24:00.869 INFO:tasks.workunit.client.1.smithi134.stdout:3/824: dwrite dc/d2a/d62/db2/d3b/f31 [4194304,4194304] 0 2024-04-22T23:24:00.869 INFO:tasks.workunit.client.1.smithi134.stdout:8/940: chown d7/db/d42/d65/d46/d9a/da3/f10f 470 1 2024-04-22T23:24:00.872 INFO:tasks.workunit.client.1.smithi134.stdout:3/825: dread - dc/d2a/d62/d86/dc1/df7/f112 zero size 2024-04-22T23:24:00.877 INFO:tasks.workunit.client.1.smithi134.stdout:8/941: chown d7/dc/d2c/d37/d47/ff4 73648272 1 2024-04-22T23:24:00.882 INFO:tasks.workunit.client.1.smithi134.stdout:3/826: creat dc/d2a/d62/d86/dc1/dd8/f133 x:0 0 0 2024-04-22T23:24:00.884 INFO:tasks.workunit.client.1.smithi134.stdout:8/942: creat d7/dc/d19/da5/f133 x:0 0 0 2024-04-22T23:24:00.885 INFO:tasks.workunit.client.1.smithi134.stdout:2/904: dwrite d2/dd/d68/f88 [0,4194304] 0 2024-04-22T23:24:00.888 INFO:tasks.workunit.client.1.smithi134.stdout:0/988: dread d1/d5/d58/f80 [0,4194304] 0 2024-04-22T23:24:00.893 INFO:tasks.workunit.client.1.smithi134.stdout:9/861: getdents d0/d2d 0 2024-04-22T23:24:00.895 INFO:tasks.workunit.client.1.smithi134.stdout:0/989: rename d1/d5/d15/fe9 to d1/d5/d58/d6b/db9/def/f148 0 2024-04-22T23:24:00.896 INFO:tasks.workunit.client.1.smithi134.stdout:8/943: unlink d7/db/d2d/d39/d58/c63 0 2024-04-22T23:24:00.897 INFO:tasks.workunit.client.1.smithi134.stdout:8/944: stat d7/db/d2d/d39/f3d 0 2024-04-22T23:24:00.902 INFO:tasks.workunit.client.1.smithi134.stdout:3/827: getdents dc/d2a/d62/db2/d34 0 2024-04-22T23:24:00.904 INFO:tasks.workunit.client.1.smithi134.stdout:2/905: symlink d2/d15/d24/d1d/d3d/d9a/d134/l135 0 2024-04-22T23:24:00.905 INFO:tasks.workunit.client.1.smithi134.stdout:9/862: symlink d0/d2d/dc1/df8/l126 0 2024-04-22T23:24:00.906 INFO:tasks.workunit.client.1.smithi134.stdout:6/994: dwrite d0/d19/de/d12/d5e/dc1/ff2 [0,4194304] 0 2024-04-22T23:24:00.907 INFO:tasks.workunit.client.1.smithi134.stdout:0/990: fsync d1/d5/d58/d6b/db9/f6d 0 2024-04-22T23:24:00.907 INFO:tasks.workunit.client.1.smithi134.stdout:2/906: write d2/dd/df/d9c/daa/ff4 [1040820,104753] 0 2024-04-22T23:24:00.918 INFO:tasks.workunit.client.1.smithi134.stdout:3/828: fsync dc/d2a/d62/db2/d3b/d29/fb8 0 2024-04-22T23:24:00.921 INFO:tasks.workunit.client.1.smithi134.stdout:9/863: fsync d0/d2d/d3d/f86 0 2024-04-22T23:24:00.956 INFO:tasks.workunit.client.1.smithi134.stdout:0/991: fdatasync d1/d2/d117/d13a/f119 0 2024-04-22T23:24:00.957 INFO:tasks.workunit.client.1.smithi134.stdout:0/992: chown d1/d2/fe2 258316757 1 2024-04-22T23:24:00.958 INFO:tasks.workunit.client.1.smithi134.stdout:2/907: rmdir d2/d15/d24/d1d/d3d 39 2024-04-22T23:24:00.960 INFO:tasks.workunit.client.1.smithi134.stdout:3/829: unlink dc/d2a/d62/db2/d34/d6b/dae/f85 0 2024-04-22T23:24:00.962 INFO:tasks.workunit.client.1.smithi134.stdout:9/864: unlink d0/d7/d23/d26/d51/d4c/cc9 0 2024-04-22T23:24:00.965 INFO:tasks.workunit.client.1.smithi134.stdout:0/993: mknod d1/d5/d58/da0/d13f/c149 0 2024-04-22T23:24:00.967 INFO:tasks.workunit.client.1.smithi134.stdout:3/830: creat dc/d2a/d62/db2/d2f/d21/d42/f134 x:0 0 0 2024-04-22T23:24:00.968 INFO:tasks.workunit.client.1.smithi134.stdout:9/865: truncate d0/d2d/d82/dbf/fd1 899691 0 2024-04-22T23:24:00.970 INFO:tasks.workunit.client.1.smithi134.stdout:3/831: write dc/d2a/d62/db2/d2f/d36/d6d/ff4 [627396,39056] 0 2024-04-22T23:24:00.970 INFO:tasks.workunit.client.1.smithi134.stdout:2/908: mkdir d2/d15/d24/d1d/d3d/dfe/d136 0 2024-04-22T23:24:00.970 INFO:tasks.workunit.client.1.smithi134.stdout:0/994: dread d1/d5/dd/d20/d2c/f4b [0,4194304] 0 2024-04-22T23:24:00.972 INFO:tasks.workunit.client.1.smithi134.stdout:0/995: chown d1/d5/d58/d22/dcb/l12a 104893194 1 2024-04-22T23:24:00.973 INFO:tasks.workunit.client.1.smithi134.stdout:9/866: mknod d0/d7/d12/d22/c127 0 2024-04-22T23:24:00.976 INFO:tasks.workunit.client.1.smithi134.stdout:3/832: symlink dc/d2a/d62/d86/df6/l135 0 2024-04-22T23:24:00.977 INFO:tasks.workunit.client.1.smithi134.stdout:0/996: mknod d1/d5/da2/c14a 0 2024-04-22T23:24:01.003 INFO:tasks.workunit.client.1.smithi134.stdout:3/833: write dc/d2a/d62/db2/d2f/d54/f7b [237620,77975] 0 2024-04-22T23:24:01.009 INFO:tasks.workunit.client.1.smithi134.stdout:2/909: dwrite d2/dd/d68/fa9 [0,4194304] 0 2024-04-22T23:24:01.012 INFO:tasks.workunit.client.1.smithi134.stdout:9/867: dwrite d0/d2d/d94/fed [0,4194304] 0 2024-04-22T23:24:01.016 INFO:tasks.workunit.client.1.smithi134.stdout:0/997: dwrite d1/f44 [4194304,4194304] 0 2024-04-22T23:24:01.016 INFO:tasks.workunit.client.1.smithi134.stdout:3/834: creat dc/d2a/d62/db2/d2f/d21/d42/d77/d8f/f136 x:0 0 0 2024-04-22T23:24:01.016 INFO:tasks.workunit.client.1.smithi134.stdout:9/868: fsync d0/d7/d12/d22/fea 0 2024-04-22T23:24:01.018 INFO:tasks.workunit.client.1.smithi134.stdout:2/910: rmdir d2/dd/df/dc2 39 2024-04-22T23:24:01.022 INFO:tasks.workunit.client.1.smithi134.stdout:9/869: creat d0/d2d/d3d/f128 x:0 0 0 2024-04-22T23:24:01.027 INFO:tasks.workunit.client.1.smithi134.stdout:3/835: rename dc/d2a/d62/d86 to dc/d2a/d62/db2/d2f/d21/d42/d77/d137 0 2024-04-22T23:24:01.031 INFO:tasks.workunit.client.1.smithi134.stdout:9/870: rename d0/d7/d12/l50 to d0/d2d/dc1/df8/l129 0 2024-04-22T23:24:01.032 INFO:tasks.workunit.client.1.smithi134.stdout:9/871: stat d0/d2d/d82/df5 0 2024-04-22T23:24:01.035 INFO:tasks.workunit.client.1.smithi134.stdout:3/836: dwrite dc/d2a/d62/db2/d2f/d36/d6d/d6f/dd9/fe0 [0,4194304] 0 2024-04-22T23:24:01.036 INFO:tasks.workunit.client.1.smithi134.stdout:2/911: link d2/d15/d24/d3b/d8a/d99/fef d2/d15/f137 0 2024-04-22T23:24:01.040 INFO:tasks.workunit.client.1.smithi134.stdout:9/872: rmdir d0/d7/d23/d26/d4d/d5d 39 2024-04-22T23:24:01.040 INFO:tasks.workunit.client.1.smithi134.stdout:3/837: dread - dc/d2a/d62/db2/d2f/d21/d42/d77/d137/dc1/f109 zero size 2024-04-22T23:24:01.041 INFO:tasks.workunit.client.1.smithi134.stdout:9/873: chown d0/d7/d23/d1b/cd3 629 1 2024-04-22T23:24:01.045 INFO:tasks.workunit.client.1.smithi134.stdout:2/912: rename d2/dd/d4a/f52 to d2/dd/d4a/f138 0 2024-04-22T23:24:01.048 INFO:tasks.workunit.client.1.smithi134.stdout:0/998: write d1/d2/f118 [912535,85239] 0 2024-04-22T23:24:01.049 INFO:tasks.workunit.client.1.smithi134.stdout:9/874: chown d0/d2d/d82/dbf/ldf 269908 1 2024-04-22T23:24:01.049 INFO:tasks.workunit.client.1.smithi134.stdout:0/999: stat d1/d5/dd/d41/f11b 0 2024-04-22T23:24:01.055 INFO:tasks.workunit.client.1.smithi134.stdout:2/913: fsync d2/d15/d24/d1d/d3d/f58 0 2024-04-22T23:24:01.060 INFO:tasks.workunit.client.1.smithi134.stdout:3/838: dwrite dc/d2a/f10e [0,4194304] 0 2024-04-22T23:24:01.066 INFO:tasks.workunit.client.1.smithi134.stdout:9/875: getdents d0/d7/d12/de8/df7 0 2024-04-22T23:24:01.069 INFO:tasks.workunit.client.1.smithi134.stdout:2/914: dwrite d2/fde [0,4194304] 0 2024-04-22T23:24:01.075 INFO:tasks.workunit.client.1.smithi134.stdout:9/876: chown d0/d7/d23/d26/d4d/d5d/c61 8759 1 2024-04-22T23:24:01.076 INFO:tasks.workunit.client.1.smithi134.stdout:2/915: read - d2/dd/df/dc2/ffc zero size 2024-04-22T23:24:01.076 INFO:tasks.workunit.client.1.smithi134.stdout:3/839: dwrite dc/d2a/d62/db2/d2f/d21/d42/d77/d137/f8a [0,4194304] 0 2024-04-22T23:24:01.077 INFO:tasks.workunit.client.1.smithi134.stdout:9/877: chown d0/d2d/f9f 229428510 1 2024-04-22T23:24:01.081 INFO:tasks.workunit.client.1.smithi134.stdout:3/840: chown dc/d2a/d62/db2/d2f/d54/f126 1 1 2024-04-22T23:24:01.085 INFO:tasks.workunit.client.1.smithi134.stdout:9/878: rename d0/d7/d23/d26/d4d/d5d/d8e/dae/db1/cba to d0/d7/d23/d26/d51/d4c/c12a 0 2024-04-22T23:24:01.085 INFO:tasks.workunit.client.1.smithi134.stdout:2/916: mknod d2/d15/d24/d3b/d8a/d99/dfb/c139 0 2024-04-22T23:24:01.088 INFO:tasks.workunit.client.1.smithi134.stdout:9/879: unlink d0/d7/d23/d26/d4d/d5d/f8f 0 2024-04-22T23:24:01.089 INFO:tasks.workunit.client.1.smithi134.stdout:2/917: fdatasync d2/d15/d24/d1d/f86 0 2024-04-22T23:24:01.091 INFO:tasks.workunit.client.1.smithi134.stdout:9/880: symlink d0/d2d/d82/da0/l12b 0 2024-04-22T23:24:01.092 INFO:tasks.workunit.client.1.smithi134.stdout:9/881: readlink d0/d2d/d82/dbf/lca 0 2024-04-22T23:24:01.092 INFO:tasks.workunit.client.1.smithi134.stdout:9/882: stat d0/d2d/c107 0 2024-04-22T23:24:01.094 INFO:tasks.workunit.client.1.smithi134.stdout:3/841: dread dc/d2a/d62/db2/d3b/d29/fb7 [0,4194304] 0 2024-04-22T23:24:01.095 INFO:tasks.workunit.client.1.smithi134.stdout:9/883: symlink d0/d7/d23/d26/l12c 0 2024-04-22T23:24:01.098 INFO:tasks.workunit.client.1.smithi134.stdout:2/918: creat d2/dd/df/d9c/dca/f13a x:0 0 0 2024-04-22T23:24:01.102 INFO:tasks.workunit.client.1.smithi134.stdout:3/842: dwrite dc/d2a/d62/db2/d2f/d21/d42/d77/d137/dc1/dd8/fe4 [0,4194304] 0 2024-04-22T23:24:01.103 INFO:tasks.workunit.client.1.smithi134.stdout:2/919: creat d2/d15/d24/f13b x:0 0 0 2024-04-22T23:24:01.107 INFO:tasks.workunit.client.1.smithi134.stdout:3/843: rmdir dc/d2a/d62/db2/d2f/d21/d42 39 2024-04-22T23:24:01.108 INFO:tasks.workunit.client.1.smithi134.stdout:2/920: truncate d2/dd/df/fb9 927772 0 2024-04-22T23:24:01.109 INFO:tasks.workunit.client.1.smithi134.stdout:2/921: write d2/fde [3312978,89317] 0 2024-04-22T23:24:01.109 INFO:tasks.workunit.client.1.smithi134.stdout:9/884: chown d0/d2d/d4e/f62 9627 1 2024-04-22T23:24:01.111 INFO:tasks.workunit.client.1.smithi134.stdout:2/922: write d2/dd/d116/f11f [924672,27147] 0 2024-04-22T23:24:01.114 INFO:tasks.workunit.client.1.smithi134.stdout:2/923: chown d2/d15/d24/d1d/d3d/d7b/f7c 1177749 1 2024-04-22T23:24:01.115 INFO:tasks.workunit.client.1.smithi134.stdout:3/844: symlink dc/d2a/d62/db2/d2f/d21/d42/d77/d137/db3/l138 0 2024-04-22T23:24:01.117 INFO:tasks.workunit.client.1.smithi134.stdout:3/845: stat dc/d2a/d62/db2/d2f/d21/d42 0 2024-04-22T23:24:01.118 INFO:tasks.workunit.client.1.smithi134.stdout:3/846: readlink dc/d2a/d62/db2/l26 0 2024-04-22T23:24:01.118 INFO:tasks.workunit.client.1.smithi134.stdout:9/885: rename d0/d7/d12/l14 to d0/d2d/l12d 0 2024-04-22T23:24:01.125 INFO:tasks.workunit.client.1.smithi134.stdout:9/886: readlink d0/d2d/dc1/df8/l126 0 2024-04-22T23:24:01.127 INFO:tasks.workunit.client.1.smithi134.stdout:2/924: write d2/dd/d68/f10f [49407,75464] 0 2024-04-22T23:24:01.130 INFO:tasks.workunit.client.1.smithi134.stdout:3/847: dwrite dc/dd/f7a [0,4194304] 0 2024-04-22T23:24:01.132 INFO:tasks.workunit.client.1.smithi134.stdout:3/848: chown dc/d2a/d62/db2/d2f/d36/d5f/l8b 158 1 2024-04-22T23:24:01.134 INFO:tasks.workunit.client.1.smithi134.stdout:9/887: rename d0/d7/d23/d1b/f101 to d0/d37/f12e 0 2024-04-22T23:24:01.137 INFO:tasks.workunit.client.1.smithi134.stdout:3/849: mknod dc/d2a/d62/db2/d2f/d21/d42/d77/c139 0 2024-04-22T23:24:01.144 INFO:tasks.workunit.client.1.smithi134.stdout:9/888: dwrite d0/d2d/d94/f106 [0,4194304] 0 2024-04-22T23:24:01.146 INFO:tasks.workunit.client.1.smithi134.stdout:3/850: creat dc/d2a/d62/db2/d2f/d21/d42/d77/d137/dc1/f13a x:0 0 0 2024-04-22T23:24:01.148 INFO:tasks.workunit.client.1.smithi134.stdout:2/925: dwrite d2/dd/d4a/d73/fdb [0,4194304] 0 2024-04-22T23:24:01.152 INFO:tasks.workunit.client.1.smithi134.stdout:9/889: creat d0/d2d/d82/f12f x:0 0 0 2024-04-22T23:24:01.155 INFO:tasks.workunit.client.1.smithi134.stdout:9/890: rmdir d0/d7/d23/d26/d4d/d5d/d8e/d105 39 2024-04-22T23:24:01.157 INFO:tasks.workunit.client.1.smithi134.stdout:2/926: getdents d2/d42/d54/d109 0 2024-04-22T23:24:01.159 INFO:tasks.workunit.client.1.smithi134.stdout:9/891: rmdir d0/d7/d23/d26 39 2024-04-22T23:24:01.166 INFO:tasks.workunit.client.1.smithi134.stdout:2/927: creat d2/dd/d4a/f13c x:0 0 0 2024-04-22T23:24:01.166 INFO:tasks.workunit.client.1.smithi134.stdout:3/851: dwrite dc/d2a/d62/f7f [4194304,4194304] 0 2024-04-22T23:24:01.168 INFO:tasks.workunit.client.1.smithi134.stdout:2/928: readlink d2/d42/d54/l8b 0 2024-04-22T23:24:01.178 INFO:tasks.workunit.client.1.smithi134.stdout:3/852: dwrite dc/d2a/d62/db2/d34/d6b/dae/f105 [0,4194304] 0 2024-04-22T23:24:01.181 INFO:tasks.workunit.client.1.smithi134.stdout:9/892: dwrite d0/d2d/d3d/f9e [0,4194304] 0 2024-04-22T23:24:01.187 INFO:tasks.workunit.client.1.smithi134.stdout:9/893: creat d0/d2d/d82/f130 x:0 0 0 2024-04-22T23:24:01.193 INFO:tasks.workunit.client.1.smithi134.stdout:3/853: dwrite dc/d2a/d62/db2/d2f/d21/d42/d77/d8f/f136 [0,4194304] 0 2024-04-22T23:24:01.198 INFO:tasks.workunit.client.1.smithi134.stdout:3/854: mknod dc/d2a/d30/d125/c13b 0 2024-04-22T23:24:01.202 INFO:tasks.workunit.client.1.smithi134.stdout:3/855: chown dc/d2a/d62/db2/d2f/d61/d6a/caf 251187 1 2024-04-22T23:24:01.203 INFO:tasks.workunit.client.1.smithi134.stdout:8/945: sync 2024-04-22T23:24:01.206 INFO:tasks.workunit.client.1.smithi134.stdout:8/946: symlink d7/db/d2d/d39/d105/l134 0 2024-04-22T23:24:01.208 INFO:tasks.workunit.client.1.smithi134.stdout:8/947: write d7/db/d2d/f12d [1015722,30080] 0 2024-04-22T23:24:01.218 INFO:tasks.workunit.client.1.smithi134.stdout:2/929: dread d2/d15/d24/d1d/d3d/d9a/fd5 [0,4194304] 0 2024-04-22T23:24:01.218 INFO:tasks.workunit.client.1.smithi134.stdout:9/894: dread d0/d7/ffc [0,4194304] 0 2024-04-22T23:24:01.225 INFO:tasks.workunit.client.1.smithi134.stdout:2/930: link d2/d15/f113 d2/dd/df/dc2/f13d 0 2024-04-22T23:24:01.226 INFO:tasks.workunit.client.1.smithi134.stdout:8/948: dwrite d7/f10c [0,4194304] 0 2024-04-22T23:24:01.230 INFO:tasks.workunit.client.1.smithi134.stdout:8/949: dread - d7/dc/d2c/d4f/ff0 zero size 2024-04-22T23:24:01.232 INFO:tasks.workunit.client.1.smithi134.stdout:9/895: dwrite d0/d7/f5b [0,4194304] 0 2024-04-22T23:24:01.238 INFO:tasks.workunit.client.1.smithi134.stdout:8/950: unlink d7/db/d2d/d39/d58/fe5 0 2024-04-22T23:24:01.243 INFO:tasks.workunit.client.1.smithi134.stdout:2/931: creat d2/d15/d24/d3b/d8a/d101/f13e x:0 0 0 2024-04-22T23:24:01.243 INFO:tasks.workunit.client.1.smithi134.stdout:9/896: dread - d0/d2d/d3d/f72 zero size 2024-04-22T23:24:01.245 INFO:tasks.workunit.client.1.smithi134.stdout:8/951: mknod d7/db/d42/c135 0 2024-04-22T23:24:01.250 INFO:tasks.workunit.client.1.smithi134.stdout:2/932: creat d2/d15/d24/d1d/d3d/d9a/f13f x:0 0 0 2024-04-22T23:24:01.250 INFO:tasks.workunit.client.1.smithi134.stdout:2/933: readlink d2/d15/d24/l5e 0 2024-04-22T23:24:01.257 INFO:tasks.workunit.client.1.smithi134.stdout:9/897: getdents d0/d7/d23/d26 0 2024-04-22T23:24:01.259 INFO:tasks.workunit.client.1.smithi134.stdout:2/934: dwrite d2/d15/d24/f13b [0,4194304] 0 2024-04-22T23:24:01.267 INFO:tasks.workunit.client.1.smithi134.stdout:9/898: unlink d0/d7/d23/d26/d4d/d5d/f10f 0 2024-04-22T23:24:01.270 INFO:tasks.workunit.client.1.smithi134.stdout:8/952: dread d7/db/d42/d65/d46/d9a/fa1 [0,4194304] 0 2024-04-22T23:24:01.273 INFO:tasks.workunit.client.1.smithi134.stdout:8/953: creat d7/db/d42/d65/dd9/f136 x:0 0 0 2024-04-22T23:24:01.277 INFO:tasks.workunit.client.1.smithi134.stdout:8/954: creat d7/db/d42/d65/d46/d9a/da3/f137 x:0 0 0 2024-04-22T23:24:01.278 INFO:tasks.workunit.client.1.smithi134.stdout:2/935: write d2/d42/d54/d33/d6e/f132 [136128,95025] 0 2024-04-22T23:24:01.284 INFO:tasks.workunit.client.1.smithi134.stdout:8/955: rename d7/db/d42/d65/dd9/dda/de7/ddd/dee/ffe to d7/db/d42/d65/d46/d120/f138 0 2024-04-22T23:24:01.293 INFO:tasks.workunit.client.1.smithi134.stdout:2/936: truncate d2/d15/d24/f13b 1844122 0 2024-04-22T23:24:01.293 INFO:tasks.workunit.client.1.smithi134.stdout:2/937: write d2/dd/d72/fae [1373513,6750] 0 2024-04-22T23:24:01.295 INFO:tasks.workunit.client.1.smithi134.stdout:2/938: fsync d2/d15/d24/d1d/d3d/d9a/f13f 0 2024-04-22T23:24:01.295 INFO:tasks.workunit.client.1.smithi134.stdout:2/939: stat d2/d42/d54/d33/c5f 0 2024-04-22T23:24:01.297 INFO:tasks.workunit.client.1.smithi134.stdout:8/956: getdents d7/dc/d2c/d37/d3a 0 2024-04-22T23:24:01.298 INFO:tasks.workunit.client.1.smithi134.stdout:2/940: mknod d2/dd/d68/c140 0 2024-04-22T23:24:01.300 INFO:tasks.workunit.client.1.smithi134.stdout:8/957: unlink d7/db/d42/d65/d46/l117 0 2024-04-22T23:24:01.303 INFO:tasks.workunit.client.1.smithi134.stdout:8/958: creat d7/db/d42/d65/dd9/f139 x:0 0 0 2024-04-22T23:24:01.303 INFO:tasks.workunit.client.1.smithi134.stdout:8/959: readlink d7/l28 0 2024-04-22T23:24:01.305 INFO:tasks.workunit.client.1.smithi134.stdout:8/960: write d7/db/d42/d65/d46/fdf [8719327,88522] 0 2024-04-22T23:24:01.306 INFO:tasks.workunit.client.1.smithi134.stdout:2/941: rename d2/dd/df/f6f to d2/d42/d54/f141 0 2024-04-22T23:24:01.309 INFO:tasks.workunit.client.1.smithi134.stdout:2/942: mkdir d2/d42/d54/d142 0 2024-04-22T23:24:01.312 INFO:tasks.workunit.client.1.smithi134.stdout:8/961: symlink d7/db/d42/d65/d46/d9a/l13a 0 2024-04-22T23:24:01.313 INFO:tasks.workunit.client.1.smithi134.stdout:8/962: write d7/db/d2d/d39/d58/f101 [242664,118571] 0 2024-04-22T23:24:01.314 INFO:tasks.workunit.client.1.smithi134.stdout:8/963: write d7/f23 [3892042,89154] 0 2024-04-22T23:24:01.321 INFO:tasks.workunit.client.1.smithi134.stdout:8/964: rename d7/db/d2d/l118 to d7/dc/de0/l13b 0 2024-04-22T23:24:01.323 INFO:tasks.workunit.client.1.smithi134.stdout:2/943: truncate d2/dd/df/fbb 48112 0 2024-04-22T23:24:01.324 INFO:tasks.workunit.client.1.smithi134.stdout:2/944: rename d2/d15/d24 to d2/d15/d24/d143 22 2024-04-22T23:24:01.328 INFO:tasks.workunit.client.1.smithi134.stdout:8/965: dwrite d7/fe9 [4194304,4194304] 0 2024-04-22T23:24:01.329 INFO:tasks.workunit.client.1.smithi134.stdout:2/945: truncate d2/dd/df/d1c/ff0 614441 0 2024-04-22T23:24:01.333 INFO:tasks.workunit.client.1.smithi134.stdout:8/966: mknod d7/db/d42/d65/dd9/dda/de4/d111/c13c 0 2024-04-22T23:24:01.336 INFO:tasks.workunit.client.1.smithi134.stdout:8/967: chown d7/dc/d2c/d4f/d6c/dd1/lfd 28 1 2024-04-22T23:24:01.338 INFO:tasks.workunit.client.1.smithi134.stdout:8/968: creat d7/dc/d2c/d4f/d6c/d128/f13d x:0 0 0 2024-04-22T23:24:01.339 INFO:tasks.workunit.client.1.smithi134.stdout:8/969: readlink d7/db/d42/d65/la4 0 2024-04-22T23:24:01.343 INFO:tasks.workunit.client.1.smithi134.stdout:8/970: fsync d7/dc/d2c/d37/d47/f123 0 2024-04-22T23:24:01.343 INFO:tasks.workunit.client.1.smithi134.stdout:8/971: chown d7/dc/f22 16369890 1 2024-04-22T23:24:01.345 INFO:tasks.workunit.client.1.smithi134.stdout:8/972: chown d7/db/d42/d65/dd9/dda/de7/fef 12623809 1 2024-04-22T23:24:01.347 INFO:tasks.workunit.client.1.smithi134.stdout:8/973: mknod d7/db/d42/d65/d46/c13e 0 2024-04-22T23:24:01.356 INFO:tasks.workunit.client.1.smithi134.stdout:8/974: link d7/dc/d2c/d4f/d6c/d6e/fca d7/dc/d2c/f13f 0 2024-04-22T23:24:01.358 INFO:tasks.workunit.client.1.smithi134.stdout:8/975: creat d7/db/d2d/d39/d105/f140 x:0 0 0 2024-04-22T23:24:01.371 INFO:tasks.workunit.client.1.smithi134.stdout:8/976: dread d7/f1c [0,4194304] 0 2024-04-22T23:24:01.372 INFO:tasks.workunit.client.1.smithi134.stdout:8/977: chown d7/dc/d19/da5/fb2 53 1 2024-04-22T23:24:01.384 INFO:tasks.workunit.client.1.smithi134.stdout:6/995: sync 2024-04-22T23:24:01.385 INFO:tasks.workunit.client.1.smithi134.stdout:6/996: dread - d0/d19/de/d12/d2a/d2d/d3a/d98/f14a zero size 2024-04-22T23:24:01.386 INFO:tasks.workunit.client.1.smithi134.stdout:8/978: dread d7/db/f3b [0,4194304] 0 2024-04-22T23:24:01.389 INFO:tasks.workunit.client.1.smithi134.stdout:8/979: dread d7/db/f3b [0,4194304] 0 2024-04-22T23:24:01.391 INFO:tasks.workunit.client.1.smithi134.stdout:6/997: rename d0/d19/de/d1f/dd5 to d0/d19/de/d12/d29/d14c 0 2024-04-22T23:24:01.392 INFO:tasks.workunit.client.1.smithi134.stdout:6/998: truncate d0/d19/de/d12/d29/d74/f8e 4946858 0 2024-04-22T23:24:01.401 INFO:tasks.workunit.client.1.smithi134.stdout:6/999: getdents d0/d19/de/d12/d2a/d2d 0 2024-04-22T23:24:01.402 INFO:tasks.workunit.client.1.smithi134.stdout:8/980: dwrite d7/db/d42/d65/f89 [0,4194304] 0 2024-04-22T23:24:01.415 INFO:tasks.workunit.client.1.smithi134.stdout:8/981: dwrite d7/db/d42/f76 [0,4194304] 0 2024-04-22T23:24:01.418 INFO:tasks.workunit.client.1.smithi134.stdout:8/982: creat d7/dc/d19/dfa/f141 x:0 0 0 2024-04-22T23:24:01.444 INFO:tasks.workunit.client.1.smithi134.stdout:8/983: dread d7/dc/d2c/fb1 [4194304,4194304] 0 2024-04-22T23:24:01.455 INFO:tasks.workunit.client.1.smithi134.stdout:8/984: write d7/db/d42/d65/dd9/dda/de7/fde [581546,97587] 0 2024-04-22T23:24:01.457 INFO:tasks.workunit.client.1.smithi134.stdout:8/985: write d7/dc/d2c/d4f/d6c/d128/f13d [384849,37050] 0 2024-04-22T23:24:01.459 INFO:tasks.workunit.client.1.smithi134.stdout:8/986: mkdir d7/dc/de0/d142 0 2024-04-22T23:24:01.464 INFO:tasks.workunit.client.1.smithi134.stdout:8/987: getdents d7/db/d42/d65/dd9/dda/de7 0 2024-04-22T23:24:01.477 INFO:tasks.workunit.client.1.smithi134.stdout:8/988: dwrite d7/dc/d2c/d4f/d6c/d6e/fc4 [0,4194304] 0 2024-04-22T23:24:01.478 INFO:tasks.workunit.client.1.smithi134.stdout:8/989: chown d7/dc/d19 81764 1 2024-04-22T23:24:01.481 INFO:tasks.workunit.client.1.smithi134.stdout:8/990: symlink d7/db/d42/d65/d46/d9a/l143 0 2024-04-22T23:24:01.487 INFO:tasks.workunit.client.1.smithi134.stdout:8/991: write d7/dc/f2b [4049234,99529] 0 2024-04-22T23:24:01.492 INFO:tasks.workunit.client.1.smithi134.stdout:8/992: read d7/db/d2d/f2e [612647,17624] 0 2024-04-22T23:24:01.499 INFO:tasks.workunit.client.1.smithi134.stdout:8/993: link d7/db/d42/d65/cfc d7/db/d2d/c144 0 2024-04-22T23:24:01.501 INFO:tasks.workunit.client.1.smithi134.stdout:8/994: fdatasync d7/db/d42/d65/dd9/f11c 0 2024-04-22T23:24:01.506 INFO:tasks.workunit.client.1.smithi134.stdout:8/995: dread d7/dc/d2c/d37/d47/f66 [0,4194304] 0 2024-04-22T23:24:01.512 INFO:tasks.workunit.client.1.smithi134.stdout:8/996: dwrite d7/db/d42/d65/d46/fdf [0,4194304] 0 2024-04-22T23:24:01.518 INFO:tasks.workunit.client.1.smithi134.stdout:8/997: write d7/f1c [4304104,99017] 0 2024-04-22T23:24:01.522 INFO:tasks.workunit.client.1.smithi134.stdout:8/998: mknod d7/db/d42/d65/dd9/dda/de4/d111/c145 0 2024-04-22T23:24:01.523 INFO:tasks.workunit.client.1.smithi134.stdout:8/999: write d7/dc/d19/f121 [948109,74221] 0 2024-04-22T23:24:01.543 INFO:tasks.workunit.client.1.smithi134.stdout:3/856: sync 2024-04-22T23:24:01.549 INFO:tasks.workunit.client.1.smithi134.stdout:3/857: link dc/d2a/d62/db2/d2f/d21/d42/d77/fb6 dc/d2a/d62/db2/d2f/d21/d42/d77/d137/dc1/f13c 0 2024-04-22T23:24:01.567 INFO:tasks.workunit.client.1.smithi134.stdout:3/858: dread dc/d2a/d62/fa6 [0,4194304] 0 2024-04-22T23:24:01.569 INFO:tasks.workunit.client.1.smithi134.stdout:3/859: mkdir dc/d2a/d62/db2/d34/d6b/d13d 0 2024-04-22T23:24:01.616 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-22T23:24:01.616 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-22T23:24:01.662 INFO:tasks.workunit.client.1.smithi134.stdout:9/899: sync 2024-04-22T23:24:01.664 INFO:tasks.workunit.client.1.smithi134.stdout:9/900: mkdir d0/d7/d12/d22/d131 0 2024-04-22T23:24:01.664 INFO:tasks.workunit.client.1.smithi134.stdout:9/901: fdatasync d0/f54 0 2024-04-22T23:24:01.667 INFO:tasks.workunit.client.1.smithi134.stdout:9/902: getdents d0/d7/d23/d26/d4d 0 2024-04-22T23:24:01.668 INFO:tasks.workunit.client.1.smithi134.stdout:9/903: truncate d0/d7/d12/de8/f123 420124 0 2024-04-22T23:24:01.670 INFO:tasks.workunit.client.1.smithi134.stdout:9/904: truncate d0/d7/d23/d26/d51/d4c/fe1 3587369 0 2024-04-22T23:24:01.672 INFO:tasks.workunit.client.1.smithi134.stdout:9/905: write d0/d7/d23/d26/d51/d88/db9/f10c [363510,72243] 0 2024-04-22T23:24:01.672 INFO:tasks.workunit.client.1.smithi134.stdout:9/906: stat d0/d2d/d94/d10e 0 2024-04-22T23:24:01.684 INFO:tasks.workunit.client.1.smithi134.stdout:9/907: dread d0/d2d/d82/f83 [0,4194304] 0 2024-04-22T23:24:01.687 INFO:tasks.workunit.client.1.smithi134.stdout:9/908: unlink d0/d7/d12/fcc 0 2024-04-22T23:24:01.688 INFO:tasks.workunit.client.1.smithi134.stdout:9/909: fdatasync d0/d7/d23/d26/d51/f9d 0 2024-04-22T23:24:01.692 INFO:tasks.workunit.client.1.smithi134.stdout:9/910: rename d0/d2d/c93 to d0/d7/d23/d26/c132 0 2024-04-22T23:24:01.696 INFO:tasks.workunit.client.1.smithi134.stdout:9/911: symlink d0/d7/d23/d1b/de2/l133 0 2024-04-22T23:24:01.698 INFO:tasks.workunit.client.1.smithi134.stdout:9/912: mknod d0/d7/d23/d26/d51/d88/c134 0 2024-04-22T23:24:01.703 INFO:tasks.workunit.client.1.smithi134.stdout:9/913: link d0/d37/d76/l115 d0/d2d/dc1/l135 0 2024-04-22T23:24:01.714 INFO:tasks.workunit.client.1.smithi134.stdout:9/914: dwrite d0/d2d/d82/fb7 [0,4194304] 0 2024-04-22T23:24:01.716 INFO:tasks.workunit.client.1.smithi134.stdout:9/915: write d0/d37/f12e [842966,116820] 0 2024-04-22T23:24:01.717 INFO:tasks.workunit.client.1.smithi134.stdout:9/916: chown d0/d7/d23/ca8 1443063555 1 2024-04-22T23:24:01.719 INFO:tasks.workunit.client.1.smithi134.stdout:9/917: mkdir d0/d7/d23/d136 0 2024-04-22T23:24:01.721 INFO:tasks.workunit.client.1.smithi134.stdout:9/918: write d0/d2d/f11f [4291862,27067] 0 2024-04-22T23:24:01.725 INFO:tasks.workunit.client.1.smithi134.stdout:9/919: rename d0/d7/c67 to d0/d7/d23/d26/d51/d88/db9/c137 0 2024-04-22T23:24:01.727 INFO:tasks.workunit.client.1.smithi134.stdout:9/920: symlink d0/d2d/d82/da0/l138 0 2024-04-22T23:24:01.732 INFO:tasks.workunit.client.1.smithi134.stdout:9/921: rename d0/d2d/d3d to d0/d7/d12/d139 0 2024-04-22T23:24:01.736 INFO:tasks.workunit.client.1.smithi134.stdout:9/922: write d0/d7/d23/d26/fb0 [2141169,16770] 0 2024-04-22T23:24:01.740 INFO:tasks.workunit.client.1.smithi134.stdout:9/923: unlink d0/d7/d23/d26/d51/d88/db9/f10c 0 2024-04-22T23:24:01.743 INFO:tasks.workunit.client.1.smithi134.stdout:9/924: link d0/d7/f6f d0/d7/d12/f13a 0 2024-04-22T23:24:01.748 INFO:tasks.workunit.client.1.smithi134.stdout:9/925: write d0/d7/d12/f13a [2508400,39617] 0 2024-04-22T23:24:01.749 INFO:tasks.workunit.client.1.smithi134.stdout:9/926: chown d0/d2d/d82/da0/l138 36389 1 2024-04-22T23:24:01.750 INFO:tasks.workunit.client.1.smithi134.stdout:9/927: stat d0/d7/d23/d26/d51/d88/c134 0 2024-04-22T23:24:01.752 INFO:tasks.workunit.client.1.smithi134.stdout:9/928: creat d0/d7/d23/d136/f13b x:0 0 0 2024-04-22T23:24:01.754 INFO:tasks.workunit.client.1.smithi134.stdout:9/929: truncate d0/d7/d23/d26/d4d/fb6 4763073 0 2024-04-22T23:24:01.757 INFO:tasks.workunit.client.1.smithi134.stdout:9/930: rename d0/d7/d12/de8 to d0/d7/d13c 0 2024-04-22T23:24:01.761 INFO:tasks.workunit.client.1.smithi134.stdout:9/931: dread d0/d7/d12/d139/f102 [0,4194304] 0 2024-04-22T23:24:01.761 INFO:tasks.workunit.client.1.smithi134.stdout:9/932: chown d0/d7/d12 4055 1 2024-04-22T23:24:01.763 INFO:tasks.workunit.client.1.smithi134.stdout:9/933: creat d0/d7/d23/d26/f13d x:0 0 0 2024-04-22T23:24:01.766 INFO:tasks.workunit.client.1.smithi134.stdout:9/934: unlink d0/d2d/d82/df5/fe7 0 2024-04-22T23:24:01.772 INFO:tasks.workunit.client.1.smithi134.stdout:9/935: dwrite d0/d7/d23/d1b/f117 [0,4194304] 0 2024-04-22T23:24:01.777 INFO:tasks.workunit.client.1.smithi134.stdout:9/936: fsync d0/d7/d12/f42 0 2024-04-22T23:24:01.780 INFO:tasks.workunit.client.1.smithi134.stdout:9/937: mknod d0/d37/c13e 0 2024-04-22T23:24:01.780 INFO:tasks.workunit.client.1.smithi134.stdout:9/938: stat d0/d7/d23/d26/d51/feb 0 2024-04-22T23:24:01.790 INFO:tasks.workunit.client.1.smithi134.stdout:9/939: write d0/d2d/d82/da0/fd5 [538881,120329] 0 2024-04-22T23:24:01.799 INFO:tasks.workunit.client.1.smithi134.stdout:9/940: dwrite d0/d37/f10a [0,4194304] 0 2024-04-22T23:24:01.804 INFO:tasks.workunit.client.1.smithi134.stdout:2/946: sync 2024-04-22T23:24:01.810 INFO:tasks.workunit.client.1.smithi134.stdout:2/947: read d2/f30 [2911092,70051] 0 2024-04-22T23:24:01.811 INFO:tasks.workunit.client.1.smithi134.stdout:9/941: dwrite d0/d7/d23/d26/d51/d4c/fe1 [0,4194304] 0 2024-04-22T23:24:01.816 INFO:tasks.workunit.client.1.smithi134.stdout:9/942: creat d0/d2d/d82/df5/d10d/f13f x:0 0 0 2024-04-22T23:24:01.818 INFO:tasks.workunit.client.1.smithi134.stdout:9/943: chown d0/d7/d12/d139/l63 61429719 1 2024-04-22T23:24:01.822 INFO:tasks.workunit.client.1.smithi134.stdout:2/948: dwrite d2/d15/d24/d1d/dba/dea/f107 [0,4194304] 0 2024-04-22T23:24:01.822 INFO:tasks.workunit.client.1.smithi134.stdout:9/944: write d0/d2d/d94/f121 [331811,57284] 0 2024-04-22T23:24:01.826 INFO:tasks.workunit.client.1.smithi134.stdout:2/949: creat d2/dd/dc9/f144 x:0 0 0 2024-04-22T23:24:01.832 INFO:tasks.workunit.client.1.smithi134.stdout:9/945: dwrite d0/d7/d13c/df7/f108 [0,4194304] 0 2024-04-22T23:24:01.838 INFO:tasks.workunit.client.1.smithi134.stdout:9/946: creat d0/d7/d13c/df7/f140 x:0 0 0 2024-04-22T23:24:01.843 INFO:tasks.workunit.client.1.smithi134.stdout:2/950: truncate d2/f3e 3441348 0 2024-04-22T23:24:01.844 INFO:tasks.workunit.client.1.smithi134.stdout:9/947: symlink d0/d7/d23/d26/l141 0 2024-04-22T23:24:01.844 INFO:tasks.workunit.client.1.smithi134.stdout:2/951: rmdir d2/d15/d24/d3b/d8a 39 2024-04-22T23:24:01.846 INFO:tasks.workunit.client.1.smithi134.stdout:9/948: truncate d0/f103 623041 0 2024-04-22T23:24:01.847 INFO:tasks.workunit.client.1.smithi134.stdout:9/949: write d0/d7/f10b [938245,54367] 0 2024-04-22T23:24:01.848 INFO:tasks.workunit.client.1.smithi134.stdout:2/952: creat d2/dd/df/f145 x:0 0 0 2024-04-22T23:24:01.851 INFO:tasks.workunit.client.1.smithi134.stdout:2/953: mknod d2/c146 0 2024-04-22T23:24:01.853 INFO:tasks.workunit.client.1.smithi134.stdout:9/950: creat d0/d7/d23/d26/d4d/d5d/d8e/f142 x:0 0 0 2024-04-22T23:24:01.855 INFO:tasks.workunit.client.1.smithi134.stdout:9/951: write d0/d7/d23/d1b/f117 [3023338,31781] 0 2024-04-22T23:24:01.855 INFO:tasks.workunit.client.1.smithi134.stdout:9/952: stat d0/d7 0 2024-04-22T23:24:01.856 INFO:tasks.workunit.client.1.smithi134.stdout:2/954: unlink d2/d15/d24/d1d/d3d/d7b/c84 0 2024-04-22T23:24:01.857 INFO:tasks.workunit.client.1.smithi134.stdout:9/953: write d0/d7/d27/fa5 [536880,11280] 0 2024-04-22T23:24:01.859 INFO:tasks.workunit.client.1.smithi134.stdout:2/955: creat d2/d42/d54/dfd/f147 x:0 0 0 2024-04-22T23:24:01.863 INFO:tasks.workunit.client.1.smithi134.stdout:9/954: write d0/d7/ffc [4725327,50802] 0 2024-04-22T23:24:01.864 INFO:tasks.workunit.client.1.smithi134.stdout:3/860: sync 2024-04-22T23:24:01.864 INFO:tasks.workunit.client.1.smithi134.stdout:2/956: stat d2/dd/df/c10 0 2024-04-22T23:24:01.873 INFO:tasks.workunit.client.1.smithi134.stdout:3/861: mkdir dc/d2a/d62/db2/d2f/d13e 0 2024-04-22T23:24:01.874 INFO:tasks.workunit.client.1.smithi134.stdout:9/955: dread d0/d37/d76/f91 [4194304,4194304] 0 2024-04-22T23:24:01.875 INFO:tasks.workunit.client.1.smithi134.stdout:9/956: readlink d0/d7/d23/d1b/de2/l133 0 2024-04-22T23:24:01.886 INFO:tasks.workunit.client.1.smithi134.stdout:9/957: dwrite d0/d7/d12/f1f [0,4194304] 0 2024-04-22T23:24:01.889 INFO:tasks.workunit.client.1.smithi134.stdout:2/957: dwrite d2/d42/d54/f59 [0,4194304] 0 2024-04-22T23:24:01.891 INFO:tasks.workunit.client.1.smithi134.stdout:9/958: symlink d0/d7/d23/d1b/d5a/l143 0 2024-04-22T23:24:01.891 INFO:tasks.workunit.client.1.smithi134.stdout:2/958: fsync d2/d15/d24/f6d 0 2024-04-22T23:24:01.894 INFO:tasks.workunit.client.1.smithi134.stdout:3/862: dwrite dc/d2a/ff3 [0,4194304] 0 2024-04-22T23:24:01.896 INFO:tasks.workunit.client.1.smithi134.stdout:2/959: truncate d2/d42/d54/f141 5692109 0 2024-04-22T23:24:01.898 INFO:tasks.workunit.client.1.smithi134.stdout:9/959: rename d0/d7/d12/d139/f102 to d0/d2d/d4e/f144 0 2024-04-22T23:24:01.901 INFO:tasks.workunit.client.1.smithi134.stdout:3/863: dread dc/d2a/d62/db2/d2f/d21/d42/d77/d137/dc1/dd8/fe4 [0,4194304] 0 2024-04-22T23:24:01.904 INFO:tasks.workunit.client.1.smithi134.stdout:9/960: write d0/d7/d12/f13a [1789042,111615] 0 2024-04-22T23:24:01.908 INFO:tasks.workunit.client.1.smithi134.stdout:9/961: symlink d0/d7/d23/d26/d51/l145 0 2024-04-22T23:24:01.909 INFO:tasks.workunit.client.1.smithi134.stdout:9/962: fsync d0/d7/d23/d1b/de2/fec 0 2024-04-22T23:24:01.927 INFO:tasks.workunit.client.1.smithi134.stdout:3/864: write dc/d2a/d62/db2/d2f/d21/d42/d77/d137/dc1/f13c [719975,4268] 0 2024-04-22T23:24:01.930 INFO:tasks.workunit.client.1.smithi134.stdout:9/963: write d0/d7/d12/f42 [2162731,62739] 0 2024-04-22T23:24:01.934 INFO:tasks.workunit.client.1.smithi134.stdout:2/960: dwrite d2/dd/df/fa0 [0,4194304] 0 2024-04-22T23:24:01.939 INFO:tasks.workunit.client.1.smithi134.stdout:2/961: dwrite d2/dd/d68/f88 [4194304,4194304] 0 2024-04-22T23:24:01.949 INFO:tasks.workunit.client.1.smithi134.stdout:2/962: dwrite d2/dd/df/d9c/dca/f13a [0,4194304] 0 2024-04-22T23:24:01.967 INFO:tasks.workunit.client.1.smithi134.stdout:3/865: mknod dc/d2a/d62/db2/d2f/d21/d42/d77/d137/dc1/df7/c13f 0 2024-04-22T23:24:01.968 INFO:tasks.workunit.client.1.smithi134.stdout:9/964: mknod d0/d2d/d82/da0/c146 0 2024-04-22T23:24:01.973 INFO:tasks.workunit.client.1.smithi134.stdout:3/866: rmdir dc/d2a/d62 39 2024-04-22T23:24:01.974 INFO:tasks.workunit.client.1.smithi134.stdout:3/867: write dc/f52 [297251,85472] 0 2024-04-22T23:24:01.975 INFO:tasks.workunit.client.1.smithi134.stdout:9/965: mkdir d0/d7/d23/d26/d4d/d5d/d8e/dae/d147 0 2024-04-22T23:24:01.975 INFO:tasks.workunit.client.1.smithi134.stdout:9/966: fsync d0/d37/f124 0 2024-04-22T23:24:01.978 INFO:tasks.workunit.client.1.smithi134.stdout:2/963: rename d2/dd/df/fb9 to d2/d42/d54/d33/f148 0 2024-04-22T23:24:01.979 INFO:tasks.workunit.client.1.smithi134.stdout:2/964: fdatasync d2/d15/d24/d1d/d3d/d9a/f13f 0 2024-04-22T23:24:01.981 INFO:tasks.workunit.client.1.smithi134.stdout:3/868: fdatasync dc/d2a/d62/db2/d2f/d21/d42/d77/fd4 0 2024-04-22T23:24:01.984 INFO:tasks.workunit.client.1.smithi134.stdout:3/869: dread - dc/d2a/d62/db2/d2f/d61/d6a/f117 zero size 2024-04-22T23:24:01.986 INFO:tasks.workunit.client.1.smithi134.stdout:2/965: dwrite d2/d94/fd4 [0,4194304] 0 2024-04-22T23:24:01.987 INFO:tasks.workunit.client.1.smithi134.stdout:9/967: symlink d0/d2d/db2/l148 0 2024-04-22T23:24:01.988 INFO:tasks.workunit.client.1.smithi134.stdout:3/870: creat dc/d2a/d30/f140 x:0 0 0 2024-04-22T23:24:01.989 INFO:tasks.workunit.client.1.smithi134.stdout:9/968: write d0/d2d/d94/fe6 [3413482,113068] 0 2024-04-22T23:24:01.989 INFO:tasks.workunit.client.1.smithi134.stdout:3/871: chown dc/d2a/d62/db2/d34/d6b/dae/dc3/dde 77 1 2024-04-22T23:24:01.990 INFO:tasks.workunit.client.1.smithi134.stdout:2/966: write d2/dd/d4a/d73/ff1 [470388,113628] 0 2024-04-22T23:24:01.991 INFO:tasks.workunit.client.1.smithi134.stdout:3/872: chown dc/d2a/d62/f7f 501577564 1 2024-04-22T23:24:01.995 INFO:tasks.workunit.client.1.smithi134.stdout:9/969: dread d0/d7/f6f [0,4194304] 0 2024-04-22T23:24:01.997 INFO:tasks.workunit.client.1.smithi134.stdout:2/967: rename d2/dd/df9 to d2/d15/d24/d1d/d3d/d149 0 2024-04-22T23:24:02.000 INFO:tasks.workunit.client.1.smithi134.stdout:9/970: fdatasync d0/d7/d23/d1b/d5a/fb3 0 2024-04-22T23:24:02.002 INFO:tasks.workunit.client.1.smithi134.stdout:2/968: mkdir d2/dd/dc9/d14a 0 2024-04-22T23:24:02.005 INFO:tasks.workunit.client.1.smithi134.stdout:2/969: readlink d2/d15/d24/d1d/l133 0 2024-04-22T23:24:02.008 INFO:tasks.workunit.client.1.smithi134.stdout:9/971: dwrite d0/d7/d27/f33 [4194304,4194304] 0 2024-04-22T23:24:02.010 INFO:tasks.workunit.client.1.smithi134.stdout:2/970: creat d2/d15/d24/d3b/d8a/d101/f14b x:0 0 0 2024-04-22T23:24:02.011 INFO:tasks.workunit.client.1.smithi134.stdout:2/971: dread - d2/d42/fe8 zero size 2024-04-22T23:24:02.013 INFO:tasks.workunit.client.1.smithi134.stdout:9/972: creat d0/d7/d23/d26/d4d/d5d/d8e/dae/db1/f149 x:0 0 0 2024-04-22T23:24:02.015 INFO:tasks.workunit.client.1.smithi134.stdout:2/972: symlink d2/d42/d54/d33/d91/d12f/l14c 0 2024-04-22T23:24:02.018 INFO:tasks.workunit.client.1.smithi134.stdout:9/973: rename d0/d2d/d82/da0/c146 to d0/d7/d23/c14a 0 2024-04-22T23:24:02.023 INFO:tasks.workunit.client.1.smithi134.stdout:2/973: link d2/dd/d68/f88 d2/d42/d54/d33/d6e/f14d 0 2024-04-22T23:24:02.025 INFO:tasks.workunit.client.1.smithi134.stdout:2/974: chown d2/dd/d68/l8f 339348 1 2024-04-22T23:24:02.029 INFO:tasks.workunit.client.1.smithi134.stdout:2/975: chown d2/d15/d24/d1d/d3d/dfe 23690788 1 2024-04-22T23:24:02.029 INFO:tasks.workunit.client.1.smithi134.stdout:3/873: write dc/d2a/d62/db2/d2f/d36/d6d/f9c [490968,63886] 0 2024-04-22T23:24:02.032 INFO:tasks.workunit.client.1.smithi134.stdout:3/874: dread dc/f40 [0,4194304] 0 2024-04-22T23:24:02.033 INFO:tasks.workunit.client.1.smithi134.stdout:2/976: dread d2/dd/df/d1c/ff0 [0,4194304] 0 2024-04-22T23:24:02.034 INFO:tasks.workunit.client.1.smithi134.stdout:3/875: chown dc/d2a/d62/db2/d2f/d36/d6d/d6f/dd9 6372 1 2024-04-22T23:24:02.034 INFO:tasks.workunit.client.1.smithi134.stdout:9/974: write d0/d7/d12/d139/fda [725656,33400] 0 2024-04-22T23:24:02.036 INFO:tasks.workunit.client.1.smithi134.stdout:2/977: mkdir d2/d15/d24/d1d/dba/dcb/d14e 0 2024-04-22T23:24:02.038 INFO:tasks.workunit.client.1.smithi134.stdout:2/978: write d2/dd/d68/f10f [315373,92870] 0 2024-04-22T23:24:02.039 INFO:tasks.workunit.client.1.smithi134.stdout:3/876: creat dc/d2a/d62/db2/d2f/d21/dec/f141 x:0 0 0 2024-04-22T23:24:02.040 INFO:tasks.workunit.client.1.smithi134.stdout:9/975: mknod d0/d2d/c14b 0 2024-04-22T23:24:02.045 INFO:tasks.workunit.client.1.smithi134.stdout:3/877: link dc/d2a/d62/db2/d2f/d21/d42/d77/d8f/lbd dc/d2a/d62/db2/d34/d72/l142 0 2024-04-22T23:24:02.050 INFO:tasks.workunit.client.1.smithi134.stdout:3/878: fdatasync dc/d2a/d62/db2/d34/d6b/fbe 0 2024-04-22T23:24:02.050 INFO:tasks.workunit.client.1.smithi134.stdout:2/979: write d2/d15/d24/f69 [873978,38453] 0 2024-04-22T23:24:02.051 INFO:tasks.workunit.client.1.smithi134.stdout:2/980: readlink d2/d42/d54/l8b 0 2024-04-22T23:24:02.055 INFO:tasks.workunit.client.1.smithi134.stdout:9/976: dread d0/d7/d12/ffe [0,4194304] 0 2024-04-22T23:24:02.057 INFO:tasks.workunit.client.1.smithi134.stdout:2/981: unlink d2/d15/d24/d1d/l2f 0 2024-04-22T23:24:02.058 INFO:tasks.workunit.client.1.smithi134.stdout:9/977: mknod d0/d2d/db2/c14c 0 2024-04-22T23:24:02.059 INFO:tasks.workunit.client.1.smithi134.stdout:3/879: truncate dc/d2a/d62/db2/d2f/d36/d5f/fd3 955615 0 2024-04-22T23:24:02.061 INFO:tasks.workunit.client.1.smithi134.stdout:3/880: dread - dc/d2a/d62/db2/d2f/d21/d42/d77/d137/dc1/f109 zero size 2024-04-22T23:24:02.062 INFO:tasks.workunit.client.1.smithi134.stdout:3/881: chown dc/d2a/d62/db2/d2f/d21/dec/dcf/fd5 12915463 1 2024-04-22T23:24:02.063 INFO:tasks.workunit.client.1.smithi134.stdout:9/978: unlink d0/d7/d23/d26/d4d/d5d/d8e/cc0 0 2024-04-22T23:24:02.064 INFO:tasks.workunit.client.1.smithi134.stdout:3/882: dread - dc/d2a/d62/db2/d2f/d21/d42/d77/d137/dc1/dd8/f133 zero size 2024-04-22T23:24:02.066 INFO:tasks.workunit.client.1.smithi134.stdout:2/982: write d2/dd/df/dc2/f13d [553099,96681] 0 2024-04-22T23:24:02.067 INFO:tasks.workunit.client.1.smithi134.stdout:3/883: creat dc/d2a/d62/db2/d34/d70/f143 x:0 0 0 2024-04-22T23:24:02.070 INFO:tasks.workunit.client.1.smithi134.stdout:2/983: creat d2/d42/d54/f14f x:0 0 0 2024-04-22T23:24:02.071 INFO:tasks.workunit.client.1.smithi134.stdout:9/979: dwrite d0/d2d/d82/df5/d10d/f112 [0,4194304] 0 2024-04-22T23:24:02.073 INFO:tasks.workunit.client.1.smithi134.stdout:3/884: write dc/d2a/d62/db2/d2f/d21/d42/d77/d137/dc1/df7/f112 [464482,19145] 0 2024-04-22T23:24:02.075 INFO:tasks.workunit.client.1.smithi134.stdout:2/984: mkdir d2/d15/d24/d3b/d8a/d99/dfb/d150 0 2024-04-22T23:24:02.077 INFO:tasks.workunit.client.1.smithi134.stdout:2/985: chown d2/d42/d54/dfd 5805 1 2024-04-22T23:24:02.079 INFO:tasks.workunit.client.1.smithi134.stdout:3/885: rename dc/d2a/d62/db2/d2f/l10 to dc/d2a/d62/db2/d2f/d21/d42/l144 0 2024-04-22T23:24:02.081 INFO:tasks.workunit.client.1.smithi134.stdout:3/886: chown dc/d2a/d62/db2/d2f/d21/dec/lda 3226 1 2024-04-22T23:24:02.082 INFO:tasks.workunit.client.1.smithi134.stdout:9/980: getdents d0/d7/d27 0 2024-04-22T23:24:02.085 INFO:tasks.workunit.client.1.smithi134.stdout:3/887: truncate dc/d2a/d62/db2/d34/d6b/fbe 1504095 0 2024-04-22T23:24:02.093 INFO:tasks.workunit.client.1.smithi134.stdout:2/986: dwrite d2/d15/d24/d1d/f36 [0,4194304] 0 2024-04-22T23:24:02.094 INFO:tasks.workunit.client.1.smithi134.stdout:2/987: readlink d2/d42/le6 0 2024-04-22T23:24:02.100 INFO:tasks.workunit.client.1.smithi134.stdout:2/988: rename d2/leb to d2/dd/d77/l151 0 2024-04-22T23:24:02.101 INFO:tasks.workunit.client.1.smithi134.stdout:2/989: fsync d2/dd/d116/f11f 0 2024-04-22T23:24:02.103 INFO:tasks.workunit.client.1.smithi134.stdout:2/990: mknod d2/d42/d54/d142/c152 0 2024-04-22T23:24:02.105 INFO:tasks.workunit.client.1.smithi134.stdout:2/991: symlink d2/d15/d24/d1d/d3d/d7b/l153 0 2024-04-22T23:24:02.107 INFO:tasks.workunit.client.1.smithi134.stdout:2/992: mknod d2/d42/d54/d33/d91/d11d/c154 0 2024-04-22T23:24:02.115 INFO:tasks.workunit.client.1.smithi134.stdout:2/993: write d2/d42/fe8 [396428,119081] 0 2024-04-22T23:24:02.119 INFO:tasks.workunit.client.1.smithi134.stdout:2/994: truncate d2/d15/d24/f6d 3865476 0 2024-04-22T23:24:02.122 INFO:tasks.workunit.client.1.smithi134.stdout:2/995: write d2/dd/d68/f88 [1415337,66212] 0 2024-04-22T23:24:02.125 INFO:tasks.workunit.client.1.smithi134.stdout:2/996: mknod d2/d42/d54/d142/c155 0 2024-04-22T23:24:02.126 INFO:tasks.workunit.client.1.smithi134.stdout:2/997: write d2/d42/d54/d109/f112 [1035561,57116] 0 2024-04-22T23:24:02.129 INFO:tasks.workunit.client.1.smithi134.stdout:2/998: creat d2/dd/d68/f156 x:0 0 0 2024-04-22T23:24:02.129 INFO:tasks.workunit.client.1.smithi134.stdout:2/999: chown d2/d15/f118 524393574 1 2024-04-22T23:24:02.397 INFO:tasks.workunit.client.1.smithi134.stdout:9/981: sync 2024-04-22T23:24:02.397 INFO:tasks.workunit.client.1.smithi134.stdout:3/888: sync 2024-04-22T23:24:02.398 INFO:tasks.workunit.client.1.smithi134.stdout:9/982: mknod d0/d7/d23/d26/d4d/d5d/d8e/d105/c14d 0 2024-04-22T23:24:02.399 INFO:tasks.workunit.client.1.smithi134.stdout:9/983: chown d0/d7/d23/d26/l141 63 1 2024-04-22T23:24:02.405 INFO:tasks.workunit.client.1.smithi134.stdout:9/984: truncate d0/d2d/d82/f83 1716698 0 2024-04-22T23:24:02.415 INFO:tasks.workunit.client.1.smithi134.stdout:9/985: dread d0/d2d/f9f [0,4194304] 0 2024-04-22T23:24:02.417 INFO:tasks.workunit.client.1.smithi134.stdout:9/986: mkdir d0/d7/d23/d1b/dc5/d14e 0 2024-04-22T23:24:02.424 INFO:tasks.workunit.client.1.smithi134.stdout:9/987: write d0/d7/d23/d1b/f64 [4726988,119964] 0 2024-04-22T23:24:02.425 INFO:tasks.workunit.client.1.smithi134.stdout:9/988: write d0/d7/d23/d1b/de2/fec [2537649,105858] 0 2024-04-22T23:24:02.426 INFO:tasks.workunit.client.1.smithi134.stdout:9/989: stat d0/d2d/db2 0 2024-04-22T23:24:02.430 INFO:tasks.workunit.client.1.smithi134.stdout:9/990: getdents d0/d7 0 2024-04-22T23:24:02.432 INFO:tasks.workunit.client.1.smithi134.stdout:9/991: chown d0/d7/d23/d26/d4d/fb6 68324 1 2024-04-22T23:24:02.435 INFO:tasks.workunit.client.1.smithi134.stdout:9/992: unlink d0/d7/d23/d26/d51/d88/db9/fcb 0 2024-04-22T23:24:02.437 INFO:tasks.workunit.client.1.smithi134.stdout:9/993: mknod d0/d7/d12/c14f 0 2024-04-22T23:24:02.439 INFO:tasks.workunit.client.1.smithi134.stdout:9/994: getdents d0/d37/d76 0 2024-04-22T23:24:02.443 INFO:tasks.workunit.client.1.smithi134.stdout:9/995: rename d0/d7/d12/d139/fda to d0/d7/d23/d26/d4d/d5d/d8e/d105/f150 0 2024-04-22T23:24:02.444 INFO:tasks.workunit.client.1.smithi134.stdout:9/996: read - d0/d7/d23/d26/d51/feb zero size 2024-04-22T23:24:02.444 INFO:tasks.workunit.client.1.smithi134.stdout:9/997: stat d0/d2d/d82/df5/d10d/f112 0 2024-04-22T23:24:02.445 INFO:tasks.workunit.client.1.smithi134.stdout:9/998: dread - d0/d2d/d82/f130 zero size 2024-04-22T23:24:02.446 INFO:tasks.workunit.client.1.smithi134.stdout:9/999: chown d0/d7/d23/d26/d51/d4c/fe1 1959 1 2024-04-22T23:24:02.496 INFO:tasks.workunit.client.1.smithi134.stdout:3/889: sync 2024-04-22T23:24:02.499 INFO:tasks.workunit.client.1.smithi134.stdout:3/890: symlink dc/d2a/d62/db2/d2f/d54/dac/d115/l145 0 2024-04-22T23:24:02.501 INFO:tasks.workunit.client.1.smithi134.stdout:3/891: mkdir dc/d7d/d146 0 2024-04-22T23:24:02.501 INFO:tasks.workunit.client.1.smithi134.stdout:3/892: chown dc/d2a/d62/db2/d2f/d36 620586 1 2024-04-22T23:24:02.503 INFO:tasks.workunit.client.1.smithi134.stdout:3/893: rmdir dc/d2a/d62/db2/d34/d72 39 2024-04-22T23:24:02.514 INFO:tasks.workunit.client.1.smithi134.stdout:3/894: dwrite dc/d2a/d62/db2/d3b/d29/fb8 [0,4194304] 0 2024-04-22T23:24:02.519 INFO:tasks.workunit.client.1.smithi134.stdout:3/895: link dc/d2a/d62/db2/d2f/d21/d42/d77/d137/fba dc/d2a/d30/f147 0 2024-04-22T23:24:02.539 INFO:tasks.workunit.client.1.smithi134.stdout:3/896: dread dc/d2a/d62/db2/d2f/d54/fd0 [0,4194304] 0 2024-04-22T23:24:02.541 INFO:tasks.workunit.client.1.smithi134.stdout:3/897: truncate dc/f40 716388 0 2024-04-22T23:24:02.541 INFO:tasks.workunit.client.1.smithi134.stdout:3/898: chown dc/d2a/d30/ca3 0 1 2024-04-22T23:24:02.542 INFO:tasks.workunit.client.1.smithi134.stdout:3/899: stat dc/d2a/d62/db2/d2f/d61/d6a/l100 0 2024-04-22T23:24:02.549 INFO:tasks.workunit.client.1.smithi134.stdout:3/900: dwrite dc/d2a/d62/db2/d2f/d21/d42/d77/d137/dc1/f13c [0,4194304] 0 2024-04-22T23:24:02.553 INFO:tasks.workunit.client.1.smithi134.stdout:3/901: creat dc/d2a/d62/db2/d2f/d36/d6d/d6f/f148 x:0 0 0 2024-04-22T23:24:02.557 INFO:tasks.workunit.client.1.smithi134.stdout:3/902: getdents dc/d2a/d62/db2/d34/d6b/dae/dc3/df2 0 2024-04-22T23:24:02.557 INFO:tasks.workunit.client.1.smithi134.stdout:3/903: chown dc/d2a/d62/db2/d2f/d21/d42 42287272 1 2024-04-22T23:24:02.559 INFO:tasks.workunit.client.1.smithi134.stdout:3/904: creat dc/f149 x:0 0 0 2024-04-22T23:24:02.593 INFO:tasks.workunit.client.1.smithi134.stdout:3/905: dread dc/d2a/d62/db2/d34/d6b/f7c [0,4194304] 0 2024-04-22T23:24:02.595 INFO:tasks.workunit.client.1.smithi134.stdout:3/906: truncate dc/d2a/d62/db2/d2f/d61/d6a/f117 762364 0 2024-04-22T23:24:02.600 INFO:tasks.workunit.client.1.smithi134.stdout:3/907: write dc/d2a/f23 [4223408,10970] 0 2024-04-22T23:24:02.603 INFO:tasks.workunit.client.1.smithi134.stdout:3/908: truncate dc/d2a/d62/db2/d2f/d21/d42/d77/fd4 1556217 0 2024-04-22T23:24:02.608 INFO:tasks.workunit.client.1.smithi134.stdout:3/909: symlink dc/d2a/d62/db2/d34/d6b/dae/dc3/l14a 0 2024-04-22T23:24:02.609 INFO:tasks.workunit.client.1.smithi134.stdout:3/910: readlink dc/d2a/d62/db2/d2f/d36/d6d/l81 0 2024-04-22T23:24:02.613 INFO:tasks.workunit.client.1.smithi134.stdout:3/911: creat dc/d2a/d62/db2/d3b/dfd/f14b x:0 0 0 2024-04-22T23:24:02.616 INFO:tasks.workunit.client.1.smithi134.stdout:3/912: dread dc/d2a/d62/db2/d2f/d36/d6d/f9c [0,4194304] 0 2024-04-22T23:24:02.618 INFO:tasks.workunit.client.1.smithi134.stdout:3/913: symlink dc/d2a/d62/db2/d2f/d54/dac/l14c 0 2024-04-22T23:24:02.619 INFO:tasks.workunit.client.1.smithi134.stdout:3/914: chown dc/d2a/d30 677682 1 2024-04-22T23:24:02.619 INFO:tasks.workunit.client.1.smithi134.stdout:3/915: chown dc/d2a/d62/db2/d2f/d36/d6d/d6f/dd9/fe0 213 1 2024-04-22T23:24:02.623 INFO:tasks.workunit.client.1.smithi134.stdout:3/916: unlink dc/d2a/d30/l56 0 2024-04-22T23:24:02.624 INFO:tasks.workunit.client.1.smithi134.stdout:3/917: write dc/d2a/d62/fa5 [2543302,114968] 0 2024-04-22T23:24:02.626 INFO:tasks.workunit.client.1.smithi134.stdout:3/918: mknod dc/d2a/d62/db2/d2f/d21/d42/c14d 0 2024-04-22T23:24:02.627 INFO:tasks.workunit.client.1.smithi134.stdout:3/919: chown dc/d2a/d62/db2/d2f/d54/fd0 78 1 2024-04-22T23:24:02.632 INFO:tasks.workunit.client.1.smithi134.stdout:3/920: dwrite dc/f149 [0,4194304] 0 2024-04-22T23:24:02.637 INFO:tasks.workunit.client.1.smithi134.stdout:3/921: creat dc/d2a/d62/db2/d2f/d21/dec/dcf/dfb/f14e x:0 0 0 2024-04-22T23:24:02.643 INFO:tasks.workunit.client.1.smithi134.stdout:3/922: rename dc/d2a/d62/db2/d3b/d29 to dc/d2a/d62/db2/d34/d6b/dfe/d14f 0 2024-04-22T23:24:02.644 INFO:tasks.workunit.client.1.smithi134.stdout:3/923: stat dc/d2a/d62/db2/d2f/d21/d42/d77/d137/dc1/dd8/f133 0 2024-04-22T23:24:02.650 INFO:tasks.workunit.client.1.smithi134.stdout:3/924: rename dc/d2a/d62/db2/d2f/d21/dec to dc/d2a/d62/db2/d34/d6b/dfe/d14f/d150 0 2024-04-22T23:24:02.652 INFO:tasks.workunit.client.1.smithi134.stdout:3/925: mknod dc/d2a/d62/db2/d34/c151 0 2024-04-22T23:24:02.653 INFO:tasks.workunit.client.1.smithi134.stdout:3/926: chown dc/d2a/d62/db2/d2f/d36/d5f/l8b 223 1 2024-04-22T23:24:02.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:02 smithi134 ceph-mon[34609]: pgmap v5: 65 pgs: 65 active+clean; 2.4 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail 2024-04-22T23:24:02.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:02 smithi134 ceph-mon[34609]: mgrmap e24: smithi134.ccxkcj(active, since 4s) 2024-04-22T23:24:02.781 INFO:tasks.workunit.client.1.smithi134.stdout:3/927: sync 2024-04-22T23:24:02.783 INFO:tasks.workunit.client.1.smithi134.stdout:3/928: symlink dc/d2a/d62/db2/d2f/d21/d42/d77/d8f/l152 0 2024-04-22T23:24:02.789 INFO:tasks.workunit.client.1.smithi134.stdout:3/929: dwrite dc/d2a/d62/db2/d34/d6b/dae/f105 [0,4194304] 0 2024-04-22T23:24:02.794 INFO:tasks.workunit.client.1.smithi134.stdout:3/930: write dc/d2a/d62/fa6 [2572666,38008] 0 2024-04-22T23:24:02.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:02 smithi067 ceph-mon[26453]: pgmap v5: 65 pgs: 65 active+clean; 2.4 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail 2024-04-22T23:24:02.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:02 smithi067 ceph-mon[26453]: mgrmap e24: smithi134.ccxkcj(active, since 4s) 2024-04-22T23:24:02.804 INFO:tasks.workunit.client.1.smithi134.stdout:3/931: truncate dc/d2a/d62/db2/d2f/d21/d42/d77/f106 2816199 0 2024-04-22T23:24:02.807 INFO:tasks.workunit.client.1.smithi134.stdout:3/932: fdatasync dc/d2a/d62/db2/d34/fdd 0 2024-04-22T23:24:02.812 INFO:tasks.workunit.client.1.smithi134.stdout:3/933: dread dc/d2a/d62/db2/d2f/d21/d42/d77/d137/dc1/f13c [0,4194304] 0 2024-04-22T23:24:02.813 INFO:tasks.workunit.client.1.smithi134.stdout:3/934: chown dc/d2a/d62/db2/d2f/d21/d42/d77/d137/dc1/dd8/f133 11357 1 2024-04-22T23:24:02.817 INFO:tasks.workunit.client.1.smithi134.stdout:3/935: creat dc/d2a/d62/db2/d34/f153 x:0 0 0 2024-04-22T23:24:02.819 INFO:tasks.workunit.client.1.smithi134.stdout:3/936: creat dc/d2a/d62/db2/d34/d6b/dae/f154 x:0 0 0 2024-04-22T23:24:02.821 INFO:tasks.workunit.client.1.smithi134.stdout:3/937: rmdir dc/d7d 39 2024-04-22T23:24:02.824 INFO:tasks.workunit.client.1.smithi134.stdout:3/938: rename dc/d2a/d62/db2/d2f/c12 to dc/d2a/d62/db2/d2f/d21/d42/d77/d137/c155 0 2024-04-22T23:24:02.825 INFO:tasks.workunit.client.1.smithi134.stdout:3/939: write dc/d2a/fe5 [3159184,39783] 0 2024-04-22T23:24:02.826 INFO:tasks.workunit.client.1.smithi134.stdout:3/940: stat dc/d2a/d62/db2/d34/f153 0 2024-04-22T23:24:02.828 INFO:tasks.workunit.client.1.smithi134.stdout:3/941: creat dc/d2a/d62/f156 x:0 0 0 2024-04-22T23:24:02.831 INFO:tasks.workunit.client.1.smithi134.stdout:3/942: creat dc/d2a/d62/d103/d11e/f157 x:0 0 0 2024-04-22T23:24:02.833 INFO:tasks.workunit.client.1.smithi134.stdout:3/943: mknod dc/d2a/d62/db2/d34/c158 0 2024-04-22T23:24:02.834 INFO:tasks.workunit.client.1.smithi134.stdout:3/944: write dc/d2a/d62/db2/d34/d6b/dfe/d14f/d150/f141 [73263,25520] 0 2024-04-22T23:24:02.851 INFO:tasks.workunit.client.1.smithi134.stdout:3/945: dread dc/d2a/d62/f65 [0,4194304] 0 2024-04-22T23:24:02.851 INFO:tasks.workunit.client.1.smithi134.stdout:3/946: stat dc/d2a/d62/db2/d2f/d61/d6a/l130 0 2024-04-22T23:24:02.853 INFO:tasks.workunit.client.1.smithi134.stdout:3/947: symlink dc/d2a/d62/db2/d2f/d21/d42/d77/d8f/l159 0 2024-04-22T23:24:02.857 INFO:tasks.workunit.client.1.smithi134.stdout:3/948: getdents dc/d2a/d62/db2/d34/d6b/dae/dc3 0 2024-04-22T23:24:02.860 INFO:tasks.workunit.client.1.smithi134.stdout:3/949: symlink dc/d2a/d62/db2/d2f/d54/dac/d115/l15a 0 2024-04-22T23:24:02.864 INFO:tasks.workunit.client.1.smithi134.stdout:3/950: rename dc/d2a/d62/db2/d2f/d36/d5f/l8b to dc/d2a/d62/db2/d34/l15b 0 2024-04-22T23:24:02.867 INFO:tasks.workunit.client.1.smithi134.stdout:3/951: creat dc/d2a/d62/db2/d2f/f15c x:0 0 0 2024-04-22T23:24:02.871 INFO:tasks.workunit.client.1.smithi134.stdout:3/952: dread dc/d2a/d62/db2/d2f/d21/d42/d77/d8f/f136 [0,4194304] 0 2024-04-22T23:24:02.875 INFO:tasks.workunit.client.1.smithi134.stdout:3/953: creat dc/d2a/d62/db2/d34/d6b/dae/def/f15d x:0 0 0 2024-04-22T23:24:02.877 INFO:tasks.workunit.client.1.smithi134.stdout:3/954: mknod dc/dd/c15e 0 2024-04-22T23:24:02.981 INFO:tasks.workunit.client.1.smithi134.stdout:3/955: sync 2024-04-22T23:24:02.998 INFO:tasks.workunit.client.1.smithi134.stdout:3/956: dwrite dc/d2a/d62/f65 [0,4194304] 0 2024-04-22T23:24:03.001 INFO:tasks.workunit.client.1.smithi134.stdout:3/957: unlink dc/d2a/d62/db2/d34/f153 0 2024-04-22T23:24:03.002 INFO:tasks.workunit.client.1.smithi134.stdout:3/958: fsync dc/d2a/d62/db2/d2f/d21/d42/d77/d137/dc1/f11d 0 2024-04-22T23:24:03.006 INFO:tasks.workunit.client.1.smithi134.stdout:3/959: dread dc/d2a/d62/db2/d34/d6b/dfe/d14f/fb8 [0,4194304] 0 2024-04-22T23:24:03.010 INFO:tasks.workunit.client.1.smithi134.stdout:3/960: symlink dc/d2a/d62/db2/d34/d6b/d13d/l15f 0 2024-04-22T23:24:03.021 INFO:tasks.workunit.client.1.smithi134.stdout:3/961: truncate dc/d2a/d62/db2/d2f/d21/d42/d77/fb6 2193523 0 2024-04-22T23:24:03.022 INFO:tasks.workunit.client.1.smithi134.stdout:3/962: creat dc/d2a/d62/d103/d11e/f160 x:0 0 0 2024-04-22T23:24:03.022 INFO:tasks.workunit.client.1.smithi134.stdout:3/963: fdatasync dc/dd/f7a 0 2024-04-22T23:24:03.025 INFO:tasks.workunit.client.1.smithi134.stdout:3/964: creat dc/d2a/d62/db2/d34/d70/f161 x:0 0 0 2024-04-22T23:24:03.026 INFO:tasks.workunit.client.1.smithi134.stdout:3/965: write dc/dd/f4f [199949,41440] 0 2024-04-22T23:24:03.027 INFO:tasks.workunit.client.1.smithi134.stdout:3/966: write dc/d2a/d62/d103/f12c [743769,69381] 0 2024-04-22T23:24:03.029 INFO:tasks.workunit.client.1.smithi134.stdout:3/967: read dc/dd/f4f [4185584,43692] 0 2024-04-22T23:24:03.030 INFO:tasks.workunit.client.1.smithi134.stdout:3/968: dread - dc/d2a/d62/db2/d34/d6b/dae/def/f15d zero size 2024-04-22T23:24:03.031 INFO:tasks.workunit.client.1.smithi134.stdout:3/969: fdatasync dc/d2a/d62/d103/f12c 0 2024-04-22T23:24:03.033 INFO:tasks.workunit.client.1.smithi134.stdout:3/970: mkdir dc/d2a/d62/db2/d2f/d36/d6d/d80/d162 0 2024-04-22T23:24:03.037 INFO:tasks.workunit.client.1.smithi134.stdout:3/971: unlink dc/d2a/d62/db2/d3b/f31 0 2024-04-22T23:24:03.037 INFO:tasks.workunit.client.1.smithi134.stdout:3/972: readlink dc/d2a/d62/l116 0 2024-04-22T23:24:03.053 INFO:tasks.workunit.client.1.smithi134.stdout:3/973: truncate dc/d2a/d62/db2/d2f/d21/d42/d77/fb6 847392 0 2024-04-22T23:24:03.053 INFO:tasks.workunit.client.1.smithi134.stdout:3/974: readlink dc/d2a/d62/db2/l26 0 2024-04-22T23:24:03.055 INFO:tasks.workunit.client.1.smithi134.stdout:3/975: write dc/d2a/d62/db2/d2f/d21/d42/f134 [650349,97215] 0 2024-04-22T23:24:03.056 INFO:tasks.workunit.client.1.smithi134.stdout:3/976: chown dc/d2a/d62/db2/d34/d6b/dae/dc3/df2 216192 1 2024-04-22T23:24:03.062 INFO:tasks.workunit.client.1.smithi134.stdout:3/977: dread dc/d2a/d62/db2/d2f/d54/fd0 [0,4194304] 0 2024-04-22T23:24:03.065 INFO:tasks.workunit.client.1.smithi134.stdout:3/978: dread dc/d2a/d62/db2/d2f/f51 [0,4194304] 0 2024-04-22T23:24:03.069 INFO:tasks.workunit.client.1.smithi134.stdout:3/979: dread dc/d2a/d62/d103/f12c [0,4194304] 0 2024-04-22T23:24:03.070 INFO:tasks.workunit.client.1.smithi134.stdout:3/980: fsync dc/f52 0 2024-04-22T23:24:03.073 INFO:tasks.workunit.client.1.smithi134.stdout:3/981: truncate dc/d2a/d62/db2/d2f/d21/d42/d77/d137/db3/fc8 698748 0 2024-04-22T23:24:03.075 INFO:tasks.workunit.client.1.smithi134.stdout:3/982: mknod dc/d2a/d62/db2/d2f/d36/c163 0 2024-04-22T23:24:03.085 INFO:tasks.workunit.client.1.smithi134.stdout:3/983: rmdir dc/d2a/d62 39 2024-04-22T23:24:03.092 INFO:tasks.workunit.client.1.smithi134.stdout:3/984: dwrite dc/d2a/d62/db2/d2f/d21/d42/d77/d137/dc1/f13a [0,4194304] 0 2024-04-22T23:24:03.104 INFO:tasks.workunit.client.1.smithi134.stdout:3/985: dwrite dc/d2a/d62/db2/d2f/d36/d6d/fb4 [0,4194304] 0 2024-04-22T23:24:03.106 INFO:tasks.workunit.client.1.smithi134.stdout:3/986: mknod dc/d2a/d62/db2/d2f/d13e/c164 0 2024-04-22T23:24:03.107 INFO:tasks.workunit.client.1.smithi134.stdout:3/987: chown dc/d2a/d62/db2/d34/d70/f161 9361254 1 2024-04-22T23:24:03.274 INFO:tasks.workunit.client.1.smithi134.stdout:3/988: sync 2024-04-22T23:24:03.275 INFO:tasks.workunit.client.1.smithi134.stdout:3/989: chown dc/d2a/d62/db2/d2f/l58 284352554 1 2024-04-22T23:24:03.277 INFO:tasks.workunit.client.1.smithi134.stdout:3/990: mknod dc/d2a/d62/db2/d3b/c165 0 2024-04-22T23:24:03.278 INFO:tasks.workunit.client.1.smithi134.stdout:3/991: chown dc/d2a/d62/db2/d34/d6b/fe9 1374 1 2024-04-22T23:24:03.279 INFO:tasks.workunit.client.1.smithi134.stdout:3/992: stat dc/d7d/laa 0 2024-04-22T23:24:03.283 INFO:tasks.workunit.client.1.smithi134.stdout:3/993: write dc/d2a/d62/db2/d34/d6b/dae/fab [572307,16986] 0 2024-04-22T23:24:03.285 INFO:tasks.workunit.client.1.smithi134.stdout:3/994: creat dc/d2a/d62/db2/d34/d6b/dfe/f166 x:0 0 0 2024-04-22T23:24:03.291 INFO:tasks.workunit.client.1.smithi134.stdout:3/995: dwrite dc/dd/f4f [0,4194304] 0 2024-04-22T23:24:03.292 INFO:tasks.workunit.client.1.smithi134.stdout:3/996: stat dc/d2a/d62/db2/d34/d6b/f9a 0 2024-04-22T23:24:03.294 INFO:tasks.workunit.client.1.smithi134.stdout:3/997: rename dc/d2a/d62/db2/d2f/d21/c12e to dc/d2a/d62/db2/d2f/d36/d5f/c167 0 2024-04-22T23:24:03.297 INFO:tasks.workunit.client.1.smithi134.stdout:3/998: link dc/d2a/d62/fa5 dc/d2a/d62/db2/d34/d6b/dae/def/f168 0 2024-04-22T23:24:03.298 INFO:tasks.workunit.client.1.smithi134.stdout:3/999: readlink dc/d2a/d62/db2/d2f/d21/d42/d77/d137/df6/l110 0 2024-04-22T23:24:03.301 INFO:tasks.workunit.client.1.smithi134.stderr:+ rm -rf -- ./tmp.FX8jKn5bsq 2024-04-22T23:24:04.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:04 smithi134 ceph-mon[34609]: pgmap v6: 65 pgs: 65 active+clean; 2.4 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 3.6 MiB/s rd, 14 MiB/s wr, 18 op/s 2024-04-22T23:24:04.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:04 smithi067 ceph-mon[26453]: pgmap v6: 65 pgs: 65 active+clean; 2.4 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 3.6 MiB/s rd, 14 MiB/s wr, 18 op/s 2024-04-22T23:24:06.628 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:06 smithi134 ceph-mon[34609]: pgmap v7: 65 pgs: 65 active+clean; 2.4 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 38 MiB/s rd, 85 MiB/s wr, 255 op/s 2024-04-22T23:24:06.629 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:06 smithi134 ceph-mon[34609]: Standby manager daemon smithi067.whdcsr started 2024-04-22T23:24:06.629 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:06 smithi134 ceph-mon[34609]: from='mgr.? 172.21.15.67:0/1253674566' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi067.whdcsr/crt"}]: dispatch 2024-04-22T23:24:06.629 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:06 smithi134 ceph-mon[34609]: from='mgr.? 172.21.15.67:0/1253674566' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T23:24:06.629 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:06 smithi134 ceph-mon[34609]: from='mgr.? 172.21.15.67:0/1253674566' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi067.whdcsr/key"}]: dispatch 2024-04-22T23:24:06.629 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:06 smithi134 ceph-mon[34609]: from='mgr.? 172.21.15.67:0/1253674566' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T23:24:06.653 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:06 smithi067 ceph-mon[26453]: pgmap v7: 65 pgs: 65 active+clean; 2.4 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 38 MiB/s rd, 85 MiB/s wr, 255 op/s 2024-04-22T23:24:06.653 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:06 smithi067 ceph-mon[26453]: Standby manager daemon smithi067.whdcsr started 2024-04-22T23:24:06.653 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:06 smithi067 ceph-mon[26453]: from='mgr.? 172.21.15.67:0/1253674566' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi067.whdcsr/crt"}]: dispatch 2024-04-22T23:24:06.653 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:06 smithi067 ceph-mon[26453]: from='mgr.? 172.21.15.67:0/1253674566' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T23:24:06.653 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:06 smithi067 ceph-mon[26453]: from='mgr.? 172.21.15.67:0/1253674566' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi067.whdcsr/key"}]: dispatch 2024-04-22T23:24:06.654 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:06 smithi067 ceph-mon[26453]: from='mgr.? 172.21.15.67:0/1253674566' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T23:24:07.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:07 smithi134 ceph-mon[34609]: mgrmap e25: smithi134.ccxkcj(active, since 9s), standbys: smithi067.whdcsr 2024-04-22T23:24:07.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:07 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "mgr metadata", "who": "smithi067.whdcsr", "id": "smithi067.whdcsr"}]: dispatch 2024-04-22T23:24:07.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:07 smithi067 ceph-mon[26453]: mgrmap e25: smithi134.ccxkcj(active, since 9s), standbys: smithi067.whdcsr 2024-04-22T23:24:07.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:07 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "mgr metadata", "who": "smithi067.whdcsr", "id": "smithi067.whdcsr"}]: dispatch 2024-04-22T23:24:08.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:08 smithi134 ceph-mon[34609]: pgmap v8: 65 pgs: 65 active+clean; 2.4 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 30 MiB/s rd, 66 MiB/s wr, 198 op/s 2024-04-22T23:24:08.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:08 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:08.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:08 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:08.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:08 smithi067 ceph-mon[26453]: pgmap v8: 65 pgs: 65 active+clean; 2.4 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 30 MiB/s rd, 66 MiB/s wr, 198 op/s 2024-04-22T23:24:08.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:08 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:08.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:08 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:10.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:10 smithi134 ceph-mon[34609]: pgmap v9: 65 pgs: 65 active+clean; 2.0 GiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 26 MiB/s rd, 57 MiB/s wr, 270 op/s 2024-04-22T23:24:10.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:10 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:10.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:10 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:10.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:10 smithi067 ceph-mon[26453]: pgmap v9: 65 pgs: 65 active+clean; 2.0 GiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 26 MiB/s rd, 57 MiB/s wr, 270 op/s 2024-04-22T23:24:10.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:10 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:10.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:10 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:12.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:12 smithi067 ceph-mon[26453]: pgmap v10: 65 pgs: 65 active+clean; 2.0 GiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 24 MiB/s rd, 52 MiB/s wr, 248 op/s 2024-04-22T23:24:12.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:12 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:24:12.848 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:12 smithi134 ceph-mon[34609]: pgmap v10: 65 pgs: 65 active+clean; 2.0 GiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 24 MiB/s rd, 52 MiB/s wr, 248 op/s 2024-04-22T23:24:12.848 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:12 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:24:14.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:14 smithi134 ceph-mon[34609]: pgmap v11: 65 pgs: 65 active+clean; 1.9 GiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 24 MiB/s rd, 52 MiB/s wr, 253 op/s 2024-04-22T23:24:14.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:14 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:14.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:14 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:14.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:14 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:24:14.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:14 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:24:14.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:14 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:24:14.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:14 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:24:14.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:14 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:24:14.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:14 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:24:14.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:14 smithi134 ceph-mon[34609]: Adjusting osd_memory_target on smithi134 to 2240M 2024-04-22T23:24:14.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:14 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:14.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:14 smithi067 ceph-mon[26453]: pgmap v11: 65 pgs: 65 active+clean; 1.9 GiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 24 MiB/s rd, 52 MiB/s wr, 253 op/s 2024-04-22T23:24:14.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:14 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:14.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:14 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:14.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:14 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:24:14.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:14 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:24:14.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:14 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:24:14.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:14 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:24:14.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:14 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:24:14.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:14 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:24:14.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:14 smithi067 ceph-mon[26453]: Adjusting osd_memory_target on smithi134 to 2240M 2024-04-22T23:24:14.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:14 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:16.511 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:16 smithi067 ceph-mon[26453]: pgmap v12: 65 pgs: 65 active+clean; 1.3 GiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 23 MiB/s rd, 47 MiB/s wr, 309 op/s 2024-04-22T23:24:16.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:16 smithi134 ceph-mon[34609]: pgmap v12: 65 pgs: 65 active+clean; 1.3 GiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 23 MiB/s rd, 47 MiB/s wr, 309 op/s 2024-04-22T23:24:18.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:18 smithi067 ceph-mon[26453]: pgmap v13: 65 pgs: 65 active+clean; 1.3 GiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 2.1 MiB/s rd, 3.4 MiB/s wr, 168 op/s 2024-04-22T23:24:18.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:18 smithi134 ceph-mon[34609]: pgmap v13: 65 pgs: 65 active+clean; 1.3 GiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 2.1 MiB/s rd, 3.4 MiB/s wr, 168 op/s 2024-04-22T23:24:19.556 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-22T23:24:19.556 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2024-04-22T23:24:19.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:19 smithi134 ceph-mon[34609]: pgmap v14: 65 pgs: 65 active+clean; 855 MiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 2.9 MiB/s rd, 4.1 MiB/s wr, 234 op/s 2024-04-22T23:24:19.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:19 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:19.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:19 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:20.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:19 smithi067 ceph-mon[26453]: pgmap v14: 65 pgs: 65 active+clean; 855 MiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 2.9 MiB/s rd, 4.1 MiB/s wr, 234 op/s 2024-04-22T23:24:20.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:19 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:20.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:19 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:21.493 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:21 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:21.493 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:21 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:21.493 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:21 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:24:21.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:21 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:24:21.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:21 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:24:21.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:21 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:24:21.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:21 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:24:21.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:21 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:24:21.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:21 smithi134 ceph-mon[34609]: Adjusting osd_memory_target on smithi067 to 1216M 2024-04-22T23:24:21.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:21 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:21.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:21 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:24:21.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:21 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:24:21.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:21 smithi134 ceph-mon[34609]: Updating smithi067:/etc/ceph/ceph.conf 2024-04-22T23:24:21.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:21 smithi134 ceph-mon[34609]: Updating smithi134:/etc/ceph/ceph.conf 2024-04-22T23:24:21.585 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:21 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:21.586 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:21 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:21.586 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:21 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:24:21.586 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:21 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:24:21.586 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:21 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:24:21.586 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:21 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:24:21.586 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:21 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:24:21.586 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:21 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T23:24:21.586 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:21 smithi067 ceph-mon[26453]: Adjusting osd_memory_target on smithi067 to 1216M 2024-04-22T23:24:21.586 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:21 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:21.586 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:21 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:24:21.587 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:21 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:24:21.587 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:21 smithi067 ceph-mon[26453]: Updating smithi067:/etc/ceph/ceph.conf 2024-04-22T23:24:21.587 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:21 smithi067 ceph-mon[26453]: Updating smithi134:/etc/ceph/ceph.conf 2024-04-22T23:24:22.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:22 smithi067 ceph-mon[26453]: Updating smithi134:/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T23:24:22.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:22 smithi067 ceph-mon[26453]: Updating smithi067:/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T23:24:22.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:22 smithi067 ceph-mon[26453]: pgmap v15: 65 pgs: 65 active+clean; 855 MiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 135 op/s 2024-04-22T23:24:22.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:22 smithi067 ceph-mon[26453]: Updating smithi134:/etc/ceph/ceph.client.admin.keyring 2024-04-22T23:24:22.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:22 smithi067 ceph-mon[26453]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-04-22T23:24:22.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:22 smithi067 ceph-mon[26453]: Updating smithi134:/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T23:24:22.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:22 smithi067 ceph-mon[26453]: Updating smithi067:/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T23:24:22.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:22 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:22.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:22 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:22.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:22 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:22.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:22 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:22.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:22 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:22.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:22 smithi134 ceph-mon[34609]: Updating smithi134:/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T23:24:22.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:22 smithi134 ceph-mon[34609]: Updating smithi067:/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T23:24:22.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:22 smithi134 ceph-mon[34609]: pgmap v15: 65 pgs: 65 active+clean; 855 MiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 135 op/s 2024-04-22T23:24:22.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:22 smithi134 ceph-mon[34609]: Updating smithi134:/etc/ceph/ceph.client.admin.keyring 2024-04-22T23:24:22.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:22 smithi134 ceph-mon[34609]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-04-22T23:24:22.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:22 smithi134 ceph-mon[34609]: Updating smithi134:/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T23:24:22.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:22 smithi134 ceph-mon[34609]: Updating smithi067:/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T23:24:22.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:22 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:22.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:22 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:22.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:22 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:22.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:22 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:22.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:22 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:23.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:23 smithi067 ceph-mon[26453]: Reconfiguring prometheus.smithi067 (dependencies changed)... 2024-04-22T23:24:23.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:23 smithi067 ceph-mon[26453]: Reconfiguring daemon prometheus.smithi067 on smithi067 2024-04-22T23:24:23.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:23 smithi134 ceph-mon[34609]: Reconfiguring prometheus.smithi067 (dependencies changed)... 2024-04-22T23:24:23.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:23 smithi134 ceph-mon[34609]: Reconfiguring daemon prometheus.smithi067 on smithi067 2024-04-22T23:24:24.788 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:24 smithi067 ceph-mon[26453]: pgmap v16: 65 pgs: 65 active+clean; 678 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 151 op/s 2024-04-22T23:24:24.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:24 smithi134 ceph-mon[34609]: pgmap v16: 65 pgs: 65 active+clean; 678 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 151 op/s 2024-04-22T23:24:26.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:26.711+0000 7f2758acb700 1 -- 172.21.15.67:0/3606777437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2754101f40 msgr2=0x7f27541023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:24:26.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:26.711+0000 7f2758acb700 1 --2- 172.21.15.67:0/3606777437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2754101f40 0x7f27541023a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2748009910 tx=0x7f2748009c20 comp rx=0 tx=0).stop 2024-04-22T23:24:26.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:26.711+0000 7f2758acb700 1 -- 172.21.15.67:0/3606777437 shutdown_connections 2024-04-22T23:24:26.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:26.711+0000 7f2758acb700 1 --2- 172.21.15.67:0/3606777437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2754101f40 0x7f27541023a0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:24:26.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:26.711+0000 7f2758acb700 1 --2- 172.21.15.67:0/3606777437 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f27540fef60 0x7f27540ff380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:24:26.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:26.711+0000 7f2758acb700 1 -- 172.21.15.67:0/3606777437 >> 172.21.15.67:0/3606777437 conn(0x7f27540fa8c0 msgr2=0x7f27540fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:24:26.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:26.711+0000 7f2758acb700 1 -- 172.21.15.67:0/3606777437 shutdown_connections 2024-04-22T23:24:26.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:26.711+0000 7f2758acb700 1 -- 172.21.15.67:0/3606777437 wait complete. 2024-04-22T23:24:26.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:26.712+0000 7f2758acb700 1 Processor -- start 2024-04-22T23:24:26.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:26.712+0000 7f2758acb700 1 -- start start 2024-04-22T23:24:26.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:26.713+0000 7f2758acb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27540fef60 0x7f27541962a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:24:26.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:26.713+0000 7f2758acb700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2754101f40 0x7f27541967e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:24:26.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:26.713+0000 7f2758acb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2754196e00 con 0x7f27540fef60 2024-04-22T23:24:26.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:26.713+0000 7f2758acb700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2754196f40 con 0x7f2754101f40 2024-04-22T23:24:26.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:26.713+0000 7f275259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27540fef60 0x7f27541962a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:24:26.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:26.713+0000 7f275259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27540fef60 0x7f27541962a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57436/0 (socket says 172.21.15.67:57436) 2024-04-22T23:24:26.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:26.713+0000 7f275259c700 1 -- 172.21.15.67:0/1174154765 learned_addr learned my addr 172.21.15.67:0/1174154765 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:24:26.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:26.713+0000 7f2751d9b700 1 --2- 172.21.15.67:0/1174154765 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2754101f40 0x7f27541967e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:24:26.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:26.713+0000 7f275259c700 1 -- 172.21.15.67:0/1174154765 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2754101f40 msgr2=0x7f27541967e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:24:26.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:26.713+0000 7f275259c700 1 --2- 172.21.15.67:0/1174154765 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2754101f40 0x7f27541967e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:24:26.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:26.713+0000 7f275259c700 1 -- 172.21.15.67:0/1174154765 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27480095f0 con 0x7f27540fef60 2024-04-22T23:24:26.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:26.714+0000 7f275259c700 1 --2- 172.21.15.67:0/1174154765 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27540fef60 0x7f27541962a0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f274400e8f0 tx=0x7f274400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:24:26.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:26.714+0000 7f27437fe700 1 -- 172.21.15.67:0/1174154765 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2744009790 con 0x7f27540fef60 2024-04-22T23:24:26.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:26.714+0000 7f2758acb700 1 -- 172.21.15.67:0/1174154765 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f275419b9f0 con 0x7f27540fef60 2024-04-22T23:24:26.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:26.714+0000 7f27437fe700 1 -- 172.21.15.67:0/1174154765 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f274400c810 con 0x7f27540fef60 2024-04-22T23:24:26.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:26.714+0000 7f2758acb700 1 -- 172.21.15.67:0/1174154765 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f275419bf40 con 0x7f27540fef60 2024-04-22T23:24:26.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:26.715+0000 7f27437fe700 1 -- 172.21.15.67:0/1174154765 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2744003d20 con 0x7f27540fef60 2024-04-22T23:24:26.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:26.715+0000 7f2758acb700 1 -- 172.21.15.67:0/1174154765 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2754109800 con 0x7f27540fef60 2024-04-22T23:24:26.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:26.716+0000 7f27437fe700 1 -- 172.21.15.67:0/1174154765 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 25) v1 ==== 91559+0+0 (secure 0 0 0) 0x7f27440041f0 con 0x7f27540fef60 2024-04-22T23:24:26.718 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:26 smithi067 ceph-mon[26453]: pgmap v17: 65 pgs: 65 active+clean; 298 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 212 op/s 2024-04-22T23:24:26.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:26.716+0000 7f27437fe700 1 --2- 172.21.15.67:0/1174154765 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f273c06d850 0x7f273c06fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:24:26.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:26.717+0000 7f27437fe700 1 -- 172.21.15.67:0/1174154765 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f2744014070 con 0x7f27540fef60 2024-04-22T23:24:26.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:26.717+0000 7f2751d9b700 1 --2- 172.21.15.67:0/1174154765 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f273c06d850 0x7f273c06fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:24:26.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:26.717+0000 7f2751d9b700 1 --2- 172.21.15.67:0/1174154765 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f273c06d850 0x7f273c06fd10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2748009fd0 tx=0x7f2748000bc0 comp rx=0 tx=0).ready entity=mgr.14622 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:24:26.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:26.726+0000 7f27437fe700 1 -- 172.21.15.67:0/1174154765 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f274405a2b0 con 0x7f27540fef60 2024-04-22T23:24:26.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:26 smithi134 ceph-mon[34609]: pgmap v17: 65 pgs: 65 active+clean; 298 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 212 op/s 2024-04-22T23:24:27.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.050+0000 7f2758acb700 1 -- 172.21.15.67:0/1174154765 --> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f275419c690 con 0x7f273c06d850 2024-04-22T23:24:27.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.052+0000 7f27437fe700 1 -- 172.21.15.67:0/1174154765 <== mgr.14622 v2:172.21.15.134:6828/1077503272 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f275419c690 con 0x7f273c06d850 2024-04-22T23:24:27.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.054+0000 7f2758acb700 1 -- 172.21.15.67:0/1174154765 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f273c06d850 msgr2=0x7f273c06fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:24:27.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.054+0000 7f2758acb700 1 --2- 172.21.15.67:0/1174154765 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f273c06d850 0x7f273c06fd10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2748009fd0 tx=0x7f2748000bc0 comp rx=0 tx=0).stop 2024-04-22T23:24:27.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.054+0000 7f2758acb700 1 -- 172.21.15.67:0/1174154765 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27540fef60 msgr2=0x7f27541962a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:24:27.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.054+0000 7f2758acb700 1 --2- 172.21.15.67:0/1174154765 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27540fef60 0x7f27541962a0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f274400e8f0 tx=0x7f274400ec00 comp rx=0 tx=0).stop 2024-04-22T23:24:27.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.054+0000 7f2758acb700 1 -- 172.21.15.67:0/1174154765 shutdown_connections 2024-04-22T23:24:27.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.054+0000 7f2758acb700 1 --2- 172.21.15.67:0/1174154765 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f273c06d850 0x7f273c06fd10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:24:27.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.054+0000 7f2758acb700 1 --2- 172.21.15.67:0/1174154765 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27540fef60 0x7f27541962a0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:24:27.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.054+0000 7f2758acb700 1 --2- 172.21.15.67:0/1174154765 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2754101f40 0x7f27541967e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:24:27.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.054+0000 7f2758acb700 1 -- 172.21.15.67:0/1174154765 >> 172.21.15.67:0/1174154765 conn(0x7f27540fa8c0 msgr2=0x7f2754105170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:24:27.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.054+0000 7f2758acb700 1 -- 172.21.15.67:0/1174154765 shutdown_connections 2024-04-22T23:24:27.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.054+0000 7f2758acb700 1 -- 172.21.15.67:0/1174154765 wait complete. 2024-04-22T23:24:27.071 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:24:27.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.188+0000 7f32e8b41700 1 -- 172.21.15.67:0/861710371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32e40fef80 msgr2=0x7f32e40ff3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:24:27.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.188+0000 7f32e8b41700 1 --2- 172.21.15.67:0/861710371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32e40fef80 0x7f32e40ff3a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f32d4009910 tx=0x7f32d4009c20 comp rx=0 tx=0).stop 2024-04-22T23:24:27.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.189+0000 7f32e8b41700 1 -- 172.21.15.67:0/861710371 shutdown_connections 2024-04-22T23:24:27.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.189+0000 7f32e8b41700 1 --2- 172.21.15.67:0/861710371 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f32e4102000 0x7f32e4102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:24:27.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.189+0000 7f32e8b41700 1 --2- 172.21.15.67:0/861710371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32e40fef80 0x7f32e40ff3a0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:24:27.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.189+0000 7f32e8b41700 1 -- 172.21.15.67:0/861710371 >> 172.21.15.67:0/861710371 conn(0x7f32e40fa8c0 msgr2=0x7f32e40fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:24:27.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.189+0000 7f32e8b41700 1 -- 172.21.15.67:0/861710371 shutdown_connections 2024-04-22T23:24:27.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.189+0000 7f32e8b41700 1 -- 172.21.15.67:0/861710371 wait complete. 2024-04-22T23:24:27.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.190+0000 7f32e8b41700 1 Processor -- start 2024-04-22T23:24:27.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.190+0000 7f32e8b41700 1 -- start start 2024-04-22T23:24:27.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.190+0000 7f32e8b41700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f32e40fef80 0x7f32e4196510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:24:27.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.190+0000 7f32e8b41700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32e4102000 0x7f32e4196a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:24:27.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.190+0000 7f32e8b41700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e4197070 con 0x7f32e4102000 2024-04-22T23:24:27.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.190+0000 7f32e8b41700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e41971b0 con 0x7f32e40fef80 2024-04-22T23:24:27.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.190+0000 7f32e1d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32e4102000 0x7f32e4196a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:24:27.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.190+0000 7f32e1d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32e4102000 0x7f32e4196a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57462/0 (socket says 172.21.15.67:57462) 2024-04-22T23:24:27.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.190+0000 7f32e1d9b700 1 -- 172.21.15.67:0/2687448337 learned_addr learned my addr 172.21.15.67:0/2687448337 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:24:27.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.190+0000 7f32e259c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f32e40fef80 0x7f32e4196510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:24:27.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.190+0000 7f32e1d9b700 1 -- 172.21.15.67:0/2687448337 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f32e40fef80 msgr2=0x7f32e4196510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:24:27.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.190+0000 7f32e1d9b700 1 --2- 172.21.15.67:0/2687448337 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f32e40fef80 0x7f32e4196510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:24:27.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.190+0000 7f32e1d9b700 1 -- 172.21.15.67:0/2687448337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d40095f0 con 0x7f32e4102000 2024-04-22T23:24:27.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.190+0000 7f32e259c700 1 --2- 172.21.15.67:0/2687448337 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f32e40fef80 0x7f32e4196510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T23:24:27.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.191+0000 7f32e1d9b700 1 --2- 172.21.15.67:0/2687448337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32e4102000 0x7f32e4196a50 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f32d800e920 tx=0x7f32d800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:24:27.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.191+0000 7f32d37fe700 1 -- 172.21.15.67:0/2687448337 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32d800c9d0 con 0x7f32e4102000 2024-04-22T23:24:27.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.191+0000 7f32e8b41700 1 -- 172.21.15.67:0/2687448337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32e419bc60 con 0x7f32e4102000 2024-04-22T23:24:27.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.191+0000 7f32d37fe700 1 -- 172.21.15.67:0/2687448337 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f32d8010430 con 0x7f32e4102000 2024-04-22T23:24:27.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.191+0000 7f32d37fe700 1 -- 172.21.15.67:0/2687448337 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32d8018650 con 0x7f32e4102000 2024-04-22T23:24:27.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.191+0000 7f32e8b41700 1 -- 172.21.15.67:0/2687448337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32e419c1b0 con 0x7f32e4102000 2024-04-22T23:24:27.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.192+0000 7f32e8b41700 1 -- 172.21.15.67:0/2687448337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32e4109800 con 0x7f32e4102000 2024-04-22T23:24:27.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.193+0000 7f32d37fe700 1 -- 172.21.15.67:0/2687448337 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 25) v1 ==== 91559+0+0 (secure 0 0 0) 0x7f32d80105a0 con 0x7f32e4102000 2024-04-22T23:24:27.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.193+0000 7f32d37fe700 1 --2- 172.21.15.67:0/2687448337 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f32cc06d850 0x7f32cc06fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:24:27.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.193+0000 7f32d37fe700 1 -- 172.21.15.67:0/2687448337 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f32d8014070 con 0x7f32e4102000 2024-04-22T23:24:27.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.193+0000 7f32e259c700 1 --2- 172.21.15.67:0/2687448337 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f32cc06d850 0x7f32cc06fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:24:27.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.194+0000 7f32e259c700 1 --2- 172.21.15.67:0/2687448337 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f32cc06d850 0x7f32cc06fd10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f32d400b6f0 tx=0x7f32d4005940 comp rx=0 tx=0).ready entity=mgr.14622 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:24:27.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.198+0000 7f32d37fe700 1 -- 172.21.15.67:0/2687448337 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f32d8053e20 con 0x7f32e4102000 2024-04-22T23:24:27.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.513+0000 7f32e8b41700 1 -- 172.21.15.67:0/2687448337 --> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f32e4106950 con 0x7f32cc06d850 2024-04-22T23:24:27.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.515+0000 7f32d37fe700 1 -- 172.21.15.67:0/2687448337 <== mgr.14622 v2:172.21.15.134:6828/1077503272 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f32e4106950 con 0x7f32cc06d850 2024-04-22T23:24:27.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.518+0000 7f32e8b41700 1 -- 172.21.15.67:0/2687448337 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f32cc06d850 msgr2=0x7f32cc06fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:24:27.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.518+0000 7f32e8b41700 1 --2- 172.21.15.67:0/2687448337 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f32cc06d850 0x7f32cc06fd10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f32d400b6f0 tx=0x7f32d4005940 comp rx=0 tx=0).stop 2024-04-22T23:24:27.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.518+0000 7f32e8b41700 1 -- 172.21.15.67:0/2687448337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32e4102000 msgr2=0x7f32e4196a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:24:27.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.518+0000 7f32e8b41700 1 --2- 172.21.15.67:0/2687448337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32e4102000 0x7f32e4196a50 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f32d800e920 tx=0x7f32d800ec30 comp rx=0 tx=0).stop 2024-04-22T23:24:27.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.519+0000 7f32e8b41700 1 -- 172.21.15.67:0/2687448337 shutdown_connections 2024-04-22T23:24:27.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.519+0000 7f32e8b41700 1 --2- 172.21.15.67:0/2687448337 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f32cc06d850 0x7f32cc06fd10 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:24:27.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.519+0000 7f32e8b41700 1 --2- 172.21.15.67:0/2687448337 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f32e40fef80 0x7f32e4196510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:24:27.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.519+0000 7f32e8b41700 1 --2- 172.21.15.67:0/2687448337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32e4102000 0x7f32e4196a50 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:24:27.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.519+0000 7f32e8b41700 1 -- 172.21.15.67:0/2687448337 >> 172.21.15.67:0/2687448337 conn(0x7f32e40fa8c0 msgr2=0x7f32e4105230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:24:27.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.521+0000 7f32e8b41700 1 -- 172.21.15.67:0/2687448337 shutdown_connections 2024-04-22T23:24:27.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.521+0000 7f32e8b41700 1 -- 172.21.15.67:0/2687448337 wait complete. 2024-04-22T23:24:27.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.710+0000 7f8ef3805700 1 -- 172.21.15.67:0/333106179 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8eec1072c0 msgr2=0x7f8eec1076e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:24:27.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.710+0000 7f8ef3805700 1 --2- 172.21.15.67:0/333106179 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8eec1072c0 0x7f8eec1076e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8ee8009910 tx=0x7f8ee8009c20 comp rx=0 tx=0).stop 2024-04-22T23:24:27.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.710+0000 7f8ef3805700 1 -- 172.21.15.67:0/333106179 shutdown_connections 2024-04-22T23:24:27.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.710+0000 7f8ef3805700 1 --2- 172.21.15.67:0/333106179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8eec1084c0 0x7f8eec108920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:24:27.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.710+0000 7f8ef3805700 1 --2- 172.21.15.67:0/333106179 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8eec1072c0 0x7f8eec1076e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:24:27.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.710+0000 7f8ef3805700 1 -- 172.21.15.67:0/333106179 >> 172.21.15.67:0/333106179 conn(0x7f8eec0763d0 msgr2=0x7f8eec078830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:24:27.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.711+0000 7f8ef3805700 1 -- 172.21.15.67:0/333106179 shutdown_connections 2024-04-22T23:24:27.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.711+0000 7f8ef3805700 1 -- 172.21.15.67:0/333106179 wait complete. 2024-04-22T23:24:27.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.712+0000 7f8ef3805700 1 Processor -- start 2024-04-22T23:24:27.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.712+0000 7f8ef3805700 1 -- start start 2024-04-22T23:24:27.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.712+0000 7f8ef3805700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8eec1072c0 0x7f8eec1a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:24:27.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.712+0000 7f8ef3805700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8eec1084c0 0x7f8eec1a5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:24:27.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.712+0000 7f8ef3805700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eec1a6060 con 0x7f8eec1072c0 2024-04-22T23:24:27.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.712+0000 7f8ef3805700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eec1aa860 con 0x7f8eec1084c0 2024-04-22T23:24:27.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.714+0000 7f8ef0da0700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8eec1084c0 0x7f8eec1a5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:24:27.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.714+0000 7f8ef0da0700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8eec1084c0 0x7f8eec1a5a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.67:44682/0 (socket says 172.21.15.67:44682) 2024-04-22T23:24:27.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.714+0000 7f8ef0da0700 1 -- 172.21.15.67:0/327272632 learned_addr learned my addr 172.21.15.67:0/327272632 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:24:27.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.714+0000 7f8ef0da0700 1 -- 172.21.15.67:0/327272632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8eec1072c0 msgr2=0x7f8eec1a5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:24:27.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.714+0000 7f8ef0da0700 1 --2- 172.21.15.67:0/327272632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8eec1072c0 0x7f8eec1a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:24:27.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.714+0000 7f8ef0da0700 1 -- 172.21.15.67:0/327272632 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ee80095f0 con 0x7f8eec1084c0 2024-04-22T23:24:27.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.715+0000 7f8ef0da0700 1 --2- 172.21.15.67:0/327272632 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8eec1084c0 0x7f8eec1a5a40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8edc00d8d0 tx=0x7f8edc00dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:24:27.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.715+0000 7f8ee27fc700 1 -- 172.21.15.67:0/327272632 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8edc009940 con 0x7f8eec1084c0 2024-04-22T23:24:27.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.715+0000 7f8ef3805700 1 -- 172.21.15.67:0/327272632 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eec1aaa60 con 0x7f8eec1084c0 2024-04-22T23:24:27.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.715+0000 7f8ee27fc700 1 -- 172.21.15.67:0/327272632 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8edc010460 con 0x7f8eec1084c0 2024-04-22T23:24:27.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.715+0000 7f8ee27fc700 1 -- 172.21.15.67:0/327272632 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8edc01b620 con 0x7f8eec1084c0 2024-04-22T23:24:27.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.715+0000 7f8ef3805700 1 -- 172.21.15.67:0/327272632 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eec1aaf60 con 0x7f8eec1084c0 2024-04-22T23:24:27.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.717+0000 7f8ef3805700 1 -- 172.21.15.67:0/327272632 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eec06b000 con 0x7f8eec1084c0 2024-04-22T23:24:27.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.718+0000 7f8ee27fc700 1 -- 172.21.15.67:0/327272632 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 25) v1 ==== 91559+0+0 (secure 0 0 0) 0x7f8edc009aa0 con 0x7f8eec1084c0 2024-04-22T23:24:27.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.718+0000 7f8ee27fc700 1 --2- 172.21.15.67:0/327272632 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f8ed806d890 0x7f8ed806fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:24:27.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.719+0000 7f8ee27fc700 1 -- 172.21.15.67:0/327272632 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f8edc08d700 con 0x7f8eec1084c0 2024-04-22T23:24:27.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.719+0000 7f8ef15a1700 1 --2- 172.21.15.67:0/327272632 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f8ed806d890 0x7f8ed806fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:24:27.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.719+0000 7f8ef15a1700 1 --2- 172.21.15.67:0/327272632 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f8ed806d890 0x7f8ed806fd50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8ee800b5c0 tx=0x7f8ee80058e0 comp rx=0 tx=0).ready entity=mgr.14622 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:24:27.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:27.726+0000 7f8ee27fc700 1 -- 172.21.15.67:0/327272632 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f8edc05a350 con 0x7f8eec1084c0 2024-04-22T23:24:27.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:27 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:24:27.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:27 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:24:28.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.086+0000 7f8ef3805700 1 -- 172.21.15.67:0/327272632 --> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f8eec10ce10 con 0x7f8ed806d890 2024-04-22T23:24:28.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.098+0000 7f8ee27fc700 1 -- 172.21.15.67:0/327272632 <== mgr.14622 v2:172.21.15.134:6828/1077503272 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f8eec10ce10 con 0x7f8ed806d890 2024-04-22T23:24:28.099 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:24:28.099 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (8m) 9s ago 9m 26.1M - 0.25.0 c8568f914cd2 8c7b7ec00657 2024-04-22T23:24:28.099 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (9m) 9s ago 9m 7843k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:24:28.099 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (8m) 20s ago 8m 7805k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:24:28.099 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (9m) 9s ago 9m 6660k - 18.2.1 5be31c24972a 14085634c5d3 2024-04-22T23:24:28.099 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (8m) 20s ago 8m 6665k - 18.2.1 5be31c24972a fa995d458353 2024-04-22T23:24:28.100 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (7m) 9s ago 9m 86.7M - 9.4.7 954c08fa6188 0d8950c9af0c 2024-04-22T23:24:28.100 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (4m) 9s ago 4m 151M - 18.2.1 5be31c24972a e24a471f0a8c 2024-04-22T23:24:28.100 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (3m) 9s ago 3m 167M - 18.2.1 5be31c24972a 0730f9fc8e61 2024-04-22T23:24:28.100 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (3m) 20s ago 3m 166M - 18.2.1 5be31c24972a b6a169586433 2024-04-22T23:24:28.100 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (3m) 20s ago 3m 148M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:24:28.100 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:9283,8765,8443 running (10m) 9s ago 10m 417M - 18.2.1 5be31c24972a 1c64cb9e36bb 2024-04-22T23:24:28.100 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (75s) 20s ago 8m 488M - 19.0.0-3252-g43be0201 a0434699c1d7 a86aaa7b9552 2024-04-22T23:24:28.100 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (11m) 9s ago 11m 51.2M 2048M 18.2.1 5be31c24972a 6789283703e0 2024-04-22T23:24:28.100 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (8m) 20s ago 8m 39.6M 2048M 18.2.1 5be31c24972a 2fb23795f7e3 2024-04-22T23:24:28.100 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (9m) 9s ago 9m 16.4M - 1.5.0 0da6a335fe13 4aa4da7089d0 2024-04-22T23:24:28.100 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (8m) 20s ago 8m 15.4M - 1.5.0 0da6a335fe13 26e43c2e756c 2024-04-22T23:24:28.100 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (7m) 9s ago 7m 205M 1216M 18.2.1 5be31c24972a a033247ee8c2 2024-04-22T23:24:28.100 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (7m) 9s ago 7m 197M 1216M 18.2.1 5be31c24972a 915b07a8edc4 2024-04-22T23:24:28.100 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (6m) 9s ago 6m 156M 1216M 18.2.1 5be31c24972a e257ecb4efa1 2024-04-22T23:24:28.101 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (6m) 20s ago 6m 234M 2240M 18.2.1 5be31c24972a 9c2c91a967ba 2024-04-22T23:24:28.101 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (6m) 20s ago 5m 201M 2240M 18.2.1 5be31c24972a c7586562ffa6 2024-04-22T23:24:28.101 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (5m) 20s ago 5m 227M 2240M 18.2.1 5be31c24972a 9fd97f1c65cb 2024-04-22T23:24:28.101 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (7m) 9s ago 8m 57.1M - 2.43.0 a07b618ecd1d d87cb15b5e55 2024-04-22T23:24:28.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.101+0000 7f8ef3805700 1 -- 172.21.15.67:0/327272632 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f8ed806d890 msgr2=0x7f8ed806fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:24:28.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.101+0000 7f8ef3805700 1 --2- 172.21.15.67:0/327272632 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f8ed806d890 0x7f8ed806fd50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8ee800b5c0 tx=0x7f8ee80058e0 comp rx=0 tx=0).stop 2024-04-22T23:24:28.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.101+0000 7f8ef3805700 1 -- 172.21.15.67:0/327272632 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8eec1084c0 msgr2=0x7f8eec1a5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:24:28.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.101+0000 7f8ef3805700 1 --2- 172.21.15.67:0/327272632 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8eec1084c0 0x7f8eec1a5a40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8edc00d8d0 tx=0x7f8edc00dc90 comp rx=0 tx=0).stop 2024-04-22T23:24:28.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.101+0000 7f8ef3805700 1 -- 172.21.15.67:0/327272632 shutdown_connections 2024-04-22T23:24:28.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.101+0000 7f8ef3805700 1 --2- 172.21.15.67:0/327272632 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f8ed806d890 0x7f8ed806fd50 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:24:28.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.101+0000 7f8ef3805700 1 --2- 172.21.15.67:0/327272632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8eec1072c0 0x7f8eec1a5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:24:28.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.101+0000 7f8ef3805700 1 --2- 172.21.15.67:0/327272632 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8eec1084c0 0x7f8eec1a5a40 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8edc00d8d0 tx=0x7f8edc00dc90 comp rx=0 tx=0).stop 2024-04-22T23:24:28.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.101+0000 7f8ef3805700 1 -- 172.21.15.67:0/327272632 >> 172.21.15.67:0/327272632 conn(0x7f8eec0763d0 msgr2=0x7f8eec10b6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:24:28.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.101+0000 7f8ef3805700 1 -- 172.21.15.67:0/327272632 shutdown_connections 2024-04-22T23:24:28.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.102+0000 7f8ef3805700 1 -- 172.21.15.67:0/327272632 wait complete. 2024-04-22T23:24:28.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.237+0000 7f8c4944d700 1 -- 172.21.15.67:0/2431404095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c440ffc10 msgr2=0x7f8c44100030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:24:28.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.237+0000 7f8c4944d700 1 --2- 172.21.15.67:0/2431404095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c440ffc10 0x7f8c44100030 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8c34009910 tx=0x7f8c34009c20 comp rx=0 tx=0).stop 2024-04-22T23:24:28.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.237+0000 7f8c4944d700 1 -- 172.21.15.67:0/2431404095 shutdown_connections 2024-04-22T23:24:28.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.237+0000 7f8c4944d700 1 --2- 172.21.15.67:0/2431404095 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c44100e10 0x7f8c44101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:24:28.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.237+0000 7f8c4944d700 1 --2- 172.21.15.67:0/2431404095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c440ffc10 0x7f8c44100030 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:24:28.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.237+0000 7f8c4944d700 1 -- 172.21.15.67:0/2431404095 >> 172.21.15.67:0/2431404095 conn(0x7f8c440fb190 msgr2=0x7f8c440fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:24:28.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.238+0000 7f8c4944d700 1 -- 172.21.15.67:0/2431404095 shutdown_connections 2024-04-22T23:24:28.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.238+0000 7f8c4944d700 1 -- 172.21.15.67:0/2431404095 wait complete. 2024-04-22T23:24:28.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.238+0000 7f8c4944d700 1 Processor -- start 2024-04-22T23:24:28.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.238+0000 7f8c4944d700 1 -- start start 2024-04-22T23:24:28.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.239+0000 7f8c4944d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c440ffc10 0x7f8c441986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:24:28.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.239+0000 7f8c4944d700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c44100e10 0x7f8c44198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:24:28.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.239+0000 7f8c4944d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c44199230 con 0x7f8c440ffc10 2024-04-22T23:24:28.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.239+0000 7f8c4944d700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c44199370 con 0x7f8c44100e10 2024-04-22T23:24:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.239+0000 7f8c42ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c440ffc10 0x7f8c441986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:24:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.239+0000 7f8c42ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c440ffc10 0x7f8c441986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57502/0 (socket says 172.21.15.67:57502) 2024-04-22T23:24:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.239+0000 7f8c42ffd700 1 -- 172.21.15.67:0/70319261 learned_addr learned my addr 172.21.15.67:0/70319261 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:24:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.239+0000 7f8c427fc700 1 --2- 172.21.15.67:0/70319261 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c44100e10 0x7f8c44198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:24:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.239+0000 7f8c42ffd700 1 -- 172.21.15.67:0/70319261 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c44100e10 msgr2=0x7f8c44198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:24:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.239+0000 7f8c42ffd700 1 --2- 172.21.15.67:0/70319261 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c44100e10 0x7f8c44198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:24:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.239+0000 7f8c42ffd700 1 -- 172.21.15.67:0/70319261 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c340095f0 con 0x7f8c440ffc10 2024-04-22T23:24:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.239+0000 7f8c42ffd700 1 --2- 172.21.15.67:0/70319261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c440ffc10 0x7f8c441986d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f8c34006010 tx=0x7f8c34004900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:24:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.239+0000 7f8c2bfff700 1 -- 172.21.15.67:0/70319261 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c34004f80 con 0x7f8c440ffc10 2024-04-22T23:24:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.240+0000 7f8c4944d700 1 -- 172.21.15.67:0/70319261 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c4419ddc0 con 0x7f8c440ffc10 2024-04-22T23:24:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.240+0000 7f8c2bfff700 1 -- 172.21.15.67:0/70319261 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8c340044d0 con 0x7f8c440ffc10 2024-04-22T23:24:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.240+0000 7f8c2bfff700 1 -- 172.21.15.67:0/70319261 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c3401d410 con 0x7f8c440ffc10 2024-04-22T23:24:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.240+0000 7f8c4944d700 1 -- 172.21.15.67:0/70319261 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c4419e290 con 0x7f8c440ffc10 2024-04-22T23:24:28.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.241+0000 7f8c4944d700 1 -- 172.21.15.67:0/70319261 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c4410ba00 con 0x7f8c440ffc10 2024-04-22T23:24:28.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.241+0000 7f8c2bfff700 1 -- 172.21.15.67:0/70319261 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 25) v1 ==== 91559+0+0 (secure 0 0 0) 0x7f8c34003810 con 0x7f8c440ffc10 2024-04-22T23:24:28.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.242+0000 7f8c2bfff700 1 --2- 172.21.15.67:0/70319261 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f8c2c06d840 0x7f8c2c06fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:24:28.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.242+0000 7f8c2bfff700 1 -- 172.21.15.67:0/70319261 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f8c3408edd0 con 0x7f8c440ffc10 2024-04-22T23:24:28.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.242+0000 7f8c427fc700 1 --2- 172.21.15.67:0/70319261 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f8c2c06d840 0x7f8c2c06fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:24:28.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.243+0000 7f8c427fc700 1 --2- 172.21.15.67:0/70319261 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f8c2c06d840 0x7f8c2c06fd00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8c38009c60 tx=0x7f8c38009260 comp rx=0 tx=0).ready entity=mgr.14622 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:24:28.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.248+0000 7f8c2bfff700 1 -- 172.21.15.67:0/70319261 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f8c3405bac0 con 0x7f8c440ffc10 2024-04-22T23:24:28.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.732+0000 7f8c4944d700 1 -- 172.21.15.67:0/70319261 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8c4419e990 con 0x7f8c440ffc10 2024-04-22T23:24:28.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.733+0000 7f8c2bfff700 1 -- 172.21.15.67:0/70319261 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+766 (secure 0 0 0) 0x7f8c3405b650 con 0x7f8c440ffc10 2024-04-22T23:24:28.735 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:24:28.735 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:24:28.735 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-22T23:24:28.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:24:28.735 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:24:28.735 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:24:28.735 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 1 2024-04-22T23:24:28.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:24:28.736 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:24:28.736 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-22T23:24:28.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:24:28.736 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:24:28.736 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-22T23:24:28.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:24:28.736 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:24:28.736 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 13, 2024-04-22T23:24:28.736 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 1 2024-04-22T23:24:28.736 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:24:28.736 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:24:28.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.736+0000 7f8c4944d700 1 -- 172.21.15.67:0/70319261 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f8c2c06d840 msgr2=0x7f8c2c06fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:24:28.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.737+0000 7f8c4944d700 1 --2- 172.21.15.67:0/70319261 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f8c2c06d840 0x7f8c2c06fd00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8c38009c60 tx=0x7f8c38009260 comp rx=0 tx=0).stop 2024-04-22T23:24:28.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.737+0000 7f8c4944d700 1 -- 172.21.15.67:0/70319261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c440ffc10 msgr2=0x7f8c441986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:24:28.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.737+0000 7f8c4944d700 1 --2- 172.21.15.67:0/70319261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c440ffc10 0x7f8c441986d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f8c34006010 tx=0x7f8c34004900 comp rx=0 tx=0).stop 2024-04-22T23:24:28.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.737+0000 7f8c4944d700 1 -- 172.21.15.67:0/70319261 shutdown_connections 2024-04-22T23:24:28.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.737+0000 7f8c4944d700 1 --2- 172.21.15.67:0/70319261 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f8c2c06d840 0x7f8c2c06fd00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:24:28.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.737+0000 7f8c4944d700 1 --2- 172.21.15.67:0/70319261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c440ffc10 0x7f8c441986d0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:24:28.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.737+0000 7f8c4944d700 1 --2- 172.21.15.67:0/70319261 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c44100e10 0x7f8c44198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:24:28.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.737+0000 7f8c4944d700 1 -- 172.21.15.67:0/70319261 >> 172.21.15.67:0/70319261 conn(0x7f8c440fb190 msgr2=0x7f8c44104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:24:28.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.737+0000 7f8c4944d700 1 -- 172.21.15.67:0/70319261 shutdown_connections 2024-04-22T23:24:28.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.737+0000 7f8c4944d700 1 -- 172.21.15.67:0/70319261 wait complete. 2024-04-22T23:24:28.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:28 smithi067 ceph-mon[26453]: pgmap v18: 65 pgs: 65 active+clean; 298 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 148 op/s 2024-04-22T23:24:28.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:28 smithi067 ceph-mon[26453]: from='client.14690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:24:28.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:28 smithi067 ceph-mon[26453]: from='client.14694 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:24:28.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.913+0000 7f9ae32d3700 1 -- 172.21.15.67:0/3730675565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9adc100ba0 msgr2=0x7f9adc101000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:24:28.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.913+0000 7f9ae32d3700 1 --2- 172.21.15.67:0/3730675565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9adc100ba0 0x7f9adc101000 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f9acc009910 tx=0x7f9acc009c20 comp rx=0 tx=0).stop 2024-04-22T23:24:28.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.914+0000 7f9ae32d3700 1 -- 172.21.15.67:0/3730675565 shutdown_connections 2024-04-22T23:24:28.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.914+0000 7f9ae32d3700 1 --2- 172.21.15.67:0/3730675565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9adc100ba0 0x7f9adc101000 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:24:28.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.914+0000 7f9ae32d3700 1 --2- 172.21.15.67:0/3730675565 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9adc0ff9a0 0x7f9adc0ffdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:24:28.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.914+0000 7f9ae32d3700 1 -- 172.21.15.67:0/3730675565 >> 172.21.15.67:0/3730675565 conn(0x7f9adc0faf80 msgr2=0x7f9adc0fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:24:28.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.915+0000 7f9ae32d3700 1 -- 172.21.15.67:0/3730675565 shutdown_connections 2024-04-22T23:24:28.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.915+0000 7f9ae32d3700 1 -- 172.21.15.67:0/3730675565 wait complete. 2024-04-22T23:24:28.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.915+0000 7f9ae32d3700 1 Processor -- start 2024-04-22T23:24:28.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.915+0000 7f9ae32d3700 1 -- start start 2024-04-22T23:24:28.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.916+0000 7f9ae32d3700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9adc0ff9a0 0x7f9adc195170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:24:28.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.916+0000 7f9ae32d3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9adc100ba0 0x7f9adc1956b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:24:28.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.916+0000 7f9ae32d3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9adc195cd0 con 0x7f9adc100ba0 2024-04-22T23:24:28.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.916+0000 7f9ae32d3700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9adc195e10 con 0x7f9adc0ff9a0 2024-04-22T23:24:28.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.916+0000 7f9ae1ad0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9adc100ba0 0x7f9adc1956b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:24:28.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.916+0000 7f9ae1ad0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9adc100ba0 0x7f9adc1956b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44798/0 (socket says 172.21.15.67:44798) 2024-04-22T23:24:28.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.916+0000 7f9ae1ad0700 1 -- 172.21.15.67:0/676596072 learned_addr learned my addr 172.21.15.67:0/676596072 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:24:28.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.917+0000 7f9ae22d1700 1 --2- 172.21.15.67:0/676596072 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9adc0ff9a0 0x7f9adc195170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:24:28.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.917+0000 7f9ae1ad0700 1 -- 172.21.15.67:0/676596072 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9adc0ff9a0 msgr2=0x7f9adc195170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:24:28.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.917+0000 7f9ae1ad0700 1 --2- 172.21.15.67:0/676596072 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9adc0ff9a0 0x7f9adc195170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:24:28.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.917+0000 7f9ae1ad0700 1 -- 172.21.15.67:0/676596072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9acc0095f0 con 0x7f9adc100ba0 2024-04-22T23:24:28.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.917+0000 7f9ae22d1700 1 --2- 172.21.15.67:0/676596072 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9adc0ff9a0 0x7f9adc195170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T23:24:28.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.918+0000 7f9ae1ad0700 1 --2- 172.21.15.67:0/676596072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9adc100ba0 0x7f9adc1956b0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9acc000c00 tx=0x7f9acc00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:24:28.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.918+0000 7f9ad37fe700 1 -- 172.21.15.67:0/676596072 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9acc01d070 con 0x7f9adc100ba0 2024-04-22T23:24:28.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.918+0000 7f9ad37fe700 1 -- 172.21.15.67:0/676596072 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f9acc005dc0 con 0x7f9adc100ba0 2024-04-22T23:24:28.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.919+0000 7f9ad37fe700 1 -- 172.21.15.67:0/676596072 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9acc0226a0 con 0x7f9adc100ba0 2024-04-22T23:24:28.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.919+0000 7f9ae32d3700 1 -- 172.21.15.67:0/676596072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9adc068f50 con 0x7f9adc100ba0 2024-04-22T23:24:28.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.919+0000 7f9ae32d3700 1 -- 172.21.15.67:0/676596072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9adc069440 con 0x7f9adc100ba0 2024-04-22T23:24:28.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.921+0000 7f9ae32d3700 1 -- 172.21.15.67:0/676596072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9adc108520 con 0x7f9adc100ba0 2024-04-22T23:24:28.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.922+0000 7f9ad37fe700 1 -- 172.21.15.67:0/676596072 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 25) v1 ==== 91559+0+0 (secure 0 0 0) 0x7f9acc0038f0 con 0x7f9adc100ba0 2024-04-22T23:24:28.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.922+0000 7f9ad37fe700 1 --2- 172.21.15.67:0/676596072 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f9ac806d950 0x7f9ac806fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:24:28.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.922+0000 7f9ad37fe700 1 -- 172.21.15.67:0/676596072 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f9acc08e780 con 0x7f9adc100ba0 2024-04-22T23:24:28.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.925+0000 7f9ae22d1700 1 --2- 172.21.15.67:0/676596072 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f9ac806d950 0x7f9ac806fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:24:28.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.926+0000 7f9ae22d1700 1 --2- 172.21.15.67:0/676596072 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f9ac806d950 0x7f9ac806fe10 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9adc100a00 tx=0x7f9ad8009260 comp rx=0 tx=0).ready entity=mgr.14622 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:24:28.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:28.930+0000 7f9ad37fe700 1 -- 172.21.15.67:0/676596072 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f9acc05b4d0 con 0x7f9adc100ba0 2024-04-22T23:24:28.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:28 smithi134 ceph-mon[34609]: pgmap v18: 65 pgs: 65 active+clean; 298 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 148 op/s 2024-04-22T23:24:28.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:28 smithi134 ceph-mon[34609]: from='client.14690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:24:28.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:28 smithi134 ceph-mon[34609]: from='client.14694 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:24:29.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:29.255+0000 7f9ae32d3700 1 -- 172.21.15.67:0/676596072 --> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9adc1054f0 con 0x7f9ac806d950 2024-04-22T23:24:29.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:29.256+0000 7f9ad37fe700 1 -- 172.21.15.67:0/676596072 <== mgr.14622 v2:172.21.15.134:6828/1077503272 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7f9adc1054f0 con 0x7f9ac806d950 2024-04-22T23:24:29.258 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:24:29.258 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:24:29.258 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:24:29.258 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-22T23:24:29.258 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [], 2024-04-22T23:24:29.258 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "1/2 daemons upgraded", 2024-04-22T23:24:29.258 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mgr daemons", 2024-04-22T23:24:29.258 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:24:29.259 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:24:29.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:29.259+0000 7f9ae32d3700 1 -- 172.21.15.67:0/676596072 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f9ac806d950 msgr2=0x7f9ac806fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:24:29.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:29.259+0000 7f9ae32d3700 1 --2- 172.21.15.67:0/676596072 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f9ac806d950 0x7f9ac806fe10 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9adc100a00 tx=0x7f9ad8009260 comp rx=0 tx=0).stop 2024-04-22T23:24:29.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:29.259+0000 7f9ae32d3700 1 -- 172.21.15.67:0/676596072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9adc100ba0 msgr2=0x7f9adc1956b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:24:29.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:29.259+0000 7f9ae32d3700 1 --2- 172.21.15.67:0/676596072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9adc100ba0 0x7f9adc1956b0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9acc000c00 tx=0x7f9acc00f690 comp rx=0 tx=0).stop 2024-04-22T23:24:29.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:29.259+0000 7f9ae32d3700 1 -- 172.21.15.67:0/676596072 shutdown_connections 2024-04-22T23:24:29.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:29.259+0000 7f9ae32d3700 1 --2- 172.21.15.67:0/676596072 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f9ac806d950 0x7f9ac806fe10 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:24:29.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:29.259+0000 7f9ae32d3700 1 --2- 172.21.15.67:0/676596072 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9adc0ff9a0 0x7f9adc195170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:24:29.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:29.259+0000 7f9ae32d3700 1 --2- 172.21.15.67:0/676596072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9adc100ba0 0x7f9adc1956b0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:24:29.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:29.259+0000 7f9ae32d3700 1 -- 172.21.15.67:0/676596072 >> 172.21.15.67:0/676596072 conn(0x7f9adc0faf80 msgr2=0x7f9adc103dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:24:29.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:29.259+0000 7f9ae32d3700 1 -- 172.21.15.67:0/676596072 shutdown_connections 2024-04-22T23:24:29.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:29.259+0000 7f9ae32d3700 1 -- 172.21.15.67:0/676596072 wait complete. 2024-04-22T23:24:29.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:29 smithi067 ceph-mon[26453]: from='client.24443 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:24:29.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:29 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:29.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:29 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:29.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:29 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/70319261' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:24:29.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:29 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T23:24:29.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:29 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:24:29.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:29 smithi134 ceph-mon[34609]: from='client.24443 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:24:29.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:29 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:29.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:29 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:29.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:29 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/70319261' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:24:29.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:29 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T23:24:29.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:29 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:24:30.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:30 smithi067 ceph-mon[26453]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T23:24:30.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:30 smithi067 ceph-mon[26453]: Upgrade: Need to upgrade myself (mgr.smithi134.ccxkcj) 2024-04-22T23:24:30.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:30 smithi067 ceph-mon[26453]: Upgrade: Need to upgrade myself (mgr.smithi134.ccxkcj) 2024-04-22T23:24:30.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:30 smithi067 ceph-mon[26453]: pgmap v19: 65 pgs: 65 active+clean; 302 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 3.0 MiB/s rd, 3.1 MiB/s wr, 226 op/s 2024-04-22T23:24:30.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:30 smithi067 ceph-mon[26453]: from='client.14704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:24:30.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:30 smithi134 ceph-mon[34609]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T23:24:30.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:30 smithi134 ceph-mon[34609]: Upgrade: Need to upgrade myself (mgr.smithi134.ccxkcj) 2024-04-22T23:24:30.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:30 smithi134 ceph-mon[34609]: Upgrade: Need to upgrade myself (mgr.smithi134.ccxkcj) 2024-04-22T23:24:30.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:30 smithi134 ceph-mon[34609]: pgmap v19: 65 pgs: 65 active+clean; 302 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 3.0 MiB/s rd, 3.1 MiB/s wr, 226 op/s 2024-04-22T23:24:30.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:30 smithi134 ceph-mon[34609]: from='client.14704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:24:32.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:31 smithi067 ceph-mon[26453]: pgmap v20: 65 pgs: 65 active+clean; 302 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 160 op/s 2024-04-22T23:24:32.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:31 smithi067 ceph-mon[26453]: Upgrade: Updating mgr.smithi067.whdcsr 2024-04-22T23:24:32.046 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:31 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:32.046 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:31 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi067.whdcsr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T23:24:32.046 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:31 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi067.whdcsr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T23:24:32.046 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:31 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T23:24:32.046 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:31 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:24:32.046 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:31 smithi067 ceph-mon[26453]: Deploying daemon mgr.smithi067.whdcsr on smithi067 2024-04-22T23:24:32.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:31 smithi134 ceph-mon[34609]: pgmap v20: 65 pgs: 65 active+clean; 302 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 160 op/s 2024-04-22T23:24:32.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:31 smithi134 ceph-mon[34609]: Upgrade: Updating mgr.smithi067.whdcsr 2024-04-22T23:24:32.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:31 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:32.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:31 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi067.whdcsr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T23:24:32.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:31 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi067.whdcsr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T23:24:32.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:31 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T23:24:32.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:31 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:24:32.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:31 smithi134 ceph-mon[34609]: Deploying daemon mgr.smithi067.whdcsr on smithi067 2024-04-22T23:24:34.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:33 smithi134 ceph-mon[34609]: pgmap v21: 65 pgs: 65 active+clean; 309 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 198 op/s 2024-04-22T23:24:34.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:33 smithi067 ceph-mon[26453]: pgmap v21: 65 pgs: 65 active+clean; 309 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 198 op/s 2024-04-22T23:24:36.633 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:36 smithi067 ceph-mon[26453]: pgmap v22: 65 pgs: 65 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 235 op/s 2024-04-22T23:24:36.682 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:36 smithi134 ceph-mon[34609]: pgmap v22: 65 pgs: 65 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 235 op/s 2024-04-22T23:24:38.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:38 smithi134 ceph-mon[34609]: pgmap v23: 65 pgs: 65 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 169 op/s 2024-04-22T23:24:38.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:38 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:38.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:38 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:38.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:38 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:24:38.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:38 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:38.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:38 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:38.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:38 smithi067 ceph-mon[26453]: pgmap v23: 65 pgs: 65 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 169 op/s 2024-04-22T23:24:38.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:38 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:38.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:38 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:38.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:38 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:24:38.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:38 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:38.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:38 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:40.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:40 smithi067 ceph-mon[26453]: pgmap v24: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 3.6 MiB/s rd, 3.5 MiB/s wr, 246 op/s 2024-04-22T23:24:40.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:40 smithi134 ceph-mon[34609]: pgmap v24: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 3.6 MiB/s rd, 3.5 MiB/s wr, 246 op/s 2024-04-22T23:24:42.592 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:42 smithi067 ceph-mon[26453]: pgmap v25: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 168 op/s 2024-04-22T23:24:42.592 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:42 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:24:42.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:42 smithi134 ceph-mon[34609]: pgmap v25: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 168 op/s 2024-04-22T23:24:42.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:42 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:24:44.480 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:44 smithi067 ceph-mon[26453]: pgmap v26: 65 pgs: 65 active+clean; 319 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 198 op/s 2024-04-22T23:24:44.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:44 smithi134 ceph-mon[34609]: pgmap v26: 65 pgs: 65 active+clean; 319 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 198 op/s 2024-04-22T23:24:46.398 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:46 smithi067 ceph-mon[26453]: pgmap v27: 65 pgs: 65 active+clean; 314 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 209 op/s 2024-04-22T23:24:46.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:46 smithi134 ceph-mon[34609]: pgmap v27: 65 pgs: 65 active+clean; 314 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 209 op/s 2024-04-22T23:24:47.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:47 smithi134 ceph-mon[34609]: Standby manager daemon smithi067.whdcsr restarted 2024-04-22T23:24:47.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:47 smithi134 ceph-mon[34609]: Standby manager daemon smithi067.whdcsr started 2024-04-22T23:24:47.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:47 smithi134 ceph-mon[34609]: from='mgr.? 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi067.whdcsr/crt"}]: dispatch 2024-04-22T23:24:47.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:47 smithi134 ceph-mon[34609]: from='mgr.? 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T23:24:47.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:47 smithi134 ceph-mon[34609]: from='mgr.? 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi067.whdcsr/key"}]: dispatch 2024-04-22T23:24:47.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:47 smithi134 ceph-mon[34609]: from='mgr.? 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T23:24:47.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:47 smithi067 ceph-mon[26453]: Standby manager daemon smithi067.whdcsr restarted 2024-04-22T23:24:47.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:47 smithi067 ceph-mon[26453]: Standby manager daemon smithi067.whdcsr started 2024-04-22T23:24:47.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:47 smithi067 ceph-mon[26453]: from='mgr.? 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi067.whdcsr/crt"}]: dispatch 2024-04-22T23:24:47.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:47 smithi067 ceph-mon[26453]: from='mgr.? 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T23:24:47.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:47 smithi067 ceph-mon[26453]: from='mgr.? 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi067.whdcsr/key"}]: dispatch 2024-04-22T23:24:47.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:47 smithi067 ceph-mon[26453]: from='mgr.? 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T23:24:48.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:48 smithi134 ceph-mon[34609]: pgmap v28: 65 pgs: 65 active+clean; 314 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 156 op/s 2024-04-22T23:24:48.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:48 smithi134 ceph-mon[34609]: mgrmap e26: smithi134.ccxkcj(active, since 50s), standbys: smithi067.whdcsr 2024-04-22T23:24:48.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:48 smithi067 ceph-mon[26453]: pgmap v28: 65 pgs: 65 active+clean; 314 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 156 op/s 2024-04-22T23:24:48.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:48 smithi067 ceph-mon[26453]: mgrmap e26: smithi134.ccxkcj(active, since 50s), standbys: smithi067.whdcsr 2024-04-22T23:24:50.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:50 smithi067 ceph-mon[26453]: pgmap v29: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 250 op/s 2024-04-22T23:24:50.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:50 smithi134 ceph-mon[34609]: pgmap v29: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 250 op/s 2024-04-22T23:24:52.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:52 smithi067 ceph-mon[26453]: pgmap v30: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.9 MiB/s rd, 1.8 MiB/s wr, 173 op/s 2024-04-22T23:24:52.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:52 smithi134 ceph-mon[34609]: pgmap v30: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.9 MiB/s rd, 1.8 MiB/s wr, 173 op/s 2024-04-22T23:24:54.387 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-04-22T23:24:54.387 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-22T23:24:54.709 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:54 smithi067 ceph-mon[26453]: pgmap v31: 65 pgs: 65 active+clean; 314 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 209 op/s 2024-04-22T23:24:54.709 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:54 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:54.709 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:54 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:54.710 DEBUG:teuthology.parallel:result is None 2024-04-22T23:24:54.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:54 smithi134 ceph-mon[34609]: pgmap v31: 65 pgs: 65 active+clean; 314 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 209 op/s 2024-04-22T23:24:54.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:54 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:54.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:54 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:56.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:56 smithi134 ceph-mon[34609]: pgmap v32: 65 pgs: 65 active+clean; 320 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 264 op/s 2024-04-22T23:24:56.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:56 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:56.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:56 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:56.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:56 smithi067 ceph-mon[26453]: pgmap v32: 65 pgs: 65 active+clean; 320 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 264 op/s 2024-04-22T23:24:56.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:56 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:56.519 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:56 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:24:57.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:57 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:24:57.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:57 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:24:58.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:24:58 smithi134 ceph-mon[34609]: pgmap v33: 65 pgs: 65 active+clean; 320 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 215 op/s 2024-04-22T23:24:58.500 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:24:58 smithi067 ceph-mon[26453]: pgmap v33: 65 pgs: 65 active+clean; 320 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 215 op/s 2024-04-22T23:24:59.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.384+0000 7f5e6c221700 1 -- 172.21.15.67:0/3428649491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e640ffc10 msgr2=0x7f5e64100030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:24:59.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.384+0000 7f5e6c221700 1 --2- 172.21.15.67:0/3428649491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e640ffc10 0x7f5e64100030 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c00b0a0 tx=0x7f5e5c00b3b0 comp rx=0 tx=0).stop 2024-04-22T23:24:59.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.384+0000 7f5e6c221700 1 -- 172.21.15.67:0/3428649491 shutdown_connections 2024-04-22T23:24:59.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.384+0000 7f5e6c221700 1 --2- 172.21.15.67:0/3428649491 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e64100e10 0x7f5e64101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:24:59.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.384+0000 7f5e6c221700 1 --2- 172.21.15.67:0/3428649491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e640ffc10 0x7f5e64100030 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:24:59.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.384+0000 7f5e6c221700 1 -- 172.21.15.67:0/3428649491 >> 172.21.15.67:0/3428649491 conn(0x7f5e640fb190 msgr2=0x7f5e640fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:24:59.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.385+0000 7f5e6c221700 1 -- 172.21.15.67:0/3428649491 shutdown_connections 2024-04-22T23:24:59.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.385+0000 7f5e6c221700 1 -- 172.21.15.67:0/3428649491 wait complete. 2024-04-22T23:24:59.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.385+0000 7f5e6c221700 1 Processor -- start 2024-04-22T23:24:59.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.385+0000 7f5e6c221700 1 -- start start 2024-04-22T23:24:59.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.385+0000 7f5e6c221700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e640ffc10 0x7f5e64103ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:24:59.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.385+0000 7f5e6c221700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e64100e10 0x7f5e641021f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:24:59.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.385+0000 7f5e6c221700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e64102730 con 0x7f5e64100e10 2024-04-22T23:24:59.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.385+0000 7f5e6c221700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e64102870 con 0x7f5e640ffc10 2024-04-22T23:24:59.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.385+0000 7f5e697bc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e64100e10 0x7f5e641021f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:24:59.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.386+0000 7f5e69fbd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e640ffc10 0x7f5e64103ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:24:59.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.386+0000 7f5e697bc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e64100e10 0x7f5e641021f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54722/0 (socket says 172.21.15.67:54722) 2024-04-22T23:24:59.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.386+0000 7f5e697bc700 1 -- 172.21.15.67:0/2831393602 learned_addr learned my addr 172.21.15.67:0/2831393602 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:24:59.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.386+0000 7f5e69fbd700 1 -- 172.21.15.67:0/2831393602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e64100e10 msgr2=0x7f5e641021f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:24:59.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.386+0000 7f5e69fbd700 1 --2- 172.21.15.67:0/2831393602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e64100e10 0x7f5e641021f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:24:59.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.386+0000 7f5e69fbd700 1 -- 172.21.15.67:0/2831393602 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5c009d00 con 0x7f5e640ffc10 2024-04-22T23:24:59.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.386+0000 7f5e697bc700 1 --2- 172.21.15.67:0/2831393602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e64100e10 0x7f5e641021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T23:24:59.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.386+0000 7f5e69fbd700 1 --2- 172.21.15.67:0/2831393602 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e640ffc10 0x7f5e64103ba0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c0077b0 tx=0x7f5e5c0096e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:24:59.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.386+0000 7f5e56ffd700 1 -- 172.21.15.67:0/2831393602 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e5c021070 con 0x7f5e640ffc10 2024-04-22T23:24:59.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.386+0000 7f5e56ffd700 1 -- 172.21.15.67:0/2831393602 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f5e5c003de0 con 0x7f5e640ffc10 2024-04-22T23:24:59.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.386+0000 7f5e6c221700 1 -- 172.21.15.67:0/2831393602 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e64102af0 con 0x7f5e640ffc10 2024-04-22T23:24:59.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.386+0000 7f5e56ffd700 1 -- 172.21.15.67:0/2831393602 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e5c00e040 con 0x7f5e640ffc10 2024-04-22T23:24:59.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.387+0000 7f5e6c221700 1 -- 172.21.15.67:0/2831393602 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e64102fb0 con 0x7f5e640ffc10 2024-04-22T23:24:59.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.387+0000 7f5e6c221700 1 -- 172.21.15.67:0/2831393602 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e64108700 con 0x7f5e640ffc10 2024-04-22T23:24:59.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.388+0000 7f5e56ffd700 1 -- 172.21.15.67:0/2831393602 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 26) v1 ==== 92702+0+0 (secure 0 0 0) 0x7f5e5c004040 con 0x7f5e640ffc10 2024-04-22T23:24:59.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.389+0000 7f5e56ffd700 1 --2- 172.21.15.67:0/2831393602 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f5e5006ea80 0x7f5e50070f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:24:59.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.389+0000 7f5e56ffd700 1 -- 172.21.15.67:0/2831393602 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f5e5c08fbc0 con 0x7f5e640ffc10 2024-04-22T23:24:59.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.389+0000 7f5e697bc700 1 --2- 172.21.15.67:0/2831393602 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f5e5006ea80 0x7f5e50070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:24:59.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.390+0000 7f5e697bc700 1 --2- 172.21.15.67:0/2831393602 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f5e5006ea80 0x7f5e50070f40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5e580104e0 tx=0x7f5e58007480 comp rx=0 tx=0).ready entity=mgr.14622 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:24:59.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.394+0000 7f5e56ffd700 1 -- 172.21.15.67:0/2831393602 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f5e5c05c890 con 0x7f5e640ffc10 2024-04-22T23:24:59.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.648+0000 7f5e6c221700 1 -- 172.21.15.67:0/2831393602 --> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5e6419ea10 con 0x7f5e5006ea80 2024-04-22T23:24:59.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.649+0000 7f5e56ffd700 1 -- 172.21.15.67:0/2831393602 <== mgr.14622 v2:172.21.15.134:6828/1077503272 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f5e6419ea10 con 0x7f5e5006ea80 2024-04-22T23:24:59.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.653+0000 7f5e6c221700 1 -- 172.21.15.67:0/2831393602 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f5e5006ea80 msgr2=0x7f5e50070f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:24:59.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.653+0000 7f5e6c221700 1 --2- 172.21.15.67:0/2831393602 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f5e5006ea80 0x7f5e50070f40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5e580104e0 tx=0x7f5e58007480 comp rx=0 tx=0).stop 2024-04-22T23:24:59.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.653+0000 7f5e6c221700 1 -- 172.21.15.67:0/2831393602 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e640ffc10 msgr2=0x7f5e64103ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:24:59.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.653+0000 7f5e6c221700 1 --2- 172.21.15.67:0/2831393602 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e640ffc10 0x7f5e64103ba0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c0077b0 tx=0x7f5e5c0096e0 comp rx=0 tx=0).stop 2024-04-22T23:24:59.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.653+0000 7f5e6c221700 1 -- 172.21.15.67:0/2831393602 shutdown_connections 2024-04-22T23:24:59.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.653+0000 7f5e6c221700 1 --2- 172.21.15.67:0/2831393602 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f5e5006ea80 0x7f5e50070f40 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:24:59.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.653+0000 7f5e6c221700 1 --2- 172.21.15.67:0/2831393602 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e640ffc10 0x7f5e64103ba0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:24:59.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.653+0000 7f5e6c221700 1 --2- 172.21.15.67:0/2831393602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e64100e10 0x7f5e641021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:24:59.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.653+0000 7f5e6c221700 1 -- 172.21.15.67:0/2831393602 >> 172.21.15.67:0/2831393602 conn(0x7f5e640fb190 msgr2=0x7f5e64068580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:24:59.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.653+0000 7f5e6c221700 1 -- 172.21.15.67:0/2831393602 shutdown_connections 2024-04-22T23:24:59.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.653+0000 7f5e6c221700 1 -- 172.21.15.67:0/2831393602 wait complete. 2024-04-22T23:24:59.666 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:24:59.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.773+0000 7f8470222700 1 -- 172.21.15.67:0/325939267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84681017a0 msgr2=0x7f8468103b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:24:59.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.773+0000 7f8470222700 1 --2- 172.21.15.67:0/325939267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84681017a0 0x7f8468103b90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f8464009910 tx=0x7f8464009c20 comp rx=0 tx=0).stop 2024-04-22T23:24:59.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.773+0000 7f8470222700 1 -- 172.21.15.67:0/325939267 shutdown_connections 2024-04-22T23:24:59.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.773+0000 7f8470222700 1 --2- 172.21.15.67:0/325939267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84681040d0 0x7f84681064c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:24:59.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.773+0000 7f8470222700 1 --2- 172.21.15.67:0/325939267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84681017a0 0x7f8468103b90 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:24:59.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.773+0000 7f8470222700 1 -- 172.21.15.67:0/325939267 >> 172.21.15.67:0/325939267 conn(0x7f84680fb190 msgr2=0x7f84680fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:24:59.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.774+0000 7f8470222700 1 -- 172.21.15.67:0/325939267 shutdown_connections 2024-04-22T23:24:59.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.774+0000 7f8470222700 1 -- 172.21.15.67:0/325939267 wait complete. 2024-04-22T23:24:59.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.774+0000 7f8470222700 1 Processor -- start 2024-04-22T23:24:59.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.774+0000 7f8470222700 1 -- start start 2024-04-22T23:24:59.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.774+0000 7f8470222700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84681017a0 0x7f8468196460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:24:59.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.774+0000 7f8470222700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84681040d0 0x7f84681969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:24:59.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.774+0000 7f8470222700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8468196fc0 con 0x7f84681017a0 2024-04-22T23:24:59.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.774+0000 7f8470222700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8468197100 con 0x7f84681040d0 2024-04-22T23:24:59.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.775+0000 7f846dfbe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84681017a0 0x7f8468196460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:24:59.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.775+0000 7f846dfbe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84681017a0 0x7f8468196460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54750/0 (socket says 172.21.15.67:54750) 2024-04-22T23:24:59.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.775+0000 7f846dfbe700 1 -- 172.21.15.67:0/2746789321 learned_addr learned my addr 172.21.15.67:0/2746789321 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:24:59.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.775+0000 7f846d7bd700 1 --2- 172.21.15.67:0/2746789321 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84681040d0 0x7f84681969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:24:59.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.775+0000 7f846dfbe700 1 -- 172.21.15.67:0/2746789321 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84681040d0 msgr2=0x7f84681969a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:24:59.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.775+0000 7f846dfbe700 1 --2- 172.21.15.67:0/2746789321 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84681040d0 0x7f84681969a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:24:59.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.775+0000 7f846dfbe700 1 -- 172.21.15.67:0/2746789321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84640095f0 con 0x7f84681017a0 2024-04-22T23:24:59.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.775+0000 7f846d7bd700 1 --2- 172.21.15.67:0/2746789321 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84681040d0 0x7f84681969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T23:24:59.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.775+0000 7f846dfbe700 1 --2- 172.21.15.67:0/2746789321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84681017a0 0x7f8468196460 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f8464000c00 tx=0x7f846400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:24:59.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.775+0000 7f845effd700 1 -- 172.21.15.67:0/2746789321 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f846401d070 con 0x7f84681017a0 2024-04-22T23:24:59.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.775+0000 7f845effd700 1 -- 172.21.15.67:0/2746789321 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8464005dc0 con 0x7f84681017a0 2024-04-22T23:24:59.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.775+0000 7f845effd700 1 -- 172.21.15.67:0/2746789321 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84640226a0 con 0x7f84681017a0 2024-04-22T23:24:59.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.776+0000 7f8470222700 1 -- 172.21.15.67:0/2746789321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84680ff590 con 0x7f84681017a0 2024-04-22T23:24:59.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.776+0000 7f8470222700 1 -- 172.21.15.67:0/2746789321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84680ffa30 con 0x7f84681017a0 2024-04-22T23:24:59.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.777+0000 7f8470222700 1 -- 172.21.15.67:0/2746789321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8468190610 con 0x7f84681017a0 2024-04-22T23:24:59.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.777+0000 7f845effd700 1 -- 172.21.15.67:0/2746789321 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 26) v1 ==== 92702+0+0 (secure 0 0 0) 0x7f84640038f0 con 0x7f84681017a0 2024-04-22T23:24:59.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.778+0000 7f845effd700 1 --2- 172.21.15.67:0/2746789321 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f8454072e80 0x7f8454075340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:24:59.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.778+0000 7f845effd700 1 -- 172.21.15.67:0/2746789321 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f846408f950 con 0x7f84681017a0 2024-04-22T23:24:59.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.778+0000 7f846d7bd700 1 --2- 172.21.15.67:0/2746789321 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f8454072e80 0x7f8454075340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:24:59.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.779+0000 7f846d7bd700 1 --2- 172.21.15.67:0/2746789321 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f8454072e80 0x7f8454075340 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8458011490 tx=0x7f8458009260 comp rx=0 tx=0).ready entity=mgr.14622 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:24:59.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:24:59.784+0000 7f845effd700 1 -- 172.21.15.67:0/2746789321 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f846405c640 con 0x7f84681017a0 2024-04-22T23:25:00.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.036+0000 7f8470222700 1 -- 172.21.15.67:0/2746789321 --> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8468100ec0 con 0x7f8454072e80 2024-04-22T23:25:00.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.038+0000 7f845effd700 1 -- 172.21.15.67:0/2746789321 <== mgr.14622 v2:172.21.15.134:6828/1077503272 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f8468100ec0 con 0x7f8454072e80 2024-04-22T23:25:00.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.040+0000 7f8470222700 1 -- 172.21.15.67:0/2746789321 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f8454072e80 msgr2=0x7f8454075340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:25:00.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.040+0000 7f8470222700 1 --2- 172.21.15.67:0/2746789321 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f8454072e80 0x7f8454075340 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8458011490 tx=0x7f8458009260 comp rx=0 tx=0).stop 2024-04-22T23:25:00.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.040+0000 7f8470222700 1 -- 172.21.15.67:0/2746789321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84681017a0 msgr2=0x7f8468196460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:25:00.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.040+0000 7f8470222700 1 --2- 172.21.15.67:0/2746789321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84681017a0 0x7f8468196460 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f8464000c00 tx=0x7f846400f690 comp rx=0 tx=0).stop 2024-04-22T23:25:00.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.040+0000 7f8470222700 1 -- 172.21.15.67:0/2746789321 shutdown_connections 2024-04-22T23:25:00.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.040+0000 7f8470222700 1 --2- 172.21.15.67:0/2746789321 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7f8454072e80 0x7f8454075340 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:00.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.040+0000 7f8470222700 1 --2- 172.21.15.67:0/2746789321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84681017a0 0x7f8468196460 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:00.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.040+0000 7f8470222700 1 --2- 172.21.15.67:0/2746789321 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f84681040d0 0x7f84681969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:00.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.040+0000 7f8470222700 1 -- 172.21.15.67:0/2746789321 >> 172.21.15.67:0/2746789321 conn(0x7f84680fb190 msgr2=0x7f84680fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:25:00.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.040+0000 7f8470222700 1 -- 172.21.15.67:0/2746789321 shutdown_connections 2024-04-22T23:25:00.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.040+0000 7f8470222700 1 -- 172.21.15.67:0/2746789321 wait complete. 2024-04-22T23:25:00.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.200+0000 7ff6b97c9700 1 -- 172.21.15.67:0/1764128511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6b4101eb0 msgr2=0x7ff6b4102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:25:00.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.200+0000 7ff6b97c9700 1 --2- 172.21.15.67:0/1764128511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6b4101eb0 0x7ff6b4102310 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff6a8009910 tx=0x7ff6a8009c20 comp rx=0 tx=0).stop 2024-04-22T23:25:00.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.200+0000 7ff6b97c9700 1 -- 172.21.15.67:0/1764128511 shutdown_connections 2024-04-22T23:25:00.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.200+0000 7ff6b97c9700 1 --2- 172.21.15.67:0/1764128511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6b4101eb0 0x7ff6b4102310 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:00.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.200+0000 7ff6b97c9700 1 --2- 172.21.15.67:0/1764128511 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff6b4100cb0 0x7ff6b41010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:00.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.200+0000 7ff6b97c9700 1 -- 172.21.15.67:0/1764128511 >> 172.21.15.67:0/1764128511 conn(0x7ff6b40fc230 msgr2=0x7ff6b40fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:25:00.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.200+0000 7ff6b97c9700 1 -- 172.21.15.67:0/1764128511 shutdown_connections 2024-04-22T23:25:00.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.200+0000 7ff6b97c9700 1 -- 172.21.15.67:0/1764128511 wait complete. 2024-04-22T23:25:00.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.200+0000 7ff6b97c9700 1 Processor -- start 2024-04-22T23:25:00.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.201+0000 7ff6b97c9700 1 -- start start 2024-04-22T23:25:00.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.201+0000 7ff6b97c9700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff6b4100cb0 0x7ff6b41964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:25:00.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.201+0000 7ff6b97c9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6b4101eb0 0x7ff6b41969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:25:00.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.201+0000 7ff6b97c9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b4197010 con 0x7ff6b4101eb0 2024-04-22T23:25:00.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.201+0000 7ff6b97c9700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b4197150 con 0x7ff6b4100cb0 2024-04-22T23:25:00.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.201+0000 7ff6b2ffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff6b4100cb0 0x7ff6b41964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:25:00.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.201+0000 7ff6b2ffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff6b4100cb0 0x7ff6b41964b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.67:56278/0 (socket says 172.21.15.67:56278) 2024-04-22T23:25:00.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.201+0000 7ff6b2ffd700 1 -- 172.21.15.67:0/2988322287 learned_addr learned my addr 172.21.15.67:0/2988322287 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:25:00.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.201+0000 7ff6b27fc700 1 --2- 172.21.15.67:0/2988322287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6b4101eb0 0x7ff6b41969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:25:00.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.201+0000 7ff6b2ffd700 1 -- 172.21.15.67:0/2988322287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6b4101eb0 msgr2=0x7ff6b41969f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:25:00.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.201+0000 7ff6b2ffd700 1 --2- 172.21.15.67:0/2988322287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6b4101eb0 0x7ff6b41969f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:00.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.201+0000 7ff6b2ffd700 1 -- 172.21.15.67:0/2988322287 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6a80095f0 con 0x7ff6b4100cb0 2024-04-22T23:25:00.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.202+0000 7ff6b2ffd700 1 --2- 172.21.15.67:0/2988322287 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff6b4100cb0 0x7ff6b41964b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff6a400e8f0 tx=0x7ff6a400ec00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:25:00.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.202+0000 7ff69bfff700 1 -- 172.21.15.67:0/2988322287 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff6a4009790 con 0x7ff6b4100cb0 2024-04-22T23:25:00.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.202+0000 7ff6b97c9700 1 -- 172.21.15.67:0/2988322287 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6b419bc00 con 0x7ff6b4100cb0 2024-04-22T23:25:00.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.202+0000 7ff69bfff700 1 -- 172.21.15.67:0/2988322287 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff6a400c810 con 0x7ff6b4100cb0 2024-04-22T23:25:00.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.202+0000 7ff6b97c9700 1 -- 172.21.15.67:0/2988322287 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6b419c150 con 0x7ff6b4100cb0 2024-04-22T23:25:00.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.202+0000 7ff69bfff700 1 -- 172.21.15.67:0/2988322287 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff6a4003d20 con 0x7ff6b4100cb0 2024-04-22T23:25:00.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.203+0000 7ff6b97c9700 1 -- 172.21.15.67:0/2988322287 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6b4109810 con 0x7ff6b4100cb0 2024-04-22T23:25:00.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.204+0000 7ff69bfff700 1 -- 172.21.15.67:0/2988322287 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 26) v1 ==== 92702+0+0 (secure 0 0 0) 0x7ff6a40041f0 con 0x7ff6b4100cb0 2024-04-22T23:25:00.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.204+0000 7ff69bfff700 1 --2- 172.21.15.67:0/2988322287 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7ff69c06ea80 0x7ff69c070f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:25:00.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.204+0000 7ff69bfff700 1 -- 172.21.15.67:0/2988322287 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5812+0+0 (secure 0 0 0) 0x7ff6a4014070 con 0x7ff6b4100cb0 2024-04-22T23:25:00.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.204+0000 7ff6b27fc700 1 --2- 172.21.15.67:0/2988322287 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7ff69c06ea80 0x7ff69c070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:25:00.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.205+0000 7ff6b27fc700 1 --2- 172.21.15.67:0/2988322287 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7ff69c06ea80 0x7ff69c070f40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff6a8009fd0 tx=0x7ff6a8005fd0 comp rx=0 tx=0).ready entity=mgr.14622 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:25:00.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.209+0000 7ff69bfff700 1 -- 172.21.15.67:0/2988322287 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7ff6a405ba00 con 0x7ff6b4100cb0 2024-04-22T23:25:00.211 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:00 smithi067 ceph-mon[26453]: pgmap v34: 65 pgs: 65 active+clean; 316 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 317 op/s 2024-04-22T23:25:00.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.455+0000 7ff6b97c9700 1 -- 172.21.15.67:0/2988322287 --> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff6b4106800 con 0x7ff69c06ea80 2024-04-22T23:25:00.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.465+0000 7ff69bfff700 1 -- 172.21.15.67:0/2988322287 <== mon.1 v2:172.21.15.134:3300/0 7 ==== mgrmap(e 27) v1 ==== 46220+0+0 (secure 0 0 0) 0x7ff6a408d1c0 con 0x7ff6b4100cb0 2024-04-22T23:25:00.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.465+0000 7ff69bfff700 1 -- 172.21.15.67:0/2988322287 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7ff69c06ea80 msgr2=0x7ff69c070f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:25:00.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:00.465+0000 7ff69bfff700 1 --2- 172.21.15.67:0/2988322287 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7ff69c06ea80 0x7ff69c070f40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff6a8009fd0 tx=0x7ff6a8005fd0 comp rx=0 tx=0).stop 2024-04-22T23:25:00.469 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:00 smithi134 ceph-mon[34609]: pgmap v34: 65 pgs: 65 active+clean; 316 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 317 op/s 2024-04-22T23:25:01.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.465+0000 7ff69bfff700 1 -- 172.21.15.67:0/2988322287 <== mon.1 v2:172.21.15.134:3300/0 8 ==== mgrmap(e 28) v1 ==== 46347+0+0 (secure 0 0 0) 0x7ff6a405bfc0 con 0x7ff6b4100cb0 2024-04-22T23:25:01.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.465+0000 7ff69bfff700 1 --2- 172.21.15.67:0/2988322287 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7ff69c005eb0 0x7ff69c036350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:25:01.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.465+0000 7ff69bfff700 1 -- 172.21.15.67:0/2988322287 --> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff69c032e10 con 0x7ff69c005eb0 2024-04-22T23:25:01.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.474+0000 7ff6b27fc700 1 --2- 172.21.15.67:0/2988322287 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7ff69c005eb0 0x7ff69c036350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:25:01.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.475+0000 7ff6b27fc700 1 --2- 172.21.15.67:0/2988322287 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7ff69c005eb0 0x7ff69c036350 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff6b419bf00 tx=0x7ff6a8023000 comp rx=0 tx=0).ready entity=mgr.14708 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:25:01.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.489+0000 7ff69bfff700 1 -- 172.21.15.67:0/2988322287 <== mgr.14708 v2:172.21.15.67:6800/2306276228 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7ff69c032e10 con 0x7ff69c005eb0 2024-04-22T23:25:01.491 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:25:01.491 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (8m) 7s ago 9m 26.1M - 0.25.0 c8568f914cd2 8c7b7ec00657 2024-04-22T23:25:01.493 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (10m) 7s ago 10m 7889k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:25:01.493 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (9m) 53s ago 9m 7805k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:25:01.493 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (10m) 7s ago 10m 6660k - 18.2.1 5be31c24972a 14085634c5d3 2024-04-22T23:25:01.493 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (9m) 53s ago 9m 6665k - 18.2.1 5be31c24972a fa995d458353 2024-04-22T23:25:01.493 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (8m) 7s ago 9m 87.8M - 9.4.7 954c08fa6188 0d8950c9af0c 2024-04-22T23:25:01.493 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (4m) 7s ago 4m 102M - 18.2.1 5be31c24972a e24a471f0a8c 2024-04-22T23:25:01.493 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (4m) 7s ago 4m 150M - 18.2.1 5be31c24972a 0730f9fc8e61 2024-04-22T23:25:01.493 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (4m) 53s ago 4m 166M - 18.2.1 5be31c24972a b6a169586433 2024-04-22T23:25:01.493 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (4m) 53s ago 4m 148M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:25:01.493 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (24s) 7s ago 11m 100M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:25:01.494 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (109s) 53s ago 9m 488M - 19.0.0-3252-g43be0201 a0434699c1d7 a86aaa7b9552 2024-04-22T23:25:01.494 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (11m) 7s ago 11m 55.7M 2048M 18.2.1 5be31c24972a 6789283703e0 2024-04-22T23:25:01.494 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (8m) 53s ago 8m 39.6M 2048M 18.2.1 5be31c24972a 2fb23795f7e3 2024-04-22T23:25:01.494 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (10m) 7s ago 10m 15.9M - 1.5.0 0da6a335fe13 4aa4da7089d0 2024-04-22T23:25:01.494 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (9m) 53s ago 9m 15.4M - 1.5.0 0da6a335fe13 26e43c2e756c 2024-04-22T23:25:01.494 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (8m) 7s ago 8m 189M 1216M 18.2.1 5be31c24972a a033247ee8c2 2024-04-22T23:25:01.494 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (7m) 7s ago 7m 194M 1216M 18.2.1 5be31c24972a 915b07a8edc4 2024-04-22T23:25:01.494 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (7m) 7s ago 7m 155M 1216M 18.2.1 5be31c24972a e257ecb4efa1 2024-04-22T23:25:01.494 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (6m) 53s ago 6m 234M 2240M 18.2.1 5be31c24972a 9c2c91a967ba 2024-04-22T23:25:01.494 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (6m) 53s ago 6m 201M 2240M 18.2.1 5be31c24972a c7586562ffa6 2024-04-22T23:25:01.494 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (6m) 53s ago 6m 227M 2240M 18.2.1 5be31c24972a 9fd97f1c65cb 2024-04-22T23:25:01.494 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (33s) 7s ago 9m 40.2M - 2.43.0 a07b618ecd1d f20b725ea6e0 2024-04-22T23:25:01.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.493+0000 7ff6b97c9700 1 -- 172.21.15.67:0/2988322287 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7ff69c005eb0 msgr2=0x7ff69c036350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:25:01.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.493+0000 7ff6b97c9700 1 --2- 172.21.15.67:0/2988322287 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7ff69c005eb0 0x7ff69c036350 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff6b419bf00 tx=0x7ff6a8023000 comp rx=0 tx=0).stop 2024-04-22T23:25:01.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.493+0000 7ff6b97c9700 1 -- 172.21.15.67:0/2988322287 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff6b4100cb0 msgr2=0x7ff6b41964b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:25:01.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.493+0000 7ff6b97c9700 1 --2- 172.21.15.67:0/2988322287 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff6b4100cb0 0x7ff6b41964b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff6a400e8f0 tx=0x7ff6a400ec00 comp rx=0 tx=0).stop 2024-04-22T23:25:01.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.493+0000 7ff6b97c9700 1 -- 172.21.15.67:0/2988322287 shutdown_connections 2024-04-22T23:25:01.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.493+0000 7ff6b97c9700 1 --2- 172.21.15.67:0/2988322287 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7ff69c005eb0 0x7ff69c036350 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:01.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.493+0000 7ff6b97c9700 1 --2- 172.21.15.67:0/2988322287 >> [v2:172.21.15.134:6828/1077503272,v1:172.21.15.134:6829/1077503272] conn(0x7ff69c06ea80 0x7ff69c070f40 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:01.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.493+0000 7ff6b97c9700 1 --2- 172.21.15.67:0/2988322287 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff6b4100cb0 0x7ff6b41964b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:01.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.493+0000 7ff6b97c9700 1 --2- 172.21.15.67:0/2988322287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6b4101eb0 0x7ff6b41969f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:01.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.493+0000 7ff6b97c9700 1 -- 172.21.15.67:0/2988322287 >> 172.21.15.67:0/2988322287 conn(0x7ff6b40fc230 msgr2=0x7ff6b41050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:25:01.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.493+0000 7ff6b97c9700 1 -- 172.21.15.67:0/2988322287 shutdown_connections 2024-04-22T23:25:01.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.493+0000 7ff6b97c9700 1 -- 172.21.15.67:0/2988322287 wait complete. 2024-04-22T23:25:01.591 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:01 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:25:01.591 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:01 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:25:01.591 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:01 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:25:01.591 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:01 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:25:01.591 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:01 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:25:01.591 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:01 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:25:01.591 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:01 smithi134 ceph-mon[34609]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "mgr fail", "who": "smithi134.ccxkcj"}]: dispatch 2024-04-22T23:25:01.591 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:01 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "mgr fail", "who": "smithi134.ccxkcj"}]: dispatch 2024-04-22T23:25:01.591 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:01 smithi134 ceph-mon[34609]: osdmap e48: 6 total, 6 up, 6 in 2024-04-22T23:25:01.591 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:01 smithi134 ceph-mon[34609]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' cmd='[{"prefix": "mgr fail", "who": "smithi134.ccxkcj"}]': finished 2024-04-22T23:25:01.591 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:01 smithi134 ceph-mon[34609]: mgrmap e27: smithi067.whdcsr(active, starting, since 0.0075544s) 2024-04-22T23:25:01.592 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:01 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-22T23:25:01.592 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:01 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-04-22T23:25:01.592 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:01 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.xyazmh"}]: dispatch 2024-04-22T23:25:01.592 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:01 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.aqlxpv"}]: dispatch 2024-04-22T23:25:01.592 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:01 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.owipad"}]: dispatch 2024-04-22T23:25:01.592 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:01 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.vuxbsj"}]: dispatch 2024-04-22T23:25:01.592 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:01 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mgr metadata", "who": "smithi067.whdcsr", "id": "smithi067.whdcsr"}]: dispatch 2024-04-22T23:25:01.592 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:01 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T23:25:01.592 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:01 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T23:25:01.592 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:01 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T23:25:01.592 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:01 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T23:25:01.592 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:01 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T23:25:01.592 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:01 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T23:25:01.592 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:01 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T23:25:01.592 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:01 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T23:25:01.593 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:01 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T23:25:01.593 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:01 smithi134 ceph-mon[34609]: Manager daemon smithi067.whdcsr is now available 2024-04-22T23:25:01.593 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:01 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:25:01.593 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:01 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:25:01.593 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:01 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.whdcsr/mirror_snapshot_schedule"}]: dispatch 2024-04-22T23:25:01.593 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:01 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.whdcsr/trash_purge_schedule"}]: dispatch 2024-04-22T23:25:01.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.618+0000 7f7872fc0700 1 -- 172.21.15.67:0/601302253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f786c100e10 msgr2=0x7f786c101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:25:01.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.618+0000 7f7872fc0700 1 --2- 172.21.15.67:0/601302253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f786c100e10 0x7f786c101270 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f7858009910 tx=0x7f7858009c20 comp rx=0 tx=0).stop 2024-04-22T23:25:01.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.618+0000 7f7872fc0700 1 -- 172.21.15.67:0/601302253 shutdown_connections 2024-04-22T23:25:01.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.618+0000 7f7872fc0700 1 --2- 172.21.15.67:0/601302253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f786c100e10 0x7f786c101270 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:01.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.618+0000 7f7872fc0700 1 --2- 172.21.15.67:0/601302253 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f786c0ffc10 0x7f786c100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:01.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.618+0000 7f7872fc0700 1 -- 172.21.15.67:0/601302253 >> 172.21.15.67:0/601302253 conn(0x7f786c0fb190 msgr2=0x7f786c0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:25:01.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.618+0000 7f7872fc0700 1 -- 172.21.15.67:0/601302253 shutdown_connections 2024-04-22T23:25:01.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.618+0000 7f7872fc0700 1 -- 172.21.15.67:0/601302253 wait complete. 2024-04-22T23:25:01.621 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:01 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:25:01.621 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:01 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:25:01.621 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:01 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:25:01.621 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:01 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:25:01.621 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:01 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' 2024-04-22T23:25:01.621 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:01 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:25:01.621 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:01 smithi067 ceph-mon[26453]: from='mgr.14622 172.21.15.134:0/2332762474' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "mgr fail", "who": "smithi134.ccxkcj"}]: dispatch 2024-04-22T23:25:01.621 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:01 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "mgr fail", "who": "smithi134.ccxkcj"}]: dispatch 2024-04-22T23:25:01.621 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:01 smithi067 ceph-mon[26453]: osdmap e48: 6 total, 6 up, 6 in 2024-04-22T23:25:01.621 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:01 smithi067 ceph-mon[26453]: from='mgr.14622 ' entity='mgr.smithi134.ccxkcj' cmd='[{"prefix": "mgr fail", "who": "smithi134.ccxkcj"}]': finished 2024-04-22T23:25:01.621 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:01 smithi067 ceph-mon[26453]: mgrmap e27: smithi067.whdcsr(active, starting, since 0.0075544s) 2024-04-22T23:25:01.621 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:01 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-22T23:25:01.621 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:01 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-04-22T23:25:01.621 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:01 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.xyazmh"}]: dispatch 2024-04-22T23:25:01.621 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:01 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.aqlxpv"}]: dispatch 2024-04-22T23:25:01.622 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:01 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.owipad"}]: dispatch 2024-04-22T23:25:01.622 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:01 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.vuxbsj"}]: dispatch 2024-04-22T23:25:01.622 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:01 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mgr metadata", "who": "smithi067.whdcsr", "id": "smithi067.whdcsr"}]: dispatch 2024-04-22T23:25:01.622 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:01 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T23:25:01.622 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:01 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T23:25:01.622 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:01 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T23:25:01.622 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:01 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T23:25:01.622 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:01 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T23:25:01.622 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:01 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T23:25:01.622 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:01 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T23:25:01.622 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:01 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T23:25:01.622 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:01 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T23:25:01.622 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:01 smithi067 ceph-mon[26453]: Manager daemon smithi067.whdcsr is now available 2024-04-22T23:25:01.622 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:01 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:25:01.623 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:01 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:25:01.623 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:01 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.whdcsr/mirror_snapshot_schedule"}]: dispatch 2024-04-22T23:25:01.623 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:01 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.whdcsr/trash_purge_schedule"}]: dispatch 2024-04-22T23:25:01.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.619+0000 7f7872fc0700 1 Processor -- start 2024-04-22T23:25:01.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.619+0000 7f7872fc0700 1 -- start start 2024-04-22T23:25:01.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.619+0000 7f7872fc0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f786c0ffc10 0x7f786c0690e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:25:01.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.619+0000 7f7872fc0700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f786c100e10 0x7f786c19b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:25:01.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.619+0000 7f7872fc0700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786c19b7d0 con 0x7f786c0ffc10 2024-04-22T23:25:01.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.619+0000 7f7872fc0700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786c19b910 con 0x7f786c100e10 2024-04-22T23:25:01.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.619+0000 7f7870d5c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f786c0ffc10 0x7f786c0690e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:25:01.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.620+0000 7f7870d5c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f786c0ffc10 0x7f786c0690e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54894/0 (socket says 172.21.15.67:54894) 2024-04-22T23:25:01.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.620+0000 7f7870d5c700 1 -- 172.21.15.67:0/77717173 learned_addr learned my addr 172.21.15.67:0/77717173 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:25:01.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.620+0000 7f786bfff700 1 --2- 172.21.15.67:0/77717173 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f786c100e10 0x7f786c19b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:25:01.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.620+0000 7f7870d5c700 1 -- 172.21.15.67:0/77717173 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f786c100e10 msgr2=0x7f786c19b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:25:01.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.620+0000 7f7870d5c700 1 --2- 172.21.15.67:0/77717173 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f786c100e10 0x7f786c19b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:01.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.620+0000 7f7870d5c700 1 -- 172.21.15.67:0/77717173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78580095f0 con 0x7f786c0ffc10 2024-04-22T23:25:01.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.620+0000 7f7870d5c700 1 --2- 172.21.15.67:0/77717173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f786c0ffc10 0x7f786c0690e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f786000d900 tx=0x7f786000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:25:01.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.620+0000 7f7869ffb700 1 -- 172.21.15.67:0/77717173 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f786000b920 con 0x7f786c0ffc10 2024-04-22T23:25:01.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.620+0000 7f7872fc0700 1 -- 172.21.15.67:0/77717173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f786c19bbf0 con 0x7f786c0ffc10 2024-04-22T23:25:01.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.621+0000 7f7869ffb700 1 -- 172.21.15.67:0/77717173 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7860010460 con 0x7f786c0ffc10 2024-04-22T23:25:01.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.621+0000 7f7869ffb700 1 -- 172.21.15.67:0/77717173 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f786000f5f0 con 0x7f786c0ffc10 2024-04-22T23:25:01.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.621+0000 7f7872fc0700 1 -- 172.21.15.67:0/77717173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f786c19c140 con 0x7f786c0ffc10 2024-04-22T23:25:01.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.622+0000 7f7869ffb700 1 -- 172.21.15.67:0/77717173 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 28) v1 ==== 46347+0+0 (secure 0 0 0) 0x7f786000f860 con 0x7f786c0ffc10 2024-04-22T23:25:01.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.622+0000 7f7869ffb700 1 --2- 172.21.15.67:0/77717173 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f785c0394c0 0x7f785c03b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:25:01.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.622+0000 7f7869ffb700 1 -- 172.21.15.67:0/77717173 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f786004d1c0 con 0x7f786c0ffc10 2024-04-22T23:25:01.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.622+0000 7f7872fc0700 1 -- 172.21.15.67:0/77717173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f786c108700 con 0x7f786c0ffc10 2024-04-22T23:25:01.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.623+0000 7f786bfff700 1 --2- 172.21.15.67:0/77717173 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f785c0394c0 0x7f785c03b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:25:01.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.623+0000 7f786bfff700 1 --2- 172.21.15.67:0/77717173 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f785c0394c0 0x7f785c03b980 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f786c069660 tx=0x7f785800b560 comp rx=0 tx=0).ready entity=mgr.14708 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:25:01.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.629+0000 7f7869ffb700 1 -- 172.21.15.67:0/77717173 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f78600228e0 con 0x7f786c0ffc10 2024-04-22T23:25:01.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.947+0000 7f7872fc0700 1 -- 172.21.15.67:0/77717173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f786c19c750 con 0x7f786c0ffc10 2024-04-22T23:25:01.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.948+0000 7f7869ffb700 1 -- 172.21.15.67:0/77717173 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7f78600222f0 con 0x7f786c0ffc10 2024-04-22T23:25:01.950 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:25:01.950 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:25:01.950 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-22T23:25:01.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:25:01.950 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:25:01.950 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 1 2024-04-22T23:25:01.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:25:01.951 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:25:01.951 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-22T23:25:01.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:25:01.951 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:25:01.951 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-22T23:25:01.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:25:01.951 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:25:01.951 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 12, 2024-04-22T23:25:01.951 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 1 2024-04-22T23:25:01.951 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:25:01.951 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:25:01.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.951+0000 7f7872fc0700 1 -- 172.21.15.67:0/77717173 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f785c0394c0 msgr2=0x7f785c03b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:25:01.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.951+0000 7f7872fc0700 1 --2- 172.21.15.67:0/77717173 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f785c0394c0 0x7f785c03b980 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f786c069660 tx=0x7f785800b560 comp rx=0 tx=0).stop 2024-04-22T23:25:01.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.951+0000 7f7872fc0700 1 -- 172.21.15.67:0/77717173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f786c0ffc10 msgr2=0x7f786c0690e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:25:01.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.951+0000 7f7872fc0700 1 --2- 172.21.15.67:0/77717173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f786c0ffc10 0x7f786c0690e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f786000d900 tx=0x7f786000dc10 comp rx=0 tx=0).stop 2024-04-22T23:25:01.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.951+0000 7f7872fc0700 1 -- 172.21.15.67:0/77717173 shutdown_connections 2024-04-22T23:25:01.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.951+0000 7f7872fc0700 1 --2- 172.21.15.67:0/77717173 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f785c0394c0 0x7f785c03b980 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:01.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.951+0000 7f7872fc0700 1 --2- 172.21.15.67:0/77717173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f786c0ffc10 0x7f786c0690e0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:01.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.951+0000 7f7872fc0700 1 --2- 172.21.15.67:0/77717173 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f786c100e10 0x7f786c19b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:01.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.951+0000 7f7872fc0700 1 -- 172.21.15.67:0/77717173 >> 172.21.15.67:0/77717173 conn(0x7f786c0fb190 msgr2=0x7f786c104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:25:01.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.951+0000 7f7872fc0700 1 -- 172.21.15.67:0/77717173 shutdown_connections 2024-04-22T23:25:01.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:01.951+0000 7f7872fc0700 1 -- 172.21.15.67:0/77717173 wait complete. 2024-04-22T23:25:02.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.069+0000 7ff69745b700 1 -- 172.21.15.67:0/3580164317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff690101eb0 msgr2=0x7ff690102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:25:02.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.069+0000 7ff69745b700 1 --2- 172.21.15.67:0/3580164317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff690101eb0 0x7ff690102310 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff680009910 tx=0x7ff680009c20 comp rx=0 tx=0).stop 2024-04-22T23:25:02.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.069+0000 7ff69745b700 1 -- 172.21.15.67:0/3580164317 shutdown_connections 2024-04-22T23:25:02.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.069+0000 7ff69745b700 1 --2- 172.21.15.67:0/3580164317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff690101eb0 0x7ff690102310 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:02.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.069+0000 7ff69745b700 1 --2- 172.21.15.67:0/3580164317 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff690100cb0 0x7ff6901010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:02.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.069+0000 7ff69745b700 1 -- 172.21.15.67:0/3580164317 >> 172.21.15.67:0/3580164317 conn(0x7ff6900fc230 msgr2=0x7ff6900fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:25:02.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.070+0000 7ff69745b700 1 -- 172.21.15.67:0/3580164317 shutdown_connections 2024-04-22T23:25:02.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.070+0000 7ff69745b700 1 -- 172.21.15.67:0/3580164317 wait complete. 2024-04-22T23:25:02.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.070+0000 7ff69745b700 1 Processor -- start 2024-04-22T23:25:02.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.070+0000 7ff69745b700 1 -- start start 2024-04-22T23:25:02.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.070+0000 7ff69745b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff690100cb0 0x7ff690074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:25:02.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.070+0000 7ff69745b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff690101eb0 0x7ff690073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:25:02.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.070+0000 7ff69745b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6900736c0 con 0x7ff690100cb0 2024-04-22T23:25:02.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.070+0000 7ff69745b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff690073800 con 0x7ff690101eb0 2024-04-22T23:25:02.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.070+0000 7ff6951f7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff690100cb0 0x7ff690074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:25:02.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.071+0000 7ff6951f7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff690100cb0 0x7ff690074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54914/0 (socket says 172.21.15.67:54914) 2024-04-22T23:25:02.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.071+0000 7ff6951f7700 1 -- 172.21.15.67:0/4131471296 learned_addr learned my addr 172.21.15.67:0/4131471296 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:25:02.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.071+0000 7ff6951f7700 1 -- 172.21.15.67:0/4131471296 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff690101eb0 msgr2=0x7ff690073180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:25:02.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.071+0000 7ff6951f7700 1 --2- 172.21.15.67:0/4131471296 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff690101eb0 0x7ff690073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:02.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.071+0000 7ff6951f7700 1 -- 172.21.15.67:0/4131471296 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6800095f0 con 0x7ff690100cb0 2024-04-22T23:25:02.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.071+0000 7ff6951f7700 1 --2- 172.21.15.67:0/4131471296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff690100cb0 0x7ff690074b30 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7ff68c00d900 tx=0x7ff68c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:25:02.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.071+0000 7ff6867fc700 1 -- 172.21.15.67:0/4131471296 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff68c0041f0 con 0x7ff690100cb0 2024-04-22T23:25:02.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.071+0000 7ff6867fc700 1 -- 172.21.15.67:0/4131471296 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7ff68c004350 con 0x7ff690100cb0 2024-04-22T23:25:02.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.071+0000 7ff6867fc700 1 -- 172.21.15.67:0/4131471296 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff68c003da0 con 0x7ff690100cb0 2024-04-22T23:25:02.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.071+0000 7ff69745b700 1 -- 172.21.15.67:0/4131471296 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff690073ae0 con 0x7ff690100cb0 2024-04-22T23:25:02.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.071+0000 7ff69745b700 1 -- 172.21.15.67:0/4131471296 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff69019e540 con 0x7ff690100cb0 2024-04-22T23:25:02.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.072+0000 7ff6867fc700 1 -- 172.21.15.67:0/4131471296 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 28) v1 ==== 46347+0+0 (secure 0 0 0) 0x7ff68c009730 con 0x7ff690100cb0 2024-04-22T23:25:02.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.072+0000 7ff69745b700 1 -- 172.21.15.67:0/4131471296 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff690109810 con 0x7ff690100cb0 2024-04-22T23:25:02.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.072+0000 7ff6867fc700 1 --2- 172.21.15.67:0/4131471296 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7ff67c0394b0 0x7ff67c03b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:25:02.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.072+0000 7ff6867fc700 1 -- 172.21.15.67:0/4131471296 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7ff68c04d770 con 0x7ff690100cb0 2024-04-22T23:25:02.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.073+0000 7ff6949f6700 1 --2- 172.21.15.67:0/4131471296 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7ff67c0394b0 0x7ff67c03b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:25:02.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.073+0000 7ff6949f6700 1 --2- 172.21.15.67:0/4131471296 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7ff67c0394b0 0x7ff67c03b970 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff68000b5c0 tx=0x7ff680005960 comp rx=0 tx=0).ready entity=mgr.14708 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:25:02.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.077+0000 7ff6867fc700 1 -- 172.21.15.67:0/4131471296 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7ff68c01a950 con 0x7ff690100cb0 2024-04-22T23:25:02.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.371+0000 7ff69745b700 1 -- 172.21.15.67:0/4131471296 --> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff690106800 con 0x7ff67c0394b0 2024-04-22T23:25:02.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.373+0000 7ff6867fc700 1 -- 172.21.15.67:0/4131471296 <== mgr.14708 v2:172.21.15.67:6800/2306276228 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7ff690106800 con 0x7ff67c0394b0 2024-04-22T23:25:02.375 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:25:02.375 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:25:02.375 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:25:02.375 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-22T23:25:02.376 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:25:02.376 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr" 2024-04-22T23:25:02.376 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:25:02.376 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "2/2 daemons upgraded", 2024-04-22T23:25:02.376 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "", 2024-04-22T23:25:02.376 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:25:02.376 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:25:02.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.375+0000 7ff69745b700 1 -- 172.21.15.67:0/4131471296 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7ff67c0394b0 msgr2=0x7ff67c03b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:25:02.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.375+0000 7ff69745b700 1 --2- 172.21.15.67:0/4131471296 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7ff67c0394b0 0x7ff67c03b970 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff68000b5c0 tx=0x7ff680005960 comp rx=0 tx=0).stop 2024-04-22T23:25:02.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.375+0000 7ff69745b700 1 -- 172.21.15.67:0/4131471296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff690100cb0 msgr2=0x7ff690074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:25:02.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.375+0000 7ff69745b700 1 --2- 172.21.15.67:0/4131471296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff690100cb0 0x7ff690074b30 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7ff68c00d900 tx=0x7ff68c00dcc0 comp rx=0 tx=0).stop 2024-04-22T23:25:02.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.375+0000 7ff69745b700 1 -- 172.21.15.67:0/4131471296 shutdown_connections 2024-04-22T23:25:02.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.375+0000 7ff69745b700 1 --2- 172.21.15.67:0/4131471296 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7ff67c0394b0 0x7ff67c03b970 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:02.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.375+0000 7ff69745b700 1 --2- 172.21.15.67:0/4131471296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff690100cb0 0x7ff690074b30 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:02.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.375+0000 7ff69745b700 1 --2- 172.21.15.67:0/4131471296 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff690101eb0 0x7ff690073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:02.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.375+0000 7ff69745b700 1 -- 172.21.15.67:0/4131471296 >> 172.21.15.67:0/4131471296 conn(0x7ff6900fc230 msgr2=0x7ff6901050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:25:02.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.375+0000 7ff69745b700 1 -- 172.21.15.67:0/4131471296 shutdown_connections 2024-04-22T23:25:02.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:02.375+0000 7ff69745b700 1 -- 172.21.15.67:0/4131471296 wait complete. 2024-04-22T23:25:02.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:02 smithi067 ceph-mon[26453]: [22/Apr/2024:23:25:01] ENGINE Bus STARTING 2024-04-22T23:25:02.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:02 smithi067 ceph-mon[26453]: [22/Apr/2024:23:25:01] ENGINE Serving on https://172.21.15.67:7150 2024-04-22T23:25:02.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:02 smithi067 ceph-mon[26453]: [22/Apr/2024:23:25:01] ENGINE Serving on http://172.21.15.67:8765 2024-04-22T23:25:02.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:02 smithi067 ceph-mon[26453]: [22/Apr/2024:23:25:01] ENGINE Bus STARTED 2024-04-22T23:25:02.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:02 smithi067 ceph-mon[26453]: mgrmap e28: smithi067.whdcsr(active, since 1.00895s) 2024-04-22T23:25:02.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:02 smithi067 ceph-mon[26453]: from='client.24467 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:25:02.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:02 smithi067 ceph-mon[26453]: pgmap v3: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:25:02.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:02 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/77717173' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:25:02.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:02 smithi134 ceph-mon[34609]: [22/Apr/2024:23:25:01] ENGINE Bus STARTING 2024-04-22T23:25:02.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:02 smithi134 ceph-mon[34609]: [22/Apr/2024:23:25:01] ENGINE Serving on https://172.21.15.67:7150 2024-04-22T23:25:02.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:02 smithi134 ceph-mon[34609]: [22/Apr/2024:23:25:01] ENGINE Serving on http://172.21.15.67:8765 2024-04-22T23:25:02.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:02 smithi134 ceph-mon[34609]: [22/Apr/2024:23:25:01] ENGINE Bus STARTED 2024-04-22T23:25:02.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:02 smithi134 ceph-mon[34609]: mgrmap e28: smithi067.whdcsr(active, since 1.00895s) 2024-04-22T23:25:02.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:02 smithi134 ceph-mon[34609]: from='client.24467 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:25:02.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:02 smithi134 ceph-mon[34609]: pgmap v3: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:25:02.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:02 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/77717173' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:25:03.733 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:03 smithi067 ceph-mon[26453]: from='client.14750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:25:03.733 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:03 smithi067 ceph-mon[26453]: mgrmap e29: smithi067.whdcsr(active, since 2s) 2024-04-22T23:25:03.733 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:03 smithi067 ceph-mon[26453]: pgmap v4: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:25:03.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:03 smithi134 ceph-mon[34609]: from='client.14750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:25:03.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:03 smithi134 ceph-mon[34609]: mgrmap e29: smithi067.whdcsr(active, since 2s) 2024-04-22T23:25:03.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:03 smithi134 ceph-mon[34609]: pgmap v4: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:25:04.730 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-04-22T23:25:04.730 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.1 /home/ubuntu/cephtest/clone.client.1 2024-04-22T23:25:05.026 DEBUG:teuthology.parallel:result is None 2024-04-22T23:25:05.026 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-22T23:25:05.079 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-22T23:25:05.080 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1 2024-04-22T23:25:05.210 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.1/client.1 2024-04-22T23:25:05.211 DEBUG:teuthology.parallel:result is None 2024-04-22T23:25:05.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:05 smithi134 ceph-mon[34609]: pgmap v5: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:25:05.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:05 smithi134 ceph-mon[34609]: mgrmap e30: smithi067.whdcsr(active, since 4s) 2024-04-22T23:25:05.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:05 smithi067 ceph-mon[26453]: pgmap v5: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:25:05.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:05 smithi067 ceph-mon[26453]: mgrmap e30: smithi067.whdcsr(active, since 4s) 2024-04-22T23:25:06.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:06 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:06.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:06 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:06.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:06 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:06.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:06 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:07.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:06 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:07.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:06 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:07.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:06 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:07.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:06 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:07.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:07 smithi134 ceph-mon[34609]: pgmap v6: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 326 KiB/s rd, 331 KiB/s wr, 42 op/s 2024-04-22T23:25:07.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:07 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:07.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:07 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:07.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:07 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:07.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:07 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:08.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:07 smithi067 ceph-mon[26453]: pgmap v6: 65 pgs: 65 active+clean; 315 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 326 KiB/s rd, 331 KiB/s wr, 42 op/s 2024-04-22T23:25:08.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:07 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:08.296 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:07 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:08.296 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:07 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:08.296 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:07 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:08.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:08 smithi134 ceph-mon[34609]: Standby manager daemon smithi134.ccxkcj started 2024-04-22T23:25:08.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:08 smithi134 ceph-mon[34609]: from='mgr.? 172.21.15.134:0/2210421924' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi134.ccxkcj/crt"}]: dispatch 2024-04-22T23:25:08.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:08 smithi134 ceph-mon[34609]: from='mgr.? 172.21.15.134:0/2210421924' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T23:25:08.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:08 smithi134 ceph-mon[34609]: from='mgr.? 172.21.15.134:0/2210421924' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi134.ccxkcj/key"}]: dispatch 2024-04-22T23:25:08.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:08 smithi134 ceph-mon[34609]: from='mgr.? 172.21.15.134:0/2210421924' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T23:25:09.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:08 smithi067 ceph-mon[26453]: Standby manager daemon smithi134.ccxkcj started 2024-04-22T23:25:09.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:08 smithi067 ceph-mon[26453]: from='mgr.? 172.21.15.134:0/2210421924' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi134.ccxkcj/crt"}]: dispatch 2024-04-22T23:25:09.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:08 smithi067 ceph-mon[26453]: from='mgr.? 172.21.15.134:0/2210421924' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T23:25:09.046 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:08 smithi067 ceph-mon[26453]: from='mgr.? 172.21.15.134:0/2210421924' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi134.ccxkcj/key"}]: dispatch 2024-04-22T23:25:09.046 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:08 smithi067 ceph-mon[26453]: from='mgr.? 172.21.15.134:0/2210421924' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T23:25:09.959 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:09 smithi067 ceph-mon[26453]: pgmap v7: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 731 KiB/s rd, 616 KiB/s wr, 148 op/s 2024-04-22T23:25:09.959 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:09 smithi067 ceph-mon[26453]: mgrmap e31: smithi067.whdcsr(active, since 8s), standbys: smithi134.ccxkcj 2024-04-22T23:25:09.960 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:09 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mgr metadata", "who": "smithi134.ccxkcj", "id": "smithi134.ccxkcj"}]: dispatch 2024-04-22T23:25:10.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:09 smithi134 ceph-mon[34609]: pgmap v7: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 731 KiB/s rd, 616 KiB/s wr, 148 op/s 2024-04-22T23:25:10.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:09 smithi134 ceph-mon[34609]: mgrmap e31: smithi067.whdcsr(active, since 8s), standbys: smithi134.ccxkcj 2024-04-22T23:25:10.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:09 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mgr metadata", "who": "smithi134.ccxkcj", "id": "smithi134.ccxkcj"}]: dispatch 2024-04-22T23:25:11.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:11 smithi134 ceph-mon[34609]: pgmap v8: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 569 KiB/s rd, 479 KiB/s wr, 115 op/s 2024-04-22T23:25:11.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:11 smithi067 ceph-mon[26453]: pgmap v8: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 569 KiB/s rd, 479 KiB/s wr, 115 op/s 2024-04-22T23:25:12.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:12 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:12.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:12 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:12.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:12 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:25:12.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:12 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:25:12.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:12 smithi067 ceph-mon[26453]: Updating smithi067:/etc/ceph/ceph.conf 2024-04-22T23:25:12.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:12 smithi067 ceph-mon[26453]: Updating smithi134:/etc/ceph/ceph.conf 2024-04-22T23:25:12.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:12 smithi067 ceph-mon[26453]: Updating smithi067:/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T23:25:12.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:12 smithi067 ceph-mon[26453]: Updating smithi134:/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T23:25:12.922 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:12 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:12.923 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:12 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:12.923 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:12 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:25:12.923 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:12 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:25:12.923 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:12 smithi134 ceph-mon[34609]: Updating smithi067:/etc/ceph/ceph.conf 2024-04-22T23:25:12.923 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:12 smithi134 ceph-mon[34609]: Updating smithi134:/etc/ceph/ceph.conf 2024-04-22T23:25:12.923 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:12 smithi134 ceph-mon[34609]: Updating smithi067:/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T23:25:12.923 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:12 smithi134 ceph-mon[34609]: Updating smithi134:/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T23:25:13.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:13 smithi067 ceph-mon[26453]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-04-22T23:25:13.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:13 smithi067 ceph-mon[26453]: Updating smithi134:/etc/ceph/ceph.client.admin.keyring 2024-04-22T23:25:13.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:13 smithi067 ceph-mon[26453]: Updating smithi067:/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T23:25:13.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:13 smithi067 ceph-mon[26453]: Updating smithi134:/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T23:25:13.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:13 smithi067 ceph-mon[26453]: pgmap v9: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 475 KiB/s rd, 401 KiB/s wr, 98 op/s 2024-04-22T23:25:13.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:13 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:13.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:13 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:13.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:13 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:13.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:13 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:13.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:13 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:13.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:13 smithi067 ceph-mon[26453]: Reconfiguring prometheus.smithi067 (dependencies changed)... 2024-04-22T23:25:13.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:13 smithi067 ceph-mon[26453]: Reconfiguring daemon prometheus.smithi067 on smithi067 2024-04-22T23:25:13.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:13 smithi134 ceph-mon[34609]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-04-22T23:25:13.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:13 smithi134 ceph-mon[34609]: Updating smithi134:/etc/ceph/ceph.client.admin.keyring 2024-04-22T23:25:13.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:13 smithi134 ceph-mon[34609]: Updating smithi067:/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T23:25:13.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:13 smithi134 ceph-mon[34609]: Updating smithi134:/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T23:25:13.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:13 smithi134 ceph-mon[34609]: pgmap v9: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 475 KiB/s rd, 401 KiB/s wr, 98 op/s 2024-04-22T23:25:13.996 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:13 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:13.996 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:13 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:13.996 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:13 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:13.996 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:13 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:13.996 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:13 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:13.996 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:13 smithi134 ceph-mon[34609]: Reconfiguring prometheus.smithi067 (dependencies changed)... 2024-04-22T23:25:13.996 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:13 smithi134 ceph-mon[34609]: Reconfiguring daemon prometheus.smithi067 on smithi067 2024-04-22T23:25:15.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:15 smithi067 ceph-mon[26453]: pgmap v10: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 435 KiB/s rd, 367 KiB/s wr, 90 op/s 2024-04-22T23:25:15.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:15 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:25:15.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:15 smithi134 ceph-mon[34609]: pgmap v10: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 435 KiB/s rd, 367 KiB/s wr, 90 op/s 2024-04-22T23:25:15.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:15 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:25:17.908 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:17 smithi067 ceph-mon[26453]: pgmap v11: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 435 KiB/s rd, 367 KiB/s wr, 90 op/s 2024-04-22T23:25:17.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:17 smithi134 ceph-mon[34609]: pgmap v11: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 435 KiB/s rd, 367 KiB/s wr, 90 op/s 2024-04-22T23:25:19.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:19 smithi134 ceph-mon[34609]: pgmap v12: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 301 KiB/s rd, 230 KiB/s wr, 74 op/s 2024-04-22T23:25:19.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:19 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:19.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:19 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:19.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:19 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T23:25:19.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:19 smithi134 ceph-mon[34609]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T23:25:19.746 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:19 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:25:19.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:19 smithi067 ceph-mon[26453]: pgmap v12: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 301 KiB/s rd, 230 KiB/s wr, 74 op/s 2024-04-22T23:25:19.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:19 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:19.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:19 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:19.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:19 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T23:25:19.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:19 smithi067 ceph-mon[26453]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T23:25:19.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:19 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:25:20.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:20 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi134.ccxkcj", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T23:25:20.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:20 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T23:25:20.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:20 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:25:20.831 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:20 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi134.ccxkcj", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T23:25:20.832 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:20 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T23:25:20.832 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:20 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:25:21.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:21 smithi067 ceph-mon[26453]: Upgrade: Updating mgr.smithi134.ccxkcj 2024-04-22T23:25:21.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:21 smithi067 ceph-mon[26453]: Deploying daemon mgr.smithi134.ccxkcj on smithi134 2024-04-22T23:25:21.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:21 smithi067 ceph-mon[26453]: pgmap v13: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 10 KiB/s rd, 8.6 KiB/s wr, 5 op/s 2024-04-22T23:25:21.913 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:21 smithi134 ceph-mon[34609]: Upgrade: Updating mgr.smithi134.ccxkcj 2024-04-22T23:25:21.913 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:21 smithi134 ceph-mon[34609]: Deploying daemon mgr.smithi134.ccxkcj on smithi134 2024-04-22T23:25:21.913 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:21 smithi134 ceph-mon[34609]: pgmap v13: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 10 KiB/s rd, 8.6 KiB/s wr, 5 op/s 2024-04-22T23:25:23.774 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:23 smithi134 ceph-mon[34609]: pgmap v14: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 11 KiB/s rd, 8.6 KiB/s wr, 7 op/s 2024-04-22T23:25:23.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:23 smithi067 ceph-mon[26453]: pgmap v14: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 11 KiB/s rd, 8.6 KiB/s wr, 7 op/s 2024-04-22T23:25:25.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:24 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:25.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:24 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:25.046 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:24 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:25:25.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:24 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:25.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:24 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:25.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:24 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:25:26.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:25 smithi067 ceph-mon[26453]: pgmap v15: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 170 B/s wr, 4 op/s 2024-04-22T23:25:26.093 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:25 smithi134 ceph-mon[34609]: pgmap v15: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 170 B/s wr, 4 op/s 2024-04-22T23:25:27.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:27 smithi134 ceph-mon[34609]: pgmap v16: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 170 B/s wr, 4 op/s 2024-04-22T23:25:27.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:27 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:27.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:27 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:28.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:27 smithi067 ceph-mon[26453]: pgmap v16: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 170 B/s wr, 4 op/s 2024-04-22T23:25:28.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:27 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:28.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:27 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:29.396 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:29 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:29.397 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:29 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:29.397 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:29 smithi134 ceph-mon[34609]: pgmap v17: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-22T23:25:29.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:29 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:29.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:29 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:29.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:29 smithi067 ceph-mon[26453]: pgmap v17: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-22T23:25:30.701 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:30 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:30.702 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:30 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:30.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:30 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:30.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:30 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:31.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:31 smithi134 ceph-mon[34609]: pgmap v18: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T23:25:31.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:31 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:25:31.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:31 smithi134 ceph-mon[34609]: Standby manager daemon smithi134.ccxkcj restarted 2024-04-22T23:25:31.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:31 smithi134 ceph-mon[34609]: Standby manager daemon smithi134.ccxkcj started 2024-04-22T23:25:31.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:31 smithi134 ceph-mon[34609]: from='mgr.? 172.21.15.134:0/48961984' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi134.ccxkcj/crt"}]: dispatch 2024-04-22T23:25:31.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:31 smithi134 ceph-mon[34609]: from='mgr.? 172.21.15.134:0/48961984' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T23:25:31.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:31 smithi134 ceph-mon[34609]: from='mgr.? 172.21.15.134:0/48961984' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi134.ccxkcj/key"}]: dispatch 2024-04-22T23:25:31.746 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:31 smithi134 ceph-mon[34609]: from='mgr.? 172.21.15.134:0/48961984' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T23:25:31.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:31 smithi067 ceph-mon[26453]: pgmap v18: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T23:25:31.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:31 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:25:31.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:31 smithi067 ceph-mon[26453]: Standby manager daemon smithi134.ccxkcj restarted 2024-04-22T23:25:31.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:31 smithi067 ceph-mon[26453]: Standby manager daemon smithi134.ccxkcj started 2024-04-22T23:25:31.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:31 smithi067 ceph-mon[26453]: from='mgr.? 172.21.15.134:0/48961984' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi134.ccxkcj/crt"}]: dispatch 2024-04-22T23:25:31.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:31 smithi067 ceph-mon[26453]: from='mgr.? 172.21.15.134:0/48961984' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T23:25:31.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:31 smithi067 ceph-mon[26453]: from='mgr.? 172.21.15.134:0/48961984' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi134.ccxkcj/key"}]: dispatch 2024-04-22T23:25:31.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:31 smithi067 ceph-mon[26453]: from='mgr.? 172.21.15.134:0/48961984' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T23:25:32.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.492+0000 7f438fc22700 1 -- 172.21.15.67:0/343996636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4388104070 msgr2=0x7f4388106460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:25:32.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.492+0000 7f438fc22700 1 --2- 172.21.15.67:0/343996636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4388104070 0x7f4388106460 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f4378009910 tx=0x7f4378009c20 comp rx=0 tx=0).stop 2024-04-22T23:25:32.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.492+0000 7f438fc22700 1 -- 172.21.15.67:0/343996636 shutdown_connections 2024-04-22T23:25:32.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.492+0000 7f438fc22700 1 --2- 172.21.15.67:0/343996636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4388104070 0x7f4388106460 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:32.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.492+0000 7f438fc22700 1 --2- 172.21.15.67:0/343996636 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4388101740 0x7f4388103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:32.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.492+0000 7f438fc22700 1 -- 172.21.15.67:0/343996636 >> 172.21.15.67:0/343996636 conn(0x7f43880fb150 msgr2=0x7f43880fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:25:32.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.492+0000 7f438fc22700 1 -- 172.21.15.67:0/343996636 shutdown_connections 2024-04-22T23:25:32.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.492+0000 7f438fc22700 1 -- 172.21.15.67:0/343996636 wait complete. 2024-04-22T23:25:32.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.493+0000 7f438fc22700 1 Processor -- start 2024-04-22T23:25:32.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.493+0000 7f438fc22700 1 -- start start 2024-04-22T23:25:32.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.493+0000 7f438fc22700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4388101740 0x7f4388196460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:25:32.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.493+0000 7f438fc22700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4388104070 0x7f43881969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:25:32.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.493+0000 7f438fc22700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4388196f30 con 0x7f4388104070 2024-04-22T23:25:32.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.493+0000 7f438fc22700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4388197070 con 0x7f4388101740 2024-04-22T23:25:32.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.493+0000 7f438d1bd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4388104070 0x7f43881969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:25:32.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.493+0000 7f438d1bd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4388104070 0x7f43881969a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38662/0 (socket says 172.21.15.67:38662) 2024-04-22T23:25:32.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.493+0000 7f438d1bd700 1 -- 172.21.15.67:0/859665741 learned_addr learned my addr 172.21.15.67:0/859665741 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:25:32.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.493+0000 7f438d9be700 1 --2- 172.21.15.67:0/859665741 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4388101740 0x7f4388196460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:25:32.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.493+0000 7f438d1bd700 1 -- 172.21.15.67:0/859665741 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4388101740 msgr2=0x7f4388196460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:25:32.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.493+0000 7f438d1bd700 1 --2- 172.21.15.67:0/859665741 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4388101740 0x7f4388196460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:32.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.493+0000 7f438d1bd700 1 -- 172.21.15.67:0/859665741 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43780095f0 con 0x7f4388104070 2024-04-22T23:25:32.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.493+0000 7f438d1bd700 1 --2- 172.21.15.67:0/859665741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4388104070 0x7f43881969a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f4378006010 tx=0x7f437800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:25:32.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.493+0000 7f437effd700 1 -- 172.21.15.67:0/859665741 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f437801d070 con 0x7f4388104070 2024-04-22T23:25:32.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.494+0000 7f437effd700 1 -- 172.21.15.67:0/859665741 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f437800f460 con 0x7f4388104070 2024-04-22T23:25:32.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.494+0000 7f437effd700 1 -- 172.21.15.67:0/859665741 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4378021640 con 0x7f4388104070 2024-04-22T23:25:32.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.494+0000 7f438fc22700 1 -- 172.21.15.67:0/859665741 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f438819bad0 con 0x7f4388104070 2024-04-22T23:25:32.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.494+0000 7f438fc22700 1 -- 172.21.15.67:0/859665741 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f438819bf40 con 0x7f4388104070 2024-04-22T23:25:32.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.495+0000 7f438fc22700 1 -- 172.21.15.67:0/859665741 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4388190620 con 0x7f4388104070 2024-04-22T23:25:32.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.495+0000 7f437effd700 1 -- 172.21.15.67:0/859665741 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 32) v1 ==== 92700+0+0 (secure 0 0 0) 0x7f437800fab0 con 0x7f4388104070 2024-04-22T23:25:32.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.496+0000 7f437effd700 1 --2- 172.21.15.67:0/859665741 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f437406ea80 0x7f4374070f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:25:32.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.496+0000 7f437effd700 1 -- 172.21.15.67:0/859665741 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f437808fef0 con 0x7f4388104070 2024-04-22T23:25:32.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.496+0000 7f438d9be700 1 --2- 172.21.15.67:0/859665741 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f437406ea80 0x7f4374070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:25:32.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.496+0000 7f438d9be700 1 --2- 172.21.15.67:0/859665741 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f437406ea80 0x7f4374070f40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4384005ea0 tx=0x7f4384005e30 comp rx=0 tx=0).ready entity=mgr.14708 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:25:32.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.500+0000 7f437effd700 1 -- 172.21.15.67:0/859665741 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f437805cb10 con 0x7f4388104070 2024-04-22T23:25:32.704 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:32 smithi067 ceph-mon[26453]: mgrmap e32: smithi067.whdcsr(active, since 30s), standbys: smithi134.ccxkcj 2024-04-22T23:25:32.704 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:32 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:32.704 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:32 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:32.704 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:32 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:25:32.705 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:32 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:25:32.705 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:32 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:32.705 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:32 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:25:32.705 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:32 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:25:32.705 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:32 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:32.705 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:32 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi067.whdcsr"}]: dispatch 2024-04-22T23:25:32.705 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:32 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi067.whdcsr"}]': finished 2024-04-22T23:25:32.705 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:32 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi134.ccxkcj"}]: dispatch 2024-04-22T23:25:32.705 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:32 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi134.ccxkcj"}]': finished 2024-04-22T23:25:32.705 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:32 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:25:32.705 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:32 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:32.705 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:32 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:25:32.705 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:32 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:32.705 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:32 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:25:32.705 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:32 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:32.705 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:32 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:25:32.705 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:32 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:32.706 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:32 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:25:32.706 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:32 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:32.706 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:32 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:25:32.706 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:32 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:32.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:32 smithi134 ceph-mon[34609]: mgrmap e32: smithi067.whdcsr(active, since 30s), standbys: smithi134.ccxkcj 2024-04-22T23:25:32.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:32 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:32.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:32 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:32.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:32 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:25:32.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:32 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:25:32.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:32 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:32.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:32 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:25:32.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:32 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:25:32.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:32 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:32.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:32 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi067.whdcsr"}]: dispatch 2024-04-22T23:25:32.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:32 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi067.whdcsr"}]': finished 2024-04-22T23:25:32.746 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:32 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi134.ccxkcj"}]: dispatch 2024-04-22T23:25:32.746 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:32 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi134.ccxkcj"}]': finished 2024-04-22T23:25:32.746 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:32 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:25:32.746 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:32 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:32.746 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:32 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:25:32.746 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:32 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:32.746 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:32 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:25:32.746 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:32 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:32.746 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:32 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:25:32.746 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:32 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:32.746 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:32 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:25:32.746 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:32 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:32.746 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:32 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:25:32.747 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:32 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:32.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.744+0000 7f438fc22700 1 -- 172.21.15.67:0/859665741 --> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f43880611d0 con 0x7f437406ea80 2024-04-22T23:25:32.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.746+0000 7f437effd700 1 -- 172.21.15.67:0/859665741 <== mgr.14708 v2:172.21.15.67:6800/2306276228 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+370 (secure 0 0 0) 0x7f43880611d0 con 0x7f437406ea80 2024-04-22T23:25:32.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.748+0000 7f438fc22700 1 -- 172.21.15.67:0/859665741 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f437406ea80 msgr2=0x7f4374070f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:25:32.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.748+0000 7f438fc22700 1 --2- 172.21.15.67:0/859665741 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f437406ea80 0x7f4374070f40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4384005ea0 tx=0x7f4384005e30 comp rx=0 tx=0).stop 2024-04-22T23:25:32.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.748+0000 7f438fc22700 1 -- 172.21.15.67:0/859665741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4388104070 msgr2=0x7f43881969a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:25:32.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.748+0000 7f438fc22700 1 --2- 172.21.15.67:0/859665741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4388104070 0x7f43881969a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f4378006010 tx=0x7f437800be20 comp rx=0 tx=0).stop 2024-04-22T23:25:32.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.748+0000 7f438fc22700 1 -- 172.21.15.67:0/859665741 shutdown_connections 2024-04-22T23:25:32.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.748+0000 7f438fc22700 1 --2- 172.21.15.67:0/859665741 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f437406ea80 0x7f4374070f40 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:32.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.748+0000 7f438fc22700 1 --2- 172.21.15.67:0/859665741 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4388101740 0x7f4388196460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:32.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.748+0000 7f438fc22700 1 --2- 172.21.15.67:0/859665741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4388104070 0x7f43881969a0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:32.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.748+0000 7f438fc22700 1 -- 172.21.15.67:0/859665741 >> 172.21.15.67:0/859665741 conn(0x7f43880fb150 msgr2=0x7f43880fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:25:32.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.748+0000 7f438fc22700 1 -- 172.21.15.67:0/859665741 shutdown_connections 2024-04-22T23:25:32.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.748+0000 7f438fc22700 1 -- 172.21.15.67:0/859665741 wait complete. 2024-04-22T23:25:32.760 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:25:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.860+0000 7f7d14986700 1 -- 172.21.15.67:0/4219479204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d100ffc10 msgr2=0x7f7d10100030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:25:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.860+0000 7f7d14986700 1 --2- 172.21.15.67:0/4219479204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d100ffc10 0x7f7d10100030 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f7d00009910 tx=0x7f7d00009c20 comp rx=0 tx=0).stop 2024-04-22T23:25:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.860+0000 7f7d14986700 1 -- 172.21.15.67:0/4219479204 shutdown_connections 2024-04-22T23:25:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.860+0000 7f7d14986700 1 --2- 172.21.15.67:0/4219479204 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d10100e10 0x7f7d10101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.860+0000 7f7d14986700 1 --2- 172.21.15.67:0/4219479204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d100ffc10 0x7f7d10100030 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.860+0000 7f7d14986700 1 -- 172.21.15.67:0/4219479204 >> 172.21.15.67:0/4219479204 conn(0x7f7d100fb190 msgr2=0x7f7d100fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:25:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.860+0000 7f7d14986700 1 -- 172.21.15.67:0/4219479204 shutdown_connections 2024-04-22T23:25:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.860+0000 7f7d14986700 1 -- 172.21.15.67:0/4219479204 wait complete. 2024-04-22T23:25:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.861+0000 7f7d14986700 1 Processor -- start 2024-04-22T23:25:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.861+0000 7f7d14986700 1 -- start start 2024-04-22T23:25:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.861+0000 7f7d14986700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d100ffc10 0x7f7d10103ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:25:32.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.861+0000 7f7d14986700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d10100e10 0x7f7d101021f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:25:32.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.861+0000 7f7d14986700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d10102730 con 0x7f7d100ffc10 2024-04-22T23:25:32.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.861+0000 7f7d14986700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d10102870 con 0x7f7d10100e10 2024-04-22T23:25:32.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.861+0000 7f7d0e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d100ffc10 0x7f7d10103ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:25:32.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.861+0000 7f7d0e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d100ffc10 0x7f7d10103ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38674/0 (socket says 172.21.15.67:38674) 2024-04-22T23:25:32.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.861+0000 7f7d0e59c700 1 -- 172.21.15.67:0/2505783495 learned_addr learned my addr 172.21.15.67:0/2505783495 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:25:32.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.861+0000 7f7d0dd9b700 1 --2- 172.21.15.67:0/2505783495 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d10100e10 0x7f7d101021f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:25:32.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.861+0000 7f7d0e59c700 1 -- 172.21.15.67:0/2505783495 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d10100e10 msgr2=0x7f7d101021f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:25:32.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.861+0000 7f7d0e59c700 1 --2- 172.21.15.67:0/2505783495 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d10100e10 0x7f7d101021f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:32.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.861+0000 7f7d0e59c700 1 -- 172.21.15.67:0/2505783495 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d000095f0 con 0x7f7d100ffc10 2024-04-22T23:25:32.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.862+0000 7f7d0e59c700 1 --2- 172.21.15.67:0/2505783495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d100ffc10 0x7f7d10103ba0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f7d0000fa00 tx=0x7f7d0000fa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:25:32.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.862+0000 7f7cff7fe700 1 -- 172.21.15.67:0/2505783495 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d0001d070 con 0x7f7d100ffc10 2024-04-22T23:25:32.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.862+0000 7f7cff7fe700 1 -- 172.21.15.67:0/2505783495 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f7d0000fe90 con 0x7f7d100ffc10 2024-04-22T23:25:32.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.862+0000 7f7d14986700 1 -- 172.21.15.67:0/2505783495 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d10102af0 con 0x7f7d100ffc10 2024-04-22T23:25:32.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.862+0000 7f7cff7fe700 1 -- 172.21.15.67:0/2505783495 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d00017630 con 0x7f7d100ffc10 2024-04-22T23:25:32.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.862+0000 7f7d14986700 1 -- 172.21.15.67:0/2505783495 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d10102fe0 con 0x7f7d100ffc10 2024-04-22T23:25:32.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.863+0000 7f7d14986700 1 -- 172.21.15.67:0/2505783495 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d10108700 con 0x7f7d100ffc10 2024-04-22T23:25:32.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.864+0000 7f7cff7fe700 1 -- 172.21.15.67:0/2505783495 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 32) v1 ==== 92700+0+0 (secure 0 0 0) 0x7f7d00003680 con 0x7f7d100ffc10 2024-04-22T23:25:32.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.864+0000 7f7cff7fe700 1 --2- 172.21.15.67:0/2505783495 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f7cf806ea80 0x7f7cf8070f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:25:32.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.864+0000 7f7cff7fe700 1 -- 172.21.15.67:0/2505783495 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f7d0008fe10 con 0x7f7d100ffc10 2024-04-22T23:25:32.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.864+0000 7f7d0dd9b700 1 --2- 172.21.15.67:0/2505783495 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f7cf806ea80 0x7f7cf8070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:25:32.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.865+0000 7f7d0dd9b700 1 --2- 172.21.15.67:0/2505783495 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f7cf806ea80 0x7f7cf8070f40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7d04009c60 tx=0x7f7d04009260 comp rx=0 tx=0).ready entity=mgr.14708 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:25:32.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:32.869+0000 7f7cff7fe700 1 -- 172.21.15.67:0/2505783495 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f7d0005ca30 con 0x7f7d100ffc10 2024-04-22T23:25:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.103+0000 7f7d14986700 1 -- 172.21.15.67:0/2505783495 --> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7d10068c90 con 0x7f7cf806ea80 2024-04-22T23:25:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.105+0000 7f7cff7fe700 1 -- 172.21.15.67:0/2505783495 <== mgr.14708 v2:172.21.15.67:6800/2306276228 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+370 (secure 0 0 0) 0x7f7d10068c90 con 0x7f7cf806ea80 2024-04-22T23:25:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.107+0000 7f7d14986700 1 -- 172.21.15.67:0/2505783495 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f7cf806ea80 msgr2=0x7f7cf8070f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:25:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.107+0000 7f7d14986700 1 --2- 172.21.15.67:0/2505783495 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f7cf806ea80 0x7f7cf8070f40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7d04009c60 tx=0x7f7d04009260 comp rx=0 tx=0).stop 2024-04-22T23:25:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.107+0000 7f7d14986700 1 -- 172.21.15.67:0/2505783495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d100ffc10 msgr2=0x7f7d10103ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:25:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.107+0000 7f7d14986700 1 --2- 172.21.15.67:0/2505783495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d100ffc10 0x7f7d10103ba0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f7d0000fa00 tx=0x7f7d0000fa30 comp rx=0 tx=0).stop 2024-04-22T23:25:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.107+0000 7f7d14986700 1 -- 172.21.15.67:0/2505783495 shutdown_connections 2024-04-22T23:25:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.107+0000 7f7d14986700 1 --2- 172.21.15.67:0/2505783495 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f7cf806ea80 0x7f7cf8070f40 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.107+0000 7f7d14986700 1 --2- 172.21.15.67:0/2505783495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d100ffc10 0x7f7d10103ba0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.107+0000 7f7d14986700 1 --2- 172.21.15.67:0/2505783495 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d10100e10 0x7f7d101021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.107+0000 7f7d14986700 1 -- 172.21.15.67:0/2505783495 >> 172.21.15.67:0/2505783495 conn(0x7f7d100fb190 msgr2=0x7f7d10068580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:25:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.107+0000 7f7d14986700 1 -- 172.21.15.67:0/2505783495 shutdown_connections 2024-04-22T23:25:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.107+0000 7f7d14986700 1 -- 172.21.15.67:0/2505783495 wait complete. 2024-04-22T23:25:33.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.213+0000 7febfd6c3700 1 -- 172.21.15.67:0/1557471805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febf81017b0 msgr2=0x7febf8103ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:25:33.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.213+0000 7febfd6c3700 1 --2- 172.21.15.67:0/1557471805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febf81017b0 0x7febf8103ba0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7febe8009910 tx=0x7febe8009c20 comp rx=0 tx=0).stop 2024-04-22T23:25:33.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.213+0000 7febfd6c3700 1 -- 172.21.15.67:0/1557471805 shutdown_connections 2024-04-22T23:25:33.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.213+0000 7febfd6c3700 1 --2- 172.21.15.67:0/1557471805 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7febf81040e0 0x7febf81064d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:33.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.213+0000 7febfd6c3700 1 --2- 172.21.15.67:0/1557471805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febf81017b0 0x7febf8103ba0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:33.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.214+0000 7febfd6c3700 1 -- 172.21.15.67:0/1557471805 >> 172.21.15.67:0/1557471805 conn(0x7febf80fb160 msgr2=0x7febf80fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:25:33.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.214+0000 7febfd6c3700 1 -- 172.21.15.67:0/1557471805 shutdown_connections 2024-04-22T23:25:33.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.214+0000 7febfd6c3700 1 -- 172.21.15.67:0/1557471805 wait complete. 2024-04-22T23:25:33.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.214+0000 7febfd6c3700 1 Processor -- start 2024-04-22T23:25:33.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.214+0000 7febfd6c3700 1 -- start start 2024-04-22T23:25:33.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.215+0000 7febfd6c3700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7febf81017b0 0x7febf81967a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:25:33.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.215+0000 7febfd6c3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febf81040e0 0x7febf8196ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:25:33.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.215+0000 7febfd6c3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf8197300 con 0x7febf81040e0 2024-04-22T23:25:33.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.215+0000 7febfd6c3700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf8197440 con 0x7febf81017b0 2024-04-22T23:25:33.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.215+0000 7febf67fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febf81040e0 0x7febf8196ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:25:33.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.215+0000 7febf67fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febf81040e0 0x7febf8196ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38686/0 (socket says 172.21.15.67:38686) 2024-04-22T23:25:33.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.215+0000 7febf67fc700 1 -- 172.21.15.67:0/1583699844 learned_addr learned my addr 172.21.15.67:0/1583699844 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:25:33.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.215+0000 7febf6ffd700 1 --2- 172.21.15.67:0/1583699844 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7febf81017b0 0x7febf81967a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:25:33.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.215+0000 7febf67fc700 1 -- 172.21.15.67:0/1583699844 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7febf81017b0 msgr2=0x7febf81967a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:25:33.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.215+0000 7febf67fc700 1 --2- 172.21.15.67:0/1583699844 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7febf81017b0 0x7febf81967a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:33.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.215+0000 7febf67fc700 1 -- 172.21.15.67:0/1583699844 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe80095f0 con 0x7febf81040e0 2024-04-22T23:25:33.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.215+0000 7febf67fc700 1 --2- 172.21.15.67:0/1583699844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febf81040e0 0x7febf8196ce0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7febec00d8d0 tx=0x7febec00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:25:33.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.215+0000 7febdffff700 1 -- 172.21.15.67:0/1583699844 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7febec0098a0 con 0x7febf81040e0 2024-04-22T23:25:33.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.215+0000 7febdffff700 1 -- 172.21.15.67:0/1583699844 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7febec010460 con 0x7febf81040e0 2024-04-22T23:25:33.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.215+0000 7febfd6c3700 1 -- 172.21.15.67:0/1583699844 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febf819bef0 con 0x7febf81040e0 2024-04-22T23:25:33.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.216+0000 7febdffff700 1 -- 172.21.15.67:0/1583699844 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7febec00f5f0 con 0x7febf81040e0 2024-04-22T23:25:33.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.216+0000 7febfd6c3700 1 -- 172.21.15.67:0/1583699844 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febf819c410 con 0x7febf81040e0 2024-04-22T23:25:33.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.217+0000 7febfd6c3700 1 -- 172.21.15.67:0/1583699844 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febf8190630 con 0x7febf81040e0 2024-04-22T23:25:33.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.217+0000 7febdffff700 1 -- 172.21.15.67:0/1583699844 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 32) v1 ==== 92700+0+0 (secure 0 0 0) 0x7febec009a00 con 0x7febf81040e0 2024-04-22T23:25:33.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.218+0000 7febdffff700 1 --2- 172.21.15.67:0/1583699844 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7febe006ea80 0x7febe0070f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:25:33.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.218+0000 7febdffff700 1 -- 172.21.15.67:0/1583699844 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7febec08e930 con 0x7febf81040e0 2024-04-22T23:25:33.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.218+0000 7febf6ffd700 1 --2- 172.21.15.67:0/1583699844 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7febe006ea80 0x7febe0070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:25:33.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.218+0000 7febf6ffd700 1 --2- 172.21.15.67:0/1583699844 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7febe006ea80 0x7febe0070f40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7febe8006010 tx=0x7febe800b540 comp rx=0 tx=0).ready entity=mgr.14708 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:25:33.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.222+0000 7febdffff700 1 -- 172.21.15.67:0/1583699844 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7febec05b550 con 0x7febf81040e0 2024-04-22T23:25:33.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.445+0000 7febfd6c3700 1 -- 172.21.15.67:0/1583699844 --> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7febf802d050 con 0x7febe006ea80 2024-04-22T23:25:33.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.455+0000 7febdffff700 1 -- 172.21.15.67:0/1583699844 <== mgr.14708 v2:172.21.15.67:6800/2306276228 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7febf802d050 con 0x7febe006ea80 2024-04-22T23:25:33.457 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:25:33.457 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (9m) 4s ago 10m 26.1M - 0.25.0 c8568f914cd2 8c7b7ec00657 2024-04-22T23:25:33.457 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (10m) 4s ago 10m 7948k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:25:33.458 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (9m) 6s ago 9m 7910k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:25:33.458 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (10m) 4s ago 10m 6660k - 18.2.1 5be31c24972a 14085634c5d3 2024-04-22T23:25:33.458 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (9m) 6s ago 9m 6665k - 18.2.1 5be31c24972a fa995d458353 2024-04-22T23:25:33.458 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (8m) 4s ago 10m 86.9M - 9.4.7 954c08fa6188 0d8950c9af0c 2024-04-22T23:25:33.458 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (5m) 4s ago 5m 62.6M - 18.2.1 5be31c24972a e24a471f0a8c 2024-04-22T23:25:33.458 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (5m) 4s ago 5m 130M - 18.2.1 5be31c24972a 0730f9fc8e61 2024-04-22T23:25:33.458 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (5m) 6s ago 5m 123M - 18.2.1 5be31c24972a b6a169586433 2024-04-22T23:25:33.458 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (4m) 6s ago 4m 67.5M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:25:33.459 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (56s) 4s ago 11m 493M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:25:33.459 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (10s) 6s ago 9m 66.0M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:25:33.459 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (12m) 4s ago 12m 55.8M 2048M 18.2.1 5be31c24972a 6789283703e0 2024-04-22T23:25:33.459 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (9m) 6s ago 9m 44.6M 2048M 18.2.1 5be31c24972a 2fb23795f7e3 2024-04-22T23:25:33.459 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (10m) 4s ago 10m 15.9M - 1.5.0 0da6a335fe13 4aa4da7089d0 2024-04-22T23:25:33.459 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (9m) 6s ago 9m 15.8M - 1.5.0 0da6a335fe13 26e43c2e756c 2024-04-22T23:25:33.460 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (8m) 4s ago 8m 203M 1216M 18.2.1 5be31c24972a a033247ee8c2 2024-04-22T23:25:33.460 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (8m) 4s ago 8m 209M 1216M 18.2.1 5be31c24972a 915b07a8edc4 2024-04-22T23:25:33.460 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (7m) 4s ago 7m 170M 1216M 18.2.1 5be31c24972a e257ecb4efa1 2024-04-22T23:25:33.460 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (7m) 6s ago 7m 248M 2240M 18.2.1 5be31c24972a 9c2c91a967ba 2024-04-22T23:25:33.460 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (7m) 6s ago 7m 217M 2240M 18.2.1 5be31c24972a c7586562ffa6 2024-04-22T23:25:33.460 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (6m) 6s ago 6m 192M 2240M 18.2.1 5be31c24972a 9fd97f1c65cb 2024-04-22T23:25:33.460 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (15s) 4s ago 10m 48.4M - 2.43.0 a07b618ecd1d e419d3fd9045 2024-04-22T23:25:33.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.458+0000 7febfd6c3700 1 -- 172.21.15.67:0/1583699844 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7febe006ea80 msgr2=0x7febe0070f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:25:33.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.458+0000 7febfd6c3700 1 --2- 172.21.15.67:0/1583699844 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7febe006ea80 0x7febe0070f40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7febe8006010 tx=0x7febe800b540 comp rx=0 tx=0).stop 2024-04-22T23:25:33.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.458+0000 7febfd6c3700 1 -- 172.21.15.67:0/1583699844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febf81040e0 msgr2=0x7febf8196ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:25:33.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.458+0000 7febfd6c3700 1 --2- 172.21.15.67:0/1583699844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febf81040e0 0x7febf8196ce0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7febec00d8d0 tx=0x7febec00dbe0 comp rx=0 tx=0).stop 2024-04-22T23:25:33.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.458+0000 7febfd6c3700 1 -- 172.21.15.67:0/1583699844 shutdown_connections 2024-04-22T23:25:33.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.458+0000 7febfd6c3700 1 --2- 172.21.15.67:0/1583699844 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7febe006ea80 0x7febe0070f40 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:33.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.458+0000 7febfd6c3700 1 --2- 172.21.15.67:0/1583699844 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7febf81017b0 0x7febf81967a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:33.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.458+0000 7febfd6c3700 1 --2- 172.21.15.67:0/1583699844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febf81040e0 0x7febf8196ce0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:33.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.458+0000 7febfd6c3700 1 -- 172.21.15.67:0/1583699844 >> 172.21.15.67:0/1583699844 conn(0x7febf80fb160 msgr2=0x7febf80fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:25:33.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.458+0000 7febfd6c3700 1 -- 172.21.15.67:0/1583699844 shutdown_connections 2024-04-22T23:25:33.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.459+0000 7febfd6c3700 1 -- 172.21.15.67:0/1583699844 wait complete. 2024-04-22T23:25:33.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.566+0000 7f67722fa700 1 -- 172.21.15.67:0/2199932873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f676c0ff5c0 msgr2=0x7f676c0ff9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:25:33.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.566+0000 7f67722fa700 1 --2- 172.21.15.67:0/2199932873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f676c0ff5c0 0x7f676c0ff9e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f675c009910 tx=0x7f675c009c20 comp rx=0 tx=0).stop 2024-04-22T23:25:33.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.566+0000 7f67722fa700 1 -- 172.21.15.67:0/2199932873 shutdown_connections 2024-04-22T23:25:33.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.566+0000 7f67722fa700 1 --2- 172.21.15.67:0/2199932873 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f676c0fff20 0x7f676c1003a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:33.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.566+0000 7f67722fa700 1 --2- 172.21.15.67:0/2199932873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f676c0ff5c0 0x7f676c0ff9e0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:33.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.566+0000 7f67722fa700 1 -- 172.21.15.67:0/2199932873 >> 172.21.15.67:0/2199932873 conn(0x7f676c0fb160 msgr2=0x7f676c0fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:25:33.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.566+0000 7f67722fa700 1 -- 172.21.15.67:0/2199932873 shutdown_connections 2024-04-22T23:25:33.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.566+0000 7f67722fa700 1 -- 172.21.15.67:0/2199932873 wait complete. 2024-04-22T23:25:33.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.566+0000 7f67722fa700 1 Processor -- start 2024-04-22T23:25:33.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.566+0000 7f67722fa700 1 -- start start 2024-04-22T23:25:33.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.567+0000 7f67722fa700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f676c0ff5c0 0x7f676c1989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:25:33.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.567+0000 7f67722fa700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f676c0fff20 0x7f676c198f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:25:33.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.567+0000 7f67722fa700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676c1994b0 con 0x7f676c0fff20 2024-04-22T23:25:33.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.567+0000 7f67722fa700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676c1995f0 con 0x7f676c0ff5c0 2024-04-22T23:25:33.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.567+0000 7f676b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f676c0fff20 0x7f676c198f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:25:33.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.567+0000 7f676b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f676c0fff20 0x7f676c198f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38712/0 (socket says 172.21.15.67:38712) 2024-04-22T23:25:33.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.567+0000 7f676b7fe700 1 -- 172.21.15.67:0/45897865 learned_addr learned my addr 172.21.15.67:0/45897865 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:25:33.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.567+0000 7f676bfff700 1 --2- 172.21.15.67:0/45897865 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f676c0ff5c0 0x7f676c1989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:25:33.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.567+0000 7f676b7fe700 1 -- 172.21.15.67:0/45897865 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f676c0ff5c0 msgr2=0x7f676c1989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:25:33.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.567+0000 7f676b7fe700 1 --2- 172.21.15.67:0/45897865 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f676c0ff5c0 0x7f676c1989e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:33.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.567+0000 7f676b7fe700 1 -- 172.21.15.67:0/45897865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f675c0095f0 con 0x7f676c0fff20 2024-04-22T23:25:33.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.568+0000 7f676b7fe700 1 --2- 172.21.15.67:0/45897865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f676c0fff20 0x7f676c198f20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f676000c930 tx=0x7f676000ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:25:33.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.568+0000 7f67697fa700 1 -- 172.21.15.67:0/45897865 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6760007ad0 con 0x7f676c0fff20 2024-04-22T23:25:33.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.568+0000 7f67697fa700 1 -- 172.21.15.67:0/45897865 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f6760007c30 con 0x7f676c0fff20 2024-04-22T23:25:33.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.568+0000 7f67722fa700 1 -- 172.21.15.67:0/45897865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f676c19e0b0 con 0x7f676c0fff20 2024-04-22T23:25:33.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.568+0000 7f67697fa700 1 -- 172.21.15.67:0/45897865 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6760018630 con 0x7f676c0fff20 2024-04-22T23:25:33.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.568+0000 7f67722fa700 1 -- 172.21.15.67:0/45897865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f676c19e5d0 con 0x7f676c0fff20 2024-04-22T23:25:33.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.569+0000 7f67722fa700 1 -- 172.21.15.67:0/45897865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f676c192830 con 0x7f676c0fff20 2024-04-22T23:25:33.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.569+0000 7f67697fa700 1 -- 172.21.15.67:0/45897865 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 32) v1 ==== 92700+0+0 (secure 0 0 0) 0x7f676000f450 con 0x7f676c0fff20 2024-04-22T23:25:33.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.570+0000 7f67697fa700 1 --2- 172.21.15.67:0/45897865 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f675406ea80 0x7f6754070f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:25:33.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.570+0000 7f67697fa700 1 -- 172.21.15.67:0/45897865 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f676008edb0 con 0x7f676c0fff20 2024-04-22T23:25:33.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.570+0000 7f676bfff700 1 --2- 172.21.15.67:0/45897865 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f675406ea80 0x7f6754070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:25:33.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.570+0000 7f676bfff700 1 --2- 172.21.15.67:0/45897865 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f675406ea80 0x7f6754070f40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f675c006010 tx=0x7f675c00b580 comp rx=0 tx=0).ready entity=mgr.14708 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:25:33.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.574+0000 7f67697fa700 1 -- 172.21.15.67:0/45897865 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f676005b950 con 0x7f676c0fff20 2024-04-22T23:25:33.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:33 smithi134 ceph-mon[34609]: Upgrade: Setting container_image for all mgr 2024-04-22T23:25:33.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:33 smithi134 ceph-mon[34609]: Upgrade: Setting container_image for all rgw 2024-04-22T23:25:33.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:33 smithi134 ceph-mon[34609]: Upgrade: Setting container_image for all rbd-mirror 2024-04-22T23:25:33.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:33 smithi134 ceph-mon[34609]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-22T23:25:33.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:33 smithi134 ceph-mon[34609]: Upgrade: Setting container_image for all iscsi 2024-04-22T23:25:33.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:33 smithi134 ceph-mon[34609]: Upgrade: Setting container_image for all nfs 2024-04-22T23:25:33.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:33 smithi134 ceph-mon[34609]: Upgrade: Setting container_image for all nvmeof 2024-04-22T23:25:33.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:33 smithi134 ceph-mon[34609]: pgmap v19: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-04-22T23:25:33.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:33 smithi134 ceph-mon[34609]: from='client.14762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:25:33.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:33 smithi067 ceph-mon[26453]: Upgrade: Setting container_image for all mgr 2024-04-22T23:25:33.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:33 smithi067 ceph-mon[26453]: Upgrade: Setting container_image for all rgw 2024-04-22T23:25:33.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:33 smithi067 ceph-mon[26453]: Upgrade: Setting container_image for all rbd-mirror 2024-04-22T23:25:33.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:33 smithi067 ceph-mon[26453]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-22T23:25:33.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:33 smithi067 ceph-mon[26453]: Upgrade: Setting container_image for all iscsi 2024-04-22T23:25:33.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:33 smithi067 ceph-mon[26453]: Upgrade: Setting container_image for all nfs 2024-04-22T23:25:33.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:33 smithi067 ceph-mon[26453]: Upgrade: Setting container_image for all nvmeof 2024-04-22T23:25:33.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:33 smithi067 ceph-mon[26453]: pgmap v19: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-04-22T23:25:33.766 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:33 smithi067 ceph-mon[26453]: from='client.14762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:25:33.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.880+0000 7f67722fa700 1 -- 172.21.15.67:0/45897865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f676c068af0 con 0x7f676c0fff20 2024-04-22T23:25:33.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.880+0000 7f67697fa700 1 -- 172.21.15.67:0/45897865 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7f676005b4e0 con 0x7f676c0fff20 2024-04-22T23:25:33.882 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:25:33.882 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:25:33.882 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-22T23:25:33.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:25:33.882 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:25:33.882 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:25:33.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:25:33.883 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:25:33.883 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-22T23:25:33.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:25:33.883 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:25:33.883 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-22T23:25:33.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:25:33.883 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:25:33.883 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 12, 2024-04-22T23:25:33.883 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:25:33.883 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:25:33.883 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:25:33.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.882+0000 7f67722fa700 1 -- 172.21.15.67:0/45897865 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f675406ea80 msgr2=0x7f6754070f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:25:33.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.882+0000 7f67722fa700 1 --2- 172.21.15.67:0/45897865 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f675406ea80 0x7f6754070f40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f675c006010 tx=0x7f675c00b580 comp rx=0 tx=0).stop 2024-04-22T23:25:33.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.882+0000 7f67722fa700 1 -- 172.21.15.67:0/45897865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f676c0fff20 msgr2=0x7f676c198f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:25:33.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.882+0000 7f67722fa700 1 --2- 172.21.15.67:0/45897865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f676c0fff20 0x7f676c198f20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f676000c930 tx=0x7f676000ccf0 comp rx=0 tx=0).stop 2024-04-22T23:25:33.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.883+0000 7f67722fa700 1 -- 172.21.15.67:0/45897865 shutdown_connections 2024-04-22T23:25:33.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.883+0000 7f67722fa700 1 --2- 172.21.15.67:0/45897865 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f675406ea80 0x7f6754070f40 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:33.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.883+0000 7f67722fa700 1 --2- 172.21.15.67:0/45897865 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f676c0ff5c0 0x7f676c1989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:33.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.883+0000 7f67722fa700 1 --2- 172.21.15.67:0/45897865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f676c0fff20 0x7f676c198f20 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:33.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.883+0000 7f67722fa700 1 -- 172.21.15.67:0/45897865 >> 172.21.15.67:0/45897865 conn(0x7f676c0fb160 msgr2=0x7f676c107cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:25:33.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.883+0000 7f67722fa700 1 -- 172.21.15.67:0/45897865 shutdown_connections 2024-04-22T23:25:33.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.883+0000 7f67722fa700 1 -- 172.21.15.67:0/45897865 wait complete. 2024-04-22T23:25:33.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.993+0000 7f4698c99700 1 -- 172.21.15.67:0/1621984472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4694102000 msgr2=0x7f4694102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:25:33.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.994+0000 7f4698c99700 1 --2- 172.21.15.67:0/1621984472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4694102000 0x7f4694102460 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f4688009910 tx=0x7f4688009c20 comp rx=0 tx=0).stop 2024-04-22T23:25:33.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.994+0000 7f4698c99700 1 -- 172.21.15.67:0/1621984472 shutdown_connections 2024-04-22T23:25:33.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.994+0000 7f4698c99700 1 --2- 172.21.15.67:0/1621984472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4694102000 0x7f4694102460 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:33.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.994+0000 7f4698c99700 1 --2- 172.21.15.67:0/1621984472 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46940fef80 0x7f46940ff3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:33.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.994+0000 7f4698c99700 1 -- 172.21.15.67:0/1621984472 >> 172.21.15.67:0/1621984472 conn(0x7f46940fa8c0 msgr2=0x7f46940fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:25:33.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.994+0000 7f4698c99700 1 -- 172.21.15.67:0/1621984472 shutdown_connections 2024-04-22T23:25:33.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.994+0000 7f4698c99700 1 -- 172.21.15.67:0/1621984472 wait complete. 2024-04-22T23:25:33.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.995+0000 7f4698c99700 1 Processor -- start 2024-04-22T23:25:33.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.995+0000 7f4698c99700 1 -- start start 2024-04-22T23:25:33.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.995+0000 7f4698c99700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46940fef80 0x7f46941964a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:25:33.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.995+0000 7f4698c99700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4694102000 0x7f46941969e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:25:33.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.995+0000 7f4698c99700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4694197000 con 0x7f4694102000 2024-04-22T23:25:33.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.995+0000 7f4698c99700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4694197140 con 0x7f46940fef80 2024-04-22T23:25:33.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.995+0000 7f4691d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4694102000 0x7f46941969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:25:33.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.995+0000 7f4691d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4694102000 0x7f46941969e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38726/0 (socket says 172.21.15.67:38726) 2024-04-22T23:25:33.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.995+0000 7f4691d9b700 1 -- 172.21.15.67:0/445262406 learned_addr learned my addr 172.21.15.67:0/445262406 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:25:33.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.995+0000 7f469259c700 1 --2- 172.21.15.67:0/445262406 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46940fef80 0x7f46941964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:25:33.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.995+0000 7f4691d9b700 1 -- 172.21.15.67:0/445262406 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46940fef80 msgr2=0x7f46941964a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:25:33.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.995+0000 7f4691d9b700 1 --2- 172.21.15.67:0/445262406 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46940fef80 0x7f46941964a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:33.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.995+0000 7f4691d9b700 1 -- 172.21.15.67:0/445262406 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46880095f0 con 0x7f4694102000 2024-04-22T23:25:33.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.995+0000 7f4691d9b700 1 --2- 172.21.15.67:0/445262406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4694102000 0x7f46941969e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f468800bda0 tx=0x7f468800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:25:33.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.996+0000 7f46837fe700 1 -- 172.21.15.67:0/445262406 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f468801d070 con 0x7f4694102000 2024-04-22T23:25:33.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.996+0000 7f4698c99700 1 -- 172.21.15.67:0/445262406 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f469419bb90 con 0x7f4694102000 2024-04-22T23:25:33.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.996+0000 7f46837fe700 1 -- 172.21.15.67:0/445262406 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f468800f460 con 0x7f4694102000 2024-04-22T23:25:33.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.996+0000 7f46837fe700 1 -- 172.21.15.67:0/445262406 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4688021640 con 0x7f4694102000 2024-04-22T23:25:34.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.996+0000 7f4698c99700 1 -- 172.21.15.67:0/445262406 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f469419c080 con 0x7f4694102000 2024-04-22T23:25:34.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.997+0000 7f4698c99700 1 -- 172.21.15.67:0/445262406 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4694109800 con 0x7f4694102000 2024-04-22T23:25:34.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.998+0000 7f46837fe700 1 -- 172.21.15.67:0/445262406 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 32) v1 ==== 92700+0+0 (secure 0 0 0) 0x7f468800fa60 con 0x7f4694102000 2024-04-22T23:25:34.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.998+0000 7f46837fe700 1 --2- 172.21.15.67:0/445262406 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f467c06ea80 0x7f467c070f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:25:34.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.998+0000 7f46837fe700 1 -- 172.21.15.67:0/445262406 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f468808fe80 con 0x7f4694102000 2024-04-22T23:25:34.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.999+0000 7f469259c700 1 --2- 172.21.15.67:0/445262406 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f467c06ea80 0x7f467c070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:25:34.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:33.999+0000 7f469259c700 1 --2- 172.21.15.67:0/445262406 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f467c06ea80 0x7f467c070f40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4684005950 tx=0x7f46840058e0 comp rx=0 tx=0).ready entity=mgr.14708 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:25:34.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:34.003+0000 7f46837fe700 1 -- 172.21.15.67:0/445262406 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f468805ca20 con 0x7f4694102000 2024-04-22T23:25:34.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:34.253+0000 7f4698c99700 1 -- 172.21.15.67:0/445262406 --> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4694106950 con 0x7f467c06ea80 2024-04-22T23:25:34.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:34.254+0000 7f46837fe700 1 -- 172.21.15.67:0/445262406 <== mgr.14708 v2:172.21.15.67:6800/2306276228 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+370 (secure 0 0 0) 0x7f4694106950 con 0x7f467c06ea80 2024-04-22T23:25:34.256 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:25:34.256 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:25:34.256 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:25:34.257 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-22T23:25:34.257 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:25:34.257 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr" 2024-04-22T23:25:34.257 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:25:34.257 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "2/2 daemons upgraded", 2024-04-22T23:25:34.257 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading node-exporter daemons", 2024-04-22T23:25:34.258 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:25:34.258 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:25:34.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:34.257+0000 7f4698c99700 1 -- 172.21.15.67:0/445262406 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f467c06ea80 msgr2=0x7f467c070f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:25:34.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:34.257+0000 7f4698c99700 1 --2- 172.21.15.67:0/445262406 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f467c06ea80 0x7f467c070f40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4684005950 tx=0x7f46840058e0 comp rx=0 tx=0).stop 2024-04-22T23:25:34.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:34.257+0000 7f4698c99700 1 -- 172.21.15.67:0/445262406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4694102000 msgr2=0x7f46941969e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:25:34.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:34.257+0000 7f4698c99700 1 --2- 172.21.15.67:0/445262406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4694102000 0x7f46941969e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f468800bda0 tx=0x7f468800be80 comp rx=0 tx=0).stop 2024-04-22T23:25:34.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:34.257+0000 7f4698c99700 1 -- 172.21.15.67:0/445262406 shutdown_connections 2024-04-22T23:25:34.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:34.257+0000 7f4698c99700 1 --2- 172.21.15.67:0/445262406 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f467c06ea80 0x7f467c070f40 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:34.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:34.257+0000 7f4698c99700 1 --2- 172.21.15.67:0/445262406 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46940fef80 0x7f46941964a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:34.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:34.257+0000 7f4698c99700 1 --2- 172.21.15.67:0/445262406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4694102000 0x7f46941969e0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:25:34.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:34.257+0000 7f4698c99700 1 -- 172.21.15.67:0/445262406 >> 172.21.15.67:0/445262406 conn(0x7f46940fa8c0 msgr2=0x7f4694105230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:25:34.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:34.257+0000 7f4698c99700 1 -- 172.21.15.67:0/445262406 shutdown_connections 2024-04-22T23:25:34.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:25:34.257+0000 7f4698c99700 1 -- 172.21.15.67:0/445262406 wait complete. 2024-04-22T23:25:34.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:34 smithi067 ceph-mon[26453]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:25:34.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:34 smithi067 ceph-mon[26453]: from='client.14770 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:25:34.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:34 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/45897865' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:25:34.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:34 smithi067 ceph-mon[26453]: Upgrade: Updating node-exporter.smithi067 (1/2) 2024-04-22T23:25:34.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:34 smithi067 ceph-mon[26453]: Deploying daemon node-exporter.smithi067 on smithi067 2024-04-22T23:25:34.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:34 smithi134 ceph-mon[34609]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:25:34.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:34 smithi134 ceph-mon[34609]: from='client.14770 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:25:34.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:34 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/45897865' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:25:34.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:34 smithi134 ceph-mon[34609]: Upgrade: Updating node-exporter.smithi067 (1/2) 2024-04-22T23:25:34.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:34 smithi134 ceph-mon[34609]: Deploying daemon node-exporter.smithi067 on smithi067 2024-04-22T23:25:35.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:35 smithi134 ceph-mon[34609]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:25:35.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:35 smithi134 ceph-mon[34609]: pgmap v20: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T23:25:35.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:35 smithi067 ceph-mon[26453]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:25:35.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:35 smithi067 ceph-mon[26453]: pgmap v20: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T23:25:37.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:37 smithi134 ceph-mon[34609]: pgmap v21: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T23:25:37.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:37 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:37.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:37 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:38.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:37 smithi067 ceph-mon[26453]: pgmap v21: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T23:25:38.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:37 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:38.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:37 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:39.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:39 smithi134 ceph-mon[34609]: Upgrade: Updating node-exporter.smithi134 (2/2) 2024-04-22T23:25:39.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:39 smithi134 ceph-mon[34609]: Deploying daemon node-exporter.smithi134 on smithi134 2024-04-22T23:25:39.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:39 smithi134 ceph-mon[34609]: pgmap v22: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-22T23:25:40.026 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:39 smithi067 ceph-mon[26453]: Upgrade: Updating node-exporter.smithi134 (2/2) 2024-04-22T23:25:40.026 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:39 smithi067 ceph-mon[26453]: Deploying daemon node-exporter.smithi134 on smithi134 2024-04-22T23:25:40.026 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:39 smithi067 ceph-mon[26453]: pgmap v22: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-22T23:25:41.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:41 smithi134 ceph-mon[34609]: pgmap v23: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T23:25:41.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:41 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:41.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:41 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:41.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:41 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:25:41.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:41 smithi067 ceph-mon[26453]: pgmap v23: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T23:25:41.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:41 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:41.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:41 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:41.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:41 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:25:42.656 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:42 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:42.656 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:42 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:43.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:42 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:43.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:42 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:43.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:43 smithi134 ceph-mon[34609]: pgmap v24: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-04-22T23:25:43.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:43 smithi067 ceph-mon[26453]: pgmap v24: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-04-22T23:25:45.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:45 smithi067 ceph-mon[26453]: pgmap v25: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T23:25:45.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:45 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:45.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:45 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:45.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:45 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:25:45.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:45 smithi134 ceph-mon[34609]: pgmap v25: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T23:25:45.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:45 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:45.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:45 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:45.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:45 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:25:46.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:46 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:46.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:46 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:46.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:46 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:46.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:46 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:47.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:46 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:47.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:46 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:47.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:46 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:47.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:46 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:47.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:47 smithi134 ceph-mon[34609]: pgmap v26: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T23:25:47.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:47 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:47.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:47 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:48.039 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:47 smithi067 ceph-mon[26453]: pgmap v26: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T23:25:48.039 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:47 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:48.039 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:47 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:49.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:49 smithi134 ceph-mon[34609]: pgmap v27: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-22T23:25:49.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:49 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:49.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:49 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:50.046 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:49 smithi067 ceph-mon[26453]: pgmap v27: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-22T23:25:50.046 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:49 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:50.046 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:49 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:51.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:51 smithi134 ceph-mon[34609]: pgmap v28: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T23:25:51.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:51 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:51.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:51 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:51.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:51 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:25:51.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:51 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:25:51.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:51 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:51.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:51 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:25:51.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:51 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:25:51.746 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:51 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:25:51.746 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:51 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:25:51.746 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:51 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:25:51.746 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:51 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:25:51.746 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:51 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:25:51.746 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:51 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:25:51.746 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:51 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:25:51.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:51 smithi067 ceph-mon[26453]: pgmap v28: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T23:25:51.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:51 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:51.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:51 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:51.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:51 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:25:51.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:51 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:25:51.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:51 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:51.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:51 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:25:51.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:51 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:25:51.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:51 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:25:51.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:51 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:25:51.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:51 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:25:51.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:51 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:25:51.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:51 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:25:51.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:51 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:25:51.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:51 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:25:53.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:53 smithi134 ceph-mon[34609]: pgmap v29: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-04-22T23:25:53.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:53 smithi134 ceph-mon[34609]: Upgrade: Updating prometheus.smithi067 2024-04-22T23:25:53.788 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:53 smithi067 ceph-mon[26453]: pgmap v29: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-04-22T23:25:53.788 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:53 smithi067 ceph-mon[26453]: Upgrade: Updating prometheus.smithi067 2024-04-22T23:25:54.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:54 smithi067 ceph-mon[26453]: Deploying daemon prometheus.smithi067 on smithi067 2024-04-22T23:25:54.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:54 smithi134 ceph-mon[34609]: Deploying daemon prometheus.smithi067 on smithi067 2024-04-22T23:25:55.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:55 smithi067 ceph-mon[26453]: pgmap v30: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T23:25:55.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:55 smithi134 ceph-mon[34609]: pgmap v30: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T23:25:57.798 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:57 smithi067 ceph-mon[26453]: pgmap v31: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T23:25:57.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:57 smithi134 ceph-mon[34609]: pgmap v31: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T23:25:59.828 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:59 smithi067 ceph-mon[26453]: pgmap v32: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-22T23:25:59.828 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:59 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:59.828 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:59 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:25:59.829 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:25:59 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:26:00.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:59 smithi134 ceph-mon[34609]: pgmap v32: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-22T23:26:00.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:59 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:00.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:59 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:00.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:25:59 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:26:01.376 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:01 smithi067 ceph-mon[26453]: pgmap v33: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T23:26:01.376 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:01 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:26:01.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:01 smithi134 ceph-mon[34609]: pgmap v33: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T23:26:01.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:01 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:26:03.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:03 smithi067 ceph-mon[26453]: pgmap v34: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-04-22T23:26:03.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:03 smithi134 ceph-mon[34609]: pgmap v34: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-04-22T23:26:04.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.368+0000 7fde8501e700 1 -- 172.21.15.67:0/4005005530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde80100e10 msgr2=0x7fde80101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:04.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.368+0000 7fde8501e700 1 --2- 172.21.15.67:0/4005005530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde80100e10 0x7fde80101270 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fde74009910 tx=0x7fde74009c20 comp rx=0 tx=0).stop 2024-04-22T23:26:04.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.368+0000 7fde8501e700 1 -- 172.21.15.67:0/4005005530 shutdown_connections 2024-04-22T23:26:04.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.368+0000 7fde8501e700 1 --2- 172.21.15.67:0/4005005530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde80100e10 0x7fde80101270 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:04.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.368+0000 7fde8501e700 1 --2- 172.21.15.67:0/4005005530 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fde800ffc10 0x7fde80100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:04.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.368+0000 7fde8501e700 1 -- 172.21.15.67:0/4005005530 >> 172.21.15.67:0/4005005530 conn(0x7fde800fb190 msgr2=0x7fde800fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:26:04.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.368+0000 7fde8501e700 1 -- 172.21.15.67:0/4005005530 shutdown_connections 2024-04-22T23:26:04.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.368+0000 7fde8501e700 1 -- 172.21.15.67:0/4005005530 wait complete. 2024-04-22T23:26:04.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.369+0000 7fde8501e700 1 Processor -- start 2024-04-22T23:26:04.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.369+0000 7fde8501e700 1 -- start start 2024-04-22T23:26:04.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.370+0000 7fde8501e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde800ffc10 0x7fde800690e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:26:04.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.370+0000 7fde8501e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fde80100e10 0x7fde8019b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:26:04.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.370+0000 7fde8501e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8019b740 con 0x7fde800ffc10 2024-04-22T23:26:04.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.370+0000 7fde8501e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8019b880 con 0x7fde80100e10 2024-04-22T23:26:04.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.370+0000 7fde7ed9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde800ffc10 0x7fde800690e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:26:04.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.370+0000 7fde7ed9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde800ffc10 0x7fde800690e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50958/0 (socket says 172.21.15.67:50958) 2024-04-22T23:26:04.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.370+0000 7fde7ed9d700 1 -- 172.21.15.67:0/3386326743 learned_addr learned my addr 172.21.15.67:0/3386326743 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:26:04.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.370+0000 7fde7e59c700 1 --2- 172.21.15.67:0/3386326743 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fde80100e10 0x7fde8019b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:26:04.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.370+0000 7fde7ed9d700 1 -- 172.21.15.67:0/3386326743 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fde80100e10 msgr2=0x7fde8019b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:04.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.370+0000 7fde7ed9d700 1 --2- 172.21.15.67:0/3386326743 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fde80100e10 0x7fde8019b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:04.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.370+0000 7fde7ed9d700 1 -- 172.21.15.67:0/3386326743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde740095f0 con 0x7fde800ffc10 2024-04-22T23:26:04.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.370+0000 7fde7ed9d700 1 --2- 172.21.15.67:0/3386326743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde800ffc10 0x7fde800690e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fde7000c910 tx=0x7fde7000ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:26:04.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.370+0000 7fde67fff700 1 -- 172.21.15.67:0/3386326743 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde700041f0 con 0x7fde800ffc10 2024-04-22T23:26:04.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.371+0000 7fde67fff700 1 -- 172.21.15.67:0/3386326743 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fde70007db0 con 0x7fde800ffc10 2024-04-22T23:26:04.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.371+0000 7fde8501e700 1 -- 172.21.15.67:0/3386326743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde8019bb60 con 0x7fde800ffc10 2024-04-22T23:26:04.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.371+0000 7fde67fff700 1 -- 172.21.15.67:0/3386326743 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde70003d90 con 0x7fde800ffc10 2024-04-22T23:26:04.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.371+0000 7fde8501e700 1 -- 172.21.15.67:0/3386326743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde8019c0b0 con 0x7fde800ffc10 2024-04-22T23:26:04.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.372+0000 7fde8501e700 1 -- 172.21.15.67:0/3386326743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde80108700 con 0x7fde800ffc10 2024-04-22T23:26:04.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.373+0000 7fde67fff700 1 -- 172.21.15.67:0/3386326743 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 32) v1 ==== 92700+0+0 (secure 0 0 0) 0x7fde70007900 con 0x7fde800ffc10 2024-04-22T23:26:04.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.373+0000 7fde67fff700 1 --2- 172.21.15.67:0/3386326743 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fde6806ea80 0x7fde68070f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:26:04.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.373+0000 7fde67fff700 1 -- 172.21.15.67:0/3386326743 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fde7008e8a0 con 0x7fde800ffc10 2024-04-22T23:26:04.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.373+0000 7fde7e59c700 1 --2- 172.21.15.67:0/3386326743 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fde6806ea80 0x7fde68070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:26:04.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.374+0000 7fde7e59c700 1 --2- 172.21.15.67:0/3386326743 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fde6806ea80 0x7fde68070f40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fde80069660 tx=0x7fde7400b580 comp rx=0 tx=0).ready entity=mgr.14708 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:26:04.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.383+0000 7fde67fff700 1 -- 172.21.15.67:0/3386326743 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7fde7005b440 con 0x7fde800ffc10 2024-04-22T23:26:04.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.614+0000 7fde8501e700 1 -- 172.21.15.67:0/3386326743 --> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fde80105760 con 0x7fde6806ea80 2024-04-22T23:26:04.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.616+0000 7fde67fff700 1 -- 172.21.15.67:0/3386326743 <== mgr.14708 v2:172.21.15.67:6800/2306276228 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7fde80105760 con 0x7fde6806ea80 2024-04-22T23:26:04.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.618+0000 7fde8501e700 1 -- 172.21.15.67:0/3386326743 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fde6806ea80 msgr2=0x7fde68070f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:04.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.618+0000 7fde8501e700 1 --2- 172.21.15.67:0/3386326743 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fde6806ea80 0x7fde68070f40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fde80069660 tx=0x7fde7400b580 comp rx=0 tx=0).stop 2024-04-22T23:26:04.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.618+0000 7fde8501e700 1 -- 172.21.15.67:0/3386326743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde800ffc10 msgr2=0x7fde800690e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:04.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.618+0000 7fde8501e700 1 --2- 172.21.15.67:0/3386326743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde800ffc10 0x7fde800690e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fde7000c910 tx=0x7fde7000ccd0 comp rx=0 tx=0).stop 2024-04-22T23:26:04.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.618+0000 7fde8501e700 1 -- 172.21.15.67:0/3386326743 shutdown_connections 2024-04-22T23:26:04.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.618+0000 7fde8501e700 1 --2- 172.21.15.67:0/3386326743 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fde6806ea80 0x7fde68070f40 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:04.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.618+0000 7fde8501e700 1 --2- 172.21.15.67:0/3386326743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde800ffc10 0x7fde800690e0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:04.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.618+0000 7fde8501e700 1 --2- 172.21.15.67:0/3386326743 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fde80100e10 0x7fde8019b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:04.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.618+0000 7fde8501e700 1 -- 172.21.15.67:0/3386326743 >> 172.21.15.67:0/3386326743 conn(0x7fde800fb190 msgr2=0x7fde80104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:26:04.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.618+0000 7fde8501e700 1 -- 172.21.15.67:0/3386326743 shutdown_connections 2024-04-22T23:26:04.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.618+0000 7fde8501e700 1 -- 172.21.15.67:0/3386326743 wait complete. 2024-04-22T23:26:04.631 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:26:04.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.731+0000 7f4bb6781700 1 -- 172.21.15.67:0/3108657691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bb00ffe00 msgr2=0x7f4bb0100280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:04.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.731+0000 7f4bb6781700 1 --2- 172.21.15.67:0/3108657691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bb00ffe00 0x7f4bb0100280 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4009910 tx=0x7f4ba4009c20 comp rx=0 tx=0).stop 2024-04-22T23:26:04.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.731+0000 7f4bb6781700 1 -- 172.21.15.67:0/3108657691 shutdown_connections 2024-04-22T23:26:04.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.731+0000 7f4bb6781700 1 --2- 172.21.15.67:0/3108657691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bb00ffe00 0x7f4bb0100280 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:04.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.731+0000 7f4bb6781700 1 --2- 172.21.15.67:0/3108657691 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4bb0101230 0x7f4bb0101650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:04.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.731+0000 7f4bb6781700 1 -- 172.21.15.67:0/3108657691 >> 172.21.15.67:0/3108657691 conn(0x7f4bb00fa8b0 msgr2=0x7f4bb00fcd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:26:04.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.731+0000 7f4bb6781700 1 -- 172.21.15.67:0/3108657691 shutdown_connections 2024-04-22T23:26:04.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.731+0000 7f4bb6781700 1 -- 172.21.15.67:0/3108657691 wait complete. 2024-04-22T23:26:04.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.731+0000 7f4bb6781700 1 Processor -- start 2024-04-22T23:26:04.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.731+0000 7f4bb6781700 1 -- start start 2024-04-22T23:26:04.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.732+0000 7f4bb6781700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4bb00ffe00 0x7f4bb01964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:26:04.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.732+0000 7f4bb6781700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bb0101230 0x7f4bb01969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:26:04.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.732+0000 7f4bb6781700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb0197010 con 0x7f4bb0101230 2024-04-22T23:26:04.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.732+0000 7f4bb6781700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb0197150 con 0x7f4bb00ffe00 2024-04-22T23:26:04.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.732+0000 7f4baf7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bb0101230 0x7f4bb01969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:26:04.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.732+0000 7f4baf7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bb0101230 0x7f4bb01969f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50976/0 (socket says 172.21.15.67:50976) 2024-04-22T23:26:04.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.732+0000 7f4baf7fe700 1 -- 172.21.15.67:0/2008556668 learned_addr learned my addr 172.21.15.67:0/2008556668 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:26:04.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.732+0000 7f4baffff700 1 --2- 172.21.15.67:0/2008556668 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4bb00ffe00 0x7f4bb01964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:26:04.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.732+0000 7f4baf7fe700 1 -- 172.21.15.67:0/2008556668 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4bb00ffe00 msgr2=0x7f4bb01964b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:04.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.732+0000 7f4baf7fe700 1 --2- 172.21.15.67:0/2008556668 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4bb00ffe00 0x7f4bb01964b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:04.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.732+0000 7f4baf7fe700 1 -- 172.21.15.67:0/2008556668 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba40095f0 con 0x7f4bb0101230 2024-04-22T23:26:04.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.733+0000 7f4baf7fe700 1 --2- 172.21.15.67:0/2008556668 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bb0101230 0x7f4bb01969f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4006010 tx=0x7f4ba400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:26:04.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.733+0000 7f4bad7fa700 1 -- 172.21.15.67:0/2008556668 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ba401d070 con 0x7f4bb0101230 2024-04-22T23:26:04.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.733+0000 7f4bb6781700 1 -- 172.21.15.67:0/2008556668 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bb019bba0 con 0x7f4bb0101230 2024-04-22T23:26:04.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.733+0000 7f4bad7fa700 1 -- 172.21.15.67:0/2008556668 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f4ba400f460 con 0x7f4bb0101230 2024-04-22T23:26:04.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.733+0000 7f4bad7fa700 1 -- 172.21.15.67:0/2008556668 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ba4021640 con 0x7f4bb0101230 2024-04-22T23:26:04.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.733+0000 7f4bb6781700 1 -- 172.21.15.67:0/2008556668 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bb019c090 con 0x7f4bb0101230 2024-04-22T23:26:04.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.734+0000 7f4bb6781700 1 -- 172.21.15.67:0/2008556668 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bb01087d0 con 0x7f4bb0101230 2024-04-22T23:26:04.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.734+0000 7f4bad7fa700 1 -- 172.21.15.67:0/2008556668 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 32) v1 ==== 92700+0+0 (secure 0 0 0) 0x7f4ba402b440 con 0x7f4bb0101230 2024-04-22T23:26:04.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.735+0000 7f4bad7fa700 1 --2- 172.21.15.67:0/2008556668 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f4b9806ea80 0x7f4b98070f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:26:04.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.735+0000 7f4bad7fa700 1 -- 172.21.15.67:0/2008556668 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f4ba408fcf0 con 0x7f4bb0101230 2024-04-22T23:26:04.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.735+0000 7f4baffff700 1 --2- 172.21.15.67:0/2008556668 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f4b9806ea80 0x7f4b98070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:26:04.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.736+0000 7f4baffff700 1 --2- 172.21.15.67:0/2008556668 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f4b9806ea80 0x7f4b98070f40 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0005950 tx=0x7f4ba000b410 comp rx=0 tx=0).ready entity=mgr.14708 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:26:04.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.740+0000 7f4bad7fa700 1 -- 172.21.15.67:0/2008556668 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f4ba405c910 con 0x7f4bb0101230 2024-04-22T23:26:04.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.977+0000 7f4bb6781700 1 -- 172.21.15.67:0/2008556668 --> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4bb00ff400 con 0x7f4b9806ea80 2024-04-22T23:26:04.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.978+0000 7f4bad7fa700 1 -- 172.21.15.67:0/2008556668 <== mgr.14708 v2:172.21.15.67:6800/2306276228 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7f4bb00ff400 con 0x7f4b9806ea80 2024-04-22T23:26:04.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.980+0000 7f4bb6781700 1 -- 172.21.15.67:0/2008556668 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f4b9806ea80 msgr2=0x7f4b98070f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:04.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.980+0000 7f4bb6781700 1 --2- 172.21.15.67:0/2008556668 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f4b9806ea80 0x7f4b98070f40 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0005950 tx=0x7f4ba000b410 comp rx=0 tx=0).stop 2024-04-22T23:26:04.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.980+0000 7f4bb6781700 1 -- 172.21.15.67:0/2008556668 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bb0101230 msgr2=0x7f4bb01969f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:04.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.980+0000 7f4bb6781700 1 --2- 172.21.15.67:0/2008556668 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bb0101230 0x7f4bb01969f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4006010 tx=0x7f4ba400bd70 comp rx=0 tx=0).stop 2024-04-22T23:26:04.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.980+0000 7f4bb6781700 1 -- 172.21.15.67:0/2008556668 shutdown_connections 2024-04-22T23:26:04.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.980+0000 7f4bb6781700 1 --2- 172.21.15.67:0/2008556668 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f4b9806ea80 0x7f4b98070f40 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:04.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.980+0000 7f4bb6781700 1 --2- 172.21.15.67:0/2008556668 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4bb00ffe00 0x7f4bb01964b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:04.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.980+0000 7f4bb6781700 1 --2- 172.21.15.67:0/2008556668 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bb0101230 0x7f4bb01969f0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:04.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.980+0000 7f4bb6781700 1 -- 172.21.15.67:0/2008556668 >> 172.21.15.67:0/2008556668 conn(0x7f4bb00fa8b0 msgr2=0x7f4bb00fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:26:04.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.980+0000 7f4bb6781700 1 -- 172.21.15.67:0/2008556668 shutdown_connections 2024-04-22T23:26:04.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:04.980+0000 7f4bb6781700 1 -- 172.21.15.67:0/2008556668 wait complete. 2024-04-22T23:26:05.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.090+0000 7f8fa90f2700 1 -- 172.21.15.67:0/901894195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fa40ffc10 msgr2=0x7f8fa4100030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:05.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.090+0000 7f8fa90f2700 1 --2- 172.21.15.67:0/901894195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fa40ffc10 0x7f8fa4100030 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8f94009910 tx=0x7f8f94009c20 comp rx=0 tx=0).stop 2024-04-22T23:26:05.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.090+0000 7f8fa90f2700 1 -- 172.21.15.67:0/901894195 shutdown_connections 2024-04-22T23:26:05.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.090+0000 7f8fa90f2700 1 --2- 172.21.15.67:0/901894195 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8fa4100e10 0x7f8fa4101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:05.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.090+0000 7f8fa90f2700 1 --2- 172.21.15.67:0/901894195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fa40ffc10 0x7f8fa4100030 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:05.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.090+0000 7f8fa90f2700 1 -- 172.21.15.67:0/901894195 >> 172.21.15.67:0/901894195 conn(0x7f8fa40fb190 msgr2=0x7f8fa40fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:26:05.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.090+0000 7f8fa90f2700 1 -- 172.21.15.67:0/901894195 shutdown_connections 2024-04-22T23:26:05.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.090+0000 7f8fa90f2700 1 -- 172.21.15.67:0/901894195 wait complete. 2024-04-22T23:26:05.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.091+0000 7f8fa90f2700 1 Processor -- start 2024-04-22T23:26:05.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.091+0000 7f8fa90f2700 1 -- start start 2024-04-22T23:26:05.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.091+0000 7f8fa90f2700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8fa40ffc10 0x7f8fa4196530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:26:05.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.091+0000 7f8fa90f2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fa4100e10 0x7f8fa4196a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:26:05.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.091+0000 7f8fa90f2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa4197090 con 0x7f8fa4100e10 2024-04-22T23:26:05.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.091+0000 7f8fa90f2700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa41971d0 con 0x7f8fa40ffc10 2024-04-22T23:26:05.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.092+0000 7f8fa259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fa4100e10 0x7f8fa4196a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:26:05.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.092+0000 7f8fa259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fa4100e10 0x7f8fa4196a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51000/0 (socket says 172.21.15.67:51000) 2024-04-22T23:26:05.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.092+0000 7f8fa259c700 1 -- 172.21.15.67:0/2143955831 learned_addr learned my addr 172.21.15.67:0/2143955831 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:26:05.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.092+0000 7f8fa2d9d700 1 --2- 172.21.15.67:0/2143955831 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8fa40ffc10 0x7f8fa4196530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:26:05.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.092+0000 7f8fa259c700 1 -- 172.21.15.67:0/2143955831 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8fa40ffc10 msgr2=0x7f8fa4196530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:05.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.092+0000 7f8fa259c700 1 --2- 172.21.15.67:0/2143955831 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8fa40ffc10 0x7f8fa4196530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:05.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.092+0000 7f8fa259c700 1 -- 172.21.15.67:0/2143955831 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f940095f0 con 0x7f8fa4100e10 2024-04-22T23:26:05.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.092+0000 7f8fa259c700 1 --2- 172.21.15.67:0/2143955831 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fa4100e10 0x7f8fa4196a70 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f8f9800b4b0 tx=0x7f8f9800b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:26:05.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.092+0000 7f8f8bfff700 1 -- 172.21.15.67:0/2143955831 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f98011800 con 0x7f8fa4100e10 2024-04-22T23:26:05.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.092+0000 7f8f8bfff700 1 -- 172.21.15.67:0/2143955831 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f8f98011e10 con 0x7f8fa4100e10 2024-04-22T23:26:05.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.092+0000 7f8fa90f2700 1 -- 172.21.15.67:0/2143955831 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fa419bc80 con 0x7f8fa4100e10 2024-04-22T23:26:05.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.093+0000 7f8f8bfff700 1 -- 172.21.15.67:0/2143955831 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f9800f2e0 con 0x7f8fa4100e10 2024-04-22T23:26:05.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.093+0000 7f8fa90f2700 1 -- 172.21.15.67:0/2143955831 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fa419c1d0 con 0x7f8fa4100e10 2024-04-22T23:26:05.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.094+0000 7f8fa90f2700 1 -- 172.21.15.67:0/2143955831 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fa4108780 con 0x7f8fa4100e10 2024-04-22T23:26:05.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.094+0000 7f8f8bfff700 1 -- 172.21.15.67:0/2143955831 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 32) v1 ==== 92700+0+0 (secure 0 0 0) 0x7f8f98010380 con 0x7f8fa4100e10 2024-04-22T23:26:05.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.095+0000 7f8f8bfff700 1 --2- 172.21.15.67:0/2143955831 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f8f8c06ea70 0x7f8f8c070f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:26:05.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.095+0000 7f8f8bfff700 1 -- 172.21.15.67:0/2143955831 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f8f9808e5d0 con 0x7f8fa4100e10 2024-04-22T23:26:05.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.095+0000 7f8fa2d9d700 1 --2- 172.21.15.67:0/2143955831 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f8f8c06ea70 0x7f8f8c070f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:26:05.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.095+0000 7f8fa2d9d700 1 --2- 172.21.15.67:0/2143955831 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f8f8c06ea70 0x7f8f8c070f30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8f94006010 tx=0x7f8f9400b560 comp rx=0 tx=0).ready entity=mgr.14708 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:26:05.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.099+0000 7f8f8bfff700 1 -- 172.21.15.67:0/2143955831 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f8f9805b280 con 0x7f8fa4100e10 2024-04-22T23:26:05.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:05 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:05.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:05 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:05.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:05 smithi067 ceph-mon[26453]: pgmap v35: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T23:26:05.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:05 smithi067 ceph-mon[26453]: from='client.14782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:26:05.296 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:05 smithi067 ceph-mon[26453]: from='client.14786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:26:05.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.349+0000 7f8fa90f2700 1 -- 172.21.15.67:0/2143955831 --> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f8fa4068c90 con 0x7f8f8c06ea70 2024-04-22T23:26:05.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.360+0000 7f8f8bfff700 1 -- 172.21.15.67:0/2143955831 <== mgr.14708 v2:172.21.15.67:6800/2306276228 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f8fa4068c90 con 0x7f8f8c06ea70 2024-04-22T23:26:05.362 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:26:05.362 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (9m) 1s ago 11m 26.1M - 0.25.0 c8568f914cd2 8c7b7ec00657 2024-04-22T23:26:05.362 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (11m) 1s ago 11m 7998k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:26:05.362 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (10m) 20s ago 10m 7931k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:26:05.362 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (11m) 1s ago 11m 6660k - 18.2.1 5be31c24972a 14085634c5d3 2024-04-22T23:26:05.362 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (10m) 20s ago 10m 6665k - 18.2.1 5be31c24972a fa995d458353 2024-04-22T23:26:05.362 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (9m) 1s ago 10m 85.8M - 9.4.7 954c08fa6188 0d8950c9af0c 2024-04-22T23:26:05.362 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (5m) 1s ago 5m 62.7M - 18.2.1 5be31c24972a e24a471f0a8c 2024-04-22T23:26:05.362 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (5m) 1s ago 5m 130M - 18.2.1 5be31c24972a 0730f9fc8e61 2024-04-22T23:26:05.362 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (5m) 20s ago 5m 123M - 18.2.1 5be31c24972a b6a169586433 2024-04-22T23:26:05.362 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (5m) 20s ago 5m 67.5M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:26:05.362 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (88s) 1s ago 12m 499M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:26:05.362 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (42s) 20s ago 10m 449M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:26:05.363 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (12m) 1s ago 12m 57.8M 2048M 18.2.1 5be31c24972a 6789283703e0 2024-04-22T23:26:05.363 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (10m) 20s ago 10m 46.6M 2048M 18.2.1 5be31c24972a 2fb23795f7e3 2024-04-22T23:26:05.363 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (29s) 1s ago 11m 8762k - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:26:05.363 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (24s) 20s ago 10m 3376k - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:26:05.363 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (9m) 1s ago 9m 204M 1216M 18.2.1 5be31c24972a a033247ee8c2 2024-04-22T23:26:05.363 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (8m) 1s ago 8m 210M 1216M 18.2.1 5be31c24972a 915b07a8edc4 2024-04-22T23:26:05.363 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (8m) 1s ago 8m 171M 1216M 18.2.1 5be31c24972a e257ecb4efa1 2024-04-22T23:26:05.363 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (8m) 20s ago 8m 248M 2240M 18.2.1 5be31c24972a 9c2c91a967ba 2024-04-22T23:26:05.363 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (7m) 20s ago 7m 217M 2240M 18.2.1 5be31c24972a c7586562ffa6 2024-04-22T23:26:05.363 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (7m) 20s ago 7m 192M 2240M 18.2.1 5be31c24972a 9fd97f1c65cb 2024-04-22T23:26:05.363 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (7s) 1s ago 10m 40.1M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:26:05.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.364+0000 7f8fa90f2700 1 -- 172.21.15.67:0/2143955831 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f8f8c06ea70 msgr2=0x7f8f8c070f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:05.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.364+0000 7f8fa90f2700 1 --2- 172.21.15.67:0/2143955831 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f8f8c06ea70 0x7f8f8c070f30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8f94006010 tx=0x7f8f9400b560 comp rx=0 tx=0).stop 2024-04-22T23:26:05.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.364+0000 7f8fa90f2700 1 -- 172.21.15.67:0/2143955831 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fa4100e10 msgr2=0x7f8fa4196a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:05.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.364+0000 7f8fa90f2700 1 --2- 172.21.15.67:0/2143955831 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fa4100e10 0x7f8fa4196a70 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f8f9800b4b0 tx=0x7f8f9800b7c0 comp rx=0 tx=0).stop 2024-04-22T23:26:05.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.364+0000 7f8fa90f2700 1 -- 172.21.15.67:0/2143955831 shutdown_connections 2024-04-22T23:26:05.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.364+0000 7f8fa90f2700 1 --2- 172.21.15.67:0/2143955831 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f8f8c06ea70 0x7f8f8c070f30 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:05.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.364+0000 7f8fa90f2700 1 --2- 172.21.15.67:0/2143955831 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8fa40ffc10 0x7f8fa4196530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:05.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.364+0000 7f8fa90f2700 1 --2- 172.21.15.67:0/2143955831 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fa4100e10 0x7f8fa4196a70 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:05.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.364+0000 7f8fa90f2700 1 -- 172.21.15.67:0/2143955831 >> 172.21.15.67:0/2143955831 conn(0x7f8fa40fb190 msgr2=0x7f8fa4068580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:26:05.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.365+0000 7f8fa90f2700 1 -- 172.21.15.67:0/2143955831 shutdown_connections 2024-04-22T23:26:05.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.365+0000 7f8fa90f2700 1 -- 172.21.15.67:0/2143955831 wait complete. 2024-04-22T23:26:05.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.475+0000 7eff10d75700 1 -- 172.21.15.67:0/2025682572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff0c1017a0 msgr2=0x7eff0c103b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:05.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.475+0000 7eff10d75700 1 --2- 172.21.15.67:0/2025682572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff0c1017a0 0x7eff0c103b90 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7efefc009910 tx=0x7efefc009c20 comp rx=0 tx=0).stop 2024-04-22T23:26:05.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.475+0000 7eff10d75700 1 -- 172.21.15.67:0/2025682572 shutdown_connections 2024-04-22T23:26:05.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.475+0000 7eff10d75700 1 --2- 172.21.15.67:0/2025682572 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff0c1040d0 0x7eff0c1064c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:05.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.475+0000 7eff10d75700 1 --2- 172.21.15.67:0/2025682572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff0c1017a0 0x7eff0c103b90 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:05.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.475+0000 7eff10d75700 1 -- 172.21.15.67:0/2025682572 >> 172.21.15.67:0/2025682572 conn(0x7eff0c0fb190 msgr2=0x7eff0c0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:26:05.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.475+0000 7eff10d75700 1 -- 172.21.15.67:0/2025682572 shutdown_connections 2024-04-22T23:26:05.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.475+0000 7eff10d75700 1 -- 172.21.15.67:0/2025682572 wait complete. 2024-04-22T23:26:05.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.475+0000 7eff10d75700 1 Processor -- start 2024-04-22T23:26:05.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.476+0000 7eff10d75700 1 -- start start 2024-04-22T23:26:05.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.476+0000 7eff10d75700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff0c1017a0 0x7eff0c198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:26:05.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.476+0000 7eff10d75700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff0c1040d0 0x7eff0c198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:26:05.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.476+0000 7eff10d75700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0c199260 con 0x7eff0c1040d0 2024-04-22T23:26:05.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.476+0000 7eff10d75700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0c1993a0 con 0x7eff0c1017a0 2024-04-22T23:26:05.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.476+0000 7eff09d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff0c1040d0 0x7eff0c198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:26:05.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.476+0000 7eff09d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff0c1040d0 0x7eff0c198c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51020/0 (socket says 172.21.15.67:51020) 2024-04-22T23:26:05.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.476+0000 7eff09d9b700 1 -- 172.21.15.67:0/2791033502 learned_addr learned my addr 172.21.15.67:0/2791033502 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:26:05.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.476+0000 7eff0a59c700 1 --2- 172.21.15.67:0/2791033502 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff0c1017a0 0x7eff0c198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:26:05.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.476+0000 7eff09d9b700 1 -- 172.21.15.67:0/2791033502 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff0c1017a0 msgr2=0x7eff0c198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:05.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.476+0000 7eff09d9b700 1 --2- 172.21.15.67:0/2791033502 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff0c1017a0 0x7eff0c198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:05.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.476+0000 7eff09d9b700 1 -- 172.21.15.67:0/2791033502 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efefc0095f0 con 0x7eff0c1040d0 2024-04-22T23:26:05.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.477+0000 7eff09d9b700 1 --2- 172.21.15.67:0/2791033502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff0c1040d0 0x7eff0c198c40 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7eff0000b4b0 tx=0x7eff0000b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:26:05.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.477+0000 7efefb7fe700 1 -- 172.21.15.67:0/2791033502 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff00011800 con 0x7eff0c1040d0 2024-04-22T23:26:05.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.477+0000 7efefb7fe700 1 -- 172.21.15.67:0/2791033502 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7eff00011e10 con 0x7eff0c1040d0 2024-04-22T23:26:05.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.477+0000 7eff10d75700 1 -- 172.21.15.67:0/2791033502 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff0c19de50 con 0x7eff0c1040d0 2024-04-22T23:26:05.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.477+0000 7efefb7fe700 1 -- 172.21.15.67:0/2791033502 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff0000f2e0 con 0x7eff0c1040d0 2024-04-22T23:26:05.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.477+0000 7eff10d75700 1 -- 172.21.15.67:0/2791033502 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff0c19e370 con 0x7eff0c1040d0 2024-04-22T23:26:05.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.478+0000 7eff10d75700 1 -- 172.21.15.67:0/2791033502 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff0c0fcd80 con 0x7eff0c1040d0 2024-04-22T23:26:05.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.478+0000 7efefb7fe700 1 -- 172.21.15.67:0/2791033502 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 32) v1 ==== 92700+0+0 (secure 0 0 0) 0x7eff00011960 con 0x7eff0c1040d0 2024-04-22T23:26:05.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.479+0000 7efefb7fe700 1 --2- 172.21.15.67:0/2791033502 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7efef406ea80 0x7efef4070f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:26:05.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.479+0000 7efefb7fe700 1 -- 172.21.15.67:0/2791033502 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7eff0008e5f0 con 0x7eff0c1040d0 2024-04-22T23:26:05.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.479+0000 7eff0a59c700 1 --2- 172.21.15.67:0/2791033502 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7efef406ea80 0x7efef4070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:26:05.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.480+0000 7eff0a59c700 1 --2- 172.21.15.67:0/2791033502 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7efef406ea80 0x7efef4070f40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7efefc006010 tx=0x7efefc00b540 comp rx=0 tx=0).ready entity=mgr.14708 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:26:05.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.484+0000 7efefb7fe700 1 -- 172.21.15.67:0/2791033502 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7eff0005b2a0 con 0x7eff0c1040d0 2024-04-22T23:26:05.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:05 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:05.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:05 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:05.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:05 smithi134 ceph-mon[34609]: pgmap v35: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T23:26:05.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:05 smithi134 ceph-mon[34609]: from='client.14782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:26:05.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:05 smithi134 ceph-mon[34609]: from='client.14786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:26:05.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.784+0000 7eff10d75700 1 -- 172.21.15.67:0/2791033502 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7eff0c02cc70 con 0x7eff0c1040d0 2024-04-22T23:26:05.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.784+0000 7efefb7fe700 1 -- 172.21.15.67:0/2791033502 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7eff0005ae30 con 0x7eff0c1040d0 2024-04-22T23:26:05.786 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:26:05.786 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:26:05.786 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-22T23:26:05.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:26:05.786 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:26:05.786 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:26:05.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:26:05.787 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:26:05.787 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-22T23:26:05.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:26:05.787 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:26:05.787 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-22T23:26:05.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:26:05.787 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:26:05.787 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 12, 2024-04-22T23:26:05.787 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:26:05.787 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:26:05.787 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:26:05.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.787+0000 7eff10d75700 1 -- 172.21.15.67:0/2791033502 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7efef406ea80 msgr2=0x7efef4070f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:05.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.787+0000 7eff10d75700 1 --2- 172.21.15.67:0/2791033502 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7efef406ea80 0x7efef4070f40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7efefc006010 tx=0x7efefc00b540 comp rx=0 tx=0).stop 2024-04-22T23:26:05.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.787+0000 7eff10d75700 1 -- 172.21.15.67:0/2791033502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff0c1040d0 msgr2=0x7eff0c198c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:05.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.787+0000 7eff10d75700 1 --2- 172.21.15.67:0/2791033502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff0c1040d0 0x7eff0c198c40 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7eff0000b4b0 tx=0x7eff0000b7c0 comp rx=0 tx=0).stop 2024-04-22T23:26:05.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.787+0000 7eff10d75700 1 -- 172.21.15.67:0/2791033502 shutdown_connections 2024-04-22T23:26:05.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.787+0000 7eff10d75700 1 --2- 172.21.15.67:0/2791033502 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7efef406ea80 0x7efef4070f40 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:05.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.787+0000 7eff10d75700 1 --2- 172.21.15.67:0/2791033502 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff0c1017a0 0x7eff0c198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:05.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.787+0000 7eff10d75700 1 --2- 172.21.15.67:0/2791033502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff0c1040d0 0x7eff0c198c40 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:05.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.787+0000 7eff10d75700 1 -- 172.21.15.67:0/2791033502 >> 172.21.15.67:0/2791033502 conn(0x7eff0c0fb190 msgr2=0x7eff0c104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:26:05.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.787+0000 7eff10d75700 1 -- 172.21.15.67:0/2791033502 shutdown_connections 2024-04-22T23:26:05.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.787+0000 7eff10d75700 1 -- 172.21.15.67:0/2791033502 wait complete. 2024-04-22T23:26:05.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.904+0000 7f22a4b3c700 1 -- 172.21.15.67:0/3049535687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22a0101730 msgr2=0x7f22a0103b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:05.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.904+0000 7f22a4b3c700 1 --2- 172.21.15.67:0/3049535687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22a0101730 0x7f22a0103b20 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f2290009910 tx=0x7f2290009c20 comp rx=0 tx=0).stop 2024-04-22T23:26:05.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.904+0000 7f22a4b3c700 1 -- 172.21.15.67:0/3049535687 shutdown_connections 2024-04-22T23:26:05.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.904+0000 7f22a4b3c700 1 --2- 172.21.15.67:0/3049535687 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f22a0104060 0x7f22a0106450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:05.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.904+0000 7f22a4b3c700 1 --2- 172.21.15.67:0/3049535687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22a0101730 0x7f22a0103b20 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:05.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.904+0000 7f22a4b3c700 1 -- 172.21.15.67:0/3049535687 >> 172.21.15.67:0/3049535687 conn(0x7f22a00fb120 msgr2=0x7f22a00fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:26:05.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.904+0000 7f22a4b3c700 1 -- 172.21.15.67:0/3049535687 shutdown_connections 2024-04-22T23:26:05.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.904+0000 7f22a4b3c700 1 -- 172.21.15.67:0/3049535687 wait complete. 2024-04-22T23:26:05.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.905+0000 7f22a4b3c700 1 Processor -- start 2024-04-22T23:26:05.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.905+0000 7f22a4b3c700 1 -- start start 2024-04-22T23:26:05.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.905+0000 7f22a4b3c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f22a0101730 0x7f22a01964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:26:05.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.905+0000 7f22a4b3c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22a0104060 0x7f22a0196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:26:05.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.905+0000 7f22a4b3c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a0197030 con 0x7f22a0104060 2024-04-22T23:26:05.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.905+0000 7f22a4b3c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a0197170 con 0x7f22a0101730 2024-04-22T23:26:05.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.906+0000 7f229dd9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22a0104060 0x7f22a0196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:26:05.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.906+0000 7f229dd9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22a0104060 0x7f22a0196a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51036/0 (socket says 172.21.15.67:51036) 2024-04-22T23:26:05.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.906+0000 7f229dd9b700 1 -- 172.21.15.67:0/1690617386 learned_addr learned my addr 172.21.15.67:0/1690617386 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:26:05.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.906+0000 7f229e59c700 1 --2- 172.21.15.67:0/1690617386 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f22a0101730 0x7f22a01964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:26:05.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.906+0000 7f229dd9b700 1 -- 172.21.15.67:0/1690617386 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f22a0101730 msgr2=0x7f22a01964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:05.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.906+0000 7f229dd9b700 1 --2- 172.21.15.67:0/1690617386 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f22a0101730 0x7f22a01964d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:05.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.906+0000 7f229dd9b700 1 -- 172.21.15.67:0/1690617386 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22900095f0 con 0x7f22a0104060 2024-04-22T23:26:05.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.906+0000 7f229dd9b700 1 --2- 172.21.15.67:0/1690617386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22a0104060 0x7f22a0196a10 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f229400b4b0 tx=0x7f229400b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:26:05.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.906+0000 7f228f7fe700 1 -- 172.21.15.67:0/1690617386 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2294011800 con 0x7f22a0104060 2024-04-22T23:26:05.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.906+0000 7f22a4b3c700 1 -- 172.21.15.67:0/1690617386 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22a019bc20 con 0x7f22a0104060 2024-04-22T23:26:05.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.906+0000 7f228f7fe700 1 -- 172.21.15.67:0/1690617386 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f2294011e10 con 0x7f22a0104060 2024-04-22T23:26:05.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.906+0000 7f22a4b3c700 1 -- 172.21.15.67:0/1690617386 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22a019c170 con 0x7f22a0104060 2024-04-22T23:26:05.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.906+0000 7f228f7fe700 1 -- 172.21.15.67:0/1690617386 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f229400f2e0 con 0x7f22a0104060 2024-04-22T23:26:05.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.907+0000 7f22a4b3c700 1 -- 172.21.15.67:0/1690617386 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22a0190690 con 0x7f22a0104060 2024-04-22T23:26:05.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.908+0000 7f228f7fe700 1 -- 172.21.15.67:0/1690617386 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 32) v1 ==== 92700+0+0 (secure 0 0 0) 0x7f2294010380 con 0x7f22a0104060 2024-04-22T23:26:05.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.908+0000 7f228f7fe700 1 --2- 172.21.15.67:0/1690617386 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f228806ea70 0x7f2288070f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:26:05.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.908+0000 7f228f7fe700 1 -- 172.21.15.67:0/1690617386 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f229408e600 con 0x7f22a0104060 2024-04-22T23:26:05.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.909+0000 7f229e59c700 1 --2- 172.21.15.67:0/1690617386 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f228806ea70 0x7f2288070f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:26:05.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.909+0000 7f229e59c700 1 --2- 172.21.15.67:0/1690617386 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f228806ea70 0x7f2288070f30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2290006010 tx=0x7f229000b560 comp rx=0 tx=0).ready entity=mgr.14708 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:26:05.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:05.913+0000 7f228f7fe700 1 -- 172.21.15.67:0/1690617386 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f229405b2b0 con 0x7f22a0104060 2024-04-22T23:26:06.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:06.147+0000 7f22a4b3c700 1 -- 172.21.15.67:0/1690617386 --> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f22a00611d0 con 0x7f228806ea70 2024-04-22T23:26:06.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:06.149+0000 7f228f7fe700 1 -- 172.21.15.67:0/1690617386 <== mgr.14708 v2:172.21.15.67:6800/2306276228 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7f22a00611d0 con 0x7f228806ea70 2024-04-22T23:26:06.150 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:26:06.150 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:26:06.150 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:26:06.150 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-22T23:26:06.150 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:26:06.150 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr" 2024-04-22T23:26:06.151 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:26:06.151 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "2/2 daemons upgraded", 2024-04-22T23:26:06.151 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading prometheus daemons", 2024-04-22T23:26:06.151 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:26:06.151 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:26:06.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:06.151+0000 7f22a4b3c700 1 -- 172.21.15.67:0/1690617386 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f228806ea70 msgr2=0x7f2288070f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:06.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:06.151+0000 7f22a4b3c700 1 --2- 172.21.15.67:0/1690617386 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f228806ea70 0x7f2288070f30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2290006010 tx=0x7f229000b560 comp rx=0 tx=0).stop 2024-04-22T23:26:06.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:06.151+0000 7f22a4b3c700 1 -- 172.21.15.67:0/1690617386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22a0104060 msgr2=0x7f22a0196a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:06.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:06.151+0000 7f22a4b3c700 1 --2- 172.21.15.67:0/1690617386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22a0104060 0x7f22a0196a10 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f229400b4b0 tx=0x7f229400b7c0 comp rx=0 tx=0).stop 2024-04-22T23:26:06.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:06.151+0000 7f22a4b3c700 1 -- 172.21.15.67:0/1690617386 shutdown_connections 2024-04-22T23:26:06.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:06.151+0000 7f22a4b3c700 1 --2- 172.21.15.67:0/1690617386 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f228806ea70 0x7f2288070f30 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:06.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:06.151+0000 7f22a4b3c700 1 --2- 172.21.15.67:0/1690617386 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f22a0101730 0x7f22a01964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:06.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:06.151+0000 7f22a4b3c700 1 --2- 172.21.15.67:0/1690617386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22a0104060 0x7f22a0196a10 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:06.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:06.151+0000 7f22a4b3c700 1 -- 172.21.15.67:0/1690617386 >> 172.21.15.67:0/1690617386 conn(0x7f22a00fb120 msgr2=0x7f22a00fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:26:06.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:06.151+0000 7f22a4b3c700 1 -- 172.21.15.67:0/1690617386 shutdown_connections 2024-04-22T23:26:06.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:06.151+0000 7f22a4b3c700 1 -- 172.21.15.67:0/1690617386 wait complete. 2024-04-22T23:26:06.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:06 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:06.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:06 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:06.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:06 smithi067 ceph-mon[26453]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:26:06.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:06 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/2791033502' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:06.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:06 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:06.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:06 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:06.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:06 smithi134 ceph-mon[34609]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:26:06.745 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:06 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/2791033502' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:07.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:07 smithi067 ceph-mon[26453]: from='client.14798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:26:07.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:07 smithi067 ceph-mon[26453]: pgmap v36: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 5 op/s 2024-04-22T23:26:07.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:07 smithi134 ceph-mon[34609]: from='client.14798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:26:07.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:07 smithi134 ceph-mon[34609]: pgmap v36: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 5 op/s 2024-04-22T23:26:09.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:09 smithi067 ceph-mon[26453]: pgmap v37: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-22T23:26:09.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:09 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:09.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:09 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:09.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:09 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:26:09.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:09 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:26:09.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:09 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:09.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:09 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T23:26:09.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:09 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:26:09.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:09 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:09.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:09 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:09.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:09 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:09.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:09 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:09.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:09 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:09.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:09 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:09.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:09 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:09.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:09 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:09.796 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:09 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:09.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:09 smithi134 ceph-mon[34609]: pgmap v37: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-22T23:26:09.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:09 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:09.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:09 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:09.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:09 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:26:09.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:09 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:26:09.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:09 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:09.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:09 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T23:26:09.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:09 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:26:09.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:09 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:09.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:09 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:09.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:09 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:09.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:09 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:09.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:09 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:09.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:09 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:09.996 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:09 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:09.996 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:09 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:09.996 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:09 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:10.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:10 smithi067 ceph-mon[26453]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T23:26:10.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:10 smithi134 ceph-mon[34609]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T23:26:11.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:11 smithi067 ceph-mon[26453]: pgmap v38: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T23:26:11.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:11 smithi067 ceph-mon[26453]: Upgrade: Updating alertmanager.smithi067 2024-04-22T23:26:11.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:11 smithi067 ceph-mon[26453]: Deploying daemon alertmanager.smithi067 on smithi067 2024-04-22T23:26:11.898 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:11 smithi134 ceph-mon[34609]: pgmap v38: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T23:26:11.898 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:11 smithi134 ceph-mon[34609]: Upgrade: Updating alertmanager.smithi067 2024-04-22T23:26:11.898 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:11 smithi134 ceph-mon[34609]: Deploying daemon alertmanager.smithi067 on smithi067 2024-04-22T23:26:13.795 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:13 smithi067 ceph-mon[26453]: pgmap v39: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-04-22T23:26:13.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:13 smithi134 ceph-mon[34609]: pgmap v39: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-04-22T23:26:15.855 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:15 smithi067 ceph-mon[26453]: pgmap v40: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T23:26:15.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:15 smithi134 ceph-mon[34609]: pgmap v40: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T23:26:16.666 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:16 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:26:16.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:16 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:26:18.083 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:17 smithi067 ceph-mon[26453]: pgmap v41: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-04-22T23:26:18.084 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:17 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:18.084 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:17 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:18.084 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:17 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:26:18.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:17 smithi134 ceph-mon[34609]: pgmap v41: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-04-22T23:26:18.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:17 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:18.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:17 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:18.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:17 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:26:20.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:19 smithi134 ceph-mon[34609]: pgmap v42: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:26:20.248 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:19 smithi067 ceph-mon[26453]: pgmap v42: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:26:21.617 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:21 smithi067 ceph-mon[26453]: pgmap v43: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T23:26:21.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:21 smithi134 ceph-mon[34609]: pgmap v43: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T23:26:23.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:23 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:23.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:23 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:23.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:23 smithi134 ceph-mon[34609]: pgmap v44: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-04-22T23:26:23.504 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:23 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:23.504 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:23 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:23.504 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:23 smithi067 ceph-mon[26453]: pgmap v44: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-04-22T23:26:24.722 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:24 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:24.722 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:24 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:24.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:24 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:24.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:24 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:25.389 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:25 smithi067 ceph-mon[26453]: pgmap v45: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T23:26:25.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:25 smithi134 ceph-mon[34609]: pgmap v45: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T23:26:27.910 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:27 smithi067 ceph-mon[26453]: pgmap v46: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-04-22T23:26:27.911 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:27 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:27.911 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:27 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:27.911 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:27 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:26:27.911 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:27 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:26:27.911 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:27 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:27.911 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:27 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T23:26:27.911 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:27 smithi067 ceph-mon[26453]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T23:26:27.911 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:27 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:26:27.911 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:27 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:27.911 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:27 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:27.911 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:27 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:27.911 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:27 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:27.911 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:27 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:27.912 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:27 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:27.912 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:27 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:27.912 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:27 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:27.912 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:27 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:27.912 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:27 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:28.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:27 smithi134 ceph-mon[34609]: pgmap v46: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-04-22T23:26:28.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:27 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:28.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:27 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:28.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:27 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:26:28.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:27 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:26:28.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:27 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:28.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:27 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T23:26:28.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:27 smithi134 ceph-mon[34609]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T23:26:28.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:27 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:26:28.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:27 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:28.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:27 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:28.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:27 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:28.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:27 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:28.245 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:27 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:28.246 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:27 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:28.246 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:27 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:28.246 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:27 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:28.246 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:27 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:28.246 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:27 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:30.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:29 smithi134 ceph-mon[34609]: pgmap v47: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T23:26:30.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:29 smithi134 ceph-mon[34609]: Upgrade: Updating grafana.smithi067 2024-04-22T23:26:30.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:29 smithi134 ceph-mon[34609]: Deploying daemon grafana.smithi067 on smithi067 2024-04-22T23:26:30.294 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:29 smithi067 ceph-mon[26453]: pgmap v47: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T23:26:30.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:29 smithi067 ceph-mon[26453]: Upgrade: Updating grafana.smithi067 2024-04-22T23:26:30.295 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:29 smithi067 ceph-mon[26453]: Deploying daemon grafana.smithi067 on smithi067 2024-04-22T23:26:31.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:31 smithi134 ceph-mon[34609]: pgmap v48: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T23:26:31.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:31 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:26:31.794 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:31 smithi067 ceph-mon[26453]: pgmap v48: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T23:26:31.794 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:31 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:26:33.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:33 smithi134 ceph-mon[34609]: pgmap v49: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-22T23:26:33.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:33 smithi134 ceph-mon[34609]: mgrmap e33: smithi067.whdcsr(active, since 92s), standbys: smithi134.ccxkcj 2024-04-22T23:26:33.794 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:33 smithi067 ceph-mon[26453]: pgmap v49: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-22T23:26:33.794 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:33 smithi067 ceph-mon[26453]: mgrmap e33: smithi067.whdcsr(active, since 92s), standbys: smithi134.ccxkcj 2024-04-22T23:26:35.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:35 smithi134 ceph-mon[34609]: pgmap v50: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-22T23:26:36.005 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:35 smithi067 ceph-mon[26453]: pgmap v50: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-22T23:26:36.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.268+0000 7f067a828700 1 -- 172.21.15.67:0/4186077609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0674100cb0 msgr2=0x7f06741010d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:36.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.268+0000 7f067a828700 1 --2- 172.21.15.67:0/4186077609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0674100cb0 0x7f06741010d0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f0664009910 tx=0x7f0664009c20 comp rx=0 tx=0).stop 2024-04-22T23:26:36.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.268+0000 7f067a828700 1 -- 172.21.15.67:0/4186077609 shutdown_connections 2024-04-22T23:26:36.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.268+0000 7f067a828700 1 --2- 172.21.15.67:0/4186077609 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0674101eb0 0x7f0674102310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:36.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.268+0000 7f067a828700 1 --2- 172.21.15.67:0/4186077609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0674100cb0 0x7f06741010d0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:36.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.268+0000 7f067a828700 1 -- 172.21.15.67:0/4186077609 >> 172.21.15.67:0/4186077609 conn(0x7f06740fc230 msgr2=0x7f06740fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:26:36.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.268+0000 7f067a828700 1 -- 172.21.15.67:0/4186077609 shutdown_connections 2024-04-22T23:26:36.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.268+0000 7f067a828700 1 -- 172.21.15.67:0/4186077609 wait complete. 2024-04-22T23:26:36.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.269+0000 7f067a828700 1 Processor -- start 2024-04-22T23:26:36.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.269+0000 7f067a828700 1 -- start start 2024-04-22T23:26:36.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.269+0000 7f067a828700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0674100cb0 0x7f0674074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:26:36.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.269+0000 7f067a828700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0674101eb0 0x7f0674073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:26:36.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.269+0000 7f067a828700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06740736c0 con 0x7f0674100cb0 2024-04-22T23:26:36.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.269+0000 7f067a828700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0674073800 con 0x7f0674101eb0 2024-04-22T23:26:36.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.269+0000 7f0673fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0674100cb0 0x7f0674074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:26:36.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.269+0000 7f0673fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0674100cb0 0x7f0674074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50012/0 (socket says 172.21.15.67:50012) 2024-04-22T23:26:36.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.269+0000 7f0673fff700 1 -- 172.21.15.67:0/2609285749 learned_addr learned my addr 172.21.15.67:0/2609285749 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:26:36.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.269+0000 7f06737fe700 1 --2- 172.21.15.67:0/2609285749 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0674101eb0 0x7f0674073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:26:36.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.270+0000 7f0673fff700 1 -- 172.21.15.67:0/2609285749 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0674101eb0 msgr2=0x7f0674073180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:36.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.270+0000 7f0673fff700 1 --2- 172.21.15.67:0/2609285749 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0674101eb0 0x7f0674073180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:36.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.270+0000 7f0673fff700 1 -- 172.21.15.67:0/2609285749 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06640095f0 con 0x7f0674100cb0 2024-04-22T23:26:36.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.270+0000 7f0673fff700 1 --2- 172.21.15.67:0/2609285749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0674100cb0 0x7f0674074b30 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0664000c00 tx=0x7f066400f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:26:36.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.270+0000 7f06717fa700 1 -- 172.21.15.67:0/2609285749 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f066401d070 con 0x7f0674100cb0 2024-04-22T23:26:36.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.270+0000 7f06717fa700 1 -- 172.21.15.67:0/2609285749 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f0664005dc0 con 0x7f0674100cb0 2024-04-22T23:26:36.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.270+0000 7f06717fa700 1 -- 172.21.15.67:0/2609285749 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06640226a0 con 0x7f0674100cb0 2024-04-22T23:26:36.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.270+0000 7f067a828700 1 -- 172.21.15.67:0/2609285749 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0674073a80 con 0x7f0674100cb0 2024-04-22T23:26:36.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.270+0000 7f067a828700 1 -- 172.21.15.67:0/2609285749 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0674073f20 con 0x7f0674100cb0 2024-04-22T23:26:36.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.271+0000 7f067a828700 1 -- 172.21.15.67:0/2609285749 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0674109830 con 0x7f0674100cb0 2024-04-22T23:26:36.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.272+0000 7f06717fa700 1 -- 172.21.15.67:0/2609285749 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 33) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f0664003910 con 0x7f0674100cb0 2024-04-22T23:26:36.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.273+0000 7f06717fa700 1 --2- 172.21.15.67:0/2609285749 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f065c06e940 0x7f065c070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:26:36.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.273+0000 7f06717fa700 1 -- 172.21.15.67:0/2609285749 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f066408ff30 con 0x7f0674100cb0 2024-04-22T23:26:36.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.273+0000 7f06737fe700 1 --2- 172.21.15.67:0/2609285749 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f065c06e940 0x7f065c070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:26:36.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.274+0000 7f06737fe700 1 --2- 172.21.15.67:0/2609285749 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f065c06e940 0x7f065c070e00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0668009bf0 tx=0x7f0668009260 comp rx=0 tx=0).ready entity=mgr.14708 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:26:36.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.279+0000 7f06717fa700 1 -- 172.21.15.67:0/2609285749 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f06640c4e20 con 0x7f0674100cb0 2024-04-22T23:26:36.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.508+0000 7f067a828700 1 -- 172.21.15.67:0/2609285749 --> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0674106800 con 0x7f065c06e940 2024-04-22T23:26:36.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.510+0000 7f06717fa700 1 -- 172.21.15.67:0/2609285749 <== mgr.14708 v2:172.21.15.67:6800/2306276228 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+364 (secure 0 0 0) 0x7f0674106800 con 0x7f065c06e940 2024-04-22T23:26:36.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.512+0000 7f067a828700 1 -- 172.21.15.67:0/2609285749 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f065c06e940 msgr2=0x7f065c070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:36.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.512+0000 7f067a828700 1 --2- 172.21.15.67:0/2609285749 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f065c06e940 0x7f065c070e00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0668009bf0 tx=0x7f0668009260 comp rx=0 tx=0).stop 2024-04-22T23:26:36.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.512+0000 7f067a828700 1 -- 172.21.15.67:0/2609285749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0674100cb0 msgr2=0x7f0674074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:36.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.512+0000 7f067a828700 1 --2- 172.21.15.67:0/2609285749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0674100cb0 0x7f0674074b30 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0664000c00 tx=0x7f066400f740 comp rx=0 tx=0).stop 2024-04-22T23:26:36.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.512+0000 7f067a828700 1 -- 172.21.15.67:0/2609285749 shutdown_connections 2024-04-22T23:26:36.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.512+0000 7f067a828700 1 --2- 172.21.15.67:0/2609285749 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f065c06e940 0x7f065c070e00 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:36.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.512+0000 7f067a828700 1 --2- 172.21.15.67:0/2609285749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0674100cb0 0x7f0674074b30 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:36.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.512+0000 7f067a828700 1 --2- 172.21.15.67:0/2609285749 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0674101eb0 0x7f0674073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:36.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.512+0000 7f067a828700 1 -- 172.21.15.67:0/2609285749 >> 172.21.15.67:0/2609285749 conn(0x7f06740fc230 msgr2=0x7f06741050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:26:36.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.512+0000 7f067a828700 1 -- 172.21.15.67:0/2609285749 shutdown_connections 2024-04-22T23:26:36.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.512+0000 7f067a828700 1 -- 172.21.15.67:0/2609285749 wait complete. 2024-04-22T23:26:36.524 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:26:36.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.622+0000 7f190fc05700 1 -- 172.21.15.67:0/1250280159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19080ffc10 msgr2=0x7f1908100030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:36.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.622+0000 7f190fc05700 1 --2- 172.21.15.67:0/1250280159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19080ffc10 0x7f1908100030 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f190400b0a0 tx=0x7f190400b3b0 comp rx=0 tx=0).stop 2024-04-22T23:26:36.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.622+0000 7f190fc05700 1 -- 172.21.15.67:0/1250280159 shutdown_connections 2024-04-22T23:26:36.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.622+0000 7f190fc05700 1 --2- 172.21.15.67:0/1250280159 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1908100e10 0x7f1908101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:36.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.622+0000 7f190fc05700 1 --2- 172.21.15.67:0/1250280159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19080ffc10 0x7f1908100030 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:36.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.622+0000 7f190fc05700 1 -- 172.21.15.67:0/1250280159 >> 172.21.15.67:0/1250280159 conn(0x7f19080fb190 msgr2=0x7f19080fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:26:36.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.622+0000 7f190fc05700 1 -- 172.21.15.67:0/1250280159 shutdown_connections 2024-04-22T23:26:36.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.622+0000 7f190fc05700 1 -- 172.21.15.67:0/1250280159 wait complete. 2024-04-22T23:26:36.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.623+0000 7f190fc05700 1 Processor -- start 2024-04-22T23:26:36.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.623+0000 7f190fc05700 1 -- start start 2024-04-22T23:26:36.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.623+0000 7f190fc05700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f19080ffc10 0x7f1908103ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:26:36.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.623+0000 7f190fc05700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1908100e10 0x7f19081021f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:26:36.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.623+0000 7f190fc05700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1908102730 con 0x7f1908100e10 2024-04-22T23:26:36.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.623+0000 7f190fc05700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1908102870 con 0x7f19080ffc10 2024-04-22T23:26:36.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.623+0000 7f190d1a0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1908100e10 0x7f19081021f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:26:36.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.623+0000 7f190d1a0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1908100e10 0x7f19081021f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50040/0 (socket says 172.21.15.67:50040) 2024-04-22T23:26:36.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.623+0000 7f190d1a0700 1 -- 172.21.15.67:0/1855515487 learned_addr learned my addr 172.21.15.67:0/1855515487 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:26:36.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.623+0000 7f190d9a1700 1 --2- 172.21.15.67:0/1855515487 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f19080ffc10 0x7f1908103ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:26:36.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.623+0000 7f190d1a0700 1 -- 172.21.15.67:0/1855515487 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f19080ffc10 msgr2=0x7f1908103ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:36.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.623+0000 7f190d1a0700 1 --2- 172.21.15.67:0/1855515487 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f19080ffc10 0x7f1908103ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:36.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.623+0000 7f190d1a0700 1 -- 172.21.15.67:0/1855515487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1904009d00 con 0x7f1908100e10 2024-04-22T23:26:36.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.623+0000 7f190d1a0700 1 --2- 172.21.15.67:0/1855515487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1908100e10 0x7f19081021f0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f18f800d8d0 tx=0x7f18f800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:26:36.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.624+0000 7f18feffd700 1 -- 172.21.15.67:0/1855515487 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18f80098a0 con 0x7f1908100e10 2024-04-22T23:26:36.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.624+0000 7f18feffd700 1 -- 172.21.15.67:0/1855515487 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f18f8010460 con 0x7f1908100e10 2024-04-22T23:26:36.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.624+0000 7f190fc05700 1 -- 172.21.15.67:0/1855515487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1908102b50 con 0x7f1908100e10 2024-04-22T23:26:36.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.624+0000 7f18feffd700 1 -- 172.21.15.67:0/1855515487 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18f800f5f0 con 0x7f1908100e10 2024-04-22T23:26:36.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.624+0000 7f190fc05700 1 -- 172.21.15.67:0/1855515487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f190819e530 con 0x7f1908100e10 2024-04-22T23:26:36.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.625+0000 7f190fc05700 1 -- 172.21.15.67:0/1855515487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1908108700 con 0x7f1908100e10 2024-04-22T23:26:36.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.625+0000 7f18feffd700 1 -- 172.21.15.67:0/1855515487 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 33) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f18f8009a00 con 0x7f1908100e10 2024-04-22T23:26:36.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.626+0000 7f18feffd700 1 --2- 172.21.15.67:0/1855515487 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f18f406e940 0x7f18f4070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:26:36.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.626+0000 7f18feffd700 1 -- 172.21.15.67:0/1855515487 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f18f808e6c0 con 0x7f1908100e10 2024-04-22T23:26:36.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.626+0000 7f190d9a1700 1 --2- 172.21.15.67:0/1855515487 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f18f406e940 0x7f18f4070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:26:36.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.627+0000 7f190d9a1700 1 --2- 172.21.15.67:0/1855515487 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f18f406e940 0x7f18f4070e00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f190400b830 tx=0x7f19040060c0 comp rx=0 tx=0).ready entity=mgr.14708 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:26:36.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.631+0000 7f18feffd700 1 -- 172.21.15.67:0/1855515487 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f18f805b500 con 0x7f1908100e10 2024-04-22T23:26:36.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.865+0000 7f190fc05700 1 -- 172.21.15.67:0/1855515487 --> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1908068c90 con 0x7f18f406e940 2024-04-22T23:26:36.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.866+0000 7f18feffd700 1 -- 172.21.15.67:0/1855515487 <== mgr.14708 v2:172.21.15.67:6800/2306276228 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+364 (secure 0 0 0) 0x7f1908068c90 con 0x7f18f406e940 2024-04-22T23:26:36.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.868+0000 7f190fc05700 1 -- 172.21.15.67:0/1855515487 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f18f406e940 msgr2=0x7f18f4070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:36.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.868+0000 7f190fc05700 1 --2- 172.21.15.67:0/1855515487 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f18f406e940 0x7f18f4070e00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f190400b830 tx=0x7f19040060c0 comp rx=0 tx=0).stop 2024-04-22T23:26:36.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.868+0000 7f190fc05700 1 -- 172.21.15.67:0/1855515487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1908100e10 msgr2=0x7f19081021f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:36.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.868+0000 7f190fc05700 1 --2- 172.21.15.67:0/1855515487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1908100e10 0x7f19081021f0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f18f800d8d0 tx=0x7f18f800dbe0 comp rx=0 tx=0).stop 2024-04-22T23:26:36.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.869+0000 7f190fc05700 1 -- 172.21.15.67:0/1855515487 shutdown_connections 2024-04-22T23:26:36.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.869+0000 7f190fc05700 1 --2- 172.21.15.67:0/1855515487 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f18f406e940 0x7f18f4070e00 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:36.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.869+0000 7f190fc05700 1 --2- 172.21.15.67:0/1855515487 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f19080ffc10 0x7f1908103ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:36.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.869+0000 7f190fc05700 1 --2- 172.21.15.67:0/1855515487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1908100e10 0x7f19081021f0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:36.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.869+0000 7f190fc05700 1 -- 172.21.15.67:0/1855515487 >> 172.21.15.67:0/1855515487 conn(0x7f19080fb190 msgr2=0x7f1908068580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:26:36.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.869+0000 7f190fc05700 1 -- 172.21.15.67:0/1855515487 shutdown_connections 2024-04-22T23:26:36.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.869+0000 7f190fc05700 1 -- 172.21.15.67:0/1855515487 wait complete. 2024-04-22T23:26:36.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.976+0000 7f3bf31d2700 1 -- 172.21.15.67:0/1883833151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bec1011b0 msgr2=0x7f3bec1015d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:36.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.976+0000 7f3bf31d2700 1 --2- 172.21.15.67:0/1883833151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bec1011b0 0x7f3bec1015d0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f3be0009910 tx=0x7f3be0009c20 comp rx=0 tx=0).stop 2024-04-22T23:26:36.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.976+0000 7f3bf31d2700 1 -- 172.21.15.67:0/1883833151 shutdown_connections 2024-04-22T23:26:36.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.976+0000 7f3bf31d2700 1 --2- 172.21.15.67:0/1883833151 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3bec0ffe10 0x7f3bec100290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:36.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.976+0000 7f3bf31d2700 1 --2- 172.21.15.67:0/1883833151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bec1011b0 0x7f3bec1015d0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:36.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.976+0000 7f3bf31d2700 1 -- 172.21.15.67:0/1883833151 >> 172.21.15.67:0/1883833151 conn(0x7f3bec0fa8c0 msgr2=0x7f3bec0fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:26:36.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.976+0000 7f3bf31d2700 1 -- 172.21.15.67:0/1883833151 shutdown_connections 2024-04-22T23:26:36.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.976+0000 7f3bf31d2700 1 -- 172.21.15.67:0/1883833151 wait complete. 2024-04-22T23:26:36.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.977+0000 7f3bf31d2700 1 Processor -- start 2024-04-22T23:26:36.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.977+0000 7f3bf31d2700 1 -- start start 2024-04-22T23:26:36.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.977+0000 7f3bf31d2700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3bec0ffe10 0x7f3bec1964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:26:36.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.977+0000 7f3bf31d2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bec1011b0 0x7f3bec1969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:26:36.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.977+0000 7f3bf31d2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bec197010 con 0x7f3bec1011b0 2024-04-22T23:26:36.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.977+0000 7f3bf31d2700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bec197150 con 0x7f3bec0ffe10 2024-04-22T23:26:36.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.977+0000 7f3bebfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bec1011b0 0x7f3bec1969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:26:36.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.977+0000 7f3bebfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bec1011b0 0x7f3bec1969f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50050/0 (socket says 172.21.15.67:50050) 2024-04-22T23:26:36.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.977+0000 7f3bebfff700 1 -- 172.21.15.67:0/3198478797 learned_addr learned my addr 172.21.15.67:0/3198478797 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:26:36.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.977+0000 7f3bf0f6e700 1 --2- 172.21.15.67:0/3198478797 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3bec0ffe10 0x7f3bec1964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:26:36.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.977+0000 7f3bebfff700 1 -- 172.21.15.67:0/3198478797 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3bec0ffe10 msgr2=0x7f3bec1964b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:36.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.977+0000 7f3bebfff700 1 --2- 172.21.15.67:0/3198478797 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3bec0ffe10 0x7f3bec1964b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:36.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.977+0000 7f3bebfff700 1 -- 172.21.15.67:0/3198478797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be00095f0 con 0x7f3bec1011b0 2024-04-22T23:26:36.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.978+0000 7f3bebfff700 1 --2- 172.21.15.67:0/3198478797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bec1011b0 0x7f3bec1969f0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f3bd800d900 tx=0x7f3bd800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:26:36.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.978+0000 7f3be9ffb700 1 -- 172.21.15.67:0/3198478797 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3bd8004a00 con 0x7f3bec1011b0 2024-04-22T23:26:36.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.978+0000 7f3be9ffb700 1 -- 172.21.15.67:0/3198478797 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7f3bd80054d0 con 0x7f3bec1011b0 2024-04-22T23:26:36.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.978+0000 7f3be9ffb700 1 -- 172.21.15.67:0/3198478797 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3bd8009d00 con 0x7f3bec1011b0 2024-04-22T23:26:36.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.978+0000 7f3bf31d2700 1 -- 172.21.15.67:0/3198478797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bec19bc00 con 0x7f3bec1011b0 2024-04-22T23:26:36.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.978+0000 7f3bf31d2700 1 -- 172.21.15.67:0/3198478797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bec19c150 con 0x7f3bec1011b0 2024-04-22T23:26:36.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.979+0000 7f3bf31d2700 1 -- 172.21.15.67:0/3198478797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bec108730 con 0x7f3bec1011b0 2024-04-22T23:26:36.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.979+0000 7f3be9ffb700 1 -- 172.21.15.67:0/3198478797 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 33) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f3bd8005020 con 0x7f3bec1011b0 2024-04-22T23:26:36.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.980+0000 7f3be9ffb700 1 --2- 172.21.15.67:0/3198478797 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f3bdc06e940 0x7f3bdc070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:26:36.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.980+0000 7f3be9ffb700 1 -- 172.21.15.67:0/3198478797 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f3bd808e770 con 0x7f3bec1011b0 2024-04-22T23:26:36.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.980+0000 7f3bf0f6e700 1 --2- 172.21.15.67:0/3198478797 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f3bdc06e940 0x7f3bdc070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:26:36.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.980+0000 7f3bf0f6e700 1 --2- 172.21.15.67:0/3198478797 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f3bdc06e940 0x7f3bdc070e00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3be0006010 tx=0x7f3be000b540 comp rx=0 tx=0).ready entity=mgr.14708 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:26:36.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:36.985+0000 7f3be9ffb700 1 -- 172.21.15.67:0/3198478797 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f3bd805b5b0 con 0x7f3bec1011b0 2024-04-22T23:26:37.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.216+0000 7f3bf31d2700 1 -- 172.21.15.67:0/3198478797 --> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3bec0ff410 con 0x7f3bdc06e940 2024-04-22T23:26:37.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.227+0000 7f3be9ffb700 1 -- 172.21.15.67:0/3198478797 <== mgr.14708 v2:172.21.15.67:6800/2306276228 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f3bec0ff410 con 0x7f3bdc06e940 2024-04-22T23:26:37.229 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:26:37.229 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (21s) 14s ago 11m 16.3M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:26:37.229 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (11m) 14s ago 11m 8014k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:26:37.229 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (10m) 52s ago 10m 7931k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:26:37.229 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (11m) 14s ago 11m 6660k - 18.2.1 5be31c24972a 14085634c5d3 2024-04-22T23:26:37.229 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (10m) 52s ago 10m 6665k - 18.2.1 5be31c24972a fa995d458353 2024-04-22T23:26:37.229 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (10m) 14s ago 11m 85.9M - 9.4.7 954c08fa6188 0d8950c9af0c 2024-04-22T23:26:37.229 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (6m) 14s ago 6m 62.4M - 18.2.1 5be31c24972a e24a471f0a8c 2024-04-22T23:26:37.229 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (6m) 14s ago 6m 130M - 18.2.1 5be31c24972a 0730f9fc8e61 2024-04-22T23:26:37.230 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (6m) 52s ago 6m 123M - 18.2.1 5be31c24972a b6a169586433 2024-04-22T23:26:37.230 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (6m) 52s ago 6m 67.5M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:26:37.230 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (2m) 14s ago 13m 499M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:26:37.230 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (73s) 52s ago 10m 449M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:26:37.230 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (13m) 14s ago 13m 58.5M 2048M 18.2.1 5be31c24972a 6789283703e0 2024-04-22T23:26:37.230 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (10m) 52s ago 10m 46.6M 2048M 18.2.1 5be31c24972a 2fb23795f7e3 2024-04-22T23:26:37.230 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (60s) 14s ago 11m 8707k - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:26:37.230 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (56s) 52s ago 10m 3376k - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:26:37.230 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (9m) 14s ago 9m 205M 1216M 18.2.1 5be31c24972a a033247ee8c2 2024-04-22T23:26:37.230 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (9m) 14s ago 9m 210M 1216M 18.2.1 5be31c24972a 915b07a8edc4 2024-04-22T23:26:37.230 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (8m) 14s ago 8m 172M 1216M 18.2.1 5be31c24972a e257ecb4efa1 2024-04-22T23:26:37.231 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (8m) 52s ago 8m 248M 2240M 18.2.1 5be31c24972a 9c2c91a967ba 2024-04-22T23:26:37.231 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (8m) 52s ago 8m 217M 2240M 18.2.1 5be31c24972a c7586562ffa6 2024-04-22T23:26:37.231 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (7m) 52s ago 7m 192M 2240M 18.2.1 5be31c24972a 9fd97f1c65cb 2024-04-22T23:26:37.231 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (39s) 14s ago 11m 51.3M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:26:37.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.229+0000 7f3bf31d2700 1 -- 172.21.15.67:0/3198478797 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f3bdc06e940 msgr2=0x7f3bdc070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:37.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.229+0000 7f3bf31d2700 1 --2- 172.21.15.67:0/3198478797 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f3bdc06e940 0x7f3bdc070e00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3be0006010 tx=0x7f3be000b540 comp rx=0 tx=0).stop 2024-04-22T23:26:37.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.229+0000 7f3bf31d2700 1 -- 172.21.15.67:0/3198478797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bec1011b0 msgr2=0x7f3bec1969f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:37.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.229+0000 7f3bf31d2700 1 --2- 172.21.15.67:0/3198478797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bec1011b0 0x7f3bec1969f0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f3bd800d900 tx=0x7f3bd800dc10 comp rx=0 tx=0).stop 2024-04-22T23:26:37.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.229+0000 7f3bf31d2700 1 -- 172.21.15.67:0/3198478797 shutdown_connections 2024-04-22T23:26:37.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.229+0000 7f3bf31d2700 1 --2- 172.21.15.67:0/3198478797 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f3bdc06e940 0x7f3bdc070e00 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:37.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.229+0000 7f3bf31d2700 1 --2- 172.21.15.67:0/3198478797 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3bec0ffe10 0x7f3bec1964b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:37.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.229+0000 7f3bf31d2700 1 --2- 172.21.15.67:0/3198478797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bec1011b0 0x7f3bec1969f0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:37.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.229+0000 7f3bf31d2700 1 -- 172.21.15.67:0/3198478797 >> 172.21.15.67:0/3198478797 conn(0x7f3bec0fa8c0 msgr2=0x7f3bec0fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:26:37.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.229+0000 7f3bf31d2700 1 -- 172.21.15.67:0/3198478797 shutdown_connections 2024-04-22T23:26:37.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.229+0000 7f3bf31d2700 1 -- 172.21.15.67:0/3198478797 wait complete. 2024-04-22T23:26:37.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.339+0000 7fe0d79e2700 1 -- 172.21.15.67:0/1100048636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0d00ffc10 msgr2=0x7fe0d0100030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:37.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.339+0000 7fe0d79e2700 1 --2- 172.21.15.67:0/1100048636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0d00ffc10 0x7fe0d0100030 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc009910 tx=0x7fe0cc009c20 comp rx=0 tx=0).stop 2024-04-22T23:26:37.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.339+0000 7fe0d79e2700 1 -- 172.21.15.67:0/1100048636 shutdown_connections 2024-04-22T23:26:37.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.339+0000 7fe0d79e2700 1 --2- 172.21.15.67:0/1100048636 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0d0100e10 0x7fe0d0101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:37.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.339+0000 7fe0d79e2700 1 --2- 172.21.15.67:0/1100048636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0d00ffc10 0x7fe0d0100030 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:37.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.339+0000 7fe0d79e2700 1 -- 172.21.15.67:0/1100048636 >> 172.21.15.67:0/1100048636 conn(0x7fe0d00fb190 msgr2=0x7fe0d00fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:26:37.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.339+0000 7fe0d79e2700 1 -- 172.21.15.67:0/1100048636 shutdown_connections 2024-04-22T23:26:37.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.339+0000 7fe0d79e2700 1 -- 172.21.15.67:0/1100048636 wait complete. 2024-04-22T23:26:37.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.340+0000 7fe0d79e2700 1 Processor -- start 2024-04-22T23:26:37.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.340+0000 7fe0d79e2700 1 -- start start 2024-04-22T23:26:37.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.340+0000 7fe0d79e2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0d00ffc10 0x7fe0d01953d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:26:37.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.340+0000 7fe0d79e2700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0d0100e10 0x7fe0d0195910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:26:37.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.340+0000 7fe0d79e2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d0195ea0 con 0x7fe0d00ffc10 2024-04-22T23:26:37.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.340+0000 7fe0d79e2700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d0195fe0 con 0x7fe0d0100e10 2024-04-22T23:26:37.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.340+0000 7fe0d577e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0d00ffc10 0x7fe0d01953d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:26:37.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.341+0000 7fe0d577e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0d00ffc10 0x7fe0d01953d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50058/0 (socket says 172.21.15.67:50058) 2024-04-22T23:26:37.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.341+0000 7fe0d577e700 1 -- 172.21.15.67:0/1621802090 learned_addr learned my addr 172.21.15.67:0/1621802090 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:26:37.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.341+0000 7fe0d4f7d700 1 --2- 172.21.15.67:0/1621802090 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0d0100e10 0x7fe0d0195910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:26:37.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.341+0000 7fe0d577e700 1 -- 172.21.15.67:0/1621802090 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0d0100e10 msgr2=0x7fe0d0195910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:37.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.341+0000 7fe0d577e700 1 --2- 172.21.15.67:0/1621802090 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0d0100e10 0x7fe0d0195910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:37.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.341+0000 7fe0d577e700 1 -- 172.21.15.67:0/1621802090 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0cc0095f0 con 0x7fe0d00ffc10 2024-04-22T23:26:37.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.341+0000 7fe0d4f7d700 1 --2- 172.21.15.67:0/1621802090 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0d0100e10 0x7fe0d0195910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:26:37.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.341+0000 7fe0d577e700 1 --2- 172.21.15.67:0/1621802090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0d00ffc10 0x7fe0d01953d0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc00bda0 tx=0x7fe0cc00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:26:37.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.341+0000 7fe0c67fc700 1 -- 172.21.15.67:0/1621802090 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0cc01d070 con 0x7fe0d00ffc10 2024-04-22T23:26:37.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.341+0000 7fe0c67fc700 1 -- 172.21.15.67:0/1621802090 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fe0cc004c50 con 0x7fe0d00ffc10 2024-04-22T23:26:37.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.341+0000 7fe0d79e2700 1 -- 172.21.15.67:0/1621802090 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0d019aa40 con 0x7fe0d00ffc10 2024-04-22T23:26:37.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.341+0000 7fe0c67fc700 1 -- 172.21.15.67:0/1621802090 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0cc00f690 con 0x7fe0d00ffc10 2024-04-22T23:26:37.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.341+0000 7fe0d79e2700 1 -- 172.21.15.67:0/1621802090 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0d019af00 con 0x7fe0d00ffc10 2024-04-22T23:26:37.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.342+0000 7fe0d79e2700 1 -- 172.21.15.67:0/1621802090 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0d0108700 con 0x7fe0d00ffc10 2024-04-22T23:26:37.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.343+0000 7fe0c67fc700 1 -- 172.21.15.67:0/1621802090 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 33) v1 ==== 92598+0+0 (secure 0 0 0) 0x7fe0cc022a20 con 0x7fe0d00ffc10 2024-04-22T23:26:37.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.343+0000 7fe0c67fc700 1 --2- 172.21.15.67:0/1621802090 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fe0bc06e940 0x7fe0bc070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:26:37.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.343+0000 7fe0c67fc700 1 -- 172.21.15.67:0/1621802090 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fe0cc08ffd0 con 0x7fe0d00ffc10 2024-04-22T23:26:37.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.343+0000 7fe0d4f7d700 1 --2- 172.21.15.67:0/1621802090 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fe0bc06e940 0x7fe0bc070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:26:37.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.344+0000 7fe0d4f7d700 1 --2- 172.21.15.67:0/1621802090 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fe0bc06e940 0x7fe0bc070e00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe0c0005fd0 tx=0x7fe0c0005e30 comp rx=0 tx=0).ready entity=mgr.14708 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:26:37.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.348+0000 7fe0c67fc700 1 -- 172.21.15.67:0/1621802090 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7fe0cc05ce10 con 0x7fe0d00ffc10 2024-04-22T23:26:37.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.650+0000 7fe0d79e2700 1 -- 172.21.15.67:0/1621802090 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe0d019b910 con 0x7fe0d00ffc10 2024-04-22T23:26:37.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.650+0000 7fe0c67fc700 1 -- 172.21.15.67:0/1621802090 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7fe0cc05c9a0 con 0x7fe0d00ffc10 2024-04-22T23:26:37.652 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:26:37.652 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:26:37.652 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-22T23:26:37.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:26:37.652 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:26:37.653 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:26:37.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:26:37.653 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:26:37.653 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-22T23:26:37.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:26:37.653 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:26:37.653 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-22T23:26:37.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:26:37.653 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:26:37.653 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 12, 2024-04-22T23:26:37.653 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:26:37.653 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:26:37.653 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:26:37.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.652+0000 7fe0d79e2700 1 -- 172.21.15.67:0/1621802090 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fe0bc06e940 msgr2=0x7fe0bc070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:37.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.652+0000 7fe0d79e2700 1 --2- 172.21.15.67:0/1621802090 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fe0bc06e940 0x7fe0bc070e00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe0c0005fd0 tx=0x7fe0c0005e30 comp rx=0 tx=0).stop 2024-04-22T23:26:37.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.652+0000 7fe0d79e2700 1 -- 172.21.15.67:0/1621802090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0d00ffc10 msgr2=0x7fe0d01953d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:37.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.652+0000 7fe0d79e2700 1 --2- 172.21.15.67:0/1621802090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0d00ffc10 0x7fe0d01953d0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc00bda0 tx=0x7fe0cc00be80 comp rx=0 tx=0).stop 2024-04-22T23:26:37.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.653+0000 7fe0d79e2700 1 -- 172.21.15.67:0/1621802090 shutdown_connections 2024-04-22T23:26:37.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.653+0000 7fe0d79e2700 1 --2- 172.21.15.67:0/1621802090 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fe0bc06e940 0x7fe0bc070e00 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:37.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.653+0000 7fe0d79e2700 1 --2- 172.21.15.67:0/1621802090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0d00ffc10 0x7fe0d01953d0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:37.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.653+0000 7fe0d79e2700 1 --2- 172.21.15.67:0/1621802090 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0d0100e10 0x7fe0d0195910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:37.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.653+0000 7fe0d79e2700 1 -- 172.21.15.67:0/1621802090 >> 172.21.15.67:0/1621802090 conn(0x7fe0d00fb190 msgr2=0x7fe0d0104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:26:37.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.653+0000 7fe0d79e2700 1 -- 172.21.15.67:0/1621802090 shutdown_connections 2024-04-22T23:26:37.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.653+0000 7fe0d79e2700 1 -- 172.21.15.67:0/1621802090 wait complete. 2024-04-22T23:26:37.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.761+0000 7fed734de700 1 -- 172.21.15.67:0/531042481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed6c104190 msgr2=0x7fed6c106580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:37.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.761+0000 7fed734de700 1 --2- 172.21.15.67:0/531042481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed6c104190 0x7fed6c106580 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fed5c009910 tx=0x7fed5c009c20 comp rx=0 tx=0).stop 2024-04-22T23:26:37.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.761+0000 7fed734de700 1 -- 172.21.15.67:0/531042481 shutdown_connections 2024-04-22T23:26:37.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.761+0000 7fed734de700 1 --2- 172.21.15.67:0/531042481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed6c104190 0x7fed6c106580 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:37.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.761+0000 7fed734de700 1 --2- 172.21.15.67:0/531042481 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed6c101860 0x7fed6c103c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:37.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.761+0000 7fed734de700 1 -- 172.21.15.67:0/531042481 >> 172.21.15.67:0/531042481 conn(0x7fed6c0fb190 msgr2=0x7fed6c0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:26:37.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.761+0000 7fed734de700 1 -- 172.21.15.67:0/531042481 shutdown_connections 2024-04-22T23:26:37.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.761+0000 7fed734de700 1 -- 172.21.15.67:0/531042481 wait complete. 2024-04-22T23:26:37.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.761+0000 7fed734de700 1 Processor -- start 2024-04-22T23:26:37.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.761+0000 7fed734de700 1 -- start start 2024-04-22T23:26:37.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.762+0000 7fed734de700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed6c101860 0x7fed6c196500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:26:37.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.762+0000 7fed734de700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed6c104190 0x7fed6c196a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:26:37.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.762+0000 7fed734de700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed6c197060 con 0x7fed6c101860 2024-04-22T23:26:37.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.762+0000 7fed734de700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed6c1971a0 con 0x7fed6c104190 2024-04-22T23:26:37.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.762+0000 7fed7127a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed6c101860 0x7fed6c196500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:26:37.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.762+0000 7fed7127a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed6c101860 0x7fed6c196500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50074/0 (socket says 172.21.15.67:50074) 2024-04-22T23:26:37.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.762+0000 7fed7127a700 1 -- 172.21.15.67:0/1478411174 learned_addr learned my addr 172.21.15.67:0/1478411174 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:26:37.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.762+0000 7fed70a79700 1 --2- 172.21.15.67:0/1478411174 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed6c104190 0x7fed6c196a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:26:37.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.762+0000 7fed7127a700 1 -- 172.21.15.67:0/1478411174 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed6c104190 msgr2=0x7fed6c196a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:37.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.762+0000 7fed7127a700 1 --2- 172.21.15.67:0/1478411174 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed6c104190 0x7fed6c196a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:37.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.762+0000 7fed7127a700 1 -- 172.21.15.67:0/1478411174 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed5c0095f0 con 0x7fed6c101860 2024-04-22T23:26:37.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.762+0000 7fed7127a700 1 --2- 172.21.15.67:0/1478411174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed6c101860 0x7fed6c196500 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fed6800b4b0 tx=0x7fed6800b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:26:37.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.762+0000 7fed627fc700 1 -- 172.21.15.67:0/1478411174 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed68011800 con 0x7fed6c101860 2024-04-22T23:26:37.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.763+0000 7fed627fc700 1 -- 172.21.15.67:0/1478411174 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1101+0+0 (secure 0 0 0) 0x7fed68011e10 con 0x7fed6c101860 2024-04-22T23:26:37.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.763+0000 7fed734de700 1 -- 172.21.15.67:0/1478411174 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed6c19bc50 con 0x7fed6c101860 2024-04-22T23:26:37.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.763+0000 7fed627fc700 1 -- 172.21.15.67:0/1478411174 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed6800f2e0 con 0x7fed6c101860 2024-04-22T23:26:37.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.763+0000 7fed734de700 1 -- 172.21.15.67:0/1478411174 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed6c19c1a0 con 0x7fed6c101860 2024-04-22T23:26:37.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.763+0000 7fed734de700 1 -- 172.21.15.67:0/1478411174 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed6c190690 con 0x7fed6c101860 2024-04-22T23:26:37.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.764+0000 7fed627fc700 1 -- 172.21.15.67:0/1478411174 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 33) v1 ==== 92598+0+0 (secure 0 0 0) 0x7fed68010380 con 0x7fed6c101860 2024-04-22T23:26:37.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.765+0000 7fed627fc700 1 --2- 172.21.15.67:0/1478411174 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fed5806e940 0x7fed58070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:26:37.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.765+0000 7fed70a79700 1 --2- 172.21.15.67:0/1478411174 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fed5806e940 0x7fed58070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:26:37.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.765+0000 7fed627fc700 1 -- 172.21.15.67:0/1478411174 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fed6808e460 con 0x7fed6c101860 2024-04-22T23:26:37.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.765+0000 7fed70a79700 1 --2- 172.21.15.67:0/1478411174 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fed5806e940 0x7fed58070e00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fed5c006010 tx=0x7fed5c00b540 comp rx=0 tx=0).ready entity=mgr.14708 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:26:37.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.769+0000 7fed627fc700 1 -- 172.21.15.67:0/1478411174 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7fed6805b220 con 0x7fed6c101860 2024-04-22T23:26:37.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:37 smithi067 ceph-mon[26453]: pgmap v51: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-22T23:26:37.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:37 smithi067 ceph-mon[26453]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:26:37.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:37 smithi067 ceph-mon[26453]: from='client.14806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:26:37.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:37 smithi134 ceph-mon[34609]: pgmap v51: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-22T23:26:37.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:37 smithi134 ceph-mon[34609]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:26:37.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:37 smithi134 ceph-mon[34609]: from='client.14806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:26:37.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.998+0000 7fed734de700 1 -- 172.21.15.67:0/1478411174 --> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fed6c0611d0 con 0x7fed5806e940 2024-04-22T23:26:38.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:37.999+0000 7fed627fc700 1 -- 172.21.15.67:0/1478411174 <== mgr.14708 v2:172.21.15.67:6800/2306276228 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+364 (secure 0 0 0) 0x7fed6c0611d0 con 0x7fed5806e940 2024-04-22T23:26:38.001 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:26:38.001 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:26:38.001 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:26:38.001 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-22T23:26:38.002 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:26:38.002 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr" 2024-04-22T23:26:38.002 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:26:38.002 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "2/2 daemons upgraded", 2024-04-22T23:26:38.002 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading grafana daemons", 2024-04-22T23:26:38.002 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:26:38.003 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:26:38.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:38.001+0000 7fed734de700 1 -- 172.21.15.67:0/1478411174 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fed5806e940 msgr2=0x7fed58070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:38.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:38.001+0000 7fed734de700 1 --2- 172.21.15.67:0/1478411174 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fed5806e940 0x7fed58070e00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fed5c006010 tx=0x7fed5c00b540 comp rx=0 tx=0).stop 2024-04-22T23:26:38.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:38.001+0000 7fed734de700 1 -- 172.21.15.67:0/1478411174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed6c101860 msgr2=0x7fed6c196500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:26:38.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:38.001+0000 7fed734de700 1 --2- 172.21.15.67:0/1478411174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed6c101860 0x7fed6c196500 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fed6800b4b0 tx=0x7fed6800b7c0 comp rx=0 tx=0).stop 2024-04-22T23:26:38.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:38.001+0000 7fed734de700 1 -- 172.21.15.67:0/1478411174 shutdown_connections 2024-04-22T23:26:38.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:38.001+0000 7fed734de700 1 --2- 172.21.15.67:0/1478411174 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fed5806e940 0x7fed58070e00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:38.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:38.001+0000 7fed734de700 1 --2- 172.21.15.67:0/1478411174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed6c101860 0x7fed6c196500 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:38.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:38.001+0000 7fed734de700 1 --2- 172.21.15.67:0/1478411174 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed6c104190 0x7fed6c196a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:26:38.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:38.001+0000 7fed734de700 1 -- 172.21.15.67:0/1478411174 >> 172.21.15.67:0/1478411174 conn(0x7fed6c0fb190 msgr2=0x7fed6c0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:26:38.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:38.001+0000 7fed734de700 1 -- 172.21.15.67:0/1478411174 shutdown_connections 2024-04-22T23:26:38.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:26:38.001+0000 7fed734de700 1 -- 172.21.15.67:0/1478411174 wait complete. 2024-04-22T23:26:38.756 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:38 smithi067 ceph-mon[26453]: from='client.14810 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:26:38.756 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:38 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/1621802090' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:38.756 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:38 smithi067 ceph-mon[26453]: from='client.14818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:26:38.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:38 smithi134 ceph-mon[34609]: from='client.14810 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:26:38.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:38 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/1621802090' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:38.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:38 smithi134 ceph-mon[34609]: from='client.14818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:26:39.771 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:39 smithi067 ceph-mon[26453]: pgmap v52: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-22T23:26:39.772 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:39 smithi067 ceph-mon[26453]: fsmap cephfs:2 {0=cephfs.smithi067.xyazmh=up:active,1=cephfs.smithi134.owipad=up:active} 2 up:standby-replay 2024-04-22T23:26:39.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:39 smithi134 ceph-mon[34609]: pgmap v52: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-22T23:26:39.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:39 smithi134 ceph-mon[34609]: fsmap cephfs:2 {0=cephfs.smithi067.xyazmh=up:active,1=cephfs.smithi134.owipad=up:active} 2 up:standby-replay 2024-04-22T23:26:41.414 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:41 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:41.414 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:41 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:41.414 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:41 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:26:41.414 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:41 smithi067 ceph-mon[26453]: pgmap v53: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-22T23:26:41.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:41 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:41.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:41 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:41.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:41 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:26:41.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:41 smithi134 ceph-mon[34609]: pgmap v53: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-22T23:26:43.750 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:43 smithi067 ceph-mon[26453]: pgmap v54: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-22T23:26:43.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:43 smithi134 ceph-mon[34609]: pgmap v54: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-22T23:26:45.794 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:45 smithi067 ceph-mon[26453]: pgmap v55: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:26:45.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:45 smithi134 ceph-mon[34609]: pgmap v55: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:26:46.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:46 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:26:46.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:46 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:46.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:46 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:47.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:46 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:26:47.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:46 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:47.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:46 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:48.012 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:47 smithi067 ceph-mon[26453]: pgmap v56: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:26:48.012 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:47 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:48.012 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:47 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:48.243 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:47 smithi134 ceph-mon[34609]: pgmap v56: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:26:48.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:47 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:48.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:47 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:50.243 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:49 smithi134 ceph-mon[34609]: pgmap v57: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:26:50.294 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:49 smithi067 ceph-mon[26453]: pgmap v57: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:26:51.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:51 smithi134 ceph-mon[34609]: pgmap v58: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:26:51.794 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:51 smithi067 ceph-mon[26453]: pgmap v58: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:26:53.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:53.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:53.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:26:53.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:26:53.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:53.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T23:26:53.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T23:26:53.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:26:53.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:53.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:53.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:53.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:53.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:53.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:53.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:53.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:53.495 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:53.496 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:53.496 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:53.496 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:53.496 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:53.496 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: Upgrade: Finalizing container_image settings 2024-04-22T23:26:53.496 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:53.496 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-22T23:26:53.496 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-22T23:26:53.496 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T23:26:53.496 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-22T23:26:53.496 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-22T23:26:53.496 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-22T23:26:53.496 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-22T23:26:53.496 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-22T23:26:53.497 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-22T23:26:53.497 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-22T23:26:53.497 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-22T23:26:53.497 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T23:26:53.497 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-22T23:26:53.497 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-22T23:26:53.497 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-22T23:26:53.497 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-22T23:26:53.497 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-22T23:26:53.497 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-22T23:26:53.497 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-22T23:26:53.497 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T23:26:53.497 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T23:26:53.497 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T23:26:53.497 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T23:26:53.498 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T23:26:53.498 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T23:26:53.498 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: Upgrade: Complete! 2024-04-22T23:26:53.498 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-22T23:26:53.498 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: pgmap v59: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:26:53.498 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-22T23:26:53.498 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:26:53.498 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:26:53.498 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:26:53.498 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:53.498 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:26:53.498 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:26:53.498 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:26:53.498 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:53 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:53.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:53.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:53.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:26:53.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:26:53.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:53.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T23:26:53.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T23:26:53.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:26:53.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:53.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:53.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:53.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:53.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:53.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:53.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:53.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:53.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:53.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:53.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:53.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:53.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:26:53.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: Upgrade: Finalizing container_image settings 2024-04-22T23:26:53.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:53.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-22T23:26:53.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-22T23:26:53.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T23:26:53.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-22T23:26:53.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-22T23:26:53.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-22T23:26:53.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-22T23:26:53.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-22T23:26:53.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-22T23:26:53.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-22T23:26:53.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-22T23:26:53.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T23:26:53.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-22T23:26:53.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-22T23:26:53.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-22T23:26:53.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-22T23:26:53.548 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-22T23:26:53.548 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-22T23:26:53.548 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-22T23:26:53.548 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T23:26:53.548 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T23:26:53.548 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T23:26:53.548 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T23:26:53.548 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T23:26:53.548 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-22T23:26:53.548 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: Upgrade: Complete! 2024-04-22T23:26:53.548 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-22T23:26:53.548 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: pgmap v59: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:26:53.548 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-22T23:26:53.548 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:26:53.548 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:26:53.549 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:26:53.549 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:53.549 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:26:53.549 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:26:53.549 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:26:53.549 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:53 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:55.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:55 smithi134 ceph-mon[34609]: pgmap v60: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:26:56.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:55 smithi067 ceph-mon[26453]: pgmap v60: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:26:56.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:56 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:57.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:56 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:26:57.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:57 smithi134 ceph-mon[34609]: pgmap v61: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:26:58.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:57 smithi067 ceph-mon[26453]: pgmap v61: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:26:59.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:26:59 smithi134 ceph-mon[34609]: pgmap v62: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:27:00.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:26:59 smithi067 ceph-mon[26453]: pgmap v62: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:27:01.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:01 smithi134 ceph-mon[34609]: pgmap v63: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:27:01.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:01 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:27:01.794 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:01 smithi067 ceph-mon[26453]: pgmap v63: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:27:01.794 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:01 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:27:03.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:03 smithi134 ceph-mon[34609]: pgmap v64: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:27:03.794 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:03 smithi067 ceph-mon[26453]: pgmap v64: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:27:05.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:05 smithi134 ceph-mon[34609]: pgmap v65: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:27:06.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:05 smithi067 ceph-mon[26453]: pgmap v65: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:27:07.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:07 smithi134 ceph-mon[34609]: pgmap v66: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:27:08.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:07 smithi067 ceph-mon[26453]: pgmap v66: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:27:08.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.125+0000 7ff9f030d700 1 -- 172.21.15.67:0/830289574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9e80ffbc0 msgr2=0x7ff9e80fffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:08.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.125+0000 7ff9f030d700 1 --2- 172.21.15.67:0/830289574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9e80ffbc0 0x7ff9e80fffe0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ff9e4009910 tx=0x7ff9e4009c20 comp rx=0 tx=0).stop 2024-04-22T23:27:08.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.125+0000 7ff9f030d700 1 -- 172.21.15.67:0/830289574 shutdown_connections 2024-04-22T23:27:08.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.125+0000 7ff9f030d700 1 --2- 172.21.15.67:0/830289574 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9e8100dc0 0x7ff9e8101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:08.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.125+0000 7ff9f030d700 1 --2- 172.21.15.67:0/830289574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9e80ffbc0 0x7ff9e80fffe0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:08.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.125+0000 7ff9f030d700 1 -- 172.21.15.67:0/830289574 >> 172.21.15.67:0/830289574 conn(0x7ff9e80fb160 msgr2=0x7ff9e80fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:27:08.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.125+0000 7ff9f030d700 1 -- 172.21.15.67:0/830289574 shutdown_connections 2024-04-22T23:27:08.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.125+0000 7ff9f030d700 1 -- 172.21.15.67:0/830289574 wait complete. 2024-04-22T23:27:08.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.126+0000 7ff9f030d700 1 Processor -- start 2024-04-22T23:27:08.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.126+0000 7ff9f030d700 1 -- start start 2024-04-22T23:27:08.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.126+0000 7ff9f030d700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9e80ffbc0 0x7ff9e81967f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:08.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.126+0000 7ff9f030d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9e8100dc0 0x7ff9e8196d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:08.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.126+0000 7ff9f030d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e81972c0 con 0x7ff9e8100dc0 2024-04-22T23:27:08.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.126+0000 7ff9f030d700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e8197400 con 0x7ff9e80ffbc0 2024-04-22T23:27:08.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.126+0000 7ff9ed8a8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9e8100dc0 0x7ff9e8196d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:08.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.126+0000 7ff9ed8a8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9e8100dc0 0x7ff9e8196d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33914/0 (socket says 172.21.15.67:33914) 2024-04-22T23:27:08.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.127+0000 7ff9ed8a8700 1 -- 172.21.15.67:0/1740816648 learned_addr learned my addr 172.21.15.67:0/1740816648 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:27:08.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.127+0000 7ff9ed8a8700 1 -- 172.21.15.67:0/1740816648 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9e80ffbc0 msgr2=0x7ff9e81967f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T23:27:08.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.127+0000 7ff9ed8a8700 1 --2- 172.21.15.67:0/1740816648 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9e80ffbc0 0x7ff9e81967f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:08.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.127+0000 7ff9ed8a8700 1 -- 172.21.15.67:0/1740816648 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e40095f0 con 0x7ff9e8100dc0 2024-04-22T23:27:08.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.127+0000 7ff9ed8a8700 1 --2- 172.21.15.67:0/1740816648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9e8100dc0 0x7ff9e8196d30 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7ff9d800c930 tx=0x7ff9d800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:27:08.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.127+0000 7ff9df7fe700 1 -- 172.21.15.67:0/1740816648 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9d8007ad0 con 0x7ff9e8100dc0 2024-04-22T23:27:08.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.127+0000 7ff9df7fe700 1 -- 172.21.15.67:0/1740816648 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff9d8007c30 con 0x7ff9e8100dc0 2024-04-22T23:27:08.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.127+0000 7ff9df7fe700 1 -- 172.21.15.67:0/1740816648 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9d8018610 con 0x7ff9e8100dc0 2024-04-22T23:27:08.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.127+0000 7ff9f030d700 1 -- 172.21.15.67:0/1740816648 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e819bec0 con 0x7ff9e8100dc0 2024-04-22T23:27:08.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.127+0000 7ff9f030d700 1 -- 172.21.15.67:0/1740816648 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e819c390 con 0x7ff9e8100dc0 2024-04-22T23:27:08.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.128+0000 7ff9f030d700 1 -- 172.21.15.67:0/1740816648 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e8068af0 con 0x7ff9e8100dc0 2024-04-22T23:27:08.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.129+0000 7ff9df7fe700 1 -- 172.21.15.67:0/1740816648 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 33) v1 ==== 92598+0+0 (secure 0 0 0) 0x7ff9d800f450 con 0x7ff9e8100dc0 2024-04-22T23:27:08.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.129+0000 7ff9df7fe700 1 --2- 172.21.15.67:0/1740816648 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7ff9d406e940 0x7ff9d4070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:08.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.129+0000 7ff9df7fe700 1 -- 172.21.15.67:0/1740816648 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7ff9d808eb10 con 0x7ff9e8100dc0 2024-04-22T23:27:08.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.129+0000 7ff9ee0a9700 1 --2- 172.21.15.67:0/1740816648 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7ff9d406e940 0x7ff9d4070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:08.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.130+0000 7ff9ee0a9700 1 --2- 172.21.15.67:0/1740816648 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7ff9d406e940 0x7ff9d4070e00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff9e400b5c0 tx=0x7ff9e4005fb0 comp rx=0 tx=0).ready entity=mgr.14708 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:27:08.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.134+0000 7ff9df7fe700 1 -- 172.21.15.67:0/1740816648 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7ff9d805b8d0 con 0x7ff9e8100dc0 2024-04-22T23:27:08.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.364+0000 7ff9f030d700 1 -- 172.21.15.67:0/1740816648 --> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff9e80695e0 con 0x7ff9d406e940 2024-04-22T23:27:08.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.366+0000 7ff9df7fe700 1 -- 172.21.15.67:0/1740816648 <== mgr.14708 v2:172.21.15.67:6800/2306276228 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7ff9e80695e0 con 0x7ff9d406e940 2024-04-22T23:27:08.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.368+0000 7ff9f030d700 1 -- 172.21.15.67:0/1740816648 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7ff9d406e940 msgr2=0x7ff9d4070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:08.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.368+0000 7ff9f030d700 1 --2- 172.21.15.67:0/1740816648 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7ff9d406e940 0x7ff9d4070e00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff9e400b5c0 tx=0x7ff9e4005fb0 comp rx=0 tx=0).stop 2024-04-22T23:27:08.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.368+0000 7ff9f030d700 1 -- 172.21.15.67:0/1740816648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9e8100dc0 msgr2=0x7ff9e8196d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:08.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.368+0000 7ff9f030d700 1 --2- 172.21.15.67:0/1740816648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9e8100dc0 0x7ff9e8196d30 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7ff9d800c930 tx=0x7ff9d800ccf0 comp rx=0 tx=0).stop 2024-04-22T23:27:08.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.368+0000 7ff9f030d700 1 -- 172.21.15.67:0/1740816648 shutdown_connections 2024-04-22T23:27:08.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.368+0000 7ff9f030d700 1 --2- 172.21.15.67:0/1740816648 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7ff9d406e940 0x7ff9d4070e00 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:08.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.368+0000 7ff9f030d700 1 --2- 172.21.15.67:0/1740816648 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9e80ffbc0 0x7ff9e81967f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:08.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.368+0000 7ff9f030d700 1 --2- 172.21.15.67:0/1740816648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9e8100dc0 0x7ff9e8196d30 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:08.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.368+0000 7ff9f030d700 1 -- 172.21.15.67:0/1740816648 >> 172.21.15.67:0/1740816648 conn(0x7ff9e80fb160 msgr2=0x7ff9e8068e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:27:08.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.368+0000 7ff9f030d700 1 -- 172.21.15.67:0/1740816648 shutdown_connections 2024-04-22T23:27:08.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:08.368+0000 7ff9f030d700 1 -- 172.21.15.67:0/1740816648 wait complete. 2024-04-22T23:27:08.999 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -e sha1=43be020184947e53516056c9931e1ac5bdbbb1a5 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-04-22T23:27:09.255 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:27:09.688 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:09 smithi067 ceph-mon[26453]: from='client.14822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:27:09.688 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:09 smithi067 ceph-mon[26453]: pgmap v67: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:27:09.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:09 smithi134 ceph-mon[34609]: from='client.14822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:27:09.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:09 smithi134 ceph-mon[34609]: pgmap v67: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:27:10.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:10.929+0000 7f03d2bc3700 1 -- 172.21.15.67:0/673292042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03cc0ffc20 msgr2=0x7f03cc100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:10.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:10.929+0000 7f03d2bc3700 1 --2- 172.21.15.67:0/673292042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03cc0ffc20 0x7f03cc100080 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f03c0009910 tx=0x7f03c0009c20 comp rx=0 tx=0).stop 2024-04-22T23:27:10.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:10.929+0000 7f03d2bc3700 1 -- 172.21.15.67:0/673292042 shutdown_connections 2024-04-22T23:27:10.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:10.929+0000 7f03d2bc3700 1 --2- 172.21.15.67:0/673292042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03cc0ffc20 0x7f03cc100080 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:10.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:10.929+0000 7f03d2bc3700 1 --2- 172.21.15.67:0/673292042 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03cc105bd0 0x7f03cc105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:10.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:10.929+0000 7f03d2bc3700 1 -- 172.21.15.67:0/673292042 >> 172.21.15.67:0/673292042 conn(0x7f03cc075670 msgr2=0x7f03cc075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:27:10.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:10.929+0000 7f03d2bc3700 1 -- 172.21.15.67:0/673292042 shutdown_connections 2024-04-22T23:27:10.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:10.929+0000 7f03d2bc3700 1 -- 172.21.15.67:0/673292042 wait complete. 2024-04-22T23:27:10.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:10.930+0000 7f03d2bc3700 1 Processor -- start 2024-04-22T23:27:10.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:10.930+0000 7f03d2bc3700 1 -- start start 2024-04-22T23:27:10.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:10.930+0000 7f03d2bc3700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03cc0ffc20 0x7f03cc19c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:10.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:10.930+0000 7f03d2bc3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03cc105bd0 0x7f03cc19cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:10.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:10.930+0000 7f03d2bc3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc19d230 con 0x7f03cc105bd0 2024-04-22T23:27:10.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:10.930+0000 7f03d2bc3700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc1966e0 con 0x7f03cc0ffc20 2024-04-22T23:27:10.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:10.930+0000 7f03cbfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03cc105bd0 0x7f03cc19cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:10.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:10.930+0000 7f03cbfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03cc105bd0 0x7f03cc19cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34152/0 (socket says 172.21.15.67:34152) 2024-04-22T23:27:10.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:10.930+0000 7f03cbfff700 1 -- 172.21.15.67:0/2991515757 learned_addr learned my addr 172.21.15.67:0/2991515757 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:27:10.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:10.930+0000 7f03d095f700 1 --2- 172.21.15.67:0/2991515757 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03cc0ffc20 0x7f03cc19c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:10.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:10.930+0000 7f03cbfff700 1 -- 172.21.15.67:0/2991515757 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03cc0ffc20 msgr2=0x7f03cc19c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:10.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:10.930+0000 7f03cbfff700 1 --2- 172.21.15.67:0/2991515757 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03cc0ffc20 0x7f03cc19c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:10.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:10.930+0000 7f03cbfff700 1 -- 172.21.15.67:0/2991515757 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c00095f0 con 0x7f03cc105bd0 2024-04-22T23:27:10.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:10.931+0000 7f03cbfff700 1 --2- 172.21.15.67:0/2991515757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03cc105bd0 0x7f03cc19cba0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f03c0006010 tx=0x7f03c000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:27:10.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:10.931+0000 7f03c9ffb700 1 -- 172.21.15.67:0/2991515757 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03c001d070 con 0x7f03cc105bd0 2024-04-22T23:27:10.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:10.931+0000 7f03c9ffb700 1 -- 172.21.15.67:0/2991515757 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f03c000f460 con 0x7f03cc105bd0 2024-04-22T23:27:10.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:10.931+0000 7f03d2bc3700 1 -- 172.21.15.67:0/2991515757 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03cc196960 con 0x7f03cc105bd0 2024-04-22T23:27:10.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:10.931+0000 7f03c9ffb700 1 -- 172.21.15.67:0/2991515757 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03c0017630 con 0x7f03cc105bd0 2024-04-22T23:27:10.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:10.931+0000 7f03d2bc3700 1 -- 172.21.15.67:0/2991515757 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03cc196e50 con 0x7f03cc105bd0 2024-04-22T23:27:10.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:10.932+0000 7f03d2bc3700 1 -- 172.21.15.67:0/2991515757 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03cc04f2e0 con 0x7f03cc105bd0 2024-04-22T23:27:10.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:10.933+0000 7f03c9ffb700 1 -- 172.21.15.67:0/2991515757 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 33) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f03c000f5d0 con 0x7f03cc105bd0 2024-04-22T23:27:10.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:10.933+0000 7f03c9ffb700 1 --2- 172.21.15.67:0/2991515757 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f03b406e940 0x7f03b4070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:10.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:10.933+0000 7f03c9ffb700 1 -- 172.21.15.67:0/2991515757 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f03c008fd20 con 0x7f03cc105bd0 2024-04-22T23:27:10.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:10.933+0000 7f03d095f700 1 --2- 172.21.15.67:0/2991515757 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f03b406e940 0x7f03b4070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:10.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:10.934+0000 7f03d095f700 1 --2- 172.21.15.67:0/2991515757 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f03b406e940 0x7f03b4070e00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f03bc005950 tx=0x7f03bc0058e0 comp rx=0 tx=0).ready entity=mgr.14708 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:27:10.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:10.938+0000 7f03c9ffb700 1 -- 172.21.15.67:0/2991515757 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f03c005cb60 con 0x7f03cc105bd0 2024-04-22T23:27:11.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:11.234+0000 7f03d2bc3700 1 -- 172.21.15.67:0/2991515757 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f03cc197bf0 con 0x7f03cc105bd0 2024-04-22T23:27:11.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:11.234+0000 7f03c9ffb700 1 -- 172.21.15.67:0/2991515757 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7f03c005c6f0 con 0x7f03cc105bd0 2024-04-22T23:27:11.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:11.236+0000 7f03d2bc3700 1 -- 172.21.15.67:0/2991515757 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f03b406e940 msgr2=0x7f03b4070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:11.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:11.236+0000 7f03d2bc3700 1 --2- 172.21.15.67:0/2991515757 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f03b406e940 0x7f03b4070e00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f03bc005950 tx=0x7f03bc0058e0 comp rx=0 tx=0).stop 2024-04-22T23:27:11.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:11.236+0000 7f03d2bc3700 1 -- 172.21.15.67:0/2991515757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03cc105bd0 msgr2=0x7f03cc19cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:11.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:11.236+0000 7f03d2bc3700 1 --2- 172.21.15.67:0/2991515757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03cc105bd0 0x7f03cc19cba0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f03c0006010 tx=0x7f03c000be20 comp rx=0 tx=0).stop 2024-04-22T23:27:11.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:11.237+0000 7f03d2bc3700 1 -- 172.21.15.67:0/2991515757 shutdown_connections 2024-04-22T23:27:11.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:11.237+0000 7f03d2bc3700 1 --2- 172.21.15.67:0/2991515757 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f03b406e940 0x7f03b4070e00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:11.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:11.237+0000 7f03d2bc3700 1 --2- 172.21.15.67:0/2991515757 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03cc0ffc20 0x7f03cc19c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:11.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:11.237+0000 7f03d2bc3700 1 --2- 172.21.15.67:0/2991515757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03cc105bd0 0x7f03cc19cba0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:11.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:11.237+0000 7f03d2bc3700 1 -- 172.21.15.67:0/2991515757 >> 172.21.15.67:0/2991515757 conn(0x7f03cc075670 msgr2=0x7f03cc0686a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:27:11.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:11.237+0000 7f03d2bc3700 1 -- 172.21.15.67:0/2991515757 shutdown_connections 2024-04-22T23:27:11.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:11.237+0000 7f03d2bc3700 1 -- 172.21.15.67:0/2991515757 wait complete. 2024-04-22T23:27:11.247 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:27:11.634 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:11 smithi067 ceph-mon[26453]: pgmap v68: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:27:11.634 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:11 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/2991515757' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:27:11.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:11 smithi134 ceph-mon[34609]: pgmap v68: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:27:11.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:11 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/2991515757' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:27:11.940 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -e sha1=43be020184947e53516056c9931e1ac5bdbbb1a5 -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-04-22T23:27:12.158 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:27:13.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:13 smithi134 ceph-mon[34609]: pgmap v69: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:27:13.794 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:13 smithi067 ceph-mon[26453]: pgmap v69: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:27:13.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:13.823+0000 7fc95a240700 1 -- 172.21.15.67:0/55440214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc954068580 msgr2=0x7fc9540689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:13.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:13.823+0000 7fc95a240700 1 --2- 172.21.15.67:0/55440214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc954068580 0x7fc9540689e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc948009910 tx=0x7fc948009c20 comp rx=0 tx=0).stop 2024-04-22T23:27:13.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:13.824+0000 7fc95a240700 1 -- 172.21.15.67:0/55440214 shutdown_connections 2024-04-22T23:27:13.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:13.824+0000 7fc95a240700 1 --2- 172.21.15.67:0/55440214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc954068580 0x7fc9540689e0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:13.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:13.824+0000 7fc95a240700 1 --2- 172.21.15.67:0/55440214 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc954106cd0 0x7fc9541070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:13.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:13.824+0000 7fc95a240700 1 -- 172.21.15.67:0/55440214 >> 172.21.15.67:0/55440214 conn(0x7fc954075670 msgr2=0x7fc954075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:27:13.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:13.824+0000 7fc95a240700 1 -- 172.21.15.67:0/55440214 shutdown_connections 2024-04-22T23:27:13.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:13.824+0000 7fc95a240700 1 -- 172.21.15.67:0/55440214 wait complete. 2024-04-22T23:27:13.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:13.824+0000 7fc95a240700 1 Processor -- start 2024-04-22T23:27:13.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:13.824+0000 7fc95a240700 1 -- start start 2024-04-22T23:27:13.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:13.825+0000 7fc95a240700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc954068580 0x7fc954196980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:13.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:13.825+0000 7fc95a240700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc954106cd0 0x7fc954196ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:13.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:13.825+0000 7fc95a240700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc954197550 con 0x7fc954068580 2024-04-22T23:27:13.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:13.825+0000 7fc95a240700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc95419b360 con 0x7fc954106cd0 2024-04-22T23:27:13.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:13.825+0000 7fc9537fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc954068580 0x7fc954196980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:13.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:13.825+0000 7fc9537fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc954068580 0x7fc954196980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34176/0 (socket says 172.21.15.67:34176) 2024-04-22T23:27:13.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:13.825+0000 7fc9537fe700 1 -- 172.21.15.67:0/2113134611 learned_addr learned my addr 172.21.15.67:0/2113134611 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:27:13.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:13.825+0000 7fc952ffd700 1 --2- 172.21.15.67:0/2113134611 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc954106cd0 0x7fc954196ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:13.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:13.825+0000 7fc9537fe700 1 -- 172.21.15.67:0/2113134611 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc954106cd0 msgr2=0x7fc954196ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:13.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:13.825+0000 7fc9537fe700 1 --2- 172.21.15.67:0/2113134611 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc954106cd0 0x7fc954196ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:13.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:13.825+0000 7fc9537fe700 1 -- 172.21.15.67:0/2113134611 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9480095f0 con 0x7fc954068580 2024-04-22T23:27:13.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:13.825+0000 7fc9537fe700 1 --2- 172.21.15.67:0/2113134611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc954068580 0x7fc954196980 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fc94400b4b0 tx=0x7fc94400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:27:13.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:13.825+0000 7fc950ff9700 1 -- 172.21.15.67:0/2113134611 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc944011800 con 0x7fc954068580 2024-04-22T23:27:13.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:13.825+0000 7fc950ff9700 1 -- 172.21.15.67:0/2113134611 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc944011e20 con 0x7fc954068580 2024-04-22T23:27:13.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:13.825+0000 7fc950ff9700 1 -- 172.21.15.67:0/2113134611 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc94400f2e0 con 0x7fc954068580 2024-04-22T23:27:13.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:13.825+0000 7fc95a240700 1 -- 172.21.15.67:0/2113134611 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc95419b640 con 0x7fc954068580 2024-04-22T23:27:13.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:13.825+0000 7fc95a240700 1 -- 172.21.15.67:0/2113134611 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc95419bb90 con 0x7fc954068580 2024-04-22T23:27:13.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:13.826+0000 7fc95a240700 1 -- 172.21.15.67:0/2113134611 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9541091d0 con 0x7fc954068580 2024-04-22T23:27:13.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:13.827+0000 7fc950ff9700 1 -- 172.21.15.67:0/2113134611 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 33) v1 ==== 92598+0+0 (secure 0 0 0) 0x7fc944010380 con 0x7fc954068580 2024-04-22T23:27:13.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:13.827+0000 7fc950ff9700 1 --2- 172.21.15.67:0/2113134611 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fc93c06e940 0x7fc93c070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:13.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:13.828+0000 7fc950ff9700 1 -- 172.21.15.67:0/2113134611 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fc94408e490 con 0x7fc954068580 2024-04-22T23:27:13.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:13.828+0000 7fc952ffd700 1 --2- 172.21.15.67:0/2113134611 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fc93c06e940 0x7fc93c070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:13.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:13.828+0000 7fc952ffd700 1 --2- 172.21.15.67:0/2113134611 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fc93c06e940 0x7fc93c070e00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc954197fa0 tx=0x7fc94801a040 comp rx=0 tx=0).ready entity=mgr.14708 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:27:13.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:13.832+0000 7fc950ff9700 1 -- 172.21.15.67:0/2113134611 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7fc94405b250 con 0x7fc954068580 2024-04-22T23:27:14.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:14.127+0000 7fc95a240700 1 -- 172.21.15.67:0/2113134611 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc95404f2e0 con 0x7fc954068580 2024-04-22T23:27:14.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:14.128+0000 7fc950ff9700 1 -- 172.21.15.67:0/2113134611 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7fc94405ade0 con 0x7fc954068580 2024-04-22T23:27:14.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:14.130+0000 7fc95a240700 1 -- 172.21.15.67:0/2113134611 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fc93c06e940 msgr2=0x7fc93c070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:14.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:14.130+0000 7fc95a240700 1 --2- 172.21.15.67:0/2113134611 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fc93c06e940 0x7fc93c070e00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc954197fa0 tx=0x7fc94801a040 comp rx=0 tx=0).stop 2024-04-22T23:27:14.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:14.130+0000 7fc95a240700 1 -- 172.21.15.67:0/2113134611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc954068580 msgr2=0x7fc954196980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:14.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:14.130+0000 7fc95a240700 1 --2- 172.21.15.67:0/2113134611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc954068580 0x7fc954196980 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fc94400b4b0 tx=0x7fc94400b870 comp rx=0 tx=0).stop 2024-04-22T23:27:14.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:14.130+0000 7fc95a240700 1 -- 172.21.15.67:0/2113134611 shutdown_connections 2024-04-22T23:27:14.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:14.130+0000 7fc95a240700 1 --2- 172.21.15.67:0/2113134611 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fc93c06e940 0x7fc93c070e00 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:14.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:14.130+0000 7fc95a240700 1 --2- 172.21.15.67:0/2113134611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc954068580 0x7fc954196980 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:14.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:14.130+0000 7fc95a240700 1 --2- 172.21.15.67:0/2113134611 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc954106cd0 0x7fc954196ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:14.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:14.130+0000 7fc95a240700 1 -- 172.21.15.67:0/2113134611 >> 172.21.15.67:0/2113134611 conn(0x7fc954075670 msgr2=0x7fc9540fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:27:14.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:14.130+0000 7fc95a240700 1 -- 172.21.15.67:0/2113134611 shutdown_connections 2024-04-22T23:27:14.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:14.131+0000 7fc95a240700 1 -- 172.21.15.67:0/2113134611 wait complete. 2024-04-22T23:27:14.142 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)" 2024-04-22T23:27:14.749 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:14 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/2113134611' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:27:14.771 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -e sha1=43be020184947e53516056c9931e1ac5bdbbb1a5 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 2'"'"'' 2024-04-22T23:27:14.987 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:27:14.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:14 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/2113134611' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:27:15.794 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:15 smithi067 ceph-mon[26453]: pgmap v70: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:27:15.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:15 smithi134 ceph-mon[34609]: pgmap v70: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:27:16.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.616+0000 7fabf538a700 1 -- 172.21.15.67:0/3966705121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabf0101850 msgr2=0x7fabf01058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:16.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.616+0000 7fabf538a700 1 --2- 172.21.15.67:0/3966705121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabf0101850 0x7fabf01058a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fabd8009910 tx=0x7fabd8009c20 comp rx=0 tx=0).stop 2024-04-22T23:27:16.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.616+0000 7fabf538a700 1 -- 172.21.15.67:0/3966705121 shutdown_connections 2024-04-22T23:27:16.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.616+0000 7fabf538a700 1 --2- 172.21.15.67:0/3966705121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabf0101850 0x7fabf01058a0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:16.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.616+0000 7fabf538a700 1 --2- 172.21.15.67:0/3966705121 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabf0100ea0 0x7fabf0101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:16.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.616+0000 7fabf538a700 1 -- 172.21.15.67:0/3966705121 >> 172.21.15.67:0/3966705121 conn(0x7fabf00fc750 msgr2=0x7fabf00feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:27:16.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.616+0000 7fabf538a700 1 -- 172.21.15.67:0/3966705121 shutdown_connections 2024-04-22T23:27:16.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.616+0000 7fabf538a700 1 -- 172.21.15.67:0/3966705121 wait complete. 2024-04-22T23:27:16.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.616+0000 7fabf538a700 1 Processor -- start 2024-04-22T23:27:16.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.617+0000 7fabf538a700 1 -- start start 2024-04-22T23:27:16.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.617+0000 7fabf538a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabf0100ea0 0x7fabf0073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:16.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.617+0000 7fabf538a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabf0101850 0x7fabf00735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:16.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.617+0000 7fabf538a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf0073b10 con 0x7fabf0101850 2024-04-22T23:27:16.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.617+0000 7fabf538a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf0073c50 con 0x7fabf0100ea0 2024-04-22T23:27:16.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.617+0000 7fabee7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabf0101850 0x7fabf00735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:16.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.617+0000 7fabee7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabf0101850 0x7fabf00735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34190/0 (socket says 172.21.15.67:34190) 2024-04-22T23:27:16.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.617+0000 7fabee7fc700 1 -- 172.21.15.67:0/742111781 learned_addr learned my addr 172.21.15.67:0/742111781 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:27:16.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.617+0000 7fabeeffd700 1 --2- 172.21.15.67:0/742111781 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabf0100ea0 0x7fabf0073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:16.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.617+0000 7fabee7fc700 1 -- 172.21.15.67:0/742111781 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabf0100ea0 msgr2=0x7fabf0073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:16.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.617+0000 7fabee7fc700 1 --2- 172.21.15.67:0/742111781 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabf0100ea0 0x7fabf0073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:16.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.617+0000 7fabee7fc700 1 -- 172.21.15.67:0/742111781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabd80095f0 con 0x7fabf0101850 2024-04-22T23:27:16.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.617+0000 7fabee7fc700 1 --2- 172.21.15.67:0/742111781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabf0101850 0x7fabf00735d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fabd80092e0 tx=0x7fabd800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:27:16.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.618+0000 7fabe7fff700 1 -- 172.21.15.67:0/742111781 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabd801d070 con 0x7fabf0101850 2024-04-22T23:27:16.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.618+0000 7fabe7fff700 1 -- 172.21.15.67:0/742111781 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fabd800f460 con 0x7fabf0101850 2024-04-22T23:27:16.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.618+0000 7fabf538a700 1 -- 172.21.15.67:0/742111781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabf0073ed0 con 0x7fabf0101850 2024-04-22T23:27:16.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.618+0000 7fabe7fff700 1 -- 172.21.15.67:0/742111781 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabd80175d0 con 0x7fabf0101850 2024-04-22T23:27:16.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.618+0000 7fabf538a700 1 -- 172.21.15.67:0/742111781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabf01a2eb0 con 0x7fabf0101850 2024-04-22T23:27:16.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.619+0000 7fabf538a700 1 -- 172.21.15.67:0/742111781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabf01091f0 con 0x7fabf0101850 2024-04-22T23:27:16.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.619+0000 7fabe7fff700 1 -- 172.21.15.67:0/742111781 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 33) v1 ==== 92598+0+0 (secure 0 0 0) 0x7fabd800f5d0 con 0x7fabf0101850 2024-04-22T23:27:16.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.620+0000 7fabe7fff700 1 --2- 172.21.15.67:0/742111781 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fabdc06e940 0x7fabdc070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:16.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.620+0000 7fabe7fff700 1 -- 172.21.15.67:0/742111781 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fabd808fca0 con 0x7fabf0101850 2024-04-22T23:27:16.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.620+0000 7fabeeffd700 1 --2- 172.21.15.67:0/742111781 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fabdc06e940 0x7fabdc070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:16.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.620+0000 7fabeeffd700 1 --2- 172.21.15.67:0/742111781 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fabdc06e940 0x7fabdc070e00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fabe800a400 tx=0x7fabe800d040 comp rx=0 tx=0).ready entity=mgr.14708 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:27:16.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.624+0000 7fabe7fff700 1 -- 172.21.15.67:0/742111781 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7fabd8094070 con 0x7fabf0101850 2024-04-22T23:27:16.794 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:16 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:27:16.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.933+0000 7fabf538a700 1 -- 172.21.15.67:0/742111781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fabf004f2e0 con 0x7fabf0101850 2024-04-22T23:27:16.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.933+0000 7fabe7fff700 1 -- 172.21.15.67:0/742111781 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7fabd8058fb0 con 0x7fabf0101850 2024-04-22T23:27:16.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.935+0000 7fabf538a700 1 -- 172.21.15.67:0/742111781 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fabdc06e940 msgr2=0x7fabdc070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:16.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.935+0000 7fabf538a700 1 --2- 172.21.15.67:0/742111781 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fabdc06e940 0x7fabdc070e00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fabe800a400 tx=0x7fabe800d040 comp rx=0 tx=0).stop 2024-04-22T23:27:16.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.935+0000 7fabf538a700 1 -- 172.21.15.67:0/742111781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabf0101850 msgr2=0x7fabf00735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:16.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.935+0000 7fabf538a700 1 --2- 172.21.15.67:0/742111781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabf0101850 0x7fabf00735d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fabd80092e0 tx=0x7fabd800bd70 comp rx=0 tx=0).stop 2024-04-22T23:27:16.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.936+0000 7fabf538a700 1 -- 172.21.15.67:0/742111781 shutdown_connections 2024-04-22T23:27:16.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.936+0000 7fabf538a700 1 --2- 172.21.15.67:0/742111781 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fabdc06e940 0x7fabdc070e00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:16.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.936+0000 7fabf538a700 1 --2- 172.21.15.67:0/742111781 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabf0100ea0 0x7fabf0073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:16.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.936+0000 7fabf538a700 1 --2- 172.21.15.67:0/742111781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabf0101850 0x7fabf00735d0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:16.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.936+0000 7fabf538a700 1 -- 172.21.15.67:0/742111781 >> 172.21.15.67:0/742111781 conn(0x7fabf00fc750 msgr2=0x7fabf00fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:27:16.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.936+0000 7fabf538a700 1 -- 172.21.15.67:0/742111781 shutdown_connections 2024-04-22T23:27:16.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:16.936+0000 7fabf538a700 1 -- 172.21.15.67:0/742111781 wait complete. 2024-04-22T23:27:16.946 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:27:16.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:16 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:27:17.581 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -e sha1=43be020184947e53516056c9931e1ac5bdbbb1a5 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 2'"'"'' 2024-04-22T23:27:17.798 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:27:17.920 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:17 smithi067 ceph-mon[26453]: pgmap v71: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:27:17.921 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:17 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/742111781' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:27:17.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:17 smithi134 ceph-mon[34609]: pgmap v71: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:27:17.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:17 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/742111781' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:27:19.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:19.383+0000 7f88edd67700 1 -- 172.21.15.67:0/1135838613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88e8100ea0 msgr2=0x7f88e8101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:19.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:19.383+0000 7f88edd67700 1 --2- 172.21.15.67:0/1135838613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88e8100ea0 0x7f88e8101280 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f88d8009910 tx=0x7f88d8009c20 comp rx=0 tx=0).stop 2024-04-22T23:27:19.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:19.384+0000 7f88edd67700 1 -- 172.21.15.67:0/1135838613 shutdown_connections 2024-04-22T23:27:19.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:19.384+0000 7f88edd67700 1 --2- 172.21.15.67:0/1135838613 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88e8101850 0x7f88e81058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:19.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:19.384+0000 7f88edd67700 1 --2- 172.21.15.67:0/1135838613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88e8100ea0 0x7f88e8101280 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:19.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:19.384+0000 7f88edd67700 1 -- 172.21.15.67:0/1135838613 >> 172.21.15.67:0/1135838613 conn(0x7f88e80fc750 msgr2=0x7f88e80feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:27:19.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:19.384+0000 7f88edd67700 1 -- 172.21.15.67:0/1135838613 shutdown_connections 2024-04-22T23:27:19.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:19.384+0000 7f88edd67700 1 -- 172.21.15.67:0/1135838613 wait complete. 2024-04-22T23:27:19.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:19.384+0000 7f88edd67700 1 Processor -- start 2024-04-22T23:27:19.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:19.384+0000 7f88edd67700 1 -- start start 2024-04-22T23:27:19.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:19.385+0000 7f88edd67700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88e8100ea0 0x7f88e8073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:19.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:19.385+0000 7f88edd67700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88e8101850 0x7f88e80735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:19.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:19.385+0000 7f88edd67700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e8073b10 con 0x7f88e8100ea0 2024-04-22T23:27:19.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:19.385+0000 7f88edd67700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e8073c50 con 0x7f88e8101850 2024-04-22T23:27:19.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:19.385+0000 7f88e77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88e8100ea0 0x7f88e8073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:19.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:19.385+0000 7f88e77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88e8100ea0 0x7f88e8073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42606/0 (socket says 172.21.15.67:42606) 2024-04-22T23:27:19.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:19.385+0000 7f88e77fe700 1 -- 172.21.15.67:0/675502737 learned_addr learned my addr 172.21.15.67:0/675502737 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:27:19.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:19.385+0000 7f88e6ffd700 1 --2- 172.21.15.67:0/675502737 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88e8101850 0x7f88e80735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:19.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:19.385+0000 7f88e77fe700 1 -- 172.21.15.67:0/675502737 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88e8101850 msgr2=0x7f88e80735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:19.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:19.385+0000 7f88e77fe700 1 --2- 172.21.15.67:0/675502737 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88e8101850 0x7f88e80735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:19.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:19.385+0000 7f88e77fe700 1 -- 172.21.15.67:0/675502737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d80095f0 con 0x7f88e8100ea0 2024-04-22T23:27:19.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:19.385+0000 7f88e77fe700 1 --2- 172.21.15.67:0/675502737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88e8100ea0 0x7f88e8073090 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f88d8006010 tx=0x7f88d800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:27:19.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:19.385+0000 7f88e4ff9700 1 -- 172.21.15.67:0/675502737 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88d801d070 con 0x7f88e8100ea0 2024-04-22T23:27:19.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:19.385+0000 7f88e4ff9700 1 -- 172.21.15.67:0/675502737 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f88d8004fb0 con 0x7f88e8100ea0 2024-04-22T23:27:19.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:19.385+0000 7f88edd67700 1 -- 172.21.15.67:0/675502737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88e8073ed0 con 0x7f88e8100ea0 2024-04-22T23:27:19.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:19.386+0000 7f88edd67700 1 -- 172.21.15.67:0/675502737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88e81a2dd0 con 0x7f88e8100ea0 2024-04-22T23:27:19.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:19.386+0000 7f88e4ff9700 1 -- 172.21.15.67:0/675502737 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88d8022ea0 con 0x7f88e8100ea0 2024-04-22T23:27:19.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:19.386+0000 7f88edd67700 1 -- 172.21.15.67:0/675502737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88e8109240 con 0x7f88e8100ea0 2024-04-22T23:27:19.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:19.387+0000 7f88e4ff9700 1 -- 172.21.15.67:0/675502737 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 33) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f88d8022470 con 0x7f88e8100ea0 2024-04-22T23:27:19.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:19.387+0000 7f88e4ff9700 1 --2- 172.21.15.67:0/675502737 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f88d006e940 0x7f88d0070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:19.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:19.388+0000 7f88e4ff9700 1 -- 172.21.15.67:0/675502737 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f88d808feb0 con 0x7f88e8100ea0 2024-04-22T23:27:19.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:19.388+0000 7f88e6ffd700 1 --2- 172.21.15.67:0/675502737 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f88d006e940 0x7f88d0070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:19.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:19.388+0000 7f88e6ffd700 1 --2- 172.21.15.67:0/675502737 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f88d006e940 0x7f88d0070e00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f88dc005ea0 tx=0x7f88dc005e30 comp rx=0 tx=0).ready entity=mgr.14708 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:27:19.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:19.392+0000 7f88e4ff9700 1 -- 172.21.15.67:0/675502737 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f88d80591f0 con 0x7f88e8100ea0 2024-04-22T23:27:19.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:19.619+0000 7f88edd67700 1 -- 172.21.15.67:0/675502737 --> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] -- mgr_command(tid 0: {"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:43be020184947e53516056c9931e1ac5bdbbb1a5", "target": ["mon-mgr", ""]}) v1 -- 0x7f88e80611d0 con 0x7f88d006e940 2024-04-22T23:27:19.620 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:19 smithi067 ceph-mon[26453]: pgmap v72: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:27:19.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:19 smithi134 ceph-mon[34609]: pgmap v72: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:27:20.866 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:20 smithi067 ceph-mon[26453]: from='client.14838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:43be020184947e53516056c9931e1ac5bdbbb1a5", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:27:21.730 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:20 smithi134 ceph-mon[34609]: from='client.14838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:43be020184947e53516056c9931e1ac5bdbbb1a5", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:27:21.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:21.639+0000 7f88e4ff9700 1 -- 172.21.15.67:0/675502737 <== mgr.14708 v2:172.21.15.67:6800/2306276228 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+5391 (secure 0 0 0) 0x7f88e80611d0 con 0x7f88d006e940 2024-04-22T23:27:21.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:21.641+0000 7f88edd67700 1 -- 172.21.15.67:0/675502737 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f88d006e940 msgr2=0x7f88d0070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:21.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:21.641+0000 7f88edd67700 1 --2- 172.21.15.67:0/675502737 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f88d006e940 0x7f88d0070e00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f88dc005ea0 tx=0x7f88dc005e30 comp rx=0 tx=0).stop 2024-04-22T23:27:21.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:21.641+0000 7f88edd67700 1 -- 172.21.15.67:0/675502737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88e8100ea0 msgr2=0x7f88e8073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:21.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:21.641+0000 7f88edd67700 1 --2- 172.21.15.67:0/675502737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88e8100ea0 0x7f88e8073090 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f88d8006010 tx=0x7f88d800be20 comp rx=0 tx=0).stop 2024-04-22T23:27:21.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:21.641+0000 7f88edd67700 1 -- 172.21.15.67:0/675502737 shutdown_connections 2024-04-22T23:27:21.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:21.641+0000 7f88edd67700 1 --2- 172.21.15.67:0/675502737 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f88d006e940 0x7f88d0070e00 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:21.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:21.641+0000 7f88edd67700 1 --2- 172.21.15.67:0/675502737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88e8100ea0 0x7f88e8073090 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:21.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:21.641+0000 7f88edd67700 1 --2- 172.21.15.67:0/675502737 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88e8101850 0x7f88e80735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:21.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:21.641+0000 7f88edd67700 1 -- 172.21.15.67:0/675502737 >> 172.21.15.67:0/675502737 conn(0x7f88e80fc750 msgr2=0x7f88e80fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:27:21.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:21.641+0000 7f88edd67700 1 -- 172.21.15.67:0/675502737 shutdown_connections 2024-04-22T23:27:21.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:21.641+0000 7f88edd67700 1 -- 172.21.15.67:0/675502737 wait complete. 2024-04-22T23:27:21.732 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:27:21.963 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:21 smithi067 ceph-mon[26453]: pgmap v73: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:27:21.963 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:21 smithi067 ceph-mon[26453]: fsmap cephfs:2 {0=cephfs.smithi067.xyazmh=up:active,1=cephfs.smithi134.owipad=up:active} 2 up:standby-replay 2024-04-22T23:27:21.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:21 smithi134 ceph-mon[34609]: pgmap v73: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:27:21.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:21 smithi134 ceph-mon[34609]: fsmap cephfs:2 {0=cephfs.smithi067.xyazmh=up:active,1=cephfs.smithi134.owipad=up:active} 2 up:standby-replay 2024-04-22T23:27:22.261 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -e sha1=43be020184947e53516056c9931e1ac5bdbbb1a5 -- bash -c 'ceph orch ps' 2024-04-22T23:27:22.476 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:27:23.757 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:23 smithi067 ceph-mon[26453]: pgmap v74: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:27:23.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:23 smithi134 ceph-mon[34609]: pgmap v74: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:27:24.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.177+0000 7f1a0c764700 1 -- 172.21.15.67:0/1674318719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a04101e00 msgr2=0x7f1a041021e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:24.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.177+0000 7f1a0c764700 1 --2- 172.21.15.67:0/1674318719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a04101e00 0x7f1a041021e0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f19f8009910 tx=0x7f19f8009c20 comp rx=0 tx=0).stop 2024-04-22T23:27:24.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.177+0000 7f1a0c764700 1 -- 172.21.15.67:0/1674318719 shutdown_connections 2024-04-22T23:27:24.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.177+0000 7f1a0c764700 1 --2- 172.21.15.67:0/1674318719 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a04102720 0x7f1a0410ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:24.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.177+0000 7f1a0c764700 1 --2- 172.21.15.67:0/1674318719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a04101e00 0x7f1a041021e0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:24.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.177+0000 7f1a0c764700 1 -- 172.21.15.67:0/1674318719 >> 172.21.15.67:0/1674318719 conn(0x7f1a040fb650 msgr2=0x7f1a040fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:27:24.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.178+0000 7f1a0c764700 1 -- 172.21.15.67:0/1674318719 shutdown_connections 2024-04-22T23:27:24.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.178+0000 7f1a0c764700 1 -- 172.21.15.67:0/1674318719 wait complete. 2024-04-22T23:27:24.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.178+0000 7f1a0c764700 1 Processor -- start 2024-04-22T23:27:24.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.178+0000 7f1a0c764700 1 -- start start 2024-04-22T23:27:24.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.178+0000 7f1a0c764700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a04101e00 0x7f1a04196950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:24.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.178+0000 7f1a0c764700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a04102720 0x7f1a04196e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:24.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.178+0000 7f1a0c764700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a04197520 con 0x7f1a04102720 2024-04-22T23:27:24.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.178+0000 7f1a0c764700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0419b330 con 0x7f1a04101e00 2024-04-22T23:27:24.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.179+0000 7f1a09cff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a04102720 0x7f1a04196e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:24.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.179+0000 7f1a09cff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a04102720 0x7f1a04196e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42622/0 (socket says 172.21.15.67:42622) 2024-04-22T23:27:24.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.179+0000 7f1a09cff700 1 -- 172.21.15.67:0/3373948246 learned_addr learned my addr 172.21.15.67:0/3373948246 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:27:24.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.179+0000 7f1a0a500700 1 --2- 172.21.15.67:0/3373948246 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a04101e00 0x7f1a04196950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:24.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.179+0000 7f1a09cff700 1 -- 172.21.15.67:0/3373948246 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a04101e00 msgr2=0x7f1a04196950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:24.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.179+0000 7f1a09cff700 1 --2- 172.21.15.67:0/3373948246 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a04101e00 0x7f1a04196950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:24.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.179+0000 7f1a09cff700 1 -- 172.21.15.67:0/3373948246 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f80095f0 con 0x7f1a04102720 2024-04-22T23:27:24.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.179+0000 7f1a09cff700 1 --2- 172.21.15.67:0/3373948246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a04102720 0x7f1a04196e90 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f1a0000d6d0 tx=0x7f1a0000d9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:27:24.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.179+0000 7f19f77fe700 1 -- 172.21.15.67:0/3373948246 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a0000f860 con 0x7f1a04102720 2024-04-22T23:27:24.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.179+0000 7f19f77fe700 1 -- 172.21.15.67:0/3373948246 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1a0000fe80 con 0x7f1a04102720 2024-04-22T23:27:24.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.179+0000 7f1a0c764700 1 -- 172.21.15.67:0/3373948246 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a0419b610 con 0x7f1a04102720 2024-04-22T23:27:24.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.180+0000 7f19f77fe700 1 -- 172.21.15.67:0/3373948246 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a00010ea0 con 0x7f1a04102720 2024-04-22T23:27:24.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.180+0000 7f1a0c764700 1 -- 172.21.15.67:0/3373948246 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a0419bb60 con 0x7f1a04102720 2024-04-22T23:27:24.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.181+0000 7f1a0c764700 1 -- 172.21.15.67:0/3373948246 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a04108310 con 0x7f1a04102720 2024-04-22T23:27:24.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.181+0000 7f19f77fe700 1 -- 172.21.15.67:0/3373948246 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 33) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f1a0000f9c0 con 0x7f1a04102720 2024-04-22T23:27:24.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.182+0000 7f19f77fe700 1 --2- 172.21.15.67:0/3373948246 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f19f006e940 0x7f19f0070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:24.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.182+0000 7f19f77fe700 1 -- 172.21.15.67:0/3373948246 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f1a0008e510 con 0x7f1a04102720 2024-04-22T23:27:24.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.182+0000 7f1a0a500700 1 --2- 172.21.15.67:0/3373948246 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f19f006e940 0x7f19f0070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:24.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.183+0000 7f1a0a500700 1 --2- 172.21.15.67:0/3373948246 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f19f006e940 0x7f19f0070e00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f19f8006010 tx=0x7f19f801a040 comp rx=0 tx=0).ready entity=mgr.14708 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:27:24.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.187+0000 7f19f77fe700 1 -- 172.21.15.67:0/3373948246 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f1a0005b350 con 0x7f1a04102720 2024-04-22T23:27:24.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.405+0000 7f1a0c764700 1 -- 172.21.15.67:0/3373948246 --> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1a04197d00 con 0x7f19f006e940 2024-04-22T23:27:24.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.416+0000 7f19f77fe700 1 -- 172.21.15.67:0/3373948246 <== mgr.14708 v2:172.21.15.67:6800/2306276228 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f1a04197d00 con 0x7f19f006e940 2024-04-22T23:27:24.417 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:27:24.417 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (68s) 38s ago 12m 16.3M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:27:24.418 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (12m) 38s ago 12m 8048k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:27:24.418 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (11m) 99s ago 11m 7931k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:27:24.418 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (12m) 38s ago 12m 6660k - 18.2.1 5be31c24972a 14085634c5d3 2024-04-22T23:27:24.418 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (11m) 99s ago 11m 6665k - 18.2.1 5be31c24972a fa995d458353 2024-04-22T23:27:24.418 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (44s) 38s ago 12m 51.2M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:27:24.418 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (6m) 38s ago 6m 62.7M - 18.2.1 5be31c24972a e24a471f0a8c 2024-04-22T23:27:24.418 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (6m) 38s ago 6m 130M - 18.2.1 5be31c24972a 0730f9fc8e61 2024-04-22T23:27:24.418 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (6m) 99s ago 6m 123M - 18.2.1 5be31c24972a b6a169586433 2024-04-22T23:27:24.418 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (6m) 99s ago 6m 67.5M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:27:24.418 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (2m) 38s ago 13m 508M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:27:24.418 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (2m) 99s ago 11m 449M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:27:24.418 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (14m) 38s ago 14m 59.0M 2048M 18.2.1 5be31c24972a 6789283703e0 2024-04-22T23:27:24.418 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (11m) 99s ago 11m 46.6M 2048M 18.2.1 5be31c24972a 2fb23795f7e3 2024-04-22T23:27:24.418 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (108s) 38s ago 12m 15.6M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:27:24.419 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (103s) 99s ago 11m 3376k - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:27:24.419 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (10m) 38s ago 10m 205M 1216M 18.2.1 5be31c24972a a033247ee8c2 2024-04-22T23:27:24.419 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (10m) 38s ago 10m 211M 1216M 18.2.1 5be31c24972a 915b07a8edc4 2024-04-22T23:27:24.419 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (9m) 38s ago 9m 172M 1216M 18.2.1 5be31c24972a e257ecb4efa1 2024-04-22T23:27:24.419 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (9m) 99s ago 9m 248M 2240M 18.2.1 5be31c24972a 9c2c91a967ba 2024-04-22T23:27:24.419 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (8m) 99s ago 8m 217M 2240M 18.2.1 5be31c24972a c7586562ffa6 2024-04-22T23:27:24.419 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (8m) 99s ago 8m 192M 2240M 18.2.1 5be31c24972a 9fd97f1c65cb 2024-04-22T23:27:24.419 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (86s) 38s ago 11m 53.8M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:27:24.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.418+0000 7f1a0c764700 1 -- 172.21.15.67:0/3373948246 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f19f006e940 msgr2=0x7f19f0070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:24.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.418+0000 7f1a0c764700 1 --2- 172.21.15.67:0/3373948246 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f19f006e940 0x7f19f0070e00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f19f8006010 tx=0x7f19f801a040 comp rx=0 tx=0).stop 2024-04-22T23:27:24.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.418+0000 7f1a0c764700 1 -- 172.21.15.67:0/3373948246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a04102720 msgr2=0x7f1a04196e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:24.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.418+0000 7f1a0c764700 1 --2- 172.21.15.67:0/3373948246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a04102720 0x7f1a04196e90 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f1a0000d6d0 tx=0x7f1a0000d9e0 comp rx=0 tx=0).stop 2024-04-22T23:27:24.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.418+0000 7f1a0c764700 1 -- 172.21.15.67:0/3373948246 shutdown_connections 2024-04-22T23:27:24.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.418+0000 7f1a0c764700 1 --2- 172.21.15.67:0/3373948246 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f19f006e940 0x7f19f0070e00 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:24.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.418+0000 7f1a0c764700 1 --2- 172.21.15.67:0/3373948246 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a04101e00 0x7f1a04196950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:24.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.418+0000 7f1a0c764700 1 --2- 172.21.15.67:0/3373948246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a04102720 0x7f1a04196e90 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:24.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.418+0000 7f1a0c764700 1 -- 172.21.15.67:0/3373948246 >> 172.21.15.67:0/3373948246 conn(0x7f1a040fb650 msgr2=0x7f1a04105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:27:24.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.418+0000 7f1a0c764700 1 -- 172.21.15.67:0/3373948246 shutdown_connections 2024-04-22T23:27:24.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:24.418+0000 7f1a0c764700 1 -- 172.21.15.67:0/3373948246 wait complete. 2024-04-22T23:27:25.068 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-22T23:27:25.069 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi067.front.sepia.ceph.com 2024-04-22T23:27:25.069 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -e sha1=43be020184947e53516056c9931e1ac5bdbbb1a5 -- bash -c 'ceph config set mgr mgr/orchestrator/fail_fs true' 2024-04-22T23:27:25.283 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:27:25.683 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:25 smithi067 ceph-mon[26453]: from='client.14842 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:27:25.683 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:25 smithi067 ceph-mon[26453]: pgmap v75: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:27:25.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:25 smithi134 ceph-mon[34609]: from='client.14842 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:27:25.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:25 smithi134 ceph-mon[34609]: pgmap v75: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:27:27.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.175+0000 7fd8a04f0700 1 -- 172.21.15.67:0/1380452838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd898100ea0 msgr2=0x7fd898101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:27.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.175+0000 7fd8a04f0700 1 --2- 172.21.15.67:0/1380452838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd898100ea0 0x7fd898101280 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fd894009910 tx=0x7fd894009c20 comp rx=0 tx=0).stop 2024-04-22T23:27:27.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.175+0000 7fd8a04f0700 1 -- 172.21.15.67:0/1380452838 shutdown_connections 2024-04-22T23:27:27.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.175+0000 7fd8a04f0700 1 --2- 172.21.15.67:0/1380452838 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd898101850 0x7fd8981058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:27.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.175+0000 7fd8a04f0700 1 --2- 172.21.15.67:0/1380452838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd898100ea0 0x7fd898101280 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:27.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.175+0000 7fd8a04f0700 1 -- 172.21.15.67:0/1380452838 >> 172.21.15.67:0/1380452838 conn(0x7fd8980fc750 msgr2=0x7fd8980feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:27:27.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.175+0000 7fd8a04f0700 1 -- 172.21.15.67:0/1380452838 shutdown_connections 2024-04-22T23:27:27.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.175+0000 7fd8a04f0700 1 -- 172.21.15.67:0/1380452838 wait complete. 2024-04-22T23:27:27.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.175+0000 7fd8a04f0700 1 Processor -- start 2024-04-22T23:27:27.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.175+0000 7fd8a04f0700 1 -- start start 2024-04-22T23:27:27.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.175+0000 7fd8a04f0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd898100ea0 0x7fd898196990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:27.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.175+0000 7fd8a04f0700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd898101850 0x7fd898196ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:27.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.175+0000 7fd8a04f0700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd898197560 con 0x7fd898100ea0 2024-04-22T23:27:27.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.175+0000 7fd8a04f0700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd89819b370 con 0x7fd898101850 2024-04-22T23:27:27.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.176+0000 7fd89e28c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd898100ea0 0x7fd898196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:27.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.176+0000 7fd89e28c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd898100ea0 0x7fd898196990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42644/0 (socket says 172.21.15.67:42644) 2024-04-22T23:27:27.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.176+0000 7fd89e28c700 1 -- 172.21.15.67:0/1356329443 learned_addr learned my addr 172.21.15.67:0/1356329443 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:27:27.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.176+0000 7fd89da8b700 1 --2- 172.21.15.67:0/1356329443 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd898101850 0x7fd898196ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:27.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.176+0000 7fd89e28c700 1 -- 172.21.15.67:0/1356329443 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd898101850 msgr2=0x7fd898196ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:27.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.176+0000 7fd89e28c700 1 --2- 172.21.15.67:0/1356329443 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd898101850 0x7fd898196ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:27.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.176+0000 7fd89e28c700 1 -- 172.21.15.67:0/1356329443 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8940095f0 con 0x7fd898100ea0 2024-04-22T23:27:27.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.176+0000 7fd89e28c700 1 --2- 172.21.15.67:0/1356329443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd898100ea0 0x7fd898196990 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fd894006010 tx=0x7fd89400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:27:27.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.176+0000 7fd88f7fe700 1 -- 172.21.15.67:0/1356329443 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd89401d070 con 0x7fd898100ea0 2024-04-22T23:27:27.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.176+0000 7fd88f7fe700 1 -- 172.21.15.67:0/1356329443 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd894004fb0 con 0x7fd898100ea0 2024-04-22T23:27:27.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.176+0000 7fd8a04f0700 1 -- 172.21.15.67:0/1356329443 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd89819b5f0 con 0x7fd898100ea0 2024-04-22T23:27:27.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.176+0000 7fd8a04f0700 1 -- 172.21.15.67:0/1356329443 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd89819bae0 con 0x7fd898100ea0 2024-04-22T23:27:27.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.177+0000 7fd88f7fe700 1 -- 172.21.15.67:0/1356329443 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd894022ea0 con 0x7fd898100ea0 2024-04-22T23:27:27.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.177+0000 7fd8a04f0700 1 -- 172.21.15.67:0/1356329443 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd87c005320 con 0x7fd898100ea0 2024-04-22T23:27:27.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.178+0000 7fd88f7fe700 1 -- 172.21.15.67:0/1356329443 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 33) v1 ==== 92598+0+0 (secure 0 0 0) 0x7fd894004af0 con 0x7fd898100ea0 2024-04-22T23:27:27.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.178+0000 7fd88f7fe700 1 --2- 172.21.15.67:0/1356329443 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fd88406e940 0x7fd884070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:27.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.179+0000 7fd88f7fe700 1 -- 172.21.15.67:0/1356329443 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fd89408fcc0 con 0x7fd898100ea0 2024-04-22T23:27:27.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.179+0000 7fd89da8b700 1 --2- 172.21.15.67:0/1356329443 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fd88406e940 0x7fd884070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:27.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.179+0000 7fd89da8b700 1 --2- 172.21.15.67:0/1356329443 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fd88406e940 0x7fd884070e00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd888006fd0 tx=0x7fd888008040 comp rx=0 tx=0).ready entity=mgr.14708 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:27:27.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.183+0000 7fd88f7fe700 1 -- 172.21.15.67:0/1356329443 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7fd89405cbb0 con 0x7fd898100ea0 2024-04-22T23:27:27.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.392+0000 7fd8a04f0700 1 -- 172.21.15.67:0/1356329443 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/orchestrator/fail_fs}] v 0) v1 -- 0x7fd87c005f70 con 0x7fd898100ea0 2024-04-22T23:27:27.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.396+0000 7fd88f7fe700 1 -- 172.21.15.67:0/1356329443 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/orchestrator/fail_fs}]=0 v44)=0 v44) v1 ==== 125+0+0 (secure 0 0 0) 0x7fd8940270c0 con 0x7fd898100ea0 2024-04-22T23:27:27.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.399+0000 7fd8a04f0700 1 -- 172.21.15.67:0/1356329443 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fd88406e940 msgr2=0x7fd884070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:27.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.399+0000 7fd8a04f0700 1 --2- 172.21.15.67:0/1356329443 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fd88406e940 0x7fd884070e00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd888006fd0 tx=0x7fd888008040 comp rx=0 tx=0).stop 2024-04-22T23:27:27.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.399+0000 7fd8a04f0700 1 -- 172.21.15.67:0/1356329443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd898100ea0 msgr2=0x7fd898196990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:27.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.399+0000 7fd8a04f0700 1 --2- 172.21.15.67:0/1356329443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd898100ea0 0x7fd898196990 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fd894006010 tx=0x7fd89400be20 comp rx=0 tx=0).stop 2024-04-22T23:27:27.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.399+0000 7fd8a04f0700 1 -- 172.21.15.67:0/1356329443 shutdown_connections 2024-04-22T23:27:27.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.399+0000 7fd8a04f0700 1 --2- 172.21.15.67:0/1356329443 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fd88406e940 0x7fd884070e00 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:27.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.399+0000 7fd8a04f0700 1 --2- 172.21.15.67:0/1356329443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd898100ea0 0x7fd898196990 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:27.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.399+0000 7fd8a04f0700 1 --2- 172.21.15.67:0/1356329443 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd898101850 0x7fd898196ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:27.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.399+0000 7fd8a04f0700 1 -- 172.21.15.67:0/1356329443 >> 172.21.15.67:0/1356329443 conn(0x7fd8980fc750 msgr2=0x7fd8980fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:27:27.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.399+0000 7fd8a04f0700 1 -- 172.21.15.67:0/1356329443 shutdown_connections 2024-04-22T23:27:27.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:27.399+0000 7fd8a04f0700 1 -- 172.21.15.67:0/1356329443 wait complete. 2024-04-22T23:27:27.694 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:27 smithi067 ceph-mon[26453]: pgmap v76: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:27:27.694 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:27 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/1356329443' entity='client.admin' 2024-04-22T23:27:27.694 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:27 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:27:27.694 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:27 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:27:27.694 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:27 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:27:27.694 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:27 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:27:27.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:27 smithi134 ceph-mon[34609]: pgmap v76: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:27:27.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:27 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/1356329443' entity='client.admin' 2024-04-22T23:27:27.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:27 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:27:27.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:27 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:27:27.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:27 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:27:27.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:27 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:27:28.012 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-22T23:27:28.013 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi067.front.sepia.ceph.com 2024-04-22T23:27:28.013 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -e sha1=43be020184947e53516056c9931e1ac5bdbbb1a5 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-22T23:27:28.231 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:27:29.872 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:29 smithi067 ceph-mon[26453]: pgmap v77: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:27:29.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:29.906+0000 7f59a60b3700 1 -- 172.21.15.67:0/220820136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59a0100ea0 msgr2=0x7f59a0101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:29.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:29.906+0000 7f59a60b3700 1 --2- 172.21.15.67:0/220820136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59a0100ea0 0x7f59a0101280 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f5990009910 tx=0x7f5990009c20 comp rx=0 tx=0).stop 2024-04-22T23:27:29.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:29.906+0000 7f59a60b3700 1 -- 172.21.15.67:0/220820136 shutdown_connections 2024-04-22T23:27:29.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:29.906+0000 7f59a60b3700 1 --2- 172.21.15.67:0/220820136 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59a0101850 0x7f59a01058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:29.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:29.906+0000 7f59a60b3700 1 --2- 172.21.15.67:0/220820136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59a0100ea0 0x7f59a0101280 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:29.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:29.906+0000 7f59a60b3700 1 -- 172.21.15.67:0/220820136 >> 172.21.15.67:0/220820136 conn(0x7f59a00fc750 msgr2=0x7f59a00feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:27:29.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:29.907+0000 7f59a60b3700 1 -- 172.21.15.67:0/220820136 shutdown_connections 2024-04-22T23:27:29.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:29.907+0000 7f59a60b3700 1 -- 172.21.15.67:0/220820136 wait complete. 2024-04-22T23:27:29.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:29.907+0000 7f59a60b3700 1 Processor -- start 2024-04-22T23:27:29.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:29.907+0000 7f59a60b3700 1 -- start start 2024-04-22T23:27:29.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:29.907+0000 7f59a60b3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59a0100ea0 0x7f59a019c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:29.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:29.908+0000 7f59a60b3700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59a0101850 0x7f59a019cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:29.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:29.908+0000 7f59a60b3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a019d230 con 0x7f59a0100ea0 2024-04-22T23:27:29.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:29.908+0000 7f59a60b3700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a0196770 con 0x7f59a0101850 2024-04-22T23:27:29.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:29.908+0000 7f599f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59a0100ea0 0x7f59a019c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:29.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:29.908+0000 7f599f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59a0100ea0 0x7f59a019c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42542/0 (socket says 172.21.15.67:42542) 2024-04-22T23:27:29.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:29.908+0000 7f599f7fe700 1 -- 172.21.15.67:0/2548122212 learned_addr learned my addr 172.21.15.67:0/2548122212 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:27:29.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:29.908+0000 7f599effd700 1 --2- 172.21.15.67:0/2548122212 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59a0101850 0x7f59a019cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:29.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:29.908+0000 7f599f7fe700 1 -- 172.21.15.67:0/2548122212 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59a0101850 msgr2=0x7f59a019cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:29.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:29.908+0000 7f599f7fe700 1 --2- 172.21.15.67:0/2548122212 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59a0101850 0x7f59a019cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:29.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:29.908+0000 7f599f7fe700 1 -- 172.21.15.67:0/2548122212 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59900095f0 con 0x7f59a0100ea0 2024-04-22T23:27:29.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:29.908+0000 7f599f7fe700 1 --2- 172.21.15.67:0/2548122212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59a0100ea0 0x7f59a019c6f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f599000b5c0 tx=0x7f599000be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:27:29.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:29.908+0000 7f599cff9700 1 -- 172.21.15.67:0/2548122212 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f599001d070 con 0x7f59a0100ea0 2024-04-22T23:27:29.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:29.908+0000 7f599cff9700 1 -- 172.21.15.67:0/2548122212 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f599000f460 con 0x7f59a0100ea0 2024-04-22T23:27:29.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:29.908+0000 7f59a60b3700 1 -- 172.21.15.67:0/2548122212 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59a01969f0 con 0x7f59a0100ea0 2024-04-22T23:27:29.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:29.908+0000 7f599cff9700 1 -- 172.21.15.67:0/2548122212 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5990017610 con 0x7f59a0100ea0 2024-04-22T23:27:29.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:29.908+0000 7f59a60b3700 1 -- 172.21.15.67:0/2548122212 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59a0196ee0 con 0x7f59a0100ea0 2024-04-22T23:27:29.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:29.910+0000 7f59a60b3700 1 -- 172.21.15.67:0/2548122212 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59a0109240 con 0x7f59a0100ea0 2024-04-22T23:27:29.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:29.910+0000 7f599cff9700 1 -- 172.21.15.67:0/2548122212 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 33) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f599000f5d0 con 0x7f59a0100ea0 2024-04-22T23:27:29.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:29.910+0000 7f599cff9700 1 --2- 172.21.15.67:0/2548122212 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f598c06e940 0x7f598c070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:29.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:29.911+0000 7f599cff9700 1 -- 172.21.15.67:0/2548122212 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f599008fe30 con 0x7f59a0100ea0 2024-04-22T23:27:29.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:29.911+0000 7f599effd700 1 --2- 172.21.15.67:0/2548122212 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f598c06e940 0x7f598c070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:29.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:29.911+0000 7f599effd700 1 --2- 172.21.15.67:0/2548122212 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f598c06e940 0x7f598c070e00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5988005ea0 tx=0x7f5988005e30 comp rx=0 tx=0).ready entity=mgr.14708 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:27:29.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:29.915+0000 7f599cff9700 1 -- 172.21.15.67:0/2548122212 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f599005cc70 con 0x7f59a0100ea0 2024-04-22T23:27:29.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:29 smithi134 ceph-mon[34609]: pgmap v77: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:27:30.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:30.128+0000 7f59a60b3700 1 -- 172.21.15.67:0/2548122212 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7f59a004f2e0 con 0x7f59a0100ea0 2024-04-22T23:27:30.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:30.128+0000 7f599cff9700 1 -- 172.21.15.67:0/2548122212 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v44)=0 v44) v1 ==== 155+0+0 (secure 0 0 0) 0x7f599005c800 con 0x7f59a0100ea0 2024-04-22T23:27:30.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:30.130+0000 7f59a60b3700 1 -- 172.21.15.67:0/2548122212 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f598c06e940 msgr2=0x7f598c070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:30.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:30.130+0000 7f59a60b3700 1 --2- 172.21.15.67:0/2548122212 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f598c06e940 0x7f598c070e00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5988005ea0 tx=0x7f5988005e30 comp rx=0 tx=0).stop 2024-04-22T23:27:30.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:30.130+0000 7f59a60b3700 1 -- 172.21.15.67:0/2548122212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59a0100ea0 msgr2=0x7f59a019c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:30.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:30.130+0000 7f59a60b3700 1 --2- 172.21.15.67:0/2548122212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59a0100ea0 0x7f59a019c6f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f599000b5c0 tx=0x7f599000be00 comp rx=0 tx=0).stop 2024-04-22T23:27:30.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:30.130+0000 7f59a60b3700 1 -- 172.21.15.67:0/2548122212 shutdown_connections 2024-04-22T23:27:30.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:30.130+0000 7f59a60b3700 1 --2- 172.21.15.67:0/2548122212 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f598c06e940 0x7f598c070e00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:30.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:30.130+0000 7f59a60b3700 1 --2- 172.21.15.67:0/2548122212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59a0100ea0 0x7f59a019c6f0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:30.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:30.130+0000 7f59a60b3700 1 --2- 172.21.15.67:0/2548122212 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59a0101850 0x7f59a019cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:30.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:30.130+0000 7f59a60b3700 1 -- 172.21.15.67:0/2548122212 >> 172.21.15.67:0/2548122212 conn(0x7f59a00fc750 msgr2=0x7f59a00fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:27:30.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:30.131+0000 7f59a60b3700 1 -- 172.21.15.67:0/2548122212 shutdown_connections 2024-04-22T23:27:30.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:30.131+0000 7f59a60b3700 1 -- 172.21.15.67:0/2548122212 wait complete. 2024-04-22T23:27:30.757 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -e sha1=43be020184947e53516056c9931e1ac5bdbbb1a5 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-22T23:27:30.971 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:27:31.639 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:31 smithi067 ceph-mon[26453]: pgmap v78: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:27:31.640 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:31 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:27:31.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:31 smithi134 ceph-mon[34609]: pgmap v78: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:27:31.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:31 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:27:32.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.706+0000 7fca42618700 1 -- 172.21.15.67:0/1073988362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca3c100ea0 msgr2=0x7fca3c101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:32.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.706+0000 7fca42618700 1 --2- 172.21.15.67:0/1073988362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca3c100ea0 0x7fca3c101280 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fca2c009910 tx=0x7fca2c009c20 comp rx=0 tx=0).stop 2024-04-22T23:27:32.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.706+0000 7fca42618700 1 -- 172.21.15.67:0/1073988362 shutdown_connections 2024-04-22T23:27:32.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.706+0000 7fca42618700 1 --2- 172.21.15.67:0/1073988362 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fca3c101850 0x7fca3c1058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:32.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.706+0000 7fca42618700 1 --2- 172.21.15.67:0/1073988362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca3c100ea0 0x7fca3c101280 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:32.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.706+0000 7fca42618700 1 -- 172.21.15.67:0/1073988362 >> 172.21.15.67:0/1073988362 conn(0x7fca3c0fc750 msgr2=0x7fca3c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:27:32.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.706+0000 7fca42618700 1 -- 172.21.15.67:0/1073988362 shutdown_connections 2024-04-22T23:27:32.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.706+0000 7fca42618700 1 -- 172.21.15.67:0/1073988362 wait complete. 2024-04-22T23:27:32.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.706+0000 7fca42618700 1 Processor -- start 2024-04-22T23:27:32.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.706+0000 7fca42618700 1 -- start start 2024-04-22T23:27:32.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.707+0000 7fca42618700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca3c100ea0 0x7fca3c19c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:32.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.707+0000 7fca42618700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fca3c101850 0x7fca3c19cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:32.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.707+0000 7fca42618700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c19d2e0 con 0x7fca3c100ea0 2024-04-22T23:27:32.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.707+0000 7fca42618700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c196790 con 0x7fca3c101850 2024-04-22T23:27:32.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.707+0000 7fca3bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca3c100ea0 0x7fca3c19c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:32.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.707+0000 7fca3bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca3c100ea0 0x7fca3c19c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42546/0 (socket says 172.21.15.67:42546) 2024-04-22T23:27:32.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.707+0000 7fca3bfff700 1 -- 172.21.15.67:0/3226117614 learned_addr learned my addr 172.21.15.67:0/3226117614 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:27:32.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.707+0000 7fca3bfff700 1 -- 172.21.15.67:0/3226117614 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fca3c101850 msgr2=0x7fca3c19cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:32.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.707+0000 7fca3b7fe700 1 --2- 172.21.15.67:0/3226117614 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fca3c101850 0x7fca3c19cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:32.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.707+0000 7fca3bfff700 1 --2- 172.21.15.67:0/3226117614 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fca3c101850 0x7fca3c19cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:32.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.707+0000 7fca3bfff700 1 -- 172.21.15.67:0/3226117614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca2c0095f0 con 0x7fca3c100ea0 2024-04-22T23:27:32.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.707+0000 7fca3b7fe700 1 --2- 172.21.15.67:0/3226117614 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fca3c101850 0x7fca3c19cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:27:32.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.707+0000 7fca3bfff700 1 --2- 172.21.15.67:0/3226117614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca3c100ea0 0x7fca3c19c710 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fca2c006010 tx=0x7fca2c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:27:32.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.708+0000 7fca397fa700 1 -- 172.21.15.67:0/3226117614 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca2c01d070 con 0x7fca3c100ea0 2024-04-22T23:27:32.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.708+0000 7fca42618700 1 -- 172.21.15.67:0/3226117614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca3c196a10 con 0x7fca3c100ea0 2024-04-22T23:27:32.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.708+0000 7fca397fa700 1 -- 172.21.15.67:0/3226117614 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fca2c004fb0 con 0x7fca3c100ea0 2024-04-22T23:27:32.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.708+0000 7fca397fa700 1 -- 172.21.15.67:0/3226117614 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca2c022ea0 con 0x7fca3c100ea0 2024-04-22T23:27:32.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.708+0000 7fca42618700 1 -- 172.21.15.67:0/3226117614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca3c196f00 con 0x7fca3c100ea0 2024-04-22T23:27:32.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.709+0000 7fca42618700 1 -- 172.21.15.67:0/3226117614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca3c109240 con 0x7fca3c100ea0 2024-04-22T23:27:32.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.709+0000 7fca397fa700 1 -- 172.21.15.67:0/3226117614 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 33) v1 ==== 92598+0+0 (secure 0 0 0) 0x7fca2c022470 con 0x7fca3c100ea0 2024-04-22T23:27:32.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.710+0000 7fca397fa700 1 --2- 172.21.15.67:0/3226117614 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fca2406e940 0x7fca24070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:32.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.710+0000 7fca3b7fe700 1 --2- 172.21.15.67:0/3226117614 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fca2406e940 0x7fca24070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:32.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.710+0000 7fca397fa700 1 -- 172.21.15.67:0/3226117614 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fca2c08fe80 con 0x7fca3c100ea0 2024-04-22T23:27:32.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.710+0000 7fca3b7fe700 1 --2- 172.21.15.67:0/3226117614 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fca2406e940 0x7fca24070e00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fca30005fd0 tx=0x7fca30005dc0 comp rx=0 tx=0).ready entity=mgr.14708 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:27:32.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.715+0000 7fca397fa700 1 -- 172.21.15.67:0/3226117614 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7fca2c05ccf0 con 0x7fca3c100ea0 2024-04-22T23:27:32.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.924+0000 7fca42618700 1 -- 172.21.15.67:0/3226117614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7fca3c197cf0 con 0x7fca3c100ea0 2024-04-22T23:27:32.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.924+0000 7fca397fa700 1 -- 172.21.15.67:0/3226117614 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v44)=0 v44) v1 ==== 163+0+0 (secure 0 0 0) 0x7fca2c022720 con 0x7fca3c100ea0 2024-04-22T23:27:32.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.926+0000 7fca42618700 1 -- 172.21.15.67:0/3226117614 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fca2406e940 msgr2=0x7fca24070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:32.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.926+0000 7fca42618700 1 --2- 172.21.15.67:0/3226117614 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fca2406e940 0x7fca24070e00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fca30005fd0 tx=0x7fca30005dc0 comp rx=0 tx=0).stop 2024-04-22T23:27:32.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.926+0000 7fca42618700 1 -- 172.21.15.67:0/3226117614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca3c100ea0 msgr2=0x7fca3c19c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:32.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.926+0000 7fca42618700 1 --2- 172.21.15.67:0/3226117614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca3c100ea0 0x7fca3c19c710 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fca2c006010 tx=0x7fca2c00be20 comp rx=0 tx=0).stop 2024-04-22T23:27:32.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.926+0000 7fca42618700 1 -- 172.21.15.67:0/3226117614 shutdown_connections 2024-04-22T23:27:32.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.926+0000 7fca42618700 1 --2- 172.21.15.67:0/3226117614 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fca2406e940 0x7fca24070e00 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:32.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.926+0000 7fca42618700 1 --2- 172.21.15.67:0/3226117614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca3c100ea0 0x7fca3c19c710 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:32.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.926+0000 7fca42618700 1 --2- 172.21.15.67:0/3226117614 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fca3c101850 0x7fca3c19cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:32.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.926+0000 7fca42618700 1 -- 172.21.15.67:0/3226117614 >> 172.21.15.67:0/3226117614 conn(0x7fca3c0fc750 msgr2=0x7fca3c0fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:27:32.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.927+0000 7fca42618700 1 -- 172.21.15.67:0/3226117614 shutdown_connections 2024-04-22T23:27:32.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:32.927+0000 7fca42618700 1 -- 172.21.15.67:0/3226117614 wait complete. 2024-04-22T23:27:33.626 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -e sha1=43be020184947e53516056c9931e1ac5bdbbb1a5 -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-22T23:27:33.842 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:27:33.859 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:33 smithi067 ceph-mon[26453]: pgmap v79: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:27:33.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:33 smithi134 ceph-mon[34609]: pgmap v79: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:27:35.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.489+0000 7f215ee1a700 1 -- 172.21.15.67:0/779857953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21581006b0 msgr2=0x7f2158100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:35.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.489+0000 7f215ee1a700 1 --2- 172.21.15.67:0/779857953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21581006b0 0x7f2158100a90 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f214c009910 tx=0x7f214c009c20 comp rx=0 tx=0).stop 2024-04-22T23:27:35.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.489+0000 7f215ee1a700 1 -- 172.21.15.67:0/779857953 shutdown_connections 2024-04-22T23:27:35.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.489+0000 7f215ee1a700 1 --2- 172.21.15.67:0/779857953 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2158101060 0x7f21581047b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:35.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.489+0000 7f215ee1a700 1 --2- 172.21.15.67:0/779857953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21581006b0 0x7f2158100a90 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:35.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.489+0000 7f215ee1a700 1 -- 172.21.15.67:0/779857953 >> 172.21.15.67:0/779857953 conn(0x7f2158074e30 msgr2=0x7f2158075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:27:35.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.490+0000 7f215ee1a700 1 -- 172.21.15.67:0/779857953 shutdown_connections 2024-04-22T23:27:35.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.490+0000 7f215ee1a700 1 -- 172.21.15.67:0/779857953 wait complete. 2024-04-22T23:27:35.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.490+0000 7f215ee1a700 1 Processor -- start 2024-04-22T23:27:35.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.490+0000 7f215ee1a700 1 -- start start 2024-04-22T23:27:35.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.490+0000 7f215ee1a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21581006b0 0x7f2158195660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:35.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.490+0000 7f215ee1a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2158101060 0x7f2158195ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:35.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.490+0000 7f215ee1a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21581960e0 con 0x7f21581006b0 2024-04-22T23:27:35.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.490+0000 7f215ee1a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2158196250 con 0x7f2158101060 2024-04-22T23:27:35.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.490+0000 7f215cbb6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21581006b0 0x7f2158195660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:35.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.490+0000 7f215cbb6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21581006b0 0x7f2158195660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42562/0 (socket says 172.21.15.67:42562) 2024-04-22T23:27:35.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.490+0000 7f215cbb6700 1 -- 172.21.15.67:0/2141290051 learned_addr learned my addr 172.21.15.67:0/2141290051 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:27:35.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.491+0000 7f2157fff700 1 --2- 172.21.15.67:0/2141290051 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2158101060 0x7f2158195ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:35.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.491+0000 7f215cbb6700 1 -- 172.21.15.67:0/2141290051 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2158101060 msgr2=0x7f2158195ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:35.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.491+0000 7f215cbb6700 1 --2- 172.21.15.67:0/2141290051 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2158101060 0x7f2158195ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:35.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.491+0000 7f215cbb6700 1 -- 172.21.15.67:0/2141290051 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f214c0095f0 con 0x7f21581006b0 2024-04-22T23:27:35.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.491+0000 7f215cbb6700 1 --2- 172.21.15.67:0/2141290051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21581006b0 0x7f2158195660 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f214c006010 tx=0x7f214c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:27:35.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.491+0000 7f2155ffb700 1 -- 172.21.15.67:0/2141290051 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f214c01d070 con 0x7f21581006b0 2024-04-22T23:27:35.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.491+0000 7f2155ffb700 1 -- 172.21.15.67:0/2141290051 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f214c004fb0 con 0x7f21581006b0 2024-04-22T23:27:35.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.491+0000 7f215ee1a700 1 -- 172.21.15.67:0/2141290051 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21581964d0 con 0x7f21581006b0 2024-04-22T23:27:35.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.491+0000 7f2155ffb700 1 -- 172.21.15.67:0/2141290051 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f214c022ea0 con 0x7f21581006b0 2024-04-22T23:27:35.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.491+0000 7f215ee1a700 1 -- 172.21.15.67:0/2141290051 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21581a2f50 con 0x7f21581006b0 2024-04-22T23:27:35.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.493+0000 7f215ee1a700 1 -- 172.21.15.67:0/2141290051 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2158108120 con 0x7f21581006b0 2024-04-22T23:27:35.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.493+0000 7f2155ffb700 1 -- 172.21.15.67:0/2141290051 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 33) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f214c022470 con 0x7f21581006b0 2024-04-22T23:27:35.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.493+0000 7f2155ffb700 1 --2- 172.21.15.67:0/2141290051 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f214806e940 0x7f2148070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:35.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.494+0000 7f2155ffb700 1 -- 172.21.15.67:0/2141290051 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f214c08fc20 con 0x7f21581006b0 2024-04-22T23:27:35.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.494+0000 7f2157fff700 1 --2- 172.21.15.67:0/2141290051 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f214806e940 0x7f2148070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:35.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.494+0000 7f2157fff700 1 --2- 172.21.15.67:0/2141290051 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f214806e940 0x7f2148070e00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2144006fd0 tx=0x7f2144008040 comp rx=0 tx=0).ready entity=mgr.14708 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:27:35.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.498+0000 7f2155ffb700 1 -- 172.21.15.67:0/2141290051 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f214c05cc30 con 0x7f21581006b0 2024-04-22T23:27:35.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.708+0000 7f215ee1a700 1 -- 172.21.15.67:0/2141290051 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7f21580ff9f0 con 0x7f21581006b0 2024-04-22T23:27:35.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.709+0000 7f2155ffb700 1 -- 172.21.15.67:0/2141290051 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v44)=0 v44) v1 ==== 135+0+0 (secure 0 0 0) 0x7f214c0270c0 con 0x7f21581006b0 2024-04-22T23:27:35.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.711+0000 7f215ee1a700 1 -- 172.21.15.67:0/2141290051 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f214806e940 msgr2=0x7f2148070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:35.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.711+0000 7f215ee1a700 1 --2- 172.21.15.67:0/2141290051 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f214806e940 0x7f2148070e00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2144006fd0 tx=0x7f2144008040 comp rx=0 tx=0).stop 2024-04-22T23:27:35.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.711+0000 7f215ee1a700 1 -- 172.21.15.67:0/2141290051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21581006b0 msgr2=0x7f2158195660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:35.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.711+0000 7f215ee1a700 1 --2- 172.21.15.67:0/2141290051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21581006b0 0x7f2158195660 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f214c006010 tx=0x7f214c00be20 comp rx=0 tx=0).stop 2024-04-22T23:27:35.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.711+0000 7f215ee1a700 1 -- 172.21.15.67:0/2141290051 shutdown_connections 2024-04-22T23:27:35.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.711+0000 7f215ee1a700 1 --2- 172.21.15.67:0/2141290051 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f214806e940 0x7f2148070e00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:35.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.711+0000 7f215ee1a700 1 --2- 172.21.15.67:0/2141290051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21581006b0 0x7f2158195660 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:35.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.711+0000 7f215ee1a700 1 --2- 172.21.15.67:0/2141290051 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2158101060 0x7f2158195ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:35.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.711+0000 7f215ee1a700 1 -- 172.21.15.67:0/2141290051 >> 172.21.15.67:0/2141290051 conn(0x7f2158074e30 msgr2=0x7f21580fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:27:35.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.711+0000 7f215ee1a700 1 -- 172.21.15.67:0/2141290051 shutdown_connections 2024-04-22T23:27:35.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:35.711+0000 7f215ee1a700 1 -- 172.21.15.67:0/2141290051 wait complete. 2024-04-22T23:27:35.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:35 smithi134 ceph-mon[34609]: pgmap v80: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:27:36.015 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:35 smithi067 ceph-mon[26453]: pgmap v80: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:27:36.340 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -e sha1=43be020184947e53516056c9931e1ac5bdbbb1a5 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-22T23:27:36.559 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:27:37.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:37 smithi067 ceph-mon[26453]: pgmap v81: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:27:37.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:37 smithi134 ceph-mon[34609]: pgmap v81: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:27:38.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.295+0000 7fa010fef700 1 -- 172.21.15.67:0/2452453129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa00c068580 msgr2=0x7fa00c0689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:38.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.295+0000 7fa010fef700 1 --2- 172.21.15.67:0/2452453129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa00c068580 0x7fa00c0689e0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fa000009910 tx=0x7fa000009c20 comp rx=0 tx=0).stop 2024-04-22T23:27:38.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.295+0000 7fa010fef700 1 -- 172.21.15.67:0/2452453129 shutdown_connections 2024-04-22T23:27:38.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.295+0000 7fa010fef700 1 --2- 172.21.15.67:0/2452453129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa00c068580 0x7fa00c0689e0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:38.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.295+0000 7fa010fef700 1 --2- 172.21.15.67:0/2452453129 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa00c106cd0 0x7fa00c1070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:38.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.295+0000 7fa010fef700 1 -- 172.21.15.67:0/2452453129 >> 172.21.15.67:0/2452453129 conn(0x7fa00c075670 msgr2=0x7fa00c075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:27:38.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.295+0000 7fa010fef700 1 -- 172.21.15.67:0/2452453129 shutdown_connections 2024-04-22T23:27:38.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.295+0000 7fa010fef700 1 -- 172.21.15.67:0/2452453129 wait complete. 2024-04-22T23:27:38.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.296+0000 7fa010fef700 1 Processor -- start 2024-04-22T23:27:38.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.296+0000 7fa010fef700 1 -- start start 2024-04-22T23:27:38.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.296+0000 7fa010fef700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa00c068580 0x7fa00c0ffcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:38.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.296+0000 7fa010fef700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa00c106cd0 0x7fa00c100230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:38.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.296+0000 7fa010fef700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa00c100770 con 0x7fa00c068580 2024-04-22T23:27:38.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.296+0000 7fa010fef700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa00c1008b0 con 0x7fa00c106cd0 2024-04-22T23:27:38.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.296+0000 7fa00a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa00c068580 0x7fa00c0ffcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:38.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.296+0000 7fa00a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa00c068580 0x7fa00c0ffcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42576/0 (socket says 172.21.15.67:42576) 2024-04-22T23:27:38.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.296+0000 7fa00a59c700 1 -- 172.21.15.67:0/3068006188 learned_addr learned my addr 172.21.15.67:0/3068006188 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:27:38.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.296+0000 7fa009d9b700 1 --2- 172.21.15.67:0/3068006188 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa00c106cd0 0x7fa00c100230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:38.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.297+0000 7fa00a59c700 1 -- 172.21.15.67:0/3068006188 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa00c106cd0 msgr2=0x7fa00c100230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:38.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.297+0000 7fa00a59c700 1 --2- 172.21.15.67:0/3068006188 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa00c106cd0 0x7fa00c100230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:38.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.297+0000 7fa00a59c700 1 -- 172.21.15.67:0/3068006188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0000095f0 con 0x7fa00c068580 2024-04-22T23:27:38.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.297+0000 7fa00a59c700 1 --2- 172.21.15.67:0/3068006188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa00c068580 0x7fa00c0ffcf0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc00c930 tx=0x7f9ffc00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:27:38.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.297+0000 7f9ffb7fe700 1 -- 172.21.15.67:0/3068006188 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ffc007ad0 con 0x7fa00c068580 2024-04-22T23:27:38.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.297+0000 7f9ffb7fe700 1 -- 172.21.15.67:0/3068006188 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ffc007c30 con 0x7fa00c068580 2024-04-22T23:27:38.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.297+0000 7f9ffb7fe700 1 -- 172.21.15.67:0/3068006188 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ffc018610 con 0x7fa00c068580 2024-04-22T23:27:38.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.297+0000 7fa010fef700 1 -- 172.21.15.67:0/3068006188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa00c1a2a40 con 0x7fa00c068580 2024-04-22T23:27:38.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.297+0000 7fa010fef700 1 -- 172.21.15.67:0/3068006188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa00c1a2da0 con 0x7fa00c068580 2024-04-22T23:27:38.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.298+0000 7fa010fef700 1 -- 172.21.15.67:0/3068006188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa00c1091d0 con 0x7fa00c068580 2024-04-22T23:27:38.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.299+0000 7f9ffb7fe700 1 -- 172.21.15.67:0/3068006188 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 33) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f9ffc00f450 con 0x7fa00c068580 2024-04-22T23:27:38.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.299+0000 7f9ffb7fe700 1 --2- 172.21.15.67:0/3068006188 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f9ff406e940 0x7f9ff4070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:38.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.299+0000 7f9ffb7fe700 1 -- 172.21.15.67:0/3068006188 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f9ffc08eb40 con 0x7fa00c068580 2024-04-22T23:27:38.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.299+0000 7fa009d9b700 1 --2- 172.21.15.67:0/3068006188 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f9ff406e940 0x7f9ff4070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:38.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.300+0000 7fa009d9b700 1 --2- 172.21.15.67:0/3068006188 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f9ff406e940 0x7f9ff4070e00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa000005f50 tx=0x7fa00001a040 comp rx=0 tx=0).ready entity=mgr.14708 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:27:38.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.304+0000 7f9ffb7fe700 1 -- 172.21.15.67:0/3068006188 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f9ffc057dd0 con 0x7fa00c068580 2024-04-22T23:27:38.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.534+0000 7fa010fef700 1 -- 172.21.15.67:0/3068006188 --> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:43be020184947e53516056c9931e1ac5bdbbb1a5", "target": ["mon-mgr", ""]}) v1 -- 0x7fa00c0611d0 con 0x7f9ff406e940 2024-04-22T23:27:38.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.540+0000 7f9ffb7fe700 1 -- 172.21.15.67:0/3068006188 <== mgr.14708 v2:172.21.15.67:6800/2306276228 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7fa00c0611d0 con 0x7f9ff406e940 2024-04-22T23:27:38.539 INFO:teuthology.orchestra.run.smithi067.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:43be020184947e53516056c9931e1ac5bdbbb1a5 2024-04-22T23:27:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.542+0000 7fa010fef700 1 -- 172.21.15.67:0/3068006188 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f9ff406e940 msgr2=0x7f9ff4070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.542+0000 7fa010fef700 1 --2- 172.21.15.67:0/3068006188 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f9ff406e940 0x7f9ff4070e00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa000005f50 tx=0x7fa00001a040 comp rx=0 tx=0).stop 2024-04-22T23:27:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.542+0000 7fa010fef700 1 -- 172.21.15.67:0/3068006188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa00c068580 msgr2=0x7fa00c0ffcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.542+0000 7fa010fef700 1 --2- 172.21.15.67:0/3068006188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa00c068580 0x7fa00c0ffcf0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc00c930 tx=0x7f9ffc00ccf0 comp rx=0 tx=0).stop 2024-04-22T23:27:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.542+0000 7fa010fef700 1 -- 172.21.15.67:0/3068006188 shutdown_connections 2024-04-22T23:27:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.542+0000 7fa010fef700 1 --2- 172.21.15.67:0/3068006188 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f9ff406e940 0x7f9ff4070e00 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.542+0000 7fa010fef700 1 --2- 172.21.15.67:0/3068006188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa00c068580 0x7fa00c0ffcf0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.542+0000 7fa010fef700 1 --2- 172.21.15.67:0/3068006188 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa00c106cd0 0x7fa00c100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:38.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.542+0000 7fa010fef700 1 -- 172.21.15.67:0/3068006188 >> 172.21.15.67:0/3068006188 conn(0x7fa00c075670 msgr2=0x7fa00c0fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:27:38.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.542+0000 7fa010fef700 1 -- 172.21.15.67:0/3068006188 shutdown_connections 2024-04-22T23:27:38.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:38.542+0000 7fa010fef700 1 -- 172.21.15.67:0/3068006188 wait complete. 2024-04-22T23:27:39.197 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-22T23:27:39.198 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi067.front.sepia.ceph.com 2024-04-22T23:27:39.198 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -e sha1=43be020184947e53516056c9931e1ac5bdbbb1a5 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-04-22T23:27:39.415 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-22T23:27:39.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:39 smithi067 ceph-mon[26453]: pgmap v82: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:27:39.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:39 smithi067 ceph-mon[26453]: from='client.14862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:43be020184947e53516056c9931e1ac5bdbbb1a5", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:27:39.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:39 smithi067 ceph-mon[26453]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:43be020184947e53516056c9931e1ac5bdbbb1a5 2024-04-22T23:27:39.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:39 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:27:39.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:39 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:27:39.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:39 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:27:39.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:39 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:27:39.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:39 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:27:39.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:39 smithi067 ceph-mon[26453]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:43be020184947e53516056c9931e1ac5bdbbb1a5 2024-04-22T23:27:39.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:39 smithi134 ceph-mon[34609]: pgmap v82: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:27:39.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:39 smithi134 ceph-mon[34609]: from='client.14862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:43be020184947e53516056c9931e1ac5bdbbb1a5", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:27:39.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:39 smithi134 ceph-mon[34609]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:43be020184947e53516056c9931e1ac5bdbbb1a5 2024-04-22T23:27:39.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:39 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:27:39.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:39 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:27:39.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:39 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:27:39.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:39 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:27:39.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:39 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:27:39.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:39 smithi134 ceph-mon[34609]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:43be020184947e53516056c9931e1ac5bdbbb1a5 2024-04-22T23:27:41.667 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:41 smithi067 ceph-mon[26453]: pgmap v83: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:27:41.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:41 smithi134 ceph-mon[34609]: pgmap v83: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:27:42.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.169+0000 7f0ffb59e700 1 -- 172.21.15.67:0/3564848553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ffc0ffa20 msgr2=0x7f0ffc0ffea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:42.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.169+0000 7f0ffb59e700 1 --2- 172.21.15.67:0/3564848553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ffc0ffa20 0x7f0ffc0ffea0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0009910 tx=0x7f0ff0009c20 comp rx=0 tx=0).stop 2024-04-22T23:27:42.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.169+0000 7f0ffb59e700 1 -- 172.21.15.67:0/3564848553 shutdown_connections 2024-04-22T23:27:42.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.169+0000 7f0ffb59e700 1 --2- 172.21.15.67:0/3564848553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ffc0ffa20 0x7f0ffc0ffea0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:42.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.169+0000 7f0ffb59e700 1 --2- 172.21.15.67:0/3564848553 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0ffc105950 0x7f0ffc105d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:42.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.169+0000 7f0ffb59e700 1 -- 172.21.15.67:0/3564848553 >> 172.21.15.67:0/3564848553 conn(0x7f0ffc075460 msgr2=0x7f0ffc075870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:27:42.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.169+0000 7f0ffb59e700 1 -- 172.21.15.67:0/3564848553 shutdown_connections 2024-04-22T23:27:42.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.169+0000 7f0ffb59e700 1 -- 172.21.15.67:0/3564848553 wait complete. 2024-04-22T23:27:42.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.170+0000 7f0ffb59e700 1 Processor -- start 2024-04-22T23:27:42.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.170+0000 7f0ffb59e700 1 -- start start 2024-04-22T23:27:42.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.170+0000 7f0ffb59e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ffc0ffa20 0x7f0ffc196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:42.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.170+0000 7f0ffb59e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0ffc105950 0x7f0ffc196b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:42.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.170+0000 7f0ffb59e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ffc197260 con 0x7f0ffc0ffa20 2024-04-22T23:27:42.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.170+0000 7f0ffb59e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ffc19aff0 con 0x7f0ffc105950 2024-04-22T23:27:42.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.171+0000 7f0ffa59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ffc0ffa20 0x7f0ffc196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:42.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.171+0000 7f0ffa59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ffc0ffa20 0x7f0ffc196640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58410/0 (socket says 172.21.15.67:58410) 2024-04-22T23:27:42.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.171+0000 7f0ffa59c700 1 -- 172.21.15.67:0/241465247 learned_addr learned my addr 172.21.15.67:0/241465247 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:27:42.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.171+0000 7f0ff9d9b700 1 --2- 172.21.15.67:0/241465247 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0ffc105950 0x7f0ffc196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:42.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.171+0000 7f0ffa59c700 1 -- 172.21.15.67:0/241465247 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0ffc105950 msgr2=0x7f0ffc196b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:42.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.171+0000 7f0ffa59c700 1 --2- 172.21.15.67:0/241465247 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0ffc105950 0x7f0ffc196b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:42.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.171+0000 7f0ffa59c700 1 -- 172.21.15.67:0/241465247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff00095f0 con 0x7f0ffc0ffa20 2024-04-22T23:27:42.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.171+0000 7f0ffa59c700 1 --2- 172.21.15.67:0/241465247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ffc0ffa20 0x7f0ffc196640 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f0fec00c8a0 tx=0x7f0fec00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:27:42.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.171+0000 7f0feb7fe700 1 -- 172.21.15.67:0/241465247 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fec0078e0 con 0x7f0ffc0ffa20 2024-04-22T23:27:42.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.171+0000 7f0feb7fe700 1 -- 172.21.15.67:0/241465247 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0fec00f450 con 0x7f0ffc0ffa20 2024-04-22T23:27:42.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.171+0000 7f0ffb59e700 1 -- 172.21.15.67:0/241465247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ffc19b2d0 con 0x7f0ffc0ffa20 2024-04-22T23:27:42.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.171+0000 7f0feb7fe700 1 -- 172.21.15.67:0/241465247 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fec00e5e0 con 0x7f0ffc0ffa20 2024-04-22T23:27:42.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.171+0000 7f0ffb59e700 1 -- 172.21.15.67:0/241465247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ffc19b820 con 0x7f0ffc0ffa20 2024-04-22T23:27:42.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.172+0000 7f0ffb59e700 1 -- 172.21.15.67:0/241465247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fdc005320 con 0x7f0ffc0ffa20 2024-04-22T23:27:42.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.173+0000 7f0feb7fe700 1 -- 172.21.15.67:0/241465247 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 33) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f0fec00fa80 con 0x7f0ffc0ffa20 2024-04-22T23:27:42.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.173+0000 7f0feb7fe700 1 --2- 172.21.15.67:0/241465247 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f0fe406e940 0x7f0fe4070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:42.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.174+0000 7f0feb7fe700 1 -- 172.21.15.67:0/241465247 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f0fec08e850 con 0x7f0ffc0ffa20 2024-04-22T23:27:42.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.174+0000 7f0ff9d9b700 1 --2- 172.21.15.67:0/241465247 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f0fe406e940 0x7f0fe4070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:42.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.174+0000 7f0ff9d9b700 1 --2- 172.21.15.67:0/241465247 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f0fe406e940 0x7f0fe4070e00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc197c60 tx=0x7f0ff001a040 comp rx=0 tx=0).ready entity=mgr.14708 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:27:42.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.178+0000 7f0feb7fe700 1 -- 172.21.15.67:0/241465247 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f0fec05b610 con 0x7f0ffc0ffa20 2024-04-22T23:27:42.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.413+0000 7f0ffb59e700 1 -- 172.21.15.67:0/241465247 --> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0fdc000bf0 con 0x7f0fe406e940 2024-04-22T23:27:42.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.414+0000 7f0feb7fe700 1 -- 172.21.15.67:0/241465247 <== mgr.14708 v2:172.21.15.67:6800/2306276228 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f0fdc000bf0 con 0x7f0fe406e940 2024-04-22T23:27:42.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.416+0000 7f0ffb59e700 1 -- 172.21.15.67:0/241465247 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f0fe406e940 msgr2=0x7f0fe4070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:42.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.416+0000 7f0ffb59e700 1 --2- 172.21.15.67:0/241465247 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f0fe406e940 0x7f0fe4070e00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc197c60 tx=0x7f0ff001a040 comp rx=0 tx=0).stop 2024-04-22T23:27:42.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.416+0000 7f0ffb59e700 1 -- 172.21.15.67:0/241465247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ffc0ffa20 msgr2=0x7f0ffc196640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:42.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.416+0000 7f0ffb59e700 1 --2- 172.21.15.67:0/241465247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ffc0ffa20 0x7f0ffc196640 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f0fec00c8a0 tx=0x7f0fec00cbb0 comp rx=0 tx=0).stop 2024-04-22T23:27:42.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.416+0000 7f0ffb59e700 1 -- 172.21.15.67:0/241465247 shutdown_connections 2024-04-22T23:27:42.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.416+0000 7f0ffb59e700 1 --2- 172.21.15.67:0/241465247 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f0fe406e940 0x7f0fe4070e00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:42.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.416+0000 7f0ffb59e700 1 --2- 172.21.15.67:0/241465247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ffc0ffa20 0x7f0ffc196640 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:42.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.416+0000 7f0ffb59e700 1 --2- 172.21.15.67:0/241465247 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0ffc105950 0x7f0ffc196b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:42.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.416+0000 7f0ffb59e700 1 -- 172.21.15.67:0/241465247 >> 172.21.15.67:0/241465247 conn(0x7f0ffc075460 msgr2=0x7f0ffc068600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:27:42.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.417+0000 7f0ffb59e700 1 -- 172.21.15.67:0/241465247 shutdown_connections 2024-04-22T23:27:42.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.417+0000 7f0ffb59e700 1 -- 172.21.15.67:0/241465247 wait complete. 2024-04-22T23:27:42.427 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:27:42.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.525+0000 7f75d2c5d700 1 -- 172.21.15.67:0/97720204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75cc101850 msgr2=0x7f75cc1058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:42.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.525+0000 7f75d2c5d700 1 --2- 172.21.15.67:0/97720204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75cc101850 0x7f75cc1058a0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f75b8009910 tx=0x7f75b8009c20 comp rx=0 tx=0).stop 2024-04-22T23:27:42.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.525+0000 7f75d2c5d700 1 -- 172.21.15.67:0/97720204 shutdown_connections 2024-04-22T23:27:42.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.525+0000 7f75d2c5d700 1 --2- 172.21.15.67:0/97720204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75cc101850 0x7f75cc1058a0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:42.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.525+0000 7f75d2c5d700 1 --2- 172.21.15.67:0/97720204 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75cc100ea0 0x7f75cc101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:42.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.525+0000 7f75d2c5d700 1 -- 172.21.15.67:0/97720204 >> 172.21.15.67:0/97720204 conn(0x7f75cc0fc750 msgr2=0x7f75cc0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:27:42.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.525+0000 7f75d2c5d700 1 -- 172.21.15.67:0/97720204 shutdown_connections 2024-04-22T23:27:42.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.525+0000 7f75d2c5d700 1 -- 172.21.15.67:0/97720204 wait complete. 2024-04-22T23:27:42.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.525+0000 7f75d2c5d700 1 Processor -- start 2024-04-22T23:27:42.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.525+0000 7f75d2c5d700 1 -- start start 2024-04-22T23:27:42.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.526+0000 7f75d2c5d700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75cc100ea0 0x7f75cc19c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:42.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.526+0000 7f75d2c5d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75cc101850 0x7f75cc19ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:42.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.526+0000 7f75d2c5d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75cc19d360 con 0x7f75cc101850 2024-04-22T23:27:42.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.526+0000 7f75d2c5d700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75cc196810 con 0x7f75cc100ea0 2024-04-22T23:27:42.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.526+0000 7f75cbfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75cc101850 0x7f75cc19ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:42.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.526+0000 7f75cbfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75cc101850 0x7f75cc19ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58424/0 (socket says 172.21.15.67:58424) 2024-04-22T23:27:42.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.526+0000 7f75cbfff700 1 -- 172.21.15.67:0/2432938018 learned_addr learned my addr 172.21.15.67:0/2432938018 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:27:42.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.526+0000 7f75d09f9700 1 --2- 172.21.15.67:0/2432938018 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75cc100ea0 0x7f75cc19c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:42.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.526+0000 7f75cbfff700 1 -- 172.21.15.67:0/2432938018 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75cc100ea0 msgr2=0x7f75cc19c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:42.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.526+0000 7f75cbfff700 1 --2- 172.21.15.67:0/2432938018 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75cc100ea0 0x7f75cc19c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:42.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.526+0000 7f75cbfff700 1 -- 172.21.15.67:0/2432938018 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b80095f0 con 0x7f75cc101850 2024-04-22T23:27:42.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.526+0000 7f75cbfff700 1 --2- 172.21.15.67:0/2432938018 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75cc101850 0x7f75cc19ccd0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f75b8000c00 tx=0x7f75b800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:27:42.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.526+0000 7f75c9ffb700 1 -- 172.21.15.67:0/2432938018 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75b801d070 con 0x7f75cc101850 2024-04-22T23:27:42.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.527+0000 7f75d2c5d700 1 -- 172.21.15.67:0/2432938018 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75cc196a90 con 0x7f75cc101850 2024-04-22T23:27:42.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.527+0000 7f75c9ffb700 1 -- 172.21.15.67:0/2432938018 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f75b8004fb0 con 0x7f75cc101850 2024-04-22T23:27:42.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.527+0000 7f75c9ffb700 1 -- 172.21.15.67:0/2432938018 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75b800f460 con 0x7f75cc101850 2024-04-22T23:27:42.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.527+0000 7f75d2c5d700 1 -- 172.21.15.67:0/2432938018 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75cc196f80 con 0x7f75cc101850 2024-04-22T23:27:42.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.528+0000 7f75d2c5d700 1 -- 172.21.15.67:0/2432938018 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75cc1091f0 con 0x7f75cc101850 2024-04-22T23:27:42.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.528+0000 7f75c9ffb700 1 -- 172.21.15.67:0/2432938018 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 33) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f75b80228a0 con 0x7f75cc101850 2024-04-22T23:27:42.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.529+0000 7f75c9ffb700 1 --2- 172.21.15.67:0/2432938018 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f75bc06e940 0x7f75bc070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:42.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.529+0000 7f75c9ffb700 1 -- 172.21.15.67:0/2432938018 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f75b808fee0 con 0x7f75cc101850 2024-04-22T23:27:42.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.529+0000 7f75d09f9700 1 --2- 172.21.15.67:0/2432938018 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f75bc06e940 0x7f75bc070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:42.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.529+0000 7f75d09f9700 1 --2- 172.21.15.67:0/2432938018 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f75bc06e940 0x7f75bc070e00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f75c0006fd0 tx=0x7f75c0008040 comp rx=0 tx=0).ready entity=mgr.14708 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:27:42.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.534+0000 7f75c9ffb700 1 -- 172.21.15.67:0/2432938018 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f75b805cd50 con 0x7f75cc101850 2024-04-22T23:27:42.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.763+0000 7f75d2c5d700 1 -- 172.21.15.67:0/2432938018 --> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f75cc197e30 con 0x7f75bc06e940 2024-04-22T23:27:42.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.764+0000 7f75c9ffb700 1 -- 172.21.15.67:0/2432938018 <== mgr.14708 v2:172.21.15.67:6800/2306276228 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f75cc197e30 con 0x7f75bc06e940 2024-04-22T23:27:42.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.766+0000 7f75d2c5d700 1 -- 172.21.15.67:0/2432938018 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f75bc06e940 msgr2=0x7f75bc070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:42.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.766+0000 7f75d2c5d700 1 --2- 172.21.15.67:0/2432938018 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f75bc06e940 0x7f75bc070e00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f75c0006fd0 tx=0x7f75c0008040 comp rx=0 tx=0).stop 2024-04-22T23:27:42.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.766+0000 7f75d2c5d700 1 -- 172.21.15.67:0/2432938018 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75cc101850 msgr2=0x7f75cc19ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:42.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.766+0000 7f75d2c5d700 1 --2- 172.21.15.67:0/2432938018 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75cc101850 0x7f75cc19ccd0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f75b8000c00 tx=0x7f75b800be20 comp rx=0 tx=0).stop 2024-04-22T23:27:42.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.766+0000 7f75d2c5d700 1 -- 172.21.15.67:0/2432938018 shutdown_connections 2024-04-22T23:27:42.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.766+0000 7f75d2c5d700 1 --2- 172.21.15.67:0/2432938018 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f75bc06e940 0x7f75bc070e00 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:42.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.766+0000 7f75d2c5d700 1 --2- 172.21.15.67:0/2432938018 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75cc100ea0 0x7f75cc19c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:42.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.766+0000 7f75d2c5d700 1 --2- 172.21.15.67:0/2432938018 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75cc101850 0x7f75cc19ccd0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:42.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.766+0000 7f75d2c5d700 1 -- 172.21.15.67:0/2432938018 >> 172.21.15.67:0/2432938018 conn(0x7f75cc0fc750 msgr2=0x7f75cc0fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:27:42.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.766+0000 7f75d2c5d700 1 -- 172.21.15.67:0/2432938018 shutdown_connections 2024-04-22T23:27:42.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.766+0000 7f75d2c5d700 1 -- 172.21.15.67:0/2432938018 wait complete. 2024-04-22T23:27:42.838 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:42 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:27:42.838 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:42 smithi067 ceph-mon[26453]: Upgrade: Target is version 19.0.0-3252-g43be0201 (squid) 2024-04-22T23:27:42.838 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:42 smithi067 ceph-mon[26453]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, digests ['quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c'] 2024-04-22T23:27:42.838 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:42 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:27:42.838 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:42 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:27:42.838 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:42 smithi067 ceph-mon[26453]: Upgrade: Setting container_image for all mgr 2024-04-22T23:27:42.838 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:42 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:27:42.839 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:42 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-22T23:27:42.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.875+0000 7ff432b67700 1 -- 172.21.15.67:0/2420314598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff42c106cd0 msgr2=0x7ff42c1070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:42.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.875+0000 7ff432b67700 1 --2- 172.21.15.67:0/2420314598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff42c106cd0 0x7ff42c1070b0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ff420009910 tx=0x7ff420009c20 comp rx=0 tx=0).stop 2024-04-22T23:27:42.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.875+0000 7ff432b67700 1 -- 172.21.15.67:0/2420314598 shutdown_connections 2024-04-22T23:27:42.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.875+0000 7ff432b67700 1 --2- 172.21.15.67:0/2420314598 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff42c068580 0x7ff42c0689e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:42.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.875+0000 7ff432b67700 1 --2- 172.21.15.67:0/2420314598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff42c106cd0 0x7ff42c1070b0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:42.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.875+0000 7ff432b67700 1 -- 172.21.15.67:0/2420314598 >> 172.21.15.67:0/2420314598 conn(0x7ff42c075670 msgr2=0x7ff42c075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:27:42.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.875+0000 7ff432b67700 1 -- 172.21.15.67:0/2420314598 shutdown_connections 2024-04-22T23:27:42.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.875+0000 7ff432b67700 1 -- 172.21.15.67:0/2420314598 wait complete. 2024-04-22T23:27:42.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.875+0000 7ff432b67700 1 Processor -- start 2024-04-22T23:27:42.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.876+0000 7ff432b67700 1 -- start start 2024-04-22T23:27:42.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.876+0000 7ff432b67700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff42c068580 0x7ff42c0ffcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:42.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.876+0000 7ff432b67700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff42c106cd0 0x7ff42c100230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:42.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.876+0000 7ff432b67700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42c100770 con 0x7ff42c106cd0 2024-04-22T23:27:42.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.876+0000 7ff432b67700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42c1008b0 con 0x7ff42c068580 2024-04-22T23:27:42.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.876+0000 7ff42bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff42c106cd0 0x7ff42c100230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:42.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.876+0000 7ff42bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff42c106cd0 0x7ff42c100230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58452/0 (socket says 172.21.15.67:58452) 2024-04-22T23:27:42.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.876+0000 7ff42bfff700 1 -- 172.21.15.67:0/266119780 learned_addr learned my addr 172.21.15.67:0/266119780 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:27:42.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.876+0000 7ff430903700 1 --2- 172.21.15.67:0/266119780 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff42c068580 0x7ff42c0ffcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:42.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.876+0000 7ff42bfff700 1 -- 172.21.15.67:0/266119780 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff42c068580 msgr2=0x7ff42c0ffcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:42.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.876+0000 7ff42bfff700 1 --2- 172.21.15.67:0/266119780 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff42c068580 0x7ff42c0ffcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:42.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.876+0000 7ff42bfff700 1 -- 172.21.15.67:0/266119780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4200095f0 con 0x7ff42c106cd0 2024-04-22T23:27:42.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.876+0000 7ff42bfff700 1 --2- 172.21.15.67:0/266119780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff42c106cd0 0x7ff42c100230 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7ff41800c930 tx=0x7ff41800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:27:42.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.877+0000 7ff429ffb700 1 -- 172.21.15.67:0/266119780 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff418007ad0 con 0x7ff42c106cd0 2024-04-22T23:27:42.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.877+0000 7ff432b67700 1 -- 172.21.15.67:0/266119780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff42c1a2a60 con 0x7ff42c106cd0 2024-04-22T23:27:42.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.877+0000 7ff429ffb700 1 -- 172.21.15.67:0/266119780 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff418007c30 con 0x7ff42c106cd0 2024-04-22T23:27:42.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.877+0000 7ff429ffb700 1 -- 172.21.15.67:0/266119780 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff418018610 con 0x7ff42c106cd0 2024-04-22T23:27:42.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.878+0000 7ff432b67700 1 -- 172.21.15.67:0/266119780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff42c1a2dc0 con 0x7ff42c106cd0 2024-04-22T23:27:42.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.878+0000 7ff429ffb700 1 -- 172.21.15.67:0/266119780 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 33) v1 ==== 92598+0+0 (secure 0 0 0) 0x7ff41800f450 con 0x7ff42c106cd0 2024-04-22T23:27:42.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.878+0000 7ff432b67700 1 -- 172.21.15.67:0/266119780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff42c1091d0 con 0x7ff42c106cd0 2024-04-22T23:27:42.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.879+0000 7ff429ffb700 1 --2- 172.21.15.67:0/266119780 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7ff41c06e940 0x7ff41c070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:42.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.879+0000 7ff429ffb700 1 -- 172.21.15.67:0/266119780 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7ff41808eb00 con 0x7ff42c106cd0 2024-04-22T23:27:42.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.879+0000 7ff430903700 1 --2- 172.21.15.67:0/266119780 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7ff41c06e940 0x7ff41c070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:42.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.880+0000 7ff430903700 1 --2- 172.21.15.67:0/266119780 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7ff41c06e940 0x7ff41c070e00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff42000b5c0 tx=0x7ff42001a040 comp rx=0 tx=0).ready entity=mgr.14708 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:27:42.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:42.888+0000 7ff429ffb700 1 -- 172.21.15.67:0/266119780 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7ff418057e10 con 0x7ff42c106cd0 2024-04-22T23:27:42.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:42 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:27:42.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:42 smithi134 ceph-mon[34609]: Upgrade: Target is version 19.0.0-3252-g43be0201 (squid) 2024-04-22T23:27:42.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:42 smithi134 ceph-mon[34609]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, digests ['quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c'] 2024-04-22T23:27:42.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:42 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:27:42.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:42 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:27:42.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:42 smithi134 ceph-mon[34609]: Upgrade: Setting container_image for all mgr 2024-04-22T23:27:42.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:42 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:27:42.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:42 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-22T23:27:43.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.116+0000 7ff432b67700 1 -- 172.21.15.67:0/266119780 --> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff42c0611d0 con 0x7ff41c06e940 2024-04-22T23:27:43.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.127+0000 7ff429ffb700 1 -- 172.21.15.67:0/266119780 <== mgr.14708 v2:172.21.15.67:6800/2306276228 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7ff42c0611d0 con 0x7ff41c06e940 2024-04-22T23:27:43.127 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:27:43.127 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (86s) 57s ago 12m 16.3M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:27:43.127 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (12m) 57s ago 12m 8048k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:27:43.127 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (11m) 118s ago 11m 7931k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:27:43.127 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (12m) 57s ago 12m 6660k - 18.2.1 5be31c24972a 14085634c5d3 2024-04-22T23:27:43.127 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (11m) 118s ago 11m 6665k - 18.2.1 5be31c24972a fa995d458353 2024-04-22T23:27:43.127 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (63s) 57s ago 12m 51.2M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:27:43.127 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (7m) 57s ago 7m 62.7M - 18.2.1 5be31c24972a e24a471f0a8c 2024-04-22T23:27:43.128 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (7m) 57s ago 7m 130M - 18.2.1 5be31c24972a 0730f9fc8e61 2024-04-22T23:27:43.128 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (7m) 118s ago 7m 123M - 18.2.1 5be31c24972a b6a169586433 2024-04-22T23:27:43.128 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (7m) 118s ago 7m 67.5M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:27:43.128 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (3m) 57s ago 14m 508M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:27:43.128 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (2m) 118s ago 11m 449M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:27:43.128 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (14m) 57s ago 14m 59.0M 2048M 18.2.1 5be31c24972a 6789283703e0 2024-04-22T23:27:43.128 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (11m) 118s ago 11m 46.6M 2048M 18.2.1 5be31c24972a 2fb23795f7e3 2024-04-22T23:27:43.128 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (2m) 57s ago 12m 15.6M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:27:43.128 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (2m) 118s ago 11m 3376k - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:27:43.128 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (10m) 57s ago 10m 205M 1216M 18.2.1 5be31c24972a a033247ee8c2 2024-04-22T23:27:43.128 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (10m) 57s ago 10m 211M 1216M 18.2.1 5be31c24972a 915b07a8edc4 2024-04-22T23:27:43.128 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (10m) 57s ago 10m 172M 1216M 18.2.1 5be31c24972a e257ecb4efa1 2024-04-22T23:27:43.128 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (9m) 118s ago 9m 248M 2240M 18.2.1 5be31c24972a 9c2c91a967ba 2024-04-22T23:27:43.128 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (9m) 118s ago 9m 217M 2240M 18.2.1 5be31c24972a c7586562ffa6 2024-04-22T23:27:43.128 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (8m) 118s ago 8m 192M 2240M 18.2.1 5be31c24972a 9fd97f1c65cb 2024-04-22T23:27:43.129 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (104s) 57s ago 12m 53.8M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:27:43.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.129+0000 7ff432b67700 1 -- 172.21.15.67:0/266119780 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7ff41c06e940 msgr2=0x7ff41c070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:43.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.129+0000 7ff432b67700 1 --2- 172.21.15.67:0/266119780 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7ff41c06e940 0x7ff41c070e00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff42000b5c0 tx=0x7ff42001a040 comp rx=0 tx=0).stop 2024-04-22T23:27:43.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.130+0000 7ff432b67700 1 -- 172.21.15.67:0/266119780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff42c106cd0 msgr2=0x7ff42c100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:43.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.130+0000 7ff432b67700 1 --2- 172.21.15.67:0/266119780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff42c106cd0 0x7ff42c100230 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7ff41800c930 tx=0x7ff41800ccf0 comp rx=0 tx=0).stop 2024-04-22T23:27:43.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.130+0000 7ff432b67700 1 -- 172.21.15.67:0/266119780 shutdown_connections 2024-04-22T23:27:43.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.130+0000 7ff432b67700 1 --2- 172.21.15.67:0/266119780 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7ff41c06e940 0x7ff41c070e00 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:43.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.130+0000 7ff432b67700 1 --2- 172.21.15.67:0/266119780 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff42c068580 0x7ff42c0ffcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:43.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.130+0000 7ff432b67700 1 --2- 172.21.15.67:0/266119780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff42c106cd0 0x7ff42c100230 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:43.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.130+0000 7ff432b67700 1 -- 172.21.15.67:0/266119780 >> 172.21.15.67:0/266119780 conn(0x7ff42c075670 msgr2=0x7ff42c0fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:27:43.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.130+0000 7ff432b67700 1 -- 172.21.15.67:0/266119780 shutdown_connections 2024-04-22T23:27:43.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.130+0000 7ff432b67700 1 -- 172.21.15.67:0/266119780 wait complete. 2024-04-22T23:27:43.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.236+0000 7f913eb12700 1 -- 172.21.15.67:0/2237143212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91380ffda0 msgr2=0x7f9138100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:43.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.236+0000 7f913eb12700 1 --2- 172.21.15.67:0/2237143212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91380ffda0 0x7f9138100180 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f912c009910 tx=0x7f912c009c20 comp rx=0 tx=0).stop 2024-04-22T23:27:43.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.236+0000 7f913eb12700 1 -- 172.21.15.67:0/2237143212 shutdown_connections 2024-04-22T23:27:43.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.236+0000 7f913eb12700 1 --2- 172.21.15.67:0/2237143212 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9138100750 0x7f91381047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:43.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.236+0000 7f913eb12700 1 --2- 172.21.15.67:0/2237143212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91380ffda0 0x7f9138100180 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:43.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.236+0000 7f913eb12700 1 -- 172.21.15.67:0/2237143212 >> 172.21.15.67:0/2237143212 conn(0x7f91380fb650 msgr2=0x7f91380fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:27:43.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.237+0000 7f913eb12700 1 -- 172.21.15.67:0/2237143212 shutdown_connections 2024-04-22T23:27:43.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.237+0000 7f913eb12700 1 -- 172.21.15.67:0/2237143212 wait complete. 2024-04-22T23:27:43.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.237+0000 7f913eb12700 1 Processor -- start 2024-04-22T23:27:43.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.237+0000 7f913eb12700 1 -- start start 2024-04-22T23:27:43.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.237+0000 7f913eb12700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91380ffda0 0x7f91381976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:43.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.237+0000 7f913eb12700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9138100750 0x7f9138197c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:43.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.237+0000 7f913eb12700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91381981a0 con 0x7f91380ffda0 2024-04-22T23:27:43.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.237+0000 7f913eb12700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9138198310 con 0x7f9138100750 2024-04-22T23:27:43.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.237+0000 7f913c8ae700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91380ffda0 0x7f91381976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:43.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.237+0000 7f913c8ae700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91380ffda0 0x7f91381976d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58466/0 (socket says 172.21.15.67:58466) 2024-04-22T23:27:43.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.238+0000 7f913c8ae700 1 -- 172.21.15.67:0/2778597947 learned_addr learned my addr 172.21.15.67:0/2778597947 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:27:43.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.238+0000 7f9137fff700 1 --2- 172.21.15.67:0/2778597947 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9138100750 0x7f9138197c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:43.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.238+0000 7f913c8ae700 1 -- 172.21.15.67:0/2778597947 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9138100750 msgr2=0x7f9138197c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:43.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.238+0000 7f913c8ae700 1 --2- 172.21.15.67:0/2778597947 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9138100750 0x7f9138197c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:43.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.238+0000 7f913c8ae700 1 -- 172.21.15.67:0/2778597947 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f912c0095f0 con 0x7f91380ffda0 2024-04-22T23:27:43.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.238+0000 7f913c8ae700 1 --2- 172.21.15.67:0/2778597947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91380ffda0 0x7f91381976d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f912c006010 tx=0x7f912c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:27:43.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.238+0000 7f9135ffb700 1 -- 172.21.15.67:0/2778597947 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f912c01d070 con 0x7f91380ffda0 2024-04-22T23:27:43.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.238+0000 7f9135ffb700 1 -- 172.21.15.67:0/2778597947 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f912c004fb0 con 0x7f91380ffda0 2024-04-22T23:27:43.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.238+0000 7f913eb12700 1 -- 172.21.15.67:0/2778597947 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91381a0740 con 0x7f91380ffda0 2024-04-22T23:27:43.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.238+0000 7f9135ffb700 1 -- 172.21.15.67:0/2778597947 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f912c022ea0 con 0x7f91380ffda0 2024-04-22T23:27:43.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.238+0000 7f913eb12700 1 -- 172.21.15.67:0/2778597947 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91381a0af0 con 0x7f91380ffda0 2024-04-22T23:27:43.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.239+0000 7f913eb12700 1 -- 172.21.15.67:0/2778597947 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9138108140 con 0x7f91380ffda0 2024-04-22T23:27:43.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.240+0000 7f9135ffb700 1 -- 172.21.15.67:0/2778597947 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 33) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f912c022470 con 0x7f91380ffda0 2024-04-22T23:27:43.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.240+0000 7f9135ffb700 1 --2- 172.21.15.67:0/2778597947 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f912806e940 0x7f9128070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:43.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.240+0000 7f9135ffb700 1 -- 172.21.15.67:0/2778597947 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f912c08fe80 con 0x7f91380ffda0 2024-04-22T23:27:43.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.240+0000 7f9137fff700 1 --2- 172.21.15.67:0/2778597947 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f912806e940 0x7f9128070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:43.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.241+0000 7f9137fff700 1 --2- 172.21.15.67:0/2778597947 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f912806e940 0x7f9128070e00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f91380696e0 tx=0x7f9124005e10 comp rx=0 tx=0).ready entity=mgr.14708 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:27:43.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.246+0000 7f9135ffb700 1 -- 172.21.15.67:0/2778597947 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f912c0591c0 con 0x7f91380ffda0 2024-04-22T23:27:43.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.546+0000 7f913eb12700 1 -- 172.21.15.67:0/2778597947 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9138069520 con 0x7f91380ffda0 2024-04-22T23:27:43.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.546+0000 7f9135ffb700 1 -- 172.21.15.67:0/2778597947 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+675 (secure 0 0 0) 0x7f912c022720 con 0x7f91380ffda0 2024-04-22T23:27:43.546 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:27:43.546 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:27:43.546 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-22T23:27:43.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:27:43.546 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:27:43.546 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:27:43.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:27:43.546 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:27:43.547 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-22T23:27:43.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:27:43.547 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:27:43.547 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-22T23:27:43.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:27:43.547 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:27:43.547 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 12, 2024-04-22T23:27:43.547 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:27:43.547 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:27:43.547 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:27:43.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.548+0000 7f913eb12700 1 -- 172.21.15.67:0/2778597947 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f912806e940 msgr2=0x7f9128070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:43.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.548+0000 7f913eb12700 1 --2- 172.21.15.67:0/2778597947 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f912806e940 0x7f9128070e00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f91380696e0 tx=0x7f9124005e10 comp rx=0 tx=0).stop 2024-04-22T23:27:43.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.548+0000 7f913eb12700 1 -- 172.21.15.67:0/2778597947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91380ffda0 msgr2=0x7f91381976d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:43.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.548+0000 7f913eb12700 1 --2- 172.21.15.67:0/2778597947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91380ffda0 0x7f91381976d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f912c006010 tx=0x7f912c00be20 comp rx=0 tx=0).stop 2024-04-22T23:27:43.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.548+0000 7f913eb12700 1 -- 172.21.15.67:0/2778597947 shutdown_connections 2024-04-22T23:27:43.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.548+0000 7f913eb12700 1 --2- 172.21.15.67:0/2778597947 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f912806e940 0x7f9128070e00 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:43.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.548+0000 7f913eb12700 1 --2- 172.21.15.67:0/2778597947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91380ffda0 0x7f91381976d0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:43.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.548+0000 7f913eb12700 1 --2- 172.21.15.67:0/2778597947 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9138100750 0x7f9138197c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:43.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.548+0000 7f913eb12700 1 -- 172.21.15.67:0/2778597947 >> 172.21.15.67:0/2778597947 conn(0x7f91380fb650 msgr2=0x7f91380fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:27:43.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.549+0000 7f913eb12700 1 -- 172.21.15.67:0/2778597947 shutdown_connections 2024-04-22T23:27:43.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.549+0000 7f913eb12700 1 -- 172.21.15.67:0/2778597947 wait complete. 2024-04-22T23:27:43.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.669+0000 7fae8098f700 1 -- 172.21.15.67:0/3432022104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae7c068580 msgr2=0x7fae7c0689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:43.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.669+0000 7fae8098f700 1 --2- 172.21.15.67:0/3432022104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae7c068580 0x7fae7c0689e0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fae64009910 tx=0x7fae64009c20 comp rx=0 tx=0).stop 2024-04-22T23:27:43.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.670+0000 7fae8098f700 1 -- 172.21.15.67:0/3432022104 shutdown_connections 2024-04-22T23:27:43.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.670+0000 7fae8098f700 1 --2- 172.21.15.67:0/3432022104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae7c068580 0x7fae7c0689e0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:43.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.670+0000 7fae8098f700 1 --2- 172.21.15.67:0/3432022104 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fae7c106cd0 0x7fae7c1070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:43.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.670+0000 7fae8098f700 1 -- 172.21.15.67:0/3432022104 >> 172.21.15.67:0/3432022104 conn(0x7fae7c075670 msgr2=0x7fae7c075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:27:43.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.670+0000 7fae8098f700 1 -- 172.21.15.67:0/3432022104 shutdown_connections 2024-04-22T23:27:43.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.670+0000 7fae8098f700 1 -- 172.21.15.67:0/3432022104 wait complete. 2024-04-22T23:27:43.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.671+0000 7fae8098f700 1 Processor -- start 2024-04-22T23:27:43.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.671+0000 7fae8098f700 1 -- start start 2024-04-22T23:27:43.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.671+0000 7fae8098f700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fae7c068580 0x7fae7c19c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:43.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.671+0000 7fae8098f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae7c106cd0 0x7fae7c19cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:43.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.671+0000 7fae8098f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7c19d200 con 0x7fae7c106cd0 2024-04-22T23:27:43.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.671+0000 7fae8098f700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7c1966b0 con 0x7fae7c068580 2024-04-22T23:27:43.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.671+0000 7fae79d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae7c106cd0 0x7fae7c19cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:43.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.671+0000 7fae79d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae7c106cd0 0x7fae7c19cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58488/0 (socket says 172.21.15.67:58488) 2024-04-22T23:27:43.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.671+0000 7fae79d9b700 1 -- 172.21.15.67:0/2972457929 learned_addr learned my addr 172.21.15.67:0/2972457929 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:27:43.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.672+0000 7fae7a59c700 1 --2- 172.21.15.67:0/2972457929 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fae7c068580 0x7fae7c19c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:43.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.672+0000 7fae79d9b700 1 -- 172.21.15.67:0/2972457929 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fae7c068580 msgr2=0x7fae7c19c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:43.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.672+0000 7fae79d9b700 1 --2- 172.21.15.67:0/2972457929 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fae7c068580 0x7fae7c19c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:43.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.672+0000 7fae79d9b700 1 -- 172.21.15.67:0/2972457929 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae640095f0 con 0x7fae7c106cd0 2024-04-22T23:27:43.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.672+0000 7fae79d9b700 1 --2- 172.21.15.67:0/2972457929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae7c106cd0 0x7fae7c19cb70 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fae64006010 tx=0x7fae6400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:27:43.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.672+0000 7fae737fe700 1 -- 172.21.15.67:0/2972457929 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae6401d070 con 0x7fae7c106cd0 2024-04-22T23:27:43.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.672+0000 7fae737fe700 1 -- 172.21.15.67:0/2972457929 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fae64005dc0 con 0x7fae7c106cd0 2024-04-22T23:27:43.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.672+0000 7fae8098f700 1 -- 172.21.15.67:0/2972457929 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae7c196930 con 0x7fae7c106cd0 2024-04-22T23:27:43.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.672+0000 7fae737fe700 1 -- 172.21.15.67:0/2972457929 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae64017400 con 0x7fae7c106cd0 2024-04-22T23:27:43.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.672+0000 7fae8098f700 1 -- 172.21.15.67:0/2972457929 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae7c196e20 con 0x7fae7c106cd0 2024-04-22T23:27:43.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.674+0000 7fae8098f700 1 -- 172.21.15.67:0/2972457929 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae7c1091d0 con 0x7fae7c106cd0 2024-04-22T23:27:43.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.674+0000 7fae737fe700 1 -- 172.21.15.67:0/2972457929 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 33) v1 ==== 92598+0+0 (secure 0 0 0) 0x7fae64003a60 con 0x7fae7c106cd0 2024-04-22T23:27:43.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.675+0000 7fae737fe700 1 --2- 172.21.15.67:0/2972457929 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fae6806e940 0x7fae68070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:43.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.675+0000 7fae737fe700 1 -- 172.21.15.67:0/2972457929 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fae6408fd80 con 0x7fae7c106cd0 2024-04-22T23:27:43.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.675+0000 7fae7a59c700 1 --2- 172.21.15.67:0/2972457929 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fae6806e940 0x7fae68070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:43.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.676+0000 7fae7a59c700 1 --2- 172.21.15.67:0/2972457929 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fae6806e940 0x7fae68070e00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fae74009550 tx=0x7fae740091b0 comp rx=0 tx=0).ready entity=mgr.14708 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:27:43.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.680+0000 7fae737fe700 1 -- 172.21.15.67:0/2972457929 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7fae6405cb40 con 0x7fae7c106cd0 2024-04-22T23:27:43.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:43 smithi067 ceph-mon[26453]: from='client.14866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:27:43.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:43 smithi067 ceph-mon[26453]: pgmap v84: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:27:43.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:43 smithi067 ceph-mon[26453]: from='client.14870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:27:43.794 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:43 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/2778597947' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:27:43.794 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:43 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:27:43.794 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:43 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T23:27:43.794 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:43 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T23:27:43.794 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:43 smithi067 ceph-mon[26453]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:27:43.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.941+0000 7fae8098f700 1 -- 172.21.15.67:0/2972457929 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fae7c04f2e0 con 0x7fae7c106cd0 2024-04-22T23:27:43.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.941+0000 7fae737fe700 1 -- 172.21.15.67:0/2972457929 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1848 (secure 0 0 0) 0x7fae6405c6d0 con 0x7fae7c106cd0 2024-04-22T23:27:43.941 INFO:teuthology.orchestra.run.smithi067.stdout:e22 2024-04-22T23:27:43.941 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:27:43.941 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:27:43.941 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:27:43.941 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:27:43.941 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:27:43.941 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:27:43.941 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 22 2024-04-22T23:27:43.941 INFO:teuthology.orchestra.run.smithi067.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T23:27:43.942 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:27:43.942 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:27:20.577198+0000 2024-04-22T23:27:43.942 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:27:43.942 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:27:43.942 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:27:43.942 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:27:43.942 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:27:43.942 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:27:43.942 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:27:43.942 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 46 2024-04-22T23:27:43.942 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:27:43.942 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:27:43.942 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:27:43.942 INFO:teuthology.orchestra.run.smithi067.stdout:up {0=14484,1=24273} 2024-04-22T23:27:43.942 INFO:teuthology.orchestra.run.smithi067.stdout:failed 2024-04-22T23:27:43.943 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:27:43.943 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:27:43.943 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:27:43.943 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:27:43.943 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:27:43.943 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:27:43.943 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:27:43.943 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:27:43.943 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{0:14484} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.67:6828/3606151860,v1:172.21.15.67:6829/3606151860] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:27:43.943 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{0:14496} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/3305304988,v1:172.21.15.67:6827/3305304988] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:27:43.943 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{1:24273} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.134:6824/535957142,v1:172.21.15.134:6825/535957142] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:27:43.943 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{1:24289} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.134:6826/1954821676,v1:172.21.15.134:6827/1954821676] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:27:43.943 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:27:43.943 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:27:43.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.945+0000 7fae8098f700 1 -- 172.21.15.67:0/2972457929 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fae6806e940 msgr2=0x7fae68070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:43.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.945+0000 7fae8098f700 1 --2- 172.21.15.67:0/2972457929 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fae6806e940 0x7fae68070e00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fae74009550 tx=0x7fae740091b0 comp rx=0 tx=0).stop 2024-04-22T23:27:43.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.945+0000 7fae8098f700 1 -- 172.21.15.67:0/2972457929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae7c106cd0 msgr2=0x7fae7c19cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:43.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.945+0000 7fae8098f700 1 --2- 172.21.15.67:0/2972457929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae7c106cd0 0x7fae7c19cb70 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fae64006010 tx=0x7fae6400bd70 comp rx=0 tx=0).stop 2024-04-22T23:27:43.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.945+0000 7fae8098f700 1 -- 172.21.15.67:0/2972457929 shutdown_connections 2024-04-22T23:27:43.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.945+0000 7fae8098f700 1 --2- 172.21.15.67:0/2972457929 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7fae6806e940 0x7fae68070e00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:43.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.945+0000 7fae8098f700 1 --2- 172.21.15.67:0/2972457929 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fae7c068580 0x7fae7c19c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:43.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.945+0000 7fae8098f700 1 --2- 172.21.15.67:0/2972457929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae7c106cd0 0x7fae7c19cb70 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:43.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.945+0000 7fae8098f700 1 -- 172.21.15.67:0/2972457929 >> 172.21.15.67:0/2972457929 conn(0x7fae7c075670 msgr2=0x7fae7c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:27:43.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.945+0000 7fae8098f700 1 -- 172.21.15.67:0/2972457929 shutdown_connections 2024-04-22T23:27:43.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:43.945+0000 7fae8098f700 1 -- 172.21.15.67:0/2972457929 wait complete. 2024-04-22T23:27:43.945 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 22 2024-04-22T23:27:43.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:43 smithi134 ceph-mon[34609]: from='client.14866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:27:43.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:43 smithi134 ceph-mon[34609]: pgmap v84: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:27:43.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:43 smithi134 ceph-mon[34609]: from='client.14870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:27:43.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:43 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/2778597947' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:27:43.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:43 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' 2024-04-22T23:27:43.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:43 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T23:27:43.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:43 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T23:27:43.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:43 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:27:44.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.085+0000 7f232d171700 1 -- 172.21.15.67:0/2404886768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23280ffc20 msgr2=0x7f2328100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:44.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.085+0000 7f232d171700 1 --2- 172.21.15.67:0/2404886768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23280ffc20 0x7f2328100080 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f231c009910 tx=0x7f231c009c20 comp rx=0 tx=0).stop 2024-04-22T23:27:44.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.085+0000 7f232d171700 1 -- 172.21.15.67:0/2404886768 shutdown_connections 2024-04-22T23:27:44.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.085+0000 7f232d171700 1 --2- 172.21.15.67:0/2404886768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23280ffc20 0x7f2328100080 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:44.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.085+0000 7f232d171700 1 --2- 172.21.15.67:0/2404886768 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2328105bd0 0x7f2328105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:44.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.085+0000 7f232d171700 1 -- 172.21.15.67:0/2404886768 >> 172.21.15.67:0/2404886768 conn(0x7f2328075670 msgr2=0x7f2328075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:27:44.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.085+0000 7f232d171700 1 -- 172.21.15.67:0/2404886768 shutdown_connections 2024-04-22T23:27:44.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.085+0000 7f232d171700 1 -- 172.21.15.67:0/2404886768 wait complete. 2024-04-22T23:27:44.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.087+0000 7f232d171700 1 Processor -- start 2024-04-22T23:27:44.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.087+0000 7f232d171700 1 -- start start 2024-04-22T23:27:44.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.087+0000 7f232d171700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23280ffc20 0x7f232819c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:44.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.087+0000 7f232d171700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2328105bd0 0x7f232819ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.087+0000 7f232d171700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f232819d4d0 con 0x7f23280ffc20 2024-04-22T23:27:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.087+0000 7f232d171700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2328196980 con 0x7f2328105bd0 2024-04-22T23:27:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.087+0000 7f2326d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23280ffc20 0x7f232819c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.087+0000 7f2326d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23280ffc20 0x7f232819c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58520/0 (socket says 172.21.15.67:58520) 2024-04-22T23:27:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.087+0000 7f2326d9d700 1 -- 172.21.15.67:0/1810612813 learned_addr learned my addr 172.21.15.67:0/1810612813 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:27:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.088+0000 7f232659c700 1 --2- 172.21.15.67:0/1810612813 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2328105bd0 0x7f232819ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.088+0000 7f2326d9d700 1 -- 172.21.15.67:0/1810612813 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2328105bd0 msgr2=0x7f232819ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.088+0000 7f2326d9d700 1 --2- 172.21.15.67:0/1810612813 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2328105bd0 0x7f232819ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.088+0000 7f2326d9d700 1 -- 172.21.15.67:0/1810612813 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f231c0095f0 con 0x7f23280ffc20 2024-04-22T23:27:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.088+0000 7f2326d9d700 1 --2- 172.21.15.67:0/1810612813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23280ffc20 0x7f232819c900 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f231800b4b0 tx=0x7f231800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:27:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.088+0000 7f230ffff700 1 -- 172.21.15.67:0/1810612813 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2318011800 con 0x7f23280ffc20 2024-04-22T23:27:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.088+0000 7f232d171700 1 -- 172.21.15.67:0/1810612813 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2328196c60 con 0x7f23280ffc20 2024-04-22T23:27:44.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.088+0000 7f230ffff700 1 -- 172.21.15.67:0/1810612813 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2318011e20 con 0x7f23280ffc20 2024-04-22T23:27:44.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.088+0000 7f230ffff700 1 -- 172.21.15.67:0/1810612813 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f231800f2e0 con 0x7f23280ffc20 2024-04-22T23:27:44.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.088+0000 7f232d171700 1 -- 172.21.15.67:0/1810612813 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23281971b0 con 0x7f23280ffc20 2024-04-22T23:27:44.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.090+0000 7f232d171700 1 -- 172.21.15.67:0/1810612813 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2328103f20 con 0x7f23280ffc20 2024-04-22T23:27:44.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.090+0000 7f230ffff700 1 -- 172.21.15.67:0/1810612813 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 33) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f2318010380 con 0x7f23280ffc20 2024-04-22T23:27:44.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.091+0000 7f230ffff700 1 --2- 172.21.15.67:0/1810612813 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f231006e940 0x7f2310070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:44.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.091+0000 7f230ffff700 1 -- 172.21.15.67:0/1810612813 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f231808e560 con 0x7f23280ffc20 2024-04-22T23:27:44.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.091+0000 7f232659c700 1 --2- 172.21.15.67:0/1810612813 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f231006e940 0x7f2310070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:44.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.092+0000 7f232659c700 1 --2- 172.21.15.67:0/1810612813 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f231006e940 0x7f2310070e00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f231c00b5c0 tx=0x7f231c005960 comp rx=0 tx=0).ready entity=mgr.14708 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:27:44.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.096+0000 7f230ffff700 1 -- 172.21.15.67:0/1810612813 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f231805b280 con 0x7f23280ffc20 2024-04-22T23:27:44.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.341+0000 7f232d171700 1 -- 172.21.15.67:0/1810612813 --> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2328197e90 con 0x7f231006e940 2024-04-22T23:27:44.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.343+0000 7f230ffff700 1 -- 172.21.15.67:0/1810612813 <== mgr.14708 v2:172.21.15.67:6800/2306276228 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f2328197e90 con 0x7f231006e940 2024-04-22T23:27:44.342 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:27:44.343 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:27:44.343 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:27:44.343 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:27:44.343 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:27:44.343 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr" 2024-04-22T23:27:44.343 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:27:44.343 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "2/23 daemons upgraded", 2024-04-22T23:27:44.343 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mon daemons", 2024-04-22T23:27:44.343 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:27:44.343 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:27:44.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.345+0000 7f232d171700 1 -- 172.21.15.67:0/1810612813 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f231006e940 msgr2=0x7f2310070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:44.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.345+0000 7f232d171700 1 --2- 172.21.15.67:0/1810612813 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f231006e940 0x7f2310070e00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f231c00b5c0 tx=0x7f231c005960 comp rx=0 tx=0).stop 2024-04-22T23:27:44.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.345+0000 7f232d171700 1 -- 172.21.15.67:0/1810612813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23280ffc20 msgr2=0x7f232819c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:44.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.345+0000 7f232d171700 1 --2- 172.21.15.67:0/1810612813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23280ffc20 0x7f232819c900 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f231800b4b0 tx=0x7f231800b870 comp rx=0 tx=0).stop 2024-04-22T23:27:44.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.345+0000 7f232d171700 1 -- 172.21.15.67:0/1810612813 shutdown_connections 2024-04-22T23:27:44.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.345+0000 7f232d171700 1 --2- 172.21.15.67:0/1810612813 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f231006e940 0x7f2310070e00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:44.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.345+0000 7f232d171700 1 --2- 172.21.15.67:0/1810612813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23280ffc20 0x7f232819c900 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:44.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.345+0000 7f232d171700 1 --2- 172.21.15.67:0/1810612813 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2328105bd0 0x7f232819ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:44.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.345+0000 7f232d171700 1 -- 172.21.15.67:0/1810612813 >> 172.21.15.67:0/1810612813 conn(0x7f2328075670 msgr2=0x7f232810a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:27:44.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.345+0000 7f232d171700 1 -- 172.21.15.67:0/1810612813 shutdown_connections 2024-04-22T23:27:44.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.345+0000 7f232d171700 1 -- 172.21.15.67:0/1810612813 wait complete. 2024-04-22T23:27:44.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.459+0000 7f1877d1e700 1 -- 172.21.15.67:0/3694833878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1870105bd0 msgr2=0x7f1870105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:44.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.459+0000 7f1877d1e700 1 --2- 172.21.15.67:0/3694833878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1870105bd0 0x7f1870105fb0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f186c009910 tx=0x7f186c009c20 comp rx=0 tx=0).stop 2024-04-22T23:27:44.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.459+0000 7f1877d1e700 1 -- 172.21.15.67:0/3694833878 shutdown_connections 2024-04-22T23:27:44.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.459+0000 7f1877d1e700 1 --2- 172.21.15.67:0/3694833878 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f18700ffc20 0x7f1870100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:44.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.459+0000 7f1877d1e700 1 --2- 172.21.15.67:0/3694833878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1870105bd0 0x7f1870105fb0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:44.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.459+0000 7f1877d1e700 1 -- 172.21.15.67:0/3694833878 >> 172.21.15.67:0/3694833878 conn(0x7f1870075670 msgr2=0x7f1870075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:27:44.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.460+0000 7f1877d1e700 1 -- 172.21.15.67:0/3694833878 shutdown_connections 2024-04-22T23:27:44.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.460+0000 7f1877d1e700 1 -- 172.21.15.67:0/3694833878 wait complete. 2024-04-22T23:27:44.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.460+0000 7f1877d1e700 1 Processor -- start 2024-04-22T23:27:44.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.460+0000 7f1877d1e700 1 -- start start 2024-04-22T23:27:44.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.460+0000 7f1877d1e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18700ffc20 0x7f1870197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:44.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.460+0000 7f1877d1e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1870105bd0 0x7f1870197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:44.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.460+0000 7f1877d1e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1870198290 con 0x7f18700ffc20 2024-04-22T23:27:44.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.460+0000 7f1877d1e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18701983d0 con 0x7f1870105bd0 2024-04-22T23:27:44.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.460+0000 7f1875aba700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18700ffc20 0x7f1870197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:44.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.460+0000 7f1875aba700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18700ffc20 0x7f1870197670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58532/0 (socket says 172.21.15.67:58532) 2024-04-22T23:27:44.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.460+0000 7f1875aba700 1 -- 172.21.15.67:0/3862188168 learned_addr learned my addr 172.21.15.67:0/3862188168 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:27:44.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.461+0000 7f1875aba700 1 -- 172.21.15.67:0/3862188168 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1870105bd0 msgr2=0x7f1870197bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:44.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.461+0000 7f1875aba700 1 --2- 172.21.15.67:0/3862188168 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1870105bd0 0x7f1870197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:44.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.461+0000 7f1875aba700 1 -- 172.21.15.67:0/3862188168 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f186c0095f0 con 0x7f18700ffc20 2024-04-22T23:27:44.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.461+0000 7f1875aba700 1 --2- 172.21.15.67:0/3862188168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18700ffc20 0x7f1870197670 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f186c006010 tx=0x7f186c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:27:44.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.461+0000 7f1866ffd700 1 -- 172.21.15.67:0/3862188168 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f186c01d070 con 0x7f18700ffc20 2024-04-22T23:27:44.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.461+0000 7f1866ffd700 1 -- 172.21.15.67:0/3862188168 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f186c004fb0 con 0x7f18700ffc20 2024-04-22T23:27:44.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.461+0000 7f1877d1e700 1 -- 172.21.15.67:0/3862188168 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18701a06e0 con 0x7f18700ffc20 2024-04-22T23:27:44.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.461+0000 7f1866ffd700 1 -- 172.21.15.67:0/3862188168 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f186c022ea0 con 0x7f18700ffc20 2024-04-22T23:27:44.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.461+0000 7f1877d1e700 1 -- 172.21.15.67:0/3862188168 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18701a0ba0 con 0x7f18700ffc20 2024-04-22T23:27:44.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.462+0000 7f1877d1e700 1 -- 172.21.15.67:0/3862188168 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18701080f0 con 0x7f18700ffc20 2024-04-22T23:27:44.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.463+0000 7f1866ffd700 1 -- 172.21.15.67:0/3862188168 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 33) v1 ==== 92598+0+0 (secure 0 0 0) 0x7f186c022470 con 0x7f18700ffc20 2024-04-22T23:27:44.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.463+0000 7f1866ffd700 1 --2- 172.21.15.67:0/3862188168 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f185c06e940 0x7f185c070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:27:44.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.463+0000 7f1866ffd700 1 -- 172.21.15.67:0/3862188168 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f186c08fde0 con 0x7f18700ffc20 2024-04-22T23:27:44.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.463+0000 7f18752b9700 1 --2- 172.21.15.67:0/3862188168 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f185c06e940 0x7f185c070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:27:44.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.464+0000 7f18752b9700 1 --2- 172.21.15.67:0/3862188168 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f185c06e940 0x7f185c070e00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1870069210 tx=0x7f1860005c80 comp rx=0 tx=0).ready entity=mgr.14708 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:27:44.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.468+0000 7f1866ffd700 1 -- 172.21.15.67:0/3862188168 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+187909 (secure 0 0 0) 0x7f186c0591a0 con 0x7f18700ffc20 2024-04-22T23:27:44.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.772+0000 7f1877d1e700 1 -- 172.21.15.67:0/3862188168 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f1870107ad0 con 0x7f18700ffc20 2024-04-22T23:27:44.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.773+0000 7f1866ffd700 1 -- 172.21.15.67:0/3862188168 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f186c022720 con 0x7f18700ffc20 2024-04-22T23:27:44.773 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T23:27:44.773 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:27:44.773 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:27:44.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.775+0000 7f1877d1e700 1 -- 172.21.15.67:0/3862188168 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f185c06e940 msgr2=0x7f185c070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:44.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.775+0000 7f1877d1e700 1 --2- 172.21.15.67:0/3862188168 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f185c06e940 0x7f185c070e00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1870069210 tx=0x7f1860005c80 comp rx=0 tx=0).stop 2024-04-22T23:27:44.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.775+0000 7f1877d1e700 1 -- 172.21.15.67:0/3862188168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18700ffc20 msgr2=0x7f1870197670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:27:44.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.775+0000 7f1877d1e700 1 --2- 172.21.15.67:0/3862188168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18700ffc20 0x7f1870197670 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f186c006010 tx=0x7f186c00be20 comp rx=0 tx=0).stop 2024-04-22T23:27:44.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.775+0000 7f1877d1e700 1 -- 172.21.15.67:0/3862188168 shutdown_connections 2024-04-22T23:27:44.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.775+0000 7f1877d1e700 1 --2- 172.21.15.67:0/3862188168 >> [v2:172.21.15.67:6800/2306276228,v1:172.21.15.67:6801/2306276228] conn(0x7f185c06e940 0x7f185c070e00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:44.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.775+0000 7f1877d1e700 1 --2- 172.21.15.67:0/3862188168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18700ffc20 0x7f1870197670 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:44.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.775+0000 7f1877d1e700 1 --2- 172.21.15.67:0/3862188168 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1870105bd0 0x7f1870197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:27:44.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.775+0000 7f1877d1e700 1 -- 172.21.15.67:0/3862188168 >> 172.21.15.67:0/3862188168 conn(0x7f1870075670 msgr2=0x7f18700feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:27:44.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.775+0000 7f1877d1e700 1 -- 172.21.15.67:0/3862188168 shutdown_connections 2024-04-22T23:27:44.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:27:44.775+0000 7f1877d1e700 1 -- 172.21.15.67:0/3862188168 wait complete. 2024-04-22T23:27:44.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:44 smithi134 ceph-mon[34609]: from='client.14874 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:27:44.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:44 smithi134 ceph-mon[34609]: Upgrade: Updating mon.smithi067 2024-04-22T23:27:44.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:44 smithi134 ceph-mon[34609]: Deploying daemon mon.smithi067 on smithi067 2024-04-22T23:27:44.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:44 smithi134 ceph-mon[34609]: from='client.? 172.21.15.67:0/2972457929' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:27:45.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:44 smithi067 ceph-mon[26453]: from='client.14874 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:27:45.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:44 smithi067 ceph-mon[26453]: Upgrade: Updating mon.smithi067 2024-04-22T23:27:45.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:44 smithi067 ceph-mon[26453]: Deploying daemon mon.smithi067 on smithi067 2024-04-22T23:27:45.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:44 smithi067 ceph-mon[26453]: from='client.? 172.21.15.67:0/2972457929' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:27:45.732 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:45 smithi067 systemd[1]: Stopping Ceph mon.smithi067 for d46c346e-00fd-11ef-bc93-c7b262605968... 2024-04-22T23:27:46.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:45 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[26449]: 2024-04-22T23:27:45.732+0000 7f2197cce700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi067 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T23:27:46.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:45 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[26449]: 2024-04-22T23:27:45.732+0000 7f2197cce700 -1 mon.smithi067@0(leader) e2 *** Got Signal Terminated *** 2024-04-22T23:27:46.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:45 smithi067 podman[112263]: 2024-04-22 23:27:45.916355283 +0000 UTC m=+0.309573932 container died 6789283703e0028c6f447f8504400ced49d85f3b57dea8cf1a25a7e126b8b58e (image=quay.io/ceph/ceph:v18.2.1, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-22T23:27:46.542 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:46 smithi067 podman[112263]: 2024-04-22 23:27:46.274615846 +0000 UTC m=+0.667834505 container cleanup 6789283703e0028c6f447f8504400ced49d85f3b57dea8cf1a25a7e126b8b58e (image=quay.io/ceph/ceph:v18.2.1, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_CLEAN=True) 2024-04-22T23:27:46.542 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:46 smithi067 bash[112263]: ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067 2024-04-22T23:27:46.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:46 smithi067 podman[112288]: 2024-04-22 23:27:46.542699518 +0000 UTC m=+0.622173672 container remove 6789283703e0028c6f447f8504400ced49d85f3b57dea8cf1a25a7e126b8b58e (image=quay.io/ceph/ceph:v18.2.1, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T23:27:46.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:46 smithi067 systemd[1]: ceph-d46c346e-00fd-11ef-bc93-c7b262605968@mon.smithi067.service: Deactivated successfully. 2024-04-22T23:27:46.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:46 smithi067 systemd[1]: Stopped Ceph mon.smithi067 for d46c346e-00fd-11ef-bc93-c7b262605968. 2024-04-22T23:27:46.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:46 smithi067 systemd[1]: ceph-d46c346e-00fd-11ef-bc93-c7b262605968@mon.smithi067.service: Consumed 15.443s CPU time. 2024-04-22T23:27:47.200 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:46 smithi067 systemd[1]: Starting Ceph mon.smithi067 for d46c346e-00fd-11ef-bc93-c7b262605968... 2024-04-22T23:27:47.200 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 podman[112419]: 2024-04-22 23:27:47.005156544 +0000 UTC m=+0.019166324 image pull a0434699c1d76af4eea1fc95ff0a22ea73d1e04e609464fbf658d6f39bc7f2fc quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c 2024-04-22T23:27:47.200 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 podman[112419]: 2024-04-22 23:27:47.118785196 +0000 UTC m=+0.132795008 container create b879412ee3f1581858c53ba16eaa5c05ccd18c2352d65fa5cfd8fe8a49196ebc (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T23:27:47.544 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 podman[112419]: 2024-04-22 23:27:47.287763854 +0000 UTC m=+0.301773607 container init b879412ee3f1581858c53ba16eaa5c05ccd18c2352d65fa5cfd8fe8a49196ebc (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T23:27:47.544 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 podman[112419]: 2024-04-22 23:27:47.291037786 +0000 UTC m=+0.305047547 container start b879412ee3f1581858c53ba16eaa5c05ccd18c2352d65fa5cfd8fe8a49196ebc (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-22-1618-43be020) 2024-04-22T23:27:47.544 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: set uid:gid to 167:167 (ceph:ceph) 2024-04-22T23:27:47.544 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev), process ceph-mon, pid 2 2024-04-22T23:27:47.544 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: pidfile_write: ignore empty --pid-file 2024-04-22T23:27:47.544 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: load: jerasure load: lrc 2024-04-22T23:27:47.544 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: RocksDB version: 7.9.2 2024-04-22T23:27:47.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Git sha 0 2024-04-22T23:27:47.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Compile date 2024-04-22 21:33:31 2024-04-22T23:27:47.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: DB SUMMARY 2024-04-22T23:27:47.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: DB Session ID: ITBB9BY8N0D9FG5CQ820 2024-04-22T23:27:47.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: CURRENT file: CURRENT 2024-04-22T23:27:47.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: IDENTITY file: IDENTITY 2024-04-22T23:27:47.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: MANIFEST file: MANIFEST-000015 size: 1509 Bytes 2024-04-22T23:27:47.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi067/store.db dir, Total Num: 1, files: 000032.sst 2024-04-22T23:27:47.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi067/store.db: 000030.log size: 5198070 ; 2024-04-22T23:27:47.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.error_if_exists: 0 2024-04-22T23:27:47.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.create_if_missing: 0 2024-04-22T23:27:47.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.paranoid_checks: 1 2024-04-22T23:27:47.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-22T23:27:47.545 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-22T23:27:47.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-22T23:27:47.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.env: 0x56485069dcc0 2024-04-22T23:27:47.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.fs: PosixFileSystem 2024-04-22T23:27:47.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.info_log: 0x564851fcbaa0 2024-04-22T23:27:47.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.max_file_opening_threads: 16 2024-04-22T23:27:47.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.statistics: (nil) 2024-04-22T23:27:47.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.use_fsync: 0 2024-04-22T23:27:47.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.max_log_file_size: 0 2024-04-22T23:27:47.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-22T23:27:47.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-22T23:27:47.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.keep_log_file_num: 1000 2024-04-22T23:27:47.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.recycle_log_file_num: 0 2024-04-22T23:27:47.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.allow_fallocate: 1 2024-04-22T23:27:47.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.allow_mmap_reads: 0 2024-04-22T23:27:47.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.allow_mmap_writes: 0 2024-04-22T23:27:47.546 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.use_direct_reads: 0 2024-04-22T23:27:47.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-22T23:27:47.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.create_missing_column_families: 0 2024-04-22T23:27:47.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.db_log_dir: 2024-04-22T23:27:47.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.wal_dir: 2024-04-22T23:27:47.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-22T23:27:47.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-22T23:27:47.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-22T23:27:47.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-22T23:27:47.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-22T23:27:47.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-22T23:27:47.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.advise_random_on_open: 1 2024-04-22T23:27:47.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.db_write_buffer_size: 0 2024-04-22T23:27:47.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.write_buffer_manager: 0x564851f5f0e0 2024-04-22T23:27:47.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-22T23:27:47.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-22T23:27:47.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-22T23:27:47.547 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.rate_limiter: (nil) 2024-04-22T23:27:47.548 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-22T23:27:47.548 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.wal_recovery_mode: 2 2024-04-22T23:27:47.548 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.enable_thread_tracking: 0 2024-04-22T23:27:47.548 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.enable_pipelined_write: 0 2024-04-22T23:27:47.548 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.unordered_write: 0 2024-04-22T23:27:47.548 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-22T23:27:47.548 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-22T23:27:47.548 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-22T23:27:47.548 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-22T23:27:47.548 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.row_cache: None 2024-04-22T23:27:47.549 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.wal_filter: None 2024-04-22T23:27:47.549 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-22T23:27:47.550 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.allow_ingest_behind: 0 2024-04-22T23:27:47.550 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.two_write_queues: 0 2024-04-22T23:27:47.550 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.manual_wal_flush: 0 2024-04-22T23:27:47.550 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.wal_compression: 0 2024-04-22T23:27:47.550 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.atomic_flush: 0 2024-04-22T23:27:47.550 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-22T23:27:47.550 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-22T23:27:47.550 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-22T23:27:47.550 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.log_readahead_size: 0 2024-04-22T23:27:47.550 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-22T23:27:47.550 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.best_efforts_recovery: 0 2024-04-22T23:27:47.550 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-22T23:27:47.550 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-22T23:27:47.550 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.allow_data_in_errors: 0 2024-04-22T23:27:47.551 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.db_host_id: __hostname__ 2024-04-22T23:27:47.551 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-22T23:27:47.551 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.max_background_jobs: 2 2024-04-22T23:27:47.551 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.max_background_compactions: -1 2024-04-22T23:27:47.551 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.max_subcompactions: 1 2024-04-22T23:27:47.551 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-22T23:27:47.551 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-22T23:27:47.551 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-22T23:27:47.551 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.max_total_wal_size: 0 2024-04-22T23:27:47.551 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-22T23:27:47.551 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-22T23:27:47.551 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-22T23:27:47.551 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-22T23:27:47.551 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.max_open_files: -1 2024-04-22T23:27:47.551 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.bytes_per_sync: 0 2024-04-22T23:27:47.551 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-22T23:27:47.551 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-22T23:27:47.552 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.compaction_readahead_size: 0 2024-04-22T23:27:47.552 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.max_background_flushes: -1 2024-04-22T23:27:47.552 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Compression algorithms supported: 2024-04-22T23:27:47.552 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-22T23:27:47.552 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: kZSTD supported: 0 2024-04-22T23:27:47.552 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: kXpressCompression supported: 0 2024-04-22T23:27:47.552 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: kLZ4HCCompression supported: 1 2024-04-22T23:27:47.552 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: kZlibCompression supported: 1 2024-04-22T23:27:47.552 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: kSnappyCompression supported: 1 2024-04-22T23:27:47.552 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: kLZ4Compression supported: 1 2024-04-22T23:27:47.552 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: kBZip2Compression supported: 0 2024-04-22T23:27:47.552 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-22T23:27:47.552 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-22T23:27:47.552 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi067/store.db/MANIFEST-000015 2024-04-22T23:27:47.552 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-22T23:27:47.552 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-22T23:27:47.553 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.merge_operator: 2024-04-22T23:27:47.553 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.compaction_filter: None 2024-04-22T23:27:47.553 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.compaction_filter_factory: None 2024-04-22T23:27:47.553 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.sst_partitioner_factory: None 2024-04-22T23:27:47.553 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-22T23:27:47.553 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-22T23:27:47.553 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564851fcbc80) 2024-04-22T23:27:47.553 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: cache_index_and_filter_blocks: 1 2024-04-22T23:27:47.553 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-22T23:27:47.553 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-22T23:27:47.553 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: pin_top_level_index_and_filter: 1 2024-04-22T23:27:47.553 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: index_type: 0 2024-04-22T23:27:47.553 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: data_block_index_type: 0 2024-04-22T23:27:47.553 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: index_shortening: 1 2024-04-22T23:27:47.554 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-22T23:27:47.554 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: checksum: 4 2024-04-22T23:27:47.554 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: no_block_cache: 0 2024-04-22T23:27:47.554 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: block_cache: 0x564851f911f0 2024-04-22T23:27:47.554 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: block_cache_name: BinnedLRUCache 2024-04-22T23:27:47.554 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: block_cache_options: 2024-04-22T23:27:47.554 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: capacity : 536870912 2024-04-22T23:27:47.554 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: num_shard_bits : 4 2024-04-22T23:27:47.554 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: strict_capacity_limit : 0 2024-04-22T23:27:47.554 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: high_pri_pool_ratio: 0.000 2024-04-22T23:27:47.554 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: block_cache_compressed: (nil) 2024-04-22T23:27:47.554 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: persistent_cache: (nil) 2024-04-22T23:27:47.554 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: block_size: 4096 2024-04-22T23:27:47.555 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: block_size_deviation: 10 2024-04-22T23:27:47.555 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: block_restart_interval: 16 2024-04-22T23:27:47.555 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: index_block_restart_interval: 1 2024-04-22T23:27:47.555 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: metadata_block_size: 4096 2024-04-22T23:27:47.555 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: partition_filters: 0 2024-04-22T23:27:47.555 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: use_delta_encoding: 1 2024-04-22T23:27:47.555 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: filter_policy: bloomfilter 2024-04-22T23:27:47.555 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: whole_key_filtering: 1 2024-04-22T23:27:47.555 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: verify_compression: 0 2024-04-22T23:27:47.555 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: read_amp_bytes_per_bit: 0 2024-04-22T23:27:47.555 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: format_version: 5 2024-04-22T23:27:47.555 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: enable_index_compression: 1 2024-04-22T23:27:47.555 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: block_align: 0 2024-04-22T23:27:47.555 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: max_auto_readahead_size: 262144 2024-04-22T23:27:47.556 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: prepopulate_block_cache: 0 2024-04-22T23:27:47.556 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: initial_auto_readahead_size: 8192 2024-04-22T23:27:47.556 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: num_file_reads_for_auto_readahead: 2 2024-04-22T23:27:47.556 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.write_buffer_size: 33554432 2024-04-22T23:27:47.556 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.max_write_buffer_number: 2 2024-04-22T23:27:47.556 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.compression: NoCompression 2024-04-22T23:27:47.556 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.bottommost_compression: Disabled 2024-04-22T23:27:47.556 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.prefix_extractor: nullptr 2024-04-22T23:27:47.556 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-22T23:27:47.556 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.num_levels: 7 2024-04-22T23:27:47.556 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-22T23:27:47.556 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-22T23:27:47.556 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-22T23:27:47.556 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-22T23:27:47.557 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-22T23:27:47.557 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-22T23:27:47.557 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-22T23:27:47.557 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-22T23:27:47.557 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-22T23:27:47.557 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-22T23:27:47.557 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-22T23:27:47.557 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-22T23:27:47.557 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-22T23:27:47.557 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.compression_opts.level: 32767 2024-04-22T23:27:47.557 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.compression_opts.strategy: 0 2024-04-22T23:27:47.557 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-22T23:27:47.557 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-22T23:27:47.557 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-22T23:27:47.557 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-22T23:27:47.557 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.compression_opts.enabled: false 2024-04-22T23:27:47.558 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-22T23:27:47.558 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-22T23:27:47.558 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-22T23:27:47.558 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-22T23:27:47.558 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.target_file_size_base: 67108864 2024-04-22T23:27:47.558 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-22T23:27:47.558 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-22T23:27:47.558 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-22T23:27:47.558 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-22T23:27:47.558 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-22T23:27:47.558 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-22T23:27:47.558 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-22T23:27:47.558 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-22T23:27:47.558 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-22T23:27:47.558 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-22T23:27:47.559 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-22T23:27:47.559 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-22T23:27:47.559 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-22T23:27:47.559 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-22T23:27:47.559 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.arena_block_size: 1048576 2024-04-22T23:27:47.559 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-22T23:27:47.559 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-22T23:27:47.559 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.disable_auto_compactions: 0 2024-04-22T23:27:47.559 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-22T23:27:47.559 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-22T23:27:47.559 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-22T23:27:47.559 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-22T23:27:47.559 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-22T23:27:47.560 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-22T23:27:47.560 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-22T23:27:47.560 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-22T23:27:47.560 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-22T23:27:47.560 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-22T23:27:47.560 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-22T23:27:47.560 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.inplace_update_support: 0 2024-04-22T23:27:47.560 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-22T23:27:47.560 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-22T23:27:47.560 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-22T23:27:47.560 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-22T23:27:47.560 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.bloom_locality: 0 2024-04-22T23:27:47.560 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.max_successive_merges: 0 2024-04-22T23:27:47.560 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-22T23:27:47.560 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.paranoid_file_checks: 0 2024-04-22T23:27:47.560 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.force_consistency_checks: 1 2024-04-22T23:27:47.560 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.report_bg_io_stats: 0 2024-04-22T23:27:47.560 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.ttl: 2592000 2024-04-22T23:27:47.561 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-22T23:27:47.561 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-22T23:27:47.561 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-22T23:27:47.561 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.enable_blob_files: false 2024-04-22T23:27:47.561 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.min_blob_size: 0 2024-04-22T23:27:47.561 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.blob_file_size: 268435456 2024-04-22T23:27:47.561 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-22T23:27:47.561 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-22T23:27:47.561 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-22T23:27:47.561 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-22T23:27:47.561 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-22T23:27:47.561 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.blob_file_starting_level: 0 2024-04-22T23:27:47.561 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-22T23:27:47.561 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-22T23:27:47.561 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi067/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 34, last_sequence is 12094, log_number is 30,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 30 2024-04-22T23:27:47.561 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 30 2024-04-22T23:27:47.562 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 3f61ccfa-96d2-469e-9f4c-f1125e9e9400 2024-04-22T23:27:47.562 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713828467321376, "job": 1, "event": "recovery_started", "wal_files": [30]} 2024-04-22T23:27:47.562 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #30 mode 2 2024-04-22T23:27:47.562 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713828467337765, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 35, "file_size": 4594320, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12095, "largest_seqno": 15436, "table_properties": {"data_size": 4581193, "index_size": 8304, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 3781, "raw_key_size": 33232, "raw_average_key_size": 22, "raw_value_size": 4551420, "raw_average_value_size": 3020, "num_data_blocks": 375, "num_entries": 1507, "num_filter_entries": 1507, "num_deletions": 259, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713828467, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "3f61ccfa-96d2-469e-9f4c-f1125e9e9400", "db_session_id": "ITBB9BY8N0D9FG5CQ820", "orig_file_number": 35, "seqno_to_time_mapping": "N/A"}} 2024-04-22T23:27:47.562 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713828467337890, "job": 1, "event": "recovery_finished"} 2024-04-22T23:27:47.562 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: [db/version_set.cc:5047] Creating manifest 37 2024-04-22T23:27:47.562 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-22T23:27:47.562 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi067/store.db/000030.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-22T23:27:47.562 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5648520a6000 2024-04-22T23:27:47.562 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: DB pointer 0x564852090000 2024-04-22T23:27:47.562 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-22T23:27:47.562 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-22T23:27:47.562 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: ** DB Stats ** 2024-04-22T23:27:47.562 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T23:27:47.562 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-22T23:27:47.562 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T23:27:47.563 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T23:27:47.563 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-22T23:27:47.563 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T23:27:47.563 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T23:27:47.563 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: 2024-04-22T23:27:47.563 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: ** Compaction Stats [default] ** 2024-04-22T23:27:47.563 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T23:27:47.563 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-22T23:27:47.563 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: L0 1/0 4.38 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 534.6 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-04-22T23:27:47.563 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: L6 1/0 4.93 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-22T23:27:47.563 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Sum 2/0 9.32 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 534.6 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-04-22T23:27:47.563 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 534.6 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-04-22T23:27:47.563 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: 2024-04-22T23:27:47.563 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: ** Compaction Stats [default] ** 2024-04-22T23:27:47.563 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T23:27:47.563 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T23:27:47.564 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 534.6 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-04-22T23:27:47.564 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: 2024-04-22T23:27:47.564 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-22T23:27:47.564 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: 2024-04-22T23:27:47.564 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T23:27:47.564 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Flush(GB): cumulative 0.004, interval 0.004 2024-04-22T23:27:47.564 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T23:27:47.564 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T23:27:47.564 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T23:27:47.564 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T23:27:47.564 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Cumulative compaction: 0.00 GB write, 229.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T23:27:47.564 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Interval compaction: 0.00 GB write, 229.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T23:27:47.564 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T23:27:47.564 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Block cache BinnedLRUCache@0x564851f911f0#2 capacity: 512.00 MB usage: 13.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-04-22T23:27:47.564 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,4.03 KB,0.0007689%) IndexBlock(1,9.08 KB,0.00173151%) Misc(1,0.00 KB,0%) 2024-04-22T23:27:47.564 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: 2024-04-22T23:27:47.564 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T23:27:47.565 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: starting mon.smithi067 rank 0 at public addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] at bind addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi067 fsid d46c346e-00fd-11ef-bc93-c7b262605968 2024-04-22T23:27:47.565 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: mon.smithi067@-1(???) e2 preinit fsid d46c346e-00fd-11ef-bc93-c7b262605968 2024-04-22T23:27:47.565 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: mon.smithi067@-1(???).mds e22 new map 2024-04-22T23:27:47.565 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: mon.smithi067@-1(???).mds e22 print_map 2024-04-22T23:27:47.565 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: e22 2024-04-22T23:27:47.565 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:27:47.565 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:27:47.565 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: legacy client fscid: 1 2024-04-22T23:27:47.565 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: 2024-04-22T23:27:47.565 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: Filesystem 'cephfs' (1) 2024-04-22T23:27:47.565 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: fs_name cephfs 2024-04-22T23:27:47.565 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: epoch 22 2024-04-22T23:27:47.565 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T23:27:47.565 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:27:47.565 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: modified 2024-04-22T23:27:20.577198+0000 2024-04-22T23:27:47.566 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: tableserver 0 2024-04-22T23:27:47.566 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: root 0 2024-04-22T23:27:47.566 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: session_timeout 60 2024-04-22T23:27:47.566 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: session_autoclose 300 2024-04-22T23:27:47.566 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: max_file_size 1099511627776 2024-04-22T23:27:47.566 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: max_xattr_size 65536 2024-04-22T23:27:47.566 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: required_client_features {} 2024-04-22T23:27:47.566 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: last_failure 0 2024-04-22T23:27:47.566 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: last_failure_osd_epoch 46 2024-04-22T23:27:47.566 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:27:47.566 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: max_mds 2 2024-04-22T23:27:47.566 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: in 0,1 2024-04-22T23:27:47.566 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: up {0=14484,1=24273} 2024-04-22T23:27:47.566 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: failed 2024-04-22T23:27:47.567 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: damaged 2024-04-22T23:27:47.567 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: stopped 2024-04-22T23:27:47.567 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: data_pools [3] 2024-04-22T23:27:47.567 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: metadata_pool 2 2024-04-22T23:27:47.567 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: inline_data enabled 2024-04-22T23:27:47.567 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: balancer 2024-04-22T23:27:47.567 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: bal_rank_mask -1 2024-04-22T23:27:47.567 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: standby_count_wanted 1 2024-04-22T23:27:47.567 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: qdb_cluster leader: 0 members: 2024-04-22T23:27:47.567 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: [mds.cephfs.smithi067.xyazmh{0:14484} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.67:6828/3606151860,v1:172.21.15.67:6829/3606151860] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:27:47.567 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: [mds.cephfs.smithi067.aqlxpv{0:14496} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/3305304988,v1:172.21.15.67:6827/3305304988] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:27:47.567 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: [mds.cephfs.smithi134.owipad{1:24273} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.134:6824/535957142,v1:172.21.15.134:6825/535957142] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:27:47.567 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: [mds.cephfs.smithi134.vuxbsj{1:24289} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.134:6826/1954821676,v1:172.21.15.134:6827/1954821676] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:27:47.567 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: 2024-04-22T23:27:47.567 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout: 2024-04-22T23:27:47.567 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: mon.smithi067@-1(???).osd e48 crush map has features 3314933000852226048, adjusting msgr requires 2024-04-22T23:27:47.567 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: mon.smithi067@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-04-22T23:27:47.568 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: mon.smithi067@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-04-22T23:27:47.568 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: mon.smithi067@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-04-22T23:27:47.568 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 ceph-mon[112438]: mon.smithi067@-1(???).paxosservice(auth 1..23) refresh upgraded, format 0 -> 3 2024-04-22T23:27:47.568 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 bash[112419]: b879412ee3f1581858c53ba16eaa5c05ccd18c2352d65fa5cfd8fe8a49196ebc 2024-04-22T23:27:47.568 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:47 smithi067 systemd[1]: Started Ceph mon.smithi067 for d46c346e-00fd-11ef-bc93-c7b262605968. 2024-04-22T23:27:49.139 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:48 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-22T23:27:49.139 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:48 smithi134 ceph-mon[34609]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-04-22T23:27:49.139 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:48 smithi134 ceph-mon[34609]: pgmap v86: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:27:49.139 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:48 smithi134 ceph-mon[34609]: mon.smithi067 calling monitor election 2024-04-22T23:27:49.140 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:48 smithi134 ceph-mon[34609]: mon.smithi067 is new leader, mons smithi067,smithi134 in quorum (ranks 0,1) 2024-04-22T23:27:49.140 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:48 smithi134 ceph-mon[34609]: monmap epoch 2 2024-04-22T23:27:49.140 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:48 smithi134 ceph-mon[34609]: fsid d46c346e-00fd-11ef-bc93-c7b262605968 2024-04-22T23:27:49.140 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:48 smithi134 ceph-mon[34609]: last_changed 2024-04-22T23:16:04.430584+0000 2024-04-22T23:27:49.140 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:48 smithi134 ceph-mon[34609]: created 2024-04-22T23:13:15.759242+0000 2024-04-22T23:27:49.140 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:48 smithi134 ceph-mon[34609]: min_mon_release 18 (reef) 2024-04-22T23:27:49.140 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:48 smithi134 ceph-mon[34609]: election_strategy: 1 2024-04-22T23:27:49.140 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:48 smithi134 ceph-mon[34609]: 0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.smithi067 2024-04-22T23:27:49.140 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:48 smithi134 ceph-mon[34609]: 1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.smithi134 2024-04-22T23:27:49.140 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:48 smithi134 ceph-mon[34609]: fsmap cephfs:2 {0=cephfs.smithi067.xyazmh=up:active,1=cephfs.smithi134.owipad=up:active} 2 up:standby-replay 2024-04-22T23:27:49.140 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:48 smithi134 ceph-mon[34609]: osdmap e48: 6 total, 6 up, 6 in 2024-04-22T23:27:49.140 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:48 smithi134 ceph-mon[34609]: mgrmap e33: smithi067.whdcsr(active, since 2m), standbys: smithi134.ccxkcj 2024-04-22T23:27:49.140 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:48 smithi134 ceph-mon[34609]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T23:27:49.140 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:48 smithi134 ceph-mon[34609]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:27:49.140 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:48 smithi134 ceph-mon[34609]: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:27:49.141 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:48 smithi134 ceph-mon[34609]: from='mgr.14708 ' entity='' 2024-04-22T23:27:49.141 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:48 smithi134 ceph-mon[34609]: mgrmap e34: smithi067.whdcsr(active, since 2m), standbys: smithi134.ccxkcj 2024-04-22T23:27:49.155 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:48 smithi067 ceph-mon[112438]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-22T23:27:49.155 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:48 smithi067 ceph-mon[112438]: from='mgr.14708 172.21.15.67:0/556051609' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-04-22T23:27:49.155 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:48 smithi067 ceph-mon[112438]: pgmap v86: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:27:49.155 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:48 smithi067 ceph-mon[112438]: mon.smithi067 calling monitor election 2024-04-22T23:27:49.155 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:48 smithi067 ceph-mon[112438]: mon.smithi067 is new leader, mons smithi067,smithi134 in quorum (ranks 0,1) 2024-04-22T23:27:49.155 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:48 smithi067 ceph-mon[112438]: monmap epoch 2 2024-04-22T23:27:49.155 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:48 smithi067 ceph-mon[112438]: fsid d46c346e-00fd-11ef-bc93-c7b262605968 2024-04-22T23:27:49.155 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:48 smithi067 ceph-mon[112438]: last_changed 2024-04-22T23:16:04.430584+0000 2024-04-22T23:27:49.155 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:48 smithi067 ceph-mon[112438]: created 2024-04-22T23:13:15.759242+0000 2024-04-22T23:27:49.156 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:48 smithi067 ceph-mon[112438]: min_mon_release 18 (reef) 2024-04-22T23:27:49.156 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:48 smithi067 ceph-mon[112438]: election_strategy: 1 2024-04-22T23:27:49.156 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:48 smithi067 ceph-mon[112438]: 0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.smithi067 2024-04-22T23:27:49.156 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:48 smithi067 ceph-mon[112438]: 1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.smithi134 2024-04-22T23:27:49.156 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:48 smithi067 ceph-mon[112438]: fsmap cephfs:2 {0=cephfs.smithi067.xyazmh=up:active,1=cephfs.smithi134.owipad=up:active} 2 up:standby-replay 2024-04-22T23:27:49.156 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:48 smithi067 ceph-mon[112438]: osdmap e48: 6 total, 6 up, 6 in 2024-04-22T23:27:49.156 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:48 smithi067 ceph-mon[112438]: mgrmap e33: smithi067.whdcsr(active, since 2m), standbys: smithi134.ccxkcj 2024-04-22T23:27:49.156 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:48 smithi067 ceph-mon[112438]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T23:27:49.156 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:48 smithi067 ceph-mon[112438]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:27:49.156 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:48 smithi067 ceph-mon[112438]: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:27:49.156 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:48 smithi067 ceph-mon[112438]: from='mgr.14708 ' entity='' 2024-04-22T23:27:49.156 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:48 smithi067 ceph-mon[112438]: mgrmap e34: smithi067.whdcsr(active, since 2m), standbys: smithi134.ccxkcj 2024-04-22T23:27:55.945 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:55 smithi067 ceph-mon[112438]: Standby manager daemon smithi134.ccxkcj restarted 2024-04-22T23:27:55.945 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:55 smithi067 ceph-mon[112438]: Standby manager daemon smithi134.ccxkcj started 2024-04-22T23:27:55.945 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:55 smithi067 ceph-mon[112438]: from='mgr.? 172.21.15.134:0/1039258521' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi134.ccxkcj/crt"}]: dispatch 2024-04-22T23:27:55.945 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:55 smithi067 ceph-mon[112438]: from='mgr.? 172.21.15.134:0/1039258521' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T23:27:55.946 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:55 smithi067 ceph-mon[112438]: from='mgr.? 172.21.15.134:0/1039258521' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi134.ccxkcj/key"}]: dispatch 2024-04-22T23:27:55.946 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:55 smithi067 ceph-mon[112438]: from='mgr.? 172.21.15.134:0/1039258521' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T23:27:55.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:55 smithi134 ceph-mon[34609]: Standby manager daemon smithi134.ccxkcj restarted 2024-04-22T23:27:55.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:55 smithi134 ceph-mon[34609]: Standby manager daemon smithi134.ccxkcj started 2024-04-22T23:27:55.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:55 smithi134 ceph-mon[34609]: from='mgr.? 172.21.15.134:0/1039258521' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi134.ccxkcj/crt"}]: dispatch 2024-04-22T23:27:55.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:55 smithi134 ceph-mon[34609]: from='mgr.? 172.21.15.134:0/1039258521' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T23:27:55.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:55 smithi134 ceph-mon[34609]: from='mgr.? 172.21.15.134:0/1039258521' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi134.ccxkcj/key"}]: dispatch 2024-04-22T23:27:55.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:55 smithi134 ceph-mon[34609]: from='mgr.? 172.21.15.134:0/1039258521' entity='mgr.smithi134.ccxkcj' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T23:27:56.960 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:56 smithi067 ceph-mon[112438]: mgrmap e35: smithi067.whdcsr(active, since 2m), standbys: smithi134.ccxkcj 2024-04-22T23:27:56.960 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:56 smithi067 ceph-mon[112438]: Active manager daemon smithi067.whdcsr restarted 2024-04-22T23:27:56.960 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:56 smithi067 ceph-mon[112438]: Activating manager daemon smithi067.whdcsr 2024-04-22T23:27:56.960 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:56 smithi067 ceph-mon[112438]: osdmap e49: 6 total, 6 up, 6 in 2024-04-22T23:27:56.960 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:56 smithi067 ceph-mon[112438]: mgrmap e36: smithi067.whdcsr(active, starting, since 0.00572067s), standbys: smithi134.ccxkcj 2024-04-22T23:27:56.960 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-22T23:27:56.960 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-04-22T23:27:56.960 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.xyazmh"}]: dispatch 2024-04-22T23:27:56.960 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.aqlxpv"}]: dispatch 2024-04-22T23:27:56.960 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.owipad"}]: dispatch 2024-04-22T23:27:56.960 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.vuxbsj"}]: dispatch 2024-04-22T23:27:56.961 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mgr metadata", "who": "smithi067.whdcsr", "id": "smithi067.whdcsr"}]: dispatch 2024-04-22T23:27:56.961 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mgr metadata", "who": "smithi134.ccxkcj", "id": "smithi134.ccxkcj"}]: dispatch 2024-04-22T23:27:56.961 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T23:27:56.961 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T23:27:56.961 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T23:27:56.961 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T23:27:56.961 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T23:27:56.961 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T23:27:56.961 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T23:27:56.961 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T23:27:56.961 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T23:27:56.961 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:56 smithi067 ceph-mon[112438]: Manager daemon smithi067.whdcsr is now available 2024-04-22T23:27:56.961 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:27:56.961 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:27:56.962 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.whdcsr/mirror_snapshot_schedule"}]: dispatch 2024-04-22T23:27:56.962 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.whdcsr/trash_purge_schedule"}]: dispatch 2024-04-22T23:27:56.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:56 smithi134 ceph-mon[34609]: mgrmap e35: smithi067.whdcsr(active, since 2m), standbys: smithi134.ccxkcj 2024-04-22T23:27:56.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:56 smithi134 ceph-mon[34609]: Active manager daemon smithi067.whdcsr restarted 2024-04-22T23:27:56.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:56 smithi134 ceph-mon[34609]: Activating manager daemon smithi067.whdcsr 2024-04-22T23:27:56.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:56 smithi134 ceph-mon[34609]: osdmap e49: 6 total, 6 up, 6 in 2024-04-22T23:27:56.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:56 smithi134 ceph-mon[34609]: mgrmap e36: smithi067.whdcsr(active, starting, since 0.00572067s), standbys: smithi134.ccxkcj 2024-04-22T23:27:56.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:56 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-22T23:27:56.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:56 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-04-22T23:27:56.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:56 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.xyazmh"}]: dispatch 2024-04-22T23:27:56.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:56 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.aqlxpv"}]: dispatch 2024-04-22T23:27:56.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:56 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.owipad"}]: dispatch 2024-04-22T23:27:56.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:56 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.vuxbsj"}]: dispatch 2024-04-22T23:27:56.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:56 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mgr metadata", "who": "smithi067.whdcsr", "id": "smithi067.whdcsr"}]: dispatch 2024-04-22T23:27:56.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:56 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mgr metadata", "who": "smithi134.ccxkcj", "id": "smithi134.ccxkcj"}]: dispatch 2024-04-22T23:27:56.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:56 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T23:27:56.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:56 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T23:27:56.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:56 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T23:27:56.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:56 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T23:27:56.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:56 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T23:27:56.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:56 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T23:27:56.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:56 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T23:27:56.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:56 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T23:27:56.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:56 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T23:27:56.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:56 smithi134 ceph-mon[34609]: Manager daemon smithi067.whdcsr is now available 2024-04-22T23:27:56.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:56 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:27:56.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:56 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:27:56.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:56 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.whdcsr/mirror_snapshot_schedule"}]: dispatch 2024-04-22T23:27:56.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:56 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi067.whdcsr/trash_purge_schedule"}]: dispatch 2024-04-22T23:27:58.243 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:57 smithi134 ceph-mon[34609]: mgrmap e37: smithi067.whdcsr(active, since 1.01147s), standbys: smithi134.ccxkcj 2024-04-22T23:27:58.243 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:57 smithi134 ceph-mon[34609]: [22/Apr/2024:23:27:57] ENGINE Bus STARTING 2024-04-22T23:27:58.243 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:57 smithi134 ceph-mon[34609]: [22/Apr/2024:23:27:57] ENGINE Serving on https://172.21.15.67:7150 2024-04-22T23:27:58.243 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:57 smithi134 ceph-mon[34609]: [22/Apr/2024:23:27:57] ENGINE Serving on http://172.21.15.67:8765 2024-04-22T23:27:58.243 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:57 smithi134 ceph-mon[34609]: [22/Apr/2024:23:27:57] ENGINE Bus STARTED 2024-04-22T23:27:58.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:57 smithi067 ceph-mon[112438]: mgrmap e37: smithi067.whdcsr(active, since 1.01147s), standbys: smithi134.ccxkcj 2024-04-22T23:27:58.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:57 smithi067 ceph-mon[112438]: [22/Apr/2024:23:27:57] ENGINE Bus STARTING 2024-04-22T23:27:58.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:57 smithi067 ceph-mon[112438]: [22/Apr/2024:23:27:57] ENGINE Serving on https://172.21.15.67:7150 2024-04-22T23:27:58.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:57 smithi067 ceph-mon[112438]: [22/Apr/2024:23:27:57] ENGINE Serving on http://172.21.15.67:8765 2024-04-22T23:27:58.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:57 smithi067 ceph-mon[112438]: [22/Apr/2024:23:27:57] ENGINE Bus STARTED 2024-04-22T23:27:59.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:58 smithi067 ceph-mon[112438]: pgmap v4: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:27:59.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:27:58 smithi067 ceph-mon[112438]: mgrmap e38: smithi067.whdcsr(active, since 2s), standbys: smithi134.ccxkcj 2024-04-22T23:27:59.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:58 smithi134 ceph-mon[34609]: pgmap v4: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:27:59.243 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:27:58 smithi134 ceph-mon[34609]: mgrmap e38: smithi067.whdcsr(active, since 2s), standbys: smithi134.ccxkcj 2024-04-22T23:28:00.421 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:00 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:00.421 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:00 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:00.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:00 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:00.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:00 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:01.543 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:01 smithi067 ceph-mon[112438]: pgmap v5: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:28:01.543 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:01 smithi067 ceph-mon[112438]: mgrmap e39: smithi067.whdcsr(active, since 4s), standbys: smithi134.ccxkcj 2024-04-22T23:28:01.543 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:01 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:01.543 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:01 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:01.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:01 smithi134 ceph-mon[34609]: pgmap v5: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:28:01.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:01 smithi134 ceph-mon[34609]: mgrmap e39: smithi067.whdcsr(active, since 4s), standbys: smithi134.ccxkcj 2024-04-22T23:28:01.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:01 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:01.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:01 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:02.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:02 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:02.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:02 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:03.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:02 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:03.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:02 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:03.945 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:03 smithi134 ceph-mon[34609]: pgmap v6: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:28:03.945 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:03 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:03.945 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:03 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:04.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:03 smithi067 ceph-mon[112438]: pgmap v6: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:28:04.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:03 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:04.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:03 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:05.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:04 smithi134 ceph-mon[34609]: pgmap v7: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 31 KiB/s rd, 0 B/s wr, 19 op/s 2024-04-22T23:28:05.243 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:04 smithi134 ceph-mon[34609]: Detected new or changed devices on smithi134 2024-04-22T23:28:05.243 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:04 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:05.243 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:04 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:05.297 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:04 smithi067 ceph-mon[112438]: pgmap v7: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 31 KiB/s rd, 0 B/s wr, 19 op/s 2024-04-22T23:28:05.297 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:04 smithi067 ceph-mon[112438]: Detected new or changed devices on smithi134 2024-04-22T23:28:05.298 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:04 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:05.298 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:04 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:07.333 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:07 smithi134 ceph-mon[34609]: pgmap v8: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 24 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-22T23:28:07.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:07 smithi134 ceph-mon[34609]: Detected new or changed devices on smithi067 2024-04-22T23:28:07.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:07 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:07.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:07 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:07.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:07 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:07.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:07 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:28:07.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:07 smithi134 ceph-mon[34609]: Updating smithi067:/etc/ceph/ceph.conf 2024-04-22T23:28:07.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:07 smithi134 ceph-mon[34609]: Updating smithi134:/etc/ceph/ceph.conf 2024-04-22T23:28:07.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:07 smithi134 ceph-mon[34609]: Updating smithi067:/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T23:28:07.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:07 smithi134 ceph-mon[34609]: Updating smithi134:/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T23:28:07.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:07 smithi134 ceph-mon[34609]: Updating smithi134:/etc/ceph/ceph.client.admin.keyring 2024-04-22T23:28:07.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:07 smithi134 ceph-mon[34609]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-04-22T23:28:07.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:07 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:07.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:07 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:07.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:07 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:07.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:07 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:07.744 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:07 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:07.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:07 smithi067 ceph-mon[112438]: pgmap v8: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 24 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-22T23:28:07.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:07 smithi067 ceph-mon[112438]: Detected new or changed devices on smithi067 2024-04-22T23:28:07.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:07 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:07.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:07 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:07.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:07 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:07.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:07 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:28:07.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:07 smithi067 ceph-mon[112438]: Updating smithi067:/etc/ceph/ceph.conf 2024-04-22T23:28:07.794 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:07 smithi067 ceph-mon[112438]: Updating smithi134:/etc/ceph/ceph.conf 2024-04-22T23:28:07.794 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:07 smithi067 ceph-mon[112438]: Updating smithi067:/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T23:28:07.794 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:07 smithi067 ceph-mon[112438]: Updating smithi134:/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T23:28:07.794 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:07 smithi067 ceph-mon[112438]: Updating smithi134:/etc/ceph/ceph.client.admin.keyring 2024-04-22T23:28:07.794 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:07 smithi067 ceph-mon[112438]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-04-22T23:28:07.794 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:07 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:07.794 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:07 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:07.794 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:07 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:07.794 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:07 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:07.794 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:07 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:08.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:08 smithi134 ceph-mon[34609]: Updating smithi134:/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T23:28:08.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:08 smithi134 ceph-mon[34609]: Updating smithi067:/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T23:28:08.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:08 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:28:08.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:08 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:28:08.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:08 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-22T23:28:08.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:08 smithi067 ceph-mon[112438]: Updating smithi134:/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T23:28:08.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:08 smithi067 ceph-mon[112438]: Updating smithi067:/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T23:28:08.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:08 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:28:08.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:08 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:28:08.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:08 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-22T23:28:09.543 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:09 smithi067 ceph-mon[112438]: pgmap v9: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 21 KiB/s rd, 0 B/s wr, 14 op/s 2024-04-22T23:28:09.543 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:09 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:09.543 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:09 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T23:28:09.544 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:09 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T23:28:09.544 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:09 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:09.596 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:09 smithi134 ceph-mon[34609]: pgmap v9: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 21 KiB/s rd, 0 B/s wr, 14 op/s 2024-04-22T23:28:09.596 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:09 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:09.596 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:09 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T23:28:09.597 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:09 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T23:28:09.597 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:09 smithi134 ceph-mon[34609]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:10.694 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:10 smithi134 ceph-mon[34609]: Upgrade: Updating mon.smithi134 2024-04-22T23:28:10.694 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:10 smithi134 ceph-mon[34609]: Deploying daemon mon.smithi134 on smithi134 2024-04-22T23:28:10.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:10 smithi067 ceph-mon[112438]: Upgrade: Updating mon.smithi134 2024-04-22T23:28:10.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:10 smithi067 ceph-mon[112438]: Deploying daemon mon.smithi134 on smithi134 2024-04-22T23:28:11.216 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:10 smithi134 systemd[1]: Stopping Ceph mon.smithi134 for d46c346e-00fd-11ef-bc93-c7b262605968... 2024-04-22T23:28:11.493 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:11 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi134[34580]: 2024-04-22T23:28:11.216+0000 7f0363b0e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi134 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T23:28:11.493 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:11 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi134[34580]: 2024-04-22T23:28:11.216+0000 7f0363b0e700 -1 mon.smithi134@1(peon) e2 *** Got Signal Terminated *** 2024-04-22T23:28:11.493 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:11 smithi134 podman[85463]: 2024-04-22 23:28:11.308661756 +0000 UTC m=+0.202731993 container died 2fb23795f7e3ee64e6f0dc5b6806658d3b8dcf49fa78aaa0b246fbc72bf339aa (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi134, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS) 2024-04-22T23:28:11.892 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:11 smithi134 podman[85463]: 2024-04-22 23:28:11.700060511 +0000 UTC m=+0.594130745 container cleanup 2fb23795f7e3ee64e6f0dc5b6806658d3b8dcf49fa78aaa0b246fbc72bf339aa (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi134, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2) 2024-04-22T23:28:11.892 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:11 smithi134 bash[85463]: ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi134 2024-04-22T23:28:12.166 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:11 smithi134 podman[85477]: 2024-04-22 23:28:11.892953892 +0000 UTC m=+0.584332790 container remove 2fb23795f7e3ee64e6f0dc5b6806658d3b8dcf49fa78aaa0b246fbc72bf339aa (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi134, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1) 2024-04-22T23:28:12.166 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:11 smithi134 systemd[1]: ceph-d46c346e-00fd-11ef-bc93-c7b262605968@mon.smithi134.service: Deactivated successfully. 2024-04-22T23:28:12.166 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:11 smithi134 systemd[1]: Stopped Ceph mon.smithi134 for d46c346e-00fd-11ef-bc93-c7b262605968. 2024-04-22T23:28:12.166 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:11 smithi134 systemd[1]: ceph-d46c346e-00fd-11ef-bc93-c7b262605968@mon.smithi134.service: Consumed 7.867s CPU time. 2024-04-22T23:28:12.417 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 systemd[1]: Starting Ceph mon.smithi134 for d46c346e-00fd-11ef-bc93-c7b262605968... 2024-04-22T23:28:12.418 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 podman[85611]: 2024-04-22 23:28:12.308922502 +0000 UTC m=+0.018501442 image pull a0434699c1d76af4eea1fc95ff0a22ea73d1e04e609464fbf658d6f39bc7f2fc quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c 2024-04-22T23:28:12.418 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 podman[85611]: 2024-04-22 23:28:12.419108433 +0000 UTC m=+0.128687354 container create 20a456e8e56257e85251cd69af703c3ae98985766033c93d6397da443ebc0ff6 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi134, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True) 2024-04-22T23:28:12.671 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 podman[85611]: 2024-04-22 23:28:12.646941479 +0000 UTC m=+0.356520401 container init 20a456e8e56257e85251cd69af703c3ae98985766033c93d6397da443ebc0ff6 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi134, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T23:28:12.672 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 podman[85611]: 2024-04-22 23:28:12.649823955 +0000 UTC m=+0.359402876 container start 20a456e8e56257e85251cd69af703c3ae98985766033c93d6397da443ebc0ff6 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi134, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T23:28:12.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: set uid:gid to 167:167 (ceph:ceph) 2024-04-22T23:28:12.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev), process ceph-mon, pid 2 2024-04-22T23:28:12.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: pidfile_write: ignore empty --pid-file 2024-04-22T23:28:12.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: load: jerasure load: lrc 2024-04-22T23:28:12.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: RocksDB version: 7.9.2 2024-04-22T23:28:12.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Git sha 0 2024-04-22T23:28:12.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Compile date 2024-04-22 21:33:31 2024-04-22T23:28:12.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: DB SUMMARY 2024-04-22T23:28:12.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: DB Session ID: 7EOXV3NKOZYWXBJIRC1J 2024-04-22T23:28:12.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: CURRENT file: CURRENT 2024-04-22T23:28:12.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: IDENTITY file: IDENTITY 2024-04-22T23:28:12.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: MANIFEST file: MANIFEST-000010 size: 1403 Bytes 2024-04-22T23:28:12.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi134/store.db dir, Total Num: 1, files: 000027.sst 2024-04-22T23:28:12.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi134/store.db: 000025.log size: 10845952 ; 2024-04-22T23:28:12.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.error_if_exists: 0 2024-04-22T23:28:12.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.create_if_missing: 0 2024-04-22T23:28:12.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.paranoid_checks: 1 2024-04-22T23:28:12.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-22T23:28:12.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-22T23:28:12.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-22T23:28:12.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.env: 0x55bc3b25bcc0 2024-04-22T23:28:12.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.fs: PosixFileSystem 2024-04-22T23:28:12.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.info_log: 0x55bc3dac1aa0 2024-04-22T23:28:12.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.max_file_opening_threads: 16 2024-04-22T23:28:12.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.statistics: (nil) 2024-04-22T23:28:12.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.use_fsync: 0 2024-04-22T23:28:12.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.max_log_file_size: 0 2024-04-22T23:28:12.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-22T23:28:12.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-22T23:28:12.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.keep_log_file_num: 1000 2024-04-22T23:28:12.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.recycle_log_file_num: 0 2024-04-22T23:28:12.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.allow_fallocate: 1 2024-04-22T23:28:12.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.allow_mmap_reads: 0 2024-04-22T23:28:12.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.allow_mmap_writes: 0 2024-04-22T23:28:12.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.use_direct_reads: 0 2024-04-22T23:28:12.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-22T23:28:12.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.create_missing_column_families: 0 2024-04-22T23:28:12.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.db_log_dir: 2024-04-22T23:28:12.995 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.wal_dir: 2024-04-22T23:28:12.996 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-22T23:28:12.997 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-22T23:28:12.997 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-22T23:28:12.997 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-22T23:28:12.997 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-22T23:28:12.997 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-22T23:28:12.997 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.advise_random_on_open: 1 2024-04-22T23:28:12.997 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.db_write_buffer_size: 0 2024-04-22T23:28:12.998 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.write_buffer_manager: 0x55bc3da550e0 2024-04-22T23:28:12.998 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-22T23:28:12.998 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-22T23:28:12.998 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-22T23:28:12.998 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.rate_limiter: (nil) 2024-04-22T23:28:12.998 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-22T23:28:12.998 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.wal_recovery_mode: 2 2024-04-22T23:28:12.998 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.enable_thread_tracking: 0 2024-04-22T23:28:12.998 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.enable_pipelined_write: 0 2024-04-22T23:28:12.998 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.unordered_write: 0 2024-04-22T23:28:12.998 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-22T23:28:12.998 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-22T23:28:12.998 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-22T23:28:12.999 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-22T23:28:12.999 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.row_cache: None 2024-04-22T23:28:12.999 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.wal_filter: None 2024-04-22T23:28:12.999 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-22T23:28:12.999 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.allow_ingest_behind: 0 2024-04-22T23:28:12.999 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.two_write_queues: 0 2024-04-22T23:28:12.999 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.manual_wal_flush: 0 2024-04-22T23:28:12.999 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.wal_compression: 0 2024-04-22T23:28:12.999 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.atomic_flush: 0 2024-04-22T23:28:12.999 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-22T23:28:12.999 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-22T23:28:12.999 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-22T23:28:12.999 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.log_readahead_size: 0 2024-04-22T23:28:12.999 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-22T23:28:12.999 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.best_efforts_recovery: 0 2024-04-22T23:28:13.000 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-22T23:28:13.000 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-22T23:28:13.000 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.allow_data_in_errors: 0 2024-04-22T23:28:13.000 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.db_host_id: __hostname__ 2024-04-22T23:28:13.000 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-22T23:28:13.000 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.max_background_jobs: 2 2024-04-22T23:28:13.000 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.max_background_compactions: -1 2024-04-22T23:28:13.000 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.max_subcompactions: 1 2024-04-22T23:28:13.000 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-22T23:28:13.000 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-22T23:28:13.000 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-22T23:28:13.000 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.max_total_wal_size: 0 2024-04-22T23:28:13.000 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-22T23:28:13.000 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-22T23:28:13.000 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-22T23:28:13.001 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-22T23:28:13.001 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.max_open_files: -1 2024-04-22T23:28:13.001 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.bytes_per_sync: 0 2024-04-22T23:28:13.001 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-22T23:28:13.001 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-22T23:28:13.001 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.compaction_readahead_size: 0 2024-04-22T23:28:13.001 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.max_background_flushes: -1 2024-04-22T23:28:13.001 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Compression algorithms supported: 2024-04-22T23:28:13.001 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-22T23:28:13.001 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: kZSTD supported: 0 2024-04-22T23:28:13.001 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: kXpressCompression supported: 0 2024-04-22T23:28:13.001 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: kLZ4HCCompression supported: 1 2024-04-22T23:28:13.001 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: kZlibCompression supported: 1 2024-04-22T23:28:13.001 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: kSnappyCompression supported: 1 2024-04-22T23:28:13.002 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: kLZ4Compression supported: 1 2024-04-22T23:28:13.002 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: kBZip2Compression supported: 0 2024-04-22T23:28:13.002 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-22T23:28:13.002 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-22T23:28:13.002 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi134/store.db/MANIFEST-000010 2024-04-22T23:28:13.002 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-22T23:28:13.002 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-22T23:28:13.002 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.merge_operator: 2024-04-22T23:28:13.002 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.compaction_filter: None 2024-04-22T23:28:13.002 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.compaction_filter_factory: None 2024-04-22T23:28:13.002 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.sst_partitioner_factory: None 2024-04-22T23:28:13.002 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-22T23:28:13.002 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-22T23:28:13.002 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55bc3dac1c80) 2024-04-22T23:28:13.003 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: cache_index_and_filter_blocks: 1 2024-04-22T23:28:13.003 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-22T23:28:13.003 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-22T23:28:13.003 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: pin_top_level_index_and_filter: 1 2024-04-22T23:28:13.003 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: index_type: 0 2024-04-22T23:28:13.003 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: data_block_index_type: 0 2024-04-22T23:28:13.003 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: index_shortening: 1 2024-04-22T23:28:13.003 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-22T23:28:13.003 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: checksum: 4 2024-04-22T23:28:13.003 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: no_block_cache: 0 2024-04-22T23:28:13.003 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: block_cache: 0x55bc3da871f0 2024-04-22T23:28:13.003 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: block_cache_name: BinnedLRUCache 2024-04-22T23:28:13.003 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: block_cache_options: 2024-04-22T23:28:13.004 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: capacity : 536870912 2024-04-22T23:28:13.004 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: num_shard_bits : 4 2024-04-22T23:28:13.004 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: strict_capacity_limit : 0 2024-04-22T23:28:13.004 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: high_pri_pool_ratio: 0.000 2024-04-22T23:28:13.004 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: block_cache_compressed: (nil) 2024-04-22T23:28:13.004 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: persistent_cache: (nil) 2024-04-22T23:28:13.004 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: block_size: 4096 2024-04-22T23:28:13.004 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: block_size_deviation: 10 2024-04-22T23:28:13.004 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: block_restart_interval: 16 2024-04-22T23:28:13.004 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: index_block_restart_interval: 1 2024-04-22T23:28:13.004 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: metadata_block_size: 4096 2024-04-22T23:28:13.004 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: partition_filters: 0 2024-04-22T23:28:13.004 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: use_delta_encoding: 1 2024-04-22T23:28:13.004 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: filter_policy: bloomfilter 2024-04-22T23:28:13.004 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: whole_key_filtering: 1 2024-04-22T23:28:13.005 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: verify_compression: 0 2024-04-22T23:28:13.005 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: read_amp_bytes_per_bit: 0 2024-04-22T23:28:13.005 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: format_version: 5 2024-04-22T23:28:13.005 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: enable_index_compression: 1 2024-04-22T23:28:13.005 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: block_align: 0 2024-04-22T23:28:13.005 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: max_auto_readahead_size: 262144 2024-04-22T23:28:13.005 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: prepopulate_block_cache: 0 2024-04-22T23:28:13.005 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: initial_auto_readahead_size: 8192 2024-04-22T23:28:13.005 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: num_file_reads_for_auto_readahead: 2 2024-04-22T23:28:13.005 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.write_buffer_size: 33554432 2024-04-22T23:28:13.005 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.max_write_buffer_number: 2 2024-04-22T23:28:13.005 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.compression: NoCompression 2024-04-22T23:28:13.005 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.bottommost_compression: Disabled 2024-04-22T23:28:13.005 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.prefix_extractor: nullptr 2024-04-22T23:28:13.006 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-22T23:28:13.006 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.num_levels: 7 2024-04-22T23:28:13.006 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-22T23:28:13.006 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-22T23:28:13.006 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-22T23:28:13.006 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-22T23:28:13.006 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-22T23:28:13.006 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-22T23:28:13.006 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-22T23:28:13.006 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-22T23:28:13.006 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-22T23:28:13.006 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-22T23:28:13.006 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-22T23:28:13.006 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-22T23:28:13.006 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-22T23:28:13.007 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.compression_opts.level: 32767 2024-04-22T23:28:13.007 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.compression_opts.strategy: 0 2024-04-22T23:28:13.007 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-22T23:28:13.007 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-22T23:28:13.007 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-22T23:28:13.007 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-22T23:28:13.007 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.compression_opts.enabled: false 2024-04-22T23:28:13.007 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-22T23:28:13.007 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-22T23:28:13.007 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-22T23:28:13.007 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-22T23:28:13.007 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.target_file_size_base: 67108864 2024-04-22T23:28:13.007 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-22T23:28:13.007 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-22T23:28:13.007 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-22T23:28:13.008 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-22T23:28:13.008 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-22T23:28:13.008 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-22T23:28:13.008 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-22T23:28:13.008 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-22T23:28:13.008 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-22T23:28:13.008 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-22T23:28:13.008 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-22T23:28:13.008 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-22T23:28:13.008 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-22T23:28:13.008 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-22T23:28:13.008 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.arena_block_size: 1048576 2024-04-22T23:28:13.008 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-22T23:28:13.008 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-22T23:28:13.008 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.disable_auto_compactions: 0 2024-04-22T23:28:13.009 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-22T23:28:13.009 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-22T23:28:13.009 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-22T23:28:13.009 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-22T23:28:13.009 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-22T23:28:13.009 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-22T23:28:13.009 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-22T23:28:13.009 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-22T23:28:13.009 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-22T23:28:13.009 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-22T23:28:13.009 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-22T23:28:13.009 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.inplace_update_support: 0 2024-04-22T23:28:13.009 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-22T23:28:13.009 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-22T23:28:13.010 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-22T23:28:13.010 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-22T23:28:13.010 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.bloom_locality: 0 2024-04-22T23:28:13.010 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.max_successive_merges: 0 2024-04-22T23:28:13.010 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-22T23:28:13.010 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.paranoid_file_checks: 0 2024-04-22T23:28:13.010 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.force_consistency_checks: 1 2024-04-22T23:28:13.010 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.report_bg_io_stats: 0 2024-04-22T23:28:13.010 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.ttl: 2592000 2024-04-22T23:28:13.010 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-22T23:28:13.010 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-22T23:28:13.010 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-22T23:28:13.010 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.enable_blob_files: false 2024-04-22T23:28:13.010 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.min_blob_size: 0 2024-04-22T23:28:13.010 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.blob_file_size: 268435456 2024-04-22T23:28:13.011 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-22T23:28:13.011 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-22T23:28:13.011 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-22T23:28:13.011 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-22T23:28:13.011 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-22T23:28:13.011 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.blob_file_starting_level: 0 2024-04-22T23:28:13.011 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-22T23:28:13.011 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-22T23:28:13.011 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi134/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 29, last_sequence is 12150, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 25 2024-04-22T23:28:13.011 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-04-22T23:28:13.011 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 859ad799-3c2f-4498-9dd5-2d736f84a4e9 2024-04-22T23:28:13.011 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713828492689821, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-04-22T23:28:13.011 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-04-22T23:28:13.011 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713828492714549, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 6393484, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12155, "largest_seqno": 16356, "table_properties": {"data_size": 6378165, "index_size": 9920, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 4357, "raw_key_size": 38075, "raw_average_key_size": 22, "raw_value_size": 6344005, "raw_average_value_size": 3696, "num_data_blocks": 444, "num_entries": 1716, "num_filter_entries": 1716, "num_deletions": 259, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713828492, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "859ad799-3c2f-4498-9dd5-2d736f84a4e9", "db_session_id": "7EOXV3NKOZYWXBJIRC1J", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-04-22T23:28:13.012 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713828492714669, "job": 1, "event": "recovery_finished"} 2024-04-22T23:28:13.012 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-04-22T23:28:13.012 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-22T23:28:13.012 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi134/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-22T23:28:13.012 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55bc3db9c000 2024-04-22T23:28:13.012 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: DB pointer 0x55bc3db86000 2024-04-22T23:28:13.012 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-22T23:28:13.012 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-22T23:28:13.012 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: ** DB Stats ** 2024-04-22T23:28:13.012 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T23:28:13.012 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-22T23:28:13.012 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T23:28:13.012 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T23:28:13.012 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-22T23:28:13.013 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T23:28:13.013 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T23:28:13.013 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: 2024-04-22T23:28:13.013 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: ** Compaction Stats [default] ** 2024-04-22T23:28:13.013 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T23:28:13.013 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-22T23:28:13.013 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: L0 1/0 6.10 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 615.2 0.01 0.00 1 0.010 0 0 0.0 0.0 2024-04-22T23:28:13.013 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: L6 1/0 4.93 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-22T23:28:13.013 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Sum 2/0 11.03 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 615.2 0.01 0.00 1 0.010 0 0 0.0 0.0 2024-04-22T23:28:13.013 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 615.2 0.01 0.00 1 0.010 0 0 0.0 0.0 2024-04-22T23:28:13.013 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: 2024-04-22T23:28:13.013 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: ** Compaction Stats [default] ** 2024-04-22T23:28:13.013 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T23:28:13.013 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T23:28:13.014 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 615.2 0.01 0.00 1 0.010 0 0 0.0 0.0 2024-04-22T23:28:13.014 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: 2024-04-22T23:28:13.014 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-22T23:28:13.014 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: 2024-04-22T23:28:13.014 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T23:28:13.014 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Flush(GB): cumulative 0.006, interval 0.006 2024-04-22T23:28:13.014 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T23:28:13.014 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T23:28:13.014 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T23:28:13.014 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T23:28:13.014 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Cumulative compaction: 0.01 GB write, 153.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T23:28:13.014 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Interval compaction: 0.01 GB write, 153.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T23:28:13.014 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T23:28:13.014 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Block cache BinnedLRUCache@0x55bc3da871f0#2 capacity: 512.00 MB usage: 14.61 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-04-22T23:28:13.014 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,4.53 KB,0.000864267%) IndexBlock(1,10.08 KB,0.00192225%) Misc(1,0.00 KB,0%) 2024-04-22T23:28:13.015 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: 2024-04-22T23:28:13.015 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T23:28:13.015 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: starting mon.smithi134 rank 1 at public addrs [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] at bind addrs [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi134 fsid d46c346e-00fd-11ef-bc93-c7b262605968 2024-04-22T23:28:13.015 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: mon.smithi134@-1(???) e2 preinit fsid d46c346e-00fd-11ef-bc93-c7b262605968 2024-04-22T23:28:13.015 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: mon.smithi134@-1(???).mds e22 new map 2024-04-22T23:28:13.015 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: mon.smithi134@-1(???).mds e22 print_map 2024-04-22T23:28:13.015 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: e22 2024-04-22T23:28:13.015 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:28:13.015 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:28:13.015 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: legacy client fscid: 1 2024-04-22T23:28:13.015 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: 2024-04-22T23:28:13.015 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: Filesystem 'cephfs' (1) 2024-04-22T23:28:13.015 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: fs_name cephfs 2024-04-22T23:28:13.015 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: epoch 22 2024-04-22T23:28:13.015 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T23:28:13.016 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:28:13.016 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: modified 2024-04-22T23:27:20.577198+0000 2024-04-22T23:28:13.016 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: tableserver 0 2024-04-22T23:28:13.016 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: root 0 2024-04-22T23:28:13.016 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: session_timeout 60 2024-04-22T23:28:13.016 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: session_autoclose 300 2024-04-22T23:28:13.016 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: max_file_size 1099511627776 2024-04-22T23:28:13.016 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: max_xattr_size 65536 2024-04-22T23:28:13.016 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: required_client_features {} 2024-04-22T23:28:13.016 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: last_failure 0 2024-04-22T23:28:13.016 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: last_failure_osd_epoch 46 2024-04-22T23:28:13.016 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:28:13.016 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: max_mds 2 2024-04-22T23:28:13.016 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: in 0,1 2024-04-22T23:28:13.017 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: up {0=14484,1=24273} 2024-04-22T23:28:13.017 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: failed 2024-04-22T23:28:13.017 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: damaged 2024-04-22T23:28:13.017 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: stopped 2024-04-22T23:28:13.017 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: data_pools [3] 2024-04-22T23:28:13.017 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: metadata_pool 2 2024-04-22T23:28:13.017 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: inline_data enabled 2024-04-22T23:28:13.017 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: balancer 2024-04-22T23:28:13.017 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: bal_rank_mask -1 2024-04-22T23:28:13.017 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: standby_count_wanted 1 2024-04-22T23:28:13.017 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: qdb_cluster leader: 0 members: 2024-04-22T23:28:13.017 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: [mds.cephfs.smithi067.xyazmh{0:14484} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.67:6828/3606151860,v1:172.21.15.67:6829/3606151860] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:28:13.017 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: [mds.cephfs.smithi067.aqlxpv{0:14496} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/3305304988,v1:172.21.15.67:6827/3305304988] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:28:13.017 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: [mds.cephfs.smithi134.owipad{1:24273} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.134:6824/535957142,v1:172.21.15.134:6825/535957142] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:28:13.017 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: [mds.cephfs.smithi134.vuxbsj{1:24289} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.134:6826/1954821676,v1:172.21.15.134:6827/1954821676] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:28:13.018 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: 2024-04-22T23:28:13.018 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout: 2024-04-22T23:28:13.018 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: mon.smithi134@-1(???).osd e49 crush map has features 3314933000852226048, adjusting msgr requires 2024-04-22T23:28:13.018 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: mon.smithi134@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-04-22T23:28:13.018 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: mon.smithi134@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-04-22T23:28:13.018 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: mon.smithi134@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-04-22T23:28:13.018 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: mon.smithi134@-1(???).paxosservice(auth 1..24) refresh upgraded, format 0 -> 3 2024-04-22T23:28:13.018 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 bash[85611]: 20a456e8e56257e85251cd69af703c3ae98985766033c93d6397da443ebc0ff6 2024-04-22T23:28:13.018 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 systemd[1]: Started Ceph mon.smithi134 for d46c346e-00fd-11ef-bc93-c7b262605968. 2024-04-22T23:28:13.018 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: pgmap v10: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-22T23:28:13.018 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:12 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:28:13.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:12 smithi067 ceph-mon[112438]: pgmap v10: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-22T23:28:13.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:12 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:28:13.899 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:13 smithi134 ceph-mon[85630]: pgmap v11: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-22T23:28:13.899 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:13 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-22T23:28:13.899 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:13 smithi134 ceph-mon[85630]: mon.smithi067 calling monitor election 2024-04-22T23:28:13.899 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:13 smithi134 ceph-mon[85630]: mon.smithi134 calling monitor election 2024-04-22T23:28:13.899 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:13 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-04-22T23:28:13.899 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:13 smithi134 ceph-mon[85630]: mon.smithi067 is new leader, mons smithi067,smithi134 in quorum (ranks 0,1) 2024-04-22T23:28:13.899 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:13 smithi134 ceph-mon[85630]: monmap epoch 3 2024-04-22T23:28:13.899 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:13 smithi134 ceph-mon[85630]: fsid d46c346e-00fd-11ef-bc93-c7b262605968 2024-04-22T23:28:13.900 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:13 smithi134 ceph-mon[85630]: last_changed 2024-04-22T23:28:12.774231+0000 2024-04-22T23:28:13.900 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:13 smithi134 ceph-mon[85630]: created 2024-04-22T23:13:15.759242+0000 2024-04-22T23:28:13.900 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:13 smithi134 ceph-mon[85630]: min_mon_release 19 (squid) 2024-04-22T23:28:13.900 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:13 smithi134 ceph-mon[85630]: election_strategy: 1 2024-04-22T23:28:13.900 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:13 smithi134 ceph-mon[85630]: 0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.smithi067 2024-04-22T23:28:13.900 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:13 smithi134 ceph-mon[85630]: 1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.smithi134 2024-04-22T23:28:13.900 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:13 smithi134 ceph-mon[85630]: fsmap cephfs:2 {0=cephfs.smithi067.xyazmh=up:active,1=cephfs.smithi134.owipad=up:active} 2 up:standby-replay 2024-04-22T23:28:13.900 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:13 smithi134 ceph-mon[85630]: osdmap e49: 6 total, 6 up, 6 in 2024-04-22T23:28:13.900 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:13 smithi134 ceph-mon[85630]: mgrmap e39: smithi067.whdcsr(active, since 16s), standbys: smithi134.ccxkcj 2024-04-22T23:28:13.900 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:13 smithi134 ceph-mon[85630]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T23:28:13.900 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:13 smithi134 ceph-mon[85630]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:28:13.900 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:13 smithi134 ceph-mon[85630]: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:28:13.900 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:13 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:13.900 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:13 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:13.900 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:13 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:28:14.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:13 smithi067 ceph-mon[112438]: pgmap v11: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-22T23:28:14.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:13 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-22T23:28:14.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:13 smithi067 ceph-mon[112438]: mon.smithi067 calling monitor election 2024-04-22T23:28:14.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:13 smithi067 ceph-mon[112438]: mon.smithi134 calling monitor election 2024-04-22T23:28:14.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:13 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-04-22T23:28:14.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:13 smithi067 ceph-mon[112438]: mon.smithi067 is new leader, mons smithi067,smithi134 in quorum (ranks 0,1) 2024-04-22T23:28:14.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:13 smithi067 ceph-mon[112438]: monmap epoch 3 2024-04-22T23:28:14.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:13 smithi067 ceph-mon[112438]: fsid d46c346e-00fd-11ef-bc93-c7b262605968 2024-04-22T23:28:14.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:13 smithi067 ceph-mon[112438]: last_changed 2024-04-22T23:28:12.774231+0000 2024-04-22T23:28:14.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:13 smithi067 ceph-mon[112438]: created 2024-04-22T23:13:15.759242+0000 2024-04-22T23:28:14.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:13 smithi067 ceph-mon[112438]: min_mon_release 19 (squid) 2024-04-22T23:28:14.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:13 smithi067 ceph-mon[112438]: election_strategy: 1 2024-04-22T23:28:14.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:13 smithi067 ceph-mon[112438]: 0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.smithi067 2024-04-22T23:28:14.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:13 smithi067 ceph-mon[112438]: 1: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.smithi134 2024-04-22T23:28:14.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:13 smithi067 ceph-mon[112438]: fsmap cephfs:2 {0=cephfs.smithi067.xyazmh=up:active,1=cephfs.smithi134.owipad=up:active} 2 up:standby-replay 2024-04-22T23:28:14.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:13 smithi067 ceph-mon[112438]: osdmap e49: 6 total, 6 up, 6 in 2024-04-22T23:28:14.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:13 smithi067 ceph-mon[112438]: mgrmap e39: smithi067.whdcsr(active, since 16s), standbys: smithi134.ccxkcj 2024-04-22T23:28:14.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:13 smithi067 ceph-mon[112438]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T23:28:14.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:13 smithi067 ceph-mon[112438]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:28:14.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:13 smithi067 ceph-mon[112438]: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:28:14.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:13 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:14.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:13 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:14.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:13 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:28:14.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:14.897+0000 7fb8b34d8700 1 -- 172.21.15.67:0/2625516167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8ac100340 msgr2=0x7fb8ac10d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:14.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:14.897+0000 7fb8b34d8700 1 --2- 172.21.15.67:0/2625516167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8ac100340 0x7fb8ac10d000 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb89c009910 tx=0x7fb89c009c20 comp rx=0 tx=0).stop 2024-04-22T23:28:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:14.897+0000 7fb8b34d8700 1 -- 172.21.15.67:0/2625516167 shutdown_connections 2024-04-22T23:28:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:14.897+0000 7fb8b34d8700 1 --2- 172.21.15.67:0/2625516167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8ac100340 0x7fb8ac10d000 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:14.897+0000 7fb8b34d8700 1 --2- 172.21.15.67:0/2625516167 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb8ac0ff990 0x7fb8ac0ffd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:14.897+0000 7fb8b34d8700 1 -- 172.21.15.67:0/2625516167 >> 172.21.15.67:0/2625516167 conn(0x7fb8ac0fb650 msgr2=0x7fb8ac0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:28:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:14.897+0000 7fb8b34d8700 1 -- 172.21.15.67:0/2625516167 shutdown_connections 2024-04-22T23:28:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:14.897+0000 7fb8b34d8700 1 -- 172.21.15.67:0/2625516167 wait complete. 2024-04-22T23:28:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:14.898+0000 7fb8b34d8700 1 Processor -- start 2024-04-22T23:28:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:14.898+0000 7fb8b34d8700 1 -- start start 2024-04-22T23:28:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:14.898+0000 7fb8b34d8700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb8ac0ff990 0x7fb8ac19e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:14.898+0000 7fb8b34d8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8ac100340 0x7fb8ac19ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:14.898+0000 7fb8b34d8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ac19f470 con 0x7fb8ac100340 2024-04-22T23:28:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:14.898+0000 7fb8b34d8700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ac198920 con 0x7fb8ac0ff990 2024-04-22T23:28:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:14.898+0000 7fb8b0a73700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8ac100340 0x7fb8ac19ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:14.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:14.898+0000 7fb8b0a73700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8ac100340 0x7fb8ac19ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45222/0 (socket says 172.21.15.67:45222) 2024-04-22T23:28:14.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:14.898+0000 7fb8b0a73700 1 -- 172.21.15.67:0/457710207 learned_addr learned my addr 172.21.15.67:0/457710207 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:28:14.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:14.898+0000 7fb8b1274700 1 --2- 172.21.15.67:0/457710207 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb8ac0ff990 0x7fb8ac19e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:14.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:14.898+0000 7fb8b1274700 1 -- 172.21.15.67:0/457710207 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb8ac0ff990 msgr2=0x7fb8ac19e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-22T23:28:14.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:14.898+0000 7fb8b1274700 1 -- 172.21.15.67:0/457710207 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb8ac0ff990 msgr2=0x7fb8ac19e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T23:28:14.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:14.898+0000 7fb8b1274700 1 --2- 172.21.15.67:0/457710207 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb8ac0ff990 0x7fb8ac19e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T23:28:14.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:14.898+0000 7fb8b0a73700 1 -- 172.21.15.67:0/457710207 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb8ac0ff990 msgr2=0x7fb8ac19e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:14.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:14.898+0000 7fb8b1274700 1 --2- 172.21.15.67:0/457710207 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb8ac0ff990 0x7fb8ac19e8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T23:28:14.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:14.899+0000 7fb8b0a73700 1 --2- 172.21.15.67:0/457710207 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb8ac0ff990 0x7fb8ac19e8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:14.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:14.899+0000 7fb8b0a73700 1 -- 172.21.15.67:0/457710207 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb89c0095f0 con 0x7fb8ac100340 2024-04-22T23:28:14.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:14.899+0000 7fb8b0a73700 1 --2- 172.21.15.67:0/457710207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8ac100340 0x7fb8ac19ede0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb89c000c00 tx=0x7fb89c0056a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:28:14.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:14.899+0000 7fb8a27fc700 1 -- 172.21.15.67:0/457710207 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb89c01d070 con 0x7fb8ac100340 2024-04-22T23:28:14.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:14.899+0000 7fb8b34d8700 1 -- 172.21.15.67:0/457710207 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8ac198ba0 con 0x7fb8ac100340 2024-04-22T23:28:14.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:14.899+0000 7fb8a27fc700 1 -- 172.21.15.67:0/457710207 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb89c00f460 con 0x7fb8ac100340 2024-04-22T23:28:14.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:14.899+0000 7fb8a27fc700 1 -- 172.21.15.67:0/457710207 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb89c021640 con 0x7fb8ac100340 2024-04-22T23:28:14.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:14.899+0000 7fb8b34d8700 1 -- 172.21.15.67:0/457710207 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8ac1990f0 con 0x7fb8ac100340 2024-04-22T23:28:14.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:14.900+0000 7fb8b34d8700 1 -- 172.21.15.67:0/457710207 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8ac10a770 con 0x7fb8ac100340 2024-04-22T23:28:14.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:14.901+0000 7fb8a27fc700 1 -- 172.21.15.67:0/457710207 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 39) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fb89c00fa60 con 0x7fb8ac100340 2024-04-22T23:28:14.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:14.901+0000 7fb8a27fc700 1 --2- 172.21.15.67:0/457710207 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb89806ee00 0x7fb8980712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:14.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:14.901+0000 7fb8a27fc700 1 -- 172.21.15.67:0/457710207 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fb89c0901e0 con 0x7fb8ac100340 2024-04-22T23:28:14.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:14.901+0000 7fb8b1274700 1 --2- 172.21.15.67:0/457710207 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb89806ee00 0x7fb8980712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:14.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:14.902+0000 7fb8b1274700 1 --2- 172.21.15.67:0/457710207 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb89806ee00 0x7fb8980712c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb8a8005950 tx=0x7fb8a800f640 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:28:14.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:14.906+0000 7fb8a27fc700 1 -- 172.21.15.67:0/457710207 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb89c05c980 con 0x7fb8ac100340 2024-04-22T23:28:15.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.137+0000 7fb8b34d8700 1 -- 172.21.15.67:0/457710207 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb8ac199ed0 con 0x7fb89806ee00 2024-04-22T23:28:15.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.139+0000 7fb8a27fc700 1 -- 172.21.15.67:0/457710207 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7fb8ac199ed0 con 0x7fb89806ee00 2024-04-22T23:28:15.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.141+0000 7fb8b34d8700 1 -- 172.21.15.67:0/457710207 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb89806ee00 msgr2=0x7fb8980712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:15.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.141+0000 7fb8b34d8700 1 --2- 172.21.15.67:0/457710207 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb89806ee00 0x7fb8980712c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb8a8005950 tx=0x7fb8a800f640 comp rx=0 tx=0).stop 2024-04-22T23:28:15.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.141+0000 7fb8b34d8700 1 -- 172.21.15.67:0/457710207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8ac100340 msgr2=0x7fb8ac19ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:15.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.141+0000 7fb8b34d8700 1 --2- 172.21.15.67:0/457710207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8ac100340 0x7fb8ac19ede0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb89c000c00 tx=0x7fb89c0056a0 comp rx=0 tx=0).stop 2024-04-22T23:28:15.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.141+0000 7fb8b34d8700 1 -- 172.21.15.67:0/457710207 shutdown_connections 2024-04-22T23:28:15.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.141+0000 7fb8b34d8700 1 --2- 172.21.15.67:0/457710207 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb89806ee00 0x7fb8980712c0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:15.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.141+0000 7fb8b34d8700 1 --2- 172.21.15.67:0/457710207 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb8ac0ff990 0x7fb8ac19e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:15.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.141+0000 7fb8b34d8700 1 --2- 172.21.15.67:0/457710207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8ac100340 0x7fb8ac19ede0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:15.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.141+0000 7fb8b34d8700 1 -- 172.21.15.67:0/457710207 >> 172.21.15.67:0/457710207 conn(0x7fb8ac0fb650 msgr2=0x7fb8ac0fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:28:15.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.141+0000 7fb8b34d8700 1 -- 172.21.15.67:0/457710207 shutdown_connections 2024-04-22T23:28:15.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.141+0000 7fb8b34d8700 1 -- 172.21.15.67:0/457710207 wait complete. 2024-04-22T23:28:15.152 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:28:15.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.251+0000 7fe1e5c34700 1 -- 172.21.15.67:0/855811513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1e00ffc20 msgr2=0x7fe1e0100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:15.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.251+0000 7fe1e5c34700 1 --2- 172.21.15.67:0/855811513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1e00ffc20 0x7fe1e0100080 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe1d0009910 tx=0x7fe1d0009c20 comp rx=0 tx=0).stop 2024-04-22T23:28:15.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.251+0000 7fe1e5c34700 1 -- 172.21.15.67:0/855811513 shutdown_connections 2024-04-22T23:28:15.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.251+0000 7fe1e5c34700 1 --2- 172.21.15.67:0/855811513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1e00ffc20 0x7fe1e0100080 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:15.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.251+0000 7fe1e5c34700 1 --2- 172.21.15.67:0/855811513 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1e0105bd0 0x7fe1e0105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:15.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.251+0000 7fe1e5c34700 1 -- 172.21.15.67:0/855811513 >> 172.21.15.67:0/855811513 conn(0x7fe1e0075670 msgr2=0x7fe1e0075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:28:15.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.251+0000 7fe1e5c34700 1 -- 172.21.15.67:0/855811513 shutdown_connections 2024-04-22T23:28:15.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.251+0000 7fe1e5c34700 1 -- 172.21.15.67:0/855811513 wait complete. 2024-04-22T23:28:15.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.252+0000 7fe1e5c34700 1 Processor -- start 2024-04-22T23:28:15.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.252+0000 7fe1e5c34700 1 -- start start 2024-04-22T23:28:15.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.252+0000 7fe1e5c34700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1e00ffc20 0x7fe1e0197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:15.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.252+0000 7fe1e5c34700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1e0105bd0 0x7fe1e0197c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:15.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.252+0000 7fe1e5c34700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e0198290 con 0x7fe1e0105bd0 2024-04-22T23:28:15.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.252+0000 7fe1e5c34700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e01983d0 con 0x7fe1e00ffc20 2024-04-22T23:28:15.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.252+0000 7fe1deffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1e0105bd0 0x7fe1e0197c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:15.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.252+0000 7fe1deffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1e0105bd0 0x7fe1e0197c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45250/0 (socket says 172.21.15.67:45250) 2024-04-22T23:28:15.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.252+0000 7fe1deffd700 1 -- 172.21.15.67:0/3092611304 learned_addr learned my addr 172.21.15.67:0/3092611304 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:28:15.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.253+0000 7fe1df7fe700 1 --2- 172.21.15.67:0/3092611304 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1e00ffc20 0x7fe1e0197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:15.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.253+0000 7fe1df7fe700 1 -- 172.21.15.67:0/3092611304 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1e00ffc20 msgr2=0x7fe1e0197700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-22T23:28:15.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.253+0000 7fe1df7fe700 1 -- 172.21.15.67:0/3092611304 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1e00ffc20 msgr2=0x7fe1e0197700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T23:28:15.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.253+0000 7fe1df7fe700 1 --2- 172.21.15.67:0/3092611304 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1e00ffc20 0x7fe1e0197700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T23:28:15.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.253+0000 7fe1df7fe700 1 --2- 172.21.15.67:0/3092611304 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1e00ffc20 0x7fe1e0197700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T23:28:15.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.253+0000 7fe1deffd700 1 -- 172.21.15.67:0/3092611304 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1e00ffc20 msgr2=0x7fe1e0197700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T23:28:15.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.253+0000 7fe1deffd700 1 --2- 172.21.15.67:0/3092611304 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1e00ffc20 0x7fe1e0197700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:15.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.253+0000 7fe1deffd700 1 -- 172.21.15.67:0/3092611304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d00095f0 con 0x7fe1e0105bd0 2024-04-22T23:28:15.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.253+0000 7fe1deffd700 1 --2- 172.21.15.67:0/3092611304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1e0105bd0 0x7fe1e0197c40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe1d000b5c0 tx=0x7fe1d000be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:28:15.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.253+0000 7fe1dcff9700 1 -- 172.21.15.67:0/3092611304 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1d001d070 con 0x7fe1e0105bd0 2024-04-22T23:28:15.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.253+0000 7fe1dcff9700 1 -- 172.21.15.67:0/3092611304 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe1d000f460 con 0x7fe1e0105bd0 2024-04-22T23:28:15.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.253+0000 7fe1e5c34700 1 -- 172.21.15.67:0/3092611304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1e01a0770 con 0x7fe1e0105bd0 2024-04-22T23:28:15.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.254+0000 7fe1dcff9700 1 -- 172.21.15.67:0/3092611304 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1d0017610 con 0x7fe1e0105bd0 2024-04-22T23:28:15.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.254+0000 7fe1e5c34700 1 -- 172.21.15.67:0/3092611304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1e01a0be0 con 0x7fe1e0105bd0 2024-04-22T23:28:15.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.255+0000 7fe1dcff9700 1 -- 172.21.15.67:0/3092611304 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 39) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fe1d0017880 con 0x7fe1e0105bd0 2024-04-22T23:28:15.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.255+0000 7fe1e5c34700 1 -- 172.21.15.67:0/3092611304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1e01080d0 con 0x7fe1e0105bd0 2024-04-22T23:28:15.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.255+0000 7fe1dcff9700 1 --2- 172.21.15.67:0/3092611304 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe1c806edf0 0x7fe1c80712b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:15.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.255+0000 7fe1dcff9700 1 -- 172.21.15.67:0/3092611304 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fe1d008ff20 con 0x7fe1e0105bd0 2024-04-22T23:28:15.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.255+0000 7fe1df7fe700 1 --2- 172.21.15.67:0/3092611304 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe1c806edf0 0x7fe1c80712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:15.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.256+0000 7fe1df7fe700 1 --2- 172.21.15.67:0/3092611304 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe1c806edf0 0x7fe1c80712b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe1d4005f50 tx=0x7fe1d400a5c0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:28:15.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.260+0000 7fe1dcff9700 1 -- 172.21.15.67:0/3092611304 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe1d0095060 con 0x7fe1e0105bd0 2024-04-22T23:28:15.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.492+0000 7fe1e5c34700 1 -- 172.21.15.67:0/3092611304 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe1e002ccc0 con 0x7fe1c806edf0 2024-04-22T23:28:15.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.493+0000 7fe1dcff9700 1 -- 172.21.15.67:0/3092611304 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7fe1e002ccc0 con 0x7fe1c806edf0 2024-04-22T23:28:15.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.496+0000 7fe1e5c34700 1 -- 172.21.15.67:0/3092611304 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe1c806edf0 msgr2=0x7fe1c80712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:15.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.496+0000 7fe1e5c34700 1 --2- 172.21.15.67:0/3092611304 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe1c806edf0 0x7fe1c80712b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe1d4005f50 tx=0x7fe1d400a5c0 comp rx=0 tx=0).stop 2024-04-22T23:28:15.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.496+0000 7fe1e5c34700 1 -- 172.21.15.67:0/3092611304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1e0105bd0 msgr2=0x7fe1e0197c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:15.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.496+0000 7fe1e5c34700 1 --2- 172.21.15.67:0/3092611304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1e0105bd0 0x7fe1e0197c40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe1d000b5c0 tx=0x7fe1d000be00 comp rx=0 tx=0).stop 2024-04-22T23:28:15.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.497+0000 7fe1e5c34700 1 -- 172.21.15.67:0/3092611304 shutdown_connections 2024-04-22T23:28:15.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.497+0000 7fe1e5c34700 1 --2- 172.21.15.67:0/3092611304 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe1c806edf0 0x7fe1c80712b0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:15.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.497+0000 7fe1e5c34700 1 --2- 172.21.15.67:0/3092611304 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1e00ffc20 0x7fe1e0197700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:15.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.497+0000 7fe1e5c34700 1 --2- 172.21.15.67:0/3092611304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1e0105bd0 0x7fe1e0197c40 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:15.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.497+0000 7fe1e5c34700 1 -- 172.21.15.67:0/3092611304 >> 172.21.15.67:0/3092611304 conn(0x7fe1e0075670 msgr2=0x7fe1e00fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:28:15.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.497+0000 7fe1e5c34700 1 -- 172.21.15.67:0/3092611304 shutdown_connections 2024-04-22T23:28:15.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.497+0000 7fe1e5c34700 1 -- 172.21.15.67:0/3092611304 wait complete. 2024-04-22T23:28:15.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.605+0000 7fd81ca6a700 1 -- 172.21.15.67:0/3294834006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd818105bf0 msgr2=0x7fd818105fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:15.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.605+0000 7fd81ca6a700 1 --2- 172.21.15.67:0/3294834006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd818105bf0 0x7fd818105fd0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd808009910 tx=0x7fd808009c20 comp rx=0 tx=0).stop 2024-04-22T23:28:15.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.605+0000 7fd81ca6a700 1 -- 172.21.15.67:0/3294834006 shutdown_connections 2024-04-22T23:28:15.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.605+0000 7fd81ca6a700 1 --2- 172.21.15.67:0/3294834006 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8180ffb40 0x7fd8180fffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:15.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.605+0000 7fd81ca6a700 1 --2- 172.21.15.67:0/3294834006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd818105bf0 0x7fd818105fd0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:15.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.605+0000 7fd81ca6a700 1 -- 172.21.15.67:0/3294834006 >> 172.21.15.67:0/3294834006 conn(0x7fd8180fb610 msgr2=0x7fd8180fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:28:15.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.606+0000 7fd81ca6a700 1 -- 172.21.15.67:0/3294834006 shutdown_connections 2024-04-22T23:28:15.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.606+0000 7fd81ca6a700 1 -- 172.21.15.67:0/3294834006 wait complete. 2024-04-22T23:28:15.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.606+0000 7fd81ca6a700 1 Processor -- start 2024-04-22T23:28:15.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.606+0000 7fd81ca6a700 1 -- start start 2024-04-22T23:28:15.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.607+0000 7fd81ca6a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8180ffb40 0x7fd81819e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:15.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.607+0000 7fd81ca6a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd818105bf0 0x7fd81819ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:15.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.607+0000 7fd81ca6a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81819f390 con 0x7fd818105bf0 2024-04-22T23:28:15.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.607+0000 7fd81ca6a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd818198840 con 0x7fd8180ffb40 2024-04-22T23:28:15.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.607+0000 7fd81659c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8180ffb40 0x7fd81819e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:15.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.607+0000 7fd81659c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8180ffb40 0x7fd81819e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.67:47346/0 (socket says 172.21.15.67:47346) 2024-04-22T23:28:15.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.607+0000 7fd815d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd818105bf0 0x7fd81819ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:15.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.607+0000 7fd81659c700 1 -- 172.21.15.67:0/2191378606 learned_addr learned my addr 172.21.15.67:0/2191378606 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:28:15.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.607+0000 7fd81659c700 1 -- 172.21.15.67:0/2191378606 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8180ffb40 msgr2=0x7fd81819e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-22T23:28:15.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.607+0000 7fd81659c700 1 -- 172.21.15.67:0/2191378606 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8180ffb40 msgr2=0x7fd81819e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T23:28:15.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.607+0000 7fd81659c700 1 --2- 172.21.15.67:0/2191378606 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8180ffb40 0x7fd81819e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T23:28:15.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.607+0000 7fd81659c700 1 --2- 172.21.15.67:0/2191378606 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8180ffb40 0x7fd81819e7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T23:28:15.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.607+0000 7fd815d9b700 1 -- 172.21.15.67:0/2191378606 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8180ffb40 msgr2=0x7fd81819e7c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T23:28:15.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.607+0000 7fd815d9b700 1 --2- 172.21.15.67:0/2191378606 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8180ffb40 0x7fd81819e7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:15.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.607+0000 7fd815d9b700 1 -- 172.21.15.67:0/2191378606 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8080095f0 con 0x7fd818105bf0 2024-04-22T23:28:15.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.607+0000 7fd815d9b700 1 --2- 172.21.15.67:0/2191378606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd818105bf0 0x7fd81819ed00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd80c00d8d0 tx=0x7fd80c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:28:15.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.607+0000 7fd8077fe700 1 -- 172.21.15.67:0/2191378606 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd80c009900 con 0x7fd818105bf0 2024-04-22T23:28:15.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.608+0000 7fd81ca6a700 1 -- 172.21.15.67:0/2191378606 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd818198b20 con 0x7fd818105bf0 2024-04-22T23:28:15.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.608+0000 7fd8077fe700 1 -- 172.21.15.67:0/2191378606 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd80c010460 con 0x7fd818105bf0 2024-04-22T23:28:15.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.608+0000 7fd8077fe700 1 -- 172.21.15.67:0/2191378606 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd80c00b5d0 con 0x7fd818105bf0 2024-04-22T23:28:15.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.608+0000 7fd81ca6a700 1 -- 172.21.15.67:0/2191378606 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd818199070 con 0x7fd818105bf0 2024-04-22T23:28:15.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.609+0000 7fd81ca6a700 1 -- 172.21.15.67:0/2191378606 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8180679c0 con 0x7fd818105bf0 2024-04-22T23:28:15.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.609+0000 7fd8077fe700 1 -- 172.21.15.67:0/2191378606 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 39) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fd80c0105d0 con 0x7fd818105bf0 2024-04-22T23:28:15.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.610+0000 7fd8077fe700 1 --2- 172.21.15.67:0/2191378606 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd80006ee00 0x7fd8000712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:15.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.610+0000 7fd8077fe700 1 -- 172.21.15.67:0/2191378606 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fd80c08edf0 con 0x7fd818105bf0 2024-04-22T23:28:15.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.610+0000 7fd81659c700 1 --2- 172.21.15.67:0/2191378606 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd80006ee00 0x7fd8000712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:15.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.610+0000 7fd81659c700 1 --2- 172.21.15.67:0/2191378606 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd80006ee00 0x7fd8000712c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd808009fd0 tx=0x7fd80801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:28:15.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.614+0000 7fd8077fe700 1 -- 172.21.15.67:0/2191378606 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fd80c05b510 con 0x7fd818105bf0 2024-04-22T23:28:15.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.862+0000 7fd81ca6a700 1 -- 172.21.15.67:0/2191378606 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd8181993a0 con 0x7fd80006ee00 2024-04-22T23:28:15.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.873+0000 7fd8077fe700 1 -- 172.21.15.67:0/2191378606 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7fd8181993a0 con 0x7fd80006ee00 2024-04-22T23:28:15.873 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:28:15.873 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (119s) 14s ago 13m 17.8M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:28:15.873 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (13m) 14s ago 13m 8150k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:28:15.873 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (12m) 0s ago 12m 8132k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:28:15.873 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (13m) 14s ago 13m 6660k - 18.2.1 5be31c24972a 14085634c5d3 2024-04-22T23:28:15.873 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (12m) 0s ago 12m 6669k - 18.2.1 5be31c24972a fa995d458353 2024-04-22T23:28:15.873 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (95s) 14s ago 12m 75.2M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:28:15.873 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (7m) 14s ago 7m 62.7M - 18.2.1 5be31c24972a e24a471f0a8c 2024-04-22T23:28:15.873 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (7m) 14s ago 7m 129M - 18.2.1 5be31c24972a 0730f9fc8e61 2024-04-22T23:28:15.873 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (7m) 0s ago 7m 123M - 18.2.1 5be31c24972a b6a169586433 2024-04-22T23:28:15.873 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (7m) 0s ago 7m 68.0M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:28:15.873 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (3m) 14s ago 14m 473M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:28:15.874 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (2m) 0s ago 12m 449M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:28:15.874 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (28s) 14s ago 14m 30.3M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:28:15.874 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (3s) 0s ago 12m 28.2M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:28:15.874 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (2m) 14s ago 13m 15.6M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:28:15.874 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (2m) 0s ago 12m 15.3M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:28:15.874 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (11m) 14s ago 11m 206M 1216M 18.2.1 5be31c24972a a033247ee8c2 2024-04-22T23:28:15.874 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (11m) 14s ago 11m 211M 1216M 18.2.1 5be31c24972a 915b07a8edc4 2024-04-22T23:28:15.874 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (10m) 14s ago 10m 173M 1216M 18.2.1 5be31c24972a e257ecb4efa1 2024-04-22T23:28:15.874 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (10m) 0s ago 10m 250M 2240M 18.2.1 5be31c24972a 9c2c91a967ba 2024-04-22T23:28:15.874 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (9m) 0s ago 9m 218M 2240M 18.2.1 5be31c24972a c7586562ffa6 2024-04-22T23:28:15.874 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (9m) 0s ago 9m 195M 2240M 18.2.1 5be31c24972a 9fd97f1c65cb 2024-04-22T23:28:15.874 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (2m) 14s ago 12m 64.7M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:28:15.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.875+0000 7fd81ca6a700 1 -- 172.21.15.67:0/2191378606 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd80006ee00 msgr2=0x7fd8000712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:15.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.875+0000 7fd81ca6a700 1 --2- 172.21.15.67:0/2191378606 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd80006ee00 0x7fd8000712c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd808009fd0 tx=0x7fd80801a040 comp rx=0 tx=0).stop 2024-04-22T23:28:15.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.876+0000 7fd81ca6a700 1 -- 172.21.15.67:0/2191378606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd818105bf0 msgr2=0x7fd81819ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:15.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.876+0000 7fd81ca6a700 1 --2- 172.21.15.67:0/2191378606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd818105bf0 0x7fd81819ed00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd80c00d8d0 tx=0x7fd80c00dc90 comp rx=0 tx=0).stop 2024-04-22T23:28:15.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.876+0000 7fd81ca6a700 1 -- 172.21.15.67:0/2191378606 shutdown_connections 2024-04-22T23:28:15.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.876+0000 7fd81ca6a700 1 --2- 172.21.15.67:0/2191378606 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd80006ee00 0x7fd8000712c0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:15.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.876+0000 7fd81ca6a700 1 --2- 172.21.15.67:0/2191378606 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8180ffb40 0x7fd81819e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:15.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.876+0000 7fd81ca6a700 1 --2- 172.21.15.67:0/2191378606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd818105bf0 0x7fd81819ed00 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:15.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.876+0000 7fd81ca6a700 1 -- 172.21.15.67:0/2191378606 >> 172.21.15.67:0/2191378606 conn(0x7fd8180fb610 msgr2=0x7fd818068f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:28:15.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.876+0000 7fd81ca6a700 1 -- 172.21.15.67:0/2191378606 shutdown_connections 2024-04-22T23:28:15.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.876+0000 7fd81ca6a700 1 -- 172.21.15.67:0/2191378606 wait complete. 2024-04-22T23:28:15.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.982+0000 7fcc2361c700 1 -- 172.21.15.67:0/979864140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc1c105bd0 msgr2=0x7fcc1c105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:15.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.982+0000 7fcc2361c700 1 --2- 172.21.15.67:0/979864140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc1c105bd0 0x7fcc1c105fb0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcc18009910 tx=0x7fcc18009c20 comp rx=0 tx=0).stop 2024-04-22T23:28:15.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.982+0000 7fcc2361c700 1 -- 172.21.15.67:0/979864140 shutdown_connections 2024-04-22T23:28:15.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.982+0000 7fcc2361c700 1 --2- 172.21.15.67:0/979864140 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc1c0ffc20 0x7fcc1c100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:15.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.982+0000 7fcc2361c700 1 --2- 172.21.15.67:0/979864140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc1c105bd0 0x7fcc1c105fb0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:15.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.982+0000 7fcc2361c700 1 -- 172.21.15.67:0/979864140 >> 172.21.15.67:0/979864140 conn(0x7fcc1c075670 msgr2=0x7fcc1c075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:28:15.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.982+0000 7fcc2361c700 1 -- 172.21.15.67:0/979864140 shutdown_connections 2024-04-22T23:28:15.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.982+0000 7fcc2361c700 1 -- 172.21.15.67:0/979864140 wait complete. 2024-04-22T23:28:15.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.983+0000 7fcc2361c700 1 Processor -- start 2024-04-22T23:28:15.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.983+0000 7fcc2361c700 1 -- start start 2024-04-22T23:28:15.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.983+0000 7fcc2361c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc1c0ffc20 0x7fcc1c101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:15.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.983+0000 7fcc2361c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc1c105bd0 0x7fcc1c101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:15.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.983+0000 7fcc2361c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1c1022c0 con 0x7fcc1c0ffc20 2024-04-22T23:28:15.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.983+0000 7fcc2361c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1c102400 con 0x7fcc1c105bd0 2024-04-22T23:28:15.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.983+0000 7fcc213b8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc1c0ffc20 0x7fcc1c101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:15.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.983+0000 7fcc213b8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc1c0ffc20 0x7fcc1c101840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45268/0 (socket says 172.21.15.67:45268) 2024-04-22T23:28:15.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.983+0000 7fcc213b8700 1 -- 172.21.15.67:0/4032929867 learned_addr learned my addr 172.21.15.67:0/4032929867 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:28:15.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.983+0000 7fcc20bb7700 1 --2- 172.21.15.67:0/4032929867 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc1c105bd0 0x7fcc1c101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:15.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.983+0000 7fcc20bb7700 1 -- 172.21.15.67:0/4032929867 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc1c105bd0 msgr2=0x7fcc1c101d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-22T23:28:15.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.983+0000 7fcc20bb7700 1 -- 172.21.15.67:0/4032929867 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc1c105bd0 msgr2=0x7fcc1c101d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T23:28:15.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.983+0000 7fcc20bb7700 1 --2- 172.21.15.67:0/4032929867 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc1c105bd0 0x7fcc1c101d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T23:28:15.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.983+0000 7fcc20bb7700 1 --2- 172.21.15.67:0/4032929867 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc1c105bd0 0x7fcc1c101d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T23:28:15.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.983+0000 7fcc213b8700 1 -- 172.21.15.67:0/4032929867 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc1c105bd0 msgr2=0x7fcc1c101d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T23:28:15.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.983+0000 7fcc213b8700 1 --2- 172.21.15.67:0/4032929867 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc1c105bd0 0x7fcc1c101d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:15.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.983+0000 7fcc213b8700 1 -- 172.21.15.67:0/4032929867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc180095f0 con 0x7fcc1c0ffc20 2024-04-22T23:28:15.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.984+0000 7fcc213b8700 1 --2- 172.21.15.67:0/4032929867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc1c0ffc20 0x7fcc1c101840 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcc18006010 tx=0x7fcc1800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:28:15.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.984+0000 7fcc127fc700 1 -- 172.21.15.67:0/4032929867 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc1801d070 con 0x7fcc1c0ffc20 2024-04-22T23:28:15.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.984+0000 7fcc127fc700 1 -- 172.21.15.67:0/4032929867 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcc1800f460 con 0x7fcc1c0ffc20 2024-04-22T23:28:15.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.984+0000 7fcc2361c700 1 -- 172.21.15.67:0/4032929867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc1c102680 con 0x7fcc1c0ffc20 2024-04-22T23:28:15.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.984+0000 7fcc127fc700 1 -- 172.21.15.67:0/4032929867 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc18017630 con 0x7fcc1c0ffc20 2024-04-22T23:28:15.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.984+0000 7fcc2361c700 1 -- 172.21.15.67:0/4032929867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc1c1a2e10 con 0x7fcc1c0ffc20 2024-04-22T23:28:15.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.985+0000 7fcc2361c700 1 -- 172.21.15.67:0/4032929867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc1c04f2e0 con 0x7fcc1c0ffc20 2024-04-22T23:28:15.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.986+0000 7fcc127fc700 1 -- 172.21.15.67:0/4032929867 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 39) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fcc1800f5d0 con 0x7fcc1c0ffc20 2024-04-22T23:28:15.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.986+0000 7fcc127fc700 1 --2- 172.21.15.67:0/4032929867 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcc0806ee00 0x7fcc080712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:15.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.986+0000 7fcc127fc700 1 -- 172.21.15.67:0/4032929867 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fcc18090520 con 0x7fcc1c0ffc20 2024-04-22T23:28:15.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.986+0000 7fcc20bb7700 1 --2- 172.21.15.67:0/4032929867 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcc0806ee00 0x7fcc080712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:15.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.987+0000 7fcc20bb7700 1 --2- 172.21.15.67:0/4032929867 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcc0806ee00 0x7fcc080712c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c005950 tx=0x7fcc0c0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:28:15.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:15.991+0000 7fcc127fc700 1 -- 172.21.15.67:0/4032929867 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fcc18059110 con 0x7fcc1c0ffc20 2024-04-22T23:28:16.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:15 smithi067 ceph-mon[112438]: pgmap v12: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 20 KiB/s rd, 0 B/s wr, 14 op/s 2024-04-22T23:28:16.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:15 smithi134 ceph-mon[85630]: pgmap v12: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 20 KiB/s rd, 0 B/s wr, 14 op/s 2024-04-22T23:28:16.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.295+0000 7fcc2361c700 1 -- 172.21.15.67:0/4032929867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fcc1c1a30b0 con 0x7fcc1c0ffc20 2024-04-22T23:28:16.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.295+0000 7fcc127fc700 1 -- 172.21.15.67:0/4032929867 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+688 (secure 0 0 0) 0x7fcc1805c7d0 con 0x7fcc1c0ffc20 2024-04-22T23:28:16.294 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:28:16.294 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:28:16.295 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:28:16.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:28:16.295 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:28:16.295 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:28:16.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:28:16.295 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:28:16.295 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-22T23:28:16.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:28:16.295 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:28:16.295 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-22T23:28:16.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:28:16.295 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:28:16.295 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 10, 2024-04-22T23:28:16.295 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 4 2024-04-22T23:28:16.296 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:28:16.296 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:28:16.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.297+0000 7fcc2361c700 1 -- 172.21.15.67:0/4032929867 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcc0806ee00 msgr2=0x7fcc080712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:16.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.297+0000 7fcc2361c700 1 --2- 172.21.15.67:0/4032929867 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcc0806ee00 0x7fcc080712c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c005950 tx=0x7fcc0c0058e0 comp rx=0 tx=0).stop 2024-04-22T23:28:16.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.297+0000 7fcc2361c700 1 -- 172.21.15.67:0/4032929867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc1c0ffc20 msgr2=0x7fcc1c101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:16.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.297+0000 7fcc2361c700 1 --2- 172.21.15.67:0/4032929867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc1c0ffc20 0x7fcc1c101840 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcc18006010 tx=0x7fcc1800be20 comp rx=0 tx=0).stop 2024-04-22T23:28:16.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.297+0000 7fcc2361c700 1 -- 172.21.15.67:0/4032929867 shutdown_connections 2024-04-22T23:28:16.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.297+0000 7fcc2361c700 1 --2- 172.21.15.67:0/4032929867 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcc0806ee00 0x7fcc080712c0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:16.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.297+0000 7fcc2361c700 1 --2- 172.21.15.67:0/4032929867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc1c0ffc20 0x7fcc1c101840 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:16.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.297+0000 7fcc2361c700 1 --2- 172.21.15.67:0/4032929867 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc1c105bd0 0x7fcc1c101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:16.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.298+0000 7fcc2361c700 1 -- 172.21.15.67:0/4032929867 >> 172.21.15.67:0/4032929867 conn(0x7fcc1c075670 msgr2=0x7fcc1c0686a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:28:16.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.298+0000 7fcc2361c700 1 -- 172.21.15.67:0/4032929867 shutdown_connections 2024-04-22T23:28:16.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.298+0000 7fcc2361c700 1 -- 172.21.15.67:0/4032929867 wait complete. 2024-04-22T23:28:16.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.417+0000 7fbfb83f7700 1 -- 172.21.15.67:0/1557780595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfb0100750 msgr2=0x7fbfb01047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:16.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.417+0000 7fbfb83f7700 1 --2- 172.21.15.67:0/1557780595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfb0100750 0x7fbfb01047a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbfac009910 tx=0x7fbfac009c20 comp rx=0 tx=0).stop 2024-04-22T23:28:16.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.417+0000 7fbfb83f7700 1 -- 172.21.15.67:0/1557780595 shutdown_connections 2024-04-22T23:28:16.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.417+0000 7fbfb83f7700 1 --2- 172.21.15.67:0/1557780595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfb0100750 0x7fbfb01047a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:16.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.417+0000 7fbfb83f7700 1 --2- 172.21.15.67:0/1557780595 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbfb00ffda0 0x7fbfb0100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:16.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.417+0000 7fbfb83f7700 1 -- 172.21.15.67:0/1557780595 >> 172.21.15.67:0/1557780595 conn(0x7fbfb00fb650 msgr2=0x7fbfb00fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:28:16.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.417+0000 7fbfb83f7700 1 -- 172.21.15.67:0/1557780595 shutdown_connections 2024-04-22T23:28:16.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.417+0000 7fbfb83f7700 1 -- 172.21.15.67:0/1557780595 wait complete. 2024-04-22T23:28:16.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.418+0000 7fbfb83f7700 1 Processor -- start 2024-04-22T23:28:16.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.418+0000 7fbfb83f7700 1 -- start start 2024-04-22T23:28:16.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.418+0000 7fbfb83f7700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbfb00ffda0 0x7fbfb019b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:16.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.418+0000 7fbfb83f7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfb0100750 0x7fbfb019bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:16.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.418+0000 7fbfb83f7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb019c1f0 con 0x7fbfb0100750 2024-04-22T23:28:16.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.418+0000 7fbfb83f7700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb01956a0 con 0x7fbfb00ffda0 2024-04-22T23:28:16.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.418+0000 7fbfb5992700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfb0100750 0x7fbfb019bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:16.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.418+0000 7fbfb5992700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfb0100750 0x7fbfb019bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45294/0 (socket says 172.21.15.67:45294) 2024-04-22T23:28:16.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.418+0000 7fbfb5992700 1 -- 172.21.15.67:0/1847938958 learned_addr learned my addr 172.21.15.67:0/1847938958 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:28:16.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.418+0000 7fbfb6193700 1 --2- 172.21.15.67:0/1847938958 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbfb00ffda0 0x7fbfb019b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:16.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.418+0000 7fbfb6193700 1 -- 172.21.15.67:0/1847938958 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbfb00ffda0 msgr2=0x7fbfb019b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-22T23:28:16.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.418+0000 7fbfb6193700 1 -- 172.21.15.67:0/1847938958 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbfb00ffda0 msgr2=0x7fbfb019b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T23:28:16.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.418+0000 7fbfb6193700 1 --2- 172.21.15.67:0/1847938958 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbfb00ffda0 0x7fbfb019b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T23:28:16.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.418+0000 7fbfb6193700 1 --2- 172.21.15.67:0/1847938958 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbfb00ffda0 0x7fbfb019b620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T23:28:16.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.418+0000 7fbfb5992700 1 -- 172.21.15.67:0/1847938958 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbfb00ffda0 msgr2=0x7fbfb019b620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T23:28:16.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.418+0000 7fbfb5992700 1 --2- 172.21.15.67:0/1847938958 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbfb00ffda0 0x7fbfb019b620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:16.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.418+0000 7fbfb5992700 1 -- 172.21.15.67:0/1847938958 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfac0095f0 con 0x7fbfb0100750 2024-04-22T23:28:16.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.419+0000 7fbfb5992700 1 --2- 172.21.15.67:0/1847938958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfb0100750 0x7fbfb019bb60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbfac00bd70 tx=0x7fbfac00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:28:16.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.419+0000 7fbfa77fe700 1 -- 172.21.15.67:0/1847938958 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbfac01d070 con 0x7fbfb0100750 2024-04-22T23:28:16.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.419+0000 7fbfb83f7700 1 -- 172.21.15.67:0/1847938958 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfb0195920 con 0x7fbfb0100750 2024-04-22T23:28:16.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.419+0000 7fbfa77fe700 1 -- 172.21.15.67:0/1847938958 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbfac005dc0 con 0x7fbfb0100750 2024-04-22T23:28:16.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.419+0000 7fbfa77fe700 1 -- 172.21.15.67:0/1847938958 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbfac017400 con 0x7fbfb0100750 2024-04-22T23:28:16.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.419+0000 7fbfb83f7700 1 -- 172.21.15.67:0/1847938958 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfb0195e10 con 0x7fbfb0100750 2024-04-22T23:28:16.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.420+0000 7fbfb83f7700 1 -- 172.21.15.67:0/1847938958 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfb01081a0 con 0x7fbfb0100750 2024-04-22T23:28:16.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.420+0000 7fbfa77fe700 1 -- 172.21.15.67:0/1847938958 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 39) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fbfac00f7f0 con 0x7fbfb0100750 2024-04-22T23:28:16.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.421+0000 7fbfa77fe700 1 --2- 172.21.15.67:0/1847938958 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbf9c06edf0 0x7fbf9c0712b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:16.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.421+0000 7fbfa77fe700 1 -- 172.21.15.67:0/1847938958 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fbfac090590 con 0x7fbfb0100750 2024-04-22T23:28:16.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.421+0000 7fbfb6193700 1 --2- 172.21.15.67:0/1847938958 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbf9c06edf0 0x7fbf9c0712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:16.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.421+0000 7fbfb6193700 1 --2- 172.21.15.67:0/1847938958 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbf9c06edf0 0x7fbf9c0712b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0005920 tx=0x7fbfa000b2c0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:28:16.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.426+0000 7fbfa77fe700 1 -- 172.21.15.67:0/1847938958 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fbfac05ccb0 con 0x7fbfb0100750 2024-04-22T23:28:16.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.683+0000 7fbfb83f7700 1 -- 172.21.15.67:0/1847938958 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fbfb004f2e0 con 0x7fbfb0100750 2024-04-22T23:28:16.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.684+0000 7fbfa77fe700 1 -- 172.21.15.67:0/1847938958 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1901 (secure 0 0 0) 0x7fbfac05c840 con 0x7fbfb0100750 2024-04-22T23:28:16.683 INFO:teuthology.orchestra.run.smithi067.stdout:e22 2024-04-22T23:28:16.683 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:28:16.683 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:28:16.683 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:28:16.684 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:28:16.684 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:28:16.684 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:28:16.684 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 22 2024-04-22T23:28:16.684 INFO:teuthology.orchestra.run.smithi067.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T23:28:16.684 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:28:16.684 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:27:20.577198+0000 2024-04-22T23:28:16.684 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:28:16.684 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:28:16.684 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:28:16.684 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:28:16.684 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:28:16.684 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:28:16.684 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:28:16.684 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:28:16.685 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 46 2024-04-22T23:28:16.685 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:28:16.685 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:28:16.685 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:28:16.685 INFO:teuthology.orchestra.run.smithi067.stdout:up {0=14484,1=24273} 2024-04-22T23:28:16.685 INFO:teuthology.orchestra.run.smithi067.stdout:failed 2024-04-22T23:28:16.685 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:28:16.685 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:28:16.685 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:28:16.685 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:28:16.685 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:28:16.686 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:28:16.686 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:28:16.686 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:28:16.686 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:28:16.686 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{0:14484} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.67:6828/3606151860,v1:172.21.15.67:6829/3606151860] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:28:16.686 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{0:14496} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/3305304988,v1:172.21.15.67:6827/3305304988] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:28:16.686 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{1:24273} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.134:6824/535957142,v1:172.21.15.134:6825/535957142] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:28:16.686 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{1:24289} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.134:6826/1954821676,v1:172.21.15.134:6827/1954821676] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:28:16.686 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:28:16.686 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:28:16.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.686+0000 7fbfb83f7700 1 -- 172.21.15.67:0/1847938958 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbf9c06edf0 msgr2=0x7fbf9c0712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:16.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.686+0000 7fbfb83f7700 1 --2- 172.21.15.67:0/1847938958 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbf9c06edf0 0x7fbf9c0712b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0005920 tx=0x7fbfa000b2c0 comp rx=0 tx=0).stop 2024-04-22T23:28:16.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.686+0000 7fbfb83f7700 1 -- 172.21.15.67:0/1847938958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfb0100750 msgr2=0x7fbfb019bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:16.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.686+0000 7fbfb83f7700 1 --2- 172.21.15.67:0/1847938958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfb0100750 0x7fbfb019bb60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbfac00bd70 tx=0x7fbfac00be50 comp rx=0 tx=0).stop 2024-04-22T23:28:16.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.686+0000 7fbfb83f7700 1 -- 172.21.15.67:0/1847938958 shutdown_connections 2024-04-22T23:28:16.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.686+0000 7fbfb83f7700 1 --2- 172.21.15.67:0/1847938958 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbf9c06edf0 0x7fbf9c0712b0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:16.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.686+0000 7fbfb83f7700 1 --2- 172.21.15.67:0/1847938958 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbfb00ffda0 0x7fbfb019b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:16.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.686+0000 7fbfb83f7700 1 --2- 172.21.15.67:0/1847938958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfb0100750 0x7fbfb019bb60 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:16.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.686+0000 7fbfb83f7700 1 -- 172.21.15.67:0/1847938958 >> 172.21.15.67:0/1847938958 conn(0x7fbfb00fb650 msgr2=0x7fbfb00fccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:28:16.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.686+0000 7fbfb83f7700 1 -- 172.21.15.67:0/1847938958 shutdown_connections 2024-04-22T23:28:16.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.687+0000 7fbfb83f7700 1 -- 172.21.15.67:0/1847938958 wait complete. 2024-04-22T23:28:16.688 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 22 2024-04-22T23:28:16.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.794+0000 7faaaa2f1700 1 -- 172.21.15.67:0/3113083844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faaa40ffc20 msgr2=0x7faaa4100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:16.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.794+0000 7faaaa2f1700 1 --2- 172.21.15.67:0/3113083844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faaa40ffc20 0x7faaa4100080 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7faa8c009910 tx=0x7faa8c009c20 comp rx=0 tx=0).stop 2024-04-22T23:28:16.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.794+0000 7faaaa2f1700 1 -- 172.21.15.67:0/3113083844 shutdown_connections 2024-04-22T23:28:16.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.794+0000 7faaaa2f1700 1 --2- 172.21.15.67:0/3113083844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faaa40ffc20 0x7faaa4100080 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:16.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.794+0000 7faaaa2f1700 1 --2- 172.21.15.67:0/3113083844 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faaa4105bd0 0x7faaa4105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:16.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.794+0000 7faaaa2f1700 1 -- 172.21.15.67:0/3113083844 >> 172.21.15.67:0/3113083844 conn(0x7faaa4075670 msgr2=0x7faaa4075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:28:16.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.794+0000 7faaaa2f1700 1 -- 172.21.15.67:0/3113083844 shutdown_connections 2024-04-22T23:28:16.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.794+0000 7faaaa2f1700 1 -- 172.21.15.67:0/3113083844 wait complete. 2024-04-22T23:28:16.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.795+0000 7faaaa2f1700 1 Processor -- start 2024-04-22T23:28:16.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.795+0000 7faaaa2f1700 1 -- start start 2024-04-22T23:28:16.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.796+0000 7faaaa2f1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faaa40ffc20 0x7faaa419b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:16.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.796+0000 7faaaa2f1700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faaa4105bd0 0x7faaa419ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:16.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.796+0000 7faaaa2f1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa419c0f0 con 0x7faaa40ffc20 2024-04-22T23:28:16.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.796+0000 7faaaa2f1700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa41955a0 con 0x7faaa4105bd0 2024-04-22T23:28:16.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.796+0000 7faaa3fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faaa40ffc20 0x7faaa419b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:16.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.796+0000 7faaa3fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faaa40ffc20 0x7faaa419b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45318/0 (socket says 172.21.15.67:45318) 2024-04-22T23:28:16.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.796+0000 7faaa3fff700 1 -- 172.21.15.67:0/3552182803 learned_addr learned my addr 172.21.15.67:0/3552182803 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:28:16.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.796+0000 7faaa37fe700 1 --2- 172.21.15.67:0/3552182803 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faaa4105bd0 0x7faaa419ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:16.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.796+0000 7faaa37fe700 1 -- 172.21.15.67:0/3552182803 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faaa4105bd0 msgr2=0x7faaa419ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-22T23:28:16.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.796+0000 7faaa37fe700 1 -- 172.21.15.67:0/3552182803 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faaa4105bd0 msgr2=0x7faaa419ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T23:28:16.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.796+0000 7faaa37fe700 1 --2- 172.21.15.67:0/3552182803 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faaa4105bd0 0x7faaa419ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T23:28:16.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.796+0000 7faaa37fe700 1 --2- 172.21.15.67:0/3552182803 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faaa4105bd0 0x7faaa419ba60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T23:28:16.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.796+0000 7faaa3fff700 1 -- 172.21.15.67:0/3552182803 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faaa4105bd0 msgr2=0x7faaa419ba60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T23:28:16.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.796+0000 7faaa3fff700 1 --2- 172.21.15.67:0/3552182803 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faaa4105bd0 0x7faaa419ba60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:16.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.796+0000 7faaa3fff700 1 -- 172.21.15.67:0/3552182803 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa8c0095f0 con 0x7faaa40ffc20 2024-04-22T23:28:16.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.796+0000 7faaa3fff700 1 --2- 172.21.15.67:0/3552182803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faaa40ffc20 0x7faaa419b520 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7faa9400d8d0 tx=0x7faa9400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:28:16.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.796+0000 7faaa17fa700 1 -- 172.21.15.67:0/3552182803 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa94004a00 con 0x7faaa40ffc20 2024-04-22T23:28:16.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.796+0000 7faaaa2f1700 1 -- 172.21.15.67:0/3552182803 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaa4195880 con 0x7faaa40ffc20 2024-04-22T23:28:16.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.797+0000 7faaaa2f1700 1 -- 172.21.15.67:0/3552182803 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaa4195dd0 con 0x7faaa40ffc20 2024-04-22T23:28:16.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.797+0000 7faaa17fa700 1 -- 172.21.15.67:0/3552182803 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faa940054e0 con 0x7faaa40ffc20 2024-04-22T23:28:16.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.797+0000 7faaa17fa700 1 -- 172.21.15.67:0/3552182803 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa94009d60 con 0x7faaa40ffc20 2024-04-22T23:28:16.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.798+0000 7faaaa2f1700 1 -- 172.21.15.67:0/3552182803 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaa41080d0 con 0x7faaa40ffc20 2024-04-22T23:28:16.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.801+0000 7faaa17fa700 1 -- 172.21.15.67:0/3552182803 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 39) v1 ==== 92847+0+0 (secure 0 0 0) 0x7faa94004f50 con 0x7faaa40ffc20 2024-04-22T23:28:16.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.801+0000 7faaa17fa700 1 --2- 172.21.15.67:0/3552182803 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faa9006ee00 0x7faa900712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:16.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.801+0000 7faaa17fa700 1 -- 172.21.15.67:0/3552182803 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6414+0+0 (secure 0 0 0) 0x7faa9408f0a0 con 0x7faaa40ffc20 2024-04-22T23:28:16.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.801+0000 7faaa37fe700 1 --2- 172.21.15.67:0/3552182803 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faa9006ee00 0x7faa900712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:16.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.802+0000 7faaa37fe700 1 --2- 172.21.15.67:0/3552182803 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faa9006ee00 0x7faa900712c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7faa8c005b40 tx=0x7faa8c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:28:16.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:16.805+0000 7faaa17fa700 1 -- 172.21.15.67:0/3552182803 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7faa9405b8e0 con 0x7faaa40ffc20 2024-04-22T23:28:17.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.036+0000 7faaaa2f1700 1 -- 172.21.15.67:0/3552182803 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7faaa4196040 con 0x7faa9006ee00 2024-04-22T23:28:17.035 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:16 smithi067 ceph-mon[112438]: from='client.34132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:28:17.035 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:16 smithi067 ceph-mon[112438]: from='client.34136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:28:17.035 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:16 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:17.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:16 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:17.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:16 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/4032929867' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:28:17.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:16 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1847938958' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:28:17.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.037+0000 7faaa17fa700 1 -- 172.21.15.67:0/3552182803 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7faaa4196040 con 0x7faa9006ee00 2024-04-22T23:28:17.036 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:28:17.036 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:28:17.037 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:28:17.037 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:28:17.037 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:28:17.037 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:28:17.037 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr" 2024-04-22T23:28:17.037 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:28:17.037 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "4/23 daemons upgraded", 2024-04-22T23:28:17.037 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mon daemons", 2024-04-22T23:28:17.037 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:28:17.037 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:28:17.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.040+0000 7faaaa2f1700 1 -- 172.21.15.67:0/3552182803 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faa9006ee00 msgr2=0x7faa900712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:17.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.040+0000 7faaaa2f1700 1 --2- 172.21.15.67:0/3552182803 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faa9006ee00 0x7faa900712c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7faa8c005b40 tx=0x7faa8c01a040 comp rx=0 tx=0).stop 2024-04-22T23:28:17.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.040+0000 7faaaa2f1700 1 -- 172.21.15.67:0/3552182803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faaa40ffc20 msgr2=0x7faaa419b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:17.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.040+0000 7faaaa2f1700 1 --2- 172.21.15.67:0/3552182803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faaa40ffc20 0x7faaa419b520 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7faa9400d8d0 tx=0x7faa9400dc90 comp rx=0 tx=0).stop 2024-04-22T23:28:17.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.041+0000 7faaaa2f1700 1 -- 172.21.15.67:0/3552182803 shutdown_connections 2024-04-22T23:28:17.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.041+0000 7faaaa2f1700 1 --2- 172.21.15.67:0/3552182803 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faa9006ee00 0x7faa900712c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:17.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.041+0000 7faaaa2f1700 1 --2- 172.21.15.67:0/3552182803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faaa40ffc20 0x7faaa419b520 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:17.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.041+0000 7faaaa2f1700 1 --2- 172.21.15.67:0/3552182803 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faaa4105bd0 0x7faaa419ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:17.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.041+0000 7faaaa2f1700 1 -- 172.21.15.67:0/3552182803 >> 172.21.15.67:0/3552182803 conn(0x7faaa4075670 msgr2=0x7faaa40feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:28:17.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.041+0000 7faaaa2f1700 1 -- 172.21.15.67:0/3552182803 shutdown_connections 2024-04-22T23:28:17.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.041+0000 7faaaa2f1700 1 -- 172.21.15.67:0/3552182803 wait complete. 2024-04-22T23:28:17.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.148+0000 7fd8a5689700 1 -- 172.21.15.67:0/1385647313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8a0102700 msgr2=0x7fd8a010abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:17.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.148+0000 7fd8a5689700 1 --2- 172.21.15.67:0/1385647313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8a0102700 0x7fd8a010abf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd894009910 tx=0x7fd894009c20 comp rx=0 tx=0).stop 2024-04-22T23:28:17.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.148+0000 7fd8a5689700 1 -- 172.21.15.67:0/1385647313 shutdown_connections 2024-04-22T23:28:17.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.148+0000 7fd8a5689700 1 --2- 172.21.15.67:0/1385647313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8a0102700 0x7fd8a010abf0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:17.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.148+0000 7fd8a5689700 1 --2- 172.21.15.67:0/1385647313 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8a0101de0 0x7fd8a01021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:17.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.148+0000 7fd8a5689700 1 -- 172.21.15.67:0/1385647313 >> 172.21.15.67:0/1385647313 conn(0x7fd8a00fb650 msgr2=0x7fd8a00fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:28:17.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.148+0000 7fd8a5689700 1 -- 172.21.15.67:0/1385647313 shutdown_connections 2024-04-22T23:28:17.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.148+0000 7fd8a5689700 1 -- 172.21.15.67:0/1385647313 wait complete. 2024-04-22T23:28:17.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.148+0000 7fd8a5689700 1 Processor -- start 2024-04-22T23:28:17.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.149+0000 7fd8a5689700 1 -- start start 2024-04-22T23:28:17.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.149+0000 7fd8a5689700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8a0101de0 0x7fd8a019c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:17.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.149+0000 7fd8a5689700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8a0102700 0x7fd8a019cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:17.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.149+0000 7fd8a5689700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a019d2d0 con 0x7fd8a0102700 2024-04-22T23:28:17.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.149+0000 7fd8a5689700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a0196780 con 0x7fd8a0101de0 2024-04-22T23:28:17.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.149+0000 7fd89e7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8a0102700 0x7fd8a019cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:17.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.149+0000 7fd89e7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8a0102700 0x7fd8a019cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45332/0 (socket says 172.21.15.67:45332) 2024-04-22T23:28:17.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.149+0000 7fd89e7fc700 1 -- 172.21.15.67:0/4099698505 learned_addr learned my addr 172.21.15.67:0/4099698505 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:28:17.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.149+0000 7fd89effd700 1 --2- 172.21.15.67:0/4099698505 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8a0101de0 0x7fd8a019c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:17.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.149+0000 7fd89effd700 1 -- 172.21.15.67:0/4099698505 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8a0101de0 msgr2=0x7fd8a019c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-22T23:28:17.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.149+0000 7fd89effd700 1 -- 172.21.15.67:0/4099698505 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8a0101de0 msgr2=0x7fd8a019c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T23:28:17.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.149+0000 7fd89effd700 1 --2- 172.21.15.67:0/4099698505 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8a0101de0 0x7fd8a019c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T23:28:17.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.149+0000 7fd89effd700 1 --2- 172.21.15.67:0/4099698505 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8a0101de0 0x7fd8a019c700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T23:28:17.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.149+0000 7fd89e7fc700 1 -- 172.21.15.67:0/4099698505 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8a0101de0 msgr2=0x7fd8a019c700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T23:28:17.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.149+0000 7fd89e7fc700 1 --2- 172.21.15.67:0/4099698505 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8a0101de0 0x7fd8a019c700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:17.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.149+0000 7fd89e7fc700 1 -- 172.21.15.67:0/4099698505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8940095f0 con 0x7fd8a0102700 2024-04-22T23:28:17.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.149+0000 7fd89e7fc700 1 --2- 172.21.15.67:0/4099698505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8a0102700 0x7fd8a019cc40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd8940092e0 tx=0x7fd89400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:28:17.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.150+0000 7fd887fff700 1 -- 172.21.15.67:0/4099698505 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd89401d070 con 0x7fd8a0102700 2024-04-22T23:28:17.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.150+0000 7fd887fff700 1 -- 172.21.15.67:0/4099698505 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd894005dc0 con 0x7fd8a0102700 2024-04-22T23:28:17.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.150+0000 7fd887fff700 1 -- 172.21.15.67:0/4099698505 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd894017400 con 0x7fd8a0102700 2024-04-22T23:28:17.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.150+0000 7fd8a5689700 1 -- 172.21.15.67:0/4099698505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8a0196a00 con 0x7fd8a0102700 2024-04-22T23:28:17.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.150+0000 7fd8a5689700 1 -- 172.21.15.67:0/4099698505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8a0196ef0 con 0x7fd8a0102700 2024-04-22T23:28:17.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.151+0000 7fd8a5689700 1 -- 172.21.15.67:0/4099698505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8a01082f0 con 0x7fd8a0102700 2024-04-22T23:28:17.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.151+0000 7fd887fff700 1 -- 172.21.15.67:0/4099698505 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 39) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fd89400f6f0 con 0x7fd8a0102700 2024-04-22T23:28:17.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.152+0000 7fd887fff700 1 --2- 172.21.15.67:0/4099698505 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd88806ee00 0x7fd8880712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:17.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.152+0000 7fd887fff700 1 -- 172.21.15.67:0/4099698505 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fd894090390 con 0x7fd8a0102700 2024-04-22T23:28:17.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.152+0000 7fd89effd700 1 --2- 172.21.15.67:0/4099698505 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd88806ee00 0x7fd8880712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:17.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.152+0000 7fd89effd700 1 --2- 172.21.15.67:0/4099698505 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd88806ee00 0x7fd8880712c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd890005950 tx=0x7fd8900058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:28:17.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.156+0000 7fd887fff700 1 -- 172.21.15.67:0/4099698505 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fd89405cb30 con 0x7fd8a0102700 2024-04-22T23:28:17.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:16 smithi134 ceph-mon[85630]: from='client.34132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:28:17.243 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:16 smithi134 ceph-mon[85630]: from='client.34136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:28:17.243 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:16 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:17.243 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:16 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:17.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:16 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/4032929867' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:28:17.244 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:16 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1847938958' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:28:17.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.453+0000 7fd8a5689700 1 -- 172.21.15.67:0/4099698505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fd8a004f2e0 con 0x7fd8a0102700 2024-04-22T23:28:17.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.453+0000 7fd887fff700 1 -- 172.21.15.67:0/4099698505 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7fd89405c6c0 con 0x7fd8a0102700 2024-04-22T23:28:17.452 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T23:28:17.453 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:28:17.453 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:28:17.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.455+0000 7fd8a5689700 1 -- 172.21.15.67:0/4099698505 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd88806ee00 msgr2=0x7fd8880712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:17.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.455+0000 7fd8a5689700 1 --2- 172.21.15.67:0/4099698505 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd88806ee00 0x7fd8880712c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd890005950 tx=0x7fd8900058e0 comp rx=0 tx=0).stop 2024-04-22T23:28:17.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.455+0000 7fd8a5689700 1 -- 172.21.15.67:0/4099698505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8a0102700 msgr2=0x7fd8a019cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:17.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.455+0000 7fd8a5689700 1 --2- 172.21.15.67:0/4099698505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8a0102700 0x7fd8a019cc40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd8940092e0 tx=0x7fd89400be20 comp rx=0 tx=0).stop 2024-04-22T23:28:17.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.455+0000 7fd8a5689700 1 -- 172.21.15.67:0/4099698505 shutdown_connections 2024-04-22T23:28:17.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.455+0000 7fd8a5689700 1 --2- 172.21.15.67:0/4099698505 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd88806ee00 0x7fd8880712c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:17.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.456+0000 7fd8a5689700 1 --2- 172.21.15.67:0/4099698505 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8a0101de0 0x7fd8a019c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:17.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.456+0000 7fd8a5689700 1 --2- 172.21.15.67:0/4099698505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8a0102700 0x7fd8a019cc40 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:17.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.456+0000 7fd8a5689700 1 -- 172.21.15.67:0/4099698505 >> 172.21.15.67:0/4099698505 conn(0x7fd8a00fb650 msgr2=0x7fd8a0105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:28:17.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.456+0000 7fd8a5689700 1 -- 172.21.15.67:0/4099698505 shutdown_connections 2024-04-22T23:28:17.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:17.456+0000 7fd8a5689700 1 -- 172.21.15.67:0/4099698505 wait complete. 2024-04-22T23:28:18.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:17 smithi067 ceph-mon[112438]: from='client.34140 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:28:18.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:17 smithi067 ceph-mon[112438]: pgmap v13: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:28:18.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:17 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:18.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:17 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:18.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:17 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/4099698505' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:28:18.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:17 smithi134 ceph-mon[85630]: from='client.34140 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:28:18.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:17 smithi134 ceph-mon[85630]: pgmap v13: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:28:18.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:17 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:18.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:17 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:18.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:17 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/4099698505' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:28:19.126 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:18 smithi134 ceph-mon[85630]: from='client.34152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:28:19.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:18 smithi067 ceph-mon[112438]: from='client.34152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:28:20.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:19 smithi134 ceph-mon[85630]: pgmap v14: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:28:20.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:19 smithi067 ceph-mon[112438]: pgmap v14: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:28:21.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:21 smithi134 ceph-mon[85630]: pgmap v15: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:28:21.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:21 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:21.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:21 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:21.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:21 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:21.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:21 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:28:21.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:21 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:21.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:21 smithi134 ceph-mon[85630]: Reconfiguring mon.smithi067 (monmap changed)... 2024-04-22T23:28:21.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:21 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T23:28:21.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:21 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T23:28:21.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:21 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:21.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:21 smithi134 ceph-mon[85630]: Reconfiguring daemon mon.smithi067 on smithi067 2024-04-22T23:28:21.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:21 smithi067 ceph-mon[112438]: pgmap v15: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:28:21.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:21 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:21.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:21 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:21.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:21 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:21.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:21 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:28:21.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:21 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:21.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:21 smithi067 ceph-mon[112438]: Reconfiguring mon.smithi067 (monmap changed)... 2024-04-22T23:28:21.794 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:21 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T23:28:21.794 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:21 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T23:28:21.794 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:21 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:21.794 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:21 smithi067 ceph-mon[112438]: Reconfiguring daemon mon.smithi067 on smithi067 2024-04-22T23:28:23.393 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:23 smithi067 ceph-mon[112438]: pgmap v16: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:28:23.393 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:23 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:23.393 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:23 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:23.393 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:23 smithi067 ceph-mon[112438]: Reconfiguring mgr.smithi067.whdcsr (monmap changed)... 2024-04-22T23:28:23.393 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:23 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi067.whdcsr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T23:28:23.393 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:23 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T23:28:23.393 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:23 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:23.393 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:23 smithi067 ceph-mon[112438]: Reconfiguring daemon mgr.smithi067.whdcsr on smithi067 2024-04-22T23:28:23.492 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:23 smithi134 ceph-mon[85630]: pgmap v16: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:28:23.492 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:23 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:23.492 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:23 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:23.492 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:23 smithi134 ceph-mon[85630]: Reconfiguring mgr.smithi067.whdcsr (monmap changed)... 2024-04-22T23:28:23.492 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:23 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi067.whdcsr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T23:28:23.492 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:23 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T23:28:23.492 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:23 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:23.493 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:23 smithi134 ceph-mon[85630]: Reconfiguring daemon mgr.smithi067.whdcsr on smithi067 2024-04-22T23:28:25.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:24 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:25.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:24 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:25.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:24 smithi067 ceph-mon[112438]: Reconfiguring ceph-exporter.smithi067 (monmap changed)... 2024-04-22T23:28:25.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:24 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi067", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T23:28:25.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:24 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi067", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T23:28:25.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:24 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi067", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-22T23:28:25.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:24 smithi067 ceph-mon[112438]: Unable to update caps for client.ceph-exporter.smithi067 2024-04-22T23:28:25.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:24 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.ceph-exporter.smithi067"}]: dispatch 2024-04-22T23:28:25.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:24 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:25.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:24 smithi067 ceph-mon[112438]: Reconfiguring daemon ceph-exporter.smithi067 on smithi067 2024-04-22T23:28:25.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:24 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:25.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:24 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:25.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:24 smithi134 ceph-mon[85630]: Reconfiguring ceph-exporter.smithi067 (monmap changed)... 2024-04-22T23:28:25.243 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:24 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi067", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T23:28:25.243 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:24 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi067", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T23:28:25.243 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:24 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi067", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-22T23:28:25.243 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:24 smithi134 ceph-mon[85630]: Unable to update caps for client.ceph-exporter.smithi067 2024-04-22T23:28:25.243 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:24 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.ceph-exporter.smithi067"}]: dispatch 2024-04-22T23:28:25.243 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:24 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:25.243 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:24 smithi134 ceph-mon[85630]: Reconfiguring daemon ceph-exporter.smithi067 on smithi067 2024-04-22T23:28:26.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:25 smithi067 ceph-mon[112438]: pgmap v17: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:28:26.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:25 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:26.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:25 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:26.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:25 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi067", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T23:28:26.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:25 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:26.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:25 smithi134 ceph-mon[85630]: pgmap v17: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:28:26.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:25 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:26.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:25 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:26.243 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:25 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi067", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T23:28:26.243 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:25 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:26.932 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:26 smithi067 ceph-mon[112438]: Reconfiguring crash.smithi067 (monmap changed)... 2024-04-22T23:28:26.932 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:26 smithi067 ceph-mon[112438]: Reconfiguring daemon crash.smithi067 on smithi067 2024-04-22T23:28:26.932 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:28:27.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:26 smithi134 ceph-mon[85630]: Reconfiguring crash.smithi067 (monmap changed)... 2024-04-22T23:28:27.243 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:26 smithi134 ceph-mon[85630]: Reconfiguring daemon crash.smithi067 on smithi067 2024-04-22T23:28:27.243 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:28:27.836 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:27 smithi067 ceph-mon[112438]: pgmap v18: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:28:27.836 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:27 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:27.836 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:27 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:27.836 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:27 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T23:28:27.836 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:27 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:28.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:27 smithi134 ceph-mon[85630]: pgmap v18: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:28:28.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:27 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:28.243 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:27 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:28.243 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:27 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T23:28:28.243 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:27 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:29.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:28 smithi067 ceph-mon[112438]: Reconfiguring osd.0 (monmap changed)... 2024-04-22T23:28:29.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:28 smithi067 ceph-mon[112438]: Reconfiguring daemon osd.0 on smithi067 2024-04-22T23:28:29.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:28 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:29.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:28 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:29.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:28 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T23:28:29.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:28 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:29.232 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:28 smithi134 ceph-mon[85630]: Reconfiguring osd.0 (monmap changed)... 2024-04-22T23:28:29.232 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:28 smithi134 ceph-mon[85630]: Reconfiguring daemon osd.0 on smithi067 2024-04-22T23:28:29.232 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:28 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:29.232 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:28 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:29.232 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:28 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T23:28:29.232 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:28 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:30.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:29 smithi067 ceph-mon[112438]: pgmap v19: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:28:30.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:29 smithi067 ceph-mon[112438]: Reconfiguring osd.1 (monmap changed)... 2024-04-22T23:28:30.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:29 smithi067 ceph-mon[112438]: Reconfiguring daemon osd.1 on smithi067 2024-04-22T23:28:30.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:29 smithi134 ceph-mon[85630]: pgmap v19: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:28:30.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:29 smithi134 ceph-mon[85630]: Reconfiguring osd.1 (monmap changed)... 2024-04-22T23:28:30.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:29 smithi134 ceph-mon[85630]: Reconfiguring daemon osd.1 on smithi067 2024-04-22T23:28:31.543 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:31 smithi067 ceph-mon[112438]: pgmap v20: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:28:31.543 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:31 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:31.543 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:31 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:31.543 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:31 smithi067 ceph-mon[112438]: Reconfiguring osd.2 (monmap changed)... 2024-04-22T23:28:31.543 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:31 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T23:28:31.543 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:31 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:31.543 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:31 smithi067 ceph-mon[112438]: Reconfiguring daemon osd.2 on smithi067 2024-04-22T23:28:31.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:31 smithi134 ceph-mon[85630]: pgmap v20: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:28:31.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:31 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:31.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:31 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:31.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:31 smithi134 ceph-mon[85630]: Reconfiguring osd.2 (monmap changed)... 2024-04-22T23:28:31.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:31 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T23:28:31.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:31 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:31.743 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:31 smithi134 ceph-mon[85630]: Reconfiguring daemon osd.2 on smithi067 2024-04-22T23:28:33.225 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:33 smithi067 ceph-mon[112438]: pgmap v21: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:28:33.225 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:33 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:33.225 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:33 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:33.226 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:33 smithi067 ceph-mon[112438]: Reconfiguring mds.cephfs.smithi067.aqlxpv (monmap changed)... 2024-04-22T23:28:33.226 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:33 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi067.aqlxpv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T23:28:33.226 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:33 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:33.226 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:33 smithi067 ceph-mon[112438]: Reconfiguring daemon mds.cephfs.smithi067.aqlxpv on smithi067 2024-04-22T23:28:33.492 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:33 smithi134 ceph-mon[85630]: pgmap v21: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:28:33.492 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:33 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:33.492 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:33 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:33.492 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:33 smithi134 ceph-mon[85630]: Reconfiguring mds.cephfs.smithi067.aqlxpv (monmap changed)... 2024-04-22T23:28:33.492 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:33 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi067.aqlxpv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T23:28:33.492 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:33 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:33.492 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:33 smithi134 ceph-mon[85630]: Reconfiguring daemon mds.cephfs.smithi067.aqlxpv on smithi067 2024-04-22T23:28:35.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:34 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:35.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:34 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:35.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:34 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi067.xyazmh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T23:28:35.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:34 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:35.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:34 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:35.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:34 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:35.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:34 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi067.xyazmh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T23:28:35.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:34 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:36.132 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:35 smithi134 ceph-mon[85630]: Reconfiguring mds.cephfs.smithi067.xyazmh (monmap changed)... 2024-04-22T23:28:36.132 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:35 smithi134 ceph-mon[85630]: Reconfiguring daemon mds.cephfs.smithi067.xyazmh on smithi067 2024-04-22T23:28:36.132 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:35 smithi134 ceph-mon[85630]: pgmap v22: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-22T23:28:36.132 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:35 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:36.132 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:35 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:36.132 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:35 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi134", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T23:28:36.133 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:35 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi134", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T23:28:36.133 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:35 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi134", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-22T23:28:36.133 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:35 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.ceph-exporter.smithi134"}]: dispatch 2024-04-22T23:28:36.133 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:35 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:36.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:35 smithi067 ceph-mon[112438]: Reconfiguring mds.cephfs.smithi067.xyazmh (monmap changed)... 2024-04-22T23:28:36.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:35 smithi067 ceph-mon[112438]: Reconfiguring daemon mds.cephfs.smithi067.xyazmh on smithi067 2024-04-22T23:28:36.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:35 smithi067 ceph-mon[112438]: pgmap v22: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-22T23:28:36.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:35 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:36.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:35 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:36.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:35 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi134", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T23:28:36.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:35 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi134", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-22T23:28:36.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:35 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "auth caps", "entity": "client.ceph-exporter.smithi134", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-22T23:28:36.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:35 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.ceph-exporter.smithi134"}]: dispatch 2024-04-22T23:28:36.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:35 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:37.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:36 smithi134 ceph-mon[85630]: Reconfiguring ceph-exporter.smithi134 (monmap changed)... 2024-04-22T23:28:37.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:36 smithi134 ceph-mon[85630]: Unable to update caps for client.ceph-exporter.smithi134 2024-04-22T23:28:37.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:36 smithi134 ceph-mon[85630]: Reconfiguring daemon ceph-exporter.smithi134 on smithi134 2024-04-22T23:28:37.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:36 smithi067 ceph-mon[112438]: Reconfiguring ceph-exporter.smithi134 (monmap changed)... 2024-04-22T23:28:37.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:36 smithi067 ceph-mon[112438]: Unable to update caps for client.ceph-exporter.smithi134 2024-04-22T23:28:37.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:36 smithi067 ceph-mon[112438]: Reconfiguring daemon ceph-exporter.smithi134 on smithi134 2024-04-22T23:28:37.955 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:37 smithi134 ceph-mon[85630]: pgmap v23: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-22T23:28:37.955 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:37 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:37.955 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:37 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:37.955 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:37 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi134", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T23:28:37.955 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:37 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:38.082 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:37 smithi067 ceph-mon[112438]: pgmap v23: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-22T23:28:38.082 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:37 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:38.082 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:37 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:38.082 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:37 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi134", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T23:28:38.082 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:37 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:38.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:38 smithi134 ceph-mon[85630]: Reconfiguring crash.smithi134 (monmap changed)... 2024-04-22T23:28:38.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:38 smithi134 ceph-mon[85630]: Reconfiguring daemon crash.smithi134 on smithi134 2024-04-22T23:28:38.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:38 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:38.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:38 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:38.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:38 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi134.ccxkcj", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T23:28:38.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:38 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T23:28:38.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:38 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:39.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:38 smithi067 ceph-mon[112438]: Reconfiguring crash.smithi134 (monmap changed)... 2024-04-22T23:28:39.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:38 smithi067 ceph-mon[112438]: Reconfiguring daemon crash.smithi134 on smithi134 2024-04-22T23:28:39.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:38 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:39.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:38 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:39.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:38 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi134.ccxkcj", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T23:28:39.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:38 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T23:28:39.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:38 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:40.174 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:39 smithi134 ceph-mon[85630]: pgmap v24: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-22T23:28:40.175 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:39 smithi134 ceph-mon[85630]: Reconfiguring mgr.smithi134.ccxkcj (monmap changed)... 2024-04-22T23:28:40.175 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:39 smithi134 ceph-mon[85630]: Reconfiguring daemon mgr.smithi134.ccxkcj on smithi134 2024-04-22T23:28:40.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:39 smithi067 ceph-mon[112438]: pgmap v24: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-22T23:28:40.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:39 smithi067 ceph-mon[112438]: Reconfiguring mgr.smithi134.ccxkcj (monmap changed)... 2024-04-22T23:28:40.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:39 smithi067 ceph-mon[112438]: Reconfiguring daemon mgr.smithi134.ccxkcj on smithi134 2024-04-22T23:28:41.492 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:41 smithi134 ceph-mon[85630]: pgmap v25: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-22T23:28:41.492 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:41.492 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:41.492 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:41 smithi134 ceph-mon[85630]: Reconfiguring mon.smithi134 (monmap changed)... 2024-04-22T23:28:41.492 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T23:28:41.492 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T23:28:41.492 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:41.492 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:41 smithi134 ceph-mon[85630]: Reconfiguring daemon mon.smithi134 on smithi134 2024-04-22T23:28:41.493 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:28:41.542 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:41 smithi067 ceph-mon[112438]: pgmap v25: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-22T23:28:41.543 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:41.543 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:41.543 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:41 smithi067 ceph-mon[112438]: Reconfiguring mon.smithi134 (monmap changed)... 2024-04-22T23:28:41.543 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T23:28:41.543 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T23:28:41.543 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:41.543 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:41 smithi067 ceph-mon[112438]: Reconfiguring daemon mon.smithi134 on smithi134 2024-04-22T23:28:41.543 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:28:43.155 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:42 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:43.155 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:42 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:43.155 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:42 smithi134 ceph-mon[85630]: Reconfiguring osd.3 (monmap changed)... 2024-04-22T23:28:43.155 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:42 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T23:28:43.155 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:42 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:43.155 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:42 smithi134 ceph-mon[85630]: Reconfiguring daemon osd.3 on smithi134 2024-04-22T23:28:43.155 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:42 smithi134 ceph-mon[85630]: pgmap v26: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-22T23:28:43.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:42 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:43.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:42 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:43.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:42 smithi067 ceph-mon[112438]: Reconfiguring osd.3 (monmap changed)... 2024-04-22T23:28:43.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:42 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T23:28:43.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:42 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:43.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:42 smithi067 ceph-mon[112438]: Reconfiguring daemon osd.3 on smithi134 2024-04-22T23:28:43.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:42 smithi067 ceph-mon[112438]: pgmap v26: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-22T23:28:44.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:44 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:44.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:44 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:44.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:44 smithi134 ceph-mon[85630]: Reconfiguring osd.4 (monmap changed)... 2024-04-22T23:28:44.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:44 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T23:28:44.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:44 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:44.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:44 smithi134 ceph-mon[85630]: Reconfiguring daemon osd.4 on smithi134 2024-04-22T23:28:44.967 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:44 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:44.967 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:44 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:44.967 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:44 smithi067 ceph-mon[112438]: Reconfiguring osd.4 (monmap changed)... 2024-04-22T23:28:44.968 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:44 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T23:28:44.968 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:44 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:44.968 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:44 smithi067 ceph-mon[112438]: Reconfiguring daemon osd.4 on smithi134 2024-04-22T23:28:45.878 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:45 smithi134 ceph-mon[85630]: pgmap v27: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-22T23:28:45.878 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:45 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:45.879 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:45 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:45.879 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:45 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T23:28:45.879 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:45 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:45.930 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:45 smithi067 ceph-mon[112438]: pgmap v27: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-22T23:28:45.930 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:45 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:45.931 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:45 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:45.931 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:45 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T23:28:45.931 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:45 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:46.884 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:46 smithi134 ceph-mon[85630]: Reconfiguring osd.5 (monmap changed)... 2024-04-22T23:28:46.884 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:46 smithi134 ceph-mon[85630]: Reconfiguring daemon osd.5 on smithi134 2024-04-22T23:28:47.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:46 smithi067 ceph-mon[112438]: Reconfiguring osd.5 (monmap changed)... 2024-04-22T23:28:47.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:46 smithi067 ceph-mon[112438]: Reconfiguring daemon osd.5 on smithi134 2024-04-22T23:28:47.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.581+0000 7fc26ee1b700 1 -- 172.21.15.67:0/3276132821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2680ffc20 msgr2=0x7fc268100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:47.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.581+0000 7fc26ee1b700 1 --2- 172.21.15.67:0/3276132821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2680ffc20 0x7fc268100080 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc254009910 tx=0x7fc254009c20 comp rx=0 tx=0).stop 2024-04-22T23:28:47.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.581+0000 7fc26ee1b700 1 -- 172.21.15.67:0/3276132821 shutdown_connections 2024-04-22T23:28:47.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.581+0000 7fc26ee1b700 1 --2- 172.21.15.67:0/3276132821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2680ffc20 0x7fc268100080 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:47.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.581+0000 7fc26ee1b700 1 --2- 172.21.15.67:0/3276132821 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc268105bd0 0x7fc268105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:47.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.581+0000 7fc26ee1b700 1 -- 172.21.15.67:0/3276132821 >> 172.21.15.67:0/3276132821 conn(0x7fc268075670 msgr2=0x7fc268075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:28:47.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.581+0000 7fc26ee1b700 1 -- 172.21.15.67:0/3276132821 shutdown_connections 2024-04-22T23:28:47.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.581+0000 7fc26ee1b700 1 -- 172.21.15.67:0/3276132821 wait complete. 2024-04-22T23:28:47.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.582+0000 7fc26ee1b700 1 Processor -- start 2024-04-22T23:28:47.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.582+0000 7fc26ee1b700 1 -- start start 2024-04-22T23:28:47.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.582+0000 7fc26ee1b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2680ffc20 0x7fc268197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:47.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.582+0000 7fc26ee1b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc268105bd0 0x7fc268197c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:47.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.582+0000 7fc26ee1b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc268198360 con 0x7fc268105bd0 2024-04-22T23:28:47.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.582+0000 7fc26ee1b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2681984d0 con 0x7fc2680ffc20 2024-04-22T23:28:47.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.582+0000 7fc267fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc268105bd0 0x7fc268197c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:47.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.582+0000 7fc267fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc268105bd0 0x7fc268197c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38262/0 (socket says 172.21.15.67:38262) 2024-04-22T23:28:47.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.582+0000 7fc267fff700 1 -- 172.21.15.67:0/3073797044 learned_addr learned my addr 172.21.15.67:0/3073797044 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:28:47.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.582+0000 7fc26cbb7700 1 --2- 172.21.15.67:0/3073797044 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2680ffc20 0x7fc268197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:47.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.582+0000 7fc267fff700 1 -- 172.21.15.67:0/3073797044 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2680ffc20 msgr2=0x7fc268197740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:47.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.582+0000 7fc267fff700 1 --2- 172.21.15.67:0/3073797044 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2680ffc20 0x7fc268197740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:47.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.582+0000 7fc267fff700 1 -- 172.21.15.67:0/3073797044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2540095f0 con 0x7fc268105bd0 2024-04-22T23:28:47.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.583+0000 7fc267fff700 1 --2- 172.21.15.67:0/3073797044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc268105bd0 0x7fc268197c80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc2540092e0 tx=0x7fc25400b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:28:47.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.583+0000 7fc265ffb700 1 -- 172.21.15.67:0/3073797044 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc25401d070 con 0x7fc268105bd0 2024-04-22T23:28:47.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.583+0000 7fc265ffb700 1 -- 172.21.15.67:0/3073797044 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc2540052a0 con 0x7fc268105bd0 2024-04-22T23:28:47.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.583+0000 7fc26ee1b700 1 -- 172.21.15.67:0/3073797044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2681a07b0 con 0x7fc268105bd0 2024-04-22T23:28:47.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.583+0000 7fc265ffb700 1 -- 172.21.15.67:0/3073797044 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc254017400 con 0x7fc268105bd0 2024-04-22T23:28:47.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.583+0000 7fc26ee1b700 1 -- 172.21.15.67:0/3073797044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2681a0c50 con 0x7fc268105bd0 2024-04-22T23:28:47.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.584+0000 7fc26ee1b700 1 -- 172.21.15.67:0/3073797044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2681080d0 con 0x7fc268105bd0 2024-04-22T23:28:47.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.585+0000 7fc265ffb700 1 -- 172.21.15.67:0/3073797044 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 39) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fc25400f820 con 0x7fc268105bd0 2024-04-22T23:28:47.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.585+0000 7fc265ffb700 1 --2- 172.21.15.67:0/3073797044 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc25806edf0 0x7fc2580712b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:47.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.585+0000 7fc265ffb700 1 -- 172.21.15.67:0/3073797044 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fc2540902d0 con 0x7fc268105bd0 2024-04-22T23:28:47.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.585+0000 7fc26cbb7700 1 --2- 172.21.15.67:0/3073797044 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc25806edf0 0x7fc2580712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:47.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.586+0000 7fc26cbb7700 1 --2- 172.21.15.67:0/3073797044 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc25806edf0 0x7fc2580712b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc25c009c60 tx=0x7fc25c009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:28:47.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.590+0000 7fc265ffb700 1 -- 172.21.15.67:0/3073797044 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc254058f40 con 0x7fc268105bd0 2024-04-22T23:28:47.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:47 smithi134 ceph-mon[85630]: pgmap v28: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:28:47.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:47 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:47.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:47 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:47.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:47 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi134.owipad", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T23:28:47.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:47 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:47.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.821+0000 7fc26ee1b700 1 -- 172.21.15.67:0/3073797044 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc268068c70 con 0x7fc25806edf0 2024-04-22T23:28:47.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.822+0000 7fc265ffb700 1 -- 172.21.15.67:0/3073797044 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7fc268068c70 con 0x7fc25806edf0 2024-04-22T23:28:47.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.825+0000 7fc26ee1b700 1 -- 172.21.15.67:0/3073797044 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc25806edf0 msgr2=0x7fc2580712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:47.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.825+0000 7fc26ee1b700 1 --2- 172.21.15.67:0/3073797044 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc25806edf0 0x7fc2580712b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc25c009c60 tx=0x7fc25c009260 comp rx=0 tx=0).stop 2024-04-22T23:28:47.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.825+0000 7fc26ee1b700 1 -- 172.21.15.67:0/3073797044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc268105bd0 msgr2=0x7fc268197c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:47.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.825+0000 7fc26ee1b700 1 --2- 172.21.15.67:0/3073797044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc268105bd0 0x7fc268197c80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc2540092e0 tx=0x7fc25400b6c0 comp rx=0 tx=0).stop 2024-04-22T23:28:47.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.825+0000 7fc26ee1b700 1 -- 172.21.15.67:0/3073797044 shutdown_connections 2024-04-22T23:28:47.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.825+0000 7fc26ee1b700 1 --2- 172.21.15.67:0/3073797044 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc25806edf0 0x7fc2580712b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:47.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.825+0000 7fc26ee1b700 1 --2- 172.21.15.67:0/3073797044 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2680ffc20 0x7fc268197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:47.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.825+0000 7fc26ee1b700 1 --2- 172.21.15.67:0/3073797044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc268105bd0 0x7fc268197c80 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:47.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.825+0000 7fc26ee1b700 1 -- 172.21.15.67:0/3073797044 >> 172.21.15.67:0/3073797044 conn(0x7fc268075670 msgr2=0x7fc2680fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:28:47.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.825+0000 7fc26ee1b700 1 -- 172.21.15.67:0/3073797044 shutdown_connections 2024-04-22T23:28:47.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.825+0000 7fc26ee1b700 1 -- 172.21.15.67:0/3073797044 wait complete. 2024-04-22T23:28:47.834 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:28:47.834 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:47 smithi067 ceph-mon[112438]: pgmap v28: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:28:47.834 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:47 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:47.834 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:47 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:47.834 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:47 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi134.owipad", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T23:28:47.834 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:47 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:47.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.932+0000 7f3e872fd700 1 -- 172.21.15.67:0/3905677743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e80101850 msgr2=0x7f3e801058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:47.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.932+0000 7f3e872fd700 1 --2- 172.21.15.67:0/3905677743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e80101850 0x7f3e801058a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3e70009910 tx=0x7f3e70009c20 comp rx=0 tx=0).stop 2024-04-22T23:28:47.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.932+0000 7f3e872fd700 1 -- 172.21.15.67:0/3905677743 shutdown_connections 2024-04-22T23:28:47.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.932+0000 7f3e872fd700 1 --2- 172.21.15.67:0/3905677743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e80101850 0x7f3e801058a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:47.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.932+0000 7f3e872fd700 1 --2- 172.21.15.67:0/3905677743 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3e80100ea0 0x7f3e80101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:47.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.932+0000 7f3e872fd700 1 -- 172.21.15.67:0/3905677743 >> 172.21.15.67:0/3905677743 conn(0x7f3e800fc750 msgr2=0x7f3e800feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:28:47.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.932+0000 7f3e872fd700 1 -- 172.21.15.67:0/3905677743 shutdown_connections 2024-04-22T23:28:47.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.932+0000 7f3e872fd700 1 -- 172.21.15.67:0/3905677743 wait complete. 2024-04-22T23:28:47.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.933+0000 7f3e872fd700 1 Processor -- start 2024-04-22T23:28:47.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.933+0000 7f3e872fd700 1 -- start start 2024-04-22T23:28:47.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.933+0000 7f3e872fd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3e80100ea0 0x7f3e8019c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:47.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.933+0000 7f3e872fd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e80101850 0x7f3e8019cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:47.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.933+0000 7f3e872fd700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8019d2d0 con 0x7f3e80101850 2024-04-22T23:28:47.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.933+0000 7f3e872fd700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e80196780 con 0x7f3e80100ea0 2024-04-22T23:28:47.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.934+0000 7f3e84898700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e80101850 0x7f3e8019cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:47.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.934+0000 7f3e84898700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e80101850 0x7f3e8019cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38292/0 (socket says 172.21.15.67:38292) 2024-04-22T23:28:47.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.934+0000 7f3e84898700 1 -- 172.21.15.67:0/2410913785 learned_addr learned my addr 172.21.15.67:0/2410913785 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:28:47.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.934+0000 7f3e85099700 1 --2- 172.21.15.67:0/2410913785 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3e80100ea0 0x7f3e8019c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:47.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.934+0000 7f3e84898700 1 -- 172.21.15.67:0/2410913785 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3e80100ea0 msgr2=0x7f3e8019c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:47.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.934+0000 7f3e84898700 1 --2- 172.21.15.67:0/2410913785 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3e80100ea0 0x7f3e8019c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:47.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.934+0000 7f3e84898700 1 -- 172.21.15.67:0/2410913785 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e700095f0 con 0x7f3e80101850 2024-04-22T23:28:47.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.934+0000 7f3e84898700 1 --2- 172.21.15.67:0/2410913785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e80101850 0x7f3e8019cc40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3e700059d0 tx=0x7f3e7000f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:28:47.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.934+0000 7f3e767fc700 1 -- 172.21.15.67:0/2410913785 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e7001d070 con 0x7f3e80101850 2024-04-22T23:28:47.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.934+0000 7f3e767fc700 1 -- 172.21.15.67:0/2410913785 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3e7000b610 con 0x7f3e80101850 2024-04-22T23:28:47.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.934+0000 7f3e872fd700 1 -- 172.21.15.67:0/2410913785 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e80196a00 con 0x7f3e80101850 2024-04-22T23:28:47.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.934+0000 7f3e872fd700 1 -- 172.21.15.67:0/2410913785 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e80196ef0 con 0x7f3e80101850 2024-04-22T23:28:47.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.934+0000 7f3e767fc700 1 -- 172.21.15.67:0/2410913785 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e70022470 con 0x7f3e80101850 2024-04-22T23:28:47.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.935+0000 7f3e872fd700 1 -- 172.21.15.67:0/2410913785 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e801092a0 con 0x7f3e80101850 2024-04-22T23:28:47.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.936+0000 7f3e767fc700 1 -- 172.21.15.67:0/2410913785 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 39) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f3e7000fc70 con 0x7f3e80101850 2024-04-22T23:28:47.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.936+0000 7f3e767fc700 1 --2- 172.21.15.67:0/2410913785 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3e6c06ee00 0x7f3e6c0712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:47.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.936+0000 7f3e767fc700 1 -- 172.21.15.67:0/2410913785 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f3e700903f0 con 0x7f3e80101850 2024-04-22T23:28:47.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.936+0000 7f3e85099700 1 --2- 172.21.15.67:0/2410913785 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3e6c06ee00 0x7f3e6c0712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:47.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.937+0000 7f3e85099700 1 --2- 172.21.15.67:0/2410913785 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3e6c06ee00 0x7f3e6c0712c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c009bf0 tx=0x7f3e7c009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:28:47.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:47.941+0000 7f3e767fc700 1 -- 172.21.15.67:0/2410913785 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f3e7005cb90 con 0x7f3e80101850 2024-04-22T23:28:48.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.172+0000 7f3e872fd700 1 -- 172.21.15.67:0/2410913785 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e80197160 con 0x7f3e6c06ee00 2024-04-22T23:28:48.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.174+0000 7f3e767fc700 1 -- 172.21.15.67:0/2410913785 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f3e80197160 con 0x7f3e6c06ee00 2024-04-22T23:28:48.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.176+0000 7f3e872fd700 1 -- 172.21.15.67:0/2410913785 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3e6c06ee00 msgr2=0x7f3e6c0712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:48.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.176+0000 7f3e872fd700 1 --2- 172.21.15.67:0/2410913785 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3e6c06ee00 0x7f3e6c0712c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c009bf0 tx=0x7f3e7c009260 comp rx=0 tx=0).stop 2024-04-22T23:28:48.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.176+0000 7f3e872fd700 1 -- 172.21.15.67:0/2410913785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e80101850 msgr2=0x7f3e8019cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:48.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.176+0000 7f3e872fd700 1 --2- 172.21.15.67:0/2410913785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e80101850 0x7f3e8019cc40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3e700059d0 tx=0x7f3e7000f740 comp rx=0 tx=0).stop 2024-04-22T23:28:48.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.176+0000 7f3e872fd700 1 -- 172.21.15.67:0/2410913785 shutdown_connections 2024-04-22T23:28:48.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.176+0000 7f3e872fd700 1 --2- 172.21.15.67:0/2410913785 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3e6c06ee00 0x7f3e6c0712c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:48.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.176+0000 7f3e872fd700 1 --2- 172.21.15.67:0/2410913785 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3e80100ea0 0x7f3e8019c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:48.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.176+0000 7f3e872fd700 1 --2- 172.21.15.67:0/2410913785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e80101850 0x7f3e8019cc40 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:48.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.176+0000 7f3e872fd700 1 -- 172.21.15.67:0/2410913785 >> 172.21.15.67:0/2410913785 conn(0x7f3e800fc750 msgr2=0x7f3e800fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:28:48.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.176+0000 7f3e872fd700 1 -- 172.21.15.67:0/2410913785 shutdown_connections 2024-04-22T23:28:48.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.176+0000 7f3e872fd700 1 -- 172.21.15.67:0/2410913785 wait complete. 2024-04-22T23:28:48.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.282+0000 7f724b306700 1 -- 172.21.15.67:0/427619203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72441027a0 msgr2=0x7f724410ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:48.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.282+0000 7f724b306700 1 --2- 172.21.15.67:0/427619203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72441027a0 0x7f724410ac90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7234009910 tx=0x7f7234009c20 comp rx=0 tx=0).stop 2024-04-22T23:28:48.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.283+0000 7f724b306700 1 -- 172.21.15.67:0/427619203 shutdown_connections 2024-04-22T23:28:48.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.283+0000 7f724b306700 1 --2- 172.21.15.67:0/427619203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72441027a0 0x7f724410ac90 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:48.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.283+0000 7f724b306700 1 --2- 172.21.15.67:0/427619203 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7244101e80 0x7f7244102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:48.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.283+0000 7f724b306700 1 -- 172.21.15.67:0/427619203 >> 172.21.15.67:0/427619203 conn(0x7f72440fb650 msgr2=0x7f72440fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:28:48.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.283+0000 7f724b306700 1 -- 172.21.15.67:0/427619203 shutdown_connections 2024-04-22T23:28:48.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.283+0000 7f724b306700 1 -- 172.21.15.67:0/427619203 wait complete. 2024-04-22T23:28:48.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.283+0000 7f724b306700 1 Processor -- start 2024-04-22T23:28:48.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.283+0000 7f724b306700 1 -- start start 2024-04-22T23:28:48.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.284+0000 7f724b306700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7244101e80 0x7f72440ffae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:48.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.284+0000 7f724b306700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f72441027a0 0x7f7244100020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:48.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.284+0000 7f724b306700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7244100560 con 0x7f7244101e80 2024-04-22T23:28:48.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.284+0000 7f724b306700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72441006a0 con 0x7f72441027a0 2024-04-22T23:28:48.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.284+0000 7f72490a2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7244101e80 0x7f72440ffae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:48.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.284+0000 7f72490a2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7244101e80 0x7f72440ffae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38310/0 (socket says 172.21.15.67:38310) 2024-04-22T23:28:48.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.284+0000 7f72490a2700 1 -- 172.21.15.67:0/537086018 learned_addr learned my addr 172.21.15.67:0/537086018 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:28:48.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.284+0000 7f72488a1700 1 --2- 172.21.15.67:0/537086018 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f72441027a0 0x7f7244100020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:48.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.284+0000 7f72490a2700 1 -- 172.21.15.67:0/537086018 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f72441027a0 msgr2=0x7f7244100020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:48.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.284+0000 7f72490a2700 1 --2- 172.21.15.67:0/537086018 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f72441027a0 0x7f7244100020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:48.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.284+0000 7f72490a2700 1 -- 172.21.15.67:0/537086018 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72340095f0 con 0x7f7244101e80 2024-04-22T23:28:48.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.284+0000 7f72490a2700 1 --2- 172.21.15.67:0/537086018 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7244101e80 0x7f72440ffae0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f724000d8d0 tx=0x7f724000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:28:48.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.284+0000 7f723a7fc700 1 -- 172.21.15.67:0/537086018 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72400098a0 con 0x7f7244101e80 2024-04-22T23:28:48.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.284+0000 7f723a7fc700 1 -- 172.21.15.67:0/537086018 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7240010460 con 0x7f7244101e80 2024-04-22T23:28:48.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.284+0000 7f724b306700 1 -- 172.21.15.67:0/537086018 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72441a2a50 con 0x7f7244101e80 2024-04-22T23:28:48.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.284+0000 7f723a7fc700 1 -- 172.21.15.67:0/537086018 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f724000f5f0 con 0x7f7244101e80 2024-04-22T23:28:48.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.285+0000 7f724b306700 1 -- 172.21.15.67:0/537086018 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72441a2e60 con 0x7f7244101e80 2024-04-22T23:28:48.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.285+0000 7f724b306700 1 -- 172.21.15.67:0/537086018 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7244108320 con 0x7f7244101e80 2024-04-22T23:28:48.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.286+0000 7f723a7fc700 1 -- 172.21.15.67:0/537086018 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 39) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f7240009a00 con 0x7f7244101e80 2024-04-22T23:28:48.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.286+0000 7f723a7fc700 1 --2- 172.21.15.67:0/537086018 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f723006ee00 0x7f72300712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:48.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.287+0000 7f723a7fc700 1 -- 172.21.15.67:0/537086018 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f724008ee60 con 0x7f7244101e80 2024-04-22T23:28:48.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.287+0000 7f72488a1700 1 --2- 172.21.15.67:0/537086018 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f723006ee00 0x7f72300712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:48.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.287+0000 7f72488a1700 1 --2- 172.21.15.67:0/537086018 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f723006ee00 0x7f72300712c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f72441012d0 tx=0x7f723401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:28:48.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.291+0000 7f723a7fc700 1 -- 172.21.15.67:0/537086018 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7240057ad0 con 0x7f7244101e80 2024-04-22T23:28:48.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.516+0000 7f724b306700 1 -- 172.21.15.67:0/537086018 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f724402cce0 con 0x7f723006ee00 2024-04-22T23:28:48.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.527+0000 7f723a7fc700 1 -- 172.21.15.67:0/537086018 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f724402cce0 con 0x7f723006ee00 2024-04-22T23:28:48.526 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:28:48.526 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (2m) 46s ago 13m 17.8M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:28:48.526 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (14m) 46s ago 14m 8150k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:28:48.526 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (13m) 32s ago 13m 8132k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:28:48.526 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (13m) 46s ago 13m 6660k - 18.2.1 5be31c24972a 14085634c5d3 2024-04-22T23:28:48.526 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (12m) 32s ago 12m 6669k - 18.2.1 5be31c24972a fa995d458353 2024-04-22T23:28:48.526 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (2m) 46s ago 13m 75.2M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:28:48.526 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (8m) 46s ago 8m 62.7M - 18.2.1 5be31c24972a e24a471f0a8c 2024-04-22T23:28:48.527 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (8m) 46s ago 8m 129M - 18.2.1 5be31c24972a 0730f9fc8e61 2024-04-22T23:28:48.527 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (8m) 32s ago 8m 123M - 18.2.1 5be31c24972a b6a169586433 2024-04-22T23:28:48.527 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (8m) 32s ago 8m 68.0M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:28:48.527 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (4m) 46s ago 15m 473M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:28:48.527 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (3m) 32s ago 12m 449M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:28:48.527 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (61s) 46s ago 15m 30.3M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:28:48.527 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (36s) 32s ago 12m 28.2M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:28:48.527 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (3m) 46s ago 13m 15.6M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:28:48.527 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (3m) 32s ago 12m 15.3M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:28:48.527 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (12m) 46s ago 12m 206M 1216M 18.2.1 5be31c24972a a033247ee8c2 2024-04-22T23:28:48.527 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (11m) 46s ago 11m 211M 1216M 18.2.1 5be31c24972a 915b07a8edc4 2024-04-22T23:28:48.527 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (11m) 46s ago 11m 173M 1216M 18.2.1 5be31c24972a e257ecb4efa1 2024-04-22T23:28:48.527 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (10m) 32s ago 10m 250M 2240M 18.2.1 5be31c24972a 9c2c91a967ba 2024-04-22T23:28:48.527 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (10m) 32s ago 10m 218M 2240M 18.2.1 5be31c24972a c7586562ffa6 2024-04-22T23:28:48.527 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (9m) 32s ago 9m 195M 2240M 18.2.1 5be31c24972a 9fd97f1c65cb 2024-04-22T23:28:48.528 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (2m) 46s ago 13m 64.7M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:28:48.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.529+0000 7f724b306700 1 -- 172.21.15.67:0/537086018 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f723006ee00 msgr2=0x7f72300712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:48.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.529+0000 7f724b306700 1 --2- 172.21.15.67:0/537086018 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f723006ee00 0x7f72300712c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f72441012d0 tx=0x7f723401a040 comp rx=0 tx=0).stop 2024-04-22T23:28:48.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.529+0000 7f724b306700 1 -- 172.21.15.67:0/537086018 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7244101e80 msgr2=0x7f72440ffae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:48.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.529+0000 7f724b306700 1 --2- 172.21.15.67:0/537086018 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7244101e80 0x7f72440ffae0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f724000d8d0 tx=0x7f724000dbe0 comp rx=0 tx=0).stop 2024-04-22T23:28:48.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.529+0000 7f724b306700 1 -- 172.21.15.67:0/537086018 shutdown_connections 2024-04-22T23:28:48.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.529+0000 7f724b306700 1 --2- 172.21.15.67:0/537086018 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f723006ee00 0x7f72300712c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:48.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.529+0000 7f724b306700 1 --2- 172.21.15.67:0/537086018 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7244101e80 0x7f72440ffae0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:48.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.529+0000 7f724b306700 1 --2- 172.21.15.67:0/537086018 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f72441027a0 0x7f7244100020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:48.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.529+0000 7f724b306700 1 -- 172.21.15.67:0/537086018 >> 172.21.15.67:0/537086018 conn(0x7f72440fb650 msgr2=0x7f7244105460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:28:48.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.530+0000 7f724b306700 1 -- 172.21.15.67:0/537086018 shutdown_connections 2024-04-22T23:28:48.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.530+0000 7f724b306700 1 -- 172.21.15.67:0/537086018 wait complete. 2024-04-22T23:28:48.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.636+0000 7f66a3564700 1 -- 172.21.15.67:0/281988285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f669c100ea0 msgr2=0x7f669c101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:48.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.636+0000 7f66a3564700 1 --2- 172.21.15.67:0/281988285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f669c100ea0 0x7f669c101280 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f668c009910 tx=0x7f668c009c20 comp rx=0 tx=0).stop 2024-04-22T23:28:48.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.636+0000 7f66a3564700 1 -- 172.21.15.67:0/281988285 shutdown_connections 2024-04-22T23:28:48.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.636+0000 7f66a3564700 1 --2- 172.21.15.67:0/281988285 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f669c101850 0x7f669c1058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:48.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.636+0000 7f66a3564700 1 --2- 172.21.15.67:0/281988285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f669c100ea0 0x7f669c101280 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:48.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.636+0000 7f66a3564700 1 -- 172.21.15.67:0/281988285 >> 172.21.15.67:0/281988285 conn(0x7f669c0fc750 msgr2=0x7f669c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:28:48.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.636+0000 7f66a3564700 1 -- 172.21.15.67:0/281988285 shutdown_connections 2024-04-22T23:28:48.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.637+0000 7f66a3564700 1 -- 172.21.15.67:0/281988285 wait complete. 2024-04-22T23:28:48.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.637+0000 7f66a3564700 1 Processor -- start 2024-04-22T23:28:48.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.637+0000 7f66a3564700 1 -- start start 2024-04-22T23:28:48.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.637+0000 7f66a3564700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f669c100ea0 0x7f669c19c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:48.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.637+0000 7f66a3564700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f669c101850 0x7f669c19cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:48.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.637+0000 7f66a3564700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f669c19d2c0 con 0x7f669c101850 2024-04-22T23:28:48.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.637+0000 7f66a3564700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f669c196770 con 0x7f669c100ea0 2024-04-22T23:28:48.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.637+0000 7f66a0aff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f669c101850 0x7f669c19cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:48.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.637+0000 7f66a0aff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f669c101850 0x7f669c19cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38340/0 (socket says 172.21.15.67:38340) 2024-04-22T23:28:48.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.638+0000 7f66a0aff700 1 -- 172.21.15.67:0/3027056565 learned_addr learned my addr 172.21.15.67:0/3027056565 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:28:48.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.638+0000 7f66a1300700 1 --2- 172.21.15.67:0/3027056565 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f669c100ea0 0x7f669c19c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:48.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.638+0000 7f66a0aff700 1 -- 172.21.15.67:0/3027056565 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f669c100ea0 msgr2=0x7f669c19c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:48.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.638+0000 7f66a0aff700 1 --2- 172.21.15.67:0/3027056565 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f669c100ea0 0x7f669c19c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:48.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.638+0000 7f66a0aff700 1 -- 172.21.15.67:0/3027056565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f668c0095f0 con 0x7f669c101850 2024-04-22T23:28:48.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.638+0000 7f66a1300700 1 --2- 172.21.15.67:0/3027056565 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f669c100ea0 0x7f669c19c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T23:28:48.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.638+0000 7f66a0aff700 1 --2- 172.21.15.67:0/3027056565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f669c101850 0x7f669c19cc30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f669800e8f0 tx=0x7f669800ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:28:48.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.638+0000 7f66927fc700 1 -- 172.21.15.67:0/3027056565 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6698009790 con 0x7f669c101850 2024-04-22T23:28:48.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.638+0000 7f66a3564700 1 -- 172.21.15.67:0/3027056565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f669c196a50 con 0x7f669c101850 2024-04-22T23:28:48.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.638+0000 7f66927fc700 1 -- 172.21.15.67:0/3027056565 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f669800c810 con 0x7f669c101850 2024-04-22T23:28:48.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.638+0000 7f66a3564700 1 -- 172.21.15.67:0/3027056565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f669c196fa0 con 0x7f669c101850 2024-04-22T23:28:48.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.638+0000 7f66927fc700 1 -- 172.21.15.67:0/3027056565 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6698005580 con 0x7f669c101850 2024-04-22T23:28:48.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.639+0000 7f66a3564700 1 -- 172.21.15.67:0/3027056565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f669c109240 con 0x7f669c101850 2024-04-22T23:28:48.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.640+0000 7f66927fc700 1 -- 172.21.15.67:0/3027056565 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 39) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f669800c980 con 0x7f669c101850 2024-04-22T23:28:48.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.640+0000 7f66927fc700 1 --2- 172.21.15.67:0/3027056565 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f668806edf0 0x7f66880712b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:48.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.640+0000 7f66927fc700 1 -- 172.21.15.67:0/3027056565 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f6698014070 con 0x7f669c101850 2024-04-22T23:28:48.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.640+0000 7f66a1300700 1 --2- 172.21.15.67:0/3027056565 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f668806edf0 0x7f66880712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:48.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.641+0000 7f66a1300700 1 --2- 172.21.15.67:0/3027056565 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f668806edf0 0x7f66880712b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f668c005b40 tx=0x7f668c005a90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:28:48.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.645+0000 7f66927fc700 1 -- 172.21.15.67:0/3027056565 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f669805ba60 con 0x7f669c101850 2024-04-22T23:28:48.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:48 smithi134 ceph-mon[85630]: Reconfiguring mds.cephfs.smithi134.owipad (monmap changed)... 2024-04-22T23:28:48.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:48 smithi134 ceph-mon[85630]: Reconfiguring daemon mds.cephfs.smithi134.owipad on smithi134 2024-04-22T23:28:48.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:48 smithi134 ceph-mon[85630]: from='client.34160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:28:48.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:48 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:48.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:48 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:48.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:48 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi134.vuxbsj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T23:28:48.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:48 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:48.792 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:48 smithi067 ceph-mon[112438]: Reconfiguring mds.cephfs.smithi134.owipad (monmap changed)... 2024-04-22T23:28:48.792 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:48 smithi067 ceph-mon[112438]: Reconfiguring daemon mds.cephfs.smithi134.owipad on smithi134 2024-04-22T23:28:48.792 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:48 smithi067 ceph-mon[112438]: from='client.34160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:28:48.792 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:48 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:48.792 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:48 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:48.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:48 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi134.vuxbsj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T23:28:48.793 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:48 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:48.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.944+0000 7f66a3564700 1 -- 172.21.15.67:0/3027056565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f669c04f2e0 con 0x7f669c101850 2024-04-22T23:28:48.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.944+0000 7f66927fc700 1 -- 172.21.15.67:0/3027056565 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+688 (secure 0 0 0) 0x7f669805b5f0 con 0x7f669c101850 2024-04-22T23:28:48.943 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:28:48.943 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:28:48.943 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:28:48.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:28:48.944 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:28:48.944 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:28:48.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:28:48.944 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:28:48.944 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-22T23:28:48.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:28:48.944 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:28:48.944 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-22T23:28:48.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:28:48.944 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:28:48.944 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 10, 2024-04-22T23:28:48.944 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 4 2024-04-22T23:28:48.944 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:28:48.944 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:28:48.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.947+0000 7f66a3564700 1 -- 172.21.15.67:0/3027056565 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f668806edf0 msgr2=0x7f66880712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:48.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.947+0000 7f66a3564700 1 --2- 172.21.15.67:0/3027056565 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f668806edf0 0x7f66880712b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f668c005b40 tx=0x7f668c005a90 comp rx=0 tx=0).stop 2024-04-22T23:28:48.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.947+0000 7f66a3564700 1 -- 172.21.15.67:0/3027056565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f669c101850 msgr2=0x7f669c19cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:48.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.947+0000 7f66a3564700 1 --2- 172.21.15.67:0/3027056565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f669c101850 0x7f669c19cc30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f669800e8f0 tx=0x7f669800ec00 comp rx=0 tx=0).stop 2024-04-22T23:28:48.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.947+0000 7f66a3564700 1 -- 172.21.15.67:0/3027056565 shutdown_connections 2024-04-22T23:28:48.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.947+0000 7f66a3564700 1 --2- 172.21.15.67:0/3027056565 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f668806edf0 0x7f66880712b0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:48.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.947+0000 7f66a3564700 1 --2- 172.21.15.67:0/3027056565 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f669c100ea0 0x7f669c19c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:48.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.947+0000 7f66a3564700 1 --2- 172.21.15.67:0/3027056565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f669c101850 0x7f669c19cc30 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:48.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.947+0000 7f66a3564700 1 -- 172.21.15.67:0/3027056565 >> 172.21.15.67:0/3027056565 conn(0x7f669c0fc750 msgr2=0x7f669c0fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:28:48.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.947+0000 7f66a3564700 1 -- 172.21.15.67:0/3027056565 shutdown_connections 2024-04-22T23:28:48.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:48.947+0000 7f66a3564700 1 -- 172.21.15.67:0/3027056565 wait complete. 2024-04-22T23:28:49.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.054+0000 7fc2f81b6700 1 -- 172.21.15.67:0/614159862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2f0101e60 msgr2=0x7fc2f0102240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:49.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.054+0000 7fc2f81b6700 1 --2- 172.21.15.67:0/614159862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2f0101e60 0x7fc2f0102240 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec009910 tx=0x7fc2ec009c20 comp rx=0 tx=0).stop 2024-04-22T23:28:49.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.054+0000 7fc2f81b6700 1 -- 172.21.15.67:0/614159862 shutdown_connections 2024-04-22T23:28:49.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.054+0000 7fc2f81b6700 1 --2- 172.21.15.67:0/614159862 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2f0102780 0x7fc2f010ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:49.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.054+0000 7fc2f81b6700 1 --2- 172.21.15.67:0/614159862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2f0101e60 0x7fc2f0102240 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:49.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.054+0000 7fc2f81b6700 1 -- 172.21.15.67:0/614159862 >> 172.21.15.67:0/614159862 conn(0x7fc2f00fb650 msgr2=0x7fc2f00fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:28:49.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.054+0000 7fc2f81b6700 1 -- 172.21.15.67:0/614159862 shutdown_connections 2024-04-22T23:28:49.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.054+0000 7fc2f81b6700 1 -- 172.21.15.67:0/614159862 wait complete. 2024-04-22T23:28:49.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.054+0000 7fc2f81b6700 1 Processor -- start 2024-04-22T23:28:49.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.055+0000 7fc2f81b6700 1 -- start start 2024-04-22T23:28:49.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.055+0000 7fc2f81b6700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2f0101e60 0x7fc2f00ffac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:49.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.055+0000 7fc2f81b6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2f0102780 0x7fc2f0100000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:49.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.055+0000 7fc2f81b6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f0100540 con 0x7fc2f0102780 2024-04-22T23:28:49.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.055+0000 7fc2f81b6700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f0100680 con 0x7fc2f0101e60 2024-04-22T23:28:49.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.055+0000 7fc2f5751700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2f0102780 0x7fc2f0100000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:49.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.055+0000 7fc2f5751700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2f0102780 0x7fc2f0100000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39234/0 (socket says 172.21.15.67:39234) 2024-04-22T23:28:49.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.055+0000 7fc2f5751700 1 -- 172.21.15.67:0/3495546184 learned_addr learned my addr 172.21.15.67:0/3495546184 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:28:49.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.055+0000 7fc2f5f52700 1 --2- 172.21.15.67:0/3495546184 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2f0101e60 0x7fc2f00ffac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:49.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.055+0000 7fc2f5751700 1 -- 172.21.15.67:0/3495546184 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2f0101e60 msgr2=0x7fc2f00ffac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:49.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.055+0000 7fc2f5751700 1 --2- 172.21.15.67:0/3495546184 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2f0101e60 0x7fc2f00ffac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:49.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.055+0000 7fc2f5751700 1 -- 172.21.15.67:0/3495546184 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2ec0095f0 con 0x7fc2f0102780 2024-04-22T23:28:49.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.055+0000 7fc2f5f52700 1 --2- 172.21.15.67:0/3495546184 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2f0101e60 0x7fc2f00ffac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:28:49.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.055+0000 7fc2f5751700 1 --2- 172.21.15.67:0/3495546184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2f0102780 0x7fc2f0100000 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc2e000d900 tx=0x7fc2e000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:28:49.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.055+0000 7fc2e6ffd700 1 -- 172.21.15.67:0/3495546184 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2e0009900 con 0x7fc2f0102780 2024-04-22T23:28:49.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.056+0000 7fc2e6ffd700 1 -- 172.21.15.67:0/3495546184 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc2e0010460 con 0x7fc2f0102780 2024-04-22T23:28:49.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.056+0000 7fc2f81b6700 1 -- 172.21.15.67:0/3495546184 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2f01a2aa0 con 0x7fc2f0102780 2024-04-22T23:28:49.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.056+0000 7fc2e6ffd700 1 -- 172.21.15.67:0/3495546184 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2e000f5f0 con 0x7fc2f0102780 2024-04-22T23:28:49.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.056+0000 7fc2f81b6700 1 -- 172.21.15.67:0/3495546184 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2f01a2e50 con 0x7fc2f0102780 2024-04-22T23:28:49.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.056+0000 7fc2f81b6700 1 -- 172.21.15.67:0/3495546184 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2f0108310 con 0x7fc2f0102780 2024-04-22T23:28:49.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.057+0000 7fc2e6ffd700 1 -- 172.21.15.67:0/3495546184 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 39) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fc2e00105d0 con 0x7fc2f0102780 2024-04-22T23:28:49.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.058+0000 7fc2e6ffd700 1 --2- 172.21.15.67:0/3495546184 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc2dc06ee00 0x7fc2dc0712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:49.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.058+0000 7fc2e6ffd700 1 -- 172.21.15.67:0/3495546184 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fc2e008ed90 con 0x7fc2f0102780 2024-04-22T23:28:49.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.058+0000 7fc2f5f52700 1 --2- 172.21.15.67:0/3495546184 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc2dc06ee00 0x7fc2dc0712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:49.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.058+0000 7fc2f5f52700 1 --2- 172.21.15.67:0/3495546184 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc2dc06ee00 0x7fc2dc0712c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec005b40 tx=0x7fc2ec01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:28:49.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.062+0000 7fc2e6ffd700 1 -- 172.21.15.67:0/3495546184 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc2e0057a00 con 0x7fc2f0102780 2024-04-22T23:28:49.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.317+0000 7fc2f81b6700 1 -- 172.21.15.67:0/3495546184 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc2f004f2e0 con 0x7fc2f0102780 2024-04-22T23:28:49.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.317+0000 7fc2e6ffd700 1 -- 172.21.15.67:0/3495546184 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1901 (secure 0 0 0) 0x7fc2e0017780 con 0x7fc2f0102780 2024-04-22T23:28:49.316 INFO:teuthology.orchestra.run.smithi067.stdout:e22 2024-04-22T23:28:49.316 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:28:49.316 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:28:49.316 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:28:49.316 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:28:49.316 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:28:49.316 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:28:49.316 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 22 2024-04-22T23:28:49.316 INFO:teuthology.orchestra.run.smithi067.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T23:28:49.316 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:28:49.317 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:27:20.577198+0000 2024-04-22T23:28:49.317 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:28:49.317 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:28:49.317 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:28:49.317 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:28:49.317 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:28:49.317 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:28:49.317 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:28:49.317 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:28:49.317 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 46 2024-04-22T23:28:49.317 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:28:49.317 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:28:49.317 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:28:49.317 INFO:teuthology.orchestra.run.smithi067.stdout:up {0=14484,1=24273} 2024-04-22T23:28:49.317 INFO:teuthology.orchestra.run.smithi067.stdout:failed 2024-04-22T23:28:49.317 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:28:49.317 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:28:49.317 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:28:49.318 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:28:49.318 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:28:49.318 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:28:49.318 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:28:49.318 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:28:49.318 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:28:49.318 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{0:14484} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.67:6828/3606151860,v1:172.21.15.67:6829/3606151860] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:28:49.318 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{0:14496} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/3305304988,v1:172.21.15.67:6827/3305304988] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:28:49.318 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{1:24273} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.134:6824/535957142,v1:172.21.15.134:6825/535957142] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:28:49.318 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{1:24289} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.134:6826/1954821676,v1:172.21.15.134:6827/1954821676] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:28:49.318 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:28:49.319 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:28:49.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.319+0000 7fc2f81b6700 1 -- 172.21.15.67:0/3495546184 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc2dc06ee00 msgr2=0x7fc2dc0712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:49.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.319+0000 7fc2f81b6700 1 --2- 172.21.15.67:0/3495546184 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc2dc06ee00 0x7fc2dc0712c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec005b40 tx=0x7fc2ec01a040 comp rx=0 tx=0).stop 2024-04-22T23:28:49.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.319+0000 7fc2f81b6700 1 -- 172.21.15.67:0/3495546184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2f0102780 msgr2=0x7fc2f0100000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:49.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.319+0000 7fc2f81b6700 1 --2- 172.21.15.67:0/3495546184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2f0102780 0x7fc2f0100000 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc2e000d900 tx=0x7fc2e000dcc0 comp rx=0 tx=0).stop 2024-04-22T23:28:49.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.319+0000 7fc2f81b6700 1 -- 172.21.15.67:0/3495546184 shutdown_connections 2024-04-22T23:28:49.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.319+0000 7fc2f81b6700 1 --2- 172.21.15.67:0/3495546184 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc2dc06ee00 0x7fc2dc0712c0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:49.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.319+0000 7fc2f81b6700 1 --2- 172.21.15.67:0/3495546184 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2f0101e60 0x7fc2f00ffac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:49.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.319+0000 7fc2f81b6700 1 --2- 172.21.15.67:0/3495546184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2f0102780 0x7fc2f0100000 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:49.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.319+0000 7fc2f81b6700 1 -- 172.21.15.67:0/3495546184 >> 172.21.15.67:0/3495546184 conn(0x7fc2f00fb650 msgr2=0x7fc2f0105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:28:49.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.319+0000 7fc2f81b6700 1 -- 172.21.15.67:0/3495546184 shutdown_connections 2024-04-22T23:28:49.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.319+0000 7fc2f81b6700 1 -- 172.21.15.67:0/3495546184 wait complete. 2024-04-22T23:28:49.320 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 22 2024-04-22T23:28:49.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.427+0000 7fba34c4b700 1 -- 172.21.15.67:0/3337657784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba30106db0 msgr2=0x7fba30107190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:49.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.427+0000 7fba34c4b700 1 --2- 172.21.15.67:0/3337657784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba30106db0 0x7fba30107190 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fba20009910 tx=0x7fba20009c20 comp rx=0 tx=0).stop 2024-04-22T23:28:49.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.427+0000 7fba34c4b700 1 -- 172.21.15.67:0/3337657784 shutdown_connections 2024-04-22T23:28:49.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.427+0000 7fba34c4b700 1 --2- 172.21.15.67:0/3337657784 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba300ff410 0x7fba300ff890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:49.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.427+0000 7fba34c4b700 1 --2- 172.21.15.67:0/3337657784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba30106db0 0x7fba30107190 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:49.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.427+0000 7fba34c4b700 1 -- 172.21.15.67:0/3337657784 >> 172.21.15.67:0/3337657784 conn(0x7fba30074e30 msgr2=0x7fba30075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:28:49.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.427+0000 7fba34c4b700 1 -- 172.21.15.67:0/3337657784 shutdown_connections 2024-04-22T23:28:49.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.427+0000 7fba34c4b700 1 -- 172.21.15.67:0/3337657784 wait complete. 2024-04-22T23:28:49.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.428+0000 7fba34c4b700 1 Processor -- start 2024-04-22T23:28:49.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.428+0000 7fba34c4b700 1 -- start start 2024-04-22T23:28:49.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.428+0000 7fba34c4b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba300ff410 0x7fba3019c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:49.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.428+0000 7fba34c4b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba30106db0 0x7fba3019cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:49.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.428+0000 7fba34c4b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3019d3f0 con 0x7fba300ff410 2024-04-22T23:28:49.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.428+0000 7fba34c4b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba301968a0 con 0x7fba30106db0 2024-04-22T23:28:49.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.428+0000 7fba2e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba300ff410 0x7fba3019c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:49.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.428+0000 7fba2e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba300ff410 0x7fba3019c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39246/0 (socket says 172.21.15.67:39246) 2024-04-22T23:28:49.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.428+0000 7fba2e59c700 1 -- 172.21.15.67:0/600290309 learned_addr learned my addr 172.21.15.67:0/600290309 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:28:49.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.428+0000 7fba2dd9b700 1 --2- 172.21.15.67:0/600290309 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba30106db0 0x7fba3019cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:49.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.428+0000 7fba2e59c700 1 -- 172.21.15.67:0/600290309 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba30106db0 msgr2=0x7fba3019cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:49.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.428+0000 7fba2e59c700 1 --2- 172.21.15.67:0/600290309 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba30106db0 0x7fba3019cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:49.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.428+0000 7fba2e59c700 1 -- 172.21.15.67:0/600290309 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba200095f0 con 0x7fba300ff410 2024-04-22T23:28:49.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.429+0000 7fba2e59c700 1 --2- 172.21.15.67:0/600290309 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba300ff410 0x7fba3019c820 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fba20006010 tx=0x7fba2000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:28:49.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.429+0000 7fba1f7fe700 1 -- 172.21.15.67:0/600290309 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba2001d070 con 0x7fba300ff410 2024-04-22T23:28:49.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.429+0000 7fba1f7fe700 1 -- 172.21.15.67:0/600290309 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fba20004fb0 con 0x7fba300ff410 2024-04-22T23:28:49.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.429+0000 7fba34c4b700 1 -- 172.21.15.67:0/600290309 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba30196b20 con 0x7fba300ff410 2024-04-22T23:28:49.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.429+0000 7fba1f7fe700 1 -- 172.21.15.67:0/600290309 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba20022ea0 con 0x7fba300ff410 2024-04-22T23:28:49.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.429+0000 7fba34c4b700 1 -- 172.21.15.67:0/600290309 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba30197010 con 0x7fba300ff410 2024-04-22T23:28:49.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.430+0000 7fba34c4b700 1 -- 172.21.15.67:0/600290309 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba30109300 con 0x7fba300ff410 2024-04-22T23:28:49.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.430+0000 7fba1f7fe700 1 -- 172.21.15.67:0/600290309 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 39) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fba20022470 con 0x7fba300ff410 2024-04-22T23:28:49.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.431+0000 7fba1f7fe700 1 --2- 172.21.15.67:0/600290309 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fba1806ee00 0x7fba180712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:49.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.431+0000 7fba1f7fe700 1 -- 172.21.15.67:0/600290309 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fba20090570 con 0x7fba300ff410 2024-04-22T23:28:49.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.431+0000 7fba2dd9b700 1 --2- 172.21.15.67:0/600290309 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fba1806ee00 0x7fba180712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:49.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.431+0000 7fba2dd9b700 1 --2- 172.21.15.67:0/600290309 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fba1806ee00 0x7fba180712c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fba24006fd0 tx=0x7fba24008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:28:49.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.436+0000 7fba1f7fe700 1 -- 172.21.15.67:0/600290309 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fba2005cdc0 con 0x7fba300ff410 2024-04-22T23:28:49.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.666+0000 7fba34c4b700 1 -- 172.21.15.67:0/600290309 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fba30197db0 con 0x7fba1806ee00 2024-04-22T23:28:49.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.668+0000 7fba1f7fe700 1 -- 172.21.15.67:0/600290309 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7fba30197db0 con 0x7fba1806ee00 2024-04-22T23:28:49.666 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:28:49.666 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:28:49.666 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:28:49.667 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:28:49.667 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:28:49.667 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:28:49.667 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr" 2024-04-22T23:28:49.667 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:28:49.667 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "4/23 daemons upgraded", 2024-04-22T23:28:49.667 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mon daemons", 2024-04-22T23:28:49.667 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:28:49.667 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:28:49.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.670+0000 7fba34c4b700 1 -- 172.21.15.67:0/600290309 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fba1806ee00 msgr2=0x7fba180712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:49.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.670+0000 7fba34c4b700 1 --2- 172.21.15.67:0/600290309 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fba1806ee00 0x7fba180712c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fba24006fd0 tx=0x7fba24008040 comp rx=0 tx=0).stop 2024-04-22T23:28:49.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.670+0000 7fba34c4b700 1 -- 172.21.15.67:0/600290309 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba300ff410 msgr2=0x7fba3019c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:49.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.670+0000 7fba34c4b700 1 --2- 172.21.15.67:0/600290309 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba300ff410 0x7fba3019c820 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fba20006010 tx=0x7fba2000be20 comp rx=0 tx=0).stop 2024-04-22T23:28:49.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.670+0000 7fba34c4b700 1 -- 172.21.15.67:0/600290309 shutdown_connections 2024-04-22T23:28:49.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.670+0000 7fba34c4b700 1 --2- 172.21.15.67:0/600290309 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fba1806ee00 0x7fba180712c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:49.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.670+0000 7fba34c4b700 1 --2- 172.21.15.67:0/600290309 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba300ff410 0x7fba3019c820 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:49.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.670+0000 7fba34c4b700 1 --2- 172.21.15.67:0/600290309 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba30106db0 0x7fba3019cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:49.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.670+0000 7fba34c4b700 1 -- 172.21.15.67:0/600290309 >> 172.21.15.67:0/600290309 conn(0x7fba30074e30 msgr2=0x7fba300fd460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:28:49.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.670+0000 7fba34c4b700 1 -- 172.21.15.67:0/600290309 shutdown_connections 2024-04-22T23:28:49.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.670+0000 7fba34c4b700 1 -- 172.21.15.67:0/600290309 wait complete. 2024-04-22T23:28:49.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:49 smithi134 ceph-mon[85630]: pgmap v29: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:28:49.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:49 smithi134 ceph-mon[85630]: from='client.34164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:28:49.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:49 smithi134 ceph-mon[85630]: Reconfiguring mds.cephfs.smithi134.vuxbsj (monmap changed)... 2024-04-22T23:28:49.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:49 smithi134 ceph-mon[85630]: Reconfiguring daemon mds.cephfs.smithi134.vuxbsj on smithi134 2024-04-22T23:28:49.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:49 smithi134 ceph-mon[85630]: from='client.34168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:28:49.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:49 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3027056565' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:28:49.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:49 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3495546184' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:28:49.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.779+0000 7f9011740700 1 -- 172.21.15.67:0/741175214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f900c105bd0 msgr2=0x7f900c105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:49.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.779+0000 7f9011740700 1 --2- 172.21.15.67:0/741175214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f900c105bd0 0x7f900c105fb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc009910 tx=0x7f8ffc009c20 comp rx=0 tx=0).stop 2024-04-22T23:28:49.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.779+0000 7f9011740700 1 -- 172.21.15.67:0/741175214 shutdown_connections 2024-04-22T23:28:49.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.779+0000 7f9011740700 1 --2- 172.21.15.67:0/741175214 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f900c0ffc20 0x7f900c100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:49.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.779+0000 7f9011740700 1 --2- 172.21.15.67:0/741175214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f900c105bd0 0x7f900c105fb0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:49.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.779+0000 7f9011740700 1 -- 172.21.15.67:0/741175214 >> 172.21.15.67:0/741175214 conn(0x7f900c075670 msgr2=0x7f900c075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:28:49.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.779+0000 7f9011740700 1 -- 172.21.15.67:0/741175214 shutdown_connections 2024-04-22T23:28:49.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.779+0000 7f9011740700 1 -- 172.21.15.67:0/741175214 wait complete. 2024-04-22T23:28:49.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.780+0000 7f9011740700 1 Processor -- start 2024-04-22T23:28:49.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.780+0000 7f9011740700 1 -- start start 2024-04-22T23:28:49.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.780+0000 7f9011740700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f900c0ffc20 0x7f900c19c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:49.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.780+0000 7f9011740700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f900c105bd0 0x7f900c19cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:49.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.780+0000 7f9011740700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f900c19d220 con 0x7f900c105bd0 2024-04-22T23:28:49.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.780+0000 7f9011740700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f900c1966a0 con 0x7f900c0ffc20 2024-04-22T23:28:49.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.780+0000 7f9003fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f900c105bd0 0x7f900c19cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:49.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.780+0000 7f9003fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f900c105bd0 0x7f900c19cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39252/0 (socket says 172.21.15.67:39252) 2024-04-22T23:28:49.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.780+0000 7f9003fff700 1 -- 172.21.15.67:0/189419756 learned_addr learned my addr 172.21.15.67:0/189419756 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:28:49.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.780+0000 7f900affd700 1 --2- 172.21.15.67:0/189419756 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f900c0ffc20 0x7f900c19c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:49.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.780+0000 7f9003fff700 1 -- 172.21.15.67:0/189419756 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f900c0ffc20 msgr2=0x7f900c19c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:49.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.780+0000 7f9003fff700 1 --2- 172.21.15.67:0/189419756 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f900c0ffc20 0x7f900c19c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:49.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.780+0000 7f9003fff700 1 -- 172.21.15.67:0/189419756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ffc0095f0 con 0x7f900c105bd0 2024-04-22T23:28:49.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.781+0000 7f9003fff700 1 --2- 172.21.15.67:0/189419756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f900c105bd0 0x7f900c19cb40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8ff400c8a0 tx=0x7f8ff400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:28:49.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.781+0000 7f9008ff9700 1 -- 172.21.15.67:0/189419756 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ff400cea0 con 0x7f900c105bd0 2024-04-22T23:28:49.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.781+0000 7f9008ff9700 1 -- 172.21.15.67:0/189419756 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ff4004830 con 0x7f900c105bd0 2024-04-22T23:28:49.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.781+0000 7f9011740700 1 -- 172.21.15.67:0/189419756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f900c196980 con 0x7f900c105bd0 2024-04-22T23:28:49.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.781+0000 7f9008ff9700 1 -- 172.21.15.67:0/189419756 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ff40055b0 con 0x7f900c105bd0 2024-04-22T23:28:49.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.781+0000 7f9011740700 1 -- 172.21.15.67:0/189419756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f900c196ed0 con 0x7f900c105bd0 2024-04-22T23:28:49.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.782+0000 7f9011740700 1 -- 172.21.15.67:0/189419756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f900c04f2e0 con 0x7f900c105bd0 2024-04-22T23:28:49.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.782+0000 7f9008ff9700 1 -- 172.21.15.67:0/189419756 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 39) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f8ff40049a0 con 0x7f900c105bd0 2024-04-22T23:28:49.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.783+0000 7f9008ff9700 1 --2- 172.21.15.67:0/189419756 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8ff806edf0 0x7f8ff80712b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:28:49.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.783+0000 7f9008ff9700 1 -- 172.21.15.67:0/189419756 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f8ff408ed20 con 0x7f900c105bd0 2024-04-22T23:28:49.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.783+0000 7f900affd700 1 --2- 172.21.15.67:0/189419756 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8ff806edf0 0x7f8ff80712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:28:49.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.783+0000 7f900affd700 1 --2- 172.21.15.67:0/189419756 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8ff806edf0 0x7f8ff80712b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc006010 tx=0x7f8ffc01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:28:49.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:49.788+0000 7f9008ff9700 1 -- 172.21.15.67:0/189419756 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f8ff405b440 con 0x7f900c105bd0 2024-04-22T23:28:50.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:49 smithi067 ceph-mon[112438]: pgmap v29: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:28:50.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:49 smithi067 ceph-mon[112438]: from='client.34164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:28:50.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:49 smithi067 ceph-mon[112438]: Reconfiguring mds.cephfs.smithi134.vuxbsj (monmap changed)... 2024-04-22T23:28:50.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:49 smithi067 ceph-mon[112438]: Reconfiguring daemon mds.cephfs.smithi134.vuxbsj on smithi134 2024-04-22T23:28:50.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:49 smithi067 ceph-mon[112438]: from='client.34168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:28:50.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:49 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3027056565' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:28:50.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:49 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3495546184' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:28:50.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:50.086+0000 7f9011740700 1 -- 172.21.15.67:0/189419756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f900c197bd0 con 0x7f900c105bd0 2024-04-22T23:28:50.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:50.086+0000 7f9008ff9700 1 -- 172.21.15.67:0/189419756 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f8ff401d0e0 con 0x7f900c105bd0 2024-04-22T23:28:50.085 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T23:28:50.085 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:28:50.085 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:28:50.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:50.088+0000 7f9011740700 1 -- 172.21.15.67:0/189419756 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8ff806edf0 msgr2=0x7f8ff80712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:50.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:50.088+0000 7f9011740700 1 --2- 172.21.15.67:0/189419756 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8ff806edf0 0x7f8ff80712b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc006010 tx=0x7f8ffc01a040 comp rx=0 tx=0).stop 2024-04-22T23:28:50.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:50.088+0000 7f9011740700 1 -- 172.21.15.67:0/189419756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f900c105bd0 msgr2=0x7f900c19cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:28:50.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:50.088+0000 7f9011740700 1 --2- 172.21.15.67:0/189419756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f900c105bd0 0x7f900c19cb40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8ff400c8a0 tx=0x7f8ff400cc60 comp rx=0 tx=0).stop 2024-04-22T23:28:50.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:50.088+0000 7f9011740700 1 -- 172.21.15.67:0/189419756 shutdown_connections 2024-04-22T23:28:50.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:50.088+0000 7f9011740700 1 --2- 172.21.15.67:0/189419756 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8ff806edf0 0x7f8ff80712b0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:50.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:50.088+0000 7f9011740700 1 --2- 172.21.15.67:0/189419756 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f900c0ffc20 0x7f900c19c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:50.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:50.088+0000 7f9011740700 1 --2- 172.21.15.67:0/189419756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f900c105bd0 0x7f900c19cb40 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:28:50.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:50.088+0000 7f9011740700 1 -- 172.21.15.67:0/189419756 >> 172.21.15.67:0/189419756 conn(0x7f900c075670 msgr2=0x7f900c103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:28:50.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:50.089+0000 7f9011740700 1 -- 172.21.15.67:0/189419756 shutdown_connections 2024-04-22T23:28:50.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:28:50.089+0000 7f9011740700 1 -- 172.21.15.67:0/189419756 wait complete. 2024-04-22T23:28:50.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:50 smithi134 ceph-mon[85630]: from='client.34180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:28:50.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:50 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/189419756' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:28:50.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:50 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:50.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:50 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:50.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:50 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:28:50.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:50 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:28:50.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:50 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:28:50.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:50 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:50.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:50 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi067"}]: dispatch 2024-04-22T23:28:50.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:50 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi067"}]': finished 2024-04-22T23:28:50.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:50 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi134"}]: dispatch 2024-04-22T23:28:50.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:50 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi134"}]': finished 2024-04-22T23:28:51.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:50 smithi067 ceph-mon[112438]: from='client.34180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:28:51.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:50 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/189419756' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:28:51.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:50 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:51.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:50 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:51.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:50 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:28:51.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:50 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:28:51.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:50 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:28:51.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:50 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:51.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:50 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi067"}]: dispatch 2024-04-22T23:28:51.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:50 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi067"}]': finished 2024-04-22T23:28:51.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:50 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi134"}]: dispatch 2024-04-22T23:28:51.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:50 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi134"}]': finished 2024-04-22T23:28:51.893 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:51 smithi067 ceph-mon[112438]: pgmap v30: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:28:51.894 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:51 smithi067 ceph-mon[112438]: Upgrade: Setting container_image for all mon 2024-04-22T23:28:51.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:51 smithi134 ceph-mon[85630]: pgmap v30: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:28:51.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:51 smithi134 ceph-mon[85630]: Upgrade: Setting container_image for all mon 2024-04-22T23:28:53.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:52 smithi134 ceph-mon[85630]: Upgrade: Updating crash.smithi067 (1/2) 2024-04-22T23:28:53.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:52 smithi134 ceph-mon[85630]: pgmap v31: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:28:53.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:52 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:53.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:52 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi067", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T23:28:53.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:52 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:53.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:52 smithi134 ceph-mon[85630]: Deploying daemon crash.smithi067 on smithi067 2024-04-22T23:28:53.250 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:52 smithi067 ceph-mon[112438]: Upgrade: Updating crash.smithi067 (1/2) 2024-04-22T23:28:53.250 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:52 smithi067 ceph-mon[112438]: pgmap v31: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:28:53.250 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:52 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:53.250 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:52 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi067", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T23:28:53.250 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:52 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:53.250 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:52 smithi067 ceph-mon[112438]: Deploying daemon crash.smithi067 on smithi067 2024-04-22T23:28:55.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:54 smithi134 ceph-mon[85630]: pgmap v32: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:28:55.255 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:54 smithi067 ceph-mon[112438]: pgmap v32: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:28:56.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:56.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:56.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:28:57.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:57.020 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:57.020 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:28:57.911 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:57 smithi134 ceph-mon[85630]: pgmap v33: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:28:57.911 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:57 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:57.912 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:57 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi134", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T23:28:57.912 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:57 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:58.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:57 smithi067 ceph-mon[112438]: pgmap v33: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:28:58.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:57 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:28:58.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:57 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi134", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-22T23:28:58.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:57 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:28:58.873 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:58 smithi134 ceph-mon[85630]: Upgrade: Updating crash.smithi134 (2/2) 2024-04-22T23:28:58.874 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:58 smithi134 ceph-mon[85630]: Deploying daemon crash.smithi134 on smithi134 2024-04-22T23:28:59.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:58 smithi067 ceph-mon[112438]: Upgrade: Updating crash.smithi134 (2/2) 2024-04-22T23:28:59.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:58 smithi067 ceph-mon[112438]: Deploying daemon crash.smithi134 on smithi134 2024-04-22T23:28:59.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:28:59 smithi134 ceph-mon[85630]: pgmap v34: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:29:00.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:28:59 smithi067 ceph-mon[112438]: pgmap v34: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:29:01.889 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:01 smithi067 ceph-mon[112438]: pgmap v35: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:29:01.890 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:01 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:01.890 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:01 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:01.890 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:01 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:29:01.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:01 smithi134 ceph-mon[85630]: pgmap v35: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:29:01.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:01 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:01.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:01 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:01.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:01 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:29:03.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:03 smithi134 ceph-mon[85630]: pgmap v36: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:29:03.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:03 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:03.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:03 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:04.032 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:03 smithi067 ceph-mon[112438]: pgmap v36: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:29:04.033 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:03 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:04.033 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:03 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:05.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:05 smithi134 ceph-mon[85630]: pgmap v37: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T23:29:05.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:05 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:05.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:05 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:06.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:05 smithi067 ceph-mon[112438]: pgmap v37: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T23:29:06.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:05 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:06.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:05 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:07.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:07 smithi067 ceph-mon[112438]: pgmap v38: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:29:07.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:07 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:07.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:07 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:07.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:07 smithi134 ceph-mon[85630]: pgmap v38: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:29:07.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:07 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:07.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:07 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:08.332 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:08 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:08.332 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:08 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:08.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:08 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:08.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:08 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:09.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:09 smithi134 ceph-mon[85630]: pgmap v39: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:29:09.492 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:09 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:09.492 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:09 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:09.542 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:09 smithi067 ceph-mon[112438]: pgmap v39: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:29:09.542 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:09 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:09.543 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:09 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:11.680 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:11 smithi067 ceph-mon[112438]: pgmap v40: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:29:11.680 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:11.680 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:11.680 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:29:11.680 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:29:11.680 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:11.680 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:29:11.680 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:29:11.680 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:29:11.680 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:29:11.680 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:11 smithi067 ceph-mon[112438]: Upgrade: Setting container_image for all crash 2024-04-22T23:29:11.680 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:11.681 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi067"}]: dispatch 2024-04-22T23:29:11.681 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi067"}]': finished 2024-04-22T23:29:11.681 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi134"}]: dispatch 2024-04-22T23:29:11.681 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi134"}]': finished 2024-04-22T23:29:11.681 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-22T23:29:11.681 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:11 smithi067 ceph-mon[112438]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-22T23:29:11.681 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:11 smithi067 ceph-mon[112438]: Upgrade: osd.0 is safe to restart 2024-04-22T23:29:11.681 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:29:11.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:11 smithi134 ceph-mon[85630]: pgmap v40: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:29:11.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:11.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:11.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:29:11.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:29:11.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:11.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:29:11.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:29:11.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:29:11.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:29:11.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:11 smithi134 ceph-mon[85630]: Upgrade: Setting container_image for all crash 2024-04-22T23:29:11.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:11.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi067"}]: dispatch 2024-04-22T23:29:11.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi067"}]': finished 2024-04-22T23:29:11.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi134"}]: dispatch 2024-04-22T23:29:11.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi134"}]': finished 2024-04-22T23:29:11.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-22T23:29:11.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:11 smithi134 ceph-mon[85630]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-22T23:29:11.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:11 smithi134 ceph-mon[85630]: Upgrade: osd.0 is safe to restart 2024-04-22T23:29:11.993 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:29:13.494 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:13 smithi067 ceph-mon[112438]: pgmap v41: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:29:13.494 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:13 smithi067 ceph-mon[112438]: Upgrade: Updating osd.0 2024-04-22T23:29:13.494 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:13 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:13.494 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:13 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T23:29:13.494 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:13 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:29:13.494 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:13 smithi067 ceph-mon[112438]: Deploying daemon osd.0 on smithi067 2024-04-22T23:29:13.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:13 smithi134 ceph-mon[85630]: pgmap v41: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:29:13.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:13 smithi134 ceph-mon[85630]: Upgrade: Updating osd.0 2024-04-22T23:29:13.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:13 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:13.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:13 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T23:29:13.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:13 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:29:13.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:13 smithi134 ceph-mon[85630]: Deploying daemon osd.0 on smithi067 2024-04-22T23:29:15.496 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:15 smithi067 ceph-mon[112438]: pgmap v42: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T23:29:15.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:15 smithi134 ceph-mon[85630]: pgmap v42: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-22T23:29:15.748 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:15 smithi067 systemd[1]: Stopping Ceph osd.0 for d46c346e-00fd-11ef-bc93-c7b262605968... 2024-04-22T23:29:16.042 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:15 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0[45701]: 2024-04-22T23:29:15.750+0000 7feb19745700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T23:29:16.042 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:15 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0[45701]: 2024-04-22T23:29:15.750+0000 7feb19745700 -1 osd.0 49 *** Got signal Terminated *** 2024-04-22T23:29:16.042 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:15 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0[45701]: 2024-04-22T23:29:15.750+0000 7feb19745700 -1 osd.0 49 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T23:29:16.542 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:16 smithi067 ceph-mon[112438]: osd.0 marked itself down and dead 2024-04-22T23:29:16.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:16 smithi134 ceph-mon[85630]: osd.0 marked itself down and dead 2024-04-22T23:29:17.042 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:16 smithi067 podman[123399]: 2024-04-22 23:29:16.768219238 +0000 UTC m=+1.116058123 container died a033247ee8c20573067537ae3269f41fc713332a766ad7fa9f4abec91719fbe3 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-22T23:29:17.424 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:17 smithi067 ceph-mon[112438]: pgmap v43: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:29:17.425 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:17 smithi067 ceph-mon[112438]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-22T23:29:17.425 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:17 smithi067 ceph-mon[112438]: osdmap e50: 6 total, 5 up, 6 in 2024-04-22T23:29:17.425 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:17 smithi067 podman[123399]: 2024-04-22 23:29:17.166964779 +0000 UTC m=+1.514803671 container cleanup a033247ee8c20573067537ae3269f41fc713332a766ad7fa9f4abec91719fbe3 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0, io.buildah.version=1.29.1, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-22T23:29:17.425 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:17 smithi067 bash[123399]: ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0 2024-04-22T23:29:17.733 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:17 smithi067 podman[123429]: 2024-04-22 23:29:17.426638132 +0000 UTC m=+0.656339456 container remove a033247ee8c20573067537ae3269f41fc713332a766ad7fa9f4abec91719fbe3 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, io.buildah.version=1.29.1) 2024-04-22T23:29:17.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:17 smithi134 ceph-mon[85630]: pgmap v43: 65 pgs: 65 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:29:17.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:17 smithi134 ceph-mon[85630]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-22T23:29:17.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:17 smithi134 ceph-mon[85630]: osdmap e50: 6 total, 5 up, 6 in 2024-04-22T23:29:18.042 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:17 smithi067 podman[123519]: 2024-04-22 23:29:17.63902688 +0000 UTC m=+0.019007560 image pull a0434699c1d76af4eea1fc95ff0a22ea73d1e04e609464fbf658d6f39bc7f2fc quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c 2024-04-22T23:29:18.042 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:17 smithi067 podman[123519]: 2024-04-22 23:29:17.744693371 +0000 UTC m=+0.124674090 container create 2b4ba7b3b97582df5fe35ff4ef519526c07d8433c87f99b85025dc39a7c02860 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-22T23:29:18.043 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:17 smithi067 podman[123519]: 2024-04-22 23:29:17.963961319 +0000 UTC m=+0.343941999 container init 2b4ba7b3b97582df5fe35ff4ef519526c07d8433c87f99b85025dc39a7c02860 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416) 2024-04-22T23:29:18.043 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:17 smithi067 podman[123519]: 2024-04-22 23:29:17.96729964 +0000 UTC m=+0.347280318 container start 2b4ba7b3b97582df5fe35ff4ef519526c07d8433c87f99b85025dc39a7c02860 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.vendor=CentOS) 2024-04-22T23:29:18.542 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:18 smithi067 ceph-mon[112438]: osdmap e51: 6 total, 5 up, 6 in 2024-04-22T23:29:18.542 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:18 smithi067 podman[123519]: 2024-04-22 23:29:18.22769779 +0000 UTC m=+0.607678513 container attach 2b4ba7b3b97582df5fe35ff4ef519526c07d8433c87f99b85025dc39a7c02860 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0-deactivate, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-22T23:29:18.542 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:18 smithi067 podman[123519]: 2024-04-22 23:29:18.277231669 +0000 UTC m=+0.657212347 container died 2b4ba7b3b97582df5fe35ff4ef519526c07d8433c87f99b85025dc39a7c02860 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-22T23:29:18.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:18 smithi134 ceph-mon[85630]: osdmap e51: 6 total, 5 up, 6 in 2024-04-22T23:29:19.293 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:18 smithi067 podman[123545]: 2024-04-22 23:29:18.834312661 +0000 UTC m=+0.550710784 container remove 2b4ba7b3b97582df5fe35ff4ef519526c07d8433c87f99b85025dc39a7c02860 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS) 2024-04-22T23:29:19.293 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:18 smithi067 systemd[1]: ceph-d46c346e-00fd-11ef-bc93-c7b262605968@osd.0.service: Deactivated successfully. 2024-04-22T23:29:19.293 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:18 smithi067 systemd[1]: Stopped Ceph osd.0 for d46c346e-00fd-11ef-bc93-c7b262605968. 2024-04-22T23:29:19.293 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:18 smithi067 systemd[1]: ceph-d46c346e-00fd-11ef-bc93-c7b262605968@osd.0.service: Consumed 57.186s CPU time. 2024-04-22T23:29:19.293 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:19 smithi067 systemd[1]: Starting Ceph osd.0 for d46c346e-00fd-11ef-bc93-c7b262605968... 2024-04-22T23:29:19.542 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:19 smithi067 ceph-mon[112438]: pgmap v46: 65 pgs: 34 peering, 31 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:29:19.543 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:19 smithi067 ceph-mon[112438]: Health check failed: Reduced data availability: 24 pgs peering (PG_AVAILABILITY) 2024-04-22T23:29:19.543 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:19 smithi067 podman[123648]: 2024-04-22 23:29:19.294127593 +0000 UTC m=+0.115019558 container create 08d82862e8720239662ad96a62ae1c6a9159e482d7ef850e09116b604eff788a (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0-activate, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T23:29:19.543 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:19 smithi067 podman[123648]: 2024-04-22 23:29:19.197798789 +0000 UTC m=+0.018690783 image pull a0434699c1d76af4eea1fc95ff0a22ea73d1e04e609464fbf658d6f39bc7f2fc quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c 2024-04-22T23:29:19.543 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:19 smithi067 podman[123648]: 2024-04-22 23:29:19.505622915 +0000 UTC m=+0.326514892 container init 08d82862e8720239662ad96a62ae1c6a9159e482d7ef850e09116b604eff788a (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0-activate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-22T23:29:19.543 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:19 smithi067 podman[123648]: 2024-04-22 23:29:19.509240782 +0000 UTC m=+0.330132756 container start 08d82862e8720239662ad96a62ae1c6a9159e482d7ef850e09116b604eff788a (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0-activate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-22T23:29:19.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:19 smithi134 ceph-mon[85630]: pgmap v46: 65 pgs: 34 peering, 31 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:29:19.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:19 smithi134 ceph-mon[85630]: Health check failed: Reduced data availability: 24 pgs peering (PG_AVAILABILITY) 2024-04-22T23:29:19.984 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:19 smithi067 podman[123648]: 2024-04-22 23:29:19.597155695 +0000 UTC m=+0.418047661 container attach 08d82862e8720239662ad96a62ae1c6a9159e482d7ef850e09116b604eff788a (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0-activate, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-22T23:29:20.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.200+0000 7f3482193700 1 -- 172.21.15.67:0/2407238533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f347c105bd0 msgr2=0x7f347c105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:20.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.200+0000 7f3482193700 1 --2- 172.21.15.67:0/2407238533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f347c105bd0 0x7f347c105fb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f346c009910 tx=0x7f346c009c20 comp rx=0 tx=0).stop 2024-04-22T23:29:20.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.200+0000 7f3482193700 1 -- 172.21.15.67:0/2407238533 shutdown_connections 2024-04-22T23:29:20.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.200+0000 7f3482193700 1 --2- 172.21.15.67:0/2407238533 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f347c0ffc20 0x7f347c100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:20.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.200+0000 7f3482193700 1 --2- 172.21.15.67:0/2407238533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f347c105bd0 0x7f347c105fb0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:20.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.200+0000 7f3482193700 1 -- 172.21.15.67:0/2407238533 >> 172.21.15.67:0/2407238533 conn(0x7f347c075670 msgr2=0x7f347c075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:29:20.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.200+0000 7f3482193700 1 -- 172.21.15.67:0/2407238533 shutdown_connections 2024-04-22T23:29:20.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.200+0000 7f3482193700 1 -- 172.21.15.67:0/2407238533 wait complete. 2024-04-22T23:29:20.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.201+0000 7f3482193700 1 Processor -- start 2024-04-22T23:29:20.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.201+0000 7f3482193700 1 -- start start 2024-04-22T23:29:20.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.201+0000 7f3482193700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f347c0ffc20 0x7f347c19c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:20.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.201+0000 7f3482193700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f347c105bd0 0x7f347c19cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:20.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.201+0000 7f3482193700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347c19d160 con 0x7f347c0ffc20 2024-04-22T23:29:20.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.201+0000 7f3482193700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347c1966a0 con 0x7f347c105bd0 2024-04-22T23:29:20.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.201+0000 7f347b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f347c0ffc20 0x7f347c19c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:20.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.201+0000 7f347b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f347c0ffc20 0x7f347c19c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35544/0 (socket says 172.21.15.67:35544) 2024-04-22T23:29:20.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.202+0000 7f347b7fe700 1 -- 172.21.15.67:0/82735838 learned_addr learned my addr 172.21.15.67:0/82735838 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:29:20.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.202+0000 7f347affd700 1 --2- 172.21.15.67:0/82735838 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f347c105bd0 0x7f347c19cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:20.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.202+0000 7f347b7fe700 1 -- 172.21.15.67:0/82735838 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f347c105bd0 msgr2=0x7f347c19cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:20.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.202+0000 7f347b7fe700 1 --2- 172.21.15.67:0/82735838 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f347c105bd0 0x7f347c19cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:20.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.202+0000 7f347b7fe700 1 -- 172.21.15.67:0/82735838 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f346c0095f0 con 0x7f347c0ffc20 2024-04-22T23:29:20.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.202+0000 7f347affd700 1 --2- 172.21.15.67:0/82735838 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f347c105bd0 0x7f347c19cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:29:20.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.202+0000 7f347b7fe700 1 --2- 172.21.15.67:0/82735838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f347c0ffc20 0x7f347c19c620 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f346c006010 tx=0x7f346c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:29:20.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.202+0000 7f3478ff9700 1 -- 172.21.15.67:0/82735838 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f346c01d070 con 0x7f347c0ffc20 2024-04-22T23:29:20.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.202+0000 7f3478ff9700 1 -- 172.21.15.67:0/82735838 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f346c004fb0 con 0x7f347c0ffc20 2024-04-22T23:29:20.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.202+0000 7f3482193700 1 -- 172.21.15.67:0/82735838 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f347c196920 con 0x7f347c0ffc20 2024-04-22T23:29:20.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.202+0000 7f3478ff9700 1 -- 172.21.15.67:0/82735838 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f346c022ea0 con 0x7f347c0ffc20 2024-04-22T23:29:20.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.202+0000 7f3482193700 1 -- 172.21.15.67:0/82735838 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f347c196e10 con 0x7f347c0ffc20 2024-04-22T23:29:20.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.203+0000 7f3482193700 1 -- 172.21.15.67:0/82735838 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f347c04f2e0 con 0x7f347c0ffc20 2024-04-22T23:29:20.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.204+0000 7f3478ff9700 1 -- 172.21.15.67:0/82735838 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 39) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f346c022470 con 0x7f347c0ffc20 2024-04-22T23:29:20.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.204+0000 7f3478ff9700 1 --2- 172.21.15.67:0/82735838 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f346406ee00 0x7f34640712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:20.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.204+0000 7f3478ff9700 1 -- 172.21.15.67:0/82735838 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f346c090620 con 0x7f347c0ffc20 2024-04-22T23:29:20.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.204+0000 7f347affd700 1 --2- 172.21.15.67:0/82735838 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f346406ee00 0x7f34640712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:20.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.205+0000 7f347affd700 1 --2- 172.21.15.67:0/82735838 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f346406ee00 0x7f34640712c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3470005fd0 tx=0x7f3470005e20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:29:20.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.209+0000 7f3478ff9700 1 -- 172.21.15.67:0/82735838 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f346c05cdf0 con 0x7f347c0ffc20 2024-04-22T23:29:20.292 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:20 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0-activate[123663]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T23:29:20.293 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:20 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0-activate[123663]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T23:29:20.293 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:20 smithi067 bash[123648]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T23:29:20.293 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:20 smithi067 bash[123648]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T23:29:20.293 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:20 smithi067 bash[123648]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-22T23:29:20.293 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:20 smithi067 bash[123648]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-22T23:29:20.293 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:20 smithi067 bash[123648]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-22T23:29:20.293 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:20 smithi067 bash[123648]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-22T23:29:20.293 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:20 smithi067 bash[123648]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-22T23:29:20.293 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:20 smithi067 bash[123648]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-22T23:29:20.293 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:20 smithi067 bash[123648]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-22T23:29:20.293 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:20 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0-activate[123663]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-22T23:29:20.293 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:20 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0-activate[123663]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-22T23:29:20.294 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:20 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0-activate[123663]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-22T23:29:20.294 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:20 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0-activate[123663]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-22T23:29:20.294 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:20 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0-activate[123663]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-22T23:29:20.294 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:20 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0-activate[123663]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-22T23:29:20.294 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:20 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0-activate[123663]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-22T23:29:20.294 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:20 smithi067 podman[123648]: 2024-04-22 23:29:20.035366799 +0000 UTC m=+0.856258776 container died 08d82862e8720239662ad96a62ae1c6a9159e482d7ef850e09116b604eff788a (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0-activate, org.label-schema.build-date=20240416, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-22T23:29:20.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.442+0000 7f3482193700 1 -- 172.21.15.67:0/82735838 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f347c1970f0 con 0x7f346406ee00 2024-04-22T23:29:20.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.443+0000 7f3478ff9700 1 -- 172.21.15.67:0/82735838 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f347c1970f0 con 0x7f346406ee00 2024-04-22T23:29:20.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.445+0000 7f3482193700 1 -- 172.21.15.67:0/82735838 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f346406ee00 msgr2=0x7f34640712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:20.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.445+0000 7f3482193700 1 --2- 172.21.15.67:0/82735838 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f346406ee00 0x7f34640712c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3470005fd0 tx=0x7f3470005e20 comp rx=0 tx=0).stop 2024-04-22T23:29:20.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.445+0000 7f3482193700 1 -- 172.21.15.67:0/82735838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f347c0ffc20 msgr2=0x7f347c19c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:20.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.445+0000 7f3482193700 1 --2- 172.21.15.67:0/82735838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f347c0ffc20 0x7f347c19c620 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f346c006010 tx=0x7f346c00be20 comp rx=0 tx=0).stop 2024-04-22T23:29:20.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.445+0000 7f3482193700 1 -- 172.21.15.67:0/82735838 shutdown_connections 2024-04-22T23:29:20.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.445+0000 7f3482193700 1 --2- 172.21.15.67:0/82735838 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f346406ee00 0x7f34640712c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:20.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.445+0000 7f3482193700 1 --2- 172.21.15.67:0/82735838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f347c0ffc20 0x7f347c19c620 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:20.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.445+0000 7f3482193700 1 --2- 172.21.15.67:0/82735838 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f347c105bd0 0x7f347c19cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:20.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.445+0000 7f3482193700 1 -- 172.21.15.67:0/82735838 >> 172.21.15.67:0/82735838 conn(0x7f347c075670 msgr2=0x7f347c0686c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:29:20.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.445+0000 7f3482193700 1 -- 172.21.15.67:0/82735838 shutdown_connections 2024-04-22T23:29:20.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.445+0000 7f3482193700 1 -- 172.21.15.67:0/82735838 wait complete. 2024-04-22T23:29:20.454 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:29:20.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.554+0000 7f2510014700 1 -- 172.21.15.67:0/1317071181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2508100ea0 msgr2=0x7f2508101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:20.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.554+0000 7f2510014700 1 --2- 172.21.15.67:0/1317071181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2508100ea0 0x7f2508101280 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f250000b0a0 tx=0x7f250000b3b0 comp rx=0 tx=0).stop 2024-04-22T23:29:20.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.554+0000 7f2510014700 1 -- 172.21.15.67:0/1317071181 shutdown_connections 2024-04-22T23:29:20.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.554+0000 7f2510014700 1 --2- 172.21.15.67:0/1317071181 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2508101850 0x7f25081058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:20.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.554+0000 7f2510014700 1 --2- 172.21.15.67:0/1317071181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2508100ea0 0x7f2508101280 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:20.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.554+0000 7f2510014700 1 -- 172.21.15.67:0/1317071181 >> 172.21.15.67:0/1317071181 conn(0x7f25080fc750 msgr2=0x7f25080feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:29:20.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.554+0000 7f2510014700 1 -- 172.21.15.67:0/1317071181 shutdown_connections 2024-04-22T23:29:20.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.554+0000 7f2510014700 1 -- 172.21.15.67:0/1317071181 wait complete. 2024-04-22T23:29:20.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.555+0000 7f2510014700 1 Processor -- start 2024-04-22T23:29:20.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.555+0000 7f2510014700 1 -- start start 2024-04-22T23:29:20.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.555+0000 7f2510014700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2508100ea0 0x7f250819c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:20.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.555+0000 7f2510014700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2508101850 0x7f250819cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:20.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.555+0000 7f2510014700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250819d2c0 con 0x7f2508100ea0 2024-04-22T23:29:20.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.555+0000 7f2510014700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2508196770 con 0x7f2508101850 2024-04-22T23:29:20.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.555+0000 7f250ddb0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2508100ea0 0x7f250819c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:20.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.555+0000 7f250ddb0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2508100ea0 0x7f250819c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35566/0 (socket says 172.21.15.67:35566) 2024-04-22T23:29:20.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.555+0000 7f250ddb0700 1 -- 172.21.15.67:0/1587458766 learned_addr learned my addr 172.21.15.67:0/1587458766 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:29:20.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.555+0000 7f250d5af700 1 --2- 172.21.15.67:0/1587458766 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2508101850 0x7f250819cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:20.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.555+0000 7f250ddb0700 1 -- 172.21.15.67:0/1587458766 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2508101850 msgr2=0x7f250819cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:20.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.555+0000 7f250ddb0700 1 --2- 172.21.15.67:0/1587458766 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2508101850 0x7f250819cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:20.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.555+0000 7f250ddb0700 1 -- 172.21.15.67:0/1587458766 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2500009d00 con 0x7f2508100ea0 2024-04-22T23:29:20.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.555+0000 7f250d5af700 1 --2- 172.21.15.67:0/1587458766 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2508101850 0x7f250819cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:29:20.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.555+0000 7f250ddb0700 1 --2- 172.21.15.67:0/1587458766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2508100ea0 0x7f250819c6f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2500000f80 tx=0x7f250000cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:29:20.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.556+0000 7f2506ffd700 1 -- 172.21.15.67:0/1587458766 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f250000e040 con 0x7f2508100ea0 2024-04-22T23:29:20.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.556+0000 7f2506ffd700 1 -- 172.21.15.67:0/1587458766 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2500003de0 con 0x7f2508100ea0 2024-04-22T23:29:20.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.556+0000 7f2510014700 1 -- 172.21.15.67:0/1587458766 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25081969f0 con 0x7f2508100ea0 2024-04-22T23:29:20.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.556+0000 7f2506ffd700 1 -- 172.21.15.67:0/1587458766 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f250001d790 con 0x7f2508100ea0 2024-04-22T23:29:20.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.556+0000 7f2510014700 1 -- 172.21.15.67:0/1587458766 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2508196ee0 con 0x7f2508100ea0 2024-04-22T23:29:20.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.557+0000 7f2510014700 1 -- 172.21.15.67:0/1587458766 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2508109240 con 0x7f2508100ea0 2024-04-22T23:29:20.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.557+0000 7f2506ffd700 1 -- 172.21.15.67:0/1587458766 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 39) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f2500004040 con 0x7f2508100ea0 2024-04-22T23:29:20.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.558+0000 7f2506ffd700 1 --2- 172.21.15.67:0/1587458766 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f24f006ee00 0x7f24f00712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:20.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.558+0000 7f2506ffd700 1 -- 172.21.15.67:0/1587458766 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f2500090890 con 0x7f2508100ea0 2024-04-22T23:29:20.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.558+0000 7f250d5af700 1 --2- 172.21.15.67:0/1587458766 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f24f006ee00 0x7f24f00712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:20.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.558+0000 7f250d5af700 1 --2- 172.21.15.67:0/1587458766 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f24f006ee00 0x7f24f00712c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f24fc005950 tx=0x7f24fc0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:29:20.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.562+0000 7f2506ffd700 1 -- 172.21.15.67:0/1587458766 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f250005cfb0 con 0x7f2508100ea0 2024-04-22T23:29:20.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.802+0000 7f2510014700 1 -- 172.21.15.67:0/1587458766 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2508197db0 con 0x7f24f006ee00 2024-04-22T23:29:20.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.803+0000 7f2506ffd700 1 -- 172.21.15.67:0/1587458766 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f2508197db0 con 0x7f24f006ee00 2024-04-22T23:29:20.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.805+0000 7f2510014700 1 -- 172.21.15.67:0/1587458766 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f24f006ee00 msgr2=0x7f24f00712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:20.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.805+0000 7f2510014700 1 --2- 172.21.15.67:0/1587458766 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f24f006ee00 0x7f24f00712c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f24fc005950 tx=0x7f24fc0058e0 comp rx=0 tx=0).stop 2024-04-22T23:29:20.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.805+0000 7f2510014700 1 -- 172.21.15.67:0/1587458766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2508100ea0 msgr2=0x7f250819c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:20.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.805+0000 7f2510014700 1 --2- 172.21.15.67:0/1587458766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2508100ea0 0x7f250819c6f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2500000f80 tx=0x7f250000cdd0 comp rx=0 tx=0).stop 2024-04-22T23:29:20.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.806+0000 7f2510014700 1 -- 172.21.15.67:0/1587458766 shutdown_connections 2024-04-22T23:29:20.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.806+0000 7f2510014700 1 --2- 172.21.15.67:0/1587458766 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f24f006ee00 0x7f24f00712c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:20.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.806+0000 7f2510014700 1 --2- 172.21.15.67:0/1587458766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2508100ea0 0x7f250819c6f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:20.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.806+0000 7f2510014700 1 --2- 172.21.15.67:0/1587458766 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2508101850 0x7f250819cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:20.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.806+0000 7f2510014700 1 -- 172.21.15.67:0/1587458766 >> 172.21.15.67:0/1587458766 conn(0x7f25080fc750 msgr2=0x7f25080fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:29:20.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.806+0000 7f2510014700 1 -- 172.21.15.67:0/1587458766 shutdown_connections 2024-04-22T23:29:20.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.806+0000 7f2510014700 1 -- 172.21.15.67:0/1587458766 wait complete. 2024-04-22T23:29:20.823 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:20 smithi067 podman[123809]: 2024-04-22 23:29:20.60647969 +0000 UTC m=+0.565287059 container remove 08d82862e8720239662ad96a62ae1c6a9159e482d7ef850e09116b604eff788a (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0-activate, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-22T23:29:20.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.916+0000 7fb127bfd700 1 -- 172.21.15.67:0/3139100875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb120068580 msgr2=0x7fb1200689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:20.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.916+0000 7fb127bfd700 1 --2- 172.21.15.67:0/3139100875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb120068580 0x7fb1200689e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb11c009910 tx=0x7fb11c009c20 comp rx=0 tx=0).stop 2024-04-22T23:29:20.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.916+0000 7fb127bfd700 1 -- 172.21.15.67:0/3139100875 shutdown_connections 2024-04-22T23:29:20.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.916+0000 7fb127bfd700 1 --2- 172.21.15.67:0/3139100875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb120068580 0x7fb1200689e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:20.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.916+0000 7fb127bfd700 1 --2- 172.21.15.67:0/3139100875 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb120106cd0 0x7fb1201070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:20.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.916+0000 7fb127bfd700 1 -- 172.21.15.67:0/3139100875 >> 172.21.15.67:0/3139100875 conn(0x7fb120075670 msgr2=0x7fb120075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:29:20.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.916+0000 7fb127bfd700 1 -- 172.21.15.67:0/3139100875 shutdown_connections 2024-04-22T23:29:20.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.916+0000 7fb127bfd700 1 -- 172.21.15.67:0/3139100875 wait complete. 2024-04-22T23:29:20.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.917+0000 7fb127bfd700 1 Processor -- start 2024-04-22T23:29:20.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.917+0000 7fb127bfd700 1 -- start start 2024-04-22T23:29:20.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.917+0000 7fb127bfd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb120068580 0x7fb1200ffcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:20.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.917+0000 7fb127bfd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb120106cd0 0x7fb120100230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:20.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.917+0000 7fb127bfd700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb120100770 con 0x7fb120068580 2024-04-22T23:29:20.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.917+0000 7fb127bfd700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1201008b0 con 0x7fb120106cd0 2024-04-22T23:29:20.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.917+0000 7fb125999700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb120068580 0x7fb1200ffcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:20.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.917+0000 7fb125999700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb120068580 0x7fb1200ffcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35594/0 (socket says 172.21.15.67:35594) 2024-04-22T23:29:20.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.917+0000 7fb125999700 1 -- 172.21.15.67:0/1600792681 learned_addr learned my addr 172.21.15.67:0/1600792681 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:29:20.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.917+0000 7fb125198700 1 --2- 172.21.15.67:0/1600792681 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb120106cd0 0x7fb120100230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:20.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.918+0000 7fb125999700 1 -- 172.21.15.67:0/1600792681 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb120106cd0 msgr2=0x7fb120100230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:20.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.918+0000 7fb125999700 1 --2- 172.21.15.67:0/1600792681 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb120106cd0 0x7fb120100230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:20.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.918+0000 7fb125999700 1 -- 172.21.15.67:0/1600792681 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb11c0095f0 con 0x7fb120068580 2024-04-22T23:29:20.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.918+0000 7fb125999700 1 --2- 172.21.15.67:0/1600792681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb120068580 0x7fb1200ffcf0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb11000e920 tx=0x7fb11000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:29:20.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.918+0000 7fb116ffd700 1 -- 172.21.15.67:0/1600792681 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb11000c9d0 con 0x7fb120068580 2024-04-22T23:29:20.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.918+0000 7fb127bfd700 1 -- 172.21.15.67:0/1600792681 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1201a2a30 con 0x7fb120068580 2024-04-22T23:29:20.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.918+0000 7fb116ffd700 1 -- 172.21.15.67:0/1600792681 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb110010430 con 0x7fb120068580 2024-04-22T23:29:20.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.918+0000 7fb116ffd700 1 -- 172.21.15.67:0/1600792681 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb110018630 con 0x7fb120068580 2024-04-22T23:29:20.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.918+0000 7fb127bfd700 1 -- 172.21.15.67:0/1600792681 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1201a2e40 con 0x7fb120068580 2024-04-22T23:29:20.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.919+0000 7fb127bfd700 1 -- 172.21.15.67:0/1600792681 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1201091d0 con 0x7fb120068580 2024-04-22T23:29:20.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.920+0000 7fb116ffd700 1 -- 172.21.15.67:0/1600792681 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 39) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fb1100105a0 con 0x7fb120068580 2024-04-22T23:29:20.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.920+0000 7fb116ffd700 1 --2- 172.21.15.67:0/1600792681 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb10c06ee00 0x7fb10c0712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:20.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.920+0000 7fb116ffd700 1 -- 172.21.15.67:0/1600792681 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fb110014070 con 0x7fb120068580 2024-04-22T23:29:20.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.920+0000 7fb125198700 1 --2- 172.21.15.67:0/1600792681 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb10c06ee00 0x7fb10c0712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:20.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.921+0000 7fb125198700 1 --2- 172.21.15.67:0/1600792681 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb10c06ee00 0x7fb10c0712c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb1201014e0 tx=0x7fb11c005940 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:29:20.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:20.925+0000 7fb116ffd700 1 -- 172.21.15.67:0/1600792681 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb110054150 con 0x7fb120068580 2024-04-22T23:29:21.088 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:20 smithi067 podman[123922]: 2024-04-22 23:29:20.729426611 +0000 UTC m=+0.018772364 image pull a0434699c1d76af4eea1fc95ff0a22ea73d1e04e609464fbf658d6f39bc7f2fc quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c 2024-04-22T23:29:21.089 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:20 smithi067 podman[123922]: 2024-04-22 23:29:20.856513733 +0000 UTC m=+0.145859458 container create a74fde1ef34edcffb85b2a275ec89aef7fcb531d7ae1abe3035519a261f301e6 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-22T23:29:21.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.150+0000 7fb127bfd700 1 -- 172.21.15.67:0/1600792681 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1200611d0 con 0x7fb10c06ee00 2024-04-22T23:29:21.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.161+0000 7fb116ffd700 1 -- 172.21.15.67:0/1600792681 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fb1200611d0 con 0x7fb10c06ee00 2024-04-22T23:29:21.159 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:29:21.160 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (3m) 15s ago 14m 26.1M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:29:21.160 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (14m) 15s ago 14m 8238k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:29:21.160 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (13m) 17s ago 13m 8200k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:29:21.160 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (25s) 15s ago 14m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:29:21.160 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (20s) 17s ago 13m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:29:21.160 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (2m) 15s ago 14m 76.2M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:29:21.160 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (8m) 15s ago 8m 62.7M - 18.2.1 5be31c24972a e24a471f0a8c 2024-04-22T23:29:21.160 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (8m) 15s ago 8m 130M - 18.2.1 5be31c24972a 0730f9fc8e61 2024-04-22T23:29:21.160 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (8m) 17s ago 8m 123M - 18.2.1 5be31c24972a b6a169586433 2024-04-22T23:29:21.160 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (8m) 17s ago 8m 67.7M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:29:21.160 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (4m) 15s ago 15m 498M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:29:21.160 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (3m) 17s ago 13m 450M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:29:21.160 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (94s) 15s ago 16m 44.6M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:29:21.161 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (68s) 17s ago 13m 38.2M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:29:21.161 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (3m) 15s ago 14m 15.5M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:29:21.161 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (3m) 17s ago 13m 15.9M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:29:21.161 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (12m) 15s ago 12m 207M 1216M 18.2.1 5be31c24972a a033247ee8c2 2024-04-22T23:29:21.161 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (12m) 15s ago 12m 213M 1216M 18.2.1 5be31c24972a 915b07a8edc4 2024-04-22T23:29:21.161 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (11m) 15s ago 11m 173M 1216M 18.2.1 5be31c24972a e257ecb4efa1 2024-04-22T23:29:21.161 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (11m) 17s ago 11m 251M 2240M 18.2.1 5be31c24972a 9c2c91a967ba 2024-04-22T23:29:21.161 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (10m) 17s ago 10m 219M 2240M 18.2.1 5be31c24972a c7586562ffa6 2024-04-22T23:29:21.161 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (10m) 17s ago 10m 196M 2240M 18.2.1 5be31c24972a 9fd97f1c65cb 2024-04-22T23:29:21.161 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (3m) 15s ago 13m 71.2M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:29:21.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.163+0000 7fb127bfd700 1 -- 172.21.15.67:0/1600792681 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb10c06ee00 msgr2=0x7fb10c0712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:21.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.163+0000 7fb127bfd700 1 --2- 172.21.15.67:0/1600792681 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb10c06ee00 0x7fb10c0712c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb1201014e0 tx=0x7fb11c005940 comp rx=0 tx=0).stop 2024-04-22T23:29:21.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.163+0000 7fb127bfd700 1 -- 172.21.15.67:0/1600792681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb120068580 msgr2=0x7fb1200ffcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:21.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.163+0000 7fb127bfd700 1 --2- 172.21.15.67:0/1600792681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb120068580 0x7fb1200ffcf0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb11000e920 tx=0x7fb11000ec30 comp rx=0 tx=0).stop 2024-04-22T23:29:21.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.163+0000 7fb127bfd700 1 -- 172.21.15.67:0/1600792681 shutdown_connections 2024-04-22T23:29:21.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.163+0000 7fb127bfd700 1 --2- 172.21.15.67:0/1600792681 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb10c06ee00 0x7fb10c0712c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:21.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.163+0000 7fb127bfd700 1 --2- 172.21.15.67:0/1600792681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb120068580 0x7fb1200ffcf0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:21.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.163+0000 7fb127bfd700 1 --2- 172.21.15.67:0/1600792681 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb120106cd0 0x7fb120100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:21.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.163+0000 7fb127bfd700 1 -- 172.21.15.67:0/1600792681 >> 172.21.15.67:0/1600792681 conn(0x7fb120075670 msgr2=0x7fb1200feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:29:21.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.163+0000 7fb127bfd700 1 -- 172.21.15.67:0/1600792681 shutdown_connections 2024-04-22T23:29:21.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.163+0000 7fb127bfd700 1 -- 172.21.15.67:0/1600792681 wait complete. 2024-04-22T23:29:21.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.290+0000 7fbbcc3ca700 1 -- 172.21.15.67:0/739494892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbc40ff5a0 msgr2=0x7fbbc40ff980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:21.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.290+0000 7fbbcc3ca700 1 --2- 172.21.15.67:0/739494892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbc40ff5a0 0x7fbbc40ff980 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbbc0009910 tx=0x7fbbc0009c20 comp rx=0 tx=0).stop 2024-04-22T23:29:21.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.290+0000 7fbbcc3ca700 1 -- 172.21.15.67:0/739494892 shutdown_connections 2024-04-22T23:29:21.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.290+0000 7fbbcc3ca700 1 --2- 172.21.15.67:0/739494892 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbbc40fff50 0x7fbbc4105840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:21.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.290+0000 7fbbcc3ca700 1 --2- 172.21.15.67:0/739494892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbc40ff5a0 0x7fbbc40ff980 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:21.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.290+0000 7fbbcc3ca700 1 -- 172.21.15.67:0/739494892 >> 172.21.15.67:0/739494892 conn(0x7fbbc4074e30 msgr2=0x7fbbc4075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:29:21.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.291+0000 7fbbcc3ca700 1 -- 172.21.15.67:0/739494892 shutdown_connections 2024-04-22T23:29:21.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.291+0000 7fbbcc3ca700 1 -- 172.21.15.67:0/739494892 wait complete. 2024-04-22T23:29:21.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.291+0000 7fbbcc3ca700 1 Processor -- start 2024-04-22T23:29:21.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.291+0000 7fbbcc3ca700 1 -- start start 2024-04-22T23:29:21.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.291+0000 7fbbcc3ca700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbbc40ff5a0 0x7fbbc4100520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:21.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.291+0000 7fbbcc3ca700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbc40fff50 0x7fbbc4100a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:21.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.291+0000 7fbbcc3ca700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc4100fa0 con 0x7fbbc40fff50 2024-04-22T23:29:21.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.291+0000 7fbbcc3ca700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc41010e0 con 0x7fbbc40ff5a0 2024-04-22T23:29:21.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.292+0000 7fbbc9965700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbc40fff50 0x7fbbc4100a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:21.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.292+0000 7fbbc9965700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbc40fff50 0x7fbbc4100a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35628/0 (socket says 172.21.15.67:35628) 2024-04-22T23:29:21.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.292+0000 7fbbca166700 1 --2- 172.21.15.67:0/3430840004 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbbc40ff5a0 0x7fbbc4100520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:21.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.292+0000 7fbbc9965700 1 -- 172.21.15.67:0/3430840004 learned_addr learned my addr 172.21.15.67:0/3430840004 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:29:21.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.292+0000 7fbbc9965700 1 -- 172.21.15.67:0/3430840004 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbbc40ff5a0 msgr2=0x7fbbc4100520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:21.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.292+0000 7fbbc9965700 1 --2- 172.21.15.67:0/3430840004 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbbc40ff5a0 0x7fbbc4100520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:21.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.292+0000 7fbbc9965700 1 -- 172.21.15.67:0/3430840004 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc00095f0 con 0x7fbbc40fff50 2024-04-22T23:29:21.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.292+0000 7fbbc9965700 1 --2- 172.21.15.67:0/3430840004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbc40fff50 0x7fbbc4100a60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fbbb400d900 tx=0x7fbbb400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:29:21.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.292+0000 7fbbbb7fe700 1 -- 172.21.15.67:0/3430840004 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbbb4009900 con 0x7fbbc40fff50 2024-04-22T23:29:21.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.292+0000 7fbbbb7fe700 1 -- 172.21.15.67:0/3430840004 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbbb4010460 con 0x7fbbc40fff50 2024-04-22T23:29:21.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.292+0000 7fbbcc3ca700 1 -- 172.21.15.67:0/3430840004 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbc41a2a80 con 0x7fbbc40fff50 2024-04-22T23:29:21.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.292+0000 7fbbbb7fe700 1 -- 172.21.15.67:0/3430840004 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbbb400f5f0 con 0x7fbbc40fff50 2024-04-22T23:29:21.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.292+0000 7fbbcc3ca700 1 -- 172.21.15.67:0/3430840004 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbc41a2e90 con 0x7fbbc40fff50 2024-04-22T23:29:21.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.294+0000 7fbbcc3ca700 1 -- 172.21.15.67:0/3430840004 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbc4109270 con 0x7fbbc40fff50 2024-04-22T23:29:21.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.294+0000 7fbbbb7fe700 1 -- 172.21.15.67:0/3430840004 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 39) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fbbb40105d0 con 0x7fbbc40fff50 2024-04-22T23:29:21.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.294+0000 7fbbbb7fe700 1 --2- 172.21.15.67:0/3430840004 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbbb006ee00 0x7fbbb00712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:21.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.295+0000 7fbbbb7fe700 1 -- 172.21.15.67:0/3430840004 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fbbb408eb40 con 0x7fbbc40fff50 2024-04-22T23:29:21.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.295+0000 7fbbca166700 1 --2- 172.21.15.67:0/3430840004 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbbb006ee00 0x7fbbb00712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:21.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.295+0000 7fbbca166700 1 --2- 172.21.15.67:0/3430840004 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbbb006ee00 0x7fbbb00712c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbbc0006010 tx=0x7fbbc001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:29:21.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.299+0000 7fbbbb7fe700 1 -- 172.21.15.67:0/3430840004 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fbbb40577b0 con 0x7fbbc40fff50 2024-04-22T23:29:21.542 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:21 smithi067 ceph-mon[112438]: pgmap v47: 65 pgs: 34 peering, 31 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:29:21.542 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:21 smithi067 ceph-mon[112438]: from='client.34188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:29:21.543 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:21 smithi067 ceph-mon[112438]: from='client.34192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:29:21.543 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:21 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:21.543 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:21 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:21.543 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:21 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:29:21.543 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:21 smithi067 podman[123922]: 2024-04-22 23:29:21.090772191 +0000 UTC m=+0.380117925 container init a74fde1ef34edcffb85b2a275ec89aef7fcb531d7ae1abe3035519a261f301e6 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-22T23:29:21.543 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:21 smithi067 podman[123922]: 2024-04-22 23:29:21.09489847 +0000 UTC m=+0.384244198 container start a74fde1ef34edcffb85b2a275ec89aef7fcb531d7ae1abe3035519a261f301e6 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-22-1618-43be020) 2024-04-22T23:29:21.543 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:21 smithi067 ceph-osd[123963]: -- 172.21.15.67:0/2401164658 <== mon.0 v2:172.21.15.67:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559a6fb57680 con 0x559a6fb21000 2024-04-22T23:29:21.543 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:21 smithi067 bash[123922]: a74fde1ef34edcffb85b2a275ec89aef7fcb531d7ae1abe3035519a261f301e6 2024-04-22T23:29:21.543 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:21 smithi067 systemd[1]: Started Ceph osd.0 for d46c346e-00fd-11ef-bc93-c7b262605968. 2024-04-22T23:29:21.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.606+0000 7fbbcc3ca700 1 -- 172.21.15.67:0/3430840004 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fbbc404f2e0 con 0x7fbbc40fff50 2024-04-22T23:29:21.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.607+0000 7fbbbb7fe700 1 -- 172.21.15.67:0/3430840004 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+687 (secure 0 0 0) 0x7fbbb40172c0 con 0x7fbbc40fff50 2024-04-22T23:29:21.605 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:29:21.606 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:29:21.606 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:29:21.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:29:21.606 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:29:21.606 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:29:21.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:29:21.606 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:29:21.606 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 5 2024-04-22T23:29:21.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:29:21.606 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:29:21.606 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-22T23:29:21.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:29:21.606 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:29:21.607 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 9, 2024-04-22T23:29:21.607 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 4 2024-04-22T23:29:21.607 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:29:21.607 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:29:21.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.610+0000 7fbbcc3ca700 1 -- 172.21.15.67:0/3430840004 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbbb006ee00 msgr2=0x7fbbb00712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:21.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.610+0000 7fbbcc3ca700 1 --2- 172.21.15.67:0/3430840004 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbbb006ee00 0x7fbbb00712c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbbc0006010 tx=0x7fbbc001a040 comp rx=0 tx=0).stop 2024-04-22T23:29:21.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.610+0000 7fbbcc3ca700 1 -- 172.21.15.67:0/3430840004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbc40fff50 msgr2=0x7fbbc4100a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:21.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.610+0000 7fbbcc3ca700 1 --2- 172.21.15.67:0/3430840004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbc40fff50 0x7fbbc4100a60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fbbb400d900 tx=0x7fbbb400dc10 comp rx=0 tx=0).stop 2024-04-22T23:29:21.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.610+0000 7fbbcc3ca700 1 -- 172.21.15.67:0/3430840004 shutdown_connections 2024-04-22T23:29:21.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.610+0000 7fbbcc3ca700 1 --2- 172.21.15.67:0/3430840004 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbbb006ee00 0x7fbbb00712c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:21.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.610+0000 7fbbcc3ca700 1 --2- 172.21.15.67:0/3430840004 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbbc40ff5a0 0x7fbbc4100520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:21.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.610+0000 7fbbcc3ca700 1 --2- 172.21.15.67:0/3430840004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbc40fff50 0x7fbbc4100a60 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:21.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.610+0000 7fbbcc3ca700 1 -- 172.21.15.67:0/3430840004 >> 172.21.15.67:0/3430840004 conn(0x7fbbc4074e30 msgr2=0x7fbbc40fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:29:21.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.610+0000 7fbbcc3ca700 1 -- 172.21.15.67:0/3430840004 shutdown_connections 2024-04-22T23:29:21.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.610+0000 7fbbcc3ca700 1 -- 172.21.15.67:0/3430840004 wait complete. 2024-04-22T23:29:21.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.724+0000 7ffa26fea700 1 -- 172.21.15.67:0/1086384819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa20102700 msgr2=0x7ffa2010abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:21.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.724+0000 7ffa26fea700 1 --2- 172.21.15.67:0/1086384819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa20102700 0x7ffa2010abf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c009910 tx=0x7ffa0c009c20 comp rx=0 tx=0).stop 2024-04-22T23:29:21.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.724+0000 7ffa26fea700 1 -- 172.21.15.67:0/1086384819 shutdown_connections 2024-04-22T23:29:21.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.724+0000 7ffa26fea700 1 --2- 172.21.15.67:0/1086384819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa20102700 0x7ffa2010abf0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:21.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.724+0000 7ffa26fea700 1 --2- 172.21.15.67:0/1086384819 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa20101de0 0x7ffa201021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:21.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.724+0000 7ffa26fea700 1 -- 172.21.15.67:0/1086384819 >> 172.21.15.67:0/1086384819 conn(0x7ffa200fb610 msgr2=0x7ffa200fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:29:21.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.724+0000 7ffa26fea700 1 -- 172.21.15.67:0/1086384819 shutdown_connections 2024-04-22T23:29:21.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.724+0000 7ffa26fea700 1 -- 172.21.15.67:0/1086384819 wait complete. 2024-04-22T23:29:21.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.725+0000 7ffa26fea700 1 Processor -- start 2024-04-22T23:29:21.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.725+0000 7ffa26fea700 1 -- start start 2024-04-22T23:29:21.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.725+0000 7ffa26fea700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa20101de0 0x7ffa2019c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:21.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.725+0000 7ffa26fea700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa20102700 0x7ffa2019cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:21.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.725+0000 7ffa26fea700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2019d220 con 0x7ffa20101de0 2024-04-22T23:29:21.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.725+0000 7ffa26fea700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa20196720 con 0x7ffa20102700 2024-04-22T23:29:21.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.725+0000 7ffa24d86700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa20101de0 0x7ffa2019c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:21.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.725+0000 7ffa24d86700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa20101de0 0x7ffa2019c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35638/0 (socket says 172.21.15.67:35638) 2024-04-22T23:29:21.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.725+0000 7ffa24d86700 1 -- 172.21.15.67:0/1309564192 learned_addr learned my addr 172.21.15.67:0/1309564192 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:29:21.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.726+0000 7ffa1ffff700 1 --2- 172.21.15.67:0/1309564192 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa20102700 0x7ffa2019cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:21.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.726+0000 7ffa24d86700 1 -- 172.21.15.67:0/1309564192 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa20102700 msgr2=0x7ffa2019cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:21.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.726+0000 7ffa24d86700 1 --2- 172.21.15.67:0/1309564192 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa20102700 0x7ffa2019cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:21.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.726+0000 7ffa24d86700 1 -- 172.21.15.67:0/1309564192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa0c0095f0 con 0x7ffa20101de0 2024-04-22T23:29:21.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.726+0000 7ffa24d86700 1 --2- 172.21.15.67:0/1309564192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa20101de0 0x7ffa2019c650 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ffa1400d900 tx=0x7ffa1400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:29:21.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.726+0000 7ffa1dffb700 1 -- 172.21.15.67:0/1309564192 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa140041f0 con 0x7ffa20101de0 2024-04-22T23:29:21.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.726+0000 7ffa1dffb700 1 -- 172.21.15.67:0/1309564192 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffa14004cf0 con 0x7ffa20101de0 2024-04-22T23:29:21.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.726+0000 7ffa1dffb700 1 -- 172.21.15.67:0/1309564192 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa1400b750 con 0x7ffa20101de0 2024-04-22T23:29:21.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.726+0000 7ffa26fea700 1 -- 172.21.15.67:0/1309564192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa20196a00 con 0x7ffa20101de0 2024-04-22T23:29:21.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.727+0000 7ffa26fea700 1 -- 172.21.15.67:0/1309564192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa20196ed0 con 0x7ffa20101de0 2024-04-22T23:29:21.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.728+0000 7ffa26fea700 1 -- 172.21.15.67:0/1309564192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa201082f0 con 0x7ffa20101de0 2024-04-22T23:29:21.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.728+0000 7ffa1dffb700 1 -- 172.21.15.67:0/1309564192 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 39) v1 ==== 92847+0+0 (secure 0 0 0) 0x7ffa14004750 con 0x7ffa20101de0 2024-04-22T23:29:21.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.728+0000 7ffa1dffb700 1 --2- 172.21.15.67:0/1309564192 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ffa1006edf0 0x7ffa100712b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:21.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.729+0000 7ffa1dffb700 1 -- 172.21.15.67:0/1309564192 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6414+0+0 (secure 0 0 0) 0x7ffa1408ef80 con 0x7ffa20101de0 2024-04-22T23:29:21.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.729+0000 7ffa1ffff700 1 --2- 172.21.15.67:0/1309564192 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ffa1006edf0 0x7ffa100712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:21.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.729+0000 7ffa1ffff700 1 --2- 172.21.15.67:0/1309564192 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ffa1006edf0 0x7ffa100712b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c006010 tx=0x7ffa0c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:29:21.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:21.734+0000 7ffa1dffb700 1 -- 172.21.15.67:0/1309564192 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ffa1405b6a0 con 0x7ffa20101de0 2024-04-22T23:29:21.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:21 smithi134 ceph-mon[85630]: pgmap v47: 65 pgs: 34 peering, 31 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:29:21.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:21 smithi134 ceph-mon[85630]: from='client.34188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:29:21.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:21 smithi134 ceph-mon[85630]: from='client.34192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:29:21.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:21 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:21.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:21 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:21.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:21 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:29:22.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.004+0000 7ffa26fea700 1 -- 172.21.15.67:0/1309564192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ffa20197cf0 con 0x7ffa20101de0 2024-04-22T23:29:22.003 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:21 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0[123959]: 2024-04-22T23:29:21.635+0000 7fb56e0ec7c0 -1 Falling back to public interface 2024-04-22T23:29:22.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.005+0000 7ffa1dffb700 1 -- 172.21.15.67:0/1309564192 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1901 (secure 0 0 0) 0x7ffa1405b230 con 0x7ffa20101de0 2024-04-22T23:29:22.004 INFO:teuthology.orchestra.run.smithi067.stdout:e22 2024-04-22T23:29:22.004 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:29:22.004 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:29:22.004 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:29:22.004 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:29:22.004 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:29:22.004 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:29:22.004 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 22 2024-04-22T23:29:22.004 INFO:teuthology.orchestra.run.smithi067.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T23:29:22.005 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:29:22.005 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:27:20.577198+0000 2024-04-22T23:29:22.005 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:29:22.005 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:29:22.005 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:29:22.005 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:29:22.005 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:29:22.005 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:29:22.005 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:29:22.005 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:29:22.005 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 46 2024-04-22T23:29:22.005 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:29:22.005 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:29:22.006 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:29:22.006 INFO:teuthology.orchestra.run.smithi067.stdout:up {0=14484,1=24273} 2024-04-22T23:29:22.006 INFO:teuthology.orchestra.run.smithi067.stdout:failed 2024-04-22T23:29:22.006 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:29:22.006 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:29:22.006 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:29:22.006 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:29:22.006 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:29:22.006 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:29:22.006 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:29:22.006 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:29:22.006 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:29:22.007 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{0:14484} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.67:6828/3606151860,v1:172.21.15.67:6829/3606151860] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:29:22.007 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{0:14496} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/3305304988,v1:172.21.15.67:6827/3305304988] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:29:22.007 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{1:24273} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.134:6824/535957142,v1:172.21.15.134:6825/535957142] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:29:22.007 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{1:24289} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.134:6826/1954821676,v1:172.21.15.134:6827/1954821676] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:29:22.007 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:29:22.007 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:29:22.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.008+0000 7ffa26fea700 1 -- 172.21.15.67:0/1309564192 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ffa1006edf0 msgr2=0x7ffa100712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:22.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.008+0000 7ffa26fea700 1 --2- 172.21.15.67:0/1309564192 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ffa1006edf0 0x7ffa100712b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c006010 tx=0x7ffa0c01a040 comp rx=0 tx=0).stop 2024-04-22T23:29:22.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.008+0000 7ffa26fea700 1 -- 172.21.15.67:0/1309564192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa20101de0 msgr2=0x7ffa2019c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:22.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.008+0000 7ffa26fea700 1 --2- 172.21.15.67:0/1309564192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa20101de0 0x7ffa2019c650 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ffa1400d900 tx=0x7ffa1400dc10 comp rx=0 tx=0).stop 2024-04-22T23:29:22.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.008+0000 7ffa26fea700 1 -- 172.21.15.67:0/1309564192 shutdown_connections 2024-04-22T23:29:22.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.008+0000 7ffa26fea700 1 --2- 172.21.15.67:0/1309564192 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ffa1006edf0 0x7ffa100712b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:22.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.008+0000 7ffa26fea700 1 --2- 172.21.15.67:0/1309564192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa20101de0 0x7ffa2019c650 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:22.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.008+0000 7ffa26fea700 1 --2- 172.21.15.67:0/1309564192 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa20102700 0x7ffa2019cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:22.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.008+0000 7ffa26fea700 1 -- 172.21.15.67:0/1309564192 >> 172.21.15.67:0/1309564192 conn(0x7ffa200fb610 msgr2=0x7ffa20105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:29:22.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.008+0000 7ffa26fea700 1 -- 172.21.15.67:0/1309564192 shutdown_connections 2024-04-22T23:29:22.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.008+0000 7ffa26fea700 1 -- 172.21.15.67:0/1309564192 wait complete. 2024-04-22T23:29:22.009 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 22 2024-04-22T23:29:22.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.117+0000 7f2217bfb700 1 -- 172.21.15.67:0/1136223992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2210101de0 msgr2=0x7f22101021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:22.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.117+0000 7f2217bfb700 1 --2- 172.21.15.67:0/1136223992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2210101de0 0x7f22101021c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f220c009910 tx=0x7f220c009c20 comp rx=0 tx=0).stop 2024-04-22T23:29:22.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.117+0000 7f2217bfb700 1 -- 172.21.15.67:0/1136223992 shutdown_connections 2024-04-22T23:29:22.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.117+0000 7f2217bfb700 1 --2- 172.21.15.67:0/1136223992 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2210102700 0x7f221010abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:22.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.117+0000 7f2217bfb700 1 --2- 172.21.15.67:0/1136223992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2210101de0 0x7f22101021c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:22.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.117+0000 7f2217bfb700 1 -- 172.21.15.67:0/1136223992 >> 172.21.15.67:0/1136223992 conn(0x7f22100fb650 msgr2=0x7f22100fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:29:22.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.117+0000 7f2217bfb700 1 -- 172.21.15.67:0/1136223992 shutdown_connections 2024-04-22T23:29:22.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.117+0000 7f2217bfb700 1 -- 172.21.15.67:0/1136223992 wait complete. 2024-04-22T23:29:22.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.118+0000 7f2217bfb700 1 Processor -- start 2024-04-22T23:29:22.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.118+0000 7f2217bfb700 1 -- start start 2024-04-22T23:29:22.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.118+0000 7f2217bfb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2210101de0 0x7f22100ffa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:22.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.118+0000 7f2217bfb700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2210102700 0x7f22100fff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:22.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.118+0000 7f2217bfb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22101004c0 con 0x7f2210101de0 2024-04-22T23:29:22.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.118+0000 7f2217bfb700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2210100600 con 0x7f2210102700 2024-04-22T23:29:22.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.118+0000 7f2215997700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2210101de0 0x7f22100ffa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:22.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.118+0000 7f2215997700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2210101de0 0x7f22100ffa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35660/0 (socket says 172.21.15.67:35660) 2024-04-22T23:29:22.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.118+0000 7f2215997700 1 -- 172.21.15.67:0/535213730 learned_addr learned my addr 172.21.15.67:0/535213730 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:29:22.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.118+0000 7f2215196700 1 --2- 172.21.15.67:0/535213730 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2210102700 0x7f22100fff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:22.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.119+0000 7f2215997700 1 -- 172.21.15.67:0/535213730 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2210102700 msgr2=0x7f22100fff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:22.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.119+0000 7f2215997700 1 --2- 172.21.15.67:0/535213730 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2210102700 0x7f22100fff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:22.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.119+0000 7f2215997700 1 -- 172.21.15.67:0/535213730 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f220c0095f0 con 0x7f2210101de0 2024-04-22T23:29:22.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.119+0000 7f2215997700 1 --2- 172.21.15.67:0/535213730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2210101de0 0x7f22100ffa40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f220c006010 tx=0x7f220c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:29:22.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.119+0000 7f2206ffd700 1 -- 172.21.15.67:0/535213730 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f220c01d070 con 0x7f2210101de0 2024-04-22T23:29:22.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.119+0000 7f2206ffd700 1 -- 172.21.15.67:0/535213730 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f220c00f460 con 0x7f2210101de0 2024-04-22T23:29:22.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.119+0000 7f2206ffd700 1 -- 172.21.15.67:0/535213730 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f220c017630 con 0x7f2210101de0 2024-04-22T23:29:22.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.119+0000 7f2217bfb700 1 -- 172.21.15.67:0/535213730 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2210100880 con 0x7f2210101de0 2024-04-22T23:29:22.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.119+0000 7f2217bfb700 1 -- 172.21.15.67:0/535213730 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22101a2d80 con 0x7f2210101de0 2024-04-22T23:29:22.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.120+0000 7f2217bfb700 1 -- 172.21.15.67:0/535213730 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22101082f0 con 0x7f2210101de0 2024-04-22T23:29:22.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.120+0000 7f2206ffd700 1 -- 172.21.15.67:0/535213730 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 39) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f220c00f5d0 con 0x7f2210101de0 2024-04-22T23:29:22.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.121+0000 7f2206ffd700 1 --2- 172.21.15.67:0/535213730 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f21fc06ee00 0x7f21fc0712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:22.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.121+0000 7f2206ffd700 1 -- 172.21.15.67:0/535213730 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f220c0904f0 con 0x7f2210101de0 2024-04-22T23:29:22.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.121+0000 7f2215196700 1 --2- 172.21.15.67:0/535213730 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f21fc06ee00 0x7f21fc0712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:22.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.122+0000 7f2215196700 1 --2- 172.21.15.67:0/535213730 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f21fc06ee00 0x7f21fc0712c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2210101230 tx=0x7f220000a2d0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:29:22.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.126+0000 7f2206ffd700 1 -- 172.21.15.67:0/535213730 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f220c059160 con 0x7f2210101de0 2024-04-22T23:29:22.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.368+0000 7f2217bfb700 1 -- 172.21.15.67:0/535213730 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f22100611d0 con 0x7f21fc06ee00 2024-04-22T23:29:22.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.370+0000 7f2206ffd700 1 -- 172.21.15.67:0/535213730 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f22100611d0 con 0x7f21fc06ee00 2024-04-22T23:29:22.368 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:29:22.368 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:29:22.368 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:29:22.368 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:29:22.368 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:29:22.368 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:29:22.368 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:29:22.368 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr" 2024-04-22T23:29:22.369 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:29:22.369 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "6/23 daemons upgraded", 2024-04-22T23:29:22.369 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading osd daemons", 2024-04-22T23:29:22.369 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:29:22.369 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:29:22.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.372+0000 7f2217bfb700 1 -- 172.21.15.67:0/535213730 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f21fc06ee00 msgr2=0x7f21fc0712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:22.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.372+0000 7f2217bfb700 1 --2- 172.21.15.67:0/535213730 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f21fc06ee00 0x7f21fc0712c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2210101230 tx=0x7f220000a2d0 comp rx=0 tx=0).stop 2024-04-22T23:29:22.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.372+0000 7f2217bfb700 1 -- 172.21.15.67:0/535213730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2210101de0 msgr2=0x7f22100ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:22.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.372+0000 7f2217bfb700 1 --2- 172.21.15.67:0/535213730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2210101de0 0x7f22100ffa40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f220c006010 tx=0x7f220c00bd70 comp rx=0 tx=0).stop 2024-04-22T23:29:22.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.372+0000 7f2217bfb700 1 -- 172.21.15.67:0/535213730 shutdown_connections 2024-04-22T23:29:22.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.372+0000 7f2217bfb700 1 --2- 172.21.15.67:0/535213730 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f21fc06ee00 0x7f21fc0712c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:22.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.372+0000 7f2217bfb700 1 --2- 172.21.15.67:0/535213730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2210101de0 0x7f22100ffa40 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:22.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.372+0000 7f2217bfb700 1 --2- 172.21.15.67:0/535213730 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2210102700 0x7f22100fff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:22.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.372+0000 7f2217bfb700 1 -- 172.21.15.67:0/535213730 >> 172.21.15.67:0/535213730 conn(0x7f22100fb650 msgr2=0x7f2210105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:29:22.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.372+0000 7f2217bfb700 1 -- 172.21.15.67:0/535213730 shutdown_connections 2024-04-22T23:29:22.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.372+0000 7f2217bfb700 1 -- 172.21.15.67:0/535213730 wait complete. 2024-04-22T23:29:22.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.482+0000 7f9916173700 1 -- 172.21.15.67:0/2527531728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9910100ea0 msgr2=0x7f9910101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:22.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.482+0000 7f9916173700 1 --2- 172.21.15.67:0/2527531728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9910100ea0 0x7f9910101280 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9900009910 tx=0x7f9900009c20 comp rx=0 tx=0).stop 2024-04-22T23:29:22.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.482+0000 7f9916173700 1 -- 172.21.15.67:0/2527531728 shutdown_connections 2024-04-22T23:29:22.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.482+0000 7f9916173700 1 --2- 172.21.15.67:0/2527531728 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9910101850 0x7f99101058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:22.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.482+0000 7f9916173700 1 --2- 172.21.15.67:0/2527531728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9910100ea0 0x7f9910101280 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:22.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.482+0000 7f9916173700 1 -- 172.21.15.67:0/2527531728 >> 172.21.15.67:0/2527531728 conn(0x7f99100fc750 msgr2=0x7f99100feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:29:22.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.482+0000 7f9916173700 1 -- 172.21.15.67:0/2527531728 shutdown_connections 2024-04-22T23:29:22.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.482+0000 7f9916173700 1 -- 172.21.15.67:0/2527531728 wait complete. 2024-04-22T23:29:22.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.482+0000 7f9916173700 1 Processor -- start 2024-04-22T23:29:22.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.483+0000 7f9916173700 1 -- start start 2024-04-22T23:29:22.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.483+0000 7f9916173700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9910100ea0 0x7f991019c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:22.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.483+0000 7f9916173700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9910101850 0x7f991019ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:22.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.483+0000 7f9916173700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991019d350 con 0x7f9910101850 2024-04-22T23:29:22.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.483+0000 7f9916173700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9910196800 con 0x7f9910100ea0 2024-04-22T23:29:22.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.483+0000 7f990effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9910101850 0x7f991019ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:22.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.483+0000 7f990effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9910101850 0x7f991019ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35674/0 (socket says 172.21.15.67:35674) 2024-04-22T23:29:22.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.483+0000 7f990effd700 1 -- 172.21.15.67:0/391697675 learned_addr learned my addr 172.21.15.67:0/391697675 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:29:22.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.483+0000 7f990f7fe700 1 --2- 172.21.15.67:0/391697675 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9910100ea0 0x7f991019c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:22.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.483+0000 7f990effd700 1 -- 172.21.15.67:0/391697675 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9910100ea0 msgr2=0x7f991019c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:22.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.483+0000 7f990effd700 1 --2- 172.21.15.67:0/391697675 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9910100ea0 0x7f991019c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:22.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.483+0000 7f990effd700 1 -- 172.21.15.67:0/391697675 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99000095f0 con 0x7f9910101850 2024-04-22T23:29:22.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.484+0000 7f990effd700 1 --2- 172.21.15.67:0/391697675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9910101850 0x7f991019ccc0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f98f800b480 tx=0x7f98f800b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:29:22.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.484+0000 7f990cff9700 1 -- 172.21.15.67:0/391697675 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98f8011800 con 0x7f9910101850 2024-04-22T23:29:22.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.484+0000 7f990cff9700 1 -- 172.21.15.67:0/391697675 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f98f8011e20 con 0x7f9910101850 2024-04-22T23:29:22.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.484+0000 7f9916173700 1 -- 172.21.15.67:0/391697675 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9910196ae0 con 0x7f9910101850 2024-04-22T23:29:22.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.484+0000 7f990cff9700 1 -- 172.21.15.67:0/391697675 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98f8010ea0 con 0x7f9910101850 2024-04-22T23:29:22.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.484+0000 7f9916173700 1 -- 172.21.15.67:0/391697675 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9910197030 con 0x7f9910101850 2024-04-22T23:29:22.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.485+0000 7f9916173700 1 -- 172.21.15.67:0/391697675 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9910109240 con 0x7f9910101850 2024-04-22T23:29:22.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.486+0000 7f990cff9700 1 -- 172.21.15.67:0/391697675 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 39) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f98f8010380 con 0x7f9910101850 2024-04-22T23:29:22.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.486+0000 7f990cff9700 1 --2- 172.21.15.67:0/391697675 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f98fc06edf0 0x7f98fc0712b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:22.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.486+0000 7f990cff9700 1 -- 172.21.15.67:0/391697675 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f98f808e9a0 con 0x7f9910101850 2024-04-22T23:29:22.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.486+0000 7f990f7fe700 1 --2- 172.21.15.67:0/391697675 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f98fc06edf0 0x7f98fc0712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:22.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.487+0000 7f990f7fe700 1 --2- 172.21.15.67:0/391697675 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f98fc06edf0 0x7f98fc0712b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9900005f50 tx=0x7f990001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:29:22.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.491+0000 7f990cff9700 1 -- 172.21.15.67:0/391697675 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f98f805b250 con 0x7f9910101850 2024-04-22T23:29:22.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:22 smithi134 ceph-mon[85630]: from='client.34196 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:29:22.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:22 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3430840004' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:29:22.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:22 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1309564192' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:29:22.792 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:22 smithi067 ceph-mon[112438]: from='client.34196 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:29:22.792 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:22 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3430840004' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:29:22.792 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:22 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1309564192' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:29:22.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.808+0000 7f9916173700 1 -- 172.21.15.67:0/391697675 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f99101972b0 con 0x7f9910101850 2024-04-22T23:29:22.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.808+0000 7f990cff9700 1 -- 172.21.15.67:0/391697675 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1518 (secure 0 0 0) 0x7f98f8015600 con 0x7f9910101850 2024-04-22T23:29:22.807 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data; 1 osds down; Degraded data redundancy: 71/363 objects degraded (19.559%), 19 pgs degraded 2024-04-22T23:29:22.807 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:29:22.807 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:29:22.807 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-22T23:29:22.807 INFO:teuthology.orchestra.run.smithi067.stdout: osd.0 (root=default,host=smithi067) is down 2024-04-22T23:29:22.807 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 71/363 objects degraded (19.559%), 19 pgs degraded 2024-04-22T23:29:22.807 INFO:teuthology.orchestra.run.smithi067.stdout: pg 1.0 is active+undersized+degraded, acting [3,1] 2024-04-22T23:29:22.807 INFO:teuthology.orchestra.run.smithi067.stdout: pg 2.0 is active+undersized+degraded, acting [3,1] 2024-04-22T23:29:22.807 INFO:teuthology.orchestra.run.smithi067.stdout: pg 2.1 is active+undersized+degraded, acting [2,1] 2024-04-22T23:29:22.807 INFO:teuthology.orchestra.run.smithi067.stdout: pg 2.2 is active+undersized+degraded, acting [5,1] 2024-04-22T23:29:22.808 INFO:teuthology.orchestra.run.smithi067.stdout: pg 2.4 is active+undersized+degraded, acting [1,4] 2024-04-22T23:29:22.808 INFO:teuthology.orchestra.run.smithi067.stdout: pg 2.5 is active+undersized+degraded, acting [3,4] 2024-04-22T23:29:22.808 INFO:teuthology.orchestra.run.smithi067.stdout: pg 2.8 is active+undersized+degraded, acting [3,5] 2024-04-22T23:29:22.808 INFO:teuthology.orchestra.run.smithi067.stdout: pg 2.9 is active+undersized+degraded, acting [1,4] 2024-04-22T23:29:22.808 INFO:teuthology.orchestra.run.smithi067.stdout: pg 2.e is active+undersized+degraded, acting [2,3] 2024-04-22T23:29:22.808 INFO:teuthology.orchestra.run.smithi067.stdout: pg 2.f is active+undersized+degraded, acting [4,5] 2024-04-22T23:29:22.808 INFO:teuthology.orchestra.run.smithi067.stdout: pg 2.10 is active+undersized+degraded, acting [2,1] 2024-04-22T23:29:22.808 INFO:teuthology.orchestra.run.smithi067.stdout: pg 2.12 is active+undersized+degraded, acting [3,1] 2024-04-22T23:29:22.808 INFO:teuthology.orchestra.run.smithi067.stdout: pg 2.13 is active+undersized+degraded, acting [4,2] 2024-04-22T23:29:22.808 INFO:teuthology.orchestra.run.smithi067.stdout: pg 2.15 is active+undersized+degraded, acting [1,3] 2024-04-22T23:29:22.808 INFO:teuthology.orchestra.run.smithi067.stdout: pg 2.19 is active+undersized+degraded, acting [4,2] 2024-04-22T23:29:22.808 INFO:teuthology.orchestra.run.smithi067.stdout: pg 2.1b is active+undersized+degraded, acting [1,5] 2024-04-22T23:29:22.808 INFO:teuthology.orchestra.run.smithi067.stdout: pg 2.1d is active+undersized+degraded, acting [3,5] 2024-04-22T23:29:22.808 INFO:teuthology.orchestra.run.smithi067.stdout: pg 2.1e is active+undersized+degraded, acting [2,5] 2024-04-22T23:29:22.809 INFO:teuthology.orchestra.run.smithi067.stdout: pg 2.1f is active+undersized+degraded, acting [3,4] 2024-04-22T23:29:22.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.810+0000 7f9916173700 1 -- 172.21.15.67:0/391697675 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f98fc06edf0 msgr2=0x7f98fc0712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:22.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.810+0000 7f9916173700 1 --2- 172.21.15.67:0/391697675 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f98fc06edf0 0x7f98fc0712b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9900005f50 tx=0x7f990001a040 comp rx=0 tx=0).stop 2024-04-22T23:29:22.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.810+0000 7f9916173700 1 -- 172.21.15.67:0/391697675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9910101850 msgr2=0x7f991019ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:22.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.810+0000 7f9916173700 1 --2- 172.21.15.67:0/391697675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9910101850 0x7f991019ccc0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f98f800b480 tx=0x7f98f800b790 comp rx=0 tx=0).stop 2024-04-22T23:29:22.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.811+0000 7f9916173700 1 -- 172.21.15.67:0/391697675 shutdown_connections 2024-04-22T23:29:22.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.811+0000 7f9916173700 1 --2- 172.21.15.67:0/391697675 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f98fc06edf0 0x7f98fc0712b0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:22.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.811+0000 7f9916173700 1 --2- 172.21.15.67:0/391697675 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9910100ea0 0x7f991019c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:22.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.811+0000 7f9916173700 1 --2- 172.21.15.67:0/391697675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9910101850 0x7f991019ccc0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:22.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.811+0000 7f9916173700 1 -- 172.21.15.67:0/391697675 >> 172.21.15.67:0/391697675 conn(0x7f99100fc750 msgr2=0x7f99100fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:29:22.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.811+0000 7f9916173700 1 -- 172.21.15.67:0/391697675 shutdown_connections 2024-04-22T23:29:22.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:22.811+0000 7f9916173700 1 -- 172.21.15.67:0/391697675 wait complete. 2024-04-22T23:29:23.503 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:23 smithi067 ceph-mon[112438]: pgmap v48: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 71/363 objects degraded (19.559%) 2024-04-22T23:29:23.503 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:23 smithi067 ceph-mon[112438]: Health check failed: Degraded data redundancy: 71/363 objects degraded (19.559%), 19 pgs degraded (PG_DEGRADED) 2024-04-22T23:29:23.503 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:23 smithi067 ceph-mon[112438]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 24 pgs peering) 2024-04-22T23:29:23.503 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:23 smithi067 ceph-mon[112438]: from='client.34210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:29:23.504 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:23 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/391697675' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:29:23.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:23 smithi134 ceph-mon[85630]: pgmap v48: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 71/363 objects degraded (19.559%) 2024-04-22T23:29:23.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:23 smithi134 ceph-mon[85630]: Health check failed: Degraded data redundancy: 71/363 objects degraded (19.559%), 19 pgs degraded (PG_DEGRADED) 2024-04-22T23:29:23.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:23 smithi134 ceph-mon[85630]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 24 pgs peering) 2024-04-22T23:29:23.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:23 smithi134 ceph-mon[85630]: from='client.34210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:29:23.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:23 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/391697675' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:29:25.542 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:25 smithi067 ceph-mon[112438]: pgmap v49: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 71/363 objects degraded (19.559%) 2024-04-22T23:29:25.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:25 smithi134 ceph-mon[85630]: pgmap v49: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 71/363 objects degraded (19.559%) 2024-04-22T23:29:26.377 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:26 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0[123959]: 2024-04-22T23:29:26.068+0000 7fb56e0ec7c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-04-22T23:29:26.792 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:26 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0[123959]: 2024-04-22T23:29:26.431+0000 7fb56e0ec7c0 -1 osd.0 49 log_to_monitors true 2024-04-22T23:29:27.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:27 smithi067 ceph-mon[112438]: pgmap v50: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 71/363 objects degraded (19.559%) 2024-04-22T23:29:27.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:27 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:27.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:27 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:29:27.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:27 smithi067 ceph-mon[112438]: from='osd.0 [v2:172.21.15.67:6802/1540469345,v1:172.21.15.67:6803/1540469345]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T23:29:27.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:27 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:27.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:27 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:27.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:27 smithi134 ceph-mon[85630]: pgmap v50: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 310 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 71/363 objects degraded (19.559%) 2024-04-22T23:29:27.492 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:27 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:27.492 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:27 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:29:27.492 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:27 smithi134 ceph-mon[85630]: from='osd.0 [v2:172.21.15.67:6802/1540469345,v1:172.21.15.67:6803/1540469345]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T23:29:27.492 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:27 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:27.492 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:27 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:27.792 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 22 23:29:27 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0[123959]: 2024-04-22T23:29:27.335+0000 7fb55f544700 -1 osd.0 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T23:29:28.542 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:28 smithi067 ceph-mon[112438]: from='osd.0 [v2:172.21.15.67:6802/1540469345,v1:172.21.15.67:6803/1540469345]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T23:29:28.542 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:28 smithi067 ceph-mon[112438]: osdmap e52: 6 total, 5 up, 6 in 2024-04-22T23:29:28.542 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:28 smithi067 ceph-mon[112438]: from='osd.0 [v2:172.21.15.67:6802/1540469345,v1:172.21.15.67:6803/1540469345]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-04-22T23:29:28.542 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:28 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:28.542 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:28 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:28.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:28 smithi134 ceph-mon[85630]: from='osd.0 [v2:172.21.15.67:6802/1540469345,v1:172.21.15.67:6803/1540469345]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T23:29:28.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:28 smithi134 ceph-mon[85630]: osdmap e52: 6 total, 5 up, 6 in 2024-04-22T23:29:28.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:28 smithi134 ceph-mon[85630]: from='osd.0 [v2:172.21.15.67:6802/1540469345,v1:172.21.15.67:6803/1540469345]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-04-22T23:29:28.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:28 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:28.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:28 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:29.658 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:29 smithi067 ceph-mon[112438]: pgmap v52: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 511 B/s wr, 4 op/s; 71/363 objects degraded (19.559%) 2024-04-22T23:29:29.658 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:29 smithi067 ceph-mon[112438]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-22T23:29:29.659 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:29 smithi067 ceph-mon[112438]: osd.0 [v2:172.21.15.67:6802/1540469345,v1:172.21.15.67:6803/1540469345] boot 2024-04-22T23:29:29.659 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:29 smithi067 ceph-mon[112438]: osdmap e53: 6 total, 6 up, 6 in 2024-04-22T23:29:29.659 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:29 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T23:29:29.659 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:29 smithi067 ceph-mon[112438]: mgrmap e40: smithi067.whdcsr(active, since 93s), standbys: smithi134.ccxkcj 2024-04-22T23:29:29.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:29 smithi134 ceph-mon[85630]: pgmap v52: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 511 B/s wr, 4 op/s; 71/363 objects degraded (19.559%) 2024-04-22T23:29:29.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:29 smithi134 ceph-mon[85630]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-22T23:29:29.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:29 smithi134 ceph-mon[85630]: osd.0 [v2:172.21.15.67:6802/1540469345,v1:172.21.15.67:6803/1540469345] boot 2024-04-22T23:29:29.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:29 smithi134 ceph-mon[85630]: osdmap e53: 6 total, 6 up, 6 in 2024-04-22T23:29:29.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:29 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T23:29:29.742 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:29 smithi134 ceph-mon[85630]: mgrmap e40: smithi067.whdcsr(active, since 93s), standbys: smithi134.ccxkcj 2024-04-22T23:29:30.542 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:30 smithi067 ceph-mon[112438]: osdmap e54: 6 total, 6 up, 6 in 2024-04-22T23:29:30.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:30 smithi134 ceph-mon[85630]: osdmap e54: 6 total, 6 up, 6 in 2024-04-22T23:29:31.542 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:31 smithi067 ceph-mon[112438]: pgmap v55: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 853 B/s wr, 4 op/s; 71/363 objects degraded (19.559%) 2024-04-22T23:29:31.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:31 smithi134 ceph-mon[85630]: pgmap v55: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 853 B/s wr, 4 op/s; 71/363 objects degraded (19.559%) 2024-04-22T23:29:32.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:32 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:32.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:32 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:32.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:32 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:29:32.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:32 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:29:32.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:32 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:32.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:32 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:29:32.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:32 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:29:32.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:32 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:29:32.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:32 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:29:32.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:32 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-22T23:29:33.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:32 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:33.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:32 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:33.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:32 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:29:33.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:32 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:29:33.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:32 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:33.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:32 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:29:33.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:32 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:29:33.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:32 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:29:33.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:32 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:29:33.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:32 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-22T23:29:33.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:33 smithi134 ceph-mon[85630]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-22T23:29:34.031 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:33 smithi134 ceph-mon[85630]: Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-04-22T23:29:34.031 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:33 smithi134 ceph-mon[85630]: pgmap v56: 65 pgs: 3 active+undersized+degraded, 6 active+undersized, 56 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 3.3 KiB/s rd, 853 B/s wr, 6 op/s; 14/363 objects degraded (3.857%); 46 B/s, 0 objects/s recovering 2024-04-22T23:29:34.031 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:33 smithi134 ceph-mon[85630]: Health check update: Degraded data redundancy: 14/363 objects degraded (3.857%), 3 pgs degraded (PG_DEGRADED) 2024-04-22T23:29:34.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:33 smithi067 ceph-mon[112438]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-22T23:29:34.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:33 smithi067 ceph-mon[112438]: Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-04-22T23:29:34.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:33 smithi067 ceph-mon[112438]: pgmap v56: 65 pgs: 3 active+undersized+degraded, 6 active+undersized, 56 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 3.3 KiB/s rd, 853 B/s wr, 6 op/s; 14/363 objects degraded (3.857%); 46 B/s, 0 objects/s recovering 2024-04-22T23:29:34.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:33 smithi067 ceph-mon[112438]: Health check update: Degraded data redundancy: 14/363 objects degraded (3.857%), 3 pgs degraded (PG_DEGRADED) 2024-04-22T23:29:35.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:34 smithi067 ceph-mon[112438]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 14/363 objects degraded (3.857%), 3 pgs degraded) 2024-04-22T23:29:35.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:34 smithi134 ceph-mon[85630]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 14/363 objects degraded (3.857%), 3 pgs degraded) 2024-04-22T23:29:36.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:35 smithi067 ceph-mon[112438]: pgmap v57: 65 pgs: 65 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s; 63 B/s, 0 objects/s recovering 2024-04-22T23:29:36.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:35 smithi134 ceph-mon[85630]: pgmap v57: 65 pgs: 65 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s; 63 B/s, 0 objects/s recovering 2024-04-22T23:29:38.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:37 smithi067 ceph-mon[112438]: pgmap v58: 65 pgs: 65 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 895 B/s rd, 1 op/s; 52 B/s, 0 objects/s recovering 2024-04-22T23:29:38.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:37 smithi134 ceph-mon[85630]: pgmap v58: 65 pgs: 65 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 895 B/s rd, 1 op/s; 52 B/s, 0 objects/s recovering 2024-04-22T23:29:40.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:39 smithi067 ceph-mon[112438]: pgmap v59: 65 pgs: 65 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s; 43 B/s, 0 objects/s recovering 2024-04-22T23:29:40.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:39 smithi134 ceph-mon[85630]: pgmap v59: 65 pgs: 65 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s; 43 B/s, 0 objects/s recovering 2024-04-22T23:29:42.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:41 smithi067 ceph-mon[112438]: pgmap v60: 65 pgs: 65 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 2 op/s; 39 B/s, 0 objects/s recovering 2024-04-22T23:29:42.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:42.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:29:42.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:42.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:41 smithi134 ceph-mon[85630]: pgmap v60: 65 pgs: 65 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 2 op/s; 39 B/s, 0 objects/s recovering 2024-04-22T23:29:42.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:42.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:29:42.043 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:44.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:43 smithi067 ceph-mon[112438]: pgmap v61: 65 pgs: 65 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 3 op/s; 35 B/s, 0 objects/s recovering 2024-04-22T23:29:44.227 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:43 smithi134 ceph-mon[85630]: pgmap v61: 65 pgs: 65 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 3 op/s; 35 B/s, 0 objects/s recovering 2024-04-22T23:29:46.041 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:45 smithi067 ceph-mon[112438]: pgmap v62: 65 pgs: 65 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s; 11 B/s, 0 objects/s recovering 2024-04-22T23:29:46.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:45 smithi134 ceph-mon[85630]: pgmap v62: 65 pgs: 65 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s; 11 B/s, 0 objects/s recovering 2024-04-22T23:29:48.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:47 smithi067 ceph-mon[112438]: pgmap v63: 65 pgs: 65 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:29:48.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:47 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-22T23:29:48.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:47 smithi134 ceph-mon[85630]: pgmap v63: 65 pgs: 65 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:29:48.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:47 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-22T23:29:49.041 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:48 smithi067 ceph-mon[112438]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-22T23:29:49.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:48 smithi067 ceph-mon[112438]: Upgrade: osd.1 is safe to restart 2024-04-22T23:29:49.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:48 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:49.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:48 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T23:29:49.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:48 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:29:49.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:48 smithi134 ceph-mon[85630]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-22T23:29:49.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:48 smithi134 ceph-mon[85630]: Upgrade: osd.1 is safe to restart 2024-04-22T23:29:49.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:48 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:49.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:48 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T23:29:49.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:48 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:29:50.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:49 smithi067 ceph-mon[112438]: pgmap v64: 65 pgs: 65 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:29:50.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:49 smithi067 ceph-mon[112438]: Upgrade: Updating osd.1 2024-04-22T23:29:50.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:49 smithi067 ceph-mon[112438]: Deploying daemon osd.1 on smithi067 2024-04-22T23:29:50.098 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:49 smithi134 ceph-mon[85630]: pgmap v64: 65 pgs: 65 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:29:50.099 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:49 smithi134 ceph-mon[85630]: Upgrade: Updating osd.1 2024-04-22T23:29:50.099 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:49 smithi134 ceph-mon[85630]: Deploying daemon osd.1 on smithi067 2024-04-22T23:29:52.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:51 smithi067 ceph-mon[112438]: pgmap v65: 65 pgs: 65 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:29:52.042 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:51 smithi067 systemd[1]: Stopping Ceph osd.1 for d46c346e-00fd-11ef-bc93-c7b262605968... 2024-04-22T23:29:52.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:51 smithi134 ceph-mon[85630]: pgmap v65: 65 pgs: 65 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:29:52.542 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:52 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1[53145]: 2024-04-22T23:29:52.149+0000 7f2c6bca5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T23:29:52.542 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:52 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1[53145]: 2024-04-22T23:29:52.149+0000 7f2c6bca5700 -1 osd.1 54 *** Got signal Terminated *** 2024-04-22T23:29:52.542 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:52 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1[53145]: 2024-04-22T23:29:52.149+0000 7f2c6bca5700 -1 osd.1 54 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T23:29:52.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:52.930+0000 7ff2bb67d700 1 -- 172.21.15.67:0/281265140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2b40ffbf0 msgr2=0x7ff2b40fffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:52.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:52.930+0000 7ff2bb67d700 1 --2- 172.21.15.67:0/281265140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2b40ffbf0 0x7ff2b40fffd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff2b0009910 tx=0x7ff2b0009c20 comp rx=0 tx=0).stop 2024-04-22T23:29:52.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:52.930+0000 7ff2bb67d700 1 -- 172.21.15.67:0/281265140 shutdown_connections 2024-04-22T23:29:52.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:52.930+0000 7ff2bb67d700 1 --2- 172.21.15.67:0/281265140 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff2b4100510 0x7ff2b41049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:52.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:52.930+0000 7ff2bb67d700 1 --2- 172.21.15.67:0/281265140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2b40ffbf0 0x7ff2b40fffd0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:52.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:52.930+0000 7ff2bb67d700 1 -- 172.21.15.67:0/281265140 >> 172.21.15.67:0/281265140 conn(0x7ff2b40fb650 msgr2=0x7ff2b40fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:29:52.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:52.930+0000 7ff2bb67d700 1 -- 172.21.15.67:0/281265140 shutdown_connections 2024-04-22T23:29:52.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:52.930+0000 7ff2bb67d700 1 -- 172.21.15.67:0/281265140 wait complete. 2024-04-22T23:29:52.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:52.931+0000 7ff2bb67d700 1 Processor -- start 2024-04-22T23:29:52.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:52.931+0000 7ff2bb67d700 1 -- start start 2024-04-22T23:29:52.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:52.931+0000 7ff2bb67d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2b40ffbf0 0x7ff2b419cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:52.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:52.931+0000 7ff2bb67d700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff2b4100510 0x7ff2b419d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:52.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:52.931+0000 7ff2bb67d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b419db80 con 0x7ff2b40ffbf0 2024-04-22T23:29:52.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:52.931+0000 7ff2bb67d700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b41a1910 con 0x7ff2b4100510 2024-04-22T23:29:52.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:52.931+0000 7ff2b9419700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2b40ffbf0 0x7ff2b419cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:52.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:52.931+0000 7ff2b9419700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2b40ffbf0 0x7ff2b419cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58162/0 (socket says 172.21.15.67:58162) 2024-04-22T23:29:52.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:52.931+0000 7ff2b9419700 1 -- 172.21.15.67:0/2621477291 learned_addr learned my addr 172.21.15.67:0/2621477291 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:29:52.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:52.931+0000 7ff2b8c18700 1 --2- 172.21.15.67:0/2621477291 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff2b4100510 0x7ff2b419d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:52.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:52.932+0000 7ff2b9419700 1 -- 172.21.15.67:0/2621477291 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff2b4100510 msgr2=0x7ff2b419d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:52.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:52.932+0000 7ff2b9419700 1 --2- 172.21.15.67:0/2621477291 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff2b4100510 0x7ff2b419d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:52.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:52.932+0000 7ff2b9419700 1 -- 172.21.15.67:0/2621477291 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2b00095f0 con 0x7ff2b40ffbf0 2024-04-22T23:29:52.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:52.932+0000 7ff2b9419700 1 --2- 172.21.15.67:0/2621477291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2b40ffbf0 0x7ff2b419cf60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff2b0006010 tx=0x7ff2b000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:29:52.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:52.932+0000 7ff2aa7fc700 1 -- 172.21.15.67:0/2621477291 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2b001d070 con 0x7ff2b40ffbf0 2024-04-22T23:29:52.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:52.932+0000 7ff2aa7fc700 1 -- 172.21.15.67:0/2621477291 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff2b0005dc0 con 0x7ff2b40ffbf0 2024-04-22T23:29:52.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:52.932+0000 7ff2bb67d700 1 -- 172.21.15.67:0/2621477291 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2b41a1b90 con 0x7ff2b40ffbf0 2024-04-22T23:29:52.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:52.932+0000 7ff2aa7fc700 1 -- 172.21.15.67:0/2621477291 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2b0017400 con 0x7ff2b40ffbf0 2024-04-22T23:29:52.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:52.932+0000 7ff2bb67d700 1 -- 172.21.15.67:0/2621477291 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2b41a2080 con 0x7ff2b40ffbf0 2024-04-22T23:29:52.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:52.933+0000 7ff2bb67d700 1 -- 172.21.15.67:0/2621477291 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2b4102090 con 0x7ff2b40ffbf0 2024-04-22T23:29:52.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:52.934+0000 7ff2aa7fc700 1 -- 172.21.15.67:0/2621477291 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff2b0003a60 con 0x7ff2b40ffbf0 2024-04-22T23:29:52.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:52.934+0000 7ff2aa7fc700 1 --2- 172.21.15.67:0/2621477291 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff2a006ecc0 0x7ff2a0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:52.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:52.934+0000 7ff2b8c18700 1 --2- 172.21.15.67:0/2621477291 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff2a006ecc0 0x7ff2a0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:52.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:52.934+0000 7ff2aa7fc700 1 -- 172.21.15.67:0/2621477291 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6414+0+0 (secure 0 0 0) 0x7ff2b00901a0 con 0x7ff2b40ffbf0 2024-04-22T23:29:52.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:52.935+0000 7ff2b8c18700 1 --2- 172.21.15.67:0/2621477291 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff2a006ecc0 0x7ff2a0071180 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff2b419e580 tx=0x7ff2a400b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:29:52.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:52.939+0000 7ff2aa7fc700 1 -- 172.21.15.67:0/2621477291 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff2b0059030 con 0x7ff2b40ffbf0 2024-04-22T23:29:53.041 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:52 smithi067 ceph-mon[112438]: osd.1 marked itself down and dead 2024-04-22T23:29:53.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.188+0000 7ff2bb67d700 1 -- 172.21.15.67:0/2621477291 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff2b419e380 con 0x7ff2a006ecc0 2024-04-22T23:29:53.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.190+0000 7ff2aa7fc700 1 -- 172.21.15.67:0/2621477291 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7ff2b419e380 con 0x7ff2a006ecc0 2024-04-22T23:29:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.192+0000 7ff2bb67d700 1 -- 172.21.15.67:0/2621477291 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff2a006ecc0 msgr2=0x7ff2a0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.192+0000 7ff2bb67d700 1 --2- 172.21.15.67:0/2621477291 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff2a006ecc0 0x7ff2a0071180 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff2b419e580 tx=0x7ff2a400b410 comp rx=0 tx=0).stop 2024-04-22T23:29:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.192+0000 7ff2bb67d700 1 -- 172.21.15.67:0/2621477291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2b40ffbf0 msgr2=0x7ff2b419cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.192+0000 7ff2bb67d700 1 --2- 172.21.15.67:0/2621477291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2b40ffbf0 0x7ff2b419cf60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff2b0006010 tx=0x7ff2b000bd70 comp rx=0 tx=0).stop 2024-04-22T23:29:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.192+0000 7ff2bb67d700 1 -- 172.21.15.67:0/2621477291 shutdown_connections 2024-04-22T23:29:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.192+0000 7ff2bb67d700 1 --2- 172.21.15.67:0/2621477291 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff2a006ecc0 0x7ff2a0071180 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.192+0000 7ff2bb67d700 1 --2- 172.21.15.67:0/2621477291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2b40ffbf0 0x7ff2b419cf60 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.192+0000 7ff2bb67d700 1 --2- 172.21.15.67:0/2621477291 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff2b4100510 0x7ff2b419d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.192+0000 7ff2bb67d700 1 -- 172.21.15.67:0/2621477291 >> 172.21.15.67:0/2621477291 conn(0x7ff2b40fb650 msgr2=0x7ff2b4111060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:29:53.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.192+0000 7ff2bb67d700 1 -- 172.21.15.67:0/2621477291 shutdown_connections 2024-04-22T23:29:53.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.192+0000 7ff2bb67d700 1 -- 172.21.15.67:0/2621477291 wait complete. 2024-04-22T23:29:53.200 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:29:53.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:52 smithi134 ceph-mon[85630]: osd.1 marked itself down and dead 2024-04-22T23:29:53.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.302+0000 7fccfd338700 1 -- 172.21.15.67:0/2661264404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccf80ff340 msgr2=0x7fccf80ff7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:53.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.302+0000 7fccfd338700 1 --2- 172.21.15.67:0/2661264404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccf80ff340 0x7fccf80ff7c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fccec009910 tx=0x7fccec009c20 comp rx=0 tx=0).stop 2024-04-22T23:29:53.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.302+0000 7fccfd338700 1 -- 172.21.15.67:0/2661264404 shutdown_connections 2024-04-22T23:29:53.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.302+0000 7fccfd338700 1 --2- 172.21.15.67:0/2661264404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccf80ff340 0x7fccf80ff7c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:53.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.302+0000 7fccfd338700 1 --2- 172.21.15.67:0/2661264404 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fccf8101bb0 0x7fccf8101f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:53.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.302+0000 7fccfd338700 1 -- 172.21.15.67:0/2661264404 >> 172.21.15.67:0/2661264404 conn(0x7fccf8074e30 msgr2=0x7fccf8075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:29:53.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.303+0000 7fccfd338700 1 -- 172.21.15.67:0/2661264404 shutdown_connections 2024-04-22T23:29:53.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.303+0000 7fccfd338700 1 -- 172.21.15.67:0/2661264404 wait complete. 2024-04-22T23:29:53.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.303+0000 7fccfd338700 1 Processor -- start 2024-04-22T23:29:53.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.303+0000 7fccfd338700 1 -- start start 2024-04-22T23:29:53.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.304+0000 7fccfd338700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccf80ff340 0x7fccf819e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:53.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.304+0000 7fccfd338700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fccf8101bb0 0x7fccf819ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:53.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.304+0000 7fccfd338700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf819f3f0 con 0x7fccf80ff340 2024-04-22T23:29:53.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.304+0000 7fccfd338700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf81988a0 con 0x7fccf8101bb0 2024-04-22T23:29:53.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.304+0000 7fccf6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccf80ff340 0x7fccf819e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:53.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.304+0000 7fccf6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccf80ff340 0x7fccf819e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58174/0 (socket says 172.21.15.67:58174) 2024-04-22T23:29:53.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.304+0000 7fccf6ffd700 1 -- 172.21.15.67:0/179705095 learned_addr learned my addr 172.21.15.67:0/179705095 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:29:53.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.304+0000 7fccf67fc700 1 --2- 172.21.15.67:0/179705095 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fccf8101bb0 0x7fccf819ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:53.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.304+0000 7fccf6ffd700 1 -- 172.21.15.67:0/179705095 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fccf8101bb0 msgr2=0x7fccf819ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:53.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.304+0000 7fccf6ffd700 1 --2- 172.21.15.67:0/179705095 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fccf8101bb0 0x7fccf819ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:53.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.304+0000 7fccf6ffd700 1 -- 172.21.15.67:0/179705095 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccec0095f0 con 0x7fccf80ff340 2024-04-22T23:29:53.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.304+0000 7fccf6ffd700 1 --2- 172.21.15.67:0/179705095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccf80ff340 0x7fccf819e820 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcce800e940 tx=0x7fcce800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:29:53.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.304+0000 7fccdffff700 1 -- 172.21.15.67:0/179705095 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcce80097b0 con 0x7fccf80ff340 2024-04-22T23:29:53.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.304+0000 7fccfd338700 1 -- 172.21.15.67:0/179705095 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccf8198b80 con 0x7fccf80ff340 2024-04-22T23:29:53.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.304+0000 7fccdffff700 1 -- 172.21.15.67:0/179705095 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcce8004cf0 con 0x7fccf80ff340 2024-04-22T23:29:53.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.304+0000 7fccdffff700 1 -- 172.21.15.67:0/179705095 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcce8005700 con 0x7fccf80ff340 2024-04-22T23:29:53.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.304+0000 7fccfd338700 1 -- 172.21.15.67:0/179705095 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccf81990d0 con 0x7fccf80ff340 2024-04-22T23:29:53.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.306+0000 7fccfd338700 1 -- 172.21.15.67:0/179705095 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccf810b230 con 0x7fccf80ff340 2024-04-22T23:29:53.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.306+0000 7fccdffff700 1 -- 172.21.15.67:0/179705095 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcce800cba0 con 0x7fccf80ff340 2024-04-22T23:29:53.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.307+0000 7fccdffff700 1 --2- 172.21.15.67:0/179705095 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcce006ecc0 0x7fcce0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:53.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.307+0000 7fccdffff700 1 -- 172.21.15.67:0/179705095 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fcce8014070 con 0x7fccf80ff340 2024-04-22T23:29:53.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.307+0000 7fccf67fc700 1 --2- 172.21.15.67:0/179705095 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcce006ecc0 0x7fcce0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:53.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.308+0000 7fccf67fc700 1 --2- 172.21.15.67:0/179705095 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcce006ecc0 0x7fcce0071180 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fccec00b760 tx=0x7fccec009f90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:29:53.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.313+0000 7fccdffff700 1 -- 172.21.15.67:0/179705095 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fcce805baf0 con 0x7fccf80ff340 2024-04-22T23:29:53.414 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:53 smithi067 podman[129294]: 2024-04-22 23:29:53.139334456 +0000 UTC m=+1.098997775 container died 915b07a8edc4c2d96f712b7dd84743172418bf62014c48e9584cd55eeb835e14 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T23:29:53.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.548+0000 7fccfd338700 1 -- 172.21.15.67:0/179705095 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fccf81992f0 con 0x7fcce006ecc0 2024-04-22T23:29:53.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.550+0000 7fccdffff700 1 -- 172.21.15.67:0/179705095 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fccf81992f0 con 0x7fcce006ecc0 2024-04-22T23:29:53.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.552+0000 7fccfd338700 1 -- 172.21.15.67:0/179705095 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcce006ecc0 msgr2=0x7fcce0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:53.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.552+0000 7fccfd338700 1 --2- 172.21.15.67:0/179705095 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcce006ecc0 0x7fcce0071180 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fccec00b760 tx=0x7fccec009f90 comp rx=0 tx=0).stop 2024-04-22T23:29:53.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.552+0000 7fccfd338700 1 -- 172.21.15.67:0/179705095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccf80ff340 msgr2=0x7fccf819e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:53.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.552+0000 7fccfd338700 1 --2- 172.21.15.67:0/179705095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccf80ff340 0x7fccf819e820 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcce800e940 tx=0x7fcce800ed00 comp rx=0 tx=0).stop 2024-04-22T23:29:53.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.552+0000 7fccfd338700 1 -- 172.21.15.67:0/179705095 shutdown_connections 2024-04-22T23:29:53.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.552+0000 7fccfd338700 1 --2- 172.21.15.67:0/179705095 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcce006ecc0 0x7fcce0071180 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:53.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.553+0000 7fccfd338700 1 --2- 172.21.15.67:0/179705095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccf80ff340 0x7fccf819e820 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:53.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.553+0000 7fccfd338700 1 --2- 172.21.15.67:0/179705095 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fccf8101bb0 0x7fccf819ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:53.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.553+0000 7fccfd338700 1 -- 172.21.15.67:0/179705095 >> 172.21.15.67:0/179705095 conn(0x7fccf8074e30 msgr2=0x7fccf80fd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:29:53.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.553+0000 7fccfd338700 1 -- 172.21.15.67:0/179705095 shutdown_connections 2024-04-22T23:29:53.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.553+0000 7fccfd338700 1 -- 172.21.15.67:0/179705095 wait complete. 2024-04-22T23:29:53.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.670+0000 7fac030a6700 1 -- 172.21.15.67:0/15367325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabfc101de0 msgr2=0x7fabfc1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:53.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.670+0000 7fac030a6700 1 --2- 172.21.15.67:0/15367325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabfc101de0 0x7fabfc1021c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fabf0009910 tx=0x7fabf0009c20 comp rx=0 tx=0).stop 2024-04-22T23:29:53.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.670+0000 7fac030a6700 1 -- 172.21.15.67:0/15367325 shutdown_connections 2024-04-22T23:29:53.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.670+0000 7fac030a6700 1 --2- 172.21.15.67:0/15367325 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabfc102700 0x7fabfc10abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:53.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.670+0000 7fac030a6700 1 --2- 172.21.15.67:0/15367325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabfc101de0 0x7fabfc1021c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:53.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.670+0000 7fac030a6700 1 -- 172.21.15.67:0/15367325 >> 172.21.15.67:0/15367325 conn(0x7fabfc0fb650 msgr2=0x7fabfc0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:29:53.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.670+0000 7fac030a6700 1 -- 172.21.15.67:0/15367325 shutdown_connections 2024-04-22T23:29:53.669 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:53 smithi067 podman[129294]: 2024-04-22 23:29:53.533170224 +0000 UTC m=+1.492833533 container cleanup 915b07a8edc4c2d96f712b7dd84743172418bf62014c48e9584cd55eeb835e14 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux ) 2024-04-22T23:29:53.669 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:53 smithi067 bash[129294]: ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1 2024-04-22T23:29:53.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.670+0000 7fac030a6700 1 -- 172.21.15.67:0/15367325 wait complete. 2024-04-22T23:29:53.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.671+0000 7fac030a6700 1 Processor -- start 2024-04-22T23:29:53.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.671+0000 7fac030a6700 1 -- start start 2024-04-22T23:29:53.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.671+0000 7fac030a6700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabfc101de0 0x7fabfc0ffa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:53.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.671+0000 7fac030a6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabfc102700 0x7fabfc0fff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:53.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.671+0000 7fac030a6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabfc1004c0 con 0x7fabfc102700 2024-04-22T23:29:53.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.671+0000 7fac030a6700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabfc100600 con 0x7fabfc101de0 2024-04-22T23:29:53.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.671+0000 7fabfbfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabfc102700 0x7fabfc0fff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:53.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.671+0000 7fabfbfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabfc102700 0x7fabfc0fff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58194/0 (socket says 172.21.15.67:58194) 2024-04-22T23:29:53.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.671+0000 7fabfbfff700 1 -- 172.21.15.67:0/2199600620 learned_addr learned my addr 172.21.15.67:0/2199600620 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:29:53.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.671+0000 7fac00e42700 1 --2- 172.21.15.67:0/2199600620 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabfc101de0 0x7fabfc0ffa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:53.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.672+0000 7fabfbfff700 1 -- 172.21.15.67:0/2199600620 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabfc101de0 msgr2=0x7fabfc0ffa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:53.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.672+0000 7fabfbfff700 1 --2- 172.21.15.67:0/2199600620 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabfc101de0 0x7fabfc0ffa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:53.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.672+0000 7fabfbfff700 1 -- 172.21.15.67:0/2199600620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf00095f0 con 0x7fabfc102700 2024-04-22T23:29:53.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.672+0000 7fabfbfff700 1 --2- 172.21.15.67:0/2199600620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabfc102700 0x7fabfc0fff80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fabe800d6d0 tx=0x7fabe800d9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:29:53.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.672+0000 7fabf9ffb700 1 -- 172.21.15.67:0/2199600620 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabe800f860 con 0x7fabfc102700 2024-04-22T23:29:53.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.672+0000 7fabf9ffb700 1 -- 172.21.15.67:0/2199600620 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fabe800fe80 con 0x7fabfc102700 2024-04-22T23:29:53.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.672+0000 7fac030a6700 1 -- 172.21.15.67:0/2199600620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabfc1a2aa0 con 0x7fabfc102700 2024-04-22T23:29:53.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.672+0000 7fabf9ffb700 1 -- 172.21.15.67:0/2199600620 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabe8010ea0 con 0x7fabfc102700 2024-04-22T23:29:53.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.672+0000 7fac030a6700 1 -- 172.21.15.67:0/2199600620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabfc1a2eb0 con 0x7fabfc102700 2024-04-22T23:29:53.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.673+0000 7fac030a6700 1 -- 172.21.15.67:0/2199600620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabfc1082f0 con 0x7fabfc102700 2024-04-22T23:29:53.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.674+0000 7fabf9ffb700 1 -- 172.21.15.67:0/2199600620 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fabe8010460 con 0x7fabfc102700 2024-04-22T23:29:53.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.674+0000 7fabf9ffb700 1 --2- 172.21.15.67:0/2199600620 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fabec06ecc0 0x7fabec071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:53.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.674+0000 7fabf9ffb700 1 -- 172.21.15.67:0/2199600620 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fabe808ea30 con 0x7fabfc102700 2024-04-22T23:29:53.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.674+0000 7fac00e42700 1 --2- 172.21.15.67:0/2199600620 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fabec06ecc0 0x7fabec071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:53.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.675+0000 7fac00e42700 1 --2- 172.21.15.67:0/2199600620 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fabec06ecc0 0x7fabec071180 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fabf000b5c0 tx=0x7fabf001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:29:53.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.679+0000 7fabf9ffb700 1 -- 172.21.15.67:0/2199600620 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fabe80578c0 con 0x7fabfc102700 2024-04-22T23:29:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.914+0000 7fac030a6700 1 -- 172.21.15.67:0/2199600620 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fabfc0611d0 con 0x7fabec06ecc0 2024-04-22T23:29:53.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.925+0000 7fabf9ffb700 1 -- 172.21.15.67:0/2199600620 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7fabfc0611d0 con 0x7fabec06ecc0 2024-04-22T23:29:53.924 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:53 smithi067 podman[129335]: 2024-04-22 23:29:53.843579677 +0000 UTC m=+0.696208123 container remove 915b07a8edc4c2d96f712b7dd84743172418bf62014c48e9584cd55eeb835e14 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-22T23:29:53.924 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:53 smithi067 ceph-mon[112438]: pgmap v66: 65 pgs: 65 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:29:53.924 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:53 smithi067 ceph-mon[112438]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-22T23:29:53.924 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:53 smithi067 ceph-mon[112438]: osdmap e55: 6 total, 5 up, 6 in 2024-04-22T23:29:53.924 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:29:53.924 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (3m) 27s ago 14m 26.1M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:29:53.924 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (15m) 27s ago 15m 8262k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:29:53.925 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (14m) 50s ago 14m 8200k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:29:53.925 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (58s) 27s ago 15m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:29:53.925 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (53s) 50s ago 14m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:29:53.925 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (3m) 27s ago 14m 76.8M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:29:53.925 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (9m) 27s ago 9m 62.7M - 18.2.1 5be31c24972a e24a471f0a8c 2024-04-22T23:29:53.925 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (9m) 27s ago 9m 130M - 18.2.1 5be31c24972a 0730f9fc8e61 2024-04-22T23:29:53.925 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (9m) 50s ago 9m 123M - 18.2.1 5be31c24972a b6a169586433 2024-04-22T23:29:53.925 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (9m) 50s ago 9m 67.7M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:29:53.925 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (5m) 27s ago 16m 500M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:29:53.925 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (4m) 50s ago 13m 450M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:29:53.926 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (2m) 27s ago 16m 46.0M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:29:53.926 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (101s) 50s ago 13m 38.2M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:29:53.926 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (4m) 27s ago 14m 15.1M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:29:53.926 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (4m) 50s ago 13m 15.9M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:29:53.926 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (33s) 27s ago 13m 12.6M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:29:53.926 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (12m) 27s ago 12m 213M 1216M 18.2.1 5be31c24972a 915b07a8edc4 2024-04-22T23:29:53.926 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (12m) 27s ago 12m 173M 1216M 18.2.1 5be31c24972a e257ecb4efa1 2024-04-22T23:29:53.926 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (11m) 50s ago 11m 251M 2240M 18.2.1 5be31c24972a 9c2c91a967ba 2024-04-22T23:29:53.926 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (11m) 50s ago 11m 219M 2240M 18.2.1 5be31c24972a c7586562ffa6 2024-04-22T23:29:53.926 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (11m) 50s ago 10m 196M 2240M 18.2.1 5be31c24972a 9fd97f1c65cb 2024-04-22T23:29:53.926 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (3m) 27s ago 14m 71.2M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:29:53.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.928+0000 7fac030a6700 1 -- 172.21.15.67:0/2199600620 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fabec06ecc0 msgr2=0x7fabec071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:53.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.928+0000 7fac030a6700 1 --2- 172.21.15.67:0/2199600620 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fabec06ecc0 0x7fabec071180 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fabf000b5c0 tx=0x7fabf001a040 comp rx=0 tx=0).stop 2024-04-22T23:29:53.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.928+0000 7fac030a6700 1 -- 172.21.15.67:0/2199600620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabfc102700 msgr2=0x7fabfc0fff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:53.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.928+0000 7fac030a6700 1 --2- 172.21.15.67:0/2199600620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabfc102700 0x7fabfc0fff80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fabe800d6d0 tx=0x7fabe800d9e0 comp rx=0 tx=0).stop 2024-04-22T23:29:53.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.928+0000 7fac030a6700 1 -- 172.21.15.67:0/2199600620 shutdown_connections 2024-04-22T23:29:53.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.928+0000 7fac030a6700 1 --2- 172.21.15.67:0/2199600620 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fabec06ecc0 0x7fabec071180 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:53.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.928+0000 7fac030a6700 1 --2- 172.21.15.67:0/2199600620 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabfc101de0 0x7fabfc0ffa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:53.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.928+0000 7fac030a6700 1 --2- 172.21.15.67:0/2199600620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabfc102700 0x7fabfc0fff80 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:53.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.928+0000 7fac030a6700 1 -- 172.21.15.67:0/2199600620 >> 172.21.15.67:0/2199600620 conn(0x7fabfc0fb650 msgr2=0x7fabfc105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:29:53.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.928+0000 7fac030a6700 1 -- 172.21.15.67:0/2199600620 shutdown_connections 2024-04-22T23:29:53.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:53.928+0000 7fac030a6700 1 -- 172.21.15.67:0/2199600620 wait complete. 2024-04-22T23:29:53.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:53 smithi134 ceph-mon[85630]: pgmap v66: 65 pgs: 65 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:29:53.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:53 smithi134 ceph-mon[85630]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-22T23:29:53.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:53 smithi134 ceph-mon[85630]: osdmap e55: 6 total, 5 up, 6 in 2024-04-22T23:29:54.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.050+0000 7fe4428be700 1 -- 172.21.15.67:0/3703121308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe43c0ffda0 msgr2=0x7fe43c100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:54.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.050+0000 7fe4428be700 1 --2- 172.21.15.67:0/3703121308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe43c0ffda0 0x7fe43c100180 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe42c009910 tx=0x7fe42c009c20 comp rx=0 tx=0).stop 2024-04-22T23:29:54.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.050+0000 7fe4428be700 1 -- 172.21.15.67:0/3703121308 shutdown_connections 2024-04-22T23:29:54.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.050+0000 7fe4428be700 1 --2- 172.21.15.67:0/3703121308 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe43c100750 0x7fe43c1047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:54.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.050+0000 7fe4428be700 1 --2- 172.21.15.67:0/3703121308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe43c0ffda0 0x7fe43c100180 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:54.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.050+0000 7fe4428be700 1 -- 172.21.15.67:0/3703121308 >> 172.21.15.67:0/3703121308 conn(0x7fe43c0fb650 msgr2=0x7fe43c0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:29:54.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.051+0000 7fe4428be700 1 -- 172.21.15.67:0/3703121308 shutdown_connections 2024-04-22T23:29:54.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.051+0000 7fe4428be700 1 -- 172.21.15.67:0/3703121308 wait complete. 2024-04-22T23:29:54.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.051+0000 7fe4428be700 1 Processor -- start 2024-04-22T23:29:54.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.052+0000 7fe4428be700 1 -- start start 2024-04-22T23:29:54.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.052+0000 7fe4428be700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe43c0ffda0 0x7fe43c19b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:54.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.052+0000 7fe4428be700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe43c100750 0x7fe43c19bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:54.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.052+0000 7fe4428be700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe43c19c150 con 0x7fe43c100750 2024-04-22T23:29:54.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.052+0000 7fe4428be700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe43c195600 con 0x7fe43c0ffda0 2024-04-22T23:29:54.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.052+0000 7fe43b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe43c100750 0x7fe43c19bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:54.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.052+0000 7fe43b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe43c100750 0x7fe43c19bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58210/0 (socket says 172.21.15.67:58210) 2024-04-22T23:29:54.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.052+0000 7fe43b7fe700 1 -- 172.21.15.67:0/1700413877 learned_addr learned my addr 172.21.15.67:0/1700413877 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:29:54.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.052+0000 7fe43bfff700 1 --2- 172.21.15.67:0/1700413877 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe43c0ffda0 0x7fe43c19b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:54.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.053+0000 7fe43b7fe700 1 -- 172.21.15.67:0/1700413877 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe43c0ffda0 msgr2=0x7fe43c19b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:54.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.053+0000 7fe43b7fe700 1 --2- 172.21.15.67:0/1700413877 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe43c0ffda0 0x7fe43c19b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:54.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.053+0000 7fe43b7fe700 1 -- 172.21.15.67:0/1700413877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe42c0095f0 con 0x7fe43c100750 2024-04-22T23:29:54.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.053+0000 7fe43bfff700 1 --2- 172.21.15.67:0/1700413877 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe43c0ffda0 0x7fe43c19b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:29:54.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.053+0000 7fe43b7fe700 1 --2- 172.21.15.67:0/1700413877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe43c100750 0x7fe43c19bac0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe43000c8f0 tx=0x7fe43000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:29:54.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.053+0000 7fe4397fa700 1 -- 172.21.15.67:0/1700413877 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe430007a50 con 0x7fe43c100750 2024-04-22T23:29:54.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.053+0000 7fe4428be700 1 -- 172.21.15.67:0/1700413877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe43c195880 con 0x7fe43c100750 2024-04-22T23:29:54.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.053+0000 7fe4397fa700 1 -- 172.21.15.67:0/1700413877 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe430007bb0 con 0x7fe43c100750 2024-04-22T23:29:54.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.053+0000 7fe4397fa700 1 -- 172.21.15.67:0/1700413877 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe430018630 con 0x7fe43c100750 2024-04-22T23:29:54.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.053+0000 7fe4428be700 1 -- 172.21.15.67:0/1700413877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe43c195dd0 con 0x7fe43c100750 2024-04-22T23:29:54.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.054+0000 7fe4428be700 1 -- 172.21.15.67:0/1700413877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe43c04f2e0 con 0x7fe43c100750 2024-04-22T23:29:54.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.055+0000 7fe4397fa700 1 -- 172.21.15.67:0/1700413877 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe430007d20 con 0x7fe43c100750 2024-04-22T23:29:54.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.056+0000 7fe4397fa700 1 --2- 172.21.15.67:0/1700413877 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe42406ecc0 0x7fe424071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:54.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.056+0000 7fe4397fa700 1 -- 172.21.15.67:0/1700413877 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fe43008ee50 con 0x7fe43c100750 2024-04-22T23:29:54.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.056+0000 7fe43bfff700 1 --2- 172.21.15.67:0/1700413877 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe42406ecc0 0x7fe424071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:54.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.057+0000 7fe43bfff700 1 --2- 172.21.15.67:0/1700413877 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe42406ecc0 0x7fe424071180 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe42c0059d0 tx=0x7fe42c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:29:54.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.061+0000 7fe4397fa700 1 -- 172.21.15.67:0/1700413877 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe43005b730 con 0x7fe43c100750 2024-04-22T23:29:54.177 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:54 smithi067 podman[129493]: 2024-04-22 23:29:54.169894363 +0000 UTC m=+0.105438512 container create 0d24dbb08d17be9b6935654e19eec0d752f5dea4bf3a8c2cbba82ae877ffc96a (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-22T23:29:54.177 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:54 smithi067 podman[129493]: 2024-04-22 23:29:54.083681549 +0000 UTC m=+0.019225691 image pull a0434699c1d76af4eea1fc95ff0a22ea73d1e04e609464fbf658d6f39bc7f2fc quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c 2024-04-22T23:29:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.379+0000 7fe4428be700 1 -- 172.21.15.67:0/1700413877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe43c196b10 con 0x7fe43c100750 2024-04-22T23:29:54.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.380+0000 7fe4397fa700 1 -- 172.21.15.67:0/1700413877 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7fe43005b2c0 con 0x7fe43c100750 2024-04-22T23:29:54.378 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:29:54.378 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:29:54.378 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:29:54.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:29:54.378 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:29:54.378 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:29:54.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:29:54.379 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:29:54.379 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4, 2024-04-22T23:29:54.379 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 1 2024-04-22T23:29:54.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:29:54.379 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:29:54.379 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-22T23:29:54.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:29:54.379 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:29:54.379 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 8, 2024-04-22T23:29:54.379 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 5 2024-04-22T23:29:54.379 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:29:54.379 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:29:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.382+0000 7fe4428be700 1 -- 172.21.15.67:0/1700413877 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe42406ecc0 msgr2=0x7fe424071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.382+0000 7fe4428be700 1 --2- 172.21.15.67:0/1700413877 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe42406ecc0 0x7fe424071180 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe42c0059d0 tx=0x7fe42c01a040 comp rx=0 tx=0).stop 2024-04-22T23:29:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.382+0000 7fe4428be700 1 -- 172.21.15.67:0/1700413877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe43c100750 msgr2=0x7fe43c19bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.382+0000 7fe4428be700 1 --2- 172.21.15.67:0/1700413877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe43c100750 0x7fe43c19bac0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe43000c8f0 tx=0x7fe43000ccb0 comp rx=0 tx=0).stop 2024-04-22T23:29:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.382+0000 7fe4428be700 1 -- 172.21.15.67:0/1700413877 shutdown_connections 2024-04-22T23:29:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.382+0000 7fe4428be700 1 --2- 172.21.15.67:0/1700413877 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe42406ecc0 0x7fe424071180 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:54.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.382+0000 7fe4428be700 1 --2- 172.21.15.67:0/1700413877 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe43c0ffda0 0x7fe43c19b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:54.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.382+0000 7fe4428be700 1 --2- 172.21.15.67:0/1700413877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe43c100750 0x7fe43c19bac0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:54.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.382+0000 7fe4428be700 1 -- 172.21.15.67:0/1700413877 >> 172.21.15.67:0/1700413877 conn(0x7fe43c0fb650 msgr2=0x7fe43c0fcd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:29:54.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.382+0000 7fe4428be700 1 -- 172.21.15.67:0/1700413877 shutdown_connections 2024-04-22T23:29:54.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.382+0000 7fe4428be700 1 -- 172.21.15.67:0/1700413877 wait complete. 2024-04-22T23:29:54.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.493+0000 7fd23f168700 1 -- 172.21.15.67:0/2600257169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2380ffc20 msgr2=0x7fd238100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:54.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.493+0000 7fd23f168700 1 --2- 172.21.15.67:0/2600257169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2380ffc20 0x7fd238100080 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd224009910 tx=0x7fd224009c20 comp rx=0 tx=0).stop 2024-04-22T23:29:54.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.493+0000 7fd23f168700 1 -- 172.21.15.67:0/2600257169 shutdown_connections 2024-04-22T23:29:54.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.493+0000 7fd23f168700 1 --2- 172.21.15.67:0/2600257169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2380ffc20 0x7fd238100080 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:54.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.493+0000 7fd23f168700 1 --2- 172.21.15.67:0/2600257169 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd238105bd0 0x7fd238105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:54.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.493+0000 7fd23f168700 1 -- 172.21.15.67:0/2600257169 >> 172.21.15.67:0/2600257169 conn(0x7fd238075670 msgr2=0x7fd238075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:29:54.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.493+0000 7fd23f168700 1 -- 172.21.15.67:0/2600257169 shutdown_connections 2024-04-22T23:29:54.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.493+0000 7fd23f168700 1 -- 172.21.15.67:0/2600257169 wait complete. 2024-04-22T23:29:54.492 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:54 smithi067 podman[129493]: 2024-04-22 23:29:54.423179853 +0000 UTC m=+0.358724007 container init 0d24dbb08d17be9b6935654e19eec0d752f5dea4bf3a8c2cbba82ae877ffc96a (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-22T23:29:54.492 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:54 smithi067 podman[129493]: 2024-04-22 23:29:54.426685035 +0000 UTC m=+0.362229189 container start 0d24dbb08d17be9b6935654e19eec0d752f5dea4bf3a8c2cbba82ae877ffc96a (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-22T23:29:54.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.494+0000 7fd23f168700 1 Processor -- start 2024-04-22T23:29:54.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.494+0000 7fd23f168700 1 -- start start 2024-04-22T23:29:54.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.494+0000 7fd23f168700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2380ffc20 0x7fd238197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:54.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.494+0000 7fd23f168700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd238105bd0 0x7fd238197c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:54.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.494+0000 7fd23f168700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2381982d0 con 0x7fd2380ffc20 2024-04-22T23:29:54.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.494+0000 7fd23f168700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd238198440 con 0x7fd238105bd0 2024-04-22T23:29:54.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.494+0000 7fd23cf04700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2380ffc20 0x7fd238197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:54.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.495+0000 7fd23cf04700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2380ffc20 0x7fd238197740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58242/0 (socket says 172.21.15.67:58242) 2024-04-22T23:29:54.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.495+0000 7fd23cf04700 1 -- 172.21.15.67:0/3137429206 learned_addr learned my addr 172.21.15.67:0/3137429206 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:29:54.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.495+0000 7fd237fff700 1 --2- 172.21.15.67:0/3137429206 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd238105bd0 0x7fd238197c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:54.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.495+0000 7fd23cf04700 1 -- 172.21.15.67:0/3137429206 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd238105bd0 msgr2=0x7fd238197c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:54.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.495+0000 7fd23cf04700 1 --2- 172.21.15.67:0/3137429206 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd238105bd0 0x7fd238197c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:54.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.495+0000 7fd23cf04700 1 -- 172.21.15.67:0/3137429206 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2240095f0 con 0x7fd2380ffc20 2024-04-22T23:29:54.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.495+0000 7fd237fff700 1 --2- 172.21.15.67:0/3137429206 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd238105bd0 0x7fd238197c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:29:54.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.495+0000 7fd23cf04700 1 --2- 172.21.15.67:0/3137429206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2380ffc20 0x7fd238197740 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd22c00c930 tx=0x7fd22c00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:29:54.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.495+0000 7fd235ffb700 1 -- 172.21.15.67:0/3137429206 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd22c007ad0 con 0x7fd2380ffc20 2024-04-22T23:29:54.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.495+0000 7fd23f168700 1 -- 172.21.15.67:0/3137429206 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2381a07b0 con 0x7fd2380ffc20 2024-04-22T23:29:54.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.495+0000 7fd235ffb700 1 -- 172.21.15.67:0/3137429206 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd22c007c30 con 0x7fd2380ffc20 2024-04-22T23:29:54.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.495+0000 7fd235ffb700 1 -- 172.21.15.67:0/3137429206 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd22c018610 con 0x7fd2380ffc20 2024-04-22T23:29:54.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.496+0000 7fd23f168700 1 -- 172.21.15.67:0/3137429206 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2381a0c70 con 0x7fd2380ffc20 2024-04-22T23:29:54.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.497+0000 7fd23f168700 1 -- 172.21.15.67:0/3137429206 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2381080d0 con 0x7fd2380ffc20 2024-04-22T23:29:54.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.497+0000 7fd235ffb700 1 -- 172.21.15.67:0/3137429206 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd22c00f450 con 0x7fd2380ffc20 2024-04-22T23:29:54.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.498+0000 7fd235ffb700 1 --2- 172.21.15.67:0/3137429206 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd22806ecc0 0x7fd228071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:54.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.498+0000 7fd235ffb700 1 -- 172.21.15.67:0/3137429206 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fd22c08efa0 con 0x7fd2380ffc20 2024-04-22T23:29:54.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.498+0000 7fd237fff700 1 --2- 172.21.15.67:0/3137429206 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd22806ecc0 0x7fd228071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:54.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.498+0000 7fd237fff700 1 --2- 172.21.15.67:0/3137429206 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd22806ecc0 0x7fd228071180 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd224005b40 tx=0x7fd22401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:29:54.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.503+0000 7fd235ffb700 1 -- 172.21.15.67:0/3137429206 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fd22c057db0 con 0x7fd2380ffc20 2024-04-22T23:29:54.769 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:54 smithi067 podman[129493]: 2024-04-22 23:29:54.497817054 +0000 UTC m=+0.433361211 container attach 0d24dbb08d17be9b6935654e19eec0d752f5dea4bf3a8c2cbba82ae877ffc96a (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T23:29:54.769 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:54 smithi067 podman[129493]: 2024-04-22 23:29:54.73457355 +0000 UTC m=+0.670117699 container died 0d24dbb08d17be9b6935654e19eec0d752f5dea4bf3a8c2cbba82ae877ffc96a (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1-deactivate, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416) 2024-04-22T23:29:54.769 INFO:teuthology.orchestra.run.smithi067.stdout:e22 2024-04-22T23:29:54.769 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:29:54.770 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:29:54.770 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:29:54.770 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:29:54.770 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:29:54.770 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:29:54.770 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 22 2024-04-22T23:29:54.770 INFO:teuthology.orchestra.run.smithi067.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T23:29:54.770 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:29:54.770 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:27:20.577198+0000 2024-04-22T23:29:54.770 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:29:54.770 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:29:54.770 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:29:54.770 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:29:54.771 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:29:54.771 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:29:54.771 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:29:54.771 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:29:54.771 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 46 2024-04-22T23:29:54.771 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:29:54.771 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:29:54.771 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:29:54.771 INFO:teuthology.orchestra.run.smithi067.stdout:up {0=14484,1=24273} 2024-04-22T23:29:54.771 INFO:teuthology.orchestra.run.smithi067.stdout:failed 2024-04-22T23:29:54.771 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:29:54.771 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:29:54.771 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:29:54.771 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:29:54.771 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:29:54.772 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:29:54.772 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:29:54.772 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:29:54.772 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:29:54.772 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{0:14484} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.67:6828/3606151860,v1:172.21.15.67:6829/3606151860] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:29:54.772 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{0:14496} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/3305304988,v1:172.21.15.67:6827/3305304988] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:29:54.772 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{1:24273} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.134:6824/535957142,v1:172.21.15.134:6825/535957142] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:29:54.772 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{1:24289} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.134:6826/1954821676,v1:172.21.15.134:6827/1954821676] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:29:54.772 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:29:54.772 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:29:54.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.770+0000 7fd23f168700 1 -- 172.21.15.67:0/3137429206 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd23804f2e0 con 0x7fd2380ffc20 2024-04-22T23:29:54.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.771+0000 7fd235ffb700 1 -- 172.21.15.67:0/3137429206 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1901 (secure 0 0 0) 0x7fd22c05b470 con 0x7fd2380ffc20 2024-04-22T23:29:54.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.774+0000 7fd23f168700 1 -- 172.21.15.67:0/3137429206 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd22806ecc0 msgr2=0x7fd228071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:54.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.774+0000 7fd23f168700 1 --2- 172.21.15.67:0/3137429206 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd22806ecc0 0x7fd228071180 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd224005b40 tx=0x7fd22401a040 comp rx=0 tx=0).stop 2024-04-22T23:29:54.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.774+0000 7fd23f168700 1 -- 172.21.15.67:0/3137429206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2380ffc20 msgr2=0x7fd238197740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:54.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.774+0000 7fd23f168700 1 --2- 172.21.15.67:0/3137429206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2380ffc20 0x7fd238197740 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd22c00c930 tx=0x7fd22c00ccf0 comp rx=0 tx=0).stop 2024-04-22T23:29:54.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.774+0000 7fd23f168700 1 -- 172.21.15.67:0/3137429206 shutdown_connections 2024-04-22T23:29:54.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.774+0000 7fd23f168700 1 --2- 172.21.15.67:0/3137429206 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd22806ecc0 0x7fd228071180 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:54.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.774+0000 7fd23f168700 1 --2- 172.21.15.67:0/3137429206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2380ffc20 0x7fd238197740 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:54.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.774+0000 7fd23f168700 1 --2- 172.21.15.67:0/3137429206 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd238105bd0 0x7fd238197c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:54.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.774+0000 7fd23f168700 1 -- 172.21.15.67:0/3137429206 >> 172.21.15.67:0/3137429206 conn(0x7fd238075670 msgr2=0x7fd2380fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:29:54.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.774+0000 7fd23f168700 1 -- 172.21.15.67:0/3137429206 shutdown_connections 2024-04-22T23:29:54.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.774+0000 7fd23f168700 1 -- 172.21.15.67:0/3137429206 wait complete. 2024-04-22T23:29:54.774 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 22 2024-04-22T23:29:54.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.885+0000 7f93cc29b700 1 -- 172.21.15.67:0/994226270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93c4100750 msgr2=0x7f93c41047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:54.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.885+0000 7f93cc29b700 1 --2- 172.21.15.67:0/994226270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93c4100750 0x7f93c41047a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f93b8009910 tx=0x7f93b8009c20 comp rx=0 tx=0).stop 2024-04-22T23:29:54.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.885+0000 7f93cc29b700 1 -- 172.21.15.67:0/994226270 shutdown_connections 2024-04-22T23:29:54.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.885+0000 7f93cc29b700 1 --2- 172.21.15.67:0/994226270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93c4100750 0x7f93c41047a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:54.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.885+0000 7f93cc29b700 1 --2- 172.21.15.67:0/994226270 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93c40ffda0 0x7f93c4100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:54.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.885+0000 7f93cc29b700 1 -- 172.21.15.67:0/994226270 >> 172.21.15.67:0/994226270 conn(0x7f93c40fb650 msgr2=0x7f93c40fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:29:54.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.885+0000 7f93cc29b700 1 -- 172.21.15.67:0/994226270 shutdown_connections 2024-04-22T23:29:54.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.886+0000 7f93cc29b700 1 -- 172.21.15.67:0/994226270 wait complete. 2024-04-22T23:29:54.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.886+0000 7f93cc29b700 1 Processor -- start 2024-04-22T23:29:54.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.886+0000 7f93cc29b700 1 -- start start 2024-04-22T23:29:54.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.886+0000 7f93cc29b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93c40ffda0 0x7f93c410ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:54.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.886+0000 7f93cc29b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93c4100750 0x7f93c410b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:54.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.886+0000 7f93cc29b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c410b590 con 0x7f93c4100750 2024-04-22T23:29:54.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.886+0000 7f93cc29b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c410b6d0 con 0x7f93c40ffda0 2024-04-22T23:29:54.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.886+0000 7f93c9836700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93c4100750 0x7f93c410b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:54.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.887+0000 7f93c9836700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93c4100750 0x7f93c410b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58270/0 (socket says 172.21.15.67:58270) 2024-04-22T23:29:54.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.887+0000 7f93c9836700 1 -- 172.21.15.67:0/794710365 learned_addr learned my addr 172.21.15.67:0/794710365 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:29:54.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.887+0000 7f93ca037700 1 --2- 172.21.15.67:0/794710365 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93c40ffda0 0x7f93c410ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:54.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.887+0000 7f93c9836700 1 -- 172.21.15.67:0/794710365 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93c40ffda0 msgr2=0x7f93c410ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:54.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.887+0000 7f93c9836700 1 --2- 172.21.15.67:0/794710365 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93c40ffda0 0x7f93c410ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:54.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.887+0000 7f93c9836700 1 -- 172.21.15.67:0/794710365 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b80095f0 con 0x7f93c4100750 2024-04-22T23:29:54.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.887+0000 7f93c9836700 1 --2- 172.21.15.67:0/794710365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93c4100750 0x7f93c410b050 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f93b80092e0 tx=0x7f93b800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:29:54.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.887+0000 7f93c37fe700 1 -- 172.21.15.67:0/794710365 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93b801d070 con 0x7f93c4100750 2024-04-22T23:29:54.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.887+0000 7f93c37fe700 1 -- 172.21.15.67:0/794710365 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f93b800f460 con 0x7f93c4100750 2024-04-22T23:29:54.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.887+0000 7f93cc29b700 1 -- 172.21.15.67:0/794710365 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93c410b950 con 0x7f93c4100750 2024-04-22T23:29:54.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.887+0000 7f93c37fe700 1 -- 172.21.15.67:0/794710365 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93b80175d0 con 0x7f93c4100750 2024-04-22T23:29:54.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.887+0000 7f93cc29b700 1 -- 172.21.15.67:0/794710365 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93c41a2e70 con 0x7f93c4100750 2024-04-22T23:29:54.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.888+0000 7f93cc29b700 1 -- 172.21.15.67:0/794710365 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93c41081a0 con 0x7f93c4100750 2024-04-22T23:29:54.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.889+0000 7f93c37fe700 1 -- 172.21.15.67:0/794710365 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f93b800f5d0 con 0x7f93c4100750 2024-04-22T23:29:54.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.889+0000 7f93c37fe700 1 --2- 172.21.15.67:0/794710365 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f93ac06ecc0 0x7f93ac071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:54.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.889+0000 7f93c37fe700 1 -- 172.21.15.67:0/794710365 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f93b8090290 con 0x7f93c4100750 2024-04-22T23:29:54.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.889+0000 7f93ca037700 1 --2- 172.21.15.67:0/794710365 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f93ac06ecc0 0x7f93ac071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:54.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.890+0000 7f93ca037700 1 --2- 172.21.15.67:0/794710365 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f93ac06ecc0 0x7f93ac071180 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f93bc005950 tx=0x7f93bc00a300 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:29:54.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:54.894+0000 7f93c37fe700 1 -- 172.21.15.67:0/794710365 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f93b80590a0 con 0x7f93c4100750 2024-04-22T23:29:55.041 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:54 smithi067 ceph-mon[112438]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:29:55.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:54 smithi067 ceph-mon[112438]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:29:55.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:54 smithi067 ceph-mon[112438]: osdmap e56: 6 total, 5 up, 6 in 2024-04-22T23:29:55.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:54 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1700413877' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:29:55.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:54 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3137429206' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:29:55.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.128+0000 7f93cc29b700 1 -- 172.21.15.67:0/794710365 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f93c410c150 con 0x7f93ac06ecc0 2024-04-22T23:29:55.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.129+0000 7f93c37fe700 1 -- 172.21.15.67:0/794710365 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f93c410c150 con 0x7f93ac06ecc0 2024-04-22T23:29:55.127 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:29:55.127 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:29:55.127 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:29:55.127 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:29:55.127 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:29:55.128 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:29:55.128 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:29:55.128 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr" 2024-04-22T23:29:55.128 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:29:55.128 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "7/23 daemons upgraded", 2024-04-22T23:29:55.128 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading osd daemons", 2024-04-22T23:29:55.128 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:29:55.128 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:29:55.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.131+0000 7f93cc29b700 1 -- 172.21.15.67:0/794710365 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f93ac06ecc0 msgr2=0x7f93ac071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:55.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.131+0000 7f93cc29b700 1 --2- 172.21.15.67:0/794710365 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f93ac06ecc0 0x7f93ac071180 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f93bc005950 tx=0x7f93bc00a300 comp rx=0 tx=0).stop 2024-04-22T23:29:55.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.131+0000 7f93cc29b700 1 -- 172.21.15.67:0/794710365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93c4100750 msgr2=0x7f93c410b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:55.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.131+0000 7f93cc29b700 1 --2- 172.21.15.67:0/794710365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93c4100750 0x7f93c410b050 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f93b80092e0 tx=0x7f93b800bd70 comp rx=0 tx=0).stop 2024-04-22T23:29:55.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.131+0000 7f93cc29b700 1 -- 172.21.15.67:0/794710365 shutdown_connections 2024-04-22T23:29:55.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.131+0000 7f93cc29b700 1 --2- 172.21.15.67:0/794710365 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f93ac06ecc0 0x7f93ac071180 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:55.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.131+0000 7f93cc29b700 1 --2- 172.21.15.67:0/794710365 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93c40ffda0 0x7f93c410ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:55.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.131+0000 7f93cc29b700 1 --2- 172.21.15.67:0/794710365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93c4100750 0x7f93c410b050 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:55.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.131+0000 7f93cc29b700 1 -- 172.21.15.67:0/794710365 >> 172.21.15.67:0/794710365 conn(0x7f93c40fb650 msgr2=0x7f93c40fcce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:29:55.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.132+0000 7f93cc29b700 1 -- 172.21.15.67:0/794710365 shutdown_connections 2024-04-22T23:29:55.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.132+0000 7f93cc29b700 1 -- 172.21.15.67:0/794710365 wait complete. 2024-04-22T23:29:55.152 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:54 smithi134 ceph-mon[85630]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:29:55.152 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:54 smithi134 ceph-mon[85630]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:29:55.153 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:54 smithi134 ceph-mon[85630]: osdmap e56: 6 total, 5 up, 6 in 2024-04-22T23:29:55.153 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:54 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1700413877' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:29:55.153 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:54 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3137429206' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:29:55.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.239+0000 7f3ae3f77700 1 -- 172.21.15.67:0/2741029800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3adc101060 msgr2=0x7f3adc1047b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:55.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.239+0000 7f3ae3f77700 1 --2- 172.21.15.67:0/2741029800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3adc101060 0x7f3adc1047b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3ad0009910 tx=0x7f3ad0009c20 comp rx=0 tx=0).stop 2024-04-22T23:29:55.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.239+0000 7f3ae3f77700 1 -- 172.21.15.67:0/2741029800 shutdown_connections 2024-04-22T23:29:55.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.239+0000 7f3ae3f77700 1 --2- 172.21.15.67:0/2741029800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3adc101060 0x7f3adc1047b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:55.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.239+0000 7f3ae3f77700 1 --2- 172.21.15.67:0/2741029800 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3adc1006b0 0x7f3adc100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:55.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.239+0000 7f3ae3f77700 1 -- 172.21.15.67:0/2741029800 >> 172.21.15.67:0/2741029800 conn(0x7f3adc074e30 msgr2=0x7f3adc075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:29:55.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.239+0000 7f3ae3f77700 1 -- 172.21.15.67:0/2741029800 shutdown_connections 2024-04-22T23:29:55.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.239+0000 7f3ae3f77700 1 -- 172.21.15.67:0/2741029800 wait complete. 2024-04-22T23:29:55.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.240+0000 7f3ae3f77700 1 Processor -- start 2024-04-22T23:29:55.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.240+0000 7f3ae3f77700 1 -- start start 2024-04-22T23:29:55.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.240+0000 7f3ae3f77700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3adc1006b0 0x7f3adc196970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:55.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.240+0000 7f3ae3f77700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3adc101060 0x7f3adc196eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:55.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.240+0000 7f3ae3f77700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3adc197540 con 0x7f3adc1006b0 2024-04-22T23:29:55.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.240+0000 7f3ae3f77700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3adc19b350 con 0x7f3adc101060 2024-04-22T23:29:55.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.240+0000 7f3ae1d13700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3adc1006b0 0x7f3adc196970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:55.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.240+0000 7f3ae1d13700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3adc1006b0 0x7f3adc196970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58290/0 (socket says 172.21.15.67:58290) 2024-04-22T23:29:55.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.240+0000 7f3ae1d13700 1 -- 172.21.15.67:0/3525213717 learned_addr learned my addr 172.21.15.67:0/3525213717 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:29:55.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.240+0000 7f3ae1512700 1 --2- 172.21.15.67:0/3525213717 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3adc101060 0x7f3adc196eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:55.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.240+0000 7f3ae1d13700 1 -- 172.21.15.67:0/3525213717 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3adc101060 msgr2=0x7f3adc196eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:55.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.240+0000 7f3ae1d13700 1 --2- 172.21.15.67:0/3525213717 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3adc101060 0x7f3adc196eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:55.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.240+0000 7f3ae1d13700 1 -- 172.21.15.67:0/3525213717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad00095f0 con 0x7f3adc1006b0 2024-04-22T23:29:55.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.240+0000 7f3ae1512700 1 --2- 172.21.15.67:0/3525213717 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3adc101060 0x7f3adc196eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:29:55.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.241+0000 7f3ae1d13700 1 --2- 172.21.15.67:0/3525213717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3adc1006b0 0x7f3adc196970 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3acc00ba50 tx=0x7f3acc00be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:29:55.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.241+0000 7f3adaffd700 1 -- 172.21.15.67:0/3525213717 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3acc00ca00 con 0x7f3adc1006b0 2024-04-22T23:29:55.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.241+0000 7f3adaffd700 1 -- 172.21.15.67:0/3525213717 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3acc00cb60 con 0x7f3adc1006b0 2024-04-22T23:29:55.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.241+0000 7f3adaffd700 1 -- 172.21.15.67:0/3525213717 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3acc0075a0 con 0x7f3adc1006b0 2024-04-22T23:29:55.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.241+0000 7f3ae3f77700 1 -- 172.21.15.67:0/3525213717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3adc19b630 con 0x7f3adc1006b0 2024-04-22T23:29:55.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.241+0000 7f3ae3f77700 1 -- 172.21.15.67:0/3525213717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3adc19bb50 con 0x7f3adc1006b0 2024-04-22T23:29:55.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.242+0000 7f3ae3f77700 1 -- 172.21.15.67:0/3525213717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3adc109150 con 0x7f3adc1006b0 2024-04-22T23:29:55.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.242+0000 7f3adaffd700 1 -- 172.21.15.67:0/3525213717 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3acc00ccd0 con 0x7f3adc1006b0 2024-04-22T23:29:55.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.243+0000 7f3adaffd700 1 --2- 172.21.15.67:0/3525213717 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3ac406ecc0 0x7f3ac4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:29:55.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.243+0000 7f3adaffd700 1 -- 172.21.15.67:0/3525213717 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6414+0+0 (secure 0 0 0) 0x7f3acc006e40 con 0x7f3adc1006b0 2024-04-22T23:29:55.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.243+0000 7f3ae1512700 1 --2- 172.21.15.67:0/3525213717 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3ac406ecc0 0x7f3ac4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:29:55.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.243+0000 7f3ae1512700 1 --2- 172.21.15.67:0/3525213717 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3ac406ecc0 0x7f3ac4071180 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3adc1978e0 tx=0x7f3ad001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:29:55.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.248+0000 7f3adaffd700 1 -- 172.21.15.67:0/3525213717 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f3acc05af80 con 0x7f3adc1006b0 2024-04-22T23:29:55.368 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:55 smithi067 podman[129541]: 2024-04-22 23:29:55.285485898 +0000 UTC m=+0.544129049 container remove 0d24dbb08d17be9b6935654e19eec0d752f5dea4bf3a8c2cbba82ae877ffc96a (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-22T23:29:55.368 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:55 smithi067 systemd[1]: ceph-d46c346e-00fd-11ef-bc93-c7b262605968@osd.1.service: Deactivated successfully. 2024-04-22T23:29:55.368 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:55 smithi067 systemd[1]: Stopped Ceph osd.1 for d46c346e-00fd-11ef-bc93-c7b262605968. 2024-04-22T23:29:55.368 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:55 smithi067 systemd[1]: ceph-d46c346e-00fd-11ef-bc93-c7b262605968@osd.1.service: Consumed 1min 2.701s CPU time. 2024-04-22T23:29:55.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.573+0000 7f3ae3f77700 1 -- 172.21.15.67:0/3525213717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f3adc04f2e0 con 0x7f3adc1006b0 2024-04-22T23:29:55.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.574+0000 7f3adaffd700 1 -- 172.21.15.67:0/3525213717 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+290 (secure 0 0 0) 0x7f3acc05ab10 con 0x7f3adc1006b0 2024-04-22T23:29:55.572 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data; 1 osds down 2024-04-22T23:29:55.572 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:29:55.572 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:29:55.572 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-22T23:29:55.572 INFO:teuthology.orchestra.run.smithi067.stdout: osd.1 (root=default,host=smithi067) is down 2024-04-22T23:29:55.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.577+0000 7f3ae3f77700 1 -- 172.21.15.67:0/3525213717 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3ac406ecc0 msgr2=0x7f3ac4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:55.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.577+0000 7f3ae3f77700 1 --2- 172.21.15.67:0/3525213717 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3ac406ecc0 0x7f3ac4071180 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3adc1978e0 tx=0x7f3ad001a040 comp rx=0 tx=0).stop 2024-04-22T23:29:55.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.577+0000 7f3ae3f77700 1 -- 172.21.15.67:0/3525213717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3adc1006b0 msgr2=0x7f3adc196970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:29:55.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.577+0000 7f3ae3f77700 1 --2- 172.21.15.67:0/3525213717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3adc1006b0 0x7f3adc196970 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3acc00ba50 tx=0x7f3acc00be10 comp rx=0 tx=0).stop 2024-04-22T23:29:55.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.577+0000 7f3ae3f77700 1 -- 172.21.15.67:0/3525213717 shutdown_connections 2024-04-22T23:29:55.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.577+0000 7f3ae3f77700 1 --2- 172.21.15.67:0/3525213717 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3ac406ecc0 0x7f3ac4071180 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:55.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.577+0000 7f3ae3f77700 1 --2- 172.21.15.67:0/3525213717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3adc1006b0 0x7f3adc196970 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:55.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.577+0000 7f3ae3f77700 1 --2- 172.21.15.67:0/3525213717 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3adc101060 0x7f3adc196eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:29:55.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.577+0000 7f3ae3f77700 1 -- 172.21.15.67:0/3525213717 >> 172.21.15.67:0/3525213717 conn(0x7f3adc074e30 msgr2=0x7f3adc103cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:29:55.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.577+0000 7f3ae3f77700 1 -- 172.21.15.67:0/3525213717 shutdown_connections 2024-04-22T23:29:55.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:29:55.577+0000 7f3ae3f77700 1 -- 172.21.15.67:0/3525213717 wait complete. 2024-04-22T23:29:55.769 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:55 smithi067 systemd[1]: Starting Ceph osd.1 for d46c346e-00fd-11ef-bc93-c7b262605968... 2024-04-22T23:29:56.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:55 smithi067 ceph-mon[112438]: from='client.34228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:29:56.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:55 smithi067 ceph-mon[112438]: pgmap v69: 65 pgs: 12 stale+active+clean, 53 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:29:56.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:55 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3525213717' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:29:56.042 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:55 smithi067 podman[129695]: 2024-04-22 23:29:55.675325399 +0000 UTC m=+0.018932408 image pull a0434699c1d76af4eea1fc95ff0a22ea73d1e04e609464fbf658d6f39bc7f2fc quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c 2024-04-22T23:29:56.042 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:55 smithi067 podman[129695]: 2024-04-22 23:29:55.796043447 +0000 UTC m=+0.139650455 container create 4afbdd2731862145fc0bdf08728af98dfd7aa877d340f942336e95960d713763 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1-activate, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T23:29:56.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:55 smithi134 ceph-mon[85630]: from='client.34228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:29:56.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:55 smithi134 ceph-mon[85630]: pgmap v69: 65 pgs: 12 stale+active+clean, 53 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:29:56.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:55 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3525213717' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:29:56.542 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:56 smithi067 podman[129695]: 2024-04-22 23:29:56.074113584 +0000 UTC m=+0.417720601 container init 4afbdd2731862145fc0bdf08728af98dfd7aa877d340f942336e95960d713763 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1-activate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, io.buildah.version=1.33.5) 2024-04-22T23:29:56.542 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:56 smithi067 podman[129695]: 2024-04-22 23:29:56.078004026 +0000 UTC m=+0.421611044 container start 4afbdd2731862145fc0bdf08728af98dfd7aa877d340f942336e95960d713763 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-22T23:29:56.542 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:56 smithi067 podman[129695]: 2024-04-22 23:29:56.148695896 +0000 UTC m=+0.492302923 container attach 4afbdd2731862145fc0bdf08728af98dfd7aa877d340f942336e95960d713763 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1-activate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True) 2024-04-22T23:29:56.891 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:56 smithi067 ceph-mon[112438]: from='client.34240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:29:56.891 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.2", "id": [0, 2]}]: dispatch 2024-04-22T23:29:56.892 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [0, 1]}]: dispatch 2024-04-22T23:29:56.892 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [3, 1]}]: dispatch 2024-04-22T23:29:56.892 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [3, 1]}]: dispatch 2024-04-22T23:29:56.892 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [0, 1]}]: dispatch 2024-04-22T23:29:56.892 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [3, 1]}]: dispatch 2024-04-22T23:29:56.892 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 1]}]: dispatch 2024-04-22T23:29:56.892 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [4, 0]}]: dispatch 2024-04-22T23:29:56.892 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 1]}]: dispatch 2024-04-22T23:29:56.892 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 2]}]: dispatch 2024-04-22T23:29:56.892 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:56.892 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:29:56.892 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:56 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1-activate[129709]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T23:29:56.892 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:56 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1-activate[129709]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T23:29:56.893 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:56 smithi067 bash[129695]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T23:29:56.893 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:56 smithi067 bash[129695]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T23:29:56.893 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:56 smithi067 bash[129695]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-22T23:29:56.893 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:56 smithi067 bash[129695]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-22T23:29:56.893 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:56 smithi067 bash[129695]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-22T23:29:56.893 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:56 smithi067 bash[129695]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-22T23:29:56.893 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:56 smithi067 bash[129695]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-22T23:29:56.893 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:56 smithi067 bash[129695]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-22T23:29:56.893 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:56 smithi067 bash[129695]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-22T23:29:56.893 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:56 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1-activate[129709]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-22T23:29:56.893 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:56 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1-activate[129709]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-22T23:29:56.893 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:56 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1-activate[129709]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-22T23:29:56.893 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:56 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1-activate[129709]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-22T23:29:56.893 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:56 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1-activate[129709]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-22T23:29:56.894 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:56 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1-activate[129709]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-22T23:29:56.894 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:56 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1-activate[129709]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-22T23:29:56.894 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:56 smithi067 podman[129695]: 2024-04-22 23:29:56.605176477 +0000 UTC m=+0.948783480 container died 4afbdd2731862145fc0bdf08728af98dfd7aa877d340f942336e95960d713763 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1-activate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T23:29:57.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:56 smithi134 ceph-mon[85630]: from='client.34240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:29:57.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.2", "id": [0, 2]}]: dispatch 2024-04-22T23:29:57.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [0, 1]}]: dispatch 2024-04-22T23:29:57.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [3, 1]}]: dispatch 2024-04-22T23:29:57.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [3, 1]}]: dispatch 2024-04-22T23:29:57.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [0, 1]}]: dispatch 2024-04-22T23:29:57.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [3, 1]}]: dispatch 2024-04-22T23:29:57.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 1]}]: dispatch 2024-04-22T23:29:57.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [4, 0]}]: dispatch 2024-04-22T23:29:57.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 1]}]: dispatch 2024-04-22T23:29:57.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 2]}]: dispatch 2024-04-22T23:29:57.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:57.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:29:57.542 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:57 smithi067 podman[129854]: 2024-04-22 23:29:57.236638207 +0000 UTC m=+0.623453367 container remove 4afbdd2731862145fc0bdf08728af98dfd7aa877d340f942336e95960d713763 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-22T23:29:57.542 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:57 smithi067 podman[129918]: 2024-04-22 23:29:57.360851979 +0000 UTC m=+0.018878572 image pull a0434699c1d76af4eea1fc95ff0a22ea73d1e04e609464fbf658d6f39bc7f2fc quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c 2024-04-22T23:29:57.542 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:57 smithi067 podman[129918]: 2024-04-22 23:29:57.479257412 +0000 UTC m=+0.137283991 container create b003cac17eac384fe520e1d5c967a6d6de8d1707adebf2ea974c9ac43c834373 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-22T23:29:58.042 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:57 smithi067 podman[129918]: 2024-04-22 23:29:57.724085315 +0000 UTC m=+0.382111896 container init b003cac17eac384fe520e1d5c967a6d6de8d1707adebf2ea974c9ac43c834373 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-22T23:29:58.042 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:57 smithi067 podman[129918]: 2024-04-22 23:29:57.727436763 +0000 UTC m=+0.385463342 container start b003cac17eac384fe520e1d5c967a6d6de8d1707adebf2ea974c9ac43c834373 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True) 2024-04-22T23:29:58.042 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:57 smithi067 ceph-osd[129937]: -- 172.21.15.67:0/4228375876 <== mon.0 v2:172.21.15.67:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564d679ab680 con 0x564d67974c00 2024-04-22T23:29:58.043 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:57 smithi067 bash[129918]: b003cac17eac384fe520e1d5c967a6d6de8d1707adebf2ea974c9ac43c834373 2024-04-22T23:29:58.043 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:57 smithi067 systemd[1]: Started Ceph osd.1 for d46c346e-00fd-11ef-bc93-c7b262605968. 2024-04-22T23:29:58.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:57 smithi067 ceph-mon[112438]: pgmap v70: 65 pgs: 12 stale+active+clean, 53 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T23:29:58.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:57 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.2", "id": [0, 2]}]': finished 2024-04-22T23:29:58.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:57 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [0, 1]}]': finished 2024-04-22T23:29:58.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:57 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [3, 1]}]': finished 2024-04-22T23:29:58.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:57 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [3, 1]}]': finished 2024-04-22T23:29:58.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:57 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [0, 1]}]': finished 2024-04-22T23:29:58.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:57 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [3, 1]}]': finished 2024-04-22T23:29:58.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:57 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 1]}]': finished 2024-04-22T23:29:58.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:57 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [4, 0]}]': finished 2024-04-22T23:29:58.044 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:57 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 1]}]': finished 2024-04-22T23:29:58.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:57 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 2]}]': finished 2024-04-22T23:29:58.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:57 smithi067 ceph-mon[112438]: osdmap e57: 6 total, 5 up, 6 in 2024-04-22T23:29:58.045 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:57 smithi067 ceph-mon[112438]: osdmap e58: 6 total, 5 up, 6 in 2024-04-22T23:29:58.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:57 smithi134 ceph-mon[85630]: pgmap v70: 65 pgs: 12 stale+active+clean, 53 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-22T23:29:58.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:57 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.2", "id": [0, 2]}]': finished 2024-04-22T23:29:58.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:57 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [0, 1]}]': finished 2024-04-22T23:29:58.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:57 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [3, 1]}]': finished 2024-04-22T23:29:58.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:57 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [3, 1]}]': finished 2024-04-22T23:29:58.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:57 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [0, 1]}]': finished 2024-04-22T23:29:58.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:57 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [3, 1]}]': finished 2024-04-22T23:29:58.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:57 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 1]}]': finished 2024-04-22T23:29:58.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:57 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [4, 0]}]': finished 2024-04-22T23:29:58.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:57 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 1]}]': finished 2024-04-22T23:29:58.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:57 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 2]}]': finished 2024-04-22T23:29:58.243 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:57 smithi134 ceph-mon[85630]: osdmap e57: 6 total, 5 up, 6 in 2024-04-22T23:29:58.243 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:57 smithi134 ceph-mon[85630]: osdmap e58: 6 total, 5 up, 6 in 2024-04-22T23:29:58.541 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:29:58 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1[129933]: 2024-04-22T23:29:58.271+0000 7fd3930417c0 -1 Falling back to public interface 2024-04-22T23:29:59.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:59.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:59.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:29:59.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:58 smithi067 ceph-mon[112438]: pgmap v73: 65 pgs: 1 unknown, 9 peering, 15 active+undersized+degraded, 16 active+undersized, 24 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 54/363 objects degraded (14.876%) 2024-04-22T23:29:59.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:58 smithi067 ceph-mon[112438]: osdmap e59: 6 total, 5 up, 6 in 2024-04-22T23:29:59.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:59.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:29:59.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:29:59.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:58 smithi134 ceph-mon[85630]: pgmap v73: 65 pgs: 1 unknown, 9 peering, 15 active+undersized+degraded, 16 active+undersized, 24 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 54/363 objects degraded (14.876%) 2024-04-22T23:29:59.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:58 smithi134 ceph-mon[85630]: osdmap e59: 6 total, 5 up, 6 in 2024-04-22T23:30:00.207 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:59 smithi134 ceph-mon[85630]: Health check failed: Degraded data redundancy: 54/363 objects degraded (14.876%), 15 pgs degraded (PG_DEGRADED) 2024-04-22T23:30:00.207 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:29:59 smithi134 ceph-mon[85630]: osdmap e60: 6 total, 5 up, 6 in 2024-04-22T23:30:00.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:59 smithi067 ceph-mon[112438]: Health check failed: Degraded data redundancy: 54/363 objects degraded (14.876%), 15 pgs degraded (PG_DEGRADED) 2024-04-22T23:30:00.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:29:59 smithi067 ceph-mon[112438]: osdmap e60: 6 total, 5 up, 6 in 2024-04-22T23:30:01.188 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:00 smithi067 ceph-mon[112438]: pgmap v76: 65 pgs: 1 unknown, 9 peering, 15 active+undersized+degraded, 16 active+undersized, 24 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 54/363 objects degraded (14.876%) 2024-04-22T23:30:01.188 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:00 smithi067 ceph-mon[112438]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data; 1 osds down; Degraded data redundancy: 54/363 objects degraded (14.876%), 15 pgs degraded 2024-04-22T23:30:01.189 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:00 smithi067 ceph-mon[112438]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:30:01.189 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:00 smithi067 ceph-mon[112438]: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:30:01.189 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:00 smithi067 ceph-mon[112438]: [WRN] OSD_DOWN: 1 osds down 2024-04-22T23:30:01.189 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:00 smithi067 ceph-mon[112438]: osd.1 (root=default,host=smithi067) is down 2024-04-22T23:30:01.189 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:00 smithi067 ceph-mon[112438]: [WRN] PG_DEGRADED: Degraded data redundancy: 54/363 objects degraded (14.876%), 15 pgs degraded 2024-04-22T23:30:01.189 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:00 smithi067 ceph-mon[112438]: pg 1.0 is active+undersized+degraded, acting [3,0] 2024-04-22T23:30:01.189 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:00 smithi067 ceph-mon[112438]: pg 2.0 is active+undersized+degraded, acting [3,0] 2024-04-22T23:30:01.189 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:00 smithi067 ceph-mon[112438]: pg 2.1 is active+undersized+degraded, acting [2,0] 2024-04-22T23:30:01.189 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:00 smithi067 ceph-mon[112438]: pg 2.3 is active+undersized+degraded, acting [5,2] 2024-04-22T23:30:01.190 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:00 smithi067 ceph-mon[112438]: pg 2.4 is active+undersized+degraded, acting [0,4] 2024-04-22T23:30:01.190 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:00 smithi067 ceph-mon[112438]: pg 2.6 is active+undersized+degraded, acting [3,4] 2024-04-22T23:30:01.190 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:00 smithi067 ceph-mon[112438]: pg 2.9 is active+undersized+degraded, acting [4,0] 2024-04-22T23:30:01.190 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:00 smithi067 ceph-mon[112438]: pg 2.a is active+undersized+degraded, acting [4,3] 2024-04-22T23:30:01.190 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:00 smithi067 ceph-mon[112438]: pg 2.d is active+undersized+degraded, acting [3,2] 2024-04-22T23:30:01.190 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:00 smithi067 ceph-mon[112438]: pg 2.10 is active+undersized+degraded, acting [2,0] 2024-04-22T23:30:01.190 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:00 smithi067 ceph-mon[112438]: pg 2.11 is active+undersized+degraded, acting [4,3] 2024-04-22T23:30:01.190 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:00 smithi067 ceph-mon[112438]: pg 2.12 is active+undersized+degraded, acting [3,0] 2024-04-22T23:30:01.190 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:00 smithi067 ceph-mon[112438]: pg 2.15 is active+undersized+degraded, acting [3,0] 2024-04-22T23:30:01.190 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:00 smithi067 ceph-mon[112438]: pg 2.17 is active+undersized+degraded, acting [5,2] 2024-04-22T23:30:01.191 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:00 smithi067 ceph-mon[112438]: pg 2.1b is active+undersized+degraded, acting [0,5] 2024-04-22T23:30:01.191 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:00 smithi067 ceph-mon[112438]: osdmap e61: 6 total, 5 up, 6 in 2024-04-22T23:30:01.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:00 smithi134 ceph-mon[85630]: pgmap v76: 65 pgs: 1 unknown, 9 peering, 15 active+undersized+degraded, 16 active+undersized, 24 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 54/363 objects degraded (14.876%) 2024-04-22T23:30:01.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:00 smithi134 ceph-mon[85630]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data; 1 osds down; Degraded data redundancy: 54/363 objects degraded (14.876%), 15 pgs degraded 2024-04-22T23:30:01.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:00 smithi134 ceph-mon[85630]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:30:01.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:00 smithi134 ceph-mon[85630]: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:30:01.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:00 smithi134 ceph-mon[85630]: [WRN] OSD_DOWN: 1 osds down 2024-04-22T23:30:01.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:00 smithi134 ceph-mon[85630]: osd.1 (root=default,host=smithi067) is down 2024-04-22T23:30:01.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:00 smithi134 ceph-mon[85630]: [WRN] PG_DEGRADED: Degraded data redundancy: 54/363 objects degraded (14.876%), 15 pgs degraded 2024-04-22T23:30:01.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:00 smithi134 ceph-mon[85630]: pg 1.0 is active+undersized+degraded, acting [3,0] 2024-04-22T23:30:01.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:00 smithi134 ceph-mon[85630]: pg 2.0 is active+undersized+degraded, acting [3,0] 2024-04-22T23:30:01.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:00 smithi134 ceph-mon[85630]: pg 2.1 is active+undersized+degraded, acting [2,0] 2024-04-22T23:30:01.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:00 smithi134 ceph-mon[85630]: pg 2.3 is active+undersized+degraded, acting [5,2] 2024-04-22T23:30:01.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:00 smithi134 ceph-mon[85630]: pg 2.4 is active+undersized+degraded, acting [0,4] 2024-04-22T23:30:01.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:00 smithi134 ceph-mon[85630]: pg 2.6 is active+undersized+degraded, acting [3,4] 2024-04-22T23:30:01.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:00 smithi134 ceph-mon[85630]: pg 2.9 is active+undersized+degraded, acting [4,0] 2024-04-22T23:30:01.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:00 smithi134 ceph-mon[85630]: pg 2.a is active+undersized+degraded, acting [4,3] 2024-04-22T23:30:01.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:00 smithi134 ceph-mon[85630]: pg 2.d is active+undersized+degraded, acting [3,2] 2024-04-22T23:30:01.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:00 smithi134 ceph-mon[85630]: pg 2.10 is active+undersized+degraded, acting [2,0] 2024-04-22T23:30:01.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:00 smithi134 ceph-mon[85630]: pg 2.11 is active+undersized+degraded, acting [4,3] 2024-04-22T23:30:01.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:00 smithi134 ceph-mon[85630]: pg 2.12 is active+undersized+degraded, acting [3,0] 2024-04-22T23:30:01.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:00 smithi134 ceph-mon[85630]: pg 2.15 is active+undersized+degraded, acting [3,0] 2024-04-22T23:30:01.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:00 smithi134 ceph-mon[85630]: pg 2.17 is active+undersized+degraded, acting [5,2] 2024-04-22T23:30:01.243 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:00 smithi134 ceph-mon[85630]: pg 2.1b is active+undersized+degraded, acting [0,5] 2024-04-22T23:30:01.243 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:00 smithi134 ceph-mon[85630]: osdmap e61: 6 total, 5 up, 6 in 2024-04-22T23:30:03.034 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:30:02 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1[129933]: 2024-04-22T23:30:02.781+0000 7fd3930417c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-04-22T23:30:03.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:03 smithi067 ceph-mon[112438]: pgmap v78: 65 pgs: 8 active+clean+remapped, 2 peering, 15 active+undersized+degraded, 16 active+undersized, 24 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 2 op/s; 54/363 objects degraded (14.876%); 18/363 objects misplaced (4.959%); 2.6 MiB/s, 51 objects/s recovering 2024-04-22T23:30:03.293 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:30:03 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1[129933]: 2024-04-22T23:30:03.159+0000 7fd3930417c0 -1 osd.1 54 log_to_monitors true 2024-04-22T23:30:03.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:03 smithi134 ceph-mon[85630]: pgmap v78: 65 pgs: 8 active+clean+remapped, 2 peering, 15 active+undersized+degraded, 16 active+undersized, 24 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 2 op/s; 54/363 objects degraded (14.876%); 18/363 objects misplaced (4.959%); 2.6 MiB/s, 51 objects/s recovering 2024-04-22T23:30:04.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:04 smithi067 ceph-mon[112438]: from='osd.1 [v2:172.21.15.67:6810/1745058179,v1:172.21.15.67:6811/1745058179]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T23:30:04.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:04 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:04.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:04 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:04.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:04 smithi134 ceph-mon[85630]: from='osd.1 [v2:172.21.15.67:6810/1745058179,v1:172.21.15.67:6811/1745058179]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T23:30:04.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:04 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:04.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:04 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:05.291 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 22 23:30:04 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1[129933]: 2024-04-22T23:30:04.987+0000 7fd380491700 -1 osd.1 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T23:30:05.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:05 smithi067 ceph-mon[112438]: pgmap v79: 65 pgs: 10 active+clean+remapped, 15 active+undersized+degraded, 16 active+undersized, 24 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 54/363 objects degraded (14.876%); 18/363 objects misplaced (4.959%); 2.0 MiB/s, 75 objects/s recovering 2024-04-22T23:30:05.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:05 smithi067 ceph-mon[112438]: from='osd.1 [v2:172.21.15.67:6810/1745058179,v1:172.21.15.67:6811/1745058179]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T23:30:05.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:05 smithi067 ceph-mon[112438]: osdmap e62: 6 total, 5 up, 6 in 2024-04-22T23:30:05.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:05 smithi067 ceph-mon[112438]: from='osd.1 [v2:172.21.15.67:6810/1745058179,v1:172.21.15.67:6811/1745058179]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-04-22T23:30:05.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:05 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:05.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:05 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:05.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:05 smithi067 ceph-mon[112438]: from='osd.1 [v2:172.21.15.67:6810/1745058179,v1:172.21.15.67:6811/1745058179]' entity='osd.1' 2024-04-22T23:30:05.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:05 smithi134 ceph-mon[85630]: pgmap v79: 65 pgs: 10 active+clean+remapped, 15 active+undersized+degraded, 16 active+undersized, 24 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 54/363 objects degraded (14.876%); 18/363 objects misplaced (4.959%); 2.0 MiB/s, 75 objects/s recovering 2024-04-22T23:30:05.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:05 smithi134 ceph-mon[85630]: from='osd.1 [v2:172.21.15.67:6810/1745058179,v1:172.21.15.67:6811/1745058179]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T23:30:05.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:05 smithi134 ceph-mon[85630]: osdmap e62: 6 total, 5 up, 6 in 2024-04-22T23:30:05.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:05 smithi134 ceph-mon[85630]: from='osd.1 [v2:172.21.15.67:6810/1745058179,v1:172.21.15.67:6811/1745058179]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-04-22T23:30:05.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:05 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:05.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:05 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:05.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:05 smithi134 ceph-mon[85630]: from='osd.1 [v2:172.21.15.67:6810/1745058179,v1:172.21.15.67:6811/1745058179]' entity='osd.1' 2024-04-22T23:30:06.411 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:06 smithi067 ceph-mon[112438]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-22T23:30:06.411 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:06 smithi067 ceph-mon[112438]: osd.1 [v2:172.21.15.67:6810/1745058179,v1:172.21.15.67:6811/1745058179] boot 2024-04-22T23:30:06.411 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:06 smithi067 ceph-mon[112438]: osdmap e63: 6 total, 6 up, 6 in 2024-04-22T23:30:06.411 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:06 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T23:30:06.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:06 smithi134 ceph-mon[85630]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-22T23:30:06.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:06 smithi134 ceph-mon[85630]: osd.1 [v2:172.21.15.67:6810/1745058179,v1:172.21.15.67:6811/1745058179] boot 2024-04-22T23:30:06.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:06 smithi134 ceph-mon[85630]: osdmap e63: 6 total, 6 up, 6 in 2024-04-22T23:30:06.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:06 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T23:30:07.407 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:07 smithi067 ceph-mon[112438]: pgmap v82: 65 pgs: 10 active+clean+remapped, 15 active+undersized+degraded, 16 active+undersized, 24 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 54/363 objects degraded (14.876%); 18/363 objects misplaced (4.959%); 2.0 MiB/s, 75 objects/s recovering 2024-04-22T23:30:07.407 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:07 smithi067 ceph-mon[112438]: osdmap e64: 6 total, 6 up, 6 in 2024-04-22T23:30:07.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:07 smithi134 ceph-mon[85630]: pgmap v82: 65 pgs: 10 active+clean+remapped, 15 active+undersized+degraded, 16 active+undersized, 24 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 54/363 objects degraded (14.876%); 18/363 objects misplaced (4.959%); 2.0 MiB/s, 75 objects/s recovering 2024-04-22T23:30:07.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:07 smithi134 ceph-mon[85630]: osdmap e64: 6 total, 6 up, 6 in 2024-04-22T23:30:09.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:08 smithi134 ceph-mon[85630]: pgmap v84: 65 pgs: 9 remapped+peering, 19 peering, 7 active+undersized+degraded, 5 active+undersized, 25 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 3 op/s; 24/363 objects degraded (6.612%); 0 B/s, 36 objects/s recovering 2024-04-22T23:30:09.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:08 smithi134 ceph-mon[85630]: osdmap e65: 6 total, 6 up, 6 in 2024-04-22T23:30:09.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:08 smithi134 ceph-mon[85630]: Health check update: Degraded data redundancy: 24/363 objects degraded (6.612%), 7 pgs degraded (PG_DEGRADED) 2024-04-22T23:30:09.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:08 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:09.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:08 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:09.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:08 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:30:09.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:08 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:30:09.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:08 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:09.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:08 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:30:09.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:08 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:30:09.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:08 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:30:09.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:08 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:30:09.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:08 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-22T23:30:09.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:08 smithi134 ceph-mon[85630]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-22T23:30:09.242 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:08 smithi134 ceph-mon[85630]: Upgrade: unsafe to stop osd(s) at this time (17 PGs are or would become offline) 2024-04-22T23:30:09.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:08 smithi067 ceph-mon[112438]: pgmap v84: 65 pgs: 9 remapped+peering, 19 peering, 7 active+undersized+degraded, 5 active+undersized, 25 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 3 op/s; 24/363 objects degraded (6.612%); 0 B/s, 36 objects/s recovering 2024-04-22T23:30:09.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:08 smithi067 ceph-mon[112438]: osdmap e65: 6 total, 6 up, 6 in 2024-04-22T23:30:09.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:08 smithi067 ceph-mon[112438]: Health check update: Degraded data redundancy: 24/363 objects degraded (6.612%), 7 pgs degraded (PG_DEGRADED) 2024-04-22T23:30:09.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:08 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:09.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:08 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:09.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:08 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:30:09.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:08 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:30:09.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:08 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:09.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:08 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:30:09.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:08 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:30:09.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:08 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:30:09.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:08 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:30:09.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:08 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-22T23:30:09.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:08 smithi067 ceph-mon[112438]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-22T23:30:09.293 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:08 smithi067 ceph-mon[112438]: Upgrade: unsafe to stop osd(s) at this time (17 PGs are or would become offline) 2024-04-22T23:30:10.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:09 smithi067 ceph-mon[112438]: osdmap e66: 6 total, 6 up, 6 in 2024-04-22T23:30:10.314 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:09 smithi134 ceph-mon[85630]: osdmap e66: 6 total, 6 up, 6 in 2024-04-22T23:30:11.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:10 smithi067 ceph-mon[112438]: pgmap v87: 65 pgs: 8 remapped+peering, 20 peering, 37 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s; 3.0 MiB/s, 1 objects/s recovering 2024-04-22T23:30:11.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:10 smithi067 ceph-mon[112438]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 24/363 objects degraded (6.612%), 7 pgs degraded) 2024-04-22T23:30:11.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:10 smithi067 ceph-mon[112438]: osdmap e67: 6 total, 6 up, 6 in 2024-04-22T23:30:11.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:10 smithi134 ceph-mon[85630]: pgmap v87: 65 pgs: 8 remapped+peering, 20 peering, 37 active+clean; 310 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s; 3.0 MiB/s, 1 objects/s recovering 2024-04-22T23:30:11.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:10 smithi134 ceph-mon[85630]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 24/363 objects degraded (6.612%), 7 pgs degraded) 2024-04-22T23:30:11.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:10 smithi134 ceph-mon[85630]: osdmap e67: 6 total, 6 up, 6 in 2024-04-22T23:30:12.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:12 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:12.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:12 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:30:12.392 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:12 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:12.393 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:12 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:30:13.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:13 smithi067 ceph-mon[112438]: pgmap v89: 65 pgs: 1 remapped+peering, 2 peering, 62 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s; 8.0 MiB/s, 49 objects/s recovering 2024-04-22T23:30:13.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:13 smithi134 ceph-mon[85630]: pgmap v89: 65 pgs: 1 remapped+peering, 2 peering, 62 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s; 8.0 MiB/s, 49 objects/s recovering 2024-04-22T23:30:15.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:15 smithi067 ceph-mon[112438]: pgmap v90: 65 pgs: 1 peering, 64 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 3 op/s; 6.7 MiB/s, 77 objects/s recovering 2024-04-22T23:30:15.368 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:15 smithi134 ceph-mon[85630]: pgmap v90: 65 pgs: 1 peering, 64 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 3 op/s; 6.7 MiB/s, 77 objects/s recovering 2024-04-22T23:30:17.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:17 smithi067 ceph-mon[112438]: pgmap v91: 65 pgs: 65 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 3 op/s; 5.0 MiB/s, 58 objects/s recovering 2024-04-22T23:30:17.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:17 smithi134 ceph-mon[85630]: pgmap v91: 65 pgs: 65 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 3 op/s; 5.0 MiB/s, 58 objects/s recovering 2024-04-22T23:30:19.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:19 smithi067 ceph-mon[112438]: pgmap v92: 65 pgs: 65 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s; 4.4 MiB/s, 51 objects/s recovering 2024-04-22T23:30:19.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:19 smithi134 ceph-mon[85630]: pgmap v92: 65 pgs: 65 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s; 4.4 MiB/s, 51 objects/s recovering 2024-04-22T23:30:21.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:21 smithi134 ceph-mon[85630]: pgmap v93: 65 pgs: 65 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s; 2.8 MiB/s, 46 objects/s recovering 2024-04-22T23:30:21.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:21 smithi067 ceph-mon[112438]: pgmap v93: 65 pgs: 65 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s; 2.8 MiB/s, 46 objects/s recovering 2024-04-22T23:30:22.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:22 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:22.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:22 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:23.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:23 smithi134 ceph-mon[85630]: pgmap v94: 65 pgs: 65 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s; 2.3 MiB/s, 38 objects/s recovering 2024-04-22T23:30:23.542 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:23 smithi067 ceph-mon[112438]: pgmap v94: 65 pgs: 65 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s; 2.3 MiB/s, 38 objects/s recovering 2024-04-22T23:30:24.230 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:24 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-22T23:30:24.230 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:24 smithi067 ceph-mon[112438]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-22T23:30:24.230 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:24 smithi067 ceph-mon[112438]: Upgrade: osd.2 is safe to restart 2024-04-22T23:30:24.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:24 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-22T23:30:24.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:24 smithi134 ceph-mon[85630]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-22T23:30:24.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:24 smithi134 ceph-mon[85630]: Upgrade: osd.2 is safe to restart 2024-04-22T23:30:25.177 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:25 smithi067 ceph-mon[112438]: pgmap v95: 65 pgs: 65 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 0 B/s, 18 objects/s recovering 2024-04-22T23:30:25.178 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:25 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:25.178 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:25 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T23:30:25.178 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:25 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:30:25.475 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:25 smithi134 ceph-mon[85630]: pgmap v95: 65 pgs: 65 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 0 B/s, 18 objects/s recovering 2024-04-22T23:30:25.475 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:25 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:25.475 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:25 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T23:30:25.475 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:25 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:30:25.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.693+0000 7f8da04a8700 1 -- 172.21.15.67:0/2538575104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d980ffc20 msgr2=0x7f8d98100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:25.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.693+0000 7f8da04a8700 1 --2- 172.21.15.67:0/2538575104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d980ffc20 0x7f8d98100080 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c009910 tx=0x7f8d8c009c20 comp rx=0 tx=0).stop 2024-04-22T23:30:25.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.693+0000 7f8da04a8700 1 -- 172.21.15.67:0/2538575104 shutdown_connections 2024-04-22T23:30:25.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.693+0000 7f8da04a8700 1 --2- 172.21.15.67:0/2538575104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d980ffc20 0x7f8d98100080 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:25.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.693+0000 7f8da04a8700 1 --2- 172.21.15.67:0/2538575104 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8d98105bd0 0x7f8d98105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:25.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.693+0000 7f8da04a8700 1 -- 172.21.15.67:0/2538575104 >> 172.21.15.67:0/2538575104 conn(0x7f8d98075670 msgr2=0x7f8d98075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:30:25.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.693+0000 7f8da04a8700 1 -- 172.21.15.67:0/2538575104 shutdown_connections 2024-04-22T23:30:25.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.693+0000 7f8da04a8700 1 -- 172.21.15.67:0/2538575104 wait complete. 2024-04-22T23:30:25.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.694+0000 7f8da04a8700 1 Processor -- start 2024-04-22T23:30:25.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.694+0000 7f8da04a8700 1 -- start start 2024-04-22T23:30:25.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.694+0000 7f8da04a8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d980ffc20 0x7f8d98101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:30:25.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.694+0000 7f8da04a8700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8d98105bd0 0x7f8d98101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:30:25.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.694+0000 7f8da04a8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d98102350 con 0x7f8d980ffc20 2024-04-22T23:30:25.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.694+0000 7f8da04a8700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d981024c0 con 0x7f8d98105bd0 2024-04-22T23:30:25.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.694+0000 7f8d9e244700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d980ffc20 0x7f8d98101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:30:25.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.694+0000 7f8d9e244700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d980ffc20 0x7f8d98101840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58646/0 (socket says 172.21.15.67:58646) 2024-04-22T23:30:25.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.694+0000 7f8d9e244700 1 -- 172.21.15.67:0/3904461919 learned_addr learned my addr 172.21.15.67:0/3904461919 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:30:25.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.694+0000 7f8d9e244700 1 -- 172.21.15.67:0/3904461919 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8d98105bd0 msgr2=0x7f8d98101d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:25.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.694+0000 7f8d9da43700 1 --2- 172.21.15.67:0/3904461919 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8d98105bd0 0x7f8d98101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:30:25.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.695+0000 7f8d9e244700 1 --2- 172.21.15.67:0/3904461919 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8d98105bd0 0x7f8d98101d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:25.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.695+0000 7f8d9e244700 1 -- 172.21.15.67:0/3904461919 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d8c0095f0 con 0x7f8d980ffc20 2024-04-22T23:30:25.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.695+0000 7f8d9e244700 1 --2- 172.21.15.67:0/3904461919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d980ffc20 0x7f8d98101840 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8d9400d900 tx=0x7f8d9400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:30:25.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.695+0000 7f8d8b7fe700 1 -- 172.21.15.67:0/3904461919 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d940041f0 con 0x7f8d980ffc20 2024-04-22T23:30:25.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.695+0000 7f8da04a8700 1 -- 172.21.15.67:0/3904461919 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d981a2ad0 con 0x7f8d980ffc20 2024-04-22T23:30:25.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.695+0000 7f8d8b7fe700 1 -- 172.21.15.67:0/3904461919 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8d94004cf0 con 0x7f8d980ffc20 2024-04-22T23:30:25.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.695+0000 7f8d8b7fe700 1 -- 172.21.15.67:0/3904461919 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d9400b750 con 0x7f8d980ffc20 2024-04-22T23:30:25.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.695+0000 7f8da04a8700 1 -- 172.21.15.67:0/3904461919 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d981a2f40 con 0x7f8d980ffc20 2024-04-22T23:30:25.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.696+0000 7f8da04a8700 1 -- 172.21.15.67:0/3904461919 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d9804f2e0 con 0x7f8d980ffc20 2024-04-22T23:30:25.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.696+0000 7f8d8b7fe700 1 -- 172.21.15.67:0/3904461919 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8d94004350 con 0x7f8d980ffc20 2024-04-22T23:30:25.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.697+0000 7f8d8b7fe700 1 --2- 172.21.15.67:0/3904461919 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8d8406ecc0 0x7f8d84071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:30:25.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.697+0000 7f8d8b7fe700 1 -- 172.21.15.67:0/3904461919 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6704+0+0 (secure 0 0 0) 0x7f8d9408f070 con 0x7f8d980ffc20 2024-04-22T23:30:25.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.697+0000 7f8d9da43700 1 --2- 172.21.15.67:0/3904461919 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8d8406ecc0 0x7f8d84071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:30:25.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.697+0000 7f8d9da43700 1 --2- 172.21.15.67:0/3904461919 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8d8406ecc0 0x7f8d84071180 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c0059d0 tx=0x7f8d8c005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:30:25.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.702+0000 7f8d8b7fe700 1 -- 172.21.15.67:0/3904461919 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f8d94057d60 con 0x7f8d980ffc20 2024-04-22T23:30:25.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.941+0000 7f8da04a8700 1 -- 172.21.15.67:0/3904461919 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d98104990 con 0x7f8d8406ecc0 2024-04-22T23:30:25.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.942+0000 7f8d8b7fe700 1 -- 172.21.15.67:0/3904461919 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f8d98104990 con 0x7f8d8406ecc0 2024-04-22T23:30:25.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.944+0000 7f8da04a8700 1 -- 172.21.15.67:0/3904461919 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8d8406ecc0 msgr2=0x7f8d84071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:25.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.944+0000 7f8da04a8700 1 --2- 172.21.15.67:0/3904461919 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8d8406ecc0 0x7f8d84071180 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c0059d0 tx=0x7f8d8c005960 comp rx=0 tx=0).stop 2024-04-22T23:30:25.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.944+0000 7f8da04a8700 1 -- 172.21.15.67:0/3904461919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d980ffc20 msgr2=0x7f8d98101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:25.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.944+0000 7f8da04a8700 1 --2- 172.21.15.67:0/3904461919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d980ffc20 0x7f8d98101840 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8d9400d900 tx=0x7f8d9400dcc0 comp rx=0 tx=0).stop 2024-04-22T23:30:25.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.944+0000 7f8da04a8700 1 -- 172.21.15.67:0/3904461919 shutdown_connections 2024-04-22T23:30:25.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.944+0000 7f8da04a8700 1 --2- 172.21.15.67:0/3904461919 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8d8406ecc0 0x7f8d84071180 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:25.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.944+0000 7f8da04a8700 1 --2- 172.21.15.67:0/3904461919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d980ffc20 0x7f8d98101840 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:25.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.944+0000 7f8da04a8700 1 --2- 172.21.15.67:0/3904461919 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8d98105bd0 0x7f8d98101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:25.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.944+0000 7f8da04a8700 1 -- 172.21.15.67:0/3904461919 >> 172.21.15.67:0/3904461919 conn(0x7f8d98075670 msgr2=0x7f8d98068700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:30:25.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.944+0000 7f8da04a8700 1 -- 172.21.15.67:0/3904461919 shutdown_connections 2024-04-22T23:30:25.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:25.944+0000 7f8da04a8700 1 -- 172.21.15.67:0/3904461919 wait complete. 2024-04-22T23:30:25.953 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:30:26.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.056+0000 7f13ad6c2700 1 -- 172.21.15.67:0/3802433037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13a8101850 msgr2=0x7f13a81058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:26.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.056+0000 7f13ad6c2700 1 --2- 172.21.15.67:0/3802433037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13a8101850 0x7f13a81058a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f139c009910 tx=0x7f139c009c20 comp rx=0 tx=0).stop 2024-04-22T23:30:26.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.056+0000 7f13ad6c2700 1 -- 172.21.15.67:0/3802433037 shutdown_connections 2024-04-22T23:30:26.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.056+0000 7f13ad6c2700 1 --2- 172.21.15.67:0/3802433037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13a8101850 0x7f13a81058a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:26.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.056+0000 7f13ad6c2700 1 --2- 172.21.15.67:0/3802433037 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13a8100ea0 0x7f13a8101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:26.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.056+0000 7f13ad6c2700 1 -- 172.21.15.67:0/3802433037 >> 172.21.15.67:0/3802433037 conn(0x7f13a80fc750 msgr2=0x7f13a80feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:30:26.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.056+0000 7f13ad6c2700 1 -- 172.21.15.67:0/3802433037 shutdown_connections 2024-04-22T23:30:26.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.056+0000 7f13ad6c2700 1 -- 172.21.15.67:0/3802433037 wait complete. 2024-04-22T23:30:26.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.056+0000 7f13ad6c2700 1 Processor -- start 2024-04-22T23:30:26.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.057+0000 7f13ad6c2700 1 -- start start 2024-04-22T23:30:26.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.057+0000 7f13ad6c2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13a8100ea0 0x7f13a819e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:30:26.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.057+0000 7f13ad6c2700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13a8101850 0x7f13a819ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:30:26.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.057+0000 7f13ad6c2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a819f410 con 0x7f13a8100ea0 2024-04-22T23:30:26.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.057+0000 7f13ad6c2700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a8198950 con 0x7f13a8101850 2024-04-22T23:30:26.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.057+0000 7f13a6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13a8100ea0 0x7f13a819e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:30:26.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.057+0000 7f13a6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13a8100ea0 0x7f13a819e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58660/0 (socket says 172.21.15.67:58660) 2024-04-22T23:30:26.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.057+0000 7f13a6ffd700 1 -- 172.21.15.67:0/2661982627 learned_addr learned my addr 172.21.15.67:0/2661982627 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:30:26.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.057+0000 7f13a67fc700 1 --2- 172.21.15.67:0/2661982627 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13a8101850 0x7f13a819ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:30:26.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.057+0000 7f13a6ffd700 1 -- 172.21.15.67:0/2661982627 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13a8101850 msgr2=0x7f13a819ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:26.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.057+0000 7f13a6ffd700 1 --2- 172.21.15.67:0/2661982627 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13a8101850 0x7f13a819ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:26.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.057+0000 7f13a6ffd700 1 -- 172.21.15.67:0/2661982627 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f139c0095f0 con 0x7f13a8100ea0 2024-04-22T23:30:26.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.058+0000 7f13a6ffd700 1 --2- 172.21.15.67:0/2661982627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13a8100ea0 0x7f13a819e8d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f139800c930 tx=0x7f139800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:30:26.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.058+0000 7f138ffff700 1 -- 172.21.15.67:0/2661982627 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1398007ad0 con 0x7f13a8100ea0 2024-04-22T23:30:26.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.058+0000 7f138ffff700 1 -- 172.21.15.67:0/2661982627 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f139800ce80 con 0x7f13a8100ea0 2024-04-22T23:30:26.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.058+0000 7f13ad6c2700 1 -- 172.21.15.67:0/2661982627 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13a8198c30 con 0x7f13a8100ea0 2024-04-22T23:30:26.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.058+0000 7f138ffff700 1 -- 172.21.15.67:0/2661982627 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1398018630 con 0x7f13a8100ea0 2024-04-22T23:30:26.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.058+0000 7f13ad6c2700 1 -- 172.21.15.67:0/2661982627 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13a8199180 con 0x7f13a8100ea0 2024-04-22T23:30:26.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.059+0000 7f13ad6c2700 1 -- 172.21.15.67:0/2661982627 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13a8105020 con 0x7f13a8100ea0 2024-04-22T23:30:26.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.059+0000 7f138ffff700 1 -- 172.21.15.67:0/2661982627 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1398007c30 con 0x7f13a8100ea0 2024-04-22T23:30:26.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.060+0000 7f138ffff700 1 --2- 172.21.15.67:0/2661982627 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f139006ecc0 0x7f1390071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:30:26.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.060+0000 7f138ffff700 1 -- 172.21.15.67:0/2661982627 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6704+0+0 (secure 0 0 0) 0x7f139808ee60 con 0x7f13a8100ea0 2024-04-22T23:30:26.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.060+0000 7f13a67fc700 1 --2- 172.21.15.67:0/2661982627 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f139006ecc0 0x7f1390071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:30:26.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.060+0000 7f13a67fc700 1 --2- 172.21.15.67:0/2661982627 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f139006ecc0 0x7f1390071180 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f13a80fcee0 tx=0x7f139c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:30:26.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.065+0000 7f138ffff700 1 -- 172.21.15.67:0/2661982627 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f139805b700 con 0x7f13a8100ea0 2024-04-22T23:30:26.250 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:26 smithi067 ceph-mon[112438]: Upgrade: Updating osd.2 2024-04-22T23:30:26.251 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:26 smithi067 ceph-mon[112438]: Deploying daemon osd.2 on smithi067 2024-04-22T23:30:26.251 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:30:26.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.303+0000 7f13ad6c2700 1 -- 172.21.15.67:0/2661982627 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f13a8199eb0 con 0x7f139006ecc0 2024-04-22T23:30:26.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.304+0000 7f138ffff700 1 -- 172.21.15.67:0/2661982627 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f13a8199eb0 con 0x7f139006ecc0 2024-04-22T23:30:26.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.306+0000 7f13ad6c2700 1 -- 172.21.15.67:0/2661982627 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f139006ecc0 msgr2=0x7f1390071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:26.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.306+0000 7f13ad6c2700 1 --2- 172.21.15.67:0/2661982627 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f139006ecc0 0x7f1390071180 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f13a80fcee0 tx=0x7f139c01a040 comp rx=0 tx=0).stop 2024-04-22T23:30:26.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.306+0000 7f13ad6c2700 1 -- 172.21.15.67:0/2661982627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13a8100ea0 msgr2=0x7f13a819e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:26.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.306+0000 7f13ad6c2700 1 --2- 172.21.15.67:0/2661982627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13a8100ea0 0x7f13a819e8d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f139800c930 tx=0x7f139800cc40 comp rx=0 tx=0).stop 2024-04-22T23:30:26.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.306+0000 7f13ad6c2700 1 -- 172.21.15.67:0/2661982627 shutdown_connections 2024-04-22T23:30:26.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.306+0000 7f13ad6c2700 1 --2- 172.21.15.67:0/2661982627 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f139006ecc0 0x7f1390071180 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:26.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.306+0000 7f13ad6c2700 1 --2- 172.21.15.67:0/2661982627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13a8100ea0 0x7f13a819e8d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:26.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.306+0000 7f13ad6c2700 1 --2- 172.21.15.67:0/2661982627 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13a8101850 0x7f13a819ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:26.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.307+0000 7f13ad6c2700 1 -- 172.21.15.67:0/2661982627 >> 172.21.15.67:0/2661982627 conn(0x7f13a80fc750 msgr2=0x7f13a80feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:30:26.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.307+0000 7f13ad6c2700 1 -- 172.21.15.67:0/2661982627 shutdown_connections 2024-04-22T23:30:26.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.307+0000 7f13ad6c2700 1 -- 172.21.15.67:0/2661982627 wait complete. 2024-04-22T23:30:26.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.414+0000 7f53113ab700 1 -- 172.21.15.67:0/1854666460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f530c0ffc20 msgr2=0x7f530c100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:26.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.414+0000 7f53113ab700 1 --2- 172.21.15.67:0/1854666460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f530c0ffc20 0x7f530c100080 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f52fc009910 tx=0x7f52fc009c20 comp rx=0 tx=0).stop 2024-04-22T23:30:26.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.414+0000 7f53113ab700 1 -- 172.21.15.67:0/1854666460 shutdown_connections 2024-04-22T23:30:26.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.414+0000 7f53113ab700 1 --2- 172.21.15.67:0/1854666460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f530c0ffc20 0x7f530c100080 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:26.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.414+0000 7f53113ab700 1 --2- 172.21.15.67:0/1854666460 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f530c105bd0 0x7f530c105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:26.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.414+0000 7f53113ab700 1 -- 172.21.15.67:0/1854666460 >> 172.21.15.67:0/1854666460 conn(0x7f530c075670 msgr2=0x7f530c075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:30:26.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.414+0000 7f53113ab700 1 -- 172.21.15.67:0/1854666460 shutdown_connections 2024-04-22T23:30:26.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.414+0000 7f53113ab700 1 -- 172.21.15.67:0/1854666460 wait complete. 2024-04-22T23:30:26.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.415+0000 7f53113ab700 1 Processor -- start 2024-04-22T23:30:26.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.415+0000 7f53113ab700 1 -- start start 2024-04-22T23:30:26.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.415+0000 7f53113ab700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f530c0ffc20 0x7f530c19c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:30:26.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.415+0000 7f53113ab700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f530c105bd0 0x7f530c19cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:30:26.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.415+0000 7f53113ab700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530c19d230 con 0x7f530c0ffc20 2024-04-22T23:30:26.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.415+0000 7f53113ab700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530c1966e0 con 0x7f530c105bd0 2024-04-22T23:30:26.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.415+0000 7f530affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f530c0ffc20 0x7f530c19c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:30:26.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.415+0000 7f530affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f530c0ffc20 0x7f530c19c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58680/0 (socket says 172.21.15.67:58680) 2024-04-22T23:30:26.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.415+0000 7f530affd700 1 -- 172.21.15.67:0/632638153 learned_addr learned my addr 172.21.15.67:0/632638153 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:30:26.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.415+0000 7f530a7fc700 1 --2- 172.21.15.67:0/632638153 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f530c105bd0 0x7f530c19cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:30:26.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.416+0000 7f530affd700 1 -- 172.21.15.67:0/632638153 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f530c105bd0 msgr2=0x7f530c19cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:26.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.416+0000 7f530affd700 1 --2- 172.21.15.67:0/632638153 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f530c105bd0 0x7f530c19cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:26.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.416+0000 7f530affd700 1 -- 172.21.15.67:0/632638153 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52fc0095f0 con 0x7f530c0ffc20 2024-04-22T23:30:26.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.416+0000 7f530a7fc700 1 --2- 172.21.15.67:0/632638153 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f530c105bd0 0x7f530c19cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:30:26.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.416+0000 7f530affd700 1 --2- 172.21.15.67:0/632638153 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f530c0ffc20 0x7f530c19c660 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f530000e920 tx=0x7f530000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:30:26.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.416+0000 7f52f3fff700 1 -- 172.21.15.67:0/632638153 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5300009790 con 0x7f530c0ffc20 2024-04-22T23:30:26.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.416+0000 7f52f3fff700 1 -- 172.21.15.67:0/632638153 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f530000cba0 con 0x7f530c0ffc20 2024-04-22T23:30:26.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.416+0000 7f53113ab700 1 -- 172.21.15.67:0/632638153 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f530c1969c0 con 0x7f530c0ffc20 2024-04-22T23:30:26.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.416+0000 7f52f3fff700 1 -- 172.21.15.67:0/632638153 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5300018650 con 0x7f530c0ffc20 2024-04-22T23:30:26.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.416+0000 7f53113ab700 1 -- 172.21.15.67:0/632638153 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f530c196f10 con 0x7f530c0ffc20 2024-04-22T23:30:26.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.417+0000 7f53113ab700 1 -- 172.21.15.67:0/632638153 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f530c04f2e0 con 0x7f530c0ffc20 2024-04-22T23:30:26.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.418+0000 7f52f3fff700 1 -- 172.21.15.67:0/632638153 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f530000cd10 con 0x7f530c0ffc20 2024-04-22T23:30:26.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.418+0000 7f52f3fff700 1 --2- 172.21.15.67:0/632638153 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f52f406ecc0 0x7f52f4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:30:26.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.418+0000 7f52f3fff700 1 -- 172.21.15.67:0/632638153 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6704+0+0 (secure 0 0 0) 0x7f5300014070 con 0x7f530c0ffc20 2024-04-22T23:30:26.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.418+0000 7f530a7fc700 1 --2- 172.21.15.67:0/632638153 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f52f406ecc0 0x7f52f4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:30:26.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.419+0000 7f530a7fc700 1 --2- 172.21.15.67:0/632638153 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f52f406ecc0 0x7f52f4071180 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f52fc009fd0 tx=0x7f52fc005940 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:30:26.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.423+0000 7f52f3fff700 1 -- 172.21.15.67:0/632638153 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f5300054150 con 0x7f530c0ffc20 2024-04-22T23:30:26.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:26 smithi134 ceph-mon[85630]: Upgrade: Updating osd.2 2024-04-22T23:30:26.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:26 smithi134 ceph-mon[85630]: Deploying daemon osd.2 on smithi067 2024-04-22T23:30:26.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:30:26.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.644+0000 7f53113ab700 1 -- 172.21.15.67:0/632638153 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f530c197240 con 0x7f52f406ecc0 2024-04-22T23:30:26.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.655+0000 7f52f3fff700 1 -- 172.21.15.67:0/632638153 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f530c197240 con 0x7f52f406ecc0 2024-04-22T23:30:26.654 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:30:26.654 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (4m) 23s ago 15m 26.1M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:30:26.654 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (15m) 23s ago 15m 8627k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:30:26.654 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (14m) 82s ago 14m 8200k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:30:26.654 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (91s) 23s ago 15m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:30:26.655 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (86s) 82s ago 14m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:30:26.655 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (3m) 23s ago 15m 80.2M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:30:26.655 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (9m) 23s ago 9m 62.7M - 18.2.1 5be31c24972a e24a471f0a8c 2024-04-22T23:30:26.655 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (9m) 23s ago 9m 130M - 18.2.1 5be31c24972a 0730f9fc8e61 2024-04-22T23:30:26.655 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (9m) 82s ago 9m 123M - 18.2.1 5be31c24972a b6a169586433 2024-04-22T23:30:26.655 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (9m) 82s ago 9m 67.7M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:30:26.656 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (5m) 23s ago 16m 505M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:30:26.656 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (5m) 82s ago 14m 450M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:30:26.656 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (2m) 23s ago 17m 48.2M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:30:26.656 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (2m) 82s ago 14m 38.2M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:30:26.656 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (4m) 23s ago 15m 15.1M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:30:26.656 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (4m) 82s ago 14m 15.9M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:30:26.656 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (65s) 23s ago 13m 81.3M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:30:26.657 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (29s) 23s ago 13m 12.6M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:30:26.657 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (12m) 23s ago 12m 173M 1216M 18.2.1 5be31c24972a e257ecb4efa1 2024-04-22T23:30:26.657 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (12m) 82s ago 12m 251M 2240M 18.2.1 5be31c24972a 9c2c91a967ba 2024-04-22T23:30:26.657 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (11m) 82s ago 11m 219M 2240M 18.2.1 5be31c24972a c7586562ffa6 2024-04-22T23:30:26.657 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (11m) 82s ago 11m 196M 2240M 18.2.1 5be31c24972a 9fd97f1c65cb 2024-04-22T23:30:26.657 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (4m) 23s ago 14m 71.2M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:30:26.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.658+0000 7f53113ab700 1 -- 172.21.15.67:0/632638153 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f52f406ecc0 msgr2=0x7f52f4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:26.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.658+0000 7f53113ab700 1 --2- 172.21.15.67:0/632638153 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f52f406ecc0 0x7f52f4071180 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f52fc009fd0 tx=0x7f52fc005940 comp rx=0 tx=0).stop 2024-04-22T23:30:26.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.658+0000 7f53113ab700 1 -- 172.21.15.67:0/632638153 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f530c0ffc20 msgr2=0x7f530c19c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:26.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.658+0000 7f53113ab700 1 --2- 172.21.15.67:0/632638153 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f530c0ffc20 0x7f530c19c660 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f530000e920 tx=0x7f530000ece0 comp rx=0 tx=0).stop 2024-04-22T23:30:26.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.658+0000 7f53113ab700 1 -- 172.21.15.67:0/632638153 shutdown_connections 2024-04-22T23:30:26.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.658+0000 7f53113ab700 1 --2- 172.21.15.67:0/632638153 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f52f406ecc0 0x7f52f4071180 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:26.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.658+0000 7f53113ab700 1 --2- 172.21.15.67:0/632638153 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f530c0ffc20 0x7f530c19c660 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:26.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.658+0000 7f53113ab700 1 --2- 172.21.15.67:0/632638153 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f530c105bd0 0x7f530c19cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:26.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.658+0000 7f53113ab700 1 -- 172.21.15.67:0/632638153 >> 172.21.15.67:0/632638153 conn(0x7f530c075670 msgr2=0x7f530c0686a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:30:26.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.658+0000 7f53113ab700 1 -- 172.21.15.67:0/632638153 shutdown_connections 2024-04-22T23:30:26.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.658+0000 7f53113ab700 1 -- 172.21.15.67:0/632638153 wait complete. 2024-04-22T23:30:26.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.772+0000 7f09f199e700 1 -- 172.21.15.67:0/4130073934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09ec100ea0 msgr2=0x7f09ec101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:26.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.772+0000 7f09f199e700 1 --2- 172.21.15.67:0/4130073934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09ec100ea0 0x7f09ec101280 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f09dc009910 tx=0x7f09dc009c20 comp rx=0 tx=0).stop 2024-04-22T23:30:26.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.772+0000 7f09f199e700 1 -- 172.21.15.67:0/4130073934 shutdown_connections 2024-04-22T23:30:26.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.772+0000 7f09f199e700 1 --2- 172.21.15.67:0/4130073934 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09ec101850 0x7f09ec1058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:26.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.772+0000 7f09f199e700 1 --2- 172.21.15.67:0/4130073934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09ec100ea0 0x7f09ec101280 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:26.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.772+0000 7f09f199e700 1 -- 172.21.15.67:0/4130073934 >> 172.21.15.67:0/4130073934 conn(0x7f09ec0fc750 msgr2=0x7f09ec0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:30:26.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.772+0000 7f09f199e700 1 -- 172.21.15.67:0/4130073934 shutdown_connections 2024-04-22T23:30:26.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.772+0000 7f09f199e700 1 -- 172.21.15.67:0/4130073934 wait complete. 2024-04-22T23:30:26.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.772+0000 7f09f199e700 1 Processor -- start 2024-04-22T23:30:26.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.773+0000 7f09f199e700 1 -- start start 2024-04-22T23:30:26.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.773+0000 7f09f199e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09ec100ea0 0x7f09ec073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:30:26.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.773+0000 7f09f199e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09ec101850 0x7f09ec0735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:30:26.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.773+0000 7f09f199e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec073b10 con 0x7f09ec100ea0 2024-04-22T23:30:26.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.773+0000 7f09f199e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec073c50 con 0x7f09ec101850 2024-04-22T23:30:26.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.773+0000 7f09eaffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09ec100ea0 0x7f09ec073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:30:26.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.773+0000 7f09eaffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09ec100ea0 0x7f09ec073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58694/0 (socket says 172.21.15.67:58694) 2024-04-22T23:30:26.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.773+0000 7f09eaffd700 1 -- 172.21.15.67:0/1571921243 learned_addr learned my addr 172.21.15.67:0/1571921243 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:30:26.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.773+0000 7f09ea7fc700 1 --2- 172.21.15.67:0/1571921243 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09ec101850 0x7f09ec0735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:30:26.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.773+0000 7f09eaffd700 1 -- 172.21.15.67:0/1571921243 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09ec101850 msgr2=0x7f09ec0735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:26.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.773+0000 7f09eaffd700 1 --2- 172.21.15.67:0/1571921243 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09ec101850 0x7f09ec0735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:26.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.773+0000 7f09eaffd700 1 -- 172.21.15.67:0/1571921243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09dc0095f0 con 0x7f09ec100ea0 2024-04-22T23:30:26.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.774+0000 7f09eaffd700 1 --2- 172.21.15.67:0/1571921243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09ec100ea0 0x7f09ec073090 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f09dc005f50 tx=0x7f09dc00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:30:26.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.774+0000 7f09f099c700 1 -- 172.21.15.67:0/1571921243 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09dc01d070 con 0x7f09ec100ea0 2024-04-22T23:30:26.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.774+0000 7f09f199e700 1 -- 172.21.15.67:0/1571921243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09ec073ed0 con 0x7f09ec100ea0 2024-04-22T23:30:26.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.774+0000 7f09f099c700 1 -- 172.21.15.67:0/1571921243 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f09dc00b610 con 0x7f09ec100ea0 2024-04-22T23:30:26.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.774+0000 7f09f099c700 1 -- 172.21.15.67:0/1571921243 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09dc022470 con 0x7f09ec100ea0 2024-04-22T23:30:26.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.774+0000 7f09f199e700 1 -- 172.21.15.67:0/1571921243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09ec1a2e20 con 0x7f09ec100ea0 2024-04-22T23:30:26.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.775+0000 7f09f199e700 1 -- 172.21.15.67:0/1571921243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09ec109240 con 0x7f09ec100ea0 2024-04-22T23:30:26.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.775+0000 7f09f099c700 1 -- 172.21.15.67:0/1571921243 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f09dc003780 con 0x7f09ec100ea0 2024-04-22T23:30:26.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.776+0000 7f09f099c700 1 --2- 172.21.15.67:0/1571921243 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f09d406ecc0 0x7f09d4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:30:26.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.776+0000 7f09f099c700 1 -- 172.21.15.67:0/1571921243 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6704+0+0 (secure 0 0 0) 0x7f09dc0902e0 con 0x7f09ec100ea0 2024-04-22T23:30:26.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.776+0000 7f09ea7fc700 1 --2- 172.21.15.67:0/1571921243 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f09d406ecc0 0x7f09d4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:30:26.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.776+0000 7f09ea7fc700 1 --2- 172.21.15.67:0/1571921243 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f09d406ecc0 0x7f09d4071180 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f09ec074880 tx=0x7f09e0009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:30:26.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:26.780+0000 7f09f099c700 1 -- 172.21.15.67:0/1571921243 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f09dc058fd0 con 0x7f09ec100ea0 2024-04-22T23:30:27.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.082+0000 7f09f199e700 1 -- 172.21.15.67:0/1571921243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f09ec04f2e0 con 0x7f09ec100ea0 2024-04-22T23:30:27.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.083+0000 7f09f099c700 1 -- 172.21.15.67:0/1571921243 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7f09dc05c690 con 0x7f09ec100ea0 2024-04-22T23:30:27.081 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:30:27.081 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:30:27.081 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:30:27.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:30:27.081 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:30:27.081 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:30:27.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:30:27.081 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:30:27.081 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4, 2024-04-22T23:30:27.081 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:30:27.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:30:27.081 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:30:27.082 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-22T23:30:27.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:30:27.082 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:30:27.082 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 8, 2024-04-22T23:30:27.082 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:30:27.082 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:30:27.082 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:30:27.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.085+0000 7f09f199e700 1 -- 172.21.15.67:0/1571921243 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f09d406ecc0 msgr2=0x7f09d4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:27.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.085+0000 7f09f199e700 1 --2- 172.21.15.67:0/1571921243 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f09d406ecc0 0x7f09d4071180 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f09ec074880 tx=0x7f09e0009260 comp rx=0 tx=0).stop 2024-04-22T23:30:27.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.085+0000 7f09f199e700 1 -- 172.21.15.67:0/1571921243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09ec100ea0 msgr2=0x7f09ec073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:27.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.085+0000 7f09f199e700 1 --2- 172.21.15.67:0/1571921243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09ec100ea0 0x7f09ec073090 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f09dc005f50 tx=0x7f09dc00f690 comp rx=0 tx=0).stop 2024-04-22T23:30:27.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.086+0000 7f09f199e700 1 -- 172.21.15.67:0/1571921243 shutdown_connections 2024-04-22T23:30:27.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.086+0000 7f09f199e700 1 --2- 172.21.15.67:0/1571921243 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f09d406ecc0 0x7f09d4071180 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:27.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.086+0000 7f09f199e700 1 --2- 172.21.15.67:0/1571921243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09ec100ea0 0x7f09ec073090 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:27.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.086+0000 7f09f199e700 1 --2- 172.21.15.67:0/1571921243 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09ec101850 0x7f09ec0735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:27.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.086+0000 7f09f199e700 1 -- 172.21.15.67:0/1571921243 >> 172.21.15.67:0/1571921243 conn(0x7f09ec0fc750 msgr2=0x7f09ec0fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:30:27.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.086+0000 7f09f199e700 1 -- 172.21.15.67:0/1571921243 shutdown_connections 2024-04-22T23:30:27.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.086+0000 7f09f199e700 1 -- 172.21.15.67:0/1571921243 wait complete. 2024-04-22T23:30:27.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.195+0000 7f7d7c9ed700 1 -- 172.21.15.67:0/3467162049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d78100750 msgr2=0x7f7d781047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:27.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.195+0000 7f7d7c9ed700 1 --2- 172.21.15.67:0/3467162049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d78100750 0x7f7d781047a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c009910 tx=0x7f7d6c009c20 comp rx=0 tx=0).stop 2024-04-22T23:30:27.193 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:27 smithi067 ceph-mon[112438]: from='client.34252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:30:27.194 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:27 smithi067 ceph-mon[112438]: pgmap v96: 65 pgs: 65 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:30:27.194 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:27 smithi067 ceph-mon[112438]: from='client.34256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:30:27.194 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:27 smithi067 ceph-mon[112438]: from='client.34260 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:30:27.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.196+0000 7f7d7c9ed700 1 -- 172.21.15.67:0/3467162049 shutdown_connections 2024-04-22T23:30:27.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.196+0000 7f7d7c9ed700 1 --2- 172.21.15.67:0/3467162049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d78100750 0x7f7d781047a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:27.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.196+0000 7f7d7c9ed700 1 --2- 172.21.15.67:0/3467162049 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d780ffda0 0x7f7d78100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:27.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.196+0000 7f7d7c9ed700 1 -- 172.21.15.67:0/3467162049 >> 172.21.15.67:0/3467162049 conn(0x7f7d780fb650 msgr2=0x7f7d780fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:30:27.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.196+0000 7f7d7c9ed700 1 -- 172.21.15.67:0/3467162049 shutdown_connections 2024-04-22T23:30:27.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.196+0000 7f7d7c9ed700 1 -- 172.21.15.67:0/3467162049 wait complete. 2024-04-22T23:30:27.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.197+0000 7f7d7c9ed700 1 Processor -- start 2024-04-22T23:30:27.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.197+0000 7f7d7c9ed700 1 -- start start 2024-04-22T23:30:27.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.197+0000 7f7d7c9ed700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d780ffda0 0x7f7d78101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:30:27.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.197+0000 7f7d7c9ed700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d78100750 0x7f7d78101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:30:27.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.197+0000 7f7d7c9ed700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d781022c0 con 0x7f7d780ffda0 2024-04-22T23:30:27.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.197+0000 7f7d7c9ed700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d78102400 con 0x7f7d78100750 2024-04-22T23:30:27.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.197+0000 7f7d7659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d780ffda0 0x7f7d78101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:30:27.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.197+0000 7f7d7659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d780ffda0 0x7f7d78101840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58716/0 (socket says 172.21.15.67:58716) 2024-04-22T23:30:27.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.197+0000 7f7d7659c700 1 -- 172.21.15.67:0/3073361697 learned_addr learned my addr 172.21.15.67:0/3073361697 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:30:27.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.197+0000 7f7d75d9b700 1 --2- 172.21.15.67:0/3073361697 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d78100750 0x7f7d78101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:30:27.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.198+0000 7f7d7659c700 1 -- 172.21.15.67:0/3073361697 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d78100750 msgr2=0x7f7d78101d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:27.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.198+0000 7f7d7659c700 1 --2- 172.21.15.67:0/3073361697 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d78100750 0x7f7d78101d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:27.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.198+0000 7f7d7659c700 1 -- 172.21.15.67:0/3073361697 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d6c0095f0 con 0x7f7d780ffda0 2024-04-22T23:30:27.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.198+0000 7f7d7659c700 1 --2- 172.21.15.67:0/3073361697 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d780ffda0 0x7f7d78101840 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7d6800b4b0 tx=0x7f7d6800b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:30:27.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.198+0000 7f7d677fe700 1 -- 172.21.15.67:0/3073361697 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d68011800 con 0x7f7d780ffda0 2024-04-22T23:30:27.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.198+0000 7f7d677fe700 1 -- 172.21.15.67:0/3073361697 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7d68011e20 con 0x7f7d780ffda0 2024-04-22T23:30:27.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.198+0000 7f7d7c9ed700 1 -- 172.21.15.67:0/3073361697 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d781a2b80 con 0x7f7d780ffda0 2024-04-22T23:30:27.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.198+0000 7f7d677fe700 1 -- 172.21.15.67:0/3073361697 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d6800f2e0 con 0x7f7d780ffda0 2024-04-22T23:30:27.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.198+0000 7f7d7c9ed700 1 -- 172.21.15.67:0/3073361697 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d781a2f30 con 0x7f7d780ffda0 2024-04-22T23:30:27.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.200+0000 7f7d7c9ed700 1 -- 172.21.15.67:0/3073361697 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d78102680 con 0x7f7d780ffda0 2024-04-22T23:30:27.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.200+0000 7f7d677fe700 1 -- 172.21.15.67:0/3073361697 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7d68010380 con 0x7f7d780ffda0 2024-04-22T23:30:27.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.201+0000 7f7d677fe700 1 --2- 172.21.15.67:0/3073361697 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7d6006ecc0 0x7f7d60071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:30:27.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.201+0000 7f7d75d9b700 1 --2- 172.21.15.67:0/3073361697 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7d6006ecc0 0x7f7d60071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:30:27.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.201+0000 7f7d677fe700 1 -- 172.21.15.67:0/3073361697 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6704+0+0 (secure 0 0 0) 0x7f7d6808e950 con 0x7f7d780ffda0 2024-04-22T23:30:27.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.201+0000 7f7d75d9b700 1 --2- 172.21.15.67:0/3073361697 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7d6006ecc0 0x7f7d60071180 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c006010 tx=0x7f7d6c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:30:27.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.205+0000 7f7d677fe700 1 -- 172.21.15.67:0/3073361697 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7d680576c0 con 0x7f7d780ffda0 2024-04-22T23:30:27.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.467+0000 7f7d7c9ed700 1 -- 172.21.15.67:0/3073361697 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f7d7804f2e0 con 0x7f7d780ffda0 2024-04-22T23:30:27.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.468+0000 7f7d677fe700 1 -- 172.21.15.67:0/3073361697 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1901 (secure 0 0 0) 0x7f7d6805ad80 con 0x7f7d780ffda0 2024-04-22T23:30:27.466 INFO:teuthology.orchestra.run.smithi067.stdout:e22 2024-04-22T23:30:27.466 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:30:27.467 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:30:27.467 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:30:27.467 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:30:27.467 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:30:27.467 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:30:27.467 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 22 2024-04-22T23:30:27.467 INFO:teuthology.orchestra.run.smithi067.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T23:30:27.467 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:30:27.467 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:27:20.577198+0000 2024-04-22T23:30:27.467 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:30:27.467 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:30:27.467 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:30:27.467 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:30:27.467 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:30:27.467 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:30:27.468 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:30:27.468 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:30:27.468 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 46 2024-04-22T23:30:27.468 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:30:27.468 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:30:27.468 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:30:27.468 INFO:teuthology.orchestra.run.smithi067.stdout:up {0=14484,1=24273} 2024-04-22T23:30:27.468 INFO:teuthology.orchestra.run.smithi067.stdout:failed 2024-04-22T23:30:27.468 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:30:27.468 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:30:27.468 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:30:27.468 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:30:27.468 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:30:27.468 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:30:27.468 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:30:27.468 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:30:27.469 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:30:27.469 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{0:14484} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.67:6828/3606151860,v1:172.21.15.67:6829/3606151860] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:30:27.469 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{0:14496} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/3305304988,v1:172.21.15.67:6827/3305304988] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:30:27.469 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{1:24273} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.134:6824/535957142,v1:172.21.15.134:6825/535957142] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:30:27.469 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{1:24289} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.134:6826/1954821676,v1:172.21.15.134:6827/1954821676] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:30:27.469 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:30:27.469 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:30:27.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.470+0000 7f7d7c9ed700 1 -- 172.21.15.67:0/3073361697 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7d6006ecc0 msgr2=0x7f7d60071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:27.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.470+0000 7f7d7c9ed700 1 --2- 172.21.15.67:0/3073361697 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7d6006ecc0 0x7f7d60071180 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c006010 tx=0x7f7d6c01a040 comp rx=0 tx=0).stop 2024-04-22T23:30:27.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.470+0000 7f7d7c9ed700 1 -- 172.21.15.67:0/3073361697 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d780ffda0 msgr2=0x7f7d78101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:27.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.470+0000 7f7d7c9ed700 1 --2- 172.21.15.67:0/3073361697 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d780ffda0 0x7f7d78101840 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7d6800b4b0 tx=0x7f7d6800b7c0 comp rx=0 tx=0).stop 2024-04-22T23:30:27.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.470+0000 7f7d7c9ed700 1 -- 172.21.15.67:0/3073361697 shutdown_connections 2024-04-22T23:30:27.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.470+0000 7f7d7c9ed700 1 --2- 172.21.15.67:0/3073361697 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7d6006ecc0 0x7f7d60071180 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:27.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.470+0000 7f7d7c9ed700 1 --2- 172.21.15.67:0/3073361697 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d780ffda0 0x7f7d78101840 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:27.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.470+0000 7f7d7c9ed700 1 --2- 172.21.15.67:0/3073361697 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d78100750 0x7f7d78101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:27.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.470+0000 7f7d7c9ed700 1 -- 172.21.15.67:0/3073361697 >> 172.21.15.67:0/3073361697 conn(0x7f7d780fb650 msgr2=0x7f7d78103ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:30:27.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.470+0000 7f7d7c9ed700 1 -- 172.21.15.67:0/3073361697 shutdown_connections 2024-04-22T23:30:27.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.470+0000 7f7d7c9ed700 1 -- 172.21.15.67:0/3073361697 wait complete. 2024-04-22T23:30:27.471 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 22 2024-04-22T23:30:27.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:27 smithi134 ceph-mon[85630]: from='client.34252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:30:27.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:27 smithi134 ceph-mon[85630]: pgmap v96: 65 pgs: 65 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:30:27.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:27 smithi134 ceph-mon[85630]: from='client.34256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:30:27.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:27 smithi134 ceph-mon[85630]: from='client.34260 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:30:27.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.581+0000 7fe8f646c700 1 -- 172.21.15.67:0/1980816178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f00ffda0 msgr2=0x7fe8f0100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:27.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.581+0000 7fe8f646c700 1 --2- 172.21.15.67:0/1980816178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f00ffda0 0x7fe8f0100180 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe8e0009910 tx=0x7fe8e0009c20 comp rx=0 tx=0).stop 2024-04-22T23:30:27.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.581+0000 7fe8f646c700 1 -- 172.21.15.67:0/1980816178 shutdown_connections 2024-04-22T23:30:27.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.581+0000 7fe8f646c700 1 --2- 172.21.15.67:0/1980816178 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8f0100750 0x7fe8f01047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:27.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.581+0000 7fe8f646c700 1 --2- 172.21.15.67:0/1980816178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f00ffda0 0x7fe8f0100180 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:27.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.581+0000 7fe8f646c700 1 -- 172.21.15.67:0/1980816178 >> 172.21.15.67:0/1980816178 conn(0x7fe8f00fb650 msgr2=0x7fe8f00fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:30:27.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.582+0000 7fe8f646c700 1 -- 172.21.15.67:0/1980816178 shutdown_connections 2024-04-22T23:30:27.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.582+0000 7fe8f646c700 1 -- 172.21.15.67:0/1980816178 wait complete. 2024-04-22T23:30:27.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.582+0000 7fe8f646c700 1 Processor -- start 2024-04-22T23:30:27.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.582+0000 7fe8f646c700 1 -- start start 2024-04-22T23:30:27.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.582+0000 7fe8f646c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f00ffda0 0x7fe8f0101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:30:27.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.583+0000 7fe8f646c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8f0100750 0x7fe8f0101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:30:27.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.583+0000 7fe8f646c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f01022c0 con 0x7fe8f00ffda0 2024-04-22T23:30:27.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.583+0000 7fe8f646c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f0102400 con 0x7fe8f0100750 2024-04-22T23:30:27.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.583+0000 7fe8effff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f00ffda0 0x7fe8f0101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:30:27.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.583+0000 7fe8effff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f00ffda0 0x7fe8f0101840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58742/0 (socket says 172.21.15.67:58742) 2024-04-22T23:30:27.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.583+0000 7fe8effff700 1 -- 172.21.15.67:0/1784067288 learned_addr learned my addr 172.21.15.67:0/1784067288 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:30:27.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.583+0000 7fe8ef7fe700 1 --2- 172.21.15.67:0/1784067288 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8f0100750 0x7fe8f0101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:30:27.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.583+0000 7fe8effff700 1 -- 172.21.15.67:0/1784067288 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8f0100750 msgr2=0x7fe8f0101d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:27.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.583+0000 7fe8effff700 1 --2- 172.21.15.67:0/1784067288 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8f0100750 0x7fe8f0101d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:27.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.583+0000 7fe8effff700 1 -- 172.21.15.67:0/1784067288 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e00095f0 con 0x7fe8f00ffda0 2024-04-22T23:30:27.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.583+0000 7fe8effff700 1 --2- 172.21.15.67:0/1784067288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f00ffda0 0x7fe8f0101840 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe8e0006010 tx=0x7fe8e000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:30:27.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.583+0000 7fe8ed7fa700 1 -- 172.21.15.67:0/1784067288 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8e001d070 con 0x7fe8f00ffda0 2024-04-22T23:30:27.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.583+0000 7fe8ed7fa700 1 -- 172.21.15.67:0/1784067288 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe8e0005dc0 con 0x7fe8f00ffda0 2024-04-22T23:30:27.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.583+0000 7fe8f646c700 1 -- 172.21.15.67:0/1784067288 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8f0102680 con 0x7fe8f00ffda0 2024-04-22T23:30:27.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.583+0000 7fe8ed7fa700 1 -- 172.21.15.67:0/1784067288 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8e0017400 con 0x7fe8f00ffda0 2024-04-22T23:30:27.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.583+0000 7fe8f646c700 1 -- 172.21.15.67:0/1784067288 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8f01a2d70 con 0x7fe8f00ffda0 2024-04-22T23:30:27.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.585+0000 7fe8f646c700 1 -- 172.21.15.67:0/1784067288 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8f004f2e0 con 0x7fe8f00ffda0 2024-04-22T23:30:27.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.585+0000 7fe8ed7fa700 1 -- 172.21.15.67:0/1784067288 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe8e0003a60 con 0x7fe8f00ffda0 2024-04-22T23:30:27.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.585+0000 7fe8ed7fa700 1 --2- 172.21.15.67:0/1784067288 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe8d806ecc0 0x7fe8d8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:30:27.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.585+0000 7fe8ed7fa700 1 -- 172.21.15.67:0/1784067288 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6704+0+0 (secure 0 0 0) 0x7fe8e00902f0 con 0x7fe8f00ffda0 2024-04-22T23:30:27.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.585+0000 7fe8ef7fe700 1 --2- 172.21.15.67:0/1784067288 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe8d806ecc0 0x7fe8d8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:30:27.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.586+0000 7fe8ef7fe700 1 --2- 172.21.15.67:0/1784067288 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe8d806ecc0 0x7fe8d8071180 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe8f0103030 tx=0x7fe8e400b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:30:27.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.590+0000 7fe8ed7fa700 1 -- 172.21.15.67:0/1784067288 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe8e0059060 con 0x7fe8f00ffda0 2024-04-22T23:30:27.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.823+0000 7fe8f646c700 1 -- 172.21.15.67:0/1784067288 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe8f00fc4e0 con 0x7fe8d806ecc0 2024-04-22T23:30:27.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.824+0000 7fe8ed7fa700 1 -- 172.21.15.67:0/1784067288 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fe8f00fc4e0 con 0x7fe8d806ecc0 2024-04-22T23:30:27.823 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:30:27.823 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:30:27.823 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:30:27.823 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:30:27.823 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:30:27.823 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:30:27.823 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:30:27.823 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr" 2024-04-22T23:30:27.823 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:30:27.823 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "8/23 daemons upgraded", 2024-04-22T23:30:27.823 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading osd daemons", 2024-04-22T23:30:27.823 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:30:27.824 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:30:27.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.826+0000 7fe8f646c700 1 -- 172.21.15.67:0/1784067288 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe8d806ecc0 msgr2=0x7fe8d8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:27.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.826+0000 7fe8f646c700 1 --2- 172.21.15.67:0/1784067288 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe8d806ecc0 0x7fe8d8071180 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe8f0103030 tx=0x7fe8e400b410 comp rx=0 tx=0).stop 2024-04-22T23:30:27.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.826+0000 7fe8f646c700 1 -- 172.21.15.67:0/1784067288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f00ffda0 msgr2=0x7fe8f0101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:27.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.826+0000 7fe8f646c700 1 --2- 172.21.15.67:0/1784067288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f00ffda0 0x7fe8f0101840 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe8e0006010 tx=0x7fe8e000bd70 comp rx=0 tx=0).stop 2024-04-22T23:30:27.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.827+0000 7fe8f646c700 1 -- 172.21.15.67:0/1784067288 shutdown_connections 2024-04-22T23:30:27.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.827+0000 7fe8f646c700 1 --2- 172.21.15.67:0/1784067288 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe8d806ecc0 0x7fe8d8071180 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:27.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.827+0000 7fe8f646c700 1 --2- 172.21.15.67:0/1784067288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f00ffda0 0x7fe8f0101840 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:27.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.827+0000 7fe8f646c700 1 --2- 172.21.15.67:0/1784067288 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8f0100750 0x7fe8f0101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:27.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.827+0000 7fe8f646c700 1 -- 172.21.15.67:0/1784067288 >> 172.21.15.67:0/1784067288 conn(0x7fe8f00fb650 msgr2=0x7fe8f0103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:30:27.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.827+0000 7fe8f646c700 1 -- 172.21.15.67:0/1784067288 shutdown_connections 2024-04-22T23:30:27.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.827+0000 7fe8f646c700 1 -- 172.21.15.67:0/1784067288 wait complete. 2024-04-22T23:30:27.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.934+0000 7f8cfd185700 1 -- 172.21.15.67:0/922393560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cf80ffc20 msgr2=0x7f8cf8100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:27.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.934+0000 7f8cfd185700 1 --2- 172.21.15.67:0/922393560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cf80ffc20 0x7f8cf8100080 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8ce0009910 tx=0x7f8ce0009c20 comp rx=0 tx=0).stop 2024-04-22T23:30:27.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.934+0000 7f8cfd185700 1 -- 172.21.15.67:0/922393560 shutdown_connections 2024-04-22T23:30:27.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.934+0000 7f8cfd185700 1 --2- 172.21.15.67:0/922393560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cf80ffc20 0x7f8cf8100080 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:27.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.934+0000 7f8cfd185700 1 --2- 172.21.15.67:0/922393560 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8cf8105bd0 0x7f8cf8105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:27.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.934+0000 7f8cfd185700 1 -- 172.21.15.67:0/922393560 >> 172.21.15.67:0/922393560 conn(0x7f8cf8075670 msgr2=0x7f8cf8075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:30:27.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.934+0000 7f8cfd185700 1 -- 172.21.15.67:0/922393560 shutdown_connections 2024-04-22T23:30:27.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.934+0000 7f8cfd185700 1 -- 172.21.15.67:0/922393560 wait complete. 2024-04-22T23:30:27.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.935+0000 7f8cfd185700 1 Processor -- start 2024-04-22T23:30:27.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.935+0000 7f8cfd185700 1 -- start start 2024-04-22T23:30:27.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.935+0000 7f8cfd185700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8cf80ffc20 0x7f8cf8101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:30:27.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.935+0000 7f8cfd185700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cf8105bd0 0x7f8cf8101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:30:27.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.935+0000 7f8cfd185700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf81022c0 con 0x7f8cf8105bd0 2024-04-22T23:30:27.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.935+0000 7f8cfd185700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf8102400 con 0x7f8cf80ffc20 2024-04-22T23:30:27.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.935+0000 7f8cf659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cf8105bd0 0x7f8cf8101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:30:27.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.936+0000 7f8cf659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cf8105bd0 0x7f8cf8101d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58768/0 (socket says 172.21.15.67:58768) 2024-04-22T23:30:27.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.936+0000 7f8cf659c700 1 -- 172.21.15.67:0/3655965241 learned_addr learned my addr 172.21.15.67:0/3655965241 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:30:27.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.936+0000 7f8cf6d9d700 1 --2- 172.21.15.67:0/3655965241 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8cf80ffc20 0x7f8cf8101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:30:27.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.936+0000 7f8cf659c700 1 -- 172.21.15.67:0/3655965241 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8cf80ffc20 msgr2=0x7f8cf8101840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:27.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.936+0000 7f8cf659c700 1 --2- 172.21.15.67:0/3655965241 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8cf80ffc20 0x7f8cf8101840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:27.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.936+0000 7f8cf659c700 1 -- 172.21.15.67:0/3655965241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce00095f0 con 0x7f8cf8105bd0 2024-04-22T23:30:27.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.936+0000 7f8cf659c700 1 --2- 172.21.15.67:0/3655965241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cf8105bd0 0x7f8cf8101d80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8ce0006010 tx=0x7f8ce000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:30:27.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.936+0000 7f8ceffff700 1 -- 172.21.15.67:0/3655965241 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ce001d070 con 0x7f8cf8105bd0 2024-04-22T23:30:27.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.936+0000 7f8cfd185700 1 -- 172.21.15.67:0/3655965241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cf8102680 con 0x7f8cf8105bd0 2024-04-22T23:30:27.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.936+0000 7f8ceffff700 1 -- 172.21.15.67:0/3655965241 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ce000f460 con 0x7f8cf8105bd0 2024-04-22T23:30:27.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.936+0000 7f8cfd185700 1 -- 172.21.15.67:0/3655965241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cf81a2d60 con 0x7f8cf8105bd0 2024-04-22T23:30:27.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.936+0000 7f8ceffff700 1 -- 172.21.15.67:0/3655965241 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ce0017610 con 0x7f8cf8105bd0 2024-04-22T23:30:27.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.937+0000 7f8cfd185700 1 -- 172.21.15.67:0/3655965241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cf804f2e0 con 0x7f8cf8105bd0 2024-04-22T23:30:27.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.938+0000 7f8ceffff700 1 -- 172.21.15.67:0/3655965241 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8ce000f5d0 con 0x7f8cf8105bd0 2024-04-22T23:30:27.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.938+0000 7f8ceffff700 1 --2- 172.21.15.67:0/3655965241 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8ce406ecc0 0x7f8ce4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:30:27.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.938+0000 7f8ceffff700 1 -- 172.21.15.67:0/3655965241 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6704+0+0 (secure 0 0 0) 0x7f8ce0090460 con 0x7f8cf8105bd0 2024-04-22T23:30:27.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.938+0000 7f8cf6d9d700 1 --2- 172.21.15.67:0/3655965241 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8ce406ecc0 0x7f8ce4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:30:27.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.939+0000 7f8cf6d9d700 1 --2- 172.21.15.67:0/3655965241 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8ce406ecc0 0x7f8ce4071180 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8005950 tx=0x7f8ce800a300 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:30:27.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:27.943+0000 7f8ceffff700 1 -- 172.21.15.67:0/3655965241 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f8ce0059150 con 0x7f8cf8105bd0 2024-04-22T23:30:28.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:28.261+0000 7f8cfd185700 1 -- 172.21.15.67:0/3655965241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8cf81a3090 con 0x7f8cf8105bd0 2024-04-22T23:30:28.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:28.262+0000 7f8ceffff700 1 -- 172.21.15.67:0/3655965241 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f8ce005c810 con 0x7f8cf8105bd0 2024-04-22T23:30:28.261 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T23:30:28.261 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:30:28.261 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:30:28.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:28.264+0000 7f8cfd185700 1 -- 172.21.15.67:0/3655965241 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8ce406ecc0 msgr2=0x7f8ce4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:28.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:28.264+0000 7f8cfd185700 1 --2- 172.21.15.67:0/3655965241 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8ce406ecc0 0x7f8ce4071180 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8005950 tx=0x7f8ce800a300 comp rx=0 tx=0).stop 2024-04-22T23:30:28.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:28.264+0000 7f8cfd185700 1 -- 172.21.15.67:0/3655965241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cf8105bd0 msgr2=0x7f8cf8101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:28.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:28.264+0000 7f8cfd185700 1 --2- 172.21.15.67:0/3655965241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cf8105bd0 0x7f8cf8101d80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8ce0006010 tx=0x7f8ce000bd70 comp rx=0 tx=0).stop 2024-04-22T23:30:28.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:28.264+0000 7f8cfd185700 1 -- 172.21.15.67:0/3655965241 shutdown_connections 2024-04-22T23:30:28.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:28.264+0000 7f8cfd185700 1 --2- 172.21.15.67:0/3655965241 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8ce406ecc0 0x7f8ce4071180 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:28.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:28.264+0000 7f8cfd185700 1 --2- 172.21.15.67:0/3655965241 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8cf80ffc20 0x7f8cf8101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:28.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:28.264+0000 7f8cfd185700 1 --2- 172.21.15.67:0/3655965241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cf8105bd0 0x7f8cf8101d80 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:28.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:28.264+0000 7f8cfd185700 1 -- 172.21.15.67:0/3655965241 >> 172.21.15.67:0/3655965241 conn(0x7f8cf8075670 msgr2=0x7f8cf8068680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:30:28.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:28.265+0000 7f8cfd185700 1 -- 172.21.15.67:0/3655965241 shutdown_connections 2024-04-22T23:30:28.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:28.265+0000 7f8cfd185700 1 -- 172.21.15.67:0/3655965241 wait complete. 2024-04-22T23:30:28.463 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:28 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1571921243' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:30:28.463 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:28 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3073361697' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:30:28.463 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:28 smithi067 ceph-mon[112438]: from='client.34272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:30:28.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:28 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1571921243' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:30:28.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:28 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3073361697' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:30:28.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:28 smithi134 ceph-mon[85630]: from='client.34272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:30:28.715 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:28 smithi067 systemd[1]: Stopping Ceph osd.2 for d46c346e-00fd-11ef-bc93-c7b262605968... 2024-04-22T23:30:29.041 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:28 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2[60972]: 2024-04-22T23:30:28.715+0000 7f93a4cba700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T23:30:29.041 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:28 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2[60972]: 2024-04-22T23:30:28.715+0000 7f93a4cba700 -1 osd.2 67 *** Got signal Terminated *** 2024-04-22T23:30:29.041 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:28 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2[60972]: 2024-04-22T23:30:28.715+0000 7f93a4cba700 -1 osd.2 67 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T23:30:29.429 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:29 smithi067 ceph-mon[112438]: pgmap v97: 65 pgs: 65 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:30:29.429 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:29 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3655965241' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:30:29.429 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:29 smithi067 ceph-mon[112438]: osd.2 marked itself down and dead 2024-04-22T23:30:29.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:29 smithi134 ceph-mon[85630]: pgmap v97: 65 pgs: 65 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:30:29.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:29 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3655965241' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:30:29.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:29 smithi134 ceph-mon[85630]: osd.2 marked itself down and dead 2024-04-22T23:30:29.747 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:29 smithi067 podman[135047]: 2024-04-22 23:30:29.432453182 +0000 UTC m=+0.819384219 container died e257ecb4efa1374dcc2e1b42264a5e3ca1d92f5bcdb5b82aa8df927245037cfd (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-22T23:30:30.041 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:29 smithi067 podman[135047]: 2024-04-22 23:30:29.75035669 +0000 UTC m=+1.137287726 container cleanup e257ecb4efa1374dcc2e1b42264a5e3ca1d92f5bcdb5b82aa8df927245037cfd (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_CLEAN=True, RELEASE=HEAD, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T23:30:30.041 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:29 smithi067 bash[135047]: ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2 2024-04-22T23:30:30.041 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:29 smithi067 podman[135066]: 2024-04-22 23:30:29.977175416 +0000 UTC m=+0.540562000 container remove e257ecb4efa1374dcc2e1b42264a5e3ca1d92f5bcdb5b82aa8df927245037cfd (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.29.1, RELEASE=HEAD) 2024-04-22T23:30:30.445 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:30 smithi067 ceph-mon[112438]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-22T23:30:30.446 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:30 smithi067 ceph-mon[112438]: osdmap e68: 6 total, 5 up, 6 in 2024-04-22T23:30:30.446 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:30 smithi067 ceph-mon[112438]: osdmap e69: 6 total, 5 up, 6 in 2024-04-22T23:30:30.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:30 smithi134 ceph-mon[85630]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-22T23:30:30.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:30 smithi134 ceph-mon[85630]: osdmap e68: 6 total, 5 up, 6 in 2024-04-22T23:30:30.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:30 smithi134 ceph-mon[85630]: osdmap e69: 6 total, 5 up, 6 in 2024-04-22T23:30:30.719 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:30 smithi067 podman[135160]: 2024-04-22 23:30:30.352576462 +0000 UTC m=+0.018832317 image pull a0434699c1d76af4eea1fc95ff0a22ea73d1e04e609464fbf658d6f39bc7f2fc quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c 2024-04-22T23:30:30.719 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:30 smithi067 podman[135160]: 2024-04-22 23:30:30.461329522 +0000 UTC m=+0.127585380 container create ca7f3546516985559d9f84c955be216b88fc3f6b73f24a4cbd289ace14867bb2 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2-deactivate, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T23:30:30.719 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:30 smithi067 podman[135160]: 2024-04-22 23:30:30.647152566 +0000 UTC m=+0.313408421 container init ca7f3546516985559d9f84c955be216b88fc3f6b73f24a4cbd289ace14867bb2 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, ceph=True, org.label-schema.schema-version=1.0) 2024-04-22T23:30:30.719 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:30 smithi067 podman[135160]: 2024-04-22 23:30:30.650461475 +0000 UTC m=+0.316717339 container start ca7f3546516985559d9f84c955be216b88fc3f6b73f24a4cbd289ace14867bb2 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-22T23:30:30.719 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:30 smithi067 podman[135160]: 2024-04-22 23:30:30.722150214 +0000 UTC m=+0.388406070 container attach ca7f3546516985559d9f84c955be216b88fc3f6b73f24a4cbd289ace14867bb2 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-22T23:30:31.041 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:30 smithi067 podman[135160]: 2024-04-22 23:30:30.968771335 +0000 UTC m=+0.635027198 container died ca7f3546516985559d9f84c955be216b88fc3f6b73f24a4cbd289ace14867bb2 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-22T23:30:31.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:31 smithi134 ceph-mon[85630]: pgmap v99: 65 pgs: 9 stale+active+clean, 56 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-22T23:30:31.505 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:31 smithi067 ceph-mon[112438]: pgmap v99: 65 pgs: 9 stale+active+clean, 56 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-22T23:30:31.791 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:31 smithi067 podman[135186]: 2024-04-22 23:30:31.509069446 +0000 UTC m=+0.533849458 container remove ca7f3546516985559d9f84c955be216b88fc3f6b73f24a4cbd289ace14867bb2 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240416, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T23:30:31.792 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:31 smithi067 systemd[1]: ceph-d46c346e-00fd-11ef-bc93-c7b262605968@osd.2.service: Deactivated successfully. 2024-04-22T23:30:31.792 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:31 smithi067 systemd[1]: Stopped Ceph osd.2 for d46c346e-00fd-11ef-bc93-c7b262605968. 2024-04-22T23:30:31.792 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:31 smithi067 systemd[1]: ceph-d46c346e-00fd-11ef-bc93-c7b262605968@osd.2.service: Consumed 49.332s CPU time. 2024-04-22T23:30:31.792 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:31 smithi067 systemd[1]: Starting Ceph osd.2 for d46c346e-00fd-11ef-bc93-c7b262605968... 2024-04-22T23:30:32.218 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:32 smithi067 ceph-mon[112438]: Health check failed: Degraded data redundancy: 20/363 objects degraded (5.510%), 6 pgs degraded (PG_DEGRADED) 2024-04-22T23:30:32.218 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:31 smithi067 podman[135289]: 2024-04-22 23:30:31.861571922 +0000 UTC m=+0.018619174 image pull a0434699c1d76af4eea1fc95ff0a22ea73d1e04e609464fbf658d6f39bc7f2fc quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c 2024-04-22T23:30:32.218 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:31 smithi067 podman[135289]: 2024-04-22 23:30:31.960754755 +0000 UTC m=+0.117802049 container create 918e1e62b257a712fb4840741d2d26b11f0018ded23586e3f34482513da11c1e (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2-activate, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-22T23:30:32.218 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:32 smithi067 podman[135289]: 2024-04-22 23:30:32.22175408 +0000 UTC m=+0.378801315 container init 918e1e62b257a712fb4840741d2d26b11f0018ded23586e3f34482513da11c1e (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2-activate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-22T23:30:32.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:32 smithi134 ceph-mon[85630]: Health check failed: Degraded data redundancy: 20/363 objects degraded (5.510%), 6 pgs degraded (PG_DEGRADED) 2024-04-22T23:30:32.541 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:32 smithi067 podman[135289]: 2024-04-22 23:30:32.22501637 +0000 UTC m=+0.382063603 container start 918e1e62b257a712fb4840741d2d26b11f0018ded23586e3f34482513da11c1e (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2-activate, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T23:30:32.541 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:32 smithi067 podman[135289]: 2024-04-22 23:30:32.305173528 +0000 UTC m=+0.462220761 container attach 918e1e62b257a712fb4840741d2d26b11f0018ded23586e3f34482513da11c1e (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2-activate, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416) 2024-04-22T23:30:33.042 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:32 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2-activate[135304]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T23:30:33.042 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:32 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2-activate[135304]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T23:30:33.042 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:32 smithi067 bash[135289]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T23:30:33.042 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:32 smithi067 bash[135289]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T23:30:33.042 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:32 smithi067 bash[135289]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-22T23:30:33.042 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:32 smithi067 bash[135289]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-22T23:30:33.043 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:32 smithi067 bash[135289]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-22T23:30:33.043 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:32 smithi067 bash[135289]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-22T23:30:33.043 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:32 smithi067 bash[135289]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-22T23:30:33.043 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:32 smithi067 bash[135289]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-22T23:30:33.043 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:32 smithi067 bash[135289]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-22T23:30:33.043 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:32 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2-activate[135304]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-22T23:30:33.043 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:32 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2-activate[135304]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-22T23:30:33.043 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:32 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2-activate[135304]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-22T23:30:33.043 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:32 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2-activate[135304]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-22T23:30:33.043 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:32 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2-activate[135304]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-22T23:30:33.043 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:32 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2-activate[135304]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-22T23:30:33.043 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:32 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2-activate[135304]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-22T23:30:33.043 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:32 smithi067 podman[135289]: 2024-04-22 23:30:32.748447704 +0000 UTC m=+0.905494940 container died 918e1e62b257a712fb4840741d2d26b11f0018ded23586e3f34482513da11c1e (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2-activate, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-22T23:30:33.448 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:33 smithi067 ceph-mon[112438]: pgmap v101: 65 pgs: 6 active+undersized+degraded, 8 stale+active+clean, 6 active+undersized, 45 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s; 20/363 objects degraded (5.510%) 2024-04-22T23:30:33.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:33 smithi134 ceph-mon[85630]: pgmap v101: 65 pgs: 6 active+undersized+degraded, 8 stale+active+clean, 6 active+undersized, 45 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s; 20/363 objects degraded (5.510%) 2024-04-22T23:30:33.780 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:33 smithi067 podman[135435]: 2024-04-22 23:30:33.451587207 +0000 UTC m=+0.697204599 container remove 918e1e62b257a712fb4840741d2d26b11f0018ded23586e3f34482513da11c1e (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2-activate, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.license=GPLv2) 2024-04-22T23:30:33.780 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:33 smithi067 podman[135500]: 2024-04-22 23:30:33.669207142 +0000 UTC m=+0.112737705 container create 76df38d881d7d98b46ac93191d7249eed8201ea900c90a4b5df239e9055803a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-22T23:30:33.780 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:33 smithi067 podman[135500]: 2024-04-22 23:30:33.575215847 +0000 UTC m=+0.018746365 image pull a0434699c1d76af4eea1fc95ff0a22ea73d1e04e609464fbf658d6f39bc7f2fc quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c 2024-04-22T23:30:34.291 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:33 smithi067 podman[135500]: 2024-04-22 23:30:33.988427825 +0000 UTC m=+0.431958353 container init 76df38d881d7d98b46ac93191d7249eed8201ea900c90a4b5df239e9055803a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T23:30:34.291 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:33 smithi067 podman[135500]: 2024-04-22 23:30:33.991771051 +0000 UTC m=+0.435301570 container start 76df38d881d7d98b46ac93191d7249eed8201ea900c90a4b5df239e9055803a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-22T23:30:34.291 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:34 smithi067 ceph-osd[135519]: -- 172.21.15.67:0/1307619636 <== mon.0 v2:172.21.15.67:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a93cc2f680 con 0x55a93cbf9000 2024-04-22T23:30:34.291 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:34 smithi067 bash[135500]: 76df38d881d7d98b46ac93191d7249eed8201ea900c90a4b5df239e9055803a3 2024-04-22T23:30:34.291 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:34 smithi067 systemd[1]: Started Ceph osd.2 for d46c346e-00fd-11ef-bc93-c7b262605968. 2024-04-22T23:30:34.791 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:34 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2[135515]: 2024-04-22T23:30:34.535+0000 7fae02fe77c0 -1 Falling back to public interface 2024-04-22T23:30:35.221 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:35 smithi067 ceph-mon[112438]: pgmap v102: 65 pgs: 10 active+undersized+degraded, 5 stale+active+clean, 11 active+undersized, 39 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 35/363 objects degraded (9.642%) 2024-04-22T23:30:35.222 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:35 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:35.222 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:35 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:35.222 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:35 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:30:35.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:35 smithi134 ceph-mon[85630]: pgmap v102: 65 pgs: 10 active+undersized+degraded, 5 stale+active+clean, 11 active+undersized, 39 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 35/363 objects degraded (9.642%) 2024-04-22T23:30:35.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:35 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:35.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:35 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:35.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:35 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:30:37.390 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:37 smithi067 ceph-mon[112438]: pgmap v103: 65 pgs: 13 active+undersized+degraded, 16 active+undersized, 36 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 48/363 objects degraded (13.223%) 2024-04-22T23:30:37.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:37 smithi134 ceph-mon[85630]: pgmap v103: 65 pgs: 13 active+undersized+degraded, 16 active+undersized, 36 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 48/363 objects degraded (13.223%) 2024-04-22T23:30:38.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:38 smithi067 ceph-mon[112438]: Health check update: Degraded data redundancy: 48/363 objects degraded (13.223%), 13 pgs degraded (PG_DEGRADED) 2024-04-22T23:30:38.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:38 smithi134 ceph-mon[85630]: Health check update: Degraded data redundancy: 48/363 objects degraded (13.223%), 13 pgs degraded (PG_DEGRADED) 2024-04-22T23:30:38.751 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:38 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2[135515]: 2024-04-22T23:30:38.460+0000 7fae02fe77c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-04-22T23:30:39.041 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:38 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2[135515]: 2024-04-22T23:30:38.781+0000 7fae02fe77c0 -1 osd.2 67 log_to_monitors true 2024-04-22T23:30:39.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:39 smithi134 ceph-mon[85630]: pgmap v104: 65 pgs: 13 active+undersized+degraded, 16 active+undersized, 36 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 48/363 objects degraded (13.223%) 2024-04-22T23:30:39.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:39 smithi134 ceph-mon[85630]: from='osd.2 [v2:172.21.15.67:6818/121705471,v1:172.21.15.67:6819/121705471]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T23:30:39.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:39 smithi067 ceph-mon[112438]: pgmap v104: 65 pgs: 13 active+undersized+degraded, 16 active+undersized, 36 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 48/363 objects degraded (13.223%) 2024-04-22T23:30:39.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:39 smithi067 ceph-mon[112438]: from='osd.2 [v2:172.21.15.67:6818/121705471,v1:172.21.15.67:6819/121705471]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T23:30:39.541 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 22 23:30:39 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2[135515]: 2024-04-22T23:30:39.156+0000 7fadf443f700 -1 osd.2 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T23:30:40.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:40 smithi134 ceph-mon[85630]: from='osd.2 [v2:172.21.15.67:6818/121705471,v1:172.21.15.67:6819/121705471]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T23:30:40.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:40 smithi134 ceph-mon[85630]: osdmap e70: 6 total, 5 up, 6 in 2024-04-22T23:30:40.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:40 smithi134 ceph-mon[85630]: from='osd.2 [v2:172.21.15.67:6818/121705471,v1:172.21.15.67:6819/121705471]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-04-22T23:30:40.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:40 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:40.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:40 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:40.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:40 smithi067 ceph-mon[112438]: from='osd.2 [v2:172.21.15.67:6818/121705471,v1:172.21.15.67:6819/121705471]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T23:30:40.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:40 smithi067 ceph-mon[112438]: osdmap e70: 6 total, 5 up, 6 in 2024-04-22T23:30:40.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:40 smithi067 ceph-mon[112438]: from='osd.2 [v2:172.21.15.67:6818/121705471,v1:172.21.15.67:6819/121705471]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-04-22T23:30:40.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:40 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:40.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:40 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:41.134 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:41 smithi067 ceph-mon[112438]: pgmap v106: 65 pgs: 13 active+undersized+degraded, 16 active+undersized, 36 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 48/363 objects degraded (13.223%) 2024-04-22T23:30:41.134 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:41 smithi067 ceph-mon[112438]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-22T23:30:41.134 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:41 smithi067 ceph-mon[112438]: osd.2 [v2:172.21.15.67:6818/121705471,v1:172.21.15.67:6819/121705471] boot 2024-04-22T23:30:41.134 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:41 smithi067 ceph-mon[112438]: osdmap e71: 6 total, 6 up, 6 in 2024-04-22T23:30:41.134 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T23:30:41.134 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:41.134 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:41.135 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:41.135 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:30:41.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:41 smithi134 ceph-mon[85630]: pgmap v106: 65 pgs: 13 active+undersized+degraded, 16 active+undersized, 36 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 48/363 objects degraded (13.223%) 2024-04-22T23:30:41.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:41 smithi134 ceph-mon[85630]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-22T23:30:41.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:41 smithi134 ceph-mon[85630]: osd.2 [v2:172.21.15.67:6818/121705471,v1:172.21.15.67:6819/121705471] boot 2024-04-22T23:30:41.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:41 smithi134 ceph-mon[85630]: osdmap e71: 6 total, 6 up, 6 in 2024-04-22T23:30:41.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T23:30:41.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:41.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:41.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:41.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:30:42.392 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:42 smithi134 ceph-mon[85630]: osdmap e72: 6 total, 6 up, 6 in 2024-04-22T23:30:42.431 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:42 smithi067 ceph-mon[112438]: osdmap e72: 6 total, 6 up, 6 in 2024-04-22T23:30:43.413 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:43 smithi067 ceph-mon[112438]: pgmap v109: 65 pgs: 7 active+undersized+degraded, 11 active+undersized, 47 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 3.2 KiB/s rd, 4 op/s; 28/363 objects degraded (7.713%) 2024-04-22T23:30:43.413 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:43 smithi067 ceph-mon[112438]: Health check update: Degraded data redundancy: 28/363 objects degraded (7.713%), 7 pgs degraded (PG_DEGRADED) 2024-04-22T23:30:43.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:43 smithi134 ceph-mon[85630]: pgmap v109: 65 pgs: 7 active+undersized+degraded, 11 active+undersized, 47 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 3.2 KiB/s rd, 4 op/s; 28/363 objects degraded (7.713%) 2024-04-22T23:30:43.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:43 smithi134 ceph-mon[85630]: Health check update: Degraded data redundancy: 28/363 objects degraded (7.713%), 7 pgs degraded (PG_DEGRADED) 2024-04-22T23:30:45.493 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:45 smithi134 ceph-mon[85630]: pgmap v110: 65 pgs: 5 active+undersized+degraded, 7 active+undersized, 53 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s; 18/363 objects degraded (4.959%) 2024-04-22T23:30:45.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:45 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:45.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:45 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:45.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:45 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:30:45.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:45 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:30:45.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:45 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:45.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:45 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:30:45.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:45 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:30:45.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:45 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:30:45.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:45 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:30:45.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:45 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-22T23:30:45.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:45 smithi134 ceph-mon[85630]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-22T23:30:45.494 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:45 smithi134 ceph-mon[85630]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-04-22T23:30:45.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:45 smithi067 ceph-mon[112438]: pgmap v110: 65 pgs: 5 active+undersized+degraded, 7 active+undersized, 53 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s; 18/363 objects degraded (4.959%) 2024-04-22T23:30:45.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:45 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:45.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:45 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:45.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:45 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:30:45.542 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:45 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:30:45.542 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:45 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:45.542 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:45 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:30:45.542 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:45 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:30:45.542 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:45 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:30:45.542 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:45 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:30:45.542 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:45 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-22T23:30:45.542 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:45 smithi067 ceph-mon[112438]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-22T23:30:45.542 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:45 smithi067 ceph-mon[112438]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-04-22T23:30:46.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:46 smithi067 ceph-mon[112438]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 18/363 objects degraded (4.959%), 5 pgs degraded) 2024-04-22T23:30:46.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:46 smithi134 ceph-mon[85630]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 18/363 objects degraded (4.959%), 5 pgs degraded) 2024-04-22T23:30:47.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:47 smithi134 ceph-mon[85630]: pgmap v111: 65 pgs: 65 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s 2024-04-22T23:30:47.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:47 smithi067 ceph-mon[112438]: pgmap v111: 65 pgs: 65 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s 2024-04-22T23:30:49.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:49 smithi134 ceph-mon[85630]: pgmap v112: 65 pgs: 65 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 3 op/s 2024-04-22T23:30:49.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:49 smithi067 ceph-mon[112438]: pgmap v112: 65 pgs: 65 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 3 op/s 2024-04-22T23:30:51.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:51 smithi067 ceph-mon[112438]: pgmap v113: 65 pgs: 65 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-22T23:30:51.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:51 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:51.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:51 smithi134 ceph-mon[85630]: pgmap v113: 65 pgs: 65 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-22T23:30:51.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:51 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:53.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:53 smithi134 ceph-mon[85630]: pgmap v114: 65 pgs: 65 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 4 op/s 2024-04-22T23:30:53.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:53 smithi067 ceph-mon[112438]: pgmap v114: 65 pgs: 65 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 4 op/s 2024-04-22T23:30:55.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:55 smithi134 ceph-mon[85630]: pgmap v115: 65 pgs: 65 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:30:55.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:55 smithi067 ceph-mon[112438]: pgmap v115: 65 pgs: 65 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:30:57.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:57 smithi067 ceph-mon[112438]: pgmap v116: 65 pgs: 65 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:30:57.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:57 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:57.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:57 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:30:57.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:57 smithi134 ceph-mon[85630]: pgmap v116: 65 pgs: 65 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:30:57.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:57 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:30:57.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:57 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:30:58.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.389+0000 7fc9e727a700 1 -- 172.21.15.67:0/65666781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9e0100750 msgr2=0x7fc9e01047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:58.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.389+0000 7fc9e727a700 1 --2- 172.21.15.67:0/65666781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9e0100750 0x7fc9e01047a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc009910 tx=0x7fc9dc009c20 comp rx=0 tx=0).stop 2024-04-22T23:30:58.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.389+0000 7fc9e727a700 1 -- 172.21.15.67:0/65666781 shutdown_connections 2024-04-22T23:30:58.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.389+0000 7fc9e727a700 1 --2- 172.21.15.67:0/65666781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9e0100750 0x7fc9e01047a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:58.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.389+0000 7fc9e727a700 1 --2- 172.21.15.67:0/65666781 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9e00ffda0 0x7fc9e0100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:58.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.389+0000 7fc9e727a700 1 -- 172.21.15.67:0/65666781 >> 172.21.15.67:0/65666781 conn(0x7fc9e00fb650 msgr2=0x7fc9e00fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:30:58.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.389+0000 7fc9e727a700 1 -- 172.21.15.67:0/65666781 shutdown_connections 2024-04-22T23:30:58.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.389+0000 7fc9e727a700 1 -- 172.21.15.67:0/65666781 wait complete. 2024-04-22T23:30:58.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.390+0000 7fc9e727a700 1 Processor -- start 2024-04-22T23:30:58.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.390+0000 7fc9e727a700 1 -- start start 2024-04-22T23:30:58.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.390+0000 7fc9e727a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9e00ffda0 0x7fc9e019c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:30:58.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.390+0000 7fc9e727a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9e0100750 0x7fc9e019cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:30:58.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.390+0000 7fc9e727a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e019d250 con 0x7fc9e00ffda0 2024-04-22T23:30:58.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.390+0000 7fc9e727a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e0196700 con 0x7fc9e0100750 2024-04-22T23:30:58.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.390+0000 7fc9e5016700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9e00ffda0 0x7fc9e019c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:30:58.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.390+0000 7fc9e5016700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9e00ffda0 0x7fc9e019c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36020/0 (socket says 172.21.15.67:36020) 2024-04-22T23:30:58.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.390+0000 7fc9e5016700 1 -- 172.21.15.67:0/938426895 learned_addr learned my addr 172.21.15.67:0/938426895 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:30:58.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.390+0000 7fc9e4815700 1 --2- 172.21.15.67:0/938426895 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9e0100750 0x7fc9e019cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:30:58.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.391+0000 7fc9e5016700 1 -- 172.21.15.67:0/938426895 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9e0100750 msgr2=0x7fc9e019cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:58.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.391+0000 7fc9e5016700 1 --2- 172.21.15.67:0/938426895 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9e0100750 0x7fc9e019cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:58.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.391+0000 7fc9e5016700 1 -- 172.21.15.67:0/938426895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9dc0095f0 con 0x7fc9e00ffda0 2024-04-22T23:30:58.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.391+0000 7fc9e5016700 1 --2- 172.21.15.67:0/938426895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9e00ffda0 0x7fc9e019c680 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc9d000d8d0 tx=0x7fc9d000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:30:58.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.391+0000 7fc9d67fc700 1 -- 172.21.15.67:0/938426895 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9d00098a0 con 0x7fc9e00ffda0 2024-04-22T23:30:58.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.391+0000 7fc9d67fc700 1 -- 172.21.15.67:0/938426895 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc9d0010460 con 0x7fc9e00ffda0 2024-04-22T23:30:58.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.391+0000 7fc9e727a700 1 -- 172.21.15.67:0/938426895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9e01969e0 con 0x7fc9e00ffda0 2024-04-22T23:30:58.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.391+0000 7fc9d67fc700 1 -- 172.21.15.67:0/938426895 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9d000f5f0 con 0x7fc9e00ffda0 2024-04-22T23:30:58.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.391+0000 7fc9e727a700 1 -- 172.21.15.67:0/938426895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9e0196f30 con 0x7fc9e00ffda0 2024-04-22T23:30:58.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.392+0000 7fc9e727a700 1 -- 172.21.15.67:0/938426895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9e004f2e0 con 0x7fc9e00ffda0 2024-04-22T23:30:58.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.393+0000 7fc9d67fc700 1 -- 172.21.15.67:0/938426895 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc9d0009a00 con 0x7fc9e00ffda0 2024-04-22T23:30:58.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.393+0000 7fc9d67fc700 1 --2- 172.21.15.67:0/938426895 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc9cc06ecc0 0x7fc9cc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:30:58.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.393+0000 7fc9d67fc700 1 -- 172.21.15.67:0/938426895 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6704+0+0 (secure 0 0 0) 0x7fc9d008ecf0 con 0x7fc9e00ffda0 2024-04-22T23:30:58.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.393+0000 7fc9e4815700 1 --2- 172.21.15.67:0/938426895 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc9cc06ecc0 0x7fc9cc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:30:58.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.394+0000 7fc9e4815700 1 --2- 172.21.15.67:0/938426895 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc9cc06ecc0 0x7fc9cc071180 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0197b70 tx=0x7fc9dc01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:30:58.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.398+0000 7fc9d67fc700 1 -- 172.21.15.67:0/938426895 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc9d005b590 con 0x7fc9e00ffda0 2024-04-22T23:30:58.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.626+0000 7fc9e727a700 1 -- 172.21.15.67:0/938426895 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc9e0197240 con 0x7fc9cc06ecc0 2024-04-22T23:30:58.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.628+0000 7fc9d67fc700 1 -- 172.21.15.67:0/938426895 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fc9e0197240 con 0x7fc9cc06ecc0 2024-04-22T23:30:58.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.630+0000 7fc9e727a700 1 -- 172.21.15.67:0/938426895 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc9cc06ecc0 msgr2=0x7fc9cc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:58.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.630+0000 7fc9e727a700 1 --2- 172.21.15.67:0/938426895 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc9cc06ecc0 0x7fc9cc071180 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0197b70 tx=0x7fc9dc01a040 comp rx=0 tx=0).stop 2024-04-22T23:30:58.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.630+0000 7fc9e727a700 1 -- 172.21.15.67:0/938426895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9e00ffda0 msgr2=0x7fc9e019c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:58.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.630+0000 7fc9e727a700 1 --2- 172.21.15.67:0/938426895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9e00ffda0 0x7fc9e019c680 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc9d000d8d0 tx=0x7fc9d000dbe0 comp rx=0 tx=0).stop 2024-04-22T23:30:58.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.630+0000 7fc9e727a700 1 -- 172.21.15.67:0/938426895 shutdown_connections 2024-04-22T23:30:58.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.630+0000 7fc9e727a700 1 --2- 172.21.15.67:0/938426895 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc9cc06ecc0 0x7fc9cc071180 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:58.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.630+0000 7fc9e727a700 1 --2- 172.21.15.67:0/938426895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9e00ffda0 0x7fc9e019c680 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:58.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.630+0000 7fc9e727a700 1 --2- 172.21.15.67:0/938426895 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9e0100750 0x7fc9e019cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:58.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.630+0000 7fc9e727a700 1 -- 172.21.15.67:0/938426895 >> 172.21.15.67:0/938426895 conn(0x7fc9e00fb650 msgr2=0x7fc9e0103f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:30:58.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.630+0000 7fc9e727a700 1 -- 172.21.15.67:0/938426895 shutdown_connections 2024-04-22T23:30:58.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.630+0000 7fc9e727a700 1 -- 172.21.15.67:0/938426895 wait complete. 2024-04-22T23:30:58.638 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:30:58.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.741+0000 7fd245a61700 1 -- 172.21.15.67:0/179426937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd240101850 msgr2=0x7fd2401058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:58.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.741+0000 7fd245a61700 1 --2- 172.21.15.67:0/179426937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd240101850 0x7fd2401058a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd234009910 tx=0x7fd234009c20 comp rx=0 tx=0).stop 2024-04-22T23:30:58.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.741+0000 7fd245a61700 1 -- 172.21.15.67:0/179426937 shutdown_connections 2024-04-22T23:30:58.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.741+0000 7fd245a61700 1 --2- 172.21.15.67:0/179426937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd240101850 0x7fd2401058a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:58.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.741+0000 7fd245a61700 1 --2- 172.21.15.67:0/179426937 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd240100ea0 0x7fd240101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:58.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.741+0000 7fd245a61700 1 -- 172.21.15.67:0/179426937 >> 172.21.15.67:0/179426937 conn(0x7fd2400fc750 msgr2=0x7fd2400feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:30:58.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.741+0000 7fd245a61700 1 -- 172.21.15.67:0/179426937 shutdown_connections 2024-04-22T23:30:58.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.741+0000 7fd245a61700 1 -- 172.21.15.67:0/179426937 wait complete. 2024-04-22T23:30:58.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.742+0000 7fd245a61700 1 Processor -- start 2024-04-22T23:30:58.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.742+0000 7fd245a61700 1 -- start start 2024-04-22T23:30:58.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.742+0000 7fd245a61700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd240100ea0 0x7fd24019c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:30:58.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.742+0000 7fd245a61700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd240101850 0x7fd24019cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:30:58.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.742+0000 7fd245a61700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24019d2d0 con 0x7fd240101850 2024-04-22T23:30:58.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.742+0000 7fd245a61700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd240196780 con 0x7fd240100ea0 2024-04-22T23:30:58.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.742+0000 7fd23e7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd240101850 0x7fd24019cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:30:58.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.742+0000 7fd23e7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd240101850 0x7fd24019cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52158/0 (socket says 172.21.15.67:52158) 2024-04-22T23:30:58.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.742+0000 7fd23e7fc700 1 -- 172.21.15.67:0/1593228927 learned_addr learned my addr 172.21.15.67:0/1593228927 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:30:58.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.742+0000 7fd23effd700 1 --2- 172.21.15.67:0/1593228927 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd240100ea0 0x7fd24019c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:30:58.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.743+0000 7fd23e7fc700 1 -- 172.21.15.67:0/1593228927 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd240100ea0 msgr2=0x7fd24019c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:58.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.743+0000 7fd23e7fc700 1 --2- 172.21.15.67:0/1593228927 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd240100ea0 0x7fd24019c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:58.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.743+0000 7fd23e7fc700 1 -- 172.21.15.67:0/1593228927 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2340095f0 con 0x7fd240101850 2024-04-22T23:30:58.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.743+0000 7fd23effd700 1 --2- 172.21.15.67:0/1593228927 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd240100ea0 0x7fd24019c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:30:58.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.743+0000 7fd23e7fc700 1 --2- 172.21.15.67:0/1593228927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd240101850 0x7fd24019cc40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd2340059d0 tx=0x7fd23400f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:30:58.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.743+0000 7fd244a5f700 1 -- 172.21.15.67:0/1593228927 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd23401d070 con 0x7fd240101850 2024-04-22T23:30:58.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.743+0000 7fd244a5f700 1 -- 172.21.15.67:0/1593228927 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd23400b610 con 0x7fd240101850 2024-04-22T23:30:58.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.743+0000 7fd245a61700 1 -- 172.21.15.67:0/1593228927 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd240196a00 con 0x7fd240101850 2024-04-22T23:30:58.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.743+0000 7fd244a5f700 1 -- 172.21.15.67:0/1593228927 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd234022470 con 0x7fd240101850 2024-04-22T23:30:58.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.743+0000 7fd245a61700 1 -- 172.21.15.67:0/1593228927 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd240196ef0 con 0x7fd240101850 2024-04-22T23:30:58.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.744+0000 7fd245a61700 1 -- 172.21.15.67:0/1593228927 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2401092a0 con 0x7fd240101850 2024-04-22T23:30:58.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.745+0000 7fd244a5f700 1 -- 172.21.15.67:0/1593228927 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd23400fc70 con 0x7fd240101850 2024-04-22T23:30:58.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.745+0000 7fd244a5f700 1 --2- 172.21.15.67:0/1593228927 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd22806ecc0 0x7fd228071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:30:58.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.745+0000 7fd23effd700 1 --2- 172.21.15.67:0/1593228927 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd22806ecc0 0x7fd228071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:30:58.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.745+0000 7fd244a5f700 1 -- 172.21.15.67:0/1593228927 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6704+0+0 (secure 0 0 0) 0x7fd2340903f0 con 0x7fd240101850 2024-04-22T23:30:58.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.746+0000 7fd23effd700 1 --2- 172.21.15.67:0/1593228927 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd22806ecc0 0x7fd228071180 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd230009cb0 tx=0x7fd230009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:30:58.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.750+0000 7fd244a5f700 1 -- 172.21.15.67:0/1593228927 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fd23405cb90 con 0x7fd240101850 2024-04-22T23:30:58.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.980+0000 7fd245a61700 1 -- 172.21.15.67:0/1593228927 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd240197110 con 0x7fd22806ecc0 2024-04-22T23:30:58.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.981+0000 7fd244a5f700 1 -- 172.21.15.67:0/1593228927 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fd240197110 con 0x7fd22806ecc0 2024-04-22T23:30:58.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.983+0000 7fd245a61700 1 -- 172.21.15.67:0/1593228927 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd22806ecc0 msgr2=0x7fd228071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:58.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.983+0000 7fd245a61700 1 --2- 172.21.15.67:0/1593228927 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd22806ecc0 0x7fd228071180 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd230009cb0 tx=0x7fd230009260 comp rx=0 tx=0).stop 2024-04-22T23:30:58.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.983+0000 7fd245a61700 1 -- 172.21.15.67:0/1593228927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd240101850 msgr2=0x7fd24019cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:58.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.983+0000 7fd245a61700 1 --2- 172.21.15.67:0/1593228927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd240101850 0x7fd24019cc40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd2340059d0 tx=0x7fd23400f740 comp rx=0 tx=0).stop 2024-04-22T23:30:58.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.983+0000 7fd245a61700 1 -- 172.21.15.67:0/1593228927 shutdown_connections 2024-04-22T23:30:58.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.983+0000 7fd245a61700 1 --2- 172.21.15.67:0/1593228927 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd22806ecc0 0x7fd228071180 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:58.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.983+0000 7fd245a61700 1 --2- 172.21.15.67:0/1593228927 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd240100ea0 0x7fd24019c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:58.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.983+0000 7fd245a61700 1 --2- 172.21.15.67:0/1593228927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd240101850 0x7fd24019cc40 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:58.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.983+0000 7fd245a61700 1 -- 172.21.15.67:0/1593228927 >> 172.21.15.67:0/1593228927 conn(0x7fd2400fc750 msgr2=0x7fd2400fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:30:58.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.983+0000 7fd245a61700 1 -- 172.21.15.67:0/1593228927 shutdown_connections 2024-04-22T23:30:58.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:58.983+0000 7fd245a61700 1 -- 172.21.15.67:0/1593228927 wait complete. 2024-04-22T23:30:59.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.092+0000 7f6c2001a700 1 -- 172.21.15.67:0/3416958980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c18100ea0 msgr2=0x7f6c18101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:59.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.092+0000 7f6c2001a700 1 --2- 172.21.15.67:0/3416958980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c18100ea0 0x7f6c18101280 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6c0800b0a0 tx=0x7f6c0800b3b0 comp rx=0 tx=0).stop 2024-04-22T23:30:59.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.092+0000 7f6c2001a700 1 -- 172.21.15.67:0/3416958980 shutdown_connections 2024-04-22T23:30:59.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.092+0000 7f6c2001a700 1 --2- 172.21.15.67:0/3416958980 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c18101850 0x7f6c181058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:59.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.092+0000 7f6c2001a700 1 --2- 172.21.15.67:0/3416958980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c18100ea0 0x7f6c18101280 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:59.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.092+0000 7f6c2001a700 1 -- 172.21.15.67:0/3416958980 >> 172.21.15.67:0/3416958980 conn(0x7f6c180fc750 msgr2=0x7f6c180feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:30:59.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.092+0000 7f6c2001a700 1 -- 172.21.15.67:0/3416958980 shutdown_connections 2024-04-22T23:30:59.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.092+0000 7f6c2001a700 1 -- 172.21.15.67:0/3416958980 wait complete. 2024-04-22T23:30:59.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.093+0000 7f6c2001a700 1 Processor -- start 2024-04-22T23:30:59.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.093+0000 7f6c2001a700 1 -- start start 2024-04-22T23:30:59.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.093+0000 7f6c2001a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c18100ea0 0x7f6c18073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:30:59.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.093+0000 7f6c2001a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c18101850 0x7f6c180735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:30:59.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.093+0000 7f6c2001a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c18073b10 con 0x7f6c18100ea0 2024-04-22T23:30:59.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.093+0000 7f6c2001a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c18073c50 con 0x7f6c18101850 2024-04-22T23:30:59.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.093+0000 7f6c1ddb6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c18100ea0 0x7f6c18073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:30:59.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.093+0000 7f6c1ddb6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c18100ea0 0x7f6c18073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52170/0 (socket says 172.21.15.67:52170) 2024-04-22T23:30:59.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.093+0000 7f6c1ddb6700 1 -- 172.21.15.67:0/4268533514 learned_addr learned my addr 172.21.15.67:0/4268533514 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:30:59.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.093+0000 7f6c1d5b5700 1 --2- 172.21.15.67:0/4268533514 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c18101850 0x7f6c180735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:30:59.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.093+0000 7f6c1ddb6700 1 -- 172.21.15.67:0/4268533514 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c18101850 msgr2=0x7f6c180735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:59.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.093+0000 7f6c1ddb6700 1 --2- 172.21.15.67:0/4268533514 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c18101850 0x7f6c180735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:59.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.093+0000 7f6c1ddb6700 1 -- 172.21.15.67:0/4268533514 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c08009d00 con 0x7f6c18100ea0 2024-04-22T23:30:59.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.094+0000 7f6c1ddb6700 1 --2- 172.21.15.67:0/4268533514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c18100ea0 0x7f6c18073090 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6c08000f80 tx=0x7f6c0800cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:30:59.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.094+0000 7f6c16ffd700 1 -- 172.21.15.67:0/4268533514 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c0800e040 con 0x7f6c18100ea0 2024-04-22T23:30:59.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.094+0000 7f6c16ffd700 1 -- 172.21.15.67:0/4268533514 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6c08003c10 con 0x7f6c18100ea0 2024-04-22T23:30:59.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.094+0000 7f6c16ffd700 1 -- 172.21.15.67:0/4268533514 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c0801d6b0 con 0x7f6c18100ea0 2024-04-22T23:30:59.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.094+0000 7f6c2001a700 1 -- 172.21.15.67:0/4268533514 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c18073ed0 con 0x7f6c18100ea0 2024-04-22T23:30:59.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.094+0000 7f6c2001a700 1 -- 172.21.15.67:0/4268533514 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c181a2e20 con 0x7f6c18100ea0 2024-04-22T23:30:59.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.095+0000 7f6c2001a700 1 -- 172.21.15.67:0/4268533514 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c18109240 con 0x7f6c18100ea0 2024-04-22T23:30:59.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.095+0000 7f6c16ffd700 1 -- 172.21.15.67:0/4268533514 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6c08004040 con 0x7f6c18100ea0 2024-04-22T23:30:59.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.096+0000 7f6c16ffd700 1 --2- 172.21.15.67:0/4268533514 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6c0006ecc0 0x7f6c00071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:30:59.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.096+0000 7f6c16ffd700 1 -- 172.21.15.67:0/4268533514 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6704+0+0 (secure 0 0 0) 0x7f6c08090220 con 0x7f6c18100ea0 2024-04-22T23:30:59.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.096+0000 7f6c1d5b5700 1 --2- 172.21.15.67:0/4268533514 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6c0006ecc0 0x7f6c00071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:30:59.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.096+0000 7f6c1d5b5700 1 --2- 172.21.15.67:0/4268533514 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6c0006ecc0 0x7f6c00071180 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6c18074880 tx=0x7f6c0c00b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:30:59.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.101+0000 7f6c16ffd700 1 -- 172.21.15.67:0/4268533514 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f6c08058f10 con 0x7f6c18100ea0 2024-04-22T23:30:59.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:59 smithi067 ceph-mon[112438]: pgmap v117: 65 pgs: 65 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:30:59.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:30:59 smithi067 ceph-mon[112438]: from='client.34284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:30:59.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.323+0000 7f6c2001a700 1 -- 172.21.15.67:0/4268533514 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6c180611d0 con 0x7f6c0006ecc0 2024-04-22T23:30:59.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.334+0000 7f6c16ffd700 1 -- 172.21.15.67:0/4268533514 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f6c180611d0 con 0x7f6c0006ecc0 2024-04-22T23:30:59.331 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:30:59.331 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (4m) 19s ago 15m 26.1M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:30:59.331 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (16m) 19s ago 16m 8728k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:30:59.332 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (15m) 115s ago 15m 8200k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:30:59.332 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (2m) 19s ago 16m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:30:59.332 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (119s) 115s ago 15m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:30:59.332 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (4m) 19s ago 15m 80.1M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:30:59.332 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (10m) 19s ago 10m 62.7M - 18.2.1 5be31c24972a e24a471f0a8c 2024-04-22T23:30:59.332 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (10m) 19s ago 10m 130M - 18.2.1 5be31c24972a 0730f9fc8e61 2024-04-22T23:30:59.332 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (10m) 115s ago 10m 123M - 18.2.1 5be31c24972a b6a169586433 2024-04-22T23:30:59.332 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (10m) 115s ago 10m 67.7M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:30:59.332 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (6m) 19s ago 17m 506M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:30:59.332 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (5m) 115s ago 15m 450M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:30:59.332 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (3m) 19s ago 17m 49.7M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:30:59.332 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (2m) 115s ago 14m 38.2M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:30:59.332 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (5m) 19s ago 16m 15.4M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:30:59.332 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (5m) 115s ago 15m 15.9M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:30:59.332 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (98s) 19s ago 14m 95.4M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:30:59.333 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (61s) 19s ago 13m 131M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:30:59.333 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (25s) 19s ago 13m 15.1M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:30:59.333 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (12m) 115s ago 12m 251M 2240M 18.2.1 5be31c24972a 9c2c91a967ba 2024-04-22T23:30:59.333 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (12m) 115s ago 12m 219M 2240M 18.2.1 5be31c24972a c7586562ffa6 2024-04-22T23:30:59.333 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (12m) 115s ago 12m 196M 2240M 18.2.1 5be31c24972a 9fd97f1c65cb 2024-04-22T23:30:59.333 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (5m) 19s ago 15m 71.3M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:30:59.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.336+0000 7f6c2001a700 1 -- 172.21.15.67:0/4268533514 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6c0006ecc0 msgr2=0x7f6c00071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:59.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.336+0000 7f6c2001a700 1 --2- 172.21.15.67:0/4268533514 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6c0006ecc0 0x7f6c00071180 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6c18074880 tx=0x7f6c0c00b410 comp rx=0 tx=0).stop 2024-04-22T23:30:59.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.336+0000 7f6c2001a700 1 -- 172.21.15.67:0/4268533514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c18100ea0 msgr2=0x7f6c18073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:59.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.336+0000 7f6c2001a700 1 --2- 172.21.15.67:0/4268533514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c18100ea0 0x7f6c18073090 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6c08000f80 tx=0x7f6c0800cd20 comp rx=0 tx=0).stop 2024-04-22T23:30:59.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.336+0000 7f6c2001a700 1 -- 172.21.15.67:0/4268533514 shutdown_connections 2024-04-22T23:30:59.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.336+0000 7f6c2001a700 1 --2- 172.21.15.67:0/4268533514 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6c0006ecc0 0x7f6c00071180 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:59.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.336+0000 7f6c2001a700 1 --2- 172.21.15.67:0/4268533514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c18100ea0 0x7f6c18073090 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:59.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.336+0000 7f6c2001a700 1 --2- 172.21.15.67:0/4268533514 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c18101850 0x7f6c180735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:59.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.336+0000 7f6c2001a700 1 -- 172.21.15.67:0/4268533514 >> 172.21.15.67:0/4268533514 conn(0x7f6c180fc750 msgr2=0x7f6c180fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:30:59.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.336+0000 7f6c2001a700 1 -- 172.21.15.67:0/4268533514 shutdown_connections 2024-04-22T23:30:59.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.336+0000 7f6c2001a700 1 -- 172.21.15.67:0/4268533514 wait complete. 2024-04-22T23:30:59.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.443+0000 7f8dff1e3700 1 -- 172.21.15.67:0/2997693085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8df8101e60 msgr2=0x7f8df8102240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:59.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.443+0000 7f8dff1e3700 1 --2- 172.21.15.67:0/2997693085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8df8101e60 0x7f8df8102240 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8de0009910 tx=0x7f8de0009c20 comp rx=0 tx=0).stop 2024-04-22T23:30:59.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.443+0000 7f8dff1e3700 1 -- 172.21.15.67:0/2997693085 shutdown_connections 2024-04-22T23:30:59.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.443+0000 7f8dff1e3700 1 --2- 172.21.15.67:0/2997693085 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8df8102780 0x7f8df810ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:59.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.443+0000 7f8dff1e3700 1 --2- 172.21.15.67:0/2997693085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8df8101e60 0x7f8df8102240 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:59.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.443+0000 7f8dff1e3700 1 -- 172.21.15.67:0/2997693085 >> 172.21.15.67:0/2997693085 conn(0x7f8df80fb650 msgr2=0x7f8df80fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:30:59.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.443+0000 7f8dff1e3700 1 -- 172.21.15.67:0/2997693085 shutdown_connections 2024-04-22T23:30:59.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.443+0000 7f8dff1e3700 1 -- 172.21.15.67:0/2997693085 wait complete. 2024-04-22T23:30:59.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.444+0000 7f8dff1e3700 1 Processor -- start 2024-04-22T23:30:59.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.444+0000 7f8dff1e3700 1 -- start start 2024-04-22T23:30:59.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.444+0000 7f8dff1e3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8df8101e60 0x7f8df80ffac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:30:59.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.444+0000 7f8dff1e3700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8df8102780 0x7f8df8100000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:30:59.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.444+0000 7f8dff1e3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df8100540 con 0x7f8df8101e60 2024-04-22T23:30:59.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.444+0000 7f8dff1e3700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df8100680 con 0x7f8df8102780 2024-04-22T23:30:59.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.444+0000 7f8dfcf7f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8df8101e60 0x7f8df80ffac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:30:59.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.444+0000 7f8dfcf7f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8df8101e60 0x7f8df80ffac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52194/0 (socket says 172.21.15.67:52194) 2024-04-22T23:30:59.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.444+0000 7f8dfcf7f700 1 -- 172.21.15.67:0/1102742321 learned_addr learned my addr 172.21.15.67:0/1102742321 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:30:59.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.445+0000 7f8df7fff700 1 --2- 172.21.15.67:0/1102742321 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8df8102780 0x7f8df8100000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:30:59.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.445+0000 7f8dfcf7f700 1 -- 172.21.15.67:0/1102742321 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8df8102780 msgr2=0x7f8df8100000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:59.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.445+0000 7f8dfcf7f700 1 --2- 172.21.15.67:0/1102742321 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8df8102780 0x7f8df8100000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:59.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.445+0000 7f8dfcf7f700 1 -- 172.21.15.67:0/1102742321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de00095f0 con 0x7f8df8101e60 2024-04-22T23:30:59.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.445+0000 7f8dfcf7f700 1 --2- 172.21.15.67:0/1102742321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8df8101e60 0x7f8df80ffac0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8de0006010 tx=0x7f8de000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:30:59.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.445+0000 7f8df5ffb700 1 -- 172.21.15.67:0/1102742321 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8de001d070 con 0x7f8df8101e60 2024-04-22T23:30:59.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.445+0000 7f8df5ffb700 1 -- 172.21.15.67:0/1102742321 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8de0005dc0 con 0x7f8df8101e60 2024-04-22T23:30:59.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.445+0000 7f8dff1e3700 1 -- 172.21.15.67:0/1102742321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8df8100900 con 0x7f8df8101e60 2024-04-22T23:30:59.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.445+0000 7f8df5ffb700 1 -- 172.21.15.67:0/1102742321 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8de0017400 con 0x7f8df8101e60 2024-04-22T23:30:59.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.445+0000 7f8dff1e3700 1 -- 172.21.15.67:0/1102742321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8df81a2d80 con 0x7f8df8101e60 2024-04-22T23:30:59.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.446+0000 7f8dff1e3700 1 -- 172.21.15.67:0/1102742321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8df8108310 con 0x7f8df8101e60 2024-04-22T23:30:59.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.447+0000 7f8df5ffb700 1 -- 172.21.15.67:0/1102742321 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8de0003a60 con 0x7f8df8101e60 2024-04-22T23:30:59.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.447+0000 7f8df5ffb700 1 --2- 172.21.15.67:0/1102742321 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8de406ecc0 0x7f8de4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:30:59.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.448+0000 7f8df7fff700 1 --2- 172.21.15.67:0/1102742321 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8de406ecc0 0x7f8de4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:30:59.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.448+0000 7f8df5ffb700 1 -- 172.21.15.67:0/1102742321 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6704+0+0 (secure 0 0 0) 0x7f8de0090270 con 0x7f8df8101e60 2024-04-22T23:30:59.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.448+0000 7f8df7fff700 1 --2- 172.21.15.67:0/1102742321 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8de406ecc0 0x7f8de4071180 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8df8101260 tx=0x7f8de80073d0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:30:59.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.452+0000 7f8df5ffb700 1 -- 172.21.15.67:0/1102742321 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f8de0058fe0 con 0x7f8df8101e60 2024-04-22T23:30:59.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:59 smithi134 ceph-mon[85630]: pgmap v117: 65 pgs: 65 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:30:59.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:30:59 smithi134 ceph-mon[85630]: from='client.34284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:30:59.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.749+0000 7f8dff1e3700 1 -- 172.21.15.67:0/1102742321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8df804f2e0 con 0x7f8df8101e60 2024-04-22T23:30:59.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.750+0000 7f8df5ffb700 1 -- 172.21.15.67:0/1102742321 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7f8de005c6a0 con 0x7f8df8101e60 2024-04-22T23:30:59.747 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:30:59.747 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:30:59.748 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:30:59.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:30:59.748 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:30:59.748 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:30:59.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:30:59.748 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:30:59.748 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 3, 2024-04-22T23:30:59.748 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 3 2024-04-22T23:30:59.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:30:59.748 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:30:59.748 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-22T23:30:59.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:30:59.748 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:30:59.749 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 7, 2024-04-22T23:30:59.749 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 7 2024-04-22T23:30:59.749 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:30:59.749 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:30:59.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.753+0000 7f8dff1e3700 1 -- 172.21.15.67:0/1102742321 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8de406ecc0 msgr2=0x7f8de4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:59.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.753+0000 7f8dff1e3700 1 --2- 172.21.15.67:0/1102742321 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8de406ecc0 0x7f8de4071180 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8df8101260 tx=0x7f8de80073d0 comp rx=0 tx=0).stop 2024-04-22T23:30:59.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.753+0000 7f8dff1e3700 1 -- 172.21.15.67:0/1102742321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8df8101e60 msgr2=0x7f8df80ffac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:59.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.753+0000 7f8dff1e3700 1 --2- 172.21.15.67:0/1102742321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8df8101e60 0x7f8df80ffac0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8de0006010 tx=0x7f8de000bd70 comp rx=0 tx=0).stop 2024-04-22T23:30:59.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.753+0000 7f8dff1e3700 1 -- 172.21.15.67:0/1102742321 shutdown_connections 2024-04-22T23:30:59.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.753+0000 7f8dff1e3700 1 --2- 172.21.15.67:0/1102742321 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8de406ecc0 0x7f8de4071180 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:59.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.753+0000 7f8dff1e3700 1 --2- 172.21.15.67:0/1102742321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8df8101e60 0x7f8df80ffac0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:59.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.753+0000 7f8dff1e3700 1 --2- 172.21.15.67:0/1102742321 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8df8102780 0x7f8df8100000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:59.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.753+0000 7f8dff1e3700 1 -- 172.21.15.67:0/1102742321 >> 172.21.15.67:0/1102742321 conn(0x7f8df80fb650 msgr2=0x7f8df8105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:30:59.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.753+0000 7f8dff1e3700 1 -- 172.21.15.67:0/1102742321 shutdown_connections 2024-04-22T23:30:59.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.753+0000 7f8dff1e3700 1 -- 172.21.15.67:0/1102742321 wait complete. 2024-04-22T23:30:59.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.860+0000 7fdfb0974700 1 -- 172.21.15.67:0/768310246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfac100750 msgr2=0x7fdfac1047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:59.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.860+0000 7fdfb0974700 1 --2- 172.21.15.67:0/768310246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfac100750 0x7fdfac1047a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fdf94009910 tx=0x7fdf94009c20 comp rx=0 tx=0).stop 2024-04-22T23:30:59.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.860+0000 7fdfb0974700 1 -- 172.21.15.67:0/768310246 shutdown_connections 2024-04-22T23:30:59.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.860+0000 7fdfb0974700 1 --2- 172.21.15.67:0/768310246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfac100750 0x7fdfac1047a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:59.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.860+0000 7fdfb0974700 1 --2- 172.21.15.67:0/768310246 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdfac0ffda0 0x7fdfac100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:59.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.860+0000 7fdfb0974700 1 -- 172.21.15.67:0/768310246 >> 172.21.15.67:0/768310246 conn(0x7fdfac0fb650 msgr2=0x7fdfac0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:30:59.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.860+0000 7fdfb0974700 1 -- 172.21.15.67:0/768310246 shutdown_connections 2024-04-22T23:30:59.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.860+0000 7fdfb0974700 1 -- 172.21.15.67:0/768310246 wait complete. 2024-04-22T23:30:59.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.861+0000 7fdfb0974700 1 Processor -- start 2024-04-22T23:30:59.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.861+0000 7fdfb0974700 1 -- start start 2024-04-22T23:30:59.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.861+0000 7fdfb0974700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfac0ffda0 0x7fdfac19e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:30:59.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.861+0000 7fdfb0974700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdfac100750 0x7fdfac19ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:30:59.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.861+0000 7fdfb0974700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfac19f490 con 0x7fdfac0ffda0 2024-04-22T23:30:59.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.861+0000 7fdfb0974700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfac198940 con 0x7fdfac100750 2024-04-22T23:30:59.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.861+0000 7fdfaa59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfac0ffda0 0x7fdfac19e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:30:59.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.861+0000 7fdfaa59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfac0ffda0 0x7fdfac19e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52206/0 (socket says 172.21.15.67:52206) 2024-04-22T23:30:59.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.861+0000 7fdfaa59c700 1 -- 172.21.15.67:0/2515454259 learned_addr learned my addr 172.21.15.67:0/2515454259 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:30:59.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.862+0000 7fdfa3fff700 1 --2- 172.21.15.67:0/2515454259 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdfac100750 0x7fdfac19ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:30:59.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.862+0000 7fdfaa59c700 1 -- 172.21.15.67:0/2515454259 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdfac100750 msgr2=0x7fdfac19ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:30:59.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.862+0000 7fdfaa59c700 1 --2- 172.21.15.67:0/2515454259 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdfac100750 0x7fdfac19ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:30:59.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.862+0000 7fdfaa59c700 1 -- 172.21.15.67:0/2515454259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf940095f0 con 0x7fdfac0ffda0 2024-04-22T23:30:59.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.862+0000 7fdfaa59c700 1 --2- 172.21.15.67:0/2515454259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfac0ffda0 0x7fdfac19e8c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c00d8d0 tx=0x7fdf9c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:30:59.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.862+0000 7fdfa37fe700 1 -- 172.21.15.67:0/2515454259 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf9c0098a0 con 0x7fdfac0ffda0 2024-04-22T23:30:59.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.862+0000 7fdfa37fe700 1 -- 172.21.15.67:0/2515454259 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdf9c010460 con 0x7fdfac0ffda0 2024-04-22T23:30:59.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.862+0000 7fdfb0974700 1 -- 172.21.15.67:0/2515454259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfac198c20 con 0x7fdfac0ffda0 2024-04-22T23:30:59.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.862+0000 7fdfa37fe700 1 -- 172.21.15.67:0/2515454259 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf9c00f5f0 con 0x7fdfac0ffda0 2024-04-22T23:30:59.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.862+0000 7fdfb0974700 1 -- 172.21.15.67:0/2515454259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfac199170 con 0x7fdfac0ffda0 2024-04-22T23:30:59.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.863+0000 7fdfb0974700 1 -- 172.21.15.67:0/2515454259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfac04f2e0 con 0x7fdfac0ffda0 2024-04-22T23:30:59.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.864+0000 7fdfa37fe700 1 -- 172.21.15.67:0/2515454259 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdf9c009a00 con 0x7fdfac0ffda0 2024-04-22T23:30:59.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.864+0000 7fdfa37fe700 1 --2- 172.21.15.67:0/2515454259 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdf9806ecc0 0x7fdf98071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:30:59.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.864+0000 7fdfa37fe700 1 -- 172.21.15.67:0/2515454259 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6704+0+0 (secure 0 0 0) 0x7fdf9c08edf0 con 0x7fdfac0ffda0 2024-04-22T23:30:59.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.864+0000 7fdfa3fff700 1 --2- 172.21.15.67:0/2515454259 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdf9806ecc0 0x7fdf98071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:30:59.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.865+0000 7fdfa3fff700 1 --2- 172.21.15.67:0/2515454259 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdf9806ecc0 0x7fdf98071180 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdfac0fe000 tx=0x7fdf9401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:30:59.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:30:59.869+0000 7fdfa37fe700 1 -- 172.21.15.67:0/2515454259 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fdf9c05b610 con 0x7fdfac0ffda0 2024-04-22T23:31:00.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.129+0000 7fdfb0974700 1 -- 172.21.15.67:0/2515454259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fdfac1994b0 con 0x7fdfac0ffda0 2024-04-22T23:31:00.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.129+0000 7fdfa37fe700 1 -- 172.21.15.67:0/2515454259 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1901 (secure 0 0 0) 0x7fdf9c05b1a0 con 0x7fdfac0ffda0 2024-04-22T23:31:00.127 INFO:teuthology.orchestra.run.smithi067.stdout:e22 2024-04-22T23:31:00.127 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:31:00.127 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:31:00.127 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:31:00.127 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:31:00.127 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:31:00.127 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:31:00.127 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 22 2024-04-22T23:31:00.127 INFO:teuthology.orchestra.run.smithi067.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T23:31:00.127 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:31:00.128 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:27:20.577198+0000 2024-04-22T23:31:00.128 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:31:00.128 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:31:00.128 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:31:00.128 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:31:00.128 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:31:00.128 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:31:00.128 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:31:00.128 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:31:00.128 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 46 2024-04-22T23:31:00.128 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:31:00.128 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:31:00.128 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:31:00.128 INFO:teuthology.orchestra.run.smithi067.stdout:up {0=14484,1=24273} 2024-04-22T23:31:00.129 INFO:teuthology.orchestra.run.smithi067.stdout:failed 2024-04-22T23:31:00.129 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:31:00.129 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:31:00.129 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:31:00.129 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:31:00.129 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:31:00.129 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:31:00.129 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:31:00.129 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:31:00.129 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:31:00.129 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{0:14484} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.67:6828/3606151860,v1:172.21.15.67:6829/3606151860] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:31:00.129 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{0:14496} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/3305304988,v1:172.21.15.67:6827/3305304988] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:31:00.129 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{1:24273} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.134:6824/535957142,v1:172.21.15.134:6825/535957142] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:31:00.129 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{1:24289} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.134:6826/1954821676,v1:172.21.15.134:6827/1954821676] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:31:00.129 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:31:00.130 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:31:00.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.131+0000 7fdfb0974700 1 -- 172.21.15.67:0/2515454259 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdf9806ecc0 msgr2=0x7fdf98071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:31:00.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.131+0000 7fdfb0974700 1 --2- 172.21.15.67:0/2515454259 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdf9806ecc0 0x7fdf98071180 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdfac0fe000 tx=0x7fdf9401a040 comp rx=0 tx=0).stop 2024-04-22T23:31:00.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.131+0000 7fdfb0974700 1 -- 172.21.15.67:0/2515454259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfac0ffda0 msgr2=0x7fdfac19e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:31:00.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.131+0000 7fdfb0974700 1 --2- 172.21.15.67:0/2515454259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfac0ffda0 0x7fdfac19e8c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c00d8d0 tx=0x7fdf9c00dbe0 comp rx=0 tx=0).stop 2024-04-22T23:31:00.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.132+0000 7fdfb0974700 1 -- 172.21.15.67:0/2515454259 shutdown_connections 2024-04-22T23:31:00.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.132+0000 7fdfb0974700 1 --2- 172.21.15.67:0/2515454259 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdf9806ecc0 0x7fdf98071180 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:00.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.132+0000 7fdfb0974700 1 --2- 172.21.15.67:0/2515454259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfac0ffda0 0x7fdfac19e8c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:00.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.132+0000 7fdfb0974700 1 --2- 172.21.15.67:0/2515454259 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdfac100750 0x7fdfac19ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:00.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.132+0000 7fdfb0974700 1 -- 172.21.15.67:0/2515454259 >> 172.21.15.67:0/2515454259 conn(0x7fdfac0fb650 msgr2=0x7fdfac103ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:31:00.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.132+0000 7fdfb0974700 1 -- 172.21.15.67:0/2515454259 shutdown_connections 2024-04-22T23:31:00.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.132+0000 7fdfb0974700 1 -- 172.21.15.67:0/2515454259 wait complete. 2024-04-22T23:31:00.135 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 22 2024-04-22T23:31:00.137 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:00 smithi134 ceph-mon[85630]: from='client.34288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:31:00.138 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:00 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-22T23:31:00.138 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:00 smithi134 ceph-mon[85630]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-22T23:31:00.138 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:00 smithi134 ceph-mon[85630]: Upgrade: osd.3 is safe to restart 2024-04-22T23:31:00.138 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:00 smithi134 ceph-mon[85630]: from='client.34292 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:31:00.138 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:00 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1102742321' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:31:00.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.240+0000 7f311bac2700 1 -- 172.21.15.67:0/1673198918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3114105bd0 msgr2=0x7f3114105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:31:00.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.240+0000 7f311bac2700 1 --2- 172.21.15.67:0/1673198918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3114105bd0 0x7f3114105fb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3110009910 tx=0x7f3110009c20 comp rx=0 tx=0).stop 2024-04-22T23:31:00.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.240+0000 7f311bac2700 1 -- 172.21.15.67:0/1673198918 shutdown_connections 2024-04-22T23:31:00.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.240+0000 7f311bac2700 1 --2- 172.21.15.67:0/1673198918 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f31140ffc20 0x7f3114100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:00.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.240+0000 7f311bac2700 1 --2- 172.21.15.67:0/1673198918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3114105bd0 0x7f3114105fb0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:00.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.240+0000 7f311bac2700 1 -- 172.21.15.67:0/1673198918 >> 172.21.15.67:0/1673198918 conn(0x7f3114075670 msgr2=0x7f3114075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:31:00.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.240+0000 7f311bac2700 1 -- 172.21.15.67:0/1673198918 shutdown_connections 2024-04-22T23:31:00.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.240+0000 7f311bac2700 1 -- 172.21.15.67:0/1673198918 wait complete. 2024-04-22T23:31:00.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.241+0000 7f311bac2700 1 Processor -- start 2024-04-22T23:31:00.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.241+0000 7f311bac2700 1 -- start start 2024-04-22T23:31:00.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.241+0000 7f311bac2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31140ffc20 0x7f311419c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:31:00.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.241+0000 7f311bac2700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3114105bd0 0x7f311419cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:31:00.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.241+0000 7f311bac2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311419d1f0 con 0x7f31140ffc20 2024-04-22T23:31:00.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.241+0000 7f311bac2700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31141966a0 con 0x7f3114105bd0 2024-04-22T23:31:00.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.241+0000 7f311985e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31140ffc20 0x7f311419c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:31:00.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.241+0000 7f311985e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31140ffc20 0x7f311419c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52234/0 (socket says 172.21.15.67:52234) 2024-04-22T23:31:00.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.241+0000 7f311985e700 1 -- 172.21.15.67:0/4242821590 learned_addr learned my addr 172.21.15.67:0/4242821590 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:31:00.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.241+0000 7f311905d700 1 --2- 172.21.15.67:0/4242821590 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3114105bd0 0x7f311419cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:31:00.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.241+0000 7f311985e700 1 -- 172.21.15.67:0/4242821590 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3114105bd0 msgr2=0x7f311419cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:31:00.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.241+0000 7f311985e700 1 --2- 172.21.15.67:0/4242821590 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3114105bd0 0x7f311419cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:00.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.241+0000 7f311985e700 1 -- 172.21.15.67:0/4242821590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31100095f0 con 0x7f31140ffc20 2024-04-22T23:31:00.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.242+0000 7f311985e700 1 --2- 172.21.15.67:0/4242821590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31140ffc20 0x7f311419c620 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3110006010 tx=0x7f311000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:31:00.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.242+0000 7f3106ffd700 1 -- 172.21.15.67:0/4242821590 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f311001d070 con 0x7f31140ffc20 2024-04-22T23:31:00.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.242+0000 7f3106ffd700 1 -- 172.21.15.67:0/4242821590 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3110005dc0 con 0x7f31140ffc20 2024-04-22T23:31:00.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.242+0000 7f311bac2700 1 -- 172.21.15.67:0/4242821590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3114196920 con 0x7f31140ffc20 2024-04-22T23:31:00.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.242+0000 7f3106ffd700 1 -- 172.21.15.67:0/4242821590 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3110017400 con 0x7f31140ffc20 2024-04-22T23:31:00.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.242+0000 7f311bac2700 1 -- 172.21.15.67:0/4242821590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3114196e10 con 0x7f31140ffc20 2024-04-22T23:31:00.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.243+0000 7f311bac2700 1 -- 172.21.15.67:0/4242821590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f311404f2e0 con 0x7f31140ffc20 2024-04-22T23:31:00.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.243+0000 7f3106ffd700 1 -- 172.21.15.67:0/4242821590 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f311000f460 con 0x7f31140ffc20 2024-04-22T23:31:00.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.244+0000 7f3106ffd700 1 --2- 172.21.15.67:0/4242821590 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f310006ecc0 0x7f3100071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:31:00.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.244+0000 7f3106ffd700 1 -- 172.21.15.67:0/4242821590 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6704+0+0 (secure 0 0 0) 0x7f3110090210 con 0x7f31140ffc20 2024-04-22T23:31:00.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.244+0000 7f311905d700 1 --2- 172.21.15.67:0/4242821590 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f310006ecc0 0x7f3100071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:31:00.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.244+0000 7f311905d700 1 --2- 172.21.15.67:0/4242821590 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f310006ecc0 0x7f3100071180 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f31140ffa40 tx=0x7f310800b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:31:00.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.249+0000 7f3106ffd700 1 -- 172.21.15.67:0/4242821590 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f311005cb70 con 0x7f31140ffc20 2024-04-22T23:31:00.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:00 smithi067 ceph-mon[112438]: from='client.34288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:31:00.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:00 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-22T23:31:00.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:00 smithi067 ceph-mon[112438]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-22T23:31:00.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:00 smithi067 ceph-mon[112438]: Upgrade: osd.3 is safe to restart 2024-04-22T23:31:00.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:00 smithi067 ceph-mon[112438]: from='client.34292 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:31:00.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:00 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1102742321' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:31:00.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.478+0000 7f311bac2700 1 -- 172.21.15.67:0/4242821590 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3114197140 con 0x7f310006ecc0 2024-04-22T23:31:00.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.479+0000 7f3106ffd700 1 -- 172.21.15.67:0/4242821590 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f3114197140 con 0x7f310006ecc0 2024-04-22T23:31:00.476 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:31:00.477 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:31:00.477 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:31:00.477 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:31:00.477 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:31:00.477 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:31:00.477 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:31:00.477 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr" 2024-04-22T23:31:00.477 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:31:00.478 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "9/23 daemons upgraded", 2024-04-22T23:31:00.478 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading osd daemons", 2024-04-22T23:31:00.478 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:31:00.478 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:31:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.481+0000 7f311bac2700 1 -- 172.21.15.67:0/4242821590 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f310006ecc0 msgr2=0x7f3100071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:31:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.481+0000 7f311bac2700 1 --2- 172.21.15.67:0/4242821590 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f310006ecc0 0x7f3100071180 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f31140ffa40 tx=0x7f310800b410 comp rx=0 tx=0).stop 2024-04-22T23:31:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.481+0000 7f311bac2700 1 -- 172.21.15.67:0/4242821590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31140ffc20 msgr2=0x7f311419c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:31:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.481+0000 7f311bac2700 1 --2- 172.21.15.67:0/4242821590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31140ffc20 0x7f311419c620 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3110006010 tx=0x7f311000bd70 comp rx=0 tx=0).stop 2024-04-22T23:31:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.481+0000 7f311bac2700 1 -- 172.21.15.67:0/4242821590 shutdown_connections 2024-04-22T23:31:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.481+0000 7f311bac2700 1 --2- 172.21.15.67:0/4242821590 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f310006ecc0 0x7f3100071180 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.481+0000 7f311bac2700 1 --2- 172.21.15.67:0/4242821590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31140ffc20 0x7f311419c620 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.482+0000 7f311bac2700 1 --2- 172.21.15.67:0/4242821590 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3114105bd0 0x7f311419cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.482+0000 7f311bac2700 1 -- 172.21.15.67:0/4242821590 >> 172.21.15.67:0/4242821590 conn(0x7f3114075670 msgr2=0x7f3114068680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:31:00.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.482+0000 7f311bac2700 1 -- 172.21.15.67:0/4242821590 shutdown_connections 2024-04-22T23:31:00.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.482+0000 7f311bac2700 1 -- 172.21.15.67:0/4242821590 wait complete. 2024-04-22T23:31:00.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.589+0000 7f4a1ac11700 1 -- 172.21.15.67:0/754432836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a14100ea0 msgr2=0x7f4a14101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:31:00.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.589+0000 7f4a1ac11700 1 --2- 172.21.15.67:0/754432836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a14100ea0 0x7f4a14101280 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4a08009910 tx=0x7f4a08009c20 comp rx=0 tx=0).stop 2024-04-22T23:31:00.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.589+0000 7f4a1ac11700 1 -- 172.21.15.67:0/754432836 shutdown_connections 2024-04-22T23:31:00.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.589+0000 7f4a1ac11700 1 --2- 172.21.15.67:0/754432836 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a14101850 0x7f4a141058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:00.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.589+0000 7f4a1ac11700 1 --2- 172.21.15.67:0/754432836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a14100ea0 0x7f4a14101280 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:00.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.589+0000 7f4a1ac11700 1 -- 172.21.15.67:0/754432836 >> 172.21.15.67:0/754432836 conn(0x7f4a140fc750 msgr2=0x7f4a140feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:31:00.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.589+0000 7f4a1ac11700 1 -- 172.21.15.67:0/754432836 shutdown_connections 2024-04-22T23:31:00.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.589+0000 7f4a1ac11700 1 -- 172.21.15.67:0/754432836 wait complete. 2024-04-22T23:31:00.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.590+0000 7f4a1ac11700 1 Processor -- start 2024-04-22T23:31:00.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.590+0000 7f4a1ac11700 1 -- start start 2024-04-22T23:31:00.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.590+0000 7f4a1ac11700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a14100ea0 0x7f4a14073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:31:00.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.590+0000 7f4a1ac11700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a14101850 0x7f4a140735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:31:00.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.590+0000 7f4a1ac11700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a14073b10 con 0x7f4a14100ea0 2024-04-22T23:31:00.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.590+0000 7f4a1ac11700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a14073c50 con 0x7f4a14101850 2024-04-22T23:31:00.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.590+0000 7f4a189ad700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a14100ea0 0x7f4a14073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:31:00.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.590+0000 7f4a189ad700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a14100ea0 0x7f4a14073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52260/0 (socket says 172.21.15.67:52260) 2024-04-22T23:31:00.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.590+0000 7f4a189ad700 1 -- 172.21.15.67:0/638712050 learned_addr learned my addr 172.21.15.67:0/638712050 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:31:00.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.590+0000 7f4a13fff700 1 --2- 172.21.15.67:0/638712050 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a14101850 0x7f4a140735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:31:00.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.590+0000 7f4a189ad700 1 -- 172.21.15.67:0/638712050 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a14101850 msgr2=0x7f4a140735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:31:00.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.590+0000 7f4a189ad700 1 --2- 172.21.15.67:0/638712050 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a14101850 0x7f4a140735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:00.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.590+0000 7f4a189ad700 1 -- 172.21.15.67:0/638712050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a080095f0 con 0x7f4a14100ea0 2024-04-22T23:31:00.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.590+0000 7f4a13fff700 1 --2- 172.21.15.67:0/638712050 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a14101850 0x7f4a140735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:31:00.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.591+0000 7f4a189ad700 1 --2- 172.21.15.67:0/638712050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a14100ea0 0x7f4a14073090 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4a08006010 tx=0x7f4a0800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:31:00.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.591+0000 7f4a11ffb700 1 -- 172.21.15.67:0/638712050 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a0801d070 con 0x7f4a14100ea0 2024-04-22T23:31:00.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.591+0000 7f4a11ffb700 1 -- 172.21.15.67:0/638712050 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4a08004fb0 con 0x7f4a14100ea0 2024-04-22T23:31:00.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.591+0000 7f4a1ac11700 1 -- 172.21.15.67:0/638712050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a14073ed0 con 0x7f4a14100ea0 2024-04-22T23:31:00.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.591+0000 7f4a11ffb700 1 -- 172.21.15.67:0/638712050 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a08022ea0 con 0x7f4a14100ea0 2024-04-22T23:31:00.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.591+0000 7f4a1ac11700 1 -- 172.21.15.67:0/638712050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a141a2dd0 con 0x7f4a14100ea0 2024-04-22T23:31:00.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.592+0000 7f4a1ac11700 1 -- 172.21.15.67:0/638712050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a14109240 con 0x7f4a14100ea0 2024-04-22T23:31:00.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.593+0000 7f4a11ffb700 1 -- 172.21.15.67:0/638712050 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4a08022470 con 0x7f4a14100ea0 2024-04-22T23:31:00.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.593+0000 7f4a11ffb700 1 --2- 172.21.15.67:0/638712050 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4a0406ecc0 0x7f4a04071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:31:00.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.593+0000 7f4a11ffb700 1 -- 172.21.15.67:0/638712050 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6704+0+0 (secure 0 0 0) 0x7f4a080904b0 con 0x7f4a14100ea0 2024-04-22T23:31:00.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.593+0000 7f4a13fff700 1 --2- 172.21.15.67:0/638712050 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4a0406ecc0 0x7f4a04071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:31:00.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.594+0000 7f4a13fff700 1 --2- 172.21.15.67:0/638712050 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4a0406ecc0 0x7f4a04071180 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4a00005fd0 tx=0x7f4a00005e20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:31:00.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.598+0000 7f4a11ffb700 1 -- 172.21.15.67:0/638712050 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f4a08059250 con 0x7f4a14100ea0 2024-04-22T23:31:00.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.892+0000 7f4a1ac11700 1 -- 172.21.15.67:0/638712050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f4a1404f2e0 con 0x7f4a14100ea0 2024-04-22T23:31:00.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.893+0000 7f4a11ffb700 1 -- 172.21.15.67:0/638712050 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f4a08022720 con 0x7f4a14100ea0 2024-04-22T23:31:00.890 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T23:31:00.890 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:31:00.890 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:31:00.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.895+0000 7f4a1ac11700 1 -- 172.21.15.67:0/638712050 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4a0406ecc0 msgr2=0x7f4a04071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:31:00.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.895+0000 7f4a1ac11700 1 --2- 172.21.15.67:0/638712050 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4a0406ecc0 0x7f4a04071180 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4a00005fd0 tx=0x7f4a00005e20 comp rx=0 tx=0).stop 2024-04-22T23:31:00.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.895+0000 7f4a1ac11700 1 -- 172.21.15.67:0/638712050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a14100ea0 msgr2=0x7f4a14073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:31:00.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.895+0000 7f4a1ac11700 1 --2- 172.21.15.67:0/638712050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a14100ea0 0x7f4a14073090 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4a08006010 tx=0x7f4a0800be20 comp rx=0 tx=0).stop 2024-04-22T23:31:00.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.895+0000 7f4a1ac11700 1 -- 172.21.15.67:0/638712050 shutdown_connections 2024-04-22T23:31:00.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.895+0000 7f4a1ac11700 1 --2- 172.21.15.67:0/638712050 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4a0406ecc0 0x7f4a04071180 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:00.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.895+0000 7f4a1ac11700 1 --2- 172.21.15.67:0/638712050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a14100ea0 0x7f4a14073090 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:00.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.895+0000 7f4a1ac11700 1 --2- 172.21.15.67:0/638712050 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a14101850 0x7f4a140735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:00.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.895+0000 7f4a1ac11700 1 -- 172.21.15.67:0/638712050 >> 172.21.15.67:0/638712050 conn(0x7f4a140fc750 msgr2=0x7f4a140fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:31:00.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.895+0000 7f4a1ac11700 1 -- 172.21.15.67:0/638712050 shutdown_connections 2024-04-22T23:31:00.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:00.895+0000 7f4a1ac11700 1 -- 172.21.15.67:0/638712050 wait complete. 2024-04-22T23:31:01.145 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:01 smithi067 ceph-mon[112438]: pgmap v118: 65 pgs: 65 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:31:01.145 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:01 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2515454259' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:31:01.145 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:01 smithi067 ceph-mon[112438]: from='client.34304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:31:01.145 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:01 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/638712050' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:31:01.146 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:01 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:01.146 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:01 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T23:31:01.146 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:01 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:31:01.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:01 smithi134 ceph-mon[85630]: pgmap v118: 65 pgs: 65 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:31:01.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:01 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2515454259' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:31:01.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:01 smithi134 ceph-mon[85630]: from='client.34304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:31:01.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:01 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/638712050' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:31:01.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:01 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:01.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:01 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T23:31:01.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:01 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:31:02.206 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:02 smithi134 ceph-mon[85630]: Upgrade: Updating osd.3 2024-04-22T23:31:02.206 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:02 smithi134 ceph-mon[85630]: Deploying daemon osd.3 on smithi134 2024-04-22T23:31:02.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:02 smithi067 ceph-mon[112438]: Upgrade: Updating osd.3 2024-04-22T23:31:02.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:02 smithi067 ceph-mon[112438]: Deploying daemon osd.3 on smithi134 2024-04-22T23:31:03.158 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:03 smithi134 ceph-mon[85630]: pgmap v119: 65 pgs: 65 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:31:03.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:03 smithi067 ceph-mon[112438]: pgmap v119: 65 pgs: 65 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:31:04.444 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:04 smithi134 systemd[1]: Stopping Ceph osd.3 for d46c346e-00fd-11ef-bc93-c7b262605968... 2024-04-22T23:31:04.740 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:04 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3[40519]: 2024-04-22T23:31:04.446+0000 7f0b2163d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T23:31:04.740 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:04 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3[40519]: 2024-04-22T23:31:04.446+0000 7f0b2163d700 -1 osd.3 72 *** Got signal Terminated *** 2024-04-22T23:31:04.740 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:04 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3[40519]: 2024-04-22T23:31:04.446+0000 7f0b2163d700 -1 osd.3 72 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T23:31:05.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:05 smithi134 ceph-mon[85630]: pgmap v120: 65 pgs: 65 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:31:05.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:05 smithi134 ceph-mon[85630]: osd.3 marked itself down and dead 2024-04-22T23:31:05.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:05 smithi067 ceph-mon[112438]: pgmap v120: 65 pgs: 65 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:31:05.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:05 smithi067 ceph-mon[112438]: osd.3 marked itself down and dead 2024-04-22T23:31:05.900 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:05 smithi134 podman[94624]: 2024-04-22 23:31:05.631692033 +0000 UTC m=+1.290983218 container died 9c2c91a967babe1ff2a048685f0d82e80ccefa1cee531416769f1633244a4301 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-22T23:31:06.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:06 smithi134 ceph-mon[85630]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-22T23:31:06.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:06 smithi134 ceph-mon[85630]: osdmap e73: 6 total, 5 up, 6 in 2024-04-22T23:31:06.240 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:06 smithi134 podman[94624]: 2024-04-22 23:31:06.005429343 +0000 UTC m=+1.664720525 container cleanup 9c2c91a967babe1ff2a048685f0d82e80ccefa1cee531416769f1633244a4301 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True) 2024-04-22T23:31:06.240 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:06 smithi134 bash[94624]: ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3 2024-04-22T23:31:06.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:06 smithi067 ceph-mon[112438]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-22T23:31:06.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:06 smithi067 ceph-mon[112438]: osdmap e73: 6 total, 5 up, 6 in 2024-04-22T23:31:06.624 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:06 smithi134 podman[94642]: 2024-04-22 23:31:06.282059825 +0000 UTC m=+0.650331955 container remove 9c2c91a967babe1ff2a048685f0d82e80ccefa1cee531416769f1633244a4301 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, ceph=True) 2024-04-22T23:31:06.882 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:06 smithi134 podman[94732]: 2024-04-22 23:31:06.530588406 +0000 UTC m=+0.018007538 image pull a0434699c1d76af4eea1fc95ff0a22ea73d1e04e609464fbf658d6f39bc7f2fc quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c 2024-04-22T23:31:06.882 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:06 smithi134 podman[94732]: 2024-04-22 23:31:06.658141342 +0000 UTC m=+0.145560456 container create 01da660594d7e761cf688407ea9041673c5b4f072516d245d480bb8cb5f13131 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T23:31:06.882 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:06 smithi134 podman[94732]: 2024-04-22 23:31:06.885632726 +0000 UTC m=+0.373051840 container init 01da660594d7e761cf688407ea9041673c5b4f072516d245d480bb8cb5f13131 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-22T23:31:07.175 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:07 smithi134 ceph-mon[85630]: pgmap v122: 65 pgs: 15 stale+active+clean, 50 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-22T23:31:07.175 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:07 smithi134 ceph-mon[85630]: osdmap e74: 6 total, 5 up, 6 in 2024-04-22T23:31:07.175 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:06 smithi134 podman[94732]: 2024-04-22 23:31:06.888849962 +0000 UTC m=+0.376269081 container start 01da660594d7e761cf688407ea9041673c5b4f072516d245d480bb8cb5f13131 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0) 2024-04-22T23:31:07.175 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:06 smithi134 podman[94732]: 2024-04-22 23:31:06.985588521 +0000 UTC m=+0.473007660 container attach 01da660594d7e761cf688407ea9041673c5b4f072516d245d480bb8cb5f13131 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-22T23:31:07.459 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:07 smithi134 podman[94732]: 2024-04-22 23:31:07.178929636 +0000 UTC m=+0.666348753 container died 01da660594d7e761cf688407ea9041673c5b4f072516d245d480bb8cb5f13131 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T23:31:07.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:07 smithi067 ceph-mon[112438]: pgmap v122: 65 pgs: 15 stale+active+clean, 50 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-22T23:31:07.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:07 smithi067 ceph-mon[112438]: osdmap e74: 6 total, 5 up, 6 in 2024-04-22T23:31:07.990 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:07 smithi134 podman[94758]: 2024-04-22 23:31:07.71426627 +0000 UTC m=+0.529591558 container remove 01da660594d7e761cf688407ea9041673c5b4f072516d245d480bb8cb5f13131 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T23:31:07.990 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:07 smithi134 systemd[1]: ceph-d46c346e-00fd-11ef-bc93-c7b262605968@osd.3.service: Deactivated successfully. 2024-04-22T23:31:07.991 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:07 smithi134 systemd[1]: Stopped Ceph osd.3 for d46c346e-00fd-11ef-bc93-c7b262605968. 2024-04-22T23:31:07.991 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:07 smithi134 systemd[1]: ceph-d46c346e-00fd-11ef-bc93-c7b262605968@osd.3.service: Consumed 1min 5.730s CPU time. 2024-04-22T23:31:07.991 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:07 smithi134 systemd[1]: Starting Ceph osd.3 for d46c346e-00fd-11ef-bc93-c7b262605968... 2024-04-22T23:31:08.422 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:08 smithi134 ceph-mon[85630]: Health check failed: Degraded data redundancy: 41/363 objects degraded (11.295%), 10 pgs degraded (PG_DEGRADED) 2024-04-22T23:31:08.422 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:08 smithi134 podman[94863]: 2024-04-22 23:31:08.050814027 +0000 UTC m=+0.017944694 image pull a0434699c1d76af4eea1fc95ff0a22ea73d1e04e609464fbf658d6f39bc7f2fc quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c 2024-04-22T23:31:08.422 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:08 smithi134 podman[94863]: 2024-04-22 23:31:08.165388846 +0000 UTC m=+0.132519489 container create 239c19694de17b4e305f61a768aab803aaf7637a2b1559b412ff9e9d0fada24f (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3-activate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416) 2024-04-22T23:31:08.422 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:08 smithi134 podman[94863]: 2024-04-22 23:31:08.35119339 +0000 UTC m=+0.318324031 container init 239c19694de17b4e305f61a768aab803aaf7637a2b1559b412ff9e9d0fada24f (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3-activate, org.label-schema.build-date=20240416, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-22T23:31:08.422 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:08 smithi134 podman[94863]: 2024-04-22 23:31:08.354363825 +0000 UTC m=+0.321494467 container start 239c19694de17b4e305f61a768aab803aaf7637a2b1559b412ff9e9d0fada24f (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3-activate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD) 2024-04-22T23:31:08.423 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:08 smithi134 podman[94863]: 2024-04-22 23:31:08.426009535 +0000 UTC m=+0.393140178 container attach 239c19694de17b4e305f61a768aab803aaf7637a2b1559b412ff9e9d0fada24f (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3-activate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, ceph=True, org.label-schema.schema-version=1.0) 2024-04-22T23:31:08.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:08 smithi067 ceph-mon[112438]: Health check failed: Degraded data redundancy: 41/363 objects degraded (11.295%), 10 pgs degraded (PG_DEGRADED) 2024-04-22T23:31:09.114 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:08 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3-activate[94878]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T23:31:09.114 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:08 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3-activate[94878]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T23:31:09.114 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:08 smithi134 bash[94863]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T23:31:09.114 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:08 smithi134 bash[94863]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T23:31:09.114 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:08 smithi134 bash[94863]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-22T23:31:09.114 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:08 smithi134 bash[94863]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-22T23:31:09.114 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:08 smithi134 bash[94863]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-22T23:31:09.114 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:08 smithi134 bash[94863]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-22T23:31:09.115 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:08 smithi134 bash[94863]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3/block 2024-04-22T23:31:09.115 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:08 smithi134 bash[94863]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-22T23:31:09.115 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:08 smithi134 bash[94863]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-22T23:31:09.115 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:08 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3-activate[94878]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-22T23:31:09.115 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:08 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3-activate[94878]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-22T23:31:09.115 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:08 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3-activate[94878]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-22T23:31:09.115 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:08 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3-activate[94878]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-22T23:31:09.115 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:08 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3-activate[94878]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3/block 2024-04-22T23:31:09.115 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:08 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3-activate[94878]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-22T23:31:09.115 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:08 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3-activate[94878]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-22T23:31:09.115 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:08 smithi134 podman[94863]: 2024-04-22 23:31:08.854355832 +0000 UTC m=+0.821486485 container died 239c19694de17b4e305f61a768aab803aaf7637a2b1559b412ff9e9d0fada24f (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3-activate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-22T23:31:09.442 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:09 smithi134 ceph-mon[85630]: pgmap v124: 65 pgs: 10 active+undersized+degraded, 3 stale+active+clean, 11 active+undersized, 41 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 41/363 objects degraded (11.295%) 2024-04-22T23:31:09.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:09 smithi067 ceph-mon[112438]: pgmap v124: 65 pgs: 10 active+undersized+degraded, 3 stale+active+clean, 11 active+undersized, 41 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 41/363 objects degraded (11.295%) 2024-04-22T23:31:09.698 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:09 smithi134 podman[95005]: 2024-04-22 23:31:09.446263324 +0000 UTC m=+0.585455874 container remove 239c19694de17b4e305f61a768aab803aaf7637a2b1559b412ff9e9d0fada24f (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3-activate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T23:31:09.699 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:09 smithi134 podman[95067]: 2024-04-22 23:31:09.56135053 +0000 UTC m=+0.018243744 image pull a0434699c1d76af4eea1fc95ff0a22ea73d1e04e609464fbf658d6f39bc7f2fc quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c 2024-04-22T23:31:09.699 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:09 smithi134 podman[95067]: 2024-04-22 23:31:09.702626203 +0000 UTC m=+0.159519440 container create 5f42b665260c42d6c323239ebc3be9b4d9ce2dd63c783bea2fc448f9b4ae2e89 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-22T23:31:09.979 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:09 smithi134 podman[95067]: 2024-04-22 23:31:09.891622429 +0000 UTC m=+0.348515624 container init 5f42b665260c42d6c323239ebc3be9b4d9ce2dd63c783bea2fc448f9b4ae2e89 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-22-1618-43be020) 2024-04-22T23:31:09.980 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:09 smithi134 podman[95067]: 2024-04-22 23:31:09.894429979 +0000 UTC m=+0.351323175 container start 5f42b665260c42d6c323239ebc3be9b4d9ce2dd63c783bea2fc448f9b4ae2e89 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-22T23:31:09.980 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:09 smithi134 ceph-osd[95086]: -- 172.21.15.134:0/2979100133 <== mon.1 v2:172.21.15.134:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556be5915680 con 0x556be58dec00 2024-04-22T23:31:10.240 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:09 smithi134 bash[95067]: 5f42b665260c42d6c323239ebc3be9b4d9ce2dd63c783bea2fc448f9b4ae2e89 2024-04-22T23:31:10.240 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:09 smithi134 systemd[1]: Started Ceph osd.3 for d46c346e-00fd-11ef-bc93-c7b262605968. 2024-04-22T23:31:10.740 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:10 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3[95082]: 2024-04-22T23:31:10.439+0000 7f2ec97107c0 -1 Falling back to public interface 2024-04-22T23:31:11.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:11 smithi134 ceph-mon[85630]: pgmap v125: 65 pgs: 17 active+undersized+degraded, 16 active+undersized, 32 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 64/363 objects degraded (17.631%) 2024-04-22T23:31:11.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:11.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:11.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:31:11.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:11.241 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:31:11.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:11 smithi067 ceph-mon[112438]: pgmap v125: 65 pgs: 17 active+undersized+degraded, 16 active+undersized, 32 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 64/363 objects degraded (17.631%) 2024-04-22T23:31:11.333 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:11.334 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:11.334 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:31:11.334 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:11.334 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:31:13.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:13 smithi067 ceph-mon[112438]: pgmap v126: 65 pgs: 17 active+undersized+degraded, 16 active+undersized, 32 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 64/363 objects degraded (17.631%) 2024-04-22T23:31:13.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:13 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:13.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:13 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:13.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:13 smithi134 ceph-mon[85630]: pgmap v126: 65 pgs: 17 active+undersized+degraded, 16 active+undersized, 32 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 64/363 objects degraded (17.631%) 2024-04-22T23:31:13.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:13 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:13.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:13 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:15.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:15 smithi134 ceph-mon[85630]: pgmap v127: 65 pgs: 17 active+undersized+degraded, 16 active+undersized, 32 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 64/363 objects degraded (17.631%) 2024-04-22T23:31:15.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:15 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:15.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:15 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:15.241 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:15 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3[95082]: 2024-04-22T23:31:15.012+0000 7f2ec97107c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-04-22T23:31:15.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:15 smithi067 ceph-mon[112438]: pgmap v127: 65 pgs: 17 active+undersized+degraded, 16 active+undersized, 32 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 64/363 objects degraded (17.631%) 2024-04-22T23:31:15.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:15 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:15.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:15 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:15.739 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:15 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3[95082]: 2024-04-22T23:31:15.357+0000 7f2ec97107c0 -1 osd.3 72 log_to_monitors true 2024-04-22T23:31:16.295 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:16 smithi134 ceph-mon[85630]: from='osd.3 [v2:172.21.15.134:6800/3344693085,v1:172.21.15.134:6801/3344693085]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T23:31:16.295 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:16 smithi134 ceph-mon[85630]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T23:31:16.328 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:16 smithi067 ceph-mon[112438]: from='osd.3 [v2:172.21.15.134:6800/3344693085,v1:172.21.15.134:6801/3344693085]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T23:31:16.328 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:16 smithi067 ceph-mon[112438]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T23:31:17.340 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 22 23:31:17 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3[95082]: 2024-04-22T23:31:17.038+0000 7f2eb6b60700 -1 osd.3 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T23:31:17.340 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:17 smithi134 ceph-mon[85630]: pgmap v128: 65 pgs: 17 active+undersized+degraded, 16 active+undersized, 32 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 64/363 objects degraded (17.631%) 2024-04-22T23:31:17.340 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:17 smithi134 ceph-mon[85630]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T23:31:17.341 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:17 smithi134 ceph-mon[85630]: from='osd.3 [v2:172.21.15.134:6800/3344693085,v1:172.21.15.134:6801/3344693085]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-22T23:31:17.341 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:17 smithi134 ceph-mon[85630]: osdmap e75: 6 total, 5 up, 6 in 2024-04-22T23:31:17.341 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:17 smithi134 ceph-mon[85630]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-22T23:31:17.341 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:17 smithi134 ceph-mon[85630]: from='osd.3 ' entity='osd.3' 2024-04-22T23:31:17.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:17 smithi067 ceph-mon[112438]: pgmap v128: 65 pgs: 17 active+undersized+degraded, 16 active+undersized, 32 active+clean; 310 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 64/363 objects degraded (17.631%) 2024-04-22T23:31:17.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:17 smithi067 ceph-mon[112438]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T23:31:17.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:17 smithi067 ceph-mon[112438]: from='osd.3 [v2:172.21.15.134:6800/3344693085,v1:172.21.15.134:6801/3344693085]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-22T23:31:17.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:17 smithi067 ceph-mon[112438]: osdmap e75: 6 total, 5 up, 6 in 2024-04-22T23:31:17.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:17 smithi067 ceph-mon[112438]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-22T23:31:17.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:17 smithi067 ceph-mon[112438]: from='osd.3 ' entity='osd.3' 2024-04-22T23:31:18.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:18 smithi067 ceph-mon[112438]: Health check update: Degraded data redundancy: 64/363 objects degraded (17.631%), 17 pgs degraded (PG_DEGRADED) 2024-04-22T23:31:18.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:18 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:18.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:18 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:18.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:18 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:31:18.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:18 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:31:18.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:18 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:18.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:18 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:31:18.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:18 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:31:18.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:18 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:31:18.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:18 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:31:18.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:18 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-22T23:31:18.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:18 smithi067 ceph-mon[112438]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-22T23:31:18.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:18 smithi067 ceph-mon[112438]: osd.3 [v2:172.21.15.134:6800/3344693085,v1:172.21.15.134:6801/3344693085] boot 2024-04-22T23:31:18.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:18 smithi067 ceph-mon[112438]: osdmap e76: 6 total, 6 up, 6 in 2024-04-22T23:31:18.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:18 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T23:31:18.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:18 smithi134 ceph-mon[85630]: Health check update: Degraded data redundancy: 64/363 objects degraded (17.631%), 17 pgs degraded (PG_DEGRADED) 2024-04-22T23:31:18.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:18 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:18.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:18 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:18.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:18 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:31:18.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:18 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:31:18.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:18 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:18.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:18 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:31:18.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:18 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:31:18.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:18 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:31:18.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:18 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:31:18.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:18 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-22T23:31:18.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:18 smithi134 ceph-mon[85630]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-22T23:31:18.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:18 smithi134 ceph-mon[85630]: osd.3 [v2:172.21.15.134:6800/3344693085,v1:172.21.15.134:6801/3344693085] boot 2024-04-22T23:31:18.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:18 smithi134 ceph-mon[85630]: osdmap e76: 6 total, 6 up, 6 in 2024-04-22T23:31:18.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:18 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T23:31:20.041 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:19 smithi067 ceph-mon[112438]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-22T23:31:20.041 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:19 smithi067 ceph-mon[112438]: Upgrade: unsafe to stop osd(s) at this time (17 PGs are or would become offline) 2024-04-22T23:31:20.041 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:19 smithi067 ceph-mon[112438]: pgmap v130: 65 pgs: 17 active+undersized+degraded, 16 active+undersized, 32 active+clean; 310 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 64/363 objects degraded (17.631%) 2024-04-22T23:31:20.041 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:19 smithi067 ceph-mon[112438]: osdmap e77: 6 total, 6 up, 6 in 2024-04-22T23:31:20.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:19 smithi134 ceph-mon[85630]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-22T23:31:20.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:19 smithi134 ceph-mon[85630]: Upgrade: unsafe to stop osd(s) at this time (17 PGs are or would become offline) 2024-04-22T23:31:20.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:19 smithi134 ceph-mon[85630]: pgmap v130: 65 pgs: 17 active+undersized+degraded, 16 active+undersized, 32 active+clean; 310 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 64/363 objects degraded (17.631%) 2024-04-22T23:31:20.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:19 smithi134 ceph-mon[85630]: osdmap e77: 6 total, 6 up, 6 in 2024-04-22T23:31:22.041 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:21 smithi067 ceph-mon[112438]: pgmap v133: 65 pgs: 12 active+undersized+degraded, 12 active+undersized, 41 active+clean; 310 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s; 49/363 objects degraded (13.499%) 2024-04-22T23:31:22.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:21 smithi134 ceph-mon[85630]: pgmap v133: 65 pgs: 12 active+undersized+degraded, 12 active+undersized, 41 active+clean; 310 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s; 49/363 objects degraded (13.499%) 2024-04-22T23:31:22.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:22 smithi134 ceph-mon[85630]: Health check update: Degraded data redundancy: 49/363 objects degraded (13.499%), 12 pgs degraded (PG_DEGRADED) 2024-04-22T23:31:23.041 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:22 smithi067 ceph-mon[112438]: Health check update: Degraded data redundancy: 49/363 objects degraded (13.499%), 12 pgs degraded (PG_DEGRADED) 2024-04-22T23:31:24.040 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:23 smithi067 ceph-mon[112438]: pgmap v134: 65 pgs: 1 active+undersized+degraded, 2 active+undersized, 62 active+clean; 310 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 102 KiB/s rd, 6 op/s; 4/363 objects degraded (1.102%) 2024-04-22T23:31:24.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:23 smithi134 ceph-mon[85630]: pgmap v134: 65 pgs: 1 active+undersized+degraded, 2 active+undersized, 62 active+clean; 310 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 102 KiB/s rd, 6 op/s; 4/363 objects degraded (1.102%) 2024-04-22T23:31:25.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:24 smithi134 ceph-mon[85630]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/363 objects degraded (1.102%), 1 pg degraded) 2024-04-22T23:31:25.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:24 smithi067 ceph-mon[112438]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/363 objects degraded (1.102%), 1 pg degraded) 2024-04-22T23:31:26.104 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:25 smithi134 ceph-mon[85630]: pgmap v135: 65 pgs: 65 active+clean; 310 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 77 KiB/s rd, 5 op/s 2024-04-22T23:31:26.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:25 smithi067 ceph-mon[112438]: pgmap v135: 65 pgs: 65 active+clean; 310 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 77 KiB/s rd, 5 op/s 2024-04-22T23:31:27.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:27 smithi067 ceph-mon[112438]: pgmap v136: 65 pgs: 65 active+clean; 310 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 75 KiB/s rd, 4 op/s 2024-04-22T23:31:27.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:27 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:27.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:27 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:31:27.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:27 smithi134 ceph-mon[85630]: pgmap v136: 65 pgs: 65 active+clean; 310 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 75 KiB/s rd, 4 op/s 2024-04-22T23:31:27.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:27 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:27.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:27 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:31:29.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:29 smithi067 ceph-mon[112438]: pgmap v137: 65 pgs: 65 active+clean; 310 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 61 KiB/s rd, 5 op/s 2024-04-22T23:31:29.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:29 smithi134 ceph-mon[85630]: pgmap v137: 65 pgs: 65 active+clean; 310 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 61 KiB/s rd, 5 op/s 2024-04-22T23:31:31.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.016+0000 7fc85a359700 1 -- 172.21.15.67:0/1568626414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc854106cd0 msgr2=0x7fc8541070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:31:31.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.016+0000 7fc85a359700 1 --2- 172.21.15.67:0/1568626414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc854106cd0 0x7fc8541070b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc844009910 tx=0x7fc844009c20 comp rx=0 tx=0).stop 2024-04-22T23:31:31.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.016+0000 7fc85a359700 1 -- 172.21.15.67:0/1568626414 shutdown_connections 2024-04-22T23:31:31.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.016+0000 7fc85a359700 1 --2- 172.21.15.67:0/1568626414 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc854068580 0x7fc8540689e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:31.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.016+0000 7fc85a359700 1 --2- 172.21.15.67:0/1568626414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc854106cd0 0x7fc8541070b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:31.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.016+0000 7fc85a359700 1 -- 172.21.15.67:0/1568626414 >> 172.21.15.67:0/1568626414 conn(0x7fc854075670 msgr2=0x7fc854075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:31:31.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.016+0000 7fc85a359700 1 -- 172.21.15.67:0/1568626414 shutdown_connections 2024-04-22T23:31:31.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.016+0000 7fc85a359700 1 -- 172.21.15.67:0/1568626414 wait complete. 2024-04-22T23:31:31.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.017+0000 7fc85a359700 1 Processor -- start 2024-04-22T23:31:31.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.017+0000 7fc85a359700 1 -- start start 2024-04-22T23:31:31.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.018+0000 7fc85a359700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc854068580 0x7fc85419c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:31:31.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.018+0000 7fc85a359700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc854106cd0 0x7fc85419cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:31:31.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.018+0000 7fc85a359700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85419d230 con 0x7fc854106cd0 2024-04-22T23:31:31.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.018+0000 7fc85a359700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8541966e0 con 0x7fc854068580 2024-04-22T23:31:31.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.018+0000 7fc8537fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc854106cd0 0x7fc85419cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:31:31.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.018+0000 7fc8537fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc854106cd0 0x7fc85419cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60348/0 (socket says 172.21.15.67:60348) 2024-04-22T23:31:31.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.018+0000 7fc8537fe700 1 -- 172.21.15.67:0/3527136754 learned_addr learned my addr 172.21.15.67:0/3527136754 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:31:31.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.018+0000 7fc853fff700 1 --2- 172.21.15.67:0/3527136754 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc854068580 0x7fc85419c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:31:31.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.018+0000 7fc8537fe700 1 -- 172.21.15.67:0/3527136754 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc854068580 msgr2=0x7fc85419c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:31:31.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.018+0000 7fc8537fe700 1 --2- 172.21.15.67:0/3527136754 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc854068580 0x7fc85419c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:31.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.018+0000 7fc8537fe700 1 -- 172.21.15.67:0/3527136754 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8440095f0 con 0x7fc854106cd0 2024-04-22T23:31:31.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.018+0000 7fc853fff700 1 --2- 172.21.15.67:0/3527136754 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc854068580 0x7fc85419c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:31:31.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.018+0000 7fc8537fe700 1 --2- 172.21.15.67:0/3527136754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc854106cd0 0x7fc85419cba0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc84800d8d0 tx=0x7fc84800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:31:31.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.018+0000 7fc8517fa700 1 -- 172.21.15.67:0/3527136754 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc848009940 con 0x7fc854106cd0 2024-04-22T23:31:31.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.019+0000 7fc8517fa700 1 -- 172.21.15.67:0/3527136754 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc848010460 con 0x7fc854106cd0 2024-04-22T23:31:31.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.019+0000 7fc85a359700 1 -- 172.21.15.67:0/3527136754 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8541969c0 con 0x7fc854106cd0 2024-04-22T23:31:31.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.019+0000 7fc8517fa700 1 -- 172.21.15.67:0/3527136754 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc84800f5f0 con 0x7fc854106cd0 2024-04-22T23:31:31.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.019+0000 7fc85a359700 1 -- 172.21.15.67:0/3527136754 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc854196f10 con 0x7fc854106cd0 2024-04-22T23:31:31.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.020+0000 7fc85a359700 1 -- 172.21.15.67:0/3527136754 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8541091d0 con 0x7fc854106cd0 2024-04-22T23:31:31.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.020+0000 7fc8517fa700 1 -- 172.21.15.67:0/3527136754 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc8480105d0 con 0x7fc854106cd0 2024-04-22T23:31:31.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.021+0000 7fc8517fa700 1 --2- 172.21.15.67:0/3527136754 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc83c06ecc0 0x7fc83c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:31:31.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.021+0000 7fc853fff700 1 --2- 172.21.15.67:0/3527136754 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc83c06ecc0 0x7fc83c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:31:31.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.021+0000 7fc8517fa700 1 -- 172.21.15.67:0/3527136754 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6704+0+0 (secure 0 0 0) 0x7fc84808ef40 con 0x7fc854106cd0 2024-04-22T23:31:31.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.021+0000 7fc853fff700 1 --2- 172.21.15.67:0/3527136754 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc83c06ecc0 0x7fc83c071180 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc844005b40 tx=0x7fc84401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:31:31.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.026+0000 7fc8517fa700 1 -- 172.21.15.67:0/3527136754 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc84805b760 con 0x7fc854106cd0 2024-04-22T23:31:31.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.259+0000 7fc85a359700 1 -- 172.21.15.67:0/3527136754 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc854197130 con 0x7fc83c06ecc0 2024-04-22T23:31:31.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.260+0000 7fc8517fa700 1 -- 172.21.15.67:0/3527136754 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fc854197130 con 0x7fc83c06ecc0 2024-04-22T23:31:31.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.262+0000 7fc85a359700 1 -- 172.21.15.67:0/3527136754 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc83c06ecc0 msgr2=0x7fc83c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:31:31.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.262+0000 7fc85a359700 1 --2- 172.21.15.67:0/3527136754 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc83c06ecc0 0x7fc83c071180 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc844005b40 tx=0x7fc84401a040 comp rx=0 tx=0).stop 2024-04-22T23:31:31.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.262+0000 7fc85a359700 1 -- 172.21.15.67:0/3527136754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc854106cd0 msgr2=0x7fc85419cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:31:31.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.262+0000 7fc85a359700 1 --2- 172.21.15.67:0/3527136754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc854106cd0 0x7fc85419cba0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc84800d8d0 tx=0x7fc84800dc90 comp rx=0 tx=0).stop 2024-04-22T23:31:31.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.263+0000 7fc85a359700 1 -- 172.21.15.67:0/3527136754 shutdown_connections 2024-04-22T23:31:31.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.263+0000 7fc85a359700 1 --2- 172.21.15.67:0/3527136754 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc83c06ecc0 0x7fc83c071180 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:31.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.263+0000 7fc85a359700 1 --2- 172.21.15.67:0/3527136754 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc854068580 0x7fc85419c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:31.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.263+0000 7fc85a359700 1 --2- 172.21.15.67:0/3527136754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc854106cd0 0x7fc85419cba0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:31.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.263+0000 7fc85a359700 1 -- 172.21.15.67:0/3527136754 >> 172.21.15.67:0/3527136754 conn(0x7fc854075670 msgr2=0x7fc8540fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:31:31.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.263+0000 7fc85a359700 1 -- 172.21.15.67:0/3527136754 shutdown_connections 2024-04-22T23:31:31.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.263+0000 7fc85a359700 1 -- 172.21.15.67:0/3527136754 wait complete. 2024-04-22T23:31:31.270 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:31:31.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:31 smithi067 ceph-mon[112438]: pgmap v138: 65 pgs: 65 active+clean; 310 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 56 KiB/s rd, 5 op/s 2024-04-22T23:31:31.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.371+0000 7fc605c4f700 1 -- 172.21.15.67:0/3141209813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6000fff50 msgr2=0x7fc600105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:31:31.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.371+0000 7fc605c4f700 1 --2- 172.21.15.67:0/3141209813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6000fff50 0x7fc600105840 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4009910 tx=0x7fc5f4009c20 comp rx=0 tx=0).stop 2024-04-22T23:31:31.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.371+0000 7fc605c4f700 1 -- 172.21.15.67:0/3141209813 shutdown_connections 2024-04-22T23:31:31.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.371+0000 7fc605c4f700 1 --2- 172.21.15.67:0/3141209813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6000fff50 0x7fc600105840 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:31.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.371+0000 7fc605c4f700 1 --2- 172.21.15.67:0/3141209813 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc6000ff5a0 0x7fc6000ff980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:31.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.371+0000 7fc605c4f700 1 -- 172.21.15.67:0/3141209813 >> 172.21.15.67:0/3141209813 conn(0x7fc600074e30 msgr2=0x7fc600075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:31:31.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.371+0000 7fc605c4f700 1 -- 172.21.15.67:0/3141209813 shutdown_connections 2024-04-22T23:31:31.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.371+0000 7fc605c4f700 1 -- 172.21.15.67:0/3141209813 wait complete. 2024-04-22T23:31:31.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.372+0000 7fc605c4f700 1 Processor -- start 2024-04-22T23:31:31.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.372+0000 7fc605c4f700 1 -- start start 2024-04-22T23:31:31.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.372+0000 7fc605c4f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6000ff5a0 0x7fc60019c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:31:31.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.372+0000 7fc605c4f700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc6000fff50 0x7fc60019cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:31:31.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.372+0000 7fc605c4f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc60019d220 con 0x7fc6000ff5a0 2024-04-22T23:31:31.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.372+0000 7fc605c4f700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6001966d0 con 0x7fc6000fff50 2024-04-22T23:31:31.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.372+0000 7fc5ff7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6000ff5a0 0x7fc60019c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:31:31.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.372+0000 7fc5ff7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6000ff5a0 0x7fc60019c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60364/0 (socket says 172.21.15.67:60364) 2024-04-22T23:31:31.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.372+0000 7fc5ff7fe700 1 -- 172.21.15.67:0/4029397248 learned_addr learned my addr 172.21.15.67:0/4029397248 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:31:31.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.372+0000 7fc5feffd700 1 --2- 172.21.15.67:0/4029397248 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc6000fff50 0x7fc60019cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:31:31.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.373+0000 7fc5ff7fe700 1 -- 172.21.15.67:0/4029397248 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc6000fff50 msgr2=0x7fc60019cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:31:31.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.373+0000 7fc5ff7fe700 1 --2- 172.21.15.67:0/4029397248 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc6000fff50 0x7fc60019cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:31.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.373+0000 7fc5ff7fe700 1 -- 172.21.15.67:0/4029397248 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5f40095f0 con 0x7fc6000ff5a0 2024-04-22T23:31:31.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.373+0000 7fc5ff7fe700 1 --2- 172.21.15.67:0/4029397248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6000ff5a0 0x7fc60019c650 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc5f000c8a0 tx=0x7fc5f000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:31:31.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.373+0000 7fc5fcff9700 1 -- 172.21.15.67:0/4029397248 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5f000cea0 con 0x7fc6000ff5a0 2024-04-22T23:31:31.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.373+0000 7fc5fcff9700 1 -- 172.21.15.67:0/4029397248 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc5f0004830 con 0x7fc6000ff5a0 2024-04-22T23:31:31.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.373+0000 7fc605c4f700 1 -- 172.21.15.67:0/4029397248 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6001969b0 con 0x7fc6000ff5a0 2024-04-22T23:31:31.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.373+0000 7fc5fcff9700 1 -- 172.21.15.67:0/4029397248 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5f00055b0 con 0x7fc6000ff5a0 2024-04-22T23:31:31.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.373+0000 7fc605c4f700 1 -- 172.21.15.67:0/4029397248 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc600196f00 con 0x7fc6000ff5a0 2024-04-22T23:31:31.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.374+0000 7fc605c4f700 1 -- 172.21.15.67:0/4029397248 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc600109200 con 0x7fc6000ff5a0 2024-04-22T23:31:31.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.375+0000 7fc5fcff9700 1 -- 172.21.15.67:0/4029397248 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc5f00049a0 con 0x7fc6000ff5a0 2024-04-22T23:31:31.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.375+0000 7fc5fcff9700 1 --2- 172.21.15.67:0/4029397248 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc5e806ecc0 0x7fc5e8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:31:31.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.375+0000 7fc5fcff9700 1 -- 172.21.15.67:0/4029397248 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6704+0+0 (secure 0 0 0) 0x7fc5f008eb10 con 0x7fc6000ff5a0 2024-04-22T23:31:31.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.375+0000 7fc5feffd700 1 --2- 172.21.15.67:0/4029397248 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc5e806ecc0 0x7fc5e8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:31:31.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.376+0000 7fc5feffd700 1 --2- 172.21.15.67:0/4029397248 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc5e806ecc0 0x7fc5e8071180 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc600197b40 tx=0x7fc5f401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:31:31.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.380+0000 7fc5fcff9700 1 -- 172.21.15.67:0/4029397248 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc5f005b3b0 con 0x7fc6000ff5a0 2024-04-22T23:31:31.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:31 smithi134 ceph-mon[85630]: pgmap v138: 65 pgs: 65 active+clean; 310 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 56 KiB/s rd, 5 op/s 2024-04-22T23:31:31.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.609+0000 7fc605c4f700 1 -- 172.21.15.67:0/4029397248 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc6001971d0 con 0x7fc5e806ecc0 2024-04-22T23:31:31.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.610+0000 7fc5fcff9700 1 -- 172.21.15.67:0/4029397248 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fc6001971d0 con 0x7fc5e806ecc0 2024-04-22T23:31:31.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.612+0000 7fc605c4f700 1 -- 172.21.15.67:0/4029397248 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc5e806ecc0 msgr2=0x7fc5e8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:31:31.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.612+0000 7fc605c4f700 1 --2- 172.21.15.67:0/4029397248 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc5e806ecc0 0x7fc5e8071180 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc600197b40 tx=0x7fc5f401a040 comp rx=0 tx=0).stop 2024-04-22T23:31:31.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.612+0000 7fc605c4f700 1 -- 172.21.15.67:0/4029397248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6000ff5a0 msgr2=0x7fc60019c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:31:31.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.612+0000 7fc605c4f700 1 --2- 172.21.15.67:0/4029397248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6000ff5a0 0x7fc60019c650 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc5f000c8a0 tx=0x7fc5f000cc60 comp rx=0 tx=0).stop 2024-04-22T23:31:31.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.612+0000 7fc605c4f700 1 -- 172.21.15.67:0/4029397248 shutdown_connections 2024-04-22T23:31:31.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.612+0000 7fc605c4f700 1 --2- 172.21.15.67:0/4029397248 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc5e806ecc0 0x7fc5e8071180 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:31.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.612+0000 7fc605c4f700 1 --2- 172.21.15.67:0/4029397248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6000ff5a0 0x7fc60019c650 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:31.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.612+0000 7fc605c4f700 1 --2- 172.21.15.67:0/4029397248 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc6000fff50 0x7fc60019cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:31.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.612+0000 7fc605c4f700 1 -- 172.21.15.67:0/4029397248 >> 172.21.15.67:0/4029397248 conn(0x7fc600074e30 msgr2=0x7fc6000fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:31:31.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.612+0000 7fc605c4f700 1 -- 172.21.15.67:0/4029397248 shutdown_connections 2024-04-22T23:31:31.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.612+0000 7fc605c4f700 1 -- 172.21.15.67:0/4029397248 wait complete. 2024-04-22T23:31:31.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.720+0000 7f8e5d649700 1 -- 172.21.15.67:0/823899714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e580ffb40 msgr2=0x7f8e580fffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:31:31.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.720+0000 7f8e5d649700 1 --2- 172.21.15.67:0/823899714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e580ffb40 0x7f8e580fffc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c009910 tx=0x7f8e4c009c20 comp rx=0 tx=0).stop 2024-04-22T23:31:31.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.720+0000 7f8e5d649700 1 -- 172.21.15.67:0/823899714 shutdown_connections 2024-04-22T23:31:31.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.720+0000 7f8e5d649700 1 --2- 172.21.15.67:0/823899714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e580ffb40 0x7f8e580fffc0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:31.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.720+0000 7f8e5d649700 1 --2- 172.21.15.67:0/823899714 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8e58105bf0 0x7f8e58105fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:31.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.720+0000 7f8e5d649700 1 -- 172.21.15.67:0/823899714 >> 172.21.15.67:0/823899714 conn(0x7f8e580fb610 msgr2=0x7f8e580fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:31:31.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.720+0000 7f8e5d649700 1 -- 172.21.15.67:0/823899714 shutdown_connections 2024-04-22T23:31:31.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.720+0000 7f8e5d649700 1 -- 172.21.15.67:0/823899714 wait complete. 2024-04-22T23:31:31.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.720+0000 7f8e5d649700 1 Processor -- start 2024-04-22T23:31:31.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.721+0000 7f8e5d649700 1 -- start start 2024-04-22T23:31:31.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.721+0000 7f8e5d649700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8e580ffb40 0x7f8e581017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:31:31.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.721+0000 7f8e5d649700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e58105bf0 0x7f8e58101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:31:31.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.721+0000 7f8e5d649700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e58102230 con 0x7f8e58105bf0 2024-04-22T23:31:31.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.721+0000 7f8e5d649700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e58102370 con 0x7f8e580ffb40 2024-04-22T23:31:31.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.721+0000 7f8e567fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e58105bf0 0x7f8e58101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:31:31.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.721+0000 7f8e567fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e58105bf0 0x7f8e58101cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60384/0 (socket says 172.21.15.67:60384) 2024-04-22T23:31:31.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.721+0000 7f8e567fc700 1 -- 172.21.15.67:0/638473750 learned_addr learned my addr 172.21.15.67:0/638473750 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:31:31.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.721+0000 7f8e567fc700 1 -- 172.21.15.67:0/638473750 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8e580ffb40 msgr2=0x7f8e581017b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:31:31.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.721+0000 7f8e567fc700 1 --2- 172.21.15.67:0/638473750 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8e580ffb40 0x7f8e581017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:31.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.721+0000 7f8e567fc700 1 -- 172.21.15.67:0/638473750 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e4c0095f0 con 0x7f8e58105bf0 2024-04-22T23:31:31.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.721+0000 7f8e567fc700 1 --2- 172.21.15.67:0/638473750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e58105bf0 0x7f8e58101cf0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c006010 tx=0x7f8e4c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:31:31.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.722+0000 7f8e3ffff700 1 -- 172.21.15.67:0/638473750 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e4c01d070 con 0x7f8e58105bf0 2024-04-22T23:31:31.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.722+0000 7f8e5d649700 1 -- 172.21.15.67:0/638473750 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e581025f0 con 0x7f8e58105bf0 2024-04-22T23:31:31.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.722+0000 7f8e3ffff700 1 -- 172.21.15.67:0/638473750 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8e4c004fb0 con 0x7f8e58105bf0 2024-04-22T23:31:31.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.722+0000 7f8e3ffff700 1 -- 172.21.15.67:0/638473750 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e4c022ea0 con 0x7f8e58105bf0 2024-04-22T23:31:31.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.722+0000 7f8e5d649700 1 -- 172.21.15.67:0/638473750 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e581a2e60 con 0x7f8e58105bf0 2024-04-22T23:31:31.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.723+0000 7f8e5d649700 1 -- 172.21.15.67:0/638473750 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e5804f2e0 con 0x7f8e58105bf0 2024-04-22T23:31:31.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.723+0000 7f8e3ffff700 1 -- 172.21.15.67:0/638473750 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8e4c022470 con 0x7f8e58105bf0 2024-04-22T23:31:31.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.724+0000 7f8e3ffff700 1 --2- 172.21.15.67:0/638473750 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8e4006ecc0 0x7f8e40071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:31:31.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.724+0000 7f8e3ffff700 1 -- 172.21.15.67:0/638473750 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6704+0+0 (secure 0 0 0) 0x7f8e4c090490 con 0x7f8e58105bf0 2024-04-22T23:31:31.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.724+0000 7f8e56ffd700 1 --2- 172.21.15.67:0/638473750 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8e4006ecc0 0x7f8e40071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:31:31.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.724+0000 7f8e56ffd700 1 --2- 172.21.15.67:0/638473750 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8e4006ecc0 0x7f8e40071180 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8e480097b0 tx=0x7f8e48006d20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:31:31.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.728+0000 7f8e3ffff700 1 -- 172.21.15.67:0/638473750 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f8e4c059230 con 0x7f8e58105bf0 2024-04-22T23:31:31.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.947+0000 7f8e5d649700 1 -- 172.21.15.67:0/638473750 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e58069600 con 0x7f8e4006ecc0 2024-04-22T23:31:31.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.958+0000 7f8e3ffff700 1 -- 172.21.15.67:0/638473750 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f8e58069600 con 0x7f8e4006ecc0 2024-04-22T23:31:31.955 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:31:31.955 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (5m) 52s ago 16m 26.1M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:31:31.955 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (16m) 52s ago 16m 8728k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:31:31.955 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (15m) 19s ago 15m 8363k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:31:31.956 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (2m) 52s ago 16m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:31:31.956 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (2m) 19s ago 15m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:31:31.956 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (4m) 52s ago 16m 80.1M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:31:31.956 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (11m) 52s ago 11m 62.7M - 18.2.1 5be31c24972a e24a471f0a8c 2024-04-22T23:31:31.956 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (10m) 52s ago 10m 130M - 18.2.1 5be31c24972a 0730f9fc8e61 2024-04-22T23:31:31.956 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (11m) 19s ago 11m 122M - 18.2.1 5be31c24972a b6a169586433 2024-04-22T23:31:31.956 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (10m) 19s ago 10m 67.9M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:31:31.956 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (6m) 52s ago 17m 506M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:31:31.956 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (6m) 19s ago 15m 452M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:31:31.957 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (3m) 52s ago 18m 49.7M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:31:31.957 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (3m) 19s ago 15m 40.2M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:31:31.957 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (5m) 52s ago 16m 15.4M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:31:31.957 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (5m) 19s ago 15m 16.0M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:31:31.957 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (2m) 52s ago 14m 95.4M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:31:31.957 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (94s) 52s ago 14m 131M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:31:31.957 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (58s) 52s ago 13m 15.1M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:31:31.957 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (22s) 19s ago 13m 11.7M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:31:31.957 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (13m) 19s ago 13m 222M 2240M 18.2.1 5be31c24972a c7586562ffa6 2024-04-22T23:31:31.958 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (12m) 19s ago 12m 207M 2240M 18.2.1 5be31c24972a 9fd97f1c65cb 2024-04-22T23:31:31.958 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (5m) 52s ago 16m 71.3M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:31:31.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.960+0000 7f8e5d649700 1 -- 172.21.15.67:0/638473750 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8e4006ecc0 msgr2=0x7f8e40071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:31:31.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.960+0000 7f8e5d649700 1 --2- 172.21.15.67:0/638473750 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8e4006ecc0 0x7f8e40071180 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8e480097b0 tx=0x7f8e48006d20 comp rx=0 tx=0).stop 2024-04-22T23:31:31.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.960+0000 7f8e5d649700 1 -- 172.21.15.67:0/638473750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e58105bf0 msgr2=0x7f8e58101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:31:31.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.960+0000 7f8e5d649700 1 --2- 172.21.15.67:0/638473750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e58105bf0 0x7f8e58101cf0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c006010 tx=0x7f8e4c00be20 comp rx=0 tx=0).stop 2024-04-22T23:31:31.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.960+0000 7f8e5d649700 1 -- 172.21.15.67:0/638473750 shutdown_connections 2024-04-22T23:31:31.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.960+0000 7f8e5d649700 1 --2- 172.21.15.67:0/638473750 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8e4006ecc0 0x7f8e40071180 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:31.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.960+0000 7f8e5d649700 1 --2- 172.21.15.67:0/638473750 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8e580ffb40 0x7f8e581017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:31.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.960+0000 7f8e5d649700 1 --2- 172.21.15.67:0/638473750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e58105bf0 0x7f8e58101cf0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:31.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.960+0000 7f8e5d649700 1 -- 172.21.15.67:0/638473750 >> 172.21.15.67:0/638473750 conn(0x7f8e580fb610 msgr2=0x7f8e58068ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:31:31.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.960+0000 7f8e5d649700 1 -- 172.21.15.67:0/638473750 shutdown_connections 2024-04-22T23:31:31.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:31.960+0000 7f8e5d649700 1 -- 172.21.15.67:0/638473750 wait complete. 2024-04-22T23:31:32.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.066+0000 7f7374da7700 1 -- 172.21.15.67:0/1257152729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7370100ea0 msgr2=0x7f7370101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:31:32.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.066+0000 7f7374da7700 1 --2- 172.21.15.67:0/1257152729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7370100ea0 0x7f7370101280 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7360009910 tx=0x7f7360009c20 comp rx=0 tx=0).stop 2024-04-22T23:31:32.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.066+0000 7f7374da7700 1 -- 172.21.15.67:0/1257152729 shutdown_connections 2024-04-22T23:31:32.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.066+0000 7f7374da7700 1 --2- 172.21.15.67:0/1257152729 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7370101850 0x7f73701058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:32.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.066+0000 7f7374da7700 1 --2- 172.21.15.67:0/1257152729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7370100ea0 0x7f7370101280 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:32.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.066+0000 7f7374da7700 1 -- 172.21.15.67:0/1257152729 >> 172.21.15.67:0/1257152729 conn(0x7f73700fc750 msgr2=0x7f73700feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:31:32.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.066+0000 7f7374da7700 1 -- 172.21.15.67:0/1257152729 shutdown_connections 2024-04-22T23:31:32.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.066+0000 7f7374da7700 1 -- 172.21.15.67:0/1257152729 wait complete. 2024-04-22T23:31:32.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.067+0000 7f7374da7700 1 Processor -- start 2024-04-22T23:31:32.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.067+0000 7f7374da7700 1 -- start start 2024-04-22T23:31:32.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.067+0000 7f7374da7700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7370100ea0 0x7f737019e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:31:32.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.067+0000 7f7374da7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7370101850 0x7f737019edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:31:32.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.067+0000 7f7374da7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f737019f3d0 con 0x7f7370101850 2024-04-22T23:31:32.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.067+0000 7f7374da7700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7370198960 con 0x7f7370100ea0 2024-04-22T23:31:32.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.067+0000 7f736dd9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7370101850 0x7f737019edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:31:32.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.067+0000 7f736dd9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7370101850 0x7f737019edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60392/0 (socket says 172.21.15.67:60392) 2024-04-22T23:31:32.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.067+0000 7f736dd9b700 1 -- 172.21.15.67:0/667391453 learned_addr learned my addr 172.21.15.67:0/667391453 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:31:32.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.068+0000 7f736e59c700 1 --2- 172.21.15.67:0/667391453 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7370100ea0 0x7f737019e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:31:32.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.068+0000 7f736dd9b700 1 -- 172.21.15.67:0/667391453 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7370100ea0 msgr2=0x7f737019e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:31:32.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.068+0000 7f736dd9b700 1 --2- 172.21.15.67:0/667391453 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7370100ea0 0x7f737019e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:32.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.068+0000 7f736dd9b700 1 -- 172.21.15.67:0/667391453 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73600095f0 con 0x7f7370101850 2024-04-22T23:31:32.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.068+0000 7f736dd9b700 1 --2- 172.21.15.67:0/667391453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7370101850 0x7f737019edb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f736400c8f0 tx=0x7f736400cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:31:32.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.068+0000 7f735f7fe700 1 -- 172.21.15.67:0/667391453 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f73640078a0 con 0x7f7370101850 2024-04-22T23:31:32.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.068+0000 7f7374da7700 1 -- 172.21.15.67:0/667391453 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7370198c40 con 0x7f7370101850 2024-04-22T23:31:32.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.068+0000 7f735f7fe700 1 -- 172.21.15.67:0/667391453 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f736400f450 con 0x7f7370101850 2024-04-22T23:31:32.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.068+0000 7f735f7fe700 1 -- 172.21.15.67:0/667391453 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7364018610 con 0x7f7370101850 2024-04-22T23:31:32.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.068+0000 7f7374da7700 1 -- 172.21.15.67:0/667391453 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7370199190 con 0x7f7370101850 2024-04-22T23:31:32.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.069+0000 7f7374da7700 1 -- 172.21.15.67:0/667391453 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7370105020 con 0x7f7370101850 2024-04-22T23:31:32.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.070+0000 7f735f7fe700 1 -- 172.21.15.67:0/667391453 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f736400f5c0 con 0x7f7370101850 2024-04-22T23:31:32.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.070+0000 7f735f7fe700 1 --2- 172.21.15.67:0/667391453 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f735806ecc0 0x7f7358071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:31:32.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.071+0000 7f735f7fe700 1 -- 172.21.15.67:0/667391453 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6704+0+0 (secure 0 0 0) 0x7f736408ee90 con 0x7f7370101850 2024-04-22T23:31:32.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.071+0000 7f736e59c700 1 --2- 172.21.15.67:0/667391453 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f735806ecc0 0x7f7358071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:31:32.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.071+0000 7f736e59c700 1 --2- 172.21.15.67:0/667391453 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f735806ecc0 0x7f7358071180 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7360006010 tx=0x7f736001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:31:32.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.077+0000 7f735f7fe700 1 -- 172.21.15.67:0/667391453 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f736405b6b0 con 0x7f7370101850 2024-04-22T23:31:32.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:32 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:32.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:32 smithi067 ceph-mon[112438]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:31:32.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:32 smithi067 ceph-mon[112438]: from='client.34320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:31:32.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.379+0000 7f7374da7700 1 -- 172.21.15.67:0/667391453 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f737004f2e0 con 0x7f7370101850 2024-04-22T23:31:32.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.379+0000 7f735f7fe700 1 -- 172.21.15.67:0/667391453 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7f736405b240 con 0x7f7370101850 2024-04-22T23:31:32.376 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:31:32.376 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:31:32.377 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:31:32.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:31:32.377 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:31:32.377 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:31:32.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:31:32.377 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:31:32.377 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2, 2024-04-22T23:31:32.377 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 4 2024-04-22T23:31:32.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:31:32.377 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:31:32.377 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-22T23:31:32.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:31:32.378 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:31:32.378 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6, 2024-04-22T23:31:32.378 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 8 2024-04-22T23:31:32.378 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:31:32.378 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:31:32.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.381+0000 7f7374da7700 1 -- 172.21.15.67:0/667391453 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f735806ecc0 msgr2=0x7f7358071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:31:32.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.381+0000 7f7374da7700 1 --2- 172.21.15.67:0/667391453 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f735806ecc0 0x7f7358071180 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7360006010 tx=0x7f736001a040 comp rx=0 tx=0).stop 2024-04-22T23:31:32.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.381+0000 7f7374da7700 1 -- 172.21.15.67:0/667391453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7370101850 msgr2=0x7f737019edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:31:32.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.381+0000 7f7374da7700 1 --2- 172.21.15.67:0/667391453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7370101850 0x7f737019edb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f736400c8f0 tx=0x7f736400cc00 comp rx=0 tx=0).stop 2024-04-22T23:31:32.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.381+0000 7f7374da7700 1 -- 172.21.15.67:0/667391453 shutdown_connections 2024-04-22T23:31:32.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.381+0000 7f7374da7700 1 --2- 172.21.15.67:0/667391453 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f735806ecc0 0x7f7358071180 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:32.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.381+0000 7f7374da7700 1 --2- 172.21.15.67:0/667391453 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7370100ea0 0x7f737019e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:32.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.381+0000 7f7374da7700 1 --2- 172.21.15.67:0/667391453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7370101850 0x7f737019edb0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:32.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.381+0000 7f7374da7700 1 -- 172.21.15.67:0/667391453 >> 172.21.15.67:0/667391453 conn(0x7f73700fc750 msgr2=0x7f73700fddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:31:32.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.382+0000 7f7374da7700 1 -- 172.21.15.67:0/667391453 shutdown_connections 2024-04-22T23:31:32.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.382+0000 7f7374da7700 1 -- 172.21.15.67:0/667391453 wait complete. 2024-04-22T23:31:32.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.488+0000 7fc3ed7b1700 1 -- 172.21.15.67:0/2661191967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3e8100e90 msgr2=0x7fc3e8101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:31:32.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.488+0000 7fc3ed7b1700 1 --2- 172.21.15.67:0/2661191967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3e8100e90 0x7fc3e8101270 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8009910 tx=0x7fc3d8009c20 comp rx=0 tx=0).stop 2024-04-22T23:31:32.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.488+0000 7fc3ed7b1700 1 -- 172.21.15.67:0/2661191967 shutdown_connections 2024-04-22T23:31:32.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.488+0000 7fc3ed7b1700 1 --2- 172.21.15.67:0/2661191967 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc3e8101840 0x7fc3e8105890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:32.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.488+0000 7fc3ed7b1700 1 --2- 172.21.15.67:0/2661191967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3e8100e90 0x7fc3e8101270 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:32.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.488+0000 7fc3ed7b1700 1 -- 172.21.15.67:0/2661191967 >> 172.21.15.67:0/2661191967 conn(0x7fc3e80fc740 msgr2=0x7fc3e80feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:31:32.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.489+0000 7fc3ed7b1700 1 -- 172.21.15.67:0/2661191967 shutdown_connections 2024-04-22T23:31:32.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.489+0000 7fc3ed7b1700 1 -- 172.21.15.67:0/2661191967 wait complete. 2024-04-22T23:31:32.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.489+0000 7fc3ed7b1700 1 Processor -- start 2024-04-22T23:31:32.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.489+0000 7fc3ed7b1700 1 -- start start 2024-04-22T23:31:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.489+0000 7fc3ed7b1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3e8100e90 0x7fc3e819e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:31:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.490+0000 7fc3ed7b1700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc3e8101840 0x7fc3e819edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:31:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.490+0000 7fc3ed7b1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e819f460 con 0x7fc3e8100e90 2024-04-22T23:31:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.490+0000 7fc3ed7b1700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e8198910 con 0x7fc3e8101840 2024-04-22T23:31:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.490+0000 7fc3e6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3e8100e90 0x7fc3e819e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:31:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.490+0000 7fc3e6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3e8100e90 0x7fc3e819e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60416/0 (socket says 172.21.15.67:60416) 2024-04-22T23:31:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.490+0000 7fc3e6ffd700 1 -- 172.21.15.67:0/2440955730 learned_addr learned my addr 172.21.15.67:0/2440955730 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:31:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.490+0000 7fc3e67fc700 1 --2- 172.21.15.67:0/2440955730 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc3e8101840 0x7fc3e819edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:31:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.490+0000 7fc3e6ffd700 1 -- 172.21.15.67:0/2440955730 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc3e8101840 msgr2=0x7fc3e819edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:31:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.490+0000 7fc3e6ffd700 1 --2- 172.21.15.67:0/2440955730 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc3e8101840 0x7fc3e819edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.490+0000 7fc3e6ffd700 1 -- 172.21.15.67:0/2440955730 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d80095f0 con 0x7fc3e8100e90 2024-04-22T23:31:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.490+0000 7fc3e6ffd700 1 --2- 172.21.15.67:0/2440955730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3e8100e90 0x7fc3e819e890 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8006010 tx=0x7fc3d800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:31:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.490+0000 7fc3cffff700 1 -- 172.21.15.67:0/2440955730 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc3d801d070 con 0x7fc3e8100e90 2024-04-22T23:31:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.490+0000 7fc3ed7b1700 1 -- 172.21.15.67:0/2440955730 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3e8198b90 con 0x7fc3e8100e90 2024-04-22T23:31:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.490+0000 7fc3cffff700 1 -- 172.21.15.67:0/2440955730 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc3d800f460 con 0x7fc3e8100e90 2024-04-22T23:31:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.490+0000 7fc3cffff700 1 -- 172.21.15.67:0/2440955730 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc3d8017610 con 0x7fc3e8100e90 2024-04-22T23:31:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.490+0000 7fc3ed7b1700 1 -- 172.21.15.67:0/2440955730 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3e8199080 con 0x7fc3e8100e90 2024-04-22T23:31:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.491+0000 7fc3ed7b1700 1 -- 172.21.15.67:0/2440955730 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3e8105010 con 0x7fc3e8100e90 2024-04-22T23:31:32.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.492+0000 7fc3cffff700 1 -- 172.21.15.67:0/2440955730 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc3d8003a60 con 0x7fc3e8100e90 2024-04-22T23:31:32.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.492+0000 7fc3cffff700 1 --2- 172.21.15.67:0/2440955730 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc3d006ecc0 0x7fc3d0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:31:32.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.492+0000 7fc3cffff700 1 -- 172.21.15.67:0/2440955730 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6704+0+0 (secure 0 0 0) 0x7fc3d8090450 con 0x7fc3e8100e90 2024-04-22T23:31:32.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.492+0000 7fc3e67fc700 1 --2- 172.21.15.67:0/2440955730 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc3d006ecc0 0x7fc3d0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:31:32.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.493+0000 7fc3e67fc700 1 --2- 172.21.15.67:0/2440955730 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc3d006ecc0 0x7fc3d0071180 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc3e80fce80 tx=0x7fc3dc00a300 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:31:32.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:32 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:32.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:32 smithi134 ceph-mon[85630]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:31:32.491 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:32 smithi134 ceph-mon[85630]: from='client.34320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:31:32.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.497+0000 7fc3cffff700 1 -- 172.21.15.67:0/2440955730 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc3d805cc70 con 0x7fc3e8100e90 2024-04-22T23:31:32.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.753+0000 7fc3ed7b1700 1 -- 172.21.15.67:0/2440955730 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc3e804f2e0 con 0x7fc3e8100e90 2024-04-22T23:31:32.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.753+0000 7fc3cffff700 1 -- 172.21.15.67:0/2440955730 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1901 (secure 0 0 0) 0x7fc3d805c800 con 0x7fc3e8100e90 2024-04-22T23:31:32.750 INFO:teuthology.orchestra.run.smithi067.stdout:e22 2024-04-22T23:31:32.750 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:31:32.751 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:31:32.751 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:31:32.751 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:31:32.751 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:31:32.751 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:31:32.751 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 22 2024-04-22T23:31:32.751 INFO:teuthology.orchestra.run.smithi067.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T23:31:32.751 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:31:32.751 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:27:20.577198+0000 2024-04-22T23:31:32.751 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:31:32.751 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:31:32.751 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:31:32.751 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:31:32.751 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:31:32.752 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:31:32.752 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:31:32.752 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:31:32.752 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 46 2024-04-22T23:31:32.752 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:31:32.752 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:31:32.752 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:31:32.752 INFO:teuthology.orchestra.run.smithi067.stdout:up {0=14484,1=24273} 2024-04-22T23:31:32.752 INFO:teuthology.orchestra.run.smithi067.stdout:failed 2024-04-22T23:31:32.752 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:31:32.752 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:31:32.752 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:31:32.752 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:31:32.752 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:31:32.752 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:31:32.753 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:31:32.753 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:31:32.753 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:31:32.753 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{0:14484} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.67:6828/3606151860,v1:172.21.15.67:6829/3606151860] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:31:32.753 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{0:14496} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/3305304988,v1:172.21.15.67:6827/3305304988] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:31:32.753 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{1:24273} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.134:6824/535957142,v1:172.21.15.134:6825/535957142] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:31:32.753 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{1:24289} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.134:6826/1954821676,v1:172.21.15.134:6827/1954821676] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:31:32.753 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:31:32.753 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:31:32.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.755+0000 7fc3ed7b1700 1 -- 172.21.15.67:0/2440955730 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc3d006ecc0 msgr2=0x7fc3d0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:31:32.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.755+0000 7fc3ed7b1700 1 --2- 172.21.15.67:0/2440955730 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc3d006ecc0 0x7fc3d0071180 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc3e80fce80 tx=0x7fc3dc00a300 comp rx=0 tx=0).stop 2024-04-22T23:31:32.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.755+0000 7fc3ed7b1700 1 -- 172.21.15.67:0/2440955730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3e8100e90 msgr2=0x7fc3e819e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:31:32.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.755+0000 7fc3ed7b1700 1 --2- 172.21.15.67:0/2440955730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3e8100e90 0x7fc3e819e890 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8006010 tx=0x7fc3d800bd70 comp rx=0 tx=0).stop 2024-04-22T23:31:32.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.755+0000 7fc3ed7b1700 1 -- 172.21.15.67:0/2440955730 shutdown_connections 2024-04-22T23:31:32.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.755+0000 7fc3ed7b1700 1 --2- 172.21.15.67:0/2440955730 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc3d006ecc0 0x7fc3d0071180 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:32.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.755+0000 7fc3ed7b1700 1 --2- 172.21.15.67:0/2440955730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3e8100e90 0x7fc3e819e890 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:32.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.755+0000 7fc3ed7b1700 1 --2- 172.21.15.67:0/2440955730 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc3e8101840 0x7fc3e819edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:32.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.755+0000 7fc3ed7b1700 1 -- 172.21.15.67:0/2440955730 >> 172.21.15.67:0/2440955730 conn(0x7fc3e80fc740 msgr2=0x7fc3e80fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:31:32.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.756+0000 7fc3ed7b1700 1 -- 172.21.15.67:0/2440955730 shutdown_connections 2024-04-22T23:31:32.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.756+0000 7fc3ed7b1700 1 -- 172.21.15.67:0/2440955730 wait complete. 2024-04-22T23:31:32.755 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 22 2024-04-22T23:31:32.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.864+0000 7fc9a1a2a700 1 -- 172.21.15.67:0/2249099501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc99c0ff670 msgr2=0x7fc99c0ffa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:31:32.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.864+0000 7fc9a1a2a700 1 --2- 172.21.15.67:0/2249099501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc99c0ff670 0x7fc99c0ffa50 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc98c009910 tx=0x7fc98c009c20 comp rx=0 tx=0).stop 2024-04-22T23:31:32.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.864+0000 7fc9a1a2a700 1 -- 172.21.15.67:0/2249099501 shutdown_connections 2024-04-22T23:31:32.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.864+0000 7fc9a1a2a700 1 --2- 172.21.15.67:0/2249099501 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc99c100020 0x7fc99c103f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:32.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.864+0000 7fc9a1a2a700 1 --2- 172.21.15.67:0/2249099501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc99c0ff670 0x7fc99c0ffa50 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:32.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.864+0000 7fc9a1a2a700 1 -- 172.21.15.67:0/2249099501 >> 172.21.15.67:0/2249099501 conn(0x7fc99c074e30 msgr2=0x7fc99c075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:31:32.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.864+0000 7fc9a1a2a700 1 -- 172.21.15.67:0/2249099501 shutdown_connections 2024-04-22T23:31:32.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.864+0000 7fc9a1a2a700 1 -- 172.21.15.67:0/2249099501 wait complete. 2024-04-22T23:31:32.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.865+0000 7fc9a1a2a700 1 Processor -- start 2024-04-22T23:31:32.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.865+0000 7fc9a1a2a700 1 -- start start 2024-04-22T23:31:32.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.865+0000 7fc9a1a2a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc99c0ff670 0x7fc99c19c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:31:32.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.865+0000 7fc9a1a2a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc99c100020 0x7fc99c19cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:31:32.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.865+0000 7fc9a1a2a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc99c196a30 con 0x7fc99c0ff670 2024-04-22T23:31:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.865+0000 7fc9a1a2a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc99c196ba0 con 0x7fc99c100020 2024-04-22T23:31:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.865+0000 7fc99affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc99c0ff670 0x7fc99c19c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:31:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.865+0000 7fc99affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc99c0ff670 0x7fc99c19c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60434/0 (socket says 172.21.15.67:60434) 2024-04-22T23:31:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.865+0000 7fc99affd700 1 -- 172.21.15.67:0/3319451878 learned_addr learned my addr 172.21.15.67:0/3319451878 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:31:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.866+0000 7fc99affd700 1 -- 172.21.15.67:0/3319451878 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc99c100020 msgr2=0x7fc99c19cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:31:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.866+0000 7fc99affd700 1 --2- 172.21.15.67:0/3319451878 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc99c100020 0x7fc99c19cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.866+0000 7fc99affd700 1 -- 172.21.15.67:0/3319451878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc98c0095f0 con 0x7fc99c0ff670 2024-04-22T23:31:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.866+0000 7fc99affd700 1 --2- 172.21.15.67:0/3319451878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc99c0ff670 0x7fc99c19c9f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc98c006010 tx=0x7fc98c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:31:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.866+0000 7fc9a0a28700 1 -- 172.21.15.67:0/3319451878 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc98c01d070 con 0x7fc99c0ff670 2024-04-22T23:31:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.866+0000 7fc9a0a28700 1 -- 172.21.15.67:0/3319451878 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc98c004fb0 con 0x7fc99c0ff670 2024-04-22T23:31:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.866+0000 7fc9a0a28700 1 -- 172.21.15.67:0/3319451878 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc98c022ea0 con 0x7fc99c0ff670 2024-04-22T23:31:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.866+0000 7fc9a1a2a700 1 -- 172.21.15.67:0/3319451878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc99c196e20 con 0x7fc99c0ff670 2024-04-22T23:31:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.866+0000 7fc9a1a2a700 1 -- 172.21.15.67:0/3319451878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc99c197310 con 0x7fc99c0ff670 2024-04-22T23:31:32.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.867+0000 7fc9a1a2a700 1 -- 172.21.15.67:0/3319451878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc99c04f2e0 con 0x7fc99c0ff670 2024-04-22T23:31:32.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.868+0000 7fc9a0a28700 1 -- 172.21.15.67:0/3319451878 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc98c022470 con 0x7fc99c0ff670 2024-04-22T23:31:32.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.868+0000 7fc9a0a28700 1 --2- 172.21.15.67:0/3319451878 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc98806ecc0 0x7fc988071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:31:32.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.868+0000 7fc9a0a28700 1 -- 172.21.15.67:0/3319451878 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6704+0+0 (secure 0 0 0) 0x7fc98c0904b0 con 0x7fc99c0ff670 2024-04-22T23:31:32.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.868+0000 7fc99a7fc700 1 --2- 172.21.15.67:0/3319451878 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc98806ecc0 0x7fc988071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:31:32.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.869+0000 7fc99a7fc700 1 --2- 172.21.15.67:0/3319451878 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc98806ecc0 0x7fc988071180 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc99c198210 tx=0x7fc984006d20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:31:32.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:32.873+0000 7fc9a0a28700 1 -- 172.21.15.67:0/3319451878 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc98c059250 con 0x7fc99c0ff670 2024-04-22T23:31:33.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.104+0000 7fc9a1a2a700 1 -- 172.21.15.67:0/3319451878 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc99c197f80 con 0x7fc98806ecc0 2024-04-22T23:31:33.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.105+0000 7fc9a0a28700 1 -- 172.21.15.67:0/3319451878 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fc99c197f80 con 0x7fc98806ecc0 2024-04-22T23:31:33.102 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:31:33.102 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:31:33.102 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:31:33.102 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:31:33.102 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:31:33.103 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:31:33.103 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:31:33.103 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr" 2024-04-22T23:31:33.103 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:31:33.103 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "10/23 daemons upgraded", 2024-04-22T23:31:33.103 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading osd daemons", 2024-04-22T23:31:33.103 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:31:33.103 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:31:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.108+0000 7fc9a1a2a700 1 -- 172.21.15.67:0/3319451878 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc98806ecc0 msgr2=0x7fc988071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:31:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.108+0000 7fc9a1a2a700 1 --2- 172.21.15.67:0/3319451878 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc98806ecc0 0x7fc988071180 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc99c198210 tx=0x7fc984006d20 comp rx=0 tx=0).stop 2024-04-22T23:31:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.108+0000 7fc9a1a2a700 1 -- 172.21.15.67:0/3319451878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc99c0ff670 msgr2=0x7fc99c19c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:31:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.108+0000 7fc9a1a2a700 1 --2- 172.21.15.67:0/3319451878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc99c0ff670 0x7fc99c19c9f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc98c006010 tx=0x7fc98c00be20 comp rx=0 tx=0).stop 2024-04-22T23:31:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.108+0000 7fc9a1a2a700 1 -- 172.21.15.67:0/3319451878 shutdown_connections 2024-04-22T23:31:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.108+0000 7fc9a1a2a700 1 --2- 172.21.15.67:0/3319451878 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc98806ecc0 0x7fc988071180 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.108+0000 7fc9a1a2a700 1 --2- 172.21.15.67:0/3319451878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc99c0ff670 0x7fc99c19c9f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.108+0000 7fc9a1a2a700 1 --2- 172.21.15.67:0/3319451878 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc99c100020 0x7fc99c19cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.108+0000 7fc9a1a2a700 1 -- 172.21.15.67:0/3319451878 >> 172.21.15.67:0/3319451878 conn(0x7fc99c074e30 msgr2=0x7fc99c106170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:31:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.108+0000 7fc9a1a2a700 1 -- 172.21.15.67:0/3319451878 shutdown_connections 2024-04-22T23:31:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.108+0000 7fc9a1a2a700 1 -- 172.21.15.67:0/3319451878 wait complete. 2024-04-22T23:31:33.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.215+0000 7f654897c700 1 -- 172.21.15.67:0/1204790050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6544102720 msgr2=0x7f654410ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:31:33.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.215+0000 7f654897c700 1 --2- 172.21.15.67:0/1204790050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6544102720 0x7f654410ac10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6538009910 tx=0x7f6538009c20 comp rx=0 tx=0).stop 2024-04-22T23:31:33.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.215+0000 7f654897c700 1 -- 172.21.15.67:0/1204790050 shutdown_connections 2024-04-22T23:31:33.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.215+0000 7f654897c700 1 --2- 172.21.15.67:0/1204790050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6544102720 0x7f654410ac10 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:33.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.215+0000 7f654897c700 1 --2- 172.21.15.67:0/1204790050 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6544101e00 0x7f65441021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:33.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.215+0000 7f654897c700 1 -- 172.21.15.67:0/1204790050 >> 172.21.15.67:0/1204790050 conn(0x7f65440fb650 msgr2=0x7f65440fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:31:33.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.215+0000 7f654897c700 1 -- 172.21.15.67:0/1204790050 shutdown_connections 2024-04-22T23:31:33.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.215+0000 7f654897c700 1 -- 172.21.15.67:0/1204790050 wait complete. 2024-04-22T23:31:33.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.215+0000 7f654897c700 1 Processor -- start 2024-04-22T23:31:33.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.216+0000 7f654897c700 1 -- start start 2024-04-22T23:31:33.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.216+0000 7f654897c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6544101e00 0x7f65440ffa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:31:33.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.216+0000 7f654897c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6544102720 0x7f65440fffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:31:33.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.216+0000 7f654897c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65441004e0 con 0x7f6544102720 2024-04-22T23:31:33.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.216+0000 7f654897c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6544100620 con 0x7f6544101e00 2024-04-22T23:31:33.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.216+0000 7f6541d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6544102720 0x7f65440fffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:31:33.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.216+0000 7f6541d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6544102720 0x7f65440fffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60450/0 (socket says 172.21.15.67:60450) 2024-04-22T23:31:33.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.216+0000 7f6541d9b700 1 -- 172.21.15.67:0/2377634406 learned_addr learned my addr 172.21.15.67:0/2377634406 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:31:33.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.216+0000 7f654259c700 1 --2- 172.21.15.67:0/2377634406 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6544101e00 0x7f65440ffa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:31:33.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.216+0000 7f6541d9b700 1 -- 172.21.15.67:0/2377634406 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6544101e00 msgr2=0x7f65440ffa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:31:33.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.216+0000 7f6541d9b700 1 --2- 172.21.15.67:0/2377634406 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6544101e00 0x7f65440ffa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:33.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.216+0000 7f6541d9b700 1 -- 172.21.15.67:0/2377634406 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65380095f0 con 0x7f6544102720 2024-04-22T23:31:33.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.216+0000 7f6541d9b700 1 --2- 172.21.15.67:0/2377634406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6544102720 0x7f65440fffa0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f653800bd70 tx=0x7f653800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:31:33.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.217+0000 7f65337fe700 1 -- 172.21.15.67:0/2377634406 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f653801d070 con 0x7f6544102720 2024-04-22T23:31:33.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.217+0000 7f654897c700 1 -- 172.21.15.67:0/2377634406 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65441008a0 con 0x7f6544102720 2024-04-22T23:31:33.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.217+0000 7f65337fe700 1 -- 172.21.15.67:0/2377634406 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6538005dc0 con 0x7f6544102720 2024-04-22T23:31:33.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.217+0000 7f65337fe700 1 -- 172.21.15.67:0/2377634406 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6538017400 con 0x7f6544102720 2024-04-22T23:31:33.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.217+0000 7f654897c700 1 -- 172.21.15.67:0/2377634406 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65441a2dc0 con 0x7f6544102720 2024-04-22T23:31:33.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.217+0000 7f654897c700 1 -- 172.21.15.67:0/2377634406 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6544108310 con 0x7f6544102720 2024-04-22T23:31:33.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.218+0000 7f65337fe700 1 -- 172.21.15.67:0/2377634406 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f653800f460 con 0x7f6544102720 2024-04-22T23:31:33.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.219+0000 7f65337fe700 1 --2- 172.21.15.67:0/2377634406 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f652c06ecc0 0x7f652c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:31:33.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.219+0000 7f65337fe700 1 -- 172.21.15.67:0/2377634406 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6704+0+0 (secure 0 0 0) 0x7f65380902a0 con 0x7f6544102720 2024-04-22T23:31:33.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.219+0000 7f654259c700 1 --2- 172.21.15.67:0/2377634406 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f652c06ecc0 0x7f652c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:31:33.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.219+0000 7f654259c700 1 --2- 172.21.15.67:0/2377634406 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f652c06ecc0 0x7f652c071180 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6534005950 tx=0x7f653400b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:31:33.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.223+0000 7f65337fe700 1 -- 172.21.15.67:0/2377634406 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f6538058f90 con 0x7f6544102720 2024-04-22T23:31:33.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:33 smithi134 ceph-mon[85630]: from='client.34324 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:31:33.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:33 smithi134 ceph-mon[85630]: pgmap v139: 65 pgs: 65 active+clean; 310 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 52 KiB/s rd, 5 op/s 2024-04-22T23:31:33.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:33 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/667391453' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:31:33.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:33 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2440955730' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:31:33.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:33 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-22T23:31:33.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.527+0000 7f654897c700 1 -- 172.21.15.67:0/2377634406 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f6544100f90 con 0x7f6544102720 2024-04-22T23:31:33.524 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:33 smithi067 ceph-mon[112438]: from='client.34324 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:31:33.524 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:33 smithi067 ceph-mon[112438]: pgmap v139: 65 pgs: 65 active+clean; 310 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 52 KiB/s rd, 5 op/s 2024-04-22T23:31:33.524 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:33 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/667391453' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:31:33.524 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:33 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2440955730' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:31:33.524 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:33 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-22T23:31:33.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.527+0000 7f65337fe700 1 -- 172.21.15.67:0/2377634406 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f653805c650 con 0x7f6544102720 2024-04-22T23:31:33.525 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T23:31:33.525 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:31:33.525 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:31:33.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.529+0000 7f654897c700 1 -- 172.21.15.67:0/2377634406 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f652c06ecc0 msgr2=0x7f652c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:31:33.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.529+0000 7f654897c700 1 --2- 172.21.15.67:0/2377634406 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f652c06ecc0 0x7f652c071180 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6534005950 tx=0x7f653400b410 comp rx=0 tx=0).stop 2024-04-22T23:31:33.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.529+0000 7f654897c700 1 -- 172.21.15.67:0/2377634406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6544102720 msgr2=0x7f65440fffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:31:33.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.529+0000 7f654897c700 1 --2- 172.21.15.67:0/2377634406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6544102720 0x7f65440fffa0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f653800bd70 tx=0x7f653800bda0 comp rx=0 tx=0).stop 2024-04-22T23:31:33.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.529+0000 7f654897c700 1 -- 172.21.15.67:0/2377634406 shutdown_connections 2024-04-22T23:31:33.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.529+0000 7f654897c700 1 --2- 172.21.15.67:0/2377634406 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f652c06ecc0 0x7f652c071180 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:33.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.529+0000 7f654897c700 1 --2- 172.21.15.67:0/2377634406 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6544101e00 0x7f65440ffa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:33.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.529+0000 7f654897c700 1 --2- 172.21.15.67:0/2377634406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6544102720 0x7f65440fffa0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:31:33.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.529+0000 7f654897c700 1 -- 172.21.15.67:0/2377634406 >> 172.21.15.67:0/2377634406 conn(0x7f65440fb650 msgr2=0x7f6544105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:31:33.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.529+0000 7f654897c700 1 -- 172.21.15.67:0/2377634406 shutdown_connections 2024-04-22T23:31:33.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:31:33.529+0000 7f654897c700 1 -- 172.21.15.67:0/2377634406 wait complete. 2024-04-22T23:31:34.371 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:34 smithi134 ceph-mon[85630]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-22T23:31:34.371 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:34 smithi134 ceph-mon[85630]: Upgrade: osd.4 is safe to restart 2024-04-22T23:31:34.371 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:34 smithi134 ceph-mon[85630]: from='client.34336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:31:34.371 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:34 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2377634406' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:31:34.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:34 smithi067 ceph-mon[112438]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-22T23:31:34.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:34 smithi067 ceph-mon[112438]: Upgrade: osd.4 is safe to restart 2024-04-22T23:31:34.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:34 smithi067 ceph-mon[112438]: from='client.34336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:31:34.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:34 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2377634406' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:31:35.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:35 smithi134 ceph-mon[85630]: pgmap v140: 65 pgs: 65 active+clean; 310 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:31:35.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:35 smithi134 ceph-mon[85630]: Upgrade: Updating osd.4 2024-04-22T23:31:35.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:35 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:35.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:35 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T23:31:35.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:35 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:31:35.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:35 smithi134 ceph-mon[85630]: Deploying daemon osd.4 on smithi134 2024-04-22T23:31:35.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:35 smithi067 ceph-mon[112438]: pgmap v140: 65 pgs: 65 active+clean; 310 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:31:35.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:35 smithi067 ceph-mon[112438]: Upgrade: Updating osd.4 2024-04-22T23:31:35.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:35 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:35.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:35 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T23:31:35.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:35 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:31:35.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:35 smithi067 ceph-mon[112438]: Deploying daemon osd.4 on smithi134 2024-04-22T23:31:37.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:37 smithi134 ceph-mon[85630]: pgmap v141: 65 pgs: 65 active+clean; 310 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:31:37.740 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:37 smithi134 systemd[1]: Stopping Ceph osd.4 for d46c346e-00fd-11ef-bc93-c7b262605968... 2024-04-22T23:31:37.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:37 smithi067 ceph-mon[112438]: pgmap v141: 65 pgs: 65 active+clean; 310 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:31:38.240 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:37 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4[47414]: 2024-04-22T23:31:37.834+0000 7f9e7acae700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T23:31:38.240 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:37 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4[47414]: 2024-04-22T23:31:37.834+0000 7f9e7acae700 -1 osd.4 77 *** Got signal Terminated *** 2024-04-22T23:31:38.240 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:37 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4[47414]: 2024-04-22T23:31:37.834+0000 7f9e7acae700 -1 osd.4 77 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T23:31:38.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:38 smithi134 ceph-mon[85630]: osd.4 marked itself down and dead 2024-04-22T23:31:38.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:38 smithi067 ceph-mon[112438]: osd.4 marked itself down and dead 2024-04-22T23:31:39.107 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:38 smithi134 podman[99521]: 2024-04-22 23:31:38.831196956 +0000 UTC m=+1.118349019 container died c7586562ffa6d8e1530701f26e129f0ac0d90b210335e2199be3c3a2327933a7 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, RELEASE=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222) 2024-04-22T23:31:39.414 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:39 smithi134 podman[99521]: 2024-04-22 23:31:39.201187182 +0000 UTC m=+1.488339241 container cleanup c7586562ffa6d8e1530701f26e129f0ac0d90b210335e2199be3c3a2327933a7 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-22T23:31:39.415 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:39 smithi134 bash[99521]: ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4 2024-04-22T23:31:39.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:39 smithi134 ceph-mon[85630]: pgmap v142: 65 pgs: 65 active+clean; 310 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:31:39.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:39 smithi134 ceph-mon[85630]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-22T23:31:39.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:39 smithi134 ceph-mon[85630]: osdmap e78: 6 total, 5 up, 6 in 2024-04-22T23:31:39.741 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:39 smithi134 podman[99549]: 2024-04-22 23:31:39.469425342 +0000 UTC m=+0.633856335 container remove c7586562ffa6d8e1530701f26e129f0ac0d90b210335e2199be3c3a2327933a7 (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD) 2024-04-22T23:31:39.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:39 smithi067 ceph-mon[112438]: pgmap v142: 65 pgs: 65 active+clean; 310 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:31:39.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:39 smithi067 ceph-mon[112438]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-22T23:31:39.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:39 smithi067 ceph-mon[112438]: osdmap e78: 6 total, 5 up, 6 in 2024-04-22T23:31:40.110 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:39 smithi134 podman[99644]: 2024-04-22 23:31:39.715892309 +0000 UTC m=+0.018022275 image pull a0434699c1d76af4eea1fc95ff0a22ea73d1e04e609464fbf658d6f39bc7f2fc quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c 2024-04-22T23:31:40.111 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:39 smithi134 podman[99644]: 2024-04-22 23:31:39.828551782 +0000 UTC m=+0.130681716 container create 5f1199e130e00893b7003d83f494a677f902a2e4dee117cc68c125a94ca27782 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, ceph=True, org.label-schema.build-date=20240416, GIT_CLEAN=True) 2024-04-22T23:31:40.111 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:40 smithi134 podman[99644]: 2024-04-22 23:31:40.039964164 +0000 UTC m=+0.342094099 container init 5f1199e130e00893b7003d83f494a677f902a2e4dee117cc68c125a94ca27782 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T23:31:40.111 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:40 smithi134 podman[99644]: 2024-04-22 23:31:40.043176304 +0000 UTC m=+0.345306239 container start 5f1199e130e00893b7003d83f494a677f902a2e4dee117cc68c125a94ca27782 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5) 2024-04-22T23:31:40.111 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:40 smithi134 podman[99644]: 2024-04-22 23:31:40.114620075 +0000 UTC m=+0.416750010 container attach 5f1199e130e00893b7003d83f494a677f902a2e4dee117cc68c125a94ca27782 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4-deactivate, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.build-date=20240416, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-22T23:31:40.425 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:40 smithi134 podman[99644]: 2024-04-22 23:31:40.33889094 +0000 UTC m=+0.641020878 container died 5f1199e130e00893b7003d83f494a677f902a2e4dee117cc68c125a94ca27782 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-22T23:31:40.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:40 smithi134 ceph-mon[85630]: osdmap e79: 6 total, 5 up, 6 in 2024-04-22T23:31:40.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:40 smithi067 ceph-mon[112438]: osdmap e79: 6 total, 5 up, 6 in 2024-04-22T23:31:41.240 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:40 smithi134 podman[99670]: 2024-04-22 23:31:40.901519328 +0000 UTC m=+0.556989721 container remove 5f1199e130e00893b7003d83f494a677f902a2e4dee117cc68c125a94ca27782 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-22T23:31:41.240 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:40 smithi134 systemd[1]: ceph-d46c346e-00fd-11ef-bc93-c7b262605968@osd.4.service: Deactivated successfully. 2024-04-22T23:31:41.240 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:40 smithi134 systemd[1]: Stopped Ceph osd.4 for d46c346e-00fd-11ef-bc93-c7b262605968. 2024-04-22T23:31:41.240 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:40 smithi134 systemd[1]: ceph-d46c346e-00fd-11ef-bc93-c7b262605968@osd.4.service: Consumed 56.506s CPU time. 2024-04-22T23:31:41.240 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:41 smithi134 systemd[1]: Starting Ceph osd.4 for d46c346e-00fd-11ef-bc93-c7b262605968... 2024-04-22T23:31:41.610 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:41 smithi134 ceph-mon[85630]: pgmap v145: 65 pgs: 11 peering, 3 stale+active+clean, 51 active+clean; 310 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 3 op/s 2024-04-22T23:31:41.610 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:41 smithi134 ceph-mon[85630]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-04-22T23:31:41.610 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:41.611 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:31:41.611 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:41 smithi134 podman[99774]: 2024-04-22 23:31:41.246181225 +0000 UTC m=+0.017906046 image pull a0434699c1d76af4eea1fc95ff0a22ea73d1e04e609464fbf658d6f39bc7f2fc quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c 2024-04-22T23:31:41.611 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:41 smithi134 podman[99774]: 2024-04-22 23:31:41.360920565 +0000 UTC m=+0.132645387 container create b2223ccd6b5cb68279ab24ec3f59a7940e38e40ec5ef242c0828d1933d6a070f (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4-activate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-22T23:31:41.611 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:41 smithi134 podman[99774]: 2024-04-22 23:31:41.614164467 +0000 UTC m=+0.385889300 container init b2223ccd6b5cb68279ab24ec3f59a7940e38e40ec5ef242c0828d1933d6a070f (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4-activate, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True) 2024-04-22T23:31:41.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:41 smithi067 ceph-mon[112438]: pgmap v145: 65 pgs: 11 peering, 3 stale+active+clean, 51 active+clean; 310 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 3 op/s 2024-04-22T23:31:41.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:41 smithi067 ceph-mon[112438]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-04-22T23:31:41.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:41.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:31:41.990 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:41 smithi134 podman[99774]: 2024-04-22 23:31:41.618052804 +0000 UTC m=+0.389777633 container start b2223ccd6b5cb68279ab24ec3f59a7940e38e40ec5ef242c0828d1933d6a070f (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4-activate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-22T23:31:42.068 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:41 smithi134 podman[99774]: 2024-04-22 23:31:41.722044172 +0000 UTC m=+0.493769011 container attach b2223ccd6b5cb68279ab24ec3f59a7940e38e40ec5ef242c0828d1933d6a070f (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-22T23:31:42.413 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:42 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4-activate[99788]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T23:31:42.413 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:42 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4-activate[99788]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T23:31:42.414 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:42 smithi134 bash[99774]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T23:31:42.414 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:42 smithi134 bash[99774]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T23:31:42.414 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:42 smithi134 bash[99774]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-22T23:31:42.414 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:42 smithi134 bash[99774]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-22T23:31:42.414 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:42 smithi134 bash[99774]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-22T23:31:42.414 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:42 smithi134 bash[99774]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-22T23:31:42.414 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:42 smithi134 bash[99774]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-4/block 2024-04-22T23:31:42.414 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:42 smithi134 bash[99774]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-22T23:31:42.414 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:42 smithi134 bash[99774]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-22T23:31:42.414 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:42 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4-activate[99788]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-22T23:31:42.414 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:42 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4-activate[99788]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-22T23:31:42.414 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:42 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4-activate[99788]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-22T23:31:42.414 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:42 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4-activate[99788]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-22T23:31:42.415 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:42 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4-activate[99788]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-4/block 2024-04-22T23:31:42.415 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:42 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4-activate[99788]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-22T23:31:42.415 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:42 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4-activate[99788]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-22T23:31:42.415 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:42 smithi134 podman[99774]: 2024-04-22 23:31:42.126997184 +0000 UTC m=+0.898722006 container died b2223ccd6b5cb68279ab24ec3f59a7940e38e40ec5ef242c0828d1933d6a070f (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4-activate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-22T23:31:42.704 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:42 smithi134 ceph-mon[85630]: Health check failed: Degraded data redundancy: 28/363 objects degraded (7.713%), 7 pgs degraded (PG_DEGRADED) 2024-04-22T23:31:42.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:42 smithi067 ceph-mon[112438]: Health check failed: Degraded data redundancy: 28/363 objects degraded (7.713%), 7 pgs degraded (PG_DEGRADED) 2024-04-22T23:31:42.990 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:42 smithi134 podman[99931]: 2024-04-22 23:31:42.70891928 +0000 UTC m=+0.574135853 container remove b2223ccd6b5cb68279ab24ec3f59a7940e38e40ec5ef242c0828d1933d6a070f (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4-activate, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-22T23:31:42.990 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:42 smithi134 podman[99995]: 2024-04-22 23:31:42.824094718 +0000 UTC m=+0.018364577 image pull a0434699c1d76af4eea1fc95ff0a22ea73d1e04e609464fbf658d6f39bc7f2fc quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c 2024-04-22T23:31:42.990 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:42 smithi134 podman[99995]: 2024-04-22 23:31:42.926348451 +0000 UTC m=+0.120618299 container create 61da66bd188211774132ac6d83cc81c2bfb0ff2e79b05f481c19da2281f379a1 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-22T23:31:43.480 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:43 smithi134 podman[99995]: 2024-04-22 23:31:43.153599356 +0000 UTC m=+0.347869207 container init 61da66bd188211774132ac6d83cc81c2bfb0ff2e79b05f481c19da2281f379a1 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240416, RELEASE=wip-adk-testing-2024-04-22-1618-43be020) 2024-04-22T23:31:43.481 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:43 smithi134 podman[99995]: 2024-04-22 23:31:43.156658344 +0000 UTC m=+0.350928193 container start 61da66bd188211774132ac6d83cc81c2bfb0ff2e79b05f481c19da2281f379a1 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=) 2024-04-22T23:31:43.481 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:43 smithi134 ceph-osd[100014]: -- 172.21.15.134:0/3526545114 <== mon.1 v2:172.21.15.134:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bebcb41680 con 0x55bebcb0ac00 2024-04-22T23:31:43.481 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:43 smithi134 bash[99995]: 61da66bd188211774132ac6d83cc81c2bfb0ff2e79b05f481c19da2281f379a1 2024-04-22T23:31:43.481 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:43 smithi134 systemd[1]: Started Ceph osd.4 for d46c346e-00fd-11ef-bc93-c7b262605968. 2024-04-22T23:31:43.740 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:43 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4[100010]: 2024-04-22T23:31:43.693+0000 7fcfd68707c0 -1 Falling back to public interface 2024-04-22T23:31:43.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:43 smithi134 ceph-mon[85630]: pgmap v146: 65 pgs: 7 active+undersized+degraded, 11 peering, 1 stale+active+clean, 8 active+undersized, 38 active+clean; 310 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 28/363 objects degraded (7.713%) 2024-04-22T23:31:43.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:43 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:43.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:43 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:43.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:43 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:31:43.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:43 smithi067 ceph-mon[112438]: pgmap v146: 65 pgs: 7 active+undersized+degraded, 11 peering, 1 stale+active+clean, 8 active+undersized, 38 active+clean; 310 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 28/363 objects degraded (7.713%) 2024-04-22T23:31:43.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:43 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:43.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:43 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:43.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:43 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:31:45.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:45 smithi134 ceph-mon[85630]: pgmap v147: 65 pgs: 10 active+undersized+degraded, 11 peering, 11 active+undersized, 33 active+clean; 310 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 42/363 objects degraded (11.570%) 2024-04-22T23:31:45.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:45 smithi067 ceph-mon[112438]: pgmap v147: 65 pgs: 10 active+undersized+degraded, 11 peering, 11 active+undersized, 33 active+clean; 310 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 42/363 objects degraded (11.570%) 2024-04-22T23:31:47.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:47 smithi134 ceph-mon[85630]: pgmap v148: 65 pgs: 16 active+undersized+degraded, 16 active+undersized, 33 active+clean; 310 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 62/363 objects degraded (17.080%) 2024-04-22T23:31:47.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:47 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:47.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:47 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:47.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:47 smithi134 ceph-mon[85630]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2024-04-22T23:31:47.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:47 smithi067 ceph-mon[112438]: pgmap v148: 65 pgs: 16 active+undersized+degraded, 16 active+undersized, 33 active+clean; 310 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 62/363 objects degraded (17.080%) 2024-04-22T23:31:47.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:47 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:47.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:47 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:47.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:47 smithi067 ceph-mon[112438]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2024-04-22T23:31:48.388 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:48 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4[100010]: 2024-04-22T23:31:48.246+0000 7fcfd68707c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-04-22T23:31:48.740 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:48 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4[100010]: 2024-04-22T23:31:48.576+0000 7fcfd68707c0 -1 osd.4 77 log_to_monitors true 2024-04-22T23:31:48.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:48 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:48.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:48 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:48.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:48 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:48.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:48 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:49.720 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:49 smithi134 ceph-mon[85630]: pgmap v149: 65 pgs: 16 active+undersized+degraded, 16 active+undersized, 33 active+clean; 310 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 62/363 objects degraded (17.080%) 2024-04-22T23:31:49.720 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:49 smithi134 ceph-mon[85630]: from='osd.4 [v2:172.21.15.134:6808/3935208416,v1:172.21.15.134:6809/3935208416]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T23:31:49.720 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:49 smithi134 ceph-mon[85630]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T23:31:49.720 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 22 23:31:49 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4[100010]: 2024-04-22T23:31:49.499+0000 7fcfc7cc8700 -1 osd.4 77 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T23:31:49.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:49 smithi067 ceph-mon[112438]: pgmap v149: 65 pgs: 16 active+undersized+degraded, 16 active+undersized, 33 active+clean; 310 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 62/363 objects degraded (17.080%) 2024-04-22T23:31:49.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:49 smithi067 ceph-mon[112438]: from='osd.4 [v2:172.21.15.134:6808/3935208416,v1:172.21.15.134:6809/3935208416]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T23:31:49.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:49 smithi067 ceph-mon[112438]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T23:31:50.706 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:50 smithi134 ceph-mon[85630]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T23:31:50.707 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:50 smithi134 ceph-mon[85630]: from='osd.4 [v2:172.21.15.134:6808/3935208416,v1:172.21.15.134:6809/3935208416]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-22T23:31:50.707 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:50 smithi134 ceph-mon[85630]: osdmap e80: 6 total, 5 up, 6 in 2024-04-22T23:31:50.707 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:50 smithi134 ceph-mon[85630]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-22T23:31:50.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:50 smithi067 ceph-mon[112438]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T23:31:50.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:50 smithi067 ceph-mon[112438]: from='osd.4 [v2:172.21.15.134:6808/3935208416,v1:172.21.15.134:6809/3935208416]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-22T23:31:50.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:50 smithi067 ceph-mon[112438]: osdmap e80: 6 total, 5 up, 6 in 2024-04-22T23:31:50.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:50 smithi067 ceph-mon[112438]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-22T23:31:51.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:51 smithi134 ceph-mon[85630]: pgmap v151: 65 pgs: 16 active+undersized+degraded, 16 active+undersized, 33 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 62/363 objects degraded (17.080%) 2024-04-22T23:31:51.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:51 smithi134 ceph-mon[85630]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-22T23:31:51.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:51 smithi134 ceph-mon[85630]: osd.4 [v2:172.21.15.134:6808/3935208416,v1:172.21.15.134:6809/3935208416] boot 2024-04-22T23:31:51.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:51 smithi134 ceph-mon[85630]: osdmap e81: 6 total, 6 up, 6 in 2024-04-22T23:31:51.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:51 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T23:31:51.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:51 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:51.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:51 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:51.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:51 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:31:51.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:51 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:31:51.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:51 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:51.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:51 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:31:51.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:51 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:31:51.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:51 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:31:51.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:51 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:31:51.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:51 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-22T23:31:51.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:51 smithi134 ceph-mon[85630]: osdmap e82: 6 total, 6 up, 6 in 2024-04-22T23:31:51.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:51 smithi067 ceph-mon[112438]: pgmap v151: 65 pgs: 16 active+undersized+degraded, 16 active+undersized, 33 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 62/363 objects degraded (17.080%) 2024-04-22T23:31:51.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:51 smithi067 ceph-mon[112438]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-22T23:31:51.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:51 smithi067 ceph-mon[112438]: osd.4 [v2:172.21.15.134:6808/3935208416,v1:172.21.15.134:6809/3935208416] boot 2024-04-22T23:31:51.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:51 smithi067 ceph-mon[112438]: osdmap e81: 6 total, 6 up, 6 in 2024-04-22T23:31:51.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:51 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T23:31:51.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:51 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:51.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:51 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:51.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:51 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:31:51.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:51 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:31:51.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:51 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:51.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:51 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:31:51.792 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:51 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:31:51.792 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:51 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:31:51.792 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:51 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:31:51.792 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:51 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-22T23:31:51.792 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:51 smithi067 ceph-mon[112438]: osdmap e82: 6 total, 6 up, 6 in 2024-04-22T23:31:52.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:52 smithi067 ceph-mon[112438]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-22T23:31:52.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:52 smithi067 ceph-mon[112438]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-04-22T23:31:52.791 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:52 smithi067 ceph-mon[112438]: Health check update: Degraded data redundancy: 62/363 objects degraded (17.080%), 16 pgs degraded (PG_DEGRADED) 2024-04-22T23:31:52.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:52 smithi134 ceph-mon[85630]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-22T23:31:52.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:52 smithi134 ceph-mon[85630]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-04-22T23:31:52.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:52 smithi134 ceph-mon[85630]: Health check update: Degraded data redundancy: 62/363 objects degraded (17.080%), 16 pgs degraded (PG_DEGRADED) 2024-04-22T23:31:53.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:53 smithi067 ceph-mon[112438]: pgmap v154: 65 pgs: 9 peering, 11 active+undersized+degraded, 8 active+undersized, 37 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 3.2 KiB/s rd, 4 op/s; 39/363 objects degraded (10.744%) 2024-04-22T23:31:53.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:53 smithi134 ceph-mon[85630]: pgmap v154: 65 pgs: 9 peering, 11 active+undersized+degraded, 8 active+undersized, 37 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 3.2 KiB/s rd, 4 op/s; 39/363 objects degraded (10.744%) 2024-04-22T23:31:55.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:55 smithi067 ceph-mon[112438]: pgmap v155: 65 pgs: 9 peering, 8 active+undersized+degraded, 6 active+undersized, 42 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s; 25/363 objects degraded (6.887%) 2024-04-22T23:31:55.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:55 smithi134 ceph-mon[85630]: pgmap v155: 65 pgs: 9 peering, 8 active+undersized+degraded, 6 active+undersized, 42 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s; 25/363 objects degraded (6.887%) 2024-04-22T23:31:57.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:57 smithi067 ceph-mon[112438]: pgmap v156: 65 pgs: 9 peering, 56 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 2 op/s 2024-04-22T23:31:57.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:57 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:57.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:57 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:31:57.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:57 smithi067 ceph-mon[112438]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 25/363 objects degraded (6.887%), 8 pgs degraded) 2024-04-22T23:31:57.489 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:57 smithi134 ceph-mon[85630]: pgmap v156: 65 pgs: 9 peering, 56 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 2 op/s 2024-04-22T23:31:57.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:57 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:31:57.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:57 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:31:57.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:57 smithi134 ceph-mon[85630]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 25/363 objects degraded (6.887%), 8 pgs degraded) 2024-04-22T23:31:59.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:31:59 smithi067 ceph-mon[112438]: pgmap v157: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 3 op/s 2024-04-22T23:31:59.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:31:59 smithi134 ceph-mon[85630]: pgmap v157: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 3 op/s 2024-04-22T23:32:01.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:01 smithi067 ceph-mon[112438]: pgmap v158: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-22T23:32:01.489 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:01 smithi134 ceph-mon[85630]: pgmap v158: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-22T23:32:03.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:03 smithi067 ceph-mon[112438]: pgmap v159: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-22T23:32:03.489 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:03 smithi134 ceph-mon[85630]: pgmap v159: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-22T23:32:03.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.650+0000 7f39ea8f4700 1 -- 172.21.15.67:0/2338874828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39e4101830 msgr2=0x7f39e4105880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:03.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.650+0000 7f39ea8f4700 1 --2- 172.21.15.67:0/2338874828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39e4101830 0x7f39e4105880 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f39d4009910 tx=0x7f39d4009c20 comp rx=0 tx=0).stop 2024-04-22T23:32:03.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.650+0000 7f39ea8f4700 1 -- 172.21.15.67:0/2338874828 shutdown_connections 2024-04-22T23:32:03.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.650+0000 7f39ea8f4700 1 --2- 172.21.15.67:0/2338874828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39e4101830 0x7f39e4105880 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:03.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.650+0000 7f39ea8f4700 1 --2- 172.21.15.67:0/2338874828 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39e4100e80 0x7f39e4101260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:03.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.650+0000 7f39ea8f4700 1 -- 172.21.15.67:0/2338874828 >> 172.21.15.67:0/2338874828 conn(0x7f39e40fc710 msgr2=0x7f39e40feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:32:03.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.650+0000 7f39ea8f4700 1 -- 172.21.15.67:0/2338874828 shutdown_connections 2024-04-22T23:32:03.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.650+0000 7f39ea8f4700 1 -- 172.21.15.67:0/2338874828 wait complete. 2024-04-22T23:32:03.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.651+0000 7f39ea8f4700 1 Processor -- start 2024-04-22T23:32:03.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.651+0000 7f39ea8f4700 1 -- start start 2024-04-22T23:32:03.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.651+0000 7f39ea8f4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39e4100e80 0x7f39e4073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:03.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.651+0000 7f39ea8f4700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39e4101830 0x7f39e40735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:03.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.651+0000 7f39ea8f4700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e4073b10 con 0x7f39e4100e80 2024-04-22T23:32:03.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.651+0000 7f39ea8f4700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e4073c50 con 0x7f39e4101830 2024-04-22T23:32:03.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.652+0000 7f39e3fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39e4100e80 0x7f39e4073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:03.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.652+0000 7f39e3fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39e4100e80 0x7f39e4073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38558/0 (socket says 172.21.15.67:38558) 2024-04-22T23:32:03.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.652+0000 7f39e3fff700 1 -- 172.21.15.67:0/3020045802 learned_addr learned my addr 172.21.15.67:0/3020045802 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:32:03.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.652+0000 7f39e37fe700 1 --2- 172.21.15.67:0/3020045802 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39e4101830 0x7f39e40735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:03.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.652+0000 7f39e3fff700 1 -- 172.21.15.67:0/3020045802 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39e4101830 msgr2=0x7f39e40735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:03.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.652+0000 7f39e3fff700 1 --2- 172.21.15.67:0/3020045802 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39e4101830 0x7f39e40735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:03.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.652+0000 7f39e3fff700 1 -- 172.21.15.67:0/3020045802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39d40095f0 con 0x7f39e4100e80 2024-04-22T23:32:03.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.652+0000 7f39e37fe700 1 --2- 172.21.15.67:0/3020045802 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39e4101830 0x7f39e40735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:32:03.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.652+0000 7f39e3fff700 1 --2- 172.21.15.67:0/3020045802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39e4100e80 0x7f39e4073090 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f39cc00b4b0 tx=0x7f39cc00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:32:03.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.652+0000 7f39e17fa700 1 -- 172.21.15.67:0/3020045802 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39cc011800 con 0x7f39e4100e80 2024-04-22T23:32:03.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.652+0000 7f39ea8f4700 1 -- 172.21.15.67:0/3020045802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39e41a2b40 con 0x7f39e4100e80 2024-04-22T23:32:03.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.652+0000 7f39e17fa700 1 -- 172.21.15.67:0/3020045802 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f39cc011e20 con 0x7f39e4100e80 2024-04-22T23:32:03.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.652+0000 7f39e17fa700 1 -- 172.21.15.67:0/3020045802 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39cc00f2e0 con 0x7f39e4100e80 2024-04-22T23:32:03.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.652+0000 7f39ea8f4700 1 -- 172.21.15.67:0/3020045802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39e41a2f50 con 0x7f39e4100e80 2024-04-22T23:32:03.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.654+0000 7f39ea8f4700 1 -- 172.21.15.67:0/3020045802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39e41091b0 con 0x7f39e4100e80 2024-04-22T23:32:03.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.654+0000 7f39e17fa700 1 -- 172.21.15.67:0/3020045802 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f39cc011960 con 0x7f39e4100e80 2024-04-22T23:32:03.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.654+0000 7f39e17fa700 1 --2- 172.21.15.67:0/3020045802 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f39d006ecc0 0x7f39d0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:03.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.654+0000 7f39e17fa700 1 -- 172.21.15.67:0/3020045802 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6704+0+0 (secure 0 0 0) 0x7f39cc08eb00 con 0x7f39e4100e80 2024-04-22T23:32:03.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.655+0000 7f39e37fe700 1 --2- 172.21.15.67:0/3020045802 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f39d006ecc0 0x7f39d0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:03.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.655+0000 7f39e37fe700 1 --2- 172.21.15.67:0/3020045802 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f39d006ecc0 0x7f39d0071180 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f39e4074880 tx=0x7f39d401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:32:03.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.659+0000 7f39e17fa700 1 -- 172.21.15.67:0/3020045802 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f39cc0577f0 con 0x7f39e4100e80 2024-04-22T23:32:03.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.890+0000 7f39ea8f4700 1 -- 172.21.15.67:0/3020045802 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f39e40611d0 con 0x7f39d006ecc0 2024-04-22T23:32:03.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.891+0000 7f39e17fa700 1 -- 172.21.15.67:0/3020045802 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f39e40611d0 con 0x7f39d006ecc0 2024-04-22T23:32:03.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.893+0000 7f39ea8f4700 1 -- 172.21.15.67:0/3020045802 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f39d006ecc0 msgr2=0x7f39d0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:03.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.893+0000 7f39ea8f4700 1 --2- 172.21.15.67:0/3020045802 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f39d006ecc0 0x7f39d0071180 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f39e4074880 tx=0x7f39d401a040 comp rx=0 tx=0).stop 2024-04-22T23:32:03.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.893+0000 7f39ea8f4700 1 -- 172.21.15.67:0/3020045802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39e4100e80 msgr2=0x7f39e4073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:03.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.893+0000 7f39ea8f4700 1 --2- 172.21.15.67:0/3020045802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39e4100e80 0x7f39e4073090 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f39cc00b4b0 tx=0x7f39cc00b870 comp rx=0 tx=0).stop 2024-04-22T23:32:03.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.894+0000 7f39ea8f4700 1 -- 172.21.15.67:0/3020045802 shutdown_connections 2024-04-22T23:32:03.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.894+0000 7f39ea8f4700 1 --2- 172.21.15.67:0/3020045802 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f39d006ecc0 0x7f39d0071180 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:03.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.894+0000 7f39ea8f4700 1 --2- 172.21.15.67:0/3020045802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39e4100e80 0x7f39e4073090 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:03.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.894+0000 7f39ea8f4700 1 --2- 172.21.15.67:0/3020045802 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39e4101830 0x7f39e40735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:03.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.894+0000 7f39ea8f4700 1 -- 172.21.15.67:0/3020045802 >> 172.21.15.67:0/3020045802 conn(0x7f39e40fc710 msgr2=0x7f39e40fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:32:03.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.894+0000 7f39ea8f4700 1 -- 172.21.15.67:0/3020045802 shutdown_connections 2024-04-22T23:32:03.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:03.894+0000 7f39ea8f4700 1 -- 172.21.15.67:0/3020045802 wait complete. 2024-04-22T23:32:03.901 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:32:04.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.002+0000 7f6c305b0700 1 -- 172.21.15.67:0/763097919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c28100ea0 msgr2=0x7f6c28101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:04.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.002+0000 7f6c305b0700 1 --2- 172.21.15.67:0/763097919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c28100ea0 0x7f6c28101280 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6c24009910 tx=0x7f6c24009c20 comp rx=0 tx=0).stop 2024-04-22T23:32:04.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.003+0000 7f6c305b0700 1 -- 172.21.15.67:0/763097919 shutdown_connections 2024-04-22T23:32:04.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.003+0000 7f6c305b0700 1 --2- 172.21.15.67:0/763097919 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c28101850 0x7f6c281058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:04.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.003+0000 7f6c305b0700 1 --2- 172.21.15.67:0/763097919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c28100ea0 0x7f6c28101280 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:04.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.003+0000 7f6c305b0700 1 -- 172.21.15.67:0/763097919 >> 172.21.15.67:0/763097919 conn(0x7f6c280fc750 msgr2=0x7f6c280feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:32:04.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.003+0000 7f6c305b0700 1 -- 172.21.15.67:0/763097919 shutdown_connections 2024-04-22T23:32:04.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.003+0000 7f6c305b0700 1 -- 172.21.15.67:0/763097919 wait complete. 2024-04-22T23:32:04.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.003+0000 7f6c305b0700 1 Processor -- start 2024-04-22T23:32:04.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.003+0000 7f6c305b0700 1 -- start start 2024-04-22T23:32:04.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.004+0000 7f6c305b0700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c28100ea0 0x7f6c2819c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:04.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.004+0000 7f6c305b0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c28101850 0x7f6c2819cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:04.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.004+0000 7f6c305b0700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2819d2c0 con 0x7f6c28101850 2024-04-22T23:32:04.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.004+0000 7f6c305b0700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c28196770 con 0x7f6c28100ea0 2024-04-22T23:32:04.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.004+0000 7f6c2db4b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c28101850 0x7f6c2819cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:04.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.004+0000 7f6c2db4b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c28101850 0x7f6c2819cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38570/0 (socket says 172.21.15.67:38570) 2024-04-22T23:32:04.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.004+0000 7f6c2db4b700 1 -- 172.21.15.67:0/442033183 learned_addr learned my addr 172.21.15.67:0/442033183 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:32:04.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.004+0000 7f6c2e34c700 1 --2- 172.21.15.67:0/442033183 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c28100ea0 0x7f6c2819c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:04.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.004+0000 7f6c2db4b700 1 -- 172.21.15.67:0/442033183 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c28100ea0 msgr2=0x7f6c2819c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:04.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.004+0000 7f6c2db4b700 1 --2- 172.21.15.67:0/442033183 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c28100ea0 0x7f6c2819c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:04.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.004+0000 7f6c2db4b700 1 -- 172.21.15.67:0/442033183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c240095f0 con 0x7f6c28101850 2024-04-22T23:32:04.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.004+0000 7f6c2e34c700 1 --2- 172.21.15.67:0/442033183 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c28100ea0 0x7f6c2819c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:32:04.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.004+0000 7f6c2db4b700 1 --2- 172.21.15.67:0/442033183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c28101850 0x7f6c2819cc30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6c1800d900 tx=0x7f6c1800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:32:04.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.004+0000 7f6c1f7fe700 1 -- 172.21.15.67:0/442033183 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c18009900 con 0x7f6c28101850 2024-04-22T23:32:04.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.004+0000 7f6c1f7fe700 1 -- 172.21.15.67:0/442033183 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6c18010460 con 0x7f6c28101850 2024-04-22T23:32:04.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.004+0000 7f6c305b0700 1 -- 172.21.15.67:0/442033183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c28196a50 con 0x7f6c28101850 2024-04-22T23:32:04.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.004+0000 7f6c1f7fe700 1 -- 172.21.15.67:0/442033183 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c1800f5f0 con 0x7f6c28101850 2024-04-22T23:32:04.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.004+0000 7f6c305b0700 1 -- 172.21.15.67:0/442033183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c28196fa0 con 0x7f6c28101850 2024-04-22T23:32:04.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.005+0000 7f6c305b0700 1 -- 172.21.15.67:0/442033183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c28109240 con 0x7f6c28101850 2024-04-22T23:32:04.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.006+0000 7f6c1f7fe700 1 -- 172.21.15.67:0/442033183 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6c180105d0 con 0x7f6c28101850 2024-04-22T23:32:04.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.006+0000 7f6c1f7fe700 1 --2- 172.21.15.67:0/442033183 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6c1406ecc0 0x7f6c14071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:04.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.007+0000 7f6c1f7fe700 1 -- 172.21.15.67:0/442033183 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6704+0+0 (secure 0 0 0) 0x7f6c1808eb80 con 0x7f6c28101850 2024-04-22T23:32:04.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.007+0000 7f6c2e34c700 1 --2- 172.21.15.67:0/442033183 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6c1406ecc0 0x7f6c14071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:04.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.007+0000 7f6c2e34c700 1 --2- 172.21.15.67:0/442033183 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6c1406ecc0 0x7f6c14071180 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6c2400b5c0 tx=0x7f6c240053a0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:32:04.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.011+0000 7f6c1f7fe700 1 -- 172.21.15.67:0/442033183 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f6c1805b3a0 con 0x7f6c28101850 2024-04-22T23:32:04.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.246+0000 7f6c305b0700 1 -- 172.21.15.67:0/442033183 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6c28197280 con 0x7f6c1406ecc0 2024-04-22T23:32:04.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.247+0000 7f6c1f7fe700 1 -- 172.21.15.67:0/442033183 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f6c28197280 con 0x7f6c1406ecc0 2024-04-22T23:32:04.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.249+0000 7f6c305b0700 1 -- 172.21.15.67:0/442033183 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6c1406ecc0 msgr2=0x7f6c14071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:04.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.249+0000 7f6c305b0700 1 --2- 172.21.15.67:0/442033183 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6c1406ecc0 0x7f6c14071180 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6c2400b5c0 tx=0x7f6c240053a0 comp rx=0 tx=0).stop 2024-04-22T23:32:04.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.249+0000 7f6c305b0700 1 -- 172.21.15.67:0/442033183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c28101850 msgr2=0x7f6c2819cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:04.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.249+0000 7f6c305b0700 1 --2- 172.21.15.67:0/442033183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c28101850 0x7f6c2819cc30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6c1800d900 tx=0x7f6c1800dcc0 comp rx=0 tx=0).stop 2024-04-22T23:32:04.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.249+0000 7f6c305b0700 1 -- 172.21.15.67:0/442033183 shutdown_connections 2024-04-22T23:32:04.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.249+0000 7f6c305b0700 1 --2- 172.21.15.67:0/442033183 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6c1406ecc0 0x7f6c14071180 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:04.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.249+0000 7f6c305b0700 1 --2- 172.21.15.67:0/442033183 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c28100ea0 0x7f6c2819c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:04.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.249+0000 7f6c305b0700 1 --2- 172.21.15.67:0/442033183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c28101850 0x7f6c2819cc30 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:04.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.249+0000 7f6c305b0700 1 -- 172.21.15.67:0/442033183 >> 172.21.15.67:0/442033183 conn(0x7f6c280fc750 msgr2=0x7f6c280fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:32:04.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.249+0000 7f6c305b0700 1 -- 172.21.15.67:0/442033183 shutdown_connections 2024-04-22T23:32:04.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.249+0000 7f6c305b0700 1 -- 172.21.15.67:0/442033183 wait complete. 2024-04-22T23:32:04.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.357+0000 7f66c9f32700 1 -- 172.21.15.67:0/3942306084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66c4102720 msgr2=0x7f66c410ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:04.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.357+0000 7f66c9f32700 1 --2- 172.21.15.67:0/3942306084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66c4102720 0x7f66c410ac10 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f66ac009910 tx=0x7f66ac009c20 comp rx=0 tx=0).stop 2024-04-22T23:32:04.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.357+0000 7f66c9f32700 1 -- 172.21.15.67:0/3942306084 shutdown_connections 2024-04-22T23:32:04.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.357+0000 7f66c9f32700 1 --2- 172.21.15.67:0/3942306084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66c4102720 0x7f66c410ac10 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:04.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.357+0000 7f66c9f32700 1 --2- 172.21.15.67:0/3942306084 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66c4101e00 0x7f66c41021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:04.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.357+0000 7f66c9f32700 1 -- 172.21.15.67:0/3942306084 >> 172.21.15.67:0/3942306084 conn(0x7f66c40fb650 msgr2=0x7f66c40fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:32:04.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.357+0000 7f66c9f32700 1 -- 172.21.15.67:0/3942306084 shutdown_connections 2024-04-22T23:32:04.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.357+0000 7f66c9f32700 1 -- 172.21.15.67:0/3942306084 wait complete. 2024-04-22T23:32:04.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.357+0000 7f66c9f32700 1 Processor -- start 2024-04-22T23:32:04.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.357+0000 7f66c9f32700 1 -- start start 2024-04-22T23:32:04.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.358+0000 7f66c9f32700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66c4101e00 0x7f66c419e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:04.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.358+0000 7f66c9f32700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66c4102720 0x7f66c419edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:04.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.358+0000 7f66c9f32700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c419f3b0 con 0x7f66c4102720 2024-04-22T23:32:04.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.358+0000 7f66c9f32700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c41988f0 con 0x7f66c4101e00 2024-04-22T23:32:04.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.358+0000 7f66c2ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66c4102720 0x7f66c419edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:04.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.358+0000 7f66c2ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66c4102720 0x7f66c419edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38592/0 (socket says 172.21.15.67:38592) 2024-04-22T23:32:04.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.358+0000 7f66c2ffd700 1 -- 172.21.15.67:0/3752549757 learned_addr learned my addr 172.21.15.67:0/3752549757 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:32:04.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.358+0000 7f66c37fe700 1 --2- 172.21.15.67:0/3752549757 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66c4101e00 0x7f66c419e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:04.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.358+0000 7f66c2ffd700 1 -- 172.21.15.67:0/3752549757 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66c4101e00 msgr2=0x7f66c419e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:04.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.358+0000 7f66c2ffd700 1 --2- 172.21.15.67:0/3752549757 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66c4101e00 0x7f66c419e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:04.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.358+0000 7f66c2ffd700 1 -- 172.21.15.67:0/3752549757 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66ac0095f0 con 0x7f66c4102720 2024-04-22T23:32:04.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.358+0000 7f66c37fe700 1 --2- 172.21.15.67:0/3752549757 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66c4101e00 0x7f66c419e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:32:04.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.358+0000 7f66c2ffd700 1 --2- 172.21.15.67:0/3752549757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66c4102720 0x7f66c419edb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f66ac00bd70 tx=0x7f66ac00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:32:04.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.358+0000 7f66c0ff9700 1 -- 172.21.15.67:0/3752549757 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66ac01d070 con 0x7f66c4102720 2024-04-22T23:32:04.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.359+0000 7f66c0ff9700 1 -- 172.21.15.67:0/3752549757 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f66ac00f460 con 0x7f66c4102720 2024-04-22T23:32:04.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.359+0000 7f66c9f32700 1 -- 172.21.15.67:0/3752549757 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66c4198b70 con 0x7f66c4102720 2024-04-22T23:32:04.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.359+0000 7f66c0ff9700 1 -- 172.21.15.67:0/3752549757 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66ac017630 con 0x7f66c4102720 2024-04-22T23:32:04.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.359+0000 7f66c9f32700 1 -- 172.21.15.67:0/3752549757 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66c4199060 con 0x7f66c4102720 2024-04-22T23:32:04.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.360+0000 7f66c9f32700 1 -- 172.21.15.67:0/3752549757 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66c410a770 con 0x7f66c4102720 2024-04-22T23:32:04.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.360+0000 7f66c0ff9700 1 -- 172.21.15.67:0/3752549757 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f66ac00fab0 con 0x7f66c4102720 2024-04-22T23:32:04.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.361+0000 7f66c0ff9700 1 --2- 172.21.15.67:0/3752549757 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f66b006ecc0 0x7f66b0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:04.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.361+0000 7f66c0ff9700 1 -- 172.21.15.67:0/3752549757 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6704+0+0 (secure 0 0 0) 0x7f66ac090350 con 0x7f66c4102720 2024-04-22T23:32:04.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.361+0000 7f66c37fe700 1 --2- 172.21.15.67:0/3752549757 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f66b006ecc0 0x7f66b0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:04.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.361+0000 7f66c37fe700 1 --2- 172.21.15.67:0/3752549757 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f66b006ecc0 0x7f66b0071180 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f66b400d340 tx=0x7f66b4009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:32:04.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.365+0000 7f66c0ff9700 1 -- 172.21.15.67:0/3752549757 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f66ac05cbf0 con 0x7f66c4102720 2024-04-22T23:32:04.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.584+0000 7f66c9f32700 1 -- 172.21.15.67:0/3752549757 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f66c4199e70 con 0x7f66b006ecc0 2024-04-22T23:32:04.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.595+0000 7f66c0ff9700 1 -- 172.21.15.67:0/3752549757 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f66c4199e70 con 0x7f66b006ecc0 2024-04-22T23:32:04.593 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:32:04.593 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (5m) 85s ago 16m 26.1M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:32:04.593 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (17m) 85s ago 17m 8728k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:32:04.593 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (16m) 18s ago 16m 8590k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:32:04.593 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (3m) 85s ago 17m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:32:04.593 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (3m) 18s ago 16m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:32:04.593 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (5m) 85s ago 16m 80.1M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:32:04.593 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (11m) 85s ago 11m 62.7M - 18.2.1 5be31c24972a e24a471f0a8c 2024-04-22T23:32:04.593 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (11m) 85s ago 11m 130M - 18.2.1 5be31c24972a 0730f9fc8e61 2024-04-22T23:32:04.593 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (11m) 18s ago 11m 123M - 18.2.1 5be31c24972a b6a169586433 2024-04-22T23:32:04.593 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (11m) 18s ago 11m 67.6M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:32:04.593 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (7m) 85s ago 18m 506M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:32:04.593 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (6m) 18s ago 16m 452M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:32:04.593 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (4m) 85s ago 18m 49.7M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:32:04.594 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (3m) 18s ago 16m 40.3M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:32:04.594 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (6m) 85s ago 17m 15.4M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:32:04.594 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (6m) 18s ago 16m 15.8M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:32:04.594 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (2m) 85s ago 15m 95.4M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:32:04.594 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (2m) 85s ago 14m 131M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:32:04.594 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (91s) 85s ago 14m 15.1M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:32:04.594 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (55s) 18s ago 14m 95.7M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:32:04.594 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (21s) 18s ago 13m 12.6M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:32:04.594 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (13m) 18s ago 13m 208M 2240M 18.2.1 5be31c24972a 9fd97f1c65cb 2024-04-22T23:32:04.594 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (6m) 85s ago 16m 71.3M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:32:04.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.597+0000 7f66c9f32700 1 -- 172.21.15.67:0/3752549757 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f66b006ecc0 msgr2=0x7f66b0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:04.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.597+0000 7f66c9f32700 1 --2- 172.21.15.67:0/3752549757 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f66b006ecc0 0x7f66b0071180 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f66b400d340 tx=0x7f66b4009040 comp rx=0 tx=0).stop 2024-04-22T23:32:04.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.598+0000 7f66c9f32700 1 -- 172.21.15.67:0/3752549757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66c4102720 msgr2=0x7f66c419edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:04.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.598+0000 7f66c9f32700 1 --2- 172.21.15.67:0/3752549757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66c4102720 0x7f66c419edb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f66ac00bd70 tx=0x7f66ac00be50 comp rx=0 tx=0).stop 2024-04-22T23:32:04.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.598+0000 7f66c9f32700 1 -- 172.21.15.67:0/3752549757 shutdown_connections 2024-04-22T23:32:04.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.598+0000 7f66c9f32700 1 --2- 172.21.15.67:0/3752549757 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f66b006ecc0 0x7f66b0071180 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:04.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.598+0000 7f66c9f32700 1 --2- 172.21.15.67:0/3752549757 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66c4101e00 0x7f66c419e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:04.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.598+0000 7f66c9f32700 1 --2- 172.21.15.67:0/3752549757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66c4102720 0x7f66c419edb0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:04.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.598+0000 7f66c9f32700 1 -- 172.21.15.67:0/3752549757 >> 172.21.15.67:0/3752549757 conn(0x7f66c40fb650 msgr2=0x7f66c4100220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:32:04.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.598+0000 7f66c9f32700 1 -- 172.21.15.67:0/3752549757 shutdown_connections 2024-04-22T23:32:04.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.598+0000 7f66c9f32700 1 -- 172.21.15.67:0/3752549757 wait complete. 2024-04-22T23:32:04.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.704+0000 7fe4993e6700 1 -- 172.21.15.67:0/4054887712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4940739d0 msgr2=0x7fe49410cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:04.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.704+0000 7fe4993e6700 1 --2- 172.21.15.67:0/4054887712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4940739d0 0x7fe49410cfc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe47c009910 tx=0x7fe47c009c20 comp rx=0 tx=0).stop 2024-04-22T23:32:04.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.704+0000 7fe4993e6700 1 -- 172.21.15.67:0/4054887712 shutdown_connections 2024-04-22T23:32:04.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.704+0000 7fe4993e6700 1 --2- 172.21.15.67:0/4054887712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4940739d0 0x7fe49410cfc0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:04.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.704+0000 7fe4993e6700 1 --2- 172.21.15.67:0/4054887712 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe4940730b0 0x7fe494073490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:04.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.704+0000 7fe4993e6700 1 -- 172.21.15.67:0/4054887712 >> 172.21.15.67:0/4054887712 conn(0x7fe4940fc750 msgr2=0x7fe4940feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:32:04.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.704+0000 7fe4993e6700 1 -- 172.21.15.67:0/4054887712 shutdown_connections 2024-04-22T23:32:04.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.704+0000 7fe4993e6700 1 -- 172.21.15.67:0/4054887712 wait complete. 2024-04-22T23:32:04.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.705+0000 7fe4993e6700 1 Processor -- start 2024-04-22T23:32:04.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.705+0000 7fe4993e6700 1 -- start start 2024-04-22T23:32:04.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.705+0000 7fe4993e6700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe4940730b0 0x7fe49419e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:04.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.705+0000 7fe4993e6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4940739d0 0x7fe49419ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:04.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.705+0000 7fe4993e6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49419f3f0 con 0x7fe4940739d0 2024-04-22T23:32:04.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.705+0000 7fe4993e6700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4941988a0 con 0x7fe4940730b0 2024-04-22T23:32:04.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.705+0000 7fe4927fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4940739d0 0x7fe49419ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:04.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.705+0000 7fe4927fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4940739d0 0x7fe49419ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38612/0 (socket says 172.21.15.67:38612) 2024-04-22T23:32:04.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.705+0000 7fe4927fc700 1 -- 172.21.15.67:0/2171362124 learned_addr learned my addr 172.21.15.67:0/2171362124 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:32:04.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.705+0000 7fe492ffd700 1 --2- 172.21.15.67:0/2171362124 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe4940730b0 0x7fe49419e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:04.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.705+0000 7fe4927fc700 1 -- 172.21.15.67:0/2171362124 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe4940730b0 msgr2=0x7fe49419e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:04.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.705+0000 7fe4927fc700 1 --2- 172.21.15.67:0/2171362124 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe4940730b0 0x7fe49419e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:04.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.706+0000 7fe4927fc700 1 -- 172.21.15.67:0/2171362124 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe47c0095f0 con 0x7fe4940739d0 2024-04-22T23:32:04.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.706+0000 7fe4927fc700 1 --2- 172.21.15.67:0/2171362124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4940739d0 0x7fe49419ed60 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe47c006010 tx=0x7fe47c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:32:04.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.706+0000 7fe48bfff700 1 -- 172.21.15.67:0/2171362124 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe47c01d070 con 0x7fe4940739d0 2024-04-22T23:32:04.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.706+0000 7fe4993e6700 1 -- 172.21.15.67:0/2171362124 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe494198b20 con 0x7fe4940739d0 2024-04-22T23:32:04.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.706+0000 7fe48bfff700 1 -- 172.21.15.67:0/2171362124 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe47c00f460 con 0x7fe4940739d0 2024-04-22T23:32:04.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.706+0000 7fe48bfff700 1 -- 172.21.15.67:0/2171362124 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe47c017610 con 0x7fe4940739d0 2024-04-22T23:32:04.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.706+0000 7fe4993e6700 1 -- 172.21.15.67:0/2171362124 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe494199010 con 0x7fe4940739d0 2024-04-22T23:32:04.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.707+0000 7fe4993e6700 1 -- 172.21.15.67:0/2171362124 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe49410a6c0 con 0x7fe4940739d0 2024-04-22T23:32:04.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.707+0000 7fe48bfff700 1 -- 172.21.15.67:0/2171362124 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe47c00f5d0 con 0x7fe4940739d0 2024-04-22T23:32:04.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.708+0000 7fe48bfff700 1 --2- 172.21.15.67:0/2171362124 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe48006ecc0 0x7fe480071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:04.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.708+0000 7fe48bfff700 1 -- 172.21.15.67:0/2171362124 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6704+0+0 (secure 0 0 0) 0x7fe47c0904a0 con 0x7fe4940739d0 2024-04-22T23:32:04.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.708+0000 7fe492ffd700 1 --2- 172.21.15.67:0/2171362124 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe48006ecc0 0x7fe480071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:04.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.708+0000 7fe492ffd700 1 --2- 172.21.15.67:0/2171362124 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe48006ecc0 0x7fe480071180 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe48c0060b0 tx=0x7fe48c00c040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:32:04.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:04.713+0000 7fe48bfff700 1 -- 172.21.15.67:0/2171362124 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe47c05ccc0 con 0x7fe4940739d0 2024-04-22T23:32:05.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.014+0000 7fe4993e6700 1 -- 172.21.15.67:0/2171362124 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe49410a0a0 con 0x7fe4940739d0 2024-04-22T23:32:05.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.014+0000 7fe48bfff700 1 -- 172.21.15.67:0/2171362124 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+791 (secure 0 0 0) 0x7fe47c05c850 con 0x7fe4940739d0 2024-04-22T23:32:05.012 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:32:05.012 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:32:05.012 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:32:05.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:32:05.012 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:32:05.012 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:32:05.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:32:05.012 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:32:05.012 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:32:05.012 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 5 2024-04-22T23:32:05.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:32:05.013 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:32:05.013 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-22T23:32:05.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:32:05.013 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:32:05.013 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 5, 2024-04-22T23:32:05.013 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 9 2024-04-22T23:32:05.013 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:32:05.013 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:32:05.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.016+0000 7fe4993e6700 1 -- 172.21.15.67:0/2171362124 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe48006ecc0 msgr2=0x7fe480071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:05.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.016+0000 7fe4993e6700 1 --2- 172.21.15.67:0/2171362124 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe48006ecc0 0x7fe480071180 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe48c0060b0 tx=0x7fe48c00c040 comp rx=0 tx=0).stop 2024-04-22T23:32:05.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.016+0000 7fe4993e6700 1 -- 172.21.15.67:0/2171362124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4940739d0 msgr2=0x7fe49419ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:05.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.016+0000 7fe4993e6700 1 --2- 172.21.15.67:0/2171362124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4940739d0 0x7fe49419ed60 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe47c006010 tx=0x7fe47c00bd70 comp rx=0 tx=0).stop 2024-04-22T23:32:05.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.017+0000 7fe4993e6700 1 -- 172.21.15.67:0/2171362124 shutdown_connections 2024-04-22T23:32:05.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.017+0000 7fe4993e6700 1 --2- 172.21.15.67:0/2171362124 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe48006ecc0 0x7fe480071180 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:05.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.017+0000 7fe4993e6700 1 --2- 172.21.15.67:0/2171362124 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe4940730b0 0x7fe49419e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:05.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.017+0000 7fe4993e6700 1 --2- 172.21.15.67:0/2171362124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4940739d0 0x7fe49419ed60 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:05.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.017+0000 7fe4993e6700 1 -- 172.21.15.67:0/2171362124 >> 172.21.15.67:0/2171362124 conn(0x7fe4940fc750 msgr2=0x7fe494107800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:32:05.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.017+0000 7fe4993e6700 1 -- 172.21.15.67:0/2171362124 shutdown_connections 2024-04-22T23:32:05.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.017+0000 7fe4993e6700 1 -- 172.21.15.67:0/2171362124 wait complete. 2024-04-22T23:32:05.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.124+0000 7fd737761700 1 -- 172.21.15.67:0/2842102415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd730101850 msgr2=0x7fd7301058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:05.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.124+0000 7fd737761700 1 --2- 172.21.15.67:0/2842102415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd730101850 0x7fd7301058a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd720009910 tx=0x7fd720009c20 comp rx=0 tx=0).stop 2024-04-22T23:32:05.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.125+0000 7fd737761700 1 -- 172.21.15.67:0/2842102415 shutdown_connections 2024-04-22T23:32:05.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.125+0000 7fd737761700 1 --2- 172.21.15.67:0/2842102415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd730101850 0x7fd7301058a0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:05.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.125+0000 7fd737761700 1 --2- 172.21.15.67:0/2842102415 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd730100ea0 0x7fd730101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:05.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.125+0000 7fd737761700 1 -- 172.21.15.67:0/2842102415 >> 172.21.15.67:0/2842102415 conn(0x7fd7300fc750 msgr2=0x7fd7300feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:32:05.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.125+0000 7fd737761700 1 -- 172.21.15.67:0/2842102415 shutdown_connections 2024-04-22T23:32:05.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.125+0000 7fd737761700 1 -- 172.21.15.67:0/2842102415 wait complete. 2024-04-22T23:32:05.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.125+0000 7fd737761700 1 Processor -- start 2024-04-22T23:32:05.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.125+0000 7fd737761700 1 -- start start 2024-04-22T23:32:05.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.125+0000 7fd737761700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd730100ea0 0x7fd730073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:05.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.125+0000 7fd737761700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd730101850 0x7fd7300735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:05.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.125+0000 7fd737761700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd730073b10 con 0x7fd730101850 2024-04-22T23:32:05.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.125+0000 7fd737761700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd730073c50 con 0x7fd730100ea0 2024-04-22T23:32:05.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.126+0000 7fd734cfc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd730101850 0x7fd7300735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:05.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.126+0000 7fd734cfc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd730101850 0x7fd7300735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38630/0 (socket says 172.21.15.67:38630) 2024-04-22T23:32:05.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.126+0000 7fd734cfc700 1 -- 172.21.15.67:0/3095176381 learned_addr learned my addr 172.21.15.67:0/3095176381 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:32:05.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.126+0000 7fd7354fd700 1 --2- 172.21.15.67:0/3095176381 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd730100ea0 0x7fd730073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:05.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.126+0000 7fd734cfc700 1 -- 172.21.15.67:0/3095176381 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd730100ea0 msgr2=0x7fd730073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:05.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.126+0000 7fd734cfc700 1 --2- 172.21.15.67:0/3095176381 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd730100ea0 0x7fd730073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:05.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.126+0000 7fd734cfc700 1 -- 172.21.15.67:0/3095176381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7200095f0 con 0x7fd730101850 2024-04-22T23:32:05.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.126+0000 7fd7354fd700 1 --2- 172.21.15.67:0/3095176381 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd730100ea0 0x7fd730073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:32:05.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.126+0000 7fd734cfc700 1 --2- 172.21.15.67:0/3095176381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd730101850 0x7fd7300735d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd7200092e0 tx=0x7fd72000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:32:05.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.126+0000 7fd7267fc700 1 -- 172.21.15.67:0/3095176381 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd72001d070 con 0x7fd730101850 2024-04-22T23:32:05.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.126+0000 7fd7267fc700 1 -- 172.21.15.67:0/3095176381 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd72000f460 con 0x7fd730101850 2024-04-22T23:32:05.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.126+0000 7fd737761700 1 -- 172.21.15.67:0/3095176381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd730073ed0 con 0x7fd730101850 2024-04-22T23:32:05.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.126+0000 7fd7267fc700 1 -- 172.21.15.67:0/3095176381 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7200175f0 con 0x7fd730101850 2024-04-22T23:32:05.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.126+0000 7fd737761700 1 -- 172.21.15.67:0/3095176381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7301a2e30 con 0x7fd730101850 2024-04-22T23:32:05.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.127+0000 7fd737761700 1 -- 172.21.15.67:0/3095176381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7301092a0 con 0x7fd730101850 2024-04-22T23:32:05.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.128+0000 7fd7267fc700 1 -- 172.21.15.67:0/3095176381 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd72000f5d0 con 0x7fd730101850 2024-04-22T23:32:05.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.128+0000 7fd7267fc700 1 --2- 172.21.15.67:0/3095176381 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd71c06ecc0 0x7fd71c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:05.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.128+0000 7fd7267fc700 1 -- 172.21.15.67:0/3095176381 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6704+0+0 (secure 0 0 0) 0x7fd720090470 con 0x7fd730101850 2024-04-22T23:32:05.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.128+0000 7fd7354fd700 1 --2- 172.21.15.67:0/3095176381 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd71c06ecc0 0x7fd71c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:05.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.129+0000 7fd7354fd700 1 --2- 172.21.15.67:0/3095176381 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd71c06ecc0 0x7fd71c071180 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd72c005950 tx=0x7fd72c005f50 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:32:05.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.133+0000 7fd7267fc700 1 -- 172.21.15.67:0/3095176381 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fd720059160 con 0x7fd730101850 2024-04-22T23:32:05.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:05 smithi067 ceph-mon[112438]: from='client.34346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:32:05.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:05 smithi067 ceph-mon[112438]: pgmap v160: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:32:05.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:05 smithi067 ceph-mon[112438]: from='client.34350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:32:05.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:05 smithi067 ceph-mon[112438]: from='client.34354 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:32:05.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:05 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2171362124' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:32:05.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.398+0000 7fd737761700 1 -- 172.21.15.67:0/3095176381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd73004f2e0 con 0x7fd730101850 2024-04-22T23:32:05.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.399+0000 7fd7267fc700 1 -- 172.21.15.67:0/3095176381 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1901 (secure 0 0 0) 0x7fd72005c820 con 0x7fd730101850 2024-04-22T23:32:05.396 INFO:teuthology.orchestra.run.smithi067.stdout:e22 2024-04-22T23:32:05.396 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:32:05.397 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:32:05.397 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:32:05.397 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:32:05.397 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:32:05.397 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:32:05.397 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 22 2024-04-22T23:32:05.397 INFO:teuthology.orchestra.run.smithi067.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T23:32:05.397 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:32:05.397 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:27:20.577198+0000 2024-04-22T23:32:05.397 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:32:05.397 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:32:05.397 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:32:05.397 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:32:05.397 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:32:05.397 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:32:05.397 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:32:05.398 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:32:05.398 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 46 2024-04-22T23:32:05.398 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:32:05.398 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:32:05.398 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:32:05.398 INFO:teuthology.orchestra.run.smithi067.stdout:up {0=14484,1=24273} 2024-04-22T23:32:05.398 INFO:teuthology.orchestra.run.smithi067.stdout:failed 2024-04-22T23:32:05.398 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:32:05.398 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:32:05.398 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:32:05.398 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:32:05.398 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:32:05.398 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:32:05.398 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:32:05.399 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:32:05.399 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:32:05.399 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{0:14484} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.67:6828/3606151860,v1:172.21.15.67:6829/3606151860] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:32:05.399 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{0:14496} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/3305304988,v1:172.21.15.67:6827/3305304988] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:32:05.399 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{1:24273} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.134:6824/535957142,v1:172.21.15.134:6825/535957142] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:32:05.399 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{1:24289} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.134:6826/1954821676,v1:172.21.15.134:6827/1954821676] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:32:05.399 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:32:05.399 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:32:05.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.401+0000 7fd737761700 1 -- 172.21.15.67:0/3095176381 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd71c06ecc0 msgr2=0x7fd71c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:05.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.401+0000 7fd737761700 1 --2- 172.21.15.67:0/3095176381 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd71c06ecc0 0x7fd71c071180 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd72c005950 tx=0x7fd72c005f50 comp rx=0 tx=0).stop 2024-04-22T23:32:05.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.401+0000 7fd737761700 1 -- 172.21.15.67:0/3095176381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd730101850 msgr2=0x7fd7300735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:05.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.401+0000 7fd737761700 1 --2- 172.21.15.67:0/3095176381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd730101850 0x7fd7300735d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd7200092e0 tx=0x7fd72000be20 comp rx=0 tx=0).stop 2024-04-22T23:32:05.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.401+0000 7fd737761700 1 -- 172.21.15.67:0/3095176381 shutdown_connections 2024-04-22T23:32:05.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.401+0000 7fd737761700 1 --2- 172.21.15.67:0/3095176381 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd71c06ecc0 0x7fd71c071180 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:05.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.401+0000 7fd737761700 1 --2- 172.21.15.67:0/3095176381 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd730100ea0 0x7fd730073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:05.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.401+0000 7fd737761700 1 --2- 172.21.15.67:0/3095176381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd730101850 0x7fd7300735d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:05.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.401+0000 7fd737761700 1 -- 172.21.15.67:0/3095176381 >> 172.21.15.67:0/3095176381 conn(0x7fd7300fc750 msgr2=0x7fd7300fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:32:05.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.401+0000 7fd737761700 1 -- 172.21.15.67:0/3095176381 shutdown_connections 2024-04-22T23:32:05.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.401+0000 7fd737761700 1 -- 172.21.15.67:0/3095176381 wait complete. 2024-04-22T23:32:05.401 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 22 2024-04-22T23:32:05.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:05 smithi134 ceph-mon[85630]: from='client.34346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:32:05.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:05 smithi134 ceph-mon[85630]: pgmap v160: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:32:05.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:05 smithi134 ceph-mon[85630]: from='client.34350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:32:05.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:05 smithi134 ceph-mon[85630]: from='client.34354 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:32:05.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:05 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2171362124' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:32:05.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.509+0000 7f735dc94700 1 -- 172.21.15.67:0/3947437383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7358068580 msgr2=0x7f7358068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:05.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.509+0000 7f735dc94700 1 --2- 172.21.15.67:0/3947437383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7358068580 0x7f7358068960 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f734800b0a0 tx=0x7f734800b3b0 comp rx=0 tx=0).stop 2024-04-22T23:32:05.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.509+0000 7f735dc94700 1 -- 172.21.15.67:0/3947437383 shutdown_connections 2024-04-22T23:32:05.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.509+0000 7f735dc94700 1 --2- 172.21.15.67:0/3947437383 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7358068ea0 0x7f735810d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:05.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.509+0000 7f735dc94700 1 --2- 172.21.15.67:0/3947437383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7358068580 0x7f7358068960 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:05.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.509+0000 7f735dc94700 1 -- 172.21.15.67:0/3947437383 >> 172.21.15.67:0/3947437383 conn(0x7f7358075670 msgr2=0x7f7358075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:32:05.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.509+0000 7f735dc94700 1 -- 172.21.15.67:0/3947437383 shutdown_connections 2024-04-22T23:32:05.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.509+0000 7f735dc94700 1 -- 172.21.15.67:0/3947437383 wait complete. 2024-04-22T23:32:05.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.510+0000 7f735dc94700 1 Processor -- start 2024-04-22T23:32:05.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.510+0000 7f735dc94700 1 -- start start 2024-04-22T23:32:05.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.510+0000 7f735dc94700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7358068580 0x7f735819e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:05.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.510+0000 7f735dc94700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7358068ea0 0x7f735819eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:05.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.510+0000 7f735dc94700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735819f3a0 con 0x7f7358068580 2024-04-22T23:32:05.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.510+0000 7f735dc94700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73581988e0 con 0x7f7358068ea0 2024-04-22T23:32:05.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.510+0000 7f73577fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7358068580 0x7f735819e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:05.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.510+0000 7f73577fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7358068580 0x7f735819e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38652/0 (socket says 172.21.15.67:38652) 2024-04-22T23:32:05.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.510+0000 7f73577fe700 1 -- 172.21.15.67:0/2823458328 learned_addr learned my addr 172.21.15.67:0/2823458328 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:32:05.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.510+0000 7f7356ffd700 1 --2- 172.21.15.67:0/2823458328 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7358068ea0 0x7f735819eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:05.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.510+0000 7f73577fe700 1 -- 172.21.15.67:0/2823458328 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7358068ea0 msgr2=0x7f735819eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:05.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.510+0000 7f73577fe700 1 --2- 172.21.15.67:0/2823458328 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7358068ea0 0x7f735819eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:05.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.510+0000 7f73577fe700 1 -- 172.21.15.67:0/2823458328 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7348009d00 con 0x7f7358068580 2024-04-22T23:32:05.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.511+0000 7f73577fe700 1 --2- 172.21.15.67:0/2823458328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7358068580 0x7f735819e860 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7348000f80 tx=0x7f734800cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:32:05.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.511+0000 7f7354ff9700 1 -- 172.21.15.67:0/2823458328 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f734800e040 con 0x7f7358068580 2024-04-22T23:32:05.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.511+0000 7f735dc94700 1 -- 172.21.15.67:0/2823458328 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7358198b60 con 0x7f7358068580 2024-04-22T23:32:05.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.511+0000 7f7354ff9700 1 -- 172.21.15.67:0/2823458328 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7348003e60 con 0x7f7358068580 2024-04-22T23:32:05.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.511+0000 7f7354ff9700 1 -- 172.21.15.67:0/2823458328 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f734801d910 con 0x7f7358068580 2024-04-22T23:32:05.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.511+0000 7f735dc94700 1 -- 172.21.15.67:0/2823458328 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7358199050 con 0x7f7358068580 2024-04-22T23:32:05.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.512+0000 7f735dc94700 1 -- 172.21.15.67:0/2823458328 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f735810aa00 con 0x7f7358068580 2024-04-22T23:32:05.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.512+0000 7f7354ff9700 1 -- 172.21.15.67:0/2823458328 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7348004820 con 0x7f7358068580 2024-04-22T23:32:05.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.513+0000 7f7354ff9700 1 --2- 172.21.15.67:0/2823458328 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f734406ecc0 0x7f7344071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:05.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.513+0000 7f7354ff9700 1 -- 172.21.15.67:0/2823458328 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6704+0+0 (secure 0 0 0) 0x7f7348090890 con 0x7f7358068580 2024-04-22T23:32:05.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.513+0000 7f7356ffd700 1 --2- 172.21.15.67:0/2823458328 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f734406ecc0 0x7f7344071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:05.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.513+0000 7f7356ffd700 1 --2- 172.21.15.67:0/2823458328 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f734406ecc0 0x7f7344071180 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7340005ea0 tx=0x7f7340005e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:32:05.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.518+0000 7f7354ff9700 1 -- 172.21.15.67:0/2823458328 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f734805d0b0 con 0x7f7358068580 2024-04-22T23:32:05.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.748+0000 7f735dc94700 1 -- 172.21.15.67:0/2823458328 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7358199e60 con 0x7f734406ecc0 2024-04-22T23:32:05.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.750+0000 7f7354ff9700 1 -- 172.21.15.67:0/2823458328 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f7358199e60 con 0x7f734406ecc0 2024-04-22T23:32:05.747 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:32:05.747 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:32:05.747 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:32:05.747 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:32:05.747 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:32:05.747 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:32:05.747 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:32:05.747 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr" 2024-04-22T23:32:05.747 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:32:05.748 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "11/23 daemons upgraded", 2024-04-22T23:32:05.748 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading osd daemons", 2024-04-22T23:32:05.748 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:32:05.748 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:32:05.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.753+0000 7f735dc94700 1 -- 172.21.15.67:0/2823458328 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f734406ecc0 msgr2=0x7f7344071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:05.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.753+0000 7f735dc94700 1 --2- 172.21.15.67:0/2823458328 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f734406ecc0 0x7f7344071180 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7340005ea0 tx=0x7f7340005e30 comp rx=0 tx=0).stop 2024-04-22T23:32:05.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.753+0000 7f735dc94700 1 -- 172.21.15.67:0/2823458328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7358068580 msgr2=0x7f735819e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:05.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.753+0000 7f735dc94700 1 --2- 172.21.15.67:0/2823458328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7358068580 0x7f735819e860 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7348000f80 tx=0x7f734800cdd0 comp rx=0 tx=0).stop 2024-04-22T23:32:05.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.753+0000 7f735dc94700 1 -- 172.21.15.67:0/2823458328 shutdown_connections 2024-04-22T23:32:05.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.753+0000 7f735dc94700 1 --2- 172.21.15.67:0/2823458328 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f734406ecc0 0x7f7344071180 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:05.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.753+0000 7f735dc94700 1 --2- 172.21.15.67:0/2823458328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7358068580 0x7f735819e860 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:05.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.753+0000 7f735dc94700 1 --2- 172.21.15.67:0/2823458328 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7358068ea0 0x7f735819eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:05.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.753+0000 7f735dc94700 1 -- 172.21.15.67:0/2823458328 >> 172.21.15.67:0/2823458328 conn(0x7f7358075670 msgr2=0x7f73580fe790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:32:05.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.753+0000 7f735dc94700 1 -- 172.21.15.67:0/2823458328 shutdown_connections 2024-04-22T23:32:05.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.753+0000 7f735dc94700 1 -- 172.21.15.67:0/2823458328 wait complete. 2024-04-22T23:32:05.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.862+0000 7f21e5881700 1 -- 172.21.15.67:0/3079706899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21e0105bf0 msgr2=0x7f21e0105fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:05.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.862+0000 7f21e5881700 1 --2- 172.21.15.67:0/3079706899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21e0105bf0 0x7f21e0105fd0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f21d800b0a0 tx=0x7f21d800b3b0 comp rx=0 tx=0).stop 2024-04-22T23:32:05.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.862+0000 7f21e5881700 1 -- 172.21.15.67:0/3079706899 shutdown_connections 2024-04-22T23:32:05.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.862+0000 7f21e5881700 1 --2- 172.21.15.67:0/3079706899 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f21e00ffb40 0x7f21e00fffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:05.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.862+0000 7f21e5881700 1 --2- 172.21.15.67:0/3079706899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21e0105bf0 0x7f21e0105fd0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:05.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.862+0000 7f21e5881700 1 -- 172.21.15.67:0/3079706899 >> 172.21.15.67:0/3079706899 conn(0x7f21e00fb610 msgr2=0x7f21e00fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:32:05.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.862+0000 7f21e5881700 1 -- 172.21.15.67:0/3079706899 shutdown_connections 2024-04-22T23:32:05.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.862+0000 7f21e5881700 1 -- 172.21.15.67:0/3079706899 wait complete. 2024-04-22T23:32:05.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.862+0000 7f21e5881700 1 Processor -- start 2024-04-22T23:32:05.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.863+0000 7f21e5881700 1 -- start start 2024-04-22T23:32:05.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.863+0000 7f21e5881700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21e00ffb40 0x7f21e019c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:05.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.863+0000 7f21e5881700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f21e0105bf0 0x7f21e019cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:05.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.863+0000 7f21e5881700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e019d290 con 0x7f21e00ffb40 2024-04-22T23:32:05.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.863+0000 7f21e5881700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e0196740 con 0x7f21e0105bf0 2024-04-22T23:32:05.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.863+0000 7f21deffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21e00ffb40 0x7f21e019c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:05.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.863+0000 7f21deffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21e00ffb40 0x7f21e019c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38672/0 (socket says 172.21.15.67:38672) 2024-04-22T23:32:05.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.863+0000 7f21deffd700 1 -- 172.21.15.67:0/3570707757 learned_addr learned my addr 172.21.15.67:0/3570707757 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:32:05.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.863+0000 7f21de7fc700 1 --2- 172.21.15.67:0/3570707757 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f21e0105bf0 0x7f21e019cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:05.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.863+0000 7f21deffd700 1 -- 172.21.15.67:0/3570707757 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f21e0105bf0 msgr2=0x7f21e019cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:05.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.863+0000 7f21deffd700 1 --2- 172.21.15.67:0/3570707757 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f21e0105bf0 0x7f21e019cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:05.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.863+0000 7f21deffd700 1 -- 172.21.15.67:0/3570707757 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d8009d00 con 0x7f21e00ffb40 2024-04-22T23:32:05.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.863+0000 7f21deffd700 1 --2- 172.21.15.67:0/3570707757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21e00ffb40 0x7f21e019c6c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f21d8000f80 tx=0x7f21d800cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:32:05.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.864+0000 7f21e487f700 1 -- 172.21.15.67:0/3570707757 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21d800e040 con 0x7f21e00ffb40 2024-04-22T23:32:05.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.864+0000 7f21e5881700 1 -- 172.21.15.67:0/3570707757 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21e01969c0 con 0x7f21e00ffb40 2024-04-22T23:32:05.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.864+0000 7f21e487f700 1 -- 172.21.15.67:0/3570707757 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f21d8003e60 con 0x7f21e00ffb40 2024-04-22T23:32:05.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.864+0000 7f21e487f700 1 -- 172.21.15.67:0/3570707757 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21d801d910 con 0x7f21e00ffb40 2024-04-22T23:32:05.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.864+0000 7f21e5881700 1 -- 172.21.15.67:0/3570707757 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21e0196eb0 con 0x7f21e00ffb40 2024-04-22T23:32:05.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.865+0000 7f21e5881700 1 -- 172.21.15.67:0/3570707757 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21e004f2e0 con 0x7f21e00ffb40 2024-04-22T23:32:05.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.865+0000 7f21e487f700 1 -- 172.21.15.67:0/3570707757 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f21d8004820 con 0x7f21e00ffb40 2024-04-22T23:32:05.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.866+0000 7f21e487f700 1 --2- 172.21.15.67:0/3570707757 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f21cc06ecc0 0x7f21cc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:05.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.866+0000 7f21e487f700 1 -- 172.21.15.67:0/3570707757 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6704+0+0 (secure 0 0 0) 0x7f21d8090890 con 0x7f21e00ffb40 2024-04-22T23:32:05.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.866+0000 7f21de7fc700 1 --2- 172.21.15.67:0/3570707757 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f21cc06ecc0 0x7f21cc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:05.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.866+0000 7f21de7fc700 1 --2- 172.21.15.67:0/3570707757 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f21cc06ecc0 0x7f21cc071180 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f21c8006fd0 tx=0x7f21c8008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:32:05.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:05.870+0000 7f21e487f700 1 -- 172.21.15.67:0/3570707757 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f21d805d0b0 con 0x7f21e00ffb40 2024-04-22T23:32:06.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:06.166+0000 7f21e5881700 1 -- 172.21.15.67:0/3570707757 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f21e0197c50 con 0x7f21e00ffb40 2024-04-22T23:32:06.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:06.167+0000 7f21e487f700 1 -- 172.21.15.67:0/3570707757 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f21d805cc40 con 0x7f21e00ffb40 2024-04-22T23:32:06.165 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-22T23:32:06.165 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:32:06.165 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:32:06.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:06.169+0000 7f21e5881700 1 -- 172.21.15.67:0/3570707757 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f21cc06ecc0 msgr2=0x7f21cc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:06.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:06.169+0000 7f21e5881700 1 --2- 172.21.15.67:0/3570707757 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f21cc06ecc0 0x7f21cc071180 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f21c8006fd0 tx=0x7f21c8008040 comp rx=0 tx=0).stop 2024-04-22T23:32:06.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:06.169+0000 7f21e5881700 1 -- 172.21.15.67:0/3570707757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21e00ffb40 msgr2=0x7f21e019c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:06.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:06.169+0000 7f21e5881700 1 --2- 172.21.15.67:0/3570707757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21e00ffb40 0x7f21e019c6c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f21d8000f80 tx=0x7f21d800cdd0 comp rx=0 tx=0).stop 2024-04-22T23:32:06.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:06.169+0000 7f21e5881700 1 -- 172.21.15.67:0/3570707757 shutdown_connections 2024-04-22T23:32:06.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:06.169+0000 7f21e5881700 1 --2- 172.21.15.67:0/3570707757 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f21cc06ecc0 0x7f21cc071180 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:06.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:06.169+0000 7f21e5881700 1 --2- 172.21.15.67:0/3570707757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21e00ffb40 0x7f21e019c6c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:06.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:06.169+0000 7f21e5881700 1 --2- 172.21.15.67:0/3570707757 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f21e0105bf0 0x7f21e019cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:06.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:06.169+0000 7f21e5881700 1 -- 172.21.15.67:0/3570707757 >> 172.21.15.67:0/3570707757 conn(0x7f21e00fb610 msgr2=0x7f21e0068f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:32:06.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:06.169+0000 7f21e5881700 1 -- 172.21.15.67:0/3570707757 shutdown_connections 2024-04-22T23:32:06.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:06.169+0000 7f21e5881700 1 -- 172.21.15.67:0/3570707757 wait complete. 2024-04-22T23:32:06.489 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:06 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3095176381' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:32:06.489 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:06 smithi134 ceph-mon[85630]: from='client.34366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:32:06.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:06 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3095176381' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:32:06.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:06 smithi067 ceph-mon[112438]: from='client.34366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:32:07.062 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:07 smithi134 ceph-mon[85630]: pgmap v161: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:32:07.063 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:07 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3570707757' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:32:07.063 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:07 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-22T23:32:07.063 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:07 smithi134 ceph-mon[85630]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-22T23:32:07.063 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:07 smithi134 ceph-mon[85630]: Upgrade: osd.5 is safe to restart 2024-04-22T23:32:07.063 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:07 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:07.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:07 smithi067 ceph-mon[112438]: pgmap v161: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:32:07.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:07 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3570707757' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:32:07.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:07 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-22T23:32:07.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:07 smithi067 ceph-mon[112438]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-22T23:32:07.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:07 smithi067 ceph-mon[112438]: Upgrade: osd.5 is safe to restart 2024-04-22T23:32:07.291 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:07 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:08.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:08 smithi134 ceph-mon[85630]: Upgrade: Updating osd.5 2024-04-22T23:32:08.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:08 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:08.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:08 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T23:32:08.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:08 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:32:08.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:08 smithi134 ceph-mon[85630]: Deploying daemon osd.5 on smithi134 2024-04-22T23:32:09.040 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:08 smithi067 ceph-mon[112438]: Upgrade: Updating osd.5 2024-04-22T23:32:09.040 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:08 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:09.041 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:08 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T23:32:09.041 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:08 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:32:09.041 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:08 smithi067 ceph-mon[112438]: Deploying daemon osd.5 on smithi134 2024-04-22T23:32:09.886 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:09 smithi134 ceph-mon[85630]: pgmap v162: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:32:10.040 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:09 smithi067 ceph-mon[112438]: pgmap v162: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:32:10.990 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:10 smithi134 systemd[1]: Stopping Ceph osd.5 for d46c346e-00fd-11ef-bc93-c7b262605968... 2024-04-22T23:32:11.489 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:11 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[54248]: 2024-04-22T23:32:11.107+0000 7f8b02e47700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T23:32:11.489 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:11 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[54248]: 2024-04-22T23:32:11.107+0000 7f8b02e47700 -1 osd.5 82 *** Got signal Terminated *** 2024-04-22T23:32:11.490 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:11 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[54248]: 2024-04-22T23:32:11.107+0000 7f8b02e47700 -1 osd.5 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T23:32:11.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:11 smithi134 ceph-mon[85630]: pgmap v163: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:32:11.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:11.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:32:11.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:11 smithi134 ceph-mon[85630]: osd.5 marked itself down and dead 2024-04-22T23:32:11.999 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:11 smithi067 ceph-mon[112438]: pgmap v163: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-22T23:32:12.000 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:12.000 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:32:12.000 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:11 smithi067 ceph-mon[112438]: osd.5 marked itself down and dead 2024-04-22T23:32:12.752 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:12 smithi134 podman[104849]: 2024-04-22 23:32:12.544958148 +0000 UTC m=+1.533793452 container died 9fd97f1c65cba0b8e314c42d008751da2f749cf16f1441a40cbf64da8b7cb69a (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=HEAD) 2024-04-22T23:32:13.040 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:12 smithi067 ceph-mon[112438]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-22T23:32:13.040 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:12 smithi067 ceph-mon[112438]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-22T23:32:13.040 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:12 smithi067 ceph-mon[112438]: osdmap e83: 6 total, 5 up, 6 in 2024-04-22T23:32:13.198 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:12 smithi134 ceph-mon[85630]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-22T23:32:13.198 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:12 smithi134 ceph-mon[85630]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-22T23:32:13.198 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:12 smithi134 ceph-mon[85630]: osdmap e83: 6 total, 5 up, 6 in 2024-04-22T23:32:13.198 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:12 smithi134 podman[104849]: 2024-04-22 23:32:12.934071707 +0000 UTC m=+1.922907011 container cleanup 9fd97f1c65cba0b8e314c42d008751da2f749cf16f1441a40cbf64da8b7cb69a (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, ceph=True) 2024-04-22T23:32:13.199 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:12 smithi134 bash[104849]: ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5 2024-04-22T23:32:13.489 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:13 smithi134 podman[104878]: 2024-04-22 23:32:13.202251299 +0000 UTC m=+0.654535801 container remove 9fd97f1c65cba0b8e314c42d008751da2f749cf16f1441a40cbf64da8b7cb69a (image=quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, GIT_BRANCH=HEAD) 2024-04-22T23:32:13.834 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:13 smithi134 ceph-mon[85630]: pgmap v164: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:32:13.835 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:13 smithi134 ceph-mon[85630]: osdmap e84: 6 total, 5 up, 6 in 2024-04-22T23:32:13.835 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:13 smithi134 podman[104966]: 2024-04-22 23:32:13.465617538 +0000 UTC m=+0.018000619 image pull a0434699c1d76af4eea1fc95ff0a22ea73d1e04e609464fbf658d6f39bc7f2fc quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c 2024-04-22T23:32:13.835 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:13 smithi134 podman[104966]: 2024-04-22 23:32:13.578112517 +0000 UTC m=+0.130495598 container create 2dedc287b2874564fd92bdbf56055848d798c0c82be1d4af607f069abf12d9aa (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T23:32:13.835 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:13 smithi134 podman[104966]: 2024-04-22 23:32:13.839192967 +0000 UTC m=+0.391576048 container init 2dedc287b2874564fd92bdbf56055848d798c0c82be1d4af607f069abf12d9aa (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240416) 2024-04-22T23:32:14.040 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:13 smithi067 ceph-mon[112438]: pgmap v164: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:32:14.041 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:13 smithi067 ceph-mon[112438]: osdmap e84: 6 total, 5 up, 6 in 2024-04-22T23:32:14.130 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:13 smithi134 podman[104966]: 2024-04-22 23:32:13.842457532 +0000 UTC m=+0.394840622 container start 2dedc287b2874564fd92bdbf56055848d798c0c82be1d4af607f069abf12d9aa (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-22T23:32:14.130 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:13 smithi134 podman[104966]: 2024-04-22 23:32:13.922500059 +0000 UTC m=+0.474883154 container attach 2dedc287b2874564fd92bdbf56055848d798c0c82be1d4af607f069abf12d9aa (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5-deactivate, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-22T23:32:14.421 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:14 smithi134 podman[104966]: 2024-04-22 23:32:14.134930822 +0000 UTC m=+0.687313907 container died 2dedc287b2874564fd92bdbf56055848d798c0c82be1d4af607f069abf12d9aa (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5-deactivate, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-22T23:32:14.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:14 smithi134 ceph-mon[85630]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-04-22T23:32:14.990 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:14 smithi134 podman[104993]: 2024-04-22 23:32:14.692357964 +0000 UTC m=+0.552753735 container remove 2dedc287b2874564fd92bdbf56055848d798c0c82be1d4af607f069abf12d9aa (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416) 2024-04-22T23:32:14.990 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:14 smithi134 systemd[1]: ceph-d46c346e-00fd-11ef-bc93-c7b262605968@osd.5.service: Deactivated successfully. 2024-04-22T23:32:14.990 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:14 smithi134 systemd[1]: Stopped Ceph osd.5 for d46c346e-00fd-11ef-bc93-c7b262605968. 2024-04-22T23:32:14.990 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:14 smithi134 systemd[1]: ceph-d46c346e-00fd-11ef-bc93-c7b262605968@osd.5.service: Consumed 54.881s CPU time. 2024-04-22T23:32:14.990 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:14 smithi134 systemd[1]: Starting Ceph osd.5 for d46c346e-00fd-11ef-bc93-c7b262605968... 2024-04-22T23:32:15.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:14 smithi067 ceph-mon[112438]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-04-22T23:32:15.425 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:15 smithi134 podman[105097]: 2024-04-22 23:32:15.031944216 +0000 UTC m=+0.018444377 image pull a0434699c1d76af4eea1fc95ff0a22ea73d1e04e609464fbf658d6f39bc7f2fc quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c 2024-04-22T23:32:15.425 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:15 smithi134 podman[105097]: 2024-04-22 23:32:15.135194975 +0000 UTC m=+0.121695111 container create 9437953d7db2ae0633dee58b3681f30774b400d0929a7b5ec95770c18e3cea2e (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5-activate, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-22T23:32:15.425 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:15 smithi134 podman[105097]: 2024-04-22 23:32:15.337537582 +0000 UTC m=+0.324037718 container init 9437953d7db2ae0633dee58b3681f30774b400d0929a7b5ec95770c18e3cea2e (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5-activate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux ) 2024-04-22T23:32:15.426 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:15 smithi134 podman[105097]: 2024-04-22 23:32:15.340706705 +0000 UTC m=+0.327206849 container start 9437953d7db2ae0633dee58b3681f30774b400d0929a7b5ec95770c18e3cea2e (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5-activate, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-22T23:32:15.426 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:15 smithi134 podman[105097]: 2024-04-22 23:32:15.429142036 +0000 UTC m=+0.415642185 container attach 9437953d7db2ae0633dee58b3681f30774b400d0929a7b5ec95770c18e3cea2e (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5-activate, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.schema-version=1.0) 2024-04-22T23:32:16.136 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:15 smithi134 ceph-mon[85630]: pgmap v167: 65 pgs: 3 peering, 11 stale+active+clean, 51 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:32:16.136 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:15 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5-activate[105111]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T23:32:16.136 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:15 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5-activate[105111]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T23:32:16.137 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:15 smithi134 bash[105097]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T23:32:16.137 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:15 smithi134 bash[105097]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-22T23:32:16.137 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:15 smithi134 bash[105097]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-22T23:32:16.137 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:15 smithi134 bash[105097]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-22T23:32:16.137 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:15 smithi134 bash[105097]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-22T23:32:16.137 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:15 smithi134 bash[105097]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-22T23:32:16.137 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:15 smithi134 bash[105097]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5/block 2024-04-22T23:32:16.137 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:15 smithi134 bash[105097]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-22T23:32:16.137 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:15 smithi134 bash[105097]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-22T23:32:16.137 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:15 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5-activate[105111]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-22T23:32:16.137 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:15 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5-activate[105111]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-22T23:32:16.137 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:15 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5-activate[105111]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-22T23:32:16.137 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:15 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5-activate[105111]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-22T23:32:16.138 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:15 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5-activate[105111]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5/block 2024-04-22T23:32:16.138 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:15 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5-activate[105111]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-22T23:32:16.138 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:15 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5-activate[105111]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-22T23:32:16.138 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:15 smithi134 podman[105097]: 2024-04-22 23:32:15.845045939 +0000 UTC m=+0.831546110 container died 9437953d7db2ae0633dee58b3681f30774b400d0929a7b5ec95770c18e3cea2e (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5-activate, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-22-1618-43be020) 2024-04-22T23:32:16.191 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:15 smithi067 ceph-mon[112438]: pgmap v167: 65 pgs: 3 peering, 11 stale+active+clean, 51 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-22T23:32:16.739 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:16 smithi134 podman[105237]: 2024-04-22 23:32:16.416010185 +0000 UTC m=+0.566136353 container remove 9437953d7db2ae0633dee58b3681f30774b400d0929a7b5ec95770c18e3cea2e (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5-activate, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T23:32:16.740 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:16 smithi134 podman[105303]: 2024-04-22 23:32:16.531540945 +0000 UTC m=+0.017760662 image pull a0434699c1d76af4eea1fc95ff0a22ea73d1e04e609464fbf658d6f39bc7f2fc quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c 2024-04-22T23:32:16.740 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:16 smithi134 podman[105303]: 2024-04-22 23:32:16.658583884 +0000 UTC m=+0.144803622 container create 93a955888c845793c0aa293386bde864bf7d19270c244d21c2a477733e0ae3d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-22-1618-43be020) 2024-04-22T23:32:17.239 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:16 smithi134 ceph-mon[85630]: Health check failed: Degraded data redundancy: 32/363 objects degraded (8.815%), 10 pgs degraded (PG_DEGRADED) 2024-04-22T23:32:17.240 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:16 smithi134 podman[105303]: 2024-04-22 23:32:16.969735388 +0000 UTC m=+0.455955081 container init 93a955888c845793c0aa293386bde864bf7d19270c244d21c2a477733e0ae3d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416) 2024-04-22T23:32:17.240 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:16 smithi134 podman[105303]: 2024-04-22 23:32:16.972837179 +0000 UTC m=+0.459056880 container start 93a955888c845793c0aa293386bde864bf7d19270c244d21c2a477733e0ae3d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-22T23:32:17.240 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:16 smithi134 ceph-osd[105321]: -- 172.21.15.134:0/1020061424 <== mon.1 v2:172.21.15.134:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c6688c5680 con 0x55c66888ec00 2024-04-22T23:32:17.240 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:17 smithi134 bash[105303]: 93a955888c845793c0aa293386bde864bf7d19270c244d21c2a477733e0ae3d5 2024-04-22T23:32:17.240 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:17 smithi134 systemd[1]: Started Ceph osd.5 for d46c346e-00fd-11ef-bc93-c7b262605968. 2024-04-22T23:32:17.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:16 smithi067 ceph-mon[112438]: Health check failed: Degraded data redundancy: 32/363 objects degraded (8.815%), 10 pgs degraded (PG_DEGRADED) 2024-04-22T23:32:17.861 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:17 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[105317]: 2024-04-22T23:32:17.521+0000 7f0dfdeac7c0 -1 Falling back to public interface 2024-04-22T23:32:18.239 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:17 smithi134 ceph-mon[85630]: pgmap v168: 65 pgs: 10 active+undersized+degraded, 3 peering, 2 stale+active+clean, 7 active+undersized, 43 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 32/363 objects degraded (8.815%) 2024-04-22T23:32:18.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:17 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:18.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:17 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:18.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:17 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:32:18.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:17 smithi067 ceph-mon[112438]: pgmap v168: 65 pgs: 10 active+undersized+degraded, 3 peering, 2 stale+active+clean, 7 active+undersized, 43 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 32/363 objects degraded (8.815%) 2024-04-22T23:32:18.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:17 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:18.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:17 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:18.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:17 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:32:19.239 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:18 smithi134 ceph-mon[85630]: pgmap v169: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 55/363 objects degraded (15.152%) 2024-04-22T23:32:19.239 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:18 smithi134 ceph-mon[85630]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-04-22T23:32:19.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:18 smithi067 ceph-mon[112438]: pgmap v169: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 55/363 objects degraded (15.152%) 2024-04-22T23:32:19.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:18 smithi067 ceph-mon[112438]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-04-22T23:32:21.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:21 smithi134 ceph-mon[85630]: pgmap v170: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 55/363 objects degraded (15.152%) 2024-04-22T23:32:21.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:21 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:21.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:21 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:21.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:21 smithi067 ceph-mon[112438]: pgmap v170: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 55/363 objects degraded (15.152%) 2024-04-22T23:32:21.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:21 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:21.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:21 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:21.961 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:21 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[105317]: 2024-04-22T23:32:21.747+0000 7f0dfdeac7c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-04-22T23:32:22.239 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:22 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:22.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:22 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:22.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:22 smithi134 ceph-mon[85630]: from='osd.5 [v2:172.21.15.134:6816/921342983,v1:172.21.15.134:6817/921342983]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T23:32:22.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:22 smithi134 ceph-mon[85630]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T23:32:22.240 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:22 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[105317]: 2024-04-22T23:32:22.042+0000 7f0dfdeac7c0 -1 osd.5 82 log_to_monitors true 2024-04-22T23:32:22.240 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 22 23:32:22 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[105317]: 2024-04-22T23:32:22.199+0000 7f0def304700 -1 osd.5 82 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T23:32:22.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:22 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:22.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:22 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:22.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:22 smithi067 ceph-mon[112438]: from='osd.5 [v2:172.21.15.134:6816/921342983,v1:172.21.15.134:6817/921342983]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T23:32:22.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:22 smithi067 ceph-mon[112438]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T23:32:23.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:23 smithi134 ceph-mon[85630]: pgmap v171: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 55/363 objects degraded (15.152%) 2024-04-22T23:32:23.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:23 smithi134 ceph-mon[85630]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T23:32:23.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:23 smithi134 ceph-mon[85630]: osdmap e85: 6 total, 5 up, 6 in 2024-04-22T23:32:23.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:23 smithi134 ceph-mon[85630]: from='osd.5 [v2:172.21.15.134:6816/921342983,v1:172.21.15.134:6817/921342983]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-22T23:32:23.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:23 smithi134 ceph-mon[85630]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-22T23:32:23.490 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:23 smithi134 ceph-mon[85630]: Health check update: Degraded data redundancy: 55/363 objects degraded (15.152%), 14 pgs degraded (PG_DEGRADED) 2024-04-22T23:32:23.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:23 smithi067 ceph-mon[112438]: pgmap v171: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 55/363 objects degraded (15.152%) 2024-04-22T23:32:23.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:23 smithi067 ceph-mon[112438]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T23:32:23.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:23 smithi067 ceph-mon[112438]: osdmap e85: 6 total, 5 up, 6 in 2024-04-22T23:32:23.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:23 smithi067 ceph-mon[112438]: from='osd.5 [v2:172.21.15.134:6816/921342983,v1:172.21.15.134:6817/921342983]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-22T23:32:23.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:23 smithi067 ceph-mon[112438]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-04-22T23:32:23.541 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:23 smithi067 ceph-mon[112438]: Health check update: Degraded data redundancy: 55/363 objects degraded (15.152%), 14 pgs degraded (PG_DEGRADED) 2024-04-22T23:32:24.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:24 smithi134 ceph-mon[85630]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-22T23:32:24.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:24 smithi134 ceph-mon[85630]: osd.5 [v2:172.21.15.134:6816/921342983,v1:172.21.15.134:6817/921342983] boot 2024-04-22T23:32:24.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:24 smithi134 ceph-mon[85630]: osdmap e86: 6 total, 6 up, 6 in 2024-04-22T23:32:24.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:24 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T23:32:24.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:24 smithi134 ceph-mon[85630]: osdmap e87: 6 total, 6 up, 6 in 2024-04-22T23:32:24.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:24 smithi067 ceph-mon[112438]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-22T23:32:24.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:24 smithi067 ceph-mon[112438]: osd.5 [v2:172.21.15.134:6816/921342983,v1:172.21.15.134:6817/921342983] boot 2024-04-22T23:32:24.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:24 smithi067 ceph-mon[112438]: osdmap e86: 6 total, 6 up, 6 in 2024-04-22T23:32:24.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:24 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T23:32:24.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:24 smithi067 ceph-mon[112438]: osdmap e87: 6 total, 6 up, 6 in 2024-04-22T23:32:25.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:25 smithi134 ceph-mon[85630]: pgmap v174: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 3 op/s; 55/363 objects degraded (15.152%) 2024-04-22T23:32:25.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:25 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:25.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:25 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:25.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:25 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:32:25.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:25 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:32:25.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:25 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:25.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:25 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:32:25.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:25 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:32:25.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:25 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:32:25.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:25 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:32:25.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:25 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:32:25.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:25 smithi134 ceph-mon[85630]: Upgrade: Setting container_image for all osd 2024-04-22T23:32:25.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:25 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:25.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:25 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-22T23:32:25.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:25 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-22T23:32:25.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:25 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-22T23:32:25.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:25 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-22T23:32:25.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:25 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-22T23:32:25.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:25 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-22T23:32:25.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:25 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-22T23:32:25.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:25 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-22T23:32:25.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:25 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-22T23:32:25.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:25 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-22T23:32:25.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:25 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-22T23:32:25.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:25 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-22T23:32:25.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:25 smithi134 ceph-mon[85630]: Upgrade: Setting require_osd_release to 19 squid 2024-04-22T23:32:25.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:25 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-22T23:32:26.041 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:25 smithi067 ceph-mon[112438]: pgmap v174: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 3 op/s; 55/363 objects degraded (15.152%) 2024-04-22T23:32:26.041 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:25 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:26.041 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:25 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:26.041 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:25 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:32:26.041 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:25 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:32:26.041 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:25 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:26.041 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:25 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:32:26.041 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:25 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:32:26.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:25 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:32:26.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:25 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:32:26.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:25 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:32:26.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:25 smithi067 ceph-mon[112438]: Upgrade: Setting container_image for all osd 2024-04-22T23:32:26.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:25 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:26.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:25 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-22T23:32:26.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:25 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-22T23:32:26.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:25 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-22T23:32:26.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:25 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-22T23:32:26.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:25 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-22T23:32:26.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:25 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-22T23:32:26.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:25 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-22T23:32:26.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:25 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-22T23:32:26.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:25 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-22T23:32:26.042 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:25 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-22T23:32:26.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:25 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-22T23:32:26.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:25 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-22T23:32:26.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:25 smithi067 ceph-mon[112438]: Upgrade: Setting require_osd_release to 19 squid 2024-04-22T23:32:26.043 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:25 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-22T23:32:26.902 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:26 smithi134 ceph-mon[85630]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-22T23:32:26.902 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-22T23:32:26.902 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:26 smithi134 ceph-mon[85630]: osdmap e88: 6 total, 6 up, 6 in 2024-04-22T23:32:26.902 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:26 smithi134 ceph-mon[85630]: Upgrade: Disabling standby-replay for filesystem cephfs 2024-04-22T23:32:26.902 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:26.902 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]: dispatch 2024-04-22T23:32:26.902 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:26.902 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:26 smithi134 ceph-mon[85630]: osdmap e89: 6 total, 6 up, 6 in 2024-04-22T23:32:26.903 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:32:26.909 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:26 smithi067 ceph-mon[112438]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-22T23:32:26.909 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-22T23:32:26.909 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:26 smithi067 ceph-mon[112438]: osdmap e88: 6 total, 6 up, 6 in 2024-04-22T23:32:26.909 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:26 smithi067 ceph-mon[112438]: Upgrade: Disabling standby-replay for filesystem cephfs 2024-04-22T23:32:26.909 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:26.909 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]: dispatch 2024-04-22T23:32:26.909 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:26.909 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:26 smithi067 ceph-mon[112438]: osdmap e89: 6 total, 6 up, 6 in 2024-04-22T23:32:26.909 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:32:27.194 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:27 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[112434]: 2024-04-22T23:32:27.026+0000 7f2013db4700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T23:32:27.885 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:27 smithi067 ceph-mon[112438]: pgmap v177: 65 pgs: 5 peering, 9 active+undersized+degraded, 11 active+undersized, 40 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2 op/s; 40/363 objects degraded (11.019%) 2024-04-22T23:32:27.886 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:27 smithi067 ceph-mon[112438]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-22T23:32:27.886 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:27 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]': finished 2024-04-22T23:32:27.886 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:27 smithi067 ceph-mon[112438]: fsmap cephfs:2 {0=cephfs.smithi067.xyazmh=up:active,1=cephfs.smithi134.owipad=up:active} 2024-04-22T23:32:27.886 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:27 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:32:27.886 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:27 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:32:27.886 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:27 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:32:27.886 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:27 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:27.886 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:27 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:32:27.886 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:27 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:32:27.886 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:27 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:32:27.886 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:27 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:32:27.886 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:27 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:32:27.886 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:27 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-22T23:32:27.887 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:27 smithi067 ceph-mon[112438]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T23:32:27.887 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:27 smithi067 ceph-mon[112438]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T23:32:27.887 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:27 smithi067 ceph-mon[112438]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-22T23:32:27.887 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:27 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-22T23:32:27.887 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:27 smithi067 ceph-mon[112438]: osdmap e90: 6 total, 6 up, 6 in 2024-04-22T23:32:27.887 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:27 smithi067 ceph-mon[112438]: mds.? [v2:172.21.15.134:6826/1827288036,v1:172.21.15.134:6827/1827288036] up:boot 2024-04-22T23:32:27.887 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:27 smithi067 ceph-mon[112438]: mds.? [v2:172.21.15.67:6826/1310921930,v1:172.21.15.67:6827/1310921930] up:boot 2024-04-22T23:32:27.887 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:27 smithi067 ceph-mon[112438]: fsmap cephfs:0/2 2 up:standby, 2 failed 2024-04-22T23:32:27.887 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:27 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.aqlxpv"}]: dispatch 2024-04-22T23:32:27.887 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:27 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.vuxbsj"}]: dispatch 2024-04-22T23:32:27.887 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:27 smithi067 ceph-mon[112438]: Health check update: Degraded data redundancy: 40/363 objects degraded (11.019%), 9 pgs degraded (PG_DEGRADED) 2024-04-22T23:32:27.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:27 smithi134 ceph-mon[85630]: pgmap v177: 65 pgs: 5 peering, 9 active+undersized+degraded, 11 active+undersized, 40 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2 op/s; 40/363 objects degraded (11.019%) 2024-04-22T23:32:27.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:27 smithi134 ceph-mon[85630]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-22T23:32:27.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:27 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]': finished 2024-04-22T23:32:27.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:27 smithi134 ceph-mon[85630]: fsmap cephfs:2 {0=cephfs.smithi067.xyazmh=up:active,1=cephfs.smithi134.owipad=up:active} 2024-04-22T23:32:27.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:27 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:32:27.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:27 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:32:27.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:27 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:32:27.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:27 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:27.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:27 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:32:27.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:27 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:32:27.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:27 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:32:27.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:27 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:32:27.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:27 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:32:27.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:27 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-22T23:32:27.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:27 smithi134 ceph-mon[85630]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T23:32:27.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:27 smithi134 ceph-mon[85630]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T23:32:27.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:27 smithi134 ceph-mon[85630]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-22T23:32:27.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:27 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-22T23:32:27.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:27 smithi134 ceph-mon[85630]: osdmap e90: 6 total, 6 up, 6 in 2024-04-22T23:32:27.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:27 smithi134 ceph-mon[85630]: mds.? [v2:172.21.15.134:6826/1827288036,v1:172.21.15.134:6827/1827288036] up:boot 2024-04-22T23:32:27.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:27 smithi134 ceph-mon[85630]: mds.? [v2:172.21.15.67:6826/1310921930,v1:172.21.15.67:6827/1310921930] up:boot 2024-04-22T23:32:27.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:27 smithi134 ceph-mon[85630]: fsmap cephfs:0/2 2 up:standby, 2 failed 2024-04-22T23:32:27.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:27 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.aqlxpv"}]: dispatch 2024-04-22T23:32:27.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:27 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.vuxbsj"}]: dispatch 2024-04-22T23:32:27.992 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:27 smithi134 ceph-mon[85630]: Health check update: Degraded data redundancy: 40/363 objects degraded (11.019%), 9 pgs degraded (PG_DEGRADED) 2024-04-22T23:32:28.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:28 smithi134 ceph-mon[85630]: Upgrade: failing fs cephfs for rapid multi-rank mds upgrade 2024-04-22T23:32:28.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:28 smithi134 ceph-mon[85630]: mds.? [v2:172.21.15.134:6824/3366499307,v1:172.21.15.134:6825/3366499307] up:boot 2024-04-22T23:32:28.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:28 smithi134 ceph-mon[85630]: mds.? [v2:172.21.15.67:6828/2016103304,v1:172.21.15.67:6829/2016103304] up:boot 2024-04-22T23:32:28.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:28 smithi134 ceph-mon[85630]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-22T23:32:28.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:28 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.xyazmh"}]: dispatch 2024-04-22T23:32:28.991 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:28 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.owipad"}]: dispatch 2024-04-22T23:32:29.040 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:28 smithi067 ceph-mon[112438]: Upgrade: failing fs cephfs for rapid multi-rank mds upgrade 2024-04-22T23:32:29.040 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:28 smithi067 ceph-mon[112438]: mds.? [v2:172.21.15.134:6824/3366499307,v1:172.21.15.134:6825/3366499307] up:boot 2024-04-22T23:32:29.040 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:28 smithi067 ceph-mon[112438]: mds.? [v2:172.21.15.67:6828/2016103304,v1:172.21.15.67:6829/2016103304] up:boot 2024-04-22T23:32:29.040 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:28 smithi067 ceph-mon[112438]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-22T23:32:29.040 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:28 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.xyazmh"}]: dispatch 2024-04-22T23:32:29.040 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:28 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.owipad"}]: dispatch 2024-04-22T23:32:29.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:29 smithi134 ceph-mon[85630]: pgmap v180: 65 pgs: 5 peering, 60 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s 2024-04-22T23:32:29.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:29 smithi134 ceph-mon[85630]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 40/363 objects degraded (11.019%), 9 pgs degraded) 2024-04-22T23:32:29.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:29 smithi134 ceph-mon[85630]: Upgrade: Updating mds.cephfs.smithi067.aqlxpv 2024-04-22T23:32:29.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:29 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:29.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:29 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi067.aqlxpv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T23:32:29.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:29 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:32:29.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:29 smithi134 ceph-mon[85630]: Deploying daemon mds.cephfs.smithi067.aqlxpv on smithi067 2024-04-22T23:32:30.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:29 smithi067 ceph-mon[112438]: pgmap v180: 65 pgs: 5 peering, 60 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s 2024-04-22T23:32:30.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:29 smithi067 ceph-mon[112438]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 40/363 objects degraded (11.019%), 9 pgs degraded) 2024-04-22T23:32:30.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:29 smithi067 ceph-mon[112438]: Upgrade: Updating mds.cephfs.smithi067.aqlxpv 2024-04-22T23:32:30.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:29 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:30.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:29 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi067.aqlxpv", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T23:32:30.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:29 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:32:30.018 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:29 smithi067 ceph-mon[112438]: Deploying daemon mds.cephfs.smithi067.aqlxpv on smithi067 2024-04-22T23:32:31.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:31 smithi134 ceph-mon[85630]: pgmap v181: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s 2024-04-22T23:32:32.040 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:31 smithi067 ceph-mon[112438]: pgmap v181: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s 2024-04-22T23:32:32.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:32 smithi134 ceph-mon[85630]: osdmap e91: 6 total, 6 up, 6 in 2024-04-22T23:32:32.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:32 smithi134 ceph-mon[85630]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-22T23:32:32.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:32 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.aqlxpv"}]: dispatch 2024-04-22T23:32:33.040 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:32 smithi067 ceph-mon[112438]: osdmap e91: 6 total, 6 up, 6 in 2024-04-22T23:32:33.040 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:32 smithi067 ceph-mon[112438]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-22T23:32:33.040 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:32 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.aqlxpv"}]: dispatch 2024-04-22T23:32:33.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:33 smithi134 ceph-mon[85630]: pgmap v183: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s 2024-04-22T23:32:34.040 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:33 smithi067 ceph-mon[112438]: pgmap v183: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s 2024-04-22T23:32:35.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:35 smithi134 ceph-mon[85630]: pgmap v184: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s 2024-04-22T23:32:36.019 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:35 smithi067 ceph-mon[112438]: pgmap v184: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s 2024-04-22T23:32:36.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.282+0000 7f4a663c8700 1 -- 172.21.15.67:0/1604433678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a60106cd0 msgr2=0x7f4a601070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:36.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.282+0000 7f4a663c8700 1 --2- 172.21.15.67:0/1604433678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a60106cd0 0x7f4a601070b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4a50009910 tx=0x7f4a50009c20 comp rx=0 tx=0).stop 2024-04-22T23:32:36.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.282+0000 7f4a663c8700 1 -- 172.21.15.67:0/1604433678 shutdown_connections 2024-04-22T23:32:36.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.282+0000 7f4a663c8700 1 --2- 172.21.15.67:0/1604433678 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a60068580 0x7f4a600689e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:36.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.282+0000 7f4a663c8700 1 --2- 172.21.15.67:0/1604433678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a60106cd0 0x7f4a601070b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:36.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.282+0000 7f4a663c8700 1 -- 172.21.15.67:0/1604433678 >> 172.21.15.67:0/1604433678 conn(0x7f4a60075670 msgr2=0x7f4a60075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:32:36.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.282+0000 7f4a663c8700 1 -- 172.21.15.67:0/1604433678 shutdown_connections 2024-04-22T23:32:36.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.282+0000 7f4a663c8700 1 -- 172.21.15.67:0/1604433678 wait complete. 2024-04-22T23:32:36.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.283+0000 7f4a663c8700 1 Processor -- start 2024-04-22T23:32:36.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.283+0000 7f4a663c8700 1 -- start start 2024-04-22T23:32:36.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.283+0000 7f4a663c8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a60068580 0x7f4a60196980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:36.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.283+0000 7f4a663c8700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a60106cd0 0x7f4a60196ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:36.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.283+0000 7f4a663c8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a60197550 con 0x7f4a60068580 2024-04-22T23:32:36.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.283+0000 7f4a663c8700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6019b360 con 0x7f4a60106cd0 2024-04-22T23:32:36.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.283+0000 7f4a5ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a60068580 0x7f4a60196980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:36.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.283+0000 7f4a5ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a60068580 0x7f4a60196980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40828/0 (socket says 172.21.15.67:40828) 2024-04-22T23:32:36.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.283+0000 7f4a5ffff700 1 -- 172.21.15.67:0/1395217924 learned_addr learned my addr 172.21.15.67:0/1395217924 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:32:36.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.284+0000 7f4a5f7fe700 1 --2- 172.21.15.67:0/1395217924 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a60106cd0 0x7f4a60196ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:36.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.284+0000 7f4a5ffff700 1 -- 172.21.15.67:0/1395217924 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a60106cd0 msgr2=0x7f4a60196ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:36.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.284+0000 7f4a5ffff700 1 --2- 172.21.15.67:0/1395217924 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a60106cd0 0x7f4a60196ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:36.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.284+0000 7f4a5ffff700 1 -- 172.21.15.67:0/1395217924 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a500095f0 con 0x7f4a60068580 2024-04-22T23:32:36.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.284+0000 7f4a5ffff700 1 --2- 172.21.15.67:0/1395217924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a60068580 0x7f4a60196980 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4a50006010 tx=0x7f4a5000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:32:36.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.284+0000 7f4a5d7fa700 1 -- 172.21.15.67:0/1395217924 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a5001d070 con 0x7f4a60068580 2024-04-22T23:32:36.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.284+0000 7f4a663c8700 1 -- 172.21.15.67:0/1395217924 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a6019b5e0 con 0x7f4a60068580 2024-04-22T23:32:36.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.284+0000 7f4a5d7fa700 1 -- 172.21.15.67:0/1395217924 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4a50004fb0 con 0x7f4a60068580 2024-04-22T23:32:36.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.284+0000 7f4a5d7fa700 1 -- 172.21.15.67:0/1395217924 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a50022ea0 con 0x7f4a60068580 2024-04-22T23:32:36.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.284+0000 7f4a663c8700 1 -- 172.21.15.67:0/1395217924 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a6019bad0 con 0x7f4a60068580 2024-04-22T23:32:36.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.285+0000 7f4a663c8700 1 -- 172.21.15.67:0/1395217924 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a601091f0 con 0x7f4a60068580 2024-04-22T23:32:36.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.286+0000 7f4a5d7fa700 1 -- 172.21.15.67:0/1395217924 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4a5000f690 con 0x7f4a60068580 2024-04-22T23:32:36.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.286+0000 7f4a5d7fa700 1 --2- 172.21.15.67:0/1395217924 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4a4c06ecc0 0x7f4a4c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:36.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.286+0000 7f4a5d7fa700 1 -- 172.21.15.67:0/1395217924 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f4a50090570 con 0x7f4a60068580 2024-04-22T23:32:36.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.286+0000 7f4a5f7fe700 1 --2- 172.21.15.67:0/1395217924 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4a4c06ecc0 0x7f4a4c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:36.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.287+0000 7f4a5f7fe700 1 --2- 172.21.15.67:0/1395217924 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4a4c06ecc0 0x7f4a4c071180 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4a48006fd0 tx=0x7f4a48008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:32:36.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.291+0000 7f4a5d7fa700 1 -- 172.21.15.67:0/1395217924 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f4a5005cd90 con 0x7f4a60068580 2024-04-22T23:32:36.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.529+0000 7f4a663c8700 1 -- 172.21.15.67:0/1395217924 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4a60197dc0 con 0x7f4a4c06ecc0 2024-04-22T23:32:36.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.530+0000 7f4a5d7fa700 1 -- 172.21.15.67:0/1395217924 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f4a60197dc0 con 0x7f4a4c06ecc0 2024-04-22T23:32:36.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.532+0000 7f4a663c8700 1 -- 172.21.15.67:0/1395217924 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4a4c06ecc0 msgr2=0x7f4a4c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:36.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.532+0000 7f4a663c8700 1 --2- 172.21.15.67:0/1395217924 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4a4c06ecc0 0x7f4a4c071180 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4a48006fd0 tx=0x7f4a48008040 comp rx=0 tx=0).stop 2024-04-22T23:32:36.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.532+0000 7f4a663c8700 1 -- 172.21.15.67:0/1395217924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a60068580 msgr2=0x7f4a60196980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:36.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.532+0000 7f4a663c8700 1 --2- 172.21.15.67:0/1395217924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a60068580 0x7f4a60196980 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4a50006010 tx=0x7f4a5000be20 comp rx=0 tx=0).stop 2024-04-22T23:32:36.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.532+0000 7f4a663c8700 1 -- 172.21.15.67:0/1395217924 shutdown_connections 2024-04-22T23:32:36.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.532+0000 7f4a663c8700 1 --2- 172.21.15.67:0/1395217924 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4a4c06ecc0 0x7f4a4c071180 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:36.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.532+0000 7f4a663c8700 1 --2- 172.21.15.67:0/1395217924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a60068580 0x7f4a60196980 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:36.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.532+0000 7f4a663c8700 1 --2- 172.21.15.67:0/1395217924 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a60106cd0 0x7f4a60196ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:36.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.532+0000 7f4a663c8700 1 -- 172.21.15.67:0/1395217924 >> 172.21.15.67:0/1395217924 conn(0x7f4a60075670 msgr2=0x7f4a600fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:32:36.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.532+0000 7f4a663c8700 1 -- 172.21.15.67:0/1395217924 shutdown_connections 2024-04-22T23:32:36.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.532+0000 7f4a663c8700 1 -- 172.21.15.67:0/1395217924 wait complete. 2024-04-22T23:32:36.540 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:32:36.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.645+0000 7f41c1b9f700 1 -- 172.21.15.67:0/856801355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41bc0ff1c0 msgr2=0x7f41bc0ff640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:36.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.645+0000 7f41c1b9f700 1 --2- 172.21.15.67:0/856801355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41bc0ff1c0 0x7f41bc0ff640 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f41a8009910 tx=0x7f41a8009c20 comp rx=0 tx=0).stop 2024-04-22T23:32:36.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.645+0000 7f41c1b9f700 1 -- 172.21.15.67:0/856801355 shutdown_connections 2024-04-22T23:32:36.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.645+0000 7f41c1b9f700 1 --2- 172.21.15.67:0/856801355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41bc0ff1c0 0x7f41bc0ff640 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:36.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.645+0000 7f41c1b9f700 1 --2- 172.21.15.67:0/856801355 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41bc102980 0x7f41bc102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:36.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.645+0000 7f41c1b9f700 1 -- 172.21.15.67:0/856801355 >> 172.21.15.67:0/856801355 conn(0x7f41bc074c20 msgr2=0x7f41bc075030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:32:36.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.645+0000 7f41c1b9f700 1 -- 172.21.15.67:0/856801355 shutdown_connections 2024-04-22T23:32:36.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.645+0000 7f41c1b9f700 1 -- 172.21.15.67:0/856801355 wait complete. 2024-04-22T23:32:36.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.646+0000 7f41c1b9f700 1 Processor -- start 2024-04-22T23:32:36.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.646+0000 7f41c1b9f700 1 -- start start 2024-04-22T23:32:36.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.646+0000 7f41c1b9f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41bc0ff1c0 0x7f41bc1003a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:36.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.646+0000 7f41c1b9f700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41bc102980 0x7f41bc1008e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:36.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.646+0000 7f41c1b9f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41bc100e20 con 0x7f41bc0ff1c0 2024-04-22T23:32:36.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.646+0000 7f41c1b9f700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41bc100f90 con 0x7f41bc102980 2024-04-22T23:32:36.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.646+0000 7f41c0b9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41bc0ff1c0 0x7f41bc1003a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:36.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.646+0000 7f41c0b9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41bc0ff1c0 0x7f41bc1003a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40872/0 (socket says 172.21.15.67:40872) 2024-04-22T23:32:36.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.646+0000 7f41c0b9d700 1 -- 172.21.15.67:0/4129277605 learned_addr learned my addr 172.21.15.67:0/4129277605 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:32:36.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.647+0000 7f41bbfff700 1 --2- 172.21.15.67:0/4129277605 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41bc102980 0x7f41bc1008e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:36.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.647+0000 7f41c0b9d700 1 -- 172.21.15.67:0/4129277605 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41bc102980 msgr2=0x7f41bc1008e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:36.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.647+0000 7f41c0b9d700 1 --2- 172.21.15.67:0/4129277605 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41bc102980 0x7f41bc1008e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:36.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.647+0000 7f41c0b9d700 1 -- 172.21.15.67:0/4129277605 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a80095f0 con 0x7f41bc0ff1c0 2024-04-22T23:32:36.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.647+0000 7f41c0b9d700 1 --2- 172.21.15.67:0/4129277605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41bc0ff1c0 0x7f41bc1003a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f41b000c930 tx=0x7f41b000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:32:36.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.647+0000 7f41b9ffb700 1 -- 172.21.15.67:0/4129277605 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f41b0007ad0 con 0x7f41bc0ff1c0 2024-04-22T23:32:36.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.647+0000 7f41c1b9f700 1 -- 172.21.15.67:0/4129277605 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41bc101210 con 0x7f41bc0ff1c0 2024-04-22T23:32:36.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.647+0000 7f41b9ffb700 1 -- 172.21.15.67:0/4129277605 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f41b000ce80 con 0x7f41bc0ff1c0 2024-04-22T23:32:36.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.647+0000 7f41b9ffb700 1 -- 172.21.15.67:0/4129277605 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f41b0018630 con 0x7f41bc0ff1c0 2024-04-22T23:32:36.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.647+0000 7f41c1b9f700 1 -- 172.21.15.67:0/4129277605 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41bc1a2ca0 con 0x7f41bc0ff1c0 2024-04-22T23:32:36.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.649+0000 7f41c1b9f700 1 -- 172.21.15.67:0/4129277605 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41bc0623c0 con 0x7f41bc0ff1c0 2024-04-22T23:32:36.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.649+0000 7f41b9ffb700 1 -- 172.21.15.67:0/4129277605 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f41b0018790 con 0x7f41bc0ff1c0 2024-04-22T23:32:36.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.650+0000 7f41b9ffb700 1 --2- 172.21.15.67:0/4129277605 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f41ac06ecc0 0x7f41ac071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:36.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.650+0000 7f41b9ffb700 1 -- 172.21.15.67:0/4129277605 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f41b008ee20 con 0x7f41bc0ff1c0 2024-04-22T23:32:36.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.650+0000 7f41bbfff700 1 --2- 172.21.15.67:0/4129277605 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f41ac06ecc0 0x7f41ac071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:36.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.650+0000 7f41bbfff700 1 --2- 172.21.15.67:0/4129277605 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f41ac06ecc0 0x7f41ac071180 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f41bc101b90 tx=0x7f41a801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:32:36.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.655+0000 7f41b9ffb700 1 -- 172.21.15.67:0/4129277605 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f41b0057a60 con 0x7f41bc0ff1c0 2024-04-22T23:32:36.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.900+0000 7f41c1b9f700 1 -- 172.21.15.67:0/4129277605 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f41bc1019e0 con 0x7f41ac06ecc0 2024-04-22T23:32:36.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.902+0000 7f41b9ffb700 1 -- 172.21.15.67:0/4129277605 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f41bc1019e0 con 0x7f41ac06ecc0 2024-04-22T23:32:36.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.904+0000 7f41c1b9f700 1 -- 172.21.15.67:0/4129277605 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f41ac06ecc0 msgr2=0x7f41ac071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:36.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.904+0000 7f41c1b9f700 1 --2- 172.21.15.67:0/4129277605 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f41ac06ecc0 0x7f41ac071180 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f41bc101b90 tx=0x7f41a801a040 comp rx=0 tx=0).stop 2024-04-22T23:32:36.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.904+0000 7f41c1b9f700 1 -- 172.21.15.67:0/4129277605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41bc0ff1c0 msgr2=0x7f41bc1003a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:36.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.904+0000 7f41c1b9f700 1 --2- 172.21.15.67:0/4129277605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41bc0ff1c0 0x7f41bc1003a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f41b000c930 tx=0x7f41b000cc40 comp rx=0 tx=0).stop 2024-04-22T23:32:36.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.904+0000 7f41c1b9f700 1 -- 172.21.15.67:0/4129277605 shutdown_connections 2024-04-22T23:32:36.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.904+0000 7f41c1b9f700 1 --2- 172.21.15.67:0/4129277605 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f41ac06ecc0 0x7f41ac071180 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:36.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.904+0000 7f41c1b9f700 1 --2- 172.21.15.67:0/4129277605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41bc0ff1c0 0x7f41bc1003a0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:36.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.904+0000 7f41c1b9f700 1 --2- 172.21.15.67:0/4129277605 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f41bc102980 0x7f41bc1008e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:36.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.904+0000 7f41c1b9f700 1 -- 172.21.15.67:0/4129277605 >> 172.21.15.67:0/4129277605 conn(0x7f41bc074c20 msgr2=0x7f41bc0fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:32:36.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.905+0000 7f41c1b9f700 1 -- 172.21.15.67:0/4129277605 shutdown_connections 2024-04-22T23:32:36.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:36.905+0000 7f41c1b9f700 1 -- 172.21.15.67:0/4129277605 wait complete. 2024-04-22T23:32:37.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.014+0000 7f16209a3700 1 -- 172.21.15.67:0/1163493364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f161c0ffc20 msgr2=0x7f161c100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:37.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.014+0000 7f16209a3700 1 --2- 172.21.15.67:0/1163493364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f161c0ffc20 0x7f161c100080 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1610009910 tx=0x7f1610009c20 comp rx=0 tx=0).stop 2024-04-22T23:32:37.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.014+0000 7f16209a3700 1 -- 172.21.15.67:0/1163493364 shutdown_connections 2024-04-22T23:32:37.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.014+0000 7f16209a3700 1 --2- 172.21.15.67:0/1163493364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f161c0ffc20 0x7f161c100080 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:37.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.014+0000 7f16209a3700 1 --2- 172.21.15.67:0/1163493364 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f161c105bd0 0x7f161c105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:37.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.014+0000 7f16209a3700 1 -- 172.21.15.67:0/1163493364 >> 172.21.15.67:0/1163493364 conn(0x7f161c075670 msgr2=0x7f161c075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:32:37.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.014+0000 7f16209a3700 1 -- 172.21.15.67:0/1163493364 shutdown_connections 2024-04-22T23:32:37.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.015+0000 7f16209a3700 1 -- 172.21.15.67:0/1163493364 wait complete. 2024-04-22T23:32:37.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.015+0000 7f16209a3700 1 Processor -- start 2024-04-22T23:32:37.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.015+0000 7f16209a3700 1 -- start start 2024-04-22T23:32:37.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.016+0000 7f16209a3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f161c0ffc20 0x7f161c19c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:37.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.016+0000 7f16209a3700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f161c105bd0 0x7f161c19cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:37.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.016+0000 7f16209a3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f161c19d240 con 0x7f161c0ffc20 2024-04-22T23:32:37.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.016+0000 7f16209a3700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f161c196760 con 0x7f161c105bd0 2024-04-22T23:32:37.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.016+0000 7f161a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f161c0ffc20 0x7f161c19c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:37.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.016+0000 7f161a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f161c0ffc20 0x7f161c19c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40886/0 (socket says 172.21.15.67:40886) 2024-04-22T23:32:37.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.016+0000 7f161a59c700 1 -- 172.21.15.67:0/1381629017 learned_addr learned my addr 172.21.15.67:0/1381629017 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:32:37.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.016+0000 7f1619d9b700 1 --2- 172.21.15.67:0/1381629017 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f161c105bd0 0x7f161c19cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:37.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.016+0000 7f161a59c700 1 -- 172.21.15.67:0/1381629017 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f161c105bd0 msgr2=0x7f161c19cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:37.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.016+0000 7f161a59c700 1 --2- 172.21.15.67:0/1381629017 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f161c105bd0 0x7f161c19cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:37.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.016+0000 7f161a59c700 1 -- 172.21.15.67:0/1381629017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16100095f0 con 0x7f161c0ffc20 2024-04-22T23:32:37.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.016+0000 7f161a59c700 1 --2- 172.21.15.67:0/1381629017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f161c0ffc20 0x7f161c19c670 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f160c00d8d0 tx=0x7f160c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:32:37.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.017+0000 7f160b7fe700 1 -- 172.21.15.67:0/1381629017 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f160c0098a0 con 0x7f161c0ffc20 2024-04-22T23:32:37.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.017+0000 7f160b7fe700 1 -- 172.21.15.67:0/1381629017 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f160c010460 con 0x7f161c0ffc20 2024-04-22T23:32:37.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.017+0000 7f16209a3700 1 -- 172.21.15.67:0/1381629017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f161c196a40 con 0x7f161c0ffc20 2024-04-22T23:32:37.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.017+0000 7f160b7fe700 1 -- 172.21.15.67:0/1381629017 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f160c00f5f0 con 0x7f161c0ffc20 2024-04-22T23:32:37.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.017+0000 7f16209a3700 1 -- 172.21.15.67:0/1381629017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f161c196f60 con 0x7f161c0ffc20 2024-04-22T23:32:37.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.018+0000 7f16209a3700 1 -- 172.21.15.67:0/1381629017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f161c04f2e0 con 0x7f161c0ffc20 2024-04-22T23:32:37.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.018+0000 7f160b7fe700 1 -- 172.21.15.67:0/1381629017 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f160c009a00 con 0x7f161c0ffc20 2024-04-22T23:32:37.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.019+0000 7f160b7fe700 1 --2- 172.21.15.67:0/1381629017 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f160406ecc0 0x7f1604071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:37.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.019+0000 7f160b7fe700 1 -- 172.21.15.67:0/1381629017 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f160c08eea0 con 0x7f161c0ffc20 2024-04-22T23:32:37.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.019+0000 7f1619d9b700 1 --2- 172.21.15.67:0/1381629017 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f160406ecc0 0x7f1604071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:37.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.019+0000 7f1619d9b700 1 --2- 172.21.15.67:0/1381629017 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f160406ecc0 0x7f1604071180 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f161000b5c0 tx=0x7f161001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:32:37.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.024+0000 7f160b7fe700 1 -- 172.21.15.67:0/1381629017 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f160c05b610 con 0x7f161c0ffc20 2024-04-22T23:32:37.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.254+0000 7f16209a3700 1 -- 172.21.15.67:0/1381629017 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f161c197290 con 0x7f160406ecc0 2024-04-22T23:32:37.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.266+0000 7f160b7fe700 1 -- 172.21.15.67:0/1381629017 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f161c197290 con 0x7f160406ecc0 2024-04-22T23:32:37.262 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:32:37.262 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (6m) 117s ago 17m 26.1M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:32:37.262 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (17m) 117s ago 17m 8728k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:32:37.262 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (16m) 17s ago 16m 8828k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:32:37.262 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (3m) 117s ago 17m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:32:37.262 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (3m) 17s ago 16m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:32:37.263 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (5m) 117s ago 17m 80.1M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:32:37.263 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 starting - - - - 2024-04-22T23:32:37.263 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (12m) 117s ago 12m 130M - 18.2.1 5be31c24972a 0730f9fc8e61 2024-04-22T23:32:37.263 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (12m) 17s ago 12m 123M - 18.2.1 5be31c24972a b6a169586433 2024-04-22T23:32:37.263 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (12m) 17s ago 12m 67.9M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:32:37.263 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (8m) 117s ago 19m 506M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:32:37.263 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (7m) 17s ago 16m 452M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:32:37.263 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (4m) 117s ago 19m 49.7M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:32:37.263 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (4m) 17s ago 16m 44.1M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:32:37.264 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (7m) 117s ago 17m 15.4M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:32:37.264 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (6m) 17s ago 16m 16.2M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:32:37.264 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (3m) 117s ago 15m 95.4M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:32:37.264 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (2m) 117s ago 15m 131M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:32:37.264 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (2m) 117s ago 14m 15.1M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:32:37.264 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (87s) 17s ago 14m 97.7M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:32:37.264 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (54s) 17s ago 14m 82.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:32:37.264 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (20s) 17s ago 13m 12.0M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:32:37.264 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (6m) 117s ago 17m 71.3M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:32:37.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.268+0000 7f16209a3700 1 -- 172.21.15.67:0/1381629017 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f160406ecc0 msgr2=0x7f1604071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:37.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.268+0000 7f16209a3700 1 --2- 172.21.15.67:0/1381629017 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f160406ecc0 0x7f1604071180 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f161000b5c0 tx=0x7f161001a040 comp rx=0 tx=0).stop 2024-04-22T23:32:37.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.268+0000 7f16209a3700 1 -- 172.21.15.67:0/1381629017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f161c0ffc20 msgr2=0x7f161c19c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:37.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.268+0000 7f16209a3700 1 --2- 172.21.15.67:0/1381629017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f161c0ffc20 0x7f161c19c670 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f160c00d8d0 tx=0x7f160c00dbe0 comp rx=0 tx=0).stop 2024-04-22T23:32:37.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.268+0000 7f16209a3700 1 -- 172.21.15.67:0/1381629017 shutdown_connections 2024-04-22T23:32:37.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.268+0000 7f16209a3700 1 --2- 172.21.15.67:0/1381629017 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f160406ecc0 0x7f1604071180 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:37.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.268+0000 7f16209a3700 1 --2- 172.21.15.67:0/1381629017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f161c0ffc20 0x7f161c19c670 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:37.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.268+0000 7f16209a3700 1 --2- 172.21.15.67:0/1381629017 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f161c105bd0 0x7f161c19cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:37.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.268+0000 7f16209a3700 1 -- 172.21.15.67:0/1381629017 >> 172.21.15.67:0/1381629017 conn(0x7f161c075670 msgr2=0x7f161c103f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:32:37.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.268+0000 7f16209a3700 1 -- 172.21.15.67:0/1381629017 shutdown_connections 2024-04-22T23:32:37.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.268+0000 7f16209a3700 1 -- 172.21.15.67:0/1381629017 wait complete. 2024-04-22T23:32:37.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.391+0000 7fc63ae07700 1 -- 172.21.15.67:0/1944992328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc634100e90 msgr2=0x7fc634101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:37.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.391+0000 7fc63ae07700 1 --2- 172.21.15.67:0/1944992328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc634100e90 0x7fc634101270 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc628009910 tx=0x7fc628009c20 comp rx=0 tx=0).stop 2024-04-22T23:32:37.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.391+0000 7fc63ae07700 1 -- 172.21.15.67:0/1944992328 shutdown_connections 2024-04-22T23:32:37.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.391+0000 7fc63ae07700 1 --2- 172.21.15.67:0/1944992328 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc634101840 0x7fc634105890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:37.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.391+0000 7fc63ae07700 1 --2- 172.21.15.67:0/1944992328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc634100e90 0x7fc634101270 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:37.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.391+0000 7fc63ae07700 1 -- 172.21.15.67:0/1944992328 >> 172.21.15.67:0/1944992328 conn(0x7fc6340fc740 msgr2=0x7fc6340feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:32:37.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.391+0000 7fc63ae07700 1 -- 172.21.15.67:0/1944992328 shutdown_connections 2024-04-22T23:32:37.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.391+0000 7fc63ae07700 1 -- 172.21.15.67:0/1944992328 wait complete. 2024-04-22T23:32:37.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.392+0000 7fc63ae07700 1 Processor -- start 2024-04-22T23:32:37.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.392+0000 7fc63ae07700 1 -- start start 2024-04-22T23:32:37.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.393+0000 7fc63ae07700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc634100e90 0x7fc63419c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:37.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.393+0000 7fc63ae07700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc634101840 0x7fc63419cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:37.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.393+0000 7fc63ae07700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63419d230 con 0x7fc634100e90 2024-04-22T23:32:37.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.393+0000 7fc63ae07700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6341966e0 con 0x7fc634101840 2024-04-22T23:32:37.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.393+0000 7fc638ba3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc634100e90 0x7fc63419c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:37.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.393+0000 7fc638ba3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc634100e90 0x7fc63419c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40902/0 (socket says 172.21.15.67:40902) 2024-04-22T23:32:37.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.393+0000 7fc638ba3700 1 -- 172.21.15.67:0/2333956612 learned_addr learned my addr 172.21.15.67:0/2333956612 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:32:37.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.393+0000 7fc633fff700 1 --2- 172.21.15.67:0/2333956612 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc634101840 0x7fc63419cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:37.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.393+0000 7fc638ba3700 1 -- 172.21.15.67:0/2333956612 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc634101840 msgr2=0x7fc63419cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:37.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.393+0000 7fc638ba3700 1 --2- 172.21.15.67:0/2333956612 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc634101840 0x7fc63419cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:37.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.393+0000 7fc638ba3700 1 -- 172.21.15.67:0/2333956612 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6280095f0 con 0x7fc634100e90 2024-04-22T23:32:37.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.393+0000 7fc638ba3700 1 --2- 172.21.15.67:0/2333956612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc634100e90 0x7fc63419c660 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc6280092e0 tx=0x7fc62800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:32:37.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.393+0000 7fc631ffb700 1 -- 172.21.15.67:0/2333956612 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc62801d070 con 0x7fc634100e90 2024-04-22T23:32:37.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.394+0000 7fc63ae07700 1 -- 172.21.15.67:0/2333956612 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc634196960 con 0x7fc634100e90 2024-04-22T23:32:37.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.394+0000 7fc631ffb700 1 -- 172.21.15.67:0/2333956612 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc628005e00 con 0x7fc634100e90 2024-04-22T23:32:37.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.394+0000 7fc631ffb700 1 -- 172.21.15.67:0/2333956612 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc628017400 con 0x7fc634100e90 2024-04-22T23:32:37.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.394+0000 7fc63ae07700 1 -- 172.21.15.67:0/2333956612 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc634196e50 con 0x7fc634100e90 2024-04-22T23:32:37.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.395+0000 7fc63ae07700 1 -- 172.21.15.67:0/2333956612 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc634109230 con 0x7fc634100e90 2024-04-22T23:32:37.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.395+0000 7fc631ffb700 1 -- 172.21.15.67:0/2333956612 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc62800f6f0 con 0x7fc634100e90 2024-04-22T23:32:37.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.396+0000 7fc631ffb700 1 --2- 172.21.15.67:0/2333956612 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc62406ecc0 0x7fc624071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:37.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.396+0000 7fc631ffb700 1 -- 172.21.15.67:0/2333956612 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fc6280903a0 con 0x7fc634100e90 2024-04-22T23:32:37.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.396+0000 7fc633fff700 1 --2- 172.21.15.67:0/2333956612 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc62406ecc0 0x7fc624071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:37.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.397+0000 7fc633fff700 1 --2- 172.21.15.67:0/2333956612 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc62406ecc0 0x7fc624071180 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc634197b50 tx=0x7fc62000b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:32:37.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.402+0000 7fc631ffb700 1 -- 172.21.15.67:0/2333956612 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc62805cb90 con 0x7fc634100e90 2024-04-22T23:32:37.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:37 smithi067 ceph-mon[112438]: pgmap v185: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s 2024-04-22T23:32:37.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:37 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:37.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:37 smithi067 ceph-mon[112438]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:32:37.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:37 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:37.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:37 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:37.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:37 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:32:37.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.703+0000 7fc63ae07700 1 -- 172.21.15.67:0/2333956612 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc634197cf0 con 0x7fc634100e90 2024-04-22T23:32:37.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.704+0000 7fc631ffb700 1 -- 172.21.15.67:0/2333956612 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fc634197cf0 con 0x7fc634100e90 2024-04-22T23:32:37.700 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:32:37.700 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:32:37.700 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:32:37.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:32:37.700 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:32:37.700 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:32:37.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:32:37.700 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:32:37.701 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:32:37.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:32:37.701 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:32:37.701 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 3, 2024-04-22T23:32:37.701 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 1 2024-04-22T23:32:37.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:32:37.701 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:32:37.701 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 3, 2024-04-22T23:32:37.701 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 11 2024-04-22T23:32:37.701 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:32:37.701 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:32:37.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.706+0000 7fc63ae07700 1 -- 172.21.15.67:0/2333956612 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc62406ecc0 msgr2=0x7fc624071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:37.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.706+0000 7fc63ae07700 1 --2- 172.21.15.67:0/2333956612 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc62406ecc0 0x7fc624071180 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc634197b50 tx=0x7fc62000b410 comp rx=0 tx=0).stop 2024-04-22T23:32:37.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.706+0000 7fc63ae07700 1 -- 172.21.15.67:0/2333956612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc634100e90 msgr2=0x7fc63419c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:37.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.706+0000 7fc63ae07700 1 --2- 172.21.15.67:0/2333956612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc634100e90 0x7fc63419c660 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc6280092e0 tx=0x7fc62800bd70 comp rx=0 tx=0).stop 2024-04-22T23:32:37.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.706+0000 7fc63ae07700 1 -- 172.21.15.67:0/2333956612 shutdown_connections 2024-04-22T23:32:37.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.706+0000 7fc63ae07700 1 --2- 172.21.15.67:0/2333956612 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc62406ecc0 0x7fc624071180 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:37.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.706+0000 7fc63ae07700 1 --2- 172.21.15.67:0/2333956612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc634100e90 0x7fc63419c660 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:37.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.706+0000 7fc63ae07700 1 --2- 172.21.15.67:0/2333956612 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc634101840 0x7fc63419cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:37.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.706+0000 7fc63ae07700 1 -- 172.21.15.67:0/2333956612 >> 172.21.15.67:0/2333956612 conn(0x7fc6340fc740 msgr2=0x7fc6340fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:32:37.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.706+0000 7fc63ae07700 1 -- 172.21.15.67:0/2333956612 shutdown_connections 2024-04-22T23:32:37.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.706+0000 7fc63ae07700 1 -- 172.21.15.67:0/2333956612 wait complete. 2024-04-22T23:32:37.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:37 smithi134 ceph-mon[85630]: pgmap v185: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s 2024-04-22T23:32:37.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:37 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:37.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:37 smithi134 ceph-mon[85630]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:32:37.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:37 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:37.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:37 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:37.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:37 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:32:37.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.817+0000 7f153705e700 1 -- 172.21.15.67:0/1347137498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1530100ea0 msgr2=0x7f1530101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:37.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.817+0000 7f153705e700 1 --2- 172.21.15.67:0/1347137498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1530100ea0 0x7f1530101280 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f152800b0a0 tx=0x7f152800b3b0 comp rx=0 tx=0).stop 2024-04-22T23:32:37.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.817+0000 7f153705e700 1 -- 172.21.15.67:0/1347137498 shutdown_connections 2024-04-22T23:32:37.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.817+0000 7f153705e700 1 --2- 172.21.15.67:0/1347137498 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1530101850 0x7f15301058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:37.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.817+0000 7f153705e700 1 --2- 172.21.15.67:0/1347137498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1530100ea0 0x7f1530101280 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:37.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.817+0000 7f153705e700 1 -- 172.21.15.67:0/1347137498 >> 172.21.15.67:0/1347137498 conn(0x7f15300fc750 msgr2=0x7f15300feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:32:37.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.817+0000 7f153705e700 1 -- 172.21.15.67:0/1347137498 shutdown_connections 2024-04-22T23:32:37.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.817+0000 7f153705e700 1 -- 172.21.15.67:0/1347137498 wait complete. 2024-04-22T23:32:37.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.817+0000 7f153705e700 1 Processor -- start 2024-04-22T23:32:37.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.818+0000 7f153705e700 1 -- start start 2024-04-22T23:32:37.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.818+0000 7f153705e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1530100ea0 0x7f1530073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:37.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.818+0000 7f153705e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1530101850 0x7f15300735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:37.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.818+0000 7f153705e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1530073b10 con 0x7f1530100ea0 2024-04-22T23:32:37.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.818+0000 7f153705e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1530073c50 con 0x7f1530101850 2024-04-22T23:32:37.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.818+0000 7f1534dfa700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1530100ea0 0x7f1530073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:37.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.818+0000 7f1534dfa700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1530100ea0 0x7f1530073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40918/0 (socket says 172.21.15.67:40918) 2024-04-22T23:32:37.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.818+0000 7f1534dfa700 1 -- 172.21.15.67:0/3567502292 learned_addr learned my addr 172.21.15.67:0/3567502292 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:32:37.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.818+0000 7f152ffff700 1 --2- 172.21.15.67:0/3567502292 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1530101850 0x7f15300735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:37.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.818+0000 7f1534dfa700 1 -- 172.21.15.67:0/3567502292 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1530101850 msgr2=0x7f15300735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:37.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.818+0000 7f1534dfa700 1 --2- 172.21.15.67:0/3567502292 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1530101850 0x7f15300735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:37.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.818+0000 7f1534dfa700 1 -- 172.21.15.67:0/3567502292 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1528009d00 con 0x7f1530100ea0 2024-04-22T23:32:37.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.818+0000 7f1534dfa700 1 --2- 172.21.15.67:0/3567502292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1530100ea0 0x7f1530073090 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f152800b070 tx=0x7f152800bc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:32:37.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.819+0000 7f152dffb700 1 -- 172.21.15.67:0/3567502292 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f152800e040 con 0x7f1530100ea0 2024-04-22T23:32:37.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.819+0000 7f152dffb700 1 -- 172.21.15.67:0/3567502292 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1528003c10 con 0x7f1530100ea0 2024-04-22T23:32:37.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.819+0000 7f153705e700 1 -- 172.21.15.67:0/3567502292 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1530073ed0 con 0x7f1530100ea0 2024-04-22T23:32:37.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.819+0000 7f152dffb700 1 -- 172.21.15.67:0/3567502292 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f152801d6a0 con 0x7f1530100ea0 2024-04-22T23:32:37.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.819+0000 7f153705e700 1 -- 172.21.15.67:0/3567502292 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15301a2e80 con 0x7f1530100ea0 2024-04-22T23:32:37.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.820+0000 7f153705e700 1 -- 172.21.15.67:0/3567502292 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1530109240 con 0x7f1530100ea0 2024-04-22T23:32:37.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.821+0000 7f152dffb700 1 -- 172.21.15.67:0/3567502292 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1528019070 con 0x7f1530100ea0 2024-04-22T23:32:37.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.821+0000 7f152dffb700 1 --2- 172.21.15.67:0/3567502292 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f151c06ecc0 0x7f151c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:37.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.821+0000 7f152dffb700 1 -- 172.21.15.67:0/3567502292 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f15280904c0 con 0x7f1530100ea0 2024-04-22T23:32:37.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.821+0000 7f152ffff700 1 --2- 172.21.15.67:0/3567502292 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f151c06ecc0 0x7f151c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:37.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.822+0000 7f152ffff700 1 --2- 172.21.15.67:0/3567502292 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f151c06ecc0 0x7f151c071180 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1530074880 tx=0x7f1520009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:32:37.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:37.826+0000 7f152dffb700 1 -- 172.21.15.67:0/3567502292 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1528059100 con 0x7f1530100ea0 2024-04-22T23:32:38.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.115+0000 7f153705e700 1 -- 172.21.15.67:0/3567502292 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f153004f2e0 con 0x7f1530100ea0 2024-04-22T23:32:38.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.116+0000 7f152dffb700 1 -- 172.21.15.67:0/3567502292 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) v1 ==== 76+0+1871 (secure 0 0 0) 0x7f152805c7c0 con 0x7f1530100ea0 2024-04-22T23:32:38.112 INFO:teuthology.orchestra.run.smithi067.stdout:e27 2024-04-22T23:32:38.112 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:32:38.112 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:32:38.112 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:32:38.112 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:32:38.112 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:32:38.112 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:32:38.112 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:32:38.112 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:32:38.113 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:32:38.113 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:32:38.113 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:32:38.113 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:32:38.113 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:32:38.113 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:32:38.113 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:32:38.113 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:32:38.113 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:32:38.113 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:32:38.113 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:32:38.113 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:32:38.113 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:32:38.113 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:32:38.114 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:32:38.114 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:32:38.114 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:32:38.114 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:32:38.114 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:32:38.114 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:32:38.114 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:32:38.114 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:32:38.114 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:32:38.114 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:32:38.114 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:32:38.114 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:32:38.114 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:32:38.114 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:32:38.114 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:32:38.115 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/2016103304,v1:172.21.15.67:6829/2016103304] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:32:38.115 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:32:38.115 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44259} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1827288036,v1:172.21.15.134:6827/1827288036] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:32:38.115 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44267} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/3366499307,v1:172.21.15.134:6825/3366499307] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:32:38.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.118+0000 7f153705e700 1 -- 172.21.15.67:0/3567502292 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f151c06ecc0 msgr2=0x7f151c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:38.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.118+0000 7f153705e700 1 --2- 172.21.15.67:0/3567502292 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f151c06ecc0 0x7f151c071180 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1530074880 tx=0x7f1520009260 comp rx=0 tx=0).stop 2024-04-22T23:32:38.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.118+0000 7f153705e700 1 -- 172.21.15.67:0/3567502292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1530100ea0 msgr2=0x7f1530073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:38.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.118+0000 7f153705e700 1 --2- 172.21.15.67:0/3567502292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1530100ea0 0x7f1530073090 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f152800b070 tx=0x7f152800bc60 comp rx=0 tx=0).stop 2024-04-22T23:32:38.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.118+0000 7f153705e700 1 -- 172.21.15.67:0/3567502292 shutdown_connections 2024-04-22T23:32:38.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.118+0000 7f153705e700 1 --2- 172.21.15.67:0/3567502292 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f151c06ecc0 0x7f151c071180 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:38.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.118+0000 7f153705e700 1 --2- 172.21.15.67:0/3567502292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1530100ea0 0x7f1530073090 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:38.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.118+0000 7f153705e700 1 --2- 172.21.15.67:0/3567502292 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1530101850 0x7f15300735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:38.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.118+0000 7f153705e700 1 -- 172.21.15.67:0/3567502292 >> 172.21.15.67:0/3567502292 conn(0x7f15300fc750 msgr2=0x7f15300fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:32:38.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.118+0000 7f153705e700 1 -- 172.21.15.67:0/3567502292 shutdown_connections 2024-04-22T23:32:38.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.119+0000 7f153705e700 1 -- 172.21.15.67:0/3567502292 wait complete. 2024-04-22T23:32:38.117 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 27 2024-04-22T23:32:38.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.244+0000 7fb98ad18700 1 -- 172.21.15.67:0/1852935918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb984101de0 msgr2=0x7fb9841021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:38.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.244+0000 7fb98ad18700 1 --2- 172.21.15.67:0/1852935918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb984101de0 0x7fb9841021c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb97c00b0a0 tx=0x7fb97c00b3b0 comp rx=0 tx=0).stop 2024-04-22T23:32:38.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.244+0000 7fb98ad18700 1 -- 172.21.15.67:0/1852935918 shutdown_connections 2024-04-22T23:32:38.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.244+0000 7fb98ad18700 1 --2- 172.21.15.67:0/1852935918 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb984102700 0x7fb98410abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:38.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.244+0000 7fb98ad18700 1 --2- 172.21.15.67:0/1852935918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb984101de0 0x7fb9841021c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:38.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.244+0000 7fb98ad18700 1 -- 172.21.15.67:0/1852935918 >> 172.21.15.67:0/1852935918 conn(0x7fb9840fb610 msgr2=0x7fb9840fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:32:38.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.244+0000 7fb98ad18700 1 -- 172.21.15.67:0/1852935918 shutdown_connections 2024-04-22T23:32:38.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.244+0000 7fb98ad18700 1 -- 172.21.15.67:0/1852935918 wait complete. 2024-04-22T23:32:38.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.245+0000 7fb98ad18700 1 Processor -- start 2024-04-22T23:32:38.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.245+0000 7fb98ad18700 1 -- start start 2024-04-22T23:32:38.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.245+0000 7fb98ad18700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb984101de0 0x7fb98419c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:38.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.245+0000 7fb98ad18700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb984102700 0x7fb98419cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:38.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.245+0000 7fb98ad18700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb98419d270 con 0x7fb984102700 2024-04-22T23:32:38.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.245+0000 7fb98ad18700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb984196720 con 0x7fb984101de0 2024-04-22T23:32:38.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.245+0000 7fb983fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb984102700 0x7fb98419cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:38.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.245+0000 7fb983fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb984102700 0x7fb98419cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40930/0 (socket says 172.21.15.67:40930) 2024-04-22T23:32:38.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.245+0000 7fb983fff700 1 -- 172.21.15.67:0/1655895544 learned_addr learned my addr 172.21.15.67:0/1655895544 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:32:38.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.245+0000 7fb988ab4700 1 --2- 172.21.15.67:0/1655895544 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb984101de0 0x7fb98419c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:38.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.246+0000 7fb983fff700 1 -- 172.21.15.67:0/1655895544 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb984101de0 msgr2=0x7fb98419c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:38.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.246+0000 7fb983fff700 1 --2- 172.21.15.67:0/1655895544 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb984101de0 0x7fb98419c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:38.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.246+0000 7fb983fff700 1 -- 172.21.15.67:0/1655895544 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb97c009d00 con 0x7fb984102700 2024-04-22T23:32:38.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.246+0000 7fb983fff700 1 --2- 172.21.15.67:0/1655895544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb984102700 0x7fb98419cbe0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb96c00d6a0 tx=0x7fb96c00d9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:32:38.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.246+0000 7fb981ffb700 1 -- 172.21.15.67:0/1655895544 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb96c004a00 con 0x7fb984102700 2024-04-22T23:32:38.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.246+0000 7fb981ffb700 1 -- 172.21.15.67:0/1655895544 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb96c0054e0 con 0x7fb984102700 2024-04-22T23:32:38.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.246+0000 7fb98ad18700 1 -- 172.21.15.67:0/1655895544 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb984196a00 con 0x7fb984102700 2024-04-22T23:32:38.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.246+0000 7fb981ffb700 1 -- 172.21.15.67:0/1655895544 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb96c00fca0 con 0x7fb984102700 2024-04-22T23:32:38.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.246+0000 7fb98ad18700 1 -- 172.21.15.67:0/1655895544 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb984196f50 con 0x7fb984102700 2024-04-22T23:32:38.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.247+0000 7fb98ad18700 1 -- 172.21.15.67:0/1655895544 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9841082f0 con 0x7fb984102700 2024-04-22T23:32:38.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.248+0000 7fb981ffb700 1 -- 172.21.15.67:0/1655895544 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb96c005020 con 0x7fb984102700 2024-04-22T23:32:38.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.248+0000 7fb981ffb700 1 --2- 172.21.15.67:0/1655895544 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb97006ecc0 0x7fb970071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:38.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.249+0000 7fb981ffb700 1 -- 172.21.15.67:0/1655895544 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fb96c08e9e0 con 0x7fb984102700 2024-04-22T23:32:38.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.249+0000 7fb988ab4700 1 --2- 172.21.15.67:0/1655895544 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb97006ecc0 0x7fb970071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:38.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.249+0000 7fb988ab4700 1 --2- 172.21.15.67:0/1655895544 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb97006ecc0 0x7fb970071180 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb97c000f80 tx=0x7fb97c0060c0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:32:38.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.253+0000 7fb981ffb700 1 -- 172.21.15.67:0/1655895544 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb96c05b150 con 0x7fb984102700 2024-04-22T23:32:38.345 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:38 smithi067 ceph-mon[112438]: from='client.34400 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:32:38.345 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:38 smithi067 ceph-mon[112438]: from='client.34404 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:32:38.345 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:38 smithi067 ceph-mon[112438]: mds.? [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] up:boot 2024-04-22T23:32:38.345 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:38 smithi067 ceph-mon[112438]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-22T23:32:38.345 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:38 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.aqlxpv"}]: dispatch 2024-04-22T23:32:38.345 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:38 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2333956612' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:32:38.345 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:38 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3567502292' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:32:38.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.499+0000 7fb98ad18700 1 -- 172.21.15.67:0/1655895544 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb984197170 con 0x7fb97006ecc0 2024-04-22T23:32:38.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.500+0000 7fb981ffb700 1 -- 172.21.15.67:0/1655895544 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb984197170 con 0x7fb97006ecc0 2024-04-22T23:32:38.496 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:32:38.496 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:32:38.496 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:32:38.496 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:32:38.496 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:32:38.496 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:32:38.496 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:32:38.496 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:32:38.496 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:32:38.497 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:32:38.497 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "12/23 daemons upgraded", 2024-04-22T23:32:38.497 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:32:38.497 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:32:38.497 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:32:38.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.502+0000 7fb98ad18700 1 -- 172.21.15.67:0/1655895544 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb97006ecc0 msgr2=0x7fb970071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:38.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.502+0000 7fb98ad18700 1 --2- 172.21.15.67:0/1655895544 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb97006ecc0 0x7fb970071180 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb97c000f80 tx=0x7fb97c0060c0 comp rx=0 tx=0).stop 2024-04-22T23:32:38.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.502+0000 7fb98ad18700 1 -- 172.21.15.67:0/1655895544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb984102700 msgr2=0x7fb98419cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:38.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.502+0000 7fb98ad18700 1 --2- 172.21.15.67:0/1655895544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb984102700 0x7fb98419cbe0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb96c00d6a0 tx=0x7fb96c00d9b0 comp rx=0 tx=0).stop 2024-04-22T23:32:38.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.502+0000 7fb98ad18700 1 -- 172.21.15.67:0/1655895544 shutdown_connections 2024-04-22T23:32:38.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.502+0000 7fb98ad18700 1 --2- 172.21.15.67:0/1655895544 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb97006ecc0 0x7fb970071180 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:38.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.502+0000 7fb98ad18700 1 --2- 172.21.15.67:0/1655895544 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb984101de0 0x7fb98419c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:38.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.502+0000 7fb98ad18700 1 --2- 172.21.15.67:0/1655895544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb984102700 0x7fb98419cbe0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:38.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.502+0000 7fb98ad18700 1 -- 172.21.15.67:0/1655895544 >> 172.21.15.67:0/1655895544 conn(0x7fb9840fb610 msgr2=0x7fb984105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:32:38.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.503+0000 7fb98ad18700 1 -- 172.21.15.67:0/1655895544 shutdown_connections 2024-04-22T23:32:38.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.503+0000 7fb98ad18700 1 -- 172.21.15.67:0/1655895544 wait complete. 2024-04-22T23:32:38.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.611+0000 7f8506ee2700 1 -- 172.21.15.67:0/314302030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8500101830 msgr2=0x7f8500105880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:38.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.611+0000 7f8506ee2700 1 --2- 172.21.15.67:0/314302030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8500101830 0x7f8500105880 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f84ec009910 tx=0x7f84ec009c20 comp rx=0 tx=0).stop 2024-04-22T23:32:38.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.611+0000 7f8506ee2700 1 -- 172.21.15.67:0/314302030 shutdown_connections 2024-04-22T23:32:38.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.611+0000 7f8506ee2700 1 --2- 172.21.15.67:0/314302030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8500101830 0x7f8500105880 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:38.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.611+0000 7f8506ee2700 1 --2- 172.21.15.67:0/314302030 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8500100e80 0x7f8500101260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:38.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.611+0000 7f8506ee2700 1 -- 172.21.15.67:0/314302030 >> 172.21.15.67:0/314302030 conn(0x7f85000fc730 msgr2=0x7f85000feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:32:38.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.611+0000 7f8506ee2700 1 -- 172.21.15.67:0/314302030 shutdown_connections 2024-04-22T23:32:38.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.611+0000 7f8506ee2700 1 -- 172.21.15.67:0/314302030 wait complete. 2024-04-22T23:32:38.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.612+0000 7f8506ee2700 1 Processor -- start 2024-04-22T23:32:38.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.612+0000 7f8506ee2700 1 -- start start 2024-04-22T23:32:38.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.612+0000 7f8506ee2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8500100e80 0x7f850019c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:38.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.612+0000 7f8506ee2700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8500101830 0x7f850019ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:38.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.612+0000 7f8506ee2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850019d380 con 0x7f8500100e80 2024-04-22T23:32:38.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.612+0000 7f8506ee2700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8500196830 con 0x7f8500101830 2024-04-22T23:32:38.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.612+0000 7f8504c7e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8500100e80 0x7f850019c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:38.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.612+0000 7f8504c7e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8500100e80 0x7f850019c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40942/0 (socket says 172.21.15.67:40942) 2024-04-22T23:32:38.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.612+0000 7f8504c7e700 1 -- 172.21.15.67:0/366365560 learned_addr learned my addr 172.21.15.67:0/366365560 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:32:38.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.613+0000 7f84fffff700 1 --2- 172.21.15.67:0/366365560 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8500101830 0x7f850019ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:38.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.613+0000 7f8504c7e700 1 -- 172.21.15.67:0/366365560 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8500101830 msgr2=0x7f850019ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:38.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.613+0000 7f8504c7e700 1 --2- 172.21.15.67:0/366365560 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8500101830 0x7f850019ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:38.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.613+0000 7f8504c7e700 1 -- 172.21.15.67:0/366365560 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84ec0095f0 con 0x7f8500100e80 2024-04-22T23:32:38.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.613+0000 7f8504c7e700 1 --2- 172.21.15.67:0/366365560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8500100e80 0x7f850019c7b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f84f400b4b0 tx=0x7f84f400b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:32:38.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.613+0000 7f84fdffb700 1 -- 172.21.15.67:0/366365560 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84f4011800 con 0x7f8500100e80 2024-04-22T23:32:38.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.613+0000 7f84fdffb700 1 -- 172.21.15.67:0/366365560 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f84f4011e20 con 0x7f8500100e80 2024-04-22T23:32:38.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.613+0000 7f84fdffb700 1 -- 172.21.15.67:0/366365560 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84f400f2e0 con 0x7f8500100e80 2024-04-22T23:32:38.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.613+0000 7f8506ee2700 1 -- 172.21.15.67:0/366365560 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8500196b10 con 0x7f8500100e80 2024-04-22T23:32:38.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.614+0000 7f8506ee2700 1 -- 172.21.15.67:0/366365560 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8500196fd0 con 0x7f8500100e80 2024-04-22T23:32:38.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.615+0000 7f8506ee2700 1 -- 172.21.15.67:0/366365560 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85001091b0 con 0x7f8500100e80 2024-04-22T23:32:38.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.615+0000 7f84fdffb700 1 -- 172.21.15.67:0/366365560 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f84f4010380 con 0x7f8500100e80 2024-04-22T23:32:38.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.616+0000 7f84fdffb700 1 --2- 172.21.15.67:0/366365560 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f84f006ecc0 0x7f84f0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:32:38.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.616+0000 7f84fffff700 1 --2- 172.21.15.67:0/366365560 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f84f006ecc0 0x7f84f0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:32:38.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.616+0000 7f84fdffb700 1 -- 172.21.15.67:0/366365560 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f84f408eaf0 con 0x7f8500100e80 2024-04-22T23:32:38.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.616+0000 7f84fffff700 1 --2- 172.21.15.67:0/366365560 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f84f006ecc0 0x7f84f0071180 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f84ec006010 tx=0x7f84ec01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:32:38.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.621+0000 7f84fdffb700 1 -- 172.21.15.67:0/366365560 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f84f405b260 con 0x7f8500100e80 2024-04-22T23:32:38.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:38 smithi134 ceph-mon[85630]: from='client.34400 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:32:38.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:38 smithi134 ceph-mon[85630]: from='client.34404 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:32:38.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:38 smithi134 ceph-mon[85630]: mds.? [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] up:boot 2024-04-22T23:32:38.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:38 smithi134 ceph-mon[85630]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-22T23:32:38.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:38 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.aqlxpv"}]: dispatch 2024-04-22T23:32:38.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:38 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2333956612' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:32:38.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:38 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3567502292' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:32:38.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.938+0000 7f8506ee2700 1 -- 172.21.15.67:0/366365560 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f850004f2e0 con 0x7f8500100e80 2024-04-22T23:32:38.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.938+0000 7f84fdffb700 1 -- 172.21.15.67:0/366365560 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f84f405adf0 con 0x7f8500100e80 2024-04-22T23:32:38.935 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:32:38.935 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:32:38.935 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:32:38.935 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:32:38.935 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:32:38.935 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:32:38.935 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:32:38.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.941+0000 7f8506ee2700 1 -- 172.21.15.67:0/366365560 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f84f006ecc0 msgr2=0x7f84f0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:38.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.941+0000 7f8506ee2700 1 --2- 172.21.15.67:0/366365560 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f84f006ecc0 0x7f84f0071180 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f84ec006010 tx=0x7f84ec01a040 comp rx=0 tx=0).stop 2024-04-22T23:32:38.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.941+0000 7f8506ee2700 1 -- 172.21.15.67:0/366365560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8500100e80 msgr2=0x7f850019c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:32:38.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.941+0000 7f8506ee2700 1 --2- 172.21.15.67:0/366365560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8500100e80 0x7f850019c7b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f84f400b4b0 tx=0x7f84f400b7c0 comp rx=0 tx=0).stop 2024-04-22T23:32:38.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.941+0000 7f8506ee2700 1 -- 172.21.15.67:0/366365560 shutdown_connections 2024-04-22T23:32:38.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.941+0000 7f8506ee2700 1 --2- 172.21.15.67:0/366365560 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f84f006ecc0 0x7f84f0071180 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:38.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.941+0000 7f8506ee2700 1 --2- 172.21.15.67:0/366365560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8500100e80 0x7f850019c7b0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:38.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.941+0000 7f8506ee2700 1 --2- 172.21.15.67:0/366365560 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8500101830 0x7f850019ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:32:38.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.941+0000 7f8506ee2700 1 -- 172.21.15.67:0/366365560 >> 172.21.15.67:0/366365560 conn(0x7f85000fc730 msgr2=0x7f85000fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:32:38.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.941+0000 7f8506ee2700 1 -- 172.21.15.67:0/366365560 shutdown_connections 2024-04-22T23:32:38.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:32:38.941+0000 7f8506ee2700 1 -- 172.21.15.67:0/366365560 wait complete. 2024-04-22T23:32:39.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:39 smithi067 ceph-mon[112438]: pgmap v186: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 102 B/s rd, 0 op/s 2024-04-22T23:32:39.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:39 smithi067 ceph-mon[112438]: from='client.34416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:32:39.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:39 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/366365560' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:32:39.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:39 smithi134 ceph-mon[85630]: pgmap v186: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 102 B/s rd, 0 op/s 2024-04-22T23:32:39.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:39 smithi134 ceph-mon[85630]: from='client.34416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:32:39.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:39 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/366365560' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:32:41.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:41 smithi067 ceph-mon[112438]: pgmap v187: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 102 B/s rd, 0 op/s 2024-04-22T23:32:41.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:41.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:32:41.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:41 smithi134 ceph-mon[85630]: pgmap v187: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 102 B/s rd, 0 op/s 2024-04-22T23:32:41.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:41.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:32:43.239 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:42 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:43.239 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:42 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:43.239 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:42 smithi134 ceph-mon[85630]: pgmap v188: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:32:43.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:42 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:43.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:42 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:43.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:42 smithi067 ceph-mon[112438]: pgmap v188: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:32:44.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:44 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:44.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:44 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:44.489 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:44 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:44.489 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:44 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:45.489 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:45 smithi134 ceph-mon[85630]: pgmap v189: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:32:45.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:45 smithi067 ceph-mon[112438]: pgmap v189: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:32:48.040 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:47 smithi067 ceph-mon[112438]: pgmap v190: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:32:48.040 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:47 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:48.040 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:47 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:48.040 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:47 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:32:48.040 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:47 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:32:48.040 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:47 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:48.040 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:47 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:32:48.040 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:47 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:32:48.040 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:47 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:32:48.041 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:47 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:32:48.041 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:47 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:32:48.239 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:47 smithi134 ceph-mon[85630]: pgmap v190: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:32:48.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:47 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:48.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:47 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:48.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:47 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:32:48.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:47 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:32:48.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:47 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:48.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:47 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:32:48.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:47 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:32:48.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:47 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:32:48.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:47 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:32:48.240 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:47 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:32:49.712 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:49 smithi067 ceph-mon[112438]: pgmap v191: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:32:49.712 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:49 smithi067 ceph-mon[112438]: Upgrade: Updating mds.cephfs.smithi067.xyazmh 2024-04-22T23:32:49.712 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:49 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:49.712 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:49 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi067.xyazmh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T23:32:49.712 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:49 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:32:49.712 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:49 smithi067 ceph-mon[112438]: Deploying daemon mds.cephfs.smithi067.xyazmh on smithi067 2024-04-22T23:32:49.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:49 smithi134 ceph-mon[85630]: pgmap v191: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:32:49.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:49 smithi134 ceph-mon[85630]: Upgrade: Updating mds.cephfs.smithi067.xyazmh 2024-04-22T23:32:49.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:49 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:49.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:49 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi067.xyazmh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T23:32:49.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:49 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:32:49.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:49 smithi134 ceph-mon[85630]: Deploying daemon mds.cephfs.smithi067.xyazmh on smithi067 2024-04-22T23:32:51.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:51 smithi134 ceph-mon[85630]: pgmap v192: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:32:51.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:51 smithi067 ceph-mon[112438]: pgmap v192: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:32:52.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:52 smithi134 ceph-mon[85630]: osdmap e92: 6 total, 6 up, 6 in 2024-04-22T23:32:52.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:52 smithi134 ceph-mon[85630]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-22T23:32:52.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:52 smithi067 ceph-mon[112438]: osdmap e92: 6 total, 6 up, 6 in 2024-04-22T23:32:52.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:52 smithi067 ceph-mon[112438]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-22T23:32:53.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:53 smithi067 ceph-mon[112438]: pgmap v194: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:32:53.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:53 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.xyazmh"}]: dispatch 2024-04-22T23:32:53.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:53 smithi134 ceph-mon[85630]: pgmap v194: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:32:53.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:53 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.xyazmh"}]: dispatch 2024-04-22T23:32:55.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:55 smithi067 ceph-mon[112438]: pgmap v195: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:32:55.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:55 smithi134 ceph-mon[85630]: pgmap v195: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:32:56.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:32:56.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:56.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:56.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:32:56.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:32:56.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:56.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:32:56.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:32:57.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:57 smithi067 ceph-mon[112438]: pgmap v196: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:32:57.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:57 smithi067 ceph-mon[112438]: mds.? [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] up:boot 2024-04-22T23:32:57.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:57 smithi067 ceph-mon[112438]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-22T23:32:57.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:57 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.xyazmh"}]: dispatch 2024-04-22T23:32:57.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:57 smithi134 ceph-mon[85630]: pgmap v196: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:32:57.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:57 smithi134 ceph-mon[85630]: mds.? [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] up:boot 2024-04-22T23:32:57.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:57 smithi134 ceph-mon[85630]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-22T23:32:57.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:57 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi067.xyazmh"}]: dispatch 2024-04-22T23:32:59.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:32:59 smithi067 ceph-mon[112438]: pgmap v197: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:32:59.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:32:59 smithi134 ceph-mon[85630]: pgmap v197: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:01.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:01 smithi067 ceph-mon[112438]: pgmap v198: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:01.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:01 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:33:01.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:01 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:33:01.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:01 smithi134 ceph-mon[85630]: pgmap v198: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:01.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:01 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:33:01.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:01 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:33:03.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:03 smithi067 ceph-mon[112438]: pgmap v199: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:03.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:03 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:33:03.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:03 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:33:03.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:03 smithi134 ceph-mon[85630]: pgmap v199: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:03.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:03 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:33:03.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:03 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:33:05.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:05 smithi067 ceph-mon[112438]: pgmap v200: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:05.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:05 smithi134 ceph-mon[85630]: pgmap v200: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:07.652 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:07 smithi067 ceph-mon[112438]: pgmap v201: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:07.652 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:07 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:33:07.652 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:07 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:33:07.652 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:07 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:33:07.653 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:07 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:33:07.653 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:07 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:33:07.653 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:07 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:33:07.653 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:07 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:33:07.653 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:07 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:33:07.653 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:07 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:33:07.653 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:07 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:33:07.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:07 smithi134 ceph-mon[85630]: pgmap v201: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:07.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:07 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:33:07.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:07 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:33:07.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:07 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:33:07.740 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:07 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:33:07.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:07 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:33:07.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:07 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:33:07.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:07 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:33:07.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:07 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:33:07.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:07 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:33:07.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:07 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:33:09.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.065+0000 7fde14c65700 1 -- 172.21.15.67:0/1276057072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde10101e40 msgr2=0x7fde10102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:09.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.065+0000 7fde14c65700 1 --2- 172.21.15.67:0/1276057072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde10101e40 0x7fde10102220 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fde0000b0d0 tx=0x7fde0000b3e0 comp rx=0 tx=0).stop 2024-04-22T23:33:09.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.065+0000 7fde14c65700 1 -- 172.21.15.67:0/1276057072 shutdown_connections 2024-04-22T23:33:09.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.065+0000 7fde14c65700 1 --2- 172.21.15.67:0/1276057072 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fde10102760 0x7fde1010ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:09.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.065+0000 7fde14c65700 1 --2- 172.21.15.67:0/1276057072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde10101e40 0x7fde10102220 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:09.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.065+0000 7fde14c65700 1 -- 172.21.15.67:0/1276057072 >> 172.21.15.67:0/1276057072 conn(0x7fde100fb650 msgr2=0x7fde100fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:33:09.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.065+0000 7fde14c65700 1 -- 172.21.15.67:0/1276057072 shutdown_connections 2024-04-22T23:33:09.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.065+0000 7fde14c65700 1 -- 172.21.15.67:0/1276057072 wait complete. 2024-04-22T23:33:09.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.066+0000 7fde14c65700 1 Processor -- start 2024-04-22T23:33:09.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.066+0000 7fde14c65700 1 -- start start 2024-04-22T23:33:09.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.066+0000 7fde14c65700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde10101e40 0x7fde1019c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:09.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.066+0000 7fde14c65700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fde10102760 0x7fde1019cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:09.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.066+0000 7fde14c65700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1019d230 con 0x7fde10101e40 2024-04-22T23:33:09.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.066+0000 7fde14c65700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde101966e0 con 0x7fde10102760 2024-04-22T23:33:09.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.066+0000 7fde0e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde10101e40 0x7fde1019c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:09.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.066+0000 7fde0e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde10101e40 0x7fde1019c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44250/0 (socket says 172.21.15.67:44250) 2024-04-22T23:33:09.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.066+0000 7fde0e59c700 1 -- 172.21.15.67:0/2070532950 learned_addr learned my addr 172.21.15.67:0/2070532950 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:33:09.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.067+0000 7fde0dd9b700 1 --2- 172.21.15.67:0/2070532950 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fde10102760 0x7fde1019cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:09.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.067+0000 7fde0e59c700 1 -- 172.21.15.67:0/2070532950 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fde10102760 msgr2=0x7fde1019cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:09.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.067+0000 7fde0e59c700 1 --2- 172.21.15.67:0/2070532950 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fde10102760 0x7fde1019cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:09.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.067+0000 7fde0e59c700 1 -- 172.21.15.67:0/2070532950 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde00009d00 con 0x7fde10101e40 2024-04-22T23:33:09.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.067+0000 7fde0e59c700 1 --2- 172.21.15.67:0/2070532950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde10101e40 0x7fde1019c660 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fde00015040 tx=0x7fde00009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:33:09.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.067+0000 7fde077fe700 1 -- 172.21.15.67:0/2070532950 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde0000e040 con 0x7fde10101e40 2024-04-22T23:33:09.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.067+0000 7fde14c65700 1 -- 172.21.15.67:0/2070532950 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde10196960 con 0x7fde10101e40 2024-04-22T23:33:09.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.067+0000 7fde077fe700 1 -- 172.21.15.67:0/2070532950 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fde00004040 con 0x7fde10101e40 2024-04-22T23:33:09.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.067+0000 7fde077fe700 1 -- 172.21.15.67:0/2070532950 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde0001d690 con 0x7fde10101e40 2024-04-22T23:33:09.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.067+0000 7fde14c65700 1 -- 172.21.15.67:0/2070532950 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde10196e50 con 0x7fde10101e40 2024-04-22T23:33:09.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.068+0000 7fde14c65700 1 -- 172.21.15.67:0/2070532950 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde10108350 con 0x7fde10101e40 2024-04-22T23:33:09.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.069+0000 7fde077fe700 1 -- 172.21.15.67:0/2070532950 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fde000041b0 con 0x7fde10101e40 2024-04-22T23:33:09.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.069+0000 7fde077fe700 1 --2- 172.21.15.67:0/2070532950 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fddfc06ecc0 0x7fddfc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:09.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.069+0000 7fde077fe700 1 -- 172.21.15.67:0/2070532950 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fde000906e0 con 0x7fde10101e40 2024-04-22T23:33:09.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.069+0000 7fde0dd9b700 1 --2- 172.21.15.67:0/2070532950 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fddfc06ecc0 0x7fddfc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:09.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.070+0000 7fde0dd9b700 1 --2- 172.21.15.67:0/2070532950 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fddfc06ecc0 0x7fddfc071180 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fde10197b50 tx=0x7fddf8009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:33:09.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.074+0000 7fde077fe700 1 -- 172.21.15.67:0/2070532950 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fde0005ce50 con 0x7fde10101e40 2024-04-22T23:33:09.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.308+0000 7fde14c65700 1 -- 172.21.15.67:0/2070532950 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fde10197070 con 0x7fddfc06ecc0 2024-04-22T23:33:09.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.309+0000 7fde077fe700 1 -- 172.21.15.67:0/2070532950 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fde10197070 con 0x7fddfc06ecc0 2024-04-22T23:33:09.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.311+0000 7fde14c65700 1 -- 172.21.15.67:0/2070532950 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fddfc06ecc0 msgr2=0x7fddfc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:09.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.311+0000 7fde14c65700 1 --2- 172.21.15.67:0/2070532950 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fddfc06ecc0 0x7fddfc071180 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fde10197b50 tx=0x7fddf8009260 comp rx=0 tx=0).stop 2024-04-22T23:33:09.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.311+0000 7fde14c65700 1 -- 172.21.15.67:0/2070532950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde10101e40 msgr2=0x7fde1019c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:09.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.311+0000 7fde14c65700 1 --2- 172.21.15.67:0/2070532950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde10101e40 0x7fde1019c660 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fde00015040 tx=0x7fde00009780 comp rx=0 tx=0).stop 2024-04-22T23:33:09.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.311+0000 7fde14c65700 1 -- 172.21.15.67:0/2070532950 shutdown_connections 2024-04-22T23:33:09.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.311+0000 7fde14c65700 1 --2- 172.21.15.67:0/2070532950 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fddfc06ecc0 0x7fddfc071180 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:09.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.311+0000 7fde14c65700 1 --2- 172.21.15.67:0/2070532950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde10101e40 0x7fde1019c660 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:09.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.311+0000 7fde14c65700 1 --2- 172.21.15.67:0/2070532950 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fde10102760 0x7fde1019cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:09.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.311+0000 7fde14c65700 1 -- 172.21.15.67:0/2070532950 >> 172.21.15.67:0/2070532950 conn(0x7fde100fb650 msgr2=0x7fde10105490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:33:09.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.312+0000 7fde14c65700 1 -- 172.21.15.67:0/2070532950 shutdown_connections 2024-04-22T23:33:09.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.312+0000 7fde14c65700 1 -- 172.21.15.67:0/2070532950 wait complete. 2024-04-22T23:33:09.318 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:33:09.318 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:09 smithi067 ceph-mon[112438]: pgmap v202: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:09.319 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:09 smithi067 ceph-mon[112438]: Upgrade: Updating mds.cephfs.smithi134.owipad 2024-04-22T23:33:09.319 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:09 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:33:09.319 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:09 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi134.owipad", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T23:33:09.319 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:09 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:33:09.319 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:09 smithi067 ceph-mon[112438]: Deploying daemon mds.cephfs.smithi134.owipad on smithi134 2024-04-22T23:33:09.374 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:09 smithi134 ceph-mon[85630]: pgmap v202: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:09.374 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:09 smithi134 ceph-mon[85630]: Upgrade: Updating mds.cephfs.smithi134.owipad 2024-04-22T23:33:09.374 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:09 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:33:09.374 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:09 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi134.owipad", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T23:33:09.375 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:09 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:33:09.375 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:09 smithi134 ceph-mon[85630]: Deploying daemon mds.cephfs.smithi134.owipad on smithi134 2024-04-22T23:33:09.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.421+0000 7f94205d1700 1 -- 172.21.15.67:0/426599659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9418101de0 msgr2=0x7f94181021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:09.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.421+0000 7f94205d1700 1 --2- 172.21.15.67:0/426599659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9418101de0 0x7f94181021c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f940800b0a0 tx=0x7f940800b3b0 comp rx=0 tx=0).stop 2024-04-22T23:33:09.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.421+0000 7f94205d1700 1 -- 172.21.15.67:0/426599659 shutdown_connections 2024-04-22T23:33:09.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.421+0000 7f94205d1700 1 --2- 172.21.15.67:0/426599659 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9418102700 0x7f941810abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:09.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.421+0000 7f94205d1700 1 --2- 172.21.15.67:0/426599659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9418101de0 0x7f94181021c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:09.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.421+0000 7f94205d1700 1 -- 172.21.15.67:0/426599659 >> 172.21.15.67:0/426599659 conn(0x7f94180fb650 msgr2=0x7f94180fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:33:09.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.421+0000 7f94205d1700 1 -- 172.21.15.67:0/426599659 shutdown_connections 2024-04-22T23:33:09.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.421+0000 7f94205d1700 1 -- 172.21.15.67:0/426599659 wait complete. 2024-04-22T23:33:09.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.422+0000 7f94205d1700 1 Processor -- start 2024-04-22T23:33:09.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.422+0000 7f94205d1700 1 -- start start 2024-04-22T23:33:09.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.422+0000 7f94205d1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9418101de0 0x7f94180ffa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:09.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.422+0000 7f94205d1700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9418102700 0x7f94180fff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:09.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.422+0000 7f94205d1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94181004c0 con 0x7f9418101de0 2024-04-22T23:33:09.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.422+0000 7f94205d1700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9418100600 con 0x7f9418102700 2024-04-22T23:33:09.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.422+0000 7f941e36d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9418101de0 0x7f94180ffa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:09.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.422+0000 7f941e36d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9418101de0 0x7f94180ffa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44278/0 (socket says 172.21.15.67:44278) 2024-04-22T23:33:09.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.422+0000 7f941e36d700 1 -- 172.21.15.67:0/2595259322 learned_addr learned my addr 172.21.15.67:0/2595259322 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:33:09.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.422+0000 7f941db6c700 1 --2- 172.21.15.67:0/2595259322 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9418102700 0x7f94180fff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:09.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.422+0000 7f941e36d700 1 -- 172.21.15.67:0/2595259322 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9418102700 msgr2=0x7f94180fff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:09.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.422+0000 7f941e36d700 1 --2- 172.21.15.67:0/2595259322 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9418102700 0x7f94180fff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:09.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.422+0000 7f941e36d700 1 -- 172.21.15.67:0/2595259322 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9408009d00 con 0x7f9418101de0 2024-04-22T23:33:09.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.423+0000 7f941e36d700 1 --2- 172.21.15.67:0/2595259322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9418101de0 0x7f94180ffa40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9408000f80 tx=0x7f9408003c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:33:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.423+0000 7f94177fe700 1 -- 172.21.15.67:0/2595259322 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f940800e040 con 0x7f9418101de0 2024-04-22T23:33:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.423+0000 7f94177fe700 1 -- 172.21.15.67:0/2595259322 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9408007c00 con 0x7f9418101de0 2024-04-22T23:33:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.423+0000 7f94177fe700 1 -- 172.21.15.67:0/2595259322 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f940801d6b0 con 0x7f9418101de0 2024-04-22T23:33:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.423+0000 7f94205d1700 1 -- 172.21.15.67:0/2595259322 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9418100880 con 0x7f9418101de0 2024-04-22T23:33:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.423+0000 7f94205d1700 1 -- 172.21.15.67:0/2595259322 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94181a2dd0 con 0x7f9418101de0 2024-04-22T23:33:09.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.424+0000 7f94205d1700 1 -- 172.21.15.67:0/2595259322 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94181082f0 con 0x7f9418101de0 2024-04-22T23:33:09.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.424+0000 7f94177fe700 1 -- 172.21.15.67:0/2595259322 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9408004590 con 0x7f9418101de0 2024-04-22T23:33:09.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.425+0000 7f94177fe700 1 --2- 172.21.15.67:0/2595259322 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f940006ecc0 0x7f9400071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:09.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.425+0000 7f94177fe700 1 -- 172.21.15.67:0/2595259322 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f94080906a0 con 0x7f9418101de0 2024-04-22T23:33:09.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.425+0000 7f941db6c700 1 --2- 172.21.15.67:0/2595259322 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f940006ecc0 0x7f9400071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:09.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.425+0000 7f941db6c700 1 --2- 172.21.15.67:0/2595259322 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f940006ecc0 0x7f9400071180 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9418101230 tx=0x7f940c00b360 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:33:09.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.430+0000 7f94177fe700 1 -- 172.21.15.67:0/2595259322 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f94080592e0 con 0x7f9418101de0 2024-04-22T23:33:09.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.661+0000 7f94205d1700 1 -- 172.21.15.67:0/2595259322 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f94180611d0 con 0x7f940006ecc0 2024-04-22T23:33:09.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.662+0000 7f94177fe700 1 -- 172.21.15.67:0/2595259322 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f94180611d0 con 0x7f940006ecc0 2024-04-22T23:33:09.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.664+0000 7f94205d1700 1 -- 172.21.15.67:0/2595259322 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f940006ecc0 msgr2=0x7f9400071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:09.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.664+0000 7f94205d1700 1 --2- 172.21.15.67:0/2595259322 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f940006ecc0 0x7f9400071180 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9418101230 tx=0x7f940c00b360 comp rx=0 tx=0).stop 2024-04-22T23:33:09.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.664+0000 7f94205d1700 1 -- 172.21.15.67:0/2595259322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9418101de0 msgr2=0x7f94180ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:09.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.664+0000 7f94205d1700 1 --2- 172.21.15.67:0/2595259322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9418101de0 0x7f94180ffa40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9408000f80 tx=0x7f9408003c30 comp rx=0 tx=0).stop 2024-04-22T23:33:09.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.664+0000 7f94205d1700 1 -- 172.21.15.67:0/2595259322 shutdown_connections 2024-04-22T23:33:09.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.664+0000 7f94205d1700 1 --2- 172.21.15.67:0/2595259322 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f940006ecc0 0x7f9400071180 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:09.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.664+0000 7f94205d1700 1 --2- 172.21.15.67:0/2595259322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9418101de0 0x7f94180ffa40 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:09.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.664+0000 7f94205d1700 1 --2- 172.21.15.67:0/2595259322 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9418102700 0x7f94180fff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:09.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.664+0000 7f94205d1700 1 -- 172.21.15.67:0/2595259322 >> 172.21.15.67:0/2595259322 conn(0x7f94180fb650 msgr2=0x7f9418105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:33:09.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.665+0000 7f94205d1700 1 -- 172.21.15.67:0/2595259322 shutdown_connections 2024-04-22T23:33:09.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.665+0000 7f94205d1700 1 -- 172.21.15.67:0/2595259322 wait complete. 2024-04-22T23:33:09.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.772+0000 7f9dc4b0f700 1 -- 172.21.15.67:0/928265292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dc00ffda0 msgr2=0x7f9dc0100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:09.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.772+0000 7f9dc4b0f700 1 --2- 172.21.15.67:0/928265292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dc00ffda0 0x7f9dc0100180 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9db4009910 tx=0x7f9db4009c20 comp rx=0 tx=0).stop 2024-04-22T23:33:09.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.772+0000 7f9dc4b0f700 1 -- 172.21.15.67:0/928265292 shutdown_connections 2024-04-22T23:33:09.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.772+0000 7f9dc4b0f700 1 --2- 172.21.15.67:0/928265292 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9dc0100750 0x7f9dc01047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:09.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.772+0000 7f9dc4b0f700 1 --2- 172.21.15.67:0/928265292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dc00ffda0 0x7f9dc0100180 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:09.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.772+0000 7f9dc4b0f700 1 -- 172.21.15.67:0/928265292 >> 172.21.15.67:0/928265292 conn(0x7f9dc00fb650 msgr2=0x7f9dc00fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:33:09.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.772+0000 7f9dc4b0f700 1 -- 172.21.15.67:0/928265292 shutdown_connections 2024-04-22T23:33:09.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.772+0000 7f9dc4b0f700 1 -- 172.21.15.67:0/928265292 wait complete. 2024-04-22T23:33:09.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.773+0000 7f9dc4b0f700 1 Processor -- start 2024-04-22T23:33:09.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.773+0000 7f9dc4b0f700 1 -- start start 2024-04-22T23:33:09.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.773+0000 7f9dc4b0f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dc00ffda0 0x7f9dc01976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:09.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.773+0000 7f9dc4b0f700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9dc0100750 0x7f9dc0197c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:09.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.773+0000 7f9dc4b0f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc0198230 con 0x7f9dc00ffda0 2024-04-22T23:33:09.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.773+0000 7f9dc4b0f700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc01983a0 con 0x7f9dc0100750 2024-04-22T23:33:09.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.773+0000 7f9dbe59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dc00ffda0 0x7f9dc01976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:09.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.773+0000 7f9dbe59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dc00ffda0 0x7f9dc01976d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44294/0 (socket says 172.21.15.67:44294) 2024-04-22T23:33:09.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.773+0000 7f9dbe59c700 1 -- 172.21.15.67:0/1572963185 learned_addr learned my addr 172.21.15.67:0/1572963185 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:33:09.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.773+0000 7f9dbdd9b700 1 --2- 172.21.15.67:0/1572963185 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9dc0100750 0x7f9dc0197c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:09.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.773+0000 7f9dbe59c700 1 -- 172.21.15.67:0/1572963185 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9dc0100750 msgr2=0x7f9dc0197c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:09.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.773+0000 7f9dbe59c700 1 --2- 172.21.15.67:0/1572963185 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9dc0100750 0x7f9dc0197c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:09.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.773+0000 7f9dbe59c700 1 -- 172.21.15.67:0/1572963185 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db40095f0 con 0x7f9dc00ffda0 2024-04-22T23:33:09.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.774+0000 7f9dbe59c700 1 --2- 172.21.15.67:0/1572963185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dc00ffda0 0x7f9dc01976d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9db4006010 tx=0x7f9db400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:33:09.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.774+0000 7f9daf7fe700 1 -- 172.21.15.67:0/1572963185 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9db401d070 con 0x7f9dc00ffda0 2024-04-22T23:33:09.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.774+0000 7f9daf7fe700 1 -- 172.21.15.67:0/1572963185 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9db4005dc0 con 0x7f9dc00ffda0 2024-04-22T23:33:09.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.774+0000 7f9daf7fe700 1 -- 172.21.15.67:0/1572963185 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9db4017400 con 0x7f9dc00ffda0 2024-04-22T23:33:09.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.774+0000 7f9dc4b0f700 1 -- 172.21.15.67:0/1572963185 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dc01a0740 con 0x7f9dc00ffda0 2024-04-22T23:33:09.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.774+0000 7f9dc4b0f700 1 -- 172.21.15.67:0/1572963185 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dc01a0bb0 con 0x7f9dc00ffda0 2024-04-22T23:33:09.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.775+0000 7f9dc4b0f700 1 -- 172.21.15.67:0/1572963185 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dc0108140 con 0x7f9dc00ffda0 2024-04-22T23:33:09.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.775+0000 7f9daf7fe700 1 -- 172.21.15.67:0/1572963185 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9db4003a60 con 0x7f9dc00ffda0 2024-04-22T23:33:09.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.776+0000 7f9daf7fe700 1 --2- 172.21.15.67:0/1572963185 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9da806ecc0 0x7f9da8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:09.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.776+0000 7f9daf7fe700 1 -- 172.21.15.67:0/1572963185 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f9db4090370 con 0x7f9dc00ffda0 2024-04-22T23:33:09.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.776+0000 7f9dbdd9b700 1 --2- 172.21.15.67:0/1572963185 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9da806ecc0 0x7f9da8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:09.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.776+0000 7f9dbdd9b700 1 --2- 172.21.15.67:0/1572963185 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9da806ecc0 0x7f9da8071180 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9dc00696e0 tx=0x7f9db000b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:33:09.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:09.781+0000 7f9daf7fe700 1 -- 172.21.15.67:0/1572963185 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f9db4059030 con 0x7f9dc00ffda0 2024-04-22T23:33:09.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.001+0000 7f9dc4b0f700 1 -- 172.21.15.67:0/1572963185 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9dc002cce0 con 0x7f9da806ecc0 2024-04-22T23:33:10.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.012+0000 7f9daf7fe700 1 -- 172.21.15.67:0/1572963185 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f9dc002cce0 con 0x7f9da806ecc0 2024-04-22T23:33:10.008 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:33:10.008 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (6m) 8s ago 18m 25.5M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:33:10.009 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (18m) 8s ago 18m 9114k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:33:10.009 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (17m) 49s ago 17m 8828k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:33:10.009 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (4m) 8s ago 18m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:33:10.009 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (4m) 49s ago 17m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:33:10.009 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (6m) 8s ago 17m 79.3M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:33:10.009 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (33s) 8s ago 12m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:33:10.009 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (14s) 8s ago 12m 12.6M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:33:10.009 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (12m) 49s ago 12m 123M - 18.2.1 5be31c24972a b6a169586433 2024-04-22T23:33:10.009 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (12m) 49s ago 12m 67.9M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:33:10.009 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (8m) 8s ago 19m 511M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:33:10.009 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (7m) 49s ago 17m 452M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:33:10.009 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (5m) 8s ago 19m 55.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:33:10.009 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (4m) 49s ago 17m 44.1M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:33:10.009 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (7m) 8s ago 18m 15.4M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:33:10.010 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (7m) 49s ago 17m 16.2M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:33:10.010 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (3m) 8s ago 16m 99.2M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:33:10.010 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (3m) 8s ago 15m 127M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:33:10.010 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (2m) 8s ago 15m 79.9M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:33:10.010 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (2m) 49s ago 15m 97.7M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:33:10.010 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (87s) 49s ago 14m 82.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:33:10.010 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (53s) 49s ago 14m 12.0M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:33:10.010 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (7m) 8s ago 17m 73.9M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:33:10.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.014+0000 7f9dc4b0f700 1 -- 172.21.15.67:0/1572963185 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9da806ecc0 msgr2=0x7f9da8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:10.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.014+0000 7f9dc4b0f700 1 --2- 172.21.15.67:0/1572963185 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9da806ecc0 0x7f9da8071180 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9dc00696e0 tx=0x7f9db000b410 comp rx=0 tx=0).stop 2024-04-22T23:33:10.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.014+0000 7f9dc4b0f700 1 -- 172.21.15.67:0/1572963185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dc00ffda0 msgr2=0x7f9dc01976d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:10.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.014+0000 7f9dc4b0f700 1 --2- 172.21.15.67:0/1572963185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dc00ffda0 0x7f9dc01976d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9db4006010 tx=0x7f9db400bd70 comp rx=0 tx=0).stop 2024-04-22T23:33:10.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.015+0000 7f9dc4b0f700 1 -- 172.21.15.67:0/1572963185 shutdown_connections 2024-04-22T23:33:10.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.015+0000 7f9dc4b0f700 1 --2- 172.21.15.67:0/1572963185 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9da806ecc0 0x7f9da8071180 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:10.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.015+0000 7f9dc4b0f700 1 --2- 172.21.15.67:0/1572963185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dc00ffda0 0x7f9dc01976d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:10.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.015+0000 7f9dc4b0f700 1 --2- 172.21.15.67:0/1572963185 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9dc0100750 0x7f9dc0197c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:10.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.015+0000 7f9dc4b0f700 1 -- 172.21.15.67:0/1572963185 >> 172.21.15.67:0/1572963185 conn(0x7f9dc00fb650 msgr2=0x7f9dc00fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:33:10.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.015+0000 7f9dc4b0f700 1 -- 172.21.15.67:0/1572963185 shutdown_connections 2024-04-22T23:33:10.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.015+0000 7f9dc4b0f700 1 -- 172.21.15.67:0/1572963185 wait complete. 2024-04-22T23:33:10.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.121+0000 7f954fb09700 1 -- 172.21.15.67:0/778332853 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9548105bf0 msgr2=0x7f9548105fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:10.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.121+0000 7f954fb09700 1 --2- 172.21.15.67:0/778332853 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9548105bf0 0x7f9548105fd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9538009910 tx=0x7f9538009c20 comp rx=0 tx=0).stop 2024-04-22T23:33:10.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.121+0000 7f954fb09700 1 -- 172.21.15.67:0/778332853 shutdown_connections 2024-04-22T23:33:10.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.121+0000 7f954fb09700 1 --2- 172.21.15.67:0/778332853 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f95480ffb40 0x7f95480fffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:10.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.121+0000 7f954fb09700 1 --2- 172.21.15.67:0/778332853 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9548105bf0 0x7f9548105fd0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:10.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.121+0000 7f954fb09700 1 -- 172.21.15.67:0/778332853 >> 172.21.15.67:0/778332853 conn(0x7f95480fb650 msgr2=0x7f95480fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:33:10.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.121+0000 7f954fb09700 1 -- 172.21.15.67:0/778332853 shutdown_connections 2024-04-22T23:33:10.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.121+0000 7f954fb09700 1 -- 172.21.15.67:0/778332853 wait complete. 2024-04-22T23:33:10.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.122+0000 7f954fb09700 1 Processor -- start 2024-04-22T23:33:10.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.122+0000 7f954fb09700 1 -- start start 2024-04-22T23:33:10.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.122+0000 7f954fb09700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95480ffb40 0x7f95481017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:10.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.122+0000 7f954fb09700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9548105bf0 0x7f9548101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:10.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.122+0000 7f954fb09700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9548102230 con 0x7f95480ffb40 2024-04-22T23:33:10.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.122+0000 7f954fb09700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9548102370 con 0x7f9548105bf0 2024-04-22T23:33:10.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.122+0000 7f954d8a5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95480ffb40 0x7f95481017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:10.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.122+0000 7f954d8a5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95480ffb40 0x7f95481017b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44312/0 (socket says 172.21.15.67:44312) 2024-04-22T23:33:10.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.122+0000 7f954d8a5700 1 -- 172.21.15.67:0/1406244860 learned_addr learned my addr 172.21.15.67:0/1406244860 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:33:10.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.122+0000 7f954d0a4700 1 --2- 172.21.15.67:0/1406244860 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9548105bf0 0x7f9548101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:10.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.122+0000 7f954d8a5700 1 -- 172.21.15.67:0/1406244860 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9548105bf0 msgr2=0x7f9548101cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:10.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.122+0000 7f954d8a5700 1 --2- 172.21.15.67:0/1406244860 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9548105bf0 0x7f9548101cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:10.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.123+0000 7f954d8a5700 1 -- 172.21.15.67:0/1406244860 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95380095f0 con 0x7f95480ffb40 2024-04-22T23:33:10.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.123+0000 7f954d8a5700 1 --2- 172.21.15.67:0/1406244860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95480ffb40 0x7f95481017b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9538006010 tx=0x7f953800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:33:10.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.123+0000 7f953effd700 1 -- 172.21.15.67:0/1406244860 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f953801d070 con 0x7f95480ffb40 2024-04-22T23:33:10.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.123+0000 7f953effd700 1 -- 172.21.15.67:0/1406244860 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9538005dc0 con 0x7f95480ffb40 2024-04-22T23:33:10.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.123+0000 7f954fb09700 1 -- 172.21.15.67:0/1406244860 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95481025f0 con 0x7f95480ffb40 2024-04-22T23:33:10.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.123+0000 7f953effd700 1 -- 172.21.15.67:0/1406244860 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9538017400 con 0x7f95480ffb40 2024-04-22T23:33:10.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.123+0000 7f954fb09700 1 -- 172.21.15.67:0/1406244860 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95481a2cd0 con 0x7f95480ffb40 2024-04-22T23:33:10.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.124+0000 7f954fb09700 1 -- 172.21.15.67:0/1406244860 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f954804f2e0 con 0x7f95480ffb40 2024-04-22T23:33:10.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.125+0000 7f953effd700 1 -- 172.21.15.67:0/1406244860 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9538003a60 con 0x7f95480ffb40 2024-04-22T23:33:10.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.125+0000 7f953effd700 1 --2- 172.21.15.67:0/1406244860 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f953406ecc0 0x7f9534071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:10.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.125+0000 7f954d0a4700 1 --2- 172.21.15.67:0/1406244860 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f953406ecc0 0x7f9534071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:10.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.125+0000 7f953effd700 1 -- 172.21.15.67:0/1406244860 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f9538090450 con 0x7f95480ffb40 2024-04-22T23:33:10.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.126+0000 7f954d0a4700 1 --2- 172.21.15.67:0/1406244860 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f953406ecc0 0x7f9534071180 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9548102fa0 tx=0x7f954400b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:33:10.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.130+0000 7f953effd700 1 -- 172.21.15.67:0/1406244860 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f9538059090 con 0x7f95480ffb40 2024-04-22T23:33:10.289 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:10 smithi067 ceph-mon[112438]: from='client.34428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:33:10.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:10 smithi067 ceph-mon[112438]: from='client.34432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:33:10.297 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:10 smithi134 ceph-mon[85630]: from='client.34428 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:33:10.297 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:10 smithi134 ceph-mon[85630]: from='client.34432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:33:10.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.429+0000 7f954fb09700 1 -- 172.21.15.67:0/1406244860 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9548102ce0 con 0x7f95480ffb40 2024-04-22T23:33:10.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.430+0000 7f953effd700 1 -- 172.21.15.67:0/1406244860 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f953805c750 con 0x7f95480ffb40 2024-04-22T23:33:10.426 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:33:10.426 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:33:10.426 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:33:10.426 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:33:10.426 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:33:10.426 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:33:10.426 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:33:10.426 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:33:10.427 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:33:10.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:33:10.427 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:33:10.427 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2, 2024-04-22T23:33:10.427 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:33:10.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:33:10.427 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:33:10.427 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2, 2024-04-22T23:33:10.427 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 12 2024-04-22T23:33:10.427 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:33:10.427 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:33:10.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.432+0000 7f954fb09700 1 -- 172.21.15.67:0/1406244860 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f953406ecc0 msgr2=0x7f9534071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:10.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.432+0000 7f954fb09700 1 --2- 172.21.15.67:0/1406244860 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f953406ecc0 0x7f9534071180 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9548102fa0 tx=0x7f954400b410 comp rx=0 tx=0).stop 2024-04-22T23:33:10.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.432+0000 7f954fb09700 1 -- 172.21.15.67:0/1406244860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95480ffb40 msgr2=0x7f95481017b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:10.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.432+0000 7f954fb09700 1 --2- 172.21.15.67:0/1406244860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95480ffb40 0x7f95481017b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9538006010 tx=0x7f953800bd70 comp rx=0 tx=0).stop 2024-04-22T23:33:10.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.432+0000 7f954fb09700 1 -- 172.21.15.67:0/1406244860 shutdown_connections 2024-04-22T23:33:10.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.432+0000 7f954fb09700 1 --2- 172.21.15.67:0/1406244860 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f953406ecc0 0x7f9534071180 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:10.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.432+0000 7f954fb09700 1 --2- 172.21.15.67:0/1406244860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95480ffb40 0x7f95481017b0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:10.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.432+0000 7f954fb09700 1 --2- 172.21.15.67:0/1406244860 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9548105bf0 0x7f9548101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:10.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.432+0000 7f954fb09700 1 -- 172.21.15.67:0/1406244860 >> 172.21.15.67:0/1406244860 conn(0x7f95480fb650 msgr2=0x7f9548068e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:33:10.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.432+0000 7f954fb09700 1 -- 172.21.15.67:0/1406244860 shutdown_connections 2024-04-22T23:33:10.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.432+0000 7f954fb09700 1 -- 172.21.15.67:0/1406244860 wait complete. 2024-04-22T23:33:10.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.539+0000 7fc244906700 1 -- 172.21.15.67:0/144269748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc240100750 msgr2=0x7fc2401047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:10.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.539+0000 7fc244906700 1 --2- 172.21.15.67:0/144269748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc240100750 0x7fc2401047a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc234009910 tx=0x7fc234009c20 comp rx=0 tx=0).stop 2024-04-22T23:33:10.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.539+0000 7fc244906700 1 -- 172.21.15.67:0/144269748 shutdown_connections 2024-04-22T23:33:10.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.539+0000 7fc244906700 1 --2- 172.21.15.67:0/144269748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc240100750 0x7fc2401047a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:10.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.539+0000 7fc244906700 1 --2- 172.21.15.67:0/144269748 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2400ffda0 0x7fc240100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:10.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.539+0000 7fc244906700 1 -- 172.21.15.67:0/144269748 >> 172.21.15.67:0/144269748 conn(0x7fc2400fb650 msgr2=0x7fc2400fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:33:10.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.539+0000 7fc244906700 1 -- 172.21.15.67:0/144269748 shutdown_connections 2024-04-22T23:33:10.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.539+0000 7fc244906700 1 -- 172.21.15.67:0/144269748 wait complete. 2024-04-22T23:33:10.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.540+0000 7fc244906700 1 Processor -- start 2024-04-22T23:33:10.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.540+0000 7fc244906700 1 -- start start 2024-04-22T23:33:10.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.540+0000 7fc244906700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2400ffda0 0x7fc24019c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:10.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.540+0000 7fc244906700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc240100750 0x7fc24019cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:10.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.540+0000 7fc244906700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc24019d240 con 0x7fc2400ffda0 2024-04-22T23:33:10.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.540+0000 7fc244906700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2401966f0 con 0x7fc240100750 2024-04-22T23:33:10.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.540+0000 7fc23e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2400ffda0 0x7fc24019c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:10.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.540+0000 7fc23e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2400ffda0 0x7fc24019c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44322/0 (socket says 172.21.15.67:44322) 2024-04-22T23:33:10.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.540+0000 7fc23e59c700 1 -- 172.21.15.67:0/3490452289 learned_addr learned my addr 172.21.15.67:0/3490452289 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:33:10.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.540+0000 7fc23dd9b700 1 --2- 172.21.15.67:0/3490452289 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc240100750 0x7fc24019cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:10.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.540+0000 7fc23e59c700 1 -- 172.21.15.67:0/3490452289 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc240100750 msgr2=0x7fc24019cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:10.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.540+0000 7fc23e59c700 1 --2- 172.21.15.67:0/3490452289 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc240100750 0x7fc24019cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:10.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.540+0000 7fc23e59c700 1 -- 172.21.15.67:0/3490452289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2340095f0 con 0x7fc2400ffda0 2024-04-22T23:33:10.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.540+0000 7fc23dd9b700 1 --2- 172.21.15.67:0/3490452289 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc240100750 0x7fc24019cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:33:10.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.541+0000 7fc23e59c700 1 --2- 172.21.15.67:0/3490452289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2400ffda0 0x7fc24019c670 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc23000d8d0 tx=0x7fc23000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:33:10.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.541+0000 7fc22f7fe700 1 -- 172.21.15.67:0/3490452289 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc230009940 con 0x7fc2400ffda0 2024-04-22T23:33:10.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.541+0000 7fc22f7fe700 1 -- 172.21.15.67:0/3490452289 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc230010460 con 0x7fc2400ffda0 2024-04-22T23:33:10.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.541+0000 7fc244906700 1 -- 172.21.15.67:0/3490452289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2401969d0 con 0x7fc2400ffda0 2024-04-22T23:33:10.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.541+0000 7fc22f7fe700 1 -- 172.21.15.67:0/3490452289 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc23000f5f0 con 0x7fc2400ffda0 2024-04-22T23:33:10.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.541+0000 7fc244906700 1 -- 172.21.15.67:0/3490452289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc240196f20 con 0x7fc2400ffda0 2024-04-22T23:33:10.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.542+0000 7fc244906700 1 -- 172.21.15.67:0/3490452289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc24004f2e0 con 0x7fc2400ffda0 2024-04-22T23:33:10.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.542+0000 7fc22f7fe700 1 -- 172.21.15.67:0/3490452289 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc2300105d0 con 0x7fc2400ffda0 2024-04-22T23:33:10.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.543+0000 7fc22f7fe700 1 --2- 172.21.15.67:0/3490452289 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc22806ecc0 0x7fc228071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:10.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.543+0000 7fc22f7fe700 1 -- 172.21.15.67:0/3490452289 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fc23008eee0 con 0x7fc2400ffda0 2024-04-22T23:33:10.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.543+0000 7fc23dd9b700 1 --2- 172.21.15.67:0/3490452289 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc22806ecc0 0x7fc228071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:10.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.543+0000 7fc23dd9b700 1 --2- 172.21.15.67:0/3490452289 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc22806ecc0 0x7fc228071180 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc240197b60 tx=0x7fc23401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:33:10.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.548+0000 7fc22f7fe700 1 -- 172.21.15.67:0/3490452289 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc23005b6d0 con 0x7fc2400ffda0 2024-04-22T23:33:10.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.807+0000 7fc244906700 1 -- 172.21.15.67:0/3490452289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc2401971c0 con 0x7fc2400ffda0 2024-04-22T23:33:10.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.808+0000 7fc22f7fe700 1 -- 172.21.15.67:0/3490452289 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) v1 ==== 76+0+1870 (secure 0 0 0) 0x7fc230016760 con 0x7fc2400ffda0 2024-04-22T23:33:10.803 INFO:teuthology.orchestra.run.smithi067.stdout:e29 2024-04-22T23:33:10.804 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:33:10.804 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:33:10.804 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:33:10.804 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:33:10.804 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:33:10.804 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:33:10.804 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:33:10.804 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:33:10.804 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:33:10.804 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:33:10.804 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:33:10.804 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:33:10.804 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:33:10.804 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:33:10.805 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:33:10.805 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:33:10.805 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:33:10.805 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:33:10.805 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:33:10.805 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:33:10.805 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:33:10.805 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:33:10.805 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:33:10.805 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:33:10.805 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:33:10.805 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:33:10.805 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:33:10.805 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:33:10.806 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:33:10.806 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:33:10.806 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:33:10.806 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:33:10.806 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:33:10.806 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:33:10.806 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:33:10.806 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:33:10.806 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:33:10.806 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:33:10.806 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:33:10.806 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44259} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1827288036,v1:172.21.15.134:6827/1827288036] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:33:10.806 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44267} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/3366499307,v1:172.21.15.134:6825/3366499307] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:33:10.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.810+0000 7fc244906700 1 -- 172.21.15.67:0/3490452289 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc22806ecc0 msgr2=0x7fc228071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:10.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.810+0000 7fc244906700 1 --2- 172.21.15.67:0/3490452289 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc22806ecc0 0x7fc228071180 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc240197b60 tx=0x7fc23401a040 comp rx=0 tx=0).stop 2024-04-22T23:33:10.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.810+0000 7fc244906700 1 -- 172.21.15.67:0/3490452289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2400ffda0 msgr2=0x7fc24019c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:10.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.810+0000 7fc244906700 1 --2- 172.21.15.67:0/3490452289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2400ffda0 0x7fc24019c670 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc23000d8d0 tx=0x7fc23000dc90 comp rx=0 tx=0).stop 2024-04-22T23:33:10.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.810+0000 7fc244906700 1 -- 172.21.15.67:0/3490452289 shutdown_connections 2024-04-22T23:33:10.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.810+0000 7fc244906700 1 --2- 172.21.15.67:0/3490452289 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc22806ecc0 0x7fc228071180 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:10.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.810+0000 7fc244906700 1 --2- 172.21.15.67:0/3490452289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2400ffda0 0x7fc24019c670 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:10.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.810+0000 7fc244906700 1 --2- 172.21.15.67:0/3490452289 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc240100750 0x7fc24019cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:10.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.810+0000 7fc244906700 1 -- 172.21.15.67:0/3490452289 >> 172.21.15.67:0/3490452289 conn(0x7fc2400fb650 msgr2=0x7fc240103f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:33:10.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.810+0000 7fc244906700 1 -- 172.21.15.67:0/3490452289 shutdown_connections 2024-04-22T23:33:10.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.810+0000 7fc244906700 1 -- 172.21.15.67:0/3490452289 wait complete. 2024-04-22T23:33:10.811 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 29 2024-04-22T23:33:10.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.919+0000 7ff791c34700 1 -- 172.21.15.67:0/950182285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff78c0ffda0 msgr2=0x7ff78c100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:10.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.919+0000 7ff791c34700 1 --2- 172.21.15.67:0/950182285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff78c0ffda0 0x7ff78c100180 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff77c009910 tx=0x7ff77c009c20 comp rx=0 tx=0).stop 2024-04-22T23:33:10.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.919+0000 7ff791c34700 1 -- 172.21.15.67:0/950182285 shutdown_connections 2024-04-22T23:33:10.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.919+0000 7ff791c34700 1 --2- 172.21.15.67:0/950182285 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff78c100750 0x7ff78c1047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:10.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.919+0000 7ff791c34700 1 --2- 172.21.15.67:0/950182285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff78c0ffda0 0x7ff78c100180 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:10.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.919+0000 7ff791c34700 1 -- 172.21.15.67:0/950182285 >> 172.21.15.67:0/950182285 conn(0x7ff78c0fb650 msgr2=0x7ff78c0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:33:10.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.919+0000 7ff791c34700 1 -- 172.21.15.67:0/950182285 shutdown_connections 2024-04-22T23:33:10.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.919+0000 7ff791c34700 1 -- 172.21.15.67:0/950182285 wait complete. 2024-04-22T23:33:10.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.920+0000 7ff791c34700 1 Processor -- start 2024-04-22T23:33:10.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.920+0000 7ff791c34700 1 -- start start 2024-04-22T23:33:10.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.920+0000 7ff791c34700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff78c0ffda0 0x7ff78c1976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:10.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.920+0000 7ff791c34700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff78c100750 0x7ff78c197c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:10.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.920+0000 7ff791c34700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78c198250 con 0x7ff78c100750 2024-04-22T23:33:10.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.920+0000 7ff791c34700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78c198390 con 0x7ff78c0ffda0 2024-04-22T23:33:10.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.920+0000 7ff78affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff78c100750 0x7ff78c197c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:10.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.920+0000 7ff78affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff78c100750 0x7ff78c197c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44328/0 (socket says 172.21.15.67:44328) 2024-04-22T23:33:10.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.920+0000 7ff78b7fe700 1 --2- 172.21.15.67:0/2625357393 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff78c0ffda0 0x7ff78c1976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:10.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.920+0000 7ff78affd700 1 -- 172.21.15.67:0/2625357393 learned_addr learned my addr 172.21.15.67:0/2625357393 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:33:10.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.920+0000 7ff78affd700 1 -- 172.21.15.67:0/2625357393 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff78c0ffda0 msgr2=0x7ff78c1976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:10.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.920+0000 7ff78affd700 1 --2- 172.21.15.67:0/2625357393 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff78c0ffda0 0x7ff78c1976f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:10.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.921+0000 7ff78affd700 1 -- 172.21.15.67:0/2625357393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff77c0095f0 con 0x7ff78c100750 2024-04-22T23:33:10.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.921+0000 7ff78affd700 1 --2- 172.21.15.67:0/2625357393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff78c100750 0x7ff78c197c30 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff78000d900 tx=0x7ff78000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:33:10.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.921+0000 7ff788ff9700 1 -- 172.21.15.67:0/2625357393 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff780009900 con 0x7ff78c100750 2024-04-22T23:33:10.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.921+0000 7ff788ff9700 1 -- 172.21.15.67:0/2625357393 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff780010460 con 0x7ff78c100750 2024-04-22T23:33:10.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.921+0000 7ff788ff9700 1 -- 172.21.15.67:0/2625357393 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff78000f5f0 con 0x7ff78c100750 2024-04-22T23:33:10.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.921+0000 7ff791c34700 1 -- 172.21.15.67:0/2625357393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff78c1a0760 con 0x7ff78c100750 2024-04-22T23:33:10.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.921+0000 7ff791c34700 1 -- 172.21.15.67:0/2625357393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff78c1a0c20 con 0x7ff78c100750 2024-04-22T23:33:10.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.922+0000 7ff791c34700 1 -- 172.21.15.67:0/2625357393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff78c108140 con 0x7ff78c100750 2024-04-22T23:33:10.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.923+0000 7ff788ff9700 1 -- 172.21.15.67:0/2625357393 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff7800105d0 con 0x7ff78c100750 2024-04-22T23:33:10.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.923+0000 7ff788ff9700 1 --2- 172.21.15.67:0/2625357393 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff77406ecc0 0x7ff774071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:10.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.923+0000 7ff788ff9700 1 -- 172.21.15.67:0/2625357393 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6876+0+0 (secure 0 0 0) 0x7ff78008ea80 con 0x7ff78c100750 2024-04-22T23:33:10.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.923+0000 7ff78b7fe700 1 --2- 172.21.15.67:0/2625357393 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff77406ecc0 0x7ff774071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:10.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.924+0000 7ff78b7fe700 1 --2- 172.21.15.67:0/2625357393 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff77406ecc0 0x7ff774071180 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff77c006010 tx=0x7ff77c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:33:10.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:10.928+0000 7ff788ff9700 1 -- 172.21.15.67:0/2625357393 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff780057740 con 0x7ff78c100750 2024-04-22T23:33:11.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.165+0000 7ff791c34700 1 -- 172.21.15.67:0/2625357393 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff78c0611d0 con 0x7ff77406ecc0 2024-04-22T23:33:11.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.166+0000 7ff788ff9700 1 -- 172.21.15.67:0/2625357393 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff78c0611d0 con 0x7ff77406ecc0 2024-04-22T23:33:11.162 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:33:11.162 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:33:11.162 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:33:11.162 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:33:11.162 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:33:11.162 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:33:11.162 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:33:11.162 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:33:11.162 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:33:11.162 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:33:11.162 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "14/23 daemons upgraded", 2024-04-22T23:33:11.162 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:33:11.162 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:33:11.163 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:33:11.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.168+0000 7ff791c34700 1 -- 172.21.15.67:0/2625357393 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff77406ecc0 msgr2=0x7ff774071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:11.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.168+0000 7ff791c34700 1 --2- 172.21.15.67:0/2625357393 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff77406ecc0 0x7ff774071180 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff77c006010 tx=0x7ff77c01a040 comp rx=0 tx=0).stop 2024-04-22T23:33:11.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.168+0000 7ff791c34700 1 -- 172.21.15.67:0/2625357393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff78c100750 msgr2=0x7ff78c197c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:11.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.168+0000 7ff791c34700 1 --2- 172.21.15.67:0/2625357393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff78c100750 0x7ff78c197c30 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff78000d900 tx=0x7ff78000dc10 comp rx=0 tx=0).stop 2024-04-22T23:33:11.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.168+0000 7ff791c34700 1 -- 172.21.15.67:0/2625357393 shutdown_connections 2024-04-22T23:33:11.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.168+0000 7ff791c34700 1 --2- 172.21.15.67:0/2625357393 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff77406ecc0 0x7ff774071180 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:11.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.168+0000 7ff791c34700 1 --2- 172.21.15.67:0/2625357393 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff78c0ffda0 0x7ff78c1976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:11.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.168+0000 7ff791c34700 1 --2- 172.21.15.67:0/2625357393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff78c100750 0x7ff78c197c30 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:11.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.168+0000 7ff791c34700 1 -- 172.21.15.67:0/2625357393 >> 172.21.15.67:0/2625357393 conn(0x7ff78c0fb650 msgr2=0x7ff78c0fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:33:11.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.168+0000 7ff791c34700 1 -- 172.21.15.67:0/2625357393 shutdown_connections 2024-04-22T23:33:11.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.168+0000 7ff791c34700 1 -- 172.21.15.67:0/2625357393 wait complete. 2024-04-22T23:33:11.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.275+0000 7fdf6cc7c700 1 -- 172.21.15.67:0/1766685380 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf68105350 msgr2=0x7fdf68105730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:11.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.275+0000 7fdf6cc7c700 1 --2- 172.21.15.67:0/1766685380 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf68105350 0x7fdf68105730 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c009910 tx=0x7fdf5c009c20 comp rx=0 tx=0).stop 2024-04-22T23:33:11.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.275+0000 7fdf6cc7c700 1 -- 172.21.15.67:0/1766685380 shutdown_connections 2024-04-22T23:33:11.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.275+0000 7fdf6cc7c700 1 --2- 172.21.15.67:0/1766685380 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf680ff340 0x7fdf680ff7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:11.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.275+0000 7fdf6cc7c700 1 --2- 172.21.15.67:0/1766685380 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf68105350 0x7fdf68105730 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:11.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.275+0000 7fdf6cc7c700 1 -- 172.21.15.67:0/1766685380 >> 172.21.15.67:0/1766685380 conn(0x7fdf68074e30 msgr2=0x7fdf68075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:33:11.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.276+0000 7fdf6cc7c700 1 -- 172.21.15.67:0/1766685380 shutdown_connections 2024-04-22T23:33:11.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.276+0000 7fdf6cc7c700 1 -- 172.21.15.67:0/1766685380 wait complete. 2024-04-22T23:33:11.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.276+0000 7fdf6cc7c700 1 Processor -- start 2024-04-22T23:33:11.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.276+0000 7fdf6cc7c700 1 -- start start 2024-04-22T23:33:11.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.276+0000 7fdf6cc7c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf680ff340 0x7fdf680fc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:11.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.277+0000 7fdf6cc7c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf68105350 0x7fdf680fc930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:11.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.277+0000 7fdf6cc7c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf680fce70 con 0x7fdf680ff340 2024-04-22T23:33:11.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.277+0000 7fdf6cc7c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf680fcfe0 con 0x7fdf68105350 2024-04-22T23:33:11.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.277+0000 7fdf6659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf680ff340 0x7fdf680fc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:11.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.277+0000 7fdf6659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf680ff340 0x7fdf680fc3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44340/0 (socket says 172.21.15.67:44340) 2024-04-22T23:33:11.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.277+0000 7fdf6659c700 1 -- 172.21.15.67:0/1085206696 learned_addr learned my addr 172.21.15.67:0/1085206696 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:33:11.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.277+0000 7fdf65d9b700 1 --2- 172.21.15.67:0/1085206696 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf68105350 0x7fdf680fc930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:11.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.277+0000 7fdf6659c700 1 -- 172.21.15.67:0/1085206696 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf68105350 msgr2=0x7fdf680fc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:11.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.277+0000 7fdf6659c700 1 --2- 172.21.15.67:0/1085206696 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf68105350 0x7fdf680fc930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:11.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.277+0000 7fdf6659c700 1 -- 172.21.15.67:0/1085206696 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf5c0095f0 con 0x7fdf680ff340 2024-04-22T23:33:11.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.277+0000 7fdf65d9b700 1 --2- 172.21.15.67:0/1085206696 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf68105350 0x7fdf680fc930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:33:11.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.277+0000 7fdf6659c700 1 --2- 172.21.15.67:0/1085206696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf680ff340 0x7fdf680fc3f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c006010 tx=0x7fdf5c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:33:11.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.277+0000 7fdf577fe700 1 -- 172.21.15.67:0/1085206696 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf5c01d070 con 0x7fdf680ff340 2024-04-22T23:33:11.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.277+0000 7fdf6cc7c700 1 -- 172.21.15.67:0/1085206696 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf680fd200 con 0x7fdf680ff340 2024-04-22T23:33:11.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.277+0000 7fdf577fe700 1 -- 172.21.15.67:0/1085206696 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdf5c004fb0 con 0x7fdf680ff340 2024-04-22T23:33:11.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.277+0000 7fdf577fe700 1 -- 172.21.15.67:0/1085206696 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf5c022ea0 con 0x7fdf680ff340 2024-04-22T23:33:11.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.277+0000 7fdf6cc7c700 1 -- 172.21.15.67:0/1085206696 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf681a2dd0 con 0x7fdf680ff340 2024-04-22T23:33:11.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.278+0000 7fdf6cc7c700 1 -- 172.21.15.67:0/1085206696 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf6804f2e0 con 0x7fdf680ff340 2024-04-22T23:33:11.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.279+0000 7fdf577fe700 1 -- 172.21.15.67:0/1085206696 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdf5c022470 con 0x7fdf680ff340 2024-04-22T23:33:11.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.279+0000 7fdf577fe700 1 --2- 172.21.15.67:0/1085206696 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdf5006ecc0 0x7fdf50071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:11.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.279+0000 7fdf577fe700 1 -- 172.21.15.67:0/1085206696 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fdf5c090570 con 0x7fdf680ff340 2024-04-22T23:33:11.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.279+0000 7fdf65d9b700 1 --2- 172.21.15.67:0/1085206696 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdf5006ecc0 0x7fdf50071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:11.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.280+0000 7fdf65d9b700 1 --2- 172.21.15.67:0/1085206696 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdf5006ecc0 0x7fdf50071180 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fdf58005fd0 tx=0x7fdf58005e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:33:11.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.284+0000 7fdf577fe700 1 -- 172.21.15.67:0/1085206696 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fdf5c059260 con 0x7fdf680ff340 2024-04-22T23:33:11.489 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:11 smithi134 ceph-mon[85630]: from='client.34436 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:33:11.489 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:11 smithi134 ceph-mon[85630]: pgmap v203: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:11.489 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:11 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1406244860' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:33:11.489 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:11 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3490452289' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:33:11.489 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:33:11.539 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:11 smithi067 ceph-mon[112438]: from='client.34436 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:33:11.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:11 smithi067 ceph-mon[112438]: pgmap v203: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:11.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:11 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1406244860' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:33:11.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:11 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3490452289' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:33:11.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:33:11.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.586+0000 7fdf6cc7c700 1 -- 172.21.15.67:0/1085206696 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fdf681a30b0 con 0x7fdf680ff340 2024-04-22T23:33:11.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.586+0000 7fdf577fe700 1 -- 172.21.15.67:0/1085206696 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fdf5c022720 con 0x7fdf680ff340 2024-04-22T23:33:11.582 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:33:11.583 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:33:11.583 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:33:11.583 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:33:11.583 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:33:11.583 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:33:11.583 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:33:11.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.588+0000 7fdf6cc7c700 1 -- 172.21.15.67:0/1085206696 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdf5006ecc0 msgr2=0x7fdf50071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:11.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.588+0000 7fdf6cc7c700 1 --2- 172.21.15.67:0/1085206696 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdf5006ecc0 0x7fdf50071180 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fdf58005fd0 tx=0x7fdf58005e30 comp rx=0 tx=0).stop 2024-04-22T23:33:11.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.588+0000 7fdf6cc7c700 1 -- 172.21.15.67:0/1085206696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf680ff340 msgr2=0x7fdf680fc3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:11.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.588+0000 7fdf6cc7c700 1 --2- 172.21.15.67:0/1085206696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf680ff340 0x7fdf680fc3f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c006010 tx=0x7fdf5c00be20 comp rx=0 tx=0).stop 2024-04-22T23:33:11.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.588+0000 7fdf6cc7c700 1 -- 172.21.15.67:0/1085206696 shutdown_connections 2024-04-22T23:33:11.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.588+0000 7fdf6cc7c700 1 --2- 172.21.15.67:0/1085206696 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdf5006ecc0 0x7fdf50071180 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:11.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.588+0000 7fdf6cc7c700 1 --2- 172.21.15.67:0/1085206696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf680ff340 0x7fdf680fc3f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:11.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.588+0000 7fdf6cc7c700 1 --2- 172.21.15.67:0/1085206696 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf68105350 0x7fdf680fc930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:11.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.588+0000 7fdf6cc7c700 1 -- 172.21.15.67:0/1085206696 >> 172.21.15.67:0/1085206696 conn(0x7fdf68074e30 msgr2=0x7fdf680febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:33:11.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.588+0000 7fdf6cc7c700 1 -- 172.21.15.67:0/1085206696 shutdown_connections 2024-04-22T23:33:11.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:11.588+0000 7fdf6cc7c700 1 -- 172.21.15.67:0/1085206696 wait complete. 2024-04-22T23:33:12.391 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:12 smithi134 ceph-mon[85630]: osdmap e93: 6 total, 6 up, 6 in 2024-04-22T23:33:12.391 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:12 smithi134 ceph-mon[85630]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-22T23:33:12.391 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:12 smithi134 ceph-mon[85630]: from='client.34448 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:33:12.391 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:12 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1085206696' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:33:12.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:12 smithi067 ceph-mon[112438]: osdmap e93: 6 total, 6 up, 6 in 2024-04-22T23:33:12.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:12 smithi067 ceph-mon[112438]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-22T23:33:12.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:12 smithi067 ceph-mon[112438]: from='client.34448 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:33:12.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:12 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1085206696' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:33:13.411 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:13 smithi134 ceph-mon[85630]: pgmap v205: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:13.411 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:13 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.owipad"}]: dispatch 2024-04-22T23:33:13.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:13 smithi067 ceph-mon[112438]: pgmap v205: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:13.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:13 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.owipad"}]: dispatch 2024-04-22T23:33:15.398 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:15 smithi134 ceph-mon[85630]: pgmap v206: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:15.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:15 smithi067 ceph-mon[112438]: pgmap v206: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:16.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:16 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:33:16.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:16 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:33:16.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:16 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:33:16.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:16 smithi134 ceph-mon[85630]: pgmap v207: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:16.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:16 smithi134 ceph-mon[85630]: mds.? [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] up:boot 2024-04-22T23:33:16.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:16 smithi134 ceph-mon[85630]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-22T23:33:16.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:16 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.owipad"}]: dispatch 2024-04-22T23:33:17.289 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:16 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:33:17.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:16 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:33:17.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:16 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:33:17.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:16 smithi067 ceph-mon[112438]: pgmap v207: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:17.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:16 smithi067 ceph-mon[112438]: mds.? [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] up:boot 2024-04-22T23:33:17.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:16 smithi067 ceph-mon[112438]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-22T23:33:17.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:16 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.owipad"}]: dispatch 2024-04-22T23:33:19.489 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:19 smithi134 ceph-mon[85630]: pgmap v208: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:19.489 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:19 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:33:19.489 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:19 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:33:19.539 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:19 smithi067 ceph-mon[112438]: pgmap v208: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:19.539 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:19 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:33:19.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:19 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:33:21.489 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:21 smithi134 ceph-mon[85630]: pgmap v209: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:21.489 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:21 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:33:21.489 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:21 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:33:21.539 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:21 smithi067 ceph-mon[112438]: pgmap v209: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:21.539 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:21 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:33:21.540 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:21 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:33:23.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:23 smithi134 ceph-mon[85630]: pgmap v210: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:23.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:23 smithi067 ceph-mon[112438]: pgmap v210: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:25.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:25 smithi134 ceph-mon[85630]: pgmap v211: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:25.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:25 smithi067 ceph-mon[112438]: pgmap v211: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:26.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:33:26.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:33:27.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:27 smithi134 ceph-mon[85630]: pgmap v212: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:27.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:27 smithi067 ceph-mon[112438]: pgmap v212: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:29.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:29 smithi134 ceph-mon[85630]: pgmap v213: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:29.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:29 smithi067 ceph-mon[112438]: pgmap v213: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:31.741 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:31 smithi134 ceph-mon[85630]: pgmap v214: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:31.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:31 smithi067 ceph-mon[112438]: pgmap v214: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:33.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:33 smithi134 ceph-mon[85630]: pgmap v215: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:33.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:33 smithi067 ceph-mon[112438]: pgmap v215: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:35.738 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:35 smithi134 ceph-mon[85630]: pgmap v216: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:35.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:35 smithi067 ceph-mon[112438]: pgmap v216: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:37.738 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:37 smithi134 ceph-mon[85630]: pgmap v217: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:37.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:37 smithi067 ceph-mon[112438]: pgmap v217: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:39.738 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:39 smithi134 ceph-mon[85630]: pgmap v218: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:39.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:39 smithi067 ceph-mon[112438]: pgmap v218: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:41.707 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:41 smithi067 ceph-mon[112438]: pgmap v219: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:41.707 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:33:41.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.711+0000 7f567a8f5700 1 -- 172.21.15.67:0/416178796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5674105bd0 msgr2=0x7f5674105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:41.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.711+0000 7f567a8f5700 1 --2- 172.21.15.67:0/416178796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5674105bd0 0x7f5674105fb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5664009910 tx=0x7f5664009c20 comp rx=0 tx=0).stop 2024-04-22T23:33:41.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.711+0000 7f567a8f5700 1 -- 172.21.15.67:0/416178796 shutdown_connections 2024-04-22T23:33:41.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.711+0000 7f567a8f5700 1 --2- 172.21.15.67:0/416178796 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f56740ffc20 0x7f5674100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:41.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.711+0000 7f567a8f5700 1 --2- 172.21.15.67:0/416178796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5674105bd0 0x7f5674105fb0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:41.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.711+0000 7f567a8f5700 1 -- 172.21.15.67:0/416178796 >> 172.21.15.67:0/416178796 conn(0x7f5674075670 msgr2=0x7f5674075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:33:41.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.711+0000 7f567a8f5700 1 -- 172.21.15.67:0/416178796 shutdown_connections 2024-04-22T23:33:41.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.711+0000 7f567a8f5700 1 -- 172.21.15.67:0/416178796 wait complete. 2024-04-22T23:33:41.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.712+0000 7f567a8f5700 1 Processor -- start 2024-04-22T23:33:41.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.712+0000 7f567a8f5700 1 -- start start 2024-04-22T23:33:41.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.712+0000 7f567a8f5700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f56740ffc20 0x7f567419e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:41.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.712+0000 7f567a8f5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5674105bd0 0x7f567419ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:41.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.712+0000 7f567a8f5700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f567419f3f0 con 0x7f5674105bd0 2024-04-22T23:33:41.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.712+0000 7f567a8f5700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56741988a0 con 0x7f56740ffc20 2024-04-22T23:33:41.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.713+0000 7f56737fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5674105bd0 0x7f567419ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:41.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.713+0000 7f56737fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5674105bd0 0x7f567419ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42806/0 (socket says 172.21.15.67:42806) 2024-04-22T23:33:41.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.713+0000 7f56737fe700 1 -- 172.21.15.67:0/3717531203 learned_addr learned my addr 172.21.15.67:0/3717531203 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:33:41.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.713+0000 7f5673fff700 1 --2- 172.21.15.67:0/3717531203 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f56740ffc20 0x7f567419e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:41.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.713+0000 7f56737fe700 1 -- 172.21.15.67:0/3717531203 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f56740ffc20 msgr2=0x7f567419e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:41.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.713+0000 7f56737fe700 1 --2- 172.21.15.67:0/3717531203 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f56740ffc20 0x7f567419e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:41.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.713+0000 7f56737fe700 1 -- 172.21.15.67:0/3717531203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56640095f0 con 0x7f5674105bd0 2024-04-22T23:33:41.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.713+0000 7f56737fe700 1 --2- 172.21.15.67:0/3717531203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5674105bd0 0x7f567419ed60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f566800d8d0 tx=0x7f566800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:33:41.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.713+0000 7f56717fa700 1 -- 172.21.15.67:0/3717531203 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56680098a0 con 0x7f5674105bd0 2024-04-22T23:33:41.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.713+0000 7f567a8f5700 1 -- 172.21.15.67:0/3717531203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5674198b80 con 0x7f5674105bd0 2024-04-22T23:33:41.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.713+0000 7f56717fa700 1 -- 172.21.15.67:0/3717531203 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5668010460 con 0x7f5674105bd0 2024-04-22T23:33:41.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.713+0000 7f56717fa700 1 -- 172.21.15.67:0/3717531203 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f566800f5f0 con 0x7f5674105bd0 2024-04-22T23:33:41.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.714+0000 7f567a8f5700 1 -- 172.21.15.67:0/3717531203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56741990d0 con 0x7f5674105bd0 2024-04-22T23:33:41.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.715+0000 7f567a8f5700 1 -- 172.21.15.67:0/3717531203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f567404f2e0 con 0x7f5674105bd0 2024-04-22T23:33:41.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.715+0000 7f56717fa700 1 -- 172.21.15.67:0/3717531203 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5668009a00 con 0x7f5674105bd0 2024-04-22T23:33:41.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.716+0000 7f56717fa700 1 --2- 172.21.15.67:0/3717531203 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f565c06ecc0 0x7f565c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:41.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.716+0000 7f56717fa700 1 -- 172.21.15.67:0/3717531203 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f566808eee0 con 0x7f5674105bd0 2024-04-22T23:33:41.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.716+0000 7f5673fff700 1 --2- 172.21.15.67:0/3717531203 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f565c06ecc0 0x7f565c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:41.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.716+0000 7f5673fff700 1 --2- 172.21.15.67:0/3717531203 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f565c06ecc0 0x7f565c071180 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5664006010 tx=0x7f566401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:33:41.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.721+0000 7f56717fa700 1 -- 172.21.15.67:0/3717531203 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f566805b650 con 0x7f5674105bd0 2024-04-22T23:33:41.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:41 smithi134 ceph-mon[85630]: pgmap v219: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:41.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:33:41.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.950+0000 7f567a8f5700 1 -- 172.21.15.67:0/3717531203 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5674199360 con 0x7f565c06ecc0 2024-04-22T23:33:41.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.951+0000 7f56717fa700 1 -- 172.21.15.67:0/3717531203 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f5674199360 con 0x7f565c06ecc0 2024-04-22T23:33:41.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.953+0000 7f567a8f5700 1 -- 172.21.15.67:0/3717531203 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f565c06ecc0 msgr2=0x7f565c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:41.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.953+0000 7f567a8f5700 1 --2- 172.21.15.67:0/3717531203 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f565c06ecc0 0x7f565c071180 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5664006010 tx=0x7f566401a040 comp rx=0 tx=0).stop 2024-04-22T23:33:41.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.953+0000 7f567a8f5700 1 -- 172.21.15.67:0/3717531203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5674105bd0 msgr2=0x7f567419ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:41.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.953+0000 7f567a8f5700 1 --2- 172.21.15.67:0/3717531203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5674105bd0 0x7f567419ed60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f566800d8d0 tx=0x7f566800dbe0 comp rx=0 tx=0).stop 2024-04-22T23:33:41.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.953+0000 7f567a8f5700 1 -- 172.21.15.67:0/3717531203 shutdown_connections 2024-04-22T23:33:41.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.953+0000 7f567a8f5700 1 --2- 172.21.15.67:0/3717531203 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f565c06ecc0 0x7f565c071180 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:41.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.953+0000 7f567a8f5700 1 --2- 172.21.15.67:0/3717531203 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f56740ffc20 0x7f567419e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:41.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.953+0000 7f567a8f5700 1 --2- 172.21.15.67:0/3717531203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5674105bd0 0x7f567419ed60 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:41.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.953+0000 7f567a8f5700 1 -- 172.21.15.67:0/3717531203 >> 172.21.15.67:0/3717531203 conn(0x7f5674075670 msgr2=0x7f5674068720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:33:41.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.953+0000 7f567a8f5700 1 -- 172.21.15.67:0/3717531203 shutdown_connections 2024-04-22T23:33:41.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:41.953+0000 7f567a8f5700 1 -- 172.21.15.67:0/3717531203 wait complete. 2024-04-22T23:33:41.960 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:33:42.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.063+0000 7fbb39561700 1 -- 172.21.15.67:0/2351216493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb340ffc20 msgr2=0x7fbb34100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:42.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.063+0000 7fbb39561700 1 --2- 172.21.15.67:0/2351216493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb340ffc20 0x7fbb34100080 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fbb28009910 tx=0x7fbb28009c20 comp rx=0 tx=0).stop 2024-04-22T23:33:42.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.063+0000 7fbb39561700 1 -- 172.21.15.67:0/2351216493 shutdown_connections 2024-04-22T23:33:42.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.063+0000 7fbb39561700 1 --2- 172.21.15.67:0/2351216493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb340ffc20 0x7fbb34100080 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:42.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.063+0000 7fbb39561700 1 --2- 172.21.15.67:0/2351216493 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb34105bd0 0x7fbb34105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:42.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.063+0000 7fbb39561700 1 -- 172.21.15.67:0/2351216493 >> 172.21.15.67:0/2351216493 conn(0x7fbb34075670 msgr2=0x7fbb34075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:33:42.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.063+0000 7fbb39561700 1 -- 172.21.15.67:0/2351216493 shutdown_connections 2024-04-22T23:33:42.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.063+0000 7fbb39561700 1 -- 172.21.15.67:0/2351216493 wait complete. 2024-04-22T23:33:42.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.064+0000 7fbb39561700 1 Processor -- start 2024-04-22T23:33:42.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.064+0000 7fbb39561700 1 -- start start 2024-04-22T23:33:42.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.064+0000 7fbb39561700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb340ffc20 0x7fbb34197760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:42.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.064+0000 7fbb39561700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb34105bd0 0x7fbb34197ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:42.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.064+0000 7fbb39561700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb341982f0 con 0x7fbb340ffc20 2024-04-22T23:33:42.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.064+0000 7fbb39561700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb34198460 con 0x7fbb34105bd0 2024-04-22T23:33:42.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.064+0000 7fbb32ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb340ffc20 0x7fbb34197760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:42.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.064+0000 7fbb32ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb340ffc20 0x7fbb34197760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42818/0 (socket says 172.21.15.67:42818) 2024-04-22T23:33:42.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.064+0000 7fbb32ffd700 1 -- 172.21.15.67:0/2550110774 learned_addr learned my addr 172.21.15.67:0/2550110774 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:33:42.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.065+0000 7fbb32ffd700 1 -- 172.21.15.67:0/2550110774 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb34105bd0 msgr2=0x7fbb34197ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:42.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.065+0000 7fbb32ffd700 1 --2- 172.21.15.67:0/2550110774 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb34105bd0 0x7fbb34197ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:42.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.065+0000 7fbb32ffd700 1 -- 172.21.15.67:0/2550110774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb280095f0 con 0x7fbb340ffc20 2024-04-22T23:33:42.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.065+0000 7fbb32ffd700 1 --2- 172.21.15.67:0/2550110774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb340ffc20 0x7fbb34197760 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbb2400c930 tx=0x7fbb2400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:33:42.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.065+0000 7fbb1bfff700 1 -- 172.21.15.67:0/2550110774 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb24007ad0 con 0x7fbb340ffc20 2024-04-22T23:33:42.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.065+0000 7fbb1bfff700 1 -- 172.21.15.67:0/2550110774 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbb24007c30 con 0x7fbb340ffc20 2024-04-22T23:33:42.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.065+0000 7fbb39561700 1 -- 172.21.15.67:0/2550110774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb341a07d0 con 0x7fbb340ffc20 2024-04-22T23:33:42.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.065+0000 7fbb1bfff700 1 -- 172.21.15.67:0/2550110774 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb24018610 con 0x7fbb340ffc20 2024-04-22T23:33:42.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.065+0000 7fbb39561700 1 -- 172.21.15.67:0/2550110774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb341a0c90 con 0x7fbb340ffc20 2024-04-22T23:33:42.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.066+0000 7fbb39561700 1 -- 172.21.15.67:0/2550110774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb341080d0 con 0x7fbb340ffc20 2024-04-22T23:33:42.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.067+0000 7fbb1bfff700 1 -- 172.21.15.67:0/2550110774 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbb2400f450 con 0x7fbb340ffc20 2024-04-22T23:33:42.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.067+0000 7fbb1bfff700 1 --2- 172.21.15.67:0/2550110774 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbb1c06ecc0 0x7fbb1c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:42.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.067+0000 7fbb1bfff700 1 -- 172.21.15.67:0/2550110774 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fbb2408f210 con 0x7fbb340ffc20 2024-04-22T23:33:42.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.067+0000 7fbb327fc700 1 --2- 172.21.15.67:0/2550110774 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbb1c06ecc0 0x7fbb1c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:42.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.067+0000 7fbb327fc700 1 --2- 172.21.15.67:0/2550110774 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbb1c06ecc0 0x7fbb1c071180 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbb2800b5c0 tx=0x7fbb28005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:33:42.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.072+0000 7fbb1bfff700 1 -- 172.21.15.67:0/2550110774 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fbb24057e50 con 0x7fbb340ffc20 2024-04-22T23:33:42.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.303+0000 7fbb39561700 1 -- 172.21.15.67:0/2550110774 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb340611d0 con 0x7fbb1c06ecc0 2024-04-22T23:33:42.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.304+0000 7fbb1bfff700 1 -- 172.21.15.67:0/2550110774 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fbb340611d0 con 0x7fbb1c06ecc0 2024-04-22T23:33:42.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.306+0000 7fbb39561700 1 -- 172.21.15.67:0/2550110774 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbb1c06ecc0 msgr2=0x7fbb1c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:42.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.306+0000 7fbb39561700 1 --2- 172.21.15.67:0/2550110774 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbb1c06ecc0 0x7fbb1c071180 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbb2800b5c0 tx=0x7fbb28005960 comp rx=0 tx=0).stop 2024-04-22T23:33:42.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.306+0000 7fbb39561700 1 -- 172.21.15.67:0/2550110774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb340ffc20 msgr2=0x7fbb34197760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:42.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.306+0000 7fbb39561700 1 --2- 172.21.15.67:0/2550110774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb340ffc20 0x7fbb34197760 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbb2400c930 tx=0x7fbb2400ccf0 comp rx=0 tx=0).stop 2024-04-22T23:33:42.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.306+0000 7fbb39561700 1 -- 172.21.15.67:0/2550110774 shutdown_connections 2024-04-22T23:33:42.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.306+0000 7fbb39561700 1 --2- 172.21.15.67:0/2550110774 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbb1c06ecc0 0x7fbb1c071180 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:42.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.306+0000 7fbb39561700 1 --2- 172.21.15.67:0/2550110774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb340ffc20 0x7fbb34197760 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:42.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.306+0000 7fbb39561700 1 --2- 172.21.15.67:0/2550110774 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb34105bd0 0x7fbb34197ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:42.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.306+0000 7fbb39561700 1 -- 172.21.15.67:0/2550110774 >> 172.21.15.67:0/2550110774 conn(0x7fbb34075670 msgr2=0x7fbb340fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:33:42.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.306+0000 7fbb39561700 1 -- 172.21.15.67:0/2550110774 shutdown_connections 2024-04-22T23:33:42.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.306+0000 7fbb39561700 1 -- 172.21.15.67:0/2550110774 wait complete. 2024-04-22T23:33:42.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.413+0000 7fe52b4d1700 1 -- 172.21.15.67:0/2850882583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe524101850 msgr2=0x7fe5241058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:42.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.413+0000 7fe52b4d1700 1 --2- 172.21.15.67:0/2850882583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe524101850 0x7fe5241058a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe514009910 tx=0x7fe514009c20 comp rx=0 tx=0).stop 2024-04-22T23:33:42.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.414+0000 7fe52b4d1700 1 -- 172.21.15.67:0/2850882583 shutdown_connections 2024-04-22T23:33:42.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.414+0000 7fe52b4d1700 1 --2- 172.21.15.67:0/2850882583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe524101850 0x7fe5241058a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:42.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.414+0000 7fe52b4d1700 1 --2- 172.21.15.67:0/2850882583 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe524100ea0 0x7fe524101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:42.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.414+0000 7fe52b4d1700 1 -- 172.21.15.67:0/2850882583 >> 172.21.15.67:0/2850882583 conn(0x7fe5240fc750 msgr2=0x7fe5240feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:33:42.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.414+0000 7fe52b4d1700 1 -- 172.21.15.67:0/2850882583 shutdown_connections 2024-04-22T23:33:42.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.414+0000 7fe52b4d1700 1 -- 172.21.15.67:0/2850882583 wait complete. 2024-04-22T23:33:42.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.414+0000 7fe52b4d1700 1 Processor -- start 2024-04-22T23:33:42.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.414+0000 7fe52b4d1700 1 -- start start 2024-04-22T23:33:42.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.415+0000 7fe52b4d1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe524100ea0 0x7fe52419c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:42.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.415+0000 7fe52b4d1700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe524101850 0x7fe52419cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:42.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.415+0000 7fe52b4d1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52419d240 con 0x7fe524100ea0 2024-04-22T23:33:42.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.415+0000 7fe52b4d1700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe524196780 con 0x7fe524101850 2024-04-22T23:33:42.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.415+0000 7fe52926d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe524100ea0 0x7fe52419c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:42.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.415+0000 7fe52926d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe524100ea0 0x7fe52419c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42838/0 (socket says 172.21.15.67:42838) 2024-04-22T23:33:42.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.415+0000 7fe52926d700 1 -- 172.21.15.67:0/716923951 learned_addr learned my addr 172.21.15.67:0/716923951 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:33:42.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.415+0000 7fe528a6c700 1 --2- 172.21.15.67:0/716923951 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe524101850 0x7fe52419cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:42.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.415+0000 7fe52926d700 1 -- 172.21.15.67:0/716923951 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe524101850 msgr2=0x7fe52419cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:42.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.415+0000 7fe52926d700 1 --2- 172.21.15.67:0/716923951 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe524101850 0x7fe52419cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:42.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.415+0000 7fe52926d700 1 -- 172.21.15.67:0/716923951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5140095f0 con 0x7fe524100ea0 2024-04-22T23:33:42.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.415+0000 7fe528a6c700 1 --2- 172.21.15.67:0/716923951 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe524101850 0x7fe52419cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:33:42.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.415+0000 7fe52926d700 1 --2- 172.21.15.67:0/716923951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe524100ea0 0x7fe52419c700 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe52000c930 tx=0x7fe52000ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:33:42.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.415+0000 7fe51a7fc700 1 -- 172.21.15.67:0/716923951 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe520007ad0 con 0x7fe524100ea0 2024-04-22T23:33:42.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.415+0000 7fe52b4d1700 1 -- 172.21.15.67:0/716923951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe524196a60 con 0x7fe524100ea0 2024-04-22T23:33:42.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.415+0000 7fe51a7fc700 1 -- 172.21.15.67:0/716923951 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe520007c30 con 0x7fe524100ea0 2024-04-22T23:33:42.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.415+0000 7fe51a7fc700 1 -- 172.21.15.67:0/716923951 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe520018610 con 0x7fe524100ea0 2024-04-22T23:33:42.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.415+0000 7fe52b4d1700 1 -- 172.21.15.67:0/716923951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe524196fb0 con 0x7fe524100ea0 2024-04-22T23:33:42.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.416+0000 7fe52b4d1700 1 -- 172.21.15.67:0/716923951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5241092a0 con 0x7fe524100ea0 2024-04-22T23:33:42.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.417+0000 7fe51a7fc700 1 -- 172.21.15.67:0/716923951 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe52000f450 con 0x7fe524100ea0 2024-04-22T23:33:42.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.417+0000 7fe51a7fc700 1 --2- 172.21.15.67:0/716923951 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe51006ecc0 0x7fe510071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:42.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.417+0000 7fe51a7fc700 1 -- 172.21.15.67:0/716923951 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fe52008f0d0 con 0x7fe524100ea0 2024-04-22T23:33:42.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.417+0000 7fe528a6c700 1 --2- 172.21.15.67:0/716923951 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe51006ecc0 0x7fe510071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:42.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.418+0000 7fe528a6c700 1 --2- 172.21.15.67:0/716923951 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe51006ecc0 0x7fe510071180 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe514006010 tx=0x7fe51400b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:33:42.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.422+0000 7fe51a7fc700 1 -- 172.21.15.67:0/716923951 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe52005b8c0 con 0x7fe524100ea0 2024-04-22T23:33:42.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.640+0000 7fe52b4d1700 1 -- 172.21.15.67:0/716923951 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe524197d50 con 0x7fe51006ecc0 2024-04-22T23:33:42.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.651+0000 7fe51a7fc700 1 -- 172.21.15.67:0/716923951 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7fe524197d50 con 0x7fe51006ecc0 2024-04-22T23:33:42.647 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:33:42.647 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (7m) 41s ago 18m 25.5M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:33:42.647 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (18m) 41s ago 18m 9114k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:33:42.647 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (17m) 23s ago 17m 9042k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:33:42.647 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (4m) 41s ago 18m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:33:42.647 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (4m) 23s ago 17m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:33:42.647 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (7m) 41s ago 18m 79.3M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:33:42.647 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (66s) 41s ago 13m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:33:42.647 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (46s) 41s ago 13m 12.6M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:33:42.648 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (27s) 23s ago 13m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:33:42.648 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (13m) 23s ago 13m 12.2M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:33:42.648 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (9m) 41s ago 20m 511M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:33:42.648 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (8m) 23s ago 17m 453M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:33:42.648 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (5m) 41s ago 20m 55.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:33:42.648 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (5m) 23s ago 17m 46.0M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:33:42.648 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (8m) 41s ago 18m 15.4M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:33:42.648 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (8m) 23s ago 17m 16.1M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:33:42.648 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (4m) 41s ago 16m 99.2M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:33:42.648 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (3m) 41s ago 16m 127M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:33:42.648 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (3m) 41s ago 16m 79.9M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:33:42.648 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (2m) 23s ago 15m 97.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:33:42.648 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (119s) 23s ago 15m 82.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:33:42.648 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (86s) 23s ago 14m 77.6M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:33:42.648 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (7m) 41s ago 18m 73.9M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:33:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.654+0000 7fe52b4d1700 1 -- 172.21.15.67:0/716923951 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe51006ecc0 msgr2=0x7fe510071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.654+0000 7fe52b4d1700 1 --2- 172.21.15.67:0/716923951 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe51006ecc0 0x7fe510071180 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe514006010 tx=0x7fe51400b540 comp rx=0 tx=0).stop 2024-04-22T23:33:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.654+0000 7fe52b4d1700 1 -- 172.21.15.67:0/716923951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe524100ea0 msgr2=0x7fe52419c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.654+0000 7fe52b4d1700 1 --2- 172.21.15.67:0/716923951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe524100ea0 0x7fe52419c700 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe52000c930 tx=0x7fe52000ccf0 comp rx=0 tx=0).stop 2024-04-22T23:33:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.654+0000 7fe52b4d1700 1 -- 172.21.15.67:0/716923951 shutdown_connections 2024-04-22T23:33:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.654+0000 7fe52b4d1700 1 --2- 172.21.15.67:0/716923951 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe51006ecc0 0x7fe510071180 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.654+0000 7fe52b4d1700 1 --2- 172.21.15.67:0/716923951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe524100ea0 0x7fe52419c700 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.654+0000 7fe52b4d1700 1 --2- 172.21.15.67:0/716923951 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe524101850 0x7fe52419cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.654+0000 7fe52b4d1700 1 -- 172.21.15.67:0/716923951 >> 172.21.15.67:0/716923951 conn(0x7fe5240fc750 msgr2=0x7fe5240fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:33:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.654+0000 7fe52b4d1700 1 -- 172.21.15.67:0/716923951 shutdown_connections 2024-04-22T23:33:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.654+0000 7fe52b4d1700 1 -- 172.21.15.67:0/716923951 wait complete. 2024-04-22T23:33:42.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.761+0000 7f7d81453700 1 -- 172.21.15.67:0/3510914230 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d7c0ffbd0 msgr2=0x7f7d7c0fffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:42.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.761+0000 7f7d81453700 1 --2- 172.21.15.67:0/3510914230 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d7c0ffbd0 0x7f7d7c0fffb0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7d64009910 tx=0x7f7d64009c20 comp rx=0 tx=0).stop 2024-04-22T23:33:42.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.761+0000 7f7d81453700 1 -- 172.21.15.67:0/3510914230 shutdown_connections 2024-04-22T23:33:42.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.761+0000 7f7d81453700 1 --2- 172.21.15.67:0/3510914230 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d7c1004f0 0x7f7d7c1049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:42.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.761+0000 7f7d81453700 1 --2- 172.21.15.67:0/3510914230 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d7c0ffbd0 0x7f7d7c0fffb0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:42.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.761+0000 7f7d81453700 1 -- 172.21.15.67:0/3510914230 >> 172.21.15.67:0/3510914230 conn(0x7f7d7c0fb650 msgr2=0x7f7d7c0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:33:42.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.761+0000 7f7d81453700 1 -- 172.21.15.67:0/3510914230 shutdown_connections 2024-04-22T23:33:42.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.761+0000 7f7d81453700 1 -- 172.21.15.67:0/3510914230 wait complete. 2024-04-22T23:33:42.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.761+0000 7f7d81453700 1 Processor -- start 2024-04-22T23:33:42.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.762+0000 7f7d81453700 1 -- start start 2024-04-22T23:33:42.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.762+0000 7f7d81453700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d7c0ffbd0 0x7f7d7c19cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:42.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.762+0000 7f7d81453700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d7c1004f0 0x7f7d7c19d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:42.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.762+0000 7f7d81453700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7c19db50 con 0x7f7d7c0ffbd0 2024-04-22T23:33:42.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.762+0000 7f7d81453700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7c1a18e0 con 0x7f7d7c1004f0 2024-04-22T23:33:42.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.762+0000 7f7d7affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d7c0ffbd0 0x7f7d7c19cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:42.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.762+0000 7f7d7affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d7c0ffbd0 0x7f7d7c19cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42862/0 (socket says 172.21.15.67:42862) 2024-04-22T23:33:42.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.762+0000 7f7d7affd700 1 -- 172.21.15.67:0/3401393971 learned_addr learned my addr 172.21.15.67:0/3401393971 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:33:42.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.762+0000 7f7d7a7fc700 1 --2- 172.21.15.67:0/3401393971 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d7c1004f0 0x7f7d7c19d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:42.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.762+0000 7f7d7affd700 1 -- 172.21.15.67:0/3401393971 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d7c1004f0 msgr2=0x7f7d7c19d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:42.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.762+0000 7f7d7affd700 1 --2- 172.21.15.67:0/3401393971 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d7c1004f0 0x7f7d7c19d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:42.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.762+0000 7f7d7affd700 1 -- 172.21.15.67:0/3401393971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d640095f0 con 0x7f7d7c0ffbd0 2024-04-22T23:33:42.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.762+0000 7f7d7a7fc700 1 --2- 172.21.15.67:0/3401393971 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d7c1004f0 0x7f7d7c19d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:33:42.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.762+0000 7f7d7affd700 1 --2- 172.21.15.67:0/3401393971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d7c0ffbd0 0x7f7d7c19cf30 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7d640092e0 tx=0x7f7d6400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:33:42.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.763+0000 7f7d73fff700 1 -- 172.21.15.67:0/3401393971 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d6401d070 con 0x7f7d7c0ffbd0 2024-04-22T23:33:42.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.763+0000 7f7d81453700 1 -- 172.21.15.67:0/3401393971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d7c1a1b60 con 0x7f7d7c0ffbd0 2024-04-22T23:33:42.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.763+0000 7f7d73fff700 1 -- 172.21.15.67:0/3401393971 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7d64005e00 con 0x7f7d7c0ffbd0 2024-04-22T23:33:42.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.763+0000 7f7d73fff700 1 -- 172.21.15.67:0/3401393971 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d64017400 con 0x7f7d7c0ffbd0 2024-04-22T23:33:42.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.763+0000 7f7d81453700 1 -- 172.21.15.67:0/3401393971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d7c1a2050 con 0x7f7d7c0ffbd0 2024-04-22T23:33:42.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.764+0000 7f7d81453700 1 -- 172.21.15.67:0/3401393971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d7c102120 con 0x7f7d7c0ffbd0 2024-04-22T23:33:42.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.764+0000 7f7d73fff700 1 -- 172.21.15.67:0/3401393971 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7d6400f460 con 0x7f7d7c0ffbd0 2024-04-22T23:33:42.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.765+0000 7f7d73fff700 1 --2- 172.21.15.67:0/3401393971 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7d6806ecc0 0x7f7d68071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:42.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.765+0000 7f7d73fff700 1 -- 172.21.15.67:0/3401393971 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f7d64090520 con 0x7f7d7c0ffbd0 2024-04-22T23:33:42.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.765+0000 7f7d7a7fc700 1 --2- 172.21.15.67:0/3401393971 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7d6806ecc0 0x7f7d68071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:42.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.765+0000 7f7d7a7fc700 1 --2- 172.21.15.67:0/3401393971 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7d6806ecc0 0x7f7d68071180 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c005950 tx=0x7f7d6c0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:33:42.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:42.770+0000 7f7d73fff700 1 -- 172.21.15.67:0/3401393971 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7d64059160 con 0x7f7d7c0ffbd0 2024-04-22T23:33:43.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.076+0000 7f7d81453700 1 -- 172.21.15.67:0/3401393971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7d7c04f2e0 con 0x7f7d7c0ffbd0 2024-04-22T23:33:43.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.076+0000 7f7d73fff700 1 -- 172.21.15.67:0/3401393971 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f7d6405c820 con 0x7f7d7c0ffbd0 2024-04-22T23:33:43.072 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:33:43.072 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:33:43.072 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:33:43.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:33:43.072 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:33:43.073 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:33:43.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:33:43.073 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:33:43.073 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:33:43.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:33:43.073 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:33:43.073 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:33:43.073 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 3 2024-04-22T23:33:43.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:33:43.073 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:33:43.073 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:33:43.073 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 13 2024-04-22T23:33:43.074 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:33:43.074 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:33:43.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.078+0000 7f7d81453700 1 -- 172.21.15.67:0/3401393971 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7d6806ecc0 msgr2=0x7f7d68071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:43.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.078+0000 7f7d81453700 1 --2- 172.21.15.67:0/3401393971 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7d6806ecc0 0x7f7d68071180 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c005950 tx=0x7f7d6c0058e0 comp rx=0 tx=0).stop 2024-04-22T23:33:43.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.078+0000 7f7d81453700 1 -- 172.21.15.67:0/3401393971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d7c0ffbd0 msgr2=0x7f7d7c19cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:43.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.079+0000 7f7d81453700 1 --2- 172.21.15.67:0/3401393971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d7c0ffbd0 0x7f7d7c19cf30 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7d640092e0 tx=0x7f7d6400be20 comp rx=0 tx=0).stop 2024-04-22T23:33:43.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.079+0000 7f7d81453700 1 -- 172.21.15.67:0/3401393971 shutdown_connections 2024-04-22T23:33:43.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.079+0000 7f7d81453700 1 --2- 172.21.15.67:0/3401393971 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7d6806ecc0 0x7f7d68071180 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:43.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.079+0000 7f7d81453700 1 --2- 172.21.15.67:0/3401393971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d7c0ffbd0 0x7f7d7c19cf30 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:43.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.079+0000 7f7d81453700 1 --2- 172.21.15.67:0/3401393971 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d7c1004f0 0x7f7d7c19d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:43.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.079+0000 7f7d81453700 1 -- 172.21.15.67:0/3401393971 >> 172.21.15.67:0/3401393971 conn(0x7f7d7c0fb650 msgr2=0x7f7d7c111040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:33:43.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.079+0000 7f7d81453700 1 -- 172.21.15.67:0/3401393971 shutdown_connections 2024-04-22T23:33:43.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.079+0000 7f7d81453700 1 -- 172.21.15.67:0/3401393971 wait complete. 2024-04-22T23:33:43.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.187+0000 7f4a8ee5c700 1 -- 172.21.15.67:0/3490056487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a88100ea0 msgr2=0x7f4a88101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:43.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.187+0000 7f4a8ee5c700 1 --2- 172.21.15.67:0/3490056487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a88100ea0 0x7f4a88101280 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4a8000b0a0 tx=0x7f4a8000b3b0 comp rx=0 tx=0).stop 2024-04-22T23:33:43.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.187+0000 7f4a8ee5c700 1 -- 172.21.15.67:0/3490056487 shutdown_connections 2024-04-22T23:33:43.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.187+0000 7f4a8ee5c700 1 --2- 172.21.15.67:0/3490056487 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a88101850 0x7f4a881058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:43.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.187+0000 7f4a8ee5c700 1 --2- 172.21.15.67:0/3490056487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a88100ea0 0x7f4a88101280 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:43.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.187+0000 7f4a8ee5c700 1 -- 172.21.15.67:0/3490056487 >> 172.21.15.67:0/3490056487 conn(0x7f4a880fc750 msgr2=0x7f4a880feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:33:43.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.187+0000 7f4a8ee5c700 1 -- 172.21.15.67:0/3490056487 shutdown_connections 2024-04-22T23:33:43.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.187+0000 7f4a8ee5c700 1 -- 172.21.15.67:0/3490056487 wait complete. 2024-04-22T23:33:43.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.188+0000 7f4a8ee5c700 1 Processor -- start 2024-04-22T23:33:43.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.188+0000 7f4a8ee5c700 1 -- start start 2024-04-22T23:33:43.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.188+0000 7f4a8ee5c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a88100ea0 0x7f4a88073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:43.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.188+0000 7f4a8ee5c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a88101850 0x7f4a880735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:43.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.188+0000 7f4a8ee5c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a88073b10 con 0x7f4a88100ea0 2024-04-22T23:33:43.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.188+0000 7f4a8ee5c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a88073c50 con 0x7f4a88101850 2024-04-22T23:33:43.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.188+0000 7f4a8cbf8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a88100ea0 0x7f4a88073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:43.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.188+0000 7f4a8cbf8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a88100ea0 0x7f4a88073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42882/0 (socket says 172.21.15.67:42882) 2024-04-22T23:33:43.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.188+0000 7f4a8cbf8700 1 -- 172.21.15.67:0/153941996 learned_addr learned my addr 172.21.15.67:0/153941996 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:33:43.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.188+0000 7f4a8cbf8700 1 -- 172.21.15.67:0/153941996 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a88101850 msgr2=0x7f4a880735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:43.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.188+0000 7f4a8cbf8700 1 --2- 172.21.15.67:0/153941996 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a88101850 0x7f4a880735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:43.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.188+0000 7f4a8cbf8700 1 -- 172.21.15.67:0/153941996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a80009d00 con 0x7f4a88100ea0 2024-04-22T23:33:43.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.189+0000 7f4a8cbf8700 1 --2- 172.21.15.67:0/153941996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a88100ea0 0x7f4a88073090 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4a8000b070 tx=0x7f4a8000cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:33:43.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.189+0000 7f4a85ffb700 1 -- 172.21.15.67:0/153941996 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a8000e040 con 0x7f4a88100ea0 2024-04-22T23:33:43.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.189+0000 7f4a85ffb700 1 -- 172.21.15.67:0/153941996 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4a80003e60 con 0x7f4a88100ea0 2024-04-22T23:33:43.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.189+0000 7f4a8ee5c700 1 -- 172.21.15.67:0/153941996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a88073ed0 con 0x7f4a88100ea0 2024-04-22T23:33:43.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.189+0000 7f4a85ffb700 1 -- 172.21.15.67:0/153941996 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a8001d910 con 0x7f4a88100ea0 2024-04-22T23:33:43.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.189+0000 7f4a8ee5c700 1 -- 172.21.15.67:0/153941996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a881a2f10 con 0x7f4a88100ea0 2024-04-22T23:33:43.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.190+0000 7f4a8ee5c700 1 -- 172.21.15.67:0/153941996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a88109240 con 0x7f4a88100ea0 2024-04-22T23:33:43.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.191+0000 7f4a85ffb700 1 -- 172.21.15.67:0/153941996 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4a80004820 con 0x7f4a88100ea0 2024-04-22T23:33:43.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.191+0000 7f4a85ffb700 1 --2- 172.21.15.67:0/153941996 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4a7006ecc0 0x7f4a70071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:43.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.191+0000 7f4a87fff700 1 --2- 172.21.15.67:0/153941996 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4a7006ecc0 0x7f4a70071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:43.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.191+0000 7f4a85ffb700 1 -- 172.21.15.67:0/153941996 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f4a80090830 con 0x7f4a88100ea0 2024-04-22T23:33:43.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.192+0000 7f4a87fff700 1 --2- 172.21.15.67:0/153941996 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4a7006ecc0 0x7f4a70071180 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4a88074880 tx=0x7f4a7c006d20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:33:43.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.196+0000 7f4a85ffb700 1 -- 172.21.15.67:0/153941996 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f4a800594f0 con 0x7f4a88100ea0 2024-04-22T23:33:43.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.456+0000 7f4a8ee5c700 1 -- 172.21.15.67:0/153941996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4a8804f2e0 con 0x7f4a88100ea0 2024-04-22T23:33:43.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.456+0000 7f4a85ffb700 1 -- 172.21.15.67:0/153941996 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1871 (secure 0 0 0) 0x7f4a8005cbb0 con 0x7f4a88100ea0 2024-04-22T23:33:43.452 INFO:teuthology.orchestra.run.smithi067.stdout:e31 2024-04-22T23:33:43.452 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:33:43.453 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:33:43.453 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:33:43.453 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:33:43.453 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:33:43.453 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:33:43.453 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:33:43.453 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:33:43.453 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:33:43.453 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:33:43.453 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:33:43.453 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:33:43.453 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:33:43.453 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:33:43.453 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:33:43.454 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:33:43.454 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:33:43.454 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:33:43.454 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:33:43.454 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:33:43.454 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:33:43.454 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:33:43.454 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:33:43.454 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:33:43.454 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:33:43.454 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:33:43.454 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:33:43.454 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:33:43.454 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:33:43.455 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:33:43.455 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:33:43.455 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:33:43.455 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:33:43.455 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:33:43.455 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:33:43.455 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:33:43.455 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:33:43.455 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:33:43.455 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:33:43.455 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44259} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1827288036,v1:172.21.15.134:6827/1827288036] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:33:43.455 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:33:43.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.459+0000 7f4a8ee5c700 1 -- 172.21.15.67:0/153941996 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4a7006ecc0 msgr2=0x7f4a70071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:43.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.459+0000 7f4a8ee5c700 1 --2- 172.21.15.67:0/153941996 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4a7006ecc0 0x7f4a70071180 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4a88074880 tx=0x7f4a7c006d20 comp rx=0 tx=0).stop 2024-04-22T23:33:43.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.459+0000 7f4a8ee5c700 1 -- 172.21.15.67:0/153941996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a88100ea0 msgr2=0x7f4a88073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:43.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.459+0000 7f4a8ee5c700 1 --2- 172.21.15.67:0/153941996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a88100ea0 0x7f4a88073090 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4a8000b070 tx=0x7f4a8000cdd0 comp rx=0 tx=0).stop 2024-04-22T23:33:43.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.459+0000 7f4a8ee5c700 1 -- 172.21.15.67:0/153941996 shutdown_connections 2024-04-22T23:33:43.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.459+0000 7f4a8ee5c700 1 --2- 172.21.15.67:0/153941996 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4a7006ecc0 0x7f4a70071180 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:43.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.459+0000 7f4a8ee5c700 1 --2- 172.21.15.67:0/153941996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a88100ea0 0x7f4a88073090 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:43.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.459+0000 7f4a8ee5c700 1 --2- 172.21.15.67:0/153941996 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a88101850 0x7f4a880735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:43.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.459+0000 7f4a8ee5c700 1 -- 172.21.15.67:0/153941996 >> 172.21.15.67:0/153941996 conn(0x7f4a880fc750 msgr2=0x7f4a880fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:33:43.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.459+0000 7f4a8ee5c700 1 -- 172.21.15.67:0/153941996 shutdown_connections 2024-04-22T23:33:43.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.459+0000 7f4a8ee5c700 1 -- 172.21.15.67:0/153941996 wait complete. 2024-04-22T23:33:43.457 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 31 2024-04-22T23:33:43.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.569+0000 7f6c1f18f700 1 -- 172.21.15.67:0/3919435301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c18100ea0 msgr2=0x7f6c18101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:43.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.569+0000 7f6c1f18f700 1 --2- 172.21.15.67:0/3919435301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c18100ea0 0x7f6c18101280 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6c00009910 tx=0x7f6c00009c20 comp rx=0 tx=0).stop 2024-04-22T23:33:43.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.569+0000 7f6c1f18f700 1 -- 172.21.15.67:0/3919435301 shutdown_connections 2024-04-22T23:33:43.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.569+0000 7f6c1f18f700 1 --2- 172.21.15.67:0/3919435301 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c18101850 0x7f6c181058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:43.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.569+0000 7f6c1f18f700 1 --2- 172.21.15.67:0/3919435301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c18100ea0 0x7f6c18101280 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:43.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.569+0000 7f6c1f18f700 1 -- 172.21.15.67:0/3919435301 >> 172.21.15.67:0/3919435301 conn(0x7f6c180fc750 msgr2=0x7f6c180feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:33:43.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.569+0000 7f6c1f18f700 1 -- 172.21.15.67:0/3919435301 shutdown_connections 2024-04-22T23:33:43.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.569+0000 7f6c1f18f700 1 -- 172.21.15.67:0/3919435301 wait complete. 2024-04-22T23:33:43.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.570+0000 7f6c1f18f700 1 Processor -- start 2024-04-22T23:33:43.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.570+0000 7f6c1f18f700 1 -- start start 2024-04-22T23:33:43.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.570+0000 7f6c1f18f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c18100ea0 0x7f6c1819c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:43.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.570+0000 7f6c1f18f700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c18101850 0x7f6c1819cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:43.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.570+0000 7f6c1f18f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1819d2c0 con 0x7f6c18100ea0 2024-04-22T23:33:43.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.570+0000 7f6c1f18f700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c18196770 con 0x7f6c18101850 2024-04-22T23:33:43.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.570+0000 7f6c1cf2b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c18100ea0 0x7f6c1819c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:43.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.570+0000 7f6c1cf2b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c18100ea0 0x7f6c1819c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42904/0 (socket says 172.21.15.67:42904) 2024-04-22T23:33:43.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.570+0000 7f6c1cf2b700 1 -- 172.21.15.67:0/2363546713 learned_addr learned my addr 172.21.15.67:0/2363546713 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:33:43.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.570+0000 7f6c17fff700 1 --2- 172.21.15.67:0/2363546713 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c18101850 0x7f6c1819cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:43.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.570+0000 7f6c1cf2b700 1 -- 172.21.15.67:0/2363546713 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c18101850 msgr2=0x7f6c1819cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:43.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.570+0000 7f6c1cf2b700 1 --2- 172.21.15.67:0/2363546713 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c18101850 0x7f6c1819cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:43.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.570+0000 7f6c1cf2b700 1 -- 172.21.15.67:0/2363546713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c000095f0 con 0x7f6c18100ea0 2024-04-22T23:33:43.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.570+0000 7f6c17fff700 1 --2- 172.21.15.67:0/2363546713 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c18101850 0x7f6c1819cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:33:43.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.570+0000 7f6c1cf2b700 1 --2- 172.21.15.67:0/2363546713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c18100ea0 0x7f6c1819c6f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6c00006010 tx=0x7f6c0000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:33:43.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.571+0000 7f6c15ffb700 1 -- 172.21.15.67:0/2363546713 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c0001d070 con 0x7f6c18100ea0 2024-04-22T23:33:43.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.571+0000 7f6c15ffb700 1 -- 172.21.15.67:0/2363546713 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6c0000f460 con 0x7f6c18100ea0 2024-04-22T23:33:43.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.571+0000 7f6c1f18f700 1 -- 172.21.15.67:0/2363546713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c181969f0 con 0x7f6c18100ea0 2024-04-22T23:33:43.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.571+0000 7f6c15ffb700 1 -- 172.21.15.67:0/2363546713 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c00017630 con 0x7f6c18100ea0 2024-04-22T23:33:43.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.571+0000 7f6c1f18f700 1 -- 172.21.15.67:0/2363546713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c18196ee0 con 0x7f6c18100ea0 2024-04-22T23:33:43.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.572+0000 7f6c1f18f700 1 -- 172.21.15.67:0/2363546713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c18109240 con 0x7f6c18100ea0 2024-04-22T23:33:43.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.572+0000 7f6c15ffb700 1 -- 172.21.15.67:0/2363546713 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6c0000f5d0 con 0x7f6c18100ea0 2024-04-22T23:33:43.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.573+0000 7f6c15ffb700 1 --2- 172.21.15.67:0/2363546713 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6c0406ecc0 0x7f6c04071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:43.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.573+0000 7f6c15ffb700 1 -- 172.21.15.67:0/2363546713 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f6c00090420 con 0x7f6c18100ea0 2024-04-22T23:33:43.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.573+0000 7f6c17fff700 1 --2- 172.21.15.67:0/2363546713 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6c0406ecc0 0x7f6c04071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:43.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.573+0000 7f6c17fff700 1 --2- 172.21.15.67:0/2363546713 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6c0406ecc0 0x7f6c04071180 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6c18197be0 tx=0x7f6c08009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:33:43.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.578+0000 7f6c15ffb700 1 -- 172.21.15.67:0/2363546713 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f6c0005cc10 con 0x7f6c18100ea0 2024-04-22T23:33:43.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:43 smithi134 ceph-mon[85630]: from='client.34456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:33:43.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:43 smithi134 ceph-mon[85630]: pgmap v220: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:43.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:43 smithi134 ceph-mon[85630]: from='client.34460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:33:43.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:43 smithi134 ceph-mon[85630]: from='client.34464 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:33:43.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:43 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3401393971' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:33:43.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:43 smithi067 ceph-mon[112438]: from='client.34456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:33:43.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:43 smithi067 ceph-mon[112438]: pgmap v220: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:43.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:43 smithi067 ceph-mon[112438]: from='client.34460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:33:43.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:43 smithi067 ceph-mon[112438]: from='client.34464 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:33:43.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:43 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3401393971' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:33:43.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.808+0000 7f6c1f18f700 1 -- 172.21.15.67:0/2363546713 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6c18197d90 con 0x7f6c0406ecc0 2024-04-22T23:33:43.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.809+0000 7f6c15ffb700 1 -- 172.21.15.67:0/2363546713 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f6c18197d90 con 0x7f6c0406ecc0 2024-04-22T23:33:43.804 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:33:43.804 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:33:43.805 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:33:43.805 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:33:43.805 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:33:43.805 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:33:43.805 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:33:43.805 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:33:43.805 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:33:43.805 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:33:43.805 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T23:33:43.805 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:33:43.805 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:33:43.805 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:33:43.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.811+0000 7f6c1f18f700 1 -- 172.21.15.67:0/2363546713 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6c0406ecc0 msgr2=0x7f6c04071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:43.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.811+0000 7f6c1f18f700 1 --2- 172.21.15.67:0/2363546713 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6c0406ecc0 0x7f6c04071180 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6c18197be0 tx=0x7f6c08009040 comp rx=0 tx=0).stop 2024-04-22T23:33:43.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.811+0000 7f6c1f18f700 1 -- 172.21.15.67:0/2363546713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c18100ea0 msgr2=0x7f6c1819c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:43.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.812+0000 7f6c1f18f700 1 --2- 172.21.15.67:0/2363546713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c18100ea0 0x7f6c1819c6f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6c00006010 tx=0x7f6c0000be20 comp rx=0 tx=0).stop 2024-04-22T23:33:43.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.812+0000 7f6c1f18f700 1 -- 172.21.15.67:0/2363546713 shutdown_connections 2024-04-22T23:33:43.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.812+0000 7f6c1f18f700 1 --2- 172.21.15.67:0/2363546713 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6c0406ecc0 0x7f6c04071180 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:43.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.812+0000 7f6c1f18f700 1 --2- 172.21.15.67:0/2363546713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c18100ea0 0x7f6c1819c6f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:43.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.812+0000 7f6c1f18f700 1 --2- 172.21.15.67:0/2363546713 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c18101850 0x7f6c1819cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:43.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.812+0000 7f6c1f18f700 1 -- 172.21.15.67:0/2363546713 >> 172.21.15.67:0/2363546713 conn(0x7f6c180fc750 msgr2=0x7f6c180fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:33:43.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.812+0000 7f6c1f18f700 1 -- 172.21.15.67:0/2363546713 shutdown_connections 2024-04-22T23:33:43.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.812+0000 7f6c1f18f700 1 -- 172.21.15.67:0/2363546713 wait complete. 2024-04-22T23:33:43.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.919+0000 7fd4c329d700 1 -- 172.21.15.67:0/1863351788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4bc105bf0 msgr2=0x7fd4bc105fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:43.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.919+0000 7fd4c329d700 1 --2- 172.21.15.67:0/1863351788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4bc105bf0 0x7fd4bc105fd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd4b8009910 tx=0x7fd4b8009c20 comp rx=0 tx=0).stop 2024-04-22T23:33:43.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.919+0000 7fd4c329d700 1 -- 172.21.15.67:0/1863351788 shutdown_connections 2024-04-22T23:33:43.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.919+0000 7fd4c329d700 1 --2- 172.21.15.67:0/1863351788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4bc0ffb40 0x7fd4bc0fffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:43.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.919+0000 7fd4c329d700 1 --2- 172.21.15.67:0/1863351788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4bc105bf0 0x7fd4bc105fd0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:43.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.919+0000 7fd4c329d700 1 -- 172.21.15.67:0/1863351788 >> 172.21.15.67:0/1863351788 conn(0x7fd4bc0fb650 msgr2=0x7fd4bc0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:33:43.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.919+0000 7fd4c329d700 1 -- 172.21.15.67:0/1863351788 shutdown_connections 2024-04-22T23:33:43.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.919+0000 7fd4c329d700 1 -- 172.21.15.67:0/1863351788 wait complete. 2024-04-22T23:33:43.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.920+0000 7fd4c329d700 1 Processor -- start 2024-04-22T23:33:43.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.920+0000 7fd4c329d700 1 -- start start 2024-04-22T23:33:43.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.920+0000 7fd4c329d700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4bc0ffb40 0x7fd4bc19c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:43.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.920+0000 7fd4c329d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4bc105bf0 0x7fd4bc19cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:43.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.920+0000 7fd4c329d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4bc19d170 con 0x7fd4bc105bf0 2024-04-22T23:33:43.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.920+0000 7fd4c329d700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4bc196620 con 0x7fd4bc0ffb40 2024-04-22T23:33:43.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.920+0000 7fd4c0838700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4bc105bf0 0x7fd4bc19cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:43.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.920+0000 7fd4c0838700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4bc105bf0 0x7fd4bc19cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42916/0 (socket says 172.21.15.67:42916) 2024-04-22T23:33:43.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.920+0000 7fd4c0838700 1 -- 172.21.15.67:0/3341391657 learned_addr learned my addr 172.21.15.67:0/3341391657 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:33:43.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.920+0000 7fd4c1039700 1 --2- 172.21.15.67:0/3341391657 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4bc0ffb40 0x7fd4bc19c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:43.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.921+0000 7fd4c0838700 1 -- 172.21.15.67:0/3341391657 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4bc0ffb40 msgr2=0x7fd4bc19c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:43.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.921+0000 7fd4c0838700 1 --2- 172.21.15.67:0/3341391657 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4bc0ffb40 0x7fd4bc19c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:43.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.921+0000 7fd4c0838700 1 -- 172.21.15.67:0/3341391657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4b80095f0 con 0x7fd4bc105bf0 2024-04-22T23:33:43.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.921+0000 7fd4c0838700 1 --2- 172.21.15.67:0/3341391657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4bc105bf0 0x7fd4bc19cae0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac00e920 tx=0x7fd4ac00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:33:43.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.921+0000 7fd4b27fc700 1 -- 172.21.15.67:0/3341391657 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4ac00c9d0 con 0x7fd4bc105bf0 2024-04-22T23:33:43.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.921+0000 7fd4b27fc700 1 -- 172.21.15.67:0/3341391657 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd4ac010430 con 0x7fd4bc105bf0 2024-04-22T23:33:43.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.921+0000 7fd4c329d700 1 -- 172.21.15.67:0/3341391657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4bc196900 con 0x7fd4bc105bf0 2024-04-22T23:33:43.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.921+0000 7fd4b27fc700 1 -- 172.21.15.67:0/3341391657 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4ac018630 con 0x7fd4bc105bf0 2024-04-22T23:33:43.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.921+0000 7fd4c329d700 1 -- 172.21.15.67:0/3341391657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4bc196e50 con 0x7fd4bc105bf0 2024-04-22T23:33:43.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.922+0000 7fd4c329d700 1 -- 172.21.15.67:0/3341391657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4bc04f2e0 con 0x7fd4bc105bf0 2024-04-22T23:33:43.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.923+0000 7fd4b27fc700 1 -- 172.21.15.67:0/3341391657 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd4ac0105a0 con 0x7fd4bc105bf0 2024-04-22T23:33:43.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.923+0000 7fd4b27fc700 1 --2- 172.21.15.67:0/3341391657 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd4a806ecc0 0x7fd4a8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:33:43.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.923+0000 7fd4b27fc700 1 -- 172.21.15.67:0/3341391657 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fd4ac014070 con 0x7fd4bc105bf0 2024-04-22T23:33:43.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.923+0000 7fd4c1039700 1 --2- 172.21.15.67:0/3341391657 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd4a806ecc0 0x7fd4a8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:33:43.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.923+0000 7fd4c1039700 1 --2- 172.21.15.67:0/3341391657 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd4a806ecc0 0x7fd4a8071180 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd4b80059d0 tx=0x7fd4b8005940 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:33:43.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:43.928+0000 7fd4b27fc700 1 -- 172.21.15.67:0/3341391657 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fd4ac054150 con 0x7fd4bc105bf0 2024-04-22T23:33:44.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:44.225+0000 7fd4c329d700 1 -- 172.21.15.67:0/3341391657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fd4bc1971e0 con 0x7fd4bc105bf0 2024-04-22T23:33:44.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:44.225+0000 7fd4b27fc700 1 -- 172.21.15.67:0/3341391657 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fd4ac062070 con 0x7fd4bc105bf0 2024-04-22T23:33:44.222 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:33:44.222 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:33:44.222 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:33:44.222 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:33:44.222 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:33:44.222 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:33:44.222 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:33:44.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:44.227+0000 7fd4c329d700 1 -- 172.21.15.67:0/3341391657 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd4a806ecc0 msgr2=0x7fd4a8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:44.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:44.227+0000 7fd4c329d700 1 --2- 172.21.15.67:0/3341391657 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd4a806ecc0 0x7fd4a8071180 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd4b80059d0 tx=0x7fd4b8005940 comp rx=0 tx=0).stop 2024-04-22T23:33:44.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:44.227+0000 7fd4c329d700 1 -- 172.21.15.67:0/3341391657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4bc105bf0 msgr2=0x7fd4bc19cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:33:44.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:44.227+0000 7fd4c329d700 1 --2- 172.21.15.67:0/3341391657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4bc105bf0 0x7fd4bc19cae0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac00e920 tx=0x7fd4ac00ec30 comp rx=0 tx=0).stop 2024-04-22T23:33:44.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:44.228+0000 7fd4c329d700 1 -- 172.21.15.67:0/3341391657 shutdown_connections 2024-04-22T23:33:44.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:44.228+0000 7fd4c329d700 1 --2- 172.21.15.67:0/3341391657 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd4a806ecc0 0x7fd4a8071180 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:44.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:44.228+0000 7fd4c329d700 1 --2- 172.21.15.67:0/3341391657 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd4bc0ffb40 0x7fd4bc19c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:44.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:44.228+0000 7fd4c329d700 1 --2- 172.21.15.67:0/3341391657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4bc105bf0 0x7fd4bc19cae0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:33:44.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:44.228+0000 7fd4c329d700 1 -- 172.21.15.67:0/3341391657 >> 172.21.15.67:0/3341391657 conn(0x7fd4bc0fb650 msgr2=0x7fd4bc068e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:33:44.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:44.228+0000 7fd4c329d700 1 -- 172.21.15.67:0/3341391657 shutdown_connections 2024-04-22T23:33:44.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:33:44.228+0000 7fd4c329d700 1 -- 172.21.15.67:0/3341391657 wait complete. 2024-04-22T23:33:44.539 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:44 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/153941996' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:33:44.539 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:44 smithi067 ceph-mon[112438]: from='client.34476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:33:44.539 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:44 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3341391657' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:33:44.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:44 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/153941996' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:33:44.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:44 smithi134 ceph-mon[85630]: from='client.34476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:33:44.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:44 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3341391657' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:33:45.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:45 smithi134 ceph-mon[85630]: pgmap v221: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:45.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:45 smithi067 ceph-mon[112438]: pgmap v221: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:47.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:47 smithi134 ceph-mon[85630]: pgmap v222: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:47.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:47 smithi067 ceph-mon[112438]: pgmap v222: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:49.738 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:49 smithi134 ceph-mon[85630]: pgmap v223: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:49.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:49 smithi067 ceph-mon[112438]: pgmap v223: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:51.738 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:51 smithi134 ceph-mon[85630]: pgmap v224: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:51.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:51 smithi067 ceph-mon[112438]: pgmap v224: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:53.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:53 smithi134 ceph-mon[85630]: pgmap v225: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:53.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:53 smithi067 ceph-mon[112438]: pgmap v225: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:55.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:55 smithi134 ceph-mon[85630]: pgmap v226: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:55.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:55 smithi067 ceph-mon[112438]: pgmap v226: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:56.738 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:33:56.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:33:57.738 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:57 smithi134 ceph-mon[85630]: pgmap v227: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:57.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:57 smithi067 ceph-mon[112438]: pgmap v227: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:59.738 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:33:59 smithi134 ceph-mon[85630]: pgmap v228: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:33:59.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:33:59 smithi067 ceph-mon[112438]: pgmap v228: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:01.738 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:01 smithi134 ceph-mon[85630]: pgmap v229: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:01.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:01 smithi067 ceph-mon[112438]: pgmap v229: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:03.539 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:03 smithi067 ceph-mon[112438]: pgmap v230: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:03.738 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:03 smithi134 ceph-mon[85630]: pgmap v230: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:05.739 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:05 smithi134 ceph-mon[85630]: pgmap v231: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:05.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:05 smithi067 ceph-mon[112438]: pgmap v231: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:07.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:07 smithi067 ceph-mon[112438]: pgmap v232: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:07.826 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:07 smithi134 ceph-mon[85630]: pgmap v232: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:09.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:09 smithi067 ceph-mon[112438]: pgmap v233: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:09.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:09 smithi134 ceph-mon[85630]: pgmap v233: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:11.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:11 smithi067 ceph-mon[112438]: pgmap v234: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:11.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:34:11.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:11 smithi134 ceph-mon[85630]: pgmap v234: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:11.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:34:13.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:13 smithi067 ceph-mon[112438]: pgmap v235: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:13.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:13 smithi134 ceph-mon[85630]: pgmap v235: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:14.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.350+0000 7f523edb5700 1 -- 172.21.15.67:0/3620857876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5238106cd0 msgr2=0x7f52381070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:14.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.350+0000 7f523edb5700 1 --2- 172.21.15.67:0/3620857876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5238106cd0 0x7f52381070b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f522c009910 tx=0x7f522c009c20 comp rx=0 tx=0).stop 2024-04-22T23:34:14.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.350+0000 7f523edb5700 1 -- 172.21.15.67:0/3620857876 shutdown_connections 2024-04-22T23:34:14.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.350+0000 7f523edb5700 1 --2- 172.21.15.67:0/3620857876 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5238068580 0x7f52380689e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:14.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.350+0000 7f523edb5700 1 --2- 172.21.15.67:0/3620857876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5238106cd0 0x7f52381070b0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:14.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.350+0000 7f523edb5700 1 -- 172.21.15.67:0/3620857876 >> 172.21.15.67:0/3620857876 conn(0x7f5238075670 msgr2=0x7f5238075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:34:14.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.350+0000 7f523edb5700 1 -- 172.21.15.67:0/3620857876 shutdown_connections 2024-04-22T23:34:14.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.350+0000 7f523edb5700 1 -- 172.21.15.67:0/3620857876 wait complete. 2024-04-22T23:34:14.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.351+0000 7f523edb5700 1 Processor -- start 2024-04-22T23:34:14.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.351+0000 7f523edb5700 1 -- start start 2024-04-22T23:34:14.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.351+0000 7f523edb5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5238068580 0x7f52380ffcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:14.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.351+0000 7f523edb5700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5238106cd0 0x7f5238100230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:14.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.351+0000 7f523edb5700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5238100770 con 0x7f5238068580 2024-04-22T23:34:14.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.351+0000 7f523edb5700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52381008b0 con 0x7f5238106cd0 2024-04-22T23:34:14.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.351+0000 7f523cb51700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5238068580 0x7f52380ffcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:14.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.351+0000 7f523cb51700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5238068580 0x7f52380ffcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53184/0 (socket says 172.21.15.67:53184) 2024-04-22T23:34:14.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.351+0000 7f523cb51700 1 -- 172.21.15.67:0/1995536559 learned_addr learned my addr 172.21.15.67:0/1995536559 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:34:14.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.351+0000 7f5237fff700 1 --2- 172.21.15.67:0/1995536559 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5238106cd0 0x7f5238100230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:14.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.352+0000 7f523cb51700 1 -- 172.21.15.67:0/1995536559 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5238106cd0 msgr2=0x7f5238100230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:14.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.352+0000 7f523cb51700 1 --2- 172.21.15.67:0/1995536559 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5238106cd0 0x7f5238100230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:14.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.352+0000 7f523cb51700 1 -- 172.21.15.67:0/1995536559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f522c0095f0 con 0x7f5238068580 2024-04-22T23:34:14.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.352+0000 7f523cb51700 1 --2- 172.21.15.67:0/1995536559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5238068580 0x7f52380ffcf0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f522c0092e0 tx=0x7f522c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:34:14.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.352+0000 7f5235ffb700 1 -- 172.21.15.67:0/1995536559 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f522c01d070 con 0x7f5238068580 2024-04-22T23:34:14.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.352+0000 7f523edb5700 1 -- 172.21.15.67:0/1995536559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5238100b30 con 0x7f5238068580 2024-04-22T23:34:14.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.352+0000 7f5235ffb700 1 -- 172.21.15.67:0/1995536559 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f522c005e00 con 0x7f5238068580 2024-04-22T23:34:14.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.352+0000 7f5235ffb700 1 -- 172.21.15.67:0/1995536559 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f522c017400 con 0x7f5238068580 2024-04-22T23:34:14.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.353+0000 7f523edb5700 1 -- 172.21.15.67:0/1995536559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52381a2d90 con 0x7f5238068580 2024-04-22T23:34:14.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.353+0000 7f523edb5700 1 -- 172.21.15.67:0/1995536559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52381091f0 con 0x7f5238068580 2024-04-22T23:34:14.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.354+0000 7f5235ffb700 1 -- 172.21.15.67:0/1995536559 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f522c00f6f0 con 0x7f5238068580 2024-04-22T23:34:14.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.354+0000 7f5235ffb700 1 --2- 172.21.15.67:0/1995536559 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f522806ecc0 0x7f5228071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:14.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.355+0000 7f5235ffb700 1 -- 172.21.15.67:0/1995536559 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f522c090480 con 0x7f5238068580 2024-04-22T23:34:14.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.355+0000 7f5237fff700 1 --2- 172.21.15.67:0/1995536559 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f522806ecc0 0x7f5228071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:14.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.355+0000 7f5237fff700 1 --2- 172.21.15.67:0/1995536559 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f522806ecc0 0x7f5228071180 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f52381014e0 tx=0x7f522400b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:34:14.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.359+0000 7f5235ffb700 1 -- 172.21.15.67:0/1995536559 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f522c0590c0 con 0x7f5238068580 2024-04-22T23:34:14.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.592+0000 7f523edb5700 1 -- 172.21.15.67:0/1995536559 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f52380611d0 con 0x7f522806ecc0 2024-04-22T23:34:14.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.593+0000 7f5235ffb700 1 -- 172.21.15.67:0/1995536559 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f52380611d0 con 0x7f522806ecc0 2024-04-22T23:34:14.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.595+0000 7f523edb5700 1 -- 172.21.15.67:0/1995536559 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f522806ecc0 msgr2=0x7f5228071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:14.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.595+0000 7f523edb5700 1 --2- 172.21.15.67:0/1995536559 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f522806ecc0 0x7f5228071180 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f52381014e0 tx=0x7f522400b410 comp rx=0 tx=0).stop 2024-04-22T23:34:14.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.595+0000 7f523edb5700 1 -- 172.21.15.67:0/1995536559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5238068580 msgr2=0x7f52380ffcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:14.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.595+0000 7f523edb5700 1 --2- 172.21.15.67:0/1995536559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5238068580 0x7f52380ffcf0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f522c0092e0 tx=0x7f522c00bd70 comp rx=0 tx=0).stop 2024-04-22T23:34:14.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.595+0000 7f523edb5700 1 -- 172.21.15.67:0/1995536559 shutdown_connections 2024-04-22T23:34:14.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.595+0000 7f523edb5700 1 --2- 172.21.15.67:0/1995536559 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f522806ecc0 0x7f5228071180 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:14.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.595+0000 7f523edb5700 1 --2- 172.21.15.67:0/1995536559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5238068580 0x7f52380ffcf0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:14.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.595+0000 7f523edb5700 1 --2- 172.21.15.67:0/1995536559 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5238106cd0 0x7f5238100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:14.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.595+0000 7f523edb5700 1 -- 172.21.15.67:0/1995536559 >> 172.21.15.67:0/1995536559 conn(0x7f5238075670 msgr2=0x7f52380fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:34:14.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.595+0000 7f523edb5700 1 -- 172.21.15.67:0/1995536559 shutdown_connections 2024-04-22T23:34:14.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.595+0000 7f523edb5700 1 -- 172.21.15.67:0/1995536559 wait complete. 2024-04-22T23:34:14.601 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:34:14.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.704+0000 7f8e2be6c700 1 -- 172.21.15.67:0/1143820302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e24100ea0 msgr2=0x7f8e24101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:14.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.704+0000 7f8e2be6c700 1 --2- 172.21.15.67:0/1143820302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e24100ea0 0x7f8e24101280 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8e20009910 tx=0x7f8e20009c20 comp rx=0 tx=0).stop 2024-04-22T23:34:14.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.704+0000 7f8e2be6c700 1 -- 172.21.15.67:0/1143820302 shutdown_connections 2024-04-22T23:34:14.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.704+0000 7f8e2be6c700 1 --2- 172.21.15.67:0/1143820302 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8e24101850 0x7f8e241058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:14.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.704+0000 7f8e2be6c700 1 --2- 172.21.15.67:0/1143820302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e24100ea0 0x7f8e24101280 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:14.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.704+0000 7f8e2be6c700 1 -- 172.21.15.67:0/1143820302 >> 172.21.15.67:0/1143820302 conn(0x7f8e240fc750 msgr2=0x7f8e240feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:34:14.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.705+0000 7f8e2be6c700 1 -- 172.21.15.67:0/1143820302 shutdown_connections 2024-04-22T23:34:14.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.705+0000 7f8e2be6c700 1 -- 172.21.15.67:0/1143820302 wait complete. 2024-04-22T23:34:14.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.705+0000 7f8e2be6c700 1 Processor -- start 2024-04-22T23:34:14.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.705+0000 7f8e2be6c700 1 -- start start 2024-04-22T23:34:14.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.705+0000 7f8e2be6c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e24100ea0 0x7f8e24196990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:14.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.705+0000 7f8e2be6c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8e24101850 0x7f8e24196ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:14.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.705+0000 7f8e2be6c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e24197560 con 0x7f8e24100ea0 2024-04-22T23:34:14.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.705+0000 7f8e2be6c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2419b370 con 0x7f8e24101850 2024-04-22T23:34:14.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.706+0000 7f8e29c08700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e24100ea0 0x7f8e24196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:14.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.706+0000 7f8e29c08700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e24100ea0 0x7f8e24196990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53194/0 (socket says 172.21.15.67:53194) 2024-04-22T23:34:14.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.706+0000 7f8e29c08700 1 -- 172.21.15.67:0/2678890726 learned_addr learned my addr 172.21.15.67:0/2678890726 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:34:14.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.706+0000 7f8e29407700 1 --2- 172.21.15.67:0/2678890726 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8e24101850 0x7f8e24196ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:14.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.706+0000 7f8e29c08700 1 -- 172.21.15.67:0/2678890726 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8e24101850 msgr2=0x7f8e24196ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:14.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.706+0000 7f8e29c08700 1 --2- 172.21.15.67:0/2678890726 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8e24101850 0x7f8e24196ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:14.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.706+0000 7f8e29c08700 1 -- 172.21.15.67:0/2678890726 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e200095f0 con 0x7f8e24100ea0 2024-04-22T23:34:14.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.706+0000 7f8e29407700 1 --2- 172.21.15.67:0/2678890726 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8e24101850 0x7f8e24196ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:34:14.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.706+0000 7f8e29c08700 1 --2- 172.21.15.67:0/2678890726 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e24100ea0 0x7f8e24196990 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8e20006010 tx=0x7f8e2000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:34:14.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.706+0000 7f8e1affd700 1 -- 172.21.15.67:0/2678890726 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e2001d070 con 0x7f8e24100ea0 2024-04-22T23:34:14.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.706+0000 7f8e1affd700 1 -- 172.21.15.67:0/2678890726 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8e20004fb0 con 0x7f8e24100ea0 2024-04-22T23:34:14.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.706+0000 7f8e2be6c700 1 -- 172.21.15.67:0/2678890726 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e2419b5f0 con 0x7f8e24100ea0 2024-04-22T23:34:14.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.706+0000 7f8e1affd700 1 -- 172.21.15.67:0/2678890726 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e20022ea0 con 0x7f8e24100ea0 2024-04-22T23:34:14.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.706+0000 7f8e2be6c700 1 -- 172.21.15.67:0/2678890726 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e2419bae0 con 0x7f8e24100ea0 2024-04-22T23:34:14.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.707+0000 7f8e2be6c700 1 -- 172.21.15.67:0/2678890726 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e24109240 con 0x7f8e24100ea0 2024-04-22T23:34:14.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.708+0000 7f8e1affd700 1 -- 172.21.15.67:0/2678890726 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8e20022470 con 0x7f8e24100ea0 2024-04-22T23:34:14.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.708+0000 7f8e1affd700 1 --2- 172.21.15.67:0/2678890726 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8e1006ecc0 0x7f8e10071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:14.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.708+0000 7f8e1affd700 1 -- 172.21.15.67:0/2678890726 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f8e200904c0 con 0x7f8e24100ea0 2024-04-22T23:34:14.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.708+0000 7f8e29407700 1 --2- 172.21.15.67:0/2678890726 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8e1006ecc0 0x7f8e10071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:14.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.709+0000 7f8e29407700 1 --2- 172.21.15.67:0/2678890726 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8e1006ecc0 0x7f8e10071180 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8e24197fb0 tx=0x7f8e14008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:34:14.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.713+0000 7f8e1affd700 1 -- 172.21.15.67:0/2678890726 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f8e2005cd60 con 0x7f8e24100ea0 2024-04-22T23:34:14.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.943+0000 7f8e2be6c700 1 -- 172.21.15.67:0/2678890726 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e24197d60 con 0x7f8e1006ecc0 2024-04-22T23:34:14.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.944+0000 7f8e1affd700 1 -- 172.21.15.67:0/2678890726 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f8e24197d60 con 0x7f8e1006ecc0 2024-04-22T23:34:14.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.946+0000 7f8e2be6c700 1 -- 172.21.15.67:0/2678890726 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8e1006ecc0 msgr2=0x7f8e10071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:14.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.946+0000 7f8e2be6c700 1 --2- 172.21.15.67:0/2678890726 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8e1006ecc0 0x7f8e10071180 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8e24197fb0 tx=0x7f8e14008040 comp rx=0 tx=0).stop 2024-04-22T23:34:14.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.946+0000 7f8e2be6c700 1 -- 172.21.15.67:0/2678890726 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e24100ea0 msgr2=0x7f8e24196990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:14.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.946+0000 7f8e2be6c700 1 --2- 172.21.15.67:0/2678890726 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e24100ea0 0x7f8e24196990 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8e20006010 tx=0x7f8e2000be20 comp rx=0 tx=0).stop 2024-04-22T23:34:14.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.946+0000 7f8e2be6c700 1 -- 172.21.15.67:0/2678890726 shutdown_connections 2024-04-22T23:34:14.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.946+0000 7f8e2be6c700 1 --2- 172.21.15.67:0/2678890726 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8e1006ecc0 0x7f8e10071180 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:14.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.946+0000 7f8e2be6c700 1 --2- 172.21.15.67:0/2678890726 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e24100ea0 0x7f8e24196990 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:14.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.946+0000 7f8e2be6c700 1 --2- 172.21.15.67:0/2678890726 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8e24101850 0x7f8e24196ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:14.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.946+0000 7f8e2be6c700 1 -- 172.21.15.67:0/2678890726 >> 172.21.15.67:0/2678890726 conn(0x7f8e240fc750 msgr2=0x7f8e240fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:34:14.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.946+0000 7f8e2be6c700 1 -- 172.21.15.67:0/2678890726 shutdown_connections 2024-04-22T23:34:14.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:14.946+0000 7f8e2be6c700 1 -- 172.21.15.67:0/2678890726 wait complete. 2024-04-22T23:34:15.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.054+0000 7fa0c95a9700 1 -- 172.21.15.67:0/2771147548 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c4105bf0 msgr2=0x7fa0c4105fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:15.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.054+0000 7fa0c95a9700 1 --2- 172.21.15.67:0/2771147548 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c4105bf0 0x7fa0c4105fd0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8009910 tx=0x7fa0b8009c20 comp rx=0 tx=0).stop 2024-04-22T23:34:15.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.054+0000 7fa0c95a9700 1 -- 172.21.15.67:0/2771147548 shutdown_connections 2024-04-22T23:34:15.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.054+0000 7fa0c95a9700 1 --2- 172.21.15.67:0/2771147548 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0c40ffb40 0x7fa0c40fffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:15.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.054+0000 7fa0c95a9700 1 --2- 172.21.15.67:0/2771147548 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c4105bf0 0x7fa0c4105fd0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:15.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.054+0000 7fa0c95a9700 1 -- 172.21.15.67:0/2771147548 >> 172.21.15.67:0/2771147548 conn(0x7fa0c40fb610 msgr2=0x7fa0c40fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:34:15.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.054+0000 7fa0c95a9700 1 -- 172.21.15.67:0/2771147548 shutdown_connections 2024-04-22T23:34:15.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.054+0000 7fa0c95a9700 1 -- 172.21.15.67:0/2771147548 wait complete. 2024-04-22T23:34:15.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.055+0000 7fa0c95a9700 1 Processor -- start 2024-04-22T23:34:15.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.055+0000 7fa0c95a9700 1 -- start start 2024-04-22T23:34:15.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.055+0000 7fa0c95a9700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0c40ffb40 0x7fa0c41954b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:15.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.055+0000 7fa0c95a9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c4105bf0 0x7fa0c41959f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:15.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.055+0000 7fa0c95a9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c4195f30 con 0x7fa0c4105bf0 2024-04-22T23:34:15.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.055+0000 7fa0c95a9700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c41960a0 con 0x7fa0c40ffb40 2024-04-22T23:34:15.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.055+0000 7fa0c27fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c4105bf0 0x7fa0c41959f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:15.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.055+0000 7fa0c27fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c4105bf0 0x7fa0c41959f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53210/0 (socket says 172.21.15.67:53210) 2024-04-22T23:34:15.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.055+0000 7fa0c27fc700 1 -- 172.21.15.67:0/1170073133 learned_addr learned my addr 172.21.15.67:0/1170073133 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:34:15.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.055+0000 7fa0c2ffd700 1 --2- 172.21.15.67:0/1170073133 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0c40ffb40 0x7fa0c41954b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:15.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.056+0000 7fa0c27fc700 1 -- 172.21.15.67:0/1170073133 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0c40ffb40 msgr2=0x7fa0c41954b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:15.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.056+0000 7fa0c27fc700 1 --2- 172.21.15.67:0/1170073133 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0c40ffb40 0x7fa0c41954b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:15.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.056+0000 7fa0c27fc700 1 -- 172.21.15.67:0/1170073133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b80095f0 con 0x7fa0c4105bf0 2024-04-22T23:34:15.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.056+0000 7fa0c27fc700 1 --2- 172.21.15.67:0/1170073133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c4105bf0 0x7fa0c41959f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa0b400c8a0 tx=0x7fa0b400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:34:15.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.056+0000 7fa0abfff700 1 -- 172.21.15.67:0/1170073133 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0b40078e0 con 0x7fa0c4105bf0 2024-04-22T23:34:15.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.056+0000 7fa0abfff700 1 -- 172.21.15.67:0/1170073133 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa0b400f450 con 0x7fa0c4105bf0 2024-04-22T23:34:15.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.056+0000 7fa0c95a9700 1 -- 172.21.15.67:0/1170073133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0c41a2b20 con 0x7fa0c4105bf0 2024-04-22T23:34:15.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.056+0000 7fa0abfff700 1 -- 172.21.15.67:0/1170073133 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0b400e5e0 con 0x7fa0c4105bf0 2024-04-22T23:34:15.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.056+0000 7fa0c95a9700 1 -- 172.21.15.67:0/1170073133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0c41a2e80 con 0x7fa0c4105bf0 2024-04-22T23:34:15.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.057+0000 7fa0c95a9700 1 -- 172.21.15.67:0/1170073133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0c4108020 con 0x7fa0c4105bf0 2024-04-22T23:34:15.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.058+0000 7fa0abfff700 1 -- 172.21.15.67:0/1170073133 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa0b400f5c0 con 0x7fa0c4105bf0 2024-04-22T23:34:15.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.058+0000 7fa0abfff700 1 --2- 172.21.15.67:0/1170073133 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa0ac06ecc0 0x7fa0ac071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:15.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.058+0000 7fa0abfff700 1 -- 172.21.15.67:0/1170073133 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fa0b408ef50 con 0x7fa0c4105bf0 2024-04-22T23:34:15.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.058+0000 7fa0c2ffd700 1 --2- 172.21.15.67:0/1170073133 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa0ac06ecc0 0x7fa0ac071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:15.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.059+0000 7fa0c2ffd700 1 --2- 172.21.15.67:0/1170073133 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa0ac06ecc0 0x7fa0ac071180 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8006010 tx=0x7fa0b801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:34:15.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.063+0000 7fa0abfff700 1 -- 172.21.15.67:0/1170073133 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fa0b40c4150 con 0x7fa0c4105bf0 2024-04-22T23:34:15.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.281+0000 7fa0c95a9700 1 -- 172.21.15.67:0/1170073133 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0c40611d0 con 0x7fa0ac06ecc0 2024-04-22T23:34:15.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.292+0000 7fa0abfff700 1 -- 172.21.15.67:0/1170073133 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7fa0c40611d0 con 0x7fa0ac06ecc0 2024-04-22T23:34:15.288 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:34:15.288 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (7m) 73s ago 19m 25.5M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:34:15.288 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (19m) 73s ago 19m 9114k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:34:15.288 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (18m) 56s ago 18m 9042k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:34:15.288 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (5m) 73s ago 19m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:34:15.288 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (5m) 56s ago 18m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:34:15.288 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (7m) 73s ago 18m 79.3M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:34:15.288 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (99s) 73s ago 13m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:34:15.288 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (79s) 73s ago 13m 12.6M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:34:15.288 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (59s) 56s ago 13m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:34:15.288 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (13m) 56s ago 13m 12.2M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:34:15.288 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (9m) 73s ago 20m 511M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:34:15.288 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (8m) 56s ago 18m 453M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:34:15.288 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (6m) 73s ago 20m 55.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:34:15.289 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (6m) 56s ago 18m 46.0M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:34:15.289 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (8m) 73s ago 19m 15.4M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:34:15.289 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (8m) 56s ago 18m 16.1M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:34:15.289 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (4m) 73s ago 17m 99.2M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:34:15.289 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (4m) 73s ago 17m 127M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:34:15.289 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (3m) 73s ago 16m 79.9M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:34:15.289 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (3m) 56s ago 16m 97.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:34:15.289 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (2m) 56s ago 15m 82.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:34:15.289 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (118s) 56s ago 15m 77.6M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:34:15.289 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (8m) 73s ago 18m 73.9M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:34:15.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.294+0000 7fa0c95a9700 1 -- 172.21.15.67:0/1170073133 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa0ac06ecc0 msgr2=0x7fa0ac071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:15.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.294+0000 7fa0c95a9700 1 --2- 172.21.15.67:0/1170073133 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa0ac06ecc0 0x7fa0ac071180 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8006010 tx=0x7fa0b801a040 comp rx=0 tx=0).stop 2024-04-22T23:34:15.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.294+0000 7fa0c95a9700 1 -- 172.21.15.67:0/1170073133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c4105bf0 msgr2=0x7fa0c41959f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:15.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.294+0000 7fa0c95a9700 1 --2- 172.21.15.67:0/1170073133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c4105bf0 0x7fa0c41959f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa0b400c8a0 tx=0x7fa0b400cbb0 comp rx=0 tx=0).stop 2024-04-22T23:34:15.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.294+0000 7fa0c95a9700 1 -- 172.21.15.67:0/1170073133 shutdown_connections 2024-04-22T23:34:15.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.294+0000 7fa0c95a9700 1 --2- 172.21.15.67:0/1170073133 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa0ac06ecc0 0x7fa0ac071180 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:15.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.294+0000 7fa0c95a9700 1 --2- 172.21.15.67:0/1170073133 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0c40ffb40 0x7fa0c41954b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:15.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.294+0000 7fa0c95a9700 1 --2- 172.21.15.67:0/1170073133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c4105bf0 0x7fa0c41959f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:15.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.294+0000 7fa0c95a9700 1 -- 172.21.15.67:0/1170073133 >> 172.21.15.67:0/1170073133 conn(0x7fa0c40fb610 msgr2=0x7fa0c40fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:34:15.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.295+0000 7fa0c95a9700 1 -- 172.21.15.67:0/1170073133 shutdown_connections 2024-04-22T23:34:15.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.295+0000 7fa0c95a9700 1 -- 172.21.15.67:0/1170073133 wait complete. 2024-04-22T23:34:15.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.401+0000 7f6635bf1700 1 -- 172.21.15.67:0/4112217202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6630101850 msgr2=0x7f66301058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:15.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.401+0000 7f6635bf1700 1 --2- 172.21.15.67:0/4112217202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6630101850 0x7f66301058a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6618009910 tx=0x7f6618009c20 comp rx=0 tx=0).stop 2024-04-22T23:34:15.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.401+0000 7f6635bf1700 1 -- 172.21.15.67:0/4112217202 shutdown_connections 2024-04-22T23:34:15.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.401+0000 7f6635bf1700 1 --2- 172.21.15.67:0/4112217202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6630101850 0x7f66301058a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:15.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.401+0000 7f6635bf1700 1 --2- 172.21.15.67:0/4112217202 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6630100ea0 0x7f6630101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:15.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.401+0000 7f6635bf1700 1 -- 172.21.15.67:0/4112217202 >> 172.21.15.67:0/4112217202 conn(0x7f66300fc750 msgr2=0x7f66300feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:34:15.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.401+0000 7f6635bf1700 1 -- 172.21.15.67:0/4112217202 shutdown_connections 2024-04-22T23:34:15.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.401+0000 7f6635bf1700 1 -- 172.21.15.67:0/4112217202 wait complete. 2024-04-22T23:34:15.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.401+0000 7f6635bf1700 1 Processor -- start 2024-04-22T23:34:15.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.402+0000 7f6635bf1700 1 -- start start 2024-04-22T23:34:15.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.402+0000 7f6635bf1700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6630100ea0 0x7f663019c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:15.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.402+0000 7f6635bf1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6630101850 0x7f663019cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:15.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.402+0000 7f6635bf1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f663019d2d0 con 0x7f6630101850 2024-04-22T23:34:15.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.402+0000 7f6635bf1700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6630196780 con 0x7f6630100ea0 2024-04-22T23:34:15.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.402+0000 7f662effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6630101850 0x7f663019cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:15.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.402+0000 7f662effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6630101850 0x7f663019cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53226/0 (socket says 172.21.15.67:53226) 2024-04-22T23:34:15.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.402+0000 7f662effd700 1 -- 172.21.15.67:0/1687716355 learned_addr learned my addr 172.21.15.67:0/1687716355 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:34:15.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.402+0000 7f662f7fe700 1 --2- 172.21.15.67:0/1687716355 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6630100ea0 0x7f663019c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:15.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.402+0000 7f662effd700 1 -- 172.21.15.67:0/1687716355 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6630100ea0 msgr2=0x7f663019c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:15.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.402+0000 7f662effd700 1 --2- 172.21.15.67:0/1687716355 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6630100ea0 0x7f663019c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:15.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.402+0000 7f662effd700 1 -- 172.21.15.67:0/1687716355 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66180095f0 con 0x7f6630101850 2024-04-22T23:34:15.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.402+0000 7f662effd700 1 --2- 172.21.15.67:0/1687716355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6630101850 0x7f663019cc40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f66180092e0 tx=0x7f661800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:34:15.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.403+0000 7f662cff9700 1 -- 172.21.15.67:0/1687716355 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f661801d070 con 0x7f6630101850 2024-04-22T23:34:15.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.403+0000 7f6635bf1700 1 -- 172.21.15.67:0/1687716355 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6630196a00 con 0x7f6630101850 2024-04-22T23:34:15.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.403+0000 7f662cff9700 1 -- 172.21.15.67:0/1687716355 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6618005dc0 con 0x7f6630101850 2024-04-22T23:34:15.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.403+0000 7f6635bf1700 1 -- 172.21.15.67:0/1687716355 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6630196ef0 con 0x7f6630101850 2024-04-22T23:34:15.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.403+0000 7f662cff9700 1 -- 172.21.15.67:0/1687716355 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6618017400 con 0x7f6630101850 2024-04-22T23:34:15.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.404+0000 7f6635bf1700 1 -- 172.21.15.67:0/1687716355 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66301092a0 con 0x7f6630101850 2024-04-22T23:34:15.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.404+0000 7f662cff9700 1 -- 172.21.15.67:0/1687716355 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f661800f6f0 con 0x7f6630101850 2024-04-22T23:34:15.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.405+0000 7f662cff9700 1 --2- 172.21.15.67:0/1687716355 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f661c06ecc0 0x7f661c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:15.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.405+0000 7f662cff9700 1 -- 172.21.15.67:0/1687716355 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f66180903c0 con 0x7f6630101850 2024-04-22T23:34:15.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.405+0000 7f662f7fe700 1 --2- 172.21.15.67:0/1687716355 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f661c06ecc0 0x7f661c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:15.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.405+0000 7f662f7fe700 1 --2- 172.21.15.67:0/1687716355 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f661c06ecc0 0x7f661c071180 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6620009550 tx=0x7f66200091b0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:34:15.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.410+0000 7f662cff9700 1 -- 172.21.15.67:0/1687716355 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f661805cb30 con 0x7f6630101850 2024-04-22T23:34:15.539 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:15 smithi067 ceph-mon[112438]: pgmap v236: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:15.539 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:15 smithi067 ceph-mon[112438]: from='client.34484 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:34:15.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.709+0000 7f6635bf1700 1 -- 172.21.15.67:0/1687716355 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f663004f2e0 con 0x7f6630101850 2024-04-22T23:34:15.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.710+0000 7f662cff9700 1 -- 172.21.15.67:0/1687716355 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f661805c6c0 con 0x7f6630101850 2024-04-22T23:34:15.705 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:34:15.705 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:34:15.705 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:34:15.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:34:15.706 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:34:15.706 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:34:15.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:34:15.706 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:34:15.706 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:34:15.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:34:15.706 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:34:15.706 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:34:15.706 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 3 2024-04-22T23:34:15.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:34:15.706 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:34:15.706 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:34:15.706 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 13 2024-04-22T23:34:15.707 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:34:15.707 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:34:15.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.712+0000 7f6635bf1700 1 -- 172.21.15.67:0/1687716355 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f661c06ecc0 msgr2=0x7f661c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:15.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.712+0000 7f6635bf1700 1 --2- 172.21.15.67:0/1687716355 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f661c06ecc0 0x7f661c071180 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6620009550 tx=0x7f66200091b0 comp rx=0 tx=0).stop 2024-04-22T23:34:15.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.712+0000 7f6635bf1700 1 -- 172.21.15.67:0/1687716355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6630101850 msgr2=0x7f663019cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:15.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.712+0000 7f6635bf1700 1 --2- 172.21.15.67:0/1687716355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6630101850 0x7f663019cc40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f66180092e0 tx=0x7f661800bd70 comp rx=0 tx=0).stop 2024-04-22T23:34:15.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.712+0000 7f6635bf1700 1 -- 172.21.15.67:0/1687716355 shutdown_connections 2024-04-22T23:34:15.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.712+0000 7f6635bf1700 1 --2- 172.21.15.67:0/1687716355 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f661c06ecc0 0x7f661c071180 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:15.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.712+0000 7f6635bf1700 1 --2- 172.21.15.67:0/1687716355 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6630100ea0 0x7f663019c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:15.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.712+0000 7f6635bf1700 1 --2- 172.21.15.67:0/1687716355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6630101850 0x7f663019cc40 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:15.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.712+0000 7f6635bf1700 1 -- 172.21.15.67:0/1687716355 >> 172.21.15.67:0/1687716355 conn(0x7f66300fc750 msgr2=0x7f66300fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:34:15.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.712+0000 7f6635bf1700 1 -- 172.21.15.67:0/1687716355 shutdown_connections 2024-04-22T23:34:15.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.712+0000 7f6635bf1700 1 -- 172.21.15.67:0/1687716355 wait complete. 2024-04-22T23:34:15.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.820+0000 7fe0de52c700 1 -- 172.21.15.67:0/2556415101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0d80ffb40 msgr2=0x7fe0d80fffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:15.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.820+0000 7fe0de52c700 1 --2- 172.21.15.67:0/2556415101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0d80ffb40 0x7fe0d80fffc0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc009910 tx=0x7fe0cc009c20 comp rx=0 tx=0).stop 2024-04-22T23:34:15.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.820+0000 7fe0de52c700 1 -- 172.21.15.67:0/2556415101 shutdown_connections 2024-04-22T23:34:15.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.820+0000 7fe0de52c700 1 --2- 172.21.15.67:0/2556415101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0d80ffb40 0x7fe0d80fffc0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:15.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.820+0000 7fe0de52c700 1 --2- 172.21.15.67:0/2556415101 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0d8105bf0 0x7fe0d8105fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:15.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.820+0000 7fe0de52c700 1 -- 172.21.15.67:0/2556415101 >> 172.21.15.67:0/2556415101 conn(0x7fe0d80fb610 msgr2=0x7fe0d80fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:34:15.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.820+0000 7fe0de52c700 1 -- 172.21.15.67:0/2556415101 shutdown_connections 2024-04-22T23:34:15.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.820+0000 7fe0de52c700 1 -- 172.21.15.67:0/2556415101 wait complete. 2024-04-22T23:34:15.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.821+0000 7fe0de52c700 1 Processor -- start 2024-04-22T23:34:15.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.821+0000 7fe0de52c700 1 -- start start 2024-04-22T23:34:15.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.821+0000 7fe0de52c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0d80ffb40 0x7fe0d819c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:15.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.821+0000 7fe0de52c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0d8105bf0 0x7fe0d819cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:15.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.821+0000 7fe0de52c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d819d1b0 con 0x7fe0d80ffb40 2024-04-22T23:34:15.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.821+0000 7fe0de52c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d8196660 con 0x7fe0d8105bf0 2024-04-22T23:34:15.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.821+0000 7fe0d7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0d80ffb40 0x7fe0d819c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:15.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.821+0000 7fe0d7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0d80ffb40 0x7fe0d819c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53248/0 (socket says 172.21.15.67:53248) 2024-04-22T23:34:15.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.821+0000 7fe0d7fff700 1 -- 172.21.15.67:0/382428067 learned_addr learned my addr 172.21.15.67:0/382428067 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:34:15.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.821+0000 7fe0d77fe700 1 --2- 172.21.15.67:0/382428067 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0d8105bf0 0x7fe0d819cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:15.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.821+0000 7fe0d7fff700 1 -- 172.21.15.67:0/382428067 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0d8105bf0 msgr2=0x7fe0d819cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:15.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.821+0000 7fe0d7fff700 1 --2- 172.21.15.67:0/382428067 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0d8105bf0 0x7fe0d819cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:15.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.821+0000 7fe0d7fff700 1 -- 172.21.15.67:0/382428067 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0cc0095f0 con 0x7fe0d80ffb40 2024-04-22T23:34:15.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.821+0000 7fe0d77fe700 1 --2- 172.21.15.67:0/382428067 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0d8105bf0 0x7fe0d819cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:34:15.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.822+0000 7fe0d7fff700 1 --2- 172.21.15.67:0/382428067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0d80ffb40 0x7fe0d819c5e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe0c800e920 tx=0x7fe0c800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:34:15.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.822+0000 7fe0d57fa700 1 -- 172.21.15.67:0/382428067 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0c8009790 con 0x7fe0d80ffb40 2024-04-22T23:34:15.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.822+0000 7fe0d57fa700 1 -- 172.21.15.67:0/382428067 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe0c800cba0 con 0x7fe0d80ffb40 2024-04-22T23:34:15.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.822+0000 7fe0de52c700 1 -- 172.21.15.67:0/382428067 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0d8196940 con 0x7fe0d80ffb40 2024-04-22T23:34:15.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.822+0000 7fe0d57fa700 1 -- 172.21.15.67:0/382428067 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0c8018650 con 0x7fe0d80ffb40 2024-04-22T23:34:15.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.822+0000 7fe0de52c700 1 -- 172.21.15.67:0/382428067 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0d8196e90 con 0x7fe0d80ffb40 2024-04-22T23:34:15.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.823+0000 7fe0de52c700 1 -- 172.21.15.67:0/382428067 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0d80679c0 con 0x7fe0d80ffb40 2024-04-22T23:34:15.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.824+0000 7fe0d57fa700 1 -- 172.21.15.67:0/382428067 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe0c800cd10 con 0x7fe0d80ffb40 2024-04-22T23:34:15.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.824+0000 7fe0d57fa700 1 --2- 172.21.15.67:0/382428067 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe0c006ecc0 0x7fe0c0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:15.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.824+0000 7fe0d57fa700 1 -- 172.21.15.67:0/382428067 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fe0c8014070 con 0x7fe0d80ffb40 2024-04-22T23:34:15.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.824+0000 7fe0d77fe700 1 --2- 172.21.15.67:0/382428067 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe0c006ecc0 0x7fe0c0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:15.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.825+0000 7fe0d77fe700 1 --2- 172.21.15.67:0/382428067 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe0c006ecc0 0x7fe0c0071180 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc00b760 tx=0x7fe0cc009f70 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:34:15.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:15.829+0000 7fe0d57fa700 1 -- 172.21.15.67:0/382428067 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe0c8054150 con 0x7fe0d80ffb40 2024-04-22T23:34:15.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:15 smithi134 ceph-mon[85630]: pgmap v236: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:15.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:15 smithi134 ceph-mon[85630]: from='client.34484 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:34:16.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.088+0000 7fe0de52c700 1 -- 172.21.15.67:0/382428067 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe0d8197b70 con 0x7fe0d80ffb40 2024-04-22T23:34:16.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.089+0000 7fe0d57fa700 1 -- 172.21.15.67:0/382428067 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1871 (secure 0 0 0) 0x7fe0c805b8b0 con 0x7fe0d80ffb40 2024-04-22T23:34:16.084 INFO:teuthology.orchestra.run.smithi067.stdout:e31 2024-04-22T23:34:16.084 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:34:16.084 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:34:16.085 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:34:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:34:16.085 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:34:16.085 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:34:16.085 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:34:16.085 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:34:16.085 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:34:16.085 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:34:16.085 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:34:16.085 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:34:16.085 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:34:16.085 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:34:16.086 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:34:16.086 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:34:16.086 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:34:16.086 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:34:16.086 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:34:16.086 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:34:16.086 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:34:16.086 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:34:16.086 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:34:16.086 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:34:16.086 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:34:16.086 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:34:16.086 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:34:16.086 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:34:16.087 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:34:16.087 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:34:16.087 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:34:16.087 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:34:16.087 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:34:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:34:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:34:16.087 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:34:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:34:16.087 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:34:16.087 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:34:16.087 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44259} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1827288036,v1:172.21.15.134:6827/1827288036] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:34:16.087 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:34:16.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.091+0000 7fe0de52c700 1 -- 172.21.15.67:0/382428067 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe0c006ecc0 msgr2=0x7fe0c0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:16.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.091+0000 7fe0de52c700 1 --2- 172.21.15.67:0/382428067 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe0c006ecc0 0x7fe0c0071180 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc00b760 tx=0x7fe0cc009f70 comp rx=0 tx=0).stop 2024-04-22T23:34:16.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.091+0000 7fe0de52c700 1 -- 172.21.15.67:0/382428067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0d80ffb40 msgr2=0x7fe0d819c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:16.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.091+0000 7fe0de52c700 1 --2- 172.21.15.67:0/382428067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0d80ffb40 0x7fe0d819c5e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe0c800e920 tx=0x7fe0c800ece0 comp rx=0 tx=0).stop 2024-04-22T23:34:16.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.091+0000 7fe0de52c700 1 -- 172.21.15.67:0/382428067 shutdown_connections 2024-04-22T23:34:16.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.091+0000 7fe0de52c700 1 --2- 172.21.15.67:0/382428067 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe0c006ecc0 0x7fe0c0071180 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:16.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.091+0000 7fe0de52c700 1 --2- 172.21.15.67:0/382428067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0d80ffb40 0x7fe0d819c5e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:16.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.091+0000 7fe0de52c700 1 --2- 172.21.15.67:0/382428067 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0d8105bf0 0x7fe0d819cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:16.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.091+0000 7fe0de52c700 1 -- 172.21.15.67:0/382428067 >> 172.21.15.67:0/382428067 conn(0x7fe0d80fb610 msgr2=0x7fe0d8068eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:34:16.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.091+0000 7fe0de52c700 1 -- 172.21.15.67:0/382428067 shutdown_connections 2024-04-22T23:34:16.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.091+0000 7fe0de52c700 1 -- 172.21.15.67:0/382428067 wait complete. 2024-04-22T23:34:16.089 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 31 2024-04-22T23:34:16.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.200+0000 7fc70a5ae700 1 -- 172.21.15.67:0/2151507060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7040ff5a0 msgr2=0x7fc7040ff980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:16.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.200+0000 7fc70a5ae700 1 --2- 172.21.15.67:0/2151507060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7040ff5a0 0x7fc7040ff980 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec009910 tx=0x7fc6ec009c20 comp rx=0 tx=0).stop 2024-04-22T23:34:16.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.200+0000 7fc70a5ae700 1 -- 172.21.15.67:0/2151507060 shutdown_connections 2024-04-22T23:34:16.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.200+0000 7fc70a5ae700 1 --2- 172.21.15.67:0/2151507060 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc7040fff50 0x7fc704105840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:16.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.200+0000 7fc70a5ae700 1 --2- 172.21.15.67:0/2151507060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7040ff5a0 0x7fc7040ff980 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:16.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.200+0000 7fc70a5ae700 1 -- 172.21.15.67:0/2151507060 >> 172.21.15.67:0/2151507060 conn(0x7fc704074e30 msgr2=0x7fc704075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:34:16.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.200+0000 7fc70a5ae700 1 -- 172.21.15.67:0/2151507060 shutdown_connections 2024-04-22T23:34:16.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.200+0000 7fc70a5ae700 1 -- 172.21.15.67:0/2151507060 wait complete. 2024-04-22T23:34:16.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.201+0000 7fc70a5ae700 1 Processor -- start 2024-04-22T23:34:16.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.201+0000 7fc70a5ae700 1 -- start start 2024-04-22T23:34:16.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.201+0000 7fc70a5ae700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7040ff5a0 0x7fc70419c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:16.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.201+0000 7fc70a5ae700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc7040fff50 0x7fc70419cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:16.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.201+0000 7fc70a5ae700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc70419d2d0 con 0x7fc7040ff5a0 2024-04-22T23:34:16.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.201+0000 7fc70a5ae700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc704196780 con 0x7fc7040fff50 2024-04-22T23:34:16.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.202+0000 7fc703fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7040ff5a0 0x7fc70419c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:16.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.202+0000 7fc703fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7040ff5a0 0x7fc70419c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53272/0 (socket says 172.21.15.67:53272) 2024-04-22T23:34:16.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.202+0000 7fc703fff700 1 -- 172.21.15.67:0/2119462058 learned_addr learned my addr 172.21.15.67:0/2119462058 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:34:16.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.202+0000 7fc7037fe700 1 --2- 172.21.15.67:0/2119462058 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc7040fff50 0x7fc70419cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:16.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.202+0000 7fc703fff700 1 -- 172.21.15.67:0/2119462058 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc7040fff50 msgr2=0x7fc70419cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:16.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.202+0000 7fc703fff700 1 --2- 172.21.15.67:0/2119462058 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc7040fff50 0x7fc70419cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:16.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.202+0000 7fc703fff700 1 -- 172.21.15.67:0/2119462058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6ec0095f0 con 0x7fc7040ff5a0 2024-04-22T23:34:16.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.202+0000 7fc703fff700 1 --2- 172.21.15.67:0/2119462058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7040ff5a0 0x7fc70419c700 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec00f690 tx=0x7fc6ec00f6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:34:16.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.202+0000 7fc7017fa700 1 -- 172.21.15.67:0/2119462058 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6ec01d070 con 0x7fc7040ff5a0 2024-04-22T23:34:16.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.202+0000 7fc7017fa700 1 -- 172.21.15.67:0/2119462058 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc6ec0050d0 con 0x7fc7040ff5a0 2024-04-22T23:34:16.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.202+0000 7fc70a5ae700 1 -- 172.21.15.67:0/2119462058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc704196a00 con 0x7fc7040ff5a0 2024-04-22T23:34:16.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.202+0000 7fc7017fa700 1 -- 172.21.15.67:0/2119462058 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6ec022470 con 0x7fc7040ff5a0 2024-04-22T23:34:16.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.203+0000 7fc70a5ae700 1 -- 172.21.15.67:0/2119462058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc704196ef0 con 0x7fc7040ff5a0 2024-04-22T23:34:16.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.203+0000 7fc70a5ae700 1 -- 172.21.15.67:0/2119462058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc704109270 con 0x7fc7040ff5a0 2024-04-22T23:34:16.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.204+0000 7fc7017fa700 1 -- 172.21.15.67:0/2119462058 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc6ec003780 con 0x7fc7040ff5a0 2024-04-22T23:34:16.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.205+0000 7fc7017fa700 1 --2- 172.21.15.67:0/2119462058 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc6f006ecc0 0x7fc6f0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:16.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.205+0000 7fc7017fa700 1 -- 172.21.15.67:0/2119462058 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fc6ec090450 con 0x7fc7040ff5a0 2024-04-22T23:34:16.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.205+0000 7fc7037fe700 1 --2- 172.21.15.67:0/2119462058 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc6f006ecc0 0x7fc6f0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:16.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.205+0000 7fc7037fe700 1 --2- 172.21.15.67:0/2119462058 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc6f006ecc0 0x7fc6f0071180 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc704197bf0 tx=0x7fc6f4009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:34:16.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.210+0000 7fc7017fa700 1 -- 172.21.15.67:0/2119462058 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc6ec05cc70 con 0x7fc7040ff5a0 2024-04-22T23:34:16.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.442+0000 7fc70a5ae700 1 -- 172.21.15.67:0/2119462058 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc704197ce0 con 0x7fc6f006ecc0 2024-04-22T23:34:16.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.443+0000 7fc7017fa700 1 -- 172.21.15.67:0/2119462058 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc704197ce0 con 0x7fc6f006ecc0 2024-04-22T23:34:16.439 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:34:16.439 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:34:16.439 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:34:16.439 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:34:16.440 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:34:16.440 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:34:16.440 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:34:16.440 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:34:16.440 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:34:16.440 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:34:16.440 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T23:34:16.441 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:34:16.441 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:34:16.441 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:34:16.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.446+0000 7fc70a5ae700 1 -- 172.21.15.67:0/2119462058 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc6f006ecc0 msgr2=0x7fc6f0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:16.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.446+0000 7fc70a5ae700 1 --2- 172.21.15.67:0/2119462058 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc6f006ecc0 0x7fc6f0071180 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc704197bf0 tx=0x7fc6f4009260 comp rx=0 tx=0).stop 2024-04-22T23:34:16.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.446+0000 7fc70a5ae700 1 -- 172.21.15.67:0/2119462058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7040ff5a0 msgr2=0x7fc70419c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:16.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.446+0000 7fc70a5ae700 1 --2- 172.21.15.67:0/2119462058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7040ff5a0 0x7fc70419c700 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec00f690 tx=0x7fc6ec00f6c0 comp rx=0 tx=0).stop 2024-04-22T23:34:16.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.446+0000 7fc70a5ae700 1 -- 172.21.15.67:0/2119462058 shutdown_connections 2024-04-22T23:34:16.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.446+0000 7fc70a5ae700 1 --2- 172.21.15.67:0/2119462058 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc6f006ecc0 0x7fc6f0071180 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:16.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.446+0000 7fc70a5ae700 1 --2- 172.21.15.67:0/2119462058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7040ff5a0 0x7fc70419c700 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:16.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.446+0000 7fc70a5ae700 1 --2- 172.21.15.67:0/2119462058 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc7040fff50 0x7fc70419cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:16.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.446+0000 7fc70a5ae700 1 -- 172.21.15.67:0/2119462058 >> 172.21.15.67:0/2119462058 conn(0x7fc704074e30 msgr2=0x7fc7040fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:34:16.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.446+0000 7fc70a5ae700 1 -- 172.21.15.67:0/2119462058 shutdown_connections 2024-04-22T23:34:16.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.446+0000 7fc70a5ae700 1 -- 172.21.15.67:0/2119462058 wait complete. 2024-04-22T23:34:16.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.554+0000 7fe65b459700 1 -- 172.21.15.67:0/2493359811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe654073130 msgr2=0x7fe654073510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:16.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.554+0000 7fe65b459700 1 --2- 172.21.15.67:0/2493359811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe654073130 0x7fe654073510 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fe644009910 tx=0x7fe644009c20 comp rx=0 tx=0).stop 2024-04-22T23:34:16.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.554+0000 7fe65b459700 1 -- 172.21.15.67:0/2493359811 shutdown_connections 2024-04-22T23:34:16.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.554+0000 7fe65b459700 1 --2- 172.21.15.67:0/2493359811 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe654073a50 0x7fe654111710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:16.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.554+0000 7fe65b459700 1 --2- 172.21.15.67:0/2493359811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe654073130 0x7fe654073510 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:16.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.554+0000 7fe65b459700 1 -- 172.21.15.67:0/2493359811 >> 172.21.15.67:0/2493359811 conn(0x7fe6540fc750 msgr2=0x7fe6540feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:34:16.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.554+0000 7fe65b459700 1 -- 172.21.15.67:0/2493359811 shutdown_connections 2024-04-22T23:34:16.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.554+0000 7fe65b459700 1 -- 172.21.15.67:0/2493359811 wait complete. 2024-04-22T23:34:16.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.555+0000 7fe65b459700 1 Processor -- start 2024-04-22T23:34:16.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.555+0000 7fe65b459700 1 -- start start 2024-04-22T23:34:16.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.555+0000 7fe65b459700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe654073130 0x7fe65419ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:16.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.555+0000 7fe65b459700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe654073a50 0x7fe65419d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:16.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.555+0000 7fe65b459700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe65419d990 con 0x7fe654073130 2024-04-22T23:34:16.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.555+0000 7fe65b459700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe65419dad0 con 0x7fe654073a50 2024-04-22T23:34:16.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.555+0000 7fe6591f5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe654073130 0x7fe65419ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:16.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.555+0000 7fe6591f5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe654073130 0x7fe65419ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53286/0 (socket says 172.21.15.67:53286) 2024-04-22T23:34:16.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.555+0000 7fe6591f5700 1 -- 172.21.15.67:0/836423020 learned_addr learned my addr 172.21.15.67:0/836423020 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:34:16.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.555+0000 7fe6589f4700 1 --2- 172.21.15.67:0/836423020 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe654073a50 0x7fe65419d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:16.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.555+0000 7fe6591f5700 1 -- 172.21.15.67:0/836423020 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe654073a50 msgr2=0x7fe65419d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:16.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.555+0000 7fe6591f5700 1 --2- 172.21.15.67:0/836423020 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe654073a50 0x7fe65419d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:16.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.555+0000 7fe6591f5700 1 -- 172.21.15.67:0/836423020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6440095f0 con 0x7fe654073130 2024-04-22T23:34:16.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.556+0000 7fe6591f5700 1 --2- 172.21.15.67:0/836423020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe654073130 0x7fe65419ce00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe644006010 tx=0x7fe64400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:34:16.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.556+0000 7fe64a7fc700 1 -- 172.21.15.67:0/836423020 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe64401d070 con 0x7fe654073130 2024-04-22T23:34:16.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.556+0000 7fe64a7fc700 1 -- 172.21.15.67:0/836423020 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe644004fb0 con 0x7fe654073130 2024-04-22T23:34:16.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.556+0000 7fe65b459700 1 -- 172.21.15.67:0/836423020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6541a1920 con 0x7fe654073130 2024-04-22T23:34:16.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.556+0000 7fe64a7fc700 1 -- 172.21.15.67:0/836423020 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe644022ea0 con 0x7fe654073130 2024-04-22T23:34:16.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.556+0000 7fe65b459700 1 -- 172.21.15.67:0/836423020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6541a1e70 con 0x7fe654073130 2024-04-22T23:34:16.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.557+0000 7fe65b459700 1 -- 172.21.15.67:0/836423020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe65410ee90 con 0x7fe654073130 2024-04-22T23:34:16.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.557+0000 7fe64a7fc700 1 -- 172.21.15.67:0/836423020 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe644022470 con 0x7fe654073130 2024-04-22T23:34:16.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.558+0000 7fe64a7fc700 1 --2- 172.21.15.67:0/836423020 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe64006ecc0 0x7fe640071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:16.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.558+0000 7fe64a7fc700 1 -- 172.21.15.67:0/836423020 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fe644090490 con 0x7fe654073130 2024-04-22T23:34:16.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.558+0000 7fe6589f4700 1 --2- 172.21.15.67:0/836423020 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe64006ecc0 0x7fe640071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:16.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.558+0000 7fe6589f4700 1 --2- 172.21.15.67:0/836423020 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe64006ecc0 0x7fe640071180 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe650005ea0 tx=0x7fe650005e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:34:16.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.563+0000 7fe64a7fc700 1 -- 172.21.15.67:0/836423020 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe644059200 con 0x7fe654073130 2024-04-22T23:34:16.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:16 smithi067 ceph-mon[112438]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:34:16.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:16 smithi067 ceph-mon[112438]: from='client.34492 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:34:16.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:16 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1687716355' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:34:16.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:16 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/382428067' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:34:16.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.858+0000 7fe65b459700 1 -- 172.21.15.67:0/836423020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe65404f2e0 con 0x7fe654073130 2024-04-22T23:34:16.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.859+0000 7fe64a7fc700 1 -- 172.21.15.67:0/836423020 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fe64405c8c0 con 0x7fe654073130 2024-04-22T23:34:16.854 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:34:16.854 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:34:16.854 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:34:16.854 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:34:16.855 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:34:16.855 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:34:16.855 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:34:16.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.861+0000 7fe65b459700 1 -- 172.21.15.67:0/836423020 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe64006ecc0 msgr2=0x7fe640071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:16.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.861+0000 7fe65b459700 1 --2- 172.21.15.67:0/836423020 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe64006ecc0 0x7fe640071180 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe650005ea0 tx=0x7fe650005e30 comp rx=0 tx=0).stop 2024-04-22T23:34:16.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.861+0000 7fe65b459700 1 -- 172.21.15.67:0/836423020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe654073130 msgr2=0x7fe65419ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:16.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.861+0000 7fe65b459700 1 --2- 172.21.15.67:0/836423020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe654073130 0x7fe65419ce00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe644006010 tx=0x7fe64400be20 comp rx=0 tx=0).stop 2024-04-22T23:34:16.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.861+0000 7fe65b459700 1 -- 172.21.15.67:0/836423020 shutdown_connections 2024-04-22T23:34:16.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.861+0000 7fe65b459700 1 --2- 172.21.15.67:0/836423020 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe64006ecc0 0x7fe640071180 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:16.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.861+0000 7fe65b459700 1 --2- 172.21.15.67:0/836423020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe654073130 0x7fe65419ce00 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:16.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.861+0000 7fe65b459700 1 --2- 172.21.15.67:0/836423020 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe654073a50 0x7fe65419d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:16.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.861+0000 7fe65b459700 1 -- 172.21.15.67:0/836423020 >> 172.21.15.67:0/836423020 conn(0x7fe6540fc750 msgr2=0x7fe654103220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:34:16.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.861+0000 7fe65b459700 1 -- 172.21.15.67:0/836423020 shutdown_connections 2024-04-22T23:34:16.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:16.861+0000 7fe65b459700 1 -- 172.21.15.67:0/836423020 wait complete. 2024-04-22T23:34:16.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:16 smithi134 ceph-mon[85630]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:34:16.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:16 smithi134 ceph-mon[85630]: from='client.34492 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:34:16.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:16 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1687716355' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:34:16.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:16 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/382428067' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:34:17.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:17 smithi067 ceph-mon[112438]: pgmap v237: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:17.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:17 smithi067 ceph-mon[112438]: from='client.34504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:34:17.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:17 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/836423020' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:34:17.933 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:17 smithi134 ceph-mon[85630]: pgmap v237: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:17.933 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:17 smithi134 ceph-mon[85630]: from='client.34504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:34:17.933 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:17 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/836423020' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:34:19.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:19 smithi067 ceph-mon[112438]: pgmap v238: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:19.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:19 smithi134 ceph-mon[85630]: pgmap v238: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:21.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:21 smithi067 ceph-mon[112438]: pgmap v239: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:21.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:21 smithi134 ceph-mon[85630]: pgmap v239: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:23.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:23 smithi067 ceph-mon[112438]: pgmap v240: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:23.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:23 smithi134 ceph-mon[85630]: pgmap v240: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:25.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:25 smithi067 ceph-mon[112438]: pgmap v241: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:25.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:25 smithi134 ceph-mon[85630]: pgmap v241: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:26.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:34:26.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:34:27.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:27 smithi067 ceph-mon[112438]: pgmap v242: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:27.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:27 smithi134 ceph-mon[85630]: pgmap v242: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:29.790 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:29 smithi067 ceph-mon[112438]: pgmap v243: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:29.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:29 smithi134 ceph-mon[85630]: pgmap v243: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:31.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:31 smithi067 ceph-mon[112438]: pgmap v244: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:31.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:31 smithi134 ceph-mon[85630]: pgmap v244: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:33.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:33 smithi067 ceph-mon[112438]: pgmap v245: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:33.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:33 smithi134 ceph-mon[85630]: pgmap v245: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:35.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:35 smithi134 ceph-mon[85630]: pgmap v246: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:36.039 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:35 smithi067 ceph-mon[112438]: pgmap v246: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:37.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:37 smithi134 ceph-mon[85630]: pgmap v247: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:38.039 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:37 smithi067 ceph-mon[112438]: pgmap v247: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:39.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:39 smithi134 ceph-mon[85630]: pgmap v248: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:40.039 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:39 smithi067 ceph-mon[112438]: pgmap v248: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:41.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:41 smithi134 ceph-mon[85630]: pgmap v249: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:41.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:34:42.039 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:41 smithi067 ceph-mon[112438]: pgmap v249: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:42.039 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:34:43.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:43 smithi134 ceph-mon[85630]: pgmap v250: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:44.039 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:43 smithi067 ceph-mon[112438]: pgmap v250: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:45.556 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:45 smithi067 ceph-mon[112438]: pgmap v251: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:45.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:45 smithi134 ceph-mon[85630]: pgmap v251: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:46.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:46.983+0000 7f2d7d25a700 1 -- 172.21.15.67:0/629323694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d78105bd0 msgr2=0x7f2d78105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:46.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:46.983+0000 7f2d7d25a700 1 --2- 172.21.15.67:0/629323694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d78105bd0 0x7f2d78105fb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2d68009910 tx=0x7f2d68009c20 comp rx=0 tx=0).stop 2024-04-22T23:34:46.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:46.983+0000 7f2d7d25a700 1 -- 172.21.15.67:0/629323694 shutdown_connections 2024-04-22T23:34:46.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:46.983+0000 7f2d7d25a700 1 --2- 172.21.15.67:0/629323694 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d780ffc20 0x7f2d78100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:46.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:46.983+0000 7f2d7d25a700 1 --2- 172.21.15.67:0/629323694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d78105bd0 0x7f2d78105fb0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:46.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:46.983+0000 7f2d7d25a700 1 -- 172.21.15.67:0/629323694 >> 172.21.15.67:0/629323694 conn(0x7f2d78075670 msgr2=0x7f2d78075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:34:46.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:46.983+0000 7f2d7d25a700 1 -- 172.21.15.67:0/629323694 shutdown_connections 2024-04-22T23:34:46.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:46.983+0000 7f2d7d25a700 1 -- 172.21.15.67:0/629323694 wait complete. 2024-04-22T23:34:46.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:46.984+0000 7f2d7d25a700 1 Processor -- start 2024-04-22T23:34:46.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:46.985+0000 7f2d7d25a700 1 -- start start 2024-04-22T23:34:46.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:46.985+0000 7f2d7d25a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d780ffc20 0x7f2d7819e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:46.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:46.985+0000 7f2d7d25a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d78105bd0 0x7f2d7819ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:46.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:46.985+0000 7f2d7d25a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7819f3f0 con 0x7f2d780ffc20 2024-04-22T23:34:46.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:46.985+0000 7f2d7d25a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d781988a0 con 0x7f2d78105bd0 2024-04-22T23:34:46.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:46.985+0000 7f2d76d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d780ffc20 0x7f2d7819e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:46.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:46.985+0000 7f2d76d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d780ffc20 0x7f2d7819e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59936/0 (socket says 172.21.15.67:59936) 2024-04-22T23:34:46.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:46.985+0000 7f2d76d9d700 1 -- 172.21.15.67:0/2225236657 learned_addr learned my addr 172.21.15.67:0/2225236657 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:34:46.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:46.985+0000 7f2d7659c700 1 --2- 172.21.15.67:0/2225236657 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d78105bd0 0x7f2d7819ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:46.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:46.985+0000 7f2d76d9d700 1 -- 172.21.15.67:0/2225236657 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d78105bd0 msgr2=0x7f2d7819ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:46.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:46.985+0000 7f2d76d9d700 1 --2- 172.21.15.67:0/2225236657 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d78105bd0 0x7f2d7819ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:46.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:46.985+0000 7f2d76d9d700 1 -- 172.21.15.67:0/2225236657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d680095f0 con 0x7f2d780ffc20 2024-04-22T23:34:46.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:46.985+0000 7f2d76d9d700 1 --2- 172.21.15.67:0/2225236657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d780ffc20 0x7f2d7819e820 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2d68006010 tx=0x7f2d6800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:34:46.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:46.986+0000 7f2d5ffff700 1 -- 172.21.15.67:0/2225236657 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d6801d070 con 0x7f2d780ffc20 2024-04-22T23:34:46.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:46.986+0000 7f2d5ffff700 1 -- 172.21.15.67:0/2225236657 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2d68005dc0 con 0x7f2d780ffc20 2024-04-22T23:34:46.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:46.986+0000 7f2d7d25a700 1 -- 172.21.15.67:0/2225236657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d78198b20 con 0x7f2d780ffc20 2024-04-22T23:34:46.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:46.986+0000 7f2d5ffff700 1 -- 172.21.15.67:0/2225236657 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d68017400 con 0x7f2d780ffc20 2024-04-22T23:34:46.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:46.986+0000 7f2d7d25a700 1 -- 172.21.15.67:0/2225236657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d78199010 con 0x7f2d780ffc20 2024-04-22T23:34:46.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:46.987+0000 7f2d7d25a700 1 -- 172.21.15.67:0/2225236657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d7804f2e0 con 0x7f2d780ffc20 2024-04-22T23:34:46.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:46.987+0000 7f2d5ffff700 1 -- 172.21.15.67:0/2225236657 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2d68003a60 con 0x7f2d780ffc20 2024-04-22T23:34:46.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:46.988+0000 7f2d5ffff700 1 --2- 172.21.15.67:0/2225236657 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2d6006ecc0 0x7f2d60071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:46.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:46.988+0000 7f2d5ffff700 1 -- 172.21.15.67:0/2225236657 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f2d68090460 con 0x7f2d780ffc20 2024-04-22T23:34:46.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:46.988+0000 7f2d7659c700 1 --2- 172.21.15.67:0/2225236657 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2d6006ecc0 0x7f2d60071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:46.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:46.988+0000 7f2d7659c700 1 --2- 172.21.15.67:0/2225236657 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2d6006ecc0 0x7f2d60071180 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2d780ffa40 tx=0x7f2d6c00b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:34:46.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:46.993+0000 7f2d5ffff700 1 -- 172.21.15.67:0/2225236657 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f2d6805cbd0 con 0x7f2d780ffc20 2024-04-22T23:34:47.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.226+0000 7f2d7d25a700 1 -- 172.21.15.67:0/2225236657 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2d781992f0 con 0x7f2d6006ecc0 2024-04-22T23:34:47.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.227+0000 7f2d5ffff700 1 -- 172.21.15.67:0/2225236657 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f2d781992f0 con 0x7f2d6006ecc0 2024-04-22T23:34:47.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.229+0000 7f2d7d25a700 1 -- 172.21.15.67:0/2225236657 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2d6006ecc0 msgr2=0x7f2d60071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:47.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.229+0000 7f2d7d25a700 1 --2- 172.21.15.67:0/2225236657 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2d6006ecc0 0x7f2d60071180 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2d780ffa40 tx=0x7f2d6c00b410 comp rx=0 tx=0).stop 2024-04-22T23:34:47.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.229+0000 7f2d7d25a700 1 -- 172.21.15.67:0/2225236657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d780ffc20 msgr2=0x7f2d7819e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:47.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.229+0000 7f2d7d25a700 1 --2- 172.21.15.67:0/2225236657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d780ffc20 0x7f2d7819e820 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2d68006010 tx=0x7f2d6800bd70 comp rx=0 tx=0).stop 2024-04-22T23:34:47.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.229+0000 7f2d7d25a700 1 -- 172.21.15.67:0/2225236657 shutdown_connections 2024-04-22T23:34:47.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.229+0000 7f2d7d25a700 1 --2- 172.21.15.67:0/2225236657 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2d6006ecc0 0x7f2d60071180 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:47.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.229+0000 7f2d7d25a700 1 --2- 172.21.15.67:0/2225236657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d780ffc20 0x7f2d7819e820 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:47.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.229+0000 7f2d7d25a700 1 --2- 172.21.15.67:0/2225236657 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d78105bd0 0x7f2d7819ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:47.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.229+0000 7f2d7d25a700 1 -- 172.21.15.67:0/2225236657 >> 172.21.15.67:0/2225236657 conn(0x7f2d78075670 msgr2=0x7f2d780686e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:34:47.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.229+0000 7f2d7d25a700 1 -- 172.21.15.67:0/2225236657 shutdown_connections 2024-04-22T23:34:47.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.229+0000 7f2d7d25a700 1 -- 172.21.15.67:0/2225236657 wait complete. 2024-04-22T23:34:47.235 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:34:47.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.338+0000 7f67e749b700 1 -- 172.21.15.67:0/1112349746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67e0101850 msgr2=0x7f67e01058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:47.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.338+0000 7f67e749b700 1 --2- 172.21.15.67:0/1112349746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67e0101850 0x7f67e01058a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f67d0009910 tx=0x7f67d0009c20 comp rx=0 tx=0).stop 2024-04-22T23:34:47.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.339+0000 7f67e749b700 1 -- 172.21.15.67:0/1112349746 shutdown_connections 2024-04-22T23:34:47.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.339+0000 7f67e749b700 1 --2- 172.21.15.67:0/1112349746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67e0101850 0x7f67e01058a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:47.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.339+0000 7f67e749b700 1 --2- 172.21.15.67:0/1112349746 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67e0100ea0 0x7f67e0101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:47.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.339+0000 7f67e749b700 1 -- 172.21.15.67:0/1112349746 >> 172.21.15.67:0/1112349746 conn(0x7f67e00fc750 msgr2=0x7f67e00feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:34:47.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.339+0000 7f67e749b700 1 -- 172.21.15.67:0/1112349746 shutdown_connections 2024-04-22T23:34:47.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.339+0000 7f67e749b700 1 -- 172.21.15.67:0/1112349746 wait complete. 2024-04-22T23:34:47.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.339+0000 7f67e749b700 1 Processor -- start 2024-04-22T23:34:47.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.339+0000 7f67e749b700 1 -- start start 2024-04-22T23:34:47.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.339+0000 7f67e749b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67e0100ea0 0x7f67e0073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:47.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.339+0000 7f67e749b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67e0101850 0x7f67e00735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:47.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.340+0000 7f67e749b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e0073b10 con 0x7f67e0101850 2024-04-22T23:34:47.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.340+0000 7f67e749b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e0073c50 con 0x7f67e0100ea0 2024-04-22T23:34:47.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.340+0000 7f67e4a36700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67e0101850 0x7f67e00735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:47.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.340+0000 7f67e4a36700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67e0101850 0x7f67e00735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59956/0 (socket says 172.21.15.67:59956) 2024-04-22T23:34:47.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.340+0000 7f67e4a36700 1 -- 172.21.15.67:0/234206032 learned_addr learned my addr 172.21.15.67:0/234206032 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:34:47.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.340+0000 7f67e5237700 1 --2- 172.21.15.67:0/234206032 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67e0100ea0 0x7f67e0073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:47.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.340+0000 7f67e4a36700 1 -- 172.21.15.67:0/234206032 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67e0100ea0 msgr2=0x7f67e0073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:47.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.340+0000 7f67e4a36700 1 --2- 172.21.15.67:0/234206032 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67e0100ea0 0x7f67e0073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:47.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.340+0000 7f67e4a36700 1 -- 172.21.15.67:0/234206032 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d00095f0 con 0x7f67e0101850 2024-04-22T23:34:47.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.340+0000 7f67e4a36700 1 --2- 172.21.15.67:0/234206032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67e0101850 0x7f67e00735d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f67d000bda0 tx=0x7f67d000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:34:47.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.340+0000 7f67d67fc700 1 -- 172.21.15.67:0/234206032 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67d001d070 con 0x7f67e0101850 2024-04-22T23:34:47.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.340+0000 7f67d67fc700 1 -- 172.21.15.67:0/234206032 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f67d0004fb0 con 0x7f67e0101850 2024-04-22T23:34:47.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.340+0000 7f67e749b700 1 -- 172.21.15.67:0/234206032 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67e0073ed0 con 0x7f67e0101850 2024-04-22T23:34:47.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.340+0000 7f67d67fc700 1 -- 172.21.15.67:0/234206032 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67d000f460 con 0x7f67e0101850 2024-04-22T23:34:47.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.340+0000 7f67e749b700 1 -- 172.21.15.67:0/234206032 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67e01a2de0 con 0x7f67e0101850 2024-04-22T23:34:47.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.341+0000 7f67e749b700 1 -- 172.21.15.67:0/234206032 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67e01092a0 con 0x7f67e0101850 2024-04-22T23:34:47.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.342+0000 7f67d67fc700 1 -- 172.21.15.67:0/234206032 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f67d0022800 con 0x7f67e0101850 2024-04-22T23:34:47.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.342+0000 7f67d67fc700 1 --2- 172.21.15.67:0/234206032 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f67cc06ecc0 0x7f67cc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:47.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.342+0000 7f67d67fc700 1 -- 172.21.15.67:0/234206032 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f67d0090460 con 0x7f67e0101850 2024-04-22T23:34:47.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.342+0000 7f67e5237700 1 --2- 172.21.15.67:0/234206032 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f67cc06ecc0 0x7f67cc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:47.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.343+0000 7f67e5237700 1 --2- 172.21.15.67:0/234206032 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f67cc06ecc0 0x7f67cc071180 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f67dc005ea0 tx=0x7f67dc005e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:34:47.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.347+0000 7f67d67fc700 1 -- 172.21.15.67:0/234206032 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f67d00592f0 con 0x7f67e0101850 2024-04-22T23:34:47.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.578+0000 7f67e749b700 1 -- 172.21.15.67:0/234206032 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f67e00746a0 con 0x7f67cc06ecc0 2024-04-22T23:34:47.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.580+0000 7f67d67fc700 1 -- 172.21.15.67:0/234206032 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f67e00746a0 con 0x7f67cc06ecc0 2024-04-22T23:34:47.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.582+0000 7f67e749b700 1 -- 172.21.15.67:0/234206032 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f67cc06ecc0 msgr2=0x7f67cc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:47.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.582+0000 7f67e749b700 1 --2- 172.21.15.67:0/234206032 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f67cc06ecc0 0x7f67cc071180 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f67dc005ea0 tx=0x7f67dc005e30 comp rx=0 tx=0).stop 2024-04-22T23:34:47.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.582+0000 7f67e749b700 1 -- 172.21.15.67:0/234206032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67e0101850 msgr2=0x7f67e00735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:47.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.582+0000 7f67e749b700 1 --2- 172.21.15.67:0/234206032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67e0101850 0x7f67e00735d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f67d000bda0 tx=0x7f67d000be80 comp rx=0 tx=0).stop 2024-04-22T23:34:47.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.582+0000 7f67e749b700 1 -- 172.21.15.67:0/234206032 shutdown_connections 2024-04-22T23:34:47.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.582+0000 7f67e749b700 1 --2- 172.21.15.67:0/234206032 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f67cc06ecc0 0x7f67cc071180 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:47.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.582+0000 7f67e749b700 1 --2- 172.21.15.67:0/234206032 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67e0100ea0 0x7f67e0073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:47.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.582+0000 7f67e749b700 1 --2- 172.21.15.67:0/234206032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67e0101850 0x7f67e00735d0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:47.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.582+0000 7f67e749b700 1 -- 172.21.15.67:0/234206032 >> 172.21.15.67:0/234206032 conn(0x7f67e00fc750 msgr2=0x7f67e00fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:34:47.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.582+0000 7f67e749b700 1 -- 172.21.15.67:0/234206032 shutdown_connections 2024-04-22T23:34:47.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.582+0000 7f67e749b700 1 -- 172.21.15.67:0/234206032 wait complete. 2024-04-22T23:34:47.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.688+0000 7f06147bf700 1 -- 172.21.15.67:0/2450903550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f060c1027e0 msgr2=0x7f060c10acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:47.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.688+0000 7f06147bf700 1 --2- 172.21.15.67:0/2450903550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f060c1027e0 0x7f060c10acd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f05fc009940 tx=0x7f05fc009c50 comp rx=0 tx=0).stop 2024-04-22T23:34:47.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.689+0000 7f06147bf700 1 -- 172.21.15.67:0/2450903550 shutdown_connections 2024-04-22T23:34:47.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.689+0000 7f06147bf700 1 --2- 172.21.15.67:0/2450903550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f060c1027e0 0x7f060c10acd0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:47.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.689+0000 7f06147bf700 1 --2- 172.21.15.67:0/2450903550 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f060c101ec0 0x7f060c1022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:47.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.689+0000 7f06147bf700 1 -- 172.21.15.67:0/2450903550 >> 172.21.15.67:0/2450903550 conn(0x7f060c0fb650 msgr2=0x7f060c0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:34:47.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.689+0000 7f06147bf700 1 -- 172.21.15.67:0/2450903550 shutdown_connections 2024-04-22T23:34:47.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.689+0000 7f06147bf700 1 -- 172.21.15.67:0/2450903550 wait complete. 2024-04-22T23:34:47.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.689+0000 7f06147bf700 1 Processor -- start 2024-04-22T23:34:47.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.689+0000 7f06147bf700 1 -- start start 2024-04-22T23:34:47.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.689+0000 7f06147bf700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f060c101ec0 0x7f060c19c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:47.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.689+0000 7f06147bf700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f060c1027e0 0x7f060c19cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:47.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.689+0000 7f06147bf700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060c19d230 con 0x7f060c1027e0 2024-04-22T23:34:47.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.689+0000 7f06147bf700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060c1966e0 con 0x7f060c101ec0 2024-04-22T23:34:47.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.690+0000 7f0611d5a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f060c1027e0 0x7f060c19cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:47.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.690+0000 7f0611d5a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f060c1027e0 0x7f060c19cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59968/0 (socket says 172.21.15.67:59968) 2024-04-22T23:34:47.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.690+0000 7f0611d5a700 1 -- 172.21.15.67:0/499549584 learned_addr learned my addr 172.21.15.67:0/499549584 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:34:47.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.690+0000 7f061255b700 1 --2- 172.21.15.67:0/499549584 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f060c101ec0 0x7f060c19c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:47.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.690+0000 7f0611d5a700 1 -- 172.21.15.67:0/499549584 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f060c101ec0 msgr2=0x7f060c19c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:47.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.690+0000 7f0611d5a700 1 --2- 172.21.15.67:0/499549584 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f060c101ec0 0x7f060c19c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:47.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.690+0000 7f0611d5a700 1 -- 172.21.15.67:0/499549584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05fc0095f0 con 0x7f060c1027e0 2024-04-22T23:34:47.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.690+0000 7f0611d5a700 1 --2- 172.21.15.67:0/499549584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f060c1027e0 0x7f060c19cba0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f05fc00b580 tx=0x7f05fc0056a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:34:47.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.690+0000 7f06037fe700 1 -- 172.21.15.67:0/499549584 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05fc01d070 con 0x7f060c1027e0 2024-04-22T23:34:47.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.690+0000 7f06037fe700 1 -- 172.21.15.67:0/499549584 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f05fc00f460 con 0x7f060c1027e0 2024-04-22T23:34:47.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.690+0000 7f06147bf700 1 -- 172.21.15.67:0/499549584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f060c196960 con 0x7f060c1027e0 2024-04-22T23:34:47.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.690+0000 7f06037fe700 1 -- 172.21.15.67:0/499549584 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05fc021640 con 0x7f060c1027e0 2024-04-22T23:34:47.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.690+0000 7f06147bf700 1 -- 172.21.15.67:0/499549584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f060c196e50 con 0x7f060c1027e0 2024-04-22T23:34:47.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.691+0000 7f06147bf700 1 -- 172.21.15.67:0/499549584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f060c108360 con 0x7f060c1027e0 2024-04-22T23:34:47.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.692+0000 7f06037fe700 1 -- 172.21.15.67:0/499549584 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f05fc00fa40 con 0x7f060c1027e0 2024-04-22T23:34:47.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.692+0000 7f06037fe700 1 --2- 172.21.15.67:0/499549584 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f05f806ecc0 0x7f05f8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:47.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.693+0000 7f061255b700 1 --2- 172.21.15.67:0/499549584 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f05f806ecc0 0x7f05f8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:47.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.693+0000 7f06037fe700 1 -- 172.21.15.67:0/499549584 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f05fc090160 con 0x7f060c1027e0 2024-04-22T23:34:47.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.693+0000 7f061255b700 1 --2- 172.21.15.67:0/499549584 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f05f806ecc0 0x7f05f8071180 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0608005950 tx=0x7f06080058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:34:47.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.698+0000 7f06037fe700 1 -- 172.21.15.67:0/499549584 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f05fc05c8f0 con 0x7f060c1027e0 2024-04-22T23:34:47.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.917+0000 7f06147bf700 1 -- 172.21.15.67:0/499549584 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f060c1970c0 con 0x7f05f806ecc0 2024-04-22T23:34:47.912 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:47 smithi067 ceph-mon[112438]: pgmap v252: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:47.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.930+0000 7f06037fe700 1 -- 172.21.15.67:0/499549584 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f060c1970c0 con 0x7f05f806ecc0 2024-04-22T23:34:47.925 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:34:47.925 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (8m) 106s ago 19m 25.5M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:34:47.925 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (20m) 106s ago 20m 9114k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:34:47.925 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (19m) 88s ago 19m 9042k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:34:47.925 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (5m) 106s ago 19m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:34:47.925 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (5m) 88s ago 18m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:34:47.925 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (8m) 106s ago 19m 79.3M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:34:47.925 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (2m) 106s ago 14m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:34:47.925 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (112s) 106s ago 14m 12.6M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:34:47.925 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (92s) 88s ago 14m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:34:47.926 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (14m) 88s ago 14m 12.2M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:34:47.926 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (10m) 106s ago 21m 511M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:34:47.926 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (9m) 88s ago 18m 453M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:34:47.926 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (7m) 106s ago 21m 55.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:34:47.926 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (6m) 88s ago 18m 46.0M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:34:47.926 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (9m) 106s ago 19m 15.4M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:34:47.926 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (9m) 88s ago 18m 16.1M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:34:47.926 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (5m) 106s ago 18m 99.2M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:34:47.926 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (4m) 106s ago 17m 127M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:34:47.926 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (4m) 106s ago 17m 79.9M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:34:47.926 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (3m) 88s ago 16m 97.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:34:47.926 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (3m) 88s ago 16m 82.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:34:47.926 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (2m) 88s ago 15m 77.6M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:34:47.926 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (8m) 106s ago 19m 73.9M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:34:47.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.932+0000 7f06147bf700 1 -- 172.21.15.67:0/499549584 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f05f806ecc0 msgr2=0x7f05f8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:47.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.932+0000 7f06147bf700 1 --2- 172.21.15.67:0/499549584 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f05f806ecc0 0x7f05f8071180 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0608005950 tx=0x7f06080058e0 comp rx=0 tx=0).stop 2024-04-22T23:34:47.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.932+0000 7f06147bf700 1 -- 172.21.15.67:0/499549584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f060c1027e0 msgr2=0x7f060c19cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:47.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.932+0000 7f06147bf700 1 --2- 172.21.15.67:0/499549584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f060c1027e0 0x7f060c19cba0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f05fc00b580 tx=0x7f05fc0056a0 comp rx=0 tx=0).stop 2024-04-22T23:34:47.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.932+0000 7f06147bf700 1 -- 172.21.15.67:0/499549584 shutdown_connections 2024-04-22T23:34:47.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.932+0000 7f06147bf700 1 --2- 172.21.15.67:0/499549584 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f05f806ecc0 0x7f05f8071180 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:47.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.932+0000 7f06147bf700 1 --2- 172.21.15.67:0/499549584 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f060c101ec0 0x7f060c19c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:47.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.932+0000 7f06147bf700 1 --2- 172.21.15.67:0/499549584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f060c1027e0 0x7f060c19cba0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:47.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.932+0000 7f06147bf700 1 -- 172.21.15.67:0/499549584 >> 172.21.15.67:0/499549584 conn(0x7f060c0fb650 msgr2=0x7f060c1054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:34:47.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.932+0000 7f06147bf700 1 -- 172.21.15.67:0/499549584 shutdown_connections 2024-04-22T23:34:47.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:47.932+0000 7f06147bf700 1 -- 172.21.15.67:0/499549584 wait complete. 2024-04-22T23:34:47.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:47 smithi134 ceph-mon[85630]: pgmap v252: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:48.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.038+0000 7fa20e4a2700 1 -- 172.21.15.67:0/3879682353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa208068580 msgr2=0x7fa2080689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:48.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.038+0000 7fa20e4a2700 1 --2- 172.21.15.67:0/3879682353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa208068580 0x7fa2080689e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc009910 tx=0x7fa1fc009c20 comp rx=0 tx=0).stop 2024-04-22T23:34:48.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.038+0000 7fa20e4a2700 1 -- 172.21.15.67:0/3879682353 shutdown_connections 2024-04-22T23:34:48.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.038+0000 7fa20e4a2700 1 --2- 172.21.15.67:0/3879682353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa208068580 0x7fa2080689e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:48.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.038+0000 7fa20e4a2700 1 --2- 172.21.15.67:0/3879682353 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa208106cd0 0x7fa2081070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:48.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.038+0000 7fa20e4a2700 1 -- 172.21.15.67:0/3879682353 >> 172.21.15.67:0/3879682353 conn(0x7fa208075670 msgr2=0x7fa208075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:34:48.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.038+0000 7fa20e4a2700 1 -- 172.21.15.67:0/3879682353 shutdown_connections 2024-04-22T23:34:48.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.038+0000 7fa20e4a2700 1 -- 172.21.15.67:0/3879682353 wait complete. 2024-04-22T23:34:48.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.039+0000 7fa20e4a2700 1 Processor -- start 2024-04-22T23:34:48.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.039+0000 7fa20e4a2700 1 -- start start 2024-04-22T23:34:48.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.039+0000 7fa20e4a2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa208068580 0x7fa20819c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:48.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.039+0000 7fa20e4a2700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa208106cd0 0x7fa20819cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:48.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.039+0000 7fa20e4a2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa20819d200 con 0x7fa208068580 2024-04-22T23:34:48.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.039+0000 7fa20e4a2700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2081966b0 con 0x7fa208106cd0 2024-04-22T23:34:48.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.039+0000 7fa207fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa208068580 0x7fa20819c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:48.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.039+0000 7fa207fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa208068580 0x7fa20819c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59976/0 (socket says 172.21.15.67:59976) 2024-04-22T23:34:48.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.039+0000 7fa207fff700 1 -- 172.21.15.67:0/184970647 learned_addr learned my addr 172.21.15.67:0/184970647 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:34:48.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.039+0000 7fa2077fe700 1 --2- 172.21.15.67:0/184970647 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa208106cd0 0x7fa20819cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:48.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.039+0000 7fa207fff700 1 -- 172.21.15.67:0/184970647 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa208106cd0 msgr2=0x7fa20819cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:48.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.039+0000 7fa207fff700 1 --2- 172.21.15.67:0/184970647 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa208106cd0 0x7fa20819cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:48.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.039+0000 7fa207fff700 1 -- 172.21.15.67:0/184970647 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1fc0095f0 con 0x7fa208068580 2024-04-22T23:34:48.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.040+0000 7fa207fff700 1 --2- 172.21.15.67:0/184970647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa208068580 0x7fa20819c630 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa1f800d900 tx=0x7fa1f800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:34:48.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.040+0000 7fa2057fa700 1 -- 172.21.15.67:0/184970647 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1f80041f0 con 0x7fa208068580 2024-04-22T23:34:48.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.040+0000 7fa2057fa700 1 -- 172.21.15.67:0/184970647 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa1f8004cf0 con 0x7fa208068580 2024-04-22T23:34:48.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.040+0000 7fa20e4a2700 1 -- 172.21.15.67:0/184970647 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa208196990 con 0x7fa208068580 2024-04-22T23:34:48.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.040+0000 7fa2057fa700 1 -- 172.21.15.67:0/184970647 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1f800b750 con 0x7fa208068580 2024-04-22T23:34:48.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.040+0000 7fa20e4a2700 1 -- 172.21.15.67:0/184970647 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa208196ee0 con 0x7fa208068580 2024-04-22T23:34:48.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.041+0000 7fa20e4a2700 1 -- 172.21.15.67:0/184970647 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2081091d0 con 0x7fa208068580 2024-04-22T23:34:48.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.042+0000 7fa2057fa700 1 -- 172.21.15.67:0/184970647 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa1f8004750 con 0x7fa208068580 2024-04-22T23:34:48.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.042+0000 7fa2057fa700 1 --2- 172.21.15.67:0/184970647 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa1f006ecc0 0x7fa1f0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:48.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.042+0000 7fa2077fe700 1 --2- 172.21.15.67:0/184970647 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa1f006ecc0 0x7fa1f0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:48.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.042+0000 7fa2057fa700 1 -- 172.21.15.67:0/184970647 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fa1f808eed0 con 0x7fa208068580 2024-04-22T23:34:48.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.043+0000 7fa2077fe700 1 --2- 172.21.15.67:0/184970647 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa1f006ecc0 0x7fa1f0071180 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa2080fe200 tx=0x7fa1fc01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:34:48.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.047+0000 7fa2057fa700 1 -- 172.21.15.67:0/184970647 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fa1f805b640 con 0x7fa208068580 2024-04-22T23:34:48.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.346+0000 7fa20e4a2700 1 -- 172.21.15.67:0/184970647 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa208197cc0 con 0x7fa208068580 2024-04-22T23:34:48.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.346+0000 7fa2057fa700 1 -- 172.21.15.67:0/184970647 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fa208197cc0 con 0x7fa208068580 2024-04-22T23:34:48.342 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:34:48.342 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:34:48.342 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:34:48.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:34:48.342 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:34:48.342 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:34:48.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:34:48.342 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:34:48.342 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:34:48.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:34:48.342 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:34:48.343 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:34:48.343 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 3 2024-04-22T23:34:48.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:34:48.343 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:34:48.343 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:34:48.343 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 13 2024-04-22T23:34:48.343 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:34:48.343 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:34:48.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.348+0000 7fa20e4a2700 1 -- 172.21.15.67:0/184970647 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa1f006ecc0 msgr2=0x7fa1f0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:48.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.348+0000 7fa20e4a2700 1 --2- 172.21.15.67:0/184970647 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa1f006ecc0 0x7fa1f0071180 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa2080fe200 tx=0x7fa1fc01a040 comp rx=0 tx=0).stop 2024-04-22T23:34:48.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.348+0000 7fa20e4a2700 1 -- 172.21.15.67:0/184970647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa208068580 msgr2=0x7fa20819c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:48.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.348+0000 7fa20e4a2700 1 --2- 172.21.15.67:0/184970647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa208068580 0x7fa20819c630 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa1f800d900 tx=0x7fa1f800dc10 comp rx=0 tx=0).stop 2024-04-22T23:34:48.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.348+0000 7fa20e4a2700 1 -- 172.21.15.67:0/184970647 shutdown_connections 2024-04-22T23:34:48.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.348+0000 7fa20e4a2700 1 --2- 172.21.15.67:0/184970647 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa1f006ecc0 0x7fa1f0071180 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:48.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.348+0000 7fa20e4a2700 1 --2- 172.21.15.67:0/184970647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa208068580 0x7fa20819c630 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:48.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.348+0000 7fa20e4a2700 1 --2- 172.21.15.67:0/184970647 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa208106cd0 0x7fa20819cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:48.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.348+0000 7fa20e4a2700 1 -- 172.21.15.67:0/184970647 >> 172.21.15.67:0/184970647 conn(0x7fa208075670 msgr2=0x7fa2080feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:34:48.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.348+0000 7fa20e4a2700 1 -- 172.21.15.67:0/184970647 shutdown_connections 2024-04-22T23:34:48.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.349+0000 7fa20e4a2700 1 -- 172.21.15.67:0/184970647 wait complete. 2024-04-22T23:34:48.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.455+0000 7fe0edccd700 1 -- 172.21.15.67:0/2847339303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0e80ffc20 msgr2=0x7fe0e8100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:48.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.455+0000 7fe0edccd700 1 --2- 172.21.15.67:0/2847339303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0e80ffc20 0x7fe0e8100080 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc009910 tx=0x7fe0dc009c20 comp rx=0 tx=0).stop 2024-04-22T23:34:48.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.456+0000 7fe0edccd700 1 -- 172.21.15.67:0/2847339303 shutdown_connections 2024-04-22T23:34:48.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.456+0000 7fe0edccd700 1 --2- 172.21.15.67:0/2847339303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0e80ffc20 0x7fe0e8100080 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:48.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.456+0000 7fe0edccd700 1 --2- 172.21.15.67:0/2847339303 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0e8105bd0 0x7fe0e8105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:48.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.456+0000 7fe0edccd700 1 -- 172.21.15.67:0/2847339303 >> 172.21.15.67:0/2847339303 conn(0x7fe0e8075670 msgr2=0x7fe0e8075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:34:48.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.456+0000 7fe0edccd700 1 -- 172.21.15.67:0/2847339303 shutdown_connections 2024-04-22T23:34:48.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.456+0000 7fe0edccd700 1 -- 172.21.15.67:0/2847339303 wait complete. 2024-04-22T23:34:48.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.456+0000 7fe0edccd700 1 Processor -- start 2024-04-22T23:34:48.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.456+0000 7fe0edccd700 1 -- start start 2024-04-22T23:34:48.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.457+0000 7fe0edccd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0e80ffc20 0x7fe0e819c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:48.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.457+0000 7fe0edccd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0e8105bd0 0x7fe0e819cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:48.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.457+0000 7fe0edccd700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e819d2b0 con 0x7fe0e8105bd0 2024-04-22T23:34:48.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.457+0000 7fe0edccd700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e8196730 con 0x7fe0e80ffc20 2024-04-22T23:34:48.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.457+0000 7fe0e6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0e8105bd0 0x7fe0e819cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:48.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.457+0000 7fe0e6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0e8105bd0 0x7fe0e819cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60006/0 (socket says 172.21.15.67:60006) 2024-04-22T23:34:48.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.457+0000 7fe0e6ffd700 1 -- 172.21.15.67:0/852666469 learned_addr learned my addr 172.21.15.67:0/852666469 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:34:48.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.457+0000 7fe0e77fe700 1 --2- 172.21.15.67:0/852666469 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0e80ffc20 0x7fe0e819c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:48.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.457+0000 7fe0e6ffd700 1 -- 172.21.15.67:0/852666469 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0e80ffc20 msgr2=0x7fe0e819c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:48.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.457+0000 7fe0e6ffd700 1 --2- 172.21.15.67:0/852666469 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0e80ffc20 0x7fe0e819c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:48.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.457+0000 7fe0e6ffd700 1 -- 172.21.15.67:0/852666469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0dc0095f0 con 0x7fe0e8105bd0 2024-04-22T23:34:48.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.457+0000 7fe0e6ffd700 1 --2- 172.21.15.67:0/852666469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0e8105bd0 0x7fe0e819cbd0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc006010 tx=0x7fe0dc00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:34:48.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.457+0000 7fe0e4ff9700 1 -- 172.21.15.67:0/852666469 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0dc01d070 con 0x7fe0e8105bd0 2024-04-22T23:34:48.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.457+0000 7fe0edccd700 1 -- 172.21.15.67:0/852666469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0e81969b0 con 0x7fe0e8105bd0 2024-04-22T23:34:48.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.457+0000 7fe0e4ff9700 1 -- 172.21.15.67:0/852666469 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe0dc00f460 con 0x7fe0e8105bd0 2024-04-22T23:34:48.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.458+0000 7fe0e4ff9700 1 -- 172.21.15.67:0/852666469 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0dc017610 con 0x7fe0e8105bd0 2024-04-22T23:34:48.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.458+0000 7fe0edccd700 1 -- 172.21.15.67:0/852666469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0e8196ea0 con 0x7fe0e8105bd0 2024-04-22T23:34:48.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.458+0000 7fe0edccd700 1 -- 172.21.15.67:0/852666469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0e804f2e0 con 0x7fe0e8105bd0 2024-04-22T23:34:48.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.459+0000 7fe0e4ff9700 1 -- 172.21.15.67:0/852666469 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe0dc00f5d0 con 0x7fe0e8105bd0 2024-04-22T23:34:48.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.459+0000 7fe0e4ff9700 1 --2- 172.21.15.67:0/852666469 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe0d006ecc0 0x7fe0d0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:48.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.460+0000 7fe0e4ff9700 1 -- 172.21.15.67:0/852666469 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fe0dc090540 con 0x7fe0e8105bd0 2024-04-22T23:34:48.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.460+0000 7fe0e77fe700 1 --2- 172.21.15.67:0/852666469 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe0d006ecc0 0x7fe0d0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:48.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.460+0000 7fe0e77fe700 1 --2- 172.21.15.67:0/852666469 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe0d006ecc0 0x7fe0d0071180 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe0d8005950 tx=0x7fe0d800a300 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:34:48.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.464+0000 7fe0e4ff9700 1 -- 172.21.15.67:0/852666469 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe0dc05ccb0 con 0x7fe0e8105bd0 2024-04-22T23:34:48.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.722+0000 7fe0edccd700 1 -- 172.21.15.67:0/852666469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe0e8197180 con 0x7fe0e8105bd0 2024-04-22T23:34:48.717 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:48 smithi067 ceph-mon[112438]: from='client.34512 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:34:48.717 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:48 smithi067 ceph-mon[112438]: from='client.34516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:34:48.717 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:48 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/184970647' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:34:48.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.722+0000 7fe0e4ff9700 1 -- 172.21.15.67:0/852666469 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1871 (secure 0 0 0) 0x7fe0dc05c840 con 0x7fe0e8105bd0 2024-04-22T23:34:48.718 INFO:teuthology.orchestra.run.smithi067.stdout:e31 2024-04-22T23:34:48.718 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:34:48.718 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:34:48.718 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:34:48.718 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:34:48.718 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:34:48.719 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:34:48.719 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:34:48.719 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:34:48.719 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:34:48.719 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:34:48.719 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:34:48.719 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:34:48.719 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:34:48.719 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:34:48.719 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:34:48.719 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:34:48.719 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:34:48.719 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:34:48.719 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:34:48.720 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:34:48.720 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:34:48.720 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:34:48.720 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:34:48.720 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:34:48.720 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:34:48.720 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:34:48.720 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:34:48.720 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:34:48.720 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:34:48.720 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:34:48.720 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:34:48.720 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:34:48.720 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:34:48.721 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:34:48.721 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:34:48.721 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:34:48.721 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:34:48.721 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:34:48.721 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:34:48.721 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44259} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1827288036,v1:172.21.15.134:6827/1827288036] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:34:48.721 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:34:48.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.725+0000 7fe0edccd700 1 -- 172.21.15.67:0/852666469 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe0d006ecc0 msgr2=0x7fe0d0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:48.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.725+0000 7fe0edccd700 1 --2- 172.21.15.67:0/852666469 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe0d006ecc0 0x7fe0d0071180 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe0d8005950 tx=0x7fe0d800a300 comp rx=0 tx=0).stop 2024-04-22T23:34:48.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.725+0000 7fe0edccd700 1 -- 172.21.15.67:0/852666469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0e8105bd0 msgr2=0x7fe0e819cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:48.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.725+0000 7fe0edccd700 1 --2- 172.21.15.67:0/852666469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0e8105bd0 0x7fe0e819cbd0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc006010 tx=0x7fe0dc00bd70 comp rx=0 tx=0).stop 2024-04-22T23:34:48.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.725+0000 7fe0edccd700 1 -- 172.21.15.67:0/852666469 shutdown_connections 2024-04-22T23:34:48.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.725+0000 7fe0edccd700 1 --2- 172.21.15.67:0/852666469 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe0d006ecc0 0x7fe0d0071180 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:48.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.725+0000 7fe0edccd700 1 --2- 172.21.15.67:0/852666469 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0e80ffc20 0x7fe0e819c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:48.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.725+0000 7fe0edccd700 1 --2- 172.21.15.67:0/852666469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0e8105bd0 0x7fe0e819cbd0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:48.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.725+0000 7fe0edccd700 1 -- 172.21.15.67:0/852666469 >> 172.21.15.67:0/852666469 conn(0x7fe0e8075670 msgr2=0x7fe0e8068700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:34:48.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.726+0000 7fe0edccd700 1 -- 172.21.15.67:0/852666469 shutdown_connections 2024-04-22T23:34:48.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.726+0000 7fe0edccd700 1 -- 172.21.15.67:0/852666469 wait complete. 2024-04-22T23:34:48.723 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 31 2024-04-22T23:34:48.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.837+0000 7f8033019700 1 -- 172.21.15.67:0/2089022659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f802c100ea0 msgr2=0x7f802c101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:48.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.837+0000 7f8033019700 1 --2- 172.21.15.67:0/2089022659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f802c100ea0 0x7f802c101280 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f802400b0a0 tx=0x7f802400b3b0 comp rx=0 tx=0).stop 2024-04-22T23:34:48.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.837+0000 7f8033019700 1 -- 172.21.15.67:0/2089022659 shutdown_connections 2024-04-22T23:34:48.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.837+0000 7f8033019700 1 --2- 172.21.15.67:0/2089022659 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f802c101850 0x7f802c1058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:48.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.837+0000 7f8033019700 1 --2- 172.21.15.67:0/2089022659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f802c100ea0 0x7f802c101280 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:48.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.837+0000 7f8033019700 1 -- 172.21.15.67:0/2089022659 >> 172.21.15.67:0/2089022659 conn(0x7f802c0fc750 msgr2=0x7f802c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:34:48.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.837+0000 7f8033019700 1 -- 172.21.15.67:0/2089022659 shutdown_connections 2024-04-22T23:34:48.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.837+0000 7f8033019700 1 -- 172.21.15.67:0/2089022659 wait complete. 2024-04-22T23:34:48.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.838+0000 7f8033019700 1 Processor -- start 2024-04-22T23:34:48.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.838+0000 7f8033019700 1 -- start start 2024-04-22T23:34:48.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.838+0000 7f8033019700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f802c100ea0 0x7f802c073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:48.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.838+0000 7f8033019700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f802c101850 0x7f802c0735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:48.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.838+0000 7f8033019700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f802c073b10 con 0x7f802c100ea0 2024-04-22T23:34:48.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.838+0000 7f8033019700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f802c073c50 con 0x7f802c101850 2024-04-22T23:34:48.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.838+0000 7f8030db5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f802c100ea0 0x7f802c073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:48.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.838+0000 7f8030db5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f802c100ea0 0x7f802c073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58454/0 (socket says 172.21.15.67:58454) 2024-04-22T23:34:48.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.838+0000 7f8030db5700 1 -- 172.21.15.67:0/4278517998 learned_addr learned my addr 172.21.15.67:0/4278517998 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:34:48.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.838+0000 7f802bfff700 1 --2- 172.21.15.67:0/4278517998 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f802c101850 0x7f802c0735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:48.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.838+0000 7f8030db5700 1 -- 172.21.15.67:0/4278517998 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f802c101850 msgr2=0x7f802c0735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:48.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.838+0000 7f8030db5700 1 --2- 172.21.15.67:0/4278517998 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f802c101850 0x7f802c0735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:48.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.838+0000 7f8030db5700 1 -- 172.21.15.67:0/4278517998 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8024009d00 con 0x7f802c100ea0 2024-04-22T23:34:48.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.838+0000 7f8030db5700 1 --2- 172.21.15.67:0/4278517998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f802c100ea0 0x7f802c073090 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f802400b070 tx=0x7f802400cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:34:48.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.839+0000 7f8029ffb700 1 -- 172.21.15.67:0/4278517998 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f802400e040 con 0x7f802c100ea0 2024-04-22T23:34:48.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.839+0000 7f8029ffb700 1 -- 172.21.15.67:0/4278517998 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8024003e60 con 0x7f802c100ea0 2024-04-22T23:34:48.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.839+0000 7f8033019700 1 -- 172.21.15.67:0/4278517998 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f802c073ed0 con 0x7f802c100ea0 2024-04-22T23:34:48.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.839+0000 7f8029ffb700 1 -- 172.21.15.67:0/4278517998 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f802401d910 con 0x7f802c100ea0 2024-04-22T23:34:48.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.839+0000 7f8033019700 1 -- 172.21.15.67:0/4278517998 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f802c1a2f10 con 0x7f802c100ea0 2024-04-22T23:34:48.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.840+0000 7f8033019700 1 -- 172.21.15.67:0/4278517998 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f802c109240 con 0x7f802c100ea0 2024-04-22T23:34:48.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.840+0000 7f8029ffb700 1 -- 172.21.15.67:0/4278517998 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8024004820 con 0x7f802c100ea0 2024-04-22T23:34:48.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.841+0000 7f8029ffb700 1 --2- 172.21.15.67:0/4278517998 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f801806ecc0 0x7f8018071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:48.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.841+0000 7f8029ffb700 1 -- 172.21.15.67:0/4278517998 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f8024090870 con 0x7f802c100ea0 2024-04-22T23:34:48.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.841+0000 7f802bfff700 1 --2- 172.21.15.67:0/4278517998 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f801806ecc0 0x7f8018071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:48.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.841+0000 7f802bfff700 1 --2- 172.21.15.67:0/4278517998 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f801806ecc0 0x7f8018071180 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f801c006fd0 tx=0x7f801c008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:34:48.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:48.846+0000 7f8029ffb700 1 -- 172.21.15.67:0/4278517998 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f8024059530 con 0x7f802c100ea0 2024-04-22T23:34:48.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:48 smithi134 ceph-mon[85630]: from='client.34512 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:34:48.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:48 smithi134 ceph-mon[85630]: from='client.34516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:34:48.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:48 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/184970647' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:34:49.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.078+0000 7f8033019700 1 -- 172.21.15.67:0/4278517998 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f802c0746d0 con 0x7f801806ecc0 2024-04-22T23:34:49.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.079+0000 7f8029ffb700 1 -- 172.21.15.67:0/4278517998 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f802c0746d0 con 0x7f801806ecc0 2024-04-22T23:34:49.074 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:34:49.074 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:34:49.074 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:34:49.075 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:34:49.075 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:34:49.075 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:34:49.075 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:34:49.075 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:34:49.075 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:34:49.075 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:34:49.075 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T23:34:49.075 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:34:49.075 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:34:49.075 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:34:49.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.082+0000 7f8033019700 1 -- 172.21.15.67:0/4278517998 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f801806ecc0 msgr2=0x7f8018071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:49.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.082+0000 7f8033019700 1 --2- 172.21.15.67:0/4278517998 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f801806ecc0 0x7f8018071180 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f801c006fd0 tx=0x7f801c008040 comp rx=0 tx=0).stop 2024-04-22T23:34:49.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.082+0000 7f8033019700 1 -- 172.21.15.67:0/4278517998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f802c100ea0 msgr2=0x7f802c073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:49.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.082+0000 7f8033019700 1 --2- 172.21.15.67:0/4278517998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f802c100ea0 0x7f802c073090 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f802400b070 tx=0x7f802400cdd0 comp rx=0 tx=0).stop 2024-04-22T23:34:49.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.082+0000 7f8033019700 1 -- 172.21.15.67:0/4278517998 shutdown_connections 2024-04-22T23:34:49.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.082+0000 7f8033019700 1 --2- 172.21.15.67:0/4278517998 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f801806ecc0 0x7f8018071180 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:49.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.082+0000 7f8033019700 1 --2- 172.21.15.67:0/4278517998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f802c100ea0 0x7f802c073090 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:49.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.082+0000 7f8033019700 1 --2- 172.21.15.67:0/4278517998 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f802c101850 0x7f802c0735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:49.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.082+0000 7f8033019700 1 -- 172.21.15.67:0/4278517998 >> 172.21.15.67:0/4278517998 conn(0x7f802c0fc750 msgr2=0x7f802c0fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:34:49.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.082+0000 7f8033019700 1 -- 172.21.15.67:0/4278517998 shutdown_connections 2024-04-22T23:34:49.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.082+0000 7f8033019700 1 -- 172.21.15.67:0/4278517998 wait complete. 2024-04-22T23:34:49.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.190+0000 7f76e59b1700 1 -- 172.21.15.67:0/1741473559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76e00ffc20 msgr2=0x7f76e0100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:49.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.190+0000 7f76e59b1700 1 --2- 172.21.15.67:0/1741473559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76e00ffc20 0x7f76e0100080 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f76d4009910 tx=0x7f76d4009c20 comp rx=0 tx=0).stop 2024-04-22T23:34:49.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.190+0000 7f76e59b1700 1 -- 172.21.15.67:0/1741473559 shutdown_connections 2024-04-22T23:34:49.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.190+0000 7f76e59b1700 1 --2- 172.21.15.67:0/1741473559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76e00ffc20 0x7f76e0100080 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:49.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.190+0000 7f76e59b1700 1 --2- 172.21.15.67:0/1741473559 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f76e0105bd0 0x7f76e0105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:49.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.190+0000 7f76e59b1700 1 -- 172.21.15.67:0/1741473559 >> 172.21.15.67:0/1741473559 conn(0x7f76e0075670 msgr2=0x7f76e0075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:34:49.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.190+0000 7f76e59b1700 1 -- 172.21.15.67:0/1741473559 shutdown_connections 2024-04-22T23:34:49.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.190+0000 7f76e59b1700 1 -- 172.21.15.67:0/1741473559 wait complete. 2024-04-22T23:34:49.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.190+0000 7f76e59b1700 1 Processor -- start 2024-04-22T23:34:49.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.191+0000 7f76e59b1700 1 -- start start 2024-04-22T23:34:49.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.191+0000 7f76e59b1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76e00ffc20 0x7f76e019c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:49.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.191+0000 7f76e59b1700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f76e0105bd0 0x7f76e019cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:49.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.191+0000 7f76e59b1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76e019d200 con 0x7f76e00ffc20 2024-04-22T23:34:49.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.191+0000 7f76e59b1700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76e01966b0 con 0x7f76e0105bd0 2024-04-22T23:34:49.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.191+0000 7f76deffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76e00ffc20 0x7f76e019c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:49.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.191+0000 7f76deffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76e00ffc20 0x7f76e019c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58472/0 (socket says 172.21.15.67:58472) 2024-04-22T23:34:49.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.191+0000 7f76deffd700 1 -- 172.21.15.67:0/2429052183 learned_addr learned my addr 172.21.15.67:0/2429052183 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:34:49.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.191+0000 7f76de7fc700 1 --2- 172.21.15.67:0/2429052183 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f76e0105bd0 0x7f76e019cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:49.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.191+0000 7f76deffd700 1 -- 172.21.15.67:0/2429052183 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f76e0105bd0 msgr2=0x7f76e019cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:49.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.191+0000 7f76deffd700 1 --2- 172.21.15.67:0/2429052183 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f76e0105bd0 0x7f76e019cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:49.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.191+0000 7f76deffd700 1 -- 172.21.15.67:0/2429052183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d40095f0 con 0x7f76e00ffc20 2024-04-22T23:34:49.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.192+0000 7f76deffd700 1 --2- 172.21.15.67:0/2429052183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76e00ffc20 0x7f76e019c630 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f76d000d8d0 tx=0x7f76d000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:34:49.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.192+0000 7f76e49af700 1 -- 172.21.15.67:0/2429052183 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76d00098a0 con 0x7f76e00ffc20 2024-04-22T23:34:49.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.192+0000 7f76e49af700 1 -- 172.21.15.67:0/2429052183 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f76d0010460 con 0x7f76e00ffc20 2024-04-22T23:34:49.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.192+0000 7f76e59b1700 1 -- 172.21.15.67:0/2429052183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76e0196990 con 0x7f76e00ffc20 2024-04-22T23:34:49.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.192+0000 7f76e49af700 1 -- 172.21.15.67:0/2429052183 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76d000f5f0 con 0x7f76e00ffc20 2024-04-22T23:34:49.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.192+0000 7f76e59b1700 1 -- 172.21.15.67:0/2429052183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76e0196ee0 con 0x7f76e00ffc20 2024-04-22T23:34:49.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.193+0000 7f76e59b1700 1 -- 172.21.15.67:0/2429052183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76e004f2e0 con 0x7f76e00ffc20 2024-04-22T23:34:49.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.193+0000 7f76e49af700 1 -- 172.21.15.67:0/2429052183 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f76d0009a00 con 0x7f76e00ffc20 2024-04-22T23:34:49.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.194+0000 7f76e49af700 1 --2- 172.21.15.67:0/2429052183 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f76c806ecc0 0x7f76c8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:34:49.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.194+0000 7f76e49af700 1 -- 172.21.15.67:0/2429052183 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f76d008eda0 con 0x7f76e00ffc20 2024-04-22T23:34:49.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.194+0000 7f76de7fc700 1 --2- 172.21.15.67:0/2429052183 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f76c806ecc0 0x7f76c8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:34:49.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.194+0000 7f76de7fc700 1 --2- 172.21.15.67:0/2429052183 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f76c806ecc0 0x7f76c8071180 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f76e00ffa60 tx=0x7f76d401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:34:49.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.199+0000 7f76e49af700 1 -- 172.21.15.67:0/2429052183 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f76d005b590 con 0x7f76e00ffc20 2024-04-22T23:34:49.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.495+0000 7f76e59b1700 1 -- 172.21.15.67:0/2429052183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f76e0197180 con 0x7f76e00ffc20 2024-04-22T23:34:49.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.495+0000 7f76e49af700 1 -- 172.21.15.67:0/2429052183 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f76d00161c0 con 0x7f76e00ffc20 2024-04-22T23:34:49.490 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:34:49.491 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:34:49.491 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:34:49.491 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:34:49.491 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:34:49.491 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:34:49.491 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:34:49.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.498+0000 7f76e59b1700 1 -- 172.21.15.67:0/2429052183 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f76c806ecc0 msgr2=0x7f76c8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:49.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.498+0000 7f76e59b1700 1 --2- 172.21.15.67:0/2429052183 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f76c806ecc0 0x7f76c8071180 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f76e00ffa60 tx=0x7f76d401a040 comp rx=0 tx=0).stop 2024-04-22T23:34:49.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.498+0000 7f76e59b1700 1 -- 172.21.15.67:0/2429052183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76e00ffc20 msgr2=0x7f76e019c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:34:49.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.498+0000 7f76e59b1700 1 --2- 172.21.15.67:0/2429052183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76e00ffc20 0x7f76e019c630 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f76d000d8d0 tx=0x7f76d000dbe0 comp rx=0 tx=0).stop 2024-04-22T23:34:49.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.498+0000 7f76e59b1700 1 -- 172.21.15.67:0/2429052183 shutdown_connections 2024-04-22T23:34:49.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.498+0000 7f76e59b1700 1 --2- 172.21.15.67:0/2429052183 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f76c806ecc0 0x7f76c8071180 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:49.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.498+0000 7f76e59b1700 1 --2- 172.21.15.67:0/2429052183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76e00ffc20 0x7f76e019c630 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:49.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.498+0000 7f76e59b1700 1 --2- 172.21.15.67:0/2429052183 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f76e0105bd0 0x7f76e019cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:34:49.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.498+0000 7f76e59b1700 1 -- 172.21.15.67:0/2429052183 >> 172.21.15.67:0/2429052183 conn(0x7f76e0075670 msgr2=0x7f76e0068680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:34:49.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.498+0000 7f76e59b1700 1 -- 172.21.15.67:0/2429052183 shutdown_connections 2024-04-22T23:34:49.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:34:49.498+0000 7f76e59b1700 1 -- 172.21.15.67:0/2429052183 wait complete. 2024-04-22T23:34:49.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:49 smithi067 ceph-mon[112438]: from='client.34520 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:34:49.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:49 smithi067 ceph-mon[112438]: pgmap v253: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:49.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:49 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/852666469' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:34:49.789 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:49 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2429052183' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:34:49.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:49 smithi134 ceph-mon[85630]: from='client.34520 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:34:49.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:49 smithi134 ceph-mon[85630]: pgmap v253: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:49.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:49 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/852666469' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:34:49.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:49 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2429052183' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:34:50.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:50 smithi134 ceph-mon[85630]: from='client.34532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:34:51.039 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:50 smithi067 ceph-mon[112438]: from='client.34532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:34:51.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:51 smithi134 ceph-mon[85630]: pgmap v254: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:52.039 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:51 smithi067 ceph-mon[112438]: pgmap v254: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:53.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:53 smithi134 ceph-mon[85630]: pgmap v255: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:54.039 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:53 smithi067 ceph-mon[112438]: pgmap v255: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:55.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:55 smithi134 ceph-mon[85630]: pgmap v256: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:56.039 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:55 smithi067 ceph-mon[112438]: pgmap v256: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:56.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:34:57.039 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:34:57.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:57 smithi134 ceph-mon[85630]: pgmap v257: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:58.039 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:57 smithi067 ceph-mon[112438]: pgmap v257: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:34:59.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:34:59 smithi134 ceph-mon[85630]: pgmap v258: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:00.039 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:34:59 smithi067 ceph-mon[112438]: pgmap v258: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:01.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:01 smithi134 ceph-mon[85630]: pgmap v259: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:02.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:01 smithi067 ceph-mon[112438]: pgmap v259: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:03.738 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:03 smithi134 ceph-mon[85630]: pgmap v260: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:03.902 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:03 smithi067 ceph-mon[112438]: pgmap v260: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:05.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:05 smithi134 ceph-mon[85630]: pgmap v261: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:06.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:05 smithi067 ceph-mon[112438]: pgmap v261: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:07.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:07 smithi134 ceph-mon[85630]: pgmap v262: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:08.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:07 smithi067 ceph-mon[112438]: pgmap v262: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:09.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:09 smithi134 ceph-mon[85630]: pgmap v263: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:10.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:09 smithi067 ceph-mon[112438]: pgmap v263: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:11.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:11 smithi134 ceph-mon[85630]: pgmap v264: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:11.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:35:12.039 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:11 smithi067 ceph-mon[112438]: pgmap v264: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:12.039 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:35:13.790 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:13 smithi134 ceph-mon[85630]: pgmap v265: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:14.010 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:13 smithi067 ceph-mon[112438]: pgmap v265: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:15.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:15 smithi134 ceph-mon[85630]: pgmap v266: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:16.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:15 smithi067 ceph-mon[112438]: pgmap v266: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:17.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:17 smithi134 ceph-mon[85630]: pgmap v267: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:18.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:17 smithi067 ceph-mon[112438]: pgmap v267: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:19.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.622+0000 7f67c8bde700 1 -- 172.21.15.67:0/2284313514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67c4102760 msgr2=0x7f67c410ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:19.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.622+0000 7f67c8bde700 1 --2- 172.21.15.67:0/2284313514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67c4102760 0x7f67c410ac50 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f67ac009940 tx=0x7f67ac009c50 comp rx=0 tx=0).stop 2024-04-22T23:35:19.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.622+0000 7f67c8bde700 1 -- 172.21.15.67:0/2284313514 shutdown_connections 2024-04-22T23:35:19.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.622+0000 7f67c8bde700 1 --2- 172.21.15.67:0/2284313514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67c4102760 0x7f67c410ac50 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:19.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.622+0000 7f67c8bde700 1 --2- 172.21.15.67:0/2284313514 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67c4101e40 0x7f67c4102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:19.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.622+0000 7f67c8bde700 1 -- 172.21.15.67:0/2284313514 >> 172.21.15.67:0/2284313514 conn(0x7f67c40fb650 msgr2=0x7f67c40fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:35:19.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.622+0000 7f67c8bde700 1 -- 172.21.15.67:0/2284313514 shutdown_connections 2024-04-22T23:35:19.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.622+0000 7f67c8bde700 1 -- 172.21.15.67:0/2284313514 wait complete. 2024-04-22T23:35:19.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.623+0000 7f67c8bde700 1 Processor -- start 2024-04-22T23:35:19.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.623+0000 7f67c8bde700 1 -- start start 2024-04-22T23:35:19.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.623+0000 7f67c8bde700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67c4101e40 0x7f67c40ffaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:19.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.623+0000 7f67c8bde700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67c4102760 0x7f67c40fffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:19.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.623+0000 7f67c8bde700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c4100520 con 0x7f67c4101e40 2024-04-22T23:35:19.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.623+0000 7f67c8bde700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c4100660 con 0x7f67c4102760 2024-04-22T23:35:19.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.623+0000 7f67c259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67c4101e40 0x7f67c40ffaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:19.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.623+0000 7f67c259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67c4101e40 0x7f67c40ffaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40234/0 (socket says 172.21.15.67:40234) 2024-04-22T23:35:19.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.623+0000 7f67c259c700 1 -- 172.21.15.67:0/1017096754 learned_addr learned my addr 172.21.15.67:0/1017096754 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:35:19.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.623+0000 7f67c1d9b700 1 --2- 172.21.15.67:0/1017096754 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67c4102760 0x7f67c40fffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:19.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.624+0000 7f67c259c700 1 -- 172.21.15.67:0/1017096754 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67c4102760 msgr2=0x7f67c40fffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:19.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.624+0000 7f67c259c700 1 --2- 172.21.15.67:0/1017096754 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67c4102760 0x7f67c40fffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:19.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.624+0000 7f67c259c700 1 -- 172.21.15.67:0/1017096754 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67ac0095f0 con 0x7f67c4101e40 2024-04-22T23:35:19.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.624+0000 7f67c259c700 1 --2- 172.21.15.67:0/1017096754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67c4101e40 0x7f67c40ffaa0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f67b400d8d0 tx=0x7f67b400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:35:19.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.626+0000 7f67bb7fe700 1 -- 172.21.15.67:0/1017096754 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67b40098a0 con 0x7f67c4101e40 2024-04-22T23:35:19.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.626+0000 7f67bb7fe700 1 -- 172.21.15.67:0/1017096754 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f67b4010460 con 0x7f67c4101e40 2024-04-22T23:35:19.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.626+0000 7f67bb7fe700 1 -- 172.21.15.67:0/1017096754 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67b400f5f0 con 0x7f67c4101e40 2024-04-22T23:35:19.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.626+0000 7f67c8bde700 1 -- 172.21.15.67:0/1017096754 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67c41a2a50 con 0x7f67c4101e40 2024-04-22T23:35:19.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.626+0000 7f67c8bde700 1 -- 172.21.15.67:0/1017096754 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67c41a2e00 con 0x7f67c4101e40 2024-04-22T23:35:19.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.627+0000 7f67c8bde700 1 -- 172.21.15.67:0/1017096754 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67c4108350 con 0x7f67c4101e40 2024-04-22T23:35:19.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.628+0000 7f67bb7fe700 1 -- 172.21.15.67:0/1017096754 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f67b400f750 con 0x7f67c4101e40 2024-04-22T23:35:19.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.628+0000 7f67bb7fe700 1 --2- 172.21.15.67:0/1017096754 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f67b006ecc0 0x7f67b0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:19.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.628+0000 7f67bb7fe700 1 -- 172.21.15.67:0/1017096754 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f67b408ede0 con 0x7f67c4101e40 2024-04-22T23:35:19.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.628+0000 7f67c1d9b700 1 --2- 172.21.15.67:0/1017096754 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f67b006ecc0 0x7f67b0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:19.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.629+0000 7f67c1d9b700 1 --2- 172.21.15.67:0/1017096754 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f67b006ecc0 0x7f67b0071180 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f67ac009910 tx=0x7f67ac005a90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:35:19.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.633+0000 7f67bb7fe700 1 -- 172.21.15.67:0/1017096754 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f67b4057aa0 con 0x7f67c4101e40 2024-04-22T23:35:19.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.863+0000 7f67c8bde700 1 -- 172.21.15.67:0/1017096754 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f67c40611d0 con 0x7f67b006ecc0 2024-04-22T23:35:19.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.864+0000 7f67bb7fe700 1 -- 172.21.15.67:0/1017096754 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f67c40611d0 con 0x7f67b006ecc0 2024-04-22T23:35:19.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.866+0000 7f67c8bde700 1 -- 172.21.15.67:0/1017096754 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f67b006ecc0 msgr2=0x7f67b0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:19.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.866+0000 7f67c8bde700 1 --2- 172.21.15.67:0/1017096754 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f67b006ecc0 0x7f67b0071180 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f67ac009910 tx=0x7f67ac005a90 comp rx=0 tx=0).stop 2024-04-22T23:35:19.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.866+0000 7f67c8bde700 1 -- 172.21.15.67:0/1017096754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67c4101e40 msgr2=0x7f67c40ffaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:19.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.866+0000 7f67c8bde700 1 --2- 172.21.15.67:0/1017096754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67c4101e40 0x7f67c40ffaa0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f67b400d8d0 tx=0x7f67b400dbe0 comp rx=0 tx=0).stop 2024-04-22T23:35:19.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.867+0000 7f67c8bde700 1 -- 172.21.15.67:0/1017096754 shutdown_connections 2024-04-22T23:35:19.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.867+0000 7f67c8bde700 1 --2- 172.21.15.67:0/1017096754 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f67b006ecc0 0x7f67b0071180 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:19.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.867+0000 7f67c8bde700 1 --2- 172.21.15.67:0/1017096754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67c4101e40 0x7f67c40ffaa0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:19.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.867+0000 7f67c8bde700 1 --2- 172.21.15.67:0/1017096754 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f67c4102760 0x7f67c40fffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:19.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.867+0000 7f67c8bde700 1 -- 172.21.15.67:0/1017096754 >> 172.21.15.67:0/1017096754 conn(0x7f67c40fb650 msgr2=0x7f67c4105490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:35:19.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.867+0000 7f67c8bde700 1 -- 172.21.15.67:0/1017096754 shutdown_connections 2024-04-22T23:35:19.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.867+0000 7f67c8bde700 1 -- 172.21.15.67:0/1017096754 wait complete. 2024-04-22T23:35:19.872 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:35:19.872 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:19 smithi067 ceph-mon[112438]: pgmap v268: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:19.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.976+0000 7f1444262700 1 -- 172.21.15.67:0/2258088965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f143c100ea0 msgr2=0x7f143c101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:19.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.976+0000 7f1444262700 1 --2- 172.21.15.67:0/2258088965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f143c100ea0 0x7f143c101280 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f142c00b0a0 tx=0x7f142c00b3b0 comp rx=0 tx=0).stop 2024-04-22T23:35:19.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.976+0000 7f1444262700 1 -- 172.21.15.67:0/2258088965 shutdown_connections 2024-04-22T23:35:19.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.976+0000 7f1444262700 1 --2- 172.21.15.67:0/2258088965 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f143c101850 0x7f143c1058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:19.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.976+0000 7f1444262700 1 --2- 172.21.15.67:0/2258088965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f143c100ea0 0x7f143c101280 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:19.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.976+0000 7f1444262700 1 -- 172.21.15.67:0/2258088965 >> 172.21.15.67:0/2258088965 conn(0x7f143c0fc750 msgr2=0x7f143c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:35:19.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.976+0000 7f1444262700 1 -- 172.21.15.67:0/2258088965 shutdown_connections 2024-04-22T23:35:19.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.976+0000 7f1444262700 1 -- 172.21.15.67:0/2258088965 wait complete. 2024-04-22T23:35:19.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.977+0000 7f1444262700 1 Processor -- start 2024-04-22T23:35:19.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.977+0000 7f1444262700 1 -- start start 2024-04-22T23:35:19.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.977+0000 7f1444262700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f143c100ea0 0x7f143c073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:19.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.977+0000 7f1444262700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f143c101850 0x7f143c0735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:19.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.977+0000 7f1444262700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f143c073b10 con 0x7f143c101850 2024-04-22T23:35:19.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.977+0000 7f1444262700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f143c073c50 con 0x7f143c100ea0 2024-04-22T23:35:19.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.977+0000 7f14417fd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f143c101850 0x7f143c0735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:19.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.977+0000 7f14417fd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f143c101850 0x7f143c0735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40244/0 (socket says 172.21.15.67:40244) 2024-04-22T23:35:19.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.977+0000 7f14417fd700 1 -- 172.21.15.67:0/4078139172 learned_addr learned my addr 172.21.15.67:0/4078139172 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:35:19.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.977+0000 7f1441ffe700 1 --2- 172.21.15.67:0/4078139172 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f143c100ea0 0x7f143c073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:19.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.977+0000 7f14417fd700 1 -- 172.21.15.67:0/4078139172 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f143c100ea0 msgr2=0x7f143c073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:19.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.977+0000 7f14417fd700 1 --2- 172.21.15.67:0/4078139172 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f143c100ea0 0x7f143c073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:19.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.977+0000 7f14417fd700 1 -- 172.21.15.67:0/4078139172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f142c009d00 con 0x7f143c101850 2024-04-22T23:35:19.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.977+0000 7f14417fd700 1 --2- 172.21.15.67:0/4078139172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f143c101850 0x7f143c0735d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f143000d900 tx=0x7f143000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:35:19.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.978+0000 7f143affd700 1 -- 172.21.15.67:0/4078139172 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1430009900 con 0x7f143c101850 2024-04-22T23:35:19.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.978+0000 7f143affd700 1 -- 172.21.15.67:0/4078139172 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1430010460 con 0x7f143c101850 2024-04-22T23:35:19.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.978+0000 7f1444262700 1 -- 172.21.15.67:0/4078139172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f143c1a2af0 con 0x7f143c101850 2024-04-22T23:35:19.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.978+0000 7f143affd700 1 -- 172.21.15.67:0/4078139172 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f143000f5f0 con 0x7f143c101850 2024-04-22T23:35:19.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.978+0000 7f1444262700 1 -- 172.21.15.67:0/4078139172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f143c1a2f00 con 0x7f143c101850 2024-04-22T23:35:19.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.979+0000 7f1444262700 1 -- 172.21.15.67:0/4078139172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f143c109240 con 0x7f143c101850 2024-04-22T23:35:19.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.979+0000 7f143affd700 1 -- 172.21.15.67:0/4078139172 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f14300105d0 con 0x7f143c101850 2024-04-22T23:35:19.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.980+0000 7f143affd700 1 --2- 172.21.15.67:0/4078139172 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f142406ecc0 0x7f1424071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:19.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.980+0000 7f143affd700 1 -- 172.21.15.67:0/4078139172 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f143008ed60 con 0x7f143c101850 2024-04-22T23:35:19.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.980+0000 7f1441ffe700 1 --2- 172.21.15.67:0/4078139172 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f142406ecc0 0x7f1424071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:19.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.980+0000 7f1441ffe700 1 --2- 172.21.15.67:0/4078139172 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f142406ecc0 0x7f1424071180 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f142c00b830 tx=0x7f142c006000 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:35:19.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:19.985+0000 7f143affd700 1 -- 172.21.15.67:0/4078139172 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1430057a20 con 0x7f143c101850 2024-04-22T23:35:19.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:19 smithi134 ceph-mon[85630]: pgmap v268: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:20.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.215+0000 7f1444262700 1 -- 172.21.15.67:0/4078139172 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f143c0611d0 con 0x7f142406ecc0 2024-04-22T23:35:20.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.216+0000 7f143affd700 1 -- 172.21.15.67:0/4078139172 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f143c0611d0 con 0x7f142406ecc0 2024-04-22T23:35:20.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.218+0000 7f1444262700 1 -- 172.21.15.67:0/4078139172 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f142406ecc0 msgr2=0x7f1424071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:20.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.218+0000 7f1444262700 1 --2- 172.21.15.67:0/4078139172 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f142406ecc0 0x7f1424071180 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f142c00b830 tx=0x7f142c006000 comp rx=0 tx=0).stop 2024-04-22T23:35:20.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.219+0000 7f1444262700 1 -- 172.21.15.67:0/4078139172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f143c101850 msgr2=0x7f143c0735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:20.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.219+0000 7f1444262700 1 --2- 172.21.15.67:0/4078139172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f143c101850 0x7f143c0735d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f143000d900 tx=0x7f143000dcc0 comp rx=0 tx=0).stop 2024-04-22T23:35:20.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.219+0000 7f1444262700 1 -- 172.21.15.67:0/4078139172 shutdown_connections 2024-04-22T23:35:20.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.219+0000 7f1444262700 1 --2- 172.21.15.67:0/4078139172 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f142406ecc0 0x7f1424071180 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:20.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.219+0000 7f1444262700 1 --2- 172.21.15.67:0/4078139172 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f143c100ea0 0x7f143c073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:20.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.219+0000 7f1444262700 1 --2- 172.21.15.67:0/4078139172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f143c101850 0x7f143c0735d0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:20.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.219+0000 7f1444262700 1 -- 172.21.15.67:0/4078139172 >> 172.21.15.67:0/4078139172 conn(0x7f143c0fc750 msgr2=0x7f143c0fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:35:20.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.219+0000 7f1444262700 1 -- 172.21.15.67:0/4078139172 shutdown_connections 2024-04-22T23:35:20.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.219+0000 7f1444262700 1 -- 172.21.15.67:0/4078139172 wait complete. 2024-04-22T23:35:20.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.328+0000 7ff3bf23e700 1 -- 172.21.15.67:0/3285460142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b8068580 msgr2=0x7ff3b80689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:20.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.328+0000 7ff3bf23e700 1 --2- 172.21.15.67:0/3285460142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b8068580 0x7ff3b80689e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff3a4009910 tx=0x7ff3a4009c20 comp rx=0 tx=0).stop 2024-04-22T23:35:20.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.328+0000 7ff3bf23e700 1 -- 172.21.15.67:0/3285460142 shutdown_connections 2024-04-22T23:35:20.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.328+0000 7ff3bf23e700 1 --2- 172.21.15.67:0/3285460142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b8068580 0x7ff3b80689e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:20.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.328+0000 7ff3bf23e700 1 --2- 172.21.15.67:0/3285460142 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff3b8106cd0 0x7ff3b81070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:20.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.328+0000 7ff3bf23e700 1 -- 172.21.15.67:0/3285460142 >> 172.21.15.67:0/3285460142 conn(0x7ff3b8075670 msgr2=0x7ff3b8075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:35:20.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.328+0000 7ff3bf23e700 1 -- 172.21.15.67:0/3285460142 shutdown_connections 2024-04-22T23:35:20.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.329+0000 7ff3bf23e700 1 -- 172.21.15.67:0/3285460142 wait complete. 2024-04-22T23:35:20.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.329+0000 7ff3bf23e700 1 Processor -- start 2024-04-22T23:35:20.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.329+0000 7ff3bf23e700 1 -- start start 2024-04-22T23:35:20.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.330+0000 7ff3bf23e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff3b8068580 0x7ff3b8196980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:20.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.330+0000 7ff3bf23e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b8106cd0 0x7ff3b8196ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:20.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.330+0000 7ff3bf23e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b8197550 con 0x7ff3b8106cd0 2024-04-22T23:35:20.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.330+0000 7ff3bf23e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b819b360 con 0x7ff3b8068580 2024-04-22T23:35:20.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.330+0000 7ff3b7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b8106cd0 0x7ff3b8196ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:20.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.330+0000 7ff3b7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b8106cd0 0x7ff3b8196ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40264/0 (socket says 172.21.15.67:40264) 2024-04-22T23:35:20.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.330+0000 7ff3b7fff700 1 -- 172.21.15.67:0/3529687830 learned_addr learned my addr 172.21.15.67:0/3529687830 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:35:20.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.330+0000 7ff3bcfda700 1 --2- 172.21.15.67:0/3529687830 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff3b8068580 0x7ff3b8196980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:20.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.330+0000 7ff3b7fff700 1 -- 172.21.15.67:0/3529687830 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff3b8068580 msgr2=0x7ff3b8196980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:20.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.330+0000 7ff3b7fff700 1 --2- 172.21.15.67:0/3529687830 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff3b8068580 0x7ff3b8196980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:20.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.330+0000 7ff3b7fff700 1 -- 172.21.15.67:0/3529687830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a40095f0 con 0x7ff3b8106cd0 2024-04-22T23:35:20.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.330+0000 7ff3b7fff700 1 --2- 172.21.15.67:0/3529687830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b8106cd0 0x7ff3b8196ec0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff3a4006010 tx=0x7ff3a400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:35:20.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.330+0000 7ff3b5ffb700 1 -- 172.21.15.67:0/3529687830 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3a401d070 con 0x7ff3b8106cd0 2024-04-22T23:35:20.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.330+0000 7ff3b5ffb700 1 -- 172.21.15.67:0/3529687830 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff3a4004fb0 con 0x7ff3b8106cd0 2024-04-22T23:35:20.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.330+0000 7ff3bf23e700 1 -- 172.21.15.67:0/3529687830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3b819b5e0 con 0x7ff3b8106cd0 2024-04-22T23:35:20.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.330+0000 7ff3b5ffb700 1 -- 172.21.15.67:0/3529687830 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3a4022ea0 con 0x7ff3b8106cd0 2024-04-22T23:35:20.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.331+0000 7ff3bf23e700 1 -- 172.21.15.67:0/3529687830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3b819bad0 con 0x7ff3b8106cd0 2024-04-22T23:35:20.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.332+0000 7ff3bf23e700 1 -- 172.21.15.67:0/3529687830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3b81091f0 con 0x7ff3b8106cd0 2024-04-22T23:35:20.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.332+0000 7ff3b5ffb700 1 -- 172.21.15.67:0/3529687830 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff3a4022470 con 0x7ff3b8106cd0 2024-04-22T23:35:20.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.332+0000 7ff3b5ffb700 1 --2- 172.21.15.67:0/3529687830 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff3a806ecc0 0x7ff3a8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:20.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.333+0000 7ff3b5ffb700 1 -- 172.21.15.67:0/3529687830 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7ff3a4090560 con 0x7ff3b8106cd0 2024-04-22T23:35:20.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.333+0000 7ff3bcfda700 1 --2- 172.21.15.67:0/3529687830 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff3a806ecc0 0x7ff3a8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:20.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.333+0000 7ff3bcfda700 1 --2- 172.21.15.67:0/3529687830 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff3a806ecc0 0x7ff3a8071180 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac006fd0 tx=0x7ff3ac008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:35:20.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.337+0000 7ff3b5ffb700 1 -- 172.21.15.67:0/3529687830 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff3a405cd80 con 0x7ff3b8106cd0 2024-04-22T23:35:20.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.554+0000 7ff3bf23e700 1 -- 172.21.15.67:0/3529687830 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff3b8197ce0 con 0x7ff3a806ecc0 2024-04-22T23:35:20.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.565+0000 7ff3b5ffb700 1 -- 172.21.15.67:0/3529687830 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7ff3b8197ce0 con 0x7ff3a806ecc0 2024-04-22T23:35:20.560 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:35:20.560 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (9m) 2m ago 20m 25.5M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:35:20.560 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (20m) 2m ago 20m 9114k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:35:20.560 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (19m) 2m ago 19m 9042k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:35:20.560 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (6m) 2m ago 20m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:35:20.561 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (6m) 2m ago 19m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:35:20.561 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (8m) 2m ago 20m 79.3M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:35:20.561 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (2m) 2m ago 14m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:35:20.561 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (2m) 2m ago 14m 12.6M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:35:20.561 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (2m) 2m ago 14m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:35:20.561 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (14m) 2m ago 14m 12.2M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:35:20.561 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (10m) 2m ago 21m 511M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:35:20.561 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (9m) 2m ago 19m 453M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:35:20.561 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (7m) 2m ago 22m 55.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:35:20.561 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (7m) 2m ago 19m 46.0M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:35:20.561 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (9m) 2m ago 20m 15.4M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:35:20.561 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (9m) 2m ago 19m 16.1M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:35:20.561 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (5m) 2m ago 18m 99.2M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:35:20.561 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (5m) 2m ago 18m 127M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:35:20.561 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (4m) 2m ago 17m 79.9M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:35:20.562 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (4m) 2m ago 17m 97.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:35:20.562 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (3m) 2m ago 16m 82.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:35:20.562 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (3m) 2m ago 16m 77.6M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:35:20.562 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (9m) 2m ago 19m 73.9M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:35:20.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.567+0000 7ff3bf23e700 1 -- 172.21.15.67:0/3529687830 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff3a806ecc0 msgr2=0x7ff3a8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:20.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.567+0000 7ff3bf23e700 1 --2- 172.21.15.67:0/3529687830 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff3a806ecc0 0x7ff3a8071180 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac006fd0 tx=0x7ff3ac008040 comp rx=0 tx=0).stop 2024-04-22T23:35:20.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.567+0000 7ff3bf23e700 1 -- 172.21.15.67:0/3529687830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b8106cd0 msgr2=0x7ff3b8196ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:20.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.567+0000 7ff3bf23e700 1 --2- 172.21.15.67:0/3529687830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b8106cd0 0x7ff3b8196ec0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff3a4006010 tx=0x7ff3a400be20 comp rx=0 tx=0).stop 2024-04-22T23:35:20.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.567+0000 7ff3bf23e700 1 -- 172.21.15.67:0/3529687830 shutdown_connections 2024-04-22T23:35:20.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.567+0000 7ff3bf23e700 1 --2- 172.21.15.67:0/3529687830 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff3a806ecc0 0x7ff3a8071180 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:20.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.567+0000 7ff3bf23e700 1 --2- 172.21.15.67:0/3529687830 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff3b8068580 0x7ff3b8196980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:20.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.567+0000 7ff3bf23e700 1 --2- 172.21.15.67:0/3529687830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b8106cd0 0x7ff3b8196ec0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:20.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.567+0000 7ff3bf23e700 1 -- 172.21.15.67:0/3529687830 >> 172.21.15.67:0/3529687830 conn(0x7ff3b8075670 msgr2=0x7ff3b80fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:35:20.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.567+0000 7ff3bf23e700 1 -- 172.21.15.67:0/3529687830 shutdown_connections 2024-04-22T23:35:20.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.567+0000 7ff3bf23e700 1 -- 172.21.15.67:0/3529687830 wait complete. 2024-04-22T23:35:20.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.677+0000 7fc1af661700 1 -- 172.21.15.67:0/3056648272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1a8068ea0 msgr2=0x7fc1a810d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:20.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.677+0000 7fc1af661700 1 --2- 172.21.15.67:0/3056648272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1a8068ea0 0x7fc1a810d290 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc19c009910 tx=0x7fc19c009c20 comp rx=0 tx=0).stop 2024-04-22T23:35:20.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.677+0000 7fc1af661700 1 -- 172.21.15.67:0/3056648272 shutdown_connections 2024-04-22T23:35:20.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.677+0000 7fc1af661700 1 --2- 172.21.15.67:0/3056648272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1a8068ea0 0x7fc1a810d290 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:20.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.677+0000 7fc1af661700 1 --2- 172.21.15.67:0/3056648272 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1a8068580 0x7fc1a8068960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:20.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.677+0000 7fc1af661700 1 -- 172.21.15.67:0/3056648272 >> 172.21.15.67:0/3056648272 conn(0x7fc1a8075670 msgr2=0x7fc1a8075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:35:20.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.677+0000 7fc1af661700 1 -- 172.21.15.67:0/3056648272 shutdown_connections 2024-04-22T23:35:20.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.677+0000 7fc1af661700 1 -- 172.21.15.67:0/3056648272 wait complete. 2024-04-22T23:35:20.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.678+0000 7fc1af661700 1 Processor -- start 2024-04-22T23:35:20.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.678+0000 7fc1af661700 1 -- start start 2024-04-22T23:35:20.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.678+0000 7fc1af661700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1a8068580 0x7fc1a819e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:20.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.678+0000 7fc1af661700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1a8068ea0 0x7fc1a819ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:20.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.678+0000 7fc1af661700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a819f4b0 con 0x7fc1a8068ea0 2024-04-22T23:35:20.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.678+0000 7fc1af661700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a8198960 con 0x7fc1a8068580 2024-04-22T23:35:20.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.678+0000 7fc1acbfc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1a8068ea0 0x7fc1a819ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:20.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.678+0000 7fc1acbfc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1a8068ea0 0x7fc1a819ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40288/0 (socket says 172.21.15.67:40288) 2024-04-22T23:35:20.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.678+0000 7fc1acbfc700 1 -- 172.21.15.67:0/4011112114 learned_addr learned my addr 172.21.15.67:0/4011112114 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:35:20.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.678+0000 7fc1ad3fd700 1 --2- 172.21.15.67:0/4011112114 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1a8068580 0x7fc1a819e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:20.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.678+0000 7fc1acbfc700 1 -- 172.21.15.67:0/4011112114 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1a8068580 msgr2=0x7fc1a819e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:20.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.678+0000 7fc1acbfc700 1 --2- 172.21.15.67:0/4011112114 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1a8068580 0x7fc1a819e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:20.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.678+0000 7fc1acbfc700 1 -- 172.21.15.67:0/4011112114 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc19c0095f0 con 0x7fc1a8068ea0 2024-04-22T23:35:20.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.679+0000 7fc1acbfc700 1 --2- 172.21.15.67:0/4011112114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1a8068ea0 0x7fc1a819ee20 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc19c006010 tx=0x7fc19c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:35:20.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.679+0000 7fc1a67fc700 1 -- 172.21.15.67:0/4011112114 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc19c01d070 con 0x7fc1a8068ea0 2024-04-22T23:35:20.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.679+0000 7fc1a67fc700 1 -- 172.21.15.67:0/4011112114 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc19c005dc0 con 0x7fc1a8068ea0 2024-04-22T23:35:20.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.679+0000 7fc1a67fc700 1 -- 172.21.15.67:0/4011112114 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc19c017400 con 0x7fc1a8068ea0 2024-04-22T23:35:20.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.679+0000 7fc1af661700 1 -- 172.21.15.67:0/4011112114 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1a8198be0 con 0x7fc1a8068ea0 2024-04-22T23:35:20.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.679+0000 7fc1af661700 1 -- 172.21.15.67:0/4011112114 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1a81990d0 con 0x7fc1a8068ea0 2024-04-22T23:35:20.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.680+0000 7fc1af661700 1 -- 172.21.15.67:0/4011112114 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1a810aa00 con 0x7fc1a8068ea0 2024-04-22T23:35:20.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.680+0000 7fc1a67fc700 1 -- 172.21.15.67:0/4011112114 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc19c003a60 con 0x7fc1a8068ea0 2024-04-22T23:35:20.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.681+0000 7fc1a67fc700 1 --2- 172.21.15.67:0/4011112114 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc19006ecc0 0x7fc190071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:20.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.681+0000 7fc1a67fc700 1 -- 172.21.15.67:0/4011112114 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fc19c090370 con 0x7fc1a8068ea0 2024-04-22T23:35:20.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.681+0000 7fc1ad3fd700 1 --2- 172.21.15.67:0/4011112114 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc19006ecc0 0x7fc190071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:20.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.681+0000 7fc1ad3fd700 1 --2- 172.21.15.67:0/4011112114 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc19006ecc0 0x7fc190071180 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc1a0005950 tx=0x7fc1a000b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:35:20.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.686+0000 7fc1a67fc700 1 -- 172.21.15.67:0/4011112114 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc19c05cb60 con 0x7fc1a8068ea0 2024-04-22T23:35:20.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.982+0000 7fc1af661700 1 -- 172.21.15.67:0/4011112114 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc1a8199e70 con 0x7fc1a8068ea0 2024-04-22T23:35:20.978 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:20 smithi067 ceph-mon[112438]: from='client.34540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:35:20.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.982+0000 7fc1a67fc700 1 -- 172.21.15.67:0/4011112114 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fc19c05c6f0 con 0x7fc1a8068ea0 2024-04-22T23:35:20.978 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:35:20.978 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:35:20.978 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:35:20.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:35:20.978 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:35:20.979 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:35:20.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:35:20.979 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:35:20.979 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:35:20.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:35:20.979 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:35:20.979 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:35:20.979 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 3 2024-04-22T23:35:20.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:35:20.979 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:35:20.979 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:35:20.979 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 13 2024-04-22T23:35:20.979 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:35:20.979 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:35:20.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.984+0000 7fc1af661700 1 -- 172.21.15.67:0/4011112114 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc19006ecc0 msgr2=0x7fc190071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:20.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.984+0000 7fc1af661700 1 --2- 172.21.15.67:0/4011112114 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc19006ecc0 0x7fc190071180 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc1a0005950 tx=0x7fc1a000b410 comp rx=0 tx=0).stop 2024-04-22T23:35:20.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.984+0000 7fc1af661700 1 -- 172.21.15.67:0/4011112114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1a8068ea0 msgr2=0x7fc1a819ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:20.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.984+0000 7fc1af661700 1 --2- 172.21.15.67:0/4011112114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1a8068ea0 0x7fc1a819ee20 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc19c006010 tx=0x7fc19c00bd70 comp rx=0 tx=0).stop 2024-04-22T23:35:20.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.985+0000 7fc1af661700 1 -- 172.21.15.67:0/4011112114 shutdown_connections 2024-04-22T23:35:20.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.985+0000 7fc1af661700 1 --2- 172.21.15.67:0/4011112114 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc19006ecc0 0x7fc190071180 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:20.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.985+0000 7fc1af661700 1 --2- 172.21.15.67:0/4011112114 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1a8068580 0x7fc1a819e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:20.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.985+0000 7fc1af661700 1 --2- 172.21.15.67:0/4011112114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1a8068ea0 0x7fc1a819ee20 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:20.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.985+0000 7fc1af661700 1 -- 172.21.15.67:0/4011112114 >> 172.21.15.67:0/4011112114 conn(0x7fc1a8075670 msgr2=0x7fc1a80fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:35:20.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.985+0000 7fc1af661700 1 -- 172.21.15.67:0/4011112114 shutdown_connections 2024-04-22T23:35:20.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:20.985+0000 7fc1af661700 1 -- 172.21.15.67:0/4011112114 wait complete. 2024-04-22T23:35:20.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:20 smithi134 ceph-mon[85630]: from='client.34540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:35:21.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.092+0000 7fb735e22700 1 -- 172.21.15.67:0/1808892721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb730106db0 msgr2=0x7fb730107190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:21.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.092+0000 7fb735e22700 1 --2- 172.21.15.67:0/1808892721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb730106db0 0x7fb730107190 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb720009910 tx=0x7fb720009c20 comp rx=0 tx=0).stop 2024-04-22T23:35:21.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.093+0000 7fb735e22700 1 -- 172.21.15.67:0/1808892721 shutdown_connections 2024-04-22T23:35:21.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.093+0000 7fb735e22700 1 --2- 172.21.15.67:0/1808892721 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb7300ff410 0x7fb7300ff890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:21.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.093+0000 7fb735e22700 1 --2- 172.21.15.67:0/1808892721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb730106db0 0x7fb730107190 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:21.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.093+0000 7fb735e22700 1 -- 172.21.15.67:0/1808892721 >> 172.21.15.67:0/1808892721 conn(0x7fb730074e30 msgr2=0x7fb730075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:35:21.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.093+0000 7fb735e22700 1 -- 172.21.15.67:0/1808892721 shutdown_connections 2024-04-22T23:35:21.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.093+0000 7fb735e22700 1 -- 172.21.15.67:0/1808892721 wait complete. 2024-04-22T23:35:21.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.093+0000 7fb735e22700 1 Processor -- start 2024-04-22T23:35:21.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.093+0000 7fb735e22700 1 -- start start 2024-04-22T23:35:21.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.094+0000 7fb735e22700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7300ff410 0x7fb73019c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:21.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.094+0000 7fb735e22700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb730106db0 0x7fb73019cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:21.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.094+0000 7fb735e22700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73019d2e0 con 0x7fb7300ff410 2024-04-22T23:35:21.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.094+0000 7fb735e22700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb730196820 con 0x7fb730106db0 2024-04-22T23:35:21.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.094+0000 7fb72f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7300ff410 0x7fb73019c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:21.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.094+0000 7fb72f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7300ff410 0x7fb73019c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40298/0 (socket says 172.21.15.67:40298) 2024-04-22T23:35:21.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.094+0000 7fb72f7fe700 1 -- 172.21.15.67:0/1611994046 learned_addr learned my addr 172.21.15.67:0/1611994046 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:35:21.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.094+0000 7fb72effd700 1 --2- 172.21.15.67:0/1611994046 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb730106db0 0x7fb73019cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:21.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.094+0000 7fb72f7fe700 1 -- 172.21.15.67:0/1611994046 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb730106db0 msgr2=0x7fb73019cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:21.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.094+0000 7fb72f7fe700 1 --2- 172.21.15.67:0/1611994046 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb730106db0 0x7fb73019cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:21.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.094+0000 7fb72f7fe700 1 -- 172.21.15.67:0/1611994046 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7200095f0 con 0x7fb7300ff410 2024-04-22T23:35:21.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.094+0000 7fb72f7fe700 1 --2- 172.21.15.67:0/1611994046 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7300ff410 0x7fb73019c7a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb72000b5c0 tx=0x7fb72000be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:35:21.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.094+0000 7fb72cff9700 1 -- 172.21.15.67:0/1611994046 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb72001d070 con 0x7fb7300ff410 2024-04-22T23:35:21.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.094+0000 7fb72cff9700 1 -- 172.21.15.67:0/1611994046 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb72000f460 con 0x7fb7300ff410 2024-04-22T23:35:21.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.094+0000 7fb735e22700 1 -- 172.21.15.67:0/1611994046 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb730196aa0 con 0x7fb7300ff410 2024-04-22T23:35:21.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.094+0000 7fb72cff9700 1 -- 172.21.15.67:0/1611994046 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb720017610 con 0x7fb7300ff410 2024-04-22T23:35:21.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.095+0000 7fb735e22700 1 -- 172.21.15.67:0/1611994046 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb730196f90 con 0x7fb7300ff410 2024-04-22T23:35:21.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.096+0000 7fb735e22700 1 -- 172.21.15.67:0/1611994046 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb730109300 con 0x7fb7300ff410 2024-04-22T23:35:21.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.096+0000 7fb72cff9700 1 -- 172.21.15.67:0/1611994046 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb72000f5d0 con 0x7fb7300ff410 2024-04-22T23:35:21.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.097+0000 7fb72cff9700 1 --2- 172.21.15.67:0/1611994046 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb71806ecc0 0x7fb718071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:21.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.097+0000 7fb72cff9700 1 -- 172.21.15.67:0/1611994046 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fb7200905e0 con 0x7fb7300ff410 2024-04-22T23:35:21.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.097+0000 7fb72effd700 1 --2- 172.21.15.67:0/1611994046 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb71806ecc0 0x7fb718071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:21.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.097+0000 7fb72effd700 1 --2- 172.21.15.67:0/1611994046 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb71806ecc0 0x7fb718071180 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb724005ea0 tx=0x7fb724005e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:35:21.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.101+0000 7fb72cff9700 1 -- 172.21.15.67:0/1611994046 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb72005cd50 con 0x7fb7300ff410 2024-04-22T23:35:21.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.357+0000 7fb735e22700 1 -- 172.21.15.67:0/1611994046 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb730197d30 con 0x7fb7300ff410 2024-04-22T23:35:21.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.358+0000 7fb72cff9700 1 -- 172.21.15.67:0/1611994046 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1871 (secure 0 0 0) 0x7fb72005c8e0 con 0x7fb7300ff410 2024-04-22T23:35:21.353 INFO:teuthology.orchestra.run.smithi067.stdout:e31 2024-04-22T23:35:21.353 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:35:21.353 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:35:21.353 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:35:21.353 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:35:21.353 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:35:21.353 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:35:21.353 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:35:21.353 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:35:21.354 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:35:21.354 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:35:21.354 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:35:21.354 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:35:21.354 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:35:21.354 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:35:21.354 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:35:21.354 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:35:21.354 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:35:21.354 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:35:21.354 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:35:21.354 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:35:21.354 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:35:21.355 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:35:21.355 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:35:21.355 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:35:21.355 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:35:21.355 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:35:21.355 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:35:21.355 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:35:21.355 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:35:21.355 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:35:21.355 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:35:21.355 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:35:21.355 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:35:21.355 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:35:21.355 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:35:21.356 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:35:21.356 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:35:21.356 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:35:21.356 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:35:21.356 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44259} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1827288036,v1:172.21.15.134:6827/1827288036] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:35:21.356 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:35:21.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.360+0000 7fb735e22700 1 -- 172.21.15.67:0/1611994046 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb71806ecc0 msgr2=0x7fb718071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:21.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.360+0000 7fb735e22700 1 --2- 172.21.15.67:0/1611994046 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb71806ecc0 0x7fb718071180 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb724005ea0 tx=0x7fb724005e30 comp rx=0 tx=0).stop 2024-04-22T23:35:21.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.360+0000 7fb735e22700 1 -- 172.21.15.67:0/1611994046 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7300ff410 msgr2=0x7fb73019c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:21.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.360+0000 7fb735e22700 1 --2- 172.21.15.67:0/1611994046 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7300ff410 0x7fb73019c7a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb72000b5c0 tx=0x7fb72000be00 comp rx=0 tx=0).stop 2024-04-22T23:35:21.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.360+0000 7fb735e22700 1 -- 172.21.15.67:0/1611994046 shutdown_connections 2024-04-22T23:35:21.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.360+0000 7fb735e22700 1 --2- 172.21.15.67:0/1611994046 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb71806ecc0 0x7fb718071180 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:21.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.360+0000 7fb735e22700 1 --2- 172.21.15.67:0/1611994046 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7300ff410 0x7fb73019c7a0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:21.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.360+0000 7fb735e22700 1 --2- 172.21.15.67:0/1611994046 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb730106db0 0x7fb73019cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:21.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.360+0000 7fb735e22700 1 -- 172.21.15.67:0/1611994046 >> 172.21.15.67:0/1611994046 conn(0x7fb730074e30 msgr2=0x7fb7300fd460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:35:21.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.360+0000 7fb735e22700 1 -- 172.21.15.67:0/1611994046 shutdown_connections 2024-04-22T23:35:21.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.360+0000 7fb735e22700 1 -- 172.21.15.67:0/1611994046 wait complete. 2024-04-22T23:35:21.357 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 31 2024-04-22T23:35:21.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.467+0000 7f2a05bf6700 1 -- 172.21.15.67:0/1396193948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a000ffda0 msgr2=0x7f2a00100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:21.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.467+0000 7f2a05bf6700 1 --2- 172.21.15.67:0/1396193948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a000ffda0 0x7f2a00100180 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f29f0009910 tx=0x7f29f0009c20 comp rx=0 tx=0).stop 2024-04-22T23:35:21.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.467+0000 7f2a05bf6700 1 -- 172.21.15.67:0/1396193948 shutdown_connections 2024-04-22T23:35:21.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.467+0000 7f2a05bf6700 1 --2- 172.21.15.67:0/1396193948 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a00100750 0x7f2a001047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:21.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.468+0000 7f2a05bf6700 1 --2- 172.21.15.67:0/1396193948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a000ffda0 0x7f2a00100180 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:21.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.468+0000 7f2a05bf6700 1 -- 172.21.15.67:0/1396193948 >> 172.21.15.67:0/1396193948 conn(0x7f2a000fb650 msgr2=0x7f2a000fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:35:21.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.468+0000 7f2a05bf6700 1 -- 172.21.15.67:0/1396193948 shutdown_connections 2024-04-22T23:35:21.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.468+0000 7f2a05bf6700 1 -- 172.21.15.67:0/1396193948 wait complete. 2024-04-22T23:35:21.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.468+0000 7f2a05bf6700 1 Processor -- start 2024-04-22T23:35:21.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.468+0000 7f2a05bf6700 1 -- start start 2024-04-22T23:35:21.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.469+0000 7f2a05bf6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a000ffda0 0x7f2a001976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:21.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.469+0000 7f2a05bf6700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a00100750 0x7f2a00197c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:21.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.469+0000 7f2a05bf6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a00198230 con 0x7f2a000ffda0 2024-04-22T23:35:21.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.469+0000 7f2a05bf6700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a001983a0 con 0x7f2a00100750 2024-04-22T23:35:21.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.469+0000 7f29ff7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a000ffda0 0x7f2a001976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:21.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.469+0000 7f29ff7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a000ffda0 0x7f2a001976d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40312/0 (socket says 172.21.15.67:40312) 2024-04-22T23:35:21.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.469+0000 7f29ff7fe700 1 -- 172.21.15.67:0/792941699 learned_addr learned my addr 172.21.15.67:0/792941699 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:35:21.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.469+0000 7f29feffd700 1 --2- 172.21.15.67:0/792941699 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a00100750 0x7f2a00197c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:21.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.469+0000 7f29ff7fe700 1 -- 172.21.15.67:0/792941699 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a00100750 msgr2=0x7f2a00197c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:21.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.469+0000 7f29ff7fe700 1 --2- 172.21.15.67:0/792941699 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a00100750 0x7f2a00197c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:21.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.469+0000 7f29ff7fe700 1 -- 172.21.15.67:0/792941699 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f00095f0 con 0x7f2a000ffda0 2024-04-22T23:35:21.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.469+0000 7f29ff7fe700 1 --2- 172.21.15.67:0/792941699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a000ffda0 0x7f2a001976d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f29f0006010 tx=0x7f29f000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:35:21.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.469+0000 7f29fcff9700 1 -- 172.21.15.67:0/792941699 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29f001d070 con 0x7f2a000ffda0 2024-04-22T23:35:21.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.469+0000 7f29fcff9700 1 -- 172.21.15.67:0/792941699 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f29f0005dc0 con 0x7f2a000ffda0 2024-04-22T23:35:21.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.469+0000 7f2a05bf6700 1 -- 172.21.15.67:0/792941699 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a001a0740 con 0x7f2a000ffda0 2024-04-22T23:35:21.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.470+0000 7f2a05bf6700 1 -- 172.21.15.67:0/792941699 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a001a0bb0 con 0x7f2a000ffda0 2024-04-22T23:35:21.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.470+0000 7f29fcff9700 1 -- 172.21.15.67:0/792941699 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29f0017400 con 0x7f2a000ffda0 2024-04-22T23:35:21.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.471+0000 7f2a05bf6700 1 -- 172.21.15.67:0/792941699 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a00108140 con 0x7f2a000ffda0 2024-04-22T23:35:21.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.471+0000 7f29fcff9700 1 -- 172.21.15.67:0/792941699 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f29f0003a60 con 0x7f2a000ffda0 2024-04-22T23:35:21.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.471+0000 7f29fcff9700 1 --2- 172.21.15.67:0/792941699 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f29e806ecc0 0x7f29e8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:21.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.472+0000 7f29fcff9700 1 -- 172.21.15.67:0/792941699 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f29f0090450 con 0x7f2a000ffda0 2024-04-22T23:35:21.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.472+0000 7f29feffd700 1 --2- 172.21.15.67:0/792941699 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f29e806ecc0 0x7f29e8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:21.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.472+0000 7f29feffd700 1 --2- 172.21.15.67:0/792941699 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f29e806ecc0 0x7f29e8071180 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2a000696e0 tx=0x7f29f400b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:35:21.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.476+0000 7f29fcff9700 1 -- 172.21.15.67:0/792941699 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f29f0059090 con 0x7f2a000ffda0 2024-04-22T23:35:21.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.706+0000 7f2a05bf6700 1 -- 172.21.15.67:0/792941699 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2a000611d0 con 0x7f29e806ecc0 2024-04-22T23:35:21.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.708+0000 7f29fcff9700 1 -- 172.21.15.67:0/792941699 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f2a000611d0 con 0x7f29e806ecc0 2024-04-22T23:35:21.703 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:35:21.703 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:35:21.703 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:35:21.703 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:35:21.703 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:35:21.703 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:35:21.703 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:35:21.703 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:35:21.703 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:35:21.703 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:35:21.703 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T23:35:21.703 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:35:21.703 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:35:21.703 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:35:21.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.710+0000 7f2a05bf6700 1 -- 172.21.15.67:0/792941699 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f29e806ecc0 msgr2=0x7f29e8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:21.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.710+0000 7f2a05bf6700 1 --2- 172.21.15.67:0/792941699 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f29e806ecc0 0x7f29e8071180 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2a000696e0 tx=0x7f29f400b410 comp rx=0 tx=0).stop 2024-04-22T23:35:21.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.710+0000 7f2a05bf6700 1 -- 172.21.15.67:0/792941699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a000ffda0 msgr2=0x7f2a001976d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:21.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.710+0000 7f2a05bf6700 1 --2- 172.21.15.67:0/792941699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a000ffda0 0x7f2a001976d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f29f0006010 tx=0x7f29f000bd70 comp rx=0 tx=0).stop 2024-04-22T23:35:21.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.710+0000 7f2a05bf6700 1 -- 172.21.15.67:0/792941699 shutdown_connections 2024-04-22T23:35:21.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.710+0000 7f2a05bf6700 1 --2- 172.21.15.67:0/792941699 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f29e806ecc0 0x7f29e8071180 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:21.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.710+0000 7f2a05bf6700 1 --2- 172.21.15.67:0/792941699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a000ffda0 0x7f2a001976d0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:21.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.710+0000 7f2a05bf6700 1 --2- 172.21.15.67:0/792941699 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a00100750 0x7f2a00197c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:21.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.710+0000 7f2a05bf6700 1 -- 172.21.15.67:0/792941699 >> 172.21.15.67:0/792941699 conn(0x7f2a000fb650 msgr2=0x7f2a000fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:35:21.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.710+0000 7f2a05bf6700 1 -- 172.21.15.67:0/792941699 shutdown_connections 2024-04-22T23:35:21.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.710+0000 7f2a05bf6700 1 -- 172.21.15.67:0/792941699 wait complete. 2024-04-22T23:35:21.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.817+0000 7f79f764f700 1 -- 172.21.15.67:0/107438842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79f0100ea0 msgr2=0x7f79f0101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:21.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.817+0000 7f79f764f700 1 --2- 172.21.15.67:0/107438842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79f0100ea0 0x7f79f0101280 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f79ec009910 tx=0x7f79ec009c20 comp rx=0 tx=0).stop 2024-04-22T23:35:21.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.817+0000 7f79f764f700 1 -- 172.21.15.67:0/107438842 shutdown_connections 2024-04-22T23:35:21.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.817+0000 7f79f764f700 1 --2- 172.21.15.67:0/107438842 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79f0101850 0x7f79f01058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:21.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.817+0000 7f79f764f700 1 --2- 172.21.15.67:0/107438842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79f0100ea0 0x7f79f0101280 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:21.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.817+0000 7f79f764f700 1 -- 172.21.15.67:0/107438842 >> 172.21.15.67:0/107438842 conn(0x7f79f00fc750 msgr2=0x7f79f00feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:35:21.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.817+0000 7f79f764f700 1 -- 172.21.15.67:0/107438842 shutdown_connections 2024-04-22T23:35:21.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.817+0000 7f79f764f700 1 -- 172.21.15.67:0/107438842 wait complete. 2024-04-22T23:35:21.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.817+0000 7f79f764f700 1 Processor -- start 2024-04-22T23:35:21.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.817+0000 7f79f764f700 1 -- start start 2024-04-22T23:35:21.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.818+0000 7f79f764f700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79f0100ea0 0x7f79f0073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:21.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.818+0000 7f79f764f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79f0101850 0x7f79f00735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:21.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.818+0000 7f79f764f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f0073b10 con 0x7f79f0101850 2024-04-22T23:35:21.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.818+0000 7f79f764f700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f0073c50 con 0x7f79f0100ea0 2024-04-22T23:35:21.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.818+0000 7f79f4bea700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79f0101850 0x7f79f00735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:21.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.818+0000 7f79f4bea700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79f0101850 0x7f79f00735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40326/0 (socket says 172.21.15.67:40326) 2024-04-22T23:35:21.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.818+0000 7f79f4bea700 1 -- 172.21.15.67:0/2201650428 learned_addr learned my addr 172.21.15.67:0/2201650428 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:35:21.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.818+0000 7f79f53eb700 1 --2- 172.21.15.67:0/2201650428 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79f0100ea0 0x7f79f0073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:21.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.818+0000 7f79f4bea700 1 -- 172.21.15.67:0/2201650428 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79f0100ea0 msgr2=0x7f79f0073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:21.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.818+0000 7f79f4bea700 1 --2- 172.21.15.67:0/2201650428 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79f0100ea0 0x7f79f0073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:21.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.818+0000 7f79f4bea700 1 -- 172.21.15.67:0/2201650428 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79ec0095f0 con 0x7f79f0101850 2024-04-22T23:35:21.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.818+0000 7f79f53eb700 1 --2- 172.21.15.67:0/2201650428 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79f0100ea0 0x7f79f0073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:35:21.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.818+0000 7f79f4bea700 1 --2- 172.21.15.67:0/2201650428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79f0101850 0x7f79f00735d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f79e000d900 tx=0x7f79e000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:35:21.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.818+0000 7f79e67fc700 1 -- 172.21.15.67:0/2201650428 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79e0009900 con 0x7f79f0101850 2024-04-22T23:35:21.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.818+0000 7f79e67fc700 1 -- 172.21.15.67:0/2201650428 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f79e0010460 con 0x7f79f0101850 2024-04-22T23:35:21.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.818+0000 7f79f764f700 1 -- 172.21.15.67:0/2201650428 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79f01a2af0 con 0x7f79f0101850 2024-04-22T23:35:21.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.819+0000 7f79e67fc700 1 -- 172.21.15.67:0/2201650428 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79e000f5f0 con 0x7f79f0101850 2024-04-22T23:35:21.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.819+0000 7f79f764f700 1 -- 172.21.15.67:0/2201650428 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79f01a2f00 con 0x7f79f0101850 2024-04-22T23:35:21.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.819+0000 7f79f764f700 1 -- 172.21.15.67:0/2201650428 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79f0109240 con 0x7f79f0101850 2024-04-22T23:35:21.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.820+0000 7f79e67fc700 1 -- 172.21.15.67:0/2201650428 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f79e00105d0 con 0x7f79f0101850 2024-04-22T23:35:21.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.820+0000 7f79e67fc700 1 --2- 172.21.15.67:0/2201650428 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f79dc06ecc0 0x7f79dc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:21.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.821+0000 7f79e67fc700 1 -- 172.21.15.67:0/2201650428 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f79e008ed50 con 0x7f79f0101850 2024-04-22T23:35:21.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.821+0000 7f79f53eb700 1 --2- 172.21.15.67:0/2201650428 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f79dc06ecc0 0x7f79dc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:21.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.821+0000 7f79f53eb700 1 --2- 172.21.15.67:0/2201650428 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f79dc06ecc0 0x7f79dc071180 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f79ec005b40 tx=0x7f79ec01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:35:21.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:21.825+0000 7f79e67fc700 1 -- 172.21.15.67:0/2201650428 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f79e0057a10 con 0x7f79f0101850 2024-04-22T23:35:21.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:21 smithi134 ceph-mon[85630]: pgmap v269: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:21.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:21 smithi134 ceph-mon[85630]: from='client.34544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:35:21.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:21 smithi134 ceph-mon[85630]: from='client.34548 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:35:21.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:21 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/4011112114' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:35:21.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:21 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1611994046' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:35:22.039 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:21 smithi067 ceph-mon[112438]: pgmap v269: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:22.039 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:21 smithi067 ceph-mon[112438]: from='client.34544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:35:22.039 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:21 smithi067 ceph-mon[112438]: from='client.34548 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:35:22.039 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:21 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/4011112114' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:35:22.039 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:21 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1611994046' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:35:22.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:22.123+0000 7f79f764f700 1 -- 172.21.15.67:0/2201650428 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f79f004f2e0 con 0x7f79f0101850 2024-04-22T23:35:22.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:22.124+0000 7f79e67fc700 1 -- 172.21.15.67:0/2201650428 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f79e00171c0 con 0x7f79f0101850 2024-04-22T23:35:22.119 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:35:22.119 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:35:22.119 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:35:22.119 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:35:22.119 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:35:22.119 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:35:22.119 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:35:22.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:22.126+0000 7f79f764f700 1 -- 172.21.15.67:0/2201650428 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f79dc06ecc0 msgr2=0x7f79dc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:22.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:22.126+0000 7f79f764f700 1 --2- 172.21.15.67:0/2201650428 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f79dc06ecc0 0x7f79dc071180 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f79ec005b40 tx=0x7f79ec01a040 comp rx=0 tx=0).stop 2024-04-22T23:35:22.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:22.126+0000 7f79f764f700 1 -- 172.21.15.67:0/2201650428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79f0101850 msgr2=0x7f79f00735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:22.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:22.126+0000 7f79f764f700 1 --2- 172.21.15.67:0/2201650428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79f0101850 0x7f79f00735d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f79e000d900 tx=0x7f79e000dcc0 comp rx=0 tx=0).stop 2024-04-22T23:35:22.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:22.126+0000 7f79f764f700 1 -- 172.21.15.67:0/2201650428 shutdown_connections 2024-04-22T23:35:22.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:22.126+0000 7f79f764f700 1 --2- 172.21.15.67:0/2201650428 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f79dc06ecc0 0x7f79dc071180 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:22.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:22.126+0000 7f79f764f700 1 --2- 172.21.15.67:0/2201650428 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79f0100ea0 0x7f79f0073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:22.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:22.126+0000 7f79f764f700 1 --2- 172.21.15.67:0/2201650428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79f0101850 0x7f79f00735d0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:22.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:22.126+0000 7f79f764f700 1 -- 172.21.15.67:0/2201650428 >> 172.21.15.67:0/2201650428 conn(0x7f79f00fc750 msgr2=0x7f79f00fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:35:22.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:22.126+0000 7f79f764f700 1 -- 172.21.15.67:0/2201650428 shutdown_connections 2024-04-22T23:35:22.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:22.126+0000 7f79f764f700 1 -- 172.21.15.67:0/2201650428 wait complete. 2024-04-22T23:35:22.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:22 smithi134 ceph-mon[85630]: from='client.34560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:35:22.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:22 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2201650428' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:35:23.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:22 smithi067 ceph-mon[112438]: from='client.34560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:35:23.039 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:22 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2201650428' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:35:23.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:23 smithi134 ceph-mon[85630]: pgmap v270: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:24.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:23 smithi067 ceph-mon[112438]: pgmap v270: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:25.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:25 smithi134 ceph-mon[85630]: pgmap v271: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:26.039 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:25 smithi067 ceph-mon[112438]: pgmap v271: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:26.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:35:27.039 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:35:27.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:27 smithi134 ceph-mon[85630]: pgmap v272: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:28.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:27 smithi067 ceph-mon[112438]: pgmap v272: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:29.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:29 smithi134 ceph-mon[85630]: pgmap v273: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:30.039 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:29 smithi067 ceph-mon[112438]: pgmap v273: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:31.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:31 smithi134 ceph-mon[85630]: pgmap v274: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:32.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:31 smithi067 ceph-mon[112438]: pgmap v274: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:33.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:33 smithi134 ceph-mon[85630]: pgmap v275: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:34.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:33 smithi067 ceph-mon[112438]: pgmap v275: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:35.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:35 smithi134 ceph-mon[85630]: pgmap v276: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:36.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:35 smithi067 ceph-mon[112438]: pgmap v276: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:37.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:37 smithi134 ceph-mon[85630]: pgmap v277: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:38.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:37 smithi067 ceph-mon[112438]: pgmap v277: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:39.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:39 smithi134 ceph-mon[85630]: pgmap v278: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:40.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:39 smithi067 ceph-mon[112438]: pgmap v278: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:41.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:41 smithi134 ceph-mon[85630]: pgmap v279: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:41.989 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:35:42.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:41 smithi067 ceph-mon[112438]: pgmap v279: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:42.039 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:35:43.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:43 smithi134 ceph-mon[85630]: pgmap v280: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:44.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:43 smithi067 ceph-mon[112438]: pgmap v280: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:45.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:45 smithi134 ceph-mon[85630]: pgmap v281: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:46.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:45 smithi067 ceph-mon[112438]: pgmap v281: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:47.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:47 smithi134 ceph-mon[85630]: pgmap v282: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:48.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:47 smithi067 ceph-mon[112438]: pgmap v282: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:49.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:49 smithi134 ceph-mon[85630]: pgmap v283: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:50.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:49 smithi067 ceph-mon[112438]: pgmap v283: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:51.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:51 smithi134 ceph-mon[85630]: pgmap v284: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:52.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:51 smithi067 ceph-mon[112438]: pgmap v284: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:52.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.248+0000 7f2cbad07700 1 -- 172.21.15.67:0/2059330862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cb4100ea0 msgr2=0x7f2cb4101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:52.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.248+0000 7f2cbad07700 1 --2- 172.21.15.67:0/2059330862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cb4100ea0 0x7f2cb4101280 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f2ca8009910 tx=0x7f2ca8009c20 comp rx=0 tx=0).stop 2024-04-22T23:35:52.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.248+0000 7f2cbad07700 1 -- 172.21.15.67:0/2059330862 shutdown_connections 2024-04-22T23:35:52.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.248+0000 7f2cbad07700 1 --2- 172.21.15.67:0/2059330862 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2cb4101850 0x7f2cb41058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:52.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.248+0000 7f2cbad07700 1 --2- 172.21.15.67:0/2059330862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cb4100ea0 0x7f2cb4101280 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:52.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.248+0000 7f2cbad07700 1 -- 172.21.15.67:0/2059330862 >> 172.21.15.67:0/2059330862 conn(0x7f2cb40fc750 msgr2=0x7f2cb40feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:35:52.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.249+0000 7f2cbad07700 1 -- 172.21.15.67:0/2059330862 shutdown_connections 2024-04-22T23:35:52.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.249+0000 7f2cbad07700 1 -- 172.21.15.67:0/2059330862 wait complete. 2024-04-22T23:35:52.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.249+0000 7f2cbad07700 1 Processor -- start 2024-04-22T23:35:52.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.249+0000 7f2cbad07700 1 -- start start 2024-04-22T23:35:52.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.249+0000 7f2cbad07700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cb4100ea0 0x7f2cb419c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:52.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.249+0000 7f2cbad07700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2cb4101850 0x7f2cb419cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:52.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.249+0000 7f2cbad07700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb419d2c0 con 0x7f2cb4100ea0 2024-04-22T23:35:52.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.249+0000 7f2cbad07700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb4196770 con 0x7f2cb4101850 2024-04-22T23:35:52.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.250+0000 7f2cb8aa3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cb4100ea0 0x7f2cb419c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:52.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.250+0000 7f2cb8aa3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cb4100ea0 0x7f2cb419c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50570/0 (socket says 172.21.15.67:50570) 2024-04-22T23:35:52.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.250+0000 7f2cb8aa3700 1 -- 172.21.15.67:0/4244447455 learned_addr learned my addr 172.21.15.67:0/4244447455 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:35:52.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.250+0000 7f2cb3fff700 1 --2- 172.21.15.67:0/4244447455 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2cb4101850 0x7f2cb419cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:52.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.250+0000 7f2cb8aa3700 1 -- 172.21.15.67:0/4244447455 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2cb4101850 msgr2=0x7f2cb419cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:52.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.250+0000 7f2cb8aa3700 1 --2- 172.21.15.67:0/4244447455 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2cb4101850 0x7f2cb419cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:52.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.250+0000 7f2cb8aa3700 1 -- 172.21.15.67:0/4244447455 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca80095f0 con 0x7f2cb4100ea0 2024-04-22T23:35:52.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.250+0000 7f2cb8aa3700 1 --2- 172.21.15.67:0/4244447455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cb4100ea0 0x7f2cb419c6f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f2ca8006010 tx=0x7f2ca800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:35:52.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.250+0000 7f2cb1ffb700 1 -- 172.21.15.67:0/4244447455 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ca801d070 con 0x7f2cb4100ea0 2024-04-22T23:35:52.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.250+0000 7f2cb1ffb700 1 -- 172.21.15.67:0/4244447455 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2ca8005dc0 con 0x7f2cb4100ea0 2024-04-22T23:35:52.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.250+0000 7f2cbad07700 1 -- 172.21.15.67:0/4244447455 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cb41969f0 con 0x7f2cb4100ea0 2024-04-22T23:35:52.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.250+0000 7f2cb1ffb700 1 -- 172.21.15.67:0/4244447455 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ca8017400 con 0x7f2cb4100ea0 2024-04-22T23:35:52.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.251+0000 7f2cbad07700 1 -- 172.21.15.67:0/4244447455 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cb4196ee0 con 0x7f2cb4100ea0 2024-04-22T23:35:52.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.252+0000 7f2cbad07700 1 -- 172.21.15.67:0/4244447455 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cb4109240 con 0x7f2cb4100ea0 2024-04-22T23:35:52.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.252+0000 7f2cb1ffb700 1 -- 172.21.15.67:0/4244447455 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2ca8003a60 con 0x7f2cb4100ea0 2024-04-22T23:35:52.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.253+0000 7f2cb1ffb700 1 --2- 172.21.15.67:0/4244447455 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2ca406ecc0 0x7f2ca4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:52.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.253+0000 7f2cb1ffb700 1 -- 172.21.15.67:0/4244447455 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f2ca8090460 con 0x7f2cb4100ea0 2024-04-22T23:35:52.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.253+0000 7f2cb3fff700 1 --2- 172.21.15.67:0/4244447455 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2ca406ecc0 0x7f2ca4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:52.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.253+0000 7f2cb3fff700 1 --2- 172.21.15.67:0/4244447455 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2ca406ecc0 0x7f2ca4071180 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f2cb4197be0 tx=0x7f2ca000b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:35:52.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.258+0000 7f2cb1ffb700 1 -- 172.21.15.67:0/4244447455 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f2ca805cbd0 con 0x7f2cb4100ea0 2024-04-22T23:35:52.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.488+0000 7f2cbad07700 1 -- 172.21.15.67:0/4244447455 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2cb4197100 con 0x7f2ca406ecc0 2024-04-22T23:35:52.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.489+0000 7f2cb1ffb700 1 -- 172.21.15.67:0/4244447455 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f2cb4197100 con 0x7f2ca406ecc0 2024-04-22T23:35:52.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.491+0000 7f2cbad07700 1 -- 172.21.15.67:0/4244447455 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2ca406ecc0 msgr2=0x7f2ca4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:52.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.491+0000 7f2cbad07700 1 --2- 172.21.15.67:0/4244447455 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2ca406ecc0 0x7f2ca4071180 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f2cb4197be0 tx=0x7f2ca000b410 comp rx=0 tx=0).stop 2024-04-22T23:35:52.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.491+0000 7f2cbad07700 1 -- 172.21.15.67:0/4244447455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cb4100ea0 msgr2=0x7f2cb419c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:52.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.491+0000 7f2cbad07700 1 --2- 172.21.15.67:0/4244447455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cb4100ea0 0x7f2cb419c6f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f2ca8006010 tx=0x7f2ca800bd70 comp rx=0 tx=0).stop 2024-04-22T23:35:52.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.491+0000 7f2cbad07700 1 -- 172.21.15.67:0/4244447455 shutdown_connections 2024-04-22T23:35:52.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.491+0000 7f2cbad07700 1 --2- 172.21.15.67:0/4244447455 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2ca406ecc0 0x7f2ca4071180 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:52.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.491+0000 7f2cbad07700 1 --2- 172.21.15.67:0/4244447455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cb4100ea0 0x7f2cb419c6f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:52.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.491+0000 7f2cbad07700 1 --2- 172.21.15.67:0/4244447455 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2cb4101850 0x7f2cb419cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:52.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.491+0000 7f2cbad07700 1 -- 172.21.15.67:0/4244447455 >> 172.21.15.67:0/4244447455 conn(0x7f2cb40fc750 msgr2=0x7f2cb40fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:35:52.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.492+0000 7f2cbad07700 1 -- 172.21.15.67:0/4244447455 shutdown_connections 2024-04-22T23:35:52.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.492+0000 7f2cbad07700 1 -- 172.21.15.67:0/4244447455 wait complete. 2024-04-22T23:35:52.498 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:35:52.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.601+0000 7f7c2e6e1700 1 -- 172.21.15.67:0/4098151426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c28068580 msgr2=0x7f7c280689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:52.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.601+0000 7f7c2e6e1700 1 --2- 172.21.15.67:0/4098151426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c28068580 0x7f7c280689e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7c18009910 tx=0x7f7c18009c20 comp rx=0 tx=0).stop 2024-04-22T23:35:52.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.601+0000 7f7c2e6e1700 1 -- 172.21.15.67:0/4098151426 shutdown_connections 2024-04-22T23:35:52.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.601+0000 7f7c2e6e1700 1 --2- 172.21.15.67:0/4098151426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c28068580 0x7f7c280689e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:52.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.601+0000 7f7c2e6e1700 1 --2- 172.21.15.67:0/4098151426 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c28106cd0 0x7f7c281070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:52.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.601+0000 7f7c2e6e1700 1 -- 172.21.15.67:0/4098151426 >> 172.21.15.67:0/4098151426 conn(0x7f7c28075670 msgr2=0x7f7c28075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:35:52.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.601+0000 7f7c2e6e1700 1 -- 172.21.15.67:0/4098151426 shutdown_connections 2024-04-22T23:35:52.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.601+0000 7f7c2e6e1700 1 -- 172.21.15.67:0/4098151426 wait complete. 2024-04-22T23:35:52.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.602+0000 7f7c2e6e1700 1 Processor -- start 2024-04-22T23:35:52.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.602+0000 7f7c2e6e1700 1 -- start start 2024-04-22T23:35:52.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.602+0000 7f7c2e6e1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c28068580 0x7f7c2819c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:52.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.602+0000 7f7c2e6e1700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c28106cd0 0x7f7c2819cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:52.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.602+0000 7f7c2e6e1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2819d200 con 0x7f7c28068580 2024-04-22T23:35:52.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.602+0000 7f7c2e6e1700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c281966b0 con 0x7f7c28106cd0 2024-04-22T23:35:52.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.602+0000 7f7c27fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c28068580 0x7f7c2819c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:52.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.602+0000 7f7c27fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c28068580 0x7f7c2819c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50592/0 (socket says 172.21.15.67:50592) 2024-04-22T23:35:52.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.602+0000 7f7c27fff700 1 -- 172.21.15.67:0/671486015 learned_addr learned my addr 172.21.15.67:0/671486015 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:35:52.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.602+0000 7f7c277fe700 1 --2- 172.21.15.67:0/671486015 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c28106cd0 0x7f7c2819cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:52.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.603+0000 7f7c27fff700 1 -- 172.21.15.67:0/671486015 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c28106cd0 msgr2=0x7f7c2819cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:52.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.603+0000 7f7c27fff700 1 --2- 172.21.15.67:0/671486015 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c28106cd0 0x7f7c2819cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:52.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.603+0000 7f7c27fff700 1 -- 172.21.15.67:0/671486015 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c180095f0 con 0x7f7c28068580 2024-04-22T23:35:52.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.603+0000 7f7c27fff700 1 --2- 172.21.15.67:0/671486015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c28068580 0x7f7c2819c630 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7c1000d8d0 tx=0x7f7c1000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:35:52.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.603+0000 7f7c257fa700 1 -- 172.21.15.67:0/671486015 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c10009940 con 0x7f7c28068580 2024-04-22T23:35:52.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.603+0000 7f7c257fa700 1 -- 172.21.15.67:0/671486015 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7c10010460 con 0x7f7c28068580 2024-04-22T23:35:52.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.603+0000 7f7c2e6e1700 1 -- 172.21.15.67:0/671486015 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c28196990 con 0x7f7c28068580 2024-04-22T23:35:52.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.603+0000 7f7c257fa700 1 -- 172.21.15.67:0/671486015 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c1000f5f0 con 0x7f7c28068580 2024-04-22T23:35:52.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.603+0000 7f7c2e6e1700 1 -- 172.21.15.67:0/671486015 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c28196ee0 con 0x7f7c28068580 2024-04-22T23:35:52.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.604+0000 7f7c2e6e1700 1 -- 172.21.15.67:0/671486015 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c281091d0 con 0x7f7c28068580 2024-04-22T23:35:52.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.605+0000 7f7c257fa700 1 -- 172.21.15.67:0/671486015 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7c100105d0 con 0x7f7c28068580 2024-04-22T23:35:52.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.605+0000 7f7c257fa700 1 --2- 172.21.15.67:0/671486015 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7c1406ecc0 0x7f7c14071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:52.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.605+0000 7f7c257fa700 1 -- 172.21.15.67:0/671486015 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f7c1008eee0 con 0x7f7c28068580 2024-04-22T23:35:52.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.605+0000 7f7c277fe700 1 --2- 172.21.15.67:0/671486015 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7c1406ecc0 0x7f7c14071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:52.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.606+0000 7f7c277fe700 1 --2- 172.21.15.67:0/671486015 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7c1406ecc0 0x7f7c14071180 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7c18005b40 tx=0x7f7c1801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:35:52.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.610+0000 7f7c257fa700 1 -- 172.21.15.67:0/671486015 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7c1005b6d0 con 0x7f7c28068580 2024-04-22T23:35:52.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.843+0000 7f7c2e6e1700 1 -- 172.21.15.67:0/671486015 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7c28197150 con 0x7f7c1406ecc0 2024-04-22T23:35:52.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.845+0000 7f7c257fa700 1 -- 172.21.15.67:0/671486015 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f7c28197150 con 0x7f7c1406ecc0 2024-04-22T23:35:52.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.847+0000 7f7c2e6e1700 1 -- 172.21.15.67:0/671486015 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7c1406ecc0 msgr2=0x7f7c14071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:52.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.847+0000 7f7c2e6e1700 1 --2- 172.21.15.67:0/671486015 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7c1406ecc0 0x7f7c14071180 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7c18005b40 tx=0x7f7c1801a040 comp rx=0 tx=0).stop 2024-04-22T23:35:52.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.847+0000 7f7c2e6e1700 1 -- 172.21.15.67:0/671486015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c28068580 msgr2=0x7f7c2819c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:52.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.847+0000 7f7c2e6e1700 1 --2- 172.21.15.67:0/671486015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c28068580 0x7f7c2819c630 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7c1000d8d0 tx=0x7f7c1000dc90 comp rx=0 tx=0).stop 2024-04-22T23:35:52.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.847+0000 7f7c2e6e1700 1 -- 172.21.15.67:0/671486015 shutdown_connections 2024-04-22T23:35:52.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.847+0000 7f7c2e6e1700 1 --2- 172.21.15.67:0/671486015 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7c1406ecc0 0x7f7c14071180 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:52.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.847+0000 7f7c2e6e1700 1 --2- 172.21.15.67:0/671486015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c28068580 0x7f7c2819c630 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:52.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.847+0000 7f7c2e6e1700 1 --2- 172.21.15.67:0/671486015 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7c28106cd0 0x7f7c2819cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:52.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.847+0000 7f7c2e6e1700 1 -- 172.21.15.67:0/671486015 >> 172.21.15.67:0/671486015 conn(0x7f7c28075670 msgr2=0x7f7c280feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:35:52.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.847+0000 7f7c2e6e1700 1 -- 172.21.15.67:0/671486015 shutdown_connections 2024-04-22T23:35:52.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.847+0000 7f7c2e6e1700 1 -- 172.21.15.67:0/671486015 wait complete. 2024-04-22T23:35:52.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.954+0000 7f00156cd700 1 -- 172.21.15.67:0/3954836160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0010106cd0 msgr2=0x7f00101070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:52.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.954+0000 7f00156cd700 1 --2- 172.21.15.67:0/3954836160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0010106cd0 0x7f00101070b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f000800b0a0 tx=0x7f000800b3b0 comp rx=0 tx=0).stop 2024-04-22T23:35:52.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.954+0000 7f00156cd700 1 -- 172.21.15.67:0/3954836160 shutdown_connections 2024-04-22T23:35:52.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.954+0000 7f00156cd700 1 --2- 172.21.15.67:0/3954836160 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0010068580 0x7f00100689e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:52.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.954+0000 7f00156cd700 1 --2- 172.21.15.67:0/3954836160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0010106cd0 0x7f00101070b0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:52.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.954+0000 7f00156cd700 1 -- 172.21.15.67:0/3954836160 >> 172.21.15.67:0/3954836160 conn(0x7f0010075670 msgr2=0x7f0010075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:35:52.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.954+0000 7f00156cd700 1 -- 172.21.15.67:0/3954836160 shutdown_connections 2024-04-22T23:35:52.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.954+0000 7f00156cd700 1 -- 172.21.15.67:0/3954836160 wait complete. 2024-04-22T23:35:52.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.955+0000 7f00156cd700 1 Processor -- start 2024-04-22T23:35:52.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.955+0000 7f00156cd700 1 -- start start 2024-04-22T23:35:52.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.955+0000 7f00156cd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0010068580 0x7f001019c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:52.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.955+0000 7f00156cd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0010106cd0 0x7f001019cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:52.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.955+0000 7f00156cd700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f001019d390 con 0x7f0010106cd0 2024-04-22T23:35:52.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.955+0000 7f00156cd700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0010196840 con 0x7f0010068580 2024-04-22T23:35:52.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.955+0000 7f000e7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0010106cd0 0x7f001019cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:52.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.955+0000 7f000e7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0010106cd0 0x7f001019cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50610/0 (socket says 172.21.15.67:50610) 2024-04-22T23:35:52.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.955+0000 7f000e7fc700 1 -- 172.21.15.67:0/4043597237 learned_addr learned my addr 172.21.15.67:0/4043597237 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:35:52.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.955+0000 7f000effd700 1 --2- 172.21.15.67:0/4043597237 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0010068580 0x7f001019c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:52.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.955+0000 7f000e7fc700 1 -- 172.21.15.67:0/4043597237 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0010068580 msgr2=0x7f001019c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:52.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.955+0000 7f000e7fc700 1 --2- 172.21.15.67:0/4043597237 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0010068580 0x7f001019c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:52.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.955+0000 7f000e7fc700 1 -- 172.21.15.67:0/4043597237 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0008009d00 con 0x7f0010106cd0 2024-04-22T23:35:52.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.956+0000 7f000e7fc700 1 --2- 172.21.15.67:0/4043597237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0010106cd0 0x7f001019cd00 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7efff800b4b0 tx=0x7efff800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:35:52.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.956+0000 7f0007fff700 1 -- 172.21.15.67:0/4043597237 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efff8011800 con 0x7f0010106cd0 2024-04-22T23:35:52.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.956+0000 7f00156cd700 1 -- 172.21.15.67:0/4043597237 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0010196b20 con 0x7f0010106cd0 2024-04-22T23:35:52.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.956+0000 7f0007fff700 1 -- 172.21.15.67:0/4043597237 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efff8011e20 con 0x7f0010106cd0 2024-04-22T23:35:52.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.956+0000 7f0007fff700 1 -- 172.21.15.67:0/4043597237 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efff800f2e0 con 0x7f0010106cd0 2024-04-22T23:35:52.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.956+0000 7f00156cd700 1 -- 172.21.15.67:0/4043597237 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0010197070 con 0x7f0010106cd0 2024-04-22T23:35:52.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.957+0000 7f00156cd700 1 -- 172.21.15.67:0/4043597237 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efff0005320 con 0x7f0010106cd0 2024-04-22T23:35:52.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.957+0000 7f0007fff700 1 -- 172.21.15.67:0/4043597237 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7efff8011960 con 0x7f0010106cd0 2024-04-22T23:35:52.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.958+0000 7f0007fff700 1 --2- 172.21.15.67:0/4043597237 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efffc06ecc0 0x7efffc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:52.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.958+0000 7f000effd700 1 --2- 172.21.15.67:0/4043597237 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efffc06ecc0 0x7efffc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:52.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.958+0000 7f0007fff700 1 -- 172.21.15.67:0/4043597237 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7efff808eb40 con 0x7f0010106cd0 2024-04-22T23:35:52.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.958+0000 7f000effd700 1 --2- 172.21.15.67:0/4043597237 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efffc06ecc0 0x7efffc071180 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0008015040 tx=0x7f0008000f40 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:35:52.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:52.963+0000 7f0007fff700 1 -- 172.21.15.67:0/4043597237 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7efff805b2b0 con 0x7f0010106cd0 2024-04-22T23:35:53.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.180+0000 7f00156cd700 1 -- 172.21.15.67:0/4043597237 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7efff0000bf0 con 0x7efffc06ecc0 2024-04-22T23:35:53.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.191+0000 7f0007fff700 1 -- 172.21.15.67:0/4043597237 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7efff0000bf0 con 0x7efffc06ecc0 2024-04-22T23:35:53.187 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:35:53.187 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (9m) 2m ago 20m 25.5M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:35:53.187 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (21m) 2m ago 21m 9114k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:35:53.187 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (20m) 2m ago 20m 9042k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:35:53.187 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (6m) 2m ago 21m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:35:53.187 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (6m) 2m ago 20m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:35:53.187 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (9m) 2m ago 20m 79.3M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:35:53.187 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (3m) 2m ago 15m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:35:53.187 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (2m) 2m ago 15m 12.6M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:35:53.188 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (2m) 2m ago 15m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:35:53.188 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (15m) 2m ago 15m 12.2M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:35:53.188 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (11m) 2m ago 22m 511M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:35:53.188 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (10m) 2m ago 19m 453M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:35:53.188 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (8m) 2m ago 22m 55.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:35:53.188 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (7m) 2m ago 19m 46.0M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:35:53.188 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (10m) 2m ago 20m 15.4M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:35:53.188 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (10m) 2m ago 19m 16.1M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:35:53.188 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (6m) 2m ago 19m 99.2M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:35:53.188 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (5m) 2m ago 18m 127M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:35:53.188 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (5m) 2m ago 18m 79.9M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:35:53.188 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (4m) 2m ago 17m 97.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:35:53.188 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (4m) 2m ago 17m 82.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:35:53.188 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (3m) 2m ago 16m 77.6M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:35:53.188 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (9m) 2m ago 20m 73.9M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:35:53.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.193+0000 7f00156cd700 1 -- 172.21.15.67:0/4043597237 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efffc06ecc0 msgr2=0x7efffc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:53.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.193+0000 7f00156cd700 1 --2- 172.21.15.67:0/4043597237 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efffc06ecc0 0x7efffc071180 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0008015040 tx=0x7f0008000f40 comp rx=0 tx=0).stop 2024-04-22T23:35:53.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.193+0000 7f00156cd700 1 -- 172.21.15.67:0/4043597237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0010106cd0 msgr2=0x7f001019cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:53.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.193+0000 7f00156cd700 1 --2- 172.21.15.67:0/4043597237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0010106cd0 0x7f001019cd00 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7efff800b4b0 tx=0x7efff800b870 comp rx=0 tx=0).stop 2024-04-22T23:35:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.193+0000 7f00156cd700 1 -- 172.21.15.67:0/4043597237 shutdown_connections 2024-04-22T23:35:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.193+0000 7f00156cd700 1 --2- 172.21.15.67:0/4043597237 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efffc06ecc0 0x7efffc071180 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.193+0000 7f00156cd700 1 --2- 172.21.15.67:0/4043597237 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0010068580 0x7f001019c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.193+0000 7f00156cd700 1 --2- 172.21.15.67:0/4043597237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0010106cd0 0x7f001019cd00 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.193+0000 7f00156cd700 1 -- 172.21.15.67:0/4043597237 >> 172.21.15.67:0/4043597237 conn(0x7f0010075670 msgr2=0x7f00100fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:35:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.194+0000 7f00156cd700 1 -- 172.21.15.67:0/4043597237 shutdown_connections 2024-04-22T23:35:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.194+0000 7f00156cd700 1 -- 172.21.15.67:0/4043597237 wait complete. 2024-04-22T23:35:53.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.300+0000 7f4872d69700 1 -- 172.21.15.67:0/3156787746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f486c101ea0 msgr2=0x7f486c102280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:53.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.300+0000 7f4872d69700 1 --2- 172.21.15.67:0/3156787746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f486c101ea0 0x7f486c102280 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4860009910 tx=0x7f4860009c20 comp rx=0 tx=0).stop 2024-04-22T23:35:53.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.300+0000 7f4872d69700 1 -- 172.21.15.67:0/3156787746 shutdown_connections 2024-04-22T23:35:53.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.300+0000 7f4872d69700 1 --2- 172.21.15.67:0/3156787746 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f486c1027c0 0x7f486c10acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:53.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.300+0000 7f4872d69700 1 --2- 172.21.15.67:0/3156787746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f486c101ea0 0x7f486c102280 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:53.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.300+0000 7f4872d69700 1 -- 172.21.15.67:0/3156787746 >> 172.21.15.67:0/3156787746 conn(0x7f486c0fb650 msgr2=0x7f486c0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:35:53.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.300+0000 7f4872d69700 1 -- 172.21.15.67:0/3156787746 shutdown_connections 2024-04-22T23:35:53.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.300+0000 7f4872d69700 1 -- 172.21.15.67:0/3156787746 wait complete. 2024-04-22T23:35:53.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.301+0000 7f4872d69700 1 Processor -- start 2024-04-22T23:35:53.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.301+0000 7f4872d69700 1 -- start start 2024-04-22T23:35:53.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.301+0000 7f4872d69700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f486c101ea0 0x7f486c19c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:53.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.301+0000 7f4872d69700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f486c1027c0 0x7f486c19cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:53.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.301+0000 7f4872d69700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f486c19d230 con 0x7f486c101ea0 2024-04-22T23:35:53.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.301+0000 7f4872d69700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f486c1966e0 con 0x7f486c1027c0 2024-04-22T23:35:53.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.301+0000 7f4870b05700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f486c101ea0 0x7f486c19c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:53.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.301+0000 7f4870b05700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f486c101ea0 0x7f486c19c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50618/0 (socket says 172.21.15.67:50618) 2024-04-22T23:35:53.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.301+0000 7f4870b05700 1 -- 172.21.15.67:0/4154751292 learned_addr learned my addr 172.21.15.67:0/4154751292 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:35:53.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.301+0000 7f486bfff700 1 --2- 172.21.15.67:0/4154751292 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f486c1027c0 0x7f486c19cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:53.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.301+0000 7f4870b05700 1 -- 172.21.15.67:0/4154751292 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f486c1027c0 msgr2=0x7f486c19cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:53.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.301+0000 7f4870b05700 1 --2- 172.21.15.67:0/4154751292 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f486c1027c0 0x7f486c19cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:53.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.301+0000 7f4870b05700 1 -- 172.21.15.67:0/4154751292 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48600095f0 con 0x7f486c101ea0 2024-04-22T23:35:53.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.301+0000 7f4870b05700 1 --2- 172.21.15.67:0/4154751292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f486c101ea0 0x7f486c19c660 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f486000b5c0 tx=0x7f486000bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:35:53.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.302+0000 7f4869ffb700 1 -- 172.21.15.67:0/4154751292 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f486001d070 con 0x7f486c101ea0 2024-04-22T23:35:53.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.302+0000 7f4869ffb700 1 -- 172.21.15.67:0/4154751292 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f486000f460 con 0x7f486c101ea0 2024-04-22T23:35:53.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.302+0000 7f4872d69700 1 -- 172.21.15.67:0/4154751292 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f486c196960 con 0x7f486c101ea0 2024-04-22T23:35:53.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.302+0000 7f4869ffb700 1 -- 172.21.15.67:0/4154751292 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4860017630 con 0x7f486c101ea0 2024-04-22T23:35:53.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.302+0000 7f4872d69700 1 -- 172.21.15.67:0/4154751292 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f486c196e50 con 0x7f486c101ea0 2024-04-22T23:35:53.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.303+0000 7f4872d69700 1 -- 172.21.15.67:0/4154751292 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f486c108350 con 0x7f486c101ea0 2024-04-22T23:35:53.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.303+0000 7f4869ffb700 1 -- 172.21.15.67:0/4154751292 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f486000f5d0 con 0x7f486c101ea0 2024-04-22T23:35:53.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.304+0000 7f4869ffb700 1 --2- 172.21.15.67:0/4154751292 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f485c06ecc0 0x7f485c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:53.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.304+0000 7f4869ffb700 1 -- 172.21.15.67:0/4154751292 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f4860090500 con 0x7f486c101ea0 2024-04-22T23:35:53.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.304+0000 7f486bfff700 1 --2- 172.21.15.67:0/4154751292 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f485c06ecc0 0x7f485c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:53.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.304+0000 7f486bfff700 1 --2- 172.21.15.67:0/4154751292 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f485c06ecc0 0x7f485c071180 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f486c197b50 tx=0x7f485800a300 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:35:53.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.309+0000 7f4869ffb700 1 -- 172.21.15.67:0/4154751292 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f486005cc70 con 0x7f486c101ea0 2024-04-22T23:35:53.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.604+0000 7f4872d69700 1 -- 172.21.15.67:0/4154751292 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f486c04f2e0 con 0x7f486c101ea0 2024-04-22T23:35:53.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.604+0000 7f4869ffb700 1 -- 172.21.15.67:0/4154751292 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f486005c800 con 0x7f486c101ea0 2024-04-22T23:35:53.600 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:35:53.600 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:35:53.600 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:35:53.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:35:53.600 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:35:53.600 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:35:53.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:35:53.600 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:35:53.600 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:35:53.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:35:53.601 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:35:53.601 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:35:53.601 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 3 2024-04-22T23:35:53.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:35:53.601 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:35:53.601 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:35:53.601 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 13 2024-04-22T23:35:53.601 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:35:53.601 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:35:53.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.606+0000 7f4872d69700 1 -- 172.21.15.67:0/4154751292 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f485c06ecc0 msgr2=0x7f485c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:53.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.606+0000 7f4872d69700 1 --2- 172.21.15.67:0/4154751292 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f485c06ecc0 0x7f485c071180 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f486c197b50 tx=0x7f485800a300 comp rx=0 tx=0).stop 2024-04-22T23:35:53.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.606+0000 7f4872d69700 1 -- 172.21.15.67:0/4154751292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f486c101ea0 msgr2=0x7f486c19c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:53.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.606+0000 7f4872d69700 1 --2- 172.21.15.67:0/4154751292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f486c101ea0 0x7f486c19c660 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f486000b5c0 tx=0x7f486000bd50 comp rx=0 tx=0).stop 2024-04-22T23:35:53.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.606+0000 7f4872d69700 1 -- 172.21.15.67:0/4154751292 shutdown_connections 2024-04-22T23:35:53.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.606+0000 7f4872d69700 1 --2- 172.21.15.67:0/4154751292 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f485c06ecc0 0x7f485c071180 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:53.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.606+0000 7f4872d69700 1 --2- 172.21.15.67:0/4154751292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f486c101ea0 0x7f486c19c660 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:53.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.606+0000 7f4872d69700 1 --2- 172.21.15.67:0/4154751292 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f486c1027c0 0x7f486c19cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:53.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.606+0000 7f4872d69700 1 -- 172.21.15.67:0/4154751292 >> 172.21.15.67:0/4154751292 conn(0x7f486c0fb650 msgr2=0x7f486c105490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:35:53.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.606+0000 7f4872d69700 1 -- 172.21.15.67:0/4154751292 shutdown_connections 2024-04-22T23:35:53.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.606+0000 7f4872d69700 1 -- 172.21.15.67:0/4154751292 wait complete. 2024-04-22T23:35:53.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.713+0000 7f59de2ac700 1 -- 172.21.15.67:0/705507474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59d80ffda0 msgr2=0x7f59d8100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:53.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.713+0000 7f59de2ac700 1 --2- 172.21.15.67:0/705507474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59d80ffda0 0x7f59d8100180 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f59c8009910 tx=0x7f59c8009c20 comp rx=0 tx=0).stop 2024-04-22T23:35:53.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.713+0000 7f59de2ac700 1 -- 172.21.15.67:0/705507474 shutdown_connections 2024-04-22T23:35:53.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.713+0000 7f59de2ac700 1 --2- 172.21.15.67:0/705507474 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59d8100750 0x7f59d81047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:53.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.713+0000 7f59de2ac700 1 --2- 172.21.15.67:0/705507474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59d80ffda0 0x7f59d8100180 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:53.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.713+0000 7f59de2ac700 1 -- 172.21.15.67:0/705507474 >> 172.21.15.67:0/705507474 conn(0x7f59d80fb650 msgr2=0x7f59d80fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:35:53.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.714+0000 7f59de2ac700 1 -- 172.21.15.67:0/705507474 shutdown_connections 2024-04-22T23:35:53.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.714+0000 7f59de2ac700 1 -- 172.21.15.67:0/705507474 wait complete. 2024-04-22T23:35:53.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.714+0000 7f59de2ac700 1 Processor -- start 2024-04-22T23:35:53.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.714+0000 7f59de2ac700 1 -- start start 2024-04-22T23:35:53.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.714+0000 7f59de2ac700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59d80ffda0 0x7f59d819e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:53.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.714+0000 7f59de2ac700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59d8100750 0x7f59d819edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:53.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.715+0000 7f59de2ac700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d819f3b0 con 0x7f59d8100750 2024-04-22T23:35:53.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.715+0000 7f59de2ac700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d81988f0 con 0x7f59d80ffda0 2024-04-22T23:35:53.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.715+0000 7f59d77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59d8100750 0x7f59d819edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:53.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.715+0000 7f59d77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59d8100750 0x7f59d819edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50638/0 (socket says 172.21.15.67:50638) 2024-04-22T23:35:53.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.715+0000 7f59d77fe700 1 -- 172.21.15.67:0/1784737865 learned_addr learned my addr 172.21.15.67:0/1784737865 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:35:53.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.715+0000 7f59d7fff700 1 --2- 172.21.15.67:0/1784737865 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59d80ffda0 0x7f59d819e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:53.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.715+0000 7f59d77fe700 1 -- 172.21.15.67:0/1784737865 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59d80ffda0 msgr2=0x7f59d819e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:53.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.715+0000 7f59d77fe700 1 --2- 172.21.15.67:0/1784737865 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59d80ffda0 0x7f59d819e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:53.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.715+0000 7f59d77fe700 1 -- 172.21.15.67:0/1784737865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c80095f0 con 0x7f59d8100750 2024-04-22T23:35:53.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.715+0000 7f59d77fe700 1 --2- 172.21.15.67:0/1784737865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59d8100750 0x7f59d819edb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f59cc00c960 tx=0x7f59cc00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:35:53.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.715+0000 7f59d57fa700 1 -- 172.21.15.67:0/1784737865 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59cc007a70 con 0x7f59d8100750 2024-04-22T23:35:53.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.715+0000 7f59de2ac700 1 -- 172.21.15.67:0/1784737865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59d8198bd0 con 0x7f59d8100750 2024-04-22T23:35:53.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.715+0000 7f59d57fa700 1 -- 172.21.15.67:0/1784737865 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f59cc007bd0 con 0x7f59d8100750 2024-04-22T23:35:53.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.715+0000 7f59d57fa700 1 -- 172.21.15.67:0/1784737865 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59cc018630 con 0x7f59d8100750 2024-04-22T23:35:53.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.715+0000 7f59de2ac700 1 -- 172.21.15.67:0/1784737865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59d8199120 con 0x7f59d8100750 2024-04-22T23:35:53.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.716+0000 7f59de2ac700 1 -- 172.21.15.67:0/1784737865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59d804f2e0 con 0x7f59d8100750 2024-04-22T23:35:53.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.717+0000 7f59d57fa700 1 -- 172.21.15.67:0/1784737865 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f59cc00f450 con 0x7f59d8100750 2024-04-22T23:35:53.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.717+0000 7f59d57fa700 1 --2- 172.21.15.67:0/1784737865 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f59c006ecc0 0x7f59c0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:53.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.718+0000 7f59d57fa700 1 -- 172.21.15.67:0/1784737865 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f59cc08ef30 con 0x7f59d8100750 2024-04-22T23:35:53.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.718+0000 7f59d7fff700 1 --2- 172.21.15.67:0/1784737865 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f59c006ecc0 0x7f59c0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:53.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.718+0000 7f59d7fff700 1 --2- 172.21.15.67:0/1784737865 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f59c006ecc0 0x7f59c0071180 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f59c8006010 tx=0x7f59c801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:35:53.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.722+0000 7f59d57fa700 1 -- 172.21.15.67:0/1784737865 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f59cc0c4150 con 0x7f59d8100750 2024-04-22T23:35:53.943 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:53 smithi134 ceph-mon[85630]: pgmap v285: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:53.943 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:53 smithi134 ceph-mon[85630]: from='client.34568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:35:53.943 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:53 smithi134 ceph-mon[85630]: from='client.34572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:35:53.943 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:53 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/4154751292' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:35:53.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.976+0000 7f59de2ac700 1 -- 172.21.15.67:0/1784737865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f59d8198d60 con 0x7f59d8100750 2024-04-22T23:35:53.972 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:53 smithi067 ceph-mon[112438]: pgmap v285: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:53.973 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:53 smithi067 ceph-mon[112438]: from='client.34568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:35:53.973 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:53 smithi067 ceph-mon[112438]: from='client.34572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:35:53.973 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:53 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/4154751292' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:35:53.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.977+0000 7f59d57fa700 1 -- 172.21.15.67:0/1784737865 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1871 (secure 0 0 0) 0x7f59d8198d60 con 0x7f59d8100750 2024-04-22T23:35:53.973 INFO:teuthology.orchestra.run.smithi067.stdout:e31 2024-04-22T23:35:53.973 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:35:53.973 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:35:53.974 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:35:53.974 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:35:53.974 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:35:53.974 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:35:53.974 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:35:53.974 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:35:53.974 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:35:53.974 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:35:53.974 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:35:53.974 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:35:53.974 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:35:53.975 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:35:53.975 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:35:53.975 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:35:53.975 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:35:53.975 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:35:53.975 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:35:53.975 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:35:53.975 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:35:53.975 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:35:53.975 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:35:53.975 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:35:53.975 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:35:53.976 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:35:53.976 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:35:53.976 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:35:53.976 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:35:53.976 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:35:53.976 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:35:53.976 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:35:53.976 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:35:53.976 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:35:53.976 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:35:53.976 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:35:53.976 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:35:53.976 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:35:53.977 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:35:53.977 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44259} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1827288036,v1:172.21.15.134:6827/1827288036] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:35:53.977 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:35:53.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.980+0000 7f59de2ac700 1 -- 172.21.15.67:0/1784737865 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f59c006ecc0 msgr2=0x7f59c0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:53.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.980+0000 7f59de2ac700 1 --2- 172.21.15.67:0/1784737865 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f59c006ecc0 0x7f59c0071180 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f59c8006010 tx=0x7f59c801a040 comp rx=0 tx=0).stop 2024-04-22T23:35:53.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.980+0000 7f59de2ac700 1 -- 172.21.15.67:0/1784737865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59d8100750 msgr2=0x7f59d819edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:53.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.980+0000 7f59de2ac700 1 --2- 172.21.15.67:0/1784737865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59d8100750 0x7f59d819edb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f59cc00c960 tx=0x7f59cc00cc70 comp rx=0 tx=0).stop 2024-04-22T23:35:53.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.980+0000 7f59de2ac700 1 -- 172.21.15.67:0/1784737865 shutdown_connections 2024-04-22T23:35:53.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.980+0000 7f59de2ac700 1 --2- 172.21.15.67:0/1784737865 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f59c006ecc0 0x7f59c0071180 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:53.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.980+0000 7f59de2ac700 1 --2- 172.21.15.67:0/1784737865 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59d80ffda0 0x7f59d819e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:53.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.980+0000 7f59de2ac700 1 --2- 172.21.15.67:0/1784737865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59d8100750 0x7f59d819edb0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:53.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.980+0000 7f59de2ac700 1 -- 172.21.15.67:0/1784737865 >> 172.21.15.67:0/1784737865 conn(0x7f59d80fb650 msgr2=0x7f59d80696c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:35:53.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.980+0000 7f59de2ac700 1 -- 172.21.15.67:0/1784737865 shutdown_connections 2024-04-22T23:35:53.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:53.980+0000 7f59de2ac700 1 -- 172.21.15.67:0/1784737865 wait complete. 2024-04-22T23:35:53.978 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 31 2024-04-22T23:35:54.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.089+0000 7f707984d700 1 -- 172.21.15.67:0/2627516092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7074101850 msgr2=0x7f70741058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:54.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.089+0000 7f707984d700 1 --2- 172.21.15.67:0/2627516092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7074101850 0x7f70741058a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7068009910 tx=0x7f7068009c20 comp rx=0 tx=0).stop 2024-04-22T23:35:54.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.089+0000 7f707984d700 1 -- 172.21.15.67:0/2627516092 shutdown_connections 2024-04-22T23:35:54.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.089+0000 7f707984d700 1 --2- 172.21.15.67:0/2627516092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7074101850 0x7f70741058a0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:54.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.089+0000 7f707984d700 1 --2- 172.21.15.67:0/2627516092 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7074100ea0 0x7f7074101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:54.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.089+0000 7f707984d700 1 -- 172.21.15.67:0/2627516092 >> 172.21.15.67:0/2627516092 conn(0x7f70740fc750 msgr2=0x7f70740feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:35:54.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.089+0000 7f707984d700 1 -- 172.21.15.67:0/2627516092 shutdown_connections 2024-04-22T23:35:54.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.089+0000 7f707984d700 1 -- 172.21.15.67:0/2627516092 wait complete. 2024-04-22T23:35:54.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.089+0000 7f707984d700 1 Processor -- start 2024-04-22T23:35:54.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.090+0000 7f707984d700 1 -- start start 2024-04-22T23:35:54.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.090+0000 7f707984d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7074100ea0 0x7f707419c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:54.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.090+0000 7f707984d700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7074101850 0x7f707419cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:54.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.090+0000 7f707984d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f707419d2d0 con 0x7f7074100ea0 2024-04-22T23:35:54.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.090+0000 7f707984d700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7074196780 con 0x7f7074101850 2024-04-22T23:35:54.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.090+0000 7f7072ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7074100ea0 0x7f707419c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:54.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.090+0000 7f7072ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7074100ea0 0x7f707419c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50660/0 (socket says 172.21.15.67:50660) 2024-04-22T23:35:54.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.090+0000 7f7072ffd700 1 -- 172.21.15.67:0/383560792 learned_addr learned my addr 172.21.15.67:0/383560792 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:35:54.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.090+0000 7f70727fc700 1 --2- 172.21.15.67:0/383560792 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7074101850 0x7f707419cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:54.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.090+0000 7f7072ffd700 1 -- 172.21.15.67:0/383560792 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7074101850 msgr2=0x7f707419cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:54.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.090+0000 7f7072ffd700 1 --2- 172.21.15.67:0/383560792 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7074101850 0x7f707419cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:54.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.090+0000 7f7072ffd700 1 -- 172.21.15.67:0/383560792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70680095f0 con 0x7f7074100ea0 2024-04-22T23:35:54.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.090+0000 7f7072ffd700 1 --2- 172.21.15.67:0/383560792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7074100ea0 0x7f707419c700 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f706400d8d0 tx=0x7f706400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:35:54.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.091+0000 7f707884b700 1 -- 172.21.15.67:0/383560792 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f70640098a0 con 0x7f7074100ea0 2024-04-22T23:35:54.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.091+0000 7f707884b700 1 -- 172.21.15.67:0/383560792 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7064010460 con 0x7f7074100ea0 2024-04-22T23:35:54.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.091+0000 7f707984d700 1 -- 172.21.15.67:0/383560792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7074196a60 con 0x7f7074100ea0 2024-04-22T23:35:54.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.091+0000 7f707884b700 1 -- 172.21.15.67:0/383560792 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f706400f5f0 con 0x7f7074100ea0 2024-04-22T23:35:54.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.091+0000 7f707984d700 1 -- 172.21.15.67:0/383560792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7074196f80 con 0x7f7074100ea0 2024-04-22T23:35:54.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.092+0000 7f707984d700 1 -- 172.21.15.67:0/383560792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70741092a0 con 0x7f7074100ea0 2024-04-22T23:35:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.092+0000 7f707884b700 1 -- 172.21.15.67:0/383560792 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7064009a00 con 0x7f7074100ea0 2024-04-22T23:35:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.093+0000 7f707884b700 1 --2- 172.21.15.67:0/383560792 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f705c06ecc0 0x7f705c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.093+0000 7f707884b700 1 -- 172.21.15.67:0/383560792 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f706408eea0 con 0x7f7074100ea0 2024-04-22T23:35:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.093+0000 7f70727fc700 1 --2- 172.21.15.67:0/383560792 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f705c06ecc0 0x7f705c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.093+0000 7f70727fc700 1 --2- 172.21.15.67:0/383560792 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f705c06ecc0 0x7f705c071180 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7068006010 tx=0x7f706801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:35:54.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.098+0000 7f707884b700 1 -- 172.21.15.67:0/383560792 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f706405b610 con 0x7f7074100ea0 2024-04-22T23:35:54.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.328+0000 7f707984d700 1 -- 172.21.15.67:0/383560792 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7074197da0 con 0x7f705c06ecc0 2024-04-22T23:35:54.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.329+0000 7f707884b700 1 -- 172.21.15.67:0/383560792 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f7074197da0 con 0x7f705c06ecc0 2024-04-22T23:35:54.325 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:35:54.325 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:35:54.325 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:35:54.325 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:35:54.325 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:35:54.325 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:35:54.325 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:35:54.325 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:35:54.326 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:35:54.326 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:35:54.326 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T23:35:54.326 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:35:54.326 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:35:54.326 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:35:54.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.332+0000 7f707984d700 1 -- 172.21.15.67:0/383560792 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f705c06ecc0 msgr2=0x7f705c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:54.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.332+0000 7f707984d700 1 --2- 172.21.15.67:0/383560792 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f705c06ecc0 0x7f705c071180 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7068006010 tx=0x7f706801a040 comp rx=0 tx=0).stop 2024-04-22T23:35:54.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.332+0000 7f707984d700 1 -- 172.21.15.67:0/383560792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7074100ea0 msgr2=0x7f707419c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:54.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.332+0000 7f707984d700 1 --2- 172.21.15.67:0/383560792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7074100ea0 0x7f707419c700 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f706400d8d0 tx=0x7f706400dbe0 comp rx=0 tx=0).stop 2024-04-22T23:35:54.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.332+0000 7f707984d700 1 -- 172.21.15.67:0/383560792 shutdown_connections 2024-04-22T23:35:54.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.332+0000 7f707984d700 1 --2- 172.21.15.67:0/383560792 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f705c06ecc0 0x7f705c071180 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:54.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.332+0000 7f707984d700 1 --2- 172.21.15.67:0/383560792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7074100ea0 0x7f707419c700 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:54.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.332+0000 7f707984d700 1 --2- 172.21.15.67:0/383560792 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7074101850 0x7f707419cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:54.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.332+0000 7f707984d700 1 -- 172.21.15.67:0/383560792 >> 172.21.15.67:0/383560792 conn(0x7f70740fc750 msgr2=0x7f70740fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:35:54.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.332+0000 7f707984d700 1 -- 172.21.15.67:0/383560792 shutdown_connections 2024-04-22T23:35:54.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.332+0000 7f707984d700 1 -- 172.21.15.67:0/383560792 wait complete. 2024-04-22T23:35:54.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.439+0000 7f5ac09c9700 1 -- 172.21.15.67:0/394843631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5abc1006b0 msgr2=0x7f5abc100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:54.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.439+0000 7f5ac09c9700 1 --2- 172.21.15.67:0/394843631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5abc1006b0 0x7f5abc100a90 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5aac009910 tx=0x7f5aac009c20 comp rx=0 tx=0).stop 2024-04-22T23:35:54.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.439+0000 7f5ac09c9700 1 -- 172.21.15.67:0/394843631 shutdown_connections 2024-04-22T23:35:54.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.439+0000 7f5ac09c9700 1 --2- 172.21.15.67:0/394843631 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5abc101060 0x7f5abc1047b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:54.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.439+0000 7f5ac09c9700 1 --2- 172.21.15.67:0/394843631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5abc1006b0 0x7f5abc100a90 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:54.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.439+0000 7f5ac09c9700 1 -- 172.21.15.67:0/394843631 >> 172.21.15.67:0/394843631 conn(0x7f5abc074e30 msgr2=0x7f5abc075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:35:54.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.439+0000 7f5ac09c9700 1 -- 172.21.15.67:0/394843631 shutdown_connections 2024-04-22T23:35:54.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.439+0000 7f5ac09c9700 1 -- 172.21.15.67:0/394843631 wait complete. 2024-04-22T23:35:54.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.440+0000 7f5ac09c9700 1 Processor -- start 2024-04-22T23:35:54.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.440+0000 7f5ac09c9700 1 -- start start 2024-04-22T23:35:54.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.440+0000 7f5ac09c9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5abc1006b0 0x7f5abc19b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:54.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.440+0000 7f5ac09c9700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5abc101060 0x7f5abc19ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:54.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.440+0000 7f5ac09c9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5abc19c070 con 0x7f5abc1006b0 2024-04-22T23:35:54.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.440+0000 7f5ac09c9700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5abc1955b0 con 0x7f5abc101060 2024-04-22T23:35:54.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.440+0000 7f5aba59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5abc1006b0 0x7f5abc19b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:54.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.440+0000 7f5aba59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5abc1006b0 0x7f5abc19b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50678/0 (socket says 172.21.15.67:50678) 2024-04-22T23:35:54.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.440+0000 7f5aba59c700 1 -- 172.21.15.67:0/3188540519 learned_addr learned my addr 172.21.15.67:0/3188540519 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:35:54.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.440+0000 7f5ab9d9b700 1 --2- 172.21.15.67:0/3188540519 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5abc101060 0x7f5abc19ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:54.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.440+0000 7f5aba59c700 1 -- 172.21.15.67:0/3188540519 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5abc101060 msgr2=0x7f5abc19ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:54.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.440+0000 7f5aba59c700 1 --2- 172.21.15.67:0/3188540519 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5abc101060 0x7f5abc19ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:54.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.440+0000 7f5aba59c700 1 -- 172.21.15.67:0/3188540519 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aac0095f0 con 0x7f5abc1006b0 2024-04-22T23:35:54.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.441+0000 7f5aba59c700 1 --2- 172.21.15.67:0/3188540519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5abc1006b0 0x7f5abc19b530 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f5aac006010 tx=0x7f5aac00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:35:54.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.441+0000 7f5ab37fe700 1 -- 172.21.15.67:0/3188540519 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5aac01f070 con 0x7f5abc1006b0 2024-04-22T23:35:54.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.441+0000 7f5ab37fe700 1 -- 172.21.15.67:0/3188540519 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5aac019610 con 0x7f5abc1006b0 2024-04-22T23:35:54.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.441+0000 7f5ac09c9700 1 -- 172.21.15.67:0/3188540519 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5abc195830 con 0x7f5abc1006b0 2024-04-22T23:35:54.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.441+0000 7f5ab37fe700 1 -- 172.21.15.67:0/3188540519 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5aac00f690 con 0x7f5abc1006b0 2024-04-22T23:35:54.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.441+0000 7f5ac09c9700 1 -- 172.21.15.67:0/3188540519 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5abc195d20 con 0x7f5abc1006b0 2024-04-22T23:35:54.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.442+0000 7f5ac09c9700 1 -- 172.21.15.67:0/3188540519 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5abc04f2e0 con 0x7f5abc1006b0 2024-04-22T23:35:54.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.442+0000 7f5ab37fe700 1 -- 172.21.15.67:0/3188540519 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5aac019780 con 0x7f5abc1006b0 2024-04-22T23:35:54.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.443+0000 7f5ab37fe700 1 --2- 172.21.15.67:0/3188540519 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5aa806ecc0 0x7f5aa8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:35:54.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.443+0000 7f5ab37fe700 1 -- 172.21.15.67:0/3188540519 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f5aac092850 con 0x7f5abc1006b0 2024-04-22T23:35:54.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.443+0000 7f5ab9d9b700 1 --2- 172.21.15.67:0/3188540519 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5aa806ecc0 0x7f5aa8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:35:54.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.443+0000 7f5ab9d9b700 1 --2- 172.21.15.67:0/3188540519 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5aa806ecc0 0x7f5aa8071180 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5abc106770 tx=0x7f5aa400a380 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:35:54.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.448+0000 7f5ab37fe700 1 -- 172.21.15.67:0/3188540519 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f5aac05efc0 con 0x7f5abc1006b0 2024-04-22T23:35:54.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.757+0000 7f5ac09c9700 1 -- 172.21.15.67:0/3188540519 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f5abc195f90 con 0x7f5abc1006b0 2024-04-22T23:35:54.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.757+0000 7f5ab37fe700 1 -- 172.21.15.67:0/3188540519 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f5aac05eb50 con 0x7f5abc1006b0 2024-04-22T23:35:54.753 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:35:54.753 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:35:54.753 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:35:54.754 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:35:54.754 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:35:54.754 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:35:54.754 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:35:54.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.759+0000 7f5ac09c9700 1 -- 172.21.15.67:0/3188540519 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5aa806ecc0 msgr2=0x7f5aa8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:54.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.759+0000 7f5ac09c9700 1 --2- 172.21.15.67:0/3188540519 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5aa806ecc0 0x7f5aa8071180 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5abc106770 tx=0x7f5aa400a380 comp rx=0 tx=0).stop 2024-04-22T23:35:54.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.759+0000 7f5ac09c9700 1 -- 172.21.15.67:0/3188540519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5abc1006b0 msgr2=0x7f5abc19b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:35:54.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.759+0000 7f5ac09c9700 1 --2- 172.21.15.67:0/3188540519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5abc1006b0 0x7f5abc19b530 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f5aac006010 tx=0x7f5aac00bd70 comp rx=0 tx=0).stop 2024-04-22T23:35:54.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.760+0000 7f5ac09c9700 1 -- 172.21.15.67:0/3188540519 shutdown_connections 2024-04-22T23:35:54.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.760+0000 7f5ac09c9700 1 --2- 172.21.15.67:0/3188540519 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5aa806ecc0 0x7f5aa8071180 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:54.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.760+0000 7f5ac09c9700 1 --2- 172.21.15.67:0/3188540519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5abc1006b0 0x7f5abc19b530 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:54.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.760+0000 7f5ac09c9700 1 --2- 172.21.15.67:0/3188540519 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5abc101060 0x7f5abc19ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:35:54.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.760+0000 7f5ac09c9700 1 -- 172.21.15.67:0/3188540519 >> 172.21.15.67:0/3188540519 conn(0x7f5abc074e30 msgr2=0x7f5abc0fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:35:54.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.760+0000 7f5ac09c9700 1 -- 172.21.15.67:0/3188540519 shutdown_connections 2024-04-22T23:35:54.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:35:54.760+0000 7f5ac09c9700 1 -- 172.21.15.67:0/3188540519 wait complete. 2024-04-22T23:35:54.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:54 smithi134 ceph-mon[85630]: from='client.34576 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:35:54.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:54 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1784737865' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:35:55.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:54 smithi067 ceph-mon[112438]: from='client.34576 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:35:55.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:54 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1784737865' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:35:55.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:55 smithi134 ceph-mon[85630]: pgmap v286: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:55.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:55 smithi134 ceph-mon[85630]: from='client.34588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:35:55.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:55 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3188540519' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:35:56.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:55 smithi067 ceph-mon[112438]: pgmap v286: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:56.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:55 smithi067 ceph-mon[112438]: from='client.34588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:35:56.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:55 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3188540519' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:35:56.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:35:57.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:35:57.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:57 smithi134 ceph-mon[85630]: pgmap v287: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:58.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:57 smithi067 ceph-mon[112438]: pgmap v287: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:59.788 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:35:59 smithi067 ceph-mon[112438]: pgmap v288: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:35:59.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:35:59 smithi134 ceph-mon[85630]: pgmap v288: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:01.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:01 smithi134 ceph-mon[85630]: pgmap v289: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:02.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:01 smithi067 ceph-mon[112438]: pgmap v289: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:03.987 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:03 smithi134 ceph-mon[85630]: pgmap v290: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:04.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:03 smithi067 ceph-mon[112438]: pgmap v290: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:05.987 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:05 smithi134 ceph-mon[85630]: pgmap v291: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:06.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:05 smithi067 ceph-mon[112438]: pgmap v291: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:07.987 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:07 smithi134 ceph-mon[85630]: pgmap v292: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:08.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:07 smithi067 ceph-mon[112438]: pgmap v292: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:09.987 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:09 smithi134 ceph-mon[85630]: pgmap v293: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:10.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:09 smithi067 ceph-mon[112438]: pgmap v293: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:11.987 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:11 smithi134 ceph-mon[85630]: pgmap v294: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:11.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:36:12.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:11 smithi067 ceph-mon[112438]: pgmap v294: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:12.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:36:13.990 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:13 smithi134 ceph-mon[85630]: pgmap v295: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:14.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:13 smithi067 ceph-mon[112438]: pgmap v295: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:15.987 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:15 smithi134 ceph-mon[85630]: pgmap v296: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:16.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:15 smithi067 ceph-mon[112438]: pgmap v296: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:17.987 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:17 smithi134 ceph-mon[85630]: pgmap v297: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:18.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:17 smithi067 ceph-mon[112438]: pgmap v297: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:19.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:19 smithi134 ceph-mon[85630]: pgmap v298: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:20.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:19 smithi067 ceph-mon[112438]: pgmap v298: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:22.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:21 smithi067 ceph-mon[112438]: pgmap v299: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:22.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:21 smithi134 ceph-mon[85630]: pgmap v299: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:24.041 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:23 smithi067 ceph-mon[112438]: pgmap v300: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:24.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:23 smithi134 ceph-mon[85630]: pgmap v300: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:24.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:24.888+0000 7f72c4155700 1 -- 172.21.15.67:0/1636024885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72bc068580 msgr2=0x7f72bc0689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:24.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:24.888+0000 7f72c4155700 1 --2- 172.21.15.67:0/1636024885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72bc068580 0x7f72bc0689e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f72b0009910 tx=0x7f72b0009c20 comp rx=0 tx=0).stop 2024-04-22T23:36:24.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:24.888+0000 7f72c4155700 1 -- 172.21.15.67:0/1636024885 shutdown_connections 2024-04-22T23:36:24.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:24.888+0000 7f72c4155700 1 --2- 172.21.15.67:0/1636024885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72bc068580 0x7f72bc0689e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:24.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:24.888+0000 7f72c4155700 1 --2- 172.21.15.67:0/1636024885 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f72bc106cd0 0x7f72bc1070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:24.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:24.888+0000 7f72c4155700 1 -- 172.21.15.67:0/1636024885 >> 172.21.15.67:0/1636024885 conn(0x7f72bc075670 msgr2=0x7f72bc075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:36:24.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:24.888+0000 7f72c4155700 1 -- 172.21.15.67:0/1636024885 shutdown_connections 2024-04-22T23:36:24.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:24.888+0000 7f72c4155700 1 -- 172.21.15.67:0/1636024885 wait complete. 2024-04-22T23:36:24.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:24.889+0000 7f72c4155700 1 Processor -- start 2024-04-22T23:36:24.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:24.889+0000 7f72c4155700 1 -- start start 2024-04-22T23:36:24.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:24.889+0000 7f72c4155700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f72bc068580 0x7f72bc0ffcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:24.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:24.889+0000 7f72c4155700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72bc106cd0 0x7f72bc100230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:24.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:24.889+0000 7f72c4155700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72bc100770 con 0x7f72bc106cd0 2024-04-22T23:36:24.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:24.889+0000 7f72c4155700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72bc1008b0 con 0x7f72bc068580 2024-04-22T23:36:24.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:24.889+0000 7f72c16f0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72bc106cd0 0x7f72bc100230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:24.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:24.890+0000 7f72c16f0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72bc106cd0 0x7f72bc100230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60958/0 (socket says 172.21.15.67:60958) 2024-04-22T23:36:24.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:24.890+0000 7f72c16f0700 1 -- 172.21.15.67:0/177599519 learned_addr learned my addr 172.21.15.67:0/177599519 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:36:24.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:24.890+0000 7f72c1ef1700 1 --2- 172.21.15.67:0/177599519 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f72bc068580 0x7f72bc0ffcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:24.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:24.890+0000 7f72c16f0700 1 -- 172.21.15.67:0/177599519 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f72bc068580 msgr2=0x7f72bc0ffcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:24.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:24.890+0000 7f72c16f0700 1 --2- 172.21.15.67:0/177599519 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f72bc068580 0x7f72bc0ffcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:24.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:24.890+0000 7f72c16f0700 1 -- 172.21.15.67:0/177599519 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b00095f0 con 0x7f72bc106cd0 2024-04-22T23:36:24.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:24.890+0000 7f72c16f0700 1 --2- 172.21.15.67:0/177599519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72bc106cd0 0x7f72bc100230 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f72b0006010 tx=0x7f72b000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:36:24.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:24.890+0000 7f72aeffd700 1 -- 172.21.15.67:0/177599519 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72b001d070 con 0x7f72bc106cd0 2024-04-22T23:36:24.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:24.890+0000 7f72c4155700 1 -- 172.21.15.67:0/177599519 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72bc100b30 con 0x7f72bc106cd0 2024-04-22T23:36:24.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:24.890+0000 7f72aeffd700 1 -- 172.21.15.67:0/177599519 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f72b0004fb0 con 0x7f72bc106cd0 2024-04-22T23:36:24.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:24.890+0000 7f72aeffd700 1 -- 172.21.15.67:0/177599519 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72b0022ea0 con 0x7f72bc106cd0 2024-04-22T23:36:24.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:24.890+0000 7f72c4155700 1 -- 172.21.15.67:0/177599519 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72bc1a2cb0 con 0x7f72bc106cd0 2024-04-22T23:36:24.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:24.891+0000 7f72c4155700 1 -- 172.21.15.67:0/177599519 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72bc1091d0 con 0x7f72bc106cd0 2024-04-22T23:36:24.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:24.892+0000 7f72aeffd700 1 -- 172.21.15.67:0/177599519 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f72b0022470 con 0x7f72bc106cd0 2024-04-22T23:36:24.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:24.892+0000 7f72aeffd700 1 --2- 172.21.15.67:0/177599519 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f72a806ecc0 0x7f72a8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:24.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:24.892+0000 7f72aeffd700 1 -- 172.21.15.67:0/177599519 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f72b0090510 con 0x7f72bc106cd0 2024-04-22T23:36:24.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:24.892+0000 7f72c1ef1700 1 --2- 172.21.15.67:0/177599519 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f72a806ecc0 0x7f72a8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:24.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:24.893+0000 7f72c1ef1700 1 --2- 172.21.15.67:0/177599519 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f72a806ecc0 0x7f72a8071180 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f72b8005ea0 tx=0x7f72b8005e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:36:24.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:24.897+0000 7f72aeffd700 1 -- 172.21.15.67:0/177599519 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f72b00591e0 con 0x7f72bc106cd0 2024-04-22T23:36:25.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.129+0000 7f72c4155700 1 -- 172.21.15.67:0/177599519 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f72bc101290 con 0x7f72a806ecc0 2024-04-22T23:36:25.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.131+0000 7f72aeffd700 1 -- 172.21.15.67:0/177599519 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f72bc101290 con 0x7f72a806ecc0 2024-04-22T23:36:25.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.132+0000 7f72c4155700 1 -- 172.21.15.67:0/177599519 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f72a806ecc0 msgr2=0x7f72a8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:25.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.132+0000 7f72c4155700 1 --2- 172.21.15.67:0/177599519 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f72a806ecc0 0x7f72a8071180 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f72b8005ea0 tx=0x7f72b8005e30 comp rx=0 tx=0).stop 2024-04-22T23:36:25.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.133+0000 7f72c4155700 1 -- 172.21.15.67:0/177599519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72bc106cd0 msgr2=0x7f72bc100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:25.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.133+0000 7f72c4155700 1 --2- 172.21.15.67:0/177599519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72bc106cd0 0x7f72bc100230 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f72b0006010 tx=0x7f72b000be20 comp rx=0 tx=0).stop 2024-04-22T23:36:25.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.133+0000 7f72c4155700 1 -- 172.21.15.67:0/177599519 shutdown_connections 2024-04-22T23:36:25.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.133+0000 7f72c4155700 1 --2- 172.21.15.67:0/177599519 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f72a806ecc0 0x7f72a8071180 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:25.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.133+0000 7f72c4155700 1 --2- 172.21.15.67:0/177599519 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f72bc068580 0x7f72bc0ffcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:25.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.133+0000 7f72c4155700 1 --2- 172.21.15.67:0/177599519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72bc106cd0 0x7f72bc100230 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:25.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.133+0000 7f72c4155700 1 -- 172.21.15.67:0/177599519 >> 172.21.15.67:0/177599519 conn(0x7f72bc075670 msgr2=0x7f72bc0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:36:25.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.133+0000 7f72c4155700 1 -- 172.21.15.67:0/177599519 shutdown_connections 2024-04-22T23:36:25.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.133+0000 7f72c4155700 1 -- 172.21.15.67:0/177599519 wait complete. 2024-04-22T23:36:25.138 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:36:25.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.241+0000 7f743f098700 1 -- 172.21.15.67:0/1326036333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7438068580 msgr2=0x7f74380689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:25.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.241+0000 7f743f098700 1 --2- 172.21.15.67:0/1326036333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7438068580 0x7f74380689e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7428009910 tx=0x7f7428009c20 comp rx=0 tx=0).stop 2024-04-22T23:36:25.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.241+0000 7f743f098700 1 -- 172.21.15.67:0/1326036333 shutdown_connections 2024-04-22T23:36:25.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.241+0000 7f743f098700 1 --2- 172.21.15.67:0/1326036333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7438068580 0x7f74380689e0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:25.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.241+0000 7f743f098700 1 --2- 172.21.15.67:0/1326036333 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7438106cd0 0x7f74381070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:25.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.241+0000 7f743f098700 1 -- 172.21.15.67:0/1326036333 >> 172.21.15.67:0/1326036333 conn(0x7f7438075670 msgr2=0x7f7438075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:36:25.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.241+0000 7f743f098700 1 -- 172.21.15.67:0/1326036333 shutdown_connections 2024-04-22T23:36:25.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.241+0000 7f743f098700 1 -- 172.21.15.67:0/1326036333 wait complete. 2024-04-22T23:36:25.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.242+0000 7f743f098700 1 Processor -- start 2024-04-22T23:36:25.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.242+0000 7f743f098700 1 -- start start 2024-04-22T23:36:25.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.242+0000 7f743f098700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7438068580 0x7f74380ffcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:25.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.242+0000 7f743f098700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7438106cd0 0x7f7438100230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:25.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.242+0000 7f743f098700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7438100770 con 0x7f7438106cd0 2024-04-22T23:36:25.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.242+0000 7f743f098700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74381008b0 con 0x7f7438068580 2024-04-22T23:36:25.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.242+0000 7f7437fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7438106cd0 0x7f7438100230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:25.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.242+0000 7f7437fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7438106cd0 0x7f7438100230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60974/0 (socket says 172.21.15.67:60974) 2024-04-22T23:36:25.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.242+0000 7f7437fff700 1 -- 172.21.15.67:0/3995949024 learned_addr learned my addr 172.21.15.67:0/3995949024 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:36:25.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.242+0000 7f743ce34700 1 --2- 172.21.15.67:0/3995949024 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7438068580 0x7f74380ffcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:25.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.242+0000 7f7437fff700 1 -- 172.21.15.67:0/3995949024 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7438068580 msgr2=0x7f74380ffcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:25.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.242+0000 7f7437fff700 1 --2- 172.21.15.67:0/3995949024 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7438068580 0x7f74380ffcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:25.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.242+0000 7f7437fff700 1 -- 172.21.15.67:0/3995949024 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74280095f0 con 0x7f7438106cd0 2024-04-22T23:36:25.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.243+0000 7f7437fff700 1 --2- 172.21.15.67:0/3995949024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7438106cd0 0x7f7438100230 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7428006010 tx=0x7f742800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:36:25.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.243+0000 7f7435ffb700 1 -- 172.21.15.67:0/3995949024 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f742801d070 con 0x7f7438106cd0 2024-04-22T23:36:25.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.243+0000 7f7435ffb700 1 -- 172.21.15.67:0/3995949024 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f742800f460 con 0x7f7438106cd0 2024-04-22T23:36:25.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.243+0000 7f743f098700 1 -- 172.21.15.67:0/3995949024 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7438100b30 con 0x7f7438106cd0 2024-04-22T23:36:25.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.243+0000 7f7435ffb700 1 -- 172.21.15.67:0/3995949024 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7428017630 con 0x7f7438106cd0 2024-04-22T23:36:25.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.243+0000 7f743f098700 1 -- 172.21.15.67:0/3995949024 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74381a2d30 con 0x7f7438106cd0 2024-04-22T23:36:25.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.244+0000 7f743f098700 1 -- 172.21.15.67:0/3995949024 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74381091d0 con 0x7f7438106cd0 2024-04-22T23:36:25.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.244+0000 7f7435ffb700 1 -- 172.21.15.67:0/3995949024 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f742800f5d0 con 0x7f7438106cd0 2024-04-22T23:36:25.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.245+0000 7f7435ffb700 1 --2- 172.21.15.67:0/3995949024 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f742406ecc0 0x7f7424071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:25.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.245+0000 7f7435ffb700 1 -- 172.21.15.67:0/3995949024 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f7428090400 con 0x7f7438106cd0 2024-04-22T23:36:25.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.245+0000 7f743ce34700 1 --2- 172.21.15.67:0/3995949024 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f742406ecc0 0x7f7424071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:25.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.245+0000 7f743ce34700 1 --2- 172.21.15.67:0/3995949024 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f742406ecc0 0x7f7424071180 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7420005950 tx=0x7f74200058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:36:25.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.250+0000 7f7435ffb700 1 -- 172.21.15.67:0/3995949024 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f74280590c0 con 0x7f7438106cd0 2024-04-22T23:36:25.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.480+0000 7f743f098700 1 -- 172.21.15.67:0/3995949024 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f74380611d0 con 0x7f742406ecc0 2024-04-22T23:36:25.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.481+0000 7f7435ffb700 1 -- 172.21.15.67:0/3995949024 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f74380611d0 con 0x7f742406ecc0 2024-04-22T23:36:25.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.483+0000 7f743f098700 1 -- 172.21.15.67:0/3995949024 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f742406ecc0 msgr2=0x7f7424071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:25.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.483+0000 7f743f098700 1 --2- 172.21.15.67:0/3995949024 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f742406ecc0 0x7f7424071180 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7420005950 tx=0x7f74200058e0 comp rx=0 tx=0).stop 2024-04-22T23:36:25.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.483+0000 7f743f098700 1 -- 172.21.15.67:0/3995949024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7438106cd0 msgr2=0x7f7438100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:25.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.483+0000 7f743f098700 1 --2- 172.21.15.67:0/3995949024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7438106cd0 0x7f7438100230 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7428006010 tx=0x7f742800be20 comp rx=0 tx=0).stop 2024-04-22T23:36:25.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.483+0000 7f743f098700 1 -- 172.21.15.67:0/3995949024 shutdown_connections 2024-04-22T23:36:25.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.483+0000 7f743f098700 1 --2- 172.21.15.67:0/3995949024 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f742406ecc0 0x7f7424071180 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:25.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.483+0000 7f743f098700 1 --2- 172.21.15.67:0/3995949024 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7438068580 0x7f74380ffcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:25.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.483+0000 7f743f098700 1 --2- 172.21.15.67:0/3995949024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7438106cd0 0x7f7438100230 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:25.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.483+0000 7f743f098700 1 -- 172.21.15.67:0/3995949024 >> 172.21.15.67:0/3995949024 conn(0x7f7438075670 msgr2=0x7f74380feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:36:25.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.483+0000 7f743f098700 1 -- 172.21.15.67:0/3995949024 shutdown_connections 2024-04-22T23:36:25.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.483+0000 7f743f098700 1 -- 172.21.15.67:0/3995949024 wait complete. 2024-04-22T23:36:25.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.590+0000 7f4efa019700 1 -- 172.21.15.67:0/2624252869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ef4101850 msgr2=0x7f4ef41058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:25.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.590+0000 7f4efa019700 1 --2- 172.21.15.67:0/2624252869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ef4101850 0x7f4ef41058a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f4edc009910 tx=0x7f4edc009c20 comp rx=0 tx=0).stop 2024-04-22T23:36:25.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.591+0000 7f4efa019700 1 -- 172.21.15.67:0/2624252869 shutdown_connections 2024-04-22T23:36:25.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.591+0000 7f4efa019700 1 --2- 172.21.15.67:0/2624252869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ef4101850 0x7f4ef41058a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:25.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.591+0000 7f4efa019700 1 --2- 172.21.15.67:0/2624252869 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4ef4100ea0 0x7f4ef4101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:25.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.591+0000 7f4efa019700 1 -- 172.21.15.67:0/2624252869 >> 172.21.15.67:0/2624252869 conn(0x7f4ef40fc750 msgr2=0x7f4ef40feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:36:25.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.591+0000 7f4efa019700 1 -- 172.21.15.67:0/2624252869 shutdown_connections 2024-04-22T23:36:25.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.591+0000 7f4efa019700 1 -- 172.21.15.67:0/2624252869 wait complete. 2024-04-22T23:36:25.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.591+0000 7f4efa019700 1 Processor -- start 2024-04-22T23:36:25.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.591+0000 7f4efa019700 1 -- start start 2024-04-22T23:36:25.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.592+0000 7f4efa019700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4ef4100ea0 0x7f4ef4073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:25.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.592+0000 7f4efa019700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ef4101850 0x7f4ef40735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:25.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.592+0000 7f4efa019700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef4073b10 con 0x7f4ef4101850 2024-04-22T23:36:25.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.592+0000 7f4efa019700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef4073c50 con 0x7f4ef4100ea0 2024-04-22T23:36:25.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.592+0000 7f4ef2ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ef4101850 0x7f4ef40735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:25.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.592+0000 7f4ef2ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ef4101850 0x7f4ef40735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60994/0 (socket says 172.21.15.67:60994) 2024-04-22T23:36:25.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.592+0000 7f4ef2ffd700 1 -- 172.21.15.67:0/719770721 learned_addr learned my addr 172.21.15.67:0/719770721 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:36:25.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.592+0000 7f4ef37fe700 1 --2- 172.21.15.67:0/719770721 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4ef4100ea0 0x7f4ef4073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:25.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.592+0000 7f4ef2ffd700 1 -- 172.21.15.67:0/719770721 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4ef4100ea0 msgr2=0x7f4ef4073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:25.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.592+0000 7f4ef2ffd700 1 --2- 172.21.15.67:0/719770721 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4ef4100ea0 0x7f4ef4073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:25.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.592+0000 7f4ef2ffd700 1 -- 172.21.15.67:0/719770721 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4edc0095f0 con 0x7f4ef4101850 2024-04-22T23:36:25.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.592+0000 7f4ef2ffd700 1 --2- 172.21.15.67:0/719770721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ef4101850 0x7f4ef40735d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4edc0092e0 tx=0x7f4edc00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:36:25.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.592+0000 7f4ef0ff9700 1 -- 172.21.15.67:0/719770721 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4edc01d070 con 0x7f4ef4101850 2024-04-22T23:36:25.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.592+0000 7f4ef0ff9700 1 -- 172.21.15.67:0/719770721 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4edc005dc0 con 0x7f4ef4101850 2024-04-22T23:36:25.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.592+0000 7f4efa019700 1 -- 172.21.15.67:0/719770721 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ef4073ed0 con 0x7f4ef4101850 2024-04-22T23:36:25.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.592+0000 7f4ef0ff9700 1 -- 172.21.15.67:0/719770721 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4edc017400 con 0x7f4ef4101850 2024-04-22T23:36:25.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.593+0000 7f4efa019700 1 -- 172.21.15.67:0/719770721 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ef41a2e30 con 0x7f4ef4101850 2024-04-22T23:36:25.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.593+0000 7f4efa019700 1 -- 172.21.15.67:0/719770721 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ef41092a0 con 0x7f4ef4101850 2024-04-22T23:36:25.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.594+0000 7f4ef0ff9700 1 -- 172.21.15.67:0/719770721 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4edc00f6f0 con 0x7f4ef4101850 2024-04-22T23:36:25.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.594+0000 7f4ef0ff9700 1 --2- 172.21.15.67:0/719770721 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4ee006ecc0 0x7f4ee0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:25.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.595+0000 7f4ef37fe700 1 --2- 172.21.15.67:0/719770721 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4ee006ecc0 0x7f4ee0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:25.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.595+0000 7f4ef0ff9700 1 -- 172.21.15.67:0/719770721 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f4edc090300 con 0x7f4ef4101850 2024-04-22T23:36:25.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.595+0000 7f4ef37fe700 1 --2- 172.21.15.67:0/719770721 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4ee006ecc0 0x7f4ee0071180 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4ee4009550 tx=0x7f4ee40091b0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:36:25.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.599+0000 7f4ef0ff9700 1 -- 172.21.15.67:0/719770721 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f4edc058fc0 con 0x7f4ef4101850 2024-04-22T23:36:25.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.823+0000 7f4efa019700 1 -- 172.21.15.67:0/719770721 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4ef4074630 con 0x7f4ee006ecc0 2024-04-22T23:36:25.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.834+0000 7f4ef0ff9700 1 -- 172.21.15.67:0/719770721 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f4ef4074630 con 0x7f4ee006ecc0 2024-04-22T23:36:25.829 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:36:25.829 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (10m) 3m ago 21m 25.5M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:36:25.829 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (21m) 3m ago 21m 9114k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:36:25.829 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (20m) 3m ago 20m 9042k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:36:25.829 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (7m) 3m ago 21m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:36:25.829 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (7m) 3m ago 20m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:36:25.829 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (9m) 3m ago 21m 79.3M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:36:25.829 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (3m) 3m ago 15m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:36:25.829 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (3m) 3m ago 15m 12.6M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:36:25.829 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (3m) 3m ago 15m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:36:25.829 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (15m) 3m ago 15m 12.2M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:36:25.829 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (11m) 3m ago 22m 511M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:36:25.830 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (11m) 3m ago 20m 453M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:36:25.830 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (8m) 3m ago 23m 55.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:36:25.830 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (8m) 3m ago 20m 46.0M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:36:25.830 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (10m) 3m ago 21m 15.4M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:36:25.830 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (10m) 3m ago 20m 16.1M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:36:25.830 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (7m) 3m ago 19m 99.2M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:36:25.830 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (6m) 3m ago 19m 127M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:36:25.830 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (5m) 3m ago 18m 79.9M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:36:25.830 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (5m) 3m ago 18m 97.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:36:25.830 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (4m) 3m ago 17m 82.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:36:25.830 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (4m) 3m ago 17m 77.6M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:36:25.830 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (10m) 3m ago 20m 73.9M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:36:25.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.836+0000 7f4efa019700 1 -- 172.21.15.67:0/719770721 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4ee006ecc0 msgr2=0x7f4ee0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:25.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.836+0000 7f4efa019700 1 --2- 172.21.15.67:0/719770721 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4ee006ecc0 0x7f4ee0071180 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4ee4009550 tx=0x7f4ee40091b0 comp rx=0 tx=0).stop 2024-04-22T23:36:25.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.836+0000 7f4efa019700 1 -- 172.21.15.67:0/719770721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ef4101850 msgr2=0x7f4ef40735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:25.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.836+0000 7f4efa019700 1 --2- 172.21.15.67:0/719770721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ef4101850 0x7f4ef40735d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4edc0092e0 tx=0x7f4edc00bd70 comp rx=0 tx=0).stop 2024-04-22T23:36:25.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.836+0000 7f4efa019700 1 -- 172.21.15.67:0/719770721 shutdown_connections 2024-04-22T23:36:25.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.836+0000 7f4efa019700 1 --2- 172.21.15.67:0/719770721 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4ee006ecc0 0x7f4ee0071180 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:25.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.836+0000 7f4efa019700 1 --2- 172.21.15.67:0/719770721 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4ef4100ea0 0x7f4ef4073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:25.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.836+0000 7f4efa019700 1 --2- 172.21.15.67:0/719770721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ef4101850 0x7f4ef40735d0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:25.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.836+0000 7f4efa019700 1 -- 172.21.15.67:0/719770721 >> 172.21.15.67:0/719770721 conn(0x7f4ef40fc750 msgr2=0x7f4ef40fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:36:25.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.836+0000 7f4efa019700 1 -- 172.21.15.67:0/719770721 shutdown_connections 2024-04-22T23:36:25.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.836+0000 7f4efa019700 1 -- 172.21.15.67:0/719770721 wait complete. 2024-04-22T23:36:25.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.942+0000 7f02b67cd700 1 -- 172.21.15.67:0/84381105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02b0102720 msgr2=0x7f02b010ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:25.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.942+0000 7f02b67cd700 1 --2- 172.21.15.67:0/84381105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02b0102720 0x7f02b010ac10 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f02a0009910 tx=0x7f02a0009c20 comp rx=0 tx=0).stop 2024-04-22T23:36:25.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.943+0000 7f02b67cd700 1 -- 172.21.15.67:0/84381105 shutdown_connections 2024-04-22T23:36:25.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.943+0000 7f02b67cd700 1 --2- 172.21.15.67:0/84381105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02b0102720 0x7f02b010ac10 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:25.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.943+0000 7f02b67cd700 1 --2- 172.21.15.67:0/84381105 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02b0101e00 0x7f02b01021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:25.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.943+0000 7f02b67cd700 1 -- 172.21.15.67:0/84381105 >> 172.21.15.67:0/84381105 conn(0x7f02b00fb650 msgr2=0x7f02b00fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:36:25.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.943+0000 7f02b67cd700 1 -- 172.21.15.67:0/84381105 shutdown_connections 2024-04-22T23:36:25.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.943+0000 7f02b67cd700 1 -- 172.21.15.67:0/84381105 wait complete. 2024-04-22T23:36:25.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.943+0000 7f02b67cd700 1 Processor -- start 2024-04-22T23:36:25.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.943+0000 7f02b67cd700 1 -- start start 2024-04-22T23:36:25.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.944+0000 7f02b67cd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02b0101e00 0x7f02b019c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:25.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.944+0000 7f02b67cd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02b0102720 0x7f02b019cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:25.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.944+0000 7f02b67cd700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b019d400 con 0x7f02b0101e00 2024-04-22T23:36:25.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.944+0000 7f02b67cd700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b01968b0 con 0x7f02b0102720 2024-04-22T23:36:25.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.944+0000 7f02affff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02b0101e00 0x7f02b019c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:25.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.944+0000 7f02affff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02b0101e00 0x7f02b019c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32772/0 (socket says 172.21.15.67:32772) 2024-04-22T23:36:25.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.944+0000 7f02affff700 1 -- 172.21.15.67:0/542607534 learned_addr learned my addr 172.21.15.67:0/542607534 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:36:25.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.944+0000 7f02af7fe700 1 --2- 172.21.15.67:0/542607534 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02b0102720 0x7f02b019cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:25.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.944+0000 7f02affff700 1 -- 172.21.15.67:0/542607534 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02b0102720 msgr2=0x7f02b019cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:25.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.944+0000 7f02affff700 1 --2- 172.21.15.67:0/542607534 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02b0102720 0x7f02b019cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:25.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.944+0000 7f02affff700 1 -- 172.21.15.67:0/542607534 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a00095f0 con 0x7f02b0101e00 2024-04-22T23:36:25.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.944+0000 7f02affff700 1 --2- 172.21.15.67:0/542607534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02b0101e00 0x7f02b019c830 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f029800b4b0 tx=0x7f029800b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:36:25.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.944+0000 7f02ad7fa700 1 -- 172.21.15.67:0/542607534 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0298011800 con 0x7f02b0101e00 2024-04-22T23:36:25.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.944+0000 7f02ad7fa700 1 -- 172.21.15.67:0/542607534 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0298011e20 con 0x7f02b0101e00 2024-04-22T23:36:25.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.944+0000 7f02ad7fa700 1 -- 172.21.15.67:0/542607534 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f029800f2e0 con 0x7f02b0101e00 2024-04-22T23:36:25.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.945+0000 7f02b67cd700 1 -- 172.21.15.67:0/542607534 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02b0196b90 con 0x7f02b0101e00 2024-04-22T23:36:25.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.945+0000 7f02b67cd700 1 -- 172.21.15.67:0/542607534 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02b0197050 con 0x7f02b0101e00 2024-04-22T23:36:25.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.945+0000 7f02b67cd700 1 -- 172.21.15.67:0/542607534 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02b0108310 con 0x7f02b0101e00 2024-04-22T23:36:25.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.946+0000 7f02ad7fa700 1 -- 172.21.15.67:0/542607534 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0298010380 con 0x7f02b0101e00 2024-04-22T23:36:25.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.947+0000 7f02ad7fa700 1 --2- 172.21.15.67:0/542607534 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f029c06ecc0 0x7f029c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:25.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.947+0000 7f02ad7fa700 1 -- 172.21.15.67:0/542607534 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f029808ea00 con 0x7f02b0101e00 2024-04-22T23:36:25.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.947+0000 7f02af7fe700 1 --2- 172.21.15.67:0/542607534 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f029c06ecc0 0x7f029c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:25.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.947+0000 7f02af7fe700 1 --2- 172.21.15.67:0/542607534 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f029c06ecc0 0x7f029c071180 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f02a0006010 tx=0x7f02a001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:36:25.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:25.951+0000 7f02ad7fa700 1 -- 172.21.15.67:0/542607534 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f029805b1f0 con 0x7f02b0101e00 2024-04-22T23:36:26.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:25 smithi067 ceph-mon[112438]: pgmap v301: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:26.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:25 smithi134 ceph-mon[85630]: pgmap v301: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:26.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.252+0000 7f02b67cd700 1 -- 172.21.15.67:0/542607534 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f02b0197dc0 con 0x7f02b0101e00 2024-04-22T23:36:26.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.252+0000 7f02ad7fa700 1 -- 172.21.15.67:0/542607534 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f029805ad80 con 0x7f02b0101e00 2024-04-22T23:36:26.247 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:36:26.247 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:36:26.247 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:36:26.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:36:26.247 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:36:26.247 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:36:26.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:36:26.247 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:36:26.247 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:36:26.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:36:26.247 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:36:26.247 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:36:26.247 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 3 2024-04-22T23:36:26.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:36:26.248 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:36:26.248 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:36:26.248 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 13 2024-04-22T23:36:26.248 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:36:26.248 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:36:26.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.255+0000 7f02b67cd700 1 -- 172.21.15.67:0/542607534 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f029c06ecc0 msgr2=0x7f029c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:26.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.255+0000 7f02b67cd700 1 --2- 172.21.15.67:0/542607534 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f029c06ecc0 0x7f029c071180 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f02a0006010 tx=0x7f02a001a040 comp rx=0 tx=0).stop 2024-04-22T23:36:26.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.255+0000 7f02b67cd700 1 -- 172.21.15.67:0/542607534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02b0101e00 msgr2=0x7f02b019c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:26.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.255+0000 7f02b67cd700 1 --2- 172.21.15.67:0/542607534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02b0101e00 0x7f02b019c830 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f029800b4b0 tx=0x7f029800b7c0 comp rx=0 tx=0).stop 2024-04-22T23:36:26.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.255+0000 7f02b67cd700 1 -- 172.21.15.67:0/542607534 shutdown_connections 2024-04-22T23:36:26.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.255+0000 7f02b67cd700 1 --2- 172.21.15.67:0/542607534 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f029c06ecc0 0x7f029c071180 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:26.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.255+0000 7f02b67cd700 1 --2- 172.21.15.67:0/542607534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02b0101e00 0x7f02b019c830 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:26.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.255+0000 7f02b67cd700 1 --2- 172.21.15.67:0/542607534 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02b0102720 0x7f02b019cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:26.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.255+0000 7f02b67cd700 1 -- 172.21.15.67:0/542607534 >> 172.21.15.67:0/542607534 conn(0x7f02b00fb650 msgr2=0x7f02b0105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:36:26.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.255+0000 7f02b67cd700 1 -- 172.21.15.67:0/542607534 shutdown_connections 2024-04-22T23:36:26.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.255+0000 7f02b67cd700 1 -- 172.21.15.67:0/542607534 wait complete. 2024-04-22T23:36:26.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.370+0000 7ff8979fd700 1 -- 172.21.15.67:0/2083609565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff890101850 msgr2=0x7ff8901058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:26.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.370+0000 7ff8979fd700 1 --2- 172.21.15.67:0/2083609565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff890101850 0x7ff8901058a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff880009910 tx=0x7ff880009c20 comp rx=0 tx=0).stop 2024-04-22T23:36:26.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.370+0000 7ff8979fd700 1 -- 172.21.15.67:0/2083609565 shutdown_connections 2024-04-22T23:36:26.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.370+0000 7ff8979fd700 1 --2- 172.21.15.67:0/2083609565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff890101850 0x7ff8901058a0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:26.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.370+0000 7ff8979fd700 1 --2- 172.21.15.67:0/2083609565 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff890100ea0 0x7ff890101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:26.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.370+0000 7ff8979fd700 1 -- 172.21.15.67:0/2083609565 >> 172.21.15.67:0/2083609565 conn(0x7ff8900fc750 msgr2=0x7ff8900feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:36:26.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.370+0000 7ff8979fd700 1 -- 172.21.15.67:0/2083609565 shutdown_connections 2024-04-22T23:36:26.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.370+0000 7ff8979fd700 1 -- 172.21.15.67:0/2083609565 wait complete. 2024-04-22T23:36:26.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.370+0000 7ff8979fd700 1 Processor -- start 2024-04-22T23:36:26.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.370+0000 7ff8979fd700 1 -- start start 2024-04-22T23:36:26.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.371+0000 7ff8979fd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff890100ea0 0x7ff89019c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:26.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.371+0000 7ff8979fd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff890101850 0x7ff89019cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:26.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.371+0000 7ff8979fd700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff89019d2d0 con 0x7ff890100ea0 2024-04-22T23:36:26.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.371+0000 7ff8979fd700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff890196780 con 0x7ff890101850 2024-04-22T23:36:26.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.371+0000 7ff895799700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff890100ea0 0x7ff89019c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:26.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.371+0000 7ff895799700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff890100ea0 0x7ff89019c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32778/0 (socket says 172.21.15.67:32778) 2024-04-22T23:36:26.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.371+0000 7ff895799700 1 -- 172.21.15.67:0/336824436 learned_addr learned my addr 172.21.15.67:0/336824436 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:36:26.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.371+0000 7ff894f98700 1 --2- 172.21.15.67:0/336824436 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff890101850 0x7ff89019cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:26.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.371+0000 7ff895799700 1 -- 172.21.15.67:0/336824436 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff890101850 msgr2=0x7ff89019cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:26.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.371+0000 7ff895799700 1 --2- 172.21.15.67:0/336824436 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff890101850 0x7ff89019cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:26.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.371+0000 7ff895799700 1 -- 172.21.15.67:0/336824436 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8800095f0 con 0x7ff890100ea0 2024-04-22T23:36:26.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.371+0000 7ff895799700 1 --2- 172.21.15.67:0/336824436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff890100ea0 0x7ff89019c700 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff88c00e920 tx=0x7ff88c00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:36:26.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.371+0000 7ff8867fc700 1 -- 172.21.15.67:0/336824436 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff88c00c9d0 con 0x7ff890100ea0 2024-04-22T23:36:26.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.372+0000 7ff8867fc700 1 -- 172.21.15.67:0/336824436 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff88c010430 con 0x7ff890100ea0 2024-04-22T23:36:26.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.372+0000 7ff8979fd700 1 -- 172.21.15.67:0/336824436 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff890196a60 con 0x7ff890100ea0 2024-04-22T23:36:26.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.372+0000 7ff8867fc700 1 -- 172.21.15.67:0/336824436 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff88c018630 con 0x7ff890100ea0 2024-04-22T23:36:26.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.372+0000 7ff8979fd700 1 -- 172.21.15.67:0/336824436 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff890196f80 con 0x7ff890100ea0 2024-04-22T23:36:26.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.372+0000 7ff8979fd700 1 -- 172.21.15.67:0/336824436 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8901092a0 con 0x7ff890100ea0 2024-04-22T23:36:26.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.373+0000 7ff8867fc700 1 -- 172.21.15.67:0/336824436 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff88c0105a0 con 0x7ff890100ea0 2024-04-22T23:36:26.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.373+0000 7ff8867fc700 1 --2- 172.21.15.67:0/336824436 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff87c06ecc0 0x7ff87c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:26.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.374+0000 7ff8867fc700 1 -- 172.21.15.67:0/336824436 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7ff88c014070 con 0x7ff890100ea0 2024-04-22T23:36:26.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.374+0000 7ff894f98700 1 --2- 172.21.15.67:0/336824436 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff87c06ecc0 0x7ff87c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:26.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.374+0000 7ff894f98700 1 --2- 172.21.15.67:0/336824436 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff87c06ecc0 0x7ff87c071180 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff8800059d0 tx=0x7ff880005940 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:36:26.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.378+0000 7ff8867fc700 1 -- 172.21.15.67:0/336824436 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff88c054150 con 0x7ff890100ea0 2024-04-22T23:36:26.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.633+0000 7ff8979fd700 1 -- 172.21.15.67:0/336824436 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff890197d30 con 0x7ff890100ea0 2024-04-22T23:36:26.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.633+0000 7ff8867fc700 1 -- 172.21.15.67:0/336824436 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1871 (secure 0 0 0) 0x7ff88c05b9b0 con 0x7ff890100ea0 2024-04-22T23:36:26.628 INFO:teuthology.orchestra.run.smithi067.stdout:e31 2024-04-22T23:36:26.628 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:36:26.628 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:36:26.628 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:36:26.628 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:36:26.628 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:36:26.628 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:36:26.629 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:36:26.629 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:36:26.629 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:36:26.629 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:36:26.629 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:36:26.629 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:36:26.629 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:36:26.629 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:36:26.629 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:36:26.629 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:36:26.629 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:36:26.629 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:36:26.629 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:36:26.629 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:36:26.629 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:36:26.630 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:36:26.630 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:36:26.630 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:36:26.630 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:36:26.630 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:36:26.630 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:36:26.630 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:36:26.630 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:36:26.630 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:36:26.630 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:36:26.630 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:36:26.630 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:36:26.630 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:36:26.630 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:36:26.631 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:36:26.631 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:36:26.631 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:36:26.631 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:36:26.631 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44259} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1827288036,v1:172.21.15.134:6827/1827288036] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:36:26.631 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:36:26.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.635+0000 7ff8979fd700 1 -- 172.21.15.67:0/336824436 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff87c06ecc0 msgr2=0x7ff87c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:26.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.635+0000 7ff8979fd700 1 --2- 172.21.15.67:0/336824436 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff87c06ecc0 0x7ff87c071180 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff8800059d0 tx=0x7ff880005940 comp rx=0 tx=0).stop 2024-04-22T23:36:26.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.635+0000 7ff8979fd700 1 -- 172.21.15.67:0/336824436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff890100ea0 msgr2=0x7ff89019c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:26.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.635+0000 7ff8979fd700 1 --2- 172.21.15.67:0/336824436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff890100ea0 0x7ff89019c700 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff88c00e920 tx=0x7ff88c00ec30 comp rx=0 tx=0).stop 2024-04-22T23:36:26.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.636+0000 7ff8979fd700 1 -- 172.21.15.67:0/336824436 shutdown_connections 2024-04-22T23:36:26.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.636+0000 7ff8979fd700 1 --2- 172.21.15.67:0/336824436 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff87c06ecc0 0x7ff87c071180 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:26.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.636+0000 7ff8979fd700 1 --2- 172.21.15.67:0/336824436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff890100ea0 0x7ff89019c700 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:26.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.636+0000 7ff8979fd700 1 --2- 172.21.15.67:0/336824436 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff890101850 0x7ff89019cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:26.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.636+0000 7ff8979fd700 1 -- 172.21.15.67:0/336824436 >> 172.21.15.67:0/336824436 conn(0x7ff8900fc750 msgr2=0x7ff8900fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:36:26.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.636+0000 7ff8979fd700 1 -- 172.21.15.67:0/336824436 shutdown_connections 2024-04-22T23:36:26.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.636+0000 7ff8979fd700 1 -- 172.21.15.67:0/336824436 wait complete. 2024-04-22T23:36:26.633 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 31 2024-04-22T23:36:26.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.744+0000 7fbd6148d700 1 -- 172.21.15.67:0/1195330292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd5c100ea0 msgr2=0x7fbd5c101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:26.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.744+0000 7fbd6148d700 1 --2- 172.21.15.67:0/1195330292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd5c100ea0 0x7fbd5c101280 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fbd54009910 tx=0x7fbd54009c20 comp rx=0 tx=0).stop 2024-04-22T23:36:26.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.744+0000 7fbd6148d700 1 -- 172.21.15.67:0/1195330292 shutdown_connections 2024-04-22T23:36:26.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.744+0000 7fbd6148d700 1 --2- 172.21.15.67:0/1195330292 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd5c101850 0x7fbd5c1058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:26.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.744+0000 7fbd6148d700 1 --2- 172.21.15.67:0/1195330292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd5c100ea0 0x7fbd5c101280 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:26.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.744+0000 7fbd6148d700 1 -- 172.21.15.67:0/1195330292 >> 172.21.15.67:0/1195330292 conn(0x7fbd5c0fc750 msgr2=0x7fbd5c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:36:26.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.744+0000 7fbd6148d700 1 -- 172.21.15.67:0/1195330292 shutdown_connections 2024-04-22T23:36:26.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.744+0000 7fbd6148d700 1 -- 172.21.15.67:0/1195330292 wait complete. 2024-04-22T23:36:26.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.744+0000 7fbd6148d700 1 Processor -- start 2024-04-22T23:36:26.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.745+0000 7fbd6148d700 1 -- start start 2024-04-22T23:36:26.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.745+0000 7fbd6148d700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd5c100ea0 0x7fbd5c19c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:26.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.745+0000 7fbd6148d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd5c101850 0x7fbd5c19cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:26.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.745+0000 7fbd6148d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5c19d2c0 con 0x7fbd5c101850 2024-04-22T23:36:26.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.745+0000 7fbd6148d700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5c196770 con 0x7fbd5c100ea0 2024-04-22T23:36:26.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.745+0000 7fbd5a7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd5c101850 0x7fbd5c19cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:26.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.745+0000 7fbd5a7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd5c101850 0x7fbd5c19cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32794/0 (socket says 172.21.15.67:32794) 2024-04-22T23:36:26.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.745+0000 7fbd5a7fc700 1 -- 172.21.15.67:0/3421529776 learned_addr learned my addr 172.21.15.67:0/3421529776 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:36:26.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.745+0000 7fbd5affd700 1 --2- 172.21.15.67:0/3421529776 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd5c100ea0 0x7fbd5c19c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:26.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.745+0000 7fbd5a7fc700 1 -- 172.21.15.67:0/3421529776 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd5c100ea0 msgr2=0x7fbd5c19c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:26.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.745+0000 7fbd5a7fc700 1 --2- 172.21.15.67:0/3421529776 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd5c100ea0 0x7fbd5c19c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:26.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.745+0000 7fbd5a7fc700 1 -- 172.21.15.67:0/3421529776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd540095f0 con 0x7fbd5c101850 2024-04-22T23:36:26.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.745+0000 7fbd5a7fc700 1 --2- 172.21.15.67:0/3421529776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd5c101850 0x7fbd5c19cc30 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fbd4400e8f0 tx=0x7fbd4400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:36:26.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.746+0000 7fbd53fff700 1 -- 172.21.15.67:0/3421529776 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd44009790 con 0x7fbd5c101850 2024-04-22T23:36:26.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.746+0000 7fbd53fff700 1 -- 172.21.15.67:0/3421529776 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbd4400c810 con 0x7fbd5c101850 2024-04-22T23:36:26.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.746+0000 7fbd6148d700 1 -- 172.21.15.67:0/3421529776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd5c196a50 con 0x7fbd5c101850 2024-04-22T23:36:26.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.746+0000 7fbd53fff700 1 -- 172.21.15.67:0/3421529776 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd44005580 con 0x7fbd5c101850 2024-04-22T23:36:26.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.746+0000 7fbd6148d700 1 -- 172.21.15.67:0/3421529776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd5c196fa0 con 0x7fbd5c101850 2024-04-22T23:36:26.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.747+0000 7fbd6148d700 1 -- 172.21.15.67:0/3421529776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd5c109240 con 0x7fbd5c101850 2024-04-22T23:36:26.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.747+0000 7fbd53fff700 1 -- 172.21.15.67:0/3421529776 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbd4400c980 con 0x7fbd5c101850 2024-04-22T23:36:26.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.748+0000 7fbd53fff700 1 --2- 172.21.15.67:0/3421529776 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbd4806ecc0 0x7fbd48071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:26.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.748+0000 7fbd53fff700 1 -- 172.21.15.67:0/3421529776 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fbd44014070 con 0x7fbd5c101850 2024-04-22T23:36:26.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.748+0000 7fbd5affd700 1 --2- 172.21.15.67:0/3421529776 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbd4806ecc0 0x7fbd48071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:26.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.748+0000 7fbd5affd700 1 --2- 172.21.15.67:0/3421529776 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbd4806ecc0 0x7fbd48071180 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbd540059d0 tx=0x7fbd54005940 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:36:26.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.752+0000 7fbd53fff700 1 -- 172.21.15.67:0/3421529776 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fbd4405b9a0 con 0x7fbd5c101850 2024-04-22T23:36:26.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.983+0000 7fbd6148d700 1 -- 172.21.15.67:0/3421529776 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbd5c197d90 con 0x7fbd4806ecc0 2024-04-22T23:36:26.978 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:26 smithi067 ceph-mon[112438]: from='client.34596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:36:26.978 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:26 smithi067 ceph-mon[112438]: from='client.34600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:36:26.978 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:26 smithi067 ceph-mon[112438]: from='client.34604 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:36:26.978 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:36:26.978 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:26 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/542607534' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:36:26.978 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:26 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/336824436' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:36:26.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.984+0000 7fbd53fff700 1 -- 172.21.15.67:0/3421529776 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fbd5c197d90 con 0x7fbd4806ecc0 2024-04-22T23:36:26.979 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:36:26.979 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:36:26.979 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:36:26.979 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:36:26.979 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:36:26.980 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:36:26.980 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:36:26.980 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:36:26.980 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:36:26.980 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:36:26.980 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T23:36:26.980 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:36:26.980 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:36:26.980 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:36:26.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.986+0000 7fbd6148d700 1 -- 172.21.15.67:0/3421529776 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbd4806ecc0 msgr2=0x7fbd48071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:26.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.986+0000 7fbd6148d700 1 --2- 172.21.15.67:0/3421529776 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbd4806ecc0 0x7fbd48071180 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbd540059d0 tx=0x7fbd54005940 comp rx=0 tx=0).stop 2024-04-22T23:36:26.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.986+0000 7fbd6148d700 1 -- 172.21.15.67:0/3421529776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd5c101850 msgr2=0x7fbd5c19cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:26.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.986+0000 7fbd6148d700 1 --2- 172.21.15.67:0/3421529776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd5c101850 0x7fbd5c19cc30 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fbd4400e8f0 tx=0x7fbd4400ec00 comp rx=0 tx=0).stop 2024-04-22T23:36:26.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.986+0000 7fbd6148d700 1 -- 172.21.15.67:0/3421529776 shutdown_connections 2024-04-22T23:36:26.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.986+0000 7fbd6148d700 1 --2- 172.21.15.67:0/3421529776 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbd4806ecc0 0x7fbd48071180 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:26.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.986+0000 7fbd6148d700 1 --2- 172.21.15.67:0/3421529776 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd5c100ea0 0x7fbd5c19c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:26.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.986+0000 7fbd6148d700 1 --2- 172.21.15.67:0/3421529776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd5c101850 0x7fbd5c19cc30 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:26.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.986+0000 7fbd6148d700 1 -- 172.21.15.67:0/3421529776 >> 172.21.15.67:0/3421529776 conn(0x7fbd5c0fc750 msgr2=0x7fbd5c0fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:36:26.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.987+0000 7fbd6148d700 1 -- 172.21.15.67:0/3421529776 shutdown_connections 2024-04-22T23:36:26.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:26.987+0000 7fbd6148d700 1 -- 172.21.15.67:0/3421529776 wait complete. 2024-04-22T23:36:27.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.094+0000 7f290adf8700 1 -- 172.21.15.67:0/1997030900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2904105350 msgr2=0x7f2904105730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:27.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.094+0000 7f290adf8700 1 --2- 172.21.15.67:0/1997030900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2904105350 0x7f2904105730 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f28f800aa20 tx=0x7f28f800ad30 comp rx=0 tx=0).stop 2024-04-22T23:36:27.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.094+0000 7f290adf8700 1 -- 172.21.15.67:0/1997030900 shutdown_connections 2024-04-22T23:36:27.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.094+0000 7f290adf8700 1 --2- 172.21.15.67:0/1997030900 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f29040ff340 0x7f29040ff7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:27.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.094+0000 7f290adf8700 1 --2- 172.21.15.67:0/1997030900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2904105350 0x7f2904105730 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:27.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.094+0000 7f290adf8700 1 -- 172.21.15.67:0/1997030900 >> 172.21.15.67:0/1997030900 conn(0x7f2904074e30 msgr2=0x7f2904075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:36:27.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.094+0000 7f290adf8700 1 -- 172.21.15.67:0/1997030900 shutdown_connections 2024-04-22T23:36:27.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.094+0000 7f290adf8700 1 -- 172.21.15.67:0/1997030900 wait complete. 2024-04-22T23:36:27.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.094+0000 7f290adf8700 1 Processor -- start 2024-04-22T23:36:27.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.094+0000 7f290adf8700 1 -- start start 2024-04-22T23:36:27.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.095+0000 7f290adf8700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f29040ff340 0x7f2904100fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:27.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.095+0000 7f290adf8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2904105350 0x7f2904101500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:27.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.095+0000 7f290adf8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2904101a40 con 0x7f2904105350 2024-04-22T23:36:27.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.095+0000 7f290adf8700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2904101b80 con 0x7f29040ff340 2024-04-22T23:36:27.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.095+0000 7f2903fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2904105350 0x7f2904101500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:27.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.095+0000 7f2903fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2904105350 0x7f2904101500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32810/0 (socket says 172.21.15.67:32810) 2024-04-22T23:36:27.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.095+0000 7f2903fff700 1 -- 172.21.15.67:0/775501909 learned_addr learned my addr 172.21.15.67:0/775501909 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:36:27.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.095+0000 7f2908b94700 1 --2- 172.21.15.67:0/775501909 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f29040ff340 0x7f2904100fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:27.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.095+0000 7f2903fff700 1 -- 172.21.15.67:0/775501909 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f29040ff340 msgr2=0x7f2904100fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:27.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.095+0000 7f2903fff700 1 --2- 172.21.15.67:0/775501909 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f29040ff340 0x7f2904100fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:27.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.095+0000 7f2903fff700 1 -- 172.21.15.67:0/775501909 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f800a700 con 0x7f2904105350 2024-04-22T23:36:27.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.095+0000 7f2903fff700 1 --2- 172.21.15.67:0/775501909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2904105350 0x7f2904101500 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f28f000b4b0 tx=0x7f28f000b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:36:27.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.095+0000 7f2901ffb700 1 -- 172.21.15.67:0/775501909 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28f0011800 con 0x7f2904105350 2024-04-22T23:36:27.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.095+0000 7f2901ffb700 1 -- 172.21.15.67:0/775501909 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f28f0011e20 con 0x7f2904105350 2024-04-22T23:36:27.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.095+0000 7f290adf8700 1 -- 172.21.15.67:0/775501909 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29041a2b10 con 0x7f2904105350 2024-04-22T23:36:27.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.096+0000 7f2901ffb700 1 -- 172.21.15.67:0/775501909 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28f000f2e0 con 0x7f2904105350 2024-04-22T23:36:27.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.096+0000 7f290adf8700 1 -- 172.21.15.67:0/775501909 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29041a2f20 con 0x7f2904105350 2024-04-22T23:36:27.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.097+0000 7f290adf8700 1 -- 172.21.15.67:0/775501909 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2904109880 con 0x7f2904105350 2024-04-22T23:36:27.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.097+0000 7f2901ffb700 1 -- 172.21.15.67:0/775501909 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f28f0010380 con 0x7f2904105350 2024-04-22T23:36:27.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.097+0000 7f2901ffb700 1 --2- 172.21.15.67:0/775501909 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f28f406ecc0 0x7f28f4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:27.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.098+0000 7f2901ffb700 1 -- 172.21.15.67:0/775501909 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f28f008eb40 con 0x7f2904105350 2024-04-22T23:36:27.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.098+0000 7f2908b94700 1 --2- 172.21.15.67:0/775501909 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f28f406ecc0 0x7f28f4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:27.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.098+0000 7f2908b94700 1 --2- 172.21.15.67:0/775501909 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f28f406ecc0 0x7f28f4071180 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f28f8000c00 tx=0x7f28f8005de0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:36:27.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.102+0000 7f2901ffb700 1 -- 172.21.15.67:0/775501909 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f28f0057780 con 0x7f2904105350 2024-04-22T23:36:27.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:26 smithi134 ceph-mon[85630]: from='client.34596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:36:27.238 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:26 smithi134 ceph-mon[85630]: from='client.34600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:36:27.238 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:26 smithi134 ceph-mon[85630]: from='client.34604 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:36:27.238 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:36:27.238 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:26 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/542607534' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:36:27.238 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:26 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/336824436' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:36:27.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.397+0000 7f290adf8700 1 -- 172.21.15.67:0/775501909 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f290404f2e0 con 0x7f2904105350 2024-04-22T23:36:27.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.397+0000 7f2901ffb700 1 -- 172.21.15.67:0/775501909 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f28f005ae40 con 0x7f2904105350 2024-04-22T23:36:27.392 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:36:27.392 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:36:27.392 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:36:27.392 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:36:27.392 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:36:27.392 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:36:27.392 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:36:27.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.399+0000 7f290adf8700 1 -- 172.21.15.67:0/775501909 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f28f406ecc0 msgr2=0x7f28f4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:27.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.399+0000 7f290adf8700 1 --2- 172.21.15.67:0/775501909 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f28f406ecc0 0x7f28f4071180 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f28f8000c00 tx=0x7f28f8005de0 comp rx=0 tx=0).stop 2024-04-22T23:36:27.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.399+0000 7f290adf8700 1 -- 172.21.15.67:0/775501909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2904105350 msgr2=0x7f2904101500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:27.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.399+0000 7f290adf8700 1 --2- 172.21.15.67:0/775501909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2904105350 0x7f2904101500 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f28f000b4b0 tx=0x7f28f000b7c0 comp rx=0 tx=0).stop 2024-04-22T23:36:27.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.399+0000 7f290adf8700 1 -- 172.21.15.67:0/775501909 shutdown_connections 2024-04-22T23:36:27.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.399+0000 7f290adf8700 1 --2- 172.21.15.67:0/775501909 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f28f406ecc0 0x7f28f4071180 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:27.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.399+0000 7f290adf8700 1 --2- 172.21.15.67:0/775501909 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f29040ff340 0x7f2904100fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:27.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.399+0000 7f290adf8700 1 --2- 172.21.15.67:0/775501909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2904105350 0x7f2904101500 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:27.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.399+0000 7f290adf8700 1 -- 172.21.15.67:0/775501909 >> 172.21.15.67:0/775501909 conn(0x7f2904074e30 msgr2=0x7f2904104760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:36:27.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.400+0000 7f290adf8700 1 -- 172.21.15.67:0/775501909 shutdown_connections 2024-04-22T23:36:27.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:27.400+0000 7f290adf8700 1 -- 172.21.15.67:0/775501909 wait complete. 2024-04-22T23:36:28.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:27 smithi067 ceph-mon[112438]: pgmap v302: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:28.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:27 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/775501909' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:36:28.238 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:27 smithi134 ceph-mon[85630]: pgmap v302: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:28.238 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:27 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/775501909' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:36:29.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:28 smithi067 ceph-mon[112438]: from='client.34616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:36:29.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:28 smithi134 ceph-mon[85630]: from='client.34616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:36:30.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:29 smithi067 ceph-mon[112438]: pgmap v303: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:30.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:29 smithi134 ceph-mon[85630]: pgmap v303: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:32.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:31 smithi067 ceph-mon[112438]: pgmap v304: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:32.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:31 smithi134 ceph-mon[85630]: pgmap v304: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:34.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:33 smithi067 ceph-mon[112438]: pgmap v305: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:34.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:33 smithi134 ceph-mon[85630]: pgmap v305: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:36.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:35 smithi067 ceph-mon[112438]: pgmap v306: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:36.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:35 smithi134 ceph-mon[85630]: pgmap v306: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:38.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:37 smithi067 ceph-mon[112438]: pgmap v307: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:38.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:37 smithi134 ceph-mon[85630]: pgmap v307: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:40.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:39 smithi067 ceph-mon[112438]: pgmap v308: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:40.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:39 smithi134 ceph-mon[85630]: pgmap v308: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:42.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:41 smithi067 ceph-mon[112438]: pgmap v309: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:42.039 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:36:42.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:41 smithi134 ceph-mon[85630]: pgmap v309: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:42.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:36:44.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:43 smithi067 ceph-mon[112438]: pgmap v310: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:44.039 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:43 smithi134 ceph-mon[85630]: pgmap v310: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:46.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:45 smithi134 ceph-mon[85630]: pgmap v311: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:46.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:45 smithi067 ceph-mon[112438]: pgmap v311: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:48.077 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:47 smithi067 ceph-mon[112438]: pgmap v312: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:48.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:47 smithi134 ceph-mon[85630]: pgmap v312: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:50.052 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:49 smithi067 ceph-mon[112438]: pgmap v313: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:50.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:49 smithi134 ceph-mon[85630]: pgmap v313: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:52.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:51 smithi134 ceph-mon[85630]: pgmap v314: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:52.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:51 smithi067 ceph-mon[112438]: pgmap v314: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:54.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:53 smithi134 ceph-mon[85630]: pgmap v315: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:54.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:53 smithi067 ceph-mon[112438]: pgmap v315: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:56.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:55 smithi134 ceph-mon[85630]: pgmap v316: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:56.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:55 smithi067 ceph-mon[112438]: pgmap v316: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:57.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:36:57.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:36:57.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.522+0000 7f1375040700 1 -- 172.21.15.67:0/3264477623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1370100750 msgr2=0x7f13701047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:57.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.522+0000 7f1375040700 1 --2- 172.21.15.67:0/3264477623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1370100750 0x7f13701047a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f1364009910 tx=0x7f1364009c20 comp rx=0 tx=0).stop 2024-04-22T23:36:57.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.522+0000 7f1375040700 1 -- 172.21.15.67:0/3264477623 shutdown_connections 2024-04-22T23:36:57.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.522+0000 7f1375040700 1 --2- 172.21.15.67:0/3264477623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1370100750 0x7f13701047a0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:57.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.522+0000 7f1375040700 1 --2- 172.21.15.67:0/3264477623 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13700ffda0 0x7f1370100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:57.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.522+0000 7f1375040700 1 -- 172.21.15.67:0/3264477623 >> 172.21.15.67:0/3264477623 conn(0x7f13700fb650 msgr2=0x7f13700fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:36:57.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.522+0000 7f1375040700 1 -- 172.21.15.67:0/3264477623 shutdown_connections 2024-04-22T23:36:57.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.522+0000 7f1375040700 1 -- 172.21.15.67:0/3264477623 wait complete. 2024-04-22T23:36:57.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.523+0000 7f1375040700 1 Processor -- start 2024-04-22T23:36:57.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.523+0000 7f1375040700 1 -- start start 2024-04-22T23:36:57.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.523+0000 7f1375040700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13700ffda0 0x7f137019c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:57.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.523+0000 7f1375040700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1370100750 0x7f137019cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:57.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.523+0000 7f1375040700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137019d240 con 0x7f1370100750 2024-04-22T23:36:57.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.523+0000 7f1375040700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13701966f0 con 0x7f13700ffda0 2024-04-22T23:36:57.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.523+0000 7f136e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1370100750 0x7f137019cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:57.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.523+0000 7f136e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1370100750 0x7f137019cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35518/0 (socket says 172.21.15.67:35518) 2024-04-22T23:36:57.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.523+0000 7f136e59c700 1 -- 172.21.15.67:0/3894961248 learned_addr learned my addr 172.21.15.67:0/3894961248 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:36:57.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.523+0000 7f136ed9d700 1 --2- 172.21.15.67:0/3894961248 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13700ffda0 0x7f137019c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:57.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.523+0000 7f136e59c700 1 -- 172.21.15.67:0/3894961248 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13700ffda0 msgr2=0x7f137019c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:57.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.523+0000 7f136e59c700 1 --2- 172.21.15.67:0/3894961248 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13700ffda0 0x7f137019c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:57.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.523+0000 7f136e59c700 1 -- 172.21.15.67:0/3894961248 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13640095f0 con 0x7f1370100750 2024-04-22T23:36:57.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.524+0000 7f136e59c700 1 --2- 172.21.15.67:0/3894961248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1370100750 0x7f137019cbb0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f13640059d0 tx=0x7f136400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:36:57.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.524+0000 7f1357fff700 1 -- 172.21.15.67:0/3894961248 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f136401d070 con 0x7f1370100750 2024-04-22T23:36:57.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.524+0000 7f1375040700 1 -- 172.21.15.67:0/3894961248 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1370196970 con 0x7f1370100750 2024-04-22T23:36:57.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.524+0000 7f1357fff700 1 -- 172.21.15.67:0/3894961248 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f136400b610 con 0x7f1370100750 2024-04-22T23:36:57.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.524+0000 7f1357fff700 1 -- 172.21.15.67:0/3894961248 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1364022470 con 0x7f1370100750 2024-04-22T23:36:57.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.524+0000 7f1375040700 1 -- 172.21.15.67:0/3894961248 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1370196e60 con 0x7f1370100750 2024-04-22T23:36:57.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.525+0000 7f1375040700 1 -- 172.21.15.67:0/3894961248 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13701091b0 con 0x7f1370100750 2024-04-22T23:36:57.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.526+0000 7f1357fff700 1 -- 172.21.15.67:0/3894961248 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1364003710 con 0x7f1370100750 2024-04-22T23:36:57.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.526+0000 7f1357fff700 1 --2- 172.21.15.67:0/3894961248 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f135806ecc0 0x7f1358071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:57.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.526+0000 7f1357fff700 1 -- 172.21.15.67:0/3894961248 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f13640903d0 con 0x7f1370100750 2024-04-22T23:36:57.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.526+0000 7f136ed9d700 1 --2- 172.21.15.67:0/3894961248 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f135806ecc0 0x7f1358071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:57.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.527+0000 7f136ed9d700 1 --2- 172.21.15.67:0/3894961248 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f135806ecc0 0x7f1358071180 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1360009c60 tx=0x7f1360009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:36:57.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.531+0000 7f1357fff700 1 -- 172.21.15.67:0/3894961248 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f136405cbc0 con 0x7f1370100750 2024-04-22T23:36:57.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.762+0000 7f1375040700 1 -- 172.21.15.67:0/3894961248 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f13701970d0 con 0x7f135806ecc0 2024-04-22T23:36:57.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.763+0000 7f1357fff700 1 -- 172.21.15.67:0/3894961248 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f13701970d0 con 0x7f135806ecc0 2024-04-22T23:36:57.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.765+0000 7f1375040700 1 -- 172.21.15.67:0/3894961248 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f135806ecc0 msgr2=0x7f1358071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:57.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.765+0000 7f1375040700 1 --2- 172.21.15.67:0/3894961248 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f135806ecc0 0x7f1358071180 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1360009c60 tx=0x7f1360009260 comp rx=0 tx=0).stop 2024-04-22T23:36:57.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.765+0000 7f1375040700 1 -- 172.21.15.67:0/3894961248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1370100750 msgr2=0x7f137019cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:57.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.765+0000 7f1375040700 1 --2- 172.21.15.67:0/3894961248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1370100750 0x7f137019cbb0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f13640059d0 tx=0x7f136400f690 comp rx=0 tx=0).stop 2024-04-22T23:36:57.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.765+0000 7f1375040700 1 -- 172.21.15.67:0/3894961248 shutdown_connections 2024-04-22T23:36:57.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.765+0000 7f1375040700 1 --2- 172.21.15.67:0/3894961248 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f135806ecc0 0x7f1358071180 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:57.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.765+0000 7f1375040700 1 --2- 172.21.15.67:0/3894961248 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13700ffda0 0x7f137019c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:57.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.765+0000 7f1375040700 1 --2- 172.21.15.67:0/3894961248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1370100750 0x7f137019cbb0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:57.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.765+0000 7f1375040700 1 -- 172.21.15.67:0/3894961248 >> 172.21.15.67:0/3894961248 conn(0x7f13700fb650 msgr2=0x7f1370069150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:36:57.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.765+0000 7f1375040700 1 -- 172.21.15.67:0/3894961248 shutdown_connections 2024-04-22T23:36:57.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.765+0000 7f1375040700 1 -- 172.21.15.67:0/3894961248 wait complete. 2024-04-22T23:36:57.770 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:36:57.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.875+0000 7f58e9075700 1 -- 172.21.15.67:0/163300352 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58e40ffc20 msgr2=0x7f58e4100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:57.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.875+0000 7f58e9075700 1 --2- 172.21.15.67:0/163300352 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58e40ffc20 0x7f58e4100080 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f58d8009910 tx=0x7f58d8009c20 comp rx=0 tx=0).stop 2024-04-22T23:36:57.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.875+0000 7f58e9075700 1 -- 172.21.15.67:0/163300352 shutdown_connections 2024-04-22T23:36:57.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.875+0000 7f58e9075700 1 --2- 172.21.15.67:0/163300352 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58e40ffc20 0x7f58e4100080 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:57.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.875+0000 7f58e9075700 1 --2- 172.21.15.67:0/163300352 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f58e4105bd0 0x7f58e4105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:57.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.875+0000 7f58e9075700 1 -- 172.21.15.67:0/163300352 >> 172.21.15.67:0/163300352 conn(0x7f58e4075670 msgr2=0x7f58e4075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:36:57.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.875+0000 7f58e9075700 1 -- 172.21.15.67:0/163300352 shutdown_connections 2024-04-22T23:36:57.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.875+0000 7f58e9075700 1 -- 172.21.15.67:0/163300352 wait complete. 2024-04-22T23:36:57.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.875+0000 7f58e9075700 1 Processor -- start 2024-04-22T23:36:57.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.876+0000 7f58e9075700 1 -- start start 2024-04-22T23:36:57.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.876+0000 7f58e9075700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58e40ffc20 0x7f58e419c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:57.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.876+0000 7f58e9075700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f58e4105bd0 0x7f58e419cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:57.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.876+0000 7f58e9075700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e419d2b0 con 0x7f58e40ffc20 2024-04-22T23:36:57.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.876+0000 7f58e9075700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e4196730 con 0x7f58e4105bd0 2024-04-22T23:36:57.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.876+0000 7f58e2d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58e40ffc20 0x7f58e419c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:57.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.876+0000 7f58e2d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58e40ffc20 0x7f58e419c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35536/0 (socket says 172.21.15.67:35536) 2024-04-22T23:36:57.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.876+0000 7f58e2d9d700 1 -- 172.21.15.67:0/1093171900 learned_addr learned my addr 172.21.15.67:0/1093171900 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:36:57.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.876+0000 7f58e259c700 1 --2- 172.21.15.67:0/1093171900 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f58e4105bd0 0x7f58e419cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:57.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.876+0000 7f58e2d9d700 1 -- 172.21.15.67:0/1093171900 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f58e4105bd0 msgr2=0x7f58e419cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:57.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.876+0000 7f58e2d9d700 1 --2- 172.21.15.67:0/1093171900 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f58e4105bd0 0x7f58e419cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:57.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.876+0000 7f58e2d9d700 1 -- 172.21.15.67:0/1093171900 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d80095f0 con 0x7f58e40ffc20 2024-04-22T23:36:57.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.877+0000 7f58e2d9d700 1 --2- 172.21.15.67:0/1093171900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58e40ffc20 0x7f58e419c690 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f58d400c8a0 tx=0x7f58d400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:36:57.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.877+0000 7f58cbfff700 1 -- 172.21.15.67:0/1093171900 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58d40078e0 con 0x7f58e40ffc20 2024-04-22T23:36:57.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.877+0000 7f58cbfff700 1 -- 172.21.15.67:0/1093171900 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f58d400f450 con 0x7f58e40ffc20 2024-04-22T23:36:57.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.877+0000 7f58e9075700 1 -- 172.21.15.67:0/1093171900 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58e4196a10 con 0x7f58e40ffc20 2024-04-22T23:36:57.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.877+0000 7f58cbfff700 1 -- 172.21.15.67:0/1093171900 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58d400e5e0 con 0x7f58e40ffc20 2024-04-22T23:36:57.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.877+0000 7f58e9075700 1 -- 172.21.15.67:0/1093171900 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58e4196f60 con 0x7f58e40ffc20 2024-04-22T23:36:57.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.878+0000 7f58e9075700 1 -- 172.21.15.67:0/1093171900 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58e404f2e0 con 0x7f58e40ffc20 2024-04-22T23:36:57.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.878+0000 7f58cbfff700 1 -- 172.21.15.67:0/1093171900 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f58d400e740 con 0x7f58e40ffc20 2024-04-22T23:36:57.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.879+0000 7f58cbfff700 1 --2- 172.21.15.67:0/1093171900 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f58cc06ecc0 0x7f58cc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:57.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.879+0000 7f58cbfff700 1 -- 172.21.15.67:0/1093171900 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f58d408efc0 con 0x7f58e40ffc20 2024-04-22T23:36:57.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.879+0000 7f58e259c700 1 --2- 172.21.15.67:0/1093171900 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f58cc06ecc0 0x7f58cc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:57.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.879+0000 7f58e259c700 1 --2- 172.21.15.67:0/1093171900 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f58cc06ecc0 0x7f58cc071180 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f58e4197f20 tx=0x7f58d801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:36:57.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:57.884+0000 7f58cbfff700 1 -- 172.21.15.67:0/1093171900 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f58d405b730 con 0x7f58e40ffc20 2024-04-22T23:36:58.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:57 smithi067 ceph-mon[112438]: pgmap v317: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:58.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.116+0000 7f58e9075700 1 -- 172.21.15.67:0/1093171900 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f58e4197350 con 0x7f58cc06ecc0 2024-04-22T23:36:58.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.117+0000 7f58cbfff700 1 -- 172.21.15.67:0/1093171900 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f58e4197350 con 0x7f58cc06ecc0 2024-04-22T23:36:58.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.120+0000 7f58e9075700 1 -- 172.21.15.67:0/1093171900 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f58cc06ecc0 msgr2=0x7f58cc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:58.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.120+0000 7f58e9075700 1 --2- 172.21.15.67:0/1093171900 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f58cc06ecc0 0x7f58cc071180 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f58e4197f20 tx=0x7f58d801a040 comp rx=0 tx=0).stop 2024-04-22T23:36:58.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.120+0000 7f58e9075700 1 -- 172.21.15.67:0/1093171900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58e40ffc20 msgr2=0x7f58e419c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:58.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.120+0000 7f58e9075700 1 --2- 172.21.15.67:0/1093171900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58e40ffc20 0x7f58e419c690 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f58d400c8a0 tx=0x7f58d400cbb0 comp rx=0 tx=0).stop 2024-04-22T23:36:58.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.120+0000 7f58e9075700 1 -- 172.21.15.67:0/1093171900 shutdown_connections 2024-04-22T23:36:58.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.120+0000 7f58e9075700 1 --2- 172.21.15.67:0/1093171900 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f58cc06ecc0 0x7f58cc071180 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:58.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.120+0000 7f58e9075700 1 --2- 172.21.15.67:0/1093171900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58e40ffc20 0x7f58e419c690 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:58.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.120+0000 7f58e9075700 1 --2- 172.21.15.67:0/1093171900 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f58e4105bd0 0x7f58e419cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:58.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.120+0000 7f58e9075700 1 -- 172.21.15.67:0/1093171900 >> 172.21.15.67:0/1093171900 conn(0x7f58e4075670 msgr2=0x7f58e4068700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:36:58.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.120+0000 7f58e9075700 1 -- 172.21.15.67:0/1093171900 shutdown_connections 2024-04-22T23:36:58.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.120+0000 7f58e9075700 1 -- 172.21.15.67:0/1093171900 wait complete. 2024-04-22T23:36:58.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.227+0000 7f95dffd8700 1 -- 172.21.15.67:0/2999960117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95d8100ea0 msgr2=0x7f95d8101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:58.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.227+0000 7f95dffd8700 1 --2- 172.21.15.67:0/2999960117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95d8100ea0 0x7f95d8101280 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f95d4009910 tx=0x7f95d4009c20 comp rx=0 tx=0).stop 2024-04-22T23:36:58.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.227+0000 7f95dffd8700 1 -- 172.21.15.67:0/2999960117 shutdown_connections 2024-04-22T23:36:58.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.227+0000 7f95dffd8700 1 --2- 172.21.15.67:0/2999960117 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f95d8101850 0x7f95d81058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:58.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.227+0000 7f95dffd8700 1 --2- 172.21.15.67:0/2999960117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95d8100ea0 0x7f95d8101280 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:58.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.227+0000 7f95dffd8700 1 -- 172.21.15.67:0/2999960117 >> 172.21.15.67:0/2999960117 conn(0x7f95d80fc750 msgr2=0x7f95d80feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:36:58.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.227+0000 7f95dffd8700 1 -- 172.21.15.67:0/2999960117 shutdown_connections 2024-04-22T23:36:58.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.227+0000 7f95dffd8700 1 -- 172.21.15.67:0/2999960117 wait complete. 2024-04-22T23:36:58.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.228+0000 7f95dffd8700 1 Processor -- start 2024-04-22T23:36:58.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.228+0000 7f95dffd8700 1 -- start start 2024-04-22T23:36:58.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.228+0000 7f95dffd8700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f95d8100ea0 0x7f95d819c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:58.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.228+0000 7f95dffd8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95d8101850 0x7f95d819cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:58.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.228+0000 7f95dffd8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d819d2e0 con 0x7f95d8101850 2024-04-22T23:36:58.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.228+0000 7f95dffd8700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d8196790 con 0x7f95d8100ea0 2024-04-22T23:36:58.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.228+0000 7f95dd573700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95d8101850 0x7f95d819cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:58.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.228+0000 7f95dd573700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95d8101850 0x7f95d819cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35552/0 (socket says 172.21.15.67:35552) 2024-04-22T23:36:58.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.228+0000 7f95dd573700 1 -- 172.21.15.67:0/2955383541 learned_addr learned my addr 172.21.15.67:0/2955383541 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:36:58.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.228+0000 7f95ddd74700 1 --2- 172.21.15.67:0/2955383541 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f95d8100ea0 0x7f95d819c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:58.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.228+0000 7f95dd573700 1 -- 172.21.15.67:0/2955383541 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f95d8100ea0 msgr2=0x7f95d819c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:58.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.228+0000 7f95dd573700 1 --2- 172.21.15.67:0/2955383541 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f95d8100ea0 0x7f95d819c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:58.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.228+0000 7f95dd573700 1 -- 172.21.15.67:0/2955383541 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d40095f0 con 0x7f95d8101850 2024-04-22T23:36:58.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.228+0000 7f95dd573700 1 --2- 172.21.15.67:0/2955383541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95d8101850 0x7f95d819cc50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f95c800d8d0 tx=0x7f95c800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:36:58.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.229+0000 7f95ceffd700 1 -- 172.21.15.67:0/2955383541 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f95c8004a00 con 0x7f95d8101850 2024-04-22T23:36:58.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.229+0000 7f95ceffd700 1 -- 172.21.15.67:0/2955383541 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f95c80054e0 con 0x7f95d8101850 2024-04-22T23:36:58.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.229+0000 7f95dffd8700 1 -- 172.21.15.67:0/2955383541 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95d8196a70 con 0x7f95d8101850 2024-04-22T23:36:58.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.229+0000 7f95ceffd700 1 -- 172.21.15.67:0/2955383541 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f95c800fc20 con 0x7f95d8101850 2024-04-22T23:36:58.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.229+0000 7f95dffd8700 1 -- 172.21.15.67:0/2955383541 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95d8196fc0 con 0x7f95d8101850 2024-04-22T23:36:58.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.230+0000 7f95dffd8700 1 -- 172.21.15.67:0/2955383541 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95d8109240 con 0x7f95d8101850 2024-04-22T23:36:58.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.230+0000 7f95ceffd700 1 -- 172.21.15.67:0/2955383541 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f95c800fd80 con 0x7f95d8101850 2024-04-22T23:36:58.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.230+0000 7f95ceffd700 1 --2- 172.21.15.67:0/2955383541 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f95c406ecc0 0x7f95c4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:58.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.231+0000 7f95ceffd700 1 -- 172.21.15.67:0/2955383541 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f95c808e840 con 0x7f95d8101850 2024-04-22T23:36:58.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.231+0000 7f95ddd74700 1 --2- 172.21.15.67:0/2955383541 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f95c406ecc0 0x7f95c4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:58.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.231+0000 7f95ddd74700 1 --2- 172.21.15.67:0/2955383541 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f95c406ecc0 0x7f95c4071180 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f95d4006010 tx=0x7f95d401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:36:58.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.235+0000 7f95ceffd700 1 -- 172.21.15.67:0/2955383541 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f95c805af90 con 0x7f95d8101850 2024-04-22T23:36:58.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:57 smithi134 ceph-mon[85630]: pgmap v317: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:58.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.452+0000 7f95dffd8700 1 -- 172.21.15.67:0/2955383541 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f95d8197db0 con 0x7f95c406ecc0 2024-04-22T23:36:58.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.464+0000 7f95ceffd700 1 -- 172.21.15.67:0/2955383541 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f95d8197db0 con 0x7f95c406ecc0 2024-04-22T23:36:58.458 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:36:58.458 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (10m) 3m ago 21m 25.5M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:36:58.458 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (22m) 3m ago 22m 9114k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:36:58.458 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (21m) 3m ago 21m 9042k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:36:58.458 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (8m) 3m ago 22m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:36:58.458 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (7m) 3m ago 21m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:36:58.458 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (10m) 3m ago 21m 79.3M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:36:58.458 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (4m) 3m ago 16m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:36:58.459 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (4m) 3m ago 16m 12.6M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:36:58.459 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (3m) 3m ago 16m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:36:58.459 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (16m) 3m ago 16m 12.2M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:36:58.459 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (12m) 3m ago 23m 511M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:36:58.459 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (11m) 3m ago 21m 453M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:36:58.459 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (9m) 3m ago 23m 55.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:36:58.459 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (8m) 3m ago 20m 46.0M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:36:58.459 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (11m) 3m ago 22m 15.4M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:36:58.459 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (11m) 3m ago 21m 16.1M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:36:58.459 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (7m) 3m ago 20m 99.2M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:36:58.459 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (7m) 3m ago 19m 127M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:36:58.459 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (6m) 3m ago 19m 79.9M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:36:58.459 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (5m) 3m ago 18m 97.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:36:58.459 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (5m) 3m ago 18m 82.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:36:58.460 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (4m) 3m ago 18m 77.6M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:36:58.460 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (11m) 3m ago 21m 73.9M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:36:58.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.466+0000 7f95dffd8700 1 -- 172.21.15.67:0/2955383541 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f95c406ecc0 msgr2=0x7f95c4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:58.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.466+0000 7f95dffd8700 1 --2- 172.21.15.67:0/2955383541 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f95c406ecc0 0x7f95c4071180 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f95d4006010 tx=0x7f95d401a040 comp rx=0 tx=0).stop 2024-04-22T23:36:58.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.466+0000 7f95dffd8700 1 -- 172.21.15.67:0/2955383541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95d8101850 msgr2=0x7f95d819cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:58.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.466+0000 7f95dffd8700 1 --2- 172.21.15.67:0/2955383541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95d8101850 0x7f95d819cc50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f95c800d8d0 tx=0x7f95c800dbe0 comp rx=0 tx=0).stop 2024-04-22T23:36:58.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.466+0000 7f95dffd8700 1 -- 172.21.15.67:0/2955383541 shutdown_connections 2024-04-22T23:36:58.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.466+0000 7f95dffd8700 1 --2- 172.21.15.67:0/2955383541 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f95c406ecc0 0x7f95c4071180 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:58.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.466+0000 7f95dffd8700 1 --2- 172.21.15.67:0/2955383541 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f95d8100ea0 0x7f95d819c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:58.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.466+0000 7f95dffd8700 1 --2- 172.21.15.67:0/2955383541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95d8101850 0x7f95d819cc50 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:58.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.466+0000 7f95dffd8700 1 -- 172.21.15.67:0/2955383541 >> 172.21.15.67:0/2955383541 conn(0x7f95d80fc750 msgr2=0x7f95d80fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:36:58.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.466+0000 7f95dffd8700 1 -- 172.21.15.67:0/2955383541 shutdown_connections 2024-04-22T23:36:58.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.466+0000 7f95dffd8700 1 -- 172.21.15.67:0/2955383541 wait complete. 2024-04-22T23:36:58.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.571+0000 7fe62a2e1700 1 -- 172.21.15.67:0/3282140457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe624068580 msgr2=0x7fe6240689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:58.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.571+0000 7fe62a2e1700 1 --2- 172.21.15.67:0/3282140457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe624068580 0x7fe6240689e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe614009910 tx=0x7fe614009c20 comp rx=0 tx=0).stop 2024-04-22T23:36:58.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.571+0000 7fe62a2e1700 1 -- 172.21.15.67:0/3282140457 shutdown_connections 2024-04-22T23:36:58.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.571+0000 7fe62a2e1700 1 --2- 172.21.15.67:0/3282140457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe624068580 0x7fe6240689e0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:58.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.571+0000 7fe62a2e1700 1 --2- 172.21.15.67:0/3282140457 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe624106cd0 0x7fe6241070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:58.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.571+0000 7fe62a2e1700 1 -- 172.21.15.67:0/3282140457 >> 172.21.15.67:0/3282140457 conn(0x7fe624075670 msgr2=0x7fe624075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:36:58.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.571+0000 7fe62a2e1700 1 -- 172.21.15.67:0/3282140457 shutdown_connections 2024-04-22T23:36:58.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.571+0000 7fe62a2e1700 1 -- 172.21.15.67:0/3282140457 wait complete. 2024-04-22T23:36:58.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.572+0000 7fe62a2e1700 1 Processor -- start 2024-04-22T23:36:58.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.572+0000 7fe62a2e1700 1 -- start start 2024-04-22T23:36:58.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.572+0000 7fe62a2e1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe624068580 0x7fe62419e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:58.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.572+0000 7fe62a2e1700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe624106cd0 0x7fe62419ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:58.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.572+0000 7fe62a2e1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe62419f400 con 0x7fe624068580 2024-04-22T23:36:58.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.572+0000 7fe62a2e1700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6241988b0 con 0x7fe624106cd0 2024-04-22T23:36:58.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.572+0000 7fe623fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe624068580 0x7fe62419e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:58.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.572+0000 7fe623fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe624068580 0x7fe62419e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35570/0 (socket says 172.21.15.67:35570) 2024-04-22T23:36:58.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.572+0000 7fe623fff700 1 -- 172.21.15.67:0/2650043430 learned_addr learned my addr 172.21.15.67:0/2650043430 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:36:58.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.572+0000 7fe6237fe700 1 --2- 172.21.15.67:0/2650043430 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe624106cd0 0x7fe62419ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:58.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.573+0000 7fe623fff700 1 -- 172.21.15.67:0/2650043430 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe624106cd0 msgr2=0x7fe62419ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.573+0000 7fe623fff700 1 --2- 172.21.15.67:0/2650043430 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe624106cd0 0x7fe62419ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.573+0000 7fe623fff700 1 -- 172.21.15.67:0/2650043430 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6140095f0 con 0x7fe624068580 2024-04-22T23:36:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.573+0000 7fe623fff700 1 --2- 172.21.15.67:0/2650043430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe624068580 0x7fe62419e830 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe60c00d8d0 tx=0x7fe60c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:36:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.573+0000 7fe6217fa700 1 -- 172.21.15.67:0/2650043430 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe60c0098a0 con 0x7fe624068580 2024-04-22T23:36:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.573+0000 7fe6217fa700 1 -- 172.21.15.67:0/2650043430 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe60c010460 con 0x7fe624068580 2024-04-22T23:36:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.573+0000 7fe62a2e1700 1 -- 172.21.15.67:0/2650043430 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe624198b90 con 0x7fe624068580 2024-04-22T23:36:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.573+0000 7fe6217fa700 1 -- 172.21.15.67:0/2650043430 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe60c00f5f0 con 0x7fe624068580 2024-04-22T23:36:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.573+0000 7fe62a2e1700 1 -- 172.21.15.67:0/2650043430 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6241990b0 con 0x7fe624068580 2024-04-22T23:36:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.574+0000 7fe62a2e1700 1 -- 172.21.15.67:0/2650043430 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe62404f2e0 con 0x7fe624068580 2024-04-22T23:36:58.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.575+0000 7fe6217fa700 1 -- 172.21.15.67:0/2650043430 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe60c009a00 con 0x7fe624068580 2024-04-22T23:36:58.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.575+0000 7fe6217fa700 1 --2- 172.21.15.67:0/2650043430 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe61006ecc0 0x7fe610071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:58.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.575+0000 7fe6217fa700 1 -- 172.21.15.67:0/2650043430 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fe60c08eea0 con 0x7fe624068580 2024-04-22T23:36:58.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.575+0000 7fe6237fe700 1 --2- 172.21.15.67:0/2650043430 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe61006ecc0 0x7fe610071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:58.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.576+0000 7fe6237fe700 1 --2- 172.21.15.67:0/2650043430 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe61006ecc0 0x7fe610071180 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe61400b5c0 tx=0x7fe61401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:36:58.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.580+0000 7fe6217fa700 1 -- 172.21.15.67:0/2650043430 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe60c05b610 con 0x7fe624068580 2024-04-22T23:36:58.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.883+0000 7fe62a2e1700 1 -- 172.21.15.67:0/2650043430 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe624199390 con 0x7fe624068580 2024-04-22T23:36:58.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.883+0000 7fe6217fa700 1 -- 172.21.15.67:0/2650043430 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fe60c05b1a0 con 0x7fe624068580 2024-04-22T23:36:58.877 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:36:58.878 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:36:58.878 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:36:58.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:36:58.878 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:36:58.878 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:36:58.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:36:58.878 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:36:58.878 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:36:58.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:36:58.878 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:36:58.878 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:36:58.878 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 3 2024-04-22T23:36:58.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:36:58.879 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:36:58.879 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:36:58.879 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 13 2024-04-22T23:36:58.879 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:36:58.879 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:36:58.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.885+0000 7fe62a2e1700 1 -- 172.21.15.67:0/2650043430 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe61006ecc0 msgr2=0x7fe610071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:58.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.885+0000 7fe62a2e1700 1 --2- 172.21.15.67:0/2650043430 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe61006ecc0 0x7fe610071180 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe61400b5c0 tx=0x7fe61401a040 comp rx=0 tx=0).stop 2024-04-22T23:36:58.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.885+0000 7fe62a2e1700 1 -- 172.21.15.67:0/2650043430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe624068580 msgr2=0x7fe62419e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:58.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.885+0000 7fe62a2e1700 1 --2- 172.21.15.67:0/2650043430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe624068580 0x7fe62419e830 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe60c00d8d0 tx=0x7fe60c00dbe0 comp rx=0 tx=0).stop 2024-04-22T23:36:58.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.886+0000 7fe62a2e1700 1 -- 172.21.15.67:0/2650043430 shutdown_connections 2024-04-22T23:36:58.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.886+0000 7fe62a2e1700 1 --2- 172.21.15.67:0/2650043430 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe61006ecc0 0x7fe610071180 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:58.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.886+0000 7fe62a2e1700 1 --2- 172.21.15.67:0/2650043430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe624068580 0x7fe62419e830 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:58.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.886+0000 7fe62a2e1700 1 --2- 172.21.15.67:0/2650043430 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe624106cd0 0x7fe62419ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:58.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.886+0000 7fe62a2e1700 1 -- 172.21.15.67:0/2650043430 >> 172.21.15.67:0/2650043430 conn(0x7fe624075670 msgr2=0x7fe6240fe9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:36:58.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.886+0000 7fe62a2e1700 1 -- 172.21.15.67:0/2650043430 shutdown_connections 2024-04-22T23:36:58.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.886+0000 7fe62a2e1700 1 -- 172.21.15.67:0/2650043430 wait complete. 2024-04-22T23:36:58.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.992+0000 7f8928547700 1 -- 172.21.15.67:0/3608336263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89200ffc20 msgr2=0x7f8920100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:58.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.992+0000 7f8928547700 1 --2- 172.21.15.67:0/3608336263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89200ffc20 0x7f8920100080 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8910009910 tx=0x7f8910009c20 comp rx=0 tx=0).stop 2024-04-22T23:36:58.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.992+0000 7f8928547700 1 -- 172.21.15.67:0/3608336263 shutdown_connections 2024-04-22T23:36:58.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.992+0000 7f8928547700 1 --2- 172.21.15.67:0/3608336263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89200ffc20 0x7f8920100080 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:58.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.992+0000 7f8928547700 1 --2- 172.21.15.67:0/3608336263 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8920105bd0 0x7f8920105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:58.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.992+0000 7f8928547700 1 -- 172.21.15.67:0/3608336263 >> 172.21.15.67:0/3608336263 conn(0x7f8920075670 msgr2=0x7f8920075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:36:58.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.992+0000 7f8928547700 1 -- 172.21.15.67:0/3608336263 shutdown_connections 2024-04-22T23:36:58.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.992+0000 7f8928547700 1 -- 172.21.15.67:0/3608336263 wait complete. 2024-04-22T23:36:58.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.993+0000 7f8928547700 1 Processor -- start 2024-04-22T23:36:58.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.993+0000 7f8928547700 1 -- start start 2024-04-22T23:36:58.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.993+0000 7f8928547700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89200ffc20 0x7f8920197760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:58.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.993+0000 7f8928547700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8920105bd0 0x7f8920197ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:58.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.993+0000 7f8928547700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8920198380 con 0x7f89200ffc20 2024-04-22T23:36:58.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.993+0000 7f8928547700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89201984f0 con 0x7f8920105bd0 2024-04-22T23:36:58.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.993+0000 7f89262e3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89200ffc20 0x7f8920197760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:58.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.993+0000 7f89262e3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89200ffc20 0x7f8920197760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53294/0 (socket says 172.21.15.67:53294) 2024-04-22T23:36:58.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.993+0000 7f89262e3700 1 -- 172.21.15.67:0/3370939150 learned_addr learned my addr 172.21.15.67:0/3370939150 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:36:58.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.993+0000 7f8925ae2700 1 --2- 172.21.15.67:0/3370939150 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8920105bd0 0x7f8920197ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:58.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.993+0000 7f89262e3700 1 -- 172.21.15.67:0/3370939150 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8920105bd0 msgr2=0x7f8920197ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:58.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.993+0000 7f89262e3700 1 --2- 172.21.15.67:0/3370939150 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8920105bd0 0x7f8920197ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:58.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.993+0000 7f89262e3700 1 -- 172.21.15.67:0/3370939150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89100095f0 con 0x7f89200ffc20 2024-04-22T23:36:58.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.994+0000 7f89262e3700 1 --2- 172.21.15.67:0/3370939150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89200ffc20 0x7f8920197760 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f891c00d8d0 tx=0x7f891c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:36:58.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.994+0000 7f89177fe700 1 -- 172.21.15.67:0/3370939150 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f891c0098a0 con 0x7f89200ffc20 2024-04-22T23:36:58.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.994+0000 7f89177fe700 1 -- 172.21.15.67:0/3370939150 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f891c010460 con 0x7f89200ffc20 2024-04-22T23:36:58.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.994+0000 7f8928547700 1 -- 172.21.15.67:0/3370939150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89201a07d0 con 0x7f89200ffc20 2024-04-22T23:36:58.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.994+0000 7f89177fe700 1 -- 172.21.15.67:0/3370939150 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f891c00f5f0 con 0x7f89200ffc20 2024-04-22T23:36:58.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.994+0000 7f8928547700 1 -- 172.21.15.67:0/3370939150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89201a0d20 con 0x7f89200ffc20 2024-04-22T23:36:58.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.995+0000 7f8928547700 1 -- 172.21.15.67:0/3370939150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89201080d0 con 0x7f89200ffc20 2024-04-22T23:36:58.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.995+0000 7f89177fe700 1 -- 172.21.15.67:0/3370939150 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f891c009a00 con 0x7f89200ffc20 2024-04-22T23:36:58.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.996+0000 7f89177fe700 1 --2- 172.21.15.67:0/3370939150 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f890c06ecc0 0x7f890c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:58.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.996+0000 7f89177fe700 1 -- 172.21.15.67:0/3370939150 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f891c08ee00 con 0x7f89200ffc20 2024-04-22T23:36:58.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.996+0000 7f8925ae2700 1 --2- 172.21.15.67:0/3370939150 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f890c06ecc0 0x7f890c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:58.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:58.996+0000 7f8925ae2700 1 --2- 172.21.15.67:0/3370939150 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f890c06ecc0 0x7f890c071180 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8920068e30 tx=0x7f891001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:36:58.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.001+0000 7f89177fe700 1 -- 172.21.15.67:0/3370939150 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f891c057a60 con 0x7f89200ffc20 2024-04-22T23:36:59.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:58 smithi134 ceph-mon[85630]: from='client.34624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:36:59.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.253+0000 7f8928547700 1 -- 172.21.15.67:0/3370939150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f892004f2e0 con 0x7f89200ffc20 2024-04-22T23:36:59.248 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:58 smithi067 ceph-mon[112438]: from='client.34624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:36:59.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.254+0000 7f89177fe700 1 -- 172.21.15.67:0/3370939150 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1871 (secure 0 0 0) 0x7f891c016760 con 0x7f89200ffc20 2024-04-22T23:36:59.248 INFO:teuthology.orchestra.run.smithi067.stdout:e31 2024-04-22T23:36:59.248 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:36:59.248 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:36:59.248 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:36:59.248 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:36:59.249 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:36:59.249 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:36:59.249 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:36:59.249 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:36:59.249 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:36:59.249 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:36:59.249 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:36:59.249 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:36:59.249 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:36:59.249 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:36:59.249 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:36:59.249 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:36:59.249 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:36:59.249 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:36:59.250 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:36:59.250 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:36:59.250 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:36:59.250 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:36:59.250 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:36:59.250 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:36:59.250 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:36:59.250 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:36:59.250 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:36:59.250 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:36:59.250 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:36:59.250 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:36:59.250 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:36:59.251 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:36:59.251 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:36:59.251 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:36:59.251 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:36:59.251 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:36:59.251 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:36:59.251 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:36:59.251 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:36:59.251 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44259} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1827288036,v1:172.21.15.134:6827/1827288036] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:36:59.251 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:36:59.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.256+0000 7f8928547700 1 -- 172.21.15.67:0/3370939150 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f890c06ecc0 msgr2=0x7f890c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:59.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.256+0000 7f8928547700 1 --2- 172.21.15.67:0/3370939150 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f890c06ecc0 0x7f890c071180 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8920068e30 tx=0x7f891001a040 comp rx=0 tx=0).stop 2024-04-22T23:36:59.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.256+0000 7f8928547700 1 -- 172.21.15.67:0/3370939150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89200ffc20 msgr2=0x7f8920197760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:59.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.256+0000 7f8928547700 1 --2- 172.21.15.67:0/3370939150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89200ffc20 0x7f8920197760 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f891c00d8d0 tx=0x7f891c00dbe0 comp rx=0 tx=0).stop 2024-04-22T23:36:59.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.256+0000 7f8928547700 1 -- 172.21.15.67:0/3370939150 shutdown_connections 2024-04-22T23:36:59.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.256+0000 7f8928547700 1 --2- 172.21.15.67:0/3370939150 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f890c06ecc0 0x7f890c071180 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:59.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.256+0000 7f8928547700 1 --2- 172.21.15.67:0/3370939150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89200ffc20 0x7f8920197760 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:59.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.256+0000 7f8928547700 1 --2- 172.21.15.67:0/3370939150 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8920105bd0 0x7f8920197ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:59.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.256+0000 7f8928547700 1 -- 172.21.15.67:0/3370939150 >> 172.21.15.67:0/3370939150 conn(0x7f8920075670 msgr2=0x7f89200fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:36:59.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.256+0000 7f8928547700 1 -- 172.21.15.67:0/3370939150 shutdown_connections 2024-04-22T23:36:59.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.256+0000 7f8928547700 1 -- 172.21.15.67:0/3370939150 wait complete. 2024-04-22T23:36:59.253 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 31 2024-04-22T23:36:59.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.364+0000 7f4487e25700 1 -- 172.21.15.67:0/1462249870 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44800ffda0 msgr2=0x7f4480100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:59.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.364+0000 7f4487e25700 1 --2- 172.21.15.67:0/1462249870 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44800ffda0 0x7f4480100180 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f447c009910 tx=0x7f447c009c20 comp rx=0 tx=0).stop 2024-04-22T23:36:59.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.364+0000 7f4487e25700 1 -- 172.21.15.67:0/1462249870 shutdown_connections 2024-04-22T23:36:59.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.364+0000 7f4487e25700 1 --2- 172.21.15.67:0/1462249870 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4480100750 0x7f44801047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:59.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.364+0000 7f4487e25700 1 --2- 172.21.15.67:0/1462249870 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44800ffda0 0x7f4480100180 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:59.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.364+0000 7f4487e25700 1 -- 172.21.15.67:0/1462249870 >> 172.21.15.67:0/1462249870 conn(0x7f44800fb650 msgr2=0x7f44800fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:36:59.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.364+0000 7f4487e25700 1 -- 172.21.15.67:0/1462249870 shutdown_connections 2024-04-22T23:36:59.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.364+0000 7f4487e25700 1 -- 172.21.15.67:0/1462249870 wait complete. 2024-04-22T23:36:59.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.365+0000 7f4487e25700 1 Processor -- start 2024-04-22T23:36:59.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.365+0000 7f4487e25700 1 -- start start 2024-04-22T23:36:59.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.365+0000 7f4487e25700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44800ffda0 0x7f448019c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:59.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.365+0000 7f4487e25700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4480100750 0x7f448019cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:59.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.365+0000 7f4487e25700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f448019d2c0 con 0x7f44800ffda0 2024-04-22T23:36:59.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.365+0000 7f4487e25700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4480196770 con 0x7f4480100750 2024-04-22T23:36:59.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.365+0000 7f4485bc1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44800ffda0 0x7f448019c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:59.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.365+0000 7f4485bc1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44800ffda0 0x7f448019c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53322/0 (socket says 172.21.15.67:53322) 2024-04-22T23:36:59.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.365+0000 7f4485bc1700 1 -- 172.21.15.67:0/3901796582 learned_addr learned my addr 172.21.15.67:0/3901796582 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:36:59.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.365+0000 7f44853c0700 1 --2- 172.21.15.67:0/3901796582 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4480100750 0x7f448019cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:59.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.366+0000 7f4485bc1700 1 -- 172.21.15.67:0/3901796582 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4480100750 msgr2=0x7f448019cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:59.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.366+0000 7f4485bc1700 1 --2- 172.21.15.67:0/3901796582 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4480100750 0x7f448019cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:59.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.366+0000 7f4485bc1700 1 -- 172.21.15.67:0/3901796582 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f447c0095f0 con 0x7f44800ffda0 2024-04-22T23:36:59.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.366+0000 7f44853c0700 1 --2- 172.21.15.67:0/3901796582 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4480100750 0x7f448019cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:36:59.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.366+0000 7f4485bc1700 1 --2- 172.21.15.67:0/3901796582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44800ffda0 0x7f448019c6f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f447c006010 tx=0x7f447c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:36:59.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.366+0000 7f4476ffd700 1 -- 172.21.15.67:0/3901796582 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f447c01d070 con 0x7f44800ffda0 2024-04-22T23:36:59.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.366+0000 7f4476ffd700 1 -- 172.21.15.67:0/3901796582 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f447c00f460 con 0x7f44800ffda0 2024-04-22T23:36:59.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.366+0000 7f4487e25700 1 -- 172.21.15.67:0/3901796582 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44801969f0 con 0x7f44800ffda0 2024-04-22T23:36:59.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.366+0000 7f4476ffd700 1 -- 172.21.15.67:0/3901796582 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f447c017630 con 0x7f44800ffda0 2024-04-22T23:36:59.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.366+0000 7f4487e25700 1 -- 172.21.15.67:0/3901796582 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4480196ee0 con 0x7f44800ffda0 2024-04-22T23:36:59.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.367+0000 7f4487e25700 1 -- 172.21.15.67:0/3901796582 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f448004f2e0 con 0x7f44800ffda0 2024-04-22T23:36:59.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.368+0000 7f4476ffd700 1 -- 172.21.15.67:0/3901796582 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f447c00f5d0 con 0x7f44800ffda0 2024-04-22T23:36:59.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.368+0000 7f4476ffd700 1 --2- 172.21.15.67:0/3901796582 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f446c06ecc0 0x7f446c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:59.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.368+0000 7f44853c0700 1 --2- 172.21.15.67:0/3901796582 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f446c06ecc0 0x7f446c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:59.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.368+0000 7f4476ffd700 1 -- 172.21.15.67:0/3901796582 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f447c090480 con 0x7f44800ffda0 2024-04-22T23:36:59.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.369+0000 7f44853c0700 1 --2- 172.21.15.67:0/3901796582 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f446c06ecc0 0x7f446c071180 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4470005950 tx=0x7f44700058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:36:59.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.373+0000 7f4476ffd700 1 -- 172.21.15.67:0/3901796582 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f447c05cbf0 con 0x7f44800ffda0 2024-04-22T23:36:59.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.603+0000 7f4487e25700 1 -- 172.21.15.67:0/3901796582 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f44801971c0 con 0x7f446c06ecc0 2024-04-22T23:36:59.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.604+0000 7f4476ffd700 1 -- 172.21.15.67:0/3901796582 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f44801971c0 con 0x7f446c06ecc0 2024-04-22T23:36:59.598 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:36:59.598 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:36:59.599 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:36:59.599 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:36:59.599 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:36:59.599 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:36:59.599 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:36:59.599 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:36:59.599 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:36:59.599 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:36:59.599 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T23:36:59.599 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:36:59.599 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:36:59.599 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:36:59.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.607+0000 7f4487e25700 1 -- 172.21.15.67:0/3901796582 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f446c06ecc0 msgr2=0x7f446c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:59.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.607+0000 7f4487e25700 1 --2- 172.21.15.67:0/3901796582 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f446c06ecc0 0x7f446c071180 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4470005950 tx=0x7f44700058e0 comp rx=0 tx=0).stop 2024-04-22T23:36:59.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.607+0000 7f4487e25700 1 -- 172.21.15.67:0/3901796582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44800ffda0 msgr2=0x7f448019c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:59.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.607+0000 7f4487e25700 1 --2- 172.21.15.67:0/3901796582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44800ffda0 0x7f448019c6f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f447c006010 tx=0x7f447c00be20 comp rx=0 tx=0).stop 2024-04-22T23:36:59.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.607+0000 7f4487e25700 1 -- 172.21.15.67:0/3901796582 shutdown_connections 2024-04-22T23:36:59.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.607+0000 7f4487e25700 1 --2- 172.21.15.67:0/3901796582 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f446c06ecc0 0x7f446c071180 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:59.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.607+0000 7f4487e25700 1 --2- 172.21.15.67:0/3901796582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44800ffda0 0x7f448019c6f0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:59.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.607+0000 7f4487e25700 1 --2- 172.21.15.67:0/3901796582 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4480100750 0x7f448019cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:59.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.607+0000 7f4487e25700 1 -- 172.21.15.67:0/3901796582 >> 172.21.15.67:0/3901796582 conn(0x7f44800fb650 msgr2=0x7f4480103fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:36:59.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.607+0000 7f4487e25700 1 -- 172.21.15.67:0/3901796582 shutdown_connections 2024-04-22T23:36:59.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.607+0000 7f4487e25700 1 -- 172.21.15.67:0/3901796582 wait complete. 2024-04-22T23:36:59.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.714+0000 7f8cfddd3700 1 -- 172.21.15.67:0/3669221850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cf8101850 msgr2=0x7f8cf81058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:59.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.714+0000 7f8cfddd3700 1 --2- 172.21.15.67:0/3669221850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cf8101850 0x7f8cf81058a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8cec009910 tx=0x7f8cec009c20 comp rx=0 tx=0).stop 2024-04-22T23:36:59.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.714+0000 7f8cfddd3700 1 -- 172.21.15.67:0/3669221850 shutdown_connections 2024-04-22T23:36:59.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.714+0000 7f8cfddd3700 1 --2- 172.21.15.67:0/3669221850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cf8101850 0x7f8cf81058a0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:59.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.714+0000 7f8cfddd3700 1 --2- 172.21.15.67:0/3669221850 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8cf8100ea0 0x7f8cf8101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:59.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.714+0000 7f8cfddd3700 1 -- 172.21.15.67:0/3669221850 >> 172.21.15.67:0/3669221850 conn(0x7f8cf80fc750 msgr2=0x7f8cf80feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:36:59.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.714+0000 7f8cfddd3700 1 -- 172.21.15.67:0/3669221850 shutdown_connections 2024-04-22T23:36:59.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.714+0000 7f8cfddd3700 1 -- 172.21.15.67:0/3669221850 wait complete. 2024-04-22T23:36:59.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.714+0000 7f8cfddd3700 1 Processor -- start 2024-04-22T23:36:59.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.714+0000 7f8cfddd3700 1 -- start start 2024-04-22T23:36:59.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.715+0000 7f8cfddd3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cf8100ea0 0x7f8cf8073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:59.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.715+0000 7f8cfddd3700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8cf8101850 0x7f8cf80735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:59.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.715+0000 7f8cfddd3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf8073b10 con 0x7f8cf8100ea0 2024-04-22T23:36:59.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.715+0000 7f8cfddd3700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf8073c50 con 0x7f8cf8101850 2024-04-22T23:36:59.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.715+0000 7f8cf77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cf8100ea0 0x7f8cf8073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:59.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.715+0000 7f8cf77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cf8100ea0 0x7f8cf8073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53340/0 (socket says 172.21.15.67:53340) 2024-04-22T23:36:59.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.715+0000 7f8cf77fe700 1 -- 172.21.15.67:0/3523320484 learned_addr learned my addr 172.21.15.67:0/3523320484 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:36:59.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.715+0000 7f8cf6ffd700 1 --2- 172.21.15.67:0/3523320484 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8cf8101850 0x7f8cf80735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:59.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.715+0000 7f8cf77fe700 1 -- 172.21.15.67:0/3523320484 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8cf8101850 msgr2=0x7f8cf80735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:36:59.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.715+0000 7f8cf77fe700 1 --2- 172.21.15.67:0/3523320484 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8cf8101850 0x7f8cf80735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:36:59.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.715+0000 7f8cf77fe700 1 -- 172.21.15.67:0/3523320484 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cec0095f0 con 0x7f8cf8100ea0 2024-04-22T23:36:59.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.715+0000 7f8cf77fe700 1 --2- 172.21.15.67:0/3523320484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cf8100ea0 0x7f8cf8073090 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8ce800d8d0 tx=0x7f8ce800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:36:59.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.715+0000 7f8cf4ff9700 1 -- 172.21.15.67:0/3523320484 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ce80098a0 con 0x7f8cf8100ea0 2024-04-22T23:36:59.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.715+0000 7f8cf4ff9700 1 -- 172.21.15.67:0/3523320484 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ce8010460 con 0x7f8cf8100ea0 2024-04-22T23:36:59.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.715+0000 7f8cfddd3700 1 -- 172.21.15.67:0/3523320484 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cf81a2b00 con 0x7f8cf8100ea0 2024-04-22T23:36:59.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.715+0000 7f8cf4ff9700 1 -- 172.21.15.67:0/3523320484 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ce800f5f0 con 0x7f8cf8100ea0 2024-04-22T23:36:59.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.716+0000 7f8cfddd3700 1 -- 172.21.15.67:0/3523320484 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cf81a2eb0 con 0x7f8cf8100ea0 2024-04-22T23:36:59.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.716+0000 7f8cfddd3700 1 -- 172.21.15.67:0/3523320484 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cf81092a0 con 0x7f8cf8100ea0 2024-04-22T23:36:59.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.717+0000 7f8cf4ff9700 1 -- 172.21.15.67:0/3523320484 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8ce8009a00 con 0x7f8cf8100ea0 2024-04-22T23:36:59.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.717+0000 7f8cf4ff9700 1 --2- 172.21.15.67:0/3523320484 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8ce006ecc0 0x7f8ce0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:36:59.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.718+0000 7f8cf4ff9700 1 -- 172.21.15.67:0/3523320484 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f8ce808edb0 con 0x7f8cf8100ea0 2024-04-22T23:36:59.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.718+0000 7f8cf6ffd700 1 --2- 172.21.15.67:0/3523320484 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8ce006ecc0 0x7f8ce0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:36:59.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.718+0000 7f8cf6ffd700 1 --2- 172.21.15.67:0/3523320484 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8ce006ecc0 0x7f8ce0071180 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8cec006010 tx=0x7f8cec01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:36:59.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:36:59.722+0000 7f8cf4ff9700 1 -- 172.21.15.67:0/3523320484 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f8ce8057a70 con 0x7f8cf8100ea0 2024-04-22T23:36:59.987 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:59 smithi134 ceph-mon[85630]: pgmap v318: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:36:59.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:59 smithi134 ceph-mon[85630]: from='client.34628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:36:59.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:59 smithi134 ceph-mon[85630]: from='client.34632 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:36:59.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:59 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2650043430' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:36:59.988 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:36:59 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3370939150' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:37:00.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:00.017+0000 7f8cfddd3700 1 -- 172.21.15.67:0/3523320484 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8cf8074690 con 0x7f8cf8100ea0 2024-04-22T23:37:00.012 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:59 smithi067 ceph-mon[112438]: pgmap v318: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:00.012 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:59 smithi067 ceph-mon[112438]: from='client.34628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:37:00.013 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:59 smithi067 ceph-mon[112438]: from='client.34632 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:37:00.013 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:59 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2650043430' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:37:00.013 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:36:59 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3370939150' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:37:00.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:00.018+0000 7f8cf4ff9700 1 -- 172.21.15.67:0/3523320484 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f8ce80161c0 con 0x7f8cf8100ea0 2024-04-22T23:37:00.013 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:37:00.013 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:37:00.013 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:37:00.013 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:37:00.013 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:37:00.014 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:37:00.014 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:37:00.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:00.020+0000 7f8cfddd3700 1 -- 172.21.15.67:0/3523320484 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8ce006ecc0 msgr2=0x7f8ce0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:37:00.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:00.020+0000 7f8cfddd3700 1 --2- 172.21.15.67:0/3523320484 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8ce006ecc0 0x7f8ce0071180 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8cec006010 tx=0x7f8cec01a040 comp rx=0 tx=0).stop 2024-04-22T23:37:00.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:00.020+0000 7f8cfddd3700 1 -- 172.21.15.67:0/3523320484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cf8100ea0 msgr2=0x7f8cf8073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:37:00.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:00.020+0000 7f8cfddd3700 1 --2- 172.21.15.67:0/3523320484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cf8100ea0 0x7f8cf8073090 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8ce800d8d0 tx=0x7f8ce800dbe0 comp rx=0 tx=0).stop 2024-04-22T23:37:00.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:00.020+0000 7f8cfddd3700 1 -- 172.21.15.67:0/3523320484 shutdown_connections 2024-04-22T23:37:00.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:00.020+0000 7f8cfddd3700 1 --2- 172.21.15.67:0/3523320484 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8ce006ecc0 0x7f8ce0071180 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:00.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:00.020+0000 7f8cfddd3700 1 --2- 172.21.15.67:0/3523320484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cf8100ea0 0x7f8cf8073090 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:00.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:00.020+0000 7f8cfddd3700 1 --2- 172.21.15.67:0/3523320484 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8cf8101850 0x7f8cf80735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:00.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:00.021+0000 7f8cfddd3700 1 -- 172.21.15.67:0/3523320484 >> 172.21.15.67:0/3523320484 conn(0x7f8cf80fc750 msgr2=0x7f8cf80fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:37:00.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:00.021+0000 7f8cfddd3700 1 -- 172.21.15.67:0/3523320484 shutdown_connections 2024-04-22T23:37:00.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:00.021+0000 7f8cfddd3700 1 -- 172.21.15.67:0/3523320484 wait complete. 2024-04-22T23:37:01.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:00 smithi134 ceph-mon[85630]: from='client.34644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:37:01.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:00 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3523320484' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:37:01.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:00 smithi067 ceph-mon[112438]: from='client.34644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:37:01.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:00 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3523320484' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:37:02.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:01 smithi134 ceph-mon[85630]: pgmap v319: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:02.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:01 smithi067 ceph-mon[112438]: pgmap v319: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:04.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:03 smithi134 ceph-mon[85630]: pgmap v320: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:04.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:03 smithi067 ceph-mon[112438]: pgmap v320: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:06.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:05 smithi134 ceph-mon[85630]: pgmap v321: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:06.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:05 smithi067 ceph-mon[112438]: pgmap v321: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:08.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:07 smithi134 ceph-mon[85630]: pgmap v322: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:08.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:07 smithi067 ceph-mon[112438]: pgmap v322: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:10.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:09 smithi134 ceph-mon[85630]: pgmap v323: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:10.265 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:09 smithi067 ceph-mon[112438]: pgmap v323: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:12.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:11 smithi134 ceph-mon[85630]: pgmap v324: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:12.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:37:12.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:11 smithi067 ceph-mon[112438]: pgmap v324: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:12.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:37:14.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:13 smithi134 ceph-mon[85630]: pgmap v325: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:14.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:13 smithi067 ceph-mon[112438]: pgmap v325: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:16.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:15 smithi134 ceph-mon[85630]: pgmap v326: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:16.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:15 smithi067 ceph-mon[112438]: pgmap v326: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:18.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:17 smithi134 ceph-mon[85630]: pgmap v327: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:18.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:17 smithi067 ceph-mon[112438]: pgmap v327: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:20.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:19 smithi134 ceph-mon[85630]: pgmap v328: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:20.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:19 smithi067 ceph-mon[112438]: pgmap v328: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:22.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:21 smithi134 ceph-mon[85630]: pgmap v329: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:22.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:21 smithi067 ceph-mon[112438]: pgmap v329: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:24.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:23 smithi134 ceph-mon[85630]: pgmap v330: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:24.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:23 smithi067 ceph-mon[112438]: pgmap v330: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:26.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:25 smithi134 ceph-mon[85630]: pgmap v331: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:26.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:25 smithi067 ceph-mon[112438]: pgmap v331: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:27.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:37:27.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:37:28.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:27 smithi134 ceph-mon[85630]: pgmap v332: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:28.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:27 smithi067 ceph-mon[112438]: pgmap v332: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:30.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.142+0000 7feb07c9c700 1 -- 172.21.15.67:0/3091412998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb000ffc20 msgr2=0x7feb00100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:37:30.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.142+0000 7feb07c9c700 1 --2- 172.21.15.67:0/3091412998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb000ffc20 0x7feb00100080 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7feaf4009910 tx=0x7feaf4009c20 comp rx=0 tx=0).stop 2024-04-22T23:37:30.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.142+0000 7feb07c9c700 1 -- 172.21.15.67:0/3091412998 shutdown_connections 2024-04-22T23:37:30.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.142+0000 7feb07c9c700 1 --2- 172.21.15.67:0/3091412998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb000ffc20 0x7feb00100080 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:30.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.142+0000 7feb07c9c700 1 --2- 172.21.15.67:0/3091412998 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb00105bd0 0x7feb00105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:30.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.142+0000 7feb07c9c700 1 -- 172.21.15.67:0/3091412998 >> 172.21.15.67:0/3091412998 conn(0x7feb00075670 msgr2=0x7feb00075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:37:30.137 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:29 smithi067 ceph-mon[112438]: pgmap v333: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:30.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.142+0000 7feb07c9c700 1 -- 172.21.15.67:0/3091412998 shutdown_connections 2024-04-22T23:37:30.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.142+0000 7feb07c9c700 1 -- 172.21.15.67:0/3091412998 wait complete. 2024-04-22T23:37:30.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.143+0000 7feb07c9c700 1 Processor -- start 2024-04-22T23:37:30.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.143+0000 7feb07c9c700 1 -- start start 2024-04-22T23:37:30.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.143+0000 7feb07c9c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb000ffc20 0x7feb0019c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:37:30.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.143+0000 7feb07c9c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb00105bd0 0x7feb0019cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:37:30.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.143+0000 7feb07c9c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb0019d200 con 0x7feb000ffc20 2024-04-22T23:37:30.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.143+0000 7feb07c9c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb001966b0 con 0x7feb00105bd0 2024-04-22T23:37:30.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.143+0000 7feb05a38700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb000ffc20 0x7feb0019c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:37:30.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.143+0000 7feb05a38700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb000ffc20 0x7feb0019c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51758/0 (socket says 172.21.15.67:51758) 2024-04-22T23:37:30.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.143+0000 7feb05a38700 1 -- 172.21.15.67:0/744878408 learned_addr learned my addr 172.21.15.67:0/744878408 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:37:30.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.144+0000 7feb05237700 1 --2- 172.21.15.67:0/744878408 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb00105bd0 0x7feb0019cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:37:30.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.144+0000 7feb05a38700 1 -- 172.21.15.67:0/744878408 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb00105bd0 msgr2=0x7feb0019cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:37:30.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.144+0000 7feb05a38700 1 --2- 172.21.15.67:0/744878408 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb00105bd0 0x7feb0019cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:30.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.144+0000 7feb05a38700 1 -- 172.21.15.67:0/744878408 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf40095f0 con 0x7feb000ffc20 2024-04-22T23:37:30.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.144+0000 7feb05a38700 1 --2- 172.21.15.67:0/744878408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb000ffc20 0x7feb0019c630 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7feaf800ada0 tx=0x7feaf800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:37:30.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.144+0000 7feafeffd700 1 -- 172.21.15.67:0/744878408 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feaf800f070 con 0x7feb000ffc20 2024-04-22T23:37:30.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.144+0000 7feafeffd700 1 -- 172.21.15.67:0/744878408 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feaf8007bf0 con 0x7feb000ffc20 2024-04-22T23:37:30.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.144+0000 7feb07c9c700 1 -- 172.21.15.67:0/744878408 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb00196990 con 0x7feb000ffc20 2024-04-22T23:37:30.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.144+0000 7feafeffd700 1 -- 172.21.15.67:0/744878408 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feaf8005690 con 0x7feb000ffc20 2024-04-22T23:37:30.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.144+0000 7feb07c9c700 1 -- 172.21.15.67:0/744878408 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb00196ee0 con 0x7feb000ffc20 2024-04-22T23:37:30.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.145+0000 7feb07c9c700 1 -- 172.21.15.67:0/744878408 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb0004f2e0 con 0x7feb000ffc20 2024-04-22T23:37:30.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.146+0000 7feafeffd700 1 -- 172.21.15.67:0/744878408 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7feaf8005850 con 0x7feb000ffc20 2024-04-22T23:37:30.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.146+0000 7feafeffd700 1 --2- 172.21.15.67:0/744878408 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7feae806ecc0 0x7feae8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:37:30.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.146+0000 7feb05237700 1 --2- 172.21.15.67:0/744878408 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7feae806ecc0 0x7feae8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:37:30.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.146+0000 7feafeffd700 1 -- 172.21.15.67:0/744878408 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7feaf808e930 con 0x7feb000ffc20 2024-04-22T23:37:30.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.147+0000 7feb05237700 1 --2- 172.21.15.67:0/744878408 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7feae806ecc0 0x7feae8071180 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7feaf40059d0 tx=0x7feaf401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:37:30.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.151+0000 7feafeffd700 1 -- 172.21.15.67:0/744878408 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7feaf805b120 con 0x7feb000ffc20 2024-04-22T23:37:30.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:29 smithi134 ceph-mon[85630]: pgmap v333: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:30.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.383+0000 7feb07c9c700 1 -- 172.21.15.67:0/744878408 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7feb001971f0 con 0x7feae806ecc0 2024-04-22T23:37:30.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.384+0000 7feafeffd700 1 -- 172.21.15.67:0/744878408 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7feb001971f0 con 0x7feae806ecc0 2024-04-22T23:37:30.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.386+0000 7feb07c9c700 1 -- 172.21.15.67:0/744878408 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7feae806ecc0 msgr2=0x7feae8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:37:30.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.386+0000 7feb07c9c700 1 --2- 172.21.15.67:0/744878408 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7feae806ecc0 0x7feae8071180 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7feaf40059d0 tx=0x7feaf401a040 comp rx=0 tx=0).stop 2024-04-22T23:37:30.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.386+0000 7feb07c9c700 1 -- 172.21.15.67:0/744878408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb000ffc20 msgr2=0x7feb0019c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:37:30.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.386+0000 7feb07c9c700 1 --2- 172.21.15.67:0/744878408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb000ffc20 0x7feb0019c630 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7feaf800ada0 tx=0x7feaf800c330 comp rx=0 tx=0).stop 2024-04-22T23:37:30.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.386+0000 7feb07c9c700 1 -- 172.21.15.67:0/744878408 shutdown_connections 2024-04-22T23:37:30.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.386+0000 7feb07c9c700 1 --2- 172.21.15.67:0/744878408 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7feae806ecc0 0x7feae8071180 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:30.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.386+0000 7feb07c9c700 1 --2- 172.21.15.67:0/744878408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb000ffc20 0x7feb0019c630 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:30.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.386+0000 7feb07c9c700 1 --2- 172.21.15.67:0/744878408 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb00105bd0 0x7feb0019cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:30.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.386+0000 7feb07c9c700 1 -- 172.21.15.67:0/744878408 >> 172.21.15.67:0/744878408 conn(0x7feb00075670 msgr2=0x7feb00068680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:37:30.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.386+0000 7feb07c9c700 1 -- 172.21.15.67:0/744878408 shutdown_connections 2024-04-22T23:37:30.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.386+0000 7feb07c9c700 1 -- 172.21.15.67:0/744878408 wait complete. 2024-04-22T23:37:30.390 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:37:30.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.494+0000 7f3bbf91a700 1 -- 172.21.15.67:0/3160429824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bb81006b0 msgr2=0x7f3bb8100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:37:30.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.494+0000 7f3bbf91a700 1 --2- 172.21.15.67:0/3160429824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bb81006b0 0x7f3bb8100a90 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f3bb4009910 tx=0x7f3bb4009c20 comp rx=0 tx=0).stop 2024-04-22T23:37:30.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.494+0000 7f3bbf91a700 1 -- 172.21.15.67:0/3160429824 shutdown_connections 2024-04-22T23:37:30.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.494+0000 7f3bbf91a700 1 --2- 172.21.15.67:0/3160429824 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3bb8101060 0x7f3bb81047b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:30.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.494+0000 7f3bbf91a700 1 --2- 172.21.15.67:0/3160429824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bb81006b0 0x7f3bb8100a90 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:30.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.494+0000 7f3bbf91a700 1 -- 172.21.15.67:0/3160429824 >> 172.21.15.67:0/3160429824 conn(0x7f3bb8074e30 msgr2=0x7f3bb8075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:37:30.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.494+0000 7f3bbf91a700 1 -- 172.21.15.67:0/3160429824 shutdown_connections 2024-04-22T23:37:30.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.494+0000 7f3bbf91a700 1 -- 172.21.15.67:0/3160429824 wait complete. 2024-04-22T23:37:30.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.495+0000 7f3bbf91a700 1 Processor -- start 2024-04-22T23:37:30.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.495+0000 7f3bbf91a700 1 -- start start 2024-04-22T23:37:30.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.495+0000 7f3bbf91a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bb81006b0 0x7f3bb81976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:37:30.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.495+0000 7f3bbf91a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3bb8101060 0x7f3bb8197c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:37:30.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.495+0000 7f3bbf91a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb81981a0 con 0x7f3bb81006b0 2024-04-22T23:37:30.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.495+0000 7f3bbf91a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb81982e0 con 0x7f3bb8101060 2024-04-22T23:37:30.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.495+0000 7f3bbd6b6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bb81006b0 0x7f3bb81976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:37:30.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.495+0000 7f3bbd6b6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bb81006b0 0x7f3bb81976d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51782/0 (socket says 172.21.15.67:51782) 2024-04-22T23:37:30.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.495+0000 7f3bbd6b6700 1 -- 172.21.15.67:0/3993970936 learned_addr learned my addr 172.21.15.67:0/3993970936 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:37:30.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.496+0000 7f3bbceb5700 1 --2- 172.21.15.67:0/3993970936 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3bb8101060 0x7f3bb8197c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:37:30.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.496+0000 7f3bbd6b6700 1 -- 172.21.15.67:0/3993970936 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3bb8101060 msgr2=0x7f3bb8197c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:37:30.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.496+0000 7f3bbd6b6700 1 --2- 172.21.15.67:0/3993970936 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3bb8101060 0x7f3bb8197c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:30.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.496+0000 7f3bbd6b6700 1 -- 172.21.15.67:0/3993970936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb40095f0 con 0x7f3bb81006b0 2024-04-22T23:37:30.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.496+0000 7f3bbd6b6700 1 --2- 172.21.15.67:0/3993970936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bb81006b0 0x7f3bb81976d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3bb4006010 tx=0x7f3bb400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:37:30.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.496+0000 7f3bae7fc700 1 -- 172.21.15.67:0/3993970936 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3bb401d070 con 0x7f3bb81006b0 2024-04-22T23:37:30.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.496+0000 7f3bbf91a700 1 -- 172.21.15.67:0/3993970936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bb81a0740 con 0x7f3bb81006b0 2024-04-22T23:37:30.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.496+0000 7f3bae7fc700 1 -- 172.21.15.67:0/3993970936 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3bb4004fb0 con 0x7f3bb81006b0 2024-04-22T23:37:30.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.497+0000 7f3bae7fc700 1 -- 172.21.15.67:0/3993970936 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3bb4022ea0 con 0x7f3bb81006b0 2024-04-22T23:37:30.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.497+0000 7f3bbf91a700 1 -- 172.21.15.67:0/3993970936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bb81a0af0 con 0x7f3bb81006b0 2024-04-22T23:37:30.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.498+0000 7f3bbf91a700 1 -- 172.21.15.67:0/3993970936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bb8108120 con 0x7f3bb81006b0 2024-04-22T23:37:30.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.498+0000 7f3bae7fc700 1 -- 172.21.15.67:0/3993970936 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3bb4022470 con 0x7f3bb81006b0 2024-04-22T23:37:30.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.499+0000 7f3bae7fc700 1 --2- 172.21.15.67:0/3993970936 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3ba406ecc0 0x7f3ba4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:37:30.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.499+0000 7f3bae7fc700 1 -- 172.21.15.67:0/3993970936 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f3bb4090520 con 0x7f3bb81006b0 2024-04-22T23:37:30.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.499+0000 7f3bbceb5700 1 --2- 172.21.15.67:0/3993970936 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3ba406ecc0 0x7f3ba4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:37:30.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.499+0000 7f3bbceb5700 1 --2- 172.21.15.67:0/3993970936 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3ba406ecc0 0x7f3ba4071180 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3ba8005ea0 tx=0x7f3ba8005e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:37:30.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.503+0000 7f3bae7fc700 1 -- 172.21.15.67:0/3993970936 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f3bb4059210 con 0x7f3bb81006b0 2024-04-22T23:37:30.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.733+0000 7f3bbf91a700 1 -- 172.21.15.67:0/3993970936 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3bb80611d0 con 0x7f3ba406ecc0 2024-04-22T23:37:30.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.735+0000 7f3bae7fc700 1 -- 172.21.15.67:0/3993970936 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f3bb80611d0 con 0x7f3ba406ecc0 2024-04-22T23:37:30.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.737+0000 7f3bbf91a700 1 -- 172.21.15.67:0/3993970936 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3ba406ecc0 msgr2=0x7f3ba4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:37:30.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.737+0000 7f3bbf91a700 1 --2- 172.21.15.67:0/3993970936 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3ba406ecc0 0x7f3ba4071180 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3ba8005ea0 tx=0x7f3ba8005e30 comp rx=0 tx=0).stop 2024-04-22T23:37:30.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.737+0000 7f3bbf91a700 1 -- 172.21.15.67:0/3993970936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bb81006b0 msgr2=0x7f3bb81976d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:37:30.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.737+0000 7f3bbf91a700 1 --2- 172.21.15.67:0/3993970936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bb81006b0 0x7f3bb81976d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3bb4006010 tx=0x7f3bb400be20 comp rx=0 tx=0).stop 2024-04-22T23:37:30.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.737+0000 7f3bbf91a700 1 -- 172.21.15.67:0/3993970936 shutdown_connections 2024-04-22T23:37:30.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.737+0000 7f3bbf91a700 1 --2- 172.21.15.67:0/3993970936 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3ba406ecc0 0x7f3ba4071180 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:30.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.737+0000 7f3bbf91a700 1 --2- 172.21.15.67:0/3993970936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bb81006b0 0x7f3bb81976d0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:30.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.737+0000 7f3bbf91a700 1 --2- 172.21.15.67:0/3993970936 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3bb8101060 0x7f3bb8197c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:30.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.737+0000 7f3bbf91a700 1 -- 172.21.15.67:0/3993970936 >> 172.21.15.67:0/3993970936 conn(0x7f3bb8074e30 msgr2=0x7f3bb80fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:37:30.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.737+0000 7f3bbf91a700 1 -- 172.21.15.67:0/3993970936 shutdown_connections 2024-04-22T23:37:30.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.737+0000 7f3bbf91a700 1 -- 172.21.15.67:0/3993970936 wait complete. 2024-04-22T23:37:30.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.844+0000 7f314ee7a700 1 -- 172.21.15.67:0/2179799207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3148100e90 msgr2=0x7f3148101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:37:30.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.844+0000 7f314ee7a700 1 --2- 172.21.15.67:0/2179799207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3148100e90 0x7f3148101270 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f313c009910 tx=0x7f313c009c20 comp rx=0 tx=0).stop 2024-04-22T23:37:30.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.844+0000 7f314ee7a700 1 -- 172.21.15.67:0/2179799207 shutdown_connections 2024-04-22T23:37:30.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.844+0000 7f314ee7a700 1 --2- 172.21.15.67:0/2179799207 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3148101840 0x7f3148105890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:30.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.844+0000 7f314ee7a700 1 --2- 172.21.15.67:0/2179799207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3148100e90 0x7f3148101270 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:30.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.844+0000 7f314ee7a700 1 -- 172.21.15.67:0/2179799207 >> 172.21.15.67:0/2179799207 conn(0x7f31480fc740 msgr2=0x7f31480feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:37:30.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.844+0000 7f314ee7a700 1 -- 172.21.15.67:0/2179799207 shutdown_connections 2024-04-22T23:37:30.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.844+0000 7f314ee7a700 1 -- 172.21.15.67:0/2179799207 wait complete. 2024-04-22T23:37:30.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.845+0000 7f314ee7a700 1 Processor -- start 2024-04-22T23:37:30.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.845+0000 7f314ee7a700 1 -- start start 2024-04-22T23:37:30.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.845+0000 7f314ee7a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3148100e90 0x7f314819c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:37:30.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.845+0000 7f314ee7a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3148101840 0x7f314819cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:37:30.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.845+0000 7f314ee7a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f314819d250 con 0x7f3148100e90 2024-04-22T23:37:30.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.845+0000 7f314ee7a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3148196700 con 0x7f3148101840 2024-04-22T23:37:30.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.845+0000 7f314cc16700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3148100e90 0x7f314819c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:37:30.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.845+0000 7f314cc16700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3148100e90 0x7f314819c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51806/0 (socket says 172.21.15.67:51806) 2024-04-22T23:37:30.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.845+0000 7f314cc16700 1 -- 172.21.15.67:0/3895652131 learned_addr learned my addr 172.21.15.67:0/3895652131 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:37:30.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.846+0000 7f3147fff700 1 --2- 172.21.15.67:0/3895652131 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3148101840 0x7f314819cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:37:30.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.846+0000 7f314cc16700 1 -- 172.21.15.67:0/3895652131 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3148101840 msgr2=0x7f314819cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:37:30.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.846+0000 7f314cc16700 1 --2- 172.21.15.67:0/3895652131 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3148101840 0x7f314819cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:30.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.846+0000 7f314cc16700 1 -- 172.21.15.67:0/3895652131 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f313c0095f0 con 0x7f3148100e90 2024-04-22T23:37:30.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.846+0000 7f314cc16700 1 --2- 172.21.15.67:0/3895652131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3148100e90 0x7f314819c680 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f313c006010 tx=0x7f313c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:37:30.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.846+0000 7f3145ffb700 1 -- 172.21.15.67:0/3895652131 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f313c01d070 con 0x7f3148100e90 2024-04-22T23:37:30.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.846+0000 7f314ee7a700 1 -- 172.21.15.67:0/3895652131 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3148196980 con 0x7f3148100e90 2024-04-22T23:37:30.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.846+0000 7f3145ffb700 1 -- 172.21.15.67:0/3895652131 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f313c004fb0 con 0x7f3148100e90 2024-04-22T23:37:30.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.846+0000 7f3145ffb700 1 -- 172.21.15.67:0/3895652131 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f313c022ea0 con 0x7f3148100e90 2024-04-22T23:37:30.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.846+0000 7f314ee7a700 1 -- 172.21.15.67:0/3895652131 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3148196e70 con 0x7f3148100e90 2024-04-22T23:37:30.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.847+0000 7f314ee7a700 1 -- 172.21.15.67:0/3895652131 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3148109230 con 0x7f3148100e90 2024-04-22T23:37:30.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.848+0000 7f3145ffb700 1 -- 172.21.15.67:0/3895652131 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f313c022470 con 0x7f3148100e90 2024-04-22T23:37:30.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.848+0000 7f3145ffb700 1 --2- 172.21.15.67:0/3895652131 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f313806ecc0 0x7f3138071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:37:30.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.848+0000 7f3145ffb700 1 -- 172.21.15.67:0/3895652131 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f313c090570 con 0x7f3148100e90 2024-04-22T23:37:30.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.848+0000 7f3147fff700 1 --2- 172.21.15.67:0/3895652131 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f313806ecc0 0x7f3138071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:37:30.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.849+0000 7f3147fff700 1 --2- 172.21.15.67:0/3895652131 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f313806ecc0 0x7f3138071180 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3134005fd0 tx=0x7f313400a560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:37:30.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:30.853+0000 7f3145ffb700 1 -- 172.21.15.67:0/3895652131 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f313c05cd90 con 0x7f3148100e90 2024-04-22T23:37:31.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.075+0000 7f314ee7a700 1 -- 172.21.15.67:0/3895652131 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3148197090 con 0x7f313806ecc0 2024-04-22T23:37:31.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.086+0000 7f3145ffb700 1 -- 172.21.15.67:0/3895652131 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f3148197090 con 0x7f313806ecc0 2024-04-22T23:37:31.080 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:37:31.080 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (11m) 4m ago 22m 25.5M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:37:31.080 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (22m) 4m ago 22m 9114k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:37:31.080 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (21m) 4m ago 21m 9042k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:37:31.080 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (8m) 4m ago 22m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:37:31.080 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (8m) 4m ago 21m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:37:31.081 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (10m) 4m ago 22m 79.3M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:37:31.081 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (4m) 4m ago 17m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:37:31.081 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (4m) 4m ago 16m 12.6M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:37:31.081 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (4m) 4m ago 17m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:37:31.081 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (16m) 4m ago 16m 12.2M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:37:31.081 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (12m) 4m ago 23m 511M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:37:31.081 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (12m) 4m ago 21m 453M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:37:31.081 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (9m) 4m ago 24m 55.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:37:31.081 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (9m) 4m ago 21m 46.0M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:37:31.081 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (11m) 4m ago 22m 15.4M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:37:31.081 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (11m) 4m ago 21m 16.1M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:37:31.081 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (8m) 4m ago 20m 99.2M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:37:31.081 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (7m) 4m ago 20m 127M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:37:31.082 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (6m) 4m ago 19m 79.9M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:37:31.082 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (6m) 4m ago 19m 97.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:37:31.082 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (5m) 4m ago 19m 82.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:37:31.082 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (5m) 4m ago 18m 77.6M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:37:31.082 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (11m) 4m ago 22m 73.9M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:37:31.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.088+0000 7f314ee7a700 1 -- 172.21.15.67:0/3895652131 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f313806ecc0 msgr2=0x7f3138071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:37:31.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.088+0000 7f314ee7a700 1 --2- 172.21.15.67:0/3895652131 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f313806ecc0 0x7f3138071180 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3134005fd0 tx=0x7f313400a560 comp rx=0 tx=0).stop 2024-04-22T23:37:31.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.088+0000 7f314ee7a700 1 -- 172.21.15.67:0/3895652131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3148100e90 msgr2=0x7f314819c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:37:31.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.088+0000 7f314ee7a700 1 --2- 172.21.15.67:0/3895652131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3148100e90 0x7f314819c680 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f313c006010 tx=0x7f313c00be20 comp rx=0 tx=0).stop 2024-04-22T23:37:31.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.088+0000 7f314ee7a700 1 -- 172.21.15.67:0/3895652131 shutdown_connections 2024-04-22T23:37:31.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.088+0000 7f314ee7a700 1 --2- 172.21.15.67:0/3895652131 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f313806ecc0 0x7f3138071180 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:31.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.088+0000 7f314ee7a700 1 --2- 172.21.15.67:0/3895652131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3148100e90 0x7f314819c680 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:31.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.088+0000 7f314ee7a700 1 --2- 172.21.15.67:0/3895652131 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3148101840 0x7f314819cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:31.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.088+0000 7f314ee7a700 1 -- 172.21.15.67:0/3895652131 >> 172.21.15.67:0/3895652131 conn(0x7f31480fc740 msgr2=0x7f31480fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:37:31.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.088+0000 7f314ee7a700 1 -- 172.21.15.67:0/3895652131 shutdown_connections 2024-04-22T23:37:31.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.088+0000 7f314ee7a700 1 -- 172.21.15.67:0/3895652131 wait complete. 2024-04-22T23:37:31.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.194+0000 7f27f9aee700 1 -- 172.21.15.67:0/192420223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27f4106cd0 msgr2=0x7f27f41070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:37:31.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.194+0000 7f27f9aee700 1 --2- 172.21.15.67:0/192420223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27f4106cd0 0x7f27f41070b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f27e4009910 tx=0x7f27e4009c20 comp rx=0 tx=0).stop 2024-04-22T23:37:31.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.194+0000 7f27f9aee700 1 -- 172.21.15.67:0/192420223 shutdown_connections 2024-04-22T23:37:31.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.194+0000 7f27f9aee700 1 --2- 172.21.15.67:0/192420223 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f27f4068580 0x7f27f40689e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:31.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.194+0000 7f27f9aee700 1 --2- 172.21.15.67:0/192420223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27f4106cd0 0x7f27f41070b0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:31.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.194+0000 7f27f9aee700 1 -- 172.21.15.67:0/192420223 >> 172.21.15.67:0/192420223 conn(0x7f27f4075670 msgr2=0x7f27f4075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:37:31.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.194+0000 7f27f9aee700 1 -- 172.21.15.67:0/192420223 shutdown_connections 2024-04-22T23:37:31.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.194+0000 7f27f9aee700 1 -- 172.21.15.67:0/192420223 wait complete. 2024-04-22T23:37:31.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.194+0000 7f27f9aee700 1 Processor -- start 2024-04-22T23:37:31.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.195+0000 7f27f9aee700 1 -- start start 2024-04-22T23:37:31.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.195+0000 7f27f9aee700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27f4068580 0x7f27f40ffcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:37:31.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.195+0000 7f27f9aee700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f27f4106cd0 0x7f27f4100230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:37:31.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.195+0000 7f27f9aee700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f4100770 con 0x7f27f4068580 2024-04-22T23:37:31.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.195+0000 7f27f9aee700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f41008b0 con 0x7f27f4106cd0 2024-04-22T23:37:31.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.195+0000 7f27f37fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27f4068580 0x7f27f40ffcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:37:31.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.195+0000 7f27f37fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27f4068580 0x7f27f40ffcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51820/0 (socket says 172.21.15.67:51820) 2024-04-22T23:37:31.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.195+0000 7f27f37fe700 1 -- 172.21.15.67:0/183705426 learned_addr learned my addr 172.21.15.67:0/183705426 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:37:31.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.195+0000 7f27f2ffd700 1 --2- 172.21.15.67:0/183705426 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f27f4106cd0 0x7f27f4100230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:37:31.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.195+0000 7f27f37fe700 1 -- 172.21.15.67:0/183705426 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f27f4106cd0 msgr2=0x7f27f4100230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:37:31.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.195+0000 7f27f37fe700 1 --2- 172.21.15.67:0/183705426 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f27f4106cd0 0x7f27f4100230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:31.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.195+0000 7f27f37fe700 1 -- 172.21.15.67:0/183705426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e40095f0 con 0x7f27f4068580 2024-04-22T23:37:31.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.195+0000 7f27f2ffd700 1 --2- 172.21.15.67:0/183705426 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f27f4106cd0 0x7f27f4100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:37:31.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.195+0000 7f27f37fe700 1 --2- 172.21.15.67:0/183705426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27f4068580 0x7f27f40ffcf0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f27e4005f50 tx=0x7f27e400f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:37:31.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.196+0000 7f27f0ff9700 1 -- 172.21.15.67:0/183705426 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27e401d070 con 0x7f27f4068580 2024-04-22T23:37:31.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.196+0000 7f27f0ff9700 1 -- 172.21.15.67:0/183705426 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f27e400b610 con 0x7f27f4068580 2024-04-22T23:37:31.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.196+0000 7f27f9aee700 1 -- 172.21.15.67:0/183705426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27f4100b30 con 0x7f27f4068580 2024-04-22T23:37:31.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.196+0000 7f27f9aee700 1 -- 172.21.15.67:0/183705426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27f41a2d90 con 0x7f27f4068580 2024-04-22T23:37:31.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.196+0000 7f27f0ff9700 1 -- 172.21.15.67:0/183705426 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27e4022470 con 0x7f27f4068580 2024-04-22T23:37:31.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.197+0000 7f27f9aee700 1 -- 172.21.15.67:0/183705426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27f41091f0 con 0x7f27f4068580 2024-04-22T23:37:31.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.197+0000 7f27f0ff9700 1 -- 172.21.15.67:0/183705426 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f27e400fca0 con 0x7f27f4068580 2024-04-22T23:37:31.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.198+0000 7f27f0ff9700 1 --2- 172.21.15.67:0/183705426 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f27dc06ecc0 0x7f27dc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:37:31.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.198+0000 7f27f2ffd700 1 --2- 172.21.15.67:0/183705426 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f27dc06ecc0 0x7f27dc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:37:31.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.198+0000 7f27f0ff9700 1 -- 172.21.15.67:0/183705426 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f27e4090360 con 0x7f27f4068580 2024-04-22T23:37:31.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.198+0000 7f27f2ffd700 1 --2- 172.21.15.67:0/183705426 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f27dc06ecc0 0x7f27dc071180 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f27f41014e0 tx=0x7f27e8009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:37:31.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.203+0000 7f27f0ff9700 1 -- 172.21.15.67:0/183705426 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f27e4059020 con 0x7f27f4068580 2024-04-22T23:37:31.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.499+0000 7f27f9aee700 1 -- 172.21.15.67:0/183705426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f27f404f2e0 con 0x7f27f4068580 2024-04-22T23:37:31.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.499+0000 7f27f0ff9700 1 -- 172.21.15.67:0/183705426 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f27e405c6e0 con 0x7f27f4068580 2024-04-22T23:37:31.493 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:37:31.493 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:37:31.494 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:37:31.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:37:31.494 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:37:31.494 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:37:31.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:37:31.494 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:37:31.494 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:37:31.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:37:31.494 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:37:31.494 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:37:31.494 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 3 2024-04-22T23:37:31.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:37:31.494 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:37:31.494 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:37:31.495 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 13 2024-04-22T23:37:31.495 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:37:31.495 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:37:31.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.501+0000 7f27f9aee700 1 -- 172.21.15.67:0/183705426 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f27dc06ecc0 msgr2=0x7f27dc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:37:31.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.501+0000 7f27f9aee700 1 --2- 172.21.15.67:0/183705426 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f27dc06ecc0 0x7f27dc071180 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f27f41014e0 tx=0x7f27e8009260 comp rx=0 tx=0).stop 2024-04-22T23:37:31.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.501+0000 7f27f9aee700 1 -- 172.21.15.67:0/183705426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27f4068580 msgr2=0x7f27f40ffcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:37:31.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.501+0000 7f27f9aee700 1 --2- 172.21.15.67:0/183705426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27f4068580 0x7f27f40ffcf0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f27e4005f50 tx=0x7f27e400f740 comp rx=0 tx=0).stop 2024-04-22T23:37:31.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.502+0000 7f27f9aee700 1 -- 172.21.15.67:0/183705426 shutdown_connections 2024-04-22T23:37:31.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.502+0000 7f27f9aee700 1 --2- 172.21.15.67:0/183705426 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f27dc06ecc0 0x7f27dc071180 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:31.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.502+0000 7f27f9aee700 1 --2- 172.21.15.67:0/183705426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27f4068580 0x7f27f40ffcf0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:31.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.502+0000 7f27f9aee700 1 --2- 172.21.15.67:0/183705426 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f27f4106cd0 0x7f27f4100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:31.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.502+0000 7f27f9aee700 1 -- 172.21.15.67:0/183705426 >> 172.21.15.67:0/183705426 conn(0x7f27f4075670 msgr2=0x7f27f40fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:37:31.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.502+0000 7f27f9aee700 1 -- 172.21.15.67:0/183705426 shutdown_connections 2024-04-22T23:37:31.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.502+0000 7f27f9aee700 1 -- 172.21.15.67:0/183705426 wait complete. 2024-04-22T23:37:31.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.609+0000 7f8ea4b12700 1 -- 172.21.15.67:0/3433675070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ea0101850 msgr2=0x7f8ea01058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:37:31.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.609+0000 7f8ea4b12700 1 --2- 172.21.15.67:0/3433675070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ea0101850 0x7f8ea01058a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8e94009910 tx=0x7f8e94009c20 comp rx=0 tx=0).stop 2024-04-22T23:37:31.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.610+0000 7f8ea4b12700 1 -- 172.21.15.67:0/3433675070 shutdown_connections 2024-04-22T23:37:31.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.610+0000 7f8ea4b12700 1 --2- 172.21.15.67:0/3433675070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ea0101850 0x7f8ea01058a0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:31.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.610+0000 7f8ea4b12700 1 --2- 172.21.15.67:0/3433675070 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8ea0100ea0 0x7f8ea0101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:31.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.610+0000 7f8ea4b12700 1 -- 172.21.15.67:0/3433675070 >> 172.21.15.67:0/3433675070 conn(0x7f8ea00fc750 msgr2=0x7f8ea00feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:37:31.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.610+0000 7f8ea4b12700 1 -- 172.21.15.67:0/3433675070 shutdown_connections 2024-04-22T23:37:31.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.610+0000 7f8ea4b12700 1 -- 172.21.15.67:0/3433675070 wait complete. 2024-04-22T23:37:31.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.610+0000 7f8ea4b12700 1 Processor -- start 2024-04-22T23:37:31.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.610+0000 7f8ea4b12700 1 -- start start 2024-04-22T23:37:31.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.611+0000 7f8ea4b12700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ea0100ea0 0x7f8ea019c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:37:31.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.611+0000 7f8ea4b12700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8ea0101850 0x7f8ea019cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:37:31.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.611+0000 7f8ea4b12700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea019d2d0 con 0x7f8ea0100ea0 2024-04-22T23:37:31.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.611+0000 7f8ea4b12700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea0196780 con 0x7f8ea0101850 2024-04-22T23:37:31.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.611+0000 7f8e9e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ea0100ea0 0x7f8ea019c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:37:31.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.611+0000 7f8e9e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ea0100ea0 0x7f8ea019c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51836/0 (socket says 172.21.15.67:51836) 2024-04-22T23:37:31.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.611+0000 7f8e9e59c700 1 -- 172.21.15.67:0/4141988 learned_addr learned my addr 172.21.15.67:0/4141988 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:37:31.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.611+0000 7f8e9dd9b700 1 --2- 172.21.15.67:0/4141988 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8ea0101850 0x7f8ea019cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:37:31.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.611+0000 7f8e9e59c700 1 -- 172.21.15.67:0/4141988 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8ea0101850 msgr2=0x7f8ea019cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:37:31.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.611+0000 7f8e9e59c700 1 --2- 172.21.15.67:0/4141988 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8ea0101850 0x7f8ea019cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:31.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.611+0000 7f8e9e59c700 1 -- 172.21.15.67:0/4141988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e940095f0 con 0x7f8ea0100ea0 2024-04-22T23:37:31.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.611+0000 7f8e9e59c700 1 --2- 172.21.15.67:0/4141988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ea0100ea0 0x7f8ea019c700 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f8e9000d900 tx=0x7f8e9000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:37:31.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.611+0000 7f8e8f7fe700 1 -- 172.21.15.67:0/4141988 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e900041f0 con 0x7f8ea0100ea0 2024-04-22T23:37:31.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.612+0000 7f8ea4b12700 1 -- 172.21.15.67:0/4141988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ea0196a60 con 0x7f8ea0100ea0 2024-04-22T23:37:31.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.612+0000 7f8e8f7fe700 1 -- 172.21.15.67:0/4141988 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8e90004cf0 con 0x7f8ea0100ea0 2024-04-22T23:37:31.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.612+0000 7f8e8f7fe700 1 -- 172.21.15.67:0/4141988 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e9000b750 con 0x7f8ea0100ea0 2024-04-22T23:37:31.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.612+0000 7f8ea4b12700 1 -- 172.21.15.67:0/4141988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ea0196fb0 con 0x7f8ea0100ea0 2024-04-22T23:37:31.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.613+0000 7f8ea4b12700 1 -- 172.21.15.67:0/4141988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ea01092a0 con 0x7f8ea0100ea0 2024-04-22T23:37:31.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.613+0000 7f8e8f7fe700 1 -- 172.21.15.67:0/4141988 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8e90004750 con 0x7f8ea0100ea0 2024-04-22T23:37:31.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.613+0000 7f8e8f7fe700 1 --2- 172.21.15.67:0/4141988 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8e8806ecc0 0x7f8e88071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:37:31.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.614+0000 7f8e8f7fe700 1 -- 172.21.15.67:0/4141988 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f8e9008edd0 con 0x7f8ea0100ea0 2024-04-22T23:37:31.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.614+0000 7f8e9dd9b700 1 --2- 172.21.15.67:0/4141988 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8e8806ecc0 0x7f8e88071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:37:31.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.614+0000 7f8e9dd9b700 1 --2- 172.21.15.67:0/4141988 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8e8806ecc0 0x7f8e88071180 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0197bf0 tx=0x7f8e9401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:37:31.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.618+0000 7f8e8f7fe700 1 -- 172.21.15.67:0/4141988 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f8e9005b5c0 con 0x7f8ea0100ea0 2024-04-22T23:37:31.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.873+0000 7f8ea4b12700 1 -- 172.21.15.67:0/4141988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f8ea0197d50 con 0x7f8ea0100ea0 2024-04-22T23:37:31.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.873+0000 7f8e8f7fe700 1 -- 172.21.15.67:0/4141988 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1871 (secure 0 0 0) 0x7f8e90016070 con 0x7f8ea0100ea0 2024-04-22T23:37:31.868 INFO:teuthology.orchestra.run.smithi067.stdout:e31 2024-04-22T23:37:31.868 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:37:31.868 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:37:31.868 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:37:31.868 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:37:31.868 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:37:31.868 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:37:31.868 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:37:31.868 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:37:31.868 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:37:31.868 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:37:31.868 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:37:31.868 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:37:31.869 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:37:31.869 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:37:31.869 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:37:31.869 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:37:31.869 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:37:31.869 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:37:31.869 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:37:31.869 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:37:31.869 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:37:31.869 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:37:31.869 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:37:31.869 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:37:31.869 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:37:31.869 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:37:31.869 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:37:31.869 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:37:31.869 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:37:31.870 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:37:31.870 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:37:31.870 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:37:31.870 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:37:31.870 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:37:31.870 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:37:31.870 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:37:31.870 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:37:31.870 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:37:31.870 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:37:31.870 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44259} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1827288036,v1:172.21.15.134:6827/1827288036] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:37:31.870 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:37:31.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.876+0000 7f8ea4b12700 1 -- 172.21.15.67:0/4141988 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8e8806ecc0 msgr2=0x7f8e88071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:37:31.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.876+0000 7f8ea4b12700 1 --2- 172.21.15.67:0/4141988 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8e8806ecc0 0x7f8e88071180 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0197bf0 tx=0x7f8e9401a040 comp rx=0 tx=0).stop 2024-04-22T23:37:31.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.876+0000 7f8ea4b12700 1 -- 172.21.15.67:0/4141988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ea0100ea0 msgr2=0x7f8ea019c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:37:31.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.876+0000 7f8ea4b12700 1 --2- 172.21.15.67:0/4141988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ea0100ea0 0x7f8ea019c700 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f8e9000d900 tx=0x7f8e9000dc10 comp rx=0 tx=0).stop 2024-04-22T23:37:31.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.877+0000 7f8ea4b12700 1 -- 172.21.15.67:0/4141988 shutdown_connections 2024-04-22T23:37:31.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.877+0000 7f8ea4b12700 1 --2- 172.21.15.67:0/4141988 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8e8806ecc0 0x7f8e88071180 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:31.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.877+0000 7f8ea4b12700 1 --2- 172.21.15.67:0/4141988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ea0100ea0 0x7f8ea019c700 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:31.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.877+0000 7f8ea4b12700 1 --2- 172.21.15.67:0/4141988 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8ea0101850 0x7f8ea019cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:31.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.877+0000 7f8ea4b12700 1 -- 172.21.15.67:0/4141988 >> 172.21.15.67:0/4141988 conn(0x7f8ea00fc750 msgr2=0x7f8ea00fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:37:31.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.877+0000 7f8ea4b12700 1 -- 172.21.15.67:0/4141988 shutdown_connections 2024-04-22T23:37:31.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.877+0000 7f8ea4b12700 1 -- 172.21.15.67:0/4141988 wait complete. 2024-04-22T23:37:31.872 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 31 2024-04-22T23:37:31.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.985+0000 7f3b72a23700 1 -- 172.21.15.67:0/2778543195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b6c068580 msgr2=0x7f3b6c0689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:37:31.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.985+0000 7f3b72a23700 1 --2- 172.21.15.67:0/2778543195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b6c068580 0x7f3b6c0689e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3b60009910 tx=0x7f3b60009c20 comp rx=0 tx=0).stop 2024-04-22T23:37:31.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.985+0000 7f3b72a23700 1 -- 172.21.15.67:0/2778543195 shutdown_connections 2024-04-22T23:37:31.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.985+0000 7f3b72a23700 1 --2- 172.21.15.67:0/2778543195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b6c068580 0x7f3b6c0689e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:31.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.985+0000 7f3b72a23700 1 --2- 172.21.15.67:0/2778543195 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3b6c106cd0 0x7f3b6c1070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:31.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.985+0000 7f3b72a23700 1 -- 172.21.15.67:0/2778543195 >> 172.21.15.67:0/2778543195 conn(0x7f3b6c075670 msgr2=0x7f3b6c075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:37:31.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.985+0000 7f3b72a23700 1 -- 172.21.15.67:0/2778543195 shutdown_connections 2024-04-22T23:37:31.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.985+0000 7f3b72a23700 1 -- 172.21.15.67:0/2778543195 wait complete. 2024-04-22T23:37:31.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.986+0000 7f3b72a23700 1 Processor -- start 2024-04-22T23:37:31.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.986+0000 7f3b72a23700 1 -- start start 2024-04-22T23:37:31.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.986+0000 7f3b72a23700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3b6c068580 0x7f3b6c0ffcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:37:31.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.986+0000 7f3b72a23700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b6c106cd0 0x7f3b6c100230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:37:31.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.986+0000 7f3b72a23700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6c100770 con 0x7f3b6c106cd0 2024-04-22T23:37:31.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.986+0000 7f3b72a23700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6c1008b0 con 0x7f3b6c068580 2024-04-22T23:37:31.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.986+0000 7f3b6b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b6c106cd0 0x7f3b6c100230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:37:31.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.986+0000 7f3b6b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b6c106cd0 0x7f3b6c100230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51844/0 (socket says 172.21.15.67:51844) 2024-04-22T23:37:31.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.986+0000 7f3b6b7fe700 1 -- 172.21.15.67:0/1964592719 learned_addr learned my addr 172.21.15.67:0/1964592719 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:37:31.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.986+0000 7f3b6bfff700 1 --2- 172.21.15.67:0/1964592719 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3b6c068580 0x7f3b6c0ffcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:37:31.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.986+0000 7f3b6b7fe700 1 -- 172.21.15.67:0/1964592719 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3b6c068580 msgr2=0x7f3b6c0ffcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:37:31.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.986+0000 7f3b6b7fe700 1 --2- 172.21.15.67:0/1964592719 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3b6c068580 0x7f3b6c0ffcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:31.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.987+0000 7f3b6b7fe700 1 -- 172.21.15.67:0/1964592719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b600095f0 con 0x7f3b6c106cd0 2024-04-22T23:37:31.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.987+0000 7f3b6b7fe700 1 --2- 172.21.15.67:0/1964592719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b6c106cd0 0x7f3b6c100230 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3b60006010 tx=0x7f3b6000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:37:31.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.987+0000 7f3b697fa700 1 -- 172.21.15.67:0/1964592719 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b6001d070 con 0x7f3b6c106cd0 2024-04-22T23:37:31.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.987+0000 7f3b697fa700 1 -- 172.21.15.67:0/1964592719 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3b6000f460 con 0x7f3b6c106cd0 2024-04-22T23:37:31.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.987+0000 7f3b72a23700 1 -- 172.21.15.67:0/1964592719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b6c100b30 con 0x7f3b6c106cd0 2024-04-22T23:37:31.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.987+0000 7f3b697fa700 1 -- 172.21.15.67:0/1964592719 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b60017610 con 0x7f3b6c106cd0 2024-04-22T23:37:31.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.987+0000 7f3b72a23700 1 -- 172.21.15.67:0/1964592719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b6c1a2d60 con 0x7f3b6c106cd0 2024-04-22T23:37:31.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.988+0000 7f3b72a23700 1 -- 172.21.15.67:0/1964592719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b6c1091d0 con 0x7f3b6c106cd0 2024-04-22T23:37:31.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.988+0000 7f3b697fa700 1 -- 172.21.15.67:0/1964592719 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3b6000f5d0 con 0x7f3b6c106cd0 2024-04-22T23:37:31.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.989+0000 7f3b697fa700 1 --2- 172.21.15.67:0/1964592719 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3b5406ecc0 0x7f3b54071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:37:31.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.989+0000 7f3b697fa700 1 -- 172.21.15.67:0/1964592719 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f3b600904b0 con 0x7f3b6c106cd0 2024-04-22T23:37:31.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.989+0000 7f3b6bfff700 1 --2- 172.21.15.67:0/1964592719 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3b5406ecc0 0x7f3b54071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:37:31.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.989+0000 7f3b6bfff700 1 --2- 172.21.15.67:0/1964592719 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3b5406ecc0 0x7f3b54071180 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c005950 tx=0x7f3b5c00a300 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:37:31.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:31.994+0000 7f3b697fa700 1 -- 172.21.15.67:0/1964592719 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f3b600590d0 con 0x7f3b6c106cd0 2024-04-22T23:37:32.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.223+0000 7f3b72a23700 1 -- 172.21.15.67:0/1964592719 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3b6c0611d0 con 0x7f3b5406ecc0 2024-04-22T23:37:32.217 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:31 smithi067 ceph-mon[112438]: pgmap v334: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:32.217 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:31 smithi067 ceph-mon[112438]: from='client.34652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:37:32.217 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:31 smithi067 ceph-mon[112438]: from='client.34656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:37:32.217 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:31 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/183705426' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:37:32.218 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:31 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/4141988' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:37:32.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.224+0000 7f3b697fa700 1 -- 172.21.15.67:0/1964592719 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f3b6c0611d0 con 0x7f3b5406ecc0 2024-04-22T23:37:32.218 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:37:32.218 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:37:32.218 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:37:32.219 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:37:32.219 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:37:32.219 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:37:32.219 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:37:32.219 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:37:32.219 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:37:32.219 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:37:32.219 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T23:37:32.219 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:37:32.219 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:37:32.219 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:37:32.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.226+0000 7f3b72a23700 1 -- 172.21.15.67:0/1964592719 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3b5406ecc0 msgr2=0x7f3b54071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:37:32.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.226+0000 7f3b72a23700 1 --2- 172.21.15.67:0/1964592719 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3b5406ecc0 0x7f3b54071180 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c005950 tx=0x7f3b5c00a300 comp rx=0 tx=0).stop 2024-04-22T23:37:32.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.226+0000 7f3b72a23700 1 -- 172.21.15.67:0/1964592719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b6c106cd0 msgr2=0x7f3b6c100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:37:32.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.226+0000 7f3b72a23700 1 --2- 172.21.15.67:0/1964592719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b6c106cd0 0x7f3b6c100230 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3b60006010 tx=0x7f3b6000bd70 comp rx=0 tx=0).stop 2024-04-22T23:37:32.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.226+0000 7f3b72a23700 1 -- 172.21.15.67:0/1964592719 shutdown_connections 2024-04-22T23:37:32.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.227+0000 7f3b72a23700 1 --2- 172.21.15.67:0/1964592719 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3b5406ecc0 0x7f3b54071180 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:32.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.227+0000 7f3b72a23700 1 --2- 172.21.15.67:0/1964592719 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3b6c068580 0x7f3b6c0ffcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:32.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.227+0000 7f3b72a23700 1 --2- 172.21.15.67:0/1964592719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b6c106cd0 0x7f3b6c100230 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:32.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.227+0000 7f3b72a23700 1 -- 172.21.15.67:0/1964592719 >> 172.21.15.67:0/1964592719 conn(0x7f3b6c075670 msgr2=0x7f3b6c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:37:32.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.227+0000 7f3b72a23700 1 -- 172.21.15.67:0/1964592719 shutdown_connections 2024-04-22T23:37:32.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.227+0000 7f3b72a23700 1 -- 172.21.15.67:0/1964592719 wait complete. 2024-04-22T23:37:32.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:31 smithi134 ceph-mon[85630]: pgmap v334: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:32.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:31 smithi134 ceph-mon[85630]: from='client.34652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:37:32.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:31 smithi134 ceph-mon[85630]: from='client.34656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:37:32.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:31 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/183705426' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:37:32.238 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:31 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/4141988' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:37:32.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.342+0000 7f73b85bf700 1 -- 172.21.15.67:0/2906705319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73b0100ea0 msgr2=0x7f73b0101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:37:32.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.342+0000 7f73b85bf700 1 --2- 172.21.15.67:0/2906705319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73b0100ea0 0x7f73b0101280 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f73a4009910 tx=0x7f73a4009c20 comp rx=0 tx=0).stop 2024-04-22T23:37:32.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.343+0000 7f73b85bf700 1 -- 172.21.15.67:0/2906705319 shutdown_connections 2024-04-22T23:37:32.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.343+0000 7f73b85bf700 1 --2- 172.21.15.67:0/2906705319 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f73b0101850 0x7f73b01058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:32.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.343+0000 7f73b85bf700 1 --2- 172.21.15.67:0/2906705319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73b0100ea0 0x7f73b0101280 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:32.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.343+0000 7f73b85bf700 1 -- 172.21.15.67:0/2906705319 >> 172.21.15.67:0/2906705319 conn(0x7f73b00fc750 msgr2=0x7f73b00feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:37:32.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.343+0000 7f73b85bf700 1 -- 172.21.15.67:0/2906705319 shutdown_connections 2024-04-22T23:37:32.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.343+0000 7f73b85bf700 1 -- 172.21.15.67:0/2906705319 wait complete. 2024-04-22T23:37:32.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.343+0000 7f73b85bf700 1 Processor -- start 2024-04-22T23:37:32.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.343+0000 7f73b85bf700 1 -- start start 2024-04-22T23:37:32.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.344+0000 7f73b85bf700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73b0100ea0 0x7f73b019c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:37:32.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.344+0000 7f73b85bf700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f73b0101850 0x7f73b019cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:37:32.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.344+0000 7f73b85bf700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b019d2c0 con 0x7f73b0100ea0 2024-04-22T23:37:32.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.344+0000 7f73b85bf700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b0196770 con 0x7f73b0101850 2024-04-22T23:37:32.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.344+0000 7f73b635b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73b0100ea0 0x7f73b019c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:37:32.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.344+0000 7f73b635b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73b0100ea0 0x7f73b019c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51870/0 (socket says 172.21.15.67:51870) 2024-04-22T23:37:32.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.344+0000 7f73b635b700 1 -- 172.21.15.67:0/143002114 learned_addr learned my addr 172.21.15.67:0/143002114 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:37:32.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.344+0000 7f73b5b5a700 1 --2- 172.21.15.67:0/143002114 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f73b0101850 0x7f73b019cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:37:32.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.344+0000 7f73b635b700 1 -- 172.21.15.67:0/143002114 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f73b0101850 msgr2=0x7f73b019cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:37:32.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.344+0000 7f73b635b700 1 --2- 172.21.15.67:0/143002114 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f73b0101850 0x7f73b019cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:32.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.344+0000 7f73b635b700 1 -- 172.21.15.67:0/143002114 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73a40095f0 con 0x7f73b0100ea0 2024-04-22T23:37:32.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.344+0000 7f73b635b700 1 --2- 172.21.15.67:0/143002114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73b0100ea0 0x7f73b019c6f0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f73a4006010 tx=0x7f73a400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:37:32.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.344+0000 7f73a37fe700 1 -- 172.21.15.67:0/143002114 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f73a401d070 con 0x7f73b0100ea0 2024-04-22T23:37:32.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.344+0000 7f73b85bf700 1 -- 172.21.15.67:0/143002114 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73b01969f0 con 0x7f73b0100ea0 2024-04-22T23:37:32.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.344+0000 7f73a37fe700 1 -- 172.21.15.67:0/143002114 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f73a4005dc0 con 0x7f73b0100ea0 2024-04-22T23:37:32.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.344+0000 7f73a37fe700 1 -- 172.21.15.67:0/143002114 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f73a4017400 con 0x7f73b0100ea0 2024-04-22T23:37:32.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.345+0000 7f73b85bf700 1 -- 172.21.15.67:0/143002114 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73b0196ee0 con 0x7f73b0100ea0 2024-04-22T23:37:32.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.346+0000 7f73b85bf700 1 -- 172.21.15.67:0/143002114 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73b0109240 con 0x7f73b0100ea0 2024-04-22T23:37:32.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.346+0000 7f73a37fe700 1 -- 172.21.15.67:0/143002114 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f73a4003a60 con 0x7f73b0100ea0 2024-04-22T23:37:32.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.346+0000 7f73a37fe700 1 --2- 172.21.15.67:0/143002114 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f739c06ecc0 0x7f739c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:37:32.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.347+0000 7f73a37fe700 1 -- 172.21.15.67:0/143002114 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f73a4090410 con 0x7f73b0100ea0 2024-04-22T23:37:32.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.347+0000 7f73b5b5a700 1 --2- 172.21.15.67:0/143002114 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f739c06ecc0 0x7f739c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:37:32.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.347+0000 7f73b5b5a700 1 --2- 172.21.15.67:0/143002114 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f739c06ecc0 0x7f739c071180 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f73b0197be0 tx=0x7f73ac00b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:37:32.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.351+0000 7f73a37fe700 1 -- 172.21.15.67:0/143002114 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f73a405cb80 con 0x7f73b0100ea0 2024-04-22T23:37:32.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.652+0000 7f73b85bf700 1 -- 172.21.15.67:0/143002114 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f73b0197cd0 con 0x7f73b0100ea0 2024-04-22T23:37:32.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.652+0000 7f73a37fe700 1 -- 172.21.15.67:0/143002114 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f73a405c710 con 0x7f73b0100ea0 2024-04-22T23:37:32.646 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:37:32.646 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:37:32.646 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:37:32.646 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:37:32.647 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:37:32.647 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:37:32.647 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:37:32.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.654+0000 7f73b85bf700 1 -- 172.21.15.67:0/143002114 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f739c06ecc0 msgr2=0x7f739c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:37:32.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.654+0000 7f73b85bf700 1 --2- 172.21.15.67:0/143002114 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f739c06ecc0 0x7f739c071180 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f73b0197be0 tx=0x7f73ac00b410 comp rx=0 tx=0).stop 2024-04-22T23:37:32.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.654+0000 7f73b85bf700 1 -- 172.21.15.67:0/143002114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73b0100ea0 msgr2=0x7f73b019c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:37:32.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.654+0000 7f73b85bf700 1 --2- 172.21.15.67:0/143002114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73b0100ea0 0x7f73b019c6f0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f73a4006010 tx=0x7f73a400bd70 comp rx=0 tx=0).stop 2024-04-22T23:37:32.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.654+0000 7f73b85bf700 1 -- 172.21.15.67:0/143002114 shutdown_connections 2024-04-22T23:37:32.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.654+0000 7f73b85bf700 1 --2- 172.21.15.67:0/143002114 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f739c06ecc0 0x7f739c071180 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:32.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.654+0000 7f73b85bf700 1 --2- 172.21.15.67:0/143002114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73b0100ea0 0x7f73b019c6f0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:32.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.654+0000 7f73b85bf700 1 --2- 172.21.15.67:0/143002114 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f73b0101850 0x7f73b019cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:37:32.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.654+0000 7f73b85bf700 1 -- 172.21.15.67:0/143002114 >> 172.21.15.67:0/143002114 conn(0x7f73b00fc750 msgr2=0x7f73b00fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:37:32.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.654+0000 7f73b85bf700 1 -- 172.21.15.67:0/143002114 shutdown_connections 2024-04-22T23:37:32.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:37:32.654+0000 7f73b85bf700 1 -- 172.21.15.67:0/143002114 wait complete. 2024-04-22T23:37:33.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:32 smithi067 ceph-mon[112438]: from='client.34660 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:37:33.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:32 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/143002114' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:37:33.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:32 smithi134 ceph-mon[85630]: from='client.34660 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:37:33.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:32 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/143002114' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:37:34.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:33 smithi134 ceph-mon[85630]: pgmap v335: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:34.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:33 smithi134 ceph-mon[85630]: from='client.34672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:37:34.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:33 smithi067 ceph-mon[112438]: pgmap v335: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:34.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:33 smithi067 ceph-mon[112438]: from='client.34672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:37:36.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:35 smithi134 ceph-mon[85630]: pgmap v336: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:36.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:35 smithi067 ceph-mon[112438]: pgmap v336: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:38.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:37 smithi134 ceph-mon[85630]: pgmap v337: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:38.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:37 smithi067 ceph-mon[112438]: pgmap v337: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:40.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:39 smithi134 ceph-mon[85630]: pgmap v338: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:40.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:39 smithi067 ceph-mon[112438]: pgmap v338: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:42.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:41 smithi134 ceph-mon[85630]: pgmap v339: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:42.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:37:42.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:41 smithi067 ceph-mon[112438]: pgmap v339: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:42.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:37:44.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:43 smithi134 ceph-mon[85630]: pgmap v340: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:44.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:43 smithi067 ceph-mon[112438]: pgmap v340: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:46.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:45 smithi134 ceph-mon[85630]: pgmap v341: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:46.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:45 smithi067 ceph-mon[112438]: pgmap v341: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:48.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:47 smithi134 ceph-mon[85630]: pgmap v342: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:48.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:47 smithi067 ceph-mon[112438]: pgmap v342: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:50.173 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:49 smithi134 ceph-mon[85630]: pgmap v343: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:50.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:49 smithi067 ceph-mon[112438]: pgmap v343: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:52.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:51 smithi134 ceph-mon[85630]: pgmap v344: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:52.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:51 smithi067 ceph-mon[112438]: pgmap v344: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:54.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:53 smithi134 ceph-mon[85630]: pgmap v345: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:54.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:53 smithi067 ceph-mon[112438]: pgmap v345: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:56.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:55 smithi067 ceph-mon[112438]: pgmap v346: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:56.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:55 smithi134 ceph-mon[85630]: pgmap v346: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:57.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:37:57.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:37:58.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:57 smithi134 ceph-mon[85630]: pgmap v347: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:37:58.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:57 smithi067 ceph-mon[112438]: pgmap v347: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:00.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:37:59 smithi134 ceph-mon[85630]: pgmap v348: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:00.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:37:59 smithi067 ceph-mon[112438]: pgmap v348: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:02.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:01 smithi134 ceph-mon[85630]: pgmap v349: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:02.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:01 smithi067 ceph-mon[112438]: pgmap v349: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:02.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:02.776+0000 7f486e849700 1 -- 172.21.15.67:0/2953129387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48681027e0 msgr2=0x7f486810acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:02.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:02.776+0000 7f486e849700 1 --2- 172.21.15.67:0/2953129387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48681027e0 0x7f486810acd0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f4850009940 tx=0x7f4850009c50 comp rx=0 tx=0).stop 2024-04-22T23:38:02.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:02.776+0000 7f486e849700 1 -- 172.21.15.67:0/2953129387 shutdown_connections 2024-04-22T23:38:02.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:02.776+0000 7f486e849700 1 --2- 172.21.15.67:0/2953129387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48681027e0 0x7f486810acd0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:02.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:02.776+0000 7f486e849700 1 --2- 172.21.15.67:0/2953129387 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4868101ec0 0x7f48681022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:02.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:02.776+0000 7f486e849700 1 -- 172.21.15.67:0/2953129387 >> 172.21.15.67:0/2953129387 conn(0x7f48680fb650 msgr2=0x7f48680fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:38:02.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:02.776+0000 7f486e849700 1 -- 172.21.15.67:0/2953129387 shutdown_connections 2024-04-22T23:38:02.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:02.776+0000 7f486e849700 1 -- 172.21.15.67:0/2953129387 wait complete. 2024-04-22T23:38:02.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:02.777+0000 7f486e849700 1 Processor -- start 2024-04-22T23:38:02.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:02.777+0000 7f486e849700 1 -- start start 2024-04-22T23:38:02.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:02.777+0000 7f486e849700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4868101ec0 0x7f486819ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:02.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:02.777+0000 7f486e849700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f48681027e0 0x7f486819d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:02.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:02.777+0000 7f486e849700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f486819dad0 con 0x7f4868101ec0 2024-04-22T23:38:02.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:02.777+0000 7f486e849700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48681a1860 con 0x7f48681027e0 2024-04-22T23:38:02.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:02.778+0000 7f4867fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4868101ec0 0x7f486819ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:02.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:02.778+0000 7f4867fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4868101ec0 0x7f486819ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59208/0 (socket says 172.21.15.67:59208) 2024-04-22T23:38:02.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:02.778+0000 7f4867fff700 1 -- 172.21.15.67:0/1366008819 learned_addr learned my addr 172.21.15.67:0/1366008819 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:38:02.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:02.778+0000 7f48677fe700 1 --2- 172.21.15.67:0/1366008819 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f48681027e0 0x7f486819d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:02.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:02.778+0000 7f4867fff700 1 -- 172.21.15.67:0/1366008819 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f48681027e0 msgr2=0x7f486819d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:02.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:02.778+0000 7f4867fff700 1 --2- 172.21.15.67:0/1366008819 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f48681027e0 0x7f486819d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:02.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:02.778+0000 7f4867fff700 1 -- 172.21.15.67:0/1366008819 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48500095f0 con 0x7f4868101ec0 2024-04-22T23:38:02.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:02.778+0000 7f48677fe700 1 --2- 172.21.15.67:0/1366008819 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f48681027e0 0x7f486819d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:38:02.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:02.778+0000 7f4867fff700 1 --2- 172.21.15.67:0/1366008819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4868101ec0 0x7f486819ceb0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f485800d8d0 tx=0x7f485800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:38:02.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:02.778+0000 7f48657fa700 1 -- 172.21.15.67:0/1366008819 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4858009940 con 0x7f4868101ec0 2024-04-22T23:38:02.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:02.778+0000 7f48657fa700 1 -- 172.21.15.67:0/1366008819 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4858010460 con 0x7f4868101ec0 2024-04-22T23:38:02.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:02.778+0000 7f486e849700 1 -- 172.21.15.67:0/1366008819 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48681a1b40 con 0x7f4868101ec0 2024-04-22T23:38:02.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:02.779+0000 7f48657fa700 1 -- 172.21.15.67:0/1366008819 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f485800f5f0 con 0x7f4868101ec0 2024-04-22T23:38:02.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:02.779+0000 7f486e849700 1 -- 172.21.15.67:0/1366008819 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48681a2060 con 0x7f4868101ec0 2024-04-22T23:38:02.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:02.780+0000 7f486e849700 1 -- 172.21.15.67:0/1366008819 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4868108360 con 0x7f4868101ec0 2024-04-22T23:38:02.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:02.780+0000 7f48657fa700 1 -- 172.21.15.67:0/1366008819 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f48580105d0 con 0x7f4868101ec0 2024-04-22T23:38:02.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:02.781+0000 7f48657fa700 1 --2- 172.21.15.67:0/1366008819 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f485406ecc0 0x7f4854071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:02.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:02.781+0000 7f48657fa700 1 -- 172.21.15.67:0/1366008819 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f485808eee0 con 0x7f4868101ec0 2024-04-22T23:38:02.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:02.781+0000 7f48677fe700 1 --2- 172.21.15.67:0/1366008819 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f485406ecc0 0x7f4854071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:02.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:02.781+0000 7f48677fe700 1 --2- 172.21.15.67:0/1366008819 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f485406ecc0 0x7f4854071180 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4850006010 tx=0x7f4850005a90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:38:02.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:02.786+0000 7f48657fa700 1 -- 172.21.15.67:0/1366008819 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f4858057ba0 con 0x7f4868101ec0 2024-04-22T23:38:03.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.014+0000 7f486e849700 1 -- 172.21.15.67:0/1366008819 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f48681a22d0 con 0x7f485406ecc0 2024-04-22T23:38:03.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.015+0000 7f48657fa700 1 -- 172.21.15.67:0/1366008819 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f48681a22d0 con 0x7f485406ecc0 2024-04-22T23:38:03.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.017+0000 7f486e849700 1 -- 172.21.15.67:0/1366008819 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f485406ecc0 msgr2=0x7f4854071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:03.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.017+0000 7f486e849700 1 --2- 172.21.15.67:0/1366008819 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f485406ecc0 0x7f4854071180 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4850006010 tx=0x7f4850005a90 comp rx=0 tx=0).stop 2024-04-22T23:38:03.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.017+0000 7f486e849700 1 -- 172.21.15.67:0/1366008819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4868101ec0 msgr2=0x7f486819ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:03.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.017+0000 7f486e849700 1 --2- 172.21.15.67:0/1366008819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4868101ec0 0x7f486819ceb0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f485800d8d0 tx=0x7f485800dc90 comp rx=0 tx=0).stop 2024-04-22T23:38:03.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.017+0000 7f486e849700 1 -- 172.21.15.67:0/1366008819 shutdown_connections 2024-04-22T23:38:03.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.017+0000 7f486e849700 1 --2- 172.21.15.67:0/1366008819 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f485406ecc0 0x7f4854071180 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:03.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.017+0000 7f486e849700 1 --2- 172.21.15.67:0/1366008819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4868101ec0 0x7f486819ceb0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:03.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.017+0000 7f486e849700 1 --2- 172.21.15.67:0/1366008819 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f48681027e0 0x7f486819d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:03.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.018+0000 7f486e849700 1 -- 172.21.15.67:0/1366008819 >> 172.21.15.67:0/1366008819 conn(0x7f48680fb650 msgr2=0x7f48681054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:38:03.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.018+0000 7f486e849700 1 -- 172.21.15.67:0/1366008819 shutdown_connections 2024-04-22T23:38:03.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.018+0000 7f486e849700 1 -- 172.21.15.67:0/1366008819 wait complete. 2024-04-22T23:38:03.022 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:38:03.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.126+0000 7fca13d18700 1 -- 172.21.15.67:0/3450491739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca0c0ffc20 msgr2=0x7fca0c100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:03.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.126+0000 7fca13d18700 1 --2- 172.21.15.67:0/3450491739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca0c0ffc20 0x7fca0c100080 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc009910 tx=0x7fc9fc009c20 comp rx=0 tx=0).stop 2024-04-22T23:38:03.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.126+0000 7fca13d18700 1 -- 172.21.15.67:0/3450491739 shutdown_connections 2024-04-22T23:38:03.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.126+0000 7fca13d18700 1 --2- 172.21.15.67:0/3450491739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca0c0ffc20 0x7fca0c100080 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:03.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.126+0000 7fca13d18700 1 --2- 172.21.15.67:0/3450491739 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fca0c105bd0 0x7fca0c105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:03.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.126+0000 7fca13d18700 1 -- 172.21.15.67:0/3450491739 >> 172.21.15.67:0/3450491739 conn(0x7fca0c075670 msgr2=0x7fca0c075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:38:03.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.126+0000 7fca13d18700 1 -- 172.21.15.67:0/3450491739 shutdown_connections 2024-04-22T23:38:03.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.126+0000 7fca13d18700 1 -- 172.21.15.67:0/3450491739 wait complete. 2024-04-22T23:38:03.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.127+0000 7fca13d18700 1 Processor -- start 2024-04-22T23:38:03.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.127+0000 7fca13d18700 1 -- start start 2024-04-22T23:38:03.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.127+0000 7fca13d18700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fca0c0ffc20 0x7fca0c19c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:03.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.127+0000 7fca13d18700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca0c105bd0 0x7fca0c19cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:03.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.127+0000 7fca13d18700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0c19d230 con 0x7fca0c105bd0 2024-04-22T23:38:03.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.127+0000 7fca13d18700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0c1966e0 con 0x7fca0c0ffc20 2024-04-22T23:38:03.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.127+0000 7fca112b3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca0c105bd0 0x7fca0c19cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:03.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.127+0000 7fca112b3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca0c105bd0 0x7fca0c19cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59232/0 (socket says 172.21.15.67:59232) 2024-04-22T23:38:03.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.127+0000 7fca112b3700 1 -- 172.21.15.67:0/1813154092 learned_addr learned my addr 172.21.15.67:0/1813154092 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:38:03.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.127+0000 7fca11ab4700 1 --2- 172.21.15.67:0/1813154092 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fca0c0ffc20 0x7fca0c19c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:03.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.127+0000 7fca112b3700 1 -- 172.21.15.67:0/1813154092 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fca0c0ffc20 msgr2=0x7fca0c19c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:03.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.127+0000 7fca112b3700 1 --2- 172.21.15.67:0/1813154092 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fca0c0ffc20 0x7fca0c19c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:03.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.127+0000 7fca112b3700 1 -- 172.21.15.67:0/1813154092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9fc0095f0 con 0x7fca0c105bd0 2024-04-22T23:38:03.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.128+0000 7fca11ab4700 1 --2- 172.21.15.67:0/1813154092 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fca0c0ffc20 0x7fca0c19c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:38:03.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.128+0000 7fca112b3700 1 --2- 172.21.15.67:0/1813154092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca0c105bd0 0x7fca0c19cba0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc006010 tx=0x7fc9fc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:38:03.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.128+0000 7fca02ffd700 1 -- 172.21.15.67:0/1813154092 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9fc01d070 con 0x7fca0c105bd0 2024-04-22T23:38:03.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.128+0000 7fca13d18700 1 -- 172.21.15.67:0/1813154092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca0c196960 con 0x7fca0c105bd0 2024-04-22T23:38:03.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.128+0000 7fca02ffd700 1 -- 172.21.15.67:0/1813154092 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc9fc00f460 con 0x7fca0c105bd0 2024-04-22T23:38:03.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.128+0000 7fca02ffd700 1 -- 172.21.15.67:0/1813154092 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9fc017630 con 0x7fca0c105bd0 2024-04-22T23:38:03.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.128+0000 7fca13d18700 1 -- 172.21.15.67:0/1813154092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca0c196e50 con 0x7fca0c105bd0 2024-04-22T23:38:03.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.129+0000 7fca13d18700 1 -- 172.21.15.67:0/1813154092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca0c04f2e0 con 0x7fca0c105bd0 2024-04-22T23:38:03.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.129+0000 7fca02ffd700 1 -- 172.21.15.67:0/1813154092 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc9fc00f5d0 con 0x7fca0c105bd0 2024-04-22T23:38:03.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.130+0000 7fca02ffd700 1 --2- 172.21.15.67:0/1813154092 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc9f806ecc0 0x7fc9f8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:03.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.130+0000 7fca02ffd700 1 -- 172.21.15.67:0/1813154092 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fc9fc090470 con 0x7fca0c105bd0 2024-04-22T23:38:03.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.130+0000 7fca11ab4700 1 --2- 172.21.15.67:0/1813154092 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc9f806ecc0 0x7fc9f8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:03.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.130+0000 7fca11ab4700 1 --2- 172.21.15.67:0/1813154092 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc9f806ecc0 0x7fc9f8071180 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fca08005fd0 tx=0x7fca08005ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:38:03.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.135+0000 7fca02ffd700 1 -- 172.21.15.67:0/1813154092 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc9fc05cbe0 con 0x7fca0c105bd0 2024-04-22T23:38:03.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.364+0000 7fca13d18700 1 -- 172.21.15.67:0/1813154092 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fca0c1970e0 con 0x7fc9f806ecc0 2024-04-22T23:38:03.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.365+0000 7fca02ffd700 1 -- 172.21.15.67:0/1813154092 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fca0c1970e0 con 0x7fc9f806ecc0 2024-04-22T23:38:03.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.367+0000 7fca13d18700 1 -- 172.21.15.67:0/1813154092 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc9f806ecc0 msgr2=0x7fc9f8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:03.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.367+0000 7fca13d18700 1 --2- 172.21.15.67:0/1813154092 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc9f806ecc0 0x7fc9f8071180 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fca08005fd0 tx=0x7fca08005ee0 comp rx=0 tx=0).stop 2024-04-22T23:38:03.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.367+0000 7fca13d18700 1 -- 172.21.15.67:0/1813154092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca0c105bd0 msgr2=0x7fca0c19cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:03.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.367+0000 7fca13d18700 1 --2- 172.21.15.67:0/1813154092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca0c105bd0 0x7fca0c19cba0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc006010 tx=0x7fc9fc00be20 comp rx=0 tx=0).stop 2024-04-22T23:38:03.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.367+0000 7fca13d18700 1 -- 172.21.15.67:0/1813154092 shutdown_connections 2024-04-22T23:38:03.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.367+0000 7fca13d18700 1 --2- 172.21.15.67:0/1813154092 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc9f806ecc0 0x7fc9f8071180 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:03.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.367+0000 7fca13d18700 1 --2- 172.21.15.67:0/1813154092 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fca0c0ffc20 0x7fca0c19c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:03.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.367+0000 7fca13d18700 1 --2- 172.21.15.67:0/1813154092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca0c105bd0 0x7fca0c19cba0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:03.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.367+0000 7fca13d18700 1 -- 172.21.15.67:0/1813154092 >> 172.21.15.67:0/1813154092 conn(0x7fca0c075670 msgr2=0x7fca0c103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:38:03.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.368+0000 7fca13d18700 1 -- 172.21.15.67:0/1813154092 shutdown_connections 2024-04-22T23:38:03.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.368+0000 7fca13d18700 1 -- 172.21.15.67:0/1813154092 wait complete. 2024-04-22T23:38:03.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.475+0000 7ff85ad68700 1 -- 172.21.15.67:0/182021210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8540ff5a0 msgr2=0x7ff8540ff980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:03.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.475+0000 7ff85ad68700 1 --2- 172.21.15.67:0/182021210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8540ff5a0 0x7ff8540ff980 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff844009910 tx=0x7ff844009c20 comp rx=0 tx=0).stop 2024-04-22T23:38:03.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.475+0000 7ff85ad68700 1 -- 172.21.15.67:0/182021210 shutdown_connections 2024-04-22T23:38:03.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.475+0000 7ff85ad68700 1 --2- 172.21.15.67:0/182021210 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8540fff50 0x7ff854105840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:03.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.475+0000 7ff85ad68700 1 --2- 172.21.15.67:0/182021210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8540ff5a0 0x7ff8540ff980 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:03.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.475+0000 7ff85ad68700 1 -- 172.21.15.67:0/182021210 >> 172.21.15.67:0/182021210 conn(0x7ff854074e30 msgr2=0x7ff854075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:38:03.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.475+0000 7ff85ad68700 1 -- 172.21.15.67:0/182021210 shutdown_connections 2024-04-22T23:38:03.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.475+0000 7ff85ad68700 1 -- 172.21.15.67:0/182021210 wait complete. 2024-04-22T23:38:03.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.475+0000 7ff85ad68700 1 Processor -- start 2024-04-22T23:38:03.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.475+0000 7ff85ad68700 1 -- start start 2024-04-22T23:38:03.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.476+0000 7ff85ad68700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8540ff5a0 0x7ff854196950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:03.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.476+0000 7ff85ad68700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8540fff50 0x7ff854196e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:03.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.476+0000 7ff85ad68700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff854197520 con 0x7ff8540fff50 2024-04-22T23:38:03.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.476+0000 7ff85ad68700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff85419b330 con 0x7ff8540ff5a0 2024-04-22T23:38:03.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.476+0000 7ff853fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8540fff50 0x7ff854196e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:03.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.476+0000 7ff853fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8540fff50 0x7ff854196e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59252/0 (socket says 172.21.15.67:59252) 2024-04-22T23:38:03.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.476+0000 7ff853fff700 1 -- 172.21.15.67:0/3258095970 learned_addr learned my addr 172.21.15.67:0/3258095970 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:38:03.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.476+0000 7ff858b04700 1 --2- 172.21.15.67:0/3258095970 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8540ff5a0 0x7ff854196950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:03.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.476+0000 7ff853fff700 1 -- 172.21.15.67:0/3258095970 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8540ff5a0 msgr2=0x7ff854196950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:03.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.476+0000 7ff853fff700 1 --2- 172.21.15.67:0/3258095970 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8540ff5a0 0x7ff854196950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:03.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.476+0000 7ff853fff700 1 -- 172.21.15.67:0/3258095970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8440095f0 con 0x7ff8540fff50 2024-04-22T23:38:03.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.476+0000 7ff853fff700 1 --2- 172.21.15.67:0/3258095970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8540fff50 0x7ff854196e90 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff84800d8d0 tx=0x7ff84800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:38:03.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.476+0000 7ff851ffb700 1 -- 172.21.15.67:0/3258095970 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff848004a00 con 0x7ff8540fff50 2024-04-22T23:38:03.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.477+0000 7ff851ffb700 1 -- 172.21.15.67:0/3258095970 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff8480054e0 con 0x7ff8540fff50 2024-04-22T23:38:03.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.477+0000 7ff85ad68700 1 -- 172.21.15.67:0/3258095970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff85419b610 con 0x7ff8540fff50 2024-04-22T23:38:03.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.477+0000 7ff851ffb700 1 -- 172.21.15.67:0/3258095970 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff84800fc20 con 0x7ff8540fff50 2024-04-22T23:38:03.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.477+0000 7ff85ad68700 1 -- 172.21.15.67:0/3258095970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff85419bb60 con 0x7ff8540fff50 2024-04-22T23:38:03.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.478+0000 7ff85ad68700 1 -- 172.21.15.67:0/3258095970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff854109270 con 0x7ff8540fff50 2024-04-22T23:38:03.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.478+0000 7ff851ffb700 1 -- 172.21.15.67:0/3258095970 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff848005020 con 0x7ff8540fff50 2024-04-22T23:38:03.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.478+0000 7ff851ffb700 1 --2- 172.21.15.67:0/3258095970 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff83c06ecc0 0x7ff83c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:03.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.479+0000 7ff851ffb700 1 -- 172.21.15.67:0/3258095970 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7ff84808e870 con 0x7ff8540fff50 2024-04-22T23:38:03.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.479+0000 7ff858b04700 1 --2- 172.21.15.67:0/3258095970 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff83c06ecc0 0x7ff83c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:03.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.479+0000 7ff858b04700 1 --2- 172.21.15.67:0/3258095970 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff83c06ecc0 0x7ff83c071180 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff844006010 tx=0x7ff84401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:38:03.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.483+0000 7ff851ffb700 1 -- 172.21.15.67:0/3258095970 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff84805afe0 con 0x7ff8540fff50 2024-04-22T23:38:03.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.703+0000 7ff85ad68700 1 -- 172.21.15.67:0/3258095970 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff854197d00 con 0x7ff83c06ecc0 2024-04-22T23:38:03.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.715+0000 7ff851ffb700 1 -- 172.21.15.67:0/3258095970 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7ff854197d00 con 0x7ff83c06ecc0 2024-04-22T23:38:03.709 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:38:03.709 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (11m) 5m ago 22m 25.5M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:38:03.709 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (23m) 5m ago 23m 9114k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:38:03.709 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (22m) 4m ago 22m 9042k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:38:03.709 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (9m) 5m ago 23m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:38:03.709 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (9m) 4m ago 22m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:38:03.709 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (11m) 5m ago 22m 79.3M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:38:03.709 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (5m) 5m ago 17m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:38:03.709 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (5m) 5m ago 17m 12.6M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:38:03.709 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (4m) 4m ago 17m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:38:03.709 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (17m) 4m ago 17m 12.2M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:38:03.710 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (13m) 5m ago 24m 511M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:38:03.710 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (12m) 4m ago 22m 453M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:38:03.710 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (10m) 5m ago 24m 55.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:38:03.710 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (9m) 4m ago 21m 46.0M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:38:03.710 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (12m) 5m ago 23m 15.4M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:38:03.710 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (12m) 4m ago 22m 16.1M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:38:03.710 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (8m) 5m ago 21m 99.2M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:38:03.710 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (8m) 5m ago 20m 127M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:38:03.710 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (7m) 5m ago 20m 79.9M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:38:03.710 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (6m) 4m ago 20m 97.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:38:03.710 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (6m) 4m ago 19m 82.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:38:03.710 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (5m) 4m ago 19m 77.6M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:38:03.710 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (12m) 5m ago 22m 73.9M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:38:03.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.717+0000 7ff85ad68700 1 -- 172.21.15.67:0/3258095970 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff83c06ecc0 msgr2=0x7ff83c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:03.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.717+0000 7ff85ad68700 1 --2- 172.21.15.67:0/3258095970 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff83c06ecc0 0x7ff83c071180 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff844006010 tx=0x7ff84401a040 comp rx=0 tx=0).stop 2024-04-22T23:38:03.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.717+0000 7ff85ad68700 1 -- 172.21.15.67:0/3258095970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8540fff50 msgr2=0x7ff854196e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:03.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.717+0000 7ff85ad68700 1 --2- 172.21.15.67:0/3258095970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8540fff50 0x7ff854196e90 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff84800d8d0 tx=0x7ff84800dbe0 comp rx=0 tx=0).stop 2024-04-22T23:38:03.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.717+0000 7ff85ad68700 1 -- 172.21.15.67:0/3258095970 shutdown_connections 2024-04-22T23:38:03.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.717+0000 7ff85ad68700 1 --2- 172.21.15.67:0/3258095970 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff83c06ecc0 0x7ff83c071180 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:03.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.717+0000 7ff85ad68700 1 --2- 172.21.15.67:0/3258095970 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8540ff5a0 0x7ff854196950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:03.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.717+0000 7ff85ad68700 1 --2- 172.21.15.67:0/3258095970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8540fff50 0x7ff854196e90 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:03.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.717+0000 7ff85ad68700 1 -- 172.21.15.67:0/3258095970 >> 172.21.15.67:0/3258095970 conn(0x7ff854074e30 msgr2=0x7ff8540fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:38:03.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.717+0000 7ff85ad68700 1 -- 172.21.15.67:0/3258095970 shutdown_connections 2024-04-22T23:38:03.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.718+0000 7ff85ad68700 1 -- 172.21.15.67:0/3258095970 wait complete. 2024-04-22T23:38:03.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.823+0000 7fc3da82a700 1 -- 172.21.15.67:0/2187379526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3d4068580 msgr2=0x7fc3d40689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:03.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.823+0000 7fc3da82a700 1 --2- 172.21.15.67:0/2187379526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3d4068580 0x7fc3d40689e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8009910 tx=0x7fc3c8009c20 comp rx=0 tx=0).stop 2024-04-22T23:38:03.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.823+0000 7fc3da82a700 1 -- 172.21.15.67:0/2187379526 shutdown_connections 2024-04-22T23:38:03.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.823+0000 7fc3da82a700 1 --2- 172.21.15.67:0/2187379526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3d4068580 0x7fc3d40689e0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:03.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.823+0000 7fc3da82a700 1 --2- 172.21.15.67:0/2187379526 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc3d4106cd0 0x7fc3d41070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:03.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.823+0000 7fc3da82a700 1 -- 172.21.15.67:0/2187379526 >> 172.21.15.67:0/2187379526 conn(0x7fc3d4075670 msgr2=0x7fc3d4075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:38:03.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.823+0000 7fc3da82a700 1 -- 172.21.15.67:0/2187379526 shutdown_connections 2024-04-22T23:38:03.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.823+0000 7fc3da82a700 1 -- 172.21.15.67:0/2187379526 wait complete. 2024-04-22T23:38:03.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.824+0000 7fc3da82a700 1 Processor -- start 2024-04-22T23:38:03.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.824+0000 7fc3da82a700 1 -- start start 2024-04-22T23:38:03.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.824+0000 7fc3da82a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc3d4068580 0x7fc3d40ffcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:03.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.824+0000 7fc3da82a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3d4106cd0 0x7fc3d4100230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:03.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.824+0000 7fc3da82a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d4100770 con 0x7fc3d4106cd0 2024-04-22T23:38:03.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.824+0000 7fc3da82a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d41008b0 con 0x7fc3d4068580 2024-04-22T23:38:03.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.825+0000 7fc3d37fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3d4106cd0 0x7fc3d4100230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:03.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.825+0000 7fc3d3fff700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc3d4068580 0x7fc3d40ffcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:03.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.825+0000 7fc3d37fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3d4106cd0 0x7fc3d4100230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59264/0 (socket says 172.21.15.67:59264) 2024-04-22T23:38:03.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.825+0000 7fc3d37fe700 1 -- 172.21.15.67:0/2960350373 learned_addr learned my addr 172.21.15.67:0/2960350373 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:38:03.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.825+0000 7fc3d37fe700 1 -- 172.21.15.67:0/2960350373 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc3d4068580 msgr2=0x7fc3d40ffcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:03.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.825+0000 7fc3d37fe700 1 --2- 172.21.15.67:0/2960350373 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc3d4068580 0x7fc3d40ffcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:03.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.825+0000 7fc3d37fe700 1 -- 172.21.15.67:0/2960350373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c80095f0 con 0x7fc3d4106cd0 2024-04-22T23:38:03.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.825+0000 7fc3d37fe700 1 --2- 172.21.15.67:0/2960350373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3d4106cd0 0x7fc3d4100230 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8000c00 tx=0x7fc3c800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:38:03.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.825+0000 7fc3d17fa700 1 -- 172.21.15.67:0/2960350373 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc3c801d070 con 0x7fc3d4106cd0 2024-04-22T23:38:03.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.825+0000 7fc3d17fa700 1 -- 172.21.15.67:0/2960350373 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc3c8005dc0 con 0x7fc3d4106cd0 2024-04-22T23:38:03.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.825+0000 7fc3da82a700 1 -- 172.21.15.67:0/2960350373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3d4100b30 con 0x7fc3d4106cd0 2024-04-22T23:38:03.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.825+0000 7fc3d17fa700 1 -- 172.21.15.67:0/2960350373 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc3c8017400 con 0x7fc3d4106cd0 2024-04-22T23:38:03.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.825+0000 7fc3da82a700 1 -- 172.21.15.67:0/2960350373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3d41a2d60 con 0x7fc3d4106cd0 2024-04-22T23:38:03.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.826+0000 7fc3da82a700 1 -- 172.21.15.67:0/2960350373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3d41091d0 con 0x7fc3d4106cd0 2024-04-22T23:38:03.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.827+0000 7fc3d17fa700 1 -- 172.21.15.67:0/2960350373 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc3c800f460 con 0x7fc3d4106cd0 2024-04-22T23:38:03.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.827+0000 7fc3d17fa700 1 --2- 172.21.15.67:0/2960350373 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc3bc06ecc0 0x7fc3bc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:03.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.827+0000 7fc3d17fa700 1 -- 172.21.15.67:0/2960350373 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fc3c80903c0 con 0x7fc3d4106cd0 2024-04-22T23:38:03.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.827+0000 7fc3d3fff700 1 --2- 172.21.15.67:0/2960350373 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc3bc06ecc0 0x7fc3bc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:03.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.828+0000 7fc3d3fff700 1 --2- 172.21.15.67:0/2960350373 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc3bc06ecc0 0x7fc3bc071180 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4005950 tx=0x7fc3c400b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:38:03.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:03.832+0000 7fc3d17fa700 1 -- 172.21.15.67:0/2960350373 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc3c8059080 con 0x7fc3d4106cd0 2024-04-22T23:38:04.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:03 smithi067 ceph-mon[112438]: pgmap v350: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:04.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.128+0000 7fc3da82a700 1 -- 172.21.15.67:0/2960350373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc3d404f2e0 con 0x7fc3d4106cd0 2024-04-22T23:38:04.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.128+0000 7fc3d17fa700 1 -- 172.21.15.67:0/2960350373 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fc3c805c740 con 0x7fc3d4106cd0 2024-04-22T23:38:04.122 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:38:04.123 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:38:04.123 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:38:04.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:38:04.123 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:38:04.123 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:38:04.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:38:04.123 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:38:04.123 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:38:04.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:38:04.123 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:38:04.123 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:38:04.123 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 3 2024-04-22T23:38:04.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:38:04.123 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:38:04.123 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:38:04.123 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 13 2024-04-22T23:38:04.124 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:38:04.124 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:38:04.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.131+0000 7fc3da82a700 1 -- 172.21.15.67:0/2960350373 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc3bc06ecc0 msgr2=0x7fc3bc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:04.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.131+0000 7fc3da82a700 1 --2- 172.21.15.67:0/2960350373 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc3bc06ecc0 0x7fc3bc071180 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4005950 tx=0x7fc3c400b410 comp rx=0 tx=0).stop 2024-04-22T23:38:04.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.131+0000 7fc3da82a700 1 -- 172.21.15.67:0/2960350373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3d4106cd0 msgr2=0x7fc3d4100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:04.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.131+0000 7fc3da82a700 1 --2- 172.21.15.67:0/2960350373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3d4106cd0 0x7fc3d4100230 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8000c00 tx=0x7fc3c800bd70 comp rx=0 tx=0).stop 2024-04-22T23:38:04.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.131+0000 7fc3da82a700 1 -- 172.21.15.67:0/2960350373 shutdown_connections 2024-04-22T23:38:04.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.131+0000 7fc3da82a700 1 --2- 172.21.15.67:0/2960350373 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc3bc06ecc0 0x7fc3bc071180 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:04.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.131+0000 7fc3da82a700 1 --2- 172.21.15.67:0/2960350373 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc3d4068580 0x7fc3d40ffcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:04.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.131+0000 7fc3da82a700 1 --2- 172.21.15.67:0/2960350373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3d4106cd0 0x7fc3d4100230 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:04.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.131+0000 7fc3da82a700 1 -- 172.21.15.67:0/2960350373 >> 172.21.15.67:0/2960350373 conn(0x7fc3d4075670 msgr2=0x7fc3d40feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:38:04.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.131+0000 7fc3da82a700 1 -- 172.21.15.67:0/2960350373 shutdown_connections 2024-04-22T23:38:04.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.131+0000 7fc3da82a700 1 -- 172.21.15.67:0/2960350373 wait complete. 2024-04-22T23:38:04.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.237+0000 7fc02280f700 1 -- 172.21.15.67:0/1414103503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc01c100750 msgr2=0x7fc01c1047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:04.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.237+0000 7fc02280f700 1 --2- 172.21.15.67:0/1414103503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc01c100750 0x7fc01c1047a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc00c009910 tx=0x7fc00c009c20 comp rx=0 tx=0).stop 2024-04-22T23:38:04.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.238+0000 7fc02280f700 1 -- 172.21.15.67:0/1414103503 shutdown_connections 2024-04-22T23:38:04.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.238+0000 7fc02280f700 1 --2- 172.21.15.67:0/1414103503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc01c100750 0x7fc01c1047a0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:04.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.238+0000 7fc02280f700 1 --2- 172.21.15.67:0/1414103503 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc01c0ffda0 0x7fc01c100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:04.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.238+0000 7fc02280f700 1 -- 172.21.15.67:0/1414103503 >> 172.21.15.67:0/1414103503 conn(0x7fc01c0fb650 msgr2=0x7fc01c0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:38:04.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.238+0000 7fc02280f700 1 -- 172.21.15.67:0/1414103503 shutdown_connections 2024-04-22T23:38:04.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.238+0000 7fc02280f700 1 -- 172.21.15.67:0/1414103503 wait complete. 2024-04-22T23:38:04.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.238+0000 7fc02280f700 1 Processor -- start 2024-04-22T23:38:04.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.238+0000 7fc02280f700 1 -- start start 2024-04-22T23:38:04.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.239+0000 7fc02280f700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc01c0ffda0 0x7fc01c19c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:04.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.239+0000 7fc02280f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc01c100750 0x7fc01c19cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:04.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.239+0000 7fc02280f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc01c19d280 con 0x7fc01c100750 2024-04-22T23:38:04.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.239+0000 7fc02280f700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc01c196730 con 0x7fc01c0ffda0 2024-04-22T23:38:04.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.239+0000 7fc01b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc01c100750 0x7fc01c19cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:04.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.239+0000 7fc01b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc01c100750 0x7fc01c19cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59290/0 (socket says 172.21.15.67:59290) 2024-04-22T23:38:04.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.239+0000 7fc01b7fe700 1 -- 172.21.15.67:0/3152094119 learned_addr learned my addr 172.21.15.67:0/3152094119 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:38:04.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.239+0000 7fc01bfff700 1 --2- 172.21.15.67:0/3152094119 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc01c0ffda0 0x7fc01c19c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:04.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.239+0000 7fc01b7fe700 1 -- 172.21.15.67:0/3152094119 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc01c0ffda0 msgr2=0x7fc01c19c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:04.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.239+0000 7fc01b7fe700 1 --2- 172.21.15.67:0/3152094119 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc01c0ffda0 0x7fc01c19c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:04.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.239+0000 7fc01b7fe700 1 -- 172.21.15.67:0/3152094119 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc00c0095f0 con 0x7fc01c100750 2024-04-22T23:38:04.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.239+0000 7fc01b7fe700 1 --2- 172.21.15.67:0/3152094119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc01c100750 0x7fc01c19cbf0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc00c00bda0 tx=0x7fc00c00bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:38:04.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.239+0000 7fc0197fa700 1 -- 172.21.15.67:0/3152094119 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc00c01d070 con 0x7fc01c100750 2024-04-22T23:38:04.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.240+0000 7fc0197fa700 1 -- 172.21.15.67:0/3152094119 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc00c005dc0 con 0x7fc01c100750 2024-04-22T23:38:04.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.240+0000 7fc0197fa700 1 -- 172.21.15.67:0/3152094119 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc00c017400 con 0x7fc01c100750 2024-04-22T23:38:04.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.240+0000 7fc02280f700 1 -- 172.21.15.67:0/3152094119 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc01c1969b0 con 0x7fc01c100750 2024-04-22T23:38:04.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.240+0000 7fc02280f700 1 -- 172.21.15.67:0/3152094119 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc01c196e50 con 0x7fc01c100750 2024-04-22T23:38:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.241+0000 7fc02280f700 1 -- 172.21.15.67:0/3152094119 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc01c04f2e0 con 0x7fc01c100750 2024-04-22T23:38:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.241+0000 7fc0197fa700 1 -- 172.21.15.67:0/3152094119 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc00c00f6f0 con 0x7fc01c100750 2024-04-22T23:38:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.242+0000 7fc0197fa700 1 --2- 172.21.15.67:0/3152094119 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc00406ecc0 0x7fc004071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.242+0000 7fc0197fa700 1 -- 172.21.15.67:0/3152094119 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fc00c090360 con 0x7fc01c100750 2024-04-22T23:38:04.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.242+0000 7fc01bfff700 1 --2- 172.21.15.67:0/3152094119 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc00406ecc0 0x7fc004071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:04.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.242+0000 7fc01bfff700 1 --2- 172.21.15.67:0/3152094119 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc00406ecc0 0x7fc004071180 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc010005950 tx=0x7fc01000b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:38:04.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:03 smithi134 ceph-mon[85630]: pgmap v350: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:04.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.247+0000 7fc0197fa700 1 -- 172.21.15.67:0/3152094119 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc00c05cad0 con 0x7fc01c100750 2024-04-22T23:38:04.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.504+0000 7fc02280f700 1 -- 172.21.15.67:0/3152094119 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc01c197c40 con 0x7fc01c100750 2024-04-22T23:38:04.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.505+0000 7fc0197fa700 1 -- 172.21.15.67:0/3152094119 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1871 (secure 0 0 0) 0x7fc00c05c660 con 0x7fc01c100750 2024-04-22T23:38:04.499 INFO:teuthology.orchestra.run.smithi067.stdout:e31 2024-04-22T23:38:04.499 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:38:04.499 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:38:04.499 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:38:04.499 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:38:04.499 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:38:04.500 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:38:04.500 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:38:04.500 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:38:04.500 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:38:04.500 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:38:04.500 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:38:04.500 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:38:04.500 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:38:04.500 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:38:04.500 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:38:04.500 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:38:04.500 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:38:04.500 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:38:04.500 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:38:04.500 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:38:04.501 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:38:04.501 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:38:04.501 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:38:04.501 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:38:04.501 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:38:04.501 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:38:04.501 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:38:04.501 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:38:04.501 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:38:04.501 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:38:04.501 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:38:04.501 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:38:04.501 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:38:04.501 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:38:04.501 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:38:04.502 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:38:04.502 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:38:04.502 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:38:04.502 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:38:04.502 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44259} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1827288036,v1:172.21.15.134:6827/1827288036] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:38:04.502 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:38:04.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.507+0000 7fc02280f700 1 -- 172.21.15.67:0/3152094119 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc00406ecc0 msgr2=0x7fc004071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:04.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.507+0000 7fc02280f700 1 --2- 172.21.15.67:0/3152094119 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc00406ecc0 0x7fc004071180 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc010005950 tx=0x7fc01000b410 comp rx=0 tx=0).stop 2024-04-22T23:38:04.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.507+0000 7fc02280f700 1 -- 172.21.15.67:0/3152094119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc01c100750 msgr2=0x7fc01c19cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:04.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.507+0000 7fc02280f700 1 --2- 172.21.15.67:0/3152094119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc01c100750 0x7fc01c19cbf0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc00c00bda0 tx=0x7fc00c00bdd0 comp rx=0 tx=0).stop 2024-04-22T23:38:04.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.507+0000 7fc02280f700 1 -- 172.21.15.67:0/3152094119 shutdown_connections 2024-04-22T23:38:04.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.507+0000 7fc02280f700 1 --2- 172.21.15.67:0/3152094119 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc00406ecc0 0x7fc004071180 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:04.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.507+0000 7fc02280f700 1 --2- 172.21.15.67:0/3152094119 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc01c0ffda0 0x7fc01c19c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:04.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.507+0000 7fc02280f700 1 --2- 172.21.15.67:0/3152094119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc01c100750 0x7fc01c19cbf0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:04.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.507+0000 7fc02280f700 1 -- 172.21.15.67:0/3152094119 >> 172.21.15.67:0/3152094119 conn(0x7fc01c0fb650 msgr2=0x7fc01c0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:38:04.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.508+0000 7fc02280f700 1 -- 172.21.15.67:0/3152094119 shutdown_connections 2024-04-22T23:38:04.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.508+0000 7fc02280f700 1 -- 172.21.15.67:0/3152094119 wait complete. 2024-04-22T23:38:04.504 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 31 2024-04-22T23:38:04.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.617+0000 7f88a02d7700 1 -- 172.21.15.67:0/2560132198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88980ffb40 msgr2=0x7f88980fffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.617+0000 7f88a02d7700 1 --2- 172.21.15.67:0/2560132198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88980ffb40 0x7f88980fffc0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8888009910 tx=0x7f8888009c20 comp rx=0 tx=0).stop 2024-04-22T23:38:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.617+0000 7f88a02d7700 1 -- 172.21.15.67:0/2560132198 shutdown_connections 2024-04-22T23:38:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.617+0000 7f88a02d7700 1 --2- 172.21.15.67:0/2560132198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88980ffb40 0x7f88980fffc0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.617+0000 7f88a02d7700 1 --2- 172.21.15.67:0/2560132198 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8898105bf0 0x7f8898105fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.617+0000 7f88a02d7700 1 -- 172.21.15.67:0/2560132198 >> 172.21.15.67:0/2560132198 conn(0x7f88980fb610 msgr2=0x7f88980fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:38:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.617+0000 7f88a02d7700 1 -- 172.21.15.67:0/2560132198 shutdown_connections 2024-04-22T23:38:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.617+0000 7f88a02d7700 1 -- 172.21.15.67:0/2560132198 wait complete. 2024-04-22T23:38:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.617+0000 7f88a02d7700 1 Processor -- start 2024-04-22T23:38:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.617+0000 7f88a02d7700 1 -- start start 2024-04-22T23:38:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.618+0000 7f88a02d7700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88980ffb40 0x7f88981017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.618+0000 7f88a02d7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8898105bf0 0x7f8898101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.618+0000 7f88a02d7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8898102230 con 0x7f8898105bf0 2024-04-22T23:38:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.618+0000 7f88a02d7700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8898102370 con 0x7f88980ffb40 2024-04-22T23:38:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.618+0000 7f889d872700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8898105bf0 0x7f8898101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.618+0000 7f889d872700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8898105bf0 0x7f8898101cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59310/0 (socket says 172.21.15.67:59310) 2024-04-22T23:38:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.618+0000 7f889d872700 1 -- 172.21.15.67:0/2286753625 learned_addr learned my addr 172.21.15.67:0/2286753625 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:38:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.618+0000 7f889e073700 1 --2- 172.21.15.67:0/2286753625 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88980ffb40 0x7f88981017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.618+0000 7f889d872700 1 -- 172.21.15.67:0/2286753625 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88980ffb40 msgr2=0x7f88981017b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.618+0000 7f889d872700 1 --2- 172.21.15.67:0/2286753625 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88980ffb40 0x7f88981017b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.618+0000 7f889d872700 1 -- 172.21.15.67:0/2286753625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88880095f0 con 0x7f8898105bf0 2024-04-22T23:38:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.618+0000 7f889d872700 1 --2- 172.21.15.67:0/2286753625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8898105bf0 0x7f8898101cf0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8888006010 tx=0x7f888800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:38:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.618+0000 7f888f7fe700 1 -- 172.21.15.67:0/2286753625 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f888801d070 con 0x7f8898105bf0 2024-04-22T23:38:04.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.618+0000 7f888f7fe700 1 -- 172.21.15.67:0/2286753625 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8888005dc0 con 0x7f8898105bf0 2024-04-22T23:38:04.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.618+0000 7f88a02d7700 1 -- 172.21.15.67:0/2286753625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88981025f0 con 0x7f8898105bf0 2024-04-22T23:38:04.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.619+0000 7f888f7fe700 1 -- 172.21.15.67:0/2286753625 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8888017400 con 0x7f8898105bf0 2024-04-22T23:38:04.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.619+0000 7f88a02d7700 1 -- 172.21.15.67:0/2286753625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88981a2d10 con 0x7f8898105bf0 2024-04-22T23:38:04.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.620+0000 7f88a02d7700 1 -- 172.21.15.67:0/2286753625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88980679c0 con 0x7f8898105bf0 2024-04-22T23:38:04.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.620+0000 7f888f7fe700 1 -- 172.21.15.67:0/2286753625 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8888003a60 con 0x7f8898105bf0 2024-04-22T23:38:04.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.620+0000 7f888f7fe700 1 --2- 172.21.15.67:0/2286753625 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f888406ecc0 0x7f8884071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:04.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.621+0000 7f889e073700 1 --2- 172.21.15.67:0/2286753625 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f888406ecc0 0x7f8884071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:04.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.621+0000 7f888f7fe700 1 -- 172.21.15.67:0/2286753625 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f8888090470 con 0x7f8898105bf0 2024-04-22T23:38:04.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.621+0000 7f889e073700 1 --2- 172.21.15.67:0/2286753625 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f888406ecc0 0x7f8884071180 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8894005950 tx=0x7f889400b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:38:04.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.625+0000 7f888f7fe700 1 -- 172.21.15.67:0/2286753625 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f88880590b0 con 0x7f8898105bf0 2024-04-22T23:38:04.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.855+0000 7f88a02d7700 1 -- 172.21.15.67:0/2286753625 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f88980695a0 con 0x7f888406ecc0 2024-04-22T23:38:04.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.857+0000 7f888f7fe700 1 -- 172.21.15.67:0/2286753625 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f88980695a0 con 0x7f888406ecc0 2024-04-22T23:38:04.851 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:38:04.851 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:38:04.851 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:38:04.851 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:38:04.851 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:38:04.851 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:38:04.851 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:38:04.851 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:38:04.851 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:38:04.851 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:38:04.851 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T23:38:04.852 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:38:04.852 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:38:04.852 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:38:04.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.859+0000 7f88a02d7700 1 -- 172.21.15.67:0/2286753625 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f888406ecc0 msgr2=0x7f8884071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:04.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.859+0000 7f88a02d7700 1 --2- 172.21.15.67:0/2286753625 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f888406ecc0 0x7f8884071180 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8894005950 tx=0x7f889400b410 comp rx=0 tx=0).stop 2024-04-22T23:38:04.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.859+0000 7f88a02d7700 1 -- 172.21.15.67:0/2286753625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8898105bf0 msgr2=0x7f8898101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:04.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.859+0000 7f88a02d7700 1 --2- 172.21.15.67:0/2286753625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8898105bf0 0x7f8898101cf0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8888006010 tx=0x7f888800bd70 comp rx=0 tx=0).stop 2024-04-22T23:38:04.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.859+0000 7f88a02d7700 1 -- 172.21.15.67:0/2286753625 shutdown_connections 2024-04-22T23:38:04.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.859+0000 7f88a02d7700 1 --2- 172.21.15.67:0/2286753625 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f888406ecc0 0x7f8884071180 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:04.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.859+0000 7f88a02d7700 1 --2- 172.21.15.67:0/2286753625 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f88980ffb40 0x7f88981017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:04.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.859+0000 7f88a02d7700 1 --2- 172.21.15.67:0/2286753625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8898105bf0 0x7f8898101cf0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:04.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.859+0000 7f88a02d7700 1 -- 172.21.15.67:0/2286753625 >> 172.21.15.67:0/2286753625 conn(0x7f88980fb610 msgr2=0x7f8898068e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:38:04.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.859+0000 7f88a02d7700 1 -- 172.21.15.67:0/2286753625 shutdown_connections 2024-04-22T23:38:04.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.859+0000 7f88a02d7700 1 -- 172.21.15.67:0/2286753625 wait complete. 2024-04-22T23:38:04.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.966+0000 7f5071f4a700 1 -- 172.21.15.67:0/439508216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f506c105bd0 msgr2=0x7f506c105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:04.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.966+0000 7f5071f4a700 1 --2- 172.21.15.67:0/439508216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f506c105bd0 0x7f506c105fb0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f505c009910 tx=0x7f505c009c20 comp rx=0 tx=0).stop 2024-04-22T23:38:04.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.966+0000 7f5071f4a700 1 -- 172.21.15.67:0/439508216 shutdown_connections 2024-04-22T23:38:04.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.966+0000 7f5071f4a700 1 --2- 172.21.15.67:0/439508216 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f506c0ffc20 0x7f506c100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:04.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.966+0000 7f5071f4a700 1 --2- 172.21.15.67:0/439508216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f506c105bd0 0x7f506c105fb0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:04.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.966+0000 7f5071f4a700 1 -- 172.21.15.67:0/439508216 >> 172.21.15.67:0/439508216 conn(0x7f506c075670 msgr2=0x7f506c075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:38:04.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.966+0000 7f5071f4a700 1 -- 172.21.15.67:0/439508216 shutdown_connections 2024-04-22T23:38:04.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.966+0000 7f5071f4a700 1 -- 172.21.15.67:0/439508216 wait complete. 2024-04-22T23:38:04.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.967+0000 7f5071f4a700 1 Processor -- start 2024-04-22T23:38:04.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.967+0000 7f5071f4a700 1 -- start start 2024-04-22T23:38:04.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.967+0000 7f5071f4a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f506c0ffc20 0x7f506c1976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:04.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.967+0000 7f5071f4a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f506c105bd0 0x7f506c197c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:04.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.967+0000 7f5071f4a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f506c1981e0 con 0x7f506c0ffc20 2024-04-22T23:38:04.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.967+0000 7f5071f4a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f506c198320 con 0x7f506c105bd0 2024-04-22T23:38:04.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.967+0000 7f506b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f506c0ffc20 0x7f506c1976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:04.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.967+0000 7f506b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f506c0ffc20 0x7f506c1976d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59330/0 (socket says 172.21.15.67:59330) 2024-04-22T23:38:04.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.967+0000 7f506b7fe700 1 -- 172.21.15.67:0/1372594728 learned_addr learned my addr 172.21.15.67:0/1372594728 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:38:04.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.967+0000 7f5062ffd700 1 --2- 172.21.15.67:0/1372594728 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f506c105bd0 0x7f506c197c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:04.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.968+0000 7f506b7fe700 1 -- 172.21.15.67:0/1372594728 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f506c105bd0 msgr2=0x7f506c197c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:04.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.968+0000 7f506b7fe700 1 --2- 172.21.15.67:0/1372594728 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f506c105bd0 0x7f506c197c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:04.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.968+0000 7f506b7fe700 1 -- 172.21.15.67:0/1372594728 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f505c0095f0 con 0x7f506c0ffc20 2024-04-22T23:38:04.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.968+0000 7f506b7fe700 1 --2- 172.21.15.67:0/1372594728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f506c0ffc20 0x7f506c1976d0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f505c005f50 tx=0x7f505c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:38:04.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.968+0000 7f50697fa700 1 -- 172.21.15.67:0/1372594728 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f505c01d070 con 0x7f506c0ffc20 2024-04-22T23:38:04.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.968+0000 7f5071f4a700 1 -- 172.21.15.67:0/1372594728 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f506c1a0740 con 0x7f506c0ffc20 2024-04-22T23:38:04.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.968+0000 7f50697fa700 1 -- 172.21.15.67:0/1372594728 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f505c00b610 con 0x7f506c0ffc20 2024-04-22T23:38:04.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.968+0000 7f50697fa700 1 -- 172.21.15.67:0/1372594728 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f505c022470 con 0x7f506c0ffc20 2024-04-22T23:38:04.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.968+0000 7f5071f4a700 1 -- 172.21.15.67:0/1372594728 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f506c1a0af0 con 0x7f506c0ffc20 2024-04-22T23:38:04.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.969+0000 7f5071f4a700 1 -- 172.21.15.67:0/1372594728 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f506c1080f0 con 0x7f506c0ffc20 2024-04-22T23:38:04.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.970+0000 7f50697fa700 1 -- 172.21.15.67:0/1372594728 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f505c003780 con 0x7f506c0ffc20 2024-04-22T23:38:04.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.970+0000 7f50697fa700 1 --2- 172.21.15.67:0/1372594728 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f505806ecc0 0x7f5058071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:04.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.970+0000 7f50697fa700 1 -- 172.21.15.67:0/1372594728 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f505c090280 con 0x7f506c0ffc20 2024-04-22T23:38:04.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.970+0000 7f5062ffd700 1 --2- 172.21.15.67:0/1372594728 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f505806ecc0 0x7f5058071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:04.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.971+0000 7f5062ffd700 1 --2- 172.21.15.67:0/1372594728 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f505806ecc0 0x7f5058071180 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f506c068dc0 tx=0x7f5054009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:38:04.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:04.975+0000 7f50697fa700 1 -- 172.21.15.67:0/1372594728 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f505c058f40 con 0x7f506c0ffc20 2024-04-22T23:38:05.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:04 smithi134 ceph-mon[85630]: from='client.34680 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:38:05.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:04 smithi134 ceph-mon[85630]: from='client.34684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:38:05.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:04 smithi134 ceph-mon[85630]: from='client.34688 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:38:05.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:04 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2960350373' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:38:05.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:04 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3152094119' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:38:05.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:05.271+0000 7f5071f4a700 1 -- 172.21.15.67:0/1372594728 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f506c04f2e0 con 0x7f506c0ffc20 2024-04-22T23:38:05.265 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:04 smithi067 ceph-mon[112438]: from='client.34680 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:38:05.265 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:04 smithi067 ceph-mon[112438]: from='client.34684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:38:05.265 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:04 smithi067 ceph-mon[112438]: from='client.34688 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:38:05.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:04 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2960350373' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:38:05.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:04 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3152094119' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:38:05.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:05.272+0000 7f50697fa700 1 -- 172.21.15.67:0/1372594728 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f505c05c600 con 0x7f506c0ffc20 2024-04-22T23:38:05.266 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:38:05.266 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:38:05.266 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:38:05.266 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:38:05.266 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:38:05.267 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:38:05.267 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:38:05.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:05.274+0000 7f5071f4a700 1 -- 172.21.15.67:0/1372594728 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f505806ecc0 msgr2=0x7f5058071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:05.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:05.274+0000 7f5071f4a700 1 --2- 172.21.15.67:0/1372594728 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f505806ecc0 0x7f5058071180 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f506c068dc0 tx=0x7f5054009260 comp rx=0 tx=0).stop 2024-04-22T23:38:05.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:05.274+0000 7f5071f4a700 1 -- 172.21.15.67:0/1372594728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f506c0ffc20 msgr2=0x7f506c1976d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:05.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:05.274+0000 7f5071f4a700 1 --2- 172.21.15.67:0/1372594728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f506c0ffc20 0x7f506c1976d0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f505c005f50 tx=0x7f505c00f690 comp rx=0 tx=0).stop 2024-04-22T23:38:05.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:05.274+0000 7f5071f4a700 1 -- 172.21.15.67:0/1372594728 shutdown_connections 2024-04-22T23:38:05.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:05.274+0000 7f5071f4a700 1 --2- 172.21.15.67:0/1372594728 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f505806ecc0 0x7f5058071180 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:05.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:05.274+0000 7f5071f4a700 1 --2- 172.21.15.67:0/1372594728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f506c0ffc20 0x7f506c1976d0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:05.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:05.274+0000 7f5071f4a700 1 --2- 172.21.15.67:0/1372594728 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f506c105bd0 0x7f506c197c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:05.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:05.274+0000 7f5071f4a700 1 -- 172.21.15.67:0/1372594728 >> 172.21.15.67:0/1372594728 conn(0x7f506c075670 msgr2=0x7f506c0feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:38:05.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:05.274+0000 7f5071f4a700 1 -- 172.21.15.67:0/1372594728 shutdown_connections 2024-04-22T23:38:05.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:05.274+0000 7f5071f4a700 1 -- 172.21.15.67:0/1372594728 wait complete. 2024-04-22T23:38:06.238 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:05 smithi134 ceph-mon[85630]: pgmap v351: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:06.238 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:05 smithi134 ceph-mon[85630]: from='client.34700 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:38:06.238 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:05 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1372594728' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:38:06.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:05 smithi067 ceph-mon[112438]: pgmap v351: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:06.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:05 smithi067 ceph-mon[112438]: from='client.34700 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:38:06.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:05 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1372594728' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:38:07.237 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:06 smithi134 ceph-mon[85630]: pgmap v352: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:07.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:06 smithi067 ceph-mon[112438]: pgmap v352: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:09.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:09 smithi134 ceph-mon[85630]: pgmap v353: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:09.537 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:09 smithi067 ceph-mon[112438]: pgmap v353: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:11.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:11 smithi067 ceph-mon[112438]: pgmap v354: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:11.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:38:11.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:11 smithi134 ceph-mon[85630]: pgmap v354: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:11.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:38:13.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:13 smithi134 ceph-mon[85630]: pgmap v355: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:13.537 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:13 smithi067 ceph-mon[112438]: pgmap v355: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:15.440 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:15 smithi134 ceph-mon[85630]: pgmap v356: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:15.537 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:15 smithi067 ceph-mon[112438]: pgmap v356: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:17.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:17 smithi134 ceph-mon[85630]: pgmap v357: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:17.538 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:17 smithi067 ceph-mon[112438]: pgmap v357: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:19.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:19 smithi134 ceph-mon[85630]: pgmap v358: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:19.538 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:19 smithi067 ceph-mon[112438]: pgmap v358: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:21.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:21 smithi067 ceph-mon[112438]: pgmap v359: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:21.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:21 smithi134 ceph-mon[85630]: pgmap v359: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:23.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:23 smithi134 ceph-mon[85630]: pgmap v360: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:23.537 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:23 smithi067 ceph-mon[112438]: pgmap v360: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:25.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:25 smithi134 ceph-mon[85630]: pgmap v361: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:25.537 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:25 smithi067 ceph-mon[112438]: pgmap v361: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:26.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:38:26.537 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:38:27.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:27 smithi134 ceph-mon[85630]: pgmap v362: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:27.537 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:27 smithi067 ceph-mon[112438]: pgmap v362: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:29.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:29 smithi134 ceph-mon[85630]: pgmap v363: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:29.537 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:29 smithi067 ceph-mon[112438]: pgmap v363: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:31.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:31 smithi134 ceph-mon[85630]: pgmap v364: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:31.537 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:31 smithi067 ceph-mon[112438]: pgmap v364: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:33.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:33 smithi134 ceph-mon[85630]: pgmap v365: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:33.537 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:33 smithi067 ceph-mon[112438]: pgmap v365: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:35.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.396+0000 7fd9b4a17700 1 -- 172.21.15.67:0/3519724163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9b0100750 msgr2=0x7fd9b01047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:35.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.396+0000 7fd9b4a17700 1 --2- 172.21.15.67:0/3519724163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9b0100750 0x7fd9b01047a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4009910 tx=0x7fd9a4009c20 comp rx=0 tx=0).stop 2024-04-22T23:38:35.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.396+0000 7fd9b4a17700 1 -- 172.21.15.67:0/3519724163 shutdown_connections 2024-04-22T23:38:35.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.396+0000 7fd9b4a17700 1 --2- 172.21.15.67:0/3519724163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9b0100750 0x7fd9b01047a0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:35.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.396+0000 7fd9b4a17700 1 --2- 172.21.15.67:0/3519724163 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd9b00ffda0 0x7fd9b0100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:35.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.396+0000 7fd9b4a17700 1 -- 172.21.15.67:0/3519724163 >> 172.21.15.67:0/3519724163 conn(0x7fd9b00fb650 msgr2=0x7fd9b00fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:38:35.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.396+0000 7fd9b4a17700 1 -- 172.21.15.67:0/3519724163 shutdown_connections 2024-04-22T23:38:35.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.396+0000 7fd9b4a17700 1 -- 172.21.15.67:0/3519724163 wait complete. 2024-04-22T23:38:35.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.397+0000 7fd9b4a17700 1 Processor -- start 2024-04-22T23:38:35.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.397+0000 7fd9b4a17700 1 -- start start 2024-04-22T23:38:35.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.397+0000 7fd9b4a17700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9b00ffda0 0x7fd9b01969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:35.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.397+0000 7fd9b4a17700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd9b0100750 0x7fd9b0196ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:35.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.397+0000 7fd9b4a17700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b0197570 con 0x7fd9b00ffda0 2024-04-22T23:38:35.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.397+0000 7fd9b4a17700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b019b380 con 0x7fd9b0100750 2024-04-22T23:38:35.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.397+0000 7fd9ae59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9b00ffda0 0x7fd9b01969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:35.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.397+0000 7fd9ae59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9b00ffda0 0x7fd9b01969a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43134/0 (socket says 172.21.15.67:43134) 2024-04-22T23:38:35.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.397+0000 7fd9ae59c700 1 -- 172.21.15.67:0/2244118133 learned_addr learned my addr 172.21.15.67:0/2244118133 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:38:35.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.397+0000 7fd9add9b700 1 --2- 172.21.15.67:0/2244118133 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd9b0100750 0x7fd9b0196ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:35.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.397+0000 7fd9ae59c700 1 -- 172.21.15.67:0/2244118133 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd9b0100750 msgr2=0x7fd9b0196ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:35.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.397+0000 7fd9ae59c700 1 --2- 172.21.15.67:0/2244118133 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd9b0100750 0x7fd9b0196ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:35.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.398+0000 7fd9ae59c700 1 -- 172.21.15.67:0/2244118133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a40095f0 con 0x7fd9b00ffda0 2024-04-22T23:38:35.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.398+0000 7fd9add9b700 1 --2- 172.21.15.67:0/2244118133 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd9b0100750 0x7fd9b0196ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:38:35.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.398+0000 7fd9ae59c700 1 --2- 172.21.15.67:0/2244118133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9b00ffda0 0x7fd9b01969a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd9a000b4b0 tx=0x7fd9a000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:38:35.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.398+0000 7fd99f7fe700 1 -- 172.21.15.67:0/2244118133 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd9a0011800 con 0x7fd9b00ffda0 2024-04-22T23:38:35.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.398+0000 7fd9b4a17700 1 -- 172.21.15.67:0/2244118133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9b019b660 con 0x7fd9b00ffda0 2024-04-22T23:38:35.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.398+0000 7fd99f7fe700 1 -- 172.21.15.67:0/2244118133 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd9a0011e20 con 0x7fd9b00ffda0 2024-04-22T23:38:35.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.398+0000 7fd99f7fe700 1 -- 172.21.15.67:0/2244118133 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd9a000f2e0 con 0x7fd9b00ffda0 2024-04-22T23:38:35.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.398+0000 7fd9b4a17700 1 -- 172.21.15.67:0/2244118133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9b019bb80 con 0x7fd9b00ffda0 2024-04-22T23:38:35.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.399+0000 7fd9b4a17700 1 -- 172.21.15.67:0/2244118133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9b004f2e0 con 0x7fd9b00ffda0 2024-04-22T23:38:35.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.400+0000 7fd99f7fe700 1 -- 172.21.15.67:0/2244118133 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd9a0010380 con 0x7fd9b00ffda0 2024-04-22T23:38:35.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.400+0000 7fd99f7fe700 1 --2- 172.21.15.67:0/2244118133 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd99806ecc0 0x7fd998071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:35.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.400+0000 7fd99f7fe700 1 -- 172.21.15.67:0/2244118133 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fd9a008eb70 con 0x7fd9b00ffda0 2024-04-22T23:38:35.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.400+0000 7fd9add9b700 1 --2- 172.21.15.67:0/2244118133 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd99806ecc0 0x7fd998071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:35.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.401+0000 7fd9add9b700 1 --2- 172.21.15.67:0/2244118133 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd99806ecc0 0x7fd998071180 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd9b0197910 tx=0x7fd9a401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:38:35.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.405+0000 7fd99f7fe700 1 -- 172.21.15.67:0/2244118133 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fd9a005b2e0 con 0x7fd9b00ffda0 2024-04-22T23:38:35.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:35 smithi134 ceph-mon[85630]: pgmap v366: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:35.537 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:35 smithi067 ceph-mon[112438]: pgmap v366: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:35.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.637+0000 7fd9b4a17700 1 -- 172.21.15.67:0/2244118133 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd9b019be70 con 0x7fd99806ecc0 2024-04-22T23:38:35.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.638+0000 7fd99f7fe700 1 -- 172.21.15.67:0/2244118133 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fd9b019be70 con 0x7fd99806ecc0 2024-04-22T23:38:35.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.640+0000 7fd9b4a17700 1 -- 172.21.15.67:0/2244118133 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd99806ecc0 msgr2=0x7fd998071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:35.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.640+0000 7fd9b4a17700 1 --2- 172.21.15.67:0/2244118133 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd99806ecc0 0x7fd998071180 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd9b0197910 tx=0x7fd9a401a040 comp rx=0 tx=0).stop 2024-04-22T23:38:35.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.640+0000 7fd9b4a17700 1 -- 172.21.15.67:0/2244118133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9b00ffda0 msgr2=0x7fd9b01969a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:35.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.640+0000 7fd9b4a17700 1 --2- 172.21.15.67:0/2244118133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9b00ffda0 0x7fd9b01969a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd9a000b4b0 tx=0x7fd9a000b870 comp rx=0 tx=0).stop 2024-04-22T23:38:35.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.640+0000 7fd9b4a17700 1 -- 172.21.15.67:0/2244118133 shutdown_connections 2024-04-22T23:38:35.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.640+0000 7fd9b4a17700 1 --2- 172.21.15.67:0/2244118133 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd99806ecc0 0x7fd998071180 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:35.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.640+0000 7fd9b4a17700 1 --2- 172.21.15.67:0/2244118133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9b00ffda0 0x7fd9b01969a0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:35.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.640+0000 7fd9b4a17700 1 --2- 172.21.15.67:0/2244118133 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd9b0100750 0x7fd9b0196ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:35.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.640+0000 7fd9b4a17700 1 -- 172.21.15.67:0/2244118133 >> 172.21.15.67:0/2244118133 conn(0x7fd9b00fb650 msgr2=0x7fd9b0103fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:38:35.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.640+0000 7fd9b4a17700 1 -- 172.21.15.67:0/2244118133 shutdown_connections 2024-04-22T23:38:35.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.640+0000 7fd9b4a17700 1 -- 172.21.15.67:0/2244118133 wait complete. 2024-04-22T23:38:35.645 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:38:35.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.749+0000 7f45624f8700 1 -- 172.21.15.67:0/1471367281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f455c101850 msgr2=0x7f455c1058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:35.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.749+0000 7f45624f8700 1 --2- 172.21.15.67:0/1471367281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f455c101850 0x7f455c1058a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f4550009910 tx=0x7f4550009c20 comp rx=0 tx=0).stop 2024-04-22T23:38:35.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.749+0000 7f45624f8700 1 -- 172.21.15.67:0/1471367281 shutdown_connections 2024-04-22T23:38:35.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.749+0000 7f45624f8700 1 --2- 172.21.15.67:0/1471367281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f455c101850 0x7f455c1058a0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:35.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.749+0000 7f45624f8700 1 --2- 172.21.15.67:0/1471367281 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f455c100ea0 0x7f455c101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:35.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.749+0000 7f45624f8700 1 -- 172.21.15.67:0/1471367281 >> 172.21.15.67:0/1471367281 conn(0x7f455c0fc750 msgr2=0x7f455c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:38:35.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.749+0000 7f45624f8700 1 -- 172.21.15.67:0/1471367281 shutdown_connections 2024-04-22T23:38:35.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.749+0000 7f45624f8700 1 -- 172.21.15.67:0/1471367281 wait complete. 2024-04-22T23:38:35.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.749+0000 7f45624f8700 1 Processor -- start 2024-04-22T23:38:35.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.749+0000 7f45624f8700 1 -- start start 2024-04-22T23:38:35.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.750+0000 7f45624f8700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f455c100ea0 0x7f455c073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:35.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.750+0000 7f45624f8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f455c101850 0x7f455c0735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:35.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.750+0000 7f45624f8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f455c073b10 con 0x7f455c101850 2024-04-22T23:38:35.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.750+0000 7f45624f8700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f455c073c50 con 0x7f455c100ea0 2024-04-22T23:38:35.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.750+0000 7f455b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f455c101850 0x7f455c0735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:35.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.750+0000 7f455b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f455c101850 0x7f455c0735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43142/0 (socket says 172.21.15.67:43142) 2024-04-22T23:38:35.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.750+0000 7f455b7fe700 1 -- 172.21.15.67:0/3589797384 learned_addr learned my addr 172.21.15.67:0/3589797384 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:38:35.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.750+0000 7f455bfff700 1 --2- 172.21.15.67:0/3589797384 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f455c100ea0 0x7f455c073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:35.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.750+0000 7f455b7fe700 1 -- 172.21.15.67:0/3589797384 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f455c100ea0 msgr2=0x7f455c073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:35.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.750+0000 7f455b7fe700 1 --2- 172.21.15.67:0/3589797384 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f455c100ea0 0x7f455c073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:35.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.750+0000 7f455b7fe700 1 -- 172.21.15.67:0/3589797384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45500095f0 con 0x7f455c101850 2024-04-22T23:38:35.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.750+0000 7f455b7fe700 1 --2- 172.21.15.67:0/3589797384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f455c101850 0x7f455c0735d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f45500092e0 tx=0x7f455000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:38:35.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.751+0000 7f45597fa700 1 -- 172.21.15.67:0/3589797384 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f455001d070 con 0x7f455c101850 2024-04-22T23:38:35.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.751+0000 7f45597fa700 1 -- 172.21.15.67:0/3589797384 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f455000f460 con 0x7f455c101850 2024-04-22T23:38:35.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.751+0000 7f45624f8700 1 -- 172.21.15.67:0/3589797384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f455c073ed0 con 0x7f455c101850 2024-04-22T23:38:35.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.751+0000 7f45597fa700 1 -- 172.21.15.67:0/3589797384 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45500175f0 con 0x7f455c101850 2024-04-22T23:38:35.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.751+0000 7f45624f8700 1 -- 172.21.15.67:0/3589797384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f455c1a2ec0 con 0x7f455c101850 2024-04-22T23:38:35.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.752+0000 7f45624f8700 1 -- 172.21.15.67:0/3589797384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f455c1091f0 con 0x7f455c101850 2024-04-22T23:38:35.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.752+0000 7f45597fa700 1 -- 172.21.15.67:0/3589797384 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f455000f5d0 con 0x7f455c101850 2024-04-22T23:38:35.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.752+0000 7f45597fa700 1 --2- 172.21.15.67:0/3589797384 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f454406ecc0 0x7f4544071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:35.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.753+0000 7f45597fa700 1 -- 172.21.15.67:0/3589797384 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f4550090440 con 0x7f455c101850 2024-04-22T23:38:35.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.753+0000 7f455bfff700 1 --2- 172.21.15.67:0/3589797384 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f454406ecc0 0x7f4544071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:35.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.753+0000 7f455bfff700 1 --2- 172.21.15.67:0/3589797384 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f454406ecc0 0x7f4544071180 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f454c006fd0 tx=0x7f454c008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:38:35.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.757+0000 7f45597fa700 1 -- 172.21.15.67:0/3589797384 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f4550059100 con 0x7f455c101850 2024-04-22T23:38:35.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.987+0000 7f45624f8700 1 -- 172.21.15.67:0/3589797384 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f455c0746d0 con 0x7f454406ecc0 2024-04-22T23:38:35.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.988+0000 7f45597fa700 1 -- 172.21.15.67:0/3589797384 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f455c0746d0 con 0x7f454406ecc0 2024-04-22T23:38:35.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.991+0000 7f45624f8700 1 -- 172.21.15.67:0/3589797384 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f454406ecc0 msgr2=0x7f4544071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:35.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.991+0000 7f45624f8700 1 --2- 172.21.15.67:0/3589797384 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f454406ecc0 0x7f4544071180 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f454c006fd0 tx=0x7f454c008040 comp rx=0 tx=0).stop 2024-04-22T23:38:35.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.991+0000 7f45624f8700 1 -- 172.21.15.67:0/3589797384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f455c101850 msgr2=0x7f455c0735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:35.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.991+0000 7f45624f8700 1 --2- 172.21.15.67:0/3589797384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f455c101850 0x7f455c0735d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f45500092e0 tx=0x7f455000be20 comp rx=0 tx=0).stop 2024-04-22T23:38:35.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.991+0000 7f45624f8700 1 -- 172.21.15.67:0/3589797384 shutdown_connections 2024-04-22T23:38:35.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.991+0000 7f45624f8700 1 --2- 172.21.15.67:0/3589797384 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f454406ecc0 0x7f4544071180 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:35.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.991+0000 7f45624f8700 1 --2- 172.21.15.67:0/3589797384 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f455c100ea0 0x7f455c073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:35.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.991+0000 7f45624f8700 1 --2- 172.21.15.67:0/3589797384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f455c101850 0x7f455c0735d0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:35.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.991+0000 7f45624f8700 1 -- 172.21.15.67:0/3589797384 >> 172.21.15.67:0/3589797384 conn(0x7f455c0fc750 msgr2=0x7f455c0fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:38:35.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.991+0000 7f45624f8700 1 -- 172.21.15.67:0/3589797384 shutdown_connections 2024-04-22T23:38:35.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:35.991+0000 7f45624f8700 1 -- 172.21.15.67:0/3589797384 wait complete. 2024-04-22T23:38:36.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.097+0000 7f1aba52c700 1 -- 172.21.15.67:0/2568661266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ab40ffb50 msgr2=0x7f1ab40fffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:36.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.097+0000 7f1aba52c700 1 --2- 172.21.15.67:0/2568661266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ab40ffb50 0x7f1ab40fffd0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1aa4009910 tx=0x7f1aa4009c20 comp rx=0 tx=0).stop 2024-04-22T23:38:36.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.097+0000 7f1aba52c700 1 -- 172.21.15.67:0/2568661266 shutdown_connections 2024-04-22T23:38:36.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.097+0000 7f1aba52c700 1 --2- 172.21.15.67:0/2568661266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ab40ffb50 0x7f1ab40fffd0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:36.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.097+0000 7f1aba52c700 1 --2- 172.21.15.67:0/2568661266 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ab4105c00 0x7f1ab4105fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:36.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.097+0000 7f1aba52c700 1 -- 172.21.15.67:0/2568661266 >> 172.21.15.67:0/2568661266 conn(0x7f1ab40fb620 msgr2=0x7f1ab40fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:38:36.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.097+0000 7f1aba52c700 1 -- 172.21.15.67:0/2568661266 shutdown_connections 2024-04-22T23:38:36.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.097+0000 7f1aba52c700 1 -- 172.21.15.67:0/2568661266 wait complete. 2024-04-22T23:38:36.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.098+0000 7f1aba52c700 1 Processor -- start 2024-04-22T23:38:36.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.098+0000 7f1aba52c700 1 -- start start 2024-04-22T23:38:36.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.098+0000 7f1aba52c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ab40ffb50 0x7f1ab419c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:36.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.098+0000 7f1aba52c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ab4105c00 0x7f1ab419cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:36.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.098+0000 7f1aba52c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab419d110 con 0x7f1ab4105c00 2024-04-22T23:38:36.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.098+0000 7f1aba52c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab4196650 con 0x7f1ab40ffb50 2024-04-22T23:38:36.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.099+0000 7f1ab37fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ab4105c00 0x7f1ab419cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:36.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.099+0000 7f1ab37fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ab4105c00 0x7f1ab419cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43154/0 (socket says 172.21.15.67:43154) 2024-04-22T23:38:36.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.099+0000 7f1ab37fe700 1 -- 172.21.15.67:0/513282922 learned_addr learned my addr 172.21.15.67:0/513282922 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:38:36.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.099+0000 7f1ab3fff700 1 --2- 172.21.15.67:0/513282922 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ab40ffb50 0x7f1ab419c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:36.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.099+0000 7f1ab37fe700 1 -- 172.21.15.67:0/513282922 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ab40ffb50 msgr2=0x7f1ab419c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:36.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.099+0000 7f1ab37fe700 1 --2- 172.21.15.67:0/513282922 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ab40ffb50 0x7f1ab419c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:36.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.099+0000 7f1ab37fe700 1 -- 172.21.15.67:0/513282922 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa40095f0 con 0x7f1ab4105c00 2024-04-22T23:38:36.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.099+0000 7f1ab37fe700 1 --2- 172.21.15.67:0/513282922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ab4105c00 0x7f1ab419cb10 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1aa4006010 tx=0x7f1aa400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:38:36.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.099+0000 7f1ab17fa700 1 -- 172.21.15.67:0/513282922 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1aa401d070 con 0x7f1ab4105c00 2024-04-22T23:38:36.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.099+0000 7f1ab17fa700 1 -- 172.21.15.67:0/513282922 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1aa400f460 con 0x7f1ab4105c00 2024-04-22T23:38:36.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.099+0000 7f1aba52c700 1 -- 172.21.15.67:0/513282922 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ab41968d0 con 0x7f1ab4105c00 2024-04-22T23:38:36.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.099+0000 7f1aba52c700 1 -- 172.21.15.67:0/513282922 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ab4196dc0 con 0x7f1ab4105c00 2024-04-22T23:38:36.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.100+0000 7f1ab17fa700 1 -- 172.21.15.67:0/513282922 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1aa4017630 con 0x7f1ab4105c00 2024-04-22T23:38:36.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.100+0000 7f1aba52c700 1 -- 172.21.15.67:0/513282922 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ab404f2e0 con 0x7f1ab4105c00 2024-04-22T23:38:36.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.101+0000 7f1ab17fa700 1 -- 172.21.15.67:0/513282922 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1aa40178a0 con 0x7f1ab4105c00 2024-04-22T23:38:36.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.101+0000 7f1ab17fa700 1 --2- 172.21.15.67:0/513282922 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1aa006ecc0 0x7f1aa0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:36.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.101+0000 7f1ab17fa700 1 -- 172.21.15.67:0/513282922 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f1aa4090450 con 0x7f1ab4105c00 2024-04-22T23:38:36.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.101+0000 7f1ab3fff700 1 --2- 172.21.15.67:0/513282922 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1aa006ecc0 0x7f1aa0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:36.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.102+0000 7f1ab3fff700 1 --2- 172.21.15.67:0/513282922 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1aa006ecc0 0x7f1aa0071180 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c00ac50 tx=0x7f1a9c00a380 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:38:36.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.106+0000 7f1ab17fa700 1 -- 172.21.15.67:0/513282922 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1aa405cc40 con 0x7f1ab4105c00 2024-04-22T23:38:36.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:36 smithi067 ceph-mon[112438]: from='client.34708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:38:36.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.325+0000 7f1aba52c700 1 -- 172.21.15.67:0/513282922 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ab41970a0 con 0x7f1aa006ecc0 2024-04-22T23:38:36.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.336+0000 7f1ab17fa700 1 -- 172.21.15.67:0/513282922 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f1ab41970a0 con 0x7f1aa006ecc0 2024-04-22T23:38:36.330 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:38:36.330 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (12m) 5m ago 23m 25.5M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:38:36.330 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (23m) 5m ago 23m 9114k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:38:36.330 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (22m) 5m ago 22m 9042k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:38:36.330 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (9m) 5m ago 23m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:38:36.330 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (9m) 5m ago 22m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:38:36.330 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (11m) 5m ago 23m 79.3M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:38:36.330 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (6m) 5m ago 18m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:38:36.331 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (5m) 5m ago 18m 12.6M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:38:36.331 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (5m) 5m ago 18m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:38:36.331 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (18m) 5m ago 18m 12.2M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:38:36.331 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (13m) 5m ago 25m 511M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:38:36.331 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (13m) 5m ago 22m 453M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:38:36.331 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (10m) 5m ago 25m 55.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:38:36.331 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (10m) 5m ago 22m 46.0M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:38:36.331 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (13m) 5m ago 23m 15.4M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:38:36.331 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (12m) 5m ago 22m 16.1M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:38:36.331 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (9m) 5m ago 21m 99.2M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:38:36.331 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (8m) 5m ago 21m 127M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:38:36.331 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (8m) 5m ago 20m 79.9M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:38:36.331 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (7m) 5m ago 20m 97.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:38:36.331 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (6m) 5m ago 20m 82.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:38:36.332 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (6m) 5m ago 19m 77.6M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:38:36.332 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (12m) 5m ago 23m 73.9M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:38:36.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.338+0000 7f1aba52c700 1 -- 172.21.15.67:0/513282922 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1aa006ecc0 msgr2=0x7f1aa0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:36.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.338+0000 7f1aba52c700 1 --2- 172.21.15.67:0/513282922 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1aa006ecc0 0x7f1aa0071180 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c00ac50 tx=0x7f1a9c00a380 comp rx=0 tx=0).stop 2024-04-22T23:38:36.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.338+0000 7f1aba52c700 1 -- 172.21.15.67:0/513282922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ab4105c00 msgr2=0x7f1ab419cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:36.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.338+0000 7f1aba52c700 1 --2- 172.21.15.67:0/513282922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ab4105c00 0x7f1ab419cb10 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1aa4006010 tx=0x7f1aa400bd70 comp rx=0 tx=0).stop 2024-04-22T23:38:36.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.338+0000 7f1aba52c700 1 -- 172.21.15.67:0/513282922 shutdown_connections 2024-04-22T23:38:36.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.338+0000 7f1aba52c700 1 --2- 172.21.15.67:0/513282922 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1aa006ecc0 0x7f1aa0071180 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:36.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.338+0000 7f1aba52c700 1 --2- 172.21.15.67:0/513282922 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ab40ffb50 0x7f1ab419c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:36.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.338+0000 7f1aba52c700 1 --2- 172.21.15.67:0/513282922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ab4105c00 0x7f1ab419cb10 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:36.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.338+0000 7f1aba52c700 1 -- 172.21.15.67:0/513282922 >> 172.21.15.67:0/513282922 conn(0x7f1ab40fb620 msgr2=0x7f1ab4068ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:38:36.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.339+0000 7f1aba52c700 1 -- 172.21.15.67:0/513282922 shutdown_connections 2024-04-22T23:38:36.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.339+0000 7f1aba52c700 1 -- 172.21.15.67:0/513282922 wait complete. 2024-04-22T23:38:36.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.444+0000 7efd31672700 1 -- 172.21.15.67:0/2353366431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd2c101de0 msgr2=0x7efd2c1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:36.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.444+0000 7efd31672700 1 --2- 172.21.15.67:0/2353366431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd2c101de0 0x7efd2c1021c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7efd1c009910 tx=0x7efd1c009c20 comp rx=0 tx=0).stop 2024-04-22T23:38:36.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.444+0000 7efd31672700 1 -- 172.21.15.67:0/2353366431 shutdown_connections 2024-04-22T23:38:36.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.444+0000 7efd31672700 1 --2- 172.21.15.67:0/2353366431 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd2c102700 0x7efd2c10abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:36.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.444+0000 7efd31672700 1 --2- 172.21.15.67:0/2353366431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd2c101de0 0x7efd2c1021c0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:36.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.444+0000 7efd31672700 1 -- 172.21.15.67:0/2353366431 >> 172.21.15.67:0/2353366431 conn(0x7efd2c0fb650 msgr2=0x7efd2c0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:38:36.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.444+0000 7efd31672700 1 -- 172.21.15.67:0/2353366431 shutdown_connections 2024-04-22T23:38:36.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.444+0000 7efd31672700 1 -- 172.21.15.67:0/2353366431 wait complete. 2024-04-22T23:38:36.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.445+0000 7efd31672700 1 Processor -- start 2024-04-22T23:38:36.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.445+0000 7efd31672700 1 -- start start 2024-04-22T23:38:36.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.446+0000 7efd31672700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd2c101de0 0x7efd2c0ffa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:36.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.446+0000 7efd31672700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd2c102700 0x7efd2c0fff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:36.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.446+0000 7efd31672700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd2c1004c0 con 0x7efd2c101de0 2024-04-22T23:38:36.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.446+0000 7efd31672700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd2c100600 con 0x7efd2c102700 2024-04-22T23:38:36.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.446+0000 7efd2affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd2c101de0 0x7efd2c0ffa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:36.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.446+0000 7efd2affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd2c101de0 0x7efd2c0ffa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43172/0 (socket says 172.21.15.67:43172) 2024-04-22T23:38:36.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.446+0000 7efd2affd700 1 -- 172.21.15.67:0/1728935867 learned_addr learned my addr 172.21.15.67:0/1728935867 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:38:36.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.446+0000 7efd2a7fc700 1 --2- 172.21.15.67:0/1728935867 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd2c102700 0x7efd2c0fff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:36.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.446+0000 7efd2affd700 1 -- 172.21.15.67:0/1728935867 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd2c102700 msgr2=0x7efd2c0fff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:36.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.446+0000 7efd2affd700 1 --2- 172.21.15.67:0/1728935867 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd2c102700 0x7efd2c0fff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:36.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.446+0000 7efd2affd700 1 -- 172.21.15.67:0/1728935867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd1c0095f0 con 0x7efd2c101de0 2024-04-22T23:38:36.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.446+0000 7efd2affd700 1 --2- 172.21.15.67:0/1728935867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd2c101de0 0x7efd2c0ffa40 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7efd1c006010 tx=0x7efd1c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:38:36.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.446+0000 7efd23fff700 1 -- 172.21.15.67:0/1728935867 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd1c01d070 con 0x7efd2c101de0 2024-04-22T23:38:36.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.446+0000 7efd31672700 1 -- 172.21.15.67:0/1728935867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd2c100880 con 0x7efd2c101de0 2024-04-22T23:38:36.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.446+0000 7efd23fff700 1 -- 172.21.15.67:0/1728935867 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efd1c00f460 con 0x7efd2c101de0 2024-04-22T23:38:36.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.446+0000 7efd23fff700 1 -- 172.21.15.67:0/1728935867 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd1c017630 con 0x7efd2c101de0 2024-04-22T23:38:36.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.447+0000 7efd31672700 1 -- 172.21.15.67:0/1728935867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd2c1a2dd0 con 0x7efd2c101de0 2024-04-22T23:38:36.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.447+0000 7efd31672700 1 -- 172.21.15.67:0/1728935867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd2c1082f0 con 0x7efd2c101de0 2024-04-22T23:38:36.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.448+0000 7efd23fff700 1 -- 172.21.15.67:0/1728935867 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7efd1c00f5d0 con 0x7efd2c101de0 2024-04-22T23:38:36.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.448+0000 7efd23fff700 1 --2- 172.21.15.67:0/1728935867 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efd1806ecc0 0x7efd18071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:36.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.448+0000 7efd23fff700 1 -- 172.21.15.67:0/1728935867 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7efd1c090450 con 0x7efd2c101de0 2024-04-22T23:38:36.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.449+0000 7efd2a7fc700 1 --2- 172.21.15.67:0/1728935867 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efd1806ecc0 0x7efd18071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:36.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.449+0000 7efd2a7fc700 1 --2- 172.21.15.67:0/1728935867 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efd1806ecc0 0x7efd18071180 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7efd14005950 tx=0x7efd140058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:38:36.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.453+0000 7efd23fff700 1 -- 172.21.15.67:0/1728935867 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7efd1c059090 con 0x7efd2c101de0 2024-04-22T23:38:36.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:36 smithi134 ceph-mon[85630]: from='client.34708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:38:36.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.751+0000 7efd31672700 1 -- 172.21.15.67:0/1728935867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7efd2c04f2e0 con 0x7efd2c101de0 2024-04-22T23:38:36.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.751+0000 7efd23fff700 1 -- 172.21.15.67:0/1728935867 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7efd1c05c750 con 0x7efd2c101de0 2024-04-22T23:38:36.745 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:38:36.745 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:38:36.745 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:38:36.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:38:36.745 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:38:36.745 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:38:36.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:38:36.746 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:38:36.746 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:38:36.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:38:36.746 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:38:36.746 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:38:36.746 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 3 2024-04-22T23:38:36.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:38:36.746 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:38:36.746 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:38:36.746 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 13 2024-04-22T23:38:36.746 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:38:36.746 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:38:36.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.754+0000 7efd31672700 1 -- 172.21.15.67:0/1728935867 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efd1806ecc0 msgr2=0x7efd18071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:36.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.754+0000 7efd31672700 1 --2- 172.21.15.67:0/1728935867 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efd1806ecc0 0x7efd18071180 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7efd14005950 tx=0x7efd140058e0 comp rx=0 tx=0).stop 2024-04-22T23:38:36.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.754+0000 7efd31672700 1 -- 172.21.15.67:0/1728935867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd2c101de0 msgr2=0x7efd2c0ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:36.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.754+0000 7efd31672700 1 --2- 172.21.15.67:0/1728935867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd2c101de0 0x7efd2c0ffa40 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7efd1c006010 tx=0x7efd1c00be20 comp rx=0 tx=0).stop 2024-04-22T23:38:36.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.754+0000 7efd31672700 1 -- 172.21.15.67:0/1728935867 shutdown_connections 2024-04-22T23:38:36.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.754+0000 7efd31672700 1 --2- 172.21.15.67:0/1728935867 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efd1806ecc0 0x7efd18071180 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:36.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.754+0000 7efd31672700 1 --2- 172.21.15.67:0/1728935867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd2c101de0 0x7efd2c0ffa40 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:36.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.754+0000 7efd31672700 1 --2- 172.21.15.67:0/1728935867 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd2c102700 0x7efd2c0fff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:36.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.754+0000 7efd31672700 1 -- 172.21.15.67:0/1728935867 >> 172.21.15.67:0/1728935867 conn(0x7efd2c0fb650 msgr2=0x7efd2c105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:38:36.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.755+0000 7efd31672700 1 -- 172.21.15.67:0/1728935867 shutdown_connections 2024-04-22T23:38:36.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.755+0000 7efd31672700 1 -- 172.21.15.67:0/1728935867 wait complete. 2024-04-22T23:38:36.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.863+0000 7feb6ebbe700 1 -- 172.21.15.67:0/2130102267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb68105bd0 msgr2=0x7feb68105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:36.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.863+0000 7feb6ebbe700 1 --2- 172.21.15.67:0/2130102267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb68105bd0 0x7feb68105fb0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7feb58009910 tx=0x7feb58009c20 comp rx=0 tx=0).stop 2024-04-22T23:38:36.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.863+0000 7feb6ebbe700 1 -- 172.21.15.67:0/2130102267 shutdown_connections 2024-04-22T23:38:36.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.863+0000 7feb6ebbe700 1 --2- 172.21.15.67:0/2130102267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb680ffc20 0x7feb68100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:36.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.863+0000 7feb6ebbe700 1 --2- 172.21.15.67:0/2130102267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb68105bd0 0x7feb68105fb0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:36.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.863+0000 7feb6ebbe700 1 -- 172.21.15.67:0/2130102267 >> 172.21.15.67:0/2130102267 conn(0x7feb68075670 msgr2=0x7feb68075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:38:36.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.863+0000 7feb6ebbe700 1 -- 172.21.15.67:0/2130102267 shutdown_connections 2024-04-22T23:38:36.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.863+0000 7feb6ebbe700 1 -- 172.21.15.67:0/2130102267 wait complete. 2024-04-22T23:38:36.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.864+0000 7feb6ebbe700 1 Processor -- start 2024-04-22T23:38:36.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.864+0000 7feb6ebbe700 1 -- start start 2024-04-22T23:38:36.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.864+0000 7feb6ebbe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb680ffc20 0x7feb68101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:36.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.864+0000 7feb6ebbe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb68105bd0 0x7feb68101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:36.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.864+0000 7feb6ebbe700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb681022c0 con 0x7feb68105bd0 2024-04-22T23:38:36.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.864+0000 7feb6ebbe700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb68102400 con 0x7feb680ffc20 2024-04-22T23:38:36.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.864+0000 7feb67fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb68105bd0 0x7feb68101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:36.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.864+0000 7feb67fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb68105bd0 0x7feb68101d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43190/0 (socket says 172.21.15.67:43190) 2024-04-22T23:38:36.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.864+0000 7feb67fff700 1 -- 172.21.15.67:0/496656632 learned_addr learned my addr 172.21.15.67:0/496656632 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:38:36.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.864+0000 7feb6c95a700 1 --2- 172.21.15.67:0/496656632 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb680ffc20 0x7feb68101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:36.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.864+0000 7feb67fff700 1 -- 172.21.15.67:0/496656632 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb680ffc20 msgr2=0x7feb68101840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:36.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.864+0000 7feb67fff700 1 --2- 172.21.15.67:0/496656632 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb680ffc20 0x7feb68101840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:36.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.864+0000 7feb67fff700 1 -- 172.21.15.67:0/496656632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb580095f0 con 0x7feb68105bd0 2024-04-22T23:38:36.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.864+0000 7feb6c95a700 1 --2- 172.21.15.67:0/496656632 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb680ffc20 0x7feb68101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:38:36.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.865+0000 7feb67fff700 1 --2- 172.21.15.67:0/496656632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb68105bd0 0x7feb68101d80 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7feb5000d8d0 tx=0x7feb5000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:38:36.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.865+0000 7feb65ffb700 1 -- 172.21.15.67:0/496656632 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb50009940 con 0x7feb68105bd0 2024-04-22T23:38:36.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.865+0000 7feb6ebbe700 1 -- 172.21.15.67:0/496656632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb681a2a20 con 0x7feb68105bd0 2024-04-22T23:38:36.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.865+0000 7feb65ffb700 1 -- 172.21.15.67:0/496656632 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feb50010460 con 0x7feb68105bd0 2024-04-22T23:38:36.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.865+0000 7feb65ffb700 1 -- 172.21.15.67:0/496656632 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb5000f5f0 con 0x7feb68105bd0 2024-04-22T23:38:36.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.865+0000 7feb6ebbe700 1 -- 172.21.15.67:0/496656632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb681a2e30 con 0x7feb68105bd0 2024-04-22T23:38:36.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.866+0000 7feb6ebbe700 1 -- 172.21.15.67:0/496656632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb6804f2e0 con 0x7feb68105bd0 2024-04-22T23:38:36.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.866+0000 7feb65ffb700 1 -- 172.21.15.67:0/496656632 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7feb500105d0 con 0x7feb68105bd0 2024-04-22T23:38:36.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.867+0000 7feb65ffb700 1 --2- 172.21.15.67:0/496656632 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7feb5406ecc0 0x7feb54071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:36.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.867+0000 7feb65ffb700 1 -- 172.21.15.67:0/496656632 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7feb5008eff0 con 0x7feb68105bd0 2024-04-22T23:38:36.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.867+0000 7feb6c95a700 1 --2- 172.21.15.67:0/496656632 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7feb5406ecc0 0x7feb54071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:36.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.867+0000 7feb6c95a700 1 --2- 172.21.15.67:0/496656632 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7feb5406ecc0 0x7feb54071180 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7feb5800b5c0 tx=0x7feb580053a0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:38:36.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:36.872+0000 7feb65ffb700 1 -- 172.21.15.67:0/496656632 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7feb50057c30 con 0x7feb68105bd0 2024-04-22T23:38:37.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.129+0000 7feb6ebbe700 1 -- 172.21.15.67:0/496656632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7feb681a3020 con 0x7feb68105bd0 2024-04-22T23:38:37.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.130+0000 7feb65ffb700 1 -- 172.21.15.67:0/496656632 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1871 (secure 0 0 0) 0x7feb5005b2f0 con 0x7feb68105bd0 2024-04-22T23:38:37.124 INFO:teuthology.orchestra.run.smithi067.stdout:e31 2024-04-22T23:38:37.124 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:38:37.124 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:38:37.124 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:38:37.124 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:38:37.124 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:38:37.124 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:38:37.125 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:38:37.125 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:38:37.125 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:38:37.125 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:38:37.125 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:38:37.125 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:38:37.125 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:38:37.125 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:38:37.125 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:38:37.125 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:38:37.125 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:38:37.125 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:38:37.126 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:38:37.126 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:38:37.126 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:38:37.126 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:38:37.126 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:38:37.126 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:38:37.126 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:38:37.126 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:38:37.126 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:38:37.126 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:38:37.126 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:38:37.127 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:38:37.127 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:38:37.127 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:38:37.127 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:38:37.127 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:38:37.127 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:38:37.127 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:38:37.127 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:38:37.127 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:38:37.127 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:38:37.127 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44259} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1827288036,v1:172.21.15.134:6827/1827288036] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:38:37.127 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:38:37.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.132+0000 7feb6ebbe700 1 -- 172.21.15.67:0/496656632 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7feb5406ecc0 msgr2=0x7feb54071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:37.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.132+0000 7feb6ebbe700 1 --2- 172.21.15.67:0/496656632 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7feb5406ecc0 0x7feb54071180 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7feb5800b5c0 tx=0x7feb580053a0 comp rx=0 tx=0).stop 2024-04-22T23:38:37.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.132+0000 7feb6ebbe700 1 -- 172.21.15.67:0/496656632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb68105bd0 msgr2=0x7feb68101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:37.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.132+0000 7feb6ebbe700 1 --2- 172.21.15.67:0/496656632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb68105bd0 0x7feb68101d80 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7feb5000d8d0 tx=0x7feb5000dc90 comp rx=0 tx=0).stop 2024-04-22T23:38:37.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.132+0000 7feb6ebbe700 1 -- 172.21.15.67:0/496656632 shutdown_connections 2024-04-22T23:38:37.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.132+0000 7feb6ebbe700 1 --2- 172.21.15.67:0/496656632 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7feb5406ecc0 0x7feb54071180 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:37.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.132+0000 7feb6ebbe700 1 --2- 172.21.15.67:0/496656632 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb680ffc20 0x7feb68101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:37.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.132+0000 7feb6ebbe700 1 --2- 172.21.15.67:0/496656632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb68105bd0 0x7feb68101d80 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:37.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.132+0000 7feb6ebbe700 1 -- 172.21.15.67:0/496656632 >> 172.21.15.67:0/496656632 conn(0x7feb68075670 msgr2=0x7feb68068680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:38:37.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.132+0000 7feb6ebbe700 1 -- 172.21.15.67:0/496656632 shutdown_connections 2024-04-22T23:38:37.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.132+0000 7feb6ebbe700 1 -- 172.21.15.67:0/496656632 wait complete. 2024-04-22T23:38:37.129 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 31 2024-04-22T23:38:37.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.241+0000 7ff65338e700 1 -- 172.21.15.67:0/4070376535 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff64c101850 msgr2=0x7ff64c1058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:37.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.241+0000 7ff65338e700 1 --2- 172.21.15.67:0/4070376535 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff64c101850 0x7ff64c1058a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff640009910 tx=0x7ff640009c20 comp rx=0 tx=0).stop 2024-04-22T23:38:37.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.241+0000 7ff65338e700 1 -- 172.21.15.67:0/4070376535 shutdown_connections 2024-04-22T23:38:37.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.241+0000 7ff65338e700 1 --2- 172.21.15.67:0/4070376535 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff64c101850 0x7ff64c1058a0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:37.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.241+0000 7ff65338e700 1 --2- 172.21.15.67:0/4070376535 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff64c100ea0 0x7ff64c101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:37.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.241+0000 7ff65338e700 1 -- 172.21.15.67:0/4070376535 >> 172.21.15.67:0/4070376535 conn(0x7ff64c0fc750 msgr2=0x7ff64c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:38:37.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.241+0000 7ff65338e700 1 -- 172.21.15.67:0/4070376535 shutdown_connections 2024-04-22T23:38:37.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.241+0000 7ff65338e700 1 -- 172.21.15.67:0/4070376535 wait complete. 2024-04-22T23:38:37.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.242+0000 7ff65338e700 1 Processor -- start 2024-04-22T23:38:37.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.242+0000 7ff65338e700 1 -- start start 2024-04-22T23:38:37.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.242+0000 7ff65338e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff64c100ea0 0x7ff64c073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:37.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.242+0000 7ff65338e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff64c101850 0x7ff64c0735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:37.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.242+0000 7ff65338e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff64c073b10 con 0x7ff64c100ea0 2024-04-22T23:38:37.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.242+0000 7ff65338e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff64c073c50 con 0x7ff64c101850 2024-04-22T23:38:37.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.242+0000 7ff65112a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff64c100ea0 0x7ff64c073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:37.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.242+0000 7ff65112a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff64c100ea0 0x7ff64c073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43198/0 (socket says 172.21.15.67:43198) 2024-04-22T23:38:37.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.242+0000 7ff65112a700 1 -- 172.21.15.67:0/1824712798 learned_addr learned my addr 172.21.15.67:0/1824712798 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:38:37.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.242+0000 7ff650929700 1 --2- 172.21.15.67:0/1824712798 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff64c101850 0x7ff64c0735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:37.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.243+0000 7ff65112a700 1 -- 172.21.15.67:0/1824712798 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff64c101850 msgr2=0x7ff64c0735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:37.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.243+0000 7ff65112a700 1 --2- 172.21.15.67:0/1824712798 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff64c101850 0x7ff64c0735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:37.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.243+0000 7ff65112a700 1 -- 172.21.15.67:0/1824712798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6400095f0 con 0x7ff64c100ea0 2024-04-22T23:38:37.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.243+0000 7ff65112a700 1 --2- 172.21.15.67:0/1824712798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff64c100ea0 0x7ff64c073090 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff63c0060b0 tx=0x7ff63c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:38:37.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.243+0000 7ff64a7fc700 1 -- 172.21.15.67:0/1824712798 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff63c00dda0 con 0x7ff64c100ea0 2024-04-22T23:38:37.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.243+0000 7ff64a7fc700 1 -- 172.21.15.67:0/1824712798 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff63c00f040 con 0x7ff64c100ea0 2024-04-22T23:38:37.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.243+0000 7ff65338e700 1 -- 172.21.15.67:0/1824712798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff64c1a2b00 con 0x7ff64c100ea0 2024-04-22T23:38:37.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.243+0000 7ff64a7fc700 1 -- 172.21.15.67:0/1824712798 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff63c0146f0 con 0x7ff64c100ea0 2024-04-22T23:38:37.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.243+0000 7ff65338e700 1 -- 172.21.15.67:0/1824712798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff64c1a2f10 con 0x7ff64c100ea0 2024-04-22T23:38:37.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.244+0000 7ff65338e700 1 -- 172.21.15.67:0/1824712798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff64c1092a0 con 0x7ff64c100ea0 2024-04-22T23:38:37.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.244+0000 7ff64a7fc700 1 -- 172.21.15.67:0/1824712798 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff63c015400 con 0x7ff64c100ea0 2024-04-22T23:38:37.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.245+0000 7ff64a7fc700 1 --2- 172.21.15.67:0/1824712798 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff63406ecc0 0x7ff634071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:37.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.245+0000 7ff64a7fc700 1 -- 172.21.15.67:0/1824712798 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7ff63c08e9b0 con 0x7ff64c100ea0 2024-04-22T23:38:37.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.245+0000 7ff650929700 1 --2- 172.21.15.67:0/1824712798 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff63406ecc0 0x7ff634071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:37.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.245+0000 7ff650929700 1 --2- 172.21.15.67:0/1824712798 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff63406ecc0 0x7ff634071180 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff64c074880 tx=0x7ff64001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:38:37.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.250+0000 7ff64a7fc700 1 -- 172.21.15.67:0/1824712798 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff63c0576a0 con 0x7ff64c100ea0 2024-04-22T23:38:37.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.481+0000 7ff65338e700 1 -- 172.21.15.67:0/1824712798 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff64c0611d0 con 0x7ff63406ecc0 2024-04-22T23:38:37.475 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:37 smithi067 ceph-mon[112438]: from='client.34712 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:38:37.475 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:37 smithi067 ceph-mon[112438]: pgmap v367: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:37.475 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:37 smithi067 ceph-mon[112438]: from='client.34716 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:38:37.475 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:37 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1728935867' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:38:37.475 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:37 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/496656632' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:38:37.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.483+0000 7ff64a7fc700 1 -- 172.21.15.67:0/1824712798 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff64c0611d0 con 0x7ff63406ecc0 2024-04-22T23:38:37.476 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:38:37.476 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:38:37.476 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:38:37.476 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:38:37.476 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:38:37.477 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:38:37.477 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:38:37.477 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:38:37.477 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:38:37.477 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:38:37.477 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T23:38:37.477 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:38:37.477 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:38:37.477 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:38:37.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.485+0000 7ff65338e700 1 -- 172.21.15.67:0/1824712798 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff63406ecc0 msgr2=0x7ff634071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:37.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.485+0000 7ff65338e700 1 --2- 172.21.15.67:0/1824712798 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff63406ecc0 0x7ff634071180 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff64c074880 tx=0x7ff64001a040 comp rx=0 tx=0).stop 2024-04-22T23:38:37.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.485+0000 7ff65338e700 1 -- 172.21.15.67:0/1824712798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff64c100ea0 msgr2=0x7ff64c073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:37.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.485+0000 7ff65338e700 1 --2- 172.21.15.67:0/1824712798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff64c100ea0 0x7ff64c073090 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff63c0060b0 tx=0x7ff63c00d3b0 comp rx=0 tx=0).stop 2024-04-22T23:38:37.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.485+0000 7ff65338e700 1 -- 172.21.15.67:0/1824712798 shutdown_connections 2024-04-22T23:38:37.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.485+0000 7ff65338e700 1 --2- 172.21.15.67:0/1824712798 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff63406ecc0 0x7ff634071180 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:37.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.485+0000 7ff65338e700 1 --2- 172.21.15.67:0/1824712798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff64c100ea0 0x7ff64c073090 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:37.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.485+0000 7ff65338e700 1 --2- 172.21.15.67:0/1824712798 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff64c101850 0x7ff64c0735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:37.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.485+0000 7ff65338e700 1 -- 172.21.15.67:0/1824712798 >> 172.21.15.67:0/1824712798 conn(0x7ff64c0fc750 msgr2=0x7ff64c0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:38:37.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.485+0000 7ff65338e700 1 -- 172.21.15.67:0/1824712798 shutdown_connections 2024-04-22T23:38:37.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.485+0000 7ff65338e700 1 -- 172.21.15.67:0/1824712798 wait complete. 2024-04-22T23:38:37.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:37 smithi134 ceph-mon[85630]: from='client.34712 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:38:37.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:37 smithi134 ceph-mon[85630]: pgmap v367: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:37.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:37 smithi134 ceph-mon[85630]: from='client.34716 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:38:37.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:37 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1728935867' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:38:37.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:37 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/496656632' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:38:37.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.592+0000 7ffbcc4be700 1 -- 172.21.15.67:0/2745786113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbc4068580 msgr2=0x7ffbc40689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:37.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.592+0000 7ffbcc4be700 1 --2- 172.21.15.67:0/2745786113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbc4068580 0x7ffbc40689e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8009910 tx=0x7ffbb8009c20 comp rx=0 tx=0).stop 2024-04-22T23:38:37.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.592+0000 7ffbcc4be700 1 -- 172.21.15.67:0/2745786113 shutdown_connections 2024-04-22T23:38:37.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.592+0000 7ffbcc4be700 1 --2- 172.21.15.67:0/2745786113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbc4068580 0x7ffbc40689e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:37.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.592+0000 7ffbcc4be700 1 --2- 172.21.15.67:0/2745786113 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffbc4106cd0 0x7ffbc41070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:37.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.592+0000 7ffbcc4be700 1 -- 172.21.15.67:0/2745786113 >> 172.21.15.67:0/2745786113 conn(0x7ffbc4075670 msgr2=0x7ffbc4075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:38:37.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.592+0000 7ffbcc4be700 1 -- 172.21.15.67:0/2745786113 shutdown_connections 2024-04-22T23:38:37.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.592+0000 7ffbcc4be700 1 -- 172.21.15.67:0/2745786113 wait complete. 2024-04-22T23:38:37.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.593+0000 7ffbcc4be700 1 Processor -- start 2024-04-22T23:38:37.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.593+0000 7ffbcc4be700 1 -- start start 2024-04-22T23:38:37.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.593+0000 7ffbcc4be700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbc4068580 0x7ffbc4196980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:37.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.593+0000 7ffbcc4be700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffbc4106cd0 0x7ffbc4196ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:37.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.593+0000 7ffbcc4be700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc4197550 con 0x7ffbc4068580 2024-04-22T23:38:37.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.593+0000 7ffbcc4be700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc419b360 con 0x7ffbc4106cd0 2024-04-22T23:38:37.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.593+0000 7ffbca25a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbc4068580 0x7ffbc4196980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:37.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.593+0000 7ffbca25a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbc4068580 0x7ffbc4196980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43208/0 (socket says 172.21.15.67:43208) 2024-04-22T23:38:37.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.593+0000 7ffbca25a700 1 -- 172.21.15.67:0/2701390733 learned_addr learned my addr 172.21.15.67:0/2701390733 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:38:37.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.593+0000 7ffbc9a59700 1 --2- 172.21.15.67:0/2701390733 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffbc4106cd0 0x7ffbc4196ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:37.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.593+0000 7ffbca25a700 1 -- 172.21.15.67:0/2701390733 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffbc4106cd0 msgr2=0x7ffbc4196ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:37.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.593+0000 7ffbca25a700 1 --2- 172.21.15.67:0/2701390733 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffbc4106cd0 0x7ffbc4196ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:37.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.593+0000 7ffbca25a700 1 -- 172.21.15.67:0/2701390733 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb80095f0 con 0x7ffbc4068580 2024-04-22T23:38:37.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.593+0000 7ffbc9a59700 1 --2- 172.21.15.67:0/2701390733 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffbc4106cd0 0x7ffbc4196ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:38:37.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.594+0000 7ffbca25a700 1 --2- 172.21.15.67:0/2701390733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbc4068580 0x7ffbc4196980 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ffbb400ba50 tx=0x7ffbb400be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:38:37.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.594+0000 7ffbc37fe700 1 -- 172.21.15.67:0/2701390733 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffbb400ca00 con 0x7ffbc4068580 2024-04-22T23:38:37.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.594+0000 7ffbc37fe700 1 -- 172.21.15.67:0/2701390733 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffbb400cb60 con 0x7ffbc4068580 2024-04-22T23:38:37.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.594+0000 7ffbcc4be700 1 -- 172.21.15.67:0/2701390733 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbc419b640 con 0x7ffbc4068580 2024-04-22T23:38:37.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.594+0000 7ffbc37fe700 1 -- 172.21.15.67:0/2701390733 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffbb40075a0 con 0x7ffbc4068580 2024-04-22T23:38:37.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.594+0000 7ffbcc4be700 1 -- 172.21.15.67:0/2701390733 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbc419bb90 con 0x7ffbc4068580 2024-04-22T23:38:37.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.595+0000 7ffbcc4be700 1 -- 172.21.15.67:0/2701390733 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbc41091f0 con 0x7ffbc4068580 2024-04-22T23:38:37.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.595+0000 7ffbc37fe700 1 -- 172.21.15.67:0/2701390733 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ffbb400ccd0 con 0x7ffbc4068580 2024-04-22T23:38:37.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.596+0000 7ffbc37fe700 1 --2- 172.21.15.67:0/2701390733 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ffbac06ecc0 0x7ffbac071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:38:37.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.596+0000 7ffbc37fe700 1 -- 172.21.15.67:0/2701390733 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7ffbb4006e40 con 0x7ffbc4068580 2024-04-22T23:38:37.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.596+0000 7ffbc9a59700 1 --2- 172.21.15.67:0/2701390733 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ffbac06ecc0 0x7ffbac071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:38:37.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.596+0000 7ffbc9a59700 1 --2- 172.21.15.67:0/2701390733 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ffbac06ecc0 0x7ffbac071180 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ffbb800b5c0 tx=0x7ffbb8005480 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:38:37.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.601+0000 7ffbc37fe700 1 -- 172.21.15.67:0/2701390733 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ffbb405af80 con 0x7ffbc4068580 2024-04-22T23:38:37.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.900+0000 7ffbcc4be700 1 -- 172.21.15.67:0/2701390733 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ffbc4197db0 con 0x7ffbc4068580 2024-04-22T23:38:37.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.901+0000 7ffbc37fe700 1 -- 172.21.15.67:0/2701390733 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7ffbb405ab10 con 0x7ffbc4068580 2024-04-22T23:38:37.895 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:38:37.895 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:38:37.895 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:38:37.895 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:38:37.895 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:38:37.895 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:38:37.895 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:38:37.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.903+0000 7ffbcc4be700 1 -- 172.21.15.67:0/2701390733 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ffbac06ecc0 msgr2=0x7ffbac071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:37.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.903+0000 7ffbcc4be700 1 --2- 172.21.15.67:0/2701390733 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ffbac06ecc0 0x7ffbac071180 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ffbb800b5c0 tx=0x7ffbb8005480 comp rx=0 tx=0).stop 2024-04-22T23:38:37.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.903+0000 7ffbcc4be700 1 -- 172.21.15.67:0/2701390733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbc4068580 msgr2=0x7ffbc4196980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:38:37.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.903+0000 7ffbcc4be700 1 --2- 172.21.15.67:0/2701390733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbc4068580 0x7ffbc4196980 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ffbb400ba50 tx=0x7ffbb400be10 comp rx=0 tx=0).stop 2024-04-22T23:38:37.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.903+0000 7ffbcc4be700 1 -- 172.21.15.67:0/2701390733 shutdown_connections 2024-04-22T23:38:37.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.903+0000 7ffbcc4be700 1 --2- 172.21.15.67:0/2701390733 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ffbac06ecc0 0x7ffbac071180 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:37.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.903+0000 7ffbcc4be700 1 --2- 172.21.15.67:0/2701390733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbc4068580 0x7ffbc4196980 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:37.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.903+0000 7ffbcc4be700 1 --2- 172.21.15.67:0/2701390733 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffbc4106cd0 0x7ffbc4196ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:38:37.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.903+0000 7ffbcc4be700 1 -- 172.21.15.67:0/2701390733 >> 172.21.15.67:0/2701390733 conn(0x7ffbc4075670 msgr2=0x7ffbc40fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:38:37.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.903+0000 7ffbcc4be700 1 -- 172.21.15.67:0/2701390733 shutdown_connections 2024-04-22T23:38:37.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:38:37.903+0000 7ffbcc4be700 1 -- 172.21.15.67:0/2701390733 wait complete. 2024-04-22T23:38:38.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:38 smithi134 ceph-mon[85630]: from='client.34728 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:38:38.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:38 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2701390733' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:38:38.537 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:38 smithi067 ceph-mon[112438]: from='client.34728 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:38:38.537 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:38 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2701390733' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:38:39.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:39 smithi134 ceph-mon[85630]: pgmap v368: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:39.537 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:39 smithi067 ceph-mon[112438]: pgmap v368: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:41.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:41 smithi134 ceph-mon[85630]: pgmap v369: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:41.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:38:41.537 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:41 smithi067 ceph-mon[112438]: pgmap v369: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:41.537 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:38:43.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:43 smithi134 ceph-mon[85630]: pgmap v370: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:43.537 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:43 smithi067 ceph-mon[112438]: pgmap v370: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:45.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:45 smithi134 ceph-mon[85630]: pgmap v371: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:45.537 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:45 smithi067 ceph-mon[112438]: pgmap v371: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:47.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:47 smithi134 ceph-mon[85630]: pgmap v372: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:47.537 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:47 smithi067 ceph-mon[112438]: pgmap v372: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:49.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:49 smithi134 ceph-mon[85630]: pgmap v373: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:49.537 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:49 smithi067 ceph-mon[112438]: pgmap v373: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:51.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:51 smithi134 ceph-mon[85630]: pgmap v374: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:51.537 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:51 smithi067 ceph-mon[112438]: pgmap v374: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:53.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:53 smithi134 ceph-mon[85630]: pgmap v375: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:53.537 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:53 smithi067 ceph-mon[112438]: pgmap v375: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:55.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:55 smithi134 ceph-mon[85630]: pgmap v376: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:55.537 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:55 smithi067 ceph-mon[112438]: pgmap v376: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:56.537 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:38:56.737 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:38:57.537 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:57 smithi067 ceph-mon[112438]: pgmap v377: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:57.736 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:57 smithi134 ceph-mon[85630]: pgmap v377: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:59.537 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:38:59 smithi067 ceph-mon[112438]: pgmap v378: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:38:59.737 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:38:59 smithi134 ceph-mon[85630]: pgmap v378: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:01.537 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:01 smithi067 ceph-mon[112438]: pgmap v379: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:01.737 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:01 smithi134 ceph-mon[85630]: pgmap v379: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:03.537 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:03 smithi067 ceph-mon[112438]: pgmap v380: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:03.737 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:03 smithi134 ceph-mon[85630]: pgmap v380: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:05.537 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:05 smithi067 ceph-mon[112438]: pgmap v381: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:05.737 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:05 smithi134 ceph-mon[85630]: pgmap v381: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:07.537 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:07 smithi067 ceph-mon[112438]: pgmap v382: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:07.737 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:07 smithi134 ceph-mon[85630]: pgmap v382: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:08.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.025+0000 7fb711991700 1 -- 172.21.15.67:0/1310838256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb70c0ffda0 msgr2=0x7fb70c100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:08.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.025+0000 7fb711991700 1 --2- 172.21.15.67:0/1310838256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb70c0ffda0 0x7fb70c100180 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc00b0a0 tx=0x7fb6fc00b3b0 comp rx=0 tx=0).stop 2024-04-22T23:39:08.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.025+0000 7fb711991700 1 -- 172.21.15.67:0/1310838256 shutdown_connections 2024-04-22T23:39:08.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.025+0000 7fb711991700 1 --2- 172.21.15.67:0/1310838256 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb70c100750 0x7fb70c1047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:08.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.025+0000 7fb711991700 1 --2- 172.21.15.67:0/1310838256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb70c0ffda0 0x7fb70c100180 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:08.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.025+0000 7fb711991700 1 -- 172.21.15.67:0/1310838256 >> 172.21.15.67:0/1310838256 conn(0x7fb70c0fb650 msgr2=0x7fb70c0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:39:08.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.025+0000 7fb711991700 1 -- 172.21.15.67:0/1310838256 shutdown_connections 2024-04-22T23:39:08.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.025+0000 7fb711991700 1 -- 172.21.15.67:0/1310838256 wait complete. 2024-04-22T23:39:08.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.026+0000 7fb711991700 1 Processor -- start 2024-04-22T23:39:08.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.026+0000 7fb711991700 1 -- start start 2024-04-22T23:39:08.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.026+0000 7fb711991700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb70c0ffda0 0x7fb70c197780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:08.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.026+0000 7fb711991700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb70c100750 0x7fb70c197cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:08.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.026+0000 7fb711991700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb70c1982e0 con 0x7fb70c0ffda0 2024-04-22T23:39:08.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.026+0000 7fb711991700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb70c198420 con 0x7fb70c100750 2024-04-22T23:39:08.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.026+0000 7fb70affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb70c0ffda0 0x7fb70c197780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:08.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.026+0000 7fb70affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb70c0ffda0 0x7fb70c197780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35188/0 (socket says 172.21.15.67:35188) 2024-04-22T23:39:08.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.026+0000 7fb70affd700 1 -- 172.21.15.67:0/985258849 learned_addr learned my addr 172.21.15.67:0/985258849 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:39:08.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.027+0000 7fb70a7fc700 1 --2- 172.21.15.67:0/985258849 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb70c100750 0x7fb70c197cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:08.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.027+0000 7fb70affd700 1 -- 172.21.15.67:0/985258849 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb70c100750 msgr2=0x7fb70c197cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:08.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.027+0000 7fb70affd700 1 --2- 172.21.15.67:0/985258849 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb70c100750 0x7fb70c197cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:08.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.027+0000 7fb70affd700 1 -- 172.21.15.67:0/985258849 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6fc009d00 con 0x7fb70c0ffda0 2024-04-22T23:39:08.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.027+0000 7fb70affd700 1 --2- 172.21.15.67:0/985258849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb70c0ffda0 0x7fb70c197780 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc00b070 tx=0x7fb6fc00cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:39:08.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.027+0000 7fb71098f700 1 -- 172.21.15.67:0/985258849 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6fc00e040 con 0x7fb70c0ffda0 2024-04-22T23:39:08.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.027+0000 7fb711991700 1 -- 172.21.15.67:0/985258849 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb70c1a07f0 con 0x7fb70c0ffda0 2024-04-22T23:39:08.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.027+0000 7fb71098f700 1 -- 172.21.15.67:0/985258849 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb6fc003de0 con 0x7fb70c0ffda0 2024-04-22T23:39:08.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.027+0000 7fb71098f700 1 -- 172.21.15.67:0/985258849 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6fc01d8e0 con 0x7fb70c0ffda0 2024-04-22T23:39:08.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.027+0000 7fb711991700 1 -- 172.21.15.67:0/985258849 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb70c1a0c00 con 0x7fb70c0ffda0 2024-04-22T23:39:08.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.028+0000 7fb711991700 1 -- 172.21.15.67:0/985258849 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb70c108140 con 0x7fb70c0ffda0 2024-04-22T23:39:08.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.029+0000 7fb71098f700 1 -- 172.21.15.67:0/985258849 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb6fc004040 con 0x7fb70c0ffda0 2024-04-22T23:39:08.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.029+0000 7fb71098f700 1 --2- 172.21.15.67:0/985258849 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb6f806ecc0 0x7fb6f8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:08.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.029+0000 7fb71098f700 1 -- 172.21.15.67:0/985258849 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fb6fc090820 con 0x7fb70c0ffda0 2024-04-22T23:39:08.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.029+0000 7fb70a7fc700 1 --2- 172.21.15.67:0/985258849 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb6f806ecc0 0x7fb6f8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:08.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.030+0000 7fb70a7fc700 1 --2- 172.21.15.67:0/985258849 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb6f806ecc0 0x7fb6f8071180 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb6f4005950 tx=0x7fb6f40058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:39:08.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.034+0000 7fb71098f700 1 -- 172.21.15.67:0/985258849 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb6fc059460 con 0x7fb70c0ffda0 2024-04-22T23:39:08.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.266+0000 7fb711991700 1 -- 172.21.15.67:0/985258849 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb70c0611d0 con 0x7fb6f806ecc0 2024-04-22T23:39:08.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.267+0000 7fb71098f700 1 -- 172.21.15.67:0/985258849 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb70c0611d0 con 0x7fb6f806ecc0 2024-04-22T23:39:08.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.269+0000 7fb711991700 1 -- 172.21.15.67:0/985258849 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb6f806ecc0 msgr2=0x7fb6f8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:08.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.269+0000 7fb711991700 1 --2- 172.21.15.67:0/985258849 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb6f806ecc0 0x7fb6f8071180 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb6f4005950 tx=0x7fb6f40058e0 comp rx=0 tx=0).stop 2024-04-22T23:39:08.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.269+0000 7fb711991700 1 -- 172.21.15.67:0/985258849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb70c0ffda0 msgr2=0x7fb70c197780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:08.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.269+0000 7fb711991700 1 --2- 172.21.15.67:0/985258849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb70c0ffda0 0x7fb70c197780 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc00b070 tx=0x7fb6fc00cdd0 comp rx=0 tx=0).stop 2024-04-22T23:39:08.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.269+0000 7fb711991700 1 -- 172.21.15.67:0/985258849 shutdown_connections 2024-04-22T23:39:08.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.269+0000 7fb711991700 1 --2- 172.21.15.67:0/985258849 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb6f806ecc0 0x7fb6f8071180 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:08.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.269+0000 7fb711991700 1 --2- 172.21.15.67:0/985258849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb70c0ffda0 0x7fb70c197780 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:08.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.269+0000 7fb711991700 1 --2- 172.21.15.67:0/985258849 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb70c100750 0x7fb70c197cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:08.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.269+0000 7fb711991700 1 -- 172.21.15.67:0/985258849 >> 172.21.15.67:0/985258849 conn(0x7fb70c0fb650 msgr2=0x7fb70c0fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:39:08.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.269+0000 7fb711991700 1 -- 172.21.15.67:0/985258849 shutdown_connections 2024-04-22T23:39:08.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.269+0000 7fb711991700 1 -- 172.21.15.67:0/985258849 wait complete. 2024-04-22T23:39:08.273 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:39:08.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.378+0000 7f0590e21700 1 -- 172.21.15.67:0/217102870 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f058c0fffb0 msgr2=0x7f058c1058e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:08.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.378+0000 7f0590e21700 1 --2- 172.21.15.67:0/217102870 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f058c0fffb0 0x7f058c1058e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f0574009910 tx=0x7f0574009c20 comp rx=0 tx=0).stop 2024-04-22T23:39:08.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.378+0000 7f0590e21700 1 -- 172.21.15.67:0/217102870 shutdown_connections 2024-04-22T23:39:08.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.378+0000 7f0590e21700 1 --2- 172.21.15.67:0/217102870 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f058c0fffb0 0x7f058c1058e0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:08.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.378+0000 7f0590e21700 1 --2- 172.21.15.67:0/217102870 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f058c0ff600 0x7f058c0ff9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:08.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.378+0000 7f0590e21700 1 -- 172.21.15.67:0/217102870 >> 172.21.15.67:0/217102870 conn(0x7f058c074df0 msgr2=0x7f058c075200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:39:08.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.378+0000 7f0590e21700 1 -- 172.21.15.67:0/217102870 shutdown_connections 2024-04-22T23:39:08.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.378+0000 7f0590e21700 1 -- 172.21.15.67:0/217102870 wait complete. 2024-04-22T23:39:08.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.379+0000 7f0590e21700 1 Processor -- start 2024-04-22T23:39:08.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.379+0000 7f0590e21700 1 -- start start 2024-04-22T23:39:08.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.379+0000 7f0590e21700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f058c0ff600 0x7f058c19c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:08.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.379+0000 7f0590e21700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f058c0fffb0 0x7f058c19cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:08.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.379+0000 7f0590e21700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058c19d290 con 0x7f058c0ff600 2024-04-22T23:39:08.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.379+0000 7f0590e21700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058c196740 con 0x7f058c0fffb0 2024-04-22T23:39:08.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.380+0000 7f058a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f058c0ff600 0x7f058c19c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:08.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.380+0000 7f058a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f058c0ff600 0x7f058c19c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35206/0 (socket says 172.21.15.67:35206) 2024-04-22T23:39:08.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.380+0000 7f058a59c700 1 -- 172.21.15.67:0/2652933691 learned_addr learned my addr 172.21.15.67:0/2652933691 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:39:08.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.380+0000 7f0589d9b700 1 --2- 172.21.15.67:0/2652933691 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f058c0fffb0 0x7f058c19cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:08.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.380+0000 7f058a59c700 1 -- 172.21.15.67:0/2652933691 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f058c0fffb0 msgr2=0x7f058c19cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:08.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.380+0000 7f058a59c700 1 --2- 172.21.15.67:0/2652933691 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f058c0fffb0 0x7f058c19cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:08.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.380+0000 7f058a59c700 1 -- 172.21.15.67:0/2652933691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05740095f0 con 0x7f058c0ff600 2024-04-22T23:39:08.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.380+0000 7f0589d9b700 1 --2- 172.21.15.67:0/2652933691 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f058c0fffb0 0x7f058c19cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:39:08.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.380+0000 7f058a59c700 1 --2- 172.21.15.67:0/2652933691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f058c0ff600 0x7f058c19c6c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f057c0060b0 tx=0x7f057c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:39:08.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.380+0000 7f05837fe700 1 -- 172.21.15.67:0/2652933691 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f057c00dda0 con 0x7f058c0ff600 2024-04-22T23:39:08.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.380+0000 7f05837fe700 1 -- 172.21.15.67:0/2652933691 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f057c00f040 con 0x7f058c0ff600 2024-04-22T23:39:08.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.380+0000 7f0590e21700 1 -- 172.21.15.67:0/2652933691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f058c196a20 con 0x7f058c0ff600 2024-04-22T23:39:08.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.380+0000 7f05837fe700 1 -- 172.21.15.67:0/2652933691 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f057c014790 con 0x7f058c0ff600 2024-04-22T23:39:08.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.380+0000 7f0590e21700 1 -- 172.21.15.67:0/2652933691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f058c196f70 con 0x7f058c0ff600 2024-04-22T23:39:08.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.381+0000 7f0590e21700 1 -- 172.21.15.67:0/2652933691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f058c1091a0 con 0x7f058c0ff600 2024-04-22T23:39:08.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.382+0000 7f05837fe700 1 -- 172.21.15.67:0/2652933691 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f057c0148f0 con 0x7f058c0ff600 2024-04-22T23:39:08.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.382+0000 7f05837fe700 1 --2- 172.21.15.67:0/2652933691 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f057806ecc0 0x7f0578071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:08.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.382+0000 7f05837fe700 1 -- 172.21.15.67:0/2652933691 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f057c08e9a0 con 0x7f058c0ff600 2024-04-22T23:39:08.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.382+0000 7f0589d9b700 1 --2- 172.21.15.67:0/2652933691 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f057806ecc0 0x7f0578071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:08.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.383+0000 7f0589d9b700 1 --2- 172.21.15.67:0/2652933691 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f057806ecc0 0x7f0578071180 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f058c197bb0 tx=0x7f057401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:39:08.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.387+0000 7f05837fe700 1 -- 172.21.15.67:0/2652933691 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f057c05b240 con 0x7f058c0ff600 2024-04-22T23:39:08.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.617+0000 7f0590e21700 1 -- 172.21.15.67:0/2652933691 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f058c197d60 con 0x7f057806ecc0 2024-04-22T23:39:08.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.618+0000 7f05837fe700 1 -- 172.21.15.67:0/2652933691 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f058c197d60 con 0x7f057806ecc0 2024-04-22T23:39:08.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.620+0000 7f0590e21700 1 -- 172.21.15.67:0/2652933691 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f057806ecc0 msgr2=0x7f0578071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:08.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.620+0000 7f0590e21700 1 --2- 172.21.15.67:0/2652933691 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f057806ecc0 0x7f0578071180 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f058c197bb0 tx=0x7f057401a040 comp rx=0 tx=0).stop 2024-04-22T23:39:08.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.620+0000 7f0590e21700 1 -- 172.21.15.67:0/2652933691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f058c0ff600 msgr2=0x7f058c19c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:08.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.620+0000 7f0590e21700 1 --2- 172.21.15.67:0/2652933691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f058c0ff600 0x7f058c19c6c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f057c0060b0 tx=0x7f057c00d3b0 comp rx=0 tx=0).stop 2024-04-22T23:39:08.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.620+0000 7f0590e21700 1 -- 172.21.15.67:0/2652933691 shutdown_connections 2024-04-22T23:39:08.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.620+0000 7f0590e21700 1 --2- 172.21.15.67:0/2652933691 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f057806ecc0 0x7f0578071180 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:08.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.620+0000 7f0590e21700 1 --2- 172.21.15.67:0/2652933691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f058c0ff600 0x7f058c19c6c0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:08.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.620+0000 7f0590e21700 1 --2- 172.21.15.67:0/2652933691 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f058c0fffb0 0x7f058c19cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:08.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.620+0000 7f0590e21700 1 -- 172.21.15.67:0/2652933691 >> 172.21.15.67:0/2652933691 conn(0x7f058c074df0 msgr2=0x7f058c0fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:39:08.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.621+0000 7f0590e21700 1 -- 172.21.15.67:0/2652933691 shutdown_connections 2024-04-22T23:39:08.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.621+0000 7f0590e21700 1 -- 172.21.15.67:0/2652933691 wait complete. 2024-04-22T23:39:08.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.727+0000 7feea6a77700 1 -- 172.21.15.67:0/898362507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feea00ffda0 msgr2=0x7feea0100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:08.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.727+0000 7feea6a77700 1 --2- 172.21.15.67:0/898362507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feea00ffda0 0x7feea0100180 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fee94009910 tx=0x7fee94009c20 comp rx=0 tx=0).stop 2024-04-22T23:39:08.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.727+0000 7feea6a77700 1 -- 172.21.15.67:0/898362507 shutdown_connections 2024-04-22T23:39:08.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.727+0000 7feea6a77700 1 --2- 172.21.15.67:0/898362507 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feea0100750 0x7feea01047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:08.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.727+0000 7feea6a77700 1 --2- 172.21.15.67:0/898362507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feea00ffda0 0x7feea0100180 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:08.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.727+0000 7feea6a77700 1 -- 172.21.15.67:0/898362507 >> 172.21.15.67:0/898362507 conn(0x7feea00fb650 msgr2=0x7feea00fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:39:08.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.728+0000 7feea6a77700 1 -- 172.21.15.67:0/898362507 shutdown_connections 2024-04-22T23:39:08.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.728+0000 7feea6a77700 1 -- 172.21.15.67:0/898362507 wait complete. 2024-04-22T23:39:08.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.728+0000 7feea6a77700 1 Processor -- start 2024-04-22T23:39:08.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.728+0000 7feea6a77700 1 -- start start 2024-04-22T23:39:08.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.729+0000 7feea6a77700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feea00ffda0 0x7feea01976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:08.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.729+0000 7feea6a77700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feea0100750 0x7feea0197c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:08.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.729+0000 7feea6a77700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea0198230 con 0x7feea0100750 2024-04-22T23:39:08.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.729+0000 7feea6a77700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea01983a0 con 0x7feea00ffda0 2024-04-22T23:39:08.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.729+0000 7fee9ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feea0100750 0x7feea0197c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:08.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.729+0000 7fee9ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feea0100750 0x7feea0197c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34798/0 (socket says 172.21.15.67:34798) 2024-04-22T23:39:08.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.729+0000 7fee9ffff700 1 -- 172.21.15.67:0/4233116087 learned_addr learned my addr 172.21.15.67:0/4233116087 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:39:08.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.729+0000 7feea4813700 1 --2- 172.21.15.67:0/4233116087 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feea00ffda0 0x7feea01976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:08.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.729+0000 7fee9ffff700 1 -- 172.21.15.67:0/4233116087 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feea00ffda0 msgr2=0x7feea01976d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:08.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.729+0000 7fee9ffff700 1 --2- 172.21.15.67:0/4233116087 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feea00ffda0 0x7feea01976d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:08.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.729+0000 7fee9ffff700 1 -- 172.21.15.67:0/4233116087 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee940095f0 con 0x7feea0100750 2024-04-22T23:39:08.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.729+0000 7fee9ffff700 1 --2- 172.21.15.67:0/4233116087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feea0100750 0x7feea0197c10 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fee8c00d8d0 tx=0x7fee8c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:39:08.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.729+0000 7fee9dffb700 1 -- 172.21.15.67:0/4233116087 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee8c0041f0 con 0x7feea0100750 2024-04-22T23:39:08.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.729+0000 7feea6a77700 1 -- 172.21.15.67:0/4233116087 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feea01a0740 con 0x7feea0100750 2024-04-22T23:39:08.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.729+0000 7fee9dffb700 1 -- 172.21.15.67:0/4233116087 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fee8c007620 con 0x7feea0100750 2024-04-22T23:39:08.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.729+0000 7fee9dffb700 1 -- 172.21.15.67:0/4233116087 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee8c0056b0 con 0x7feea0100750 2024-04-22T23:39:08.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.730+0000 7feea6a77700 1 -- 172.21.15.67:0/4233116087 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feea01a0c30 con 0x7feea0100750 2024-04-22T23:39:08.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.731+0000 7feea6a77700 1 -- 172.21.15.67:0/4233116087 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feea0108140 con 0x7feea0100750 2024-04-22T23:39:08.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.731+0000 7fee9dffb700 1 -- 172.21.15.67:0/4233116087 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fee8c00f770 con 0x7feea0100750 2024-04-22T23:39:08.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.731+0000 7fee9dffb700 1 --2- 172.21.15.67:0/4233116087 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fee9006ecc0 0x7fee90071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:08.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.732+0000 7fee9dffb700 1 -- 172.21.15.67:0/4233116087 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fee8c08ee40 con 0x7feea0100750 2024-04-22T23:39:08.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.732+0000 7feea4813700 1 --2- 172.21.15.67:0/4233116087 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fee9006ecc0 0x7fee90071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:08.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.732+0000 7feea4813700 1 --2- 172.21.15.67:0/4233116087 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fee9006ecc0 0x7fee90071180 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fee94006010 tx=0x7fee9401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:39:08.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.736+0000 7fee9dffb700 1 -- 172.21.15.67:0/4233116087 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fee8c057a80 con 0x7feea0100750 2024-04-22T23:39:08.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.954+0000 7feea6a77700 1 -- 172.21.15.67:0/4233116087 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7feea002cce0 con 0x7fee9006ecc0 2024-04-22T23:39:08.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.966+0000 7fee9dffb700 1 -- 172.21.15.67:0/4233116087 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7feea002cce0 con 0x7fee9006ecc0 2024-04-22T23:39:08.960 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:39:08.960 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (12m) 6m ago 24m 25.5M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:39:08.960 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (24m) 6m ago 24m 9114k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:39:08.960 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (23m) 5m ago 23m 9042k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:39:08.960 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (10m) 6m ago 24m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:39:08.960 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (10m) 5m ago 23m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:39:08.960 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (12m) 6m ago 23m 79.3M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:39:08.960 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (6m) 6m ago 18m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:39:08.960 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (6m) 6m ago 18m 12.6M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:39:08.960 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (5m) 5m ago 18m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:39:08.960 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (18m) 5m ago 18m 12.2M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:39:08.960 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (14m) 6m ago 25m 511M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:39:08.960 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (13m) 5m ago 23m 453M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:39:08.961 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (11m) 6m ago 25m 55.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:39:08.961 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (10m) 5m ago 23m 46.0M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:39:08.961 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (13m) 6m ago 24m 15.4M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:39:08.961 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (13m) 5m ago 23m 16.1M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:39:08.961 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (9m) 6m ago 22m 99.2M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:39:08.961 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (9m) 6m ago 21m 127M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:39:08.961 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (8m) 6m ago 21m 79.9M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:39:08.961 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (7m) 5m ago 21m 97.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:39:08.961 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (7m) 5m ago 20m 82.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:39:08.961 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (6m) 5m ago 20m 77.6M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:39:08.961 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (13m) 6m ago 23m 73.9M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:39:08.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.968+0000 7feea6a77700 1 -- 172.21.15.67:0/4233116087 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fee9006ecc0 msgr2=0x7fee90071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:08.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.968+0000 7feea6a77700 1 --2- 172.21.15.67:0/4233116087 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fee9006ecc0 0x7fee90071180 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fee94006010 tx=0x7fee9401a040 comp rx=0 tx=0).stop 2024-04-22T23:39:08.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.968+0000 7feea6a77700 1 -- 172.21.15.67:0/4233116087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feea0100750 msgr2=0x7feea0197c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:08.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.968+0000 7feea6a77700 1 --2- 172.21.15.67:0/4233116087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feea0100750 0x7feea0197c10 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fee8c00d8d0 tx=0x7fee8c00dbe0 comp rx=0 tx=0).stop 2024-04-22T23:39:08.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.968+0000 7feea6a77700 1 -- 172.21.15.67:0/4233116087 shutdown_connections 2024-04-22T23:39:08.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.968+0000 7feea6a77700 1 --2- 172.21.15.67:0/4233116087 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fee9006ecc0 0x7fee90071180 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:08.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.968+0000 7feea6a77700 1 --2- 172.21.15.67:0/4233116087 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feea00ffda0 0x7feea01976d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:08.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.968+0000 7feea6a77700 1 --2- 172.21.15.67:0/4233116087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feea0100750 0x7feea0197c10 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:08.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.968+0000 7feea6a77700 1 -- 172.21.15.67:0/4233116087 >> 172.21.15.67:0/4233116087 conn(0x7feea00fb650 msgr2=0x7feea00fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:39:08.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.968+0000 7feea6a77700 1 -- 172.21.15.67:0/4233116087 shutdown_connections 2024-04-22T23:39:08.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:08.968+0000 7feea6a77700 1 -- 172.21.15.67:0/4233116087 wait complete. 2024-04-22T23:39:09.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.074+0000 7fbcdd3f7700 1 -- 172.21.15.67:0/1753564292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd8102700 msgr2=0x7fbcd810abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:09.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.074+0000 7fbcdd3f7700 1 --2- 172.21.15.67:0/1753564292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd8102700 0x7fbcd810abf0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fbcc0009910 tx=0x7fbcc0009c20 comp rx=0 tx=0).stop 2024-04-22T23:39:09.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.074+0000 7fbcdd3f7700 1 -- 172.21.15.67:0/1753564292 shutdown_connections 2024-04-22T23:39:09.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.074+0000 7fbcdd3f7700 1 --2- 172.21.15.67:0/1753564292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd8102700 0x7fbcd810abf0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:09.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.074+0000 7fbcdd3f7700 1 --2- 172.21.15.67:0/1753564292 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbcd8101de0 0x7fbcd81021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:09.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.074+0000 7fbcdd3f7700 1 -- 172.21.15.67:0/1753564292 >> 172.21.15.67:0/1753564292 conn(0x7fbcd80fb610 msgr2=0x7fbcd80fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:39:09.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.074+0000 7fbcdd3f7700 1 -- 172.21.15.67:0/1753564292 shutdown_connections 2024-04-22T23:39:09.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.074+0000 7fbcdd3f7700 1 -- 172.21.15.67:0/1753564292 wait complete. 2024-04-22T23:39:09.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.075+0000 7fbcdd3f7700 1 Processor -- start 2024-04-22T23:39:09.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.075+0000 7fbcdd3f7700 1 -- start start 2024-04-22T23:39:09.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.075+0000 7fbcdd3f7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd8101de0 0x7fbcd80ffa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:09.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.075+0000 7fbcdd3f7700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbcd8102700 0x7fbcd80fff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:09.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.075+0000 7fbcdd3f7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd81004c0 con 0x7fbcd8101de0 2024-04-22T23:39:09.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.075+0000 7fbcdd3f7700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd8100600 con 0x7fbcd8102700 2024-04-22T23:39:09.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.075+0000 7fbcd6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd8101de0 0x7fbcd80ffa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:09.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.075+0000 7fbcd6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd8101de0 0x7fbcd80ffa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34816/0 (socket says 172.21.15.67:34816) 2024-04-22T23:39:09.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.075+0000 7fbcd6ffd700 1 -- 172.21.15.67:0/876027220 learned_addr learned my addr 172.21.15.67:0/876027220 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:39:09.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.076+0000 7fbcd67fc700 1 --2- 172.21.15.67:0/876027220 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbcd8102700 0x7fbcd80fff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:09.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.076+0000 7fbcd6ffd700 1 -- 172.21.15.67:0/876027220 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbcd8102700 msgr2=0x7fbcd80fff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:09.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.076+0000 7fbcd6ffd700 1 --2- 172.21.15.67:0/876027220 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbcd8102700 0x7fbcd80fff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:09.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.076+0000 7fbcd6ffd700 1 -- 172.21.15.67:0/876027220 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc00095f0 con 0x7fbcd8101de0 2024-04-22T23:39:09.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.076+0000 7fbcd67fc700 1 --2- 172.21.15.67:0/876027220 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbcd8102700 0x7fbcd80fff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:39:09.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.076+0000 7fbcd6ffd700 1 --2- 172.21.15.67:0/876027220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd8101de0 0x7fbcd80ffa40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fbcd00060b0 tx=0x7fbcd000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:39:09.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.076+0000 7fbccffff700 1 -- 172.21.15.67:0/876027220 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbcd000dda0 con 0x7fbcd8101de0 2024-04-22T23:39:09.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.076+0000 7fbcdd3f7700 1 -- 172.21.15.67:0/876027220 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcd81a2aa0 con 0x7fbcd8101de0 2024-04-22T23:39:09.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.076+0000 7fbccffff700 1 -- 172.21.15.67:0/876027220 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbcd000f040 con 0x7fbcd8101de0 2024-04-22T23:39:09.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.076+0000 7fbccffff700 1 -- 172.21.15.67:0/876027220 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbcd0014790 con 0x7fbcd8101de0 2024-04-22T23:39:09.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.076+0000 7fbcdd3f7700 1 -- 172.21.15.67:0/876027220 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcd81a2e50 con 0x7fbcd8101de0 2024-04-22T23:39:09.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.077+0000 7fbcdd3f7700 1 -- 172.21.15.67:0/876027220 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcd81082f0 con 0x7fbcd8101de0 2024-04-22T23:39:09.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.078+0000 7fbccffff700 1 -- 172.21.15.67:0/876027220 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbcd0014980 con 0x7fbcd8101de0 2024-04-22T23:39:09.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.078+0000 7fbccffff700 1 --2- 172.21.15.67:0/876027220 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbcc406ecc0 0x7fbcc4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:09.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.078+0000 7fbccffff700 1 -- 172.21.15.67:0/876027220 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fbcd008e9e0 con 0x7fbcd8101de0 2024-04-22T23:39:09.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.078+0000 7fbcd67fc700 1 --2- 172.21.15.67:0/876027220 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbcc406ecc0 0x7fbcc4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:09.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.079+0000 7fbcd67fc700 1 --2- 172.21.15.67:0/876027220 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbcc406ecc0 0x7fbcc4071180 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbcc000b5c0 tx=0x7fbcc00053a0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:39:09.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.083+0000 7fbccffff700 1 -- 172.21.15.67:0/876027220 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fbcd00576d0 con 0x7fbcd8101de0 2024-04-22T23:39:09.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.385+0000 7fbcdd3f7700 1 -- 172.21.15.67:0/876027220 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fbcd8107cd0 con 0x7fbcd8101de0 2024-04-22T23:39:09.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.385+0000 7fbccffff700 1 -- 172.21.15.67:0/876027220 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fbcd005ad90 con 0x7fbcd8101de0 2024-04-22T23:39:09.379 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:39:09.379 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:39:09.379 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:39:09.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:39:09.379 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:39:09.379 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:39:09.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:39:09.380 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:39:09.380 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:39:09.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:39:09.380 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:39:09.380 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:39:09.380 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 3 2024-04-22T23:39:09.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:39:09.380 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:39:09.380 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:39:09.380 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 13 2024-04-22T23:39:09.380 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:39:09.381 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:39:09.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.387+0000 7fbcdd3f7700 1 -- 172.21.15.67:0/876027220 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbcc406ecc0 msgr2=0x7fbcc4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:09.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.387+0000 7fbcdd3f7700 1 --2- 172.21.15.67:0/876027220 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbcc406ecc0 0x7fbcc4071180 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbcc000b5c0 tx=0x7fbcc00053a0 comp rx=0 tx=0).stop 2024-04-22T23:39:09.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.388+0000 7fbcdd3f7700 1 -- 172.21.15.67:0/876027220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd8101de0 msgr2=0x7fbcd80ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:09.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.388+0000 7fbcdd3f7700 1 --2- 172.21.15.67:0/876027220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd8101de0 0x7fbcd80ffa40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fbcd00060b0 tx=0x7fbcd000d3b0 comp rx=0 tx=0).stop 2024-04-22T23:39:09.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.388+0000 7fbcdd3f7700 1 -- 172.21.15.67:0/876027220 shutdown_connections 2024-04-22T23:39:09.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.388+0000 7fbcdd3f7700 1 --2- 172.21.15.67:0/876027220 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbcc406ecc0 0x7fbcc4071180 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:09.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.388+0000 7fbcdd3f7700 1 --2- 172.21.15.67:0/876027220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd8101de0 0x7fbcd80ffa40 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:09.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.388+0000 7fbcdd3f7700 1 --2- 172.21.15.67:0/876027220 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbcd8102700 0x7fbcd80fff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:09.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.388+0000 7fbcdd3f7700 1 -- 172.21.15.67:0/876027220 >> 172.21.15.67:0/876027220 conn(0x7fbcd80fb610 msgr2=0x7fbcd8105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:39:09.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.388+0000 7fbcdd3f7700 1 -- 172.21.15.67:0/876027220 shutdown_connections 2024-04-22T23:39:09.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.388+0000 7fbcdd3f7700 1 -- 172.21.15.67:0/876027220 wait complete. 2024-04-22T23:39:09.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.496+0000 7fbbef972700 1 -- 172.21.15.67:0/3881839715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbe8105bd0 msgr2=0x7fbbe8105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:09.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.496+0000 7fbbef972700 1 --2- 172.21.15.67:0/3881839715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbe8105bd0 0x7fbbe8105fb0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fbbe4009910 tx=0x7fbbe4009c20 comp rx=0 tx=0).stop 2024-04-22T23:39:09.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.496+0000 7fbbef972700 1 -- 172.21.15.67:0/3881839715 shutdown_connections 2024-04-22T23:39:09.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.496+0000 7fbbef972700 1 --2- 172.21.15.67:0/3881839715 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbbe80ffc20 0x7fbbe8100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:09.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.496+0000 7fbbef972700 1 --2- 172.21.15.67:0/3881839715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbe8105bd0 0x7fbbe8105fb0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:09.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.496+0000 7fbbef972700 1 -- 172.21.15.67:0/3881839715 >> 172.21.15.67:0/3881839715 conn(0x7fbbe8075670 msgr2=0x7fbbe8075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:39:09.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.496+0000 7fbbef972700 1 -- 172.21.15.67:0/3881839715 shutdown_connections 2024-04-22T23:39:09.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.496+0000 7fbbef972700 1 -- 172.21.15.67:0/3881839715 wait complete. 2024-04-22T23:39:09.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.496+0000 7fbbef972700 1 Processor -- start 2024-04-22T23:39:09.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.496+0000 7fbbef972700 1 -- start start 2024-04-22T23:39:09.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.497+0000 7fbbef972700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbe80ffc20 0x7fbbe8101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:09.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.497+0000 7fbbef972700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbbe8105bd0 0x7fbbe8101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:09.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.497+0000 7fbbef972700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe81022c0 con 0x7fbbe80ffc20 2024-04-22T23:39:09.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.497+0000 7fbbef972700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe8102400 con 0x7fbbe8105bd0 2024-04-22T23:39:09.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.497+0000 7fbbed70e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbe80ffc20 0x7fbbe8101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:09.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.497+0000 7fbbed70e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbe80ffc20 0x7fbbe8101840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34838/0 (socket says 172.21.15.67:34838) 2024-04-22T23:39:09.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.497+0000 7fbbed70e700 1 -- 172.21.15.67:0/3258180196 learned_addr learned my addr 172.21.15.67:0/3258180196 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:39:09.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.497+0000 7fbbecf0d700 1 --2- 172.21.15.67:0/3258180196 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbbe8105bd0 0x7fbbe8101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:09.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.497+0000 7fbbed70e700 1 -- 172.21.15.67:0/3258180196 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbbe8105bd0 msgr2=0x7fbbe8101d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:09.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.497+0000 7fbbed70e700 1 --2- 172.21.15.67:0/3258180196 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbbe8105bd0 0x7fbbe8101d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:09.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.497+0000 7fbbed70e700 1 -- 172.21.15.67:0/3258180196 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe40095f0 con 0x7fbbe80ffc20 2024-04-22T23:39:09.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.497+0000 7fbbecf0d700 1 --2- 172.21.15.67:0/3258180196 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbbe8105bd0 0x7fbbe8101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:39:09.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.497+0000 7fbbed70e700 1 --2- 172.21.15.67:0/3258180196 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbe80ffc20 0x7fbbe8101840 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fbbe4006010 tx=0x7fbbe400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:39:09.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.497+0000 7fbbde7fc700 1 -- 172.21.15.67:0/3258180196 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbbe401d070 con 0x7fbbe80ffc20 2024-04-22T23:39:09.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.497+0000 7fbbef972700 1 -- 172.21.15.67:0/3258180196 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbe8102680 con 0x7fbbe80ffc20 2024-04-22T23:39:09.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.497+0000 7fbbde7fc700 1 -- 172.21.15.67:0/3258180196 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbbe4004fb0 con 0x7fbbe80ffc20 2024-04-22T23:39:09.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.497+0000 7fbbde7fc700 1 -- 172.21.15.67:0/3258180196 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbbe4022ea0 con 0x7fbbe80ffc20 2024-04-22T23:39:09.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.498+0000 7fbbef972700 1 -- 172.21.15.67:0/3258180196 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbe81a2d00 con 0x7fbbe80ffc20 2024-04-22T23:39:09.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.498+0000 7fbbef972700 1 -- 172.21.15.67:0/3258180196 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbe804f2e0 con 0x7fbbe80ffc20 2024-04-22T23:39:09.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.499+0000 7fbbde7fc700 1 -- 172.21.15.67:0/3258180196 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbbe4022470 con 0x7fbbe80ffc20 2024-04-22T23:39:09.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.499+0000 7fbbde7fc700 1 --2- 172.21.15.67:0/3258180196 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbbd406ecc0 0x7fbbd4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:09.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.500+0000 7fbbde7fc700 1 -- 172.21.15.67:0/3258180196 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fbbe4090570 con 0x7fbbe80ffc20 2024-04-22T23:39:09.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.500+0000 7fbbecf0d700 1 --2- 172.21.15.67:0/3258180196 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbbd406ecc0 0x7fbbd4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:09.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.500+0000 7fbbecf0d700 1 --2- 172.21.15.67:0/3258180196 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbbd406ecc0 0x7fbbd4071180 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fbbd8005fd0 tx=0x7fbbd8005e20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:39:09.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.504+0000 7fbbde7fc700 1 -- 172.21.15.67:0/3258180196 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fbbe4059260 con 0x7fbbe80ffc20 2024-04-22T23:39:09.737 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:09 smithi134 ceph-mon[85630]: pgmap v383: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:09.737 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:09 smithi134 ceph-mon[85630]: from='client.34736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:39:09.737 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:09 smithi134 ceph-mon[85630]: from='client.34740 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:39:09.737 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:09 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/876027220' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:39:09.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.756+0000 7fbbef972700 1 -- 172.21.15.67:0/3258180196 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fbbe81a2fb0 con 0x7fbbe80ffc20 2024-04-22T23:39:09.750 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:09 smithi067 ceph-mon[112438]: pgmap v383: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:09.750 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:09 smithi067 ceph-mon[112438]: from='client.34736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:39:09.751 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:09 smithi067 ceph-mon[112438]: from='client.34740 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:39:09.751 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:09 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/876027220' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:39:09.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.757+0000 7fbbde7fc700 1 -- 172.21.15.67:0/3258180196 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1871 (secure 0 0 0) 0x7fbbe4027760 con 0x7fbbe80ffc20 2024-04-22T23:39:09.751 INFO:teuthology.orchestra.run.smithi067.stdout:e31 2024-04-22T23:39:09.751 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:39:09.751 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:39:09.751 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:39:09.751 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:39:09.751 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:39:09.752 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:39:09.752 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:39:09.752 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:39:09.752 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:39:09.752 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:39:09.752 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:39:09.752 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:39:09.752 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:39:09.752 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:39:09.752 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:39:09.752 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:39:09.752 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:39:09.752 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:39:09.752 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:39:09.752 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:39:09.753 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:39:09.753 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:39:09.753 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:39:09.753 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:39:09.753 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:39:09.753 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:39:09.753 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:39:09.753 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:39:09.753 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:39:09.753 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:39:09.753 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:39:09.753 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:39:09.753 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:39:09.753 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:39:09.753 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:39:09.753 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:39:09.754 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:39:09.754 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:39:09.754 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:39:09.754 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44259} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1827288036,v1:172.21.15.134:6827/1827288036] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:39:09.754 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:39:09.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.760+0000 7fbbef972700 1 -- 172.21.15.67:0/3258180196 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbbd406ecc0 msgr2=0x7fbbd4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:09.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.760+0000 7fbbef972700 1 --2- 172.21.15.67:0/3258180196 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbbd406ecc0 0x7fbbd4071180 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fbbd8005fd0 tx=0x7fbbd8005e20 comp rx=0 tx=0).stop 2024-04-22T23:39:09.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.760+0000 7fbbef972700 1 -- 172.21.15.67:0/3258180196 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbe80ffc20 msgr2=0x7fbbe8101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:09.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.760+0000 7fbbef972700 1 --2- 172.21.15.67:0/3258180196 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbe80ffc20 0x7fbbe8101840 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fbbe4006010 tx=0x7fbbe400be20 comp rx=0 tx=0).stop 2024-04-22T23:39:09.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.760+0000 7fbbef972700 1 -- 172.21.15.67:0/3258180196 shutdown_connections 2024-04-22T23:39:09.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.760+0000 7fbbef972700 1 --2- 172.21.15.67:0/3258180196 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbbd406ecc0 0x7fbbd4071180 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:09.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.760+0000 7fbbef972700 1 --2- 172.21.15.67:0/3258180196 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbe80ffc20 0x7fbbe8101840 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:09.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.760+0000 7fbbef972700 1 --2- 172.21.15.67:0/3258180196 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbbe8105bd0 0x7fbbe8101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:09.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.760+0000 7fbbef972700 1 -- 172.21.15.67:0/3258180196 >> 172.21.15.67:0/3258180196 conn(0x7fbbe8075670 msgr2=0x7fbbe80686a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:39:09.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.760+0000 7fbbef972700 1 -- 172.21.15.67:0/3258180196 shutdown_connections 2024-04-22T23:39:09.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.760+0000 7fbbef972700 1 -- 172.21.15.67:0/3258180196 wait complete. 2024-04-22T23:39:09.755 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 31 2024-04-22T23:39:09.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.870+0000 7f542bb91700 1 -- 172.21.15.67:0/3199975509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54240ffb40 msgr2=0x7f54240fffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:09.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.870+0000 7f542bb91700 1 --2- 172.21.15.67:0/3199975509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54240ffb40 0x7f54240fffc0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f5420009910 tx=0x7f5420009c20 comp rx=0 tx=0).stop 2024-04-22T23:39:09.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.870+0000 7f542bb91700 1 -- 172.21.15.67:0/3199975509 shutdown_connections 2024-04-22T23:39:09.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.870+0000 7f542bb91700 1 --2- 172.21.15.67:0/3199975509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54240ffb40 0x7f54240fffc0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:09.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.870+0000 7f542bb91700 1 --2- 172.21.15.67:0/3199975509 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5424105bf0 0x7f5424105fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:09.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.870+0000 7f542bb91700 1 -- 172.21.15.67:0/3199975509 >> 172.21.15.67:0/3199975509 conn(0x7f54240fb650 msgr2=0x7f54240fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:39:09.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.870+0000 7f542bb91700 1 -- 172.21.15.67:0/3199975509 shutdown_connections 2024-04-22T23:39:09.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.870+0000 7f542bb91700 1 -- 172.21.15.67:0/3199975509 wait complete. 2024-04-22T23:39:09.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.871+0000 7f542bb91700 1 Processor -- start 2024-04-22T23:39:09.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.871+0000 7f542bb91700 1 -- start start 2024-04-22T23:39:09.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.871+0000 7f542bb91700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54240ffb40 0x7f542419e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:09.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.871+0000 7f542bb91700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5424105bf0 0x7f542419ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:09.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.871+0000 7f542bb91700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f542419f370 con 0x7f54240ffb40 2024-04-22T23:39:09.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.871+0000 7f542bb91700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5424198820 con 0x7f5424105bf0 2024-04-22T23:39:09.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.871+0000 7f542992d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54240ffb40 0x7f542419e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:09.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.871+0000 7f542992d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54240ffb40 0x7f542419e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34848/0 (socket says 172.21.15.67:34848) 2024-04-22T23:39:09.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.871+0000 7f542992d700 1 -- 172.21.15.67:0/4267392286 learned_addr learned my addr 172.21.15.67:0/4267392286 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:39:09.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.872+0000 7f542912c700 1 --2- 172.21.15.67:0/4267392286 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5424105bf0 0x7f542419ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:09.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.872+0000 7f542992d700 1 -- 172.21.15.67:0/4267392286 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5424105bf0 msgr2=0x7f542419ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:09.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.872+0000 7f542992d700 1 --2- 172.21.15.67:0/4267392286 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5424105bf0 0x7f542419ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:09.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.872+0000 7f542992d700 1 -- 172.21.15.67:0/4267392286 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54200095f0 con 0x7f54240ffb40 2024-04-22T23:39:09.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.872+0000 7f542992d700 1 --2- 172.21.15.67:0/4267392286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54240ffb40 0x7f542419e7a0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f541400d8d0 tx=0x7f541400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:39:09.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.872+0000 7f541affd700 1 -- 172.21.15.67:0/4267392286 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5414009940 con 0x7f54240ffb40 2024-04-22T23:39:09.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.872+0000 7f541affd700 1 -- 172.21.15.67:0/4267392286 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5414010460 con 0x7f54240ffb40 2024-04-22T23:39:09.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.872+0000 7f542bb91700 1 -- 172.21.15.67:0/4267392286 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5424198b00 con 0x7f54240ffb40 2024-04-22T23:39:09.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.872+0000 7f541affd700 1 -- 172.21.15.67:0/4267392286 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f541400f5f0 con 0x7f54240ffb40 2024-04-22T23:39:09.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.872+0000 7f542bb91700 1 -- 172.21.15.67:0/4267392286 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5424199050 con 0x7f54240ffb40 2024-04-22T23:39:09.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.873+0000 7f542bb91700 1 -- 172.21.15.67:0/4267392286 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f542404f2e0 con 0x7f54240ffb40 2024-04-22T23:39:09.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.874+0000 7f541affd700 1 -- 172.21.15.67:0/4267392286 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f54140105d0 con 0x7f54240ffb40 2024-04-22T23:39:09.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.874+0000 7f541affd700 1 --2- 172.21.15.67:0/4267392286 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f541006ecc0 0x7f5410071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:09.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.874+0000 7f542912c700 1 --2- 172.21.15.67:0/4267392286 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f541006ecc0 0x7f5410071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:09.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.874+0000 7f541affd700 1 -- 172.21.15.67:0/4267392286 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f541408eff0 con 0x7f54240ffb40 2024-04-22T23:39:09.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.875+0000 7f542912c700 1 --2- 172.21.15.67:0/4267392286 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f541006ecc0 0x7f5410071180 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f542000b5c0 tx=0x7f5420005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:39:09.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:09.879+0000 7f541affd700 1 -- 172.21.15.67:0/4267392286 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f541405b760 con 0x7f54240ffb40 2024-04-22T23:39:10.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.111+0000 7f542bb91700 1 -- 172.21.15.67:0/4267392286 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5424199380 con 0x7f541006ecc0 2024-04-22T23:39:10.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.113+0000 7f541affd700 1 -- 172.21.15.67:0/4267392286 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f5424199380 con 0x7f541006ecc0 2024-04-22T23:39:10.106 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:39:10.107 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:39:10.107 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:39:10.107 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:39:10.107 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:39:10.107 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:39:10.107 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:39:10.107 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:39:10.107 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:39:10.107 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:39:10.107 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T23:39:10.107 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:39:10.107 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:39:10.107 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:39:10.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.115+0000 7f542bb91700 1 -- 172.21.15.67:0/4267392286 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f541006ecc0 msgr2=0x7f5410071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:10.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.115+0000 7f542bb91700 1 --2- 172.21.15.67:0/4267392286 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f541006ecc0 0x7f5410071180 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f542000b5c0 tx=0x7f5420005960 comp rx=0 tx=0).stop 2024-04-22T23:39:10.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.115+0000 7f542bb91700 1 -- 172.21.15.67:0/4267392286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54240ffb40 msgr2=0x7f542419e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:10.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.115+0000 7f542bb91700 1 --2- 172.21.15.67:0/4267392286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54240ffb40 0x7f542419e7a0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f541400d8d0 tx=0x7f541400dc90 comp rx=0 tx=0).stop 2024-04-22T23:39:10.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.115+0000 7f542bb91700 1 -- 172.21.15.67:0/4267392286 shutdown_connections 2024-04-22T23:39:10.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.115+0000 7f542bb91700 1 --2- 172.21.15.67:0/4267392286 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f541006ecc0 0x7f5410071180 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:10.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.115+0000 7f542bb91700 1 --2- 172.21.15.67:0/4267392286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54240ffb40 0x7f542419e7a0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:10.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.115+0000 7f542bb91700 1 --2- 172.21.15.67:0/4267392286 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5424105bf0 0x7f542419ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:10.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.115+0000 7f542bb91700 1 -- 172.21.15.67:0/4267392286 >> 172.21.15.67:0/4267392286 conn(0x7f54240fb650 msgr2=0x7f5424068ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:39:10.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.115+0000 7f542bb91700 1 -- 172.21.15.67:0/4267392286 shutdown_connections 2024-04-22T23:39:10.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.115+0000 7f542bb91700 1 -- 172.21.15.67:0/4267392286 wait complete. 2024-04-22T23:39:10.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.223+0000 7fa80f935700 1 -- 172.21.15.67:0/2398052237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa808068580 msgr2=0x7fa8080689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:10.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.223+0000 7fa80f935700 1 --2- 172.21.15.67:0/2398052237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa808068580 0x7fa8080689e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa804009910 tx=0x7fa804009c20 comp rx=0 tx=0).stop 2024-04-22T23:39:10.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.223+0000 7fa80f935700 1 -- 172.21.15.67:0/2398052237 shutdown_connections 2024-04-22T23:39:10.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.223+0000 7fa80f935700 1 --2- 172.21.15.67:0/2398052237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa808068580 0x7fa8080689e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:10.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.223+0000 7fa80f935700 1 --2- 172.21.15.67:0/2398052237 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa808106cd0 0x7fa8081070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:10.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.223+0000 7fa80f935700 1 -- 172.21.15.67:0/2398052237 >> 172.21.15.67:0/2398052237 conn(0x7fa808075670 msgr2=0x7fa808075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:39:10.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.223+0000 7fa80f935700 1 -- 172.21.15.67:0/2398052237 shutdown_connections 2024-04-22T23:39:10.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.223+0000 7fa80f935700 1 -- 172.21.15.67:0/2398052237 wait complete. 2024-04-22T23:39:10.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.224+0000 7fa80f935700 1 Processor -- start 2024-04-22T23:39:10.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.224+0000 7fa80f935700 1 -- start start 2024-04-22T23:39:10.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.224+0000 7fa80f935700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa808068580 0x7fa80819c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:10.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.224+0000 7fa80f935700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa808106cd0 0x7fa80819cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:10.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.224+0000 7fa80f935700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa80819d200 con 0x7fa808106cd0 2024-04-22T23:39:10.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.224+0000 7fa80f935700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8081966b0 con 0x7fa808068580 2024-04-22T23:39:10.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.224+0000 7fa80ced0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa808106cd0 0x7fa80819cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:10.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.224+0000 7fa80ced0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa808106cd0 0x7fa80819cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34866/0 (socket says 172.21.15.67:34866) 2024-04-22T23:39:10.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.224+0000 7fa80ced0700 1 -- 172.21.15.67:0/1289415060 learned_addr learned my addr 172.21.15.67:0/1289415060 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:39:10.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.224+0000 7fa80d6d1700 1 --2- 172.21.15.67:0/1289415060 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa808068580 0x7fa80819c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:10.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.225+0000 7fa80ced0700 1 -- 172.21.15.67:0/1289415060 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa808068580 msgr2=0x7fa80819c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:10.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.225+0000 7fa80ced0700 1 --2- 172.21.15.67:0/1289415060 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa808068580 0x7fa80819c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:10.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.225+0000 7fa80ced0700 1 -- 172.21.15.67:0/1289415060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8040095f0 con 0x7fa808106cd0 2024-04-22T23:39:10.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.225+0000 7fa80ced0700 1 --2- 172.21.15.67:0/1289415060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa808106cd0 0x7fa80819cb70 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa80400b5c0 tx=0x7fa80400be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:39:10.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.225+0000 7fa7fa7fc700 1 -- 172.21.15.67:0/1289415060 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa80401d070 con 0x7fa808106cd0 2024-04-22T23:39:10.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.225+0000 7fa80f935700 1 -- 172.21.15.67:0/1289415060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa808196930 con 0x7fa808106cd0 2024-04-22T23:39:10.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.225+0000 7fa7fa7fc700 1 -- 172.21.15.67:0/1289415060 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa80400f460 con 0x7fa808106cd0 2024-04-22T23:39:10.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.225+0000 7fa7fa7fc700 1 -- 172.21.15.67:0/1289415060 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa804017610 con 0x7fa808106cd0 2024-04-22T23:39:10.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.225+0000 7fa80f935700 1 -- 172.21.15.67:0/1289415060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa808196e20 con 0x7fa808106cd0 2024-04-22T23:39:10.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.226+0000 7fa80f935700 1 -- 172.21.15.67:0/1289415060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8081091d0 con 0x7fa808106cd0 2024-04-22T23:39:10.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.227+0000 7fa7fa7fc700 1 -- 172.21.15.67:0/1289415060 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa80400f5d0 con 0x7fa808106cd0 2024-04-22T23:39:10.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.227+0000 7fa7fa7fc700 1 --2- 172.21.15.67:0/1289415060 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa7f406ecc0 0x7fa7f4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:10.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.227+0000 7fa7fa7fc700 1 -- 172.21.15.67:0/1289415060 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fa8040907b0 con 0x7fa808106cd0 2024-04-22T23:39:10.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.227+0000 7fa80d6d1700 1 --2- 172.21.15.67:0/1289415060 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa7f406ecc0 0x7fa7f4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:10.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.228+0000 7fa80d6d1700 1 --2- 172.21.15.67:0/1289415060 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa7f406ecc0 0x7fa7f4071180 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc005fd0 tx=0x7fa7fc00a560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:39:10.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.231+0000 7fa7fa7fc700 1 -- 172.21.15.67:0/1289415060 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fa80405cfa0 con 0x7fa808106cd0 2024-04-22T23:39:10.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.532+0000 7fa80f935700 1 -- 172.21.15.67:0/1289415060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa80804f2e0 con 0x7fa808106cd0 2024-04-22T23:39:10.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.533+0000 7fa7fa7fc700 1 -- 172.21.15.67:0/1289415060 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fa80405cb30 con 0x7fa808106cd0 2024-04-22T23:39:10.526 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:39:10.527 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:39:10.527 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:39:10.527 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:39:10.527 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:39:10.527 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:39:10.527 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:39:10.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.535+0000 7fa80f935700 1 -- 172.21.15.67:0/1289415060 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa7f406ecc0 msgr2=0x7fa7f4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:10.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.535+0000 7fa80f935700 1 --2- 172.21.15.67:0/1289415060 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa7f406ecc0 0x7fa7f4071180 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc005fd0 tx=0x7fa7fc00a560 comp rx=0 tx=0).stop 2024-04-22T23:39:10.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.535+0000 7fa80f935700 1 -- 172.21.15.67:0/1289415060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa808106cd0 msgr2=0x7fa80819cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:10.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.535+0000 7fa80f935700 1 --2- 172.21.15.67:0/1289415060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa808106cd0 0x7fa80819cb70 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa80400b5c0 tx=0x7fa80400be00 comp rx=0 tx=0).stop 2024-04-22T23:39:10.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.535+0000 7fa80f935700 1 -- 172.21.15.67:0/1289415060 shutdown_connections 2024-04-22T23:39:10.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.535+0000 7fa80f935700 1 --2- 172.21.15.67:0/1289415060 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa7f406ecc0 0x7fa7f4071180 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:10.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.535+0000 7fa80f935700 1 --2- 172.21.15.67:0/1289415060 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa808068580 0x7fa80819c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:10.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.535+0000 7fa80f935700 1 --2- 172.21.15.67:0/1289415060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa808106cd0 0x7fa80819cb70 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:10.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.535+0000 7fa80f935700 1 -- 172.21.15.67:0/1289415060 >> 172.21.15.67:0/1289415060 conn(0x7fa808075670 msgr2=0x7fa8080feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:39:10.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.535+0000 7fa80f935700 1 -- 172.21.15.67:0/1289415060 shutdown_connections 2024-04-22T23:39:10.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:10.535+0000 7fa80f935700 1 -- 172.21.15.67:0/1289415060 wait complete. 2024-04-22T23:39:10.736 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:10 smithi134 ceph-mon[85630]: from='client.34744 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:39:10.737 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:10 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3258180196' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:39:10.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:10 smithi067 ceph-mon[112438]: from='client.34744 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:39:10.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:10 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3258180196' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:39:11.737 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:11 smithi134 ceph-mon[85630]: from='client.34756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:39:11.737 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:11 smithi134 ceph-mon[85630]: pgmap v384: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:11.737 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:11 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1289415060' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:39:11.737 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:39:11.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:11 smithi067 ceph-mon[112438]: from='client.34756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:39:11.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:11 smithi067 ceph-mon[112438]: pgmap v384: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:11.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:11 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1289415060' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:39:11.788 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:39:13.736 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:13 smithi134 ceph-mon[85630]: pgmap v385: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:13.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:13 smithi067 ceph-mon[112438]: pgmap v385: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:15.737 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:15 smithi134 ceph-mon[85630]: pgmap v386: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:15.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:15 smithi067 ceph-mon[112438]: pgmap v386: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:17.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:17 smithi067 ceph-mon[112438]: pgmap v387: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:17.987 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:17 smithi134 ceph-mon[85630]: pgmap v387: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:19.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:19 smithi067 ceph-mon[112438]: pgmap v388: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:19.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:19 smithi134 ceph-mon[85630]: pgmap v388: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:21.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:21 smithi067 ceph-mon[112438]: pgmap v389: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:21.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:21 smithi134 ceph-mon[85630]: pgmap v389: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:23.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:23 smithi067 ceph-mon[112438]: pgmap v390: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:23.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:23 smithi134 ceph-mon[85630]: pgmap v390: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:25.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:25 smithi067 ceph-mon[112438]: pgmap v391: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:25.987 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:25 smithi134 ceph-mon[85630]: pgmap v391: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:26.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:39:26.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:39:27.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:27 smithi067 ceph-mon[112438]: pgmap v392: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:27.987 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:27 smithi134 ceph-mon[85630]: pgmap v392: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:29.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:29 smithi067 ceph-mon[112438]: pgmap v393: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:29.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:29 smithi134 ceph-mon[85630]: pgmap v393: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:31.779 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:31 smithi067 ceph-mon[112438]: pgmap v394: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:31.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:31 smithi134 ceph-mon[85630]: pgmap v394: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:33.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:33 smithi067 ceph-mon[112438]: pgmap v395: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:33.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:33 smithi134 ceph-mon[85630]: pgmap v395: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:35.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:35 smithi067 ceph-mon[112438]: pgmap v396: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:35.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:35 smithi134 ceph-mon[85630]: pgmap v396: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:37.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:37 smithi067 ceph-mon[112438]: pgmap v397: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:37.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:37 smithi134 ceph-mon[85630]: pgmap v397: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:39.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:39 smithi067 ceph-mon[112438]: pgmap v398: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:39.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:39 smithi134 ceph-mon[85630]: pgmap v398: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:40.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.656+0000 7f00bd70f700 1 -- 172.21.15.67:0/932073443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00b8100750 msgr2=0x7f00b81047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:40.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.656+0000 7f00bd70f700 1 --2- 172.21.15.67:0/932073443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00b8100750 0x7f00b81047a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f00a0009910 tx=0x7f00a0009c20 comp rx=0 tx=0).stop 2024-04-22T23:39:40.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.656+0000 7f00bd70f700 1 -- 172.21.15.67:0/932073443 shutdown_connections 2024-04-22T23:39:40.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.656+0000 7f00bd70f700 1 --2- 172.21.15.67:0/932073443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00b8100750 0x7f00b81047a0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:40.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.656+0000 7f00bd70f700 1 --2- 172.21.15.67:0/932073443 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f00b80ffda0 0x7f00b8100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:40.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.656+0000 7f00bd70f700 1 -- 172.21.15.67:0/932073443 >> 172.21.15.67:0/932073443 conn(0x7f00b80fb650 msgr2=0x7f00b80fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:39:40.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.656+0000 7f00bd70f700 1 -- 172.21.15.67:0/932073443 shutdown_connections 2024-04-22T23:39:40.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.656+0000 7f00bd70f700 1 -- 172.21.15.67:0/932073443 wait complete. 2024-04-22T23:39:40.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.657+0000 7f00bd70f700 1 Processor -- start 2024-04-22T23:39:40.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.657+0000 7f00bd70f700 1 -- start start 2024-04-22T23:39:40.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.657+0000 7f00bd70f700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f00b80ffda0 0x7f00b81977c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:40.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.657+0000 7f00bd70f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00b8100750 0x7f00b8197d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:40.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.657+0000 7f00bd70f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b8198320 con 0x7f00b8100750 2024-04-22T23:39:40.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.657+0000 7f00bd70f700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b8198460 con 0x7f00b80ffda0 2024-04-22T23:39:40.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.657+0000 7f00affff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00b8100750 0x7f00b8197d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:40.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.657+0000 7f00affff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00b8100750 0x7f00b8197d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43586/0 (socket says 172.21.15.67:43586) 2024-04-22T23:39:40.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.657+0000 7f00affff700 1 -- 172.21.15.67:0/3356491848 learned_addr learned my addr 172.21.15.67:0/3356491848 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:39:40.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.658+0000 7f00b6ffd700 1 --2- 172.21.15.67:0/3356491848 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f00b80ffda0 0x7f00b81977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:40.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.658+0000 7f00affff700 1 -- 172.21.15.67:0/3356491848 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f00b80ffda0 msgr2=0x7f00b81977c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:40.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.658+0000 7f00affff700 1 --2- 172.21.15.67:0/3356491848 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f00b80ffda0 0x7f00b81977c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:40.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.658+0000 7f00affff700 1 -- 172.21.15.67:0/3356491848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a00095f0 con 0x7f00b8100750 2024-04-22T23:39:40.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.658+0000 7f00b6ffd700 1 --2- 172.21.15.67:0/3356491848 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f00b80ffda0 0x7f00b81977c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:39:40.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.658+0000 7f00affff700 1 --2- 172.21.15.67:0/3356491848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00b8100750 0x7f00b8197d00 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f00a00059d0 tx=0x7f00a000f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:39:40.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.658+0000 7f00b4ff9700 1 -- 172.21.15.67:0/3356491848 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00a001d070 con 0x7f00b8100750 2024-04-22T23:39:40.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.658+0000 7f00bd70f700 1 -- 172.21.15.67:0/3356491848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00b81a0830 con 0x7f00b8100750 2024-04-22T23:39:40.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.658+0000 7f00b4ff9700 1 -- 172.21.15.67:0/3356491848 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f00a000b610 con 0x7f00b8100750 2024-04-22T23:39:40.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.658+0000 7f00b4ff9700 1 -- 172.21.15.67:0/3356491848 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00a0022470 con 0x7f00b8100750 2024-04-22T23:39:40.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.658+0000 7f00bd70f700 1 -- 172.21.15.67:0/3356491848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00b81a0c40 con 0x7f00b8100750 2024-04-22T23:39:40.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.659+0000 7f00bd70f700 1 -- 172.21.15.67:0/3356491848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00b81081a0 con 0x7f00b8100750 2024-04-22T23:39:40.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.660+0000 7f00b4ff9700 1 -- 172.21.15.67:0/3356491848 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f00a000fc70 con 0x7f00b8100750 2024-04-22T23:39:40.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.660+0000 7f00b4ff9700 1 --2- 172.21.15.67:0/3356491848 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f00a406ecc0 0x7f00a4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:40.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.660+0000 7f00b4ff9700 1 -- 172.21.15.67:0/3356491848 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f00a0090420 con 0x7f00b8100750 2024-04-22T23:39:40.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.660+0000 7f00b6ffd700 1 --2- 172.21.15.67:0/3356491848 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f00a406ecc0 0x7f00a4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:40.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.661+0000 7f00b6ffd700 1 --2- 172.21.15.67:0/3356491848 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f00a406ecc0 0x7f00a4071180 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f00a8009be0 tx=0x7f00a8009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:39:40.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.665+0000 7f00b4ff9700 1 -- 172.21.15.67:0/3356491848 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f00a0059060 con 0x7f00b8100750 2024-04-22T23:39:40.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.893+0000 7f00bd70f700 1 -- 172.21.15.67:0/3356491848 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f00b80611d0 con 0x7f00a406ecc0 2024-04-22T23:39:40.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.895+0000 7f00b4ff9700 1 -- 172.21.15.67:0/3356491848 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f00b80611d0 con 0x7f00a406ecc0 2024-04-22T23:39:40.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.897+0000 7f00bd70f700 1 -- 172.21.15.67:0/3356491848 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f00a406ecc0 msgr2=0x7f00a4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:40.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.897+0000 7f00bd70f700 1 --2- 172.21.15.67:0/3356491848 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f00a406ecc0 0x7f00a4071180 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f00a8009be0 tx=0x7f00a8009260 comp rx=0 tx=0).stop 2024-04-22T23:39:40.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.897+0000 7f00bd70f700 1 -- 172.21.15.67:0/3356491848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00b8100750 msgr2=0x7f00b8197d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:40.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.897+0000 7f00bd70f700 1 --2- 172.21.15.67:0/3356491848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00b8100750 0x7f00b8197d00 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f00a00059d0 tx=0x7f00a000f740 comp rx=0 tx=0).stop 2024-04-22T23:39:40.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.897+0000 7f00bd70f700 1 -- 172.21.15.67:0/3356491848 shutdown_connections 2024-04-22T23:39:40.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.897+0000 7f00bd70f700 1 --2- 172.21.15.67:0/3356491848 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f00a406ecc0 0x7f00a4071180 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:40.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.897+0000 7f00bd70f700 1 --2- 172.21.15.67:0/3356491848 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f00b80ffda0 0x7f00b81977c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:40.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.897+0000 7f00bd70f700 1 --2- 172.21.15.67:0/3356491848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00b8100750 0x7f00b8197d00 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:40.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.897+0000 7f00bd70f700 1 -- 172.21.15.67:0/3356491848 >> 172.21.15.67:0/3356491848 conn(0x7f00b80fb650 msgr2=0x7f00b80fccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:39:40.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.897+0000 7f00bd70f700 1 -- 172.21.15.67:0/3356491848 shutdown_connections 2024-04-22T23:39:40.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:40.897+0000 7f00bd70f700 1 -- 172.21.15.67:0/3356491848 wait complete. 2024-04-22T23:39:40.902 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:39:41.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.006+0000 7f7d9eb4e700 1 -- 172.21.15.67:0/3634279882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d98100ea0 msgr2=0x7f7d98101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:41.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.006+0000 7f7d9eb4e700 1 --2- 172.21.15.67:0/3634279882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d98100ea0 0x7f7d98101280 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c009910 tx=0x7f7d8c009c20 comp rx=0 tx=0).stop 2024-04-22T23:39:41.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.006+0000 7f7d9eb4e700 1 -- 172.21.15.67:0/3634279882 shutdown_connections 2024-04-22T23:39:41.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.006+0000 7f7d9eb4e700 1 --2- 172.21.15.67:0/3634279882 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d98101850 0x7f7d981058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:41.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.006+0000 7f7d9eb4e700 1 --2- 172.21.15.67:0/3634279882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d98100ea0 0x7f7d98101280 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:41.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.006+0000 7f7d9eb4e700 1 -- 172.21.15.67:0/3634279882 >> 172.21.15.67:0/3634279882 conn(0x7f7d980fc750 msgr2=0x7f7d980feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:39:41.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.006+0000 7f7d9eb4e700 1 -- 172.21.15.67:0/3634279882 shutdown_connections 2024-04-22T23:39:41.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.006+0000 7f7d9eb4e700 1 -- 172.21.15.67:0/3634279882 wait complete. 2024-04-22T23:39:41.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.006+0000 7f7d9eb4e700 1 Processor -- start 2024-04-22T23:39:41.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.006+0000 7f7d9eb4e700 1 -- start start 2024-04-22T23:39:41.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.007+0000 7f7d9eb4e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d98100ea0 0x7f7d9819c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:41.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.007+0000 7f7d9eb4e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d98101850 0x7f7d9819cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:41.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.007+0000 7f7d9eb4e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9819d2c0 con 0x7f7d98100ea0 2024-04-22T23:39:41.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.007+0000 7f7d9eb4e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d98196770 con 0x7f7d98101850 2024-04-22T23:39:41.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.007+0000 7f7d9c8ea700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d98100ea0 0x7f7d9819c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:41.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.007+0000 7f7d9c8ea700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d98100ea0 0x7f7d9819c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43602/0 (socket says 172.21.15.67:43602) 2024-04-22T23:39:41.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.007+0000 7f7d9c8ea700 1 -- 172.21.15.67:0/302139703 learned_addr learned my addr 172.21.15.67:0/302139703 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:39:41.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.007+0000 7f7d97fff700 1 --2- 172.21.15.67:0/302139703 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d98101850 0x7f7d9819cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:41.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.007+0000 7f7d9c8ea700 1 -- 172.21.15.67:0/302139703 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d98101850 msgr2=0x7f7d9819cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:41.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.007+0000 7f7d9c8ea700 1 --2- 172.21.15.67:0/302139703 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d98101850 0x7f7d9819cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:41.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.007+0000 7f7d9c8ea700 1 -- 172.21.15.67:0/302139703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d8c0095f0 con 0x7f7d98100ea0 2024-04-22T23:39:41.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.007+0000 7f7d9c8ea700 1 --2- 172.21.15.67:0/302139703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d98100ea0 0x7f7d9819c6f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c006010 tx=0x7f7d8c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:39:41.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.007+0000 7f7d95ffb700 1 -- 172.21.15.67:0/302139703 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d8c01d070 con 0x7f7d98100ea0 2024-04-22T23:39:41.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.007+0000 7f7d95ffb700 1 -- 172.21.15.67:0/302139703 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7d8c004fb0 con 0x7f7d98100ea0 2024-04-22T23:39:41.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.007+0000 7f7d9eb4e700 1 -- 172.21.15.67:0/302139703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d981969f0 con 0x7f7d98100ea0 2024-04-22T23:39:41.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.008+0000 7f7d95ffb700 1 -- 172.21.15.67:0/302139703 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d8c022ea0 con 0x7f7d98100ea0 2024-04-22T23:39:41.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.008+0000 7f7d9eb4e700 1 -- 172.21.15.67:0/302139703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d98196ee0 con 0x7f7d98100ea0 2024-04-22T23:39:41.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.009+0000 7f7d9eb4e700 1 -- 172.21.15.67:0/302139703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d98109240 con 0x7f7d98100ea0 2024-04-22T23:39:41.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.009+0000 7f7d95ffb700 1 -- 172.21.15.67:0/302139703 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7d8c022470 con 0x7f7d98100ea0 2024-04-22T23:39:41.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.009+0000 7f7d95ffb700 1 --2- 172.21.15.67:0/302139703 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7d8806ecc0 0x7f7d88071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:41.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.010+0000 7f7d95ffb700 1 -- 172.21.15.67:0/302139703 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f7d8c0904c0 con 0x7f7d98100ea0 2024-04-22T23:39:41.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.010+0000 7f7d97fff700 1 --2- 172.21.15.67:0/302139703 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7d8806ecc0 0x7f7d88071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:41.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.010+0000 7f7d97fff700 1 --2- 172.21.15.67:0/302139703 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7d8806ecc0 0x7f7d88071180 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7d84005fd0 tx=0x7f7d8400a560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:39:41.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.014+0000 7f7d95ffb700 1 -- 172.21.15.67:0/302139703 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7d8c05cd60 con 0x7f7d98100ea0 2024-04-22T23:39:41.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.253+0000 7f7d9eb4e700 1 -- 172.21.15.67:0/302139703 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7d98197150 con 0x7f7d8806ecc0 2024-04-22T23:39:41.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.254+0000 7f7d95ffb700 1 -- 172.21.15.67:0/302139703 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f7d98197150 con 0x7f7d8806ecc0 2024-04-22T23:39:41.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.256+0000 7f7d9eb4e700 1 -- 172.21.15.67:0/302139703 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7d8806ecc0 msgr2=0x7f7d88071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:41.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.256+0000 7f7d9eb4e700 1 --2- 172.21.15.67:0/302139703 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7d8806ecc0 0x7f7d88071180 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7d84005fd0 tx=0x7f7d8400a560 comp rx=0 tx=0).stop 2024-04-22T23:39:41.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.256+0000 7f7d9eb4e700 1 -- 172.21.15.67:0/302139703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d98100ea0 msgr2=0x7f7d9819c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:41.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.256+0000 7f7d9eb4e700 1 --2- 172.21.15.67:0/302139703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d98100ea0 0x7f7d9819c6f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c006010 tx=0x7f7d8c00be20 comp rx=0 tx=0).stop 2024-04-22T23:39:41.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.256+0000 7f7d9eb4e700 1 -- 172.21.15.67:0/302139703 shutdown_connections 2024-04-22T23:39:41.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.256+0000 7f7d9eb4e700 1 --2- 172.21.15.67:0/302139703 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7d8806ecc0 0x7f7d88071180 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:41.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.256+0000 7f7d9eb4e700 1 --2- 172.21.15.67:0/302139703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d98100ea0 0x7f7d9819c6f0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:41.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.256+0000 7f7d9eb4e700 1 --2- 172.21.15.67:0/302139703 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d98101850 0x7f7d9819cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:41.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.256+0000 7f7d9eb4e700 1 -- 172.21.15.67:0/302139703 >> 172.21.15.67:0/302139703 conn(0x7f7d980fc750 msgr2=0x7f7d980fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:39:41.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.257+0000 7f7d9eb4e700 1 -- 172.21.15.67:0/302139703 shutdown_connections 2024-04-22T23:39:41.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.257+0000 7f7d9eb4e700 1 -- 172.21.15.67:0/302139703 wait complete. 2024-04-22T23:39:41.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.371+0000 7fe47625e700 1 -- 172.21.15.67:0/519964833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe470101850 msgr2=0x7fe4701058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:41.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.371+0000 7fe47625e700 1 --2- 172.21.15.67:0/519964833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe470101850 0x7fe4701058a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe464009910 tx=0x7fe464009c20 comp rx=0 tx=0).stop 2024-04-22T23:39:41.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.372+0000 7fe47625e700 1 -- 172.21.15.67:0/519964833 shutdown_connections 2024-04-22T23:39:41.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.372+0000 7fe47625e700 1 --2- 172.21.15.67:0/519964833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe470101850 0x7fe4701058a0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:41.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.372+0000 7fe47625e700 1 --2- 172.21.15.67:0/519964833 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe470100ea0 0x7fe470101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:41.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.372+0000 7fe47625e700 1 -- 172.21.15.67:0/519964833 >> 172.21.15.67:0/519964833 conn(0x7fe4700fc750 msgr2=0x7fe4700feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:39:41.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.372+0000 7fe47625e700 1 -- 172.21.15.67:0/519964833 shutdown_connections 2024-04-22T23:39:41.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.372+0000 7fe47625e700 1 -- 172.21.15.67:0/519964833 wait complete. 2024-04-22T23:39:41.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.372+0000 7fe47625e700 1 Processor -- start 2024-04-22T23:39:41.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.372+0000 7fe47625e700 1 -- start start 2024-04-22T23:39:41.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.373+0000 7fe47625e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe470100ea0 0x7fe470073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:41.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.373+0000 7fe47625e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe470101850 0x7fe4700735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:41.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.373+0000 7fe47625e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe470073b10 con 0x7fe470100ea0 2024-04-22T23:39:41.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.373+0000 7fe47625e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe470073c50 con 0x7fe470101850 2024-04-22T23:39:41.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.373+0000 7fe46f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe470100ea0 0x7fe470073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:41.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.373+0000 7fe46f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe470100ea0 0x7fe470073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43622/0 (socket says 172.21.15.67:43622) 2024-04-22T23:39:41.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.373+0000 7fe46f7fe700 1 -- 172.21.15.67:0/3560186446 learned_addr learned my addr 172.21.15.67:0/3560186446 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:39:41.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.373+0000 7fe46effd700 1 --2- 172.21.15.67:0/3560186446 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe470101850 0x7fe4700735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:41.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.373+0000 7fe46f7fe700 1 -- 172.21.15.67:0/3560186446 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe470101850 msgr2=0x7fe4700735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:41.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.373+0000 7fe46f7fe700 1 --2- 172.21.15.67:0/3560186446 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe470101850 0x7fe4700735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:41.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.373+0000 7fe46f7fe700 1 -- 172.21.15.67:0/3560186446 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4640095f0 con 0x7fe470100ea0 2024-04-22T23:39:41.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.373+0000 7fe46effd700 1 --2- 172.21.15.67:0/3560186446 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe470101850 0x7fe4700735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T23:39:41.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.373+0000 7fe46f7fe700 1 --2- 172.21.15.67:0/3560186446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe470100ea0 0x7fe470073090 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe46000d8d0 tx=0x7fe46000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:39:41.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.373+0000 7fe46cff9700 1 -- 172.21.15.67:0/3560186446 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4600098a0 con 0x7fe470100ea0 2024-04-22T23:39:41.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.374+0000 7fe46cff9700 1 -- 172.21.15.67:0/3560186446 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe460010460 con 0x7fe470100ea0 2024-04-22T23:39:41.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.374+0000 7fe47625e700 1 -- 172.21.15.67:0/3560186446 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4701a2b00 con 0x7fe470100ea0 2024-04-22T23:39:41.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.374+0000 7fe46cff9700 1 -- 172.21.15.67:0/3560186446 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe46000f5f0 con 0x7fe470100ea0 2024-04-22T23:39:41.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.374+0000 7fe47625e700 1 -- 172.21.15.67:0/3560186446 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4701a2f10 con 0x7fe470100ea0 2024-04-22T23:39:41.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.375+0000 7fe47625e700 1 -- 172.21.15.67:0/3560186446 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4701092a0 con 0x7fe470100ea0 2024-04-22T23:39:41.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.375+0000 7fe46cff9700 1 -- 172.21.15.67:0/3560186446 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe460009a00 con 0x7fe470100ea0 2024-04-22T23:39:41.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.376+0000 7fe46cff9700 1 --2- 172.21.15.67:0/3560186446 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe45806ecc0 0x7fe458071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:41.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.376+0000 7fe46cff9700 1 -- 172.21.15.67:0/3560186446 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fe46008eea0 con 0x7fe470100ea0 2024-04-22T23:39:41.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.376+0000 7fe46effd700 1 --2- 172.21.15.67:0/3560186446 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe45806ecc0 0x7fe458071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:41.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.376+0000 7fe46effd700 1 --2- 172.21.15.67:0/3560186446 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe45806ecc0 0x7fe458071180 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe470074880 tx=0x7fe46401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:39:41.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.380+0000 7fe46cff9700 1 -- 172.21.15.67:0/3560186446 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe460057ae0 con 0x7fe470100ea0 2024-04-22T23:39:41.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.598+0000 7fe47625e700 1 -- 172.21.15.67:0/3560186446 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe4700611d0 con 0x7fe45806ecc0 2024-04-22T23:39:41.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.609+0000 7fe46cff9700 1 -- 172.21.15.67:0/3560186446 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fe4700611d0 con 0x7fe45806ecc0 2024-04-22T23:39:41.604 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:39:41.604 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (13m) 6m ago 24m 25.5M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:39:41.604 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (24m) 6m ago 24m 9114k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:39:41.604 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (23m) 6m ago 23m 9042k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:39:41.604 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (10m) 6m ago 24m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:39:41.604 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (10m) 6m ago 23m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:39:41.604 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (13m) 6m ago 24m 79.3M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:39:41.604 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (7m) 6m ago 19m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:39:41.604 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (6m) 6m ago 19m 12.6M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:39:41.604 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (6m) 6m ago 19m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:39:41.604 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (19m) 6m ago 19m 12.2M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:39:41.604 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (15m) 6m ago 26m 511M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:39:41.604 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (14m) 6m ago 23m 453M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:39:41.605 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (11m) 6m ago 26m 55.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:39:41.605 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (11m) 6m ago 23m 46.0M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:39:41.605 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (14m) 6m ago 24m 15.4M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:39:41.605 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (14m) 6m ago 23m 16.1M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:39:41.605 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (10m) 6m ago 22m 99.2M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:39:41.605 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (9m) 6m ago 22m 127M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:39:41.605 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (9m) 6m ago 22m 79.9M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:39:41.605 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (8m) 6m ago 21m 97.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:39:41.605 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (7m) 6m ago 21m 82.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:39:41.605 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (7m) 6m ago 20m 77.6M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:39:41.605 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (13m) 6m ago 24m 73.9M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:39:41.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.612+0000 7fe47625e700 1 -- 172.21.15.67:0/3560186446 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe45806ecc0 msgr2=0x7fe458071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:41.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.612+0000 7fe47625e700 1 --2- 172.21.15.67:0/3560186446 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe45806ecc0 0x7fe458071180 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe470074880 tx=0x7fe46401a040 comp rx=0 tx=0).stop 2024-04-22T23:39:41.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.612+0000 7fe47625e700 1 -- 172.21.15.67:0/3560186446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe470100ea0 msgr2=0x7fe470073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:41.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.612+0000 7fe47625e700 1 --2- 172.21.15.67:0/3560186446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe470100ea0 0x7fe470073090 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe46000d8d0 tx=0x7fe46000dbe0 comp rx=0 tx=0).stop 2024-04-22T23:39:41.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.612+0000 7fe47625e700 1 -- 172.21.15.67:0/3560186446 shutdown_connections 2024-04-22T23:39:41.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.612+0000 7fe47625e700 1 --2- 172.21.15.67:0/3560186446 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe45806ecc0 0x7fe458071180 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:41.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.612+0000 7fe47625e700 1 --2- 172.21.15.67:0/3560186446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe470100ea0 0x7fe470073090 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:41.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.612+0000 7fe47625e700 1 --2- 172.21.15.67:0/3560186446 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe470101850 0x7fe4700735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:41.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.612+0000 7fe47625e700 1 -- 172.21.15.67:0/3560186446 >> 172.21.15.67:0/3560186446 conn(0x7fe4700fc750 msgr2=0x7fe4700fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:39:41.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.612+0000 7fe47625e700 1 -- 172.21.15.67:0/3560186446 shutdown_connections 2024-04-22T23:39:41.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.612+0000 7fe47625e700 1 -- 172.21.15.67:0/3560186446 wait complete. 2024-04-22T23:39:41.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.718+0000 7fa94b492700 1 -- 172.21.15.67:0/82833619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa944101e60 msgr2=0x7fa944102240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:41.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.718+0000 7fa94b492700 1 --2- 172.21.15.67:0/82833619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa944101e60 0x7fa944102240 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fa93c00b0a0 tx=0x7fa93c00b3b0 comp rx=0 tx=0).stop 2024-04-22T23:39:41.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.718+0000 7fa94b492700 1 -- 172.21.15.67:0/82833619 shutdown_connections 2024-04-22T23:39:41.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.718+0000 7fa94b492700 1 --2- 172.21.15.67:0/82833619 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa944102780 0x7fa94410ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:41.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.718+0000 7fa94b492700 1 --2- 172.21.15.67:0/82833619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa944101e60 0x7fa944102240 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:41.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.718+0000 7fa94b492700 1 -- 172.21.15.67:0/82833619 >> 172.21.15.67:0/82833619 conn(0x7fa9440fb650 msgr2=0x7fa9440fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:39:41.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.718+0000 7fa94b492700 1 -- 172.21.15.67:0/82833619 shutdown_connections 2024-04-22T23:39:41.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.718+0000 7fa94b492700 1 -- 172.21.15.67:0/82833619 wait complete. 2024-04-22T23:39:41.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.719+0000 7fa94b492700 1 Processor -- start 2024-04-22T23:39:41.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.719+0000 7fa94b492700 1 -- start start 2024-04-22T23:39:41.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.719+0000 7fa94b492700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa944101e60 0x7fa94419c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:41.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.719+0000 7fa94b492700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa944102780 0x7fa94419cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:41.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.719+0000 7fa94b492700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94419d220 con 0x7fa944101e60 2024-04-22T23:39:41.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.719+0000 7fa94b492700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa944196720 con 0x7fa944102780 2024-04-22T23:39:41.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.719+0000 7fa94922e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa944101e60 0x7fa94419c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:41.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.719+0000 7fa94922e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa944101e60 0x7fa94419c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43630/0 (socket says 172.21.15.67:43630) 2024-04-22T23:39:41.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.719+0000 7fa94922e700 1 -- 172.21.15.67:0/1754542263 learned_addr learned my addr 172.21.15.67:0/1754542263 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:39:41.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.719+0000 7fa948a2d700 1 --2- 172.21.15.67:0/1754542263 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa944102780 0x7fa94419cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:41.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.720+0000 7fa94922e700 1 -- 172.21.15.67:0/1754542263 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa944102780 msgr2=0x7fa94419cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:41.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.720+0000 7fa94922e700 1 --2- 172.21.15.67:0/1754542263 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa944102780 0x7fa94419cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:41.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.720+0000 7fa94922e700 1 -- 172.21.15.67:0/1754542263 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa93c009d00 con 0x7fa944101e60 2024-04-22T23:39:41.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.720+0000 7fa948a2d700 1 --2- 172.21.15.67:0/1754542263 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa944102780 0x7fa94419cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:39:41.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.720+0000 7fa94922e700 1 --2- 172.21.15.67:0/1754542263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa944101e60 0x7fa94419c650 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fa93c000f80 tx=0x7fa93c00cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:39:41.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.720+0000 7fa9427fc700 1 -- 172.21.15.67:0/1754542263 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa93c00e040 con 0x7fa944101e60 2024-04-22T23:39:41.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.720+0000 7fa9427fc700 1 -- 172.21.15.67:0/1754542263 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa93c003de0 con 0x7fa944101e60 2024-04-22T23:39:41.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.720+0000 7fa9427fc700 1 -- 172.21.15.67:0/1754542263 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa93c01d790 con 0x7fa944101e60 2024-04-22T23:39:41.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.720+0000 7fa94b492700 1 -- 172.21.15.67:0/1754542263 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9441969a0 con 0x7fa944101e60 2024-04-22T23:39:41.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.720+0000 7fa94b492700 1 -- 172.21.15.67:0/1754542263 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa944196e40 con 0x7fa944101e60 2024-04-22T23:39:41.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.721+0000 7fa94b492700 1 -- 172.21.15.67:0/1754542263 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa944108310 con 0x7fa944101e60 2024-04-22T23:39:41.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.722+0000 7fa9427fc700 1 -- 172.21.15.67:0/1754542263 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa93c004040 con 0x7fa944101e60 2024-04-22T23:39:41.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.722+0000 7fa9427fc700 1 --2- 172.21.15.67:0/1754542263 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa92c06ecc0 0x7fa92c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:41.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.722+0000 7fa9427fc700 1 -- 172.21.15.67:0/1754542263 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fa93c0907e0 con 0x7fa944101e60 2024-04-22T23:39:41.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.722+0000 7fa948a2d700 1 --2- 172.21.15.67:0/1754542263 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa92c06ecc0 0x7fa92c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:41.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.722+0000 7fa948a2d700 1 --2- 172.21.15.67:0/1754542263 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa92c06ecc0 0x7fa92c071180 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa938005950 tx=0x7fa9380058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:39:41.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:41.727+0000 7fa9427fc700 1 -- 172.21.15.67:0/1754542263 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fa93c05cf50 con 0x7fa944101e60 2024-04-22T23:39:41.736 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:41 smithi134 ceph-mon[85630]: pgmap v399: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:41.736 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:41 smithi134 ceph-mon[85630]: from='client.34764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:39:41.736 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:39:41.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:41 smithi067 ceph-mon[112438]: pgmap v399: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:41.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:41 smithi067 ceph-mon[112438]: from='client.34764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:39:41.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:39:42.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.053+0000 7fa94b492700 1 -- 172.21.15.67:0/1754542263 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa944197ce0 con 0x7fa944101e60 2024-04-22T23:39:42.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.053+0000 7fa9427fc700 1 -- 172.21.15.67:0/1754542263 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fa944197ce0 con 0x7fa944101e60 2024-04-22T23:39:42.048 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:39:42.048 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:39:42.048 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:39:42.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:39:42.048 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:39:42.048 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:39:42.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:39:42.048 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:39:42.048 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:39:42.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:39:42.048 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:39:42.048 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:39:42.049 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 3 2024-04-22T23:39:42.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:39:42.049 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:39:42.049 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:39:42.049 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 13 2024-04-22T23:39:42.049 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:39:42.049 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:39:42.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.055+0000 7fa94b492700 1 -- 172.21.15.67:0/1754542263 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa92c06ecc0 msgr2=0x7fa92c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:42.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.055+0000 7fa94b492700 1 --2- 172.21.15.67:0/1754542263 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa92c06ecc0 0x7fa92c071180 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa938005950 tx=0x7fa9380058e0 comp rx=0 tx=0).stop 2024-04-22T23:39:42.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.055+0000 7fa94b492700 1 -- 172.21.15.67:0/1754542263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa944101e60 msgr2=0x7fa94419c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:42.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.055+0000 7fa94b492700 1 --2- 172.21.15.67:0/1754542263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa944101e60 0x7fa94419c650 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fa93c000f80 tx=0x7fa93c00cdd0 comp rx=0 tx=0).stop 2024-04-22T23:39:42.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.056+0000 7fa94b492700 1 -- 172.21.15.67:0/1754542263 shutdown_connections 2024-04-22T23:39:42.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.056+0000 7fa94b492700 1 --2- 172.21.15.67:0/1754542263 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa92c06ecc0 0x7fa92c071180 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:42.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.056+0000 7fa94b492700 1 --2- 172.21.15.67:0/1754542263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa944101e60 0x7fa94419c650 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:42.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.056+0000 7fa94b492700 1 --2- 172.21.15.67:0/1754542263 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa944102780 0x7fa94419cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:42.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.056+0000 7fa94b492700 1 -- 172.21.15.67:0/1754542263 >> 172.21.15.67:0/1754542263 conn(0x7fa9440fb650 msgr2=0x7fa944105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:39:42.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.056+0000 7fa94b492700 1 -- 172.21.15.67:0/1754542263 shutdown_connections 2024-04-22T23:39:42.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.056+0000 7fa94b492700 1 -- 172.21.15.67:0/1754542263 wait complete. 2024-04-22T23:39:42.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.163+0000 7fe2bcd46700 1 -- 172.21.15.67:0/3700821440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2b80ffda0 msgr2=0x7fe2b8100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:42.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.163+0000 7fe2bcd46700 1 --2- 172.21.15.67:0/3700821440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2b80ffda0 0x7fe2b8100180 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fe2a8009910 tx=0x7fe2a8009c20 comp rx=0 tx=0).stop 2024-04-22T23:39:42.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.163+0000 7fe2bcd46700 1 -- 172.21.15.67:0/3700821440 shutdown_connections 2024-04-22T23:39:42.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.163+0000 7fe2bcd46700 1 --2- 172.21.15.67:0/3700821440 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe2b8100750 0x7fe2b81047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:42.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.163+0000 7fe2bcd46700 1 --2- 172.21.15.67:0/3700821440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2b80ffda0 0x7fe2b8100180 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:42.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.163+0000 7fe2bcd46700 1 -- 172.21.15.67:0/3700821440 >> 172.21.15.67:0/3700821440 conn(0x7fe2b80fb650 msgr2=0x7fe2b80fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:39:42.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.163+0000 7fe2bcd46700 1 -- 172.21.15.67:0/3700821440 shutdown_connections 2024-04-22T23:39:42.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.163+0000 7fe2bcd46700 1 -- 172.21.15.67:0/3700821440 wait complete. 2024-04-22T23:39:42.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.164+0000 7fe2bcd46700 1 Processor -- start 2024-04-22T23:39:42.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.164+0000 7fe2bcd46700 1 -- start start 2024-04-22T23:39:42.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.164+0000 7fe2bcd46700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe2b80ffda0 0x7fe2b819c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:42.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.164+0000 7fe2bcd46700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2b8100750 0x7fe2b819cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:42.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.164+0000 7fe2bcd46700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b819d2b0 con 0x7fe2b8100750 2024-04-22T23:39:42.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.164+0000 7fe2bcd46700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b8196760 con 0x7fe2b80ffda0 2024-04-22T23:39:42.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.164+0000 7fe2b5d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2b8100750 0x7fe2b819cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:42.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.164+0000 7fe2b5d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2b8100750 0x7fe2b819cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43644/0 (socket says 172.21.15.67:43644) 2024-04-22T23:39:42.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.164+0000 7fe2b5d9b700 1 -- 172.21.15.67:0/2620467293 learned_addr learned my addr 172.21.15.67:0/2620467293 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:39:42.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.164+0000 7fe2b659c700 1 --2- 172.21.15.67:0/2620467293 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe2b80ffda0 0x7fe2b819c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:42.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.164+0000 7fe2b5d9b700 1 -- 172.21.15.67:0/2620467293 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe2b80ffda0 msgr2=0x7fe2b819c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:42.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.164+0000 7fe2b5d9b700 1 --2- 172.21.15.67:0/2620467293 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe2b80ffda0 0x7fe2b819c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:42.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.165+0000 7fe2b5d9b700 1 -- 172.21.15.67:0/2620467293 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a80095f0 con 0x7fe2b8100750 2024-04-22T23:39:42.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.165+0000 7fe2b5d9b700 1 --2- 172.21.15.67:0/2620467293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2b8100750 0x7fe2b819cc20 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac00c8f0 tx=0x7fe2ac00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:39:42.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.165+0000 7fe2a77fe700 1 -- 172.21.15.67:0/2620467293 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2ac007a50 con 0x7fe2b8100750 2024-04-22T23:39:42.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.165+0000 7fe2a77fe700 1 -- 172.21.15.67:0/2620467293 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe2ac007bb0 con 0x7fe2b8100750 2024-04-22T23:39:42.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.165+0000 7fe2bcd46700 1 -- 172.21.15.67:0/2620467293 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2b8196a40 con 0x7fe2b8100750 2024-04-22T23:39:42.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.165+0000 7fe2a77fe700 1 -- 172.21.15.67:0/2620467293 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2ac018630 con 0x7fe2b8100750 2024-04-22T23:39:42.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.165+0000 7fe2bcd46700 1 -- 172.21.15.67:0/2620467293 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2b8196f90 con 0x7fe2b8100750 2024-04-22T23:39:42.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.166+0000 7fe2bcd46700 1 -- 172.21.15.67:0/2620467293 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2b804f2e0 con 0x7fe2b8100750 2024-04-22T23:39:42.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.167+0000 7fe2a77fe700 1 -- 172.21.15.67:0/2620467293 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe2ac007d20 con 0x7fe2b8100750 2024-04-22T23:39:42.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.167+0000 7fe2a77fe700 1 --2- 172.21.15.67:0/2620467293 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe2a006ecc0 0x7fe2a0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:42.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.168+0000 7fe2a77fe700 1 -- 172.21.15.67:0/2620467293 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fe2ac08eeb0 con 0x7fe2b8100750 2024-04-22T23:39:42.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.168+0000 7fe2b659c700 1 --2- 172.21.15.67:0/2620467293 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe2a006ecc0 0x7fe2a0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:42.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.168+0000 7fe2b659c700 1 --2- 172.21.15.67:0/2620467293 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe2a006ecc0 0x7fe2a0071180 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe2a8006010 tx=0x7fe2a801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:39:42.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.172+0000 7fe2a77fe700 1 -- 172.21.15.67:0/2620467293 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe2ac05b6a0 con 0x7fe2b8100750 2024-04-22T23:39:42.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.430+0000 7fe2bcd46700 1 -- 172.21.15.67:0/2620467293 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe2b8197c70 con 0x7fe2b8100750 2024-04-22T23:39:42.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.431+0000 7fe2a77fe700 1 -- 172.21.15.67:0/2620467293 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1871 (secure 0 0 0) 0x7fe2ac05b230 con 0x7fe2b8100750 2024-04-22T23:39:42.426 INFO:teuthology.orchestra.run.smithi067.stdout:e31 2024-04-22T23:39:42.426 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:39:42.426 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:39:42.426 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:39:42.426 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:39:42.426 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:39:42.426 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:39:42.426 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:39:42.426 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:39:42.426 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:39:42.426 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:39:42.426 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:39:42.426 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:39:42.427 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:39:42.427 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:39:42.427 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:39:42.427 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:39:42.427 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:39:42.427 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:39:42.427 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:39:42.427 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:39:42.427 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:39:42.427 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:39:42.427 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:39:42.427 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:39:42.427 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:39:42.427 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:39:42.428 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:39:42.428 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:39:42.428 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:39:42.428 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:39:42.428 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:39:42.428 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:39:42.428 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:39:42.428 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:39:42.428 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:39:42.428 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:39:42.428 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:39:42.428 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:39:42.428 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:39:42.428 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44259} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1827288036,v1:172.21.15.134:6827/1827288036] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:39:42.428 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:39:42.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.433+0000 7fe2bcd46700 1 -- 172.21.15.67:0/2620467293 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe2a006ecc0 msgr2=0x7fe2a0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:42.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.433+0000 7fe2bcd46700 1 --2- 172.21.15.67:0/2620467293 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe2a006ecc0 0x7fe2a0071180 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe2a8006010 tx=0x7fe2a801a040 comp rx=0 tx=0).stop 2024-04-22T23:39:42.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.433+0000 7fe2bcd46700 1 -- 172.21.15.67:0/2620467293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2b8100750 msgr2=0x7fe2b819cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:42.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.433+0000 7fe2bcd46700 1 --2- 172.21.15.67:0/2620467293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2b8100750 0x7fe2b819cc20 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac00c8f0 tx=0x7fe2ac00ccb0 comp rx=0 tx=0).stop 2024-04-22T23:39:42.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.433+0000 7fe2bcd46700 1 -- 172.21.15.67:0/2620467293 shutdown_connections 2024-04-22T23:39:42.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.434+0000 7fe2bcd46700 1 --2- 172.21.15.67:0/2620467293 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe2a006ecc0 0x7fe2a0071180 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:42.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.434+0000 7fe2bcd46700 1 --2- 172.21.15.67:0/2620467293 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe2b80ffda0 0x7fe2b819c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:42.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.434+0000 7fe2bcd46700 1 --2- 172.21.15.67:0/2620467293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2b8100750 0x7fe2b819cc20 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:42.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.434+0000 7fe2bcd46700 1 -- 172.21.15.67:0/2620467293 >> 172.21.15.67:0/2620467293 conn(0x7fe2b80fb650 msgr2=0x7fe2b8103f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:39:42.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.434+0000 7fe2bcd46700 1 -- 172.21.15.67:0/2620467293 shutdown_connections 2024-04-22T23:39:42.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.434+0000 7fe2bcd46700 1 -- 172.21.15.67:0/2620467293 wait complete. 2024-04-22T23:39:42.430 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 31 2024-04-22T23:39:42.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.542+0000 7fe434162700 1 -- 172.21.15.67:0/3119783339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe42c101850 msgr2=0x7fe42c1058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:42.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.542+0000 7fe434162700 1 --2- 172.21.15.67:0/3119783339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe42c101850 0x7fe42c1058a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fe420009910 tx=0x7fe420009c20 comp rx=0 tx=0).stop 2024-04-22T23:39:42.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.542+0000 7fe434162700 1 -- 172.21.15.67:0/3119783339 shutdown_connections 2024-04-22T23:39:42.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.542+0000 7fe434162700 1 --2- 172.21.15.67:0/3119783339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe42c101850 0x7fe42c1058a0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:42.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.542+0000 7fe434162700 1 --2- 172.21.15.67:0/3119783339 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe42c100ea0 0x7fe42c101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:42.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.542+0000 7fe434162700 1 -- 172.21.15.67:0/3119783339 >> 172.21.15.67:0/3119783339 conn(0x7fe42c0fc750 msgr2=0x7fe42c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:39:42.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.542+0000 7fe434162700 1 -- 172.21.15.67:0/3119783339 shutdown_connections 2024-04-22T23:39:42.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.542+0000 7fe434162700 1 -- 172.21.15.67:0/3119783339 wait complete. 2024-04-22T23:39:42.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.543+0000 7fe434162700 1 Processor -- start 2024-04-22T23:39:42.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.543+0000 7fe434162700 1 -- start start 2024-04-22T23:39:42.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.543+0000 7fe434162700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe42c100ea0 0x7fe42c19c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:42.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.543+0000 7fe434162700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe42c101850 0x7fe42c19cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:42.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.543+0000 7fe434162700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe42c19d2d0 con 0x7fe42c101850 2024-04-22T23:39:42.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.543+0000 7fe434162700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe42c196780 con 0x7fe42c100ea0 2024-04-22T23:39:42.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.543+0000 7fe4316fd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe42c101850 0x7fe42c19cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:42.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.543+0000 7fe4316fd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe42c101850 0x7fe42c19cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43652/0 (socket says 172.21.15.67:43652) 2024-04-22T23:39:42.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.543+0000 7fe4316fd700 1 -- 172.21.15.67:0/245183253 learned_addr learned my addr 172.21.15.67:0/245183253 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:39:42.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.543+0000 7fe431efe700 1 --2- 172.21.15.67:0/245183253 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe42c100ea0 0x7fe42c19c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:42.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.543+0000 7fe4316fd700 1 -- 172.21.15.67:0/245183253 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe42c100ea0 msgr2=0x7fe42c19c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:42.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.543+0000 7fe4316fd700 1 --2- 172.21.15.67:0/245183253 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe42c100ea0 0x7fe42c19c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:42.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.543+0000 7fe4316fd700 1 -- 172.21.15.67:0/245183253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4200095f0 con 0x7fe42c101850 2024-04-22T23:39:42.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.543+0000 7fe4316fd700 1 --2- 172.21.15.67:0/245183253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe42c101850 0x7fe42c19cc40 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe42000bd70 tx=0x7fe42000bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:39:42.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.544+0000 7fe42affd700 1 -- 172.21.15.67:0/245183253 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe42001d070 con 0x7fe42c101850 2024-04-22T23:39:42.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.544+0000 7fe42affd700 1 -- 172.21.15.67:0/245183253 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe420005dc0 con 0x7fe42c101850 2024-04-22T23:39:42.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.544+0000 7fe434162700 1 -- 172.21.15.67:0/245183253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe42c196a00 con 0x7fe42c101850 2024-04-22T23:39:42.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.544+0000 7fe42affd700 1 -- 172.21.15.67:0/245183253 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe420017400 con 0x7fe42c101850 2024-04-22T23:39:42.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.544+0000 7fe434162700 1 -- 172.21.15.67:0/245183253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe42c196ef0 con 0x7fe42c101850 2024-04-22T23:39:42.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.545+0000 7fe434162700 1 -- 172.21.15.67:0/245183253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe42c1092a0 con 0x7fe42c101850 2024-04-22T23:39:42.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.545+0000 7fe42affd700 1 -- 172.21.15.67:0/245183253 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe42000f460 con 0x7fe42c101850 2024-04-22T23:39:42.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.546+0000 7fe42affd700 1 --2- 172.21.15.67:0/245183253 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe41406ecc0 0x7fe414071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:42.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.546+0000 7fe431efe700 1 --2- 172.21.15.67:0/245183253 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe41406ecc0 0x7fe414071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:42.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.546+0000 7fe42affd700 1 -- 172.21.15.67:0/245183253 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fe420090340 con 0x7fe42c101850 2024-04-22T23:39:42.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.546+0000 7fe431efe700 1 --2- 172.21.15.67:0/245183253 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe41406ecc0 0x7fe414071180 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe424009550 tx=0x7fe4240091f0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:39:42.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.551+0000 7fe42affd700 1 -- 172.21.15.67:0/245183253 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe42005cb30 con 0x7fe42c101850 2024-04-22T23:39:42.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.780+0000 7fe434162700 1 -- 172.21.15.67:0/245183253 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe42c197160 con 0x7fe41406ecc0 2024-04-22T23:39:42.775 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:42 smithi067 ceph-mon[112438]: from='client.34768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:39:42.775 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:42 smithi067 ceph-mon[112438]: from='client.34772 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:39:42.775 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:42 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1754542263' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:39:42.775 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:42 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2620467293' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:39:42.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.782+0000 7fe42affd700 1 -- 172.21.15.67:0/245183253 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe42c197160 con 0x7fe41406ecc0 2024-04-22T23:39:42.776 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:39:42.776 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:39:42.776 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:39:42.776 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:39:42.776 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:39:42.777 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:39:42.777 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:39:42.777 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:39:42.777 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:39:42.777 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:39:42.777 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T23:39:42.777 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:39:42.777 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:39:42.777 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:39:42.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.784+0000 7fe434162700 1 -- 172.21.15.67:0/245183253 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe41406ecc0 msgr2=0x7fe414071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:42.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.784+0000 7fe434162700 1 --2- 172.21.15.67:0/245183253 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe41406ecc0 0x7fe414071180 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe424009550 tx=0x7fe4240091f0 comp rx=0 tx=0).stop 2024-04-22T23:39:42.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.784+0000 7fe434162700 1 -- 172.21.15.67:0/245183253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe42c101850 msgr2=0x7fe42c19cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:42.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.784+0000 7fe434162700 1 --2- 172.21.15.67:0/245183253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe42c101850 0x7fe42c19cc40 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe42000bd70 tx=0x7fe42000bda0 comp rx=0 tx=0).stop 2024-04-22T23:39:42.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.784+0000 7fe434162700 1 -- 172.21.15.67:0/245183253 shutdown_connections 2024-04-22T23:39:42.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.784+0000 7fe434162700 1 --2- 172.21.15.67:0/245183253 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe41406ecc0 0x7fe414071180 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:42.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.784+0000 7fe434162700 1 --2- 172.21.15.67:0/245183253 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe42c100ea0 0x7fe42c19c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:42.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.784+0000 7fe434162700 1 --2- 172.21.15.67:0/245183253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe42c101850 0x7fe42c19cc40 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:42.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.784+0000 7fe434162700 1 -- 172.21.15.67:0/245183253 >> 172.21.15.67:0/245183253 conn(0x7fe42c0fc750 msgr2=0x7fe42c0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:39:42.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.784+0000 7fe434162700 1 -- 172.21.15.67:0/245183253 shutdown_connections 2024-04-22T23:39:42.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.784+0000 7fe434162700 1 -- 172.21.15.67:0/245183253 wait complete. 2024-04-22T23:39:42.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.891+0000 7f06cc471700 1 -- 172.21.15.67:0/3988579125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06c4100ea0 msgr2=0x7f06c4101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:42.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.891+0000 7f06cc471700 1 --2- 172.21.15.67:0/3988579125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06c4100ea0 0x7f06c4101280 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f06c0009910 tx=0x7f06c0009c20 comp rx=0 tx=0).stop 2024-04-22T23:39:42.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.891+0000 7f06cc471700 1 -- 172.21.15.67:0/3988579125 shutdown_connections 2024-04-22T23:39:42.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.891+0000 7f06cc471700 1 --2- 172.21.15.67:0/3988579125 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06c4101850 0x7f06c41058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:42.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.891+0000 7f06cc471700 1 --2- 172.21.15.67:0/3988579125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06c4100ea0 0x7f06c4101280 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:42.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.891+0000 7f06cc471700 1 -- 172.21.15.67:0/3988579125 >> 172.21.15.67:0/3988579125 conn(0x7f06c40fc750 msgr2=0x7f06c40feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:39:42.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.891+0000 7f06cc471700 1 -- 172.21.15.67:0/3988579125 shutdown_connections 2024-04-22T23:39:42.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.891+0000 7f06cc471700 1 -- 172.21.15.67:0/3988579125 wait complete. 2024-04-22T23:39:42.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.892+0000 7f06cc471700 1 Processor -- start 2024-04-22T23:39:42.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.892+0000 7f06cc471700 1 -- start start 2024-04-22T23:39:42.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.892+0000 7f06cc471700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06c4100ea0 0x7f06c419c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:42.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.892+0000 7f06cc471700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06c4101850 0x7f06c419cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:42.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.892+0000 7f06cc471700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c419d2c0 con 0x7f06c4100ea0 2024-04-22T23:39:42.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.892+0000 7f06cc471700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c4196770 con 0x7f06c4101850 2024-04-22T23:39:42.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.892+0000 7f06ca20d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06c4100ea0 0x7f06c419c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:42.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.892+0000 7f06ca20d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06c4100ea0 0x7f06c419c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43670/0 (socket says 172.21.15.67:43670) 2024-04-22T23:39:42.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.892+0000 7f06ca20d700 1 -- 172.21.15.67:0/1099358895 learned_addr learned my addr 172.21.15.67:0/1099358895 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:39:42.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.892+0000 7f06c9a0c700 1 --2- 172.21.15.67:0/1099358895 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06c4101850 0x7f06c419cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:42.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.892+0000 7f06ca20d700 1 -- 172.21.15.67:0/1099358895 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06c4101850 msgr2=0x7f06c419cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:42.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.892+0000 7f06ca20d700 1 --2- 172.21.15.67:0/1099358895 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06c4101850 0x7f06c419cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:42.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.892+0000 7f06ca20d700 1 -- 172.21.15.67:0/1099358895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c00095f0 con 0x7f06c4100ea0 2024-04-22T23:39:42.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.892+0000 7f06c9a0c700 1 --2- 172.21.15.67:0/1099358895 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06c4101850 0x7f06c419cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:39:42.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.892+0000 7f06ca20d700 1 --2- 172.21.15.67:0/1099358895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06c4100ea0 0x7f06c419c6f0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f06c0006010 tx=0x7f06c000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:39:42.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.893+0000 7f06bb7fe700 1 -- 172.21.15.67:0/1099358895 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06c001d070 con 0x7f06c4100ea0 2024-04-22T23:39:42.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.893+0000 7f06bb7fe700 1 -- 172.21.15.67:0/1099358895 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f06c000f460 con 0x7f06c4100ea0 2024-04-22T23:39:42.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.893+0000 7f06cc471700 1 -- 172.21.15.67:0/1099358895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06c41969f0 con 0x7f06c4100ea0 2024-04-22T23:39:42.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.893+0000 7f06bb7fe700 1 -- 172.21.15.67:0/1099358895 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06c0017630 con 0x7f06c4100ea0 2024-04-22T23:39:42.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.893+0000 7f06cc471700 1 -- 172.21.15.67:0/1099358895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06c4196ee0 con 0x7f06c4100ea0 2024-04-22T23:39:42.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.894+0000 7f06cc471700 1 -- 172.21.15.67:0/1099358895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06c4109240 con 0x7f06c4100ea0 2024-04-22T23:39:42.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.894+0000 7f06bb7fe700 1 -- 172.21.15.67:0/1099358895 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f06c000f5d0 con 0x7f06c4100ea0 2024-04-22T23:39:42.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.895+0000 7f06bb7fe700 1 --2- 172.21.15.67:0/1099358895 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f06b006ecc0 0x7f06b0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:39:42.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.895+0000 7f06bb7fe700 1 -- 172.21.15.67:0/1099358895 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f06c0090400 con 0x7f06c4100ea0 2024-04-22T23:39:42.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.895+0000 7f06c9a0c700 1 --2- 172.21.15.67:0/1099358895 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f06b006ecc0 0x7f06b0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:39:42.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.895+0000 7f06c9a0c700 1 --2- 172.21.15.67:0/1099358895 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f06b006ecc0 0x7f06b0071180 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f06b4005fd0 tx=0x7f06b4005ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:39:42.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:42.899+0000 7f06bb7fe700 1 -- 172.21.15.67:0/1099358895 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f06c005cbf0 con 0x7f06c4100ea0 2024-04-22T23:39:42.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:42 smithi134 ceph-mon[85630]: from='client.34768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:39:42.987 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:42 smithi134 ceph-mon[85630]: from='client.34772 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:39:42.987 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:42 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1754542263' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:39:42.987 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:42 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2620467293' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:39:43.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:43.203+0000 7f06cc471700 1 -- 172.21.15.67:0/1099358895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f06c4197d40 con 0x7f06c4100ea0 2024-04-22T23:39:43.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:43.203+0000 7f06bb7fe700 1 -- 172.21.15.67:0/1099358895 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f06c005c780 con 0x7f06c4100ea0 2024-04-22T23:39:43.198 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:39:43.198 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:39:43.198 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:39:43.198 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:39:43.198 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:39:43.199 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:39:43.199 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:39:43.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:43.207+0000 7f06cc471700 1 -- 172.21.15.67:0/1099358895 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f06b006ecc0 msgr2=0x7f06b0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:43.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:43.207+0000 7f06cc471700 1 --2- 172.21.15.67:0/1099358895 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f06b006ecc0 0x7f06b0071180 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f06b4005fd0 tx=0x7f06b4005ee0 comp rx=0 tx=0).stop 2024-04-22T23:39:43.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:43.207+0000 7f06cc471700 1 -- 172.21.15.67:0/1099358895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06c4100ea0 msgr2=0x7f06c419c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:39:43.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:43.207+0000 7f06cc471700 1 --2- 172.21.15.67:0/1099358895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06c4100ea0 0x7f06c419c6f0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f06c0006010 tx=0x7f06c000be20 comp rx=0 tx=0).stop 2024-04-22T23:39:43.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:43.207+0000 7f06cc471700 1 -- 172.21.15.67:0/1099358895 shutdown_connections 2024-04-22T23:39:43.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:43.207+0000 7f06cc471700 1 --2- 172.21.15.67:0/1099358895 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f06b006ecc0 0x7f06b0071180 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:43.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:43.207+0000 7f06cc471700 1 --2- 172.21.15.67:0/1099358895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06c4100ea0 0x7f06c419c6f0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:43.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:43.207+0000 7f06cc471700 1 --2- 172.21.15.67:0/1099358895 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06c4101850 0x7f06c419cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:39:43.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:43.207+0000 7f06cc471700 1 -- 172.21.15.67:0/1099358895 >> 172.21.15.67:0/1099358895 conn(0x7f06c40fc750 msgr2=0x7f06c40fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:39:43.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:43.207+0000 7f06cc471700 1 -- 172.21.15.67:0/1099358895 shutdown_connections 2024-04-22T23:39:43.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:39:43.207+0000 7f06cc471700 1 -- 172.21.15.67:0/1099358895 wait complete. 2024-04-22T23:39:43.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:43 smithi067 ceph-mon[112438]: pgmap v400: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:43.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:43 smithi067 ceph-mon[112438]: from='client.34784 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:39:43.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:43 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1099358895' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:39:43.987 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:43 smithi134 ceph-mon[85630]: pgmap v400: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:43.987 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:43 smithi134 ceph-mon[85630]: from='client.34784 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:39:43.987 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:43 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1099358895' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:39:45.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:45 smithi067 ceph-mon[112438]: pgmap v401: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:45.788 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:45 smithi134 ceph-mon[85630]: pgmap v401: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:47.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:47 smithi134 ceph-mon[85630]: pgmap v402: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:48.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:47 smithi067 ceph-mon[112438]: pgmap v402: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:49.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:49 smithi134 ceph-mon[85630]: pgmap v403: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:50.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:49 smithi067 ceph-mon[112438]: pgmap v403: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:51.737 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:51 smithi134 ceph-mon[85630]: pgmap v404: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:52.015 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:51 smithi067 ceph-mon[112438]: pgmap v404: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:53.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:53 smithi134 ceph-mon[85630]: pgmap v405: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:54.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:53 smithi067 ceph-mon[112438]: pgmap v405: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:55.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:55 smithi134 ceph-mon[85630]: pgmap v406: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:56.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:55 smithi067 ceph-mon[112438]: pgmap v406: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:56.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:39:57.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:39:57.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:57 smithi134 ceph-mon[85630]: pgmap v407: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:58.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:57 smithi067 ceph-mon[112438]: pgmap v407: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:59.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:39:59 smithi134 ceph-mon[85630]: pgmap v408: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:39:59.992 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:39:59 smithi067 ceph-mon[112438]: pgmap v408: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:00.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:00 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[112434]: 2024-04-22T23:39:59.999+0000 7f20165b9700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:40:00.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:00 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[112434]: 2024-04-22T23:39:59.999+0000 7f20165b9700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:40:00.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:00 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[112434]: 2024-04-22T23:39:59.999+0000 7f20165b9700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2024-04-22T23:40:00.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:00 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[112434]: 2024-04-22T23:39:59.999+0000 7f20165b9700 -1 log_channel(cluster) log [ERR] : [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:40:00.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:00 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[112434]: 2024-04-22T23:39:59.999+0000 7f20165b9700 -1 log_channel(cluster) log [ERR] : fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:40:00.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:00 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[112434]: 2024-04-22T23:39:59.999+0000 7f20165b9700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:40:00.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:00 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[112434]: 2024-04-22T23:39:59.999+0000 7f20165b9700 -1 log_channel(cluster) log [ERR] : fs cephfs is offline because no MDS is active for it. 2024-04-22T23:40:00.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:00 smithi134 ceph-mon[85630]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:40:00.987 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:00 smithi134 ceph-mon[85630]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:40:00.987 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:00 smithi134 ceph-mon[85630]: fs cephfs is degraded 2024-04-22T23:40:00.987 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:00 smithi134 ceph-mon[85630]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:40:00.987 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:00 smithi134 ceph-mon[85630]: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:40:00.987 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:00 smithi134 ceph-mon[85630]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:40:00.987 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:00 smithi134 ceph-mon[85630]: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:40:01.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:00 smithi067 ceph-mon[112438]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:40:01.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:00 smithi067 ceph-mon[112438]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:40:01.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:00 smithi067 ceph-mon[112438]: fs cephfs is degraded 2024-04-22T23:40:01.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:00 smithi067 ceph-mon[112438]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:40:01.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:00 smithi067 ceph-mon[112438]: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:40:01.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:00 smithi067 ceph-mon[112438]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:40:01.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:00 smithi067 ceph-mon[112438]: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:40:01.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:01 smithi134 ceph-mon[85630]: pgmap v409: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:02.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:01 smithi067 ceph-mon[112438]: pgmap v409: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:03.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:03 smithi134 ceph-mon[85630]: pgmap v410: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:04.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:03 smithi067 ceph-mon[112438]: pgmap v410: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:05.987 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:05 smithi134 ceph-mon[85630]: pgmap v411: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:06.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:05 smithi067 ceph-mon[112438]: pgmap v411: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:07.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:07 smithi134 ceph-mon[85630]: pgmap v412: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:08.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:07 smithi067 ceph-mon[112438]: pgmap v412: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:09.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:09 smithi134 ceph-mon[85630]: pgmap v413: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:10.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:09 smithi067 ceph-mon[112438]: pgmap v413: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:11.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:11 smithi134 ceph-mon[85630]: pgmap v414: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:11.987 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:40:12.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:11 smithi067 ceph-mon[112438]: pgmap v414: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:12.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:40:13.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.329+0000 7fa43f3d6700 1 -- 172.21.15.67:0/1644452218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa438105bf0 msgr2=0x7fa438105fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:13.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.329+0000 7fa43f3d6700 1 --2- 172.21.15.67:0/1644452218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa438105bf0 0x7fa438105fd0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa434009910 tx=0x7fa434009c20 comp rx=0 tx=0).stop 2024-04-22T23:40:13.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.329+0000 7fa43f3d6700 1 -- 172.21.15.67:0/1644452218 shutdown_connections 2024-04-22T23:40:13.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.329+0000 7fa43f3d6700 1 --2- 172.21.15.67:0/1644452218 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa4380ffb40 0x7fa4380fffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:13.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.329+0000 7fa43f3d6700 1 --2- 172.21.15.67:0/1644452218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa438105bf0 0x7fa438105fd0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:13.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.329+0000 7fa43f3d6700 1 -- 172.21.15.67:0/1644452218 >> 172.21.15.67:0/1644452218 conn(0x7fa4380fb650 msgr2=0x7fa4380fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:40:13.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.329+0000 7fa43f3d6700 1 -- 172.21.15.67:0/1644452218 shutdown_connections 2024-04-22T23:40:13.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.329+0000 7fa43f3d6700 1 -- 172.21.15.67:0/1644452218 wait complete. 2024-04-22T23:40:13.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.330+0000 7fa43f3d6700 1 Processor -- start 2024-04-22T23:40:13.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.330+0000 7fa43f3d6700 1 -- start start 2024-04-22T23:40:13.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.330+0000 7fa43f3d6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4380ffb40 0x7fa43819c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:13.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.330+0000 7fa43f3d6700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa438105bf0 0x7fa43819cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:13.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.330+0000 7fa43f3d6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa43819d170 con 0x7fa4380ffb40 2024-04-22T23:40:13.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.330+0000 7fa43f3d6700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa438196620 con 0x7fa438105bf0 2024-04-22T23:40:13.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.330+0000 7fa43d172700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4380ffb40 0x7fa43819c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:13.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.330+0000 7fa43d172700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4380ffb40 0x7fa43819c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45078/0 (socket says 172.21.15.67:45078) 2024-04-22T23:40:13.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.330+0000 7fa43d172700 1 -- 172.21.15.67:0/3612383582 learned_addr learned my addr 172.21.15.67:0/3612383582 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:40:13.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.330+0000 7fa43c971700 1 --2- 172.21.15.67:0/3612383582 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa438105bf0 0x7fa43819cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:13.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.330+0000 7fa43d172700 1 -- 172.21.15.67:0/3612383582 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa438105bf0 msgr2=0x7fa43819cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:13.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.330+0000 7fa43d172700 1 --2- 172.21.15.67:0/3612383582 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa438105bf0 0x7fa43819cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:13.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.330+0000 7fa43d172700 1 -- 172.21.15.67:0/3612383582 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4340095f0 con 0x7fa4380ffb40 2024-04-22T23:40:13.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.330+0000 7fa43c971700 1 --2- 172.21.15.67:0/3612383582 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa438105bf0 0x7fa43819cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:40:13.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.331+0000 7fa43d172700 1 --2- 172.21.15.67:0/3612383582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4380ffb40 0x7fa43819c5a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fa434006010 tx=0x7fa43400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:40:13.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.331+0000 7fa42e7fc700 1 -- 172.21.15.67:0/3612383582 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa43401d070 con 0x7fa4380ffb40 2024-04-22T23:40:13.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.331+0000 7fa42e7fc700 1 -- 172.21.15.67:0/3612383582 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa43400f460 con 0x7fa4380ffb40 2024-04-22T23:40:13.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.331+0000 7fa43f3d6700 1 -- 172.21.15.67:0/3612383582 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4381968a0 con 0x7fa4380ffb40 2024-04-22T23:40:13.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.331+0000 7fa42e7fc700 1 -- 172.21.15.67:0/3612383582 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa434017630 con 0x7fa4380ffb40 2024-04-22T23:40:13.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.331+0000 7fa43f3d6700 1 -- 172.21.15.67:0/3612383582 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa438196d90 con 0x7fa4380ffb40 2024-04-22T23:40:13.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.332+0000 7fa43f3d6700 1 -- 172.21.15.67:0/3612383582 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa43804f2e0 con 0x7fa4380ffb40 2024-04-22T23:40:13.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.332+0000 7fa42e7fc700 1 -- 172.21.15.67:0/3612383582 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa43400f5d0 con 0x7fa4380ffb40 2024-04-22T23:40:13.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.333+0000 7fa42e7fc700 1 --2- 172.21.15.67:0/3612383582 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa42406ecc0 0x7fa424071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:13.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.333+0000 7fa42e7fc700 1 -- 172.21.15.67:0/3612383582 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fa434090470 con 0x7fa4380ffb40 2024-04-22T23:40:13.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.333+0000 7fa43c971700 1 --2- 172.21.15.67:0/3612383582 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa42406ecc0 0x7fa424071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:13.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.333+0000 7fa43c971700 1 --2- 172.21.15.67:0/3612383582 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa42406ecc0 0x7fa424071180 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa428005fd0 tx=0x7fa428005ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:40:13.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.338+0000 7fa42e7fc700 1 -- 172.21.15.67:0/3612383582 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fa43405cbe0 con 0x7fa4380ffb40 2024-04-22T23:40:13.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.569+0000 7fa43f3d6700 1 -- 172.21.15.67:0/3612383582 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa438197190 con 0x7fa42406ecc0 2024-04-22T23:40:13.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.570+0000 7fa42e7fc700 1 -- 172.21.15.67:0/3612383582 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa438197190 con 0x7fa42406ecc0 2024-04-22T23:40:13.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.572+0000 7fa43f3d6700 1 -- 172.21.15.67:0/3612383582 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa42406ecc0 msgr2=0x7fa424071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:13.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.572+0000 7fa43f3d6700 1 --2- 172.21.15.67:0/3612383582 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa42406ecc0 0x7fa424071180 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa428005fd0 tx=0x7fa428005ee0 comp rx=0 tx=0).stop 2024-04-22T23:40:13.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.572+0000 7fa43f3d6700 1 -- 172.21.15.67:0/3612383582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4380ffb40 msgr2=0x7fa43819c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:13.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.572+0000 7fa43f3d6700 1 --2- 172.21.15.67:0/3612383582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4380ffb40 0x7fa43819c5a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fa434006010 tx=0x7fa43400be20 comp rx=0 tx=0).stop 2024-04-22T23:40:13.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.572+0000 7fa43f3d6700 1 -- 172.21.15.67:0/3612383582 shutdown_connections 2024-04-22T23:40:13.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.572+0000 7fa43f3d6700 1 --2- 172.21.15.67:0/3612383582 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa42406ecc0 0x7fa424071180 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:13.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.572+0000 7fa43f3d6700 1 --2- 172.21.15.67:0/3612383582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4380ffb40 0x7fa43819c5a0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:13.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.572+0000 7fa43f3d6700 1 --2- 172.21.15.67:0/3612383582 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa438105bf0 0x7fa43819cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:13.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.572+0000 7fa43f3d6700 1 -- 172.21.15.67:0/3612383582 >> 172.21.15.67:0/3612383582 conn(0x7fa4380fb650 msgr2=0x7fa438068e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:40:13.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.572+0000 7fa43f3d6700 1 -- 172.21.15.67:0/3612383582 shutdown_connections 2024-04-22T23:40:13.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.572+0000 7fa43f3d6700 1 -- 172.21.15.67:0/3612383582 wait complete. 2024-04-22T23:40:13.576 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:40:13.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.682+0000 7f128f80a700 1 -- 172.21.15.67:0/34199589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1288102720 msgr2=0x7f128810ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:13.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.682+0000 7f128f80a700 1 --2- 172.21.15.67:0/34199589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1288102720 0x7f128810ac10 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f127c009910 tx=0x7f127c009c20 comp rx=0 tx=0).stop 2024-04-22T23:40:13.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.682+0000 7f128f80a700 1 -- 172.21.15.67:0/34199589 shutdown_connections 2024-04-22T23:40:13.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.682+0000 7f128f80a700 1 --2- 172.21.15.67:0/34199589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1288102720 0x7f128810ac10 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:13.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.682+0000 7f128f80a700 1 --2- 172.21.15.67:0/34199589 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1288101e00 0x7f12881021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:13.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.682+0000 7f128f80a700 1 -- 172.21.15.67:0/34199589 >> 172.21.15.67:0/34199589 conn(0x7f12880fb650 msgr2=0x7f12880fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:40:13.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.682+0000 7f128f80a700 1 -- 172.21.15.67:0/34199589 shutdown_connections 2024-04-22T23:40:13.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.682+0000 7f128f80a700 1 -- 172.21.15.67:0/34199589 wait complete. 2024-04-22T23:40:13.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.683+0000 7f128f80a700 1 Processor -- start 2024-04-22T23:40:13.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.683+0000 7f128f80a700 1 -- start start 2024-04-22T23:40:13.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.683+0000 7f128f80a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1288101e00 0x7f12880ffa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:13.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.683+0000 7f128f80a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1288102720 0x7f12880fffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:13.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.683+0000 7f128f80a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12881004e0 con 0x7f1288101e00 2024-04-22T23:40:13.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.683+0000 7f128f80a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1288100620 con 0x7f1288102720 2024-04-22T23:40:13.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.683+0000 7f128d5a6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1288101e00 0x7f12880ffa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:13.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.683+0000 7f128d5a6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1288101e00 0x7f12880ffa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45104/0 (socket says 172.21.15.67:45104) 2024-04-22T23:40:13.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.683+0000 7f128d5a6700 1 -- 172.21.15.67:0/1475871249 learned_addr learned my addr 172.21.15.67:0/1475871249 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:40:13.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.683+0000 7f128d5a6700 1 -- 172.21.15.67:0/1475871249 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1288102720 msgr2=0x7f12880fffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:13.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.683+0000 7f128cda5700 1 --2- 172.21.15.67:0/1475871249 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1288102720 0x7f12880fffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:13.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.683+0000 7f128d5a6700 1 --2- 172.21.15.67:0/1475871249 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1288102720 0x7f12880fffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:13.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.683+0000 7f128d5a6700 1 -- 172.21.15.67:0/1475871249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f127c0095f0 con 0x7f1288101e00 2024-04-22T23:40:13.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.683+0000 7f128d5a6700 1 --2- 172.21.15.67:0/1475871249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1288101e00 0x7f12880ffa60 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f12780060b0 tx=0x7f127800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:40:13.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.684+0000 7f12867fc700 1 -- 172.21.15.67:0/1475871249 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f127800dda0 con 0x7f1288101e00 2024-04-22T23:40:13.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.684+0000 7f12867fc700 1 -- 172.21.15.67:0/1475871249 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f127800f040 con 0x7f1288101e00 2024-04-22T23:40:13.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.684+0000 7f128f80a700 1 -- 172.21.15.67:0/1475871249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12881a2a90 con 0x7f1288101e00 2024-04-22T23:40:13.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.684+0000 7f12867fc700 1 -- 172.21.15.67:0/1475871249 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1278014790 con 0x7f1288101e00 2024-04-22T23:40:13.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.684+0000 7f128f80a700 1 -- 172.21.15.67:0/1475871249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12881a2ea0 con 0x7f1288101e00 2024-04-22T23:40:13.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.685+0000 7f128f80a700 1 -- 172.21.15.67:0/1475871249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1288108310 con 0x7f1288101e00 2024-04-22T23:40:13.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.685+0000 7f12867fc700 1 -- 172.21.15.67:0/1475871249 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1278014980 con 0x7f1288101e00 2024-04-22T23:40:13.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.686+0000 7f12867fc700 1 --2- 172.21.15.67:0/1475871249 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f127006ecc0 0x7f1270071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:13.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.686+0000 7f12867fc700 1 -- 172.21.15.67:0/1475871249 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f127808ea80 con 0x7f1288101e00 2024-04-22T23:40:13.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.686+0000 7f128cda5700 1 --2- 172.21.15.67:0/1475871249 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f127006ecc0 0x7f1270071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:13.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.686+0000 7f128cda5700 1 --2- 172.21.15.67:0/1475871249 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f127006ecc0 0x7f1270071180 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f127c006010 tx=0x7f127c005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:40:13.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.690+0000 7f12867fc700 1 -- 172.21.15.67:0/1475871249 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1278057770 con 0x7f1288101e00 2024-04-22T23:40:13.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.921+0000 7f128f80a700 1 -- 172.21.15.67:0/1475871249 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f12880611d0 con 0x7f127006ecc0 2024-04-22T23:40:13.914 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:13 smithi067 ceph-mon[112438]: pgmap v415: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:13.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.922+0000 7f12867fc700 1 -- 172.21.15.67:0/1475871249 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f12880611d0 con 0x7f127006ecc0 2024-04-22T23:40:13.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.924+0000 7f128f80a700 1 -- 172.21.15.67:0/1475871249 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f127006ecc0 msgr2=0x7f1270071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:13.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.924+0000 7f128f80a700 1 --2- 172.21.15.67:0/1475871249 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f127006ecc0 0x7f1270071180 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f127c006010 tx=0x7f127c005960 comp rx=0 tx=0).stop 2024-04-22T23:40:13.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.924+0000 7f128f80a700 1 -- 172.21.15.67:0/1475871249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1288101e00 msgr2=0x7f12880ffa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:13.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.924+0000 7f128f80a700 1 --2- 172.21.15.67:0/1475871249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1288101e00 0x7f12880ffa60 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f12780060b0 tx=0x7f127800d3b0 comp rx=0 tx=0).stop 2024-04-22T23:40:13.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.924+0000 7f128f80a700 1 -- 172.21.15.67:0/1475871249 shutdown_connections 2024-04-22T23:40:13.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.924+0000 7f128f80a700 1 --2- 172.21.15.67:0/1475871249 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f127006ecc0 0x7f1270071180 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:13.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.924+0000 7f128f80a700 1 --2- 172.21.15.67:0/1475871249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1288101e00 0x7f12880ffa60 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:13.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.924+0000 7f128f80a700 1 --2- 172.21.15.67:0/1475871249 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1288102720 0x7f12880fffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:13.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.924+0000 7f128f80a700 1 -- 172.21.15.67:0/1475871249 >> 172.21.15.67:0/1475871249 conn(0x7f12880fb650 msgr2=0x7f1288105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:40:13.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.924+0000 7f128f80a700 1 -- 172.21.15.67:0/1475871249 shutdown_connections 2024-04-22T23:40:13.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:13.924+0000 7f128f80a700 1 -- 172.21.15.67:0/1475871249 wait complete. 2024-04-22T23:40:13.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:13 smithi134 ceph-mon[85630]: pgmap v415: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:14.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.031+0000 7fae29aec700 1 -- 172.21.15.67:0/4043361734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae240ff340 msgr2=0x7fae240ff7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:14.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.031+0000 7fae29aec700 1 --2- 172.21.15.67:0/4043361734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae240ff340 0x7fae240ff7a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fae18009910 tx=0x7fae18009c20 comp rx=0 tx=0).stop 2024-04-22T23:40:14.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.031+0000 7fae29aec700 1 -- 172.21.15.67:0/4043361734 shutdown_connections 2024-04-22T23:40:14.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.032+0000 7fae29aec700 1 --2- 172.21.15.67:0/4043361734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae240ff340 0x7fae240ff7a0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:14.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.032+0000 7fae29aec700 1 --2- 172.21.15.67:0/4043361734 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fae24105350 0x7fae24105730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:14.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.032+0000 7fae29aec700 1 -- 172.21.15.67:0/4043361734 >> 172.21.15.67:0/4043361734 conn(0x7fae24074e30 msgr2=0x7fae24075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:40:14.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.032+0000 7fae29aec700 1 -- 172.21.15.67:0/4043361734 shutdown_connections 2024-04-22T23:40:14.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.032+0000 7fae29aec700 1 -- 172.21.15.67:0/4043361734 wait complete. 2024-04-22T23:40:14.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.032+0000 7fae29aec700 1 Processor -- start 2024-04-22T23:40:14.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.032+0000 7fae29aec700 1 -- start start 2024-04-22T23:40:14.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.033+0000 7fae29aec700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae240ff340 0x7fae241976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:14.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.033+0000 7fae29aec700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fae24105350 0x7fae24197c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:14.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.033+0000 7fae29aec700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae24198300 con 0x7fae240ff340 2024-04-22T23:40:14.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.033+0000 7fae29aec700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae24198440 con 0x7fae24105350 2024-04-22T23:40:14.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.033+0000 7fae237fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae240ff340 0x7fae241976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:14.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.033+0000 7fae237fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae240ff340 0x7fae241976e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45118/0 (socket says 172.21.15.67:45118) 2024-04-22T23:40:14.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.033+0000 7fae237fe700 1 -- 172.21.15.67:0/3753193570 learned_addr learned my addr 172.21.15.67:0/3753193570 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:40:14.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.033+0000 7fae22ffd700 1 --2- 172.21.15.67:0/3753193570 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fae24105350 0x7fae24197c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:14.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.033+0000 7fae237fe700 1 -- 172.21.15.67:0/3753193570 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fae24105350 msgr2=0x7fae24197c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:14.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.033+0000 7fae237fe700 1 --2- 172.21.15.67:0/3753193570 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fae24105350 0x7fae24197c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:14.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.033+0000 7fae237fe700 1 -- 172.21.15.67:0/3753193570 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae180095f0 con 0x7fae240ff340 2024-04-22T23:40:14.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.033+0000 7fae237fe700 1 --2- 172.21.15.67:0/3753193570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae240ff340 0x7fae241976e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fae1400c8a0 tx=0x7fae1400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:40:14.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.033+0000 7fae20ff9700 1 -- 172.21.15.67:0/3753193570 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae1400cea0 con 0x7fae240ff340 2024-04-22T23:40:14.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.033+0000 7fae20ff9700 1 -- 172.21.15.67:0/3753193570 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fae14004830 con 0x7fae240ff340 2024-04-22T23:40:14.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.033+0000 7fae29aec700 1 -- 172.21.15.67:0/3753193570 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae241a07b0 con 0x7fae240ff340 2024-04-22T23:40:14.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.034+0000 7fae20ff9700 1 -- 172.21.15.67:0/3753193570 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae140055b0 con 0x7fae240ff340 2024-04-22T23:40:14.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.034+0000 7fae29aec700 1 -- 172.21.15.67:0/3753193570 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae241a0cd0 con 0x7fae240ff340 2024-04-22T23:40:14.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.034+0000 7fae29aec700 1 -- 172.21.15.67:0/3753193570 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae2404f2e0 con 0x7fae240ff340 2024-04-22T23:40:14.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.035+0000 7fae20ff9700 1 -- 172.21.15.67:0/3753193570 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fae140049a0 con 0x7fae240ff340 2024-04-22T23:40:14.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.035+0000 7fae20ff9700 1 --2- 172.21.15.67:0/3753193570 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fae0c06ecc0 0x7fae0c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:14.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.035+0000 7fae20ff9700 1 -- 172.21.15.67:0/3753193570 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fae1408ecc0 con 0x7fae240ff340 2024-04-22T23:40:14.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.036+0000 7fae22ffd700 1 --2- 172.21.15.67:0/3753193570 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fae0c06ecc0 0x7fae0c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:14.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.036+0000 7fae22ffd700 1 --2- 172.21.15.67:0/3753193570 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fae0c06ecc0 0x7fae0c071180 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fae180059d0 tx=0x7fae1801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:40:14.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.040+0000 7fae20ff9700 1 -- 172.21.15.67:0/3753193570 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fae14057900 con 0x7fae240ff340 2024-04-22T23:40:14.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.260+0000 7fae29aec700 1 -- 172.21.15.67:0/3753193570 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fae24104180 con 0x7fae0c06ecc0 2024-04-22T23:40:14.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.271+0000 7fae20ff9700 1 -- 172.21.15.67:0/3753193570 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fae24104180 con 0x7fae0c06ecc0 2024-04-22T23:40:14.264 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:40:14.264 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (13m) 7m ago 25m 25.5M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:40:14.264 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (25m) 7m ago 25m 9114k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:40:14.264 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (24m) 6m ago 24m 9042k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:40:14.264 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (11m) 7m ago 25m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:40:14.264 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (11m) 6m ago 24m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:40:14.264 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (13m) 7m ago 24m 79.3M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:40:14.265 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (7m) 7m ago 19m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:40:14.265 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (7m) 7m ago 19m 12.6M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:40:14.265 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (6m) 6m ago 19m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:40:14.265 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (19m) 6m ago 19m 12.2M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:40:14.265 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (15m) 7m ago 26m 511M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:40:14.265 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (14m) 6m ago 24m 453M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:40:14.265 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (12m) 7m ago 26m 55.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:40:14.265 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (12m) 6m ago 24m 46.0M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:40:14.265 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (14m) 7m ago 25m 15.4M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:40:14.265 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (14m) 6m ago 24m 16.1M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:40:14.265 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (10m) 7m ago 23m 99.2M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:40:14.265 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (10m) 7m ago 23m 127M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:40:14.265 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (9m) 7m ago 22m 79.9M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:40:14.265 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (9m) 6m ago 22m 97.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:40:14.265 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (8m) 6m ago 21m 82.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:40:14.265 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (7m) 6m ago 21m 77.6M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:40:14.266 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (14m) 7m ago 24m 73.9M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:40:14.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.274+0000 7fae29aec700 1 -- 172.21.15.67:0/3753193570 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fae0c06ecc0 msgr2=0x7fae0c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:14.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.274+0000 7fae29aec700 1 --2- 172.21.15.67:0/3753193570 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fae0c06ecc0 0x7fae0c071180 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fae180059d0 tx=0x7fae1801a040 comp rx=0 tx=0).stop 2024-04-22T23:40:14.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.274+0000 7fae29aec700 1 -- 172.21.15.67:0/3753193570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae240ff340 msgr2=0x7fae241976e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:14.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.274+0000 7fae29aec700 1 --2- 172.21.15.67:0/3753193570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae240ff340 0x7fae241976e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fae1400c8a0 tx=0x7fae1400cc60 comp rx=0 tx=0).stop 2024-04-22T23:40:14.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.274+0000 7fae29aec700 1 -- 172.21.15.67:0/3753193570 shutdown_connections 2024-04-22T23:40:14.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.274+0000 7fae29aec700 1 --2- 172.21.15.67:0/3753193570 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fae0c06ecc0 0x7fae0c071180 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:14.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.274+0000 7fae29aec700 1 --2- 172.21.15.67:0/3753193570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae240ff340 0x7fae241976e0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:14.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.274+0000 7fae29aec700 1 --2- 172.21.15.67:0/3753193570 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fae24105350 0x7fae24197c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:14.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.274+0000 7fae29aec700 1 -- 172.21.15.67:0/3753193570 >> 172.21.15.67:0/3753193570 conn(0x7fae24074e30 msgr2=0x7fae24103a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:40:14.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.274+0000 7fae29aec700 1 -- 172.21.15.67:0/3753193570 shutdown_connections 2024-04-22T23:40:14.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.274+0000 7fae29aec700 1 -- 172.21.15.67:0/3753193570 wait complete. 2024-04-22T23:40:14.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.381+0000 7fd456d2a700 1 -- 172.21.15.67:0/1555238324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4500ffda0 msgr2=0x7fd450100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:14.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.381+0000 7fd456d2a700 1 --2- 172.21.15.67:0/1555238324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4500ffda0 0x7fd450100180 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd440009910 tx=0x7fd440009c20 comp rx=0 tx=0).stop 2024-04-22T23:40:14.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.381+0000 7fd456d2a700 1 -- 172.21.15.67:0/1555238324 shutdown_connections 2024-04-22T23:40:14.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.381+0000 7fd456d2a700 1 --2- 172.21.15.67:0/1555238324 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd450100750 0x7fd4501047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:14.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.381+0000 7fd456d2a700 1 --2- 172.21.15.67:0/1555238324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4500ffda0 0x7fd450100180 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:14.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.381+0000 7fd456d2a700 1 -- 172.21.15.67:0/1555238324 >> 172.21.15.67:0/1555238324 conn(0x7fd4500fb650 msgr2=0x7fd4500fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:40:14.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.381+0000 7fd456d2a700 1 -- 172.21.15.67:0/1555238324 shutdown_connections 2024-04-22T23:40:14.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.381+0000 7fd456d2a700 1 -- 172.21.15.67:0/1555238324 wait complete. 2024-04-22T23:40:14.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.381+0000 7fd456d2a700 1 Processor -- start 2024-04-22T23:40:14.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.382+0000 7fd456d2a700 1 -- start start 2024-04-22T23:40:14.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.382+0000 7fd456d2a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4500ffda0 0x7fd45019c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:14.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.382+0000 7fd456d2a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd450100750 0x7fd45019cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:14.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.382+0000 7fd456d2a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd45019d180 con 0x7fd4500ffda0 2024-04-22T23:40:14.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.382+0000 7fd456d2a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4501966c0 con 0x7fd450100750 2024-04-22T23:40:14.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.382+0000 7fd454ac6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4500ffda0 0x7fd45019c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:14.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.382+0000 7fd454ac6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4500ffda0 0x7fd45019c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45138/0 (socket says 172.21.15.67:45138) 2024-04-22T23:40:14.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.382+0000 7fd454ac6700 1 -- 172.21.15.67:0/2004690551 learned_addr learned my addr 172.21.15.67:0/2004690551 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:40:14.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.382+0000 7fd454ac6700 1 -- 172.21.15.67:0/2004690551 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd450100750 msgr2=0x7fd45019cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:14.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.382+0000 7fd454ac6700 1 --2- 172.21.15.67:0/2004690551 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd450100750 0x7fd45019cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:14.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.382+0000 7fd454ac6700 1 -- 172.21.15.67:0/2004690551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4400095f0 con 0x7fd4500ffda0 2024-04-22T23:40:14.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.382+0000 7fd454ac6700 1 --2- 172.21.15.67:0/2004690551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4500ffda0 0x7fd45019c640 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fd440006010 tx=0x7fd44000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:40:14.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.382+0000 7fd44dffb700 1 -- 172.21.15.67:0/2004690551 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd44001d070 con 0x7fd4500ffda0 2024-04-22T23:40:14.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.382+0000 7fd44dffb700 1 -- 172.21.15.67:0/2004690551 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd440004fb0 con 0x7fd4500ffda0 2024-04-22T23:40:14.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.383+0000 7fd44dffb700 1 -- 172.21.15.67:0/2004690551 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd440022ea0 con 0x7fd4500ffda0 2024-04-22T23:40:14.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.383+0000 7fd456d2a700 1 -- 172.21.15.67:0/2004690551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd450196940 con 0x7fd4500ffda0 2024-04-22T23:40:14.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.383+0000 7fd456d2a700 1 -- 172.21.15.67:0/2004690551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd450196e30 con 0x7fd4500ffda0 2024-04-22T23:40:14.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.384+0000 7fd456d2a700 1 -- 172.21.15.67:0/2004690551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd45004f2e0 con 0x7fd4500ffda0 2024-04-22T23:40:14.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.384+0000 7fd44dffb700 1 -- 172.21.15.67:0/2004690551 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd440022470 con 0x7fd4500ffda0 2024-04-22T23:40:14.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.385+0000 7fd44dffb700 1 --2- 172.21.15.67:0/2004690551 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd43806ecc0 0x7fd438071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:14.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.385+0000 7fd44dffb700 1 -- 172.21.15.67:0/2004690551 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fd440090570 con 0x7fd4500ffda0 2024-04-22T23:40:14.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.385+0000 7fd44ffff700 1 --2- 172.21.15.67:0/2004690551 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd43806ecc0 0x7fd438071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:14.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.385+0000 7fd44ffff700 1 --2- 172.21.15.67:0/2004690551 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd43806ecc0 0x7fd438071180 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd444005fd0 tx=0x7fd444005d00 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:40:14.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.390+0000 7fd44dffb700 1 -- 172.21.15.67:0/2004690551 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fd44005cd90 con 0x7fd4500ffda0 2024-04-22T23:40:14.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.689+0000 7fd456d2a700 1 -- 172.21.15.67:0/2004690551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd450197bd0 con 0x7fd4500ffda0 2024-04-22T23:40:14.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.690+0000 7fd44dffb700 1 -- 172.21.15.67:0/2004690551 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fd440022720 con 0x7fd4500ffda0 2024-04-22T23:40:14.683 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:40:14.683 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:40:14.683 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:40:14.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:40:14.683 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:40:14.683 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:40:14.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:40:14.683 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:40:14.683 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:40:14.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:40:14.684 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:40:14.684 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:40:14.684 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 3 2024-04-22T23:40:14.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:40:14.684 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:40:14.684 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:40:14.684 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 13 2024-04-22T23:40:14.684 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:40:14.684 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:40:14.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.692+0000 7fd456d2a700 1 -- 172.21.15.67:0/2004690551 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd43806ecc0 msgr2=0x7fd438071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:14.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.692+0000 7fd456d2a700 1 --2- 172.21.15.67:0/2004690551 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd43806ecc0 0x7fd438071180 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd444005fd0 tx=0x7fd444005d00 comp rx=0 tx=0).stop 2024-04-22T23:40:14.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.692+0000 7fd456d2a700 1 -- 172.21.15.67:0/2004690551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4500ffda0 msgr2=0x7fd45019c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:14.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.692+0000 7fd456d2a700 1 --2- 172.21.15.67:0/2004690551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4500ffda0 0x7fd45019c640 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fd440006010 tx=0x7fd44000be20 comp rx=0 tx=0).stop 2024-04-22T23:40:14.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.692+0000 7fd456d2a700 1 -- 172.21.15.67:0/2004690551 shutdown_connections 2024-04-22T23:40:14.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.692+0000 7fd456d2a700 1 --2- 172.21.15.67:0/2004690551 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd43806ecc0 0x7fd438071180 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:14.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.692+0000 7fd456d2a700 1 --2- 172.21.15.67:0/2004690551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4500ffda0 0x7fd45019c640 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:14.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.692+0000 7fd456d2a700 1 --2- 172.21.15.67:0/2004690551 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd450100750 0x7fd45019cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:14.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.692+0000 7fd456d2a700 1 -- 172.21.15.67:0/2004690551 >> 172.21.15.67:0/2004690551 conn(0x7fd4500fb650 msgr2=0x7fd450103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:40:14.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.692+0000 7fd456d2a700 1 -- 172.21.15.67:0/2004690551 shutdown_connections 2024-04-22T23:40:14.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.692+0000 7fd456d2a700 1 -- 172.21.15.67:0/2004690551 wait complete. 2024-04-22T23:40:14.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.799+0000 7fc2f7e68700 1 -- 172.21.15.67:0/984822151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2f0100020 msgr2=0x7fc2f0103f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:14.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.799+0000 7fc2f7e68700 1 --2- 172.21.15.67:0/984822151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2f0100020 0x7fc2f0103f10 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fc2e0009910 tx=0x7fc2e0009c20 comp rx=0 tx=0).stop 2024-04-22T23:40:14.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.799+0000 7fc2f7e68700 1 -- 172.21.15.67:0/984822151 shutdown_connections 2024-04-22T23:40:14.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.799+0000 7fc2f7e68700 1 --2- 172.21.15.67:0/984822151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2f0100020 0x7fc2f0103f10 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:14.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.799+0000 7fc2f7e68700 1 --2- 172.21.15.67:0/984822151 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2f00ff670 0x7fc2f00ffa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:14.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.799+0000 7fc2f7e68700 1 -- 172.21.15.67:0/984822151 >> 172.21.15.67:0/984822151 conn(0x7fc2f0074e30 msgr2=0x7fc2f0075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:40:14.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.799+0000 7fc2f7e68700 1 -- 172.21.15.67:0/984822151 shutdown_connections 2024-04-22T23:40:14.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.799+0000 7fc2f7e68700 1 -- 172.21.15.67:0/984822151 wait complete. 2024-04-22T23:40:14.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.799+0000 7fc2f7e68700 1 Processor -- start 2024-04-22T23:40:14.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.800+0000 7fc2f7e68700 1 -- start start 2024-04-22T23:40:14.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.800+0000 7fc2f7e68700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2f00ff670 0x7fc2f0100fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:14.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.800+0000 7fc2f7e68700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2f0100020 0x7fc2f01014f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:14.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.800+0000 7fc2f7e68700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f0101a30 con 0x7fc2f00ff670 2024-04-22T23:40:14.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.800+0000 7fc2f7e68700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f0101b70 con 0x7fc2f0100020 2024-04-22T23:40:14.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.800+0000 7fc2f5c04700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2f00ff670 0x7fc2f0100fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:14.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.800+0000 7fc2f5c04700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2f00ff670 0x7fc2f0100fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45166/0 (socket says 172.21.15.67:45166) 2024-04-22T23:40:14.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.800+0000 7fc2f5c04700 1 -- 172.21.15.67:0/865237907 learned_addr learned my addr 172.21.15.67:0/865237907 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:40:14.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.800+0000 7fc2f5c04700 1 -- 172.21.15.67:0/865237907 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2f0100020 msgr2=0x7fc2f01014f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:14.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.800+0000 7fc2f5c04700 1 --2- 172.21.15.67:0/865237907 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2f0100020 0x7fc2f01014f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:14.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.800+0000 7fc2f5c04700 1 -- 172.21.15.67:0/865237907 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e00095f0 con 0x7fc2f00ff670 2024-04-22T23:40:14.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.800+0000 7fc2f5c04700 1 --2- 172.21.15.67:0/865237907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2f00ff670 0x7fc2f0100fb0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec00dd70 tx=0x7fc2ec00a530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:40:14.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.800+0000 7fc2e6ffd700 1 -- 172.21.15.67:0/865237907 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2ec00ad50 con 0x7fc2f00ff670 2024-04-22T23:40:14.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.800+0000 7fc2f7e68700 1 -- 172.21.15.67:0/865237907 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2f01a2a50 con 0x7fc2f00ff670 2024-04-22T23:40:14.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.801+0000 7fc2e6ffd700 1 -- 172.21.15.67:0/865237907 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc2ec004cf0 con 0x7fc2f00ff670 2024-04-22T23:40:14.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.801+0000 7fc2e6ffd700 1 -- 172.21.15.67:0/865237907 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2ec00e680 con 0x7fc2f00ff670 2024-04-22T23:40:14.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.801+0000 7fc2f7e68700 1 -- 172.21.15.67:0/865237907 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2f01a2e60 con 0x7fc2f00ff670 2024-04-22T23:40:14.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.802+0000 7fc2f7e68700 1 -- 172.21.15.67:0/865237907 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2f0109830 con 0x7fc2f00ff670 2024-04-22T23:40:14.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.802+0000 7fc2e6ffd700 1 -- 172.21.15.67:0/865237907 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc2ec004750 con 0x7fc2f00ff670 2024-04-22T23:40:14.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.803+0000 7fc2e6ffd700 1 --2- 172.21.15.67:0/865237907 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc2dc06ecc0 0x7fc2dc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:14.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.803+0000 7fc2e6ffd700 1 -- 172.21.15.67:0/865237907 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fc2ec08eff0 con 0x7fc2f00ff670 2024-04-22T23:40:14.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.803+0000 7fc2f5403700 1 --2- 172.21.15.67:0/865237907 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc2dc06ecc0 0x7fc2dc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:14.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.803+0000 7fc2f5403700 1 --2- 172.21.15.67:0/865237907 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc2dc06ecc0 0x7fc2dc071180 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc2e0006010 tx=0x7fc2e000b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:40:14.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:14.808+0000 7fc2e6ffd700 1 -- 172.21.15.67:0/865237907 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc2ec057c30 con 0x7fc2f00ff670 2024-04-22T23:40:14.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:14 smithi134 ceph-mon[85630]: from='client.34792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:40:14.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:14 smithi134 ceph-mon[85630]: from='client.34796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:40:15.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:14 smithi067 ceph-mon[112438]: from='client.34792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:40:15.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:14 smithi067 ceph-mon[112438]: from='client.34796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:40:15.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.064+0000 7fc2f7e68700 1 -- 172.21.15.67:0/865237907 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc2f002cc70 con 0x7fc2f00ff670 2024-04-22T23:40:15.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.064+0000 7fc2e6ffd700 1 -- 172.21.15.67:0/865237907 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1871 (secure 0 0 0) 0x7fc2ec05b2f0 con 0x7fc2f00ff670 2024-04-22T23:40:15.057 INFO:teuthology.orchestra.run.smithi067.stdout:e31 2024-04-22T23:40:15.057 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:40:15.058 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:40:15.058 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:40:15.058 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:40:15.058 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:40:15.058 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:40:15.058 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:40:15.058 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:40:15.058 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:40:15.058 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:40:15.058 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:40:15.058 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:40:15.058 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:40:15.058 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:40:15.059 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:40:15.059 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:40:15.059 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:40:15.059 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:40:15.059 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:40:15.059 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:40:15.059 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:40:15.059 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:40:15.059 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:40:15.059 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:40:15.059 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:40:15.059 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:40:15.059 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:40:15.059 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:40:15.059 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:40:15.059 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:40:15.059 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:40:15.060 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:40:15.060 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:40:15.060 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:40:15.060 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:40:15.060 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:40:15.060 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:40:15.060 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:40:15.060 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:40:15.060 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44259} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1827288036,v1:172.21.15.134:6827/1827288036] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:40:15.060 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:40:15.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.067+0000 7fc2f7e68700 1 -- 172.21.15.67:0/865237907 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc2dc06ecc0 msgr2=0x7fc2dc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:15.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.067+0000 7fc2f7e68700 1 --2- 172.21.15.67:0/865237907 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc2dc06ecc0 0x7fc2dc071180 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc2e0006010 tx=0x7fc2e000b540 comp rx=0 tx=0).stop 2024-04-22T23:40:15.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.067+0000 7fc2f7e68700 1 -- 172.21.15.67:0/865237907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2f00ff670 msgr2=0x7fc2f0100fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:15.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.067+0000 7fc2f7e68700 1 --2- 172.21.15.67:0/865237907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2f00ff670 0x7fc2f0100fb0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec00dd70 tx=0x7fc2ec00a530 comp rx=0 tx=0).stop 2024-04-22T23:40:15.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.067+0000 7fc2f7e68700 1 -- 172.21.15.67:0/865237907 shutdown_connections 2024-04-22T23:40:15.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.067+0000 7fc2f7e68700 1 --2- 172.21.15.67:0/865237907 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc2dc06ecc0 0x7fc2dc071180 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:15.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.067+0000 7fc2f7e68700 1 --2- 172.21.15.67:0/865237907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2f00ff670 0x7fc2f0100fb0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:15.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.067+0000 7fc2f7e68700 1 --2- 172.21.15.67:0/865237907 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2f0100020 0x7fc2f01014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:15.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.067+0000 7fc2f7e68700 1 -- 172.21.15.67:0/865237907 >> 172.21.15.67:0/865237907 conn(0x7fc2f0074e30 msgr2=0x7fc2f00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:40:15.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.067+0000 7fc2f7e68700 1 -- 172.21.15.67:0/865237907 shutdown_connections 2024-04-22T23:40:15.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.067+0000 7fc2f7e68700 1 -- 172.21.15.67:0/865237907 wait complete. 2024-04-22T23:40:15.062 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 31 2024-04-22T23:40:15.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.175+0000 7f560b511700 1 -- 172.21.15.67:0/2015570348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5604100ea0 msgr2=0x7f5604101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:15.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.175+0000 7f560b511700 1 --2- 172.21.15.67:0/2015570348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5604100ea0 0x7f5604101280 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f55f4009910 tx=0x7f55f4009c20 comp rx=0 tx=0).stop 2024-04-22T23:40:15.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.175+0000 7f560b511700 1 -- 172.21.15.67:0/2015570348 shutdown_connections 2024-04-22T23:40:15.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.175+0000 7f560b511700 1 --2- 172.21.15.67:0/2015570348 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5604101850 0x7f56041058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:15.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.175+0000 7f560b511700 1 --2- 172.21.15.67:0/2015570348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5604100ea0 0x7f5604101280 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:15.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.175+0000 7f560b511700 1 -- 172.21.15.67:0/2015570348 >> 172.21.15.67:0/2015570348 conn(0x7f56040fc750 msgr2=0x7f56040feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:40:15.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.175+0000 7f560b511700 1 -- 172.21.15.67:0/2015570348 shutdown_connections 2024-04-22T23:40:15.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.176+0000 7f560b511700 1 -- 172.21.15.67:0/2015570348 wait complete. 2024-04-22T23:40:15.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.176+0000 7f560b511700 1 Processor -- start 2024-04-22T23:40:15.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.176+0000 7f560b511700 1 -- start start 2024-04-22T23:40:15.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.176+0000 7f560b511700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5604100ea0 0x7f560419c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:15.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.176+0000 7f560b511700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5604101850 0x7f560419cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:15.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.176+0000 7f560b511700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560419d2c0 con 0x7f5604100ea0 2024-04-22T23:40:15.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.176+0000 7f560b511700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5604196770 con 0x7f5604101850 2024-04-22T23:40:15.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.177+0000 7f56092ad700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5604100ea0 0x7f560419c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:15.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.177+0000 7f56092ad700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5604100ea0 0x7f560419c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45182/0 (socket says 172.21.15.67:45182) 2024-04-22T23:40:15.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.177+0000 7f56092ad700 1 -- 172.21.15.67:0/3497865898 learned_addr learned my addr 172.21.15.67:0/3497865898 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:40:15.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.177+0000 7f5608aac700 1 --2- 172.21.15.67:0/3497865898 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5604101850 0x7f560419cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:15.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.177+0000 7f56092ad700 1 -- 172.21.15.67:0/3497865898 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5604101850 msgr2=0x7f560419cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:15.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.177+0000 7f56092ad700 1 --2- 172.21.15.67:0/3497865898 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5604101850 0x7f560419cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:15.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.177+0000 7f56092ad700 1 -- 172.21.15.67:0/3497865898 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55f40095f0 con 0x7f5604100ea0 2024-04-22T23:40:15.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.177+0000 7f5608aac700 1 --2- 172.21.15.67:0/3497865898 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5604101850 0x7f560419cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:40:15.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.177+0000 7f56092ad700 1 --2- 172.21.15.67:0/3497865898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5604100ea0 0x7f560419c6f0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f55f4006010 tx=0x7f55f400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:40:15.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.177+0000 7f55fa7fc700 1 -- 172.21.15.67:0/3497865898 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55f401d070 con 0x7f5604100ea0 2024-04-22T23:40:15.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.177+0000 7f560b511700 1 -- 172.21.15.67:0/3497865898 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56041969f0 con 0x7f5604100ea0 2024-04-22T23:40:15.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.177+0000 7f55fa7fc700 1 -- 172.21.15.67:0/3497865898 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f55f4004fb0 con 0x7f5604100ea0 2024-04-22T23:40:15.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.177+0000 7f55fa7fc700 1 -- 172.21.15.67:0/3497865898 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55f400f690 con 0x7f5604100ea0 2024-04-22T23:40:15.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.177+0000 7f560b511700 1 -- 172.21.15.67:0/3497865898 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5604196ee0 con 0x7f5604100ea0 2024-04-22T23:40:15.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.178+0000 7f560b511700 1 -- 172.21.15.67:0/3497865898 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5604109240 con 0x7f5604100ea0 2024-04-22T23:40:15.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.179+0000 7f55fa7fc700 1 -- 172.21.15.67:0/3497865898 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f55f40228a0 con 0x7f5604100ea0 2024-04-22T23:40:15.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.179+0000 7f55fa7fc700 1 --2- 172.21.15.67:0/3497865898 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f55f006ecc0 0x7f55f0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:15.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.179+0000 7f55fa7fc700 1 -- 172.21.15.67:0/3497865898 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f55f4090840 con 0x7f5604100ea0 2024-04-22T23:40:15.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.179+0000 7f5608aac700 1 --2- 172.21.15.67:0/3497865898 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f55f006ecc0 0x7f55f0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:15.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.180+0000 7f5608aac700 1 --2- 172.21.15.67:0/3497865898 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f55f006ecc0 0x7f55f0071180 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5604197be0 tx=0x7f5600005f50 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:40:15.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.184+0000 7f55fa7fc700 1 -- 172.21.15.67:0/3497865898 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f55f405cfb0 con 0x7f5604100ea0 2024-04-22T23:40:15.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.413+0000 7f560b511700 1 -- 172.21.15.67:0/3497865898 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5604197cd0 con 0x7f55f006ecc0 2024-04-22T23:40:15.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.415+0000 7f55fa7fc700 1 -- 172.21.15.67:0/3497865898 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f5604197cd0 con 0x7f55f006ecc0 2024-04-22T23:40:15.408 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:40:15.408 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:40:15.408 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:40:15.408 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:40:15.408 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:40:15.408 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:40:15.408 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:40:15.408 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:40:15.408 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:40:15.409 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:40:15.409 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T23:40:15.409 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:40:15.409 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:40:15.409 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:40:15.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.417+0000 7f560b511700 1 -- 172.21.15.67:0/3497865898 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f55f006ecc0 msgr2=0x7f55f0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:15.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.417+0000 7f560b511700 1 --2- 172.21.15.67:0/3497865898 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f55f006ecc0 0x7f55f0071180 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5604197be0 tx=0x7f5600005f50 comp rx=0 tx=0).stop 2024-04-22T23:40:15.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.417+0000 7f560b511700 1 -- 172.21.15.67:0/3497865898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5604100ea0 msgr2=0x7f560419c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:15.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.417+0000 7f560b511700 1 --2- 172.21.15.67:0/3497865898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5604100ea0 0x7f560419c6f0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f55f4006010 tx=0x7f55f400be20 comp rx=0 tx=0).stop 2024-04-22T23:40:15.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.417+0000 7f560b511700 1 -- 172.21.15.67:0/3497865898 shutdown_connections 2024-04-22T23:40:15.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.417+0000 7f560b511700 1 --2- 172.21.15.67:0/3497865898 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f55f006ecc0 0x7f55f0071180 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:15.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.417+0000 7f560b511700 1 --2- 172.21.15.67:0/3497865898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5604100ea0 0x7f560419c6f0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:15.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.417+0000 7f560b511700 1 --2- 172.21.15.67:0/3497865898 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5604101850 0x7f560419cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:15.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.417+0000 7f560b511700 1 -- 172.21.15.67:0/3497865898 >> 172.21.15.67:0/3497865898 conn(0x7f56040fc750 msgr2=0x7f56040fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:40:15.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.417+0000 7f560b511700 1 -- 172.21.15.67:0/3497865898 shutdown_connections 2024-04-22T23:40:15.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.417+0000 7f560b511700 1 -- 172.21.15.67:0/3497865898 wait complete. 2024-04-22T23:40:15.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.524+0000 7f7cccd49700 1 -- 172.21.15.67:0/1149488744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cc8102700 msgr2=0x7f7cc810abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:15.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.524+0000 7f7cccd49700 1 --2- 172.21.15.67:0/1149488744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cc8102700 0x7f7cc810abf0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0009910 tx=0x7f7cb0009c20 comp rx=0 tx=0).stop 2024-04-22T23:40:15.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.525+0000 7f7cccd49700 1 -- 172.21.15.67:0/1149488744 shutdown_connections 2024-04-22T23:40:15.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.525+0000 7f7cccd49700 1 --2- 172.21.15.67:0/1149488744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cc8102700 0x7f7cc810abf0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:15.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.525+0000 7f7cccd49700 1 --2- 172.21.15.67:0/1149488744 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7cc8101de0 0x7f7cc81021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:15.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.525+0000 7f7cccd49700 1 -- 172.21.15.67:0/1149488744 >> 172.21.15.67:0/1149488744 conn(0x7f7cc80fb610 msgr2=0x7f7cc80fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:40:15.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.525+0000 7f7cccd49700 1 -- 172.21.15.67:0/1149488744 shutdown_connections 2024-04-22T23:40:15.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.525+0000 7f7cccd49700 1 -- 172.21.15.67:0/1149488744 wait complete. 2024-04-22T23:40:15.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.525+0000 7f7cccd49700 1 Processor -- start 2024-04-22T23:40:15.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.526+0000 7f7cccd49700 1 -- start start 2024-04-22T23:40:15.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.526+0000 7f7cccd49700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cc8101de0 0x7f7cc819c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:15.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.526+0000 7f7cccd49700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7cc8102700 0x7f7cc819cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:15.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.526+0000 7f7cccd49700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc819d220 con 0x7f7cc8101de0 2024-04-22T23:40:15.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.526+0000 7f7cccd49700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc8196720 con 0x7f7cc8102700 2024-04-22T23:40:15.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.526+0000 7f7cc659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cc8101de0 0x7f7cc819c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:15.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.526+0000 7f7cc659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cc8101de0 0x7f7cc819c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45206/0 (socket says 172.21.15.67:45206) 2024-04-22T23:40:15.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.526+0000 7f7cc659c700 1 -- 172.21.15.67:0/2561457689 learned_addr learned my addr 172.21.15.67:0/2561457689 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:40:15.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.526+0000 7f7cc5d9b700 1 --2- 172.21.15.67:0/2561457689 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7cc8102700 0x7f7cc819cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:15.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.526+0000 7f7cc659c700 1 -- 172.21.15.67:0/2561457689 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7cc8102700 msgr2=0x7f7cc819cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:15.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.526+0000 7f7cc659c700 1 --2- 172.21.15.67:0/2561457689 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7cc8102700 0x7f7cc819cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:15.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.526+0000 7f7cc659c700 1 -- 172.21.15.67:0/2561457689 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb00095f0 con 0x7f7cc8101de0 2024-04-22T23:40:15.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.526+0000 7f7cc659c700 1 --2- 172.21.15.67:0/2561457689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cc8101de0 0x7f7cc819c650 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7cc000ef10 tx=0x7f7cc000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:40:15.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.527+0000 7f7cbf7fe700 1 -- 172.21.15.67:0/2561457689 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7cc000c9f0 con 0x7f7cc8101de0 2024-04-22T23:40:15.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.527+0000 7f7cbf7fe700 1 -- 172.21.15.67:0/2561457689 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7cc000f040 con 0x7f7cc8101de0 2024-04-22T23:40:15.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.527+0000 7f7cccd49700 1 -- 172.21.15.67:0/2561457689 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cc81969a0 con 0x7f7cc8101de0 2024-04-22T23:40:15.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.527+0000 7f7cbf7fe700 1 -- 172.21.15.67:0/2561457689 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7cc00146b0 con 0x7f7cc8101de0 2024-04-22T23:40:15.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.527+0000 7f7cccd49700 1 -- 172.21.15.67:0/2561457689 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cc8196ec0 con 0x7f7cc8101de0 2024-04-22T23:40:15.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.528+0000 7f7cccd49700 1 -- 172.21.15.67:0/2561457689 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cc81082f0 con 0x7f7cc8101de0 2024-04-22T23:40:15.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.528+0000 7f7cbf7fe700 1 -- 172.21.15.67:0/2561457689 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7cc000cb50 con 0x7f7cc8101de0 2024-04-22T23:40:15.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.529+0000 7f7cbf7fe700 1 --2- 172.21.15.67:0/2561457689 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7cb406ecc0 0x7f7cb4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:15.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.529+0000 7f7cbf7fe700 1 -- 172.21.15.67:0/2561457689 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f7cc008f320 con 0x7f7cc8101de0 2024-04-22T23:40:15.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.529+0000 7f7cc5d9b700 1 --2- 172.21.15.67:0/2561457689 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7cb406ecc0 0x7f7cb4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:15.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.529+0000 7f7cc5d9b700 1 --2- 172.21.15.67:0/2561457689 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7cb406ecc0 0x7f7cb4071180 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0005b40 tx=0x7f7cb0005a90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:40:15.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.534+0000 7f7cbf7fe700 1 -- 172.21.15.67:0/2561457689 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7cc005ba90 con 0x7f7cc8101de0 2024-04-22T23:40:15.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:15 smithi067 ceph-mon[112438]: pgmap v416: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:15.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:15 smithi067 ceph-mon[112438]: from='client.34800 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:40:15.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:15 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2004690551' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:40:15.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:15 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/865237907' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:40:15.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.834+0000 7f7cccd49700 1 -- 172.21.15.67:0/2561457689 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7cc80679c0 con 0x7f7cc8101de0 2024-04-22T23:40:15.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.834+0000 7f7cbf7fe700 1 -- 172.21.15.67:0/2561457689 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f7cc0007e60 con 0x7f7cc8101de0 2024-04-22T23:40:15.827 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:40:15.827 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:40:15.827 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:40:15.828 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:40:15.828 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:40:15.828 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:40:15.828 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:40:15.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.836+0000 7f7cccd49700 1 -- 172.21.15.67:0/2561457689 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7cb406ecc0 msgr2=0x7f7cb4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:15.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.836+0000 7f7cccd49700 1 --2- 172.21.15.67:0/2561457689 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7cb406ecc0 0x7f7cb4071180 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0005b40 tx=0x7f7cb0005a90 comp rx=0 tx=0).stop 2024-04-22T23:40:15.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.836+0000 7f7cccd49700 1 -- 172.21.15.67:0/2561457689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cc8101de0 msgr2=0x7f7cc819c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:15.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.836+0000 7f7cccd49700 1 --2- 172.21.15.67:0/2561457689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cc8101de0 0x7f7cc819c650 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7cc000ef10 tx=0x7f7cc000c3a0 comp rx=0 tx=0).stop 2024-04-22T23:40:15.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.836+0000 7f7cccd49700 1 -- 172.21.15.67:0/2561457689 shutdown_connections 2024-04-22T23:40:15.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.836+0000 7f7cccd49700 1 --2- 172.21.15.67:0/2561457689 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7cb406ecc0 0x7f7cb4071180 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:15.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.836+0000 7f7cccd49700 1 --2- 172.21.15.67:0/2561457689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cc8101de0 0x7f7cc819c650 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:15.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.836+0000 7f7cccd49700 1 --2- 172.21.15.67:0/2561457689 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7cc8102700 0x7f7cc819cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:15.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.836+0000 7f7cccd49700 1 -- 172.21.15.67:0/2561457689 >> 172.21.15.67:0/2561457689 conn(0x7f7cc80fb610 msgr2=0x7f7cc8105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:40:15.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.836+0000 7f7cccd49700 1 -- 172.21.15.67:0/2561457689 shutdown_connections 2024-04-22T23:40:15.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:15.836+0000 7f7cccd49700 1 -- 172.21.15.67:0/2561457689 wait complete. 2024-04-22T23:40:15.888 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:15 smithi134 ceph-mon[85630]: pgmap v416: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:15.888 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:15 smithi134 ceph-mon[85630]: from='client.34800 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:40:15.888 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:15 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2004690551' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:40:15.888 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:15 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/865237907' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:40:16.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:16 smithi134 ceph-mon[85630]: from='client.34812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:40:16.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:16 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2561457689' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:40:17.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:16 smithi067 ceph-mon[112438]: from='client.34812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:40:17.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:16 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2561457689' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:40:17.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:17 smithi134 ceph-mon[85630]: pgmap v417: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:18.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:17 smithi067 ceph-mon[112438]: pgmap v417: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:19.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:19 smithi134 ceph-mon[85630]: pgmap v418: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:20.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:19 smithi067 ceph-mon[112438]: pgmap v418: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:21.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:21 smithi134 ceph-mon[85630]: pgmap v419: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:22.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:21 smithi067 ceph-mon[112438]: pgmap v419: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:23.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:23 smithi134 ceph-mon[85630]: pgmap v420: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:24.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:23 smithi067 ceph-mon[112438]: pgmap v420: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:25.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:25 smithi134 ceph-mon[85630]: pgmap v421: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:26.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:25 smithi067 ceph-mon[112438]: pgmap v421: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:26.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:40:27.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:40:27.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:27 smithi067 ceph-mon[112438]: pgmap v422: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:27.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:27 smithi134 ceph-mon[85630]: pgmap v422: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:29.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:29 smithi134 ceph-mon[85630]: pgmap v423: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:30.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:29 smithi067 ceph-mon[112438]: pgmap v423: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:31.883 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:31 smithi134 ceph-mon[85630]: pgmap v424: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:32.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:31 smithi067 ceph-mon[112438]: pgmap v424: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:33.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:33 smithi134 ceph-mon[85630]: pgmap v425: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:34.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:33 smithi067 ceph-mon[112438]: pgmap v425: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:35.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:35 smithi134 ceph-mon[85630]: pgmap v426: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:36.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:35 smithi067 ceph-mon[112438]: pgmap v426: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:37.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:37 smithi067 ceph-mon[112438]: pgmap v427: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:37.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:37 smithi134 ceph-mon[85630]: pgmap v427: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:39.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:39 smithi134 ceph-mon[85630]: pgmap v428: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:40.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:39 smithi067 ceph-mon[112438]: pgmap v428: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:41.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:41 smithi134 ceph-mon[85630]: pgmap v429: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:41.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:40:42.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:41 smithi067 ceph-mon[112438]: pgmap v429: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:42.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:40:43.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:43 smithi134 ceph-mon[85630]: pgmap v430: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:44.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:43 smithi067 ceph-mon[112438]: pgmap v430: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:45.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:45.959+0000 7f1aa6e4c700 1 -- 172.21.15.67:0/975740534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1aa00ffda0 msgr2=0x7f1aa0100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:45.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:45.959+0000 7f1aa6e4c700 1 --2- 172.21.15.67:0/975740534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1aa00ffda0 0x7f1aa0100180 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f1a94009910 tx=0x7f1a94009c20 comp rx=0 tx=0).stop 2024-04-22T23:40:45.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:45.959+0000 7f1aa6e4c700 1 -- 172.21.15.67:0/975740534 shutdown_connections 2024-04-22T23:40:45.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:45.959+0000 7f1aa6e4c700 1 --2- 172.21.15.67:0/975740534 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1aa0100750 0x7f1aa01047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:45.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:45.959+0000 7f1aa6e4c700 1 --2- 172.21.15.67:0/975740534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1aa00ffda0 0x7f1aa0100180 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:45.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:45.959+0000 7f1aa6e4c700 1 -- 172.21.15.67:0/975740534 >> 172.21.15.67:0/975740534 conn(0x7f1aa00fb650 msgr2=0x7f1aa00fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:40:45.953 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:45 smithi067 ceph-mon[112438]: pgmap v431: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:45.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:45.959+0000 7f1aa6e4c700 1 -- 172.21.15.67:0/975740534 shutdown_connections 2024-04-22T23:40:45.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:45.959+0000 7f1aa6e4c700 1 -- 172.21.15.67:0/975740534 wait complete. 2024-04-22T23:40:45.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:45.959+0000 7f1aa6e4c700 1 Processor -- start 2024-04-22T23:40:45.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:45.959+0000 7f1aa6e4c700 1 -- start start 2024-04-22T23:40:45.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:45.960+0000 7f1aa6e4c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1aa00ffda0 0x7f1aa019c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:45.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:45.960+0000 7f1aa6e4c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1aa0100750 0x7f1aa019cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:45.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:45.960+0000 7f1aa6e4c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa019d3b0 con 0x7f1aa00ffda0 2024-04-22T23:40:45.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:45.960+0000 7f1aa6e4c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa0196860 con 0x7f1aa0100750 2024-04-22T23:40:45.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:45.960+0000 7f1aa4be8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1aa00ffda0 0x7f1aa019c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:45.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:45.960+0000 7f1aa4be8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1aa00ffda0 0x7f1aa019c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40128/0 (socket says 172.21.15.67:40128) 2024-04-22T23:40:45.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:45.960+0000 7f1aa4be8700 1 -- 172.21.15.67:0/3900299427 learned_addr learned my addr 172.21.15.67:0/3900299427 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:40:45.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:45.960+0000 7f1a9ffff700 1 --2- 172.21.15.67:0/3900299427 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1aa0100750 0x7f1aa019cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:45.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:45.960+0000 7f1aa4be8700 1 -- 172.21.15.67:0/3900299427 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1aa0100750 msgr2=0x7f1aa019cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:45.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:45.960+0000 7f1aa4be8700 1 --2- 172.21.15.67:0/3900299427 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1aa0100750 0x7f1aa019cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:45.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:45.960+0000 7f1aa4be8700 1 -- 172.21.15.67:0/3900299427 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a940095f0 con 0x7f1aa00ffda0 2024-04-22T23:40:45.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:45.960+0000 7f1a9ffff700 1 --2- 172.21.15.67:0/3900299427 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1aa0100750 0x7f1aa019cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:40:45.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:45.960+0000 7f1aa4be8700 1 --2- 172.21.15.67:0/3900299427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1aa00ffda0 0x7f1aa019c7e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f1a94006010 tx=0x7f1a9400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:40:45.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:45.961+0000 7f1a9dffb700 1 -- 172.21.15.67:0/3900299427 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a9401d070 con 0x7f1aa00ffda0 2024-04-22T23:40:45.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:45.961+0000 7f1a9dffb700 1 -- 172.21.15.67:0/3900299427 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1a94004fb0 con 0x7f1aa00ffda0 2024-04-22T23:40:45.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:45.961+0000 7f1aa6e4c700 1 -- 172.21.15.67:0/3900299427 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aa0196ae0 con 0x7f1aa00ffda0 2024-04-22T23:40:45.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:45.961+0000 7f1a9dffb700 1 -- 172.21.15.67:0/3900299427 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a94022ea0 con 0x7f1aa00ffda0 2024-04-22T23:40:45.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:45.961+0000 7f1aa6e4c700 1 -- 172.21.15.67:0/3900299427 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aa0196fd0 con 0x7f1aa00ffda0 2024-04-22T23:40:45.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:45.962+0000 7f1aa6e4c700 1 -- 172.21.15.67:0/3900299427 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aa004f2e0 con 0x7f1aa00ffda0 2024-04-22T23:40:45.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:45.963+0000 7f1a9dffb700 1 -- 172.21.15.67:0/3900299427 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1a94022470 con 0x7f1aa00ffda0 2024-04-22T23:40:45.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:45.963+0000 7f1a9dffb700 1 --2- 172.21.15.67:0/3900299427 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1a9006ecc0 0x7f1a90071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:45.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:45.963+0000 7f1a9dffb700 1 -- 172.21.15.67:0/3900299427 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f1a94090490 con 0x7f1aa00ffda0 2024-04-22T23:40:45.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:45.963+0000 7f1a9ffff700 1 --2- 172.21.15.67:0/3900299427 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1a9006ecc0 0x7f1a90071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:45.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:45.964+0000 7f1a9ffff700 1 --2- 172.21.15.67:0/3900299427 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1a9006ecc0 0x7f1a90071180 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1aa0106760 tx=0x7f1a8c008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:40:45.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:45.968+0000 7f1a9dffb700 1 -- 172.21.15.67:0/3900299427 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1a9405cd30 con 0x7f1aa00ffda0 2024-04-22T23:40:45.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:45 smithi134 ceph-mon[85630]: pgmap v431: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:46.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.198+0000 7f1aa6e4c700 1 -- 172.21.15.67:0/3900299427 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1aa0197300 con 0x7f1a9006ecc0 2024-04-22T23:40:46.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.199+0000 7f1a9dffb700 1 -- 172.21.15.67:0/3900299427 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f1aa0197300 con 0x7f1a9006ecc0 2024-04-22T23:40:46.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.201+0000 7f1aa6e4c700 1 -- 172.21.15.67:0/3900299427 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1a9006ecc0 msgr2=0x7f1a90071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:46.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.201+0000 7f1aa6e4c700 1 --2- 172.21.15.67:0/3900299427 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1a9006ecc0 0x7f1a90071180 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1aa0106760 tx=0x7f1a8c008040 comp rx=0 tx=0).stop 2024-04-22T23:40:46.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.201+0000 7f1aa6e4c700 1 -- 172.21.15.67:0/3900299427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1aa00ffda0 msgr2=0x7f1aa019c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:46.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.201+0000 7f1aa6e4c700 1 --2- 172.21.15.67:0/3900299427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1aa00ffda0 0x7f1aa019c7e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f1a94006010 tx=0x7f1a9400be20 comp rx=0 tx=0).stop 2024-04-22T23:40:46.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.202+0000 7f1aa6e4c700 1 -- 172.21.15.67:0/3900299427 shutdown_connections 2024-04-22T23:40:46.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.202+0000 7f1aa6e4c700 1 --2- 172.21.15.67:0/3900299427 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1a9006ecc0 0x7f1a90071180 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:46.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.202+0000 7f1aa6e4c700 1 --2- 172.21.15.67:0/3900299427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1aa00ffda0 0x7f1aa019c7e0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:46.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.202+0000 7f1aa6e4c700 1 --2- 172.21.15.67:0/3900299427 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1aa0100750 0x7f1aa019cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:46.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.202+0000 7f1aa6e4c700 1 -- 172.21.15.67:0/3900299427 >> 172.21.15.67:0/3900299427 conn(0x7f1aa00fb650 msgr2=0x7f1aa0103f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:40:46.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.202+0000 7f1aa6e4c700 1 -- 172.21.15.67:0/3900299427 shutdown_connections 2024-04-22T23:40:46.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.202+0000 7f1aa6e4c700 1 -- 172.21.15.67:0/3900299427 wait complete. 2024-04-22T23:40:46.205 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:40:46.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.310+0000 7f12e0172700 1 -- 172.21.15.67:0/3314781479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12d8101de0 msgr2=0x7f12d81021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:46.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.310+0000 7f12e0172700 1 --2- 172.21.15.67:0/3314781479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12d8101de0 0x7f12d81021c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f12d4009910 tx=0x7f12d4009c20 comp rx=0 tx=0).stop 2024-04-22T23:40:46.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.310+0000 7f12e0172700 1 -- 172.21.15.67:0/3314781479 shutdown_connections 2024-04-22T23:40:46.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.310+0000 7f12e0172700 1 --2- 172.21.15.67:0/3314781479 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12d8102700 0x7f12d810abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:46.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.310+0000 7f12e0172700 1 --2- 172.21.15.67:0/3314781479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12d8101de0 0x7f12d81021c0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:46.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.310+0000 7f12e0172700 1 -- 172.21.15.67:0/3314781479 >> 172.21.15.67:0/3314781479 conn(0x7f12d80fb650 msgr2=0x7f12d80fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:40:46.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.310+0000 7f12e0172700 1 -- 172.21.15.67:0/3314781479 shutdown_connections 2024-04-22T23:40:46.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.310+0000 7f12e0172700 1 -- 172.21.15.67:0/3314781479 wait complete. 2024-04-22T23:40:46.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.310+0000 7f12e0172700 1 Processor -- start 2024-04-22T23:40:46.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.310+0000 7f12e0172700 1 -- start start 2024-04-22T23:40:46.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.311+0000 7f12e0172700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12d8101de0 0x7f12d819c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:46.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.311+0000 7f12e0172700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12d8102700 0x7f12d819cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:46.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.311+0000 7f12e0172700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d819d270 con 0x7f12d8102700 2024-04-22T23:40:46.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.311+0000 7f12e0172700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d8196720 con 0x7f12d8101de0 2024-04-22T23:40:46.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.311+0000 7f12dd70d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12d8102700 0x7f12d819cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:46.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.311+0000 7f12dd70d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12d8102700 0x7f12d819cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40148/0 (socket says 172.21.15.67:40148) 2024-04-22T23:40:46.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.311+0000 7f12dd70d700 1 -- 172.21.15.67:0/2960649602 learned_addr learned my addr 172.21.15.67:0/2960649602 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:40:46.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.311+0000 7f12ddf0e700 1 --2- 172.21.15.67:0/2960649602 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12d8101de0 0x7f12d819c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:46.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.311+0000 7f12dd70d700 1 -- 172.21.15.67:0/2960649602 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12d8101de0 msgr2=0x7f12d819c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:46.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.311+0000 7f12dd70d700 1 --2- 172.21.15.67:0/2960649602 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12d8101de0 0x7f12d819c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:46.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.311+0000 7f12dd70d700 1 -- 172.21.15.67:0/2960649602 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12d40095f0 con 0x7f12d8102700 2024-04-22T23:40:46.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.311+0000 7f12dd70d700 1 --2- 172.21.15.67:0/2960649602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12d8102700 0x7f12d819cbe0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f12c800d6d0 tx=0x7f12c800d9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:40:46.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.311+0000 7f12ceffd700 1 -- 172.21.15.67:0/2960649602 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f12c800f860 con 0x7f12d8102700 2024-04-22T23:40:46.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.312+0000 7f12ceffd700 1 -- 172.21.15.67:0/2960649602 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f12c800fe80 con 0x7f12d8102700 2024-04-22T23:40:46.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.312+0000 7f12e0172700 1 -- 172.21.15.67:0/2960649602 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12d8196a00 con 0x7f12d8102700 2024-04-22T23:40:46.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.312+0000 7f12ceffd700 1 -- 172.21.15.67:0/2960649602 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f12c8010ea0 con 0x7f12d8102700 2024-04-22T23:40:46.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.312+0000 7f12e0172700 1 -- 172.21.15.67:0/2960649602 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12d8196f50 con 0x7f12d8102700 2024-04-22T23:40:46.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.312+0000 7f12e0172700 1 -- 172.21.15.67:0/2960649602 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12d81082f0 con 0x7f12d8102700 2024-04-22T23:40:46.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.313+0000 7f12ceffd700 1 -- 172.21.15.67:0/2960649602 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f12c8010460 con 0x7f12d8102700 2024-04-22T23:40:46.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.314+0000 7f12ceffd700 1 --2- 172.21.15.67:0/2960649602 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f12c406ecc0 0x7f12c4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:46.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.314+0000 7f12ceffd700 1 -- 172.21.15.67:0/2960649602 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f12c808f150 con 0x7f12d8102700 2024-04-22T23:40:46.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.314+0000 7f12ddf0e700 1 --2- 172.21.15.67:0/2960649602 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f12c406ecc0 0x7f12c4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:46.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.314+0000 7f12ddf0e700 1 --2- 172.21.15.67:0/2960649602 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f12c406ecc0 0x7f12c4071180 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f12d4006010 tx=0x7f12d401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:40:46.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.318+0000 7f12ceffd700 1 -- 172.21.15.67:0/2960649602 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f12c805b920 con 0x7f12d8102700 2024-04-22T23:40:46.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.547+0000 7f12e0172700 1 -- 172.21.15.67:0/2960649602 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f12d8197220 con 0x7f12c406ecc0 2024-04-22T23:40:46.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.548+0000 7f12ceffd700 1 -- 172.21.15.67:0/2960649602 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f12d8197220 con 0x7f12c406ecc0 2024-04-22T23:40:46.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.550+0000 7f12e0172700 1 -- 172.21.15.67:0/2960649602 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f12c406ecc0 msgr2=0x7f12c4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:46.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.550+0000 7f12e0172700 1 --2- 172.21.15.67:0/2960649602 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f12c406ecc0 0x7f12c4071180 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f12d4006010 tx=0x7f12d401a040 comp rx=0 tx=0).stop 2024-04-22T23:40:46.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.550+0000 7f12e0172700 1 -- 172.21.15.67:0/2960649602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12d8102700 msgr2=0x7f12d819cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:46.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.550+0000 7f12e0172700 1 --2- 172.21.15.67:0/2960649602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12d8102700 0x7f12d819cbe0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f12c800d6d0 tx=0x7f12c800d9e0 comp rx=0 tx=0).stop 2024-04-22T23:40:46.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.550+0000 7f12e0172700 1 -- 172.21.15.67:0/2960649602 shutdown_connections 2024-04-22T23:40:46.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.550+0000 7f12e0172700 1 --2- 172.21.15.67:0/2960649602 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f12c406ecc0 0x7f12c4071180 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:46.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.550+0000 7f12e0172700 1 --2- 172.21.15.67:0/2960649602 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12d8101de0 0x7f12d819c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:46.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.550+0000 7f12e0172700 1 --2- 172.21.15.67:0/2960649602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12d8102700 0x7f12d819cbe0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:46.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.550+0000 7f12e0172700 1 -- 172.21.15.67:0/2960649602 >> 172.21.15.67:0/2960649602 conn(0x7f12d80fb650 msgr2=0x7f12d8105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:40:46.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.550+0000 7f12e0172700 1 -- 172.21.15.67:0/2960649602 shutdown_connections 2024-04-22T23:40:46.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.550+0000 7f12e0172700 1 -- 172.21.15.67:0/2960649602 wait complete. 2024-04-22T23:40:46.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.657+0000 7f81107c6700 1 -- 172.21.15.67:0/4242840568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8108101e00 msgr2=0x7f81081021e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:46.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.657+0000 7f81107c6700 1 --2- 172.21.15.67:0/4242840568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8108101e00 0x7f81081021e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f8104009910 tx=0x7f8104009c20 comp rx=0 tx=0).stop 2024-04-22T23:40:46.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.657+0000 7f81107c6700 1 -- 172.21.15.67:0/4242840568 shutdown_connections 2024-04-22T23:40:46.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.657+0000 7f81107c6700 1 --2- 172.21.15.67:0/4242840568 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8108102720 0x7f810810ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:46.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.657+0000 7f81107c6700 1 --2- 172.21.15.67:0/4242840568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8108101e00 0x7f81081021e0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:46.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.657+0000 7f81107c6700 1 -- 172.21.15.67:0/4242840568 >> 172.21.15.67:0/4242840568 conn(0x7f81080fb650 msgr2=0x7f81080fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:40:46.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.657+0000 7f81107c6700 1 -- 172.21.15.67:0/4242840568 shutdown_connections 2024-04-22T23:40:46.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.657+0000 7f81107c6700 1 -- 172.21.15.67:0/4242840568 wait complete. 2024-04-22T23:40:46.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.658+0000 7f81107c6700 1 Processor -- start 2024-04-22T23:40:46.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.658+0000 7f81107c6700 1 -- start start 2024-04-22T23:40:46.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.658+0000 7f81107c6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8108101e00 0x7f810819c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:46.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.658+0000 7f81107c6700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8108102720 0x7f810819cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:46.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.658+0000 7f81107c6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810819d260 con 0x7f8108101e00 2024-04-22T23:40:46.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.658+0000 7f81107c6700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8108196710 con 0x7f8108102720 2024-04-22T23:40:46.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.658+0000 7f810e562700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8108101e00 0x7f810819c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:46.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.658+0000 7f810e562700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8108101e00 0x7f810819c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40156/0 (socket says 172.21.15.67:40156) 2024-04-22T23:40:46.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.658+0000 7f810e562700 1 -- 172.21.15.67:0/250311843 learned_addr learned my addr 172.21.15.67:0/250311843 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:40:46.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.658+0000 7f810dd61700 1 --2- 172.21.15.67:0/250311843 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8108102720 0x7f810819cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:46.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.658+0000 7f810e562700 1 -- 172.21.15.67:0/250311843 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8108102720 msgr2=0x7f810819cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:46.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.658+0000 7f810e562700 1 --2- 172.21.15.67:0/250311843 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8108102720 0x7f810819cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:46.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.658+0000 7f810e562700 1 -- 172.21.15.67:0/250311843 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81040095f0 con 0x7f8108101e00 2024-04-22T23:40:46.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.659+0000 7f810e562700 1 --2- 172.21.15.67:0/250311843 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8108101e00 0x7f810819c690 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8104005f50 tx=0x7f810400f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:40:46.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.660+0000 7f80ff7fe700 1 -- 172.21.15.67:0/250311843 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f810401d070 con 0x7f8108101e00 2024-04-22T23:40:46.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.660+0000 7f80ff7fe700 1 -- 172.21.15.67:0/250311843 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f810400b610 con 0x7f8108101e00 2024-04-22T23:40:46.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.660+0000 7f81107c6700 1 -- 172.21.15.67:0/250311843 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8108196990 con 0x7f8108101e00 2024-04-22T23:40:46.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.660+0000 7f80ff7fe700 1 -- 172.21.15.67:0/250311843 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8104022470 con 0x7f8108101e00 2024-04-22T23:40:46.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.660+0000 7f81107c6700 1 -- 172.21.15.67:0/250311843 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8108196e80 con 0x7f8108101e00 2024-04-22T23:40:46.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.661+0000 7f81107c6700 1 -- 172.21.15.67:0/250311843 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8108108310 con 0x7f8108101e00 2024-04-22T23:40:46.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.661+0000 7f80ff7fe700 1 -- 172.21.15.67:0/250311843 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f810400fca0 con 0x7f8108101e00 2024-04-22T23:40:46.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.662+0000 7f80ff7fe700 1 --2- 172.21.15.67:0/250311843 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f80f406ecc0 0x7f80f4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:46.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.662+0000 7f80ff7fe700 1 -- 172.21.15.67:0/250311843 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f8104090440 con 0x7f8108101e00 2024-04-22T23:40:46.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.662+0000 7f810dd61700 1 --2- 172.21.15.67:0/250311843 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f80f406ecc0 0x7f80f4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:46.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.662+0000 7f810dd61700 1 --2- 172.21.15.67:0/250311843 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f80f406ecc0 0x7f80f4071180 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8108197b80 tx=0x7f80f8009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:40:46.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.667+0000 7f80ff7fe700 1 -- 172.21.15.67:0/250311843 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f810405cbb0 con 0x7f8108101e00 2024-04-22T23:40:46.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.884+0000 7f81107c6700 1 -- 172.21.15.67:0/250311843 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f8108197d30 con 0x7f80f406ecc0 2024-04-22T23:40:46.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.896+0000 7f80ff7fe700 1 -- 172.21.15.67:0/250311843 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f8108197d30 con 0x7f80f406ecc0 2024-04-22T23:40:46.889 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:40:46.889 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (14m) 7m ago 25m 25.5M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:40:46.889 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (26m) 7m ago 26m 9114k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:40:46.889 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (24m) 7m ago 24m 9042k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:40:46.889 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (11m) 7m ago 25m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:40:46.889 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (11m) 7m ago 24m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:40:46.890 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (14m) 7m ago 25m 79.3M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:40:46.890 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (8m) 7m ago 20m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:40:46.890 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (7m) 7m ago 20m 12.6M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:40:46.890 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (7m) 7m ago 20m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:40:46.890 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (20m) 7m ago 20m 12.2M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:40:46.890 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (16m) 7m ago 27m 511M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:40:46.890 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (15m) 7m ago 24m 453M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:40:46.890 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (12m) 7m ago 27m 55.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:40:46.890 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (12m) 7m ago 24m 46.0M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:40:46.890 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (15m) 7m ago 25m 15.4M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:40:46.890 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (15m) 7m ago 24m 16.1M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:40:46.890 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (11m) 7m ago 24m 99.2M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:40:46.890 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (10m) 7m ago 23m 127M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:40:46.890 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (10m) 7m ago 23m 79.9M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:40:46.891 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (9m) 7m ago 22m 97.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:40:46.891 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (9m) 7m ago 22m 82.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:40:46.891 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (8m) 7m ago 21m 77.6M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:40:46.891 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (14m) 7m ago 25m 73.9M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:40:46.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.898+0000 7f81107c6700 1 -- 172.21.15.67:0/250311843 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f80f406ecc0 msgr2=0x7f80f4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:46.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.898+0000 7f81107c6700 1 --2- 172.21.15.67:0/250311843 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f80f406ecc0 0x7f80f4071180 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8108197b80 tx=0x7f80f8009260 comp rx=0 tx=0).stop 2024-04-22T23:40:46.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.898+0000 7f81107c6700 1 -- 172.21.15.67:0/250311843 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8108101e00 msgr2=0x7f810819c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:46.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.898+0000 7f81107c6700 1 --2- 172.21.15.67:0/250311843 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8108101e00 0x7f810819c690 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8104005f50 tx=0x7f810400f740 comp rx=0 tx=0).stop 2024-04-22T23:40:46.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.898+0000 7f81107c6700 1 -- 172.21.15.67:0/250311843 shutdown_connections 2024-04-22T23:40:46.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.898+0000 7f81107c6700 1 --2- 172.21.15.67:0/250311843 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f80f406ecc0 0x7f80f4071180 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:46.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.898+0000 7f81107c6700 1 --2- 172.21.15.67:0/250311843 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8108101e00 0x7f810819c690 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:46.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.898+0000 7f81107c6700 1 --2- 172.21.15.67:0/250311843 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8108102720 0x7f810819cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:46.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.898+0000 7f81107c6700 1 -- 172.21.15.67:0/250311843 >> 172.21.15.67:0/250311843 conn(0x7f81080fb650 msgr2=0x7f8108105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:40:46.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.898+0000 7f81107c6700 1 -- 172.21.15.67:0/250311843 shutdown_connections 2024-04-22T23:40:46.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:46.898+0000 7f81107c6700 1 -- 172.21.15.67:0/250311843 wait complete. 2024-04-22T23:40:46.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.004+0000 7fcb9f330700 1 -- 172.21.15.67:0/4014507280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb98102720 msgr2=0x7fcb9810ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:46.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.004+0000 7fcb9f330700 1 --2- 172.21.15.67:0/4014507280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb98102720 0x7fcb9810ac10 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fcb88009910 tx=0x7fcb88009c20 comp rx=0 tx=0).stop 2024-04-22T23:40:46.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.004+0000 7fcb9f330700 1 -- 172.21.15.67:0/4014507280 shutdown_connections 2024-04-22T23:40:46.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.004+0000 7fcb9f330700 1 --2- 172.21.15.67:0/4014507280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb98102720 0x7fcb9810ac10 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:46.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.004+0000 7fcb9f330700 1 --2- 172.21.15.67:0/4014507280 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb98101e00 0x7fcb981021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:46.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.004+0000 7fcb9f330700 1 -- 172.21.15.67:0/4014507280 >> 172.21.15.67:0/4014507280 conn(0x7fcb980fb650 msgr2=0x7fcb980fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:40:46.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.004+0000 7fcb9f330700 1 -- 172.21.15.67:0/4014507280 shutdown_connections 2024-04-22T23:40:46.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.004+0000 7fcb9f330700 1 -- 172.21.15.67:0/4014507280 wait complete. 2024-04-22T23:40:46.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.005+0000 7fcb9f330700 1 Processor -- start 2024-04-22T23:40:46.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.005+0000 7fcb9f330700 1 -- start start 2024-04-22T23:40:46.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.005+0000 7fcb9f330700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb98101e00 0x7fcb980ffa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:46.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.005+0000 7fcb9f330700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb98102720 0x7fcb980fffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:46.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.005+0000 7fcb9f330700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb981004e0 con 0x7fcb98101e00 2024-04-22T23:40:46.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.005+0000 7fcb9f330700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb98100620 con 0x7fcb98102720 2024-04-22T23:40:46.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.005+0000 7fcb9d0cc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb98101e00 0x7fcb980ffa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:46.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.005+0000 7fcb9d0cc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb98101e00 0x7fcb980ffa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40174/0 (socket says 172.21.15.67:40174) 2024-04-22T23:40:46.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.005+0000 7fcb9d0cc700 1 -- 172.21.15.67:0/1008177468 learned_addr learned my addr 172.21.15.67:0/1008177468 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:40:46.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.005+0000 7fcb9c8cb700 1 --2- 172.21.15.67:0/1008177468 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb98102720 0x7fcb980fffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:46.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.005+0000 7fcb9d0cc700 1 -- 172.21.15.67:0/1008177468 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb98102720 msgr2=0x7fcb980fffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:46.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.005+0000 7fcb9d0cc700 1 --2- 172.21.15.67:0/1008177468 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb98102720 0x7fcb980fffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:46.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.005+0000 7fcb9d0cc700 1 -- 172.21.15.67:0/1008177468 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb880095f0 con 0x7fcb98101e00 2024-04-22T23:40:47.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.006+0000 7fcb9d0cc700 1 --2- 172.21.15.67:0/1008177468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb98101e00 0x7fcb980ffa60 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fcb9400c930 tx=0x7fcb9400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:40:47.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.006+0000 7fcb8e7fc700 1 -- 172.21.15.67:0/1008177468 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb94007ad0 con 0x7fcb98101e00 2024-04-22T23:40:47.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.006+0000 7fcb8e7fc700 1 -- 172.21.15.67:0/1008177468 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcb9400ce80 con 0x7fcb98101e00 2024-04-22T23:40:47.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.006+0000 7fcb9f330700 1 -- 172.21.15.67:0/1008177468 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb981a2b20 con 0x7fcb98101e00 2024-04-22T23:40:47.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.006+0000 7fcb8e7fc700 1 -- 172.21.15.67:0/1008177468 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb94018630 con 0x7fcb98101e00 2024-04-22T23:40:47.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.006+0000 7fcb9f330700 1 -- 172.21.15.67:0/1008177468 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb981a2e80 con 0x7fcb98101e00 2024-04-22T23:40:47.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.007+0000 7fcb9f330700 1 -- 172.21.15.67:0/1008177468 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb98108310 con 0x7fcb98101e00 2024-04-22T23:40:47.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.007+0000 7fcb8e7fc700 1 -- 172.21.15.67:0/1008177468 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcb94007c30 con 0x7fcb98101e00 2024-04-22T23:40:47.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.008+0000 7fcb8e7fc700 1 --2- 172.21.15.67:0/1008177468 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcb8406ecc0 0x7fcb84071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:47.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.008+0000 7fcb8e7fc700 1 -- 172.21.15.67:0/1008177468 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fcb9408ef00 con 0x7fcb98101e00 2024-04-22T23:40:47.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.008+0000 7fcb9c8cb700 1 --2- 172.21.15.67:0/1008177468 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcb8406ecc0 0x7fcb84071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:47.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.008+0000 7fcb9c8cb700 1 --2- 172.21.15.67:0/1008177468 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcb8406ecc0 0x7fcb84071180 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcb98101250 tx=0x7fcb8801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:40:47.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.012+0000 7fcb8e7fc700 1 -- 172.21.15.67:0/1008177468 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fcb94057bc0 con 0x7fcb98101e00 2024-04-22T23:40:47.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.313+0000 7fcb9f330700 1 -- 172.21.15.67:0/1008177468 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fcb9804f2e0 con 0x7fcb98101e00 2024-04-22T23:40:47.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.313+0000 7fcb8e7fc700 1 -- 172.21.15.67:0/1008177468 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fcb94060e60 con 0x7fcb98101e00 2024-04-22T23:40:47.306 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:40:47.306 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:40:47.306 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:40:47.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:40:47.306 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:40:47.307 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:40:47.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:40:47.307 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:40:47.307 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:40:47.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:40:47.307 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:40:47.307 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:40:47.307 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 3 2024-04-22T23:40:47.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:40:47.307 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:40:47.307 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:40:47.307 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 13 2024-04-22T23:40:47.307 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:40:47.307 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:40:47.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.316+0000 7fcb9f330700 1 -- 172.21.15.67:0/1008177468 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcb8406ecc0 msgr2=0x7fcb84071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:47.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.316+0000 7fcb9f330700 1 --2- 172.21.15.67:0/1008177468 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcb8406ecc0 0x7fcb84071180 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcb98101250 tx=0x7fcb8801a040 comp rx=0 tx=0).stop 2024-04-22T23:40:47.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.316+0000 7fcb9f330700 1 -- 172.21.15.67:0/1008177468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb98101e00 msgr2=0x7fcb980ffa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:47.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.316+0000 7fcb9f330700 1 --2- 172.21.15.67:0/1008177468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb98101e00 0x7fcb980ffa60 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fcb9400c930 tx=0x7fcb9400cc40 comp rx=0 tx=0).stop 2024-04-22T23:40:47.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.316+0000 7fcb9f330700 1 -- 172.21.15.67:0/1008177468 shutdown_connections 2024-04-22T23:40:47.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.316+0000 7fcb9f330700 1 --2- 172.21.15.67:0/1008177468 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcb8406ecc0 0x7fcb84071180 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:47.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.316+0000 7fcb9f330700 1 --2- 172.21.15.67:0/1008177468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb98101e00 0x7fcb980ffa60 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:47.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.316+0000 7fcb9f330700 1 --2- 172.21.15.67:0/1008177468 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcb98102720 0x7fcb980fffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:47.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.316+0000 7fcb9f330700 1 -- 172.21.15.67:0/1008177468 >> 172.21.15.67:0/1008177468 conn(0x7fcb980fb650 msgr2=0x7fcb98105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:40:47.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.316+0000 7fcb9f330700 1 -- 172.21.15.67:0/1008177468 shutdown_connections 2024-04-22T23:40:47.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.316+0000 7fcb9f330700 1 -- 172.21.15.67:0/1008177468 wait complete. 2024-04-22T23:40:47.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.422+0000 7f744010e700 1 -- 172.21.15.67:0/3731406244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7438100750 msgr2=0x7f74381047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:47.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.422+0000 7f744010e700 1 --2- 172.21.15.67:0/3731406244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7438100750 0x7f74381047a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f7434009910 tx=0x7f7434009c20 comp rx=0 tx=0).stop 2024-04-22T23:40:47.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.422+0000 7f744010e700 1 -- 172.21.15.67:0/3731406244 shutdown_connections 2024-04-22T23:40:47.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.422+0000 7f744010e700 1 --2- 172.21.15.67:0/3731406244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7438100750 0x7f74381047a0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:47.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.422+0000 7f744010e700 1 --2- 172.21.15.67:0/3731406244 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74380ffda0 0x7f7438100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:47.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.422+0000 7f744010e700 1 -- 172.21.15.67:0/3731406244 >> 172.21.15.67:0/3731406244 conn(0x7f74380fb650 msgr2=0x7f74380fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:40:47.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.423+0000 7f744010e700 1 -- 172.21.15.67:0/3731406244 shutdown_connections 2024-04-22T23:40:47.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.423+0000 7f744010e700 1 -- 172.21.15.67:0/3731406244 wait complete. 2024-04-22T23:40:47.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.423+0000 7f744010e700 1 Processor -- start 2024-04-22T23:40:47.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.423+0000 7f744010e700 1 -- start start 2024-04-22T23:40:47.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.423+0000 7f744010e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74380ffda0 0x7f743819c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:47.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.423+0000 7f744010e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7438100750 0x7f743819cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:47.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.423+0000 7f744010e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f743819d240 con 0x7f74380ffda0 2024-04-22T23:40:47.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.423+0000 7f744010e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74381966f0 con 0x7f7438100750 2024-04-22T23:40:47.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.424+0000 7f743deaa700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74380ffda0 0x7f743819c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:47.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.424+0000 7f743deaa700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74380ffda0 0x7f743819c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40190/0 (socket says 172.21.15.67:40190) 2024-04-22T23:40:47.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.424+0000 7f743deaa700 1 -- 172.21.15.67:0/2466371262 learned_addr learned my addr 172.21.15.67:0/2466371262 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:40:47.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.424+0000 7f743d6a9700 1 --2- 172.21.15.67:0/2466371262 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7438100750 0x7f743819cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:47.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.424+0000 7f743deaa700 1 -- 172.21.15.67:0/2466371262 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7438100750 msgr2=0x7f743819cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:47.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.424+0000 7f743deaa700 1 --2- 172.21.15.67:0/2466371262 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7438100750 0x7f743819cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:47.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.424+0000 7f743deaa700 1 -- 172.21.15.67:0/2466371262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74340095f0 con 0x7f74380ffda0 2024-04-22T23:40:47.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.424+0000 7f743deaa700 1 --2- 172.21.15.67:0/2466371262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74380ffda0 0x7f743819c670 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f742c00c8a0 tx=0x7f742c00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:40:47.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.424+0000 7f742affd700 1 -- 172.21.15.67:0/2466371262 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f742c0078e0 con 0x7f74380ffda0 2024-04-22T23:40:47.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.424+0000 7f742affd700 1 -- 172.21.15.67:0/2466371262 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f742c00f450 con 0x7f74380ffda0 2024-04-22T23:40:47.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.424+0000 7f744010e700 1 -- 172.21.15.67:0/2466371262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74381969d0 con 0x7f74380ffda0 2024-04-22T23:40:47.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.424+0000 7f742affd700 1 -- 172.21.15.67:0/2466371262 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f742c00e5e0 con 0x7f74380ffda0 2024-04-22T23:40:47.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.424+0000 7f744010e700 1 -- 172.21.15.67:0/2466371262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7438196ef0 con 0x7f74380ffda0 2024-04-22T23:40:47.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.425+0000 7f744010e700 1 -- 172.21.15.67:0/2466371262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74381091b0 con 0x7f74380ffda0 2024-04-22T23:40:47.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.426+0000 7f742affd700 1 -- 172.21.15.67:0/2466371262 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f742c00f5c0 con 0x7f74380ffda0 2024-04-22T23:40:47.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.426+0000 7f742affd700 1 --2- 172.21.15.67:0/2466371262 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f742406ecc0 0x7f7424071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:47.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.426+0000 7f742affd700 1 -- 172.21.15.67:0/2466371262 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f742c08ee30 con 0x7f74380ffda0 2024-04-22T23:40:47.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.426+0000 7f743d6a9700 1 --2- 172.21.15.67:0/2466371262 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f742406ecc0 0x7f7424071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:47.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.427+0000 7f743d6a9700 1 --2- 172.21.15.67:0/2466371262 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f742406ecc0 0x7f7424071180 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7434006010 tx=0x7f743401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:40:47.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.431+0000 7f742affd700 1 -- 172.21.15.67:0/2466371262 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f742c05b620 con 0x7f74380ffda0 2024-04-22T23:40:47.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.688+0000 7f744010e700 1 -- 172.21.15.67:0/2466371262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f743804f2e0 con 0x7f74380ffda0 2024-04-22T23:40:47.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.688+0000 7f742affd700 1 -- 172.21.15.67:0/2466371262 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1871 (secure 0 0 0) 0x7f742c05b1b0 con 0x7f74380ffda0 2024-04-22T23:40:47.681 INFO:teuthology.orchestra.run.smithi067.stdout:e31 2024-04-22T23:40:47.681 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:40:47.681 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:40:47.682 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:40:47.682 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:40:47.682 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:40:47.682 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:40:47.682 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:40:47.682 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:40:47.682 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:40:47.682 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:40:47.682 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:40:47.682 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:40:47.682 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:40:47.682 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:40:47.682 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:40:47.682 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:40:47.682 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:40:47.682 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:40:47.683 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:40:47.683 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:40:47.683 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:40:47.683 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:40:47.683 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:40:47.683 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:40:47.683 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:40:47.683 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:40:47.683 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:40:47.683 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:40:47.683 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:40:47.683 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:40:47.683 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:40:47.683 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:40:47.683 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:40:47.683 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:40:47.684 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:40:47.684 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:40:47.684 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:40:47.684 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:40:47.684 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:40:47.684 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44259} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1827288036,v1:172.21.15.134:6827/1827288036] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:40:47.684 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:40:47.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.690+0000 7f744010e700 1 -- 172.21.15.67:0/2466371262 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f742406ecc0 msgr2=0x7f7424071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:47.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.690+0000 7f744010e700 1 --2- 172.21.15.67:0/2466371262 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f742406ecc0 0x7f7424071180 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7434006010 tx=0x7f743401a040 comp rx=0 tx=0).stop 2024-04-22T23:40:47.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.690+0000 7f744010e700 1 -- 172.21.15.67:0/2466371262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74380ffda0 msgr2=0x7f743819c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:47.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.690+0000 7f744010e700 1 --2- 172.21.15.67:0/2466371262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74380ffda0 0x7f743819c670 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f742c00c8a0 tx=0x7f742c00cbb0 comp rx=0 tx=0).stop 2024-04-22T23:40:47.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.690+0000 7f744010e700 1 -- 172.21.15.67:0/2466371262 shutdown_connections 2024-04-22T23:40:47.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.690+0000 7f744010e700 1 --2- 172.21.15.67:0/2466371262 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f742406ecc0 0x7f7424071180 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:47.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.690+0000 7f744010e700 1 --2- 172.21.15.67:0/2466371262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74380ffda0 0x7f743819c670 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:47.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.690+0000 7f744010e700 1 --2- 172.21.15.67:0/2466371262 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7438100750 0x7f743819cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:47.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.690+0000 7f744010e700 1 -- 172.21.15.67:0/2466371262 >> 172.21.15.67:0/2466371262 conn(0x7f74380fb650 msgr2=0x7f7438069150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:40:47.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.690+0000 7f744010e700 1 -- 172.21.15.67:0/2466371262 shutdown_connections 2024-04-22T23:40:47.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.690+0000 7f744010e700 1 -- 172.21.15.67:0/2466371262 wait complete. 2024-04-22T23:40:47.685 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 31 2024-04-22T23:40:47.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.798+0000 7fe67a6cc700 1 -- 172.21.15.67:0/1202477548 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe674103a30 msgr2=0x7fe674107a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:47.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.798+0000 7fe67a6cc700 1 --2- 172.21.15.67:0/1202477548 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe674103a30 0x7fe674107a80 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fe668009910 tx=0x7fe668009c20 comp rx=0 tx=0).stop 2024-04-22T23:40:47.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.798+0000 7fe67a6cc700 1 -- 172.21.15.67:0/1202477548 shutdown_connections 2024-04-22T23:40:47.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.798+0000 7fe67a6cc700 1 --2- 172.21.15.67:0/1202477548 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe674103a30 0x7fe674107a80 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:47.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.798+0000 7fe67a6cc700 1 --2- 172.21.15.67:0/1202477548 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe674103080 0x7fe674103460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:47.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.798+0000 7fe67a6cc700 1 -- 172.21.15.67:0/1202477548 >> 172.21.15.67:0/1202477548 conn(0x7fe6740fe910 msgr2=0x7fe674100d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:40:47.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.798+0000 7fe67a6cc700 1 -- 172.21.15.67:0/1202477548 shutdown_connections 2024-04-22T23:40:47.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.798+0000 7fe67a6cc700 1 -- 172.21.15.67:0/1202477548 wait complete. 2024-04-22T23:40:47.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.799+0000 7fe67a6cc700 1 Processor -- start 2024-04-22T23:40:47.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.799+0000 7fe67a6cc700 1 -- start start 2024-04-22T23:40:47.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.799+0000 7fe67a6cc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe674103080 0x7fe67419e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:47.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.799+0000 7fe67a6cc700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe674103a30 0x7fe67419eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:47.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.799+0000 7fe67a6cc700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67419f430 con 0x7fe674103080 2024-04-22T23:40:47.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.799+0000 7fe67a6cc700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6741988e0 con 0x7fe674103a30 2024-04-22T23:40:47.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.799+0000 7fe673fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe674103080 0x7fe67419e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:47.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.799+0000 7fe673fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe674103080 0x7fe67419e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40202/0 (socket says 172.21.15.67:40202) 2024-04-22T23:40:47.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.799+0000 7fe673fff700 1 -- 172.21.15.67:0/2814936432 learned_addr learned my addr 172.21.15.67:0/2814936432 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:40:47.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.799+0000 7fe6737fe700 1 --2- 172.21.15.67:0/2814936432 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe674103a30 0x7fe67419eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:47.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.800+0000 7fe673fff700 1 -- 172.21.15.67:0/2814936432 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe674103a30 msgr2=0x7fe67419eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:47.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.800+0000 7fe673fff700 1 --2- 172.21.15.67:0/2814936432 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe674103a30 0x7fe67419eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:47.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.800+0000 7fe673fff700 1 -- 172.21.15.67:0/2814936432 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6680095f0 con 0x7fe674103080 2024-04-22T23:40:47.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.800+0000 7fe673fff700 1 --2- 172.21.15.67:0/2814936432 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe674103080 0x7fe67419e860 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fe66400d8d0 tx=0x7fe66400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:40:47.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.800+0000 7fe6717fa700 1 -- 172.21.15.67:0/2814936432 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe6640098a0 con 0x7fe674103080 2024-04-22T23:40:47.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.800+0000 7fe6717fa700 1 -- 172.21.15.67:0/2814936432 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe664010460 con 0x7fe674103080 2024-04-22T23:40:47.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.800+0000 7fe67a6cc700 1 -- 172.21.15.67:0/2814936432 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe674198bc0 con 0x7fe674103080 2024-04-22T23:40:47.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.800+0000 7fe6717fa700 1 -- 172.21.15.67:0/2814936432 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe66400f5f0 con 0x7fe674103080 2024-04-22T23:40:47.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.800+0000 7fe67a6cc700 1 -- 172.21.15.67:0/2814936432 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe674199110 con 0x7fe674103080 2024-04-22T23:40:47.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.801+0000 7fe67a6cc700 1 -- 172.21.15.67:0/2814936432 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe67410b480 con 0x7fe674103080 2024-04-22T23:40:47.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.802+0000 7fe6717fa700 1 -- 172.21.15.67:0/2814936432 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe664009a00 con 0x7fe674103080 2024-04-22T23:40:47.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.802+0000 7fe6717fa700 1 --2- 172.21.15.67:0/2814936432 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe65c06ecc0 0x7fe65c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:47.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.802+0000 7fe6717fa700 1 -- 172.21.15.67:0/2814936432 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fe66408ee80 con 0x7fe674103080 2024-04-22T23:40:47.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.802+0000 7fe6737fe700 1 --2- 172.21.15.67:0/2814936432 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe65c06ecc0 0x7fe65c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:47.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.803+0000 7fe6737fe700 1 --2- 172.21.15.67:0/2814936432 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe65c06ecc0 0x7fe65c071180 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe6740ff130 tx=0x7fe66801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:40:47.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:47.807+0000 7fe6717fa700 1 -- 172.21.15.67:0/2814936432 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe66405b5f0 con 0x7fe674103080 2024-04-22T23:40:47.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:47 smithi134 ceph-mon[85630]: pgmap v432: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:47.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:47 smithi134 ceph-mon[85630]: from='client.34820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:40:47.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:47 smithi134 ceph-mon[85630]: from='client.34824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:40:47.987 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:47 smithi134 ceph-mon[85630]: from='client.34828 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:40:47.987 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:47 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1008177468' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:40:48.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.043+0000 7fe67a6cc700 1 -- 172.21.15.67:0/2814936432 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe674199380 con 0x7fe65c06ecc0 2024-04-22T23:40:48.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:47 smithi067 ceph-mon[112438]: pgmap v432: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:48.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:47 smithi067 ceph-mon[112438]: from='client.34820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:40:48.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:47 smithi067 ceph-mon[112438]: from='client.34824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:40:48.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:47 smithi067 ceph-mon[112438]: from='client.34828 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:40:48.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:47 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1008177468' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:40:48.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.045+0000 7fe6717fa700 1 -- 172.21.15.67:0/2814936432 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe674199380 con 0x7fe65c06ecc0 2024-04-22T23:40:48.038 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:40:48.038 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:40:48.038 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:40:48.038 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:40:48.038 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:40:48.038 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:40:48.038 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:40:48.038 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:40:48.038 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:40:48.038 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:40:48.038 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T23:40:48.038 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:40:48.038 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:40:48.039 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:40:48.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.047+0000 7fe67a6cc700 1 -- 172.21.15.67:0/2814936432 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe65c06ecc0 msgr2=0x7fe65c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:48.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.047+0000 7fe67a6cc700 1 --2- 172.21.15.67:0/2814936432 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe65c06ecc0 0x7fe65c071180 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe6740ff130 tx=0x7fe66801a040 comp rx=0 tx=0).stop 2024-04-22T23:40:48.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.047+0000 7fe67a6cc700 1 -- 172.21.15.67:0/2814936432 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe674103080 msgr2=0x7fe67419e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:48.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.047+0000 7fe67a6cc700 1 --2- 172.21.15.67:0/2814936432 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe674103080 0x7fe67419e860 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fe66400d8d0 tx=0x7fe66400dbe0 comp rx=0 tx=0).stop 2024-04-22T23:40:48.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.047+0000 7fe67a6cc700 1 -- 172.21.15.67:0/2814936432 shutdown_connections 2024-04-22T23:40:48.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.047+0000 7fe67a6cc700 1 --2- 172.21.15.67:0/2814936432 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe65c06ecc0 0x7fe65c071180 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:48.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.047+0000 7fe67a6cc700 1 --2- 172.21.15.67:0/2814936432 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe674103080 0x7fe67419e860 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:48.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.047+0000 7fe67a6cc700 1 --2- 172.21.15.67:0/2814936432 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe674103a30 0x7fe67419eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:48.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.047+0000 7fe67a6cc700 1 -- 172.21.15.67:0/2814936432 >> 172.21.15.67:0/2814936432 conn(0x7fe6740fe910 msgr2=0x7fe6740fffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:40:48.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.047+0000 7fe67a6cc700 1 -- 172.21.15.67:0/2814936432 shutdown_connections 2024-04-22T23:40:48.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.047+0000 7fe67a6cc700 1 -- 172.21.15.67:0/2814936432 wait complete. 2024-04-22T23:40:48.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.154+0000 7fc0058b1700 1 -- 172.21.15.67:0/919042418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0000fff50 msgr2=0x7fc000105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:48.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.154+0000 7fc0058b1700 1 --2- 172.21.15.67:0/919042418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0000fff50 0x7fc000105840 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fbff4009910 tx=0x7fbff4009c20 comp rx=0 tx=0).stop 2024-04-22T23:40:48.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.154+0000 7fc0058b1700 1 -- 172.21.15.67:0/919042418 shutdown_connections 2024-04-22T23:40:48.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.154+0000 7fc0058b1700 1 --2- 172.21.15.67:0/919042418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0000fff50 0x7fc000105840 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:48.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.154+0000 7fc0058b1700 1 --2- 172.21.15.67:0/919042418 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc0000ff5a0 0x7fc0000ff980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:48.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.154+0000 7fc0058b1700 1 -- 172.21.15.67:0/919042418 >> 172.21.15.67:0/919042418 conn(0x7fc000074e30 msgr2=0x7fc000075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:40:48.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.155+0000 7fc0058b1700 1 -- 172.21.15.67:0/919042418 shutdown_connections 2024-04-22T23:40:48.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.155+0000 7fc0058b1700 1 -- 172.21.15.67:0/919042418 wait complete. 2024-04-22T23:40:48.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.155+0000 7fc0058b1700 1 Processor -- start 2024-04-22T23:40:48.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.155+0000 7fc0058b1700 1 -- start start 2024-04-22T23:40:48.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.156+0000 7fc0058b1700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc0000ff5a0 0x7fc00019c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:48.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.156+0000 7fc0058b1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0000fff50 0x7fc00019cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:48.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.156+0000 7fc0058b1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc00019d210 con 0x7fc0000fff50 2024-04-22T23:40:48.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.156+0000 7fc0058b1700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0001966c0 con 0x7fc0000ff5a0 2024-04-22T23:40:48.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.156+0000 7fbffe7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0000fff50 0x7fc00019cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:48.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.156+0000 7fbffe7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0000fff50 0x7fc00019cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40220/0 (socket says 172.21.15.67:40220) 2024-04-22T23:40:48.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.156+0000 7fbffe7fc700 1 -- 172.21.15.67:0/2307589622 learned_addr learned my addr 172.21.15.67:0/2307589622 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:40:48.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.156+0000 7fbffeffd700 1 --2- 172.21.15.67:0/2307589622 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc0000ff5a0 0x7fc00019c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:48.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.156+0000 7fbffe7fc700 1 -- 172.21.15.67:0/2307589622 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc0000ff5a0 msgr2=0x7fc00019c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:48.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.156+0000 7fbffe7fc700 1 --2- 172.21.15.67:0/2307589622 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc0000ff5a0 0x7fc00019c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:48.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.156+0000 7fbffe7fc700 1 -- 172.21.15.67:0/2307589622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff40095f0 con 0x7fc0000fff50 2024-04-22T23:40:48.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.156+0000 7fbffe7fc700 1 --2- 172.21.15.67:0/2307589622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0000fff50 0x7fc00019cb80 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fbff40092e0 tx=0x7fbff400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:40:48.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.156+0000 7fc0048af700 1 -- 172.21.15.67:0/2307589622 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbff401d070 con 0x7fc0000fff50 2024-04-22T23:40:48.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.156+0000 7fc0048af700 1 -- 172.21.15.67:0/2307589622 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbff4005dc0 con 0x7fc0000fff50 2024-04-22T23:40:48.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.156+0000 7fc0048af700 1 -- 172.21.15.67:0/2307589622 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbff4017400 con 0x7fc0000fff50 2024-04-22T23:40:48.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.156+0000 7fc0058b1700 1 -- 172.21.15.67:0/2307589622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc000196940 con 0x7fc0000fff50 2024-04-22T23:40:48.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.157+0000 7fc0058b1700 1 -- 172.21.15.67:0/2307589622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc000196e30 con 0x7fc0000fff50 2024-04-22T23:40:48.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.157+0000 7fc0058b1700 1 -- 172.21.15.67:0/2307589622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc000109200 con 0x7fc0000fff50 2024-04-22T23:40:48.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.158+0000 7fc0048af700 1 -- 172.21.15.67:0/2307589622 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbff400f460 con 0x7fc0000fff50 2024-04-22T23:40:48.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.158+0000 7fc0048af700 1 --2- 172.21.15.67:0/2307589622 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbfe806ecc0 0x7fbfe8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:40:48.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.158+0000 7fc0048af700 1 -- 172.21.15.67:0/2307589622 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fbff40904e0 con 0x7fc0000fff50 2024-04-22T23:40:48.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.158+0000 7fbffeffd700 1 --2- 172.21.15.67:0/2307589622 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbfe806ecc0 0x7fbfe8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:40:48.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.159+0000 7fbffeffd700 1 --2- 172.21.15.67:0/2307589622 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbfe806ecc0 0x7fbfe8071180 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbff0005950 tx=0x7fbff00058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:40:48.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.163+0000 7fc0048af700 1 -- 172.21.15.67:0/2307589622 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fbff405cc50 con 0x7fc0000fff50 2024-04-22T23:40:48.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.464+0000 7fc0058b1700 1 -- 172.21.15.67:0/2307589622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc00004f2e0 con 0x7fc0000fff50 2024-04-22T23:40:48.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.464+0000 7fc0048af700 1 -- 172.21.15.67:0/2307589622 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fbff405c7e0 con 0x7fc0000fff50 2024-04-22T23:40:48.457 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:40:48.457 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:40:48.457 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:40:48.458 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:40:48.458 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:40:48.458 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:40:48.458 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:40:48.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.466+0000 7fc0058b1700 1 -- 172.21.15.67:0/2307589622 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbfe806ecc0 msgr2=0x7fbfe8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:48.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.466+0000 7fc0058b1700 1 --2- 172.21.15.67:0/2307589622 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbfe806ecc0 0x7fbfe8071180 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbff0005950 tx=0x7fbff00058e0 comp rx=0 tx=0).stop 2024-04-22T23:40:48.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.466+0000 7fc0058b1700 1 -- 172.21.15.67:0/2307589622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0000fff50 msgr2=0x7fc00019cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:40:48.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.466+0000 7fc0058b1700 1 --2- 172.21.15.67:0/2307589622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0000fff50 0x7fc00019cb80 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fbff40092e0 tx=0x7fbff400be20 comp rx=0 tx=0).stop 2024-04-22T23:40:48.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.466+0000 7fc0058b1700 1 -- 172.21.15.67:0/2307589622 shutdown_connections 2024-04-22T23:40:48.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.466+0000 7fc0058b1700 1 --2- 172.21.15.67:0/2307589622 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbfe806ecc0 0x7fbfe8071180 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:48.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.466+0000 7fc0058b1700 1 --2- 172.21.15.67:0/2307589622 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc0000ff5a0 0x7fc00019c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:48.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.466+0000 7fc0058b1700 1 --2- 172.21.15.67:0/2307589622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0000fff50 0x7fc00019cb80 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:40:48.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.466+0000 7fc0058b1700 1 -- 172.21.15.67:0/2307589622 >> 172.21.15.67:0/2307589622 conn(0x7fc000074e30 msgr2=0x7fc0000fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:40:48.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.466+0000 7fc0058b1700 1 -- 172.21.15.67:0/2307589622 shutdown_connections 2024-04-22T23:40:48.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:40:48.466+0000 7fc0058b1700 1 -- 172.21.15.67:0/2307589622 wait complete. 2024-04-22T23:40:48.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:48 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2466371262' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:40:48.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:48 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2307589622' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:40:48.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:48 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2466371262' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:40:48.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:48 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2307589622' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:40:49.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:49 smithi134 ceph-mon[85630]: from='client.34840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:40:49.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:49 smithi134 ceph-mon[85630]: pgmap v433: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:50.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:49 smithi067 ceph-mon[112438]: from='client.34840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:40:50.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:49 smithi067 ceph-mon[112438]: pgmap v433: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:51.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:51 smithi134 ceph-mon[85630]: pgmap v434: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:52.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:51 smithi067 ceph-mon[112438]: pgmap v434: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:53.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:53 smithi134 ceph-mon[85630]: pgmap v435: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:54.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:53 smithi067 ceph-mon[112438]: pgmap v435: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:55.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:55 smithi134 ceph-mon[85630]: pgmap v436: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:56.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:55 smithi067 ceph-mon[112438]: pgmap v436: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:56.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:40:57.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:40:57.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:57 smithi134 ceph-mon[85630]: pgmap v437: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:58.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:57 smithi067 ceph-mon[112438]: pgmap v437: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:40:59.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:40:59 smithi134 ceph-mon[85630]: pgmap v438: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:00.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:40:59 smithi067 ceph-mon[112438]: pgmap v438: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:01.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:01 smithi134 ceph-mon[85630]: pgmap v439: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:02.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:01 smithi067 ceph-mon[112438]: pgmap v439: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:03.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:03 smithi134 ceph-mon[85630]: pgmap v440: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:04.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:03 smithi067 ceph-mon[112438]: pgmap v440: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:05.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:05 smithi134 ceph-mon[85630]: pgmap v441: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:06.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:05 smithi067 ceph-mon[112438]: pgmap v441: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:07.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:07 smithi134 ceph-mon[85630]: pgmap v442: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:08.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:07 smithi067 ceph-mon[112438]: pgmap v442: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:09.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:09 smithi134 ceph-mon[85630]: pgmap v443: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:10.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:09 smithi067 ceph-mon[112438]: pgmap v443: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:11.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:11 smithi134 ceph-mon[85630]: pgmap v444: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:11.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:41:12.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:11 smithi067 ceph-mon[112438]: pgmap v444: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:12.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:41:13.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:13 smithi134 ceph-mon[85630]: pgmap v445: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:14.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:13 smithi067 ceph-mon[112438]: pgmap v445: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:15.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:15 smithi134 ceph-mon[85630]: pgmap v446: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:16.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:15 smithi067 ceph-mon[112438]: pgmap v446: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:17.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:17 smithi134 ceph-mon[85630]: pgmap v447: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:18.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:17 smithi067 ceph-mon[112438]: pgmap v447: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:18.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.588+0000 7fb380946700 1 -- 172.21.15.67:0/793229991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb37c100750 msgr2=0x7fb37c1047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:18.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.588+0000 7fb380946700 1 --2- 172.21.15.67:0/793229991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb37c100750 0x7fb37c1047a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fb36c009910 tx=0x7fb36c009c20 comp rx=0 tx=0).stop 2024-04-22T23:41:18.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.588+0000 7fb380946700 1 -- 172.21.15.67:0/793229991 shutdown_connections 2024-04-22T23:41:18.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.588+0000 7fb380946700 1 --2- 172.21.15.67:0/793229991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb37c100750 0x7fb37c1047a0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:18.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.588+0000 7fb380946700 1 --2- 172.21.15.67:0/793229991 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb37c0ffda0 0x7fb37c100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:18.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.588+0000 7fb380946700 1 -- 172.21.15.67:0/793229991 >> 172.21.15.67:0/793229991 conn(0x7fb37c0fb650 msgr2=0x7fb37c0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:41:18.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.589+0000 7fb380946700 1 -- 172.21.15.67:0/793229991 shutdown_connections 2024-04-22T23:41:18.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.589+0000 7fb380946700 1 -- 172.21.15.67:0/793229991 wait complete. 2024-04-22T23:41:18.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.589+0000 7fb380946700 1 Processor -- start 2024-04-22T23:41:18.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.589+0000 7fb380946700 1 -- start start 2024-04-22T23:41:18.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.590+0000 7fb380946700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb37c0ffda0 0x7fb37c196950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:18.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.590+0000 7fb380946700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb37c100750 0x7fb37c196e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:18.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.590+0000 7fb380946700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb37c197520 con 0x7fb37c100750 2024-04-22T23:41:18.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.590+0000 7fb380946700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb37c19b330 con 0x7fb37c0ffda0 2024-04-22T23:41:18.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.590+0000 7fb379d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb37c100750 0x7fb37c196e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:18.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.590+0000 7fb379d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb37c100750 0x7fb37c196e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50690/0 (socket says 172.21.15.67:50690) 2024-04-22T23:41:18.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.590+0000 7fb379d9b700 1 -- 172.21.15.67:0/2986476253 learned_addr learned my addr 172.21.15.67:0/2986476253 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:41:18.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.590+0000 7fb37a59c700 1 --2- 172.21.15.67:0/2986476253 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb37c0ffda0 0x7fb37c196950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:18.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.590+0000 7fb379d9b700 1 -- 172.21.15.67:0/2986476253 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb37c0ffda0 msgr2=0x7fb37c196950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:18.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.590+0000 7fb379d9b700 1 --2- 172.21.15.67:0/2986476253 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb37c0ffda0 0x7fb37c196950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:18.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.590+0000 7fb379d9b700 1 -- 172.21.15.67:0/2986476253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb36c0095f0 con 0x7fb37c100750 2024-04-22T23:41:18.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.590+0000 7fb37a59c700 1 --2- 172.21.15.67:0/2986476253 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb37c0ffda0 0x7fb37c196950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:41:18.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.590+0000 7fb379d9b700 1 --2- 172.21.15.67:0/2986476253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb37c100750 0x7fb37c196e90 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fb36c00bd70 tx=0x7fb36c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:41:18.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.591+0000 7fb3737fe700 1 -- 172.21.15.67:0/2986476253 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb36c01d070 con 0x7fb37c100750 2024-04-22T23:41:18.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.591+0000 7fb380946700 1 -- 172.21.15.67:0/2986476253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb37c19b5b0 con 0x7fb37c100750 2024-04-22T23:41:18.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.591+0000 7fb3737fe700 1 -- 172.21.15.67:0/2986476253 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb36c005dc0 con 0x7fb37c100750 2024-04-22T23:41:18.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.591+0000 7fb3737fe700 1 -- 172.21.15.67:0/2986476253 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb36c017400 con 0x7fb37c100750 2024-04-22T23:41:18.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.591+0000 7fb380946700 1 -- 172.21.15.67:0/2986476253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb37c19baa0 con 0x7fb37c100750 2024-04-22T23:41:18.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.592+0000 7fb380946700 1 -- 172.21.15.67:0/2986476253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb37c04f2e0 con 0x7fb37c100750 2024-04-22T23:41:18.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.592+0000 7fb3737fe700 1 -- 172.21.15.67:0/2986476253 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb36c00f7f0 con 0x7fb37c100750 2024-04-22T23:41:18.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.593+0000 7fb3737fe700 1 --2- 172.21.15.67:0/2986476253 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb36806ecc0 0x7fb368071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:18.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.593+0000 7fb3737fe700 1 -- 172.21.15.67:0/2986476253 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fb36c0904e0 con 0x7fb37c100750 2024-04-22T23:41:18.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.593+0000 7fb37a59c700 1 --2- 172.21.15.67:0/2986476253 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb36806ecc0 0x7fb368071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:18.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.593+0000 7fb37a59c700 1 --2- 172.21.15.67:0/2986476253 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb36806ecc0 0x7fb368071180 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb364005fd0 tx=0x7fb364005ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:41:18.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.598+0000 7fb3737fe700 1 -- 172.21.15.67:0/2986476253 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb36c05cc50 con 0x7fb37c100750 2024-04-22T23:41:18.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.826+0000 7fb380946700 1 -- 172.21.15.67:0/2986476253 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb37c19bcc0 con 0x7fb36806ecc0 2024-04-22T23:41:18.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.828+0000 7fb3737fe700 1 -- 172.21.15.67:0/2986476253 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb37c19bcc0 con 0x7fb36806ecc0 2024-04-22T23:41:18.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.830+0000 7fb380946700 1 -- 172.21.15.67:0/2986476253 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb36806ecc0 msgr2=0x7fb368071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:18.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.830+0000 7fb380946700 1 --2- 172.21.15.67:0/2986476253 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb36806ecc0 0x7fb368071180 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb364005fd0 tx=0x7fb364005ee0 comp rx=0 tx=0).stop 2024-04-22T23:41:18.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.830+0000 7fb380946700 1 -- 172.21.15.67:0/2986476253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb37c100750 msgr2=0x7fb37c196e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:18.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.830+0000 7fb380946700 1 --2- 172.21.15.67:0/2986476253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb37c100750 0x7fb37c196e90 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fb36c00bd70 tx=0x7fb36c00be50 comp rx=0 tx=0).stop 2024-04-22T23:41:18.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.830+0000 7fb380946700 1 -- 172.21.15.67:0/2986476253 shutdown_connections 2024-04-22T23:41:18.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.830+0000 7fb380946700 1 --2- 172.21.15.67:0/2986476253 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb36806ecc0 0x7fb368071180 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:18.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.830+0000 7fb380946700 1 --2- 172.21.15.67:0/2986476253 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb37c0ffda0 0x7fb37c196950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:18.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.830+0000 7fb380946700 1 --2- 172.21.15.67:0/2986476253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb37c100750 0x7fb37c196e90 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:18.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.830+0000 7fb380946700 1 -- 172.21.15.67:0/2986476253 >> 172.21.15.67:0/2986476253 conn(0x7fb37c0fb650 msgr2=0x7fb37c103fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:41:18.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.830+0000 7fb380946700 1 -- 172.21.15.67:0/2986476253 shutdown_connections 2024-04-22T23:41:18.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.830+0000 7fb380946700 1 -- 172.21.15.67:0/2986476253 wait complete. 2024-04-22T23:41:18.833 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:41:18.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.938+0000 7fe1fec45700 1 -- 172.21.15.67:0/6016882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1f8102780 msgr2=0x7fe1f810ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:18.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.938+0000 7fe1fec45700 1 --2- 172.21.15.67:0/6016882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1f8102780 0x7fe1f810ac70 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fe1e4009910 tx=0x7fe1e4009c20 comp rx=0 tx=0).stop 2024-04-22T23:41:18.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.939+0000 7fe1fec45700 1 -- 172.21.15.67:0/6016882 shutdown_connections 2024-04-22T23:41:18.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.939+0000 7fe1fec45700 1 --2- 172.21.15.67:0/6016882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1f8102780 0x7fe1f810ac70 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:18.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.939+0000 7fe1fec45700 1 --2- 172.21.15.67:0/6016882 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1f8101e60 0x7fe1f8102240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:18.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.939+0000 7fe1fec45700 1 -- 172.21.15.67:0/6016882 >> 172.21.15.67:0/6016882 conn(0x7fe1f80fb650 msgr2=0x7fe1f80fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:41:18.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.939+0000 7fe1fec45700 1 -- 172.21.15.67:0/6016882 shutdown_connections 2024-04-22T23:41:18.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.939+0000 7fe1fec45700 1 -- 172.21.15.67:0/6016882 wait complete. 2024-04-22T23:41:18.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.939+0000 7fe1fec45700 1 Processor -- start 2024-04-22T23:41:18.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.939+0000 7fe1fec45700 1 -- start start 2024-04-22T23:41:18.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.939+0000 7fe1fec45700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1f8101e60 0x7fe1f819c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:18.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.939+0000 7fe1fec45700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1f8102780 0x7fe1f819cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:18.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.939+0000 7fe1fec45700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f819d210 con 0x7fe1f8102780 2024-04-22T23:41:18.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.939+0000 7fe1fec45700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f81966c0 con 0x7fe1f8101e60 2024-04-22T23:41:18.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.940+0000 7fe1f7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1f8102780 0x7fe1f819cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:18.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.940+0000 7fe1f7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1f8102780 0x7fe1f819cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47064/0 (socket says 172.21.15.67:47064) 2024-04-22T23:41:18.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.940+0000 7fe1f7fff700 1 -- 172.21.15.67:0/3234467455 learned_addr learned my addr 172.21.15.67:0/3234467455 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:41:18.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.940+0000 7fe1fc9e1700 1 --2- 172.21.15.67:0/3234467455 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1f8101e60 0x7fe1f819c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:18.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.940+0000 7fe1f7fff700 1 -- 172.21.15.67:0/3234467455 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1f8101e60 msgr2=0x7fe1f819c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:18.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.940+0000 7fe1f7fff700 1 --2- 172.21.15.67:0/3234467455 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1f8101e60 0x7fe1f819c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:18.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.940+0000 7fe1f7fff700 1 -- 172.21.15.67:0/3234467455 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e40095f0 con 0x7fe1f8102780 2024-04-22T23:41:18.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.940+0000 7fe1f7fff700 1 --2- 172.21.15.67:0/3234467455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1f8102780 0x7fe1f819cb80 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fe1e400bd70 tx=0x7fe1e400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:41:18.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.940+0000 7fe1f5ffb700 1 -- 172.21.15.67:0/3234467455 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1e401d070 con 0x7fe1f8102780 2024-04-22T23:41:18.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.940+0000 7fe1fec45700 1 -- 172.21.15.67:0/3234467455 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1f8196940 con 0x7fe1f8102780 2024-04-22T23:41:18.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.940+0000 7fe1f5ffb700 1 -- 172.21.15.67:0/3234467455 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe1e4005dc0 con 0x7fe1f8102780 2024-04-22T23:41:18.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.940+0000 7fe1f5ffb700 1 -- 172.21.15.67:0/3234467455 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1e4017400 con 0x7fe1f8102780 2024-04-22T23:41:18.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.940+0000 7fe1fec45700 1 -- 172.21.15.67:0/3234467455 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1f8196e30 con 0x7fe1f8102780 2024-04-22T23:41:18.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.942+0000 7fe1fec45700 1 -- 172.21.15.67:0/3234467455 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1f8108300 con 0x7fe1f8102780 2024-04-22T23:41:18.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.942+0000 7fe1f5ffb700 1 -- 172.21.15.67:0/3234467455 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe1e400f7f0 con 0x7fe1f8102780 2024-04-22T23:41:18.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.942+0000 7fe1f5ffb700 1 --2- 172.21.15.67:0/3234467455 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe1e806ecc0 0x7fe1e8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:18.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.942+0000 7fe1f5ffb700 1 -- 172.21.15.67:0/3234467455 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fe1e40904e0 con 0x7fe1f8102780 2024-04-22T23:41:18.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.943+0000 7fe1fc9e1700 1 --2- 172.21.15.67:0/3234467455 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe1e806ecc0 0x7fe1e8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:18.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.943+0000 7fe1fc9e1700 1 --2- 172.21.15.67:0/3234467455 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe1e806ecc0 0x7fe1e8071180 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec005950 tx=0x7fe1ec0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:41:18.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:18.947+0000 7fe1f5ffb700 1 -- 172.21.15.67:0/3234467455 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe1e405cc50 con 0x7fe1f8102780 2024-04-22T23:41:19.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.180+0000 7fe1fec45700 1 -- 172.21.15.67:0/3234467455 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe1f8197050 con 0x7fe1e806ecc0 2024-04-22T23:41:19.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.182+0000 7fe1f5ffb700 1 -- 172.21.15.67:0/3234467455 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe1f8197050 con 0x7fe1e806ecc0 2024-04-22T23:41:19.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.184+0000 7fe1fec45700 1 -- 172.21.15.67:0/3234467455 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe1e806ecc0 msgr2=0x7fe1e8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:19.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.184+0000 7fe1fec45700 1 --2- 172.21.15.67:0/3234467455 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe1e806ecc0 0x7fe1e8071180 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec005950 tx=0x7fe1ec0058e0 comp rx=0 tx=0).stop 2024-04-22T23:41:19.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.184+0000 7fe1fec45700 1 -- 172.21.15.67:0/3234467455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1f8102780 msgr2=0x7fe1f819cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:19.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.184+0000 7fe1fec45700 1 --2- 172.21.15.67:0/3234467455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1f8102780 0x7fe1f819cb80 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fe1e400bd70 tx=0x7fe1e400be50 comp rx=0 tx=0).stop 2024-04-22T23:41:19.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.184+0000 7fe1fec45700 1 -- 172.21.15.67:0/3234467455 shutdown_connections 2024-04-22T23:41:19.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.184+0000 7fe1fec45700 1 --2- 172.21.15.67:0/3234467455 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe1e806ecc0 0x7fe1e8071180 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:19.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.184+0000 7fe1fec45700 1 --2- 172.21.15.67:0/3234467455 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1f8101e60 0x7fe1f819c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:19.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.184+0000 7fe1fec45700 1 --2- 172.21.15.67:0/3234467455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1f8102780 0x7fe1f819cb80 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:19.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.184+0000 7fe1fec45700 1 -- 172.21.15.67:0/3234467455 >> 172.21.15.67:0/3234467455 conn(0x7fe1f80fb650 msgr2=0x7fe1f8105440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:41:19.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.184+0000 7fe1fec45700 1 -- 172.21.15.67:0/3234467455 shutdown_connections 2024-04-22T23:41:19.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.184+0000 7fe1fec45700 1 -- 172.21.15.67:0/3234467455 wait complete. 2024-04-22T23:41:19.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.291+0000 7fc1b32e4700 1 -- 172.21.15.67:0/2985324718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1ac101850 msgr2=0x7fc1ac1058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:19.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.291+0000 7fc1b32e4700 1 --2- 172.21.15.67:0/2985324718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1ac101850 0x7fc1ac1058a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc1a0009910 tx=0x7fc1a0009c20 comp rx=0 tx=0).stop 2024-04-22T23:41:19.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.291+0000 7fc1b32e4700 1 -- 172.21.15.67:0/2985324718 shutdown_connections 2024-04-22T23:41:19.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.291+0000 7fc1b32e4700 1 --2- 172.21.15.67:0/2985324718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1ac101850 0x7fc1ac1058a0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:19.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.291+0000 7fc1b32e4700 1 --2- 172.21.15.67:0/2985324718 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1ac100ea0 0x7fc1ac101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:19.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.291+0000 7fc1b32e4700 1 -- 172.21.15.67:0/2985324718 >> 172.21.15.67:0/2985324718 conn(0x7fc1ac0fc750 msgr2=0x7fc1ac0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:41:19.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.291+0000 7fc1b32e4700 1 -- 172.21.15.67:0/2985324718 shutdown_connections 2024-04-22T23:41:19.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.291+0000 7fc1b32e4700 1 -- 172.21.15.67:0/2985324718 wait complete. 2024-04-22T23:41:19.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.292+0000 7fc1b32e4700 1 Processor -- start 2024-04-22T23:41:19.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.292+0000 7fc1b32e4700 1 -- start start 2024-04-22T23:41:19.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.292+0000 7fc1b32e4700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1ac100ea0 0x7fc1ac073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:19.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.292+0000 7fc1b32e4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1ac101850 0x7fc1ac0735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:19.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.292+0000 7fc1b32e4700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ac073b10 con 0x7fc1ac101850 2024-04-22T23:41:19.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.292+0000 7fc1b32e4700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ac073c50 con 0x7fc1ac100ea0 2024-04-22T23:41:19.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.292+0000 7fc1b087f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1ac101850 0x7fc1ac0735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:19.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.292+0000 7fc1b087f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1ac101850 0x7fc1ac0735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47088/0 (socket says 172.21.15.67:47088) 2024-04-22T23:41:19.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.292+0000 7fc1b087f700 1 -- 172.21.15.67:0/4010543358 learned_addr learned my addr 172.21.15.67:0/4010543358 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:41:19.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.292+0000 7fc1b1080700 1 --2- 172.21.15.67:0/4010543358 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1ac100ea0 0x7fc1ac073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:19.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.292+0000 7fc1b087f700 1 -- 172.21.15.67:0/4010543358 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1ac100ea0 msgr2=0x7fc1ac073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:19.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.292+0000 7fc1b087f700 1 --2- 172.21.15.67:0/4010543358 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1ac100ea0 0x7fc1ac073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:19.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.292+0000 7fc1b087f700 1 -- 172.21.15.67:0/4010543358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a00095f0 con 0x7fc1ac101850 2024-04-22T23:41:19.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.292+0000 7fc1b1080700 1 --2- 172.21.15.67:0/4010543358 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1ac100ea0 0x7fc1ac073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:41:19.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.293+0000 7fc1b087f700 1 --2- 172.21.15.67:0/4010543358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1ac101850 0x7fc1ac0735d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc1a00092e0 tx=0x7fc1a000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:41:19.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.293+0000 7fc1aa7fc700 1 -- 172.21.15.67:0/4010543358 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1a001d070 con 0x7fc1ac101850 2024-04-22T23:41:19.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.293+0000 7fc1aa7fc700 1 -- 172.21.15.67:0/4010543358 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc1a0005dc0 con 0x7fc1ac101850 2024-04-22T23:41:19.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.293+0000 7fc1aa7fc700 1 -- 172.21.15.67:0/4010543358 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1a0017400 con 0x7fc1ac101850 2024-04-22T23:41:19.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.293+0000 7fc1b32e4700 1 -- 172.21.15.67:0/4010543358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1ac073ed0 con 0x7fc1ac101850 2024-04-22T23:41:19.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.293+0000 7fc1b32e4700 1 -- 172.21.15.67:0/4010543358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1ac1a2d70 con 0x7fc1ac101850 2024-04-22T23:41:19.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.294+0000 7fc1b32e4700 1 -- 172.21.15.67:0/4010543358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1ac1092a0 con 0x7fc1ac101850 2024-04-22T23:41:19.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.294+0000 7fc1aa7fc700 1 -- 172.21.15.67:0/4010543358 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc1a000f460 con 0x7fc1ac101850 2024-04-22T23:41:19.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.295+0000 7fc1aa7fc700 1 --2- 172.21.15.67:0/4010543358 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc19406ecc0 0x7fc194071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:19.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.295+0000 7fc1aa7fc700 1 -- 172.21.15.67:0/4010543358 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fc1a0090570 con 0x7fc1ac101850 2024-04-22T23:41:19.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.295+0000 7fc1b1080700 1 --2- 172.21.15.67:0/4010543358 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc19406ecc0 0x7fc194071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:19.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.295+0000 7fc1b1080700 1 --2- 172.21.15.67:0/4010543358 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc19406ecc0 0x7fc194071180 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc19c0094f0 tx=0x7fc19c009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:41:19.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.300+0000 7fc1aa7fc700 1 -- 172.21.15.67:0/4010543358 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc1a0059110 con 0x7fc1ac101850 2024-04-22T23:41:19.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.516+0000 7fc1b32e4700 1 -- 172.21.15.67:0/4010543358 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc1ac0611d0 con 0x7fc19406ecc0 2024-04-22T23:41:19.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.530+0000 7fc1aa7fc700 1 -- 172.21.15.67:0/4010543358 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fc1ac0611d0 con 0x7fc19406ecc0 2024-04-22T23:41:19.523 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:41:19.523 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (15m) 8m ago 26m 25.5M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:41:19.523 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (26m) 8m ago 26m 9114k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:41:19.523 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (25m) 8m ago 25m 9042k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:41:19.523 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (12m) 8m ago 26m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:41:19.523 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (12m) 8m ago 25m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:41:19.523 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (14m) 8m ago 26m 79.3M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:41:19.523 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (8m) 8m ago 20m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:41:19.523 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (8m) 8m ago 20m 12.6M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:41:19.523 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (8m) 8m ago 20m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:41:19.523 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (20m) 8m ago 20m 12.2M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:41:19.524 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (16m) 8m ago 27m 511M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:41:19.524 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (15m) 8m ago 25m 453M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:41:19.524 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (13m) 8m ago 28m 55.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:41:19.524 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (13m) 8m ago 25m 46.0M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:41:19.524 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (15m) 8m ago 26m 15.4M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:41:19.524 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (15m) 8m ago 25m 16.1M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:41:19.524 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (11m) 8m ago 24m 99.2M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:41:19.524 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (11m) 8m ago 24m 127M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:41:19.524 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (10m) 8m ago 23m 79.9M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:41:19.524 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (10m) 8m ago 23m 97.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:41:19.525 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (9m) 8m ago 22m 82.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:41:19.525 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (9m) 8m ago 22m 77.6M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:41:19.525 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (15m) 8m ago 25m 73.9M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:41:19.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.532+0000 7fc1b32e4700 1 -- 172.21.15.67:0/4010543358 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc19406ecc0 msgr2=0x7fc194071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:19.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.532+0000 7fc1b32e4700 1 --2- 172.21.15.67:0/4010543358 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc19406ecc0 0x7fc194071180 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc19c0094f0 tx=0x7fc19c009040 comp rx=0 tx=0).stop 2024-04-22T23:41:19.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.532+0000 7fc1b32e4700 1 -- 172.21.15.67:0/4010543358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1ac101850 msgr2=0x7fc1ac0735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:19.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.532+0000 7fc1b32e4700 1 --2- 172.21.15.67:0/4010543358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1ac101850 0x7fc1ac0735d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc1a00092e0 tx=0x7fc1a000be20 comp rx=0 tx=0).stop 2024-04-22T23:41:19.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.532+0000 7fc1b32e4700 1 -- 172.21.15.67:0/4010543358 shutdown_connections 2024-04-22T23:41:19.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.532+0000 7fc1b32e4700 1 --2- 172.21.15.67:0/4010543358 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc19406ecc0 0x7fc194071180 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:19.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.532+0000 7fc1b32e4700 1 --2- 172.21.15.67:0/4010543358 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1ac100ea0 0x7fc1ac073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:19.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.532+0000 7fc1b32e4700 1 --2- 172.21.15.67:0/4010543358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1ac101850 0x7fc1ac0735d0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:19.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.532+0000 7fc1b32e4700 1 -- 172.21.15.67:0/4010543358 >> 172.21.15.67:0/4010543358 conn(0x7fc1ac0fc750 msgr2=0x7fc1ac0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:41:19.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.532+0000 7fc1b32e4700 1 -- 172.21.15.67:0/4010543358 shutdown_connections 2024-04-22T23:41:19.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.532+0000 7fc1b32e4700 1 -- 172.21.15.67:0/4010543358 wait complete. 2024-04-22T23:41:19.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.638+0000 7efca13a6700 1 -- 172.21.15.67:0/4234250038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc9c0ffda0 msgr2=0x7efc9c100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:19.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.638+0000 7efca13a6700 1 --2- 172.21.15.67:0/4234250038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc9c0ffda0 0x7efc9c100180 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7efc8c00b0a0 tx=0x7efc8c00b3b0 comp rx=0 tx=0).stop 2024-04-22T23:41:19.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.638+0000 7efca13a6700 1 -- 172.21.15.67:0/4234250038 shutdown_connections 2024-04-22T23:41:19.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.638+0000 7efca13a6700 1 --2- 172.21.15.67:0/4234250038 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc9c100750 0x7efc9c1047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:19.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.638+0000 7efca13a6700 1 --2- 172.21.15.67:0/4234250038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc9c0ffda0 0x7efc9c100180 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:19.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.638+0000 7efca13a6700 1 -- 172.21.15.67:0/4234250038 >> 172.21.15.67:0/4234250038 conn(0x7efc9c0fb650 msgr2=0x7efc9c0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:41:19.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.638+0000 7efca13a6700 1 -- 172.21.15.67:0/4234250038 shutdown_connections 2024-04-22T23:41:19.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.638+0000 7efca13a6700 1 -- 172.21.15.67:0/4234250038 wait complete. 2024-04-22T23:41:19.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.639+0000 7efca13a6700 1 Processor -- start 2024-04-22T23:41:19.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.639+0000 7efca13a6700 1 -- start start 2024-04-22T23:41:19.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.639+0000 7efca13a6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc9c0ffda0 0x7efc9c197800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:19.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.639+0000 7efca13a6700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc9c100750 0x7efc9c197d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:19.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.639+0000 7efca13a6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c198360 con 0x7efc9c0ffda0 2024-04-22T23:41:19.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.639+0000 7efca13a6700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c1984d0 con 0x7efc9c100750 2024-04-22T23:41:19.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.639+0000 7efc9affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc9c0ffda0 0x7efc9c197800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:19.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.639+0000 7efc9affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc9c0ffda0 0x7efc9c197800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47102/0 (socket says 172.21.15.67:47102) 2024-04-22T23:41:19.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.639+0000 7efc9affd700 1 -- 172.21.15.67:0/1944896788 learned_addr learned my addr 172.21.15.67:0/1944896788 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:41:19.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.640+0000 7efc9a7fc700 1 --2- 172.21.15.67:0/1944896788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc9c100750 0x7efc9c197d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:19.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.640+0000 7efc9affd700 1 -- 172.21.15.67:0/1944896788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc9c100750 msgr2=0x7efc9c197d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:19.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.640+0000 7efc9affd700 1 --2- 172.21.15.67:0/1944896788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc9c100750 0x7efc9c197d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:19.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.640+0000 7efc9affd700 1 -- 172.21.15.67:0/1944896788 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc8c009d00 con 0x7efc9c0ffda0 2024-04-22T23:41:19.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.640+0000 7efc9affd700 1 --2- 172.21.15.67:0/1944896788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc9c0ffda0 0x7efc9c197800 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7efc8c000f80 tx=0x7efc8c00cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:41:19.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.640+0000 7efc93fff700 1 -- 172.21.15.67:0/1944896788 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc8c00e040 con 0x7efc9c0ffda0 2024-04-22T23:41:19.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.640+0000 7efc93fff700 1 -- 172.21.15.67:0/1944896788 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efc8c003e60 con 0x7efc9c0ffda0 2024-04-22T23:41:19.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.640+0000 7efca13a6700 1 -- 172.21.15.67:0/1944896788 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc9c1a0870 con 0x7efc9c0ffda0 2024-04-22T23:41:19.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.640+0000 7efc93fff700 1 -- 172.21.15.67:0/1944896788 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc8c01d910 con 0x7efc9c0ffda0 2024-04-22T23:41:19.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.640+0000 7efca13a6700 1 -- 172.21.15.67:0/1944896788 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc9c1a0ce0 con 0x7efc9c0ffda0 2024-04-22T23:41:19.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.641+0000 7efca13a6700 1 -- 172.21.15.67:0/1944896788 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc9c108140 con 0x7efc9c0ffda0 2024-04-22T23:41:19.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.642+0000 7efc93fff700 1 -- 172.21.15.67:0/1944896788 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7efc8c004820 con 0x7efc9c0ffda0 2024-04-22T23:41:19.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.642+0000 7efc93fff700 1 --2- 172.21.15.67:0/1944896788 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efc8806ecc0 0x7efc88071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:19.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.642+0000 7efc93fff700 1 -- 172.21.15.67:0/1944896788 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7efc8c0907c0 con 0x7efc9c0ffda0 2024-04-22T23:41:19.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.642+0000 7efc9a7fc700 1 --2- 172.21.15.67:0/1944896788 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efc8806ecc0 0x7efc88071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:19.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.643+0000 7efc9a7fc700 1 --2- 172.21.15.67:0/1944896788 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efc8806ecc0 0x7efc88071180 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7efc84006fd0 tx=0x7efc84008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:41:19.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.647+0000 7efc93fff700 1 -- 172.21.15.67:0/1944896788 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7efc8c059480 con 0x7efc9c0ffda0 2024-04-22T23:41:19.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:19 smithi067 ceph-mon[112438]: pgmap v448: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:19.787 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:19 smithi067 ceph-mon[112438]: from='client.34848 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:41:19.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.944+0000 7efca13a6700 1 -- 172.21.15.67:0/1944896788 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7efc9c107b20 con 0x7efc9c0ffda0 2024-04-22T23:41:19.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.945+0000 7efc93fff700 1 -- 172.21.15.67:0/1944896788 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7efc8c05cb40 con 0x7efc9c0ffda0 2024-04-22T23:41:19.938 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:41:19.938 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:41:19.938 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:41:19.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:41:19.938 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:41:19.938 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:41:19.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:41:19.938 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:41:19.938 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:41:19.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:41:19.939 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:41:19.939 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:41:19.939 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 3 2024-04-22T23:41:19.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:41:19.939 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:41:19.939 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:41:19.939 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 13 2024-04-22T23:41:19.939 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:41:19.939 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:41:19.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.947+0000 7efca13a6700 1 -- 172.21.15.67:0/1944896788 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efc8806ecc0 msgr2=0x7efc88071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:19.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.947+0000 7efca13a6700 1 --2- 172.21.15.67:0/1944896788 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efc8806ecc0 0x7efc88071180 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7efc84006fd0 tx=0x7efc84008040 comp rx=0 tx=0).stop 2024-04-22T23:41:19.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.947+0000 7efca13a6700 1 -- 172.21.15.67:0/1944896788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc9c0ffda0 msgr2=0x7efc9c197800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:19.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.947+0000 7efca13a6700 1 --2- 172.21.15.67:0/1944896788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc9c0ffda0 0x7efc9c197800 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7efc8c000f80 tx=0x7efc8c00cdd0 comp rx=0 tx=0).stop 2024-04-22T23:41:19.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.947+0000 7efca13a6700 1 -- 172.21.15.67:0/1944896788 shutdown_connections 2024-04-22T23:41:19.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.947+0000 7efca13a6700 1 --2- 172.21.15.67:0/1944896788 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efc8806ecc0 0x7efc88071180 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:19.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.947+0000 7efca13a6700 1 --2- 172.21.15.67:0/1944896788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc9c0ffda0 0x7efc9c197800 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:19.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.947+0000 7efca13a6700 1 --2- 172.21.15.67:0/1944896788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc9c100750 0x7efc9c197d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:19.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.947+0000 7efca13a6700 1 -- 172.21.15.67:0/1944896788 >> 172.21.15.67:0/1944896788 conn(0x7efc9c0fb650 msgr2=0x7efc9c0fcdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:41:19.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.947+0000 7efca13a6700 1 -- 172.21.15.67:0/1944896788 shutdown_connections 2024-04-22T23:41:19.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:19.947+0000 7efca13a6700 1 -- 172.21.15.67:0/1944896788 wait complete. 2024-04-22T23:41:19.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:19 smithi134 ceph-mon[85630]: pgmap v448: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:19.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:19 smithi134 ceph-mon[85630]: from='client.34848 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:41:20.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.054+0000 7f1775245700 1 -- 172.21.15.67:0/417895970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1770105bd0 msgr2=0x7f1770105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:20.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.054+0000 7f1775245700 1 --2- 172.21.15.67:0/417895970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1770105bd0 0x7f1770105fb0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f1760009910 tx=0x7f1760009c20 comp rx=0 tx=0).stop 2024-04-22T23:41:20.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.054+0000 7f1775245700 1 -- 172.21.15.67:0/417895970 shutdown_connections 2024-04-22T23:41:20.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.054+0000 7f1775245700 1 --2- 172.21.15.67:0/417895970 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17700ffc20 0x7f1770100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:20.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.054+0000 7f1775245700 1 --2- 172.21.15.67:0/417895970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1770105bd0 0x7f1770105fb0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:20.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.054+0000 7f1775245700 1 -- 172.21.15.67:0/417895970 >> 172.21.15.67:0/417895970 conn(0x7f1770075670 msgr2=0x7f1770075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:41:20.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.054+0000 7f1775245700 1 -- 172.21.15.67:0/417895970 shutdown_connections 2024-04-22T23:41:20.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.054+0000 7f1775245700 1 -- 172.21.15.67:0/417895970 wait complete. 2024-04-22T23:41:20.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.055+0000 7f1775245700 1 Processor -- start 2024-04-22T23:41:20.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.055+0000 7f1775245700 1 -- start start 2024-04-22T23:41:20.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.055+0000 7f1775245700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17700ffc20 0x7f177019c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:20.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.055+0000 7f1775245700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1770105bd0 0x7f177019cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:20.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.055+0000 7f1775245700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f177019d1f0 con 0x7f17700ffc20 2024-04-22T23:41:20.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.055+0000 7f1775245700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17701966a0 con 0x7f1770105bd0 2024-04-22T23:41:20.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.055+0000 7f176ed9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17700ffc20 0x7f177019c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:20.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.055+0000 7f176ed9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17700ffc20 0x7f177019c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47108/0 (socket says 172.21.15.67:47108) 2024-04-22T23:41:20.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.055+0000 7f176ed9d700 1 -- 172.21.15.67:0/2647818896 learned_addr learned my addr 172.21.15.67:0/2647818896 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:41:20.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.056+0000 7f176e59c700 1 --2- 172.21.15.67:0/2647818896 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1770105bd0 0x7f177019cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:20.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.056+0000 7f176ed9d700 1 -- 172.21.15.67:0/2647818896 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1770105bd0 msgr2=0x7f177019cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:20.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.056+0000 7f176ed9d700 1 --2- 172.21.15.67:0/2647818896 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1770105bd0 0x7f177019cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:20.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.056+0000 7f176ed9d700 1 -- 172.21.15.67:0/2647818896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17600095f0 con 0x7f17700ffc20 2024-04-22T23:41:20.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.056+0000 7f176e59c700 1 --2- 172.21.15.67:0/2647818896 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1770105bd0 0x7f177019cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:41:20.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.056+0000 7f176ed9d700 1 --2- 172.21.15.67:0/2647818896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17700ffc20 0x7f177019c620 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f1760006010 tx=0x7f176000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:41:20.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.056+0000 7f1767fff700 1 -- 172.21.15.67:0/2647818896 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f176001d070 con 0x7f17700ffc20 2024-04-22T23:41:20.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.056+0000 7f1775245700 1 -- 172.21.15.67:0/2647818896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1770196920 con 0x7f17700ffc20 2024-04-22T23:41:20.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.056+0000 7f1767fff700 1 -- 172.21.15.67:0/2647818896 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1760004fb0 con 0x7f17700ffc20 2024-04-22T23:41:20.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.056+0000 7f1767fff700 1 -- 172.21.15.67:0/2647818896 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1760022ea0 con 0x7f17700ffc20 2024-04-22T23:41:20.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.056+0000 7f1775245700 1 -- 172.21.15.67:0/2647818896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1770196e10 con 0x7f17700ffc20 2024-04-22T23:41:20.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.057+0000 7f1775245700 1 -- 172.21.15.67:0/2647818896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f177004f2e0 con 0x7f17700ffc20 2024-04-22T23:41:20.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.058+0000 7f1767fff700 1 -- 172.21.15.67:0/2647818896 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1760022470 con 0x7f17700ffc20 2024-04-22T23:41:20.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.058+0000 7f1767fff700 1 --2- 172.21.15.67:0/2647818896 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f175c06ecc0 0x7f175c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:20.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.058+0000 7f176e59c700 1 --2- 172.21.15.67:0/2647818896 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f175c06ecc0 0x7f175c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:20.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.058+0000 7f1767fff700 1 -- 172.21.15.67:0/2647818896 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f1760090560 con 0x7f17700ffc20 2024-04-22T23:41:20.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.059+0000 7f176e59c700 1 --2- 172.21.15.67:0/2647818896 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f175c06ecc0 0x7f175c071180 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1758005fd0 tx=0x7f1758005dc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:41:20.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.063+0000 7f1767fff700 1 -- 172.21.15.67:0/2647818896 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f176005cd80 con 0x7f17700ffc20 2024-04-22T23:41:20.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.322+0000 7f1775245700 1 -- 172.21.15.67:0/2647818896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f17701970f0 con 0x7f17700ffc20 2024-04-22T23:41:20.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.322+0000 7f1767fff700 1 -- 172.21.15.67:0/2647818896 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1871 (secure 0 0 0) 0x7f1760027760 con 0x7f17700ffc20 2024-04-22T23:41:20.315 INFO:teuthology.orchestra.run.smithi067.stdout:e31 2024-04-22T23:41:20.316 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:41:20.316 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:41:20.316 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:41:20.316 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:41:20.316 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:41:20.316 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:41:20.316 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:41:20.316 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:41:20.316 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:41:20.316 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:41:20.316 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:41:20.316 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:41:20.316 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:41:20.317 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:41:20.317 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:41:20.317 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:41:20.317 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:41:20.317 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:41:20.317 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:41:20.317 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:41:20.317 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:41:20.317 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:41:20.317 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:41:20.317 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:41:20.317 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:41:20.317 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:41:20.318 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:41:20.318 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:41:20.318 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:41:20.318 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:41:20.318 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:41:20.318 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:41:20.318 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:41:20.318 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:41:20.318 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:41:20.318 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:41:20.318 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:41:20.318 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:41:20.319 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:41:20.319 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44259} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1827288036,v1:172.21.15.134:6827/1827288036] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:41:20.319 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:41:20.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.324+0000 7f1775245700 1 -- 172.21.15.67:0/2647818896 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f175c06ecc0 msgr2=0x7f175c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:20.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.324+0000 7f1775245700 1 --2- 172.21.15.67:0/2647818896 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f175c06ecc0 0x7f175c071180 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1758005fd0 tx=0x7f1758005dc0 comp rx=0 tx=0).stop 2024-04-22T23:41:20.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.324+0000 7f1775245700 1 -- 172.21.15.67:0/2647818896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17700ffc20 msgr2=0x7f177019c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:20.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.324+0000 7f1775245700 1 --2- 172.21.15.67:0/2647818896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17700ffc20 0x7f177019c620 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f1760006010 tx=0x7f176000be20 comp rx=0 tx=0).stop 2024-04-22T23:41:20.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.325+0000 7f1775245700 1 -- 172.21.15.67:0/2647818896 shutdown_connections 2024-04-22T23:41:20.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.325+0000 7f1775245700 1 --2- 172.21.15.67:0/2647818896 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f175c06ecc0 0x7f175c071180 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:20.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.325+0000 7f1775245700 1 --2- 172.21.15.67:0/2647818896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17700ffc20 0x7f177019c620 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:20.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.325+0000 7f1775245700 1 --2- 172.21.15.67:0/2647818896 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1770105bd0 0x7f177019cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:20.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.325+0000 7f1775245700 1 -- 172.21.15.67:0/2647818896 >> 172.21.15.67:0/2647818896 conn(0x7f1770075670 msgr2=0x7f1770068680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:41:20.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.325+0000 7f1775245700 1 -- 172.21.15.67:0/2647818896 shutdown_connections 2024-04-22T23:41:20.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.325+0000 7f1775245700 1 -- 172.21.15.67:0/2647818896 wait complete. 2024-04-22T23:41:20.320 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 31 2024-04-22T23:41:20.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.432+0000 7fc23fb51700 1 -- 172.21.15.67:0/229283306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc238105bd0 msgr2=0x7fc238105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:20.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.432+0000 7fc23fb51700 1 --2- 172.21.15.67:0/229283306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc238105bd0 0x7fc238105fb0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc234009910 tx=0x7fc234009c20 comp rx=0 tx=0).stop 2024-04-22T23:41:20.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.432+0000 7fc23fb51700 1 -- 172.21.15.67:0/229283306 shutdown_connections 2024-04-22T23:41:20.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.432+0000 7fc23fb51700 1 --2- 172.21.15.67:0/229283306 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2380ffc20 0x7fc238100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:20.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.432+0000 7fc23fb51700 1 --2- 172.21.15.67:0/229283306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc238105bd0 0x7fc238105fb0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:20.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.432+0000 7fc23fb51700 1 -- 172.21.15.67:0/229283306 >> 172.21.15.67:0/229283306 conn(0x7fc238075670 msgr2=0x7fc238075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:41:20.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.432+0000 7fc23fb51700 1 -- 172.21.15.67:0/229283306 shutdown_connections 2024-04-22T23:41:20.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.432+0000 7fc23fb51700 1 -- 172.21.15.67:0/229283306 wait complete. 2024-04-22T23:41:20.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.433+0000 7fc23fb51700 1 Processor -- start 2024-04-22T23:41:20.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.433+0000 7fc23fb51700 1 -- start start 2024-04-22T23:41:20.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.433+0000 7fc23fb51700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2380ffc20 0x7fc2381976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:20.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.433+0000 7fc23fb51700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc238105bd0 0x7fc238197c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:20.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.433+0000 7fc23fb51700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc238198260 con 0x7fc238105bd0 2024-04-22T23:41:20.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.433+0000 7fc23fb51700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2381983d0 con 0x7fc2380ffc20 2024-04-22T23:41:20.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.433+0000 7fc23d0ec700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc238105bd0 0x7fc238197c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:20.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.433+0000 7fc23d0ec700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc238105bd0 0x7fc238197c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47126/0 (socket says 172.21.15.67:47126) 2024-04-22T23:41:20.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.433+0000 7fc23d0ec700 1 -- 172.21.15.67:0/1081525373 learned_addr learned my addr 172.21.15.67:0/1081525373 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:41:20.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.433+0000 7fc23d8ed700 1 --2- 172.21.15.67:0/1081525373 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2380ffc20 0x7fc2381976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:20.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.433+0000 7fc23d0ec700 1 -- 172.21.15.67:0/1081525373 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2380ffc20 msgr2=0x7fc2381976d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:20.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.433+0000 7fc23d0ec700 1 --2- 172.21.15.67:0/1081525373 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2380ffc20 0x7fc2381976d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:20.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.433+0000 7fc23d0ec700 1 -- 172.21.15.67:0/1081525373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2340095f0 con 0x7fc238105bd0 2024-04-22T23:41:20.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.434+0000 7fc23d0ec700 1 --2- 172.21.15.67:0/1081525373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc238105bd0 0x7fc238197c10 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc22c00c930 tx=0x7fc22c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:41:20.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.434+0000 7fc22affd700 1 -- 172.21.15.67:0/1081525373 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc22c007ad0 con 0x7fc238105bd0 2024-04-22T23:41:20.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.434+0000 7fc23fb51700 1 -- 172.21.15.67:0/1081525373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2381a0740 con 0x7fc238105bd0 2024-04-22T23:41:20.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.434+0000 7fc22affd700 1 -- 172.21.15.67:0/1081525373 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc22c00ce80 con 0x7fc238105bd0 2024-04-22T23:41:20.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.434+0000 7fc22affd700 1 -- 172.21.15.67:0/1081525373 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc22c018630 con 0x7fc238105bd0 2024-04-22T23:41:20.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.434+0000 7fc23fb51700 1 -- 172.21.15.67:0/1081525373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2381a0c00 con 0x7fc238105bd0 2024-04-22T23:41:20.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.435+0000 7fc23fb51700 1 -- 172.21.15.67:0/1081525373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2381080d0 con 0x7fc238105bd0 2024-04-22T23:41:20.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.436+0000 7fc22affd700 1 -- 172.21.15.67:0/1081525373 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc22c007c30 con 0x7fc238105bd0 2024-04-22T23:41:20.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.436+0000 7fc22affd700 1 --2- 172.21.15.67:0/1081525373 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc22406ecc0 0x7fc224071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:20.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.436+0000 7fc22affd700 1 -- 172.21.15.67:0/1081525373 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fc22c08f010 con 0x7fc238105bd0 2024-04-22T23:41:20.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.436+0000 7fc23d8ed700 1 --2- 172.21.15.67:0/1081525373 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc22406ecc0 0x7fc224071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:20.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.437+0000 7fc23d8ed700 1 --2- 172.21.15.67:0/1081525373 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc22406ecc0 0x7fc224071180 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc234006010 tx=0x7fc23401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:41:20.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.441+0000 7fc22affd700 1 -- 172.21.15.67:0/1081525373 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc22c057c50 con 0x7fc238105bd0 2024-04-22T23:41:20.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.672+0000 7fc23fb51700 1 -- 172.21.15.67:0/1081525373 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc238068c80 con 0x7fc22406ecc0 2024-04-22T23:41:20.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.673+0000 7fc22affd700 1 -- 172.21.15.67:0/1081525373 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc238068c80 con 0x7fc22406ecc0 2024-04-22T23:41:20.666 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:41:20.666 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:41:20.666 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:41:20.666 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:41:20.666 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:41:20.666 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:41:20.667 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:41:20.667 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:41:20.667 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:41:20.667 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:41:20.667 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T23:41:20.667 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:41:20.667 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:41:20.667 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:41:20.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.675+0000 7fc23fb51700 1 -- 172.21.15.67:0/1081525373 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc22406ecc0 msgr2=0x7fc224071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:20.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.675+0000 7fc23fb51700 1 --2- 172.21.15.67:0/1081525373 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc22406ecc0 0x7fc224071180 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc234006010 tx=0x7fc23401a040 comp rx=0 tx=0).stop 2024-04-22T23:41:20.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.675+0000 7fc23fb51700 1 -- 172.21.15.67:0/1081525373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc238105bd0 msgr2=0x7fc238197c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:20.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.675+0000 7fc23fb51700 1 --2- 172.21.15.67:0/1081525373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc238105bd0 0x7fc238197c10 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc22c00c930 tx=0x7fc22c00cc40 comp rx=0 tx=0).stop 2024-04-22T23:41:20.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.675+0000 7fc23fb51700 1 -- 172.21.15.67:0/1081525373 shutdown_connections 2024-04-22T23:41:20.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.675+0000 7fc23fb51700 1 --2- 172.21.15.67:0/1081525373 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc22406ecc0 0x7fc224071180 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:20.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.675+0000 7fc23fb51700 1 --2- 172.21.15.67:0/1081525373 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2380ffc20 0x7fc2381976d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:20.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.675+0000 7fc23fb51700 1 --2- 172.21.15.67:0/1081525373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc238105bd0 0x7fc238197c10 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:20.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.675+0000 7fc23fb51700 1 -- 172.21.15.67:0/1081525373 >> 172.21.15.67:0/1081525373 conn(0x7fc238075670 msgr2=0x7fc2380feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:41:20.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.675+0000 7fc23fb51700 1 -- 172.21.15.67:0/1081525373 shutdown_connections 2024-04-22T23:41:20.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.675+0000 7fc23fb51700 1 -- 172.21.15.67:0/1081525373 wait complete. 2024-04-22T23:41:20.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.783+0000 7f21784bc700 1 -- 172.21.15.67:0/2016812597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2170068580 msgr2=0x7f2170068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:20.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.783+0000 7f21784bc700 1 --2- 172.21.15.67:0/2016812597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2170068580 0x7f2170068960 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f216c009910 tx=0x7f216c009c20 comp rx=0 tx=0).stop 2024-04-22T23:41:20.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.783+0000 7f21784bc700 1 -- 172.21.15.67:0/2016812597 shutdown_connections 2024-04-22T23:41:20.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.783+0000 7f21784bc700 1 --2- 172.21.15.67:0/2016812597 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2170068ea0 0x7f217010d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:20.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.783+0000 7f21784bc700 1 --2- 172.21.15.67:0/2016812597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2170068580 0x7f2170068960 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:20.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.783+0000 7f21784bc700 1 -- 172.21.15.67:0/2016812597 >> 172.21.15.67:0/2016812597 conn(0x7f2170075670 msgr2=0x7f2170075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:41:20.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.783+0000 7f21784bc700 1 -- 172.21.15.67:0/2016812597 shutdown_connections 2024-04-22T23:41:20.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.783+0000 7f21784bc700 1 -- 172.21.15.67:0/2016812597 wait complete. 2024-04-22T23:41:20.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.783+0000 7f21784bc700 1 Processor -- start 2024-04-22T23:41:20.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.783+0000 7f21784bc700 1 -- start start 2024-04-22T23:41:20.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.784+0000 7f21784bc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2170068580 0x7f217019e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:20.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.784+0000 7f21784bc700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2170068ea0 0x7f217019eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:20.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.784+0000 7f21784bc700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f217019f3a0 con 0x7f2170068580 2024-04-22T23:41:20.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.784+0000 7f21784bc700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21701988e0 con 0x7f2170068ea0 2024-04-22T23:41:20.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.784+0000 7f2176258700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2170068580 0x7f217019e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:20.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.784+0000 7f2176258700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2170068580 0x7f217019e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47134/0 (socket says 172.21.15.67:47134) 2024-04-22T23:41:20.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.784+0000 7f2176258700 1 -- 172.21.15.67:0/1400107939 learned_addr learned my addr 172.21.15.67:0/1400107939 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:41:20.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.784+0000 7f2175a57700 1 --2- 172.21.15.67:0/1400107939 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2170068ea0 0x7f217019eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:20.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.784+0000 7f2176258700 1 -- 172.21.15.67:0/1400107939 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2170068ea0 msgr2=0x7f217019eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:20.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.784+0000 7f2176258700 1 --2- 172.21.15.67:0/1400107939 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2170068ea0 0x7f217019eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:20.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.784+0000 7f2176258700 1 -- 172.21.15.67:0/1400107939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f216c0095f0 con 0x7f2170068580 2024-04-22T23:41:20.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.784+0000 7f2175a57700 1 --2- 172.21.15.67:0/1400107939 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2170068ea0 0x7f217019eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:41:20.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.784+0000 7f2176258700 1 --2- 172.21.15.67:0/1400107939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2170068580 0x7f217019e860 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f216c006010 tx=0x7f216c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:41:20.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.785+0000 7f21677fe700 1 -- 172.21.15.67:0/1400107939 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f216c01d070 con 0x7f2170068580 2024-04-22T23:41:20.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.785+0000 7f21677fe700 1 -- 172.21.15.67:0/1400107939 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f216c004fb0 con 0x7f2170068580 2024-04-22T23:41:20.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.785+0000 7f21784bc700 1 -- 172.21.15.67:0/1400107939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2170198b60 con 0x7f2170068580 2024-04-22T23:41:20.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.785+0000 7f21677fe700 1 -- 172.21.15.67:0/1400107939 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f216c022ea0 con 0x7f2170068580 2024-04-22T23:41:20.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.785+0000 7f21784bc700 1 -- 172.21.15.67:0/1400107939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2170199050 con 0x7f2170068580 2024-04-22T23:41:20.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.786+0000 7f21784bc700 1 -- 172.21.15.67:0/1400107939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f217010aa00 con 0x7f2170068580 2024-04-22T23:41:20.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.786+0000 7f21677fe700 1 -- 172.21.15.67:0/1400107939 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f216c022470 con 0x7f2170068580 2024-04-22T23:41:20.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.787+0000 7f21677fe700 1 --2- 172.21.15.67:0/1400107939 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f215c06ecc0 0x7f215c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:20.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.787+0000 7f21677fe700 1 -- 172.21.15.67:0/1400107939 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f216c090570 con 0x7f2170068580 2024-04-22T23:41:20.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.787+0000 7f2175a57700 1 --2- 172.21.15.67:0/1400107939 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f215c06ecc0 0x7f215c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:20.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.787+0000 7f2175a57700 1 --2- 172.21.15.67:0/1400107939 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f215c06ecc0 0x7f215c071180 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2160005fd0 tx=0x7f2160005e20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:41:20.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:20.792+0000 7f21677fe700 1 -- 172.21.15.67:0/1400107939 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f216c05cd90 con 0x7f2170068580 2024-04-22T23:41:20.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:20 smithi134 ceph-mon[85630]: from='client.34852 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:41:20.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:20 smithi134 ceph-mon[85630]: from='client.34856 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:41:20.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:20 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1944896788' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:41:20.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:20 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2647818896' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:41:21.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:20 smithi067 ceph-mon[112438]: from='client.34852 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:41:21.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:20 smithi067 ceph-mon[112438]: from='client.34856 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:41:21.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:20 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1944896788' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:41:21.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:20 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2647818896' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:41:21.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:21.088+0000 7f21784bc700 1 -- 172.21.15.67:0/1400107939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f2170199df0 con 0x7f2170068580 2024-04-22T23:41:21.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:21.088+0000 7f21677fe700 1 -- 172.21.15.67:0/1400107939 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f216c022720 con 0x7f2170068580 2024-04-22T23:41:21.081 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:41:21.082 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:41:21.082 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:41:21.082 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:41:21.082 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:41:21.082 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:41:21.082 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:41:21.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:21.090+0000 7f21784bc700 1 -- 172.21.15.67:0/1400107939 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f215c06ecc0 msgr2=0x7f215c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:21.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:21.090+0000 7f21784bc700 1 --2- 172.21.15.67:0/1400107939 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f215c06ecc0 0x7f215c071180 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2160005fd0 tx=0x7f2160005e20 comp rx=0 tx=0).stop 2024-04-22T23:41:21.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:21.090+0000 7f21784bc700 1 -- 172.21.15.67:0/1400107939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2170068580 msgr2=0x7f217019e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:21.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:21.090+0000 7f21784bc700 1 --2- 172.21.15.67:0/1400107939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2170068580 0x7f217019e860 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f216c006010 tx=0x7f216c00be20 comp rx=0 tx=0).stop 2024-04-22T23:41:21.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:21.090+0000 7f21784bc700 1 -- 172.21.15.67:0/1400107939 shutdown_connections 2024-04-22T23:41:21.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:21.090+0000 7f21784bc700 1 --2- 172.21.15.67:0/1400107939 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f215c06ecc0 0x7f215c071180 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:21.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:21.090+0000 7f21784bc700 1 --2- 172.21.15.67:0/1400107939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2170068580 0x7f217019e860 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:21.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:21.090+0000 7f21784bc700 1 --2- 172.21.15.67:0/1400107939 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2170068ea0 0x7f217019eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:21.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:21.090+0000 7f21784bc700 1 -- 172.21.15.67:0/1400107939 >> 172.21.15.67:0/1400107939 conn(0x7f2170075670 msgr2=0x7f21700fe790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:41:21.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:21.091+0000 7f21784bc700 1 -- 172.21.15.67:0/1400107939 shutdown_connections 2024-04-22T23:41:21.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:21.091+0000 7f21784bc700 1 -- 172.21.15.67:0/1400107939 wait complete. 2024-04-22T23:41:21.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:21 smithi134 ceph-mon[85630]: pgmap v449: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:21.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:21 smithi134 ceph-mon[85630]: from='client.34868 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:41:21.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:21 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1400107939' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:41:22.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:21 smithi067 ceph-mon[112438]: pgmap v449: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:22.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:21 smithi067 ceph-mon[112438]: from='client.34868 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:41:22.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:21 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1400107939' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:41:23.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:23 smithi134 ceph-mon[85630]: pgmap v450: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:24.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:23 smithi067 ceph-mon[112438]: pgmap v450: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:25.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:25 smithi134 ceph-mon[85630]: pgmap v451: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:26.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:25 smithi067 ceph-mon[112438]: pgmap v451: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:26.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:41:27.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:41:27.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:27 smithi134 ceph-mon[85630]: pgmap v452: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:28.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:27 smithi067 ceph-mon[112438]: pgmap v452: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:29.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:29 smithi134 ceph-mon[85630]: pgmap v453: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:30.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:29 smithi067 ceph-mon[112438]: pgmap v453: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:31.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:31 smithi134 ceph-mon[85630]: pgmap v454: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:32.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:31 smithi067 ceph-mon[112438]: pgmap v454: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:34.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:33 smithi067 ceph-mon[112438]: pgmap v455: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:34.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:33 smithi134 ceph-mon[85630]: pgmap v455: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:36.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:35 smithi067 ceph-mon[112438]: pgmap v456: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:36.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:35 smithi134 ceph-mon[85630]: pgmap v456: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:37.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:37 smithi134 ceph-mon[85630]: pgmap v457: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:38.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:37 smithi067 ceph-mon[112438]: pgmap v457: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:40.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:39 smithi067 ceph-mon[112438]: pgmap v458: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:40.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:39 smithi134 ceph-mon[85630]: pgmap v458: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:42.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:41 smithi067 ceph-mon[112438]: pgmap v459: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:42.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:41:42.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:41 smithi134 ceph-mon[85630]: pgmap v459: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:42.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:41:44.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:43 smithi067 ceph-mon[112438]: pgmap v460: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:44.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:43 smithi134 ceph-mon[85630]: pgmap v460: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:46.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:45 smithi067 ceph-mon[112438]: pgmap v461: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:46.038 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:45 smithi134 ceph-mon[85630]: pgmap v461: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:47.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:47 smithi134 ceph-mon[85630]: pgmap v462: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:48.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:47 smithi067 ceph-mon[112438]: pgmap v462: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:50.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:49 smithi067 ceph-mon[112438]: pgmap v463: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:50.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:49 smithi134 ceph-mon[85630]: pgmap v463: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:51.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.211+0000 7f2e388bd700 1 -- 172.21.15.67:0/2113402593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e34100ea0 msgr2=0x7f2e34101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:51.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.211+0000 7f2e388bd700 1 --2- 172.21.15.67:0/2113402593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e34100ea0 0x7f2e34101280 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f2e24009910 tx=0x7f2e24009c20 comp rx=0 tx=0).stop 2024-04-22T23:41:51.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.211+0000 7f2e388bd700 1 -- 172.21.15.67:0/2113402593 shutdown_connections 2024-04-22T23:41:51.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.211+0000 7f2e388bd700 1 --2- 172.21.15.67:0/2113402593 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e34101850 0x7f2e341058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:51.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.211+0000 7f2e388bd700 1 --2- 172.21.15.67:0/2113402593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e34100ea0 0x7f2e34101280 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:51.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.211+0000 7f2e388bd700 1 -- 172.21.15.67:0/2113402593 >> 172.21.15.67:0/2113402593 conn(0x7f2e340fc750 msgr2=0x7f2e340feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:41:51.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.211+0000 7f2e388bd700 1 -- 172.21.15.67:0/2113402593 shutdown_connections 2024-04-22T23:41:51.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.211+0000 7f2e388bd700 1 -- 172.21.15.67:0/2113402593 wait complete. 2024-04-22T23:41:51.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.212+0000 7f2e388bd700 1 Processor -- start 2024-04-22T23:41:51.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.212+0000 7f2e388bd700 1 -- start start 2024-04-22T23:41:51.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.212+0000 7f2e388bd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e34100ea0 0x7f2e3419c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:51.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.212+0000 7f2e388bd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e34101850 0x7f2e3419cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:51.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.212+0000 7f2e388bd700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3419d2c0 con 0x7f2e34101850 2024-04-22T23:41:51.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.212+0000 7f2e388bd700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e34196770 con 0x7f2e34100ea0 2024-04-22T23:41:51.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.212+0000 7f2e31d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e34101850 0x7f2e3419cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:51.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.212+0000 7f2e31d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e34101850 0x7f2e3419cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47116/0 (socket says 172.21.15.67:47116) 2024-04-22T23:41:51.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.212+0000 7f2e31d9b700 1 -- 172.21.15.67:0/3603373935 learned_addr learned my addr 172.21.15.67:0/3603373935 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:41:51.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.213+0000 7f2e3259c700 1 --2- 172.21.15.67:0/3603373935 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e34100ea0 0x7f2e3419c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:51.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.213+0000 7f2e31d9b700 1 -- 172.21.15.67:0/3603373935 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e34100ea0 msgr2=0x7f2e3419c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:51.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.213+0000 7f2e31d9b700 1 --2- 172.21.15.67:0/3603373935 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e34100ea0 0x7f2e3419c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:51.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.213+0000 7f2e31d9b700 1 -- 172.21.15.67:0/3603373935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e240095f0 con 0x7f2e34101850 2024-04-22T23:41:51.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.213+0000 7f2e3259c700 1 --2- 172.21.15.67:0/3603373935 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e34100ea0 0x7f2e3419c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:41:51.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.213+0000 7f2e31d9b700 1 --2- 172.21.15.67:0/3603373935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e34101850 0x7f2e3419cc30 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f2e2800e8f0 tx=0x7f2e2800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:41:51.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.213+0000 7f2e237fe700 1 -- 172.21.15.67:0/3603373935 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e28009790 con 0x7f2e34101850 2024-04-22T23:41:51.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.213+0000 7f2e237fe700 1 -- 172.21.15.67:0/3603373935 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2e28004cf0 con 0x7f2e34101850 2024-04-22T23:41:51.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.213+0000 7f2e237fe700 1 -- 172.21.15.67:0/3603373935 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e28005690 con 0x7f2e34101850 2024-04-22T23:41:51.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.213+0000 7f2e388bd700 1 -- 172.21.15.67:0/3603373935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e34196a50 con 0x7f2e34101850 2024-04-22T23:41:51.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.213+0000 7f2e388bd700 1 -- 172.21.15.67:0/3603373935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e34196f50 con 0x7f2e34101850 2024-04-22T23:41:51.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.214+0000 7f2e388bd700 1 -- 172.21.15.67:0/3603373935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e34109240 con 0x7f2e34101850 2024-04-22T23:41:51.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.215+0000 7f2e237fe700 1 -- 172.21.15.67:0/3603373935 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2e28004750 con 0x7f2e34101850 2024-04-22T23:41:51.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.215+0000 7f2e237fe700 1 --2- 172.21.15.67:0/3603373935 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2e1c06ecc0 0x7f2e1c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:51.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.215+0000 7f2e3259c700 1 --2- 172.21.15.67:0/3603373935 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2e1c06ecc0 0x7f2e1c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:51.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.215+0000 7f2e237fe700 1 -- 172.21.15.67:0/3603373935 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f2e28014070 con 0x7f2e34101850 2024-04-22T23:41:51.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.216+0000 7f2e3259c700 1 --2- 172.21.15.67:0/3603373935 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2e1c06ecc0 0x7f2e1c071180 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2e2400b760 tx=0x7f2e24005a90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:41:51.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.220+0000 7f2e237fe700 1 -- 172.21.15.67:0/3603373935 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f2e2805bb30 con 0x7f2e34101850 2024-04-22T23:41:51.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.451+0000 7f2e388bd700 1 -- 172.21.15.67:0/3603373935 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e34197d90 con 0x7f2e1c06ecc0 2024-04-22T23:41:51.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.452+0000 7f2e237fe700 1 -- 172.21.15.67:0/3603373935 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f2e34197d90 con 0x7f2e1c06ecc0 2024-04-22T23:41:51.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.454+0000 7f2e388bd700 1 -- 172.21.15.67:0/3603373935 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2e1c06ecc0 msgr2=0x7f2e1c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:51.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.454+0000 7f2e388bd700 1 --2- 172.21.15.67:0/3603373935 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2e1c06ecc0 0x7f2e1c071180 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2e2400b760 tx=0x7f2e24005a90 comp rx=0 tx=0).stop 2024-04-22T23:41:51.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.454+0000 7f2e388bd700 1 -- 172.21.15.67:0/3603373935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e34101850 msgr2=0x7f2e3419cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:51.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.454+0000 7f2e388bd700 1 --2- 172.21.15.67:0/3603373935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e34101850 0x7f2e3419cc30 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f2e2800e8f0 tx=0x7f2e2800ecb0 comp rx=0 tx=0).stop 2024-04-22T23:41:51.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.454+0000 7f2e388bd700 1 -- 172.21.15.67:0/3603373935 shutdown_connections 2024-04-22T23:41:51.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.454+0000 7f2e388bd700 1 --2- 172.21.15.67:0/3603373935 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2e1c06ecc0 0x7f2e1c071180 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:51.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.454+0000 7f2e388bd700 1 --2- 172.21.15.67:0/3603373935 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e34100ea0 0x7f2e3419c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:51.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.454+0000 7f2e388bd700 1 --2- 172.21.15.67:0/3603373935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e34101850 0x7f2e3419cc30 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:51.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.454+0000 7f2e388bd700 1 -- 172.21.15.67:0/3603373935 >> 172.21.15.67:0/3603373935 conn(0x7f2e340fc750 msgr2=0x7f2e340fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:41:51.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.454+0000 7f2e388bd700 1 -- 172.21.15.67:0/3603373935 shutdown_connections 2024-04-22T23:41:51.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.454+0000 7f2e388bd700 1 -- 172.21.15.67:0/3603373935 wait complete. 2024-04-22T23:41:51.458 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:41:51.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.564+0000 7f7979a2a700 1 -- 172.21.15.67:0/4235037205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7974100750 msgr2=0x7f79741047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:51.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.564+0000 7f7979a2a700 1 --2- 172.21.15.67:0/4235037205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7974100750 0x7f79741047a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f795c009910 tx=0x7f795c009c20 comp rx=0 tx=0).stop 2024-04-22T23:41:51.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.564+0000 7f7979a2a700 1 -- 172.21.15.67:0/4235037205 shutdown_connections 2024-04-22T23:41:51.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.564+0000 7f7979a2a700 1 --2- 172.21.15.67:0/4235037205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7974100750 0x7f79741047a0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:51.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.564+0000 7f7979a2a700 1 --2- 172.21.15.67:0/4235037205 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79740ffda0 0x7f7974100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:51.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.564+0000 7f7979a2a700 1 -- 172.21.15.67:0/4235037205 >> 172.21.15.67:0/4235037205 conn(0x7f79740fb650 msgr2=0x7f79740fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:41:51.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.564+0000 7f7979a2a700 1 -- 172.21.15.67:0/4235037205 shutdown_connections 2024-04-22T23:41:51.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.564+0000 7f7979a2a700 1 -- 172.21.15.67:0/4235037205 wait complete. 2024-04-22T23:41:51.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.564+0000 7f7979a2a700 1 Processor -- start 2024-04-22T23:41:51.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.564+0000 7f7979a2a700 1 -- start start 2024-04-22T23:41:51.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.565+0000 7f7979a2a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79740ffda0 0x7f7974196950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:51.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.565+0000 7f7979a2a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7974100750 0x7f7974196e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:51.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.565+0000 7f7979a2a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7974197490 con 0x7f7974100750 2024-04-22T23:41:51.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.565+0000 7f7979a2a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7974197600 con 0x7f79740ffda0 2024-04-22T23:41:51.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.565+0000 7f796bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7974100750 0x7f7974196e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:51.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.565+0000 7f796bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7974100750 0x7f7974196e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47144/0 (socket says 172.21.15.67:47144) 2024-04-22T23:41:51.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.565+0000 7f796bfff700 1 -- 172.21.15.67:0/1031478155 learned_addr learned my addr 172.21.15.67:0/1031478155 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:41:51.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.565+0000 7f7972ffd700 1 --2- 172.21.15.67:0/1031478155 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79740ffda0 0x7f7974196950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:51.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.565+0000 7f796bfff700 1 -- 172.21.15.67:0/1031478155 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79740ffda0 msgr2=0x7f7974196950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:51.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.565+0000 7f796bfff700 1 --2- 172.21.15.67:0/1031478155 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79740ffda0 0x7f7974196950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:51.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.565+0000 7f796bfff700 1 -- 172.21.15.67:0/1031478155 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f795c0095f0 con 0x7f7974100750 2024-04-22T23:41:51.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.565+0000 7f796bfff700 1 --2- 172.21.15.67:0/1031478155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7974100750 0x7f7974196e90 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f795c00bda0 tx=0x7f795c00bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:41:51.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.565+0000 7f7978a28700 1 -- 172.21.15.67:0/1031478155 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f795c01d070 con 0x7f7974100750 2024-04-22T23:41:51.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.565+0000 7f7978a28700 1 -- 172.21.15.67:0/1031478155 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f795c00f460 con 0x7f7974100750 2024-04-22T23:41:51.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.565+0000 7f7978a28700 1 -- 172.21.15.67:0/1031478155 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f795c017630 con 0x7f7974100750 2024-04-22T23:41:51.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.565+0000 7f7979a2a700 1 -- 172.21.15.67:0/1031478155 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f797419b420 con 0x7f7974100750 2024-04-22T23:41:51.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.566+0000 7f7979a2a700 1 -- 172.21.15.67:0/1031478155 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f797419b8c0 con 0x7f7974100750 2024-04-22T23:41:51.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.566+0000 7f7979a2a700 1 -- 172.21.15.67:0/1031478155 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f797404f2e0 con 0x7f7974100750 2024-04-22T23:41:51.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.567+0000 7f7978a28700 1 -- 172.21.15.67:0/1031478155 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f795c00fab0 con 0x7f7974100750 2024-04-22T23:41:51.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.567+0000 7f7978a28700 1 --2- 172.21.15.67:0/1031478155 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f796006ecc0 0x7f7960071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:51.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.567+0000 7f7978a28700 1 -- 172.21.15.67:0/1031478155 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f795c090440 con 0x7f7974100750 2024-04-22T23:41:51.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.567+0000 7f7972ffd700 1 --2- 172.21.15.67:0/1031478155 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f796006ecc0 0x7f7960071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:51.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.568+0000 7f7972ffd700 1 --2- 172.21.15.67:0/1031478155 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f796006ecc0 0x7f7960071180 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f796400ac50 tx=0x7f796400a380 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:41:51.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.572+0000 7f7978a28700 1 -- 172.21.15.67:0/1031478155 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f795c05cbb0 con 0x7f7974100750 2024-04-22T23:41:51.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.802+0000 7f7979a2a700 1 -- 172.21.15.67:0/1031478155 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f797419bba0 con 0x7f796006ecc0 2024-04-22T23:41:51.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.803+0000 7f7978a28700 1 -- 172.21.15.67:0/1031478155 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f797419bba0 con 0x7f796006ecc0 2024-04-22T23:41:51.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.805+0000 7f7979a2a700 1 -- 172.21.15.67:0/1031478155 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f796006ecc0 msgr2=0x7f7960071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:51.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.805+0000 7f7979a2a700 1 --2- 172.21.15.67:0/1031478155 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f796006ecc0 0x7f7960071180 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f796400ac50 tx=0x7f796400a380 comp rx=0 tx=0).stop 2024-04-22T23:41:51.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.805+0000 7f7979a2a700 1 -- 172.21.15.67:0/1031478155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7974100750 msgr2=0x7f7974196e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:51.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.805+0000 7f7979a2a700 1 --2- 172.21.15.67:0/1031478155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7974100750 0x7f7974196e90 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f795c00bda0 tx=0x7f795c00bdd0 comp rx=0 tx=0).stop 2024-04-22T23:41:51.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.805+0000 7f7979a2a700 1 -- 172.21.15.67:0/1031478155 shutdown_connections 2024-04-22T23:41:51.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.805+0000 7f7979a2a700 1 --2- 172.21.15.67:0/1031478155 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f796006ecc0 0x7f7960071180 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:51.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.805+0000 7f7979a2a700 1 --2- 172.21.15.67:0/1031478155 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79740ffda0 0x7f7974196950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:51.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.805+0000 7f7979a2a700 1 --2- 172.21.15.67:0/1031478155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7974100750 0x7f7974196e90 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:51.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.805+0000 7f7979a2a700 1 -- 172.21.15.67:0/1031478155 >> 172.21.15.67:0/1031478155 conn(0x7f79740fb650 msgr2=0x7f7974103fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:41:51.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.805+0000 7f7979a2a700 1 -- 172.21.15.67:0/1031478155 shutdown_connections 2024-04-22T23:41:51.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.805+0000 7f7979a2a700 1 -- 172.21.15.67:0/1031478155 wait complete. 2024-04-22T23:41:51.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.912+0000 7f0dae438700 1 -- 172.21.15.67:0/3252447649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0da80fff50 msgr2=0x7f0da8105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:51.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.912+0000 7f0dae438700 1 --2- 172.21.15.67:0/3252447649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0da80fff50 0x7f0da8105840 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c009910 tx=0x7f0d9c009c20 comp rx=0 tx=0).stop 2024-04-22T23:41:51.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.912+0000 7f0dae438700 1 -- 172.21.15.67:0/3252447649 shutdown_connections 2024-04-22T23:41:51.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.912+0000 7f0dae438700 1 --2- 172.21.15.67:0/3252447649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0da80fff50 0x7f0da8105840 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:51.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.912+0000 7f0dae438700 1 --2- 172.21.15.67:0/3252447649 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0da80ff5a0 0x7f0da80ff980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:51.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.912+0000 7f0dae438700 1 -- 172.21.15.67:0/3252447649 >> 172.21.15.67:0/3252447649 conn(0x7f0da8074e30 msgr2=0x7f0da8075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:41:51.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.913+0000 7f0dae438700 1 -- 172.21.15.67:0/3252447649 shutdown_connections 2024-04-22T23:41:51.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.913+0000 7f0dae438700 1 -- 172.21.15.67:0/3252447649 wait complete. 2024-04-22T23:41:51.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.913+0000 7f0dae438700 1 Processor -- start 2024-04-22T23:41:51.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.913+0000 7f0dae438700 1 -- start start 2024-04-22T23:41:51.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.914+0000 7f0dae438700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0da80ff5a0 0x7f0da8100520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:51.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.914+0000 7f0dae438700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0da80fff50 0x7f0da8100a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:51.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.914+0000 7f0dae438700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da8100fa0 con 0x7f0da80ff5a0 2024-04-22T23:41:51.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.914+0000 7f0dae438700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da81010e0 con 0x7f0da80fff50 2024-04-22T23:41:51.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.914+0000 7f0da7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0da80ff5a0 0x7f0da8100520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:51.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.914+0000 7f0da7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0da80ff5a0 0x7f0da8100520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47156/0 (socket says 172.21.15.67:47156) 2024-04-22T23:41:51.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.914+0000 7f0da7fff700 1 -- 172.21.15.67:0/3699474490 learned_addr learned my addr 172.21.15.67:0/3699474490 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:41:51.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.914+0000 7f0da77fe700 1 --2- 172.21.15.67:0/3699474490 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0da80fff50 0x7f0da8100a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:51.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.914+0000 7f0da7fff700 1 -- 172.21.15.67:0/3699474490 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0da80fff50 msgr2=0x7f0da8100a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:51.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.914+0000 7f0da7fff700 1 --2- 172.21.15.67:0/3699474490 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0da80fff50 0x7f0da8100a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:51.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.914+0000 7f0da7fff700 1 -- 172.21.15.67:0/3699474490 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d9c0095f0 con 0x7f0da80ff5a0 2024-04-22T23:41:51.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.914+0000 7f0da7fff700 1 --2- 172.21.15.67:0/3699474490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0da80ff5a0 0x7f0da8100520 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f0d9800c930 tx=0x7f0d9800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:41:51.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.914+0000 7f0da57fa700 1 -- 172.21.15.67:0/3699474490 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d98007ad0 con 0x7f0da80ff5a0 2024-04-22T23:41:51.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.914+0000 7f0da57fa700 1 -- 172.21.15.67:0/3699474490 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0d9800ce80 con 0x7f0da80ff5a0 2024-04-22T23:41:51.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.914+0000 7f0dae438700 1 -- 172.21.15.67:0/3699474490 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0da81a2a40 con 0x7f0da80ff5a0 2024-04-22T23:41:51.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.914+0000 7f0da57fa700 1 -- 172.21.15.67:0/3699474490 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d98018630 con 0x7f0da80ff5a0 2024-04-22T23:41:51.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.915+0000 7f0dae438700 1 -- 172.21.15.67:0/3699474490 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0da81a2da0 con 0x7f0da80ff5a0 2024-04-22T23:41:51.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.915+0000 7f0dae438700 1 -- 172.21.15.67:0/3699474490 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0da8109200 con 0x7f0da80ff5a0 2024-04-22T23:41:51.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.916+0000 7f0da57fa700 1 -- 172.21.15.67:0/3699474490 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0d98007c30 con 0x7f0da80ff5a0 2024-04-22T23:41:51.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.916+0000 7f0da57fa700 1 --2- 172.21.15.67:0/3699474490 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0d9006ecc0 0x7f0d90071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:51.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.917+0000 7f0da57fa700 1 -- 172.21.15.67:0/3699474490 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f0d9808f010 con 0x7f0da80ff5a0 2024-04-22T23:41:51.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.917+0000 7f0da77fe700 1 --2- 172.21.15.67:0/3699474490 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0d9006ecc0 0x7f0d90071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:51.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.917+0000 7f0da77fe700 1 --2- 172.21.15.67:0/3699474490 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0d9006ecc0 0x7f0d90071180 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0da8101d10 tx=0x7f0d9c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:41:51.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:51.921+0000 7f0da57fa700 1 -- 172.21.15.67:0/3699474490 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f0d98057c50 con 0x7f0da80ff5a0 2024-04-22T23:41:52.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:51 smithi067 ceph-mon[112438]: pgmap v464: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:52.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.140+0000 7f0dae438700 1 -- 172.21.15.67:0/3699474490 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0da80611d0 con 0x7f0d9006ecc0 2024-04-22T23:41:52.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.151+0000 7f0da57fa700 1 -- 172.21.15.67:0/3699474490 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f0da80611d0 con 0x7f0d9006ecc0 2024-04-22T23:41:52.145 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:41:52.145 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (15m) 8m ago 26m 25.5M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:41:52.145 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (27m) 8m ago 27m 9114k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:41:52.145 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (26m) 8m ago 26m 9042k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:41:52.145 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (12m) 8m ago 27m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:41:52.145 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (12m) 8m ago 26m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:41:52.145 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (15m) 8m ago 26m 79.3M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:41:52.145 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (9m) 8m ago 21m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:41:52.146 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (8m) 8m ago 21m 12.6M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:41:52.146 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (8m) 8m ago 21m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:41:52.146 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (21m) 8m ago 21m 12.2M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:41:52.146 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (17m) 8m ago 28m 511M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:41:52.146 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (16m) 8m ago 25m 453M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:41:52.146 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (14m) 8m ago 28m 55.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:41:52.146 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (13m) 8m ago 25m 46.0M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:41:52.146 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (16m) 8m ago 26m 15.4M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:41:52.146 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (16m) 8m ago 25m 16.1M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:41:52.146 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (12m) 8m ago 25m 99.2M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:41:52.146 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (11m) 8m ago 24m 127M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:41:52.146 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (11m) 8m ago 24m 79.9M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:41:52.146 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (10m) 8m ago 23m 97.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:41:52.146 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (10m) 8m ago 23m 82.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:41:52.147 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (9m) 8m ago 22m 77.6M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:41:52.147 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (15m) 8m ago 26m 73.9M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:41:52.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.153+0000 7f0dae438700 1 -- 172.21.15.67:0/3699474490 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0d9006ecc0 msgr2=0x7f0d90071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:52.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.153+0000 7f0dae438700 1 --2- 172.21.15.67:0/3699474490 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0d9006ecc0 0x7f0d90071180 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0da8101d10 tx=0x7f0d9c01a040 comp rx=0 tx=0).stop 2024-04-22T23:41:52.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.153+0000 7f0dae438700 1 -- 172.21.15.67:0/3699474490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0da80ff5a0 msgr2=0x7f0da8100520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:52.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.153+0000 7f0dae438700 1 --2- 172.21.15.67:0/3699474490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0da80ff5a0 0x7f0da8100520 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f0d9800c930 tx=0x7f0d9800cc40 comp rx=0 tx=0).stop 2024-04-22T23:41:52.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.153+0000 7f0dae438700 1 -- 172.21.15.67:0/3699474490 shutdown_connections 2024-04-22T23:41:52.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.153+0000 7f0dae438700 1 --2- 172.21.15.67:0/3699474490 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0d9006ecc0 0x7f0d90071180 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:52.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.153+0000 7f0dae438700 1 --2- 172.21.15.67:0/3699474490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0da80ff5a0 0x7f0da8100520 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:52.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.153+0000 7f0dae438700 1 --2- 172.21.15.67:0/3699474490 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0da80fff50 0x7f0da8100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:52.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.153+0000 7f0dae438700 1 -- 172.21.15.67:0/3699474490 >> 172.21.15.67:0/3699474490 conn(0x7f0da8074e30 msgr2=0x7f0da80fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:41:52.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.153+0000 7f0dae438700 1 -- 172.21.15.67:0/3699474490 shutdown_connections 2024-04-22T23:41:52.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.153+0000 7f0dae438700 1 -- 172.21.15.67:0/3699474490 wait complete. 2024-04-22T23:41:52.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:51 smithi134 ceph-mon[85630]: pgmap v464: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:52.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.260+0000 7f1a46498700 1 -- 172.21.15.67:0/2706644991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a40106db0 msgr2=0x7f1a40107190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:52.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.260+0000 7f1a46498700 1 --2- 172.21.15.67:0/2706644991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a40106db0 0x7f1a40107190 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f1a30009910 tx=0x7f1a30009c20 comp rx=0 tx=0).stop 2024-04-22T23:41:52.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.260+0000 7f1a46498700 1 -- 172.21.15.67:0/2706644991 shutdown_connections 2024-04-22T23:41:52.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.260+0000 7f1a46498700 1 --2- 172.21.15.67:0/2706644991 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a400ff410 0x7f1a400ff890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:52.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.260+0000 7f1a46498700 1 --2- 172.21.15.67:0/2706644991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a40106db0 0x7f1a40107190 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:52.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.260+0000 7f1a46498700 1 -- 172.21.15.67:0/2706644991 >> 172.21.15.67:0/2706644991 conn(0x7f1a40074e30 msgr2=0x7f1a40075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:41:52.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.260+0000 7f1a46498700 1 -- 172.21.15.67:0/2706644991 shutdown_connections 2024-04-22T23:41:52.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.260+0000 7f1a46498700 1 -- 172.21.15.67:0/2706644991 wait complete. 2024-04-22T23:41:52.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.261+0000 7f1a46498700 1 Processor -- start 2024-04-22T23:41:52.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.261+0000 7f1a46498700 1 -- start start 2024-04-22T23:41:52.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.262+0000 7f1a46498700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a400ff410 0x7f1a401005f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:52.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.262+0000 7f1a46498700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a40106db0 0x7f1a40100b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:52.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.262+0000 7f1a46498700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a40101100 con 0x7f1a400ff410 2024-04-22T23:41:52.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.262+0000 7f1a46498700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a40101270 con 0x7f1a40106db0 2024-04-22T23:41:52.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.262+0000 7f1a3ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a400ff410 0x7f1a401005f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:52.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.262+0000 7f1a3ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a400ff410 0x7f1a401005f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47166/0 (socket says 172.21.15.67:47166) 2024-04-22T23:41:52.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.262+0000 7f1a3ffff700 1 -- 172.21.15.67:0/483853499 learned_addr learned my addr 172.21.15.67:0/483853499 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:41:52.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.262+0000 7f1a3f7fe700 1 --2- 172.21.15.67:0/483853499 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a40106db0 0x7f1a40100b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:52.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.262+0000 7f1a3ffff700 1 -- 172.21.15.67:0/483853499 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a40106db0 msgr2=0x7f1a40100b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:52.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.262+0000 7f1a3ffff700 1 --2- 172.21.15.67:0/483853499 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a40106db0 0x7f1a40100b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:52.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.262+0000 7f1a3ffff700 1 -- 172.21.15.67:0/483853499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a300095f0 con 0x7f1a400ff410 2024-04-22T23:41:52.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.262+0000 7f1a3ffff700 1 --2- 172.21.15.67:0/483853499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a400ff410 0x7f1a401005f0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f1a300092e0 tx=0x7f1a3000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:41:52.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.262+0000 7f1a3d7fa700 1 -- 172.21.15.67:0/483853499 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a3001d070 con 0x7f1a400ff410 2024-04-22T23:41:52.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.262+0000 7f1a3d7fa700 1 -- 172.21.15.67:0/483853499 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1a30005e00 con 0x7f1a400ff410 2024-04-22T23:41:52.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.262+0000 7f1a46498700 1 -- 172.21.15.67:0/483853499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a401a2b80 con 0x7f1a400ff410 2024-04-22T23:41:52.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.263+0000 7f1a3d7fa700 1 -- 172.21.15.67:0/483853499 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a30017400 con 0x7f1a400ff410 2024-04-22T23:41:52.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.263+0000 7f1a46498700 1 -- 172.21.15.67:0/483853499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a401a2f30 con 0x7f1a400ff410 2024-04-22T23:41:52.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.263+0000 7f1a46498700 1 -- 172.21.15.67:0/483853499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a40109300 con 0x7f1a400ff410 2024-04-22T23:41:52.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.264+0000 7f1a3d7fa700 1 -- 172.21.15.67:0/483853499 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1a3000f460 con 0x7f1a400ff410 2024-04-22T23:41:52.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.264+0000 7f1a3d7fa700 1 --2- 172.21.15.67:0/483853499 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1a2806ecc0 0x7f1a28071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:52.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.265+0000 7f1a3d7fa700 1 -- 172.21.15.67:0/483853499 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f1a30090400 con 0x7f1a400ff410 2024-04-22T23:41:52.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.265+0000 7f1a3f7fe700 1 --2- 172.21.15.67:0/483853499 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1a2806ecc0 0x7f1a28071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:52.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.265+0000 7f1a3f7fe700 1 --2- 172.21.15.67:0/483853499 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1a2806ecc0 0x7f1a28071180 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1a34005950 tx=0x7f1a340058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:41:52.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.269+0000 7f1a3d7fa700 1 -- 172.21.15.67:0/483853499 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1a300590c0 con 0x7f1a400ff410 2024-04-22T23:41:52.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.568+0000 7f1a46498700 1 -- 172.21.15.67:0/483853499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1a4004f2e0 con 0x7f1a400ff410 2024-04-22T23:41:52.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.569+0000 7f1a3d7fa700 1 -- 172.21.15.67:0/483853499 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f1a3005c780 con 0x7f1a400ff410 2024-04-22T23:41:52.563 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:41:52.563 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:41:52.563 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:41:52.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:41:52.563 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:41:52.563 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:41:52.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:41:52.563 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:41:52.563 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:41:52.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:41:52.564 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:41:52.564 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:41:52.564 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 3 2024-04-22T23:41:52.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:41:52.564 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:41:52.564 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:41:52.564 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 13 2024-04-22T23:41:52.564 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:41:52.564 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:41:52.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.571+0000 7f1a46498700 1 -- 172.21.15.67:0/483853499 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1a2806ecc0 msgr2=0x7f1a28071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:52.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.571+0000 7f1a46498700 1 --2- 172.21.15.67:0/483853499 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1a2806ecc0 0x7f1a28071180 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1a34005950 tx=0x7f1a340058e0 comp rx=0 tx=0).stop 2024-04-22T23:41:52.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.571+0000 7f1a46498700 1 -- 172.21.15.67:0/483853499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a400ff410 msgr2=0x7f1a401005f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:52.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.571+0000 7f1a46498700 1 --2- 172.21.15.67:0/483853499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a400ff410 0x7f1a401005f0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f1a300092e0 tx=0x7f1a3000be20 comp rx=0 tx=0).stop 2024-04-22T23:41:52.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.571+0000 7f1a46498700 1 -- 172.21.15.67:0/483853499 shutdown_connections 2024-04-22T23:41:52.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.571+0000 7f1a46498700 1 --2- 172.21.15.67:0/483853499 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1a2806ecc0 0x7f1a28071180 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:52.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.571+0000 7f1a46498700 1 --2- 172.21.15.67:0/483853499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a400ff410 0x7f1a401005f0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:52.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.571+0000 7f1a46498700 1 --2- 172.21.15.67:0/483853499 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a40106db0 0x7f1a40100b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:52.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.571+0000 7f1a46498700 1 -- 172.21.15.67:0/483853499 >> 172.21.15.67:0/483853499 conn(0x7f1a40074e30 msgr2=0x7f1a400fd420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:41:52.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.571+0000 7f1a46498700 1 -- 172.21.15.67:0/483853499 shutdown_connections 2024-04-22T23:41:52.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.571+0000 7f1a46498700 1 -- 172.21.15.67:0/483853499 wait complete. 2024-04-22T23:41:52.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.678+0000 7f57cc96d700 1 -- 172.21.15.67:0/1033425505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57c8068580 msgr2=0x7f57c80689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:52.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.678+0000 7f57cc96d700 1 --2- 172.21.15.67:0/1033425505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57c8068580 0x7f57c80689e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f57b0009910 tx=0x7f57b0009c20 comp rx=0 tx=0).stop 2024-04-22T23:41:52.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.678+0000 7f57cc96d700 1 -- 172.21.15.67:0/1033425505 shutdown_connections 2024-04-22T23:41:52.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.678+0000 7f57cc96d700 1 --2- 172.21.15.67:0/1033425505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57c8068580 0x7f57c80689e0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:52.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.678+0000 7f57cc96d700 1 --2- 172.21.15.67:0/1033425505 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f57c8106cd0 0x7f57c81070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:52.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.678+0000 7f57cc96d700 1 -- 172.21.15.67:0/1033425505 >> 172.21.15.67:0/1033425505 conn(0x7f57c8075670 msgr2=0x7f57c8075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:41:52.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.678+0000 7f57cc96d700 1 -- 172.21.15.67:0/1033425505 shutdown_connections 2024-04-22T23:41:52.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.678+0000 7f57cc96d700 1 -- 172.21.15.67:0/1033425505 wait complete. 2024-04-22T23:41:52.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.679+0000 7f57cc96d700 1 Processor -- start 2024-04-22T23:41:52.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.679+0000 7f57cc96d700 1 -- start start 2024-04-22T23:41:52.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.679+0000 7f57cc96d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57c8068580 0x7f57c819c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:52.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.679+0000 7f57cc96d700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f57c8106cd0 0x7f57c819cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:52.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.679+0000 7f57cc96d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c819d200 con 0x7f57c8068580 2024-04-22T23:41:52.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.679+0000 7f57cc96d700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c81966b0 con 0x7f57c8106cd0 2024-04-22T23:41:52.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.679+0000 7f57c659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57c8068580 0x7f57c819c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:52.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.679+0000 7f57c659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57c8068580 0x7f57c819c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47182/0 (socket says 172.21.15.67:47182) 2024-04-22T23:41:52.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.679+0000 7f57c659c700 1 -- 172.21.15.67:0/2440524955 learned_addr learned my addr 172.21.15.67:0/2440524955 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:41:52.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.680+0000 7f57c659c700 1 -- 172.21.15.67:0/2440524955 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f57c8106cd0 msgr2=0x7f57c819cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:52.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.680+0000 7f57c659c700 1 --2- 172.21.15.67:0/2440524955 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f57c8106cd0 0x7f57c819cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:52.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.680+0000 7f57c659c700 1 -- 172.21.15.67:0/2440524955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b00095f0 con 0x7f57c8068580 2024-04-22T23:41:52.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.680+0000 7f57c659c700 1 --2- 172.21.15.67:0/2440524955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57c8068580 0x7f57c819c630 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f57c000ef10 tx=0x7f57c000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:41:52.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.680+0000 7f57bf7fe700 1 -- 172.21.15.67:0/2440524955 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57c000ca50 con 0x7f57c8068580 2024-04-22T23:41:52.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.680+0000 7f57cc96d700 1 -- 172.21.15.67:0/2440524955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57c8196930 con 0x7f57c8068580 2024-04-22T23:41:52.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.680+0000 7f57bf7fe700 1 -- 172.21.15.67:0/2440524955 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f57c000f040 con 0x7f57c8068580 2024-04-22T23:41:52.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.680+0000 7f57bf7fe700 1 -- 172.21.15.67:0/2440524955 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57c00146b0 con 0x7f57c8068580 2024-04-22T23:41:52.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.680+0000 7f57cc96d700 1 -- 172.21.15.67:0/2440524955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57c8196e80 con 0x7f57c8068580 2024-04-22T23:41:52.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.681+0000 7f57cc96d700 1 -- 172.21.15.67:0/2440524955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57c81091d0 con 0x7f57c8068580 2024-04-22T23:41:52.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.681+0000 7f57bf7fe700 1 -- 172.21.15.67:0/2440524955 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f57c000cbb0 con 0x7f57c8068580 2024-04-22T23:41:52.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.682+0000 7f57bf7fe700 1 --2- 172.21.15.67:0/2440524955 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f57b406ecc0 0x7f57b4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:52.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.682+0000 7f57bf7fe700 1 -- 172.21.15.67:0/2440524955 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f57c008f3e0 con 0x7f57c8068580 2024-04-22T23:41:52.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.682+0000 7f57c5d9b700 1 --2- 172.21.15.67:0/2440524955 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f57b406ecc0 0x7f57b4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:52.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.682+0000 7f57c5d9b700 1 --2- 172.21.15.67:0/2440524955 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f57b406ecc0 0x7f57b4071180 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f57b0009fd0 tx=0x7f57b0000bc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:41:52.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.687+0000 7f57bf7fe700 1 -- 172.21.15.67:0/2440524955 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f57c005bb50 con 0x7f57c8068580 2024-04-22T23:41:52.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.943+0000 7f57cc96d700 1 -- 172.21.15.67:0/2440524955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f57c804f2e0 con 0x7f57c8068580 2024-04-22T23:41:52.937 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:52 smithi067 ceph-mon[112438]: from='client.34876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:41:52.937 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:52 smithi067 ceph-mon[112438]: from='client.34880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:41:52.937 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:52 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/483853499' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:41:52.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.943+0000 7f57bf7fe700 1 -- 172.21.15.67:0/2440524955 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1871 (secure 0 0 0) 0x7f57c005b6e0 con 0x7f57c8068580 2024-04-22T23:41:52.938 INFO:teuthology.orchestra.run.smithi067.stdout:e31 2024-04-22T23:41:52.938 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:41:52.938 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:41:52.938 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:41:52.938 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:41:52.938 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:41:52.938 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:41:52.938 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:41:52.938 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:41:52.939 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:41:52.939 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:41:52.939 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:41:52.939 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:41:52.939 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:41:52.939 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:41:52.939 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:41:52.939 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:41:52.939 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:41:52.939 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:41:52.939 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:41:52.939 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:41:52.939 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:41:52.939 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:41:52.939 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:41:52.939 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:41:52.939 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:41:52.940 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:41:52.940 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:41:52.940 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:41:52.940 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:41:52.940 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:41:52.940 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:41:52.940 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:41:52.940 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:41:52.940 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:41:52.940 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:41:52.940 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:41:52.940 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:41:52.940 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:41:52.940 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:41:52.940 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44259} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1827288036,v1:172.21.15.134:6827/1827288036] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:41:52.940 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:41:52.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.947+0000 7f57cc96d700 1 -- 172.21.15.67:0/2440524955 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f57b406ecc0 msgr2=0x7f57b4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:52.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.947+0000 7f57cc96d700 1 --2- 172.21.15.67:0/2440524955 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f57b406ecc0 0x7f57b4071180 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f57b0009fd0 tx=0x7f57b0000bc0 comp rx=0 tx=0).stop 2024-04-22T23:41:52.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.947+0000 7f57cc96d700 1 -- 172.21.15.67:0/2440524955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57c8068580 msgr2=0x7f57c819c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:52.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.947+0000 7f57cc96d700 1 --2- 172.21.15.67:0/2440524955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57c8068580 0x7f57c819c630 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f57c000ef10 tx=0x7f57c000c3a0 comp rx=0 tx=0).stop 2024-04-22T23:41:52.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.947+0000 7f57cc96d700 1 -- 172.21.15.67:0/2440524955 shutdown_connections 2024-04-22T23:41:52.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.947+0000 7f57cc96d700 1 --2- 172.21.15.67:0/2440524955 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f57b406ecc0 0x7f57b4071180 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:52.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.947+0000 7f57cc96d700 1 --2- 172.21.15.67:0/2440524955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57c8068580 0x7f57c819c630 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:52.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.947+0000 7f57cc96d700 1 --2- 172.21.15.67:0/2440524955 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f57c8106cd0 0x7f57c819cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:52.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.947+0000 7f57cc96d700 1 -- 172.21.15.67:0/2440524955 >> 172.21.15.67:0/2440524955 conn(0x7f57c8075670 msgr2=0x7f57c80feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:41:52.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.947+0000 7f57cc96d700 1 -- 172.21.15.67:0/2440524955 shutdown_connections 2024-04-22T23:41:52.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:52.947+0000 7f57cc96d700 1 -- 172.21.15.67:0/2440524955 wait complete. 2024-04-22T23:41:52.942 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 31 2024-04-22T23:41:53.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.057+0000 7facb5b10700 1 -- 172.21.15.67:0/1745639794 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7facb0100750 msgr2=0x7facb01047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:53.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.057+0000 7facb5b10700 1 --2- 172.21.15.67:0/1745639794 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7facb0100750 0x7facb01047a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7faca4009910 tx=0x7faca4009c20 comp rx=0 tx=0).stop 2024-04-22T23:41:53.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.057+0000 7facb5b10700 1 -- 172.21.15.67:0/1745639794 shutdown_connections 2024-04-22T23:41:53.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.057+0000 7facb5b10700 1 --2- 172.21.15.67:0/1745639794 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7facb0100750 0x7facb01047a0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:53.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.057+0000 7facb5b10700 1 --2- 172.21.15.67:0/1745639794 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7facb00ffda0 0x7facb0100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:53.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.057+0000 7facb5b10700 1 -- 172.21.15.67:0/1745639794 >> 172.21.15.67:0/1745639794 conn(0x7facb00fb650 msgr2=0x7facb00fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:41:53.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.057+0000 7facb5b10700 1 -- 172.21.15.67:0/1745639794 shutdown_connections 2024-04-22T23:41:53.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.057+0000 7facb5b10700 1 -- 172.21.15.67:0/1745639794 wait complete. 2024-04-22T23:41:53.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.057+0000 7facb5b10700 1 Processor -- start 2024-04-22T23:41:53.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.058+0000 7facb5b10700 1 -- start start 2024-04-22T23:41:53.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.058+0000 7facb5b10700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7facb00ffda0 0x7facb0196950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:53.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.058+0000 7facb5b10700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7facb0100750 0x7facb0196e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:53.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.058+0000 7facb5b10700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb0197520 con 0x7facb0100750 2024-04-22T23:41:53.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.058+0000 7facb5b10700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb019b330 con 0x7facb00ffda0 2024-04-22T23:41:53.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.058+0000 7facaeffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7facb0100750 0x7facb0196e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:53.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.058+0000 7facaeffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7facb0100750 0x7facb0196e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47206/0 (socket says 172.21.15.67:47206) 2024-04-22T23:41:53.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.058+0000 7facaeffd700 1 -- 172.21.15.67:0/3158155574 learned_addr learned my addr 172.21.15.67:0/3158155574 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:41:53.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.058+0000 7facaf7fe700 1 --2- 172.21.15.67:0/3158155574 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7facb00ffda0 0x7facb0196950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:53.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.058+0000 7facaeffd700 1 -- 172.21.15.67:0/3158155574 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7facb00ffda0 msgr2=0x7facb0196950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:53.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.058+0000 7facaeffd700 1 --2- 172.21.15.67:0/3158155574 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7facb00ffda0 0x7facb0196950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:53.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.058+0000 7facaeffd700 1 -- 172.21.15.67:0/3158155574 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faca40095f0 con 0x7facb0100750 2024-04-22T23:41:53.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.058+0000 7facaf7fe700 1 --2- 172.21.15.67:0/3158155574 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7facb00ffda0 0x7facb0196950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:41:53.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.059+0000 7facaeffd700 1 --2- 172.21.15.67:0/3158155574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7facb0100750 0x7facb0196e90 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7faca40092e0 tx=0x7faca400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:41:53.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.059+0000 7facacff9700 1 -- 172.21.15.67:0/3158155574 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faca401d070 con 0x7facb0100750 2024-04-22T23:41:53.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.059+0000 7facacff9700 1 -- 172.21.15.67:0/3158155574 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faca4005dc0 con 0x7facb0100750 2024-04-22T23:41:53.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.059+0000 7facacff9700 1 -- 172.21.15.67:0/3158155574 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faca4017400 con 0x7facb0100750 2024-04-22T23:41:53.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.059+0000 7facb5b10700 1 -- 172.21.15.67:0/3158155574 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facb019b5b0 con 0x7facb0100750 2024-04-22T23:41:53.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.059+0000 7facb5b10700 1 -- 172.21.15.67:0/3158155574 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facb019baa0 con 0x7facb0100750 2024-04-22T23:41:53.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.060+0000 7facb5b10700 1 -- 172.21.15.67:0/3158155574 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facb004f2e0 con 0x7facb0100750 2024-04-22T23:41:53.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.060+0000 7facacff9700 1 -- 172.21.15.67:0/3158155574 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7faca400f460 con 0x7facb0100750 2024-04-22T23:41:53.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.061+0000 7facacff9700 1 --2- 172.21.15.67:0/3158155574 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fac9806ecc0 0x7fac98071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:53.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.061+0000 7facacff9700 1 -- 172.21.15.67:0/3158155574 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7faca4090430 con 0x7facb0100750 2024-04-22T23:41:53.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.061+0000 7facaf7fe700 1 --2- 172.21.15.67:0/3158155574 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fac9806ecc0 0x7fac98071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:53.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.061+0000 7facaf7fe700 1 --2- 172.21.15.67:0/3158155574 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fac9806ecc0 0x7fac98071180 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7faca0005fd0 tx=0x7faca0005ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:41:53.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.066+0000 7facacff9700 1 -- 172.21.15.67:0/3158155574 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7faca405cc20 con 0x7facb0100750 2024-04-22T23:41:53.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:52 smithi134 ceph-mon[85630]: from='client.34876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:41:53.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:52 smithi134 ceph-mon[85630]: from='client.34880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:41:53.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:52 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/483853499' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:41:53.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.295+0000 7facb5b10700 1 -- 172.21.15.67:0/3158155574 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7facb019bd10 con 0x7fac9806ecc0 2024-04-22T23:41:53.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.297+0000 7facacff9700 1 -- 172.21.15.67:0/3158155574 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7facb019bd10 con 0x7fac9806ecc0 2024-04-22T23:41:53.291 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:41:53.291 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:41:53.291 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:41:53.291 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:41:53.291 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:41:53.291 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:41:53.291 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:41:53.291 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:41:53.291 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:41:53.291 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:41:53.291 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T23:41:53.291 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:41:53.291 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:41:53.291 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:41:53.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.299+0000 7facb5b10700 1 -- 172.21.15.67:0/3158155574 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fac9806ecc0 msgr2=0x7fac98071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:53.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.299+0000 7facb5b10700 1 --2- 172.21.15.67:0/3158155574 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fac9806ecc0 0x7fac98071180 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7faca0005fd0 tx=0x7faca0005ee0 comp rx=0 tx=0).stop 2024-04-22T23:41:53.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.299+0000 7facb5b10700 1 -- 172.21.15.67:0/3158155574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7facb0100750 msgr2=0x7facb0196e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:53.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.299+0000 7facb5b10700 1 --2- 172.21.15.67:0/3158155574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7facb0100750 0x7facb0196e90 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7faca40092e0 tx=0x7faca400be20 comp rx=0 tx=0).stop 2024-04-22T23:41:53.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.299+0000 7facb5b10700 1 -- 172.21.15.67:0/3158155574 shutdown_connections 2024-04-22T23:41:53.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.299+0000 7facb5b10700 1 --2- 172.21.15.67:0/3158155574 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fac9806ecc0 0x7fac98071180 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:53.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.299+0000 7facb5b10700 1 --2- 172.21.15.67:0/3158155574 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7facb00ffda0 0x7facb0196950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:53.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.299+0000 7facb5b10700 1 --2- 172.21.15.67:0/3158155574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7facb0100750 0x7facb0196e90 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:53.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.299+0000 7facb5b10700 1 -- 172.21.15.67:0/3158155574 >> 172.21.15.67:0/3158155574 conn(0x7facb00fb650 msgr2=0x7facb0103fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:41:53.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.299+0000 7facb5b10700 1 -- 172.21.15.67:0/3158155574 shutdown_connections 2024-04-22T23:41:53.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.299+0000 7facb5b10700 1 -- 172.21.15.67:0/3158155574 wait complete. 2024-04-22T23:41:53.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.408+0000 7fd52f00a700 1 -- 172.21.15.67:0/1468475959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd528101850 msgr2=0x7fd5281058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:53.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.408+0000 7fd52f00a700 1 --2- 172.21.15.67:0/1468475959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd528101850 0x7fd5281058a0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fd514009910 tx=0x7fd514009c20 comp rx=0 tx=0).stop 2024-04-22T23:41:53.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.408+0000 7fd52f00a700 1 -- 172.21.15.67:0/1468475959 shutdown_connections 2024-04-22T23:41:53.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.408+0000 7fd52f00a700 1 --2- 172.21.15.67:0/1468475959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd528101850 0x7fd5281058a0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:53.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.408+0000 7fd52f00a700 1 --2- 172.21.15.67:0/1468475959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd528100ea0 0x7fd528101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:53.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.408+0000 7fd52f00a700 1 -- 172.21.15.67:0/1468475959 >> 172.21.15.67:0/1468475959 conn(0x7fd5280fc750 msgr2=0x7fd5280feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:41:53.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.408+0000 7fd52f00a700 1 -- 172.21.15.67:0/1468475959 shutdown_connections 2024-04-22T23:41:53.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.408+0000 7fd52f00a700 1 -- 172.21.15.67:0/1468475959 wait complete. 2024-04-22T23:41:53.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.409+0000 7fd52f00a700 1 Processor -- start 2024-04-22T23:41:53.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.409+0000 7fd52f00a700 1 -- start start 2024-04-22T23:41:53.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.409+0000 7fd52f00a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd528100ea0 0x7fd52819c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:53.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.409+0000 7fd52f00a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd528101850 0x7fd52819cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:53.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.409+0000 7fd52f00a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd52819d2d0 con 0x7fd528101850 2024-04-22T23:41:53.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.409+0000 7fd52f00a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd528196780 con 0x7fd528100ea0 2024-04-22T23:41:53.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.409+0000 7fd527fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd528101850 0x7fd52819cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:53.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.409+0000 7fd527fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd528101850 0x7fd52819cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47234/0 (socket says 172.21.15.67:47234) 2024-04-22T23:41:53.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.409+0000 7fd527fff700 1 -- 172.21.15.67:0/2867765787 learned_addr learned my addr 172.21.15.67:0/2867765787 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:41:53.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.409+0000 7fd52cda6700 1 --2- 172.21.15.67:0/2867765787 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd528100ea0 0x7fd52819c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:53.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.409+0000 7fd527fff700 1 -- 172.21.15.67:0/2867765787 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd528100ea0 msgr2=0x7fd52819c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:53.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.409+0000 7fd527fff700 1 --2- 172.21.15.67:0/2867765787 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd528100ea0 0x7fd52819c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:53.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.409+0000 7fd527fff700 1 -- 172.21.15.67:0/2867765787 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5140095f0 con 0x7fd528101850 2024-04-22T23:41:53.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.409+0000 7fd52cda6700 1 --2- 172.21.15.67:0/2867765787 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd528100ea0 0x7fd52819c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:41:53.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.410+0000 7fd527fff700 1 --2- 172.21.15.67:0/2867765787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd528101850 0x7fd52819cc40 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd514009fd0 tx=0x7fd51400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:41:53.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.410+0000 7fd525ffb700 1 -- 172.21.15.67:0/2867765787 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd51401d070 con 0x7fd528101850 2024-04-22T23:41:53.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.410+0000 7fd525ffb700 1 -- 172.21.15.67:0/2867765787 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd514022470 con 0x7fd528101850 2024-04-22T23:41:53.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.410+0000 7fd52f00a700 1 -- 172.21.15.67:0/2867765787 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd528196a00 con 0x7fd528101850 2024-04-22T23:41:53.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.410+0000 7fd525ffb700 1 -- 172.21.15.67:0/2867765787 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd514017610 con 0x7fd528101850 2024-04-22T23:41:53.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.410+0000 7fd52f00a700 1 -- 172.21.15.67:0/2867765787 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd528196e90 con 0x7fd528101850 2024-04-22T23:41:53.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.411+0000 7fd52f00a700 1 -- 172.21.15.67:0/2867765787 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5281092a0 con 0x7fd528101850 2024-04-22T23:41:53.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.412+0000 7fd525ffb700 1 -- 172.21.15.67:0/2867765787 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd5140225e0 con 0x7fd528101850 2024-04-22T23:41:53.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.412+0000 7fd525ffb700 1 --2- 172.21.15.67:0/2867765787 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd51806ecc0 0x7fd518071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:41:53.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.412+0000 7fd525ffb700 1 -- 172.21.15.67:0/2867765787 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fd5140905e0 con 0x7fd528101850 2024-04-22T23:41:53.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.412+0000 7fd52cda6700 1 --2- 172.21.15.67:0/2867765787 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd51806ecc0 0x7fd518071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:41:53.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.413+0000 7fd52cda6700 1 --2- 172.21.15.67:0/2867765787 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd51806ecc0 0x7fd518071180 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd51c009cb0 tx=0x7fd51c009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:41:53.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.417+0000 7fd525ffb700 1 -- 172.21.15.67:0/2867765787 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fd51405cd50 con 0x7fd528101850 2024-04-22T23:41:53.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.714+0000 7fd52f00a700 1 -- 172.21.15.67:0/2867765787 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fd52804f2e0 con 0x7fd528101850 2024-04-22T23:41:53.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.715+0000 7fd525ffb700 1 -- 172.21.15.67:0/2867765787 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fd51405c8e0 con 0x7fd528101850 2024-04-22T23:41:53.709 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:41:53.709 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:41:53.709 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:41:53.709 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:41:53.709 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:41:53.709 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:41:53.709 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:41:53.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.718+0000 7fd52f00a700 1 -- 172.21.15.67:0/2867765787 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd51806ecc0 msgr2=0x7fd518071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:53.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.718+0000 7fd52f00a700 1 --2- 172.21.15.67:0/2867765787 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd51806ecc0 0x7fd518071180 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd51c009cb0 tx=0x7fd51c009260 comp rx=0 tx=0).stop 2024-04-22T23:41:53.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.718+0000 7fd52f00a700 1 -- 172.21.15.67:0/2867765787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd528101850 msgr2=0x7fd52819cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:41:53.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.718+0000 7fd52f00a700 1 --2- 172.21.15.67:0/2867765787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd528101850 0x7fd52819cc40 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd514009fd0 tx=0x7fd51400f690 comp rx=0 tx=0).stop 2024-04-22T23:41:53.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.718+0000 7fd52f00a700 1 -- 172.21.15.67:0/2867765787 shutdown_connections 2024-04-22T23:41:53.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.718+0000 7fd52f00a700 1 --2- 172.21.15.67:0/2867765787 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd51806ecc0 0x7fd518071180 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:53.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.718+0000 7fd52f00a700 1 --2- 172.21.15.67:0/2867765787 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd528100ea0 0x7fd52819c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:53.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.718+0000 7fd52f00a700 1 --2- 172.21.15.67:0/2867765787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd528101850 0x7fd52819cc40 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:41:53.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.718+0000 7fd52f00a700 1 -- 172.21.15.67:0/2867765787 >> 172.21.15.67:0/2867765787 conn(0x7fd5280fc750 msgr2=0x7fd5280fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:41:53.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.718+0000 7fd52f00a700 1 -- 172.21.15.67:0/2867765787 shutdown_connections 2024-04-22T23:41:53.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:41:53.718+0000 7fd52f00a700 1 -- 172.21.15.67:0/2867765787 wait complete. 2024-04-22T23:41:54.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:53 smithi067 ceph-mon[112438]: from='client.34884 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:41:54.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:53 smithi067 ceph-mon[112438]: pgmap v465: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:54.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:53 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2440524955' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:41:54.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:53 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2867765787' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:41:54.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:53 smithi134 ceph-mon[85630]: from='client.34884 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:41:54.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:53 smithi134 ceph-mon[85630]: pgmap v465: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:54.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:53 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2440524955' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:41:54.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:53 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2867765787' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:41:55.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:54 smithi067 ceph-mon[112438]: from='client.34896 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:41:55.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:54 smithi134 ceph-mon[85630]: from='client.34896 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:41:56.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:55 smithi067 ceph-mon[112438]: pgmap v466: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:56.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:55 smithi134 ceph-mon[85630]: pgmap v466: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:57.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:41:57.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:41:58.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:57 smithi067 ceph-mon[112438]: pgmap v467: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:41:58.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:57 smithi134 ceph-mon[85630]: pgmap v467: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:00.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:41:59 smithi067 ceph-mon[112438]: pgmap v468: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:00.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:41:59 smithi134 ceph-mon[85630]: pgmap v468: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:02.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:01 smithi134 ceph-mon[85630]: pgmap v469: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:02.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:01 smithi067 ceph-mon[112438]: pgmap v469: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:04.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:03 smithi134 ceph-mon[85630]: pgmap v470: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:04.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:03 smithi067 ceph-mon[112438]: pgmap v470: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:06.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:05 smithi134 ceph-mon[85630]: pgmap v471: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:06.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:05 smithi067 ceph-mon[112438]: pgmap v471: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:08.076 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:07 smithi067 ceph-mon[112438]: pgmap v472: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:08.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:07 smithi134 ceph-mon[85630]: pgmap v472: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:09.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:09 smithi134 ceph-mon[85630]: pgmap v473: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:10.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:09 smithi067 ceph-mon[112438]: pgmap v473: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:12.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:11 smithi134 ceph-mon[85630]: pgmap v474: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:12.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:42:12.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:11 smithi067 ceph-mon[112438]: pgmap v474: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:12.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:42:14.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:13 smithi134 ceph-mon[85630]: pgmap v475: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:14.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:13 smithi067 ceph-mon[112438]: pgmap v475: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:16.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:15 smithi134 ceph-mon[85630]: pgmap v476: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:16.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:15 smithi067 ceph-mon[112438]: pgmap v476: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:18.075 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:17 smithi067 ceph-mon[112438]: pgmap v477: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:18.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:17 smithi134 ceph-mon[85630]: pgmap v477: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:20.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:19 smithi134 ceph-mon[85630]: pgmap v478: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:20.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:19 smithi067 ceph-mon[112438]: pgmap v478: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:22.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:21 smithi134 ceph-mon[85630]: pgmap v479: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:22.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:21 smithi067 ceph-mon[112438]: pgmap v479: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:23.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:23.835+0000 7f0f5fa70700 1 -- 172.21.15.67:0/3307538407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f58101850 msgr2=0x7f0f581058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:23.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:23.835+0000 7f0f5fa70700 1 --2- 172.21.15.67:0/3307538407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f58101850 0x7f0f581058a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c009910 tx=0x7f0f4c009c20 comp rx=0 tx=0).stop 2024-04-22T23:42:23.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:23.835+0000 7f0f5fa70700 1 -- 172.21.15.67:0/3307538407 shutdown_connections 2024-04-22T23:42:23.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:23.835+0000 7f0f5fa70700 1 --2- 172.21.15.67:0/3307538407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f58101850 0x7f0f581058a0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:23.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:23.835+0000 7f0f5fa70700 1 --2- 172.21.15.67:0/3307538407 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0f58100ea0 0x7f0f58101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:23.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:23.835+0000 7f0f5fa70700 1 -- 172.21.15.67:0/3307538407 >> 172.21.15.67:0/3307538407 conn(0x7f0f580fc750 msgr2=0x7f0f580feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:42:23.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:23.835+0000 7f0f5fa70700 1 -- 172.21.15.67:0/3307538407 shutdown_connections 2024-04-22T23:42:23.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:23.835+0000 7f0f5fa70700 1 -- 172.21.15.67:0/3307538407 wait complete. 2024-04-22T23:42:23.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:23.836+0000 7f0f5fa70700 1 Processor -- start 2024-04-22T23:42:23.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:23.836+0000 7f0f5fa70700 1 -- start start 2024-04-22T23:42:23.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:23.836+0000 7f0f5fa70700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f58100ea0 0x7f0f5819c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:23.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:23.836+0000 7f0f5fa70700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0f58101850 0x7f0f5819cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:23.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:23.836+0000 7f0f5fa70700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5819d210 con 0x7f0f58100ea0 2024-04-22T23:42:23.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:23.836+0000 7f0f5fa70700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f581966c0 con 0x7f0f58101850 2024-04-22T23:42:23.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:23.836+0000 7f0f5d80c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f58100ea0 0x7f0f5819c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:23.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:23.836+0000 7f0f5d80c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f58100ea0 0x7f0f5819c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47424/0 (socket says 172.21.15.67:47424) 2024-04-22T23:42:23.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:23.836+0000 7f0f5d80c700 1 -- 172.21.15.67:0/916687711 learned_addr learned my addr 172.21.15.67:0/916687711 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:42:23.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:23.836+0000 7f0f5d00b700 1 --2- 172.21.15.67:0/916687711 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0f58101850 0x7f0f5819cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:23.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:23.836+0000 7f0f5d80c700 1 -- 172.21.15.67:0/916687711 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0f58101850 msgr2=0x7f0f5819cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:23.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:23.836+0000 7f0f5d80c700 1 --2- 172.21.15.67:0/916687711 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0f58101850 0x7f0f5819cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:23.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:23.836+0000 7f0f5d80c700 1 -- 172.21.15.67:0/916687711 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f4c0095f0 con 0x7f0f58100ea0 2024-04-22T23:42:23.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:23.837+0000 7f0f5d80c700 1 --2- 172.21.15.67:0/916687711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f58100ea0 0x7f0f5819c640 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f0f4800ef10 tx=0x7f0f4800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:42:23.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:23.837+0000 7f0f56ffd700 1 -- 172.21.15.67:0/916687711 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f4800c9f0 con 0x7f0f58100ea0 2024-04-22T23:42:23.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:23.837+0000 7f0f56ffd700 1 -- 172.21.15.67:0/916687711 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0f4800f040 con 0x7f0f58100ea0 2024-04-22T23:42:23.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:23.837+0000 7f0f56ffd700 1 -- 172.21.15.67:0/916687711 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f480146b0 con 0x7f0f58100ea0 2024-04-22T23:42:23.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:23.837+0000 7f0f5fa70700 1 -- 172.21.15.67:0/916687711 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f58196940 con 0x7f0f58100ea0 2024-04-22T23:42:23.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:23.837+0000 7f0f5fa70700 1 -- 172.21.15.67:0/916687711 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f58196e60 con 0x7f0f58100ea0 2024-04-22T23:42:23.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:23.838+0000 7f0f5fa70700 1 -- 172.21.15.67:0/916687711 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f581092a0 con 0x7f0f58100ea0 2024-04-22T23:42:23.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:23.838+0000 7f0f56ffd700 1 -- 172.21.15.67:0/916687711 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0f4800cb50 con 0x7f0f58100ea0 2024-04-22T23:42:23.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:23.839+0000 7f0f56ffd700 1 --2- 172.21.15.67:0/916687711 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0f4006ecc0 0x7f0f40071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:23.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:23.839+0000 7f0f56ffd700 1 -- 172.21.15.67:0/916687711 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f0f4808f260 con 0x7f0f58100ea0 2024-04-22T23:42:23.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:23.839+0000 7f0f5d00b700 1 --2- 172.21.15.67:0/916687711 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0f4006ecc0 0x7f0f40071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:23.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:23.839+0000 7f0f5d00b700 1 --2- 172.21.15.67:0/916687711 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0f4006ecc0 0x7f0f40071180 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c0059d0 tx=0x7f0f4c005940 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:42:23.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:23.844+0000 7f0f56ffd700 1 -- 172.21.15.67:0/916687711 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f0f4805ba50 con 0x7f0f58100ea0 2024-04-22T23:42:24.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:23 smithi067 ceph-mon[112438]: pgmap v480: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:24.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.073+0000 7f0f5fa70700 1 -- 172.21.15.67:0/916687711 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0f581970d0 con 0x7f0f4006ecc0 2024-04-22T23:42:24.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.075+0000 7f0f56ffd700 1 -- 172.21.15.67:0/916687711 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f0f581970d0 con 0x7f0f4006ecc0 2024-04-22T23:42:24.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.077+0000 7f0f5fa70700 1 -- 172.21.15.67:0/916687711 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0f4006ecc0 msgr2=0x7f0f40071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:24.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.077+0000 7f0f5fa70700 1 --2- 172.21.15.67:0/916687711 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0f4006ecc0 0x7f0f40071180 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c0059d0 tx=0x7f0f4c005940 comp rx=0 tx=0).stop 2024-04-22T23:42:24.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.077+0000 7f0f5fa70700 1 -- 172.21.15.67:0/916687711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f58100ea0 msgr2=0x7f0f5819c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:24.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.077+0000 7f0f5fa70700 1 --2- 172.21.15.67:0/916687711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f58100ea0 0x7f0f5819c640 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f0f4800ef10 tx=0x7f0f4800c3a0 comp rx=0 tx=0).stop 2024-04-22T23:42:24.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.077+0000 7f0f5fa70700 1 -- 172.21.15.67:0/916687711 shutdown_connections 2024-04-22T23:42:24.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.077+0000 7f0f5fa70700 1 --2- 172.21.15.67:0/916687711 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0f4006ecc0 0x7f0f40071180 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:24.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.077+0000 7f0f5fa70700 1 --2- 172.21.15.67:0/916687711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f58100ea0 0x7f0f5819c640 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:24.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.077+0000 7f0f5fa70700 1 --2- 172.21.15.67:0/916687711 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0f58101850 0x7f0f5819cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:24.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.077+0000 7f0f5fa70700 1 -- 172.21.15.67:0/916687711 >> 172.21.15.67:0/916687711 conn(0x7f0f580fc750 msgr2=0x7f0f580feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:42:24.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.077+0000 7f0f5fa70700 1 -- 172.21.15.67:0/916687711 shutdown_connections 2024-04-22T23:42:24.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.077+0000 7f0f5fa70700 1 -- 172.21.15.67:0/916687711 wait complete. 2024-04-22T23:42:24.080 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:42:24.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.185+0000 7fbfe8fde700 1 -- 172.21.15.67:0/566075125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfe40fff50 msgr2=0x7fbfe4105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:24.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.185+0000 7fbfe8fde700 1 --2- 172.21.15.67:0/566075125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfe40fff50 0x7fbfe4105840 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fbfd8009910 tx=0x7fbfd8009c20 comp rx=0 tx=0).stop 2024-04-22T23:42:24.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.185+0000 7fbfe8fde700 1 -- 172.21.15.67:0/566075125 shutdown_connections 2024-04-22T23:42:24.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.185+0000 7fbfe8fde700 1 --2- 172.21.15.67:0/566075125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfe40fff50 0x7fbfe4105840 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:24.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.185+0000 7fbfe8fde700 1 --2- 172.21.15.67:0/566075125 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbfe40ff5a0 0x7fbfe40ff980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:24.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.185+0000 7fbfe8fde700 1 -- 172.21.15.67:0/566075125 >> 172.21.15.67:0/566075125 conn(0x7fbfe4074e30 msgr2=0x7fbfe4075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:42:24.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.185+0000 7fbfe8fde700 1 -- 172.21.15.67:0/566075125 shutdown_connections 2024-04-22T23:42:24.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.185+0000 7fbfe8fde700 1 -- 172.21.15.67:0/566075125 wait complete. 2024-04-22T23:42:24.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.186+0000 7fbfe8fde700 1 Processor -- start 2024-04-22T23:42:24.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.186+0000 7fbfe8fde700 1 -- start start 2024-04-22T23:42:24.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.186+0000 7fbfe8fde700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbfe40ff5a0 0x7fbfe419c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:24.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.186+0000 7fbfe8fde700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfe40fff50 0x7fbfe419cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:24.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.186+0000 7fbfe8fde700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe419d190 con 0x7fbfe40fff50 2024-04-22T23:42:24.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.186+0000 7fbfe8fde700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe41966d0 con 0x7fbfe40ff5a0 2024-04-22T23:42:24.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.186+0000 7fbfe1d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfe40fff50 0x7fbfe419cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:24.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.186+0000 7fbfe1d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfe40fff50 0x7fbfe419cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47430/0 (socket says 172.21.15.67:47430) 2024-04-22T23:42:24.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.186+0000 7fbfe1d9b700 1 -- 172.21.15.67:0/3128053439 learned_addr learned my addr 172.21.15.67:0/3128053439 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:42:24.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.186+0000 7fbfe259c700 1 --2- 172.21.15.67:0/3128053439 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbfe40ff5a0 0x7fbfe419c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:24.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.186+0000 7fbfe1d9b700 1 -- 172.21.15.67:0/3128053439 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbfe40ff5a0 msgr2=0x7fbfe419c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:24.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.186+0000 7fbfe1d9b700 1 --2- 172.21.15.67:0/3128053439 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbfe40ff5a0 0x7fbfe419c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:24.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.186+0000 7fbfe1d9b700 1 -- 172.21.15.67:0/3128053439 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd80095f0 con 0x7fbfe40fff50 2024-04-22T23:42:24.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.186+0000 7fbfe1d9b700 1 --2- 172.21.15.67:0/3128053439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfe40fff50 0x7fbfe419cb90 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fbfd80092e0 tx=0x7fbfd800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:42:24.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.187+0000 7fbfd37fe700 1 -- 172.21.15.67:0/3128053439 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbfd801d070 con 0x7fbfe40fff50 2024-04-22T23:42:24.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.187+0000 7fbfd37fe700 1 -- 172.21.15.67:0/3128053439 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbfd800f460 con 0x7fbfe40fff50 2024-04-22T23:42:24.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.187+0000 7fbfe8fde700 1 -- 172.21.15.67:0/3128053439 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfe4196950 con 0x7fbfe40fff50 2024-04-22T23:42:24.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.187+0000 7fbfd37fe700 1 -- 172.21.15.67:0/3128053439 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbfd80175f0 con 0x7fbfe40fff50 2024-04-22T23:42:24.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.187+0000 7fbfe8fde700 1 -- 172.21.15.67:0/3128053439 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfe4196e40 con 0x7fbfe40fff50 2024-04-22T23:42:24.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.188+0000 7fbfe8fde700 1 -- 172.21.15.67:0/3128053439 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfe4109200 con 0x7fbfe40fff50 2024-04-22T23:42:24.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.188+0000 7fbfd37fe700 1 -- 172.21.15.67:0/3128053439 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbfd800f5d0 con 0x7fbfe40fff50 2024-04-22T23:42:24.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.189+0000 7fbfd37fe700 1 --2- 172.21.15.67:0/3128053439 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbfcc06ecc0 0x7fbfcc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:24.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.189+0000 7fbfd37fe700 1 -- 172.21.15.67:0/3128053439 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fbfd8090450 con 0x7fbfe40fff50 2024-04-22T23:42:24.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.189+0000 7fbfe259c700 1 --2- 172.21.15.67:0/3128053439 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbfcc06ecc0 0x7fbfcc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:24.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.189+0000 7fbfe259c700 1 --2- 172.21.15.67:0/3128053439 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbfcc06ecc0 0x7fbfcc071180 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4005ea0 tx=0x7fbfd4005e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:42:24.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.193+0000 7fbfd37fe700 1 -- 172.21.15.67:0/3128053439 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fbfd805cc40 con 0x7fbfe40fff50 2024-04-22T23:42:24.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:23 smithi134 ceph-mon[85630]: pgmap v480: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:24.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.427+0000 7fbfe8fde700 1 -- 172.21.15.67:0/3128053439 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbfe41970b0 con 0x7fbfcc06ecc0 2024-04-22T23:42:24.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.428+0000 7fbfd37fe700 1 -- 172.21.15.67:0/3128053439 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fbfe41970b0 con 0x7fbfcc06ecc0 2024-04-22T23:42:24.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.430+0000 7fbfe8fde700 1 -- 172.21.15.67:0/3128053439 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbfcc06ecc0 msgr2=0x7fbfcc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:24.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.430+0000 7fbfe8fde700 1 --2- 172.21.15.67:0/3128053439 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbfcc06ecc0 0x7fbfcc071180 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4005ea0 tx=0x7fbfd4005e30 comp rx=0 tx=0).stop 2024-04-22T23:42:24.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.430+0000 7fbfe8fde700 1 -- 172.21.15.67:0/3128053439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfe40fff50 msgr2=0x7fbfe419cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:24.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.430+0000 7fbfe8fde700 1 --2- 172.21.15.67:0/3128053439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfe40fff50 0x7fbfe419cb90 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fbfd80092e0 tx=0x7fbfd800be20 comp rx=0 tx=0).stop 2024-04-22T23:42:24.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.430+0000 7fbfe8fde700 1 -- 172.21.15.67:0/3128053439 shutdown_connections 2024-04-22T23:42:24.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.430+0000 7fbfe8fde700 1 --2- 172.21.15.67:0/3128053439 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbfcc06ecc0 0x7fbfcc071180 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:24.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.430+0000 7fbfe8fde700 1 --2- 172.21.15.67:0/3128053439 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbfe40ff5a0 0x7fbfe419c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:24.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.430+0000 7fbfe8fde700 1 --2- 172.21.15.67:0/3128053439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfe40fff50 0x7fbfe419cb90 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:24.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.430+0000 7fbfe8fde700 1 -- 172.21.15.67:0/3128053439 >> 172.21.15.67:0/3128053439 conn(0x7fbfe4074e30 msgr2=0x7fbfe40fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:42:24.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.430+0000 7fbfe8fde700 1 -- 172.21.15.67:0/3128053439 shutdown_connections 2024-04-22T23:42:24.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.430+0000 7fbfe8fde700 1 -- 172.21.15.67:0/3128053439 wait complete. 2024-04-22T23:42:24.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.537+0000 7f96e8c28700 1 -- 172.21.15.67:0/1250456407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96e40ff5a0 msgr2=0x7f96e40ff980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:24.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.537+0000 7f96e8c28700 1 --2- 172.21.15.67:0/1250456407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96e40ff5a0 0x7f96e40ff980 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f96d4009910 tx=0x7f96d4009c20 comp rx=0 tx=0).stop 2024-04-22T23:42:24.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.538+0000 7f96e8c28700 1 -- 172.21.15.67:0/1250456407 shutdown_connections 2024-04-22T23:42:24.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.538+0000 7f96e8c28700 1 --2- 172.21.15.67:0/1250456407 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f96e40fff50 0x7f96e4105840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:24.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.538+0000 7f96e8c28700 1 --2- 172.21.15.67:0/1250456407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96e40ff5a0 0x7f96e40ff980 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:24.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.538+0000 7f96e8c28700 1 -- 172.21.15.67:0/1250456407 >> 172.21.15.67:0/1250456407 conn(0x7f96e4074e30 msgr2=0x7f96e4075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:42:24.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.538+0000 7f96e8c28700 1 -- 172.21.15.67:0/1250456407 shutdown_connections 2024-04-22T23:42:24.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.538+0000 7f96e8c28700 1 -- 172.21.15.67:0/1250456407 wait complete. 2024-04-22T23:42:24.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.538+0000 7f96e8c28700 1 Processor -- start 2024-04-22T23:42:24.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.538+0000 7f96e8c28700 1 -- start start 2024-04-22T23:42:24.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.539+0000 7f96e8c28700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96e40ff5a0 0x7f96e419c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:24.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.539+0000 7f96e8c28700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f96e40fff50 0x7f96e419cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:24.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.539+0000 7f96e8c28700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e419d3b0 con 0x7f96e40ff5a0 2024-04-22T23:42:24.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.539+0000 7f96e8c28700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e4196860 con 0x7f96e40fff50 2024-04-22T23:42:24.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.539+0000 7f96e259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96e40ff5a0 0x7f96e419c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:24.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.539+0000 7f96e259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96e40ff5a0 0x7f96e419c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47450/0 (socket says 172.21.15.67:47450) 2024-04-22T23:42:24.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.539+0000 7f96e259c700 1 -- 172.21.15.67:0/1029094382 learned_addr learned my addr 172.21.15.67:0/1029094382 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:42:24.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.539+0000 7f96e1d9b700 1 --2- 172.21.15.67:0/1029094382 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f96e40fff50 0x7f96e419cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:24.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.539+0000 7f96e259c700 1 -- 172.21.15.67:0/1029094382 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f96e40fff50 msgr2=0x7f96e419cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:24.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.539+0000 7f96e259c700 1 --2- 172.21.15.67:0/1029094382 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f96e40fff50 0x7f96e419cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:24.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.539+0000 7f96e259c700 1 -- 172.21.15.67:0/1029094382 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96d40095f0 con 0x7f96e40ff5a0 2024-04-22T23:42:24.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.539+0000 7f96e259c700 1 --2- 172.21.15.67:0/1029094382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96e40ff5a0 0x7f96e419c7e0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f96d4006010 tx=0x7f96d400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:42:24.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.539+0000 7f96d37fe700 1 -- 172.21.15.67:0/1029094382 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96d401d070 con 0x7f96e40ff5a0 2024-04-22T23:42:24.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.539+0000 7f96e8c28700 1 -- 172.21.15.67:0/1029094382 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96e4196ae0 con 0x7f96e40ff5a0 2024-04-22T23:42:24.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.539+0000 7f96d37fe700 1 -- 172.21.15.67:0/1029094382 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f96d4004fb0 con 0x7f96e40ff5a0 2024-04-22T23:42:24.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.539+0000 7f96d37fe700 1 -- 172.21.15.67:0/1029094382 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96d400f690 con 0x7f96e40ff5a0 2024-04-22T23:42:24.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.539+0000 7f96e8c28700 1 -- 172.21.15.67:0/1029094382 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96e4196fd0 con 0x7f96e40ff5a0 2024-04-22T23:42:24.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.540+0000 7f96e8c28700 1 -- 172.21.15.67:0/1029094382 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96e4109270 con 0x7f96e40ff5a0 2024-04-22T23:42:24.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.541+0000 7f96d37fe700 1 -- 172.21.15.67:0/1029094382 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f96d40228a0 con 0x7f96e40ff5a0 2024-04-22T23:42:24.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.541+0000 7f96d37fe700 1 --2- 172.21.15.67:0/1029094382 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f96cc06ecc0 0x7f96cc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:24.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.541+0000 7f96d37fe700 1 -- 172.21.15.67:0/1029094382 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f96d4090840 con 0x7f96e40ff5a0 2024-04-22T23:42:24.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.541+0000 7f96e1d9b700 1 --2- 172.21.15.67:0/1029094382 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f96cc06ecc0 0x7f96cc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:24.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.542+0000 7f96e1d9b700 1 --2- 172.21.15.67:0/1029094382 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f96cc06ecc0 0x7f96cc071180 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f96d8006fd0 tx=0x7f96d8008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:42:24.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.546+0000 7f96d37fe700 1 -- 172.21.15.67:0/1029094382 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f96d405cfb0 con 0x7f96e40ff5a0 2024-04-22T23:42:24.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.765+0000 7f96e8c28700 1 -- 172.21.15.67:0/1029094382 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f96e4197e30 con 0x7f96cc06ecc0 2024-04-22T23:42:24.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.776+0000 7f96d37fe700 1 -- 172.21.15.67:0/1029094382 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f96e4197e30 con 0x7f96cc06ecc0 2024-04-22T23:42:24.769 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:42:24.769 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (16m) 9m ago 27m 25.5M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:42:24.769 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (27m) 9m ago 27m 9114k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:42:24.769 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (26m) 9m ago 26m 9042k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:42:24.769 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (13m) 9m ago 27m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:42:24.769 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (13m) 9m ago 26m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:42:24.769 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (15m) 9m ago 27m 79.3M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:42:24.769 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (9m) 9m ago 21m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:42:24.769 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (9m) 9m ago 21m 12.6M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:42:24.769 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (9m) 9m ago 21m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:42:24.769 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (21m) 9m ago 21m 12.2M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:42:24.769 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (17m) 9m ago 28m 511M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:42:24.770 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (17m) 9m ago 26m 453M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:42:24.770 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (14m) 9m ago 29m 55.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:42:24.770 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (14m) 9m ago 26m 46.0M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:42:24.770 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (16m) 9m ago 27m 15.4M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:42:24.770 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (16m) 9m ago 26m 16.1M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:42:24.770 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (13m) 9m ago 25m 99.2M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:42:24.770 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (12m) 9m ago 25m 127M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:42:24.770 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (11m) 9m ago 24m 79.9M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:42:24.770 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (11m) 9m ago 24m 97.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:42:24.770 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (10m) 9m ago 23m 82.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:42:24.770 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (10m) 9m ago 23m 77.6M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:42:24.770 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (16m) 9m ago 26m 73.9M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:42:24.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.779+0000 7f96e8c28700 1 -- 172.21.15.67:0/1029094382 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f96cc06ecc0 msgr2=0x7f96cc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:24.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.779+0000 7f96e8c28700 1 --2- 172.21.15.67:0/1029094382 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f96cc06ecc0 0x7f96cc071180 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f96d8006fd0 tx=0x7f96d8008040 comp rx=0 tx=0).stop 2024-04-22T23:42:24.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.779+0000 7f96e8c28700 1 -- 172.21.15.67:0/1029094382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96e40ff5a0 msgr2=0x7f96e419c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:24.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.779+0000 7f96e8c28700 1 --2- 172.21.15.67:0/1029094382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96e40ff5a0 0x7f96e419c7e0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f96d4006010 tx=0x7f96d400be20 comp rx=0 tx=0).stop 2024-04-22T23:42:24.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.779+0000 7f96e8c28700 1 -- 172.21.15.67:0/1029094382 shutdown_connections 2024-04-22T23:42:24.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.779+0000 7f96e8c28700 1 --2- 172.21.15.67:0/1029094382 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f96cc06ecc0 0x7f96cc071180 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:24.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.779+0000 7f96e8c28700 1 --2- 172.21.15.67:0/1029094382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96e40ff5a0 0x7f96e419c7e0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:24.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.779+0000 7f96e8c28700 1 --2- 172.21.15.67:0/1029094382 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f96e40fff50 0x7f96e419cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:24.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.779+0000 7f96e8c28700 1 -- 172.21.15.67:0/1029094382 >> 172.21.15.67:0/1029094382 conn(0x7f96e4074e30 msgr2=0x7f96e40fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:42:24.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.779+0000 7f96e8c28700 1 -- 172.21.15.67:0/1029094382 shutdown_connections 2024-04-22T23:42:24.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.779+0000 7f96e8c28700 1 -- 172.21.15.67:0/1029094382 wait complete. 2024-04-22T23:42:24.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.885+0000 7f911d399700 1 -- 172.21.15.67:0/334170208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9118102700 msgr2=0x7f911810abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:24.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.885+0000 7f911d399700 1 --2- 172.21.15.67:0/334170208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9118102700 0x7f911810abf0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f9100009910 tx=0x7f9100009c20 comp rx=0 tx=0).stop 2024-04-22T23:42:24.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.885+0000 7f911d399700 1 -- 172.21.15.67:0/334170208 shutdown_connections 2024-04-22T23:42:24.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.885+0000 7f911d399700 1 --2- 172.21.15.67:0/334170208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9118102700 0x7f911810abf0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:24.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.885+0000 7f911d399700 1 --2- 172.21.15.67:0/334170208 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9118101de0 0x7f91181021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:24.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.885+0000 7f911d399700 1 -- 172.21.15.67:0/334170208 >> 172.21.15.67:0/334170208 conn(0x7f91180fb610 msgr2=0x7f91180fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:42:24.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.885+0000 7f911d399700 1 -- 172.21.15.67:0/334170208 shutdown_connections 2024-04-22T23:42:24.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.885+0000 7f911d399700 1 -- 172.21.15.67:0/334170208 wait complete. 2024-04-22T23:42:24.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.885+0000 7f911d399700 1 Processor -- start 2024-04-22T23:42:24.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.886+0000 7f911d399700 1 -- start start 2024-04-22T23:42:24.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.886+0000 7f911d399700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9118101de0 0x7f911819c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:24.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.886+0000 7f911d399700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9118102700 0x7f911819cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:24.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.886+0000 7f911d399700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f911819d220 con 0x7f9118102700 2024-04-22T23:42:24.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.886+0000 7f911d399700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9118196720 con 0x7f9118101de0 2024-04-22T23:42:24.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.886+0000 7f91167fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9118102700 0x7f911819cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:24.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.886+0000 7f91167fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9118102700 0x7f911819cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47468/0 (socket says 172.21.15.67:47468) 2024-04-22T23:42:24.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.886+0000 7f91167fc700 1 -- 172.21.15.67:0/1945179220 learned_addr learned my addr 172.21.15.67:0/1945179220 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:42:24.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.886+0000 7f9116ffd700 1 --2- 172.21.15.67:0/1945179220 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9118101de0 0x7f911819c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:24.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.886+0000 7f91167fc700 1 -- 172.21.15.67:0/1945179220 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9118101de0 msgr2=0x7f911819c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:24.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.886+0000 7f91167fc700 1 --2- 172.21.15.67:0/1945179220 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9118101de0 0x7f911819c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:24.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.886+0000 7f91167fc700 1 -- 172.21.15.67:0/1945179220 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91000095f0 con 0x7f9118102700 2024-04-22T23:42:24.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.886+0000 7f9116ffd700 1 --2- 172.21.15.67:0/1945179220 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9118101de0 0x7f911819c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:42:24.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.886+0000 7f91167fc700 1 --2- 172.21.15.67:0/1945179220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9118102700 0x7f911819cb90 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f91000092e0 tx=0x7f910000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:42:24.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.887+0000 7f910ffff700 1 -- 172.21.15.67:0/1945179220 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f910001d070 con 0x7f9118102700 2024-04-22T23:42:24.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.887+0000 7f910ffff700 1 -- 172.21.15.67:0/1945179220 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9100005dc0 con 0x7f9118102700 2024-04-22T23:42:24.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.887+0000 7f910ffff700 1 -- 172.21.15.67:0/1945179220 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9100017400 con 0x7f9118102700 2024-04-22T23:42:24.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.887+0000 7f911d399700 1 -- 172.21.15.67:0/1945179220 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91181969a0 con 0x7f9118102700 2024-04-22T23:42:24.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.887+0000 7f911d399700 1 -- 172.21.15.67:0/1945179220 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9118196e40 con 0x7f9118102700 2024-04-22T23:42:24.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.888+0000 7f911d399700 1 -- 172.21.15.67:0/1945179220 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91181082f0 con 0x7f9118102700 2024-04-22T23:42:24.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.888+0000 7f910ffff700 1 -- 172.21.15.67:0/1945179220 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f910000f6f0 con 0x7f9118102700 2024-04-22T23:42:24.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.889+0000 7f910ffff700 1 --2- 172.21.15.67:0/1945179220 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f910406ecc0 0x7f9104071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:24.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.889+0000 7f910ffff700 1 -- 172.21.15.67:0/1945179220 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f91000903c0 con 0x7f9118102700 2024-04-22T23:42:24.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.889+0000 7f9116ffd700 1 --2- 172.21.15.67:0/1945179220 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f910406ecc0 0x7f9104071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:24.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.889+0000 7f9116ffd700 1 --2- 172.21.15.67:0/1945179220 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f910406ecc0 0x7f9104071180 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9108009dd0 tx=0x7f9108009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:42:24.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:24.893+0000 7f910ffff700 1 -- 172.21.15.67:0/1945179220 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f910005cb30 con 0x7f9118102700 2024-04-22T23:42:25.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.190+0000 7f911d399700 1 -- 172.21.15.67:0/1945179220 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f91180679c0 con 0x7f9118102700 2024-04-22T23:42:25.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.191+0000 7f910ffff700 1 -- 172.21.15.67:0/1945179220 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f910005c6c0 con 0x7f9118102700 2024-04-22T23:42:25.183 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:42:25.183 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:42:25.184 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:42:25.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:42:25.184 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:42:25.184 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:42:25.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:42:25.184 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:42:25.184 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:42:25.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:42:25.184 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:42:25.184 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:42:25.184 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 3 2024-04-22T23:42:25.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:42:25.184 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:42:25.184 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:42:25.184 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 13 2024-04-22T23:42:25.185 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:42:25.185 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:42:25.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.193+0000 7f911d399700 1 -- 172.21.15.67:0/1945179220 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f910406ecc0 msgr2=0x7f9104071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:25.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.193+0000 7f911d399700 1 --2- 172.21.15.67:0/1945179220 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f910406ecc0 0x7f9104071180 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9108009dd0 tx=0x7f9108009040 comp rx=0 tx=0).stop 2024-04-22T23:42:25.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.193+0000 7f911d399700 1 -- 172.21.15.67:0/1945179220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9118102700 msgr2=0x7f911819cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:25.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.193+0000 7f911d399700 1 --2- 172.21.15.67:0/1945179220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9118102700 0x7f911819cb90 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f91000092e0 tx=0x7f910000be20 comp rx=0 tx=0).stop 2024-04-22T23:42:25.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.193+0000 7f911d399700 1 -- 172.21.15.67:0/1945179220 shutdown_connections 2024-04-22T23:42:25.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.193+0000 7f911d399700 1 --2- 172.21.15.67:0/1945179220 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f910406ecc0 0x7f9104071180 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:25.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.193+0000 7f911d399700 1 --2- 172.21.15.67:0/1945179220 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9118101de0 0x7f911819c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:25.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.193+0000 7f911d399700 1 --2- 172.21.15.67:0/1945179220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9118102700 0x7f911819cb90 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:25.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.193+0000 7f911d399700 1 -- 172.21.15.67:0/1945179220 >> 172.21.15.67:0/1945179220 conn(0x7f91180fb610 msgr2=0x7f9118105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:42:25.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.193+0000 7f911d399700 1 -- 172.21.15.67:0/1945179220 shutdown_connections 2024-04-22T23:42:25.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.193+0000 7f911d399700 1 -- 172.21.15.67:0/1945179220 wait complete. 2024-04-22T23:42:25.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.300+0000 7fbecc418700 1 -- 172.21.15.67:0/2731831592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbec4100750 msgr2=0x7fbec41047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:25.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.300+0000 7fbecc418700 1 --2- 172.21.15.67:0/2731831592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbec4100750 0x7fbec41047a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4009910 tx=0x7fbeb4009c20 comp rx=0 tx=0).stop 2024-04-22T23:42:25.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.300+0000 7fbecc418700 1 -- 172.21.15.67:0/2731831592 shutdown_connections 2024-04-22T23:42:25.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.300+0000 7fbecc418700 1 --2- 172.21.15.67:0/2731831592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbec4100750 0x7fbec41047a0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:25.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.300+0000 7fbecc418700 1 --2- 172.21.15.67:0/2731831592 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbec40ffda0 0x7fbec4100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:25.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.300+0000 7fbecc418700 1 -- 172.21.15.67:0/2731831592 >> 172.21.15.67:0/2731831592 conn(0x7fbec40fb650 msgr2=0x7fbec40fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:42:25.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.300+0000 7fbecc418700 1 -- 172.21.15.67:0/2731831592 shutdown_connections 2024-04-22T23:42:25.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.300+0000 7fbecc418700 1 -- 172.21.15.67:0/2731831592 wait complete. 2024-04-22T23:42:25.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.301+0000 7fbecc418700 1 Processor -- start 2024-04-22T23:42:25.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.301+0000 7fbecc418700 1 -- start start 2024-04-22T23:42:25.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.301+0000 7fbecc418700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbec40ffda0 0x7fbec419e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:25.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.301+0000 7fbecc418700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbec4100750 0x7fbec419ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:25.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.301+0000 7fbecc418700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec419f4c0 con 0x7fbec4100750 2024-04-22T23:42:25.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.301+0000 7fbecc418700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec4198970 con 0x7fbec40ffda0 2024-04-22T23:42:25.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.301+0000 7fbec99b3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbec4100750 0x7fbec419ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:25.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.301+0000 7fbec99b3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbec4100750 0x7fbec419ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47482/0 (socket says 172.21.15.67:47482) 2024-04-22T23:42:25.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.301+0000 7fbec99b3700 1 -- 172.21.15.67:0/220287606 learned_addr learned my addr 172.21.15.67:0/220287606 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:42:25.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.301+0000 7fbeca1b4700 1 --2- 172.21.15.67:0/220287606 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbec40ffda0 0x7fbec419e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:25.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.301+0000 7fbec99b3700 1 -- 172.21.15.67:0/220287606 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbec40ffda0 msgr2=0x7fbec419e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:25.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.301+0000 7fbec99b3700 1 --2- 172.21.15.67:0/220287606 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbec40ffda0 0x7fbec419e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:25.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.301+0000 7fbec99b3700 1 -- 172.21.15.67:0/220287606 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb40095f0 con 0x7fbec4100750 2024-04-22T23:42:25.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.301+0000 7fbec99b3700 1 --2- 172.21.15.67:0/220287606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbec4100750 0x7fbec419ee30 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fbeb400bda0 tx=0x7fbeb400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:42:25.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.302+0000 7fbebb7fe700 1 -- 172.21.15.67:0/220287606 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbeb401d070 con 0x7fbec4100750 2024-04-22T23:42:25.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.302+0000 7fbecc418700 1 -- 172.21.15.67:0/220287606 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbec4198bf0 con 0x7fbec4100750 2024-04-22T23:42:25.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.302+0000 7fbebb7fe700 1 -- 172.21.15.67:0/220287606 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbeb4004fb0 con 0x7fbec4100750 2024-04-22T23:42:25.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.302+0000 7fbebb7fe700 1 -- 172.21.15.67:0/220287606 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbeb400f460 con 0x7fbec4100750 2024-04-22T23:42:25.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.302+0000 7fbecc418700 1 -- 172.21.15.67:0/220287606 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbec41990e0 con 0x7fbec4100750 2024-04-22T23:42:25.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.303+0000 7fbecc418700 1 -- 172.21.15.67:0/220287606 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbec404f2e0 con 0x7fbec4100750 2024-04-22T23:42:25.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.303+0000 7fbebb7fe700 1 -- 172.21.15.67:0/220287606 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbeb4022800 con 0x7fbec4100750 2024-04-22T23:42:25.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.304+0000 7fbebb7fe700 1 --2- 172.21.15.67:0/220287606 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbeb006ecc0 0x7fbeb0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:25.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.304+0000 7fbebb7fe700 1 -- 172.21.15.67:0/220287606 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fbeb4090510 con 0x7fbec4100750 2024-04-22T23:42:25.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.304+0000 7fbeca1b4700 1 --2- 172.21.15.67:0/220287606 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbeb006ecc0 0x7fbeb0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:25.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.304+0000 7fbeca1b4700 1 --2- 172.21.15.67:0/220287606 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbeb006ecc0 0x7fbeb0071180 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fbec0005950 tx=0x7fbec0005eb0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:42:25.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.308+0000 7fbebb7fe700 1 -- 172.21.15.67:0/220287606 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fbeb405ce50 con 0x7fbec4100750 2024-04-22T23:42:25.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.562+0000 7fbecc418700 1 -- 172.21.15.67:0/220287606 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fbec41993c0 con 0x7fbec4100750 2024-04-22T23:42:25.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.563+0000 7fbebb7fe700 1 -- 172.21.15.67:0/220287606 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1871 (secure 0 0 0) 0x7fbeb405c9e0 con 0x7fbec4100750 2024-04-22T23:42:25.556 INFO:teuthology.orchestra.run.smithi067.stdout:e31 2024-04-22T23:42:25.556 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:42:25.556 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:42:25.556 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:42:25.556 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:42:25.556 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:42:25.556 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:42:25.556 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:42:25.556 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:42:25.556 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:42:25.556 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:42:25.556 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:42:25.556 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:42:25.556 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:42:25.557 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:42:25.557 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:42:25.557 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:42:25.557 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:42:25.557 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:42:25.557 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:42:25.557 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:42:25.557 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:42:25.557 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:42:25.557 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:42:25.557 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:42:25.557 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:42:25.557 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:42:25.557 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:42:25.557 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:42:25.557 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:42:25.557 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:42:25.558 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:42:25.558 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:42:25.558 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:42:25.558 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:42:25.558 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:42:25.558 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:42:25.558 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:42:25.558 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:42:25.558 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:42:25.558 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44259} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1827288036,v1:172.21.15.134:6827/1827288036] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:42:25.558 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:42:25.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.565+0000 7fbecc418700 1 -- 172.21.15.67:0/220287606 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbeb006ecc0 msgr2=0x7fbeb0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:25.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.565+0000 7fbecc418700 1 --2- 172.21.15.67:0/220287606 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbeb006ecc0 0x7fbeb0071180 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fbec0005950 tx=0x7fbec0005eb0 comp rx=0 tx=0).stop 2024-04-22T23:42:25.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.565+0000 7fbecc418700 1 -- 172.21.15.67:0/220287606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbec4100750 msgr2=0x7fbec419ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:25.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.565+0000 7fbecc418700 1 --2- 172.21.15.67:0/220287606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbec4100750 0x7fbec419ee30 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fbeb400bda0 tx=0x7fbeb400be80 comp rx=0 tx=0).stop 2024-04-22T23:42:25.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.565+0000 7fbecc418700 1 -- 172.21.15.67:0/220287606 shutdown_connections 2024-04-22T23:42:25.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.565+0000 7fbecc418700 1 --2- 172.21.15.67:0/220287606 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbeb006ecc0 0x7fbeb0071180 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:25.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.565+0000 7fbecc418700 1 --2- 172.21.15.67:0/220287606 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbec40ffda0 0x7fbec419e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:25.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.565+0000 7fbecc418700 1 --2- 172.21.15.67:0/220287606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbec4100750 0x7fbec419ee30 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:25.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.565+0000 7fbecc418700 1 -- 172.21.15.67:0/220287606 >> 172.21.15.67:0/220287606 conn(0x7fbec40fb650 msgr2=0x7fbec4068e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:42:25.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.565+0000 7fbecc418700 1 -- 172.21.15.67:0/220287606 shutdown_connections 2024-04-22T23:42:25.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.565+0000 7fbecc418700 1 -- 172.21.15.67:0/220287606 wait complete. 2024-04-22T23:42:25.559 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 31 2024-04-22T23:42:25.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.672+0000 7f8ec39d3700 1 -- 172.21.15.67:0/126556353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ebc101850 msgr2=0x7f8ebc1058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:25.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.672+0000 7f8ec39d3700 1 --2- 172.21.15.67:0/126556353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ebc101850 0x7f8ebc1058a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8009910 tx=0x7f8eb8009c20 comp rx=0 tx=0).stop 2024-04-22T23:42:25.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.672+0000 7f8ec39d3700 1 -- 172.21.15.67:0/126556353 shutdown_connections 2024-04-22T23:42:25.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.672+0000 7f8ec39d3700 1 --2- 172.21.15.67:0/126556353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ebc101850 0x7f8ebc1058a0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:25.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.672+0000 7f8ec39d3700 1 --2- 172.21.15.67:0/126556353 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8ebc100ea0 0x7f8ebc101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:25.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.672+0000 7f8ec39d3700 1 -- 172.21.15.67:0/126556353 >> 172.21.15.67:0/126556353 conn(0x7f8ebc0fc750 msgr2=0x7f8ebc0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:42:25.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.672+0000 7f8ec39d3700 1 -- 172.21.15.67:0/126556353 shutdown_connections 2024-04-22T23:42:25.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.672+0000 7f8ec39d3700 1 -- 172.21.15.67:0/126556353 wait complete. 2024-04-22T23:42:25.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.673+0000 7f8ec39d3700 1 Processor -- start 2024-04-22T23:42:25.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.673+0000 7f8ec39d3700 1 -- start start 2024-04-22T23:42:25.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.673+0000 7f8ec39d3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ebc100ea0 0x7f8ebc19c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:25.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.673+0000 7f8ec39d3700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8ebc101850 0x7f8ebc19cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:25.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.673+0000 7f8ec39d3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc19d240 con 0x7f8ebc100ea0 2024-04-22T23:42:25.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.673+0000 7f8ec39d3700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc196780 con 0x7f8ebc101850 2024-04-22T23:42:25.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.674+0000 7f8ec176f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ebc100ea0 0x7f8ebc19c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:25.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.674+0000 7f8ec176f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ebc100ea0 0x7f8ebc19c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47504/0 (socket says 172.21.15.67:47504) 2024-04-22T23:42:25.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.674+0000 7f8ec176f700 1 -- 172.21.15.67:0/801403789 learned_addr learned my addr 172.21.15.67:0/801403789 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:42:25.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.674+0000 7f8ec0f6e700 1 --2- 172.21.15.67:0/801403789 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8ebc101850 0x7f8ebc19cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:25.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.674+0000 7f8ec176f700 1 -- 172.21.15.67:0/801403789 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8ebc101850 msgr2=0x7f8ebc19cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:25.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.674+0000 7f8ec176f700 1 --2- 172.21.15.67:0/801403789 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8ebc101850 0x7f8ebc19cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:25.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.674+0000 7f8ec176f700 1 -- 172.21.15.67:0/801403789 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb80095f0 con 0x7f8ebc100ea0 2024-04-22T23:42:25.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.674+0000 7f8ec176f700 1 --2- 172.21.15.67:0/801403789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ebc100ea0 0x7f8ebc19c700 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f8eac00c930 tx=0x7f8eac00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:42:25.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.674+0000 7f8eb27fc700 1 -- 172.21.15.67:0/801403789 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8eac007ad0 con 0x7f8ebc100ea0 2024-04-22T23:42:25.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.674+0000 7f8eb27fc700 1 -- 172.21.15.67:0/801403789 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8eac00ce80 con 0x7f8ebc100ea0 2024-04-22T23:42:25.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.674+0000 7f8ec39d3700 1 -- 172.21.15.67:0/801403789 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ebc196a60 con 0x7f8ebc100ea0 2024-04-22T23:42:25.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.674+0000 7f8eb27fc700 1 -- 172.21.15.67:0/801403789 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8eac018630 con 0x7f8ebc100ea0 2024-04-22T23:42:25.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.674+0000 7f8ec39d3700 1 -- 172.21.15.67:0/801403789 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ebc196fb0 con 0x7f8ebc100ea0 2024-04-22T23:42:25.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.675+0000 7f8ec39d3700 1 -- 172.21.15.67:0/801403789 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ebc1092a0 con 0x7f8ebc100ea0 2024-04-22T23:42:25.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.676+0000 7f8eb27fc700 1 -- 172.21.15.67:0/801403789 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8eac007c30 con 0x7f8ebc100ea0 2024-04-22T23:42:25.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.676+0000 7f8eb27fc700 1 --2- 172.21.15.67:0/801403789 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8ea806ecc0 0x7f8ea8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:25.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.677+0000 7f8eb27fc700 1 -- 172.21.15.67:0/801403789 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f8eac08f010 con 0x7f8ebc100ea0 2024-04-22T23:42:25.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.677+0000 7f8ec0f6e700 1 --2- 172.21.15.67:0/801403789 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8ea806ecc0 0x7f8ea8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:25.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.677+0000 7f8ec0f6e700 1 --2- 172.21.15.67:0/801403789 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8ea806ecc0 0x7f8ea8071180 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc197bf0 tx=0x7f8eb801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:42:25.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.681+0000 7f8eb27fc700 1 -- 172.21.15.67:0/801403789 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f8eac05b780 con 0x7f8ebc100ea0 2024-04-22T23:42:25.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.915+0000 7f8ec39d3700 1 -- 172.21.15.67:0/801403789 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ebc197d50 con 0x7f8ea806ecc0 2024-04-22T23:42:25.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.917+0000 7f8eb27fc700 1 -- 172.21.15.67:0/801403789 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f8ebc197d50 con 0x7f8ea806ecc0 2024-04-22T23:42:25.909 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:42:25.909 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:42:25.910 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:42:25.910 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:42:25.910 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:42:25.910 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:42:25.910 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:42:25.910 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:42:25.910 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:42:25.910 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:42:25.910 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T23:42:25.910 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:42:25.910 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:42:25.910 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:42:25.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.919+0000 7f8ec39d3700 1 -- 172.21.15.67:0/801403789 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8ea806ecc0 msgr2=0x7f8ea8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:25.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.919+0000 7f8ec39d3700 1 --2- 172.21.15.67:0/801403789 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8ea806ecc0 0x7f8ea8071180 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc197bf0 tx=0x7f8eb801a040 comp rx=0 tx=0).stop 2024-04-22T23:42:25.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.919+0000 7f8ec39d3700 1 -- 172.21.15.67:0/801403789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ebc100ea0 msgr2=0x7f8ebc19c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:25.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.919+0000 7f8ec39d3700 1 --2- 172.21.15.67:0/801403789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ebc100ea0 0x7f8ebc19c700 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f8eac00c930 tx=0x7f8eac00cc40 comp rx=0 tx=0).stop 2024-04-22T23:42:25.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.919+0000 7f8ec39d3700 1 -- 172.21.15.67:0/801403789 shutdown_connections 2024-04-22T23:42:25.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.919+0000 7f8ec39d3700 1 --2- 172.21.15.67:0/801403789 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8ea806ecc0 0x7f8ea8071180 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:25.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.919+0000 7f8ec39d3700 1 --2- 172.21.15.67:0/801403789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ebc100ea0 0x7f8ebc19c700 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:25.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.919+0000 7f8ec39d3700 1 --2- 172.21.15.67:0/801403789 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8ebc101850 0x7f8ebc19cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:25.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.919+0000 7f8ec39d3700 1 -- 172.21.15.67:0/801403789 >> 172.21.15.67:0/801403789 conn(0x7f8ebc0fc750 msgr2=0x7f8ebc0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:42:25.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.919+0000 7f8ec39d3700 1 -- 172.21.15.67:0/801403789 shutdown_connections 2024-04-22T23:42:25.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:25.919+0000 7f8ec39d3700 1 -- 172.21.15.67:0/801403789 wait complete. 2024-04-22T23:42:26.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.026+0000 7ffbdfe6b700 1 -- 172.21.15.67:0/352291471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbd8102720 msgr2=0x7ffbd810ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:26.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.026+0000 7ffbdfe6b700 1 --2- 172.21.15.67:0/352291471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbd8102720 0x7ffbd810ac10 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8009910 tx=0x7ffbc8009c20 comp rx=0 tx=0).stop 2024-04-22T23:42:26.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.027+0000 7ffbdfe6b700 1 -- 172.21.15.67:0/352291471 shutdown_connections 2024-04-22T23:42:26.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.027+0000 7ffbdfe6b700 1 --2- 172.21.15.67:0/352291471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbd8102720 0x7ffbd810ac10 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:26.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.027+0000 7ffbdfe6b700 1 --2- 172.21.15.67:0/352291471 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffbd8101e00 0x7ffbd81021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:26.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.027+0000 7ffbdfe6b700 1 -- 172.21.15.67:0/352291471 >> 172.21.15.67:0/352291471 conn(0x7ffbd80fb650 msgr2=0x7ffbd80fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:42:26.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.027+0000 7ffbdfe6b700 1 -- 172.21.15.67:0/352291471 shutdown_connections 2024-04-22T23:42:26.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.027+0000 7ffbdfe6b700 1 -- 172.21.15.67:0/352291471 wait complete. 2024-04-22T23:42:26.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.027+0000 7ffbdfe6b700 1 Processor -- start 2024-04-22T23:42:26.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.027+0000 7ffbdfe6b700 1 -- start start 2024-04-22T23:42:26.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.027+0000 7ffbdfe6b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbd8101e00 0x7ffbd819e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:26.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.028+0000 7ffbdfe6b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffbd8102720 0x7ffbd819edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:26.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.028+0000 7ffbdfe6b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd819f3b0 con 0x7ffbd8101e00 2024-04-22T23:42:26.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.028+0000 7ffbdfe6b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd81988f0 con 0x7ffbd8102720 2024-04-22T23:42:26.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.028+0000 7ffbddc07700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbd8101e00 0x7ffbd819e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:26.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.028+0000 7ffbddc07700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbd8101e00 0x7ffbd819e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47522/0 (socket says 172.21.15.67:47522) 2024-04-22T23:42:26.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.028+0000 7ffbddc07700 1 -- 172.21.15.67:0/3545644344 learned_addr learned my addr 172.21.15.67:0/3545644344 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:42:26.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.028+0000 7ffbdd406700 1 --2- 172.21.15.67:0/3545644344 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffbd8102720 0x7ffbd819edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:26.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.028+0000 7ffbddc07700 1 -- 172.21.15.67:0/3545644344 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffbd8102720 msgr2=0x7ffbd819edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:26.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.028+0000 7ffbddc07700 1 --2- 172.21.15.67:0/3545644344 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffbd8102720 0x7ffbd819edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:26.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.028+0000 7ffbddc07700 1 -- 172.21.15.67:0/3545644344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc80095f0 con 0x7ffbd8101e00 2024-04-22T23:42:26.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.028+0000 7ffbddc07700 1 --2- 172.21.15.67:0/3545644344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbd8101e00 0x7ffbd819e870 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7ffbd400c930 tx=0x7ffbd400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:42:26.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.028+0000 7ffbceffd700 1 -- 172.21.15.67:0/3545644344 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffbd4007ad0 con 0x7ffbd8101e00 2024-04-22T23:42:26.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.028+0000 7ffbceffd700 1 -- 172.21.15.67:0/3545644344 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffbd400ce80 con 0x7ffbd8101e00 2024-04-22T23:42:26.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.028+0000 7ffbdfe6b700 1 -- 172.21.15.67:0/3545644344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbd8198bd0 con 0x7ffbd8101e00 2024-04-22T23:42:26.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.028+0000 7ffbceffd700 1 -- 172.21.15.67:0/3545644344 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffbd4018630 con 0x7ffbd8101e00 2024-04-22T23:42:26.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.028+0000 7ffbdfe6b700 1 -- 172.21.15.67:0/3545644344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbd81990f0 con 0x7ffbd8101e00 2024-04-22T23:42:26.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.029+0000 7ffbdfe6b700 1 -- 172.21.15.67:0/3545644344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbd810a770 con 0x7ffbd8101e00 2024-04-22T23:42:26.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.030+0000 7ffbceffd700 1 -- 172.21.15.67:0/3545644344 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ffbd4007c30 con 0x7ffbd8101e00 2024-04-22T23:42:26.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.030+0000 7ffbceffd700 1 --2- 172.21.15.67:0/3545644344 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ffbc406ecc0 0x7ffbc4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:26.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.030+0000 7ffbceffd700 1 -- 172.21.15.67:0/3545644344 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7ffbd408ef10 con 0x7ffbd8101e00 2024-04-22T23:42:26.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.030+0000 7ffbdd406700 1 --2- 172.21.15.67:0/3545644344 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ffbc406ecc0 0x7ffbc4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:26.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.031+0000 7ffbdd406700 1 --2- 172.21.15.67:0/3545644344 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ffbc406ecc0 0x7ffbc4071180 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ffbc800b5c0 tx=0x7ffbc801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:42:26.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.035+0000 7ffbceffd700 1 -- 172.21.15.67:0/3545644344 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ffbd405b700 con 0x7ffbd8101e00 2024-04-22T23:42:26.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:25 smithi134 ceph-mon[85630]: from='client.34904 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:42:26.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:25 smithi134 ceph-mon[85630]: pgmap v481: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:26.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:25 smithi134 ceph-mon[85630]: from='client.34908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:42:26.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:25 smithi134 ceph-mon[85630]: from='client.34912 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:42:26.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:25 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1945179220' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:42:26.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:25 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/220287606' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:42:26.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:25 smithi067 ceph-mon[112438]: from='client.34904 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:42:26.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:25 smithi067 ceph-mon[112438]: pgmap v481: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:26.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:25 smithi067 ceph-mon[112438]: from='client.34908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:42:26.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:25 smithi067 ceph-mon[112438]: from='client.34912 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:42:26.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:25 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1945179220' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:42:26.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:25 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/220287606' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:42:26.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.337+0000 7ffbdfe6b700 1 -- 172.21.15.67:0/3545644344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ffbd804f2e0 con 0x7ffbd8101e00 2024-04-22T23:42:26.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.338+0000 7ffbceffd700 1 -- 172.21.15.67:0/3545644344 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7ffbd804f2e0 con 0x7ffbd8101e00 2024-04-22T23:42:26.331 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:42:26.331 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:42:26.331 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:42:26.331 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:42:26.331 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:42:26.331 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:42:26.331 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:42:26.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.340+0000 7ffbdfe6b700 1 -- 172.21.15.67:0/3545644344 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ffbc406ecc0 msgr2=0x7ffbc4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:26.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.340+0000 7ffbdfe6b700 1 --2- 172.21.15.67:0/3545644344 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ffbc406ecc0 0x7ffbc4071180 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ffbc800b5c0 tx=0x7ffbc801a040 comp rx=0 tx=0).stop 2024-04-22T23:42:26.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.340+0000 7ffbdfe6b700 1 -- 172.21.15.67:0/3545644344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbd8101e00 msgr2=0x7ffbd819e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:26.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.340+0000 7ffbdfe6b700 1 --2- 172.21.15.67:0/3545644344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbd8101e00 0x7ffbd819e870 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7ffbd400c930 tx=0x7ffbd400cc40 comp rx=0 tx=0).stop 2024-04-22T23:42:26.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.340+0000 7ffbdfe6b700 1 -- 172.21.15.67:0/3545644344 shutdown_connections 2024-04-22T23:42:26.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.340+0000 7ffbdfe6b700 1 --2- 172.21.15.67:0/3545644344 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ffbc406ecc0 0x7ffbc4071180 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:26.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.340+0000 7ffbdfe6b700 1 --2- 172.21.15.67:0/3545644344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbd8101e00 0x7ffbd819e870 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:26.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.340+0000 7ffbdfe6b700 1 --2- 172.21.15.67:0/3545644344 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffbd8102720 0x7ffbd819edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:26.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.340+0000 7ffbdfe6b700 1 -- 172.21.15.67:0/3545644344 >> 172.21.15.67:0/3545644344 conn(0x7ffbd80fb650 msgr2=0x7ffbd8100220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:42:26.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.340+0000 7ffbdfe6b700 1 -- 172.21.15.67:0/3545644344 shutdown_connections 2024-04-22T23:42:26.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:26.340+0000 7ffbdfe6b700 1 -- 172.21.15.67:0/3545644344 wait complete. 2024-04-22T23:42:27.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:26 smithi134 ceph-mon[85630]: from='client.34924 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:42:27.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:42:27.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:26 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3545644344' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:42:27.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:26 smithi067 ceph-mon[112438]: from='client.34924 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:42:27.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:42:27.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:26 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3545644344' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:42:28.115 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:27 smithi134 ceph-mon[85630]: pgmap v482: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:28.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:27 smithi067 ceph-mon[112438]: pgmap v482: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:30.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:29 smithi134 ceph-mon[85630]: pgmap v483: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:30.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:29 smithi067 ceph-mon[112438]: pgmap v483: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:32.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:31 smithi134 ceph-mon[85630]: pgmap v484: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:32.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:31 smithi067 ceph-mon[112438]: pgmap v484: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:34.037 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:33 smithi067 ceph-mon[112438]: pgmap v485: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:34.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:33 smithi134 ceph-mon[85630]: pgmap v485: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:36.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:35 smithi134 ceph-mon[85630]: pgmap v486: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:36.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:35 smithi067 ceph-mon[112438]: pgmap v486: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:38.223 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:37 smithi134 ceph-mon[85630]: pgmap v487: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:38.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:37 smithi067 ceph-mon[112438]: pgmap v487: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:40.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:39 smithi134 ceph-mon[85630]: pgmap v488: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:40.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:39 smithi067 ceph-mon[112438]: pgmap v488: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:42.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:41 smithi134 ceph-mon[85630]: pgmap v489: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:42.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:42:42.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:41 smithi067 ceph-mon[112438]: pgmap v489: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:42.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:42:44.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:43 smithi134 ceph-mon[85630]: pgmap v490: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:44.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:43 smithi067 ceph-mon[112438]: pgmap v490: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:46.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:45 smithi134 ceph-mon[85630]: pgmap v491: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:46.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:45 smithi067 ceph-mon[112438]: pgmap v491: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:48.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:47 smithi134 ceph-mon[85630]: pgmap v492: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:48.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:47 smithi067 ceph-mon[112438]: pgmap v492: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:50.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:49 smithi134 ceph-mon[85630]: pgmap v493: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:50.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:49 smithi067 ceph-mon[112438]: pgmap v493: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:52.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:51 smithi134 ceph-mon[85630]: pgmap v494: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:52.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:51 smithi067 ceph-mon[112438]: pgmap v494: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:54.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:53 smithi134 ceph-mon[85630]: pgmap v495: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:54.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:53 smithi067 ceph-mon[112438]: pgmap v495: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:56.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:55 smithi134 ceph-mon[85630]: pgmap v496: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:56.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:55 smithi067 ceph-mon[112438]: pgmap v496: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:56.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.457+0000 7f19fe323700 1 -- 172.21.15.67:0/60367211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19f8101850 msgr2=0x7f19f81058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:56.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.457+0000 7f19fe323700 1 --2- 172.21.15.67:0/60367211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19f8101850 0x7f19f81058a0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f19e0009910 tx=0x7f19e0009c20 comp rx=0 tx=0).stop 2024-04-22T23:42:56.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.457+0000 7f19fe323700 1 -- 172.21.15.67:0/60367211 shutdown_connections 2024-04-22T23:42:56.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.457+0000 7f19fe323700 1 --2- 172.21.15.67:0/60367211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19f8101850 0x7f19f81058a0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:56.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.457+0000 7f19fe323700 1 --2- 172.21.15.67:0/60367211 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f19f8100ea0 0x7f19f8101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:56.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.457+0000 7f19fe323700 1 -- 172.21.15.67:0/60367211 >> 172.21.15.67:0/60367211 conn(0x7f19f80fc750 msgr2=0x7f19f80feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:42:56.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.457+0000 7f19fe323700 1 -- 172.21.15.67:0/60367211 shutdown_connections 2024-04-22T23:42:56.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.457+0000 7f19fe323700 1 -- 172.21.15.67:0/60367211 wait complete. 2024-04-22T23:42:56.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.458+0000 7f19fe323700 1 Processor -- start 2024-04-22T23:42:56.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.458+0000 7f19fe323700 1 -- start start 2024-04-22T23:42:56.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.458+0000 7f19fe323700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f19f8100ea0 0x7f19f8073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:56.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.458+0000 7f19fe323700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19f8101850 0x7f19f80735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:56.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.458+0000 7f19fe323700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f8073b10 con 0x7f19f8101850 2024-04-22T23:42:56.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.458+0000 7f19fe323700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f8073c50 con 0x7f19f8100ea0 2024-04-22T23:42:56.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.459+0000 7f19f77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19f8101850 0x7f19f80735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:56.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.459+0000 7f19f77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19f8101850 0x7f19f80735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52358/0 (socket says 172.21.15.67:52358) 2024-04-22T23:42:56.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.459+0000 7f19f77fe700 1 -- 172.21.15.67:0/2399217395 learned_addr learned my addr 172.21.15.67:0/2399217395 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:42:56.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.459+0000 7f19f7fff700 1 --2- 172.21.15.67:0/2399217395 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f19f8100ea0 0x7f19f8073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:56.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.459+0000 7f19f77fe700 1 -- 172.21.15.67:0/2399217395 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f19f8100ea0 msgr2=0x7f19f8073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:56.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.459+0000 7f19f77fe700 1 --2- 172.21.15.67:0/2399217395 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f19f8100ea0 0x7f19f8073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:56.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.459+0000 7f19f77fe700 1 -- 172.21.15.67:0/2399217395 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e00095f0 con 0x7f19f8101850 2024-04-22T23:42:56.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.459+0000 7f19f77fe700 1 --2- 172.21.15.67:0/2399217395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19f8101850 0x7f19f80735d0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f19e0005f50 tx=0x7f19e000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:42:56.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.459+0000 7f19f57fa700 1 -- 172.21.15.67:0/2399217395 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19e001d070 con 0x7f19f8101850 2024-04-22T23:42:56.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.459+0000 7f19fe323700 1 -- 172.21.15.67:0/2399217395 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19f8073ed0 con 0x7f19f8101850 2024-04-22T23:42:56.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.459+0000 7f19f57fa700 1 -- 172.21.15.67:0/2399217395 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f19e000f460 con 0x7f19f8101850 2024-04-22T23:42:56.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.459+0000 7f19f57fa700 1 -- 172.21.15.67:0/2399217395 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19e0017630 con 0x7f19f8101850 2024-04-22T23:42:56.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.459+0000 7f19fe323700 1 -- 172.21.15.67:0/2399217395 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19f81a2ec0 con 0x7f19f8101850 2024-04-22T23:42:56.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.460+0000 7f19fe323700 1 -- 172.21.15.67:0/2399217395 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19f81091f0 con 0x7f19f8101850 2024-04-22T23:42:56.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.461+0000 7f19f57fa700 1 -- 172.21.15.67:0/2399217395 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f19e000f5d0 con 0x7f19f8101850 2024-04-22T23:42:56.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.461+0000 7f19f57fa700 1 --2- 172.21.15.67:0/2399217395 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f19e406ecc0 0x7f19e4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:56.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.461+0000 7f19f57fa700 1 -- 172.21.15.67:0/2399217395 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f19e00904e0 con 0x7f19f8101850 2024-04-22T23:42:56.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.461+0000 7f19f7fff700 1 --2- 172.21.15.67:0/2399217395 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f19e406ecc0 0x7f19e4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:56.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.462+0000 7f19f7fff700 1 --2- 172.21.15.67:0/2399217395 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f19e406ecc0 0x7f19e4071180 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f19e8009a10 tx=0x7f19e8009190 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:42:56.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.466+0000 7f19f57fa700 1 -- 172.21.15.67:0/2399217395 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f19e0059120 con 0x7f19f8101850 2024-04-22T23:42:56.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.696+0000 7f19fe323700 1 -- 172.21.15.67:0/2399217395 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f19f80746a0 con 0x7f19e406ecc0 2024-04-22T23:42:56.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.697+0000 7f19f57fa700 1 -- 172.21.15.67:0/2399217395 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f19f80746a0 con 0x7f19e406ecc0 2024-04-22T23:42:56.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.699+0000 7f19fe323700 1 -- 172.21.15.67:0/2399217395 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f19e406ecc0 msgr2=0x7f19e4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:56.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.699+0000 7f19fe323700 1 --2- 172.21.15.67:0/2399217395 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f19e406ecc0 0x7f19e4071180 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f19e8009a10 tx=0x7f19e8009190 comp rx=0 tx=0).stop 2024-04-22T23:42:56.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.699+0000 7f19fe323700 1 -- 172.21.15.67:0/2399217395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19f8101850 msgr2=0x7f19f80735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:56.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.699+0000 7f19fe323700 1 --2- 172.21.15.67:0/2399217395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19f8101850 0x7f19f80735d0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f19e0005f50 tx=0x7f19e000bd70 comp rx=0 tx=0).stop 2024-04-22T23:42:56.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.699+0000 7f19fe323700 1 -- 172.21.15.67:0/2399217395 shutdown_connections 2024-04-22T23:42:56.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.699+0000 7f19fe323700 1 --2- 172.21.15.67:0/2399217395 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f19e406ecc0 0x7f19e4071180 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:56.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.699+0000 7f19fe323700 1 --2- 172.21.15.67:0/2399217395 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f19f8100ea0 0x7f19f8073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:56.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.699+0000 7f19fe323700 1 --2- 172.21.15.67:0/2399217395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19f8101850 0x7f19f80735d0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:56.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.699+0000 7f19fe323700 1 -- 172.21.15.67:0/2399217395 >> 172.21.15.67:0/2399217395 conn(0x7f19f80fc750 msgr2=0x7f19f80fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:42:56.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.700+0000 7f19fe323700 1 -- 172.21.15.67:0/2399217395 shutdown_connections 2024-04-22T23:42:56.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.700+0000 7f19fe323700 1 -- 172.21.15.67:0/2399217395 wait complete. 2024-04-22T23:42:56.703 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:42:56.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.808+0000 7f7ae3b2e700 1 -- 172.21.15.67:0/2338909735 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7adc0ffb40 msgr2=0x7f7adc0fffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:56.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.808+0000 7f7ae3b2e700 1 --2- 172.21.15.67:0/2338909735 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7adc0ffb40 0x7f7adc0fffc0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7acc009910 tx=0x7f7acc009c20 comp rx=0 tx=0).stop 2024-04-22T23:42:56.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.808+0000 7f7ae3b2e700 1 -- 172.21.15.67:0/2338909735 shutdown_connections 2024-04-22T23:42:56.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.808+0000 7f7ae3b2e700 1 --2- 172.21.15.67:0/2338909735 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7adc0ffb40 0x7f7adc0fffc0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:56.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.808+0000 7f7ae3b2e700 1 --2- 172.21.15.67:0/2338909735 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7adc105bf0 0x7f7adc105fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:56.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.808+0000 7f7ae3b2e700 1 -- 172.21.15.67:0/2338909735 >> 172.21.15.67:0/2338909735 conn(0x7f7adc0fb650 msgr2=0x7f7adc0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:42:56.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.808+0000 7f7ae3b2e700 1 -- 172.21.15.67:0/2338909735 shutdown_connections 2024-04-22T23:42:56.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.808+0000 7f7ae3b2e700 1 -- 172.21.15.67:0/2338909735 wait complete. 2024-04-22T23:42:56.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.809+0000 7f7ae3b2e700 1 Processor -- start 2024-04-22T23:42:56.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.809+0000 7f7ae3b2e700 1 -- start start 2024-04-22T23:42:56.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.809+0000 7f7ae3b2e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7adc0ffb40 0x7f7adc19c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:56.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.809+0000 7f7ae3b2e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7adc105bf0 0x7f7adc19caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:56.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.809+0000 7f7ae3b2e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc19d180 con 0x7f7adc105bf0 2024-04-22T23:42:56.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.809+0000 7f7ae3b2e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc196630 con 0x7f7adc0ffb40 2024-04-22T23:42:56.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.809+0000 7f7ae10c9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7adc105bf0 0x7f7adc19caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:56.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.809+0000 7f7ae10c9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7adc105bf0 0x7f7adc19caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52376/0 (socket says 172.21.15.67:52376) 2024-04-22T23:42:56.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.809+0000 7f7ae10c9700 1 -- 172.21.15.67:0/738136434 learned_addr learned my addr 172.21.15.67:0/738136434 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:42:56.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.809+0000 7f7ae10c9700 1 -- 172.21.15.67:0/738136434 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7adc0ffb40 msgr2=0x7f7adc19c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:56.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.809+0000 7f7ae10c9700 1 --2- 172.21.15.67:0/738136434 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7adc0ffb40 0x7f7adc19c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:56.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.809+0000 7f7ae10c9700 1 -- 172.21.15.67:0/738136434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7acc0095f0 con 0x7f7adc105bf0 2024-04-22T23:42:56.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.809+0000 7f7ae10c9700 1 --2- 172.21.15.67:0/738136434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7adc105bf0 0x7f7adc19caf0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f7acc00b5c0 tx=0x7f7acc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:42:56.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.809+0000 7f7ad2ffd700 1 -- 172.21.15.67:0/738136434 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7acc01d070 con 0x7f7adc105bf0 2024-04-22T23:42:56.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.810+0000 7f7ae3b2e700 1 -- 172.21.15.67:0/738136434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7adc1968b0 con 0x7f7adc105bf0 2024-04-22T23:42:56.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.810+0000 7f7ad2ffd700 1 -- 172.21.15.67:0/738136434 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7acc005e00 con 0x7f7adc105bf0 2024-04-22T23:42:56.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.810+0000 7f7ad2ffd700 1 -- 172.21.15.67:0/738136434 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7acc017400 con 0x7f7adc105bf0 2024-04-22T23:42:56.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.810+0000 7f7ae3b2e700 1 -- 172.21.15.67:0/738136434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7adc196da0 con 0x7f7adc105bf0 2024-04-22T23:42:56.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.810+0000 7f7ae3b2e700 1 -- 172.21.15.67:0/738136434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7adc04f2e0 con 0x7f7adc105bf0 2024-04-22T23:42:56.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.811+0000 7f7ad2ffd700 1 -- 172.21.15.67:0/738136434 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7acc00f460 con 0x7f7adc105bf0 2024-04-22T23:42:56.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.811+0000 7f7ad2ffd700 1 --2- 172.21.15.67:0/738136434 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7ac806ecc0 0x7f7ac8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:56.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.812+0000 7f7ad2ffd700 1 -- 172.21.15.67:0/738136434 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f7acc090530 con 0x7f7adc105bf0 2024-04-22T23:42:56.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.812+0000 7f7ae18ca700 1 --2- 172.21.15.67:0/738136434 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7ac806ecc0 0x7f7ac8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:56.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.812+0000 7f7ae18ca700 1 --2- 172.21.15.67:0/738136434 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7ac806ecc0 0x7f7ac8071180 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7ad8005fd0 tx=0x7f7ad8005f60 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:42:56.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:56.816+0000 7f7ad2ffd700 1 -- 172.21.15.67:0/738136434 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7acc05cca0 con 0x7f7adc105bf0 2024-04-22T23:42:57.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:42:57.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.048+0000 7f7ae3b2e700 1 -- 172.21.15.67:0/738136434 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7adc197180 con 0x7f7ac806ecc0 2024-04-22T23:42:57.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.049+0000 7f7ad2ffd700 1 -- 172.21.15.67:0/738136434 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f7adc197180 con 0x7f7ac806ecc0 2024-04-22T23:42:57.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.051+0000 7f7ae3b2e700 1 -- 172.21.15.67:0/738136434 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7ac806ecc0 msgr2=0x7f7ac8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:57.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.051+0000 7f7ae3b2e700 1 --2- 172.21.15.67:0/738136434 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7ac806ecc0 0x7f7ac8071180 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7ad8005fd0 tx=0x7f7ad8005f60 comp rx=0 tx=0).stop 2024-04-22T23:42:57.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.052+0000 7f7ae3b2e700 1 -- 172.21.15.67:0/738136434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7adc105bf0 msgr2=0x7f7adc19caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:57.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.052+0000 7f7ae3b2e700 1 --2- 172.21.15.67:0/738136434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7adc105bf0 0x7f7adc19caf0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f7acc00b5c0 tx=0x7f7acc00be20 comp rx=0 tx=0).stop 2024-04-22T23:42:57.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.052+0000 7f7ae3b2e700 1 -- 172.21.15.67:0/738136434 shutdown_connections 2024-04-22T23:42:57.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.052+0000 7f7ae3b2e700 1 --2- 172.21.15.67:0/738136434 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7ac806ecc0 0x7f7ac8071180 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:57.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.052+0000 7f7ae3b2e700 1 --2- 172.21.15.67:0/738136434 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7adc0ffb40 0x7f7adc19c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:57.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.052+0000 7f7ae3b2e700 1 --2- 172.21.15.67:0/738136434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7adc105bf0 0x7f7adc19caf0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:57.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.052+0000 7f7ae3b2e700 1 -- 172.21.15.67:0/738136434 >> 172.21.15.67:0/738136434 conn(0x7f7adc0fb650 msgr2=0x7f7adc068eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:42:57.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.052+0000 7f7ae3b2e700 1 -- 172.21.15.67:0/738136434 shutdown_connections 2024-04-22T23:42:57.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.052+0000 7f7ae3b2e700 1 -- 172.21.15.67:0/738136434 wait complete. 2024-04-22T23:42:57.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.159+0000 7f16e0169700 1 -- 172.21.15.67:0/3623245288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16d80fff50 msgr2=0x7f16d8105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:57.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.159+0000 7f16e0169700 1 --2- 172.21.15.67:0/3623245288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16d80fff50 0x7f16d8105840 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f16cc009910 tx=0x7f16cc009c20 comp rx=0 tx=0).stop 2024-04-22T23:42:57.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.159+0000 7f16e0169700 1 -- 172.21.15.67:0/3623245288 shutdown_connections 2024-04-22T23:42:57.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.159+0000 7f16e0169700 1 --2- 172.21.15.67:0/3623245288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16d80fff50 0x7f16d8105840 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:57.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.159+0000 7f16e0169700 1 --2- 172.21.15.67:0/3623245288 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f16d80ff5a0 0x7f16d80ff980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:57.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.159+0000 7f16e0169700 1 -- 172.21.15.67:0/3623245288 >> 172.21.15.67:0/3623245288 conn(0x7f16d8074e30 msgr2=0x7f16d8075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:42:57.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.159+0000 7f16e0169700 1 -- 172.21.15.67:0/3623245288 shutdown_connections 2024-04-22T23:42:57.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.159+0000 7f16e0169700 1 -- 172.21.15.67:0/3623245288 wait complete. 2024-04-22T23:42:57.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.160+0000 7f16e0169700 1 Processor -- start 2024-04-22T23:42:57.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.160+0000 7f16e0169700 1 -- start start 2024-04-22T23:42:57.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.160+0000 7f16e0169700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16d80ff5a0 0x7f16d819c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:57.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.160+0000 7f16e0169700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f16d80fff50 0x7f16d819cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:57.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.160+0000 7f16e0169700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d819d260 con 0x7f16d80ff5a0 2024-04-22T23:42:57.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.160+0000 7f16e0169700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d8196710 con 0x7f16d80fff50 2024-04-22T23:42:57.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.160+0000 7f16ddf05700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16d80ff5a0 0x7f16d819c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:57.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.160+0000 7f16ddf05700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16d80ff5a0 0x7f16d819c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52384/0 (socket says 172.21.15.67:52384) 2024-04-22T23:42:57.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.160+0000 7f16ddf05700 1 -- 172.21.15.67:0/2775153483 learned_addr learned my addr 172.21.15.67:0/2775153483 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:42:57.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.160+0000 7f16dd704700 1 --2- 172.21.15.67:0/2775153483 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f16d80fff50 0x7f16d819cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:57.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.160+0000 7f16ddf05700 1 -- 172.21.15.67:0/2775153483 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f16d80fff50 msgr2=0x7f16d819cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:57.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.161+0000 7f16ddf05700 1 --2- 172.21.15.67:0/2775153483 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f16d80fff50 0x7f16d819cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:57.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.161+0000 7f16ddf05700 1 -- 172.21.15.67:0/2775153483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16cc0095f0 con 0x7f16d80ff5a0 2024-04-22T23:42:57.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.161+0000 7f16dd704700 1 --2- 172.21.15.67:0/2775153483 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f16d80fff50 0x7f16d819cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:42:57.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.161+0000 7f16ddf05700 1 --2- 172.21.15.67:0/2775153483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16d80ff5a0 0x7f16d819c690 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f16d400c8a0 tx=0x7f16d400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:42:57.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.161+0000 7f16caffd700 1 -- 172.21.15.67:0/2775153483 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16d400cea0 con 0x7f16d80ff5a0 2024-04-22T23:42:57.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.161+0000 7f16e0169700 1 -- 172.21.15.67:0/2775153483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16d81969f0 con 0x7f16d80ff5a0 2024-04-22T23:42:57.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.161+0000 7f16caffd700 1 -- 172.21.15.67:0/2775153483 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f16d4004830 con 0x7f16d80ff5a0 2024-04-22T23:42:57.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.161+0000 7f16e0169700 1 -- 172.21.15.67:0/2775153483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16d8196f40 con 0x7f16d80ff5a0 2024-04-22T23:42:57.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.161+0000 7f16caffd700 1 -- 172.21.15.67:0/2775153483 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16d40055b0 con 0x7f16d80ff5a0 2024-04-22T23:42:57.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.162+0000 7f16e0169700 1 -- 172.21.15.67:0/2775153483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16bc005320 con 0x7f16d80ff5a0 2024-04-22T23:42:57.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.162+0000 7f16caffd700 1 -- 172.21.15.67:0/2775153483 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f16d40049a0 con 0x7f16d80ff5a0 2024-04-22T23:42:57.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.163+0000 7f16caffd700 1 --2- 172.21.15.67:0/2775153483 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f16c406ecc0 0x7f16c4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:57.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.163+0000 7f16caffd700 1 -- 172.21.15.67:0/2775153483 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f16d408eb80 con 0x7f16d80ff5a0 2024-04-22T23:42:57.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.163+0000 7f16dd704700 1 --2- 172.21.15.67:0/2775153483 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f16c406ecc0 0x7f16c4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:57.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.163+0000 7f16dd704700 1 --2- 172.21.15.67:0/2775153483 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f16c406ecc0 0x7f16c4071180 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f16d8197b80 tx=0x7f16cc01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:42:57.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.168+0000 7f16caffd700 1 -- 172.21.15.67:0/2775153483 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f16d405b370 con 0x7f16d80ff5a0 2024-04-22T23:42:57.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:42:57.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.386+0000 7f16e0169700 1 -- 172.21.15.67:0/2775153483 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f16bc000bf0 con 0x7f16c406ecc0 2024-04-22T23:42:57.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.398+0000 7f16caffd700 1 -- 172.21.15.67:0/2775153483 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f16bc000bf0 con 0x7f16c406ecc0 2024-04-22T23:42:57.391 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:42:57.391 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (16m) 9m ago 27m 25.5M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:42:57.391 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (28m) 9m ago 28m 9114k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:42:57.391 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (27m) 9m ago 27m 9042k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:42:57.391 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (14m) 9m ago 28m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:42:57.391 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (13m) 9m ago 27m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:42:57.392 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (16m) 9m ago 27m 79.3M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:42:57.392 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (10m) 9m ago 22m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:42:57.392 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (10m) 9m ago 22m 12.6M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:42:57.392 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (9m) 9m ago 22m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:42:57.392 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (22m) 9m ago 22m 12.2M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:42:57.392 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (18m) 9m ago 29m 511M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:42:57.392 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (17m) 9m ago 26m 453M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:42:57.392 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (15m) 9m ago 29m 55.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:42:57.392 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (14m) 9m ago 26m 46.0M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:42:57.392 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (17m) 9m ago 28m 15.4M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:42:57.392 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (17m) 9m ago 27m 16.1M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:42:57.392 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (13m) 9m ago 26m 99.2M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:42:57.392 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (13m) 9m ago 25m 127M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:42:57.392 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (12m) 9m ago 25m 79.9M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:42:57.392 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (11m) 9m ago 24m 97.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:42:57.393 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (11m) 9m ago 24m 82.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:42:57.393 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (10m) 9m ago 24m 77.6M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:42:57.393 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (16m) 9m ago 27m 73.9M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:42:57.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.401+0000 7f16e0169700 1 -- 172.21.15.67:0/2775153483 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f16c406ecc0 msgr2=0x7f16c4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:57.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.401+0000 7f16e0169700 1 --2- 172.21.15.67:0/2775153483 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f16c406ecc0 0x7f16c4071180 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f16d8197b80 tx=0x7f16cc01a040 comp rx=0 tx=0).stop 2024-04-22T23:42:57.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.401+0000 7f16e0169700 1 -- 172.21.15.67:0/2775153483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16d80ff5a0 msgr2=0x7f16d819c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:57.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.401+0000 7f16e0169700 1 --2- 172.21.15.67:0/2775153483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16d80ff5a0 0x7f16d819c690 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f16d400c8a0 tx=0x7f16d400cc60 comp rx=0 tx=0).stop 2024-04-22T23:42:57.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.401+0000 7f16e0169700 1 -- 172.21.15.67:0/2775153483 shutdown_connections 2024-04-22T23:42:57.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.401+0000 7f16e0169700 1 --2- 172.21.15.67:0/2775153483 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f16c406ecc0 0x7f16c4071180 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:57.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.401+0000 7f16e0169700 1 --2- 172.21.15.67:0/2775153483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16d80ff5a0 0x7f16d819c690 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:57.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.401+0000 7f16e0169700 1 --2- 172.21.15.67:0/2775153483 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f16d80fff50 0x7f16d819cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:57.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.401+0000 7f16e0169700 1 -- 172.21.15.67:0/2775153483 >> 172.21.15.67:0/2775153483 conn(0x7f16d8074e30 msgr2=0x7f16d80fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:42:57.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.401+0000 7f16e0169700 1 -- 172.21.15.67:0/2775153483 shutdown_connections 2024-04-22T23:42:57.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.401+0000 7f16e0169700 1 -- 172.21.15.67:0/2775153483 wait complete. 2024-04-22T23:42:57.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.506+0000 7ff365708700 1 -- 172.21.15.67:0/1793468614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff360101e00 msgr2=0x7ff3601021e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:57.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.506+0000 7ff365708700 1 --2- 172.21.15.67:0/1793468614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff360101e00 0x7ff3601021e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ff350009910 tx=0x7ff350009c20 comp rx=0 tx=0).stop 2024-04-22T23:42:57.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.506+0000 7ff365708700 1 -- 172.21.15.67:0/1793468614 shutdown_connections 2024-04-22T23:42:57.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.506+0000 7ff365708700 1 --2- 172.21.15.67:0/1793468614 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff360102720 0x7ff36010ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:57.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.506+0000 7ff365708700 1 --2- 172.21.15.67:0/1793468614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff360101e00 0x7ff3601021e0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:57.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.507+0000 7ff365708700 1 -- 172.21.15.67:0/1793468614 >> 172.21.15.67:0/1793468614 conn(0x7ff3600fb650 msgr2=0x7ff3600fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:42:57.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.507+0000 7ff365708700 1 -- 172.21.15.67:0/1793468614 shutdown_connections 2024-04-22T23:42:57.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.507+0000 7ff365708700 1 -- 172.21.15.67:0/1793468614 wait complete. 2024-04-22T23:42:57.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.507+0000 7ff365708700 1 Processor -- start 2024-04-22T23:42:57.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.507+0000 7ff365708700 1 -- start start 2024-04-22T23:42:57.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.508+0000 7ff365708700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff360101e00 0x7ff3600ffa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:57.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.508+0000 7ff365708700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff360102720 0x7ff3600fffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:57.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.508+0000 7ff365708700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3601004e0 con 0x7ff360102720 2024-04-22T23:42:57.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.508+0000 7ff365708700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff360100620 con 0x7ff360101e00 2024-04-22T23:42:57.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.508+0000 7ff35e7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff360102720 0x7ff3600fffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:57.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.508+0000 7ff35e7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff360102720 0x7ff3600fffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52402/0 (socket says 172.21.15.67:52402) 2024-04-22T23:42:57.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.508+0000 7ff35e7fc700 1 -- 172.21.15.67:0/3042853415 learned_addr learned my addr 172.21.15.67:0/3042853415 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:42:57.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.508+0000 7ff35effd700 1 --2- 172.21.15.67:0/3042853415 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff360101e00 0x7ff3600ffa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:57.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.508+0000 7ff35e7fc700 1 -- 172.21.15.67:0/3042853415 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff360101e00 msgr2=0x7ff3600ffa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:57.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.508+0000 7ff35e7fc700 1 --2- 172.21.15.67:0/3042853415 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff360101e00 0x7ff3600ffa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:57.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.508+0000 7ff35e7fc700 1 -- 172.21.15.67:0/3042853415 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3500095f0 con 0x7ff360102720 2024-04-22T23:42:57.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.508+0000 7ff35e7fc700 1 --2- 172.21.15.67:0/3042853415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff360102720 0x7ff3600fffa0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff35400d8d0 tx=0x7ff35400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:42:57.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.508+0000 7ff347fff700 1 -- 172.21.15.67:0/3042853415 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff354009900 con 0x7ff360102720 2024-04-22T23:42:57.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.508+0000 7ff365708700 1 -- 172.21.15.67:0/3042853415 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3601a2ae0 con 0x7ff360102720 2024-04-22T23:42:57.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.508+0000 7ff347fff700 1 -- 172.21.15.67:0/3042853415 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff354010460 con 0x7ff360102720 2024-04-22T23:42:57.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.508+0000 7ff347fff700 1 -- 172.21.15.67:0/3042853415 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff35400f5f0 con 0x7ff360102720 2024-04-22T23:42:57.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.509+0000 7ff365708700 1 -- 172.21.15.67:0/3042853415 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3601a2e90 con 0x7ff360102720 2024-04-22T23:42:57.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.509+0000 7ff365708700 1 -- 172.21.15.67:0/3042853415 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff360108310 con 0x7ff360102720 2024-04-22T23:42:57.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.510+0000 7ff347fff700 1 -- 172.21.15.67:0/3042853415 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff3540105d0 con 0x7ff360102720 2024-04-22T23:42:57.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.510+0000 7ff347fff700 1 --2- 172.21.15.67:0/3042853415 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff34806ecc0 0x7ff348071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:57.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.511+0000 7ff347fff700 1 -- 172.21.15.67:0/3042853415 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7ff35408ec60 con 0x7ff360102720 2024-04-22T23:42:57.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.511+0000 7ff35effd700 1 --2- 172.21.15.67:0/3042853415 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff34806ecc0 0x7ff348071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:57.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.511+0000 7ff35effd700 1 --2- 172.21.15.67:0/3042853415 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff34806ecc0 0x7ff348071180 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff35000b5c0 tx=0x7ff35001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:42:57.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.515+0000 7ff347fff700 1 -- 172.21.15.67:0/3042853415 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff3540578a0 con 0x7ff360102720 2024-04-22T23:42:57.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.817+0000 7ff365708700 1 -- 172.21.15.67:0/3042853415 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff3601007e0 con 0x7ff360102720 2024-04-22T23:42:57.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.817+0000 7ff347fff700 1 -- 172.21.15.67:0/3042853415 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7ff354057070 con 0x7ff360102720 2024-04-22T23:42:57.810 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:42:57.810 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:42:57.810 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:42:57.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:42:57.810 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:42:57.810 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:42:57.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:42:57.810 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:42:57.810 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:42:57.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:42:57.811 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:42:57.811 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:42:57.811 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 3 2024-04-22T23:42:57.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:42:57.811 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:42:57.811 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:42:57.811 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 13 2024-04-22T23:42:57.811 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:42:57.811 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:42:57.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.819+0000 7ff365708700 1 -- 172.21.15.67:0/3042853415 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff34806ecc0 msgr2=0x7ff348071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:57.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.819+0000 7ff365708700 1 --2- 172.21.15.67:0/3042853415 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff34806ecc0 0x7ff348071180 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff35000b5c0 tx=0x7ff35001a040 comp rx=0 tx=0).stop 2024-04-22T23:42:57.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.819+0000 7ff365708700 1 -- 172.21.15.67:0/3042853415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff360102720 msgr2=0x7ff3600fffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:57.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.819+0000 7ff365708700 1 --2- 172.21.15.67:0/3042853415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff360102720 0x7ff3600fffa0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff35400d8d0 tx=0x7ff35400dc90 comp rx=0 tx=0).stop 2024-04-22T23:42:57.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.820+0000 7ff365708700 1 -- 172.21.15.67:0/3042853415 shutdown_connections 2024-04-22T23:42:57.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.820+0000 7ff365708700 1 --2- 172.21.15.67:0/3042853415 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff34806ecc0 0x7ff348071180 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:57.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.820+0000 7ff365708700 1 --2- 172.21.15.67:0/3042853415 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff360101e00 0x7ff3600ffa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:57.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.820+0000 7ff365708700 1 --2- 172.21.15.67:0/3042853415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff360102720 0x7ff3600fffa0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:57.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.820+0000 7ff365708700 1 -- 172.21.15.67:0/3042853415 >> 172.21.15.67:0/3042853415 conn(0x7ff3600fb650 msgr2=0x7ff360105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:42:57.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.820+0000 7ff365708700 1 -- 172.21.15.67:0/3042853415 shutdown_connections 2024-04-22T23:42:57.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.820+0000 7ff365708700 1 -- 172.21.15.67:0/3042853415 wait complete. 2024-04-22T23:42:57.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.926+0000 7fe19f5c9700 1 -- 172.21.15.67:0/4110758719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe198100750 msgr2=0x7fe1981047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:57.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.926+0000 7fe19f5c9700 1 --2- 172.21.15.67:0/4110758719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe198100750 0x7fe1981047a0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fe188009910 tx=0x7fe188009c20 comp rx=0 tx=0).stop 2024-04-22T23:42:57.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.926+0000 7fe19f5c9700 1 -- 172.21.15.67:0/4110758719 shutdown_connections 2024-04-22T23:42:57.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.926+0000 7fe19f5c9700 1 --2- 172.21.15.67:0/4110758719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe198100750 0x7fe1981047a0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:57.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.926+0000 7fe19f5c9700 1 --2- 172.21.15.67:0/4110758719 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1980ffda0 0x7fe198100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:57.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.926+0000 7fe19f5c9700 1 -- 172.21.15.67:0/4110758719 >> 172.21.15.67:0/4110758719 conn(0x7fe1980fb650 msgr2=0x7fe1980fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:42:57.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.927+0000 7fe19f5c9700 1 -- 172.21.15.67:0/4110758719 shutdown_connections 2024-04-22T23:42:57.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.927+0000 7fe19f5c9700 1 -- 172.21.15.67:0/4110758719 wait complete. 2024-04-22T23:42:57.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.927+0000 7fe19f5c9700 1 Processor -- start 2024-04-22T23:42:57.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.927+0000 7fe19f5c9700 1 -- start start 2024-04-22T23:42:57.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.927+0000 7fe19f5c9700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1980ffda0 0x7fe198101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:57.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.927+0000 7fe19f5c9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe198100750 0x7fe198101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:57.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.927+0000 7fe19f5c9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1981022c0 con 0x7fe198100750 2024-04-22T23:42:57.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.927+0000 7fe19f5c9700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe198102400 con 0x7fe1980ffda0 2024-04-22T23:42:57.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.928+0000 7fe19cb64700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe198100750 0x7fe198101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:57.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.928+0000 7fe19cb64700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe198100750 0x7fe198101d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52422/0 (socket says 172.21.15.67:52422) 2024-04-22T23:42:57.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.928+0000 7fe19cb64700 1 -- 172.21.15.67:0/100501811 learned_addr learned my addr 172.21.15.67:0/100501811 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:42:57.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.928+0000 7fe19d365700 1 --2- 172.21.15.67:0/100501811 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1980ffda0 0x7fe198101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:57.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.928+0000 7fe19cb64700 1 -- 172.21.15.67:0/100501811 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1980ffda0 msgr2=0x7fe198101840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:57.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.928+0000 7fe19cb64700 1 --2- 172.21.15.67:0/100501811 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1980ffda0 0x7fe198101840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:57.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.928+0000 7fe19cb64700 1 -- 172.21.15.67:0/100501811 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1880095f0 con 0x7fe198100750 2024-04-22T23:42:57.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.928+0000 7fe19d365700 1 --2- 172.21.15.67:0/100501811 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1980ffda0 0x7fe198101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:42:57.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.928+0000 7fe19cb64700 1 --2- 172.21.15.67:0/100501811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe198100750 0x7fe198101d80 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fe18800bda0 tx=0x7fe18800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:42:57.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.928+0000 7fe18e7fc700 1 -- 172.21.15.67:0/100501811 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe18801d070 con 0x7fe198100750 2024-04-22T23:42:57.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.928+0000 7fe18e7fc700 1 -- 172.21.15.67:0/100501811 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe188004fb0 con 0x7fe198100750 2024-04-22T23:42:57.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.928+0000 7fe19f5c9700 1 -- 172.21.15.67:0/100501811 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe198102680 con 0x7fe198100750 2024-04-22T23:42:57.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.928+0000 7fe18e7fc700 1 -- 172.21.15.67:0/100501811 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe18800f480 con 0x7fe198100750 2024-04-22T23:42:57.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.928+0000 7fe19f5c9700 1 -- 172.21.15.67:0/100501811 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1981a2d50 con 0x7fe198100750 2024-04-22T23:42:57.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.929+0000 7fe19f5c9700 1 -- 172.21.15.67:0/100501811 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1981091b0 con 0x7fe198100750 2024-04-22T23:42:57.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.930+0000 7fe18e7fc700 1 -- 172.21.15.67:0/100501811 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe188022870 con 0x7fe198100750 2024-04-22T23:42:57.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.930+0000 7fe18e7fc700 1 --2- 172.21.15.67:0/100501811 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe18406ecc0 0x7fe184071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:57.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.930+0000 7fe18e7fc700 1 -- 172.21.15.67:0/100501811 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fe188090420 con 0x7fe198100750 2024-04-22T23:42:57.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.930+0000 7fe19d365700 1 --2- 172.21.15.67:0/100501811 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe18406ecc0 0x7fe184071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:57.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.931+0000 7fe19d365700 1 --2- 172.21.15.67:0/100501811 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe18406ecc0 0x7fe184071180 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe194005fd0 tx=0x7fe194005e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:42:57.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:57.935+0000 7fe18e7fc700 1 -- 172.21.15.67:0/100501811 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe188059190 con 0x7fe198100750 2024-04-22T23:42:58.075 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:57 smithi067 ceph-mon[112438]: pgmap v497: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:58.075 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:57 smithi067 ceph-mon[112438]: from='client.34932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:42:58.075 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:57 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3042853415' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:42:58.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.191+0000 7fe19f5c9700 1 -- 172.21.15.67:0/100501811 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe198108b90 con 0x7fe198100750 2024-04-22T23:42:58.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.191+0000 7fe18e7fc700 1 -- 172.21.15.67:0/100501811 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1871 (secure 0 0 0) 0x7fe188027760 con 0x7fe198100750 2024-04-22T23:42:58.184 INFO:teuthology.orchestra.run.smithi067.stdout:e31 2024-04-22T23:42:58.184 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:42:58.184 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:42:58.185 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:42:58.185 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:42:58.185 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:42:58.185 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:42:58.185 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:42:58.185 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:42:58.185 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:42:58.185 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:42:58.185 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:42:58.185 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:42:58.185 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:42:58.185 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:42:58.185 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:42:58.185 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:42:58.185 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:42:58.185 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:42:58.186 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:42:58.186 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:42:58.186 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:42:58.186 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:42:58.186 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:42:58.186 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:42:58.186 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:42:58.186 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:42:58.186 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:42:58.186 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:42:58.186 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:42:58.186 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:42:58.186 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:42:58.186 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:42:58.187 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:42:58.187 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:42:58.187 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:42:58.187 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:42:58.187 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:42:58.187 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:42:58.187 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:42:58.187 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44259} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1827288036,v1:172.21.15.134:6827/1827288036] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:42:58.187 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:42:58.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.194+0000 7fe19f5c9700 1 -- 172.21.15.67:0/100501811 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe18406ecc0 msgr2=0x7fe184071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:58.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.194+0000 7fe19f5c9700 1 --2- 172.21.15.67:0/100501811 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe18406ecc0 0x7fe184071180 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe194005fd0 tx=0x7fe194005e30 comp rx=0 tx=0).stop 2024-04-22T23:42:58.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.194+0000 7fe19f5c9700 1 -- 172.21.15.67:0/100501811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe198100750 msgr2=0x7fe198101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:58.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.194+0000 7fe19f5c9700 1 --2- 172.21.15.67:0/100501811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe198100750 0x7fe198101d80 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fe18800bda0 tx=0x7fe18800be80 comp rx=0 tx=0).stop 2024-04-22T23:42:58.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.194+0000 7fe19f5c9700 1 -- 172.21.15.67:0/100501811 shutdown_connections 2024-04-22T23:42:58.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.194+0000 7fe19f5c9700 1 --2- 172.21.15.67:0/100501811 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe18406ecc0 0x7fe184071180 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:58.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.194+0000 7fe19f5c9700 1 --2- 172.21.15.67:0/100501811 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1980ffda0 0x7fe198101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:58.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.194+0000 7fe19f5c9700 1 --2- 172.21.15.67:0/100501811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe198100750 0x7fe198101d80 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:58.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.194+0000 7fe19f5c9700 1 -- 172.21.15.67:0/100501811 >> 172.21.15.67:0/100501811 conn(0x7fe1980fb650 msgr2=0x7fe198069150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:42:58.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.194+0000 7fe19f5c9700 1 -- 172.21.15.67:0/100501811 shutdown_connections 2024-04-22T23:42:58.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.194+0000 7fe19f5c9700 1 -- 172.21.15.67:0/100501811 wait complete. 2024-04-22T23:42:58.189 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 31 2024-04-22T23:42:58.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:57 smithi134 ceph-mon[85630]: pgmap v497: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:42:58.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:57 smithi134 ceph-mon[85630]: from='client.34932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:42:58.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:57 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3042853415' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:42:58.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.302+0000 7fef26895700 1 -- 172.21.15.67:0/311379691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef20102780 msgr2=0x7fef2010ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:58.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.302+0000 7fef26895700 1 --2- 172.21.15.67:0/311379691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef20102780 0x7fef2010ac70 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fef10009940 tx=0x7fef10009c50 comp rx=0 tx=0).stop 2024-04-22T23:42:58.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.302+0000 7fef26895700 1 -- 172.21.15.67:0/311379691 shutdown_connections 2024-04-22T23:42:58.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.302+0000 7fef26895700 1 --2- 172.21.15.67:0/311379691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef20102780 0x7fef2010ac70 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:58.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.302+0000 7fef26895700 1 --2- 172.21.15.67:0/311379691 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef20101e60 0x7fef20102240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:58.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.302+0000 7fef26895700 1 -- 172.21.15.67:0/311379691 >> 172.21.15.67:0/311379691 conn(0x7fef200fb5f0 msgr2=0x7fef200fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:42:58.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.302+0000 7fef26895700 1 -- 172.21.15.67:0/311379691 shutdown_connections 2024-04-22T23:42:58.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.302+0000 7fef26895700 1 -- 172.21.15.67:0/311379691 wait complete. 2024-04-22T23:42:58.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.302+0000 7fef26895700 1 Processor -- start 2024-04-22T23:42:58.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.303+0000 7fef26895700 1 -- start start 2024-04-22T23:42:58.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.303+0000 7fef26895700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef20101e60 0x7fef2019c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:58.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.303+0000 7fef26895700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef20102780 0x7fef2019cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:58.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.303+0000 7fef26895700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2019d410 con 0x7fef20101e60 2024-04-22T23:42:58.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.303+0000 7fef26895700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef201968c0 con 0x7fef20102780 2024-04-22T23:42:58.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.303+0000 7fef1ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef20101e60 0x7fef2019c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:58.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.303+0000 7fef1ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef20101e60 0x7fef2019c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52450/0 (socket says 172.21.15.67:52450) 2024-04-22T23:42:58.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.303+0000 7fef1ffff700 1 -- 172.21.15.67:0/4123545925 learned_addr learned my addr 172.21.15.67:0/4123545925 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:42:58.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.303+0000 7fef1ffff700 1 -- 172.21.15.67:0/4123545925 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef20102780 msgr2=0x7fef2019cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:58.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.303+0000 7fef1ffff700 1 --2- 172.21.15.67:0/4123545925 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef20102780 0x7fef2019cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:58.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.303+0000 7fef1ffff700 1 -- 172.21.15.67:0/4123545925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef100095f0 con 0x7fef20101e60 2024-04-22T23:42:58.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.303+0000 7fef1ffff700 1 --2- 172.21.15.67:0/4123545925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef20101e60 0x7fef2019c840 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fef1400b4b0 tx=0x7fef1400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:42:58.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.304+0000 7fef1d7fa700 1 -- 172.21.15.67:0/4123545925 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef14011800 con 0x7fef20101e60 2024-04-22T23:42:58.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.304+0000 7fef1d7fa700 1 -- 172.21.15.67:0/4123545925 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef14011e20 con 0x7fef20101e60 2024-04-22T23:42:58.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.304+0000 7fef26895700 1 -- 172.21.15.67:0/4123545925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef20196ba0 con 0x7fef20101e60 2024-04-22T23:42:58.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.304+0000 7fef1d7fa700 1 -- 172.21.15.67:0/4123545925 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef1400f2e0 con 0x7fef20101e60 2024-04-22T23:42:58.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.304+0000 7fef26895700 1 -- 172.21.15.67:0/4123545925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef201970f0 con 0x7fef20101e60 2024-04-22T23:42:58.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.305+0000 7fef26895700 1 -- 172.21.15.67:0/4123545925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef20108370 con 0x7fef20101e60 2024-04-22T23:42:58.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.305+0000 7fef1d7fa700 1 -- 172.21.15.67:0/4123545925 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fef14010380 con 0x7fef20101e60 2024-04-22T23:42:58.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.306+0000 7fef1d7fa700 1 --2- 172.21.15.67:0/4123545925 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fef0806ecc0 0x7fef08071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:58.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.306+0000 7fef1d7fa700 1 -- 172.21.15.67:0/4123545925 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fef1408eb00 con 0x7fef20101e60 2024-04-22T23:42:58.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.306+0000 7fef1f7fe700 1 --2- 172.21.15.67:0/4123545925 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fef0806ecc0 0x7fef08071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:58.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.306+0000 7fef1f7fe700 1 --2- 172.21.15.67:0/4123545925 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fef0806ecc0 0x7fef08071180 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fef10006010 tx=0x7fef10000bc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:42:58.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.311+0000 7fef1d7fa700 1 -- 172.21.15.67:0/4123545925 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fef1405b3a0 con 0x7fef20101e60 2024-04-22T23:42:58.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.544+0000 7fef26895700 1 -- 172.21.15.67:0/4123545925 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fef20197e70 con 0x7fef0806ecc0 2024-04-22T23:42:58.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.545+0000 7fef1d7fa700 1 -- 172.21.15.67:0/4123545925 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fef20197e70 con 0x7fef0806ecc0 2024-04-22T23:42:58.538 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:42:58.538 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:42:58.538 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:42:58.538 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:42:58.538 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:42:58.538 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:42:58.538 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:42:58.538 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:42:58.539 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:42:58.539 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:42:58.539 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T23:42:58.539 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:42:58.539 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:42:58.539 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:42:58.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.548+0000 7fef26895700 1 -- 172.21.15.67:0/4123545925 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fef0806ecc0 msgr2=0x7fef08071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:58.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.548+0000 7fef26895700 1 --2- 172.21.15.67:0/4123545925 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fef0806ecc0 0x7fef08071180 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fef10006010 tx=0x7fef10000bc0 comp rx=0 tx=0).stop 2024-04-22T23:42:58.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.548+0000 7fef26895700 1 -- 172.21.15.67:0/4123545925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef20101e60 msgr2=0x7fef2019c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:58.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.548+0000 7fef26895700 1 --2- 172.21.15.67:0/4123545925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef20101e60 0x7fef2019c840 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fef1400b4b0 tx=0x7fef1400b870 comp rx=0 tx=0).stop 2024-04-22T23:42:58.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.548+0000 7fef26895700 1 -- 172.21.15.67:0/4123545925 shutdown_connections 2024-04-22T23:42:58.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.548+0000 7fef26895700 1 --2- 172.21.15.67:0/4123545925 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fef0806ecc0 0x7fef08071180 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:58.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.548+0000 7fef26895700 1 --2- 172.21.15.67:0/4123545925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef20101e60 0x7fef2019c840 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:58.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.548+0000 7fef26895700 1 --2- 172.21.15.67:0/4123545925 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef20102780 0x7fef2019cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:58.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.548+0000 7fef26895700 1 -- 172.21.15.67:0/4123545925 >> 172.21.15.67:0/4123545925 conn(0x7fef200fb5f0 msgr2=0x7fef201054b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:42:58.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.548+0000 7fef26895700 1 -- 172.21.15.67:0/4123545925 shutdown_connections 2024-04-22T23:42:58.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.548+0000 7fef26895700 1 -- 172.21.15.67:0/4123545925 wait complete. 2024-04-22T23:42:58.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.656+0000 7f098c8e9700 1 -- 172.21.15.67:0/3397449326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09880ffc20 msgr2=0x7f0988100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:58.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.656+0000 7f098c8e9700 1 --2- 172.21.15.67:0/3397449326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09880ffc20 0x7f0988100080 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f097c009910 tx=0x7f097c009c20 comp rx=0 tx=0).stop 2024-04-22T23:42:58.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.656+0000 7f098c8e9700 1 -- 172.21.15.67:0/3397449326 shutdown_connections 2024-04-22T23:42:58.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.656+0000 7f098c8e9700 1 --2- 172.21.15.67:0/3397449326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09880ffc20 0x7f0988100080 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:58.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.656+0000 7f098c8e9700 1 --2- 172.21.15.67:0/3397449326 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0988105bd0 0x7f0988105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:58.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.656+0000 7f098c8e9700 1 -- 172.21.15.67:0/3397449326 >> 172.21.15.67:0/3397449326 conn(0x7f0988075670 msgr2=0x7f0988075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:42:58.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.656+0000 7f098c8e9700 1 -- 172.21.15.67:0/3397449326 shutdown_connections 2024-04-22T23:42:58.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.656+0000 7f098c8e9700 1 -- 172.21.15.67:0/3397449326 wait complete. 2024-04-22T23:42:58.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.657+0000 7f098c8e9700 1 Processor -- start 2024-04-22T23:42:58.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.657+0000 7f098c8e9700 1 -- start start 2024-04-22T23:42:58.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.658+0000 7f098c8e9700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09880ffc20 0x7f0988101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:58.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.658+0000 7f098c8e9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0988105bd0 0x7f0988101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:58.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.658+0000 7f098c8e9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09881022c0 con 0x7f0988105bd0 2024-04-22T23:42:58.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.658+0000 7f098c8e9700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0988102400 con 0x7f09880ffc20 2024-04-22T23:42:58.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.658+0000 7f0985d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0988105bd0 0x7f0988101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:58.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.658+0000 7f0985d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0988105bd0 0x7f0988101d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52482/0 (socket says 172.21.15.67:52482) 2024-04-22T23:42:58.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.658+0000 7f0985d9b700 1 -- 172.21.15.67:0/3320577979 learned_addr learned my addr 172.21.15.67:0/3320577979 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:42:58.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.658+0000 7f098659c700 1 --2- 172.21.15.67:0/3320577979 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09880ffc20 0x7f0988101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:58.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.658+0000 7f0985d9b700 1 -- 172.21.15.67:0/3320577979 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09880ffc20 msgr2=0x7f0988101840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:58.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.658+0000 7f0985d9b700 1 --2- 172.21.15.67:0/3320577979 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09880ffc20 0x7f0988101840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:58.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.658+0000 7f0985d9b700 1 -- 172.21.15.67:0/3320577979 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f097c0095f0 con 0x7f0988105bd0 2024-04-22T23:42:58.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.658+0000 7f0985d9b700 1 --2- 172.21.15.67:0/3320577979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0988105bd0 0x7f0988101d80 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f097c006010 tx=0x7f097c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:42:58.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.658+0000 7f09777fe700 1 -- 172.21.15.67:0/3320577979 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f097c01d070 con 0x7f0988105bd0 2024-04-22T23:42:58.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.659+0000 7f098c8e9700 1 -- 172.21.15.67:0/3320577979 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0988102680 con 0x7f0988105bd0 2024-04-22T23:42:58.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.659+0000 7f09777fe700 1 -- 172.21.15.67:0/3320577979 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f097c00f460 con 0x7f0988105bd0 2024-04-22T23:42:58.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.659+0000 7f09777fe700 1 -- 172.21.15.67:0/3320577979 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f097c017610 con 0x7f0988105bd0 2024-04-22T23:42:58.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.659+0000 7f098c8e9700 1 -- 172.21.15.67:0/3320577979 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09881a2eb0 con 0x7f0988105bd0 2024-04-22T23:42:58.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.660+0000 7f098c8e9700 1 -- 172.21.15.67:0/3320577979 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f098804f2e0 con 0x7f0988105bd0 2024-04-22T23:42:58.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.660+0000 7f09777fe700 1 -- 172.21.15.67:0/3320577979 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f097c00f5d0 con 0x7f0988105bd0 2024-04-22T23:42:58.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.660+0000 7f09777fe700 1 --2- 172.21.15.67:0/3320577979 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f097006ecc0 0x7f0970071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:42:58.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.661+0000 7f09777fe700 1 -- 172.21.15.67:0/3320577979 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f097c090510 con 0x7f0988105bd0 2024-04-22T23:42:58.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.661+0000 7f098659c700 1 --2- 172.21.15.67:0/3320577979 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f097006ecc0 0x7f0970071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:42:58.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.661+0000 7f098659c700 1 --2- 172.21.15.67:0/3320577979 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f097006ecc0 0x7f0970071180 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0978009a10 tx=0x7f0978009190 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:42:58.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.665+0000 7f09777fe700 1 -- 172.21.15.67:0/3320577979 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f097c059150 con 0x7f0988105bd0 2024-04-22T23:42:58.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.965+0000 7f098c8e9700 1 -- 172.21.15.67:0/3320577979 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f09881a30d0 con 0x7f0988105bd0 2024-04-22T23:42:58.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.965+0000 7f09777fe700 1 -- 172.21.15.67:0/3320577979 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f097c05c810 con 0x7f0988105bd0 2024-04-22T23:42:58.958 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:42:58.958 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:42:58.958 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:42:58.958 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:42:58.958 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:42:58.958 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:42:58.958 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:42:58.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.967+0000 7f098c8e9700 1 -- 172.21.15.67:0/3320577979 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f097006ecc0 msgr2=0x7f0970071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:58.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.967+0000 7f098c8e9700 1 --2- 172.21.15.67:0/3320577979 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f097006ecc0 0x7f0970071180 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0978009a10 tx=0x7f0978009190 comp rx=0 tx=0).stop 2024-04-22T23:42:58.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.967+0000 7f098c8e9700 1 -- 172.21.15.67:0/3320577979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0988105bd0 msgr2=0x7f0988101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:42:58.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.967+0000 7f098c8e9700 1 --2- 172.21.15.67:0/3320577979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0988105bd0 0x7f0988101d80 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f097c006010 tx=0x7f097c00bd70 comp rx=0 tx=0).stop 2024-04-22T23:42:58.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.967+0000 7f098c8e9700 1 -- 172.21.15.67:0/3320577979 shutdown_connections 2024-04-22T23:42:58.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.967+0000 7f098c8e9700 1 --2- 172.21.15.67:0/3320577979 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f097006ecc0 0x7f0970071180 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:58.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.967+0000 7f098c8e9700 1 --2- 172.21.15.67:0/3320577979 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f09880ffc20 0x7f0988101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:58.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.967+0000 7f098c8e9700 1 --2- 172.21.15.67:0/3320577979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0988105bd0 0x7f0988101d80 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:42:58.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.967+0000 7f098c8e9700 1 -- 172.21.15.67:0/3320577979 >> 172.21.15.67:0/3320577979 conn(0x7f0988075670 msgr2=0x7f09880686e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:42:58.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.967+0000 7f098c8e9700 1 -- 172.21.15.67:0/3320577979 shutdown_connections 2024-04-22T23:42:58.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:42:58.967+0000 7f098c8e9700 1 -- 172.21.15.67:0/3320577979 wait complete. 2024-04-22T23:42:59.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:58 smithi134 ceph-mon[85630]: from='client.34936 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:42:59.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:58 smithi134 ceph-mon[85630]: from='client.34940 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:42:59.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:58 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/100501811' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:42:59.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:58 smithi067 ceph-mon[112438]: from='client.34936 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:42:59.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:58 smithi067 ceph-mon[112438]: from='client.34940 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:42:59.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:58 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/100501811' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:43:00.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:59 smithi134 ceph-mon[85630]: pgmap v498: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:00.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:59 smithi134 ceph-mon[85630]: from='client.34952 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:43:00.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:42:59 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3320577979' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:43:00.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:59 smithi067 ceph-mon[112438]: pgmap v498: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:00.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:59 smithi067 ceph-mon[112438]: from='client.34952 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:43:00.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:42:59 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3320577979' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:43:02.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:01 smithi134 ceph-mon[85630]: pgmap v499: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:02.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:01 smithi067 ceph-mon[112438]: pgmap v499: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:04.238 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:03 smithi134 ceph-mon[85630]: pgmap v500: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:04.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:03 smithi067 ceph-mon[112438]: pgmap v500: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:06.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:05 smithi134 ceph-mon[85630]: pgmap v501: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:06.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:05 smithi067 ceph-mon[112438]: pgmap v501: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:08.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:07 smithi134 ceph-mon[85630]: pgmap v502: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:08.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:07 smithi067 ceph-mon[112438]: pgmap v502: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:10.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:09 smithi134 ceph-mon[85630]: pgmap v503: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:10.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:09 smithi067 ceph-mon[112438]: pgmap v503: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:12.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:11 smithi134 ceph-mon[85630]: pgmap v504: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:12.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:43:12.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:11 smithi067 ceph-mon[112438]: pgmap v504: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:12.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:43:14.173 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:13 smithi067 ceph-mon[112438]: pgmap v505: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:14.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:13 smithi134 ceph-mon[85630]: pgmap v505: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:16.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:15 smithi134 ceph-mon[85630]: pgmap v506: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:16.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:15 smithi067 ceph-mon[112438]: pgmap v506: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:18.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:17 smithi134 ceph-mon[85630]: pgmap v507: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:18.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:17 smithi067 ceph-mon[112438]: pgmap v507: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:20.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:19 smithi134 ceph-mon[85630]: pgmap v508: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:20.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:19 smithi067 ceph-mon[112438]: pgmap v508: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:22.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:21 smithi134 ceph-mon[85630]: pgmap v509: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:22.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:21 smithi067 ceph-mon[112438]: pgmap v509: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:23.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:23 smithi134 ceph-mon[85630]: pgmap v510: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:24.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:23 smithi067 ceph-mon[112438]: pgmap v510: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:26.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:25 smithi134 ceph-mon[85630]: pgmap v511: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:26.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:25 smithi067 ceph-mon[112438]: pgmap v511: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:27.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:43:27.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:43:28.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:27 smithi134 ceph-mon[85630]: pgmap v512: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:28.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:27 smithi067 ceph-mon[112438]: pgmap v512: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:29.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.085+0000 7f9284099700 1 -- 172.21.15.67:0/364815351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f927c100ea0 msgr2=0x7f927c101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:43:29.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.085+0000 7f9284099700 1 --2- 172.21.15.67:0/364815351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f927c100ea0 0x7f927c101280 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f9278009910 tx=0x7f9278009c20 comp rx=0 tx=0).stop 2024-04-22T23:43:29.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.085+0000 7f9284099700 1 -- 172.21.15.67:0/364815351 shutdown_connections 2024-04-22T23:43:29.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.085+0000 7f9284099700 1 --2- 172.21.15.67:0/364815351 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f927c101850 0x7f927c1058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:29.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.085+0000 7f9284099700 1 --2- 172.21.15.67:0/364815351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f927c100ea0 0x7f927c101280 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:29.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.085+0000 7f9284099700 1 -- 172.21.15.67:0/364815351 >> 172.21.15.67:0/364815351 conn(0x7f927c0fc750 msgr2=0x7f927c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:43:29.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.085+0000 7f9284099700 1 -- 172.21.15.67:0/364815351 shutdown_connections 2024-04-22T23:43:29.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.085+0000 7f9284099700 1 -- 172.21.15.67:0/364815351 wait complete. 2024-04-22T23:43:29.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.086+0000 7f9284099700 1 Processor -- start 2024-04-22T23:43:29.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.086+0000 7f9284099700 1 -- start start 2024-04-22T23:43:29.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.086+0000 7f9284099700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f927c100ea0 0x7f927c19e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:43:29.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.086+0000 7f9284099700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f927c101850 0x7f927c19edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:43:29.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.086+0000 7f9284099700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f927c19f440 con 0x7f927c100ea0 2024-04-22T23:43:29.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.086+0000 7f9284099700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f927c1988f0 con 0x7f927c101850 2024-04-22T23:43:29.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.086+0000 7f9281e35700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f927c100ea0 0x7f927c19e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:43:29.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.086+0000 7f9281e35700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f927c100ea0 0x7f927c19e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41302/0 (socket says 172.21.15.67:41302) 2024-04-22T23:43:29.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.086+0000 7f9281e35700 1 -- 172.21.15.67:0/3269308910 learned_addr learned my addr 172.21.15.67:0/3269308910 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:43:29.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.086+0000 7f9281634700 1 --2- 172.21.15.67:0/3269308910 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f927c101850 0x7f927c19edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:43:29.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.086+0000 7f9281e35700 1 -- 172.21.15.67:0/3269308910 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f927c101850 msgr2=0x7f927c19edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:43:29.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.086+0000 7f9281e35700 1 --2- 172.21.15.67:0/3269308910 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f927c101850 0x7f927c19edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:29.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.086+0000 7f9281e35700 1 -- 172.21.15.67:0/3269308910 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92780095f0 con 0x7f927c100ea0 2024-04-22T23:43:29.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.087+0000 7f9281e35700 1 --2- 172.21.15.67:0/3269308910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f927c100ea0 0x7f927c19e870 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f9278006010 tx=0x7f927800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:43:29.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.087+0000 7f9272ffd700 1 -- 172.21.15.67:0/3269308910 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f927801d070 con 0x7f927c100ea0 2024-04-22T23:43:29.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.087+0000 7f9272ffd700 1 -- 172.21.15.67:0/3269308910 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9278004fb0 con 0x7f927c100ea0 2024-04-22T23:43:29.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.087+0000 7f9284099700 1 -- 172.21.15.67:0/3269308910 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f927c198b70 con 0x7f927c100ea0 2024-04-22T23:43:29.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.087+0000 7f9272ffd700 1 -- 172.21.15.67:0/3269308910 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9278022ea0 con 0x7f927c100ea0 2024-04-22T23:43:29.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.087+0000 7f9284099700 1 -- 172.21.15.67:0/3269308910 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f927c199060 con 0x7f927c100ea0 2024-04-22T23:43:29.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.088+0000 7f9284099700 1 -- 172.21.15.67:0/3269308910 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f927c105020 con 0x7f927c100ea0 2024-04-22T23:43:29.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.088+0000 7f9272ffd700 1 -- 172.21.15.67:0/3269308910 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9278022470 con 0x7f927c100ea0 2024-04-22T23:43:29.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.089+0000 7f9272ffd700 1 --2- 172.21.15.67:0/3269308910 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f926806ecc0 0x7f9268071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:43:29.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.089+0000 7f9272ffd700 1 -- 172.21.15.67:0/3269308910 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f92780904c0 con 0x7f927c100ea0 2024-04-22T23:43:29.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.089+0000 7f9281634700 1 --2- 172.21.15.67:0/3269308910 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f926806ecc0 0x7f9268071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:43:29.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.089+0000 7f9281634700 1 --2- 172.21.15.67:0/3269308910 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f926806ecc0 0x7f9268071180 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f926c005fd0 tx=0x7f926c00a560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:43:29.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.094+0000 7f9272ffd700 1 -- 172.21.15.67:0/3269308910 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f927805cd60 con 0x7f927c100ea0 2024-04-22T23:43:29.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.325+0000 7f9284099700 1 -- 172.21.15.67:0/3269308910 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f927c199ec0 con 0x7f926806ecc0 2024-04-22T23:43:29.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.326+0000 7f9272ffd700 1 -- 172.21.15.67:0/3269308910 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f927c199ec0 con 0x7f926806ecc0 2024-04-22T23:43:29.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.328+0000 7f9284099700 1 -- 172.21.15.67:0/3269308910 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f926806ecc0 msgr2=0x7f9268071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:43:29.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.328+0000 7f9284099700 1 --2- 172.21.15.67:0/3269308910 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f926806ecc0 0x7f9268071180 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f926c005fd0 tx=0x7f926c00a560 comp rx=0 tx=0).stop 2024-04-22T23:43:29.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.328+0000 7f9284099700 1 -- 172.21.15.67:0/3269308910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f927c100ea0 msgr2=0x7f927c19e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:43:29.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.328+0000 7f9284099700 1 --2- 172.21.15.67:0/3269308910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f927c100ea0 0x7f927c19e870 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f9278006010 tx=0x7f927800be20 comp rx=0 tx=0).stop 2024-04-22T23:43:29.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.328+0000 7f9284099700 1 -- 172.21.15.67:0/3269308910 shutdown_connections 2024-04-22T23:43:29.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.328+0000 7f9284099700 1 --2- 172.21.15.67:0/3269308910 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f926806ecc0 0x7f9268071180 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:29.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.328+0000 7f9284099700 1 --2- 172.21.15.67:0/3269308910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f927c100ea0 0x7f927c19e870 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:29.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.328+0000 7f9284099700 1 --2- 172.21.15.67:0/3269308910 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f927c101850 0x7f927c19edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:29.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.328+0000 7f9284099700 1 -- 172.21.15.67:0/3269308910 >> 172.21.15.67:0/3269308910 conn(0x7f927c0fc750 msgr2=0x7f927c0fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:43:29.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.329+0000 7f9284099700 1 -- 172.21.15.67:0/3269308910 shutdown_connections 2024-04-22T23:43:29.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.329+0000 7f9284099700 1 -- 172.21.15.67:0/3269308910 wait complete. 2024-04-22T23:43:29.331 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:43:29.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.438+0000 7f7efafc8700 1 -- 172.21.15.67:0/3731365263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ef40ff340 msgr2=0x7f7ef40ff7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:43:29.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.438+0000 7f7efafc8700 1 --2- 172.21.15.67:0/3731365263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ef40ff340 0x7f7ef40ff7a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f7ee800aa20 tx=0x7f7ee800ad30 comp rx=0 tx=0).stop 2024-04-22T23:43:29.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.438+0000 7f7efafc8700 1 -- 172.21.15.67:0/3731365263 shutdown_connections 2024-04-22T23:43:29.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.438+0000 7f7efafc8700 1 --2- 172.21.15.67:0/3731365263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ef40ff340 0x7f7ef40ff7a0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:29.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.438+0000 7f7efafc8700 1 --2- 172.21.15.67:0/3731365263 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7ef4105350 0x7f7ef4105730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:29.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.438+0000 7f7efafc8700 1 -- 172.21.15.67:0/3731365263 >> 172.21.15.67:0/3731365263 conn(0x7f7ef4074e30 msgr2=0x7f7ef4075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:43:29.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.438+0000 7f7efafc8700 1 -- 172.21.15.67:0/3731365263 shutdown_connections 2024-04-22T23:43:29.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.438+0000 7f7efafc8700 1 -- 172.21.15.67:0/3731365263 wait complete. 2024-04-22T23:43:29.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.438+0000 7f7efafc8700 1 Processor -- start 2024-04-22T23:43:29.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.439+0000 7f7efafc8700 1 -- start start 2024-04-22T23:43:29.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.439+0000 7f7efafc8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ef40ff340 0x7f7ef419c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:43:29.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.439+0000 7f7efafc8700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7ef4105350 0x7f7ef419cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:43:29.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.439+0000 7f7efafc8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef419d1f0 con 0x7f7ef40ff340 2024-04-22T23:43:29.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.439+0000 7f7efafc8700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef41966a0 con 0x7f7ef4105350 2024-04-22T23:43:29.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.439+0000 7f7ef8d64700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ef40ff340 0x7f7ef419c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:43:29.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.439+0000 7f7ef8d64700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ef40ff340 0x7f7ef419c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41332/0 (socket says 172.21.15.67:41332) 2024-04-22T23:43:29.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.439+0000 7f7ef8d64700 1 -- 172.21.15.67:0/3766790706 learned_addr learned my addr 172.21.15.67:0/3766790706 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:43:29.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.439+0000 7f7ef3fff700 1 --2- 172.21.15.67:0/3766790706 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7ef4105350 0x7f7ef419cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:43:29.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.439+0000 7f7ef8d64700 1 -- 172.21.15.67:0/3766790706 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7ef4105350 msgr2=0x7f7ef419cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:43:29.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.439+0000 7f7ef8d64700 1 --2- 172.21.15.67:0/3766790706 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7ef4105350 0x7f7ef419cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:29.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.439+0000 7f7ef8d64700 1 -- 172.21.15.67:0/3766790706 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee800a700 con 0x7f7ef40ff340 2024-04-22T23:43:29.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.439+0000 7f7ef8d64700 1 --2- 172.21.15.67:0/3766790706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ef40ff340 0x7f7ef419c620 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f7ee400c8a0 tx=0x7f7ee400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:43:29.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.440+0000 7f7ef1ffb700 1 -- 172.21.15.67:0/3766790706 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ee40078e0 con 0x7f7ef40ff340 2024-04-22T23:43:29.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.440+0000 7f7ef1ffb700 1 -- 172.21.15.67:0/3766790706 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7ee400f450 con 0x7f7ef40ff340 2024-04-22T23:43:29.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.440+0000 7f7ef1ffb700 1 -- 172.21.15.67:0/3766790706 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ee400e5e0 con 0x7f7ef40ff340 2024-04-22T23:43:29.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.440+0000 7f7efafc8700 1 -- 172.21.15.67:0/3766790706 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ef4196980 con 0x7f7ef40ff340 2024-04-22T23:43:29.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.440+0000 7f7efafc8700 1 -- 172.21.15.67:0/3766790706 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ef4196ea0 con 0x7f7ef40ff340 2024-04-22T23:43:29.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.441+0000 7f7efafc8700 1 -- 172.21.15.67:0/3766790706 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ef4109860 con 0x7f7ef40ff340 2024-04-22T23:43:29.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.441+0000 7f7ef1ffb700 1 -- 172.21.15.67:0/3766790706 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7ee400f5c0 con 0x7f7ef40ff340 2024-04-22T23:43:29.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.442+0000 7f7ef1ffb700 1 --2- 172.21.15.67:0/3766790706 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7edc06ecc0 0x7f7edc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:43:29.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.442+0000 7f7ef3fff700 1 --2- 172.21.15.67:0/3766790706 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7edc06ecc0 0x7f7edc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:43:29.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.442+0000 7f7ef1ffb700 1 -- 172.21.15.67:0/3766790706 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f7ee408ee30 con 0x7f7ef40ff340 2024-04-22T23:43:29.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.442+0000 7f7ef3fff700 1 --2- 172.21.15.67:0/3766790706 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7edc06ecc0 0x7f7edc071180 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8000c00 tx=0x7f7ee8005de0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:43:29.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.447+0000 7f7ef1ffb700 1 -- 172.21.15.67:0/3766790706 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7ee405b620 con 0x7f7ef40ff340 2024-04-22T23:43:29.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.676+0000 7f7efafc8700 1 -- 172.21.15.67:0/3766790706 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ef4197170 con 0x7f7edc06ecc0 2024-04-22T23:43:29.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.678+0000 7f7ef1ffb700 1 -- 172.21.15.67:0/3766790706 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f7ef4197170 con 0x7f7edc06ecc0 2024-04-22T23:43:29.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.680+0000 7f7efafc8700 1 -- 172.21.15.67:0/3766790706 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7edc06ecc0 msgr2=0x7f7edc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:43:29.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.680+0000 7f7efafc8700 1 --2- 172.21.15.67:0/3766790706 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7edc06ecc0 0x7f7edc071180 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8000c00 tx=0x7f7ee8005de0 comp rx=0 tx=0).stop 2024-04-22T23:43:29.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.680+0000 7f7efafc8700 1 -- 172.21.15.67:0/3766790706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ef40ff340 msgr2=0x7f7ef419c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:43:29.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.680+0000 7f7efafc8700 1 --2- 172.21.15.67:0/3766790706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ef40ff340 0x7f7ef419c620 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f7ee400c8a0 tx=0x7f7ee400cbb0 comp rx=0 tx=0).stop 2024-04-22T23:43:29.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.680+0000 7f7efafc8700 1 -- 172.21.15.67:0/3766790706 shutdown_connections 2024-04-22T23:43:29.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.680+0000 7f7efafc8700 1 --2- 172.21.15.67:0/3766790706 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7edc06ecc0 0x7f7edc071180 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:29.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.680+0000 7f7efafc8700 1 --2- 172.21.15.67:0/3766790706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ef40ff340 0x7f7ef419c620 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:29.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.680+0000 7f7efafc8700 1 --2- 172.21.15.67:0/3766790706 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7ef4105350 0x7f7ef419cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:29.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.680+0000 7f7efafc8700 1 -- 172.21.15.67:0/3766790706 >> 172.21.15.67:0/3766790706 conn(0x7f7ef4074e30 msgr2=0x7f7ef4103870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:43:29.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.680+0000 7f7efafc8700 1 -- 172.21.15.67:0/3766790706 shutdown_connections 2024-04-22T23:43:29.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.680+0000 7f7efafc8700 1 -- 172.21.15.67:0/3766790706 wait complete. 2024-04-22T23:43:29.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.787+0000 7fc864dad700 1 -- 172.21.15.67:0/2607670907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc860102720 msgr2=0x7fc86010ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:43:29.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.787+0000 7fc864dad700 1 --2- 172.21.15.67:0/2607670907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc860102720 0x7fc86010ac10 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fc848009910 tx=0x7fc848009c20 comp rx=0 tx=0).stop 2024-04-22T23:43:29.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.787+0000 7fc864dad700 1 -- 172.21.15.67:0/2607670907 shutdown_connections 2024-04-22T23:43:29.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.787+0000 7fc864dad700 1 --2- 172.21.15.67:0/2607670907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc860102720 0x7fc86010ac10 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:29.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.787+0000 7fc864dad700 1 --2- 172.21.15.67:0/2607670907 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc860101e00 0x7fc8601021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:29.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.787+0000 7fc864dad700 1 -- 172.21.15.67:0/2607670907 >> 172.21.15.67:0/2607670907 conn(0x7fc8600fb650 msgr2=0x7fc8600fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:43:29.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.787+0000 7fc864dad700 1 -- 172.21.15.67:0/2607670907 shutdown_connections 2024-04-22T23:43:29.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.787+0000 7fc864dad700 1 -- 172.21.15.67:0/2607670907 wait complete. 2024-04-22T23:43:29.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.788+0000 7fc864dad700 1 Processor -- start 2024-04-22T23:43:29.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.788+0000 7fc864dad700 1 -- start start 2024-04-22T23:43:29.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.788+0000 7fc864dad700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc860101e00 0x7fc86019e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:43:29.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.788+0000 7fc864dad700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc860102720 0x7fc86019edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:43:29.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.788+0000 7fc864dad700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc86019f440 con 0x7fc860102720 2024-04-22T23:43:29.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.788+0000 7fc864dad700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8601988f0 con 0x7fc860101e00 2024-04-22T23:43:29.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.788+0000 7fc85dd9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc860102720 0x7fc86019edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:43:29.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.788+0000 7fc85dd9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc860102720 0x7fc86019edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41356/0 (socket says 172.21.15.67:41356) 2024-04-22T23:43:29.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.788+0000 7fc85dd9b700 1 -- 172.21.15.67:0/1182599892 learned_addr learned my addr 172.21.15.67:0/1182599892 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:43:29.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.789+0000 7fc85e59c700 1 --2- 172.21.15.67:0/1182599892 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc860101e00 0x7fc86019e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:43:29.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.789+0000 7fc85dd9b700 1 -- 172.21.15.67:0/1182599892 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc860101e00 msgr2=0x7fc86019e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:43:29.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.789+0000 7fc85dd9b700 1 --2- 172.21.15.67:0/1182599892 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc860101e00 0x7fc86019e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:29.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.789+0000 7fc85dd9b700 1 -- 172.21.15.67:0/1182599892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8480095f0 con 0x7fc860102720 2024-04-22T23:43:29.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.789+0000 7fc85dd9b700 1 --2- 172.21.15.67:0/1182599892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc860102720 0x7fc86019edb0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fc84800bd70 tx=0x7fc84800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:43:29.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.789+0000 7fc8577fe700 1 -- 172.21.15.67:0/1182599892 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc84801d070 con 0x7fc860102720 2024-04-22T23:43:29.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.789+0000 7fc8577fe700 1 -- 172.21.15.67:0/1182599892 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc848005dc0 con 0x7fc860102720 2024-04-22T23:43:29.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.789+0000 7fc864dad700 1 -- 172.21.15.67:0/1182599892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc860198b70 con 0x7fc860102720 2024-04-22T23:43:29.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.789+0000 7fc8577fe700 1 -- 172.21.15.67:0/1182599892 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc848017400 con 0x7fc860102720 2024-04-22T23:43:29.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.789+0000 7fc864dad700 1 -- 172.21.15.67:0/1182599892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc860199060 con 0x7fc860102720 2024-04-22T23:43:29.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.790+0000 7fc864dad700 1 -- 172.21.15.67:0/1182599892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc86010a770 con 0x7fc860102720 2024-04-22T23:43:29.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.790+0000 7fc8577fe700 1 -- 172.21.15.67:0/1182599892 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc848017560 con 0x7fc860102720 2024-04-22T23:43:29.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.791+0000 7fc8577fe700 1 --2- 172.21.15.67:0/1182599892 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc84c06ecc0 0x7fc84c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:43:29.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.791+0000 7fc8577fe700 1 -- 172.21.15.67:0/1182599892 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fc848090420 con 0x7fc860102720 2024-04-22T23:43:29.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.791+0000 7fc85e59c700 1 --2- 172.21.15.67:0/1182599892 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc84c06ecc0 0x7fc84c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:43:29.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.791+0000 7fc85e59c700 1 --2- 172.21.15.67:0/1182599892 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc84c06ecc0 0x7fc84c071180 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc858009c10 tx=0x7fc858009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:43:29.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:29.796+0000 7fc8577fe700 1 -- 172.21.15.67:0/1182599892 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc84805cbb0 con 0x7fc860102720 2024-04-22T23:43:30.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.015+0000 7fc864dad700 1 -- 172.21.15.67:0/1182599892 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc860199390 con 0x7fc84c06ecc0 2024-04-22T23:43:30.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.027+0000 7fc8577fe700 1 -- 172.21.15.67:0/1182599892 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fc860199390 con 0x7fc84c06ecc0 2024-04-22T23:43:30.019 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:43:30.019 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (17m) 10m ago 28m 25.5M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:43:30.019 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (28m) 10m ago 28m 9114k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:43:30.019 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (27m) 10m ago 27m 9042k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:43:30.019 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (14m) 10m ago 28m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:43:30.020 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (14m) 10m ago 27m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:43:30.020 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (16m) 10m ago 28m 79.3M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:43:30.020 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (10m) 10m ago 23m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:43:30.020 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (10m) 10m ago 22m 12.6M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:43:30.020 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (10m) 10m ago 23m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:43:30.020 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (22m) 10m ago 22m 12.2M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:43:30.020 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (18m) 10m ago 29m 511M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:43:30.020 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (18m) 10m ago 27m 453M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:43:30.020 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (15m) 10m ago 30m 55.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:43:30.020 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (15m) 10m ago 27m 46.0M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:43:30.020 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (17m) 10m ago 28m 15.4M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:43:30.020 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (17m) 10m ago 27m 16.1M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:43:30.020 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (14m) 10m ago 26m 99.2M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:43:30.020 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (13m) 10m ago 26m 127M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:43:30.020 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (12m) 10m ago 25m 79.9M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:43:30.021 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (12m) 10m ago 25m 97.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:43:30.021 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (11m) 10m ago 25m 82.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:43:30.021 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (11m) 10m ago 24m 77.6M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:43:30.021 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (17m) 10m ago 28m 73.9M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:43:30.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.029+0000 7fc864dad700 1 -- 172.21.15.67:0/1182599892 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc84c06ecc0 msgr2=0x7fc84c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:43:30.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.029+0000 7fc864dad700 1 --2- 172.21.15.67:0/1182599892 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc84c06ecc0 0x7fc84c071180 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc858009c10 tx=0x7fc858009040 comp rx=0 tx=0).stop 2024-04-22T23:43:30.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.029+0000 7fc864dad700 1 -- 172.21.15.67:0/1182599892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc860102720 msgr2=0x7fc86019edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:43:30.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.029+0000 7fc864dad700 1 --2- 172.21.15.67:0/1182599892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc860102720 0x7fc86019edb0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fc84800bd70 tx=0x7fc84800be50 comp rx=0 tx=0).stop 2024-04-22T23:43:30.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.029+0000 7fc864dad700 1 -- 172.21.15.67:0/1182599892 shutdown_connections 2024-04-22T23:43:30.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.029+0000 7fc864dad700 1 --2- 172.21.15.67:0/1182599892 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc84c06ecc0 0x7fc84c071180 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:30.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.029+0000 7fc864dad700 1 --2- 172.21.15.67:0/1182599892 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc860101e00 0x7fc86019e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:30.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.029+0000 7fc864dad700 1 --2- 172.21.15.67:0/1182599892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc860102720 0x7fc86019edb0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:30.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.029+0000 7fc864dad700 1 -- 172.21.15.67:0/1182599892 >> 172.21.15.67:0/1182599892 conn(0x7fc8600fb650 msgr2=0x7fc860100220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:43:30.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.030+0000 7fc864dad700 1 -- 172.21.15.67:0/1182599892 shutdown_connections 2024-04-22T23:43:30.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.030+0000 7fc864dad700 1 -- 172.21.15.67:0/1182599892 wait complete. 2024-04-22T23:43:30.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.135+0000 7f35b35e8700 1 -- 172.21.15.67:0/1529332423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35ac0ffc20 msgr2=0x7f35ac100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:43:30.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.135+0000 7f35b35e8700 1 --2- 172.21.15.67:0/1529332423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35ac0ffc20 0x7f35ac100080 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f35a0009910 tx=0x7f35a0009c20 comp rx=0 tx=0).stop 2024-04-22T23:43:30.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.135+0000 7f35b35e8700 1 -- 172.21.15.67:0/1529332423 shutdown_connections 2024-04-22T23:43:30.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.135+0000 7f35b35e8700 1 --2- 172.21.15.67:0/1529332423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35ac0ffc20 0x7f35ac100080 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:30.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.135+0000 7f35b35e8700 1 --2- 172.21.15.67:0/1529332423 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f35ac105bd0 0x7f35ac105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:30.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.135+0000 7f35b35e8700 1 -- 172.21.15.67:0/1529332423 >> 172.21.15.67:0/1529332423 conn(0x7f35ac075670 msgr2=0x7f35ac075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:43:30.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.135+0000 7f35b35e8700 1 -- 172.21.15.67:0/1529332423 shutdown_connections 2024-04-22T23:43:30.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.135+0000 7f35b35e8700 1 -- 172.21.15.67:0/1529332423 wait complete. 2024-04-22T23:43:30.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.136+0000 7f35b35e8700 1 Processor -- start 2024-04-22T23:43:30.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.136+0000 7f35b35e8700 1 -- start start 2024-04-22T23:43:30.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.136+0000 7f35b35e8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35ac0ffc20 0x7f35ac101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:43:30.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.136+0000 7f35b35e8700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f35ac105bd0 0x7f35ac101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:43:30.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.136+0000 7f35b35e8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ac1022c0 con 0x7f35ac0ffc20 2024-04-22T23:43:30.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.136+0000 7f35b35e8700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ac102400 con 0x7f35ac105bd0 2024-04-22T23:43:30.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.136+0000 7f35b1384700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35ac0ffc20 0x7f35ac101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:43:30.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.136+0000 7f35b1384700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35ac0ffc20 0x7f35ac101840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41372/0 (socket says 172.21.15.67:41372) 2024-04-22T23:43:30.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.136+0000 7f35b1384700 1 -- 172.21.15.67:0/1048546026 learned_addr learned my addr 172.21.15.67:0/1048546026 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:43:30.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.137+0000 7f35b0b83700 1 --2- 172.21.15.67:0/1048546026 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f35ac105bd0 0x7f35ac101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:43:30.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.137+0000 7f35b1384700 1 -- 172.21.15.67:0/1048546026 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f35ac105bd0 msgr2=0x7f35ac101d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:43:30.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.137+0000 7f35b1384700 1 --2- 172.21.15.67:0/1048546026 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f35ac105bd0 0x7f35ac101d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:30.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.137+0000 7f35b1384700 1 -- 172.21.15.67:0/1048546026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a00095f0 con 0x7f35ac0ffc20 2024-04-22T23:43:30.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.137+0000 7f35b1384700 1 --2- 172.21.15.67:0/1048546026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35ac0ffc20 0x7f35ac101840 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f35a400c930 tx=0x7f35a400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:43:30.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.137+0000 7f35aa7fc700 1 -- 172.21.15.67:0/1048546026 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35a4007ad0 con 0x7f35ac0ffc20 2024-04-22T23:43:30.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.137+0000 7f35aa7fc700 1 -- 172.21.15.67:0/1048546026 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f35a4007c30 con 0x7f35ac0ffc20 2024-04-22T23:43:30.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.137+0000 7f35b35e8700 1 -- 172.21.15.67:0/1048546026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35ac1a2a60 con 0x7f35ac0ffc20 2024-04-22T23:43:30.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.137+0000 7f35aa7fc700 1 -- 172.21.15.67:0/1048546026 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35a4018610 con 0x7f35ac0ffc20 2024-04-22T23:43:30.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.137+0000 7f35b35e8700 1 -- 172.21.15.67:0/1048546026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35ac1a2dc0 con 0x7f35ac0ffc20 2024-04-22T23:43:30.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.138+0000 7f35b35e8700 1 -- 172.21.15.67:0/1048546026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35ac04f2e0 con 0x7f35ac0ffc20 2024-04-22T23:43:30.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.139+0000 7f35aa7fc700 1 -- 172.21.15.67:0/1048546026 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f35a400f450 con 0x7f35ac0ffc20 2024-04-22T23:43:30.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.139+0000 7f35aa7fc700 1 --2- 172.21.15.67:0/1048546026 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f359406ecc0 0x7f3594071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:43:30.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.139+0000 7f35aa7fc700 1 -- 172.21.15.67:0/1048546026 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f35a408f120 con 0x7f35ac0ffc20 2024-04-22T23:43:30.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.139+0000 7f35b0b83700 1 --2- 172.21.15.67:0/1048546026 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f359406ecc0 0x7f3594071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:43:30.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.140+0000 7f35b0b83700 1 --2- 172.21.15.67:0/1048546026 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f359406ecc0 0x7f3594071180 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f35a000b5c0 tx=0x7f35a0005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:43:30.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.144+0000 7f35aa7fc700 1 -- 172.21.15.67:0/1048546026 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f35a4057de0 con 0x7f35ac0ffc20 2024-04-22T23:43:30.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:29 smithi134 ceph-mon[85630]: pgmap v513: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:30.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:29 smithi067 ceph-mon[112438]: pgmap v513: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:30.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.445+0000 7f35b35e8700 1 -- 172.21.15.67:0/1048546026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f35ac1025c0 con 0x7f35ac0ffc20 2024-04-22T23:43:30.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.446+0000 7f35aa7fc700 1 -- 172.21.15.67:0/1048546026 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f35a405b4a0 con 0x7f35ac0ffc20 2024-04-22T23:43:30.438 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:43:30.438 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:43:30.438 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:43:30.438 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:43:30.439 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:43:30.439 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:43:30.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:43:30.439 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:43:30.439 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:43:30.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:43:30.439 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:43:30.439 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:43:30.439 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 3 2024-04-22T23:43:30.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:43:30.439 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:43:30.439 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:43:30.439 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 13 2024-04-22T23:43:30.439 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:43:30.440 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:43:30.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.448+0000 7f35b35e8700 1 -- 172.21.15.67:0/1048546026 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f359406ecc0 msgr2=0x7f3594071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:43:30.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.448+0000 7f35b35e8700 1 --2- 172.21.15.67:0/1048546026 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f359406ecc0 0x7f3594071180 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f35a000b5c0 tx=0x7f35a0005960 comp rx=0 tx=0).stop 2024-04-22T23:43:30.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.448+0000 7f35b35e8700 1 -- 172.21.15.67:0/1048546026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35ac0ffc20 msgr2=0x7f35ac101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:43:30.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.448+0000 7f35b35e8700 1 --2- 172.21.15.67:0/1048546026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35ac0ffc20 0x7f35ac101840 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f35a400c930 tx=0x7f35a400ccf0 comp rx=0 tx=0).stop 2024-04-22T23:43:30.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.448+0000 7f35b35e8700 1 -- 172.21.15.67:0/1048546026 shutdown_connections 2024-04-22T23:43:30.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.448+0000 7f35b35e8700 1 --2- 172.21.15.67:0/1048546026 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f359406ecc0 0x7f3594071180 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:30.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.448+0000 7f35b35e8700 1 --2- 172.21.15.67:0/1048546026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35ac0ffc20 0x7f35ac101840 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:30.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.448+0000 7f35b35e8700 1 --2- 172.21.15.67:0/1048546026 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f35ac105bd0 0x7f35ac101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:30.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.448+0000 7f35b35e8700 1 -- 172.21.15.67:0/1048546026 >> 172.21.15.67:0/1048546026 conn(0x7f35ac075670 msgr2=0x7f35ac0686a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:43:30.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.448+0000 7f35b35e8700 1 -- 172.21.15.67:0/1048546026 shutdown_connections 2024-04-22T23:43:30.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.448+0000 7f35b35e8700 1 -- 172.21.15.67:0/1048546026 wait complete. 2024-04-22T23:43:30.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.554+0000 7febef760700 1 -- 172.21.15.67:0/2614347064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe8105bf0 msgr2=0x7febe8105fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:43:30.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.554+0000 7febef760700 1 --2- 172.21.15.67:0/2614347064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe8105bf0 0x7febe8105fd0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7febd8009910 tx=0x7febd8009c20 comp rx=0 tx=0).stop 2024-04-22T23:43:30.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.554+0000 7febef760700 1 -- 172.21.15.67:0/2614347064 shutdown_connections 2024-04-22T23:43:30.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.554+0000 7febef760700 1 --2- 172.21.15.67:0/2614347064 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7febe80ffb40 0x7febe80fffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:30.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.554+0000 7febef760700 1 --2- 172.21.15.67:0/2614347064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe8105bf0 0x7febe8105fd0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:30.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.554+0000 7febef760700 1 -- 172.21.15.67:0/2614347064 >> 172.21.15.67:0/2614347064 conn(0x7febe80fb610 msgr2=0x7febe80fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:43:30.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.554+0000 7febef760700 1 -- 172.21.15.67:0/2614347064 shutdown_connections 2024-04-22T23:43:30.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.554+0000 7febef760700 1 -- 172.21.15.67:0/2614347064 wait complete. 2024-04-22T23:43:30.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.555+0000 7febef760700 1 Processor -- start 2024-04-22T23:43:30.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.555+0000 7febef760700 1 -- start start 2024-04-22T23:43:30.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.555+0000 7febef760700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7febe80ffb40 0x7febe819c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:43:30.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.555+0000 7febef760700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe8105bf0 0x7febe819cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:43:30.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.555+0000 7febef760700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe819d0e0 con 0x7febe8105bf0 2024-04-22T23:43:30.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.555+0000 7febef760700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe8196670 con 0x7febe80ffb40 2024-04-22T23:43:30.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.555+0000 7febeccfb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe8105bf0 0x7febe819cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:43:30.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.556+0000 7febeccfb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe8105bf0 0x7febe819cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41388/0 (socket says 172.21.15.67:41388) 2024-04-22T23:43:30.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.556+0000 7febeccfb700 1 -- 172.21.15.67:0/765670676 learned_addr learned my addr 172.21.15.67:0/765670676 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:43:30.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.556+0000 7febed4fc700 1 --2- 172.21.15.67:0/765670676 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7febe80ffb40 0x7febe819c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:43:30.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.556+0000 7febeccfb700 1 -- 172.21.15.67:0/765670676 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7febe80ffb40 msgr2=0x7febe819c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:43:30.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.556+0000 7febeccfb700 1 --2- 172.21.15.67:0/765670676 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7febe80ffb40 0x7febe819c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:30.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.556+0000 7febeccfb700 1 -- 172.21.15.67:0/765670676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd80095f0 con 0x7febe8105bf0 2024-04-22T23:43:30.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.556+0000 7febeccfb700 1 --2- 172.21.15.67:0/765670676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe8105bf0 0x7febe819cae0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7febe400c930 tx=0x7febe400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:43:30.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.556+0000 7febde7fc700 1 -- 172.21.15.67:0/765670676 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7febe4007ad0 con 0x7febe8105bf0 2024-04-22T23:43:30.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.556+0000 7febde7fc700 1 -- 172.21.15.67:0/765670676 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7febe400ce80 con 0x7febe8105bf0 2024-04-22T23:43:30.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.556+0000 7febde7fc700 1 -- 172.21.15.67:0/765670676 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7febe4018630 con 0x7febe8105bf0 2024-04-22T23:43:30.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.556+0000 7febef760700 1 -- 172.21.15.67:0/765670676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febe8196950 con 0x7febe8105bf0 2024-04-22T23:43:30.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.556+0000 7febef760700 1 -- 172.21.15.67:0/765670676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febe8196ea0 con 0x7febe8105bf0 2024-04-22T23:43:30.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.557+0000 7febef760700 1 -- 172.21.15.67:0/765670676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febe804f2e0 con 0x7febe8105bf0 2024-04-22T23:43:30.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.558+0000 7febde7fc700 1 -- 172.21.15.67:0/765670676 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7febe4007c30 con 0x7febe8105bf0 2024-04-22T23:43:30.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.558+0000 7febde7fc700 1 --2- 172.21.15.67:0/765670676 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7febd406ecc0 0x7febd4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:43:30.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.558+0000 7febde7fc700 1 -- 172.21.15.67:0/765670676 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7febe408ef40 con 0x7febe8105bf0 2024-04-22T23:43:30.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.558+0000 7febed4fc700 1 --2- 172.21.15.67:0/765670676 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7febd406ecc0 0x7febd4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:43:30.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.559+0000 7febed4fc700 1 --2- 172.21.15.67:0/765670676 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7febd406ecc0 0x7febd4071180 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7febd8006010 tx=0x7febd801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:43:30.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.563+0000 7febde7fc700 1 -- 172.21.15.67:0/765670676 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7febe40c4150 con 0x7febe8105bf0 2024-04-22T23:43:30.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.819+0000 7febef760700 1 -- 172.21.15.67:0/765670676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7febe8197b30 con 0x7febe8105bf0 2024-04-22T23:43:30.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.819+0000 7febde7fc700 1 -- 172.21.15.67:0/765670676 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1871 (secure 0 0 0) 0x7febe405b730 con 0x7febe8105bf0 2024-04-22T23:43:30.812 INFO:teuthology.orchestra.run.smithi067.stdout:e31 2024-04-22T23:43:30.812 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:43:30.812 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:43:30.812 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:43:30.812 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:43:30.812 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:43:30.812 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:43:30.812 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:43:30.812 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:43:30.812 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:43:30.813 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:43:30.813 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:43:30.813 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:43:30.813 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:43:30.813 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:43:30.813 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:43:30.813 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:43:30.813 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:43:30.813 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:43:30.813 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:43:30.813 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:43:30.813 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:43:30.813 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:43:30.813 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:43:30.814 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:43:30.814 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:43:30.814 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:43:30.814 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:43:30.814 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:43:30.814 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:43:30.814 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:43:30.814 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:43:30.814 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:43:30.814 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:43:30.814 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:43:30.814 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:43:30.814 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:43:30.814 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:43:30.815 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:43:30.815 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:43:30.815 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44259} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1827288036,v1:172.21.15.134:6827/1827288036] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:43:30.815 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:43:30.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.821+0000 7febef760700 1 -- 172.21.15.67:0/765670676 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7febd406ecc0 msgr2=0x7febd4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:43:30.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.821+0000 7febef760700 1 --2- 172.21.15.67:0/765670676 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7febd406ecc0 0x7febd4071180 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7febd8006010 tx=0x7febd801a040 comp rx=0 tx=0).stop 2024-04-22T23:43:30.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.821+0000 7febef760700 1 -- 172.21.15.67:0/765670676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe8105bf0 msgr2=0x7febe819cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:43:30.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.821+0000 7febef760700 1 --2- 172.21.15.67:0/765670676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe8105bf0 0x7febe819cae0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7febe400c930 tx=0x7febe400cc40 comp rx=0 tx=0).stop 2024-04-22T23:43:30.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.821+0000 7febef760700 1 -- 172.21.15.67:0/765670676 shutdown_connections 2024-04-22T23:43:30.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.821+0000 7febef760700 1 --2- 172.21.15.67:0/765670676 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7febd406ecc0 0x7febd4071180 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:30.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.821+0000 7febef760700 1 --2- 172.21.15.67:0/765670676 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7febe80ffb40 0x7febe819c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:30.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.821+0000 7febef760700 1 --2- 172.21.15.67:0/765670676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe8105bf0 0x7febe819cae0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:30.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.821+0000 7febef760700 1 -- 172.21.15.67:0/765670676 >> 172.21.15.67:0/765670676 conn(0x7febe80fb610 msgr2=0x7febe8068f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:43:30.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.821+0000 7febef760700 1 -- 172.21.15.67:0/765670676 shutdown_connections 2024-04-22T23:43:30.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.821+0000 7febef760700 1 -- 172.21.15.67:0/765670676 wait complete. 2024-04-22T23:43:30.816 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 31 2024-04-22T23:43:30.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.929+0000 7f8bfda9d700 1 -- 172.21.15.67:0/3406207916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bf8101e00 msgr2=0x7f8bf81021e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:43:30.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.929+0000 7f8bfda9d700 1 --2- 172.21.15.67:0/3406207916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bf8101e00 0x7f8bf81021e0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f8be800b0a0 tx=0x7f8be800b3b0 comp rx=0 tx=0).stop 2024-04-22T23:43:30.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.929+0000 7f8bfda9d700 1 -- 172.21.15.67:0/3406207916 shutdown_connections 2024-04-22T23:43:30.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.929+0000 7f8bfda9d700 1 --2- 172.21.15.67:0/3406207916 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8bf8102720 0x7f8bf810ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:30.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.929+0000 7f8bfda9d700 1 --2- 172.21.15.67:0/3406207916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bf8101e00 0x7f8bf81021e0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:30.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.929+0000 7f8bfda9d700 1 -- 172.21.15.67:0/3406207916 >> 172.21.15.67:0/3406207916 conn(0x7f8bf80fb650 msgr2=0x7f8bf80fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:43:30.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.929+0000 7f8bfda9d700 1 -- 172.21.15.67:0/3406207916 shutdown_connections 2024-04-22T23:43:30.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.929+0000 7f8bfda9d700 1 -- 172.21.15.67:0/3406207916 wait complete. 2024-04-22T23:43:30.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.929+0000 7f8bfda9d700 1 Processor -- start 2024-04-22T23:43:30.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.930+0000 7f8bfda9d700 1 -- start start 2024-04-22T23:43:30.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.930+0000 7f8bfda9d700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8bf8101e00 0x7f8bf819c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:43:30.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.930+0000 7f8bfda9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bf8102720 0x7f8bf819cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:43:30.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.930+0000 7f8bfda9d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf819d260 con 0x7f8bf8102720 2024-04-22T23:43:30.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.930+0000 7f8bfda9d700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf8196710 con 0x7f8bf8101e00 2024-04-22T23:43:30.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.930+0000 7f8bf6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bf8102720 0x7f8bf819cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:43:30.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.930+0000 7f8bf6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bf8102720 0x7f8bf819cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41410/0 (socket says 172.21.15.67:41410) 2024-04-22T23:43:30.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.930+0000 7f8bf6ffd700 1 -- 172.21.15.67:0/3314029879 learned_addr learned my addr 172.21.15.67:0/3314029879 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:43:30.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.930+0000 7f8bf77fe700 1 --2- 172.21.15.67:0/3314029879 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8bf8101e00 0x7f8bf819c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:43:30.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.930+0000 7f8bf6ffd700 1 -- 172.21.15.67:0/3314029879 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8bf8101e00 msgr2=0x7f8bf819c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:43:30.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.930+0000 7f8bf6ffd700 1 --2- 172.21.15.67:0/3314029879 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8bf8101e00 0x7f8bf819c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:30.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.930+0000 7f8bf6ffd700 1 -- 172.21.15.67:0/3314029879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be8009d00 con 0x7f8bf8102720 2024-04-22T23:43:30.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.930+0000 7f8bf6ffd700 1 --2- 172.21.15.67:0/3314029879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bf8102720 0x7f8bf819cbd0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f8be000d8d0 tx=0x7f8be000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:43:30.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.931+0000 7f8bf4ff9700 1 -- 172.21.15.67:0/3314029879 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8be00041f0 con 0x7f8bf8102720 2024-04-22T23:43:30.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.931+0000 7f8bf4ff9700 1 -- 172.21.15.67:0/3314029879 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8be0004350 con 0x7f8bf8102720 2024-04-22T23:43:30.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.931+0000 7f8bfda9d700 1 -- 172.21.15.67:0/3314029879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bf81969f0 con 0x7f8bf8102720 2024-04-22T23:43:30.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.931+0000 7f8bf4ff9700 1 -- 172.21.15.67:0/3314029879 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8be0003c80 con 0x7f8bf8102720 2024-04-22T23:43:30.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.931+0000 7f8bfda9d700 1 -- 172.21.15.67:0/3314029879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bf8196f40 con 0x7f8bf8102720 2024-04-22T23:43:30.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.932+0000 7f8bfda9d700 1 -- 172.21.15.67:0/3314029879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bf8108310 con 0x7f8bf8102720 2024-04-22T23:43:30.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.932+0000 7f8bf4ff9700 1 -- 172.21.15.67:0/3314029879 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8be000f690 con 0x7f8bf8102720 2024-04-22T23:43:30.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.933+0000 7f8bf4ff9700 1 --2- 172.21.15.67:0/3314029879 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8be406ecc0 0x7f8be4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:43:30.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.933+0000 7f8bf77fe700 1 --2- 172.21.15.67:0/3314029879 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8be406ecc0 0x7f8be4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:43:30.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.933+0000 7f8bf4ff9700 1 -- 172.21.15.67:0/3314029879 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f8be008eac0 con 0x7f8bf8102720 2024-04-22T23:43:30.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.933+0000 7f8bf77fe700 1 --2- 172.21.15.67:0/3314029879 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8be406ecc0 0x7f8be4071180 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8be8000f80 tx=0x7f8be80060c0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:43:30.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:30.938+0000 7f8bf4ff9700 1 -- 172.21.15.67:0/3314029879 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f8be005b2b0 con 0x7f8bf8102720 2024-04-22T23:43:31.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.168+0000 7f8bfda9d700 1 -- 172.21.15.67:0/3314029879 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8bf8197270 con 0x7f8be406ecc0 2024-04-22T23:43:31.161 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:30 smithi067 ceph-mon[112438]: from='client.34960 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:43:31.161 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:30 smithi067 ceph-mon[112438]: from='client.34964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:43:31.161 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:30 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1048546026' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:43:31.161 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:30 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/765670676' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:43:31.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.169+0000 7f8bf4ff9700 1 -- 172.21.15.67:0/3314029879 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f8bf8197270 con 0x7f8be406ecc0 2024-04-22T23:43:31.162 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:43:31.162 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:43:31.162 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:43:31.162 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:43:31.162 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:43:31.162 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:43:31.162 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:43:31.162 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:43:31.162 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:43:31.162 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:43:31.162 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T23:43:31.162 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:43:31.162 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:43:31.163 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:43:31.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.172+0000 7f8bfda9d700 1 -- 172.21.15.67:0/3314029879 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8be406ecc0 msgr2=0x7f8be4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:43:31.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.172+0000 7f8bfda9d700 1 --2- 172.21.15.67:0/3314029879 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8be406ecc0 0x7f8be4071180 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8be8000f80 tx=0x7f8be80060c0 comp rx=0 tx=0).stop 2024-04-22T23:43:31.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.172+0000 7f8bfda9d700 1 -- 172.21.15.67:0/3314029879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bf8102720 msgr2=0x7f8bf819cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:43:31.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.172+0000 7f8bfda9d700 1 --2- 172.21.15.67:0/3314029879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bf8102720 0x7f8bf819cbd0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f8be000d8d0 tx=0x7f8be000dbe0 comp rx=0 tx=0).stop 2024-04-22T23:43:31.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.173+0000 7f8bfda9d700 1 -- 172.21.15.67:0/3314029879 shutdown_connections 2024-04-22T23:43:31.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.173+0000 7f8bfda9d700 1 --2- 172.21.15.67:0/3314029879 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8be406ecc0 0x7f8be4071180 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:31.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.173+0000 7f8bfda9d700 1 --2- 172.21.15.67:0/3314029879 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8bf8101e00 0x7f8bf819c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:31.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.173+0000 7f8bfda9d700 1 --2- 172.21.15.67:0/3314029879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bf8102720 0x7f8bf819cbd0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:31.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.173+0000 7f8bfda9d700 1 -- 172.21.15.67:0/3314029879 >> 172.21.15.67:0/3314029879 conn(0x7f8bf80fb650 msgr2=0x7f8bf8105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:43:31.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.173+0000 7f8bfda9d700 1 -- 172.21.15.67:0/3314029879 shutdown_connections 2024-04-22T23:43:31.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.173+0000 7f8bfda9d700 1 -- 172.21.15.67:0/3314029879 wait complete. 2024-04-22T23:43:31.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:30 smithi134 ceph-mon[85630]: from='client.34960 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:43:31.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:30 smithi134 ceph-mon[85630]: from='client.34964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:43:31.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:30 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1048546026' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:43:31.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:30 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/765670676' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:43:31.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.281+0000 7ff599398700 1 -- 172.21.15.67:0/3739963217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff594105a20 msgr2=0x7ff594105e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:43:31.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.281+0000 7ff599398700 1 --2- 172.21.15.67:0/3739963217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff594105a20 0x7ff594105e00 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7ff584009910 tx=0x7ff584009c20 comp rx=0 tx=0).stop 2024-04-22T23:43:31.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.281+0000 7ff599398700 1 -- 172.21.15.67:0/3739963217 shutdown_connections 2024-04-22T23:43:31.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.281+0000 7ff599398700 1 --2- 172.21.15.67:0/3739963217 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff5940ffb60 0x7ff5940fffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:31.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.281+0000 7ff599398700 1 --2- 172.21.15.67:0/3739963217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff594105a20 0x7ff594105e00 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:31.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.281+0000 7ff599398700 1 -- 172.21.15.67:0/3739963217 >> 172.21.15.67:0/3739963217 conn(0x7ff5940754a0 msgr2=0x7ff5940758b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:43:31.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.281+0000 7ff599398700 1 -- 172.21.15.67:0/3739963217 shutdown_connections 2024-04-22T23:43:31.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.281+0000 7ff599398700 1 -- 172.21.15.67:0/3739963217 wait complete. 2024-04-22T23:43:31.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.282+0000 7ff599398700 1 Processor -- start 2024-04-22T23:43:31.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.282+0000 7ff599398700 1 -- start start 2024-04-22T23:43:31.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.282+0000 7ff599398700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5940ffb60 0x7ff5941966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:43:31.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.282+0000 7ff599398700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff594105a20 0x7ff594196c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:43:31.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.282+0000 7ff599398700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5941972a0 con 0x7ff5940ffb60 2024-04-22T23:43:31.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.282+0000 7ff599398700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff594199fd0 con 0x7ff594105a20 2024-04-22T23:43:31.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.282+0000 7ff592ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5940ffb60 0x7ff5941966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:43:31.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.282+0000 7ff592ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5940ffb60 0x7ff5941966d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41436/0 (socket says 172.21.15.67:41436) 2024-04-22T23:43:31.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.282+0000 7ff592ffd700 1 -- 172.21.15.67:0/3246123625 learned_addr learned my addr 172.21.15.67:0/3246123625 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:43:31.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.283+0000 7ff58bfff700 1 --2- 172.21.15.67:0/3246123625 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff594105a20 0x7ff594196c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:43:31.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.283+0000 7ff592ffd700 1 -- 172.21.15.67:0/3246123625 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff594105a20 msgr2=0x7ff594196c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:43:31.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.283+0000 7ff592ffd700 1 --2- 172.21.15.67:0/3246123625 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff594105a20 0x7ff594196c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:31.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.283+0000 7ff592ffd700 1 -- 172.21.15.67:0/3246123625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5840095f0 con 0x7ff5940ffb60 2024-04-22T23:43:31.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.283+0000 7ff58bfff700 1 --2- 172.21.15.67:0/3246123625 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff594105a20 0x7ff594196c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:43:31.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.283+0000 7ff592ffd700 1 --2- 172.21.15.67:0/3246123625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5940ffb60 0x7ff5941966d0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7ff584006010 tx=0x7ff58400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:43:31.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.283+0000 7ff590ff9700 1 -- 172.21.15.67:0/3246123625 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff58401d070 con 0x7ff5940ffb60 2024-04-22T23:43:31.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.283+0000 7ff590ff9700 1 -- 172.21.15.67:0/3246123625 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff58400f460 con 0x7ff5940ffb60 2024-04-22T23:43:31.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.283+0000 7ff590ff9700 1 -- 172.21.15.67:0/3246123625 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff584017630 con 0x7ff5940ffb60 2024-04-22T23:43:31.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.283+0000 7ff599398700 1 -- 172.21.15.67:0/3246123625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff59419a250 con 0x7ff5940ffb60 2024-04-22T23:43:31.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.283+0000 7ff599398700 1 -- 172.21.15.67:0/3246123625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff59419a740 con 0x7ff5940ffb60 2024-04-22T23:43:31.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.284+0000 7ff599398700 1 -- 172.21.15.67:0/3246123625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff59404f2e0 con 0x7ff5940ffb60 2024-04-22T23:43:31.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.285+0000 7ff590ff9700 1 -- 172.21.15.67:0/3246123625 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff58400f5d0 con 0x7ff5940ffb60 2024-04-22T23:43:31.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.285+0000 7ff590ff9700 1 --2- 172.21.15.67:0/3246123625 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff58006ecc0 0x7ff580071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:43:31.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.285+0000 7ff590ff9700 1 -- 172.21.15.67:0/3246123625 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7ff584090480 con 0x7ff5940ffb60 2024-04-22T23:43:31.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.285+0000 7ff58bfff700 1 --2- 172.21.15.67:0/3246123625 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff58006ecc0 0x7ff580071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:43:31.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.286+0000 7ff58bfff700 1 --2- 172.21.15.67:0/3246123625 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff58006ecc0 0x7ff580071180 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff57c005950 tx=0x7ff57c0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:43:31.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.290+0000 7ff590ff9700 1 -- 172.21.15.67:0/3246123625 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff5840590c0 con 0x7ff5940ffb60 2024-04-22T23:43:31.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.590+0000 7ff599398700 1 -- 172.21.15.67:0/3246123625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff59419aa60 con 0x7ff5940ffb60 2024-04-22T23:43:31.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.591+0000 7ff590ff9700 1 -- 172.21.15.67:0/3246123625 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7ff58405c780 con 0x7ff5940ffb60 2024-04-22T23:43:31.583 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:43:31.583 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:43:31.584 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:43:31.584 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:43:31.584 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:43:31.584 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:43:31.584 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:43:31.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.593+0000 7ff599398700 1 -- 172.21.15.67:0/3246123625 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff58006ecc0 msgr2=0x7ff580071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:43:31.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.593+0000 7ff599398700 1 --2- 172.21.15.67:0/3246123625 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff58006ecc0 0x7ff580071180 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff57c005950 tx=0x7ff57c0058e0 comp rx=0 tx=0).stop 2024-04-22T23:43:31.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.593+0000 7ff599398700 1 -- 172.21.15.67:0/3246123625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5940ffb60 msgr2=0x7ff5941966d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:43:31.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.593+0000 7ff599398700 1 --2- 172.21.15.67:0/3246123625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5940ffb60 0x7ff5941966d0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7ff584006010 tx=0x7ff58400be20 comp rx=0 tx=0).stop 2024-04-22T23:43:31.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.593+0000 7ff599398700 1 -- 172.21.15.67:0/3246123625 shutdown_connections 2024-04-22T23:43:31.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.593+0000 7ff599398700 1 --2- 172.21.15.67:0/3246123625 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff58006ecc0 0x7ff580071180 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:31.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.593+0000 7ff599398700 1 --2- 172.21.15.67:0/3246123625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5940ffb60 0x7ff5941966d0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:31.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.593+0000 7ff599398700 1 --2- 172.21.15.67:0/3246123625 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff594105a20 0x7ff594196c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:43:31.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.593+0000 7ff599398700 1 -- 172.21.15.67:0/3246123625 >> 172.21.15.67:0/3246123625 conn(0x7ff5940754a0 msgr2=0x7ff5940686a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:43:31.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.593+0000 7ff599398700 1 -- 172.21.15.67:0/3246123625 shutdown_connections 2024-04-22T23:43:31.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:43:31.593+0000 7ff599398700 1 -- 172.21.15.67:0/3246123625 wait complete. 2024-04-22T23:43:32.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:31 smithi134 ceph-mon[85630]: from='client.34968 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:43:32.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:31 smithi134 ceph-mon[85630]: pgmap v514: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:32.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:31 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3246123625' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:43:32.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:31 smithi067 ceph-mon[112438]: from='client.34968 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:43:32.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:31 smithi067 ceph-mon[112438]: pgmap v514: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:32.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:31 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3246123625' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:43:33.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:32 smithi134 ceph-mon[85630]: from='client.34980 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:43:33.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:32 smithi067 ceph-mon[112438]: from='client.34980 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:43:34.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:33 smithi134 ceph-mon[85630]: pgmap v515: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:34.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:33 smithi067 ceph-mon[112438]: pgmap v515: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:36.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:35 smithi134 ceph-mon[85630]: pgmap v516: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:36.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:35 smithi067 ceph-mon[112438]: pgmap v516: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:38.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:37 smithi134 ceph-mon[85630]: pgmap v517: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:38.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:37 smithi067 ceph-mon[112438]: pgmap v517: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:40.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:39 smithi134 ceph-mon[85630]: pgmap v518: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:40.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:39 smithi067 ceph-mon[112438]: pgmap v518: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:42.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:41 smithi134 ceph-mon[85630]: pgmap v519: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:42.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:43:42.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:41 smithi067 ceph-mon[112438]: pgmap v519: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:42.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:43:44.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:43 smithi134 ceph-mon[85630]: pgmap v520: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:44.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:43 smithi067 ceph-mon[112438]: pgmap v520: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:46.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:45 smithi134 ceph-mon[85630]: pgmap v521: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:46.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:45 smithi067 ceph-mon[112438]: pgmap v521: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:48.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:47 smithi134 ceph-mon[85630]: pgmap v522: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:48.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:47 smithi067 ceph-mon[112438]: pgmap v522: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:50.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:49 smithi134 ceph-mon[85630]: pgmap v523: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:50.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:49 smithi067 ceph-mon[112438]: pgmap v523: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:52.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:51 smithi134 ceph-mon[85630]: pgmap v524: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:52.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:51 smithi067 ceph-mon[112438]: pgmap v524: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:54.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:53 smithi134 ceph-mon[85630]: pgmap v525: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:54.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:53 smithi067 ceph-mon[112438]: pgmap v525: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:56.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:55 smithi067 ceph-mon[112438]: pgmap v526: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:56.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:55 smithi134 ceph-mon[85630]: pgmap v526: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:57.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:57 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:43:57.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:57 smithi067 ceph-mon[112438]: pgmap v527: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:57.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:57 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:43:57.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:57 smithi134 ceph-mon[85630]: pgmap v527: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:59.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:43:59 smithi134 ceph-mon[85630]: pgmap v528: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:43:59.536 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:43:59 smithi067 ceph-mon[112438]: pgmap v528: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:01.536 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:01 smithi067 ceph-mon[112438]: pgmap v529: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:01.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.714+0000 7f910277c700 1 -- 172.21.15.67:0/547020616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90fc101df0 msgr2=0x7f90fc1021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.714+0000 7f910277c700 1 --2- 172.21.15.67:0/547020616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90fc101df0 0x7f90fc1021d0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f90e4009910 tx=0x7f90e4009c20 comp rx=0 tx=0).stop 2024-04-22T23:44:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.714+0000 7f910277c700 1 -- 172.21.15.67:0/547020616 shutdown_connections 2024-04-22T23:44:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.714+0000 7f910277c700 1 --2- 172.21.15.67:0/547020616 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f90fc102710 0x7f90fc10ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.714+0000 7f910277c700 1 --2- 172.21.15.67:0/547020616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90fc101df0 0x7f90fc1021d0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.714+0000 7f910277c700 1 -- 172.21.15.67:0/547020616 >> 172.21.15.67:0/547020616 conn(0x7f90fc0fb620 msgr2=0x7f90fc0fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:44:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.714+0000 7f910277c700 1 -- 172.21.15.67:0/547020616 shutdown_connections 2024-04-22T23:44:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.714+0000 7f910277c700 1 -- 172.21.15.67:0/547020616 wait complete. 2024-04-22T23:44:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.714+0000 7f910277c700 1 Processor -- start 2024-04-22T23:44:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.714+0000 7f910277c700 1 -- start start 2024-04-22T23:44:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.715+0000 7f910277c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f90fc101df0 0x7f90fc19c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.715+0000 7f910277c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90fc102710 0x7f90fc19cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.715+0000 7f910277c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc19d250 con 0x7f90fc102710 2024-04-22T23:44:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.715+0000 7f910277c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc196700 con 0x7f90fc101df0 2024-04-22T23:44:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.715+0000 7f90fb7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90fc102710 0x7f90fc19cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.715+0000 7f90fb7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90fc102710 0x7f90fc19cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34272/0 (socket says 172.21.15.67:34272) 2024-04-22T23:44:01.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.715+0000 7f90fb7fe700 1 -- 172.21.15.67:0/2289541155 learned_addr learned my addr 172.21.15.67:0/2289541155 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:44:01.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.715+0000 7f90fbfff700 1 --2- 172.21.15.67:0/2289541155 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f90fc101df0 0x7f90fc19c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:01.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.715+0000 7f90fb7fe700 1 -- 172.21.15.67:0/2289541155 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f90fc101df0 msgr2=0x7f90fc19c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:01.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.715+0000 7f90fb7fe700 1 --2- 172.21.15.67:0/2289541155 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f90fc101df0 0x7f90fc19c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:01.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.715+0000 7f90fb7fe700 1 -- 172.21.15.67:0/2289541155 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e40095f0 con 0x7f90fc102710 2024-04-22T23:44:01.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.716+0000 7f90fb7fe700 1 --2- 172.21.15.67:0/2289541155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90fc102710 0x7f90fc19cbc0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f90ec00d8d0 tx=0x7f90ec00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:44:01.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.716+0000 7f90f97fa700 1 -- 172.21.15.67:0/2289541155 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90ec0041f0 con 0x7f90fc102710 2024-04-22T23:44:01.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.716+0000 7f90f97fa700 1 -- 172.21.15.67:0/2289541155 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f90ec004350 con 0x7f90fc102710 2024-04-22T23:44:01.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.716+0000 7f910277c700 1 -- 172.21.15.67:0/2289541155 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90fc1969e0 con 0x7f90fc102710 2024-04-22T23:44:01.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.716+0000 7f90f97fa700 1 -- 172.21.15.67:0/2289541155 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90ec003d60 con 0x7f90fc102710 2024-04-22T23:44:01.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.716+0000 7f910277c700 1 -- 172.21.15.67:0/2289541155 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90fc196f30 con 0x7f90fc102710 2024-04-22T23:44:01.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.717+0000 7f910277c700 1 -- 172.21.15.67:0/2289541155 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90fc108300 con 0x7f90fc102710 2024-04-22T23:44:01.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.718+0000 7f90f97fa700 1 -- 172.21.15.67:0/2289541155 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f90ec009730 con 0x7f90fc102710 2024-04-22T23:44:01.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.718+0000 7f90f97fa700 1 --2- 172.21.15.67:0/2289541155 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f90e806ecc0 0x7f90e8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:01.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.718+0000 7f90f97fa700 1 -- 172.21.15.67:0/2289541155 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f90ec08eb70 con 0x7f90fc102710 2024-04-22T23:44:01.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.718+0000 7f90fbfff700 1 --2- 172.21.15.67:0/2289541155 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f90e806ecc0 0x7f90e8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:01.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.719+0000 7f90fbfff700 1 --2- 172.21.15.67:0/2289541155 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f90e806ecc0 0x7f90e8071180 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f90e4006010 tx=0x7f90e401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:44:01.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.723+0000 7f90f97fa700 1 -- 172.21.15.67:0/2289541155 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f90ec05b360 con 0x7f90fc102710 2024-04-22T23:44:01.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:01 smithi134 ceph-mon[85630]: pgmap v529: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:01.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.954+0000 7f910277c700 1 -- 172.21.15.67:0/2289541155 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f90fc197180 con 0x7f90e806ecc0 2024-04-22T23:44:01.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.955+0000 7f90f97fa700 1 -- 172.21.15.67:0/2289541155 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f90fc197180 con 0x7f90e806ecc0 2024-04-22T23:44:01.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.957+0000 7f910277c700 1 -- 172.21.15.67:0/2289541155 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f90e806ecc0 msgr2=0x7f90e8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:01.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.957+0000 7f910277c700 1 --2- 172.21.15.67:0/2289541155 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f90e806ecc0 0x7f90e8071180 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f90e4006010 tx=0x7f90e401a040 comp rx=0 tx=0).stop 2024-04-22T23:44:01.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.957+0000 7f910277c700 1 -- 172.21.15.67:0/2289541155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90fc102710 msgr2=0x7f90fc19cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:01.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.957+0000 7f910277c700 1 --2- 172.21.15.67:0/2289541155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90fc102710 0x7f90fc19cbc0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f90ec00d8d0 tx=0x7f90ec00dc90 comp rx=0 tx=0).stop 2024-04-22T23:44:01.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.957+0000 7f910277c700 1 -- 172.21.15.67:0/2289541155 shutdown_connections 2024-04-22T23:44:01.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.957+0000 7f910277c700 1 --2- 172.21.15.67:0/2289541155 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f90e806ecc0 0x7f90e8071180 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:01.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.957+0000 7f910277c700 1 --2- 172.21.15.67:0/2289541155 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f90fc101df0 0x7f90fc19c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:01.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.957+0000 7f910277c700 1 --2- 172.21.15.67:0/2289541155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90fc102710 0x7f90fc19cbc0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:01.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.957+0000 7f910277c700 1 -- 172.21.15.67:0/2289541155 >> 172.21.15.67:0/2289541155 conn(0x7f90fc0fb620 msgr2=0x7f90fc105440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:44:01.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.957+0000 7f910277c700 1 -- 172.21.15.67:0/2289541155 shutdown_connections 2024-04-22T23:44:01.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:01.957+0000 7f910277c700 1 -- 172.21.15.67:0/2289541155 wait complete. 2024-04-22T23:44:01.960 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:44:02.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.065+0000 7f4f55384700 1 -- 172.21.15.67:0/1008812967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f50101e60 msgr2=0x7f4f50102240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:02.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.065+0000 7f4f55384700 1 --2- 172.21.15.67:0/1008812967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f50101e60 0x7f4f50102240 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f4f38009910 tx=0x7f4f38009c20 comp rx=0 tx=0).stop 2024-04-22T23:44:02.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.065+0000 7f4f55384700 1 -- 172.21.15.67:0/1008812967 shutdown_connections 2024-04-22T23:44:02.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.065+0000 7f4f55384700 1 --2- 172.21.15.67:0/1008812967 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4f50102780 0x7f4f5010ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:02.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.065+0000 7f4f55384700 1 --2- 172.21.15.67:0/1008812967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f50101e60 0x7f4f50102240 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:02.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.065+0000 7f4f55384700 1 -- 172.21.15.67:0/1008812967 >> 172.21.15.67:0/1008812967 conn(0x7f4f500fb650 msgr2=0x7f4f500fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:44:02.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.065+0000 7f4f55384700 1 -- 172.21.15.67:0/1008812967 shutdown_connections 2024-04-22T23:44:02.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.065+0000 7f4f55384700 1 -- 172.21.15.67:0/1008812967 wait complete. 2024-04-22T23:44:02.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.066+0000 7f4f55384700 1 Processor -- start 2024-04-22T23:44:02.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.066+0000 7f4f55384700 1 -- start start 2024-04-22T23:44:02.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.066+0000 7f4f55384700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f50101e60 0x7f4f500ffac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:02.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.066+0000 7f4f55384700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4f50102780 0x7f4f50100000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:02.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.066+0000 7f4f55384700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f50100540 con 0x7f4f50101e60 2024-04-22T23:44:02.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.066+0000 7f4f55384700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f50100680 con 0x7f4f50102780 2024-04-22T23:44:02.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.066+0000 7f4f4effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f50101e60 0x7f4f500ffac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:02.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.066+0000 7f4f4effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f50101e60 0x7f4f500ffac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34296/0 (socket says 172.21.15.67:34296) 2024-04-22T23:44:02.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.066+0000 7f4f4effd700 1 -- 172.21.15.67:0/1402100416 learned_addr learned my addr 172.21.15.67:0/1402100416 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:44:02.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.066+0000 7f4f4e7fc700 1 --2- 172.21.15.67:0/1402100416 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4f50102780 0x7f4f50100000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:02.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.066+0000 7f4f4effd700 1 -- 172.21.15.67:0/1402100416 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4f50102780 msgr2=0x7f4f50100000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:02.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.066+0000 7f4f4effd700 1 --2- 172.21.15.67:0/1402100416 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4f50102780 0x7f4f50100000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:02.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.066+0000 7f4f4effd700 1 -- 172.21.15.67:0/1402100416 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f380095f0 con 0x7f4f50101e60 2024-04-22T23:44:02.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.067+0000 7f4f4effd700 1 --2- 172.21.15.67:0/1402100416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f50101e60 0x7f4f500ffac0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f4f3800b5c0 tx=0x7f4f3800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:44:02.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.067+0000 7f4f47fff700 1 -- 172.21.15.67:0/1402100416 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f3801d070 con 0x7f4f50101e60 2024-04-22T23:44:02.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.067+0000 7f4f55384700 1 -- 172.21.15.67:0/1402100416 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f50100900 con 0x7f4f50101e60 2024-04-22T23:44:02.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.067+0000 7f4f47fff700 1 -- 172.21.15.67:0/1402100416 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4f38005e00 con 0x7f4f50101e60 2024-04-22T23:44:02.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.067+0000 7f4f47fff700 1 -- 172.21.15.67:0/1402100416 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f38017400 con 0x7f4f50101e60 2024-04-22T23:44:02.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.067+0000 7f4f55384700 1 -- 172.21.15.67:0/1402100416 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f501a2d80 con 0x7f4f50101e60 2024-04-22T23:44:02.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.068+0000 7f4f55384700 1 -- 172.21.15.67:0/1402100416 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f50108310 con 0x7f4f50101e60 2024-04-22T23:44:02.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.068+0000 7f4f47fff700 1 -- 172.21.15.67:0/1402100416 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4f3800f6f0 con 0x7f4f50101e60 2024-04-22T23:44:02.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.069+0000 7f4f47fff700 1 --2- 172.21.15.67:0/1402100416 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4f3c06ecc0 0x7f4f3c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:02.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.069+0000 7f4f47fff700 1 -- 172.21.15.67:0/1402100416 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f4f380904b0 con 0x7f4f50101e60 2024-04-22T23:44:02.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.069+0000 7f4f4e7fc700 1 --2- 172.21.15.67:0/1402100416 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4f3c06ecc0 0x7f4f3c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:02.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.069+0000 7f4f4e7fc700 1 --2- 172.21.15.67:0/1402100416 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4f3c06ecc0 0x7f4f3c071180 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4f50101260 tx=0x7f4f400073d0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:44:02.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.073+0000 7f4f47fff700 1 -- 172.21.15.67:0/1402100416 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f4f380590f0 con 0x7f4f50101e60 2024-04-22T23:44:02.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.306+0000 7f4f55384700 1 -- 172.21.15.67:0/1402100416 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f500611d0 con 0x7f4f3c06ecc0 2024-04-22T23:44:02.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.307+0000 7f4f47fff700 1 -- 172.21.15.67:0/1402100416 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f4f500611d0 con 0x7f4f3c06ecc0 2024-04-22T23:44:02.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.310+0000 7f4f55384700 1 -- 172.21.15.67:0/1402100416 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4f3c06ecc0 msgr2=0x7f4f3c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:02.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.310+0000 7f4f55384700 1 --2- 172.21.15.67:0/1402100416 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4f3c06ecc0 0x7f4f3c071180 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4f50101260 tx=0x7f4f400073d0 comp rx=0 tx=0).stop 2024-04-22T23:44:02.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.310+0000 7f4f55384700 1 -- 172.21.15.67:0/1402100416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f50101e60 msgr2=0x7f4f500ffac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:02.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.310+0000 7f4f55384700 1 --2- 172.21.15.67:0/1402100416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f50101e60 0x7f4f500ffac0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f4f3800b5c0 tx=0x7f4f3800bd70 comp rx=0 tx=0).stop 2024-04-22T23:44:02.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.310+0000 7f4f55384700 1 -- 172.21.15.67:0/1402100416 shutdown_connections 2024-04-22T23:44:02.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.310+0000 7f4f55384700 1 --2- 172.21.15.67:0/1402100416 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4f3c06ecc0 0x7f4f3c071180 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:02.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.310+0000 7f4f55384700 1 --2- 172.21.15.67:0/1402100416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f50101e60 0x7f4f500ffac0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:02.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.310+0000 7f4f55384700 1 --2- 172.21.15.67:0/1402100416 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4f50102780 0x7f4f50100000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:02.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.310+0000 7f4f55384700 1 -- 172.21.15.67:0/1402100416 >> 172.21.15.67:0/1402100416 conn(0x7f4f500fb650 msgr2=0x7f4f50105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:44:02.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.310+0000 7f4f55384700 1 -- 172.21.15.67:0/1402100416 shutdown_connections 2024-04-22T23:44:02.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.310+0000 7f4f55384700 1 -- 172.21.15.67:0/1402100416 wait complete. 2024-04-22T23:44:02.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.417+0000 7fb89cc18700 1 -- 172.21.15.67:0/1638667247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb898101850 msgr2=0x7fb8981058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:02.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.417+0000 7fb89cc18700 1 --2- 172.21.15.67:0/1638667247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb898101850 0x7fb8981058a0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fb888009910 tx=0x7fb888009c20 comp rx=0 tx=0).stop 2024-04-22T23:44:02.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.417+0000 7fb89cc18700 1 -- 172.21.15.67:0/1638667247 shutdown_connections 2024-04-22T23:44:02.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.417+0000 7fb89cc18700 1 --2- 172.21.15.67:0/1638667247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb898101850 0x7fb8981058a0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:02.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.417+0000 7fb89cc18700 1 --2- 172.21.15.67:0/1638667247 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb898100ea0 0x7fb898101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:02.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.417+0000 7fb89cc18700 1 -- 172.21.15.67:0/1638667247 >> 172.21.15.67:0/1638667247 conn(0x7fb8980fc750 msgr2=0x7fb8980feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:44:02.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.417+0000 7fb89cc18700 1 -- 172.21.15.67:0/1638667247 shutdown_connections 2024-04-22T23:44:02.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.417+0000 7fb89cc18700 1 -- 172.21.15.67:0/1638667247 wait complete. 2024-04-22T23:44:02.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.418+0000 7fb89cc18700 1 Processor -- start 2024-04-22T23:44:02.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.418+0000 7fb89cc18700 1 -- start start 2024-04-22T23:44:02.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.418+0000 7fb89cc18700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb898100ea0 0x7fb898073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:02.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.418+0000 7fb89cc18700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb898101850 0x7fb8980735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:02.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.418+0000 7fb89cc18700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb898073b10 con 0x7fb898100ea0 2024-04-22T23:44:02.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.418+0000 7fb89cc18700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb898073c50 con 0x7fb898101850 2024-04-22T23:44:02.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.418+0000 7fb89659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb898100ea0 0x7fb898073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:02.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.418+0000 7fb89659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb898100ea0 0x7fb898073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34316/0 (socket says 172.21.15.67:34316) 2024-04-22T23:44:02.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.418+0000 7fb89659c700 1 -- 172.21.15.67:0/1826748033 learned_addr learned my addr 172.21.15.67:0/1826748033 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:44:02.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.418+0000 7fb895d9b700 1 --2- 172.21.15.67:0/1826748033 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb898101850 0x7fb8980735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:02.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.418+0000 7fb89659c700 1 -- 172.21.15.67:0/1826748033 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb898101850 msgr2=0x7fb8980735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:02.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.418+0000 7fb89659c700 1 --2- 172.21.15.67:0/1826748033 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb898101850 0x7fb8980735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:02.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.418+0000 7fb89659c700 1 -- 172.21.15.67:0/1826748033 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8880095f0 con 0x7fb898100ea0 2024-04-22T23:44:02.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.419+0000 7fb89659c700 1 --2- 172.21.15.67:0/1826748033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb898100ea0 0x7fb898073090 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fb88000b4b0 tx=0x7fb88000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:44:02.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.419+0000 7fb88f7fe700 1 -- 172.21.15.67:0/1826748033 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb880011800 con 0x7fb898100ea0 2024-04-22T23:44:02.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.419+0000 7fb89cc18700 1 -- 172.21.15.67:0/1826748033 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8981a2b90 con 0x7fb898100ea0 2024-04-22T23:44:02.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.419+0000 7fb88f7fe700 1 -- 172.21.15.67:0/1826748033 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb880011e20 con 0x7fb898100ea0 2024-04-22T23:44:02.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.419+0000 7fb88f7fe700 1 -- 172.21.15.67:0/1826748033 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb88000f2e0 con 0x7fb898100ea0 2024-04-22T23:44:02.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.419+0000 7fb89cc18700 1 -- 172.21.15.67:0/1826748033 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8981a2fa0 con 0x7fb898100ea0 2024-04-22T23:44:02.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.420+0000 7fb89cc18700 1 -- 172.21.15.67:0/1826748033 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8981091d0 con 0x7fb898100ea0 2024-04-22T23:44:02.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.420+0000 7fb88f7fe700 1 -- 172.21.15.67:0/1826748033 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb880010380 con 0x7fb898100ea0 2024-04-22T23:44:02.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.421+0000 7fb88f7fe700 1 --2- 172.21.15.67:0/1826748033 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb88406ecc0 0x7fb884071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:02.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.421+0000 7fb88f7fe700 1 -- 172.21.15.67:0/1826748033 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fb88008ebd0 con 0x7fb898100ea0 2024-04-22T23:44:02.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.421+0000 7fb895d9b700 1 --2- 172.21.15.67:0/1826748033 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb88406ecc0 0x7fb884071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:02.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.421+0000 7fb895d9b700 1 --2- 172.21.15.67:0/1826748033 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb88406ecc0 0x7fb884071180 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb898074880 tx=0x7fb88801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:44:02.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.425+0000 7fb88f7fe700 1 -- 172.21.15.67:0/1826748033 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb8800577b0 con 0x7fb898100ea0 2024-04-22T23:44:02.536 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:02 smithi067 ceph-mon[112438]: from='client.34988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:44:02.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.644+0000 7fb89cc18700 1 -- 172.21.15.67:0/1826748033 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb8980611d0 con 0x7fb88406ecc0 2024-04-22T23:44:02.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.655+0000 7fb88f7fe700 1 -- 172.21.15.67:0/1826748033 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fb8980611d0 con 0x7fb88406ecc0 2024-04-22T23:44:02.647 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:44:02.647 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (17m) 11m ago 28m 25.5M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:44:02.647 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (29m) 11m ago 29m 9114k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:44:02.648 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (28m) 10m ago 28m 9042k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:44:02.648 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (15m) 11m ago 29m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:44:02.648 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (15m) 10m ago 28m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:44:02.648 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (17m) 11m ago 28m 79.3M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:44:02.648 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (11m) 11m ago 23m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:44:02.648 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (11m) 11m ago 23m 12.6M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:44:02.648 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (10m) 10m ago 23m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:44:02.648 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (23m) 10m ago 23m 12.2M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:44:02.648 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (19m) 11m ago 30m 511M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:44:02.648 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (18m) 10m ago 28m 453M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:44:02.648 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (16m) 11m ago 30m 55.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:44:02.648 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (15m) 10m ago 27m 46.0M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:44:02.648 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (18m) 11m ago 29m 15.4M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:44:02.648 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (18m) 10m ago 28m 16.1M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:44:02.648 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (14m) 11m ago 27m 99.2M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:44:02.648 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (14m) 11m ago 26m 127M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:44:02.649 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (13m) 11m ago 26m 79.9M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:44:02.649 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (12m) 10m ago 25m 97.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:44:02.649 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (12m) 10m ago 25m 82.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:44:02.649 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (11m) 10m ago 25m 77.6M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:44:02.649 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (18m) 11m ago 28m 73.9M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:44:02.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.657+0000 7fb89cc18700 1 -- 172.21.15.67:0/1826748033 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb88406ecc0 msgr2=0x7fb884071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:02.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.657+0000 7fb89cc18700 1 --2- 172.21.15.67:0/1826748033 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb88406ecc0 0x7fb884071180 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb898074880 tx=0x7fb88801a040 comp rx=0 tx=0).stop 2024-04-22T23:44:02.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.657+0000 7fb89cc18700 1 -- 172.21.15.67:0/1826748033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb898100ea0 msgr2=0x7fb898073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:02.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.657+0000 7fb89cc18700 1 --2- 172.21.15.67:0/1826748033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb898100ea0 0x7fb898073090 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fb88000b4b0 tx=0x7fb88000b870 comp rx=0 tx=0).stop 2024-04-22T23:44:02.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.657+0000 7fb89cc18700 1 -- 172.21.15.67:0/1826748033 shutdown_connections 2024-04-22T23:44:02.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.657+0000 7fb89cc18700 1 --2- 172.21.15.67:0/1826748033 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb88406ecc0 0x7fb884071180 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:02.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.657+0000 7fb89cc18700 1 --2- 172.21.15.67:0/1826748033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb898100ea0 0x7fb898073090 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:02.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.657+0000 7fb89cc18700 1 --2- 172.21.15.67:0/1826748033 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb898101850 0x7fb8980735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:02.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.657+0000 7fb89cc18700 1 -- 172.21.15.67:0/1826748033 >> 172.21.15.67:0/1826748033 conn(0x7fb8980fc750 msgr2=0x7fb8980fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:44:02.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.658+0000 7fb89cc18700 1 -- 172.21.15.67:0/1826748033 shutdown_connections 2024-04-22T23:44:02.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.658+0000 7fb89cc18700 1 -- 172.21.15.67:0/1826748033 wait complete. 2024-04-22T23:44:02.736 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:02 smithi134 ceph-mon[85630]: from='client.34988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:44:02.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.763+0000 7f1b203a3700 1 -- 172.21.15.67:0/1563767201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b18105bd0 msgr2=0x7f1b18105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:02.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.763+0000 7f1b203a3700 1 --2- 172.21.15.67:0/1563767201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b18105bd0 0x7f1b18105fb0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c009910 tx=0x7f1b0c009c20 comp rx=0 tx=0).stop 2024-04-22T23:44:02.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.763+0000 7f1b203a3700 1 -- 172.21.15.67:0/1563767201 shutdown_connections 2024-04-22T23:44:02.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.763+0000 7f1b203a3700 1 --2- 172.21.15.67:0/1563767201 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1b180ffc20 0x7f1b18100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:02.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.763+0000 7f1b203a3700 1 --2- 172.21.15.67:0/1563767201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b18105bd0 0x7f1b18105fb0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:02.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.763+0000 7f1b203a3700 1 -- 172.21.15.67:0/1563767201 >> 172.21.15.67:0/1563767201 conn(0x7f1b18075670 msgr2=0x7f1b18075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:44:02.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.763+0000 7f1b203a3700 1 -- 172.21.15.67:0/1563767201 shutdown_connections 2024-04-22T23:44:02.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.763+0000 7f1b203a3700 1 -- 172.21.15.67:0/1563767201 wait complete. 2024-04-22T23:44:02.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.764+0000 7f1b203a3700 1 Processor -- start 2024-04-22T23:44:02.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.764+0000 7f1b203a3700 1 -- start start 2024-04-22T23:44:02.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.764+0000 7f1b203a3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b180ffc20 0x7f1b1819b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:02.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.764+0000 7f1b203a3700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1b18105bd0 0x7f1b1819bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:02.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.764+0000 7f1b203a3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b18195590 con 0x7f1b180ffc20 2024-04-22T23:44:02.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.764+0000 7f1b203a3700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b18195700 con 0x7f1b18105bd0 2024-04-22T23:44:02.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.764+0000 7f1b1e13f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b180ffc20 0x7f1b1819b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:02.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.764+0000 7f1b1e13f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b180ffc20 0x7f1b1819b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34342/0 (socket says 172.21.15.67:34342) 2024-04-22T23:44:02.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.765+0000 7f1b1e13f700 1 -- 172.21.15.67:0/2515037890 learned_addr learned my addr 172.21.15.67:0/2515037890 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:44:02.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.765+0000 7f1b1d93e700 1 --2- 172.21.15.67:0/2515037890 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1b18105bd0 0x7f1b1819bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:02.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.765+0000 7f1b1e13f700 1 -- 172.21.15.67:0/2515037890 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1b18105bd0 msgr2=0x7f1b1819bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:02.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.765+0000 7f1b1e13f700 1 --2- 172.21.15.67:0/2515037890 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1b18105bd0 0x7f1b1819bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:02.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.765+0000 7f1b1e13f700 1 -- 172.21.15.67:0/2515037890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b0c0095f0 con 0x7f1b180ffc20 2024-04-22T23:44:02.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.765+0000 7f1b1e13f700 1 --2- 172.21.15.67:0/2515037890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b180ffc20 0x7f1b1819b5a0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c006010 tx=0x7f1b0c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:44:02.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.765+0000 7f1b0b7fe700 1 -- 172.21.15.67:0/2515037890 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b0c01d070 con 0x7f1b180ffc20 2024-04-22T23:44:02.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.765+0000 7f1b0b7fe700 1 -- 172.21.15.67:0/2515037890 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1b0c004fb0 con 0x7f1b180ffc20 2024-04-22T23:44:02.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.765+0000 7f1b203a3700 1 -- 172.21.15.67:0/2515037890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b18195980 con 0x7f1b180ffc20 2024-04-22T23:44:02.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.765+0000 7f1b0b7fe700 1 -- 172.21.15.67:0/2515037890 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b0c022ea0 con 0x7f1b180ffc20 2024-04-22T23:44:02.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.765+0000 7f1b203a3700 1 -- 172.21.15.67:0/2515037890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b18195e70 con 0x7f1b180ffc20 2024-04-22T23:44:02.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.766+0000 7f1b203a3700 1 -- 172.21.15.67:0/2515037890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b181080f0 con 0x7f1b180ffc20 2024-04-22T23:44:02.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.767+0000 7f1b0b7fe700 1 -- 172.21.15.67:0/2515037890 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1b0c022470 con 0x7f1b180ffc20 2024-04-22T23:44:02.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.767+0000 7f1b0b7fe700 1 --2- 172.21.15.67:0/2515037890 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1b0406ecc0 0x7f1b04071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:02.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.767+0000 7f1b0b7fe700 1 -- 172.21.15.67:0/2515037890 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f1b0c0904c0 con 0x7f1b180ffc20 2024-04-22T23:44:02.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.767+0000 7f1b1d93e700 1 --2- 172.21.15.67:0/2515037890 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1b0406ecc0 0x7f1b04071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:02.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.768+0000 7f1b1d93e700 1 --2- 172.21.15.67:0/2515037890 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1b0406ecc0 0x7f1b04071180 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1b18196e30 tx=0x7f1b14005cf0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:44:02.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:02.772+0000 7f1b0b7fe700 1 -- 172.21.15.67:0/2515037890 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1b0c059230 con 0x7f1b180ffc20 2024-04-22T23:44:03.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.071+0000 7f1b203a3700 1 -- 172.21.15.67:0/2515037890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1b18196b60 con 0x7f1b180ffc20 2024-04-22T23:44:03.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.071+0000 7f1b0b7fe700 1 -- 172.21.15.67:0/2515037890 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f1b0c022720 con 0x7f1b180ffc20 2024-04-22T23:44:03.064 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:44:03.064 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:44:03.064 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:44:03.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:44:03.064 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:44:03.064 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:44:03.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:44:03.065 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:44:03.065 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:44:03.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:44:03.065 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:44:03.065 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:44:03.065 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 3 2024-04-22T23:44:03.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:44:03.065 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:44:03.065 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:44:03.065 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 13 2024-04-22T23:44:03.065 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:44:03.065 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:44:03.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.073+0000 7f1b203a3700 1 -- 172.21.15.67:0/2515037890 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1b0406ecc0 msgr2=0x7f1b04071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:03.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.074+0000 7f1b203a3700 1 --2- 172.21.15.67:0/2515037890 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1b0406ecc0 0x7f1b04071180 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1b18196e30 tx=0x7f1b14005cf0 comp rx=0 tx=0).stop 2024-04-22T23:44:03.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.074+0000 7f1b203a3700 1 -- 172.21.15.67:0/2515037890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b180ffc20 msgr2=0x7f1b1819b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:03.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.074+0000 7f1b203a3700 1 --2- 172.21.15.67:0/2515037890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b180ffc20 0x7f1b1819b5a0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c006010 tx=0x7f1b0c00be20 comp rx=0 tx=0).stop 2024-04-22T23:44:03.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.074+0000 7f1b203a3700 1 -- 172.21.15.67:0/2515037890 shutdown_connections 2024-04-22T23:44:03.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.074+0000 7f1b203a3700 1 --2- 172.21.15.67:0/2515037890 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1b0406ecc0 0x7f1b04071180 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:03.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.074+0000 7f1b203a3700 1 --2- 172.21.15.67:0/2515037890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b180ffc20 0x7f1b1819b5a0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:03.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.074+0000 7f1b203a3700 1 --2- 172.21.15.67:0/2515037890 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1b18105bd0 0x7f1b1819bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:03.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.074+0000 7f1b203a3700 1 -- 172.21.15.67:0/2515037890 >> 172.21.15.67:0/2515037890 conn(0x7f1b18075670 msgr2=0x7f1b180feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:44:03.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.074+0000 7f1b203a3700 1 -- 172.21.15.67:0/2515037890 shutdown_connections 2024-04-22T23:44:03.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.074+0000 7f1b203a3700 1 -- 172.21.15.67:0/2515037890 wait complete. 2024-04-22T23:44:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.180+0000 7ffaa4105700 1 -- 172.21.15.67:0/5961260 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa9c0ffda0 msgr2=0x7ffa9c100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.180+0000 7ffaa4105700 1 --2- 172.21.15.67:0/5961260 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa9c0ffda0 0x7ffa9c100180 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c009910 tx=0x7ffa8c009c20 comp rx=0 tx=0).stop 2024-04-22T23:44:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.180+0000 7ffaa4105700 1 -- 172.21.15.67:0/5961260 shutdown_connections 2024-04-22T23:44:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.180+0000 7ffaa4105700 1 --2- 172.21.15.67:0/5961260 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa9c100750 0x7ffa9c1047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.180+0000 7ffaa4105700 1 --2- 172.21.15.67:0/5961260 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa9c0ffda0 0x7ffa9c100180 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.180+0000 7ffaa4105700 1 -- 172.21.15.67:0/5961260 >> 172.21.15.67:0/5961260 conn(0x7ffa9c0fb650 msgr2=0x7ffa9c0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:44:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.180+0000 7ffaa4105700 1 -- 172.21.15.67:0/5961260 shutdown_connections 2024-04-22T23:44:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.180+0000 7ffaa4105700 1 -- 172.21.15.67:0/5961260 wait complete. 2024-04-22T23:44:03.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.181+0000 7ffaa4105700 1 Processor -- start 2024-04-22T23:44:03.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.181+0000 7ffaa4105700 1 -- start start 2024-04-22T23:44:03.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.181+0000 7ffaa4105700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa9c0ffda0 0x7ffa9c19e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:03.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.181+0000 7ffaa4105700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa9c100750 0x7ffa9c19edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:03.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.181+0000 7ffaa4105700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9c19f440 con 0x7ffa9c0ffda0 2024-04-22T23:44:03.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.181+0000 7ffaa4105700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9c1988f0 con 0x7ffa9c100750 2024-04-22T23:44:03.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.181+0000 7ffaa1ea1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa9c0ffda0 0x7ffa9c19e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:03.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.181+0000 7ffaa1ea1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa9c0ffda0 0x7ffa9c19e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34354/0 (socket says 172.21.15.67:34354) 2024-04-22T23:44:03.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.181+0000 7ffaa1ea1700 1 -- 172.21.15.67:0/2109989590 learned_addr learned my addr 172.21.15.67:0/2109989590 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:44:03.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.182+0000 7ffaa16a0700 1 --2- 172.21.15.67:0/2109989590 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa9c100750 0x7ffa9c19edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:03.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.182+0000 7ffaa1ea1700 1 -- 172.21.15.67:0/2109989590 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa9c100750 msgr2=0x7ffa9c19edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:03.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.182+0000 7ffaa1ea1700 1 --2- 172.21.15.67:0/2109989590 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa9c100750 0x7ffa9c19edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:03.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.182+0000 7ffaa1ea1700 1 -- 172.21.15.67:0/2109989590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa8c0095f0 con 0x7ffa9c0ffda0 2024-04-22T23:44:03.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.182+0000 7ffaa1ea1700 1 --2- 172.21.15.67:0/2109989590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa9c0ffda0 0x7ffa9c19e870 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c005f50 tx=0x7ffa8c00f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:44:03.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.182+0000 7ffa92ffd700 1 -- 172.21.15.67:0/2109989590 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa8c01d070 con 0x7ffa9c0ffda0 2024-04-22T23:44:03.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.182+0000 7ffa92ffd700 1 -- 172.21.15.67:0/2109989590 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffa8c00b610 con 0x7ffa9c0ffda0 2024-04-22T23:44:03.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.182+0000 7ffaa4105700 1 -- 172.21.15.67:0/2109989590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa9c198b70 con 0x7ffa9c0ffda0 2024-04-22T23:44:03.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.182+0000 7ffa92ffd700 1 -- 172.21.15.67:0/2109989590 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa8c022470 con 0x7ffa9c0ffda0 2024-04-22T23:44:03.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.182+0000 7ffaa4105700 1 -- 172.21.15.67:0/2109989590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa9c199060 con 0x7ffa9c0ffda0 2024-04-22T23:44:03.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.183+0000 7ffaa4105700 1 -- 172.21.15.67:0/2109989590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa9c04f2e0 con 0x7ffa9c0ffda0 2024-04-22T23:44:03.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.184+0000 7ffa92ffd700 1 -- 172.21.15.67:0/2109989590 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ffa8c00fca0 con 0x7ffa9c0ffda0 2024-04-22T23:44:03.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.184+0000 7ffa92ffd700 1 --2- 172.21.15.67:0/2109989590 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ffa8806ecc0 0x7ffa88071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:03.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.184+0000 7ffa92ffd700 1 -- 172.21.15.67:0/2109989590 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7ffa8c090370 con 0x7ffa9c0ffda0 2024-04-22T23:44:03.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.184+0000 7ffaa16a0700 1 --2- 172.21.15.67:0/2109989590 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ffa8806ecc0 0x7ffa88071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:03.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.185+0000 7ffaa16a0700 1 --2- 172.21.15.67:0/2109989590 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ffa8806ecc0 0x7ffa88071180 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c104a20 tx=0x7ffa98009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:44:03.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.189+0000 7ffa92ffd700 1 -- 172.21.15.67:0/2109989590 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ffa8c05cb60 con 0x7ffa9c0ffda0 2024-04-22T23:44:03.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.444+0000 7ffaa4105700 1 -- 172.21.15.67:0/2109989590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ffa9c1992d0 con 0x7ffa9c0ffda0 2024-04-22T23:44:03.436 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:03 smithi067 ceph-mon[112438]: pgmap v530: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:03.436 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:03 smithi067 ceph-mon[112438]: from='client.34992 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:44:03.436 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:03 smithi067 ceph-mon[112438]: from='client.34996 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:44:03.437 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:03 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2515037890' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:44:03.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.444+0000 7ffa92ffd700 1 -- 172.21.15.67:0/2109989590 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1871 (secure 0 0 0) 0x7ffa8c05c6f0 con 0x7ffa9c0ffda0 2024-04-22T23:44:03.437 INFO:teuthology.orchestra.run.smithi067.stdout:e31 2024-04-22T23:44:03.437 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:44:03.437 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:44:03.437 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:44:03.437 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:44:03.437 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:44:03.437 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:44:03.437 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:44:03.438 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:44:03.438 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:44:03.438 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:44:03.438 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:44:03.438 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:44:03.438 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:44:03.438 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:44:03.438 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:44:03.438 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:44:03.438 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:44:03.438 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:44:03.438 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:44:03.438 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:44:03.438 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:44:03.439 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:44:03.439 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:44:03.439 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:44:03.439 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:44:03.439 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:44:03.439 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:44:03.439 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:44:03.439 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:44:03.439 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:44:03.439 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:44:03.439 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:44:03.439 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:44:03.439 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:44:03.439 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:44:03.439 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:44:03.439 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:44:03.440 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:44:03.440 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:44:03.440 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44259} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1827288036,v1:172.21.15.134:6827/1827288036] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:44:03.440 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:44:03.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.447+0000 7ffaa4105700 1 -- 172.21.15.67:0/2109989590 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ffa8806ecc0 msgr2=0x7ffa88071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:03.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.447+0000 7ffaa4105700 1 --2- 172.21.15.67:0/2109989590 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ffa8806ecc0 0x7ffa88071180 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c104a20 tx=0x7ffa98009260 comp rx=0 tx=0).stop 2024-04-22T23:44:03.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.447+0000 7ffaa4105700 1 -- 172.21.15.67:0/2109989590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa9c0ffda0 msgr2=0x7ffa9c19e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:03.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.447+0000 7ffaa4105700 1 --2- 172.21.15.67:0/2109989590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa9c0ffda0 0x7ffa9c19e870 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c005f50 tx=0x7ffa8c00f740 comp rx=0 tx=0).stop 2024-04-22T23:44:03.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.447+0000 7ffaa4105700 1 -- 172.21.15.67:0/2109989590 shutdown_connections 2024-04-22T23:44:03.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.447+0000 7ffaa4105700 1 --2- 172.21.15.67:0/2109989590 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ffa8806ecc0 0x7ffa88071180 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:03.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.447+0000 7ffaa4105700 1 --2- 172.21.15.67:0/2109989590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa9c0ffda0 0x7ffa9c19e870 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:03.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.447+0000 7ffaa4105700 1 --2- 172.21.15.67:0/2109989590 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa9c100750 0x7ffa9c19edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:03.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.447+0000 7ffaa4105700 1 -- 172.21.15.67:0/2109989590 >> 172.21.15.67:0/2109989590 conn(0x7ffa9c0fb650 msgr2=0x7ffa9c0696c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:44:03.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.447+0000 7ffaa4105700 1 -- 172.21.15.67:0/2109989590 shutdown_connections 2024-04-22T23:44:03.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.447+0000 7ffaa4105700 1 -- 172.21.15.67:0/2109989590 wait complete. 2024-04-22T23:44:03.441 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 31 2024-04-22T23:44:03.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.555+0000 7f35d57cf700 1 -- 172.21.15.67:0/3648476469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35d0100ea0 msgr2=0x7f35d0101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:03.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.555+0000 7f35d57cf700 1 --2- 172.21.15.67:0/3648476469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35d0100ea0 0x7f35d0101280 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f35b8009910 tx=0x7f35b8009c20 comp rx=0 tx=0).stop 2024-04-22T23:44:03.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.555+0000 7f35d57cf700 1 -- 172.21.15.67:0/3648476469 shutdown_connections 2024-04-22T23:44:03.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.555+0000 7f35d57cf700 1 --2- 172.21.15.67:0/3648476469 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f35d0101850 0x7f35d01058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:03.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.555+0000 7f35d57cf700 1 --2- 172.21.15.67:0/3648476469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35d0100ea0 0x7f35d0101280 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:03.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.555+0000 7f35d57cf700 1 -- 172.21.15.67:0/3648476469 >> 172.21.15.67:0/3648476469 conn(0x7f35d00fc750 msgr2=0x7f35d00feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:44:03.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.555+0000 7f35d57cf700 1 -- 172.21.15.67:0/3648476469 shutdown_connections 2024-04-22T23:44:03.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.555+0000 7f35d57cf700 1 -- 172.21.15.67:0/3648476469 wait complete. 2024-04-22T23:44:03.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.556+0000 7f35d57cf700 1 Processor -- start 2024-04-22T23:44:03.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.556+0000 7f35d57cf700 1 -- start start 2024-04-22T23:44:03.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.556+0000 7f35d57cf700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35d0100ea0 0x7f35d019c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:03.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.556+0000 7f35d57cf700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f35d0101850 0x7f35d019cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:03.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.556+0000 7f35d57cf700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d019d2c0 con 0x7f35d0100ea0 2024-04-22T23:44:03.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.556+0000 7f35d57cf700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d0196770 con 0x7f35d0101850 2024-04-22T23:44:03.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.557+0000 7f35ceffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35d0100ea0 0x7f35d019c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:03.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.557+0000 7f35ceffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35d0100ea0 0x7f35d019c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34370/0 (socket says 172.21.15.67:34370) 2024-04-22T23:44:03.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.557+0000 7f35ceffd700 1 -- 172.21.15.67:0/3892267500 learned_addr learned my addr 172.21.15.67:0/3892267500 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:44:03.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.557+0000 7f35ce7fc700 1 --2- 172.21.15.67:0/3892267500 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f35d0101850 0x7f35d019cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:03.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.557+0000 7f35ceffd700 1 -- 172.21.15.67:0/3892267500 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f35d0101850 msgr2=0x7f35d019cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:03.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.557+0000 7f35ceffd700 1 --2- 172.21.15.67:0/3892267500 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f35d0101850 0x7f35d019cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:03.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.557+0000 7f35ceffd700 1 -- 172.21.15.67:0/3892267500 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b80095f0 con 0x7f35d0100ea0 2024-04-22T23:44:03.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.557+0000 7f35ce7fc700 1 --2- 172.21.15.67:0/3892267500 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f35d0101850 0x7f35d019cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:44:03.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.557+0000 7f35ceffd700 1 --2- 172.21.15.67:0/3892267500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35d0100ea0 0x7f35d019c6f0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f35b8006010 tx=0x7f35b800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:44:03.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.557+0000 7f35c7fff700 1 -- 172.21.15.67:0/3892267500 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35b801d070 con 0x7f35d0100ea0 2024-04-22T23:44:03.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.557+0000 7f35c7fff700 1 -- 172.21.15.67:0/3892267500 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f35b8004fb0 con 0x7f35d0100ea0 2024-04-22T23:44:03.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.557+0000 7f35d57cf700 1 -- 172.21.15.67:0/3892267500 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35d01969f0 con 0x7f35d0100ea0 2024-04-22T23:44:03.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.557+0000 7f35c7fff700 1 -- 172.21.15.67:0/3892267500 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35b8022ea0 con 0x7f35d0100ea0 2024-04-22T23:44:03.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.557+0000 7f35d57cf700 1 -- 172.21.15.67:0/3892267500 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35d0196ee0 con 0x7f35d0100ea0 2024-04-22T23:44:03.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.558+0000 7f35d57cf700 1 -- 172.21.15.67:0/3892267500 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35d0109240 con 0x7f35d0100ea0 2024-04-22T23:44:03.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.559+0000 7f35c7fff700 1 -- 172.21.15.67:0/3892267500 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f35b8022470 con 0x7f35d0100ea0 2024-04-22T23:44:03.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.559+0000 7f35c7fff700 1 --2- 172.21.15.67:0/3892267500 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f35bc06ecc0 0x7f35bc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:03.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.559+0000 7f35c7fff700 1 -- 172.21.15.67:0/3892267500 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f35b8090560 con 0x7f35d0100ea0 2024-04-22T23:44:03.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.559+0000 7f35ce7fc700 1 --2- 172.21.15.67:0/3892267500 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f35bc06ecc0 0x7f35bc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:03.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.560+0000 7f35ce7fc700 1 --2- 172.21.15.67:0/3892267500 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f35bc06ecc0 0x7f35bc071180 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f35d0197be0 tx=0x7f35c0005f50 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:44:03.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.564+0000 7f35c7fff700 1 -- 172.21.15.67:0/3892267500 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f35b805cd80 con 0x7f35d0100ea0 2024-04-22T23:44:03.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:03 smithi134 ceph-mon[85630]: pgmap v530: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:03.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:03 smithi134 ceph-mon[85630]: from='client.34992 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:44:03.736 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:03 smithi134 ceph-mon[85630]: from='client.34996 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:44:03.736 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:03 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2515037890' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:44:03.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.793+0000 7f35d57cf700 1 -- 172.21.15.67:0/3892267500 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f35d0197d40 con 0x7f35bc06ecc0 2024-04-22T23:44:03.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.794+0000 7f35c7fff700 1 -- 172.21.15.67:0/3892267500 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f35d0197d40 con 0x7f35bc06ecc0 2024-04-22T23:44:03.786 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:44:03.786 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:44:03.786 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:44:03.787 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:44:03.787 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:44:03.787 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:44:03.787 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:44:03.787 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:44:03.787 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:44:03.787 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:44:03.787 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T23:44:03.787 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:44:03.787 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:44:03.787 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:44:03.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.796+0000 7f35d57cf700 1 -- 172.21.15.67:0/3892267500 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f35bc06ecc0 msgr2=0x7f35bc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:03.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.796+0000 7f35d57cf700 1 --2- 172.21.15.67:0/3892267500 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f35bc06ecc0 0x7f35bc071180 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f35d0197be0 tx=0x7f35c0005f50 comp rx=0 tx=0).stop 2024-04-22T23:44:03.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.796+0000 7f35d57cf700 1 -- 172.21.15.67:0/3892267500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35d0100ea0 msgr2=0x7f35d019c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:03.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.796+0000 7f35d57cf700 1 --2- 172.21.15.67:0/3892267500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35d0100ea0 0x7f35d019c6f0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f35b8006010 tx=0x7f35b800be20 comp rx=0 tx=0).stop 2024-04-22T23:44:03.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.796+0000 7f35d57cf700 1 -- 172.21.15.67:0/3892267500 shutdown_connections 2024-04-22T23:44:03.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.796+0000 7f35d57cf700 1 --2- 172.21.15.67:0/3892267500 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f35bc06ecc0 0x7f35bc071180 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:03.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.796+0000 7f35d57cf700 1 --2- 172.21.15.67:0/3892267500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35d0100ea0 0x7f35d019c6f0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:03.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.796+0000 7f35d57cf700 1 --2- 172.21.15.67:0/3892267500 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f35d0101850 0x7f35d019cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:03.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.796+0000 7f35d57cf700 1 -- 172.21.15.67:0/3892267500 >> 172.21.15.67:0/3892267500 conn(0x7f35d00fc750 msgr2=0x7f35d00fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:44:03.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.796+0000 7f35d57cf700 1 -- 172.21.15.67:0/3892267500 shutdown_connections 2024-04-22T23:44:03.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.796+0000 7f35d57cf700 1 -- 172.21.15.67:0/3892267500 wait complete. 2024-04-22T23:44:03.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.904+0000 7fdb4c623700 1 -- 172.21.15.67:0/3099958484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb44100ea0 msgr2=0x7fdb44101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:03.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.904+0000 7fdb4c623700 1 --2- 172.21.15.67:0/3099958484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb44100ea0 0x7fdb44101280 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fdb40009910 tx=0x7fdb40009c20 comp rx=0 tx=0).stop 2024-04-22T23:44:03.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.904+0000 7fdb4c623700 1 -- 172.21.15.67:0/3099958484 shutdown_connections 2024-04-22T23:44:03.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.904+0000 7fdb4c623700 1 --2- 172.21.15.67:0/3099958484 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdb44101850 0x7fdb441058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:03.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.904+0000 7fdb4c623700 1 --2- 172.21.15.67:0/3099958484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb44100ea0 0x7fdb44101280 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:03.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.904+0000 7fdb4c623700 1 -- 172.21.15.67:0/3099958484 >> 172.21.15.67:0/3099958484 conn(0x7fdb440fc750 msgr2=0x7fdb440feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:44:03.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.904+0000 7fdb4c623700 1 -- 172.21.15.67:0/3099958484 shutdown_connections 2024-04-22T23:44:03.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.904+0000 7fdb4c623700 1 -- 172.21.15.67:0/3099958484 wait complete. 2024-04-22T23:44:03.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.904+0000 7fdb4c623700 1 Processor -- start 2024-04-22T23:44:03.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.904+0000 7fdb4c623700 1 -- start start 2024-04-22T23:44:03.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.905+0000 7fdb4c623700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdb44100ea0 0x7fdb44073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:03.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.905+0000 7fdb4c623700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb44101850 0x7fdb440735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:03.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.905+0000 7fdb4c623700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb44073b10 con 0x7fdb44101850 2024-04-22T23:44:03.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.905+0000 7fdb4c623700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb44073c50 con 0x7fdb44100ea0 2024-04-22T23:44:03.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.905+0000 7fdb49bbe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb44101850 0x7fdb440735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:03.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.905+0000 7fdb49bbe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb44101850 0x7fdb440735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34376/0 (socket says 172.21.15.67:34376) 2024-04-22T23:44:03.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.905+0000 7fdb49bbe700 1 -- 172.21.15.67:0/105665181 learned_addr learned my addr 172.21.15.67:0/105665181 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:44:03.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.905+0000 7fdb4a3bf700 1 --2- 172.21.15.67:0/105665181 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdb44100ea0 0x7fdb44073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:03.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.905+0000 7fdb49bbe700 1 -- 172.21.15.67:0/105665181 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdb44100ea0 msgr2=0x7fdb44073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:03.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.905+0000 7fdb49bbe700 1 --2- 172.21.15.67:0/105665181 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdb44100ea0 0x7fdb44073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:03.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.905+0000 7fdb49bbe700 1 -- 172.21.15.67:0/105665181 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb400095f0 con 0x7fdb44101850 2024-04-22T23:44:03.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.905+0000 7fdb4a3bf700 1 --2- 172.21.15.67:0/105665181 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdb44100ea0 0x7fdb44073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:44:03.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.905+0000 7fdb49bbe700 1 --2- 172.21.15.67:0/105665181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb44101850 0x7fdb440735d0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fdb3800c960 tx=0x7fdb3800cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:44:03.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.906+0000 7fdb377fe700 1 -- 172.21.15.67:0/105665181 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb38007a70 con 0x7fdb44101850 2024-04-22T23:44:03.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.906+0000 7fdb377fe700 1 -- 172.21.15.67:0/105665181 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdb38007bd0 con 0x7fdb44101850 2024-04-22T23:44:03.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.906+0000 7fdb4c623700 1 -- 172.21.15.67:0/105665181 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb441a2af0 con 0x7fdb44101850 2024-04-22T23:44:03.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.906+0000 7fdb377fe700 1 -- 172.21.15.67:0/105665181 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb380186c0 con 0x7fdb44101850 2024-04-22T23:44:03.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.906+0000 7fdb4c623700 1 -- 172.21.15.67:0/105665181 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb441a2e50 con 0x7fdb44101850 2024-04-22T23:44:03.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.907+0000 7fdb4c623700 1 -- 172.21.15.67:0/105665181 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb44109240 con 0x7fdb44101850 2024-04-22T23:44:03.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.907+0000 7fdb377fe700 1 -- 172.21.15.67:0/105665181 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdb3800faa0 con 0x7fdb44101850 2024-04-22T23:44:03.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.908+0000 7fdb377fe700 1 --2- 172.21.15.67:0/105665181 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdb3006ecc0 0x7fdb30071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:03.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.908+0000 7fdb377fe700 1 -- 172.21.15.67:0/105665181 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fdb3808ef70 con 0x7fdb44101850 2024-04-22T23:44:03.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.908+0000 7fdb4a3bf700 1 --2- 172.21.15.67:0/105665181 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdb3006ecc0 0x7fdb30071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:03.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.908+0000 7fdb4a3bf700 1 --2- 172.21.15.67:0/105665181 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdb3006ecc0 0x7fdb30071180 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fdb4000b5c0 tx=0x7fdb40005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:44:03.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:03.913+0000 7fdb377fe700 1 -- 172.21.15.67:0/105665181 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fdb380c4150 con 0x7fdb44101850 2024-04-22T23:44:04.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:04.210+0000 7fdb4c623700 1 -- 172.21.15.67:0/105665181 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fdb4404f2e0 con 0x7fdb44101850 2024-04-22T23:44:04.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:04.210+0000 7fdb377fe700 1 -- 172.21.15.67:0/105665181 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fdb3805b760 con 0x7fdb44101850 2024-04-22T23:44:04.203 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:44:04.203 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:44:04.203 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:44:04.203 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:44:04.203 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:44:04.203 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:44:04.203 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:44:04.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:04.213+0000 7fdb4c623700 1 -- 172.21.15.67:0/105665181 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdb3006ecc0 msgr2=0x7fdb30071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:04.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:04.213+0000 7fdb4c623700 1 --2- 172.21.15.67:0/105665181 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdb3006ecc0 0x7fdb30071180 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fdb4000b5c0 tx=0x7fdb40005960 comp rx=0 tx=0).stop 2024-04-22T23:44:04.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:04.213+0000 7fdb4c623700 1 -- 172.21.15.67:0/105665181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb44101850 msgr2=0x7fdb440735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:04.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:04.213+0000 7fdb4c623700 1 --2- 172.21.15.67:0/105665181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb44101850 0x7fdb440735d0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fdb3800c960 tx=0x7fdb3800cd20 comp rx=0 tx=0).stop 2024-04-22T23:44:04.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:04.213+0000 7fdb4c623700 1 -- 172.21.15.67:0/105665181 shutdown_connections 2024-04-22T23:44:04.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:04.213+0000 7fdb4c623700 1 --2- 172.21.15.67:0/105665181 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdb3006ecc0 0x7fdb30071180 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:04.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:04.213+0000 7fdb4c623700 1 --2- 172.21.15.67:0/105665181 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdb44100ea0 0x7fdb44073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:04.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:04.213+0000 7fdb4c623700 1 --2- 172.21.15.67:0/105665181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb44101850 0x7fdb440735d0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:04.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:04.213+0000 7fdb4c623700 1 -- 172.21.15.67:0/105665181 >> 172.21.15.67:0/105665181 conn(0x7fdb440fc750 msgr2=0x7fdb440fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:44:04.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:04.213+0000 7fdb4c623700 1 -- 172.21.15.67:0/105665181 shutdown_connections 2024-04-22T23:44:04.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:04.213+0000 7fdb4c623700 1 -- 172.21.15.67:0/105665181 wait complete. 2024-04-22T23:44:04.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:04 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2109989590' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:44:04.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:04 smithi134 ceph-mon[85630]: from='client.35008 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:44:04.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:04 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/105665181' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:44:04.536 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:04 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2109989590' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:44:04.536 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:04 smithi067 ceph-mon[112438]: from='client.35008 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:44:04.536 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:04 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/105665181' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:44:05.536 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:05 smithi067 ceph-mon[112438]: pgmap v531: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:05.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:05 smithi134 ceph-mon[85630]: pgmap v531: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:07.536 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:07 smithi067 ceph-mon[112438]: pgmap v532: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:07.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:07 smithi134 ceph-mon[85630]: pgmap v532: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:09.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:09 smithi134 ceph-mon[85630]: pgmap v533: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:09.536 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:09 smithi067 ceph-mon[112438]: pgmap v533: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:11.536 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:11 smithi067 ceph-mon[112438]: pgmap v534: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:11.536 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:44:11.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:11 smithi134 ceph-mon[85630]: pgmap v534: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:11.736 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:44:13.536 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:13 smithi067 ceph-mon[112438]: pgmap v535: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:13.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:13 smithi134 ceph-mon[85630]: pgmap v535: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:15.536 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:15 smithi067 ceph-mon[112438]: pgmap v536: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:15.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:15 smithi134 ceph-mon[85630]: pgmap v536: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:17.536 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:17 smithi067 ceph-mon[112438]: pgmap v537: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:17.537 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:17 smithi134 ceph-mon[85630]: pgmap v537: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:19.536 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:19 smithi067 ceph-mon[112438]: pgmap v538: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:19.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:19 smithi134 ceph-mon[85630]: pgmap v538: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:21.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:21 smithi134 ceph-mon[85630]: pgmap v539: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:21.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:21 smithi067 ceph-mon[112438]: pgmap v539: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:23.736 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:23 smithi134 ceph-mon[85630]: pgmap v540: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:23.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:23 smithi067 ceph-mon[112438]: pgmap v540: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:25.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:25 smithi134 ceph-mon[85630]: pgmap v541: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:25.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:25 smithi067 ceph-mon[112438]: pgmap v541: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:26.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:44:26.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:44:27.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:27 smithi134 ceph-mon[85630]: pgmap v542: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:27.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:27 smithi067 ceph-mon[112438]: pgmap v542: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:29.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:29 smithi134 ceph-mon[85630]: pgmap v543: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:29.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:29 smithi067 ceph-mon[112438]: pgmap v543: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:31.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:31 smithi134 ceph-mon[85630]: pgmap v544: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:31.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:31 smithi067 ceph-mon[112438]: pgmap v544: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:33.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:33 smithi134 ceph-mon[85630]: pgmap v545: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:33.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:33 smithi067 ceph-mon[112438]: pgmap v545: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:34.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.335+0000 7f48493d8700 1 -- 172.21.15.67:0/839025416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4844100ea0 msgr2=0x7f4844101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:34.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.335+0000 7f48493d8700 1 --2- 172.21.15.67:0/839025416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4844100ea0 0x7f4844101280 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f483c00b0a0 tx=0x7f483c00b3b0 comp rx=0 tx=0).stop 2024-04-22T23:44:34.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.335+0000 7f48493d8700 1 -- 172.21.15.67:0/839025416 shutdown_connections 2024-04-22T23:44:34.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.335+0000 7f48493d8700 1 --2- 172.21.15.67:0/839025416 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4844101850 0x7f48441058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:34.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.335+0000 7f48493d8700 1 --2- 172.21.15.67:0/839025416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4844100ea0 0x7f4844101280 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:34.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.335+0000 7f48493d8700 1 -- 172.21.15.67:0/839025416 >> 172.21.15.67:0/839025416 conn(0x7f48440fc750 msgr2=0x7f48440feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:44:34.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.335+0000 7f48493d8700 1 -- 172.21.15.67:0/839025416 shutdown_connections 2024-04-22T23:44:34.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.335+0000 7f48493d8700 1 -- 172.21.15.67:0/839025416 wait complete. 2024-04-22T23:44:34.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.335+0000 7f48493d8700 1 Processor -- start 2024-04-22T23:44:34.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.336+0000 7f48493d8700 1 -- start start 2024-04-22T23:44:34.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.336+0000 7f48493d8700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4844100ea0 0x7f484419c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:34.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.336+0000 7f48493d8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4844101850 0x7f484419cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:34.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.336+0000 7f48493d8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484419d2e0 con 0x7f4844101850 2024-04-22T23:44:34.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.336+0000 7f48493d8700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4844196790 con 0x7f4844100ea0 2024-04-22T23:44:34.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.336+0000 7f48427fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4844101850 0x7f484419cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:34.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.336+0000 7f48427fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4844101850 0x7f484419cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40960/0 (socket says 172.21.15.67:40960) 2024-04-22T23:44:34.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.336+0000 7f4842ffd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4844100ea0 0x7f484419c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:34.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.336+0000 7f48427fc700 1 -- 172.21.15.67:0/446735186 learned_addr learned my addr 172.21.15.67:0/446735186 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:44:34.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.336+0000 7f48427fc700 1 -- 172.21.15.67:0/446735186 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4844100ea0 msgr2=0x7f484419c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:34.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.336+0000 7f48427fc700 1 --2- 172.21.15.67:0/446735186 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4844100ea0 0x7f484419c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:34.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.337+0000 7f48427fc700 1 -- 172.21.15.67:0/446735186 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f483c009d00 con 0x7f4844101850 2024-04-22T23:44:34.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.337+0000 7f48427fc700 1 --2- 172.21.15.67:0/446735186 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4844101850 0x7f484419cc50 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f482c00d8d0 tx=0x7f482c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:44:34.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.337+0000 7f483bfff700 1 -- 172.21.15.67:0/446735186 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f482c004a00 con 0x7f4844101850 2024-04-22T23:44:34.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.337+0000 7f48493d8700 1 -- 172.21.15.67:0/446735186 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4844196a70 con 0x7f4844101850 2024-04-22T23:44:34.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.337+0000 7f483bfff700 1 -- 172.21.15.67:0/446735186 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f482c0054e0 con 0x7f4844101850 2024-04-22T23:44:34.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.337+0000 7f483bfff700 1 -- 172.21.15.67:0/446735186 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f482c00fc20 con 0x7f4844101850 2024-04-22T23:44:34.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.337+0000 7f48493d8700 1 -- 172.21.15.67:0/446735186 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4844196fc0 con 0x7f4844101850 2024-04-22T23:44:34.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.338+0000 7f48493d8700 1 -- 172.21.15.67:0/446735186 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4844109240 con 0x7f4844101850 2024-04-22T23:44:34.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.339+0000 7f483bfff700 1 -- 172.21.15.67:0/446735186 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f482c005020 con 0x7f4844101850 2024-04-22T23:44:34.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.339+0000 7f483bfff700 1 --2- 172.21.15.67:0/446735186 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f483006ecc0 0x7f4830071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:34.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.339+0000 7f483bfff700 1 -- 172.21.15.67:0/446735186 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f482c08e870 con 0x7f4844101850 2024-04-22T23:44:34.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.339+0000 7f4842ffd700 1 --2- 172.21.15.67:0/446735186 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f483006ecc0 0x7f4830071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:34.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.340+0000 7f4842ffd700 1 --2- 172.21.15.67:0/446735186 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f483006ecc0 0x7f4830071180 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f483c000f80 tx=0x7f483c0060c0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:44:34.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.344+0000 7f483bfff700 1 -- 172.21.15.67:0/446735186 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f482c05afe0 con 0x7f4844101850 2024-04-22T23:44:34.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.573+0000 7f48493d8700 1 -- 172.21.15.67:0/446735186 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4844197db0 con 0x7f483006ecc0 2024-04-22T23:44:34.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.574+0000 7f483bfff700 1 -- 172.21.15.67:0/446735186 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f4844197db0 con 0x7f483006ecc0 2024-04-22T23:44:34.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.576+0000 7f48493d8700 1 -- 172.21.15.67:0/446735186 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f483006ecc0 msgr2=0x7f4830071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:34.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.576+0000 7f48493d8700 1 --2- 172.21.15.67:0/446735186 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f483006ecc0 0x7f4830071180 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f483c000f80 tx=0x7f483c0060c0 comp rx=0 tx=0).stop 2024-04-22T23:44:34.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.576+0000 7f48493d8700 1 -- 172.21.15.67:0/446735186 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4844101850 msgr2=0x7f484419cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:34.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.576+0000 7f48493d8700 1 --2- 172.21.15.67:0/446735186 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4844101850 0x7f484419cc50 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f482c00d8d0 tx=0x7f482c00dbe0 comp rx=0 tx=0).stop 2024-04-22T23:44:34.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.576+0000 7f48493d8700 1 -- 172.21.15.67:0/446735186 shutdown_connections 2024-04-22T23:44:34.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.576+0000 7f48493d8700 1 --2- 172.21.15.67:0/446735186 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f483006ecc0 0x7f4830071180 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:34.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.576+0000 7f48493d8700 1 --2- 172.21.15.67:0/446735186 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4844100ea0 0x7f484419c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:34.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.576+0000 7f48493d8700 1 --2- 172.21.15.67:0/446735186 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4844101850 0x7f484419cc50 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:34.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.576+0000 7f48493d8700 1 -- 172.21.15.67:0/446735186 >> 172.21.15.67:0/446735186 conn(0x7f48440fc750 msgr2=0x7f48440fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:44:34.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.576+0000 7f48493d8700 1 -- 172.21.15.67:0/446735186 shutdown_connections 2024-04-22T23:44:34.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.576+0000 7f48493d8700 1 -- 172.21.15.67:0/446735186 wait complete. 2024-04-22T23:44:34.579 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:44:34.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.684+0000 7fa31a7dc700 1 -- 172.21.15.67:0/4045440585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa314101850 msgr2=0x7fa3141058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:34.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.684+0000 7fa31a7dc700 1 --2- 172.21.15.67:0/4045440585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa314101850 0x7fa3141058a0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fa308009910 tx=0x7fa308009c20 comp rx=0 tx=0).stop 2024-04-22T23:44:34.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.684+0000 7fa31a7dc700 1 -- 172.21.15.67:0/4045440585 shutdown_connections 2024-04-22T23:44:34.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.684+0000 7fa31a7dc700 1 --2- 172.21.15.67:0/4045440585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa314101850 0x7fa3141058a0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:34.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.684+0000 7fa31a7dc700 1 --2- 172.21.15.67:0/4045440585 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa314100ea0 0x7fa314101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:34.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.684+0000 7fa31a7dc700 1 -- 172.21.15.67:0/4045440585 >> 172.21.15.67:0/4045440585 conn(0x7fa3140fc750 msgr2=0x7fa3140feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:44:34.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.684+0000 7fa31a7dc700 1 -- 172.21.15.67:0/4045440585 shutdown_connections 2024-04-22T23:44:34.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.684+0000 7fa31a7dc700 1 -- 172.21.15.67:0/4045440585 wait complete. 2024-04-22T23:44:34.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.684+0000 7fa31a7dc700 1 Processor -- start 2024-04-22T23:44:34.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.684+0000 7fa31a7dc700 1 -- start start 2024-04-22T23:44:34.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.685+0000 7fa31a7dc700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa314100ea0 0x7fa31419e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:34.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.685+0000 7fa31a7dc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa314101850 0x7fa31419ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:34.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.685+0000 7fa31a7dc700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31419f520 con 0x7fa314101850 2024-04-22T23:44:34.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.685+0000 7fa31a7dc700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3141989d0 con 0x7fa314100ea0 2024-04-22T23:44:34.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.685+0000 7fa3137fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa314101850 0x7fa31419ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:34.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.685+0000 7fa3137fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa314101850 0x7fa31419ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40978/0 (socket says 172.21.15.67:40978) 2024-04-22T23:44:34.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.685+0000 7fa3137fe700 1 -- 172.21.15.67:0/2873612640 learned_addr learned my addr 172.21.15.67:0/2873612640 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:44:34.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.685+0000 7fa313fff700 1 --2- 172.21.15.67:0/2873612640 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa314100ea0 0x7fa31419e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:34.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.685+0000 7fa3137fe700 1 -- 172.21.15.67:0/2873612640 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa314100ea0 msgr2=0x7fa31419e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:34.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.685+0000 7fa3137fe700 1 --2- 172.21.15.67:0/2873612640 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa314100ea0 0x7fa31419e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:34.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.685+0000 7fa3137fe700 1 -- 172.21.15.67:0/2873612640 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3080095f0 con 0x7fa314101850 2024-04-22T23:44:34.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.685+0000 7fa3137fe700 1 --2- 172.21.15.67:0/2873612640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa314101850 0x7fa31419ee90 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fa3080092e0 tx=0x7fa30800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:44:34.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.686+0000 7fa3117fa700 1 -- 172.21.15.67:0/2873612640 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa30801d070 con 0x7fa314101850 2024-04-22T23:44:34.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.686+0000 7fa3117fa700 1 -- 172.21.15.67:0/2873612640 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa30800f460 con 0x7fa314101850 2024-04-22T23:44:34.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.686+0000 7fa3117fa700 1 -- 172.21.15.67:0/2873612640 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3080175d0 con 0x7fa314101850 2024-04-22T23:44:34.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.686+0000 7fa31a7dc700 1 -- 172.21.15.67:0/2873612640 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa314198c50 con 0x7fa314101850 2024-04-22T23:44:34.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.686+0000 7fa31a7dc700 1 -- 172.21.15.67:0/2873612640 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa314199140 con 0x7fa314101850 2024-04-22T23:44:34.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.687+0000 7fa31a7dc700 1 -- 172.21.15.67:0/2873612640 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa314105020 con 0x7fa314101850 2024-04-22T23:44:34.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.687+0000 7fa3117fa700 1 -- 172.21.15.67:0/2873612640 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa30800f5d0 con 0x7fa314101850 2024-04-22T23:44:34.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.688+0000 7fa3117fa700 1 --2- 172.21.15.67:0/2873612640 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa2fc06ecc0 0x7fa2fc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:34.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.688+0000 7fa3117fa700 1 -- 172.21.15.67:0/2873612640 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fa3080903d0 con 0x7fa314101850 2024-04-22T23:44:34.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.688+0000 7fa313fff700 1 --2- 172.21.15.67:0/2873612640 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa2fc06ecc0 0x7fa2fc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:34.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.688+0000 7fa313fff700 1 --2- 172.21.15.67:0/2873612640 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa2fc06ecc0 0x7fa2fc071180 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa304009a10 tx=0x7fa304009190 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:44:34.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.693+0000 7fa3117fa700 1 -- 172.21.15.67:0/2873612640 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fa30805cbc0 con 0x7fa314101850 2024-04-22T23:44:34.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.922+0000 7fa31a7dc700 1 -- 172.21.15.67:0/2873612640 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa3141993b0 con 0x7fa2fc06ecc0 2024-04-22T23:44:34.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.923+0000 7fa3117fa700 1 -- 172.21.15.67:0/2873612640 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa3141993b0 con 0x7fa2fc06ecc0 2024-04-22T23:44:34.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.926+0000 7fa31a7dc700 1 -- 172.21.15.67:0/2873612640 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa2fc06ecc0 msgr2=0x7fa2fc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:34.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.926+0000 7fa31a7dc700 1 --2- 172.21.15.67:0/2873612640 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa2fc06ecc0 0x7fa2fc071180 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa304009a10 tx=0x7fa304009190 comp rx=0 tx=0).stop 2024-04-22T23:44:34.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.926+0000 7fa31a7dc700 1 -- 172.21.15.67:0/2873612640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa314101850 msgr2=0x7fa31419ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:34.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.926+0000 7fa31a7dc700 1 --2- 172.21.15.67:0/2873612640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa314101850 0x7fa31419ee90 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fa3080092e0 tx=0x7fa30800bd70 comp rx=0 tx=0).stop 2024-04-22T23:44:34.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.926+0000 7fa31a7dc700 1 -- 172.21.15.67:0/2873612640 shutdown_connections 2024-04-22T23:44:34.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.926+0000 7fa31a7dc700 1 --2- 172.21.15.67:0/2873612640 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa2fc06ecc0 0x7fa2fc071180 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:34.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.926+0000 7fa31a7dc700 1 --2- 172.21.15.67:0/2873612640 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa314100ea0 0x7fa31419e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:34.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.926+0000 7fa31a7dc700 1 --2- 172.21.15.67:0/2873612640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa314101850 0x7fa31419ee90 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:34.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.926+0000 7fa31a7dc700 1 -- 172.21.15.67:0/2873612640 >> 172.21.15.67:0/2873612640 conn(0x7fa3140fc750 msgr2=0x7fa3140fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:44:34.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.926+0000 7fa31a7dc700 1 -- 172.21.15.67:0/2873612640 shutdown_connections 2024-04-22T23:44:34.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:34.926+0000 7fa31a7dc700 1 -- 172.21.15.67:0/2873612640 wait complete. 2024-04-22T23:44:35.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.033+0000 7f83d4d61700 1 -- 172.21.15.67:0/1059058879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83d00739d0 msgr2=0x7f83d010cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:35.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.033+0000 7f83d4d61700 1 --2- 172.21.15.67:0/1059058879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83d00739d0 0x7f83d010cfb0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f83b8009910 tx=0x7f83b8009c20 comp rx=0 tx=0).stop 2024-04-22T23:44:35.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.033+0000 7f83d4d61700 1 -- 172.21.15.67:0/1059058879 shutdown_connections 2024-04-22T23:44:35.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.033+0000 7f83d4d61700 1 --2- 172.21.15.67:0/1059058879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83d00739d0 0x7f83d010cfb0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:35.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.033+0000 7f83d4d61700 1 --2- 172.21.15.67:0/1059058879 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f83d00730b0 0x7f83d0073490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:35.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.033+0000 7f83d4d61700 1 -- 172.21.15.67:0/1059058879 >> 172.21.15.67:0/1059058879 conn(0x7f83d00fc740 msgr2=0x7f83d00feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:44:35.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.033+0000 7f83d4d61700 1 -- 172.21.15.67:0/1059058879 shutdown_connections 2024-04-22T23:44:35.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.033+0000 7f83d4d61700 1 -- 172.21.15.67:0/1059058879 wait complete. 2024-04-22T23:44:35.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.034+0000 7f83d4d61700 1 Processor -- start 2024-04-22T23:44:35.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.034+0000 7f83d4d61700 1 -- start start 2024-04-22T23:44:35.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.034+0000 7f83d4d61700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f83d00730b0 0x7f83d019e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:35.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.034+0000 7f83d4d61700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83d00739d0 0x7f83d019ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:35.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.034+0000 7f83d4d61700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83d019f3e0 con 0x7f83d00739d0 2024-04-22T23:44:35.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.034+0000 7f83d4d61700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83d01988e0 con 0x7f83d00730b0 2024-04-22T23:44:35.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.034+0000 7f83cdd9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83d00739d0 0x7f83d019ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:35.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.034+0000 7f83cdd9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83d00739d0 0x7f83d019ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41010/0 (socket says 172.21.15.67:41010) 2024-04-22T23:44:35.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.034+0000 7f83cdd9b700 1 -- 172.21.15.67:0/1057180727 learned_addr learned my addr 172.21.15.67:0/1057180727 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:44:35.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.034+0000 7f83ce59c700 1 --2- 172.21.15.67:0/1057180727 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f83d00730b0 0x7f83d019e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:35.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.034+0000 7f83cdd9b700 1 -- 172.21.15.67:0/1057180727 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f83d00730b0 msgr2=0x7f83d019e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:35.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.034+0000 7f83cdd9b700 1 --2- 172.21.15.67:0/1057180727 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f83d00730b0 0x7f83d019e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:35.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.034+0000 7f83cdd9b700 1 -- 172.21.15.67:0/1057180727 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b80095f0 con 0x7f83d00739d0 2024-04-22T23:44:35.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.035+0000 7f83cdd9b700 1 --2- 172.21.15.67:0/1057180727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83d00739d0 0x7f83d019ed50 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f83b8006010 tx=0x7f83b800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:44:35.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.035+0000 7f83c77fe700 1 -- 172.21.15.67:0/1057180727 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83b801d070 con 0x7f83d00739d0 2024-04-22T23:44:35.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.035+0000 7f83d4d61700 1 -- 172.21.15.67:0/1057180727 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83d0198b60 con 0x7f83d00739d0 2024-04-22T23:44:35.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.035+0000 7f83c77fe700 1 -- 172.21.15.67:0/1057180727 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f83b800f460 con 0x7f83d00739d0 2024-04-22T23:44:35.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.035+0000 7f83c77fe700 1 -- 172.21.15.67:0/1057180727 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83b8017630 con 0x7f83d00739d0 2024-04-22T23:44:35.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.035+0000 7f83d4d61700 1 -- 172.21.15.67:0/1057180727 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83d0199050 con 0x7f83d00739d0 2024-04-22T23:44:35.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.036+0000 7f83d4d61700 1 -- 172.21.15.67:0/1057180727 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83d010a6b0 con 0x7f83d00739d0 2024-04-22T23:44:35.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.036+0000 7f83c77fe700 1 -- 172.21.15.67:0/1057180727 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f83b800f5d0 con 0x7f83d00739d0 2024-04-22T23:44:35.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.037+0000 7f83c77fe700 1 --2- 172.21.15.67:0/1057180727 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f83bc06ecc0 0x7f83bc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:35.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.037+0000 7f83c77fe700 1 -- 172.21.15.67:0/1057180727 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f83b8090470 con 0x7f83d00739d0 2024-04-22T23:44:35.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.037+0000 7f83ce59c700 1 --2- 172.21.15.67:0/1057180727 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f83bc06ecc0 0x7f83bc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:35.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.037+0000 7f83ce59c700 1 --2- 172.21.15.67:0/1057180727 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f83bc06ecc0 0x7f83bc071180 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f83c8009d10 tx=0x7f83c8009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:44:35.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.042+0000 7f83c77fe700 1 -- 172.21.15.67:0/1057180727 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f83b805cbe0 con 0x7f83d00739d0 2024-04-22T23:44:35.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.273+0000 7f83d4d61700 1 -- 172.21.15.67:0/1057180727 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f83d0199270 con 0x7f83bc06ecc0 2024-04-22T23:44:35.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.284+0000 7f83c77fe700 1 -- 172.21.15.67:0/1057180727 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f83d0199270 con 0x7f83bc06ecc0 2024-04-22T23:44:35.277 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:44:35.277 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (18m) 11m ago 29m 25.5M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:44:35.277 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (29m) 11m ago 29m 9114k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:44:35.277 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (28m) 11m ago 28m 9042k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:44:35.277 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (15m) 11m ago 29m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:44:35.277 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (15m) 11m ago 28m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:44:35.277 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (17m) 11m ago 29m 79.3M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:44:35.277 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (11m) 11m ago 24m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:44:35.277 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (11m) 11m ago 24m 12.6M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:44:35.277 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (11m) 11m ago 24m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:44:35.277 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (24m) 11m ago 23m 12.2M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:44:35.278 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (19m) 11m ago 30m 511M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:44:35.278 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (19m) 11m ago 28m 453M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:44:35.278 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (16m) 11m ago 31m 55.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:44:35.278 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (16m) 11m ago 28m 46.0M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:44:35.278 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (18m) 11m ago 29m 15.4M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:44:35.278 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (18m) 11m ago 28m 16.1M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:44:35.278 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (15m) 11m ago 27m 99.2M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:44:35.278 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (14m) 11m ago 27m 127M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:44:35.278 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (14m) 11m ago 26m 79.9M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:44:35.278 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (13m) 11m ago 26m 97.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:44:35.278 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (12m) 11m ago 26m 82.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:44:35.278 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (12m) 11m ago 25m 77.6M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:44:35.278 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (18m) 11m ago 29m 73.9M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:44:35.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.286+0000 7f83d4d61700 1 -- 172.21.15.67:0/1057180727 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f83bc06ecc0 msgr2=0x7f83bc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:35.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.286+0000 7f83d4d61700 1 --2- 172.21.15.67:0/1057180727 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f83bc06ecc0 0x7f83bc071180 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f83c8009d10 tx=0x7f83c8009040 comp rx=0 tx=0).stop 2024-04-22T23:44:35.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.286+0000 7f83d4d61700 1 -- 172.21.15.67:0/1057180727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83d00739d0 msgr2=0x7f83d019ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:35.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.286+0000 7f83d4d61700 1 --2- 172.21.15.67:0/1057180727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83d00739d0 0x7f83d019ed50 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f83b8006010 tx=0x7f83b800be20 comp rx=0 tx=0).stop 2024-04-22T23:44:35.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.287+0000 7f83d4d61700 1 -- 172.21.15.67:0/1057180727 shutdown_connections 2024-04-22T23:44:35.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.287+0000 7f83d4d61700 1 --2- 172.21.15.67:0/1057180727 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f83bc06ecc0 0x7f83bc071180 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:35.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.287+0000 7f83d4d61700 1 --2- 172.21.15.67:0/1057180727 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f83d00730b0 0x7f83d019e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:35.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.287+0000 7f83d4d61700 1 --2- 172.21.15.67:0/1057180727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83d00739d0 0x7f83d019ed50 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:35.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.287+0000 7f83d4d61700 1 -- 172.21.15.67:0/1057180727 >> 172.21.15.67:0/1057180727 conn(0x7f83d00fc740 msgr2=0x7f83d01077f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:44:35.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.287+0000 7f83d4d61700 1 -- 172.21.15.67:0/1057180727 shutdown_connections 2024-04-22T23:44:35.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.287+0000 7f83d4d61700 1 -- 172.21.15.67:0/1057180727 wait complete. 2024-04-22T23:44:35.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.395+0000 7f2f48b1f700 1 -- 172.21.15.67:0/3981455160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f44101850 msgr2=0x7f2f441058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:35.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.395+0000 7f2f48b1f700 1 --2- 172.21.15.67:0/3981455160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f44101850 0x7f2f441058a0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f2f34009910 tx=0x7f2f34009c20 comp rx=0 tx=0).stop 2024-04-22T23:44:35.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.395+0000 7f2f48b1f700 1 -- 172.21.15.67:0/3981455160 shutdown_connections 2024-04-22T23:44:35.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.395+0000 7f2f48b1f700 1 --2- 172.21.15.67:0/3981455160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f44101850 0x7f2f441058a0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:35.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.395+0000 7f2f48b1f700 1 --2- 172.21.15.67:0/3981455160 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f44100ea0 0x7f2f44101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:35.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.395+0000 7f2f48b1f700 1 -- 172.21.15.67:0/3981455160 >> 172.21.15.67:0/3981455160 conn(0x7f2f440fc750 msgr2=0x7f2f440feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:44:35.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.395+0000 7f2f48b1f700 1 -- 172.21.15.67:0/3981455160 shutdown_connections 2024-04-22T23:44:35.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.395+0000 7f2f48b1f700 1 -- 172.21.15.67:0/3981455160 wait complete. 2024-04-22T23:44:35.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.396+0000 7f2f48b1f700 1 Processor -- start 2024-04-22T23:44:35.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.396+0000 7f2f48b1f700 1 -- start start 2024-04-22T23:44:35.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.396+0000 7f2f48b1f700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f44100ea0 0x7f2f4419e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:35.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.396+0000 7f2f48b1f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f44101850 0x7f2f4419ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:35.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.396+0000 7f2f48b1f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4419f4a0 con 0x7f2f44101850 2024-04-22T23:44:35.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.396+0000 7f2f48b1f700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f44198950 con 0x7f2f44100ea0 2024-04-22T23:44:35.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.396+0000 7f2f41d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f44101850 0x7f2f4419ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:35.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.396+0000 7f2f41d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f44101850 0x7f2f4419ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41016/0 (socket says 172.21.15.67:41016) 2024-04-22T23:44:35.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.396+0000 7f2f41d9b700 1 -- 172.21.15.67:0/3182823374 learned_addr learned my addr 172.21.15.67:0/3182823374 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:44:35.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.396+0000 7f2f4259c700 1 --2- 172.21.15.67:0/3182823374 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f44100ea0 0x7f2f4419e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:35.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.397+0000 7f2f41d9b700 1 -- 172.21.15.67:0/3182823374 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f44100ea0 msgr2=0x7f2f4419e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:35.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.397+0000 7f2f41d9b700 1 --2- 172.21.15.67:0/3182823374 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f44100ea0 0x7f2f4419e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:35.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.397+0000 7f2f41d9b700 1 -- 172.21.15.67:0/3182823374 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f340095f0 con 0x7f2f44101850 2024-04-22T23:44:35.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.397+0000 7f2f4259c700 1 --2- 172.21.15.67:0/3182823374 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f44100ea0 0x7f2f4419e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:44:35.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.397+0000 7f2f41d9b700 1 --2- 172.21.15.67:0/3182823374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f44101850 0x7f2f4419ee10 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f2f3400bd70 tx=0x7f2f3400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:44:35.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.397+0000 7f2f337fe700 1 -- 172.21.15.67:0/3182823374 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f3401d070 con 0x7f2f44101850 2024-04-22T23:44:35.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.397+0000 7f2f337fe700 1 -- 172.21.15.67:0/3182823374 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2f34005dc0 con 0x7f2f44101850 2024-04-22T23:44:35.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.397+0000 7f2f48b1f700 1 -- 172.21.15.67:0/3182823374 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f44198bd0 con 0x7f2f44101850 2024-04-22T23:44:35.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.397+0000 7f2f337fe700 1 -- 172.21.15.67:0/3182823374 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f34017400 con 0x7f2f44101850 2024-04-22T23:44:35.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.397+0000 7f2f48b1f700 1 -- 172.21.15.67:0/3182823374 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f441990c0 con 0x7f2f44101850 2024-04-22T23:44:35.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.398+0000 7f2f48b1f700 1 -- 172.21.15.67:0/3182823374 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f44105020 con 0x7f2f44101850 2024-04-22T23:44:35.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.399+0000 7f2f337fe700 1 -- 172.21.15.67:0/3182823374 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2f3400f7f0 con 0x7f2f44101850 2024-04-22T23:44:35.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.399+0000 7f2f337fe700 1 --2- 172.21.15.67:0/3182823374 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2f2c06ecc0 0x7f2f2c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:35.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.399+0000 7f2f337fe700 1 -- 172.21.15.67:0/3182823374 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f2f34090420 con 0x7f2f44101850 2024-04-22T23:44:35.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.399+0000 7f2f4259c700 1 --2- 172.21.15.67:0/3182823374 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2f2c06ecc0 0x7f2f2c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:35.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.400+0000 7f2f4259c700 1 --2- 172.21.15.67:0/3182823374 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2f2c06ecc0 0x7f2f2c071180 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2f38005fd0 tx=0x7f2f38005ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:44:35.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.404+0000 7f2f337fe700 1 -- 172.21.15.67:0/3182823374 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f2f3405cc10 con 0x7f2f44101850 2024-04-22T23:44:35.536 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:35 smithi067 ceph-mon[112438]: pgmap v546: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:35.536 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:35 smithi067 ceph-mon[112438]: from='client.35016 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:44:35.536 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:35 smithi067 ceph-mon[112438]: from='client.35020 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:44:35.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.706+0000 7f2f48b1f700 1 -- 172.21.15.67:0/3182823374 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2f4404f2e0 con 0x7f2f44101850 2024-04-22T23:44:35.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.707+0000 7f2f337fe700 1 -- 172.21.15.67:0/3182823374 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f2f3405c7a0 con 0x7f2f44101850 2024-04-22T23:44:35.699 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:44:35.699 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:44:35.699 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:44:35.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:44:35.699 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:44:35.699 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:44:35.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:44:35.699 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:44:35.699 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:44:35.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:44:35.700 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:44:35.700 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:44:35.700 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 3 2024-04-22T23:44:35.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:44:35.700 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:44:35.700 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:44:35.700 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 13 2024-04-22T23:44:35.700 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:44:35.700 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:44:35.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.709+0000 7f2f48b1f700 1 -- 172.21.15.67:0/3182823374 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2f2c06ecc0 msgr2=0x7f2f2c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:35.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.709+0000 7f2f48b1f700 1 --2- 172.21.15.67:0/3182823374 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2f2c06ecc0 0x7f2f2c071180 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2f38005fd0 tx=0x7f2f38005ee0 comp rx=0 tx=0).stop 2024-04-22T23:44:35.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.709+0000 7f2f48b1f700 1 -- 172.21.15.67:0/3182823374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f44101850 msgr2=0x7f2f4419ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:35.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.709+0000 7f2f48b1f700 1 --2- 172.21.15.67:0/3182823374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f44101850 0x7f2f4419ee10 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f2f3400bd70 tx=0x7f2f3400be50 comp rx=0 tx=0).stop 2024-04-22T23:44:35.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.709+0000 7f2f48b1f700 1 -- 172.21.15.67:0/3182823374 shutdown_connections 2024-04-22T23:44:35.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.709+0000 7f2f48b1f700 1 --2- 172.21.15.67:0/3182823374 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2f2c06ecc0 0x7f2f2c071180 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:35.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.709+0000 7f2f48b1f700 1 --2- 172.21.15.67:0/3182823374 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2f44100ea0 0x7f2f4419e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:35.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.709+0000 7f2f48b1f700 1 --2- 172.21.15.67:0/3182823374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f44101850 0x7f2f4419ee10 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:35.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.709+0000 7f2f48b1f700 1 -- 172.21.15.67:0/3182823374 >> 172.21.15.67:0/3182823374 conn(0x7f2f440fc750 msgr2=0x7f2f440feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:44:35.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.709+0000 7f2f48b1f700 1 -- 172.21.15.67:0/3182823374 shutdown_connections 2024-04-22T23:44:35.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.709+0000 7f2f48b1f700 1 -- 172.21.15.67:0/3182823374 wait complete. 2024-04-22T23:44:35.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:35 smithi134 ceph-mon[85630]: pgmap v546: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:35.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:35 smithi134 ceph-mon[85630]: from='client.35016 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:44:35.736 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:35 smithi134 ceph-mon[85630]: from='client.35020 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:44:35.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.816+0000 7fa8a177e700 1 -- 172.21.15.67:0/3709722892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa89c100ea0 msgr2=0x7fa89c101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:35.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.816+0000 7fa8a177e700 1 --2- 172.21.15.67:0/3709722892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa89c100ea0 0x7fa89c101280 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fa88c009910 tx=0x7fa88c009c20 comp rx=0 tx=0).stop 2024-04-22T23:44:35.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.816+0000 7fa8a177e700 1 -- 172.21.15.67:0/3709722892 shutdown_connections 2024-04-22T23:44:35.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.816+0000 7fa8a177e700 1 --2- 172.21.15.67:0/3709722892 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa89c101850 0x7fa89c1058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:35.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.816+0000 7fa8a177e700 1 --2- 172.21.15.67:0/3709722892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa89c100ea0 0x7fa89c101280 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:35.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.816+0000 7fa8a177e700 1 -- 172.21.15.67:0/3709722892 >> 172.21.15.67:0/3709722892 conn(0x7fa89c0fc750 msgr2=0x7fa89c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:44:35.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.816+0000 7fa8a177e700 1 -- 172.21.15.67:0/3709722892 shutdown_connections 2024-04-22T23:44:35.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.816+0000 7fa8a177e700 1 -- 172.21.15.67:0/3709722892 wait complete. 2024-04-22T23:44:35.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.816+0000 7fa8a177e700 1 Processor -- start 2024-04-22T23:44:35.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.817+0000 7fa8a177e700 1 -- start start 2024-04-22T23:44:35.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.817+0000 7fa8a177e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa89c100ea0 0x7fa89c19c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:35.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.817+0000 7fa8a177e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa89c101850 0x7fa89c19cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:35.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.817+0000 7fa8a177e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa89c19d2c0 con 0x7fa89c101850 2024-04-22T23:44:35.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.817+0000 7fa8a177e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa89c196770 con 0x7fa89c100ea0 2024-04-22T23:44:35.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.817+0000 7fa89a7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa89c101850 0x7fa89c19cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:35.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.817+0000 7fa89a7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa89c101850 0x7fa89c19cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41040/0 (socket says 172.21.15.67:41040) 2024-04-22T23:44:35.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.817+0000 7fa89a7fc700 1 -- 172.21.15.67:0/3714868872 learned_addr learned my addr 172.21.15.67:0/3714868872 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:44:35.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.817+0000 7fa89affd700 1 --2- 172.21.15.67:0/3714868872 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa89c100ea0 0x7fa89c19c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:35.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.817+0000 7fa89a7fc700 1 -- 172.21.15.67:0/3714868872 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa89c100ea0 msgr2=0x7fa89c19c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:35.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.817+0000 7fa89a7fc700 1 --2- 172.21.15.67:0/3714868872 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa89c100ea0 0x7fa89c19c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:35.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.817+0000 7fa89a7fc700 1 -- 172.21.15.67:0/3714868872 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa88c0095f0 con 0x7fa89c101850 2024-04-22T23:44:35.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.817+0000 7fa89a7fc700 1 --2- 172.21.15.67:0/3714868872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa89c101850 0x7fa89c19cc30 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fa89000d900 tx=0x7fa89000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:44:35.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.817+0000 7fa883fff700 1 -- 172.21.15.67:0/3714868872 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa890009900 con 0x7fa89c101850 2024-04-22T23:44:35.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.818+0000 7fa883fff700 1 -- 172.21.15.67:0/3714868872 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa890010460 con 0x7fa89c101850 2024-04-22T23:44:35.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.818+0000 7fa8a177e700 1 -- 172.21.15.67:0/3714868872 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa89c196a50 con 0x7fa89c101850 2024-04-22T23:44:35.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.818+0000 7fa883fff700 1 -- 172.21.15.67:0/3714868872 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa89000f5f0 con 0x7fa89c101850 2024-04-22T23:44:35.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.818+0000 7fa8a177e700 1 -- 172.21.15.67:0/3714868872 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa89c196fa0 con 0x7fa89c101850 2024-04-22T23:44:35.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.818+0000 7fa8a177e700 1 -- 172.21.15.67:0/3714868872 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa89c109240 con 0x7fa89c101850 2024-04-22T23:44:35.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.819+0000 7fa883fff700 1 -- 172.21.15.67:0/3714868872 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa8900105d0 con 0x7fa89c101850 2024-04-22T23:44:35.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.819+0000 7fa883fff700 1 --2- 172.21.15.67:0/3714868872 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa88406ecc0 0x7fa884071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:35.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.820+0000 7fa883fff700 1 -- 172.21.15.67:0/3714868872 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fa89008ee30 con 0x7fa89c101850 2024-04-22T23:44:35.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.820+0000 7fa89affd700 1 --2- 172.21.15.67:0/3714868872 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa88406ecc0 0x7fa884071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:35.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.820+0000 7fa89affd700 1 --2- 172.21.15.67:0/3714868872 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa88406ecc0 0x7fa884071180 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa88c00b5c0 tx=0x7fa88c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:44:35.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:35.824+0000 7fa883fff700 1 -- 172.21.15.67:0/3714868872 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fa89005b5a0 con 0x7fa89c101850 2024-04-22T23:44:36.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.083+0000 7fa8a177e700 1 -- 172.21.15.67:0/3714868872 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa89c04f2e0 con 0x7fa89c101850 2024-04-22T23:44:36.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.084+0000 7fa883fff700 1 -- 172.21.15.67:0/3714868872 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1871 (secure 0 0 0) 0x7fa89005b130 con 0x7fa89c101850 2024-04-22T23:44:36.076 INFO:teuthology.orchestra.run.smithi067.stdout:e31 2024-04-22T23:44:36.076 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:44:36.077 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:44:36.077 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:44:36.077 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:44:36.077 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:44:36.077 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:44:36.077 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:44:36.077 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:44:36.077 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:44:36.077 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:44:36.077 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:44:36.077 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:44:36.077 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:44:36.077 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:44:36.077 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:44:36.078 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:44:36.078 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:44:36.078 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:44:36.078 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:44:36.078 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:44:36.078 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:44:36.078 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:44:36.078 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:44:36.078 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:44:36.078 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:44:36.078 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:44:36.078 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:44:36.079 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:44:36.079 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:44:36.079 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:44:36.079 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:44:36.079 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:44:36.079 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:44:36.079 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:44:36.079 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:44:36.079 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:44:36.079 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:44:36.079 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:44:36.079 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:44:36.079 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44259} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1827288036,v1:172.21.15.134:6827/1827288036] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:44:36.079 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:44:36.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.086+0000 7fa8a177e700 1 -- 172.21.15.67:0/3714868872 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa88406ecc0 msgr2=0x7fa884071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:36.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.086+0000 7fa8a177e700 1 --2- 172.21.15.67:0/3714868872 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa88406ecc0 0x7fa884071180 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa88c00b5c0 tx=0x7fa88c01a040 comp rx=0 tx=0).stop 2024-04-22T23:44:36.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.086+0000 7fa8a177e700 1 -- 172.21.15.67:0/3714868872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa89c101850 msgr2=0x7fa89c19cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:36.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.086+0000 7fa8a177e700 1 --2- 172.21.15.67:0/3714868872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa89c101850 0x7fa89c19cc30 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fa89000d900 tx=0x7fa89000dcc0 comp rx=0 tx=0).stop 2024-04-22T23:44:36.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.086+0000 7fa8a177e700 1 -- 172.21.15.67:0/3714868872 shutdown_connections 2024-04-22T23:44:36.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.086+0000 7fa8a177e700 1 --2- 172.21.15.67:0/3714868872 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa88406ecc0 0x7fa884071180 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:36.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.086+0000 7fa8a177e700 1 --2- 172.21.15.67:0/3714868872 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa89c100ea0 0x7fa89c19c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:36.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.086+0000 7fa8a177e700 1 --2- 172.21.15.67:0/3714868872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa89c101850 0x7fa89c19cc30 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:36.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.086+0000 7fa8a177e700 1 -- 172.21.15.67:0/3714868872 >> 172.21.15.67:0/3714868872 conn(0x7fa89c0fc750 msgr2=0x7fa89c0fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:44:36.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.086+0000 7fa8a177e700 1 -- 172.21.15.67:0/3714868872 shutdown_connections 2024-04-22T23:44:36.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.086+0000 7fa8a177e700 1 -- 172.21.15.67:0/3714868872 wait complete. 2024-04-22T23:44:36.081 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 31 2024-04-22T23:44:36.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.195+0000 7fe0735a4700 1 -- 172.21.15.67:0/2644840244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe06c106db0 msgr2=0x7fe06c107190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:36.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.195+0000 7fe0735a4700 1 --2- 172.21.15.67:0/2644840244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe06c106db0 0x7fe06c107190 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fe068009910 tx=0x7fe068009c20 comp rx=0 tx=0).stop 2024-04-22T23:44:36.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.195+0000 7fe0735a4700 1 -- 172.21.15.67:0/2644840244 shutdown_connections 2024-04-22T23:44:36.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.195+0000 7fe0735a4700 1 --2- 172.21.15.67:0/2644840244 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe06c0ff410 0x7fe06c0ff890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:36.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.195+0000 7fe0735a4700 1 --2- 172.21.15.67:0/2644840244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe06c106db0 0x7fe06c107190 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:36.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.195+0000 7fe0735a4700 1 -- 172.21.15.67:0/2644840244 >> 172.21.15.67:0/2644840244 conn(0x7fe06c074e30 msgr2=0x7fe06c075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:44:36.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.195+0000 7fe0735a4700 1 -- 172.21.15.67:0/2644840244 shutdown_connections 2024-04-22T23:44:36.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.195+0000 7fe0735a4700 1 -- 172.21.15.67:0/2644840244 wait complete. 2024-04-22T23:44:36.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.195+0000 7fe0735a4700 1 Processor -- start 2024-04-22T23:44:36.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.195+0000 7fe0735a4700 1 -- start start 2024-04-22T23:44:36.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.195+0000 7fe0735a4700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe06c0ff410 0x7fe06c1005f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:36.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.195+0000 7fe0735a4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe06c106db0 0x7fe06c100b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:36.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.196+0000 7fe0735a4700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe06c101070 con 0x7fe06c106db0 2024-04-22T23:44:36.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.196+0000 7fe0735a4700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe06c1011b0 con 0x7fe06c0ff410 2024-04-22T23:44:36.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.196+0000 7fe070b3f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe06c106db0 0x7fe06c100b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:36.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.196+0000 7fe070b3f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe06c106db0 0x7fe06c100b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41044/0 (socket says 172.21.15.67:41044) 2024-04-22T23:44:36.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.196+0000 7fe070b3f700 1 -- 172.21.15.67:0/2552023112 learned_addr learned my addr 172.21.15.67:0/2552023112 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:44:36.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.196+0000 7fe071340700 1 --2- 172.21.15.67:0/2552023112 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe06c0ff410 0x7fe06c1005f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:36.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.196+0000 7fe070b3f700 1 -- 172.21.15.67:0/2552023112 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe06c0ff410 msgr2=0x7fe06c1005f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:36.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.196+0000 7fe070b3f700 1 --2- 172.21.15.67:0/2552023112 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe06c0ff410 0x7fe06c1005f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:36.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.196+0000 7fe070b3f700 1 -- 172.21.15.67:0/2552023112 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0680095f0 con 0x7fe06c106db0 2024-04-22T23:44:36.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.196+0000 7fe070b3f700 1 --2- 172.21.15.67:0/2552023112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe06c106db0 0x7fe06c100b30 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe05c00d8d0 tx=0x7fe05c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:44:36.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.196+0000 7fe0627fc700 1 -- 172.21.15.67:0/2552023112 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe05c0098a0 con 0x7fe06c106db0 2024-04-22T23:44:36.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.196+0000 7fe0627fc700 1 -- 172.21.15.67:0/2552023112 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe05c010460 con 0x7fe06c106db0 2024-04-22T23:44:36.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.196+0000 7fe0735a4700 1 -- 172.21.15.67:0/2552023112 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe06c1a2ba0 con 0x7fe06c106db0 2024-04-22T23:44:36.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.196+0000 7fe0627fc700 1 -- 172.21.15.67:0/2552023112 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe05c00f5f0 con 0x7fe06c106db0 2024-04-22T23:44:36.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.197+0000 7fe0735a4700 1 -- 172.21.15.67:0/2552023112 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe06c1a2f00 con 0x7fe06c106db0 2024-04-22T23:44:36.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.197+0000 7fe0735a4700 1 -- 172.21.15.67:0/2552023112 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe06c1013d0 con 0x7fe06c106db0 2024-04-22T23:44:36.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.198+0000 7fe0627fc700 1 -- 172.21.15.67:0/2552023112 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe05c009a00 con 0x7fe06c106db0 2024-04-22T23:44:36.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.198+0000 7fe0627fc700 1 --2- 172.21.15.67:0/2552023112 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe05806ecc0 0x7fe058071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:36.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.199+0000 7fe0627fc700 1 -- 172.21.15.67:0/2552023112 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fe05c08edb0 con 0x7fe06c106db0 2024-04-22T23:44:36.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.199+0000 7fe071340700 1 --2- 172.21.15.67:0/2552023112 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe05806ecc0 0x7fe058071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:36.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.199+0000 7fe071340700 1 --2- 172.21.15.67:0/2552023112 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe05806ecc0 0x7fe058071180 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe068006010 tx=0x7fe06801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:44:36.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.203+0000 7fe0627fc700 1 -- 172.21.15.67:0/2552023112 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe05c057a70 con 0x7fe06c106db0 2024-04-22T23:44:36.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.432+0000 7fe0735a4700 1 -- 172.21.15.67:0/2552023112 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe06c101b20 con 0x7fe05806ecc0 2024-04-22T23:44:36.425 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:36 smithi067 ceph-mon[112438]: from='client.35024 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:44:36.425 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:36 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3182823374' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:44:36.425 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:36 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3714868872' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:44:36.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.434+0000 7fe0627fc700 1 -- 172.21.15.67:0/2552023112 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe06c101b20 con 0x7fe05806ecc0 2024-04-22T23:44:36.426 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:44:36.426 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:44:36.426 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:44:36.426 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:44:36.426 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:44:36.426 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:44:36.426 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:44:36.426 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:44:36.426 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:44:36.427 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:44:36.427 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T23:44:36.427 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:44:36.427 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:44:36.427 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:44:36.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.436+0000 7fe0735a4700 1 -- 172.21.15.67:0/2552023112 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe05806ecc0 msgr2=0x7fe058071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:36.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.436+0000 7fe0735a4700 1 --2- 172.21.15.67:0/2552023112 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe05806ecc0 0x7fe058071180 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe068006010 tx=0x7fe06801a040 comp rx=0 tx=0).stop 2024-04-22T23:44:36.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.436+0000 7fe0735a4700 1 -- 172.21.15.67:0/2552023112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe06c106db0 msgr2=0x7fe06c100b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:36.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.436+0000 7fe0735a4700 1 --2- 172.21.15.67:0/2552023112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe06c106db0 0x7fe06c100b30 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe05c00d8d0 tx=0x7fe05c00dbe0 comp rx=0 tx=0).stop 2024-04-22T23:44:36.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.436+0000 7fe0735a4700 1 -- 172.21.15.67:0/2552023112 shutdown_connections 2024-04-22T23:44:36.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.436+0000 7fe0735a4700 1 --2- 172.21.15.67:0/2552023112 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe05806ecc0 0x7fe058071180 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:36.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.436+0000 7fe0735a4700 1 --2- 172.21.15.67:0/2552023112 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe06c0ff410 0x7fe06c1005f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:36.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.436+0000 7fe0735a4700 1 --2- 172.21.15.67:0/2552023112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe06c106db0 0x7fe06c100b30 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:36.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.436+0000 7fe0735a4700 1 -- 172.21.15.67:0/2552023112 >> 172.21.15.67:0/2552023112 conn(0x7fe06c074e30 msgr2=0x7fe06c0fd460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:44:36.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.436+0000 7fe0735a4700 1 -- 172.21.15.67:0/2552023112 shutdown_connections 2024-04-22T23:44:36.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.437+0000 7fe0735a4700 1 -- 172.21.15.67:0/2552023112 wait complete. 2024-04-22T23:44:36.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.543+0000 7f7baee77700 1 -- 172.21.15.67:0/804508298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ba8101850 msgr2=0x7f7ba81058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:36.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.543+0000 7f7baee77700 1 --2- 172.21.15.67:0/804508298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ba8101850 0x7f7ba81058a0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f7b94009910 tx=0x7f7b94009c20 comp rx=0 tx=0).stop 2024-04-22T23:44:36.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.543+0000 7f7baee77700 1 -- 172.21.15.67:0/804508298 shutdown_connections 2024-04-22T23:44:36.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.543+0000 7f7baee77700 1 --2- 172.21.15.67:0/804508298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ba8101850 0x7f7ba81058a0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:36.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.543+0000 7f7baee77700 1 --2- 172.21.15.67:0/804508298 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7ba8100ea0 0x7f7ba8101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:36.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.543+0000 7f7baee77700 1 -- 172.21.15.67:0/804508298 >> 172.21.15.67:0/804508298 conn(0x7f7ba80fc750 msgr2=0x7f7ba80feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:44:36.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.543+0000 7f7baee77700 1 -- 172.21.15.67:0/804508298 shutdown_connections 2024-04-22T23:44:36.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.543+0000 7f7baee77700 1 -- 172.21.15.67:0/804508298 wait complete. 2024-04-22T23:44:36.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.544+0000 7f7baee77700 1 Processor -- start 2024-04-22T23:44:36.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.544+0000 7f7baee77700 1 -- start start 2024-04-22T23:44:36.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.544+0000 7f7baee77700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7ba8100ea0 0x7f7ba819c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:36.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.544+0000 7f7baee77700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ba8101850 0x7f7ba819cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:36.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.544+0000 7f7baee77700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba819d2d0 con 0x7f7ba8101850 2024-04-22T23:44:36.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.544+0000 7f7baee77700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba8196780 con 0x7f7ba8100ea0 2024-04-22T23:44:36.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.544+0000 7f7ba7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ba8101850 0x7f7ba819cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:36.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.544+0000 7f7ba7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ba8101850 0x7f7ba819cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41068/0 (socket says 172.21.15.67:41068) 2024-04-22T23:44:36.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.544+0000 7f7ba7fff700 1 -- 172.21.15.67:0/2020441091 learned_addr learned my addr 172.21.15.67:0/2020441091 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:44:36.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.544+0000 7f7bacc13700 1 --2- 172.21.15.67:0/2020441091 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7ba8100ea0 0x7f7ba819c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:36.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.544+0000 7f7ba7fff700 1 -- 172.21.15.67:0/2020441091 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7ba8100ea0 msgr2=0x7f7ba819c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:36.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.544+0000 7f7ba7fff700 1 --2- 172.21.15.67:0/2020441091 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7ba8100ea0 0x7f7ba819c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:36.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.544+0000 7f7ba7fff700 1 -- 172.21.15.67:0/2020441091 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b940095f0 con 0x7f7ba8101850 2024-04-22T23:44:36.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.544+0000 7f7ba7fff700 1 --2- 172.21.15.67:0/2020441091 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ba8101850 0x7f7ba819cc40 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f7b940092e0 tx=0x7f7b9400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:44:36.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.545+0000 7f7ba5ffb700 1 -- 172.21.15.67:0/2020441091 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b9401d070 con 0x7f7ba8101850 2024-04-22T23:44:36.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.545+0000 7f7baee77700 1 -- 172.21.15.67:0/2020441091 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ba8196a00 con 0x7f7ba8101850 2024-04-22T23:44:36.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.545+0000 7f7ba5ffb700 1 -- 172.21.15.67:0/2020441091 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b94005dc0 con 0x7f7ba8101850 2024-04-22T23:44:36.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.545+0000 7f7ba5ffb700 1 -- 172.21.15.67:0/2020441091 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b94017400 con 0x7f7ba8101850 2024-04-22T23:44:36.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.545+0000 7f7baee77700 1 -- 172.21.15.67:0/2020441091 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ba8196ef0 con 0x7f7ba8101850 2024-04-22T23:44:36.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.546+0000 7f7baee77700 1 -- 172.21.15.67:0/2020441091 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ba81092a0 con 0x7f7ba8101850 2024-04-22T23:44:36.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.546+0000 7f7ba5ffb700 1 -- 172.21.15.67:0/2020441091 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7b9400f6f0 con 0x7f7ba8101850 2024-04-22T23:44:36.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.547+0000 7f7ba5ffb700 1 --2- 172.21.15.67:0/2020441091 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7b9806ecc0 0x7f7b98071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:44:36.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.547+0000 7f7ba5ffb700 1 -- 172.21.15.67:0/2020441091 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f7b940903c0 con 0x7f7ba8101850 2024-04-22T23:44:36.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.547+0000 7f7bacc13700 1 --2- 172.21.15.67:0/2020441091 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7b9806ecc0 0x7f7b98071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:44:36.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.548+0000 7f7bacc13700 1 --2- 172.21.15.67:0/2020441091 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7b9806ecc0 0x7f7b98071180 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c005950 tx=0x7f7b9c00b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:44:36.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.552+0000 7f7ba5ffb700 1 -- 172.21.15.67:0/2020441091 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7b9405cb30 con 0x7f7ba8101850 2024-04-22T23:44:36.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:36 smithi134 ceph-mon[85630]: from='client.35024 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:44:36.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:36 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3182823374' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:44:36.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:36 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3714868872' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:44:36.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.846+0000 7f7baee77700 1 -- 172.21.15.67:0/2020441091 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7ba804f2e0 con 0x7f7ba8101850 2024-04-22T23:44:36.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.846+0000 7f7ba5ffb700 1 -- 172.21.15.67:0/2020441091 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f7b9405c6c0 con 0x7f7ba8101850 2024-04-22T23:44:36.838 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:44:36.838 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:44:36.838 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:44:36.839 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:44:36.839 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:44:36.839 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:44:36.839 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:44:36.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.848+0000 7f7baee77700 1 -- 172.21.15.67:0/2020441091 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7b9806ecc0 msgr2=0x7f7b98071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:36.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.848+0000 7f7baee77700 1 --2- 172.21.15.67:0/2020441091 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7b9806ecc0 0x7f7b98071180 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c005950 tx=0x7f7b9c00b410 comp rx=0 tx=0).stop 2024-04-22T23:44:36.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.848+0000 7f7baee77700 1 -- 172.21.15.67:0/2020441091 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ba8101850 msgr2=0x7f7ba819cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:44:36.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.848+0000 7f7baee77700 1 --2- 172.21.15.67:0/2020441091 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ba8101850 0x7f7ba819cc40 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f7b940092e0 tx=0x7f7b9400bd70 comp rx=0 tx=0).stop 2024-04-22T23:44:36.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.848+0000 7f7baee77700 1 -- 172.21.15.67:0/2020441091 shutdown_connections 2024-04-22T23:44:36.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.848+0000 7f7baee77700 1 --2- 172.21.15.67:0/2020441091 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7b9806ecc0 0x7f7b98071180 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:36.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.848+0000 7f7baee77700 1 --2- 172.21.15.67:0/2020441091 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7ba8100ea0 0x7f7ba819c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:36.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.848+0000 7f7baee77700 1 --2- 172.21.15.67:0/2020441091 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ba8101850 0x7f7ba819cc40 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:44:36.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.848+0000 7f7baee77700 1 -- 172.21.15.67:0/2020441091 >> 172.21.15.67:0/2020441091 conn(0x7f7ba80fc750 msgr2=0x7f7ba80fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:44:36.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.848+0000 7f7baee77700 1 -- 172.21.15.67:0/2020441091 shutdown_connections 2024-04-22T23:44:36.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:44:36.848+0000 7f7baee77700 1 -- 172.21.15.67:0/2020441091 wait complete. 2024-04-22T23:44:37.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:37 smithi134 ceph-mon[85630]: pgmap v547: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:37.736 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:37 smithi134 ceph-mon[85630]: from='client.35036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:44:37.736 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:37 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2020441091' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:44:37.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:37 smithi067 ceph-mon[112438]: pgmap v547: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:37.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:37 smithi067 ceph-mon[112438]: from='client.35036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:44:37.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:37 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2020441091' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:44:39.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:39 smithi134 ceph-mon[85630]: pgmap v548: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:39.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:39 smithi067 ceph-mon[112438]: pgmap v548: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:41.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:41 smithi134 ceph-mon[85630]: pgmap v549: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:41.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:44:41.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:41 smithi067 ceph-mon[112438]: pgmap v549: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:41.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:44:43.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:43 smithi134 ceph-mon[85630]: pgmap v550: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:43.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:43 smithi067 ceph-mon[112438]: pgmap v550: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:45.536 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:45 smithi067 ceph-mon[112438]: pgmap v551: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:45.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:45 smithi134 ceph-mon[85630]: pgmap v551: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:47.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:47 smithi134 ceph-mon[85630]: pgmap v552: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:47.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:47 smithi067 ceph-mon[112438]: pgmap v552: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:49.613 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:49 smithi134 ceph-mon[85630]: pgmap v553: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:49.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:49 smithi067 ceph-mon[112438]: pgmap v553: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:51.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:51 smithi134 ceph-mon[85630]: pgmap v554: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:51.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:51 smithi067 ceph-mon[112438]: pgmap v554: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:53.736 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:53 smithi134 ceph-mon[85630]: pgmap v555: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:53.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:53 smithi067 ceph-mon[112438]: pgmap v555: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:55.536 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:55 smithi067 ceph-mon[112438]: pgmap v556: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:55.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:55 smithi134 ceph-mon[85630]: pgmap v556: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:56.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:44:56.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:44:57.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:57 smithi134 ceph-mon[85630]: pgmap v557: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:57.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:57 smithi067 ceph-mon[112438]: pgmap v557: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:59.719 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:44:59 smithi134 ceph-mon[85630]: pgmap v558: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:44:59.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:44:59 smithi067 ceph-mon[112438]: pgmap v558: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:01.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:01 smithi134 ceph-mon[85630]: pgmap v559: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:01.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:01 smithi067 ceph-mon[112438]: pgmap v559: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:03.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:03 smithi134 ceph-mon[85630]: pgmap v560: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:03.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:03 smithi067 ceph-mon[112438]: pgmap v560: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:05.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:05 smithi134 ceph-mon[85630]: pgmap v561: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:05.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:05 smithi067 ceph-mon[112438]: pgmap v561: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:06.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:06.970+0000 7f0d399a2700 1 -- 172.21.15.67:0/1998940802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d34106a50 msgr2=0x7f0d34106e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:06.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:06.970+0000 7f0d399a2700 1 --2- 172.21.15.67:0/1998940802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d34106a50 0x7f0d34106e30 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c00b0a0 tx=0x7f0d2c00b3b0 comp rx=0 tx=0).stop 2024-04-22T23:45:06.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:06.970+0000 7f0d399a2700 1 -- 172.21.15.67:0/1998940802 shutdown_connections 2024-04-22T23:45:06.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:06.970+0000 7f0d399a2700 1 --2- 172.21.15.67:0/1998940802 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0d34068580 0x7f0d34068a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:06.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:06.970+0000 7f0d399a2700 1 --2- 172.21.15.67:0/1998940802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d34106a50 0x7f0d34106e30 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:06.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:06.970+0000 7f0d399a2700 1 -- 172.21.15.67:0/1998940802 >> 172.21.15.67:0/1998940802 conn(0x7f0d34075460 msgr2=0x7f0d34075870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:45:06.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:06.970+0000 7f0d399a2700 1 -- 172.21.15.67:0/1998940802 shutdown_connections 2024-04-22T23:45:06.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:06.970+0000 7f0d399a2700 1 -- 172.21.15.67:0/1998940802 wait complete. 2024-04-22T23:45:06.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:06.971+0000 7f0d399a2700 1 Processor -- start 2024-04-22T23:45:06.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:06.971+0000 7f0d399a2700 1 -- start start 2024-04-22T23:45:06.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:06.971+0000 7f0d399a2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d34068580 0x7f0d340ffaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:06.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:06.971+0000 7f0d399a2700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0d34106a50 0x7f0d34100030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:06.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:06.971+0000 7f0d399a2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d34100570 con 0x7f0d34068580 2024-04-22T23:45:06.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:06.971+0000 7f0d399a2700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d341006b0 con 0x7f0d34106a50 2024-04-22T23:45:06.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:06.971+0000 7f0d389a0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d34068580 0x7f0d340ffaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:06.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:06.971+0000 7f0d389a0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d34068580 0x7f0d340ffaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36960/0 (socket says 172.21.15.67:36960) 2024-04-22T23:45:06.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:06.971+0000 7f0d389a0700 1 -- 172.21.15.67:0/2551113727 learned_addr learned my addr 172.21.15.67:0/2551113727 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:45:06.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:06.971+0000 7f0d33fff700 1 --2- 172.21.15.67:0/2551113727 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0d34106a50 0x7f0d34100030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:06.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:06.971+0000 7f0d389a0700 1 -- 172.21.15.67:0/2551113727 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0d34106a50 msgr2=0x7f0d34100030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:06.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:06.971+0000 7f0d389a0700 1 --2- 172.21.15.67:0/2551113727 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0d34106a50 0x7f0d34100030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:06.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:06.971+0000 7f0d389a0700 1 -- 172.21.15.67:0/2551113727 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d2c009d00 con 0x7f0d34068580 2024-04-22T23:45:06.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:06.971+0000 7f0d33fff700 1 --2- 172.21.15.67:0/2551113727 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0d34106a50 0x7f0d34100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:45:06.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:06.972+0000 7f0d389a0700 1 --2- 172.21.15.67:0/2551113727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d34068580 0x7f0d340ffaf0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c000f80 tx=0x7f0d2c003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:45:06.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:06.972+0000 7f0d31ffb700 1 -- 172.21.15.67:0/2551113727 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d2c00e040 con 0x7f0d34068580 2024-04-22T23:45:06.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:06.972+0000 7f0d31ffb700 1 -- 172.21.15.67:0/2551113727 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0d2c007c00 con 0x7f0d34068580 2024-04-22T23:45:06.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:06.972+0000 7f0d31ffb700 1 -- 172.21.15.67:0/2551113727 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d2c01d740 con 0x7f0d34068580 2024-04-22T23:45:06.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:06.972+0000 7f0d399a2700 1 -- 172.21.15.67:0/2551113727 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d34100930 con 0x7f0d34068580 2024-04-22T23:45:06.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:06.972+0000 7f0d399a2700 1 -- 172.21.15.67:0/2551113727 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d341a2c30 con 0x7f0d34068580 2024-04-22T23:45:06.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:06.973+0000 7f0d399a2700 1 -- 172.21.15.67:0/2551113727 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d34108f90 con 0x7f0d34068580 2024-04-22T23:45:06.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:06.974+0000 7f0d31ffb700 1 -- 172.21.15.67:0/2551113727 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0d2c004040 con 0x7f0d34068580 2024-04-22T23:45:06.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:06.974+0000 7f0d31ffb700 1 --2- 172.21.15.67:0/2551113727 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0d1c06ecc0 0x7f0d1c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:06.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:06.974+0000 7f0d31ffb700 1 -- 172.21.15.67:0/2551113727 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f0d2c090780 con 0x7f0d34068580 2024-04-22T23:45:06.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:06.974+0000 7f0d33fff700 1 --2- 172.21.15.67:0/2551113727 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0d1c06ecc0 0x7f0d1c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:06.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:06.975+0000 7f0d33fff700 1 --2- 172.21.15.67:0/2551113727 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0d1c06ecc0 0x7f0d1c071180 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f0d34101290 tx=0x7f0d28008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:45:06.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:06.979+0000 7f0d31ffb700 1 -- 172.21.15.67:0/2551113727 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f0d2c059440 con 0x7f0d34068580 2024-04-22T23:45:07.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.210+0000 7f0d399a2700 1 -- 172.21.15.67:0/2551113727 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d340611d0 con 0x7f0d1c06ecc0 2024-04-22T23:45:07.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.211+0000 7f0d31ffb700 1 -- 172.21.15.67:0/2551113727 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f0d340611d0 con 0x7f0d1c06ecc0 2024-04-22T23:45:07.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.213+0000 7f0d399a2700 1 -- 172.21.15.67:0/2551113727 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0d1c06ecc0 msgr2=0x7f0d1c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:07.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.213+0000 7f0d399a2700 1 --2- 172.21.15.67:0/2551113727 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0d1c06ecc0 0x7f0d1c071180 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f0d34101290 tx=0x7f0d28008040 comp rx=0 tx=0).stop 2024-04-22T23:45:07.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.213+0000 7f0d399a2700 1 -- 172.21.15.67:0/2551113727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d34068580 msgr2=0x7f0d340ffaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:07.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.213+0000 7f0d399a2700 1 --2- 172.21.15.67:0/2551113727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d34068580 0x7f0d340ffaf0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c000f80 tx=0x7f0d2c003ce0 comp rx=0 tx=0).stop 2024-04-22T23:45:07.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.214+0000 7f0d399a2700 1 -- 172.21.15.67:0/2551113727 shutdown_connections 2024-04-22T23:45:07.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.214+0000 7f0d399a2700 1 --2- 172.21.15.67:0/2551113727 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0d1c06ecc0 0x7f0d1c071180 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:07.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.214+0000 7f0d399a2700 1 --2- 172.21.15.67:0/2551113727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d34068580 0x7f0d340ffaf0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:07.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.214+0000 7f0d399a2700 1 --2- 172.21.15.67:0/2551113727 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0d34106a50 0x7f0d34100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:07.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.214+0000 7f0d399a2700 1 -- 172.21.15.67:0/2551113727 >> 172.21.15.67:0/2551113727 conn(0x7f0d34075460 msgr2=0x7f0d340feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:45:07.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.214+0000 7f0d399a2700 1 -- 172.21.15.67:0/2551113727 shutdown_connections 2024-04-22T23:45:07.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.214+0000 7f0d399a2700 1 -- 172.21.15.67:0/2551113727 wait complete. 2024-04-22T23:45:07.216 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:45:07.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.322+0000 7f30cb8eb700 1 -- 172.21.15.67:0/121359025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30c4068580 msgr2=0x7f30c40689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:07.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.322+0000 7f30cb8eb700 1 --2- 172.21.15.67:0/121359025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30c4068580 0x7f30c40689e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f30c0009910 tx=0x7f30c0009c20 comp rx=0 tx=0).stop 2024-04-22T23:45:07.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.322+0000 7f30cb8eb700 1 -- 172.21.15.67:0/121359025 shutdown_connections 2024-04-22T23:45:07.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.322+0000 7f30cb8eb700 1 --2- 172.21.15.67:0/121359025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30c4068580 0x7f30c40689e0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:07.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.322+0000 7f30cb8eb700 1 --2- 172.21.15.67:0/121359025 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f30c4106cd0 0x7f30c41070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:07.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.322+0000 7f30cb8eb700 1 -- 172.21.15.67:0/121359025 >> 172.21.15.67:0/121359025 conn(0x7f30c4075670 msgr2=0x7f30c4075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:45:07.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.322+0000 7f30cb8eb700 1 -- 172.21.15.67:0/121359025 shutdown_connections 2024-04-22T23:45:07.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.322+0000 7f30cb8eb700 1 -- 172.21.15.67:0/121359025 wait complete. 2024-04-22T23:45:07.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.323+0000 7f30cb8eb700 1 Processor -- start 2024-04-22T23:45:07.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.323+0000 7f30cb8eb700 1 -- start start 2024-04-22T23:45:07.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.323+0000 7f30cb8eb700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f30c4068580 0x7f30c40ffcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:07.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.323+0000 7f30cb8eb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30c4106cd0 0x7f30c4100230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:07.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.323+0000 7f30cb8eb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c4100770 con 0x7f30c4106cd0 2024-04-22T23:45:07.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.323+0000 7f30cb8eb700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c41008b0 con 0x7f30c4068580 2024-04-22T23:45:07.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.323+0000 7f30c8e86700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30c4106cd0 0x7f30c4100230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:07.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.323+0000 7f30c8e86700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30c4106cd0 0x7f30c4100230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36980/0 (socket says 172.21.15.67:36980) 2024-04-22T23:45:07.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.323+0000 7f30c8e86700 1 -- 172.21.15.67:0/1401808953 learned_addr learned my addr 172.21.15.67:0/1401808953 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:45:07.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.323+0000 7f30c9687700 1 --2- 172.21.15.67:0/1401808953 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f30c4068580 0x7f30c40ffcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:07.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.323+0000 7f30c8e86700 1 -- 172.21.15.67:0/1401808953 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f30c4068580 msgr2=0x7f30c40ffcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:07.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.323+0000 7f30c8e86700 1 --2- 172.21.15.67:0/1401808953 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f30c4068580 0x7f30c40ffcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:07.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.324+0000 7f30c8e86700 1 -- 172.21.15.67:0/1401808953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c00095f0 con 0x7f30c4106cd0 2024-04-22T23:45:07.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.324+0000 7f30c8e86700 1 --2- 172.21.15.67:0/1401808953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30c4106cd0 0x7f30c4100230 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f30c0006010 tx=0x7f30c000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:45:07.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.324+0000 7f30b67fc700 1 -- 172.21.15.67:0/1401808953 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30c001d070 con 0x7f30c4106cd0 2024-04-22T23:45:07.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.324+0000 7f30b67fc700 1 -- 172.21.15.67:0/1401808953 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f30c0004fb0 con 0x7f30c4106cd0 2024-04-22T23:45:07.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.324+0000 7f30cb8eb700 1 -- 172.21.15.67:0/1401808953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30c4100b30 con 0x7f30c4106cd0 2024-04-22T23:45:07.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.324+0000 7f30b67fc700 1 -- 172.21.15.67:0/1401808953 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30c0022ea0 con 0x7f30c4106cd0 2024-04-22T23:45:07.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.324+0000 7f30cb8eb700 1 -- 172.21.15.67:0/1401808953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30c41a2e80 con 0x7f30c4106cd0 2024-04-22T23:45:07.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.325+0000 7f30cb8eb700 1 -- 172.21.15.67:0/1401808953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30c41091f0 con 0x7f30c4106cd0 2024-04-22T23:45:07.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.325+0000 7f30b67fc700 1 -- 172.21.15.67:0/1401808953 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f30c0022470 con 0x7f30c4106cd0 2024-04-22T23:45:07.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.326+0000 7f30b67fc700 1 --2- 172.21.15.67:0/1401808953 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f30b006ecc0 0x7f30b0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:07.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.326+0000 7f30b67fc700 1 -- 172.21.15.67:0/1401808953 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f30c0090490 con 0x7f30c4106cd0 2024-04-22T23:45:07.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.326+0000 7f30c9687700 1 --2- 172.21.15.67:0/1401808953 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f30b006ecc0 0x7f30b0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:07.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.326+0000 7f30c9687700 1 --2- 172.21.15.67:0/1401808953 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f30b006ecc0 0x7f30b0071180 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f30b8006fd0 tx=0x7f30b8008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:45:07.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.331+0000 7f30b67fc700 1 -- 172.21.15.67:0/1401808953 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f30c0059200 con 0x7f30c4106cd0 2024-04-22T23:45:07.536 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:07 smithi067 ceph-mon[112438]: pgmap v562: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:07.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.563+0000 7f30cb8eb700 1 -- 172.21.15.67:0/1401808953 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f30c40611d0 con 0x7f30b006ecc0 2024-04-22T23:45:07.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.564+0000 7f30b67fc700 1 -- 172.21.15.67:0/1401808953 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f30c40611d0 con 0x7f30b006ecc0 2024-04-22T23:45:07.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.566+0000 7f30cb8eb700 1 -- 172.21.15.67:0/1401808953 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f30b006ecc0 msgr2=0x7f30b0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:07.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.566+0000 7f30cb8eb700 1 --2- 172.21.15.67:0/1401808953 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f30b006ecc0 0x7f30b0071180 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f30b8006fd0 tx=0x7f30b8008040 comp rx=0 tx=0).stop 2024-04-22T23:45:07.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.566+0000 7f30cb8eb700 1 -- 172.21.15.67:0/1401808953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30c4106cd0 msgr2=0x7f30c4100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:07.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.566+0000 7f30cb8eb700 1 --2- 172.21.15.67:0/1401808953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30c4106cd0 0x7f30c4100230 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f30c0006010 tx=0x7f30c000be20 comp rx=0 tx=0).stop 2024-04-22T23:45:07.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.566+0000 7f30cb8eb700 1 -- 172.21.15.67:0/1401808953 shutdown_connections 2024-04-22T23:45:07.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.566+0000 7f30cb8eb700 1 --2- 172.21.15.67:0/1401808953 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f30b006ecc0 0x7f30b0071180 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:07.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.566+0000 7f30cb8eb700 1 --2- 172.21.15.67:0/1401808953 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f30c4068580 0x7f30c40ffcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:07.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.566+0000 7f30cb8eb700 1 --2- 172.21.15.67:0/1401808953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30c4106cd0 0x7f30c4100230 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:07.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.566+0000 7f30cb8eb700 1 -- 172.21.15.67:0/1401808953 >> 172.21.15.67:0/1401808953 conn(0x7f30c4075670 msgr2=0x7f30c40fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:45:07.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.567+0000 7f30cb8eb700 1 -- 172.21.15.67:0/1401808953 shutdown_connections 2024-04-22T23:45:07.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.567+0000 7f30cb8eb700 1 -- 172.21.15.67:0/1401808953 wait complete. 2024-04-22T23:45:07.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.674+0000 7f50847a4700 1 -- 172.21.15.67:0/1161626278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f507c1027c0 msgr2=0x7f507c10acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:07.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.674+0000 7f50847a4700 1 --2- 172.21.15.67:0/1161626278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f507c1027c0 0x7f507c10acb0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f506c009910 tx=0x7f506c009c20 comp rx=0 tx=0).stop 2024-04-22T23:45:07.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.674+0000 7f50847a4700 1 -- 172.21.15.67:0/1161626278 shutdown_connections 2024-04-22T23:45:07.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.674+0000 7f50847a4700 1 --2- 172.21.15.67:0/1161626278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f507c1027c0 0x7f507c10acb0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:07.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.674+0000 7f50847a4700 1 --2- 172.21.15.67:0/1161626278 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f507c101ea0 0x7f507c102280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:07.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.674+0000 7f50847a4700 1 -- 172.21.15.67:0/1161626278 >> 172.21.15.67:0/1161626278 conn(0x7f507c0fb650 msgr2=0x7f507c0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:45:07.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.674+0000 7f50847a4700 1 -- 172.21.15.67:0/1161626278 shutdown_connections 2024-04-22T23:45:07.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.674+0000 7f50847a4700 1 -- 172.21.15.67:0/1161626278 wait complete. 2024-04-22T23:45:07.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.674+0000 7f50847a4700 1 Processor -- start 2024-04-22T23:45:07.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.674+0000 7f50847a4700 1 -- start start 2024-04-22T23:45:07.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.675+0000 7f50847a4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f507c101ea0 0x7f507c0ffb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:07.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.675+0000 7f50847a4700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f507c1027c0 0x7f507c100040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:07.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.675+0000 7f50847a4700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507c100580 con 0x7f507c101ea0 2024-04-22T23:45:07.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.675+0000 7f50847a4700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507c1006c0 con 0x7f507c1027c0 2024-04-22T23:45:07.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.675+0000 7f5082540700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f507c101ea0 0x7f507c0ffb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:07.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.675+0000 7f5082540700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f507c101ea0 0x7f507c0ffb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37002/0 (socket says 172.21.15.67:37002) 2024-04-22T23:45:07.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.675+0000 7f5082540700 1 -- 172.21.15.67:0/2749596464 learned_addr learned my addr 172.21.15.67:0/2749596464 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:45:07.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.675+0000 7f5081d3f700 1 --2- 172.21.15.67:0/2749596464 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f507c1027c0 0x7f507c100040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:07.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.675+0000 7f5082540700 1 -- 172.21.15.67:0/2749596464 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f507c1027c0 msgr2=0x7f507c100040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:07.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.675+0000 7f5082540700 1 --2- 172.21.15.67:0/2749596464 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f507c1027c0 0x7f507c100040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:07.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.675+0000 7f5082540700 1 -- 172.21.15.67:0/2749596464 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f506c0095f0 con 0x7f507c101ea0 2024-04-22T23:45:07.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.675+0000 7f5082540700 1 --2- 172.21.15.67:0/2749596464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f507c101ea0 0x7f507c0ffb00 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f507800d8d0 tx=0x7f507800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:45:07.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.675+0000 7f50737fe700 1 -- 172.21.15.67:0/2749596464 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5078009940 con 0x7f507c101ea0 2024-04-22T23:45:07.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.675+0000 7f50737fe700 1 -- 172.21.15.67:0/2749596464 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5078010460 con 0x7f507c101ea0 2024-04-22T23:45:07.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.675+0000 7f50847a4700 1 -- 172.21.15.67:0/2749596464 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f507c1a2a60 con 0x7f507c101ea0 2024-04-22T23:45:07.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.675+0000 7f50737fe700 1 -- 172.21.15.67:0/2749596464 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f507800f5f0 con 0x7f507c101ea0 2024-04-22T23:45:07.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.675+0000 7f50847a4700 1 -- 172.21.15.67:0/2749596464 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f507c1a2e70 con 0x7f507c101ea0 2024-04-22T23:45:07.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.676+0000 7f50847a4700 1 -- 172.21.15.67:0/2749596464 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f507c108340 con 0x7f507c101ea0 2024-04-22T23:45:07.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.677+0000 7f50737fe700 1 -- 172.21.15.67:0/2749596464 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f50780105d0 con 0x7f507c101ea0 2024-04-22T23:45:07.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.677+0000 7f50737fe700 1 --2- 172.21.15.67:0/2749596464 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f506806ecc0 0x7f5068071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:07.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.678+0000 7f5081d3f700 1 --2- 172.21.15.67:0/2749596464 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f506806ecc0 0x7f5068071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:07.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.678+0000 7f50737fe700 1 -- 172.21.15.67:0/2749596464 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f507808eef0 con 0x7f507c101ea0 2024-04-22T23:45:07.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.678+0000 7f5081d3f700 1 --2- 172.21.15.67:0/2749596464 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f506806ecc0 0x7f5068071180 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f506c0059d0 tx=0x7f506c005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:45:07.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.682+0000 7f50737fe700 1 -- 172.21.15.67:0/2749596464 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f5078057bb0 con 0x7f507c101ea0 2024-04-22T23:45:07.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:07 smithi134 ceph-mon[85630]: pgmap v562: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:07.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.903+0000 7f50847a4700 1 -- 172.21.15.67:0/2749596464 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f507c0611d0 con 0x7f506806ecc0 2024-04-22T23:45:07.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.916+0000 7f50737fe700 1 -- 172.21.15.67:0/2749596464 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f507c0611d0 con 0x7f506806ecc0 2024-04-22T23:45:07.909 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:45:07.909 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (18m) 12m ago 30m 25.5M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:45:07.909 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (30m) 12m ago 30m 9114k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:45:07.909 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (29m) 11m ago 29m 9042k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:45:07.909 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (16m) 12m ago 30m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:45:07.909 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (16m) 11m ago 29m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:45:07.909 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (18m) 12m ago 29m 79.3M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:45:07.909 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (12m) 12m ago 24m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:45:07.909 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (12m) 12m ago 24m 12.6M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:45:07.909 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (11m) 11m ago 24m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:45:07.909 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (24m) 11m ago 24m 12.2M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:45:07.909 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (20m) 12m ago 31m 511M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:45:07.909 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (19m) 11m ago 29m 453M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:45:07.910 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (17m) 12m ago 31m 55.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:45:07.910 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (16m) 11m ago 29m 46.0M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:45:07.910 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (19m) 12m ago 30m 15.4M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:45:07.910 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (19m) 11m ago 29m 16.1M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:45:07.910 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (15m) 12m ago 28m 99.2M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:45:07.910 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (15m) 12m ago 27m 127M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:45:07.910 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (14m) 12m ago 27m 79.9M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:45:07.910 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (13m) 11m ago 27m 97.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:45:07.910 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (13m) 11m ago 26m 82.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:45:07.910 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (12m) 11m ago 26m 77.6M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:45:07.910 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (19m) 12m ago 29m 73.9M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:45:07.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.918+0000 7f50847a4700 1 -- 172.21.15.67:0/2749596464 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f506806ecc0 msgr2=0x7f5068071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:07.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.918+0000 7f50847a4700 1 --2- 172.21.15.67:0/2749596464 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f506806ecc0 0x7f5068071180 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f506c0059d0 tx=0x7f506c005960 comp rx=0 tx=0).stop 2024-04-22T23:45:07.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.918+0000 7f50847a4700 1 -- 172.21.15.67:0/2749596464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f507c101ea0 msgr2=0x7f507c0ffb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:07.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.918+0000 7f50847a4700 1 --2- 172.21.15.67:0/2749596464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f507c101ea0 0x7f507c0ffb00 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f507800d8d0 tx=0x7f507800dc90 comp rx=0 tx=0).stop 2024-04-22T23:45:07.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.918+0000 7f50847a4700 1 -- 172.21.15.67:0/2749596464 shutdown_connections 2024-04-22T23:45:07.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.918+0000 7f50847a4700 1 --2- 172.21.15.67:0/2749596464 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f506806ecc0 0x7f5068071180 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:07.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.918+0000 7f50847a4700 1 --2- 172.21.15.67:0/2749596464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f507c101ea0 0x7f507c0ffb00 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:07.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.918+0000 7f50847a4700 1 --2- 172.21.15.67:0/2749596464 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f507c1027c0 0x7f507c100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:07.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.918+0000 7f50847a4700 1 -- 172.21.15.67:0/2749596464 >> 172.21.15.67:0/2749596464 conn(0x7f507c0fb650 msgr2=0x7f507c105480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:45:07.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.919+0000 7f50847a4700 1 -- 172.21.15.67:0/2749596464 shutdown_connections 2024-04-22T23:45:07.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:07.919+0000 7f50847a4700 1 -- 172.21.15.67:0/2749596464 wait complete. 2024-04-22T23:45:08.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.025+0000 7f6ef69d9700 1 -- 172.21.15.67:0/3994413919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ef00ffda0 msgr2=0x7f6ef0100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:08.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.025+0000 7f6ef69d9700 1 --2- 172.21.15.67:0/3994413919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ef00ffda0 0x7f6ef0100180 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f6ed8009910 tx=0x7f6ed8009c20 comp rx=0 tx=0).stop 2024-04-22T23:45:08.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.025+0000 7f6ef69d9700 1 -- 172.21.15.67:0/3994413919 shutdown_connections 2024-04-22T23:45:08.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.025+0000 7f6ef69d9700 1 --2- 172.21.15.67:0/3994413919 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ef0100750 0x7f6ef01047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:08.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.025+0000 7f6ef69d9700 1 --2- 172.21.15.67:0/3994413919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ef00ffda0 0x7f6ef0100180 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:08.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.025+0000 7f6ef69d9700 1 -- 172.21.15.67:0/3994413919 >> 172.21.15.67:0/3994413919 conn(0x7f6ef00fb650 msgr2=0x7f6ef00fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:45:08.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.025+0000 7f6ef69d9700 1 -- 172.21.15.67:0/3994413919 shutdown_connections 2024-04-22T23:45:08.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.025+0000 7f6ef69d9700 1 -- 172.21.15.67:0/3994413919 wait complete. 2024-04-22T23:45:08.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.026+0000 7f6ef69d9700 1 Processor -- start 2024-04-22T23:45:08.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.026+0000 7f6ef69d9700 1 -- start start 2024-04-22T23:45:08.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.026+0000 7f6ef69d9700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ef00ffda0 0x7f6ef0101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:08.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.026+0000 7f6ef69d9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ef0100750 0x7f6ef0101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:08.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.026+0000 7f6ef69d9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef01022c0 con 0x7f6ef0100750 2024-04-22T23:45:08.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.026+0000 7f6ef69d9700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef0102400 con 0x7f6ef00ffda0 2024-04-22T23:45:08.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.026+0000 7f6eef7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ef0100750 0x7f6ef0101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:08.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.026+0000 7f6eef7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ef0100750 0x7f6ef0101d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37026/0 (socket says 172.21.15.67:37026) 2024-04-22T23:45:08.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.026+0000 7f6eef7fe700 1 -- 172.21.15.67:0/480819586 learned_addr learned my addr 172.21.15.67:0/480819586 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:45:08.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.026+0000 7f6eeffff700 1 --2- 172.21.15.67:0/480819586 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ef00ffda0 0x7f6ef0101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:08.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.026+0000 7f6eef7fe700 1 -- 172.21.15.67:0/480819586 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ef00ffda0 msgr2=0x7f6ef0101840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:08.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.026+0000 7f6eef7fe700 1 --2- 172.21.15.67:0/480819586 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ef00ffda0 0x7f6ef0101840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:08.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.026+0000 7f6eef7fe700 1 -- 172.21.15.67:0/480819586 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed80095f0 con 0x7f6ef0100750 2024-04-22T23:45:08.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.027+0000 7f6eef7fe700 1 --2- 172.21.15.67:0/480819586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ef0100750 0x7f6ef0101d80 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f6ee000d900 tx=0x7f6ee000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:45:08.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.027+0000 7f6eed7fa700 1 -- 172.21.15.67:0/480819586 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ee0009900 con 0x7f6ef0100750 2024-04-22T23:45:08.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.027+0000 7f6eed7fa700 1 -- 172.21.15.67:0/480819586 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6ee0010460 con 0x7f6ef0100750 2024-04-22T23:45:08.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.027+0000 7f6ef69d9700 1 -- 172.21.15.67:0/480819586 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ef01a2ae0 con 0x7f6ef0100750 2024-04-22T23:45:08.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.027+0000 7f6eed7fa700 1 -- 172.21.15.67:0/480819586 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ee000f5f0 con 0x7f6ef0100750 2024-04-22T23:45:08.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.027+0000 7f6ef69d9700 1 -- 172.21.15.67:0/480819586 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ef01a2ef0 con 0x7f6ef0100750 2024-04-22T23:45:08.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.028+0000 7f6ef69d9700 1 -- 172.21.15.67:0/480819586 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ef004f2e0 con 0x7f6ef0100750 2024-04-22T23:45:08.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.028+0000 7f6eed7fa700 1 -- 172.21.15.67:0/480819586 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6ee000f870 con 0x7f6ef0100750 2024-04-22T23:45:08.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.029+0000 7f6eed7fa700 1 --2- 172.21.15.67:0/480819586 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6edc06ecc0 0x7f6edc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:08.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.029+0000 7f6eed7fa700 1 -- 172.21.15.67:0/480819586 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f6ee008ec50 con 0x7f6ef0100750 2024-04-22T23:45:08.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.029+0000 7f6eeffff700 1 --2- 172.21.15.67:0/480819586 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6edc06ecc0 0x7f6edc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:08.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.029+0000 7f6eeffff700 1 --2- 172.21.15.67:0/480819586 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6edc06ecc0 0x7f6edc071180 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6ed800b5c0 tx=0x7f6ed801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:45:08.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.034+0000 7f6eed7fa700 1 -- 172.21.15.67:0/480819586 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f6ee0057900 con 0x7f6ef0100750 2024-04-22T23:45:08.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.340+0000 7f6ef69d9700 1 -- 172.21.15.67:0/480819586 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6ef01a31a0 con 0x7f6ef0100750 2024-04-22T23:45:08.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.340+0000 7f6eed7fa700 1 -- 172.21.15.67:0/480819586 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f6ee005afc0 con 0x7f6ef0100750 2024-04-22T23:45:08.332 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:45:08.333 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:45:08.333 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:45:08.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:45:08.333 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:45:08.333 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:45:08.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:45:08.333 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:45:08.333 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:45:08.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:45:08.333 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:45:08.333 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:45:08.333 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 3 2024-04-22T23:45:08.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:45:08.333 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:45:08.334 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:45:08.334 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 13 2024-04-22T23:45:08.334 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:45:08.334 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:45:08.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.343+0000 7f6ef69d9700 1 -- 172.21.15.67:0/480819586 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6edc06ecc0 msgr2=0x7f6edc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:08.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.343+0000 7f6ef69d9700 1 --2- 172.21.15.67:0/480819586 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6edc06ecc0 0x7f6edc071180 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6ed800b5c0 tx=0x7f6ed801a040 comp rx=0 tx=0).stop 2024-04-22T23:45:08.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.343+0000 7f6ef69d9700 1 -- 172.21.15.67:0/480819586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ef0100750 msgr2=0x7f6ef0101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:08.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.343+0000 7f6ef69d9700 1 --2- 172.21.15.67:0/480819586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ef0100750 0x7f6ef0101d80 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f6ee000d900 tx=0x7f6ee000dcc0 comp rx=0 tx=0).stop 2024-04-22T23:45:08.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.343+0000 7f6ef69d9700 1 -- 172.21.15.67:0/480819586 shutdown_connections 2024-04-22T23:45:08.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.343+0000 7f6ef69d9700 1 --2- 172.21.15.67:0/480819586 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6edc06ecc0 0x7f6edc071180 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:08.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.343+0000 7f6ef69d9700 1 --2- 172.21.15.67:0/480819586 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ef00ffda0 0x7f6ef0101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:08.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.343+0000 7f6ef69d9700 1 --2- 172.21.15.67:0/480819586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ef0100750 0x7f6ef0101d80 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:08.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.343+0000 7f6ef69d9700 1 -- 172.21.15.67:0/480819586 >> 172.21.15.67:0/480819586 conn(0x7f6ef00fb650 msgr2=0x7f6ef0103f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:45:08.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.343+0000 7f6ef69d9700 1 -- 172.21.15.67:0/480819586 shutdown_connections 2024-04-22T23:45:08.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.343+0000 7f6ef69d9700 1 -- 172.21.15.67:0/480819586 wait complete. 2024-04-22T23:45:08.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.450+0000 7fdd03d04700 1 -- 172.21.15.67:0/4281989223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcfc106cd0 msgr2=0x7fdcfc1070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:08.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.450+0000 7fdd03d04700 1 --2- 172.21.15.67:0/4281989223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcfc106cd0 0x7fdcfc1070b0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fdcf8009910 tx=0x7fdcf8009c20 comp rx=0 tx=0).stop 2024-04-22T23:45:08.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.450+0000 7fdd03d04700 1 -- 172.21.15.67:0/4281989223 shutdown_connections 2024-04-22T23:45:08.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.450+0000 7fdd03d04700 1 --2- 172.21.15.67:0/4281989223 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdcfc068580 0x7fdcfc0689e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:08.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.450+0000 7fdd03d04700 1 --2- 172.21.15.67:0/4281989223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcfc106cd0 0x7fdcfc1070b0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:08.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.450+0000 7fdd03d04700 1 -- 172.21.15.67:0/4281989223 >> 172.21.15.67:0/4281989223 conn(0x7fdcfc075670 msgr2=0x7fdcfc075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:45:08.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.450+0000 7fdd03d04700 1 -- 172.21.15.67:0/4281989223 shutdown_connections 2024-04-22T23:45:08.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.450+0000 7fdd03d04700 1 -- 172.21.15.67:0/4281989223 wait complete. 2024-04-22T23:45:08.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.451+0000 7fdd03d04700 1 Processor -- start 2024-04-22T23:45:08.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.451+0000 7fdd03d04700 1 -- start start 2024-04-22T23:45:08.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.451+0000 7fdd03d04700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcfc068580 0x7fdcfc196980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:08.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.451+0000 7fdd03d04700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdcfc106cd0 0x7fdcfc196ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:08.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.451+0000 7fdd03d04700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcfc197550 con 0x7fdcfc068580 2024-04-22T23:45:08.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.451+0000 7fdd03d04700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcfc19b360 con 0x7fdcfc106cd0 2024-04-22T23:45:08.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.451+0000 7fdd01aa0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcfc068580 0x7fdcfc196980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:08.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.451+0000 7fdd01aa0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcfc068580 0x7fdcfc196980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37034/0 (socket says 172.21.15.67:37034) 2024-04-22T23:45:08.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.451+0000 7fdd01aa0700 1 -- 172.21.15.67:0/3083211865 learned_addr learned my addr 172.21.15.67:0/3083211865 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:45:08.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.451+0000 7fdd0129f700 1 --2- 172.21.15.67:0/3083211865 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdcfc106cd0 0x7fdcfc196ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:08.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.451+0000 7fdd01aa0700 1 -- 172.21.15.67:0/3083211865 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdcfc106cd0 msgr2=0x7fdcfc196ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:08.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.451+0000 7fdd01aa0700 1 --2- 172.21.15.67:0/3083211865 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdcfc106cd0 0x7fdcfc196ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:08.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.451+0000 7fdd01aa0700 1 -- 172.21.15.67:0/3083211865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcf80095f0 con 0x7fdcfc068580 2024-04-22T23:45:08.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.452+0000 7fdd01aa0700 1 --2- 172.21.15.67:0/3083211865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcfc068580 0x7fdcfc196980 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fdcf8006010 tx=0x7fdcf800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:45:08.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.452+0000 7fdcf2ffd700 1 -- 172.21.15.67:0/3083211865 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdcf801d070 con 0x7fdcfc068580 2024-04-22T23:45:08.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.452+0000 7fdcf2ffd700 1 -- 172.21.15.67:0/3083211865 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdcf8004fb0 con 0x7fdcfc068580 2024-04-22T23:45:08.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.452+0000 7fdd03d04700 1 -- 172.21.15.67:0/3083211865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcfc19b5e0 con 0x7fdcfc068580 2024-04-22T23:45:08.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.452+0000 7fdcf2ffd700 1 -- 172.21.15.67:0/3083211865 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdcf8022ea0 con 0x7fdcfc068580 2024-04-22T23:45:08.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.452+0000 7fdd03d04700 1 -- 172.21.15.67:0/3083211865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcfc19bad0 con 0x7fdcfc068580 2024-04-22T23:45:08.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.453+0000 7fdd03d04700 1 -- 172.21.15.67:0/3083211865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcfc1091f0 con 0x7fdcfc068580 2024-04-22T23:45:08.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.453+0000 7fdcf2ffd700 1 -- 172.21.15.67:0/3083211865 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdcf8022470 con 0x7fdcfc068580 2024-04-22T23:45:08.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.454+0000 7fdcf2ffd700 1 --2- 172.21.15.67:0/3083211865 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdce806ecc0 0x7fdce8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:08.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.454+0000 7fdd0129f700 1 --2- 172.21.15.67:0/3083211865 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdce806ecc0 0x7fdce8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:08.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.454+0000 7fdcf2ffd700 1 -- 172.21.15.67:0/3083211865 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fdcf8090560 con 0x7fdcfc068580 2024-04-22T23:45:08.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.455+0000 7fdd0129f700 1 --2- 172.21.15.67:0/3083211865 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdce806ecc0 0x7fdce8071180 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fdcec006fd0 tx=0x7fdcec008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:45:08.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.459+0000 7fdcf2ffd700 1 -- 172.21.15.67:0/3083211865 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fdcf805cd80 con 0x7fdcfc068580 2024-04-22T23:45:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.712+0000 7fdd03d04700 1 -- 172.21.15.67:0/3083211865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fdcfc197ce0 con 0x7fdcfc068580 2024-04-22T23:45:08.705 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:08 smithi067 ceph-mon[112438]: from='client.35044 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:45:08.705 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:08 smithi067 ceph-mon[112438]: from='client.35048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:45:08.705 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:08 smithi067 ceph-mon[112438]: from='client.35052 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:45:08.705 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:08 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/480819586' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:45:08.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.713+0000 7fdcf2ffd700 1 -- 172.21.15.67:0/3083211865 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1871 (secure 0 0 0) 0x7fdcf8027760 con 0x7fdcfc068580 2024-04-22T23:45:08.706 INFO:teuthology.orchestra.run.smithi067.stdout:e31 2024-04-22T23:45:08.706 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:45:08.706 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:45:08.706 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:45:08.706 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:45:08.706 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:45:08.706 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:45:08.706 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:45:08.706 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:45:08.706 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:45:08.707 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:45:08.707 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:45:08.707 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:45:08.707 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:45:08.707 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:45:08.707 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:45:08.707 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:45:08.707 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:45:08.707 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:45:08.707 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:45:08.707 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:45:08.707 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:45:08.707 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:45:08.707 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:45:08.707 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:45:08.708 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:45:08.708 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:45:08.708 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:45:08.708 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:45:08.708 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:45:08.708 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:45:08.708 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:45:08.708 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:45:08.708 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:45:08.708 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:45:08.708 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:45:08.708 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:45:08.708 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:45:08.708 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:45:08.708 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:45:08.708 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44259} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1827288036,v1:172.21.15.134:6827/1827288036] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:45:08.709 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:45:08.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.715+0000 7fdd03d04700 1 -- 172.21.15.67:0/3083211865 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdce806ecc0 msgr2=0x7fdce8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:08.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.715+0000 7fdd03d04700 1 --2- 172.21.15.67:0/3083211865 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdce806ecc0 0x7fdce8071180 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fdcec006fd0 tx=0x7fdcec008040 comp rx=0 tx=0).stop 2024-04-22T23:45:08.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.715+0000 7fdd03d04700 1 -- 172.21.15.67:0/3083211865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcfc068580 msgr2=0x7fdcfc196980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:08.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.715+0000 7fdd03d04700 1 --2- 172.21.15.67:0/3083211865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcfc068580 0x7fdcfc196980 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fdcf8006010 tx=0x7fdcf800be20 comp rx=0 tx=0).stop 2024-04-22T23:45:08.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.715+0000 7fdd03d04700 1 -- 172.21.15.67:0/3083211865 shutdown_connections 2024-04-22T23:45:08.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.715+0000 7fdd03d04700 1 --2- 172.21.15.67:0/3083211865 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdce806ecc0 0x7fdce8071180 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:08.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.715+0000 7fdd03d04700 1 --2- 172.21.15.67:0/3083211865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcfc068580 0x7fdcfc196980 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:08.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.715+0000 7fdd03d04700 1 --2- 172.21.15.67:0/3083211865 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdcfc106cd0 0x7fdcfc196ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:08.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.715+0000 7fdd03d04700 1 -- 172.21.15.67:0/3083211865 >> 172.21.15.67:0/3083211865 conn(0x7fdcfc075670 msgr2=0x7fdcfc0fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:45:08.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.715+0000 7fdd03d04700 1 -- 172.21.15.67:0/3083211865 shutdown_connections 2024-04-22T23:45:08.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.715+0000 7fdd03d04700 1 -- 172.21.15.67:0/3083211865 wait complete. 2024-04-22T23:45:08.711 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 31 2024-04-22T23:45:08.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:08 smithi134 ceph-mon[85630]: from='client.35044 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:45:08.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:08 smithi134 ceph-mon[85630]: from='client.35048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:45:08.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:08 smithi134 ceph-mon[85630]: from='client.35052 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:45:08.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:08 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/480819586' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:45:08.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.824+0000 7f5b5e11d700 1 -- 172.21.15.67:0/3632339368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b58105bd0 msgr2=0x7f5b58105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:08.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.824+0000 7f5b5e11d700 1 --2- 172.21.15.67:0/3632339368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b58105bd0 0x7f5b58105fb0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f5b50009910 tx=0x7f5b50009c20 comp rx=0 tx=0).stop 2024-04-22T23:45:08.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.824+0000 7f5b5e11d700 1 -- 172.21.15.67:0/3632339368 shutdown_connections 2024-04-22T23:45:08.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.824+0000 7f5b5e11d700 1 --2- 172.21.15.67:0/3632339368 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b580ffc20 0x7f5b58100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:08.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.824+0000 7f5b5e11d700 1 --2- 172.21.15.67:0/3632339368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b58105bd0 0x7f5b58105fb0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:08.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.824+0000 7f5b5e11d700 1 -- 172.21.15.67:0/3632339368 >> 172.21.15.67:0/3632339368 conn(0x7f5b58075670 msgr2=0x7f5b58075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:45:08.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.824+0000 7f5b5e11d700 1 -- 172.21.15.67:0/3632339368 shutdown_connections 2024-04-22T23:45:08.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.824+0000 7f5b5e11d700 1 -- 172.21.15.67:0/3632339368 wait complete. 2024-04-22T23:45:08.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.824+0000 7f5b5e11d700 1 Processor -- start 2024-04-22T23:45:08.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.825+0000 7f5b5e11d700 1 -- start start 2024-04-22T23:45:08.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.825+0000 7f5b5e11d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b580ffc20 0x7f5b5819b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:08.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.825+0000 7f5b5e11d700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b58105bd0 0x7f5b5819bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:08.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.825+0000 7f5b5e11d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b58195590 con 0x7f5b580ffc20 2024-04-22T23:45:08.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.825+0000 7f5b5e11d700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b58195700 con 0x7f5b58105bd0 2024-04-22T23:45:08.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.825+0000 7f5b577fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b580ffc20 0x7f5b5819b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:08.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.825+0000 7f5b577fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b580ffc20 0x7f5b5819b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40610/0 (socket says 172.21.15.67:40610) 2024-04-22T23:45:08.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.825+0000 7f5b577fe700 1 -- 172.21.15.67:0/2603882767 learned_addr learned my addr 172.21.15.67:0/2603882767 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:45:08.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.825+0000 7f5b56ffd700 1 --2- 172.21.15.67:0/2603882767 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b58105bd0 0x7f5b5819bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:08.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.825+0000 7f5b577fe700 1 -- 172.21.15.67:0/2603882767 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b58105bd0 msgr2=0x7f5b5819bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:08.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.825+0000 7f5b577fe700 1 --2- 172.21.15.67:0/2603882767 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b58105bd0 0x7f5b5819bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:08.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.825+0000 7f5b577fe700 1 -- 172.21.15.67:0/2603882767 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b500095f0 con 0x7f5b580ffc20 2024-04-22T23:45:08.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.825+0000 7f5b577fe700 1 --2- 172.21.15.67:0/2603882767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b580ffc20 0x7f5b5819b5a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f5b50006010 tx=0x7f5b5000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:45:08.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.826+0000 7f5b54ff9700 1 -- 172.21.15.67:0/2603882767 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b5001d070 con 0x7f5b580ffc20 2024-04-22T23:45:08.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.826+0000 7f5b54ff9700 1 -- 172.21.15.67:0/2603882767 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5b50005dc0 con 0x7f5b580ffc20 2024-04-22T23:45:08.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.826+0000 7f5b5e11d700 1 -- 172.21.15.67:0/2603882767 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b58195920 con 0x7f5b580ffc20 2024-04-22T23:45:08.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.826+0000 7f5b54ff9700 1 -- 172.21.15.67:0/2603882767 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b50017400 con 0x7f5b580ffc20 2024-04-22T23:45:08.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.826+0000 7f5b5e11d700 1 -- 172.21.15.67:0/2603882767 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b58195e70 con 0x7f5b580ffc20 2024-04-22T23:45:08.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.827+0000 7f5b5e11d700 1 -- 172.21.15.67:0/2603882767 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b5804f2e0 con 0x7f5b580ffc20 2024-04-22T23:45:08.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.827+0000 7f5b54ff9700 1 -- 172.21.15.67:0/2603882767 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5b50003a60 con 0x7f5b580ffc20 2024-04-22T23:45:08.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.828+0000 7f5b54ff9700 1 --2- 172.21.15.67:0/2603882767 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5b4406ecc0 0x7f5b44071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:08.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.828+0000 7f5b54ff9700 1 -- 172.21.15.67:0/2603882767 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f5b50090370 con 0x7f5b580ffc20 2024-04-22T23:45:08.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.828+0000 7f5b56ffd700 1 --2- 172.21.15.67:0/2603882767 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5b4406ecc0 0x7f5b44071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:08.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.828+0000 7f5b56ffd700 1 --2- 172.21.15.67:0/2603882767 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5b4406ecc0 0x7f5b44071180 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5b58196e30 tx=0x7f5b4000b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:45:08.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:08.833+0000 7f5b54ff9700 1 -- 172.21.15.67:0/2603882767 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f5b50059030 con 0x7f5b580ffc20 2024-04-22T23:45:09.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.062+0000 7f5b5e11d700 1 -- 172.21.15.67:0/2603882767 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b581960e0 con 0x7f5b4406ecc0 2024-04-22T23:45:09.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.064+0000 7f5b54ff9700 1 -- 172.21.15.67:0/2603882767 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f5b581960e0 con 0x7f5b4406ecc0 2024-04-22T23:45:09.056 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:45:09.056 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:45:09.056 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:45:09.056 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:45:09.056 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:45:09.056 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:45:09.056 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:45:09.056 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:45:09.056 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:45:09.056 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:45:09.056 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T23:45:09.056 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:45:09.057 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:45:09.057 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:45:09.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.066+0000 7f5b5e11d700 1 -- 172.21.15.67:0/2603882767 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5b4406ecc0 msgr2=0x7f5b44071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:09.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.066+0000 7f5b5e11d700 1 --2- 172.21.15.67:0/2603882767 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5b4406ecc0 0x7f5b44071180 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5b58196e30 tx=0x7f5b4000b410 comp rx=0 tx=0).stop 2024-04-22T23:45:09.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.066+0000 7f5b5e11d700 1 -- 172.21.15.67:0/2603882767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b580ffc20 msgr2=0x7f5b5819b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:09.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.066+0000 7f5b5e11d700 1 --2- 172.21.15.67:0/2603882767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b580ffc20 0x7f5b5819b5a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f5b50006010 tx=0x7f5b5000bd70 comp rx=0 tx=0).stop 2024-04-22T23:45:09.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.066+0000 7f5b5e11d700 1 -- 172.21.15.67:0/2603882767 shutdown_connections 2024-04-22T23:45:09.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.066+0000 7f5b5e11d700 1 --2- 172.21.15.67:0/2603882767 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5b4406ecc0 0x7f5b44071180 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:09.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.066+0000 7f5b5e11d700 1 --2- 172.21.15.67:0/2603882767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b580ffc20 0x7f5b5819b5a0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:09.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.066+0000 7f5b5e11d700 1 --2- 172.21.15.67:0/2603882767 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b58105bd0 0x7f5b5819bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:09.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.066+0000 7f5b5e11d700 1 -- 172.21.15.67:0/2603882767 >> 172.21.15.67:0/2603882767 conn(0x7f5b58075670 msgr2=0x7f5b580feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:45:09.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.066+0000 7f5b5e11d700 1 -- 172.21.15.67:0/2603882767 shutdown_connections 2024-04-22T23:45:09.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.066+0000 7f5b5e11d700 1 -- 172.21.15.67:0/2603882767 wait complete. 2024-04-22T23:45:09.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.173+0000 7fcecf30c700 1 -- 172.21.15.67:0/415558126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcec8105350 msgr2=0x7fcec8105730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:09.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.173+0000 7fcecf30c700 1 --2- 172.21.15.67:0/415558126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcec8105350 0x7fcec8105730 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fcebc00aa20 tx=0x7fcebc00ad30 comp rx=0 tx=0).stop 2024-04-22T23:45:09.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.173+0000 7fcecf30c700 1 -- 172.21.15.67:0/415558126 shutdown_connections 2024-04-22T23:45:09.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.173+0000 7fcecf30c700 1 --2- 172.21.15.67:0/415558126 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcec80ff340 0x7fcec80ff7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:09.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.173+0000 7fcecf30c700 1 --2- 172.21.15.67:0/415558126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcec8105350 0x7fcec8105730 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:09.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.173+0000 7fcecf30c700 1 -- 172.21.15.67:0/415558126 >> 172.21.15.67:0/415558126 conn(0x7fcec8074e30 msgr2=0x7fcec8075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:45:09.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.173+0000 7fcecf30c700 1 -- 172.21.15.67:0/415558126 shutdown_connections 2024-04-22T23:45:09.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.173+0000 7fcecf30c700 1 -- 172.21.15.67:0/415558126 wait complete. 2024-04-22T23:45:09.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.174+0000 7fcecf30c700 1 Processor -- start 2024-04-22T23:45:09.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.174+0000 7fcecf30c700 1 -- start start 2024-04-22T23:45:09.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.174+0000 7fcecf30c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcec80ff340 0x7fcec8100fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:09.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.174+0000 7fcecf30c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcec8105350 0x7fcec8101500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:09.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.174+0000 7fcecf30c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec8101a40 con 0x7fcec8105350 2024-04-22T23:45:09.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.174+0000 7fcecf30c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec8101b80 con 0x7fcec80ff340 2024-04-22T23:45:09.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.175+0000 7fcecc8a7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcec8105350 0x7fcec8101500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:09.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.175+0000 7fcecc8a7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcec8105350 0x7fcec8101500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40630/0 (socket says 172.21.15.67:40630) 2024-04-22T23:45:09.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.175+0000 7fcecc8a7700 1 -- 172.21.15.67:0/2100868217 learned_addr learned my addr 172.21.15.67:0/2100868217 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:45:09.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.175+0000 7fcecd0a8700 1 --2- 172.21.15.67:0/2100868217 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcec80ff340 0x7fcec8100fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:09.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.175+0000 7fcecc8a7700 1 -- 172.21.15.67:0/2100868217 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcec80ff340 msgr2=0x7fcec8100fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:09.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.175+0000 7fcecc8a7700 1 --2- 172.21.15.67:0/2100868217 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcec80ff340 0x7fcec8100fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:09.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.175+0000 7fcecc8a7700 1 -- 172.21.15.67:0/2100868217 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcebc00a700 con 0x7fcec8105350 2024-04-22T23:45:09.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.175+0000 7fcecc8a7700 1 --2- 172.21.15.67:0/2100868217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcec8105350 0x7fcec8101500 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fcec400d8d0 tx=0x7fcec400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:45:09.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.175+0000 7fceba7fc700 1 -- 172.21.15.67:0/2100868217 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcec4009940 con 0x7fcec8105350 2024-04-22T23:45:09.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.175+0000 7fceba7fc700 1 -- 172.21.15.67:0/2100868217 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcec4010460 con 0x7fcec8105350 2024-04-22T23:45:09.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.175+0000 7fcecf30c700 1 -- 172.21.15.67:0/2100868217 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcec81a2a70 con 0x7fcec8105350 2024-04-22T23:45:09.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.175+0000 7fceba7fc700 1 -- 172.21.15.67:0/2100868217 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcec400f5f0 con 0x7fcec8105350 2024-04-22T23:45:09.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.175+0000 7fcecf30c700 1 -- 172.21.15.67:0/2100868217 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcec81a2e80 con 0x7fcec8105350 2024-04-22T23:45:09.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.176+0000 7fcecf30c700 1 -- 172.21.15.67:0/2100868217 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcec8109880 con 0x7fcec8105350 2024-04-22T23:45:09.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.177+0000 7fceba7fc700 1 -- 172.21.15.67:0/2100868217 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcec40105d0 con 0x7fcec8105350 2024-04-22T23:45:09.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.177+0000 7fceba7fc700 1 --2- 172.21.15.67:0/2100868217 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fceb406ecc0 0x7fceb4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:09.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.177+0000 7fceba7fc700 1 -- 172.21.15.67:0/2100868217 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fcec408eee0 con 0x7fcec8105350 2024-04-22T23:45:09.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.177+0000 7fcecd0a8700 1 --2- 172.21.15.67:0/2100868217 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fceb406ecc0 0x7fceb4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:09.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.178+0000 7fcecd0a8700 1 --2- 172.21.15.67:0/2100868217 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fceb406ecc0 0x7fceb4071180 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fcebc00d010 tx=0x7fcebc00c650 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:45:09.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.182+0000 7fceba7fc700 1 -- 172.21.15.67:0/2100868217 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fcec4057ba0 con 0x7fcec8105350 2024-04-22T23:45:09.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.478+0000 7fcecf30c700 1 -- 172.21.15.67:0/2100868217 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fcec804f2e0 con 0x7fcec8105350 2024-04-22T23:45:09.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.478+0000 7fceba7fc700 1 -- 172.21.15.67:0/2100868217 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7fcec40161c0 con 0x7fcec8105350 2024-04-22T23:45:09.470 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:45:09.470 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:45:09.470 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:45:09.470 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:45:09.471 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:45:09.471 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:45:09.471 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:45:09.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.480+0000 7fcecf30c700 1 -- 172.21.15.67:0/2100868217 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fceb406ecc0 msgr2=0x7fceb4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:09.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.480+0000 7fcecf30c700 1 --2- 172.21.15.67:0/2100868217 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fceb406ecc0 0x7fceb4071180 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fcebc00d010 tx=0x7fcebc00c650 comp rx=0 tx=0).stop 2024-04-22T23:45:09.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.480+0000 7fcecf30c700 1 -- 172.21.15.67:0/2100868217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcec8105350 msgr2=0x7fcec8101500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:09.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.480+0000 7fcecf30c700 1 --2- 172.21.15.67:0/2100868217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcec8105350 0x7fcec8101500 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fcec400d8d0 tx=0x7fcec400dc90 comp rx=0 tx=0).stop 2024-04-22T23:45:09.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.480+0000 7fcecf30c700 1 -- 172.21.15.67:0/2100868217 shutdown_connections 2024-04-22T23:45:09.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.480+0000 7fcecf30c700 1 --2- 172.21.15.67:0/2100868217 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fceb406ecc0 0x7fceb4071180 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:09.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.480+0000 7fcecf30c700 1 --2- 172.21.15.67:0/2100868217 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcec80ff340 0x7fcec8100fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:09.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.480+0000 7fcecf30c700 1 --2- 172.21.15.67:0/2100868217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcec8105350 0x7fcec8101500 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:09.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.480+0000 7fcecf30c700 1 -- 172.21.15.67:0/2100868217 >> 172.21.15.67:0/2100868217 conn(0x7fcec8074e30 msgr2=0x7fcec81038b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:45:09.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.480+0000 7fcecf30c700 1 -- 172.21.15.67:0/2100868217 shutdown_connections 2024-04-22T23:45:09.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:09.480+0000 7fcecf30c700 1 -- 172.21.15.67:0/2100868217 wait complete. 2024-04-22T23:45:09.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:09 smithi134 ceph-mon[85630]: pgmap v563: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:09.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:09 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3083211865' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:45:09.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:09 smithi067 ceph-mon[112438]: pgmap v563: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:09.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:09 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3083211865' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:45:10.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:10 smithi134 ceph-mon[85630]: from='client.35064 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:45:10.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:10 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2100868217' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:45:10.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:10 smithi067 ceph-mon[112438]: from='client.35064 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:45:10.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:10 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2100868217' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:45:11.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:11 smithi134 ceph-mon[85630]: pgmap v564: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:11.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:45:11.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:11 smithi067 ceph-mon[112438]: pgmap v564: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:11.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:45:13.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:13 smithi134 ceph-mon[85630]: pgmap v565: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:13.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:13 smithi067 ceph-mon[112438]: pgmap v565: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:15.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:15 smithi134 ceph-mon[85630]: pgmap v566: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:15.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:15 smithi067 ceph-mon[112438]: pgmap v566: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:17.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:17 smithi134 ceph-mon[85630]: pgmap v567: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:17.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:17 smithi067 ceph-mon[112438]: pgmap v567: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:19.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:19 smithi134 ceph-mon[85630]: pgmap v568: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:19.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:19 smithi067 ceph-mon[112438]: pgmap v568: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:21.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:21 smithi134 ceph-mon[85630]: pgmap v569: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:21.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:21 smithi067 ceph-mon[112438]: pgmap v569: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:23.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:23 smithi067 ceph-mon[112438]: pgmap v570: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:23.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:23 smithi134 ceph-mon[85630]: pgmap v570: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:25.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:25 smithi134 ceph-mon[85630]: pgmap v571: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:26.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:25 smithi067 ceph-mon[112438]: pgmap v571: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:26.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:45:27.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:45:27.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:27 smithi134 ceph-mon[85630]: pgmap v572: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:28.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:27 smithi067 ceph-mon[112438]: pgmap v572: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:29.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:29 smithi134 ceph-mon[85630]: pgmap v573: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:30.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:29 smithi067 ceph-mon[112438]: pgmap v573: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:31.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:31 smithi134 ceph-mon[85630]: pgmap v574: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:32.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:31 smithi067 ceph-mon[112438]: pgmap v574: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:33.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:33 smithi134 ceph-mon[85630]: pgmap v575: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:34.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:33 smithi067 ceph-mon[112438]: pgmap v575: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:35.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:35 smithi134 ceph-mon[85630]: pgmap v576: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:36.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:35 smithi067 ceph-mon[112438]: pgmap v576: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:37.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:37 smithi134 ceph-mon[85630]: pgmap v577: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:38.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:37 smithi067 ceph-mon[112438]: pgmap v577: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:39.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.601+0000 7f2d3701a700 1 -- 172.21.15.67:0/1079816343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d300ffda0 msgr2=0x7f2d30100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:39.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.601+0000 7f2d3701a700 1 --2- 172.21.15.67:0/1079816343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d300ffda0 0x7f2d30100180 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f2d2800b0a0 tx=0x7f2d2800b3b0 comp rx=0 tx=0).stop 2024-04-22T23:45:39.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.602+0000 7f2d3701a700 1 -- 172.21.15.67:0/1079816343 shutdown_connections 2024-04-22T23:45:39.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.602+0000 7f2d3701a700 1 --2- 172.21.15.67:0/1079816343 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d30100750 0x7f2d301047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:39.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.602+0000 7f2d3701a700 1 --2- 172.21.15.67:0/1079816343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d300ffda0 0x7f2d30100180 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:39.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.602+0000 7f2d3701a700 1 -- 172.21.15.67:0/1079816343 >> 172.21.15.67:0/1079816343 conn(0x7f2d300fb650 msgr2=0x7f2d300fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:45:39.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.602+0000 7f2d3701a700 1 -- 172.21.15.67:0/1079816343 shutdown_connections 2024-04-22T23:45:39.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.602+0000 7f2d3701a700 1 -- 172.21.15.67:0/1079816343 wait complete. 2024-04-22T23:45:39.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.602+0000 7f2d3701a700 1 Processor -- start 2024-04-22T23:45:39.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.603+0000 7f2d3701a700 1 -- start start 2024-04-22T23:45:39.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.603+0000 7f2d3701a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d300ffda0 0x7f2d3019b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:39.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.603+0000 7f2d3701a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d30100750 0x7f2d3019ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:39.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.603+0000 7f2d3701a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d3019c090 con 0x7f2d300ffda0 2024-04-22T23:45:39.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.603+0000 7f2d3701a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d301955d0 con 0x7f2d30100750 2024-04-22T23:45:39.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.603+0000 7f2d34db6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d300ffda0 0x7f2d3019b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:39.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.603+0000 7f2d34db6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d300ffda0 0x7f2d3019b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50438/0 (socket says 172.21.15.67:50438) 2024-04-22T23:45:39.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.603+0000 7f2d34db6700 1 -- 172.21.15.67:0/542812724 learned_addr learned my addr 172.21.15.67:0/542812724 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:45:39.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.603+0000 7f2d2ffff700 1 --2- 172.21.15.67:0/542812724 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d30100750 0x7f2d3019ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:39.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.603+0000 7f2d34db6700 1 -- 172.21.15.67:0/542812724 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d30100750 msgr2=0x7f2d3019ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:39.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.603+0000 7f2d34db6700 1 --2- 172.21.15.67:0/542812724 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d30100750 0x7f2d3019ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:39.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.603+0000 7f2d34db6700 1 -- 172.21.15.67:0/542812724 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d28009d00 con 0x7f2d300ffda0 2024-04-22T23:45:39.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.603+0000 7f2d34db6700 1 --2- 172.21.15.67:0/542812724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d300ffda0 0x7f2d3019b550 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f2d28000f80 tx=0x7f2d2800cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:45:39.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.604+0000 7f2d2dffb700 1 -- 172.21.15.67:0/542812724 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d2800e040 con 0x7f2d300ffda0 2024-04-22T23:45:39.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.604+0000 7f2d3701a700 1 -- 172.21.15.67:0/542812724 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d30195850 con 0x7f2d300ffda0 2024-04-22T23:45:39.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.604+0000 7f2d2dffb700 1 -- 172.21.15.67:0/542812724 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2d28003de0 con 0x7f2d300ffda0 2024-04-22T23:45:39.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.604+0000 7f2d2dffb700 1 -- 172.21.15.67:0/542812724 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d2801d7d0 con 0x7f2d300ffda0 2024-04-22T23:45:39.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.604+0000 7f2d3701a700 1 -- 172.21.15.67:0/542812724 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d30195d40 con 0x7f2d300ffda0 2024-04-22T23:45:39.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.605+0000 7f2d3701a700 1 -- 172.21.15.67:0/542812724 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d30108140 con 0x7f2d300ffda0 2024-04-22T23:45:39.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.605+0000 7f2d2dffb700 1 -- 172.21.15.67:0/542812724 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2d28004040 con 0x7f2d300ffda0 2024-04-22T23:45:39.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.606+0000 7f2d2dffb700 1 --2- 172.21.15.67:0/542812724 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2d1c06ecc0 0x7f2d1c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:39.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.606+0000 7f2d2dffb700 1 -- 172.21.15.67:0/542812724 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f2d28090930 con 0x7f2d300ffda0 2024-04-22T23:45:39.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.606+0000 7f2d2ffff700 1 --2- 172.21.15.67:0/542812724 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2d1c06ecc0 0x7f2d1c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:39.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.607+0000 7f2d2ffff700 1 --2- 172.21.15.67:0/542812724 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2d1c06ecc0 0x7f2d1c071180 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2d30104a20 tx=0x7f2d2000a380 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:45:39.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.611+0000 7f2d2dffb700 1 -- 172.21.15.67:0/542812724 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f2d2805d0a0 con 0x7f2d300ffda0 2024-04-22T23:45:39.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.840+0000 7f2d3701a700 1 -- 172.21.15.67:0/542812724 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2d30196b10 con 0x7f2d1c06ecc0 2024-04-22T23:45:39.834 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:39 smithi067 ceph-mon[112438]: pgmap v578: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:39.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.842+0000 7f2d2dffb700 1 -- 172.21.15.67:0/542812724 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f2d30196b10 con 0x7f2d1c06ecc0 2024-04-22T23:45:39.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.844+0000 7f2d3701a700 1 -- 172.21.15.67:0/542812724 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2d1c06ecc0 msgr2=0x7f2d1c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:39.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.844+0000 7f2d3701a700 1 --2- 172.21.15.67:0/542812724 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2d1c06ecc0 0x7f2d1c071180 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2d30104a20 tx=0x7f2d2000a380 comp rx=0 tx=0).stop 2024-04-22T23:45:39.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.844+0000 7f2d3701a700 1 -- 172.21.15.67:0/542812724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d300ffda0 msgr2=0x7f2d3019b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:39.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.844+0000 7f2d3701a700 1 --2- 172.21.15.67:0/542812724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d300ffda0 0x7f2d3019b550 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f2d28000f80 tx=0x7f2d2800cd20 comp rx=0 tx=0).stop 2024-04-22T23:45:39.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.844+0000 7f2d3701a700 1 -- 172.21.15.67:0/542812724 shutdown_connections 2024-04-22T23:45:39.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.845+0000 7f2d3701a700 1 --2- 172.21.15.67:0/542812724 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2d1c06ecc0 0x7f2d1c071180 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:39.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.845+0000 7f2d3701a700 1 --2- 172.21.15.67:0/542812724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d300ffda0 0x7f2d3019b550 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:39.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.845+0000 7f2d3701a700 1 --2- 172.21.15.67:0/542812724 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d30100750 0x7f2d3019ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:39.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.845+0000 7f2d3701a700 1 -- 172.21.15.67:0/542812724 >> 172.21.15.67:0/542812724 conn(0x7f2d300fb650 msgr2=0x7f2d300fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:45:39.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.845+0000 7f2d3701a700 1 -- 172.21.15.67:0/542812724 shutdown_connections 2024-04-22T23:45:39.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.845+0000 7f2d3701a700 1 -- 172.21.15.67:0/542812724 wait complete. 2024-04-22T23:45:39.848 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:45:39.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.953+0000 7f2aceded700 1 -- 172.21.15.67:0/4068767233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ac8100ea0 msgr2=0x7f2ac8101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:39.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.953+0000 7f2aceded700 1 --2- 172.21.15.67:0/4068767233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ac8100ea0 0x7f2ac8101280 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f2ab8009910 tx=0x7f2ab8009c20 comp rx=0 tx=0).stop 2024-04-22T23:45:39.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.953+0000 7f2aceded700 1 -- 172.21.15.67:0/4068767233 shutdown_connections 2024-04-22T23:45:39.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.953+0000 7f2aceded700 1 --2- 172.21.15.67:0/4068767233 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ac8101850 0x7f2ac81058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:39.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.953+0000 7f2aceded700 1 --2- 172.21.15.67:0/4068767233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ac8100ea0 0x7f2ac8101280 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:39.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.953+0000 7f2aceded700 1 -- 172.21.15.67:0/4068767233 >> 172.21.15.67:0/4068767233 conn(0x7f2ac80fc750 msgr2=0x7f2ac80feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:45:39.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.953+0000 7f2aceded700 1 -- 172.21.15.67:0/4068767233 shutdown_connections 2024-04-22T23:45:39.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.953+0000 7f2aceded700 1 -- 172.21.15.67:0/4068767233 wait complete. 2024-04-22T23:45:39.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.953+0000 7f2aceded700 1 Processor -- start 2024-04-22T23:45:39.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.953+0000 7f2aceded700 1 -- start start 2024-04-22T23:45:39.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.954+0000 7f2aceded700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ac8100ea0 0x7f2ac819c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:39.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.954+0000 7f2aceded700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ac8101850 0x7f2ac819cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:39.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.954+0000 7f2aceded700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac819d2c0 con 0x7f2ac8100ea0 2024-04-22T23:45:39.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.954+0000 7f2aceded700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac8196770 con 0x7f2ac8101850 2024-04-22T23:45:39.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.954+0000 7f2accb89700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ac8100ea0 0x7f2ac819c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:39.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.954+0000 7f2accb89700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ac8100ea0 0x7f2ac819c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50456/0 (socket says 172.21.15.67:50456) 2024-04-22T23:45:39.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.954+0000 7f2accb89700 1 -- 172.21.15.67:0/1537384171 learned_addr learned my addr 172.21.15.67:0/1537384171 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:45:39.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.954+0000 7f2ac7fff700 1 --2- 172.21.15.67:0/1537384171 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ac8101850 0x7f2ac819cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:39.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.954+0000 7f2accb89700 1 -- 172.21.15.67:0/1537384171 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ac8101850 msgr2=0x7f2ac819cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:39.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.954+0000 7f2accb89700 1 --2- 172.21.15.67:0/1537384171 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ac8101850 0x7f2ac819cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.954+0000 7f2accb89700 1 -- 172.21.15.67:0/1537384171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab80095f0 con 0x7f2ac8100ea0 2024-04-22T23:45:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.954+0000 7f2ac7fff700 1 --2- 172.21.15.67:0/1537384171 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ac8101850 0x7f2ac819cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:45:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.954+0000 7f2accb89700 1 --2- 172.21.15.67:0/1537384171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ac8100ea0 0x7f2ac819c6f0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f2ab8006010 tx=0x7f2ab800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:45:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.954+0000 7f2ac5ffb700 1 -- 172.21.15.67:0/1537384171 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ab801d070 con 0x7f2ac8100ea0 2024-04-22T23:45:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.954+0000 7f2ac5ffb700 1 -- 172.21.15.67:0/1537384171 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2ab800f460 con 0x7f2ac8100ea0 2024-04-22T23:45:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.955+0000 7f2aceded700 1 -- 172.21.15.67:0/1537384171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ac81969f0 con 0x7f2ac8100ea0 2024-04-22T23:45:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.955+0000 7f2ac5ffb700 1 -- 172.21.15.67:0/1537384171 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ab8017630 con 0x7f2ac8100ea0 2024-04-22T23:45:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.955+0000 7f2aceded700 1 -- 172.21.15.67:0/1537384171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ac8196ee0 con 0x7f2ac8100ea0 2024-04-22T23:45:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.956+0000 7f2aceded700 1 -- 172.21.15.67:0/1537384171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ac8109240 con 0x7f2ac8100ea0 2024-04-22T23:45:39.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.956+0000 7f2ac5ffb700 1 -- 172.21.15.67:0/1537384171 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2ab800f5d0 con 0x7f2ac8100ea0 2024-04-22T23:45:39.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.957+0000 7f2ac5ffb700 1 --2- 172.21.15.67:0/1537384171 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2ab006ecc0 0x7f2ab0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:39.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.957+0000 7f2ac5ffb700 1 -- 172.21.15.67:0/1537384171 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f2ab8090400 con 0x7f2ac8100ea0 2024-04-22T23:45:39.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.957+0000 7f2ac7fff700 1 --2- 172.21.15.67:0/1537384171 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2ab006ecc0 0x7f2ab0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:39.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.957+0000 7f2ac7fff700 1 --2- 172.21.15.67:0/1537384171 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2ab006ecc0 0x7f2ab0071180 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2abc005d90 tx=0x7f2abc005d00 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:45:39.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:39.961+0000 7f2ac5ffb700 1 -- 172.21.15.67:0/1537384171 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f2ab805cbf0 con 0x7f2ac8100ea0 2024-04-22T23:45:39.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:39 smithi134 ceph-mon[85630]: pgmap v578: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:40.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.194+0000 7f2aceded700 1 -- 172.21.15.67:0/1537384171 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2ac8197d90 con 0x7f2ab006ecc0 2024-04-22T23:45:40.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.196+0000 7f2ac5ffb700 1 -- 172.21.15.67:0/1537384171 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f2ac8197d90 con 0x7f2ab006ecc0 2024-04-22T23:45:40.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.198+0000 7f2aceded700 1 -- 172.21.15.67:0/1537384171 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2ab006ecc0 msgr2=0x7f2ab0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:40.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.198+0000 7f2aceded700 1 --2- 172.21.15.67:0/1537384171 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2ab006ecc0 0x7f2ab0071180 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2abc005d90 tx=0x7f2abc005d00 comp rx=0 tx=0).stop 2024-04-22T23:45:40.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.198+0000 7f2aceded700 1 -- 172.21.15.67:0/1537384171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ac8100ea0 msgr2=0x7f2ac819c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:40.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.198+0000 7f2aceded700 1 --2- 172.21.15.67:0/1537384171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ac8100ea0 0x7f2ac819c6f0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f2ab8006010 tx=0x7f2ab800be20 comp rx=0 tx=0).stop 2024-04-22T23:45:40.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.198+0000 7f2aceded700 1 -- 172.21.15.67:0/1537384171 shutdown_connections 2024-04-22T23:45:40.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.198+0000 7f2aceded700 1 --2- 172.21.15.67:0/1537384171 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2ab006ecc0 0x7f2ab0071180 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:40.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.198+0000 7f2aceded700 1 --2- 172.21.15.67:0/1537384171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ac8100ea0 0x7f2ac819c6f0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:40.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.198+0000 7f2aceded700 1 --2- 172.21.15.67:0/1537384171 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ac8101850 0x7f2ac819cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:40.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.198+0000 7f2aceded700 1 -- 172.21.15.67:0/1537384171 >> 172.21.15.67:0/1537384171 conn(0x7f2ac80fc750 msgr2=0x7f2ac80fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:45:40.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.198+0000 7f2aceded700 1 -- 172.21.15.67:0/1537384171 shutdown_connections 2024-04-22T23:45:40.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.198+0000 7f2aceded700 1 -- 172.21.15.67:0/1537384171 wait complete. 2024-04-22T23:45:40.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.304+0000 7ff77d703700 1 -- 172.21.15.67:0/1839268926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff778101850 msgr2=0x7ff7781058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:40.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.304+0000 7ff77d703700 1 --2- 172.21.15.67:0/1839268926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff778101850 0x7ff7781058a0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ff76c009910 tx=0x7ff76c009c20 comp rx=0 tx=0).stop 2024-04-22T23:45:40.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.305+0000 7ff77d703700 1 -- 172.21.15.67:0/1839268926 shutdown_connections 2024-04-22T23:45:40.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.305+0000 7ff77d703700 1 --2- 172.21.15.67:0/1839268926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff778101850 0x7ff7781058a0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:40.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.305+0000 7ff77d703700 1 --2- 172.21.15.67:0/1839268926 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff778100ea0 0x7ff778101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:40.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.305+0000 7ff77d703700 1 -- 172.21.15.67:0/1839268926 >> 172.21.15.67:0/1839268926 conn(0x7ff7780fc750 msgr2=0x7ff7780feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:45:40.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.305+0000 7ff77d703700 1 -- 172.21.15.67:0/1839268926 shutdown_connections 2024-04-22T23:45:40.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.305+0000 7ff77d703700 1 -- 172.21.15.67:0/1839268926 wait complete. 2024-04-22T23:45:40.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.305+0000 7ff77d703700 1 Processor -- start 2024-04-22T23:45:40.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.305+0000 7ff77d703700 1 -- start start 2024-04-22T23:45:40.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.306+0000 7ff77d703700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff778100ea0 0x7ff778073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:40.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.306+0000 7ff77d703700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff778101850 0x7ff7780735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:40.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.306+0000 7ff77d703700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff778073b10 con 0x7ff778101850 2024-04-22T23:45:40.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.306+0000 7ff77d703700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff778073c50 con 0x7ff778100ea0 2024-04-22T23:45:40.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.306+0000 7ff7767fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff778101850 0x7ff7780735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:40.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.306+0000 7ff7767fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff778101850 0x7ff7780735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50474/0 (socket says 172.21.15.67:50474) 2024-04-22T23:45:40.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.306+0000 7ff7767fc700 1 -- 172.21.15.67:0/2457377294 learned_addr learned my addr 172.21.15.67:0/2457377294 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:45:40.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.306+0000 7ff776ffd700 1 --2- 172.21.15.67:0/2457377294 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff778100ea0 0x7ff778073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:40.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.306+0000 7ff7767fc700 1 -- 172.21.15.67:0/2457377294 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff778100ea0 msgr2=0x7ff778073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:40.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.306+0000 7ff7767fc700 1 --2- 172.21.15.67:0/2457377294 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff778100ea0 0x7ff778073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:40.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.306+0000 7ff7767fc700 1 -- 172.21.15.67:0/2457377294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff76c0095f0 con 0x7ff778101850 2024-04-22T23:45:40.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.306+0000 7ff7767fc700 1 --2- 172.21.15.67:0/2457377294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff778101850 0x7ff7780735d0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ff76c0092e0 tx=0x7ff76c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:45:40.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.306+0000 7ff75ffff700 1 -- 172.21.15.67:0/2457377294 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff76c01d070 con 0x7ff778101850 2024-04-22T23:45:40.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.306+0000 7ff77d703700 1 -- 172.21.15.67:0/2457377294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff778073ed0 con 0x7ff778101850 2024-04-22T23:45:40.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.306+0000 7ff75ffff700 1 -- 172.21.15.67:0/2457377294 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff76c00f460 con 0x7ff778101850 2024-04-22T23:45:40.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.306+0000 7ff75ffff700 1 -- 172.21.15.67:0/2457377294 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff76c0175f0 con 0x7ff778101850 2024-04-22T23:45:40.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.307+0000 7ff77d703700 1 -- 172.21.15.67:0/2457377294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7781a2e30 con 0x7ff778101850 2024-04-22T23:45:40.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.307+0000 7ff77d703700 1 -- 172.21.15.67:0/2457377294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7781092a0 con 0x7ff778101850 2024-04-22T23:45:40.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.308+0000 7ff75ffff700 1 -- 172.21.15.67:0/2457377294 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff76c00f5d0 con 0x7ff778101850 2024-04-22T23:45:40.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.308+0000 7ff75ffff700 1 --2- 172.21.15.67:0/2457377294 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff76006ecc0 0x7ff760071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:40.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.308+0000 7ff75ffff700 1 -- 172.21.15.67:0/2457377294 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7ff76c090500 con 0x7ff778101850 2024-04-22T23:45:40.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.308+0000 7ff776ffd700 1 --2- 172.21.15.67:0/2457377294 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff76006ecc0 0x7ff760071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:40.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.309+0000 7ff776ffd700 1 --2- 172.21.15.67:0/2457377294 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff76006ecc0 0x7ff760071180 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff768005fd0 tx=0x7ff76800a560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:45:40.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.313+0000 7ff75ffff700 1 -- 172.21.15.67:0/2457377294 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff76c059140 con 0x7ff778101850 2024-04-22T23:45:40.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.532+0000 7ff77d703700 1 -- 172.21.15.67:0/2457377294 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff778074630 con 0x7ff76006ecc0 2024-04-22T23:45:40.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.543+0000 7ff75ffff700 1 -- 172.21.15.67:0/2457377294 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7ff778074630 con 0x7ff76006ecc0 2024-04-22T23:45:40.536 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:45:40.536 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (19m) 12m ago 30m 25.5M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:45:40.536 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (30m) 12m ago 30m 9114k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:45:40.536 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (29m) 12m ago 29m 9042k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:45:40.536 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (16m) 12m ago 30m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:45:40.536 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (16m) 12m ago 29m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:45:40.537 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (19m) 12m ago 30m 79.3M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:45:40.537 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (13m) 12m ago 25m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:45:40.537 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (12m) 12m ago 25m 12.6M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:45:40.537 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (12m) 12m ago 25m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:45:40.537 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (25m) 12m ago 25m 12.2M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:45:40.537 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (21m) 12m ago 32m 511M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:45:40.537 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (20m) 12m ago 29m 453M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:45:40.537 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (17m) 12m ago 32m 55.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:45:40.537 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (17m) 12m ago 29m 46.0M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:45:40.537 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (20m) 12m ago 30m 15.4M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:45:40.537 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (19m) 12m ago 29m 16.1M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:45:40.537 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (16m) 12m ago 28m 99.2M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:45:40.537 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (15m) 12m ago 28m 127M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:45:40.537 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (15m) 12m ago 28m 79.9M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:45:40.538 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (14m) 12m ago 27m 97.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:45:40.538 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (13m) 12m ago 27m 82.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:45:40.538 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (13m) 12m ago 26m 77.6M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:45:40.538 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (19m) 12m ago 30m 73.9M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:45:40.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.546+0000 7ff77d703700 1 -- 172.21.15.67:0/2457377294 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff76006ecc0 msgr2=0x7ff760071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:40.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.546+0000 7ff77d703700 1 --2- 172.21.15.67:0/2457377294 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff76006ecc0 0x7ff760071180 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff768005fd0 tx=0x7ff76800a560 comp rx=0 tx=0).stop 2024-04-22T23:45:40.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.546+0000 7ff77d703700 1 -- 172.21.15.67:0/2457377294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff778101850 msgr2=0x7ff7780735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:40.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.546+0000 7ff77d703700 1 --2- 172.21.15.67:0/2457377294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff778101850 0x7ff7780735d0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ff76c0092e0 tx=0x7ff76c00be20 comp rx=0 tx=0).stop 2024-04-22T23:45:40.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.546+0000 7ff77d703700 1 -- 172.21.15.67:0/2457377294 shutdown_connections 2024-04-22T23:45:40.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.546+0000 7ff77d703700 1 --2- 172.21.15.67:0/2457377294 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff76006ecc0 0x7ff760071180 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:40.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.546+0000 7ff77d703700 1 --2- 172.21.15.67:0/2457377294 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff778100ea0 0x7ff778073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:40.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.546+0000 7ff77d703700 1 --2- 172.21.15.67:0/2457377294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff778101850 0x7ff7780735d0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:40.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.546+0000 7ff77d703700 1 -- 172.21.15.67:0/2457377294 >> 172.21.15.67:0/2457377294 conn(0x7ff7780fc750 msgr2=0x7ff7780fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:45:40.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.546+0000 7ff77d703700 1 -- 172.21.15.67:0/2457377294 shutdown_connections 2024-04-22T23:45:40.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.546+0000 7ff77d703700 1 -- 172.21.15.67:0/2457377294 wait complete. 2024-04-22T23:45:40.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.652+0000 7fb15009f700 1 -- 172.21.15.67:0/3316876073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1480ffc20 msgr2=0x7fb148100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:40.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.652+0000 7fb15009f700 1 --2- 172.21.15.67:0/3316876073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1480ffc20 0x7fb148100080 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fb13c009910 tx=0x7fb13c009c20 comp rx=0 tx=0).stop 2024-04-22T23:45:40.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.652+0000 7fb15009f700 1 -- 172.21.15.67:0/3316876073 shutdown_connections 2024-04-22T23:45:40.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.652+0000 7fb15009f700 1 --2- 172.21.15.67:0/3316876073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1480ffc20 0x7fb148100080 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:40.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.652+0000 7fb15009f700 1 --2- 172.21.15.67:0/3316876073 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb148105bd0 0x7fb148105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:40.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.652+0000 7fb15009f700 1 -- 172.21.15.67:0/3316876073 >> 172.21.15.67:0/3316876073 conn(0x7fb148075670 msgr2=0x7fb148075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:45:40.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.652+0000 7fb15009f700 1 -- 172.21.15.67:0/3316876073 shutdown_connections 2024-04-22T23:45:40.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.652+0000 7fb15009f700 1 -- 172.21.15.67:0/3316876073 wait complete. 2024-04-22T23:45:40.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.653+0000 7fb15009f700 1 Processor -- start 2024-04-22T23:45:40.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.653+0000 7fb15009f700 1 -- start start 2024-04-22T23:45:40.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.653+0000 7fb15009f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1480ffc20 0x7fb14819b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:40.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.653+0000 7fb15009f700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb148105bd0 0x7fb14819bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:40.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.653+0000 7fb15009f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb148195620 con 0x7fb1480ffc20 2024-04-22T23:45:40.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.653+0000 7fb15009f700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb148195790 con 0x7fb148105bd0 2024-04-22T23:45:40.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.653+0000 7fb14de3b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1480ffc20 0x7fb14819b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:40.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.653+0000 7fb14de3b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1480ffc20 0x7fb14819b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50504/0 (socket says 172.21.15.67:50504) 2024-04-22T23:45:40.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.653+0000 7fb14de3b700 1 -- 172.21.15.67:0/2762801326 learned_addr learned my addr 172.21.15.67:0/2762801326 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:45:40.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.653+0000 7fb14d63a700 1 --2- 172.21.15.67:0/2762801326 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb148105bd0 0x7fb14819bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:40.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.653+0000 7fb14de3b700 1 -- 172.21.15.67:0/2762801326 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb148105bd0 msgr2=0x7fb14819bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:40.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.653+0000 7fb14de3b700 1 --2- 172.21.15.67:0/2762801326 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb148105bd0 0x7fb14819bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:40.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.653+0000 7fb14de3b700 1 -- 172.21.15.67:0/2762801326 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb13c0095f0 con 0x7fb1480ffc20 2024-04-22T23:45:40.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.654+0000 7fb14de3b700 1 --2- 172.21.15.67:0/2762801326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1480ffc20 0x7fb14819b630 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fb14400d8d0 tx=0x7fb14400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:45:40.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.654+0000 7fb13affd700 1 -- 172.21.15.67:0/2762801326 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb144009940 con 0x7fb1480ffc20 2024-04-22T23:45:40.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.654+0000 7fb13affd700 1 -- 172.21.15.67:0/2762801326 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb144010460 con 0x7fb1480ffc20 2024-04-22T23:45:40.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.654+0000 7fb13affd700 1 -- 172.21.15.67:0/2762801326 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb14400f5f0 con 0x7fb1480ffc20 2024-04-22T23:45:40.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.654+0000 7fb15009f700 1 -- 172.21.15.67:0/2762801326 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb148195990 con 0x7fb1480ffc20 2024-04-22T23:45:40.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.654+0000 7fb15009f700 1 -- 172.21.15.67:0/2762801326 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb148195e90 con 0x7fb1480ffc20 2024-04-22T23:45:40.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.655+0000 7fb15009f700 1 -- 172.21.15.67:0/2762801326 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb14804f2e0 con 0x7fb1480ffc20 2024-04-22T23:45:40.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.655+0000 7fb13affd700 1 -- 172.21.15.67:0/2762801326 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb1440105d0 con 0x7fb1480ffc20 2024-04-22T23:45:40.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.656+0000 7fb13affd700 1 --2- 172.21.15.67:0/2762801326 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb13406ecc0 0x7fb134071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:40.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.656+0000 7fb14d63a700 1 --2- 172.21.15.67:0/2762801326 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb13406ecc0 0x7fb134071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:40.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.656+0000 7fb13affd700 1 -- 172.21.15.67:0/2762801326 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fb14408eff0 con 0x7fb1480ffc20 2024-04-22T23:45:40.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.657+0000 7fb14d63a700 1 --2- 172.21.15.67:0/2762801326 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb13406ecc0 0x7fb134071180 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb148196ec0 tx=0x7fb13c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:45:40.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.661+0000 7fb13affd700 1 -- 172.21.15.67:0/2762801326 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb144057c30 con 0x7fb1480ffc20 2024-04-22T23:45:40.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:40 smithi067 ceph-mon[112438]: from='client.35072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:45:40.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.961+0000 7fb15009f700 1 -- 172.21.15.67:0/2762801326 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb1481960e0 con 0x7fb1480ffc20 2024-04-22T23:45:40.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.961+0000 7fb13affd700 1 -- 172.21.15.67:0/2762801326 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fb14405b2f0 con 0x7fb1480ffc20 2024-04-22T23:45:40.955 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:45:40.955 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:45:40.955 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:45:40.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:45:40.955 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:45:40.955 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:45:40.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:45:40.955 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:45:40.955 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:45:40.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:45:40.956 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:45:40.956 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:45:40.956 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 3 2024-04-22T23:45:40.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:45:40.956 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:45:40.956 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:45:40.956 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 13 2024-04-22T23:45:40.956 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:45:40.956 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:45:40.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.963+0000 7fb15009f700 1 -- 172.21.15.67:0/2762801326 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb13406ecc0 msgr2=0x7fb134071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:40.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.963+0000 7fb15009f700 1 --2- 172.21.15.67:0/2762801326 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb13406ecc0 0x7fb134071180 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb148196ec0 tx=0x7fb13c01a040 comp rx=0 tx=0).stop 2024-04-22T23:45:40.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.963+0000 7fb15009f700 1 -- 172.21.15.67:0/2762801326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1480ffc20 msgr2=0x7fb14819b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:40.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.963+0000 7fb15009f700 1 --2- 172.21.15.67:0/2762801326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1480ffc20 0x7fb14819b630 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fb14400d8d0 tx=0x7fb14400dc90 comp rx=0 tx=0).stop 2024-04-22T23:45:40.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.964+0000 7fb15009f700 1 -- 172.21.15.67:0/2762801326 shutdown_connections 2024-04-22T23:45:40.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.964+0000 7fb15009f700 1 --2- 172.21.15.67:0/2762801326 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb13406ecc0 0x7fb134071180 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:40.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.964+0000 7fb15009f700 1 --2- 172.21.15.67:0/2762801326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1480ffc20 0x7fb14819b630 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:40.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.964+0000 7fb15009f700 1 --2- 172.21.15.67:0/2762801326 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb148105bd0 0x7fb14819bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:40.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.964+0000 7fb15009f700 1 -- 172.21.15.67:0/2762801326 >> 172.21.15.67:0/2762801326 conn(0x7fb148075670 msgr2=0x7fb1480fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:45:40.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.964+0000 7fb15009f700 1 -- 172.21.15.67:0/2762801326 shutdown_connections 2024-04-22T23:45:40.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:40.964+0000 7fb15009f700 1 -- 172.21.15.67:0/2762801326 wait complete. 2024-04-22T23:45:40.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:40 smithi134 ceph-mon[85630]: from='client.35072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:45:41.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.071+0000 7ff9f9deb700 1 -- 172.21.15.67:0/618139632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9f4101850 msgr2=0x7ff9f41058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:41.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.071+0000 7ff9f9deb700 1 --2- 172.21.15.67:0/618139632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9f4101850 0x7ff9f41058a0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc009910 tx=0x7ff9dc009c20 comp rx=0 tx=0).stop 2024-04-22T23:45:41.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.071+0000 7ff9f9deb700 1 -- 172.21.15.67:0/618139632 shutdown_connections 2024-04-22T23:45:41.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.071+0000 7ff9f9deb700 1 --2- 172.21.15.67:0/618139632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9f4101850 0x7ff9f41058a0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:41.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.071+0000 7ff9f9deb700 1 --2- 172.21.15.67:0/618139632 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9f4100ea0 0x7ff9f4101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:41.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.071+0000 7ff9f9deb700 1 -- 172.21.15.67:0/618139632 >> 172.21.15.67:0/618139632 conn(0x7ff9f40fc750 msgr2=0x7ff9f40feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:45:41.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.071+0000 7ff9f9deb700 1 -- 172.21.15.67:0/618139632 shutdown_connections 2024-04-22T23:45:41.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.071+0000 7ff9f9deb700 1 -- 172.21.15.67:0/618139632 wait complete. 2024-04-22T23:45:41.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.072+0000 7ff9f9deb700 1 Processor -- start 2024-04-22T23:45:41.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.072+0000 7ff9f9deb700 1 -- start start 2024-04-22T23:45:41.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.072+0000 7ff9f9deb700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9f4100ea0 0x7ff9f419c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:41.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.072+0000 7ff9f9deb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9f4101850 0x7ff9f419cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:41.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.072+0000 7ff9f9deb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f419d2d0 con 0x7ff9f4101850 2024-04-22T23:45:41.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.072+0000 7ff9f9deb700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f4196780 con 0x7ff9f4100ea0 2024-04-22T23:45:41.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.072+0000 7ff9eaffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9f4101850 0x7ff9f419cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:41.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.072+0000 7ff9eaffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9f4101850 0x7ff9f419cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50534/0 (socket says 172.21.15.67:50534) 2024-04-22T23:45:41.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.072+0000 7ff9eaffd700 1 -- 172.21.15.67:0/1219107710 learned_addr learned my addr 172.21.15.67:0/1219107710 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:45:41.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.072+0000 7ff9f37fe700 1 --2- 172.21.15.67:0/1219107710 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9f4100ea0 0x7ff9f419c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:41.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.073+0000 7ff9eaffd700 1 -- 172.21.15.67:0/1219107710 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9f4100ea0 msgr2=0x7ff9f419c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:41.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.073+0000 7ff9eaffd700 1 --2- 172.21.15.67:0/1219107710 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9f4100ea0 0x7ff9f419c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:41.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.073+0000 7ff9eaffd700 1 -- 172.21.15.67:0/1219107710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9dc0095f0 con 0x7ff9f4101850 2024-04-22T23:45:41.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.073+0000 7ff9f37fe700 1 --2- 172.21.15.67:0/1219107710 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9f4100ea0 0x7ff9f419c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:45:41.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.073+0000 7ff9eaffd700 1 --2- 172.21.15.67:0/1219107710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9f4101850 0x7ff9f419cc40 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc000c00 tx=0x7ff9dc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:45:41.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.073+0000 7ff9f17fa700 1 -- 172.21.15.67:0/1219107710 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9dc01d070 con 0x7ff9f4101850 2024-04-22T23:45:41.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.073+0000 7ff9f17fa700 1 -- 172.21.15.67:0/1219107710 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff9dc005dc0 con 0x7ff9f4101850 2024-04-22T23:45:41.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.073+0000 7ff9f9deb700 1 -- 172.21.15.67:0/1219107710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9f4196a00 con 0x7ff9f4101850 2024-04-22T23:45:41.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.073+0000 7ff9f17fa700 1 -- 172.21.15.67:0/1219107710 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9dc017400 con 0x7ff9f4101850 2024-04-22T23:45:41.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.073+0000 7ff9f9deb700 1 -- 172.21.15.67:0/1219107710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9f4196ef0 con 0x7ff9f4101850 2024-04-22T23:45:41.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.074+0000 7ff9f9deb700 1 -- 172.21.15.67:0/1219107710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9f41092a0 con 0x7ff9f4101850 2024-04-22T23:45:41.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.075+0000 7ff9f17fa700 1 -- 172.21.15.67:0/1219107710 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff9dc00f7f0 con 0x7ff9f4101850 2024-04-22T23:45:41.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.075+0000 7ff9f17fa700 1 --2- 172.21.15.67:0/1219107710 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff9e006ecc0 0x7ff9e0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:41.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.075+0000 7ff9f37fe700 1 --2- 172.21.15.67:0/1219107710 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff9e006ecc0 0x7ff9e0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:41.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.075+0000 7ff9f17fa700 1 -- 172.21.15.67:0/1219107710 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7ff9dc0903f0 con 0x7ff9f4101850 2024-04-22T23:45:41.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.076+0000 7ff9f37fe700 1 --2- 172.21.15.67:0/1219107710 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff9e006ecc0 0x7ff9e0071180 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff9e4005fd0 tx=0x7ff9e4005ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:45:41.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.080+0000 7ff9f17fa700 1 -- 172.21.15.67:0/1219107710 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff9dc05cbe0 con 0x7ff9f4101850 2024-04-22T23:45:41.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.341+0000 7ff9f9deb700 1 -- 172.21.15.67:0/1219107710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff9f404f2e0 con 0x7ff9f4101850 2024-04-22T23:45:41.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.341+0000 7ff9f17fa700 1 -- 172.21.15.67:0/1219107710 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1871 (secure 0 0 0) 0x7ff9dc05c770 con 0x7ff9f4101850 2024-04-22T23:45:41.334 INFO:teuthology.orchestra.run.smithi067.stdout:e31 2024-04-22T23:45:41.335 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:45:41.335 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:45:41.335 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:45:41.335 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:45:41.335 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:45:41.335 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:45:41.336 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:45:41.336 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:45:41.336 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:45:41.336 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:45:41.336 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:45:41.336 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:45:41.336 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:45:41.336 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:45:41.336 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:45:41.336 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:45:41.336 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:45:41.336 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:45:41.336 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:45:41.336 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:45:41.336 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:45:41.337 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:45:41.337 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:45:41.337 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:45:41.337 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:45:41.337 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:45:41.337 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:45:41.337 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:45:41.337 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:45:41.337 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:45:41.337 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:45:41.337 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:45:41.337 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:45:41.337 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:45:41.337 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:45:41.337 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:45:41.337 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:45:41.338 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:45:41.338 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:45:41.338 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44259} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1827288036,v1:172.21.15.134:6827/1827288036] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:45:41.338 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:45:41.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.343+0000 7ff9f9deb700 1 -- 172.21.15.67:0/1219107710 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff9e006ecc0 msgr2=0x7ff9e0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:41.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.343+0000 7ff9f9deb700 1 --2- 172.21.15.67:0/1219107710 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff9e006ecc0 0x7ff9e0071180 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff9e4005fd0 tx=0x7ff9e4005ee0 comp rx=0 tx=0).stop 2024-04-22T23:45:41.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.343+0000 7ff9f9deb700 1 -- 172.21.15.67:0/1219107710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9f4101850 msgr2=0x7ff9f419cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:41.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.343+0000 7ff9f9deb700 1 --2- 172.21.15.67:0/1219107710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9f4101850 0x7ff9f419cc40 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc000c00 tx=0x7ff9dc00be20 comp rx=0 tx=0).stop 2024-04-22T23:45:41.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.343+0000 7ff9f9deb700 1 -- 172.21.15.67:0/1219107710 shutdown_connections 2024-04-22T23:45:41.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.343+0000 7ff9f9deb700 1 --2- 172.21.15.67:0/1219107710 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff9e006ecc0 0x7ff9e0071180 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:41.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.343+0000 7ff9f9deb700 1 --2- 172.21.15.67:0/1219107710 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9f4100ea0 0x7ff9f419c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:41.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.343+0000 7ff9f9deb700 1 --2- 172.21.15.67:0/1219107710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9f4101850 0x7ff9f419cc40 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:41.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.343+0000 7ff9f9deb700 1 -- 172.21.15.67:0/1219107710 >> 172.21.15.67:0/1219107710 conn(0x7ff9f40fc750 msgr2=0x7ff9f40fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:45:41.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.344+0000 7ff9f9deb700 1 -- 172.21.15.67:0/1219107710 shutdown_connections 2024-04-22T23:45:41.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.344+0000 7ff9f9deb700 1 -- 172.21.15.67:0/1219107710 wait complete. 2024-04-22T23:45:41.339 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 31 2024-04-22T23:45:41.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.452+0000 7f13a09cd700 1 -- 172.21.15.67:0/2599538634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f139c106cd0 msgr2=0x7f139c1070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:41.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.452+0000 7f13a09cd700 1 --2- 172.21.15.67:0/2599538634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f139c106cd0 0x7f139c1070b0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f138c009910 tx=0x7f138c009c20 comp rx=0 tx=0).stop 2024-04-22T23:45:41.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.452+0000 7f13a09cd700 1 -- 172.21.15.67:0/2599538634 shutdown_connections 2024-04-22T23:45:41.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.452+0000 7f13a09cd700 1 --2- 172.21.15.67:0/2599538634 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f139c068580 0x7f139c0689e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:41.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.452+0000 7f13a09cd700 1 --2- 172.21.15.67:0/2599538634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f139c106cd0 0x7f139c1070b0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:41.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.452+0000 7f13a09cd700 1 -- 172.21.15.67:0/2599538634 >> 172.21.15.67:0/2599538634 conn(0x7f139c075670 msgr2=0x7f139c075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:45:41.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.452+0000 7f13a09cd700 1 -- 172.21.15.67:0/2599538634 shutdown_connections 2024-04-22T23:45:41.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.452+0000 7f13a09cd700 1 -- 172.21.15.67:0/2599538634 wait complete. 2024-04-22T23:45:41.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.453+0000 7f13a09cd700 1 Processor -- start 2024-04-22T23:45:41.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.453+0000 7f13a09cd700 1 -- start start 2024-04-22T23:45:41.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.453+0000 7f13a09cd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f139c068580 0x7f139c19c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:41.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.453+0000 7f13a09cd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f139c106cd0 0x7f139c19cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:41.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.453+0000 7f13a09cd700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f139c19d230 con 0x7f139c106cd0 2024-04-22T23:45:41.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.453+0000 7f13a09cd700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f139c1966e0 con 0x7f139c068580 2024-04-22T23:45:41.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.453+0000 7f1399d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f139c106cd0 0x7f139c19cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:41.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.453+0000 7f1399d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f139c106cd0 0x7f139c19cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50538/0 (socket says 172.21.15.67:50538) 2024-04-22T23:45:41.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.453+0000 7f1399d9b700 1 -- 172.21.15.67:0/2318391712 learned_addr learned my addr 172.21.15.67:0/2318391712 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:45:41.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.453+0000 7f139a59c700 1 --2- 172.21.15.67:0/2318391712 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f139c068580 0x7f139c19c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:41.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.453+0000 7f1399d9b700 1 -- 172.21.15.67:0/2318391712 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f139c068580 msgr2=0x7f139c19c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:41.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.453+0000 7f1399d9b700 1 --2- 172.21.15.67:0/2318391712 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f139c068580 0x7f139c19c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:41.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.453+0000 7f1399d9b700 1 -- 172.21.15.67:0/2318391712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f138c0095f0 con 0x7f139c106cd0 2024-04-22T23:45:41.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.454+0000 7f1399d9b700 1 --2- 172.21.15.67:0/2318391712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f139c106cd0 0x7f139c19cba0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f139000d8d0 tx=0x7f139000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:45:41.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.454+0000 7f138b7fe700 1 -- 172.21.15.67:0/2318391712 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1390009940 con 0x7f139c106cd0 2024-04-22T23:45:41.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.454+0000 7f13a09cd700 1 -- 172.21.15.67:0/2318391712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f139c1969c0 con 0x7f139c106cd0 2024-04-22T23:45:41.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.454+0000 7f138b7fe700 1 -- 172.21.15.67:0/2318391712 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1390010460 con 0x7f139c106cd0 2024-04-22T23:45:41.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.454+0000 7f138b7fe700 1 -- 172.21.15.67:0/2318391712 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f139000f5f0 con 0x7f139c106cd0 2024-04-22T23:45:41.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.454+0000 7f13a09cd700 1 -- 172.21.15.67:0/2318391712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f139c196f10 con 0x7f139c106cd0 2024-04-22T23:45:41.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.455+0000 7f13a09cd700 1 -- 172.21.15.67:0/2318391712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f139c1091d0 con 0x7f139c106cd0 2024-04-22T23:45:41.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.455+0000 7f138b7fe700 1 -- 172.21.15.67:0/2318391712 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f13900105d0 con 0x7f139c106cd0 2024-04-22T23:45:41.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.456+0000 7f138b7fe700 1 --2- 172.21.15.67:0/2318391712 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f138406ecc0 0x7f1384071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:41.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.456+0000 7f138b7fe700 1 -- 172.21.15.67:0/2318391712 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f139008eff0 con 0x7f139c106cd0 2024-04-22T23:45:41.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.456+0000 7f139a59c700 1 --2- 172.21.15.67:0/2318391712 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f138406ecc0 0x7f1384071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:41.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.456+0000 7f139a59c700 1 --2- 172.21.15.67:0/2318391712 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f138406ecc0 0x7f1384071180 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f138c006010 tx=0x7f138c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:45:41.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.461+0000 7f138b7fe700 1 -- 172.21.15.67:0/2318391712 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f139005b760 con 0x7f139c106cd0 2024-04-22T23:45:41.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.696+0000 7f13a09cd700 1 -- 172.21.15.67:0/2318391712 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f139c1971f0 con 0x7f138406ecc0 2024-04-22T23:45:41.689 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:41 smithi067 ceph-mon[112438]: from='client.35076 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:45:41.689 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:41 smithi067 ceph-mon[112438]: pgmap v579: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:41.689 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:41 smithi067 ceph-mon[112438]: from='client.35080 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:45:41.689 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:41 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2762801326' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:45:41.690 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:45:41.690 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:41 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1219107710' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:45:41.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.697+0000 7f138b7fe700 1 -- 172.21.15.67:0/2318391712 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f139c1971f0 con 0x7f138406ecc0 2024-04-22T23:45:41.690 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:45:41.690 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:45:41.690 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:45:41.690 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:45:41.691 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:45:41.691 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:45:41.691 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:45:41.691 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:45:41.691 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:45:41.691 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:45:41.691 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T23:45:41.691 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:45:41.691 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:45:41.691 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:45:41.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.699+0000 7f13a09cd700 1 -- 172.21.15.67:0/2318391712 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f138406ecc0 msgr2=0x7f1384071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:41.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.699+0000 7f13a09cd700 1 --2- 172.21.15.67:0/2318391712 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f138406ecc0 0x7f1384071180 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f138c006010 tx=0x7f138c01a040 comp rx=0 tx=0).stop 2024-04-22T23:45:41.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.699+0000 7f13a09cd700 1 -- 172.21.15.67:0/2318391712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f139c106cd0 msgr2=0x7f139c19cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:41.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.699+0000 7f13a09cd700 1 --2- 172.21.15.67:0/2318391712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f139c106cd0 0x7f139c19cba0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f139000d8d0 tx=0x7f139000dc90 comp rx=0 tx=0).stop 2024-04-22T23:45:41.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.699+0000 7f13a09cd700 1 -- 172.21.15.67:0/2318391712 shutdown_connections 2024-04-22T23:45:41.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.699+0000 7f13a09cd700 1 --2- 172.21.15.67:0/2318391712 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f138406ecc0 0x7f1384071180 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:41.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.699+0000 7f13a09cd700 1 --2- 172.21.15.67:0/2318391712 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f139c068580 0x7f139c19c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:41.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.699+0000 7f13a09cd700 1 --2- 172.21.15.67:0/2318391712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f139c106cd0 0x7f139c19cba0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:41.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.699+0000 7f13a09cd700 1 -- 172.21.15.67:0/2318391712 >> 172.21.15.67:0/2318391712 conn(0x7f139c075670 msgr2=0x7f139c0fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:45:41.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.700+0000 7f13a09cd700 1 -- 172.21.15.67:0/2318391712 shutdown_connections 2024-04-22T23:45:41.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.700+0000 7f13a09cd700 1 -- 172.21.15.67:0/2318391712 wait complete. 2024-04-22T23:45:41.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.807+0000 7f05eaae7700 1 -- 172.21.15.67:0/3299373111 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05e40ffb40 msgr2=0x7f05e40fffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:41.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.807+0000 7f05eaae7700 1 --2- 172.21.15.67:0/3299373111 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05e40ffb40 0x7f05e40fffc0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f05d0009910 tx=0x7f05d0009c20 comp rx=0 tx=0).stop 2024-04-22T23:45:41.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.807+0000 7f05eaae7700 1 -- 172.21.15.67:0/3299373111 shutdown_connections 2024-04-22T23:45:41.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.807+0000 7f05eaae7700 1 --2- 172.21.15.67:0/3299373111 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05e40ffb40 0x7f05e40fffc0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:41.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.807+0000 7f05eaae7700 1 --2- 172.21.15.67:0/3299373111 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f05e4105bf0 0x7f05e4105fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:41.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.807+0000 7f05eaae7700 1 -- 172.21.15.67:0/3299373111 >> 172.21.15.67:0/3299373111 conn(0x7f05e40fb610 msgr2=0x7f05e40fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:45:41.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.807+0000 7f05eaae7700 1 -- 172.21.15.67:0/3299373111 shutdown_connections 2024-04-22T23:45:41.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.807+0000 7f05eaae7700 1 -- 172.21.15.67:0/3299373111 wait complete. 2024-04-22T23:45:41.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.808+0000 7f05eaae7700 1 Processor -- start 2024-04-22T23:45:41.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.808+0000 7f05eaae7700 1 -- start start 2024-04-22T23:45:41.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.808+0000 7f05eaae7700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f05e40ffb40 0x7f05e419c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:41.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.808+0000 7f05eaae7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05e4105bf0 0x7f05e419cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:41.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.808+0000 7f05eaae7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05e419d1d0 con 0x7f05e4105bf0 2024-04-22T23:45:41.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.808+0000 7f05eaae7700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05e4196680 con 0x7f05e40ffb40 2024-04-22T23:45:41.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.808+0000 7f05e3fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05e4105bf0 0x7f05e419cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:41.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.808+0000 7f05e3fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05e4105bf0 0x7f05e419cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50558/0 (socket says 172.21.15.67:50558) 2024-04-22T23:45:41.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.808+0000 7f05e3fff700 1 -- 172.21.15.67:0/664906 learned_addr learned my addr 172.21.15.67:0/664906 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:45:41.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.808+0000 7f05e3fff700 1 -- 172.21.15.67:0/664906 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f05e40ffb40 msgr2=0x7f05e419c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:41.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.808+0000 7f05e8883700 1 --2- 172.21.15.67:0/664906 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f05e40ffb40 0x7f05e419c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:41.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.808+0000 7f05e3fff700 1 --2- 172.21.15.67:0/664906 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f05e40ffb40 0x7f05e419c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:41.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.808+0000 7f05e3fff700 1 -- 172.21.15.67:0/664906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d00095f0 con 0x7f05e4105bf0 2024-04-22T23:45:41.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.808+0000 7f05e8883700 1 --2- 172.21.15.67:0/664906 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f05e40ffb40 0x7f05e419c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:45:41.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.809+0000 7f05e3fff700 1 --2- 172.21.15.67:0/664906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05e4105bf0 0x7f05e419cb40 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f05d00092e0 tx=0x7f05d000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:45:41.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.809+0000 7f05e1ffb700 1 -- 172.21.15.67:0/664906 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05d001d070 con 0x7f05e4105bf0 2024-04-22T23:45:41.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.809+0000 7f05e1ffb700 1 -- 172.21.15.67:0/664906 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f05d0005e00 con 0x7f05e4105bf0 2024-04-22T23:45:41.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.809+0000 7f05eaae7700 1 -- 172.21.15.67:0/664906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05e4196900 con 0x7f05e4105bf0 2024-04-22T23:45:41.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.809+0000 7f05e1ffb700 1 -- 172.21.15.67:0/664906 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05d0017400 con 0x7f05e4105bf0 2024-04-22T23:45:41.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.809+0000 7f05eaae7700 1 -- 172.21.15.67:0/664906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05e4196df0 con 0x7f05e4105bf0 2024-04-22T23:45:41.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.810+0000 7f05eaae7700 1 -- 172.21.15.67:0/664906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05e40679c0 con 0x7f05e4105bf0 2024-04-22T23:45:41.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.811+0000 7f05e1ffb700 1 -- 172.21.15.67:0/664906 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f05d000f460 con 0x7f05e4105bf0 2024-04-22T23:45:41.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.811+0000 7f05e1ffb700 1 --2- 172.21.15.67:0/664906 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f05d406ecc0 0x7f05d4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:45:41.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.811+0000 7f05e1ffb700 1 -- 172.21.15.67:0/664906 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f05d0090400 con 0x7f05e4105bf0 2024-04-22T23:45:41.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.811+0000 7f05e8883700 1 --2- 172.21.15.67:0/664906 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f05d406ecc0 0x7f05d4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:45:41.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.812+0000 7f05e8883700 1 --2- 172.21.15.67:0/664906 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f05d406ecc0 0x7f05d4071180 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f05d8005fd0 tx=0x7f05d8005ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:45:41.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:41.816+0000 7f05e1ffb700 1 -- 172.21.15.67:0/664906 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f05d005cbf0 con 0x7f05e4105bf0 2024-04-22T23:45:41.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:41 smithi134 ceph-mon[85630]: from='client.35076 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:45:41.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:41 smithi134 ceph-mon[85630]: pgmap v579: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:41.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:41 smithi134 ceph-mon[85630]: from='client.35080 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:45:41.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:41 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2762801326' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:45:41.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:45:41.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:41 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1219107710' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:45:42.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:42.119+0000 7f05eaae7700 1 -- 172.21.15.67:0/664906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f05e404f2e0 con 0x7f05e4105bf0 2024-04-22T23:45:42.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:42.119+0000 7f05e1ffb700 1 -- 172.21.15.67:0/664906 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f05d005c780 con 0x7f05e4105bf0 2024-04-22T23:45:42.111 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:45:42.111 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:45:42.111 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:45:42.111 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:45:42.111 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:45:42.111 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:45:42.112 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:45:42.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:42.121+0000 7f05eaae7700 1 -- 172.21.15.67:0/664906 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f05d406ecc0 msgr2=0x7f05d4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:42.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:42.121+0000 7f05eaae7700 1 --2- 172.21.15.67:0/664906 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f05d406ecc0 0x7f05d4071180 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f05d8005fd0 tx=0x7f05d8005ee0 comp rx=0 tx=0).stop 2024-04-22T23:45:42.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:42.121+0000 7f05eaae7700 1 -- 172.21.15.67:0/664906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05e4105bf0 msgr2=0x7f05e419cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:45:42.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:42.121+0000 7f05eaae7700 1 --2- 172.21.15.67:0/664906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05e4105bf0 0x7f05e419cb40 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f05d00092e0 tx=0x7f05d000be20 comp rx=0 tx=0).stop 2024-04-22T23:45:42.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:42.121+0000 7f05eaae7700 1 -- 172.21.15.67:0/664906 shutdown_connections 2024-04-22T23:45:42.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:42.121+0000 7f05eaae7700 1 --2- 172.21.15.67:0/664906 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f05d406ecc0 0x7f05d4071180 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:42.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:42.121+0000 7f05eaae7700 1 --2- 172.21.15.67:0/664906 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f05e40ffb40 0x7f05e419c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:42.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:42.121+0000 7f05eaae7700 1 --2- 172.21.15.67:0/664906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05e4105bf0 0x7f05e419cb40 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:45:42.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:42.121+0000 7f05eaae7700 1 -- 172.21.15.67:0/664906 >> 172.21.15.67:0/664906 conn(0x7f05e40fb610 msgr2=0x7f05e4068e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:45:42.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:42.121+0000 7f05eaae7700 1 -- 172.21.15.67:0/664906 shutdown_connections 2024-04-22T23:45:42.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:45:42.122+0000 7f05eaae7700 1 -- 172.21.15.67:0/664906 wait complete. 2024-04-22T23:45:42.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:42 smithi134 ceph-mon[85630]: from='client.35092 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:45:42.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:42 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/664906' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:45:43.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:42 smithi067 ceph-mon[112438]: from='client.35092 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:45:43.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:42 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/664906' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:45:43.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:43 smithi134 ceph-mon[85630]: pgmap v580: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:44.035 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:43 smithi067 ceph-mon[112438]: pgmap v580: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:45.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:45 smithi134 ceph-mon[85630]: pgmap v581: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:46.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:45 smithi067 ceph-mon[112438]: pgmap v581: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:47.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:47 smithi134 ceph-mon[85630]: pgmap v582: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:48.035 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:47 smithi067 ceph-mon[112438]: pgmap v582: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:49.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:49 smithi134 ceph-mon[85630]: pgmap v583: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:50.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:49 smithi067 ceph-mon[112438]: pgmap v583: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:51.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:51 smithi134 ceph-mon[85630]: pgmap v584: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:52.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:51 smithi067 ceph-mon[112438]: pgmap v584: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:53.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:53 smithi134 ceph-mon[85630]: pgmap v585: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:54.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:53 smithi067 ceph-mon[112438]: pgmap v585: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:55.908 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:55 smithi067 ceph-mon[112438]: pgmap v586: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:55.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:55 smithi134 ceph-mon[85630]: pgmap v586: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:56.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:45:57.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:45:57.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:57 smithi134 ceph-mon[85630]: pgmap v587: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:58.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:57 smithi067 ceph-mon[112438]: pgmap v587: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:45:59.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:45:59 smithi134 ceph-mon[85630]: pgmap v588: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:00.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:45:59 smithi067 ceph-mon[112438]: pgmap v588: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:01.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:01 smithi134 ceph-mon[85630]: pgmap v589: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:02.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:01 smithi067 ceph-mon[112438]: pgmap v589: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:03.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:03 smithi134 ceph-mon[85630]: pgmap v590: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:04.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:03 smithi067 ceph-mon[112438]: pgmap v590: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:05.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:05 smithi134 ceph-mon[85630]: pgmap v591: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:06.016 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:05 smithi067 ceph-mon[112438]: pgmap v591: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:07.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:07 smithi134 ceph-mon[85630]: pgmap v592: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:08.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:07 smithi067 ceph-mon[112438]: pgmap v592: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:09.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:09 smithi134 ceph-mon[85630]: pgmap v593: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:10.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:09 smithi067 ceph-mon[112438]: pgmap v593: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:11.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:11 smithi134 ceph-mon[85630]: pgmap v594: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:11.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:46:12.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:11 smithi067 ceph-mon[112438]: pgmap v594: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:12.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:46:12.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.243+0000 7f1a33bee700 1 -- 172.21.15.67:0/786707423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a2c101850 msgr2=0x7f1a2c1058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:12.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.243+0000 7f1a33bee700 1 --2- 172.21.15.67:0/786707423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a2c101850 0x7f1a2c1058a0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c009910 tx=0x7f1a1c009c20 comp rx=0 tx=0).stop 2024-04-22T23:46:12.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.243+0000 7f1a33bee700 1 -- 172.21.15.67:0/786707423 shutdown_connections 2024-04-22T23:46:12.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.243+0000 7f1a33bee700 1 --2- 172.21.15.67:0/786707423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a2c101850 0x7f1a2c1058a0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:12.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.243+0000 7f1a33bee700 1 --2- 172.21.15.67:0/786707423 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a2c100ea0 0x7f1a2c101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:12.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.243+0000 7f1a33bee700 1 -- 172.21.15.67:0/786707423 >> 172.21.15.67:0/786707423 conn(0x7f1a2c0fc750 msgr2=0x7f1a2c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:46:12.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.243+0000 7f1a33bee700 1 -- 172.21.15.67:0/786707423 shutdown_connections 2024-04-22T23:46:12.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.243+0000 7f1a33bee700 1 -- 172.21.15.67:0/786707423 wait complete. 2024-04-22T23:46:12.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.244+0000 7f1a33bee700 1 Processor -- start 2024-04-22T23:46:12.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.244+0000 7f1a33bee700 1 -- start start 2024-04-22T23:46:12.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.244+0000 7f1a33bee700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a2c100ea0 0x7f1a2c19c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:12.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.244+0000 7f1a33bee700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a2c101850 0x7f1a2c19cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:12.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.244+0000 7f1a33bee700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2c19d2d0 con 0x7f1a2c100ea0 2024-04-22T23:46:12.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.244+0000 7f1a33bee700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2c196780 con 0x7f1a2c101850 2024-04-22T23:46:12.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.244+0000 7f1a3198a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a2c100ea0 0x7f1a2c19c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:12.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.244+0000 7f1a3198a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a2c100ea0 0x7f1a2c19c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45058/0 (socket says 172.21.15.67:45058) 2024-04-22T23:46:12.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.244+0000 7f1a3198a700 1 -- 172.21.15.67:0/3306914461 learned_addr learned my addr 172.21.15.67:0/3306914461 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:46:12.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.244+0000 7f1a31189700 1 --2- 172.21.15.67:0/3306914461 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a2c101850 0x7f1a2c19cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:12.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.245+0000 7f1a3198a700 1 -- 172.21.15.67:0/3306914461 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a2c101850 msgr2=0x7f1a2c19cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:12.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.245+0000 7f1a3198a700 1 --2- 172.21.15.67:0/3306914461 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a2c101850 0x7f1a2c19cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:12.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.245+0000 7f1a3198a700 1 -- 172.21.15.67:0/3306914461 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a1c0095f0 con 0x7f1a2c100ea0 2024-04-22T23:46:12.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.245+0000 7f1a31189700 1 --2- 172.21.15.67:0/3306914461 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a2c101850 0x7f1a2c19cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:46:12.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.245+0000 7f1a3198a700 1 --2- 172.21.15.67:0/3306914461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a2c100ea0 0x7f1a2c19c700 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f1a2800d8d0 tx=0x7f1a2800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:46:12.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.245+0000 7f1a22ffd700 1 -- 172.21.15.67:0/3306914461 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a28009940 con 0x7f1a2c100ea0 2024-04-22T23:46:12.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.245+0000 7f1a22ffd700 1 -- 172.21.15.67:0/3306914461 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1a28010460 con 0x7f1a2c100ea0 2024-04-22T23:46:12.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.245+0000 7f1a33bee700 1 -- 172.21.15.67:0/3306914461 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a2c196a60 con 0x7f1a2c100ea0 2024-04-22T23:46:12.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.245+0000 7f1a22ffd700 1 -- 172.21.15.67:0/3306914461 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a2800f5f0 con 0x7f1a2c100ea0 2024-04-22T23:46:12.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.245+0000 7f1a33bee700 1 -- 172.21.15.67:0/3306914461 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a2c196fb0 con 0x7f1a2c100ea0 2024-04-22T23:46:12.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.246+0000 7f1a33bee700 1 -- 172.21.15.67:0/3306914461 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a2c1092a0 con 0x7f1a2c100ea0 2024-04-22T23:46:12.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.247+0000 7f1a22ffd700 1 -- 172.21.15.67:0/3306914461 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1a280105d0 con 0x7f1a2c100ea0 2024-04-22T23:46:12.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.247+0000 7f1a22ffd700 1 --2- 172.21.15.67:0/3306914461 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1a1806ecc0 0x7f1a18071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:12.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.247+0000 7f1a22ffd700 1 -- 172.21.15.67:0/3306914461 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f1a2808eee0 con 0x7f1a2c100ea0 2024-04-22T23:46:12.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.247+0000 7f1a31189700 1 --2- 172.21.15.67:0/3306914461 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1a1806ecc0 0x7f1a18071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:12.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.248+0000 7f1a31189700 1 --2- 172.21.15.67:0/3306914461 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1a1806ecc0 0x7f1a18071180 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c197bf0 tx=0x7f1a1c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:46:12.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.252+0000 7f1a22ffd700 1 -- 172.21.15.67:0/3306914461 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1a2805b6d0 con 0x7f1a2c100ea0 2024-04-22T23:46:12.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.485+0000 7f1a33bee700 1 -- 172.21.15.67:0/3306914461 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1a2c197da0 con 0x7f1a1806ecc0 2024-04-22T23:46:12.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.486+0000 7f1a22ffd700 1 -- 172.21.15.67:0/3306914461 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f1a2c197da0 con 0x7f1a1806ecc0 2024-04-22T23:46:12.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.488+0000 7f1a33bee700 1 -- 172.21.15.67:0/3306914461 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1a1806ecc0 msgr2=0x7f1a18071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:12.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.488+0000 7f1a33bee700 1 --2- 172.21.15.67:0/3306914461 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1a1806ecc0 0x7f1a18071180 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c197bf0 tx=0x7f1a1c01a040 comp rx=0 tx=0).stop 2024-04-22T23:46:12.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.488+0000 7f1a33bee700 1 -- 172.21.15.67:0/3306914461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a2c100ea0 msgr2=0x7f1a2c19c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:12.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.488+0000 7f1a33bee700 1 --2- 172.21.15.67:0/3306914461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a2c100ea0 0x7f1a2c19c700 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f1a2800d8d0 tx=0x7f1a2800dc90 comp rx=0 tx=0).stop 2024-04-22T23:46:12.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.488+0000 7f1a33bee700 1 -- 172.21.15.67:0/3306914461 shutdown_connections 2024-04-22T23:46:12.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.488+0000 7f1a33bee700 1 --2- 172.21.15.67:0/3306914461 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1a1806ecc0 0x7f1a18071180 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:12.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.488+0000 7f1a33bee700 1 --2- 172.21.15.67:0/3306914461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a2c100ea0 0x7f1a2c19c700 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:12.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.488+0000 7f1a33bee700 1 --2- 172.21.15.67:0/3306914461 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a2c101850 0x7f1a2c19cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:12.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.488+0000 7f1a33bee700 1 -- 172.21.15.67:0/3306914461 >> 172.21.15.67:0/3306914461 conn(0x7f1a2c0fc750 msgr2=0x7f1a2c0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:46:12.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.488+0000 7f1a33bee700 1 -- 172.21.15.67:0/3306914461 shutdown_connections 2024-04-22T23:46:12.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.488+0000 7f1a33bee700 1 -- 172.21.15.67:0/3306914461 wait complete. 2024-04-22T23:46:12.490 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:46:12.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.596+0000 7fb02d7e1700 1 -- 172.21.15.67:0/2160145269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb028100750 msgr2=0x7fb0281047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:12.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.596+0000 7fb02d7e1700 1 --2- 172.21.15.67:0/2160145269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb028100750 0x7fb0281047a0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fb018009910 tx=0x7fb018009c20 comp rx=0 tx=0).stop 2024-04-22T23:46:12.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.596+0000 7fb02d7e1700 1 -- 172.21.15.67:0/2160145269 shutdown_connections 2024-04-22T23:46:12.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.596+0000 7fb02d7e1700 1 --2- 172.21.15.67:0/2160145269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb028100750 0x7fb0281047a0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:12.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.596+0000 7fb02d7e1700 1 --2- 172.21.15.67:0/2160145269 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb0280ffda0 0x7fb028100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:12.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.596+0000 7fb02d7e1700 1 -- 172.21.15.67:0/2160145269 >> 172.21.15.67:0/2160145269 conn(0x7fb0280fb650 msgr2=0x7fb0280fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:46:12.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.596+0000 7fb02d7e1700 1 -- 172.21.15.67:0/2160145269 shutdown_connections 2024-04-22T23:46:12.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.596+0000 7fb02d7e1700 1 -- 172.21.15.67:0/2160145269 wait complete. 2024-04-22T23:46:12.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.597+0000 7fb02d7e1700 1 Processor -- start 2024-04-22T23:46:12.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.597+0000 7fb02d7e1700 1 -- start start 2024-04-22T23:46:12.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.597+0000 7fb02d7e1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0280ffda0 0x7fb02819c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:12.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.597+0000 7fb02d7e1700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb028100750 0x7fb02819cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:12.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.597+0000 7fb02d7e1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02819d160 con 0x7fb0280ffda0 2024-04-22T23:46:12.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.597+0000 7fb02d7e1700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0281966f0 con 0x7fb028100750 2024-04-22T23:46:12.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.598+0000 7fb026ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0280ffda0 0x7fb02819c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:12.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.598+0000 7fb026ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0280ffda0 0x7fb02819c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45074/0 (socket says 172.21.15.67:45074) 2024-04-22T23:46:12.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.598+0000 7fb026ffd700 1 -- 172.21.15.67:0/2990042867 learned_addr learned my addr 172.21.15.67:0/2990042867 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:46:12.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.598+0000 7fb0267fc700 1 --2- 172.21.15.67:0/2990042867 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb028100750 0x7fb02819cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:12.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.598+0000 7fb026ffd700 1 -- 172.21.15.67:0/2990042867 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb028100750 msgr2=0x7fb02819cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:12.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.598+0000 7fb026ffd700 1 --2- 172.21.15.67:0/2990042867 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb028100750 0x7fb02819cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:12.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.598+0000 7fb026ffd700 1 -- 172.21.15.67:0/2990042867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0180095f0 con 0x7fb0280ffda0 2024-04-22T23:46:12.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.598+0000 7fb0267fc700 1 --2- 172.21.15.67:0/2990042867 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb028100750 0x7fb02819cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:46:12.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.598+0000 7fb026ffd700 1 --2- 172.21.15.67:0/2990042867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0280ffda0 0x7fb02819c600 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fb01000d8d0 tx=0x7fb01000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:46:12.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.598+0000 7fb01ffff700 1 -- 172.21.15.67:0/2990042867 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb010009940 con 0x7fb0280ffda0 2024-04-22T23:46:12.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.598+0000 7fb02d7e1700 1 -- 172.21.15.67:0/2990042867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0281969d0 con 0x7fb0280ffda0 2024-04-22T23:46:12.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.598+0000 7fb01ffff700 1 -- 172.21.15.67:0/2990042867 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb010010460 con 0x7fb0280ffda0 2024-04-22T23:46:12.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.598+0000 7fb01ffff700 1 -- 172.21.15.67:0/2990042867 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb01000f5f0 con 0x7fb0280ffda0 2024-04-22T23:46:12.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.598+0000 7fb02d7e1700 1 -- 172.21.15.67:0/2990042867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb028196f20 con 0x7fb0280ffda0 2024-04-22T23:46:12.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.599+0000 7fb02d7e1700 1 -- 172.21.15.67:0/2990042867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0281091b0 con 0x7fb0280ffda0 2024-04-22T23:46:12.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.600+0000 7fb01ffff700 1 -- 172.21.15.67:0/2990042867 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb0100105d0 con 0x7fb0280ffda0 2024-04-22T23:46:12.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.600+0000 7fb01ffff700 1 --2- 172.21.15.67:0/2990042867 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb01406ecc0 0x7fb014071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:12.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.600+0000 7fb01ffff700 1 -- 172.21.15.67:0/2990042867 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fb01008eff0 con 0x7fb0280ffda0 2024-04-22T23:46:12.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.600+0000 7fb0267fc700 1 --2- 172.21.15.67:0/2990042867 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb01406ecc0 0x7fb014071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:12.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.601+0000 7fb0267fc700 1 --2- 172.21.15.67:0/2990042867 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb01406ecc0 0x7fb014071180 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb01800b5c0 tx=0x7fb0180053a0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:46:12.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.605+0000 7fb01ffff700 1 -- 172.21.15.67:0/2990042867 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb01005b760 con 0x7fb0280ffda0 2024-04-22T23:46:12.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.835+0000 7fb02d7e1700 1 -- 172.21.15.67:0/2990042867 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb028197140 con 0x7fb01406ecc0 2024-04-22T23:46:12.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.836+0000 7fb01ffff700 1 -- 172.21.15.67:0/2990042867 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb028197140 con 0x7fb01406ecc0 2024-04-22T23:46:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.838+0000 7fb02d7e1700 1 -- 172.21.15.67:0/2990042867 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb01406ecc0 msgr2=0x7fb014071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.838+0000 7fb02d7e1700 1 --2- 172.21.15.67:0/2990042867 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb01406ecc0 0x7fb014071180 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb01800b5c0 tx=0x7fb0180053a0 comp rx=0 tx=0).stop 2024-04-22T23:46:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.838+0000 7fb02d7e1700 1 -- 172.21.15.67:0/2990042867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0280ffda0 msgr2=0x7fb02819c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.838+0000 7fb02d7e1700 1 --2- 172.21.15.67:0/2990042867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0280ffda0 0x7fb02819c600 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fb01000d8d0 tx=0x7fb01000dc90 comp rx=0 tx=0).stop 2024-04-22T23:46:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.838+0000 7fb02d7e1700 1 -- 172.21.15.67:0/2990042867 shutdown_connections 2024-04-22T23:46:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.838+0000 7fb02d7e1700 1 --2- 172.21.15.67:0/2990042867 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb01406ecc0 0x7fb014071180 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.838+0000 7fb02d7e1700 1 --2- 172.21.15.67:0/2990042867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0280ffda0 0x7fb02819c600 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.838+0000 7fb02d7e1700 1 --2- 172.21.15.67:0/2990042867 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb028100750 0x7fb02819cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.838+0000 7fb02d7e1700 1 -- 172.21.15.67:0/2990042867 >> 172.21.15.67:0/2990042867 conn(0x7fb0280fb650 msgr2=0x7fb028069150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:46:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.839+0000 7fb02d7e1700 1 -- 172.21.15.67:0/2990042867 shutdown_connections 2024-04-22T23:46:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.839+0000 7fb02d7e1700 1 -- 172.21.15.67:0/2990042867 wait complete. 2024-04-22T23:46:12.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.945+0000 7f909f164700 1 -- 172.21.15.67:0/216511039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9098105bd0 msgr2=0x7f9098105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:12.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.945+0000 7f909f164700 1 --2- 172.21.15.67:0/216511039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9098105bd0 0x7f9098105fb0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f908c009910 tx=0x7f908c009c20 comp rx=0 tx=0).stop 2024-04-22T23:46:12.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.945+0000 7f909f164700 1 -- 172.21.15.67:0/216511039 shutdown_connections 2024-04-22T23:46:12.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.945+0000 7f909f164700 1 --2- 172.21.15.67:0/216511039 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f90980ffc20 0x7f9098100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:12.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.945+0000 7f909f164700 1 --2- 172.21.15.67:0/216511039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9098105bd0 0x7f9098105fb0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:12.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.945+0000 7f909f164700 1 -- 172.21.15.67:0/216511039 >> 172.21.15.67:0/216511039 conn(0x7f9098075670 msgr2=0x7f9098075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:46:12.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.946+0000 7f909f164700 1 -- 172.21.15.67:0/216511039 shutdown_connections 2024-04-22T23:46:12.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.946+0000 7f909f164700 1 -- 172.21.15.67:0/216511039 wait complete. 2024-04-22T23:46:12.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.946+0000 7f909f164700 1 Processor -- start 2024-04-22T23:46:12.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.946+0000 7f909f164700 1 -- start start 2024-04-22T23:46:12.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.946+0000 7f909f164700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90980ffc20 0x7f909819c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:12.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.946+0000 7f909f164700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9098105bd0 0x7f909819cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:12.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.946+0000 7f909f164700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909819d1f0 con 0x7f90980ffc20 2024-04-22T23:46:12.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.946+0000 7f909f164700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90981966a0 con 0x7f9098105bd0 2024-04-22T23:46:12.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.947+0000 7f909cf00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90980ffc20 0x7f909819c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:12.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.947+0000 7f909cf00700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90980ffc20 0x7f909819c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45084/0 (socket says 172.21.15.67:45084) 2024-04-22T23:46:12.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.947+0000 7f909cf00700 1 -- 172.21.15.67:0/3445314953 learned_addr learned my addr 172.21.15.67:0/3445314953 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:46:12.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.947+0000 7f9097fff700 1 --2- 172.21.15.67:0/3445314953 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9098105bd0 0x7f909819cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:12.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.947+0000 7f909cf00700 1 -- 172.21.15.67:0/3445314953 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9098105bd0 msgr2=0x7f909819cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:12.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.947+0000 7f909cf00700 1 --2- 172.21.15.67:0/3445314953 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9098105bd0 0x7f909819cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:12.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.947+0000 7f909cf00700 1 -- 172.21.15.67:0/3445314953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f908c0095f0 con 0x7f90980ffc20 2024-04-22T23:46:12.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.947+0000 7f909cf00700 1 --2- 172.21.15.67:0/3445314953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90980ffc20 0x7f909819c620 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f908c006010 tx=0x7f908c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:46:12.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.947+0000 7f9095ffb700 1 -- 172.21.15.67:0/3445314953 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f908c01d070 con 0x7f90980ffc20 2024-04-22T23:46:12.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.947+0000 7f9095ffb700 1 -- 172.21.15.67:0/3445314953 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f908c004fb0 con 0x7f90980ffc20 2024-04-22T23:46:12.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.947+0000 7f9095ffb700 1 -- 172.21.15.67:0/3445314953 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f908c00f690 con 0x7f90980ffc20 2024-04-22T23:46:12.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.947+0000 7f909f164700 1 -- 172.21.15.67:0/3445314953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9098196920 con 0x7f90980ffc20 2024-04-22T23:46:12.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.947+0000 7f909f164700 1 -- 172.21.15.67:0/3445314953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9098196e10 con 0x7f90980ffc20 2024-04-22T23:46:12.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.948+0000 7f909f164700 1 -- 172.21.15.67:0/3445314953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f909804f2e0 con 0x7f90980ffc20 2024-04-22T23:46:12.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.949+0000 7f9095ffb700 1 -- 172.21.15.67:0/3445314953 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f908c0228a0 con 0x7f90980ffc20 2024-04-22T23:46:12.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.949+0000 7f9095ffb700 1 --2- 172.21.15.67:0/3445314953 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f908806ecc0 0x7f9088071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:12.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.950+0000 7f9095ffb700 1 -- 172.21.15.67:0/3445314953 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f908c0907d0 con 0x7f90980ffc20 2024-04-22T23:46:12.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.950+0000 7f9097fff700 1 --2- 172.21.15.67:0/3445314953 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f908806ecc0 0x7f9088071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:12.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.950+0000 7f9097fff700 1 --2- 172.21.15.67:0/3445314953 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f908806ecc0 0x7f9088071180 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9084005fd0 tx=0x7f908400a560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:46:12.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:12.954+0000 7f9095ffb700 1 -- 172.21.15.67:0/3445314953 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f908c05cfc0 con 0x7f90980ffc20 2024-04-22T23:46:13.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.172+0000 7f909f164700 1 -- 172.21.15.67:0/3445314953 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9098197120 con 0x7f908806ecc0 2024-04-22T23:46:13.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.183+0000 7f9095ffb700 1 -- 172.21.15.67:0/3445314953 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f9098197120 con 0x7f908806ecc0 2024-04-22T23:46:13.175 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:46:13.175 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (19m) 13m ago 31m 25.5M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:46:13.175 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (31m) 13m ago 31m 9114k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:46:13.175 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (30m) 12m ago 30m 9042k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:46:13.175 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (17m) 13m ago 31m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:46:13.175 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (17m) 12m ago 30m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:46:13.175 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (19m) 13m ago 30m 79.3M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:46:13.176 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (13m) 13m ago 25m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:46:13.176 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (13m) 13m ago 25m 12.6M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:46:13.176 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (12m) 12m ago 25m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:46:13.176 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (25m) 12m ago 25m 12.2M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:46:13.176 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (21m) 13m ago 32m 511M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:46:13.176 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (20m) 12m ago 30m 453M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:46:13.176 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (18m) 13m ago 32m 55.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:46:13.176 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (18m) 12m ago 30m 46.0M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:46:13.176 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (20m) 13m ago 31m 15.4M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:46:13.176 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (20m) 12m ago 30m 16.1M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:46:13.176 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (16m) 13m ago 29m 99.2M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:46:13.176 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (16m) 13m ago 28m 127M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:46:13.176 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (15m) 13m ago 28m 79.9M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:46:13.176 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (15m) 12m ago 28m 97.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:46:13.176 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (14m) 12m ago 27m 82.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:46:13.177 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (13m) 12m ago 27m 77.6M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:46:13.177 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (20m) 13m ago 30m 73.9M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:46:13.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.186+0000 7f909f164700 1 -- 172.21.15.67:0/3445314953 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f908806ecc0 msgr2=0x7f9088071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:13.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.186+0000 7f909f164700 1 --2- 172.21.15.67:0/3445314953 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f908806ecc0 0x7f9088071180 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9084005fd0 tx=0x7f908400a560 comp rx=0 tx=0).stop 2024-04-22T23:46:13.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.186+0000 7f909f164700 1 -- 172.21.15.67:0/3445314953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90980ffc20 msgr2=0x7f909819c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:13.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.186+0000 7f909f164700 1 --2- 172.21.15.67:0/3445314953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90980ffc20 0x7f909819c620 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f908c006010 tx=0x7f908c00be20 comp rx=0 tx=0).stop 2024-04-22T23:46:13.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.186+0000 7f909f164700 1 -- 172.21.15.67:0/3445314953 shutdown_connections 2024-04-22T23:46:13.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.186+0000 7f909f164700 1 --2- 172.21.15.67:0/3445314953 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f908806ecc0 0x7f9088071180 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:13.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.186+0000 7f909f164700 1 --2- 172.21.15.67:0/3445314953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90980ffc20 0x7f909819c620 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:13.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.186+0000 7f909f164700 1 --2- 172.21.15.67:0/3445314953 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9098105bd0 0x7f909819cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:13.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.186+0000 7f909f164700 1 -- 172.21.15.67:0/3445314953 >> 172.21.15.67:0/3445314953 conn(0x7f9098075670 msgr2=0x7f9098068680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:46:13.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.186+0000 7f909f164700 1 -- 172.21.15.67:0/3445314953 shutdown_connections 2024-04-22T23:46:13.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.186+0000 7f909f164700 1 -- 172.21.15.67:0/3445314953 wait complete. 2024-04-22T23:46:13.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.291+0000 7f81ae5d7700 1 -- 172.21.15.67:0/610693632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81a8101750 msgr2=0x7f81a81058b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:13.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.291+0000 7f81ae5d7700 1 --2- 172.21.15.67:0/610693632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81a8101750 0x7f81a81058b0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f8198009910 tx=0x7f8198009c20 comp rx=0 tx=0).stop 2024-04-22T23:46:13.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.291+0000 7f81ae5d7700 1 -- 172.21.15.67:0/610693632 shutdown_connections 2024-04-22T23:46:13.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.291+0000 7f81ae5d7700 1 --2- 172.21.15.67:0/610693632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81a8101750 0x7f81a81058b0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:13.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.291+0000 7f81ae5d7700 1 --2- 172.21.15.67:0/610693632 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f81a8100da0 0x7f81a8101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:13.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.291+0000 7f81ae5d7700 1 -- 172.21.15.67:0/610693632 >> 172.21.15.67:0/610693632 conn(0x7f81a80fc670 msgr2=0x7f81a80fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:46:13.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.291+0000 7f81ae5d7700 1 -- 172.21.15.67:0/610693632 shutdown_connections 2024-04-22T23:46:13.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.291+0000 7f81ae5d7700 1 -- 172.21.15.67:0/610693632 wait complete. 2024-04-22T23:46:13.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.292+0000 7f81ae5d7700 1 Processor -- start 2024-04-22T23:46:13.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.292+0000 7f81ae5d7700 1 -- start start 2024-04-22T23:46:13.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.292+0000 7f81ae5d7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81a8100da0 0x7f81a819c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:13.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.292+0000 7f81ae5d7700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f81a8101750 0x7f81a819cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:13.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.292+0000 7f81ae5d7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a819d160 con 0x7f81a8100da0 2024-04-22T23:46:13.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.292+0000 7f81ae5d7700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a81966f0 con 0x7f81a8101750 2024-04-22T23:46:13.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.292+0000 7f81a7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81a8100da0 0x7f81a819c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:13.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.292+0000 7f81a7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81a8100da0 0x7f81a819c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45108/0 (socket says 172.21.15.67:45108) 2024-04-22T23:46:13.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.292+0000 7f81a7fff700 1 -- 172.21.15.67:0/77702662 learned_addr learned my addr 172.21.15.67:0/77702662 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:46:13.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.292+0000 7f81a7fff700 1 -- 172.21.15.67:0/77702662 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f81a8101750 msgr2=0x7f81a819cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:13.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.292+0000 7f81a7fff700 1 --2- 172.21.15.67:0/77702662 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f81a8101750 0x7f81a819cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:13.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.292+0000 7f81a7fff700 1 -- 172.21.15.67:0/77702662 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81980095f0 con 0x7f81a8100da0 2024-04-22T23:46:13.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.293+0000 7f81a7fff700 1 --2- 172.21.15.67:0/77702662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81a8100da0 0x7f81a819c600 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f819000d8d0 tx=0x7f819000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:46:13.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.293+0000 7f81a57fa700 1 -- 172.21.15.67:0/77702662 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8190009940 con 0x7f81a8100da0 2024-04-22T23:46:13.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.293+0000 7f81a57fa700 1 -- 172.21.15.67:0/77702662 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8190010460 con 0x7f81a8100da0 2024-04-22T23:46:13.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.293+0000 7f81ae5d7700 1 -- 172.21.15.67:0/77702662 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81a81969d0 con 0x7f81a8100da0 2024-04-22T23:46:13.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.293+0000 7f81a57fa700 1 -- 172.21.15.67:0/77702662 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f819000f5f0 con 0x7f81a8100da0 2024-04-22T23:46:13.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.293+0000 7f81ae5d7700 1 -- 172.21.15.67:0/77702662 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81a8196f20 con 0x7f81a8100da0 2024-04-22T23:46:13.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.294+0000 7f81ae5d7700 1 -- 172.21.15.67:0/77702662 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81a81092b0 con 0x7f81a8100da0 2024-04-22T23:46:13.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.295+0000 7f81a57fa700 1 -- 172.21.15.67:0/77702662 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f81900105d0 con 0x7f81a8100da0 2024-04-22T23:46:13.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.295+0000 7f81a57fa700 1 --2- 172.21.15.67:0/77702662 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f819406ecc0 0x7f8194071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:13.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.295+0000 7f81a57fa700 1 -- 172.21.15.67:0/77702662 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f819008eff0 con 0x7f81a8100da0 2024-04-22T23:46:13.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.295+0000 7f81a77fe700 1 --2- 172.21.15.67:0/77702662 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f819406ecc0 0x7f8194071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:13.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.296+0000 7f81a77fe700 1 --2- 172.21.15.67:0/77702662 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f819406ecc0 0x7f8194071180 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f819800b5c0 tx=0x7f8198005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:46:13.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.300+0000 7f81a57fa700 1 -- 172.21.15.67:0/77702662 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f819005b760 con 0x7f81a8100da0 2024-04-22T23:46:13.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.601+0000 7f81ae5d7700 1 -- 172.21.15.67:0/77702662 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f81a8197ca0 con 0x7f81a8100da0 2024-04-22T23:46:13.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.601+0000 7f81a57fa700 1 -- 172.21.15.67:0/77702662 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f819005b2f0 con 0x7f81a8100da0 2024-04-22T23:46:13.593 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:46:13.593 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:46:13.593 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:46:13.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:46:13.594 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:46:13.594 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:46:13.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:46:13.594 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:46:13.594 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:46:13.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:46:13.594 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:46:13.594 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:46:13.594 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 3 2024-04-22T23:46:13.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:46:13.594 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:46:13.594 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:46:13.594 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 13 2024-04-22T23:46:13.595 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:46:13.595 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:46:13.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.603+0000 7f81ae5d7700 1 -- 172.21.15.67:0/77702662 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f819406ecc0 msgr2=0x7f8194071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:13.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.603+0000 7f81ae5d7700 1 --2- 172.21.15.67:0/77702662 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f819406ecc0 0x7f8194071180 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f819800b5c0 tx=0x7f8198005960 comp rx=0 tx=0).stop 2024-04-22T23:46:13.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.603+0000 7f81ae5d7700 1 -- 172.21.15.67:0/77702662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81a8100da0 msgr2=0x7f81a819c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:13.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.603+0000 7f81ae5d7700 1 --2- 172.21.15.67:0/77702662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81a8100da0 0x7f81a819c600 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f819000d8d0 tx=0x7f819000dc90 comp rx=0 tx=0).stop 2024-04-22T23:46:13.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.603+0000 7f81ae5d7700 1 -- 172.21.15.67:0/77702662 shutdown_connections 2024-04-22T23:46:13.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.603+0000 7f81ae5d7700 1 --2- 172.21.15.67:0/77702662 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f819406ecc0 0x7f8194071180 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:13.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.603+0000 7f81ae5d7700 1 --2- 172.21.15.67:0/77702662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81a8100da0 0x7f81a819c600 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:13.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.603+0000 7f81ae5d7700 1 --2- 172.21.15.67:0/77702662 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f81a8101750 0x7f81a819cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:13.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.603+0000 7f81ae5d7700 1 -- 172.21.15.67:0/77702662 >> 172.21.15.67:0/77702662 conn(0x7f81a80fc670 msgr2=0x7f81a80fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:46:13.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.603+0000 7f81ae5d7700 1 -- 172.21.15.67:0/77702662 shutdown_connections 2024-04-22T23:46:13.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.603+0000 7f81ae5d7700 1 -- 172.21.15.67:0/77702662 wait complete. 2024-04-22T23:46:13.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.711+0000 7fa5a7ffb700 1 -- 172.21.15.67:0/2790332144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5a0105bf0 msgr2=0x7fa5a0105fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:13.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.711+0000 7fa5a7ffb700 1 --2- 172.21.15.67:0/2790332144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5a0105bf0 0x7fa5a0105fd0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fa59c009910 tx=0x7fa59c009c20 comp rx=0 tx=0).stop 2024-04-22T23:46:13.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.711+0000 7fa5a7ffb700 1 -- 172.21.15.67:0/2790332144 shutdown_connections 2024-04-22T23:46:13.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.711+0000 7fa5a7ffb700 1 --2- 172.21.15.67:0/2790332144 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa5a00ffb40 0x7fa5a00fffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:13.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.711+0000 7fa5a7ffb700 1 --2- 172.21.15.67:0/2790332144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5a0105bf0 0x7fa5a0105fd0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:13.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.711+0000 7fa5a7ffb700 1 -- 172.21.15.67:0/2790332144 >> 172.21.15.67:0/2790332144 conn(0x7fa5a00fb610 msgr2=0x7fa5a00fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:46:13.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.711+0000 7fa5a7ffb700 1 -- 172.21.15.67:0/2790332144 shutdown_connections 2024-04-22T23:46:13.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.711+0000 7fa5a7ffb700 1 -- 172.21.15.67:0/2790332144 wait complete. 2024-04-22T23:46:13.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.711+0000 7fa5a7ffb700 1 Processor -- start 2024-04-22T23:46:13.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.711+0000 7fa5a7ffb700 1 -- start start 2024-04-22T23:46:13.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.712+0000 7fa5a7ffb700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa5a00ffb40 0x7fa5a01017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:13.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.712+0000 7fa5a7ffb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5a0105bf0 0x7fa5a0101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:13.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.712+0000 7fa5a7ffb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a0102230 con 0x7fa5a0105bf0 2024-04-22T23:46:13.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.712+0000 7fa5a7ffb700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a0102370 con 0x7fa5a00ffb40 2024-04-22T23:46:13.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.712+0000 7fa5a5596700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5a0105bf0 0x7fa5a0101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:13.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.712+0000 7fa5a5596700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5a0105bf0 0x7fa5a0101cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45116/0 (socket says 172.21.15.67:45116) 2024-04-22T23:46:13.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.712+0000 7fa5a5d97700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa5a00ffb40 0x7fa5a01017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:13.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.712+0000 7fa5a5596700 1 -- 172.21.15.67:0/926557342 learned_addr learned my addr 172.21.15.67:0/926557342 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:46:13.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.712+0000 7fa5a5596700 1 -- 172.21.15.67:0/926557342 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa5a00ffb40 msgr2=0x7fa5a01017b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:13.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.712+0000 7fa5a5596700 1 --2- 172.21.15.67:0/926557342 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa5a00ffb40 0x7fa5a01017b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:13.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.712+0000 7fa5a5596700 1 -- 172.21.15.67:0/926557342 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa59c0095f0 con 0x7fa5a0105bf0 2024-04-22T23:46:13.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.712+0000 7fa5a5596700 1 --2- 172.21.15.67:0/926557342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5a0105bf0 0x7fa5a0101cf0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fa59000c8a0 tx=0x7fa59000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:46:13.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.712+0000 7fa596ffd700 1 -- 172.21.15.67:0/926557342 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa5900078e0 con 0x7fa5a0105bf0 2024-04-22T23:46:13.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.712+0000 7fa5a7ffb700 1 -- 172.21.15.67:0/926557342 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5a01a2990 con 0x7fa5a0105bf0 2024-04-22T23:46:13.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.712+0000 7fa596ffd700 1 -- 172.21.15.67:0/926557342 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa59000f450 con 0x7fa5a0105bf0 2024-04-22T23:46:13.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.713+0000 7fa596ffd700 1 -- 172.21.15.67:0/926557342 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa59000e5e0 con 0x7fa5a0105bf0 2024-04-22T23:46:13.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.713+0000 7fa5a7ffb700 1 -- 172.21.15.67:0/926557342 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5a01a2da0 con 0x7fa5a0105bf0 2024-04-22T23:46:13.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.714+0000 7fa5a7ffb700 1 -- 172.21.15.67:0/926557342 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5a004f2e0 con 0x7fa5a0105bf0 2024-04-22T23:46:13.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.714+0000 7fa596ffd700 1 -- 172.21.15.67:0/926557342 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa59000f5c0 con 0x7fa5a0105bf0 2024-04-22T23:46:13.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.715+0000 7fa596ffd700 1 --2- 172.21.15.67:0/926557342 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa58c06ecc0 0x7fa58c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:13.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.715+0000 7fa596ffd700 1 -- 172.21.15.67:0/926557342 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fa59008efb0 con 0x7fa5a0105bf0 2024-04-22T23:46:13.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.715+0000 7fa5a5d97700 1 --2- 172.21.15.67:0/926557342 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa58c06ecc0 0x7fa58c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:13.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.715+0000 7fa5a5d97700 1 --2- 172.21.15.67:0/926557342 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa58c06ecc0 0x7fa58c071180 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa59c006010 tx=0x7fa59c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:46:13.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.719+0000 7fa596ffd700 1 -- 172.21.15.67:0/926557342 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fa590057bf0 con 0x7fa5a0105bf0 2024-04-22T23:46:13.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.974+0000 7fa5a7ffb700 1 -- 172.21.15.67:0/926557342 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa5a0102ce0 con 0x7fa5a0105bf0 2024-04-22T23:46:13.967 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:13 smithi067 ceph-mon[112438]: pgmap v595: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:13.967 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:13 smithi067 ceph-mon[112438]: from='client.35100 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:46:13.967 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:13 smithi067 ceph-mon[112438]: from='client.35104 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:46:13.967 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:13 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/77702662' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:46:13.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.975+0000 7fa596ffd700 1 -- 172.21.15.67:0/926557342 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1871 (secure 0 0 0) 0x7fa59005b2b0 con 0x7fa5a0105bf0 2024-04-22T23:46:13.968 INFO:teuthology.orchestra.run.smithi067.stdout:e31 2024-04-22T23:46:13.968 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:46:13.968 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:46:13.968 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:46:13.968 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:46:13.968 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:46:13.968 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:46:13.968 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:46:13.968 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:46:13.968 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:46:13.968 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:46:13.968 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:46:13.968 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:46:13.969 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:46:13.969 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:46:13.969 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:46:13.969 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:46:13.969 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:46:13.969 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:46:13.969 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:46:13.969 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:46:13.969 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:46:13.969 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:46:13.969 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:46:13.969 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:46:13.969 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:46:13.969 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:46:13.969 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:46:13.970 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:46:13.970 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:46:13.970 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:46:13.970 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:46:13.970 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:46:13.970 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:46:13.970 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:46:13.970 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:46:13.970 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:46:13.970 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:46:13.970 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:46:13.970 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:46:13.970 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44259} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1827288036,v1:172.21.15.134:6827/1827288036] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:46:13.970 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:46:13.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.977+0000 7fa5a7ffb700 1 -- 172.21.15.67:0/926557342 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa58c06ecc0 msgr2=0x7fa58c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:13.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.977+0000 7fa5a7ffb700 1 --2- 172.21.15.67:0/926557342 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa58c06ecc0 0x7fa58c071180 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa59c006010 tx=0x7fa59c01a040 comp rx=0 tx=0).stop 2024-04-22T23:46:13.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.977+0000 7fa5a7ffb700 1 -- 172.21.15.67:0/926557342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5a0105bf0 msgr2=0x7fa5a0101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:13.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.977+0000 7fa5a7ffb700 1 --2- 172.21.15.67:0/926557342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5a0105bf0 0x7fa5a0101cf0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fa59000c8a0 tx=0x7fa59000cbb0 comp rx=0 tx=0).stop 2024-04-22T23:46:13.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.977+0000 7fa5a7ffb700 1 -- 172.21.15.67:0/926557342 shutdown_connections 2024-04-22T23:46:13.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.977+0000 7fa5a7ffb700 1 --2- 172.21.15.67:0/926557342 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa58c06ecc0 0x7fa58c071180 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:13.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.977+0000 7fa5a7ffb700 1 --2- 172.21.15.67:0/926557342 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa5a00ffb40 0x7fa5a01017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:13.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.977+0000 7fa5a7ffb700 1 --2- 172.21.15.67:0/926557342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5a0105bf0 0x7fa5a0101cf0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:13.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.977+0000 7fa5a7ffb700 1 -- 172.21.15.67:0/926557342 >> 172.21.15.67:0/926557342 conn(0x7fa5a00fb610 msgr2=0x7fa5a0068ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:46:13.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.977+0000 7fa5a7ffb700 1 -- 172.21.15.67:0/926557342 shutdown_connections 2024-04-22T23:46:13.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:13.977+0000 7fa5a7ffb700 1 -- 172.21.15.67:0/926557342 wait complete. 2024-04-22T23:46:13.972 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 31 2024-04-22T23:46:13.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:13 smithi134 ceph-mon[85630]: pgmap v595: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:13.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:13 smithi134 ceph-mon[85630]: from='client.35100 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:46:13.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:13 smithi134 ceph-mon[85630]: from='client.35104 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:46:13.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:13 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/77702662' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:46:14.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.085+0000 7f15d71a1700 1 -- 172.21.15.67:0/1981843545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15d00fff50 msgr2=0x7f15d0105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:14.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.085+0000 7f15d71a1700 1 --2- 172.21.15.67:0/1981843545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15d00fff50 0x7f15d0105840 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f15c4009910 tx=0x7f15c4009c20 comp rx=0 tx=0).stop 2024-04-22T23:46:14.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.085+0000 7f15d71a1700 1 -- 172.21.15.67:0/1981843545 shutdown_connections 2024-04-22T23:46:14.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.085+0000 7f15d71a1700 1 --2- 172.21.15.67:0/1981843545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15d00fff50 0x7f15d0105840 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:14.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.085+0000 7f15d71a1700 1 --2- 172.21.15.67:0/1981843545 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15d00ff5a0 0x7f15d00ff980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:14.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.085+0000 7f15d71a1700 1 -- 172.21.15.67:0/1981843545 >> 172.21.15.67:0/1981843545 conn(0x7f15d0074e30 msgr2=0x7f15d0075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:46:14.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.085+0000 7f15d71a1700 1 -- 172.21.15.67:0/1981843545 shutdown_connections 2024-04-22T23:46:14.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.085+0000 7f15d71a1700 1 -- 172.21.15.67:0/1981843545 wait complete. 2024-04-22T23:46:14.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.085+0000 7f15d71a1700 1 Processor -- start 2024-04-22T23:46:14.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.086+0000 7f15d71a1700 1 -- start start 2024-04-22T23:46:14.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.086+0000 7f15d71a1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15d00ff5a0 0x7f15d019c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:14.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.086+0000 7f15d71a1700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15d00fff50 0x7f15d019cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:14.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.086+0000 7f15d71a1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d019d210 con 0x7f15d00ff5a0 2024-04-22T23:46:14.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.086+0000 7f15d71a1700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d01966c0 con 0x7f15d00fff50 2024-04-22T23:46:14.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.086+0000 7f15d4f3d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15d00ff5a0 0x7f15d019c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:14.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.086+0000 7f15d4f3d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15d00ff5a0 0x7f15d019c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45132/0 (socket says 172.21.15.67:45132) 2024-04-22T23:46:14.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.086+0000 7f15d4f3d700 1 -- 172.21.15.67:0/1678037973 learned_addr learned my addr 172.21.15.67:0/1678037973 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:46:14.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.086+0000 7f15cffff700 1 --2- 172.21.15.67:0/1678037973 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15d00fff50 0x7f15d019cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:14.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.086+0000 7f15d4f3d700 1 -- 172.21.15.67:0/1678037973 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15d00fff50 msgr2=0x7f15d019cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:14.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.086+0000 7f15d4f3d700 1 --2- 172.21.15.67:0/1678037973 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15d00fff50 0x7f15d019cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:14.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.086+0000 7f15d4f3d700 1 -- 172.21.15.67:0/1678037973 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c40095f0 con 0x7f15d00ff5a0 2024-04-22T23:46:14.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.086+0000 7f15d4f3d700 1 --2- 172.21.15.67:0/1678037973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15d00ff5a0 0x7f15d019c640 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f15c000d8d0 tx=0x7f15c000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:46:14.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.086+0000 7f15cdffb700 1 -- 172.21.15.67:0/1678037973 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15c0009940 con 0x7f15d00ff5a0 2024-04-22T23:46:14.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.087+0000 7f15cdffb700 1 -- 172.21.15.67:0/1678037973 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f15c0010460 con 0x7f15d00ff5a0 2024-04-22T23:46:14.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.087+0000 7f15d71a1700 1 -- 172.21.15.67:0/1678037973 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15d01969a0 con 0x7f15d00ff5a0 2024-04-22T23:46:14.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.087+0000 7f15cdffb700 1 -- 172.21.15.67:0/1678037973 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15c000f5f0 con 0x7f15d00ff5a0 2024-04-22T23:46:14.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.087+0000 7f15d71a1700 1 -- 172.21.15.67:0/1678037973 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15d0196ef0 con 0x7f15d00ff5a0 2024-04-22T23:46:14.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.088+0000 7f15d71a1700 1 -- 172.21.15.67:0/1678037973 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15d0109200 con 0x7f15d00ff5a0 2024-04-22T23:46:14.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.088+0000 7f15cdffb700 1 -- 172.21.15.67:0/1678037973 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f15c00105d0 con 0x7f15d00ff5a0 2024-04-22T23:46:14.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.088+0000 7f15cdffb700 1 --2- 172.21.15.67:0/1678037973 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f15b806ecc0 0x7f15b8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:14.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.089+0000 7f15cdffb700 1 -- 172.21.15.67:0/1678037973 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f15c008eee0 con 0x7f15d00ff5a0 2024-04-22T23:46:14.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.089+0000 7f15cffff700 1 --2- 172.21.15.67:0/1678037973 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f15b806ecc0 0x7f15b8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:14.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.089+0000 7f15cffff700 1 --2- 172.21.15.67:0/1678037973 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f15b806ecc0 0x7f15b8071180 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f15c4000c00 tx=0x7f15c4005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:46:14.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.093+0000 7f15cdffb700 1 -- 172.21.15.67:0/1678037973 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f15c005b6d0 con 0x7f15d00ff5a0 2024-04-22T23:46:14.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.321+0000 7f15d71a1700 1 -- 172.21.15.67:0/1678037973 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f15d0197160 con 0x7f15b806ecc0 2024-04-22T23:46:14.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.323+0000 7f15cdffb700 1 -- 172.21.15.67:0/1678037973 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f15d0197160 con 0x7f15b806ecc0 2024-04-22T23:46:14.315 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:46:14.315 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:46:14.315 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:46:14.315 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:46:14.315 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:46:14.315 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:46:14.315 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:46:14.315 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:46:14.315 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:46:14.315 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:46:14.315 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T23:46:14.315 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:46:14.316 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:46:14.316 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:46:14.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.325+0000 7f15d71a1700 1 -- 172.21.15.67:0/1678037973 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f15b806ecc0 msgr2=0x7f15b8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:14.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.325+0000 7f15d71a1700 1 --2- 172.21.15.67:0/1678037973 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f15b806ecc0 0x7f15b8071180 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f15c4000c00 tx=0x7f15c4005fd0 comp rx=0 tx=0).stop 2024-04-22T23:46:14.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.325+0000 7f15d71a1700 1 -- 172.21.15.67:0/1678037973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15d00ff5a0 msgr2=0x7f15d019c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:14.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.325+0000 7f15d71a1700 1 --2- 172.21.15.67:0/1678037973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15d00ff5a0 0x7f15d019c640 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f15c000d8d0 tx=0x7f15c000dc90 comp rx=0 tx=0).stop 2024-04-22T23:46:14.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.325+0000 7f15d71a1700 1 -- 172.21.15.67:0/1678037973 shutdown_connections 2024-04-22T23:46:14.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.325+0000 7f15d71a1700 1 --2- 172.21.15.67:0/1678037973 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f15b806ecc0 0x7f15b8071180 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:14.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.325+0000 7f15d71a1700 1 --2- 172.21.15.67:0/1678037973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15d00ff5a0 0x7f15d019c640 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:14.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.325+0000 7f15d71a1700 1 --2- 172.21.15.67:0/1678037973 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15d00fff50 0x7f15d019cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:14.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.325+0000 7f15d71a1700 1 -- 172.21.15.67:0/1678037973 >> 172.21.15.67:0/1678037973 conn(0x7f15d0074e30 msgr2=0x7f15d00fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:46:14.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.325+0000 7f15d71a1700 1 -- 172.21.15.67:0/1678037973 shutdown_connections 2024-04-22T23:46:14.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.325+0000 7f15d71a1700 1 -- 172.21.15.67:0/1678037973 wait complete. 2024-04-22T23:46:14.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.431+0000 7feb56c3e700 1 -- 172.21.15.67:0/3437755816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb50101850 msgr2=0x7feb501058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:14.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.431+0000 7feb56c3e700 1 --2- 172.21.15.67:0/3437755816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb50101850 0x7feb501058a0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7feb3c009910 tx=0x7feb3c009c20 comp rx=0 tx=0).stop 2024-04-22T23:46:14.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.431+0000 7feb56c3e700 1 -- 172.21.15.67:0/3437755816 shutdown_connections 2024-04-22T23:46:14.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.431+0000 7feb56c3e700 1 --2- 172.21.15.67:0/3437755816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb50101850 0x7feb501058a0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:14.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.431+0000 7feb56c3e700 1 --2- 172.21.15.67:0/3437755816 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb50100ea0 0x7feb50101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:14.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.431+0000 7feb56c3e700 1 -- 172.21.15.67:0/3437755816 >> 172.21.15.67:0/3437755816 conn(0x7feb500fc750 msgr2=0x7feb500feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:46:14.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.431+0000 7feb56c3e700 1 -- 172.21.15.67:0/3437755816 shutdown_connections 2024-04-22T23:46:14.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.431+0000 7feb56c3e700 1 -- 172.21.15.67:0/3437755816 wait complete. 2024-04-22T23:46:14.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.432+0000 7feb56c3e700 1 Processor -- start 2024-04-22T23:46:14.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.432+0000 7feb56c3e700 1 -- start start 2024-04-22T23:46:14.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.432+0000 7feb56c3e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb50100ea0 0x7feb5019c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:14.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.432+0000 7feb56c3e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb50101850 0x7feb5019cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:14.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.432+0000 7feb56c3e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb5019d2d0 con 0x7feb50100ea0 2024-04-22T23:46:14.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.432+0000 7feb56c3e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb50196780 con 0x7feb50101850 2024-04-22T23:46:14.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.433+0000 7feb549da700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb50100ea0 0x7feb5019c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:14.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.433+0000 7feb549da700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb50100ea0 0x7feb5019c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45138/0 (socket says 172.21.15.67:45138) 2024-04-22T23:46:14.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.433+0000 7feb549da700 1 -- 172.21.15.67:0/2000782143 learned_addr learned my addr 172.21.15.67:0/2000782143 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:46:14.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.433+0000 7feb4ffff700 1 --2- 172.21.15.67:0/2000782143 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb50101850 0x7feb5019cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:14.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.433+0000 7feb549da700 1 -- 172.21.15.67:0/2000782143 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb50101850 msgr2=0x7feb5019cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:14.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.433+0000 7feb549da700 1 --2- 172.21.15.67:0/2000782143 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb50101850 0x7feb5019cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:14.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.433+0000 7feb549da700 1 -- 172.21.15.67:0/2000782143 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb3c0095f0 con 0x7feb50100ea0 2024-04-22T23:46:14.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.433+0000 7feb549da700 1 --2- 172.21.15.67:0/2000782143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb50100ea0 0x7feb5019c700 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7feb4400c8a0 tx=0x7feb4400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:46:14.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.433+0000 7feb4dffb700 1 -- 172.21.15.67:0/2000782143 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb4400cea0 con 0x7feb50100ea0 2024-04-22T23:46:14.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.433+0000 7feb4dffb700 1 -- 172.21.15.67:0/2000782143 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feb44004830 con 0x7feb50100ea0 2024-04-22T23:46:14.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.433+0000 7feb4dffb700 1 -- 172.21.15.67:0/2000782143 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb440055b0 con 0x7feb50100ea0 2024-04-22T23:46:14.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.433+0000 7feb56c3e700 1 -- 172.21.15.67:0/2000782143 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb50196a60 con 0x7feb50100ea0 2024-04-22T23:46:14.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.433+0000 7feb56c3e700 1 -- 172.21.15.67:0/2000782143 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb50196fb0 con 0x7feb50100ea0 2024-04-22T23:46:14.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.434+0000 7feb56c3e700 1 -- 172.21.15.67:0/2000782143 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb501092a0 con 0x7feb50100ea0 2024-04-22T23:46:14.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.435+0000 7feb4dffb700 1 -- 172.21.15.67:0/2000782143 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7feb440049a0 con 0x7feb50100ea0 2024-04-22T23:46:14.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.435+0000 7feb4dffb700 1 --2- 172.21.15.67:0/2000782143 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7feb4006ecc0 0x7feb40071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:14.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.435+0000 7feb4dffb700 1 -- 172.21.15.67:0/2000782143 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7feb4408ecc0 con 0x7feb50100ea0 2024-04-22T23:46:14.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.435+0000 7feb4ffff700 1 --2- 172.21.15.67:0/2000782143 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7feb4006ecc0 0x7feb40071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:14.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.436+0000 7feb4ffff700 1 --2- 172.21.15.67:0/2000782143 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7feb4006ecc0 0x7feb40071180 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7feb50197bf0 tx=0x7feb3c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:46:14.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.440+0000 7feb4dffb700 1 -- 172.21.15.67:0/2000782143 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7feb4405b430 con 0x7feb50100ea0 2024-04-22T23:46:14.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.739+0000 7feb56c3e700 1 -- 172.21.15.67:0/2000782143 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7feb50197d30 con 0x7feb50100ea0 2024-04-22T23:46:14.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.740+0000 7feb4dffb700 1 -- 172.21.15.67:0/2000782143 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7feb4401d1c0 con 0x7feb50100ea0 2024-04-22T23:46:14.732 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:46:14.732 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:46:14.733 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:46:14.733 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:46:14.733 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:46:14.733 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:46:14.733 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:46:14.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.742+0000 7feb56c3e700 1 -- 172.21.15.67:0/2000782143 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7feb4006ecc0 msgr2=0x7feb40071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:14.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.742+0000 7feb56c3e700 1 --2- 172.21.15.67:0/2000782143 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7feb4006ecc0 0x7feb40071180 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7feb50197bf0 tx=0x7feb3c01a040 comp rx=0 tx=0).stop 2024-04-22T23:46:14.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.742+0000 7feb56c3e700 1 -- 172.21.15.67:0/2000782143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb50100ea0 msgr2=0x7feb5019c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:14.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.742+0000 7feb56c3e700 1 --2- 172.21.15.67:0/2000782143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb50100ea0 0x7feb5019c700 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7feb4400c8a0 tx=0x7feb4400cc60 comp rx=0 tx=0).stop 2024-04-22T23:46:14.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.742+0000 7feb56c3e700 1 -- 172.21.15.67:0/2000782143 shutdown_connections 2024-04-22T23:46:14.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.742+0000 7feb56c3e700 1 --2- 172.21.15.67:0/2000782143 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7feb4006ecc0 0x7feb40071180 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:14.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.742+0000 7feb56c3e700 1 --2- 172.21.15.67:0/2000782143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb50100ea0 0x7feb5019c700 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:14.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.742+0000 7feb56c3e700 1 --2- 172.21.15.67:0/2000782143 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feb50101850 0x7feb5019cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:14.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.742+0000 7feb56c3e700 1 -- 172.21.15.67:0/2000782143 >> 172.21.15.67:0/2000782143 conn(0x7feb500fc750 msgr2=0x7feb500fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:46:14.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.742+0000 7feb56c3e700 1 -- 172.21.15.67:0/2000782143 shutdown_connections 2024-04-22T23:46:14.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:14.742+0000 7feb56c3e700 1 -- 172.21.15.67:0/2000782143 wait complete. 2024-04-22T23:46:14.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:14 smithi134 ceph-mon[85630]: from='client.35108 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:46:14.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:14 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/926557342' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:46:15.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:14 smithi067 ceph-mon[112438]: from='client.35108 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:46:15.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:14 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/926557342' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:46:15.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:15 smithi134 ceph-mon[85630]: pgmap v596: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:15.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:15 smithi134 ceph-mon[85630]: from='client.35120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:46:15.735 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:15 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2000782143' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:46:16.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:15 smithi067 ceph-mon[112438]: pgmap v596: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:16.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:15 smithi067 ceph-mon[112438]: from='client.35120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:46:16.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:15 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2000782143' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:46:17.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:17 smithi134 ceph-mon[85630]: pgmap v597: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:18.035 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:17 smithi067 ceph-mon[112438]: pgmap v597: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:19.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:19 smithi134 ceph-mon[85630]: pgmap v598: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:20.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:19 smithi067 ceph-mon[112438]: pgmap v598: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:21.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:21 smithi134 ceph-mon[85630]: pgmap v599: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:22.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:21 smithi067 ceph-mon[112438]: pgmap v599: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:23.986 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:23 smithi134 ceph-mon[85630]: pgmap v600: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:24.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:23 smithi067 ceph-mon[112438]: pgmap v600: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:25.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:25 smithi134 ceph-mon[85630]: pgmap v601: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:26.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:25 smithi067 ceph-mon[112438]: pgmap v601: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:26.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:46:27.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:46:27.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:27 smithi134 ceph-mon[85630]: pgmap v602: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:28.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:27 smithi067 ceph-mon[112438]: pgmap v602: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:29.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:29 smithi134 ceph-mon[85630]: pgmap v603: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:30.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:29 smithi067 ceph-mon[112438]: pgmap v603: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:31.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:31 smithi134 ceph-mon[85630]: pgmap v604: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:32.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:31 smithi067 ceph-mon[112438]: pgmap v604: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:33.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:33 smithi134 ceph-mon[85630]: pgmap v605: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:34.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:33 smithi067 ceph-mon[112438]: pgmap v605: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:35.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:35 smithi134 ceph-mon[85630]: pgmap v606: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:36.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:35 smithi067 ceph-mon[112438]: pgmap v606: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:37.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:37 smithi134 ceph-mon[85630]: pgmap v607: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:38.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:37 smithi067 ceph-mon[112438]: pgmap v607: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:39.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:39 smithi134 ceph-mon[85630]: pgmap v608: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:40.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:39 smithi067 ceph-mon[112438]: pgmap v608: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:41.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:41 smithi134 ceph-mon[85630]: pgmap v609: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:41.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:46:42.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:41 smithi067 ceph-mon[112438]: pgmap v609: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:42.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:46:43.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:43 smithi134 ceph-mon[85630]: pgmap v610: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:44.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:43 smithi067 ceph-mon[112438]: pgmap v610: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:44.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:44.863+0000 7f4401327700 1 -- 172.21.15.67:0/3633955732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43fc102700 msgr2=0x7f43fc10abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:44.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:44.863+0000 7f4401327700 1 --2- 172.21.15.67:0/3633955732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43fc102700 0x7f43fc10abf0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f43e4009910 tx=0x7f43e4009c20 comp rx=0 tx=0).stop 2024-04-22T23:46:44.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:44.863+0000 7f4401327700 1 -- 172.21.15.67:0/3633955732 shutdown_connections 2024-04-22T23:46:44.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:44.863+0000 7f4401327700 1 --2- 172.21.15.67:0/3633955732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43fc102700 0x7f43fc10abf0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:44.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:44.863+0000 7f4401327700 1 --2- 172.21.15.67:0/3633955732 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f43fc101de0 0x7f43fc1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:44.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:44.863+0000 7f4401327700 1 -- 172.21.15.67:0/3633955732 >> 172.21.15.67:0/3633955732 conn(0x7f43fc0fb610 msgr2=0x7f43fc0fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:46:44.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:44.863+0000 7f4401327700 1 -- 172.21.15.67:0/3633955732 shutdown_connections 2024-04-22T23:46:44.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:44.863+0000 7f4401327700 1 -- 172.21.15.67:0/3633955732 wait complete. 2024-04-22T23:46:44.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:44.864+0000 7f4401327700 1 Processor -- start 2024-04-22T23:46:44.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:44.864+0000 7f4401327700 1 -- start start 2024-04-22T23:46:44.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:44.864+0000 7f4401327700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43fc101de0 0x7f43fc19e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:44.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:44.864+0000 7f4401327700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f43fc102700 0x7f43fc19eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:44.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:44.864+0000 7f4401327700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43fc19f430 con 0x7f43fc101de0 2024-04-22T23:46:44.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:44.864+0000 7f4401327700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43fc1988e0 con 0x7f43fc102700 2024-04-22T23:46:44.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:44.864+0000 7f43faffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43fc101de0 0x7f43fc19e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:44.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:44.864+0000 7f43faffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43fc101de0 0x7f43fc19e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56698/0 (socket says 172.21.15.67:56698) 2024-04-22T23:46:44.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:44.864+0000 7f43faffd700 1 -- 172.21.15.67:0/973952237 learned_addr learned my addr 172.21.15.67:0/973952237 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:46:44.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:44.865+0000 7f43fa7fc700 1 --2- 172.21.15.67:0/973952237 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f43fc102700 0x7f43fc19eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:44.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:44.865+0000 7f43faffd700 1 -- 172.21.15.67:0/973952237 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f43fc102700 msgr2=0x7f43fc19eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:44.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:44.865+0000 7f43faffd700 1 --2- 172.21.15.67:0/973952237 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f43fc102700 0x7f43fc19eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:44.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:44.865+0000 7f43faffd700 1 -- 172.21.15.67:0/973952237 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e40095f0 con 0x7f43fc101de0 2024-04-22T23:46:44.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:44.865+0000 7f43faffd700 1 --2- 172.21.15.67:0/973952237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43fc101de0 0x7f43fc19e860 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f43ec00ada0 tx=0x7f43ec00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:46:44.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:44.865+0000 7f43f3fff700 1 -- 172.21.15.67:0/973952237 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43ec00f070 con 0x7f43fc101de0 2024-04-22T23:46:44.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:44.865+0000 7f43f3fff700 1 -- 172.21.15.67:0/973952237 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f43ec007bf0 con 0x7f43fc101de0 2024-04-22T23:46:44.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:44.865+0000 7f4401327700 1 -- 172.21.15.67:0/973952237 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43fc198bc0 con 0x7f43fc101de0 2024-04-22T23:46:44.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:44.865+0000 7f43f3fff700 1 -- 172.21.15.67:0/973952237 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43ec005690 con 0x7f43fc101de0 2024-04-22T23:46:44.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:44.865+0000 7f4401327700 1 -- 172.21.15.67:0/973952237 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43fc199110 con 0x7f43fc101de0 2024-04-22T23:46:44.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:44.866+0000 7f4401327700 1 -- 172.21.15.67:0/973952237 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43fc10a750 con 0x7f43fc101de0 2024-04-22T23:46:44.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:44.867+0000 7f43f3fff700 1 -- 172.21.15.67:0/973952237 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f43ec005900 con 0x7f43fc101de0 2024-04-22T23:46:44.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:44.867+0000 7f43f3fff700 1 --2- 172.21.15.67:0/973952237 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f43e806ecc0 0x7f43e8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:44.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:44.867+0000 7f43f3fff700 1 -- 172.21.15.67:0/973952237 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f43ec08ea10 con 0x7f43fc101de0 2024-04-22T23:46:44.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:44.867+0000 7f43fa7fc700 1 --2- 172.21.15.67:0/973952237 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f43e806ecc0 0x7f43e8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:44.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:44.868+0000 7f43fa7fc700 1 --2- 172.21.15.67:0/973952237 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f43e806ecc0 0x7f43e8071180 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f43fc0fcd80 tx=0x7f43e401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:46:44.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:44.872+0000 7f43f3fff700 1 -- 172.21.15.67:0/973952237 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f43ec05b180 con 0x7f43fc101de0 2024-04-22T23:46:45.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.102+0000 7f4401327700 1 -- 172.21.15.67:0/973952237 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f43fc199330 con 0x7f43e806ecc0 2024-04-22T23:46:45.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.104+0000 7f43f3fff700 1 -- 172.21.15.67:0/973952237 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f43fc199330 con 0x7f43e806ecc0 2024-04-22T23:46:45.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.106+0000 7f4401327700 1 -- 172.21.15.67:0/973952237 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f43e806ecc0 msgr2=0x7f43e8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:45.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.106+0000 7f4401327700 1 --2- 172.21.15.67:0/973952237 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f43e806ecc0 0x7f43e8071180 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f43fc0fcd80 tx=0x7f43e401a040 comp rx=0 tx=0).stop 2024-04-22T23:46:45.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.106+0000 7f4401327700 1 -- 172.21.15.67:0/973952237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43fc101de0 msgr2=0x7f43fc19e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:45.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.106+0000 7f4401327700 1 --2- 172.21.15.67:0/973952237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43fc101de0 0x7f43fc19e860 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f43ec00ada0 tx=0x7f43ec00c330 comp rx=0 tx=0).stop 2024-04-22T23:46:45.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.106+0000 7f4401327700 1 -- 172.21.15.67:0/973952237 shutdown_connections 2024-04-22T23:46:45.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.106+0000 7f4401327700 1 --2- 172.21.15.67:0/973952237 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f43e806ecc0 0x7f43e8071180 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:45.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.106+0000 7f4401327700 1 --2- 172.21.15.67:0/973952237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43fc101de0 0x7f43fc19e860 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:45.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.106+0000 7f4401327700 1 --2- 172.21.15.67:0/973952237 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f43fc102700 0x7f43fc19eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:45.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.106+0000 7f4401327700 1 -- 172.21.15.67:0/973952237 >> 172.21.15.67:0/973952237 conn(0x7f43fc0fb610 msgr2=0x7f43fc1001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:46:45.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.106+0000 7f4401327700 1 -- 172.21.15.67:0/973952237 shutdown_connections 2024-04-22T23:46:45.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.106+0000 7f4401327700 1 -- 172.21.15.67:0/973952237 wait complete. 2024-04-22T23:46:45.108 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:46:45.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.213+0000 7fb1fcbfa700 1 -- 172.21.15.67:0/3150841168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1f8068580 msgr2=0x7fb1f80689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:45.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.213+0000 7fb1fcbfa700 1 --2- 172.21.15.67:0/3150841168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1f8068580 0x7fb1f80689e0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec009910 tx=0x7fb1ec009c20 comp rx=0 tx=0).stop 2024-04-22T23:46:45.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.214+0000 7fb1fcbfa700 1 -- 172.21.15.67:0/3150841168 shutdown_connections 2024-04-22T23:46:45.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.214+0000 7fb1fcbfa700 1 --2- 172.21.15.67:0/3150841168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1f8068580 0x7fb1f80689e0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:45.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.214+0000 7fb1fcbfa700 1 --2- 172.21.15.67:0/3150841168 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1f8106cd0 0x7fb1f81070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:45.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.214+0000 7fb1fcbfa700 1 -- 172.21.15.67:0/3150841168 >> 172.21.15.67:0/3150841168 conn(0x7fb1f8075670 msgr2=0x7fb1f8075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:46:45.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.214+0000 7fb1fcbfa700 1 -- 172.21.15.67:0/3150841168 shutdown_connections 2024-04-22T23:46:45.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.214+0000 7fb1fcbfa700 1 -- 172.21.15.67:0/3150841168 wait complete. 2024-04-22T23:46:45.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.214+0000 7fb1fcbfa700 1 Processor -- start 2024-04-22T23:46:45.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.214+0000 7fb1fcbfa700 1 -- start start 2024-04-22T23:46:45.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.215+0000 7fb1fcbfa700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1f8068580 0x7fb1f819e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:45.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.215+0000 7fb1fcbfa700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1f8106cd0 0x7fb1f819ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:45.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.215+0000 7fb1fcbfa700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f819f400 con 0x7fb1f8068580 2024-04-22T23:46:45.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.215+0000 7fb1fcbfa700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f81988b0 con 0x7fb1f8106cd0 2024-04-22T23:46:45.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.215+0000 7fb1f659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1f8068580 0x7fb1f819e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:45.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.215+0000 7fb1f659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1f8068580 0x7fb1f819e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56726/0 (socket says 172.21.15.67:56726) 2024-04-22T23:46:45.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.215+0000 7fb1f659c700 1 -- 172.21.15.67:0/1953337238 learned_addr learned my addr 172.21.15.67:0/1953337238 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:46:45.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.215+0000 7fb1f5d9b700 1 --2- 172.21.15.67:0/1953337238 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1f8106cd0 0x7fb1f819ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:45.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.215+0000 7fb1f659c700 1 -- 172.21.15.67:0/1953337238 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1f8106cd0 msgr2=0x7fb1f819ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:45.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.215+0000 7fb1f659c700 1 --2- 172.21.15.67:0/1953337238 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1f8106cd0 0x7fb1f819ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:45.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.215+0000 7fb1f659c700 1 -- 172.21.15.67:0/1953337238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1ec0095f0 con 0x7fb1f8068580 2024-04-22T23:46:45.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.215+0000 7fb1f659c700 1 --2- 172.21.15.67:0/1953337238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1f8068580 0x7fb1f819e830 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fb1e800c8a0 tx=0x7fb1e800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:46:45.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.215+0000 7fb1e77fe700 1 -- 172.21.15.67:0/1953337238 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1e80078e0 con 0x7fb1f8068580 2024-04-22T23:46:45.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.215+0000 7fb1fcbfa700 1 -- 172.21.15.67:0/1953337238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1f8198b90 con 0x7fb1f8068580 2024-04-22T23:46:45.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.215+0000 7fb1e77fe700 1 -- 172.21.15.67:0/1953337238 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb1e800f450 con 0x7fb1f8068580 2024-04-22T23:46:45.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.216+0000 7fb1e77fe700 1 -- 172.21.15.67:0/1953337238 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1e800e5e0 con 0x7fb1f8068580 2024-04-22T23:46:45.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.216+0000 7fb1fcbfa700 1 -- 172.21.15.67:0/1953337238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1f81990e0 con 0x7fb1f8068580 2024-04-22T23:46:45.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.217+0000 7fb1fcbfa700 1 -- 172.21.15.67:0/1953337238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1f804f2e0 con 0x7fb1f8068580 2024-04-22T23:46:45.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.217+0000 7fb1e77fe700 1 -- 172.21.15.67:0/1953337238 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb1e800f5c0 con 0x7fb1f8068580 2024-04-22T23:46:45.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.217+0000 7fb1e77fe700 1 --2- 172.21.15.67:0/1953337238 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb1e006ecc0 0x7fb1e0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:45.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.218+0000 7fb1e77fe700 1 -- 172.21.15.67:0/1953337238 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fb1e808eff0 con 0x7fb1f8068580 2024-04-22T23:46:45.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.218+0000 7fb1f5d9b700 1 --2- 172.21.15.67:0/1953337238 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb1e006ecc0 0x7fb1e0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:45.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.218+0000 7fb1f5d9b700 1 --2- 172.21.15.67:0/1953337238 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb1e006ecc0 0x7fb1e0071180 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb1f80fe3c0 tx=0x7fb1ec01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:46:45.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.222+0000 7fb1e77fe700 1 -- 172.21.15.67:0/1953337238 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb1e805b760 con 0x7fb1f8068580 2024-04-22T23:46:45.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.451+0000 7fb1fcbfa700 1 -- 172.21.15.67:0/1953337238 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1f81993d0 con 0x7fb1e006ecc0 2024-04-22T23:46:45.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.452+0000 7fb1e77fe700 1 -- 172.21.15.67:0/1953337238 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb1f81993d0 con 0x7fb1e006ecc0 2024-04-22T23:46:45.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.454+0000 7fb1fcbfa700 1 -- 172.21.15.67:0/1953337238 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb1e006ecc0 msgr2=0x7fb1e0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:45.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.454+0000 7fb1fcbfa700 1 --2- 172.21.15.67:0/1953337238 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb1e006ecc0 0x7fb1e0071180 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb1f80fe3c0 tx=0x7fb1ec01a040 comp rx=0 tx=0).stop 2024-04-22T23:46:45.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.454+0000 7fb1fcbfa700 1 -- 172.21.15.67:0/1953337238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1f8068580 msgr2=0x7fb1f819e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:45.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.454+0000 7fb1fcbfa700 1 --2- 172.21.15.67:0/1953337238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1f8068580 0x7fb1f819e830 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fb1e800c8a0 tx=0x7fb1e800cbb0 comp rx=0 tx=0).stop 2024-04-22T23:46:45.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.454+0000 7fb1fcbfa700 1 -- 172.21.15.67:0/1953337238 shutdown_connections 2024-04-22T23:46:45.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.454+0000 7fb1fcbfa700 1 --2- 172.21.15.67:0/1953337238 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb1e006ecc0 0x7fb1e0071180 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:45.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.454+0000 7fb1fcbfa700 1 --2- 172.21.15.67:0/1953337238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1f8068580 0x7fb1f819e830 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:45.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.454+0000 7fb1fcbfa700 1 --2- 172.21.15.67:0/1953337238 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1f8106cd0 0x7fb1f819ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:45.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.454+0000 7fb1fcbfa700 1 -- 172.21.15.67:0/1953337238 >> 172.21.15.67:0/1953337238 conn(0x7fb1f8075670 msgr2=0x7fb1f80fe9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:46:45.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.454+0000 7fb1fcbfa700 1 -- 172.21.15.67:0/1953337238 shutdown_connections 2024-04-22T23:46:45.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.455+0000 7fb1fcbfa700 1 -- 172.21.15.67:0/1953337238 wait complete. 2024-04-22T23:46:45.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.560+0000 7ff59e46f700 1 -- 172.21.15.67:0/3454256860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5980ffc20 msgr2=0x7ff598100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:45.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.560+0000 7ff59e46f700 1 --2- 172.21.15.67:0/3454256860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5980ffc20 0x7ff598100080 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7ff580009910 tx=0x7ff580009c20 comp rx=0 tx=0).stop 2024-04-22T23:46:45.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.560+0000 7ff59e46f700 1 -- 172.21.15.67:0/3454256860 shutdown_connections 2024-04-22T23:46:45.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.560+0000 7ff59e46f700 1 --2- 172.21.15.67:0/3454256860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5980ffc20 0x7ff598100080 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:45.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.560+0000 7ff59e46f700 1 --2- 172.21.15.67:0/3454256860 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff598105bd0 0x7ff598105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:45.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.560+0000 7ff59e46f700 1 -- 172.21.15.67:0/3454256860 >> 172.21.15.67:0/3454256860 conn(0x7ff598075670 msgr2=0x7ff598075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:46:45.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.560+0000 7ff59e46f700 1 -- 172.21.15.67:0/3454256860 shutdown_connections 2024-04-22T23:46:45.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.560+0000 7ff59e46f700 1 -- 172.21.15.67:0/3454256860 wait complete. 2024-04-22T23:46:45.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.561+0000 7ff59e46f700 1 Processor -- start 2024-04-22T23:46:45.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.561+0000 7ff59e46f700 1 -- start start 2024-04-22T23:46:45.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.561+0000 7ff59e46f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5980ffc20 0x7ff59819c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:45.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.561+0000 7ff59e46f700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff598105bd0 0x7ff59819ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:45.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.561+0000 7ff59e46f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff59819d350 con 0x7ff5980ffc20 2024-04-22T23:46:45.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.561+0000 7ff59e46f700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff598196800 con 0x7ff598105bd0 2024-04-22T23:46:45.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.561+0000 7ff597fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5980ffc20 0x7ff59819c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:45.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.561+0000 7ff597fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5980ffc20 0x7ff59819c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56750/0 (socket says 172.21.15.67:56750) 2024-04-22T23:46:45.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.561+0000 7ff597fff700 1 -- 172.21.15.67:0/3185073084 learned_addr learned my addr 172.21.15.67:0/3185073084 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:46:45.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.562+0000 7ff5977fe700 1 --2- 172.21.15.67:0/3185073084 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff598105bd0 0x7ff59819ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:45.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.562+0000 7ff597fff700 1 -- 172.21.15.67:0/3185073084 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff598105bd0 msgr2=0x7ff59819ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:45.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.562+0000 7ff597fff700 1 --2- 172.21.15.67:0/3185073084 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff598105bd0 0x7ff59819ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:45.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.562+0000 7ff597fff700 1 -- 172.21.15.67:0/3185073084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5800095f0 con 0x7ff5980ffc20 2024-04-22T23:46:45.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.562+0000 7ff597fff700 1 --2- 172.21.15.67:0/3185073084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5980ffc20 0x7ff59819c780 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7ff58800b4b0 tx=0x7ff58800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:46:45.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.562+0000 7ff5957fa700 1 -- 172.21.15.67:0/3185073084 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff588011800 con 0x7ff5980ffc20 2024-04-22T23:46:45.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.562+0000 7ff5957fa700 1 -- 172.21.15.67:0/3185073084 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff588011e20 con 0x7ff5980ffc20 2024-04-22T23:46:45.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.562+0000 7ff59e46f700 1 -- 172.21.15.67:0/3185073084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff598196ae0 con 0x7ff5980ffc20 2024-04-22T23:46:45.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.562+0000 7ff5957fa700 1 -- 172.21.15.67:0/3185073084 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff58800f2e0 con 0x7ff5980ffc20 2024-04-22T23:46:45.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.562+0000 7ff59e46f700 1 -- 172.21.15.67:0/3185073084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff598197030 con 0x7ff5980ffc20 2024-04-22T23:46:45.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.563+0000 7ff59e46f700 1 -- 172.21.15.67:0/3185073084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff59804f2e0 con 0x7ff5980ffc20 2024-04-22T23:46:45.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.564+0000 7ff5957fa700 1 -- 172.21.15.67:0/3185073084 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff588010380 con 0x7ff5980ffc20 2024-04-22T23:46:45.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.564+0000 7ff5957fa700 1 --2- 172.21.15.67:0/3185073084 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff58406ecc0 0x7ff584071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:45.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.564+0000 7ff5957fa700 1 -- 172.21.15.67:0/3185073084 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7ff58808eb70 con 0x7ff5980ffc20 2024-04-22T23:46:45.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.564+0000 7ff5977fe700 1 --2- 172.21.15.67:0/3185073084 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff58406ecc0 0x7ff584071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:45.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.565+0000 7ff5977fe700 1 --2- 172.21.15.67:0/3185073084 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff58406ecc0 0x7ff584071180 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff5800059d0 tx=0x7ff58001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:46:45.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.569+0000 7ff5957fa700 1 -- 172.21.15.67:0/3185073084 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff58805b2e0 con 0x7ff5980ffc20 2024-04-22T23:46:45.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.786+0000 7ff59e46f700 1 -- 172.21.15.67:0/3185073084 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff598197310 con 0x7ff58406ecc0 2024-04-22T23:46:45.778 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:45 smithi067 ceph-mon[112438]: pgmap v611: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:45.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.797+0000 7ff5957fa700 1 -- 172.21.15.67:0/3185073084 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7ff598197310 con 0x7ff58406ecc0 2024-04-22T23:46:45.789 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:46:45.789 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (20m) 13m ago 31m 25.5M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:46:45.790 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (32m) 13m ago 31m 9114k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:46:45.790 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (30m) 13m ago 30m 9042k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:46:45.790 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (17m) 13m ago 31m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:46:45.790 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (17m) 13m ago 30m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:46:45.790 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (20m) 13m ago 31m 79.3M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:46:45.790 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (14m) 13m ago 26m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:46:45.790 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (13m) 13m ago 26m 12.6M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:46:45.790 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (13m) 13m ago 26m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:46:45.790 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (26m) 13m ago 26m 12.2M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:46:45.790 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (22m) 13m ago 33m 511M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:46:45.790 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (21m) 13m ago 30m 453M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:46:45.791 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (18m) 13m ago 33m 55.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:46:45.791 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (18m) 13m ago 30m 46.0M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:46:45.791 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (21m) 13m ago 31m 15.4M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:46:45.791 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (21m) 13m ago 30m 16.1M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:46:45.791 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (17m) 13m ago 29m 99.2M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:46:45.791 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (16m) 13m ago 29m 127M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:46:45.791 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (16m) 13m ago 29m 79.9M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:46:45.791 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (15m) 13m ago 28m 97.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:46:45.791 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (15m) 13m ago 28m 82.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:46:45.791 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (14m) 13m ago 27m 77.6M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:46:45.791 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (20m) 13m ago 31m 73.9M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:46:45.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.799+0000 7ff59e46f700 1 -- 172.21.15.67:0/3185073084 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff58406ecc0 msgr2=0x7ff584071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:45.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.799+0000 7ff59e46f700 1 --2- 172.21.15.67:0/3185073084 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff58406ecc0 0x7ff584071180 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff5800059d0 tx=0x7ff58001a040 comp rx=0 tx=0).stop 2024-04-22T23:46:45.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.799+0000 7ff59e46f700 1 -- 172.21.15.67:0/3185073084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5980ffc20 msgr2=0x7ff59819c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:45.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.799+0000 7ff59e46f700 1 --2- 172.21.15.67:0/3185073084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5980ffc20 0x7ff59819c780 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7ff58800b4b0 tx=0x7ff58800b870 comp rx=0 tx=0).stop 2024-04-22T23:46:45.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.799+0000 7ff59e46f700 1 -- 172.21.15.67:0/3185073084 shutdown_connections 2024-04-22T23:46:45.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.800+0000 7ff59e46f700 1 --2- 172.21.15.67:0/3185073084 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff58406ecc0 0x7ff584071180 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:45.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.800+0000 7ff59e46f700 1 --2- 172.21.15.67:0/3185073084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5980ffc20 0x7ff59819c780 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:45.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.800+0000 7ff59e46f700 1 --2- 172.21.15.67:0/3185073084 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff598105bd0 0x7ff59819ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:45.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.800+0000 7ff59e46f700 1 -- 172.21.15.67:0/3185073084 >> 172.21.15.67:0/3185073084 conn(0x7ff598075670 msgr2=0x7ff5980686e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:46:45.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.800+0000 7ff59e46f700 1 -- 172.21.15.67:0/3185073084 shutdown_connections 2024-04-22T23:46:45.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.800+0000 7ff59e46f700 1 -- 172.21.15.67:0/3185073084 wait complete. 2024-04-22T23:46:45.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.906+0000 7f1584ddc700 1 -- 172.21.15.67:0/2548360045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15800ffda0 msgr2=0x7f1580100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:45.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.906+0000 7f1584ddc700 1 --2- 172.21.15.67:0/2548360045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15800ffda0 0x7f1580100180 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f1570009910 tx=0x7f1570009c20 comp rx=0 tx=0).stop 2024-04-22T23:46:45.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.906+0000 7f1584ddc700 1 -- 172.21.15.67:0/2548360045 shutdown_connections 2024-04-22T23:46:45.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.906+0000 7f1584ddc700 1 --2- 172.21.15.67:0/2548360045 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1580100750 0x7f15801047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:45.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.906+0000 7f1584ddc700 1 --2- 172.21.15.67:0/2548360045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15800ffda0 0x7f1580100180 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:45.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.906+0000 7f1584ddc700 1 -- 172.21.15.67:0/2548360045 >> 172.21.15.67:0/2548360045 conn(0x7f15800fb650 msgr2=0x7f15800fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:46:45.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.906+0000 7f1584ddc700 1 -- 172.21.15.67:0/2548360045 shutdown_connections 2024-04-22T23:46:45.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.906+0000 7f1584ddc700 1 -- 172.21.15.67:0/2548360045 wait complete. 2024-04-22T23:46:45.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.906+0000 7f1584ddc700 1 Processor -- start 2024-04-22T23:46:45.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.907+0000 7f1584ddc700 1 -- start start 2024-04-22T23:46:45.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.907+0000 7f1584ddc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15800ffda0 0x7f158019c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:45.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.907+0000 7f1584ddc700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1580100750 0x7f158019cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:45.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.907+0000 7f1584ddc700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f158019d200 con 0x7f15800ffda0 2024-04-22T23:46:45.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.907+0000 7f1584ddc700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15801966b0 con 0x7f1580100750 2024-04-22T23:46:45.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.907+0000 7f157e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15800ffda0 0x7f158019c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:45.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.907+0000 7f157e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15800ffda0 0x7f158019c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56768/0 (socket says 172.21.15.67:56768) 2024-04-22T23:46:45.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.907+0000 7f157e59c700 1 -- 172.21.15.67:0/3445333446 learned_addr learned my addr 172.21.15.67:0/3445333446 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:46:45.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.907+0000 7f157dd9b700 1 --2- 172.21.15.67:0/3445333446 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1580100750 0x7f158019cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:45.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.907+0000 7f157e59c700 1 -- 172.21.15.67:0/3445333446 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1580100750 msgr2=0x7f158019cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:45.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.907+0000 7f157e59c700 1 --2- 172.21.15.67:0/3445333446 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1580100750 0x7f158019cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:45.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.907+0000 7f157e59c700 1 -- 172.21.15.67:0/3445333446 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15700095f0 con 0x7f15800ffda0 2024-04-22T23:46:45.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.907+0000 7f157dd9b700 1 --2- 172.21.15.67:0/3445333446 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1580100750 0x7f158019cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:46:45.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.907+0000 7f157e59c700 1 --2- 172.21.15.67:0/3445333446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15800ffda0 0x7f158019c630 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f1570006010 tx=0x7f157000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:46:45.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.908+0000 7f156f7fe700 1 -- 172.21.15.67:0/3445333446 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f157001d070 con 0x7f15800ffda0 2024-04-22T23:46:45.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.908+0000 7f156f7fe700 1 -- 172.21.15.67:0/3445333446 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f157000f460 con 0x7f15800ffda0 2024-04-22T23:46:45.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.908+0000 7f156f7fe700 1 -- 172.21.15.67:0/3445333446 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1570017630 con 0x7f15800ffda0 2024-04-22T23:46:45.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.908+0000 7f1584ddc700 1 -- 172.21.15.67:0/3445333446 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1580196930 con 0x7f15800ffda0 2024-04-22T23:46:45.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.908+0000 7f1584ddc700 1 -- 172.21.15.67:0/3445333446 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1580196e20 con 0x7f15800ffda0 2024-04-22T23:46:45.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.909+0000 7f1584ddc700 1 -- 172.21.15.67:0/3445333446 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f158004f2e0 con 0x7f15800ffda0 2024-04-22T23:46:45.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.910+0000 7f156f7fe700 1 -- 172.21.15.67:0/3445333446 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f157000f5d0 con 0x7f15800ffda0 2024-04-22T23:46:45.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.910+0000 7f156f7fe700 1 --2- 172.21.15.67:0/3445333446 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f156806ecc0 0x7f1568071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:45.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.910+0000 7f156f7fe700 1 -- 172.21.15.67:0/3445333446 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f15700904a0 con 0x7f15800ffda0 2024-04-22T23:46:45.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.910+0000 7f157dd9b700 1 --2- 172.21.15.67:0/3445333446 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f156806ecc0 0x7f1568071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:45.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.911+0000 7f157dd9b700 1 --2- 172.21.15.67:0/3445333446 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f156806ecc0 0x7f1568071180 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1574005950 tx=0x7f15740058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:46:45.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:45.915+0000 7f156f7fe700 1 -- 172.21.15.67:0/3445333446 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f157005cc10 con 0x7f15800ffda0 2024-04-22T23:46:45.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:45 smithi134 ceph-mon[85630]: pgmap v611: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:46.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.214+0000 7f1584ddc700 1 -- 172.21.15.67:0/3445333446 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1580197100 con 0x7f15800ffda0 2024-04-22T23:46:46.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.215+0000 7f156f7fe700 1 -- 172.21.15.67:0/3445333446 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f157005c7a0 con 0x7f15800ffda0 2024-04-22T23:46:46.207 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:46:46.207 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:46:46.207 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:46:46.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:46:46.207 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:46:46.208 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:46:46.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:46:46.208 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:46:46.208 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:46:46.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:46:46.208 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:46:46.208 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:46:46.208 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 3 2024-04-22T23:46:46.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:46:46.208 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:46:46.208 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:46:46.208 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 13 2024-04-22T23:46:46.208 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:46:46.208 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:46:46.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.217+0000 7f1584ddc700 1 -- 172.21.15.67:0/3445333446 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f156806ecc0 msgr2=0x7f1568071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:46.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.217+0000 7f1584ddc700 1 --2- 172.21.15.67:0/3445333446 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f156806ecc0 0x7f1568071180 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1574005950 tx=0x7f15740058e0 comp rx=0 tx=0).stop 2024-04-22T23:46:46.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.217+0000 7f1584ddc700 1 -- 172.21.15.67:0/3445333446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15800ffda0 msgr2=0x7f158019c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:46.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.217+0000 7f1584ddc700 1 --2- 172.21.15.67:0/3445333446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15800ffda0 0x7f158019c630 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f1570006010 tx=0x7f157000be20 comp rx=0 tx=0).stop 2024-04-22T23:46:46.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.217+0000 7f1584ddc700 1 -- 172.21.15.67:0/3445333446 shutdown_connections 2024-04-22T23:46:46.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.217+0000 7f1584ddc700 1 --2- 172.21.15.67:0/3445333446 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f156806ecc0 0x7f1568071180 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:46.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.217+0000 7f1584ddc700 1 --2- 172.21.15.67:0/3445333446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15800ffda0 0x7f158019c630 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:46.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.217+0000 7f1584ddc700 1 --2- 172.21.15.67:0/3445333446 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1580100750 0x7f158019cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:46.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.217+0000 7f1584ddc700 1 -- 172.21.15.67:0/3445333446 >> 172.21.15.67:0/3445333446 conn(0x7f15800fb650 msgr2=0x7f1580103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:46:46.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.217+0000 7f1584ddc700 1 -- 172.21.15.67:0/3445333446 shutdown_connections 2024-04-22T23:46:46.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.217+0000 7f1584ddc700 1 -- 172.21.15.67:0/3445333446 wait complete. 2024-04-22T23:46:46.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.324+0000 7fbedc3f0700 1 -- 172.21.15.67:0/922185557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbed4102760 msgr2=0x7fbed410ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:46.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.324+0000 7fbedc3f0700 1 --2- 172.21.15.67:0/922185557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbed4102760 0x7fbed410ac50 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fbec4009940 tx=0x7fbec4009c50 comp rx=0 tx=0).stop 2024-04-22T23:46:46.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.324+0000 7fbedc3f0700 1 -- 172.21.15.67:0/922185557 shutdown_connections 2024-04-22T23:46:46.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.324+0000 7fbedc3f0700 1 --2- 172.21.15.67:0/922185557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbed4102760 0x7fbed410ac50 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:46.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.324+0000 7fbedc3f0700 1 --2- 172.21.15.67:0/922185557 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbed4101e40 0x7fbed4102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:46.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.324+0000 7fbedc3f0700 1 -- 172.21.15.67:0/922185557 >> 172.21.15.67:0/922185557 conn(0x7fbed40fb610 msgr2=0x7fbed40fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:46:46.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.324+0000 7fbedc3f0700 1 -- 172.21.15.67:0/922185557 shutdown_connections 2024-04-22T23:46:46.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.324+0000 7fbedc3f0700 1 -- 172.21.15.67:0/922185557 wait complete. 2024-04-22T23:46:46.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.325+0000 7fbedc3f0700 1 Processor -- start 2024-04-22T23:46:46.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.325+0000 7fbedc3f0700 1 -- start start 2024-04-22T23:46:46.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.325+0000 7fbedc3f0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbed4101e40 0x7fbed40ffaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:46.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.325+0000 7fbedc3f0700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbed4102760 0x7fbed40fffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:46.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.325+0000 7fbedc3f0700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed4100520 con 0x7fbed4101e40 2024-04-22T23:46:46.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.325+0000 7fbedc3f0700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed4100660 con 0x7fbed4102760 2024-04-22T23:46:46.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.325+0000 7fbeda18c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbed4101e40 0x7fbed40ffaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:46.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.325+0000 7fbeda18c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbed4101e40 0x7fbed40ffaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56792/0 (socket says 172.21.15.67:56792) 2024-04-22T23:46:46.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.325+0000 7fbeda18c700 1 -- 172.21.15.67:0/3026621661 learned_addr learned my addr 172.21.15.67:0/3026621661 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:46:46.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.325+0000 7fbed998b700 1 --2- 172.21.15.67:0/3026621661 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbed4102760 0x7fbed40fffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:46.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.326+0000 7fbeda18c700 1 -- 172.21.15.67:0/3026621661 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbed4102760 msgr2=0x7fbed40fffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:46.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.326+0000 7fbeda18c700 1 --2- 172.21.15.67:0/3026621661 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbed4102760 0x7fbed40fffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:46.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.326+0000 7fbeda18c700 1 -- 172.21.15.67:0/3026621661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbec40095f0 con 0x7fbed4101e40 2024-04-22T23:46:46.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.326+0000 7fbeda18c700 1 --2- 172.21.15.67:0/3026621661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbed4101e40 0x7fbed40ffaa0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fbed000d8d0 tx=0x7fbed000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:46:46.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.326+0000 7fbecb7fe700 1 -- 172.21.15.67:0/3026621661 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbed00098a0 con 0x7fbed4101e40 2024-04-22T23:46:46.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.326+0000 7fbecb7fe700 1 -- 172.21.15.67:0/3026621661 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbed0010460 con 0x7fbed4101e40 2024-04-22T23:46:46.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.326+0000 7fbedc3f0700 1 -- 172.21.15.67:0/3026621661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbed41a2a80 con 0x7fbed4101e40 2024-04-22T23:46:46.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.326+0000 7fbecb7fe700 1 -- 172.21.15.67:0/3026621661 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbed000f5f0 con 0x7fbed4101e40 2024-04-22T23:46:46.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.326+0000 7fbedc3f0700 1 -- 172.21.15.67:0/3026621661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbed41a2e30 con 0x7fbed4101e40 2024-04-22T23:46:46.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.327+0000 7fbedc3f0700 1 -- 172.21.15.67:0/3026621661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbed4108350 con 0x7fbed4101e40 2024-04-22T23:46:46.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.328+0000 7fbecb7fe700 1 -- 172.21.15.67:0/3026621661 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbed0009a00 con 0x7fbed4101e40 2024-04-22T23:46:46.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.328+0000 7fbecb7fe700 1 --2- 172.21.15.67:0/3026621661 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbec006ecc0 0x7fbec0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:46.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.328+0000 7fbed998b700 1 --2- 172.21.15.67:0/3026621661 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbec006ecc0 0x7fbec0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:46.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.328+0000 7fbecb7fe700 1 -- 172.21.15.67:0/3026621661 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fbed008edb0 con 0x7fbed4101e40 2024-04-22T23:46:46.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.329+0000 7fbed998b700 1 --2- 172.21.15.67:0/3026621661 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbec006ecc0 0x7fbec0071180 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fbec4005b40 tx=0x7fbec4005a90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:46:46.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.333+0000 7fbecb7fe700 1 -- 172.21.15.67:0/3026621661 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fbed0057a70 con 0x7fbed4101e40 2024-04-22T23:46:46.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.596+0000 7fbedc3f0700 1 -- 172.21.15.67:0/3026621661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fbed4107d30 con 0x7fbed4101e40 2024-04-22T23:46:46.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.596+0000 7fbecb7fe700 1 -- 172.21.15.67:0/3026621661 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1871 (secure 0 0 0) 0x7fbed0016760 con 0x7fbed4101e40 2024-04-22T23:46:46.589 INFO:teuthology.orchestra.run.smithi067.stdout:e31 2024-04-22T23:46:46.589 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:46:46.589 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:46:46.589 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:46:46.589 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:46:46.589 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:46:46.589 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:46:46.589 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:46:46.589 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:46:46.589 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:46:46.589 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:46:46.589 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:46:46.590 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:46:46.590 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:46:46.590 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:46:46.590 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:46:46.590 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:46:46.590 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:46:46.590 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:46:46.590 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:46:46.590 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:46:46.590 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:46:46.590 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:46:46.590 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:46:46.590 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:46:46.590 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:46:46.590 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:46:46.591 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:46:46.591 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:46:46.591 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:46:46.591 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:46:46.591 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:46:46.591 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:46:46.591 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:46:46.591 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:46:46.591 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:46:46.591 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:46:46.591 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:46:46.591 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:46:46.591 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:46:46.591 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44259} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1827288036,v1:172.21.15.134:6827/1827288036] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:46:46.592 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:46:46.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.598+0000 7fbedc3f0700 1 -- 172.21.15.67:0/3026621661 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbec006ecc0 msgr2=0x7fbec0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:46.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.598+0000 7fbedc3f0700 1 --2- 172.21.15.67:0/3026621661 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbec006ecc0 0x7fbec0071180 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fbec4005b40 tx=0x7fbec4005a90 comp rx=0 tx=0).stop 2024-04-22T23:46:46.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.599+0000 7fbedc3f0700 1 -- 172.21.15.67:0/3026621661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbed4101e40 msgr2=0x7fbed40ffaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:46.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.599+0000 7fbedc3f0700 1 --2- 172.21.15.67:0/3026621661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbed4101e40 0x7fbed40ffaa0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fbed000d8d0 tx=0x7fbed000dbe0 comp rx=0 tx=0).stop 2024-04-22T23:46:46.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.599+0000 7fbedc3f0700 1 -- 172.21.15.67:0/3026621661 shutdown_connections 2024-04-22T23:46:46.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.599+0000 7fbedc3f0700 1 --2- 172.21.15.67:0/3026621661 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbec006ecc0 0x7fbec0071180 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:46.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.599+0000 7fbedc3f0700 1 --2- 172.21.15.67:0/3026621661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbed4101e40 0x7fbed40ffaa0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:46.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.599+0000 7fbedc3f0700 1 --2- 172.21.15.67:0/3026621661 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbed4102760 0x7fbed40fffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:46.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.599+0000 7fbedc3f0700 1 -- 172.21.15.67:0/3026621661 >> 172.21.15.67:0/3026621661 conn(0x7fbed40fb610 msgr2=0x7fbed4105490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:46:46.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.599+0000 7fbedc3f0700 1 -- 172.21.15.67:0/3026621661 shutdown_connections 2024-04-22T23:46:46.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.599+0000 7fbedc3f0700 1 -- 172.21.15.67:0/3026621661 wait complete. 2024-04-22T23:46:46.593 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 31 2024-04-22T23:46:46.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.707+0000 7f24cdad6700 1 -- 172.21.15.67:0/968587395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24c8073160 msgr2=0x7f24c8073540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:46.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.707+0000 7f24cdad6700 1 --2- 172.21.15.67:0/968587395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24c8073160 0x7f24c8073540 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f24b8009910 tx=0x7f24b8009c20 comp rx=0 tx=0).stop 2024-04-22T23:46:46.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.708+0000 7f24cdad6700 1 -- 172.21.15.67:0/968587395 shutdown_connections 2024-04-22T23:46:46.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.708+0000 7f24cdad6700 1 --2- 172.21.15.67:0/968587395 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f24c8073a80 0x7f24c810cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:46.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.708+0000 7f24cdad6700 1 --2- 172.21.15.67:0/968587395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24c8073160 0x7f24c8073540 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:46.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.708+0000 7f24cdad6700 1 -- 172.21.15.67:0/968587395 >> 172.21.15.67:0/968587395 conn(0x7f24c80fc750 msgr2=0x7f24c80feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:46:46.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.708+0000 7f24cdad6700 1 -- 172.21.15.67:0/968587395 shutdown_connections 2024-04-22T23:46:46.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.708+0000 7f24cdad6700 1 -- 172.21.15.67:0/968587395 wait complete. 2024-04-22T23:46:46.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.708+0000 7f24cdad6700 1 Processor -- start 2024-04-22T23:46:46.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.708+0000 7f24cdad6700 1 -- start start 2024-04-22T23:46:46.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.709+0000 7f24cdad6700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f24c8073160 0x7f24c819e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:46.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.709+0000 7f24cdad6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24c8073a80 0x7f24c819edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:46.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.709+0000 7f24cdad6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c819f440 con 0x7f24c8073a80 2024-04-22T23:46:46.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.709+0000 7f24cdad6700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c81988f0 con 0x7f24c8073160 2024-04-22T23:46:46.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.709+0000 7f24c6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24c8073a80 0x7f24c819edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:46.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.709+0000 7f24c6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24c8073a80 0x7f24c819edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56804/0 (socket says 172.21.15.67:56804) 2024-04-22T23:46:46.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.709+0000 7f24c6ffd700 1 -- 172.21.15.67:0/4021324235 learned_addr learned my addr 172.21.15.67:0/4021324235 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:46:46.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.709+0000 7f24c77fe700 1 --2- 172.21.15.67:0/4021324235 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f24c8073160 0x7f24c819e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:46.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.709+0000 7f24c6ffd700 1 -- 172.21.15.67:0/4021324235 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f24c8073160 msgr2=0x7f24c819e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:46.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.709+0000 7f24c6ffd700 1 --2- 172.21.15.67:0/4021324235 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f24c8073160 0x7f24c819e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:46.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.709+0000 7f24c6ffd700 1 -- 172.21.15.67:0/4021324235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b80095f0 con 0x7f24c8073a80 2024-04-22T23:46:46.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.709+0000 7f24c6ffd700 1 --2- 172.21.15.67:0/4021324235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24c8073a80 0x7f24c819edb0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f24bc00d8d0 tx=0x7f24bc00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:46:46.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.709+0000 7f24c4ff9700 1 -- 172.21.15.67:0/4021324235 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f24bc0098a0 con 0x7f24c8073a80 2024-04-22T23:46:46.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.709+0000 7f24cdad6700 1 -- 172.21.15.67:0/4021324235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24c8198bd0 con 0x7f24c8073a80 2024-04-22T23:46:46.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.710+0000 7f24c4ff9700 1 -- 172.21.15.67:0/4021324235 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f24bc010460 con 0x7f24c8073a80 2024-04-22T23:46:46.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.710+0000 7f24c4ff9700 1 -- 172.21.15.67:0/4021324235 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f24bc00f5f0 con 0x7f24c8073a80 2024-04-22T23:46:46.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.710+0000 7f24cdad6700 1 -- 172.21.15.67:0/4021324235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24c8199120 con 0x7f24c8073a80 2024-04-22T23:46:46.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.710+0000 7f24cdad6700 1 -- 172.21.15.67:0/4021324235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24c810a6a0 con 0x7f24c8073a80 2024-04-22T23:46:46.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.711+0000 7f24c4ff9700 1 -- 172.21.15.67:0/4021324235 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f24bc009a00 con 0x7f24c8073a80 2024-04-22T23:46:46.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.711+0000 7f24c4ff9700 1 --2- 172.21.15.67:0/4021324235 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f24b006ecc0 0x7f24b0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:46.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.712+0000 7f24c4ff9700 1 -- 172.21.15.67:0/4021324235 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f24bc08eea0 con 0x7f24c8073a80 2024-04-22T23:46:46.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.712+0000 7f24c77fe700 1 --2- 172.21.15.67:0/4021324235 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f24b006ecc0 0x7f24b0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:46.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.712+0000 7f24c77fe700 1 --2- 172.21.15.67:0/4021324235 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f24b006ecc0 0x7f24b0071180 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f24b8006010 tx=0x7f24b801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:46:46.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.716+0000 7f24c4ff9700 1 -- 172.21.15.67:0/4021324235 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f24bc05b610 con 0x7f24c8073a80 2024-04-22T23:46:46.708 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:46 smithi067 ceph-mon[112438]: from='client.35128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:46:46.708 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:46 smithi067 ceph-mon[112438]: from='client.35132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:46:46.708 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:46 smithi067 ceph-mon[112438]: from='client.35136 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:46:46.709 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:46 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3445333446' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:46:46.709 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:46 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3026621661' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:46:46.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.944+0000 7f24cdad6700 1 -- 172.21.15.67:0/4021324235 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f24c8199340 con 0x7f24b006ecc0 2024-04-22T23:46:46.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.946+0000 7f24c4ff9700 1 -- 172.21.15.67:0/4021324235 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f24c8199340 con 0x7f24b006ecc0 2024-04-22T23:46:46.937 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:46:46.938 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:46:46.938 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:46:46.938 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:46:46.938 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:46:46.938 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:46:46.938 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:46:46.938 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:46:46.938 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:46:46.938 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:46:46.938 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T23:46:46.938 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:46:46.938 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:46:46.938 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:46:46.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.948+0000 7f24cdad6700 1 -- 172.21.15.67:0/4021324235 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f24b006ecc0 msgr2=0x7f24b0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:46.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.948+0000 7f24cdad6700 1 --2- 172.21.15.67:0/4021324235 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f24b006ecc0 0x7f24b0071180 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f24b8006010 tx=0x7f24b801a040 comp rx=0 tx=0).stop 2024-04-22T23:46:46.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.948+0000 7f24cdad6700 1 -- 172.21.15.67:0/4021324235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24c8073a80 msgr2=0x7f24c819edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:46.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.948+0000 7f24cdad6700 1 --2- 172.21.15.67:0/4021324235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24c8073a80 0x7f24c819edb0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f24bc00d8d0 tx=0x7f24bc00dbe0 comp rx=0 tx=0).stop 2024-04-22T23:46:46.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.948+0000 7f24cdad6700 1 -- 172.21.15.67:0/4021324235 shutdown_connections 2024-04-22T23:46:46.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.948+0000 7f24cdad6700 1 --2- 172.21.15.67:0/4021324235 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f24b006ecc0 0x7f24b0071180 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:46.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.948+0000 7f24cdad6700 1 --2- 172.21.15.67:0/4021324235 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f24c8073160 0x7f24c819e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:46.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.948+0000 7f24cdad6700 1 --2- 172.21.15.67:0/4021324235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24c8073a80 0x7f24c819edb0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:46.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.948+0000 7f24cdad6700 1 -- 172.21.15.67:0/4021324235 >> 172.21.15.67:0/4021324235 conn(0x7f24c80fc750 msgr2=0x7f24c81077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:46:46.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.948+0000 7f24cdad6700 1 -- 172.21.15.67:0/4021324235 shutdown_connections 2024-04-22T23:46:46.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:46.948+0000 7f24cdad6700 1 -- 172.21.15.67:0/4021324235 wait complete. 2024-04-22T23:46:46.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:46 smithi134 ceph-mon[85630]: from='client.35128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:46:46.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:46 smithi134 ceph-mon[85630]: from='client.35132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:46:46.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:46 smithi134 ceph-mon[85630]: from='client.35136 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:46:46.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:46 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3445333446' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:46:46.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:46 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3026621661' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:46:47.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.055+0000 7f78dee66700 1 -- 172.21.15.67:0/1927431254 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78d80ffc20 msgr2=0x7f78d8100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:47.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.055+0000 7f78dee66700 1 --2- 172.21.15.67:0/1927431254 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78d80ffc20 0x7f78d8100080 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f78cc009910 tx=0x7f78cc009c20 comp rx=0 tx=0).stop 2024-04-22T23:46:47.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.055+0000 7f78dee66700 1 -- 172.21.15.67:0/1927431254 shutdown_connections 2024-04-22T23:46:47.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.055+0000 7f78dee66700 1 --2- 172.21.15.67:0/1927431254 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78d80ffc20 0x7f78d8100080 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:47.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.055+0000 7f78dee66700 1 --2- 172.21.15.67:0/1927431254 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78d8105bd0 0x7f78d8105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:47.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.055+0000 7f78dee66700 1 -- 172.21.15.67:0/1927431254 >> 172.21.15.67:0/1927431254 conn(0x7f78d8075670 msgr2=0x7f78d8075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:46:47.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.055+0000 7f78dee66700 1 -- 172.21.15.67:0/1927431254 shutdown_connections 2024-04-22T23:46:47.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.055+0000 7f78dee66700 1 -- 172.21.15.67:0/1927431254 wait complete. 2024-04-22T23:46:47.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.056+0000 7f78dee66700 1 Processor -- start 2024-04-22T23:46:47.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.056+0000 7f78dee66700 1 -- start start 2024-04-22T23:46:47.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.056+0000 7f78dee66700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78d80ffc20 0x7f78d8197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:47.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.056+0000 7f78dee66700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78d8105bd0 0x7f78d8197c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:47.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.056+0000 7f78dee66700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d8198360 con 0x7f78d8105bd0 2024-04-22T23:46:47.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.056+0000 7f78dee66700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d81984d0 con 0x7f78d80ffc20 2024-04-22T23:46:47.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.056+0000 7f78d7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78d8105bd0 0x7f78d8197c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:47.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.056+0000 7f78d7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78d8105bd0 0x7f78d8197c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56826/0 (socket says 172.21.15.67:56826) 2024-04-22T23:46:47.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.056+0000 7f78d7fff700 1 -- 172.21.15.67:0/855148764 learned_addr learned my addr 172.21.15.67:0/855148764 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:46:47.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.056+0000 7f78dcc02700 1 --2- 172.21.15.67:0/855148764 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78d80ffc20 0x7f78d8197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:47.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.056+0000 7f78d7fff700 1 -- 172.21.15.67:0/855148764 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78d80ffc20 msgr2=0x7f78d8197740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:47.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.056+0000 7f78d7fff700 1 --2- 172.21.15.67:0/855148764 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78d80ffc20 0x7f78d8197740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:47.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.056+0000 7f78d7fff700 1 -- 172.21.15.67:0/855148764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78cc0095f0 con 0x7f78d8105bd0 2024-04-22T23:46:47.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.056+0000 7f78dcc02700 1 --2- 172.21.15.67:0/855148764 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78d80ffc20 0x7f78d8197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:46:47.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.057+0000 7f78d7fff700 1 --2- 172.21.15.67:0/855148764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78d8105bd0 0x7f78d8197c80 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f78cc006010 tx=0x7f78cc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:46:47.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.057+0000 7f78d5ffb700 1 -- 172.21.15.67:0/855148764 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78cc01d070 con 0x7f78d8105bd0 2024-04-22T23:46:47.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.057+0000 7f78d5ffb700 1 -- 172.21.15.67:0/855148764 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f78cc004fb0 con 0x7f78d8105bd0 2024-04-22T23:46:47.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.057+0000 7f78dee66700 1 -- 172.21.15.67:0/855148764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78d81a07b0 con 0x7f78d8105bd0 2024-04-22T23:46:47.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.057+0000 7f78d5ffb700 1 -- 172.21.15.67:0/855148764 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78cc022ea0 con 0x7f78d8105bd0 2024-04-22T23:46:47.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.057+0000 7f78dee66700 1 -- 172.21.15.67:0/855148764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78d81a0c50 con 0x7f78d8105bd0 2024-04-22T23:46:47.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.058+0000 7f78dee66700 1 -- 172.21.15.67:0/855148764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78d81080d0 con 0x7f78d8105bd0 2024-04-22T23:46:47.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.059+0000 7f78d5ffb700 1 -- 172.21.15.67:0/855148764 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f78cc022470 con 0x7f78d8105bd0 2024-04-22T23:46:47.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.059+0000 7f78d5ffb700 1 --2- 172.21.15.67:0/855148764 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f78c006ecc0 0x7f78c0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:46:47.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.059+0000 7f78d5ffb700 1 -- 172.21.15.67:0/855148764 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f78cc0904c0 con 0x7f78d8105bd0 2024-04-22T23:46:47.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.059+0000 7f78dcc02700 1 --2- 172.21.15.67:0/855148764 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f78c006ecc0 0x7f78c0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:46:47.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.060+0000 7f78dcc02700 1 --2- 172.21.15.67:0/855148764 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f78c006ecc0 0x7f78c0071180 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f78d00060b0 tx=0x7f78d00076a0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:46:47.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.064+0000 7f78d5ffb700 1 -- 172.21.15.67:0/855148764 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f78cc059230 con 0x7f78d8105bd0 2024-04-22T23:46:47.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.362+0000 7f78dee66700 1 -- 172.21.15.67:0/855148764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f78d804f2e0 con 0x7f78d8105bd0 2024-04-22T23:46:47.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.362+0000 7f78d5ffb700 1 -- 172.21.15.67:0/855148764 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f78cc022720 con 0x7f78d8105bd0 2024-04-22T23:46:47.354 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:46:47.354 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:46:47.355 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:46:47.355 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:46:47.355 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:46:47.355 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:46:47.355 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:46:47.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.365+0000 7f78dee66700 1 -- 172.21.15.67:0/855148764 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f78c006ecc0 msgr2=0x7f78c0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:47.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.365+0000 7f78dee66700 1 --2- 172.21.15.67:0/855148764 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f78c006ecc0 0x7f78c0071180 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f78d00060b0 tx=0x7f78d00076a0 comp rx=0 tx=0).stop 2024-04-22T23:46:47.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.365+0000 7f78dee66700 1 -- 172.21.15.67:0/855148764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78d8105bd0 msgr2=0x7f78d8197c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:46:47.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.365+0000 7f78dee66700 1 --2- 172.21.15.67:0/855148764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78d8105bd0 0x7f78d8197c80 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f78cc006010 tx=0x7f78cc00be20 comp rx=0 tx=0).stop 2024-04-22T23:46:47.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.365+0000 7f78dee66700 1 -- 172.21.15.67:0/855148764 shutdown_connections 2024-04-22T23:46:47.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.365+0000 7f78dee66700 1 --2- 172.21.15.67:0/855148764 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f78c006ecc0 0x7f78c0071180 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:47.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.365+0000 7f78dee66700 1 --2- 172.21.15.67:0/855148764 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78d80ffc20 0x7f78d8197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:47.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.365+0000 7f78dee66700 1 --2- 172.21.15.67:0/855148764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78d8105bd0 0x7f78d8197c80 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:46:47.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.365+0000 7f78dee66700 1 -- 172.21.15.67:0/855148764 >> 172.21.15.67:0/855148764 conn(0x7f78d8075670 msgr2=0x7f78d80fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:46:47.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.365+0000 7f78dee66700 1 -- 172.21.15.67:0/855148764 shutdown_connections 2024-04-22T23:46:47.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:46:47.365+0000 7f78dee66700 1 -- 172.21.15.67:0/855148764 wait complete. 2024-04-22T23:46:47.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:47 smithi134 ceph-mon[85630]: pgmap v612: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:47.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:47 smithi134 ceph-mon[85630]: from='client.35148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:46:47.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:47 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/855148764' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:46:48.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:47 smithi067 ceph-mon[112438]: pgmap v612: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:48.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:47 smithi067 ceph-mon[112438]: from='client.35148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:46:48.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:47 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/855148764' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:46:49.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:49 smithi134 ceph-mon[85630]: pgmap v613: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:50.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:49 smithi067 ceph-mon[112438]: pgmap v613: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:51.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:51 smithi067 ceph-mon[112438]: pgmap v614: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:51.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:51 smithi134 ceph-mon[85630]: pgmap v614: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:53.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:53 smithi134 ceph-mon[85630]: pgmap v615: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:54.035 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:53 smithi067 ceph-mon[112438]: pgmap v615: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:55.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:55 smithi134 ceph-mon[85630]: pgmap v616: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:56.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:55 smithi067 ceph-mon[112438]: pgmap v616: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:56.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:46:57.035 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:46:57.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:57 smithi134 ceph-mon[85630]: pgmap v617: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:58.035 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:57 smithi067 ceph-mon[112438]: pgmap v617: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:46:59.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:46:59 smithi134 ceph-mon[85630]: pgmap v618: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:00.035 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:46:59 smithi067 ceph-mon[112438]: pgmap v618: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:01.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:01 smithi134 ceph-mon[85630]: pgmap v619: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:02.035 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:01 smithi067 ceph-mon[112438]: pgmap v619: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:03.994 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:03 smithi134 ceph-mon[85630]: pgmap v620: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:04.149 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:03 smithi067 ceph-mon[112438]: pgmap v620: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:05.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:05 smithi134 ceph-mon[85630]: pgmap v621: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:06.035 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:05 smithi067 ceph-mon[112438]: pgmap v621: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:07.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:07 smithi134 ceph-mon[85630]: pgmap v622: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:08.035 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:07 smithi067 ceph-mon[112438]: pgmap v622: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:09.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:09 smithi134 ceph-mon[85630]: pgmap v623: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:10.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:09 smithi067 ceph-mon[112438]: pgmap v623: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:11.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:11 smithi134 ceph-mon[85630]: pgmap v624: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:11.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:47:12.035 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:11 smithi067 ceph-mon[112438]: pgmap v624: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:12.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:47:14.035 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:13 smithi067 ceph-mon[112438]: pgmap v625: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:14.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:13 smithi134 ceph-mon[85630]: pgmap v625: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:16.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:15 smithi067 ceph-mon[112438]: pgmap v626: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:16.152 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:15 smithi134 ceph-mon[85630]: pgmap v626: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:17.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.487+0000 7fbf56021700 1 -- 172.21.15.67:0/2498074591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf500fff50 msgr2=0x7fbf50105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:17.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.487+0000 7fbf56021700 1 --2- 172.21.15.67:0/2498074591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf500fff50 0x7fbf50105840 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fbf40009910 tx=0x7fbf40009c20 comp rx=0 tx=0).stop 2024-04-22T23:47:17.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.487+0000 7fbf56021700 1 -- 172.21.15.67:0/2498074591 shutdown_connections 2024-04-22T23:47:17.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.487+0000 7fbf56021700 1 --2- 172.21.15.67:0/2498074591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf500fff50 0x7fbf50105840 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:17.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.487+0000 7fbf56021700 1 --2- 172.21.15.67:0/2498074591 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbf500ff5a0 0x7fbf500ff980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:17.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.487+0000 7fbf56021700 1 -- 172.21.15.67:0/2498074591 >> 172.21.15.67:0/2498074591 conn(0x7fbf50074e30 msgr2=0x7fbf50075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:47:17.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.487+0000 7fbf56021700 1 -- 172.21.15.67:0/2498074591 shutdown_connections 2024-04-22T23:47:17.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.487+0000 7fbf56021700 1 -- 172.21.15.67:0/2498074591 wait complete. 2024-04-22T23:47:17.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.488+0000 7fbf56021700 1 Processor -- start 2024-04-22T23:47:17.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.488+0000 7fbf56021700 1 -- start start 2024-04-22T23:47:17.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.488+0000 7fbf56021700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbf500ff5a0 0x7fbf50100520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:17.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.488+0000 7fbf56021700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf500fff50 0x7fbf50100a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:17.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.488+0000 7fbf56021700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf50100fa0 con 0x7fbf500fff50 2024-04-22T23:47:17.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.488+0000 7fbf56021700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf501010e0 con 0x7fbf500ff5a0 2024-04-22T23:47:17.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.488+0000 7fbf4effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf500fff50 0x7fbf50100a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:17.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.488+0000 7fbf4effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf500fff50 0x7fbf50100a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34764/0 (socket says 172.21.15.67:34764) 2024-04-22T23:47:17.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.488+0000 7fbf4effd700 1 -- 172.21.15.67:0/3481728859 learned_addr learned my addr 172.21.15.67:0/3481728859 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:47:17.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.489+0000 7fbf4f7fe700 1 --2- 172.21.15.67:0/3481728859 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbf500ff5a0 0x7fbf50100520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:17.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.489+0000 7fbf4effd700 1 -- 172.21.15.67:0/3481728859 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbf500ff5a0 msgr2=0x7fbf50100520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:17.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.489+0000 7fbf4effd700 1 --2- 172.21.15.67:0/3481728859 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbf500ff5a0 0x7fbf50100520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:17.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.489+0000 7fbf4effd700 1 -- 172.21.15.67:0/3481728859 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf400095f0 con 0x7fbf500fff50 2024-04-22T23:47:17.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.489+0000 7fbf4effd700 1 --2- 172.21.15.67:0/3481728859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf500fff50 0x7fbf50100a60 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fbf4000bda0 tx=0x7fbf4000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:47:17.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.489+0000 7fbf4cff9700 1 -- 172.21.15.67:0/3481728859 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf4001d070 con 0x7fbf500fff50 2024-04-22T23:47:17.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.489+0000 7fbf4cff9700 1 -- 172.21.15.67:0/3481728859 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbf40004fb0 con 0x7fbf500fff50 2024-04-22T23:47:17.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.489+0000 7fbf56021700 1 -- 172.21.15.67:0/3481728859 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf50101360 con 0x7fbf500fff50 2024-04-22T23:47:17.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.489+0000 7fbf4cff9700 1 -- 172.21.15.67:0/3481728859 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf4000f480 con 0x7fbf500fff50 2024-04-22T23:47:17.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.489+0000 7fbf56021700 1 -- 172.21.15.67:0/3481728859 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf501a2d20 con 0x7fbf500fff50 2024-04-22T23:47:17.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.490+0000 7fbf56021700 1 -- 172.21.15.67:0/3481728859 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf50109200 con 0x7fbf500fff50 2024-04-22T23:47:17.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.491+0000 7fbf4cff9700 1 -- 172.21.15.67:0/3481728859 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbf40022870 con 0x7fbf500fff50 2024-04-22T23:47:17.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.491+0000 7fbf4cff9700 1 --2- 172.21.15.67:0/3481728859 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbf3c06ecc0 0x7fbf3c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:17.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.491+0000 7fbf4cff9700 1 -- 172.21.15.67:0/3481728859 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fbf40090420 con 0x7fbf500fff50 2024-04-22T23:47:17.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.491+0000 7fbf4f7fe700 1 --2- 172.21.15.67:0/3481728859 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbf3c06ecc0 0x7fbf3c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:17.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.492+0000 7fbf4f7fe700 1 --2- 172.21.15.67:0/3481728859 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbf3c06ecc0 0x7fbf3c071180 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fbf3800ba60 tx=0x7fbf38005d50 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:47:17.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.496+0000 7fbf4cff9700 1 -- 172.21.15.67:0/3481728859 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fbf40059190 con 0x7fbf500fff50 2024-04-22T23:47:17.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.728+0000 7fbf56021700 1 -- 172.21.15.67:0/3481728859 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbf500611d0 con 0x7fbf3c06ecc0 2024-04-22T23:47:17.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.729+0000 7fbf4cff9700 1 -- 172.21.15.67:0/3481728859 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fbf500611d0 con 0x7fbf3c06ecc0 2024-04-22T23:47:17.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.731+0000 7fbf56021700 1 -- 172.21.15.67:0/3481728859 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbf3c06ecc0 msgr2=0x7fbf3c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:17.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.731+0000 7fbf56021700 1 --2- 172.21.15.67:0/3481728859 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbf3c06ecc0 0x7fbf3c071180 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fbf3800ba60 tx=0x7fbf38005d50 comp rx=0 tx=0).stop 2024-04-22T23:47:17.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.731+0000 7fbf56021700 1 -- 172.21.15.67:0/3481728859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf500fff50 msgr2=0x7fbf50100a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:17.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.731+0000 7fbf56021700 1 --2- 172.21.15.67:0/3481728859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf500fff50 0x7fbf50100a60 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fbf4000bda0 tx=0x7fbf4000be80 comp rx=0 tx=0).stop 2024-04-22T23:47:17.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.731+0000 7fbf56021700 1 -- 172.21.15.67:0/3481728859 shutdown_connections 2024-04-22T23:47:17.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.731+0000 7fbf56021700 1 --2- 172.21.15.67:0/3481728859 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbf3c06ecc0 0x7fbf3c071180 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:17.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.731+0000 7fbf56021700 1 --2- 172.21.15.67:0/3481728859 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbf500ff5a0 0x7fbf50100520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:17.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.731+0000 7fbf56021700 1 --2- 172.21.15.67:0/3481728859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf500fff50 0x7fbf50100a60 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:17.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.731+0000 7fbf56021700 1 -- 172.21.15.67:0/3481728859 >> 172.21.15.67:0/3481728859 conn(0x7fbf50074e30 msgr2=0x7fbf500fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:47:17.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.732+0000 7fbf56021700 1 -- 172.21.15.67:0/3481728859 shutdown_connections 2024-04-22T23:47:17.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.732+0000 7fbf56021700 1 -- 172.21.15.67:0/3481728859 wait complete. 2024-04-22T23:47:17.734 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:47:17.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.840+0000 7f55d18b6700 1 -- 172.21.15.67:0/3708781565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55cc101850 msgr2=0x7f55cc1058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:17.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.840+0000 7f55d18b6700 1 --2- 172.21.15.67:0/3708781565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55cc101850 0x7f55cc1058a0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f55c0009910 tx=0x7f55c0009c20 comp rx=0 tx=0).stop 2024-04-22T23:47:17.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.840+0000 7f55d18b6700 1 -- 172.21.15.67:0/3708781565 shutdown_connections 2024-04-22T23:47:17.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.840+0000 7f55d18b6700 1 --2- 172.21.15.67:0/3708781565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55cc101850 0x7f55cc1058a0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:17.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.840+0000 7f55d18b6700 1 --2- 172.21.15.67:0/3708781565 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f55cc100ea0 0x7f55cc101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:17.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.840+0000 7f55d18b6700 1 -- 172.21.15.67:0/3708781565 >> 172.21.15.67:0/3708781565 conn(0x7f55cc0fc750 msgr2=0x7f55cc0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:47:17.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.841+0000 7f55d18b6700 1 -- 172.21.15.67:0/3708781565 shutdown_connections 2024-04-22T23:47:17.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.841+0000 7f55d18b6700 1 -- 172.21.15.67:0/3708781565 wait complete. 2024-04-22T23:47:17.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.841+0000 7f55d18b6700 1 Processor -- start 2024-04-22T23:47:17.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.841+0000 7f55d18b6700 1 -- start start 2024-04-22T23:47:17.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.841+0000 7f55d18b6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55cc100ea0 0x7f55cc19c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:17.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.841+0000 7f55d18b6700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f55cc101850 0x7f55cc19cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:17.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.841+0000 7f55d18b6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55cc19d2d0 con 0x7f55cc100ea0 2024-04-22T23:47:17.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.841+0000 7f55d18b6700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55cc196780 con 0x7f55cc101850 2024-04-22T23:47:17.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.842+0000 7f55caffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55cc100ea0 0x7f55cc19c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:17.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.842+0000 7f55caffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55cc100ea0 0x7f55cc19c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34770/0 (socket says 172.21.15.67:34770) 2024-04-22T23:47:17.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.842+0000 7f55caffd700 1 -- 172.21.15.67:0/2326986047 learned_addr learned my addr 172.21.15.67:0/2326986047 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:47:17.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.842+0000 7f55ca7fc700 1 --2- 172.21.15.67:0/2326986047 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f55cc101850 0x7f55cc19cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:17.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.842+0000 7f55caffd700 1 -- 172.21.15.67:0/2326986047 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f55cc101850 msgr2=0x7f55cc19cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:17.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.842+0000 7f55caffd700 1 --2- 172.21.15.67:0/2326986047 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f55cc101850 0x7f55cc19cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:17.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.842+0000 7f55caffd700 1 -- 172.21.15.67:0/2326986047 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c00095f0 con 0x7f55cc100ea0 2024-04-22T23:47:17.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.842+0000 7f55caffd700 1 --2- 172.21.15.67:0/2326986047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55cc100ea0 0x7f55cc19c700 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f55bc00d8d0 tx=0x7f55bc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:47:17.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.842+0000 7f55d08b4700 1 -- 172.21.15.67:0/2326986047 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55bc009940 con 0x7f55cc100ea0 2024-04-22T23:47:17.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.842+0000 7f55d08b4700 1 -- 172.21.15.67:0/2326986047 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f55bc010460 con 0x7f55cc100ea0 2024-04-22T23:47:17.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.842+0000 7f55d18b6700 1 -- 172.21.15.67:0/2326986047 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55cc196a60 con 0x7f55cc100ea0 2024-04-22T23:47:17.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.842+0000 7f55d08b4700 1 -- 172.21.15.67:0/2326986047 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55bc00f5f0 con 0x7f55cc100ea0 2024-04-22T23:47:17.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.842+0000 7f55d18b6700 1 -- 172.21.15.67:0/2326986047 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55cc196fb0 con 0x7f55cc100ea0 2024-04-22T23:47:17.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.843+0000 7f55d18b6700 1 -- 172.21.15.67:0/2326986047 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55cc1092a0 con 0x7f55cc100ea0 2024-04-22T23:47:17.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.844+0000 7f55d08b4700 1 -- 172.21.15.67:0/2326986047 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f55bc009aa0 con 0x7f55cc100ea0 2024-04-22T23:47:17.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.844+0000 7f55d08b4700 1 --2- 172.21.15.67:0/2326986047 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f55b406ecc0 0x7f55b4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:17.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.844+0000 7f55ca7fc700 1 --2- 172.21.15.67:0/2326986047 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f55b406ecc0 0x7f55b4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:17.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.844+0000 7f55d08b4700 1 -- 172.21.15.67:0/2326986047 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f55bc08efc0 con 0x7f55cc100ea0 2024-04-22T23:47:17.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.845+0000 7f55ca7fc700 1 --2- 172.21.15.67:0/2326986047 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f55b406ecc0 0x7f55b4071180 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f55cc197bf0 tx=0x7f55c001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:47:17.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:17.849+0000 7f55d08b4700 1 -- 172.21.15.67:0/2326986047 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f55bc05b730 con 0x7f55cc100ea0 2024-04-22T23:47:18.035 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:17 smithi067 ceph-mon[112438]: pgmap v627: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:18.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.081+0000 7f55d18b6700 1 -- 172.21.15.67:0/2326986047 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f55cc197da0 con 0x7f55b406ecc0 2024-04-22T23:47:18.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.083+0000 7f55d08b4700 1 -- 172.21.15.67:0/2326986047 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f55cc197da0 con 0x7f55b406ecc0 2024-04-22T23:47:18.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.085+0000 7f55d18b6700 1 -- 172.21.15.67:0/2326986047 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f55b406ecc0 msgr2=0x7f55b4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:18.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.085+0000 7f55d18b6700 1 --2- 172.21.15.67:0/2326986047 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f55b406ecc0 0x7f55b4071180 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f55cc197bf0 tx=0x7f55c001a040 comp rx=0 tx=0).stop 2024-04-22T23:47:18.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.085+0000 7f55d18b6700 1 -- 172.21.15.67:0/2326986047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55cc100ea0 msgr2=0x7f55cc19c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:18.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.085+0000 7f55d18b6700 1 --2- 172.21.15.67:0/2326986047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55cc100ea0 0x7f55cc19c700 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f55bc00d8d0 tx=0x7f55bc00dc90 comp rx=0 tx=0).stop 2024-04-22T23:47:18.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.085+0000 7f55d18b6700 1 -- 172.21.15.67:0/2326986047 shutdown_connections 2024-04-22T23:47:18.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.085+0000 7f55d18b6700 1 --2- 172.21.15.67:0/2326986047 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f55b406ecc0 0x7f55b4071180 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:18.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.085+0000 7f55d18b6700 1 --2- 172.21.15.67:0/2326986047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55cc100ea0 0x7f55cc19c700 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:18.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.085+0000 7f55d18b6700 1 --2- 172.21.15.67:0/2326986047 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f55cc101850 0x7f55cc19cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:18.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.085+0000 7f55d18b6700 1 -- 172.21.15.67:0/2326986047 >> 172.21.15.67:0/2326986047 conn(0x7f55cc0fc750 msgr2=0x7f55cc0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:47:18.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.085+0000 7f55d18b6700 1 -- 172.21.15.67:0/2326986047 shutdown_connections 2024-04-22T23:47:18.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.085+0000 7f55d18b6700 1 -- 172.21.15.67:0/2326986047 wait complete. 2024-04-22T23:47:18.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.191+0000 7fa51604b700 1 -- 172.21.15.67:0/3756807659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5101027e0 msgr2=0x7fa51010acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:18.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.191+0000 7fa51604b700 1 --2- 172.21.15.67:0/3756807659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5101027e0 0x7fa51010acd0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fa500009940 tx=0x7fa500009c50 comp rx=0 tx=0).stop 2024-04-22T23:47:18.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.191+0000 7fa51604b700 1 -- 172.21.15.67:0/3756807659 shutdown_connections 2024-04-22T23:47:18.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.191+0000 7fa51604b700 1 --2- 172.21.15.67:0/3756807659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5101027e0 0x7fa51010acd0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:18.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.191+0000 7fa51604b700 1 --2- 172.21.15.67:0/3756807659 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa510101ec0 0x7fa5101022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:18.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.191+0000 7fa51604b700 1 -- 172.21.15.67:0/3756807659 >> 172.21.15.67:0/3756807659 conn(0x7fa5100fb650 msgr2=0x7fa5100fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:47:18.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.191+0000 7fa51604b700 1 -- 172.21.15.67:0/3756807659 shutdown_connections 2024-04-22T23:47:18.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.191+0000 7fa51604b700 1 -- 172.21.15.67:0/3756807659 wait complete. 2024-04-22T23:47:18.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.192+0000 7fa51604b700 1 Processor -- start 2024-04-22T23:47:18.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.192+0000 7fa51604b700 1 -- start start 2024-04-22T23:47:18.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.192+0000 7fa51604b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa510101ec0 0x7fa51019c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:18.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.192+0000 7fa51604b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5101027e0 0x7fa51019cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:18.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.192+0000 7fa51604b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51019d230 con 0x7fa5101027e0 2024-04-22T23:47:18.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.192+0000 7fa51604b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5101966e0 con 0x7fa510101ec0 2024-04-22T23:47:18.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.193+0000 7fa50effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5101027e0 0x7fa51019cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:18.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.193+0000 7fa50effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5101027e0 0x7fa51019cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34788/0 (socket says 172.21.15.67:34788) 2024-04-22T23:47:18.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.193+0000 7fa50effd700 1 -- 172.21.15.67:0/2822279421 learned_addr learned my addr 172.21.15.67:0/2822279421 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:47:18.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.193+0000 7fa50f7fe700 1 --2- 172.21.15.67:0/2822279421 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa510101ec0 0x7fa51019c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:18.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.193+0000 7fa50effd700 1 -- 172.21.15.67:0/2822279421 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa510101ec0 msgr2=0x7fa51019c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:18.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.193+0000 7fa50effd700 1 --2- 172.21.15.67:0/2822279421 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa510101ec0 0x7fa51019c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:18.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.193+0000 7fa50effd700 1 -- 172.21.15.67:0/2822279421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5000095f0 con 0x7fa5101027e0 2024-04-22T23:47:18.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.193+0000 7fa50f7fe700 1 --2- 172.21.15.67:0/2822279421 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa510101ec0 0x7fa51019c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:47:18.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.193+0000 7fa50effd700 1 --2- 172.21.15.67:0/2822279421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5101027e0 0x7fa51019cba0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fa50000b580 tx=0x7fa5000056a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:47:18.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.193+0000 7fa50cff9700 1 -- 172.21.15.67:0/2822279421 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa50001d070 con 0x7fa5101027e0 2024-04-22T23:47:18.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.193+0000 7fa51604b700 1 -- 172.21.15.67:0/2822279421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa510196960 con 0x7fa5101027e0 2024-04-22T23:47:18.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.193+0000 7fa51604b700 1 -- 172.21.15.67:0/2822279421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa510196e50 con 0x7fa5101027e0 2024-04-22T23:47:18.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.193+0000 7fa50cff9700 1 -- 172.21.15.67:0/2822279421 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa500005dc0 con 0x7fa5101027e0 2024-04-22T23:47:18.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.193+0000 7fa50cff9700 1 -- 172.21.15.67:0/2822279421 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa500021410 con 0x7fa5101027e0 2024-04-22T23:47:18.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.195+0000 7fa51604b700 1 -- 172.21.15.67:0/2822279421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4fc005320 con 0x7fa5101027e0 2024-04-22T23:47:18.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.195+0000 7fa50cff9700 1 -- 172.21.15.67:0/2822279421 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa500017400 con 0x7fa5101027e0 2024-04-22T23:47:18.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.195+0000 7fa50cff9700 1 --2- 172.21.15.67:0/2822279421 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa4f806ecc0 0x7fa4f8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:18.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.195+0000 7fa50cff9700 1 -- 172.21.15.67:0/2822279421 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fa50008ffe0 con 0x7fa5101027e0 2024-04-22T23:47:18.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.195+0000 7fa50f7fe700 1 --2- 172.21.15.67:0/2822279421 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa4f806ecc0 0x7fa4f8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:18.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.196+0000 7fa50f7fe700 1 --2- 172.21.15.67:0/2822279421 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa4f806ecc0 0x7fa4f8071180 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa504005950 tx=0x7fa5040058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:47:18.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.200+0000 7fa50cff9700 1 -- 172.21.15.67:0/2822279421 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fa50005c750 con 0x7fa5101027e0 2024-04-22T23:47:18.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:17 smithi134 ceph-mon[85630]: pgmap v627: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:18.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.419+0000 7fa51604b700 1 -- 172.21.15.67:0/2822279421 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4fc000bf0 con 0x7fa4f806ecc0 2024-04-22T23:47:18.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.430+0000 7fa50cff9700 1 -- 172.21.15.67:0/2822279421 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fa4fc000bf0 con 0x7fa4f806ecc0 2024-04-22T23:47:18.425 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:47:18.425 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (21m) 14m ago 32m 25.5M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:47:18.425 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (32m) 14m ago 32m 9114k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:47:18.426 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (31m) 13m ago 31m 9042k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:47:18.427 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (18m) 14m ago 32m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:47:18.427 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (18m) 13m ago 31m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:47:18.427 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (20m) 14m ago 31m 79.3M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:47:18.427 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (14m) 14m ago 26m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:47:18.427 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (14m) 14m ago 26m 12.6M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:47:18.427 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (14m) 13m ago 26m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:47:18.427 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (26m) 13m ago 26m 12.2M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:47:18.427 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (22m) 14m ago 33m 511M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:47:18.427 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (21m) 13m ago 31m 453M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:47:18.427 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (19m) 14m ago 33m 55.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:47:18.427 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (19m) 13m ago 31m 46.0M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:47:18.427 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (21m) 14m ago 32m 15.4M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:47:18.428 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (21m) 13m ago 31m 16.1M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:47:18.428 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (17m) 14m ago 30m 99.2M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:47:18.428 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (17m) 14m ago 30m 127M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:47:18.428 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (16m) 14m ago 29m 79.9M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:47:18.428 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (16m) 13m ago 29m 97.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:47:18.428 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (15m) 13m ago 28m 82.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:47:18.428 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (15m) 13m ago 28m 77.6M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:47:18.428 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (21m) 14m ago 31m 73.9M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:47:18.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.433+0000 7fa51604b700 1 -- 172.21.15.67:0/2822279421 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa4f806ecc0 msgr2=0x7fa4f8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:18.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.433+0000 7fa51604b700 1 --2- 172.21.15.67:0/2822279421 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa4f806ecc0 0x7fa4f8071180 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa504005950 tx=0x7fa5040058e0 comp rx=0 tx=0).stop 2024-04-22T23:47:18.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.433+0000 7fa51604b700 1 -- 172.21.15.67:0/2822279421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5101027e0 msgr2=0x7fa51019cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:18.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.433+0000 7fa51604b700 1 --2- 172.21.15.67:0/2822279421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5101027e0 0x7fa51019cba0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fa50000b580 tx=0x7fa5000056a0 comp rx=0 tx=0).stop 2024-04-22T23:47:18.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.433+0000 7fa51604b700 1 -- 172.21.15.67:0/2822279421 shutdown_connections 2024-04-22T23:47:18.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.433+0000 7fa51604b700 1 --2- 172.21.15.67:0/2822279421 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa4f806ecc0 0x7fa4f8071180 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:18.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.433+0000 7fa51604b700 1 --2- 172.21.15.67:0/2822279421 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa510101ec0 0x7fa51019c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:18.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.433+0000 7fa51604b700 1 --2- 172.21.15.67:0/2822279421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5101027e0 0x7fa51019cba0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:18.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.433+0000 7fa51604b700 1 -- 172.21.15.67:0/2822279421 >> 172.21.15.67:0/2822279421 conn(0x7fa5100fb650 msgr2=0x7fa5101054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:47:18.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.433+0000 7fa51604b700 1 -- 172.21.15.67:0/2822279421 shutdown_connections 2024-04-22T23:47:18.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.433+0000 7fa51604b700 1 -- 172.21.15.67:0/2822279421 wait complete. 2024-04-22T23:47:18.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.538+0000 7fef630f6700 1 -- 172.21.15.67:0/4133010699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c0ffb40 msgr2=0x7fef5c0fffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:18.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.538+0000 7fef630f6700 1 --2- 172.21.15.67:0/4133010699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c0ffb40 0x7fef5c0fffc0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fef4c009910 tx=0x7fef4c009c20 comp rx=0 tx=0).stop 2024-04-22T23:47:18.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.538+0000 7fef630f6700 1 -- 172.21.15.67:0/4133010699 shutdown_connections 2024-04-22T23:47:18.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.538+0000 7fef630f6700 1 --2- 172.21.15.67:0/4133010699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c0ffb40 0x7fef5c0fffc0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:18.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.538+0000 7fef630f6700 1 --2- 172.21.15.67:0/4133010699 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef5c105bf0 0x7fef5c105fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:18.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.538+0000 7fef630f6700 1 -- 172.21.15.67:0/4133010699 >> 172.21.15.67:0/4133010699 conn(0x7fef5c0fb650 msgr2=0x7fef5c0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:47:18.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.538+0000 7fef630f6700 1 -- 172.21.15.67:0/4133010699 shutdown_connections 2024-04-22T23:47:18.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.539+0000 7fef630f6700 1 -- 172.21.15.67:0/4133010699 wait complete. 2024-04-22T23:47:18.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.539+0000 7fef630f6700 1 Processor -- start 2024-04-22T23:47:18.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.539+0000 7fef630f6700 1 -- start start 2024-04-22T23:47:18.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.539+0000 7fef630f6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c0ffb40 0x7fef5c19e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:18.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.539+0000 7fef630f6700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef5c105bf0 0x7fef5c19ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:18.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.539+0000 7fef630f6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef5c19f500 con 0x7fef5c0ffb40 2024-04-22T23:47:18.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.539+0000 7fef630f6700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef5c1989b0 con 0x7fef5c105bf0 2024-04-22T23:47:18.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.539+0000 7fef60e92700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c0ffb40 0x7fef5c19e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:18.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.540+0000 7fef60e92700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c0ffb40 0x7fef5c19e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34806/0 (socket says 172.21.15.67:34806) 2024-04-22T23:47:18.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.540+0000 7fef60e92700 1 -- 172.21.15.67:0/3932027978 learned_addr learned my addr 172.21.15.67:0/3932027978 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:47:18.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.540+0000 7fef5bfff700 1 --2- 172.21.15.67:0/3932027978 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef5c105bf0 0x7fef5c19ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:18.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.540+0000 7fef60e92700 1 -- 172.21.15.67:0/3932027978 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef5c105bf0 msgr2=0x7fef5c19ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:18.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.540+0000 7fef60e92700 1 --2- 172.21.15.67:0/3932027978 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef5c105bf0 0x7fef5c19ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:18.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.540+0000 7fef60e92700 1 -- 172.21.15.67:0/3932027978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef4c0095f0 con 0x7fef5c0ffb40 2024-04-22T23:47:18.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.540+0000 7fef60e92700 1 --2- 172.21.15.67:0/3932027978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c0ffb40 0x7fef5c19e930 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fef4400b4b0 tx=0x7fef4400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:47:18.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.540+0000 7fef59ffb700 1 -- 172.21.15.67:0/3932027978 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef44011800 con 0x7fef5c0ffb40 2024-04-22T23:47:18.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.540+0000 7fef59ffb700 1 -- 172.21.15.67:0/3932027978 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef44011e20 con 0x7fef5c0ffb40 2024-04-22T23:47:18.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.540+0000 7fef630f6700 1 -- 172.21.15.67:0/3932027978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef5c198c90 con 0x7fef5c0ffb40 2024-04-22T23:47:18.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.540+0000 7fef59ffb700 1 -- 172.21.15.67:0/3932027978 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef4400f2e0 con 0x7fef5c0ffb40 2024-04-22T23:47:18.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.540+0000 7fef630f6700 1 -- 172.21.15.67:0/3932027978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef5c1991e0 con 0x7fef5c0ffb40 2024-04-22T23:47:18.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.541+0000 7fef630f6700 1 -- 172.21.15.67:0/3932027978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef5c04f2e0 con 0x7fef5c0ffb40 2024-04-22T23:47:18.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.542+0000 7fef59ffb700 1 -- 172.21.15.67:0/3932027978 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fef44010380 con 0x7fef5c0ffb40 2024-04-22T23:47:18.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.542+0000 7fef59ffb700 1 --2- 172.21.15.67:0/3932027978 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fef4806ecc0 0x7fef48071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:18.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.542+0000 7fef59ffb700 1 -- 172.21.15.67:0/3932027978 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fef4408eb00 con 0x7fef5c0ffb40 2024-04-22T23:47:18.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.542+0000 7fef5bfff700 1 --2- 172.21.15.67:0/3932027978 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fef4806ecc0 0x7fef48071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:18.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.543+0000 7fef5bfff700 1 --2- 172.21.15.67:0/3932027978 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fef4806ecc0 0x7fef48071180 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fef4c005f50 tx=0x7fef4c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:47:18.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.547+0000 7fef59ffb700 1 -- 172.21.15.67:0/3932027978 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fef4405b3a0 con 0x7fef5c0ffb40 2024-04-22T23:47:18.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.849+0000 7fef630f6700 1 -- 172.21.15.67:0/3932027978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fef5c1994b0 con 0x7fef5c0ffb40 2024-04-22T23:47:18.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.849+0000 7fef59ffb700 1 -- 172.21.15.67:0/3932027978 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7fef4405af30 con 0x7fef5c0ffb40 2024-04-22T23:47:18.841 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:47:18.841 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:47:18.841 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:47:18.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:47:18.841 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:47:18.841 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:47:18.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:47:18.841 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:47:18.842 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:47:18.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:47:18.842 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:47:18.842 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:47:18.842 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 3 2024-04-22T23:47:18.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:47:18.842 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:47:18.842 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:47:18.842 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 13 2024-04-22T23:47:18.842 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:47:18.842 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:47:18.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.852+0000 7fef630f6700 1 -- 172.21.15.67:0/3932027978 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fef4806ecc0 msgr2=0x7fef48071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:18.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.852+0000 7fef630f6700 1 --2- 172.21.15.67:0/3932027978 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fef4806ecc0 0x7fef48071180 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fef4c005f50 tx=0x7fef4c01a040 comp rx=0 tx=0).stop 2024-04-22T23:47:18.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.852+0000 7fef630f6700 1 -- 172.21.15.67:0/3932027978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c0ffb40 msgr2=0x7fef5c19e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:18.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.852+0000 7fef630f6700 1 --2- 172.21.15.67:0/3932027978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c0ffb40 0x7fef5c19e930 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fef4400b4b0 tx=0x7fef4400b870 comp rx=0 tx=0).stop 2024-04-22T23:47:18.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.852+0000 7fef630f6700 1 -- 172.21.15.67:0/3932027978 shutdown_connections 2024-04-22T23:47:18.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.852+0000 7fef630f6700 1 --2- 172.21.15.67:0/3932027978 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fef4806ecc0 0x7fef48071180 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:18.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.852+0000 7fef630f6700 1 --2- 172.21.15.67:0/3932027978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c0ffb40 0x7fef5c19e930 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:18.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.852+0000 7fef630f6700 1 --2- 172.21.15.67:0/3932027978 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fef5c105bf0 0x7fef5c19ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:18.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.852+0000 7fef630f6700 1 -- 172.21.15.67:0/3932027978 >> 172.21.15.67:0/3932027978 conn(0x7fef5c0fb650 msgr2=0x7fef5c0690b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:47:18.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.852+0000 7fef630f6700 1 -- 172.21.15.67:0/3932027978 shutdown_connections 2024-04-22T23:47:18.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.852+0000 7fef630f6700 1 -- 172.21.15.67:0/3932027978 wait complete. 2024-04-22T23:47:18.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.959+0000 7f6c17a5f700 1 -- 172.21.15.67:0/1100040661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c10100ea0 msgr2=0x7f6c10101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:18.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.959+0000 7f6c17a5f700 1 --2- 172.21.15.67:0/1100040661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c10100ea0 0x7f6c10101280 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c009910 tx=0x7f6c0c009c20 comp rx=0 tx=0).stop 2024-04-22T23:47:18.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.959+0000 7f6c17a5f700 1 -- 172.21.15.67:0/1100040661 shutdown_connections 2024-04-22T23:47:18.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.959+0000 7f6c17a5f700 1 --2- 172.21.15.67:0/1100040661 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c10101850 0x7f6c101058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:18.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.959+0000 7f6c17a5f700 1 --2- 172.21.15.67:0/1100040661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c10100ea0 0x7f6c10101280 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:18.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.959+0000 7f6c17a5f700 1 -- 172.21.15.67:0/1100040661 >> 172.21.15.67:0/1100040661 conn(0x7f6c100fc750 msgr2=0x7f6c100feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:47:18.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.959+0000 7f6c17a5f700 1 -- 172.21.15.67:0/1100040661 shutdown_connections 2024-04-22T23:47:18.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.959+0000 7f6c17a5f700 1 -- 172.21.15.67:0/1100040661 wait complete. 2024-04-22T23:47:18.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.960+0000 7f6c17a5f700 1 Processor -- start 2024-04-22T23:47:18.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.960+0000 7f6c17a5f700 1 -- start start 2024-04-22T23:47:18.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.960+0000 7f6c17a5f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c10100ea0 0x7f6c10073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:18.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.960+0000 7f6c17a5f700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c10101850 0x7f6c100735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:18.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.960+0000 7f6c17a5f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c10073b10 con 0x7f6c10100ea0 2024-04-22T23:47:18.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.960+0000 7f6c17a5f700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c10073c50 con 0x7f6c10101850 2024-04-22T23:47:18.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.960+0000 7f6c157fb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c10100ea0 0x7f6c10073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:18.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.960+0000 7f6c157fb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c10100ea0 0x7f6c10073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41528/0 (socket says 172.21.15.67:41528) 2024-04-22T23:47:18.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.960+0000 7f6c157fb700 1 -- 172.21.15.67:0/2451539304 learned_addr learned my addr 172.21.15.67:0/2451539304 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:47:18.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.960+0000 7f6c14ffa700 1 --2- 172.21.15.67:0/2451539304 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c10101850 0x7f6c100735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:18.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.960+0000 7f6c157fb700 1 -- 172.21.15.67:0/2451539304 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c10101850 msgr2=0x7f6c100735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:18.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.960+0000 7f6c157fb700 1 --2- 172.21.15.67:0/2451539304 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c10101850 0x7f6c100735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:18.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.960+0000 7f6c157fb700 1 -- 172.21.15.67:0/2451539304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c0c0095f0 con 0x7f6c10100ea0 2024-04-22T23:47:18.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.961+0000 7f6c157fb700 1 --2- 172.21.15.67:0/2451539304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c10100ea0 0x7f6c10073090 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c005f50 tx=0x7f6c0c00f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:47:18.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.961+0000 7f6c067fc700 1 -- 172.21.15.67:0/2451539304 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c0c01d070 con 0x7f6c10100ea0 2024-04-22T23:47:18.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.961+0000 7f6c067fc700 1 -- 172.21.15.67:0/2451539304 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6c0c00b610 con 0x7f6c10100ea0 2024-04-22T23:47:18.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.961+0000 7f6c17a5f700 1 -- 172.21.15.67:0/2451539304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c10073ed0 con 0x7f6c10100ea0 2024-04-22T23:47:18.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.961+0000 7f6c17a5f700 1 -- 172.21.15.67:0/2451539304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c101a2e20 con 0x7f6c10100ea0 2024-04-22T23:47:18.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.961+0000 7f6c067fc700 1 -- 172.21.15.67:0/2451539304 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c0c022470 con 0x7f6c10100ea0 2024-04-22T23:47:18.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.962+0000 7f6c17a5f700 1 -- 172.21.15.67:0/2451539304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c10109240 con 0x7f6c10100ea0 2024-04-22T23:47:18.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.962+0000 7f6c067fc700 1 -- 172.21.15.67:0/2451539304 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6c0c00fca0 con 0x7f6c10100ea0 2024-04-22T23:47:18.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.963+0000 7f6c067fc700 1 --2- 172.21.15.67:0/2451539304 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6bfc06ecc0 0x7f6bfc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:18.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.963+0000 7f6c067fc700 1 -- 172.21.15.67:0/2451539304 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f6c0c0903c0 con 0x7f6c10100ea0 2024-04-22T23:47:18.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.963+0000 7f6c14ffa700 1 --2- 172.21.15.67:0/2451539304 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6bfc06ecc0 0x7f6bfc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:18.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.963+0000 7f6c14ffa700 1 --2- 172.21.15.67:0/2451539304 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6bfc06ecc0 0x7f6bfc071180 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f6c10074880 tx=0x7f6c00009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:47:18.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:18.967+0000 7f6c067fc700 1 -- 172.21.15.67:0/2451539304 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f6c0c059020 con 0x7f6c10100ea0 2024-04-22T23:47:19.038 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:18 smithi067 ceph-mon[112438]: from='client.35156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:47:19.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.230+0000 7f6c17a5f700 1 -- 172.21.15.67:0/2451539304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f6c1004f2e0 con 0x7f6c10100ea0 2024-04-22T23:47:19.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.230+0000 7f6c067fc700 1 -- 172.21.15.67:0/2451539304 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1871 (secure 0 0 0) 0x7f6c0c05c6e0 con 0x7f6c10100ea0 2024-04-22T23:47:19.222 INFO:teuthology.orchestra.run.smithi067.stdout:e31 2024-04-22T23:47:19.222 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:47:19.222 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:47:19.222 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:47:19.222 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:47:19.222 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:47:19.222 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:47:19.223 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:47:19.223 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:47:19.223 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:47:19.223 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:47:19.223 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:47:19.223 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:47:19.223 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:47:19.223 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:47:19.223 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:47:19.223 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:47:19.223 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:47:19.223 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:47:19.223 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:47:19.223 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:47:19.223 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:47:19.224 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:47:19.224 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:47:19.224 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:47:19.224 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:47:19.224 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:47:19.224 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:47:19.224 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:47:19.224 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:47:19.224 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:47:19.224 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:47:19.224 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:47:19.224 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:47:19.224 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:47:19.224 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:47:19.224 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:47:19.225 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:47:19.225 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:47:19.225 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:47:19.225 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44259} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1827288036,v1:172.21.15.134:6827/1827288036] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:47:19.225 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:47:19.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.232+0000 7f6c17a5f700 1 -- 172.21.15.67:0/2451539304 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6bfc06ecc0 msgr2=0x7f6bfc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:19.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.232+0000 7f6c17a5f700 1 --2- 172.21.15.67:0/2451539304 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6bfc06ecc0 0x7f6bfc071180 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f6c10074880 tx=0x7f6c00009260 comp rx=0 tx=0).stop 2024-04-22T23:47:19.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.232+0000 7f6c17a5f700 1 -- 172.21.15.67:0/2451539304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c10100ea0 msgr2=0x7f6c10073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:19.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.232+0000 7f6c17a5f700 1 --2- 172.21.15.67:0/2451539304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c10100ea0 0x7f6c10073090 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c005f50 tx=0x7f6c0c00f740 comp rx=0 tx=0).stop 2024-04-22T23:47:19.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.232+0000 7f6c17a5f700 1 -- 172.21.15.67:0/2451539304 shutdown_connections 2024-04-22T23:47:19.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.232+0000 7f6c17a5f700 1 --2- 172.21.15.67:0/2451539304 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6bfc06ecc0 0x7f6bfc071180 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:19.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.232+0000 7f6c17a5f700 1 --2- 172.21.15.67:0/2451539304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c10100ea0 0x7f6c10073090 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:19.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.232+0000 7f6c17a5f700 1 --2- 172.21.15.67:0/2451539304 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c10101850 0x7f6c100735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:19.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.232+0000 7f6c17a5f700 1 -- 172.21.15.67:0/2451539304 >> 172.21.15.67:0/2451539304 conn(0x7f6c100fc750 msgr2=0x7f6c100fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:47:19.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.232+0000 7f6c17a5f700 1 -- 172.21.15.67:0/2451539304 shutdown_connections 2024-04-22T23:47:19.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.232+0000 7f6c17a5f700 1 -- 172.21.15.67:0/2451539304 wait complete. 2024-04-22T23:47:19.227 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 31 2024-04-22T23:47:19.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:18 smithi134 ceph-mon[85630]: from='client.35156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:47:19.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.339+0000 7fa099446700 1 -- 172.21.15.67:0/536565564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa094101850 msgr2=0x7fa0941058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:19.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.339+0000 7fa099446700 1 --2- 172.21.15.67:0/536565564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa094101850 0x7fa0941058a0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fa084009910 tx=0x7fa084009c20 comp rx=0 tx=0).stop 2024-04-22T23:47:19.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.339+0000 7fa099446700 1 -- 172.21.15.67:0/536565564 shutdown_connections 2024-04-22T23:47:19.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.339+0000 7fa099446700 1 --2- 172.21.15.67:0/536565564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa094101850 0x7fa0941058a0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:19.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.339+0000 7fa099446700 1 --2- 172.21.15.67:0/536565564 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa094100ea0 0x7fa094101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:19.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.339+0000 7fa099446700 1 -- 172.21.15.67:0/536565564 >> 172.21.15.67:0/536565564 conn(0x7fa0940fc750 msgr2=0x7fa0940feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:47:19.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.339+0000 7fa099446700 1 -- 172.21.15.67:0/536565564 shutdown_connections 2024-04-22T23:47:19.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.339+0000 7fa099446700 1 -- 172.21.15.67:0/536565564 wait complete. 2024-04-22T23:47:19.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.340+0000 7fa099446700 1 Processor -- start 2024-04-22T23:47:19.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.340+0000 7fa099446700 1 -- start start 2024-04-22T23:47:19.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.340+0000 7fa099446700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa094100ea0 0x7fa094073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:19.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.340+0000 7fa099446700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa094101850 0x7fa0940735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:19.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.340+0000 7fa099446700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa094073b10 con 0x7fa094100ea0 2024-04-22T23:47:19.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.340+0000 7fa099446700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa094073c50 con 0x7fa094101850 2024-04-22T23:47:19.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.340+0000 7fa092ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa094100ea0 0x7fa094073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:19.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.340+0000 7fa092ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa094100ea0 0x7fa094073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41538/0 (socket says 172.21.15.67:41538) 2024-04-22T23:47:19.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.340+0000 7fa092ffd700 1 -- 172.21.15.67:0/2762748905 learned_addr learned my addr 172.21.15.67:0/2762748905 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:47:19.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.341+0000 7fa0927fc700 1 --2- 172.21.15.67:0/2762748905 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa094101850 0x7fa0940735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:19.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.341+0000 7fa092ffd700 1 -- 172.21.15.67:0/2762748905 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa094101850 msgr2=0x7fa0940735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:19.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.341+0000 7fa092ffd700 1 --2- 172.21.15.67:0/2762748905 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa094101850 0x7fa0940735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:19.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.341+0000 7fa092ffd700 1 -- 172.21.15.67:0/2762748905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0840095f0 con 0x7fa094100ea0 2024-04-22T23:47:19.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.341+0000 7fa092ffd700 1 --2- 172.21.15.67:0/2762748905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa094100ea0 0x7fa094073090 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fa08800d8d0 tx=0x7fa08800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:47:19.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.341+0000 7fa07bfff700 1 -- 172.21.15.67:0/2762748905 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0880098a0 con 0x7fa094100ea0 2024-04-22T23:47:19.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.341+0000 7fa07bfff700 1 -- 172.21.15.67:0/2762748905 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa088010460 con 0x7fa094100ea0 2024-04-22T23:47:19.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.341+0000 7fa099446700 1 -- 172.21.15.67:0/2762748905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0941a2b00 con 0x7fa094100ea0 2024-04-22T23:47:19.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.341+0000 7fa099446700 1 -- 172.21.15.67:0/2762748905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0941a2f10 con 0x7fa094100ea0 2024-04-22T23:47:19.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.341+0000 7fa07bfff700 1 -- 172.21.15.67:0/2762748905 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa08800f5f0 con 0x7fa094100ea0 2024-04-22T23:47:19.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.342+0000 7fa099446700 1 -- 172.21.15.67:0/2762748905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa080005320 con 0x7fa094100ea0 2024-04-22T23:47:19.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.343+0000 7fa07bfff700 1 -- 172.21.15.67:0/2762748905 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa088009a00 con 0x7fa094100ea0 2024-04-22T23:47:19.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.343+0000 7fa07bfff700 1 --2- 172.21.15.67:0/2762748905 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa07c06ecc0 0x7fa07c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:19.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.343+0000 7fa07bfff700 1 -- 172.21.15.67:0/2762748905 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fa08808ee30 con 0x7fa094100ea0 2024-04-22T23:47:19.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.343+0000 7fa0927fc700 1 --2- 172.21.15.67:0/2762748905 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa07c06ecc0 0x7fa07c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:19.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.344+0000 7fa0927fc700 1 --2- 172.21.15.67:0/2762748905 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa07c06ecc0 0x7fa07c071180 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa094074880 tx=0x7fa08401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:47:19.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.348+0000 7fa07bfff700 1 -- 172.21.15.67:0/2762748905 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fa088057a70 con 0x7fa094100ea0 2024-04-22T23:47:19.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.577+0000 7fa099446700 1 -- 172.21.15.67:0/2762748905 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa080000bf0 con 0x7fa07c06ecc0 2024-04-22T23:47:19.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.579+0000 7fa07bfff700 1 -- 172.21.15.67:0/2762748905 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa080000bf0 con 0x7fa07c06ecc0 2024-04-22T23:47:19.571 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:47:19.571 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:47:19.571 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:47:19.571 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:47:19.571 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:47:19.571 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:47:19.571 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:47:19.571 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:47:19.571 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:47:19.571 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:47:19.571 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T23:47:19.572 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:47:19.572 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:47:19.572 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:47:19.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.581+0000 7fa099446700 1 -- 172.21.15.67:0/2762748905 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa07c06ecc0 msgr2=0x7fa07c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:19.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.581+0000 7fa099446700 1 --2- 172.21.15.67:0/2762748905 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa07c06ecc0 0x7fa07c071180 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa094074880 tx=0x7fa08401a040 comp rx=0 tx=0).stop 2024-04-22T23:47:19.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.581+0000 7fa099446700 1 -- 172.21.15.67:0/2762748905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa094100ea0 msgr2=0x7fa094073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:19.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.581+0000 7fa099446700 1 --2- 172.21.15.67:0/2762748905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa094100ea0 0x7fa094073090 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fa08800d8d0 tx=0x7fa08800dbe0 comp rx=0 tx=0).stop 2024-04-22T23:47:19.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.581+0000 7fa099446700 1 -- 172.21.15.67:0/2762748905 shutdown_connections 2024-04-22T23:47:19.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.581+0000 7fa099446700 1 --2- 172.21.15.67:0/2762748905 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa07c06ecc0 0x7fa07c071180 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:19.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.581+0000 7fa099446700 1 --2- 172.21.15.67:0/2762748905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa094100ea0 0x7fa094073090 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:19.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.581+0000 7fa099446700 1 --2- 172.21.15.67:0/2762748905 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa094101850 0x7fa0940735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:19.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.581+0000 7fa099446700 1 -- 172.21.15.67:0/2762748905 >> 172.21.15.67:0/2762748905 conn(0x7fa0940fc750 msgr2=0x7fa0940fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:47:19.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.581+0000 7fa099446700 1 -- 172.21.15.67:0/2762748905 shutdown_connections 2024-04-22T23:47:19.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.581+0000 7fa099446700 1 -- 172.21.15.67:0/2762748905 wait complete. 2024-04-22T23:47:19.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.687+0000 7f54396cb700 1 -- 172.21.15.67:0/1683303738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5434101de0 msgr2=0x7f54341021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:19.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.687+0000 7f54396cb700 1 --2- 172.21.15.67:0/1683303738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5434101de0 0x7f54341021c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f5424009910 tx=0x7f5424009c20 comp rx=0 tx=0).stop 2024-04-22T23:47:19.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.687+0000 7f54396cb700 1 -- 172.21.15.67:0/1683303738 shutdown_connections 2024-04-22T23:47:19.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.687+0000 7f54396cb700 1 --2- 172.21.15.67:0/1683303738 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5434102700 0x7f543410abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:19.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.687+0000 7f54396cb700 1 --2- 172.21.15.67:0/1683303738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5434101de0 0x7f54341021c0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:19.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.687+0000 7f54396cb700 1 -- 172.21.15.67:0/1683303738 >> 172.21.15.67:0/1683303738 conn(0x7f54340fb650 msgr2=0x7f54340fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:47:19.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.688+0000 7f54396cb700 1 -- 172.21.15.67:0/1683303738 shutdown_connections 2024-04-22T23:47:19.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.688+0000 7f54396cb700 1 -- 172.21.15.67:0/1683303738 wait complete. 2024-04-22T23:47:19.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.688+0000 7f54396cb700 1 Processor -- start 2024-04-22T23:47:19.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.688+0000 7f54396cb700 1 -- start start 2024-04-22T23:47:19.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.688+0000 7f54396cb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5434101de0 0x7f543419c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:19.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.688+0000 7f54396cb700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5434102700 0x7f543419cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:19.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.688+0000 7f54396cb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f543419d270 con 0x7f5434101de0 2024-04-22T23:47:19.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.688+0000 7f54396cb700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5434196720 con 0x7f5434102700 2024-04-22T23:47:19.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.689+0000 7f5432ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5434101de0 0x7f543419c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:19.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.689+0000 7f5432ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5434101de0 0x7f543419c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41548/0 (socket says 172.21.15.67:41548) 2024-04-22T23:47:19.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.689+0000 7f5432ffd700 1 -- 172.21.15.67:0/1605331173 learned_addr learned my addr 172.21.15.67:0/1605331173 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:47:19.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.689+0000 7f54327fc700 1 --2- 172.21.15.67:0/1605331173 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5434102700 0x7f543419cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:19.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.689+0000 7f5432ffd700 1 -- 172.21.15.67:0/1605331173 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5434102700 msgr2=0x7f543419cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:19.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.689+0000 7f5432ffd700 1 --2- 172.21.15.67:0/1605331173 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5434102700 0x7f543419cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:19.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.689+0000 7f5432ffd700 1 -- 172.21.15.67:0/1605331173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54240095f0 con 0x7f5434101de0 2024-04-22T23:47:19.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.689+0000 7f5432ffd700 1 --2- 172.21.15.67:0/1605331173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5434101de0 0x7f543419c6a0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f5424006010 tx=0x7f542400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:47:19.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.689+0000 7f541bfff700 1 -- 172.21.15.67:0/1605331173 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f542401d070 con 0x7f5434101de0 2024-04-22T23:47:19.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.689+0000 7f541bfff700 1 -- 172.21.15.67:0/1605331173 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f542400f460 con 0x7f5434101de0 2024-04-22T23:47:19.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.689+0000 7f54396cb700 1 -- 172.21.15.67:0/1605331173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54341969a0 con 0x7f5434101de0 2024-04-22T23:47:19.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.689+0000 7f541bfff700 1 -- 172.21.15.67:0/1605331173 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5424017610 con 0x7f5434101de0 2024-04-22T23:47:19.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.689+0000 7f54396cb700 1 -- 172.21.15.67:0/1605331173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5434196e90 con 0x7f5434101de0 2024-04-22T23:47:19.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.690+0000 7f54396cb700 1 -- 172.21.15.67:0/1605331173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54341082f0 con 0x7f5434101de0 2024-04-22T23:47:19.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.691+0000 7f541bfff700 1 -- 172.21.15.67:0/1605331173 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f542400f5d0 con 0x7f5434101de0 2024-04-22T23:47:19.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.691+0000 7f541bfff700 1 --2- 172.21.15.67:0/1605331173 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f541c06ecc0 0x7f541c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:19.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.692+0000 7f541bfff700 1 -- 172.21.15.67:0/1605331173 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f5424090710 con 0x7f5434101de0 2024-04-22T23:47:19.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.692+0000 7f54327fc700 1 --2- 172.21.15.67:0/1605331173 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f541c06ecc0 0x7f541c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:19.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.692+0000 7f54327fc700 1 --2- 172.21.15.67:0/1605331173 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f541c06ecc0 0x7f541c071180 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f5434197b90 tx=0x7f542800a250 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:47:19.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.696+0000 7f541bfff700 1 -- 172.21.15.67:0/1605331173 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f542405cf00 con 0x7f5434101de0 2024-04-22T23:47:19.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.993+0000 7f54396cb700 1 -- 172.21.15.67:0/1605331173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f543404f2e0 con 0x7f5434101de0 2024-04-22T23:47:19.985 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:19 smithi067 ceph-mon[112438]: from='client.35160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:47:19.985 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:19 smithi067 ceph-mon[112438]: pgmap v628: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:19.985 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:19 smithi067 ceph-mon[112438]: from='client.35164 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:47:19.985 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:19 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3932027978' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:47:19.985 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:19 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2451539304' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:47:19.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.993+0000 7f541bfff700 1 -- 172.21.15.67:0/1605331173 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f542405ca90 con 0x7f5434101de0 2024-04-22T23:47:19.986 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:47:19.986 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:47:19.986 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:47:19.986 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:47:19.986 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:47:19.986 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:47:19.987 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:47:19.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.995+0000 7f54396cb700 1 -- 172.21.15.67:0/1605331173 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f541c06ecc0 msgr2=0x7f541c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:19.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.995+0000 7f54396cb700 1 --2- 172.21.15.67:0/1605331173 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f541c06ecc0 0x7f541c071180 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f5434197b90 tx=0x7f542800a250 comp rx=0 tx=0).stop 2024-04-22T23:47:19.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.995+0000 7f54396cb700 1 -- 172.21.15.67:0/1605331173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5434101de0 msgr2=0x7f543419c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:19.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.995+0000 7f54396cb700 1 --2- 172.21.15.67:0/1605331173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5434101de0 0x7f543419c6a0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f5424006010 tx=0x7f542400bd70 comp rx=0 tx=0).stop 2024-04-22T23:47:19.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.995+0000 7f54396cb700 1 -- 172.21.15.67:0/1605331173 shutdown_connections 2024-04-22T23:47:19.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.995+0000 7f54396cb700 1 --2- 172.21.15.67:0/1605331173 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f541c06ecc0 0x7f541c071180 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:19.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.995+0000 7f54396cb700 1 --2- 172.21.15.67:0/1605331173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5434101de0 0x7f543419c6a0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:19.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.995+0000 7f54396cb700 1 --2- 172.21.15.67:0/1605331173 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5434102700 0x7f543419cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:19.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.995+0000 7f54396cb700 1 -- 172.21.15.67:0/1605331173 >> 172.21.15.67:0/1605331173 conn(0x7f54340fb650 msgr2=0x7f5434105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:47:19.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.995+0000 7f54396cb700 1 -- 172.21.15.67:0/1605331173 shutdown_connections 2024-04-22T23:47:19.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:19.995+0000 7f54396cb700 1 -- 172.21.15.67:0/1605331173 wait complete. 2024-04-22T23:47:20.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:19 smithi134 ceph-mon[85630]: from='client.35160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:47:20.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:19 smithi134 ceph-mon[85630]: pgmap v628: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:20.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:19 smithi134 ceph-mon[85630]: from='client.35164 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:47:20.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:19 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3932027978' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:47:20.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:19 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2451539304' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:47:21.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:20 smithi067 ceph-mon[112438]: from='client.35176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:47:21.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:20 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1605331173' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:47:21.205 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:20 smithi134 ceph-mon[85630]: from='client.35176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:47:21.205 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:20 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1605331173' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:47:22.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:21 smithi067 ceph-mon[112438]: pgmap v629: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:22.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:21 smithi134 ceph-mon[85630]: pgmap v629: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:24.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:23 smithi067 ceph-mon[112438]: pgmap v630: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:24.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:23 smithi134 ceph-mon[85630]: pgmap v630: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:26.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:25 smithi067 ceph-mon[112438]: pgmap v631: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:26.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:25 smithi134 ceph-mon[85630]: pgmap v631: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:27.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:47:27.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:47:28.035 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:27 smithi067 ceph-mon[112438]: pgmap v632: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:28.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:27 smithi134 ceph-mon[85630]: pgmap v632: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:30.035 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:29 smithi067 ceph-mon[112438]: pgmap v633: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:30.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:29 smithi134 ceph-mon[85630]: pgmap v633: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:32.035 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:31 smithi067 ceph-mon[112438]: pgmap v634: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:32.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:31 smithi134 ceph-mon[85630]: pgmap v634: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:34.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:33 smithi067 ceph-mon[112438]: pgmap v635: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:34.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:33 smithi134 ceph-mon[85630]: pgmap v635: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:36.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:35 smithi067 ceph-mon[112438]: pgmap v636: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:36.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:35 smithi134 ceph-mon[85630]: pgmap v636: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:38.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:37 smithi067 ceph-mon[112438]: pgmap v637: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:38.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:37 smithi134 ceph-mon[85630]: pgmap v637: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:40.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:39 smithi134 ceph-mon[85630]: pgmap v638: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:40.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:39 smithi067 ceph-mon[112438]: pgmap v638: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:42.060 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:41 smithi067 ceph-mon[112438]: pgmap v639: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:42.060 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:47:42.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:41 smithi134 ceph-mon[85630]: pgmap v639: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:42.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:47:44.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:43 smithi134 ceph-mon[85630]: pgmap v640: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:44.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:43 smithi067 ceph-mon[112438]: pgmap v640: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:46.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:45 smithi134 ceph-mon[85630]: pgmap v641: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:46.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:45 smithi067 ceph-mon[112438]: pgmap v641: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:48.075 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:47 smithi067 ceph-mon[112438]: pgmap v642: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:48.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:47 smithi134 ceph-mon[85630]: pgmap v642: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:50.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.117+0000 7fbe4142a700 1 -- 172.21.15.67:0/2131805564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe3c102700 msgr2=0x7fbe3c10abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:50.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.117+0000 7fbe4142a700 1 --2- 172.21.15.67:0/2131805564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe3c102700 0x7fbe3c10abf0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fbe30009910 tx=0x7fbe30009c20 comp rx=0 tx=0).stop 2024-04-22T23:47:50.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.117+0000 7fbe4142a700 1 -- 172.21.15.67:0/2131805564 shutdown_connections 2024-04-22T23:47:50.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.117+0000 7fbe4142a700 1 --2- 172.21.15.67:0/2131805564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe3c102700 0x7fbe3c10abf0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:50.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.117+0000 7fbe4142a700 1 --2- 172.21.15.67:0/2131805564 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbe3c101de0 0x7fbe3c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:50.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.117+0000 7fbe4142a700 1 -- 172.21.15.67:0/2131805564 >> 172.21.15.67:0/2131805564 conn(0x7fbe3c0fb650 msgr2=0x7fbe3c0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:47:50.111 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:49 smithi067 ceph-mon[112438]: pgmap v643: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:50.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.117+0000 7fbe4142a700 1 -- 172.21.15.67:0/2131805564 shutdown_connections 2024-04-22T23:47:50.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.117+0000 7fbe4142a700 1 -- 172.21.15.67:0/2131805564 wait complete. 2024-04-22T23:47:50.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.118+0000 7fbe4142a700 1 Processor -- start 2024-04-22T23:47:50.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.118+0000 7fbe4142a700 1 -- start start 2024-04-22T23:47:50.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.118+0000 7fbe4142a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe3c101de0 0x7fbe3c19c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:50.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.118+0000 7fbe4142a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbe3c102700 0x7fbe3c19cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:50.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.118+0000 7fbe4142a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3c19d270 con 0x7fbe3c101de0 2024-04-22T23:47:50.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.118+0000 7fbe4142a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3c196720 con 0x7fbe3c102700 2024-04-22T23:47:50.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.118+0000 7fbe3affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe3c101de0 0x7fbe3c19c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:50.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.118+0000 7fbe3affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe3c101de0 0x7fbe3c19c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42492/0 (socket says 172.21.15.67:42492) 2024-04-22T23:47:50.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.118+0000 7fbe3affd700 1 -- 172.21.15.67:0/3767002584 learned_addr learned my addr 172.21.15.67:0/3767002584 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:47:50.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.119+0000 7fbe3a7fc700 1 --2- 172.21.15.67:0/3767002584 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbe3c102700 0x7fbe3c19cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:50.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.119+0000 7fbe3affd700 1 -- 172.21.15.67:0/3767002584 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbe3c102700 msgr2=0x7fbe3c19cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:50.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.119+0000 7fbe3affd700 1 --2- 172.21.15.67:0/3767002584 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbe3c102700 0x7fbe3c19cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:50.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.119+0000 7fbe3affd700 1 -- 172.21.15.67:0/3767002584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe300095f0 con 0x7fbe3c101de0 2024-04-22T23:47:50.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.119+0000 7fbe3affd700 1 --2- 172.21.15.67:0/3767002584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe3c101de0 0x7fbe3c19c6a0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c00d8d0 tx=0x7fbe2c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:47:50.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.119+0000 7fbe23fff700 1 -- 172.21.15.67:0/3767002584 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe2c0098a0 con 0x7fbe3c101de0 2024-04-22T23:47:50.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.119+0000 7fbe4142a700 1 -- 172.21.15.67:0/3767002584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe3c196a00 con 0x7fbe3c101de0 2024-04-22T23:47:50.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.119+0000 7fbe23fff700 1 -- 172.21.15.67:0/3767002584 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbe2c010460 con 0x7fbe3c101de0 2024-04-22T23:47:50.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.119+0000 7fbe23fff700 1 -- 172.21.15.67:0/3767002584 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe2c00f5f0 con 0x7fbe3c101de0 2024-04-22T23:47:50.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.119+0000 7fbe4142a700 1 -- 172.21.15.67:0/3767002584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe3c196f50 con 0x7fbe3c101de0 2024-04-22T23:47:50.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.120+0000 7fbe4142a700 1 -- 172.21.15.67:0/3767002584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe3c1082f0 con 0x7fbe3c101de0 2024-04-22T23:47:50.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.121+0000 7fbe23fff700 1 -- 172.21.15.67:0/3767002584 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbe2c009a00 con 0x7fbe3c101de0 2024-04-22T23:47:50.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.121+0000 7fbe23fff700 1 --2- 172.21.15.67:0/3767002584 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbe2406ecc0 0x7fbe24071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:50.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.122+0000 7fbe23fff700 1 -- 172.21.15.67:0/3767002584 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fbe2c08eea0 con 0x7fbe3c101de0 2024-04-22T23:47:50.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.122+0000 7fbe3a7fc700 1 --2- 172.21.15.67:0/3767002584 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbe2406ecc0 0x7fbe24071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:50.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.122+0000 7fbe3a7fc700 1 --2- 172.21.15.67:0/3767002584 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbe2406ecc0 0x7fbe24071180 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c197b90 tx=0x7fbe3001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:47:50.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.126+0000 7fbe23fff700 1 -- 172.21.15.67:0/3767002584 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fbe2c05b610 con 0x7fbe3c101de0 2024-04-22T23:47:50.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:49 smithi134 ceph-mon[85630]: pgmap v643: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:50.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.358+0000 7fbe4142a700 1 -- 172.21.15.67:0/3767002584 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe3c197170 con 0x7fbe2406ecc0 2024-04-22T23:47:50.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.359+0000 7fbe23fff700 1 -- 172.21.15.67:0/3767002584 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fbe3c197170 con 0x7fbe2406ecc0 2024-04-22T23:47:50.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.361+0000 7fbe4142a700 1 -- 172.21.15.67:0/3767002584 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbe2406ecc0 msgr2=0x7fbe24071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:50.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.361+0000 7fbe4142a700 1 --2- 172.21.15.67:0/3767002584 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbe2406ecc0 0x7fbe24071180 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c197b90 tx=0x7fbe3001a040 comp rx=0 tx=0).stop 2024-04-22T23:47:50.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.361+0000 7fbe4142a700 1 -- 172.21.15.67:0/3767002584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe3c101de0 msgr2=0x7fbe3c19c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:50.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.361+0000 7fbe4142a700 1 --2- 172.21.15.67:0/3767002584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe3c101de0 0x7fbe3c19c6a0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c00d8d0 tx=0x7fbe2c00dbe0 comp rx=0 tx=0).stop 2024-04-22T23:47:50.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.361+0000 7fbe4142a700 1 -- 172.21.15.67:0/3767002584 shutdown_connections 2024-04-22T23:47:50.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.361+0000 7fbe4142a700 1 --2- 172.21.15.67:0/3767002584 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbe2406ecc0 0x7fbe24071180 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:50.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.361+0000 7fbe4142a700 1 --2- 172.21.15.67:0/3767002584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe3c101de0 0x7fbe3c19c6a0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:50.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.361+0000 7fbe4142a700 1 --2- 172.21.15.67:0/3767002584 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbe3c102700 0x7fbe3c19cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:50.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.361+0000 7fbe4142a700 1 -- 172.21.15.67:0/3767002584 >> 172.21.15.67:0/3767002584 conn(0x7fbe3c0fb650 msgr2=0x7fbe3c105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:47:50.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.361+0000 7fbe4142a700 1 -- 172.21.15.67:0/3767002584 shutdown_connections 2024-04-22T23:47:50.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.361+0000 7fbe4142a700 1 -- 172.21.15.67:0/3767002584 wait complete. 2024-04-22T23:47:50.365 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:47:50.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.469+0000 7fe82321a700 1 -- 172.21.15.67:0/2944615275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe81c068580 msgr2=0x7fe81c0689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:50.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.469+0000 7fe82321a700 1 --2- 172.21.15.67:0/2944615275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe81c068580 0x7fe81c0689e0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fe810009910 tx=0x7fe810009c20 comp rx=0 tx=0).stop 2024-04-22T23:47:50.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.469+0000 7fe82321a700 1 -- 172.21.15.67:0/2944615275 shutdown_connections 2024-04-22T23:47:50.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.469+0000 7fe82321a700 1 --2- 172.21.15.67:0/2944615275 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe81c068580 0x7fe81c0689e0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:50.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.469+0000 7fe82321a700 1 --2- 172.21.15.67:0/2944615275 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe81c106cd0 0x7fe81c1070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:50.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.469+0000 7fe82321a700 1 -- 172.21.15.67:0/2944615275 >> 172.21.15.67:0/2944615275 conn(0x7fe81c075670 msgr2=0x7fe81c075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:47:50.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.469+0000 7fe82321a700 1 -- 172.21.15.67:0/2944615275 shutdown_connections 2024-04-22T23:47:50.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.469+0000 7fe82321a700 1 -- 172.21.15.67:0/2944615275 wait complete. 2024-04-22T23:47:50.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.470+0000 7fe82321a700 1 Processor -- start 2024-04-22T23:47:50.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.470+0000 7fe82321a700 1 -- start start 2024-04-22T23:47:50.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.470+0000 7fe82321a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe81c068580 0x7fe81c19c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:50.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.470+0000 7fe82321a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe81c106cd0 0x7fe81c19cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:50.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.470+0000 7fe82321a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe81c19d120 con 0x7fe81c106cd0 2024-04-22T23:47:50.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.470+0000 7fe82321a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe81c1966b0 con 0x7fe81c068580 2024-04-22T23:47:50.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.470+0000 7fe81bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe81c106cd0 0x7fe81c19cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:50.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.470+0000 7fe81bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe81c106cd0 0x7fe81c19cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42514/0 (socket says 172.21.15.67:42514) 2024-04-22T23:47:50.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.470+0000 7fe81bfff700 1 -- 172.21.15.67:0/1954678515 learned_addr learned my addr 172.21.15.67:0/1954678515 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:47:50.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.470+0000 7fe820fb6700 1 --2- 172.21.15.67:0/1954678515 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe81c068580 0x7fe81c19c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:50.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.470+0000 7fe81bfff700 1 -- 172.21.15.67:0/1954678515 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe81c068580 msgr2=0x7fe81c19c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:50.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.470+0000 7fe81bfff700 1 --2- 172.21.15.67:0/1954678515 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe81c068580 0x7fe81c19c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:50.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.470+0000 7fe81bfff700 1 -- 172.21.15.67:0/1954678515 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8100095f0 con 0x7fe81c106cd0 2024-04-22T23:47:50.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.470+0000 7fe81bfff700 1 --2- 172.21.15.67:0/1954678515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe81c106cd0 0x7fe81c19cb00 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fe810006010 tx=0x7fe81000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:47:50.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.471+0000 7fe819ffb700 1 -- 172.21.15.67:0/1954678515 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe81001d070 con 0x7fe81c106cd0 2024-04-22T23:47:50.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.471+0000 7fe819ffb700 1 -- 172.21.15.67:0/1954678515 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe810005dc0 con 0x7fe81c106cd0 2024-04-22T23:47:50.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.471+0000 7fe819ffb700 1 -- 172.21.15.67:0/1954678515 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe810017400 con 0x7fe81c106cd0 2024-04-22T23:47:50.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.471+0000 7fe82321a700 1 -- 172.21.15.67:0/1954678515 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe81c196930 con 0x7fe81c106cd0 2024-04-22T23:47:50.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.471+0000 7fe82321a700 1 -- 172.21.15.67:0/1954678515 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe81c196e20 con 0x7fe81c106cd0 2024-04-22T23:47:50.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.472+0000 7fe82321a700 1 -- 172.21.15.67:0/1954678515 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe81c1091d0 con 0x7fe81c106cd0 2024-04-22T23:47:50.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.472+0000 7fe819ffb700 1 -- 172.21.15.67:0/1954678515 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe810003a60 con 0x7fe81c106cd0 2024-04-22T23:47:50.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.473+0000 7fe819ffb700 1 --2- 172.21.15.67:0/1954678515 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe80406ecc0 0x7fe804071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:50.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.473+0000 7fe819ffb700 1 -- 172.21.15.67:0/1954678515 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fe810090410 con 0x7fe81c106cd0 2024-04-22T23:47:50.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.473+0000 7fe820fb6700 1 --2- 172.21.15.67:0/1954678515 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe80406ecc0 0x7fe804071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:50.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.473+0000 7fe820fb6700 1 --2- 172.21.15.67:0/1954678515 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe80406ecc0 0x7fe804071180 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe80c005950 tx=0x7fe80c00b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:47:50.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.477+0000 7fe819ffb700 1 -- 172.21.15.67:0/1954678515 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe81005cb60 con 0x7fe81c106cd0 2024-04-22T23:47:50.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.708+0000 7fe82321a700 1 -- 172.21.15.67:0/1954678515 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe81c197090 con 0x7fe80406ecc0 2024-04-22T23:47:50.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.710+0000 7fe819ffb700 1 -- 172.21.15.67:0/1954678515 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe81c197090 con 0x7fe80406ecc0 2024-04-22T23:47:50.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.712+0000 7fe82321a700 1 -- 172.21.15.67:0/1954678515 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe80406ecc0 msgr2=0x7fe804071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:50.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.712+0000 7fe82321a700 1 --2- 172.21.15.67:0/1954678515 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe80406ecc0 0x7fe804071180 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe80c005950 tx=0x7fe80c00b410 comp rx=0 tx=0).stop 2024-04-22T23:47:50.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.712+0000 7fe82321a700 1 -- 172.21.15.67:0/1954678515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe81c106cd0 msgr2=0x7fe81c19cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:50.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.712+0000 7fe82321a700 1 --2- 172.21.15.67:0/1954678515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe81c106cd0 0x7fe81c19cb00 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fe810006010 tx=0x7fe81000bd70 comp rx=0 tx=0).stop 2024-04-22T23:47:50.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.712+0000 7fe82321a700 1 -- 172.21.15.67:0/1954678515 shutdown_connections 2024-04-22T23:47:50.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.712+0000 7fe82321a700 1 --2- 172.21.15.67:0/1954678515 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe80406ecc0 0x7fe804071180 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:50.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.712+0000 7fe82321a700 1 --2- 172.21.15.67:0/1954678515 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe81c068580 0x7fe81c19c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:50.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.712+0000 7fe82321a700 1 --2- 172.21.15.67:0/1954678515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe81c106cd0 0x7fe81c19cb00 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:50.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.712+0000 7fe82321a700 1 -- 172.21.15.67:0/1954678515 >> 172.21.15.67:0/1954678515 conn(0x7fe81c075670 msgr2=0x7fe81c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:47:50.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.712+0000 7fe82321a700 1 -- 172.21.15.67:0/1954678515 shutdown_connections 2024-04-22T23:47:50.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.712+0000 7fe82321a700 1 -- 172.21.15.67:0/1954678515 wait complete. 2024-04-22T23:47:50.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.820+0000 7fc004267700 1 -- 172.21.15.67:0/1879459613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbffc101e00 msgr2=0x7fbffc1021e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:50.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.820+0000 7fc004267700 1 --2- 172.21.15.67:0/1879459613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbffc101e00 0x7fbffc1021e0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fbfec009910 tx=0x7fbfec009c20 comp rx=0 tx=0).stop 2024-04-22T23:47:50.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.820+0000 7fc004267700 1 -- 172.21.15.67:0/1879459613 shutdown_connections 2024-04-22T23:47:50.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.820+0000 7fc004267700 1 --2- 172.21.15.67:0/1879459613 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbffc102720 0x7fbffc10ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:50.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.820+0000 7fc004267700 1 --2- 172.21.15.67:0/1879459613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbffc101e00 0x7fbffc1021e0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:50.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.820+0000 7fc004267700 1 -- 172.21.15.67:0/1879459613 >> 172.21.15.67:0/1879459613 conn(0x7fbffc0fb650 msgr2=0x7fbffc0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:47:50.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.820+0000 7fc004267700 1 -- 172.21.15.67:0/1879459613 shutdown_connections 2024-04-22T23:47:50.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.821+0000 7fc004267700 1 -- 172.21.15.67:0/1879459613 wait complete. 2024-04-22T23:47:50.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.821+0000 7fc004267700 1 Processor -- start 2024-04-22T23:47:50.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.821+0000 7fc004267700 1 -- start start 2024-04-22T23:47:50.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.821+0000 7fc004267700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbffc101e00 0x7fbffc19c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:50.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.821+0000 7fc004267700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbffc102720 0x7fbffc19cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:50.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.821+0000 7fc004267700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbffc19d260 con 0x7fbffc102720 2024-04-22T23:47:50.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.821+0000 7fc004267700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbffc196710 con 0x7fbffc101e00 2024-04-22T23:47:50.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.822+0000 7fc001802700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbffc102720 0x7fbffc19cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:50.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.822+0000 7fc001802700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbffc102720 0x7fbffc19cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42536/0 (socket says 172.21.15.67:42536) 2024-04-22T23:47:50.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.822+0000 7fc001802700 1 -- 172.21.15.67:0/3981963239 learned_addr learned my addr 172.21.15.67:0/3981963239 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:47:50.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.822+0000 7fc002003700 1 --2- 172.21.15.67:0/3981963239 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbffc101e00 0x7fbffc19c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:50.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.822+0000 7fc001802700 1 -- 172.21.15.67:0/3981963239 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbffc101e00 msgr2=0x7fbffc19c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:50.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.822+0000 7fc001802700 1 --2- 172.21.15.67:0/3981963239 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbffc101e00 0x7fbffc19c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:50.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.822+0000 7fc001802700 1 -- 172.21.15.67:0/3981963239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfec0095f0 con 0x7fbffc102720 2024-04-22T23:47:50.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.822+0000 7fc001802700 1 --2- 172.21.15.67:0/3981963239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbffc102720 0x7fbffc19cbd0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fbff800d8d0 tx=0x7fbff800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:47:50.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.822+0000 7fbff37fe700 1 -- 172.21.15.67:0/3981963239 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbff80041f0 con 0x7fbffc102720 2024-04-22T23:47:50.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.822+0000 7fbff37fe700 1 -- 172.21.15.67:0/3981963239 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbff8004350 con 0x7fbffc102720 2024-04-22T23:47:50.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.822+0000 7fbff37fe700 1 -- 172.21.15.67:0/3981963239 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbff8003c80 con 0x7fbffc102720 2024-04-22T23:47:50.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.822+0000 7fc004267700 1 -- 172.21.15.67:0/3981963239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbffc1969f0 con 0x7fbffc102720 2024-04-22T23:47:50.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.822+0000 7fc004267700 1 -- 172.21.15.67:0/3981963239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbffc196f40 con 0x7fbffc102720 2024-04-22T23:47:50.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.823+0000 7fc004267700 1 -- 172.21.15.67:0/3981963239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbffc108310 con 0x7fbffc102720 2024-04-22T23:47:50.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.824+0000 7fbff37fe700 1 -- 172.21.15.67:0/3981963239 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbff800f690 con 0x7fbffc102720 2024-04-22T23:47:50.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.824+0000 7fbff37fe700 1 --2- 172.21.15.67:0/3981963239 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbfe806ecc0 0x7fbfe8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:50.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.825+0000 7fbff37fe700 1 -- 172.21.15.67:0/3981963239 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fbff808eac0 con 0x7fbffc102720 2024-04-22T23:47:50.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.825+0000 7fc002003700 1 --2- 172.21.15.67:0/3981963239 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbfe806ecc0 0x7fbfe8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:50.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.825+0000 7fc002003700 1 --2- 172.21.15.67:0/3981963239 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbfe806ecc0 0x7fbfe8071180 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fbfec00b5c0 tx=0x7fbfec01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:47:50.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:50.829+0000 7fbff37fe700 1 -- 172.21.15.67:0/3981963239 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fbff805b2b0 con 0x7fbffc102720 2024-04-22T23:47:51.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.050+0000 7fc004267700 1 -- 172.21.15.67:0/3981963239 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fbffc197270 con 0x7fbfe806ecc0 2024-04-22T23:47:51.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.061+0000 7fbff37fe700 1 -- 172.21.15.67:0/3981963239 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fbffc197270 con 0x7fbfe806ecc0 2024-04-22T23:47:51.054 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:47:51.055 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (21m) 14m ago 32m 25.5M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:47:51.055 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (33m) 14m ago 33m 9114k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:47:51.055 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (32m) 14m ago 32m 9042k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:47:51.055 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (18m) 14m ago 33m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:47:51.055 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (18m) 14m ago 32m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:47:51.055 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (21m) 14m ago 32m 79.3M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:47:51.056 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (15m) 14m ago 27m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:47:51.056 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (14m) 14m ago 27m 12.6M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:47:51.056 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (14m) 14m ago 27m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:47:51.056 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (27m) 14m ago 27m 12.2M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:47:51.056 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (23m) 14m ago 34m 511M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:47:51.056 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (22m) 14m ago 31m 453M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:47:51.056 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (20m) 14m ago 34m 55.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:47:51.056 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (19m) 14m ago 31m 46.0M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:47:51.057 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (22m) 14m ago 32m 15.4M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:47:51.057 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (22m) 14m ago 31m 16.1M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:47:51.057 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (18m) 14m ago 31m 99.2M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:47:51.057 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (17m) 14m ago 30m 127M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:47:51.057 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (17m) 14m ago 30m 79.9M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:47:51.057 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (16m) 14m ago 29m 97.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:47:51.057 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (16m) 14m ago 29m 82.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:47:51.057 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (15m) 14m ago 28m 77.6M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:47:51.058 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (21m) 14m ago 32m 73.9M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:47:51.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.063+0000 7fc004267700 1 -- 172.21.15.67:0/3981963239 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbfe806ecc0 msgr2=0x7fbfe8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:51.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.063+0000 7fc004267700 1 --2- 172.21.15.67:0/3981963239 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbfe806ecc0 0x7fbfe8071180 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fbfec00b5c0 tx=0x7fbfec01a040 comp rx=0 tx=0).stop 2024-04-22T23:47:51.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.063+0000 7fc004267700 1 -- 172.21.15.67:0/3981963239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbffc102720 msgr2=0x7fbffc19cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:51.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.063+0000 7fc004267700 1 --2- 172.21.15.67:0/3981963239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbffc102720 0x7fbffc19cbd0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fbff800d8d0 tx=0x7fbff800dbe0 comp rx=0 tx=0).stop 2024-04-22T23:47:51.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.063+0000 7fc004267700 1 -- 172.21.15.67:0/3981963239 shutdown_connections 2024-04-22T23:47:51.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.063+0000 7fc004267700 1 --2- 172.21.15.67:0/3981963239 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbfe806ecc0 0x7fbfe8071180 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:51.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.063+0000 7fc004267700 1 --2- 172.21.15.67:0/3981963239 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbffc101e00 0x7fbffc19c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:51.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.063+0000 7fc004267700 1 --2- 172.21.15.67:0/3981963239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbffc102720 0x7fbffc19cbd0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:51.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.063+0000 7fc004267700 1 -- 172.21.15.67:0/3981963239 >> 172.21.15.67:0/3981963239 conn(0x7fbffc0fb650 msgr2=0x7fbffc105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:47:51.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.063+0000 7fc004267700 1 -- 172.21.15.67:0/3981963239 shutdown_connections 2024-04-22T23:47:51.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.063+0000 7fc004267700 1 -- 172.21.15.67:0/3981963239 wait complete. 2024-04-22T23:47:51.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.170+0000 7f604aab9700 1 -- 172.21.15.67:0/2467417836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6044101850 msgr2=0x7f60441058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:51.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.170+0000 7f604aab9700 1 --2- 172.21.15.67:0/2467417836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6044101850 0x7f60441058a0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f6038009910 tx=0x7f6038009c20 comp rx=0 tx=0).stop 2024-04-22T23:47:51.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.170+0000 7f604aab9700 1 -- 172.21.15.67:0/2467417836 shutdown_connections 2024-04-22T23:47:51.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.170+0000 7f604aab9700 1 --2- 172.21.15.67:0/2467417836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6044101850 0x7f60441058a0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:51.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.170+0000 7f604aab9700 1 --2- 172.21.15.67:0/2467417836 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6044100ea0 0x7f6044101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:51.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.170+0000 7f604aab9700 1 -- 172.21.15.67:0/2467417836 >> 172.21.15.67:0/2467417836 conn(0x7f60440fc750 msgr2=0x7f60440feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:47:51.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.170+0000 7f604aab9700 1 -- 172.21.15.67:0/2467417836 shutdown_connections 2024-04-22T23:47:51.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.170+0000 7f604aab9700 1 -- 172.21.15.67:0/2467417836 wait complete. 2024-04-22T23:47:51.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.170+0000 7f604aab9700 1 Processor -- start 2024-04-22T23:47:51.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.170+0000 7f604aab9700 1 -- start start 2024-04-22T23:47:51.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.171+0000 7f604aab9700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6044100ea0 0x7f604419c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:51.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.171+0000 7f604aab9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6044101850 0x7f604419cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:51.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.171+0000 7f604aab9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604419d2d0 con 0x7f6044101850 2024-04-22T23:47:51.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.171+0000 7f604aab9700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6044196780 con 0x7f6044100ea0 2024-04-22T23:47:51.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.171+0000 7f6043fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6044101850 0x7f604419cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:51.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.171+0000 7f6043fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6044101850 0x7f604419cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42554/0 (socket says 172.21.15.67:42554) 2024-04-22T23:47:51.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.171+0000 7f6043fff700 1 -- 172.21.15.67:0/3649778267 learned_addr learned my addr 172.21.15.67:0/3649778267 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:47:51.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.171+0000 7f6048855700 1 --2- 172.21.15.67:0/3649778267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6044100ea0 0x7f604419c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:51.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.171+0000 7f6043fff700 1 -- 172.21.15.67:0/3649778267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6044100ea0 msgr2=0x7f604419c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:51.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.171+0000 7f6043fff700 1 --2- 172.21.15.67:0/3649778267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6044100ea0 0x7f604419c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:51.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.171+0000 7f6043fff700 1 -- 172.21.15.67:0/3649778267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60380095f0 con 0x7f6044101850 2024-04-22T23:47:51.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.171+0000 7f6043fff700 1 --2- 172.21.15.67:0/3649778267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6044101850 0x7f604419cc40 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f603800bda0 tx=0x7f603800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:47:51.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.171+0000 7f6041ffb700 1 -- 172.21.15.67:0/3649778267 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f603801d070 con 0x7f6044101850 2024-04-22T23:47:51.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.171+0000 7f6041ffb700 1 -- 172.21.15.67:0/3649778267 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f603800f460 con 0x7f6044101850 2024-04-22T23:47:51.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.172+0000 7f604aab9700 1 -- 172.21.15.67:0/3649778267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6044196a00 con 0x7f6044101850 2024-04-22T23:47:51.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.172+0000 7f6041ffb700 1 -- 172.21.15.67:0/3649778267 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6038017610 con 0x7f6044101850 2024-04-22T23:47:51.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.172+0000 7f604aab9700 1 -- 172.21.15.67:0/3649778267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6044196ef0 con 0x7f6044101850 2024-04-22T23:47:51.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.173+0000 7f604aab9700 1 -- 172.21.15.67:0/3649778267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60441092a0 con 0x7f6044101850 2024-04-22T23:47:51.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.173+0000 7f6041ffb700 1 -- 172.21.15.67:0/3649778267 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f603800fa40 con 0x7f6044101850 2024-04-22T23:47:51.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.174+0000 7f6041ffb700 1 --2- 172.21.15.67:0/3649778267 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f602c06ecc0 0x7f602c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:51.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.174+0000 7f6041ffb700 1 -- 172.21.15.67:0/3649778267 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f60380903b0 con 0x7f6044101850 2024-04-22T23:47:51.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.174+0000 7f6048855700 1 --2- 172.21.15.67:0/3649778267 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f602c06ecc0 0x7f602c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:51.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.174+0000 7f6048855700 1 --2- 172.21.15.67:0/3649778267 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f602c06ecc0 0x7f602c071180 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6034005950 tx=0x7f60340058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:47:51.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.178+0000 7f6041ffb700 1 -- 172.21.15.67:0/3649778267 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f603805cba0 con 0x7f6044101850 2024-04-22T23:47:51.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.476+0000 7f604aab9700 1 -- 172.21.15.67:0/3649778267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f604404f2e0 con 0x7f6044101850 2024-04-22T23:47:51.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.476+0000 7f6041ffb700 1 -- 172.21.15.67:0/3649778267 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+805 (secure 0 0 0) 0x7f603805c730 con 0x7f6044101850 2024-04-22T23:47:51.469 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:47:51.469 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:47:51.469 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:47:51.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:47:51.470 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:47:51.470 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:47:51.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:47:51.470 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:47:51.470 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:47:51.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:47:51.470 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:47:51.470 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:47:51.470 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 3 2024-04-22T23:47:51.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:47:51.470 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:47:51.470 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-22T23:47:51.470 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 13 2024-04-22T23:47:51.471 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:47:51.471 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:47:51.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.479+0000 7f604aab9700 1 -- 172.21.15.67:0/3649778267 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f602c06ecc0 msgr2=0x7f602c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:51.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.479+0000 7f604aab9700 1 --2- 172.21.15.67:0/3649778267 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f602c06ecc0 0x7f602c071180 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6034005950 tx=0x7f60340058e0 comp rx=0 tx=0).stop 2024-04-22T23:47:51.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.479+0000 7f604aab9700 1 -- 172.21.15.67:0/3649778267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6044101850 msgr2=0x7f604419cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:51.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.479+0000 7f604aab9700 1 --2- 172.21.15.67:0/3649778267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6044101850 0x7f604419cc40 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f603800bda0 tx=0x7f603800be80 comp rx=0 tx=0).stop 2024-04-22T23:47:51.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.479+0000 7f604aab9700 1 -- 172.21.15.67:0/3649778267 shutdown_connections 2024-04-22T23:47:51.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.479+0000 7f604aab9700 1 --2- 172.21.15.67:0/3649778267 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f602c06ecc0 0x7f602c071180 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:51.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.479+0000 7f604aab9700 1 --2- 172.21.15.67:0/3649778267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6044100ea0 0x7f604419c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:51.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.479+0000 7f604aab9700 1 --2- 172.21.15.67:0/3649778267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6044101850 0x7f604419cc40 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:51.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.479+0000 7f604aab9700 1 -- 172.21.15.67:0/3649778267 >> 172.21.15.67:0/3649778267 conn(0x7f60440fc750 msgr2=0x7f60440fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:47:51.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.479+0000 7f604aab9700 1 -- 172.21.15.67:0/3649778267 shutdown_connections 2024-04-22T23:47:51.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.479+0000 7f604aab9700 1 -- 172.21.15.67:0/3649778267 wait complete. 2024-04-22T23:47:51.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.588+0000 7f6e70bea700 1 -- 172.21.15.67:0/38344012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e6c0ff5a0 msgr2=0x7f6e6c0ff980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:51.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.588+0000 7f6e70bea700 1 --2- 172.21.15.67:0/38344012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e6c0ff5a0 0x7f6e6c0ff980 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c009910 tx=0x7f6e5c009c20 comp rx=0 tx=0).stop 2024-04-22T23:47:51.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.588+0000 7f6e70bea700 1 -- 172.21.15.67:0/38344012 shutdown_connections 2024-04-22T23:47:51.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.588+0000 7f6e70bea700 1 --2- 172.21.15.67:0/38344012 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e6c0fff50 0x7f6e6c105840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:51.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.588+0000 7f6e70bea700 1 --2- 172.21.15.67:0/38344012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e6c0ff5a0 0x7f6e6c0ff980 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:51.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.588+0000 7f6e70bea700 1 -- 172.21.15.67:0/38344012 >> 172.21.15.67:0/38344012 conn(0x7f6e6c074e30 msgr2=0x7f6e6c075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:47:51.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.588+0000 7f6e70bea700 1 -- 172.21.15.67:0/38344012 shutdown_connections 2024-04-22T23:47:51.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.588+0000 7f6e70bea700 1 -- 172.21.15.67:0/38344012 wait complete. 2024-04-22T23:47:51.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.589+0000 7f6e70bea700 1 Processor -- start 2024-04-22T23:47:51.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.589+0000 7f6e70bea700 1 -- start start 2024-04-22T23:47:51.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.589+0000 7f6e70bea700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e6c0ff5a0 0x7f6e6c19c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:51.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.589+0000 7f6e70bea700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e6c0fff50 0x7f6e6c19cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:51.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.589+0000 7f6e70bea700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c19d250 con 0x7f6e6c0ff5a0 2024-04-22T23:47:51.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.589+0000 7f6e70bea700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c196700 con 0x7f6e6c0fff50 2024-04-22T23:47:51.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.589+0000 7f6e6a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e6c0ff5a0 0x7f6e6c19c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:51.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.589+0000 7f6e6a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e6c0ff5a0 0x7f6e6c19c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42578/0 (socket says 172.21.15.67:42578) 2024-04-22T23:47:51.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.589+0000 7f6e6a59c700 1 -- 172.21.15.67:0/2183704594 learned_addr learned my addr 172.21.15.67:0/2183704594 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:47:51.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.589+0000 7f6e6a59c700 1 -- 172.21.15.67:0/2183704594 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e6c0fff50 msgr2=0x7f6e6c19cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:51.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.589+0000 7f6e69d9b700 1 --2- 172.21.15.67:0/2183704594 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e6c0fff50 0x7f6e6c19cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:51.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.589+0000 7f6e6a59c700 1 --2- 172.21.15.67:0/2183704594 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e6c0fff50 0x7f6e6c19cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:51.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.589+0000 7f6e6a59c700 1 -- 172.21.15.67:0/2183704594 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e5c0095f0 con 0x7f6e6c0ff5a0 2024-04-22T23:47:51.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.589+0000 7f6e6a59c700 1 --2- 172.21.15.67:0/2183704594 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e6c0ff5a0 0x7f6e6c19c680 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c005f50 tx=0x7f6e5c00f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:47:51.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.590+0000 7f6e5b7fe700 1 -- 172.21.15.67:0/2183704594 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e5c01d070 con 0x7f6e6c0ff5a0 2024-04-22T23:47:51.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.590+0000 7f6e5b7fe700 1 -- 172.21.15.67:0/2183704594 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6e5c00b610 con 0x7f6e6c0ff5a0 2024-04-22T23:47:51.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.590+0000 7f6e70bea700 1 -- 172.21.15.67:0/2183704594 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e6c196980 con 0x7f6e6c0ff5a0 2024-04-22T23:47:51.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.590+0000 7f6e5b7fe700 1 -- 172.21.15.67:0/2183704594 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e5c022470 con 0x7f6e6c0ff5a0 2024-04-22T23:47:51.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.590+0000 7f6e70bea700 1 -- 172.21.15.67:0/2183704594 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e6c196e70 con 0x7f6e6c0ff5a0 2024-04-22T23:47:51.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.591+0000 7f6e70bea700 1 -- 172.21.15.67:0/2183704594 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e6c109270 con 0x7f6e6c0ff5a0 2024-04-22T23:47:51.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.591+0000 7f6e5b7fe700 1 -- 172.21.15.67:0/2183704594 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6e5c00fca0 con 0x7f6e6c0ff5a0 2024-04-22T23:47:51.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.592+0000 7f6e5b7fe700 1 --2- 172.21.15.67:0/2183704594 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6e5406ecc0 0x7f6e54071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:51.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.592+0000 7f6e5b7fe700 1 -- 172.21.15.67:0/2183704594 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f6e5c090440 con 0x7f6e6c0ff5a0 2024-04-22T23:47:51.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.592+0000 7f6e69d9b700 1 --2- 172.21.15.67:0/2183704594 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6e5406ecc0 0x7f6e54071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:51.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.592+0000 7f6e69d9b700 1 --2- 172.21.15.67:0/2183704594 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6e5406ecc0 0x7f6e54071180 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c197b70 tx=0x7f6e60009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:47:51.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.596+0000 7f6e5b7fe700 1 -- 172.21.15.67:0/2183704594 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f6e5c05cbb0 con 0x7f6e6c0ff5a0 2024-04-22T23:47:51.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.854+0000 7f6e70bea700 1 -- 172.21.15.67:0/2183704594 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f6e6c197d10 con 0x7f6e6c0ff5a0 2024-04-22T23:47:51.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.854+0000 7f6e5b7fe700 1 -- 172.21.15.67:0/2183704594 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1871 (secure 0 0 0) 0x7f6e6c197d10 con 0x7f6e6c0ff5a0 2024-04-22T23:47:51.847 INFO:teuthology.orchestra.run.smithi067.stdout:e31 2024-04-22T23:47:51.847 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:47:51.847 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:47:51.847 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:47:51.847 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:47:51.847 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:47:51.847 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:47:51.847 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:47:51.848 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:47:51.848 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:47:51.848 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:47:51.848 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:47:51.848 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:47:51.848 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:47:51.848 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:47:51.848 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:47:51.848 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:47:51.848 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:47:51.848 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:47:51.848 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:47:51.848 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:47:51.848 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:47:51.849 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:47:51.849 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:47:51.849 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:47:51.849 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:47:51.849 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:47:51.849 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:47:51.849 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:47:51.849 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:47:51.849 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:47:51.849 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:47:51.849 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:47:51.849 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:47:51.849 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:47:51.849 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:47:51.850 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:47:51.850 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:47:51.850 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:47:51.850 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:47:51.850 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44259} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1827288036,v1:172.21.15.134:6827/1827288036] compat {c=[1],r=[1],i=[7ff]}] 2024-04-22T23:47:51.850 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:47:51.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.856+0000 7f6e70bea700 1 -- 172.21.15.67:0/2183704594 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6e5406ecc0 msgr2=0x7f6e54071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:51.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.856+0000 7f6e70bea700 1 --2- 172.21.15.67:0/2183704594 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6e5406ecc0 0x7f6e54071180 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c197b70 tx=0x7f6e60009260 comp rx=0 tx=0).stop 2024-04-22T23:47:51.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.856+0000 7f6e70bea700 1 -- 172.21.15.67:0/2183704594 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e6c0ff5a0 msgr2=0x7f6e6c19c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:51.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.856+0000 7f6e70bea700 1 --2- 172.21.15.67:0/2183704594 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e6c0ff5a0 0x7f6e6c19c680 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c005f50 tx=0x7f6e5c00f740 comp rx=0 tx=0).stop 2024-04-22T23:47:51.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.856+0000 7f6e70bea700 1 -- 172.21.15.67:0/2183704594 shutdown_connections 2024-04-22T23:47:51.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.856+0000 7f6e70bea700 1 --2- 172.21.15.67:0/2183704594 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6e5406ecc0 0x7f6e54071180 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:51.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.856+0000 7f6e70bea700 1 --2- 172.21.15.67:0/2183704594 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e6c0ff5a0 0x7f6e6c19c680 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:51.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.856+0000 7f6e70bea700 1 --2- 172.21.15.67:0/2183704594 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e6c0fff50 0x7f6e6c19cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:51.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.856+0000 7f6e70bea700 1 -- 172.21.15.67:0/2183704594 >> 172.21.15.67:0/2183704594 conn(0x7f6e6c074e30 msgr2=0x7f6e6c0fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:47:51.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.856+0000 7f6e70bea700 1 -- 172.21.15.67:0/2183704594 shutdown_connections 2024-04-22T23:47:51.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.856+0000 7f6e70bea700 1 -- 172.21.15.67:0/2183704594 wait complete. 2024-04-22T23:47:51.852 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 31 2024-04-22T23:47:51.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.964+0000 7fb3e6f9a700 1 -- 172.21.15.67:0/3307434723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3e0101de0 msgr2=0x7fb3e01021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:51.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.964+0000 7fb3e6f9a700 1 --2- 172.21.15.67:0/3307434723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3e0101de0 0x7fb3e01021c0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fb3d800b0a0 tx=0x7fb3d800b3b0 comp rx=0 tx=0).stop 2024-04-22T23:47:51.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.964+0000 7fb3e6f9a700 1 -- 172.21.15.67:0/3307434723 shutdown_connections 2024-04-22T23:47:51.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.964+0000 7fb3e6f9a700 1 --2- 172.21.15.67:0/3307434723 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb3e0102700 0x7fb3e010abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:51.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.964+0000 7fb3e6f9a700 1 --2- 172.21.15.67:0/3307434723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3e0101de0 0x7fb3e01021c0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:51.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.964+0000 7fb3e6f9a700 1 -- 172.21.15.67:0/3307434723 >> 172.21.15.67:0/3307434723 conn(0x7fb3e00fb610 msgr2=0x7fb3e00fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:47:51.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.964+0000 7fb3e6f9a700 1 -- 172.21.15.67:0/3307434723 shutdown_connections 2024-04-22T23:47:51.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.964+0000 7fb3e6f9a700 1 -- 172.21.15.67:0/3307434723 wait complete. 2024-04-22T23:47:51.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.965+0000 7fb3e6f9a700 1 Processor -- start 2024-04-22T23:47:51.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.965+0000 7fb3e6f9a700 1 -- start start 2024-04-22T23:47:51.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.965+0000 7fb3e6f9a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3e0101de0 0x7fb3e00ffa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:51.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.965+0000 7fb3e6f9a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb3e0102700 0x7fb3e00fff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:51.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.965+0000 7fb3e6f9a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e01004c0 con 0x7fb3e0101de0 2024-04-22T23:47:51.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.965+0000 7fb3e6f9a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e0100600 con 0x7fb3e0102700 2024-04-22T23:47:51.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.965+0000 7fb3e4d36700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3e0101de0 0x7fb3e00ffa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:51.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.965+0000 7fb3e4d36700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3e0101de0 0x7fb3e00ffa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42602/0 (socket says 172.21.15.67:42602) 2024-04-22T23:47:51.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.965+0000 7fb3e4d36700 1 -- 172.21.15.67:0/3940743788 learned_addr learned my addr 172.21.15.67:0/3940743788 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:47:51.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.965+0000 7fb3dffff700 1 --2- 172.21.15.67:0/3940743788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb3e0102700 0x7fb3e00fff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:51.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.965+0000 7fb3e4d36700 1 -- 172.21.15.67:0/3940743788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb3e0102700 msgr2=0x7fb3e00fff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:51.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.965+0000 7fb3e4d36700 1 --2- 172.21.15.67:0/3940743788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb3e0102700 0x7fb3e00fff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:51.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.965+0000 7fb3e4d36700 1 -- 172.21.15.67:0/3940743788 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d8009d00 con 0x7fb3e0101de0 2024-04-22T23:47:51.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.965+0000 7fb3dffff700 1 --2- 172.21.15.67:0/3940743788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb3e0102700 0x7fb3e00fff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:47:51.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.965+0000 7fb3e4d36700 1 --2- 172.21.15.67:0/3940743788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3e0101de0 0x7fb3e00ffa40 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fb3d8000f80 tx=0x7fb3d8003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:47:51.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.966+0000 7fb3ddffb700 1 -- 172.21.15.67:0/3940743788 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3d800e040 con 0x7fb3e0101de0 2024-04-22T23:47:51.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.966+0000 7fb3ddffb700 1 -- 172.21.15.67:0/3940743788 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb3d8007c00 con 0x7fb3e0101de0 2024-04-22T23:47:51.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.966+0000 7fb3e6f9a700 1 -- 172.21.15.67:0/3940743788 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3e0100880 con 0x7fb3e0101de0 2024-04-22T23:47:51.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.966+0000 7fb3ddffb700 1 -- 172.21.15.67:0/3940743788 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3d801d6b0 con 0x7fb3e0101de0 2024-04-22T23:47:51.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.966+0000 7fb3e6f9a700 1 -- 172.21.15.67:0/3940743788 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3e01a2da0 con 0x7fb3e0101de0 2024-04-22T23:47:51.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.967+0000 7fb3e6f9a700 1 -- 172.21.15.67:0/3940743788 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3e01082f0 con 0x7fb3e0101de0 2024-04-22T23:47:51.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.968+0000 7fb3ddffb700 1 -- 172.21.15.67:0/3940743788 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb3d8004040 con 0x7fb3e0101de0 2024-04-22T23:47:51.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.968+0000 7fb3ddffb700 1 --2- 172.21.15.67:0/3940743788 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb3cc06ecc0 0x7fb3cc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:51.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.968+0000 7fb3ddffb700 1 -- 172.21.15.67:0/3940743788 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fb3d8090620 con 0x7fb3e0101de0 2024-04-22T23:47:51.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.968+0000 7fb3dffff700 1 --2- 172.21.15.67:0/3940743788 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb3cc06ecc0 0x7fb3cc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:51.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.968+0000 7fb3dffff700 1 --2- 172.21.15.67:0/3940743788 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb3cc06ecc0 0x7fb3cc071180 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb3d0005950 tx=0x7fb3d00058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:47:51.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:51.973+0000 7fb3ddffb700 1 -- 172.21.15.67:0/3940743788 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb3d80592e0 con 0x7fb3e0101de0 2024-04-22T23:47:52.169 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:51 smithi067 ceph-mon[112438]: pgmap v644: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:52.169 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:51 smithi067 ceph-mon[112438]: from='client.35184 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:47:52.169 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:51 smithi067 ceph-mon[112438]: from='client.35188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:47:52.169 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:51 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3649778267' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:47:52.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.202+0000 7fb3e6f9a700 1 -- 172.21.15.67:0/3940743788 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3e00611d0 con 0x7fb3cc06ecc0 2024-04-22T23:47:52.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.204+0000 7fb3ddffb700 1 -- 172.21.15.67:0/3940743788 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb3e00611d0 con 0x7fb3cc06ecc0 2024-04-22T23:47:52.197 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:47:52.197 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:47:52.197 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:47:52.197 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:47:52.197 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:47:52.197 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:47:52.197 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:47:52.197 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:47:52.197 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:47:52.197 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:47:52.197 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T23:47:52.197 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:47:52.198 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:47:52.198 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:47:52.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.207+0000 7fb3e6f9a700 1 -- 172.21.15.67:0/3940743788 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb3cc06ecc0 msgr2=0x7fb3cc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:52.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.207+0000 7fb3e6f9a700 1 --2- 172.21.15.67:0/3940743788 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb3cc06ecc0 0x7fb3cc071180 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb3d0005950 tx=0x7fb3d00058e0 comp rx=0 tx=0).stop 2024-04-22T23:47:52.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.207+0000 7fb3e6f9a700 1 -- 172.21.15.67:0/3940743788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3e0101de0 msgr2=0x7fb3e00ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:52.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.207+0000 7fb3e6f9a700 1 --2- 172.21.15.67:0/3940743788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3e0101de0 0x7fb3e00ffa40 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fb3d8000f80 tx=0x7fb3d8003ce0 comp rx=0 tx=0).stop 2024-04-22T23:47:52.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.207+0000 7fb3e6f9a700 1 -- 172.21.15.67:0/3940743788 shutdown_connections 2024-04-22T23:47:52.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.207+0000 7fb3e6f9a700 1 --2- 172.21.15.67:0/3940743788 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb3cc06ecc0 0x7fb3cc071180 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:52.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.207+0000 7fb3e6f9a700 1 --2- 172.21.15.67:0/3940743788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3e0101de0 0x7fb3e00ffa40 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:52.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.207+0000 7fb3e6f9a700 1 --2- 172.21.15.67:0/3940743788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb3e0102700 0x7fb3e00fff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:52.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.207+0000 7fb3e6f9a700 1 -- 172.21.15.67:0/3940743788 >> 172.21.15.67:0/3940743788 conn(0x7fb3e00fb610 msgr2=0x7fb3e0105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:47:52.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.207+0000 7fb3e6f9a700 1 -- 172.21.15.67:0/3940743788 shutdown_connections 2024-04-22T23:47:52.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.207+0000 7fb3e6f9a700 1 -- 172.21.15.67:0/3940743788 wait complete. 2024-04-22T23:47:52.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:51 smithi134 ceph-mon[85630]: pgmap v644: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:52.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:51 smithi134 ceph-mon[85630]: from='client.35184 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:47:52.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:51 smithi134 ceph-mon[85630]: from='client.35188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:47:52.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:51 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3649778267' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:47:52.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.315+0000 7f461d7be700 1 -- 172.21.15.67:0/68621551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4618100320 msgr2=0x7f461810cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:52.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.315+0000 7f461d7be700 1 --2- 172.21.15.67:0/68621551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4618100320 0x7f461810cfe0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f4600009910 tx=0x7f4600009c20 comp rx=0 tx=0).stop 2024-04-22T23:47:52.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.315+0000 7f461d7be700 1 -- 172.21.15.67:0/68621551 shutdown_connections 2024-04-22T23:47:52.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.315+0000 7f461d7be700 1 --2- 172.21.15.67:0/68621551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4618100320 0x7f461810cfe0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:52.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.315+0000 7f461d7be700 1 --2- 172.21.15.67:0/68621551 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46180ff970 0x7f46180ffd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:52.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.315+0000 7f461d7be700 1 -- 172.21.15.67:0/68621551 >> 172.21.15.67:0/68621551 conn(0x7f46180fb610 msgr2=0x7f46180fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:47:52.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.315+0000 7f461d7be700 1 -- 172.21.15.67:0/68621551 shutdown_connections 2024-04-22T23:47:52.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.315+0000 7f461d7be700 1 -- 172.21.15.67:0/68621551 wait complete. 2024-04-22T23:47:52.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.315+0000 7f461d7be700 1 Processor -- start 2024-04-22T23:47:52.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.316+0000 7f461d7be700 1 -- start start 2024-04-22T23:47:52.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.316+0000 7f461d7be700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46180ff970 0x7f461819e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:52.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.316+0000 7f461d7be700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4618100320 0x7f461819ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:52.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.316+0000 7f461d7be700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f461819f380 con 0x7f4618100320 2024-04-22T23:47:52.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.316+0000 7f461d7be700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46181988c0 con 0x7f46180ff970 2024-04-22T23:47:52.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.316+0000 7f46167fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4618100320 0x7f461819ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:52.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.316+0000 7f46167fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4618100320 0x7f461819ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42632/0 (socket says 172.21.15.67:42632) 2024-04-22T23:47:52.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.316+0000 7f46167fc700 1 -- 172.21.15.67:0/853136385 learned_addr learned my addr 172.21.15.67:0/853136385 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:47:52.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.316+0000 7f4616ffd700 1 --2- 172.21.15.67:0/853136385 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46180ff970 0x7f461819e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:52.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.316+0000 7f46167fc700 1 -- 172.21.15.67:0/853136385 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46180ff970 msgr2=0x7f461819e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:52.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.316+0000 7f46167fc700 1 --2- 172.21.15.67:0/853136385 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46180ff970 0x7f461819e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:52.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.316+0000 7f46167fc700 1 -- 172.21.15.67:0/853136385 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46000095f0 con 0x7f4618100320 2024-04-22T23:47:52.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.316+0000 7f4616ffd700 1 --2- 172.21.15.67:0/853136385 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46180ff970 0x7f461819e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:47:52.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.317+0000 7f46167fc700 1 --2- 172.21.15.67:0/853136385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4618100320 0x7f461819ed80 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f4600005b40 tx=0x7f46000056a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:47:52.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.317+0000 7f460ffff700 1 -- 172.21.15.67:0/853136385 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f460001d070 con 0x7f4618100320 2024-04-22T23:47:52.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.317+0000 7f460ffff700 1 -- 172.21.15.67:0/853136385 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f460000f460 con 0x7f4618100320 2024-04-22T23:47:52.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.317+0000 7f461d7be700 1 -- 172.21.15.67:0/853136385 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4618198b40 con 0x7f4618100320 2024-04-22T23:47:52.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.317+0000 7f460ffff700 1 -- 172.21.15.67:0/853136385 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4600021640 con 0x7f4618100320 2024-04-22T23:47:52.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.317+0000 7f461d7be700 1 -- 172.21.15.67:0/853136385 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4618199090 con 0x7f4618100320 2024-04-22T23:47:52.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.318+0000 7f461d7be700 1 -- 172.21.15.67:0/853136385 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f461810a750 con 0x7f4618100320 2024-04-22T23:47:52.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.318+0000 7f460ffff700 1 -- 172.21.15.67:0/853136385 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f460000f5d0 con 0x7f4618100320 2024-04-22T23:47:52.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.319+0000 7f460ffff700 1 --2- 172.21.15.67:0/853136385 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f460406ecc0 0x7f4604071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:47:52.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.319+0000 7f4616ffd700 1 --2- 172.21.15.67:0/853136385 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f460406ecc0 0x7f4604071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:47:52.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.319+0000 7f460ffff700 1 -- 172.21.15.67:0/853136385 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f4600090250 con 0x7f4618100320 2024-04-22T23:47:52.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.320+0000 7f4616ffd700 1 --2- 172.21.15.67:0/853136385 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f460406ecc0 0x7f4604071180 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f4608005fd0 tx=0x7f4608005e20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:47:52.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.324+0000 7f460ffff700 1 -- 172.21.15.67:0/853136385 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f460005ca40 con 0x7f4618100320 2024-04-22T23:47:52.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.623+0000 7f461d7be700 1 -- 172.21.15.67:0/853136385 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f46180679c0 con 0x7f4618100320 2024-04-22T23:47:52.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.623+0000 7f460ffff700 1 -- 172.21.15.67:0/853136385 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+423 (secure 0 0 0) 0x7f460005c5d0 con 0x7f4618100320 2024-04-22T23:47:52.616 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:47:52.616 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:47:52.616 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:47:52.616 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:47:52.617 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:47:52.617 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:47:52.617 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:47:52.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.625+0000 7f461d7be700 1 -- 172.21.15.67:0/853136385 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f460406ecc0 msgr2=0x7f4604071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:52.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.625+0000 7f461d7be700 1 --2- 172.21.15.67:0/853136385 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f460406ecc0 0x7f4604071180 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f4608005fd0 tx=0x7f4608005e20 comp rx=0 tx=0).stop 2024-04-22T23:47:52.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.625+0000 7f461d7be700 1 -- 172.21.15.67:0/853136385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4618100320 msgr2=0x7f461819ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:47:52.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.625+0000 7f461d7be700 1 --2- 172.21.15.67:0/853136385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4618100320 0x7f461819ed80 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f4600005b40 tx=0x7f46000056a0 comp rx=0 tx=0).stop 2024-04-22T23:47:52.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.625+0000 7f461d7be700 1 -- 172.21.15.67:0/853136385 shutdown_connections 2024-04-22T23:47:52.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.625+0000 7f461d7be700 1 --2- 172.21.15.67:0/853136385 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f460406ecc0 0x7f4604071180 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:52.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.625+0000 7f461d7be700 1 --2- 172.21.15.67:0/853136385 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f46180ff970 0x7f461819e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:52.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.625+0000 7f461d7be700 1 --2- 172.21.15.67:0/853136385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4618100320 0x7f461819ed80 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:47:52.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.625+0000 7f461d7be700 1 -- 172.21.15.67:0/853136385 >> 172.21.15.67:0/853136385 conn(0x7f46180fb610 msgr2=0x7f46180fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:47:52.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.626+0000 7f461d7be700 1 -- 172.21.15.67:0/853136385 shutdown_connections 2024-04-22T23:47:52.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:47:52.626+0000 7f461d7be700 1 -- 172.21.15.67:0/853136385 wait complete. 2024-04-22T23:47:53.035 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:52 smithi067 ceph-mon[112438]: from='client.35192 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:47:53.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:52 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2183704594' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:47:53.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:52 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/853136385' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:47:53.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:52 smithi134 ceph-mon[85630]: from='client.35192 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:47:53.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:52 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2183704594' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:47:53.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:52 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/853136385' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:47:54.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:53 smithi134 ceph-mon[85630]: from='client.35204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:47:54.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:53 smithi134 ceph-mon[85630]: pgmap v645: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:54.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:53 smithi067 ceph-mon[112438]: from='client.35204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:47:54.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:53 smithi067 ceph-mon[112438]: pgmap v645: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:56.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:55 smithi134 ceph-mon[85630]: pgmap v646: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:56.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:55 smithi067 ceph-mon[112438]: pgmap v646: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:56.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:47:57.220 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:47:58.074 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:57 smithi067 ceph-mon[112438]: pgmap v647: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:47:58.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:57 smithi134 ceph-mon[85630]: pgmap v647: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:00.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:47:59 smithi134 ceph-mon[85630]: pgmap v648: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:00.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:47:59 smithi067 ceph-mon[112438]: pgmap v648: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:01.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:01 smithi134 ceph-mon[85630]: pgmap v649: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:02.275 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:01 smithi067 ceph-mon[112438]: pgmap v649: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:04.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:03 smithi134 ceph-mon[85630]: pgmap v650: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:04.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:03 smithi067 ceph-mon[112438]: pgmap v650: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:06.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:05 smithi134 ceph-mon[85630]: pgmap v651: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:06.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:05 smithi067 ceph-mon[112438]: pgmap v651: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:08.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:07 smithi134 ceph-mon[85630]: pgmap v652: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:08.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:07 smithi067 ceph-mon[112438]: pgmap v652: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:10.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:09 smithi134 ceph-mon[85630]: pgmap v653: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:10.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:09 smithi067 ceph-mon[112438]: pgmap v653: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:12.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:11 smithi134 ceph-mon[85630]: pgmap v654: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:12.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:48:12.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:11 smithi067 ceph-mon[112438]: pgmap v654: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:12.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:48:14.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:13 smithi134 ceph-mon[85630]: pgmap v655: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:14.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:13 smithi067 ceph-mon[112438]: pgmap v655: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:16.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:15 smithi134 ceph-mon[85630]: pgmap v656: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:16.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:15 smithi067 ceph-mon[112438]: pgmap v656: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:17.936 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:17 smithi134 ceph-mon[85630]: pgmap v657: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:17.936 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:17 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:48:17.936 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:17 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:48:17.936 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:17 smithi134 ceph-mon[85630]: pgmap v658: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:17.936 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:17 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:48:17.936 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:17 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:48:17.936 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:17 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:48:17.936 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:17 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:48:17.936 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:17 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:48:17.937 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:17 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:48:18.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:17 smithi067 ceph-mon[112438]: pgmap v657: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:18.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:17 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:48:18.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:17 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T23:48:18.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:17 smithi067 ceph-mon[112438]: pgmap v658: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:18.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:17 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:48:18.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:17 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:48:18.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:17 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:48:18.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:17 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:48:18.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:17 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:48:18.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:17 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:48:18.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:18 smithi134 ceph-mon[85630]: Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2024-04-22T23:48:18.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:18 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:48:18.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:18 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi134.vuxbsj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T23:48:18.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:18 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:48:19.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:18 smithi067 ceph-mon[112438]: Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2024-04-22T23:48:19.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:18 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:48:19.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:18 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi134.vuxbsj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T23:48:19.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:18 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T23:48:20.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:19 smithi134 ceph-mon[85630]: Upgrade: Updating mds.cephfs.smithi134.vuxbsj 2024-04-22T23:48:20.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:19 smithi134 ceph-mon[85630]: Deploying daemon mds.cephfs.smithi134.vuxbsj on smithi134 2024-04-22T23:48:20.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:19 smithi134 ceph-mon[85630]: pgmap v659: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:20.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:19 smithi067 ceph-mon[112438]: Upgrade: Updating mds.cephfs.smithi134.vuxbsj 2024-04-22T23:48:20.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:19 smithi067 ceph-mon[112438]: Deploying daemon mds.cephfs.smithi134.vuxbsj on smithi134 2024-04-22T23:48:20.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:19 smithi067 ceph-mon[112438]: pgmap v659: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:22.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:21 smithi134 ceph-mon[85630]: osdmap e94: 6 total, 6 up, 6 in 2024-04-22T23:48:22.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:21 smithi134 ceph-mon[85630]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-22T23:48:22.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:21 smithi134 ceph-mon[85630]: pgmap v661: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:22.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:21 smithi067 ceph-mon[112438]: osdmap e94: 6 total, 6 up, 6 in 2024-04-22T23:48:22.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:21 smithi067 ceph-mon[112438]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-22T23:48:22.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:21 smithi067 ceph-mon[112438]: pgmap v661: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:22.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.745+0000 7f8c11343700 1 -- 172.21.15.67:0/1055499454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c0c106cd0 msgr2=0x7f8c0c1070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:22.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.745+0000 7f8c11343700 1 --2- 172.21.15.67:0/1055499454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c0c106cd0 0x7f8c0c1070b0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc009910 tx=0x7f8bfc009c20 comp rx=0 tx=0).stop 2024-04-22T23:48:22.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.745+0000 7f8c11343700 1 -- 172.21.15.67:0/1055499454 shutdown_connections 2024-04-22T23:48:22.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.745+0000 7f8c11343700 1 --2- 172.21.15.67:0/1055499454 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c0c068580 0x7f8c0c0689e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:22.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.745+0000 7f8c11343700 1 --2- 172.21.15.67:0/1055499454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c0c106cd0 0x7f8c0c1070b0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:22.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.745+0000 7f8c11343700 1 -- 172.21.15.67:0/1055499454 >> 172.21.15.67:0/1055499454 conn(0x7f8c0c075670 msgr2=0x7f8c0c075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:48:22.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.745+0000 7f8c11343700 1 -- 172.21.15.67:0/1055499454 shutdown_connections 2024-04-22T23:48:22.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.745+0000 7f8c11343700 1 -- 172.21.15.67:0/1055499454 wait complete. 2024-04-22T23:48:22.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.746+0000 7f8c11343700 1 Processor -- start 2024-04-22T23:48:22.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.746+0000 7f8c11343700 1 -- start start 2024-04-22T23:48:22.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.746+0000 7f8c11343700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c0c068580 0x7f8c0c0ffcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:22.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.746+0000 7f8c11343700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c0c106cd0 0x7f8c0c100230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:22.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.746+0000 7f8c11343700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0c100770 con 0x7f8c0c106cd0 2024-04-22T23:48:22.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.746+0000 7f8c11343700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0c1008b0 con 0x7f8c0c068580 2024-04-22T23:48:22.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.746+0000 7f8c0a7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c0c106cd0 0x7f8c0c100230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:22.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.746+0000 7f8c0a7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c0c106cd0 0x7f8c0c100230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54880/0 (socket says 172.21.15.67:54880) 2024-04-22T23:48:22.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.746+0000 7f8c0a7fc700 1 -- 172.21.15.67:0/2562984082 learned_addr learned my addr 172.21.15.67:0/2562984082 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:48:22.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.746+0000 7f8c0affd700 1 --2- 172.21.15.67:0/2562984082 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c0c068580 0x7f8c0c0ffcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:22.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.747+0000 7f8c0a7fc700 1 -- 172.21.15.67:0/2562984082 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c0c068580 msgr2=0x7f8c0c0ffcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:22.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.747+0000 7f8c0a7fc700 1 --2- 172.21.15.67:0/2562984082 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c0c068580 0x7f8c0c0ffcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:22.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.747+0000 7f8c0a7fc700 1 -- 172.21.15.67:0/2562984082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bfc0095f0 con 0x7f8c0c106cd0 2024-04-22T23:48:22.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.747+0000 7f8c0a7fc700 1 --2- 172.21.15.67:0/2562984082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c0c106cd0 0x7f8c0c100230 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f8c0000c930 tx=0x7f8c0000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:48:22.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.747+0000 7f8bf3fff700 1 -- 172.21.15.67:0/2562984082 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c00007ad0 con 0x7f8c0c106cd0 2024-04-22T23:48:22.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.747+0000 7f8bf3fff700 1 -- 172.21.15.67:0/2562984082 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8c0000ce80 con 0x7f8c0c106cd0 2024-04-22T23:48:22.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.747+0000 7f8c11343700 1 -- 172.21.15.67:0/2562984082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c0c1a2a60 con 0x7f8c0c106cd0 2024-04-22T23:48:22.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.747+0000 7f8bf3fff700 1 -- 172.21.15.67:0/2562984082 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c00018630 con 0x7f8c0c106cd0 2024-04-22T23:48:22.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.747+0000 7f8c11343700 1 -- 172.21.15.67:0/2562984082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c0c1a2dc0 con 0x7f8c0c106cd0 2024-04-22T23:48:22.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.748+0000 7f8c11343700 1 -- 172.21.15.67:0/2562984082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c0c1091d0 con 0x7f8c0c106cd0 2024-04-22T23:48:22.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.749+0000 7f8bf3fff700 1 -- 172.21.15.67:0/2562984082 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8c00007c30 con 0x7f8c0c106cd0 2024-04-22T23:48:22.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.749+0000 7f8bf3fff700 1 --2- 172.21.15.67:0/2562984082 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8bf406ecc0 0x7f8bf4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:22.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.749+0000 7f8bf3fff700 1 -- 172.21.15.67:0/2562984082 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f8c0008f010 con 0x7f8c0c106cd0 2024-04-22T23:48:22.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.749+0000 7f8c0affd700 1 --2- 172.21.15.67:0/2562984082 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8bf406ecc0 0x7f8bf4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:22.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.750+0000 7f8c0affd700 1 --2- 172.21.15.67:0/2562984082 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8bf406ecc0 0x7f8bf4071180 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc006010 tx=0x7f8bfc01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:48:22.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.754+0000 7f8bf3fff700 1 -- 172.21.15.67:0/2562984082 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f8c00057c50 con 0x7f8c0c106cd0 2024-04-22T23:48:22.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.985+0000 7f8c11343700 1 -- 172.21.15.67:0/2562984082 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8c0c02cd30 con 0x7f8bf406ecc0 2024-04-22T23:48:22.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.987+0000 7f8bf3fff700 1 -- 172.21.15.67:0/2562984082 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f8c0c02cd30 con 0x7f8bf406ecc0 2024-04-22T23:48:22.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.989+0000 7f8c11343700 1 -- 172.21.15.67:0/2562984082 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8bf406ecc0 msgr2=0x7f8bf4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:22.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.989+0000 7f8c11343700 1 --2- 172.21.15.67:0/2562984082 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8bf406ecc0 0x7f8bf4071180 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc006010 tx=0x7f8bfc01a040 comp rx=0 tx=0).stop 2024-04-22T23:48:22.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.989+0000 7f8c11343700 1 -- 172.21.15.67:0/2562984082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c0c106cd0 msgr2=0x7f8c0c100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:22.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.989+0000 7f8c11343700 1 --2- 172.21.15.67:0/2562984082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c0c106cd0 0x7f8c0c100230 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f8c0000c930 tx=0x7f8c0000cc40 comp rx=0 tx=0).stop 2024-04-22T23:48:22.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.989+0000 7f8c11343700 1 -- 172.21.15.67:0/2562984082 shutdown_connections 2024-04-22T23:48:22.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.989+0000 7f8c11343700 1 --2- 172.21.15.67:0/2562984082 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8bf406ecc0 0x7f8bf4071180 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:22.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.989+0000 7f8c11343700 1 --2- 172.21.15.67:0/2562984082 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c0c068580 0x7f8c0c0ffcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:22.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.989+0000 7f8c11343700 1 --2- 172.21.15.67:0/2562984082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c0c106cd0 0x7f8c0c100230 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:22.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.989+0000 7f8c11343700 1 -- 172.21.15.67:0/2562984082 >> 172.21.15.67:0/2562984082 conn(0x7f8c0c075670 msgr2=0x7f8c0c0fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:48:22.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.989+0000 7f8c11343700 1 -- 172.21.15.67:0/2562984082 shutdown_connections 2024-04-22T23:48:22.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:22.989+0000 7f8c11343700 1 -- 172.21.15.67:0/2562984082 wait complete. 2024-04-22T23:48:22.991 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:48:23.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.098+0000 7fe275a61700 1 -- 172.21.15.67:0/393817700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe270068580 msgr2=0x7fe2700689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:23.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.098+0000 7fe275a61700 1 --2- 172.21.15.67:0/393817700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe270068580 0x7fe2700689e0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fe258009910 tx=0x7fe258009c20 comp rx=0 tx=0).stop 2024-04-22T23:48:23.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.098+0000 7fe275a61700 1 -- 172.21.15.67:0/393817700 shutdown_connections 2024-04-22T23:48:23.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.098+0000 7fe275a61700 1 --2- 172.21.15.67:0/393817700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe270068580 0x7fe2700689e0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:23.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.098+0000 7fe275a61700 1 --2- 172.21.15.67:0/393817700 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe270106cd0 0x7fe2701070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:23.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.098+0000 7fe275a61700 1 -- 172.21.15.67:0/393817700 >> 172.21.15.67:0/393817700 conn(0x7fe270075670 msgr2=0x7fe270075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:48:23.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.098+0000 7fe275a61700 1 -- 172.21.15.67:0/393817700 shutdown_connections 2024-04-22T23:48:23.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.098+0000 7fe275a61700 1 -- 172.21.15.67:0/393817700 wait complete. 2024-04-22T23:48:23.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.099+0000 7fe275a61700 1 Processor -- start 2024-04-22T23:48:23.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.099+0000 7fe275a61700 1 -- start start 2024-04-22T23:48:23.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.099+0000 7fe275a61700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe270068580 0x7fe2700ffcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:23.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.099+0000 7fe275a61700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe270106cd0 0x7fe270100230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:23.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.099+0000 7fe275a61700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe270100770 con 0x7fe270106cd0 2024-04-22T23:48:23.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.099+0000 7fe275a61700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2701008b0 con 0x7fe270068580 2024-04-22T23:48:23.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.099+0000 7fe26e7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe270106cd0 0x7fe270100230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:23.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.099+0000 7fe26e7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe270106cd0 0x7fe270100230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54910/0 (socket says 172.21.15.67:54910) 2024-04-22T23:48:23.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.099+0000 7fe26e7fc700 1 -- 172.21.15.67:0/3352168302 learned_addr learned my addr 172.21.15.67:0/3352168302 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:48:23.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.099+0000 7fe26effd700 1 --2- 172.21.15.67:0/3352168302 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe270068580 0x7fe2700ffcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:23.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.099+0000 7fe26e7fc700 1 -- 172.21.15.67:0/3352168302 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe270068580 msgr2=0x7fe2700ffcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:23.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.099+0000 7fe26e7fc700 1 --2- 172.21.15.67:0/3352168302 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe270068580 0x7fe2700ffcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:23.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.099+0000 7fe26e7fc700 1 -- 172.21.15.67:0/3352168302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2580095f0 con 0x7fe270106cd0 2024-04-22T23:48:23.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.100+0000 7fe26e7fc700 1 --2- 172.21.15.67:0/3352168302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe270106cd0 0x7fe270100230 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fe258006010 tx=0x7fe25800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:48:23.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.100+0000 7fe274a5f700 1 -- 172.21.15.67:0/3352168302 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe25801d070 con 0x7fe270106cd0 2024-04-22T23:48:23.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.100+0000 7fe275a61700 1 -- 172.21.15.67:0/3352168302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe270100b30 con 0x7fe270106cd0 2024-04-22T23:48:23.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.100+0000 7fe274a5f700 1 -- 172.21.15.67:0/3352168302 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe258004fb0 con 0x7fe270106cd0 2024-04-22T23:48:23.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.100+0000 7fe274a5f700 1 -- 172.21.15.67:0/3352168302 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe258022ea0 con 0x7fe270106cd0 2024-04-22T23:48:23.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.100+0000 7fe275a61700 1 -- 172.21.15.67:0/3352168302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2701a2d10 con 0x7fe270106cd0 2024-04-22T23:48:23.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.101+0000 7fe275a61700 1 -- 172.21.15.67:0/3352168302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2701091d0 con 0x7fe270106cd0 2024-04-22T23:48:23.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.101+0000 7fe274a5f700 1 -- 172.21.15.67:0/3352168302 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe258022470 con 0x7fe270106cd0 2024-04-22T23:48:23.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.102+0000 7fe274a5f700 1 --2- 172.21.15.67:0/3352168302 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe25c06ecc0 0x7fe25c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:23.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.102+0000 7fe274a5f700 1 -- 172.21.15.67:0/3352168302 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fe2580904c0 con 0x7fe270106cd0 2024-04-22T23:48:23.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.102+0000 7fe26effd700 1 --2- 172.21.15.67:0/3352168302 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe25c06ecc0 0x7fe25c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:23.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.102+0000 7fe26effd700 1 --2- 172.21.15.67:0/3352168302 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe25c06ecc0 0x7fe25c071180 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe268005ea0 tx=0x7fe268005e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:48:23.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.107+0000 7fe274a5f700 1 -- 172.21.15.67:0/3352168302 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe258059230 con 0x7fe270106cd0 2024-04-22T23:48:23.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.336+0000 7fe275a61700 1 -- 172.21.15.67:0/3352168302 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe2700611d0 con 0x7fe25c06ecc0 2024-04-22T23:48:23.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.337+0000 7fe274a5f700 1 -- 172.21.15.67:0/3352168302 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe2700611d0 con 0x7fe25c06ecc0 2024-04-22T23:48:23.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.339+0000 7fe275a61700 1 -- 172.21.15.67:0/3352168302 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe25c06ecc0 msgr2=0x7fe25c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:23.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.339+0000 7fe275a61700 1 --2- 172.21.15.67:0/3352168302 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe25c06ecc0 0x7fe25c071180 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe268005ea0 tx=0x7fe268005e30 comp rx=0 tx=0).stop 2024-04-22T23:48:23.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.339+0000 7fe275a61700 1 -- 172.21.15.67:0/3352168302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe270106cd0 msgr2=0x7fe270100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:23.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.339+0000 7fe275a61700 1 --2- 172.21.15.67:0/3352168302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe270106cd0 0x7fe270100230 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fe258006010 tx=0x7fe25800be20 comp rx=0 tx=0).stop 2024-04-22T23:48:23.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.339+0000 7fe275a61700 1 -- 172.21.15.67:0/3352168302 shutdown_connections 2024-04-22T23:48:23.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.339+0000 7fe275a61700 1 --2- 172.21.15.67:0/3352168302 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe25c06ecc0 0x7fe25c071180 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:23.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.339+0000 7fe275a61700 1 --2- 172.21.15.67:0/3352168302 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe270068580 0x7fe2700ffcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:23.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.339+0000 7fe275a61700 1 --2- 172.21.15.67:0/3352168302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe270106cd0 0x7fe270100230 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:23.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.339+0000 7fe275a61700 1 -- 172.21.15.67:0/3352168302 >> 172.21.15.67:0/3352168302 conn(0x7fe270075670 msgr2=0x7fe2700feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:48:23.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.339+0000 7fe275a61700 1 -- 172.21.15.67:0/3352168302 shutdown_connections 2024-04-22T23:48:23.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.340+0000 7fe275a61700 1 -- 172.21.15.67:0/3352168302 wait complete. 2024-04-22T23:48:23.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.446+0000 7ff5c1dfe700 1 -- 172.21.15.67:0/795556619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5bc100ea0 msgr2=0x7ff5bc101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:23.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.446+0000 7ff5c1dfe700 1 --2- 172.21.15.67:0/795556619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5bc100ea0 0x7ff5bc101280 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac009910 tx=0x7ff5ac009c20 comp rx=0 tx=0).stop 2024-04-22T23:48:23.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.447+0000 7ff5c1dfe700 1 -- 172.21.15.67:0/795556619 shutdown_connections 2024-04-22T23:48:23.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.447+0000 7ff5c1dfe700 1 --2- 172.21.15.67:0/795556619 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff5bc101850 0x7ff5bc1058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:23.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.447+0000 7ff5c1dfe700 1 --2- 172.21.15.67:0/795556619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5bc100ea0 0x7ff5bc101280 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:23.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.447+0000 7ff5c1dfe700 1 -- 172.21.15.67:0/795556619 >> 172.21.15.67:0/795556619 conn(0x7ff5bc0fc750 msgr2=0x7ff5bc0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:48:23.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.447+0000 7ff5c1dfe700 1 -- 172.21.15.67:0/795556619 shutdown_connections 2024-04-22T23:48:23.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.447+0000 7ff5c1dfe700 1 -- 172.21.15.67:0/795556619 wait complete. 2024-04-22T23:48:23.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.447+0000 7ff5c1dfe700 1 Processor -- start 2024-04-22T23:48:23.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.447+0000 7ff5c1dfe700 1 -- start start 2024-04-22T23:48:23.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.448+0000 7ff5c1dfe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5bc100ea0 0x7ff5bc19c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:23.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.448+0000 7ff5c1dfe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff5bc101850 0x7ff5bc19cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:23.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.448+0000 7ff5c1dfe700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5bc19d2c0 con 0x7ff5bc100ea0 2024-04-22T23:48:23.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.448+0000 7ff5c1dfe700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5bc196770 con 0x7ff5bc101850 2024-04-22T23:48:23.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.448+0000 7ff5bb7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5bc100ea0 0x7ff5bc19c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:23.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.448+0000 7ff5bb7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5bc100ea0 0x7ff5bc19c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54916/0 (socket says 172.21.15.67:54916) 2024-04-22T23:48:23.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.448+0000 7ff5bb7fe700 1 -- 172.21.15.67:0/675792743 learned_addr learned my addr 172.21.15.67:0/675792743 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:48:23.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.448+0000 7ff5baffd700 1 --2- 172.21.15.67:0/675792743 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff5bc101850 0x7ff5bc19cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:23.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.448+0000 7ff5bb7fe700 1 -- 172.21.15.67:0/675792743 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff5bc101850 msgr2=0x7ff5bc19cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:23.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.448+0000 7ff5bb7fe700 1 --2- 172.21.15.67:0/675792743 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff5bc101850 0x7ff5bc19cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:23.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.448+0000 7ff5bb7fe700 1 -- 172.21.15.67:0/675792743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5ac0095f0 con 0x7ff5bc100ea0 2024-04-22T23:48:23.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.448+0000 7ff5bb7fe700 1 --2- 172.21.15.67:0/675792743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5bc100ea0 0x7ff5bc19c6f0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac006010 tx=0x7ff5ac00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:48:23.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.449+0000 7ff5b8ff9700 1 -- 172.21.15.67:0/675792743 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5ac01d070 con 0x7ff5bc100ea0 2024-04-22T23:48:23.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.449+0000 7ff5b8ff9700 1 -- 172.21.15.67:0/675792743 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff5ac005dc0 con 0x7ff5bc100ea0 2024-04-22T23:48:23.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.449+0000 7ff5c1dfe700 1 -- 172.21.15.67:0/675792743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5bc1969f0 con 0x7ff5bc100ea0 2024-04-22T23:48:23.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.449+0000 7ff5b8ff9700 1 -- 172.21.15.67:0/675792743 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5ac017400 con 0x7ff5bc100ea0 2024-04-22T23:48:23.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.449+0000 7ff5c1dfe700 1 -- 172.21.15.67:0/675792743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5bc196ee0 con 0x7ff5bc100ea0 2024-04-22T23:48:23.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.450+0000 7ff5c1dfe700 1 -- 172.21.15.67:0/675792743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5bc109240 con 0x7ff5bc100ea0 2024-04-22T23:48:23.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.450+0000 7ff5b8ff9700 1 -- 172.21.15.67:0/675792743 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff5ac00f460 con 0x7ff5bc100ea0 2024-04-22T23:48:23.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.451+0000 7ff5b8ff9700 1 --2- 172.21.15.67:0/675792743 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff5a406ecc0 0x7ff5a4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:23.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.451+0000 7ff5b8ff9700 1 -- 172.21.15.67:0/675792743 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7ff5ac0903d0 con 0x7ff5bc100ea0 2024-04-22T23:48:23.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.451+0000 7ff5baffd700 1 --2- 172.21.15.67:0/675792743 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff5a406ecc0 0x7ff5a4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:23.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.451+0000 7ff5baffd700 1 --2- 172.21.15.67:0/675792743 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff5a406ecc0 0x7ff5a4071180 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc197be0 tx=0x7ff5b000b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:48:23.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.456+0000 7ff5b8ff9700 1 -- 172.21.15.67:0/675792743 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff5ac05cc00 con 0x7ff5bc100ea0 2024-04-22T23:48:23.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.678+0000 7ff5c1dfe700 1 -- 172.21.15.67:0/675792743 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff5bc197d40 con 0x7ff5a406ecc0 2024-04-22T23:48:23.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.689+0000 7ff5b8ff9700 1 -- 172.21.15.67:0/675792743 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7ff5bc197d40 con 0x7ff5a406ecc0 2024-04-22T23:48:23.681 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:48:23.681 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (22m) 15m ago 33m 25.5M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:48:23.681 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (33m) 15m ago 33m 9114k - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:48:23.681 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (32m) 15m ago 32m 9042k - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:48:23.681 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (19m) 15m ago 33m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:48:23.681 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (19m) 15m ago 32m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:48:23.681 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (21m) 15m ago 33m 79.3M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:48:23.681 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (15m) 15m ago 27m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:48:23.681 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (15m) 15m ago 27m 12.6M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:48:23.681 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (15m) 15m ago 27m 12.5M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:48:23.681 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (27m) 15m ago 27m 12.2M - 18.2.1 5be31c24972a 9e89d982d925 2024-04-22T23:48:23.681 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (23m) 15m ago 34m 511M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:48:23.682 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (23m) 15m ago 32m 453M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:48:23.682 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (20m) 15m ago 35m 55.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:48:23.682 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (20m) 15m ago 32m 46.0M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:48:23.682 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (22m) 15m ago 33m 15.4M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:48:23.682 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (22m) 15m ago 32m 16.1M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:48:23.682 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (19m) 15m ago 31m 99.2M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:48:23.682 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (18m) 15m ago 31m 127M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:48:23.682 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (17m) 15m ago 30m 79.9M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:48:23.682 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (17m) 15m ago 30m 97.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:48:23.682 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (16m) 15m ago 29m 82.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:48:23.682 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (16m) 15m ago 29m 77.6M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:48:23.682 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (22m) 15m ago 32m 73.9M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:48:23.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.691+0000 7ff5c1dfe700 1 -- 172.21.15.67:0/675792743 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff5a406ecc0 msgr2=0x7ff5a4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:23.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.691+0000 7ff5c1dfe700 1 --2- 172.21.15.67:0/675792743 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff5a406ecc0 0x7ff5a4071180 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc197be0 tx=0x7ff5b000b410 comp rx=0 tx=0).stop 2024-04-22T23:48:23.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.691+0000 7ff5c1dfe700 1 -- 172.21.15.67:0/675792743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5bc100ea0 msgr2=0x7ff5bc19c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:23.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.691+0000 7ff5c1dfe700 1 --2- 172.21.15.67:0/675792743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5bc100ea0 0x7ff5bc19c6f0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac006010 tx=0x7ff5ac00bd70 comp rx=0 tx=0).stop 2024-04-22T23:48:23.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.691+0000 7ff5c1dfe700 1 -- 172.21.15.67:0/675792743 shutdown_connections 2024-04-22T23:48:23.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.691+0000 7ff5c1dfe700 1 --2- 172.21.15.67:0/675792743 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff5a406ecc0 0x7ff5a4071180 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:23.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.691+0000 7ff5c1dfe700 1 --2- 172.21.15.67:0/675792743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5bc100ea0 0x7ff5bc19c6f0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:23.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.691+0000 7ff5c1dfe700 1 --2- 172.21.15.67:0/675792743 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff5bc101850 0x7ff5bc19cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:23.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.691+0000 7ff5c1dfe700 1 -- 172.21.15.67:0/675792743 >> 172.21.15.67:0/675792743 conn(0x7ff5bc0fc750 msgr2=0x7ff5bc0fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:48:23.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.691+0000 7ff5c1dfe700 1 -- 172.21.15.67:0/675792743 shutdown_connections 2024-04-22T23:48:23.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.691+0000 7ff5c1dfe700 1 -- 172.21.15.67:0/675792743 wait complete. 2024-04-22T23:48:23.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.797+0000 7f7ee5b7a700 1 -- 172.21.15.67:0/955018292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ee0105bd0 msgr2=0x7f7ee0105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:23.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.797+0000 7f7ee5b7a700 1 --2- 172.21.15.67:0/955018292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ee0105bd0 0x7f7ee0105fb0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f7ed0009910 tx=0x7f7ed0009c20 comp rx=0 tx=0).stop 2024-04-22T23:48:23.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.797+0000 7f7ee5b7a700 1 -- 172.21.15.67:0/955018292 shutdown_connections 2024-04-22T23:48:23.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.797+0000 7f7ee5b7a700 1 --2- 172.21.15.67:0/955018292 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7ee00ffc20 0x7f7ee0100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:23.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.797+0000 7f7ee5b7a700 1 --2- 172.21.15.67:0/955018292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ee0105bd0 0x7f7ee0105fb0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:23.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.797+0000 7f7ee5b7a700 1 -- 172.21.15.67:0/955018292 >> 172.21.15.67:0/955018292 conn(0x7f7ee0075670 msgr2=0x7f7ee0075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:48:23.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.797+0000 7f7ee5b7a700 1 -- 172.21.15.67:0/955018292 shutdown_connections 2024-04-22T23:48:23.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.797+0000 7f7ee5b7a700 1 -- 172.21.15.67:0/955018292 wait complete. 2024-04-22T23:48:23.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.798+0000 7f7ee5b7a700 1 Processor -- start 2024-04-22T23:48:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.798+0000 7f7ee5b7a700 1 -- start start 2024-04-22T23:48:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.798+0000 7f7ee5b7a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7ee00ffc20 0x7f7ee019c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.798+0000 7f7ee5b7a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ee0105bd0 0x7f7ee019cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.798+0000 7f7ee5b7a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee019d1f0 con 0x7f7ee0105bd0 2024-04-22T23:48:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.798+0000 7f7ee5b7a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee01966a0 con 0x7f7ee00ffc20 2024-04-22T23:48:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.798+0000 7f7edeffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ee0105bd0 0x7f7ee019cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.798+0000 7f7edeffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ee0105bd0 0x7f7ee019cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54932/0 (socket says 172.21.15.67:54932) 2024-04-22T23:48:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.798+0000 7f7edeffd700 1 -- 172.21.15.67:0/3742390442 learned_addr learned my addr 172.21.15.67:0/3742390442 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:48:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.798+0000 7f7edf7fe700 1 --2- 172.21.15.67:0/3742390442 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7ee00ffc20 0x7f7ee019c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.799+0000 7f7edeffd700 1 -- 172.21.15.67:0/3742390442 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7ee00ffc20 msgr2=0x7f7ee019c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.799+0000 7f7edeffd700 1 --2- 172.21.15.67:0/3742390442 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7ee00ffc20 0x7f7ee019c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.799+0000 7f7edeffd700 1 -- 172.21.15.67:0/3742390442 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed00095f0 con 0x7f7ee0105bd0 2024-04-22T23:48:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.799+0000 7f7edeffd700 1 --2- 172.21.15.67:0/3742390442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ee0105bd0 0x7f7ee019cb60 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f7ed400e920 tx=0x7f7ed400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:48:23.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.799+0000 7f7edcff9700 1 -- 172.21.15.67:0/3742390442 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ed400c9d0 con 0x7f7ee0105bd0 2024-04-22T23:48:23.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.799+0000 7f7edcff9700 1 -- 172.21.15.67:0/3742390442 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7ed4010430 con 0x7f7ee0105bd0 2024-04-22T23:48:23.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.799+0000 7f7ee5b7a700 1 -- 172.21.15.67:0/3742390442 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ee0196980 con 0x7f7ee0105bd0 2024-04-22T23:48:23.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.799+0000 7f7edcff9700 1 -- 172.21.15.67:0/3742390442 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ed4018630 con 0x7f7ee0105bd0 2024-04-22T23:48:23.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.799+0000 7f7ee5b7a700 1 -- 172.21.15.67:0/3742390442 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ee0196ed0 con 0x7f7ee0105bd0 2024-04-22T23:48:23.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.800+0000 7f7ee5b7a700 1 -- 172.21.15.67:0/3742390442 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ecc005320 con 0x7f7ee0105bd0 2024-04-22T23:48:23.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.801+0000 7f7edcff9700 1 -- 172.21.15.67:0/3742390442 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7ed400cb30 con 0x7f7ee0105bd0 2024-04-22T23:48:23.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.801+0000 7f7edcff9700 1 --2- 172.21.15.67:0/3742390442 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7ec806ecc0 0x7f7ec8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:23.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.801+0000 7f7edf7fe700 1 --2- 172.21.15.67:0/3742390442 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7ec806ecc0 0x7f7ec8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:23.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.801+0000 7f7edcff9700 1 -- 172.21.15.67:0/3742390442 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f7ed4014070 con 0x7f7ee0105bd0 2024-04-22T23:48:23.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.802+0000 7f7edf7fe700 1 --2- 172.21.15.67:0/3742390442 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7ec806ecc0 0x7f7ec8071180 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7ed00059d0 tx=0x7f7ed0005940 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:48:23.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:23.806+0000 7f7edcff9700 1 -- 172.21.15.67:0/3742390442 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7ed4054150 con 0x7f7ee0105bd0 2024-04-22T23:48:24.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.107+0000 7f7ee5b7a700 1 -- 172.21.15.67:0/3742390442 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7ecc006200 con 0x7f7ee0105bd0 2024-04-22T23:48:24.099 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:23 smithi067 ceph-mon[112438]: pgmap v662: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:24.099 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:23 smithi067 ceph-mon[112438]: from='client.35212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:48:24.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.108+0000 7f7edcff9700 1 -- 172.21.15.67:0/3742390442 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f7ed405b670 con 0x7f7ee0105bd0 2024-04-22T23:48:24.100 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:48:24.100 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:48:24.100 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:48:24.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:48:24.100 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:48:24.100 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:48:24.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:48:24.101 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:48:24.101 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:48:24.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:48:24.101 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:48:24.101 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 3 2024-04-22T23:48:24.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:48:24.101 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:48:24.101 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 13 2024-04-22T23:48:24.101 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:48:24.101 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:48:24.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.110+0000 7f7ee5b7a700 1 -- 172.21.15.67:0/3742390442 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7ec806ecc0 msgr2=0x7f7ec8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:24.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.110+0000 7f7ee5b7a700 1 --2- 172.21.15.67:0/3742390442 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7ec806ecc0 0x7f7ec8071180 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7ed00059d0 tx=0x7f7ed0005940 comp rx=0 tx=0).stop 2024-04-22T23:48:24.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.110+0000 7f7ee5b7a700 1 -- 172.21.15.67:0/3742390442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ee0105bd0 msgr2=0x7f7ee019cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:24.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.110+0000 7f7ee5b7a700 1 --2- 172.21.15.67:0/3742390442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ee0105bd0 0x7f7ee019cb60 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f7ed400e920 tx=0x7f7ed400ec30 comp rx=0 tx=0).stop 2024-04-22T23:48:24.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.110+0000 7f7ee5b7a700 1 -- 172.21.15.67:0/3742390442 shutdown_connections 2024-04-22T23:48:24.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.110+0000 7f7ee5b7a700 1 --2- 172.21.15.67:0/3742390442 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7ec806ecc0 0x7f7ec8071180 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:24.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.110+0000 7f7ee5b7a700 1 --2- 172.21.15.67:0/3742390442 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7ee00ffc20 0x7f7ee019c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:24.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.110+0000 7f7ee5b7a700 1 --2- 172.21.15.67:0/3742390442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ee0105bd0 0x7f7ee019cb60 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:24.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.110+0000 7f7ee5b7a700 1 -- 172.21.15.67:0/3742390442 >> 172.21.15.67:0/3742390442 conn(0x7f7ee0075670 msgr2=0x7f7ee00686c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:48:24.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.111+0000 7f7ee5b7a700 1 -- 172.21.15.67:0/3742390442 shutdown_connections 2024-04-22T23:48:24.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.111+0000 7f7ee5b7a700 1 -- 172.21.15.67:0/3742390442 wait complete. 2024-04-22T23:48:24.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.217+0000 7fa039c3a700 1 -- 172.21.15.67:0/3931438409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa034101e00 msgr2=0x7fa0341021e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:24.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.217+0000 7fa039c3a700 1 --2- 172.21.15.67:0/3931438409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa034101e00 0x7fa0341021e0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fa024009910 tx=0x7fa024009c20 comp rx=0 tx=0).stop 2024-04-22T23:48:24.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.218+0000 7fa039c3a700 1 -- 172.21.15.67:0/3931438409 shutdown_connections 2024-04-22T23:48:24.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.218+0000 7fa039c3a700 1 --2- 172.21.15.67:0/3931438409 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa034102720 0x7fa03410ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:24.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.218+0000 7fa039c3a700 1 --2- 172.21.15.67:0/3931438409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa034101e00 0x7fa0341021e0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:24.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.218+0000 7fa039c3a700 1 -- 172.21.15.67:0/3931438409 >> 172.21.15.67:0/3931438409 conn(0x7fa0340fb650 msgr2=0x7fa0340fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:48:24.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.218+0000 7fa039c3a700 1 -- 172.21.15.67:0/3931438409 shutdown_connections 2024-04-22T23:48:24.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.218+0000 7fa039c3a700 1 -- 172.21.15.67:0/3931438409 wait complete. 2024-04-22T23:48:24.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.218+0000 7fa039c3a700 1 Processor -- start 2024-04-22T23:48:24.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.218+0000 7fa039c3a700 1 -- start start 2024-04-22T23:48:24.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.219+0000 7fa039c3a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa034101e00 0x7fa0340ffa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:24.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.219+0000 7fa039c3a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa034102720 0x7fa0340fffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:24.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.219+0000 7fa039c3a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0341004e0 con 0x7fa034101e00 2024-04-22T23:48:24.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.219+0000 7fa039c3a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa034100620 con 0x7fa034102720 2024-04-22T23:48:24.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.219+0000 7fa0337fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa034101e00 0x7fa0340ffa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:24.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.219+0000 7fa0337fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa034101e00 0x7fa0340ffa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54942/0 (socket says 172.21.15.67:54942) 2024-04-22T23:48:24.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.219+0000 7fa0337fe700 1 -- 172.21.15.67:0/3650620268 learned_addr learned my addr 172.21.15.67:0/3650620268 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:48:24.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.219+0000 7fa032ffd700 1 --2- 172.21.15.67:0/3650620268 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa034102720 0x7fa0340fffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:24.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.219+0000 7fa0337fe700 1 -- 172.21.15.67:0/3650620268 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa034102720 msgr2=0x7fa0340fffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:24.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.219+0000 7fa0337fe700 1 --2- 172.21.15.67:0/3650620268 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa034102720 0x7fa0340fffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:24.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.219+0000 7fa0337fe700 1 -- 172.21.15.67:0/3650620268 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0240095f0 con 0x7fa034101e00 2024-04-22T23:48:24.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.219+0000 7fa0337fe700 1 --2- 172.21.15.67:0/3650620268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa034101e00 0x7fa0340ffa60 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fa024006010 tx=0x7fa02400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:48:24.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.219+0000 7fa030ff9700 1 -- 172.21.15.67:0/3650620268 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa02401d070 con 0x7fa034101e00 2024-04-22T23:48:24.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.219+0000 7fa030ff9700 1 -- 172.21.15.67:0/3650620268 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa024005dc0 con 0x7fa034101e00 2024-04-22T23:48:24.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.219+0000 7fa039c3a700 1 -- 172.21.15.67:0/3650620268 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0341008a0 con 0x7fa034101e00 2024-04-22T23:48:24.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.220+0000 7fa030ff9700 1 -- 172.21.15.67:0/3650620268 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa024017400 con 0x7fa034101e00 2024-04-22T23:48:24.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.220+0000 7fa039c3a700 1 -- 172.21.15.67:0/3650620268 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0341a2dc0 con 0x7fa034101e00 2024-04-22T23:48:24.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.220+0000 7fa039c3a700 1 -- 172.21.15.67:0/3650620268 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa034108310 con 0x7fa034101e00 2024-04-22T23:48:24.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.221+0000 7fa030ff9700 1 -- 172.21.15.67:0/3650620268 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa024003a60 con 0x7fa034101e00 2024-04-22T23:48:24.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.221+0000 7fa030ff9700 1 --2- 172.21.15.67:0/3650620268 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa01c06ecc0 0x7fa01c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:24.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.221+0000 7fa030ff9700 1 -- 172.21.15.67:0/3650620268 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fa024090370 con 0x7fa034101e00 2024-04-22T23:48:24.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.221+0000 7fa032ffd700 1 --2- 172.21.15.67:0/3650620268 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa01c06ecc0 0x7fa01c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:24.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.222+0000 7fa032ffd700 1 --2- 172.21.15.67:0/3650620268 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa01c06ecc0 0x7fa01c071180 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa034101250 tx=0x7fa02800b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:48:24.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.226+0000 7fa030ff9700 1 -- 172.21.15.67:0/3650620268 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fa024059030 con 0x7fa034101e00 2024-04-22T23:48:24.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:23 smithi134 ceph-mon[85630]: pgmap v662: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:24.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:23 smithi134 ceph-mon[85630]: from='client.35212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:48:24.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.483+0000 7fa039c3a700 1 -- 172.21.15.67:0/3650620268 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa03404f2e0 con 0x7fa034101e00 2024-04-22T23:48:24.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.483+0000 7fa030ff9700 1 -- 172.21.15.67:0/3650620268 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1693 (secure 0 0 0) 0x7fa02405c6f0 con 0x7fa034101e00 2024-04-22T23:48:24.476 INFO:teuthology.orchestra.run.smithi067.stdout:e32 2024-04-22T23:48:24.476 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:48:24.476 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:48:24.476 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:48:24.476 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:48:24.476 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:48:24.476 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:48:24.476 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:48:24.476 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:48:24.476 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:48:24.476 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:48:24.476 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:48:24.477 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:48:24.477 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:48:24.477 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:48:24.477 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:48:24.477 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:48:24.477 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:48:24.477 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:48:24.477 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:48:24.477 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:48:24.477 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:48:24.477 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:48:24.477 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:48:24.477 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:48:24.477 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:48:24.478 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:48:24.478 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:48:24.478 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:48:24.478 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:48:24.478 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:48:24.478 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:48:24.478 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:48:24.478 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:48:24.478 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:48:24.478 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:48:24.479 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:48:24.479 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:48:24.479 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:48:24.479 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:48:24.479 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:48:24.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.486+0000 7fa039c3a700 1 -- 172.21.15.67:0/3650620268 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa01c06ecc0 msgr2=0x7fa01c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:24.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.486+0000 7fa039c3a700 1 --2- 172.21.15.67:0/3650620268 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa01c06ecc0 0x7fa01c071180 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa034101250 tx=0x7fa02800b410 comp rx=0 tx=0).stop 2024-04-22T23:48:24.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.486+0000 7fa039c3a700 1 -- 172.21.15.67:0/3650620268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa034101e00 msgr2=0x7fa0340ffa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:24.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.486+0000 7fa039c3a700 1 --2- 172.21.15.67:0/3650620268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa034101e00 0x7fa0340ffa60 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fa024006010 tx=0x7fa02400bd70 comp rx=0 tx=0).stop 2024-04-22T23:48:24.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.487+0000 7fa039c3a700 1 -- 172.21.15.67:0/3650620268 shutdown_connections 2024-04-22T23:48:24.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.487+0000 7fa039c3a700 1 --2- 172.21.15.67:0/3650620268 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa01c06ecc0 0x7fa01c071180 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:24.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.487+0000 7fa039c3a700 1 --2- 172.21.15.67:0/3650620268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa034101e00 0x7fa0340ffa60 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:24.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.487+0000 7fa039c3a700 1 --2- 172.21.15.67:0/3650620268 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa034102720 0x7fa0340fffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:24.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.487+0000 7fa039c3a700 1 -- 172.21.15.67:0/3650620268 >> 172.21.15.67:0/3650620268 conn(0x7fa0340fb650 msgr2=0x7fa034105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:48:24.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.487+0000 7fa039c3a700 1 -- 172.21.15.67:0/3650620268 shutdown_connections 2024-04-22T23:48:24.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.487+0000 7fa039c3a700 1 -- 172.21.15.67:0/3650620268 wait complete. 2024-04-22T23:48:24.481 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 32 2024-04-22T23:48:24.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.596+0000 7f7e87708700 1 -- 172.21.15.67:0/2115082619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e80101e00 msgr2=0x7f7e801021e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:24.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.596+0000 7f7e87708700 1 --2- 172.21.15.67:0/2115082619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e80101e00 0x7f7e801021e0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c009910 tx=0x7f7e7c009c20 comp rx=0 tx=0).stop 2024-04-22T23:48:24.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.596+0000 7f7e87708700 1 -- 172.21.15.67:0/2115082619 shutdown_connections 2024-04-22T23:48:24.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.596+0000 7f7e87708700 1 --2- 172.21.15.67:0/2115082619 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7e80102720 0x7f7e8010ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:24.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.596+0000 7f7e87708700 1 --2- 172.21.15.67:0/2115082619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e80101e00 0x7f7e801021e0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:24.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.596+0000 7f7e87708700 1 -- 172.21.15.67:0/2115082619 >> 172.21.15.67:0/2115082619 conn(0x7f7e800fb650 msgr2=0x7f7e800fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:48:24.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.596+0000 7f7e87708700 1 -- 172.21.15.67:0/2115082619 shutdown_connections 2024-04-22T23:48:24.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.596+0000 7f7e87708700 1 -- 172.21.15.67:0/2115082619 wait complete. 2024-04-22T23:48:24.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.596+0000 7f7e87708700 1 Processor -- start 2024-04-22T23:48:24.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.597+0000 7f7e87708700 1 -- start start 2024-04-22T23:48:24.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.597+0000 7f7e87708700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7e80101e00 0x7f7e800ffa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:24.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.597+0000 7f7e87708700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e80102720 0x7f7e800fffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:24.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.597+0000 7f7e87708700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e801004e0 con 0x7f7e80102720 2024-04-22T23:48:24.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.597+0000 7f7e87708700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e80100620 con 0x7f7e80101e00 2024-04-22T23:48:24.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.597+0000 7f7e84ca3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e80102720 0x7f7e800fffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:24.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.597+0000 7f7e84ca3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e80102720 0x7f7e800fffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54948/0 (socket says 172.21.15.67:54948) 2024-04-22T23:48:24.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.597+0000 7f7e84ca3700 1 -- 172.21.15.67:0/3193203321 learned_addr learned my addr 172.21.15.67:0/3193203321 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:48:24.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.597+0000 7f7e854a4700 1 --2- 172.21.15.67:0/3193203321 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7e80101e00 0x7f7e800ffa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:24.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.597+0000 7f7e84ca3700 1 -- 172.21.15.67:0/3193203321 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7e80101e00 msgr2=0x7f7e800ffa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:24.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.597+0000 7f7e84ca3700 1 --2- 172.21.15.67:0/3193203321 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7e80101e00 0x7f7e800ffa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:24.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.597+0000 7f7e84ca3700 1 -- 172.21.15.67:0/3193203321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7c0095f0 con 0x7f7e80102720 2024-04-22T23:48:24.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.597+0000 7f7e84ca3700 1 --2- 172.21.15.67:0/3193203321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e80102720 0x7f7e800fffa0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f7e7000e8c0 tx=0x7f7e7000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:48:24.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.597+0000 7f7e767fc700 1 -- 172.21.15.67:0/3193203321 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e700041f0 con 0x7f7e80102720 2024-04-22T23:48:24.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.598+0000 7f7e87708700 1 -- 172.21.15.67:0/3193203321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e801a2a90 con 0x7f7e80102720 2024-04-22T23:48:24.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.598+0000 7f7e767fc700 1 -- 172.21.15.67:0/3193203321 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7e7000cca0 con 0x7f7e80102720 2024-04-22T23:48:24.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.598+0000 7f7e767fc700 1 -- 172.21.15.67:0/3193203321 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e70003de0 con 0x7f7e80102720 2024-04-22T23:48:24.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.598+0000 7f7e87708700 1 -- 172.21.15.67:0/3193203321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e801a2ea0 con 0x7f7e80102720 2024-04-22T23:48:24.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.599+0000 7f7e87708700 1 -- 172.21.15.67:0/3193203321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e80108310 con 0x7f7e80102720 2024-04-22T23:48:24.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.599+0000 7f7e767fc700 1 -- 172.21.15.67:0/3193203321 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7e7000c7e0 con 0x7f7e80102720 2024-04-22T23:48:24.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.599+0000 7f7e767fc700 1 --2- 172.21.15.67:0/3193203321 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7e6c06ecc0 0x7f7e6c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:24.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.600+0000 7f7e767fc700 1 -- 172.21.15.67:0/3193203321 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f7e70014070 con 0x7f7e80102720 2024-04-22T23:48:24.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.600+0000 7f7e854a4700 1 --2- 172.21.15.67:0/3193203321 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7e6c06ecc0 0x7f7e6c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:24.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.600+0000 7f7e854a4700 1 --2- 172.21.15.67:0/3193203321 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7e6c06ecc0 0x7f7e6c071180 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c0059d0 tx=0x7f7e7c005940 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:48:24.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.604+0000 7f7e767fc700 1 -- 172.21.15.67:0/3193203321 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7e70057fd0 con 0x7f7e80102720 2024-04-22T23:48:24.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.845+0000 7f7e87708700 1 -- 172.21.15.67:0/3193203321 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e800611d0 con 0x7f7e6c06ecc0 2024-04-22T23:48:24.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.846+0000 7f7e767fc700 1 -- 172.21.15.67:0/3193203321 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f7e800611d0 con 0x7f7e6c06ecc0 2024-04-22T23:48:24.838 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:48:24.838 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:48:24.838 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:48:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:48:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:48:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:48:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:48:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:48:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:48:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:48:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "15/23 daemons upgraded", 2024-04-22T23:48:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:48:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:48:24.839 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:48:24.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.848+0000 7f7e87708700 1 -- 172.21.15.67:0/3193203321 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7e6c06ecc0 msgr2=0x7f7e6c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:24.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.848+0000 7f7e87708700 1 --2- 172.21.15.67:0/3193203321 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7e6c06ecc0 0x7f7e6c071180 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c0059d0 tx=0x7f7e7c005940 comp rx=0 tx=0).stop 2024-04-22T23:48:24.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.848+0000 7f7e87708700 1 -- 172.21.15.67:0/3193203321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e80102720 msgr2=0x7f7e800fffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:24.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.848+0000 7f7e87708700 1 --2- 172.21.15.67:0/3193203321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e80102720 0x7f7e800fffa0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f7e7000e8c0 tx=0x7f7e7000ebd0 comp rx=0 tx=0).stop 2024-04-22T23:48:24.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.848+0000 7f7e87708700 1 -- 172.21.15.67:0/3193203321 shutdown_connections 2024-04-22T23:48:24.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.848+0000 7f7e87708700 1 --2- 172.21.15.67:0/3193203321 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7e6c06ecc0 0x7f7e6c071180 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:24.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.848+0000 7f7e87708700 1 --2- 172.21.15.67:0/3193203321 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7e80101e00 0x7f7e800ffa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:24.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.848+0000 7f7e87708700 1 --2- 172.21.15.67:0/3193203321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e80102720 0x7f7e800fffa0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:24.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.848+0000 7f7e87708700 1 -- 172.21.15.67:0/3193203321 >> 172.21.15.67:0/3193203321 conn(0x7f7e800fb650 msgr2=0x7f7e80105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:48:24.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.849+0000 7f7e87708700 1 -- 172.21.15.67:0/3193203321 shutdown_connections 2024-04-22T23:48:24.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.849+0000 7f7e87708700 1 -- 172.21.15.67:0/3193203321 wait complete. 2024-04-22T23:48:24.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.957+0000 7f965f330700 1 -- 172.21.15.67:0/3689100334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96580ffda0 msgr2=0x7f9658100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:24.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.957+0000 7f965f330700 1 --2- 172.21.15.67:0/3689100334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96580ffda0 0x7f9658100180 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f9654009910 tx=0x7f9654009c20 comp rx=0 tx=0).stop 2024-04-22T23:48:24.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.958+0000 7f965f330700 1 -- 172.21.15.67:0/3689100334 shutdown_connections 2024-04-22T23:48:24.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.958+0000 7f965f330700 1 --2- 172.21.15.67:0/3689100334 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9658100750 0x7f96581047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:24.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.958+0000 7f965f330700 1 --2- 172.21.15.67:0/3689100334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96580ffda0 0x7f9658100180 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:24.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.958+0000 7f965f330700 1 -- 172.21.15.67:0/3689100334 >> 172.21.15.67:0/3689100334 conn(0x7f96580fb650 msgr2=0x7f96580fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:48:24.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.958+0000 7f965f330700 1 -- 172.21.15.67:0/3689100334 shutdown_connections 2024-04-22T23:48:24.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.958+0000 7f965f330700 1 -- 172.21.15.67:0/3689100334 wait complete. 2024-04-22T23:48:24.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.958+0000 7f965f330700 1 Processor -- start 2024-04-22T23:48:24.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.958+0000 7f965f330700 1 -- start start 2024-04-22T23:48:24.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.959+0000 7f965f330700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f96580ffda0 0x7f9658101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:24.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.959+0000 7f965f330700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9658100750 0x7f9658101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:24.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.959+0000 7f965f330700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96581022c0 con 0x7f9658100750 2024-04-22T23:48:24.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.959+0000 7f965f330700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9658102400 con 0x7f96580ffda0 2024-04-22T23:48:24.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.959+0000 7f965c8cb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9658100750 0x7f9658101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:24.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.959+0000 7f965c8cb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9658100750 0x7f9658101d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54966/0 (socket says 172.21.15.67:54966) 2024-04-22T23:48:24.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.959+0000 7f965c8cb700 1 -- 172.21.15.67:0/4183819480 learned_addr learned my addr 172.21.15.67:0/4183819480 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:48:24.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.959+0000 7f965d0cc700 1 --2- 172.21.15.67:0/4183819480 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f96580ffda0 0x7f9658101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:24.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.959+0000 7f965c8cb700 1 -- 172.21.15.67:0/4183819480 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f96580ffda0 msgr2=0x7f9658101840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:24.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.959+0000 7f965c8cb700 1 --2- 172.21.15.67:0/4183819480 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f96580ffda0 0x7f9658101840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:24.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.959+0000 7f965c8cb700 1 -- 172.21.15.67:0/4183819480 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96540095f0 con 0x7f9658100750 2024-04-22T23:48:24.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.959+0000 7f965c8cb700 1 --2- 172.21.15.67:0/4183819480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9658100750 0x7f9658101d80 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f964800d900 tx=0x7f964800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:48:24.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.960+0000 7f964e7fc700 1 -- 172.21.15.67:0/4183819480 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9648009900 con 0x7f9658100750 2024-04-22T23:48:24.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.960+0000 7f965f330700 1 -- 172.21.15.67:0/4183819480 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96581a2ae0 con 0x7f9658100750 2024-04-22T23:48:24.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.960+0000 7f964e7fc700 1 -- 172.21.15.67:0/4183819480 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9648010460 con 0x7f9658100750 2024-04-22T23:48:24.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.960+0000 7f964e7fc700 1 -- 172.21.15.67:0/4183819480 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f964800f5f0 con 0x7f9658100750 2024-04-22T23:48:24.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.960+0000 7f965f330700 1 -- 172.21.15.67:0/4183819480 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96581a2ef0 con 0x7f9658100750 2024-04-22T23:48:24.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.961+0000 7f965f330700 1 -- 172.21.15.67:0/4183819480 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f965804f2e0 con 0x7f9658100750 2024-04-22T23:48:24.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.961+0000 7f964e7fc700 1 -- 172.21.15.67:0/4183819480 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f96480105d0 con 0x7f9658100750 2024-04-22T23:48:24.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.962+0000 7f964e7fc700 1 --2- 172.21.15.67:0/4183819480 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f964406ecc0 0x7f9644071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:24.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.962+0000 7f964e7fc700 1 -- 172.21.15.67:0/4183819480 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f964808eb60 con 0x7f9658100750 2024-04-22T23:48:24.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.962+0000 7f965d0cc700 1 --2- 172.21.15.67:0/4183819480 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f964406ecc0 0x7f9644071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:24.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.962+0000 7f965d0cc700 1 --2- 172.21.15.67:0/4183819480 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f964406ecc0 0x7f9644071180 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f9654006010 tx=0x7f965401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:48:24.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:24.967+0000 7f964e7fc700 1 -- 172.21.15.67:0/4183819480 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f96480577a0 con 0x7f9658100750 2024-04-22T23:48:25.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:24 smithi134 ceph-mon[85630]: from='client.35216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:48:25.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:24 smithi134 ceph-mon[85630]: from='client.35220 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:48:25.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:24 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3742390442' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:48:25.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:24 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3650620268' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:48:25.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:24 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:48:25.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:24 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:48:25.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:24 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:48:25.265 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:24 smithi067 ceph-mon[112438]: from='client.35216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:48:25.265 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:24 smithi067 ceph-mon[112438]: from='client.35220 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:48:25.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:24 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3742390442' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:48:25.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:24 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3650620268' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:48:25.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:24 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:48:25.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:24 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:48:25.266 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:24 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T23:48:25.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:25.273+0000 7f965f330700 1 -- 172.21.15.67:0/4183819480 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f9658102d70 con 0x7f9658100750 2024-04-22T23:48:25.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:25.274+0000 7f964e7fc700 1 -- 172.21.15.67:0/4183819480 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4505 (secure 0 0 0) 0x7f964805ae60 con 0x7f9658100750 2024-04-22T23:48:25.266 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:48:25.267 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T23:48:25.267 INFO:teuthology.orchestra.run.smithi067.stdout: host smithi134 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-22T23:48:25.267 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T23:48:25.267 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:48:25.267 INFO:teuthology.orchestra.run.smithi067.stdout:Traceback (most recent call last): 2024-04-22T23:48:25.267 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T23:48:25.267 INFO:teuthology.orchestra.run.smithi067.stdout: return _run_code(code, main_globals, None, 2024-04-22T23:48:25.267 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T23:48:25.267 INFO:teuthology.orchestra.run.smithi067.stdout: exec(code, run_globals) 2024-04-22T23:48:25.267 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5642, in 2024-04-22T23:48:25.267 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5630, in main 2024-04-22T23:48:25.267 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 408, in _infer_config 2024-04-22T23:48:25.267 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 323, in _infer_fsid 2024-04-22T23:48:25.267 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 436, in _infer_image 2024-04-22T23:48:25.268 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 310, in _validate_fsid 2024-04-22T23:48:25.268 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 3380, in command_ceph_volume 2024-04-22T23:48:25.268 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T23:48:25.268 INFO:teuthology.orchestra.run.smithi067.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:48:25.268 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:48:25.268 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:48:25.268 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:48:25.268 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:48:25.268 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:48:25.268 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:48:25.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:25.276+0000 7f965f330700 1 -- 172.21.15.67:0/4183819480 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f964406ecc0 msgr2=0x7f9644071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:25.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:25.276+0000 7f965f330700 1 --2- 172.21.15.67:0/4183819480 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f964406ecc0 0x7f9644071180 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f9654006010 tx=0x7f965401a040 comp rx=0 tx=0).stop 2024-04-22T23:48:25.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:25.276+0000 7f965f330700 1 -- 172.21.15.67:0/4183819480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9658100750 msgr2=0x7f9658101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:25.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:25.276+0000 7f965f330700 1 --2- 172.21.15.67:0/4183819480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9658100750 0x7f9658101d80 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f964800d900 tx=0x7f964800dc10 comp rx=0 tx=0).stop 2024-04-22T23:48:25.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:25.276+0000 7f965f330700 1 -- 172.21.15.67:0/4183819480 shutdown_connections 2024-04-22T23:48:25.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:25.276+0000 7f965f330700 1 --2- 172.21.15.67:0/4183819480 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f964406ecc0 0x7f9644071180 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:25.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:25.276+0000 7f965f330700 1 --2- 172.21.15.67:0/4183819480 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f96580ffda0 0x7f9658101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:25.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:25.276+0000 7f965f330700 1 --2- 172.21.15.67:0/4183819480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9658100750 0x7f9658101d80 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:25.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:25.276+0000 7f965f330700 1 -- 172.21.15.67:0/4183819480 >> 172.21.15.67:0/4183819480 conn(0x7f96580fb650 msgr2=0x7f9658103f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:48:25.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:25.276+0000 7f965f330700 1 -- 172.21.15.67:0/4183819480 shutdown_connections 2024-04-22T23:48:25.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:25.277+0000 7f965f330700 1 -- 172.21.15.67:0/4183819480 wait complete. 2024-04-22T23:48:26.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:25 smithi134 ceph-mon[85630]: from='client.35236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:48:26.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:25 smithi134 ceph-mon[85630]: pgmap v663: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:26.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:25 smithi134 ceph-mon[85630]: mds.? [v2:172.21.15.134:6826/1849648622,v1:172.21.15.134:6827/1849648622] up:boot 2024-04-22T23:48:26.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:25 smithi134 ceph-mon[85630]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-22T23:48:26.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:25 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.vuxbsj"}]: dispatch 2024-04-22T23:48:26.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:25 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:48:26.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:25 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:48:26.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:25 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:48:26.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:25 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:48:26.236 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:25 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/4183819480' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:48:26.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:25 smithi067 ceph-mon[112438]: from='client.35236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:48:26.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:25 smithi067 ceph-mon[112438]: pgmap v663: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:26.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:25 smithi067 ceph-mon[112438]: mds.? [v2:172.21.15.134:6826/1849648622,v1:172.21.15.134:6827/1849648622] up:boot 2024-04-22T23:48:26.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:25 smithi067 ceph-mon[112438]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-22T23:48:26.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:25 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi134.vuxbsj"}]: dispatch 2024-04-22T23:48:26.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:25 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:48:26.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:25 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:48:26.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:25 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:48:26.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:25 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:48:26.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:25 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/4183819480' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:48:27.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:27 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:48:27.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:27 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:48:27.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:27 smithi067 ceph-mon[112438]: pgmap v664: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:27.296 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:27 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:48:27.297 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:27 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:48:27.297 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:27 smithi134 ceph-mon[85630]: pgmap v664: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:29.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:29 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:48:29.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:29 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:48:29.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:29 smithi134 ceph-mon[85630]: pgmap v665: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:29.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:29 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:48:29.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:29 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:48:29.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:29 smithi067 ceph-mon[112438]: pgmap v665: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:30.413 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:30 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:48:30.413 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:30 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:48:30.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:30 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:48:30.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:30 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:48:31.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:31 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:48:31.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:31 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:48:31.786 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:31 smithi067 ceph-mon[112438]: pgmap v666: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:31.951 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:31 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:48:31.952 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:31 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-22T23:48:31.952 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:31 smithi134 ceph-mon[85630]: pgmap v666: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:34.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:33 smithi134 ceph-mon[85630]: pgmap v667: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:34.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:33 smithi067 ceph-mon[112438]: pgmap v667: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:36.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:35 smithi134 ceph-mon[85630]: pgmap v668: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:36.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:35 smithi067 ceph-mon[112438]: pgmap v668: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:38.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:37 smithi134 ceph-mon[85630]: pgmap v669: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:38.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:37 smithi067 ceph-mon[112438]: pgmap v669: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:40.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:39 smithi134 ceph-mon[85630]: pgmap v670: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:40.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:39 smithi067 ceph-mon[112438]: pgmap v670: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:42.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:41 smithi067 ceph-mon[112438]: pgmap v671: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:42.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:48:42.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:41 smithi134 ceph-mon[85630]: pgmap v671: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:42.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:48:44.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:43 smithi067 ceph-mon[112438]: pgmap v672: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:44.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:43 smithi134 ceph-mon[85630]: pgmap v672: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:46.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:46 smithi067 ceph-mon[112438]: pgmap v673: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:46.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:46 smithi134 ceph-mon[85630]: pgmap v673: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:48.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:48 smithi067 ceph-mon[112438]: pgmap v674: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:48.386 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:48 smithi134 ceph-mon[85630]: pgmap v674: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:50.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:50 smithi067 ceph-mon[112438]: pgmap v675: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:50.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:50 smithi134 ceph-mon[85630]: pgmap v675: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:52.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:52 smithi067 ceph-mon[112438]: pgmap v676: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:52.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:52 smithi134 ceph-mon[85630]: pgmap v676: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:54.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:54 smithi067 ceph-mon[112438]: pgmap v677: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:54.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:54 smithi134 ceph-mon[85630]: pgmap v677: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:55.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.399+0000 7f399fa74700 1 -- 172.21.15.67:0/3015712860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3998068580 msgr2=0x7f3998068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:55.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.399+0000 7f399fa74700 1 --2- 172.21.15.67:0/3015712860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3998068580 0x7f3998068960 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f3994009910 tx=0x7f3994009c20 comp rx=0 tx=0).stop 2024-04-22T23:48:55.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.399+0000 7f399fa74700 1 -- 172.21.15.67:0/3015712860 shutdown_connections 2024-04-22T23:48:55.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.399+0000 7f399fa74700 1 --2- 172.21.15.67:0/3015712860 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3998068ea0 0x7f399810d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:55.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.399+0000 7f399fa74700 1 --2- 172.21.15.67:0/3015712860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3998068580 0x7f3998068960 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:55.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.399+0000 7f399fa74700 1 -- 172.21.15.67:0/3015712860 >> 172.21.15.67:0/3015712860 conn(0x7f3998075670 msgr2=0x7f3998075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:48:55.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.399+0000 7f399fa74700 1 -- 172.21.15.67:0/3015712860 shutdown_connections 2024-04-22T23:48:55.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.399+0000 7f399fa74700 1 -- 172.21.15.67:0/3015712860 wait complete. 2024-04-22T23:48:55.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.400+0000 7f399fa74700 1 Processor -- start 2024-04-22T23:48:55.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.400+0000 7f399fa74700 1 -- start start 2024-04-22T23:48:55.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.400+0000 7f399fa74700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3998068580 0x7f399819e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:55.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.400+0000 7f399fa74700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3998068ea0 0x7f399819eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:55.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.400+0000 7f399fa74700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f399819f430 con 0x7f3998068580 2024-04-22T23:48:55.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.400+0000 7f399fa74700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39981988e0 con 0x7f3998068ea0 2024-04-22T23:48:55.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.401+0000 7f399d810700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3998068580 0x7f399819e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:55.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.401+0000 7f399d810700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3998068580 0x7f399819e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44982/0 (socket says 172.21.15.67:44982) 2024-04-22T23:48:55.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.401+0000 7f399d810700 1 -- 172.21.15.67:0/1400696494 learned_addr learned my addr 172.21.15.67:0/1400696494 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:48:55.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.401+0000 7f399d00f700 1 --2- 172.21.15.67:0/1400696494 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3998068ea0 0x7f399819eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:55.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.401+0000 7f399d810700 1 -- 172.21.15.67:0/1400696494 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3998068ea0 msgr2=0x7f399819eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:55.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.401+0000 7f399d810700 1 --2- 172.21.15.67:0/1400696494 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3998068ea0 0x7f399819eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:55.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.401+0000 7f399d810700 1 -- 172.21.15.67:0/1400696494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39940095f0 con 0x7f3998068580 2024-04-22T23:48:55.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.401+0000 7f399d00f700 1 --2- 172.21.15.67:0/1400696494 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3998068ea0 0x7f399819eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:48:55.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.401+0000 7f399d810700 1 --2- 172.21.15.67:0/1400696494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3998068580 0x7f399819e860 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f399400b5c0 tx=0x7f399400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:48:55.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.401+0000 7f398effd700 1 -- 172.21.15.67:0/1400696494 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f399401d070 con 0x7f3998068580 2024-04-22T23:48:55.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.401+0000 7f398effd700 1 -- 172.21.15.67:0/1400696494 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3994005e00 con 0x7f3998068580 2024-04-22T23:48:55.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.401+0000 7f399fa74700 1 -- 172.21.15.67:0/1400696494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3998198b60 con 0x7f3998068580 2024-04-22T23:48:55.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.401+0000 7f398effd700 1 -- 172.21.15.67:0/1400696494 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3994017400 con 0x7f3998068580 2024-04-22T23:48:55.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.401+0000 7f399fa74700 1 -- 172.21.15.67:0/1400696494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3998199050 con 0x7f3998068580 2024-04-22T23:48:55.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.403+0000 7f399fa74700 1 -- 172.21.15.67:0/1400696494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f399810aa00 con 0x7f3998068580 2024-04-22T23:48:55.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.403+0000 7f398effd700 1 -- 172.21.15.67:0/1400696494 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f399400f460 con 0x7f3998068580 2024-04-22T23:48:55.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.403+0000 7f398effd700 1 --2- 172.21.15.67:0/1400696494 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f398406ecc0 0x7f3984071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:55.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.404+0000 7f399d00f700 1 --2- 172.21.15.67:0/1400696494 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f398406ecc0 0x7f3984071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:55.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.404+0000 7f398effd700 1 -- 172.21.15.67:0/1400696494 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f3994090550 con 0x7f3998068580 2024-04-22T23:48:55.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.404+0000 7f399d00f700 1 --2- 172.21.15.67:0/1400696494 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f398406ecc0 0x7f3984071180 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3988005950 tx=0x7f39880058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:48:55.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.408+0000 7f398effd700 1 -- 172.21.15.67:0/1400696494 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f399405ccc0 con 0x7f3998068580 2024-04-22T23:48:55.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.641+0000 7f399fa74700 1 -- 172.21.15.67:0/1400696494 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3998199eb0 con 0x7f398406ecc0 2024-04-22T23:48:55.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.642+0000 7f398effd700 1 -- 172.21.15.67:0/1400696494 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3998199eb0 con 0x7f398406ecc0 2024-04-22T23:48:55.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.644+0000 7f399fa74700 1 -- 172.21.15.67:0/1400696494 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f398406ecc0 msgr2=0x7f3984071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:55.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.644+0000 7f399fa74700 1 --2- 172.21.15.67:0/1400696494 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f398406ecc0 0x7f3984071180 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3988005950 tx=0x7f39880058e0 comp rx=0 tx=0).stop 2024-04-22T23:48:55.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.644+0000 7f399fa74700 1 -- 172.21.15.67:0/1400696494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3998068580 msgr2=0x7f399819e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:55.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.644+0000 7f399fa74700 1 --2- 172.21.15.67:0/1400696494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3998068580 0x7f399819e860 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f399400b5c0 tx=0x7f399400be20 comp rx=0 tx=0).stop 2024-04-22T23:48:55.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.645+0000 7f399fa74700 1 -- 172.21.15.67:0/1400696494 shutdown_connections 2024-04-22T23:48:55.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.645+0000 7f399fa74700 1 --2- 172.21.15.67:0/1400696494 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f398406ecc0 0x7f3984071180 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:55.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.645+0000 7f399fa74700 1 --2- 172.21.15.67:0/1400696494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3998068580 0x7f399819e860 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:55.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.645+0000 7f399fa74700 1 --2- 172.21.15.67:0/1400696494 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3998068ea0 0x7f399819eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:55.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.645+0000 7f399fa74700 1 -- 172.21.15.67:0/1400696494 >> 172.21.15.67:0/1400696494 conn(0x7f3998075670 msgr2=0x7f39980fe790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:48:55.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.645+0000 7f399fa74700 1 -- 172.21.15.67:0/1400696494 shutdown_connections 2024-04-22T23:48:55.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.645+0000 7f399fa74700 1 -- 172.21.15.67:0/1400696494 wait complete. 2024-04-22T23:48:55.647 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:48:55.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.752+0000 7f540d84f700 1 -- 172.21.15.67:0/1059629133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5408100750 msgr2=0x7f54081047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:55.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.752+0000 7f540d84f700 1 --2- 172.21.15.67:0/1059629133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5408100750 0x7f54081047a0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f53f8009910 tx=0x7f53f8009c20 comp rx=0 tx=0).stop 2024-04-22T23:48:55.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.752+0000 7f540d84f700 1 -- 172.21.15.67:0/1059629133 shutdown_connections 2024-04-22T23:48:55.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.752+0000 7f540d84f700 1 --2- 172.21.15.67:0/1059629133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5408100750 0x7f54081047a0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:55.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.752+0000 7f540d84f700 1 --2- 172.21.15.67:0/1059629133 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f54080ffda0 0x7f5408100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:55.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.752+0000 7f540d84f700 1 -- 172.21.15.67:0/1059629133 >> 172.21.15.67:0/1059629133 conn(0x7f54080fb650 msgr2=0x7f54080fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:48:55.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.752+0000 7f540d84f700 1 -- 172.21.15.67:0/1059629133 shutdown_connections 2024-04-22T23:48:55.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.752+0000 7f540d84f700 1 -- 172.21.15.67:0/1059629133 wait complete. 2024-04-22T23:48:55.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.753+0000 7f540d84f700 1 Processor -- start 2024-04-22T23:48:55.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.753+0000 7f540d84f700 1 -- start start 2024-04-22T23:48:55.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.753+0000 7f540d84f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54080ffda0 0x7f540819c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:55.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.753+0000 7f540d84f700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5408100750 0x7f540819cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:55.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.753+0000 7f540d84f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540819d240 con 0x7f54080ffda0 2024-04-22T23:48:55.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.753+0000 7f540d84f700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54081966f0 con 0x7f5408100750 2024-04-22T23:48:55.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.754+0000 7f5406ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54080ffda0 0x7f540819c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:55.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.754+0000 7f5406ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54080ffda0 0x7f540819c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45010/0 (socket says 172.21.15.67:45010) 2024-04-22T23:48:55.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.754+0000 7f5406ffd700 1 -- 172.21.15.67:0/3633764381 learned_addr learned my addr 172.21.15.67:0/3633764381 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:48:55.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.754+0000 7f54067fc700 1 --2- 172.21.15.67:0/3633764381 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5408100750 0x7f540819cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:55.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.754+0000 7f5406ffd700 1 -- 172.21.15.67:0/3633764381 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5408100750 msgr2=0x7f540819cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:55.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.754+0000 7f5406ffd700 1 --2- 172.21.15.67:0/3633764381 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5408100750 0x7f540819cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:55.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.754+0000 7f5406ffd700 1 -- 172.21.15.67:0/3633764381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f80095f0 con 0x7f54080ffda0 2024-04-22T23:48:55.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.754+0000 7f5406ffd700 1 --2- 172.21.15.67:0/3633764381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54080ffda0 0x7f540819c670 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f53f000e920 tx=0x7f53f000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:48:55.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.754+0000 7f540c84d700 1 -- 172.21.15.67:0/3633764381 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53f000c9d0 con 0x7f54080ffda0 2024-04-22T23:48:55.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.754+0000 7f540c84d700 1 -- 172.21.15.67:0/3633764381 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f53f0010430 con 0x7f54080ffda0 2024-04-22T23:48:55.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.754+0000 7f540d84f700 1 -- 172.21.15.67:0/3633764381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54081969d0 con 0x7f54080ffda0 2024-04-22T23:48:55.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.754+0000 7f540c84d700 1 -- 172.21.15.67:0/3633764381 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53f0018630 con 0x7f54080ffda0 2024-04-22T23:48:55.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.754+0000 7f540d84f700 1 -- 172.21.15.67:0/3633764381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5408196ef0 con 0x7f54080ffda0 2024-04-22T23:48:55.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.755+0000 7f540d84f700 1 -- 172.21.15.67:0/3633764381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54081091b0 con 0x7f54080ffda0 2024-04-22T23:48:55.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.756+0000 7f540c84d700 1 -- 172.21.15.67:0/3633764381 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f53f00105a0 con 0x7f54080ffda0 2024-04-22T23:48:55.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.756+0000 7f540c84d700 1 --2- 172.21.15.67:0/3633764381 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f53f406ecc0 0x7f53f4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:55.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.756+0000 7f540c84d700 1 -- 172.21.15.67:0/3633764381 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f53f0014070 con 0x7f54080ffda0 2024-04-22T23:48:55.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.756+0000 7f54067fc700 1 --2- 172.21.15.67:0/3633764381 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f53f406ecc0 0x7f53f4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:55.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.757+0000 7f54067fc700 1 --2- 172.21.15.67:0/3633764381 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f53f406ecc0 0x7f53f4071180 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f53f80059d0 tx=0x7f53f8005940 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:48:55.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.761+0000 7f540c84d700 1 -- 172.21.15.67:0/3633764381 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f53f0054150 con 0x7f54080ffda0 2024-04-22T23:48:55.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.988+0000 7f540d84f700 1 -- 172.21.15.67:0/3633764381 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5408197110 con 0x7f53f406ecc0 2024-04-22T23:48:55.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.990+0000 7f540c84d700 1 -- 172.21.15.67:0/3633764381 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f5408197110 con 0x7f53f406ecc0 2024-04-22T23:48:55.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.992+0000 7f540d84f700 1 -- 172.21.15.67:0/3633764381 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f53f406ecc0 msgr2=0x7f53f4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:55.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.992+0000 7f540d84f700 1 --2- 172.21.15.67:0/3633764381 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f53f406ecc0 0x7f53f4071180 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f53f80059d0 tx=0x7f53f8005940 comp rx=0 tx=0).stop 2024-04-22T23:48:55.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.992+0000 7f540d84f700 1 -- 172.21.15.67:0/3633764381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54080ffda0 msgr2=0x7f540819c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:55.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.992+0000 7f540d84f700 1 --2- 172.21.15.67:0/3633764381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54080ffda0 0x7f540819c670 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f53f000e920 tx=0x7f53f000ec30 comp rx=0 tx=0).stop 2024-04-22T23:48:55.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.992+0000 7f540d84f700 1 -- 172.21.15.67:0/3633764381 shutdown_connections 2024-04-22T23:48:55.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.992+0000 7f540d84f700 1 --2- 172.21.15.67:0/3633764381 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f53f406ecc0 0x7f53f4071180 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:55.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.992+0000 7f540d84f700 1 --2- 172.21.15.67:0/3633764381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54080ffda0 0x7f540819c670 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:55.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.992+0000 7f540d84f700 1 --2- 172.21.15.67:0/3633764381 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5408100750 0x7f540819cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:55.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.992+0000 7f540d84f700 1 -- 172.21.15.67:0/3633764381 >> 172.21.15.67:0/3633764381 conn(0x7f54080fb650 msgr2=0x7f5408069150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:48:55.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.992+0000 7f540d84f700 1 -- 172.21.15.67:0/3633764381 shutdown_connections 2024-04-22T23:48:55.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:55.992+0000 7f540d84f700 1 -- 172.21.15.67:0/3633764381 wait complete. 2024-04-22T23:48:56.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.100+0000 7f6be3e70700 1 -- 172.21.15.67:0/3315882829 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bdc0fff50 msgr2=0x7f6bdc105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:56.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.100+0000 7f6be3e70700 1 --2- 172.21.15.67:0/3315882829 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bdc0fff50 0x7f6bdc105840 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc009910 tx=0x7f6bcc009c20 comp rx=0 tx=0).stop 2024-04-22T23:48:56.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.100+0000 7f6be3e70700 1 -- 172.21.15.67:0/3315882829 shutdown_connections 2024-04-22T23:48:56.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.100+0000 7f6be3e70700 1 --2- 172.21.15.67:0/3315882829 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bdc0fff50 0x7f6bdc105840 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:56.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.100+0000 7f6be3e70700 1 --2- 172.21.15.67:0/3315882829 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6bdc0ff5a0 0x7f6bdc0ff980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:56.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.100+0000 7f6be3e70700 1 -- 172.21.15.67:0/3315882829 >> 172.21.15.67:0/3315882829 conn(0x7f6bdc074e30 msgr2=0x7f6bdc075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:48:56.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.100+0000 7f6be3e70700 1 -- 172.21.15.67:0/3315882829 shutdown_connections 2024-04-22T23:48:56.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.100+0000 7f6be3e70700 1 -- 172.21.15.67:0/3315882829 wait complete. 2024-04-22T23:48:56.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.100+0000 7f6be3e70700 1 Processor -- start 2024-04-22T23:48:56.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.100+0000 7f6be3e70700 1 -- start start 2024-04-22T23:48:56.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.101+0000 7f6be3e70700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bdc0ff5a0 0x7f6bdc196990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:56.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.101+0000 7f6be3e70700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6bdc0fff50 0x7f6bdc196ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:56.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.101+0000 7f6be3e70700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bdc197560 con 0x7f6bdc0ff5a0 2024-04-22T23:48:56.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.101+0000 7f6be3e70700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bdc19b370 con 0x7f6bdc0fff50 2024-04-22T23:48:56.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.101+0000 7f6be1c0c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bdc0ff5a0 0x7f6bdc196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:56.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.101+0000 7f6be1c0c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bdc0ff5a0 0x7f6bdc196990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45026/0 (socket says 172.21.15.67:45026) 2024-04-22T23:48:56.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.101+0000 7f6be1c0c700 1 -- 172.21.15.67:0/1838917360 learned_addr learned my addr 172.21.15.67:0/1838917360 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:48:56.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.101+0000 7f6be140b700 1 --2- 172.21.15.67:0/1838917360 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6bdc0fff50 0x7f6bdc196ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:56.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.101+0000 7f6be1c0c700 1 -- 172.21.15.67:0/1838917360 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6bdc0fff50 msgr2=0x7f6bdc196ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:56.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.101+0000 7f6be1c0c700 1 --2- 172.21.15.67:0/1838917360 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6bdc0fff50 0x7f6bdc196ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:56.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.101+0000 7f6be1c0c700 1 -- 172.21.15.67:0/1838917360 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bcc0095f0 con 0x7f6bdc0ff5a0 2024-04-22T23:48:56.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.101+0000 7f6be140b700 1 --2- 172.21.15.67:0/1838917360 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6bdc0fff50 0x7f6bdc196ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:48:56.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.101+0000 7f6be1c0c700 1 --2- 172.21.15.67:0/1838917360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bdc0ff5a0 0x7f6bdc196990 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f6bd800b4b0 tx=0x7f6bd800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:48:56.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.101+0000 7f6bd2ffd700 1 -- 172.21.15.67:0/1838917360 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6bd8011800 con 0x7f6bdc0ff5a0 2024-04-22T23:48:56.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.101+0000 7f6be3e70700 1 -- 172.21.15.67:0/1838917360 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bdc19b650 con 0x7f6bdc0ff5a0 2024-04-22T23:48:56.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.101+0000 7f6bd2ffd700 1 -- 172.21.15.67:0/1838917360 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6bd8011e20 con 0x7f6bdc0ff5a0 2024-04-22T23:48:56.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.101+0000 7f6bd2ffd700 1 -- 172.21.15.67:0/1838917360 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6bd800f2e0 con 0x7f6bdc0ff5a0 2024-04-22T23:48:56.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.102+0000 7f6be3e70700 1 -- 172.21.15.67:0/1838917360 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bdc19bb70 con 0x7f6bdc0ff5a0 2024-04-22T23:48:56.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.102+0000 7f6be3e70700 1 -- 172.21.15.67:0/1838917360 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bdc109200 con 0x7f6bdc0ff5a0 2024-04-22T23:48:56.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.103+0000 7f6bd2ffd700 1 -- 172.21.15.67:0/1838917360 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6bd8010380 con 0x7f6bdc0ff5a0 2024-04-22T23:48:56.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.103+0000 7f6bd2ffd700 1 --2- 172.21.15.67:0/1838917360 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6bc806ecc0 0x7f6bc8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:56.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.104+0000 7f6bd2ffd700 1 -- 172.21.15.67:0/1838917360 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f6bd808eb70 con 0x7f6bdc0ff5a0 2024-04-22T23:48:56.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.104+0000 7f6be140b700 1 --2- 172.21.15.67:0/1838917360 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6bc806ecc0 0x7f6bc8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:56.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.104+0000 7f6be140b700 1 --2- 172.21.15.67:0/1838917360 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6bc806ecc0 0x7f6bc8071180 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc197900 tx=0x7f6bcc01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:48:56.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.108+0000 7f6bd2ffd700 1 -- 172.21.15.67:0/1838917360 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f6bd805b2e0 con 0x7f6bdc0ff5a0 2024-04-22T23:48:56.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:56 smithi067 ceph-mon[112438]: pgmap v678: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:56.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.330+0000 7f6be3e70700 1 -- 172.21.15.67:0/1838917360 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6bdc197dd0 con 0x7f6bc806ecc0 2024-04-22T23:48:56.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.341+0000 7f6bd2ffd700 1 -- 172.21.15.67:0/1838917360 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f6bdc197dd0 con 0x7f6bc806ecc0 2024-04-22T23:48:56.334 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:48:56.334 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (22m) 25s ago 33m 24.4M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:48:56.334 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (34m) 25s ago 34m 10.0M - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:48:56.334 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (33m) 28s ago 33m 10.0M - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:48:56.335 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (20m) 25s ago 34m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:48:56.335 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (19m) 28s ago 33m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:48:56.335 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (22m) 25s ago 33m 81.2M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:48:56.335 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (16m) 25s ago 28m 19.4M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:48:56.335 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (16m) 25s ago 28m 20.3M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:48:56.335 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (15m) 28s ago 28m 20.1M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:48:56.336 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (32s) 28s ago 28m 14.0M - 19.0.0-3252-g43be0201 a0434699c1d7 46d3d7e86fee 2024-04-22T23:48:56.336 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (24m) 25s ago 35m 514M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:48:56.336 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (23m) 28s ago 32m 459M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:48:56.336 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (21m) 25s ago 35m 74.2M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:48:56.336 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (20m) 28s ago 32m 54.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:48:56.336 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (23m) 25s ago 34m 15.2M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:48:56.336 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (23m) 28s ago 33m 16.0M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:48:56.337 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (19m) 25s ago 32m 102M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:48:56.337 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (18m) 25s ago 31m 128M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:48:56.337 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (18m) 25s ago 31m 83.8M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:48:56.337 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (17m) 28s ago 30m 101M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:48:56.337 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (17m) 28s ago 30m 86.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:48:56.337 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (16m) 28s ago 30m 81.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:48:56.337 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (22m) 25s ago 33m 78.7M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:48:56.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.343+0000 7f6be3e70700 1 -- 172.21.15.67:0/1838917360 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6bc806ecc0 msgr2=0x7f6bc8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:56.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.343+0000 7f6be3e70700 1 --2- 172.21.15.67:0/1838917360 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6bc806ecc0 0x7f6bc8071180 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc197900 tx=0x7f6bcc01a040 comp rx=0 tx=0).stop 2024-04-22T23:48:56.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.344+0000 7f6be3e70700 1 -- 172.21.15.67:0/1838917360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bdc0ff5a0 msgr2=0x7f6bdc196990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:56.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.344+0000 7f6be3e70700 1 --2- 172.21.15.67:0/1838917360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bdc0ff5a0 0x7f6bdc196990 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f6bd800b4b0 tx=0x7f6bd800b870 comp rx=0 tx=0).stop 2024-04-22T23:48:56.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.344+0000 7f6be3e70700 1 -- 172.21.15.67:0/1838917360 shutdown_connections 2024-04-22T23:48:56.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.344+0000 7f6be3e70700 1 --2- 172.21.15.67:0/1838917360 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6bc806ecc0 0x7f6bc8071180 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:56.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.344+0000 7f6be3e70700 1 --2- 172.21.15.67:0/1838917360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bdc0ff5a0 0x7f6bdc196990 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:56.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.344+0000 7f6be3e70700 1 --2- 172.21.15.67:0/1838917360 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6bdc0fff50 0x7f6bdc196ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:56.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.344+0000 7f6be3e70700 1 -- 172.21.15.67:0/1838917360 >> 172.21.15.67:0/1838917360 conn(0x7f6bdc074e30 msgr2=0x7f6bdc0fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:48:56.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.344+0000 7f6be3e70700 1 -- 172.21.15.67:0/1838917360 shutdown_connections 2024-04-22T23:48:56.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.344+0000 7f6be3e70700 1 -- 172.21.15.67:0/1838917360 wait complete. 2024-04-22T23:48:56.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.449+0000 7f5f583a1700 1 -- 172.21.15.67:0/1181473387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f50101e00 msgr2=0x7f5f501021e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:56.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.449+0000 7f5f583a1700 1 --2- 172.21.15.67:0/1181473387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f50101e00 0x7f5f501021e0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c009910 tx=0x7f5f4c009c20 comp rx=0 tx=0).stop 2024-04-22T23:48:56.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.450+0000 7f5f583a1700 1 -- 172.21.15.67:0/1181473387 shutdown_connections 2024-04-22T23:48:56.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.450+0000 7f5f583a1700 1 --2- 172.21.15.67:0/1181473387 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5f50102720 0x7f5f5010ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:56.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.450+0000 7f5f583a1700 1 --2- 172.21.15.67:0/1181473387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f50101e00 0x7f5f501021e0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:56.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.450+0000 7f5f583a1700 1 -- 172.21.15.67:0/1181473387 >> 172.21.15.67:0/1181473387 conn(0x7f5f500fb650 msgr2=0x7f5f500fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:48:56.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.450+0000 7f5f583a1700 1 -- 172.21.15.67:0/1181473387 shutdown_connections 2024-04-22T23:48:56.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.450+0000 7f5f583a1700 1 -- 172.21.15.67:0/1181473387 wait complete. 2024-04-22T23:48:56.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.450+0000 7f5f583a1700 1 Processor -- start 2024-04-22T23:48:56.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.450+0000 7f5f583a1700 1 -- start start 2024-04-22T23:48:56.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.450+0000 7f5f583a1700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5f50101e00 0x7f5f500ffa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:56.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.450+0000 7f5f583a1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f50102720 0x7f5f500fffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:56.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.450+0000 7f5f583a1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f501004e0 con 0x7f5f50102720 2024-04-22T23:48:56.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.450+0000 7f5f583a1700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f50100620 con 0x7f5f50101e00 2024-04-22T23:48:56.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.451+0000 7f5f5593c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f50102720 0x7f5f500fffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:56.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.451+0000 7f5f5593c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f50102720 0x7f5f500fffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45032/0 (socket says 172.21.15.67:45032) 2024-04-22T23:48:56.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.451+0000 7f5f5593c700 1 -- 172.21.15.67:0/3207023861 learned_addr learned my addr 172.21.15.67:0/3207023861 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:48:56.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.451+0000 7f5f5613d700 1 --2- 172.21.15.67:0/3207023861 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5f50101e00 0x7f5f500ffa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:56.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.451+0000 7f5f5593c700 1 -- 172.21.15.67:0/3207023861 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5f50101e00 msgr2=0x7f5f500ffa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:56.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.451+0000 7f5f5593c700 1 --2- 172.21.15.67:0/3207023861 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5f50101e00 0x7f5f500ffa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:56.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.451+0000 7f5f5593c700 1 -- 172.21.15.67:0/3207023861 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f4c0095f0 con 0x7f5f50102720 2024-04-22T23:48:56.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.451+0000 7f5f5613d700 1 --2- 172.21.15.67:0/3207023861 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5f50101e00 0x7f5f500ffa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:48:56.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.451+0000 7f5f5593c700 1 --2- 172.21.15.67:0/3207023861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f50102720 0x7f5f500fffa0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f5f4000d8d0 tx=0x7f5f4000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:48:56.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.451+0000 7f5f477fe700 1 -- 172.21.15.67:0/3207023861 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f400041f0 con 0x7f5f50102720 2024-04-22T23:48:56.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.451+0000 7f5f477fe700 1 -- 172.21.15.67:0/3207023861 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5f40004350 con 0x7f5f50102720 2024-04-22T23:48:56.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.451+0000 7f5f583a1700 1 -- 172.21.15.67:0/3207023861 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f501a2a90 con 0x7f5f50102720 2024-04-22T23:48:56.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.451+0000 7f5f477fe700 1 -- 172.21.15.67:0/3207023861 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f40003d60 con 0x7f5f50102720 2024-04-22T23:48:56.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.451+0000 7f5f583a1700 1 -- 172.21.15.67:0/3207023861 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f501a2ea0 con 0x7f5f50102720 2024-04-22T23:48:56.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.452+0000 7f5f583a1700 1 -- 172.21.15.67:0/3207023861 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f50108310 con 0x7f5f50102720 2024-04-22T23:48:56.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.453+0000 7f5f477fe700 1 -- 172.21.15.67:0/3207023861 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5f40009730 con 0x7f5f50102720 2024-04-22T23:48:56.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.453+0000 7f5f477fe700 1 --2- 172.21.15.67:0/3207023861 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5f3c06ecc0 0x7f5f3c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:56.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.453+0000 7f5f477fe700 1 -- 172.21.15.67:0/3207023861 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f5f4008eb70 con 0x7f5f50102720 2024-04-22T23:48:56.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.453+0000 7f5f5613d700 1 --2- 172.21.15.67:0/3207023861 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5f3c06ecc0 0x7f5f3c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:56.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.454+0000 7f5f5613d700 1 --2- 172.21.15.67:0/3207023861 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5f3c06ecc0 0x7f5f3c071180 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c00b5c0 tx=0x7f5f4c0053a0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:48:56.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.458+0000 7f5f477fe700 1 -- 172.21.15.67:0/3207023861 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f5f40057830 con 0x7f5f50102720 2024-04-22T23:48:56.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:56 smithi134 ceph-mon[85630]: pgmap v678: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:56.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.756+0000 7f5f583a1700 1 -- 172.21.15.67:0/3207023861 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5f5004f2e0 con 0x7f5f50102720 2024-04-22T23:48:56.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.756+0000 7f5f477fe700 1 -- 172.21.15.67:0/3207023861 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f5f40017280 con 0x7f5f50102720 2024-04-22T23:48:56.748 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:48:56.748 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:48:56.748 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:48:56.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:48:56.748 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:48:56.749 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:48:56.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:48:56.749 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:48:56.749 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:48:56.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:48:56.749 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:48:56.749 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 4 2024-04-22T23:48:56.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:48:56.749 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:48:56.749 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 14 2024-04-22T23:48:56.749 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:48:56.749 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:48:56.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.759+0000 7f5f583a1700 1 -- 172.21.15.67:0/3207023861 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5f3c06ecc0 msgr2=0x7f5f3c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:56.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.759+0000 7f5f583a1700 1 --2- 172.21.15.67:0/3207023861 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5f3c06ecc0 0x7f5f3c071180 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c00b5c0 tx=0x7f5f4c0053a0 comp rx=0 tx=0).stop 2024-04-22T23:48:56.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.759+0000 7f5f583a1700 1 -- 172.21.15.67:0/3207023861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f50102720 msgr2=0x7f5f500fffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:56.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.759+0000 7f5f583a1700 1 --2- 172.21.15.67:0/3207023861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f50102720 0x7f5f500fffa0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f5f4000d8d0 tx=0x7f5f4000dc90 comp rx=0 tx=0).stop 2024-04-22T23:48:56.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.759+0000 7f5f583a1700 1 -- 172.21.15.67:0/3207023861 shutdown_connections 2024-04-22T23:48:56.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.759+0000 7f5f583a1700 1 --2- 172.21.15.67:0/3207023861 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5f3c06ecc0 0x7f5f3c071180 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:56.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.759+0000 7f5f583a1700 1 --2- 172.21.15.67:0/3207023861 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5f50101e00 0x7f5f500ffa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:56.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.759+0000 7f5f583a1700 1 --2- 172.21.15.67:0/3207023861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f50102720 0x7f5f500fffa0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:56.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.759+0000 7f5f583a1700 1 -- 172.21.15.67:0/3207023861 >> 172.21.15.67:0/3207023861 conn(0x7f5f500fb650 msgr2=0x7f5f50105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:48:56.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.759+0000 7f5f583a1700 1 -- 172.21.15.67:0/3207023861 shutdown_connections 2024-04-22T23:48:56.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.759+0000 7f5f583a1700 1 -- 172.21.15.67:0/3207023861 wait complete. 2024-04-22T23:48:56.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.866+0000 7fd153a37700 1 -- 172.21.15.67:0/12745018 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd14c101850 msgr2=0x7fd14c1058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:56.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.866+0000 7fd153a37700 1 --2- 172.21.15.67:0/12745018 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd14c101850 0x7fd14c1058a0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fd148009910 tx=0x7fd148009c20 comp rx=0 tx=0).stop 2024-04-22T23:48:56.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.866+0000 7fd153a37700 1 -- 172.21.15.67:0/12745018 shutdown_connections 2024-04-22T23:48:56.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.866+0000 7fd153a37700 1 --2- 172.21.15.67:0/12745018 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd14c101850 0x7fd14c1058a0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:56.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.866+0000 7fd153a37700 1 --2- 172.21.15.67:0/12745018 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd14c100ea0 0x7fd14c101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:56.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.866+0000 7fd153a37700 1 -- 172.21.15.67:0/12745018 >> 172.21.15.67:0/12745018 conn(0x7fd14c0fc750 msgr2=0x7fd14c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:48:56.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.866+0000 7fd153a37700 1 -- 172.21.15.67:0/12745018 shutdown_connections 2024-04-22T23:48:56.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.866+0000 7fd153a37700 1 -- 172.21.15.67:0/12745018 wait complete. 2024-04-22T23:48:56.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.867+0000 7fd153a37700 1 Processor -- start 2024-04-22T23:48:56.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.867+0000 7fd153a37700 1 -- start start 2024-04-22T23:48:56.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.867+0000 7fd153a37700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd14c100ea0 0x7fd14c19c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:56.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.867+0000 7fd153a37700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd14c101850 0x7fd14c19cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:56.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.867+0000 7fd153a37700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14c19d240 con 0x7fd14c101850 2024-04-22T23:48:56.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.867+0000 7fd153a37700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14c196780 con 0x7fd14c100ea0 2024-04-22T23:48:56.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.867+0000 7fd150fd2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd14c101850 0x7fd14c19cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:56.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.867+0000 7fd150fd2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd14c101850 0x7fd14c19cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45050/0 (socket says 172.21.15.67:45050) 2024-04-22T23:48:56.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.867+0000 7fd150fd2700 1 -- 172.21.15.67:0/4029148074 learned_addr learned my addr 172.21.15.67:0/4029148074 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:48:56.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.868+0000 7fd1517d3700 1 --2- 172.21.15.67:0/4029148074 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd14c100ea0 0x7fd14c19c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:56.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.868+0000 7fd150fd2700 1 -- 172.21.15.67:0/4029148074 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd14c100ea0 msgr2=0x7fd14c19c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:56.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.868+0000 7fd150fd2700 1 --2- 172.21.15.67:0/4029148074 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd14c100ea0 0x7fd14c19c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:56.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.868+0000 7fd150fd2700 1 -- 172.21.15.67:0/4029148074 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1480095f0 con 0x7fd14c101850 2024-04-22T23:48:56.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.868+0000 7fd150fd2700 1 --2- 172.21.15.67:0/4029148074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd14c101850 0x7fd14c19cc40 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fd14800bda0 tx=0x7fd14800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:48:56.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.868+0000 7fd1427fc700 1 -- 172.21.15.67:0/4029148074 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd14801d070 con 0x7fd14c101850 2024-04-22T23:48:56.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.868+0000 7fd1427fc700 1 -- 172.21.15.67:0/4029148074 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd148004fb0 con 0x7fd14c101850 2024-04-22T23:48:56.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.868+0000 7fd153a37700 1 -- 172.21.15.67:0/4029148074 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd14c196a00 con 0x7fd14c101850 2024-04-22T23:48:56.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.868+0000 7fd1427fc700 1 -- 172.21.15.67:0/4029148074 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd14800f480 con 0x7fd14c101850 2024-04-22T23:48:56.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.868+0000 7fd153a37700 1 -- 172.21.15.67:0/4029148074 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd14c196ef0 con 0x7fd14c101850 2024-04-22T23:48:56.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.869+0000 7fd153a37700 1 -- 172.21.15.67:0/4029148074 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd14c1092a0 con 0x7fd14c101850 2024-04-22T23:48:56.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.869+0000 7fd1427fc700 1 -- 172.21.15.67:0/4029148074 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd148022870 con 0x7fd14c101850 2024-04-22T23:48:56.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.870+0000 7fd1427fc700 1 --2- 172.21.15.67:0/4029148074 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd13806ecc0 0x7fd138071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:56.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.870+0000 7fd1427fc700 1 -- 172.21.15.67:0/4029148074 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fd148090440 con 0x7fd14c101850 2024-04-22T23:48:56.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.870+0000 7fd1517d3700 1 --2- 172.21.15.67:0/4029148074 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd13806ecc0 0x7fd138071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:56.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.870+0000 7fd1517d3700 1 --2- 172.21.15.67:0/4029148074 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd13806ecc0 0x7fd138071180 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd13c00ba60 tx=0x7fd13c005d50 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:48:56.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:56.875+0000 7fd1427fc700 1 -- 172.21.15.67:0/4029148074 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fd14805cce0 con 0x7fd14c101850 2024-04-22T23:48:57.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.131+0000 7fd153a37700 1 -- 172.21.15.67:0/4029148074 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd14c197c90 con 0x7fd14c101850 2024-04-22T23:48:57.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.131+0000 7fd1427fc700 1 -- 172.21.15.67:0/4029148074 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1872 (secure 0 0 0) 0x7fd14805c870 con 0x7fd14c101850 2024-04-22T23:48:57.123 INFO:teuthology.orchestra.run.smithi067.stdout:e33 2024-04-22T23:48:57.123 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:48:57.123 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:48:57.123 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:48:57.123 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:48:57.123 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:48:57.124 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:48:57.124 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:48:57.124 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:48:57.124 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:48:57.124 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:48:57.124 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:48:57.124 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:48:57.124 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:48:57.124 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:48:57.124 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:48:57.124 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:48:57.124 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:48:57.124 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:48:57.124 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:48:57.124 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:48:57.125 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:48:57.125 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:48:57.125 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:48:57.125 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:48:57.125 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:48:57.125 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:48:57.125 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:48:57.125 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:48:57.125 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:48:57.125 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:48:57.125 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:48:57.125 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:48:57.125 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:48:57.125 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:48:57.125 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:48:57.126 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:48:57.126 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:48:57.126 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:48:57.126 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:48:57.126 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:48:57.126 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44851} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1849648622,v1:172.21.15.134:6827/1849648622] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:48:57.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.133+0000 7fd153a37700 1 -- 172.21.15.67:0/4029148074 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd13806ecc0 msgr2=0x7fd138071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:57.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.133+0000 7fd153a37700 1 --2- 172.21.15.67:0/4029148074 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd13806ecc0 0x7fd138071180 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd13c00ba60 tx=0x7fd13c005d50 comp rx=0 tx=0).stop 2024-04-22T23:48:57.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.133+0000 7fd153a37700 1 -- 172.21.15.67:0/4029148074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd14c101850 msgr2=0x7fd14c19cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:57.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.133+0000 7fd153a37700 1 --2- 172.21.15.67:0/4029148074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd14c101850 0x7fd14c19cc40 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fd14800bda0 tx=0x7fd14800be80 comp rx=0 tx=0).stop 2024-04-22T23:48:57.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.133+0000 7fd153a37700 1 -- 172.21.15.67:0/4029148074 shutdown_connections 2024-04-22T23:48:57.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.133+0000 7fd153a37700 1 --2- 172.21.15.67:0/4029148074 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd13806ecc0 0x7fd138071180 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:57.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.133+0000 7fd153a37700 1 --2- 172.21.15.67:0/4029148074 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd14c100ea0 0x7fd14c19c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:57.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.133+0000 7fd153a37700 1 --2- 172.21.15.67:0/4029148074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd14c101850 0x7fd14c19cc40 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:57.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.133+0000 7fd153a37700 1 -- 172.21.15.67:0/4029148074 >> 172.21.15.67:0/4029148074 conn(0x7fd14c0fc750 msgr2=0x7fd14c0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:48:57.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.133+0000 7fd153a37700 1 -- 172.21.15.67:0/4029148074 shutdown_connections 2024-04-22T23:48:57.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.133+0000 7fd153a37700 1 -- 172.21.15.67:0/4029148074 wait complete. 2024-04-22T23:48:57.127 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 33 2024-04-22T23:48:57.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.241+0000 7f0a5d175700 1 -- 172.21.15.67:0/378101619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a580ffda0 msgr2=0x7f0a58100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:57.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.241+0000 7f0a5d175700 1 --2- 172.21.15.67:0/378101619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a580ffda0 0x7f0a58100180 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f0a48009910 tx=0x7f0a48009c20 comp rx=0 tx=0).stop 2024-04-22T23:48:57.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.241+0000 7f0a5d175700 1 -- 172.21.15.67:0/378101619 shutdown_connections 2024-04-22T23:48:57.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.241+0000 7f0a5d175700 1 --2- 172.21.15.67:0/378101619 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0a58100750 0x7f0a581047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:57.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.241+0000 7f0a5d175700 1 --2- 172.21.15.67:0/378101619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a580ffda0 0x7f0a58100180 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:57.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.241+0000 7f0a5d175700 1 -- 172.21.15.67:0/378101619 >> 172.21.15.67:0/378101619 conn(0x7f0a580fb650 msgr2=0x7f0a580fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:48:57.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.241+0000 7f0a5d175700 1 -- 172.21.15.67:0/378101619 shutdown_connections 2024-04-22T23:48:57.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.242+0000 7f0a5d175700 1 -- 172.21.15.67:0/378101619 wait complete. 2024-04-22T23:48:57.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.242+0000 7f0a5d175700 1 Processor -- start 2024-04-22T23:48:57.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.242+0000 7f0a5d175700 1 -- start start 2024-04-22T23:48:57.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.243+0000 7f0a5d175700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a580ffda0 0x7f0a5819c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:57.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.243+0000 7f0a5d175700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0a58100750 0x7f0a5819cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:57.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.243+0000 7f0a5d175700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5819d230 con 0x7f0a580ffda0 2024-04-22T23:48:57.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.243+0000 7f0a5d175700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a581966e0 con 0x7f0a58100750 2024-04-22T23:48:57.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.243+0000 7f0a56d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a580ffda0 0x7f0a5819c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:57.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.243+0000 7f0a56d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a580ffda0 0x7f0a5819c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45064/0 (socket says 172.21.15.67:45064) 2024-04-22T23:48:57.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.243+0000 7f0a56d9d700 1 -- 172.21.15.67:0/2390851033 learned_addr learned my addr 172.21.15.67:0/2390851033 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:48:57.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.243+0000 7f0a5659c700 1 --2- 172.21.15.67:0/2390851033 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0a58100750 0x7f0a5819cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:57.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.243+0000 7f0a56d9d700 1 -- 172.21.15.67:0/2390851033 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0a58100750 msgr2=0x7f0a5819cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:57.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.243+0000 7f0a56d9d700 1 --2- 172.21.15.67:0/2390851033 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0a58100750 0x7f0a5819cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:57.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.243+0000 7f0a56d9d700 1 -- 172.21.15.67:0/2390851033 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a480095f0 con 0x7f0a580ffda0 2024-04-22T23:48:57.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.243+0000 7f0a5659c700 1 --2- 172.21.15.67:0/2390851033 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0a58100750 0x7f0a5819cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:48:57.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.243+0000 7f0a56d9d700 1 --2- 172.21.15.67:0/2390851033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a580ffda0 0x7f0a5819c660 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f0a48006010 tx=0x7f0a4800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:48:57.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.243+0000 7f0a3ffff700 1 -- 172.21.15.67:0/2390851033 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a4801d070 con 0x7f0a580ffda0 2024-04-22T23:48:57.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.243+0000 7f0a3ffff700 1 -- 172.21.15.67:0/2390851033 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0a4800f460 con 0x7f0a580ffda0 2024-04-22T23:48:57.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.243+0000 7f0a5d175700 1 -- 172.21.15.67:0/2390851033 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a58196960 con 0x7f0a580ffda0 2024-04-22T23:48:57.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.244+0000 7f0a3ffff700 1 -- 172.21.15.67:0/2390851033 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a48017630 con 0x7f0a580ffda0 2024-04-22T23:48:57.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.244+0000 7f0a5d175700 1 -- 172.21.15.67:0/2390851033 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a58196e50 con 0x7f0a580ffda0 2024-04-22T23:48:57.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.245+0000 7f0a5d175700 1 -- 172.21.15.67:0/2390851033 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a58108140 con 0x7f0a580ffda0 2024-04-22T23:48:57.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.245+0000 7f0a3ffff700 1 -- 172.21.15.67:0/2390851033 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0a4800f5d0 con 0x7f0a580ffda0 2024-04-22T23:48:57.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.246+0000 7f0a3ffff700 1 --2- 172.21.15.67:0/2390851033 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0a4006ecc0 0x7f0a40071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:57.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.246+0000 7f0a3ffff700 1 -- 172.21.15.67:0/2390851033 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f0a48090460 con 0x7f0a580ffda0 2024-04-22T23:48:57.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.246+0000 7f0a5659c700 1 --2- 172.21.15.67:0/2390851033 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0a4006ecc0 0x7f0a40071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:57.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.246+0000 7f0a5659c700 1 --2- 172.21.15.67:0/2390851033 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0a4006ecc0 0x7f0a40071180 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c005950 tx=0x7f0a4c0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:48:57.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.250+0000 7f0a3ffff700 1 -- 172.21.15.67:0/2390851033 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f0a4805cbd0 con 0x7f0a580ffda0 2024-04-22T23:48:57.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:57 smithi067 ceph-mon[112438]: from='client.35244 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:48:57.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:57 smithi067 ceph-mon[112438]: from='client.35248 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:48:57.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:57 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:48:57.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:57 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3207023861' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:48:57.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.485+0000 7f0a5d175700 1 -- 172.21.15.67:0/2390851033 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a58197c60 con 0x7f0a4006ecc0 2024-04-22T23:48:57.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.486+0000 7f0a3ffff700 1 -- 172.21.15.67:0/2390851033 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f0a58197c60 con 0x7f0a4006ecc0 2024-04-22T23:48:57.478 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:48:57.478 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:48:57.478 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:48:57.478 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:48:57.478 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:48:57.478 INFO:teuthology.orchestra.run.smithi067.stdout: "mds", 2024-04-22T23:48:57.478 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:48:57.478 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:48:57.478 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:48:57.478 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:48:57.478 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:48:57.478 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T23:48:57.478 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:48:57.478 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:48:57.479 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:48:57.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.488+0000 7f0a5d175700 1 -- 172.21.15.67:0/2390851033 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0a4006ecc0 msgr2=0x7f0a40071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:57.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.488+0000 7f0a5d175700 1 --2- 172.21.15.67:0/2390851033 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0a4006ecc0 0x7f0a40071180 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c005950 tx=0x7f0a4c0058e0 comp rx=0 tx=0).stop 2024-04-22T23:48:57.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.488+0000 7f0a5d175700 1 -- 172.21.15.67:0/2390851033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a580ffda0 msgr2=0x7f0a5819c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:57.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.488+0000 7f0a5d175700 1 --2- 172.21.15.67:0/2390851033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a580ffda0 0x7f0a5819c660 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f0a48006010 tx=0x7f0a4800be20 comp rx=0 tx=0).stop 2024-04-22T23:48:57.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.488+0000 7f0a5d175700 1 -- 172.21.15.67:0/2390851033 shutdown_connections 2024-04-22T23:48:57.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.488+0000 7f0a5d175700 1 --2- 172.21.15.67:0/2390851033 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0a4006ecc0 0x7f0a40071180 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:57.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.488+0000 7f0a5d175700 1 --2- 172.21.15.67:0/2390851033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a580ffda0 0x7f0a5819c660 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:57.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.488+0000 7f0a5d175700 1 --2- 172.21.15.67:0/2390851033 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0a58100750 0x7f0a5819cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:57.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.488+0000 7f0a5d175700 1 -- 172.21.15.67:0/2390851033 >> 172.21.15.67:0/2390851033 conn(0x7f0a580fb650 msgr2=0x7f0a580fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:48:57.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.489+0000 7f0a5d175700 1 -- 172.21.15.67:0/2390851033 shutdown_connections 2024-04-22T23:48:57.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.489+0000 7f0a5d175700 1 -- 172.21.15.67:0/2390851033 wait complete. 2024-04-22T23:48:57.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:57 smithi134 ceph-mon[85630]: from='client.35244 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:48:57.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:57 smithi134 ceph-mon[85630]: from='client.35248 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:48:57.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:57 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:48:57.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:57 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3207023861' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:48:57.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.595+0000 7fce0f0c4700 1 -- 172.21.15.67:0/2938446498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce08106db0 msgr2=0x7fce08107190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:57.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.595+0000 7fce0f0c4700 1 --2- 172.21.15.67:0/2938446498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce08106db0 0x7fce08107190 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc009910 tx=0x7fcdfc009c20 comp rx=0 tx=0).stop 2024-04-22T23:48:57.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.595+0000 7fce0f0c4700 1 -- 172.21.15.67:0/2938446498 shutdown_connections 2024-04-22T23:48:57.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.595+0000 7fce0f0c4700 1 --2- 172.21.15.67:0/2938446498 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce080ff410 0x7fce080ff890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:57.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.595+0000 7fce0f0c4700 1 --2- 172.21.15.67:0/2938446498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce08106db0 0x7fce08107190 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:57.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.595+0000 7fce0f0c4700 1 -- 172.21.15.67:0/2938446498 >> 172.21.15.67:0/2938446498 conn(0x7fce08074e30 msgr2=0x7fce08075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:48:57.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.596+0000 7fce0f0c4700 1 -- 172.21.15.67:0/2938446498 shutdown_connections 2024-04-22T23:48:57.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.596+0000 7fce0f0c4700 1 -- 172.21.15.67:0/2938446498 wait complete. 2024-04-22T23:48:57.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.596+0000 7fce0f0c4700 1 Processor -- start 2024-04-22T23:48:57.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.596+0000 7fce0f0c4700 1 -- start start 2024-04-22T23:48:57.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.596+0000 7fce0f0c4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce080ff410 0x7fce0819c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:57.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.596+0000 7fce0f0c4700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce08106db0 0x7fce0819cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:57.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.596+0000 7fce0f0c4700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce0819d370 con 0x7fce080ff410 2024-04-22T23:48:57.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.596+0000 7fce0f0c4700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce08196820 con 0x7fce08106db0 2024-04-22T23:48:57.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.597+0000 7fce0ce60700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce080ff410 0x7fce0819c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:57.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.597+0000 7fce0ce60700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce080ff410 0x7fce0819c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45086/0 (socket says 172.21.15.67:45086) 2024-04-22T23:48:57.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.597+0000 7fce0ce60700 1 -- 172.21.15.67:0/1207956378 learned_addr learned my addr 172.21.15.67:0/1207956378 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:48:57.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.597+0000 7fce07fff700 1 --2- 172.21.15.67:0/1207956378 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce08106db0 0x7fce0819cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:57.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.597+0000 7fce0ce60700 1 -- 172.21.15.67:0/1207956378 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce08106db0 msgr2=0x7fce0819cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:57.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.597+0000 7fce0ce60700 1 --2- 172.21.15.67:0/1207956378 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce08106db0 0x7fce0819cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:57.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.597+0000 7fce0ce60700 1 -- 172.21.15.67:0/1207956378 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdfc0095f0 con 0x7fce080ff410 2024-04-22T23:48:57.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.597+0000 7fce07fff700 1 --2- 172.21.15.67:0/1207956378 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce08106db0 0x7fce0819cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:48:57.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.597+0000 7fce0ce60700 1 --2- 172.21.15.67:0/1207956378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce080ff410 0x7fce0819c7a0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc006010 tx=0x7fcdfc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:48:57.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.597+0000 7fce05ffb700 1 -- 172.21.15.67:0/1207956378 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdfc01d070 con 0x7fce080ff410 2024-04-22T23:48:57.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.597+0000 7fce05ffb700 1 -- 172.21.15.67:0/1207956378 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcdfc004fb0 con 0x7fce080ff410 2024-04-22T23:48:57.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.597+0000 7fce0f0c4700 1 -- 172.21.15.67:0/1207956378 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce08196aa0 con 0x7fce080ff410 2024-04-22T23:48:57.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.597+0000 7fce05ffb700 1 -- 172.21.15.67:0/1207956378 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdfc022ea0 con 0x7fce080ff410 2024-04-22T23:48:57.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.597+0000 7fce0f0c4700 1 -- 172.21.15.67:0/1207956378 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce08196f90 con 0x7fce080ff410 2024-04-22T23:48:57.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.598+0000 7fce0f0c4700 1 -- 172.21.15.67:0/1207956378 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce08109300 con 0x7fce080ff410 2024-04-22T23:48:57.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.599+0000 7fce05ffb700 1 -- 172.21.15.67:0/1207956378 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcdfc022470 con 0x7fce080ff410 2024-04-22T23:48:57.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.599+0000 7fce05ffb700 1 --2- 172.21.15.67:0/1207956378 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcdf806ecc0 0x7fcdf8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:48:57.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.600+0000 7fce05ffb700 1 -- 172.21.15.67:0/1207956378 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fcdfc0904c0 con 0x7fce080ff410 2024-04-22T23:48:57.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.600+0000 7fce07fff700 1 --2- 172.21.15.67:0/1207956378 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcdf806ecc0 0x7fcdf8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:48:57.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.600+0000 7fce07fff700 1 --2- 172.21.15.67:0/1207956378 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcdf806ecc0 0x7fcdf8071180 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fce08075490 tx=0x7fcdf4005f50 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:48:57.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.604+0000 7fce05ffb700 1 -- 172.21.15.67:0/1207956378 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fcdfc05cd60 con 0x7fce080ff410 2024-04-22T23:48:57.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.912+0000 7fce0f0c4700 1 -- 172.21.15.67:0/1207956378 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fce0804f2e0 con 0x7fce080ff410 2024-04-22T23:48:57.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.913+0000 7fce05ffb700 1 -- 172.21.15.67:0/1207956378 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4505 (secure 0 0 0) 0x7fcdfc022720 con 0x7fce080ff410 2024-04-22T23:48:57.904 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:48:57.904 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T23:48:57.904 INFO:teuthology.orchestra.run.smithi067.stdout: host smithi134 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-22T23:48:57.905 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T23:48:57.905 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:48:57.905 INFO:teuthology.orchestra.run.smithi067.stdout:Traceback (most recent call last): 2024-04-22T23:48:57.905 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T23:48:57.905 INFO:teuthology.orchestra.run.smithi067.stdout: return _run_code(code, main_globals, None, 2024-04-22T23:48:57.905 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T23:48:57.905 INFO:teuthology.orchestra.run.smithi067.stdout: exec(code, run_globals) 2024-04-22T23:48:57.905 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5642, in 2024-04-22T23:48:57.905 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5630, in main 2024-04-22T23:48:57.905 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 408, in _infer_config 2024-04-22T23:48:57.905 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 323, in _infer_fsid 2024-04-22T23:48:57.905 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 436, in _infer_image 2024-04-22T23:48:57.905 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 310, in _validate_fsid 2024-04-22T23:48:57.905 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 3380, in command_ceph_volume 2024-04-22T23:48:57.906 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T23:48:57.906 INFO:teuthology.orchestra.run.smithi067.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:48:57.906 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:48:57.906 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:48:57.906 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:48:57.906 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:48:57.906 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:48:57.906 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:48:57.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.915+0000 7fce0f0c4700 1 -- 172.21.15.67:0/1207956378 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcdf806ecc0 msgr2=0x7fcdf8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:57.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.915+0000 7fce0f0c4700 1 --2- 172.21.15.67:0/1207956378 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcdf806ecc0 0x7fcdf8071180 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fce08075490 tx=0x7fcdf4005f50 comp rx=0 tx=0).stop 2024-04-22T23:48:57.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.915+0000 7fce0f0c4700 1 -- 172.21.15.67:0/1207956378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce080ff410 msgr2=0x7fce0819c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:48:57.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.915+0000 7fce0f0c4700 1 --2- 172.21.15.67:0/1207956378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce080ff410 0x7fce0819c7a0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc006010 tx=0x7fcdfc00be20 comp rx=0 tx=0).stop 2024-04-22T23:48:57.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.915+0000 7fce0f0c4700 1 -- 172.21.15.67:0/1207956378 shutdown_connections 2024-04-22T23:48:57.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.915+0000 7fce0f0c4700 1 --2- 172.21.15.67:0/1207956378 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcdf806ecc0 0x7fcdf8071180 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:57.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.915+0000 7fce0f0c4700 1 --2- 172.21.15.67:0/1207956378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce080ff410 0x7fce0819c7a0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:57.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.915+0000 7fce0f0c4700 1 --2- 172.21.15.67:0/1207956378 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce08106db0 0x7fce0819cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:48:57.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.915+0000 7fce0f0c4700 1 -- 172.21.15.67:0/1207956378 >> 172.21.15.67:0/1207956378 conn(0x7fce08074e30 msgr2=0x7fce080fd460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:48:57.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.915+0000 7fce0f0c4700 1 -- 172.21.15.67:0/1207956378 shutdown_connections 2024-04-22T23:48:57.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:48:57.915+0000 7fce0f0c4700 1 -- 172.21.15.67:0/1207956378 wait complete. 2024-04-22T23:48:58.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:58 smithi067 ceph-mon[112438]: from='client.35252 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:48:58.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:58 smithi067 ceph-mon[112438]: pgmap v679: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:58.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:58 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/4029148074' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:48:58.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:58 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1207956378' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:48:58.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:58 smithi134 ceph-mon[85630]: from='client.35252 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:48:58.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:58 smithi134 ceph-mon[85630]: pgmap v679: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:48:58.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:58 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/4029148074' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:48:58.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:58 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1207956378' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:48:59.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:48:59 smithi067 ceph-mon[112438]: from='client.35264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:48:59.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:48:59 smithi134 ceph-mon[85630]: from='client.35264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:49:00.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:00 smithi067 ceph-mon[112438]: pgmap v680: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:00.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:00 smithi134 ceph-mon[85630]: pgmap v680: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:02.278 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:02 smithi134 ceph-mon[85630]: pgmap v681: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:02.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:02 smithi067 ceph-mon[112438]: pgmap v681: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:04.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:04 smithi067 ceph-mon[112438]: pgmap v682: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:04.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:04 smithi134 ceph-mon[85630]: pgmap v682: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:06.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:06 smithi067 ceph-mon[112438]: pgmap v683: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:06.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:06 smithi134 ceph-mon[85630]: pgmap v683: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:08.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:08 smithi067 ceph-mon[112438]: pgmap v684: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:08.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:08 smithi134 ceph-mon[85630]: pgmap v684: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:10.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:10 smithi067 ceph-mon[112438]: pgmap v685: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:10.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:10 smithi134 ceph-mon[85630]: pgmap v685: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:12.386 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:12 smithi134 ceph-mon[85630]: pgmap v686: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:12.386 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:12 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:49:12.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:12 smithi067 ceph-mon[112438]: pgmap v686: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:12.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:12 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:49:14.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:14 smithi134 ceph-mon[85630]: pgmap v687: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:14.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:14 smithi067 ceph-mon[112438]: pgmap v687: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:15.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:15 smithi134 ceph-mon[85630]: pgmap v688: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:15.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:15 smithi067 ceph-mon[112438]: pgmap v688: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:18.239 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:17 smithi067 ceph-mon[112438]: pgmap v689: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:18.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:17 smithi134 ceph-mon[85630]: pgmap v689: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:20.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:19 smithi067 ceph-mon[112438]: pgmap v690: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:20.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:19 smithi134 ceph-mon[85630]: pgmap v690: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:22.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:22 smithi067 ceph-mon[112438]: pgmap v691: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:22.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:22 smithi134 ceph-mon[85630]: pgmap v691: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:23.985 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:23 smithi134 ceph-mon[85630]: pgmap v692: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:24.035 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:23 smithi067 ceph-mon[112438]: pgmap v692: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:26.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:25 smithi067 ceph-mon[112438]: pgmap v693: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:26.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:25 smithi134 ceph-mon[85630]: pgmap v693: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:27.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:27 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:49:27.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:27 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:49:28.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.039+0000 7f2a31af9700 1 -- 172.21.15.67:0/1945174692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a2c101850 msgr2=0x7f2a2c1058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:49:28.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.039+0000 7f2a31af9700 1 --2- 172.21.15.67:0/1945174692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a2c101850 0x7f2a2c1058a0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f2a20009910 tx=0x7f2a20009c20 comp rx=0 tx=0).stop 2024-04-22T23:49:28.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.039+0000 7f2a31af9700 1 -- 172.21.15.67:0/1945174692 shutdown_connections 2024-04-22T23:49:28.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.039+0000 7f2a31af9700 1 --2- 172.21.15.67:0/1945174692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a2c101850 0x7f2a2c1058a0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:28.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.039+0000 7f2a31af9700 1 --2- 172.21.15.67:0/1945174692 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a2c100ea0 0x7f2a2c101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:28.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.039+0000 7f2a31af9700 1 -- 172.21.15.67:0/1945174692 >> 172.21.15.67:0/1945174692 conn(0x7f2a2c0fc750 msgr2=0x7f2a2c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:49:28.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.039+0000 7f2a31af9700 1 -- 172.21.15.67:0/1945174692 shutdown_connections 2024-04-22T23:49:28.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.039+0000 7f2a31af9700 1 -- 172.21.15.67:0/1945174692 wait complete. 2024-04-22T23:49:28.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.039+0000 7f2a31af9700 1 Processor -- start 2024-04-22T23:49:28.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.040+0000 7f2a31af9700 1 -- start start 2024-04-22T23:49:28.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.040+0000 7f2a31af9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a2c100ea0 0x7f2a2c19c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:49:28.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.040+0000 7f2a31af9700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a2c101850 0x7f2a2c19cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:49:28.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.040+0000 7f2a31af9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2c19d2d0 con 0x7f2a2c100ea0 2024-04-22T23:49:28.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.040+0000 7f2a31af9700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2c196780 con 0x7f2a2c101850 2024-04-22T23:49:28.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.040+0000 7f2a2b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a2c100ea0 0x7f2a2c19c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:49:28.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.040+0000 7f2a2b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a2c100ea0 0x7f2a2c19c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36174/0 (socket says 172.21.15.67:36174) 2024-04-22T23:49:28.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.040+0000 7f2a2b7fe700 1 -- 172.21.15.67:0/779695223 learned_addr learned my addr 172.21.15.67:0/779695223 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:49:28.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.040+0000 7f2a2affd700 1 --2- 172.21.15.67:0/779695223 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a2c101850 0x7f2a2c19cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:49:28.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.041+0000 7f2a2b7fe700 1 -- 172.21.15.67:0/779695223 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a2c101850 msgr2=0x7f2a2c19cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:49:28.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.041+0000 7f2a2b7fe700 1 --2- 172.21.15.67:0/779695223 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a2c101850 0x7f2a2c19cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:28.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.041+0000 7f2a2b7fe700 1 -- 172.21.15.67:0/779695223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a200095f0 con 0x7f2a2c100ea0 2024-04-22T23:49:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.041+0000 7f2a2affd700 1 --2- 172.21.15.67:0/779695223 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a2c101850 0x7f2a2c19cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:49:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.041+0000 7f2a2b7fe700 1 --2- 172.21.15.67:0/779695223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a2c100ea0 0x7f2a2c19c700 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c00d8d0 tx=0x7f2a1c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:49:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.041+0000 7f2a28ff9700 1 -- 172.21.15.67:0/779695223 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a1c009940 con 0x7f2a2c100ea0 2024-04-22T23:49:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.041+0000 7f2a28ff9700 1 -- 172.21.15.67:0/779695223 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2a1c010460 con 0x7f2a2c100ea0 2024-04-22T23:49:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.041+0000 7f2a31af9700 1 -- 172.21.15.67:0/779695223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a2c196a60 con 0x7f2a2c100ea0 2024-04-22T23:49:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.041+0000 7f2a28ff9700 1 -- 172.21.15.67:0/779695223 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a1c00f5f0 con 0x7f2a2c100ea0 2024-04-22T23:49:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.041+0000 7f2a31af9700 1 -- 172.21.15.67:0/779695223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a2c196fb0 con 0x7f2a2c100ea0 2024-04-22T23:49:28.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.042+0000 7f2a31af9700 1 -- 172.21.15.67:0/779695223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a2c1092a0 con 0x7f2a2c100ea0 2024-04-22T23:49:28.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.043+0000 7f2a28ff9700 1 -- 172.21.15.67:0/779695223 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2a1c0105d0 con 0x7f2a2c100ea0 2024-04-22T23:49:28.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.043+0000 7f2a28ff9700 1 --2- 172.21.15.67:0/779695223 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2a1406ecc0 0x7f2a14071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:49:28.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.043+0000 7f2a28ff9700 1 -- 172.21.15.67:0/779695223 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f2a1c08eff0 con 0x7f2a2c100ea0 2024-04-22T23:49:28.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.043+0000 7f2a2affd700 1 --2- 172.21.15.67:0/779695223 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2a1406ecc0 0x7f2a14071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:49:28.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.044+0000 7f2a2affd700 1 --2- 172.21.15.67:0/779695223 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2a1406ecc0 0x7f2a14071180 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c197bf0 tx=0x7f2a2001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:49:28.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.049+0000 7f2a28ff9700 1 -- 172.21.15.67:0/779695223 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f2a1c05b760 con 0x7f2a2c100ea0 2024-04-22T23:49:28.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.280+0000 7f2a31af9700 1 -- 172.21.15.67:0/779695223 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2a2c197d50 con 0x7f2a1406ecc0 2024-04-22T23:49:28.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:28 smithi067 ceph-mon[112438]: pgmap v694: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:28.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.282+0000 7f2a28ff9700 1 -- 172.21.15.67:0/779695223 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f2a2c197d50 con 0x7f2a1406ecc0 2024-04-22T23:49:28.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.284+0000 7f2a31af9700 1 -- 172.21.15.67:0/779695223 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2a1406ecc0 msgr2=0x7f2a14071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:49:28.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.284+0000 7f2a31af9700 1 --2- 172.21.15.67:0/779695223 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2a1406ecc0 0x7f2a14071180 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c197bf0 tx=0x7f2a2001a040 comp rx=0 tx=0).stop 2024-04-22T23:49:28.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.284+0000 7f2a31af9700 1 -- 172.21.15.67:0/779695223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a2c100ea0 msgr2=0x7f2a2c19c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:49:28.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.284+0000 7f2a31af9700 1 --2- 172.21.15.67:0/779695223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a2c100ea0 0x7f2a2c19c700 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c00d8d0 tx=0x7f2a1c00dc90 comp rx=0 tx=0).stop 2024-04-22T23:49:28.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.284+0000 7f2a31af9700 1 -- 172.21.15.67:0/779695223 shutdown_connections 2024-04-22T23:49:28.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.284+0000 7f2a31af9700 1 --2- 172.21.15.67:0/779695223 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2a1406ecc0 0x7f2a14071180 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:28.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.284+0000 7f2a31af9700 1 --2- 172.21.15.67:0/779695223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a2c100ea0 0x7f2a2c19c700 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:28.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.284+0000 7f2a31af9700 1 --2- 172.21.15.67:0/779695223 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2a2c101850 0x7f2a2c19cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:28.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.284+0000 7f2a31af9700 1 -- 172.21.15.67:0/779695223 >> 172.21.15.67:0/779695223 conn(0x7f2a2c0fc750 msgr2=0x7f2a2c0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:49:28.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.284+0000 7f2a31af9700 1 -- 172.21.15.67:0/779695223 shutdown_connections 2024-04-22T23:49:28.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.284+0000 7f2a31af9700 1 -- 172.21.15.67:0/779695223 wait complete. 2024-04-22T23:49:28.286 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:49:28.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.393+0000 7f39e20c7700 1 -- 172.21.15.67:0/220755696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39dc100ea0 msgr2=0x7f39dc101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:49:28.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.393+0000 7f39e20c7700 1 --2- 172.21.15.67:0/220755696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39dc100ea0 0x7f39dc101280 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f39cc009910 tx=0x7f39cc009c20 comp rx=0 tx=0).stop 2024-04-22T23:49:28.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.393+0000 7f39e20c7700 1 -- 172.21.15.67:0/220755696 shutdown_connections 2024-04-22T23:49:28.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.393+0000 7f39e20c7700 1 --2- 172.21.15.67:0/220755696 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39dc101850 0x7f39dc1058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:28.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.393+0000 7f39e20c7700 1 --2- 172.21.15.67:0/220755696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39dc100ea0 0x7f39dc101280 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:28.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.393+0000 7f39e20c7700 1 -- 172.21.15.67:0/220755696 >> 172.21.15.67:0/220755696 conn(0x7f39dc0fc750 msgr2=0x7f39dc0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:49:28.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.393+0000 7f39e20c7700 1 -- 172.21.15.67:0/220755696 shutdown_connections 2024-04-22T23:49:28.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.393+0000 7f39e20c7700 1 -- 172.21.15.67:0/220755696 wait complete. 2024-04-22T23:49:28.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.393+0000 7f39e20c7700 1 Processor -- start 2024-04-22T23:49:28.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.394+0000 7f39e20c7700 1 -- start start 2024-04-22T23:49:28.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.394+0000 7f39e20c7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39dc100ea0 0x7f39dc196990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:49:28.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.394+0000 7f39e20c7700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39dc101850 0x7f39dc196ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:49:28.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.394+0000 7f39e20c7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39dc197560 con 0x7f39dc100ea0 2024-04-22T23:49:28.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.394+0000 7f39e20c7700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39dc19b370 con 0x7f39dc101850 2024-04-22T23:49:28.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.394+0000 7f39db7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39dc100ea0 0x7f39dc196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:49:28.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.394+0000 7f39db7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39dc100ea0 0x7f39dc196990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36194/0 (socket says 172.21.15.67:36194) 2024-04-22T23:49:28.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.394+0000 7f39db7fe700 1 -- 172.21.15.67:0/625472149 learned_addr learned my addr 172.21.15.67:0/625472149 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:49:28.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.394+0000 7f39daffd700 1 --2- 172.21.15.67:0/625472149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39dc101850 0x7f39dc196ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:49:28.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.394+0000 7f39db7fe700 1 -- 172.21.15.67:0/625472149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39dc101850 msgr2=0x7f39dc196ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:49:28.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.394+0000 7f39db7fe700 1 --2- 172.21.15.67:0/625472149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39dc101850 0x7f39dc196ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:28.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.394+0000 7f39db7fe700 1 -- 172.21.15.67:0/625472149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39cc0095f0 con 0x7f39dc100ea0 2024-04-22T23:49:28.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.394+0000 7f39daffd700 1 --2- 172.21.15.67:0/625472149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39dc101850 0x7f39dc196ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:49:28.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.394+0000 7f39db7fe700 1 --2- 172.21.15.67:0/625472149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39dc100ea0 0x7f39dc196990 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f39cc006010 tx=0x7f39cc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:49:28.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.395+0000 7f39d8ff9700 1 -- 172.21.15.67:0/625472149 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39cc01d070 con 0x7f39dc100ea0 2024-04-22T23:49:28.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.395+0000 7f39d8ff9700 1 -- 172.21.15.67:0/625472149 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f39cc004fb0 con 0x7f39dc100ea0 2024-04-22T23:49:28.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.395+0000 7f39d8ff9700 1 -- 172.21.15.67:0/625472149 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39cc022ea0 con 0x7f39dc100ea0 2024-04-22T23:49:28.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.395+0000 7f39e20c7700 1 -- 172.21.15.67:0/625472149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39dc19b5f0 con 0x7f39dc100ea0 2024-04-22T23:49:28.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.395+0000 7f39e20c7700 1 -- 172.21.15.67:0/625472149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39dc19bae0 con 0x7f39dc100ea0 2024-04-22T23:49:28.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.396+0000 7f39e20c7700 1 -- 172.21.15.67:0/625472149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39dc109240 con 0x7f39dc100ea0 2024-04-22T23:49:28.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.396+0000 7f39d8ff9700 1 -- 172.21.15.67:0/625472149 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f39cc022470 con 0x7f39dc100ea0 2024-04-22T23:49:28.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.397+0000 7f39d8ff9700 1 --2- 172.21.15.67:0/625472149 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f39c406ecc0 0x7f39c4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:49:28.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.397+0000 7f39d8ff9700 1 -- 172.21.15.67:0/625472149 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f39cc0904c0 con 0x7f39dc100ea0 2024-04-22T23:49:28.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.397+0000 7f39daffd700 1 --2- 172.21.15.67:0/625472149 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f39c406ecc0 0x7f39c4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:49:28.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.397+0000 7f39daffd700 1 --2- 172.21.15.67:0/625472149 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f39c406ecc0 0x7f39c4071180 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f39dc197fb0 tx=0x7f39d0008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:49:28.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.401+0000 7f39d8ff9700 1 -- 172.21.15.67:0/625472149 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f39cc05cd60 con 0x7f39dc100ea0 2024-04-22T23:49:28.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:28 smithi134 ceph-mon[85630]: pgmap v694: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:28.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.632+0000 7f39e20c7700 1 -- 172.21.15.67:0/625472149 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f39dc197e00 con 0x7f39c406ecc0 2024-04-22T23:49:28.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.633+0000 7f39d8ff9700 1 -- 172.21.15.67:0/625472149 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f39dc197e00 con 0x7f39c406ecc0 2024-04-22T23:49:28.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.635+0000 7f39e20c7700 1 -- 172.21.15.67:0/625472149 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f39c406ecc0 msgr2=0x7f39c4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:49:28.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.635+0000 7f39e20c7700 1 --2- 172.21.15.67:0/625472149 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f39c406ecc0 0x7f39c4071180 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f39dc197fb0 tx=0x7f39d0008040 comp rx=0 tx=0).stop 2024-04-22T23:49:28.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.635+0000 7f39e20c7700 1 -- 172.21.15.67:0/625472149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39dc100ea0 msgr2=0x7f39dc196990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:49:28.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.635+0000 7f39e20c7700 1 --2- 172.21.15.67:0/625472149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39dc100ea0 0x7f39dc196990 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f39cc006010 tx=0x7f39cc00be20 comp rx=0 tx=0).stop 2024-04-22T23:49:28.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.635+0000 7f39e20c7700 1 -- 172.21.15.67:0/625472149 shutdown_connections 2024-04-22T23:49:28.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.635+0000 7f39e20c7700 1 --2- 172.21.15.67:0/625472149 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f39c406ecc0 0x7f39c4071180 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:28.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.635+0000 7f39e20c7700 1 --2- 172.21.15.67:0/625472149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39dc100ea0 0x7f39dc196990 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:28.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.635+0000 7f39e20c7700 1 --2- 172.21.15.67:0/625472149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39dc101850 0x7f39dc196ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:28.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.635+0000 7f39e20c7700 1 -- 172.21.15.67:0/625472149 >> 172.21.15.67:0/625472149 conn(0x7f39dc0fc750 msgr2=0x7f39dc0fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:49:28.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.635+0000 7f39e20c7700 1 -- 172.21.15.67:0/625472149 shutdown_connections 2024-04-22T23:49:28.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.635+0000 7f39e20c7700 1 -- 172.21.15.67:0/625472149 wait complete. 2024-04-22T23:49:28.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.743+0000 7f1ffd86e700 1 -- 172.21.15.67:0/2288640109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ff8105bd0 msgr2=0x7f1ff8105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:49:28.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.743+0000 7f1ffd86e700 1 --2- 172.21.15.67:0/2288640109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ff8105bd0 0x7f1ff8105fb0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f1ff000b0a0 tx=0x7f1ff000b3b0 comp rx=0 tx=0).stop 2024-04-22T23:49:28.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.743+0000 7f1ffd86e700 1 -- 172.21.15.67:0/2288640109 shutdown_connections 2024-04-22T23:49:28.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.743+0000 7f1ffd86e700 1 --2- 172.21.15.67:0/2288640109 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ff80ffc20 0x7f1ff8100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:28.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.743+0000 7f1ffd86e700 1 --2- 172.21.15.67:0/2288640109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ff8105bd0 0x7f1ff8105fb0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:28.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.743+0000 7f1ffd86e700 1 -- 172.21.15.67:0/2288640109 >> 172.21.15.67:0/2288640109 conn(0x7f1ff8075670 msgr2=0x7f1ff8075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:49:28.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.743+0000 7f1ffd86e700 1 -- 172.21.15.67:0/2288640109 shutdown_connections 2024-04-22T23:49:28.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.743+0000 7f1ffd86e700 1 -- 172.21.15.67:0/2288640109 wait complete. 2024-04-22T23:49:28.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.743+0000 7f1ffd86e700 1 Processor -- start 2024-04-22T23:49:28.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.744+0000 7f1ffd86e700 1 -- start start 2024-04-22T23:49:28.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.744+0000 7f1ffd86e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ff80ffc20 0x7f1ff8101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:49:28.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.744+0000 7f1ffd86e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ff8105bd0 0x7f1ff8101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:49:28.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.744+0000 7f1ffd86e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff81022c0 con 0x7f1ff80ffc20 2024-04-22T23:49:28.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.744+0000 7f1ffd86e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff8102400 con 0x7f1ff8105bd0 2024-04-22T23:49:28.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.744+0000 7f1ff6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ff80ffc20 0x7f1ff8101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:49:28.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.744+0000 7f1ff6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ff80ffc20 0x7f1ff8101840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60654/0 (socket says 172.21.15.67:60654) 2024-04-22T23:49:28.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.744+0000 7f1ff6ffd700 1 -- 172.21.15.67:0/509128362 learned_addr learned my addr 172.21.15.67:0/509128362 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:49:28.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.744+0000 7f1ff67fc700 1 --2- 172.21.15.67:0/509128362 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ff8105bd0 0x7f1ff8101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:49:28.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.744+0000 7f1ff6ffd700 1 -- 172.21.15.67:0/509128362 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ff8105bd0 msgr2=0x7f1ff8101d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:49:28.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.744+0000 7f1ff6ffd700 1 --2- 172.21.15.67:0/509128362 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ff8105bd0 0x7f1ff8101d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:28.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.744+0000 7f1ff6ffd700 1 -- 172.21.15.67:0/509128362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff0009d00 con 0x7f1ff80ffc20 2024-04-22T23:49:28.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.744+0000 7f1ff6ffd700 1 --2- 172.21.15.67:0/509128362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ff80ffc20 0x7f1ff8101840 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0000f80 tx=0x7f1ff000cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:49:28.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.745+0000 7f1ffc86c700 1 -- 172.21.15.67:0/509128362 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ff000e040 con 0x7f1ff80ffc20 2024-04-22T23:49:28.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.745+0000 7f1ffc86c700 1 -- 172.21.15.67:0/509128362 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ff0003c10 con 0x7f1ff80ffc20 2024-04-22T23:49:28.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.745+0000 7f1ffd86e700 1 -- 172.21.15.67:0/509128362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ff8102680 con 0x7f1ff80ffc20 2024-04-22T23:49:28.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.745+0000 7f1ffc86c700 1 -- 172.21.15.67:0/509128362 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ff001d6b0 con 0x7f1ff80ffc20 2024-04-22T23:49:28.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.745+0000 7f1ffd86e700 1 -- 172.21.15.67:0/509128362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ff81a2da0 con 0x7f1ff80ffc20 2024-04-22T23:49:28.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.746+0000 7f1ffd86e700 1 -- 172.21.15.67:0/509128362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ff804f2e0 con 0x7f1ff80ffc20 2024-04-22T23:49:28.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.746+0000 7f1ffc86c700 1 -- 172.21.15.67:0/509128362 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1ff0004040 con 0x7f1ff80ffc20 2024-04-22T23:49:28.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.746+0000 7f1ffc86c700 1 --2- 172.21.15.67:0/509128362 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1fe406ecc0 0x7f1fe4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:49:28.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.747+0000 7f1ffc86c700 1 -- 172.21.15.67:0/509128362 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f1ff0090760 con 0x7f1ff80ffc20 2024-04-22T23:49:28.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.747+0000 7f1ff67fc700 1 --2- 172.21.15.67:0/509128362 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1fe406ecc0 0x7f1fe4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:49:28.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.747+0000 7f1ff67fc700 1 --2- 172.21.15.67:0/509128362 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1fe406ecc0 0x7f1fe4071180 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8103030 tx=0x7f1fe000b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:49:28.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.751+0000 7f1ffc86c700 1 -- 172.21.15.67:0/509128362 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1ff00593a0 con 0x7f1ff80ffc20 2024-04-22T23:49:28.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.969+0000 7f1ffd86e700 1 -- 172.21.15.67:0/509128362 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ff8068d70 con 0x7f1fe406ecc0 2024-04-22T23:49:28.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.980+0000 7f1ffc86c700 1 -- 172.21.15.67:0/509128362 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f1ff8068d70 con 0x7f1fe406ecc0 2024-04-22T23:49:28.972 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:49:28.972 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (23m) 58s ago 34m 24.4M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:49:28.972 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (34m) 58s ago 34m 10.0M - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:49:28.972 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (33m) 60s ago 33m 10.0M - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:49:28.972 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (20m) 58s ago 34m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:49:28.972 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (20m) 60s ago 33m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:49:28.972 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (22m) 58s ago 34m 81.2M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:49:28.972 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (16m) 58s ago 29m 19.4M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:49:28.972 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (16m) 58s ago 28m 20.3M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:49:28.972 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (16m) 60s ago 28m 20.1M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:49:28.972 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (64s) 60s ago 28m 14.0M - 19.0.0-3252-g43be0201 a0434699c1d7 46d3d7e86fee 2024-04-22T23:49:28.972 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (24m) 58s ago 35m 514M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:49:28.972 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (24m) 60s ago 33m 459M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:49:28.973 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (21m) 58s ago 36m 74.2M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:49:28.973 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (21m) 60s ago 33m 54.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:49:28.973 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (23m) 58s ago 34m 15.2M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:49:28.973 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (23m) 60s ago 33m 16.0M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:49:28.973 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (20m) 58s ago 32m 102M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:49:28.973 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (19m) 58s ago 32m 128M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:49:28.973 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (18m) 58s ago 31m 83.8M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:49:28.973 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (18m) 60s ago 31m 101M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:49:28.973 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (17m) 60s ago 31m 86.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:49:28.973 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (17m) 60s ago 30m 81.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:49:28.973 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (23m) 58s ago 33m 78.7M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:49:28.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.982+0000 7f1ffd86e700 1 -- 172.21.15.67:0/509128362 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1fe406ecc0 msgr2=0x7f1fe4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:49:28.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.982+0000 7f1ffd86e700 1 --2- 172.21.15.67:0/509128362 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1fe406ecc0 0x7f1fe4071180 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8103030 tx=0x7f1fe000b410 comp rx=0 tx=0).stop 2024-04-22T23:49:28.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.982+0000 7f1ffd86e700 1 -- 172.21.15.67:0/509128362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ff80ffc20 msgr2=0x7f1ff8101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:49:28.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.982+0000 7f1ffd86e700 1 --2- 172.21.15.67:0/509128362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ff80ffc20 0x7f1ff8101840 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0000f80 tx=0x7f1ff000cd20 comp rx=0 tx=0).stop 2024-04-22T23:49:28.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.982+0000 7f1ffd86e700 1 -- 172.21.15.67:0/509128362 shutdown_connections 2024-04-22T23:49:28.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.982+0000 7f1ffd86e700 1 --2- 172.21.15.67:0/509128362 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1fe406ecc0 0x7f1fe4071180 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:28.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.982+0000 7f1ffd86e700 1 --2- 172.21.15.67:0/509128362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ff80ffc20 0x7f1ff8101840 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:28.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.982+0000 7f1ffd86e700 1 --2- 172.21.15.67:0/509128362 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ff8105bd0 0x7f1ff8101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:28.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.982+0000 7f1ffd86e700 1 -- 172.21.15.67:0/509128362 >> 172.21.15.67:0/509128362 conn(0x7f1ff8075670 msgr2=0x7f1ff8068660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:49:28.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.982+0000 7f1ffd86e700 1 -- 172.21.15.67:0/509128362 shutdown_connections 2024-04-22T23:49:28.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:28.982+0000 7f1ffd86e700 1 -- 172.21.15.67:0/509128362 wait complete. 2024-04-22T23:49:29.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.088+0000 7f38dbb63700 1 -- 172.21.15.67:0/3871700815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38d4101850 msgr2=0x7f38d41058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:49:29.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.088+0000 7f38dbb63700 1 --2- 172.21.15.67:0/3871700815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38d4101850 0x7f38d41058a0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f38d0009910 tx=0x7f38d0009c20 comp rx=0 tx=0).stop 2024-04-22T23:49:29.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.088+0000 7f38dbb63700 1 -- 172.21.15.67:0/3871700815 shutdown_connections 2024-04-22T23:49:29.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.088+0000 7f38dbb63700 1 --2- 172.21.15.67:0/3871700815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38d4101850 0x7f38d41058a0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:29.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.088+0000 7f38dbb63700 1 --2- 172.21.15.67:0/3871700815 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f38d4100ea0 0x7f38d4101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:29.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.088+0000 7f38dbb63700 1 -- 172.21.15.67:0/3871700815 >> 172.21.15.67:0/3871700815 conn(0x7f38d40fc750 msgr2=0x7f38d40feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:49:29.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.088+0000 7f38dbb63700 1 -- 172.21.15.67:0/3871700815 shutdown_connections 2024-04-22T23:49:29.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.088+0000 7f38dbb63700 1 -- 172.21.15.67:0/3871700815 wait complete. 2024-04-22T23:49:29.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.089+0000 7f38dbb63700 1 Processor -- start 2024-04-22T23:49:29.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.089+0000 7f38dbb63700 1 -- start start 2024-04-22T23:49:29.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.089+0000 7f38dbb63700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38d4100ea0 0x7f38d4073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:49:29.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.089+0000 7f38dbb63700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f38d4101850 0x7f38d40735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:49:29.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.089+0000 7f38dbb63700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d4073b10 con 0x7f38d4100ea0 2024-04-22T23:49:29.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.089+0000 7f38dbb63700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d4073c50 con 0x7f38d4101850 2024-04-22T23:49:29.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.089+0000 7f38d98ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38d4100ea0 0x7f38d4073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:49:29.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.089+0000 7f38d98ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38d4100ea0 0x7f38d4073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60670/0 (socket says 172.21.15.67:60670) 2024-04-22T23:49:29.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.089+0000 7f38d98ff700 1 -- 172.21.15.67:0/3112110103 learned_addr learned my addr 172.21.15.67:0/3112110103 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:49:29.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.089+0000 7f38d90fe700 1 --2- 172.21.15.67:0/3112110103 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f38d4101850 0x7f38d40735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:49:29.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.090+0000 7f38d98ff700 1 -- 172.21.15.67:0/3112110103 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f38d4101850 msgr2=0x7f38d40735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:49:29.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.090+0000 7f38d98ff700 1 --2- 172.21.15.67:0/3112110103 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f38d4101850 0x7f38d40735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:29.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.090+0000 7f38d98ff700 1 -- 172.21.15.67:0/3112110103 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d00095f0 con 0x7f38d4100ea0 2024-04-22T23:49:29.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.090+0000 7f38d98ff700 1 --2- 172.21.15.67:0/3112110103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38d4100ea0 0x7f38d4073090 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f38c400d8d0 tx=0x7f38c400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:49:29.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.090+0000 7f38caffd700 1 -- 172.21.15.67:0/3112110103 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38c40098a0 con 0x7f38d4100ea0 2024-04-22T23:49:29.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.090+0000 7f38caffd700 1 -- 172.21.15.67:0/3112110103 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f38c4010460 con 0x7f38d4100ea0 2024-04-22T23:49:29.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.090+0000 7f38dbb63700 1 -- 172.21.15.67:0/3112110103 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38d41a2b00 con 0x7f38d4100ea0 2024-04-22T23:49:29.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.090+0000 7f38caffd700 1 -- 172.21.15.67:0/3112110103 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38c400f5f0 con 0x7f38d4100ea0 2024-04-22T23:49:29.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.090+0000 7f38dbb63700 1 -- 172.21.15.67:0/3112110103 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38d41a2f10 con 0x7f38d4100ea0 2024-04-22T23:49:29.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.091+0000 7f38dbb63700 1 -- 172.21.15.67:0/3112110103 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38d41092a0 con 0x7f38d4100ea0 2024-04-22T23:49:29.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.091+0000 7f38caffd700 1 -- 172.21.15.67:0/3112110103 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f38c4009a00 con 0x7f38d4100ea0 2024-04-22T23:49:29.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.092+0000 7f38caffd700 1 --2- 172.21.15.67:0/3112110103 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f38c006ecc0 0x7f38c0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:49:29.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.092+0000 7f38caffd700 1 -- 172.21.15.67:0/3112110103 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f38c408edb0 con 0x7f38d4100ea0 2024-04-22T23:49:29.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.092+0000 7f38d90fe700 1 --2- 172.21.15.67:0/3112110103 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f38c006ecc0 0x7f38c0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:49:29.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.092+0000 7f38d90fe700 1 --2- 172.21.15.67:0/3112110103 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f38c006ecc0 0x7f38c0071180 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f38d4074880 tx=0x7f38d001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:49:29.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.097+0000 7f38caffd700 1 -- 172.21.15.67:0/3112110103 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f38c4057a70 con 0x7f38d4100ea0 2024-04-22T23:49:29.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.399+0000 7f38dbb63700 1 -- 172.21.15.67:0/3112110103 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f38d404f2e0 con 0x7f38d4100ea0 2024-04-22T23:49:29.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.399+0000 7f38caffd700 1 -- 172.21.15.67:0/3112110103 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f38c405b130 con 0x7f38d4100ea0 2024-04-22T23:49:29.391 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:49:29.391 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:49:29.391 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:49:29.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:49:29.391 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:49:29.391 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:49:29.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:49:29.391 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:49:29.391 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:49:29.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:49:29.392 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:49:29.392 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 4 2024-04-22T23:49:29.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:49:29.392 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:49:29.392 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 14 2024-04-22T23:49:29.392 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:49:29.392 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:49:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.401+0000 7f38dbb63700 1 -- 172.21.15.67:0/3112110103 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f38c006ecc0 msgr2=0x7f38c0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:49:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.401+0000 7f38dbb63700 1 --2- 172.21.15.67:0/3112110103 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f38c006ecc0 0x7f38c0071180 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f38d4074880 tx=0x7f38d001a040 comp rx=0 tx=0).stop 2024-04-22T23:49:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.401+0000 7f38dbb63700 1 -- 172.21.15.67:0/3112110103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38d4100ea0 msgr2=0x7f38d4073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:49:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.401+0000 7f38dbb63700 1 --2- 172.21.15.67:0/3112110103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38d4100ea0 0x7f38d4073090 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f38c400d8d0 tx=0x7f38c400dbe0 comp rx=0 tx=0).stop 2024-04-22T23:49:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.401+0000 7f38dbb63700 1 -- 172.21.15.67:0/3112110103 shutdown_connections 2024-04-22T23:49:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.401+0000 7f38dbb63700 1 --2- 172.21.15.67:0/3112110103 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f38c006ecc0 0x7f38c0071180 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.401+0000 7f38dbb63700 1 --2- 172.21.15.67:0/3112110103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38d4100ea0 0x7f38d4073090 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.401+0000 7f38dbb63700 1 --2- 172.21.15.67:0/3112110103 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f38d4101850 0x7f38d40735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.401+0000 7f38dbb63700 1 -- 172.21.15.67:0/3112110103 >> 172.21.15.67:0/3112110103 conn(0x7f38d40fc750 msgr2=0x7f38d40fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:49:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.401+0000 7f38dbb63700 1 -- 172.21.15.67:0/3112110103 shutdown_connections 2024-04-22T23:49:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.401+0000 7f38dbb63700 1 -- 172.21.15.67:0/3112110103 wait complete. 2024-04-22T23:49:29.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.508+0000 7ffa2b946700 1 -- 172.21.15.67:0/1838142188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa240ffda0 msgr2=0x7ffa24100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:49:29.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.508+0000 7ffa2b946700 1 --2- 172.21.15.67:0/1838142188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa240ffda0 0x7ffa24100180 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7ffa14009910 tx=0x7ffa14009c20 comp rx=0 tx=0).stop 2024-04-22T23:49:29.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.508+0000 7ffa2b946700 1 -- 172.21.15.67:0/1838142188 shutdown_connections 2024-04-22T23:49:29.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.508+0000 7ffa2b946700 1 --2- 172.21.15.67:0/1838142188 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa24100750 0x7ffa241047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:29.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.508+0000 7ffa2b946700 1 --2- 172.21.15.67:0/1838142188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa240ffda0 0x7ffa24100180 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:29.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.508+0000 7ffa2b946700 1 -- 172.21.15.67:0/1838142188 >> 172.21.15.67:0/1838142188 conn(0x7ffa240fb650 msgr2=0x7ffa240fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:49:29.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.508+0000 7ffa2b946700 1 -- 172.21.15.67:0/1838142188 shutdown_connections 2024-04-22T23:49:29.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.508+0000 7ffa2b946700 1 -- 172.21.15.67:0/1838142188 wait complete. 2024-04-22T23:49:29.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.509+0000 7ffa2b946700 1 Processor -- start 2024-04-22T23:49:29.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.509+0000 7ffa2b946700 1 -- start start 2024-04-22T23:49:29.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.509+0000 7ffa2b946700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa240ffda0 0x7ffa24101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:49:29.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.509+0000 7ffa2b946700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa24100750 0x7ffa24101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:49:29.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.509+0000 7ffa2b946700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa24102350 con 0x7ffa24100750 2024-04-22T23:49:29.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.509+0000 7ffa2b946700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa241024c0 con 0x7ffa240ffda0 2024-04-22T23:49:29.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.509+0000 7ffa28ee1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa24100750 0x7ffa24101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:49:29.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.509+0000 7ffa28ee1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa24100750 0x7ffa24101d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60690/0 (socket says 172.21.15.67:60690) 2024-04-22T23:49:29.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.509+0000 7ffa28ee1700 1 -- 172.21.15.67:0/27442406 learned_addr learned my addr 172.21.15.67:0/27442406 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:49:29.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.509+0000 7ffa296e2700 1 --2- 172.21.15.67:0/27442406 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa240ffda0 0x7ffa24101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:49:29.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.509+0000 7ffa28ee1700 1 -- 172.21.15.67:0/27442406 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa240ffda0 msgr2=0x7ffa24101840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:49:29.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.509+0000 7ffa28ee1700 1 --2- 172.21.15.67:0/27442406 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa240ffda0 0x7ffa24101840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:29.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.509+0000 7ffa28ee1700 1 -- 172.21.15.67:0/27442406 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa140095f0 con 0x7ffa24100750 2024-04-22T23:49:29.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.510+0000 7ffa28ee1700 1 --2- 172.21.15.67:0/27442406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa24100750 0x7ffa24101d80 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7ffa2000b480 tx=0x7ffa2000b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:49:29.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.510+0000 7ffa1a7fc700 1 -- 172.21.15.67:0/27442406 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa20011800 con 0x7ffa24100750 2024-04-22T23:49:29.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.510+0000 7ffa1a7fc700 1 -- 172.21.15.67:0/27442406 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffa20011e20 con 0x7ffa24100750 2024-04-22T23:49:29.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.510+0000 7ffa2b946700 1 -- 172.21.15.67:0/27442406 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa241a2bf0 con 0x7ffa24100750 2024-04-22T23:49:29.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.510+0000 7ffa1a7fc700 1 -- 172.21.15.67:0/27442406 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa20010ea0 con 0x7ffa24100750 2024-04-22T23:49:29.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.510+0000 7ffa2b946700 1 -- 172.21.15.67:0/27442406 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa241a3060 con 0x7ffa24100750 2024-04-22T23:49:29.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.511+0000 7ffa2b946700 1 -- 172.21.15.67:0/27442406 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa2404f2e0 con 0x7ffa24100750 2024-04-22T23:49:29.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.512+0000 7ffa1a7fc700 1 -- 172.21.15.67:0/27442406 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ffa20010380 con 0x7ffa24100750 2024-04-22T23:49:29.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.512+0000 7ffa1a7fc700 1 --2- 172.21.15.67:0/27442406 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ffa1006ecc0 0x7ffa10071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:49:29.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.512+0000 7ffa296e2700 1 --2- 172.21.15.67:0/27442406 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ffa1006ecc0 0x7ffa10071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:49:29.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.512+0000 7ffa1a7fc700 1 -- 172.21.15.67:0/27442406 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7ffa2008ea00 con 0x7ffa24100750 2024-04-22T23:49:29.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.513+0000 7ffa296e2700 1 --2- 172.21.15.67:0/27442406 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ffa1006ecc0 0x7ffa10071180 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ffa14006010 tx=0x7ffa1401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:49:29.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.517+0000 7ffa1a7fc700 1 -- 172.21.15.67:0/27442406 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ffa200576c0 con 0x7ffa24100750 2024-04-22T23:49:29.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.771+0000 7ffa2b946700 1 -- 172.21.15.67:0/27442406 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ffa241a32b0 con 0x7ffa24100750 2024-04-22T23:49:29.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.771+0000 7ffa1a7fc700 1 -- 172.21.15.67:0/27442406 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1872 (secure 0 0 0) 0x7ffa2005ad80 con 0x7ffa24100750 2024-04-22T23:49:29.764 INFO:teuthology.orchestra.run.smithi067.stdout:e33 2024-04-22T23:49:29.764 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:49:29.764 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:49:29.764 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:49:29.764 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:49:29.764 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:49:29.764 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:49:29.765 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:49:29.765 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:49:29.765 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:49:29.765 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:49:29.765 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:49:29.765 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:49:29.765 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:49:29.765 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:49:29.766 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:49:29.766 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:49:29.766 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:49:29.766 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:49:29.766 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:49:29.766 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:49:29.766 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:49:29.766 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:49:29.767 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:49:29.767 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:49:29.767 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:49:29.767 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:49:29.767 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:49:29.767 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:49:29.767 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:49:29.767 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:49:29.768 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:49:29.768 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:49:29.768 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:49:29.768 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:49:29.768 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:49:29.768 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:49:29.769 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:49:29.769 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:49:29.769 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:49:29.769 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:49:29.769 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44851} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1849648622,v1:172.21.15.134:6827/1849648622] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:49:29.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.774+0000 7ffa2b946700 1 -- 172.21.15.67:0/27442406 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ffa1006ecc0 msgr2=0x7ffa10071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:49:29.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.774+0000 7ffa2b946700 1 --2- 172.21.15.67:0/27442406 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ffa1006ecc0 0x7ffa10071180 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ffa14006010 tx=0x7ffa1401a040 comp rx=0 tx=0).stop 2024-04-22T23:49:29.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.774+0000 7ffa2b946700 1 -- 172.21.15.67:0/27442406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa24100750 msgr2=0x7ffa24101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:49:29.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.774+0000 7ffa2b946700 1 --2- 172.21.15.67:0/27442406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa24100750 0x7ffa24101d80 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7ffa2000b480 tx=0x7ffa2000b790 comp rx=0 tx=0).stop 2024-04-22T23:49:29.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.774+0000 7ffa2b946700 1 -- 172.21.15.67:0/27442406 shutdown_connections 2024-04-22T23:49:29.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.774+0000 7ffa2b946700 1 --2- 172.21.15.67:0/27442406 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ffa1006ecc0 0x7ffa10071180 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:29.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.774+0000 7ffa2b946700 1 --2- 172.21.15.67:0/27442406 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa240ffda0 0x7ffa24101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:29.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.774+0000 7ffa2b946700 1 --2- 172.21.15.67:0/27442406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa24100750 0x7ffa24101d80 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:29.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.774+0000 7ffa2b946700 1 -- 172.21.15.67:0/27442406 >> 172.21.15.67:0/27442406 conn(0x7ffa240fb650 msgr2=0x7ffa24104060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:49:29.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.774+0000 7ffa2b946700 1 -- 172.21.15.67:0/27442406 shutdown_connections 2024-04-22T23:49:29.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.774+0000 7ffa2b946700 1 -- 172.21.15.67:0/27442406 wait complete. 2024-04-22T23:49:29.772 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 33 2024-04-22T23:49:29.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.882+0000 7faa6bb84700 1 -- 172.21.15.67:0/222108760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa64101e60 msgr2=0x7faa64102240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:49:29.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.882+0000 7faa6bb84700 1 --2- 172.21.15.67:0/222108760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa64101e60 0x7faa64102240 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7faa60009910 tx=0x7faa60009c20 comp rx=0 tx=0).stop 2024-04-22T23:49:29.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.882+0000 7faa6bb84700 1 -- 172.21.15.67:0/222108760 shutdown_connections 2024-04-22T23:49:29.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.882+0000 7faa6bb84700 1 --2- 172.21.15.67:0/222108760 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faa64102780 0x7faa6410ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:29.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.882+0000 7faa6bb84700 1 --2- 172.21.15.67:0/222108760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa64101e60 0x7faa64102240 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:29.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.882+0000 7faa6bb84700 1 -- 172.21.15.67:0/222108760 >> 172.21.15.67:0/222108760 conn(0x7faa640fb650 msgr2=0x7faa640fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:49:29.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.882+0000 7faa6bb84700 1 -- 172.21.15.67:0/222108760 shutdown_connections 2024-04-22T23:49:29.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.882+0000 7faa6bb84700 1 -- 172.21.15.67:0/222108760 wait complete. 2024-04-22T23:49:29.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.883+0000 7faa6bb84700 1 Processor -- start 2024-04-22T23:49:29.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.883+0000 7faa6bb84700 1 -- start start 2024-04-22T23:49:29.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.883+0000 7faa6bb84700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa64101e60 0x7faa640ffac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:49:29.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.883+0000 7faa6bb84700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faa64102780 0x7faa64100000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:49:29.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.883+0000 7faa6bb84700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa64100540 con 0x7faa64101e60 2024-04-22T23:49:29.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.883+0000 7faa6bb84700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa64100680 con 0x7faa64102780 2024-04-22T23:49:29.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.883+0000 7faa69920700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa64101e60 0x7faa640ffac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:49:29.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.883+0000 7faa69920700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa64101e60 0x7faa640ffac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60700/0 (socket says 172.21.15.67:60700) 2024-04-22T23:49:29.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.883+0000 7faa69920700 1 -- 172.21.15.67:0/2924210036 learned_addr learned my addr 172.21.15.67:0/2924210036 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:49:29.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.883+0000 7faa6911f700 1 --2- 172.21.15.67:0/2924210036 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faa64102780 0x7faa64100000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:49:29.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.883+0000 7faa69920700 1 -- 172.21.15.67:0/2924210036 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faa64102780 msgr2=0x7faa64100000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:49:29.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.883+0000 7faa69920700 1 --2- 172.21.15.67:0/2924210036 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faa64102780 0x7faa64100000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:29.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.883+0000 7faa69920700 1 -- 172.21.15.67:0/2924210036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa600095f0 con 0x7faa64101e60 2024-04-22T23:49:29.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.884+0000 7faa69920700 1 --2- 172.21.15.67:0/2924210036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa64101e60 0x7faa640ffac0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7faa6000b5c0 tx=0x7faa6000be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:49:29.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.884+0000 7faa5affd700 1 -- 172.21.15.67:0/2924210036 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa6001d070 con 0x7faa64101e60 2024-04-22T23:49:29.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.884+0000 7faa5affd700 1 -- 172.21.15.67:0/2924210036 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faa6000f460 con 0x7faa64101e60 2024-04-22T23:49:29.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.884+0000 7faa6bb84700 1 -- 172.21.15.67:0/2924210036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa64100900 con 0x7faa64101e60 2024-04-22T23:49:29.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.884+0000 7faa5affd700 1 -- 172.21.15.67:0/2924210036 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa60017610 con 0x7faa64101e60 2024-04-22T23:49:29.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.884+0000 7faa6bb84700 1 -- 172.21.15.67:0/2924210036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa641a2d80 con 0x7faa64101e60 2024-04-22T23:49:29.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.885+0000 7faa6bb84700 1 -- 172.21.15.67:0/2924210036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa64108310 con 0x7faa64101e60 2024-04-22T23:49:29.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.885+0000 7faa5affd700 1 -- 172.21.15.67:0/2924210036 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7faa6000f5d0 con 0x7faa64101e60 2024-04-22T23:49:29.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.886+0000 7faa5affd700 1 --2- 172.21.15.67:0/2924210036 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faa5006ecc0 0x7faa50071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:49:29.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.886+0000 7faa5affd700 1 -- 172.21.15.67:0/2924210036 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7faa600905c0 con 0x7faa64101e60 2024-04-22T23:49:29.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.886+0000 7faa6911f700 1 --2- 172.21.15.67:0/2924210036 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faa5006ecc0 0x7faa50071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:49:29.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.886+0000 7faa6911f700 1 --2- 172.21.15.67:0/2924210036 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faa5006ecc0 0x7faa50071180 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7faa54005fd0 tx=0x7faa5400a560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:49:29.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:29.890+0000 7faa5affd700 1 -- 172.21.15.67:0/2924210036 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7faa60059200 con 0x7faa64101e60 2024-04-22T23:49:30.035 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:30 smithi067 ceph-mon[112438]: from='client.35272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:49:30.035 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:30 smithi067 ceph-mon[112438]: from='client.35276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:49:30.035 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:30 smithi067 ceph-mon[112438]: pgmap v695: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:30.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:30 smithi067 ceph-mon[112438]: from='client.35280 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:49:30.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:30 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3112110103' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:49:30.036 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:30 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/27442406' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:49:30.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.121+0000 7faa6bb84700 1 -- 172.21.15.67:0/2924210036 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7faa640611d0 con 0x7faa5006ecc0 2024-04-22T23:49:30.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.123+0000 7faa5affd700 1 -- 172.21.15.67:0/2924210036 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7faa640611d0 con 0x7faa5006ecc0 2024-04-22T23:49:30.114 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:49:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:49:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:49:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:49:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:49:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: "mds", 2024-04-22T23:49:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:49:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:49:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:49:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:49:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:49:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T23:49:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:49:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:49:30.115 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:49:30.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.125+0000 7faa6bb84700 1 -- 172.21.15.67:0/2924210036 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faa5006ecc0 msgr2=0x7faa50071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:49:30.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.125+0000 7faa6bb84700 1 --2- 172.21.15.67:0/2924210036 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faa5006ecc0 0x7faa50071180 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7faa54005fd0 tx=0x7faa5400a560 comp rx=0 tx=0).stop 2024-04-22T23:49:30.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.125+0000 7faa6bb84700 1 -- 172.21.15.67:0/2924210036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa64101e60 msgr2=0x7faa640ffac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:49:30.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.125+0000 7faa6bb84700 1 --2- 172.21.15.67:0/2924210036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa64101e60 0x7faa640ffac0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7faa6000b5c0 tx=0x7faa6000be00 comp rx=0 tx=0).stop 2024-04-22T23:49:30.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.125+0000 7faa6bb84700 1 -- 172.21.15.67:0/2924210036 shutdown_connections 2024-04-22T23:49:30.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.125+0000 7faa6bb84700 1 --2- 172.21.15.67:0/2924210036 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faa5006ecc0 0x7faa50071180 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:30.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.125+0000 7faa6bb84700 1 --2- 172.21.15.67:0/2924210036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa64101e60 0x7faa640ffac0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:30.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.125+0000 7faa6bb84700 1 --2- 172.21.15.67:0/2924210036 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faa64102780 0x7faa64100000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:30.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.125+0000 7faa6bb84700 1 -- 172.21.15.67:0/2924210036 >> 172.21.15.67:0/2924210036 conn(0x7faa640fb650 msgr2=0x7faa64105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:49:30.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.125+0000 7faa6bb84700 1 -- 172.21.15.67:0/2924210036 shutdown_connections 2024-04-22T23:49:30.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.125+0000 7faa6bb84700 1 -- 172.21.15.67:0/2924210036 wait complete. 2024-04-22T23:49:30.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.231+0000 7fe6081fe700 1 -- 172.21.15.67:0/415207597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe600100ea0 msgr2=0x7fe600101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:49:30.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.231+0000 7fe6081fe700 1 --2- 172.21.15.67:0/415207597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe600100ea0 0x7fe600101280 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc009910 tx=0x7fe5fc009c20 comp rx=0 tx=0).stop 2024-04-22T23:49:30.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.231+0000 7fe6081fe700 1 -- 172.21.15.67:0/415207597 shutdown_connections 2024-04-22T23:49:30.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.231+0000 7fe6081fe700 1 --2- 172.21.15.67:0/415207597 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe600101850 0x7fe6001058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:30.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.231+0000 7fe6081fe700 1 --2- 172.21.15.67:0/415207597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe600100ea0 0x7fe600101280 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:30.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.231+0000 7fe6081fe700 1 -- 172.21.15.67:0/415207597 >> 172.21.15.67:0/415207597 conn(0x7fe6000fc750 msgr2=0x7fe6000feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:49:30.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.232+0000 7fe6081fe700 1 -- 172.21.15.67:0/415207597 shutdown_connections 2024-04-22T23:49:30.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.232+0000 7fe6081fe700 1 -- 172.21.15.67:0/415207597 wait complete. 2024-04-22T23:49:30.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.232+0000 7fe6081fe700 1 Processor -- start 2024-04-22T23:49:30.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.232+0000 7fe6081fe700 1 -- start start 2024-04-22T23:49:30.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.232+0000 7fe6081fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe600100ea0 0x7fe600073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:49:30.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.232+0000 7fe6081fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe600101850 0x7fe6000735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:49:30.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.232+0000 7fe6081fe700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe600073b10 con 0x7fe600100ea0 2024-04-22T23:49:30.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.232+0000 7fe6081fe700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe600073c50 con 0x7fe600101850 2024-04-22T23:49:30.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.233+0000 7fe605f9a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe600100ea0 0x7fe600073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:49:30.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.233+0000 7fe605f9a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe600100ea0 0x7fe600073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60718/0 (socket says 172.21.15.67:60718) 2024-04-22T23:49:30.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.233+0000 7fe605f9a700 1 -- 172.21.15.67:0/2906415523 learned_addr learned my addr 172.21.15.67:0/2906415523 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:49:30.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.233+0000 7fe605799700 1 --2- 172.21.15.67:0/2906415523 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe600101850 0x7fe6000735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:49:30.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.233+0000 7fe605f9a700 1 -- 172.21.15.67:0/2906415523 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe600101850 msgr2=0x7fe6000735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:49:30.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.233+0000 7fe605f9a700 1 --2- 172.21.15.67:0/2906415523 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe600101850 0x7fe6000735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:30.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.233+0000 7fe605f9a700 1 -- 172.21.15.67:0/2906415523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5fc0095f0 con 0x7fe600100ea0 2024-04-22T23:49:30.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.233+0000 7fe605f9a700 1 --2- 172.21.15.67:0/2906415523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe600100ea0 0x7fe600073090 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc006010 tx=0x7fe5fc00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:49:30.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.233+0000 7fe5f6ffd700 1 -- 172.21.15.67:0/2906415523 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5fc01d070 con 0x7fe600100ea0 2024-04-22T23:49:30.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.233+0000 7fe5f6ffd700 1 -- 172.21.15.67:0/2906415523 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe5fc00f460 con 0x7fe600100ea0 2024-04-22T23:49:30.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.233+0000 7fe6081fe700 1 -- 172.21.15.67:0/2906415523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe600073ed0 con 0x7fe600100ea0 2024-04-22T23:49:30.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.233+0000 7fe5f6ffd700 1 -- 172.21.15.67:0/2906415523 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5fc017610 con 0x7fe600100ea0 2024-04-22T23:49:30.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.233+0000 7fe6081fe700 1 -- 172.21.15.67:0/2906415523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6001a2e40 con 0x7fe600100ea0 2024-04-22T23:49:30.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.234+0000 7fe6081fe700 1 -- 172.21.15.67:0/2906415523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe600109240 con 0x7fe600100ea0 2024-04-22T23:49:30.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.235+0000 7fe5f6ffd700 1 -- 172.21.15.67:0/2906415523 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe5fc00f5d0 con 0x7fe600100ea0 2024-04-22T23:49:30.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.235+0000 7fe5f6ffd700 1 --2- 172.21.15.67:0/2906415523 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe5ec06ecc0 0x7fe5ec071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:49:30.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.236+0000 7fe5f6ffd700 1 -- 172.21.15.67:0/2906415523 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fe5fc090550 con 0x7fe600100ea0 2024-04-22T23:49:30.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.236+0000 7fe605799700 1 --2- 172.21.15.67:0/2906415523 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe5ec06ecc0 0x7fe5ec071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:49:30.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.236+0000 7fe605799700 1 --2- 172.21.15.67:0/2906415523 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe5ec06ecc0 0x7fe5ec071180 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe600074880 tx=0x7fe5f000a300 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:49:30.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.240+0000 7fe5f6ffd700 1 -- 172.21.15.67:0/2906415523 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe5fc059190 con 0x7fe600100ea0 2024-04-22T23:49:30.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:30 smithi134 ceph-mon[85630]: from='client.35272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:49:30.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:30 smithi134 ceph-mon[85630]: from='client.35276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:49:30.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:30 smithi134 ceph-mon[85630]: pgmap v695: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:30.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:30 smithi134 ceph-mon[85630]: from='client.35280 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:49:30.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:30 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3112110103' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:49:30.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:30 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/27442406' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:49:30.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.537+0000 7fe6081fe700 1 -- 172.21.15.67:0/2906415523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe600108c20 con 0x7fe600100ea0 2024-04-22T23:49:30.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.538+0000 7fe5f6ffd700 1 -- 172.21.15.67:0/2906415523 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4505 (secure 0 0 0) 0x7fe5fc05c850 con 0x7fe600100ea0 2024-04-22T23:49:30.529 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:49:30.530 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T23:49:30.530 INFO:teuthology.orchestra.run.smithi067.stdout: host smithi134 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-22T23:49:30.530 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T23:49:30.530 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:49:30.530 INFO:teuthology.orchestra.run.smithi067.stdout:Traceback (most recent call last): 2024-04-22T23:49:30.530 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T23:49:30.530 INFO:teuthology.orchestra.run.smithi067.stdout: return _run_code(code, main_globals, None, 2024-04-22T23:49:30.530 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T23:49:30.530 INFO:teuthology.orchestra.run.smithi067.stdout: exec(code, run_globals) 2024-04-22T23:49:30.530 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5642, in 2024-04-22T23:49:30.530 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5630, in main 2024-04-22T23:49:30.530 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 408, in _infer_config 2024-04-22T23:49:30.530 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 323, in _infer_fsid 2024-04-22T23:49:30.531 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 436, in _infer_image 2024-04-22T23:49:30.531 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 310, in _validate_fsid 2024-04-22T23:49:30.531 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 3380, in command_ceph_volume 2024-04-22T23:49:30.531 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T23:49:30.531 INFO:teuthology.orchestra.run.smithi067.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:49:30.531 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:49:30.531 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:49:30.531 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:49:30.531 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:49:30.531 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:49:30.531 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:49:30.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.541+0000 7fe6081fe700 1 -- 172.21.15.67:0/2906415523 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe5ec06ecc0 msgr2=0x7fe5ec071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:49:30.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.541+0000 7fe6081fe700 1 --2- 172.21.15.67:0/2906415523 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe5ec06ecc0 0x7fe5ec071180 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe600074880 tx=0x7fe5f000a300 comp rx=0 tx=0).stop 2024-04-22T23:49:30.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.541+0000 7fe6081fe700 1 -- 172.21.15.67:0/2906415523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe600100ea0 msgr2=0x7fe600073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:49:30.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.541+0000 7fe6081fe700 1 --2- 172.21.15.67:0/2906415523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe600100ea0 0x7fe600073090 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc006010 tx=0x7fe5fc00bd70 comp rx=0 tx=0).stop 2024-04-22T23:49:30.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.541+0000 7fe6081fe700 1 -- 172.21.15.67:0/2906415523 shutdown_connections 2024-04-22T23:49:30.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.541+0000 7fe6081fe700 1 --2- 172.21.15.67:0/2906415523 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe5ec06ecc0 0x7fe5ec071180 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:30.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.541+0000 7fe6081fe700 1 --2- 172.21.15.67:0/2906415523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe600100ea0 0x7fe600073090 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:30.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.541+0000 7fe6081fe700 1 --2- 172.21.15.67:0/2906415523 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe600101850 0x7fe6000735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:49:30.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.541+0000 7fe6081fe700 1 -- 172.21.15.67:0/2906415523 >> 172.21.15.67:0/2906415523 conn(0x7fe6000fc750 msgr2=0x7fe6000fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:49:30.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.541+0000 7fe6081fe700 1 -- 172.21.15.67:0/2906415523 shutdown_connections 2024-04-22T23:49:30.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:49:30.541+0000 7fe6081fe700 1 -- 172.21.15.67:0/2906415523 wait complete. 2024-04-22T23:49:31.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:31 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2906415523' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:49:31.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:31 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2906415523' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:49:32.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:32 smithi067 ceph-mon[112438]: from='client.35292 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:49:32.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:32 smithi067 ceph-mon[112438]: pgmap v696: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:32.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:32 smithi134 ceph-mon[85630]: from='client.35292 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:49:32.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:32 smithi134 ceph-mon[85630]: pgmap v696: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:34.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:34 smithi067 ceph-mon[112438]: pgmap v697: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:34.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:34 smithi134 ceph-mon[85630]: pgmap v697: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:36.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:36 smithi067 ceph-mon[112438]: pgmap v698: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:36.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:36 smithi134 ceph-mon[85630]: pgmap v698: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:38.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:38 smithi067 ceph-mon[112438]: pgmap v699: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:38.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:38 smithi134 ceph-mon[85630]: pgmap v699: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:40.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:40 smithi067 ceph-mon[112438]: pgmap v700: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:40.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:40 smithi134 ceph-mon[85630]: pgmap v700: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:42.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:42 smithi067 ceph-mon[112438]: pgmap v701: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:42.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:42 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:49:42.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:42 smithi134 ceph-mon[85630]: pgmap v701: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:42.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:42 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:49:44.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:44 smithi067 ceph-mon[112438]: pgmap v702: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:44.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:44 smithi134 ceph-mon[85630]: pgmap v702: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:46.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:46 smithi067 ceph-mon[112438]: pgmap v703: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:46.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:46 smithi134 ceph-mon[85630]: pgmap v703: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:48.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:48 smithi067 ceph-mon[112438]: pgmap v704: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:48.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:48 smithi134 ceph-mon[85630]: pgmap v704: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:50.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:50 smithi067 ceph-mon[112438]: pgmap v705: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:50.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:50 smithi134 ceph-mon[85630]: pgmap v705: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:52.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:52 smithi134 ceph-mon[85630]: pgmap v706: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:52.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:52 smithi067 ceph-mon[112438]: pgmap v706: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:54.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:54 smithi134 ceph-mon[85630]: pgmap v707: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:54.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:54 smithi067 ceph-mon[112438]: pgmap v707: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:56.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:56 smithi134 ceph-mon[85630]: pgmap v708: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:56.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:49:56.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:56 smithi067 ceph-mon[112438]: pgmap v708: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:56.536 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:49:57.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:49:57 smithi134 ceph-mon[85630]: pgmap v709: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:49:57.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:49:57 smithi067 ceph-mon[112438]: pgmap v709: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:00.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[112434]: 2024-04-22T23:49:59.999+0000 7f20165b9700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:50:00.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[112434]: 2024-04-22T23:49:59.999+0000 7f20165b9700 -1 log_channel(cluster) log [ERR] : [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T23:50:00.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[112434]: 2024-04-22T23:49:59.999+0000 7f20165b9700 -1 log_channel(cluster) log [ERR] : host smithi134 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-22T23:50:00.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[112434]: 2024-04-22T23:49:59.999+0000 7f20165b9700 -1 log_channel(cluster) log [ERR] : /usr/bin/podman: timeout after 895 seconds 2024-04-22T23:50:00.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[112434]: 2024-04-22T23:49:59.999+0000 7f20165b9700 -1 log_channel(cluster) log [ERR] : Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:50:00.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[112434]: 2024-04-22T23:49:59.999+0000 7f20165b9700 -1 log_channel(cluster) log [ERR] : Traceback (most recent call last): 2024-04-22T23:50:00.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[112434]: 2024-04-22T23:49:59.999+0000 7f20165b9700 -1 log_channel(cluster) log [ERR] : File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T23:50:00.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[112434]: 2024-04-22T23:49:59.999+0000 7f20165b9700 -1 log_channel(cluster) log [ERR] : return _run_code(code, main_globals, None, 2024-04-22T23:50:00.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[112434]: 2024-04-22T23:49:59.999+0000 7f20165b9700 -1 log_channel(cluster) log [ERR] : File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T23:50:00.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[112434]: 2024-04-22T23:49:59.999+0000 7f20165b9700 -1 log_channel(cluster) log [ERR] : exec(code, run_globals) 2024-04-22T23:50:00.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[112434]: 2024-04-22T23:49:59.999+0000 7f20165b9700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5642, in 2024-04-22T23:50:00.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[112434]: 2024-04-22T23:49:59.999+0000 7f20165b9700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5630, in main 2024-04-22T23:50:00.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[112434]: 2024-04-22T23:49:59.999+0000 7f20165b9700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 408, in _infer_config 2024-04-22T23:50:00.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[112434]: 2024-04-22T23:49:59.999+0000 7f20165b9700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 323, in _infer_fsid 2024-04-22T23:50:00.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[112434]: 2024-04-22T23:49:59.999+0000 7f20165b9700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 436, in _infer_image 2024-04-22T23:50:00.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[112434]: 2024-04-22T23:49:59.999+0000 7f20165b9700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 310, in _validate_fsid 2024-04-22T23:50:00.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[112434]: 2024-04-22T23:49:59.999+0000 7f20165b9700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 3380, in command_ceph_volume 2024-04-22T23:50:00.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[112434]: 2024-04-22T23:49:59.999+0000 7f20165b9700 -1 log_channel(cluster) log [ERR] : File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T23:50:00.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[112434]: 2024-04-22T23:49:59.999+0000 7f20165b9700 -1 log_channel(cluster) log [ERR] : RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:50:00.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[112434]: 2024-04-22T23:49:59.999+0000 7f20165b9700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:50:00.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[112434]: 2024-04-22T23:49:59.999+0000 7f20165b9700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2024-04-22T23:50:00.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[112434]: 2024-04-22T23:49:59.999+0000 7f20165b9700 -1 log_channel(cluster) log [ERR] : [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:50:00.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[112434]: 2024-04-22T23:49:59.999+0000 7f20165b9700 -1 log_channel(cluster) log [ERR] : fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:50:00.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[112434]: 2024-04-22T23:49:59.999+0000 7f20165b9700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:50:00.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[112434]: 2024-04-22T23:49:59.999+0000 7f20165b9700 -1 log_channel(cluster) log [ERR] : fs cephfs is offline because no MDS is active for it. 2024-04-22T23:50:00.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-mon[112438]: pgmap v710: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:00.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-mon[112438]: Health detail: HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:50:00.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-mon[112438]: [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T23:50:00.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-mon[112438]: host smithi134 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-22T23:50:00.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-mon[112438]: /usr/bin/podman: timeout after 895 seconds 2024-04-22T23:50:00.288 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-mon[112438]: Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:50:00.289 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-mon[112438]: Traceback (most recent call last): 2024-04-22T23:50:00.289 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-mon[112438]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T23:50:00.289 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-mon[112438]: return _run_code(code, main_globals, None, 2024-04-22T23:50:00.289 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-mon[112438]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T23:50:00.289 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-mon[112438]: exec(code, run_globals) 2024-04-22T23:50:00.289 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-mon[112438]: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5642, in 2024-04-22T23:50:00.289 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-mon[112438]: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5630, in main 2024-04-22T23:50:00.289 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-mon[112438]: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 408, in _infer_config 2024-04-22T23:50:00.289 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-mon[112438]: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 323, in _infer_fsid 2024-04-22T23:50:00.289 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-mon[112438]: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 436, in _infer_image 2024-04-22T23:50:00.289 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-mon[112438]: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 310, in _validate_fsid 2024-04-22T23:50:00.289 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-mon[112438]: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 3380, in command_ceph_volume 2024-04-22T23:50:00.289 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-mon[112438]: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T23:50:00.289 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-mon[112438]: RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:50:00.289 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-mon[112438]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:50:00.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-mon[112438]: fs cephfs is degraded 2024-04-22T23:50:00.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-mon[112438]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:50:00.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-mon[112438]: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:50:00.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-mon[112438]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:50:00.290 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:00 smithi067 ceph-mon[112438]: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:50:00.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:00 smithi134 ceph-mon[85630]: pgmap v710: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:00.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:00 smithi134 ceph-mon[85630]: Health detail: HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:50:00.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:00 smithi134 ceph-mon[85630]: [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T23:50:00.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:00 smithi134 ceph-mon[85630]: host smithi134 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-22T23:50:00.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:00 smithi134 ceph-mon[85630]: /usr/bin/podman: timeout after 895 seconds 2024-04-22T23:50:00.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:00 smithi134 ceph-mon[85630]: Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:50:00.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:00 smithi134 ceph-mon[85630]: Traceback (most recent call last): 2024-04-22T23:50:00.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:00 smithi134 ceph-mon[85630]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T23:50:00.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:00 smithi134 ceph-mon[85630]: return _run_code(code, main_globals, None, 2024-04-22T23:50:00.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:00 smithi134 ceph-mon[85630]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T23:50:00.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:00 smithi134 ceph-mon[85630]: exec(code, run_globals) 2024-04-22T23:50:00.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:00 smithi134 ceph-mon[85630]: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5642, in 2024-04-22T23:50:00.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:00 smithi134 ceph-mon[85630]: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5630, in main 2024-04-22T23:50:00.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:00 smithi134 ceph-mon[85630]: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 408, in _infer_config 2024-04-22T23:50:00.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:00 smithi134 ceph-mon[85630]: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 323, in _infer_fsid 2024-04-22T23:50:00.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:00 smithi134 ceph-mon[85630]: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 436, in _infer_image 2024-04-22T23:50:00.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:00 smithi134 ceph-mon[85630]: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 310, in _validate_fsid 2024-04-22T23:50:00.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:00 smithi134 ceph-mon[85630]: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 3380, in command_ceph_volume 2024-04-22T23:50:00.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:00 smithi134 ceph-mon[85630]: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T23:50:00.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:00 smithi134 ceph-mon[85630]: RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:50:00.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:00 smithi134 ceph-mon[85630]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:50:00.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:00 smithi134 ceph-mon[85630]: fs cephfs is degraded 2024-04-22T23:50:00.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:00 smithi134 ceph-mon[85630]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:50:00.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:00 smithi134 ceph-mon[85630]: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:50:00.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:00 smithi134 ceph-mon[85630]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:50:00.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:00 smithi134 ceph-mon[85630]: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:50:00.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.664+0000 7f2544876700 1 -- 172.21.15.67:0/1171625401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25400ffb40 msgr2=0x7f25400fffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:00.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.664+0000 7f2544876700 1 --2- 172.21.15.67:0/1171625401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25400ffb40 0x7f25400fffc0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f2530009910 tx=0x7f2530009c20 comp rx=0 tx=0).stop 2024-04-22T23:50:00.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.664+0000 7f2544876700 1 -- 172.21.15.67:0/1171625401 shutdown_connections 2024-04-22T23:50:00.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.664+0000 7f2544876700 1 --2- 172.21.15.67:0/1171625401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25400ffb40 0x7f25400fffc0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:00.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.664+0000 7f2544876700 1 --2- 172.21.15.67:0/1171625401 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2540105bf0 0x7f2540105fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:00.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.664+0000 7f2544876700 1 -- 172.21.15.67:0/1171625401 >> 172.21.15.67:0/1171625401 conn(0x7f25400fb650 msgr2=0x7f25400fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:50:00.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.664+0000 7f2544876700 1 -- 172.21.15.67:0/1171625401 shutdown_connections 2024-04-22T23:50:00.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.664+0000 7f2544876700 1 -- 172.21.15.67:0/1171625401 wait complete. 2024-04-22T23:50:00.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.665+0000 7f2544876700 1 Processor -- start 2024-04-22T23:50:00.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.665+0000 7f2544876700 1 -- start start 2024-04-22T23:50:00.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.665+0000 7f2544876700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f25400ffb40 0x7f254019e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:00.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.665+0000 7f2544876700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2540105bf0 0x7f254019ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:00.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.665+0000 7f2544876700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f254019f390 con 0x7f2540105bf0 2024-04-22T23:50:00.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.665+0000 7f2544876700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2540198840 con 0x7f25400ffb40 2024-04-22T23:50:00.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.665+0000 7f253dd9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2540105bf0 0x7f254019ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:00.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.665+0000 7f253dd9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2540105bf0 0x7f254019ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60362/0 (socket says 172.21.15.67:60362) 2024-04-22T23:50:00.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.665+0000 7f253dd9b700 1 -- 172.21.15.67:0/2272975693 learned_addr learned my addr 172.21.15.67:0/2272975693 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:50:00.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.665+0000 7f253e59c700 1 --2- 172.21.15.67:0/2272975693 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f25400ffb40 0x7f254019e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:00.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.665+0000 7f253dd9b700 1 -- 172.21.15.67:0/2272975693 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f25400ffb40 msgr2=0x7f254019e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:00.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.665+0000 7f253dd9b700 1 --2- 172.21.15.67:0/2272975693 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f25400ffb40 0x7f254019e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:00.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.665+0000 7f253dd9b700 1 -- 172.21.15.67:0/2272975693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25300095f0 con 0x7f2540105bf0 2024-04-22T23:50:00.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.666+0000 7f253dd9b700 1 --2- 172.21.15.67:0/2272975693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2540105bf0 0x7f254019ed00 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f25300092e0 tx=0x7f253000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:50:00.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.666+0000 7f252f7fe700 1 -- 172.21.15.67:0/2272975693 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f253001d070 con 0x7f2540105bf0 2024-04-22T23:50:00.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.666+0000 7f2544876700 1 -- 172.21.15.67:0/2272975693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2540198ac0 con 0x7f2540105bf0 2024-04-22T23:50:00.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.666+0000 7f252f7fe700 1 -- 172.21.15.67:0/2272975693 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2530005e00 con 0x7f2540105bf0 2024-04-22T23:50:00.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.666+0000 7f252f7fe700 1 -- 172.21.15.67:0/2272975693 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2530017400 con 0x7f2540105bf0 2024-04-22T23:50:00.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.666+0000 7f2544876700 1 -- 172.21.15.67:0/2272975693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2540198fb0 con 0x7f2540105bf0 2024-04-22T23:50:00.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.667+0000 7f2544876700 1 -- 172.21.15.67:0/2272975693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2540108020 con 0x7f2540105bf0 2024-04-22T23:50:00.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.667+0000 7f252f7fe700 1 -- 172.21.15.67:0/2272975693 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f253000f6f0 con 0x7f2540105bf0 2024-04-22T23:50:00.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.668+0000 7f252f7fe700 1 --2- 172.21.15.67:0/2272975693 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f252806ecc0 0x7f2528071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:00.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.668+0000 7f252f7fe700 1 -- 172.21.15.67:0/2272975693 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f25300904b0 con 0x7f2540105bf0 2024-04-22T23:50:00.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.668+0000 7f253e59c700 1 --2- 172.21.15.67:0/2272975693 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f252806ecc0 0x7f2528071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:00.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.669+0000 7f253e59c700 1 --2- 172.21.15.67:0/2272975693 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f252806ecc0 0x7f2528071180 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f2534005950 tx=0x7f253400b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:50:00.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.673+0000 7f252f7fe700 1 -- 172.21.15.67:0/2272975693 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f253005cc20 con 0x7f2540105bf0 2024-04-22T23:50:00.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.904+0000 7f2544876700 1 -- 172.21.15.67:0/2272975693 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2540199290 con 0x7f252806ecc0 2024-04-22T23:50:00.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.906+0000 7f252f7fe700 1 -- 172.21.15.67:0/2272975693 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f2540199290 con 0x7f252806ecc0 2024-04-22T23:50:00.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.908+0000 7f2544876700 1 -- 172.21.15.67:0/2272975693 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f252806ecc0 msgr2=0x7f2528071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:00.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.908+0000 7f2544876700 1 --2- 172.21.15.67:0/2272975693 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f252806ecc0 0x7f2528071180 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f2534005950 tx=0x7f253400b410 comp rx=0 tx=0).stop 2024-04-22T23:50:00.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.908+0000 7f2544876700 1 -- 172.21.15.67:0/2272975693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2540105bf0 msgr2=0x7f254019ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:00.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.908+0000 7f2544876700 1 --2- 172.21.15.67:0/2272975693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2540105bf0 0x7f254019ed00 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f25300092e0 tx=0x7f253000bd70 comp rx=0 tx=0).stop 2024-04-22T23:50:00.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.908+0000 7f2544876700 1 -- 172.21.15.67:0/2272975693 shutdown_connections 2024-04-22T23:50:00.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.908+0000 7f2544876700 1 --2- 172.21.15.67:0/2272975693 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f252806ecc0 0x7f2528071180 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:00.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.908+0000 7f2544876700 1 --2- 172.21.15.67:0/2272975693 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f25400ffb40 0x7f254019e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:00.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.908+0000 7f2544876700 1 --2- 172.21.15.67:0/2272975693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2540105bf0 0x7f254019ed00 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:00.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.908+0000 7f2544876700 1 -- 172.21.15.67:0/2272975693 >> 172.21.15.67:0/2272975693 conn(0x7f25400fb650 msgr2=0x7f25400fd150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:50:00.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.908+0000 7f2544876700 1 -- 172.21.15.67:0/2272975693 shutdown_connections 2024-04-22T23:50:00.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:00.908+0000 7f2544876700 1 -- 172.21.15.67:0/2272975693 wait complete. 2024-04-22T23:50:00.910 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:50:01.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.016+0000 7f75cffa0700 1 -- 172.21.15.67:0/1946450874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75c81027a0 msgr2=0x7f75c810ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:01.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.016+0000 7f75cffa0700 1 --2- 172.21.15.67:0/1946450874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75c81027a0 0x7f75c810ac90 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f75b8009910 tx=0x7f75b8009c20 comp rx=0 tx=0).stop 2024-04-22T23:50:01.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.016+0000 7f75cffa0700 1 -- 172.21.15.67:0/1946450874 shutdown_connections 2024-04-22T23:50:01.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.016+0000 7f75cffa0700 1 --2- 172.21.15.67:0/1946450874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75c81027a0 0x7f75c810ac90 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:01.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.016+0000 7f75cffa0700 1 --2- 172.21.15.67:0/1946450874 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75c8101e80 0x7f75c8102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:01.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.016+0000 7f75cffa0700 1 -- 172.21.15.67:0/1946450874 >> 172.21.15.67:0/1946450874 conn(0x7f75c80fb650 msgr2=0x7f75c80fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:50:01.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.016+0000 7f75cffa0700 1 -- 172.21.15.67:0/1946450874 shutdown_connections 2024-04-22T23:50:01.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.016+0000 7f75cffa0700 1 -- 172.21.15.67:0/1946450874 wait complete. 2024-04-22T23:50:01.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.016+0000 7f75cffa0700 1 Processor -- start 2024-04-22T23:50:01.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.017+0000 7f75cffa0700 1 -- start start 2024-04-22T23:50:01.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.017+0000 7f75cffa0700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75c8101e80 0x7f75c819e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:01.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.017+0000 7f75cffa0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75c81027a0 0x7f75c819ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:01.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.017+0000 7f75cffa0700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c819f400 con 0x7f75c81027a0 2024-04-22T23:50:01.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.017+0000 7f75cffa0700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c81988b0 con 0x7f75c8101e80 2024-04-22T23:50:01.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.017+0000 7f75cd53b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75c81027a0 0x7f75c819ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:01.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.017+0000 7f75cd53b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75c81027a0 0x7f75c819ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60380/0 (socket says 172.21.15.67:60380) 2024-04-22T23:50:01.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.017+0000 7f75cd53b700 1 -- 172.21.15.67:0/2688525168 learned_addr learned my addr 172.21.15.67:0/2688525168 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:50:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.017+0000 7f75cdd3c700 1 --2- 172.21.15.67:0/2688525168 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75c8101e80 0x7f75c819e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.017+0000 7f75cd53b700 1 -- 172.21.15.67:0/2688525168 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75c8101e80 msgr2=0x7f75c819e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.017+0000 7f75cd53b700 1 --2- 172.21.15.67:0/2688525168 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75c8101e80 0x7f75c819e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.017+0000 7f75cd53b700 1 -- 172.21.15.67:0/2688525168 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b80095f0 con 0x7f75c81027a0 2024-04-22T23:50:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.017+0000 7f75cd53b700 1 --2- 172.21.15.67:0/2688525168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75c81027a0 0x7f75c819ed70 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f75b80092e0 tx=0x7f75b800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:50:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.018+0000 7f75beffd700 1 -- 172.21.15.67:0/2688525168 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75b801d070 con 0x7f75c81027a0 2024-04-22T23:50:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.018+0000 7f75cffa0700 1 -- 172.21.15.67:0/2688525168 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c8198b30 con 0x7f75c81027a0 2024-04-22T23:50:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.018+0000 7f75beffd700 1 -- 172.21.15.67:0/2688525168 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f75b8005dc0 con 0x7f75c81027a0 2024-04-22T23:50:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.018+0000 7f75beffd700 1 -- 172.21.15.67:0/2688525168 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75b8017400 con 0x7f75c81027a0 2024-04-22T23:50:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.018+0000 7f75cffa0700 1 -- 172.21.15.67:0/2688525168 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c8199020 con 0x7f75c81027a0 2024-04-22T23:50:01.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.019+0000 7f75cffa0700 1 -- 172.21.15.67:0/2688525168 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75c810a720 con 0x7f75c81027a0 2024-04-22T23:50:01.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.019+0000 7f75beffd700 1 -- 172.21.15.67:0/2688525168 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f75b800f460 con 0x7f75c81027a0 2024-04-22T23:50:01.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.020+0000 7f75beffd700 1 --2- 172.21.15.67:0/2688525168 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f75b406ecc0 0x7f75b4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:01.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.020+0000 7f75beffd700 1 -- 172.21.15.67:0/2688525168 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f75b80904f0 con 0x7f75c81027a0 2024-04-22T23:50:01.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.020+0000 7f75cdd3c700 1 --2- 172.21.15.67:0/2688525168 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f75b406ecc0 0x7f75b4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:01.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.020+0000 7f75cdd3c700 1 --2- 172.21.15.67:0/2688525168 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f75b406ecc0 0x7f75b4071180 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f75c4005950 tx=0x7f75c40058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:50:01.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.024+0000 7f75beffd700 1 -- 172.21.15.67:0/2688525168 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f75b805cc60 con 0x7f75c81027a0 2024-04-22T23:50:01.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.255+0000 7f75cffa0700 1 -- 172.21.15.67:0/2688525168 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f75c8199240 con 0x7f75b406ecc0 2024-04-22T23:50:01.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.256+0000 7f75beffd700 1 -- 172.21.15.67:0/2688525168 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f75c8199240 con 0x7f75b406ecc0 2024-04-22T23:50:01.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.258+0000 7f75cffa0700 1 -- 172.21.15.67:0/2688525168 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f75b406ecc0 msgr2=0x7f75b4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:01.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.258+0000 7f75cffa0700 1 --2- 172.21.15.67:0/2688525168 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f75b406ecc0 0x7f75b4071180 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f75c4005950 tx=0x7f75c40058e0 comp rx=0 tx=0).stop 2024-04-22T23:50:01.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.258+0000 7f75cffa0700 1 -- 172.21.15.67:0/2688525168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75c81027a0 msgr2=0x7f75c819ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:01.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.258+0000 7f75cffa0700 1 --2- 172.21.15.67:0/2688525168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75c81027a0 0x7f75c819ed70 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f75b80092e0 tx=0x7f75b800be20 comp rx=0 tx=0).stop 2024-04-22T23:50:01.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.259+0000 7f75cffa0700 1 -- 172.21.15.67:0/2688525168 shutdown_connections 2024-04-22T23:50:01.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.259+0000 7f75cffa0700 1 --2- 172.21.15.67:0/2688525168 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f75b406ecc0 0x7f75b4071180 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:01.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.259+0000 7f75cffa0700 1 --2- 172.21.15.67:0/2688525168 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75c8101e80 0x7f75c819e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:01.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.259+0000 7f75cffa0700 1 --2- 172.21.15.67:0/2688525168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75c81027a0 0x7f75c819ed70 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:01.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.259+0000 7f75cffa0700 1 -- 172.21.15.67:0/2688525168 >> 172.21.15.67:0/2688525168 conn(0x7f75c80fb650 msgr2=0x7f75c8107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:50:01.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.259+0000 7f75cffa0700 1 -- 172.21.15.67:0/2688525168 shutdown_connections 2024-04-22T23:50:01.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.259+0000 7f75cffa0700 1 -- 172.21.15.67:0/2688525168 wait complete. 2024-04-22T23:50:01.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.366+0000 7fba08dfa700 1 -- 172.21.15.67:0/3981309967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba04100340 msgr2=0x7fba0410d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:01.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.366+0000 7fba08dfa700 1 --2- 172.21.15.67:0/3981309967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba04100340 0x7fba0410d000 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8009910 tx=0x7fb9f8009c20 comp rx=0 tx=0).stop 2024-04-22T23:50:01.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.366+0000 7fba08dfa700 1 -- 172.21.15.67:0/3981309967 shutdown_connections 2024-04-22T23:50:01.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.366+0000 7fba08dfa700 1 --2- 172.21.15.67:0/3981309967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba04100340 0x7fba0410d000 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:01.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.366+0000 7fba08dfa700 1 --2- 172.21.15.67:0/3981309967 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba040ff990 0x7fba040ffd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:01.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.366+0000 7fba08dfa700 1 -- 172.21.15.67:0/3981309967 >> 172.21.15.67:0/3981309967 conn(0x7fba040fb650 msgr2=0x7fba040fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:50:01.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.366+0000 7fba08dfa700 1 -- 172.21.15.67:0/3981309967 shutdown_connections 2024-04-22T23:50:01.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.366+0000 7fba08dfa700 1 -- 172.21.15.67:0/3981309967 wait complete. 2024-04-22T23:50:01.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.367+0000 7fba08dfa700 1 Processor -- start 2024-04-22T23:50:01.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.367+0000 7fba08dfa700 1 -- start start 2024-04-22T23:50:01.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.367+0000 7fba08dfa700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba040ff990 0x7fba0419e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:01.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.367+0000 7fba08dfa700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba04100340 0x7fba0419eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:01.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.367+0000 7fba08dfa700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba0419f430 con 0x7fba04100340 2024-04-22T23:50:01.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.367+0000 7fba08dfa700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba041988e0 con 0x7fba040ff990 2024-04-22T23:50:01.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.367+0000 7fba01d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba04100340 0x7fba0419eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:01.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.367+0000 7fba01d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba04100340 0x7fba0419eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60400/0 (socket says 172.21.15.67:60400) 2024-04-22T23:50:01.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.367+0000 7fba01d9b700 1 -- 172.21.15.67:0/132442974 learned_addr learned my addr 172.21.15.67:0/132442974 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:50:01.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.367+0000 7fba0259c700 1 --2- 172.21.15.67:0/132442974 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba040ff990 0x7fba0419e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:01.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.367+0000 7fba01d9b700 1 -- 172.21.15.67:0/132442974 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba040ff990 msgr2=0x7fba0419e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:01.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.367+0000 7fba01d9b700 1 --2- 172.21.15.67:0/132442974 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba040ff990 0x7fba0419e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:01.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.367+0000 7fba01d9b700 1 -- 172.21.15.67:0/132442974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f80095f0 con 0x7fba04100340 2024-04-22T23:50:01.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.367+0000 7fba01d9b700 1 --2- 172.21.15.67:0/132442974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba04100340 0x7fba0419eda0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8000c00 tx=0x7fb9f80056a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:50:01.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.368+0000 7fb9f37fe700 1 -- 172.21.15.67:0/132442974 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9f801d070 con 0x7fba04100340 2024-04-22T23:50:01.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.368+0000 7fb9f37fe700 1 -- 172.21.15.67:0/132442974 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb9f800f460 con 0x7fba04100340 2024-04-22T23:50:01.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.368+0000 7fba08dfa700 1 -- 172.21.15.67:0/132442974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba04198b60 con 0x7fba04100340 2024-04-22T23:50:01.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.368+0000 7fb9f37fe700 1 -- 172.21.15.67:0/132442974 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9f8021640 con 0x7fba04100340 2024-04-22T23:50:01.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.368+0000 7fba08dfa700 1 -- 172.21.15.67:0/132442974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba041990b0 con 0x7fba04100340 2024-04-22T23:50:01.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.369+0000 7fba08dfa700 1 -- 172.21.15.67:0/132442974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba0410a770 con 0x7fba04100340 2024-04-22T23:50:01.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.369+0000 7fb9f37fe700 1 -- 172.21.15.67:0/132442974 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb9f800fa60 con 0x7fba04100340 2024-04-22T23:50:01.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.370+0000 7fb9f37fe700 1 --2- 172.21.15.67:0/132442974 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb9ec06ecc0 0x7fb9ec071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:01.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.370+0000 7fb9f37fe700 1 -- 172.21.15.67:0/132442974 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fb9f8090120 con 0x7fba04100340 2024-04-22T23:50:01.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.370+0000 7fba0259c700 1 --2- 172.21.15.67:0/132442974 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb9ec06ecc0 0x7fb9ec071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:01.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.370+0000 7fba0259c700 1 --2- 172.21.15.67:0/132442974 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb9ec06ecc0 0x7fb9ec071180 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4005950 tx=0x7fb9f40058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:50:01.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.375+0000 7fb9f37fe700 1 -- 172.21.15.67:0/132442974 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb9f805c910 con 0x7fba04100340 2024-04-22T23:50:01.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.594+0000 7fba08dfa700 1 -- 172.21.15.67:0/132442974 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fba04199380 con 0x7fb9ec06ecc0 2024-04-22T23:50:01.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.605+0000 7fb9f37fe700 1 -- 172.21.15.67:0/132442974 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fba04199380 con 0x7fb9ec06ecc0 2024-04-22T23:50:01.599 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:50:01.599 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (23m) 91s ago 34m 24.4M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:50:01.599 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (35m) 91s ago 35m 10.0M - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:50:01.599 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (34m) 93s ago 34m 10.0M - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:50:01.599 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (21m) 91s ago 35m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:50:01.599 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (21m) 93s ago 34m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:50:01.599 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (23m) 91s ago 34m 81.2M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:50:01.599 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (17m) 91s ago 29m 19.4M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:50:01.599 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (17m) 91s ago 29m 20.3M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:50:01.600 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (16m) 93s ago 29m 20.1M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:50:01.600 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (97s) 93s ago 29m 14.0M - 19.0.0-3252-g43be0201 a0434699c1d7 46d3d7e86fee 2024-04-22T23:50:01.600 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (25m) 91s ago 36m 514M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:50:01.600 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (24m) 93s ago 34m 459M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:50:01.600 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (22m) 91s ago 36m 74.2M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:50:01.600 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (21m) 93s ago 33m 54.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:50:01.600 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (24m) 91s ago 35m 15.2M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:50:01.600 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (24m) 93s ago 34m 16.0M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:50:01.600 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (20m) 91s ago 33m 102M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:50:01.600 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (20m) 91s ago 32m 128M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:50:01.600 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (19m) 91s ago 32m 83.8M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:50:01.600 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (18m) 93s ago 31m 101M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:50:01.600 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (18m) 93s ago 31m 86.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:50:01.600 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (17m) 93s ago 31m 81.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:50:01.600 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (24m) 91s ago 34m 78.7M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:50:01.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.608+0000 7fba08dfa700 1 -- 172.21.15.67:0/132442974 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb9ec06ecc0 msgr2=0x7fb9ec071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:01.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.608+0000 7fba08dfa700 1 --2- 172.21.15.67:0/132442974 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb9ec06ecc0 0x7fb9ec071180 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4005950 tx=0x7fb9f40058e0 comp rx=0 tx=0).stop 2024-04-22T23:50:01.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.608+0000 7fba08dfa700 1 -- 172.21.15.67:0/132442974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba04100340 msgr2=0x7fba0419eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:01.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.608+0000 7fba08dfa700 1 --2- 172.21.15.67:0/132442974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba04100340 0x7fba0419eda0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8000c00 tx=0x7fb9f80056a0 comp rx=0 tx=0).stop 2024-04-22T23:50:01.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.608+0000 7fba08dfa700 1 -- 172.21.15.67:0/132442974 shutdown_connections 2024-04-22T23:50:01.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.608+0000 7fba08dfa700 1 --2- 172.21.15.67:0/132442974 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb9ec06ecc0 0x7fb9ec071180 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:01.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.608+0000 7fba08dfa700 1 --2- 172.21.15.67:0/132442974 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba040ff990 0x7fba0419e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:01.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.608+0000 7fba08dfa700 1 --2- 172.21.15.67:0/132442974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba04100340 0x7fba0419eda0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:01.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.608+0000 7fba08dfa700 1 -- 172.21.15.67:0/132442974 >> 172.21.15.67:0/132442974 conn(0x7fba040fb650 msgr2=0x7fba040fcca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:50:01.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.608+0000 7fba08dfa700 1 -- 172.21.15.67:0/132442974 shutdown_connections 2024-04-22T23:50:01.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.608+0000 7fba08dfa700 1 -- 172.21.15.67:0/132442974 wait complete. 2024-04-22T23:50:01.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.714+0000 7efca2794700 1 -- 172.21.15.67:0/2333574284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc9c1027e0 msgr2=0x7efc9c10acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:01.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.714+0000 7efca2794700 1 --2- 172.21.15.67:0/2333574284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc9c1027e0 0x7efc9c10acd0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7efc90009940 tx=0x7efc90009c50 comp rx=0 tx=0).stop 2024-04-22T23:50:01.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.714+0000 7efca2794700 1 -- 172.21.15.67:0/2333574284 shutdown_connections 2024-04-22T23:50:01.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.714+0000 7efca2794700 1 --2- 172.21.15.67:0/2333574284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc9c1027e0 0x7efc9c10acd0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:01.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.714+0000 7efca2794700 1 --2- 172.21.15.67:0/2333574284 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc9c101ec0 0x7efc9c1022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:01.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.714+0000 7efca2794700 1 -- 172.21.15.67:0/2333574284 >> 172.21.15.67:0/2333574284 conn(0x7efc9c0fb650 msgr2=0x7efc9c0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:50:01.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.714+0000 7efca2794700 1 -- 172.21.15.67:0/2333574284 shutdown_connections 2024-04-22T23:50:01.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.714+0000 7efca2794700 1 -- 172.21.15.67:0/2333574284 wait complete. 2024-04-22T23:50:01.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.714+0000 7efca2794700 1 Processor -- start 2024-04-22T23:50:01.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.714+0000 7efca2794700 1 -- start start 2024-04-22T23:50:01.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.715+0000 7efca2794700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc9c101ec0 0x7efc9c0ffb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:01.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.715+0000 7efca2794700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc9c1027e0 0x7efc9c100060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:01.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.715+0000 7efca2794700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c1005a0 con 0x7efc9c101ec0 2024-04-22T23:50:01.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.715+0000 7efca2794700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c1006e0 con 0x7efc9c1027e0 2024-04-22T23:50:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.715+0000 7efc9bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc9c101ec0 0x7efc9c0ffb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.715+0000 7efc9bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc9c101ec0 0x7efc9c0ffb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60408/0 (socket says 172.21.15.67:60408) 2024-04-22T23:50:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.715+0000 7efc9bfff700 1 -- 172.21.15.67:0/2993128656 learned_addr learned my addr 172.21.15.67:0/2993128656 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:50:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.715+0000 7efc9b7fe700 1 --2- 172.21.15.67:0/2993128656 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc9c1027e0 0x7efc9c100060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.715+0000 7efc9bfff700 1 -- 172.21.15.67:0/2993128656 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc9c1027e0 msgr2=0x7efc9c100060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.715+0000 7efc9bfff700 1 --2- 172.21.15.67:0/2993128656 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc9c1027e0 0x7efc9c100060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.715+0000 7efc9bfff700 1 -- 172.21.15.67:0/2993128656 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc900095f0 con 0x7efc9c101ec0 2024-04-22T23:50:01.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.715+0000 7efc9bfff700 1 --2- 172.21.15.67:0/2993128656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc9c101ec0 0x7efc9c0ffb20 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7efc8c00e920 tx=0x7efc8c00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:50:01.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.716+0000 7efc997fa700 1 -- 172.21.15.67:0/2993128656 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc8c00c9d0 con 0x7efc9c101ec0 2024-04-22T23:50:01.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.716+0000 7efc997fa700 1 -- 172.21.15.67:0/2993128656 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efc8c010430 con 0x7efc9c101ec0 2024-04-22T23:50:01.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.716+0000 7efca2794700 1 -- 172.21.15.67:0/2993128656 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc9c1a2a60 con 0x7efc9c101ec0 2024-04-22T23:50:01.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.716+0000 7efc997fa700 1 -- 172.21.15.67:0/2993128656 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc8c018630 con 0x7efc9c101ec0 2024-04-22T23:50:01.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.716+0000 7efca2794700 1 -- 172.21.15.67:0/2993128656 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc9c1a2e10 con 0x7efc9c101ec0 2024-04-22T23:50:01.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.717+0000 7efca2794700 1 -- 172.21.15.67:0/2993128656 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc9c108360 con 0x7efc9c101ec0 2024-04-22T23:50:01.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.717+0000 7efc997fa700 1 -- 172.21.15.67:0/2993128656 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7efc8c0105a0 con 0x7efc9c101ec0 2024-04-22T23:50:01.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.718+0000 7efc997fa700 1 --2- 172.21.15.67:0/2993128656 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efc8406ecc0 0x7efc84071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:01.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.718+0000 7efc997fa700 1 -- 172.21.15.67:0/2993128656 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7efc8c014070 con 0x7efc9c101ec0 2024-04-22T23:50:01.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.718+0000 7efc9b7fe700 1 --2- 172.21.15.67:0/2993128656 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efc8406ecc0 0x7efc84071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:01.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.718+0000 7efc9b7fe700 1 --2- 172.21.15.67:0/2993128656 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efc8406ecc0 0x7efc84071180 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7efc90006010 tx=0x7efc9000b5a0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:50:01.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:01.722+0000 7efc997fa700 1 -- 172.21.15.67:0/2993128656 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7efc8c054150 con 0x7efc9c101ec0 2024-04-22T23:50:02.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.020+0000 7efca2794700 1 -- 172.21.15.67:0/2993128656 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7efc9c101190 con 0x7efc9c101ec0 2024-04-22T23:50:02.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.020+0000 7efc997fa700 1 -- 172.21.15.67:0/2993128656 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7efc8c057e80 con 0x7efc9c101ec0 2024-04-22T23:50:02.012 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:50:02.012 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:50:02.012 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:50:02.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:50:02.012 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:50:02.012 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:50:02.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:50:02.012 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:50:02.012 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:50:02.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:50:02.013 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:50:02.013 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 4 2024-04-22T23:50:02.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:50:02.013 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:50:02.013 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 14 2024-04-22T23:50:02.013 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:50:02.013 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:50:02.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.023+0000 7efca2794700 1 -- 172.21.15.67:0/2993128656 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efc8406ecc0 msgr2=0x7efc84071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:02.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.023+0000 7efca2794700 1 --2- 172.21.15.67:0/2993128656 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efc8406ecc0 0x7efc84071180 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7efc90006010 tx=0x7efc9000b5a0 comp rx=0 tx=0).stop 2024-04-22T23:50:02.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.023+0000 7efca2794700 1 -- 172.21.15.67:0/2993128656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc9c101ec0 msgr2=0x7efc9c0ffb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:02.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.023+0000 7efca2794700 1 --2- 172.21.15.67:0/2993128656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc9c101ec0 0x7efc9c0ffb20 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7efc8c00e920 tx=0x7efc8c00ec30 comp rx=0 tx=0).stop 2024-04-22T23:50:02.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.023+0000 7efca2794700 1 -- 172.21.15.67:0/2993128656 shutdown_connections 2024-04-22T23:50:02.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.023+0000 7efca2794700 1 --2- 172.21.15.67:0/2993128656 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efc8406ecc0 0x7efc84071180 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:02.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.023+0000 7efca2794700 1 --2- 172.21.15.67:0/2993128656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc9c101ec0 0x7efc9c0ffb20 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:02.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.023+0000 7efca2794700 1 --2- 172.21.15.67:0/2993128656 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efc9c1027e0 0x7efc9c100060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:02.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.023+0000 7efca2794700 1 -- 172.21.15.67:0/2993128656 >> 172.21.15.67:0/2993128656 conn(0x7efc9c0fb650 msgr2=0x7efc9c1054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:50:02.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.023+0000 7efca2794700 1 -- 172.21.15.67:0/2993128656 shutdown_connections 2024-04-22T23:50:02.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.023+0000 7efca2794700 1 -- 172.21.15.67:0/2993128656 wait complete. 2024-04-22T23:50:02.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.130+0000 7fe8ad52d700 1 -- 172.21.15.67:0/4089699975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8a80ffc20 msgr2=0x7fe8a8100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:02.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.130+0000 7fe8ad52d700 1 --2- 172.21.15.67:0/4089699975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8a80ffc20 0x7fe8a8100080 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fe890009910 tx=0x7fe890009c20 comp rx=0 tx=0).stop 2024-04-22T23:50:02.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.130+0000 7fe8ad52d700 1 -- 172.21.15.67:0/4089699975 shutdown_connections 2024-04-22T23:50:02.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.130+0000 7fe8ad52d700 1 --2- 172.21.15.67:0/4089699975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8a80ffc20 0x7fe8a8100080 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:02.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.130+0000 7fe8ad52d700 1 --2- 172.21.15.67:0/4089699975 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8a8105bd0 0x7fe8a8105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:02.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.130+0000 7fe8ad52d700 1 -- 172.21.15.67:0/4089699975 >> 172.21.15.67:0/4089699975 conn(0x7fe8a8075670 msgr2=0x7fe8a8075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:50:02.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.130+0000 7fe8ad52d700 1 -- 172.21.15.67:0/4089699975 shutdown_connections 2024-04-22T23:50:02.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.130+0000 7fe8ad52d700 1 -- 172.21.15.67:0/4089699975 wait complete. 2024-04-22T23:50:02.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.130+0000 7fe8ad52d700 1 Processor -- start 2024-04-22T23:50:02.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.131+0000 7fe8ad52d700 1 -- start start 2024-04-22T23:50:02.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.131+0000 7fe8ad52d700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8a80ffc20 0x7fe8a819c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:02.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.131+0000 7fe8ad52d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8a8105bd0 0x7fe8a819cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:02.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.131+0000 7fe8ad52d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a819d230 con 0x7fe8a8105bd0 2024-04-22T23:50:02.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.131+0000 7fe8ad52d700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a81966e0 con 0x7fe8a80ffc20 2024-04-22T23:50:02.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.131+0000 7fe8a67fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8a8105bd0 0x7fe8a819cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:02.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.131+0000 7fe8a67fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8a8105bd0 0x7fe8a819cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60416/0 (socket says 172.21.15.67:60416) 2024-04-22T23:50:02.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.131+0000 7fe8a67fc700 1 -- 172.21.15.67:0/4202227149 learned_addr learned my addr 172.21.15.67:0/4202227149 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:50:02.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.131+0000 7fe8a6ffd700 1 --2- 172.21.15.67:0/4202227149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8a80ffc20 0x7fe8a819c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:02.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.131+0000 7fe8a67fc700 1 -- 172.21.15.67:0/4202227149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8a80ffc20 msgr2=0x7fe8a819c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:02.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.131+0000 7fe8a67fc700 1 --2- 172.21.15.67:0/4202227149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8a80ffc20 0x7fe8a819c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:02.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.131+0000 7fe8a67fc700 1 -- 172.21.15.67:0/4202227149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8900095f0 con 0x7fe8a8105bd0 2024-04-22T23:50:02.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.132+0000 7fe8a67fc700 1 --2- 172.21.15.67:0/4202227149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8a8105bd0 0x7fe8a819cba0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fe8900092e0 tx=0x7fe89000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:50:02.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.132+0000 7fe89ffff700 1 -- 172.21.15.67:0/4202227149 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe89001d070 con 0x7fe8a8105bd0 2024-04-22T23:50:02.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.132+0000 7fe89ffff700 1 -- 172.21.15.67:0/4202227149 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe890005e00 con 0x7fe8a8105bd0 2024-04-22T23:50:02.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.132+0000 7fe89ffff700 1 -- 172.21.15.67:0/4202227149 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe890017400 con 0x7fe8a8105bd0 2024-04-22T23:50:02.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.132+0000 7fe8ad52d700 1 -- 172.21.15.67:0/4202227149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8a8196960 con 0x7fe8a8105bd0 2024-04-22T23:50:02.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.132+0000 7fe8ad52d700 1 -- 172.21.15.67:0/4202227149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8a8196e50 con 0x7fe8a8105bd0 2024-04-22T23:50:02.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.133+0000 7fe8ad52d700 1 -- 172.21.15.67:0/4202227149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8a804f2e0 con 0x7fe8a8105bd0 2024-04-22T23:50:02.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.133+0000 7fe89ffff700 1 -- 172.21.15.67:0/4202227149 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe89000f6f0 con 0x7fe8a8105bd0 2024-04-22T23:50:02.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.134+0000 7fe89ffff700 1 --2- 172.21.15.67:0/4202227149 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe89406ecc0 0x7fe894071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:02.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.134+0000 7fe8a6ffd700 1 --2- 172.21.15.67:0/4202227149 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe89406ecc0 0x7fe894071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:02.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.134+0000 7fe89ffff700 1 -- 172.21.15.67:0/4202227149 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fe890090480 con 0x7fe8a8105bd0 2024-04-22T23:50:02.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.134+0000 7fe8a6ffd700 1 --2- 172.21.15.67:0/4202227149 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe89406ecc0 0x7fe894071180 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe8a0009550 tx=0x7fe8a00091b0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:50:02.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.139+0000 7fe89ffff700 1 -- 172.21.15.67:0/4202227149 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe89005cbf0 con 0x7fe8a8105bd0 2024-04-22T23:50:02.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:02 smithi067 ceph-mon[112438]: from='client.35300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:50:02.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:02 smithi067 ceph-mon[112438]: pgmap v711: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:02.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.395+0000 7fe8ad52d700 1 -- 172.21.15.67:0/4202227149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe8a8197bf0 con 0x7fe8a8105bd0 2024-04-22T23:50:02.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.395+0000 7fe89ffff700 1 -- 172.21.15.67:0/4202227149 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1872 (secure 0 0 0) 0x7fe89005c780 con 0x7fe8a8105bd0 2024-04-22T23:50:02.388 INFO:teuthology.orchestra.run.smithi067.stdout:e33 2024-04-22T23:50:02.388 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:50:02.388 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:50:02.388 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:50:02.388 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:50:02.388 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:50:02.388 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:50:02.388 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:50:02.388 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:50:02.388 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:50:02.388 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:50:02.388 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:50:02.389 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:50:02.389 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:50:02.389 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:50:02.389 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:50:02.389 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:50:02.389 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:50:02.389 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:50:02.389 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:50:02.389 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:50:02.389 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:50:02.389 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:50:02.389 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:50:02.389 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:50:02.389 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:50:02.389 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:50:02.390 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:50:02.390 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:50:02.390 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:50:02.390 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:50:02.390 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:50:02.390 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:50:02.390 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:50:02.390 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:50:02.390 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:50:02.390 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:50:02.390 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:50:02.390 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:50:02.390 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:50:02.390 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:50:02.391 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44851} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1849648622,v1:172.21.15.134:6827/1849648622] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:50:02.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.397+0000 7fe8ad52d700 1 -- 172.21.15.67:0/4202227149 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe89406ecc0 msgr2=0x7fe894071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:02.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.397+0000 7fe8ad52d700 1 --2- 172.21.15.67:0/4202227149 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe89406ecc0 0x7fe894071180 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe8a0009550 tx=0x7fe8a00091b0 comp rx=0 tx=0).stop 2024-04-22T23:50:02.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.397+0000 7fe8ad52d700 1 -- 172.21.15.67:0/4202227149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8a8105bd0 msgr2=0x7fe8a819cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:02.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.397+0000 7fe8ad52d700 1 --2- 172.21.15.67:0/4202227149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8a8105bd0 0x7fe8a819cba0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fe8900092e0 tx=0x7fe89000bd70 comp rx=0 tx=0).stop 2024-04-22T23:50:02.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.397+0000 7fe8ad52d700 1 -- 172.21.15.67:0/4202227149 shutdown_connections 2024-04-22T23:50:02.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.397+0000 7fe8ad52d700 1 --2- 172.21.15.67:0/4202227149 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe89406ecc0 0x7fe894071180 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:02.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.397+0000 7fe8ad52d700 1 --2- 172.21.15.67:0/4202227149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe8a80ffc20 0x7fe8a819c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:02.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.397+0000 7fe8ad52d700 1 --2- 172.21.15.67:0/4202227149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8a8105bd0 0x7fe8a819cba0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:02.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.397+0000 7fe8ad52d700 1 -- 172.21.15.67:0/4202227149 >> 172.21.15.67:0/4202227149 conn(0x7fe8a8075670 msgr2=0x7fe8a80686a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:50:02.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.397+0000 7fe8ad52d700 1 -- 172.21.15.67:0/4202227149 shutdown_connections 2024-04-22T23:50:02.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.397+0000 7fe8ad52d700 1 -- 172.21.15.67:0/4202227149 wait complete. 2024-04-22T23:50:02.393 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 33 2024-04-22T23:50:02.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:02 smithi134 ceph-mon[85630]: from='client.35300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:50:02.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:02 smithi134 ceph-mon[85630]: pgmap v711: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:02.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.505+0000 7f635cb10700 1 -- 172.21.15.67:0/52956162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6358101de0 msgr2=0x7f63581021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:02.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.505+0000 7f635cb10700 1 --2- 172.21.15.67:0/52956162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6358101de0 0x7f63581021c0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f635000b0a0 tx=0x7f635000b3b0 comp rx=0 tx=0).stop 2024-04-22T23:50:02.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.505+0000 7f635cb10700 1 -- 172.21.15.67:0/52956162 shutdown_connections 2024-04-22T23:50:02.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.505+0000 7f635cb10700 1 --2- 172.21.15.67:0/52956162 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6358102700 0x7f635810abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:02.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.505+0000 7f635cb10700 1 --2- 172.21.15.67:0/52956162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6358101de0 0x7f63581021c0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:02.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.505+0000 7f635cb10700 1 -- 172.21.15.67:0/52956162 >> 172.21.15.67:0/52956162 conn(0x7f63580fb610 msgr2=0x7f63580fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:50:02.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.505+0000 7f635cb10700 1 -- 172.21.15.67:0/52956162 shutdown_connections 2024-04-22T23:50:02.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.505+0000 7f635cb10700 1 -- 172.21.15.67:0/52956162 wait complete. 2024-04-22T23:50:02.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.506+0000 7f635cb10700 1 Processor -- start 2024-04-22T23:50:02.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.506+0000 7f635cb10700 1 -- start start 2024-04-22T23:50:02.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.506+0000 7f635cb10700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6358101de0 0x7f635819c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:02.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.506+0000 7f635cb10700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6358102700 0x7f635819cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:02.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.506+0000 7f635cb10700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635819d2b0 con 0x7f6358101de0 2024-04-22T23:50:02.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.506+0000 7f635cb10700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6358196760 con 0x7f6358102700 2024-04-22T23:50:02.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.506+0000 7f635659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6358101de0 0x7f635819c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:02.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.506+0000 7f635659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6358101de0 0x7f635819c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60448/0 (socket says 172.21.15.67:60448) 2024-04-22T23:50:02.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.506+0000 7f635659c700 1 -- 172.21.15.67:0/725390251 learned_addr learned my addr 172.21.15.67:0/725390251 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:50:02.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.506+0000 7f6355d9b700 1 --2- 172.21.15.67:0/725390251 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6358102700 0x7f635819cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:02.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.507+0000 7f635659c700 1 -- 172.21.15.67:0/725390251 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6358102700 msgr2=0x7f635819cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:02.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.507+0000 7f635659c700 1 --2- 172.21.15.67:0/725390251 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6358102700 0x7f635819cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:02.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.507+0000 7f635659c700 1 -- 172.21.15.67:0/725390251 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6350009d00 con 0x7f6358101de0 2024-04-22T23:50:02.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.507+0000 7f635659c700 1 --2- 172.21.15.67:0/725390251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6358101de0 0x7f635819c6e0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f635000b070 tx=0x7f6350003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:50:02.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.507+0000 7f634f7fe700 1 -- 172.21.15.67:0/725390251 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f635000e040 con 0x7f6358101de0 2024-04-22T23:50:02.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.507+0000 7f634f7fe700 1 -- 172.21.15.67:0/725390251 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f63500096e0 con 0x7f6358101de0 2024-04-22T23:50:02.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.507+0000 7f635cb10700 1 -- 172.21.15.67:0/725390251 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63581969e0 con 0x7f6358101de0 2024-04-22T23:50:02.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.507+0000 7f634f7fe700 1 -- 172.21.15.67:0/725390251 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f635001d6b0 con 0x7f6358101de0 2024-04-22T23:50:02.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.507+0000 7f635cb10700 1 -- 172.21.15.67:0/725390251 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6358196ed0 con 0x7f6358101de0 2024-04-22T23:50:02.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.508+0000 7f635cb10700 1 -- 172.21.15.67:0/725390251 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63581082f0 con 0x7f6358101de0 2024-04-22T23:50:02.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.508+0000 7f634f7fe700 1 -- 172.21.15.67:0/725390251 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6350004590 con 0x7f6358101de0 2024-04-22T23:50:02.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.509+0000 7f634f7fe700 1 --2- 172.21.15.67:0/725390251 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f634406ecc0 0x7f6344071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:02.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.509+0000 7f634f7fe700 1 -- 172.21.15.67:0/725390251 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f63500906b0 con 0x7f6358101de0 2024-04-22T23:50:02.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.509+0000 7f6355d9b700 1 --2- 172.21.15.67:0/725390251 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f634406ecc0 0x7f6344071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:02.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.509+0000 7f6355d9b700 1 --2- 172.21.15.67:0/725390251 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f634406ecc0 0x7f6344071180 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f6340005950 tx=0x7f63400058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:50:02.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.514+0000 7f634f7fe700 1 -- 172.21.15.67:0/725390251 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f635005ce20 con 0x7f6358101de0 2024-04-22T23:50:02.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.745+0000 7f635cb10700 1 -- 172.21.15.67:0/725390251 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6358197d80 con 0x7f634406ecc0 2024-04-22T23:50:02.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.746+0000 7f634f7fe700 1 -- 172.21.15.67:0/725390251 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6358197d80 con 0x7f634406ecc0 2024-04-22T23:50:02.740 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:50:02.740 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:50:02.740 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:50:02.740 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:50:02.740 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:50:02.740 INFO:teuthology.orchestra.run.smithi067.stdout: "mds", 2024-04-22T23:50:02.740 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:50:02.740 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:50:02.740 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:50:02.740 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:50:02.740 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:50:02.740 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T23:50:02.741 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:50:02.741 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:50:02.741 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:50:02.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.749+0000 7f635cb10700 1 -- 172.21.15.67:0/725390251 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f634406ecc0 msgr2=0x7f6344071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:02.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.749+0000 7f635cb10700 1 --2- 172.21.15.67:0/725390251 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f634406ecc0 0x7f6344071180 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f6340005950 tx=0x7f63400058e0 comp rx=0 tx=0).stop 2024-04-22T23:50:02.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.749+0000 7f635cb10700 1 -- 172.21.15.67:0/725390251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6358101de0 msgr2=0x7f635819c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:02.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.749+0000 7f635cb10700 1 --2- 172.21.15.67:0/725390251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6358101de0 0x7f635819c6e0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f635000b070 tx=0x7f6350003ce0 comp rx=0 tx=0).stop 2024-04-22T23:50:02.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.749+0000 7f635cb10700 1 -- 172.21.15.67:0/725390251 shutdown_connections 2024-04-22T23:50:02.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.749+0000 7f635cb10700 1 --2- 172.21.15.67:0/725390251 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f634406ecc0 0x7f6344071180 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:02.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.749+0000 7f635cb10700 1 --2- 172.21.15.67:0/725390251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6358101de0 0x7f635819c6e0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:02.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.749+0000 7f635cb10700 1 --2- 172.21.15.67:0/725390251 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6358102700 0x7f635819cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:02.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.749+0000 7f635cb10700 1 -- 172.21.15.67:0/725390251 >> 172.21.15.67:0/725390251 conn(0x7f63580fb610 msgr2=0x7f6358105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:50:02.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.749+0000 7f635cb10700 1 -- 172.21.15.67:0/725390251 shutdown_connections 2024-04-22T23:50:02.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.749+0000 7f635cb10700 1 -- 172.21.15.67:0/725390251 wait complete. 2024-04-22T23:50:02.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.855+0000 7fac1fd35700 1 -- 172.21.15.67:0/1468962693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac180ffda0 msgr2=0x7fac18100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:02.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.855+0000 7fac1fd35700 1 --2- 172.21.15.67:0/1468962693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac180ffda0 0x7fac18100180 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fac14009910 tx=0x7fac14009c20 comp rx=0 tx=0).stop 2024-04-22T23:50:02.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.856+0000 7fac1fd35700 1 -- 172.21.15.67:0/1468962693 shutdown_connections 2024-04-22T23:50:02.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.856+0000 7fac1fd35700 1 --2- 172.21.15.67:0/1468962693 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac18100750 0x7fac181047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:02.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.856+0000 7fac1fd35700 1 --2- 172.21.15.67:0/1468962693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac180ffda0 0x7fac18100180 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:02.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.856+0000 7fac1fd35700 1 -- 172.21.15.67:0/1468962693 >> 172.21.15.67:0/1468962693 conn(0x7fac180fb650 msgr2=0x7fac180fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:50:02.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.856+0000 7fac1fd35700 1 -- 172.21.15.67:0/1468962693 shutdown_connections 2024-04-22T23:50:02.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.856+0000 7fac1fd35700 1 -- 172.21.15.67:0/1468962693 wait complete. 2024-04-22T23:50:02.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.856+0000 7fac1fd35700 1 Processor -- start 2024-04-22T23:50:02.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.857+0000 7fac1fd35700 1 -- start start 2024-04-22T23:50:02.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.857+0000 7fac1fd35700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac180ffda0 0x7fac181954e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:02.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.857+0000 7fac1fd35700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac18100750 0x7fac18195a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:02.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.857+0000 7fac1fd35700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac18195f60 con 0x7fac180ffda0 2024-04-22T23:50:02.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.857+0000 7fac1fd35700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac181960d0 con 0x7fac18100750 2024-04-22T23:50:02.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.857+0000 7fac1dad1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac180ffda0 0x7fac181954e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:02.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.857+0000 7fac1dad1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac180ffda0 0x7fac181954e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60470/0 (socket says 172.21.15.67:60470) 2024-04-22T23:50:02.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.857+0000 7fac1dad1700 1 -- 172.21.15.67:0/2392038000 learned_addr learned my addr 172.21.15.67:0/2392038000 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:50:02.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.857+0000 7fac1d2d0700 1 --2- 172.21.15.67:0/2392038000 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac18100750 0x7fac18195a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:02.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.857+0000 7fac1dad1700 1 -- 172.21.15.67:0/2392038000 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac18100750 msgr2=0x7fac18195a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:02.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.857+0000 7fac1dad1700 1 --2- 172.21.15.67:0/2392038000 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac18100750 0x7fac18195a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:02.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.857+0000 7fac1dad1700 1 -- 172.21.15.67:0/2392038000 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac140095f0 con 0x7fac180ffda0 2024-04-22T23:50:02.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.857+0000 7fac1d2d0700 1 --2- 172.21.15.67:0/2392038000 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac18100750 0x7fac18195a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:50:02.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.857+0000 7fac1dad1700 1 --2- 172.21.15.67:0/2392038000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac180ffda0 0x7fac181954e0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fac1400b5c0 tx=0x7fac1400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:50:02.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.858+0000 7fac0effd700 1 -- 172.21.15.67:0/2392038000 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac1401d070 con 0x7fac180ffda0 2024-04-22T23:50:02.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.858+0000 7fac0effd700 1 -- 172.21.15.67:0/2392038000 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fac14005e00 con 0x7fac180ffda0 2024-04-22T23:50:02.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.858+0000 7fac1fd35700 1 -- 172.21.15.67:0/2392038000 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac18196350 con 0x7fac180ffda0 2024-04-22T23:50:02.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.858+0000 7fac0effd700 1 -- 172.21.15.67:0/2392038000 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac14017400 con 0x7fac180ffda0 2024-04-22T23:50:02.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.858+0000 7fac1fd35700 1 -- 172.21.15.67:0/2392038000 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac181a2dd0 con 0x7fac180ffda0 2024-04-22T23:50:02.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.859+0000 7fac1fd35700 1 -- 172.21.15.67:0/2392038000 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac18108140 con 0x7fac180ffda0 2024-04-22T23:50:02.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.860+0000 7fac0effd700 1 -- 172.21.15.67:0/2392038000 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fac1400f460 con 0x7fac180ffda0 2024-04-22T23:50:02.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.860+0000 7fac0effd700 1 --2- 172.21.15.67:0/2392038000 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fac0406ecc0 0x7fac04071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:02.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.860+0000 7fac0effd700 1 -- 172.21.15.67:0/2392038000 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fac14090270 con 0x7fac180ffda0 2024-04-22T23:50:02.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.860+0000 7fac1d2d0700 1 --2- 172.21.15.67:0/2392038000 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fac0406ecc0 0x7fac04071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:02.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.861+0000 7fac1d2d0700 1 --2- 172.21.15.67:0/2392038000 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fac0406ecc0 0x7fac04071180 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fac08005d90 tx=0x7fac08005d00 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:50:02.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:02.865+0000 7fac0effd700 1 -- 172.21.15.67:0/2392038000 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fac1405ca60 con 0x7fac180ffda0 2024-04-22T23:50:03.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:03.167+0000 7fac1fd35700 1 -- 172.21.15.67:0/2392038000 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fac18196350 con 0x7fac180ffda0 2024-04-22T23:50:03.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:03.167+0000 7fac0effd700 1 -- 172.21.15.67:0/2392038000 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4505 (secure 0 0 0) 0x7fac18196350 con 0x7fac180ffda0 2024-04-22T23:50:03.160 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:50:03.160 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T23:50:03.160 INFO:teuthology.orchestra.run.smithi067.stdout: host smithi134 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-22T23:50:03.160 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T23:50:03.160 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:50:03.160 INFO:teuthology.orchestra.run.smithi067.stdout:Traceback (most recent call last): 2024-04-22T23:50:03.161 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T23:50:03.161 INFO:teuthology.orchestra.run.smithi067.stdout: return _run_code(code, main_globals, None, 2024-04-22T23:50:03.161 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T23:50:03.161 INFO:teuthology.orchestra.run.smithi067.stdout: exec(code, run_globals) 2024-04-22T23:50:03.161 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5642, in 2024-04-22T23:50:03.161 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5630, in main 2024-04-22T23:50:03.161 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 408, in _infer_config 2024-04-22T23:50:03.161 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 323, in _infer_fsid 2024-04-22T23:50:03.161 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 436, in _infer_image 2024-04-22T23:50:03.161 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 310, in _validate_fsid 2024-04-22T23:50:03.161 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 3380, in command_ceph_volume 2024-04-22T23:50:03.161 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T23:50:03.161 INFO:teuthology.orchestra.run.smithi067.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:50:03.161 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:50:03.162 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:50:03.162 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:50:03.162 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:50:03.162 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:50:03.162 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:50:03.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:03.169+0000 7fac1fd35700 1 -- 172.21.15.67:0/2392038000 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fac0406ecc0 msgr2=0x7fac04071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:03.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:03.169+0000 7fac1fd35700 1 --2- 172.21.15.67:0/2392038000 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fac0406ecc0 0x7fac04071180 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fac08005d90 tx=0x7fac08005d00 comp rx=0 tx=0).stop 2024-04-22T23:50:03.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:03.169+0000 7fac1fd35700 1 -- 172.21.15.67:0/2392038000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac180ffda0 msgr2=0x7fac181954e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:03.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:03.169+0000 7fac1fd35700 1 --2- 172.21.15.67:0/2392038000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac180ffda0 0x7fac181954e0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fac1400b5c0 tx=0x7fac1400be20 comp rx=0 tx=0).stop 2024-04-22T23:50:03.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:03.169+0000 7fac1fd35700 1 -- 172.21.15.67:0/2392038000 shutdown_connections 2024-04-22T23:50:03.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:03.169+0000 7fac1fd35700 1 --2- 172.21.15.67:0/2392038000 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fac0406ecc0 0x7fac04071180 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:03.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:03.169+0000 7fac1fd35700 1 --2- 172.21.15.67:0/2392038000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac180ffda0 0x7fac181954e0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:03.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:03.169+0000 7fac1fd35700 1 --2- 172.21.15.67:0/2392038000 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fac18100750 0x7fac18195a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:03.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:03.169+0000 7fac1fd35700 1 -- 172.21.15.67:0/2392038000 >> 172.21.15.67:0/2392038000 conn(0x7fac180fb650 msgr2=0x7fac180fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:50:03.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:03.169+0000 7fac1fd35700 1 -- 172.21.15.67:0/2392038000 shutdown_connections 2024-04-22T23:50:03.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:03.169+0000 7fac1fd35700 1 -- 172.21.15.67:0/2392038000 wait complete. 2024-04-22T23:50:03.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:03 smithi134 ceph-mon[85630]: from='client.35304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:50:03.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:03 smithi134 ceph-mon[85630]: from='client.35308 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:50:03.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:03 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2993128656' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:50:03.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:03 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/4202227149' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:50:03.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:03 smithi067 ceph-mon[112438]: from='client.35304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:50:03.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:03 smithi067 ceph-mon[112438]: from='client.35308 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:50:03.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:03 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2993128656' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:50:03.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:03 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/4202227149' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:50:04.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:04 smithi067 ceph-mon[112438]: from='client.35320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:50:04.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:04 smithi067 ceph-mon[112438]: pgmap v712: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:04.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:04 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2392038000' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:50:04.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:04 smithi134 ceph-mon[85630]: from='client.35320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:50:04.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:04 smithi134 ceph-mon[85630]: pgmap v712: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:04.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:04 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2392038000' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:50:06.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:06 smithi067 ceph-mon[112438]: pgmap v713: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:06.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:06 smithi134 ceph-mon[85630]: pgmap v713: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:08.235 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:08 smithi134 ceph-mon[85630]: pgmap v714: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:08.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:08 smithi067 ceph-mon[112438]: pgmap v714: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:10.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:10 smithi067 ceph-mon[112438]: pgmap v715: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:10.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:10 smithi134 ceph-mon[85630]: pgmap v715: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:12.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:12 smithi067 ceph-mon[112438]: pgmap v716: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:12.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:12 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:50:12.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:12 smithi134 ceph-mon[85630]: pgmap v716: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:12.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:12 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:50:14.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:14 smithi067 ceph-mon[112438]: pgmap v717: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:14.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:14 smithi134 ceph-mon[85630]: pgmap v717: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:16.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:16 smithi067 ceph-mon[112438]: pgmap v718: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:16.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:16 smithi134 ceph-mon[85630]: pgmap v718: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:18.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:18 smithi067 ceph-mon[112438]: pgmap v719: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:18.286 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:18 smithi134 ceph-mon[85630]: pgmap v719: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:20.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:20 smithi067 ceph-mon[112438]: pgmap v720: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:20.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:20 smithi134 ceph-mon[85630]: pgmap v720: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:22.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:22 smithi067 ceph-mon[112438]: pgmap v721: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:22.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:22 smithi134 ceph-mon[85630]: pgmap v721: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:24.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:24 smithi067 ceph-mon[112438]: pgmap v722: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:24.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:24 smithi134 ceph-mon[85630]: pgmap v722: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:26.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:26 smithi134 ceph-mon[85630]: pgmap v723: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:26.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:26 smithi067 ceph-mon[112438]: pgmap v723: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:27.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:27 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:50:27.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:27 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:50:28.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:28 smithi134 ceph-mon[85630]: pgmap v724: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:28.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:28 smithi067 ceph-mon[112438]: pgmap v724: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:30.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:30 smithi134 ceph-mon[85630]: pgmap v725: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:30.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:30 smithi067 ceph-mon[112438]: pgmap v725: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:31.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:31 smithi134 ceph-mon[85630]: pgmap v726: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:31.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:31 smithi067 ceph-mon[112438]: pgmap v726: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:33.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.293+0000 7f3bafae5700 1 -- 172.21.15.67:0/380448307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ba80ff340 msgr2=0x7f3ba80ff7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:33.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.293+0000 7f3bafae5700 1 --2- 172.21.15.67:0/380448307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ba80ff340 0x7f3ba80ff7a0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f3b98009910 tx=0x7f3b98009c20 comp rx=0 tx=0).stop 2024-04-22T23:50:33.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.293+0000 7f3bafae5700 1 -- 172.21.15.67:0/380448307 shutdown_connections 2024-04-22T23:50:33.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.293+0000 7f3bafae5700 1 --2- 172.21.15.67:0/380448307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ba80ff340 0x7f3ba80ff7a0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:33.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.293+0000 7f3bafae5700 1 --2- 172.21.15.67:0/380448307 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3ba8105350 0x7f3ba8105730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:33.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.293+0000 7f3bafae5700 1 -- 172.21.15.67:0/380448307 >> 172.21.15.67:0/380448307 conn(0x7f3ba8074e30 msgr2=0x7f3ba8075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:50:33.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.293+0000 7f3bafae5700 1 -- 172.21.15.67:0/380448307 shutdown_connections 2024-04-22T23:50:33.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.293+0000 7f3bafae5700 1 -- 172.21.15.67:0/380448307 wait complete. 2024-04-22T23:50:33.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.294+0000 7f3bafae5700 1 Processor -- start 2024-04-22T23:50:33.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.294+0000 7f3bafae5700 1 -- start start 2024-04-22T23:50:33.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.294+0000 7f3bafae5700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3ba80ff340 0x7f3ba8100fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:33.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.294+0000 7f3bafae5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ba8105350 0x7f3ba8101500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:33.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.294+0000 7f3bafae5700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba8101a40 con 0x7f3ba8105350 2024-04-22T23:50:33.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.294+0000 7f3bafae5700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba8101b80 con 0x7f3ba80ff340 2024-04-22T23:50:33.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.294+0000 7f3bad080700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ba8105350 0x7f3ba8101500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:33.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.294+0000 7f3bad080700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ba8105350 0x7f3ba8101500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49762/0 (socket says 172.21.15.67:49762) 2024-04-22T23:50:33.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.294+0000 7f3bad080700 1 -- 172.21.15.67:0/1790324031 learned_addr learned my addr 172.21.15.67:0/1790324031 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:50:33.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.294+0000 7f3bad881700 1 --2- 172.21.15.67:0/1790324031 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3ba80ff340 0x7f3ba8100fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:33.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.294+0000 7f3bad080700 1 -- 172.21.15.67:0/1790324031 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3ba80ff340 msgr2=0x7f3ba8100fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:33.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.294+0000 7f3bad080700 1 --2- 172.21.15.67:0/1790324031 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3ba80ff340 0x7f3ba8100fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:33.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.294+0000 7f3bad080700 1 -- 172.21.15.67:0/1790324031 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b980095f0 con 0x7f3ba8105350 2024-04-22T23:50:33.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.295+0000 7f3bad080700 1 --2- 172.21.15.67:0/1790324031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ba8105350 0x7f3ba8101500 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f3b98006010 tx=0x7f3b9800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:50:33.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.295+0000 7f3b9effd700 1 -- 172.21.15.67:0/1790324031 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b9801d070 con 0x7f3ba8105350 2024-04-22T23:50:33.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.295+0000 7f3bafae5700 1 -- 172.21.15.67:0/1790324031 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ba8101e00 con 0x7f3ba8105350 2024-04-22T23:50:33.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.295+0000 7f3b9effd700 1 -- 172.21.15.67:0/1790324031 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3b98005dc0 con 0x7f3ba8105350 2024-04-22T23:50:33.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.295+0000 7f3b9effd700 1 -- 172.21.15.67:0/1790324031 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b98017400 con 0x7f3ba8105350 2024-04-22T23:50:33.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.295+0000 7f3bafae5700 1 -- 172.21.15.67:0/1790324031 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ba81a2d30 con 0x7f3ba8105350 2024-04-22T23:50:33.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.296+0000 7f3bafae5700 1 -- 172.21.15.67:0/1790324031 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ba8109860 con 0x7f3ba8105350 2024-04-22T23:50:33.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.296+0000 7f3b9effd700 1 -- 172.21.15.67:0/1790324031 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3b98003a60 con 0x7f3ba8105350 2024-04-22T23:50:33.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.297+0000 7f3b9effd700 1 --2- 172.21.15.67:0/1790324031 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3b9406ecc0 0x7f3b94071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:33.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.297+0000 7f3b9effd700 1 -- 172.21.15.67:0/1790324031 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f3b98090450 con 0x7f3ba8105350 2024-04-22T23:50:33.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.297+0000 7f3bad881700 1 --2- 172.21.15.67:0/1790324031 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3b9406ecc0 0x7f3b94071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:33.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.297+0000 7f3bad881700 1 --2- 172.21.15.67:0/1790324031 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3b9406ecc0 0x7f3b94071180 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4005950 tx=0x7f3ba400b510 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:50:33.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.302+0000 7f3b9effd700 1 -- 172.21.15.67:0/1790324031 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f3b98059090 con 0x7f3ba8105350 2024-04-22T23:50:33.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.534+0000 7f3bafae5700 1 -- 172.21.15.67:0/1790324031 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ba80611d0 con 0x7f3b9406ecc0 2024-04-22T23:50:33.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.535+0000 7f3b9effd700 1 -- 172.21.15.67:0/1790324031 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3ba80611d0 con 0x7f3b9406ecc0 2024-04-22T23:50:33.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.537+0000 7f3bafae5700 1 -- 172.21.15.67:0/1790324031 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3b9406ecc0 msgr2=0x7f3b94071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:33.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.537+0000 7f3bafae5700 1 --2- 172.21.15.67:0/1790324031 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3b9406ecc0 0x7f3b94071180 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4005950 tx=0x7f3ba400b510 comp rx=0 tx=0).stop 2024-04-22T23:50:33.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.537+0000 7f3bafae5700 1 -- 172.21.15.67:0/1790324031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ba8105350 msgr2=0x7f3ba8101500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:33.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.537+0000 7f3bafae5700 1 --2- 172.21.15.67:0/1790324031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ba8105350 0x7f3ba8101500 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f3b98006010 tx=0x7f3b9800bd70 comp rx=0 tx=0).stop 2024-04-22T23:50:33.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.537+0000 7f3bafae5700 1 -- 172.21.15.67:0/1790324031 shutdown_connections 2024-04-22T23:50:33.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.537+0000 7f3bafae5700 1 --2- 172.21.15.67:0/1790324031 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3b9406ecc0 0x7f3b94071180 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:33.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.537+0000 7f3bafae5700 1 --2- 172.21.15.67:0/1790324031 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3ba80ff340 0x7f3ba8100fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:33.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.537+0000 7f3bafae5700 1 --2- 172.21.15.67:0/1790324031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ba8105350 0x7f3ba8101500 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:33.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.537+0000 7f3bafae5700 1 -- 172.21.15.67:0/1790324031 >> 172.21.15.67:0/1790324031 conn(0x7f3ba8074e30 msgr2=0x7f3ba8103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:50:33.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.538+0000 7f3bafae5700 1 -- 172.21.15.67:0/1790324031 shutdown_connections 2024-04-22T23:50:33.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.538+0000 7f3bafae5700 1 -- 172.21.15.67:0/1790324031 wait complete. 2024-04-22T23:50:33.540 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:50:33.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.645+0000 7fc8ae619700 1 -- 172.21.15.67:0/2506492910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8a8105bd0 msgr2=0x7fc8a8105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:33.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.645+0000 7fc8ae619700 1 --2- 172.21.15.67:0/2506492910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8a8105bd0 0x7fc8a8105fb0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fc89c009910 tx=0x7fc89c009c20 comp rx=0 tx=0).stop 2024-04-22T23:50:33.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.645+0000 7fc8ae619700 1 -- 172.21.15.67:0/2506492910 shutdown_connections 2024-04-22T23:50:33.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.645+0000 7fc8ae619700 1 --2- 172.21.15.67:0/2506492910 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc8a80ffc20 0x7fc8a8100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:33.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.645+0000 7fc8ae619700 1 --2- 172.21.15.67:0/2506492910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8a8105bd0 0x7fc8a8105fb0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:33.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.645+0000 7fc8ae619700 1 -- 172.21.15.67:0/2506492910 >> 172.21.15.67:0/2506492910 conn(0x7fc8a8075670 msgr2=0x7fc8a8075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:50:33.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.645+0000 7fc8ae619700 1 -- 172.21.15.67:0/2506492910 shutdown_connections 2024-04-22T23:50:33.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.645+0000 7fc8ae619700 1 -- 172.21.15.67:0/2506492910 wait complete. 2024-04-22T23:50:33.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.646+0000 7fc8ae619700 1 Processor -- start 2024-04-22T23:50:33.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.646+0000 7fc8ae619700 1 -- start start 2024-04-22T23:50:33.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.646+0000 7fc8ae619700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc8a80ffc20 0x7fc8a819c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:33.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.646+0000 7fc8ae619700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8a8105bd0 0x7fc8a819cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:33.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.646+0000 7fc8ae619700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a819d1f0 con 0x7fc8a8105bd0 2024-04-22T23:50:33.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.646+0000 7fc8ae619700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a81966a0 con 0x7fc8a80ffc20 2024-04-22T23:50:33.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.646+0000 7fc8a77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8a8105bd0 0x7fc8a819cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:33.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.646+0000 7fc8a77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8a8105bd0 0x7fc8a819cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49782/0 (socket says 172.21.15.67:49782) 2024-04-22T23:50:33.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.646+0000 7fc8a77fe700 1 -- 172.21.15.67:0/2757504737 learned_addr learned my addr 172.21.15.67:0/2757504737 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:50:33.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.646+0000 7fc8a7fff700 1 --2- 172.21.15.67:0/2757504737 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc8a80ffc20 0x7fc8a819c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:33.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.646+0000 7fc8a77fe700 1 -- 172.21.15.67:0/2757504737 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc8a80ffc20 msgr2=0x7fc8a819c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:33.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.647+0000 7fc8a77fe700 1 --2- 172.21.15.67:0/2757504737 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc8a80ffc20 0x7fc8a819c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:33.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.647+0000 7fc8a77fe700 1 -- 172.21.15.67:0/2757504737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc89c0095f0 con 0x7fc8a8105bd0 2024-04-22T23:50:33.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.647+0000 7fc8a77fe700 1 --2- 172.21.15.67:0/2757504737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8a8105bd0 0x7fc8a819cb60 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fc89800c8a0 tx=0x7fc89800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:50:33.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.647+0000 7fc8a57fa700 1 -- 172.21.15.67:0/2757504737 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc89800cea0 con 0x7fc8a8105bd0 2024-04-22T23:50:33.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.647+0000 7fc8ae619700 1 -- 172.21.15.67:0/2757504737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8a8196980 con 0x7fc8a8105bd0 2024-04-22T23:50:33.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.647+0000 7fc8a57fa700 1 -- 172.21.15.67:0/2757504737 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc898004830 con 0x7fc8a8105bd0 2024-04-22T23:50:33.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.647+0000 7fc8a57fa700 1 -- 172.21.15.67:0/2757504737 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8980055b0 con 0x7fc8a8105bd0 2024-04-22T23:50:33.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.647+0000 7fc8ae619700 1 -- 172.21.15.67:0/2757504737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8a8196ed0 con 0x7fc8a8105bd0 2024-04-22T23:50:33.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.648+0000 7fc8ae619700 1 -- 172.21.15.67:0/2757504737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8a804f2e0 con 0x7fc8a8105bd0 2024-04-22T23:50:33.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.649+0000 7fc8a57fa700 1 -- 172.21.15.67:0/2757504737 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc8980049a0 con 0x7fc8a8105bd0 2024-04-22T23:50:33.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.649+0000 7fc8a57fa700 1 --2- 172.21.15.67:0/2757504737 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc89006ecc0 0x7fc890071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:33.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.650+0000 7fc8a57fa700 1 -- 172.21.15.67:0/2757504737 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fc89808f080 con 0x7fc8a8105bd0 2024-04-22T23:50:33.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.650+0000 7fc8a7fff700 1 --2- 172.21.15.67:0/2757504737 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc89006ecc0 0x7fc890071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:33.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.650+0000 7fc8a7fff700 1 --2- 172.21.15.67:0/2757504737 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc89006ecc0 0x7fc890071180 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc89c00b5c0 tx=0x7fc89c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:50:33.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.654+0000 7fc8a57fa700 1 -- 172.21.15.67:0/2757504737 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc89805b910 con 0x7fc8a8105bd0 2024-04-22T23:50:33.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.883+0000 7fc8ae619700 1 -- 172.21.15.67:0/2757504737 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc8a8197270 con 0x7fc89006ecc0 2024-04-22T23:50:33.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.884+0000 7fc8a57fa700 1 -- 172.21.15.67:0/2757504737 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc8a8197270 con 0x7fc89006ecc0 2024-04-22T23:50:33.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.886+0000 7fc8ae619700 1 -- 172.21.15.67:0/2757504737 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc89006ecc0 msgr2=0x7fc890071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:33.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.886+0000 7fc8ae619700 1 --2- 172.21.15.67:0/2757504737 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc89006ecc0 0x7fc890071180 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc89c00b5c0 tx=0x7fc89c01a040 comp rx=0 tx=0).stop 2024-04-22T23:50:33.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.886+0000 7fc8ae619700 1 -- 172.21.15.67:0/2757504737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8a8105bd0 msgr2=0x7fc8a819cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:33.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.886+0000 7fc8ae619700 1 --2- 172.21.15.67:0/2757504737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8a8105bd0 0x7fc8a819cb60 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fc89800c8a0 tx=0x7fc89800cc60 comp rx=0 tx=0).stop 2024-04-22T23:50:33.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.886+0000 7fc8ae619700 1 -- 172.21.15.67:0/2757504737 shutdown_connections 2024-04-22T23:50:33.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.886+0000 7fc8ae619700 1 --2- 172.21.15.67:0/2757504737 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc89006ecc0 0x7fc890071180 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:33.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.886+0000 7fc8ae619700 1 --2- 172.21.15.67:0/2757504737 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc8a80ffc20 0x7fc8a819c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:33.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.886+0000 7fc8ae619700 1 --2- 172.21.15.67:0/2757504737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8a8105bd0 0x7fc8a819cb60 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:33.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.886+0000 7fc8ae619700 1 -- 172.21.15.67:0/2757504737 >> 172.21.15.67:0/2757504737 conn(0x7fc8a8075670 msgr2=0x7fc8a80686c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:50:33.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.886+0000 7fc8ae619700 1 -- 172.21.15.67:0/2757504737 shutdown_connections 2024-04-22T23:50:33.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.887+0000 7fc8ae619700 1 -- 172.21.15.67:0/2757504737 wait complete. 2024-04-22T23:50:33.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.994+0000 7f07b413a700 1 -- 172.21.15.67:0/1148539146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07ac100ea0 msgr2=0x7f07ac101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:33.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.994+0000 7f07b413a700 1 --2- 172.21.15.67:0/1148539146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07ac100ea0 0x7f07ac101280 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f079c009910 tx=0x7f079c009c20 comp rx=0 tx=0).stop 2024-04-22T23:50:33.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.994+0000 7f07b413a700 1 -- 172.21.15.67:0/1148539146 shutdown_connections 2024-04-22T23:50:33.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.994+0000 7f07b413a700 1 --2- 172.21.15.67:0/1148539146 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07ac101850 0x7f07ac1058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:33.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.994+0000 7f07b413a700 1 --2- 172.21.15.67:0/1148539146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07ac100ea0 0x7f07ac101280 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:33.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.994+0000 7f07b413a700 1 -- 172.21.15.67:0/1148539146 >> 172.21.15.67:0/1148539146 conn(0x7f07ac0fc750 msgr2=0x7f07ac0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:50:33.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.994+0000 7f07b413a700 1 -- 172.21.15.67:0/1148539146 shutdown_connections 2024-04-22T23:50:33.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.994+0000 7f07b413a700 1 -- 172.21.15.67:0/1148539146 wait complete. 2024-04-22T23:50:33.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.995+0000 7f07b413a700 1 Processor -- start 2024-04-22T23:50:33.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.995+0000 7f07b413a700 1 -- start start 2024-04-22T23:50:33.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.995+0000 7f07b413a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07ac100ea0 0x7f07ac19c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:33.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.995+0000 7f07b413a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07ac101850 0x7f07ac19cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:33.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.995+0000 7f07b413a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07ac19d230 con 0x7f07ac100ea0 2024-04-22T23:50:33.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.995+0000 7f07b413a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07ac196770 con 0x7f07ac101850 2024-04-22T23:50:33.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.995+0000 7f07b1ed6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07ac100ea0 0x7f07ac19c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:33.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.995+0000 7f07b1ed6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07ac100ea0 0x7f07ac19c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49794/0 (socket says 172.21.15.67:49794) 2024-04-22T23:50:33.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.995+0000 7f07b1ed6700 1 -- 172.21.15.67:0/1956110830 learned_addr learned my addr 172.21.15.67:0/1956110830 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:50:33.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.996+0000 7f07b16d5700 1 --2- 172.21.15.67:0/1956110830 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07ac101850 0x7f07ac19cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:33.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.996+0000 7f07b1ed6700 1 -- 172.21.15.67:0/1956110830 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07ac101850 msgr2=0x7f07ac19cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:33.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.996+0000 7f07b1ed6700 1 --2- 172.21.15.67:0/1956110830 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07ac101850 0x7f07ac19cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:33.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.996+0000 7f07b1ed6700 1 -- 172.21.15.67:0/1956110830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f079c0095f0 con 0x7f07ac100ea0 2024-04-22T23:50:33.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.996+0000 7f07b16d5700 1 --2- 172.21.15.67:0/1956110830 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07ac101850 0x7f07ac19cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:50:33.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.996+0000 7f07b1ed6700 1 --2- 172.21.15.67:0/1956110830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07ac100ea0 0x7f07ac19c6f0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f079c006010 tx=0x7f079c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:50:33.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.996+0000 7f07a2ffd700 1 -- 172.21.15.67:0/1956110830 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f079c01d070 con 0x7f07ac100ea0 2024-04-22T23:50:33.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.996+0000 7f07a2ffd700 1 -- 172.21.15.67:0/1956110830 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f079c004fb0 con 0x7f07ac100ea0 2024-04-22T23:50:33.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.996+0000 7f07a2ffd700 1 -- 172.21.15.67:0/1956110830 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f079c022ea0 con 0x7f07ac100ea0 2024-04-22T23:50:33.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.996+0000 7f07b413a700 1 -- 172.21.15.67:0/1956110830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07ac1969f0 con 0x7f07ac100ea0 2024-04-22T23:50:33.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.996+0000 7f07b413a700 1 -- 172.21.15.67:0/1956110830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07ac196ee0 con 0x7f07ac100ea0 2024-04-22T23:50:33.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.997+0000 7f07b413a700 1 -- 172.21.15.67:0/1956110830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07ac109240 con 0x7f07ac100ea0 2024-04-22T23:50:33.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.998+0000 7f07a2ffd700 1 -- 172.21.15.67:0/1956110830 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f079c022470 con 0x7f07ac100ea0 2024-04-22T23:50:33.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.998+0000 7f07a2ffd700 1 --2- 172.21.15.67:0/1956110830 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f079806ecc0 0x7f0798071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:33.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.998+0000 7f07a2ffd700 1 -- 172.21.15.67:0/1956110830 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f079c090490 con 0x7f07ac100ea0 2024-04-22T23:50:33.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.998+0000 7f07b16d5700 1 --2- 172.21.15.67:0/1956110830 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f079806ecc0 0x7f0798071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:33.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:33.999+0000 7f07b16d5700 1 --2- 172.21.15.67:0/1956110830 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f079806ecc0 0x7f0798071180 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f07a8005fd0 tx=0x7f07a8005e20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:50:33.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.003+0000 7f07a2ffd700 1 -- 172.21.15.67:0/1956110830 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f079c05cd30 con 0x7f07ac100ea0 2024-04-22T23:50:34.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.221+0000 7f07b413a700 1 -- 172.21.15.67:0/1956110830 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f07ac197cf0 con 0x7f079806ecc0 2024-04-22T23:50:34.213 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:34 smithi067 ceph-mon[112438]: pgmap v727: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:34.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.233+0000 7f07a2ffd700 1 -- 172.21.15.67:0/1956110830 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f07ac197cf0 con 0x7f079806ecc0 2024-04-22T23:50:34.225 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:50:34.225 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (24m) 2m ago 35m 24.4M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:50:34.225 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (35m) 2m ago 35m 10.0M - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:50:34.225 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (34m) 2m ago 34m 10.0M - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:50:34.225 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (21m) 2m ago 35m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:50:34.225 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (21m) 2m ago 34m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:50:34.225 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (23m) 2m ago 35m 81.2M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:50:34.225 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (17m) 2m ago 30m 19.4M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:50:34.225 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (17m) 2m ago 30m 20.3M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:50:34.225 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (17m) 2m ago 30m 20.1M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:50:34.225 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (2m) 2m ago 29m 14.0M - 19.0.0-3252-g43be0201 a0434699c1d7 46d3d7e86fee 2024-04-22T23:50:34.225 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (25m) 2m ago 36m 514M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:50:34.226 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (25m) 2m ago 34m 459M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:50:34.226 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (22m) 2m ago 37m 74.2M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:50:34.226 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (22m) 2m ago 34m 54.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:50:34.226 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (24m) 2m ago 35m 15.2M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:50:34.226 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (24m) 2m ago 34m 16.0M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:50:34.226 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (21m) 2m ago 33m 102M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:50:34.226 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (20m) 2m ago 33m 128M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:50:34.226 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (20m) 2m ago 32m 83.8M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:50:34.228 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (19m) 2m ago 32m 101M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:50:34.229 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (18m) 2m ago 32m 86.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:50:34.230 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (18m) 2m ago 31m 81.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:50:34.230 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (24m) 2m ago 35m 78.7M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:50:34.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.235+0000 7f07b413a700 1 -- 172.21.15.67:0/1956110830 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f079806ecc0 msgr2=0x7f0798071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:34.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.235+0000 7f07b413a700 1 --2- 172.21.15.67:0/1956110830 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f079806ecc0 0x7f0798071180 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f07a8005fd0 tx=0x7f07a8005e20 comp rx=0 tx=0).stop 2024-04-22T23:50:34.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.235+0000 7f07b413a700 1 -- 172.21.15.67:0/1956110830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07ac100ea0 msgr2=0x7f07ac19c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:34.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.235+0000 7f07b413a700 1 --2- 172.21.15.67:0/1956110830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07ac100ea0 0x7f07ac19c6f0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f079c006010 tx=0x7f079c00be20 comp rx=0 tx=0).stop 2024-04-22T23:50:34.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.235+0000 7f07b413a700 1 -- 172.21.15.67:0/1956110830 shutdown_connections 2024-04-22T23:50:34.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.235+0000 7f07b413a700 1 --2- 172.21.15.67:0/1956110830 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f079806ecc0 0x7f0798071180 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:34.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.235+0000 7f07b413a700 1 --2- 172.21.15.67:0/1956110830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07ac100ea0 0x7f07ac19c6f0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:34.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.235+0000 7f07b413a700 1 --2- 172.21.15.67:0/1956110830 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07ac101850 0x7f07ac19cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:34.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.235+0000 7f07b413a700 1 -- 172.21.15.67:0/1956110830 >> 172.21.15.67:0/1956110830 conn(0x7f07ac0fc750 msgr2=0x7f07ac0fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:50:34.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.235+0000 7f07b413a700 1 -- 172.21.15.67:0/1956110830 shutdown_connections 2024-04-22T23:50:34.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.235+0000 7f07b413a700 1 -- 172.21.15.67:0/1956110830 wait complete. 2024-04-22T23:50:34.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.341+0000 7f6a73c55700 1 -- 172.21.15.67:0/1127790227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a6c100750 msgr2=0x7f6a6c1047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:34.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.341+0000 7f6a73c55700 1 --2- 172.21.15.67:0/1127790227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a6c100750 0x7f6a6c1047a0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009910 tx=0x7f6a60009c20 comp rx=0 tx=0).stop 2024-04-22T23:50:34.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.341+0000 7f6a73c55700 1 -- 172.21.15.67:0/1127790227 shutdown_connections 2024-04-22T23:50:34.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.341+0000 7f6a73c55700 1 --2- 172.21.15.67:0/1127790227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a6c100750 0x7f6a6c1047a0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:34.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.341+0000 7f6a73c55700 1 --2- 172.21.15.67:0/1127790227 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6a6c0ffda0 0x7f6a6c100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:34.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.341+0000 7f6a73c55700 1 -- 172.21.15.67:0/1127790227 >> 172.21.15.67:0/1127790227 conn(0x7f6a6c0fb650 msgr2=0x7f6a6c0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:50:34.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.341+0000 7f6a73c55700 1 -- 172.21.15.67:0/1127790227 shutdown_connections 2024-04-22T23:50:34.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.341+0000 7f6a73c55700 1 -- 172.21.15.67:0/1127790227 wait complete. 2024-04-22T23:50:34.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.341+0000 7f6a73c55700 1 Processor -- start 2024-04-22T23:50:34.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.342+0000 7f6a73c55700 1 -- start start 2024-04-22T23:50:34.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.342+0000 7f6a73c55700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a6c0ffda0 0x7f6a6c19c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:34.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.342+0000 7f6a73c55700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6a6c100750 0x7f6a6c19cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:34.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.342+0000 7f6a73c55700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6c19d240 con 0x7f6a6c0ffda0 2024-04-22T23:50:34.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.342+0000 7f6a73c55700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6c1966f0 con 0x7f6a6c100750 2024-04-22T23:50:34.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.342+0000 7f6a719f1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a6c0ffda0 0x7f6a6c19c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:34.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.342+0000 7f6a719f1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a6c0ffda0 0x7f6a6c19c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49802/0 (socket says 172.21.15.67:49802) 2024-04-22T23:50:34.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.342+0000 7f6a719f1700 1 -- 172.21.15.67:0/707668918 learned_addr learned my addr 172.21.15.67:0/707668918 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:50:34.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.342+0000 7f6a711f0700 1 --2- 172.21.15.67:0/707668918 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6a6c100750 0x7f6a6c19cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:34.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.342+0000 7f6a719f1700 1 -- 172.21.15.67:0/707668918 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6a6c100750 msgr2=0x7f6a6c19cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:34.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.342+0000 7f6a719f1700 1 --2- 172.21.15.67:0/707668918 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6a6c100750 0x7f6a6c19cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:34.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.342+0000 7f6a719f1700 1 -- 172.21.15.67:0/707668918 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a600095f0 con 0x7f6a6c0ffda0 2024-04-22T23:50:34.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.342+0000 7f6a719f1700 1 --2- 172.21.15.67:0/707668918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a6c0ffda0 0x7f6a6c19c670 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f6a6400d8d0 tx=0x7f6a6400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:50:34.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.343+0000 7f6a6affd700 1 -- 172.21.15.67:0/707668918 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a64009940 con 0x7f6a6c0ffda0 2024-04-22T23:50:34.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.343+0000 7f6a6affd700 1 -- 172.21.15.67:0/707668918 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6a64010460 con 0x7f6a6c0ffda0 2024-04-22T23:50:34.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.343+0000 7f6a73c55700 1 -- 172.21.15.67:0/707668918 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a6c1969d0 con 0x7f6a6c0ffda0 2024-04-22T23:50:34.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.343+0000 7f6a6affd700 1 -- 172.21.15.67:0/707668918 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a6400f5f0 con 0x7f6a6c0ffda0 2024-04-22T23:50:34.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.343+0000 7f6a73c55700 1 -- 172.21.15.67:0/707668918 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a6c196f20 con 0x7f6a6c0ffda0 2024-04-22T23:50:34.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.344+0000 7f6a73c55700 1 -- 172.21.15.67:0/707668918 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a6c04f2e0 con 0x7f6a6c0ffda0 2024-04-22T23:50:34.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.344+0000 7f6a6affd700 1 -- 172.21.15.67:0/707668918 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6a640105d0 con 0x7f6a6c0ffda0 2024-04-22T23:50:34.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.345+0000 7f6a6affd700 1 --2- 172.21.15.67:0/707668918 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6a5406ecc0 0x7f6a54071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:34.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.345+0000 7f6a6affd700 1 -- 172.21.15.67:0/707668918 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f6a6408eef0 con 0x7f6a6c0ffda0 2024-04-22T23:50:34.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.345+0000 7f6a711f0700 1 --2- 172.21.15.67:0/707668918 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6a5406ecc0 0x7f6a54071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:34.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.345+0000 7f6a711f0700 1 --2- 172.21.15.67:0/707668918 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6a5406ecc0 0x7f6a54071180 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6a60000c00 tx=0x7f6a60005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:50:34.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.349+0000 7f6a6affd700 1 -- 172.21.15.67:0/707668918 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f6a6405b6e0 con 0x7f6a6c0ffda0 2024-04-22T23:50:34.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:34 smithi134 ceph-mon[85630]: pgmap v727: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:34.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.650+0000 7f6a73c55700 1 -- 172.21.15.67:0/707668918 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6a6c197c00 con 0x7f6a6c0ffda0 2024-04-22T23:50:34.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.650+0000 7f6a6affd700 1 -- 172.21.15.67:0/707668918 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f6a6405b270 con 0x7f6a6c0ffda0 2024-04-22T23:50:34.642 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:50:34.642 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:50:34.642 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:50:34.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:50:34.642 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:50:34.642 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:50:34.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:50:34.642 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:50:34.642 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:50:34.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:50:34.643 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:50:34.643 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 4 2024-04-22T23:50:34.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:50:34.643 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:50:34.643 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 14 2024-04-22T23:50:34.643 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:50:34.643 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:50:34.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.652+0000 7f6a73c55700 1 -- 172.21.15.67:0/707668918 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6a5406ecc0 msgr2=0x7f6a54071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:34.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.652+0000 7f6a73c55700 1 --2- 172.21.15.67:0/707668918 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6a5406ecc0 0x7f6a54071180 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6a60000c00 tx=0x7f6a60005fd0 comp rx=0 tx=0).stop 2024-04-22T23:50:34.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.652+0000 7f6a73c55700 1 -- 172.21.15.67:0/707668918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a6c0ffda0 msgr2=0x7f6a6c19c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:34.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.652+0000 7f6a73c55700 1 --2- 172.21.15.67:0/707668918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a6c0ffda0 0x7f6a6c19c670 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f6a6400d8d0 tx=0x7f6a6400dc90 comp rx=0 tx=0).stop 2024-04-22T23:50:34.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.652+0000 7f6a73c55700 1 -- 172.21.15.67:0/707668918 shutdown_connections 2024-04-22T23:50:34.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.652+0000 7f6a73c55700 1 --2- 172.21.15.67:0/707668918 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6a5406ecc0 0x7f6a54071180 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:34.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.652+0000 7f6a73c55700 1 --2- 172.21.15.67:0/707668918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a6c0ffda0 0x7f6a6c19c670 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:34.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.652+0000 7f6a73c55700 1 --2- 172.21.15.67:0/707668918 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6a6c100750 0x7f6a6c19cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:34.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.652+0000 7f6a73c55700 1 -- 172.21.15.67:0/707668918 >> 172.21.15.67:0/707668918 conn(0x7f6a6c0fb650 msgr2=0x7f6a6c103f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:50:34.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.653+0000 7f6a73c55700 1 -- 172.21.15.67:0/707668918 shutdown_connections 2024-04-22T23:50:34.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.653+0000 7f6a73c55700 1 -- 172.21.15.67:0/707668918 wait complete. 2024-04-22T23:50:34.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.759+0000 7f0163338700 1 -- 172.21.15.67:0/3102238277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f015c102720 msgr2=0x7f015c10ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:34.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.759+0000 7f0163338700 1 --2- 172.21.15.67:0/3102238277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f015c102720 0x7f015c10ac10 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f014c009910 tx=0x7f014c009c20 comp rx=0 tx=0).stop 2024-04-22T23:50:34.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.759+0000 7f0163338700 1 -- 172.21.15.67:0/3102238277 shutdown_connections 2024-04-22T23:50:34.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.759+0000 7f0163338700 1 --2- 172.21.15.67:0/3102238277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f015c102720 0x7f015c10ac10 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:34.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.759+0000 7f0163338700 1 --2- 172.21.15.67:0/3102238277 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f015c101e00 0x7f015c1021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:34.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.759+0000 7f0163338700 1 -- 172.21.15.67:0/3102238277 >> 172.21.15.67:0/3102238277 conn(0x7f015c0fb650 msgr2=0x7f015c0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:50:34.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.759+0000 7f0163338700 1 -- 172.21.15.67:0/3102238277 shutdown_connections 2024-04-22T23:50:34.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.759+0000 7f0163338700 1 -- 172.21.15.67:0/3102238277 wait complete. 2024-04-22T23:50:34.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.760+0000 7f0163338700 1 Processor -- start 2024-04-22T23:50:34.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.760+0000 7f0163338700 1 -- start start 2024-04-22T23:50:34.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.760+0000 7f0163338700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f015c101e00 0x7f015c0ffa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:34.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.760+0000 7f0163338700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f015c102720 0x7f015c0fffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:34.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.760+0000 7f0163338700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c1004e0 con 0x7f015c101e00 2024-04-22T23:50:34.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.760+0000 7f0163338700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c100620 con 0x7f015c102720 2024-04-22T23:50:34.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.760+0000 7f01610d4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f015c101e00 0x7f015c0ffa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:34.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.760+0000 7f01610d4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f015c101e00 0x7f015c0ffa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49810/0 (socket says 172.21.15.67:49810) 2024-04-22T23:50:34.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.760+0000 7f01610d4700 1 -- 172.21.15.67:0/1709243339 learned_addr learned my addr 172.21.15.67:0/1709243339 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:50:34.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.760+0000 7f01608d3700 1 --2- 172.21.15.67:0/1709243339 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f015c102720 0x7f015c0fffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:34.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.760+0000 7f01610d4700 1 -- 172.21.15.67:0/1709243339 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f015c102720 msgr2=0x7f015c0fffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:34.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.760+0000 7f01610d4700 1 --2- 172.21.15.67:0/1709243339 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f015c102720 0x7f015c0fffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:34.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.760+0000 7f01610d4700 1 -- 172.21.15.67:0/1709243339 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f014c0095f0 con 0x7f015c101e00 2024-04-22T23:50:34.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.760+0000 7f01610d4700 1 --2- 172.21.15.67:0/1709243339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f015c101e00 0x7f015c0ffa60 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f015800c930 tx=0x7f015800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:50:34.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.761+0000 7f01527fc700 1 -- 172.21.15.67:0/1709243339 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0158007ad0 con 0x7f015c101e00 2024-04-22T23:50:34.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.761+0000 7f0163338700 1 -- 172.21.15.67:0/1709243339 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f015c1a2a90 con 0x7f015c101e00 2024-04-22T23:50:34.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.761+0000 7f01527fc700 1 -- 172.21.15.67:0/1709243339 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0158007c30 con 0x7f015c101e00 2024-04-22T23:50:34.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.761+0000 7f01527fc700 1 -- 172.21.15.67:0/1709243339 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0158018610 con 0x7f015c101e00 2024-04-22T23:50:34.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.761+0000 7f0163338700 1 -- 172.21.15.67:0/1709243339 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f015c1a2df0 con 0x7f015c101e00 2024-04-22T23:50:34.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.762+0000 7f0163338700 1 -- 172.21.15.67:0/1709243339 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f015c108310 con 0x7f015c101e00 2024-04-22T23:50:34.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.762+0000 7f01527fc700 1 -- 172.21.15.67:0/1709243339 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f015800f450 con 0x7f015c101e00 2024-04-22T23:50:34.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.763+0000 7f01527fc700 1 --2- 172.21.15.67:0/1709243339 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f014806ecc0 0x7f0148071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:34.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.763+0000 7f01527fc700 1 -- 172.21.15.67:0/1709243339 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f015808f120 con 0x7f015c101e00 2024-04-22T23:50:34.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.763+0000 7f01608d3700 1 --2- 172.21.15.67:0/1709243339 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f014806ecc0 0x7f0148071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:34.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.763+0000 7f01608d3700 1 --2- 172.21.15.67:0/1709243339 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f014806ecc0 0x7f0148071180 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f014c006010 tx=0x7f014c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:50:34.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:34.767+0000 7f01527fc700 1 -- 172.21.15.67:0/1709243339 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f0158057de0 con 0x7f015c101e00 2024-04-22T23:50:35.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.025+0000 7f0163338700 1 -- 172.21.15.67:0/1709243339 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f015c107cf0 con 0x7f015c101e00 2024-04-22T23:50:35.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.026+0000 7f01527fc700 1 -- 172.21.15.67:0/1709243339 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1872 (secure 0 0 0) 0x7f015805b4a0 con 0x7f015c101e00 2024-04-22T23:50:35.018 INFO:teuthology.orchestra.run.smithi067.stdout:e33 2024-04-22T23:50:35.018 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:50:35.018 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:50:35.018 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:50:35.018 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:50:35.018 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:50:35.018 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:50:35.018 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:50:35.018 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:50:35.018 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:50:35.018 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:50:35.018 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:50:35.018 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:50:35.019 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:50:35.019 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:50:35.019 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:50:35.019 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:50:35.019 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:50:35.019 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:50:35.019 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:50:35.019 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:50:35.019 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:50:35.019 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:50:35.019 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:50:35.020 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:50:35.020 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:50:35.020 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:50:35.020 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:50:35.020 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:50:35.020 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:50:35.020 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:50:35.020 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:50:35.020 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:50:35.020 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:50:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:50:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:50:35.020 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:50:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:50:35.021 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:50:35.021 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:50:35.021 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:50:35.021 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44851} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1849648622,v1:172.21.15.134:6827/1849648622] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:50:35.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.028+0000 7f0163338700 1 -- 172.21.15.67:0/1709243339 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f014806ecc0 msgr2=0x7f0148071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:35.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.028+0000 7f0163338700 1 --2- 172.21.15.67:0/1709243339 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f014806ecc0 0x7f0148071180 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f014c006010 tx=0x7f014c01a040 comp rx=0 tx=0).stop 2024-04-22T23:50:35.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.028+0000 7f0163338700 1 -- 172.21.15.67:0/1709243339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f015c101e00 msgr2=0x7f015c0ffa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:35.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.028+0000 7f0163338700 1 --2- 172.21.15.67:0/1709243339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f015c101e00 0x7f015c0ffa60 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f015800c930 tx=0x7f015800ccf0 comp rx=0 tx=0).stop 2024-04-22T23:50:35.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.028+0000 7f0163338700 1 -- 172.21.15.67:0/1709243339 shutdown_connections 2024-04-22T23:50:35.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.028+0000 7f0163338700 1 --2- 172.21.15.67:0/1709243339 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f014806ecc0 0x7f0148071180 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:35.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.028+0000 7f0163338700 1 --2- 172.21.15.67:0/1709243339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f015c101e00 0x7f015c0ffa60 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:35.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.028+0000 7f0163338700 1 --2- 172.21.15.67:0/1709243339 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f015c102720 0x7f015c0fffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:35.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.028+0000 7f0163338700 1 -- 172.21.15.67:0/1709243339 >> 172.21.15.67:0/1709243339 conn(0x7f015c0fb650 msgr2=0x7f015c105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:50:35.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.028+0000 7f0163338700 1 -- 172.21.15.67:0/1709243339 shutdown_connections 2024-04-22T23:50:35.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.028+0000 7f0163338700 1 -- 172.21.15.67:0/1709243339 wait complete. 2024-04-22T23:50:35.022 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 33 2024-04-22T23:50:35.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.136+0000 7f7fb789a700 1 -- 172.21.15.67:0/1376754532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fb0100750 msgr2=0x7f7fb01047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:35.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.136+0000 7f7fb789a700 1 --2- 172.21.15.67:0/1376754532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fb0100750 0x7f7fb01047a0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f7fa0009910 tx=0x7f7fa0009c20 comp rx=0 tx=0).stop 2024-04-22T23:50:35.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.136+0000 7f7fb789a700 1 -- 172.21.15.67:0/1376754532 shutdown_connections 2024-04-22T23:50:35.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.136+0000 7f7fb789a700 1 --2- 172.21.15.67:0/1376754532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fb0100750 0x7f7fb01047a0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:35.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.136+0000 7f7fb789a700 1 --2- 172.21.15.67:0/1376754532 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7fb00ffda0 0x7f7fb0100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:35.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.136+0000 7f7fb789a700 1 -- 172.21.15.67:0/1376754532 >> 172.21.15.67:0/1376754532 conn(0x7f7fb00fb650 msgr2=0x7f7fb00fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:50:35.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.136+0000 7f7fb789a700 1 -- 172.21.15.67:0/1376754532 shutdown_connections 2024-04-22T23:50:35.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.136+0000 7f7fb789a700 1 -- 172.21.15.67:0/1376754532 wait complete. 2024-04-22T23:50:35.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.137+0000 7f7fb789a700 1 Processor -- start 2024-04-22T23:50:35.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.137+0000 7f7fb789a700 1 -- start start 2024-04-22T23:50:35.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.137+0000 7f7fb789a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7fb00ffda0 0x7f7fb0196950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:35.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.137+0000 7f7fb789a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fb0100750 0x7f7fb0196e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:35.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.137+0000 7f7fb789a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb0197520 con 0x7f7fb0100750 2024-04-22T23:50:35.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.137+0000 7f7fb789a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb019b330 con 0x7f7fb00ffda0 2024-04-22T23:50:35.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.137+0000 7f7fb4e35700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fb0100750 0x7f7fb0196e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:35.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.137+0000 7f7fb4e35700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fb0100750 0x7f7fb0196e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49832/0 (socket says 172.21.15.67:49832) 2024-04-22T23:50:35.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.137+0000 7f7fb4e35700 1 -- 172.21.15.67:0/481297346 learned_addr learned my addr 172.21.15.67:0/481297346 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:50:35.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.137+0000 7f7fb5636700 1 --2- 172.21.15.67:0/481297346 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7fb00ffda0 0x7f7fb0196950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:35.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.137+0000 7f7fb4e35700 1 -- 172.21.15.67:0/481297346 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7fb00ffda0 msgr2=0x7f7fb0196950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:35.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.137+0000 7f7fb4e35700 1 --2- 172.21.15.67:0/481297346 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7fb00ffda0 0x7f7fb0196950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:35.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.137+0000 7f7fb4e35700 1 -- 172.21.15.67:0/481297346 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa00095f0 con 0x7f7fb0100750 2024-04-22T23:50:35.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.137+0000 7f7fb4e35700 1 --2- 172.21.15.67:0/481297346 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fb0100750 0x7f7fb0196e90 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f7fa000bda0 tx=0x7f7fa000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:50:35.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.138+0000 7f7fa67fc700 1 -- 172.21.15.67:0/481297346 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7fa001d070 con 0x7f7fb0100750 2024-04-22T23:50:35.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.138+0000 7f7fa67fc700 1 -- 172.21.15.67:0/481297346 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7fa000f460 con 0x7f7fb0100750 2024-04-22T23:50:35.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.138+0000 7f7fb789a700 1 -- 172.21.15.67:0/481297346 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fb019b5b0 con 0x7f7fb0100750 2024-04-22T23:50:35.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.138+0000 7f7fa67fc700 1 -- 172.21.15.67:0/481297346 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7fa0017630 con 0x7f7fb0100750 2024-04-22T23:50:35.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.138+0000 7f7fb789a700 1 -- 172.21.15.67:0/481297346 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fb019baa0 con 0x7f7fb0100750 2024-04-22T23:50:35.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.139+0000 7f7fb789a700 1 -- 172.21.15.67:0/481297346 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fb004f2e0 con 0x7f7fb0100750 2024-04-22T23:50:35.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.139+0000 7f7fa67fc700 1 -- 172.21.15.67:0/481297346 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7fa000fa40 con 0x7f7fb0100750 2024-04-22T23:50:35.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.140+0000 7f7fa67fc700 1 --2- 172.21.15.67:0/481297346 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7f9c06ecc0 0x7f7f9c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:35.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.140+0000 7f7fa67fc700 1 -- 172.21.15.67:0/481297346 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f7fa0090410 con 0x7f7fb0100750 2024-04-22T23:50:35.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.140+0000 7f7fb5636700 1 --2- 172.21.15.67:0/481297346 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7f9c06ecc0 0x7f7f9c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:35.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.140+0000 7f7fb5636700 1 --2- 172.21.15.67:0/481297346 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7f9c06ecc0 0x7f7f9c071180 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7fac005950 tx=0x7f7fac0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:50:35.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.145+0000 7f7fa67fc700 1 -- 172.21.15.67:0/481297346 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7fa005cb80 con 0x7f7fb0100750 2024-04-22T23:50:35.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:35 smithi067 ceph-mon[112438]: from='client.35328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:50:35.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:35 smithi067 ceph-mon[112438]: from='client.35332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:50:35.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:35 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/707668918' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:50:35.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.380+0000 7f7fb789a700 1 -- 172.21.15.67:0/481297346 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7fb019bcc0 con 0x7f7f9c06ecc0 2024-04-22T23:50:35.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.381+0000 7f7fa67fc700 1 -- 172.21.15.67:0/481297346 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f7fb019bcc0 con 0x7f7f9c06ecc0 2024-04-22T23:50:35.373 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:50:35.373 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:50:35.373 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:50:35.373 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:50:35.373 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:50:35.373 INFO:teuthology.orchestra.run.smithi067.stdout: "mds", 2024-04-22T23:50:35.373 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:50:35.373 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:50:35.373 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:50:35.373 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:50:35.373 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:50:35.373 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T23:50:35.373 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:50:35.374 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:50:35.374 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:50:35.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.383+0000 7f7fb789a700 1 -- 172.21.15.67:0/481297346 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7f9c06ecc0 msgr2=0x7f7f9c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:35.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.383+0000 7f7fb789a700 1 --2- 172.21.15.67:0/481297346 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7f9c06ecc0 0x7f7f9c071180 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7fac005950 tx=0x7f7fac0058e0 comp rx=0 tx=0).stop 2024-04-22T23:50:35.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.383+0000 7f7fb789a700 1 -- 172.21.15.67:0/481297346 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fb0100750 msgr2=0x7f7fb0196e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:35.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.383+0000 7f7fb789a700 1 --2- 172.21.15.67:0/481297346 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fb0100750 0x7f7fb0196e90 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f7fa000bda0 tx=0x7f7fa000be80 comp rx=0 tx=0).stop 2024-04-22T23:50:35.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.383+0000 7f7fb789a700 1 -- 172.21.15.67:0/481297346 shutdown_connections 2024-04-22T23:50:35.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.383+0000 7f7fb789a700 1 --2- 172.21.15.67:0/481297346 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7f9c06ecc0 0x7f7f9c071180 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:35.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.383+0000 7f7fb789a700 1 --2- 172.21.15.67:0/481297346 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7fb00ffda0 0x7f7fb0196950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:35.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.383+0000 7f7fb789a700 1 --2- 172.21.15.67:0/481297346 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fb0100750 0x7f7fb0196e90 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:35.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.383+0000 7f7fb789a700 1 -- 172.21.15.67:0/481297346 >> 172.21.15.67:0/481297346 conn(0x7f7fb00fb650 msgr2=0x7f7fb0103fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:50:35.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.383+0000 7f7fb789a700 1 -- 172.21.15.67:0/481297346 shutdown_connections 2024-04-22T23:50:35.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.383+0000 7f7fb789a700 1 -- 172.21.15.67:0/481297346 wait complete. 2024-04-22T23:50:35.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.490+0000 7fe7f2282700 1 -- 172.21.15.67:0/862419573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7ec100310 msgr2=0x7fe7ec10d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:35.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.490+0000 7fe7f2282700 1 --2- 172.21.15.67:0/862419573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7ec100310 0x7fe7ec10d060 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc009910 tx=0x7fe7dc009c20 comp rx=0 tx=0).stop 2024-04-22T23:50:35.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.490+0000 7fe7f2282700 1 -- 172.21.15.67:0/862419573 shutdown_connections 2024-04-22T23:50:35.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.490+0000 7fe7f2282700 1 --2- 172.21.15.67:0/862419573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7ec100310 0x7fe7ec10d060 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:35.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.490+0000 7fe7f2282700 1 --2- 172.21.15.67:0/862419573 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe7ec0ff9f0 0x7fe7ec0ffdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:35.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.490+0000 7fe7f2282700 1 -- 172.21.15.67:0/862419573 >> 172.21.15.67:0/862419573 conn(0x7fe7ec0fb650 msgr2=0x7fe7ec0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:50:35.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.490+0000 7fe7f2282700 1 -- 172.21.15.67:0/862419573 shutdown_connections 2024-04-22T23:50:35.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.490+0000 7fe7f2282700 1 -- 172.21.15.67:0/862419573 wait complete. 2024-04-22T23:50:35.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.490+0000 7fe7f2282700 1 Processor -- start 2024-04-22T23:50:35.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.490+0000 7fe7f2282700 1 -- start start 2024-04-22T23:50:35.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.491+0000 7fe7f2282700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7ec0ff9f0 0x7fe7ec19e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:35.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.491+0000 7fe7f2282700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe7ec100310 0x7fe7ec19eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:35.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.491+0000 7fe7f2282700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ec19f430 con 0x7fe7ec0ff9f0 2024-04-22T23:50:35.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.491+0000 7fe7f2282700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ec1988e0 con 0x7fe7ec100310 2024-04-22T23:50:35.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.491+0000 7fe7ebfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7ec0ff9f0 0x7fe7ec19e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:35.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.491+0000 7fe7ebfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7ec0ff9f0 0x7fe7ec19e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49852/0 (socket says 172.21.15.67:49852) 2024-04-22T23:50:35.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.491+0000 7fe7ebfff700 1 -- 172.21.15.67:0/3748418686 learned_addr learned my addr 172.21.15.67:0/3748418686 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:50:35.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.491+0000 7fe7eb7fe700 1 --2- 172.21.15.67:0/3748418686 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe7ec100310 0x7fe7ec19eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:35.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.491+0000 7fe7ebfff700 1 -- 172.21.15.67:0/3748418686 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe7ec100310 msgr2=0x7fe7ec19eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:35.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.491+0000 7fe7ebfff700 1 --2- 172.21.15.67:0/3748418686 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe7ec100310 0x7fe7ec19eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:35.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.491+0000 7fe7ebfff700 1 -- 172.21.15.67:0/3748418686 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7dc0095f0 con 0x7fe7ec0ff9f0 2024-04-22T23:50:35.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.491+0000 7fe7eb7fe700 1 --2- 172.21.15.67:0/3748418686 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe7ec100310 0x7fe7ec19eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:50:35.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.492+0000 7fe7ebfff700 1 --2- 172.21.15.67:0/3748418686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7ec0ff9f0 0x7fe7ec19e860 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fe7e000d8d0 tx=0x7fe7e000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:50:35.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.492+0000 7fe7e97fa700 1 -- 172.21.15.67:0/3748418686 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe7e0009940 con 0x7fe7ec0ff9f0 2024-04-22T23:50:35.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.492+0000 7fe7e97fa700 1 -- 172.21.15.67:0/3748418686 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe7e0010460 con 0x7fe7ec0ff9f0 2024-04-22T23:50:35.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.492+0000 7fe7e97fa700 1 -- 172.21.15.67:0/3748418686 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe7e000f5f0 con 0x7fe7ec0ff9f0 2024-04-22T23:50:35.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.492+0000 7fe7f2282700 1 -- 172.21.15.67:0/3748418686 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7ec198bc0 con 0x7fe7ec0ff9f0 2024-04-22T23:50:35.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.492+0000 7fe7f2282700 1 -- 172.21.15.67:0/3748418686 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7ec199110 con 0x7fe7ec0ff9f0 2024-04-22T23:50:35.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:35 smithi134 ceph-mon[85630]: from='client.35328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:50:35.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:35 smithi134 ceph-mon[85630]: from='client.35332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:50:35.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:35 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/707668918' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:50:35.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.493+0000 7fe7f2282700 1 -- 172.21.15.67:0/3748418686 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7ec10a7d0 con 0x7fe7ec0ff9f0 2024-04-22T23:50:35.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.493+0000 7fe7e97fa700 1 -- 172.21.15.67:0/3748418686 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe7e00105d0 con 0x7fe7ec0ff9f0 2024-04-22T23:50:35.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.494+0000 7fe7e97fa700 1 --2- 172.21.15.67:0/3748418686 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe7d406ecc0 0x7fe7d4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:50:35.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.494+0000 7fe7e97fa700 1 -- 172.21.15.67:0/3748418686 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fe7e008eee0 con 0x7fe7ec0ff9f0 2024-04-22T23:50:35.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.494+0000 7fe7eb7fe700 1 --2- 172.21.15.67:0/3748418686 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe7d406ecc0 0x7fe7d4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:50:35.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.494+0000 7fe7eb7fe700 1 --2- 172.21.15.67:0/3748418686 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe7d406ecc0 0x7fe7d4071180 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc005b40 tx=0x7fe7dc005a90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:50:35.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.499+0000 7fe7e97fa700 1 -- 172.21.15.67:0/3748418686 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe7e005b6d0 con 0x7fe7ec0ff9f0 2024-04-22T23:50:35.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.794+0000 7fe7f2282700 1 -- 172.21.15.67:0/3748418686 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe7ec04f2e0 con 0x7fe7ec0ff9f0 2024-04-22T23:50:35.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.795+0000 7fe7e97fa700 1 -- 172.21.15.67:0/3748418686 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4505 (secure 0 0 0) 0x7fe7e005b260 con 0x7fe7ec0ff9f0 2024-04-22T23:50:35.786 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:50:35.786 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T23:50:35.786 INFO:teuthology.orchestra.run.smithi067.stdout: host smithi134 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-22T23:50:35.787 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T23:50:35.787 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:50:35.787 INFO:teuthology.orchestra.run.smithi067.stdout:Traceback (most recent call last): 2024-04-22T23:50:35.787 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T23:50:35.787 INFO:teuthology.orchestra.run.smithi067.stdout: return _run_code(code, main_globals, None, 2024-04-22T23:50:35.787 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T23:50:35.787 INFO:teuthology.orchestra.run.smithi067.stdout: exec(code, run_globals) 2024-04-22T23:50:35.787 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5642, in 2024-04-22T23:50:35.787 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5630, in main 2024-04-22T23:50:35.787 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 408, in _infer_config 2024-04-22T23:50:35.787 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 323, in _infer_fsid 2024-04-22T23:50:35.787 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 436, in _infer_image 2024-04-22T23:50:35.787 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 310, in _validate_fsid 2024-04-22T23:50:35.787 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 3380, in command_ceph_volume 2024-04-22T23:50:35.788 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T23:50:35.788 INFO:teuthology.orchestra.run.smithi067.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:50:35.788 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:50:35.788 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:50:35.788 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:50:35.788 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:50:35.788 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:50:35.788 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:50:35.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.797+0000 7fe7f2282700 1 -- 172.21.15.67:0/3748418686 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe7d406ecc0 msgr2=0x7fe7d4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:35.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.797+0000 7fe7f2282700 1 --2- 172.21.15.67:0/3748418686 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe7d406ecc0 0x7fe7d4071180 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc005b40 tx=0x7fe7dc005a90 comp rx=0 tx=0).stop 2024-04-22T23:50:35.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.797+0000 7fe7f2282700 1 -- 172.21.15.67:0/3748418686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7ec0ff9f0 msgr2=0x7fe7ec19e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:50:35.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.797+0000 7fe7f2282700 1 --2- 172.21.15.67:0/3748418686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7ec0ff9f0 0x7fe7ec19e860 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fe7e000d8d0 tx=0x7fe7e000dc90 comp rx=0 tx=0).stop 2024-04-22T23:50:35.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.797+0000 7fe7f2282700 1 -- 172.21.15.67:0/3748418686 shutdown_connections 2024-04-22T23:50:35.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.797+0000 7fe7f2282700 1 --2- 172.21.15.67:0/3748418686 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe7d406ecc0 0x7fe7d4071180 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:35.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.797+0000 7fe7f2282700 1 --2- 172.21.15.67:0/3748418686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7ec0ff9f0 0x7fe7ec19e860 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:35.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.797+0000 7fe7f2282700 1 --2- 172.21.15.67:0/3748418686 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe7ec100310 0x7fe7ec19eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:50:35.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.797+0000 7fe7f2282700 1 -- 172.21.15.67:0/3748418686 >> 172.21.15.67:0/3748418686 conn(0x7fe7ec0fb650 msgr2=0x7fe7ec0fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:50:35.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.797+0000 7fe7f2282700 1 -- 172.21.15.67:0/3748418686 shutdown_connections 2024-04-22T23:50:35.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:50:35.797+0000 7fe7f2282700 1 -- 172.21.15.67:0/3748418686 wait complete. 2024-04-22T23:50:36.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:36 smithi067 ceph-mon[112438]: from='client.35336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:50:36.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:36 smithi067 ceph-mon[112438]: pgmap v728: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:36.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:36 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1709243339' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:50:36.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:36 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3748418686' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:50:36.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:36 smithi134 ceph-mon[85630]: from='client.35336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:50:36.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:36 smithi134 ceph-mon[85630]: pgmap v728: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:36.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:36 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1709243339' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:50:36.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:36 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3748418686' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:50:37.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:37 smithi067 ceph-mon[112438]: from='client.35348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:50:37.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:37 smithi134 ceph-mon[85630]: from='client.35348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:50:38.287 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:38 smithi067 ceph-mon[112438]: pgmap v729: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:38.298 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:38 smithi134 ceph-mon[85630]: pgmap v729: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:40.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:40 smithi067 ceph-mon[112438]: pgmap v730: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:40.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:40 smithi134 ceph-mon[85630]: pgmap v730: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:42.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:42 smithi067 ceph-mon[112438]: pgmap v731: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:42.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:42 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:50:42.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:42 smithi134 ceph-mon[85630]: pgmap v731: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:42.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:42 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:50:44.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:44 smithi067 ceph-mon[112438]: pgmap v732: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:44.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:44 smithi134 ceph-mon[85630]: pgmap v732: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:46.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:46 smithi067 ceph-mon[112438]: pgmap v733: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:46.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:46 smithi134 ceph-mon[85630]: pgmap v733: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:48.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:48 smithi067 ceph-mon[112438]: pgmap v734: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:48.286 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:48 smithi134 ceph-mon[85630]: pgmap v734: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:50.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:50 smithi134 ceph-mon[85630]: pgmap v735: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:50.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:50 smithi067 ceph-mon[112438]: pgmap v735: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:52.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:52 smithi134 ceph-mon[85630]: pgmap v736: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:52.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:52 smithi067 ceph-mon[112438]: pgmap v736: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:54.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:54 smithi134 ceph-mon[85630]: pgmap v737: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:54.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:54 smithi067 ceph-mon[112438]: pgmap v737: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:56.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:56 smithi134 ceph-mon[85630]: pgmap v738: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:56.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:50:56.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:56 smithi067 ceph-mon[112438]: pgmap v738: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:56.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:50:58.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:50:58 smithi134 ceph-mon[85630]: pgmap v739: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:50:58.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:50:58 smithi067 ceph-mon[112438]: pgmap v739: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:00.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:00 smithi134 ceph-mon[85630]: pgmap v740: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:00.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:00 smithi067 ceph-mon[112438]: pgmap v740: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:01.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:01 smithi134 ceph-mon[85630]: pgmap v741: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:01.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:01 smithi067 ceph-mon[112438]: pgmap v741: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:04.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:04 smithi067 ceph-mon[112438]: pgmap v742: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:04.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:04 smithi134 ceph-mon[85630]: pgmap v742: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:05.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:05.918+0000 7fba79f30700 1 -- 172.21.15.67:0/3195430269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba740ff5a0 msgr2=0x7fba740ff980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:05.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:05.918+0000 7fba79f30700 1 --2- 172.21.15.67:0/3195430269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba740ff5a0 0x7fba740ff980 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fba64009910 tx=0x7fba64009c20 comp rx=0 tx=0).stop 2024-04-22T23:51:05.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:05.919+0000 7fba79f30700 1 -- 172.21.15.67:0/3195430269 shutdown_connections 2024-04-22T23:51:05.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:05.919+0000 7fba79f30700 1 --2- 172.21.15.67:0/3195430269 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba740fff50 0x7fba74105840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:05.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:05.919+0000 7fba79f30700 1 --2- 172.21.15.67:0/3195430269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba740ff5a0 0x7fba740ff980 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:05.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:05.919+0000 7fba79f30700 1 -- 172.21.15.67:0/3195430269 >> 172.21.15.67:0/3195430269 conn(0x7fba74074e30 msgr2=0x7fba74075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:51:05.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:05.919+0000 7fba79f30700 1 -- 172.21.15.67:0/3195430269 shutdown_connections 2024-04-22T23:51:05.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:05.919+0000 7fba79f30700 1 -- 172.21.15.67:0/3195430269 wait complete. 2024-04-22T23:51:05.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:05.919+0000 7fba79f30700 1 Processor -- start 2024-04-22T23:51:05.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:05.919+0000 7fba79f30700 1 -- start start 2024-04-22T23:51:05.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:05.920+0000 7fba79f30700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba740ff5a0 0x7fba74100520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:05.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:05.920+0000 7fba79f30700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba740fff50 0x7fba74100a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:05.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:05.920+0000 7fba79f30700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba74100fa0 con 0x7fba740fff50 2024-04-22T23:51:05.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:05.920+0000 7fba79f30700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba741010e0 con 0x7fba740ff5a0 2024-04-22T23:51:05.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:05.920+0000 7fba72ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba740fff50 0x7fba74100a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:05.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:05.920+0000 7fba72ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba740fff50 0x7fba74100a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36478/0 (socket says 172.21.15.67:36478) 2024-04-22T23:51:05.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:05.920+0000 7fba72ffd700 1 -- 172.21.15.67:0/1391243316 learned_addr learned my addr 172.21.15.67:0/1391243316 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:51:05.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:05.920+0000 7fba737fe700 1 --2- 172.21.15.67:0/1391243316 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba740ff5a0 0x7fba74100520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:05.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:05.920+0000 7fba72ffd700 1 -- 172.21.15.67:0/1391243316 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba740ff5a0 msgr2=0x7fba74100520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:05.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:05.920+0000 7fba72ffd700 1 --2- 172.21.15.67:0/1391243316 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba740ff5a0 0x7fba74100520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:05.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:05.920+0000 7fba72ffd700 1 -- 172.21.15.67:0/1391243316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba640095f0 con 0x7fba740fff50 2024-04-22T23:51:05.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:05.920+0000 7fba737fe700 1 --2- 172.21.15.67:0/1391243316 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba740ff5a0 0x7fba74100520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:51:05.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:05.920+0000 7fba72ffd700 1 --2- 172.21.15.67:0/1391243316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba740fff50 0x7fba74100a60 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fba6800e8f0 tx=0x7fba6800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:51:05.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:05.921+0000 7fba70ff9700 1 -- 172.21.15.67:0/1391243316 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba68009790 con 0x7fba740fff50 2024-04-22T23:51:05.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:05.921+0000 7fba70ff9700 1 -- 172.21.15.67:0/1391243316 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fba68004cf0 con 0x7fba740fff50 2024-04-22T23:51:05.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:05.921+0000 7fba79f30700 1 -- 172.21.15.67:0/1391243316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba741a2b00 con 0x7fba740fff50 2024-04-22T23:51:05.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:05.921+0000 7fba70ff9700 1 -- 172.21.15.67:0/1391243316 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba68005690 con 0x7fba740fff50 2024-04-22T23:51:05.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:05.921+0000 7fba79f30700 1 -- 172.21.15.67:0/1391243316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba741a2f10 con 0x7fba740fff50 2024-04-22T23:51:05.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:05.922+0000 7fba79f30700 1 -- 172.21.15.67:0/1391243316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba74109270 con 0x7fba740fff50 2024-04-22T23:51:05.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:05.922+0000 7fba70ff9700 1 -- 172.21.15.67:0/1391243316 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fba68004750 con 0x7fba740fff50 2024-04-22T23:51:05.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:05.923+0000 7fba70ff9700 1 --2- 172.21.15.67:0/1391243316 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fba5c06ecc0 0x7fba5c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:05.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:05.923+0000 7fba737fe700 1 --2- 172.21.15.67:0/1391243316 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fba5c06ecc0 0x7fba5c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:05.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:05.923+0000 7fba70ff9700 1 -- 172.21.15.67:0/1391243316 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fba68014070 con 0x7fba740fff50 2024-04-22T23:51:05.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:05.923+0000 7fba737fe700 1 --2- 172.21.15.67:0/1391243316 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fba5c06ecc0 0x7fba5c071180 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fba64000c00 tx=0x7fba64009f70 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:51:05.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:05.928+0000 7fba70ff9700 1 -- 172.21.15.67:0/1391243316 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fba68058030 con 0x7fba740fff50 2024-04-22T23:51:06.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.158+0000 7fba79f30700 1 -- 172.21.15.67:0/1391243316 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fba740611d0 con 0x7fba5c06ecc0 2024-04-22T23:51:06.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.160+0000 7fba70ff9700 1 -- 172.21.15.67:0/1391243316 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fba740611d0 con 0x7fba5c06ecc0 2024-04-22T23:51:06.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.162+0000 7fba79f30700 1 -- 172.21.15.67:0/1391243316 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fba5c06ecc0 msgr2=0x7fba5c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:06.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.162+0000 7fba79f30700 1 --2- 172.21.15.67:0/1391243316 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fba5c06ecc0 0x7fba5c071180 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fba64000c00 tx=0x7fba64009f70 comp rx=0 tx=0).stop 2024-04-22T23:51:06.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.162+0000 7fba79f30700 1 -- 172.21.15.67:0/1391243316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba740fff50 msgr2=0x7fba74100a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:06.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.162+0000 7fba79f30700 1 --2- 172.21.15.67:0/1391243316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba740fff50 0x7fba74100a60 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fba6800e8f0 tx=0x7fba6800ecb0 comp rx=0 tx=0).stop 2024-04-22T23:51:06.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.162+0000 7fba79f30700 1 -- 172.21.15.67:0/1391243316 shutdown_connections 2024-04-22T23:51:06.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.162+0000 7fba79f30700 1 --2- 172.21.15.67:0/1391243316 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fba5c06ecc0 0x7fba5c071180 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:06.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.162+0000 7fba79f30700 1 --2- 172.21.15.67:0/1391243316 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba740ff5a0 0x7fba74100520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:06.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.162+0000 7fba79f30700 1 --2- 172.21.15.67:0/1391243316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba740fff50 0x7fba74100a60 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:06.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.162+0000 7fba79f30700 1 -- 172.21.15.67:0/1391243316 >> 172.21.15.67:0/1391243316 conn(0x7fba74074e30 msgr2=0x7fba740fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:51:06.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.162+0000 7fba79f30700 1 -- 172.21.15.67:0/1391243316 shutdown_connections 2024-04-22T23:51:06.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.162+0000 7fba79f30700 1 -- 172.21.15.67:0/1391243316 wait complete. 2024-04-22T23:51:06.164 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:51:06.164 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:06 smithi067 ceph-mon[112438]: pgmap v743: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:06.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.269+0000 7f7efcb4c700 1 -- 172.21.15.67:0/4127714595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ef8100750 msgr2=0x7f7ef81047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:06.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.269+0000 7f7efcb4c700 1 --2- 172.21.15.67:0/4127714595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ef8100750 0x7f7ef81047a0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f7eec009910 tx=0x7f7eec009c20 comp rx=0 tx=0).stop 2024-04-22T23:51:06.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.270+0000 7f7efcb4c700 1 -- 172.21.15.67:0/4127714595 shutdown_connections 2024-04-22T23:51:06.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.270+0000 7f7efcb4c700 1 --2- 172.21.15.67:0/4127714595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ef8100750 0x7f7ef81047a0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:06.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.270+0000 7f7efcb4c700 1 --2- 172.21.15.67:0/4127714595 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7ef80ffda0 0x7f7ef8100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:06.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.270+0000 7f7efcb4c700 1 -- 172.21.15.67:0/4127714595 >> 172.21.15.67:0/4127714595 conn(0x7f7ef80fb650 msgr2=0x7f7ef80fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:51:06.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.270+0000 7f7efcb4c700 1 -- 172.21.15.67:0/4127714595 shutdown_connections 2024-04-22T23:51:06.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.270+0000 7f7efcb4c700 1 -- 172.21.15.67:0/4127714595 wait complete. 2024-04-22T23:51:06.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.270+0000 7f7efcb4c700 1 Processor -- start 2024-04-22T23:51:06.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.270+0000 7f7efcb4c700 1 -- start start 2024-04-22T23:51:06.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.271+0000 7f7efcb4c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7ef80ffda0 0x7f7ef8101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:06.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.271+0000 7f7efcb4c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ef8100750 0x7f7ef8101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:06.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.271+0000 7f7efcb4c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef81022c0 con 0x7f7ef8100750 2024-04-22T23:51:06.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.271+0000 7f7efcb4c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef8102400 con 0x7f7ef80ffda0 2024-04-22T23:51:06.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.271+0000 7f7ef5d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ef8100750 0x7f7ef8101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:06.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.271+0000 7f7ef5d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ef8100750 0x7f7ef8101d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36488/0 (socket says 172.21.15.67:36488) 2024-04-22T23:51:06.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.271+0000 7f7ef5d9b700 1 -- 172.21.15.67:0/466648471 learned_addr learned my addr 172.21.15.67:0/466648471 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:51:06.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.271+0000 7f7ef659c700 1 --2- 172.21.15.67:0/466648471 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7ef80ffda0 0x7f7ef8101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:06.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.271+0000 7f7ef5d9b700 1 -- 172.21.15.67:0/466648471 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7ef80ffda0 msgr2=0x7f7ef8101840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:06.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.271+0000 7f7ef5d9b700 1 --2- 172.21.15.67:0/466648471 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7ef80ffda0 0x7f7ef8101840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:06.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.271+0000 7f7ef5d9b700 1 -- 172.21.15.67:0/466648471 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eec0095f0 con 0x7f7ef8100750 2024-04-22T23:51:06.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.271+0000 7f7ef5d9b700 1 --2- 172.21.15.67:0/466648471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ef8100750 0x7f7ef8101d80 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f7eec0092e0 tx=0x7f7eec00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:51:06.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.271+0000 7f7ee77fe700 1 -- 172.21.15.67:0/466648471 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7eec01d070 con 0x7f7ef8100750 2024-04-22T23:51:06.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.271+0000 7f7efcb4c700 1 -- 172.21.15.67:0/466648471 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ef8102680 con 0x7f7ef8100750 2024-04-22T23:51:06.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.271+0000 7f7ee77fe700 1 -- 172.21.15.67:0/466648471 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7eec005dc0 con 0x7f7ef8100750 2024-04-22T23:51:06.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.272+0000 7f7ee77fe700 1 -- 172.21.15.67:0/466648471 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7eec017400 con 0x7f7ef8100750 2024-04-22T23:51:06.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.272+0000 7f7efcb4c700 1 -- 172.21.15.67:0/466648471 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ef81a2da0 con 0x7f7ef8100750 2024-04-22T23:51:06.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.272+0000 7f7efcb4c700 1 -- 172.21.15.67:0/466648471 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ef81091b0 con 0x7f7ef8100750 2024-04-22T23:51:06.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.273+0000 7f7ee77fe700 1 -- 172.21.15.67:0/466648471 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7eec00f6f0 con 0x7f7ef8100750 2024-04-22T23:51:06.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.274+0000 7f7ee77fe700 1 --2- 172.21.15.67:0/466648471 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7ee006ecc0 0x7f7ee0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:06.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.274+0000 7f7ef659c700 1 --2- 172.21.15.67:0/466648471 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7ee006ecc0 0x7f7ee0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:06.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.274+0000 7f7ee77fe700 1 -- 172.21.15.67:0/466648471 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f7eec0903c0 con 0x7f7ef8100750 2024-04-22T23:51:06.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.274+0000 7f7ef659c700 1 --2- 172.21.15.67:0/466648471 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7ee006ecc0 0x7f7ee0071180 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8005950 tx=0x7f7ee800b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:51:06.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.279+0000 7f7ee77fe700 1 -- 172.21.15.67:0/466648471 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7eec059000 con 0x7f7ef8100750 2024-04-22T23:51:06.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:06 smithi134 ceph-mon[85630]: pgmap v743: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:06.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.509+0000 7f7efcb4c700 1 -- 172.21.15.67:0/466648471 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ef802cce0 con 0x7f7ee006ecc0 2024-04-22T23:51:06.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.511+0000 7f7ee77fe700 1 -- 172.21.15.67:0/466648471 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f7ef802cce0 con 0x7f7ee006ecc0 2024-04-22T23:51:06.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.513+0000 7f7efcb4c700 1 -- 172.21.15.67:0/466648471 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7ee006ecc0 msgr2=0x7f7ee0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:06.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.513+0000 7f7efcb4c700 1 --2- 172.21.15.67:0/466648471 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7ee006ecc0 0x7f7ee0071180 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8005950 tx=0x7f7ee800b410 comp rx=0 tx=0).stop 2024-04-22T23:51:06.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.513+0000 7f7efcb4c700 1 -- 172.21.15.67:0/466648471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ef8100750 msgr2=0x7f7ef8101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:06.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.513+0000 7f7efcb4c700 1 --2- 172.21.15.67:0/466648471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ef8100750 0x7f7ef8101d80 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f7eec0092e0 tx=0x7f7eec00bd70 comp rx=0 tx=0).stop 2024-04-22T23:51:06.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.513+0000 7f7efcb4c700 1 -- 172.21.15.67:0/466648471 shutdown_connections 2024-04-22T23:51:06.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.513+0000 7f7efcb4c700 1 --2- 172.21.15.67:0/466648471 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7ee006ecc0 0x7f7ee0071180 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:06.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.513+0000 7f7efcb4c700 1 --2- 172.21.15.67:0/466648471 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7ef80ffda0 0x7f7ef8101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:06.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.513+0000 7f7efcb4c700 1 --2- 172.21.15.67:0/466648471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ef8100750 0x7f7ef8101d80 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:06.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.513+0000 7f7efcb4c700 1 -- 172.21.15.67:0/466648471 >> 172.21.15.67:0/466648471 conn(0x7f7ef80fb650 msgr2=0x7f7ef8069150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:51:06.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.513+0000 7f7efcb4c700 1 -- 172.21.15.67:0/466648471 shutdown_connections 2024-04-22T23:51:06.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.513+0000 7f7efcb4c700 1 -- 172.21.15.67:0/466648471 wait complete. 2024-04-22T23:51:06.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.619+0000 7f2c0d63f700 1 -- 172.21.15.67:0/2343756192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c08101850 msgr2=0x7f2c081058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:06.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.619+0000 7f2c0d63f700 1 --2- 172.21.15.67:0/2343756192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c08101850 0x7f2c081058a0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f2bf8009910 tx=0x7f2bf8009c20 comp rx=0 tx=0).stop 2024-04-22T23:51:06.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.619+0000 7f2c0d63f700 1 -- 172.21.15.67:0/2343756192 shutdown_connections 2024-04-22T23:51:06.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.619+0000 7f2c0d63f700 1 --2- 172.21.15.67:0/2343756192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c08101850 0x7f2c081058a0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:06.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.619+0000 7f2c0d63f700 1 --2- 172.21.15.67:0/2343756192 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c08100ea0 0x7f2c08101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:06.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.619+0000 7f2c0d63f700 1 -- 172.21.15.67:0/2343756192 >> 172.21.15.67:0/2343756192 conn(0x7f2c080fc750 msgr2=0x7f2c080feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:51:06.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.619+0000 7f2c0d63f700 1 -- 172.21.15.67:0/2343756192 shutdown_connections 2024-04-22T23:51:06.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.620+0000 7f2c0d63f700 1 -- 172.21.15.67:0/2343756192 wait complete. 2024-04-22T23:51:06.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.620+0000 7f2c0d63f700 1 Processor -- start 2024-04-22T23:51:06.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.620+0000 7f2c0d63f700 1 -- start start 2024-04-22T23:51:06.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.620+0000 7f2c0d63f700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c08100ea0 0x7f2c08073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:06.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.621+0000 7f2c0d63f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c08101850 0x7f2c080735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:06.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.621+0000 7f2c0d63f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c08073b10 con 0x7f2c08101850 2024-04-22T23:51:06.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.621+0000 7f2c0d63f700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c08073c50 con 0x7f2c08100ea0 2024-04-22T23:51:06.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.621+0000 7f2c067fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c08101850 0x7f2c080735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:06.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.621+0000 7f2c067fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c08101850 0x7f2c080735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36506/0 (socket says 172.21.15.67:36506) 2024-04-22T23:51:06.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.621+0000 7f2c067fc700 1 -- 172.21.15.67:0/2295943984 learned_addr learned my addr 172.21.15.67:0/2295943984 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:51:06.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.621+0000 7f2c06ffd700 1 --2- 172.21.15.67:0/2295943984 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c08100ea0 0x7f2c08073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:06.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.621+0000 7f2c067fc700 1 -- 172.21.15.67:0/2295943984 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c08100ea0 msgr2=0x7f2c08073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:06.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.621+0000 7f2c067fc700 1 --2- 172.21.15.67:0/2295943984 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c08100ea0 0x7f2c08073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:06.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.621+0000 7f2c067fc700 1 -- 172.21.15.67:0/2295943984 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf80095f0 con 0x7f2c08101850 2024-04-22T23:51:06.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.621+0000 7f2c067fc700 1 --2- 172.21.15.67:0/2295943984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c08101850 0x7f2c080735d0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f2bf80059d0 tx=0x7f2bf800f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:51:06.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.621+0000 7f2beffff700 1 -- 172.21.15.67:0/2295943984 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2bf801d070 con 0x7f2c08101850 2024-04-22T23:51:06.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.621+0000 7f2beffff700 1 -- 172.21.15.67:0/2295943984 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2bf800b610 con 0x7f2c08101850 2024-04-22T23:51:06.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.621+0000 7f2c0d63f700 1 -- 172.21.15.67:0/2295943984 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c08073ed0 con 0x7f2c08101850 2024-04-22T23:51:06.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.621+0000 7f2beffff700 1 -- 172.21.15.67:0/2295943984 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2bf8022470 con 0x7f2c08101850 2024-04-22T23:51:06.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.621+0000 7f2c0d63f700 1 -- 172.21.15.67:0/2295943984 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c081a2e30 con 0x7f2c08101850 2024-04-22T23:51:06.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.622+0000 7f2c0d63f700 1 -- 172.21.15.67:0/2295943984 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c081092a0 con 0x7f2c08101850 2024-04-22T23:51:06.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.623+0000 7f2beffff700 1 -- 172.21.15.67:0/2295943984 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2bf800fc70 con 0x7f2c08101850 2024-04-22T23:51:06.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.623+0000 7f2beffff700 1 --2- 172.21.15.67:0/2295943984 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2bf006ecc0 0x7f2bf0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:06.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.623+0000 7f2beffff700 1 -- 172.21.15.67:0/2295943984 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f2bf8090340 con 0x7f2c08101850 2024-04-22T23:51:06.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.624+0000 7f2c06ffd700 1 --2- 172.21.15.67:0/2295943984 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2bf006ecc0 0x7f2bf0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:06.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.624+0000 7f2c06ffd700 1 --2- 172.21.15.67:0/2295943984 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2bf006ecc0 0x7f2bf0071180 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f2bfc009bf0 tx=0x7f2bfc009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:51:06.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.628+0000 7f2beffff700 1 -- 172.21.15.67:0/2295943984 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f2bf8059000 con 0x7f2c08101850 2024-04-22T23:51:06.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.852+0000 7f2c0d63f700 1 -- 172.21.15.67:0/2295943984 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2c080746d0 con 0x7f2bf006ecc0 2024-04-22T23:51:06.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.863+0000 7f2beffff700 1 -- 172.21.15.67:0/2295943984 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f2c080746d0 con 0x7f2bf006ecc0 2024-04-22T23:51:06.854 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:51:06.854 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (24m) 2m ago 36m 24.4M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:51:06.854 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (36m) 2m ago 36m 10.0M - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:51:06.854 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (35m) 2m ago 35m 10.0M - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:51:06.854 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (22m) 2m ago 36m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:51:06.855 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (22m) 2m ago 35m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:51:06.855 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (24m) 2m ago 35m 81.2M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:51:06.855 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (18m) 2m ago 30m 19.4M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:51:06.855 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (18m) 2m ago 30m 20.3M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:51:06.855 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (17m) 2m ago 30m 20.1M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:51:06.855 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (2m) 2m ago 30m 14.0M - 19.0.0-3252-g43be0201 a0434699c1d7 46d3d7e86fee 2024-04-22T23:51:06.855 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (26m) 2m ago 37m 514M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:51:06.855 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (25m) 2m ago 35m 459M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:51:06.855 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (23m) 2m ago 37m 74.2M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:51:06.855 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (22m) 2m ago 35m 54.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:51:06.855 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (25m) 2m ago 36m 15.2M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:51:06.855 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (25m) 2m ago 35m 16.0M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:51:06.855 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (21m) 2m ago 34m 102M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:51:06.856 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (21m) 2m ago 33m 128M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:51:06.856 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (20m) 2m ago 33m 83.8M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:51:06.856 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (19m) 2m ago 33m 101M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:51:06.856 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (19m) 2m ago 32m 86.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:51:06.856 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (18m) 2m ago 32m 81.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:51:06.856 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (25m) 2m ago 35m 78.7M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:51:06.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.865+0000 7f2c0d63f700 1 -- 172.21.15.67:0/2295943984 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2bf006ecc0 msgr2=0x7f2bf0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:06.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.865+0000 7f2c0d63f700 1 --2- 172.21.15.67:0/2295943984 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2bf006ecc0 0x7f2bf0071180 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f2bfc009bf0 tx=0x7f2bfc009260 comp rx=0 tx=0).stop 2024-04-22T23:51:06.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.865+0000 7f2c0d63f700 1 -- 172.21.15.67:0/2295943984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c08101850 msgr2=0x7f2c080735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:06.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.865+0000 7f2c0d63f700 1 --2- 172.21.15.67:0/2295943984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c08101850 0x7f2c080735d0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f2bf80059d0 tx=0x7f2bf800f740 comp rx=0 tx=0).stop 2024-04-22T23:51:06.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.865+0000 7f2c0d63f700 1 -- 172.21.15.67:0/2295943984 shutdown_connections 2024-04-22T23:51:06.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.865+0000 7f2c0d63f700 1 --2- 172.21.15.67:0/2295943984 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2bf006ecc0 0x7f2bf0071180 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:06.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.865+0000 7f2c0d63f700 1 --2- 172.21.15.67:0/2295943984 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c08100ea0 0x7f2c08073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:06.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.865+0000 7f2c0d63f700 1 --2- 172.21.15.67:0/2295943984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c08101850 0x7f2c080735d0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:06.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.865+0000 7f2c0d63f700 1 -- 172.21.15.67:0/2295943984 >> 172.21.15.67:0/2295943984 conn(0x7f2c080fc750 msgr2=0x7f2c080fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:51:06.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.865+0000 7f2c0d63f700 1 -- 172.21.15.67:0/2295943984 shutdown_connections 2024-04-22T23:51:06.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.865+0000 7f2c0d63f700 1 -- 172.21.15.67:0/2295943984 wait complete. 2024-04-22T23:51:06.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.970+0000 7f8a73d35700 1 -- 172.21.15.67:0/1518803812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a6c1027a0 msgr2=0x7f8a6c10ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:06.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.970+0000 7f8a73d35700 1 --2- 172.21.15.67:0/1518803812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a6c1027a0 0x7f8a6c10ac90 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c009910 tx=0x7f8a5c009c20 comp rx=0 tx=0).stop 2024-04-22T23:51:06.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.970+0000 7f8a73d35700 1 -- 172.21.15.67:0/1518803812 shutdown_connections 2024-04-22T23:51:06.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.970+0000 7f8a73d35700 1 --2- 172.21.15.67:0/1518803812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a6c1027a0 0x7f8a6c10ac90 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:06.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.970+0000 7f8a73d35700 1 --2- 172.21.15.67:0/1518803812 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a6c101e80 0x7f8a6c102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:06.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.970+0000 7f8a73d35700 1 -- 172.21.15.67:0/1518803812 >> 172.21.15.67:0/1518803812 conn(0x7f8a6c0fb650 msgr2=0x7f8a6c0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:51:06.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.971+0000 7f8a73d35700 1 -- 172.21.15.67:0/1518803812 shutdown_connections 2024-04-22T23:51:06.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.971+0000 7f8a73d35700 1 -- 172.21.15.67:0/1518803812 wait complete. 2024-04-22T23:51:06.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.971+0000 7f8a73d35700 1 Processor -- start 2024-04-22T23:51:06.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.971+0000 7f8a73d35700 1 -- start start 2024-04-22T23:51:06.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.971+0000 7f8a73d35700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a6c101e80 0x7f8a6c19c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:06.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.971+0000 7f8a73d35700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a6c1027a0 0x7f8a6c19cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:06.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.971+0000 7f8a73d35700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a6c19d220 con 0x7f8a6c101e80 2024-04-22T23:51:06.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.971+0000 7f8a73d35700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a6c1966d0 con 0x7f8a6c1027a0 2024-04-22T23:51:06.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.972+0000 7f8a71ad1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a6c101e80 0x7f8a6c19c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:06.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.972+0000 7f8a71ad1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a6c101e80 0x7f8a6c19c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36526/0 (socket says 172.21.15.67:36526) 2024-04-22T23:51:06.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.972+0000 7f8a71ad1700 1 -- 172.21.15.67:0/919267788 learned_addr learned my addr 172.21.15.67:0/919267788 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:51:06.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.972+0000 7f8a712d0700 1 --2- 172.21.15.67:0/919267788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a6c1027a0 0x7f8a6c19cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:06.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.972+0000 7f8a71ad1700 1 -- 172.21.15.67:0/919267788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a6c1027a0 msgr2=0x7f8a6c19cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:06.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.972+0000 7f8a71ad1700 1 --2- 172.21.15.67:0/919267788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a6c1027a0 0x7f8a6c19cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:06.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.972+0000 7f8a71ad1700 1 -- 172.21.15.67:0/919267788 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a5c0095f0 con 0x7f8a6c101e80 2024-04-22T23:51:06.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.972+0000 7f8a712d0700 1 --2- 172.21.15.67:0/919267788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a6c1027a0 0x7f8a6c19cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:51:06.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.972+0000 7f8a71ad1700 1 --2- 172.21.15.67:0/919267788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a6c101e80 0x7f8a6c19c650 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f8a6800d8d0 tx=0x7f8a6800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:51:06.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.972+0000 7f8a62ffd700 1 -- 172.21.15.67:0/919267788 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a68009940 con 0x7f8a6c101e80 2024-04-22T23:51:06.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.972+0000 7f8a73d35700 1 -- 172.21.15.67:0/919267788 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a6c1969b0 con 0x7f8a6c101e80 2024-04-22T23:51:06.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.972+0000 7f8a73d35700 1 -- 172.21.15.67:0/919267788 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a6c196f00 con 0x7f8a6c101e80 2024-04-22T23:51:06.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.973+0000 7f8a62ffd700 1 -- 172.21.15.67:0/919267788 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8a68010460 con 0x7f8a6c101e80 2024-04-22T23:51:06.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.973+0000 7f8a62ffd700 1 -- 172.21.15.67:0/919267788 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a6800f5f0 con 0x7f8a6c101e80 2024-04-22T23:51:06.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.973+0000 7f8a73d35700 1 -- 172.21.15.67:0/919267788 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a50005320 con 0x7f8a6c101e80 2024-04-22T23:51:06.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.974+0000 7f8a62ffd700 1 -- 172.21.15.67:0/919267788 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8a68010ad0 con 0x7f8a6c101e80 2024-04-22T23:51:06.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.975+0000 7f8a62ffd700 1 --2- 172.21.15.67:0/919267788 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8a5806ecc0 0x7f8a58071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:06.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.975+0000 7f8a62ffd700 1 -- 172.21.15.67:0/919267788 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f8a6808ee30 con 0x7f8a6c101e80 2024-04-22T23:51:06.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.975+0000 7f8a712d0700 1 --2- 172.21.15.67:0/919267788 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8a5806ecc0 0x7f8a58071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:06.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.975+0000 7f8a712d0700 1 --2- 172.21.15.67:0/919267788 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8a5806ecc0 0x7f8a58071180 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c197b40 tx=0x7f8a5c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:51:06.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:06.979+0000 7f8a62ffd700 1 -- 172.21.15.67:0/919267788 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f8a6805b770 con 0x7f8a6c101e80 2024-04-22T23:51:07.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.280+0000 7f8a73d35700 1 -- 172.21.15.67:0/919267788 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8a50005cc0 con 0x7f8a6c101e80 2024-04-22T23:51:07.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.281+0000 7f8a62ffd700 1 -- 172.21.15.67:0/919267788 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f8a6805b300 con 0x7f8a6c101e80 2024-04-22T23:51:07.272 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:51:07.273 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:51:07.273 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:51:07.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:51:07.273 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:51:07.273 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:51:07.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:51:07.273 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:51:07.273 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:51:07.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:51:07.273 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:51:07.273 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 4 2024-04-22T23:51:07.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:51:07.273 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:51:07.274 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 14 2024-04-22T23:51:07.274 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:51:07.274 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:51:07.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.283+0000 7f8a73d35700 1 -- 172.21.15.67:0/919267788 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8a5806ecc0 msgr2=0x7f8a58071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:07.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.283+0000 7f8a73d35700 1 --2- 172.21.15.67:0/919267788 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8a5806ecc0 0x7f8a58071180 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c197b40 tx=0x7f8a5c01a040 comp rx=0 tx=0).stop 2024-04-22T23:51:07.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.283+0000 7f8a73d35700 1 -- 172.21.15.67:0/919267788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a6c101e80 msgr2=0x7f8a6c19c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:07.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.283+0000 7f8a73d35700 1 --2- 172.21.15.67:0/919267788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a6c101e80 0x7f8a6c19c650 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f8a6800d8d0 tx=0x7f8a6800dc90 comp rx=0 tx=0).stop 2024-04-22T23:51:07.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.284+0000 7f8a73d35700 1 -- 172.21.15.67:0/919267788 shutdown_connections 2024-04-22T23:51:07.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.284+0000 7f8a73d35700 1 --2- 172.21.15.67:0/919267788 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8a5806ecc0 0x7f8a58071180 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:07.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.284+0000 7f8a73d35700 1 --2- 172.21.15.67:0/919267788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a6c101e80 0x7f8a6c19c650 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:07.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.284+0000 7f8a73d35700 1 --2- 172.21.15.67:0/919267788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a6c1027a0 0x7f8a6c19cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:07.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.284+0000 7f8a73d35700 1 -- 172.21.15.67:0/919267788 >> 172.21.15.67:0/919267788 conn(0x7f8a6c0fb650 msgr2=0x7f8a6c105460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:51:07.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.284+0000 7f8a73d35700 1 -- 172.21.15.67:0/919267788 shutdown_connections 2024-04-22T23:51:07.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.284+0000 7f8a73d35700 1 -- 172.21.15.67:0/919267788 wait complete. 2024-04-22T23:51:07.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.392+0000 7f16c2c12700 1 -- 172.21.15.67:0/2164675555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16bc101850 msgr2=0x7f16bc1058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:07.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.392+0000 7f16c2c12700 1 --2- 172.21.15.67:0/2164675555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16bc101850 0x7f16bc1058a0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f16b0009910 tx=0x7f16b0009c20 comp rx=0 tx=0).stop 2024-04-22T23:51:07.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.392+0000 7f16c2c12700 1 -- 172.21.15.67:0/2164675555 shutdown_connections 2024-04-22T23:51:07.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.392+0000 7f16c2c12700 1 --2- 172.21.15.67:0/2164675555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16bc101850 0x7f16bc1058a0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:07.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.392+0000 7f16c2c12700 1 --2- 172.21.15.67:0/2164675555 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f16bc100ea0 0x7f16bc101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:07.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.392+0000 7f16c2c12700 1 -- 172.21.15.67:0/2164675555 >> 172.21.15.67:0/2164675555 conn(0x7f16bc0fc750 msgr2=0x7f16bc0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:51:07.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.392+0000 7f16c2c12700 1 -- 172.21.15.67:0/2164675555 shutdown_connections 2024-04-22T23:51:07.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.392+0000 7f16c2c12700 1 -- 172.21.15.67:0/2164675555 wait complete. 2024-04-22T23:51:07.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.392+0000 7f16c2c12700 1 Processor -- start 2024-04-22T23:51:07.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.392+0000 7f16c2c12700 1 -- start start 2024-04-22T23:51:07.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.393+0000 7f16c2c12700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16bc100ea0 0x7f16bc19c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:07.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.393+0000 7f16c2c12700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f16bc101850 0x7f16bc19cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:07.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.393+0000 7f16c2c12700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16bc19d2d0 con 0x7f16bc100ea0 2024-04-22T23:51:07.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.393+0000 7f16c2c12700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16bc196780 con 0x7f16bc101850 2024-04-22T23:51:07.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.393+0000 7f16c09ae700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16bc100ea0 0x7f16bc19c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:07.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.393+0000 7f16c09ae700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16bc100ea0 0x7f16bc19c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36550/0 (socket says 172.21.15.67:36550) 2024-04-22T23:51:07.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.393+0000 7f16c09ae700 1 -- 172.21.15.67:0/4198933739 learned_addr learned my addr 172.21.15.67:0/4198933739 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:51:07.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.393+0000 7f16bbfff700 1 --2- 172.21.15.67:0/4198933739 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f16bc101850 0x7f16bc19cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:07.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.393+0000 7f16c09ae700 1 -- 172.21.15.67:0/4198933739 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f16bc101850 msgr2=0x7f16bc19cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:07.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.393+0000 7f16c09ae700 1 --2- 172.21.15.67:0/4198933739 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f16bc101850 0x7f16bc19cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:07.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.393+0000 7f16c09ae700 1 -- 172.21.15.67:0/4198933739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b00095f0 con 0x7f16bc100ea0 2024-04-22T23:51:07.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.393+0000 7f16c09ae700 1 --2- 172.21.15.67:0/4198933739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16bc100ea0 0x7f16bc19c700 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f16ac00c8a0 tx=0x7f16ac00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:51:07.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.393+0000 7f16b9ffb700 1 -- 172.21.15.67:0/4198933739 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16ac00cea0 con 0x7f16bc100ea0 2024-04-22T23:51:07.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.393+0000 7f16b9ffb700 1 -- 172.21.15.67:0/4198933739 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f16ac004830 con 0x7f16bc100ea0 2024-04-22T23:51:07.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.393+0000 7f16c2c12700 1 -- 172.21.15.67:0/4198933739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16bc196a60 con 0x7f16bc100ea0 2024-04-22T23:51:07.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.393+0000 7f16b9ffb700 1 -- 172.21.15.67:0/4198933739 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16ac0055b0 con 0x7f16bc100ea0 2024-04-22T23:51:07.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.394+0000 7f16c2c12700 1 -- 172.21.15.67:0/4198933739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16bc196fb0 con 0x7f16bc100ea0 2024-04-22T23:51:07.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.395+0000 7f16c2c12700 1 -- 172.21.15.67:0/4198933739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16bc1092a0 con 0x7f16bc100ea0 2024-04-22T23:51:07.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.395+0000 7f16b9ffb700 1 -- 172.21.15.67:0/4198933739 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f16ac0049a0 con 0x7f16bc100ea0 2024-04-22T23:51:07.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.395+0000 7f16b9ffb700 1 --2- 172.21.15.67:0/4198933739 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f16a406ecc0 0x7f16a4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:07.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.396+0000 7f16b9ffb700 1 -- 172.21.15.67:0/4198933739 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f16ac08ebc0 con 0x7f16bc100ea0 2024-04-22T23:51:07.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.396+0000 7f16bbfff700 1 --2- 172.21.15.67:0/4198933739 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f16a406ecc0 0x7f16a4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:07.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.396+0000 7f16bbfff700 1 --2- 172.21.15.67:0/4198933739 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f16a406ecc0 0x7f16a4071180 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f16bc197bf0 tx=0x7f16b001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:51:07.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.400+0000 7f16b9ffb700 1 -- 172.21.15.67:0/4198933739 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f16ac05b3b0 con 0x7f16bc100ea0 2024-04-22T23:51:07.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.654+0000 7f16c2c12700 1 -- 172.21.15.67:0/4198933739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f16bc04f2e0 con 0x7f16bc100ea0 2024-04-22T23:51:07.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.655+0000 7f16b9ffb700 1 -- 172.21.15.67:0/4198933739 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1872 (secure 0 0 0) 0x7f16ac01d070 con 0x7f16bc100ea0 2024-04-22T23:51:07.646 INFO:teuthology.orchestra.run.smithi067.stdout:e33 2024-04-22T23:51:07.647 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:51:07.647 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:51:07.647 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:51:07.647 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:51:07.647 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:51:07.647 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:51:07.647 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:51:07.647 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:51:07.647 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:51:07.647 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:51:07.647 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:51:07.647 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:51:07.647 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:51:07.647 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:51:07.648 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:51:07.648 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:51:07.648 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:51:07.648 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:51:07.648 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:51:07.648 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:51:07.648 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:51:07.648 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:51:07.648 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:51:07.648 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:51:07.648 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:51:07.648 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:51:07.648 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:51:07.648 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:51:07.648 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:51:07.649 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:51:07.649 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:51:07.649 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:51:07.649 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:51:07.649 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:51:07.649 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:51:07.649 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:51:07.649 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:51:07.649 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:51:07.649 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:51:07.649 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:51:07.649 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44851} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1849648622,v1:172.21.15.134:6827/1849648622] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:51:07.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.657+0000 7f16c2c12700 1 -- 172.21.15.67:0/4198933739 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f16a406ecc0 msgr2=0x7f16a4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:07.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.657+0000 7f16c2c12700 1 --2- 172.21.15.67:0/4198933739 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f16a406ecc0 0x7f16a4071180 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f16bc197bf0 tx=0x7f16b001a040 comp rx=0 tx=0).stop 2024-04-22T23:51:07.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.657+0000 7f16c2c12700 1 -- 172.21.15.67:0/4198933739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16bc100ea0 msgr2=0x7f16bc19c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:07.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.657+0000 7f16c2c12700 1 --2- 172.21.15.67:0/4198933739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16bc100ea0 0x7f16bc19c700 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f16ac00c8a0 tx=0x7f16ac00cc60 comp rx=0 tx=0).stop 2024-04-22T23:51:07.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.657+0000 7f16c2c12700 1 -- 172.21.15.67:0/4198933739 shutdown_connections 2024-04-22T23:51:07.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.657+0000 7f16c2c12700 1 --2- 172.21.15.67:0/4198933739 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f16a406ecc0 0x7f16a4071180 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:07.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.657+0000 7f16c2c12700 1 --2- 172.21.15.67:0/4198933739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16bc100ea0 0x7f16bc19c700 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:07.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.657+0000 7f16c2c12700 1 --2- 172.21.15.67:0/4198933739 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f16bc101850 0x7f16bc19cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:07.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.657+0000 7f16c2c12700 1 -- 172.21.15.67:0/4198933739 >> 172.21.15.67:0/4198933739 conn(0x7f16bc0fc750 msgr2=0x7f16bc0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:51:07.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.657+0000 7f16c2c12700 1 -- 172.21.15.67:0/4198933739 shutdown_connections 2024-04-22T23:51:07.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.657+0000 7f16c2c12700 1 -- 172.21.15.67:0/4198933739 wait complete. 2024-04-22T23:51:07.651 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 33 2024-04-22T23:51:07.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.765+0000 7f77a4ef9700 1 -- 172.21.15.67:0/984687646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77a0101de0 msgr2=0x7f77a01021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:07.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.765+0000 7f77a4ef9700 1 --2- 172.21.15.67:0/984687646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77a0101de0 0x7f77a01021c0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f779800b0a0 tx=0x7f779800b3b0 comp rx=0 tx=0).stop 2024-04-22T23:51:07.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.765+0000 7f77a4ef9700 1 -- 172.21.15.67:0/984687646 shutdown_connections 2024-04-22T23:51:07.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.765+0000 7f77a4ef9700 1 --2- 172.21.15.67:0/984687646 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77a0102700 0x7f77a010abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:07.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.765+0000 7f77a4ef9700 1 --2- 172.21.15.67:0/984687646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77a0101de0 0x7f77a01021c0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:07.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.765+0000 7f77a4ef9700 1 -- 172.21.15.67:0/984687646 >> 172.21.15.67:0/984687646 conn(0x7f77a00fb610 msgr2=0x7f77a00fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:51:07.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.765+0000 7f77a4ef9700 1 -- 172.21.15.67:0/984687646 shutdown_connections 2024-04-22T23:51:07.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.765+0000 7f77a4ef9700 1 -- 172.21.15.67:0/984687646 wait complete. 2024-04-22T23:51:07.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.766+0000 7f77a4ef9700 1 Processor -- start 2024-04-22T23:51:07.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.766+0000 7f77a4ef9700 1 -- start start 2024-04-22T23:51:07.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.766+0000 7f77a4ef9700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77a0101de0 0x7f77a019e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:07.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.766+0000 7f77a4ef9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77a0102700 0x7f77a019eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:07.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.766+0000 7f77a4ef9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a019f550 con 0x7f77a0102700 2024-04-22T23:51:07.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.766+0000 7f77a4ef9700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a0198a00 con 0x7f77a0101de0 2024-04-22T23:51:07.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.766+0000 7f779dd9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77a0102700 0x7f77a019eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:07.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.766+0000 7f779dd9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77a0102700 0x7f77a019eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36568/0 (socket says 172.21.15.67:36568) 2024-04-22T23:51:07.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.766+0000 7f779dd9b700 1 -- 172.21.15.67:0/3759133321 learned_addr learned my addr 172.21.15.67:0/3759133321 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:51:07.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.766+0000 7f779e59c700 1 --2- 172.21.15.67:0/3759133321 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77a0101de0 0x7f77a019e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:07.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.766+0000 7f779dd9b700 1 -- 172.21.15.67:0/3759133321 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77a0101de0 msgr2=0x7f77a019e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:07.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.766+0000 7f779dd9b700 1 --2- 172.21.15.67:0/3759133321 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77a0101de0 0x7f77a019e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:07.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.766+0000 7f779dd9b700 1 -- 172.21.15.67:0/3759133321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7798009d00 con 0x7f77a0102700 2024-04-22T23:51:07.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.767+0000 7f779dd9b700 1 --2- 172.21.15.67:0/3759133321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77a0102700 0x7f77a019eec0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f778800b3d0 tx=0x7f778800b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:51:07.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.767+0000 7f77977fe700 1 -- 172.21.15.67:0/3759133321 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7788011800 con 0x7f77a0102700 2024-04-22T23:51:07.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.767+0000 7f77977fe700 1 -- 172.21.15.67:0/3759133321 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7788011e20 con 0x7f77a0102700 2024-04-22T23:51:07.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.767+0000 7f77a4ef9700 1 -- 172.21.15.67:0/3759133321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77a0198ce0 con 0x7f77a0102700 2024-04-22T23:51:07.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.767+0000 7f77977fe700 1 -- 172.21.15.67:0/3759133321 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7788010ea0 con 0x7f77a0102700 2024-04-22T23:51:07.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.767+0000 7f77a4ef9700 1 -- 172.21.15.67:0/3759133321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77a0199230 con 0x7f77a0102700 2024-04-22T23:51:07.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.768+0000 7f77a4ef9700 1 -- 172.21.15.67:0/3759133321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77a010a750 con 0x7f77a0102700 2024-04-22T23:51:07.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.769+0000 7f77977fe700 1 -- 172.21.15.67:0/3759133321 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7788010380 con 0x7f77a0102700 2024-04-22T23:51:07.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.769+0000 7f77977fe700 1 --2- 172.21.15.67:0/3759133321 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f778c06ecc0 0x7f778c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:07.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.769+0000 7f77977fe700 1 -- 172.21.15.67:0/3759133321 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f778808eac0 con 0x7f77a0102700 2024-04-22T23:51:07.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.769+0000 7f779e59c700 1 --2- 172.21.15.67:0/3759133321 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f778c06ecc0 0x7f778c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:07.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.769+0000 7f779e59c700 1 --2- 172.21.15.67:0/3759133321 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f778c06ecc0 0x7f778c071180 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7798015040 tx=0x7f7798006000 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:51:07.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:07.774+0000 7f77977fe700 1 -- 172.21.15.67:0/3759133321 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f778805b230 con 0x7f77a0102700 2024-04-22T23:51:07.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.001+0000 7f77a4ef9700 1 -- 172.21.15.67:0/3759133321 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f77a0199fb0 con 0x7f778c06ecc0 2024-04-22T23:51:07.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.002+0000 7f77977fe700 1 -- 172.21.15.67:0/3759133321 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f77a0199fb0 con 0x7f778c06ecc0 2024-04-22T23:51:07.994 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:51:07.994 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:51:07.994 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:51:07.994 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:51:07.994 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:51:07.994 INFO:teuthology.orchestra.run.smithi067.stdout: "mds", 2024-04-22T23:51:07.994 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:51:07.994 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:51:07.994 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:51:07.994 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:51:07.994 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:51:07.994 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T23:51:07.994 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:51:07.995 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:51:07.995 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:51:07.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.005+0000 7f77a4ef9700 1 -- 172.21.15.67:0/3759133321 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f778c06ecc0 msgr2=0x7f778c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:07.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.005+0000 7f77a4ef9700 1 --2- 172.21.15.67:0/3759133321 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f778c06ecc0 0x7f778c071180 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7798015040 tx=0x7f7798006000 comp rx=0 tx=0).stop 2024-04-22T23:51:07.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.005+0000 7f77a4ef9700 1 -- 172.21.15.67:0/3759133321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77a0102700 msgr2=0x7f77a019eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:07.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.005+0000 7f77a4ef9700 1 --2- 172.21.15.67:0/3759133321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77a0102700 0x7f77a019eec0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f778800b3d0 tx=0x7f778800b790 comp rx=0 tx=0).stop 2024-04-22T23:51:07.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.005+0000 7f77a4ef9700 1 -- 172.21.15.67:0/3759133321 shutdown_connections 2024-04-22T23:51:07.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.005+0000 7f77a4ef9700 1 --2- 172.21.15.67:0/3759133321 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f778c06ecc0 0x7f778c071180 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:07.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.005+0000 7f77a4ef9700 1 --2- 172.21.15.67:0/3759133321 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77a0101de0 0x7f77a019e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:07.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.005+0000 7f77a4ef9700 1 --2- 172.21.15.67:0/3759133321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77a0102700 0x7f77a019eec0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:07.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.005+0000 7f77a4ef9700 1 -- 172.21.15.67:0/3759133321 >> 172.21.15.67:0/3759133321 conn(0x7f77a00fb610 msgr2=0x7f77a0100200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:51:07.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.006+0000 7f77a4ef9700 1 -- 172.21.15.67:0/3759133321 shutdown_connections 2024-04-22T23:51:07.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.006+0000 7f77a4ef9700 1 -- 172.21.15.67:0/3759133321 wait complete. 2024-04-22T23:51:08.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.115+0000 7f17a8e78700 1 -- 172.21.15.67:0/292033401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17a4101ec0 msgr2=0x7f17a41022a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:08.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.115+0000 7f17a8e78700 1 --2- 172.21.15.67:0/292033401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17a4101ec0 0x7f17a41022a0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f178c009940 tx=0x7f178c009c50 comp rx=0 tx=0).stop 2024-04-22T23:51:08.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.115+0000 7f17a8e78700 1 -- 172.21.15.67:0/292033401 shutdown_connections 2024-04-22T23:51:08.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.115+0000 7f17a8e78700 1 --2- 172.21.15.67:0/292033401 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17a41027e0 0x7f17a410acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:08.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.115+0000 7f17a8e78700 1 --2- 172.21.15.67:0/292033401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17a4101ec0 0x7f17a41022a0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:08.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.115+0000 7f17a8e78700 1 -- 172.21.15.67:0/292033401 >> 172.21.15.67:0/292033401 conn(0x7f17a40fb650 msgr2=0x7f17a40fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:51:08.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.115+0000 7f17a8e78700 1 -- 172.21.15.67:0/292033401 shutdown_connections 2024-04-22T23:51:08.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.115+0000 7f17a8e78700 1 -- 172.21.15.67:0/292033401 wait complete. 2024-04-22T23:51:08.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.116+0000 7f17a8e78700 1 Processor -- start 2024-04-22T23:51:08.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.116+0000 7f17a8e78700 1 -- start start 2024-04-22T23:51:08.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.116+0000 7f17a8e78700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17a4101ec0 0x7f17a419c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:08.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.116+0000 7f17a8e78700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17a41027e0 0x7f17a419cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:08.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.116+0000 7f17a8e78700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a419d260 con 0x7f17a4101ec0 2024-04-22T23:51:08.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.116+0000 7f17a8e78700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a4196710 con 0x7f17a41027e0 2024-04-22T23:51:08.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.116+0000 7f17a259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17a4101ec0 0x7f17a419c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:08.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.116+0000 7f17a259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17a4101ec0 0x7f17a419c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36580/0 (socket says 172.21.15.67:36580) 2024-04-22T23:51:08.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.116+0000 7f17a259c700 1 -- 172.21.15.67:0/3169346195 learned_addr learned my addr 172.21.15.67:0/3169346195 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:51:08.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.117+0000 7f17a1d9b700 1 --2- 172.21.15.67:0/3169346195 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17a41027e0 0x7f17a419cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:08.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.117+0000 7f17a259c700 1 -- 172.21.15.67:0/3169346195 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17a41027e0 msgr2=0x7f17a419cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:08.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.117+0000 7f17a259c700 1 --2- 172.21.15.67:0/3169346195 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17a41027e0 0x7f17a419cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:08.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.117+0000 7f17a259c700 1 -- 172.21.15.67:0/3169346195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f178c0095f0 con 0x7f17a4101ec0 2024-04-22T23:51:08.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.117+0000 7f17a1d9b700 1 --2- 172.21.15.67:0/3169346195 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17a41027e0 0x7f17a419cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:51:08.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.117+0000 7f17a259c700 1 --2- 172.21.15.67:0/3169346195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17a4101ec0 0x7f17a419c690 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f178c0055f0 tx=0x7f178c0056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:51:08.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.117+0000 7f179b7fe700 1 -- 172.21.15.67:0/3169346195 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f178c01d070 con 0x7f17a4101ec0 2024-04-22T23:51:08.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.117+0000 7f17a8e78700 1 -- 172.21.15.67:0/3169346195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17a4196990 con 0x7f17a4101ec0 2024-04-22T23:51:08.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.117+0000 7f179b7fe700 1 -- 172.21.15.67:0/3169346195 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f178c00f460 con 0x7f17a4101ec0 2024-04-22T23:51:08.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.117+0000 7f179b7fe700 1 -- 172.21.15.67:0/3169346195 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f178c021620 con 0x7f17a4101ec0 2024-04-22T23:51:08.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.117+0000 7f17a8e78700 1 -- 172.21.15.67:0/3169346195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17a4196e80 con 0x7f17a4101ec0 2024-04-22T23:51:08.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.118+0000 7f17a8e78700 1 -- 172.21.15.67:0/3169346195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17a41083a0 con 0x7f17a4101ec0 2024-04-22T23:51:08.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.119+0000 7f179b7fe700 1 -- 172.21.15.67:0/3169346195 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f178c00fa60 con 0x7f17a4101ec0 2024-04-22T23:51:08.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.119+0000 7f179b7fe700 1 --2- 172.21.15.67:0/3169346195 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f179006ecc0 0x7f1790071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:08.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.119+0000 7f179b7fe700 1 -- 172.21.15.67:0/3169346195 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f178c090350 con 0x7f17a4101ec0 2024-04-22T23:51:08.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.119+0000 7f17a1d9b700 1 --2- 172.21.15.67:0/3169346195 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f179006ecc0 0x7f1790071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:08.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.120+0000 7f17a1d9b700 1 --2- 172.21.15.67:0/3169346195 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f179006ecc0 0x7f1790071180 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f17a4197b80 tx=0x7f1794005f30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:51:08.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.124+0000 7f179b7fe700 1 -- 172.21.15.67:0/3169346195 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f178c05cac0 con 0x7f17a4101ec0 2024-04-22T23:51:08.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:08 smithi067 ceph-mon[112438]: from='client.35356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:51:08.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:08 smithi067 ceph-mon[112438]: from='client.35360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:51:08.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:08 smithi067 ceph-mon[112438]: from='client.35364 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:51:08.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:08 smithi067 ceph-mon[112438]: pgmap v744: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:08.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:08 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/919267788' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:51:08.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:08 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/4198933739' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:51:08.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.424+0000 7f17a8e78700 1 -- 172.21.15.67:0/3169346195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f17a4197c20 con 0x7f17a4101ec0 2024-04-22T23:51:08.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.425+0000 7f179b7fe700 1 -- 172.21.15.67:0/3169346195 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4505 (secure 0 0 0) 0x7f178c05c650 con 0x7f17a4101ec0 2024-04-22T23:51:08.416 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:51:08.417 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T23:51:08.417 INFO:teuthology.orchestra.run.smithi067.stdout: host smithi134 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-22T23:51:08.417 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T23:51:08.417 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:51:08.417 INFO:teuthology.orchestra.run.smithi067.stdout:Traceback (most recent call last): 2024-04-22T23:51:08.417 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T23:51:08.417 INFO:teuthology.orchestra.run.smithi067.stdout: return _run_code(code, main_globals, None, 2024-04-22T23:51:08.417 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T23:51:08.417 INFO:teuthology.orchestra.run.smithi067.stdout: exec(code, run_globals) 2024-04-22T23:51:08.417 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5642, in 2024-04-22T23:51:08.417 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5630, in main 2024-04-22T23:51:08.417 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 408, in _infer_config 2024-04-22T23:51:08.417 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 323, in _infer_fsid 2024-04-22T23:51:08.417 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 436, in _infer_image 2024-04-22T23:51:08.418 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 310, in _validate_fsid 2024-04-22T23:51:08.418 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 3380, in command_ceph_volume 2024-04-22T23:51:08.418 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T23:51:08.418 INFO:teuthology.orchestra.run.smithi067.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:51:08.418 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:51:08.418 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:51:08.418 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:51:08.418 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:51:08.418 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:51:08.418 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:51:08.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.427+0000 7f17a8e78700 1 -- 172.21.15.67:0/3169346195 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f179006ecc0 msgr2=0x7f1790071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:08.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.427+0000 7f17a8e78700 1 --2- 172.21.15.67:0/3169346195 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f179006ecc0 0x7f1790071180 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f17a4197b80 tx=0x7f1794005f30 comp rx=0 tx=0).stop 2024-04-22T23:51:08.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.427+0000 7f17a8e78700 1 -- 172.21.15.67:0/3169346195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17a4101ec0 msgr2=0x7f17a419c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:08.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.427+0000 7f17a8e78700 1 --2- 172.21.15.67:0/3169346195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17a4101ec0 0x7f17a419c690 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f178c0055f0 tx=0x7f178c0056d0 comp rx=0 tx=0).stop 2024-04-22T23:51:08.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.427+0000 7f17a8e78700 1 -- 172.21.15.67:0/3169346195 shutdown_connections 2024-04-22T23:51:08.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.427+0000 7f17a8e78700 1 --2- 172.21.15.67:0/3169346195 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f179006ecc0 0x7f1790071180 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:08.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.427+0000 7f17a8e78700 1 --2- 172.21.15.67:0/3169346195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17a4101ec0 0x7f17a419c690 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:08.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.427+0000 7f17a8e78700 1 --2- 172.21.15.67:0/3169346195 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17a41027e0 0x7f17a419cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:08.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.427+0000 7f17a8e78700 1 -- 172.21.15.67:0/3169346195 >> 172.21.15.67:0/3169346195 conn(0x7f17a40fb650 msgr2=0x7f17a41054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:51:08.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.427+0000 7f17a8e78700 1 -- 172.21.15.67:0/3169346195 shutdown_connections 2024-04-22T23:51:08.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:08.427+0000 7f17a8e78700 1 -- 172.21.15.67:0/3169346195 wait complete. 2024-04-22T23:51:08.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:08 smithi134 ceph-mon[85630]: from='client.35356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:51:08.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:08 smithi134 ceph-mon[85630]: from='client.35360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:51:08.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:08 smithi134 ceph-mon[85630]: from='client.35364 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:51:08.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:08 smithi134 ceph-mon[85630]: pgmap v744: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:08.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:08 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/919267788' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:51:08.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:08 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/4198933739' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:51:09.277 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:09 smithi067 ceph-mon[112438]: from='client.35376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:51:09.277 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:09 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3169346195' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:51:09.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:09 smithi134 ceph-mon[85630]: from='client.35376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:51:09.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:09 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3169346195' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:51:10.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:10 smithi067 ceph-mon[112438]: pgmap v745: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:10.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:10 smithi134 ceph-mon[85630]: pgmap v745: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:12.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:12 smithi067 ceph-mon[112438]: pgmap v746: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:12.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:12 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:51:12.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:12 smithi134 ceph-mon[85630]: pgmap v746: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:12.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:12 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:51:14.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:14 smithi067 ceph-mon[112438]: pgmap v747: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:14.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:14 smithi134 ceph-mon[85630]: pgmap v747: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:16.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:16 smithi134 ceph-mon[85630]: pgmap v748: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:16.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:16 smithi067 ceph-mon[112438]: pgmap v748: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:18.337 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:18 smithi067 ceph-mon[112438]: pgmap v749: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:18.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:18 smithi134 ceph-mon[85630]: pgmap v749: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:20.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:20 smithi134 ceph-mon[85630]: pgmap v750: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:20.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:20 smithi067 ceph-mon[112438]: pgmap v750: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:22.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:22 smithi134 ceph-mon[85630]: pgmap v751: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:22.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:22 smithi067 ceph-mon[112438]: pgmap v751: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:24.439 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:24 smithi067 ceph-mon[112438]: pgmap v752: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:24.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:24 smithi134 ceph-mon[85630]: pgmap v752: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:26.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:26 smithi134 ceph-mon[85630]: pgmap v753: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:26.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:51:26.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:26 smithi067 ceph-mon[112438]: pgmap v753: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:26.536 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:51:28.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:28 smithi134 ceph-mon[85630]: pgmap v754: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:28.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:28 smithi067 ceph-mon[112438]: pgmap v754: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:30.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:30 smithi134 ceph-mon[85630]: pgmap v755: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:30.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:30 smithi067 ceph-mon[112438]: pgmap v755: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:31.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:31 smithi134 ceph-mon[85630]: pgmap v756: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:31.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:31 smithi067 ceph-mon[112438]: pgmap v756: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:34.234 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:34 smithi134 ceph-mon[85630]: pgmap v757: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:34.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:34 smithi067 ceph-mon[112438]: pgmap v757: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:36.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:36 smithi067 ceph-mon[112438]: pgmap v758: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:36.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:36 smithi134 ceph-mon[85630]: pgmap v758: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:38.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:38 smithi067 ceph-mon[112438]: pgmap v759: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:38.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:38 smithi134 ceph-mon[85630]: pgmap v759: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.547+0000 7f24d93d6700 1 -- 172.21.15.67:0/1649936648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24d4068580 msgr2=0x7f24d40689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.547+0000 7f24d93d6700 1 --2- 172.21.15.67:0/1649936648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24d4068580 0x7f24d40689e0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f24bc009910 tx=0x7f24bc009c20 comp rx=0 tx=0).stop 2024-04-22T23:51:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.547+0000 7f24d93d6700 1 -- 172.21.15.67:0/1649936648 shutdown_connections 2024-04-22T23:51:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.547+0000 7f24d93d6700 1 --2- 172.21.15.67:0/1649936648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24d4068580 0x7f24d40689e0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.547+0000 7f24d93d6700 1 --2- 172.21.15.67:0/1649936648 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f24d4106cd0 0x7f24d41070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.547+0000 7f24d93d6700 1 -- 172.21.15.67:0/1649936648 >> 172.21.15.67:0/1649936648 conn(0x7f24d4075670 msgr2=0x7f24d4075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:51:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.547+0000 7f24d93d6700 1 -- 172.21.15.67:0/1649936648 shutdown_connections 2024-04-22T23:51:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.547+0000 7f24d93d6700 1 -- 172.21.15.67:0/1649936648 wait complete. 2024-04-22T23:51:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.548+0000 7f24d93d6700 1 Processor -- start 2024-04-22T23:51:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.548+0000 7f24d93d6700 1 -- start start 2024-04-22T23:51:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.548+0000 7f24d93d6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24d4068580 0x7f24d419c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.548+0000 7f24d93d6700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f24d4106cd0 0x7f24d419cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.548+0000 7f24d93d6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d419d200 con 0x7f24d4068580 2024-04-22T23:51:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.548+0000 7f24d93d6700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d41966b0 con 0x7f24d4106cd0 2024-04-22T23:51:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.548+0000 7f24d2ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24d4068580 0x7f24d419c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.548+0000 7f24d2ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24d4068580 0x7f24d419c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59808/0 (socket says 172.21.15.67:59808) 2024-04-22T23:51:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.548+0000 7f24d2ffd700 1 -- 172.21.15.67:0/4251590695 learned_addr learned my addr 172.21.15.67:0/4251590695 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:51:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.549+0000 7f24d2ffd700 1 -- 172.21.15.67:0/4251590695 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f24d4106cd0 msgr2=0x7f24d419cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.549+0000 7f24d27fc700 1 --2- 172.21.15.67:0/4251590695 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f24d4106cd0 0x7f24d419cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:38.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.549+0000 7f24d2ffd700 1 --2- 172.21.15.67:0/4251590695 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f24d4106cd0 0x7f24d419cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:38.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.549+0000 7f24d2ffd700 1 -- 172.21.15.67:0/4251590695 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24bc0095f0 con 0x7f24d4068580 2024-04-22T23:51:38.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.549+0000 7f24d2ffd700 1 --2- 172.21.15.67:0/4251590695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24d4068580 0x7f24d419c630 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f24cc0060b0 tx=0x7f24cc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:51:38.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.549+0000 7f24cbfff700 1 -- 172.21.15.67:0/4251590695 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f24cc00dda0 con 0x7f24d4068580 2024-04-22T23:51:38.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.549+0000 7f24d93d6700 1 -- 172.21.15.67:0/4251590695 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24d4196990 con 0x7f24d4068580 2024-04-22T23:51:38.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.549+0000 7f24cbfff700 1 -- 172.21.15.67:0/4251590695 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f24cc00f040 con 0x7f24d4068580 2024-04-22T23:51:38.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.549+0000 7f24cbfff700 1 -- 172.21.15.67:0/4251590695 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f24cc014790 con 0x7f24d4068580 2024-04-22T23:51:38.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.549+0000 7f24d93d6700 1 -- 172.21.15.67:0/4251590695 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24d4196ee0 con 0x7f24d4068580 2024-04-22T23:51:38.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.550+0000 7f24d93d6700 1 -- 172.21.15.67:0/4251590695 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24d41091d0 con 0x7f24d4068580 2024-04-22T23:51:38.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.551+0000 7f24cbfff700 1 -- 172.21.15.67:0/4251590695 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f24cc014980 con 0x7f24d4068580 2024-04-22T23:51:38.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.551+0000 7f24cbfff700 1 --2- 172.21.15.67:0/4251590695 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f24c006ecc0 0x7f24c0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:38.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.551+0000 7f24cbfff700 1 -- 172.21.15.67:0/4251590695 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f24cc08ea70 con 0x7f24d4068580 2024-04-22T23:51:38.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.551+0000 7f24d27fc700 1 --2- 172.21.15.67:0/4251590695 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f24c006ecc0 0x7f24c0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:38.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.552+0000 7f24d27fc700 1 --2- 172.21.15.67:0/4251590695 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f24c006ecc0 0x7f24c0071180 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f24bc00b5c0 tx=0x7f24bc005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:51:38.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.556+0000 7f24cbfff700 1 -- 172.21.15.67:0/4251590695 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f24cc05b290 con 0x7f24d4068580 2024-04-22T23:51:38.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.788+0000 7f24d93d6700 1 -- 172.21.15.67:0/4251590695 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f24d4197160 con 0x7f24c006ecc0 2024-04-22T23:51:38.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.789+0000 7f24cbfff700 1 -- 172.21.15.67:0/4251590695 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f24d4197160 con 0x7f24c006ecc0 2024-04-22T23:51:38.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.791+0000 7f24d93d6700 1 -- 172.21.15.67:0/4251590695 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f24c006ecc0 msgr2=0x7f24c0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:38.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.791+0000 7f24d93d6700 1 --2- 172.21.15.67:0/4251590695 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f24c006ecc0 0x7f24c0071180 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f24bc00b5c0 tx=0x7f24bc005960 comp rx=0 tx=0).stop 2024-04-22T23:51:38.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.791+0000 7f24d93d6700 1 -- 172.21.15.67:0/4251590695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24d4068580 msgr2=0x7f24d419c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:38.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.791+0000 7f24d93d6700 1 --2- 172.21.15.67:0/4251590695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24d4068580 0x7f24d419c630 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f24cc0060b0 tx=0x7f24cc00d3b0 comp rx=0 tx=0).stop 2024-04-22T23:51:38.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.791+0000 7f24d93d6700 1 -- 172.21.15.67:0/4251590695 shutdown_connections 2024-04-22T23:51:38.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.791+0000 7f24d93d6700 1 --2- 172.21.15.67:0/4251590695 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f24c006ecc0 0x7f24c0071180 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:38.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.791+0000 7f24d93d6700 1 --2- 172.21.15.67:0/4251590695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24d4068580 0x7f24d419c630 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:38.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.791+0000 7f24d93d6700 1 --2- 172.21.15.67:0/4251590695 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f24d4106cd0 0x7f24d419cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:38.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.791+0000 7f24d93d6700 1 -- 172.21.15.67:0/4251590695 >> 172.21.15.67:0/4251590695 conn(0x7f24d4075670 msgr2=0x7f24d40feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:51:38.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.791+0000 7f24d93d6700 1 -- 172.21.15.67:0/4251590695 shutdown_connections 2024-04-22T23:51:38.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.792+0000 7f24d93d6700 1 -- 172.21.15.67:0/4251590695 wait complete. 2024-04-22T23:51:38.794 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:51:38.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.900+0000 7fdbde7d6700 1 -- 172.21.15.67:0/185024976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbd80ffc20 msgr2=0x7fdbd8100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:38.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.900+0000 7fdbde7d6700 1 --2- 172.21.15.67:0/185024976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbd80ffc20 0x7fdbd8100080 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fdbc0009910 tx=0x7fdbc0009c20 comp rx=0 tx=0).stop 2024-04-22T23:51:38.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.900+0000 7fdbde7d6700 1 -- 172.21.15.67:0/185024976 shutdown_connections 2024-04-22T23:51:38.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.900+0000 7fdbde7d6700 1 --2- 172.21.15.67:0/185024976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbd80ffc20 0x7fdbd8100080 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:38.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.900+0000 7fdbde7d6700 1 --2- 172.21.15.67:0/185024976 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdbd8105bd0 0x7fdbd8105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:38.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.900+0000 7fdbde7d6700 1 -- 172.21.15.67:0/185024976 >> 172.21.15.67:0/185024976 conn(0x7fdbd8075670 msgr2=0x7fdbd8075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:51:38.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.900+0000 7fdbde7d6700 1 -- 172.21.15.67:0/185024976 shutdown_connections 2024-04-22T23:51:38.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.900+0000 7fdbde7d6700 1 -- 172.21.15.67:0/185024976 wait complete. 2024-04-22T23:51:38.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.901+0000 7fdbde7d6700 1 Processor -- start 2024-04-22T23:51:38.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.901+0000 7fdbde7d6700 1 -- start start 2024-04-22T23:51:38.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.901+0000 7fdbde7d6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbd80ffc20 0x7fdbd819c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:38.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.901+0000 7fdbde7d6700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdbd8105bd0 0x7fdbd819cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:38.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.901+0000 7fdbde7d6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd819d200 con 0x7fdbd80ffc20 2024-04-22T23:51:38.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.901+0000 7fdbde7d6700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd81966b0 con 0x7fdbd8105bd0 2024-04-22T23:51:38.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.901+0000 7fdbd7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbd80ffc20 0x7fdbd819c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:38.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.901+0000 7fdbd7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbd80ffc20 0x7fdbd819c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44404/0 (socket says 172.21.15.67:44404) 2024-04-22T23:51:38.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.901+0000 7fdbd7fff700 1 -- 172.21.15.67:0/56121873 learned_addr learned my addr 172.21.15.67:0/56121873 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:51:38.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.901+0000 7fdbcffff700 1 --2- 172.21.15.67:0/56121873 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdbd8105bd0 0x7fdbd819cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:38.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.901+0000 7fdbd7fff700 1 -- 172.21.15.67:0/56121873 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdbd8105bd0 msgr2=0x7fdbd819cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:38.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.901+0000 7fdbd7fff700 1 --2- 172.21.15.67:0/56121873 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdbd8105bd0 0x7fdbd819cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:38.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.901+0000 7fdbd7fff700 1 -- 172.21.15.67:0/56121873 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc00095f0 con 0x7fdbd80ffc20 2024-04-22T23:51:38.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.902+0000 7fdbd7fff700 1 --2- 172.21.15.67:0/56121873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbd80ffc20 0x7fdbd819c630 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fdbc800d8d0 tx=0x7fdbc800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:51:38.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.902+0000 7fdbd5ffb700 1 -- 172.21.15.67:0/56121873 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdbc8009940 con 0x7fdbd80ffc20 2024-04-22T23:51:38.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.902+0000 7fdbde7d6700 1 -- 172.21.15.67:0/56121873 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbd8196990 con 0x7fdbd80ffc20 2024-04-22T23:51:38.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.902+0000 7fdbd5ffb700 1 -- 172.21.15.67:0/56121873 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdbc8010460 con 0x7fdbd80ffc20 2024-04-22T23:51:38.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.902+0000 7fdbd5ffb700 1 -- 172.21.15.67:0/56121873 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdbc800f5f0 con 0x7fdbd80ffc20 2024-04-22T23:51:38.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.902+0000 7fdbde7d6700 1 -- 172.21.15.67:0/56121873 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbd8196ee0 con 0x7fdbd80ffc20 2024-04-22T23:51:38.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.903+0000 7fdbde7d6700 1 -- 172.21.15.67:0/56121873 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbd804f2e0 con 0x7fdbd80ffc20 2024-04-22T23:51:38.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.903+0000 7fdbd5ffb700 1 -- 172.21.15.67:0/56121873 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdbc80105d0 con 0x7fdbd80ffc20 2024-04-22T23:51:38.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.904+0000 7fdbd5ffb700 1 --2- 172.21.15.67:0/56121873 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdbc406ecc0 0x7fdbc4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:38.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.904+0000 7fdbd5ffb700 1 -- 172.21.15.67:0/56121873 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fdbc808efe0 con 0x7fdbd80ffc20 2024-04-22T23:51:38.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.904+0000 7fdbcffff700 1 --2- 172.21.15.67:0/56121873 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdbc406ecc0 0x7fdbc4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:38.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.904+0000 7fdbcffff700 1 --2- 172.21.15.67:0/56121873 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdbc406ecc0 0x7fdbc4071180 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fdbd80ffa60 tx=0x7fdbc001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:51:38.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:38.909+0000 7fdbd5ffb700 1 -- 172.21.15.67:0/56121873 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fdbc805b750 con 0x7fdbd80ffc20 2024-04-22T23:51:39.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.146+0000 7fdbde7d6700 1 -- 172.21.15.67:0/56121873 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdbd81971c0 con 0x7fdbc406ecc0 2024-04-22T23:51:39.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.147+0000 7fdbd5ffb700 1 -- 172.21.15.67:0/56121873 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fdbd81971c0 con 0x7fdbc406ecc0 2024-04-22T23:51:39.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.151+0000 7fdbde7d6700 1 -- 172.21.15.67:0/56121873 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdbc406ecc0 msgr2=0x7fdbc4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:39.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.151+0000 7fdbde7d6700 1 --2- 172.21.15.67:0/56121873 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdbc406ecc0 0x7fdbc4071180 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fdbd80ffa60 tx=0x7fdbc001a040 comp rx=0 tx=0).stop 2024-04-22T23:51:39.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.151+0000 7fdbde7d6700 1 -- 172.21.15.67:0/56121873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbd80ffc20 msgr2=0x7fdbd819c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:39.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.151+0000 7fdbde7d6700 1 --2- 172.21.15.67:0/56121873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbd80ffc20 0x7fdbd819c630 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fdbc800d8d0 tx=0x7fdbc800dc90 comp rx=0 tx=0).stop 2024-04-22T23:51:39.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.151+0000 7fdbde7d6700 1 -- 172.21.15.67:0/56121873 shutdown_connections 2024-04-22T23:51:39.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.151+0000 7fdbde7d6700 1 --2- 172.21.15.67:0/56121873 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdbc406ecc0 0x7fdbc4071180 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:39.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.151+0000 7fdbde7d6700 1 --2- 172.21.15.67:0/56121873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbd80ffc20 0x7fdbd819c630 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:39.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.151+0000 7fdbde7d6700 1 --2- 172.21.15.67:0/56121873 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdbd8105bd0 0x7fdbd819cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:39.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.151+0000 7fdbde7d6700 1 -- 172.21.15.67:0/56121873 >> 172.21.15.67:0/56121873 conn(0x7fdbd8075670 msgr2=0x7fdbd8068680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:51:39.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.151+0000 7fdbde7d6700 1 -- 172.21.15.67:0/56121873 shutdown_connections 2024-04-22T23:51:39.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.151+0000 7fdbde7d6700 1 -- 172.21.15.67:0/56121873 wait complete. 2024-04-22T23:51:39.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.257+0000 7ff57b0ec700 1 -- 172.21.15.67:0/4163659222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff574100ea0 msgr2=0x7ff574101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:39.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.257+0000 7ff57b0ec700 1 --2- 172.21.15.67:0/4163659222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff574100ea0 0x7ff574101280 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7ff564009910 tx=0x7ff564009c20 comp rx=0 tx=0).stop 2024-04-22T23:51:39.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.257+0000 7ff57b0ec700 1 -- 172.21.15.67:0/4163659222 shutdown_connections 2024-04-22T23:51:39.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.257+0000 7ff57b0ec700 1 --2- 172.21.15.67:0/4163659222 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff574101850 0x7ff5741058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:39.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.257+0000 7ff57b0ec700 1 --2- 172.21.15.67:0/4163659222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff574100ea0 0x7ff574101280 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:39.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.257+0000 7ff57b0ec700 1 -- 172.21.15.67:0/4163659222 >> 172.21.15.67:0/4163659222 conn(0x7ff5740fc750 msgr2=0x7ff5740feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:51:39.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.257+0000 7ff57b0ec700 1 -- 172.21.15.67:0/4163659222 shutdown_connections 2024-04-22T23:51:39.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.257+0000 7ff57b0ec700 1 -- 172.21.15.67:0/4163659222 wait complete. 2024-04-22T23:51:39.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.257+0000 7ff57b0ec700 1 Processor -- start 2024-04-22T23:51:39.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.257+0000 7ff57b0ec700 1 -- start start 2024-04-22T23:51:39.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.258+0000 7ff57b0ec700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff574100ea0 0x7ff57419e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:39.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.258+0000 7ff57b0ec700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff574101850 0x7ff57419ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:39.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.258+0000 7ff57b0ec700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff57419f4b0 con 0x7ff574100ea0 2024-04-22T23:51:39.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.258+0000 7ff57b0ec700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff574198960 con 0x7ff574101850 2024-04-22T23:51:39.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.258+0000 7ff578e88700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff574100ea0 0x7ff57419e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:39.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.258+0000 7ff578e88700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff574100ea0 0x7ff57419e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44426/0 (socket says 172.21.15.67:44426) 2024-04-22T23:51:39.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.258+0000 7ff578e88700 1 -- 172.21.15.67:0/4138652168 learned_addr learned my addr 172.21.15.67:0/4138652168 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:51:39.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.258+0000 7ff573fff700 1 --2- 172.21.15.67:0/4138652168 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff574101850 0x7ff57419ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:39.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.258+0000 7ff578e88700 1 -- 172.21.15.67:0/4138652168 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff574101850 msgr2=0x7ff57419ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:39.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.258+0000 7ff578e88700 1 --2- 172.21.15.67:0/4138652168 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff574101850 0x7ff57419ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:39.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.258+0000 7ff578e88700 1 -- 172.21.15.67:0/4138652168 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5640095f0 con 0x7ff574100ea0 2024-04-22T23:51:39.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.258+0000 7ff573fff700 1 --2- 172.21.15.67:0/4138652168 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff574101850 0x7ff57419ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:51:39.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.258+0000 7ff578e88700 1 --2- 172.21.15.67:0/4138652168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff574100ea0 0x7ff57419e8e0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7ff564006010 tx=0x7ff56400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:51:39.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.258+0000 7ff571ffb700 1 -- 172.21.15.67:0/4138652168 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff56401d070 con 0x7ff574100ea0 2024-04-22T23:51:39.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.258+0000 7ff571ffb700 1 -- 172.21.15.67:0/4138652168 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff564004fb0 con 0x7ff574100ea0 2024-04-22T23:51:39.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.258+0000 7ff57b0ec700 1 -- 172.21.15.67:0/4138652168 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff574198be0 con 0x7ff574100ea0 2024-04-22T23:51:39.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.259+0000 7ff571ffb700 1 -- 172.21.15.67:0/4138652168 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff564022ea0 con 0x7ff574100ea0 2024-04-22T23:51:39.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.259+0000 7ff57b0ec700 1 -- 172.21.15.67:0/4138652168 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5741990d0 con 0x7ff574100ea0 2024-04-22T23:51:39.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.260+0000 7ff57b0ec700 1 -- 172.21.15.67:0/4138652168 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff574105020 con 0x7ff574100ea0 2024-04-22T23:51:39.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.260+0000 7ff571ffb700 1 -- 172.21.15.67:0/4138652168 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff564022470 con 0x7ff574100ea0 2024-04-22T23:51:39.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.261+0000 7ff571ffb700 1 --2- 172.21.15.67:0/4138652168 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff55c06ecc0 0x7ff55c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:39.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.261+0000 7ff571ffb700 1 -- 172.21.15.67:0/4138652168 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7ff5640904c0 con 0x7ff574100ea0 2024-04-22T23:51:39.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.261+0000 7ff573fff700 1 --2- 172.21.15.67:0/4138652168 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff55c06ecc0 0x7ff55c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:39.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.261+0000 7ff573fff700 1 --2- 172.21.15.67:0/4138652168 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff55c06ecc0 0x7ff55c071180 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7ff5740fce90 tx=0x7ff568005f50 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:51:39.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.267+0000 7ff571ffb700 1 -- 172.21.15.67:0/4138652168 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff56405cd60 con 0x7ff574100ea0 2024-04-22T23:51:39.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.484+0000 7ff57b0ec700 1 -- 172.21.15.67:0/4138652168 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff574199ec0 con 0x7ff55c06ecc0 2024-04-22T23:51:39.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.496+0000 7ff571ffb700 1 -- 172.21.15.67:0/4138652168 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7ff574199ec0 con 0x7ff55c06ecc0 2024-04-22T23:51:39.487 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:51:39.488 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (25m) 3m ago 36m 24.4M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:51:39.488 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (36m) 3m ago 36m 10.0M - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:51:39.488 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (35m) 3m ago 35m 10.0M - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:51:39.488 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (22m) 3m ago 36m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:51:39.488 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (22m) 3m ago 35m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:51:39.488 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (24m) 3m ago 36m 81.2M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:51:39.488 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (19m) 3m ago 31m 19.4M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:51:39.488 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (18m) 3m ago 31m 20.3M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:51:39.488 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (18m) 3m ago 31m 20.1M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:51:39.488 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (3m) 3m ago 31m 14.0M - 19.0.0-3252-g43be0201 a0434699c1d7 46d3d7e86fee 2024-04-22T23:51:39.488 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (27m) 3m ago 38m 514M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:51:39.488 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (26m) 3m ago 35m 459M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:51:39.488 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (23m) 3m ago 38m 74.2M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:51:39.489 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (23m) 3m ago 35m 54.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:51:39.489 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (26m) 3m ago 36m 15.2M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:51:39.489 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (25m) 3m ago 35m 16.0M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:51:39.489 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (22m) 3m ago 34m 102M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:51:39.489 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (21m) 3m ago 34m 128M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:51:39.489 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (21m) 3m ago 33m 83.8M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:51:39.489 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (20m) 3m ago 33m 101M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:51:39.489 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (19m) 3m ago 33m 86.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:51:39.489 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (19m) 3m ago 32m 81.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:51:39.489 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (25m) 3m ago 36m 78.7M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:51:39.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.498+0000 7ff57b0ec700 1 -- 172.21.15.67:0/4138652168 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff55c06ecc0 msgr2=0x7ff55c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:39.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.498+0000 7ff57b0ec700 1 --2- 172.21.15.67:0/4138652168 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff55c06ecc0 0x7ff55c071180 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7ff5740fce90 tx=0x7ff568005f50 comp rx=0 tx=0).stop 2024-04-22T23:51:39.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.498+0000 7ff57b0ec700 1 -- 172.21.15.67:0/4138652168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff574100ea0 msgr2=0x7ff57419e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:39.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.498+0000 7ff57b0ec700 1 --2- 172.21.15.67:0/4138652168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff574100ea0 0x7ff57419e8e0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7ff564006010 tx=0x7ff56400be20 comp rx=0 tx=0).stop 2024-04-22T23:51:39.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.498+0000 7ff57b0ec700 1 -- 172.21.15.67:0/4138652168 shutdown_connections 2024-04-22T23:51:39.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.498+0000 7ff57b0ec700 1 --2- 172.21.15.67:0/4138652168 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff55c06ecc0 0x7ff55c071180 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:39.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.498+0000 7ff57b0ec700 1 --2- 172.21.15.67:0/4138652168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff574100ea0 0x7ff57419e8e0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:39.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.498+0000 7ff57b0ec700 1 --2- 172.21.15.67:0/4138652168 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff574101850 0x7ff57419ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:39.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.498+0000 7ff57b0ec700 1 -- 172.21.15.67:0/4138652168 >> 172.21.15.67:0/4138652168 conn(0x7ff5740fc750 msgr2=0x7ff5740fddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:51:39.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.498+0000 7ff57b0ec700 1 -- 172.21.15.67:0/4138652168 shutdown_connections 2024-04-22T23:51:39.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.498+0000 7ff57b0ec700 1 -- 172.21.15.67:0/4138652168 wait complete. 2024-04-22T23:51:39.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.604+0000 7f000c8f6700 1 -- 172.21.15.67:0/881437670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0008101850 msgr2=0x7f00081058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:39.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.604+0000 7f000c8f6700 1 --2- 172.21.15.67:0/881437670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0008101850 0x7f00081058a0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7efff8009910 tx=0x7efff8009c20 comp rx=0 tx=0).stop 2024-04-22T23:51:39.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.604+0000 7f000c8f6700 1 -- 172.21.15.67:0/881437670 shutdown_connections 2024-04-22T23:51:39.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.604+0000 7f000c8f6700 1 --2- 172.21.15.67:0/881437670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0008101850 0x7f00081058a0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:39.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.604+0000 7f000c8f6700 1 --2- 172.21.15.67:0/881437670 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0008100ea0 0x7f0008101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:39.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.604+0000 7f000c8f6700 1 -- 172.21.15.67:0/881437670 >> 172.21.15.67:0/881437670 conn(0x7f00080fc750 msgr2=0x7f00080feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:51:39.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.604+0000 7f000c8f6700 1 -- 172.21.15.67:0/881437670 shutdown_connections 2024-04-22T23:51:39.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.604+0000 7f000c8f6700 1 -- 172.21.15.67:0/881437670 wait complete. 2024-04-22T23:51:39.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.605+0000 7f000c8f6700 1 Processor -- start 2024-04-22T23:51:39.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.605+0000 7f000c8f6700 1 -- start start 2024-04-22T23:51:39.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.605+0000 7f000c8f6700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0008100ea0 0x7f000819c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:39.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.605+0000 7f000c8f6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0008101850 0x7f000819cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:39.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.605+0000 7f000c8f6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f000819d210 con 0x7f0008101850 2024-04-22T23:51:39.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.605+0000 7f000c8f6700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00081966c0 con 0x7f0008100ea0 2024-04-22T23:51:39.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.606+0000 7f0005d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0008101850 0x7f000819cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:39.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.606+0000 7f0005d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0008101850 0x7f000819cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44452/0 (socket says 172.21.15.67:44452) 2024-04-22T23:51:39.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.606+0000 7f0005d9b700 1 -- 172.21.15.67:0/1840743021 learned_addr learned my addr 172.21.15.67:0/1840743021 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:51:39.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.606+0000 7f000659c700 1 --2- 172.21.15.67:0/1840743021 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0008100ea0 0x7f000819c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:39.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.606+0000 7f0005d9b700 1 -- 172.21.15.67:0/1840743021 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0008100ea0 msgr2=0x7f000819c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:39.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.606+0000 7f0005d9b700 1 --2- 172.21.15.67:0/1840743021 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0008100ea0 0x7f000819c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:39.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.606+0000 7f0005d9b700 1 -- 172.21.15.67:0/1840743021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff80095f0 con 0x7f0008101850 2024-04-22T23:51:39.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.606+0000 7f0005d9b700 1 --2- 172.21.15.67:0/1840743021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0008101850 0x7f000819cb80 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7efff80092e0 tx=0x7efff800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:51:39.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.606+0000 7effff7fe700 1 -- 172.21.15.67:0/1840743021 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efff801d070 con 0x7f0008101850 2024-04-22T23:51:39.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.606+0000 7effff7fe700 1 -- 172.21.15.67:0/1840743021 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efff8005dc0 con 0x7f0008101850 2024-04-22T23:51:39.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.606+0000 7f000c8f6700 1 -- 172.21.15.67:0/1840743021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0008196940 con 0x7f0008101850 2024-04-22T23:51:39.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.606+0000 7effff7fe700 1 -- 172.21.15.67:0/1840743021 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efff8017400 con 0x7f0008101850 2024-04-22T23:51:39.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.606+0000 7f000c8f6700 1 -- 172.21.15.67:0/1840743021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0008196e30 con 0x7f0008101850 2024-04-22T23:51:39.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.607+0000 7f000c8f6700 1 -- 172.21.15.67:0/1840743021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00081092a0 con 0x7f0008101850 2024-04-22T23:51:39.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.608+0000 7effff7fe700 1 -- 172.21.15.67:0/1840743021 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7efff800f6f0 con 0x7f0008101850 2024-04-22T23:51:39.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.608+0000 7effff7fe700 1 --2- 172.21.15.67:0/1840743021 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efff406ecc0 0x7efff4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:39.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.608+0000 7effff7fe700 1 -- 172.21.15.67:0/1840743021 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7efff80902e0 con 0x7f0008101850 2024-04-22T23:51:39.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.608+0000 7f000659c700 1 --2- 172.21.15.67:0/1840743021 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efff406ecc0 0x7efff4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:39.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.609+0000 7f000659c700 1 --2- 172.21.15.67:0/1840743021 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efff406ecc0 0x7efff4071180 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7efff0005950 tx=0x7efff000b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:51:39.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.614+0000 7effff7fe700 1 -- 172.21.15.67:0/1840743021 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7efff805cad0 con 0x7f0008101850 2024-04-22T23:51:39.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.913+0000 7f000c8f6700 1 -- 172.21.15.67:0/1840743021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f000804f2e0 con 0x7f0008101850 2024-04-22T23:51:39.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.914+0000 7effff7fe700 1 -- 172.21.15.67:0/1840743021 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7efff805c660 con 0x7f0008101850 2024-04-22T23:51:39.905 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:51:39.905 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:51:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:51:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:51:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:51:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:51:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:51:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:51:39.906 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:51:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:51:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:51:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 4 2024-04-22T23:51:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:51:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:51:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 14 2024-04-22T23:51:39.907 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:51:39.908 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:51:39.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.915+0000 7f000c8f6700 1 -- 172.21.15.67:0/1840743021 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efff406ecc0 msgr2=0x7efff4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:39.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.916+0000 7f000c8f6700 1 --2- 172.21.15.67:0/1840743021 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efff406ecc0 0x7efff4071180 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7efff0005950 tx=0x7efff000b410 comp rx=0 tx=0).stop 2024-04-22T23:51:39.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.916+0000 7f000c8f6700 1 -- 172.21.15.67:0/1840743021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0008101850 msgr2=0x7f000819cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:39.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.916+0000 7f000c8f6700 1 --2- 172.21.15.67:0/1840743021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0008101850 0x7f000819cb80 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7efff80092e0 tx=0x7efff800bd70 comp rx=0 tx=0).stop 2024-04-22T23:51:39.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.916+0000 7f000c8f6700 1 -- 172.21.15.67:0/1840743021 shutdown_connections 2024-04-22T23:51:39.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.916+0000 7f000c8f6700 1 --2- 172.21.15.67:0/1840743021 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efff406ecc0 0x7efff4071180 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:39.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.916+0000 7f000c8f6700 1 --2- 172.21.15.67:0/1840743021 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0008100ea0 0x7f000819c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:39.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.916+0000 7f000c8f6700 1 --2- 172.21.15.67:0/1840743021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0008101850 0x7f000819cb80 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:39.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.916+0000 7f000c8f6700 1 -- 172.21.15.67:0/1840743021 >> 172.21.15.67:0/1840743021 conn(0x7f00080fc750 msgr2=0x7f00080feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:51:39.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.916+0000 7f000c8f6700 1 -- 172.21.15.67:0/1840743021 shutdown_connections 2024-04-22T23:51:39.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:39.916+0000 7f000c8f6700 1 -- 172.21.15.67:0/1840743021 wait complete. 2024-04-22T23:51:40.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.023+0000 7fdb86b15700 1 -- 172.21.15.67:0/620582473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb80101850 msgr2=0x7fdb801058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:40.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.023+0000 7fdb86b15700 1 --2- 172.21.15.67:0/620582473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb80101850 0x7fdb801058a0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c009910 tx=0x7fdb6c009c20 comp rx=0 tx=0).stop 2024-04-22T23:51:40.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.023+0000 7fdb86b15700 1 -- 172.21.15.67:0/620582473 shutdown_connections 2024-04-22T23:51:40.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.023+0000 7fdb86b15700 1 --2- 172.21.15.67:0/620582473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb80101850 0x7fdb801058a0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:40.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.023+0000 7fdb86b15700 1 --2- 172.21.15.67:0/620582473 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdb80100ea0 0x7fdb80101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:40.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.023+0000 7fdb86b15700 1 -- 172.21.15.67:0/620582473 >> 172.21.15.67:0/620582473 conn(0x7fdb800fc750 msgr2=0x7fdb800feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:51:40.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.023+0000 7fdb86b15700 1 -- 172.21.15.67:0/620582473 shutdown_connections 2024-04-22T23:51:40.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.023+0000 7fdb86b15700 1 -- 172.21.15.67:0/620582473 wait complete. 2024-04-22T23:51:40.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.023+0000 7fdb86b15700 1 Processor -- start 2024-04-22T23:51:40.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.023+0000 7fdb86b15700 1 -- start start 2024-04-22T23:51:40.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.024+0000 7fdb86b15700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdb80100ea0 0x7fdb80073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:40.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.024+0000 7fdb86b15700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb80101850 0x7fdb800735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:40.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.024+0000 7fdb86b15700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb80073b10 con 0x7fdb80101850 2024-04-22T23:51:40.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.024+0000 7fdb86b15700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb80073c50 con 0x7fdb80100ea0 2024-04-22T23:51:40.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.024+0000 7fdb7ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb80101850 0x7fdb800735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:40.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.024+0000 7fdb7ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb80101850 0x7fdb800735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44464/0 (socket says 172.21.15.67:44464) 2024-04-22T23:51:40.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.024+0000 7fdb7ffff700 1 -- 172.21.15.67:0/2036984595 learned_addr learned my addr 172.21.15.67:0/2036984595 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:51:40.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.024+0000 7fdb848b1700 1 --2- 172.21.15.67:0/2036984595 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdb80100ea0 0x7fdb80073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:40.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.024+0000 7fdb7ffff700 1 -- 172.21.15.67:0/2036984595 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdb80100ea0 msgr2=0x7fdb80073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:40.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.024+0000 7fdb7ffff700 1 --2- 172.21.15.67:0/2036984595 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdb80100ea0 0x7fdb80073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:40.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.024+0000 7fdb7ffff700 1 -- 172.21.15.67:0/2036984595 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb6c0095f0 con 0x7fdb80101850 2024-04-22T23:51:40.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.024+0000 7fdb7ffff700 1 --2- 172.21.15.67:0/2036984595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb80101850 0x7fdb800735d0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c00bda0 tx=0x7fdb6c00bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:51:40.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.024+0000 7fdb7dffb700 1 -- 172.21.15.67:0/2036984595 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb6c01d070 con 0x7fdb80101850 2024-04-22T23:51:40.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.024+0000 7fdb7dffb700 1 -- 172.21.15.67:0/2036984595 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdb6c00f460 con 0x7fdb80101850 2024-04-22T23:51:40.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.024+0000 7fdb86b15700 1 -- 172.21.15.67:0/2036984595 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb80073ed0 con 0x7fdb80101850 2024-04-22T23:51:40.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.025+0000 7fdb7dffb700 1 -- 172.21.15.67:0/2036984595 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb6c017630 con 0x7fdb80101850 2024-04-22T23:51:40.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.025+0000 7fdb86b15700 1 -- 172.21.15.67:0/2036984595 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb801a2de0 con 0x7fdb80101850 2024-04-22T23:51:40.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.026+0000 7fdb86b15700 1 -- 172.21.15.67:0/2036984595 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb801092a0 con 0x7fdb80101850 2024-04-22T23:51:40.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.026+0000 7fdb7dffb700 1 -- 172.21.15.67:0/2036984595 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdb6c00fab0 con 0x7fdb80101850 2024-04-22T23:51:40.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.027+0000 7fdb7dffb700 1 --2- 172.21.15.67:0/2036984595 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdb7006ecc0 0x7fdb70071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:40.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.027+0000 7fdb7dffb700 1 -- 172.21.15.67:0/2036984595 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fdb6c090480 con 0x7fdb80101850 2024-04-22T23:51:40.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.027+0000 7fdb848b1700 1 --2- 172.21.15.67:0/2036984595 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdb7006ecc0 0x7fdb70071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:40.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.027+0000 7fdb848b1700 1 --2- 172.21.15.67:0/2036984595 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdb7006ecc0 0x7fdb70071180 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fdb7400ac50 tx=0x7fdb7400a380 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:51:40.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.031+0000 7fdb7dffb700 1 -- 172.21.15.67:0/2036984595 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fdb6c0590c0 con 0x7fdb80101850 2024-04-22T23:51:40.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.293+0000 7fdb86b15700 1 -- 172.21.15.67:0/2036984595 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fdb800745c0 con 0x7fdb80101850 2024-04-22T23:51:40.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:40 smithi067 ceph-mon[112438]: from='client.35384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:51:40.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:40 smithi067 ceph-mon[112438]: pgmap v760: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:40.286 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:40 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1840743021' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:51:40.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.293+0000 7fdb7dffb700 1 -- 172.21.15.67:0/2036984595 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1872 (secure 0 0 0) 0x7fdb6c05c780 con 0x7fdb80101850 2024-04-22T23:51:40.287 INFO:teuthology.orchestra.run.smithi067.stdout:e33 2024-04-22T23:51:40.287 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:51:40.287 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:51:40.287 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:51:40.287 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:51:40.287 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:51:40.287 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:51:40.287 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:51:40.287 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:51:40.288 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:51:40.288 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:51:40.288 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:51:40.288 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:51:40.288 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:51:40.288 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:51:40.288 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:51:40.288 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:51:40.288 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:51:40.288 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:51:40.288 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:51:40.288 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:51:40.288 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:51:40.288 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:51:40.288 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:51:40.289 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:51:40.289 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:51:40.289 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:51:40.289 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:51:40.289 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:51:40.289 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:51:40.289 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:51:40.289 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:51:40.289 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:51:40.289 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:51:40.289 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:51:40.289 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:51:40.289 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:51:40.289 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:51:40.289 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:51:40.290 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:51:40.290 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:51:40.290 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44851} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1849648622,v1:172.21.15.134:6827/1849648622] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:51:40.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.296+0000 7fdb86b15700 1 -- 172.21.15.67:0/2036984595 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdb7006ecc0 msgr2=0x7fdb70071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:40.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.296+0000 7fdb86b15700 1 --2- 172.21.15.67:0/2036984595 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdb7006ecc0 0x7fdb70071180 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fdb7400ac50 tx=0x7fdb7400a380 comp rx=0 tx=0).stop 2024-04-22T23:51:40.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.296+0000 7fdb86b15700 1 -- 172.21.15.67:0/2036984595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb80101850 msgr2=0x7fdb800735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:40.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.296+0000 7fdb86b15700 1 --2- 172.21.15.67:0/2036984595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb80101850 0x7fdb800735d0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c00bda0 tx=0x7fdb6c00bdd0 comp rx=0 tx=0).stop 2024-04-22T23:51:40.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.296+0000 7fdb86b15700 1 -- 172.21.15.67:0/2036984595 shutdown_connections 2024-04-22T23:51:40.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.296+0000 7fdb86b15700 1 --2- 172.21.15.67:0/2036984595 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdb7006ecc0 0x7fdb70071180 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:40.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.296+0000 7fdb86b15700 1 --2- 172.21.15.67:0/2036984595 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdb80100ea0 0x7fdb80073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:40.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.296+0000 7fdb86b15700 1 --2- 172.21.15.67:0/2036984595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb80101850 0x7fdb800735d0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:40.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.296+0000 7fdb86b15700 1 -- 172.21.15.67:0/2036984595 >> 172.21.15.67:0/2036984595 conn(0x7fdb800fc750 msgr2=0x7fdb800fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:51:40.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.296+0000 7fdb86b15700 1 -- 172.21.15.67:0/2036984595 shutdown_connections 2024-04-22T23:51:40.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.296+0000 7fdb86b15700 1 -- 172.21.15.67:0/2036984595 wait complete. 2024-04-22T23:51:40.291 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 33 2024-04-22T23:51:40.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.404+0000 7efeeb59e700 1 -- 172.21.15.67:0/875210797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeec1025d0 msgr2=0x7efeec106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:40.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.404+0000 7efeeb59e700 1 --2- 172.21.15.67:0/875210797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeec1025d0 0x7efeec106a80 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7efee0009910 tx=0x7efee0009c20 comp rx=0 tx=0).stop 2024-04-22T23:51:40.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.404+0000 7efeeb59e700 1 -- 172.21.15.67:0/875210797 shutdown_connections 2024-04-22T23:51:40.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.404+0000 7efeeb59e700 1 --2- 172.21.15.67:0/875210797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeec1025d0 0x7efeec106a80 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:40.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.404+0000 7efeeb59e700 1 --2- 172.21.15.67:0/875210797 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efeec101c20 0x7efeec102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:40.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.404+0000 7efeeb59e700 1 -- 172.21.15.67:0/875210797 >> 172.21.15.67:0/875210797 conn(0x7efeec0fb4b0 msgr2=0x7efeec0fd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:51:40.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.404+0000 7efeeb59e700 1 -- 172.21.15.67:0/875210797 shutdown_connections 2024-04-22T23:51:40.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.404+0000 7efeeb59e700 1 -- 172.21.15.67:0/875210797 wait complete. 2024-04-22T23:51:40.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.405+0000 7efeeb59e700 1 Processor -- start 2024-04-22T23:51:40.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.405+0000 7efeeb59e700 1 -- start start 2024-04-22T23:51:40.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.405+0000 7efeeb59e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efeec101c20 0x7efeec19c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:40.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.405+0000 7efeeb59e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeec1025d0 0x7efeec19cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:40.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.405+0000 7efeeb59e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeec19d110 con 0x7efeec1025d0 2024-04-22T23:51:40.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.405+0000 7efeeb59e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeec196650 con 0x7efeec101c20 2024-04-22T23:51:40.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.405+0000 7efee9d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeec1025d0 0x7efeec19cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:40.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.405+0000 7efee9d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeec1025d0 0x7efeec19cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44484/0 (socket says 172.21.15.67:44484) 2024-04-22T23:51:40.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.405+0000 7efee9d9b700 1 -- 172.21.15.67:0/3077102764 learned_addr learned my addr 172.21.15.67:0/3077102764 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:51:40.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.406+0000 7efeea59c700 1 --2- 172.21.15.67:0/3077102764 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efeec101c20 0x7efeec19c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:40.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.406+0000 7efee9d9b700 1 -- 172.21.15.67:0/3077102764 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efeec101c20 msgr2=0x7efeec19c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:40.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.406+0000 7efee9d9b700 1 --2- 172.21.15.67:0/3077102764 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efeec101c20 0x7efeec19c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:40.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.406+0000 7efee9d9b700 1 -- 172.21.15.67:0/3077102764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee00095f0 con 0x7efeec1025d0 2024-04-22T23:51:40.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.406+0000 7efee9d9b700 1 --2- 172.21.15.67:0/3077102764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeec1025d0 0x7efeec19cb10 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7efee000bda0 tx=0x7efee000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:51:40.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.406+0000 7efedb7fe700 1 -- 172.21.15.67:0/3077102764 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efee001d070 con 0x7efeec1025d0 2024-04-22T23:51:40.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.406+0000 7efeeb59e700 1 -- 172.21.15.67:0/3077102764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeec1968d0 con 0x7efeec1025d0 2024-04-22T23:51:40.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.406+0000 7efedb7fe700 1 -- 172.21.15.67:0/3077102764 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efee0004fb0 con 0x7efeec1025d0 2024-04-22T23:51:40.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.406+0000 7efedb7fe700 1 -- 172.21.15.67:0/3077102764 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efee000f460 con 0x7efeec1025d0 2024-04-22T23:51:40.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.406+0000 7efeeb59e700 1 -- 172.21.15.67:0/3077102764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeec196dc0 con 0x7efeec1025d0 2024-04-22T23:51:40.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.407+0000 7efeeb59e700 1 -- 172.21.15.67:0/3077102764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeec1041e0 con 0x7efeec1025d0 2024-04-22T23:51:40.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.407+0000 7efedb7fe700 1 -- 172.21.15.67:0/3077102764 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7efee0022800 con 0x7efeec1025d0 2024-04-22T23:51:40.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.408+0000 7efedb7fe700 1 --2- 172.21.15.67:0/3077102764 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efed406ecc0 0x7efed4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:40.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.408+0000 7efedb7fe700 1 -- 172.21.15.67:0/3077102764 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7efee00904c0 con 0x7efeec1025d0 2024-04-22T23:51:40.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.408+0000 7efeea59c700 1 --2- 172.21.15.67:0/3077102764 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efed406ecc0 0x7efed4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:40.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.408+0000 7efeea59c700 1 --2- 172.21.15.67:0/3077102764 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efed406ecc0 0x7efed4071180 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7efedc005fd0 tx=0x7efedc005e50 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:51:40.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.413+0000 7efedb7fe700 1 -- 172.21.15.67:0/3077102764 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7efee005ce20 con 0x7efeec1025d0 2024-04-22T23:51:40.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:40 smithi134 ceph-mon[85630]: from='client.35384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:51:40.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:40 smithi134 ceph-mon[85630]: pgmap v760: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:40.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:40 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1840743021' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:51:40.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.644+0000 7efeeb59e700 1 -- 172.21.15.67:0/3077102764 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efeec197b60 con 0x7efed406ecc0 2024-04-22T23:51:40.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.645+0000 7efedb7fe700 1 -- 172.21.15.67:0/3077102764 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7efeec197b60 con 0x7efed406ecc0 2024-04-22T23:51:40.636 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:51:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:51:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:51:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:51:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:51:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: "mds", 2024-04-22T23:51:40.637 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:51:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:51:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:51:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:51:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:51:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T23:51:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:51:40.638 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:51:40.639 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:51:40.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.647+0000 7efeeb59e700 1 -- 172.21.15.67:0/3077102764 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efed406ecc0 msgr2=0x7efed4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:40.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.647+0000 7efeeb59e700 1 --2- 172.21.15.67:0/3077102764 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efed406ecc0 0x7efed4071180 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7efedc005fd0 tx=0x7efedc005e50 comp rx=0 tx=0).stop 2024-04-22T23:51:40.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.647+0000 7efeeb59e700 1 -- 172.21.15.67:0/3077102764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeec1025d0 msgr2=0x7efeec19cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:40.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.647+0000 7efeeb59e700 1 --2- 172.21.15.67:0/3077102764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeec1025d0 0x7efeec19cb10 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7efee000bda0 tx=0x7efee000be80 comp rx=0 tx=0).stop 2024-04-22T23:51:40.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.647+0000 7efeeb59e700 1 -- 172.21.15.67:0/3077102764 shutdown_connections 2024-04-22T23:51:40.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.647+0000 7efeeb59e700 1 --2- 172.21.15.67:0/3077102764 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efed406ecc0 0x7efed4071180 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:40.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.647+0000 7efeeb59e700 1 --2- 172.21.15.67:0/3077102764 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efeec101c20 0x7efeec19c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:40.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.647+0000 7efeeb59e700 1 --2- 172.21.15.67:0/3077102764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeec1025d0 0x7efeec19cb10 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:40.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.647+0000 7efeeb59e700 1 -- 172.21.15.67:0/3077102764 >> 172.21.15.67:0/3077102764 conn(0x7efeec0fb4b0 msgr2=0x7efeec10a300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:51:40.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.647+0000 7efeeb59e700 1 -- 172.21.15.67:0/3077102764 shutdown_connections 2024-04-22T23:51:40.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.647+0000 7efeeb59e700 1 -- 172.21.15.67:0/3077102764 wait complete. 2024-04-22T23:51:40.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.753+0000 7fd3254b1700 1 -- 172.21.15.67:0/3544155250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3200ff340 msgr2=0x7fd3200ff7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:40.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.753+0000 7fd3254b1700 1 --2- 172.21.15.67:0/3544155250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3200ff340 0x7fd3200ff7a0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fd314009910 tx=0x7fd314009c20 comp rx=0 tx=0).stop 2024-04-22T23:51:40.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.754+0000 7fd3254b1700 1 -- 172.21.15.67:0/3544155250 shutdown_connections 2024-04-22T23:51:40.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.754+0000 7fd3254b1700 1 --2- 172.21.15.67:0/3544155250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3200ff340 0x7fd3200ff7a0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:40.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.754+0000 7fd3254b1700 1 --2- 172.21.15.67:0/3544155250 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd320105350 0x7fd320105730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:40.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.754+0000 7fd3254b1700 1 -- 172.21.15.67:0/3544155250 >> 172.21.15.67:0/3544155250 conn(0x7fd320074e30 msgr2=0x7fd320075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:51:40.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.754+0000 7fd3254b1700 1 -- 172.21.15.67:0/3544155250 shutdown_connections 2024-04-22T23:51:40.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.754+0000 7fd3254b1700 1 -- 172.21.15.67:0/3544155250 wait complete. 2024-04-22T23:51:40.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.754+0000 7fd3254b1700 1 Processor -- start 2024-04-22T23:51:40.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.754+0000 7fd3254b1700 1 -- start start 2024-04-22T23:51:40.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.755+0000 7fd3254b1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3200ff340 0x7fd320100fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:40.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.755+0000 7fd3254b1700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd320105350 0x7fd320101500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:40.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.755+0000 7fd3254b1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd320101ad0 con 0x7fd3200ff340 2024-04-22T23:51:40.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.755+0000 7fd3254b1700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd320101c40 con 0x7fd320105350 2024-04-22T23:51:40.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.755+0000 7fd31effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3200ff340 0x7fd320100fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:40.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.755+0000 7fd31effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3200ff340 0x7fd320100fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44508/0 (socket says 172.21.15.67:44508) 2024-04-22T23:51:40.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.755+0000 7fd31effd700 1 -- 172.21.15.67:0/111318267 learned_addr learned my addr 172.21.15.67:0/111318267 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:51:40.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.755+0000 7fd31e7fc700 1 --2- 172.21.15.67:0/111318267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd320105350 0x7fd320101500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:40.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.755+0000 7fd31effd700 1 -- 172.21.15.67:0/111318267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd320105350 msgr2=0x7fd320101500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:40.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.755+0000 7fd31effd700 1 --2- 172.21.15.67:0/111318267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd320105350 0x7fd320101500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:40.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.755+0000 7fd31effd700 1 -- 172.21.15.67:0/111318267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3140095f0 con 0x7fd3200ff340 2024-04-22T23:51:40.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.755+0000 7fd31e7fc700 1 --2- 172.21.15.67:0/111318267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd320105350 0x7fd320101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:51:40.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.755+0000 7fd31effd700 1 --2- 172.21.15.67:0/111318267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3200ff340 0x7fd320100fc0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fd31000b4b0 tx=0x7fd31000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:51:40.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.755+0000 7fd307fff700 1 -- 172.21.15.67:0/111318267 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd310011800 con 0x7fd3200ff340 2024-04-22T23:51:40.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.755+0000 7fd307fff700 1 -- 172.21.15.67:0/111318267 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd310011e20 con 0x7fd3200ff340 2024-04-22T23:51:40.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.755+0000 7fd3254b1700 1 -- 172.21.15.67:0/111318267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3201a2b70 con 0x7fd3200ff340 2024-04-22T23:51:40.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.755+0000 7fd307fff700 1 -- 172.21.15.67:0/111318267 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd31000f2e0 con 0x7fd3200ff340 2024-04-22T23:51:40.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.756+0000 7fd3254b1700 1 -- 172.21.15.67:0/111318267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3201a2fe0 con 0x7fd3200ff340 2024-04-22T23:51:40.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.756+0000 7fd3254b1700 1 -- 172.21.15.67:0/111318267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd320109880 con 0x7fd3200ff340 2024-04-22T23:51:40.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.757+0000 7fd307fff700 1 -- 172.21.15.67:0/111318267 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd310010380 con 0x7fd3200ff340 2024-04-22T23:51:40.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.757+0000 7fd307fff700 1 --2- 172.21.15.67:0/111318267 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd30806ecc0 0x7fd308071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:51:40.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.758+0000 7fd307fff700 1 -- 172.21.15.67:0/111318267 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fd31008eb70 con 0x7fd3200ff340 2024-04-22T23:51:40.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.758+0000 7fd31e7fc700 1 --2- 172.21.15.67:0/111318267 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd30806ecc0 0x7fd308071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:51:40.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.758+0000 7fd31e7fc700 1 --2- 172.21.15.67:0/111318267 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd30806ecc0 0x7fd308071180 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd314006010 tx=0x7fd31400b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:51:40.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:40.762+0000 7fd307fff700 1 -- 172.21.15.67:0/111318267 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fd3100577b0 con 0x7fd3200ff340 2024-04-22T23:51:41.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:41.060+0000 7fd3254b1700 1 -- 172.21.15.67:0/111318267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fd320109260 con 0x7fd3200ff340 2024-04-22T23:51:41.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:41.060+0000 7fd307fff700 1 -- 172.21.15.67:0/111318267 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4505 (secure 0 0 0) 0x7fd31005ae70 con 0x7fd3200ff340 2024-04-22T23:51:41.052 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:51:41.052 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T23:51:41.052 INFO:teuthology.orchestra.run.smithi067.stdout: host smithi134 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-22T23:51:41.052 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T23:51:41.052 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:51:41.052 INFO:teuthology.orchestra.run.smithi067.stdout:Traceback (most recent call last): 2024-04-22T23:51:41.052 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T23:51:41.052 INFO:teuthology.orchestra.run.smithi067.stdout: return _run_code(code, main_globals, None, 2024-04-22T23:51:41.053 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T23:51:41.053 INFO:teuthology.orchestra.run.smithi067.stdout: exec(code, run_globals) 2024-04-22T23:51:41.053 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5642, in 2024-04-22T23:51:41.053 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5630, in main 2024-04-22T23:51:41.053 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 408, in _infer_config 2024-04-22T23:51:41.053 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 323, in _infer_fsid 2024-04-22T23:51:41.053 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 436, in _infer_image 2024-04-22T23:51:41.053 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 310, in _validate_fsid 2024-04-22T23:51:41.053 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 3380, in command_ceph_volume 2024-04-22T23:51:41.053 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T23:51:41.053 INFO:teuthology.orchestra.run.smithi067.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:51:41.053 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:51:41.054 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:51:41.054 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:51:41.054 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:51:41.054 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:51:41.054 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:51:41.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:41.062+0000 7fd3254b1700 1 -- 172.21.15.67:0/111318267 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd30806ecc0 msgr2=0x7fd308071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:41.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:41.062+0000 7fd3254b1700 1 --2- 172.21.15.67:0/111318267 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd30806ecc0 0x7fd308071180 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd314006010 tx=0x7fd31400b540 comp rx=0 tx=0).stop 2024-04-22T23:51:41.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:41.062+0000 7fd3254b1700 1 -- 172.21.15.67:0/111318267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3200ff340 msgr2=0x7fd320100fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:51:41.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:41.062+0000 7fd3254b1700 1 --2- 172.21.15.67:0/111318267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3200ff340 0x7fd320100fc0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fd31000b4b0 tx=0x7fd31000b870 comp rx=0 tx=0).stop 2024-04-22T23:51:41.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:41.062+0000 7fd3254b1700 1 -- 172.21.15.67:0/111318267 shutdown_connections 2024-04-22T23:51:41.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:41.063+0000 7fd3254b1700 1 --2- 172.21.15.67:0/111318267 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd30806ecc0 0x7fd308071180 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:41.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:41.063+0000 7fd3254b1700 1 --2- 172.21.15.67:0/111318267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3200ff340 0x7fd320100fc0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:41.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:41.063+0000 7fd3254b1700 1 --2- 172.21.15.67:0/111318267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd320105350 0x7fd320101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:51:41.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:41.063+0000 7fd3254b1700 1 -- 172.21.15.67:0/111318267 >> 172.21.15.67:0/111318267 conn(0x7fd320074e30 msgr2=0x7fd320103100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:51:41.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:41.063+0000 7fd3254b1700 1 -- 172.21.15.67:0/111318267 shutdown_connections 2024-04-22T23:51:41.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:51:41.063+0000 7fd3254b1700 1 -- 172.21.15.67:0/111318267 wait complete. 2024-04-22T23:51:41.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:41 smithi134 ceph-mon[85630]: from='client.35388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:51:41.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:41 smithi134 ceph-mon[85630]: from='client.35392 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:51:41.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:41 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2036984595' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:51:41.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:41 smithi067 ceph-mon[112438]: from='client.35388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:51:41.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:41 smithi067 ceph-mon[112438]: from='client.35392 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:51:41.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:41 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2036984595' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:51:42.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:42 smithi134 ceph-mon[85630]: from='client.35404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:51:42.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:42 smithi134 ceph-mon[85630]: pgmap v761: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:42.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:42 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:51:42.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:42 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/111318267' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:51:42.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:42 smithi067 ceph-mon[112438]: from='client.35404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:51:42.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:42 smithi067 ceph-mon[112438]: pgmap v761: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:42.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:42 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:51:42.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:42 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/111318267' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:51:44.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:44 smithi134 ceph-mon[85630]: pgmap v762: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:44.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:44 smithi067 ceph-mon[112438]: pgmap v762: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:46.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:46 smithi134 ceph-mon[85630]: pgmap v763: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:46.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:46 smithi067 ceph-mon[112438]: pgmap v763: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:48.337 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:48 smithi067 ceph-mon[112438]: pgmap v764: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:48.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:48 smithi134 ceph-mon[85630]: pgmap v764: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:50.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:50 smithi134 ceph-mon[85630]: pgmap v765: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:50.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:50 smithi067 ceph-mon[112438]: pgmap v765: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:51.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:51 smithi134 ceph-mon[85630]: pgmap v766: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:51.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:51 smithi067 ceph-mon[112438]: pgmap v766: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:54.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:54 smithi067 ceph-mon[112438]: pgmap v767: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:54.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:54 smithi134 ceph-mon[85630]: pgmap v767: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:56.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:56 smithi067 ceph-mon[112438]: pgmap v768: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:56.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:56 smithi134 ceph-mon[85630]: pgmap v768: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:57.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:57 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:51:57.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:57 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:51:58.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:51:58 smithi134 ceph-mon[85630]: pgmap v769: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:51:58.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:51:58 smithi067 ceph-mon[112438]: pgmap v769: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:00.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:00 smithi067 ceph-mon[112438]: pgmap v770: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:00.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:00 smithi134 ceph-mon[85630]: pgmap v770: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:02.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:02 smithi134 ceph-mon[85630]: pgmap v771: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:02.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:02 smithi067 ceph-mon[112438]: pgmap v771: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:04.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:04 smithi134 ceph-mon[85630]: pgmap v772: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:04.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:04 smithi067 ceph-mon[112438]: pgmap v772: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:06.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:06 smithi134 ceph-mon[85630]: pgmap v773: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:06.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:06 smithi067 ceph-mon[112438]: pgmap v773: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:08.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:08 smithi134 ceph-mon[85630]: pgmap v774: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:08.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:08 smithi067 ceph-mon[112438]: pgmap v774: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:10.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:10 smithi067 ceph-mon[112438]: pgmap v775: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:10.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:10 smithi134 ceph-mon[85630]: pgmap v775: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:11.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.181+0000 7f747a585700 1 -- 172.21.15.67:0/2284738639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7474102720 msgr2=0x7f747410ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:11.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.181+0000 7f747a585700 1 --2- 172.21.15.67:0/2284738639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7474102720 0x7f747410ac10 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f745c009910 tx=0x7f745c009c20 comp rx=0 tx=0).stop 2024-04-22T23:52:11.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.181+0000 7f747a585700 1 -- 172.21.15.67:0/2284738639 shutdown_connections 2024-04-22T23:52:11.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.181+0000 7f747a585700 1 --2- 172.21.15.67:0/2284738639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7474102720 0x7f747410ac10 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:11.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.181+0000 7f747a585700 1 --2- 172.21.15.67:0/2284738639 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7474101e00 0x7f74741021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:11.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.181+0000 7f747a585700 1 -- 172.21.15.67:0/2284738639 >> 172.21.15.67:0/2284738639 conn(0x7f74740fb650 msgr2=0x7f74740fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:52:11.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.181+0000 7f747a585700 1 -- 172.21.15.67:0/2284738639 shutdown_connections 2024-04-22T23:52:11.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.181+0000 7f747a585700 1 -- 172.21.15.67:0/2284738639 wait complete. 2024-04-22T23:52:11.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.182+0000 7f747a585700 1 Processor -- start 2024-04-22T23:52:11.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.182+0000 7f747a585700 1 -- start start 2024-04-22T23:52:11.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.182+0000 7f747a585700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7474101e00 0x7f747419c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:11.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.182+0000 7f747a585700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7474102720 0x7f747419cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:11.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.182+0000 7f747a585700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747419d260 con 0x7f7474102720 2024-04-22T23:52:11.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.182+0000 7f747a585700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7474196710 con 0x7f7474101e00 2024-04-22T23:52:11.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.183+0000 7f74737fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7474102720 0x7f747419cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:11.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.183+0000 7f74737fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7474102720 0x7f747419cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52894/0 (socket says 172.21.15.67:52894) 2024-04-22T23:52:11.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.183+0000 7f74737fe700 1 -- 172.21.15.67:0/399906002 learned_addr learned my addr 172.21.15.67:0/399906002 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:52:11.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.183+0000 7f7473fff700 1 --2- 172.21.15.67:0/399906002 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7474101e00 0x7f747419c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:11.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.183+0000 7f74737fe700 1 -- 172.21.15.67:0/399906002 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7474101e00 msgr2=0x7f747419c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:11.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.183+0000 7f74737fe700 1 --2- 172.21.15.67:0/399906002 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7474101e00 0x7f747419c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:11.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.183+0000 7f74737fe700 1 -- 172.21.15.67:0/399906002 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f745c0095f0 con 0x7f7474102720 2024-04-22T23:52:11.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.183+0000 7f7473fff700 1 --2- 172.21.15.67:0/399906002 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7474101e00 0x7f747419c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:52:11.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.183+0000 7f74737fe700 1 --2- 172.21.15.67:0/399906002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7474102720 0x7f747419cbd0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f745c00bd70 tx=0x7f745c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:52:11.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.183+0000 7f74717fa700 1 -- 172.21.15.67:0/399906002 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f745c01d070 con 0x7f7474102720 2024-04-22T23:52:11.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.183+0000 7f74717fa700 1 -- 172.21.15.67:0/399906002 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f745c005dc0 con 0x7f7474102720 2024-04-22T23:52:11.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.183+0000 7f747a585700 1 -- 172.21.15.67:0/399906002 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7474196990 con 0x7f7474102720 2024-04-22T23:52:11.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.183+0000 7f74717fa700 1 -- 172.21.15.67:0/399906002 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f745c017400 con 0x7f7474102720 2024-04-22T23:52:11.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.184+0000 7f747a585700 1 -- 172.21.15.67:0/399906002 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7474196e80 con 0x7f7474102720 2024-04-22T23:52:11.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.184+0000 7f747a585700 1 -- 172.21.15.67:0/399906002 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7474108310 con 0x7f7474102720 2024-04-22T23:52:11.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.185+0000 7f74717fa700 1 -- 172.21.15.67:0/399906002 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f745c00f7f0 con 0x7f7474102720 2024-04-22T23:52:11.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.185+0000 7f74717fa700 1 --2- 172.21.15.67:0/399906002 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f746006ecc0 0x7f7460071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:11.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.186+0000 7f74717fa700 1 -- 172.21.15.67:0/399906002 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f745c090340 con 0x7f7474102720 2024-04-22T23:52:11.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.186+0000 7f7473fff700 1 --2- 172.21.15.67:0/399906002 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f746006ecc0 0x7f7460071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:11.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.186+0000 7f7473fff700 1 --2- 172.21.15.67:0/399906002 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f746006ecc0 0x7f7460071180 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f746400bc10 tx=0x7f746400b5f0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:52:11.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.190+0000 7f74717fa700 1 -- 172.21.15.67:0/399906002 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f745c05cb30 con 0x7f7474102720 2024-04-22T23:52:11.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.423+0000 7f747a585700 1 -- 172.21.15.67:0/399906002 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f74741970f0 con 0x7f746006ecc0 2024-04-22T23:52:11.415 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:52:11.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.424+0000 7f74717fa700 1 -- 172.21.15.67:0/399906002 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f74741970f0 con 0x7f746006ecc0 2024-04-22T23:52:11.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.426+0000 7f747a585700 1 -- 172.21.15.67:0/399906002 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f746006ecc0 msgr2=0x7f7460071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:11.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.426+0000 7f747a585700 1 --2- 172.21.15.67:0/399906002 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f746006ecc0 0x7f7460071180 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f746400bc10 tx=0x7f746400b5f0 comp rx=0 tx=0).stop 2024-04-22T23:52:11.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.426+0000 7f747a585700 1 -- 172.21.15.67:0/399906002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7474102720 msgr2=0x7f747419cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:11.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.426+0000 7f747a585700 1 --2- 172.21.15.67:0/399906002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7474102720 0x7f747419cbd0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f745c00bd70 tx=0x7f745c00be50 comp rx=0 tx=0).stop 2024-04-22T23:52:11.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.426+0000 7f747a585700 1 -- 172.21.15.67:0/399906002 shutdown_connections 2024-04-22T23:52:11.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.426+0000 7f747a585700 1 --2- 172.21.15.67:0/399906002 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f746006ecc0 0x7f7460071180 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:11.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.426+0000 7f747a585700 1 --2- 172.21.15.67:0/399906002 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7474101e00 0x7f747419c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:11.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.426+0000 7f747a585700 1 --2- 172.21.15.67:0/399906002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7474102720 0x7f747419cbd0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:11.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.426+0000 7f747a585700 1 -- 172.21.15.67:0/399906002 >> 172.21.15.67:0/399906002 conn(0x7f74740fb650 msgr2=0x7f7474105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:52:11.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.426+0000 7f747a585700 1 -- 172.21.15.67:0/399906002 shutdown_connections 2024-04-22T23:52:11.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.426+0000 7f747a585700 1 -- 172.21.15.67:0/399906002 wait complete. 2024-04-22T23:52:11.428 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:52:11.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:52:11.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.533+0000 7fc1e7f14700 1 -- 172.21.15.67:0/3955492792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1e00ffda0 msgr2=0x7fc1e0100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:11.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.533+0000 7fc1e7f14700 1 --2- 172.21.15.67:0/3955492792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1e00ffda0 0x7fc1e0100180 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc009910 tx=0x7fc1dc009c20 comp rx=0 tx=0).stop 2024-04-22T23:52:11.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.534+0000 7fc1e7f14700 1 -- 172.21.15.67:0/3955492792 shutdown_connections 2024-04-22T23:52:11.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.534+0000 7fc1e7f14700 1 --2- 172.21.15.67:0/3955492792 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1e0100750 0x7fc1e01047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:11.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.534+0000 7fc1e7f14700 1 --2- 172.21.15.67:0/3955492792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1e00ffda0 0x7fc1e0100180 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:11.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.534+0000 7fc1e7f14700 1 -- 172.21.15.67:0/3955492792 >> 172.21.15.67:0/3955492792 conn(0x7fc1e00fb650 msgr2=0x7fc1e00fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:52:11.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.534+0000 7fc1e7f14700 1 -- 172.21.15.67:0/3955492792 shutdown_connections 2024-04-22T23:52:11.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.534+0000 7fc1e7f14700 1 -- 172.21.15.67:0/3955492792 wait complete. 2024-04-22T23:52:11.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.534+0000 7fc1e7f14700 1 Processor -- start 2024-04-22T23:52:11.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.534+0000 7fc1e7f14700 1 -- start start 2024-04-22T23:52:11.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.535+0000 7fc1e7f14700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1e00ffda0 0x7fc1e0197820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:11.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.535+0000 7fc1e7f14700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1e0100750 0x7fc1e0197d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:11.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.535+0000 7fc1e7f14700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e0198380 con 0x7fc1e0100750 2024-04-22T23:52:11.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.535+0000 7fc1e7f14700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e01984c0 con 0x7fc1e00ffda0 2024-04-22T23:52:11.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.535+0000 7fc1e54af700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1e0100750 0x7fc1e0197d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:11.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.535+0000 7fc1e54af700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1e0100750 0x7fc1e0197d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52902/0 (socket says 172.21.15.67:52902) 2024-04-22T23:52:11.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.535+0000 7fc1e54af700 1 -- 172.21.15.67:0/4163641725 learned_addr learned my addr 172.21.15.67:0/4163641725 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:52:11.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.535+0000 7fc1e5cb0700 1 --2- 172.21.15.67:0/4163641725 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1e00ffda0 0x7fc1e0197820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:11.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.535+0000 7fc1e54af700 1 -- 172.21.15.67:0/4163641725 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1e00ffda0 msgr2=0x7fc1e0197820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:11.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.535+0000 7fc1e54af700 1 --2- 172.21.15.67:0/4163641725 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1e00ffda0 0x7fc1e0197820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:11.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.535+0000 7fc1e54af700 1 -- 172.21.15.67:0/4163641725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1dc0095f0 con 0x7fc1e0100750 2024-04-22T23:52:11.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.535+0000 7fc1e5cb0700 1 --2- 172.21.15.67:0/4163641725 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1e00ffda0 0x7fc1e0197820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:52:11.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.535+0000 7fc1e54af700 1 --2- 172.21.15.67:0/4163641725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1e0100750 0x7fc1e0197d60 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fc1d000b480 tx=0x7fc1d000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:52:11.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.535+0000 7fc1d6ffd700 1 -- 172.21.15.67:0/4163641725 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1d00118c0 con 0x7fc1e0100750 2024-04-22T23:52:11.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.535+0000 7fc1d6ffd700 1 -- 172.21.15.67:0/4163641725 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc1d0010380 con 0x7fc1e0100750 2024-04-22T23:52:11.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.535+0000 7fc1e7f14700 1 -- 172.21.15.67:0/4163641725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1e01a0890 con 0x7fc1e0100750 2024-04-22T23:52:11.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.536+0000 7fc1d6ffd700 1 -- 172.21.15.67:0/4163641725 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1d000f4f0 con 0x7fc1e0100750 2024-04-22T23:52:11.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.536+0000 7fc1e7f14700 1 -- 172.21.15.67:0/4163641725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1e01a0d50 con 0x7fc1e0100750 2024-04-22T23:52:11.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.537+0000 7fc1e7f14700 1 -- 172.21.15.67:0/4163641725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1e0108140 con 0x7fc1e0100750 2024-04-22T23:52:11.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.537+0000 7fc1d6ffd700 1 -- 172.21.15.67:0/4163641725 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc1d00104f0 con 0x7fc1e0100750 2024-04-22T23:52:11.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.537+0000 7fc1d6ffd700 1 --2- 172.21.15.67:0/4163641725 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc1cc06ecc0 0x7fc1cc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:11.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.538+0000 7fc1d6ffd700 1 -- 172.21.15.67:0/4163641725 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fc1d008eb40 con 0x7fc1e0100750 2024-04-22T23:52:11.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.538+0000 7fc1e5cb0700 1 --2- 172.21.15.67:0/4163641725 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc1cc06ecc0 0x7fc1cc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:11.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.538+0000 7fc1e5cb0700 1 --2- 172.21.15.67:0/4163641725 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc1cc06ecc0 0x7fc1cc071180 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc0059d0 tx=0x7fc1dc01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:52:11.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.542+0000 7fc1d6ffd700 1 -- 172.21.15.67:0/4163641725 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc1d0057800 con 0x7fc1e0100750 2024-04-22T23:52:11.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.775+0000 7fc1e7f14700 1 -- 172.21.15.67:0/4163641725 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc1e00611d0 con 0x7fc1cc06ecc0 2024-04-22T23:52:11.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.777+0000 7fc1d6ffd700 1 -- 172.21.15.67:0/4163641725 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc1e00611d0 con 0x7fc1cc06ecc0 2024-04-22T23:52:11.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.780+0000 7fc1e7f14700 1 -- 172.21.15.67:0/4163641725 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc1cc06ecc0 msgr2=0x7fc1cc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:11.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.780+0000 7fc1e7f14700 1 --2- 172.21.15.67:0/4163641725 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc1cc06ecc0 0x7fc1cc071180 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc0059d0 tx=0x7fc1dc01a040 comp rx=0 tx=0).stop 2024-04-22T23:52:11.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.780+0000 7fc1e7f14700 1 -- 172.21.15.67:0/4163641725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1e0100750 msgr2=0x7fc1e0197d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:11.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.780+0000 7fc1e7f14700 1 --2- 172.21.15.67:0/4163641725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1e0100750 0x7fc1e0197d60 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fc1d000b480 tx=0x7fc1d000b840 comp rx=0 tx=0).stop 2024-04-22T23:52:11.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.780+0000 7fc1e7f14700 1 -- 172.21.15.67:0/4163641725 shutdown_connections 2024-04-22T23:52:11.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.780+0000 7fc1e7f14700 1 --2- 172.21.15.67:0/4163641725 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc1cc06ecc0 0x7fc1cc071180 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:11.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.780+0000 7fc1e7f14700 1 --2- 172.21.15.67:0/4163641725 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1e00ffda0 0x7fc1e0197820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:11.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.780+0000 7fc1e7f14700 1 --2- 172.21.15.67:0/4163641725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1e0100750 0x7fc1e0197d60 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:11.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.780+0000 7fc1e7f14700 1 -- 172.21.15.67:0/4163641725 >> 172.21.15.67:0/4163641725 conn(0x7fc1e00fb650 msgr2=0x7fc1e00fcdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:52:11.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.780+0000 7fc1e7f14700 1 -- 172.21.15.67:0/4163641725 shutdown_connections 2024-04-22T23:52:11.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.780+0000 7fc1e7f14700 1 -- 172.21.15.67:0/4163641725 wait complete. 2024-04-22T23:52:11.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.887+0000 7f281e33b700 1 -- 172.21.15.67:0/18119590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28180fff50 msgr2=0x7f2818105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:11.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.887+0000 7f281e33b700 1 --2- 172.21.15.67:0/18119590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28180fff50 0x7f2818105840 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f280c009910 tx=0x7f280c009c20 comp rx=0 tx=0).stop 2024-04-22T23:52:11.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.887+0000 7f281e33b700 1 -- 172.21.15.67:0/18119590 shutdown_connections 2024-04-22T23:52:11.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.887+0000 7f281e33b700 1 --2- 172.21.15.67:0/18119590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28180fff50 0x7f2818105840 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:11.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.887+0000 7f281e33b700 1 --2- 172.21.15.67:0/18119590 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f28180ff5a0 0x7f28180ff980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:11.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.887+0000 7f281e33b700 1 -- 172.21.15.67:0/18119590 >> 172.21.15.67:0/18119590 conn(0x7f2818074e30 msgr2=0x7f2818075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:52:11.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.887+0000 7f281e33b700 1 -- 172.21.15.67:0/18119590 shutdown_connections 2024-04-22T23:52:11.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.887+0000 7f281e33b700 1 -- 172.21.15.67:0/18119590 wait complete. 2024-04-22T23:52:11.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.888+0000 7f281e33b700 1 Processor -- start 2024-04-22T23:52:11.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.888+0000 7f281e33b700 1 -- start start 2024-04-22T23:52:11.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.889+0000 7f281e33b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28180ff5a0 0x7f281819c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:11.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.889+0000 7f281e33b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f28180fff50 0x7f281819cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:11.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.889+0000 7f281e33b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f281819d210 con 0x7f28180ff5a0 2024-04-22T23:52:11.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.889+0000 7f281e33b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28181966c0 con 0x7f28180fff50 2024-04-22T23:52:11.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.889+0000 7f2817fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28180ff5a0 0x7f281819c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:11.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.889+0000 7f2817fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28180ff5a0 0x7f281819c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52920/0 (socket says 172.21.15.67:52920) 2024-04-22T23:52:11.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.889+0000 7f2817fff700 1 -- 172.21.15.67:0/1807766109 learned_addr learned my addr 172.21.15.67:0/1807766109 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:52:11.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.889+0000 7f28177fe700 1 --2- 172.21.15.67:0/1807766109 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f28180fff50 0x7f281819cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:11.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.889+0000 7f2817fff700 1 -- 172.21.15.67:0/1807766109 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f28180fff50 msgr2=0x7f281819cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:11.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.889+0000 7f2817fff700 1 --2- 172.21.15.67:0/1807766109 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f28180fff50 0x7f281819cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:11.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.889+0000 7f2817fff700 1 -- 172.21.15.67:0/1807766109 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f280c0095f0 con 0x7f28180ff5a0 2024-04-22T23:52:11.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.889+0000 7f28177fe700 1 --2- 172.21.15.67:0/1807766109 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f28180fff50 0x7f281819cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:52:11.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.889+0000 7f2817fff700 1 --2- 172.21.15.67:0/1807766109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28180ff5a0 0x7f281819c640 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f280800e920 tx=0x7f280800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:52:11.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.889+0000 7f28157fa700 1 -- 172.21.15.67:0/1807766109 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2808009790 con 0x7f28180ff5a0 2024-04-22T23:52:11.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.889+0000 7f28157fa700 1 -- 172.21.15.67:0/1807766109 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f280800cba0 con 0x7f28180ff5a0 2024-04-22T23:52:11.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.889+0000 7f281e33b700 1 -- 172.21.15.67:0/1807766109 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28181969a0 con 0x7f28180ff5a0 2024-04-22T23:52:11.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.889+0000 7f28157fa700 1 -- 172.21.15.67:0/1807766109 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2808018650 con 0x7f28180ff5a0 2024-04-22T23:52:11.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.889+0000 7f281e33b700 1 -- 172.21.15.67:0/1807766109 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2818196ef0 con 0x7f28180ff5a0 2024-04-22T23:52:11.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.890+0000 7f281e33b700 1 -- 172.21.15.67:0/1807766109 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2818109200 con 0x7f28180ff5a0 2024-04-22T23:52:11.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.891+0000 7f28157fa700 1 -- 172.21.15.67:0/1807766109 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f280800cd10 con 0x7f28180ff5a0 2024-04-22T23:52:11.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.891+0000 7f28157fa700 1 --2- 172.21.15.67:0/1807766109 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f280006ecc0 0x7f2800071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:11.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.892+0000 7f28157fa700 1 -- 172.21.15.67:0/1807766109 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f2808014070 con 0x7f28180ff5a0 2024-04-22T23:52:11.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.892+0000 7f28177fe700 1 --2- 172.21.15.67:0/1807766109 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f280006ecc0 0x7f2800071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:11.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.892+0000 7f28177fe700 1 --2- 172.21.15.67:0/1807766109 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f280006ecc0 0x7f2800071180 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f280c006010 tx=0x7f280c009f90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:52:11.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:11.896+0000 7f28157fa700 1 -- 172.21.15.67:0/1807766109 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f2808054150 con 0x7f28180ff5a0 2024-04-22T23:52:12.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.115+0000 7f281e33b700 1 -- 172.21.15.67:0/1807766109 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f28181971d0 con 0x7f280006ecc0 2024-04-22T23:52:12.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.126+0000 7f28157fa700 1 -- 172.21.15.67:0/1807766109 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f28181971d0 con 0x7f280006ecc0 2024-04-22T23:52:12.117 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:52:12.117 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (25m) 3m ago 37m 24.4M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:52:12.117 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (37m) 3m ago 37m 10.0M - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:52:12.117 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (36m) 3m ago 36m 10.0M - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:52:12.117 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (23m) 3m ago 37m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:52:12.118 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (23m) 3m ago 36m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:52:12.118 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (25m) 3m ago 36m 81.2M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:52:12.118 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (19m) 3m ago 31m 19.4M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:52:12.118 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (19m) 3m ago 31m 20.3M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:52:12.118 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (18m) 3m ago 31m 20.1M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:52:12.118 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (3m) 3m ago 31m 14.0M - 19.0.0-3252-g43be0201 a0434699c1d7 46d3d7e86fee 2024-04-22T23:52:12.118 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (27m) 3m ago 38m 514M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:52:12.118 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (26m) 3m ago 36m 459M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:52:12.118 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (24m) 3m ago 38m 74.2M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:52:12.118 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (23m) 3m ago 36m 54.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:52:12.118 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (26m) 3m ago 37m 15.2M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:52:12.118 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (26m) 3m ago 36m 16.0M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:52:12.118 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (22m) 3m ago 35m 102M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:52:12.118 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (22m) 3m ago 34m 128M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:52:12.118 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (21m) 3m ago 34m 83.8M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:52:12.119 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (21m) 3m ago 34m 101M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:52:12.119 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (20m) 3m ago 33m 86.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:52:12.119 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (19m) 3m ago 33m 81.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:52:12.119 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (26m) 3m ago 36m 78.7M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:52:12.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.128+0000 7f281e33b700 1 -- 172.21.15.67:0/1807766109 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f280006ecc0 msgr2=0x7f2800071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:12.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.128+0000 7f281e33b700 1 --2- 172.21.15.67:0/1807766109 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f280006ecc0 0x7f2800071180 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f280c006010 tx=0x7f280c009f90 comp rx=0 tx=0).stop 2024-04-22T23:52:12.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.128+0000 7f281e33b700 1 -- 172.21.15.67:0/1807766109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28180ff5a0 msgr2=0x7f281819c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:12.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.128+0000 7f281e33b700 1 --2- 172.21.15.67:0/1807766109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28180ff5a0 0x7f281819c640 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f280800e920 tx=0x7f280800ece0 comp rx=0 tx=0).stop 2024-04-22T23:52:12.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.128+0000 7f281e33b700 1 -- 172.21.15.67:0/1807766109 shutdown_connections 2024-04-22T23:52:12.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.128+0000 7f281e33b700 1 --2- 172.21.15.67:0/1807766109 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f280006ecc0 0x7f2800071180 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:12.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.128+0000 7f281e33b700 1 --2- 172.21.15.67:0/1807766109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28180ff5a0 0x7f281819c640 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:12.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.128+0000 7f281e33b700 1 --2- 172.21.15.67:0/1807766109 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f28180fff50 0x7f281819cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:12.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.128+0000 7f281e33b700 1 -- 172.21.15.67:0/1807766109 >> 172.21.15.67:0/1807766109 conn(0x7f2818074e30 msgr2=0x7f28180fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:52:12.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.128+0000 7f281e33b700 1 -- 172.21.15.67:0/1807766109 shutdown_connections 2024-04-22T23:52:12.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.128+0000 7f281e33b700 1 -- 172.21.15.67:0/1807766109 wait complete. 2024-04-22T23:52:12.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.233+0000 7ff25e3e8700 1 -- 172.21.15.67:0/965673942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff258101850 msgr2=0x7ff2581058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:12.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.233+0000 7ff25e3e8700 1 --2- 172.21.15.67:0/965673942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff258101850 0x7ff2581058a0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff24c009910 tx=0x7ff24c009c20 comp rx=0 tx=0).stop 2024-04-22T23:52:12.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.234+0000 7ff25e3e8700 1 -- 172.21.15.67:0/965673942 shutdown_connections 2024-04-22T23:52:12.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.234+0000 7ff25e3e8700 1 --2- 172.21.15.67:0/965673942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff258101850 0x7ff2581058a0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:12.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.234+0000 7ff25e3e8700 1 --2- 172.21.15.67:0/965673942 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff258100ea0 0x7ff258101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:12.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.234+0000 7ff25e3e8700 1 -- 172.21.15.67:0/965673942 >> 172.21.15.67:0/965673942 conn(0x7ff2580fc750 msgr2=0x7ff2580feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:52:12.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.234+0000 7ff25e3e8700 1 -- 172.21.15.67:0/965673942 shutdown_connections 2024-04-22T23:52:12.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.234+0000 7ff25e3e8700 1 -- 172.21.15.67:0/965673942 wait complete. 2024-04-22T23:52:12.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.234+0000 7ff25e3e8700 1 Processor -- start 2024-04-22T23:52:12.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.234+0000 7ff25e3e8700 1 -- start start 2024-04-22T23:52:12.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.235+0000 7ff25e3e8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff258100ea0 0x7ff258073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:12.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.235+0000 7ff25e3e8700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff258101850 0x7ff2580735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:12.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.235+0000 7ff25e3e8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff258073b10 con 0x7ff258100ea0 2024-04-22T23:52:12.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.235+0000 7ff25e3e8700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff258073c50 con 0x7ff258101850 2024-04-22T23:52:12.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.235+0000 7ff257fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff258100ea0 0x7ff258073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:12.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.235+0000 7ff257fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff258100ea0 0x7ff258073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52942/0 (socket says 172.21.15.67:52942) 2024-04-22T23:52:12.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.235+0000 7ff257fff700 1 -- 172.21.15.67:0/649036614 learned_addr learned my addr 172.21.15.67:0/649036614 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:52:12.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.235+0000 7ff2577fe700 1 --2- 172.21.15.67:0/649036614 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff258101850 0x7ff2580735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:12.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.235+0000 7ff257fff700 1 -- 172.21.15.67:0/649036614 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff258101850 msgr2=0x7ff2580735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:12.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.235+0000 7ff257fff700 1 --2- 172.21.15.67:0/649036614 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff258101850 0x7ff2580735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:12.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.235+0000 7ff257fff700 1 -- 172.21.15.67:0/649036614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff24c0095f0 con 0x7ff258100ea0 2024-04-22T23:52:12.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.235+0000 7ff257fff700 1 --2- 172.21.15.67:0/649036614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff258100ea0 0x7ff258073090 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7ff24800b4b0 tx=0x7ff24800b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:52:12.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.235+0000 7ff2557fa700 1 -- 172.21.15.67:0/649036614 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff248011800 con 0x7ff258100ea0 2024-04-22T23:52:12.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.236+0000 7ff25e3e8700 1 -- 172.21.15.67:0/649036614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2581a2b90 con 0x7ff258100ea0 2024-04-22T23:52:12.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.236+0000 7ff2557fa700 1 -- 172.21.15.67:0/649036614 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff248011e20 con 0x7ff258100ea0 2024-04-22T23:52:12.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.236+0000 7ff2557fa700 1 -- 172.21.15.67:0/649036614 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff24800f2e0 con 0x7ff258100ea0 2024-04-22T23:52:12.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.236+0000 7ff25e3e8700 1 -- 172.21.15.67:0/649036614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2581a2fa0 con 0x7ff258100ea0 2024-04-22T23:52:12.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.237+0000 7ff25e3e8700 1 -- 172.21.15.67:0/649036614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2581091d0 con 0x7ff258100ea0 2024-04-22T23:52:12.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.237+0000 7ff2557fa700 1 -- 172.21.15.67:0/649036614 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff248010380 con 0x7ff258100ea0 2024-04-22T23:52:12.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.238+0000 7ff2557fa700 1 --2- 172.21.15.67:0/649036614 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff24006ecc0 0x7ff240071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:12.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.238+0000 7ff2557fa700 1 -- 172.21.15.67:0/649036614 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7ff24808ea20 con 0x7ff258100ea0 2024-04-22T23:52:12.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.238+0000 7ff2577fe700 1 --2- 172.21.15.67:0/649036614 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff24006ecc0 0x7ff240071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:12.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.238+0000 7ff2577fe700 1 --2- 172.21.15.67:0/649036614 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff24006ecc0 0x7ff240071180 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff258074880 tx=0x7ff24c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:52:12.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.242+0000 7ff2557fa700 1 -- 172.21.15.67:0/649036614 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff2480576e0 con 0x7ff258100ea0 2024-04-22T23:52:12.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:12 smithi134 ceph-mon[85630]: pgmap v776: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:12.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:12 smithi134 ceph-mon[85630]: from='client.35412 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:52:12.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:12 smithi134 ceph-mon[85630]: from='client.35416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:52:12.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.539+0000 7ff25e3e8700 1 -- 172.21.15.67:0/649036614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff258073e10 con 0x7ff258100ea0 2024-04-22T23:52:12.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:12 smithi067 ceph-mon[112438]: pgmap v776: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:12.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:12 smithi067 ceph-mon[112438]: from='client.35412 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:52:12.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:12 smithi067 ceph-mon[112438]: from='client.35416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:52:12.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.540+0000 7ff2557fa700 1 -- 172.21.15.67:0/649036614 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7ff24805ada0 con 0x7ff258100ea0 2024-04-22T23:52:12.532 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:52:12.532 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:52:12.532 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:52:12.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:52:12.532 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:52:12.532 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:52:12.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:52:12.532 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:52:12.532 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:52:12.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:52:12.532 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:52:12.532 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 4 2024-04-22T23:52:12.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:52:12.533 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:52:12.533 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 14 2024-04-22T23:52:12.533 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:52:12.533 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:52:12.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.543+0000 7ff25e3e8700 1 -- 172.21.15.67:0/649036614 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff24006ecc0 msgr2=0x7ff240071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:12.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.543+0000 7ff25e3e8700 1 --2- 172.21.15.67:0/649036614 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff24006ecc0 0x7ff240071180 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff258074880 tx=0x7ff24c01a040 comp rx=0 tx=0).stop 2024-04-22T23:52:12.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.543+0000 7ff25e3e8700 1 -- 172.21.15.67:0/649036614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff258100ea0 msgr2=0x7ff258073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:12.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.543+0000 7ff25e3e8700 1 --2- 172.21.15.67:0/649036614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff258100ea0 0x7ff258073090 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7ff24800b4b0 tx=0x7ff24800b7c0 comp rx=0 tx=0).stop 2024-04-22T23:52:12.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.543+0000 7ff25e3e8700 1 -- 172.21.15.67:0/649036614 shutdown_connections 2024-04-22T23:52:12.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.543+0000 7ff25e3e8700 1 --2- 172.21.15.67:0/649036614 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff24006ecc0 0x7ff240071180 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:12.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.543+0000 7ff25e3e8700 1 --2- 172.21.15.67:0/649036614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff258100ea0 0x7ff258073090 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:12.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.543+0000 7ff25e3e8700 1 --2- 172.21.15.67:0/649036614 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff258101850 0x7ff2580735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:12.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.543+0000 7ff25e3e8700 1 -- 172.21.15.67:0/649036614 >> 172.21.15.67:0/649036614 conn(0x7ff2580fc750 msgr2=0x7ff2580fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:52:12.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.543+0000 7ff25e3e8700 1 -- 172.21.15.67:0/649036614 shutdown_connections 2024-04-22T23:52:12.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.543+0000 7ff25e3e8700 1 -- 172.21.15.67:0/649036614 wait complete. 2024-04-22T23:52:12.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.650+0000 7f2b021a7700 1 -- 172.21.15.67:0/2399169306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2afc101850 msgr2=0x7f2afc1058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:12.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.650+0000 7f2b021a7700 1 --2- 172.21.15.67:0/2399169306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2afc101850 0x7f2afc1058a0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f2af0009910 tx=0x7f2af0009c20 comp rx=0 tx=0).stop 2024-04-22T23:52:12.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.650+0000 7f2b021a7700 1 -- 172.21.15.67:0/2399169306 shutdown_connections 2024-04-22T23:52:12.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.650+0000 7f2b021a7700 1 --2- 172.21.15.67:0/2399169306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2afc101850 0x7f2afc1058a0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:12.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.650+0000 7f2b021a7700 1 --2- 172.21.15.67:0/2399169306 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2afc100ea0 0x7f2afc101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:12.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.650+0000 7f2b021a7700 1 -- 172.21.15.67:0/2399169306 >> 172.21.15.67:0/2399169306 conn(0x7f2afc0fc750 msgr2=0x7f2afc0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:52:12.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.651+0000 7f2b021a7700 1 -- 172.21.15.67:0/2399169306 shutdown_connections 2024-04-22T23:52:12.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.651+0000 7f2b021a7700 1 -- 172.21.15.67:0/2399169306 wait complete. 2024-04-22T23:52:12.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.651+0000 7f2b021a7700 1 Processor -- start 2024-04-22T23:52:12.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.651+0000 7f2b021a7700 1 -- start start 2024-04-22T23:52:12.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.651+0000 7f2b021a7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2afc100ea0 0x7f2afc19c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:12.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.652+0000 7f2b021a7700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2afc101850 0x7f2afc19ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:12.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.652+0000 7f2b021a7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2afc19d360 con 0x7f2afc100ea0 2024-04-22T23:52:12.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.652+0000 7f2b021a7700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2afc196810 con 0x7f2afc101850 2024-04-22T23:52:12.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.652+0000 7f2afb7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2afc100ea0 0x7f2afc19c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:12.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.652+0000 7f2afb7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2afc100ea0 0x7f2afc19c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52962/0 (socket says 172.21.15.67:52962) 2024-04-22T23:52:12.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.652+0000 7f2afb7fe700 1 -- 172.21.15.67:0/4089906330 learned_addr learned my addr 172.21.15.67:0/4089906330 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:52:12.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.652+0000 7f2afaffd700 1 --2- 172.21.15.67:0/4089906330 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2afc101850 0x7f2afc19ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:12.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.652+0000 7f2afb7fe700 1 -- 172.21.15.67:0/4089906330 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2afc101850 msgr2=0x7f2afc19ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:12.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.652+0000 7f2afb7fe700 1 --2- 172.21.15.67:0/4089906330 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2afc101850 0x7f2afc19ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:12.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.652+0000 7f2afb7fe700 1 -- 172.21.15.67:0/4089906330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af00095f0 con 0x7f2afc100ea0 2024-04-22T23:52:12.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.652+0000 7f2afb7fe700 1 --2- 172.21.15.67:0/4089906330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2afc100ea0 0x7f2afc19c790 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f2aec00b4b0 tx=0x7f2aec00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:52:12.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.652+0000 7f2af8ff9700 1 -- 172.21.15.67:0/4089906330 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2aec011800 con 0x7f2afc100ea0 2024-04-22T23:52:12.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.652+0000 7f2af8ff9700 1 -- 172.21.15.67:0/4089906330 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2aec011e20 con 0x7f2afc100ea0 2024-04-22T23:52:12.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.652+0000 7f2b021a7700 1 -- 172.21.15.67:0/4089906330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2afc196af0 con 0x7f2afc100ea0 2024-04-22T23:52:12.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.652+0000 7f2af8ff9700 1 -- 172.21.15.67:0/4089906330 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2aec00f2e0 con 0x7f2afc100ea0 2024-04-22T23:52:12.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.652+0000 7f2b021a7700 1 -- 172.21.15.67:0/4089906330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2afc197040 con 0x7f2afc100ea0 2024-04-22T23:52:12.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.653+0000 7f2b021a7700 1 -- 172.21.15.67:0/4089906330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2afc1091d0 con 0x7f2afc100ea0 2024-04-22T23:52:12.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.654+0000 7f2af8ff9700 1 -- 172.21.15.67:0/4089906330 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2aec010380 con 0x7f2afc100ea0 2024-04-22T23:52:12.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.654+0000 7f2af8ff9700 1 --2- 172.21.15.67:0/4089906330 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2ae406ecc0 0x7f2ae4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:12.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.654+0000 7f2af8ff9700 1 -- 172.21.15.67:0/4089906330 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f2aec08ebc0 con 0x7f2afc100ea0 2024-04-22T23:52:12.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.654+0000 7f2afaffd700 1 --2- 172.21.15.67:0/4089906330 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2ae406ecc0 0x7f2ae4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:12.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.655+0000 7f2afaffd700 1 --2- 172.21.15.67:0/4089906330 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2ae406ecc0 0x7f2ae4071180 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2afc0fd160 tx=0x7f2af001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:52:12.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.659+0000 7f2af8ff9700 1 -- 172.21.15.67:0/4089906330 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f2aec05b330 con 0x7f2afc100ea0 2024-04-22T23:52:12.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.913+0000 7f2b021a7700 1 -- 172.21.15.67:0/4089906330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2afc04f2e0 con 0x7f2afc100ea0 2024-04-22T23:52:12.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.913+0000 7f2af8ff9700 1 -- 172.21.15.67:0/4089906330 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1872 (secure 0 0 0) 0x7f2aec05aec0 con 0x7f2afc100ea0 2024-04-22T23:52:12.905 INFO:teuthology.orchestra.run.smithi067.stdout:e33 2024-04-22T23:52:12.905 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:52:12.905 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:52:12.905 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:52:12.905 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:52:12.905 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:52:12.905 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:52:12.906 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:52:12.906 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:52:12.906 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:52:12.906 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:52:12.906 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:52:12.906 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:52:12.906 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:52:12.906 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:52:12.906 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:52:12.906 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:52:12.906 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:52:12.906 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:52:12.906 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:52:12.906 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:52:12.907 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:52:12.907 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:52:12.907 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:52:12.907 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:52:12.907 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:52:12.907 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:52:12.907 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:52:12.907 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:52:12.907 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:52:12.907 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:52:12.907 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:52:12.907 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:52:12.907 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:52:12.907 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:52:12.907 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:52:12.908 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:52:12.908 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:52:12.908 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:52:12.908 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:52:12.908 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:52:12.908 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44851} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1849648622,v1:172.21.15.134:6827/1849648622] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:52:12.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.915+0000 7f2b021a7700 1 -- 172.21.15.67:0/4089906330 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2ae406ecc0 msgr2=0x7f2ae4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:12.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.915+0000 7f2b021a7700 1 --2- 172.21.15.67:0/4089906330 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2ae406ecc0 0x7f2ae4071180 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2afc0fd160 tx=0x7f2af001a040 comp rx=0 tx=0).stop 2024-04-22T23:52:12.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.915+0000 7f2b021a7700 1 -- 172.21.15.67:0/4089906330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2afc100ea0 msgr2=0x7f2afc19c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:12.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.915+0000 7f2b021a7700 1 --2- 172.21.15.67:0/4089906330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2afc100ea0 0x7f2afc19c790 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f2aec00b4b0 tx=0x7f2aec00b870 comp rx=0 tx=0).stop 2024-04-22T23:52:12.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.916+0000 7f2b021a7700 1 -- 172.21.15.67:0/4089906330 shutdown_connections 2024-04-22T23:52:12.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.916+0000 7f2b021a7700 1 --2- 172.21.15.67:0/4089906330 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2ae406ecc0 0x7f2ae4071180 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:12.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.916+0000 7f2b021a7700 1 --2- 172.21.15.67:0/4089906330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2afc100ea0 0x7f2afc19c790 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:12.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.916+0000 7f2b021a7700 1 --2- 172.21.15.67:0/4089906330 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2afc101850 0x7f2afc19ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:12.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.916+0000 7f2b021a7700 1 -- 172.21.15.67:0/4089906330 >> 172.21.15.67:0/4089906330 conn(0x7f2afc0fc750 msgr2=0x7f2afc0fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:52:12.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.916+0000 7f2b021a7700 1 -- 172.21.15.67:0/4089906330 shutdown_connections 2024-04-22T23:52:12.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:12.916+0000 7f2b021a7700 1 -- 172.21.15.67:0/4089906330 wait complete. 2024-04-22T23:52:12.909 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 33 2024-04-22T23:52:13.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.023+0000 7f8127dea700 1 -- 172.21.15.67:0/115820816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81200730b0 msgr2=0x7f8120073490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:13.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.024+0000 7f8127dea700 1 --2- 172.21.15.67:0/115820816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81200730b0 0x7f8120073490 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f8110009910 tx=0x7f8110009c20 comp rx=0 tx=0).stop 2024-04-22T23:52:13.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.024+0000 7f8127dea700 1 -- 172.21.15.67:0/115820816 shutdown_connections 2024-04-22T23:52:13.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.024+0000 7f8127dea700 1 --2- 172.21.15.67:0/115820816 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f81200739d0 0x7f812010cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:13.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.024+0000 7f8127dea700 1 --2- 172.21.15.67:0/115820816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81200730b0 0x7f8120073490 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:13.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.024+0000 7f8127dea700 1 -- 172.21.15.67:0/115820816 >> 172.21.15.67:0/115820816 conn(0x7f81200fc750 msgr2=0x7f81200feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:52:13.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.024+0000 7f8127dea700 1 -- 172.21.15.67:0/115820816 shutdown_connections 2024-04-22T23:52:13.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.024+0000 7f8127dea700 1 -- 172.21.15.67:0/115820816 wait complete. 2024-04-22T23:52:13.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.024+0000 7f8127dea700 1 Processor -- start 2024-04-22T23:52:13.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.024+0000 7f8127dea700 1 -- start start 2024-04-22T23:52:13.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.025+0000 7f8127dea700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f81200730b0 0x7f812019e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:13.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.025+0000 7f8127dea700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81200739d0 0x7f812019eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:13.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.025+0000 7f8127dea700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f812019f530 con 0x7f81200739d0 2024-04-22T23:52:13.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.025+0000 7f8127dea700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81201989e0 con 0x7f81200730b0 2024-04-22T23:52:13.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.025+0000 7f8125385700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81200739d0 0x7f812019eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:13.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.025+0000 7f8125385700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81200739d0 0x7f812019eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52982/0 (socket says 172.21.15.67:52982) 2024-04-22T23:52:13.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.025+0000 7f8125385700 1 -- 172.21.15.67:0/1029419951 learned_addr learned my addr 172.21.15.67:0/1029419951 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:52:13.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.025+0000 7f8125b86700 1 --2- 172.21.15.67:0/1029419951 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f81200730b0 0x7f812019e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:13.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.025+0000 7f8125385700 1 -- 172.21.15.67:0/1029419951 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f81200730b0 msgr2=0x7f812019e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:13.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.025+0000 7f8125385700 1 --2- 172.21.15.67:0/1029419951 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f81200730b0 0x7f812019e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:13.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.025+0000 7f8125385700 1 -- 172.21.15.67:0/1029419951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81100095f0 con 0x7f81200739d0 2024-04-22T23:52:13.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.025+0000 7f8125385700 1 --2- 172.21.15.67:0/1029419951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81200739d0 0x7f812019eea0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f811c00b4b0 tx=0x7f811c00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:52:13.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.025+0000 7f8116ffd700 1 -- 172.21.15.67:0/1029419951 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f811c011800 con 0x7f81200739d0 2024-04-22T23:52:13.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.025+0000 7f8116ffd700 1 -- 172.21.15.67:0/1029419951 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f811c011e20 con 0x7f81200739d0 2024-04-22T23:52:13.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.025+0000 7f8127dea700 1 -- 172.21.15.67:0/1029419951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8120198cc0 con 0x7f81200739d0 2024-04-22T23:52:13.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.025+0000 7f8116ffd700 1 -- 172.21.15.67:0/1029419951 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f811c00f2e0 con 0x7f81200739d0 2024-04-22T23:52:13.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.025+0000 7f8127dea700 1 -- 172.21.15.67:0/1029419951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8120199210 con 0x7f81200739d0 2024-04-22T23:52:13.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.026+0000 7f8127dea700 1 -- 172.21.15.67:0/1029419951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f812010a6c0 con 0x7f81200739d0 2024-04-22T23:52:13.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.027+0000 7f8116ffd700 1 -- 172.21.15.67:0/1029419951 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f811c010380 con 0x7f81200739d0 2024-04-22T23:52:13.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.027+0000 7f8116ffd700 1 --2- 172.21.15.67:0/1029419951 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f810c06ecc0 0x7f810c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:13.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.028+0000 7f8116ffd700 1 -- 172.21.15.67:0/1029419951 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f811c08eb00 con 0x7f81200739d0 2024-04-22T23:52:13.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.028+0000 7f8125b86700 1 --2- 172.21.15.67:0/1029419951 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f810c06ecc0 0x7f810c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:13.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.028+0000 7f8125b86700 1 --2- 172.21.15.67:0/1029419951 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f810c06ecc0 0x7f810c071180 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f811000b5c0 tx=0x7f811001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:52:13.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.032+0000 7f8116ffd700 1 -- 172.21.15.67:0/1029419951 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f811c05b3a0 con 0x7f81200739d0 2024-04-22T23:52:13.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.264+0000 7f8127dea700 1 -- 172.21.15.67:0/1029419951 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8120199540 con 0x7f810c06ecc0 2024-04-22T23:52:13.256 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:13 smithi067 ceph-mon[112438]: from='client.35420 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:52:13.256 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:13 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/649036614' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:52:13.256 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:13 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/4089906330' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:52:13.256 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:13 smithi067 ceph-mon[112438]: pgmap v777: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:13.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.265+0000 7f8116ffd700 1 -- 172.21.15.67:0/1029419951 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8120199540 con 0x7f810c06ecc0 2024-04-22T23:52:13.256 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:52:13.257 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:52:13.257 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:52:13.257 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:52:13.257 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:52:13.257 INFO:teuthology.orchestra.run.smithi067.stdout: "mds", 2024-04-22T23:52:13.257 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:52:13.257 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:52:13.257 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:52:13.257 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:52:13.257 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:52:13.257 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T23:52:13.257 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:52:13.257 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:52:13.257 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:52:13.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.268+0000 7f8127dea700 1 -- 172.21.15.67:0/1029419951 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f810c06ecc0 msgr2=0x7f810c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:13.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.268+0000 7f8127dea700 1 --2- 172.21.15.67:0/1029419951 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f810c06ecc0 0x7f810c071180 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f811000b5c0 tx=0x7f811001a040 comp rx=0 tx=0).stop 2024-04-22T23:52:13.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.268+0000 7f8127dea700 1 -- 172.21.15.67:0/1029419951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81200739d0 msgr2=0x7f812019eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:13.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.268+0000 7f8127dea700 1 --2- 172.21.15.67:0/1029419951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81200739d0 0x7f812019eea0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f811c00b4b0 tx=0x7f811c00b870 comp rx=0 tx=0).stop 2024-04-22T23:52:13.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.268+0000 7f8127dea700 1 -- 172.21.15.67:0/1029419951 shutdown_connections 2024-04-22T23:52:13.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.268+0000 7f8127dea700 1 --2- 172.21.15.67:0/1029419951 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f810c06ecc0 0x7f810c071180 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:13.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.268+0000 7f8127dea700 1 --2- 172.21.15.67:0/1029419951 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f81200730b0 0x7f812019e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:13.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.268+0000 7f8127dea700 1 --2- 172.21.15.67:0/1029419951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81200739d0 0x7f812019eea0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:13.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.268+0000 7f8127dea700 1 -- 172.21.15.67:0/1029419951 >> 172.21.15.67:0/1029419951 conn(0x7f81200fc750 msgr2=0x7f8120107800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:52:13.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.268+0000 7f8127dea700 1 -- 172.21.15.67:0/1029419951 shutdown_connections 2024-04-22T23:52:13.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.268+0000 7f8127dea700 1 -- 172.21.15.67:0/1029419951 wait complete. 2024-04-22T23:52:13.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.375+0000 7f61871b5700 1 -- 172.21.15.67:0/2112959991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6180101e00 msgr2=0x7f61801021e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:13.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.375+0000 7f61871b5700 1 --2- 172.21.15.67:0/2112959991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6180101e00 0x7f61801021e0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f6174009910 tx=0x7f6174009c20 comp rx=0 tx=0).stop 2024-04-22T23:52:13.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.375+0000 7f61871b5700 1 -- 172.21.15.67:0/2112959991 shutdown_connections 2024-04-22T23:52:13.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.375+0000 7f61871b5700 1 --2- 172.21.15.67:0/2112959991 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6180102720 0x7f618010ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:13.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.375+0000 7f61871b5700 1 --2- 172.21.15.67:0/2112959991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6180101e00 0x7f61801021e0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:13.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.375+0000 7f61871b5700 1 -- 172.21.15.67:0/2112959991 >> 172.21.15.67:0/2112959991 conn(0x7f61800fb650 msgr2=0x7f61800fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:52:13.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.375+0000 7f61871b5700 1 -- 172.21.15.67:0/2112959991 shutdown_connections 2024-04-22T23:52:13.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.375+0000 7f61871b5700 1 -- 172.21.15.67:0/2112959991 wait complete. 2024-04-22T23:52:13.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.376+0000 7f61871b5700 1 Processor -- start 2024-04-22T23:52:13.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.376+0000 7f61871b5700 1 -- start start 2024-04-22T23:52:13.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.376+0000 7f61871b5700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6180101e00 0x7f61800ffa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:13.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.376+0000 7f61871b5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6180102720 0x7f61800fffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:13.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.376+0000 7f61871b5700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61801004e0 con 0x7f6180102720 2024-04-22T23:52:13.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.376+0000 7f61871b5700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6180100620 con 0x7f6180101e00 2024-04-22T23:52:13.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.376+0000 7f617ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6180102720 0x7f61800fffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:13.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.376+0000 7f617ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6180102720 0x7f61800fffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52994/0 (socket says 172.21.15.67:52994) 2024-04-22T23:52:13.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.376+0000 7f617ffff700 1 -- 172.21.15.67:0/3163481716 learned_addr learned my addr 172.21.15.67:0/3163481716 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:52:13.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.377+0000 7f6184f51700 1 --2- 172.21.15.67:0/3163481716 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6180101e00 0x7f61800ffa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:13.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.377+0000 7f617ffff700 1 -- 172.21.15.67:0/3163481716 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6180101e00 msgr2=0x7f61800ffa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:13.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.377+0000 7f617ffff700 1 --2- 172.21.15.67:0/3163481716 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6180101e00 0x7f61800ffa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:13.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.377+0000 7f617ffff700 1 -- 172.21.15.67:0/3163481716 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61740095f0 con 0x7f6180102720 2024-04-22T23:52:13.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.377+0000 7f617ffff700 1 --2- 172.21.15.67:0/3163481716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6180102720 0x7f61800fffa0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f616c00d8d0 tx=0x7f616c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:52:13.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.377+0000 7f617dffb700 1 -- 172.21.15.67:0/3163481716 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f616c0041f0 con 0x7f6180102720 2024-04-22T23:52:13.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.377+0000 7f617dffb700 1 -- 172.21.15.67:0/3163481716 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f616c004350 con 0x7f6180102720 2024-04-22T23:52:13.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.377+0000 7f61871b5700 1 -- 172.21.15.67:0/3163481716 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61801a2a90 con 0x7f6180102720 2024-04-22T23:52:13.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.377+0000 7f617dffb700 1 -- 172.21.15.67:0/3163481716 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f616c003d60 con 0x7f6180102720 2024-04-22T23:52:13.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.377+0000 7f61871b5700 1 -- 172.21.15.67:0/3163481716 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61801a2ea0 con 0x7f6180102720 2024-04-22T23:52:13.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.378+0000 7f61871b5700 1 -- 172.21.15.67:0/3163481716 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6180108310 con 0x7f6180102720 2024-04-22T23:52:13.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.379+0000 7f617dffb700 1 -- 172.21.15.67:0/3163481716 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f616c009730 con 0x7f6180102720 2024-04-22T23:52:13.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.379+0000 7f617dffb700 1 --2- 172.21.15.67:0/3163481716 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f617006ecc0 0x7f6170071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:13.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.379+0000 7f617dffb700 1 -- 172.21.15.67:0/3163481716 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f616c08eb70 con 0x7f6180102720 2024-04-22T23:52:13.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.379+0000 7f6184f51700 1 --2- 172.21.15.67:0/3163481716 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f617006ecc0 0x7f6170071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:13.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.380+0000 7f6184f51700 1 --2- 172.21.15.67:0/3163481716 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f617006ecc0 0x7f6170071180 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f617400b5c0 tx=0x7f617401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:52:13.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.384+0000 7f617dffb700 1 -- 172.21.15.67:0/3163481716 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f616c057830 con 0x7f6180102720 2024-04-22T23:52:13.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:13 smithi134 ceph-mon[85630]: from='client.35420 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:52:13.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:13 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/649036614' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:52:13.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:13 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/4089906330' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:52:13.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:13 smithi134 ceph-mon[85630]: pgmap v777: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:13.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.688+0000 7f61871b5700 1 -- 172.21.15.67:0/3163481716 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f618004f2e0 con 0x7f6180102720 2024-04-22T23:52:13.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.688+0000 7f617dffb700 1 -- 172.21.15.67:0/3163481716 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4505 (secure 0 0 0) 0x7f616c017090 con 0x7f6180102720 2024-04-22T23:52:13.680 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:52:13.680 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T23:52:13.680 INFO:teuthology.orchestra.run.smithi067.stdout: host smithi134 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-22T23:52:13.680 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T23:52:13.680 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:52:13.680 INFO:teuthology.orchestra.run.smithi067.stdout:Traceback (most recent call last): 2024-04-22T23:52:13.680 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T23:52:13.680 INFO:teuthology.orchestra.run.smithi067.stdout: return _run_code(code, main_globals, None, 2024-04-22T23:52:13.680 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T23:52:13.680 INFO:teuthology.orchestra.run.smithi067.stdout: exec(code, run_globals) 2024-04-22T23:52:13.680 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5642, in 2024-04-22T23:52:13.680 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5630, in main 2024-04-22T23:52:13.681 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 408, in _infer_config 2024-04-22T23:52:13.681 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 323, in _infer_fsid 2024-04-22T23:52:13.681 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 436, in _infer_image 2024-04-22T23:52:13.681 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 310, in _validate_fsid 2024-04-22T23:52:13.681 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 3380, in command_ceph_volume 2024-04-22T23:52:13.681 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T23:52:13.681 INFO:teuthology.orchestra.run.smithi067.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:52:13.681 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:52:13.681 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:52:13.681 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:52:13.681 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:52:13.681 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:52:13.681 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:52:13.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.690+0000 7f61871b5700 1 -- 172.21.15.67:0/3163481716 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f617006ecc0 msgr2=0x7f6170071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:13.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.690+0000 7f61871b5700 1 --2- 172.21.15.67:0/3163481716 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f617006ecc0 0x7f6170071180 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f617400b5c0 tx=0x7f617401a040 comp rx=0 tx=0).stop 2024-04-22T23:52:13.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.690+0000 7f61871b5700 1 -- 172.21.15.67:0/3163481716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6180102720 msgr2=0x7f61800fffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:13.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.690+0000 7f61871b5700 1 --2- 172.21.15.67:0/3163481716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6180102720 0x7f61800fffa0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f616c00d8d0 tx=0x7f616c00dc90 comp rx=0 tx=0).stop 2024-04-22T23:52:13.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.690+0000 7f61871b5700 1 -- 172.21.15.67:0/3163481716 shutdown_connections 2024-04-22T23:52:13.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.690+0000 7f61871b5700 1 --2- 172.21.15.67:0/3163481716 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f617006ecc0 0x7f6170071180 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:13.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.690+0000 7f61871b5700 1 --2- 172.21.15.67:0/3163481716 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6180101e00 0x7f61800ffa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:13.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.690+0000 7f61871b5700 1 --2- 172.21.15.67:0/3163481716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6180102720 0x7f61800fffa0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:13.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.690+0000 7f61871b5700 1 -- 172.21.15.67:0/3163481716 >> 172.21.15.67:0/3163481716 conn(0x7f61800fb650 msgr2=0x7f6180105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:52:13.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.690+0000 7f61871b5700 1 -- 172.21.15.67:0/3163481716 shutdown_connections 2024-04-22T23:52:13.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:13.690+0000 7f61871b5700 1 -- 172.21.15.67:0/3163481716 wait complete. 2024-04-22T23:52:14.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:14 smithi134 ceph-mon[85630]: from='client.35432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:52:14.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:14 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3163481716' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:52:14.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:14 smithi067 ceph-mon[112438]: from='client.35432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:52:14.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:14 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3163481716' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:52:15.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:15 smithi067 ceph-mon[112438]: pgmap v778: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:15.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:15 smithi134 ceph-mon[85630]: pgmap v778: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:18.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:18 smithi134 ceph-mon[85630]: pgmap v779: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:18.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:18 smithi067 ceph-mon[112438]: pgmap v779: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:20.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:20 smithi067 ceph-mon[112438]: pgmap v780: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:20.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:20 smithi134 ceph-mon[85630]: pgmap v780: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:22.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:22 smithi134 ceph-mon[85630]: pgmap v781: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:22.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:22 smithi067 ceph-mon[112438]: pgmap v781: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:24.296 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:24 smithi134 ceph-mon[85630]: pgmap v782: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:24.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:24 smithi067 ceph-mon[112438]: pgmap v782: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:26.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:26 smithi134 ceph-mon[85630]: pgmap v783: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:26.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:52:26.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:26 smithi067 ceph-mon[112438]: pgmap v783: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:26.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:52:28.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:28 smithi134 ceph-mon[85630]: pgmap v784: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:28.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:28 smithi067 ceph-mon[112438]: pgmap v784: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:30.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:30 smithi134 ceph-mon[85630]: pgmap v785: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:30.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:30 smithi067 ceph-mon[112438]: pgmap v785: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:32.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:32 smithi134 ceph-mon[85630]: pgmap v786: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:32.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:32 smithi067 ceph-mon[112438]: pgmap v786: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:34.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:34 smithi134 ceph-mon[85630]: pgmap v787: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:34.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:34 smithi067 ceph-mon[112438]: pgmap v787: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:36.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:36 smithi134 ceph-mon[85630]: pgmap v788: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:36.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:36 smithi067 ceph-mon[112438]: pgmap v788: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:38.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:38 smithi134 ceph-mon[85630]: pgmap v789: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:38.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:38 smithi067 ceph-mon[112438]: pgmap v789: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:39.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:39 smithi134 ceph-mon[85630]: pgmap v790: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:39.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:39 smithi067 ceph-mon[112438]: pgmap v790: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:41.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:41 smithi134 ceph-mon[85630]: pgmap v791: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:41.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:52:41.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:41 smithi067 ceph-mon[112438]: pgmap v791: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:41.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:52:43.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:43.808+0000 7f4674310700 1 -- 172.21.15.67:0/1948835176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c100ea0 msgr2=0x7f466c101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:43.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:43.808+0000 7f4674310700 1 --2- 172.21.15.67:0/1948835176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c100ea0 0x7f466c101280 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f466400b0a0 tx=0x7f466400b3b0 comp rx=0 tx=0).stop 2024-04-22T23:52:43.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:43.808+0000 7f4674310700 1 -- 172.21.15.67:0/1948835176 shutdown_connections 2024-04-22T23:52:43.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:43.808+0000 7f4674310700 1 --2- 172.21.15.67:0/1948835176 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f466c101850 0x7f466c1058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:43.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:43.808+0000 7f4674310700 1 --2- 172.21.15.67:0/1948835176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c100ea0 0x7f466c101280 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:43.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:43.808+0000 7f4674310700 1 -- 172.21.15.67:0/1948835176 >> 172.21.15.67:0/1948835176 conn(0x7f466c0fc750 msgr2=0x7f466c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:52:43.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:43.808+0000 7f4674310700 1 -- 172.21.15.67:0/1948835176 shutdown_connections 2024-04-22T23:52:43.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:43.808+0000 7f4674310700 1 -- 172.21.15.67:0/1948835176 wait complete. 2024-04-22T23:52:43.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:43.809+0000 7f4674310700 1 Processor -- start 2024-04-22T23:52:43.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:43.809+0000 7f4674310700 1 -- start start 2024-04-22T23:52:43.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:43.809+0000 7f4674310700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f466c100ea0 0x7f466c19c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:43.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:43.809+0000 7f4674310700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c101850 0x7f466c19cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:43.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:43.809+0000 7f4674310700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466c19d2e0 con 0x7f466c101850 2024-04-22T23:52:43.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:43.809+0000 7f4674310700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466c196790 con 0x7f466c100ea0 2024-04-22T23:52:43.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:43.809+0000 7f46718ab700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c101850 0x7f466c19cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:43.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:43.809+0000 7f46718ab700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c101850 0x7f466c19cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50986/0 (socket says 172.21.15.67:50986) 2024-04-22T23:52:43.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:43.809+0000 7f46718ab700 1 -- 172.21.15.67:0/1192462541 learned_addr learned my addr 172.21.15.67:0/1192462541 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:52:43.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:43.810+0000 7f46720ac700 1 --2- 172.21.15.67:0/1192462541 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f466c100ea0 0x7f466c19c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:43.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:43.810+0000 7f46718ab700 1 -- 172.21.15.67:0/1192462541 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f466c100ea0 msgr2=0x7f466c19c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:43.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:43.810+0000 7f46718ab700 1 --2- 172.21.15.67:0/1192462541 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f466c100ea0 0x7f466c19c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:43.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:43.810+0000 7f46718ab700 1 -- 172.21.15.67:0/1192462541 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4664009d00 con 0x7f466c101850 2024-04-22T23:52:43.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:43.810+0000 7f46718ab700 1 --2- 172.21.15.67:0/1192462541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c101850 0x7f466c19cc50 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f466000d900 tx=0x7f466000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:52:43.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:43.810+0000 7f466b7fe700 1 -- 172.21.15.67:0/1192462541 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4660009900 con 0x7f466c101850 2024-04-22T23:52:43.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:43.810+0000 7f4674310700 1 -- 172.21.15.67:0/1192462541 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f466c196a70 con 0x7f466c101850 2024-04-22T23:52:43.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:43.810+0000 7f466b7fe700 1 -- 172.21.15.67:0/1192462541 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4660010460 con 0x7f466c101850 2024-04-22T23:52:43.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:43.810+0000 7f466b7fe700 1 -- 172.21.15.67:0/1192462541 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f466000f5f0 con 0x7f466c101850 2024-04-22T23:52:43.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:43.810+0000 7f4674310700 1 -- 172.21.15.67:0/1192462541 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f466c196fc0 con 0x7f466c101850 2024-04-22T23:52:43.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:43.811+0000 7f4674310700 1 -- 172.21.15.67:0/1192462541 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f466c109240 con 0x7f466c101850 2024-04-22T23:52:43.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:43.812+0000 7f466b7fe700 1 -- 172.21.15.67:0/1192462541 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f46600105d0 con 0x7f466c101850 2024-04-22T23:52:43.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:43.812+0000 7f466b7fe700 1 --2- 172.21.15.67:0/1192462541 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f465406ecc0 0x7f4654071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:43.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:43.812+0000 7f466b7fe700 1 -- 172.21.15.67:0/1192462541 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f466008ee30 con 0x7f466c101850 2024-04-22T23:52:43.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:43.812+0000 7f46720ac700 1 --2- 172.21.15.67:0/1192462541 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f465406ecc0 0x7f4654071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:43.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:43.813+0000 7f46720ac700 1 --2- 172.21.15.67:0/1192462541 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f465406ecc0 0x7f4654071180 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f466400b830 tx=0x7f46640061f0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:52:43.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:43.817+0000 7f466b7fe700 1 -- 172.21.15.67:0/1192462541 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f466005b5a0 con 0x7f466c101850 2024-04-22T23:52:44.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.046+0000 7f4674310700 1 -- 172.21.15.67:0/1192462541 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f466c197db0 con 0x7f465406ecc0 2024-04-22T23:52:44.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.048+0000 7f466b7fe700 1 -- 172.21.15.67:0/1192462541 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f466c197db0 con 0x7f465406ecc0 2024-04-22T23:52:44.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.050+0000 7f4674310700 1 -- 172.21.15.67:0/1192462541 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f465406ecc0 msgr2=0x7f4654071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:44.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.050+0000 7f4674310700 1 --2- 172.21.15.67:0/1192462541 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f465406ecc0 0x7f4654071180 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f466400b830 tx=0x7f46640061f0 comp rx=0 tx=0).stop 2024-04-22T23:52:44.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.050+0000 7f4674310700 1 -- 172.21.15.67:0/1192462541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c101850 msgr2=0x7f466c19cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:44.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.050+0000 7f4674310700 1 --2- 172.21.15.67:0/1192462541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c101850 0x7f466c19cc50 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f466000d900 tx=0x7f466000dcc0 comp rx=0 tx=0).stop 2024-04-22T23:52:44.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.050+0000 7f4674310700 1 -- 172.21.15.67:0/1192462541 shutdown_connections 2024-04-22T23:52:44.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.050+0000 7f4674310700 1 --2- 172.21.15.67:0/1192462541 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f465406ecc0 0x7f4654071180 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:44.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.050+0000 7f4674310700 1 --2- 172.21.15.67:0/1192462541 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f466c100ea0 0x7f466c19c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:44.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.050+0000 7f4674310700 1 --2- 172.21.15.67:0/1192462541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f466c101850 0x7f466c19cc50 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:44.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.050+0000 7f4674310700 1 -- 172.21.15.67:0/1192462541 >> 172.21.15.67:0/1192462541 conn(0x7f466c0fc750 msgr2=0x7f466c0fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:52:44.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.050+0000 7f4674310700 1 -- 172.21.15.67:0/1192462541 shutdown_connections 2024-04-22T23:52:44.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.050+0000 7f4674310700 1 -- 172.21.15.67:0/1192462541 wait complete. 2024-04-22T23:52:44.052 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:52:44.052 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:44 smithi067 ceph-mon[112438]: pgmap v792: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:44.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.158+0000 7f0af7947700 1 -- 172.21.15.67:0/3499513555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0af0101850 msgr2=0x7f0af01058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:44.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.158+0000 7f0af7947700 1 --2- 172.21.15.67:0/3499513555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0af0101850 0x7f0af01058a0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f0ae0009910 tx=0x7f0ae0009c20 comp rx=0 tx=0).stop 2024-04-22T23:52:44.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.158+0000 7f0af7947700 1 -- 172.21.15.67:0/3499513555 shutdown_connections 2024-04-22T23:52:44.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.158+0000 7f0af7947700 1 --2- 172.21.15.67:0/3499513555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0af0101850 0x7f0af01058a0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:44.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.158+0000 7f0af7947700 1 --2- 172.21.15.67:0/3499513555 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0af0100ea0 0x7f0af0101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:44.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.158+0000 7f0af7947700 1 -- 172.21.15.67:0/3499513555 >> 172.21.15.67:0/3499513555 conn(0x7f0af00fc750 msgr2=0x7f0af00feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:52:44.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.158+0000 7f0af7947700 1 -- 172.21.15.67:0/3499513555 shutdown_connections 2024-04-22T23:52:44.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.158+0000 7f0af7947700 1 -- 172.21.15.67:0/3499513555 wait complete. 2024-04-22T23:52:44.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.158+0000 7f0af7947700 1 Processor -- start 2024-04-22T23:52:44.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.158+0000 7f0af7947700 1 -- start start 2024-04-22T23:52:44.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.159+0000 7f0af7947700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0af0100ea0 0x7f0af019c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:44.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.159+0000 7f0af7947700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0af0101850 0x7f0af019cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:44.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.159+0000 7f0af7947700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af019d2d0 con 0x7f0af0100ea0 2024-04-22T23:52:44.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.159+0000 7f0af7947700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af0196780 con 0x7f0af0101850 2024-04-22T23:52:44.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.159+0000 7f0af56e3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0af0100ea0 0x7f0af019c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:44.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.159+0000 7f0af56e3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0af0100ea0 0x7f0af019c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51006/0 (socket says 172.21.15.67:51006) 2024-04-22T23:52:44.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.159+0000 7f0af56e3700 1 -- 172.21.15.67:0/2412375120 learned_addr learned my addr 172.21.15.67:0/2412375120 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:52:44.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.159+0000 7f0af4ee2700 1 --2- 172.21.15.67:0/2412375120 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0af0101850 0x7f0af019cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:44.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.159+0000 7f0af56e3700 1 -- 172.21.15.67:0/2412375120 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0af0101850 msgr2=0x7f0af019cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:44.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.159+0000 7f0af56e3700 1 --2- 172.21.15.67:0/2412375120 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0af0101850 0x7f0af019cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:44.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.159+0000 7f0af56e3700 1 -- 172.21.15.67:0/2412375120 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae00095f0 con 0x7f0af0100ea0 2024-04-22T23:52:44.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.159+0000 7f0af56e3700 1 --2- 172.21.15.67:0/2412375120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0af0100ea0 0x7f0af019c700 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f0aec00d8d0 tx=0x7f0aec00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:52:44.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.159+0000 7f0ae67fc700 1 -- 172.21.15.67:0/2412375120 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0aec009940 con 0x7f0af0100ea0 2024-04-22T23:52:44.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.159+0000 7f0ae67fc700 1 -- 172.21.15.67:0/2412375120 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0aec010460 con 0x7f0af0100ea0 2024-04-22T23:52:44.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.159+0000 7f0af7947700 1 -- 172.21.15.67:0/2412375120 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0af0196a60 con 0x7f0af0100ea0 2024-04-22T23:52:44.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.159+0000 7f0ae67fc700 1 -- 172.21.15.67:0/2412375120 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0aec00f5f0 con 0x7f0af0100ea0 2024-04-22T23:52:44.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.159+0000 7f0af7947700 1 -- 172.21.15.67:0/2412375120 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0af0196fb0 con 0x7f0af0100ea0 2024-04-22T23:52:44.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.160+0000 7f0af7947700 1 -- 172.21.15.67:0/2412375120 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0af01092a0 con 0x7f0af0100ea0 2024-04-22T23:52:44.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.161+0000 7f0ae67fc700 1 -- 172.21.15.67:0/2412375120 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0aec0105d0 con 0x7f0af0100ea0 2024-04-22T23:52:44.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.161+0000 7f0ae67fc700 1 --2- 172.21.15.67:0/2412375120 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0adc06ecc0 0x7f0adc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:44.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.161+0000 7f0ae67fc700 1 -- 172.21.15.67:0/2412375120 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f0aec08efe0 con 0x7f0af0100ea0 2024-04-22T23:52:44.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.161+0000 7f0af4ee2700 1 --2- 172.21.15.67:0/2412375120 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0adc06ecc0 0x7f0adc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:44.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.162+0000 7f0af4ee2700 1 --2- 172.21.15.67:0/2412375120 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0adc06ecc0 0x7f0adc071180 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f0af0197bf0 tx=0x7f0ae001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:52:44.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.166+0000 7f0ae67fc700 1 -- 172.21.15.67:0/2412375120 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f0aec05b750 con 0x7f0af0100ea0 2024-04-22T23:52:44.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.398+0000 7f0af7947700 1 -- 172.21.15.67:0/2412375120 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0af0197da0 con 0x7f0adc06ecc0 2024-04-22T23:52:44.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.400+0000 7f0ae67fc700 1 -- 172.21.15.67:0/2412375120 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f0af0197da0 con 0x7f0adc06ecc0 2024-04-22T23:52:44.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.402+0000 7f0af7947700 1 -- 172.21.15.67:0/2412375120 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0adc06ecc0 msgr2=0x7f0adc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:44.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.402+0000 7f0af7947700 1 --2- 172.21.15.67:0/2412375120 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0adc06ecc0 0x7f0adc071180 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f0af0197bf0 tx=0x7f0ae001a040 comp rx=0 tx=0).stop 2024-04-22T23:52:44.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.402+0000 7f0af7947700 1 -- 172.21.15.67:0/2412375120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0af0100ea0 msgr2=0x7f0af019c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:44.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.402+0000 7f0af7947700 1 --2- 172.21.15.67:0/2412375120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0af0100ea0 0x7f0af019c700 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f0aec00d8d0 tx=0x7f0aec00dc90 comp rx=0 tx=0).stop 2024-04-22T23:52:44.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.402+0000 7f0af7947700 1 -- 172.21.15.67:0/2412375120 shutdown_connections 2024-04-22T23:52:44.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.402+0000 7f0af7947700 1 --2- 172.21.15.67:0/2412375120 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0adc06ecc0 0x7f0adc071180 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:44.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.402+0000 7f0af7947700 1 --2- 172.21.15.67:0/2412375120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0af0100ea0 0x7f0af019c700 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:44.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.402+0000 7f0af7947700 1 --2- 172.21.15.67:0/2412375120 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0af0101850 0x7f0af019cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:44.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.402+0000 7f0af7947700 1 -- 172.21.15.67:0/2412375120 >> 172.21.15.67:0/2412375120 conn(0x7f0af00fc750 msgr2=0x7f0af00fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:52:44.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.402+0000 7f0af7947700 1 -- 172.21.15.67:0/2412375120 shutdown_connections 2024-04-22T23:52:44.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.402+0000 7f0af7947700 1 -- 172.21.15.67:0/2412375120 wait complete. 2024-04-22T23:52:44.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:44 smithi134 ceph-mon[85630]: pgmap v792: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:44.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.509+0000 7fb079434700 1 -- 172.21.15.67:0/2668854456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb074106cd0 msgr2=0x7fb0741070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:44.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.509+0000 7fb079434700 1 --2- 172.21.15.67:0/2668854456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb074106cd0 0x7fb0741070b0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fb06c00b0a0 tx=0x7fb06c00b3b0 comp rx=0 tx=0).stop 2024-04-22T23:52:44.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.509+0000 7fb079434700 1 -- 172.21.15.67:0/2668854456 shutdown_connections 2024-04-22T23:52:44.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.509+0000 7fb079434700 1 --2- 172.21.15.67:0/2668854456 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb074068580 0x7fb0740689e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:44.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.509+0000 7fb079434700 1 --2- 172.21.15.67:0/2668854456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb074106cd0 0x7fb0741070b0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:44.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.509+0000 7fb079434700 1 -- 172.21.15.67:0/2668854456 >> 172.21.15.67:0/2668854456 conn(0x7fb074075670 msgr2=0x7fb074075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:52:44.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.509+0000 7fb079434700 1 -- 172.21.15.67:0/2668854456 shutdown_connections 2024-04-22T23:52:44.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.509+0000 7fb079434700 1 -- 172.21.15.67:0/2668854456 wait complete. 2024-04-22T23:52:44.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.510+0000 7fb079434700 1 Processor -- start 2024-04-22T23:52:44.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.510+0000 7fb079434700 1 -- start start 2024-04-22T23:52:44.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.510+0000 7fb079434700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb074068580 0x7fb07419c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:44.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.510+0000 7fb079434700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb074106cd0 0x7fb07419cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:44.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.510+0000 7fb079434700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07419d260 con 0x7fb074068580 2024-04-22T23:52:44.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.510+0000 7fb079434700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb074196710 con 0x7fb074106cd0 2024-04-22T23:52:44.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.510+0000 7fb072ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb074068580 0x7fb07419c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:44.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.510+0000 7fb072ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb074068580 0x7fb07419c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51022/0 (socket says 172.21.15.67:51022) 2024-04-22T23:52:44.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.510+0000 7fb072ffd700 1 -- 172.21.15.67:0/1730596064 learned_addr learned my addr 172.21.15.67:0/1730596064 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:52:44.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.510+0000 7fb0727fc700 1 --2- 172.21.15.67:0/1730596064 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb074106cd0 0x7fb07419cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:44.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.511+0000 7fb072ffd700 1 -- 172.21.15.67:0/1730596064 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb074106cd0 msgr2=0x7fb07419cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:44.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.511+0000 7fb072ffd700 1 --2- 172.21.15.67:0/1730596064 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb074106cd0 0x7fb07419cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:44.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.511+0000 7fb072ffd700 1 -- 172.21.15.67:0/1730596064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb06c009d00 con 0x7fb074068580 2024-04-22T23:52:44.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.511+0000 7fb072ffd700 1 --2- 172.21.15.67:0/1730596064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb074068580 0x7fb07419c690 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fb06c00b070 tx=0x7fb06c00cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:52:44.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.511+0000 7fb06bfff700 1 -- 172.21.15.67:0/1730596064 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb06c00e040 con 0x7fb074068580 2024-04-22T23:52:44.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.511+0000 7fb079434700 1 -- 172.21.15.67:0/1730596064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb074196990 con 0x7fb074068580 2024-04-22T23:52:44.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.511+0000 7fb06bfff700 1 -- 172.21.15.67:0/1730596064 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb06c003c10 con 0x7fb074068580 2024-04-22T23:52:44.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.511+0000 7fb06bfff700 1 -- 172.21.15.67:0/1730596064 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb06c01d740 con 0x7fb074068580 2024-04-22T23:52:44.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.511+0000 7fb079434700 1 -- 172.21.15.67:0/1730596064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb074196e80 con 0x7fb074068580 2024-04-22T23:52:44.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.512+0000 7fb079434700 1 -- 172.21.15.67:0/1730596064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0741091f0 con 0x7fb074068580 2024-04-22T23:52:44.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.512+0000 7fb06bfff700 1 -- 172.21.15.67:0/1730596064 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb06c01d9b0 con 0x7fb074068580 2024-04-22T23:52:44.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.513+0000 7fb06bfff700 1 --2- 172.21.15.67:0/1730596064 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb06006ecc0 0x7fb060071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:44.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.513+0000 7fb06bfff700 1 -- 172.21.15.67:0/1730596064 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fb06c0907b0 con 0x7fb074068580 2024-04-22T23:52:44.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.513+0000 7fb0727fc700 1 --2- 172.21.15.67:0/1730596064 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb06006ecc0 0x7fb060071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:44.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.513+0000 7fb0727fc700 1 --2- 172.21.15.67:0/1730596064 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb06006ecc0 0x7fb060071180 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fb074197b80 tx=0x7fb05c00b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:52:44.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.518+0000 7fb06bfff700 1 -- 172.21.15.67:0/1730596064 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb06c05cf90 con 0x7fb074068580 2024-04-22T23:52:44.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.736+0000 7fb079434700 1 -- 172.21.15.67:0/1730596064 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb0741970a0 con 0x7fb06006ecc0 2024-04-22T23:52:44.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.747+0000 7fb06bfff700 1 -- 172.21.15.67:0/1730596064 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fb0741970a0 con 0x7fb06006ecc0 2024-04-22T23:52:44.739 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:52:44.739 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (26m) 4m ago 37m 24.4M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:52:44.739 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (37m) 4m ago 37m 10.0M - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:52:44.739 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (36m) 4m ago 36m 10.0M - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:52:44.739 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (23m) 4m ago 37m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:52:44.739 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (23m) 4m ago 36m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:52:44.739 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (26m) 4m ago 37m 81.2M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:52:44.739 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (20m) 4m ago 32m 19.4M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:52:44.739 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (19m) 4m ago 32m 20.3M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:52:44.739 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (19m) 4m ago 32m 20.1M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:52:44.740 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (4m) 4m ago 32m 14.0M - 19.0.0-3252-g43be0201 a0434699c1d7 46d3d7e86fee 2024-04-22T23:52:44.740 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (28m) 4m ago 39m 514M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:52:44.740 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (27m) 4m ago 36m 459M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:52:44.740 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (24m) 4m ago 39m 74.2M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:52:44.740 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (24m) 4m ago 36m 54.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:52:44.740 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (27m) 4m ago 37m 15.2M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:52:44.740 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (27m) 4m ago 36m 16.0M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:52:44.740 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (23m) 4m ago 35m 102M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:52:44.740 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (22m) 4m ago 35m 128M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:52:44.740 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (22m) 4m ago 35m 83.8M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:52:44.740 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (21m) 4m ago 34m 101M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:52:44.740 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (21m) 4m ago 34m 86.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:52:44.741 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (20m) 4m ago 33m 81.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:52:44.741 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (26m) 4m ago 37m 78.7M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:52:44.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.750+0000 7fb079434700 1 -- 172.21.15.67:0/1730596064 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb06006ecc0 msgr2=0x7fb060071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:44.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.750+0000 7fb079434700 1 --2- 172.21.15.67:0/1730596064 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb06006ecc0 0x7fb060071180 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fb074197b80 tx=0x7fb05c00b410 comp rx=0 tx=0).stop 2024-04-22T23:52:44.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.750+0000 7fb079434700 1 -- 172.21.15.67:0/1730596064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb074068580 msgr2=0x7fb07419c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:44.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.750+0000 7fb079434700 1 --2- 172.21.15.67:0/1730596064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb074068580 0x7fb07419c690 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fb06c00b070 tx=0x7fb06c00cd20 comp rx=0 tx=0).stop 2024-04-22T23:52:44.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.750+0000 7fb079434700 1 -- 172.21.15.67:0/1730596064 shutdown_connections 2024-04-22T23:52:44.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.750+0000 7fb079434700 1 --2- 172.21.15.67:0/1730596064 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb06006ecc0 0x7fb060071180 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:44.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.750+0000 7fb079434700 1 --2- 172.21.15.67:0/1730596064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb074068580 0x7fb07419c690 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:44.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.750+0000 7fb079434700 1 --2- 172.21.15.67:0/1730596064 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb074106cd0 0x7fb07419cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:44.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.750+0000 7fb079434700 1 -- 172.21.15.67:0/1730596064 >> 172.21.15.67:0/1730596064 conn(0x7fb074075670 msgr2=0x7fb0740fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:52:44.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.750+0000 7fb079434700 1 -- 172.21.15.67:0/1730596064 shutdown_connections 2024-04-22T23:52:44.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.750+0000 7fb079434700 1 -- 172.21.15.67:0/1730596064 wait complete. 2024-04-22T23:52:44.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.855+0000 7fd2416a9700 1 -- 172.21.15.67:0/1478896256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd23c100ea0 msgr2=0x7fd23c101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:44.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.855+0000 7fd2416a9700 1 --2- 172.21.15.67:0/1478896256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd23c100ea0 0x7fd23c101280 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fd230009910 tx=0x7fd230009c20 comp rx=0 tx=0).stop 2024-04-22T23:52:44.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.855+0000 7fd2416a9700 1 -- 172.21.15.67:0/1478896256 shutdown_connections 2024-04-22T23:52:44.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.855+0000 7fd2416a9700 1 --2- 172.21.15.67:0/1478896256 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd23c101850 0x7fd23c1058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:44.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.855+0000 7fd2416a9700 1 --2- 172.21.15.67:0/1478896256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd23c100ea0 0x7fd23c101280 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:44.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.855+0000 7fd2416a9700 1 -- 172.21.15.67:0/1478896256 >> 172.21.15.67:0/1478896256 conn(0x7fd23c0fc750 msgr2=0x7fd23c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:52:44.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.855+0000 7fd2416a9700 1 -- 172.21.15.67:0/1478896256 shutdown_connections 2024-04-22T23:52:44.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.855+0000 7fd2416a9700 1 -- 172.21.15.67:0/1478896256 wait complete. 2024-04-22T23:52:44.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.856+0000 7fd2416a9700 1 Processor -- start 2024-04-22T23:52:44.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.856+0000 7fd2416a9700 1 -- start start 2024-04-22T23:52:44.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.856+0000 7fd2416a9700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd23c100ea0 0x7fd23c196990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:44.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.856+0000 7fd2416a9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd23c101850 0x7fd23c196ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:44.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.856+0000 7fd2416a9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd23c197560 con 0x7fd23c101850 2024-04-22T23:52:44.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.856+0000 7fd2416a9700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd23c19b370 con 0x7fd23c100ea0 2024-04-22T23:52:44.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.856+0000 7fd23a7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd23c101850 0x7fd23c196ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:44.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.856+0000 7fd23a7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd23c101850 0x7fd23c196ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51042/0 (socket says 172.21.15.67:51042) 2024-04-22T23:52:44.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.856+0000 7fd23a7fc700 1 -- 172.21.15.67:0/3165984494 learned_addr learned my addr 172.21.15.67:0/3165984494 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:52:44.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.857+0000 7fd23affd700 1 --2- 172.21.15.67:0/3165984494 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd23c100ea0 0x7fd23c196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:44.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.857+0000 7fd23a7fc700 1 -- 172.21.15.67:0/3165984494 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd23c100ea0 msgr2=0x7fd23c196990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:44.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.857+0000 7fd23a7fc700 1 --2- 172.21.15.67:0/3165984494 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd23c100ea0 0x7fd23c196990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:44.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.857+0000 7fd23a7fc700 1 -- 172.21.15.67:0/3165984494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2300095f0 con 0x7fd23c101850 2024-04-22T23:52:44.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.857+0000 7fd23a7fc700 1 --2- 172.21.15.67:0/3165984494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd23c101850 0x7fd23c196ed0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fd22c00b480 tx=0x7fd22c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:52:44.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.857+0000 7fd223fff700 1 -- 172.21.15.67:0/3165984494 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd22c0118c0 con 0x7fd23c101850 2024-04-22T23:52:44.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.857+0000 7fd223fff700 1 -- 172.21.15.67:0/3165984494 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd22c010380 con 0x7fd23c101850 2024-04-22T23:52:44.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.857+0000 7fd2416a9700 1 -- 172.21.15.67:0/3165984494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd23c19b650 con 0x7fd23c101850 2024-04-22T23:52:44.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.857+0000 7fd223fff700 1 -- 172.21.15.67:0/3165984494 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd22c00f4f0 con 0x7fd23c101850 2024-04-22T23:52:44.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.857+0000 7fd2416a9700 1 -- 172.21.15.67:0/3165984494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd23c19bba0 con 0x7fd23c101850 2024-04-22T23:52:44.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.858+0000 7fd2416a9700 1 -- 172.21.15.67:0/3165984494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd23c109240 con 0x7fd23c101850 2024-04-22T23:52:44.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.859+0000 7fd223fff700 1 -- 172.21.15.67:0/3165984494 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd22c0104f0 con 0x7fd23c101850 2024-04-22T23:52:44.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.859+0000 7fd223fff700 1 --2- 172.21.15.67:0/3165984494 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd22406ecc0 0x7fd224071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:44.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.859+0000 7fd223fff700 1 -- 172.21.15.67:0/3165984494 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fd22c08ec00 con 0x7fd23c101850 2024-04-22T23:52:44.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.859+0000 7fd23affd700 1 --2- 172.21.15.67:0/3165984494 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd22406ecc0 0x7fd224071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:44.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.860+0000 7fd23affd700 1 --2- 172.21.15.67:0/3165984494 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd22406ecc0 0x7fd224071180 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd23000b5c0 tx=0x7fd23001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:52:44.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:44.864+0000 7fd223fff700 1 -- 172.21.15.67:0/3165984494 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fd22c094020 con 0x7fd23c101850 2024-04-22T23:52:45.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.163+0000 7fd2416a9700 1 -- 172.21.15.67:0/3165984494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd23c04f2e0 con 0x7fd23c101850 2024-04-22T23:52:45.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.164+0000 7fd223fff700 1 -- 172.21.15.67:0/3165984494 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fd22c05b370 con 0x7fd23c101850 2024-04-22T23:52:45.155 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:52:45.155 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:52:45.155 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:52:45.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:52:45.155 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:52:45.155 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:52:45.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:52:45.156 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:52:45.156 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:52:45.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:52:45.156 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:52:45.156 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 4 2024-04-22T23:52:45.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:52:45.156 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:52:45.156 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 14 2024-04-22T23:52:45.156 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:52:45.156 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:52:45.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.166+0000 7fd2416a9700 1 -- 172.21.15.67:0/3165984494 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd22406ecc0 msgr2=0x7fd224071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:45.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.166+0000 7fd2416a9700 1 --2- 172.21.15.67:0/3165984494 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd22406ecc0 0x7fd224071180 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd23000b5c0 tx=0x7fd23001a040 comp rx=0 tx=0).stop 2024-04-22T23:52:45.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.166+0000 7fd2416a9700 1 -- 172.21.15.67:0/3165984494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd23c101850 msgr2=0x7fd23c196ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:45.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.166+0000 7fd2416a9700 1 --2- 172.21.15.67:0/3165984494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd23c101850 0x7fd23c196ed0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fd22c00b480 tx=0x7fd22c00b840 comp rx=0 tx=0).stop 2024-04-22T23:52:45.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.166+0000 7fd2416a9700 1 -- 172.21.15.67:0/3165984494 shutdown_connections 2024-04-22T23:52:45.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.166+0000 7fd2416a9700 1 --2- 172.21.15.67:0/3165984494 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd22406ecc0 0x7fd224071180 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:45.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.166+0000 7fd2416a9700 1 --2- 172.21.15.67:0/3165984494 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd23c100ea0 0x7fd23c196990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:45.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.166+0000 7fd2416a9700 1 --2- 172.21.15.67:0/3165984494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd23c101850 0x7fd23c196ed0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:45.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.166+0000 7fd2416a9700 1 -- 172.21.15.67:0/3165984494 >> 172.21.15.67:0/3165984494 conn(0x7fd23c0fc750 msgr2=0x7fd23c0fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:52:45.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.167+0000 7fd2416a9700 1 -- 172.21.15.67:0/3165984494 shutdown_connections 2024-04-22T23:52:45.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.167+0000 7fd2416a9700 1 -- 172.21.15.67:0/3165984494 wait complete. 2024-04-22T23:52:45.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.273+0000 7f6070688700 1 -- 172.21.15.67:0/614112815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6068101850 msgr2=0x7f60681058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:45.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.273+0000 7f6070688700 1 --2- 172.21.15.67:0/614112815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6068101850 0x7f60681058a0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f6064009910 tx=0x7f6064009c20 comp rx=0 tx=0).stop 2024-04-22T23:52:45.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.273+0000 7f6070688700 1 -- 172.21.15.67:0/614112815 shutdown_connections 2024-04-22T23:52:45.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.273+0000 7f6070688700 1 --2- 172.21.15.67:0/614112815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6068101850 0x7f60681058a0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:45.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.273+0000 7f6070688700 1 --2- 172.21.15.67:0/614112815 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6068100ea0 0x7f6068101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:45.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.273+0000 7f6070688700 1 -- 172.21.15.67:0/614112815 >> 172.21.15.67:0/614112815 conn(0x7f60680fc750 msgr2=0x7f60680feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:52:45.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.273+0000 7f6070688700 1 -- 172.21.15.67:0/614112815 shutdown_connections 2024-04-22T23:52:45.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.273+0000 7f6070688700 1 -- 172.21.15.67:0/614112815 wait complete. 2024-04-22T23:52:45.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.274+0000 7f6070688700 1 Processor -- start 2024-04-22T23:52:45.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.274+0000 7f6070688700 1 -- start start 2024-04-22T23:52:45.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.274+0000 7f6070688700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6068100ea0 0x7f606819c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:45.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.274+0000 7f6070688700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6068101850 0x7f606819cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:45.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.274+0000 7f6070688700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606819d250 con 0x7f6068101850 2024-04-22T23:52:45.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.274+0000 7f6070688700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6068196700 con 0x7f6068100ea0 2024-04-22T23:52:45.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.274+0000 7f606dc23700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6068101850 0x7f606819cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:45.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.274+0000 7f606dc23700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6068101850 0x7f606819cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51060/0 (socket says 172.21.15.67:51060) 2024-04-22T23:52:45.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.274+0000 7f606dc23700 1 -- 172.21.15.67:0/4050413174 learned_addr learned my addr 172.21.15.67:0/4050413174 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:52:45.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.274+0000 7f606e424700 1 --2- 172.21.15.67:0/4050413174 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6068100ea0 0x7f606819c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:45.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.275+0000 7f606dc23700 1 -- 172.21.15.67:0/4050413174 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6068100ea0 msgr2=0x7f606819c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:45.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.275+0000 7f606dc23700 1 --2- 172.21.15.67:0/4050413174 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6068100ea0 0x7f606819c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:45.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.275+0000 7f606dc23700 1 -- 172.21.15.67:0/4050413174 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60640095f0 con 0x7f6068101850 2024-04-22T23:52:45.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.275+0000 7f606dc23700 1 --2- 172.21.15.67:0/4050413174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6068101850 0x7f606819cbc0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f60640059d0 tx=0x7f606400f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:52:45.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.275+0000 7f605b7fe700 1 -- 172.21.15.67:0/4050413174 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f606401d070 con 0x7f6068101850 2024-04-22T23:52:45.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.275+0000 7f605b7fe700 1 -- 172.21.15.67:0/4050413174 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f606400b610 con 0x7f6068101850 2024-04-22T23:52:45.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.275+0000 7f605b7fe700 1 -- 172.21.15.67:0/4050413174 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6064022470 con 0x7f6068101850 2024-04-22T23:52:45.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.275+0000 7f6070688700 1 -- 172.21.15.67:0/4050413174 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6068196980 con 0x7f6068101850 2024-04-22T23:52:45.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.275+0000 7f6070688700 1 -- 172.21.15.67:0/4050413174 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6068196e20 con 0x7f6068101850 2024-04-22T23:52:45.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.276+0000 7f6070688700 1 -- 172.21.15.67:0/4050413174 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60681091b0 con 0x7f6068101850 2024-04-22T23:52:45.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.276+0000 7f605b7fe700 1 -- 172.21.15.67:0/4050413174 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f606400fc70 con 0x7f6068101850 2024-04-22T23:52:45.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.277+0000 7f605b7fe700 1 --2- 172.21.15.67:0/4050413174 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f605406ecc0 0x7f6054071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:45.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.277+0000 7f606e424700 1 --2- 172.21.15.67:0/4050413174 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f605406ecc0 0x7f6054071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:45.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.277+0000 7f605b7fe700 1 -- 172.21.15.67:0/4050413174 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f6064090420 con 0x7f6068101850 2024-04-22T23:52:45.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.277+0000 7f606e424700 1 --2- 172.21.15.67:0/4050413174 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f605406ecc0 0x7f6054071180 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f605c009bf0 tx=0x7f605c009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:52:45.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.282+0000 7f605b7fe700 1 -- 172.21.15.67:0/4050413174 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f606405cb90 con 0x7f6068101850 2024-04-22T23:52:45.305 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:45 smithi067 ceph-mon[112438]: from='client.35440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:52:45.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:45 smithi134 ceph-mon[85630]: from='client.35440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:52:45.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.541+0000 7f6070688700 1 -- 172.21.15.67:0/4050413174 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f606804f2e0 con 0x7f6068101850 2024-04-22T23:52:45.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.542+0000 7f605b7fe700 1 -- 172.21.15.67:0/4050413174 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1872 (secure 0 0 0) 0x7f606405c720 con 0x7f6068101850 2024-04-22T23:52:45.534 INFO:teuthology.orchestra.run.smithi067.stdout:e33 2024-04-22T23:52:45.534 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:52:45.534 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:52:45.534 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:52:45.534 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:52:45.534 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:52:45.534 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:52:45.534 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:52:45.534 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:52:45.534 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:52:45.535 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:52:45.535 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:52:45.535 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:52:45.535 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:52:45.535 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:52:45.535 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:52:45.535 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:52:45.535 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:52:45.535 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:52:45.535 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:52:45.535 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:52:45.535 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:52:45.535 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:52:45.535 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:52:45.535 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:52:45.536 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:52:45.536 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:52:45.536 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:52:45.536 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:52:45.536 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:52:45.536 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:52:45.536 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:52:45.536 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:52:45.536 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:52:45.536 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:52:45.536 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:52:45.536 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:52:45.536 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:52:45.536 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:52:45.536 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:52:45.537 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:52:45.537 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44851} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1849648622,v1:172.21.15.134:6827/1849648622] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:52:45.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.544+0000 7f6070688700 1 -- 172.21.15.67:0/4050413174 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f605406ecc0 msgr2=0x7f6054071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:45.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.544+0000 7f6070688700 1 --2- 172.21.15.67:0/4050413174 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f605406ecc0 0x7f6054071180 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f605c009bf0 tx=0x7f605c009260 comp rx=0 tx=0).stop 2024-04-22T23:52:45.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.544+0000 7f6070688700 1 -- 172.21.15.67:0/4050413174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6068101850 msgr2=0x7f606819cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:45.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.544+0000 7f6070688700 1 --2- 172.21.15.67:0/4050413174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6068101850 0x7f606819cbc0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f60640059d0 tx=0x7f606400f740 comp rx=0 tx=0).stop 2024-04-22T23:52:45.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.544+0000 7f6070688700 1 -- 172.21.15.67:0/4050413174 shutdown_connections 2024-04-22T23:52:45.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.544+0000 7f6070688700 1 --2- 172.21.15.67:0/4050413174 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f605406ecc0 0x7f6054071180 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:45.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.544+0000 7f6070688700 1 --2- 172.21.15.67:0/4050413174 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6068100ea0 0x7f606819c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:45.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.544+0000 7f6070688700 1 --2- 172.21.15.67:0/4050413174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6068101850 0x7f606819cbc0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:45.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.544+0000 7f6070688700 1 -- 172.21.15.67:0/4050413174 >> 172.21.15.67:0/4050413174 conn(0x7f60680fc750 msgr2=0x7f6068105110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:52:45.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.544+0000 7f6070688700 1 -- 172.21.15.67:0/4050413174 shutdown_connections 2024-04-22T23:52:45.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.544+0000 7f6070688700 1 -- 172.21.15.67:0/4050413174 wait complete. 2024-04-22T23:52:45.538 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 33 2024-04-22T23:52:45.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.653+0000 7fc5f14e4700 1 -- 172.21.15.67:0/446353685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5ec101850 msgr2=0x7fc5ec1058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:45.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.653+0000 7fc5f14e4700 1 --2- 172.21.15.67:0/446353685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5ec101850 0x7fc5ec1058a0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0009910 tx=0x7fc5e0009c20 comp rx=0 tx=0).stop 2024-04-22T23:52:45.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.653+0000 7fc5f14e4700 1 -- 172.21.15.67:0/446353685 shutdown_connections 2024-04-22T23:52:45.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.653+0000 7fc5f14e4700 1 --2- 172.21.15.67:0/446353685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5ec101850 0x7fc5ec1058a0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:45.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.653+0000 7fc5f14e4700 1 --2- 172.21.15.67:0/446353685 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5ec100ea0 0x7fc5ec101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:45.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.653+0000 7fc5f14e4700 1 -- 172.21.15.67:0/446353685 >> 172.21.15.67:0/446353685 conn(0x7fc5ec0fc750 msgr2=0x7fc5ec0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:52:45.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.653+0000 7fc5f14e4700 1 -- 172.21.15.67:0/446353685 shutdown_connections 2024-04-22T23:52:45.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.653+0000 7fc5f14e4700 1 -- 172.21.15.67:0/446353685 wait complete. 2024-04-22T23:52:45.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.654+0000 7fc5f14e4700 1 Processor -- start 2024-04-22T23:52:45.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.654+0000 7fc5f14e4700 1 -- start start 2024-04-22T23:52:45.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.654+0000 7fc5f14e4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5ec100ea0 0x7fc5ec073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:45.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.654+0000 7fc5f14e4700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5ec101850 0x7fc5ec0735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:45.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.654+0000 7fc5f14e4700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec073b10 con 0x7fc5ec100ea0 2024-04-22T23:52:45.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.654+0000 7fc5f14e4700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec073c50 con 0x7fc5ec101850 2024-04-22T23:52:45.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.655+0000 7fc5eaffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5ec100ea0 0x7fc5ec073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:45.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.655+0000 7fc5eaffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5ec100ea0 0x7fc5ec073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51072/0 (socket says 172.21.15.67:51072) 2024-04-22T23:52:45.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.655+0000 7fc5eaffd700 1 -- 172.21.15.67:0/406601901 learned_addr learned my addr 172.21.15.67:0/406601901 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:52:45.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.655+0000 7fc5ea7fc700 1 --2- 172.21.15.67:0/406601901 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5ec101850 0x7fc5ec0735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:45.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.655+0000 7fc5eaffd700 1 -- 172.21.15.67:0/406601901 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5ec101850 msgr2=0x7fc5ec0735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:45.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.655+0000 7fc5eaffd700 1 --2- 172.21.15.67:0/406601901 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5ec101850 0x7fc5ec0735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:45.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.655+0000 7fc5eaffd700 1 -- 172.21.15.67:0/406601901 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e00095f0 con 0x7fc5ec100ea0 2024-04-22T23:52:45.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.655+0000 7fc5eaffd700 1 --2- 172.21.15.67:0/406601901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5ec100ea0 0x7fc5ec073090 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc00d8d0 tx=0x7fc5dc00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:52:45.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.655+0000 7fc5d3fff700 1 -- 172.21.15.67:0/406601901 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5dc0098a0 con 0x7fc5ec100ea0 2024-04-22T23:52:45.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.655+0000 7fc5f14e4700 1 -- 172.21.15.67:0/406601901 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5ec1a2b00 con 0x7fc5ec100ea0 2024-04-22T23:52:45.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.655+0000 7fc5d3fff700 1 -- 172.21.15.67:0/406601901 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc5dc010460 con 0x7fc5ec100ea0 2024-04-22T23:52:45.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.655+0000 7fc5d3fff700 1 -- 172.21.15.67:0/406601901 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5dc00f5f0 con 0x7fc5ec100ea0 2024-04-22T23:52:45.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.655+0000 7fc5f14e4700 1 -- 172.21.15.67:0/406601901 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5ec1a2f10 con 0x7fc5ec100ea0 2024-04-22T23:52:45.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.656+0000 7fc5f14e4700 1 -- 172.21.15.67:0/406601901 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5ec1092a0 con 0x7fc5ec100ea0 2024-04-22T23:52:45.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.657+0000 7fc5d3fff700 1 -- 172.21.15.67:0/406601901 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc5dc009a00 con 0x7fc5ec100ea0 2024-04-22T23:52:45.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.658+0000 7fc5d3fff700 1 --2- 172.21.15.67:0/406601901 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc5d406ecc0 0x7fc5d4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:45.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.658+0000 7fc5ea7fc700 1 --2- 172.21.15.67:0/406601901 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc5d406ecc0 0x7fc5d4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:45.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.658+0000 7fc5d3fff700 1 -- 172.21.15.67:0/406601901 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fc5dc08eea0 con 0x7fc5ec100ea0 2024-04-22T23:52:45.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.658+0000 7fc5ea7fc700 1 --2- 172.21.15.67:0/406601901 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc5d406ecc0 0x7fc5d4071180 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec074880 tx=0x7fc5e001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:52:45.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.662+0000 7fc5d3fff700 1 -- 172.21.15.67:0/406601901 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc5dc057ae0 con 0x7fc5ec100ea0 2024-04-22T23:52:45.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.891+0000 7fc5f14e4700 1 -- 172.21.15.67:0/406601901 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc5ec0611d0 con 0x7fc5d406ecc0 2024-04-22T23:52:45.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.892+0000 7fc5d3fff700 1 -- 172.21.15.67:0/406601901 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc5ec0611d0 con 0x7fc5d406ecc0 2024-04-22T23:52:45.883 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:52:45.883 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:52:45.883 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:52:45.883 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:52:45.883 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:52:45.883 INFO:teuthology.orchestra.run.smithi067.stdout: "mds", 2024-04-22T23:52:45.884 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:52:45.884 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:52:45.884 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:52:45.884 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:52:45.884 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:52:45.884 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T23:52:45.884 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:52:45.884 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:52:45.884 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:52:45.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.894+0000 7fc5f14e4700 1 -- 172.21.15.67:0/406601901 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc5d406ecc0 msgr2=0x7fc5d4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:45.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.894+0000 7fc5f14e4700 1 --2- 172.21.15.67:0/406601901 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc5d406ecc0 0x7fc5d4071180 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec074880 tx=0x7fc5e001a040 comp rx=0 tx=0).stop 2024-04-22T23:52:45.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.894+0000 7fc5f14e4700 1 -- 172.21.15.67:0/406601901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5ec100ea0 msgr2=0x7fc5ec073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:45.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.894+0000 7fc5f14e4700 1 --2- 172.21.15.67:0/406601901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5ec100ea0 0x7fc5ec073090 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc00d8d0 tx=0x7fc5dc00dbe0 comp rx=0 tx=0).stop 2024-04-22T23:52:45.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.894+0000 7fc5f14e4700 1 -- 172.21.15.67:0/406601901 shutdown_connections 2024-04-22T23:52:45.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.894+0000 7fc5f14e4700 1 --2- 172.21.15.67:0/406601901 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc5d406ecc0 0x7fc5d4071180 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:45.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.894+0000 7fc5f14e4700 1 --2- 172.21.15.67:0/406601901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5ec100ea0 0x7fc5ec073090 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:45.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.894+0000 7fc5f14e4700 1 --2- 172.21.15.67:0/406601901 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5ec101850 0x7fc5ec0735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:45.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.894+0000 7fc5f14e4700 1 -- 172.21.15.67:0/406601901 >> 172.21.15.67:0/406601901 conn(0x7fc5ec0fc750 msgr2=0x7fc5ec0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:52:45.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.894+0000 7fc5f14e4700 1 -- 172.21.15.67:0/406601901 shutdown_connections 2024-04-22T23:52:45.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:45.894+0000 7fc5f14e4700 1 -- 172.21.15.67:0/406601901 wait complete. 2024-04-22T23:52:45.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.001+0000 7f77e6607700 1 -- 172.21.15.67:0/2485704243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77e0100ea0 msgr2=0x7f77e0101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:45.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.001+0000 7f77e6607700 1 --2- 172.21.15.67:0/2485704243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77e0100ea0 0x7f77e0101280 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f77d0009910 tx=0x7f77d0009c20 comp rx=0 tx=0).stop 2024-04-22T23:52:45.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.001+0000 7f77e6607700 1 -- 172.21.15.67:0/2485704243 shutdown_connections 2024-04-22T23:52:45.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.001+0000 7f77e6607700 1 --2- 172.21.15.67:0/2485704243 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77e0101850 0x7f77e01058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:45.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.001+0000 7f77e6607700 1 --2- 172.21.15.67:0/2485704243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77e0100ea0 0x7f77e0101280 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:45.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.001+0000 7f77e6607700 1 -- 172.21.15.67:0/2485704243 >> 172.21.15.67:0/2485704243 conn(0x7f77e00fc750 msgr2=0x7f77e00feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:52:45.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.001+0000 7f77e6607700 1 -- 172.21.15.67:0/2485704243 shutdown_connections 2024-04-22T23:52:45.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.001+0000 7f77e6607700 1 -- 172.21.15.67:0/2485704243 wait complete. 2024-04-22T23:52:45.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.002+0000 7f77e6607700 1 Processor -- start 2024-04-22T23:52:45.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.002+0000 7f77e6607700 1 -- start start 2024-04-22T23:52:45.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.002+0000 7f77e6607700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77e0100ea0 0x7f77e019c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:45.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.002+0000 7f77e6607700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77e0101850 0x7f77e019cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:45.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.002+0000 7f77e6607700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e019d2c0 con 0x7f77e0101850 2024-04-22T23:52:45.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.002+0000 7f77e6607700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e0196770 con 0x7f77e0100ea0 2024-04-22T23:52:45.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.002+0000 7f77df7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77e0101850 0x7f77e019cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:45.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.003+0000 7f77df7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77e0101850 0x7f77e019cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51104/0 (socket says 172.21.15.67:51104) 2024-04-22T23:52:45.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.003+0000 7f77df7fe700 1 -- 172.21.15.67:0/2661043626 learned_addr learned my addr 172.21.15.67:0/2661043626 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:52:45.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.003+0000 7f77dffff700 1 --2- 172.21.15.67:0/2661043626 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77e0100ea0 0x7f77e019c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:45.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.003+0000 7f77df7fe700 1 -- 172.21.15.67:0/2661043626 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77e0100ea0 msgr2=0x7f77e019c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:45.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.003+0000 7f77df7fe700 1 --2- 172.21.15.67:0/2661043626 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77e0100ea0 0x7f77e019c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:45.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.003+0000 7f77df7fe700 1 -- 172.21.15.67:0/2661043626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d00095f0 con 0x7f77e0101850 2024-04-22T23:52:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.003+0000 7f77df7fe700 1 --2- 172.21.15.67:0/2661043626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77e0101850 0x7f77e019cc30 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f77d400e8f0 tx=0x7f77d400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:52:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.003+0000 7f77dd7fa700 1 -- 172.21.15.67:0/2661043626 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77d4009790 con 0x7f77e0101850 2024-04-22T23:52:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.003+0000 7f77dd7fa700 1 -- 172.21.15.67:0/2661043626 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f77d400c810 con 0x7f77e0101850 2024-04-22T23:52:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.003+0000 7f77dd7fa700 1 -- 172.21.15.67:0/2661043626 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77d4005580 con 0x7f77e0101850 2024-04-22T23:52:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.003+0000 7f77e6607700 1 -- 172.21.15.67:0/2661043626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77e0196a50 con 0x7f77e0101850 2024-04-22T23:52:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.003+0000 7f77e6607700 1 -- 172.21.15.67:0/2661043626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77e0196fa0 con 0x7f77e0101850 2024-04-22T23:52:45.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.005+0000 7f77e6607700 1 -- 172.21.15.67:0/2661043626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77e0109240 con 0x7f77e0101850 2024-04-22T23:52:45.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.005+0000 7f77dd7fa700 1 -- 172.21.15.67:0/2661043626 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f77d400c980 con 0x7f77e0101850 2024-04-22T23:52:45.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.006+0000 7f77dd7fa700 1 --2- 172.21.15.67:0/2661043626 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f77c806ecc0 0x7f77c8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:52:45.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.006+0000 7f77dd7fa700 1 -- 172.21.15.67:0/2661043626 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f77d4014070 con 0x7f77e0101850 2024-04-22T23:52:45.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.006+0000 7f77dffff700 1 --2- 172.21.15.67:0/2661043626 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f77c806ecc0 0x7f77c8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:52:45.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.006+0000 7f77dffff700 1 --2- 172.21.15.67:0/2661043626 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f77c806ecc0 0x7f77c8071180 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f77d00059d0 tx=0x7f77d0005940 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:52:46.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.010+0000 7f77dd7fa700 1 -- 172.21.15.67:0/2661043626 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f77d405ba00 con 0x7f77e0101850 2024-04-22T23:52:46.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:46 smithi067 ceph-mon[112438]: from='client.35444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:52:46.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:46 smithi067 ceph-mon[112438]: from='client.35448 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:52:46.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:46 smithi067 ceph-mon[112438]: pgmap v793: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:46.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:46 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3165984494' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:52:46.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:46 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/4050413174' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:52:46.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.311+0000 7f77e6607700 1 -- 172.21.15.67:0/2661043626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f77e004f2e0 con 0x7f77e0101850 2024-04-22T23:52:46.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.311+0000 7f77dd7fa700 1 -- 172.21.15.67:0/2661043626 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4505 (secure 0 0 0) 0x7f77d4062030 con 0x7f77e0101850 2024-04-22T23:52:46.303 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:52:46.303 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T23:52:46.303 INFO:teuthology.orchestra.run.smithi067.stdout: host smithi134 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-22T23:52:46.303 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T23:52:46.303 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:52:46.303 INFO:teuthology.orchestra.run.smithi067.stdout:Traceback (most recent call last): 2024-04-22T23:52:46.303 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T23:52:46.303 INFO:teuthology.orchestra.run.smithi067.stdout: return _run_code(code, main_globals, None, 2024-04-22T23:52:46.303 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T23:52:46.303 INFO:teuthology.orchestra.run.smithi067.stdout: exec(code, run_globals) 2024-04-22T23:52:46.303 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5642, in 2024-04-22T23:52:46.304 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5630, in main 2024-04-22T23:52:46.304 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 408, in _infer_config 2024-04-22T23:52:46.304 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 323, in _infer_fsid 2024-04-22T23:52:46.304 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 436, in _infer_image 2024-04-22T23:52:46.304 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 310, in _validate_fsid 2024-04-22T23:52:46.304 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 3380, in command_ceph_volume 2024-04-22T23:52:46.304 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T23:52:46.304 INFO:teuthology.orchestra.run.smithi067.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:52:46.304 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:52:46.304 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:52:46.304 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:52:46.304 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:52:46.304 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:52:46.304 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:52:46.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.314+0000 7f77e6607700 1 -- 172.21.15.67:0/2661043626 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f77c806ecc0 msgr2=0x7f77c8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:46.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.314+0000 7f77e6607700 1 --2- 172.21.15.67:0/2661043626 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f77c806ecc0 0x7f77c8071180 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f77d00059d0 tx=0x7f77d0005940 comp rx=0 tx=0).stop 2024-04-22T23:52:46.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.314+0000 7f77e6607700 1 -- 172.21.15.67:0/2661043626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77e0101850 msgr2=0x7f77e019cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:52:46.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.314+0000 7f77e6607700 1 --2- 172.21.15.67:0/2661043626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77e0101850 0x7f77e019cc30 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f77d400e8f0 tx=0x7f77d400ec00 comp rx=0 tx=0).stop 2024-04-22T23:52:46.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.314+0000 7f77e6607700 1 -- 172.21.15.67:0/2661043626 shutdown_connections 2024-04-22T23:52:46.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.314+0000 7f77e6607700 1 --2- 172.21.15.67:0/2661043626 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f77c806ecc0 0x7f77c8071180 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:46.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.314+0000 7f77e6607700 1 --2- 172.21.15.67:0/2661043626 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77e0100ea0 0x7f77e019c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:46.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.314+0000 7f77e6607700 1 --2- 172.21.15.67:0/2661043626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77e0101850 0x7f77e019cc30 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:52:46.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.314+0000 7f77e6607700 1 -- 172.21.15.67:0/2661043626 >> 172.21.15.67:0/2661043626 conn(0x7f77e00fc750 msgr2=0x7f77e00fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:52:46.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.314+0000 7f77e6607700 1 -- 172.21.15.67:0/2661043626 shutdown_connections 2024-04-22T23:52:46.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:52:46.314+0000 7f77e6607700 1 -- 172.21.15.67:0/2661043626 wait complete. 2024-04-22T23:52:46.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:46 smithi134 ceph-mon[85630]: from='client.35444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:52:46.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:46 smithi134 ceph-mon[85630]: from='client.35448 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:52:46.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:46 smithi134 ceph-mon[85630]: pgmap v793: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:46.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:46 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3165984494' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:52:46.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:46 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/4050413174' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:52:47.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:47 smithi134 ceph-mon[85630]: from='client.35460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:52:47.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:47 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2661043626' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:52:47.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:47 smithi067 ceph-mon[112438]: from='client.35460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:52:47.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:47 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2661043626' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:52:48.386 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:48 smithi134 ceph-mon[85630]: pgmap v794: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:48.436 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:48 smithi067 ceph-mon[112438]: pgmap v794: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:50.358 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:50 smithi067 ceph-mon[112438]: pgmap v795: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:50.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:50 smithi134 ceph-mon[85630]: pgmap v795: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:52.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:52 smithi134 ceph-mon[85630]: pgmap v796: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:52.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:52 smithi067 ceph-mon[112438]: pgmap v796: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:54.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:54 smithi134 ceph-mon[85630]: pgmap v797: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:54.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:54 smithi067 ceph-mon[112438]: pgmap v797: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:55.410 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:55 smithi067 ceph-mon[112438]: pgmap v798: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:55.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:55 smithi134 ceph-mon[85630]: pgmap v798: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:56.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:52:56.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:52:57.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:52:57 smithi134 ceph-mon[85630]: pgmap v799: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:52:57.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:52:57 smithi067 ceph-mon[112438]: pgmap v799: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:00.465 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:00 smithi067 ceph-mon[112438]: pgmap v800: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:00.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:00 smithi134 ceph-mon[85630]: pgmap v800: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:02.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:02 smithi134 ceph-mon[85630]: pgmap v801: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:02.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:02 smithi067 ceph-mon[112438]: pgmap v801: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:04.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:04 smithi134 ceph-mon[85630]: pgmap v802: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:04.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:04 smithi067 ceph-mon[112438]: pgmap v802: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:06.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:06 smithi134 ceph-mon[85630]: pgmap v803: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:06.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:06 smithi067 ceph-mon[112438]: pgmap v803: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:08.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:08 smithi134 ceph-mon[85630]: pgmap v804: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:08.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:08 smithi067 ceph-mon[112438]: pgmap v804: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:10.234 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:10 smithi134 ceph-mon[85630]: pgmap v805: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:10.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:10 smithi067 ceph-mon[112438]: pgmap v805: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:11.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:53:11.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:53:12.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:12 smithi134 ceph-mon[85630]: pgmap v806: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:12.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:12 smithi067 ceph-mon[112438]: pgmap v806: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:13.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:13 smithi134 ceph-mon[85630]: pgmap v807: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:13.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:13 smithi067 ceph-mon[112438]: pgmap v807: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:16.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.436+0000 7f5b1af63700 1 -- 172.21.15.67:0/870127674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b140ffda0 msgr2=0x7f5b14100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:16.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.436+0000 7f5b1af63700 1 --2- 172.21.15.67:0/870127674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b140ffda0 0x7f5b14100180 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f5b08009910 tx=0x7f5b08009c20 comp rx=0 tx=0).stop 2024-04-22T23:53:16.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.436+0000 7f5b1af63700 1 -- 172.21.15.67:0/870127674 shutdown_connections 2024-04-22T23:53:16.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.436+0000 7f5b1af63700 1 --2- 172.21.15.67:0/870127674 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b14100750 0x7f5b141047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:16.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.436+0000 7f5b1af63700 1 --2- 172.21.15.67:0/870127674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b140ffda0 0x7f5b14100180 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:16.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.436+0000 7f5b1af63700 1 -- 172.21.15.67:0/870127674 >> 172.21.15.67:0/870127674 conn(0x7f5b140fb650 msgr2=0x7f5b140fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:53:16.428 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:16 smithi067 ceph-mon[112438]: pgmap v808: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:16.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.436+0000 7f5b1af63700 1 -- 172.21.15.67:0/870127674 shutdown_connections 2024-04-22T23:53:16.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.436+0000 7f5b1af63700 1 -- 172.21.15.67:0/870127674 wait complete. 2024-04-22T23:53:16.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.437+0000 7f5b1af63700 1 Processor -- start 2024-04-22T23:53:16.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.437+0000 7f5b1af63700 1 -- start start 2024-04-22T23:53:16.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.437+0000 7f5b1af63700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b140ffda0 0x7f5b1419c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:16.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.437+0000 7f5b1af63700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b14100750 0x7f5b1419cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:16.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.437+0000 7f5b1af63700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1419d2c0 con 0x7f5b140ffda0 2024-04-22T23:53:16.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.437+0000 7f5b1af63700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b14196770 con 0x7f5b14100750 2024-04-22T23:53:16.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.437+0000 7f5b18cff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b140ffda0 0x7f5b1419c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:16.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.437+0000 7f5b18cff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b140ffda0 0x7f5b1419c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55308/0 (socket says 172.21.15.67:55308) 2024-04-22T23:53:16.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.437+0000 7f5b18cff700 1 -- 172.21.15.67:0/3166582482 learned_addr learned my addr 172.21.15.67:0/3166582482 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:53:16.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.437+0000 7f5b13fff700 1 --2- 172.21.15.67:0/3166582482 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b14100750 0x7f5b1419cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:16.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.437+0000 7f5b18cff700 1 -- 172.21.15.67:0/3166582482 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b14100750 msgr2=0x7f5b1419cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:16.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.437+0000 7f5b18cff700 1 --2- 172.21.15.67:0/3166582482 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b14100750 0x7f5b1419cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:16.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.437+0000 7f5b18cff700 1 -- 172.21.15.67:0/3166582482 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b080095f0 con 0x7f5b140ffda0 2024-04-22T23:53:16.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.437+0000 7f5b13fff700 1 --2- 172.21.15.67:0/3166582482 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b14100750 0x7f5b1419cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:53:16.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.438+0000 7f5b18cff700 1 --2- 172.21.15.67:0/3166582482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b140ffda0 0x7f5b1419c6f0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f5b08006010 tx=0x7f5b0800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:53:16.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.438+0000 7f5b11ffb700 1 -- 172.21.15.67:0/3166582482 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b0801d070 con 0x7f5b140ffda0 2024-04-22T23:53:16.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.438+0000 7f5b11ffb700 1 -- 172.21.15.67:0/3166582482 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5b0800f460 con 0x7f5b140ffda0 2024-04-22T23:53:16.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.438+0000 7f5b1af63700 1 -- 172.21.15.67:0/3166582482 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b141969f0 con 0x7f5b140ffda0 2024-04-22T23:53:16.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.438+0000 7f5b11ffb700 1 -- 172.21.15.67:0/3166582482 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b08017630 con 0x7f5b140ffda0 2024-04-22T23:53:16.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.438+0000 7f5b1af63700 1 -- 172.21.15.67:0/3166582482 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b14196ee0 con 0x7f5b140ffda0 2024-04-22T23:53:16.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.439+0000 7f5b1af63700 1 -- 172.21.15.67:0/3166582482 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b1404f2e0 con 0x7f5b140ffda0 2024-04-22T23:53:16.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.440+0000 7f5b11ffb700 1 -- 172.21.15.67:0/3166582482 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5b0800f5d0 con 0x7f5b140ffda0 2024-04-22T23:53:16.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.440+0000 7f5b11ffb700 1 --2- 172.21.15.67:0/3166582482 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5b0406ecc0 0x7f5b04071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:16.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.441+0000 7f5b11ffb700 1 -- 172.21.15.67:0/3166582482 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f5b08090420 con 0x7f5b140ffda0 2024-04-22T23:53:16.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.441+0000 7f5b13fff700 1 --2- 172.21.15.67:0/3166582482 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5b0406ecc0 0x7f5b04071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:16.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.441+0000 7f5b13fff700 1 --2- 172.21.15.67:0/3166582482 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5b0406ecc0 0x7f5b04071180 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f5b00005fd0 tx=0x7f5b00005ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:53:16.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.446+0000 7f5b11ffb700 1 -- 172.21.15.67:0/3166582482 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f5b0805cc10 con 0x7f5b140ffda0 2024-04-22T23:53:16.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:16 smithi134 ceph-mon[85630]: pgmap v808: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:16.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.679+0000 7f5b1af63700 1 -- 172.21.15.67:0/3166582482 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b14197170 con 0x7f5b0406ecc0 2024-04-22T23:53:16.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.680+0000 7f5b11ffb700 1 -- 172.21.15.67:0/3166582482 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f5b14197170 con 0x7f5b0406ecc0 2024-04-22T23:53:16.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.682+0000 7f5b1af63700 1 -- 172.21.15.67:0/3166582482 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5b0406ecc0 msgr2=0x7f5b04071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:16.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.682+0000 7f5b1af63700 1 --2- 172.21.15.67:0/3166582482 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5b0406ecc0 0x7f5b04071180 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f5b00005fd0 tx=0x7f5b00005ee0 comp rx=0 tx=0).stop 2024-04-22T23:53:16.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.682+0000 7f5b1af63700 1 -- 172.21.15.67:0/3166582482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b140ffda0 msgr2=0x7f5b1419c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:16.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.682+0000 7f5b1af63700 1 --2- 172.21.15.67:0/3166582482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b140ffda0 0x7f5b1419c6f0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f5b08006010 tx=0x7f5b0800be20 comp rx=0 tx=0).stop 2024-04-22T23:53:16.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.682+0000 7f5b1af63700 1 -- 172.21.15.67:0/3166582482 shutdown_connections 2024-04-22T23:53:16.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.682+0000 7f5b1af63700 1 --2- 172.21.15.67:0/3166582482 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5b0406ecc0 0x7f5b04071180 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:16.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.682+0000 7f5b1af63700 1 --2- 172.21.15.67:0/3166582482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b140ffda0 0x7f5b1419c6f0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:16.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.682+0000 7f5b1af63700 1 --2- 172.21.15.67:0/3166582482 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b14100750 0x7f5b1419cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:16.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.682+0000 7f5b1af63700 1 -- 172.21.15.67:0/3166582482 >> 172.21.15.67:0/3166582482 conn(0x7f5b140fb650 msgr2=0x7f5b14103fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:53:16.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.682+0000 7f5b1af63700 1 -- 172.21.15.67:0/3166582482 shutdown_connections 2024-04-22T23:53:16.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.682+0000 7f5b1af63700 1 -- 172.21.15.67:0/3166582482 wait complete. 2024-04-22T23:53:16.684 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:53:16.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.790+0000 7f7ad2f5a700 1 -- 172.21.15.67:0/916418888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7acc0ffda0 msgr2=0x7f7acc100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:16.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.790+0000 7f7ad2f5a700 1 --2- 172.21.15.67:0/916418888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7acc0ffda0 0x7f7acc100180 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f7ab4009910 tx=0x7f7ab4009c20 comp rx=0 tx=0).stop 2024-04-22T23:53:16.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.790+0000 7f7ad2f5a700 1 -- 172.21.15.67:0/916418888 shutdown_connections 2024-04-22T23:53:16.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.790+0000 7f7ad2f5a700 1 --2- 172.21.15.67:0/916418888 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7acc100750 0x7f7acc1047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:16.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.790+0000 7f7ad2f5a700 1 --2- 172.21.15.67:0/916418888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7acc0ffda0 0x7f7acc100180 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:16.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.790+0000 7f7ad2f5a700 1 -- 172.21.15.67:0/916418888 >> 172.21.15.67:0/916418888 conn(0x7f7acc0fb650 msgr2=0x7f7acc0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:53:16.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.790+0000 7f7ad2f5a700 1 -- 172.21.15.67:0/916418888 shutdown_connections 2024-04-22T23:53:16.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.790+0000 7f7ad2f5a700 1 -- 172.21.15.67:0/916418888 wait complete. 2024-04-22T23:53:16.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.791+0000 7f7ad2f5a700 1 Processor -- start 2024-04-22T23:53:16.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.791+0000 7f7ad2f5a700 1 -- start start 2024-04-22T23:53:16.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.791+0000 7f7ad2f5a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7acc0ffda0 0x7f7acc1976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:16.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.791+0000 7f7ad2f5a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7acc100750 0x7f7acc197c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:16.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.791+0000 7f7ad2f5a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc1981c0 con 0x7f7acc0ffda0 2024-04-22T23:53:16.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.791+0000 7f7ad2f5a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc198300 con 0x7f7acc100750 2024-04-22T23:53:16.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.791+0000 7f7ad0cf6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7acc0ffda0 0x7f7acc1976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:16.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.791+0000 7f7ad0cf6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7acc0ffda0 0x7f7acc1976f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55320/0 (socket says 172.21.15.67:55320) 2024-04-22T23:53:16.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.791+0000 7f7ad0cf6700 1 -- 172.21.15.67:0/3386154049 learned_addr learned my addr 172.21.15.67:0/3386154049 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:53:16.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.792+0000 7f7acbfff700 1 --2- 172.21.15.67:0/3386154049 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7acc100750 0x7f7acc197c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:16.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.792+0000 7f7ad0cf6700 1 -- 172.21.15.67:0/3386154049 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7acc100750 msgr2=0x7f7acc197c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:16.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.792+0000 7f7ad0cf6700 1 --2- 172.21.15.67:0/3386154049 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7acc100750 0x7f7acc197c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:16.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.792+0000 7f7ad0cf6700 1 -- 172.21.15.67:0/3386154049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ab40095f0 con 0x7f7acc0ffda0 2024-04-22T23:53:16.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.792+0000 7f7acbfff700 1 --2- 172.21.15.67:0/3386154049 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7acc100750 0x7f7acc197c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:53:16.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.792+0000 7f7ad0cf6700 1 --2- 172.21.15.67:0/3386154049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7acc0ffda0 0x7f7acc1976f0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f7ab4006010 tx=0x7f7ab400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:53:16.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.792+0000 7f7ac9ffb700 1 -- 172.21.15.67:0/3386154049 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ab401d070 con 0x7f7acc0ffda0 2024-04-22T23:53:16.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.792+0000 7f7ac9ffb700 1 -- 172.21.15.67:0/3386154049 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7ab4004fb0 con 0x7f7acc0ffda0 2024-04-22T23:53:16.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.792+0000 7f7ad2f5a700 1 -- 172.21.15.67:0/3386154049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7acc1a0760 con 0x7f7acc0ffda0 2024-04-22T23:53:16.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.792+0000 7f7ac9ffb700 1 -- 172.21.15.67:0/3386154049 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ab4022ea0 con 0x7f7acc0ffda0 2024-04-22T23:53:16.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.792+0000 7f7ad2f5a700 1 -- 172.21.15.67:0/3386154049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7acc1a0b10 con 0x7f7acc0ffda0 2024-04-22T23:53:16.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.793+0000 7f7ad2f5a700 1 -- 172.21.15.67:0/3386154049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7acc108140 con 0x7f7acc0ffda0 2024-04-22T23:53:16.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.794+0000 7f7ac9ffb700 1 -- 172.21.15.67:0/3386154049 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7ab4004af0 con 0x7f7acc0ffda0 2024-04-22T23:53:16.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.794+0000 7f7ac9ffb700 1 --2- 172.21.15.67:0/3386154049 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7ab806ecc0 0x7f7ab8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:16.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.794+0000 7f7ac9ffb700 1 -- 172.21.15.67:0/3386154049 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f7ab40904c0 con 0x7f7acc0ffda0 2024-04-22T23:53:16.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.794+0000 7f7acbfff700 1 --2- 172.21.15.67:0/3386154049 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7ab806ecc0 0x7f7ab8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:16.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.795+0000 7f7acbfff700 1 --2- 172.21.15.67:0/3386154049 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7ab806ecc0 0x7f7ab8071180 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f7abc005fd0 tx=0x7f7abc005e40 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:53:16.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:16.799+0000 7f7ac9ffb700 1 -- 172.21.15.67:0/3386154049 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7ab4059230 con 0x7f7acc0ffda0 2024-04-22T23:53:17.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.029+0000 7f7ad2f5a700 1 -- 172.21.15.67:0/3386154049 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7acc0611d0 con 0x7f7ab806ecc0 2024-04-22T23:53:17.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.031+0000 7f7ac9ffb700 1 -- 172.21.15.67:0/3386154049 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f7acc0611d0 con 0x7f7ab806ecc0 2024-04-22T23:53:17.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.033+0000 7f7ad2f5a700 1 -- 172.21.15.67:0/3386154049 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7ab806ecc0 msgr2=0x7f7ab8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:17.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.033+0000 7f7ad2f5a700 1 --2- 172.21.15.67:0/3386154049 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7ab806ecc0 0x7f7ab8071180 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f7abc005fd0 tx=0x7f7abc005e40 comp rx=0 tx=0).stop 2024-04-22T23:53:17.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.033+0000 7f7ad2f5a700 1 -- 172.21.15.67:0/3386154049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7acc0ffda0 msgr2=0x7f7acc1976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:17.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.033+0000 7f7ad2f5a700 1 --2- 172.21.15.67:0/3386154049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7acc0ffda0 0x7f7acc1976f0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f7ab4006010 tx=0x7f7ab400be20 comp rx=0 tx=0).stop 2024-04-22T23:53:17.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.033+0000 7f7ad2f5a700 1 -- 172.21.15.67:0/3386154049 shutdown_connections 2024-04-22T23:53:17.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.033+0000 7f7ad2f5a700 1 --2- 172.21.15.67:0/3386154049 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7ab806ecc0 0x7f7ab8071180 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:17.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.033+0000 7f7ad2f5a700 1 --2- 172.21.15.67:0/3386154049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7acc0ffda0 0x7f7acc1976f0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:17.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.033+0000 7f7ad2f5a700 1 --2- 172.21.15.67:0/3386154049 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7acc100750 0x7f7acc197c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:17.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.033+0000 7f7ad2f5a700 1 -- 172.21.15.67:0/3386154049 >> 172.21.15.67:0/3386154049 conn(0x7f7acc0fb650 msgr2=0x7f7acc0fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:53:17.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.033+0000 7f7ad2f5a700 1 -- 172.21.15.67:0/3386154049 shutdown_connections 2024-04-22T23:53:17.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.033+0000 7f7ad2f5a700 1 -- 172.21.15.67:0/3386154049 wait complete. 2024-04-22T23:53:17.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.140+0000 7feeedc6b700 1 -- 172.21.15.67:0/215229273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feee8105bd0 msgr2=0x7feee8105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:17.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.140+0000 7feeedc6b700 1 --2- 172.21.15.67:0/215229273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feee8105bd0 0x7feee8105fb0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7feed0009910 tx=0x7feed0009c20 comp rx=0 tx=0).stop 2024-04-22T23:53:17.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.140+0000 7feeedc6b700 1 -- 172.21.15.67:0/215229273 shutdown_connections 2024-04-22T23:53:17.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.140+0000 7feeedc6b700 1 --2- 172.21.15.67:0/215229273 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feee80ffc20 0x7feee8100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:17.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.140+0000 7feeedc6b700 1 --2- 172.21.15.67:0/215229273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feee8105bd0 0x7feee8105fb0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:17.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.140+0000 7feeedc6b700 1 -- 172.21.15.67:0/215229273 >> 172.21.15.67:0/215229273 conn(0x7feee8075670 msgr2=0x7feee8075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:53:17.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.140+0000 7feeedc6b700 1 -- 172.21.15.67:0/215229273 shutdown_connections 2024-04-22T23:53:17.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.140+0000 7feeedc6b700 1 -- 172.21.15.67:0/215229273 wait complete. 2024-04-22T23:53:17.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.141+0000 7feeedc6b700 1 Processor -- start 2024-04-22T23:53:17.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.141+0000 7feeedc6b700 1 -- start start 2024-04-22T23:53:17.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.141+0000 7feeedc6b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feee80ffc20 0x7feee819e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:17.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.141+0000 7feeedc6b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feee8105bd0 0x7feee819ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:17.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.141+0000 7feeedc6b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee819f360 con 0x7feee8105bd0 2024-04-22T23:53:17.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.141+0000 7feeedc6b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee81988a0 con 0x7feee80ffc20 2024-04-22T23:53:17.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.141+0000 7feee6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feee8105bd0 0x7feee819ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:17.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.141+0000 7feee6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feee8105bd0 0x7feee819ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55342/0 (socket says 172.21.15.67:55342) 2024-04-22T23:53:17.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.141+0000 7feee6ffd700 1 -- 172.21.15.67:0/1403750674 learned_addr learned my addr 172.21.15.67:0/1403750674 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:53:17.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.141+0000 7feee77fe700 1 --2- 172.21.15.67:0/1403750674 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feee80ffc20 0x7feee819e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:17.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.142+0000 7feee6ffd700 1 -- 172.21.15.67:0/1403750674 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feee80ffc20 msgr2=0x7feee819e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:17.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.142+0000 7feee6ffd700 1 --2- 172.21.15.67:0/1403750674 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feee80ffc20 0x7feee819e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:17.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.142+0000 7feee6ffd700 1 -- 172.21.15.67:0/1403750674 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed00095f0 con 0x7feee8105bd0 2024-04-22T23:53:17.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.142+0000 7feee6ffd700 1 --2- 172.21.15.67:0/1403750674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feee8105bd0 0x7feee819ed60 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7feed800c930 tx=0x7feed800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:53:17.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.142+0000 7feee4ff9700 1 -- 172.21.15.67:0/1403750674 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feed8007ad0 con 0x7feee8105bd0 2024-04-22T23:53:17.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.142+0000 7feee4ff9700 1 -- 172.21.15.67:0/1403750674 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feed800ce80 con 0x7feee8105bd0 2024-04-22T23:53:17.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.142+0000 7feeedc6b700 1 -- 172.21.15.67:0/1403750674 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee8198b80 con 0x7feee8105bd0 2024-04-22T23:53:17.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.142+0000 7feee4ff9700 1 -- 172.21.15.67:0/1403750674 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feed8018630 con 0x7feee8105bd0 2024-04-22T23:53:17.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.142+0000 7feeedc6b700 1 -- 172.21.15.67:0/1403750674 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feee81990d0 con 0x7feee8105bd0 2024-04-22T23:53:17.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.143+0000 7feeedc6b700 1 -- 172.21.15.67:0/1403750674 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee804f2e0 con 0x7feee8105bd0 2024-04-22T23:53:17.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.144+0000 7feee4ff9700 1 -- 172.21.15.67:0/1403750674 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7feed8007c30 con 0x7feee8105bd0 2024-04-22T23:53:17.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.144+0000 7feee4ff9700 1 --2- 172.21.15.67:0/1403750674 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7feed406ecc0 0x7feed4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:17.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.144+0000 7feee4ff9700 1 -- 172.21.15.67:0/1403750674 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7feed808ef50 con 0x7feee8105bd0 2024-04-22T23:53:17.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.144+0000 7feee77fe700 1 --2- 172.21.15.67:0/1403750674 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7feed406ecc0 0x7feed4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:17.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.145+0000 7feee77fe700 1 --2- 172.21.15.67:0/1403750674 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7feed406ecc0 0x7feed4071180 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7feed0006010 tx=0x7feed001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:53:17.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.149+0000 7feee4ff9700 1 -- 172.21.15.67:0/1403750674 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7feed80c4150 con 0x7feee8105bd0 2024-04-22T23:53:17.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.372+0000 7feeedc6b700 1 -- 172.21.15.67:0/1403750674 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7feee8199400 con 0x7feed406ecc0 2024-04-22T23:53:17.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.383+0000 7feee4ff9700 1 -- 172.21.15.67:0/1403750674 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7feee8199400 con 0x7feed406ecc0 2024-04-22T23:53:17.374 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:53:17.375 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (27m) 4m ago 38m 24.4M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:53:17.375 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (38m) 4m ago 38m 10.0M - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:53:17.375 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (37m) 4m ago 37m 10.0M - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:53:17.375 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (24m) 4m ago 38m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:53:17.375 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (24m) 4m ago 37m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:53:17.375 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (26m) 4m ago 37m 81.2M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:53:17.375 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (20m) 4m ago 32m 19.4M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:53:17.375 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (20m) 4m ago 32m 20.3M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:53:17.375 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (20m) 4m ago 32m 20.1M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:53:17.375 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (4m) 4m ago 32m 14.0M - 19.0.0-3252-g43be0201 a0434699c1d7 46d3d7e86fee 2024-04-22T23:53:17.375 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (28m) 4m ago 39m 514M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:53:17.375 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (27m) 4m ago 37m 459M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:53:17.375 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (25m) 4m ago 39m 74.2M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:53:17.376 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (25m) 4m ago 37m 54.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:53:17.376 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (27m) 4m ago 38m 15.2M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:53:17.376 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (27m) 4m ago 37m 16.0M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:53:17.376 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (23m) 4m ago 36m 102M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:53:17.376 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (23m) 4m ago 36m 128M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:53:17.376 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (22m) 4m ago 35m 83.8M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:53:17.376 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (22m) 4m ago 35m 101M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:53:17.376 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (21m) 4m ago 34m 86.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:53:17.376 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (21m) 4m ago 34m 81.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:53:17.376 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (27m) 4m ago 37m 78.7M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:53:17.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.385+0000 7feeedc6b700 1 -- 172.21.15.67:0/1403750674 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7feed406ecc0 msgr2=0x7feed4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:17.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.385+0000 7feeedc6b700 1 --2- 172.21.15.67:0/1403750674 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7feed406ecc0 0x7feed4071180 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7feed0006010 tx=0x7feed001a040 comp rx=0 tx=0).stop 2024-04-22T23:53:17.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.385+0000 7feeedc6b700 1 -- 172.21.15.67:0/1403750674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feee8105bd0 msgr2=0x7feee819ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:17.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.385+0000 7feeedc6b700 1 --2- 172.21.15.67:0/1403750674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feee8105bd0 0x7feee819ed60 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7feed800c930 tx=0x7feed800cc40 comp rx=0 tx=0).stop 2024-04-22T23:53:17.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.385+0000 7feeedc6b700 1 -- 172.21.15.67:0/1403750674 shutdown_connections 2024-04-22T23:53:17.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.385+0000 7feeedc6b700 1 --2- 172.21.15.67:0/1403750674 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7feed406ecc0 0x7feed4071180 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:17.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.385+0000 7feeedc6b700 1 --2- 172.21.15.67:0/1403750674 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7feee80ffc20 0x7feee819e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:17.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.385+0000 7feeedc6b700 1 --2- 172.21.15.67:0/1403750674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feee8105bd0 0x7feee819ed60 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:17.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.385+0000 7feeedc6b700 1 -- 172.21.15.67:0/1403750674 >> 172.21.15.67:0/1403750674 conn(0x7feee8075670 msgr2=0x7feee80686c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:53:17.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.385+0000 7feeedc6b700 1 -- 172.21.15.67:0/1403750674 shutdown_connections 2024-04-22T23:53:17.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.385+0000 7feeedc6b700 1 -- 172.21.15.67:0/1403750674 wait complete. 2024-04-22T23:53:17.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.491+0000 7f5fc8430700 1 -- 172.21.15.67:0/2145423412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fc0101e80 msgr2=0x7f5fc0102260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:17.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.491+0000 7f5fc8430700 1 --2- 172.21.15.67:0/2145423412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fc0101e80 0x7f5fc0102260 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f5fb800b0a0 tx=0x7f5fb800b3b0 comp rx=0 tx=0).stop 2024-04-22T23:53:17.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.491+0000 7f5fc8430700 1 -- 172.21.15.67:0/2145423412 shutdown_connections 2024-04-22T23:53:17.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.491+0000 7f5fc8430700 1 --2- 172.21.15.67:0/2145423412 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5fc01027a0 0x7f5fc010ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:17.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.491+0000 7f5fc8430700 1 --2- 172.21.15.67:0/2145423412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fc0101e80 0x7f5fc0102260 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:17.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.491+0000 7f5fc8430700 1 -- 172.21.15.67:0/2145423412 >> 172.21.15.67:0/2145423412 conn(0x7f5fc00fb650 msgr2=0x7f5fc00fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:53:17.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.491+0000 7f5fc8430700 1 -- 172.21.15.67:0/2145423412 shutdown_connections 2024-04-22T23:53:17.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.491+0000 7f5fc8430700 1 -- 172.21.15.67:0/2145423412 wait complete. 2024-04-22T23:53:17.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.492+0000 7f5fc8430700 1 Processor -- start 2024-04-22T23:53:17.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.492+0000 7f5fc8430700 1 -- start start 2024-04-22T23:53:17.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.492+0000 7f5fc8430700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5fc0101e80 0x7f5fc00ffae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:17.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.492+0000 7f5fc8430700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fc01027a0 0x7f5fc0100020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:17.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.492+0000 7f5fc8430700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc0100560 con 0x7f5fc01027a0 2024-04-22T23:53:17.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.492+0000 7f5fc8430700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc01006a0 con 0x7f5fc0101e80 2024-04-22T23:53:17.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.492+0000 7f5fc59cb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fc01027a0 0x7f5fc0100020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:17.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.492+0000 7f5fc59cb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fc01027a0 0x7f5fc0100020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55346/0 (socket says 172.21.15.67:55346) 2024-04-22T23:53:17.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.492+0000 7f5fc59cb700 1 -- 172.21.15.67:0/2437281599 learned_addr learned my addr 172.21.15.67:0/2437281599 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:53:17.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.492+0000 7f5fc59cb700 1 -- 172.21.15.67:0/2437281599 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5fc0101e80 msgr2=0x7f5fc00ffae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:17.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.492+0000 7f5fc61cc700 1 --2- 172.21.15.67:0/2437281599 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5fc0101e80 0x7f5fc00ffae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:17.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.492+0000 7f5fc59cb700 1 --2- 172.21.15.67:0/2437281599 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5fc0101e80 0x7f5fc00ffae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:17.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.492+0000 7f5fc59cb700 1 -- 172.21.15.67:0/2437281599 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb8009d00 con 0x7f5fc01027a0 2024-04-22T23:53:17.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.492+0000 7f5fc59cb700 1 --2- 172.21.15.67:0/2437281599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fc01027a0 0x7f5fc0100020 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f5fb400d900 tx=0x7f5fb400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:53:17.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.493+0000 7f5fbf7fe700 1 -- 172.21.15.67:0/2437281599 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5fb4009900 con 0x7f5fc01027a0 2024-04-22T23:53:17.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.493+0000 7f5fc8430700 1 -- 172.21.15.67:0/2437281599 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fc01a2b40 con 0x7f5fc01027a0 2024-04-22T23:53:17.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.493+0000 7f5fbf7fe700 1 -- 172.21.15.67:0/2437281599 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5fb4010460 con 0x7f5fc01027a0 2024-04-22T23:53:17.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.493+0000 7f5fbf7fe700 1 -- 172.21.15.67:0/2437281599 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5fb400f5f0 con 0x7f5fc01027a0 2024-04-22T23:53:17.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.493+0000 7f5fc8430700 1 -- 172.21.15.67:0/2437281599 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fc01a2f50 con 0x7f5fc01027a0 2024-04-22T23:53:17.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.494+0000 7f5fc8430700 1 -- 172.21.15.67:0/2437281599 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fc0108330 con 0x7f5fc01027a0 2024-04-22T23:53:17.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.494+0000 7f5fbf7fe700 1 -- 172.21.15.67:0/2437281599 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5fb40105d0 con 0x7f5fc01027a0 2024-04-22T23:53:17.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.495+0000 7f5fbf7fe700 1 --2- 172.21.15.67:0/2437281599 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5fa806ecc0 0x7f5fa8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:17.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.495+0000 7f5fbf7fe700 1 -- 172.21.15.67:0/2437281599 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f5fb408ed50 con 0x7f5fc01027a0 2024-04-22T23:53:17.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.495+0000 7f5fc61cc700 1 --2- 172.21.15.67:0/2437281599 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5fa806ecc0 0x7f5fa8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:17.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.495+0000 7f5fc61cc700 1 --2- 172.21.15.67:0/2437281599 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5fa806ecc0 0x7f5fa8071180 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f5fb8015040 tx=0x7f5fb8000f40 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:53:17.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.500+0000 7f5fbf7fe700 1 -- 172.21.15.67:0/2437281599 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f5fb4057a10 con 0x7f5fc01027a0 2024-04-22T23:53:17.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.797+0000 7f5fc8430700 1 -- 172.21.15.67:0/2437281599 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5fc004f2e0 con 0x7f5fc01027a0 2024-04-22T23:53:17.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.798+0000 7f5fbf7fe700 1 -- 172.21.15.67:0/2437281599 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f5fb405b0d0 con 0x7f5fc01027a0 2024-04-22T23:53:17.789 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:53:17.789 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:53:17.789 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:53:17.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:53:17.789 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:53:17.790 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:53:17.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:53:17.790 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:53:17.790 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:53:17.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:53:17.790 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:53:17.790 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 4 2024-04-22T23:53:17.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:53:17.790 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:53:17.790 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 14 2024-04-22T23:53:17.790 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:53:17.790 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:53:17.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.800+0000 7f5fc8430700 1 -- 172.21.15.67:0/2437281599 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5fa806ecc0 msgr2=0x7f5fa8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:17.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.800+0000 7f5fc8430700 1 --2- 172.21.15.67:0/2437281599 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5fa806ecc0 0x7f5fa8071180 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f5fb8015040 tx=0x7f5fb8000f40 comp rx=0 tx=0).stop 2024-04-22T23:53:17.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.800+0000 7f5fc8430700 1 -- 172.21.15.67:0/2437281599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fc01027a0 msgr2=0x7f5fc0100020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:17.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.800+0000 7f5fc8430700 1 --2- 172.21.15.67:0/2437281599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fc01027a0 0x7f5fc0100020 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f5fb400d900 tx=0x7f5fb400dcc0 comp rx=0 tx=0).stop 2024-04-22T23:53:17.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.800+0000 7f5fc8430700 1 -- 172.21.15.67:0/2437281599 shutdown_connections 2024-04-22T23:53:17.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.800+0000 7f5fc8430700 1 --2- 172.21.15.67:0/2437281599 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5fa806ecc0 0x7f5fa8071180 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:17.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.800+0000 7f5fc8430700 1 --2- 172.21.15.67:0/2437281599 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5fc0101e80 0x7f5fc00ffae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:17.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.800+0000 7f5fc8430700 1 --2- 172.21.15.67:0/2437281599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5fc01027a0 0x7f5fc0100020 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:17.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.800+0000 7f5fc8430700 1 -- 172.21.15.67:0/2437281599 >> 172.21.15.67:0/2437281599 conn(0x7f5fc00fb650 msgr2=0x7f5fc0105470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:53:17.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.800+0000 7f5fc8430700 1 -- 172.21.15.67:0/2437281599 shutdown_connections 2024-04-22T23:53:17.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.800+0000 7f5fc8430700 1 -- 172.21.15.67:0/2437281599 wait complete. 2024-04-22T23:53:17.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.907+0000 7fee529db700 1 -- 172.21.15.67:0/1596326666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee4c0ffda0 msgr2=0x7fee4c100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.907+0000 7fee529db700 1 --2- 172.21.15.67:0/1596326666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee4c0ffda0 0x7fee4c100180 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fee3c009910 tx=0x7fee3c009c20 comp rx=0 tx=0).stop 2024-04-22T23:53:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.907+0000 7fee529db700 1 -- 172.21.15.67:0/1596326666 shutdown_connections 2024-04-22T23:53:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.907+0000 7fee529db700 1 --2- 172.21.15.67:0/1596326666 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee4c100750 0x7fee4c1047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.907+0000 7fee529db700 1 --2- 172.21.15.67:0/1596326666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee4c0ffda0 0x7fee4c100180 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.907+0000 7fee529db700 1 -- 172.21.15.67:0/1596326666 >> 172.21.15.67:0/1596326666 conn(0x7fee4c0fb650 msgr2=0x7fee4c0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:53:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.907+0000 7fee529db700 1 -- 172.21.15.67:0/1596326666 shutdown_connections 2024-04-22T23:53:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.907+0000 7fee529db700 1 -- 172.21.15.67:0/1596326666 wait complete. 2024-04-22T23:53:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.908+0000 7fee529db700 1 Processor -- start 2024-04-22T23:53:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.908+0000 7fee529db700 1 -- start start 2024-04-22T23:53:17.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.908+0000 7fee529db700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee4c0ffda0 0x7fee4c19e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:17.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.908+0000 7fee529db700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee4c100750 0x7fee4c19ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:17.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.908+0000 7fee529db700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4c19f4d0 con 0x7fee4c0ffda0 2024-04-22T23:53:17.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.908+0000 7fee529db700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4c198980 con 0x7fee4c100750 2024-04-22T23:53:17.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.908+0000 7fee4bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee4c0ffda0 0x7fee4c19e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:17.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.908+0000 7fee4bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee4c0ffda0 0x7fee4c19e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55362/0 (socket says 172.21.15.67:55362) 2024-04-22T23:53:17.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.908+0000 7fee4bfff700 1 -- 172.21.15.67:0/1030351165 learned_addr learned my addr 172.21.15.67:0/1030351165 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:53:17.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.908+0000 7fee43fff700 1 --2- 172.21.15.67:0/1030351165 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee4c100750 0x7fee4c19ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:17.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.908+0000 7fee4bfff700 1 -- 172.21.15.67:0/1030351165 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee4c100750 msgr2=0x7fee4c19ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:17.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.908+0000 7fee4bfff700 1 --2- 172.21.15.67:0/1030351165 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee4c100750 0x7fee4c19ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:17.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.908+0000 7fee4bfff700 1 -- 172.21.15.67:0/1030351165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee3c0095f0 con 0x7fee4c0ffda0 2024-04-22T23:53:17.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.908+0000 7fee43fff700 1 --2- 172.21.15.67:0/1030351165 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee4c100750 0x7fee4c19ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:53:17.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.909+0000 7fee4bfff700 1 --2- 172.21.15.67:0/1030351165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee4c0ffda0 0x7fee4c19e900 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fee3c006010 tx=0x7fee3c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:53:17.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.909+0000 7fee49ffb700 1 -- 172.21.15.67:0/1030351165 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee3c01d070 con 0x7fee4c0ffda0 2024-04-22T23:53:17.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.909+0000 7fee49ffb700 1 -- 172.21.15.67:0/1030351165 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fee3c00f460 con 0x7fee4c0ffda0 2024-04-22T23:53:17.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.909+0000 7fee529db700 1 -- 172.21.15.67:0/1030351165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee4c198c00 con 0x7fee4c0ffda0 2024-04-22T23:53:17.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.909+0000 7fee49ffb700 1 -- 172.21.15.67:0/1030351165 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee3c017630 con 0x7fee4c0ffda0 2024-04-22T23:53:17.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.909+0000 7fee529db700 1 -- 172.21.15.67:0/1030351165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee4c1990f0 con 0x7fee4c0ffda0 2024-04-22T23:53:17.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.910+0000 7fee529db700 1 -- 172.21.15.67:0/1030351165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee4c04f2e0 con 0x7fee4c0ffda0 2024-04-22T23:53:17.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.910+0000 7fee49ffb700 1 -- 172.21.15.67:0/1030351165 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fee3c00f5d0 con 0x7fee4c0ffda0 2024-04-22T23:53:17.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.911+0000 7fee49ffb700 1 --2- 172.21.15.67:0/1030351165 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fee3806ecc0 0x7fee38071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:17.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.911+0000 7fee49ffb700 1 -- 172.21.15.67:0/1030351165 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fee3c090400 con 0x7fee4c0ffda0 2024-04-22T23:53:17.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.911+0000 7fee43fff700 1 --2- 172.21.15.67:0/1030351165 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fee3806ecc0 0x7fee38071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:17.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.911+0000 7fee43fff700 1 --2- 172.21.15.67:0/1030351165 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fee3806ecc0 0x7fee38071180 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fee34005fd0 tx=0x7fee34005ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:53:17.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:17.916+0000 7fee49ffb700 1 -- 172.21.15.67:0/1030351165 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fee3c05cbf0 con 0x7fee4c0ffda0 2024-04-22T23:53:18.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.175+0000 7fee529db700 1 -- 172.21.15.67:0/1030351165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fee4c199310 con 0x7fee4c0ffda0 2024-04-22T23:53:18.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.176+0000 7fee49ffb700 1 -- 172.21.15.67:0/1030351165 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1872 (secure 0 0 0) 0x7fee3c05c780 con 0x7fee4c0ffda0 2024-04-22T23:53:18.167 INFO:teuthology.orchestra.run.smithi067.stdout:e33 2024-04-22T23:53:18.167 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:53:18.167 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:53:18.167 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:53:18.168 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:53:18.168 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:53:18.168 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:53:18.168 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:53:18.168 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:53:18.168 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:53:18.168 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:53:18.168 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:53:18.168 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:53:18.168 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:53:18.168 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:53:18.168 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:53:18.168 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:53:18.168 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:53:18.168 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:53:18.169 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:53:18.169 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:53:18.169 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:53:18.169 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:53:18.169 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:53:18.169 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:53:18.169 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:53:18.169 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:53:18.169 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:53:18.169 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:53:18.169 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:53:18.169 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:53:18.169 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:53:18.169 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:53:18.169 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:53:18.170 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:53:18.170 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:53:18.170 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:53:18.170 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:53:18.170 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:53:18.170 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:53:18.170 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:53:18.170 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44851} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1849648622,v1:172.21.15.134:6827/1849648622] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:53:18.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.178+0000 7fee529db700 1 -- 172.21.15.67:0/1030351165 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fee3806ecc0 msgr2=0x7fee38071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:18.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.178+0000 7fee529db700 1 --2- 172.21.15.67:0/1030351165 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fee3806ecc0 0x7fee38071180 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fee34005fd0 tx=0x7fee34005ee0 comp rx=0 tx=0).stop 2024-04-22T23:53:18.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.178+0000 7fee529db700 1 -- 172.21.15.67:0/1030351165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee4c0ffda0 msgr2=0x7fee4c19e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:18.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.178+0000 7fee529db700 1 --2- 172.21.15.67:0/1030351165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee4c0ffda0 0x7fee4c19e900 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fee3c006010 tx=0x7fee3c00be20 comp rx=0 tx=0).stop 2024-04-22T23:53:18.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.178+0000 7fee529db700 1 -- 172.21.15.67:0/1030351165 shutdown_connections 2024-04-22T23:53:18.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.178+0000 7fee529db700 1 --2- 172.21.15.67:0/1030351165 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fee3806ecc0 0x7fee38071180 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:18.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.178+0000 7fee529db700 1 --2- 172.21.15.67:0/1030351165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee4c0ffda0 0x7fee4c19e900 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:18.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.178+0000 7fee529db700 1 --2- 172.21.15.67:0/1030351165 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee4c100750 0x7fee4c19ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:18.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.178+0000 7fee529db700 1 -- 172.21.15.67:0/1030351165 >> 172.21.15.67:0/1030351165 conn(0x7fee4c0fb650 msgr2=0x7fee4c069700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:53:18.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.178+0000 7fee529db700 1 -- 172.21.15.67:0/1030351165 shutdown_connections 2024-04-22T23:53:18.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.178+0000 7fee529db700 1 -- 172.21.15.67:0/1030351165 wait complete. 2024-04-22T23:53:18.172 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 33 2024-04-22T23:53:18.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.286+0000 7fdfedd8a700 1 -- 172.21.15.67:0/1104957883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfe8106cd0 msgr2=0x7fdfe81070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:18.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.286+0000 7fdfedd8a700 1 --2- 172.21.15.67:0/1104957883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfe8106cd0 0x7fdfe81070b0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8009910 tx=0x7fdfd8009c20 comp rx=0 tx=0).stop 2024-04-22T23:53:18.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.286+0000 7fdfedd8a700 1 -- 172.21.15.67:0/1104957883 shutdown_connections 2024-04-22T23:53:18.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.286+0000 7fdfedd8a700 1 --2- 172.21.15.67:0/1104957883 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdfe8068580 0x7fdfe80689e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:18.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.286+0000 7fdfedd8a700 1 --2- 172.21.15.67:0/1104957883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfe8106cd0 0x7fdfe81070b0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:18.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.286+0000 7fdfedd8a700 1 -- 172.21.15.67:0/1104957883 >> 172.21.15.67:0/1104957883 conn(0x7fdfe8075670 msgr2=0x7fdfe8075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:53:18.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.286+0000 7fdfedd8a700 1 -- 172.21.15.67:0/1104957883 shutdown_connections 2024-04-22T23:53:18.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.286+0000 7fdfedd8a700 1 -- 172.21.15.67:0/1104957883 wait complete. 2024-04-22T23:53:18.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.287+0000 7fdfedd8a700 1 Processor -- start 2024-04-22T23:53:18.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.287+0000 7fdfedd8a700 1 -- start start 2024-04-22T23:53:18.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.287+0000 7fdfedd8a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfe8068580 0x7fdfe819e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:18.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.287+0000 7fdfedd8a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdfe8106cd0 0x7fdfe819ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:18.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.287+0000 7fdfedd8a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe819f410 con 0x7fdfe8068580 2024-04-22T23:53:18.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.287+0000 7fdfedd8a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe81988c0 con 0x7fdfe8106cd0 2024-04-22T23:53:18.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.287+0000 7fdfe77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfe8068580 0x7fdfe819e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:18.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.287+0000 7fdfe77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfe8068580 0x7fdfe819e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55388/0 (socket says 172.21.15.67:55388) 2024-04-22T23:53:18.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.287+0000 7fdfe77fe700 1 -- 172.21.15.67:0/1326462689 learned_addr learned my addr 172.21.15.67:0/1326462689 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:53:18.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.287+0000 7fdfe6ffd700 1 --2- 172.21.15.67:0/1326462689 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdfe8106cd0 0x7fdfe819ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:18.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.287+0000 7fdfe77fe700 1 -- 172.21.15.67:0/1326462689 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdfe8106cd0 msgr2=0x7fdfe819ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:18.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.288+0000 7fdfe77fe700 1 --2- 172.21.15.67:0/1326462689 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdfe8106cd0 0x7fdfe819ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:18.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.288+0000 7fdfe77fe700 1 -- 172.21.15.67:0/1326462689 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd80095f0 con 0x7fdfe8068580 2024-04-22T23:53:18.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.288+0000 7fdfe6ffd700 1 --2- 172.21.15.67:0/1326462689 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdfe8106cd0 0x7fdfe819ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:53:18.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.288+0000 7fdfe77fe700 1 --2- 172.21.15.67:0/1326462689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfe8068580 0x7fdfe819e840 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8006010 tx=0x7fdfd800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:53:18.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.288+0000 7fdfe4ff9700 1 -- 172.21.15.67:0/1326462689 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfd801d070 con 0x7fdfe8068580 2024-04-22T23:53:18.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.288+0000 7fdfe4ff9700 1 -- 172.21.15.67:0/1326462689 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdfd8004fb0 con 0x7fdfe8068580 2024-04-22T23:53:18.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.288+0000 7fdfedd8a700 1 -- 172.21.15.67:0/1326462689 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfe8198b40 con 0x7fdfe8068580 2024-04-22T23:53:18.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.288+0000 7fdfe4ff9700 1 -- 172.21.15.67:0/1326462689 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfd8022ea0 con 0x7fdfe8068580 2024-04-22T23:53:18.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.288+0000 7fdfedd8a700 1 -- 172.21.15.67:0/1326462689 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfe8199030 con 0x7fdfe8068580 2024-04-22T23:53:18.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.289+0000 7fdfedd8a700 1 -- 172.21.15.67:0/1326462689 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfe804f2e0 con 0x7fdfe8068580 2024-04-22T23:53:18.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.290+0000 7fdfe4ff9700 1 -- 172.21.15.67:0/1326462689 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdfd8022470 con 0x7fdfe8068580 2024-04-22T23:53:18.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.290+0000 7fdfe4ff9700 1 --2- 172.21.15.67:0/1326462689 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdfd406ecc0 0x7fdfd4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:18.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.290+0000 7fdfe4ff9700 1 -- 172.21.15.67:0/1326462689 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fdfd8090560 con 0x7fdfe8068580 2024-04-22T23:53:18.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.290+0000 7fdfe6ffd700 1 --2- 172.21.15.67:0/1326462689 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdfd406ecc0 0x7fdfd4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:18.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.291+0000 7fdfe6ffd700 1 --2- 172.21.15.67:0/1326462689 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdfd406ecc0 0x7fdfd4071180 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0005fd0 tx=0x7fdfd0005dc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:53:18.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.295+0000 7fdfe4ff9700 1 -- 172.21.15.67:0/1326462689 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fdfd805cd80 con 0x7fdfe8068580 2024-04-22T23:53:18.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:18 smithi134 ceph-mon[85630]: from='client.35468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:53:18.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:18 smithi134 ceph-mon[85630]: pgmap v809: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:18.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:18 smithi134 ceph-mon[85630]: from='client.35472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:53:18.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:18 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2437281599' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:53:18.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.523+0000 7fdfedd8a700 1 -- 172.21.15.67:0/1326462689 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdfe8199310 con 0x7fdfd406ecc0 2024-04-22T23:53:18.515 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:18 smithi067 ceph-mon[112438]: from='client.35468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:53:18.515 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:18 smithi067 ceph-mon[112438]: pgmap v809: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:18.515 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:18 smithi067 ceph-mon[112438]: from='client.35472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:53:18.515 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:18 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2437281599' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:53:18.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.525+0000 7fdfe4ff9700 1 -- 172.21.15.67:0/1326462689 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fdfe8199310 con 0x7fdfd406ecc0 2024-04-22T23:53:18.516 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:53:18.516 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:53:18.516 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:53:18.516 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:53:18.516 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:53:18.516 INFO:teuthology.orchestra.run.smithi067.stdout: "mds", 2024-04-22T23:53:18.516 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:53:18.516 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:53:18.516 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:53:18.516 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:53:18.517 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:53:18.517 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T23:53:18.517 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:53:18.517 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:53:18.517 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:53:18.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.527+0000 7fdfedd8a700 1 -- 172.21.15.67:0/1326462689 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdfd406ecc0 msgr2=0x7fdfd4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:18.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.527+0000 7fdfedd8a700 1 --2- 172.21.15.67:0/1326462689 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdfd406ecc0 0x7fdfd4071180 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0005fd0 tx=0x7fdfd0005dc0 comp rx=0 tx=0).stop 2024-04-22T23:53:18.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.527+0000 7fdfedd8a700 1 -- 172.21.15.67:0/1326462689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfe8068580 msgr2=0x7fdfe819e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:18.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.527+0000 7fdfedd8a700 1 --2- 172.21.15.67:0/1326462689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfe8068580 0x7fdfe819e840 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8006010 tx=0x7fdfd800be20 comp rx=0 tx=0).stop 2024-04-22T23:53:18.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.527+0000 7fdfedd8a700 1 -- 172.21.15.67:0/1326462689 shutdown_connections 2024-04-22T23:53:18.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.527+0000 7fdfedd8a700 1 --2- 172.21.15.67:0/1326462689 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdfd406ecc0 0x7fdfd4071180 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:18.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.527+0000 7fdfedd8a700 1 --2- 172.21.15.67:0/1326462689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfe8068580 0x7fdfe819e840 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:18.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.527+0000 7fdfedd8a700 1 --2- 172.21.15.67:0/1326462689 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdfe8106cd0 0x7fdfe819ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:18.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.527+0000 7fdfedd8a700 1 -- 172.21.15.67:0/1326462689 >> 172.21.15.67:0/1326462689 conn(0x7fdfe8075670 msgr2=0x7fdfe80fea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:53:18.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.527+0000 7fdfedd8a700 1 -- 172.21.15.67:0/1326462689 shutdown_connections 2024-04-22T23:53:18.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.527+0000 7fdfedd8a700 1 -- 172.21.15.67:0/1326462689 wait complete. 2024-04-22T23:53:18.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.633+0000 7efdd1877700 1 -- 172.21.15.67:0/3971237053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efdcc100ea0 msgr2=0x7efdcc101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:18.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.633+0000 7efdd1877700 1 --2- 172.21.15.67:0/3971237053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efdcc100ea0 0x7efdcc101280 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7efdbc009910 tx=0x7efdbc009c20 comp rx=0 tx=0).stop 2024-04-22T23:53:18.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.633+0000 7efdd1877700 1 -- 172.21.15.67:0/3971237053 shutdown_connections 2024-04-22T23:53:18.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.633+0000 7efdd1877700 1 --2- 172.21.15.67:0/3971237053 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdcc101850 0x7efdcc1058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:18.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.633+0000 7efdd1877700 1 --2- 172.21.15.67:0/3971237053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efdcc100ea0 0x7efdcc101280 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:18.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.633+0000 7efdd1877700 1 -- 172.21.15.67:0/3971237053 >> 172.21.15.67:0/3971237053 conn(0x7efdcc0fc750 msgr2=0x7efdcc0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:53:18.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.634+0000 7efdd1877700 1 -- 172.21.15.67:0/3971237053 shutdown_connections 2024-04-22T23:53:18.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.634+0000 7efdd1877700 1 -- 172.21.15.67:0/3971237053 wait complete. 2024-04-22T23:53:18.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.634+0000 7efdd1877700 1 Processor -- start 2024-04-22T23:53:18.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.634+0000 7efdd1877700 1 -- start start 2024-04-22T23:53:18.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.635+0000 7efdd1877700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efdcc100ea0 0x7efdcc19e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:18.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.635+0000 7efdd1877700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdcc101850 0x7efdcc19ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:18.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.635+0000 7efdd1877700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdcc19f4b0 con 0x7efdcc100ea0 2024-04-22T23:53:18.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.635+0000 7efdd1877700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdcc198960 con 0x7efdcc101850 2024-04-22T23:53:18.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.635+0000 7efdcaffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efdcc100ea0 0x7efdcc19e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:18.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.635+0000 7efdcaffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efdcc100ea0 0x7efdcc19e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55402/0 (socket says 172.21.15.67:55402) 2024-04-22T23:53:18.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.635+0000 7efdcaffd700 1 -- 172.21.15.67:0/1408066356 learned_addr learned my addr 172.21.15.67:0/1408066356 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:53:18.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.635+0000 7efdca7fc700 1 --2- 172.21.15.67:0/1408066356 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdcc101850 0x7efdcc19ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:18.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.635+0000 7efdcaffd700 1 -- 172.21.15.67:0/1408066356 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdcc101850 msgr2=0x7efdcc19ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:18.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.635+0000 7efdcaffd700 1 --2- 172.21.15.67:0/1408066356 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdcc101850 0x7efdcc19ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:18.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.635+0000 7efdcaffd700 1 -- 172.21.15.67:0/1408066356 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdbc0095f0 con 0x7efdcc100ea0 2024-04-22T23:53:18.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.635+0000 7efdca7fc700 1 --2- 172.21.15.67:0/1408066356 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdcc101850 0x7efdcc19ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:53:18.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.635+0000 7efdcaffd700 1 --2- 172.21.15.67:0/1408066356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efdcc100ea0 0x7efdcc19e8e0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7efdbc006010 tx=0x7efdbc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:53:18.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.635+0000 7efdd0875700 1 -- 172.21.15.67:0/1408066356 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efdbc01d070 con 0x7efdcc100ea0 2024-04-22T23:53:18.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.635+0000 7efdd0875700 1 -- 172.21.15.67:0/1408066356 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efdbc00f460 con 0x7efdcc100ea0 2024-04-22T23:53:18.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.635+0000 7efdd0875700 1 -- 172.21.15.67:0/1408066356 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efdbc017630 con 0x7efdcc100ea0 2024-04-22T23:53:18.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.635+0000 7efdd1877700 1 -- 172.21.15.67:0/1408066356 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdcc198be0 con 0x7efdcc100ea0 2024-04-22T23:53:18.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.636+0000 7efdd1877700 1 -- 172.21.15.67:0/1408066356 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdcc1990d0 con 0x7efdcc100ea0 2024-04-22T23:53:18.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.636+0000 7efdd1877700 1 -- 172.21.15.67:0/1408066356 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdcc105020 con 0x7efdcc100ea0 2024-04-22T23:53:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.637+0000 7efdd0875700 1 -- 172.21.15.67:0/1408066356 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7efdbc00f5d0 con 0x7efdcc100ea0 2024-04-22T23:53:18.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.637+0000 7efdd0875700 1 --2- 172.21.15.67:0/1408066356 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efdb406ecc0 0x7efdb4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:18.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.638+0000 7efdd0875700 1 -- 172.21.15.67:0/1408066356 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7efdbc090400 con 0x7efdcc100ea0 2024-04-22T23:53:18.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.638+0000 7efdca7fc700 1 --2- 172.21.15.67:0/1408066356 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efdb406ecc0 0x7efdb4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:18.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.638+0000 7efdca7fc700 1 --2- 172.21.15.67:0/1408066356 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efdb406ecc0 0x7efdb4071180 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7efdc0005950 tx=0x7efdc00058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:53:18.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.642+0000 7efdd0875700 1 -- 172.21.15.67:0/1408066356 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7efdbc05cbf0 con 0x7efdcc100ea0 2024-04-22T23:53:18.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.939+0000 7efdd1877700 1 -- 172.21.15.67:0/1408066356 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7efdcc04f2e0 con 0x7efdcc100ea0 2024-04-22T23:53:18.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.940+0000 7efdd0875700 1 -- 172.21.15.67:0/1408066356 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4505 (secure 0 0 0) 0x7efdbc05c780 con 0x7efdcc100ea0 2024-04-22T23:53:18.931 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:53:18.931 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T23:53:18.931 INFO:teuthology.orchestra.run.smithi067.stdout: host smithi134 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-22T23:53:18.931 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T23:53:18.932 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:53:18.932 INFO:teuthology.orchestra.run.smithi067.stdout:Traceback (most recent call last): 2024-04-22T23:53:18.932 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T23:53:18.932 INFO:teuthology.orchestra.run.smithi067.stdout: return _run_code(code, main_globals, None, 2024-04-22T23:53:18.932 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T23:53:18.932 INFO:teuthology.orchestra.run.smithi067.stdout: exec(code, run_globals) 2024-04-22T23:53:18.932 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5642, in 2024-04-22T23:53:18.932 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5630, in main 2024-04-22T23:53:18.932 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 408, in _infer_config 2024-04-22T23:53:18.932 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 323, in _infer_fsid 2024-04-22T23:53:18.932 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 436, in _infer_image 2024-04-22T23:53:18.933 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 310, in _validate_fsid 2024-04-22T23:53:18.933 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 3380, in command_ceph_volume 2024-04-22T23:53:18.933 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T23:53:18.933 INFO:teuthology.orchestra.run.smithi067.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:53:18.933 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:53:18.933 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:53:18.933 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:53:18.933 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:53:18.933 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:53:18.933 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:53:18.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.942+0000 7efdd1877700 1 -- 172.21.15.67:0/1408066356 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efdb406ecc0 msgr2=0x7efdb4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:18.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.942+0000 7efdd1877700 1 --2- 172.21.15.67:0/1408066356 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efdb406ecc0 0x7efdb4071180 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7efdc0005950 tx=0x7efdc00058e0 comp rx=0 tx=0).stop 2024-04-22T23:53:18.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.942+0000 7efdd1877700 1 -- 172.21.15.67:0/1408066356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efdcc100ea0 msgr2=0x7efdcc19e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:18.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.942+0000 7efdd1877700 1 --2- 172.21.15.67:0/1408066356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efdcc100ea0 0x7efdcc19e8e0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7efdbc006010 tx=0x7efdbc00be20 comp rx=0 tx=0).stop 2024-04-22T23:53:18.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.942+0000 7efdd1877700 1 -- 172.21.15.67:0/1408066356 shutdown_connections 2024-04-22T23:53:18.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.942+0000 7efdd1877700 1 --2- 172.21.15.67:0/1408066356 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efdb406ecc0 0x7efdb4071180 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:18.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.942+0000 7efdd1877700 1 --2- 172.21.15.67:0/1408066356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efdcc100ea0 0x7efdcc19e8e0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:18.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.942+0000 7efdd1877700 1 --2- 172.21.15.67:0/1408066356 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efdcc101850 0x7efdcc19ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:18.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.942+0000 7efdd1877700 1 -- 172.21.15.67:0/1408066356 >> 172.21.15.67:0/1408066356 conn(0x7efdcc0fc750 msgr2=0x7efdcc0fddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:53:18.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.942+0000 7efdd1877700 1 -- 172.21.15.67:0/1408066356 shutdown_connections 2024-04-22T23:53:18.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:18.942+0000 7efdd1877700 1 -- 172.21.15.67:0/1408066356 wait complete. 2024-04-22T23:53:19.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:19 smithi067 ceph-mon[112438]: from='client.35476 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:53:19.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:19 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1030351165' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:53:19.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:19 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1408066356' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:53:19.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:19 smithi134 ceph-mon[85630]: from='client.35476 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:53:19.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:19 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1030351165' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:53:19.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:19 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1408066356' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:53:20.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:20 smithi134 ceph-mon[85630]: from='client.35488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:53:20.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:20 smithi134 ceph-mon[85630]: pgmap v810: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:20.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:20 smithi067 ceph-mon[112438]: from='client.35488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:53:20.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:20 smithi067 ceph-mon[112438]: pgmap v810: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:22.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:22 smithi134 ceph-mon[85630]: pgmap v811: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:22.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:22 smithi067 ceph-mon[112438]: pgmap v811: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:23.984 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:23 smithi134 ceph-mon[85630]: pgmap v812: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:24.035 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:23 smithi067 ceph-mon[112438]: pgmap v812: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:26.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:26 smithi134 ceph-mon[85630]: pgmap v813: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:26.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:53:26.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:26 smithi067 ceph-mon[112438]: pgmap v813: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:26.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:53:28.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:28 smithi134 ceph-mon[85630]: pgmap v814: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:28.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:28 smithi067 ceph-mon[112438]: pgmap v814: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:30.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:30 smithi134 ceph-mon[85630]: pgmap v815: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:30.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:30 smithi067 ceph-mon[112438]: pgmap v815: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:32.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:32 smithi134 ceph-mon[85630]: pgmap v816: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:32.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:32 smithi067 ceph-mon[112438]: pgmap v816: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:34.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:34 smithi134 ceph-mon[85630]: pgmap v817: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:34.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:34 smithi067 ceph-mon[112438]: pgmap v817: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:36.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:36 smithi067 ceph-mon[112438]: pgmap v818: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:36.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:36 smithi134 ceph-mon[85630]: pgmap v818: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:38.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:38 smithi134 ceph-mon[85630]: pgmap v819: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:38.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:38 smithi067 ceph-mon[112438]: pgmap v819: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:39.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:39 smithi134 ceph-mon[85630]: pgmap v820: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:39.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:39 smithi067 ceph-mon[112438]: pgmap v820: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:41.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:41 smithi067 ceph-mon[112438]: pgmap v821: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:41.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:53:41.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:41 smithi134 ceph-mon[85630]: pgmap v821: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:41.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:53:44.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:44 smithi134 ceph-mon[85630]: pgmap v822: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:44.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:44 smithi067 ceph-mon[112438]: pgmap v822: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:46.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:46 smithi067 ceph-mon[112438]: pgmap v823: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:46.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:46 smithi134 ceph-mon[85630]: pgmap v823: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:48.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:48 smithi134 ceph-mon[85630]: pgmap v824: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:48.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:48 smithi067 ceph-mon[112438]: pgmap v824: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:49.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.063+0000 7f8a721b9700 1 -- 172.21.15.67:0/2024542356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a6c100750 msgr2=0x7f8a6c1047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:49.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.063+0000 7f8a721b9700 1 --2- 172.21.15.67:0/2024542356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a6c100750 0x7f8a6c1047a0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f8a60009910 tx=0x7f8a60009c20 comp rx=0 tx=0).stop 2024-04-22T23:53:49.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.063+0000 7f8a721b9700 1 -- 172.21.15.67:0/2024542356 shutdown_connections 2024-04-22T23:53:49.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.063+0000 7f8a721b9700 1 --2- 172.21.15.67:0/2024542356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a6c100750 0x7f8a6c1047a0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:49.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.063+0000 7f8a721b9700 1 --2- 172.21.15.67:0/2024542356 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a6c0ffda0 0x7f8a6c100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:49.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.063+0000 7f8a721b9700 1 -- 172.21.15.67:0/2024542356 >> 172.21.15.67:0/2024542356 conn(0x7f8a6c0fb650 msgr2=0x7f8a6c0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:53:49.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.063+0000 7f8a721b9700 1 -- 172.21.15.67:0/2024542356 shutdown_connections 2024-04-22T23:53:49.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.063+0000 7f8a721b9700 1 -- 172.21.15.67:0/2024542356 wait complete. 2024-04-22T23:53:49.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.064+0000 7f8a721b9700 1 Processor -- start 2024-04-22T23:53:49.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.064+0000 7f8a721b9700 1 -- start start 2024-04-22T23:53:49.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.064+0000 7f8a721b9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a6c0ffda0 0x7f8a6c1977a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:49.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.064+0000 7f8a721b9700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a6c100750 0x7f8a6c197ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:49.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.064+0000 7f8a721b9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a6c198300 con 0x7f8a6c0ffda0 2024-04-22T23:53:49.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.064+0000 7f8a721b9700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a6c198470 con 0x7f8a6c100750 2024-04-22T23:53:49.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.065+0000 7f8a6b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a6c0ffda0 0x7f8a6c1977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:49.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.065+0000 7f8a6b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a6c0ffda0 0x7f8a6c1977a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43450/0 (socket says 172.21.15.67:43450) 2024-04-22T23:53:49.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.065+0000 7f8a6b7fe700 1 -- 172.21.15.67:0/3536515676 learned_addr learned my addr 172.21.15.67:0/3536515676 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:53:49.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.065+0000 7f8a6affd700 1 --2- 172.21.15.67:0/3536515676 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a6c100750 0x7f8a6c197ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:49.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.065+0000 7f8a6b7fe700 1 -- 172.21.15.67:0/3536515676 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a6c100750 msgr2=0x7f8a6c197ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:49.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.065+0000 7f8a6b7fe700 1 --2- 172.21.15.67:0/3536515676 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a6c100750 0x7f8a6c197ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:49.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.065+0000 7f8a6b7fe700 1 -- 172.21.15.67:0/3536515676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a600095f0 con 0x7f8a6c0ffda0 2024-04-22T23:53:49.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.065+0000 7f8a6b7fe700 1 --2- 172.21.15.67:0/3536515676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a6c0ffda0 0x7f8a6c1977a0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c00d8d0 tx=0x7f8a5c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:53:49.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.065+0000 7f8a68ff9700 1 -- 172.21.15.67:0/3536515676 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a5c0098a0 con 0x7f8a6c0ffda0 2024-04-22T23:53:49.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.065+0000 7f8a68ff9700 1 -- 172.21.15.67:0/3536515676 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8a5c010460 con 0x7f8a6c0ffda0 2024-04-22T23:53:49.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.065+0000 7f8a721b9700 1 -- 172.21.15.67:0/3536515676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a6c1a0810 con 0x7f8a6c0ffda0 2024-04-22T23:53:49.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.065+0000 7f8a68ff9700 1 -- 172.21.15.67:0/3536515676 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a5c00f5f0 con 0x7f8a6c0ffda0 2024-04-22T23:53:49.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.065+0000 7f8a721b9700 1 -- 172.21.15.67:0/3536515676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a6c1a0cd0 con 0x7f8a6c0ffda0 2024-04-22T23:53:49.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.066+0000 7f8a721b9700 1 -- 172.21.15.67:0/3536515676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a6c1081a0 con 0x7f8a6c0ffda0 2024-04-22T23:53:49.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.067+0000 7f8a68ff9700 1 -- 172.21.15.67:0/3536515676 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8a5c009a00 con 0x7f8a6c0ffda0 2024-04-22T23:53:49.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.067+0000 7f8a68ff9700 1 --2- 172.21.15.67:0/3536515676 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8a5406ecc0 0x7f8a54071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:49.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.068+0000 7f8a6affd700 1 --2- 172.21.15.67:0/3536515676 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8a5406ecc0 0x7f8a54071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:49.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.068+0000 7f8a68ff9700 1 -- 172.21.15.67:0/3536515676 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f8a5c08ee00 con 0x7f8a6c0ffda0 2024-04-22T23:53:49.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.068+0000 7f8a6affd700 1 --2- 172.21.15.67:0/3536515676 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8a5406ecc0 0x7f8a54071180 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f8a60006010 tx=0x7f8a6001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:53:49.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.072+0000 7f8a68ff9700 1 -- 172.21.15.67:0/3536515676 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f8a5c057a60 con 0x7f8a6c0ffda0 2024-04-22T23:53:49.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.307+0000 7f8a721b9700 1 -- 172.21.15.67:0/3536515676 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8a6c0611d0 con 0x7f8a5406ecc0 2024-04-22T23:53:49.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.308+0000 7f8a68ff9700 1 -- 172.21.15.67:0/3536515676 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8a6c0611d0 con 0x7f8a5406ecc0 2024-04-22T23:53:49.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.310+0000 7f8a721b9700 1 -- 172.21.15.67:0/3536515676 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8a5406ecc0 msgr2=0x7f8a54071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:49.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.310+0000 7f8a721b9700 1 --2- 172.21.15.67:0/3536515676 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8a5406ecc0 0x7f8a54071180 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f8a60006010 tx=0x7f8a6001a040 comp rx=0 tx=0).stop 2024-04-22T23:53:49.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.310+0000 7f8a721b9700 1 -- 172.21.15.67:0/3536515676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a6c0ffda0 msgr2=0x7f8a6c1977a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:49.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.310+0000 7f8a721b9700 1 --2- 172.21.15.67:0/3536515676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a6c0ffda0 0x7f8a6c1977a0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c00d8d0 tx=0x7f8a5c00dbe0 comp rx=0 tx=0).stop 2024-04-22T23:53:49.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.311+0000 7f8a721b9700 1 -- 172.21.15.67:0/3536515676 shutdown_connections 2024-04-22T23:53:49.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.311+0000 7f8a721b9700 1 --2- 172.21.15.67:0/3536515676 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f8a5406ecc0 0x7f8a54071180 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:49.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.311+0000 7f8a721b9700 1 --2- 172.21.15.67:0/3536515676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a6c0ffda0 0x7f8a6c1977a0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:49.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.311+0000 7f8a721b9700 1 --2- 172.21.15.67:0/3536515676 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a6c100750 0x7f8a6c197ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:49.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.311+0000 7f8a721b9700 1 -- 172.21.15.67:0/3536515676 >> 172.21.15.67:0/3536515676 conn(0x7f8a6c0fb650 msgr2=0x7f8a6c0fccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:53:49.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.311+0000 7f8a721b9700 1 -- 172.21.15.67:0/3536515676 shutdown_connections 2024-04-22T23:53:49.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.311+0000 7f8a721b9700 1 -- 172.21.15.67:0/3536515676 wait complete. 2024-04-22T23:53:49.313 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:53:49.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.419+0000 7f9d19360700 1 -- 172.21.15.67:0/1206243622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d141027c0 msgr2=0x7f9d1410acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:49.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.419+0000 7f9d19360700 1 --2- 172.21.15.67:0/1206243622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d141027c0 0x7f9d1410acb0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc009910 tx=0x7f9cfc009c20 comp rx=0 tx=0).stop 2024-04-22T23:53:49.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.419+0000 7f9d19360700 1 -- 172.21.15.67:0/1206243622 shutdown_connections 2024-04-22T23:53:49.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.419+0000 7f9d19360700 1 --2- 172.21.15.67:0/1206243622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d141027c0 0x7f9d1410acb0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:49.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.419+0000 7f9d19360700 1 --2- 172.21.15.67:0/1206243622 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9d14101ea0 0x7f9d14102280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:49.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.419+0000 7f9d19360700 1 -- 172.21.15.67:0/1206243622 >> 172.21.15.67:0/1206243622 conn(0x7f9d140fb650 msgr2=0x7f9d140fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:53:49.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.419+0000 7f9d19360700 1 -- 172.21.15.67:0/1206243622 shutdown_connections 2024-04-22T23:53:49.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.419+0000 7f9d19360700 1 -- 172.21.15.67:0/1206243622 wait complete. 2024-04-22T23:53:49.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.420+0000 7f9d19360700 1 Processor -- start 2024-04-22T23:53:49.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.420+0000 7f9d19360700 1 -- start start 2024-04-22T23:53:49.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.420+0000 7f9d19360700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d14101ea0 0x7f9d140ffb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:49.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.420+0000 7f9d19360700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9d141027c0 0x7f9d14100040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:49.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.420+0000 7f9d19360700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d14100580 con 0x7f9d14101ea0 2024-04-22T23:53:49.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.420+0000 7f9d19360700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d141006c0 con 0x7f9d141027c0 2024-04-22T23:53:49.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.420+0000 7f9d12ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d14101ea0 0x7f9d140ffb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:49.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.420+0000 7f9d12ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d14101ea0 0x7f9d140ffb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43480/0 (socket says 172.21.15.67:43480) 2024-04-22T23:53:49.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.420+0000 7f9d12ffd700 1 -- 172.21.15.67:0/243485857 learned_addr learned my addr 172.21.15.67:0/243485857 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:53:49.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.421+0000 7f9d12ffd700 1 -- 172.21.15.67:0/243485857 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9d141027c0 msgr2=0x7f9d14100040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:49.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.421+0000 7f9d12ffd700 1 --2- 172.21.15.67:0/243485857 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9d141027c0 0x7f9d14100040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:49.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.421+0000 7f9d12ffd700 1 -- 172.21.15.67:0/243485857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cfc0095f0 con 0x7f9d14101ea0 2024-04-22T23:53:49.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.421+0000 7f9d12ffd700 1 --2- 172.21.15.67:0/243485857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d14101ea0 0x7f9d140ffb00 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f9d0400d8d0 tx=0x7f9d0400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:53:49.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.421+0000 7f9d0bfff700 1 -- 172.21.15.67:0/243485857 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d04009940 con 0x7f9d14101ea0 2024-04-22T23:53:49.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.421+0000 7f9d0bfff700 1 -- 172.21.15.67:0/243485857 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9d04010460 con 0x7f9d14101ea0 2024-04-22T23:53:49.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.421+0000 7f9d19360700 1 -- 172.21.15.67:0/243485857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d141a2a60 con 0x7f9d14101ea0 2024-04-22T23:53:49.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.421+0000 7f9d0bfff700 1 -- 172.21.15.67:0/243485857 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d0400f5f0 con 0x7f9d14101ea0 2024-04-22T23:53:49.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.421+0000 7f9d19360700 1 -- 172.21.15.67:0/243485857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d141a2e70 con 0x7f9d14101ea0 2024-04-22T23:53:49.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.422+0000 7f9d19360700 1 -- 172.21.15.67:0/243485857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d14108340 con 0x7f9d14101ea0 2024-04-22T23:53:49.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.423+0000 7f9d0bfff700 1 -- 172.21.15.67:0/243485857 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9d040105d0 con 0x7f9d14101ea0 2024-04-22T23:53:49.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.423+0000 7f9d0bfff700 1 --2- 172.21.15.67:0/243485857 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9d0006ecc0 0x7f9d00071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:49.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.423+0000 7f9d0bfff700 1 -- 172.21.15.67:0/243485857 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f9d0408eff0 con 0x7f9d14101ea0 2024-04-22T23:53:49.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.423+0000 7f9d127fc700 1 --2- 172.21.15.67:0/243485857 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9d0006ecc0 0x7f9d00071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:49.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.424+0000 7f9d127fc700 1 --2- 172.21.15.67:0/243485857 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9d0006ecc0 0x7f9d00071180 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc00b5c0 tx=0x7f9cfc005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:53:49.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.428+0000 7f9d0bfff700 1 -- 172.21.15.67:0/243485857 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f9d04057c30 con 0x7f9d14101ea0 2024-04-22T23:53:49.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.656+0000 7f9d19360700 1 -- 172.21.15.67:0/243485857 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d14100880 con 0x7f9d0006ecc0 2024-04-22T23:53:49.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.658+0000 7f9d0bfff700 1 -- 172.21.15.67:0/243485857 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9d14100880 con 0x7f9d0006ecc0 2024-04-22T23:53:49.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.660+0000 7f9d19360700 1 -- 172.21.15.67:0/243485857 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9d0006ecc0 msgr2=0x7f9d00071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:49.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.660+0000 7f9d19360700 1 --2- 172.21.15.67:0/243485857 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9d0006ecc0 0x7f9d00071180 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc00b5c0 tx=0x7f9cfc005960 comp rx=0 tx=0).stop 2024-04-22T23:53:49.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.660+0000 7f9d19360700 1 -- 172.21.15.67:0/243485857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d14101ea0 msgr2=0x7f9d140ffb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:49.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.660+0000 7f9d19360700 1 --2- 172.21.15.67:0/243485857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d14101ea0 0x7f9d140ffb00 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f9d0400d8d0 tx=0x7f9d0400dc90 comp rx=0 tx=0).stop 2024-04-22T23:53:49.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.660+0000 7f9d19360700 1 -- 172.21.15.67:0/243485857 shutdown_connections 2024-04-22T23:53:49.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.660+0000 7f9d19360700 1 --2- 172.21.15.67:0/243485857 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9d0006ecc0 0x7f9d00071180 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:49.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.660+0000 7f9d19360700 1 --2- 172.21.15.67:0/243485857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d14101ea0 0x7f9d140ffb00 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:49.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.660+0000 7f9d19360700 1 --2- 172.21.15.67:0/243485857 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9d141027c0 0x7f9d14100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:49.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.660+0000 7f9d19360700 1 -- 172.21.15.67:0/243485857 >> 172.21.15.67:0/243485857 conn(0x7f9d140fb650 msgr2=0x7f9d14105480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:53:49.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.660+0000 7f9d19360700 1 -- 172.21.15.67:0/243485857 shutdown_connections 2024-04-22T23:53:49.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.660+0000 7f9d19360700 1 -- 172.21.15.67:0/243485857 wait complete. 2024-04-22T23:53:49.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.767+0000 7f80c7ca4700 1 -- 172.21.15.67:0/2745258488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80c0068580 msgr2=0x7f80c00689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:49.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.767+0000 7f80c7ca4700 1 --2- 172.21.15.67:0/2745258488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80c0068580 0x7f80c00689e0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f80bc009910 tx=0x7f80bc009c20 comp rx=0 tx=0).stop 2024-04-22T23:53:49.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.767+0000 7f80c7ca4700 1 -- 172.21.15.67:0/2745258488 shutdown_connections 2024-04-22T23:53:49.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.767+0000 7f80c7ca4700 1 --2- 172.21.15.67:0/2745258488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80c0068580 0x7f80c00689e0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:49.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.767+0000 7f80c7ca4700 1 --2- 172.21.15.67:0/2745258488 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f80c0106cd0 0x7f80c01070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:49.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.767+0000 7f80c7ca4700 1 -- 172.21.15.67:0/2745258488 >> 172.21.15.67:0/2745258488 conn(0x7f80c0075670 msgr2=0x7f80c0075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:53:49.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.767+0000 7f80c7ca4700 1 -- 172.21.15.67:0/2745258488 shutdown_connections 2024-04-22T23:53:49.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.767+0000 7f80c7ca4700 1 -- 172.21.15.67:0/2745258488 wait complete. 2024-04-22T23:53:49.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.768+0000 7f80c7ca4700 1 Processor -- start 2024-04-22T23:53:49.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.768+0000 7f80c7ca4700 1 -- start start 2024-04-22T23:53:49.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.768+0000 7f80c7ca4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80c0068580 0x7f80c019e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:49.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.768+0000 7f80c7ca4700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f80c0106cd0 0x7f80c019ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:49.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.768+0000 7f80c7ca4700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c019f400 con 0x7f80c0068580 2024-04-22T23:53:49.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.768+0000 7f80c7ca4700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c01988b0 con 0x7f80c0106cd0 2024-04-22T23:53:49.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.768+0000 7f80c5a40700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80c0068580 0x7f80c019e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:49.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.768+0000 7f80c5a40700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80c0068580 0x7f80c019e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43488/0 (socket says 172.21.15.67:43488) 2024-04-22T23:53:49.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.768+0000 7f80c5a40700 1 -- 172.21.15.67:0/1766689951 learned_addr learned my addr 172.21.15.67:0/1766689951 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:53:49.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.768+0000 7f80c5a40700 1 -- 172.21.15.67:0/1766689951 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f80c0106cd0 msgr2=0x7f80c019ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:49.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.768+0000 7f80c523f700 1 --2- 172.21.15.67:0/1766689951 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f80c0106cd0 0x7f80c019ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:49.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.768+0000 7f80c5a40700 1 --2- 172.21.15.67:0/1766689951 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f80c0106cd0 0x7f80c019ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:49.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.768+0000 7f80c5a40700 1 -- 172.21.15.67:0/1766689951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80bc0095f0 con 0x7f80c0068580 2024-04-22T23:53:49.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.769+0000 7f80c5a40700 1 --2- 172.21.15.67:0/1766689951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80c0068580 0x7f80c019e830 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f80b400d8d0 tx=0x7f80b400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:53:49.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.769+0000 7f80b2ffd700 1 -- 172.21.15.67:0/1766689951 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f80b4009940 con 0x7f80c0068580 2024-04-22T23:53:49.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.769+0000 7f80b2ffd700 1 -- 172.21.15.67:0/1766689951 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f80b4010460 con 0x7f80c0068580 2024-04-22T23:53:49.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.769+0000 7f80c7ca4700 1 -- 172.21.15.67:0/1766689951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80c0198b90 con 0x7f80c0068580 2024-04-22T23:53:49.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.769+0000 7f80c7ca4700 1 -- 172.21.15.67:0/1766689951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80c01990e0 con 0x7f80c0068580 2024-04-22T23:53:49.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.769+0000 7f80b2ffd700 1 -- 172.21.15.67:0/1766689951 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f80b400f5f0 con 0x7f80c0068580 2024-04-22T23:53:49.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.770+0000 7f80c7ca4700 1 -- 172.21.15.67:0/1766689951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80c004f2e0 con 0x7f80c0068580 2024-04-22T23:53:49.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.770+0000 7f80b2ffd700 1 -- 172.21.15.67:0/1766689951 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f80b40105d0 con 0x7f80c0068580 2024-04-22T23:53:49.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.771+0000 7f80b2ffd700 1 --2- 172.21.15.67:0/1766689951 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f80ac06ecc0 0x7f80ac071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:49.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.771+0000 7f80b2ffd700 1 -- 172.21.15.67:0/1766689951 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f80b408eff0 con 0x7f80c0068580 2024-04-22T23:53:49.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.771+0000 7f80c523f700 1 --2- 172.21.15.67:0/1766689951 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f80ac06ecc0 0x7f80ac071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:49.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.771+0000 7f80c523f700 1 --2- 172.21.15.67:0/1766689951 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f80ac06ecc0 0x7f80ac071180 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f80bc00b5c0 tx=0x7f80bc005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:53:49.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.776+0000 7f80b2ffd700 1 -- 172.21.15.67:0/1766689951 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f80b405b760 con 0x7f80c0068580 2024-04-22T23:53:49.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:49.993+0000 7f80c7ca4700 1 -- 172.21.15.67:0/1766689951 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f80c0199410 con 0x7f80ac06ecc0 2024-04-22T23:53:49.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.005+0000 7f80b2ffd700 1 -- 172.21.15.67:0/1766689951 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f80c0199410 con 0x7f80ac06ecc0 2024-04-22T23:53:49.998 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:53:49.998 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (27m) 5m ago 38m 24.4M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:53:49.998 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (39m) 5m ago 39m 10.0M - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:53:49.998 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (38m) 5m ago 38m 10.0M - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:53:49.998 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (24m) 5m ago 38m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:53:49.998 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (24m) 5m ago 37m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:53:49.998 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (27m) 5m ago 38m 81.2M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:53:49.998 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (21m) 5m ago 33m 19.4M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:53:49.998 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (20m) 5m ago 33m 20.3M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:53:49.998 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (20m) 5m ago 33m 20.1M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:53:49.998 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (5m) 5m ago 33m 14.0M - 19.0.0-3252-g43be0201 a0434699c1d7 46d3d7e86fee 2024-04-22T23:53:49.999 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (29m) 5m ago 40m 514M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:53:49.999 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (28m) 5m ago 37m 459M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:53:49.999 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (26m) 5m ago 40m 74.2M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:53:49.999 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (25m) 5m ago 37m 54.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:53:49.999 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (28m) 5m ago 38m 15.2M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:53:49.999 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (28m) 5m ago 37m 16.0M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:53:49.999 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (24m) 5m ago 37m 102M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:53:49.999 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (23m) 5m ago 36m 128M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:53:49.999 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (23m) 5m ago 36m 83.8M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:53:49.999 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (22m) 5m ago 35m 101M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:53:49.999 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (22m) 5m ago 35m 86.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:53:49.999 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (21m) 5m ago 34m 81.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:53:49.999 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (27m) 5m ago 38m 78.7M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:53:50.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.008+0000 7f80c7ca4700 1 -- 172.21.15.67:0/1766689951 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f80ac06ecc0 msgr2=0x7f80ac071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:50.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.008+0000 7f80c7ca4700 1 --2- 172.21.15.67:0/1766689951 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f80ac06ecc0 0x7f80ac071180 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f80bc00b5c0 tx=0x7f80bc005960 comp rx=0 tx=0).stop 2024-04-22T23:53:50.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.008+0000 7f80c7ca4700 1 -- 172.21.15.67:0/1766689951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80c0068580 msgr2=0x7f80c019e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:50.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.008+0000 7f80c7ca4700 1 --2- 172.21.15.67:0/1766689951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80c0068580 0x7f80c019e830 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f80b400d8d0 tx=0x7f80b400dc90 comp rx=0 tx=0).stop 2024-04-22T23:53:50.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.008+0000 7f80c7ca4700 1 -- 172.21.15.67:0/1766689951 shutdown_connections 2024-04-22T23:53:50.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.008+0000 7f80c7ca4700 1 --2- 172.21.15.67:0/1766689951 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f80ac06ecc0 0x7f80ac071180 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:50.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.008+0000 7f80c7ca4700 1 --2- 172.21.15.67:0/1766689951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80c0068580 0x7f80c019e830 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:50.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.008+0000 7f80c7ca4700 1 --2- 172.21.15.67:0/1766689951 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f80c0106cd0 0x7f80c019ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:50.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.008+0000 7f80c7ca4700 1 -- 172.21.15.67:0/1766689951 >> 172.21.15.67:0/1766689951 conn(0x7f80c0075670 msgr2=0x7f80c00fe9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:53:50.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.008+0000 7f80c7ca4700 1 -- 172.21.15.67:0/1766689951 shutdown_connections 2024-04-22T23:53:50.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.008+0000 7f80c7ca4700 1 -- 172.21.15.67:0/1766689951 wait complete. 2024-04-22T23:53:50.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.117+0000 7f5c76350700 1 -- 172.21.15.67:0/1224156544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c700ffda0 msgr2=0x7f5c70100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:50.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.117+0000 7f5c76350700 1 --2- 172.21.15.67:0/1224156544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c700ffda0 0x7f5c70100180 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f5c60009910 tx=0x7f5c60009c20 comp rx=0 tx=0).stop 2024-04-22T23:53:50.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.117+0000 7f5c76350700 1 -- 172.21.15.67:0/1224156544 shutdown_connections 2024-04-22T23:53:50.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.117+0000 7f5c76350700 1 --2- 172.21.15.67:0/1224156544 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c70100750 0x7f5c701047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:50.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.117+0000 7f5c76350700 1 --2- 172.21.15.67:0/1224156544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c700ffda0 0x7f5c70100180 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:50.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.117+0000 7f5c76350700 1 -- 172.21.15.67:0/1224156544 >> 172.21.15.67:0/1224156544 conn(0x7f5c700fb650 msgr2=0x7f5c700fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:53:50.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.117+0000 7f5c76350700 1 -- 172.21.15.67:0/1224156544 shutdown_connections 2024-04-22T23:53:50.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.117+0000 7f5c76350700 1 -- 172.21.15.67:0/1224156544 wait complete. 2024-04-22T23:53:50.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.118+0000 7f5c76350700 1 Processor -- start 2024-04-22T23:53:50.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.118+0000 7f5c76350700 1 -- start start 2024-04-22T23:53:50.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.118+0000 7f5c76350700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c700ffda0 0x7f5c7019c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:50.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.118+0000 7f5c76350700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c70100750 0x7f5c7019cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:50.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.118+0000 7f5c76350700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7019d2b0 con 0x7f5c700ffda0 2024-04-22T23:53:50.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.118+0000 7f5c76350700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c70196760 con 0x7f5c70100750 2024-04-22T23:53:50.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.118+0000 7f5c6ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c700ffda0 0x7f5c7019c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:50.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.118+0000 7f5c6ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c700ffda0 0x7f5c7019c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43498/0 (socket says 172.21.15.67:43498) 2024-04-22T23:53:50.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.118+0000 7f5c6ffff700 1 -- 172.21.15.67:0/3295570991 learned_addr learned my addr 172.21.15.67:0/3295570991 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:53:50.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.118+0000 7f5c6f7fe700 1 --2- 172.21.15.67:0/3295570991 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c70100750 0x7f5c7019cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:50.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.118+0000 7f5c6ffff700 1 -- 172.21.15.67:0/3295570991 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c70100750 msgr2=0x7f5c7019cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:50.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.118+0000 7f5c6ffff700 1 --2- 172.21.15.67:0/3295570991 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c70100750 0x7f5c7019cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:50.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.118+0000 7f5c6ffff700 1 -- 172.21.15.67:0/3295570991 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c600095f0 con 0x7f5c700ffda0 2024-04-22T23:53:50.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.119+0000 7f5c6ffff700 1 --2- 172.21.15.67:0/3295570991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c700ffda0 0x7f5c7019c6e0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f5c60006010 tx=0x7f5c6000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:53:50.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.119+0000 7f5c6d7fa700 1 -- 172.21.15.67:0/3295570991 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c6001d070 con 0x7f5c700ffda0 2024-04-22T23:53:50.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.119+0000 7f5c6d7fa700 1 -- 172.21.15.67:0/3295570991 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5c6000f460 con 0x7f5c700ffda0 2024-04-22T23:53:50.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.119+0000 7f5c76350700 1 -- 172.21.15.67:0/3295570991 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c701969e0 con 0x7f5c700ffda0 2024-04-22T23:53:50.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.119+0000 7f5c6d7fa700 1 -- 172.21.15.67:0/3295570991 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c60017630 con 0x7f5c700ffda0 2024-04-22T23:53:50.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.119+0000 7f5c76350700 1 -- 172.21.15.67:0/3295570991 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c70196ed0 con 0x7f5c700ffda0 2024-04-22T23:53:50.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.120+0000 7f5c76350700 1 -- 172.21.15.67:0/3295570991 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c7004f2e0 con 0x7f5c700ffda0 2024-04-22T23:53:50.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.120+0000 7f5c6d7fa700 1 -- 172.21.15.67:0/3295570991 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5c6000f5d0 con 0x7f5c700ffda0 2024-04-22T23:53:50.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.121+0000 7f5c6d7fa700 1 --2- 172.21.15.67:0/3295570991 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5c5c06ecc0 0x7f5c5c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:50.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.121+0000 7f5c6d7fa700 1 -- 172.21.15.67:0/3295570991 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f5c60090470 con 0x7f5c700ffda0 2024-04-22T23:53:50.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.121+0000 7f5c6f7fe700 1 --2- 172.21.15.67:0/3295570991 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5c5c06ecc0 0x7f5c5c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:50.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.121+0000 7f5c6f7fe700 1 --2- 172.21.15.67:0/3295570991 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5c5c06ecc0 0x7f5c5c071180 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5c58005950 tx=0x7f5c580058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:53:50.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.126+0000 7f5c6d7fa700 1 -- 172.21.15.67:0/3295570991 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f5c6005cbe0 con 0x7f5c700ffda0 2024-04-22T23:53:50.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:50 smithi067 ceph-mon[112438]: pgmap v825: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:50.334 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:50 smithi134 ceph-mon[85630]: pgmap v825: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:50.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.436+0000 7f5c76350700 1 -- 172.21.15.67:0/3295570991 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5c701970f0 con 0x7f5c700ffda0 2024-04-22T23:53:50.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.436+0000 7f5c6d7fa700 1 -- 172.21.15.67:0/3295570991 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f5c6005c770 con 0x7f5c700ffda0 2024-04-22T23:53:50.428 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:53:50.429 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:53:50.429 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:53:50.429 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:53:50.429 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:53:50.429 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:53:50.429 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:53:50.429 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:53:50.429 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:53:50.429 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:53:50.429 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:53:50.429 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 4 2024-04-22T23:53:50.429 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:53:50.429 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:53:50.429 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 14 2024-04-22T23:53:50.429 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:53:50.430 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:53:50.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.438+0000 7f5c76350700 1 -- 172.21.15.67:0/3295570991 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5c5c06ecc0 msgr2=0x7f5c5c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:50.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.438+0000 7f5c76350700 1 --2- 172.21.15.67:0/3295570991 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5c5c06ecc0 0x7f5c5c071180 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5c58005950 tx=0x7f5c580058e0 comp rx=0 tx=0).stop 2024-04-22T23:53:50.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.438+0000 7f5c76350700 1 -- 172.21.15.67:0/3295570991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c700ffda0 msgr2=0x7f5c7019c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:50.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.438+0000 7f5c76350700 1 --2- 172.21.15.67:0/3295570991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c700ffda0 0x7f5c7019c6e0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f5c60006010 tx=0x7f5c6000be20 comp rx=0 tx=0).stop 2024-04-22T23:53:50.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.439+0000 7f5c76350700 1 -- 172.21.15.67:0/3295570991 shutdown_connections 2024-04-22T23:53:50.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.439+0000 7f5c76350700 1 --2- 172.21.15.67:0/3295570991 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5c5c06ecc0 0x7f5c5c071180 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:50.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.439+0000 7f5c76350700 1 --2- 172.21.15.67:0/3295570991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c700ffda0 0x7f5c7019c6e0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:50.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.439+0000 7f5c76350700 1 --2- 172.21.15.67:0/3295570991 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c70100750 0x7f5c7019cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:50.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.439+0000 7f5c76350700 1 -- 172.21.15.67:0/3295570991 >> 172.21.15.67:0/3295570991 conn(0x7f5c700fb650 msgr2=0x7f5c70103f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:53:50.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.439+0000 7f5c76350700 1 -- 172.21.15.67:0/3295570991 shutdown_connections 2024-04-22T23:53:50.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.439+0000 7f5c76350700 1 -- 172.21.15.67:0/3295570991 wait complete. 2024-04-22T23:53:50.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.545+0000 7ff429b8c700 1 -- 172.21.15.67:0/2347363261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff424100e90 msgr2=0x7ff424101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:50.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.545+0000 7ff429b8c700 1 --2- 172.21.15.67:0/2347363261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff424100e90 0x7ff424101270 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7ff41c00b0a0 tx=0x7ff41c00b3b0 comp rx=0 tx=0).stop 2024-04-22T23:53:50.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.545+0000 7ff429b8c700 1 -- 172.21.15.67:0/2347363261 shutdown_connections 2024-04-22T23:53:50.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.545+0000 7ff429b8c700 1 --2- 172.21.15.67:0/2347363261 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff424101840 0x7ff424105890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:50.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.545+0000 7ff429b8c700 1 --2- 172.21.15.67:0/2347363261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff424100e90 0x7ff424101270 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:50.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.545+0000 7ff429b8c700 1 -- 172.21.15.67:0/2347363261 >> 172.21.15.67:0/2347363261 conn(0x7ff4240fc740 msgr2=0x7ff4240feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:53:50.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.545+0000 7ff429b8c700 1 -- 172.21.15.67:0/2347363261 shutdown_connections 2024-04-22T23:53:50.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.545+0000 7ff429b8c700 1 -- 172.21.15.67:0/2347363261 wait complete. 2024-04-22T23:53:50.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.546+0000 7ff429b8c700 1 Processor -- start 2024-04-22T23:53:50.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.546+0000 7ff429b8c700 1 -- start start 2024-04-22T23:53:50.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.546+0000 7ff429b8c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff424100e90 0x7ff42419c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:50.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.546+0000 7ff429b8c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff424101840 0x7ff42419cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:50.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.546+0000 7ff429b8c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42419d230 con 0x7ff424100e90 2024-04-22T23:53:50.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.546+0000 7ff429b8c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4241966e0 con 0x7ff424101840 2024-04-22T23:53:50.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.546+0000 7ff4237fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff424100e90 0x7ff42419c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:50.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.546+0000 7ff4237fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff424100e90 0x7ff42419c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43516/0 (socket says 172.21.15.67:43516) 2024-04-22T23:53:50.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.546+0000 7ff4237fe700 1 -- 172.21.15.67:0/1819888495 learned_addr learned my addr 172.21.15.67:0/1819888495 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:53:50.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.546+0000 7ff422ffd700 1 --2- 172.21.15.67:0/1819888495 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff424101840 0x7ff42419cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:50.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.546+0000 7ff4237fe700 1 -- 172.21.15.67:0/1819888495 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff424101840 msgr2=0x7ff42419cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:50.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.546+0000 7ff4237fe700 1 --2- 172.21.15.67:0/1819888495 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff424101840 0x7ff42419cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:50.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.546+0000 7ff4237fe700 1 -- 172.21.15.67:0/1819888495 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff41c009d00 con 0x7ff424100e90 2024-04-22T23:53:50.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.547+0000 7ff4237fe700 1 --2- 172.21.15.67:0/1819888495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff424100e90 0x7ff42419c660 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7ff41c000f80 tx=0x7ff41c00cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:53:50.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.547+0000 7ff420ff9700 1 -- 172.21.15.67:0/1819888495 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff41c00e040 con 0x7ff424100e90 2024-04-22T23:53:50.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.547+0000 7ff420ff9700 1 -- 172.21.15.67:0/1819888495 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff41c003c10 con 0x7ff424100e90 2024-04-22T23:53:50.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.547+0000 7ff429b8c700 1 -- 172.21.15.67:0/1819888495 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff424196960 con 0x7ff424100e90 2024-04-22T23:53:50.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.547+0000 7ff420ff9700 1 -- 172.21.15.67:0/1819888495 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff41c01d6b0 con 0x7ff424100e90 2024-04-22T23:53:50.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.547+0000 7ff429b8c700 1 -- 172.21.15.67:0/1819888495 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff424196e50 con 0x7ff424100e90 2024-04-22T23:53:50.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.548+0000 7ff429b8c700 1 -- 172.21.15.67:0/1819888495 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff424109230 con 0x7ff424100e90 2024-04-22T23:53:50.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.548+0000 7ff420ff9700 1 -- 172.21.15.67:0/1819888495 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff41c004040 con 0x7ff424100e90 2024-04-22T23:53:50.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.549+0000 7ff420ff9700 1 --2- 172.21.15.67:0/1819888495 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff41006ecc0 0x7ff410071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:50.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.549+0000 7ff422ffd700 1 --2- 172.21.15.67:0/1819888495 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff41006ecc0 0x7ff410071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:50.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.549+0000 7ff420ff9700 1 -- 172.21.15.67:0/1819888495 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7ff41c090700 con 0x7ff424100e90 2024-04-22T23:53:50.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.549+0000 7ff422ffd700 1 --2- 172.21.15.67:0/1819888495 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff41006ecc0 0x7ff410071180 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7ff424197b50 tx=0x7ff40c00b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:53:50.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.554+0000 7ff420ff9700 1 -- 172.21.15.67:0/1819888495 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff41c05cef0 con 0x7ff424100e90 2024-04-22T23:53:50.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.815+0000 7ff429b8c700 1 -- 172.21.15.67:0/1819888495 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff424197c90 con 0x7ff424100e90 2024-04-22T23:53:50.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.815+0000 7ff420ff9700 1 -- 172.21.15.67:0/1819888495 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1872 (secure 0 0 0) 0x7ff41c05ca80 con 0x7ff424100e90 2024-04-22T23:53:50.808 INFO:teuthology.orchestra.run.smithi067.stdout:e33 2024-04-22T23:53:50.808 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:53:50.808 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:53:50.808 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:53:50.808 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:53:50.808 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:53:50.808 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:53:50.808 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:53:50.808 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:53:50.808 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:53:50.808 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:53:50.808 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:53:50.808 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:53:50.808 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:53:50.809 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:53:50.809 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:53:50.809 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:53:50.809 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:53:50.809 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:53:50.809 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:53:50.809 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:53:50.809 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:53:50.809 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:53:50.809 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:53:50.809 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:53:50.809 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:53:50.809 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:53:50.809 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:53:50.809 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:53:50.809 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:53:50.810 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:53:50.810 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:53:50.810 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:53:50.810 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:53:50.810 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:53:50.810 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:53:50.810 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:53:50.810 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:53:50.810 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:53:50.810 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:53:50.810 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:53:50.810 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44851} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1849648622,v1:172.21.15.134:6827/1849648622] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:53:50.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.817+0000 7ff429b8c700 1 -- 172.21.15.67:0/1819888495 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff41006ecc0 msgr2=0x7ff410071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:50.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.817+0000 7ff429b8c700 1 --2- 172.21.15.67:0/1819888495 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff41006ecc0 0x7ff410071180 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7ff424197b50 tx=0x7ff40c00b410 comp rx=0 tx=0).stop 2024-04-22T23:53:50.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.817+0000 7ff429b8c700 1 -- 172.21.15.67:0/1819888495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff424100e90 msgr2=0x7ff42419c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:50.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.817+0000 7ff429b8c700 1 --2- 172.21.15.67:0/1819888495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff424100e90 0x7ff42419c660 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7ff41c000f80 tx=0x7ff41c00cd20 comp rx=0 tx=0).stop 2024-04-22T23:53:50.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.817+0000 7ff429b8c700 1 -- 172.21.15.67:0/1819888495 shutdown_connections 2024-04-22T23:53:50.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.817+0000 7ff429b8c700 1 --2- 172.21.15.67:0/1819888495 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff41006ecc0 0x7ff410071180 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:50.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.817+0000 7ff429b8c700 1 --2- 172.21.15.67:0/1819888495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff424100e90 0x7ff42419c660 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:50.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.817+0000 7ff429b8c700 1 --2- 172.21.15.67:0/1819888495 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff424101840 0x7ff42419cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:50.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.817+0000 7ff429b8c700 1 -- 172.21.15.67:0/1819888495 >> 172.21.15.67:0/1819888495 conn(0x7ff4240fc740 msgr2=0x7ff4240fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:53:50.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.818+0000 7ff429b8c700 1 -- 172.21.15.67:0/1819888495 shutdown_connections 2024-04-22T23:53:50.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.818+0000 7ff429b8c700 1 -- 172.21.15.67:0/1819888495 wait complete. 2024-04-22T23:53:50.812 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 33 2024-04-22T23:53:50.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.925+0000 7f71e5ac8700 1 -- 172.21.15.67:0/2081604631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71e0068580 msgr2=0x7f71e00689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:50.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.925+0000 7f71e5ac8700 1 --2- 172.21.15.67:0/2081604631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71e0068580 0x7f71e00689e0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f71c8009910 tx=0x7f71c8009c20 comp rx=0 tx=0).stop 2024-04-22T23:53:50.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.925+0000 7f71e5ac8700 1 -- 172.21.15.67:0/2081604631 shutdown_connections 2024-04-22T23:53:50.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.925+0000 7f71e5ac8700 1 --2- 172.21.15.67:0/2081604631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71e0068580 0x7f71e00689e0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:50.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.925+0000 7f71e5ac8700 1 --2- 172.21.15.67:0/2081604631 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f71e0106cd0 0x7f71e01070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:50.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.925+0000 7f71e5ac8700 1 -- 172.21.15.67:0/2081604631 >> 172.21.15.67:0/2081604631 conn(0x7f71e0075670 msgr2=0x7f71e0075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:53:50.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.925+0000 7f71e5ac8700 1 -- 172.21.15.67:0/2081604631 shutdown_connections 2024-04-22T23:53:50.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.925+0000 7f71e5ac8700 1 -- 172.21.15.67:0/2081604631 wait complete. 2024-04-22T23:53:50.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.926+0000 7f71e5ac8700 1 Processor -- start 2024-04-22T23:53:50.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.926+0000 7f71e5ac8700 1 -- start start 2024-04-22T23:53:50.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.926+0000 7f71e5ac8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71e0068580 0x7f71e019c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:50.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.926+0000 7f71e5ac8700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f71e0106cd0 0x7f71e019cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:50.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.926+0000 7f71e5ac8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e019d220 con 0x7f71e0068580 2024-04-22T23:53:50.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.926+0000 7f71e5ac8700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e01966d0 con 0x7f71e0106cd0 2024-04-22T23:53:50.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.926+0000 7f71df7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71e0068580 0x7f71e019c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:50.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.926+0000 7f71df7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71e0068580 0x7f71e019c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43530/0 (socket says 172.21.15.67:43530) 2024-04-22T23:53:50.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.926+0000 7f71df7fe700 1 -- 172.21.15.67:0/3436384010 learned_addr learned my addr 172.21.15.67:0/3436384010 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:53:50.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.926+0000 7f71deffd700 1 --2- 172.21.15.67:0/3436384010 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f71e0106cd0 0x7f71e019cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:50.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.926+0000 7f71df7fe700 1 -- 172.21.15.67:0/3436384010 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f71e0106cd0 msgr2=0x7f71e019cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:50.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.926+0000 7f71df7fe700 1 --2- 172.21.15.67:0/3436384010 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f71e0106cd0 0x7f71e019cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:50.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.926+0000 7f71df7fe700 1 -- 172.21.15.67:0/3436384010 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c80095f0 con 0x7f71e0068580 2024-04-22T23:53:50.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.927+0000 7f71df7fe700 1 --2- 172.21.15.67:0/3436384010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71e0068580 0x7f71e019c650 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f71d0007e90 tx=0x7f71d000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:53:50.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.927+0000 7f71dcff9700 1 -- 172.21.15.67:0/3436384010 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71d0017070 con 0x7f71e0068580 2024-04-22T23:53:50.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.927+0000 7f71dcff9700 1 -- 172.21.15.67:0/3436384010 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f71d000f040 con 0x7f71e0068580 2024-04-22T23:53:50.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.927+0000 7f71dcff9700 1 -- 172.21.15.67:0/3436384010 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71d0005890 con 0x7f71e0068580 2024-04-22T23:53:50.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.927+0000 7f71e5ac8700 1 -- 172.21.15.67:0/3436384010 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71e01969b0 con 0x7f71e0068580 2024-04-22T23:53:50.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.927+0000 7f71e5ac8700 1 -- 172.21.15.67:0/3436384010 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71e0196eb0 con 0x7f71e0068580 2024-04-22T23:53:50.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.928+0000 7f71e5ac8700 1 -- 172.21.15.67:0/3436384010 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71e01091d0 con 0x7f71e0068580 2024-04-22T23:53:50.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.929+0000 7f71dcff9700 1 -- 172.21.15.67:0/3436384010 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f71d0004c40 con 0x7f71e0068580 2024-04-22T23:53:50.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.929+0000 7f71dcff9700 1 --2- 172.21.15.67:0/3436384010 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f71cc06ecc0 0x7f71cc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:50.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.929+0000 7f71dcff9700 1 -- 172.21.15.67:0/3436384010 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f71d008ea00 con 0x7f71e0068580 2024-04-22T23:53:50.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.929+0000 7f71deffd700 1 --2- 172.21.15.67:0/3436384010 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f71cc06ecc0 0x7f71cc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:50.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.930+0000 7f71deffd700 1 --2- 172.21.15.67:0/3436384010 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f71cc06ecc0 0x7f71cc071180 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f71e0197b40 tx=0x7f71c801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:53:50.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:50.934+0000 7f71dcff9700 1 -- 172.21.15.67:0/3436384010 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f71d005b170 con 0x7f71e0068580 2024-04-22T23:53:51.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.167+0000 7f71e5ac8700 1 -- 172.21.15.67:0/3436384010 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f71e0197120 con 0x7f71cc06ecc0 2024-04-22T23:53:51.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.168+0000 7f71dcff9700 1 -- 172.21.15.67:0/3436384010 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f71e0197120 con 0x7f71cc06ecc0 2024-04-22T23:53:51.161 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:53:51.161 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:53:51.161 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:53:51.161 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:53:51.161 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:53:51.161 INFO:teuthology.orchestra.run.smithi067.stdout: "mds", 2024-04-22T23:53:51.161 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:53:51.161 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:53:51.161 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:53:51.161 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:53:51.161 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:53:51.161 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T23:53:51.161 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:53:51.161 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:53:51.161 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:53:51.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.170+0000 7f71e5ac8700 1 -- 172.21.15.67:0/3436384010 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f71cc06ecc0 msgr2=0x7f71cc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:51.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.170+0000 7f71e5ac8700 1 --2- 172.21.15.67:0/3436384010 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f71cc06ecc0 0x7f71cc071180 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f71e0197b40 tx=0x7f71c801a040 comp rx=0 tx=0).stop 2024-04-22T23:53:51.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.171+0000 7f71e5ac8700 1 -- 172.21.15.67:0/3436384010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71e0068580 msgr2=0x7f71e019c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:51.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.171+0000 7f71e5ac8700 1 --2- 172.21.15.67:0/3436384010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71e0068580 0x7f71e019c650 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f71d0007e90 tx=0x7f71d000d3b0 comp rx=0 tx=0).stop 2024-04-22T23:53:51.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.171+0000 7f71e5ac8700 1 -- 172.21.15.67:0/3436384010 shutdown_connections 2024-04-22T23:53:51.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.171+0000 7f71e5ac8700 1 --2- 172.21.15.67:0/3436384010 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f71cc06ecc0 0x7f71cc071180 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:51.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.171+0000 7f71e5ac8700 1 --2- 172.21.15.67:0/3436384010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71e0068580 0x7f71e019c650 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:51.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.171+0000 7f71e5ac8700 1 --2- 172.21.15.67:0/3436384010 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f71e0106cd0 0x7f71e019cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:51.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.171+0000 7f71e5ac8700 1 -- 172.21.15.67:0/3436384010 >> 172.21.15.67:0/3436384010 conn(0x7f71e0075670 msgr2=0x7f71e00feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:53:51.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.171+0000 7f71e5ac8700 1 -- 172.21.15.67:0/3436384010 shutdown_connections 2024-04-22T23:53:51.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.171+0000 7f71e5ac8700 1 -- 172.21.15.67:0/3436384010 wait complete. 2024-04-22T23:53:51.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.277+0000 7fc9b4867700 1 -- 172.21.15.67:0/3049135358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9b01027a0 msgr2=0x7fc9b010ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:51.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.277+0000 7fc9b4867700 1 --2- 172.21.15.67:0/3049135358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9b01027a0 0x7fc9b010ac90 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fc998009910 tx=0x7fc998009c20 comp rx=0 tx=0).stop 2024-04-22T23:53:51.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.277+0000 7fc9b4867700 1 -- 172.21.15.67:0/3049135358 shutdown_connections 2024-04-22T23:53:51.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.277+0000 7fc9b4867700 1 --2- 172.21.15.67:0/3049135358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9b01027a0 0x7fc9b010ac90 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:51.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.277+0000 7fc9b4867700 1 --2- 172.21.15.67:0/3049135358 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9b0101e80 0x7fc9b0102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:51.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.277+0000 7fc9b4867700 1 -- 172.21.15.67:0/3049135358 >> 172.21.15.67:0/3049135358 conn(0x7fc9b00fb610 msgr2=0x7fc9b00fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:53:51.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.277+0000 7fc9b4867700 1 -- 172.21.15.67:0/3049135358 shutdown_connections 2024-04-22T23:53:51.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.277+0000 7fc9b4867700 1 -- 172.21.15.67:0/3049135358 wait complete. 2024-04-22T23:53:51.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.278+0000 7fc9b4867700 1 Processor -- start 2024-04-22T23:53:51.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.278+0000 7fc9b4867700 1 -- start start 2024-04-22T23:53:51.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.278+0000 7fc9b4867700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9b0101e80 0x7fc9b019e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:51.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.278+0000 7fc9b4867700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9b01027a0 0x7fc9b019edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:51.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.278+0000 7fc9b4867700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b019f440 con 0x7fc9b01027a0 2024-04-22T23:53:51.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.278+0000 7fc9b4867700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b01988f0 con 0x7fc9b0101e80 2024-04-22T23:53:51.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.278+0000 7fc9add9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9b01027a0 0x7fc9b019edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:51.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.278+0000 7fc9add9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9b01027a0 0x7fc9b019edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43550/0 (socket says 172.21.15.67:43550) 2024-04-22T23:53:51.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.278+0000 7fc9add9b700 1 -- 172.21.15.67:0/186073752 learned_addr learned my addr 172.21.15.67:0/186073752 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:53:51.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.278+0000 7fc9ae59c700 1 --2- 172.21.15.67:0/186073752 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9b0101e80 0x7fc9b019e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:51.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.278+0000 7fc9add9b700 1 -- 172.21.15.67:0/186073752 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9b0101e80 msgr2=0x7fc9b019e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:51.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.278+0000 7fc9add9b700 1 --2- 172.21.15.67:0/186073752 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9b0101e80 0x7fc9b019e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:51.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.278+0000 7fc9add9b700 1 -- 172.21.15.67:0/186073752 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9980095f0 con 0x7fc9b01027a0 2024-04-22T23:53:51.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.279+0000 7fc9add9b700 1 --2- 172.21.15.67:0/186073752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9b01027a0 0x7fc9b019edb0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fc99800bd70 tx=0x7fc99800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:53:51.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.279+0000 7fc9a77fe700 1 -- 172.21.15.67:0/186073752 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc99801d070 con 0x7fc9b01027a0 2024-04-22T23:53:51.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.279+0000 7fc9a77fe700 1 -- 172.21.15.67:0/186073752 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc99800f460 con 0x7fc9b01027a0 2024-04-22T23:53:51.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.279+0000 7fc9b4867700 1 -- 172.21.15.67:0/186073752 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9b0198b70 con 0x7fc9b01027a0 2024-04-22T23:53:51.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.279+0000 7fc9a77fe700 1 -- 172.21.15.67:0/186073752 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc998017630 con 0x7fc9b01027a0 2024-04-22T23:53:51.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.279+0000 7fc9b4867700 1 -- 172.21.15.67:0/186073752 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9b0199060 con 0x7fc9b01027a0 2024-04-22T23:53:51.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.280+0000 7fc9b4867700 1 -- 172.21.15.67:0/186073752 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9b010a7f0 con 0x7fc9b01027a0 2024-04-22T23:53:51.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.280+0000 7fc9a77fe700 1 -- 172.21.15.67:0/186073752 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc99800f5d0 con 0x7fc9b01027a0 2024-04-22T23:53:51.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.281+0000 7fc9a77fe700 1 --2- 172.21.15.67:0/186073752 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc99c06ecc0 0x7fc99c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:53:51.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.281+0000 7fc9a77fe700 1 -- 172.21.15.67:0/186073752 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fc998090510 con 0x7fc9b01027a0 2024-04-22T23:53:51.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.281+0000 7fc9ae59c700 1 --2- 172.21.15.67:0/186073752 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc99c06ecc0 0x7fc99c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:53:51.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.281+0000 7fc9ae59c700 1 --2- 172.21.15.67:0/186073752 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc99c06ecc0 0x7fc99c071180 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fc9a80060b0 tx=0x7fc9a8008640 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:53:51.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.286+0000 7fc9a77fe700 1 -- 172.21.15.67:0/186073752 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc99805cca0 con 0x7fc9b01027a0 2024-04-22T23:53:51.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:51 smithi134 ceph-mon[85630]: from='client.35496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:53:51.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:51 smithi134 ceph-mon[85630]: from='client.35500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:53:51.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:51 smithi134 ceph-mon[85630]: from='client.35504 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:53:51.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:51 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3295570991' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:53:51.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:51 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1819888495' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:53:51.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:51 smithi067 ceph-mon[112438]: from='client.35496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:53:51.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:51 smithi067 ceph-mon[112438]: from='client.35500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:53:51.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:51 smithi067 ceph-mon[112438]: from='client.35504 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:53:51.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:51 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3295570991' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:53:51.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:51 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1819888495' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:53:51.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.585+0000 7fc9b4867700 1 -- 172.21.15.67:0/186073752 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc9b0199e60 con 0x7fc9b01027a0 2024-04-22T23:53:51.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.585+0000 7fc9a77fe700 1 -- 172.21.15.67:0/186073752 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4505 (secure 0 0 0) 0x7fc99805c830 con 0x7fc9b01027a0 2024-04-22T23:53:51.577 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:53:51.577 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T23:53:51.578 INFO:teuthology.orchestra.run.smithi067.stdout: host smithi134 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-22T23:53:51.578 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T23:53:51.578 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:53:51.578 INFO:teuthology.orchestra.run.smithi067.stdout:Traceback (most recent call last): 2024-04-22T23:53:51.578 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T23:53:51.578 INFO:teuthology.orchestra.run.smithi067.stdout: return _run_code(code, main_globals, None, 2024-04-22T23:53:51.578 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T23:53:51.578 INFO:teuthology.orchestra.run.smithi067.stdout: exec(code, run_globals) 2024-04-22T23:53:51.578 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5642, in 2024-04-22T23:53:51.578 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5630, in main 2024-04-22T23:53:51.578 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 408, in _infer_config 2024-04-22T23:53:51.578 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 323, in _infer_fsid 2024-04-22T23:53:51.578 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 436, in _infer_image 2024-04-22T23:53:51.579 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 310, in _validate_fsid 2024-04-22T23:53:51.579 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 3380, in command_ceph_volume 2024-04-22T23:53:51.579 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T23:53:51.579 INFO:teuthology.orchestra.run.smithi067.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:53:51.579 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:53:51.579 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:53:51.579 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:53:51.579 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:53:51.579 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:53:51.579 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:53:51.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.587+0000 7fc9b4867700 1 -- 172.21.15.67:0/186073752 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc99c06ecc0 msgr2=0x7fc99c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:51.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.587+0000 7fc9b4867700 1 --2- 172.21.15.67:0/186073752 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc99c06ecc0 0x7fc99c071180 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fc9a80060b0 tx=0x7fc9a8008640 comp rx=0 tx=0).stop 2024-04-22T23:53:51.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.587+0000 7fc9b4867700 1 -- 172.21.15.67:0/186073752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9b01027a0 msgr2=0x7fc9b019edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:53:51.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.587+0000 7fc9b4867700 1 --2- 172.21.15.67:0/186073752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9b01027a0 0x7fc9b019edb0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fc99800bd70 tx=0x7fc99800be50 comp rx=0 tx=0).stop 2024-04-22T23:53:51.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.587+0000 7fc9b4867700 1 -- 172.21.15.67:0/186073752 shutdown_connections 2024-04-22T23:53:51.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.587+0000 7fc9b4867700 1 --2- 172.21.15.67:0/186073752 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc99c06ecc0 0x7fc99c071180 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:51.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.587+0000 7fc9b4867700 1 --2- 172.21.15.67:0/186073752 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc9b0101e80 0x7fc9b019e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:51.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.587+0000 7fc9b4867700 1 --2- 172.21.15.67:0/186073752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9b01027a0 0x7fc9b019edb0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:53:51.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.587+0000 7fc9b4867700 1 -- 172.21.15.67:0/186073752 >> 172.21.15.67:0/186073752 conn(0x7fc9b00fb610 msgr2=0x7fc9b01002a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:53:51.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.587+0000 7fc9b4867700 1 -- 172.21.15.67:0/186073752 shutdown_connections 2024-04-22T23:53:51.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:53:51.587+0000 7fc9b4867700 1 -- 172.21.15.67:0/186073752 wait complete. 2024-04-22T23:53:52.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:52 smithi134 ceph-mon[85630]: pgmap v826: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:52.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:52 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/186073752' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:53:52.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:52 smithi067 ceph-mon[112438]: pgmap v826: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:52.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:52 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/186073752' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:53:53.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:53 smithi134 ceph-mon[85630]: from='client.35516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:53:53.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:53 smithi134 ceph-mon[85630]: pgmap v827: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:53.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:53 smithi067 ceph-mon[112438]: from='client.35516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:53:53.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:53 smithi067 ceph-mon[112438]: pgmap v827: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:56.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:56 smithi134 ceph-mon[85630]: pgmap v828: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:56.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:53:56.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:56 smithi067 ceph-mon[112438]: pgmap v828: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:56.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:53:58.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:53:58 smithi134 ceph-mon[85630]: pgmap v829: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:53:58.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:53:58 smithi067 ceph-mon[112438]: pgmap v829: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:00.441 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:00 smithi134 ceph-mon[85630]: pgmap v830: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:00.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:00 smithi067 ceph-mon[112438]: pgmap v830: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:02.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:02 smithi134 ceph-mon[85630]: pgmap v831: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:02.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:02 smithi067 ceph-mon[112438]: pgmap v831: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:04.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:04 smithi134 ceph-mon[85630]: pgmap v832: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:04.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:04 smithi067 ceph-mon[112438]: pgmap v832: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:05.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:05 smithi134 ceph-mon[85630]: pgmap v833: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:05.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:05 smithi067 ceph-mon[112438]: pgmap v833: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:08.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:08 smithi134 ceph-mon[85630]: pgmap v834: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:08.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:08 smithi067 ceph-mon[112438]: pgmap v834: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:10.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:10 smithi134 ceph-mon[85630]: pgmap v835: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:10.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:10 smithi067 ceph-mon[112438]: pgmap v835: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:11.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:54:11.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:54:12.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:12 smithi134 ceph-mon[85630]: pgmap v836: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:12.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:12 smithi067 ceph-mon[112438]: pgmap v836: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:13.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:13 smithi134 ceph-mon[85630]: pgmap v837: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:13.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:13 smithi067 ceph-mon[112438]: pgmap v837: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:16.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:16 smithi134 ceph-mon[85630]: pgmap v838: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:16.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:16 smithi067 ceph-mon[112438]: pgmap v838: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:18.337 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:18 smithi067 ceph-mon[112438]: pgmap v839: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:18.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:18 smithi134 ceph-mon[85630]: pgmap v839: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:20.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:20 smithi134 ceph-mon[85630]: pgmap v840: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:20.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:20 smithi067 ceph-mon[112438]: pgmap v840: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:21.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.709+0000 7fd5452c2700 1 -- 172.21.15.67:0/3643973855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd540100750 msgr2=0x7fd5401047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:21.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.709+0000 7fd5452c2700 1 --2- 172.21.15.67:0/3643973855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd540100750 0x7fd5401047a0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fd528009910 tx=0x7fd528009c20 comp rx=0 tx=0).stop 2024-04-22T23:54:21.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.709+0000 7fd5452c2700 1 -- 172.21.15.67:0/3643973855 shutdown_connections 2024-04-22T23:54:21.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.709+0000 7fd5452c2700 1 --2- 172.21.15.67:0/3643973855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd540100750 0x7fd5401047a0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:21.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.709+0000 7fd5452c2700 1 --2- 172.21.15.67:0/3643973855 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd5400ffda0 0x7fd540100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:21.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.709+0000 7fd5452c2700 1 -- 172.21.15.67:0/3643973855 >> 172.21.15.67:0/3643973855 conn(0x7fd5400fb650 msgr2=0x7fd5400fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:54:21.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.709+0000 7fd5452c2700 1 -- 172.21.15.67:0/3643973855 shutdown_connections 2024-04-22T23:54:21.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.710+0000 7fd5452c2700 1 -- 172.21.15.67:0/3643973855 wait complete. 2024-04-22T23:54:21.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.710+0000 7fd5452c2700 1 Processor -- start 2024-04-22T23:54:21.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.710+0000 7fd5452c2700 1 -- start start 2024-04-22T23:54:21.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.711+0000 7fd5452c2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5400ffda0 0x7fd540101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:21.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.711+0000 7fd5452c2700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd540100750 0x7fd540101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:21.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.711+0000 7fd5452c2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5401022c0 con 0x7fd5400ffda0 2024-04-22T23:54:21.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.711+0000 7fd5452c2700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd540102400 con 0x7fd540100750 2024-04-22T23:54:21.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.711+0000 7fd53effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5400ffda0 0x7fd540101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:21.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.711+0000 7fd53effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5400ffda0 0x7fd540101840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42260/0 (socket says 172.21.15.67:42260) 2024-04-22T23:54:21.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.711+0000 7fd53effd700 1 -- 172.21.15.67:0/2579453942 learned_addr learned my addr 172.21.15.67:0/2579453942 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:54:21.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.711+0000 7fd53e7fc700 1 --2- 172.21.15.67:0/2579453942 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd540100750 0x7fd540101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:21.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.711+0000 7fd53effd700 1 -- 172.21.15.67:0/2579453942 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd540100750 msgr2=0x7fd540101d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:21.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.711+0000 7fd53effd700 1 --2- 172.21.15.67:0/2579453942 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd540100750 0x7fd540101d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:21.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.711+0000 7fd53effd700 1 -- 172.21.15.67:0/2579453942 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5280095f0 con 0x7fd5400ffda0 2024-04-22T23:54:21.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.711+0000 7fd53effd700 1 --2- 172.21.15.67:0/2579453942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5400ffda0 0x7fd540101840 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fd53000d8d0 tx=0x7fd53000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:54:21.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.711+0000 7fd537fff700 1 -- 172.21.15.67:0/2579453942 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd530009940 con 0x7fd5400ffda0 2024-04-22T23:54:21.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.712+0000 7fd5452c2700 1 -- 172.21.15.67:0/2579453942 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5401a2a70 con 0x7fd5400ffda0 2024-04-22T23:54:21.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.712+0000 7fd537fff700 1 -- 172.21.15.67:0/2579453942 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd530010460 con 0x7fd5400ffda0 2024-04-22T23:54:21.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.712+0000 7fd537fff700 1 -- 172.21.15.67:0/2579453942 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd53000f5f0 con 0x7fd5400ffda0 2024-04-22T23:54:21.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.712+0000 7fd5452c2700 1 -- 172.21.15.67:0/2579453942 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5401a2e80 con 0x7fd5400ffda0 2024-04-22T23:54:21.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.713+0000 7fd5452c2700 1 -- 172.21.15.67:0/2579453942 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5401091b0 con 0x7fd5400ffda0 2024-04-22T23:54:21.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.713+0000 7fd537fff700 1 -- 172.21.15.67:0/2579453942 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd5300105d0 con 0x7fd5400ffda0 2024-04-22T23:54:21.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.714+0000 7fd537fff700 1 --2- 172.21.15.67:0/2579453942 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd52c06ecc0 0x7fd52c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:21.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.714+0000 7fd537fff700 1 -- 172.21.15.67:0/2579453942 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fd53008eff0 con 0x7fd5400ffda0 2024-04-22T23:54:21.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.714+0000 7fd53e7fc700 1 --2- 172.21.15.67:0/2579453942 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd52c06ecc0 0x7fd52c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:21.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.714+0000 7fd53e7fc700 1 --2- 172.21.15.67:0/2579453942 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd52c06ecc0 0x7fd52c071180 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fd540103030 tx=0x7fd52801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:54:21.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.719+0000 7fd537fff700 1 -- 172.21.15.67:0/2579453942 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fd530057c30 con 0x7fd5400ffda0 2024-04-22T23:54:21.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.951+0000 7fd5452c2700 1 -- 172.21.15.67:0/2579453942 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd5400611d0 con 0x7fd52c06ecc0 2024-04-22T23:54:21.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.952+0000 7fd537fff700 1 -- 172.21.15.67:0/2579453942 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd5400611d0 con 0x7fd52c06ecc0 2024-04-22T23:54:21.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.954+0000 7fd5452c2700 1 -- 172.21.15.67:0/2579453942 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd52c06ecc0 msgr2=0x7fd52c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:21.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.954+0000 7fd5452c2700 1 --2- 172.21.15.67:0/2579453942 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd52c06ecc0 0x7fd52c071180 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fd540103030 tx=0x7fd52801a040 comp rx=0 tx=0).stop 2024-04-22T23:54:21.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.954+0000 7fd5452c2700 1 -- 172.21.15.67:0/2579453942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5400ffda0 msgr2=0x7fd540101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:21.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.954+0000 7fd5452c2700 1 --2- 172.21.15.67:0/2579453942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5400ffda0 0x7fd540101840 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fd53000d8d0 tx=0x7fd53000dc90 comp rx=0 tx=0).stop 2024-04-22T23:54:21.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.954+0000 7fd5452c2700 1 -- 172.21.15.67:0/2579453942 shutdown_connections 2024-04-22T23:54:21.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.954+0000 7fd5452c2700 1 --2- 172.21.15.67:0/2579453942 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd52c06ecc0 0x7fd52c071180 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:21.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.954+0000 7fd5452c2700 1 --2- 172.21.15.67:0/2579453942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5400ffda0 0x7fd540101840 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:21.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.954+0000 7fd5452c2700 1 --2- 172.21.15.67:0/2579453942 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd540100750 0x7fd540101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:21.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.954+0000 7fd5452c2700 1 -- 172.21.15.67:0/2579453942 >> 172.21.15.67:0/2579453942 conn(0x7fd5400fb650 msgr2=0x7fd540069150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:54:21.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.954+0000 7fd5452c2700 1 -- 172.21.15.67:0/2579453942 shutdown_connections 2024-04-22T23:54:21.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:21.954+0000 7fd5452c2700 1 -- 172.21.15.67:0/2579453942 wait complete. 2024-04-22T23:54:21.956 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:54:22.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.063+0000 7f597405b700 1 -- 172.21.15.67:0/249078248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f596c100ea0 msgr2=0x7f596c101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:22.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.063+0000 7f597405b700 1 --2- 172.21.15.67:0/249078248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f596c100ea0 0x7f596c101280 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f5964009910 tx=0x7f5964009c20 comp rx=0 tx=0).stop 2024-04-22T23:54:22.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.063+0000 7f597405b700 1 -- 172.21.15.67:0/249078248 shutdown_connections 2024-04-22T23:54:22.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.063+0000 7f597405b700 1 --2- 172.21.15.67:0/249078248 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f596c101850 0x7f596c1058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:22.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.063+0000 7f597405b700 1 --2- 172.21.15.67:0/249078248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f596c100ea0 0x7f596c101280 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:22.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.063+0000 7f597405b700 1 -- 172.21.15.67:0/249078248 >> 172.21.15.67:0/249078248 conn(0x7f596c0fc750 msgr2=0x7f596c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:54:22.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.063+0000 7f597405b700 1 -- 172.21.15.67:0/249078248 shutdown_connections 2024-04-22T23:54:22.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.063+0000 7f597405b700 1 -- 172.21.15.67:0/249078248 wait complete. 2024-04-22T23:54:22.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.064+0000 7f597405b700 1 Processor -- start 2024-04-22T23:54:22.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.064+0000 7f597405b700 1 -- start start 2024-04-22T23:54:22.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.064+0000 7f597405b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f596c100ea0 0x7f596c073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:22.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.064+0000 7f597405b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f596c101850 0x7f596c0735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:22.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.064+0000 7f597405b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596c073b10 con 0x7f596c100ea0 2024-04-22T23:54:22.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.064+0000 7f597405b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596c073c50 con 0x7f596c101850 2024-04-22T23:54:22.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.064+0000 7f5971df7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f596c100ea0 0x7f596c073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:22.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.064+0000 7f5971df7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f596c100ea0 0x7f596c073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42286/0 (socket says 172.21.15.67:42286) 2024-04-22T23:54:22.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.064+0000 7f5971df7700 1 -- 172.21.15.67:0/115508386 learned_addr learned my addr 172.21.15.67:0/115508386 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:54:22.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.064+0000 7f59715f6700 1 --2- 172.21.15.67:0/115508386 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f596c101850 0x7f596c0735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:22.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.064+0000 7f5971df7700 1 -- 172.21.15.67:0/115508386 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f596c101850 msgr2=0x7f596c0735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:22.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.064+0000 7f5971df7700 1 --2- 172.21.15.67:0/115508386 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f596c101850 0x7f596c0735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:22.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.064+0000 7f5971df7700 1 -- 172.21.15.67:0/115508386 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59640095f0 con 0x7f596c100ea0 2024-04-22T23:54:22.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.065+0000 7f5971df7700 1 --2- 172.21.15.67:0/115508386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f596c100ea0 0x7f596c073090 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f5964006010 tx=0x7f596400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:54:22.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.065+0000 7f596affd700 1 -- 172.21.15.67:0/115508386 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f596401d070 con 0x7f596c100ea0 2024-04-22T23:54:22.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.065+0000 7f596affd700 1 -- 172.21.15.67:0/115508386 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f596400f460 con 0x7f596c100ea0 2024-04-22T23:54:22.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.065+0000 7f597405b700 1 -- 172.21.15.67:0/115508386 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f596c073ed0 con 0x7f596c100ea0 2024-04-22T23:54:22.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.065+0000 7f596affd700 1 -- 172.21.15.67:0/115508386 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5964017630 con 0x7f596c100ea0 2024-04-22T23:54:22.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.065+0000 7f597405b700 1 -- 172.21.15.67:0/115508386 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f596c1a2e40 con 0x7f596c100ea0 2024-04-22T23:54:22.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.066+0000 7f597405b700 1 -- 172.21.15.67:0/115508386 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f596c109240 con 0x7f596c100ea0 2024-04-22T23:54:22.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.066+0000 7f596affd700 1 -- 172.21.15.67:0/115508386 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f596400f5d0 con 0x7f596c100ea0 2024-04-22T23:54:22.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.067+0000 7f596affd700 1 --2- 172.21.15.67:0/115508386 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f595406ecc0 0x7f5954071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:22.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.067+0000 7f596affd700 1 -- 172.21.15.67:0/115508386 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f59640903f0 con 0x7f596c100ea0 2024-04-22T23:54:22.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.067+0000 7f59715f6700 1 --2- 172.21.15.67:0/115508386 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f595406ecc0 0x7f5954071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:22.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.067+0000 7f59715f6700 1 --2- 172.21.15.67:0/115508386 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f595406ecc0 0x7f5954071180 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f5960005950 tx=0x7f59600058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:54:22.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.072+0000 7f596affd700 1 -- 172.21.15.67:0/115508386 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f59640590b0 con 0x7f596c100ea0 2024-04-22T23:54:22.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:22 smithi067 ceph-mon[112438]: pgmap v841: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:22.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.303+0000 7f597405b700 1 -- 172.21.15.67:0/115508386 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f596c0611d0 con 0x7f595406ecc0 2024-04-22T23:54:22.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.304+0000 7f596affd700 1 -- 172.21.15.67:0/115508386 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f596c0611d0 con 0x7f595406ecc0 2024-04-22T23:54:22.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.307+0000 7f597405b700 1 -- 172.21.15.67:0/115508386 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f595406ecc0 msgr2=0x7f5954071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:22.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.307+0000 7f597405b700 1 --2- 172.21.15.67:0/115508386 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f595406ecc0 0x7f5954071180 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f5960005950 tx=0x7f59600058e0 comp rx=0 tx=0).stop 2024-04-22T23:54:22.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.307+0000 7f597405b700 1 -- 172.21.15.67:0/115508386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f596c100ea0 msgr2=0x7f596c073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:22.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.307+0000 7f597405b700 1 --2- 172.21.15.67:0/115508386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f596c100ea0 0x7f596c073090 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f5964006010 tx=0x7f596400be20 comp rx=0 tx=0).stop 2024-04-22T23:54:22.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.307+0000 7f597405b700 1 -- 172.21.15.67:0/115508386 shutdown_connections 2024-04-22T23:54:22.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.307+0000 7f597405b700 1 --2- 172.21.15.67:0/115508386 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f595406ecc0 0x7f5954071180 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:22.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.307+0000 7f597405b700 1 --2- 172.21.15.67:0/115508386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f596c100ea0 0x7f596c073090 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:22.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.307+0000 7f597405b700 1 --2- 172.21.15.67:0/115508386 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f596c101850 0x7f596c0735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:22.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.307+0000 7f597405b700 1 -- 172.21.15.67:0/115508386 >> 172.21.15.67:0/115508386 conn(0x7f596c0fc750 msgr2=0x7f596c0fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:54:22.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.307+0000 7f597405b700 1 -- 172.21.15.67:0/115508386 shutdown_connections 2024-04-22T23:54:22.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.307+0000 7f597405b700 1 -- 172.21.15.67:0/115508386 wait complete. 2024-04-22T23:54:22.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.414+0000 7ffa2d2f0700 1 -- 172.21.15.67:0/3983005014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa28101850 msgr2=0x7ffa281058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:22.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.414+0000 7ffa2d2f0700 1 --2- 172.21.15.67:0/3983005014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa28101850 0x7ffa281058a0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c009910 tx=0x7ffa1c009c20 comp rx=0 tx=0).stop 2024-04-22T23:54:22.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.414+0000 7ffa2d2f0700 1 -- 172.21.15.67:0/3983005014 shutdown_connections 2024-04-22T23:54:22.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.414+0000 7ffa2d2f0700 1 --2- 172.21.15.67:0/3983005014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa28101850 0x7ffa281058a0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:22.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.414+0000 7ffa2d2f0700 1 --2- 172.21.15.67:0/3983005014 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa28100ea0 0x7ffa28101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:22.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.414+0000 7ffa2d2f0700 1 -- 172.21.15.67:0/3983005014 >> 172.21.15.67:0/3983005014 conn(0x7ffa280fc750 msgr2=0x7ffa280feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:54:22.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.414+0000 7ffa2d2f0700 1 -- 172.21.15.67:0/3983005014 shutdown_connections 2024-04-22T23:54:22.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.414+0000 7ffa2d2f0700 1 -- 172.21.15.67:0/3983005014 wait complete. 2024-04-22T23:54:22.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.414+0000 7ffa2d2f0700 1 Processor -- start 2024-04-22T23:54:22.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.415+0000 7ffa2d2f0700 1 -- start start 2024-04-22T23:54:22.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.415+0000 7ffa2d2f0700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa28100ea0 0x7ffa28073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:22.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.415+0000 7ffa2d2f0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa28101850 0x7ffa280735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:22.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.415+0000 7ffa2d2f0700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa28073b10 con 0x7ffa28101850 2024-04-22T23:54:22.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.415+0000 7ffa2d2f0700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa28073c50 con 0x7ffa28100ea0 2024-04-22T23:54:22.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.415+0000 7ffa267fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa28101850 0x7ffa280735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:22.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.415+0000 7ffa267fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa28101850 0x7ffa280735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42304/0 (socket says 172.21.15.67:42304) 2024-04-22T23:54:22.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.415+0000 7ffa267fc700 1 -- 172.21.15.67:0/3379759156 learned_addr learned my addr 172.21.15.67:0/3379759156 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:54:22.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.415+0000 7ffa26ffd700 1 --2- 172.21.15.67:0/3379759156 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa28100ea0 0x7ffa28073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:22.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.415+0000 7ffa267fc700 1 -- 172.21.15.67:0/3379759156 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa28100ea0 msgr2=0x7ffa28073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:22.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.415+0000 7ffa267fc700 1 --2- 172.21.15.67:0/3379759156 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa28100ea0 0x7ffa28073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:22.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.415+0000 7ffa267fc700 1 -- 172.21.15.67:0/3379759156 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa1c0095f0 con 0x7ffa28101850 2024-04-22T23:54:22.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.415+0000 7ffa26ffd700 1 --2- 172.21.15.67:0/3379759156 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa28100ea0 0x7ffa28073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:54:22.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.415+0000 7ffa267fc700 1 --2- 172.21.15.67:0/3379759156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa28101850 0x7ffa280735d0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c00bda0 tx=0x7ffa1c00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:54:22.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.416+0000 7ffa0ffff700 1 -- 172.21.15.67:0/3379759156 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa1c01d070 con 0x7ffa28101850 2024-04-22T23:54:22.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.416+0000 7ffa0ffff700 1 -- 172.21.15.67:0/3379759156 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffa1c004fb0 con 0x7ffa28101850 2024-04-22T23:54:22.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.416+0000 7ffa0ffff700 1 -- 172.21.15.67:0/3379759156 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa1c00f480 con 0x7ffa28101850 2024-04-22T23:54:22.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.416+0000 7ffa2d2f0700 1 -- 172.21.15.67:0/3379759156 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa28073ed0 con 0x7ffa28101850 2024-04-22T23:54:22.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.416+0000 7ffa2d2f0700 1 -- 172.21.15.67:0/3379759156 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa281a2e30 con 0x7ffa28101850 2024-04-22T23:54:22.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.417+0000 7ffa2d2f0700 1 -- 172.21.15.67:0/3379759156 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa281092a0 con 0x7ffa28101850 2024-04-22T23:54:22.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.417+0000 7ffa0ffff700 1 -- 172.21.15.67:0/3379759156 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ffa1c022870 con 0x7ffa28101850 2024-04-22T23:54:22.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.418+0000 7ffa0ffff700 1 --2- 172.21.15.67:0/3379759156 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ffa1006ecc0 0x7ffa10071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:22.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.418+0000 7ffa26ffd700 1 --2- 172.21.15.67:0/3379759156 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ffa1006ecc0 0x7ffa10071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:22.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.418+0000 7ffa0ffff700 1 -- 172.21.15.67:0/3379759156 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7ffa1c090470 con 0x7ffa28101850 2024-04-22T23:54:22.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.418+0000 7ffa26ffd700 1 --2- 172.21.15.67:0/3379759156 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ffa1006ecc0 0x7ffa10071180 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7ffa18005950 tx=0x7ffa18005f50 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:54:22.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.423+0000 7ffa0ffff700 1 -- 172.21.15.67:0/3379759156 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ffa1c0591e0 con 0x7ffa28101850 2024-04-22T23:54:22.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:22 smithi134 ceph-mon[85630]: pgmap v841: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:22.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.640+0000 7ffa2d2f0700 1 -- 172.21.15.67:0/3379759156 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ffa28074660 con 0x7ffa1006ecc0 2024-04-22T23:54:22.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.652+0000 7ffa0ffff700 1 -- 172.21.15.67:0/3379759156 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7ffa28074660 con 0x7ffa1006ecc0 2024-04-22T23:54:22.643 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:54:22.643 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (28m) 5m ago 39m 24.4M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:54:22.643 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (39m) 5m ago 39m 10.0M - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:54:22.643 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (38m) 5m ago 38m 10.0M - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:54:22.643 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (25m) 5m ago 39m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:54:22.643 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (25m) 5m ago 38m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:54:22.643 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (27m) 5m ago 39m 81.2M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:54:22.643 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (21m) 5m ago 33m 19.4M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:54:22.643 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (21m) 5m ago 33m 20.3M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:54:22.643 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (21m) 5m ago 33m 20.1M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:54:22.644 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (5m) 5m ago 33m 14.0M - 19.0.0-3252-g43be0201 a0434699c1d7 46d3d7e86fee 2024-04-22T23:54:22.644 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (29m) 5m ago 40m 514M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:54:22.644 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (28m) 5m ago 38m 459M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:54:22.644 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (26m) 5m ago 41m 74.2M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:54:22.644 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (26m) 5m ago 38m 54.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:54:22.644 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (28m) 5m ago 39m 15.2M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:54:22.644 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (28m) 5m ago 38m 16.0M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:54:22.644 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (25m) 5m ago 37m 102M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:54:22.644 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (24m) 5m ago 37m 128M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:54:22.644 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (23m) 5m ago 36m 83.8M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:54:22.644 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (23m) 5m ago 36m 101M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:54:22.644 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (22m) 5m ago 35m 86.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:54:22.644 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (22m) 5m ago 35m 81.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:54:22.644 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (28m) 5m ago 38m 78.7M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:54:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.654+0000 7ffa2d2f0700 1 -- 172.21.15.67:0/3379759156 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ffa1006ecc0 msgr2=0x7ffa10071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.654+0000 7ffa2d2f0700 1 --2- 172.21.15.67:0/3379759156 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ffa1006ecc0 0x7ffa10071180 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7ffa18005950 tx=0x7ffa18005f50 comp rx=0 tx=0).stop 2024-04-22T23:54:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.654+0000 7ffa2d2f0700 1 -- 172.21.15.67:0/3379759156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa28101850 msgr2=0x7ffa280735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.654+0000 7ffa2d2f0700 1 --2- 172.21.15.67:0/3379759156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa28101850 0x7ffa280735d0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c00bda0 tx=0x7ffa1c00be80 comp rx=0 tx=0).stop 2024-04-22T23:54:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.655+0000 7ffa2d2f0700 1 -- 172.21.15.67:0/3379759156 shutdown_connections 2024-04-22T23:54:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.655+0000 7ffa2d2f0700 1 --2- 172.21.15.67:0/3379759156 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ffa1006ecc0 0x7ffa10071180 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.655+0000 7ffa2d2f0700 1 --2- 172.21.15.67:0/3379759156 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa28100ea0 0x7ffa28073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.655+0000 7ffa2d2f0700 1 --2- 172.21.15.67:0/3379759156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa28101850 0x7ffa280735d0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.655+0000 7ffa2d2f0700 1 -- 172.21.15.67:0/3379759156 >> 172.21.15.67:0/3379759156 conn(0x7ffa280fc750 msgr2=0x7ffa280fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:54:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.655+0000 7ffa2d2f0700 1 -- 172.21.15.67:0/3379759156 shutdown_connections 2024-04-22T23:54:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.655+0000 7ffa2d2f0700 1 -- 172.21.15.67:0/3379759156 wait complete. 2024-04-22T23:54:22.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.761+0000 7fcdb9be1700 1 -- 172.21.15.67:0/636149405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcdb40ff5a0 msgr2=0x7fcdb40ff980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:22.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.761+0000 7fcdb9be1700 1 --2- 172.21.15.67:0/636149405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcdb40ff5a0 0x7fcdb40ff980 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fcda4009910 tx=0x7fcda4009c20 comp rx=0 tx=0).stop 2024-04-22T23:54:22.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.761+0000 7fcdb9be1700 1 -- 172.21.15.67:0/636149405 shutdown_connections 2024-04-22T23:54:22.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.761+0000 7fcdb9be1700 1 --2- 172.21.15.67:0/636149405 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcdb40fff50 0x7fcdb4105840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:22.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.761+0000 7fcdb9be1700 1 --2- 172.21.15.67:0/636149405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcdb40ff5a0 0x7fcdb40ff980 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:22.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.761+0000 7fcdb9be1700 1 -- 172.21.15.67:0/636149405 >> 172.21.15.67:0/636149405 conn(0x7fcdb4074e30 msgr2=0x7fcdb4075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:54:22.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.761+0000 7fcdb9be1700 1 -- 172.21.15.67:0/636149405 shutdown_connections 2024-04-22T23:54:22.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.761+0000 7fcdb9be1700 1 -- 172.21.15.67:0/636149405 wait complete. 2024-04-22T23:54:22.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.762+0000 7fcdb9be1700 1 Processor -- start 2024-04-22T23:54:22.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.762+0000 7fcdb9be1700 1 -- start start 2024-04-22T23:54:22.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.762+0000 7fcdb9be1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcdb40ff5a0 0x7fcdb419c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:22.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.762+0000 7fcdb9be1700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcdb40fff50 0x7fcdb419cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:22.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.762+0000 7fcdb9be1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb419d2d0 con 0x7fcdb40ff5a0 2024-04-22T23:54:22.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.762+0000 7fcdb9be1700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb4196780 con 0x7fcdb40fff50 2024-04-22T23:54:22.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.762+0000 7fcdb37fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcdb40ff5a0 0x7fcdb419c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:22.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.762+0000 7fcdb37fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcdb40ff5a0 0x7fcdb419c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42326/0 (socket says 172.21.15.67:42326) 2024-04-22T23:54:22.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.762+0000 7fcdb37fe700 1 -- 172.21.15.67:0/2353229598 learned_addr learned my addr 172.21.15.67:0/2353229598 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:54:22.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.762+0000 7fcdb2ffd700 1 --2- 172.21.15.67:0/2353229598 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcdb40fff50 0x7fcdb419cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:22.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.763+0000 7fcdb37fe700 1 -- 172.21.15.67:0/2353229598 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcdb40fff50 msgr2=0x7fcdb419cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:22.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.763+0000 7fcdb37fe700 1 --2- 172.21.15.67:0/2353229598 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcdb40fff50 0x7fcdb419cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:22.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.763+0000 7fcdb37fe700 1 -- 172.21.15.67:0/2353229598 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda40095f0 con 0x7fcdb40ff5a0 2024-04-22T23:54:22.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.763+0000 7fcdb2ffd700 1 --2- 172.21.15.67:0/2353229598 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcdb40fff50 0x7fcdb419cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:54:22.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.763+0000 7fcdb37fe700 1 --2- 172.21.15.67:0/2353229598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcdb40ff5a0 0x7fcdb419c700 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fcda4006010 tx=0x7fcda400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:54:22.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.763+0000 7fcdb0ff9700 1 -- 172.21.15.67:0/2353229598 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcda401d070 con 0x7fcdb40ff5a0 2024-04-22T23:54:22.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.763+0000 7fcdb0ff9700 1 -- 172.21.15.67:0/2353229598 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcda4004fb0 con 0x7fcdb40ff5a0 2024-04-22T23:54:22.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.763+0000 7fcdb9be1700 1 -- 172.21.15.67:0/2353229598 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdb4196a00 con 0x7fcdb40ff5a0 2024-04-22T23:54:22.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.763+0000 7fcdb0ff9700 1 -- 172.21.15.67:0/2353229598 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcda4022ea0 con 0x7fcdb40ff5a0 2024-04-22T23:54:22.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.763+0000 7fcdb9be1700 1 -- 172.21.15.67:0/2353229598 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdb4196ef0 con 0x7fcdb40ff5a0 2024-04-22T23:54:22.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.764+0000 7fcdb9be1700 1 -- 172.21.15.67:0/2353229598 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdb4109270 con 0x7fcdb40ff5a0 2024-04-22T23:54:22.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.765+0000 7fcdb0ff9700 1 -- 172.21.15.67:0/2353229598 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcda4022470 con 0x7fcdb40ff5a0 2024-04-22T23:54:22.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.765+0000 7fcdb0ff9700 1 --2- 172.21.15.67:0/2353229598 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcd9c06ecc0 0x7fcd9c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:22.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.765+0000 7fcdb0ff9700 1 -- 172.21.15.67:0/2353229598 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fcda4090560 con 0x7fcdb40ff5a0 2024-04-22T23:54:22.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.765+0000 7fcdb2ffd700 1 --2- 172.21.15.67:0/2353229598 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcd9c06ecc0 0x7fcd9c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:22.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.766+0000 7fcdb2ffd700 1 --2- 172.21.15.67:0/2353229598 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcd9c06ecc0 0x7fcd9c071180 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fcda8005fd0 tx=0x7fcda8005dc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:54:22.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:22.770+0000 7fcdb0ff9700 1 -- 172.21.15.67:0/2353229598 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fcda405cd80 con 0x7fcdb40ff5a0 2024-04-22T23:54:23.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.068+0000 7fcdb9be1700 1 -- 172.21.15.67:0/2353229598 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fcdb4197d50 con 0x7fcdb40ff5a0 2024-04-22T23:54:23.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.068+0000 7fcdb0ff9700 1 -- 172.21.15.67:0/2353229598 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fcda4022720 con 0x7fcdb40ff5a0 2024-04-22T23:54:23.060 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:54:23.060 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:54:23.060 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:54:23.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:54:23.060 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:54:23.060 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:54:23.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:54:23.060 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:54:23.060 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:54:23.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:54:23.060 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:54:23.060 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 4 2024-04-22T23:54:23.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:54:23.061 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:54:23.061 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 14 2024-04-22T23:54:23.061 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:54:23.061 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:54:23.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.071+0000 7fcdb9be1700 1 -- 172.21.15.67:0/2353229598 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcd9c06ecc0 msgr2=0x7fcd9c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:23.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.071+0000 7fcdb9be1700 1 --2- 172.21.15.67:0/2353229598 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcd9c06ecc0 0x7fcd9c071180 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fcda8005fd0 tx=0x7fcda8005dc0 comp rx=0 tx=0).stop 2024-04-22T23:54:23.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.071+0000 7fcdb9be1700 1 -- 172.21.15.67:0/2353229598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcdb40ff5a0 msgr2=0x7fcdb419c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:23.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.071+0000 7fcdb9be1700 1 --2- 172.21.15.67:0/2353229598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcdb40ff5a0 0x7fcdb419c700 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fcda4006010 tx=0x7fcda400be20 comp rx=0 tx=0).stop 2024-04-22T23:54:23.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.071+0000 7fcdb9be1700 1 -- 172.21.15.67:0/2353229598 shutdown_connections 2024-04-22T23:54:23.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.071+0000 7fcdb9be1700 1 --2- 172.21.15.67:0/2353229598 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcd9c06ecc0 0x7fcd9c071180 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:23.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.071+0000 7fcdb9be1700 1 --2- 172.21.15.67:0/2353229598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcdb40ff5a0 0x7fcdb419c700 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:23.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.071+0000 7fcdb9be1700 1 --2- 172.21.15.67:0/2353229598 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcdb40fff50 0x7fcdb419cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:23.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.071+0000 7fcdb9be1700 1 -- 172.21.15.67:0/2353229598 >> 172.21.15.67:0/2353229598 conn(0x7fcdb4074e30 msgr2=0x7fcdb40fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:54:23.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.071+0000 7fcdb9be1700 1 -- 172.21.15.67:0/2353229598 shutdown_connections 2024-04-22T23:54:23.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.071+0000 7fcdb9be1700 1 -- 172.21.15.67:0/2353229598 wait complete. 2024-04-22T23:54:23.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.178+0000 7f7d5e1d2700 1 -- 172.21.15.67:0/4147472237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d58102720 msgr2=0x7f7d5810ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:23.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.178+0000 7f7d5e1d2700 1 --2- 172.21.15.67:0/4147472237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d58102720 0x7f7d5810ac10 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c009910 tx=0x7f7d4c009c20 comp rx=0 tx=0).stop 2024-04-22T23:54:23.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.179+0000 7f7d5e1d2700 1 -- 172.21.15.67:0/4147472237 shutdown_connections 2024-04-22T23:54:23.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.179+0000 7f7d5e1d2700 1 --2- 172.21.15.67:0/4147472237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d58102720 0x7f7d5810ac10 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:23.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.179+0000 7f7d5e1d2700 1 --2- 172.21.15.67:0/4147472237 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d58101e00 0x7f7d581021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:23.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.179+0000 7f7d5e1d2700 1 -- 172.21.15.67:0/4147472237 >> 172.21.15.67:0/4147472237 conn(0x7f7d580fb650 msgr2=0x7f7d580fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:54:23.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.179+0000 7f7d5e1d2700 1 -- 172.21.15.67:0/4147472237 shutdown_connections 2024-04-22T23:54:23.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.179+0000 7f7d5e1d2700 1 -- 172.21.15.67:0/4147472237 wait complete. 2024-04-22T23:54:23.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.179+0000 7f7d5e1d2700 1 Processor -- start 2024-04-22T23:54:23.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.179+0000 7f7d5e1d2700 1 -- start start 2024-04-22T23:54:23.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.180+0000 7f7d5e1d2700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d58101e00 0x7f7d5819c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:23.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.180+0000 7f7d5e1d2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d58102720 0x7f7d5819cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:23.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.180+0000 7f7d5e1d2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5819d260 con 0x7f7d58102720 2024-04-22T23:54:23.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.180+0000 7f7d5e1d2700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d58196710 con 0x7f7d58101e00 2024-04-22T23:54:23.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.180+0000 7f7d56ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d58102720 0x7f7d5819cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:23.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.180+0000 7f7d56ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d58102720 0x7f7d5819cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42340/0 (socket says 172.21.15.67:42340) 2024-04-22T23:54:23.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.180+0000 7f7d56ffd700 1 -- 172.21.15.67:0/1956183499 learned_addr learned my addr 172.21.15.67:0/1956183499 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:54:23.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.180+0000 7f7d577fe700 1 --2- 172.21.15.67:0/1956183499 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d58101e00 0x7f7d5819c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:23.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.180+0000 7f7d56ffd700 1 -- 172.21.15.67:0/1956183499 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d58101e00 msgr2=0x7f7d5819c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:23.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.180+0000 7f7d56ffd700 1 --2- 172.21.15.67:0/1956183499 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d58101e00 0x7f7d5819c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:23.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.180+0000 7f7d56ffd700 1 -- 172.21.15.67:0/1956183499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d4c0095f0 con 0x7f7d58102720 2024-04-22T23:54:23.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.180+0000 7f7d56ffd700 1 --2- 172.21.15.67:0/1956183499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d58102720 0x7f7d5819cbd0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c00bd70 tx=0x7f7d4c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:54:23.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.180+0000 7f7d54ff9700 1 -- 172.21.15.67:0/1956183499 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d4c01d070 con 0x7f7d58102720 2024-04-22T23:54:23.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.180+0000 7f7d54ff9700 1 -- 172.21.15.67:0/1956183499 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7d4c00f460 con 0x7f7d58102720 2024-04-22T23:54:23.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.180+0000 7f7d5e1d2700 1 -- 172.21.15.67:0/1956183499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d58196990 con 0x7f7d58102720 2024-04-22T23:54:23.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.181+0000 7f7d54ff9700 1 -- 172.21.15.67:0/1956183499 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d4c017630 con 0x7f7d58102720 2024-04-22T23:54:23.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.181+0000 7f7d5e1d2700 1 -- 172.21.15.67:0/1956183499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d58196e80 con 0x7f7d58102720 2024-04-22T23:54:23.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.181+0000 7f7d5e1d2700 1 -- 172.21.15.67:0/1956183499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d58108310 con 0x7f7d58102720 2024-04-22T23:54:23.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.182+0000 7f7d54ff9700 1 -- 172.21.15.67:0/1956183499 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7d4c00f5d0 con 0x7f7d58102720 2024-04-22T23:54:23.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.182+0000 7f7d54ff9700 1 --2- 172.21.15.67:0/1956183499 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7d4006ecc0 0x7f7d40071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:23.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.183+0000 7f7d54ff9700 1 -- 172.21.15.67:0/1956183499 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f7d4c090590 con 0x7f7d58102720 2024-04-22T23:54:23.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.183+0000 7f7d577fe700 1 --2- 172.21.15.67:0/1956183499 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7d4006ecc0 0x7f7d40071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:23.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.183+0000 7f7d577fe700 1 --2- 172.21.15.67:0/1956183499 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7d4006ecc0 0x7f7d40071180 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f7d48005fd0 tx=0x7f7d4800a560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:54:23.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.187+0000 7f7d54ff9700 1 -- 172.21.15.67:0/1956183499 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7d4c05cd00 con 0x7f7d58102720 2024-04-22T23:54:23.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.448+0000 7f7d5e1d2700 1 -- 172.21.15.67:0/1956183499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f7d58197cc0 con 0x7f7d58102720 2024-04-22T23:54:23.440 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:23 smithi067 ceph-mon[112438]: from='client.35524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:54:23.440 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:23 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2353229598' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:54:23.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.449+0000 7f7d54ff9700 1 -- 172.21.15.67:0/1956183499 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1872 (secure 0 0 0) 0x7f7d4c05c890 con 0x7f7d58102720 2024-04-22T23:54:23.440 INFO:teuthology.orchestra.run.smithi067.stdout:e33 2024-04-22T23:54:23.440 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:54:23.441 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:54:23.441 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:54:23.441 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:54:23.441 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:54:23.441 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:54:23.441 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:54:23.441 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:54:23.441 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:54:23.441 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:54:23.441 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:54:23.441 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:54:23.442 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:54:23.442 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:54:23.442 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:54:23.442 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:54:23.442 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:54:23.442 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:54:23.442 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:54:23.442 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:54:23.442 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:54:23.442 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:54:23.442 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:54:23.442 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:54:23.442 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:54:23.443 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:54:23.443 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:54:23.443 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:54:23.443 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:54:23.443 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:54:23.443 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:54:23.443 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:54:23.443 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:54:23.443 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:54:23.443 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:54:23.443 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:54:23.443 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:54:23.443 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:54:23.443 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:54:23.443 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:54:23.443 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44851} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1849648622,v1:172.21.15.134:6827/1849648622] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:54:23.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.451+0000 7f7d5e1d2700 1 -- 172.21.15.67:0/1956183499 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7d4006ecc0 msgr2=0x7f7d40071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:23.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.451+0000 7f7d5e1d2700 1 --2- 172.21.15.67:0/1956183499 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7d4006ecc0 0x7f7d40071180 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f7d48005fd0 tx=0x7f7d4800a560 comp rx=0 tx=0).stop 2024-04-22T23:54:23.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.451+0000 7f7d5e1d2700 1 -- 172.21.15.67:0/1956183499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d58102720 msgr2=0x7f7d5819cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:23.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.451+0000 7f7d5e1d2700 1 --2- 172.21.15.67:0/1956183499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d58102720 0x7f7d5819cbd0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c00bd70 tx=0x7f7d4c00be50 comp rx=0 tx=0).stop 2024-04-22T23:54:23.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.451+0000 7f7d5e1d2700 1 -- 172.21.15.67:0/1956183499 shutdown_connections 2024-04-22T23:54:23.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.451+0000 7f7d5e1d2700 1 --2- 172.21.15.67:0/1956183499 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7d4006ecc0 0x7f7d40071180 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:23.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.451+0000 7f7d5e1d2700 1 --2- 172.21.15.67:0/1956183499 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d58101e00 0x7f7d5819c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:23.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.451+0000 7f7d5e1d2700 1 --2- 172.21.15.67:0/1956183499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d58102720 0x7f7d5819cbd0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:23.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.451+0000 7f7d5e1d2700 1 -- 172.21.15.67:0/1956183499 >> 172.21.15.67:0/1956183499 conn(0x7f7d580fb650 msgr2=0x7f7d58105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:54:23.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.451+0000 7f7d5e1d2700 1 -- 172.21.15.67:0/1956183499 shutdown_connections 2024-04-22T23:54:23.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.451+0000 7f7d5e1d2700 1 -- 172.21.15.67:0/1956183499 wait complete. 2024-04-22T23:54:23.445 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 33 2024-04-22T23:54:23.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:23 smithi134 ceph-mon[85630]: from='client.35524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:54:23.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:23 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2353229598' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:54:23.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.561+0000 7f198f5b5700 1 -- 172.21.15.67:0/3164557489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1988101850 msgr2=0x7f19881058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:23.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.561+0000 7f198f5b5700 1 --2- 172.21.15.67:0/3164557489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1988101850 0x7f19881058a0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f197c009910 tx=0x7f197c009c20 comp rx=0 tx=0).stop 2024-04-22T23:54:23.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.561+0000 7f198f5b5700 1 -- 172.21.15.67:0/3164557489 shutdown_connections 2024-04-22T23:54:23.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.562+0000 7f198f5b5700 1 --2- 172.21.15.67:0/3164557489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1988101850 0x7f19881058a0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:23.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.562+0000 7f198f5b5700 1 --2- 172.21.15.67:0/3164557489 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1988100ea0 0x7f1988101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:23.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.562+0000 7f198f5b5700 1 -- 172.21.15.67:0/3164557489 >> 172.21.15.67:0/3164557489 conn(0x7f19880fc750 msgr2=0x7f19880feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:54:23.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.562+0000 7f198f5b5700 1 -- 172.21.15.67:0/3164557489 shutdown_connections 2024-04-22T23:54:23.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.562+0000 7f198f5b5700 1 -- 172.21.15.67:0/3164557489 wait complete. 2024-04-22T23:54:23.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.562+0000 7f198f5b5700 1 Processor -- start 2024-04-22T23:54:23.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.562+0000 7f198f5b5700 1 -- start start 2024-04-22T23:54:23.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.562+0000 7f198f5b5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1988100ea0 0x7f1988194660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:23.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.562+0000 7f198f5b5700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1988101850 0x7f1988194ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:23.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.562+0000 7f198f5b5700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1988195280 con 0x7f1988100ea0 2024-04-22T23:54:23.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.562+0000 7f198f5b5700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1988199010 con 0x7f1988101850 2024-04-22T23:54:23.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.563+0000 7f198d351700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1988100ea0 0x7f1988194660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:23.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.563+0000 7f198d351700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1988100ea0 0x7f1988194660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42356/0 (socket says 172.21.15.67:42356) 2024-04-22T23:54:23.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.563+0000 7f198d351700 1 -- 172.21.15.67:0/3321981945 learned_addr learned my addr 172.21.15.67:0/3321981945 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:54:23.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.563+0000 7f198cb50700 1 --2- 172.21.15.67:0/3321981945 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1988101850 0x7f1988194ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:23.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.563+0000 7f198d351700 1 -- 172.21.15.67:0/3321981945 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1988101850 msgr2=0x7f1988194ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:23.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.563+0000 7f198d351700 1 --2- 172.21.15.67:0/3321981945 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1988101850 0x7f1988194ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:23.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.563+0000 7f198d351700 1 -- 172.21.15.67:0/3321981945 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f197c0095f0 con 0x7f1988100ea0 2024-04-22T23:54:23.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.563+0000 7f198d351700 1 --2- 172.21.15.67:0/3321981945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1988100ea0 0x7f1988194660 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f198400d8d0 tx=0x7f198400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:54:23.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.563+0000 7f197a7fc700 1 -- 172.21.15.67:0/3321981945 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19840098a0 con 0x7f1988100ea0 2024-04-22T23:54:23.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.563+0000 7f198f5b5700 1 -- 172.21.15.67:0/3321981945 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19881992f0 con 0x7f1988100ea0 2024-04-22T23:54:23.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.563+0000 7f197a7fc700 1 -- 172.21.15.67:0/3321981945 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1984010460 con 0x7f1988100ea0 2024-04-22T23:54:23.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.563+0000 7f197a7fc700 1 -- 172.21.15.67:0/3321981945 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f198400f5f0 con 0x7f1988100ea0 2024-04-22T23:54:23.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.563+0000 7f198f5b5700 1 -- 172.21.15.67:0/3321981945 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1988199810 con 0x7f1988100ea0 2024-04-22T23:54:23.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.564+0000 7f198f5b5700 1 -- 172.21.15.67:0/3321981945 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19881092a0 con 0x7f1988100ea0 2024-04-22T23:54:23.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.565+0000 7f197a7fc700 1 -- 172.21.15.67:0/3321981945 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1984009a00 con 0x7f1988100ea0 2024-04-22T23:54:23.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.565+0000 7f197a7fc700 1 --2- 172.21.15.67:0/3321981945 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f19740730d0 0x7f1974075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:23.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.565+0000 7f197a7fc700 1 -- 172.21.15.67:0/3321981945 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f198408eea0 con 0x7f1988100ea0 2024-04-22T23:54:23.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.565+0000 7f198cb50700 1 --2- 172.21.15.67:0/3321981945 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f19740730d0 0x7f1974075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:23.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.566+0000 7f198cb50700 1 --2- 172.21.15.67:0/3321981945 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f19740730d0 0x7f1974075590 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f197c006010 tx=0x7f197c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:54:23.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.571+0000 7f197a7fc700 1 -- 172.21.15.67:0/3321981945 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1984057ae0 con 0x7f1988100ea0 2024-04-22T23:54:23.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.808+0000 7f198f5b5700 1 -- 172.21.15.67:0/3321981945 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1988199a30 con 0x7f19740730d0 2024-04-22T23:54:23.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.809+0000 7f197a7fc700 1 -- 172.21.15.67:0/3321981945 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f1988199a30 con 0x7f19740730d0 2024-04-22T23:54:23.800 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:54:23.800 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:54:23.800 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:54:23.800 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:54:23.800 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:54:23.800 INFO:teuthology.orchestra.run.smithi067.stdout: "mds", 2024-04-22T23:54:23.800 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:54:23.800 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:54:23.800 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:54:23.800 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:54:23.801 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:54:23.801 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T23:54:23.801 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:54:23.801 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:54:23.801 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:54:23.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.811+0000 7f198f5b5700 1 -- 172.21.15.67:0/3321981945 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f19740730d0 msgr2=0x7f1974075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:23.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.811+0000 7f198f5b5700 1 --2- 172.21.15.67:0/3321981945 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f19740730d0 0x7f1974075590 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f197c006010 tx=0x7f197c01a040 comp rx=0 tx=0).stop 2024-04-22T23:54:23.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.811+0000 7f198f5b5700 1 -- 172.21.15.67:0/3321981945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1988100ea0 msgr2=0x7f1988194660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:23.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.811+0000 7f198f5b5700 1 --2- 172.21.15.67:0/3321981945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1988100ea0 0x7f1988194660 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f198400d8d0 tx=0x7f198400dbe0 comp rx=0 tx=0).stop 2024-04-22T23:54:23.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.811+0000 7f198f5b5700 1 -- 172.21.15.67:0/3321981945 shutdown_connections 2024-04-22T23:54:23.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.811+0000 7f198f5b5700 1 --2- 172.21.15.67:0/3321981945 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f19740730d0 0x7f1974075590 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:23.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.811+0000 7f198f5b5700 1 --2- 172.21.15.67:0/3321981945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1988100ea0 0x7f1988194660 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:23.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.811+0000 7f198f5b5700 1 --2- 172.21.15.67:0/3321981945 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1988101850 0x7f1988194ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:23.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.811+0000 7f198f5b5700 1 -- 172.21.15.67:0/3321981945 >> 172.21.15.67:0/3321981945 conn(0x7f19880fc750 msgr2=0x7f19880feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:54:23.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.811+0000 7f198f5b5700 1 -- 172.21.15.67:0/3321981945 shutdown_connections 2024-04-22T23:54:23.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.811+0000 7f198f5b5700 1 -- 172.21.15.67:0/3321981945 wait complete. 2024-04-22T23:54:23.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.918+0000 7f12363e3700 1 -- 172.21.15.67:0/812864250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12300ffc20 msgr2=0x7f1230100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:23.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.918+0000 7f12363e3700 1 --2- 172.21.15.67:0/812864250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12300ffc20 0x7f1230100080 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f1220009910 tx=0x7f1220009c20 comp rx=0 tx=0).stop 2024-04-22T23:54:23.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.918+0000 7f12363e3700 1 -- 172.21.15.67:0/812864250 shutdown_connections 2024-04-22T23:54:23.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.918+0000 7f12363e3700 1 --2- 172.21.15.67:0/812864250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12300ffc20 0x7f1230100080 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:23.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.918+0000 7f12363e3700 1 --2- 172.21.15.67:0/812864250 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1230105bd0 0x7f1230105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:23.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.918+0000 7f12363e3700 1 -- 172.21.15.67:0/812864250 >> 172.21.15.67:0/812864250 conn(0x7f1230075670 msgr2=0x7f1230075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:54:23.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.918+0000 7f12363e3700 1 -- 172.21.15.67:0/812864250 shutdown_connections 2024-04-22T23:54:23.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.918+0000 7f12363e3700 1 -- 172.21.15.67:0/812864250 wait complete. 2024-04-22T23:54:23.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.919+0000 7f12363e3700 1 Processor -- start 2024-04-22T23:54:23.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.919+0000 7f12363e3700 1 -- start start 2024-04-22T23:54:23.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.919+0000 7f12363e3700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12300ffc20 0x7f1230197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:23.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.919+0000 7f12363e3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1230105bd0 0x7f1230197c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:23.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.919+0000 7f12363e3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12301982d0 con 0x7f1230105bd0 2024-04-22T23:54:23.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.919+0000 7f12363e3700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1230198440 con 0x7f12300ffc20 2024-04-22T23:54:23.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.919+0000 7f122f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1230105bd0 0x7f1230197c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:23.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.919+0000 7f122f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1230105bd0 0x7f1230197c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42370/0 (socket says 172.21.15.67:42370) 2024-04-22T23:54:23.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.919+0000 7f122f7fe700 1 -- 172.21.15.67:0/4173848216 learned_addr learned my addr 172.21.15.67:0/4173848216 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:54:23.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.919+0000 7f122f7fe700 1 -- 172.21.15.67:0/4173848216 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12300ffc20 msgr2=0x7f1230197740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:23.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.919+0000 7f122f7fe700 1 --2- 172.21.15.67:0/4173848216 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12300ffc20 0x7f1230197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:23.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.920+0000 7f122f7fe700 1 -- 172.21.15.67:0/4173848216 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12200095f0 con 0x7f1230105bd0 2024-04-22T23:54:23.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.920+0000 7f122f7fe700 1 --2- 172.21.15.67:0/4173848216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1230105bd0 0x7f1230197c80 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f1220006010 tx=0x7f122000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:54:23.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.920+0000 7f122d7fa700 1 -- 172.21.15.67:0/4173848216 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f122001d070 con 0x7f1230105bd0 2024-04-22T23:54:23.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.920+0000 7f12363e3700 1 -- 172.21.15.67:0/4173848216 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12301a07b0 con 0x7f1230105bd0 2024-04-22T23:54:23.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.920+0000 7f122d7fa700 1 -- 172.21.15.67:0/4173848216 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1220004fb0 con 0x7f1230105bd0 2024-04-22T23:54:23.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.920+0000 7f122d7fa700 1 -- 172.21.15.67:0/4173848216 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1220022ea0 con 0x7f1230105bd0 2024-04-22T23:54:23.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.920+0000 7f12363e3700 1 -- 172.21.15.67:0/4173848216 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12301a0bc0 con 0x7f1230105bd0 2024-04-22T23:54:23.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.921+0000 7f12363e3700 1 -- 172.21.15.67:0/4173848216 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12301080d0 con 0x7f1230105bd0 2024-04-22T23:54:23.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.922+0000 7f122d7fa700 1 -- 172.21.15.67:0/4173848216 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1220022470 con 0x7f1230105bd0 2024-04-22T23:54:23.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.922+0000 7f122d7fa700 1 --2- 172.21.15.67:0/4173848216 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f121c06ecc0 0x7f121c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:23.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.922+0000 7f122d7fa700 1 -- 172.21.15.67:0/4173848216 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f1220090590 con 0x7f1230105bd0 2024-04-22T23:54:23.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.922+0000 7f122ffff700 1 --2- 172.21.15.67:0/4173848216 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f121c06ecc0 0x7f121c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:23.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.923+0000 7f122ffff700 1 --2- 172.21.15.67:0/4173848216 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f121c06ecc0 0x7f121c071180 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f1218005fd0 tx=0x7f1218005d00 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:54:23.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:23.927+0000 7f122d7fa700 1 -- 172.21.15.67:0/4173848216 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1220059280 con 0x7f1230105bd0 2024-04-22T23:54:24.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:24.226+0000 7f12363e3700 1 -- 172.21.15.67:0/4173848216 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f123004f2e0 con 0x7f1230105bd0 2024-04-22T23:54:24.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:24.227+0000 7f122d7fa700 1 -- 172.21.15.67:0/4173848216 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4505 (secure 0 0 0) 0x7f1220022720 con 0x7f1230105bd0 2024-04-22T23:54:24.218 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:54:24.218 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T23:54:24.218 INFO:teuthology.orchestra.run.smithi067.stdout: host smithi134 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-22T23:54:24.218 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T23:54:24.218 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:54:24.218 INFO:teuthology.orchestra.run.smithi067.stdout:Traceback (most recent call last): 2024-04-22T23:54:24.218 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T23:54:24.218 INFO:teuthology.orchestra.run.smithi067.stdout: return _run_code(code, main_globals, None, 2024-04-22T23:54:24.219 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T23:54:24.219 INFO:teuthology.orchestra.run.smithi067.stdout: exec(code, run_globals) 2024-04-22T23:54:24.219 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5642, in 2024-04-22T23:54:24.219 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5630, in main 2024-04-22T23:54:24.219 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 408, in _infer_config 2024-04-22T23:54:24.219 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 323, in _infer_fsid 2024-04-22T23:54:24.219 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 436, in _infer_image 2024-04-22T23:54:24.219 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 310, in _validate_fsid 2024-04-22T23:54:24.219 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 3380, in command_ceph_volume 2024-04-22T23:54:24.219 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T23:54:24.219 INFO:teuthology.orchestra.run.smithi067.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:54:24.219 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:54:24.219 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:54:24.219 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:54:24.220 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:54:24.220 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:54:24.220 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:54:24.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:24.229+0000 7f12363e3700 1 -- 172.21.15.67:0/4173848216 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f121c06ecc0 msgr2=0x7f121c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:24.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:24.229+0000 7f12363e3700 1 --2- 172.21.15.67:0/4173848216 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f121c06ecc0 0x7f121c071180 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f1218005fd0 tx=0x7f1218005d00 comp rx=0 tx=0).stop 2024-04-22T23:54:24.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:24.229+0000 7f12363e3700 1 -- 172.21.15.67:0/4173848216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1230105bd0 msgr2=0x7f1230197c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:24.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:24.229+0000 7f12363e3700 1 --2- 172.21.15.67:0/4173848216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1230105bd0 0x7f1230197c80 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f1220006010 tx=0x7f122000be20 comp rx=0 tx=0).stop 2024-04-22T23:54:24.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:24.229+0000 7f12363e3700 1 -- 172.21.15.67:0/4173848216 shutdown_connections 2024-04-22T23:54:24.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:24.229+0000 7f12363e3700 1 --2- 172.21.15.67:0/4173848216 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f121c06ecc0 0x7f121c071180 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:24.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:24.229+0000 7f12363e3700 1 --2- 172.21.15.67:0/4173848216 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12300ffc20 0x7f1230197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:24.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:24.229+0000 7f12363e3700 1 --2- 172.21.15.67:0/4173848216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1230105bd0 0x7f1230197c80 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:24.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:24.229+0000 7f12363e3700 1 -- 172.21.15.67:0/4173848216 >> 172.21.15.67:0/4173848216 conn(0x7f1230075670 msgr2=0x7f12300fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:54:24.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:24.229+0000 7f12363e3700 1 -- 172.21.15.67:0/4173848216 shutdown_connections 2024-04-22T23:54:24.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:24.229+0000 7f12363e3700 1 -- 172.21.15.67:0/4173848216 wait complete. 2024-04-22T23:54:24.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:24 smithi134 ceph-mon[85630]: from='client.35528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:54:24.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:24 smithi134 ceph-mon[85630]: from='client.35532 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:54:24.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:24 smithi134 ceph-mon[85630]: pgmap v842: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:24.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:24 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1956183499' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:54:24.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:24 smithi134 ceph-mon[85630]: from='client.35544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:54:24.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:24 smithi067 ceph-mon[112438]: from='client.35528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:54:24.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:24 smithi067 ceph-mon[112438]: from='client.35532 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:54:24.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:24 smithi067 ceph-mon[112438]: pgmap v842: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:24.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:24 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1956183499' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:54:24.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:24 smithi067 ceph-mon[112438]: from='client.35544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:54:25.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:25 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/4173848216' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:54:25.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:25 smithi134 ceph-mon[85630]: pgmap v843: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:25.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:25 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/4173848216' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:54:25.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:25 smithi067 ceph-mon[112438]: pgmap v843: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:26.379 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:54:26.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:54:27.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:27 smithi134 ceph-mon[85630]: pgmap v844: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:27.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:27 smithi067 ceph-mon[112438]: pgmap v844: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:30.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:30 smithi134 ceph-mon[85630]: pgmap v845: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:30.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:30 smithi067 ceph-mon[112438]: pgmap v845: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:32.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:32 smithi134 ceph-mon[85630]: pgmap v846: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:32.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:32 smithi067 ceph-mon[112438]: pgmap v846: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:34.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:34 smithi134 ceph-mon[85630]: pgmap v847: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:34.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:34 smithi067 ceph-mon[112438]: pgmap v847: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:36.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:36 smithi134 ceph-mon[85630]: pgmap v848: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:36.487 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:36 smithi067 ceph-mon[112438]: pgmap v848: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:37.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:37 smithi134 ceph-mon[85630]: pgmap v849: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:37.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:37 smithi067 ceph-mon[112438]: pgmap v849: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:40.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:40 smithi134 ceph-mon[85630]: pgmap v850: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:40.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:40 smithi067 ceph-mon[112438]: pgmap v850: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:41.234 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:54:41.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:54:42.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:42 smithi134 ceph-mon[85630]: pgmap v851: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:42.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:42 smithi067 ceph-mon[112438]: pgmap v851: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:43.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:43 smithi134 ceph-mon[85630]: pgmap v852: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:43.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:43 smithi067 ceph-mon[112438]: pgmap v852: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:46.234 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:46 smithi134 ceph-mon[85630]: pgmap v853: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:46.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:46 smithi067 ceph-mon[112438]: pgmap v853: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:48.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:48 smithi134 ceph-mon[85630]: pgmap v854: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:48.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:48 smithi067 ceph-mon[112438]: pgmap v854: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:49.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:49 smithi134 ceph-mon[85630]: pgmap v855: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:49.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:49 smithi067 ceph-mon[112438]: pgmap v855: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:52.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:52 smithi134 ceph-mon[85630]: pgmap v856: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:52.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:52 smithi067 ceph-mon[112438]: pgmap v856: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:54.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.350+0000 7f83f03f2700 1 -- 172.21.15.67:0/1698227700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83e8100ea0 msgr2=0x7f83e8101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:54.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.350+0000 7f83f03f2700 1 --2- 172.21.15.67:0/1698227700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83e8100ea0 0x7f83e8101280 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f83e4009910 tx=0x7f83e4009c20 comp rx=0 tx=0).stop 2024-04-22T23:54:54.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.350+0000 7f83f03f2700 1 -- 172.21.15.67:0/1698227700 shutdown_connections 2024-04-22T23:54:54.342 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:54 smithi067 ceph-mon[112438]: pgmap v857: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:54.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.350+0000 7f83f03f2700 1 --2- 172.21.15.67:0/1698227700 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f83e8101850 0x7f83e81058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:54.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.350+0000 7f83f03f2700 1 --2- 172.21.15.67:0/1698227700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83e8100ea0 0x7f83e8101280 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:54.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.350+0000 7f83f03f2700 1 -- 172.21.15.67:0/1698227700 >> 172.21.15.67:0/1698227700 conn(0x7f83e80fc750 msgr2=0x7f83e80feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:54:54.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.350+0000 7f83f03f2700 1 -- 172.21.15.67:0/1698227700 shutdown_connections 2024-04-22T23:54:54.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.350+0000 7f83f03f2700 1 -- 172.21.15.67:0/1698227700 wait complete. 2024-04-22T23:54:54.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.351+0000 7f83f03f2700 1 Processor -- start 2024-04-22T23:54:54.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.351+0000 7f83f03f2700 1 -- start start 2024-04-22T23:54:54.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.351+0000 7f83f03f2700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f83e8100ea0 0x7f83e8073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:54.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.351+0000 7f83f03f2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83e8101850 0x7f83e80735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:54.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.351+0000 7f83f03f2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e8073b10 con 0x7f83e8101850 2024-04-22T23:54:54.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.351+0000 7f83f03f2700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e8073c50 con 0x7f83e8100ea0 2024-04-22T23:54:54.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.351+0000 7f83ed98d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83e8101850 0x7f83e80735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:54.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.351+0000 7f83ed98d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83e8101850 0x7f83e80735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45486/0 (socket says 172.21.15.67:45486) 2024-04-22T23:54:54.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.351+0000 7f83ed98d700 1 -- 172.21.15.67:0/344215030 learned_addr learned my addr 172.21.15.67:0/344215030 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:54:54.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.351+0000 7f83ee18e700 1 --2- 172.21.15.67:0/344215030 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f83e8100ea0 0x7f83e8073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:54.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.351+0000 7f83ed98d700 1 -- 172.21.15.67:0/344215030 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f83e8100ea0 msgr2=0x7f83e8073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:54.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.351+0000 7f83ed98d700 1 --2- 172.21.15.67:0/344215030 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f83e8100ea0 0x7f83e8073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:54.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.351+0000 7f83ed98d700 1 -- 172.21.15.67:0/344215030 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e40095f0 con 0x7f83e8101850 2024-04-22T23:54:54.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.352+0000 7f83ed98d700 1 --2- 172.21.15.67:0/344215030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83e8101850 0x7f83e80735d0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f83d800d900 tx=0x7f83d800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:54:54.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.352+0000 7f83df7fe700 1 -- 172.21.15.67:0/344215030 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83d8009900 con 0x7f83e8101850 2024-04-22T23:54:54.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.352+0000 7f83f03f2700 1 -- 172.21.15.67:0/344215030 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83e81a2af0 con 0x7f83e8101850 2024-04-22T23:54:54.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.352+0000 7f83df7fe700 1 -- 172.21.15.67:0/344215030 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f83d8010460 con 0x7f83e8101850 2024-04-22T23:54:54.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.352+0000 7f83df7fe700 1 -- 172.21.15.67:0/344215030 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83d800f5f0 con 0x7f83e8101850 2024-04-22T23:54:54.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.352+0000 7f83f03f2700 1 -- 172.21.15.67:0/344215030 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83e81a2f00 con 0x7f83e8101850 2024-04-22T23:54:54.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.353+0000 7f83f03f2700 1 -- 172.21.15.67:0/344215030 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83e8109240 con 0x7f83e8101850 2024-04-22T23:54:54.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.353+0000 7f83df7fe700 1 -- 172.21.15.67:0/344215030 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f83d80105d0 con 0x7f83e8101850 2024-04-22T23:54:54.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.354+0000 7f83df7fe700 1 --2- 172.21.15.67:0/344215030 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f83d406ecc0 0x7f83d4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:54.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.354+0000 7f83df7fe700 1 -- 172.21.15.67:0/344215030 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f83d808eb70 con 0x7f83e8101850 2024-04-22T23:54:54.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.354+0000 7f83ee18e700 1 --2- 172.21.15.67:0/344215030 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f83d406ecc0 0x7f83d4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:54.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.354+0000 7f83ee18e700 1 --2- 172.21.15.67:0/344215030 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f83d406ecc0 0x7f83d4071180 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f83e4006010 tx=0x7f83e401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:54:54.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.359+0000 7f83df7fe700 1 -- 172.21.15.67:0/344215030 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f83d80577b0 con 0x7f83e8101850 2024-04-22T23:54:54.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:54 smithi134 ceph-mon[85630]: pgmap v857: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:54.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.590+0000 7f83f03f2700 1 -- 172.21.15.67:0/344215030 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f83e80611d0 con 0x7f83d406ecc0 2024-04-22T23:54:54.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.591+0000 7f83df7fe700 1 -- 172.21.15.67:0/344215030 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f83e80611d0 con 0x7f83d406ecc0 2024-04-22T23:54:54.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.593+0000 7f83f03f2700 1 -- 172.21.15.67:0/344215030 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f83d406ecc0 msgr2=0x7f83d4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:54.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.593+0000 7f83f03f2700 1 --2- 172.21.15.67:0/344215030 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f83d406ecc0 0x7f83d4071180 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f83e4006010 tx=0x7f83e401a040 comp rx=0 tx=0).stop 2024-04-22T23:54:54.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.593+0000 7f83f03f2700 1 -- 172.21.15.67:0/344215030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83e8101850 msgr2=0x7f83e80735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:54.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.593+0000 7f83f03f2700 1 --2- 172.21.15.67:0/344215030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83e8101850 0x7f83e80735d0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f83d800d900 tx=0x7f83d800dc10 comp rx=0 tx=0).stop 2024-04-22T23:54:54.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.594+0000 7f83f03f2700 1 -- 172.21.15.67:0/344215030 shutdown_connections 2024-04-22T23:54:54.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.594+0000 7f83f03f2700 1 --2- 172.21.15.67:0/344215030 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f83d406ecc0 0x7f83d4071180 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:54.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.594+0000 7f83f03f2700 1 --2- 172.21.15.67:0/344215030 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f83e8100ea0 0x7f83e8073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:54.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.594+0000 7f83f03f2700 1 --2- 172.21.15.67:0/344215030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83e8101850 0x7f83e80735d0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:54.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.594+0000 7f83f03f2700 1 -- 172.21.15.67:0/344215030 >> 172.21.15.67:0/344215030 conn(0x7f83e80fc750 msgr2=0x7f83e80fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:54:54.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.594+0000 7f83f03f2700 1 -- 172.21.15.67:0/344215030 shutdown_connections 2024-04-22T23:54:54.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.594+0000 7f83f03f2700 1 -- 172.21.15.67:0/344215030 wait complete. 2024-04-22T23:54:54.595 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:54:54.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.701+0000 7f01b0de2700 1 -- 172.21.15.67:0/1415488319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01ac106cd0 msgr2=0x7f01ac1070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:54.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.701+0000 7f01b0de2700 1 --2- 172.21.15.67:0/1415488319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01ac106cd0 0x7f01ac1070b0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f019c00b0a0 tx=0x7f019c00b3b0 comp rx=0 tx=0).stop 2024-04-22T23:54:54.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.701+0000 7f01b0de2700 1 -- 172.21.15.67:0/1415488319 shutdown_connections 2024-04-22T23:54:54.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.701+0000 7f01b0de2700 1 --2- 172.21.15.67:0/1415488319 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f01ac068580 0x7f01ac0689e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:54.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.701+0000 7f01b0de2700 1 --2- 172.21.15.67:0/1415488319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01ac106cd0 0x7f01ac1070b0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:54.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.701+0000 7f01b0de2700 1 -- 172.21.15.67:0/1415488319 >> 172.21.15.67:0/1415488319 conn(0x7f01ac075670 msgr2=0x7f01ac075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:54:54.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.701+0000 7f01b0de2700 1 -- 172.21.15.67:0/1415488319 shutdown_connections 2024-04-22T23:54:54.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.701+0000 7f01b0de2700 1 -- 172.21.15.67:0/1415488319 wait complete. 2024-04-22T23:54:54.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.702+0000 7f01b0de2700 1 Processor -- start 2024-04-22T23:54:54.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.702+0000 7f01b0de2700 1 -- start start 2024-04-22T23:54:54.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.702+0000 7f01b0de2700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f01ac068580 0x7f01ac0ffcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:54.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.702+0000 7f01b0de2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01ac106cd0 0x7f01ac100230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:54.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.702+0000 7f01b0de2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ac100770 con 0x7f01ac106cd0 2024-04-22T23:54:54.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.702+0000 7f01b0de2700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ac1008b0 con 0x7f01ac068580 2024-04-22T23:54:54.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.702+0000 7f01a9d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01ac106cd0 0x7f01ac100230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:54.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.702+0000 7f01a9d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01ac106cd0 0x7f01ac100230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45502/0 (socket says 172.21.15.67:45502) 2024-04-22T23:54:54.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.702+0000 7f01a9d9b700 1 -- 172.21.15.67:0/3801170799 learned_addr learned my addr 172.21.15.67:0/3801170799 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:54:54.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.702+0000 7f01aa59c700 1 --2- 172.21.15.67:0/3801170799 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f01ac068580 0x7f01ac0ffcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:54.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.703+0000 7f01a9d9b700 1 -- 172.21.15.67:0/3801170799 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f01ac068580 msgr2=0x7f01ac0ffcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:54.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.703+0000 7f01a9d9b700 1 --2- 172.21.15.67:0/3801170799 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f01ac068580 0x7f01ac0ffcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:54.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.703+0000 7f01a9d9b700 1 -- 172.21.15.67:0/3801170799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f019c009d00 con 0x7f01ac106cd0 2024-04-22T23:54:54.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.703+0000 7f01a9d9b700 1 --2- 172.21.15.67:0/3801170799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01ac106cd0 0x7f01ac100230 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f019400d8d0 tx=0x7f019400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:54:54.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.703+0000 7f01a37fe700 1 -- 172.21.15.67:0/3801170799 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0194009940 con 0x7f01ac106cd0 2024-04-22T23:54:54.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.703+0000 7f01a37fe700 1 -- 172.21.15.67:0/3801170799 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0194010460 con 0x7f01ac106cd0 2024-04-22T23:54:54.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.703+0000 7f01b0de2700 1 -- 172.21.15.67:0/3801170799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01ac1a2a90 con 0x7f01ac106cd0 2024-04-22T23:54:54.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.703+0000 7f01a37fe700 1 -- 172.21.15.67:0/3801170799 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f019400f5f0 con 0x7f01ac106cd0 2024-04-22T23:54:54.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.703+0000 7f01b0de2700 1 -- 172.21.15.67:0/3801170799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01ac1a2ea0 con 0x7f01ac106cd0 2024-04-22T23:54:54.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.704+0000 7f01b0de2700 1 -- 172.21.15.67:0/3801170799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01ac1091d0 con 0x7f01ac106cd0 2024-04-22T23:54:54.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.705+0000 7f01a37fe700 1 -- 172.21.15.67:0/3801170799 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f01940105d0 con 0x7f01ac106cd0 2024-04-22T23:54:54.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.705+0000 7f01a37fe700 1 --2- 172.21.15.67:0/3801170799 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f019806ecc0 0x7f0198071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:54.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.705+0000 7f01a37fe700 1 -- 172.21.15.67:0/3801170799 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f019408eff0 con 0x7f01ac106cd0 2024-04-22T23:54:54.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.705+0000 7f01aa59c700 1 --2- 172.21.15.67:0/3801170799 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f019806ecc0 0x7f0198071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:54.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.706+0000 7f01aa59c700 1 --2- 172.21.15.67:0/3801170799 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f019806ecc0 0x7f0198071180 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f019c009480 tx=0x7f019c00b7f0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:54:54.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.710+0000 7f01a37fe700 1 -- 172.21.15.67:0/3801170799 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f0194057c30 con 0x7f01ac106cd0 2024-04-22T23:54:54.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.939+0000 7f01b0de2700 1 -- 172.21.15.67:0/3801170799 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f01ac0611d0 con 0x7f019806ecc0 2024-04-22T23:54:54.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.941+0000 7f01a37fe700 1 -- 172.21.15.67:0/3801170799 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f01ac0611d0 con 0x7f019806ecc0 2024-04-22T23:54:54.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.943+0000 7f01b0de2700 1 -- 172.21.15.67:0/3801170799 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f019806ecc0 msgr2=0x7f0198071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:54.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.943+0000 7f01b0de2700 1 --2- 172.21.15.67:0/3801170799 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f019806ecc0 0x7f0198071180 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f019c009480 tx=0x7f019c00b7f0 comp rx=0 tx=0).stop 2024-04-22T23:54:54.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.943+0000 7f01b0de2700 1 -- 172.21.15.67:0/3801170799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01ac106cd0 msgr2=0x7f01ac100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:54.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.943+0000 7f01b0de2700 1 --2- 172.21.15.67:0/3801170799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01ac106cd0 0x7f01ac100230 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f019400d8d0 tx=0x7f019400dc90 comp rx=0 tx=0).stop 2024-04-22T23:54:54.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.943+0000 7f01b0de2700 1 -- 172.21.15.67:0/3801170799 shutdown_connections 2024-04-22T23:54:54.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.943+0000 7f01b0de2700 1 --2- 172.21.15.67:0/3801170799 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f019806ecc0 0x7f0198071180 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:54.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.943+0000 7f01b0de2700 1 --2- 172.21.15.67:0/3801170799 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f01ac068580 0x7f01ac0ffcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:54.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.943+0000 7f01b0de2700 1 --2- 172.21.15.67:0/3801170799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01ac106cd0 0x7f01ac100230 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:54.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.943+0000 7f01b0de2700 1 -- 172.21.15.67:0/3801170799 >> 172.21.15.67:0/3801170799 conn(0x7f01ac075670 msgr2=0x7f01ac0fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:54:54.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.943+0000 7f01b0de2700 1 -- 172.21.15.67:0/3801170799 shutdown_connections 2024-04-22T23:54:54.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:54.943+0000 7f01b0de2700 1 -- 172.21.15.67:0/3801170799 wait complete. 2024-04-22T23:54:55.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.051+0000 7f1a37016700 1 -- 172.21.15.67:0/2537465967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a30102700 msgr2=0x7f1a3010abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:55.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.051+0000 7f1a37016700 1 --2- 172.21.15.67:0/2537465967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a30102700 0x7f1a3010abf0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c009910 tx=0x7f1a1c009c20 comp rx=0 tx=0).stop 2024-04-22T23:54:55.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.051+0000 7f1a37016700 1 -- 172.21.15.67:0/2537465967 shutdown_connections 2024-04-22T23:54:55.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.051+0000 7f1a37016700 1 --2- 172.21.15.67:0/2537465967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a30102700 0x7f1a3010abf0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:55.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.051+0000 7f1a37016700 1 --2- 172.21.15.67:0/2537465967 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a30101de0 0x7f1a301021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:55.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.051+0000 7f1a37016700 1 -- 172.21.15.67:0/2537465967 >> 172.21.15.67:0/2537465967 conn(0x7f1a300fb650 msgr2=0x7f1a300fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:54:55.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.051+0000 7f1a37016700 1 -- 172.21.15.67:0/2537465967 shutdown_connections 2024-04-22T23:54:55.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.051+0000 7f1a37016700 1 -- 172.21.15.67:0/2537465967 wait complete. 2024-04-22T23:54:55.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.051+0000 7f1a37016700 1 Processor -- start 2024-04-22T23:54:55.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.051+0000 7f1a37016700 1 -- start start 2024-04-22T23:54:55.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.052+0000 7f1a37016700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a30101de0 0x7f1a3019e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:55.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.052+0000 7f1a37016700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a30102700 0x7f1a3019eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:55.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.052+0000 7f1a37016700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3019f540 con 0x7f1a30102700 2024-04-22T23:54:55.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.052+0000 7f1a37016700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a301989f0 con 0x7f1a30101de0 2024-04-22T23:54:55.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.052+0000 7f1a2ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a30102700 0x7f1a3019eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:55.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.052+0000 7f1a2ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a30102700 0x7f1a3019eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45524/0 (socket says 172.21.15.67:45524) 2024-04-22T23:54:55.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.052+0000 7f1a2ffff700 1 -- 172.21.15.67:0/568501841 learned_addr learned my addr 172.21.15.67:0/568501841 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:54:55.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.052+0000 7f1a34db2700 1 --2- 172.21.15.67:0/568501841 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a30101de0 0x7f1a3019e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:55.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.052+0000 7f1a2ffff700 1 -- 172.21.15.67:0/568501841 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a30101de0 msgr2=0x7f1a3019e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:55.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.052+0000 7f1a2ffff700 1 --2- 172.21.15.67:0/568501841 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a30101de0 0x7f1a3019e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:55.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.052+0000 7f1a2ffff700 1 -- 172.21.15.67:0/568501841 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a1c0095f0 con 0x7f1a30102700 2024-04-22T23:54:55.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.052+0000 7f1a2ffff700 1 --2- 172.21.15.67:0/568501841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a30102700 0x7f1a3019eeb0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c0092e0 tx=0x7f1a1c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:54:55.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.053+0000 7f1a2dffb700 1 -- 172.21.15.67:0/568501841 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a1c01d070 con 0x7f1a30102700 2024-04-22T23:54:55.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.053+0000 7f1a2dffb700 1 -- 172.21.15.67:0/568501841 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1a1c00f460 con 0x7f1a30102700 2024-04-22T23:54:55.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.053+0000 7f1a37016700 1 -- 172.21.15.67:0/568501841 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a30198c70 con 0x7f1a30102700 2024-04-22T23:54:55.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.053+0000 7f1a2dffb700 1 -- 172.21.15.67:0/568501841 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a1c017610 con 0x7f1a30102700 2024-04-22T23:54:55.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.053+0000 7f1a37016700 1 -- 172.21.15.67:0/568501841 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a30199160 con 0x7f1a30102700 2024-04-22T23:54:55.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.054+0000 7f1a37016700 1 -- 172.21.15.67:0/568501841 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a3010a750 con 0x7f1a30102700 2024-04-22T23:54:55.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.054+0000 7f1a2dffb700 1 -- 172.21.15.67:0/568501841 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1a1c00f5d0 con 0x7f1a30102700 2024-04-22T23:54:55.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.055+0000 7f1a2dffb700 1 --2- 172.21.15.67:0/568501841 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1a2006ecc0 0x7f1a20071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:55.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.055+0000 7f1a2dffb700 1 -- 172.21.15.67:0/568501841 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f1a1c0903b0 con 0x7f1a30102700 2024-04-22T23:54:55.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.055+0000 7f1a34db2700 1 --2- 172.21.15.67:0/568501841 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1a2006ecc0 0x7f1a20071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:55.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.055+0000 7f1a34db2700 1 --2- 172.21.15.67:0/568501841 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1a2006ecc0 0x7f1a20071180 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f1a24006fd0 tx=0x7f1a24008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:54:55.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.060+0000 7f1a2dffb700 1 -- 172.21.15.67:0/568501841 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1a1c05cba0 con 0x7f1a30102700 2024-04-22T23:54:55.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.281+0000 7f1a37016700 1 -- 172.21.15.67:0/568501841 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1a301993d0 con 0x7f1a2006ecc0 2024-04-22T23:54:55.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:55 smithi067 ceph-mon[112438]: from='client.35552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:54:55.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:55 smithi067 ceph-mon[112438]: from='client.35556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:54:55.272 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:55 smithi067 ceph-mon[112438]: pgmap v858: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:55.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.292+0000 7f1a2dffb700 1 -- 172.21.15.67:0/568501841 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f1a301993d0 con 0x7f1a2006ecc0 2024-04-22T23:54:55.284 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:54:55.284 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (28m) 6m ago 39m 24.4M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:54:55.284 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (40m) 6m ago 40m 10.0M - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:54:55.284 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (39m) 6m ago 39m 10.0M - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:54:55.284 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (26m) 6m ago 40m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:54:55.285 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (25m) 6m ago 39m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:54:55.285 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (28m) 6m ago 39m 81.2M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:54:55.285 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (22m) 6m ago 34m 19.4M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:54:55.285 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (21m) 6m ago 34m 20.3M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:54:55.285 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (21m) 6m ago 34m 20.1M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:54:55.285 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (6m) 6m ago 34m 14.0M - 19.0.0-3252-g43be0201 a0434699c1d7 46d3d7e86fee 2024-04-22T23:54:55.285 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (30m) 6m ago 41m 514M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:54:55.285 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (29m) 6m ago 38m 459M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:54:55.285 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (27m) 6m ago 41m 74.2M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:54:55.285 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (26m) 6m ago 38m 54.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:54:55.285 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (29m) 6m ago 39m 15.2M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:54:55.285 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (29m) 6m ago 39m 16.0M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:54:55.285 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (25m) 6m ago 38m 102M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:54:55.285 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (24m) 6m ago 37m 128M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:54:55.285 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (24m) 6m ago 37m 83.8M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:54:55.286 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (23m) 6m ago 36m 101M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:54:55.286 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (23m) 6m ago 36m 86.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:54:55.286 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (22m) 6m ago 36m 81.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:54:55.286 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (28m) 6m ago 39m 78.7M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:54:55.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.295+0000 7f1a37016700 1 -- 172.21.15.67:0/568501841 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1a2006ecc0 msgr2=0x7f1a20071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:55.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.295+0000 7f1a37016700 1 --2- 172.21.15.67:0/568501841 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1a2006ecc0 0x7f1a20071180 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f1a24006fd0 tx=0x7f1a24008040 comp rx=0 tx=0).stop 2024-04-22T23:54:55.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.295+0000 7f1a37016700 1 -- 172.21.15.67:0/568501841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a30102700 msgr2=0x7f1a3019eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:55.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.295+0000 7f1a37016700 1 --2- 172.21.15.67:0/568501841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a30102700 0x7f1a3019eeb0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c0092e0 tx=0x7f1a1c00be20 comp rx=0 tx=0).stop 2024-04-22T23:54:55.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.295+0000 7f1a37016700 1 -- 172.21.15.67:0/568501841 shutdown_connections 2024-04-22T23:54:55.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.295+0000 7f1a37016700 1 --2- 172.21.15.67:0/568501841 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1a2006ecc0 0x7f1a20071180 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:55.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.295+0000 7f1a37016700 1 --2- 172.21.15.67:0/568501841 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a30101de0 0x7f1a3019e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:55.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.295+0000 7f1a37016700 1 --2- 172.21.15.67:0/568501841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a30102700 0x7f1a3019eeb0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:55.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.295+0000 7f1a37016700 1 -- 172.21.15.67:0/568501841 >> 172.21.15.67:0/568501841 conn(0x7f1a300fb650 msgr2=0x7f1a30100200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:54:55.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.295+0000 7f1a37016700 1 -- 172.21.15.67:0/568501841 shutdown_connections 2024-04-22T23:54:55.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.295+0000 7f1a37016700 1 -- 172.21.15.67:0/568501841 wait complete. 2024-04-22T23:54:55.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.401+0000 7fc5aacf9700 1 -- 172.21.15.67:0/4221965856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5a4102700 msgr2=0x7fc5a410abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:55.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.401+0000 7fc5aacf9700 1 --2- 172.21.15.67:0/4221965856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5a4102700 0x7fc5a410abf0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fc598009910 tx=0x7fc598009c20 comp rx=0 tx=0).stop 2024-04-22T23:54:55.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.401+0000 7fc5aacf9700 1 -- 172.21.15.67:0/4221965856 shutdown_connections 2024-04-22T23:54:55.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.401+0000 7fc5aacf9700 1 --2- 172.21.15.67:0/4221965856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5a4102700 0x7fc5a410abf0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:55.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.401+0000 7fc5aacf9700 1 --2- 172.21.15.67:0/4221965856 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5a4101de0 0x7fc5a41021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:55.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.401+0000 7fc5aacf9700 1 -- 172.21.15.67:0/4221965856 >> 172.21.15.67:0/4221965856 conn(0x7fc5a40fb610 msgr2=0x7fc5a40fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:54:55.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.401+0000 7fc5aacf9700 1 -- 172.21.15.67:0/4221965856 shutdown_connections 2024-04-22T23:54:55.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.401+0000 7fc5aacf9700 1 -- 172.21.15.67:0/4221965856 wait complete. 2024-04-22T23:54:55.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.402+0000 7fc5aacf9700 1 Processor -- start 2024-04-22T23:54:55.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.402+0000 7fc5aacf9700 1 -- start start 2024-04-22T23:54:55.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.402+0000 7fc5aacf9700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5a4101de0 0x7fc5a419c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:55.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.402+0000 7fc5aacf9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5a4102700 0x7fc5a419cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:55.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.402+0000 7fc5aacf9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a419d220 con 0x7fc5a4102700 2024-04-22T23:54:55.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.402+0000 7fc5aacf9700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a4196720 con 0x7fc5a4101de0 2024-04-22T23:54:55.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.402+0000 7fc5a3fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5a4102700 0x7fc5a419cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:55.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.402+0000 7fc5a3fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5a4102700 0x7fc5a419cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45546/0 (socket says 172.21.15.67:45546) 2024-04-22T23:54:55.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.402+0000 7fc5a3fff700 1 -- 172.21.15.67:0/3977237199 learned_addr learned my addr 172.21.15.67:0/3977237199 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:54:55.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.402+0000 7fc5a8a95700 1 --2- 172.21.15.67:0/3977237199 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5a4101de0 0x7fc5a419c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:55.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.402+0000 7fc5a3fff700 1 -- 172.21.15.67:0/3977237199 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5a4101de0 msgr2=0x7fc5a419c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:55.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.402+0000 7fc5a3fff700 1 --2- 172.21.15.67:0/3977237199 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5a4101de0 0x7fc5a419c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:55.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.402+0000 7fc5a3fff700 1 -- 172.21.15.67:0/3977237199 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5980095f0 con 0x7fc5a4102700 2024-04-22T23:54:55.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.403+0000 7fc5a3fff700 1 --2- 172.21.15.67:0/3977237199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5a4102700 0x7fc5a419cb90 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fc5980092e0 tx=0x7fc59800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:54:55.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.403+0000 7fc5a1ffb700 1 -- 172.21.15.67:0/3977237199 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc59801d070 con 0x7fc5a4102700 2024-04-22T23:54:55.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.403+0000 7fc5a1ffb700 1 -- 172.21.15.67:0/3977237199 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc598005dc0 con 0x7fc5a4102700 2024-04-22T23:54:55.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.403+0000 7fc5aacf9700 1 -- 172.21.15.67:0/3977237199 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5a41969a0 con 0x7fc5a4102700 2024-04-22T23:54:55.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.403+0000 7fc5a1ffb700 1 -- 172.21.15.67:0/3977237199 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc598017400 con 0x7fc5a4102700 2024-04-22T23:54:55.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.403+0000 7fc5aacf9700 1 -- 172.21.15.67:0/3977237199 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5a4196e90 con 0x7fc5a4102700 2024-04-22T23:54:55.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.404+0000 7fc5aacf9700 1 -- 172.21.15.67:0/3977237199 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5a41082f0 con 0x7fc5a4102700 2024-04-22T23:54:55.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.404+0000 7fc5a1ffb700 1 -- 172.21.15.67:0/3977237199 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc59800f6f0 con 0x7fc5a4102700 2024-04-22T23:54:55.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.405+0000 7fc5a1ffb700 1 --2- 172.21.15.67:0/3977237199 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc58c06ecc0 0x7fc58c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:55.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.405+0000 7fc5a1ffb700 1 -- 172.21.15.67:0/3977237199 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fc598090420 con 0x7fc5a4102700 2024-04-22T23:54:55.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.405+0000 7fc5a8a95700 1 --2- 172.21.15.67:0/3977237199 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc58c06ecc0 0x7fc58c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:55.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.405+0000 7fc5a8a95700 1 --2- 172.21.15.67:0/3977237199 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc58c06ecc0 0x7fc58c071180 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fc594005950 tx=0x7fc5940058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:54:55.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.410+0000 7fc5a1ffb700 1 -- 172.21.15.67:0/3977237199 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc59805cb30 con 0x7fc5a4102700 2024-04-22T23:54:55.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:55 smithi134 ceph-mon[85630]: from='client.35552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:54:55.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:55 smithi134 ceph-mon[85630]: from='client.35556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:54:55.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:55 smithi134 ceph-mon[85630]: pgmap v858: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:55.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.706+0000 7fc5aacf9700 1 -- 172.21.15.67:0/3977237199 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc5a40679c0 con 0x7fc5a4102700 2024-04-22T23:54:55.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.706+0000 7fc5a1ffb700 1 -- 172.21.15.67:0/3977237199 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fc59805c6c0 con 0x7fc5a4102700 2024-04-22T23:54:55.697 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:54:55.697 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:54:55.697 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:54:55.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:54:55.697 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:54:55.697 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:54:55.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:54:55.697 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:54:55.698 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:54:55.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:54:55.698 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:54:55.698 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 4 2024-04-22T23:54:55.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:54:55.698 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:54:55.698 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 14 2024-04-22T23:54:55.698 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:54:55.698 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:54:55.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.708+0000 7fc5aacf9700 1 -- 172.21.15.67:0/3977237199 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc58c06ecc0 msgr2=0x7fc58c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:55.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.708+0000 7fc5aacf9700 1 --2- 172.21.15.67:0/3977237199 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc58c06ecc0 0x7fc58c071180 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fc594005950 tx=0x7fc5940058e0 comp rx=0 tx=0).stop 2024-04-22T23:54:55.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.708+0000 7fc5aacf9700 1 -- 172.21.15.67:0/3977237199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5a4102700 msgr2=0x7fc5a419cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:55.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.708+0000 7fc5aacf9700 1 --2- 172.21.15.67:0/3977237199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5a4102700 0x7fc5a419cb90 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fc5980092e0 tx=0x7fc59800be20 comp rx=0 tx=0).stop 2024-04-22T23:54:55.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.709+0000 7fc5aacf9700 1 -- 172.21.15.67:0/3977237199 shutdown_connections 2024-04-22T23:54:55.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.709+0000 7fc5aacf9700 1 --2- 172.21.15.67:0/3977237199 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc58c06ecc0 0x7fc58c071180 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:55.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.709+0000 7fc5aacf9700 1 --2- 172.21.15.67:0/3977237199 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5a4101de0 0x7fc5a419c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:55.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.709+0000 7fc5aacf9700 1 --2- 172.21.15.67:0/3977237199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5a4102700 0x7fc5a419cb90 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:55.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.709+0000 7fc5aacf9700 1 -- 172.21.15.67:0/3977237199 >> 172.21.15.67:0/3977237199 conn(0x7fc5a40fb610 msgr2=0x7fc5a4105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:54:55.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.709+0000 7fc5aacf9700 1 -- 172.21.15.67:0/3977237199 shutdown_connections 2024-04-22T23:54:55.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.709+0000 7fc5aacf9700 1 -- 172.21.15.67:0/3977237199 wait complete. 2024-04-22T23:54:55.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.815+0000 7f4a7984f700 1 -- 172.21.15.67:0/644050717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a74100750 msgr2=0x7f4a741047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:55.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.815+0000 7f4a7984f700 1 --2- 172.21.15.67:0/644050717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a74100750 0x7f4a741047a0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f4a64009910 tx=0x7f4a64009c20 comp rx=0 tx=0).stop 2024-04-22T23:54:55.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.815+0000 7f4a7984f700 1 -- 172.21.15.67:0/644050717 shutdown_connections 2024-04-22T23:54:55.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.815+0000 7f4a7984f700 1 --2- 172.21.15.67:0/644050717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a74100750 0x7f4a741047a0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:55.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.815+0000 7f4a7984f700 1 --2- 172.21.15.67:0/644050717 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a740ffda0 0x7f4a74100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:55.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.815+0000 7f4a7984f700 1 -- 172.21.15.67:0/644050717 >> 172.21.15.67:0/644050717 conn(0x7f4a740fb650 msgr2=0x7f4a740fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:54:55.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.815+0000 7f4a7984f700 1 -- 172.21.15.67:0/644050717 shutdown_connections 2024-04-22T23:54:55.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.815+0000 7f4a7984f700 1 -- 172.21.15.67:0/644050717 wait complete. 2024-04-22T23:54:55.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.816+0000 7f4a7984f700 1 Processor -- start 2024-04-22T23:54:55.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.816+0000 7f4a7984f700 1 -- start start 2024-04-22T23:54:55.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.816+0000 7f4a7984f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a740ffda0 0x7f4a741977c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:55.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.816+0000 7f4a7984f700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a74100750 0x7f4a74197d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:55.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.816+0000 7f4a7984f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a74198320 con 0x7f4a740ffda0 2024-04-22T23:54:55.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.816+0000 7f4a7984f700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a74198460 con 0x7f4a74100750 2024-04-22T23:54:55.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.816+0000 7f4a72ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a740ffda0 0x7f4a741977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:55.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.817+0000 7f4a72ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a740ffda0 0x7f4a741977c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45564/0 (socket says 172.21.15.67:45564) 2024-04-22T23:54:55.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.817+0000 7f4a72ffd700 1 -- 172.21.15.67:0/2185100092 learned_addr learned my addr 172.21.15.67:0/2185100092 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:54:55.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.817+0000 7f4a727fc700 1 --2- 172.21.15.67:0/2185100092 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a74100750 0x7f4a74197d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:55.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.817+0000 7f4a72ffd700 1 -- 172.21.15.67:0/2185100092 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a74100750 msgr2=0x7f4a74197d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:55.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.817+0000 7f4a72ffd700 1 --2- 172.21.15.67:0/2185100092 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a74100750 0x7f4a74197d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:55.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.817+0000 7f4a72ffd700 1 -- 172.21.15.67:0/2185100092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a640095f0 con 0x7f4a740ffda0 2024-04-22T23:54:55.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.817+0000 7f4a72ffd700 1 --2- 172.21.15.67:0/2185100092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a740ffda0 0x7f4a741977c0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f4a6800d8d0 tx=0x7f4a6800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:54:55.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.817+0000 7f4a7884d700 1 -- 172.21.15.67:0/2185100092 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a680098a0 con 0x7f4a740ffda0 2024-04-22T23:54:55.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.817+0000 7f4a7884d700 1 -- 172.21.15.67:0/2185100092 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4a68010460 con 0x7f4a740ffda0 2024-04-22T23:54:55.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.817+0000 7f4a7984f700 1 -- 172.21.15.67:0/2185100092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a741a0830 con 0x7f4a740ffda0 2024-04-22T23:54:55.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.817+0000 7f4a7884d700 1 -- 172.21.15.67:0/2185100092 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a6800f5f0 con 0x7f4a740ffda0 2024-04-22T23:54:55.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.817+0000 7f4a7984f700 1 -- 172.21.15.67:0/2185100092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a741a0cf0 con 0x7f4a740ffda0 2024-04-22T23:54:55.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.818+0000 7f4a7984f700 1 -- 172.21.15.67:0/2185100092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a741081a0 con 0x7f4a740ffda0 2024-04-22T23:54:55.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.819+0000 7f4a7884d700 1 -- 172.21.15.67:0/2185100092 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4a68009a00 con 0x7f4a740ffda0 2024-04-22T23:54:55.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.819+0000 7f4a7884d700 1 --2- 172.21.15.67:0/2185100092 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4a5c06ecc0 0x7f4a5c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:55.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.819+0000 7f4a7884d700 1 -- 172.21.15.67:0/2185100092 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f4a6808eda0 con 0x7f4a740ffda0 2024-04-22T23:54:55.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.819+0000 7f4a727fc700 1 --2- 172.21.15.67:0/2185100092 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4a5c06ecc0 0x7f4a5c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:55.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.820+0000 7f4a727fc700 1 --2- 172.21.15.67:0/2185100092 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4a5c06ecc0 0x7f4a5c071180 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f4a64006010 tx=0x7f4a6401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:54:55.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:55.824+0000 7f4a7884d700 1 -- 172.21.15.67:0/2185100092 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f4a68057a60 con 0x7f4a740ffda0 2024-04-22T23:54:56.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.080+0000 7f4a7984f700 1 -- 172.21.15.67:0/2185100092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4a740694a0 con 0x7f4a740ffda0 2024-04-22T23:54:56.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.080+0000 7f4a7884d700 1 -- 172.21.15.67:0/2185100092 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1872 (secure 0 0 0) 0x7f4a68016760 con 0x7f4a740ffda0 2024-04-22T23:54:56.071 INFO:teuthology.orchestra.run.smithi067.stdout:e33 2024-04-22T23:54:56.071 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:54:56.072 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:54:56.072 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:54:56.072 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:54:56.072 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:54:56.072 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:54:56.072 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:54:56.072 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:54:56.072 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:54:56.072 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:54:56.072 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:54:56.072 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:54:56.072 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:54:56.072 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:54:56.072 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:54:56.073 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:54:56.073 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:54:56.073 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:54:56.073 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:54:56.073 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:54:56.073 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:54:56.073 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:54:56.073 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:54:56.073 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:54:56.073 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:54:56.073 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:54:56.073 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:54:56.073 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:54:56.074 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:54:56.074 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:54:56.074 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:54:56.074 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:54:56.074 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:54:56.074 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:54:56.074 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:54:56.074 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:54:56.074 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:54:56.074 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:54:56.074 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:54:56.074 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:54:56.074 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44851} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1849648622,v1:172.21.15.134:6827/1849648622] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:54:56.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.082+0000 7f4a7984f700 1 -- 172.21.15.67:0/2185100092 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4a5c06ecc0 msgr2=0x7f4a5c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:56.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.082+0000 7f4a7984f700 1 --2- 172.21.15.67:0/2185100092 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4a5c06ecc0 0x7f4a5c071180 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f4a64006010 tx=0x7f4a6401a040 comp rx=0 tx=0).stop 2024-04-22T23:54:56.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.082+0000 7f4a7984f700 1 -- 172.21.15.67:0/2185100092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a740ffda0 msgr2=0x7f4a741977c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:56.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.082+0000 7f4a7984f700 1 --2- 172.21.15.67:0/2185100092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a740ffda0 0x7f4a741977c0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f4a6800d8d0 tx=0x7f4a6800dbe0 comp rx=0 tx=0).stop 2024-04-22T23:54:56.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.082+0000 7f4a7984f700 1 -- 172.21.15.67:0/2185100092 shutdown_connections 2024-04-22T23:54:56.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.082+0000 7f4a7984f700 1 --2- 172.21.15.67:0/2185100092 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4a5c06ecc0 0x7f4a5c071180 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:56.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.082+0000 7f4a7984f700 1 --2- 172.21.15.67:0/2185100092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a740ffda0 0x7f4a741977c0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:56.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.082+0000 7f4a7984f700 1 --2- 172.21.15.67:0/2185100092 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4a74100750 0x7f4a74197d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:56.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.082+0000 7f4a7984f700 1 -- 172.21.15.67:0/2185100092 >> 172.21.15.67:0/2185100092 conn(0x7f4a740fb650 msgr2=0x7f4a740fccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:54:56.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.083+0000 7f4a7984f700 1 -- 172.21.15.67:0/2185100092 shutdown_connections 2024-04-22T23:54:56.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.083+0000 7f4a7984f700 1 -- 172.21.15.67:0/2185100092 wait complete. 2024-04-22T23:54:56.076 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 33 2024-04-22T23:54:56.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.193+0000 7f2d0ec9b700 1 -- 172.21.15.67:0/1180846200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d080ffb40 msgr2=0x7f2d080fffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:56.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.193+0000 7f2d0ec9b700 1 --2- 172.21.15.67:0/1180846200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d080ffb40 0x7f2d080fffc0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f2cf4009910 tx=0x7f2cf4009c20 comp rx=0 tx=0).stop 2024-04-22T23:54:56.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.193+0000 7f2d0ec9b700 1 -- 172.21.15.67:0/1180846200 shutdown_connections 2024-04-22T23:54:56.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.193+0000 7f2d0ec9b700 1 --2- 172.21.15.67:0/1180846200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d080ffb40 0x7f2d080fffc0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:56.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.193+0000 7f2d0ec9b700 1 --2- 172.21.15.67:0/1180846200 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d08105bf0 0x7f2d08105fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:56.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.193+0000 7f2d0ec9b700 1 -- 172.21.15.67:0/1180846200 >> 172.21.15.67:0/1180846200 conn(0x7f2d080fb610 msgr2=0x7f2d080fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:54:56.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.193+0000 7f2d0ec9b700 1 -- 172.21.15.67:0/1180846200 shutdown_connections 2024-04-22T23:54:56.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.193+0000 7f2d0ec9b700 1 -- 172.21.15.67:0/1180846200 wait complete. 2024-04-22T23:54:56.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.193+0000 7f2d0ec9b700 1 Processor -- start 2024-04-22T23:54:56.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.193+0000 7f2d0ec9b700 1 -- start start 2024-04-22T23:54:56.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.194+0000 7f2d0ec9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d080ffb40 0x7f2d0819c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:56.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.194+0000 7f2d0ec9b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d08105bf0 0x7f2d0819cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:56.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.194+0000 7f2d0ec9b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d0819d130 con 0x7f2d080ffb40 2024-04-22T23:54:56.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.194+0000 7f2d0ec9b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d08196670 con 0x7f2d08105bf0 2024-04-22T23:54:56.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.194+0000 7f2d0ca37700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d080ffb40 0x7f2d0819c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:56.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.194+0000 7f2d0ca37700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d080ffb40 0x7f2d0819c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45586/0 (socket says 172.21.15.67:45586) 2024-04-22T23:54:56.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.194+0000 7f2d0ca37700 1 -- 172.21.15.67:0/3987198053 learned_addr learned my addr 172.21.15.67:0/3987198053 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:54:56.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.194+0000 7f2d07fff700 1 --2- 172.21.15.67:0/3987198053 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d08105bf0 0x7f2d0819cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:56.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.194+0000 7f2d0ca37700 1 -- 172.21.15.67:0/3987198053 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d08105bf0 msgr2=0x7f2d0819cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:56.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.194+0000 7f2d0ca37700 1 --2- 172.21.15.67:0/3987198053 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d08105bf0 0x7f2d0819cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:56.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.194+0000 7f2d0ca37700 1 -- 172.21.15.67:0/3987198053 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf40095f0 con 0x7f2d080ffb40 2024-04-22T23:54:56.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.195+0000 7f2d0ca37700 1 --2- 172.21.15.67:0/3987198053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d080ffb40 0x7f2d0819c5f0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc00c930 tx=0x7f2cfc00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:54:56.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.195+0000 7f2d05ffb700 1 -- 172.21.15.67:0/3987198053 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2cfc007ad0 con 0x7f2d080ffb40 2024-04-22T23:54:56.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.195+0000 7f2d05ffb700 1 -- 172.21.15.67:0/3987198053 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2cfc007c30 con 0x7f2d080ffb40 2024-04-22T23:54:56.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.195+0000 7f2d0ec9b700 1 -- 172.21.15.67:0/3987198053 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d08196950 con 0x7f2d080ffb40 2024-04-22T23:54:56.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.195+0000 7f2d05ffb700 1 -- 172.21.15.67:0/3987198053 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2cfc018610 con 0x7f2d080ffb40 2024-04-22T23:54:56.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.195+0000 7f2d0ec9b700 1 -- 172.21.15.67:0/3987198053 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d08196e50 con 0x7f2d080ffb40 2024-04-22T23:54:56.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.196+0000 7f2d0ec9b700 1 -- 172.21.15.67:0/3987198053 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d080679c0 con 0x7f2d080ffb40 2024-04-22T23:54:56.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.196+0000 7f2d05ffb700 1 -- 172.21.15.67:0/3987198053 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2cfc00f450 con 0x7f2d080ffb40 2024-04-22T23:54:56.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.197+0000 7f2d05ffb700 1 --2- 172.21.15.67:0/3987198053 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2cf806ecc0 0x7f2cf8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:56.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.197+0000 7f2d05ffb700 1 -- 172.21.15.67:0/3987198053 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f2cfc08f250 con 0x7f2d080ffb40 2024-04-22T23:54:56.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.197+0000 7f2d07fff700 1 --2- 172.21.15.67:0/3987198053 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2cf806ecc0 0x7f2cf8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:56.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.197+0000 7f2d07fff700 1 --2- 172.21.15.67:0/3987198053 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2cf806ecc0 0x7f2cf8071180 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2d080feae0 tx=0x7f2cf401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:54:56.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.202+0000 7f2d05ffb700 1 -- 172.21.15.67:0/3987198053 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f2cfc05b960 con 0x7f2d080ffb40 2024-04-22T23:54:56.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.431+0000 7f2d0ec9b700 1 -- 172.21.15.67:0/3987198053 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2d08197190 con 0x7f2cf806ecc0 2024-04-22T23:54:56.423 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:56 smithi067 ceph-mon[112438]: from='client.35560 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:54:56.423 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:56 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3977237199' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:54:56.423 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:54:56.423 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:56 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2185100092' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:54:56.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.433+0000 7f2d05ffb700 1 -- 172.21.15.67:0/3987198053 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f2d08197190 con 0x7f2cf806ecc0 2024-04-22T23:54:56.423 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:54:56.424 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:54:56.424 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:54:56.424 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:54:56.424 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:54:56.424 INFO:teuthology.orchestra.run.smithi067.stdout: "mds", 2024-04-22T23:54:56.424 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:54:56.424 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:54:56.424 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:54:56.424 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:54:56.424 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:54:56.424 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T23:54:56.424 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:54:56.424 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:54:56.424 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:54:56.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.435+0000 7f2d0ec9b700 1 -- 172.21.15.67:0/3987198053 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2cf806ecc0 msgr2=0x7f2cf8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:56.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.435+0000 7f2d0ec9b700 1 --2- 172.21.15.67:0/3987198053 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2cf806ecc0 0x7f2cf8071180 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2d080feae0 tx=0x7f2cf401a040 comp rx=0 tx=0).stop 2024-04-22T23:54:56.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.435+0000 7f2d0ec9b700 1 -- 172.21.15.67:0/3987198053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d080ffb40 msgr2=0x7f2d0819c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:56.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.435+0000 7f2d0ec9b700 1 --2- 172.21.15.67:0/3987198053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d080ffb40 0x7f2d0819c5f0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc00c930 tx=0x7f2cfc00ccf0 comp rx=0 tx=0).stop 2024-04-22T23:54:56.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.436+0000 7f2d0ec9b700 1 -- 172.21.15.67:0/3987198053 shutdown_connections 2024-04-22T23:54:56.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.436+0000 7f2d0ec9b700 1 --2- 172.21.15.67:0/3987198053 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2cf806ecc0 0x7f2cf8071180 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:56.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.436+0000 7f2d0ec9b700 1 --2- 172.21.15.67:0/3987198053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d080ffb40 0x7f2d0819c5f0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:56.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.436+0000 7f2d0ec9b700 1 --2- 172.21.15.67:0/3987198053 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2d08105bf0 0x7f2d0819cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:56.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.436+0000 7f2d0ec9b700 1 -- 172.21.15.67:0/3987198053 >> 172.21.15.67:0/3987198053 conn(0x7f2d080fb610 msgr2=0x7f2d08068e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:54:56.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.436+0000 7f2d0ec9b700 1 -- 172.21.15.67:0/3987198053 shutdown_connections 2024-04-22T23:54:56.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.436+0000 7f2d0ec9b700 1 -- 172.21.15.67:0/3987198053 wait complete. 2024-04-22T23:54:56.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:56 smithi134 ceph-mon[85630]: from='client.35560 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:54:56.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:56 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3977237199' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:54:56.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:54:56.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:56 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2185100092' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:54:56.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.543+0000 7f838baa0700 1 -- 172.21.15.67:0/2102914791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8384102720 msgr2=0x7f838410ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:56.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.543+0000 7f838baa0700 1 --2- 172.21.15.67:0/2102914791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8384102720 0x7f838410ac10 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f8374009910 tx=0x7f8374009c20 comp rx=0 tx=0).stop 2024-04-22T23:54:56.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.543+0000 7f838baa0700 1 -- 172.21.15.67:0/2102914791 shutdown_connections 2024-04-22T23:54:56.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.543+0000 7f838baa0700 1 --2- 172.21.15.67:0/2102914791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8384102720 0x7f838410ac10 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:56.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.543+0000 7f838baa0700 1 --2- 172.21.15.67:0/2102914791 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8384101e00 0x7f83841021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:56.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.543+0000 7f838baa0700 1 -- 172.21.15.67:0/2102914791 >> 172.21.15.67:0/2102914791 conn(0x7f83840fb650 msgr2=0x7f83840fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:54:56.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.543+0000 7f838baa0700 1 -- 172.21.15.67:0/2102914791 shutdown_connections 2024-04-22T23:54:56.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.543+0000 7f838baa0700 1 -- 172.21.15.67:0/2102914791 wait complete. 2024-04-22T23:54:56.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.543+0000 7f838baa0700 1 Processor -- start 2024-04-22T23:54:56.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.543+0000 7f838baa0700 1 -- start start 2024-04-22T23:54:56.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.543+0000 7f838baa0700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8384101e00 0x7f838419c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:56.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.544+0000 7f838baa0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8384102720 0x7f838419cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:56.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.544+0000 7f838baa0700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838419d260 con 0x7f8384102720 2024-04-22T23:54:56.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.544+0000 7f838baa0700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8384196710 con 0x7f8384101e00 2024-04-22T23:54:56.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.544+0000 7f838903b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8384102720 0x7f838419cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:56.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.544+0000 7f838903b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8384102720 0x7f838419cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45602/0 (socket says 172.21.15.67:45602) 2024-04-22T23:54:56.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.544+0000 7f838903b700 1 -- 172.21.15.67:0/548720508 learned_addr learned my addr 172.21.15.67:0/548720508 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:54:56.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.544+0000 7f838983c700 1 --2- 172.21.15.67:0/548720508 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8384101e00 0x7f838419c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:56.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.544+0000 7f838903b700 1 -- 172.21.15.67:0/548720508 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8384101e00 msgr2=0x7f838419c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:56.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.544+0000 7f838903b700 1 --2- 172.21.15.67:0/548720508 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8384101e00 0x7f838419c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:56.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.544+0000 7f838903b700 1 -- 172.21.15.67:0/548720508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83740095f0 con 0x7f8384102720 2024-04-22T23:54:56.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.544+0000 7f838903b700 1 --2- 172.21.15.67:0/548720508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8384102720 0x7f838419cbd0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f837400bd70 tx=0x7f837400bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:54:56.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.544+0000 7f837affd700 1 -- 172.21.15.67:0/548720508 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f837401d070 con 0x7f8384102720 2024-04-22T23:54:56.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.544+0000 7f838baa0700 1 -- 172.21.15.67:0/548720508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8384196990 con 0x7f8384102720 2024-04-22T23:54:56.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.544+0000 7f837affd700 1 -- 172.21.15.67:0/548720508 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8374005dc0 con 0x7f8384102720 2024-04-22T23:54:56.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.544+0000 7f837affd700 1 -- 172.21.15.67:0/548720508 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8374017400 con 0x7f8384102720 2024-04-22T23:54:56.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.544+0000 7f838baa0700 1 -- 172.21.15.67:0/548720508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8384196e80 con 0x7f8384102720 2024-04-22T23:54:56.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.545+0000 7f838baa0700 1 -- 172.21.15.67:0/548720508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8384108310 con 0x7f8384102720 2024-04-22T23:54:56.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.546+0000 7f837affd700 1 -- 172.21.15.67:0/548720508 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f837400f460 con 0x7f8384102720 2024-04-22T23:54:56.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.546+0000 7f837affd700 1 --2- 172.21.15.67:0/548720508 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f837006ecc0 0x7f8370071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:54:56.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.547+0000 7f838983c700 1 --2- 172.21.15.67:0/548720508 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f837006ecc0 0x7f8370071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:54:56.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.547+0000 7f837affd700 1 -- 172.21.15.67:0/548720508 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f83740903f0 con 0x7f8384102720 2024-04-22T23:54:56.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.547+0000 7f838983c700 1 --2- 172.21.15.67:0/548720508 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f837006ecc0 0x7f8370071180 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f8380005950 tx=0x7f838000b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:54:56.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.551+0000 7f837affd700 1 -- 172.21.15.67:0/548720508 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f837405cac0 con 0x7f8384102720 2024-04-22T23:54:56.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.848+0000 7f838baa0700 1 -- 172.21.15.67:0/548720508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8384197cc0 con 0x7f8384102720 2024-04-22T23:54:56.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.849+0000 7f837affd700 1 -- 172.21.15.67:0/548720508 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4505 (secure 0 0 0) 0x7f837405c650 con 0x7f8384102720 2024-04-22T23:54:56.840 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:54:56.840 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T23:54:56.840 INFO:teuthology.orchestra.run.smithi067.stdout: host smithi134 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-22T23:54:56.840 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T23:54:56.840 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:54:56.840 INFO:teuthology.orchestra.run.smithi067.stdout:Traceback (most recent call last): 2024-04-22T23:54:56.840 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T23:54:56.840 INFO:teuthology.orchestra.run.smithi067.stdout: return _run_code(code, main_globals, None, 2024-04-22T23:54:56.840 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T23:54:56.841 INFO:teuthology.orchestra.run.smithi067.stdout: exec(code, run_globals) 2024-04-22T23:54:56.841 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5642, in 2024-04-22T23:54:56.841 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5630, in main 2024-04-22T23:54:56.841 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 408, in _infer_config 2024-04-22T23:54:56.841 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 323, in _infer_fsid 2024-04-22T23:54:56.841 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 436, in _infer_image 2024-04-22T23:54:56.841 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 310, in _validate_fsid 2024-04-22T23:54:56.841 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 3380, in command_ceph_volume 2024-04-22T23:54:56.841 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T23:54:56.841 INFO:teuthology.orchestra.run.smithi067.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:54:56.841 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:54:56.841 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:54:56.841 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:54:56.841 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:54:56.841 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:54:56.842 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:54:56.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.851+0000 7f838baa0700 1 -- 172.21.15.67:0/548720508 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f837006ecc0 msgr2=0x7f8370071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:56.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.851+0000 7f838baa0700 1 --2- 172.21.15.67:0/548720508 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f837006ecc0 0x7f8370071180 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f8380005950 tx=0x7f838000b410 comp rx=0 tx=0).stop 2024-04-22T23:54:56.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.851+0000 7f838baa0700 1 -- 172.21.15.67:0/548720508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8384102720 msgr2=0x7f838419cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:54:56.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.851+0000 7f838baa0700 1 --2- 172.21.15.67:0/548720508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8384102720 0x7f838419cbd0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f837400bd70 tx=0x7f837400bda0 comp rx=0 tx=0).stop 2024-04-22T23:54:56.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.851+0000 7f838baa0700 1 -- 172.21.15.67:0/548720508 shutdown_connections 2024-04-22T23:54:56.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.851+0000 7f838baa0700 1 --2- 172.21.15.67:0/548720508 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f837006ecc0 0x7f8370071180 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:56.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.851+0000 7f838baa0700 1 --2- 172.21.15.67:0/548720508 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8384101e00 0x7f838419c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:56.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.851+0000 7f838baa0700 1 --2- 172.21.15.67:0/548720508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8384102720 0x7f838419cbd0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:54:56.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.851+0000 7f838baa0700 1 -- 172.21.15.67:0/548720508 >> 172.21.15.67:0/548720508 conn(0x7f83840fb650 msgr2=0x7f8384105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:54:56.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.851+0000 7f838baa0700 1 -- 172.21.15.67:0/548720508 shutdown_connections 2024-04-22T23:54:56.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:54:56.851+0000 7f838baa0700 1 -- 172.21.15.67:0/548720508 wait complete. 2024-04-22T23:54:57.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:57 smithi134 ceph-mon[85630]: from='client.35572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:54:57.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:57 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/548720508' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:54:57.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:54:57 smithi134 ceph-mon[85630]: pgmap v859: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:54:57.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:57 smithi067 ceph-mon[112438]: from='client.35572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:54:57.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:57 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/548720508' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:54:57.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:54:57 smithi067 ceph-mon[112438]: pgmap v859: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:00.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:00 smithi134 ceph-mon[85630]: pgmap v860: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:00.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:00 smithi067 ceph-mon[112438]: pgmap v860: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:01.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:01 smithi134 ceph-mon[85630]: pgmap v861: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:01.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:01 smithi067 ceph-mon[112438]: pgmap v861: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:04.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:04 smithi134 ceph-mon[85630]: pgmap v862: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:04.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:04 smithi067 ceph-mon[112438]: pgmap v862: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:05.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:05 smithi134 ceph-mon[85630]: pgmap v863: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:05.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:05 smithi067 ceph-mon[112438]: pgmap v863: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:08.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:08 smithi134 ceph-mon[85630]: pgmap v864: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:08.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:08 smithi067 ceph-mon[112438]: pgmap v864: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:09.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:09 smithi134 ceph-mon[85630]: pgmap v865: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:09.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:09 smithi067 ceph-mon[112438]: pgmap v865: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:11.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:11 smithi134 ceph-mon[85630]: pgmap v866: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:11.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:55:11.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:11 smithi067 ceph-mon[112438]: pgmap v866: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:11.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:55:14.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:14 smithi134 ceph-mon[85630]: pgmap v867: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:14.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:14 smithi067 ceph-mon[112438]: pgmap v867: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:15.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:15 smithi134 ceph-mon[85630]: pgmap v868: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:15.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:15 smithi067 ceph-mon[112438]: pgmap v868: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:17.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:17 smithi067 ceph-mon[112438]: pgmap v869: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:17.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:17 smithi134 ceph-mon[85630]: pgmap v869: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:19.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:19 smithi134 ceph-mon[85630]: pgmap v870: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:19.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:19 smithi067 ceph-mon[112438]: pgmap v870: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:21.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:21 smithi134 ceph-mon[85630]: pgmap v871: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:21.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:21 smithi067 ceph-mon[112438]: pgmap v871: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:23.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:23 smithi134 ceph-mon[85630]: pgmap v872: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:23.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:23 smithi067 ceph-mon[112438]: pgmap v872: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:25.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:25 smithi134 ceph-mon[85630]: pgmap v873: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:25.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:25 smithi067 ceph-mon[112438]: pgmap v873: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:26.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:55:26.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:55:26.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:26.973+0000 7f1135f1c700 1 -- 172.21.15.67:0/1248670541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1130100e90 msgr2=0x7f1130101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:55:26.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:26.973+0000 7f1135f1c700 1 --2- 172.21.15.67:0/1248670541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1130100e90 0x7f1130101270 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f112000b0a0 tx=0x7f112000b3b0 comp rx=0 tx=0).stop 2024-04-22T23:55:26.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:26.973+0000 7f1135f1c700 1 -- 172.21.15.67:0/1248670541 shutdown_connections 2024-04-22T23:55:26.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:26.973+0000 7f1135f1c700 1 --2- 172.21.15.67:0/1248670541 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1130101840 0x7f1130105890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:26.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:26.973+0000 7f1135f1c700 1 --2- 172.21.15.67:0/1248670541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1130100e90 0x7f1130101270 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:26.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:26.973+0000 7f1135f1c700 1 -- 172.21.15.67:0/1248670541 >> 172.21.15.67:0/1248670541 conn(0x7f11300fc740 msgr2=0x7f11300feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:55:26.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:26.973+0000 7f1135f1c700 1 -- 172.21.15.67:0/1248670541 shutdown_connections 2024-04-22T23:55:26.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:26.973+0000 7f1135f1c700 1 -- 172.21.15.67:0/1248670541 wait complete. 2024-04-22T23:55:26.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:26.974+0000 7f1135f1c700 1 Processor -- start 2024-04-22T23:55:26.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:26.974+0000 7f1135f1c700 1 -- start start 2024-04-22T23:55:26.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:26.974+0000 7f1135f1c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1130100e90 0x7f113019c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:55:26.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:26.974+0000 7f1135f1c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1130101840 0x7f113019cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:55:26.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:26.974+0000 7f1135f1c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113019d2b0 con 0x7f1130101840 2024-04-22T23:55:26.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:26.974+0000 7f1135f1c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1130196760 con 0x7f1130100e90 2024-04-22T23:55:26.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:26.975+0000 7f112effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1130101840 0x7f113019cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:55:26.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:26.975+0000 7f112effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1130101840 0x7f113019cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38722/0 (socket says 172.21.15.67:38722) 2024-04-22T23:55:26.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:26.975+0000 7f112effd700 1 -- 172.21.15.67:0/1914907371 learned_addr learned my addr 172.21.15.67:0/1914907371 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:55:26.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:26.975+0000 7f112f7fe700 1 --2- 172.21.15.67:0/1914907371 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1130100e90 0x7f113019c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:55:26.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:26.975+0000 7f112effd700 1 -- 172.21.15.67:0/1914907371 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1130100e90 msgr2=0x7f113019c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:55:26.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:26.975+0000 7f112effd700 1 --2- 172.21.15.67:0/1914907371 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1130100e90 0x7f113019c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:26.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:26.975+0000 7f112effd700 1 -- 172.21.15.67:0/1914907371 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1120009d00 con 0x7f1130101840 2024-04-22T23:55:26.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:26.975+0000 7f112f7fe700 1 --2- 172.21.15.67:0/1914907371 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1130100e90 0x7f113019c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:55:26.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:26.975+0000 7f112effd700 1 --2- 172.21.15.67:0/1914907371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1130101840 0x7f113019cc20 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f111800c8f0 tx=0x7f111800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:55:26.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:26.975+0000 7f112cff9700 1 -- 172.21.15.67:0/1914907371 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1118007a50 con 0x7f1130101840 2024-04-22T23:55:26.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:26.975+0000 7f112cff9700 1 -- 172.21.15.67:0/1914907371 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1118007bb0 con 0x7f1130101840 2024-04-22T23:55:26.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:26.975+0000 7f1135f1c700 1 -- 172.21.15.67:0/1914907371 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1130196a40 con 0x7f1130101840 2024-04-22T23:55:26.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:26.975+0000 7f112cff9700 1 -- 172.21.15.67:0/1914907371 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1118018630 con 0x7f1130101840 2024-04-22T23:55:26.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:26.976+0000 7f1135f1c700 1 -- 172.21.15.67:0/1914907371 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1130196f90 con 0x7f1130101840 2024-04-22T23:55:26.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:26.977+0000 7f1135f1c700 1 -- 172.21.15.67:0/1914907371 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1130109230 con 0x7f1130101840 2024-04-22T23:55:26.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:26.977+0000 7f112cff9700 1 -- 172.21.15.67:0/1914907371 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1118007d20 con 0x7f1130101840 2024-04-22T23:55:26.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:26.978+0000 7f112cff9700 1 --2- 172.21.15.67:0/1914907371 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f111c06ecc0 0x7f111c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:55:26.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:26.978+0000 7f112cff9700 1 -- 172.21.15.67:0/1914907371 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f111808efc0 con 0x7f1130101840 2024-04-22T23:55:26.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:26.978+0000 7f112f7fe700 1 --2- 172.21.15.67:0/1914907371 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f111c06ecc0 0x7f111c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:55:26.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:26.978+0000 7f112f7fe700 1 --2- 172.21.15.67:0/1914907371 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f111c06ecc0 0x7f111c071180 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f11200062a0 tx=0x7f1120006210 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:55:26.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:26.983+0000 7f112cff9700 1 -- 172.21.15.67:0/1914907371 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f111805b730 con 0x7f1130101840 2024-04-22T23:55:27.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.219+0000 7f1135f1c700 1 -- 172.21.15.67:0/1914907371 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1130197210 con 0x7f111c06ecc0 2024-04-22T23:55:27.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.220+0000 7f112cff9700 1 -- 172.21.15.67:0/1914907371 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f1130197210 con 0x7f111c06ecc0 2024-04-22T23:55:27.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.222+0000 7f1135f1c700 1 -- 172.21.15.67:0/1914907371 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f111c06ecc0 msgr2=0x7f111c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:55:27.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.222+0000 7f1135f1c700 1 --2- 172.21.15.67:0/1914907371 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f111c06ecc0 0x7f111c071180 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f11200062a0 tx=0x7f1120006210 comp rx=0 tx=0).stop 2024-04-22T23:55:27.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.222+0000 7f1135f1c700 1 -- 172.21.15.67:0/1914907371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1130101840 msgr2=0x7f113019cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:55:27.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.222+0000 7f1135f1c700 1 --2- 172.21.15.67:0/1914907371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1130101840 0x7f113019cc20 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f111800c8f0 tx=0x7f111800ccb0 comp rx=0 tx=0).stop 2024-04-22T23:55:27.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.222+0000 7f1135f1c700 1 -- 172.21.15.67:0/1914907371 shutdown_connections 2024-04-22T23:55:27.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.222+0000 7f1135f1c700 1 --2- 172.21.15.67:0/1914907371 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f111c06ecc0 0x7f111c071180 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:27.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.222+0000 7f1135f1c700 1 --2- 172.21.15.67:0/1914907371 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1130100e90 0x7f113019c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:27.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.222+0000 7f1135f1c700 1 --2- 172.21.15.67:0/1914907371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1130101840 0x7f113019cc20 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:27.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.222+0000 7f1135f1c700 1 -- 172.21.15.67:0/1914907371 >> 172.21.15.67:0/1914907371 conn(0x7f11300fc740 msgr2=0x7f11300fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:55:27.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.222+0000 7f1135f1c700 1 -- 172.21.15.67:0/1914907371 shutdown_connections 2024-04-22T23:55:27.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.222+0000 7f1135f1c700 1 -- 172.21.15.67:0/1914907371 wait complete. 2024-04-22T23:55:27.223 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:55:27.224 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:27 smithi067 ceph-mon[112438]: pgmap v874: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:27.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.330+0000 7faf84ace700 1 -- 172.21.15.67:0/3678308294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf80101850 msgr2=0x7faf801058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:55:27.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.330+0000 7faf84ace700 1 --2- 172.21.15.67:0/3678308294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf80101850 0x7faf801058a0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7faf74009910 tx=0x7faf74009c20 comp rx=0 tx=0).stop 2024-04-22T23:55:27.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.330+0000 7faf84ace700 1 -- 172.21.15.67:0/3678308294 shutdown_connections 2024-04-22T23:55:27.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.330+0000 7faf84ace700 1 --2- 172.21.15.67:0/3678308294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf80101850 0x7faf801058a0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:27.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.330+0000 7faf84ace700 1 --2- 172.21.15.67:0/3678308294 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf80100ea0 0x7faf80101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:27.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.330+0000 7faf84ace700 1 -- 172.21.15.67:0/3678308294 >> 172.21.15.67:0/3678308294 conn(0x7faf800fc750 msgr2=0x7faf800feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:55:27.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.330+0000 7faf84ace700 1 -- 172.21.15.67:0/3678308294 shutdown_connections 2024-04-22T23:55:27.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.330+0000 7faf84ace700 1 -- 172.21.15.67:0/3678308294 wait complete. 2024-04-22T23:55:27.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.330+0000 7faf84ace700 1 Processor -- start 2024-04-22T23:55:27.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.331+0000 7faf84ace700 1 -- start start 2024-04-22T23:55:27.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.331+0000 7faf84ace700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf80100ea0 0x7faf8019c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:55:27.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.331+0000 7faf84ace700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf80101850 0x7faf8019cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:55:27.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.331+0000 7faf84ace700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8019d2d0 con 0x7faf80100ea0 2024-04-22T23:55:27.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.331+0000 7faf84ace700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf80196780 con 0x7faf80101850 2024-04-22T23:55:27.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.331+0000 7faf7e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf80100ea0 0x7faf8019c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:55:27.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.331+0000 7faf7e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf80100ea0 0x7faf8019c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38742/0 (socket says 172.21.15.67:38742) 2024-04-22T23:55:27.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.331+0000 7faf7e59c700 1 -- 172.21.15.67:0/2810786508 learned_addr learned my addr 172.21.15.67:0/2810786508 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:55:27.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.331+0000 7faf7dd9b700 1 --2- 172.21.15.67:0/2810786508 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf80101850 0x7faf8019cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:55:27.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.331+0000 7faf7e59c700 1 -- 172.21.15.67:0/2810786508 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf80101850 msgr2=0x7faf8019cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:55:27.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.331+0000 7faf7e59c700 1 --2- 172.21.15.67:0/2810786508 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf80101850 0x7faf8019cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:27.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.331+0000 7faf7e59c700 1 -- 172.21.15.67:0/2810786508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf740095f0 con 0x7faf80100ea0 2024-04-22T23:55:27.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.331+0000 7faf7e59c700 1 --2- 172.21.15.67:0/2810786508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf80100ea0 0x7faf8019c700 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7faf7000d8d0 tx=0x7faf7000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:55:27.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.332+0000 7faf6f7fe700 1 -- 172.21.15.67:0/2810786508 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf700098a0 con 0x7faf80100ea0 2024-04-22T23:55:27.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.332+0000 7faf6f7fe700 1 -- 172.21.15.67:0/2810786508 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faf70010460 con 0x7faf80100ea0 2024-04-22T23:55:27.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.332+0000 7faf84ace700 1 -- 172.21.15.67:0/2810786508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf80196a60 con 0x7faf80100ea0 2024-04-22T23:55:27.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.332+0000 7faf6f7fe700 1 -- 172.21.15.67:0/2810786508 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf7000f5f0 con 0x7faf80100ea0 2024-04-22T23:55:27.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.332+0000 7faf84ace700 1 -- 172.21.15.67:0/2810786508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf80196fb0 con 0x7faf80100ea0 2024-04-22T23:55:27.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.333+0000 7faf84ace700 1 -- 172.21.15.67:0/2810786508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf801092a0 con 0x7faf80100ea0 2024-04-22T23:55:27.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.333+0000 7faf6f7fe700 1 -- 172.21.15.67:0/2810786508 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7faf70009a00 con 0x7faf80100ea0 2024-04-22T23:55:27.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.334+0000 7faf6f7fe700 1 --2- 172.21.15.67:0/2810786508 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faf6806ecc0 0x7faf68071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:55:27.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.334+0000 7faf6f7fe700 1 -- 172.21.15.67:0/2810786508 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7faf7008eea0 con 0x7faf80100ea0 2024-04-22T23:55:27.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.334+0000 7faf7dd9b700 1 --2- 172.21.15.67:0/2810786508 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faf6806ecc0 0x7faf68071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:55:27.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.334+0000 7faf7dd9b700 1 --2- 172.21.15.67:0/2810786508 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faf6806ecc0 0x7faf68071180 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7faf80197bf0 tx=0x7faf7401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:55:27.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.338+0000 7faf6f7fe700 1 -- 172.21.15.67:0/2810786508 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7faf7005b610 con 0x7faf80100ea0 2024-04-22T23:55:27.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:27 smithi134 ceph-mon[85630]: pgmap v874: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:27.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.567+0000 7faf84ace700 1 -- 172.21.15.67:0/2810786508 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7faf80197da0 con 0x7faf6806ecc0 2024-04-22T23:55:27.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.569+0000 7faf6f7fe700 1 -- 172.21.15.67:0/2810786508 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7faf80197da0 con 0x7faf6806ecc0 2024-04-22T23:55:27.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.571+0000 7faf84ace700 1 -- 172.21.15.67:0/2810786508 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faf6806ecc0 msgr2=0x7faf68071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:55:27.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.571+0000 7faf84ace700 1 --2- 172.21.15.67:0/2810786508 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faf6806ecc0 0x7faf68071180 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7faf80197bf0 tx=0x7faf7401a040 comp rx=0 tx=0).stop 2024-04-22T23:55:27.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.571+0000 7faf84ace700 1 -- 172.21.15.67:0/2810786508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf80100ea0 msgr2=0x7faf8019c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:55:27.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.571+0000 7faf84ace700 1 --2- 172.21.15.67:0/2810786508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf80100ea0 0x7faf8019c700 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7faf7000d8d0 tx=0x7faf7000dbe0 comp rx=0 tx=0).stop 2024-04-22T23:55:27.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.571+0000 7faf84ace700 1 -- 172.21.15.67:0/2810786508 shutdown_connections 2024-04-22T23:55:27.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.571+0000 7faf84ace700 1 --2- 172.21.15.67:0/2810786508 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faf6806ecc0 0x7faf68071180 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:27.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.571+0000 7faf84ace700 1 --2- 172.21.15.67:0/2810786508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf80100ea0 0x7faf8019c700 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:27.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.571+0000 7faf84ace700 1 --2- 172.21.15.67:0/2810786508 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf80101850 0x7faf8019cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:27.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.571+0000 7faf84ace700 1 -- 172.21.15.67:0/2810786508 >> 172.21.15.67:0/2810786508 conn(0x7faf800fc750 msgr2=0x7faf800fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:55:27.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.571+0000 7faf84ace700 1 -- 172.21.15.67:0/2810786508 shutdown_connections 2024-04-22T23:55:27.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.571+0000 7faf84ace700 1 -- 172.21.15.67:0/2810786508 wait complete. 2024-04-22T23:55:27.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.678+0000 7fb433552700 1 -- 172.21.15.67:0/810532225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb42c101de0 msgr2=0x7fb42c1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:55:27.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.678+0000 7fb433552700 1 --2- 172.21.15.67:0/810532225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb42c101de0 0x7fb42c1021c0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fb428009910 tx=0x7fb428009c20 comp rx=0 tx=0).stop 2024-04-22T23:55:27.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.678+0000 7fb433552700 1 -- 172.21.15.67:0/810532225 shutdown_connections 2024-04-22T23:55:27.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.678+0000 7fb433552700 1 --2- 172.21.15.67:0/810532225 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb42c102700 0x7fb42c10abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:27.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.678+0000 7fb433552700 1 --2- 172.21.15.67:0/810532225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb42c101de0 0x7fb42c1021c0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:27.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.678+0000 7fb433552700 1 -- 172.21.15.67:0/810532225 >> 172.21.15.67:0/810532225 conn(0x7fb42c0fb650 msgr2=0x7fb42c0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:55:27.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.678+0000 7fb433552700 1 -- 172.21.15.67:0/810532225 shutdown_connections 2024-04-22T23:55:27.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.678+0000 7fb433552700 1 -- 172.21.15.67:0/810532225 wait complete. 2024-04-22T23:55:27.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.679+0000 7fb433552700 1 Processor -- start 2024-04-22T23:55:27.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.679+0000 7fb433552700 1 -- start start 2024-04-22T23:55:27.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.679+0000 7fb433552700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb42c101de0 0x7fb42c19c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:55:27.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.679+0000 7fb433552700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb42c102700 0x7fb42c19cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:55:27.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.679+0000 7fb433552700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c19d270 con 0x7fb42c102700 2024-04-22T23:55:27.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.679+0000 7fb433552700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c196720 con 0x7fb42c101de0 2024-04-22T23:55:27.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.679+0000 7fb430aed700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb42c102700 0x7fb42c19cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:55:27.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.679+0000 7fb430aed700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb42c102700 0x7fb42c19cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38770/0 (socket says 172.21.15.67:38770) 2024-04-22T23:55:27.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.679+0000 7fb430aed700 1 -- 172.21.15.67:0/2668367744 learned_addr learned my addr 172.21.15.67:0/2668367744 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:55:27.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.679+0000 7fb4312ee700 1 --2- 172.21.15.67:0/2668367744 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb42c101de0 0x7fb42c19c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:55:27.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.679+0000 7fb430aed700 1 -- 172.21.15.67:0/2668367744 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb42c101de0 msgr2=0x7fb42c19c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:55:27.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.679+0000 7fb430aed700 1 --2- 172.21.15.67:0/2668367744 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb42c101de0 0x7fb42c19c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:27.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.680+0000 7fb430aed700 1 -- 172.21.15.67:0/2668367744 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4280095f0 con 0x7fb42c102700 2024-04-22T23:55:27.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.680+0000 7fb430aed700 1 --2- 172.21.15.67:0/2668367744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb42c102700 0x7fb42c19cbe0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fb41c00d6d0 tx=0x7fb41c00d9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:55:27.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.680+0000 7fb4227fc700 1 -- 172.21.15.67:0/2668367744 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb41c00f860 con 0x7fb42c102700 2024-04-22T23:55:27.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.680+0000 7fb4227fc700 1 -- 172.21.15.67:0/2668367744 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb41c00fe80 con 0x7fb42c102700 2024-04-22T23:55:27.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.680+0000 7fb433552700 1 -- 172.21.15.67:0/2668367744 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb42c196a00 con 0x7fb42c102700 2024-04-22T23:55:27.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.680+0000 7fb4227fc700 1 -- 172.21.15.67:0/2668367744 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb41c010ea0 con 0x7fb42c102700 2024-04-22T23:55:27.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.680+0000 7fb433552700 1 -- 172.21.15.67:0/2668367744 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb42c196f50 con 0x7fb42c102700 2024-04-22T23:55:27.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.681+0000 7fb433552700 1 -- 172.21.15.67:0/2668367744 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb42c1082f0 con 0x7fb42c102700 2024-04-22T23:55:27.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.681+0000 7fb4227fc700 1 -- 172.21.15.67:0/2668367744 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb41c010460 con 0x7fb42c102700 2024-04-22T23:55:27.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.682+0000 7fb4227fc700 1 --2- 172.21.15.67:0/2668367744 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb41806ecc0 0x7fb418071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:55:27.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.682+0000 7fb4227fc700 1 -- 172.21.15.67:0/2668367744 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fb41c08ec20 con 0x7fb42c102700 2024-04-22T23:55:27.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.682+0000 7fb4312ee700 1 --2- 172.21.15.67:0/2668367744 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb41806ecc0 0x7fb418071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:55:27.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.682+0000 7fb4312ee700 1 --2- 172.21.15.67:0/2668367744 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb41806ecc0 0x7fb418071180 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb428006010 tx=0x7fb42801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:55:27.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.687+0000 7fb4227fc700 1 -- 172.21.15.67:0/2668367744 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb41c05b410 con 0x7fb42c102700 2024-04-22T23:55:27.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.907+0000 7fb433552700 1 -- 172.21.15.67:0/2668367744 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb42c197220 con 0x7fb41806ecc0 2024-04-22T23:55:27.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.918+0000 7fb4227fc700 1 -- 172.21.15.67:0/2668367744 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fb42c197220 con 0x7fb41806ecc0 2024-04-22T23:55:27.909 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:55:27.909 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (29m) 6m ago 40m 24.4M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:55:27.909 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (40m) 6m ago 40m 10.0M - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:55:27.909 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (39m) 6m ago 39m 10.0M - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:55:27.909 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (26m) 6m ago 40m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:55:27.910 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (26m) 6m ago 39m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:55:27.910 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (28m) 6m ago 40m 81.2M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:55:27.910 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (22m) 6m ago 35m 19.4M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:55:27.910 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (22m) 6m ago 34m 20.3M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:55:27.910 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (22m) 6m ago 34m 20.1M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:55:27.910 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (7m) 6m ago 34m 14.0M - 19.0.0-3252-g43be0201 a0434699c1d7 46d3d7e86fee 2024-04-22T23:55:27.910 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (30m) 6m ago 41m 514M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:55:27.910 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (30m) 6m ago 39m 459M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:55:27.910 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (27m) 6m ago 42m 74.2M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:55:27.910 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (27m) 6m ago 39m 54.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:55:27.910 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (29m) 6m ago 40m 15.2M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:55:27.910 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (29m) 6m ago 39m 16.0M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:55:27.910 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (26m) 6m ago 38m 102M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:55:27.910 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (25m) 6m ago 38m 128M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:55:27.911 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (24m) 6m ago 37m 83.8M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:55:27.911 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (24m) 6m ago 37m 101M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:55:27.911 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (23m) 6m ago 36m 86.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:55:27.911 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (23m) 6m ago 36m 81.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:55:27.911 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (29m) 6m ago 39m 78.7M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:55:27.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.921+0000 7fb433552700 1 -- 172.21.15.67:0/2668367744 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb41806ecc0 msgr2=0x7fb418071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:55:27.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.921+0000 7fb433552700 1 --2- 172.21.15.67:0/2668367744 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb41806ecc0 0x7fb418071180 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb428006010 tx=0x7fb42801a040 comp rx=0 tx=0).stop 2024-04-22T23:55:27.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.921+0000 7fb433552700 1 -- 172.21.15.67:0/2668367744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb42c102700 msgr2=0x7fb42c19cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:55:27.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.921+0000 7fb433552700 1 --2- 172.21.15.67:0/2668367744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb42c102700 0x7fb42c19cbe0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fb41c00d6d0 tx=0x7fb41c00d9e0 comp rx=0 tx=0).stop 2024-04-22T23:55:27.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.921+0000 7fb433552700 1 -- 172.21.15.67:0/2668367744 shutdown_connections 2024-04-22T23:55:27.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.921+0000 7fb433552700 1 --2- 172.21.15.67:0/2668367744 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb41806ecc0 0x7fb418071180 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:27.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.921+0000 7fb433552700 1 --2- 172.21.15.67:0/2668367744 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb42c101de0 0x7fb42c19c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:27.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.921+0000 7fb433552700 1 --2- 172.21.15.67:0/2668367744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb42c102700 0x7fb42c19cbe0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:27.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.921+0000 7fb433552700 1 -- 172.21.15.67:0/2668367744 >> 172.21.15.67:0/2668367744 conn(0x7fb42c0fb650 msgr2=0x7fb42c105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:55:27.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.921+0000 7fb433552700 1 -- 172.21.15.67:0/2668367744 shutdown_connections 2024-04-22T23:55:27.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:27.921+0000 7fb433552700 1 -- 172.21.15.67:0/2668367744 wait complete. 2024-04-22T23:55:28.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.027+0000 7fe383622700 1 -- 172.21.15.67:0/3315690336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe37c068ea0 msgr2=0x7fe37c10d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:55:28.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.027+0000 7fe383622700 1 --2- 172.21.15.67:0/3315690336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe37c068ea0 0x7fe37c10d290 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fe36c009910 tx=0x7fe36c009c20 comp rx=0 tx=0).stop 2024-04-22T23:55:28.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.027+0000 7fe383622700 1 -- 172.21.15.67:0/3315690336 shutdown_connections 2024-04-22T23:55:28.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.027+0000 7fe383622700 1 --2- 172.21.15.67:0/3315690336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe37c068ea0 0x7fe37c10d290 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:28.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.027+0000 7fe383622700 1 --2- 172.21.15.67:0/3315690336 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe37c068580 0x7fe37c068960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:28.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.027+0000 7fe383622700 1 -- 172.21.15.67:0/3315690336 >> 172.21.15.67:0/3315690336 conn(0x7fe37c075670 msgr2=0x7fe37c075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:55:28.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.027+0000 7fe383622700 1 -- 172.21.15.67:0/3315690336 shutdown_connections 2024-04-22T23:55:28.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.027+0000 7fe383622700 1 -- 172.21.15.67:0/3315690336 wait complete. 2024-04-22T23:55:28.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.028+0000 7fe383622700 1 Processor -- start 2024-04-22T23:55:28.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.028+0000 7fe383622700 1 -- start start 2024-04-22T23:55:28.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.028+0000 7fe383622700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe37c068580 0x7fe37c19e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:55:28.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.028+0000 7fe383622700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe37c068ea0 0x7fe37c19ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:55:28.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.028+0000 7fe383622700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe37c19f4b0 con 0x7fe37c068ea0 2024-04-22T23:55:28.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.028+0000 7fe383622700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe37c198960 con 0x7fe37c068580 2024-04-22T23:55:28.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.028+0000 7fe380bbd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe37c068ea0 0x7fe37c19ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:55:28.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.028+0000 7fe380bbd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe37c068ea0 0x7fe37c19ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38788/0 (socket says 172.21.15.67:38788) 2024-04-22T23:55:28.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.028+0000 7fe380bbd700 1 -- 172.21.15.67:0/3360756330 learned_addr learned my addr 172.21.15.67:0/3360756330 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:55:28.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.028+0000 7fe3813be700 1 --2- 172.21.15.67:0/3360756330 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe37c068580 0x7fe37c19e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:55:28.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.028+0000 7fe380bbd700 1 -- 172.21.15.67:0/3360756330 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe37c068580 msgr2=0x7fe37c19e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:55:28.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.028+0000 7fe380bbd700 1 --2- 172.21.15.67:0/3360756330 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe37c068580 0x7fe37c19e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:28.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.028+0000 7fe380bbd700 1 -- 172.21.15.67:0/3360756330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe36c0095f0 con 0x7fe37c068ea0 2024-04-22T23:55:28.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.028+0000 7fe3813be700 1 --2- 172.21.15.67:0/3360756330 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe37c068580 0x7fe37c19e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:55:28.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.028+0000 7fe380bbd700 1 --2- 172.21.15.67:0/3360756330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe37c068ea0 0x7fe37c19ee20 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fe36c006010 tx=0x7fe36c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:55:28.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.029+0000 7fe3727fc700 1 -- 172.21.15.67:0/3360756330 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe36c01d070 con 0x7fe37c068ea0 2024-04-22T23:55:28.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.029+0000 7fe3727fc700 1 -- 172.21.15.67:0/3360756330 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe36c004fb0 con 0x7fe37c068ea0 2024-04-22T23:55:28.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.029+0000 7fe3727fc700 1 -- 172.21.15.67:0/3360756330 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe36c022ea0 con 0x7fe37c068ea0 2024-04-22T23:55:28.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.029+0000 7fe383622700 1 -- 172.21.15.67:0/3360756330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe37c198be0 con 0x7fe37c068ea0 2024-04-22T23:55:28.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.029+0000 7fe383622700 1 -- 172.21.15.67:0/3360756330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe37c1990d0 con 0x7fe37c068ea0 2024-04-22T23:55:28.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.030+0000 7fe383622700 1 -- 172.21.15.67:0/3360756330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe37c10aa00 con 0x7fe37c068ea0 2024-04-22T23:55:28.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.030+0000 7fe3727fc700 1 -- 172.21.15.67:0/3360756330 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe36c022470 con 0x7fe37c068ea0 2024-04-22T23:55:28.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.031+0000 7fe3727fc700 1 --2- 172.21.15.67:0/3360756330 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe36806ecc0 0x7fe368071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:55:28.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.031+0000 7fe3727fc700 1 -- 172.21.15.67:0/3360756330 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fe36c090560 con 0x7fe37c068ea0 2024-04-22T23:55:28.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.031+0000 7fe3813be700 1 --2- 172.21.15.67:0/3360756330 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe36806ecc0 0x7fe368071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:55:28.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.031+0000 7fe3813be700 1 --2- 172.21.15.67:0/3360756330 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe36806ecc0 0x7fe368071180 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fe378005950 tx=0x7fe378005f50 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:55:28.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.035+0000 7fe3727fc700 1 -- 172.21.15.67:0/3360756330 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe36c05cd60 con 0x7fe37c068ea0 2024-04-22T23:55:28.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.335+0000 7fe383622700 1 -- 172.21.15.67:0/3360756330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe37c04f2e0 con 0x7fe37c068ea0 2024-04-22T23:55:28.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.336+0000 7fe3727fc700 1 -- 172.21.15.67:0/3360756330 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fe36c022720 con 0x7fe37c068ea0 2024-04-22T23:55:28.327 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:55:28.327 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:55:28.327 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:55:28.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:55:28.327 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:55:28.327 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:55:28.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:55:28.327 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:55:28.327 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:55:28.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:55:28.327 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:55:28.327 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 4 2024-04-22T23:55:28.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:55:28.328 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:55:28.328 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 14 2024-04-22T23:55:28.328 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:55:28.328 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:55:28.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.338+0000 7fe383622700 1 -- 172.21.15.67:0/3360756330 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe36806ecc0 msgr2=0x7fe368071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:55:28.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.338+0000 7fe383622700 1 --2- 172.21.15.67:0/3360756330 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe36806ecc0 0x7fe368071180 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fe378005950 tx=0x7fe378005f50 comp rx=0 tx=0).stop 2024-04-22T23:55:28.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.338+0000 7fe383622700 1 -- 172.21.15.67:0/3360756330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe37c068ea0 msgr2=0x7fe37c19ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:55:28.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.338+0000 7fe383622700 1 --2- 172.21.15.67:0/3360756330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe37c068ea0 0x7fe37c19ee20 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fe36c006010 tx=0x7fe36c00be20 comp rx=0 tx=0).stop 2024-04-22T23:55:28.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.338+0000 7fe383622700 1 -- 172.21.15.67:0/3360756330 shutdown_connections 2024-04-22T23:55:28.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.338+0000 7fe383622700 1 --2- 172.21.15.67:0/3360756330 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe36806ecc0 0x7fe368071180 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:28.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.338+0000 7fe383622700 1 --2- 172.21.15.67:0/3360756330 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe37c068580 0x7fe37c19e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:28.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.338+0000 7fe383622700 1 --2- 172.21.15.67:0/3360756330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe37c068ea0 0x7fe37c19ee20 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:28.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.338+0000 7fe383622700 1 -- 172.21.15.67:0/3360756330 >> 172.21.15.67:0/3360756330 conn(0x7fe37c075670 msgr2=0x7fe37c0fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:55:28.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.338+0000 7fe383622700 1 -- 172.21.15.67:0/3360756330 shutdown_connections 2024-04-22T23:55:28.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.338+0000 7fe383622700 1 -- 172.21.15.67:0/3360756330 wait complete. 2024-04-22T23:55:28.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.445+0000 7fdc98cb3700 1 -- 172.21.15.67:0/1250693670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc94101850 msgr2=0x7fdc941058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:55:28.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.445+0000 7fdc98cb3700 1 --2- 172.21.15.67:0/1250693670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc94101850 0x7fdc941058a0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fdc88009910 tx=0x7fdc88009c20 comp rx=0 tx=0).stop 2024-04-22T23:55:28.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.445+0000 7fdc98cb3700 1 -- 172.21.15.67:0/1250693670 shutdown_connections 2024-04-22T23:55:28.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.445+0000 7fdc98cb3700 1 --2- 172.21.15.67:0/1250693670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc94101850 0x7fdc941058a0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:28.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.445+0000 7fdc98cb3700 1 --2- 172.21.15.67:0/1250693670 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdc94100ea0 0x7fdc94101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:28.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.445+0000 7fdc98cb3700 1 -- 172.21.15.67:0/1250693670 >> 172.21.15.67:0/1250693670 conn(0x7fdc940fc750 msgr2=0x7fdc940feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:55:28.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.445+0000 7fdc98cb3700 1 -- 172.21.15.67:0/1250693670 shutdown_connections 2024-04-22T23:55:28.437 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:28 smithi067 ceph-mon[112438]: from='client.35580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:55:28.437 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:28 smithi067 ceph-mon[112438]: from='client.35584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:55:28.437 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:28 smithi067 ceph-mon[112438]: from='client.35588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:55:28.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.445+0000 7fdc98cb3700 1 -- 172.21.15.67:0/1250693670 wait complete. 2024-04-22T23:55:28.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.446+0000 7fdc98cb3700 1 Processor -- start 2024-04-22T23:55:28.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.446+0000 7fdc98cb3700 1 -- start start 2024-04-22T23:55:28.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.446+0000 7fdc98cb3700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdc94100ea0 0x7fdc9419e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:55:28.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.446+0000 7fdc98cb3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc94101850 0x7fdc9419ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:55:28.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.446+0000 7fdc98cb3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc9419f410 con 0x7fdc94101850 2024-04-22T23:55:28.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.446+0000 7fdc98cb3700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc94198950 con 0x7fdc94100ea0 2024-04-22T23:55:28.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.446+0000 7fdc91d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc94101850 0x7fdc9419ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:55:28.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.446+0000 7fdc91d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc94101850 0x7fdc9419ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38816/0 (socket says 172.21.15.67:38816) 2024-04-22T23:55:28.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.446+0000 7fdc91d9b700 1 -- 172.21.15.67:0/4099898597 learned_addr learned my addr 172.21.15.67:0/4099898597 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:55:28.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.446+0000 7fdc9259c700 1 --2- 172.21.15.67:0/4099898597 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdc94100ea0 0x7fdc9419e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:55:28.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.446+0000 7fdc91d9b700 1 -- 172.21.15.67:0/4099898597 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdc94100ea0 msgr2=0x7fdc9419e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:55:28.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.446+0000 7fdc91d9b700 1 --2- 172.21.15.67:0/4099898597 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdc94100ea0 0x7fdc9419e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:28.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.446+0000 7fdc91d9b700 1 -- 172.21.15.67:0/4099898597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc880095f0 con 0x7fdc94101850 2024-04-22T23:55:28.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.447+0000 7fdc91d9b700 1 --2- 172.21.15.67:0/4099898597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc94101850 0x7fdc9419ee10 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fdc8800bda0 tx=0x7fdc8800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:55:28.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.447+0000 7fdc837fe700 1 -- 172.21.15.67:0/4099898597 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc8801d070 con 0x7fdc94101850 2024-04-22T23:55:28.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.447+0000 7fdc837fe700 1 -- 172.21.15.67:0/4099898597 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdc88004fb0 con 0x7fdc94101850 2024-04-22T23:55:28.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.447+0000 7fdc837fe700 1 -- 172.21.15.67:0/4099898597 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc8800f480 con 0x7fdc94101850 2024-04-22T23:55:28.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.447+0000 7fdc98cb3700 1 -- 172.21.15.67:0/4099898597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc94198bd0 con 0x7fdc94101850 2024-04-22T23:55:28.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.447+0000 7fdc98cb3700 1 -- 172.21.15.67:0/4099898597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc941990c0 con 0x7fdc94101850 2024-04-22T23:55:28.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.448+0000 7fdc98cb3700 1 -- 172.21.15.67:0/4099898597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc94105020 con 0x7fdc94101850 2024-04-22T23:55:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.448+0000 7fdc837fe700 1 -- 172.21.15.67:0/4099898597 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdc88004af0 con 0x7fdc94101850 2024-04-22T23:55:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.449+0000 7fdc837fe700 1 --2- 172.21.15.67:0/4099898597 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdc7c06ecc0 0x7fdc7c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:55:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.449+0000 7fdc837fe700 1 -- 172.21.15.67:0/4099898597 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fdc88090450 con 0x7fdc94101850 2024-04-22T23:55:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.449+0000 7fdc9259c700 1 --2- 172.21.15.67:0/4099898597 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdc7c06ecc0 0x7fdc7c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:55:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.450+0000 7fdc9259c700 1 --2- 172.21.15.67:0/4099898597 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdc7c06ecc0 0x7fdc7c071180 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fdc84005ea0 tx=0x7fdc84005e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:55:28.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.454+0000 7fdc837fe700 1 -- 172.21.15.67:0/4099898597 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fdc8805ccf0 con 0x7fdc94101850 2024-04-22T23:55:28.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:28 smithi134 ceph-mon[85630]: from='client.35580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:55:28.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:28 smithi134 ceph-mon[85630]: from='client.35584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:55:28.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:28 smithi134 ceph-mon[85630]: from='client.35588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:55:28.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.710+0000 7fdc98cb3700 1 -- 172.21.15.67:0/4099898597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fdc94199e60 con 0x7fdc94101850 2024-04-22T23:55:28.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.711+0000 7fdc837fe700 1 -- 172.21.15.67:0/4099898597 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1872 (secure 0 0 0) 0x7fdc88027760 con 0x7fdc94101850 2024-04-22T23:55:28.702 INFO:teuthology.orchestra.run.smithi067.stdout:e33 2024-04-22T23:55:28.702 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:55:28.702 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:55:28.702 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:55:28.702 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:55:28.702 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:55:28.702 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:55:28.702 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:55:28.702 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:55:28.702 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:55:28.703 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:55:28.703 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:55:28.703 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:55:28.703 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:55:28.703 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:55:28.703 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:55:28.703 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:55:28.703 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:55:28.703 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:55:28.703 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:55:28.703 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:55:28.703 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:55:28.703 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:55:28.703 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:55:28.703 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:55:28.704 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:55:28.704 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:55:28.704 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:55:28.704 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:55:28.704 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:55:28.704 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:55:28.704 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:55:28.704 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:55:28.704 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:55:28.704 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:55:28.704 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:55:28.704 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:55:28.704 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:55:28.704 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:55:28.705 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:55:28.705 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:55:28.705 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44851} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1849648622,v1:172.21.15.134:6827/1849648622] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:55:28.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.713+0000 7fdc98cb3700 1 -- 172.21.15.67:0/4099898597 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdc7c06ecc0 msgr2=0x7fdc7c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:55:28.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.713+0000 7fdc98cb3700 1 --2- 172.21.15.67:0/4099898597 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdc7c06ecc0 0x7fdc7c071180 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fdc84005ea0 tx=0x7fdc84005e30 comp rx=0 tx=0).stop 2024-04-22T23:55:28.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.713+0000 7fdc98cb3700 1 -- 172.21.15.67:0/4099898597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc94101850 msgr2=0x7fdc9419ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:55:28.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.713+0000 7fdc98cb3700 1 --2- 172.21.15.67:0/4099898597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc94101850 0x7fdc9419ee10 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fdc8800bda0 tx=0x7fdc8800be80 comp rx=0 tx=0).stop 2024-04-22T23:55:28.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.713+0000 7fdc98cb3700 1 -- 172.21.15.67:0/4099898597 shutdown_connections 2024-04-22T23:55:28.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.713+0000 7fdc98cb3700 1 --2- 172.21.15.67:0/4099898597 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdc7c06ecc0 0x7fdc7c071180 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:28.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.713+0000 7fdc98cb3700 1 --2- 172.21.15.67:0/4099898597 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdc94100ea0 0x7fdc9419e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:28.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.713+0000 7fdc98cb3700 1 --2- 172.21.15.67:0/4099898597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc94101850 0x7fdc9419ee10 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:28.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.713+0000 7fdc98cb3700 1 -- 172.21.15.67:0/4099898597 >> 172.21.15.67:0/4099898597 conn(0x7fdc940fc750 msgr2=0x7fdc940feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:55:28.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.713+0000 7fdc98cb3700 1 -- 172.21.15.67:0/4099898597 shutdown_connections 2024-04-22T23:55:28.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.713+0000 7fdc98cb3700 1 -- 172.21.15.67:0/4099898597 wait complete. 2024-04-22T23:55:28.706 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 33 2024-04-22T23:55:28.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.820+0000 7f50a59c9700 1 -- 172.21.15.67:0/3837973613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a0102700 msgr2=0x7f50a010abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:55:28.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.820+0000 7f50a59c9700 1 --2- 172.21.15.67:0/3837973613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a0102700 0x7f50a010abf0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f5094009910 tx=0x7f5094009c20 comp rx=0 tx=0).stop 2024-04-22T23:55:28.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.820+0000 7f50a59c9700 1 -- 172.21.15.67:0/3837973613 shutdown_connections 2024-04-22T23:55:28.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.820+0000 7f50a59c9700 1 --2- 172.21.15.67:0/3837973613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a0102700 0x7f50a010abf0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:28.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.820+0000 7f50a59c9700 1 --2- 172.21.15.67:0/3837973613 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f50a0101de0 0x7f50a01021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:28.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.820+0000 7f50a59c9700 1 -- 172.21.15.67:0/3837973613 >> 172.21.15.67:0/3837973613 conn(0x7f50a00fb650 msgr2=0x7f50a00fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:55:28.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.820+0000 7f50a59c9700 1 -- 172.21.15.67:0/3837973613 shutdown_connections 2024-04-22T23:55:28.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.820+0000 7f50a59c9700 1 -- 172.21.15.67:0/3837973613 wait complete. 2024-04-22T23:55:28.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.821+0000 7f50a59c9700 1 Processor -- start 2024-04-22T23:55:28.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.821+0000 7f50a59c9700 1 -- start start 2024-04-22T23:55:28.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.821+0000 7f50a59c9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a0101de0 0x7f50a019c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:55:28.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.821+0000 7f50a59c9700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f50a0102700 0x7f50a019cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:55:28.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.821+0000 7f50a59c9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a019d270 con 0x7f50a0101de0 2024-04-22T23:55:28.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.821+0000 7f50a59c9700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a0196720 con 0x7f50a0102700 2024-04-22T23:55:28.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.822+0000 7f509effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a0101de0 0x7f50a019c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:55:28.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.822+0000 7f509effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a0101de0 0x7f50a019c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42976/0 (socket says 172.21.15.67:42976) 2024-04-22T23:55:28.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.822+0000 7f509effd700 1 -- 172.21.15.67:0/2491263141 learned_addr learned my addr 172.21.15.67:0/2491263141 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:55:28.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.822+0000 7f509effd700 1 -- 172.21.15.67:0/2491263141 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f50a0102700 msgr2=0x7f50a019cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:55:28.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.822+0000 7f509effd700 1 --2- 172.21.15.67:0/2491263141 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f50a0102700 0x7f50a019cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:28.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.822+0000 7f509effd700 1 -- 172.21.15.67:0/2491263141 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50940095f0 con 0x7f50a0101de0 2024-04-22T23:55:28.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.822+0000 7f509effd700 1 --2- 172.21.15.67:0/2491263141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a0101de0 0x7f50a019c6a0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f509000c8a0 tx=0x7f509000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:55:28.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.822+0000 7f50a49c7700 1 -- 172.21.15.67:0/2491263141 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f509000cea0 con 0x7f50a0101de0 2024-04-22T23:55:28.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.822+0000 7f50a59c9700 1 -- 172.21.15.67:0/2491263141 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50a0196a00 con 0x7f50a0101de0 2024-04-22T23:55:28.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.822+0000 7f50a49c7700 1 -- 172.21.15.67:0/2491263141 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5090004830 con 0x7f50a0101de0 2024-04-22T23:55:28.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.822+0000 7f50a49c7700 1 -- 172.21.15.67:0/2491263141 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50900055b0 con 0x7f50a0101de0 2024-04-22T23:55:28.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.822+0000 7f50a59c9700 1 -- 172.21.15.67:0/2491263141 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50a0196f50 con 0x7f50a0101de0 2024-04-22T23:55:28.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.823+0000 7f50a59c9700 1 -- 172.21.15.67:0/2491263141 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50a01082f0 con 0x7f50a0101de0 2024-04-22T23:55:28.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.824+0000 7f50a49c7700 1 -- 172.21.15.67:0/2491263141 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f50900049a0 con 0x7f50a0101de0 2024-04-22T23:55:28.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.824+0000 7f50a49c7700 1 --2- 172.21.15.67:0/2491263141 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f508806ecc0 0x7f5088071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:55:28.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.824+0000 7f50a49c7700 1 -- 172.21.15.67:0/2491263141 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f509008ecc0 con 0x7f50a0101de0 2024-04-22T23:55:28.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.824+0000 7f509e7fc700 1 --2- 172.21.15.67:0/2491263141 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f508806ecc0 0x7f5088071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:55:28.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.825+0000 7f509e7fc700 1 --2- 172.21.15.67:0/2491263141 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f508806ecc0 0x7f5088071180 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f509400b5c0 tx=0x7f5094005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:55:28.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:28.829+0000 7f50a49c7700 1 -- 172.21.15.67:0/2491263141 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f509005b430 con 0x7f50a0101de0 2024-04-22T23:55:29.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.061+0000 7f50a59c9700 1 -- 172.21.15.67:0/2491263141 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f50a0197220 con 0x7f508806ecc0 2024-04-22T23:55:29.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.062+0000 7f50a49c7700 1 -- 172.21.15.67:0/2491263141 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f50a0197220 con 0x7f508806ecc0 2024-04-22T23:55:29.053 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:55:29.053 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:55:29.053 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:55:29.053 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:55:29.054 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:55:29.054 INFO:teuthology.orchestra.run.smithi067.stdout: "mds", 2024-04-22T23:55:29.054 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:55:29.054 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:55:29.054 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:55:29.054 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:55:29.054 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:55:29.054 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T23:55:29.054 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:55:29.054 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:55:29.054 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:55:29.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.065+0000 7f50a59c9700 1 -- 172.21.15.67:0/2491263141 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f508806ecc0 msgr2=0x7f5088071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:55:29.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.065+0000 7f50a59c9700 1 --2- 172.21.15.67:0/2491263141 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f508806ecc0 0x7f5088071180 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f509400b5c0 tx=0x7f5094005960 comp rx=0 tx=0).stop 2024-04-22T23:55:29.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.065+0000 7f50a59c9700 1 -- 172.21.15.67:0/2491263141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a0101de0 msgr2=0x7f50a019c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:55:29.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.065+0000 7f50a59c9700 1 --2- 172.21.15.67:0/2491263141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a0101de0 0x7f50a019c6a0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f509000c8a0 tx=0x7f509000cc60 comp rx=0 tx=0).stop 2024-04-22T23:55:29.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.065+0000 7f50a59c9700 1 -- 172.21.15.67:0/2491263141 shutdown_connections 2024-04-22T23:55:29.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.065+0000 7f50a59c9700 1 --2- 172.21.15.67:0/2491263141 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f508806ecc0 0x7f5088071180 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:29.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.065+0000 7f50a59c9700 1 --2- 172.21.15.67:0/2491263141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a0101de0 0x7f50a019c6a0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:29.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.065+0000 7f50a59c9700 1 --2- 172.21.15.67:0/2491263141 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f50a0102700 0x7f50a019cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:29.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.065+0000 7f50a59c9700 1 -- 172.21.15.67:0/2491263141 >> 172.21.15.67:0/2491263141 conn(0x7f50a00fb650 msgr2=0x7f50a0105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:55:29.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.065+0000 7f50a59c9700 1 -- 172.21.15.67:0/2491263141 shutdown_connections 2024-04-22T23:55:29.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.065+0000 7f50a59c9700 1 -- 172.21.15.67:0/2491263141 wait complete. 2024-04-22T23:55:29.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.171+0000 7fbd2affb700 1 -- 172.21.15.67:0/3611638280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd240ff5a0 msgr2=0x7fbd240ff980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:55:29.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.171+0000 7fbd2affb700 1 --2- 172.21.15.67:0/3611638280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd240ff5a0 0x7fbd240ff980 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c009910 tx=0x7fbd0c009c20 comp rx=0 tx=0).stop 2024-04-22T23:55:29.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.171+0000 7fbd2affb700 1 -- 172.21.15.67:0/3611638280 shutdown_connections 2024-04-22T23:55:29.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.171+0000 7fbd2affb700 1 --2- 172.21.15.67:0/3611638280 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd240fff50 0x7fbd24105840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:29.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.171+0000 7fbd2affb700 1 --2- 172.21.15.67:0/3611638280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd240ff5a0 0x7fbd240ff980 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:29.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.171+0000 7fbd2affb700 1 -- 172.21.15.67:0/3611638280 >> 172.21.15.67:0/3611638280 conn(0x7fbd24074e30 msgr2=0x7fbd24075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:55:29.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.171+0000 7fbd2affb700 1 -- 172.21.15.67:0/3611638280 shutdown_connections 2024-04-22T23:55:29.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.171+0000 7fbd2affb700 1 -- 172.21.15.67:0/3611638280 wait complete. 2024-04-22T23:55:29.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.172+0000 7fbd2affb700 1 Processor -- start 2024-04-22T23:55:29.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.172+0000 7fbd2affb700 1 -- start start 2024-04-22T23:55:29.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.172+0000 7fbd2affb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd240ff5a0 0x7fbd2419c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:55:29.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.172+0000 7fbd2affb700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd240fff50 0x7fbd2419cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:55:29.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.172+0000 7fbd2affb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2419d250 con 0x7fbd240ff5a0 2024-04-22T23:55:29.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.172+0000 7fbd2affb700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd24196700 con 0x7fbd240fff50 2024-04-22T23:55:29.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.172+0000 7fbd28d97700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd240ff5a0 0x7fbd2419c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:55:29.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.172+0000 7fbd28d97700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd240ff5a0 0x7fbd2419c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42994/0 (socket says 172.21.15.67:42994) 2024-04-22T23:55:29.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.172+0000 7fbd28d97700 1 -- 172.21.15.67:0/2377691507 learned_addr learned my addr 172.21.15.67:0/2377691507 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:55:29.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.172+0000 7fbd23fff700 1 --2- 172.21.15.67:0/2377691507 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd240fff50 0x7fbd2419cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:55:29.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.173+0000 7fbd28d97700 1 -- 172.21.15.67:0/2377691507 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd240fff50 msgr2=0x7fbd2419cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:55:29.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.173+0000 7fbd28d97700 1 --2- 172.21.15.67:0/2377691507 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd240fff50 0x7fbd2419cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:29.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.173+0000 7fbd28d97700 1 -- 172.21.15.67:0/2377691507 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd0c0095f0 con 0x7fbd240ff5a0 2024-04-22T23:55:29.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.173+0000 7fbd28d97700 1 --2- 172.21.15.67:0/2377691507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd240ff5a0 0x7fbd2419c680 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c0092e0 tx=0x7fbd0c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:55:29.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.173+0000 7fbd21ffb700 1 -- 172.21.15.67:0/2377691507 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd0c01d070 con 0x7fbd240ff5a0 2024-04-22T23:55:29.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.173+0000 7fbd2affb700 1 -- 172.21.15.67:0/2377691507 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd24196980 con 0x7fbd240ff5a0 2024-04-22T23:55:29.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.173+0000 7fbd21ffb700 1 -- 172.21.15.67:0/2377691507 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbd0c005e00 con 0x7fbd240ff5a0 2024-04-22T23:55:29.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.173+0000 7fbd21ffb700 1 -- 172.21.15.67:0/2377691507 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd0c017400 con 0x7fbd240ff5a0 2024-04-22T23:55:29.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.173+0000 7fbd2affb700 1 -- 172.21.15.67:0/2377691507 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd24196e70 con 0x7fbd240ff5a0 2024-04-22T23:55:29.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.174+0000 7fbd2affb700 1 -- 172.21.15.67:0/2377691507 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd24109270 con 0x7fbd240ff5a0 2024-04-22T23:55:29.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.175+0000 7fbd21ffb700 1 -- 172.21.15.67:0/2377691507 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbd0c00f460 con 0x7fbd240ff5a0 2024-04-22T23:55:29.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.175+0000 7fbd21ffb700 1 --2- 172.21.15.67:0/2377691507 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbd1006ecc0 0x7fbd10071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:55:29.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.175+0000 7fbd21ffb700 1 -- 172.21.15.67:0/2377691507 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fbd0c090500 con 0x7fbd240ff5a0 2024-04-22T23:55:29.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.175+0000 7fbd23fff700 1 --2- 172.21.15.67:0/2377691507 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbd1006ecc0 0x7fbd10071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:55:29.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.176+0000 7fbd23fff700 1 --2- 172.21.15.67:0/2377691507 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbd1006ecc0 0x7fbd10071180 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fbd14005fd0 tx=0x7fbd14005f60 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:55:29.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.180+0000 7fbd21ffb700 1 -- 172.21.15.67:0/2377691507 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fbd0c05cc70 con 0x7fbd240ff5a0 2024-04-22T23:55:29.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.478+0000 7fbd2affb700 1 -- 172.21.15.67:0/2377691507 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fbd24197d10 con 0x7fbd240ff5a0 2024-04-22T23:55:29.469 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:29 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3360756330' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:55:29.469 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:29 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/4099898597' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:55:29.469 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:29 smithi067 ceph-mon[112438]: pgmap v875: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:29.469 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:29 smithi067 ceph-mon[112438]: from='client.35600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:55:29.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.478+0000 7fbd21ffb700 1 -- 172.21.15.67:0/2377691507 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4505 (secure 0 0 0) 0x7fbd24197d10 con 0x7fbd240ff5a0 2024-04-22T23:55:29.470 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:55:29.470 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T23:55:29.470 INFO:teuthology.orchestra.run.smithi067.stdout: host smithi134 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-22T23:55:29.470 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T23:55:29.470 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:55:29.470 INFO:teuthology.orchestra.run.smithi067.stdout:Traceback (most recent call last): 2024-04-22T23:55:29.470 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T23:55:29.470 INFO:teuthology.orchestra.run.smithi067.stdout: return _run_code(code, main_globals, None, 2024-04-22T23:55:29.470 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T23:55:29.470 INFO:teuthology.orchestra.run.smithi067.stdout: exec(code, run_globals) 2024-04-22T23:55:29.470 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5642, in 2024-04-22T23:55:29.470 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5630, in main 2024-04-22T23:55:29.470 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 408, in _infer_config 2024-04-22T23:55:29.471 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 323, in _infer_fsid 2024-04-22T23:55:29.471 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 436, in _infer_image 2024-04-22T23:55:29.471 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 310, in _validate_fsid 2024-04-22T23:55:29.471 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 3380, in command_ceph_volume 2024-04-22T23:55:29.471 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T23:55:29.471 INFO:teuthology.orchestra.run.smithi067.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:55:29.471 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:55:29.471 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:55:29.471 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:55:29.471 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:55:29.471 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:55:29.471 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:55:29.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.480+0000 7fbd2affb700 1 -- 172.21.15.67:0/2377691507 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbd1006ecc0 msgr2=0x7fbd10071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:55:29.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.480+0000 7fbd2affb700 1 --2- 172.21.15.67:0/2377691507 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbd1006ecc0 0x7fbd10071180 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fbd14005fd0 tx=0x7fbd14005f60 comp rx=0 tx=0).stop 2024-04-22T23:55:29.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.480+0000 7fbd2affb700 1 -- 172.21.15.67:0/2377691507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd240ff5a0 msgr2=0x7fbd2419c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:55:29.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.480+0000 7fbd2affb700 1 --2- 172.21.15.67:0/2377691507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd240ff5a0 0x7fbd2419c680 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c0092e0 tx=0x7fbd0c00be20 comp rx=0 tx=0).stop 2024-04-22T23:55:29.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.480+0000 7fbd2affb700 1 -- 172.21.15.67:0/2377691507 shutdown_connections 2024-04-22T23:55:29.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.480+0000 7fbd2affb700 1 --2- 172.21.15.67:0/2377691507 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbd1006ecc0 0x7fbd10071180 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:29.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.480+0000 7fbd2affb700 1 --2- 172.21.15.67:0/2377691507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd240ff5a0 0x7fbd2419c680 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:29.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.480+0000 7fbd2affb700 1 --2- 172.21.15.67:0/2377691507 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd240fff50 0x7fbd2419cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:29.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.480+0000 7fbd2affb700 1 -- 172.21.15.67:0/2377691507 >> 172.21.15.67:0/2377691507 conn(0x7fbd24074e30 msgr2=0x7fbd240fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:55:29.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.480+0000 7fbd2affb700 1 -- 172.21.15.67:0/2377691507 shutdown_connections 2024-04-22T23:55:29.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:29.481+0000 7fbd2affb700 1 -- 172.21.15.67:0/2377691507 wait complete. 2024-04-22T23:55:29.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:29 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3360756330' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:55:29.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:29 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/4099898597' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:55:29.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:29 smithi134 ceph-mon[85630]: pgmap v875: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:29.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:29 smithi134 ceph-mon[85630]: from='client.35600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:55:30.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:30 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2377691507' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:55:30.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:30 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2377691507' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:55:31.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:31 smithi134 ceph-mon[85630]: pgmap v876: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:31.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:31 smithi067 ceph-mon[112438]: pgmap v876: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:33.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:33 smithi134 ceph-mon[85630]: pgmap v877: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:33.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:33 smithi067 ceph-mon[112438]: pgmap v877: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:35.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:35 smithi134 ceph-mon[85630]: pgmap v878: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:35.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:35 smithi067 ceph-mon[112438]: pgmap v878: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:37.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:37 smithi134 ceph-mon[85630]: pgmap v879: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:37.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:37 smithi067 ceph-mon[112438]: pgmap v879: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:39.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:39 smithi134 ceph-mon[85630]: pgmap v880: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:39.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:39 smithi067 ceph-mon[112438]: pgmap v880: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:41.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:41 smithi134 ceph-mon[85630]: pgmap v881: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:41.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:55:41.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:41 smithi067 ceph-mon[112438]: pgmap v881: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:41.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:55:43.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:43 smithi134 ceph-mon[85630]: pgmap v882: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:43.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:43 smithi067 ceph-mon[112438]: pgmap v882: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:45.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:45 smithi134 ceph-mon[85630]: pgmap v883: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:45.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:45 smithi067 ceph-mon[112438]: pgmap v883: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:47.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:47 smithi134 ceph-mon[85630]: pgmap v884: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:47.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:47 smithi067 ceph-mon[112438]: pgmap v884: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:49.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:49 smithi134 ceph-mon[85630]: pgmap v885: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:49.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:49 smithi067 ceph-mon[112438]: pgmap v885: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:51.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:51 smithi134 ceph-mon[85630]: pgmap v886: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:51.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:51 smithi067 ceph-mon[112438]: pgmap v886: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:53.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:53 smithi134 ceph-mon[85630]: pgmap v887: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:53.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:53 smithi067 ceph-mon[112438]: pgmap v887: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:55.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:55 smithi134 ceph-mon[85630]: pgmap v888: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:55.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:55 smithi067 ceph-mon[112438]: pgmap v888: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:56.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:55:56.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:55:57.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:57 smithi134 ceph-mon[85630]: pgmap v889: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:57.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:57 smithi067 ceph-mon[112438]: pgmap v889: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:59.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:55:59 smithi134 ceph-mon[85630]: pgmap v890: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:59.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:55:59 smithi067 ceph-mon[112438]: pgmap v890: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:55:59.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.602+0000 7f9cf5f96700 1 -- 172.21.15.67:0/1220181915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf0068580 msgr2=0x7f9cf00689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:55:59.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.602+0000 7f9cf5f96700 1 --2- 172.21.15.67:0/1220181915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf0068580 0x7f9cf00689e0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f9ce4009910 tx=0x7f9ce4009c20 comp rx=0 tx=0).stop 2024-04-22T23:55:59.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.603+0000 7f9cf5f96700 1 -- 172.21.15.67:0/1220181915 shutdown_connections 2024-04-22T23:55:59.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.603+0000 7f9cf5f96700 1 --2- 172.21.15.67:0/1220181915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf0068580 0x7f9cf00689e0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:59.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.603+0000 7f9cf5f96700 1 --2- 172.21.15.67:0/1220181915 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9cf0106cd0 0x7f9cf01070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:59.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.603+0000 7f9cf5f96700 1 -- 172.21.15.67:0/1220181915 >> 172.21.15.67:0/1220181915 conn(0x7f9cf0075670 msgr2=0x7f9cf0075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:55:59.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.603+0000 7f9cf5f96700 1 -- 172.21.15.67:0/1220181915 shutdown_connections 2024-04-22T23:55:59.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.603+0000 7f9cf5f96700 1 -- 172.21.15.67:0/1220181915 wait complete. 2024-04-22T23:55:59.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.603+0000 7f9cf5f96700 1 Processor -- start 2024-04-22T23:55:59.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.603+0000 7f9cf5f96700 1 -- start start 2024-04-22T23:55:59.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.604+0000 7f9cf5f96700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9cf0068580 0x7f9cf00ffcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:55:59.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.604+0000 7f9cf5f96700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf0106cd0 0x7f9cf0100230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:55:59.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.604+0000 7f9cf5f96700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf0100770 con 0x7f9cf0106cd0 2024-04-22T23:55:59.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.604+0000 7f9cf5f96700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf01008b0 con 0x7f9cf0068580 2024-04-22T23:55:59.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.604+0000 7f9ceeffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf0106cd0 0x7f9cf0100230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:55:59.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.604+0000 7f9ceeffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf0106cd0 0x7f9cf0100230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41730/0 (socket says 172.21.15.67:41730) 2024-04-22T23:55:59.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.604+0000 7f9ceeffd700 1 -- 172.21.15.67:0/147889042 learned_addr learned my addr 172.21.15.67:0/147889042 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:55:59.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.604+0000 7f9cef7fe700 1 --2- 172.21.15.67:0/147889042 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9cf0068580 0x7f9cf00ffcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:55:59.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.604+0000 7f9ceeffd700 1 -- 172.21.15.67:0/147889042 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9cf0068580 msgr2=0x7f9cf00ffcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:55:59.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.604+0000 7f9ceeffd700 1 --2- 172.21.15.67:0/147889042 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9cf0068580 0x7f9cf00ffcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:59.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.604+0000 7f9ceeffd700 1 -- 172.21.15.67:0/147889042 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce40095f0 con 0x7f9cf0106cd0 2024-04-22T23:55:59.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.604+0000 7f9ceeffd700 1 --2- 172.21.15.67:0/147889042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf0106cd0 0x7f9cf0100230 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f9ce4006010 tx=0x7f9ce400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:55:59.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.605+0000 7f9cecff9700 1 -- 172.21.15.67:0/147889042 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ce401d070 con 0x7f9cf0106cd0 2024-04-22T23:55:59.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.605+0000 7f9cecff9700 1 -- 172.21.15.67:0/147889042 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ce4004fb0 con 0x7f9cf0106cd0 2024-04-22T23:55:59.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.605+0000 7f9cf5f96700 1 -- 172.21.15.67:0/147889042 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cf0100b30 con 0x7f9cf0106cd0 2024-04-22T23:55:59.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.605+0000 7f9cecff9700 1 -- 172.21.15.67:0/147889042 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ce400f690 con 0x7f9cf0106cd0 2024-04-22T23:55:59.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.605+0000 7f9cf5f96700 1 -- 172.21.15.67:0/147889042 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cf01a2d60 con 0x7f9cf0106cd0 2024-04-22T23:55:59.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.606+0000 7f9cf5f96700 1 -- 172.21.15.67:0/147889042 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cf01091d0 con 0x7f9cf0106cd0 2024-04-22T23:55:59.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.606+0000 7f9cecff9700 1 -- 172.21.15.67:0/147889042 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9ce40228a0 con 0x7f9cf0106cd0 2024-04-22T23:55:59.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.607+0000 7f9cecff9700 1 --2- 172.21.15.67:0/147889042 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9cd806ecc0 0x7f9cd8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:55:59.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.607+0000 7f9cecff9700 1 -- 172.21.15.67:0/147889042 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f9ce40907a0 con 0x7f9cf0106cd0 2024-04-22T23:55:59.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.607+0000 7f9cef7fe700 1 --2- 172.21.15.67:0/147889042 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9cd806ecc0 0x7f9cd8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:55:59.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.607+0000 7f9cef7fe700 1 --2- 172.21.15.67:0/147889042 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9cd806ecc0 0x7f9cd8071180 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f9ce0005fd0 tx=0x7f9ce000a560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:55:59.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.612+0000 7f9cecff9700 1 -- 172.21.15.67:0/147889042 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f9ce4059460 con 0x7f9cf0106cd0 2024-04-22T23:55:59.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.842+0000 7f9cf5f96700 1 -- 172.21.15.67:0/147889042 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9cf00611d0 con 0x7f9cd806ecc0 2024-04-22T23:55:59.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.843+0000 7f9cecff9700 1 -- 172.21.15.67:0/147889042 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9cf00611d0 con 0x7f9cd806ecc0 2024-04-22T23:55:59.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.845+0000 7f9cf5f96700 1 -- 172.21.15.67:0/147889042 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9cd806ecc0 msgr2=0x7f9cd8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:55:59.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.845+0000 7f9cf5f96700 1 --2- 172.21.15.67:0/147889042 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9cd806ecc0 0x7f9cd8071180 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f9ce0005fd0 tx=0x7f9ce000a560 comp rx=0 tx=0).stop 2024-04-22T23:55:59.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.845+0000 7f9cf5f96700 1 -- 172.21.15.67:0/147889042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf0106cd0 msgr2=0x7f9cf0100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:55:59.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.845+0000 7f9cf5f96700 1 --2- 172.21.15.67:0/147889042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf0106cd0 0x7f9cf0100230 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f9ce4006010 tx=0x7f9ce400be20 comp rx=0 tx=0).stop 2024-04-22T23:55:59.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.845+0000 7f9cf5f96700 1 -- 172.21.15.67:0/147889042 shutdown_connections 2024-04-22T23:55:59.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.845+0000 7f9cf5f96700 1 --2- 172.21.15.67:0/147889042 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9cd806ecc0 0x7f9cd8071180 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:59.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.845+0000 7f9cf5f96700 1 --2- 172.21.15.67:0/147889042 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9cf0068580 0x7f9cf00ffcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:59.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.845+0000 7f9cf5f96700 1 --2- 172.21.15.67:0/147889042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf0106cd0 0x7f9cf0100230 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:59.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.845+0000 7f9cf5f96700 1 -- 172.21.15.67:0/147889042 >> 172.21.15.67:0/147889042 conn(0x7f9cf0075670 msgr2=0x7f9cf00feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:55:59.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.845+0000 7f9cf5f96700 1 -- 172.21.15.67:0/147889042 shutdown_connections 2024-04-22T23:55:59.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.845+0000 7f9cf5f96700 1 -- 172.21.15.67:0/147889042 wait complete. 2024-04-22T23:55:59.846 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:55:59.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.954+0000 7f9b0fd68700 1 -- 172.21.15.67:0/1815091585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b081006b0 msgr2=0x7f9b08100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:55:59.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.954+0000 7f9b0fd68700 1 --2- 172.21.15.67:0/1815091585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b081006b0 0x7f9b08100a90 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f9b04009910 tx=0x7f9b04009c20 comp rx=0 tx=0).stop 2024-04-22T23:55:59.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.954+0000 7f9b0fd68700 1 -- 172.21.15.67:0/1815091585 shutdown_connections 2024-04-22T23:55:59.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.954+0000 7f9b0fd68700 1 --2- 172.21.15.67:0/1815091585 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9b08101060 0x7f9b081047b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:59.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.954+0000 7f9b0fd68700 1 --2- 172.21.15.67:0/1815091585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b081006b0 0x7f9b08100a90 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:59.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.954+0000 7f9b0fd68700 1 -- 172.21.15.67:0/1815091585 >> 172.21.15.67:0/1815091585 conn(0x7f9b08074e30 msgr2=0x7f9b08075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:55:59.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.954+0000 7f9b0fd68700 1 -- 172.21.15.67:0/1815091585 shutdown_connections 2024-04-22T23:55:59.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.954+0000 7f9b0fd68700 1 -- 172.21.15.67:0/1815091585 wait complete. 2024-04-22T23:55:59.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.954+0000 7f9b0fd68700 1 Processor -- start 2024-04-22T23:55:59.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.954+0000 7f9b0fd68700 1 -- start start 2024-04-22T23:55:59.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.955+0000 7f9b0fd68700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b081006b0 0x7f9b0819c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:55:59.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.955+0000 7f9b0fd68700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9b08101060 0x7f9b0819cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:55:59.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.955+0000 7f9b0fd68700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0819d270 con 0x7f9b081006b0 2024-04-22T23:55:59.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.955+0000 7f9b0fd68700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b081966f0 con 0x7f9b08101060 2024-04-22T23:55:59.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.955+0000 7f9b0db04700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b081006b0 0x7f9b0819c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:55:59.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.955+0000 7f9b0db04700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b081006b0 0x7f9b0819c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41740/0 (socket says 172.21.15.67:41740) 2024-04-22T23:55:59.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.955+0000 7f9b0db04700 1 -- 172.21.15.67:0/1306059372 learned_addr learned my addr 172.21.15.67:0/1306059372 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:55:59.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.955+0000 7f9b0d303700 1 --2- 172.21.15.67:0/1306059372 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9b08101060 0x7f9b0819cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:55:59.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.955+0000 7f9b0db04700 1 -- 172.21.15.67:0/1306059372 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9b08101060 msgr2=0x7f9b0819cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:55:59.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.955+0000 7f9b0db04700 1 --2- 172.21.15.67:0/1306059372 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9b08101060 0x7f9b0819cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:55:59.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.955+0000 7f9b0db04700 1 -- 172.21.15.67:0/1306059372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b040095f0 con 0x7f9b081006b0 2024-04-22T23:55:59.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.955+0000 7f9b0d303700 1 --2- 172.21.15.67:0/1306059372 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9b08101060 0x7f9b0819cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:55:59.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.955+0000 7f9b0db04700 1 --2- 172.21.15.67:0/1306059372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b081006b0 0x7f9b0819c650 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f9b04006010 tx=0x7f9b0400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:55:59.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.956+0000 7f9afeffd700 1 -- 172.21.15.67:0/1306059372 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b0401d070 con 0x7f9b081006b0 2024-04-22T23:55:59.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.956+0000 7f9b0fd68700 1 -- 172.21.15.67:0/1306059372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b08196970 con 0x7f9b081006b0 2024-04-22T23:55:59.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.956+0000 7f9afeffd700 1 -- 172.21.15.67:0/1306059372 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9b04004fb0 con 0x7f9b081006b0 2024-04-22T23:55:59.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.956+0000 7f9afeffd700 1 -- 172.21.15.67:0/1306059372 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b0400f690 con 0x7f9b081006b0 2024-04-22T23:55:59.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.956+0000 7f9b0fd68700 1 -- 172.21.15.67:0/1306059372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b08196e60 con 0x7f9b081006b0 2024-04-22T23:55:59.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.957+0000 7f9b0fd68700 1 -- 172.21.15.67:0/1306059372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b08109130 con 0x7f9b081006b0 2024-04-22T23:55:59.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.957+0000 7f9afeffd700 1 -- 172.21.15.67:0/1306059372 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9b040228a0 con 0x7f9b081006b0 2024-04-22T23:55:59.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.957+0000 7f9afeffd700 1 --2- 172.21.15.67:0/1306059372 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9af406ecc0 0x7f9af4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:55:59.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.958+0000 7f9afeffd700 1 -- 172.21.15.67:0/1306059372 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f9b04090840 con 0x7f9b081006b0 2024-04-22T23:55:59.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.958+0000 7f9b0d303700 1 --2- 172.21.15.67:0/1306059372 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9af406ecc0 0x7f9af4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:55:59.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.958+0000 7f9b0d303700 1 --2- 172.21.15.67:0/1306059372 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9af406ecc0 0x7f9af4071180 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f9b08197ee0 tx=0x7f9af8005f50 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:55:59.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:55:59.962+0000 7f9afeffd700 1 -- 172.21.15.67:0/1306059372 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f9b0405cfb0 con 0x7f9b081006b0 2024-04-22T23:56:00.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.194+0000 7f9b0fd68700 1 -- 172.21.15.67:0/1306059372 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9b08197080 con 0x7f9af406ecc0 2024-04-22T23:56:00.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.195+0000 7f9afeffd700 1 -- 172.21.15.67:0/1306059372 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9b08197080 con 0x7f9af406ecc0 2024-04-22T23:56:00.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.197+0000 7f9b0fd68700 1 -- 172.21.15.67:0/1306059372 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9af406ecc0 msgr2=0x7f9af4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:00.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.197+0000 7f9b0fd68700 1 --2- 172.21.15.67:0/1306059372 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9af406ecc0 0x7f9af4071180 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f9b08197ee0 tx=0x7f9af8005f50 comp rx=0 tx=0).stop 2024-04-22T23:56:00.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.197+0000 7f9b0fd68700 1 -- 172.21.15.67:0/1306059372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b081006b0 msgr2=0x7f9b0819c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:00.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.197+0000 7f9b0fd68700 1 --2- 172.21.15.67:0/1306059372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b081006b0 0x7f9b0819c650 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f9b04006010 tx=0x7f9b0400be20 comp rx=0 tx=0).stop 2024-04-22T23:56:00.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.197+0000 7f9b0fd68700 1 -- 172.21.15.67:0/1306059372 shutdown_connections 2024-04-22T23:56:00.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.197+0000 7f9b0fd68700 1 --2- 172.21.15.67:0/1306059372 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9af406ecc0 0x7f9af4071180 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:00.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.197+0000 7f9b0fd68700 1 --2- 172.21.15.67:0/1306059372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b081006b0 0x7f9b0819c650 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:00.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.197+0000 7f9b0fd68700 1 --2- 172.21.15.67:0/1306059372 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9b08101060 0x7f9b0819cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:00.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.197+0000 7f9b0fd68700 1 -- 172.21.15.67:0/1306059372 >> 172.21.15.67:0/1306059372 conn(0x7f9b08074e30 msgr2=0x7f9b08103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:56:00.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.197+0000 7f9b0fd68700 1 -- 172.21.15.67:0/1306059372 shutdown_connections 2024-04-22T23:56:00.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.197+0000 7f9b0fd68700 1 -- 172.21.15.67:0/1306059372 wait complete. 2024-04-22T23:56:00.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.306+0000 7f20d5e25700 1 -- 172.21.15.67:0/3259358254 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20d0100750 msgr2=0x7f20d01047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:00.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.306+0000 7f20d5e25700 1 --2- 172.21.15.67:0/3259358254 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20d0100750 0x7f20d01047a0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f20c4009910 tx=0x7f20c4009c20 comp rx=0 tx=0).stop 2024-04-22T23:56:00.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.306+0000 7f20d5e25700 1 -- 172.21.15.67:0/3259358254 shutdown_connections 2024-04-22T23:56:00.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.306+0000 7f20d5e25700 1 --2- 172.21.15.67:0/3259358254 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20d0100750 0x7f20d01047a0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:00.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.306+0000 7f20d5e25700 1 --2- 172.21.15.67:0/3259358254 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f20d00ffda0 0x7f20d0100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:00.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.306+0000 7f20d5e25700 1 -- 172.21.15.67:0/3259358254 >> 172.21.15.67:0/3259358254 conn(0x7f20d00fb650 msgr2=0x7f20d00fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:56:00.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.306+0000 7f20d5e25700 1 -- 172.21.15.67:0/3259358254 shutdown_connections 2024-04-22T23:56:00.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.306+0000 7f20d5e25700 1 -- 172.21.15.67:0/3259358254 wait complete. 2024-04-22T23:56:00.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.307+0000 7f20d5e25700 1 Processor -- start 2024-04-22T23:56:00.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.307+0000 7f20d5e25700 1 -- start start 2024-04-22T23:56:00.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.307+0000 7f20d5e25700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20d00ffda0 0x7f20d019e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:56:00.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.307+0000 7f20d5e25700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f20d0100750 0x7f20d019edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:56:00.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.307+0000 7f20d5e25700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d019f450 con 0x7f20d00ffda0 2024-04-22T23:56:00.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.307+0000 7f20d5e25700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d0198900 con 0x7f20d0100750 2024-04-22T23:56:00.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.308+0000 7f20cf7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20d00ffda0 0x7f20d019e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:56:00.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.308+0000 7f20cf7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20d00ffda0 0x7f20d019e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41770/0 (socket says 172.21.15.67:41770) 2024-04-22T23:56:00.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.308+0000 7f20cf7fe700 1 -- 172.21.15.67:0/229045729 learned_addr learned my addr 172.21.15.67:0/229045729 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:56:00.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.308+0000 7f20ceffd700 1 --2- 172.21.15.67:0/229045729 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f20d0100750 0x7f20d019edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:56:00.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.308+0000 7f20cf7fe700 1 -- 172.21.15.67:0/229045729 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f20d0100750 msgr2=0x7f20d019edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:00.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.308+0000 7f20cf7fe700 1 --2- 172.21.15.67:0/229045729 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f20d0100750 0x7f20d019edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:00.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.308+0000 7f20cf7fe700 1 -- 172.21.15.67:0/229045729 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c40095f0 con 0x7f20d00ffda0 2024-04-22T23:56:00.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.308+0000 7f20cf7fe700 1 --2- 172.21.15.67:0/229045729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20d00ffda0 0x7f20d019e880 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f20c000d8d0 tx=0x7f20c000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:56:00.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.308+0000 7f20ccff9700 1 -- 172.21.15.67:0/229045729 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20c00098a0 con 0x7f20d00ffda0 2024-04-22T23:56:00.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.308+0000 7f20d5e25700 1 -- 172.21.15.67:0/229045729 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20d0198be0 con 0x7f20d00ffda0 2024-04-22T23:56:00.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.308+0000 7f20ccff9700 1 -- 172.21.15.67:0/229045729 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f20c0010460 con 0x7f20d00ffda0 2024-04-22T23:56:00.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.308+0000 7f20ccff9700 1 -- 172.21.15.67:0/229045729 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20c000f5f0 con 0x7f20d00ffda0 2024-04-22T23:56:00.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.308+0000 7f20d5e25700 1 -- 172.21.15.67:0/229045729 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20d0199130 con 0x7f20d00ffda0 2024-04-22T23:56:00.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.309+0000 7f20d5e25700 1 -- 172.21.15.67:0/229045729 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20d004f2e0 con 0x7f20d00ffda0 2024-04-22T23:56:00.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.310+0000 7f20ccff9700 1 -- 172.21.15.67:0/229045729 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f20c0009a00 con 0x7f20d00ffda0 2024-04-22T23:56:00.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.310+0000 7f20ccff9700 1 --2- 172.21.15.67:0/229045729 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f20b806ecc0 0x7f20b8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:56:00.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.310+0000 7f20ccff9700 1 -- 172.21.15.67:0/229045729 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f20c008eea0 con 0x7f20d00ffda0 2024-04-22T23:56:00.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.310+0000 7f20ceffd700 1 --2- 172.21.15.67:0/229045729 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f20b806ecc0 0x7f20b8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:56:00.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.311+0000 7f20ceffd700 1 --2- 172.21.15.67:0/229045729 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f20b806ecc0 0x7f20b8071180 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f20d0104a90 tx=0x7f20c401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:56:00.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.315+0000 7f20ccff9700 1 -- 172.21.15.67:0/229045729 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f20c005b610 con 0x7f20d00ffda0 2024-04-22T23:56:00.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:00 smithi134 ceph-mon[85630]: from='client.35608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:56:00.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.535+0000 7f20d5e25700 1 -- 172.21.15.67:0/229045729 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f20d01993f0 con 0x7f20b806ecc0 2024-04-22T23:56:00.526 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:00 smithi067 ceph-mon[112438]: from='client.35608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:56:00.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.547+0000 7f20ccff9700 1 -- 172.21.15.67:0/229045729 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f20d01993f0 con 0x7f20b806ecc0 2024-04-22T23:56:00.538 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:56:00.538 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (29m) 7m ago 40m 24.4M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:56:00.538 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (41m) 7m ago 41m 10.0M - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:56:00.538 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (40m) 7m ago 40m 10.0M - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:56:00.538 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (27m) 7m ago 41m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:56:00.538 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (27m) 7m ago 40m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:56:00.538 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (29m) 7m ago 40m 81.2M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:56:00.538 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (23m) 7m ago 35m 19.4M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:56:00.538 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (23m) 7m ago 35m 20.3M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:56:00.538 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (22m) 7m ago 35m 20.1M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:56:00.538 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (7m) 7m ago 35m 14.0M - 19.0.0-3252-g43be0201 a0434699c1d7 46d3d7e86fee 2024-04-22T23:56:00.538 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (31m) 7m ago 42m 514M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:56:00.539 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (30m) 7m ago 40m 459M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:56:00.539 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (28m) 7m ago 42m 74.2M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:56:00.539 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (27m) 7m ago 39m 54.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:56:00.539 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (30m) 7m ago 41m 15.2M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:56:00.539 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (30m) 7m ago 40m 16.0M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:56:00.539 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (26m) 7m ago 39m 102M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:56:00.539 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (26m) 7m ago 38m 128M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:56:00.539 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (25m) 7m ago 38m 83.8M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:56:00.539 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (24m) 7m ago 37m 101M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:56:00.539 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (24m) 7m ago 37m 86.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:56:00.539 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (23m) 7m ago 37m 81.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:56:00.539 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (30m) 7m ago 40m 78.7M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:56:00.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.549+0000 7f20d5e25700 1 -- 172.21.15.67:0/229045729 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f20b806ecc0 msgr2=0x7f20b8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:00.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.549+0000 7f20d5e25700 1 --2- 172.21.15.67:0/229045729 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f20b806ecc0 0x7f20b8071180 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f20d0104a90 tx=0x7f20c401a040 comp rx=0 tx=0).stop 2024-04-22T23:56:00.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.549+0000 7f20d5e25700 1 -- 172.21.15.67:0/229045729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20d00ffda0 msgr2=0x7f20d019e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:00.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.549+0000 7f20d5e25700 1 --2- 172.21.15.67:0/229045729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20d00ffda0 0x7f20d019e880 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f20c000d8d0 tx=0x7f20c000dbe0 comp rx=0 tx=0).stop 2024-04-22T23:56:00.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.549+0000 7f20d5e25700 1 -- 172.21.15.67:0/229045729 shutdown_connections 2024-04-22T23:56:00.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.549+0000 7f20d5e25700 1 --2- 172.21.15.67:0/229045729 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f20b806ecc0 0x7f20b8071180 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:00.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.549+0000 7f20d5e25700 1 --2- 172.21.15.67:0/229045729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20d00ffda0 0x7f20d019e880 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:00.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.549+0000 7f20d5e25700 1 --2- 172.21.15.67:0/229045729 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f20d0100750 0x7f20d019edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:00.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.549+0000 7f20d5e25700 1 -- 172.21.15.67:0/229045729 >> 172.21.15.67:0/229045729 conn(0x7f20d00fb650 msgr2=0x7f20d0103780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:56:00.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.549+0000 7f20d5e25700 1 -- 172.21.15.67:0/229045729 shutdown_connections 2024-04-22T23:56:00.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.549+0000 7f20d5e25700 1 -- 172.21.15.67:0/229045729 wait complete. 2024-04-22T23:56:00.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.655+0000 7f42a1265700 1 -- 172.21.15.67:0/1323004706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f429c0ffc20 msgr2=0x7f429c100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:00.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.655+0000 7f42a1265700 1 --2- 172.21.15.67:0/1323004706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f429c0ffc20 0x7f429c100080 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f4284009910 tx=0x7f4284009c20 comp rx=0 tx=0).stop 2024-04-22T23:56:00.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.655+0000 7f42a1265700 1 -- 172.21.15.67:0/1323004706 shutdown_connections 2024-04-22T23:56:00.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.655+0000 7f42a1265700 1 --2- 172.21.15.67:0/1323004706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f429c0ffc20 0x7f429c100080 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:00.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.655+0000 7f42a1265700 1 --2- 172.21.15.67:0/1323004706 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f429c105bd0 0x7f429c105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:00.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.655+0000 7f42a1265700 1 -- 172.21.15.67:0/1323004706 >> 172.21.15.67:0/1323004706 conn(0x7f429c075670 msgr2=0x7f429c075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:56:00.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.655+0000 7f42a1265700 1 -- 172.21.15.67:0/1323004706 shutdown_connections 2024-04-22T23:56:00.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.655+0000 7f42a1265700 1 -- 172.21.15.67:0/1323004706 wait complete. 2024-04-22T23:56:00.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.655+0000 7f42a1265700 1 Processor -- start 2024-04-22T23:56:00.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.656+0000 7f42a1265700 1 -- start start 2024-04-22T23:56:00.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.656+0000 7f42a1265700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f429c0ffc20 0x7f429c101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:56:00.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.656+0000 7f42a1265700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f429c105bd0 0x7f429c101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:56:00.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.656+0000 7f42a1265700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429c1022c0 con 0x7f429c105bd0 2024-04-22T23:56:00.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.656+0000 7f42a1265700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429c102400 con 0x7f429c0ffc20 2024-04-22T23:56:00.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.656+0000 7f429a7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f429c105bd0 0x7f429c101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:56:00.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.656+0000 7f429a7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f429c105bd0 0x7f429c101d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41782/0 (socket says 172.21.15.67:41782) 2024-04-22T23:56:00.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.656+0000 7f429a7fc700 1 -- 172.21.15.67:0/2834261862 learned_addr learned my addr 172.21.15.67:0/2834261862 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:56:00.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.656+0000 7f429affd700 1 --2- 172.21.15.67:0/2834261862 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f429c0ffc20 0x7f429c101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:56:00.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.656+0000 7f429a7fc700 1 -- 172.21.15.67:0/2834261862 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f429c0ffc20 msgr2=0x7f429c101840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:00.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.656+0000 7f429a7fc700 1 --2- 172.21.15.67:0/2834261862 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f429c0ffc20 0x7f429c101840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:00.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.656+0000 7f429a7fc700 1 -- 172.21.15.67:0/2834261862 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42840095f0 con 0x7f429c105bd0 2024-04-22T23:56:00.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.656+0000 7f429affd700 1 --2- 172.21.15.67:0/2834261862 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f429c0ffc20 0x7f429c101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:56:00.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.657+0000 7f429a7fc700 1 --2- 172.21.15.67:0/2834261862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f429c105bd0 0x7f429c101d80 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f4284006010 tx=0x7f428400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:56:00.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.657+0000 7f4293fff700 1 -- 172.21.15.67:0/2834261862 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f428401d070 con 0x7f429c105bd0 2024-04-22T23:56:00.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.657+0000 7f4293fff700 1 -- 172.21.15.67:0/2834261862 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f428400f460 con 0x7f429c105bd0 2024-04-22T23:56:00.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.657+0000 7f42a1265700 1 -- 172.21.15.67:0/2834261862 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f429c102680 con 0x7f429c105bd0 2024-04-22T23:56:00.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.657+0000 7f4293fff700 1 -- 172.21.15.67:0/2834261862 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4284017630 con 0x7f429c105bd0 2024-04-22T23:56:00.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.657+0000 7f42a1265700 1 -- 172.21.15.67:0/2834261862 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f429c1a2d90 con 0x7f429c105bd0 2024-04-22T23:56:00.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.658+0000 7f42a1265700 1 -- 172.21.15.67:0/2834261862 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f429c04f2e0 con 0x7f429c105bd0 2024-04-22T23:56:00.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.658+0000 7f4293fff700 1 -- 172.21.15.67:0/2834261862 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f428400f5d0 con 0x7f429c105bd0 2024-04-22T23:56:00.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.659+0000 7f4293fff700 1 --2- 172.21.15.67:0/2834261862 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f428806ecc0 0x7f4288071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:56:00.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.659+0000 7f4293fff700 1 -- 172.21.15.67:0/2834261862 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f4284090470 con 0x7f429c105bd0 2024-04-22T23:56:00.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.659+0000 7f429affd700 1 --2- 172.21.15.67:0/2834261862 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f428806ecc0 0x7f4288071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:56:00.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.659+0000 7f429affd700 1 --2- 172.21.15.67:0/2834261862 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f428806ecc0 0x7f4288071180 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f4294009dd0 tx=0x7f4294009040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:56:00.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.664+0000 7f4293fff700 1 -- 172.21.15.67:0/2834261862 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f42840590b0 con 0x7f429c105bd0 2024-04-22T23:56:00.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.962+0000 7f42a1265700 1 -- 172.21.15.67:0/2834261862 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f429c1a2fb0 con 0x7f429c105bd0 2024-04-22T23:56:00.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.963+0000 7f4293fff700 1 -- 172.21.15.67:0/2834261862 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f428405c770 con 0x7f429c105bd0 2024-04-22T23:56:00.954 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:56:00.954 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:56:00.954 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:56:00.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:56:00.954 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:56:00.954 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:56:00.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:56:00.954 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:56:00.954 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:56:00.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:56:00.954 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:56:00.954 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 4 2024-04-22T23:56:00.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:56:00.955 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:56:00.955 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 14 2024-04-22T23:56:00.955 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:56:00.955 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:56:00.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.965+0000 7f42a1265700 1 -- 172.21.15.67:0/2834261862 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f428806ecc0 msgr2=0x7f4288071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:00.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.965+0000 7f42a1265700 1 --2- 172.21.15.67:0/2834261862 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f428806ecc0 0x7f4288071180 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f4294009dd0 tx=0x7f4294009040 comp rx=0 tx=0).stop 2024-04-22T23:56:00.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.965+0000 7f42a1265700 1 -- 172.21.15.67:0/2834261862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f429c105bd0 msgr2=0x7f429c101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:00.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.965+0000 7f42a1265700 1 --2- 172.21.15.67:0/2834261862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f429c105bd0 0x7f429c101d80 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f4284006010 tx=0x7f428400be20 comp rx=0 tx=0).stop 2024-04-22T23:56:00.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.965+0000 7f42a1265700 1 -- 172.21.15.67:0/2834261862 shutdown_connections 2024-04-22T23:56:00.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.965+0000 7f42a1265700 1 --2- 172.21.15.67:0/2834261862 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f428806ecc0 0x7f4288071180 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:00.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.965+0000 7f42a1265700 1 --2- 172.21.15.67:0/2834261862 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f429c0ffc20 0x7f429c101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:00.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.965+0000 7f42a1265700 1 --2- 172.21.15.67:0/2834261862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f429c105bd0 0x7f429c101d80 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:00.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.965+0000 7f42a1265700 1 -- 172.21.15.67:0/2834261862 >> 172.21.15.67:0/2834261862 conn(0x7f429c075670 msgr2=0x7f429c068640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:56:00.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.965+0000 7f42a1265700 1 -- 172.21.15.67:0/2834261862 shutdown_connections 2024-04-22T23:56:00.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:00.965+0000 7f42a1265700 1 -- 172.21.15.67:0/2834261862 wait complete. 2024-04-22T23:56:01.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.072+0000 7f54ae3f5700 1 -- 172.21.15.67:0/1314777806 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a8102720 msgr2=0x7f54a810ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:01.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.072+0000 7f54ae3f5700 1 --2- 172.21.15.67:0/1314777806 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a8102720 0x7f54a810ac10 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f549c009910 tx=0x7f549c009c20 comp rx=0 tx=0).stop 2024-04-22T23:56:01.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.072+0000 7f54ae3f5700 1 -- 172.21.15.67:0/1314777806 shutdown_connections 2024-04-22T23:56:01.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.072+0000 7f54ae3f5700 1 --2- 172.21.15.67:0/1314777806 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a8102720 0x7f54a810ac10 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:01.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.072+0000 7f54ae3f5700 1 --2- 172.21.15.67:0/1314777806 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f54a8101e00 0x7f54a81021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:01.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.072+0000 7f54ae3f5700 1 -- 172.21.15.67:0/1314777806 >> 172.21.15.67:0/1314777806 conn(0x7f54a80fb650 msgr2=0x7f54a80fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:56:01.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.073+0000 7f54ae3f5700 1 -- 172.21.15.67:0/1314777806 shutdown_connections 2024-04-22T23:56:01.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.073+0000 7f54ae3f5700 1 -- 172.21.15.67:0/1314777806 wait complete. 2024-04-22T23:56:01.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.073+0000 7f54ae3f5700 1 Processor -- start 2024-04-22T23:56:01.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.073+0000 7f54ae3f5700 1 -- start start 2024-04-22T23:56:01.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.074+0000 7f54ae3f5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a8101e00 0x7f54a80ffa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:56:01.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.074+0000 7f54ae3f5700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f54a8102720 0x7f54a80fffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:56:01.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.074+0000 7f54ae3f5700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a81004e0 con 0x7f54a8101e00 2024-04-22T23:56:01.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.074+0000 7f54ae3f5700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a8100620 con 0x7f54a8102720 2024-04-22T23:56:01.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.074+0000 7f54a7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a8101e00 0x7f54a80ffa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:56:01.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.074+0000 7f54a7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a8101e00 0x7f54a80ffa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41798/0 (socket says 172.21.15.67:41798) 2024-04-22T23:56:01.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.074+0000 7f54a7fff700 1 -- 172.21.15.67:0/2590505759 learned_addr learned my addr 172.21.15.67:0/2590505759 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:56:01.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.074+0000 7f54a77fe700 1 --2- 172.21.15.67:0/2590505759 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f54a8102720 0x7f54a80fffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:56:01.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.074+0000 7f54a7fff700 1 -- 172.21.15.67:0/2590505759 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f54a8102720 msgr2=0x7f54a80fffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:01.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.074+0000 7f54a7fff700 1 --2- 172.21.15.67:0/2590505759 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f54a8102720 0x7f54a80fffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:01.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.074+0000 7f54a7fff700 1 -- 172.21.15.67:0/2590505759 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f549c0095f0 con 0x7f54a8101e00 2024-04-22T23:56:01.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.074+0000 7f54a7fff700 1 --2- 172.21.15.67:0/2590505759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a8101e00 0x7f54a80ffa60 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f549800c8a0 tx=0x7f549800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:56:01.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.074+0000 7f54a57fa700 1 -- 172.21.15.67:0/2590505759 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54980078e0 con 0x7f54a8101e00 2024-04-22T23:56:01.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.074+0000 7f54ae3f5700 1 -- 172.21.15.67:0/2590505759 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54a81a2a90 con 0x7f54a8101e00 2024-04-22T23:56:01.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.074+0000 7f54a57fa700 1 -- 172.21.15.67:0/2590505759 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f549800f450 con 0x7f54a8101e00 2024-04-22T23:56:01.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.074+0000 7f54a57fa700 1 -- 172.21.15.67:0/2590505759 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f549800e5e0 con 0x7f54a8101e00 2024-04-22T23:56:01.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.075+0000 7f54ae3f5700 1 -- 172.21.15.67:0/2590505759 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54a81a2e40 con 0x7f54a8101e00 2024-04-22T23:56:01.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.076+0000 7f54ae3f5700 1 -- 172.21.15.67:0/2590505759 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54a8108310 con 0x7f54a8101e00 2024-04-22T23:56:01.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.076+0000 7f54a57fa700 1 -- 172.21.15.67:0/2590505759 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5498007a40 con 0x7f54a8101e00 2024-04-22T23:56:01.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.076+0000 7f54a57fa700 1 --2- 172.21.15.67:0/2590505759 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f549006ecc0 0x7f5490071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:56:01.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.076+0000 7f54a57fa700 1 -- 172.21.15.67:0/2590505759 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f549808efd0 con 0x7f54a8101e00 2024-04-22T23:56:01.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.076+0000 7f54a77fe700 1 --2- 172.21.15.67:0/2590505759 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f549006ecc0 0x7f5490071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:56:01.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.077+0000 7f54a77fe700 1 --2- 172.21.15.67:0/2590505759 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f549006ecc0 0x7f5490071180 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f549c006010 tx=0x7f549c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:56:01.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.081+0000 7f54a57fa700 1 -- 172.21.15.67:0/2590505759 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f5498057c10 con 0x7f54a8101e00 2024-04-22T23:56:01.284 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:01 smithi067 ceph-mon[112438]: from='client.35612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:56:01.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:01 smithi067 ceph-mon[112438]: from='client.35616 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:56:01.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:01 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2834261862' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:56:01.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:01 smithi067 ceph-mon[112438]: pgmap v891: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:01.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.340+0000 7f54ae3f5700 1 -- 172.21.15.67:0/2590505759 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f54a81008d0 con 0x7f54a8101e00 2024-04-22T23:56:01.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.341+0000 7f54a57fa700 1 -- 172.21.15.67:0/2590505759 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1872 (secure 0 0 0) 0x7f54a81008d0 con 0x7f54a8101e00 2024-04-22T23:56:01.332 INFO:teuthology.orchestra.run.smithi067.stdout:e33 2024-04-22T23:56:01.332 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:56:01.332 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:56:01.332 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:56:01.332 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:56:01.332 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:56:01.332 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:56:01.332 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:56:01.332 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:56:01.332 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:56:01.332 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:56:01.332 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:56:01.333 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:56:01.333 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:56:01.333 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:56:01.333 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:56:01.333 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:56:01.333 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:56:01.333 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:56:01.333 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:56:01.333 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:56:01.333 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:56:01.333 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:56:01.333 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:56:01.333 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:56:01.333 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:56:01.333 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:56:01.333 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:56:01.333 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:56:01.334 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:56:01.334 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:56:01.334 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:56:01.334 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:56:01.334 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:56:01.334 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:56:01.334 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:56:01.334 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:56:01.334 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:56:01.334 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:56:01.334 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:56:01.334 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:56:01.334 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44851} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1849648622,v1:172.21.15.134:6827/1849648622] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:56:01.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.343+0000 7f54ae3f5700 1 -- 172.21.15.67:0/2590505759 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f549006ecc0 msgr2=0x7f5490071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:01.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.343+0000 7f54ae3f5700 1 --2- 172.21.15.67:0/2590505759 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f549006ecc0 0x7f5490071180 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f549c006010 tx=0x7f549c01a040 comp rx=0 tx=0).stop 2024-04-22T23:56:01.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.343+0000 7f54ae3f5700 1 -- 172.21.15.67:0/2590505759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a8101e00 msgr2=0x7f54a80ffa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:01.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.343+0000 7f54ae3f5700 1 --2- 172.21.15.67:0/2590505759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a8101e00 0x7f54a80ffa60 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f549800c8a0 tx=0x7f549800cbb0 comp rx=0 tx=0).stop 2024-04-22T23:56:01.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.343+0000 7f54ae3f5700 1 -- 172.21.15.67:0/2590505759 shutdown_connections 2024-04-22T23:56:01.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.343+0000 7f54ae3f5700 1 --2- 172.21.15.67:0/2590505759 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f549006ecc0 0x7f5490071180 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:01.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.343+0000 7f54ae3f5700 1 --2- 172.21.15.67:0/2590505759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a8101e00 0x7f54a80ffa60 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:01.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.343+0000 7f54ae3f5700 1 --2- 172.21.15.67:0/2590505759 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f54a8102720 0x7f54a80fffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:01.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.343+0000 7f54ae3f5700 1 -- 172.21.15.67:0/2590505759 >> 172.21.15.67:0/2590505759 conn(0x7f54a80fb650 msgr2=0x7f54a8105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:56:01.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.343+0000 7f54ae3f5700 1 -- 172.21.15.67:0/2590505759 shutdown_connections 2024-04-22T23:56:01.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.343+0000 7f54ae3f5700 1 -- 172.21.15.67:0/2590505759 wait complete. 2024-04-22T23:56:01.336 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 33 2024-04-22T23:56:01.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.451+0000 7faa69634700 1 -- 172.21.15.67:0/2169071987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa64100320 msgr2=0x7faa6410cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:01.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.451+0000 7faa69634700 1 --2- 172.21.15.67:0/2169071987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa64100320 0x7faa6410cfe0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7faa58009910 tx=0x7faa58009c20 comp rx=0 tx=0).stop 2024-04-22T23:56:01.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.451+0000 7faa69634700 1 -- 172.21.15.67:0/2169071987 shutdown_connections 2024-04-22T23:56:01.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.451+0000 7faa69634700 1 --2- 172.21.15.67:0/2169071987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa64100320 0x7faa6410cfe0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:01.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.451+0000 7faa69634700 1 --2- 172.21.15.67:0/2169071987 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faa640ff970 0x7faa640ffd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:01.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.451+0000 7faa69634700 1 -- 172.21.15.67:0/2169071987 >> 172.21.15.67:0/2169071987 conn(0x7faa640fb650 msgr2=0x7faa640fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:56:01.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.452+0000 7faa69634700 1 -- 172.21.15.67:0/2169071987 shutdown_connections 2024-04-22T23:56:01.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.452+0000 7faa69634700 1 -- 172.21.15.67:0/2169071987 wait complete. 2024-04-22T23:56:01.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.452+0000 7faa69634700 1 Processor -- start 2024-04-22T23:56:01.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.452+0000 7faa69634700 1 -- start start 2024-04-22T23:56:01.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.452+0000 7faa69634700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faa640ff970 0x7faa6419e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:56:01.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.452+0000 7faa69634700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa64100320 0x7faa6419ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:56:01.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.453+0000 7faa69634700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6419f520 con 0x7faa64100320 2024-04-22T23:56:01.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.453+0000 7faa69634700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa641989d0 con 0x7faa640ff970 2024-04-22T23:56:01.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.453+0000 7faa627fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa64100320 0x7faa6419ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:56:01.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.453+0000 7faa627fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa64100320 0x7faa6419ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41818/0 (socket says 172.21.15.67:41818) 2024-04-22T23:56:01.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.453+0000 7faa627fc700 1 -- 172.21.15.67:0/2341069175 learned_addr learned my addr 172.21.15.67:0/2341069175 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:56:01.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.453+0000 7faa62ffd700 1 --2- 172.21.15.67:0/2341069175 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faa640ff970 0x7faa6419e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:56:01.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.453+0000 7faa627fc700 1 -- 172.21.15.67:0/2341069175 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faa640ff970 msgr2=0x7faa6419e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:01.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.453+0000 7faa627fc700 1 --2- 172.21.15.67:0/2341069175 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faa640ff970 0x7faa6419e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:01.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.453+0000 7faa627fc700 1 -- 172.21.15.67:0/2341069175 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa580095f0 con 0x7faa64100320 2024-04-22T23:56:01.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.453+0000 7faa627fc700 1 --2- 172.21.15.67:0/2341069175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa64100320 0x7faa6419ee90 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7faa58000c00 tx=0x7faa580055f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:56:01.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.453+0000 7faa4bfff700 1 -- 172.21.15.67:0/2341069175 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa5801d070 con 0x7faa64100320 2024-04-22T23:56:01.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.453+0000 7faa69634700 1 -- 172.21.15.67:0/2341069175 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa64198c50 con 0x7faa64100320 2024-04-22T23:56:01.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.453+0000 7faa4bfff700 1 -- 172.21.15.67:0/2341069175 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faa5800f460 con 0x7faa64100320 2024-04-22T23:56:01.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.453+0000 7faa4bfff700 1 -- 172.21.15.67:0/2341069175 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa58021640 con 0x7faa64100320 2024-04-22T23:56:01.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.454+0000 7faa69634700 1 -- 172.21.15.67:0/2341069175 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa641991a0 con 0x7faa64100320 2024-04-22T23:56:01.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.455+0000 7faa69634700 1 -- 172.21.15.67:0/2341069175 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa6410a750 con 0x7faa64100320 2024-04-22T23:56:01.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.455+0000 7faa4bfff700 1 -- 172.21.15.67:0/2341069175 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7faa5800fa60 con 0x7faa64100320 2024-04-22T23:56:01.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.456+0000 7faa4bfff700 1 --2- 172.21.15.67:0/2341069175 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faa4c06ecc0 0x7faa4c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:56:01.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.456+0000 7faa4bfff700 1 -- 172.21.15.67:0/2341069175 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7faa58090250 con 0x7faa64100320 2024-04-22T23:56:01.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.456+0000 7faa62ffd700 1 --2- 172.21.15.67:0/2341069175 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faa4c06ecc0 0x7faa4c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:56:01.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.456+0000 7faa62ffd700 1 --2- 172.21.15.67:0/2341069175 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faa4c06ecc0 0x7faa4c071180 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7faa54009a10 tx=0x7faa54009190 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:56:01.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.460+0000 7faa4bfff700 1 -- 172.21.15.67:0/2341069175 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7faa5805c9c0 con 0x7faa64100320 2024-04-22T23:56:01.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:01 smithi134 ceph-mon[85630]: from='client.35612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:56:01.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:01 smithi134 ceph-mon[85630]: from='client.35616 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:56:01.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:01 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2834261862' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:56:01.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:01 smithi134 ceph-mon[85630]: pgmap v891: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:01.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.689+0000 7faa69634700 1 -- 172.21.15.67:0/2341069175 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7faa641993c0 con 0x7faa4c06ecc0 2024-04-22T23:56:01.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.690+0000 7faa4bfff700 1 -- 172.21.15.67:0/2341069175 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7faa641993c0 con 0x7faa4c06ecc0 2024-04-22T23:56:01.681 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:56:01.681 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:56:01.681 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:56:01.681 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:56:01.681 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:56:01.681 INFO:teuthology.orchestra.run.smithi067.stdout: "mds", 2024-04-22T23:56:01.681 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:56:01.681 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:56:01.681 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:56:01.681 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:56:01.681 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:56:01.682 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T23:56:01.682 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:56:01.682 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:56:01.682 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:56:01.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.692+0000 7faa69634700 1 -- 172.21.15.67:0/2341069175 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faa4c06ecc0 msgr2=0x7faa4c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:01.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.692+0000 7faa69634700 1 --2- 172.21.15.67:0/2341069175 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faa4c06ecc0 0x7faa4c071180 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7faa54009a10 tx=0x7faa54009190 comp rx=0 tx=0).stop 2024-04-22T23:56:01.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.692+0000 7faa69634700 1 -- 172.21.15.67:0/2341069175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa64100320 msgr2=0x7faa6419ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:01.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.692+0000 7faa69634700 1 --2- 172.21.15.67:0/2341069175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa64100320 0x7faa6419ee90 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7faa58000c00 tx=0x7faa580055f0 comp rx=0 tx=0).stop 2024-04-22T23:56:01.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.692+0000 7faa69634700 1 -- 172.21.15.67:0/2341069175 shutdown_connections 2024-04-22T23:56:01.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.692+0000 7faa69634700 1 --2- 172.21.15.67:0/2341069175 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faa4c06ecc0 0x7faa4c071180 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:01.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.692+0000 7faa69634700 1 --2- 172.21.15.67:0/2341069175 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faa640ff970 0x7faa6419e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:01.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.692+0000 7faa69634700 1 --2- 172.21.15.67:0/2341069175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa64100320 0x7faa6419ee90 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:01.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.692+0000 7faa69634700 1 -- 172.21.15.67:0/2341069175 >> 172.21.15.67:0/2341069175 conn(0x7faa640fb650 msgr2=0x7faa640fcd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:56:01.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.693+0000 7faa69634700 1 -- 172.21.15.67:0/2341069175 shutdown_connections 2024-04-22T23:56:01.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.693+0000 7faa69634700 1 -- 172.21.15.67:0/2341069175 wait complete. 2024-04-22T23:56:01.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.799+0000 7f08990af700 1 -- 172.21.15.67:0/496135607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08940ff340 msgr2=0x7f08940ff7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:01.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.799+0000 7f08990af700 1 --2- 172.21.15.67:0/496135607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08940ff340 0x7f08940ff7a0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f088800aa20 tx=0x7f088800ad30 comp rx=0 tx=0).stop 2024-04-22T23:56:01.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.799+0000 7f08990af700 1 -- 172.21.15.67:0/496135607 shutdown_connections 2024-04-22T23:56:01.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.799+0000 7f08990af700 1 --2- 172.21.15.67:0/496135607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08940ff340 0x7f08940ff7a0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:01.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.799+0000 7f08990af700 1 --2- 172.21.15.67:0/496135607 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0894105350 0x7f0894105730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:01.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.799+0000 7f08990af700 1 -- 172.21.15.67:0/496135607 >> 172.21.15.67:0/496135607 conn(0x7f0894074e30 msgr2=0x7f0894075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:56:01.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.799+0000 7f08990af700 1 -- 172.21.15.67:0/496135607 shutdown_connections 2024-04-22T23:56:01.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.799+0000 7f08990af700 1 -- 172.21.15.67:0/496135607 wait complete. 2024-04-22T23:56:01.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.800+0000 7f08990af700 1 Processor -- start 2024-04-22T23:56:01.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.800+0000 7f08990af700 1 -- start start 2024-04-22T23:56:01.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.800+0000 7f08990af700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08940ff340 0x7f089419c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:56:01.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.800+0000 7f08990af700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0894105350 0x7f089419cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:56:01.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.800+0000 7f08990af700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f089419d1f0 con 0x7f0894105350 2024-04-22T23:56:01.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.800+0000 7f08990af700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08941966a0 con 0x7f08940ff340 2024-04-22T23:56:01.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.800+0000 7f089259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0894105350 0x7f089419cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:56:01.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.800+0000 7f089259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0894105350 0x7f089419cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41836/0 (socket says 172.21.15.67:41836) 2024-04-22T23:56:01.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.800+0000 7f089259c700 1 -- 172.21.15.67:0/2961842705 learned_addr learned my addr 172.21.15.67:0/2961842705 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:56:01.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.800+0000 7f0892d9d700 1 --2- 172.21.15.67:0/2961842705 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08940ff340 0x7f089419c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:56:01.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.801+0000 7f089259c700 1 -- 172.21.15.67:0/2961842705 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08940ff340 msgr2=0x7f089419c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:01.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.801+0000 7f089259c700 1 --2- 172.21.15.67:0/2961842705 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08940ff340 0x7f089419c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:01.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.801+0000 7f089259c700 1 -- 172.21.15.67:0/2961842705 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f088800a700 con 0x7f0894105350 2024-04-22T23:56:01.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.801+0000 7f0892d9d700 1 --2- 172.21.15.67:0/2961842705 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08940ff340 0x7f089419c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:56:01.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.801+0000 7f089259c700 1 --2- 172.21.15.67:0/2961842705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0894105350 0x7f089419cb60 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f0888006010 tx=0x7f0888005750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:56:01.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.801+0000 7f087bfff700 1 -- 172.21.15.67:0/2961842705 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0888004bf0 con 0x7f0894105350 2024-04-22T23:56:01.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.801+0000 7f087bfff700 1 -- 172.21.15.67:0/2961842705 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0888004d50 con 0x7f0894105350 2024-04-22T23:56:01.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.801+0000 7f08990af700 1 -- 172.21.15.67:0/2961842705 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0894196920 con 0x7f0894105350 2024-04-22T23:56:01.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.801+0000 7f087bfff700 1 -- 172.21.15.67:0/2961842705 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0888016920 con 0x7f0894105350 2024-04-22T23:56:01.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.801+0000 7f08990af700 1 -- 172.21.15.67:0/2961842705 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0894196e10 con 0x7f0894105350 2024-04-22T23:56:01.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.802+0000 7f08990af700 1 -- 172.21.15.67:0/2961842705 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0894109860 con 0x7f0894105350 2024-04-22T23:56:01.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.803+0000 7f087bfff700 1 -- 172.21.15.67:0/2961842705 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0888016a80 con 0x7f0894105350 2024-04-22T23:56:01.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.803+0000 7f087bfff700 1 --2- 172.21.15.67:0/2961842705 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f087c06ecc0 0x7f087c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:56:01.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.803+0000 7f087bfff700 1 -- 172.21.15.67:0/2961842705 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f0888012070 con 0x7f0894105350 2024-04-22T23:56:01.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.803+0000 7f0892d9d700 1 --2- 172.21.15.67:0/2961842705 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f087c06ecc0 0x7f087c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:56:01.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.804+0000 7f0892d9d700 1 --2- 172.21.15.67:0/2961842705 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f087c06ecc0 0x7f087c071180 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f088400bc10 tx=0x7f088400b5f0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:56:01.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:01.808+0000 7f087bfff700 1 -- 172.21.15.67:0/2961842705 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f088805d7a0 con 0x7f0894105350 2024-04-22T23:56:02.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:02.108+0000 7f08990af700 1 -- 172.21.15.67:0/2961842705 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f089404f2e0 con 0x7f0894105350 2024-04-22T23:56:02.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:02.108+0000 7f087bfff700 1 -- 172.21.15.67:0/2961842705 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4505 (secure 0 0 0) 0x7f088805d330 con 0x7f0894105350 2024-04-22T23:56:02.100 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:56:02.100 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T23:56:02.100 INFO:teuthology.orchestra.run.smithi067.stdout: host smithi134 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-22T23:56:02.100 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T23:56:02.100 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:56:02.100 INFO:teuthology.orchestra.run.smithi067.stdout:Traceback (most recent call last): 2024-04-22T23:56:02.101 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T23:56:02.101 INFO:teuthology.orchestra.run.smithi067.stdout: return _run_code(code, main_globals, None, 2024-04-22T23:56:02.101 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T23:56:02.101 INFO:teuthology.orchestra.run.smithi067.stdout: exec(code, run_globals) 2024-04-22T23:56:02.101 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5642, in 2024-04-22T23:56:02.101 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5630, in main 2024-04-22T23:56:02.101 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 408, in _infer_config 2024-04-22T23:56:02.101 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 323, in _infer_fsid 2024-04-22T23:56:02.101 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 436, in _infer_image 2024-04-22T23:56:02.101 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 310, in _validate_fsid 2024-04-22T23:56:02.101 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 3380, in command_ceph_volume 2024-04-22T23:56:02.101 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T23:56:02.101 INFO:teuthology.orchestra.run.smithi067.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:56:02.101 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:56:02.101 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:56:02.102 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:56:02.102 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:56:02.102 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:56:02.102 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:56:02.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:02.110+0000 7f08990af700 1 -- 172.21.15.67:0/2961842705 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f087c06ecc0 msgr2=0x7f087c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:02.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:02.110+0000 7f08990af700 1 --2- 172.21.15.67:0/2961842705 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f087c06ecc0 0x7f087c071180 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f088400bc10 tx=0x7f088400b5f0 comp rx=0 tx=0).stop 2024-04-22T23:56:02.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:02.110+0000 7f08990af700 1 -- 172.21.15.67:0/2961842705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0894105350 msgr2=0x7f089419cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:02.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:02.110+0000 7f08990af700 1 --2- 172.21.15.67:0/2961842705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0894105350 0x7f089419cb60 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f0888006010 tx=0x7f0888005750 comp rx=0 tx=0).stop 2024-04-22T23:56:02.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:02.110+0000 7f08990af700 1 -- 172.21.15.67:0/2961842705 shutdown_connections 2024-04-22T23:56:02.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:02.110+0000 7f08990af700 1 --2- 172.21.15.67:0/2961842705 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f087c06ecc0 0x7f087c071180 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:02.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:02.110+0000 7f08990af700 1 --2- 172.21.15.67:0/2961842705 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08940ff340 0x7f089419c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:02.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:02.110+0000 7f08990af700 1 --2- 172.21.15.67:0/2961842705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0894105350 0x7f089419cb60 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:02.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:02.110+0000 7f08990af700 1 -- 172.21.15.67:0/2961842705 >> 172.21.15.67:0/2961842705 conn(0x7f0894074e30 msgr2=0x7f0894103870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:56:02.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:02.110+0000 7f08990af700 1 -- 172.21.15.67:0/2961842705 shutdown_connections 2024-04-22T23:56:02.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:02.110+0000 7f08990af700 1 -- 172.21.15.67:0/2961842705 wait complete. 2024-04-22T23:56:02.407 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:02 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2590505759' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:56:02.407 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:02 smithi067 ceph-mon[112438]: from='client.35628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:56:02.407 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:02 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2961842705' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:56:02.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:02 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2590505759' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:56:02.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:02 smithi134 ceph-mon[85630]: from='client.35628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:56:02.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:02 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2961842705' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:56:03.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:03 smithi134 ceph-mon[85630]: pgmap v892: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:03.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:03 smithi067 ceph-mon[112438]: pgmap v892: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:05.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:05 smithi134 ceph-mon[85630]: pgmap v893: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:05.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:05 smithi067 ceph-mon[112438]: pgmap v893: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:07.457 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:07 smithi067 ceph-mon[112438]: pgmap v894: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:07.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:07 smithi134 ceph-mon[85630]: pgmap v894: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:09.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:09 smithi134 ceph-mon[85630]: pgmap v895: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:09.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:09 smithi067 ceph-mon[112438]: pgmap v895: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:11.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:56:11.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:56:12.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:12 smithi134 ceph-mon[85630]: pgmap v896: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:12.511 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:12 smithi067 ceph-mon[112438]: pgmap v896: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:13.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:13 smithi134 ceph-mon[85630]: pgmap v897: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:13.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:13 smithi067 ceph-mon[112438]: pgmap v897: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:15.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:15 smithi134 ceph-mon[85630]: pgmap v898: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:15.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:15 smithi067 ceph-mon[112438]: pgmap v898: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:17.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:17 smithi134 ceph-mon[85630]: pgmap v899: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:17.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:17 smithi067 ceph-mon[112438]: pgmap v899: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:19.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:19 smithi134 ceph-mon[85630]: pgmap v900: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:19.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:19 smithi067 ceph-mon[112438]: pgmap v900: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:21.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:21 smithi134 ceph-mon[85630]: pgmap v901: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:21.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:21 smithi067 ceph-mon[112438]: pgmap v901: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:23.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:23 smithi134 ceph-mon[85630]: pgmap v902: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:23.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:23 smithi067 ceph-mon[112438]: pgmap v902: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:25.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:25 smithi134 ceph-mon[85630]: pgmap v903: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:25.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:25 smithi067 ceph-mon[112438]: pgmap v903: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:26.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:56:26.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:56:27.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:27 smithi134 ceph-mon[85630]: pgmap v904: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:27.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:27 smithi067 ceph-mon[112438]: pgmap v904: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:29.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:29 smithi134 ceph-mon[85630]: pgmap v905: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:29.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:29 smithi067 ceph-mon[112438]: pgmap v905: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:31.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:31 smithi134 ceph-mon[85630]: pgmap v906: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:31.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:31 smithi067 ceph-mon[112438]: pgmap v906: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:32.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.233+0000 7f7bdce09700 1 -- 172.21.15.67:0/3862563708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bd8100320 msgr2=0x7f7bd810cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:32.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.233+0000 7f7bdce09700 1 --2- 172.21.15.67:0/3862563708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bd8100320 0x7f7bd810cfe0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc009910 tx=0x7f7bcc009c20 comp rx=0 tx=0).stop 2024-04-22T23:56:32.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.233+0000 7f7bdce09700 1 -- 172.21.15.67:0/3862563708 shutdown_connections 2024-04-22T23:56:32.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.233+0000 7f7bdce09700 1 --2- 172.21.15.67:0/3862563708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bd8100320 0x7f7bd810cfe0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:32.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.233+0000 7f7bdce09700 1 --2- 172.21.15.67:0/3862563708 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7bd80ff970 0x7f7bd80ffd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:32.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.233+0000 7f7bdce09700 1 -- 172.21.15.67:0/3862563708 >> 172.21.15.67:0/3862563708 conn(0x7f7bd80fb650 msgr2=0x7f7bd80fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:56:32.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.233+0000 7f7bdce09700 1 -- 172.21.15.67:0/3862563708 shutdown_connections 2024-04-22T23:56:32.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.233+0000 7f7bdce09700 1 -- 172.21.15.67:0/3862563708 wait complete. 2024-04-22T23:56:32.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.234+0000 7f7bdce09700 1 Processor -- start 2024-04-22T23:56:32.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.234+0000 7f7bdce09700 1 -- start start 2024-04-22T23:56:32.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.234+0000 7f7bdce09700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7bd80ff970 0x7f7bd819e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:56:32.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.234+0000 7f7bdce09700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bd8100320 0x7f7bd819ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:56:32.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.234+0000 7f7bdce09700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd819f360 con 0x7f7bd8100320 2024-04-22T23:56:32.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.234+0000 7f7bdce09700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd81988a0 con 0x7f7bd80ff970 2024-04-22T23:56:32.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.234+0000 7f7bd5d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bd8100320 0x7f7bd819ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:56:32.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.234+0000 7f7bd5d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bd8100320 0x7f7bd819ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60240/0 (socket says 172.21.15.67:60240) 2024-04-22T23:56:32.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.234+0000 7f7bd5d9b700 1 -- 172.21.15.67:0/80186330 learned_addr learned my addr 172.21.15.67:0/80186330 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:56:32.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.234+0000 7f7bd659c700 1 --2- 172.21.15.67:0/80186330 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7bd80ff970 0x7f7bd819e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:56:32.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.235+0000 7f7bd5d9b700 1 -- 172.21.15.67:0/80186330 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7bd80ff970 msgr2=0x7f7bd819e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:32.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.235+0000 7f7bd5d9b700 1 --2- 172.21.15.67:0/80186330 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7bd80ff970 0x7f7bd819e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:32.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.235+0000 7f7bd5d9b700 1 -- 172.21.15.67:0/80186330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bcc0095f0 con 0x7f7bd8100320 2024-04-22T23:56:32.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.235+0000 7f7bd5d9b700 1 --2- 172.21.15.67:0/80186330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bd8100320 0x7f7bd819ed60 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc006010 tx=0x7f7bcc0055f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:56:32.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.235+0000 7f7bc77fe700 1 -- 172.21.15.67:0/80186330 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7bcc01d070 con 0x7f7bd8100320 2024-04-22T23:56:32.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.235+0000 7f7bc77fe700 1 -- 172.21.15.67:0/80186330 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7bcc005dc0 con 0x7f7bd8100320 2024-04-22T23:56:32.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.235+0000 7f7bdce09700 1 -- 172.21.15.67:0/80186330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bd8198b20 con 0x7f7bd8100320 2024-04-22T23:56:32.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.235+0000 7f7bc77fe700 1 -- 172.21.15.67:0/80186330 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7bcc021410 con 0x7f7bd8100320 2024-04-22T23:56:32.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.235+0000 7f7bdce09700 1 -- 172.21.15.67:0/80186330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bd8199070 con 0x7f7bd8100320 2024-04-22T23:56:32.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.236+0000 7f7bdce09700 1 -- 172.21.15.67:0/80186330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bd810a750 con 0x7f7bd8100320 2024-04-22T23:56:32.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.237+0000 7f7bc77fe700 1 -- 172.21.15.67:0/80186330 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7bcc00f460 con 0x7f7bd8100320 2024-04-22T23:56:32.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.237+0000 7f7bc77fe700 1 --2- 172.21.15.67:0/80186330 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7bc006ecc0 0x7f7bc0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:56:32.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.237+0000 7f7bc77fe700 1 -- 172.21.15.67:0/80186330 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f7bcc090150 con 0x7f7bd8100320 2024-04-22T23:56:32.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.237+0000 7f7bd659c700 1 --2- 172.21.15.67:0/80186330 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7bc006ecc0 0x7f7bc0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:56:32.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.238+0000 7f7bd659c700 1 --2- 172.21.15.67:0/80186330 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7bc006ecc0 0x7f7bc0071180 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f7bc800ac50 tx=0x7f7bc800a380 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:56:32.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.242+0000 7f7bc77fe700 1 -- 172.21.15.67:0/80186330 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7bcc05c940 con 0x7f7bd8100320 2024-04-22T23:56:32.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.476+0000 7f7bdce09700 1 -- 172.21.15.67:0/80186330 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7bd8199e00 con 0x7f7bc006ecc0 2024-04-22T23:56:32.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.477+0000 7f7bc77fe700 1 -- 172.21.15.67:0/80186330 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f7bd8199e00 con 0x7f7bc006ecc0 2024-04-22T23:56:32.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.479+0000 7f7bdce09700 1 -- 172.21.15.67:0/80186330 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7bc006ecc0 msgr2=0x7f7bc0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:32.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.479+0000 7f7bdce09700 1 --2- 172.21.15.67:0/80186330 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7bc006ecc0 0x7f7bc0071180 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f7bc800ac50 tx=0x7f7bc800a380 comp rx=0 tx=0).stop 2024-04-22T23:56:32.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.479+0000 7f7bdce09700 1 -- 172.21.15.67:0/80186330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bd8100320 msgr2=0x7f7bd819ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:32.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.479+0000 7f7bdce09700 1 --2- 172.21.15.67:0/80186330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bd8100320 0x7f7bd819ed60 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc006010 tx=0x7f7bcc0055f0 comp rx=0 tx=0).stop 2024-04-22T23:56:32.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.479+0000 7f7bdce09700 1 -- 172.21.15.67:0/80186330 shutdown_connections 2024-04-22T23:56:32.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.479+0000 7f7bdce09700 1 --2- 172.21.15.67:0/80186330 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7bc006ecc0 0x7f7bc0071180 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:32.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.479+0000 7f7bdce09700 1 --2- 172.21.15.67:0/80186330 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7bd80ff970 0x7f7bd819e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:32.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.479+0000 7f7bdce09700 1 --2- 172.21.15.67:0/80186330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bd8100320 0x7f7bd819ed60 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:32.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.479+0000 7f7bdce09700 1 -- 172.21.15.67:0/80186330 >> 172.21.15.67:0/80186330 conn(0x7f7bd80fb650 msgr2=0x7f7bd80fcc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:56:32.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.479+0000 7f7bdce09700 1 -- 172.21.15.67:0/80186330 shutdown_connections 2024-04-22T23:56:32.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.479+0000 7f7bdce09700 1 -- 172.21.15.67:0/80186330 wait complete. 2024-04-22T23:56:32.480 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:56:32.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.587+0000 7f031f70b700 1 -- 172.21.15.67:0/3749015089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03180ffc20 msgr2=0x7f0318100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:32.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.587+0000 7f031f70b700 1 --2- 172.21.15.67:0/3749015089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03180ffc20 0x7f0318100080 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f0308009910 tx=0x7f0308009c20 comp rx=0 tx=0).stop 2024-04-22T23:56:32.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.587+0000 7f031f70b700 1 -- 172.21.15.67:0/3749015089 shutdown_connections 2024-04-22T23:56:32.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.587+0000 7f031f70b700 1 --2- 172.21.15.67:0/3749015089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03180ffc20 0x7f0318100080 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:32.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.587+0000 7f031f70b700 1 --2- 172.21.15.67:0/3749015089 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0318105bd0 0x7f0318105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:32.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.587+0000 7f031f70b700 1 -- 172.21.15.67:0/3749015089 >> 172.21.15.67:0/3749015089 conn(0x7f0318075670 msgr2=0x7f0318075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:56:32.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.587+0000 7f031f70b700 1 -- 172.21.15.67:0/3749015089 shutdown_connections 2024-04-22T23:56:32.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.587+0000 7f031f70b700 1 -- 172.21.15.67:0/3749015089 wait complete. 2024-04-22T23:56:32.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.588+0000 7f031f70b700 1 Processor -- start 2024-04-22T23:56:32.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.588+0000 7f031f70b700 1 -- start start 2024-04-22T23:56:32.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.588+0000 7f031f70b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03180ffc20 0x7f031819c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:56:32.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.588+0000 7f031f70b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0318105bd0 0x7f031819cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:56:32.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.588+0000 7f031f70b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f031819d170 con 0x7f0318105bd0 2024-04-22T23:56:32.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.588+0000 7f031f70b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03181966b0 con 0x7f03180ffc20 2024-04-22T23:56:32.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.588+0000 7f031cca6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0318105bd0 0x7f031819cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:56:32.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.588+0000 7f031cca6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0318105bd0 0x7f031819cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60252/0 (socket says 172.21.15.67:60252) 2024-04-22T23:56:32.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.588+0000 7f031cca6700 1 -- 172.21.15.67:0/1940909343 learned_addr learned my addr 172.21.15.67:0/1940909343 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:56:32.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.588+0000 7f031d4a7700 1 --2- 172.21.15.67:0/1940909343 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03180ffc20 0x7f031819c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:56:32.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.588+0000 7f031cca6700 1 -- 172.21.15.67:0/1940909343 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03180ffc20 msgr2=0x7f031819c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:32.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.588+0000 7f031cca6700 1 --2- 172.21.15.67:0/1940909343 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03180ffc20 0x7f031819c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:32.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.588+0000 7f031cca6700 1 -- 172.21.15.67:0/1940909343 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03080095f0 con 0x7f0318105bd0 2024-04-22T23:56:32.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.589+0000 7f031cca6700 1 --2- 172.21.15.67:0/1940909343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0318105bd0 0x7f031819cb70 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f0308006010 tx=0x7f030800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:56:32.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.589+0000 7f030e7fc700 1 -- 172.21.15.67:0/1940909343 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f030801d070 con 0x7f0318105bd0 2024-04-22T23:56:32.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.589+0000 7f030e7fc700 1 -- 172.21.15.67:0/1940909343 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f030800f460 con 0x7f0318105bd0 2024-04-22T23:56:32.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.589+0000 7f030e7fc700 1 -- 172.21.15.67:0/1940909343 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0308017630 con 0x7f0318105bd0 2024-04-22T23:56:32.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.589+0000 7f031f70b700 1 -- 172.21.15.67:0/1940909343 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0318196930 con 0x7f0318105bd0 2024-04-22T23:56:32.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.589+0000 7f031f70b700 1 -- 172.21.15.67:0/1940909343 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0318196e20 con 0x7f0318105bd0 2024-04-22T23:56:32.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.590+0000 7f031f70b700 1 -- 172.21.15.67:0/1940909343 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f031804f2e0 con 0x7f0318105bd0 2024-04-22T23:56:32.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.590+0000 7f030e7fc700 1 -- 172.21.15.67:0/1940909343 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f030800f5d0 con 0x7f0318105bd0 2024-04-22T23:56:32.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.591+0000 7f030e7fc700 1 --2- 172.21.15.67:0/1940909343 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f030406ecc0 0x7f0304071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:56:32.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.591+0000 7f030e7fc700 1 -- 172.21.15.67:0/1940909343 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f03080904e0 con 0x7f0318105bd0 2024-04-22T23:56:32.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.591+0000 7f031d4a7700 1 --2- 172.21.15.67:0/1940909343 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f030406ecc0 0x7f0304071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:56:32.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.591+0000 7f031d4a7700 1 --2- 172.21.15.67:0/1940909343 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f030406ecc0 0x7f0304071180 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f031400ac50 tx=0x7f031400a380 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:56:32.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.596+0000 7f030e7fc700 1 -- 172.21.15.67:0/1940909343 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f030805cc30 con 0x7f0318105bd0 2024-04-22T23:56:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.826+0000 7f031f70b700 1 -- 172.21.15.67:0/1940909343 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0318197090 con 0x7f030406ecc0 2024-04-22T23:56:32.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.828+0000 7f030e7fc700 1 -- 172.21.15.67:0/1940909343 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f0318197090 con 0x7f030406ecc0 2024-04-22T23:56:32.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.830+0000 7f031f70b700 1 -- 172.21.15.67:0/1940909343 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f030406ecc0 msgr2=0x7f0304071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:32.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.830+0000 7f031f70b700 1 --2- 172.21.15.67:0/1940909343 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f030406ecc0 0x7f0304071180 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f031400ac50 tx=0x7f031400a380 comp rx=0 tx=0).stop 2024-04-22T23:56:32.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.830+0000 7f031f70b700 1 -- 172.21.15.67:0/1940909343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0318105bd0 msgr2=0x7f031819cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:32.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.830+0000 7f031f70b700 1 --2- 172.21.15.67:0/1940909343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0318105bd0 0x7f031819cb70 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f0308006010 tx=0x7f030800bd70 comp rx=0 tx=0).stop 2024-04-22T23:56:32.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.830+0000 7f031f70b700 1 -- 172.21.15.67:0/1940909343 shutdown_connections 2024-04-22T23:56:32.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.830+0000 7f031f70b700 1 --2- 172.21.15.67:0/1940909343 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f030406ecc0 0x7f0304071180 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:32.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.830+0000 7f031f70b700 1 --2- 172.21.15.67:0/1940909343 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f03180ffc20 0x7f031819c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:32.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.830+0000 7f031f70b700 1 --2- 172.21.15.67:0/1940909343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0318105bd0 0x7f031819cb70 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:32.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.830+0000 7f031f70b700 1 -- 172.21.15.67:0/1940909343 >> 172.21.15.67:0/1940909343 conn(0x7f0318075670 msgr2=0x7f0318068680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:56:32.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.830+0000 7f031f70b700 1 -- 172.21.15.67:0/1940909343 shutdown_connections 2024-04-22T23:56:32.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.830+0000 7f031f70b700 1 -- 172.21.15.67:0/1940909343 wait complete. 2024-04-22T23:56:32.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.936+0000 7f3b88349700 1 -- 172.21.15.67:0/3526293358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b80100750 msgr2=0x7f3b801047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:32.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.936+0000 7f3b88349700 1 --2- 172.21.15.67:0/3526293358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b80100750 0x7f3b801047a0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f3b70009910 tx=0x7f3b70009c20 comp rx=0 tx=0).stop 2024-04-22T23:56:32.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.937+0000 7f3b88349700 1 -- 172.21.15.67:0/3526293358 shutdown_connections 2024-04-22T23:56:32.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.937+0000 7f3b88349700 1 --2- 172.21.15.67:0/3526293358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b80100750 0x7f3b801047a0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:32.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.937+0000 7f3b88349700 1 --2- 172.21.15.67:0/3526293358 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3b800ffda0 0x7f3b80100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:32.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.937+0000 7f3b88349700 1 -- 172.21.15.67:0/3526293358 >> 172.21.15.67:0/3526293358 conn(0x7f3b800fb650 msgr2=0x7f3b800fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:56:32.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.937+0000 7f3b88349700 1 -- 172.21.15.67:0/3526293358 shutdown_connections 2024-04-22T23:56:32.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.937+0000 7f3b88349700 1 -- 172.21.15.67:0/3526293358 wait complete. 2024-04-22T23:56:32.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.937+0000 7f3b88349700 1 Processor -- start 2024-04-22T23:56:32.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.937+0000 7f3b88349700 1 -- start start 2024-04-22T23:56:32.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.937+0000 7f3b88349700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3b800ffda0 0x7f3b8019c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:56:32.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.937+0000 7f3b88349700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b80100750 0x7f3b8019cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:56:32.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.937+0000 7f3b88349700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8019d240 con 0x7f3b80100750 2024-04-22T23:56:32.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.937+0000 7f3b88349700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b801966f0 con 0x7f3b800ffda0 2024-04-22T23:56:32.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.938+0000 7f3b858e4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b80100750 0x7f3b8019cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:56:32.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.938+0000 7f3b858e4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b80100750 0x7f3b8019cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60276/0 (socket says 172.21.15.67:60276) 2024-04-22T23:56:32.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.938+0000 7f3b858e4700 1 -- 172.21.15.67:0/4245636880 learned_addr learned my addr 172.21.15.67:0/4245636880 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:56:32.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.938+0000 7f3b860e5700 1 --2- 172.21.15.67:0/4245636880 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3b800ffda0 0x7f3b8019c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:56:32.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.938+0000 7f3b858e4700 1 -- 172.21.15.67:0/4245636880 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3b800ffda0 msgr2=0x7f3b8019c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:32.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.938+0000 7f3b858e4700 1 --2- 172.21.15.67:0/4245636880 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3b800ffda0 0x7f3b8019c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:32.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.938+0000 7f3b858e4700 1 -- 172.21.15.67:0/4245636880 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b700095f0 con 0x7f3b80100750 2024-04-22T23:56:32.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.938+0000 7f3b860e5700 1 --2- 172.21.15.67:0/4245636880 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3b800ffda0 0x7f3b8019c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:56:32.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.938+0000 7f3b858e4700 1 --2- 172.21.15.67:0/4245636880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b80100750 0x7f3b8019cbb0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f3b700092e0 tx=0x7f3b7000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:56:32.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.938+0000 7f3b777fe700 1 -- 172.21.15.67:0/4245636880 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b7001d070 con 0x7f3b80100750 2024-04-22T23:56:32.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.938+0000 7f3b88349700 1 -- 172.21.15.67:0/4245636880 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b80196970 con 0x7f3b80100750 2024-04-22T23:56:32.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.938+0000 7f3b777fe700 1 -- 172.21.15.67:0/4245636880 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3b70005dc0 con 0x7f3b80100750 2024-04-22T23:56:32.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.938+0000 7f3b777fe700 1 -- 172.21.15.67:0/4245636880 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b70017400 con 0x7f3b80100750 2024-04-22T23:56:32.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.938+0000 7f3b88349700 1 -- 172.21.15.67:0/4245636880 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b80196e60 con 0x7f3b80100750 2024-04-22T23:56:32.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.939+0000 7f3b88349700 1 -- 172.21.15.67:0/4245636880 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b801091b0 con 0x7f3b80100750 2024-04-22T23:56:32.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.940+0000 7f3b777fe700 1 -- 172.21.15.67:0/4245636880 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3b7000f460 con 0x7f3b80100750 2024-04-22T23:56:32.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.940+0000 7f3b777fe700 1 --2- 172.21.15.67:0/4245636880 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3b6c06ecc0 0x7f3b6c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:56:32.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.941+0000 7f3b777fe700 1 -- 172.21.15.67:0/4245636880 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f3b700904f0 con 0x7f3b80100750 2024-04-22T23:56:32.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.941+0000 7f3b860e5700 1 --2- 172.21.15.67:0/4245636880 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3b6c06ecc0 0x7f3b6c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:56:32.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.941+0000 7f3b860e5700 1 --2- 172.21.15.67:0/4245636880 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3b6c06ecc0 0x7f3b6c071180 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c005950 tx=0x7f3b7c0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:56:32.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:32.945+0000 7f3b777fe700 1 -- 172.21.15.67:0/4245636880 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f3b7005cc60 con 0x7f3b80100750 2024-04-22T23:56:33.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.166+0000 7f3b88349700 1 -- 172.21.15.67:0/4245636880 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3b80197140 con 0x7f3b6c06ecc0 2024-04-22T23:56:33.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.177+0000 7f3b777fe700 1 -- 172.21.15.67:0/4245636880 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f3b80197140 con 0x7f3b6c06ecc0 2024-04-22T23:56:33.168 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:56:33.168 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (30m) 8m ago 41m 24.4M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:56:33.168 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (41m) 8m ago 41m 10.0M - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:56:33.168 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (40m) 8m ago 40m 10.0M - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:56:33.168 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (27m) 8m ago 41m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:56:33.168 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (27m) 8m ago 40m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:56:33.168 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (29m) 8m ago 41m 81.2M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:56:33.168 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (23m) 8m ago 36m 19.4M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:56:33.169 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (23m) 8m ago 36m 20.3M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:56:33.169 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (23m) 8m ago 36m 20.1M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:56:33.169 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (8m) 8m ago 35m 14.0M - 19.0.0-3252-g43be0201 a0434699c1d7 46d3d7e86fee 2024-04-22T23:56:33.169 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (31m) 8m ago 42m 514M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:56:33.169 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (31m) 8m ago 40m 459M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:56:33.169 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (28m) 8m ago 43m 74.2M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:56:33.169 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (28m) 8m ago 40m 54.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:56:33.169 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (30m) 8m ago 41m 15.2M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:56:33.169 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (30m) 8m ago 40m 16.0M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:56:33.169 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (27m) 8m ago 39m 102M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:56:33.169 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (26m) 8m ago 39m 128M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:56:33.169 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (25m) 8m ago 38m 83.8M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:56:33.169 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (25m) 8m ago 38m 101M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:56:33.169 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (24m) 8m ago 38m 86.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:56:33.169 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (24m) 8m ago 37m 81.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:56:33.169 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (30m) 8m ago 41m 78.7M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:56:33.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.179+0000 7f3b88349700 1 -- 172.21.15.67:0/4245636880 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3b6c06ecc0 msgr2=0x7f3b6c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:33.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.179+0000 7f3b88349700 1 --2- 172.21.15.67:0/4245636880 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3b6c06ecc0 0x7f3b6c071180 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c005950 tx=0x7f3b7c0058e0 comp rx=0 tx=0).stop 2024-04-22T23:56:33.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.179+0000 7f3b88349700 1 -- 172.21.15.67:0/4245636880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b80100750 msgr2=0x7f3b8019cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:33.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.179+0000 7f3b88349700 1 --2- 172.21.15.67:0/4245636880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b80100750 0x7f3b8019cbb0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f3b700092e0 tx=0x7f3b7000be20 comp rx=0 tx=0).stop 2024-04-22T23:56:33.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.179+0000 7f3b88349700 1 -- 172.21.15.67:0/4245636880 shutdown_connections 2024-04-22T23:56:33.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.179+0000 7f3b88349700 1 --2- 172.21.15.67:0/4245636880 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3b6c06ecc0 0x7f3b6c071180 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:33.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.179+0000 7f3b88349700 1 --2- 172.21.15.67:0/4245636880 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3b800ffda0 0x7f3b8019c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:33.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.179+0000 7f3b88349700 1 --2- 172.21.15.67:0/4245636880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b80100750 0x7f3b8019cbb0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:33.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.179+0000 7f3b88349700 1 -- 172.21.15.67:0/4245636880 >> 172.21.15.67:0/4245636880 conn(0x7f3b800fb650 msgr2=0x7f3b80069150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:56:33.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.179+0000 7f3b88349700 1 -- 172.21.15.67:0/4245636880 shutdown_connections 2024-04-22T23:56:33.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.179+0000 7f3b88349700 1 -- 172.21.15.67:0/4245636880 wait complete. 2024-04-22T23:56:33.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.285+0000 7faf3e369700 1 -- 172.21.15.67:0/293688613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf38100750 msgr2=0x7faf381047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:33.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.285+0000 7faf3e369700 1 --2- 172.21.15.67:0/293688613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf38100750 0x7faf381047a0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7faf20009910 tx=0x7faf20009c20 comp rx=0 tx=0).stop 2024-04-22T23:56:33.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.285+0000 7faf3e369700 1 -- 172.21.15.67:0/293688613 shutdown_connections 2024-04-22T23:56:33.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.285+0000 7faf3e369700 1 --2- 172.21.15.67:0/293688613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf38100750 0x7faf381047a0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:33.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.285+0000 7faf3e369700 1 --2- 172.21.15.67:0/293688613 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf380ffda0 0x7faf38100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:33.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.285+0000 7faf3e369700 1 -- 172.21.15.67:0/293688613 >> 172.21.15.67:0/293688613 conn(0x7faf380fb650 msgr2=0x7faf380fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:56:33.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.285+0000 7faf3e369700 1 -- 172.21.15.67:0/293688613 shutdown_connections 2024-04-22T23:56:33.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.285+0000 7faf3e369700 1 -- 172.21.15.67:0/293688613 wait complete. 2024-04-22T23:56:33.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.286+0000 7faf3e369700 1 Processor -- start 2024-04-22T23:56:33.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.286+0000 7faf3e369700 1 -- start start 2024-04-22T23:56:33.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.286+0000 7faf3e369700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf380ffda0 0x7faf3819c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:56:33.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.286+0000 7faf3e369700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf38100750 0x7faf3819cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:56:33.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.286+0000 7faf3e369700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf3819d240 con 0x7faf38100750 2024-04-22T23:56:33.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.286+0000 7faf3e369700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf381966f0 con 0x7faf380ffda0 2024-04-22T23:56:33.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.286+0000 7faf377fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf38100750 0x7faf3819cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:56:33.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.286+0000 7faf377fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf38100750 0x7faf3819cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60292/0 (socket says 172.21.15.67:60292) 2024-04-22T23:56:33.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.286+0000 7faf377fe700 1 -- 172.21.15.67:0/1014477537 learned_addr learned my addr 172.21.15.67:0/1014477537 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:56:33.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.286+0000 7faf37fff700 1 --2- 172.21.15.67:0/1014477537 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf380ffda0 0x7faf3819c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:56:33.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.286+0000 7faf377fe700 1 -- 172.21.15.67:0/1014477537 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf380ffda0 msgr2=0x7faf3819c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:33.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.286+0000 7faf377fe700 1 --2- 172.21.15.67:0/1014477537 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf380ffda0 0x7faf3819c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:33.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.286+0000 7faf377fe700 1 -- 172.21.15.67:0/1014477537 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf200095f0 con 0x7faf38100750 2024-04-22T23:56:33.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.287+0000 7faf377fe700 1 --2- 172.21.15.67:0/1014477537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf38100750 0x7faf3819cbb0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7faf2000bda0 tx=0x7faf2000bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:56:33.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.287+0000 7faf357fa700 1 -- 172.21.15.67:0/1014477537 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf2001d070 con 0x7faf38100750 2024-04-22T23:56:33.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.287+0000 7faf357fa700 1 -- 172.21.15.67:0/1014477537 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faf20005dc0 con 0x7faf38100750 2024-04-22T23:56:33.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.287+0000 7faf3e369700 1 -- 172.21.15.67:0/1014477537 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf38196970 con 0x7faf38100750 2024-04-22T23:56:33.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.287+0000 7faf357fa700 1 -- 172.21.15.67:0/1014477537 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf20017400 con 0x7faf38100750 2024-04-22T23:56:33.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.287+0000 7faf3e369700 1 -- 172.21.15.67:0/1014477537 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf38196e60 con 0x7faf38100750 2024-04-22T23:56:33.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.288+0000 7faf3e369700 1 -- 172.21.15.67:0/1014477537 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf381091b0 con 0x7faf38100750 2024-04-22T23:56:33.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.288+0000 7faf357fa700 1 -- 172.21.15.67:0/1014477537 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7faf2000f460 con 0x7faf38100750 2024-04-22T23:56:33.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.289+0000 7faf357fa700 1 --2- 172.21.15.67:0/1014477537 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faf2406ecc0 0x7faf24071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:56:33.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.289+0000 7faf357fa700 1 -- 172.21.15.67:0/1014477537 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7faf200903b0 con 0x7faf38100750 2024-04-22T23:56:33.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.289+0000 7faf37fff700 1 --2- 172.21.15.67:0/1014477537 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faf2406ecc0 0x7faf24071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:56:33.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.289+0000 7faf37fff700 1 --2- 172.21.15.67:0/1014477537 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faf2406ecc0 0x7faf24071180 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7faf28007be0 tx=0x7faf280073d0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:56:33.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.294+0000 7faf357fa700 1 -- 172.21.15.67:0/1014477537 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7faf2005cb20 con 0x7faf38100750 2024-04-22T23:56:33.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:33 smithi134 ceph-mon[85630]: from='client.35636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:56:33.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:33 smithi134 ceph-mon[85630]: from='client.35640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:56:33.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:33 smithi134 ceph-mon[85630]: pgmap v907: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:33.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:33 smithi067 ceph-mon[112438]: from='client.35636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:56:33.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:33 smithi067 ceph-mon[112438]: from='client.35640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:56:33.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:33 smithi067 ceph-mon[112438]: pgmap v907: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:33.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.591+0000 7faf3e369700 1 -- 172.21.15.67:0/1014477537 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7faf3804f2e0 con 0x7faf38100750 2024-04-22T23:56:33.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.592+0000 7faf357fa700 1 -- 172.21.15.67:0/1014477537 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7faf2005c6b0 con 0x7faf38100750 2024-04-22T23:56:33.583 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:56:33.583 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:56:33.583 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:56:33.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:56:33.583 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:56:33.583 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:56:33.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:56:33.583 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:56:33.583 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:56:33.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:56:33.583 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:56:33.583 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 4 2024-04-22T23:56:33.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:56:33.584 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:56:33.584 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 14 2024-04-22T23:56:33.584 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:56:33.584 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:56:33.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.594+0000 7faf3e369700 1 -- 172.21.15.67:0/1014477537 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faf2406ecc0 msgr2=0x7faf24071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:33.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.594+0000 7faf3e369700 1 --2- 172.21.15.67:0/1014477537 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faf2406ecc0 0x7faf24071180 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7faf28007be0 tx=0x7faf280073d0 comp rx=0 tx=0).stop 2024-04-22T23:56:33.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.594+0000 7faf3e369700 1 -- 172.21.15.67:0/1014477537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf38100750 msgr2=0x7faf3819cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:33.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.594+0000 7faf3e369700 1 --2- 172.21.15.67:0/1014477537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf38100750 0x7faf3819cbb0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7faf2000bda0 tx=0x7faf2000bdd0 comp rx=0 tx=0).stop 2024-04-22T23:56:33.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.594+0000 7faf3e369700 1 -- 172.21.15.67:0/1014477537 shutdown_connections 2024-04-22T23:56:33.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.594+0000 7faf3e369700 1 --2- 172.21.15.67:0/1014477537 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faf2406ecc0 0x7faf24071180 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:33.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.594+0000 7faf3e369700 1 --2- 172.21.15.67:0/1014477537 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf380ffda0 0x7faf3819c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:33.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.594+0000 7faf3e369700 1 --2- 172.21.15.67:0/1014477537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf38100750 0x7faf3819cbb0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:33.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.594+0000 7faf3e369700 1 -- 172.21.15.67:0/1014477537 >> 172.21.15.67:0/1014477537 conn(0x7faf380fb650 msgr2=0x7faf38069150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:56:33.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.594+0000 7faf3e369700 1 -- 172.21.15.67:0/1014477537 shutdown_connections 2024-04-22T23:56:33.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.594+0000 7faf3e369700 1 -- 172.21.15.67:0/1014477537 wait complete. 2024-04-22T23:56:33.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.700+0000 7f46138e3700 1 -- 172.21.15.67:0/256216513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f460c0ffda0 msgr2=0x7f460c100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:33.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.700+0000 7f46138e3700 1 --2- 172.21.15.67:0/256216513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f460c0ffda0 0x7f460c100180 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f4608009910 tx=0x7f4608009c20 comp rx=0 tx=0).stop 2024-04-22T23:56:33.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.700+0000 7f46138e3700 1 -- 172.21.15.67:0/256216513 shutdown_connections 2024-04-22T23:56:33.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.700+0000 7f46138e3700 1 --2- 172.21.15.67:0/256216513 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f460c100750 0x7f460c1047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:33.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.700+0000 7f46138e3700 1 --2- 172.21.15.67:0/256216513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f460c0ffda0 0x7f460c100180 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:33.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.700+0000 7f46138e3700 1 -- 172.21.15.67:0/256216513 >> 172.21.15.67:0/256216513 conn(0x7f460c0fb650 msgr2=0x7f460c0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:56:33.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.701+0000 7f46138e3700 1 -- 172.21.15.67:0/256216513 shutdown_connections 2024-04-22T23:56:33.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.701+0000 7f46138e3700 1 -- 172.21.15.67:0/256216513 wait complete. 2024-04-22T23:56:33.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.701+0000 7f46138e3700 1 Processor -- start 2024-04-22T23:56:33.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.701+0000 7f46138e3700 1 -- start start 2024-04-22T23:56:33.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.701+0000 7f46138e3700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f460c0ffda0 0x7f460c101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:56:33.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.701+0000 7f46138e3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f460c100750 0x7f460c101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:56:33.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.701+0000 7f46138e3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f460c1022c0 con 0x7f460c100750 2024-04-22T23:56:33.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.701+0000 7f46138e3700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f460c102400 con 0x7f460c0ffda0 2024-04-22T23:56:33.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.702+0000 7f4610e7e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f460c100750 0x7f460c101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:56:33.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.702+0000 7f4610e7e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f460c100750 0x7f460c101d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60314/0 (socket says 172.21.15.67:60314) 2024-04-22T23:56:33.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.702+0000 7f4610e7e700 1 -- 172.21.15.67:0/217857057 learned_addr learned my addr 172.21.15.67:0/217857057 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:56:33.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.702+0000 7f461167f700 1 --2- 172.21.15.67:0/217857057 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f460c0ffda0 0x7f460c101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:56:33.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.702+0000 7f4610e7e700 1 -- 172.21.15.67:0/217857057 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f460c0ffda0 msgr2=0x7f460c101840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:33.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.702+0000 7f4610e7e700 1 --2- 172.21.15.67:0/217857057 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f460c0ffda0 0x7f460c101840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:33.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.702+0000 7f4610e7e700 1 -- 172.21.15.67:0/217857057 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46080095f0 con 0x7f460c100750 2024-04-22T23:56:33.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.702+0000 7f461167f700 1 --2- 172.21.15.67:0/217857057 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f460c0ffda0 0x7f460c101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:56:33.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.702+0000 7f4610e7e700 1 --2- 172.21.15.67:0/217857057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f460c100750 0x7f460c101d80 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f45fc00d900 tx=0x7f45fc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:56:33.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.702+0000 7f46027fc700 1 -- 172.21.15.67:0/217857057 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45fc009900 con 0x7f460c100750 2024-04-22T23:56:33.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.702+0000 7f46027fc700 1 -- 172.21.15.67:0/217857057 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f45fc010460 con 0x7f460c100750 2024-04-22T23:56:33.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.702+0000 7f46138e3700 1 -- 172.21.15.67:0/217857057 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f460c1a2af0 con 0x7f460c100750 2024-04-22T23:56:33.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.702+0000 7f46027fc700 1 -- 172.21.15.67:0/217857057 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45fc00f5f0 con 0x7f460c100750 2024-04-22T23:56:33.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.702+0000 7f46138e3700 1 -- 172.21.15.67:0/217857057 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f460c1a2f00 con 0x7f460c100750 2024-04-22T23:56:33.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.703+0000 7f46138e3700 1 -- 172.21.15.67:0/217857057 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f460c04f2e0 con 0x7f460c100750 2024-04-22T23:56:33.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.704+0000 7f46027fc700 1 -- 172.21.15.67:0/217857057 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f45fc0105d0 con 0x7f460c100750 2024-04-22T23:56:33.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.704+0000 7f46027fc700 1 --2- 172.21.15.67:0/217857057 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f45f806ecc0 0x7f45f8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:56:33.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.704+0000 7f46027fc700 1 -- 172.21.15.67:0/217857057 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f45fc08ed60 con 0x7f460c100750 2024-04-22T23:56:33.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.704+0000 7f461167f700 1 --2- 172.21.15.67:0/217857057 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f45f806ecc0 0x7f45f8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:56:33.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.705+0000 7f461167f700 1 --2- 172.21.15.67:0/217857057 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f45f806ecc0 0x7f45f8071180 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f4608005b40 tx=0x7f460801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:56:33.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.709+0000 7f46027fc700 1 -- 172.21.15.67:0/217857057 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f45fc057a20 con 0x7f460c100750 2024-04-22T23:56:33.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.966+0000 7f46138e3700 1 -- 172.21.15.67:0/217857057 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f460c102d70 con 0x7f460c100750 2024-04-22T23:56:33.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.966+0000 7f46027fc700 1 -- 172.21.15.67:0/217857057 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1872 (secure 0 0 0) 0x7f45fc017760 con 0x7f460c100750 2024-04-22T23:56:33.957 INFO:teuthology.orchestra.run.smithi067.stdout:e33 2024-04-22T23:56:33.957 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:56:33.957 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:56:33.957 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:56:33.957 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:56:33.958 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:56:33.958 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:56:33.958 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:56:33.958 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:56:33.958 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:56:33.958 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:56:33.958 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:56:33.958 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:56:33.958 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:56:33.958 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:56:33.958 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:56:33.958 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:56:33.958 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:56:33.958 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:56:33.958 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:56:33.958 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:56:33.959 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:56:33.959 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:56:33.959 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:56:33.959 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:56:33.959 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:56:33.959 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:56:33.959 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:56:33.959 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:56:33.959 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:56:33.959 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:56:33.959 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:56:33.959 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:56:33.959 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:56:33.959 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:56:33.959 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:56:33.960 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:56:33.960 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:56:33.960 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:56:33.960 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:56:33.960 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:56:33.960 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44851} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1849648622,v1:172.21.15.134:6827/1849648622] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:56:33.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.968+0000 7f46138e3700 1 -- 172.21.15.67:0/217857057 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f45f806ecc0 msgr2=0x7f45f8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:33.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.968+0000 7f46138e3700 1 --2- 172.21.15.67:0/217857057 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f45f806ecc0 0x7f45f8071180 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f4608005b40 tx=0x7f460801a040 comp rx=0 tx=0).stop 2024-04-22T23:56:33.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.968+0000 7f46138e3700 1 -- 172.21.15.67:0/217857057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f460c100750 msgr2=0x7f460c101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:33.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.968+0000 7f46138e3700 1 --2- 172.21.15.67:0/217857057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f460c100750 0x7f460c101d80 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f45fc00d900 tx=0x7f45fc00dcc0 comp rx=0 tx=0).stop 2024-04-22T23:56:33.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.969+0000 7f46138e3700 1 -- 172.21.15.67:0/217857057 shutdown_connections 2024-04-22T23:56:33.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.969+0000 7f46138e3700 1 --2- 172.21.15.67:0/217857057 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f45f806ecc0 0x7f45f8071180 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:33.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.969+0000 7f46138e3700 1 --2- 172.21.15.67:0/217857057 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f460c0ffda0 0x7f460c101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:33.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.969+0000 7f46138e3700 1 --2- 172.21.15.67:0/217857057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f460c100750 0x7f460c101d80 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:33.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.969+0000 7f46138e3700 1 -- 172.21.15.67:0/217857057 >> 172.21.15.67:0/217857057 conn(0x7f460c0fb650 msgr2=0x7f460c103fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:56:33.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.969+0000 7f46138e3700 1 -- 172.21.15.67:0/217857057 shutdown_connections 2024-04-22T23:56:33.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:33.969+0000 7f46138e3700 1 -- 172.21.15.67:0/217857057 wait complete. 2024-04-22T23:56:33.961 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 33 2024-04-22T23:56:34.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.077+0000 7f691cb14700 1 -- 172.21.15.67:0/2068387793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6918101850 msgr2=0x7f69181058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:34.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.077+0000 7f691cb14700 1 --2- 172.21.15.67:0/2068387793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6918101850 0x7f69181058a0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f6900009910 tx=0x7f6900009c20 comp rx=0 tx=0).stop 2024-04-22T23:56:34.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.077+0000 7f691cb14700 1 -- 172.21.15.67:0/2068387793 shutdown_connections 2024-04-22T23:56:34.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.077+0000 7f691cb14700 1 --2- 172.21.15.67:0/2068387793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6918101850 0x7f69181058a0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:34.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.077+0000 7f691cb14700 1 --2- 172.21.15.67:0/2068387793 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6918100ea0 0x7f6918101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:34.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.078+0000 7f691cb14700 1 -- 172.21.15.67:0/2068387793 >> 172.21.15.67:0/2068387793 conn(0x7f69180fc750 msgr2=0x7f69180feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:56:34.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.078+0000 7f691cb14700 1 -- 172.21.15.67:0/2068387793 shutdown_connections 2024-04-22T23:56:34.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.078+0000 7f691cb14700 1 -- 172.21.15.67:0/2068387793 wait complete. 2024-04-22T23:56:34.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.078+0000 7f691cb14700 1 Processor -- start 2024-04-22T23:56:34.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.078+0000 7f691cb14700 1 -- start start 2024-04-22T23:56:34.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.079+0000 7f691cb14700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6918100ea0 0x7f691819c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:56:34.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.079+0000 7f691cb14700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6918101850 0x7f691819cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:56:34.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.079+0000 7f691cb14700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691819d2d0 con 0x7f6918100ea0 2024-04-22T23:56:34.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.079+0000 7f691cb14700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6918196780 con 0x7f6918101850 2024-04-22T23:56:34.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.079+0000 7f691659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6918100ea0 0x7f691819c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:56:34.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.079+0000 7f691659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6918100ea0 0x7f691819c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60334/0 (socket says 172.21.15.67:60334) 2024-04-22T23:56:34.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.079+0000 7f691659c700 1 -- 172.21.15.67:0/3531092472 learned_addr learned my addr 172.21.15.67:0/3531092472 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:56:34.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.079+0000 7f6915d9b700 1 --2- 172.21.15.67:0/3531092472 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6918101850 0x7f691819cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:56:34.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.079+0000 7f691659c700 1 -- 172.21.15.67:0/3531092472 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6918101850 msgr2=0x7f691819cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:34.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.079+0000 7f691659c700 1 --2- 172.21.15.67:0/3531092472 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6918101850 0x7f691819cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:34.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.079+0000 7f691659c700 1 -- 172.21.15.67:0/3531092472 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69000095f0 con 0x7f6918100ea0 2024-04-22T23:56:34.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.079+0000 7f6915d9b700 1 --2- 172.21.15.67:0/3531092472 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6918101850 0x7f691819cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:56:34.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.079+0000 7f691659c700 1 --2- 172.21.15.67:0/3531092472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6918100ea0 0x7f691819c700 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f690800d8d0 tx=0x7f690800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:56:34.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.079+0000 7f690f7fe700 1 -- 172.21.15.67:0/3531092472 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6908009940 con 0x7f6918100ea0 2024-04-22T23:56:34.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.079+0000 7f691cb14700 1 -- 172.21.15.67:0/3531092472 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6918196a60 con 0x7f6918100ea0 2024-04-22T23:56:34.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.079+0000 7f690f7fe700 1 -- 172.21.15.67:0/3531092472 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6908010460 con 0x7f6918100ea0 2024-04-22T23:56:34.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.080+0000 7f690f7fe700 1 -- 172.21.15.67:0/3531092472 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f690800f5f0 con 0x7f6918100ea0 2024-04-22T23:56:34.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.080+0000 7f691cb14700 1 -- 172.21.15.67:0/3531092472 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6918196fb0 con 0x7f6918100ea0 2024-04-22T23:56:34.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.080+0000 7f691cb14700 1 -- 172.21.15.67:0/3531092472 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69181092a0 con 0x7f6918100ea0 2024-04-22T23:56:34.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.081+0000 7f690f7fe700 1 -- 172.21.15.67:0/3531092472 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f69080105d0 con 0x7f6918100ea0 2024-04-22T23:56:34.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.081+0000 7f690f7fe700 1 --2- 172.21.15.67:0/3531092472 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f690406ecc0 0x7f6904071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:56:34.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.082+0000 7f690f7fe700 1 -- 172.21.15.67:0/3531092472 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f690808efe0 con 0x7f6918100ea0 2024-04-22T23:56:34.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.082+0000 7f6915d9b700 1 --2- 172.21.15.67:0/3531092472 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f690406ecc0 0x7f6904071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:56:34.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.082+0000 7f6915d9b700 1 --2- 172.21.15.67:0/3531092472 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f690406ecc0 0x7f6904071180 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f6918197bf0 tx=0x7f690001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:56:34.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.086+0000 7f690f7fe700 1 -- 172.21.15.67:0/3531092472 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f690805b750 con 0x7f6918100ea0 2024-04-22T23:56:34.284 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:34 smithi067 ceph-mon[112438]: from='client.35644 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:56:34.284 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:34 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1014477537' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:56:34.284 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:34 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/217857057' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:56:34.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.322+0000 7f691cb14700 1 -- 172.21.15.67:0/3531092472 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6918197da0 con 0x7f690406ecc0 2024-04-22T23:56:34.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.323+0000 7f690f7fe700 1 -- 172.21.15.67:0/3531092472 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6918197da0 con 0x7f690406ecc0 2024-04-22T23:56:34.314 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:56:34.314 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:56:34.314 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:56:34.314 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:56:34.315 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:56:34.315 INFO:teuthology.orchestra.run.smithi067.stdout: "mds", 2024-04-22T23:56:34.315 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:56:34.315 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:56:34.315 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:56:34.315 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:56:34.315 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:56:34.315 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T23:56:34.315 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:56:34.315 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:56:34.315 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:56:34.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.325+0000 7f691cb14700 1 -- 172.21.15.67:0/3531092472 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f690406ecc0 msgr2=0x7f6904071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:34.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.325+0000 7f691cb14700 1 --2- 172.21.15.67:0/3531092472 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f690406ecc0 0x7f6904071180 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f6918197bf0 tx=0x7f690001a040 comp rx=0 tx=0).stop 2024-04-22T23:56:34.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.325+0000 7f691cb14700 1 -- 172.21.15.67:0/3531092472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6918100ea0 msgr2=0x7f691819c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:34.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.325+0000 7f691cb14700 1 --2- 172.21.15.67:0/3531092472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6918100ea0 0x7f691819c700 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f690800d8d0 tx=0x7f690800dc90 comp rx=0 tx=0).stop 2024-04-22T23:56:34.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.325+0000 7f691cb14700 1 -- 172.21.15.67:0/3531092472 shutdown_connections 2024-04-22T23:56:34.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.325+0000 7f691cb14700 1 --2- 172.21.15.67:0/3531092472 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f690406ecc0 0x7f6904071180 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:34.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.325+0000 7f691cb14700 1 --2- 172.21.15.67:0/3531092472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6918100ea0 0x7f691819c700 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:34.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.325+0000 7f691cb14700 1 --2- 172.21.15.67:0/3531092472 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6918101850 0x7f691819cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:34.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.325+0000 7f691cb14700 1 -- 172.21.15.67:0/3531092472 >> 172.21.15.67:0/3531092472 conn(0x7f69180fc750 msgr2=0x7f69180fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:56:34.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.326+0000 7f691cb14700 1 -- 172.21.15.67:0/3531092472 shutdown_connections 2024-04-22T23:56:34.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.326+0000 7f691cb14700 1 -- 172.21.15.67:0/3531092472 wait complete. 2024-04-22T23:56:34.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.432+0000 7f101125b700 1 -- 172.21.15.67:0/1699619691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f100c101840 msgr2=0x7f100c105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:34.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.432+0000 7f101125b700 1 --2- 172.21.15.67:0/1699619691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f100c101840 0x7f100c105890 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc009910 tx=0x7f0ffc009c20 comp rx=0 tx=0).stop 2024-04-22T23:56:34.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.432+0000 7f101125b700 1 -- 172.21.15.67:0/1699619691 shutdown_connections 2024-04-22T23:56:34.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.432+0000 7f101125b700 1 --2- 172.21.15.67:0/1699619691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f100c101840 0x7f100c105890 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:34.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.432+0000 7f101125b700 1 --2- 172.21.15.67:0/1699619691 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f100c100e90 0x7f100c101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:34.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.432+0000 7f101125b700 1 -- 172.21.15.67:0/1699619691 >> 172.21.15.67:0/1699619691 conn(0x7f100c0fc740 msgr2=0x7f100c0feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:56:34.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.432+0000 7f101125b700 1 -- 172.21.15.67:0/1699619691 shutdown_connections 2024-04-22T23:56:34.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.432+0000 7f101125b700 1 -- 172.21.15.67:0/1699619691 wait complete. 2024-04-22T23:56:34.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.433+0000 7f101125b700 1 Processor -- start 2024-04-22T23:56:34.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.433+0000 7f101125b700 1 -- start start 2024-04-22T23:56:34.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.433+0000 7f101125b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f100c100e90 0x7f100c073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:56:34.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.433+0000 7f101125b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f100c101840 0x7f100c0735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:56:34.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.433+0000 7f101125b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100c073b10 con 0x7f100c100e90 2024-04-22T23:56:34.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.433+0000 7f101125b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100c073c50 con 0x7f100c101840 2024-04-22T23:56:34.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.433+0000 7f100ad9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f100c100e90 0x7f100c073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:56:34.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.433+0000 7f100ad9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f100c100e90 0x7f100c073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60352/0 (socket says 172.21.15.67:60352) 2024-04-22T23:56:34.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.433+0000 7f100ad9d700 1 -- 172.21.15.67:0/4065182215 learned_addr learned my addr 172.21.15.67:0/4065182215 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:56:34.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.434+0000 7f100a59c700 1 --2- 172.21.15.67:0/4065182215 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f100c101840 0x7f100c0735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:56:34.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.434+0000 7f100ad9d700 1 -- 172.21.15.67:0/4065182215 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f100c101840 msgr2=0x7f100c0735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:34.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.434+0000 7f100ad9d700 1 --2- 172.21.15.67:0/4065182215 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f100c101840 0x7f100c0735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:34.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.434+0000 7f100ad9d700 1 -- 172.21.15.67:0/4065182215 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ffc0095f0 con 0x7f100c100e90 2024-04-22T23:56:34.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.434+0000 7f100a59c700 1 --2- 172.21.15.67:0/4065182215 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f100c101840 0x7f100c0735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:56:34.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.434+0000 7f100ad9d700 1 --2- 172.21.15.67:0/4065182215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f100c100e90 0x7f100c073090 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f100000c930 tx=0x7f100000ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:56:34.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.434+0000 7f0ff3fff700 1 -- 172.21.15.67:0/4065182215 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1000007ad0 con 0x7f100c100e90 2024-04-22T23:56:34.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.434+0000 7f101125b700 1 -- 172.21.15.67:0/4065182215 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f100c1a2a70 con 0x7f100c100e90 2024-04-22T23:56:34.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.434+0000 7f0ff3fff700 1 -- 172.21.15.67:0/4065182215 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1000007c30 con 0x7f100c100e90 2024-04-22T23:56:34.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.434+0000 7f0ff3fff700 1 -- 172.21.15.67:0/4065182215 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1000018610 con 0x7f100c100e90 2024-04-22T23:56:34.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.434+0000 7f101125b700 1 -- 172.21.15.67:0/4065182215 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f100c1a2dd0 con 0x7f100c100e90 2024-04-22T23:56:34.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.435+0000 7f101125b700 1 -- 172.21.15.67:0/4065182215 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f100c109290 con 0x7f100c100e90 2024-04-22T23:56:34.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.436+0000 7f0ff3fff700 1 -- 172.21.15.67:0/4065182215 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1000018880 con 0x7f100c100e90 2024-04-22T23:56:34.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.436+0000 7f0ff3fff700 1 --2- 172.21.15.67:0/4065182215 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0ff406ecc0 0x7f0ff4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:56:34.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.436+0000 7f0ff3fff700 1 -- 172.21.15.67:0/4065182215 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f100008f1c0 con 0x7f100c100e90 2024-04-22T23:56:34.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.436+0000 7f100a59c700 1 --2- 172.21.15.67:0/4065182215 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0ff406ecc0 0x7f0ff4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:56:34.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.437+0000 7f100a59c700 1 --2- 172.21.15.67:0/4065182215 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0ff406ecc0 0x7f0ff4071180 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc006010 tx=0x7f0ffc005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:56:34.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.441+0000 7f0ff3fff700 1 -- 172.21.15.67:0/4065182215 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1000057e00 con 0x7f100c100e90 2024-04-22T23:56:34.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:34 smithi134 ceph-mon[85630]: from='client.35644 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:56:34.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:34 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1014477537' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:56:34.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:34 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/217857057' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:56:34.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.735+0000 7f101125b700 1 -- 172.21.15.67:0/4065182215 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f100c04f2e0 con 0x7f100c100e90 2024-04-22T23:56:34.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.736+0000 7f0ff3fff700 1 -- 172.21.15.67:0/4065182215 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4505 (secure 0 0 0) 0x7f100005b4c0 con 0x7f100c100e90 2024-04-22T23:56:34.727 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:56:34.727 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T23:56:34.727 INFO:teuthology.orchestra.run.smithi067.stdout: host smithi134 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-22T23:56:34.727 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T23:56:34.727 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:56:34.727 INFO:teuthology.orchestra.run.smithi067.stdout:Traceback (most recent call last): 2024-04-22T23:56:34.727 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T23:56:34.727 INFO:teuthology.orchestra.run.smithi067.stdout: return _run_code(code, main_globals, None, 2024-04-22T23:56:34.727 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T23:56:34.727 INFO:teuthology.orchestra.run.smithi067.stdout: exec(code, run_globals) 2024-04-22T23:56:34.728 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5642, in 2024-04-22T23:56:34.728 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5630, in main 2024-04-22T23:56:34.728 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 408, in _infer_config 2024-04-22T23:56:34.728 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 323, in _infer_fsid 2024-04-22T23:56:34.728 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 436, in _infer_image 2024-04-22T23:56:34.728 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 310, in _validate_fsid 2024-04-22T23:56:34.728 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 3380, in command_ceph_volume 2024-04-22T23:56:34.728 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T23:56:34.728 INFO:teuthology.orchestra.run.smithi067.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:56:34.728 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:56:34.728 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:56:34.728 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:56:34.728 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:56:34.728 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:56:34.728 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:56:34.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.738+0000 7f101125b700 1 -- 172.21.15.67:0/4065182215 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0ff406ecc0 msgr2=0x7f0ff4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:34.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.738+0000 7f101125b700 1 --2- 172.21.15.67:0/4065182215 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0ff406ecc0 0x7f0ff4071180 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc006010 tx=0x7f0ffc005960 comp rx=0 tx=0).stop 2024-04-22T23:56:34.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.738+0000 7f101125b700 1 -- 172.21.15.67:0/4065182215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f100c100e90 msgr2=0x7f100c073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:56:34.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.738+0000 7f101125b700 1 --2- 172.21.15.67:0/4065182215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f100c100e90 0x7f100c073090 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f100000c930 tx=0x7f100000ccf0 comp rx=0 tx=0).stop 2024-04-22T23:56:34.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.738+0000 7f101125b700 1 -- 172.21.15.67:0/4065182215 shutdown_connections 2024-04-22T23:56:34.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.738+0000 7f101125b700 1 --2- 172.21.15.67:0/4065182215 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0ff406ecc0 0x7f0ff4071180 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:34.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.738+0000 7f101125b700 1 --2- 172.21.15.67:0/4065182215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f100c100e90 0x7f100c073090 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:34.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.738+0000 7f101125b700 1 --2- 172.21.15.67:0/4065182215 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f100c101840 0x7f100c0735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:56:34.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.738+0000 7f101125b700 1 -- 172.21.15.67:0/4065182215 >> 172.21.15.67:0/4065182215 conn(0x7f100c0fc740 msgr2=0x7f100c0fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:56:34.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.738+0000 7f101125b700 1 -- 172.21.15.67:0/4065182215 shutdown_connections 2024-04-22T23:56:34.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:56:34.738+0000 7f101125b700 1 -- 172.21.15.67:0/4065182215 wait complete. 2024-04-22T23:56:35.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:35 smithi134 ceph-mon[85630]: from='client.35656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:56:35.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:35 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/4065182215' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:56:35.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:35 smithi134 ceph-mon[85630]: pgmap v908: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:35.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:35 smithi067 ceph-mon[112438]: from='client.35656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:56:35.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:35 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/4065182215' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:56:35.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:35 smithi067 ceph-mon[112438]: pgmap v908: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:37.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:37 smithi134 ceph-mon[85630]: pgmap v909: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:37.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:37 smithi067 ceph-mon[112438]: pgmap v909: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:39.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:39 smithi134 ceph-mon[85630]: pgmap v910: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:39.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:39 smithi067 ceph-mon[112438]: pgmap v910: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:41.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:56:41.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:56:42.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:42 smithi134 ceph-mon[85630]: pgmap v911: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:42.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:42 smithi067 ceph-mon[112438]: pgmap v911: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:44.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:44 smithi134 ceph-mon[85630]: pgmap v912: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:44.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:44 smithi067 ceph-mon[112438]: pgmap v912: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:45.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:45 smithi134 ceph-mon[85630]: pgmap v913: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:45.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:45 smithi067 ceph-mon[112438]: pgmap v913: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:47.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:47 smithi134 ceph-mon[85630]: pgmap v914: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:47.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:47 smithi067 ceph-mon[112438]: pgmap v914: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:49.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:49 smithi134 ceph-mon[85630]: pgmap v915: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:49.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:49 smithi067 ceph-mon[112438]: pgmap v915: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:51.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:51 smithi134 ceph-mon[85630]: pgmap v916: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:51.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:51 smithi067 ceph-mon[112438]: pgmap v916: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:53.284 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:53 smithi067 ceph-mon[112438]: pgmap v917: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:53.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:53 smithi134 ceph-mon[85630]: pgmap v917: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:55.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:55 smithi134 ceph-mon[85630]: pgmap v918: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:55.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:55 smithi067 ceph-mon[112438]: pgmap v918: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:56.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:56:56.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:56:57.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:57 smithi134 ceph-mon[85630]: pgmap v919: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:57.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:57 smithi067 ceph-mon[112438]: pgmap v919: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:59.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:56:59 smithi134 ceph-mon[85630]: pgmap v920: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:56:59.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:56:59 smithi067 ceph-mon[112438]: pgmap v920: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:01.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:01 smithi134 ceph-mon[85630]: pgmap v921: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:01.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:01 smithi067 ceph-mon[112438]: pgmap v921: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:03.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:03 smithi134 ceph-mon[85630]: pgmap v922: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:03.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:03 smithi067 ceph-mon[112438]: pgmap v922: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:04.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:04.859+0000 7f0494ebb700 1 -- 172.21.15.67:0/944171097 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0490101850 msgr2=0x7f04901058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:04.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:04.859+0000 7f0494ebb700 1 --2- 172.21.15.67:0/944171097 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0490101850 0x7f04901058a0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f0484009910 tx=0x7f0484009c20 comp rx=0 tx=0).stop 2024-04-22T23:57:04.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:04.859+0000 7f0494ebb700 1 -- 172.21.15.67:0/944171097 shutdown_connections 2024-04-22T23:57:04.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:04.859+0000 7f0494ebb700 1 --2- 172.21.15.67:0/944171097 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0490101850 0x7f04901058a0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:04.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:04.859+0000 7f0494ebb700 1 --2- 172.21.15.67:0/944171097 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0490100ea0 0x7f0490101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:04.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:04.859+0000 7f0494ebb700 1 -- 172.21.15.67:0/944171097 >> 172.21.15.67:0/944171097 conn(0x7f04900fc750 msgr2=0x7f04900feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:57:04.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:04.859+0000 7f0494ebb700 1 -- 172.21.15.67:0/944171097 shutdown_connections 2024-04-22T23:57:04.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:04.859+0000 7f0494ebb700 1 -- 172.21.15.67:0/944171097 wait complete. 2024-04-22T23:57:04.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:04.860+0000 7f0494ebb700 1 Processor -- start 2024-04-22T23:57:04.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:04.860+0000 7f0494ebb700 1 -- start start 2024-04-22T23:57:04.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:04.860+0000 7f0494ebb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0490100ea0 0x7f0490073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:04.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:04.860+0000 7f0494ebb700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0490101850 0x7f04900735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:04.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:04.860+0000 7f0494ebb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0490073b10 con 0x7f0490100ea0 2024-04-22T23:57:04.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:04.860+0000 7f0494ebb700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0490073c50 con 0x7f0490101850 2024-04-22T23:57:04.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:04.860+0000 7f048e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0490100ea0 0x7f0490073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:04.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:04.860+0000 7f048e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0490100ea0 0x7f0490073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41220/0 (socket says 172.21.15.67:41220) 2024-04-22T23:57:04.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:04.860+0000 7f048e59c700 1 -- 172.21.15.67:0/906863102 learned_addr learned my addr 172.21.15.67:0/906863102 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:57:04.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:04.861+0000 7f048dd9b700 1 --2- 172.21.15.67:0/906863102 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0490101850 0x7f04900735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:04.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:04.861+0000 7f048e59c700 1 -- 172.21.15.67:0/906863102 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0490101850 msgr2=0x7f04900735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:04.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:04.861+0000 7f048e59c700 1 --2- 172.21.15.67:0/906863102 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0490101850 0x7f04900735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:04.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:04.861+0000 7f048e59c700 1 -- 172.21.15.67:0/906863102 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04840095f0 con 0x7f0490100ea0 2024-04-22T23:57:04.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:04.861+0000 7f048e59c700 1 --2- 172.21.15.67:0/906863102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0490100ea0 0x7f0490073090 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f048000c930 tx=0x7f048000ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:57:04.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:04.861+0000 7f047f7fe700 1 -- 172.21.15.67:0/906863102 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0480007ad0 con 0x7f0490100ea0 2024-04-22T23:57:04.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:04.861+0000 7f047f7fe700 1 -- 172.21.15.67:0/906863102 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0480007c30 con 0x7f0490100ea0 2024-04-22T23:57:04.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:04.861+0000 7f0494ebb700 1 -- 172.21.15.67:0/906863102 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04901a2b00 con 0x7f0490100ea0 2024-04-22T23:57:04.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:04.861+0000 7f047f7fe700 1 -- 172.21.15.67:0/906863102 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0480018610 con 0x7f0490100ea0 2024-04-22T23:57:04.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:04.861+0000 7f0494ebb700 1 -- 172.21.15.67:0/906863102 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04901a2e60 con 0x7f0490100ea0 2024-04-22T23:57:04.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:04.862+0000 7f0494ebb700 1 -- 172.21.15.67:0/906863102 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04901092a0 con 0x7f0490100ea0 2024-04-22T23:57:04.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:04.863+0000 7f047f7fe700 1 -- 172.21.15.67:0/906863102 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f048000f450 con 0x7f0490100ea0 2024-04-22T23:57:04.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:04.863+0000 7f047f7fe700 1 --2- 172.21.15.67:0/906863102 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f047806ecc0 0x7f0478071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:04.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:04.863+0000 7f047f7fe700 1 -- 172.21.15.67:0/906863102 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f048008f1f0 con 0x7f0490100ea0 2024-04-22T23:57:04.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:04.864+0000 7f048dd9b700 1 --2- 172.21.15.67:0/906863102 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f047806ecc0 0x7f0478071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:04.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:04.864+0000 7f048dd9b700 1 --2- 172.21.15.67:0/906863102 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f047806ecc0 0x7f0478071180 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f0490074880 tx=0x7f048401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:57:04.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:04.868+0000 7f047f7fe700 1 -- 172.21.15.67:0/906863102 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f0480057e30 con 0x7f0490100ea0 2024-04-22T23:57:05.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.097+0000 7f0494ebb700 1 -- 172.21.15.67:0/906863102 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f04900611d0 con 0x7f047806ecc0 2024-04-22T23:57:05.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.098+0000 7f047f7fe700 1 -- 172.21.15.67:0/906863102 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f04900611d0 con 0x7f047806ecc0 2024-04-22T23:57:05.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.100+0000 7f0494ebb700 1 -- 172.21.15.67:0/906863102 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f047806ecc0 msgr2=0x7f0478071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:05.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.100+0000 7f0494ebb700 1 --2- 172.21.15.67:0/906863102 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f047806ecc0 0x7f0478071180 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f0490074880 tx=0x7f048401a040 comp rx=0 tx=0).stop 2024-04-22T23:57:05.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.100+0000 7f0494ebb700 1 -- 172.21.15.67:0/906863102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0490100ea0 msgr2=0x7f0490073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:05.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.100+0000 7f0494ebb700 1 --2- 172.21.15.67:0/906863102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0490100ea0 0x7f0490073090 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f048000c930 tx=0x7f048000ccf0 comp rx=0 tx=0).stop 2024-04-22T23:57:05.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.100+0000 7f0494ebb700 1 -- 172.21.15.67:0/906863102 shutdown_connections 2024-04-22T23:57:05.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.100+0000 7f0494ebb700 1 --2- 172.21.15.67:0/906863102 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f047806ecc0 0x7f0478071180 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:05.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.100+0000 7f0494ebb700 1 --2- 172.21.15.67:0/906863102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0490100ea0 0x7f0490073090 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:05.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.100+0000 7f0494ebb700 1 --2- 172.21.15.67:0/906863102 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0490101850 0x7f04900735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:05.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.100+0000 7f0494ebb700 1 -- 172.21.15.67:0/906863102 >> 172.21.15.67:0/906863102 conn(0x7f04900fc750 msgr2=0x7f04900fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:57:05.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.100+0000 7f0494ebb700 1 -- 172.21.15.67:0/906863102 shutdown_connections 2024-04-22T23:57:05.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.100+0000 7f0494ebb700 1 -- 172.21.15.67:0/906863102 wait complete. 2024-04-22T23:57:05.101 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:57:05.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.210+0000 7fa11d50d700 1 -- 172.21.15.67:0/3014971010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa118100750 msgr2=0x7fa1181047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:05.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.210+0000 7fa11d50d700 1 --2- 172.21.15.67:0/3014971010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa118100750 0x7fa1181047a0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fa10c009910 tx=0x7fa10c009c20 comp rx=0 tx=0).stop 2024-04-22T23:57:05.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.210+0000 7fa11d50d700 1 -- 172.21.15.67:0/3014971010 shutdown_connections 2024-04-22T23:57:05.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.210+0000 7fa11d50d700 1 --2- 172.21.15.67:0/3014971010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa118100750 0x7fa1181047a0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:05.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.210+0000 7fa11d50d700 1 --2- 172.21.15.67:0/3014971010 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa1180ffda0 0x7fa118100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:05.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.210+0000 7fa11d50d700 1 -- 172.21.15.67:0/3014971010 >> 172.21.15.67:0/3014971010 conn(0x7fa1180fb650 msgr2=0x7fa1180fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:57:05.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.210+0000 7fa11d50d700 1 -- 172.21.15.67:0/3014971010 shutdown_connections 2024-04-22T23:57:05.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.210+0000 7fa11d50d700 1 -- 172.21.15.67:0/3014971010 wait complete. 2024-04-22T23:57:05.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.211+0000 7fa11d50d700 1 Processor -- start 2024-04-22T23:57:05.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.211+0000 7fa11d50d700 1 -- start start 2024-04-22T23:57:05.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.211+0000 7fa11d50d700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa1180ffda0 0x7fa118101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:05.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.211+0000 7fa11d50d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa118100750 0x7fa118101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:05.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.211+0000 7fa11d50d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1181022c0 con 0x7fa118100750 2024-04-22T23:57:05.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.211+0000 7fa11d50d700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa118102400 con 0x7fa1180ffda0 2024-04-22T23:57:05.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.211+0000 7fa1167fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa118100750 0x7fa118101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:05.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.211+0000 7fa1167fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa118100750 0x7fa118101d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41230/0 (socket says 172.21.15.67:41230) 2024-04-22T23:57:05.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.211+0000 7fa1167fc700 1 -- 172.21.15.67:0/2331426186 learned_addr learned my addr 172.21.15.67:0/2331426186 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:57:05.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.212+0000 7fa116ffd700 1 --2- 172.21.15.67:0/2331426186 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa1180ffda0 0x7fa118101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:05.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.212+0000 7fa1167fc700 1 -- 172.21.15.67:0/2331426186 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa1180ffda0 msgr2=0x7fa118101840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:05.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.212+0000 7fa1167fc700 1 --2- 172.21.15.67:0/2331426186 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa1180ffda0 0x7fa118101840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:05.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.212+0000 7fa1167fc700 1 -- 172.21.15.67:0/2331426186 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa10c0095f0 con 0x7fa118100750 2024-04-22T23:57:05.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.212+0000 7fa1167fc700 1 --2- 172.21.15.67:0/2331426186 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa118100750 0x7fa118101d80 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fa10c0092e0 tx=0x7fa10c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:57:05.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.212+0000 7fa0fffff700 1 -- 172.21.15.67:0/2331426186 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa10c01d070 con 0x7fa118100750 2024-04-22T23:57:05.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.212+0000 7fa0fffff700 1 -- 172.21.15.67:0/2331426186 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa10c005dc0 con 0x7fa118100750 2024-04-22T23:57:05.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.212+0000 7fa11d50d700 1 -- 172.21.15.67:0/2331426186 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa118102680 con 0x7fa118100750 2024-04-22T23:57:05.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.212+0000 7fa0fffff700 1 -- 172.21.15.67:0/2331426186 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa10c017400 con 0x7fa118100750 2024-04-22T23:57:05.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.212+0000 7fa11d50d700 1 -- 172.21.15.67:0/2331426186 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1181a2da0 con 0x7fa118100750 2024-04-22T23:57:05.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.213+0000 7fa11d50d700 1 -- 172.21.15.67:0/2331426186 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1181091b0 con 0x7fa118100750 2024-04-22T23:57:05.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.214+0000 7fa0fffff700 1 -- 172.21.15.67:0/2331426186 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa10c00f6f0 con 0x7fa118100750 2024-04-22T23:57:05.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.214+0000 7fa0fffff700 1 --2- 172.21.15.67:0/2331426186 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa10006ecc0 0x7fa100071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:05.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.214+0000 7fa0fffff700 1 -- 172.21.15.67:0/2331426186 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fa10c0903c0 con 0x7fa118100750 2024-04-22T23:57:05.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.214+0000 7fa116ffd700 1 --2- 172.21.15.67:0/2331426186 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa10006ecc0 0x7fa100071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:05.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.215+0000 7fa116ffd700 1 --2- 172.21.15.67:0/2331426186 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa10006ecc0 0x7fa100071180 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa108005950 tx=0x7fa10800b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:57:05.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.219+0000 7fa0fffff700 1 -- 172.21.15.67:0/2331426186 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fa10c059000 con 0x7fa118100750 2024-04-22T23:57:05.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.455+0000 7fa11d50d700 1 -- 172.21.15.67:0/2331426186 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa11802cce0 con 0x7fa10006ecc0 2024-04-22T23:57:05.446 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:05 smithi067 ceph-mon[112438]: pgmap v923: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:05.446 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:05 smithi067 ceph-mon[112438]: from='client.35664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:57:05.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.457+0000 7fa0fffff700 1 -- 172.21.15.67:0/2331426186 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa11802cce0 con 0x7fa10006ecc0 2024-04-22T23:57:05.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.459+0000 7fa11d50d700 1 -- 172.21.15.67:0/2331426186 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa10006ecc0 msgr2=0x7fa100071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:05.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.459+0000 7fa11d50d700 1 --2- 172.21.15.67:0/2331426186 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa10006ecc0 0x7fa100071180 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa108005950 tx=0x7fa10800b410 comp rx=0 tx=0).stop 2024-04-22T23:57:05.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.459+0000 7fa11d50d700 1 -- 172.21.15.67:0/2331426186 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa118100750 msgr2=0x7fa118101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:05.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.459+0000 7fa11d50d700 1 --2- 172.21.15.67:0/2331426186 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa118100750 0x7fa118101d80 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fa10c0092e0 tx=0x7fa10c00bd70 comp rx=0 tx=0).stop 2024-04-22T23:57:05.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.459+0000 7fa11d50d700 1 -- 172.21.15.67:0/2331426186 shutdown_connections 2024-04-22T23:57:05.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.459+0000 7fa11d50d700 1 --2- 172.21.15.67:0/2331426186 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa10006ecc0 0x7fa100071180 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:05.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.459+0000 7fa11d50d700 1 --2- 172.21.15.67:0/2331426186 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa1180ffda0 0x7fa118101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:05.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.459+0000 7fa11d50d700 1 --2- 172.21.15.67:0/2331426186 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa118100750 0x7fa118101d80 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:05.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.459+0000 7fa11d50d700 1 -- 172.21.15.67:0/2331426186 >> 172.21.15.67:0/2331426186 conn(0x7fa1180fb650 msgr2=0x7fa118069150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:57:05.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.459+0000 7fa11d50d700 1 -- 172.21.15.67:0/2331426186 shutdown_connections 2024-04-22T23:57:05.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.459+0000 7fa11d50d700 1 -- 172.21.15.67:0/2331426186 wait complete. 2024-04-22T23:57:05.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:05 smithi134 ceph-mon[85630]: pgmap v923: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:05.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:05 smithi134 ceph-mon[85630]: from='client.35664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:57:05.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.565+0000 7f911e982700 1 -- 172.21.15.67:0/542690934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91180ffc20 msgr2=0x7f9118100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:05.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.565+0000 7f911e982700 1 --2- 172.21.15.67:0/542690934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91180ffc20 0x7f9118100080 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f9100009910 tx=0x7f9100009c20 comp rx=0 tx=0).stop 2024-04-22T23:57:05.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.565+0000 7f911e982700 1 -- 172.21.15.67:0/542690934 shutdown_connections 2024-04-22T23:57:05.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.565+0000 7f911e982700 1 --2- 172.21.15.67:0/542690934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91180ffc20 0x7f9118100080 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:05.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.565+0000 7f911e982700 1 --2- 172.21.15.67:0/542690934 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9118105bd0 0x7f9118105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:05.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.565+0000 7f911e982700 1 -- 172.21.15.67:0/542690934 >> 172.21.15.67:0/542690934 conn(0x7f9118075670 msgr2=0x7f9118075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:57:05.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.565+0000 7f911e982700 1 -- 172.21.15.67:0/542690934 shutdown_connections 2024-04-22T23:57:05.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.565+0000 7f911e982700 1 -- 172.21.15.67:0/542690934 wait complete. 2024-04-22T23:57:05.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.566+0000 7f911e982700 1 Processor -- start 2024-04-22T23:57:05.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.566+0000 7f911e982700 1 -- start start 2024-04-22T23:57:05.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.566+0000 7f911e982700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91180ffc20 0x7f911819c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:05.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.566+0000 7f911e982700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9118105bd0 0x7f911819cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:05.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.566+0000 7f911e982700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f911819d200 con 0x7f91180ffc20 2024-04-22T23:57:05.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.566+0000 7f911e982700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91181966b0 con 0x7f9118105bd0 2024-04-22T23:57:05.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.567+0000 7f9117fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91180ffc20 0x7f911819c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:05.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.567+0000 7f9117fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91180ffc20 0x7f911819c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41244/0 (socket says 172.21.15.67:41244) 2024-04-22T23:57:05.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.567+0000 7f9117fff700 1 -- 172.21.15.67:0/3181154051 learned_addr learned my addr 172.21.15.67:0/3181154051 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:57:05.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.567+0000 7f91177fe700 1 --2- 172.21.15.67:0/3181154051 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9118105bd0 0x7f911819cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:05.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.567+0000 7f9117fff700 1 -- 172.21.15.67:0/3181154051 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9118105bd0 msgr2=0x7f911819cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:05.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.567+0000 7f9117fff700 1 --2- 172.21.15.67:0/3181154051 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9118105bd0 0x7f911819cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:05.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.567+0000 7f9117fff700 1 -- 172.21.15.67:0/3181154051 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91000095f0 con 0x7f91180ffc20 2024-04-22T23:57:05.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.567+0000 7f9117fff700 1 --2- 172.21.15.67:0/3181154051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91180ffc20 0x7f911819c630 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f910800d8d0 tx=0x7f910800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:57:05.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.567+0000 7f91157fa700 1 -- 172.21.15.67:0/3181154051 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9108009940 con 0x7f91180ffc20 2024-04-22T23:57:05.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.567+0000 7f91157fa700 1 -- 172.21.15.67:0/3181154051 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9108010460 con 0x7f91180ffc20 2024-04-22T23:57:05.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.567+0000 7f911e982700 1 -- 172.21.15.67:0/3181154051 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9118196990 con 0x7f91180ffc20 2024-04-22T23:57:05.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.567+0000 7f91157fa700 1 -- 172.21.15.67:0/3181154051 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f910800f5f0 con 0x7f91180ffc20 2024-04-22T23:57:05.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.567+0000 7f911e982700 1 -- 172.21.15.67:0/3181154051 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9118196ee0 con 0x7f91180ffc20 2024-04-22T23:57:05.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.568+0000 7f911e982700 1 -- 172.21.15.67:0/3181154051 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f911804f2e0 con 0x7f91180ffc20 2024-04-22T23:57:05.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.569+0000 7f91157fa700 1 -- 172.21.15.67:0/3181154051 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f91080105d0 con 0x7f91180ffc20 2024-04-22T23:57:05.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.569+0000 7f91157fa700 1 --2- 172.21.15.67:0/3181154051 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f910406ecc0 0x7f9104071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:05.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.569+0000 7f91157fa700 1 -- 172.21.15.67:0/3181154051 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f910808eff0 con 0x7f91180ffc20 2024-04-22T23:57:05.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.569+0000 7f91177fe700 1 --2- 172.21.15.67:0/3181154051 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f910406ecc0 0x7f9104071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:05.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.570+0000 7f91177fe700 1 --2- 172.21.15.67:0/3181154051 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f910406ecc0 0x7f9104071180 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f910000b5c0 tx=0x7f9100005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:57:05.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.574+0000 7f91157fa700 1 -- 172.21.15.67:0/3181154051 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f910805b760 con 0x7f91180ffc20 2024-04-22T23:57:05.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.793+0000 7f911e982700 1 -- 172.21.15.67:0/3181154051 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f91181971a0 con 0x7f910406ecc0 2024-04-22T23:57:05.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.804+0000 7f91157fa700 1 -- 172.21.15.67:0/3181154051 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f91181971a0 con 0x7f910406ecc0 2024-04-22T23:57:05.795 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:57:05.795 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (30m) 8m ago 42m 24.4M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:57:05.796 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (42m) 8m ago 42m 10.0M - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:57:05.796 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (41m) 8m ago 41m 10.0M - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:57:05.796 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (28m) 8m ago 42m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:57:05.796 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (28m) 8m ago 41m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:57:05.796 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (30m) 8m ago 41m 81.2M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:57:05.796 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (24m) 8m ago 36m 19.4M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:57:05.796 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (24m) 8m ago 36m 20.3M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:57:05.796 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (23m) 8m ago 36m 20.1M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:57:05.796 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (8m) 8m ago 36m 14.0M - 19.0.0-3252-g43be0201 a0434699c1d7 46d3d7e86fee 2024-04-22T23:57:05.796 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (32m) 8m ago 43m 514M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:57:05.796 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (31m) 8m ago 41m 459M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:57:05.796 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (29m) 8m ago 43m 74.2M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:57:05.796 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (28m) 8m ago 41m 54.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:57:05.796 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (31m) 8m ago 42m 15.2M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:57:05.797 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (31m) 8m ago 41m 16.0M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:57:05.797 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (27m) 8m ago 40m 102M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:57:05.797 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (27m) 8m ago 39m 128M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:57:05.797 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (26m) 8m ago 39m 83.8M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:57:05.797 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (25m) 8m ago 39m 101M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:57:05.797 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (25m) 8m ago 38m 86.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:57:05.797 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (24m) 8m ago 38m 81.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:57:05.797 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (31m) 8m ago 41m 78.7M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:57:05.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.806+0000 7f911e982700 1 -- 172.21.15.67:0/3181154051 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f910406ecc0 msgr2=0x7f9104071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:05.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.806+0000 7f911e982700 1 --2- 172.21.15.67:0/3181154051 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f910406ecc0 0x7f9104071180 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f910000b5c0 tx=0x7f9100005960 comp rx=0 tx=0).stop 2024-04-22T23:57:05.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.806+0000 7f911e982700 1 -- 172.21.15.67:0/3181154051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91180ffc20 msgr2=0x7f911819c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:05.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.806+0000 7f911e982700 1 --2- 172.21.15.67:0/3181154051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91180ffc20 0x7f911819c630 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f910800d8d0 tx=0x7f910800dc90 comp rx=0 tx=0).stop 2024-04-22T23:57:05.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.807+0000 7f911e982700 1 -- 172.21.15.67:0/3181154051 shutdown_connections 2024-04-22T23:57:05.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.807+0000 7f911e982700 1 --2- 172.21.15.67:0/3181154051 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f910406ecc0 0x7f9104071180 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:05.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.807+0000 7f911e982700 1 --2- 172.21.15.67:0/3181154051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91180ffc20 0x7f911819c630 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:05.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.807+0000 7f911e982700 1 --2- 172.21.15.67:0/3181154051 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9118105bd0 0x7f911819cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:05.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.807+0000 7f911e982700 1 -- 172.21.15.67:0/3181154051 >> 172.21.15.67:0/3181154051 conn(0x7f9118075670 msgr2=0x7f9118068680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:57:05.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.807+0000 7f911e982700 1 -- 172.21.15.67:0/3181154051 shutdown_connections 2024-04-22T23:57:05.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.807+0000 7f911e982700 1 -- 172.21.15.67:0/3181154051 wait complete. 2024-04-22T23:57:05.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.912+0000 7f411df20700 1 -- 172.21.15.67:0/527401396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4118101850 msgr2=0x7f41181058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:05.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.912+0000 7f411df20700 1 --2- 172.21.15.67:0/527401396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4118101850 0x7f41181058a0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f410c009910 tx=0x7f410c009c20 comp rx=0 tx=0).stop 2024-04-22T23:57:05.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.912+0000 7f411df20700 1 -- 172.21.15.67:0/527401396 shutdown_connections 2024-04-22T23:57:05.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.912+0000 7f411df20700 1 --2- 172.21.15.67:0/527401396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4118101850 0x7f41181058a0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:05.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.912+0000 7f411df20700 1 --2- 172.21.15.67:0/527401396 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4118100ea0 0x7f4118101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:05.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.912+0000 7f411df20700 1 -- 172.21.15.67:0/527401396 >> 172.21.15.67:0/527401396 conn(0x7f41180fc750 msgr2=0x7f41180feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:57:05.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.912+0000 7f411df20700 1 -- 172.21.15.67:0/527401396 shutdown_connections 2024-04-22T23:57:05.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.912+0000 7f411df20700 1 -- 172.21.15.67:0/527401396 wait complete. 2024-04-22T23:57:05.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.913+0000 7f411df20700 1 Processor -- start 2024-04-22T23:57:05.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.913+0000 7f411df20700 1 -- start start 2024-04-22T23:57:05.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.913+0000 7f411df20700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4118100ea0 0x7f411819c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:05.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.913+0000 7f411df20700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4118101850 0x7f411819cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:05.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.913+0000 7f411df20700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f411819d2d0 con 0x7f4118100ea0 2024-04-22T23:57:05.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.913+0000 7f411df20700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4118196780 con 0x7f4118101850 2024-04-22T23:57:05.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.913+0000 7f41177fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4118100ea0 0x7f411819c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:05.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.913+0000 7f41177fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4118100ea0 0x7f411819c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41254/0 (socket says 172.21.15.67:41254) 2024-04-22T23:57:05.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.913+0000 7f41177fe700 1 -- 172.21.15.67:0/2088779391 learned_addr learned my addr 172.21.15.67:0/2088779391 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:57:05.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.914+0000 7f4116ffd700 1 --2- 172.21.15.67:0/2088779391 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4118101850 0x7f411819cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:05.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.914+0000 7f41177fe700 1 -- 172.21.15.67:0/2088779391 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4118101850 msgr2=0x7f411819cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:05.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.914+0000 7f41177fe700 1 --2- 172.21.15.67:0/2088779391 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4118101850 0x7f411819cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:05.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.914+0000 7f41177fe700 1 -- 172.21.15.67:0/2088779391 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f410c0095f0 con 0x7f4118100ea0 2024-04-22T23:57:05.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.914+0000 7f41177fe700 1 --2- 172.21.15.67:0/2088779391 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4118100ea0 0x7f411819c700 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f410800d8d0 tx=0x7f410800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:57:05.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.914+0000 7f4114ff9700 1 -- 172.21.15.67:0/2088779391 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4108009940 con 0x7f4118100ea0 2024-04-22T23:57:05.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.914+0000 7f4114ff9700 1 -- 172.21.15.67:0/2088779391 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4108010460 con 0x7f4118100ea0 2024-04-22T23:57:05.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.914+0000 7f411df20700 1 -- 172.21.15.67:0/2088779391 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4118196a60 con 0x7f4118100ea0 2024-04-22T23:57:05.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.914+0000 7f4114ff9700 1 -- 172.21.15.67:0/2088779391 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f410800f5f0 con 0x7f4118100ea0 2024-04-22T23:57:05.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.914+0000 7f411df20700 1 -- 172.21.15.67:0/2088779391 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4118196fb0 con 0x7f4118100ea0 2024-04-22T23:57:05.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.915+0000 7f411df20700 1 -- 172.21.15.67:0/2088779391 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41181092a0 con 0x7f4118100ea0 2024-04-22T23:57:05.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.916+0000 7f4114ff9700 1 -- 172.21.15.67:0/2088779391 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f41080105d0 con 0x7f4118100ea0 2024-04-22T23:57:05.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.916+0000 7f4114ff9700 1 --2- 172.21.15.67:0/2088779391 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f410006ecc0 0x7f4100071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:05.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.916+0000 7f4114ff9700 1 -- 172.21.15.67:0/2088779391 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f410808eef0 con 0x7f4118100ea0 2024-04-22T23:57:05.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.916+0000 7f4116ffd700 1 --2- 172.21.15.67:0/2088779391 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f410006ecc0 0x7f4100071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:05.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.917+0000 7f4116ffd700 1 --2- 172.21.15.67:0/2088779391 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f410006ecc0 0x7f4100071180 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f4118197bf0 tx=0x7f410c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:57:05.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:05.921+0000 7f4114ff9700 1 -- 172.21.15.67:0/2088779391 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f410805b6e0 con 0x7f4118100ea0 2024-04-22T23:57:06.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.217+0000 7f411df20700 1 -- 172.21.15.67:0/2088779391 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f4118197d50 con 0x7f4118100ea0 2024-04-22T23:57:06.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.217+0000 7f4114ff9700 1 -- 172.21.15.67:0/2088779391 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f410805b270 con 0x7f4118100ea0 2024-04-22T23:57:06.208 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:57:06.208 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:57:06.209 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:57:06.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:57:06.209 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:57:06.209 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:57:06.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:57:06.209 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:57:06.209 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:57:06.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:57:06.209 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:57:06.209 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 4 2024-04-22T23:57:06.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:57:06.209 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:57:06.210 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 14 2024-04-22T23:57:06.210 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:57:06.210 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:57:06.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.220+0000 7f411df20700 1 -- 172.21.15.67:0/2088779391 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f410006ecc0 msgr2=0x7f4100071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:06.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.220+0000 7f411df20700 1 --2- 172.21.15.67:0/2088779391 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f410006ecc0 0x7f4100071180 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f4118197bf0 tx=0x7f410c01a040 comp rx=0 tx=0).stop 2024-04-22T23:57:06.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.220+0000 7f411df20700 1 -- 172.21.15.67:0/2088779391 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4118100ea0 msgr2=0x7f411819c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:06.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.220+0000 7f411df20700 1 --2- 172.21.15.67:0/2088779391 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4118100ea0 0x7f411819c700 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f410800d8d0 tx=0x7f410800dc90 comp rx=0 tx=0).stop 2024-04-22T23:57:06.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.220+0000 7f411df20700 1 -- 172.21.15.67:0/2088779391 shutdown_connections 2024-04-22T23:57:06.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.220+0000 7f411df20700 1 --2- 172.21.15.67:0/2088779391 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f410006ecc0 0x7f4100071180 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:06.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.220+0000 7f411df20700 1 --2- 172.21.15.67:0/2088779391 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4118100ea0 0x7f411819c700 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:06.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.220+0000 7f411df20700 1 --2- 172.21.15.67:0/2088779391 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4118101850 0x7f411819cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:06.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.220+0000 7f411df20700 1 -- 172.21.15.67:0/2088779391 >> 172.21.15.67:0/2088779391 conn(0x7f41180fc750 msgr2=0x7f41180fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:57:06.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.220+0000 7f411df20700 1 -- 172.21.15.67:0/2088779391 shutdown_connections 2024-04-22T23:57:06.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.220+0000 7f411df20700 1 -- 172.21.15.67:0/2088779391 wait complete. 2024-04-22T23:57:06.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.327+0000 7f9ce2c48700 1 -- 172.21.15.67:0/717591732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cdc0ffc20 msgr2=0x7f9cdc100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:06.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.327+0000 7f9ce2c48700 1 --2- 172.21.15.67:0/717591732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cdc0ffc20 0x7f9cdc100080 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc00b0a0 tx=0x7f9ccc00b3b0 comp rx=0 tx=0).stop 2024-04-22T23:57:06.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.327+0000 7f9ce2c48700 1 -- 172.21.15.67:0/717591732 shutdown_connections 2024-04-22T23:57:06.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.327+0000 7f9ce2c48700 1 --2- 172.21.15.67:0/717591732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cdc0ffc20 0x7f9cdc100080 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:06.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.327+0000 7f9ce2c48700 1 --2- 172.21.15.67:0/717591732 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9cdc105bd0 0x7f9cdc105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:06.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.327+0000 7f9ce2c48700 1 -- 172.21.15.67:0/717591732 >> 172.21.15.67:0/717591732 conn(0x7f9cdc075670 msgr2=0x7f9cdc075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:57:06.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.327+0000 7f9ce2c48700 1 -- 172.21.15.67:0/717591732 shutdown_connections 2024-04-22T23:57:06.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.327+0000 7f9ce2c48700 1 -- 172.21.15.67:0/717591732 wait complete. 2024-04-22T23:57:06.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.328+0000 7f9ce2c48700 1 Processor -- start 2024-04-22T23:57:06.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.328+0000 7f9ce2c48700 1 -- start start 2024-04-22T23:57:06.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.328+0000 7f9ce2c48700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cdc0ffc20 0x7f9cdc19c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:06.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.328+0000 7f9ce2c48700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9cdc105bd0 0x7f9cdc19cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:06.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.328+0000 7f9ce2c48700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc19d1b0 con 0x7f9cdc0ffc20 2024-04-22T23:57:06.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.328+0000 7f9ce2c48700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc1966f0 con 0x7f9cdc105bd0 2024-04-22T23:57:06.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.328+0000 7f9ce09e4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cdc0ffc20 0x7f9cdc19c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:06.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.328+0000 7f9ce09e4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cdc0ffc20 0x7f9cdc19c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41274/0 (socket says 172.21.15.67:41274) 2024-04-22T23:57:06.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.328+0000 7f9ce09e4700 1 -- 172.21.15.67:0/2901553834 learned_addr learned my addr 172.21.15.67:0/2901553834 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:57:06.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.329+0000 7f9cdbfff700 1 --2- 172.21.15.67:0/2901553834 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9cdc105bd0 0x7f9cdc19cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:06.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.329+0000 7f9ce09e4700 1 -- 172.21.15.67:0/2901553834 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9cdc105bd0 msgr2=0x7f9cdc19cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:06.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.329+0000 7f9ce09e4700 1 --2- 172.21.15.67:0/2901553834 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9cdc105bd0 0x7f9cdc19cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:06.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.329+0000 7f9ce09e4700 1 -- 172.21.15.67:0/2901553834 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ccc009d00 con 0x7f9cdc0ffc20 2024-04-22T23:57:06.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.329+0000 7f9ce09e4700 1 --2- 172.21.15.67:0/2901553834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cdc0ffc20 0x7f9cdc19c670 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f9cc400c930 tx=0x7f9cc400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:57:06.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.329+0000 7f9cd9ffb700 1 -- 172.21.15.67:0/2901553834 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9cc4007ad0 con 0x7f9cdc0ffc20 2024-04-22T23:57:06.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.329+0000 7f9cd9ffb700 1 -- 172.21.15.67:0/2901553834 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9cc400ce80 con 0x7f9cdc0ffc20 2024-04-22T23:57:06.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.329+0000 7f9ce2c48700 1 -- 172.21.15.67:0/2901553834 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cdc1969d0 con 0x7f9cdc0ffc20 2024-04-22T23:57:06.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.329+0000 7f9cd9ffb700 1 -- 172.21.15.67:0/2901553834 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9cc4018630 con 0x7f9cdc0ffc20 2024-04-22T23:57:06.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.329+0000 7f9ce2c48700 1 -- 172.21.15.67:0/2901553834 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cdc196ef0 con 0x7f9cdc0ffc20 2024-04-22T23:57:06.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.330+0000 7f9ce2c48700 1 -- 172.21.15.67:0/2901553834 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cdc04f2e0 con 0x7f9cdc0ffc20 2024-04-22T23:57:06.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.331+0000 7f9cd9ffb700 1 -- 172.21.15.67:0/2901553834 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9cc4007c30 con 0x7f9cdc0ffc20 2024-04-22T23:57:06.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.331+0000 7f9cd9ffb700 1 --2- 172.21.15.67:0/2901553834 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9cc806ecc0 0x7f9cc8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:06.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.331+0000 7f9cd9ffb700 1 -- 172.21.15.67:0/2901553834 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f9cc408f010 con 0x7f9cdc0ffc20 2024-04-22T23:57:06.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.331+0000 7f9cdbfff700 1 --2- 172.21.15.67:0/2901553834 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9cc806ecc0 0x7f9cc8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:06.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.332+0000 7f9cdbfff700 1 --2- 172.21.15.67:0/2901553834 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9cc806ecc0 0x7f9cc8071180 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc000f80 tx=0x7f9ccc0097e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:57:06.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.336+0000 7f9cd9ffb700 1 -- 172.21.15.67:0/2901553834 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f9cc405b780 con 0x7f9cdc0ffc20 2024-04-22T23:57:06.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:06 smithi134 ceph-mon[85630]: from='client.35668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:57:06.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:06 smithi134 ceph-mon[85630]: from='client.35672 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:57:06.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:06 smithi067 ceph-mon[112438]: from='client.35668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:57:06.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:06 smithi067 ceph-mon[112438]: from='client.35672 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:57:06.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.593+0000 7f9ce2c48700 1 -- 172.21.15.67:0/2901553834 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f9cdc197c00 con 0x7f9cdc0ffc20 2024-04-22T23:57:06.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.593+0000 7f9cd9ffb700 1 -- 172.21.15.67:0/2901553834 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1872 (secure 0 0 0) 0x7f9cc405b310 con 0x7f9cdc0ffc20 2024-04-22T23:57:06.584 INFO:teuthology.orchestra.run.smithi067.stdout:e33 2024-04-22T23:57:06.584 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:57:06.584 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:57:06.584 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:57:06.585 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:57:06.585 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:57:06.585 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:57:06.585 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:57:06.585 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:57:06.585 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:57:06.585 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:57:06.585 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:57:06.585 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:57:06.585 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:57:06.585 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:57:06.585 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:57:06.585 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:57:06.586 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:57:06.586 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:57:06.586 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:57:06.586 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:57:06.586 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:57:06.586 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:57:06.586 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:57:06.586 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:57:06.586 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:57:06.586 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:57:06.586 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:57:06.586 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:57:06.586 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:57:06.586 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:57:06.586 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:57:06.587 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:57:06.587 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:57:06.587 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:57:06.587 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:57:06.587 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:57:06.587 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:57:06.587 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:57:06.587 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:57:06.587 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:57:06.587 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44851} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1849648622,v1:172.21.15.134:6827/1849648622] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:57:06.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.595+0000 7f9ce2c48700 1 -- 172.21.15.67:0/2901553834 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9cc806ecc0 msgr2=0x7f9cc8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:06.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.595+0000 7f9ce2c48700 1 --2- 172.21.15.67:0/2901553834 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9cc806ecc0 0x7f9cc8071180 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc000f80 tx=0x7f9ccc0097e0 comp rx=0 tx=0).stop 2024-04-22T23:57:06.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.595+0000 7f9ce2c48700 1 -- 172.21.15.67:0/2901553834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cdc0ffc20 msgr2=0x7f9cdc19c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:06.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.595+0000 7f9ce2c48700 1 --2- 172.21.15.67:0/2901553834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cdc0ffc20 0x7f9cdc19c670 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f9cc400c930 tx=0x7f9cc400cc40 comp rx=0 tx=0).stop 2024-04-22T23:57:06.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.595+0000 7f9ce2c48700 1 -- 172.21.15.67:0/2901553834 shutdown_connections 2024-04-22T23:57:06.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.595+0000 7f9ce2c48700 1 --2- 172.21.15.67:0/2901553834 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9cc806ecc0 0x7f9cc8071180 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:06.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.595+0000 7f9ce2c48700 1 --2- 172.21.15.67:0/2901553834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cdc0ffc20 0x7f9cdc19c670 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:06.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.595+0000 7f9ce2c48700 1 --2- 172.21.15.67:0/2901553834 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9cdc105bd0 0x7f9cdc19cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:06.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.595+0000 7f9ce2c48700 1 -- 172.21.15.67:0/2901553834 >> 172.21.15.67:0/2901553834 conn(0x7f9cdc075670 msgr2=0x7f9cdc0686e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:57:06.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.596+0000 7f9ce2c48700 1 -- 172.21.15.67:0/2901553834 shutdown_connections 2024-04-22T23:57:06.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.596+0000 7f9ce2c48700 1 -- 172.21.15.67:0/2901553834 wait complete. 2024-04-22T23:57:06.589 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 33 2024-04-22T23:57:06.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.703+0000 7fc8c1999700 1 -- 172.21.15.67:0/3730017192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8bc102720 msgr2=0x7fc8bc10ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:06.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.703+0000 7fc8c1999700 1 --2- 172.21.15.67:0/3730017192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8bc102720 0x7fc8bc10ac10 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fc8a4009910 tx=0x7fc8a4009c20 comp rx=0 tx=0).stop 2024-04-22T23:57:06.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.703+0000 7fc8c1999700 1 -- 172.21.15.67:0/3730017192 shutdown_connections 2024-04-22T23:57:06.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.703+0000 7fc8c1999700 1 --2- 172.21.15.67:0/3730017192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8bc102720 0x7fc8bc10ac10 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:06.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.703+0000 7fc8c1999700 1 --2- 172.21.15.67:0/3730017192 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc8bc101e00 0x7fc8bc1021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:06.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.703+0000 7fc8c1999700 1 -- 172.21.15.67:0/3730017192 >> 172.21.15.67:0/3730017192 conn(0x7fc8bc0fb650 msgr2=0x7fc8bc0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:57:06.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.704+0000 7fc8c1999700 1 -- 172.21.15.67:0/3730017192 shutdown_connections 2024-04-22T23:57:06.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.704+0000 7fc8c1999700 1 -- 172.21.15.67:0/3730017192 wait complete. 2024-04-22T23:57:06.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.704+0000 7fc8c1999700 1 Processor -- start 2024-04-22T23:57:06.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.704+0000 7fc8c1999700 1 -- start start 2024-04-22T23:57:06.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.704+0000 7fc8c1999700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8bc101e00 0x7fc8bc19c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:06.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.704+0000 7fc8c1999700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc8bc102720 0x7fc8bc19cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:06.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.704+0000 7fc8c1999700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8bc19d260 con 0x7fc8bc101e00 2024-04-22T23:57:06.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.704+0000 7fc8c1999700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8bc196710 con 0x7fc8bc102720 2024-04-22T23:57:06.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.704+0000 7fc8baffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8bc101e00 0x7fc8bc19c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:06.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.705+0000 7fc8baffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8bc101e00 0x7fc8bc19c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41278/0 (socket says 172.21.15.67:41278) 2024-04-22T23:57:06.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.705+0000 7fc8baffd700 1 -- 172.21.15.67:0/1636623258 learned_addr learned my addr 172.21.15.67:0/1636623258 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:57:06.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.705+0000 7fc8b3fff700 1 --2- 172.21.15.67:0/1636623258 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc8bc102720 0x7fc8bc19cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:06.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.705+0000 7fc8baffd700 1 -- 172.21.15.67:0/1636623258 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc8bc102720 msgr2=0x7fc8bc19cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:06.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.705+0000 7fc8baffd700 1 --2- 172.21.15.67:0/1636623258 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc8bc102720 0x7fc8bc19cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:06.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.705+0000 7fc8baffd700 1 -- 172.21.15.67:0/1636623258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a40095f0 con 0x7fc8bc101e00 2024-04-22T23:57:06.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.705+0000 7fc8baffd700 1 --2- 172.21.15.67:0/1636623258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8bc101e00 0x7fc8bc19c690 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac00d8d0 tx=0x7fc8ac00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:57:06.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.705+0000 7fc8c0997700 1 -- 172.21.15.67:0/1636623258 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8ac009940 con 0x7fc8bc101e00 2024-04-22T23:57:06.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.705+0000 7fc8c0997700 1 -- 172.21.15.67:0/1636623258 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc8ac010460 con 0x7fc8bc101e00 2024-04-22T23:57:06.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.705+0000 7fc8c1999700 1 -- 172.21.15.67:0/1636623258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8bc1969f0 con 0x7fc8bc101e00 2024-04-22T23:57:06.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.705+0000 7fc8c0997700 1 -- 172.21.15.67:0/1636623258 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8ac00f5f0 con 0x7fc8bc101e00 2024-04-22T23:57:06.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.705+0000 7fc8c1999700 1 -- 172.21.15.67:0/1636623258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8bc196f40 con 0x7fc8bc101e00 2024-04-22T23:57:06.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.706+0000 7fc8c1999700 1 -- 172.21.15.67:0/1636623258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8bc108310 con 0x7fc8bc101e00 2024-04-22T23:57:06.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.707+0000 7fc8c0997700 1 -- 172.21.15.67:0/1636623258 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc8ac0105d0 con 0x7fc8bc101e00 2024-04-22T23:57:06.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.707+0000 7fc8c0997700 1 --2- 172.21.15.67:0/1636623258 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc8a806ecc0 0x7fc8a8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:06.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.707+0000 7fc8c0997700 1 -- 172.21.15.67:0/1636623258 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fc8ac08eee0 con 0x7fc8bc101e00 2024-04-22T23:57:06.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.707+0000 7fc8b3fff700 1 --2- 172.21.15.67:0/1636623258 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc8a806ecc0 0x7fc8a8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:06.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.708+0000 7fc8b3fff700 1 --2- 172.21.15.67:0/1636623258 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc8a806ecc0 0x7fc8a8071180 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fc8a4006010 tx=0x7fc8a4005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:57:06.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.712+0000 7fc8c0997700 1 -- 172.21.15.67:0/1636623258 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc8ac05b6d0 con 0x7fc8bc101e00 2024-04-22T23:57:06.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.942+0000 7fc8c1999700 1 -- 172.21.15.67:0/1636623258 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc8bc1971b0 con 0x7fc8a806ecc0 2024-04-22T23:57:06.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.943+0000 7fc8c0997700 1 -- 172.21.15.67:0/1636623258 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc8bc1971b0 con 0x7fc8a806ecc0 2024-04-22T23:57:06.934 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:57:06.934 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:57:06.934 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:57:06.934 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:57:06.934 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:57:06.934 INFO:teuthology.orchestra.run.smithi067.stdout: "mds", 2024-04-22T23:57:06.934 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:57:06.934 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:57:06.934 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:57:06.934 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:57:06.935 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:57:06.935 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T23:57:06.935 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:57:06.935 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:57:06.935 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:57:06.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.945+0000 7fc8c1999700 1 -- 172.21.15.67:0/1636623258 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc8a806ecc0 msgr2=0x7fc8a8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:06.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.945+0000 7fc8c1999700 1 --2- 172.21.15.67:0/1636623258 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc8a806ecc0 0x7fc8a8071180 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fc8a4006010 tx=0x7fc8a4005960 comp rx=0 tx=0).stop 2024-04-22T23:57:06.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.945+0000 7fc8c1999700 1 -- 172.21.15.67:0/1636623258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8bc101e00 msgr2=0x7fc8bc19c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:06.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.945+0000 7fc8c1999700 1 --2- 172.21.15.67:0/1636623258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8bc101e00 0x7fc8bc19c690 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac00d8d0 tx=0x7fc8ac00dc90 comp rx=0 tx=0).stop 2024-04-22T23:57:06.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.945+0000 7fc8c1999700 1 -- 172.21.15.67:0/1636623258 shutdown_connections 2024-04-22T23:57:06.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.945+0000 7fc8c1999700 1 --2- 172.21.15.67:0/1636623258 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc8a806ecc0 0x7fc8a8071180 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:06.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.945+0000 7fc8c1999700 1 --2- 172.21.15.67:0/1636623258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8bc101e00 0x7fc8bc19c690 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:06.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.945+0000 7fc8c1999700 1 --2- 172.21.15.67:0/1636623258 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc8bc102720 0x7fc8bc19cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:06.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.945+0000 7fc8c1999700 1 -- 172.21.15.67:0/1636623258 >> 172.21.15.67:0/1636623258 conn(0x7fc8bc0fb650 msgr2=0x7fc8bc105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:57:06.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.946+0000 7fc8c1999700 1 -- 172.21.15.67:0/1636623258 shutdown_connections 2024-04-22T23:57:06.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:06.946+0000 7fc8c1999700 1 -- 172.21.15.67:0/1636623258 wait complete. 2024-04-22T23:57:07.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.052+0000 7f88878c6700 1 -- 172.21.15.67:0/2591696830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8880102720 msgr2=0x7f888010ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:07.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.052+0000 7f88878c6700 1 --2- 172.21.15.67:0/2591696830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8880102720 0x7f888010ac10 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f887c009910 tx=0x7f887c009c20 comp rx=0 tx=0).stop 2024-04-22T23:57:07.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.052+0000 7f88878c6700 1 -- 172.21.15.67:0/2591696830 shutdown_connections 2024-04-22T23:57:07.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.052+0000 7f88878c6700 1 --2- 172.21.15.67:0/2591696830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8880102720 0x7f888010ac10 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:07.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.053+0000 7f88878c6700 1 --2- 172.21.15.67:0/2591696830 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8880101e00 0x7f88801021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:07.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.053+0000 7f88878c6700 1 -- 172.21.15.67:0/2591696830 >> 172.21.15.67:0/2591696830 conn(0x7f88800fb650 msgr2=0x7f88800fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:57:07.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.053+0000 7f88878c6700 1 -- 172.21.15.67:0/2591696830 shutdown_connections 2024-04-22T23:57:07.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.053+0000 7f88878c6700 1 -- 172.21.15.67:0/2591696830 wait complete. 2024-04-22T23:57:07.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.053+0000 7f88878c6700 1 Processor -- start 2024-04-22T23:57:07.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.053+0000 7f88878c6700 1 -- start start 2024-04-22T23:57:07.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.053+0000 7f88878c6700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8880101e00 0x7f888019c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:07.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.053+0000 7f88878c6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8880102720 0x7f888019cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:07.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.054+0000 7f88878c6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888019d270 con 0x7f8880102720 2024-04-22T23:57:07.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.054+0000 7f88878c6700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8880196720 con 0x7f8880101e00 2024-04-22T23:57:07.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.054+0000 7f8884e61700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8880102720 0x7f888019cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:07.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.054+0000 7f8884e61700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8880102720 0x7f888019cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41290/0 (socket says 172.21.15.67:41290) 2024-04-22T23:57:07.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.054+0000 7f8884e61700 1 -- 172.21.15.67:0/4240762666 learned_addr learned my addr 172.21.15.67:0/4240762666 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:57:07.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.054+0000 7f8885662700 1 --2- 172.21.15.67:0/4240762666 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8880101e00 0x7f888019c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:07.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.054+0000 7f8884e61700 1 -- 172.21.15.67:0/4240762666 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8880101e00 msgr2=0x7f888019c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:07.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.054+0000 7f8884e61700 1 --2- 172.21.15.67:0/4240762666 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8880101e00 0x7f888019c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:07.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.054+0000 7f8884e61700 1 -- 172.21.15.67:0/4240762666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f887c0095f0 con 0x7f8880102720 2024-04-22T23:57:07.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.054+0000 7f8884e61700 1 --2- 172.21.15.67:0/4240762666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8880102720 0x7f888019cbe0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f887c00bd70 tx=0x7f887c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:57:07.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.054+0000 7f88767fc700 1 -- 172.21.15.67:0/4240762666 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f887c01d070 con 0x7f8880102720 2024-04-22T23:57:07.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.054+0000 7f88767fc700 1 -- 172.21.15.67:0/4240762666 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f887c005dc0 con 0x7f8880102720 2024-04-22T23:57:07.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.054+0000 7f88878c6700 1 -- 172.21.15.67:0/4240762666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88801969a0 con 0x7f8880102720 2024-04-22T23:57:07.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.054+0000 7f88767fc700 1 -- 172.21.15.67:0/4240762666 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f887c017400 con 0x7f8880102720 2024-04-22T23:57:07.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.054+0000 7f88878c6700 1 -- 172.21.15.67:0/4240762666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8880196e90 con 0x7f8880102720 2024-04-22T23:57:07.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.055+0000 7f88878c6700 1 -- 172.21.15.67:0/4240762666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8880108310 con 0x7f8880102720 2024-04-22T23:57:07.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.056+0000 7f88767fc700 1 -- 172.21.15.67:0/4240762666 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f887c00f7b0 con 0x7f8880102720 2024-04-22T23:57:07.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.056+0000 7f88767fc700 1 --2- 172.21.15.67:0/4240762666 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f886c06ecc0 0x7f886c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:07.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.056+0000 7f88767fc700 1 -- 172.21.15.67:0/4240762666 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f887c090410 con 0x7f8880102720 2024-04-22T23:57:07.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.056+0000 7f8885662700 1 --2- 172.21.15.67:0/4240762666 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f886c06ecc0 0x7f886c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:07.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.057+0000 7f8885662700 1 --2- 172.21.15.67:0/4240762666 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f886c06ecc0 0x7f886c071180 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f8870005950 tx=0x7f88700058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:57:07.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.061+0000 7f88767fc700 1 -- 172.21.15.67:0/4240762666 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f887c05cc00 con 0x7f8880102720 2024-04-22T23:57:07.284 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:07 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2088779391' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:57:07.284 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:07 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2901553834' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:57:07.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:07 smithi067 ceph-mon[112438]: from='client.35684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:57:07.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:07 smithi067 ceph-mon[112438]: pgmap v924: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:07.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.362+0000 7f88878c6700 1 -- 172.21.15.67:0/4240762666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8880197cd0 con 0x7f8880102720 2024-04-22T23:57:07.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.362+0000 7f88767fc700 1 -- 172.21.15.67:0/4240762666 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4505 (secure 0 0 0) 0x7f887c05c790 con 0x7f8880102720 2024-04-22T23:57:07.353 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:57:07.353 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T23:57:07.353 INFO:teuthology.orchestra.run.smithi067.stdout: host smithi134 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-22T23:57:07.353 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T23:57:07.353 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:57:07.353 INFO:teuthology.orchestra.run.smithi067.stdout:Traceback (most recent call last): 2024-04-22T23:57:07.353 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T23:57:07.353 INFO:teuthology.orchestra.run.smithi067.stdout: return _run_code(code, main_globals, None, 2024-04-22T23:57:07.354 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T23:57:07.354 INFO:teuthology.orchestra.run.smithi067.stdout: exec(code, run_globals) 2024-04-22T23:57:07.354 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5642, in 2024-04-22T23:57:07.354 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5630, in main 2024-04-22T23:57:07.354 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 408, in _infer_config 2024-04-22T23:57:07.354 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 323, in _infer_fsid 2024-04-22T23:57:07.354 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 436, in _infer_image 2024-04-22T23:57:07.354 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 310, in _validate_fsid 2024-04-22T23:57:07.354 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 3380, in command_ceph_volume 2024-04-22T23:57:07.354 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T23:57:07.354 INFO:teuthology.orchestra.run.smithi067.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:57:07.354 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:57:07.354 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:57:07.355 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:57:07.355 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:57:07.355 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:57:07.355 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:57:07.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.364+0000 7f88878c6700 1 -- 172.21.15.67:0/4240762666 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f886c06ecc0 msgr2=0x7f886c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:07.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.364+0000 7f88878c6700 1 --2- 172.21.15.67:0/4240762666 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f886c06ecc0 0x7f886c071180 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f8870005950 tx=0x7f88700058e0 comp rx=0 tx=0).stop 2024-04-22T23:57:07.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.364+0000 7f88878c6700 1 -- 172.21.15.67:0/4240762666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8880102720 msgr2=0x7f888019cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:07.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.364+0000 7f88878c6700 1 --2- 172.21.15.67:0/4240762666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8880102720 0x7f888019cbe0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f887c00bd70 tx=0x7f887c00be50 comp rx=0 tx=0).stop 2024-04-22T23:57:07.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.364+0000 7f88878c6700 1 -- 172.21.15.67:0/4240762666 shutdown_connections 2024-04-22T23:57:07.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.364+0000 7f88878c6700 1 --2- 172.21.15.67:0/4240762666 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f886c06ecc0 0x7f886c071180 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:07.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.364+0000 7f88878c6700 1 --2- 172.21.15.67:0/4240762666 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8880101e00 0x7f888019c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:07.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.364+0000 7f88878c6700 1 --2- 172.21.15.67:0/4240762666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8880102720 0x7f888019cbe0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:07.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.364+0000 7f88878c6700 1 -- 172.21.15.67:0/4240762666 >> 172.21.15.67:0/4240762666 conn(0x7f88800fb650 msgr2=0x7f8880105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:57:07.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.364+0000 7f88878c6700 1 -- 172.21.15.67:0/4240762666 shutdown_connections 2024-04-22T23:57:07.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:07.364+0000 7f88878c6700 1 -- 172.21.15.67:0/4240762666 wait complete. 2024-04-22T23:57:07.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:07 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2088779391' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:57:07.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:07 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2901553834' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:57:07.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:07 smithi134 ceph-mon[85630]: from='client.35684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:57:07.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:07 smithi134 ceph-mon[85630]: pgmap v924: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:08.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:08 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/4240762666' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:57:08.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:08 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/4240762666' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:57:09.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:09 smithi134 ceph-mon[85630]: pgmap v925: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:09.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:09 smithi067 ceph-mon[112438]: pgmap v925: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:11.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:57:11.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:57:12.468 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:12 smithi134 ceph-mon[85630]: pgmap v926: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:12.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:12 smithi067 ceph-mon[112438]: pgmap v926: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:14.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:14 smithi134 ceph-mon[85630]: pgmap v927: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:14.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:14 smithi067 ceph-mon[112438]: pgmap v927: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:16.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:16 smithi134 ceph-mon[85630]: pgmap v928: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:16.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:16 smithi067 ceph-mon[112438]: pgmap v928: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:17.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:17 smithi134 ceph-mon[85630]: pgmap v929: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:17.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:17 smithi067 ceph-mon[112438]: pgmap v929: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:19.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:19 smithi134 ceph-mon[85630]: pgmap v930: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:19.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:19 smithi067 ceph-mon[112438]: pgmap v930: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:21.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:21 smithi134 ceph-mon[85630]: pgmap v931: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:21.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:21 smithi067 ceph-mon[112438]: pgmap v931: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:23.984 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:23 smithi134 ceph-mon[85630]: pgmap v932: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:24.034 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:23 smithi067 ceph-mon[112438]: pgmap v932: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:25.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:25 smithi134 ceph-mon[85630]: pgmap v933: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:25.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:25 smithi067 ceph-mon[112438]: pgmap v933: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:26.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:57:26.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:57:27.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:27 smithi134 ceph-mon[85630]: pgmap v934: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:27.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:27 smithi067 ceph-mon[112438]: pgmap v934: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:29.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:29 smithi134 ceph-mon[85630]: pgmap v935: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:29.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:29 smithi067 ceph-mon[112438]: pgmap v935: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:31.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:31 smithi134 ceph-mon[85630]: pgmap v936: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:31.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:31 smithi067 ceph-mon[112438]: pgmap v936: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:33.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:33 smithi134 ceph-mon[85630]: pgmap v937: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:33.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:33 smithi067 ceph-mon[112438]: pgmap v937: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:35.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:35 smithi134 ceph-mon[85630]: pgmap v938: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:35.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:35 smithi067 ceph-mon[112438]: pgmap v938: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:37.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.484+0000 7f3a7a30a700 1 -- 172.21.15.67:0/1593965427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a74100750 msgr2=0x7f3a741047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:37.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.484+0000 7f3a7a30a700 1 --2- 172.21.15.67:0/1593965427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a74100750 0x7f3a741047a0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c009910 tx=0x7f3a5c009c20 comp rx=0 tx=0).stop 2024-04-22T23:57:37.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.484+0000 7f3a7a30a700 1 -- 172.21.15.67:0/1593965427 shutdown_connections 2024-04-22T23:57:37.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.484+0000 7f3a7a30a700 1 --2- 172.21.15.67:0/1593965427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a74100750 0x7f3a741047a0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:37.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.484+0000 7f3a7a30a700 1 --2- 172.21.15.67:0/1593965427 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a740ffda0 0x7f3a74100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:37.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.484+0000 7f3a7a30a700 1 -- 172.21.15.67:0/1593965427 >> 172.21.15.67:0/1593965427 conn(0x7f3a740fb650 msgr2=0x7f3a740fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:57:37.476 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:37 smithi067 ceph-mon[112438]: pgmap v939: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:37.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.484+0000 7f3a7a30a700 1 -- 172.21.15.67:0/1593965427 shutdown_connections 2024-04-22T23:57:37.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.484+0000 7f3a7a30a700 1 -- 172.21.15.67:0/1593965427 wait complete. 2024-04-22T23:57:37.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.485+0000 7f3a7a30a700 1 Processor -- start 2024-04-22T23:57:37.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.485+0000 7f3a7a30a700 1 -- start start 2024-04-22T23:57:37.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.485+0000 7f3a7a30a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a740ffda0 0x7f3a7419e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:37.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.485+0000 7f3a7a30a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a74100750 0x7f3a7419edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:37.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.485+0000 7f3a7a30a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7419f450 con 0x7f3a74100750 2024-04-22T23:57:37.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.485+0000 7f3a7a30a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a74198900 con 0x7f3a740ffda0 2024-04-22T23:57:37.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.485+0000 7f3a737fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a74100750 0x7f3a7419edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:37.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.485+0000 7f3a737fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a74100750 0x7f3a7419edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52826/0 (socket says 172.21.15.67:52826) 2024-04-22T23:57:37.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.485+0000 7f3a737fe700 1 -- 172.21.15.67:0/1603829828 learned_addr learned my addr 172.21.15.67:0/1603829828 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:57:37.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.486+0000 7f3a73fff700 1 --2- 172.21.15.67:0/1603829828 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a740ffda0 0x7f3a7419e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:37.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.486+0000 7f3a737fe700 1 -- 172.21.15.67:0/1603829828 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a740ffda0 msgr2=0x7f3a7419e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:37.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.486+0000 7f3a737fe700 1 --2- 172.21.15.67:0/1603829828 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a740ffda0 0x7f3a7419e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:37.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.486+0000 7f3a737fe700 1 -- 172.21.15.67:0/1603829828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a5c0095f0 con 0x7f3a74100750 2024-04-22T23:57:37.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.486+0000 7f3a737fe700 1 --2- 172.21.15.67:0/1603829828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a74100750 0x7f3a7419edc0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c0092e0 tx=0x7f3a5c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:57:37.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.486+0000 7f3a717fa700 1 -- 172.21.15.67:0/1603829828 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a5c01d070 con 0x7f3a74100750 2024-04-22T23:57:37.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.486+0000 7f3a717fa700 1 -- 172.21.15.67:0/1603829828 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3a5c005dc0 con 0x7f3a74100750 2024-04-22T23:57:37.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.486+0000 7f3a717fa700 1 -- 172.21.15.67:0/1603829828 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a5c017400 con 0x7f3a74100750 2024-04-22T23:57:37.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.486+0000 7f3a7a30a700 1 -- 172.21.15.67:0/1603829828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a74198b80 con 0x7f3a74100750 2024-04-22T23:57:37.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.486+0000 7f3a7a30a700 1 -- 172.21.15.67:0/1603829828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a74199070 con 0x7f3a74100750 2024-04-22T23:57:37.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.487+0000 7f3a7a30a700 1 -- 172.21.15.67:0/1603829828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a7404f2e0 con 0x7f3a74100750 2024-04-22T23:57:37.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.488+0000 7f3a717fa700 1 -- 172.21.15.67:0/1603829828 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3a5c00f460 con 0x7f3a74100750 2024-04-22T23:57:37.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.488+0000 7f3a717fa700 1 --2- 172.21.15.67:0/1603829828 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3a6006ecc0 0x7f3a60071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:37.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.488+0000 7f3a717fa700 1 -- 172.21.15.67:0/1603829828 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f3a5c0904e0 con 0x7f3a74100750 2024-04-22T23:57:37.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.488+0000 7f3a73fff700 1 --2- 172.21.15.67:0/1603829828 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3a6006ecc0 0x7f3a60071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:37.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.489+0000 7f3a73fff700 1 --2- 172.21.15.67:0/1603829828 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3a6006ecc0 0x7f3a60071180 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f3a64005950 tx=0x7f3a640058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:57:37.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:37 smithi134 ceph-mon[85630]: pgmap v939: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:37.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.493+0000 7f3a717fa700 1 -- 172.21.15.67:0/1603829828 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f3a5c05cc50 con 0x7f3a74100750 2024-04-22T23:57:37.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.725+0000 7f3a7a30a700 1 -- 172.21.15.67:0/1603829828 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a74199330 con 0x7f3a6006ecc0 2024-04-22T23:57:37.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.726+0000 7f3a717fa700 1 -- 172.21.15.67:0/1603829828 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3a74199330 con 0x7f3a6006ecc0 2024-04-22T23:57:37.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.728+0000 7f3a7a30a700 1 -- 172.21.15.67:0/1603829828 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3a6006ecc0 msgr2=0x7f3a60071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:37.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.728+0000 7f3a7a30a700 1 --2- 172.21.15.67:0/1603829828 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3a6006ecc0 0x7f3a60071180 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f3a64005950 tx=0x7f3a640058e0 comp rx=0 tx=0).stop 2024-04-22T23:57:37.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.728+0000 7f3a7a30a700 1 -- 172.21.15.67:0/1603829828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a74100750 msgr2=0x7f3a7419edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:37.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.728+0000 7f3a7a30a700 1 --2- 172.21.15.67:0/1603829828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a74100750 0x7f3a7419edc0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c0092e0 tx=0x7f3a5c00be20 comp rx=0 tx=0).stop 2024-04-22T23:57:37.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.729+0000 7f3a7a30a700 1 -- 172.21.15.67:0/1603829828 shutdown_connections 2024-04-22T23:57:37.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.729+0000 7f3a7a30a700 1 --2- 172.21.15.67:0/1603829828 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3a6006ecc0 0x7f3a60071180 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:37.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.729+0000 7f3a7a30a700 1 --2- 172.21.15.67:0/1603829828 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a740ffda0 0x7f3a7419e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:37.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.729+0000 7f3a7a30a700 1 --2- 172.21.15.67:0/1603829828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a74100750 0x7f3a7419edc0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:37.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.729+0000 7f3a7a30a700 1 -- 172.21.15.67:0/1603829828 >> 172.21.15.67:0/1603829828 conn(0x7f3a740fb650 msgr2=0x7f3a74103780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:57:37.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.729+0000 7f3a7a30a700 1 -- 172.21.15.67:0/1603829828 shutdown_connections 2024-04-22T23:57:37.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.729+0000 7f3a7a30a700 1 -- 172.21.15.67:0/1603829828 wait complete. 2024-04-22T23:57:37.731 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:57:37.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.836+0000 7f9ea7734700 1 -- 172.21.15.67:0/3849512749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ea0105bd0 msgr2=0x7f9ea0105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:37.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.836+0000 7f9ea7734700 1 --2- 172.21.15.67:0/3849512749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ea0105bd0 0x7f9ea0105fb0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f9e90009910 tx=0x7f9e90009c20 comp rx=0 tx=0).stop 2024-04-22T23:57:37.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.836+0000 7f9ea7734700 1 -- 172.21.15.67:0/3849512749 shutdown_connections 2024-04-22T23:57:37.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.836+0000 7f9ea7734700 1 --2- 172.21.15.67:0/3849512749 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ea00ffc20 0x7f9ea0100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:37.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.836+0000 7f9ea7734700 1 --2- 172.21.15.67:0/3849512749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ea0105bd0 0x7f9ea0105fb0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:37.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.836+0000 7f9ea7734700 1 -- 172.21.15.67:0/3849512749 >> 172.21.15.67:0/3849512749 conn(0x7f9ea0075670 msgr2=0x7f9ea0075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:57:37.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.836+0000 7f9ea7734700 1 -- 172.21.15.67:0/3849512749 shutdown_connections 2024-04-22T23:57:37.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.836+0000 7f9ea7734700 1 -- 172.21.15.67:0/3849512749 wait complete. 2024-04-22T23:57:37.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.837+0000 7f9ea7734700 1 Processor -- start 2024-04-22T23:57:37.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.837+0000 7f9ea7734700 1 -- start start 2024-04-22T23:57:37.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.837+0000 7f9ea7734700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ea00ffc20 0x7f9ea0101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:37.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.837+0000 7f9ea7734700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ea0105bd0 0x7f9ea0101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:37.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.837+0000 7f9ea7734700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea01022c0 con 0x7f9ea00ffc20 2024-04-22T23:57:37.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.837+0000 7f9ea7734700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea0102400 con 0x7f9ea0105bd0 2024-04-22T23:57:37.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.837+0000 7f9ea54d0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ea00ffc20 0x7f9ea0101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:37.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.837+0000 7f9ea54d0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ea00ffc20 0x7f9ea0101840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52838/0 (socket says 172.21.15.67:52838) 2024-04-22T23:57:37.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.837+0000 7f9ea54d0700 1 -- 172.21.15.67:0/1054033149 learned_addr learned my addr 172.21.15.67:0/1054033149 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:57:37.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.837+0000 7f9ea4ccf700 1 --2- 172.21.15.67:0/1054033149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ea0105bd0 0x7f9ea0101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:37.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.837+0000 7f9ea54d0700 1 -- 172.21.15.67:0/1054033149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ea0105bd0 msgr2=0x7f9ea0101d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:37.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.837+0000 7f9ea54d0700 1 --2- 172.21.15.67:0/1054033149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ea0105bd0 0x7f9ea0101d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:37.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.837+0000 7f9ea54d0700 1 -- 172.21.15.67:0/1054033149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e900095f0 con 0x7f9ea00ffc20 2024-04-22T23:57:37.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.838+0000 7f9ea54d0700 1 --2- 172.21.15.67:0/1054033149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ea00ffc20 0x7f9ea0101840 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f9e90006010 tx=0x7f9e9000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:57:37.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.838+0000 7f9e967fc700 1 -- 172.21.15.67:0/1054033149 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e9001d070 con 0x7f9ea00ffc20 2024-04-22T23:57:37.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.838+0000 7f9e967fc700 1 -- 172.21.15.67:0/1054033149 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9e90005dc0 con 0x7f9ea00ffc20 2024-04-22T23:57:37.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.838+0000 7f9e967fc700 1 -- 172.21.15.67:0/1054033149 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e90017400 con 0x7f9ea00ffc20 2024-04-22T23:57:37.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.838+0000 7f9ea7734700 1 -- 172.21.15.67:0/1054033149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ea0102680 con 0x7f9ea00ffc20 2024-04-22T23:57:37.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.838+0000 7f9ea7734700 1 -- 172.21.15.67:0/1054033149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ea01a2d50 con 0x7f9ea00ffc20 2024-04-22T23:57:37.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.839+0000 7f9ea7734700 1 -- 172.21.15.67:0/1054033149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ea004f2e0 con 0x7f9ea00ffc20 2024-04-22T23:57:37.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.839+0000 7f9e967fc700 1 -- 172.21.15.67:0/1054033149 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9e90003a60 con 0x7f9ea00ffc20 2024-04-22T23:57:37.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.840+0000 7f9e967fc700 1 --2- 172.21.15.67:0/1054033149 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9e8c06ecc0 0x7f9e8c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:37.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.840+0000 7f9e967fc700 1 -- 172.21.15.67:0/1054033149 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f9e90090370 con 0x7f9ea00ffc20 2024-04-22T23:57:37.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.840+0000 7f9ea4ccf700 1 --2- 172.21.15.67:0/1054033149 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9e8c06ecc0 0x7f9e8c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:37.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.840+0000 7f9ea4ccf700 1 --2- 172.21.15.67:0/1054033149 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9e8c06ecc0 0x7f9e8c071180 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f9ea0103030 tx=0x7f9e9c00b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:57:37.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:37.845+0000 7f9e967fc700 1 -- 172.21.15.67:0/1054033149 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f9e90059030 con 0x7f9ea00ffc20 2024-04-22T23:57:38.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.076+0000 7f9ea7734700 1 -- 172.21.15.67:0/1054033149 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ea0068d90 con 0x7f9e8c06ecc0 2024-04-22T23:57:38.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.077+0000 7f9e967fc700 1 -- 172.21.15.67:0/1054033149 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9ea0068d90 con 0x7f9e8c06ecc0 2024-04-22T23:57:38.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.079+0000 7f9ea7734700 1 -- 172.21.15.67:0/1054033149 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9e8c06ecc0 msgr2=0x7f9e8c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:38.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.079+0000 7f9ea7734700 1 --2- 172.21.15.67:0/1054033149 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9e8c06ecc0 0x7f9e8c071180 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f9ea0103030 tx=0x7f9e9c00b410 comp rx=0 tx=0).stop 2024-04-22T23:57:38.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.079+0000 7f9ea7734700 1 -- 172.21.15.67:0/1054033149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ea00ffc20 msgr2=0x7f9ea0101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:38.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.079+0000 7f9ea7734700 1 --2- 172.21.15.67:0/1054033149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ea00ffc20 0x7f9ea0101840 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f9e90006010 tx=0x7f9e9000bd70 comp rx=0 tx=0).stop 2024-04-22T23:57:38.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.079+0000 7f9ea7734700 1 -- 172.21.15.67:0/1054033149 shutdown_connections 2024-04-22T23:57:38.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.079+0000 7f9ea7734700 1 --2- 172.21.15.67:0/1054033149 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9e8c06ecc0 0x7f9e8c071180 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:38.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.079+0000 7f9ea7734700 1 --2- 172.21.15.67:0/1054033149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ea00ffc20 0x7f9ea0101840 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:38.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.079+0000 7f9ea7734700 1 --2- 172.21.15.67:0/1054033149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ea0105bd0 0x7f9ea0101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:38.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.079+0000 7f9ea7734700 1 -- 172.21.15.67:0/1054033149 >> 172.21.15.67:0/1054033149 conn(0x7f9ea0075670 msgr2=0x7f9ea0068680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:57:38.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.079+0000 7f9ea7734700 1 -- 172.21.15.67:0/1054033149 shutdown_connections 2024-04-22T23:57:38.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.079+0000 7f9ea7734700 1 -- 172.21.15.67:0/1054033149 wait complete. 2024-04-22T23:57:38.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.188+0000 7fc58be08700 1 -- 172.21.15.67:0/537501139 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc584101e80 msgr2=0x7fc584102260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:38.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.188+0000 7fc58be08700 1 --2- 172.21.15.67:0/537501139 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc584101e80 0x7fc584102260 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fc57400b0a0 tx=0x7fc57400b3b0 comp rx=0 tx=0).stop 2024-04-22T23:57:38.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.188+0000 7fc58be08700 1 -- 172.21.15.67:0/537501139 shutdown_connections 2024-04-22T23:57:38.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.188+0000 7fc58be08700 1 --2- 172.21.15.67:0/537501139 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5841027a0 0x7fc58410ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:38.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.188+0000 7fc58be08700 1 --2- 172.21.15.67:0/537501139 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc584101e80 0x7fc584102260 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:38.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.188+0000 7fc58be08700 1 -- 172.21.15.67:0/537501139 >> 172.21.15.67:0/537501139 conn(0x7fc5840fb610 msgr2=0x7fc5840fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:57:38.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.188+0000 7fc58be08700 1 -- 172.21.15.67:0/537501139 shutdown_connections 2024-04-22T23:57:38.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.188+0000 7fc58be08700 1 -- 172.21.15.67:0/537501139 wait complete. 2024-04-22T23:57:38.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.188+0000 7fc58be08700 1 Processor -- start 2024-04-22T23:57:38.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.189+0000 7fc58be08700 1 -- start start 2024-04-22T23:57:38.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.189+0000 7fc58be08700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc584101e80 0x7fc5840ffae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:38.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.189+0000 7fc58be08700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5841027a0 0x7fc584100020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:38.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.189+0000 7fc58be08700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc584100560 con 0x7fc584101e80 2024-04-22T23:57:38.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.189+0000 7fc58be08700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5841006a0 con 0x7fc5841027a0 2024-04-22T23:57:38.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.189+0000 7fc589ba4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc584101e80 0x7fc5840ffae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:38.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.189+0000 7fc589ba4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc584101e80 0x7fc5840ffae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52866/0 (socket says 172.21.15.67:52866) 2024-04-22T23:57:38.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.189+0000 7fc589ba4700 1 -- 172.21.15.67:0/2104990343 learned_addr learned my addr 172.21.15.67:0/2104990343 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:57:38.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.189+0000 7fc5893a3700 1 --2- 172.21.15.67:0/2104990343 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5841027a0 0x7fc584100020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:38.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.189+0000 7fc589ba4700 1 -- 172.21.15.67:0/2104990343 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5841027a0 msgr2=0x7fc584100020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:38.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.189+0000 7fc589ba4700 1 --2- 172.21.15.67:0/2104990343 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5841027a0 0x7fc584100020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:38.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.189+0000 7fc589ba4700 1 -- 172.21.15.67:0/2104990343 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc574009d00 con 0x7fc584101e80 2024-04-22T23:57:38.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.189+0000 7fc5893a3700 1 --2- 172.21.15.67:0/2104990343 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5841027a0 0x7fc584100020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:57:38.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.189+0000 7fc589ba4700 1 --2- 172.21.15.67:0/2104990343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc584101e80 0x7fc5840ffae0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fc574000f80 tx=0x7fc574003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:57:38.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.190+0000 7fc582ffd700 1 -- 172.21.15.67:0/2104990343 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc57400e040 con 0x7fc584101e80 2024-04-22T23:57:38.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.190+0000 7fc582ffd700 1 -- 172.21.15.67:0/2104990343 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc574007c00 con 0x7fc584101e80 2024-04-22T23:57:38.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.190+0000 7fc58be08700 1 -- 172.21.15.67:0/2104990343 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc584100920 con 0x7fc584101e80 2024-04-22T23:57:38.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.190+0000 7fc582ffd700 1 -- 172.21.15.67:0/2104990343 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc57401d6b0 con 0x7fc584101e80 2024-04-22T23:57:38.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.190+0000 7fc58be08700 1 -- 172.21.15.67:0/2104990343 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5841a2df0 con 0x7fc584101e80 2024-04-22T23:57:38.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.191+0000 7fc58be08700 1 -- 172.21.15.67:0/2104990343 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc584108390 con 0x7fc584101e80 2024-04-22T23:57:38.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.191+0000 7fc582ffd700 1 -- 172.21.15.67:0/2104990343 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc574004040 con 0x7fc584101e80 2024-04-22T23:57:38.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.191+0000 7fc582ffd700 1 --2- 172.21.15.67:0/2104990343 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc56c06ecc0 0x7fc56c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:38.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.192+0000 7fc582ffd700 1 -- 172.21.15.67:0/2104990343 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fc574090610 con 0x7fc584101e80 2024-04-22T23:57:38.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.192+0000 7fc5893a3700 1 --2- 172.21.15.67:0/2104990343 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc56c06ecc0 0x7fc56c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:38.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.192+0000 7fc5893a3700 1 --2- 172.21.15.67:0/2104990343 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc56c06ecc0 0x7fc56c071180 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fc578005fd0 tx=0x7fc578005ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:57:38.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.197+0000 7fc582ffd700 1 -- 172.21.15.67:0/2104990343 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc5740592d0 con 0x7fc584101e80 2024-04-22T23:57:38.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.419+0000 7fc58be08700 1 -- 172.21.15.67:0/2104990343 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc5840611d0 con 0x7fc56c06ecc0 2024-04-22T23:57:38.411 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:38 smithi067 ceph-mon[112438]: from='client.35692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:57:38.411 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:38 smithi067 ceph-mon[112438]: from='client.35696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:57:38.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.432+0000 7fc582ffd700 1 -- 172.21.15.67:0/2104990343 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fc5840611d0 con 0x7fc56c06ecc0 2024-04-22T23:57:38.424 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:57:38.424 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (31m) 9m ago 42m 24.4M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:57:38.424 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (42m) 9m ago 42m 10.0M - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:57:38.424 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (41m) 9m ago 41m 10.0M - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:57:38.424 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (28m) 9m ago 42m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:57:38.424 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (28m) 9m ago 41m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:57:38.424 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (30m) 9m ago 42m 81.2M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:57:38.424 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (25m) 9m ago 37m 19.4M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:57:38.424 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (24m) 9m ago 37m 20.3M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:57:38.424 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (24m) 9m ago 37m 20.1M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:57:38.424 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (9m) 9m ago 37m 14.0M - 19.0.0-3252-g43be0201 a0434699c1d7 46d3d7e86fee 2024-04-22T23:57:38.425 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (33m) 9m ago 44m 514M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:57:38.425 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (32m) 9m ago 41m 459M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:57:38.425 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (29m) 9m ago 44m 74.2M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:57:38.425 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (29m) 9m ago 41m 54.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:57:38.425 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (32m) 9m ago 42m 15.2M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:57:38.425 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (31m) 9m ago 41m 16.0M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:57:38.425 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (28m) 9m ago 40m 102M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:57:38.425 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (27m) 9m ago 40m 128M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:57:38.425 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (27m) 9m ago 39m 83.8M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:57:38.425 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (26m) 9m ago 39m 101M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:57:38.425 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (25m) 9m ago 39m 86.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:57:38.425 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (25m) 9m ago 38m 81.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:57:38.425 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (31m) 9m ago 42m 78.7M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:57:38.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.434+0000 7fc58be08700 1 -- 172.21.15.67:0/2104990343 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc56c06ecc0 msgr2=0x7fc56c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:38.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.434+0000 7fc58be08700 1 --2- 172.21.15.67:0/2104990343 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc56c06ecc0 0x7fc56c071180 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fc578005fd0 tx=0x7fc578005ee0 comp rx=0 tx=0).stop 2024-04-22T23:57:38.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.434+0000 7fc58be08700 1 -- 172.21.15.67:0/2104990343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc584101e80 msgr2=0x7fc5840ffae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:38.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.434+0000 7fc58be08700 1 --2- 172.21.15.67:0/2104990343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc584101e80 0x7fc5840ffae0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fc574000f80 tx=0x7fc574003ce0 comp rx=0 tx=0).stop 2024-04-22T23:57:38.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.434+0000 7fc58be08700 1 -- 172.21.15.67:0/2104990343 shutdown_connections 2024-04-22T23:57:38.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.434+0000 7fc58be08700 1 --2- 172.21.15.67:0/2104990343 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc56c06ecc0 0x7fc56c071180 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:38.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.434+0000 7fc58be08700 1 --2- 172.21.15.67:0/2104990343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc584101e80 0x7fc5840ffae0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:38.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.434+0000 7fc58be08700 1 --2- 172.21.15.67:0/2104990343 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc5841027a0 0x7fc584100020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:38.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.434+0000 7fc58be08700 1 -- 172.21.15.67:0/2104990343 >> 172.21.15.67:0/2104990343 conn(0x7fc5840fb610 msgr2=0x7fc5841054d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:57:38.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.434+0000 7fc58be08700 1 -- 172.21.15.67:0/2104990343 shutdown_connections 2024-04-22T23:57:38.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.434+0000 7fc58be08700 1 -- 172.21.15.67:0/2104990343 wait complete. 2024-04-22T23:57:38.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:38 smithi134 ceph-mon[85630]: from='client.35692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:57:38.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:38 smithi134 ceph-mon[85630]: from='client.35696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:57:38.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.545+0000 7ff2180ba700 1 -- 172.21.15.67:0/3158742820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff210100ea0 msgr2=0x7ff210101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:38.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.545+0000 7ff2180ba700 1 --2- 172.21.15.67:0/3158742820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff210100ea0 0x7ff210101280 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7ff20c009910 tx=0x7ff20c009c20 comp rx=0 tx=0).stop 2024-04-22T23:57:38.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.545+0000 7ff2180ba700 1 -- 172.21.15.67:0/3158742820 shutdown_connections 2024-04-22T23:57:38.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.545+0000 7ff2180ba700 1 --2- 172.21.15.67:0/3158742820 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff210101850 0x7ff2101058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:38.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.545+0000 7ff2180ba700 1 --2- 172.21.15.67:0/3158742820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff210100ea0 0x7ff210101280 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:38.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.545+0000 7ff2180ba700 1 -- 172.21.15.67:0/3158742820 >> 172.21.15.67:0/3158742820 conn(0x7ff2100fc750 msgr2=0x7ff2100feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:57:38.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.545+0000 7ff2180ba700 1 -- 172.21.15.67:0/3158742820 shutdown_connections 2024-04-22T23:57:38.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.545+0000 7ff2180ba700 1 -- 172.21.15.67:0/3158742820 wait complete. 2024-04-22T23:57:38.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.546+0000 7ff2180ba700 1 Processor -- start 2024-04-22T23:57:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.546+0000 7ff2180ba700 1 -- start start 2024-04-22T23:57:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.546+0000 7ff2180ba700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff210100ea0 0x7ff21019c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.546+0000 7ff2180ba700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff210101850 0x7ff21019cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.546+0000 7ff2180ba700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21019d2c0 con 0x7ff210100ea0 2024-04-22T23:57:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.546+0000 7ff2180ba700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff210196770 con 0x7ff210101850 2024-04-22T23:57:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.547+0000 7ff215e56700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff210100ea0 0x7ff21019c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.547+0000 7ff215e56700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff210100ea0 0x7ff21019c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52882/0 (socket says 172.21.15.67:52882) 2024-04-22T23:57:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.547+0000 7ff215e56700 1 -- 172.21.15.67:0/2512736223 learned_addr learned my addr 172.21.15.67:0/2512736223 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:57:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.547+0000 7ff215655700 1 --2- 172.21.15.67:0/2512736223 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff210101850 0x7ff21019cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.547+0000 7ff215e56700 1 -- 172.21.15.67:0/2512736223 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff210101850 msgr2=0x7ff21019cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.547+0000 7ff215e56700 1 --2- 172.21.15.67:0/2512736223 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff210101850 0x7ff21019cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.547+0000 7ff215e56700 1 -- 172.21.15.67:0/2512736223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff20c0095f0 con 0x7ff210100ea0 2024-04-22T23:57:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.547+0000 7ff215e56700 1 --2- 172.21.15.67:0/2512736223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff210100ea0 0x7ff21019c6f0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7ff20c006010 tx=0x7ff20c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:57:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.547+0000 7ff202ffd700 1 -- 172.21.15.67:0/2512736223 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff20c01d070 con 0x7ff210100ea0 2024-04-22T23:57:38.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.547+0000 7ff202ffd700 1 -- 172.21.15.67:0/2512736223 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff20c005dc0 con 0x7ff210100ea0 2024-04-22T23:57:38.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.547+0000 7ff2180ba700 1 -- 172.21.15.67:0/2512736223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2101969f0 con 0x7ff210100ea0 2024-04-22T23:57:38.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.547+0000 7ff202ffd700 1 -- 172.21.15.67:0/2512736223 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff20c017400 con 0x7ff210100ea0 2024-04-22T23:57:38.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.547+0000 7ff2180ba700 1 -- 172.21.15.67:0/2512736223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff210196ee0 con 0x7ff210100ea0 2024-04-22T23:57:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.548+0000 7ff2180ba700 1 -- 172.21.15.67:0/2512736223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff210109240 con 0x7ff210100ea0 2024-04-22T23:57:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.549+0000 7ff202ffd700 1 -- 172.21.15.67:0/2512736223 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff20c003a60 con 0x7ff210100ea0 2024-04-22T23:57:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.549+0000 7ff202ffd700 1 --2- 172.21.15.67:0/2512736223 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff1fc06ecc0 0x7ff1fc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.549+0000 7ff202ffd700 1 -- 172.21.15.67:0/2512736223 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7ff20c090370 con 0x7ff210100ea0 2024-04-22T23:57:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.549+0000 7ff215655700 1 --2- 172.21.15.67:0/2512736223 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff1fc06ecc0 0x7ff1fc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.550+0000 7ff215655700 1 --2- 172.21.15.67:0/2512736223 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff1fc06ecc0 0x7ff1fc071180 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7ff210197be0 tx=0x7ff20400b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:57:38.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.554+0000 7ff202ffd700 1 -- 172.21.15.67:0/2512736223 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff20c05cb60 con 0x7ff210100ea0 2024-04-22T23:57:38.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.854+0000 7ff2180ba700 1 -- 172.21.15.67:0/2512736223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff21004f2e0 con 0x7ff210100ea0 2024-04-22T23:57:38.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.854+0000 7ff202ffd700 1 -- 172.21.15.67:0/2512736223 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7ff20c05c6f0 con 0x7ff210100ea0 2024-04-22T23:57:38.846 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:57:38.846 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:57:38.846 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:57:38.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:57:38.847 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:57:38.847 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:57:38.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:57:38.847 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:57:38.847 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:57:38.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:57:38.847 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:57:38.847 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 4 2024-04-22T23:57:38.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:57:38.847 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:57:38.847 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 14 2024-04-22T23:57:38.847 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:57:38.847 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:57:38.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.857+0000 7ff2180ba700 1 -- 172.21.15.67:0/2512736223 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff1fc06ecc0 msgr2=0x7ff1fc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:38.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.857+0000 7ff2180ba700 1 --2- 172.21.15.67:0/2512736223 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff1fc06ecc0 0x7ff1fc071180 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7ff210197be0 tx=0x7ff20400b410 comp rx=0 tx=0).stop 2024-04-22T23:57:38.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.857+0000 7ff2180ba700 1 -- 172.21.15.67:0/2512736223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff210100ea0 msgr2=0x7ff21019c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:38.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.857+0000 7ff2180ba700 1 --2- 172.21.15.67:0/2512736223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff210100ea0 0x7ff21019c6f0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7ff20c006010 tx=0x7ff20c00bd70 comp rx=0 tx=0).stop 2024-04-22T23:57:38.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.857+0000 7ff2180ba700 1 -- 172.21.15.67:0/2512736223 shutdown_connections 2024-04-22T23:57:38.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.857+0000 7ff2180ba700 1 --2- 172.21.15.67:0/2512736223 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff1fc06ecc0 0x7ff1fc071180 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:38.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.857+0000 7ff2180ba700 1 --2- 172.21.15.67:0/2512736223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff210100ea0 0x7ff21019c6f0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:38.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.857+0000 7ff2180ba700 1 --2- 172.21.15.67:0/2512736223 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff210101850 0x7ff21019cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:38.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.857+0000 7ff2180ba700 1 -- 172.21.15.67:0/2512736223 >> 172.21.15.67:0/2512736223 conn(0x7ff2100fc750 msgr2=0x7ff2100fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:57:38.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.857+0000 7ff2180ba700 1 -- 172.21.15.67:0/2512736223 shutdown_connections 2024-04-22T23:57:38.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.857+0000 7ff2180ba700 1 -- 172.21.15.67:0/2512736223 wait complete. 2024-04-22T23:57:38.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.963+0000 7f2226c58700 1 -- 172.21.15.67:0/4237402804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22200ffb40 msgr2=0x7f222010ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:38.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.963+0000 7f2226c58700 1 --2- 172.21.15.67:0/4237402804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22200ffb40 0x7f222010ab40 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f220c009910 tx=0x7f220c009c20 comp rx=0 tx=0).stop 2024-04-22T23:57:38.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.963+0000 7f2226c58700 1 -- 172.21.15.67:0/4237402804 shutdown_connections 2024-04-22T23:57:38.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.963+0000 7f2226c58700 1 --2- 172.21.15.67:0/4237402804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22200ffb40 0x7f222010ab40 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:38.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.963+0000 7f2226c58700 1 --2- 172.21.15.67:0/4237402804 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f22200ff190 0x7f22200ff570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:38.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.963+0000 7f2226c58700 1 -- 172.21.15.67:0/4237402804 >> 172.21.15.67:0/4237402804 conn(0x7f2220075e10 msgr2=0x7f2220076220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:57:38.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.964+0000 7f2226c58700 1 -- 172.21.15.67:0/4237402804 shutdown_connections 2024-04-22T23:57:38.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.964+0000 7f2226c58700 1 -- 172.21.15.67:0/4237402804 wait complete. 2024-04-22T23:57:38.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.964+0000 7f2226c58700 1 Processor -- start 2024-04-22T23:57:38.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.964+0000 7f2226c58700 1 -- start start 2024-04-22T23:57:38.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.965+0000 7f2226c58700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f22200ff190 0x7f222019c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:38.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.965+0000 7f2226c58700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22200ffb40 0x7f222019cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:38.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.965+0000 7f2226c58700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f222019d200 con 0x7f22200ffb40 2024-04-22T23:57:38.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.965+0000 7f2226c58700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22201966b0 con 0x7f22200ff190 2024-04-22T23:57:38.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.965+0000 7f221ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22200ffb40 0x7f222019cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:38.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.965+0000 7f221ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22200ffb40 0x7f222019cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46454/0 (socket says 172.21.15.67:46454) 2024-04-22T23:57:38.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.965+0000 7f221ffff700 1 -- 172.21.15.67:0/3548668207 learned_addr learned my addr 172.21.15.67:0/3548668207 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:57:38.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.965+0000 7f22249f4700 1 --2- 172.21.15.67:0/3548668207 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f22200ff190 0x7f222019c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:38.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.965+0000 7f221ffff700 1 -- 172.21.15.67:0/3548668207 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f22200ff190 msgr2=0x7f222019c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:38.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.965+0000 7f221ffff700 1 --2- 172.21.15.67:0/3548668207 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f22200ff190 0x7f222019c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:38.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.965+0000 7f221ffff700 1 -- 172.21.15.67:0/3548668207 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f220c0095f0 con 0x7f22200ffb40 2024-04-22T23:57:38.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.965+0000 7f221ffff700 1 --2- 172.21.15.67:0/3548668207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22200ffb40 0x7f222019cb70 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f220c005f50 tx=0x7f220c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:57:38.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.965+0000 7f221dffb700 1 -- 172.21.15.67:0/3548668207 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f220c01d070 con 0x7f22200ffb40 2024-04-22T23:57:38.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.965+0000 7f221dffb700 1 -- 172.21.15.67:0/3548668207 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f220c00f460 con 0x7f22200ffb40 2024-04-22T23:57:38.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.965+0000 7f221dffb700 1 -- 172.21.15.67:0/3548668207 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f220c017630 con 0x7f22200ffb40 2024-04-22T23:57:38.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.965+0000 7f2226c58700 1 -- 172.21.15.67:0/3548668207 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2220196930 con 0x7f22200ffb40 2024-04-22T23:57:38.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.966+0000 7f2226c58700 1 -- 172.21.15.67:0/3548668207 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2220196e20 con 0x7f22200ffb40 2024-04-22T23:57:38.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.966+0000 7f2226c58700 1 -- 172.21.15.67:0/3548668207 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2220108240 con 0x7f22200ffb40 2024-04-22T23:57:38.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.967+0000 7f221dffb700 1 -- 172.21.15.67:0/3548668207 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f220c00f5d0 con 0x7f22200ffb40 2024-04-22T23:57:38.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.967+0000 7f221dffb700 1 --2- 172.21.15.67:0/3548668207 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f221006ecc0 0x7f2210071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:38.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.968+0000 7f221dffb700 1 -- 172.21.15.67:0/3548668207 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f220c0905b0 con 0x7f22200ffb40 2024-04-22T23:57:38.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.968+0000 7f22249f4700 1 --2- 172.21.15.67:0/3548668207 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f221006ecc0 0x7f2210071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:38.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.968+0000 7f22249f4700 1 --2- 172.21.15.67:0/3548668207 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f221006ecc0 0x7f2210071180 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f2214005950 tx=0x7f221400a300 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:57:38.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:38.972+0000 7f221dffb700 1 -- 172.21.15.67:0/3548668207 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f220c05cc80 con 0x7f22200ffb40 2024-04-22T23:57:39.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.228+0000 7f2226c58700 1 -- 172.21.15.67:0/3548668207 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2220197c60 con 0x7f22200ffb40 2024-04-22T23:57:39.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.228+0000 7f221dffb700 1 -- 172.21.15.67:0/3548668207 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1872 (secure 0 0 0) 0x7f220c05c810 con 0x7f22200ffb40 2024-04-22T23:57:39.220 INFO:teuthology.orchestra.run.smithi067.stdout:e33 2024-04-22T23:57:39.220 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:57:39.221 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:57:39.221 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:57:39.221 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:57:39.221 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:57:39.221 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:57:39.221 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:57:39.221 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:57:39.221 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:57:39.221 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:57:39.221 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:57:39.221 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:57:39.221 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:57:39.221 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:57:39.221 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:57:39.221 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:57:39.222 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:57:39.222 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:57:39.222 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:57:39.222 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:57:39.222 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:57:39.222 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:57:39.222 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:57:39.222 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:57:39.222 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:57:39.222 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:57:39.222 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:57:39.222 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:57:39.222 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:57:39.222 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:57:39.222 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:57:39.223 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:57:39.223 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:57:39.223 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:57:39.223 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:57:39.223 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:57:39.223 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:57:39.223 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:57:39.223 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:57:39.223 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:57:39.223 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44851} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1849648622,v1:172.21.15.134:6827/1849648622] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:57:39.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.231+0000 7f2226c58700 1 -- 172.21.15.67:0/3548668207 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f221006ecc0 msgr2=0x7f2210071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:39.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.231+0000 7f2226c58700 1 --2- 172.21.15.67:0/3548668207 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f221006ecc0 0x7f2210071180 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f2214005950 tx=0x7f221400a300 comp rx=0 tx=0).stop 2024-04-22T23:57:39.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.231+0000 7f2226c58700 1 -- 172.21.15.67:0/3548668207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22200ffb40 msgr2=0x7f222019cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:39.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.231+0000 7f2226c58700 1 --2- 172.21.15.67:0/3548668207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22200ffb40 0x7f222019cb70 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f220c005f50 tx=0x7f220c00bd70 comp rx=0 tx=0).stop 2024-04-22T23:57:39.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.231+0000 7f2226c58700 1 -- 172.21.15.67:0/3548668207 shutdown_connections 2024-04-22T23:57:39.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.231+0000 7f2226c58700 1 --2- 172.21.15.67:0/3548668207 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f221006ecc0 0x7f2210071180 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:39.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.231+0000 7f2226c58700 1 --2- 172.21.15.67:0/3548668207 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f22200ff190 0x7f222019c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:39.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.231+0000 7f2226c58700 1 --2- 172.21.15.67:0/3548668207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22200ffb40 0x7f222019cb70 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:39.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.231+0000 7f2226c58700 1 -- 172.21.15.67:0/3548668207 >> 172.21.15.67:0/3548668207 conn(0x7f2220075e10 msgr2=0x7f2220105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:57:39.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.231+0000 7f2226c58700 1 -- 172.21.15.67:0/3548668207 shutdown_connections 2024-04-22T23:57:39.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.231+0000 7f2226c58700 1 -- 172.21.15.67:0/3548668207 wait complete. 2024-04-22T23:57:39.225 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 33 2024-04-22T23:57:39.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.338+0000 7f07c356b700 1 -- 172.21.15.67:0/56294504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07bc100750 msgr2=0x7f07bc1047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:39.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.338+0000 7f07c356b700 1 --2- 172.21.15.67:0/56294504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07bc100750 0x7f07bc1047a0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f07b8009910 tx=0x7f07b8009c20 comp rx=0 tx=0).stop 2024-04-22T23:57:39.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.338+0000 7f07c356b700 1 -- 172.21.15.67:0/56294504 shutdown_connections 2024-04-22T23:57:39.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.338+0000 7f07c356b700 1 --2- 172.21.15.67:0/56294504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07bc100750 0x7f07bc1047a0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:39.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.338+0000 7f07c356b700 1 --2- 172.21.15.67:0/56294504 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07bc0ffda0 0x7f07bc100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:39.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.338+0000 7f07c356b700 1 -- 172.21.15.67:0/56294504 >> 172.21.15.67:0/56294504 conn(0x7f07bc0fb650 msgr2=0x7f07bc0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:57:39.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.338+0000 7f07c356b700 1 -- 172.21.15.67:0/56294504 shutdown_connections 2024-04-22T23:57:39.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.338+0000 7f07c356b700 1 -- 172.21.15.67:0/56294504 wait complete. 2024-04-22T23:57:39.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.339+0000 7f07c356b700 1 Processor -- start 2024-04-22T23:57:39.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.339+0000 7f07c356b700 1 -- start start 2024-04-22T23:57:39.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.339+0000 7f07c356b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07bc0ffda0 0x7f07bc101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:39.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.339+0000 7f07c356b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07bc100750 0x7f07bc101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:39.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.339+0000 7f07c356b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07bc1022c0 con 0x7f07bc100750 2024-04-22T23:57:39.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.339+0000 7f07c356b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07bc102400 con 0x7f07bc0ffda0 2024-04-22T23:57:39.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.339+0000 7f07c0b06700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07bc100750 0x7f07bc101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:39.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.339+0000 7f07c0b06700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07bc100750 0x7f07bc101d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46476/0 (socket says 172.21.15.67:46476) 2024-04-22T23:57:39.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.339+0000 7f07c0b06700 1 -- 172.21.15.67:0/2216808057 learned_addr learned my addr 172.21.15.67:0/2216808057 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:57:39.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.340+0000 7f07c1307700 1 --2- 172.21.15.67:0/2216808057 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07bc0ffda0 0x7f07bc101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:39.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.340+0000 7f07c0b06700 1 -- 172.21.15.67:0/2216808057 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07bc0ffda0 msgr2=0x7f07bc101840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:39.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.340+0000 7f07c0b06700 1 --2- 172.21.15.67:0/2216808057 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07bc0ffda0 0x7f07bc101840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:39.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.340+0000 7f07c0b06700 1 -- 172.21.15.67:0/2216808057 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b80095f0 con 0x7f07bc100750 2024-04-22T23:57:39.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.340+0000 7f07c0b06700 1 --2- 172.21.15.67:0/2216808057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07bc100750 0x7f07bc101d80 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f07b800bda0 tx=0x7f07b800bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:57:39.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.340+0000 7f07ae7fc700 1 -- 172.21.15.67:0/2216808057 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07b801d070 con 0x7f07bc100750 2024-04-22T23:57:39.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.340+0000 7f07ae7fc700 1 -- 172.21.15.67:0/2216808057 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f07b800f460 con 0x7f07bc100750 2024-04-22T23:57:39.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.340+0000 7f07c356b700 1 -- 172.21.15.67:0/2216808057 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07bc102680 con 0x7f07bc100750 2024-04-22T23:57:39.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.340+0000 7f07ae7fc700 1 -- 172.21.15.67:0/2216808057 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07b8017630 con 0x7f07bc100750 2024-04-22T23:57:39.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.340+0000 7f07c356b700 1 -- 172.21.15.67:0/2216808057 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07bc1a2d50 con 0x7f07bc100750 2024-04-22T23:57:39.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.341+0000 7f07c356b700 1 -- 172.21.15.67:0/2216808057 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07bc1091b0 con 0x7f07bc100750 2024-04-22T23:57:39.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.342+0000 7f07ae7fc700 1 -- 172.21.15.67:0/2216808057 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f07b800fab0 con 0x7f07bc100750 2024-04-22T23:57:39.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.342+0000 7f07ae7fc700 1 --2- 172.21.15.67:0/2216808057 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f07a806ecc0 0x7f07a8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:39.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.342+0000 7f07c1307700 1 --2- 172.21.15.67:0/2216808057 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f07a806ecc0 0x7f07a8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:39.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.342+0000 7f07ae7fc700 1 -- 172.21.15.67:0/2216808057 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f07b80903a0 con 0x7f07bc100750 2024-04-22T23:57:39.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.343+0000 7f07c1307700 1 --2- 172.21.15.67:0/2216808057 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f07a806ecc0 0x7f07a8071180 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f07b000ac50 tx=0x7f07b000a380 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:57:39.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.348+0000 7f07ae7fc700 1 -- 172.21.15.67:0/2216808057 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f07b8059060 con 0x7f07bc100750 2024-04-22T23:57:39.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:39 smithi134 ceph-mon[85630]: from='client.35700 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:57:39.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:39 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2512736223' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:57:39.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:39 smithi134 ceph-mon[85630]: pgmap v940: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:39.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:39 smithi067 ceph-mon[112438]: from='client.35700 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:57:39.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:39 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2512736223' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:57:39.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:39 smithi067 ceph-mon[112438]: pgmap v940: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:39.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.580+0000 7f07c356b700 1 -- 172.21.15.67:0/2216808057 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f07bc102e80 con 0x7f07a806ecc0 2024-04-22T23:57:39.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.582+0000 7f07ae7fc700 1 -- 172.21.15.67:0/2216808057 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f07bc102e80 con 0x7f07a806ecc0 2024-04-22T23:57:39.573 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:57:39.573 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:57:39.573 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:57:39.574 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:57:39.574 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:57:39.574 INFO:teuthology.orchestra.run.smithi067.stdout: "mds", 2024-04-22T23:57:39.574 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:57:39.574 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:57:39.574 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:57:39.574 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:57:39.574 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:57:39.574 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T23:57:39.574 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:57:39.574 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:57:39.574 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:57:39.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.584+0000 7f07c356b700 1 -- 172.21.15.67:0/2216808057 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f07a806ecc0 msgr2=0x7f07a8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:39.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.584+0000 7f07c356b700 1 --2- 172.21.15.67:0/2216808057 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f07a806ecc0 0x7f07a8071180 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f07b000ac50 tx=0x7f07b000a380 comp rx=0 tx=0).stop 2024-04-22T23:57:39.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.584+0000 7f07c356b700 1 -- 172.21.15.67:0/2216808057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07bc100750 msgr2=0x7f07bc101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:39.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.584+0000 7f07c356b700 1 --2- 172.21.15.67:0/2216808057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07bc100750 0x7f07bc101d80 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f07b800bda0 tx=0x7f07b800bdd0 comp rx=0 tx=0).stop 2024-04-22T23:57:39.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.584+0000 7f07c356b700 1 -- 172.21.15.67:0/2216808057 shutdown_connections 2024-04-22T23:57:39.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.584+0000 7f07c356b700 1 --2- 172.21.15.67:0/2216808057 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f07a806ecc0 0x7f07a8071180 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:39.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.584+0000 7f07c356b700 1 --2- 172.21.15.67:0/2216808057 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07bc0ffda0 0x7f07bc101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:39.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.584+0000 7f07c356b700 1 --2- 172.21.15.67:0/2216808057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07bc100750 0x7f07bc101d80 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:39.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.584+0000 7f07c356b700 1 -- 172.21.15.67:0/2216808057 >> 172.21.15.67:0/2216808057 conn(0x7f07bc0fb650 msgr2=0x7f07bc069150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:57:39.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.584+0000 7f07c356b700 1 -- 172.21.15.67:0/2216808057 shutdown_connections 2024-04-22T23:57:39.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.584+0000 7f07c356b700 1 -- 172.21.15.67:0/2216808057 wait complete. 2024-04-22T23:57:39.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.691+0000 7f6d39184700 1 -- 172.21.15.67:0/3118710972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d34106cd0 msgr2=0x7f6d341070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:39.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.691+0000 7f6d39184700 1 --2- 172.21.15.67:0/3118710972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d34106cd0 0x7f6d341070b0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f6d2400b0a0 tx=0x7f6d2400b3b0 comp rx=0 tx=0).stop 2024-04-22T23:57:39.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.691+0000 7f6d39184700 1 -- 172.21.15.67:0/3118710972 shutdown_connections 2024-04-22T23:57:39.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.691+0000 7f6d39184700 1 --2- 172.21.15.67:0/3118710972 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6d34068580 0x7f6d340689e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:39.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.691+0000 7f6d39184700 1 --2- 172.21.15.67:0/3118710972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d34106cd0 0x7f6d341070b0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:39.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.691+0000 7f6d39184700 1 -- 172.21.15.67:0/3118710972 >> 172.21.15.67:0/3118710972 conn(0x7f6d34075670 msgr2=0x7f6d34075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:57:39.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.691+0000 7f6d39184700 1 -- 172.21.15.67:0/3118710972 shutdown_connections 2024-04-22T23:57:39.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.691+0000 7f6d39184700 1 -- 172.21.15.67:0/3118710972 wait complete. 2024-04-22T23:57:39.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.692+0000 7f6d39184700 1 Processor -- start 2024-04-22T23:57:39.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.692+0000 7f6d39184700 1 -- start start 2024-04-22T23:57:39.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.692+0000 7f6d39184700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6d34068580 0x7f6d3419c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:39.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.692+0000 7f6d39184700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d34106cd0 0x7f6d3419cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:39.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.692+0000 7f6d39184700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3419d230 con 0x7f6d34106cd0 2024-04-22T23:57:39.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.692+0000 7f6d39184700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d341966e0 con 0x7f6d34068580 2024-04-22T23:57:39.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.692+0000 7f6d3259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d34106cd0 0x7f6d3419cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:39.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.692+0000 7f6d3259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d34106cd0 0x7f6d3419cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46498/0 (socket says 172.21.15.67:46498) 2024-04-22T23:57:39.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.692+0000 7f6d3259c700 1 -- 172.21.15.67:0/3215138257 learned_addr learned my addr 172.21.15.67:0/3215138257 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:57:39.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.692+0000 7f6d32d9d700 1 --2- 172.21.15.67:0/3215138257 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6d34068580 0x7f6d3419c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:39.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.692+0000 7f6d3259c700 1 -- 172.21.15.67:0/3215138257 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6d34068580 msgr2=0x7f6d3419c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:39.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.692+0000 7f6d3259c700 1 --2- 172.21.15.67:0/3215138257 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6d34068580 0x7f6d3419c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:39.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.692+0000 7f6d3259c700 1 -- 172.21.15.67:0/3215138257 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d24009d00 con 0x7f6d34106cd0 2024-04-22T23:57:39.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.692+0000 7f6d32d9d700 1 --2- 172.21.15.67:0/3215138257 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6d34068580 0x7f6d3419c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:57:39.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.693+0000 7f6d3259c700 1 --2- 172.21.15.67:0/3215138257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d34106cd0 0x7f6d3419cba0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c00d8d0 tx=0x7f6d1c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:57:39.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.693+0000 7f6d2bfff700 1 -- 172.21.15.67:0/3215138257 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d1c009940 con 0x7f6d34106cd0 2024-04-22T23:57:39.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.693+0000 7f6d39184700 1 -- 172.21.15.67:0/3215138257 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d341969c0 con 0x7f6d34106cd0 2024-04-22T23:57:39.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.693+0000 7f6d2bfff700 1 -- 172.21.15.67:0/3215138257 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6d1c010460 con 0x7f6d34106cd0 2024-04-22T23:57:39.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.693+0000 7f6d2bfff700 1 -- 172.21.15.67:0/3215138257 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d1c00f5f0 con 0x7f6d34106cd0 2024-04-22T23:57:39.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.693+0000 7f6d39184700 1 -- 172.21.15.67:0/3215138257 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d34196f10 con 0x7f6d34106cd0 2024-04-22T23:57:39.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.694+0000 7f6d39184700 1 -- 172.21.15.67:0/3215138257 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d341091d0 con 0x7f6d34106cd0 2024-04-22T23:57:39.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.694+0000 7f6d2bfff700 1 -- 172.21.15.67:0/3215138257 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6d1c0105d0 con 0x7f6d34106cd0 2024-04-22T23:57:39.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.695+0000 7f6d2bfff700 1 --2- 172.21.15.67:0/3215138257 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6d2006ecc0 0x7f6d20071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:57:39.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.695+0000 7f6d2bfff700 1 -- 172.21.15.67:0/3215138257 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f6d1c08eff0 con 0x7f6d34106cd0 2024-04-22T23:57:39.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.695+0000 7f6d32d9d700 1 --2- 172.21.15.67:0/3215138257 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6d2006ecc0 0x7f6d20071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:57:39.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.696+0000 7f6d32d9d700 1 --2- 172.21.15.67:0/3215138257 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6d2006ecc0 0x7f6d20071180 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f6d24009480 tx=0x7f6d24006000 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:57:39.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.700+0000 7f6d2bfff700 1 -- 172.21.15.67:0/3215138257 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f6d1c05b760 con 0x7f6d34106cd0 2024-04-22T23:57:39.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.996+0000 7f6d39184700 1 -- 172.21.15.67:0/3215138257 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f6d3404f2e0 con 0x7f6d34106cd0 2024-04-22T23:57:39.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.996+0000 7f6d2bfff700 1 -- 172.21.15.67:0/3215138257 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4505 (secure 0 0 0) 0x7f6d1c05b2f0 con 0x7f6d34106cd0 2024-04-22T23:57:39.987 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:57:39.987 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T23:57:39.987 INFO:teuthology.orchestra.run.smithi067.stdout: host smithi134 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-22T23:57:39.987 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T23:57:39.987 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:57:39.987 INFO:teuthology.orchestra.run.smithi067.stdout:Traceback (most recent call last): 2024-04-22T23:57:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T23:57:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: return _run_code(code, main_globals, None, 2024-04-22T23:57:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T23:57:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: exec(code, run_globals) 2024-04-22T23:57:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5642, in 2024-04-22T23:57:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5630, in main 2024-04-22T23:57:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 408, in _infer_config 2024-04-22T23:57:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 323, in _infer_fsid 2024-04-22T23:57:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 436, in _infer_image 2024-04-22T23:57:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 310, in _validate_fsid 2024-04-22T23:57:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 3380, in command_ceph_volume 2024-04-22T23:57:39.988 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T23:57:39.988 INFO:teuthology.orchestra.run.smithi067.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:57:39.988 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:57:39.989 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:57:39.989 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:57:39.989 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:57:39.989 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:57:39.989 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:57:39.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.998+0000 7f6d39184700 1 -- 172.21.15.67:0/3215138257 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6d2006ecc0 msgr2=0x7f6d20071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:39.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.998+0000 7f6d39184700 1 --2- 172.21.15.67:0/3215138257 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6d2006ecc0 0x7f6d20071180 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f6d24009480 tx=0x7f6d24006000 comp rx=0 tx=0).stop 2024-04-22T23:57:39.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.998+0000 7f6d39184700 1 -- 172.21.15.67:0/3215138257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d34106cd0 msgr2=0x7f6d3419cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:57:39.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.998+0000 7f6d39184700 1 --2- 172.21.15.67:0/3215138257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d34106cd0 0x7f6d3419cba0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c00d8d0 tx=0x7f6d1c00dc90 comp rx=0 tx=0).stop 2024-04-22T23:57:39.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.998+0000 7f6d39184700 1 -- 172.21.15.67:0/3215138257 shutdown_connections 2024-04-22T23:57:39.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.998+0000 7f6d39184700 1 --2- 172.21.15.67:0/3215138257 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6d2006ecc0 0x7f6d20071180 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:39.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.998+0000 7f6d39184700 1 --2- 172.21.15.67:0/3215138257 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6d34068580 0x7f6d3419c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:39.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.998+0000 7f6d39184700 1 --2- 172.21.15.67:0/3215138257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d34106cd0 0x7f6d3419cba0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:57:39.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.998+0000 7f6d39184700 1 -- 172.21.15.67:0/3215138257 >> 172.21.15.67:0/3215138257 conn(0x7f6d34075670 msgr2=0x7f6d340fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:57:39.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.999+0000 7f6d39184700 1 -- 172.21.15.67:0/3215138257 shutdown_connections 2024-04-22T23:57:39.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:57:39.999+0000 7f6d39184700 1 -- 172.21.15.67:0/3215138257 wait complete. 2024-04-22T23:57:40.284 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:40 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3548668207' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:57:40.284 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:40 smithi067 ceph-mon[112438]: from='client.35712 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:57:40.285 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:40 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3215138257' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:57:40.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:40 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3548668207' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:57:40.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:40 smithi134 ceph-mon[85630]: from='client.35712 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:57:40.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:40 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3215138257' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:57:41.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:57:41.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:41 smithi134 ceph-mon[85630]: pgmap v941: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:41.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:57:41.537 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:41 smithi067 ceph-mon[112438]: pgmap v941: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:43.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:43 smithi134 ceph-mon[85630]: pgmap v942: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:43.488 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:43 smithi067 ceph-mon[112438]: pgmap v942: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:45.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:45 smithi134 ceph-mon[85630]: pgmap v943: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:45.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:45 smithi067 ceph-mon[112438]: pgmap v943: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:47.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:47 smithi134 ceph-mon[85630]: pgmap v944: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:47.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:47 smithi067 ceph-mon[112438]: pgmap v944: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:49.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:49 smithi134 ceph-mon[85630]: pgmap v945: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:49.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:49 smithi067 ceph-mon[112438]: pgmap v945: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:51.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:51 smithi134 ceph-mon[85630]: pgmap v946: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:51.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:51 smithi067 ceph-mon[112438]: pgmap v946: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:53.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:53 smithi134 ceph-mon[85630]: pgmap v947: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:53.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:53 smithi067 ceph-mon[112438]: pgmap v947: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:55.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:55 smithi134 ceph-mon[85630]: pgmap v948: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:55.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:55 smithi067 ceph-mon[112438]: pgmap v948: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:56.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:57:56.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:57:57.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:57 smithi134 ceph-mon[85630]: pgmap v949: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:57.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:57 smithi067 ceph-mon[112438]: pgmap v949: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:59.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:57:59 smithi134 ceph-mon[85630]: pgmap v950: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:57:59.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:57:59 smithi067 ceph-mon[112438]: pgmap v950: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:01.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:01 smithi134 ceph-mon[85630]: pgmap v951: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:01.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:01 smithi067 ceph-mon[112438]: pgmap v951: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:03.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:03 smithi134 ceph-mon[85630]: pgmap v952: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:03.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:03 smithi067 ceph-mon[112438]: pgmap v952: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:05.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:05 smithi134 ceph-mon[85630]: pgmap v953: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:05.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:05 smithi067 ceph-mon[112438]: pgmap v953: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:07.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:07 smithi134 ceph-mon[85630]: pgmap v954: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:07.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:07 smithi067 ceph-mon[112438]: pgmap v954: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:09.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:09 smithi134 ceph-mon[85630]: pgmap v955: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:09.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:09 smithi067 ceph-mon[112438]: pgmap v955: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:10.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.121+0000 7f0dc0463700 1 -- 172.21.15.67:0/812478429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0db80ffda0 msgr2=0x7f0db8100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:10.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.121+0000 7f0dc0463700 1 --2- 172.21.15.67:0/812478429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0db80ffda0 0x7f0db8100180 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f0db4009910 tx=0x7f0db4009c20 comp rx=0 tx=0).stop 2024-04-22T23:58:10.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.121+0000 7f0dc0463700 1 -- 172.21.15.67:0/812478429 shutdown_connections 2024-04-22T23:58:10.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.121+0000 7f0dc0463700 1 --2- 172.21.15.67:0/812478429 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0db8100750 0x7f0db81047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:10.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.121+0000 7f0dc0463700 1 --2- 172.21.15.67:0/812478429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0db80ffda0 0x7f0db8100180 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:10.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.121+0000 7f0dc0463700 1 -- 172.21.15.67:0/812478429 >> 172.21.15.67:0/812478429 conn(0x7f0db80fb650 msgr2=0x7f0db80fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:58:10.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.121+0000 7f0dc0463700 1 -- 172.21.15.67:0/812478429 shutdown_connections 2024-04-22T23:58:10.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.121+0000 7f0dc0463700 1 -- 172.21.15.67:0/812478429 wait complete. 2024-04-22T23:58:10.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.122+0000 7f0dc0463700 1 Processor -- start 2024-04-22T23:58:10.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.122+0000 7f0dc0463700 1 -- start start 2024-04-22T23:58:10.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.122+0000 7f0dc0463700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0db80ffda0 0x7f0db8101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:10.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.122+0000 7f0dc0463700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0db8100750 0x7f0db8101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:10.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.122+0000 7f0dc0463700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db8102350 con 0x7f0db80ffda0 2024-04-22T23:58:10.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.122+0000 7f0dc0463700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db81024c0 con 0x7f0db8100750 2024-04-22T23:58:10.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.122+0000 7f0dbe1ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0db80ffda0 0x7f0db8101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:10.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.122+0000 7f0dbe1ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0db80ffda0 0x7f0db8101840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57362/0 (socket says 172.21.15.67:57362) 2024-04-22T23:58:10.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.122+0000 7f0dbe1ff700 1 -- 172.21.15.67:0/2787842316 learned_addr learned my addr 172.21.15.67:0/2787842316 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:58:10.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.123+0000 7f0dbd9fe700 1 --2- 172.21.15.67:0/2787842316 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0db8100750 0x7f0db8101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:10.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.123+0000 7f0dbe1ff700 1 -- 172.21.15.67:0/2787842316 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0db8100750 msgr2=0x7f0db8101d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:10.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.123+0000 7f0dbe1ff700 1 --2- 172.21.15.67:0/2787842316 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0db8100750 0x7f0db8101d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:10.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.123+0000 7f0dbe1ff700 1 -- 172.21.15.67:0/2787842316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db40095f0 con 0x7f0db80ffda0 2024-04-22T23:58:10.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.123+0000 7f0dbe1ff700 1 --2- 172.21.15.67:0/2787842316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0db80ffda0 0x7f0db8101840 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f0db4006010 tx=0x7f0db400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:58:10.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.123+0000 7f0daf7fe700 1 -- 172.21.15.67:0/2787842316 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0db401d070 con 0x7f0db80ffda0 2024-04-22T23:58:10.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.123+0000 7f0dc0463700 1 -- 172.21.15.67:0/2787842316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0db81a2aa0 con 0x7f0db80ffda0 2024-04-22T23:58:10.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.123+0000 7f0daf7fe700 1 -- 172.21.15.67:0/2787842316 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0db4005dc0 con 0x7f0db80ffda0 2024-04-22T23:58:10.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.123+0000 7f0daf7fe700 1 -- 172.21.15.67:0/2787842316 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0db4017400 con 0x7f0db80ffda0 2024-04-22T23:58:10.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.123+0000 7f0dc0463700 1 -- 172.21.15.67:0/2787842316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0db81a2e50 con 0x7f0db80ffda0 2024-04-22T23:58:10.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.124+0000 7f0dc0463700 1 -- 172.21.15.67:0/2787842316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0db804f2e0 con 0x7f0db80ffda0 2024-04-22T23:58:10.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.125+0000 7f0daf7fe700 1 -- 172.21.15.67:0/2787842316 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0db4003a60 con 0x7f0db80ffda0 2024-04-22T23:58:10.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.125+0000 7f0daf7fe700 1 --2- 172.21.15.67:0/2787842316 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0da406ecc0 0x7f0da4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:10.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.126+0000 7f0dbd9fe700 1 --2- 172.21.15.67:0/2787842316 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0da406ecc0 0x7f0da4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:10.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.126+0000 7f0daf7fe700 1 -- 172.21.15.67:0/2787842316 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f0db4090430 con 0x7f0db80ffda0 2024-04-22T23:58:10.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.126+0000 7f0dbd9fe700 1 --2- 172.21.15.67:0/2787842316 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0da406ecc0 0x7f0da4071180 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f0db8103030 tx=0x7f0da80073d0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:58:10.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.130+0000 7f0daf7fe700 1 -- 172.21.15.67:0/2787842316 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f0db4059070 con 0x7f0db80ffda0 2024-04-22T23:58:10.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.361+0000 7f0dc0463700 1 -- 172.21.15.67:0/2787842316 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0db80fc4c0 con 0x7f0da406ecc0 2024-04-22T23:58:10.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.363+0000 7f0daf7fe700 1 -- 172.21.15.67:0/2787842316 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f0db80fc4c0 con 0x7f0da406ecc0 2024-04-22T23:58:10.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.365+0000 7f0dc0463700 1 -- 172.21.15.67:0/2787842316 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0da406ecc0 msgr2=0x7f0da4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:10.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.365+0000 7f0dc0463700 1 --2- 172.21.15.67:0/2787842316 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0da406ecc0 0x7f0da4071180 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f0db8103030 tx=0x7f0da80073d0 comp rx=0 tx=0).stop 2024-04-22T23:58:10.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.365+0000 7f0dc0463700 1 -- 172.21.15.67:0/2787842316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0db80ffda0 msgr2=0x7f0db8101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:10.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.365+0000 7f0dc0463700 1 --2- 172.21.15.67:0/2787842316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0db80ffda0 0x7f0db8101840 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f0db4006010 tx=0x7f0db400bd70 comp rx=0 tx=0).stop 2024-04-22T23:58:10.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.365+0000 7f0dc0463700 1 -- 172.21.15.67:0/2787842316 shutdown_connections 2024-04-22T23:58:10.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.365+0000 7f0dc0463700 1 --2- 172.21.15.67:0/2787842316 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0da406ecc0 0x7f0da4071180 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:10.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.365+0000 7f0dc0463700 1 --2- 172.21.15.67:0/2787842316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0db80ffda0 0x7f0db8101840 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:10.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.365+0000 7f0dc0463700 1 --2- 172.21.15.67:0/2787842316 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0db8100750 0x7f0db8101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:10.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.365+0000 7f0dc0463700 1 -- 172.21.15.67:0/2787842316 >> 172.21.15.67:0/2787842316 conn(0x7f0db80fb650 msgr2=0x7f0db8103fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:58:10.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.365+0000 7f0dc0463700 1 -- 172.21.15.67:0/2787842316 shutdown_connections 2024-04-22T23:58:10.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.365+0000 7f0dc0463700 1 -- 172.21.15.67:0/2787842316 wait complete. 2024-04-22T23:58:10.366 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:58:10.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.473+0000 7f007b286700 1 -- 172.21.15.67:0/2859187593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0074105bf0 msgr2=0x7f0074105fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:10.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.473+0000 7f007b286700 1 --2- 172.21.15.67:0/2859187593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0074105bf0 0x7f0074105fd0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f0070009910 tx=0x7f0070009c20 comp rx=0 tx=0).stop 2024-04-22T23:58:10.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.473+0000 7f007b286700 1 -- 172.21.15.67:0/2859187593 shutdown_connections 2024-04-22T23:58:10.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.473+0000 7f007b286700 1 --2- 172.21.15.67:0/2859187593 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f00740ffb40 0x7f00740fffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:10.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.473+0000 7f007b286700 1 --2- 172.21.15.67:0/2859187593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0074105bf0 0x7f0074105fd0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:10.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.473+0000 7f007b286700 1 -- 172.21.15.67:0/2859187593 >> 172.21.15.67:0/2859187593 conn(0x7f00740fb650 msgr2=0x7f00740fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:58:10.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.473+0000 7f007b286700 1 -- 172.21.15.67:0/2859187593 shutdown_connections 2024-04-22T23:58:10.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.473+0000 7f007b286700 1 -- 172.21.15.67:0/2859187593 wait complete. 2024-04-22T23:58:10.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.474+0000 7f007b286700 1 Processor -- start 2024-04-22T23:58:10.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.474+0000 7f007b286700 1 -- start start 2024-04-22T23:58:10.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.474+0000 7f007b286700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f00740ffb40 0x7f00741017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:10.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.474+0000 7f007b286700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0074105bf0 0x7f0074101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:10.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.474+0000 7f007b286700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0074102230 con 0x7f0074105bf0 2024-04-22T23:58:10.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.474+0000 7f007b286700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0074102370 con 0x7f00740ffb40 2024-04-22T23:58:10.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.474+0000 7f0078821700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0074105bf0 0x7f0074101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:10.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.474+0000 7f0078821700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0074105bf0 0x7f0074101cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57382/0 (socket says 172.21.15.67:57382) 2024-04-22T23:58:10.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.474+0000 7f0078821700 1 -- 172.21.15.67:0/4061266527 learned_addr learned my addr 172.21.15.67:0/4061266527 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:58:10.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.475+0000 7f0079022700 1 --2- 172.21.15.67:0/4061266527 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f00740ffb40 0x7f00741017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:10.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.475+0000 7f0078821700 1 -- 172.21.15.67:0/4061266527 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f00740ffb40 msgr2=0x7f00741017b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:10.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.475+0000 7f0078821700 1 --2- 172.21.15.67:0/4061266527 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f00740ffb40 0x7f00741017b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:10.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.475+0000 7f0078821700 1 -- 172.21.15.67:0/4061266527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00700095f0 con 0x7f0074105bf0 2024-04-22T23:58:10.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.475+0000 7f0078821700 1 --2- 172.21.15.67:0/4061266527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0074105bf0 0x7f0074101cf0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f006400b4b0 tx=0x7f006400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:58:10.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.475+0000 7f006a7fc700 1 -- 172.21.15.67:0/4061266527 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0064011800 con 0x7f0074105bf0 2024-04-22T23:58:10.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.475+0000 7f006a7fc700 1 -- 172.21.15.67:0/4061266527 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0064011e20 con 0x7f0074105bf0 2024-04-22T23:58:10.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.475+0000 7f007b286700 1 -- 172.21.15.67:0/4061266527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00741a2b00 con 0x7f0074105bf0 2024-04-22T23:58:10.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.475+0000 7f006a7fc700 1 -- 172.21.15.67:0/4061266527 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f006400f2e0 con 0x7f0074105bf0 2024-04-22T23:58:10.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.475+0000 7f007b286700 1 -- 172.21.15.67:0/4061266527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00741a2f10 con 0x7f0074105bf0 2024-04-22T23:58:10.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.476+0000 7f007b286700 1 -- 172.21.15.67:0/4061266527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f007404f2e0 con 0x7f0074105bf0 2024-04-22T23:58:10.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.477+0000 7f006a7fc700 1 -- 172.21.15.67:0/4061266527 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0064010380 con 0x7f0074105bf0 2024-04-22T23:58:10.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.477+0000 7f006a7fc700 1 --2- 172.21.15.67:0/4061266527 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f006006ecc0 0x7f0060071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:10.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.477+0000 7f006a7fc700 1 -- 172.21.15.67:0/4061266527 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f006408eb70 con 0x7f0074105bf0 2024-04-22T23:58:10.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.477+0000 7f0079022700 1 --2- 172.21.15.67:0/4061266527 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f006006ecc0 0x7f0060071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:10.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.477+0000 7f0079022700 1 --2- 172.21.15.67:0/4061266527 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f006006ecc0 0x7f0060071180 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f007000b5c0 tx=0x7f007001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:58:10.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.482+0000 7f006a7fc700 1 -- 172.21.15.67:0/4061266527 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f00640577b0 con 0x7f0074105bf0 2024-04-22T23:58:10.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.712+0000 7f007b286700 1 -- 172.21.15.67:0/4061266527 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0074069660 con 0x7f006006ecc0 2024-04-22T23:58:10.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.713+0000 7f006a7fc700 1 -- 172.21.15.67:0/4061266527 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f0074069660 con 0x7f006006ecc0 2024-04-22T23:58:10.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.716+0000 7f007b286700 1 -- 172.21.15.67:0/4061266527 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f006006ecc0 msgr2=0x7f0060071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:10.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.716+0000 7f007b286700 1 --2- 172.21.15.67:0/4061266527 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f006006ecc0 0x7f0060071180 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f007000b5c0 tx=0x7f007001a040 comp rx=0 tx=0).stop 2024-04-22T23:58:10.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.716+0000 7f007b286700 1 -- 172.21.15.67:0/4061266527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0074105bf0 msgr2=0x7f0074101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:10.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.716+0000 7f007b286700 1 --2- 172.21.15.67:0/4061266527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0074105bf0 0x7f0074101cf0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f006400b4b0 tx=0x7f006400b870 comp rx=0 tx=0).stop 2024-04-22T23:58:10.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.716+0000 7f007b286700 1 -- 172.21.15.67:0/4061266527 shutdown_connections 2024-04-22T23:58:10.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.716+0000 7f007b286700 1 --2- 172.21.15.67:0/4061266527 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f006006ecc0 0x7f0060071180 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:10.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.716+0000 7f007b286700 1 --2- 172.21.15.67:0/4061266527 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f00740ffb40 0x7f00741017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:10.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.716+0000 7f007b286700 1 --2- 172.21.15.67:0/4061266527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0074105bf0 0x7f0074101cf0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:10.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.716+0000 7f007b286700 1 -- 172.21.15.67:0/4061266527 >> 172.21.15.67:0/4061266527 conn(0x7f00740fb650 msgr2=0x7f0074068f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:58:10.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.716+0000 7f007b286700 1 -- 172.21.15.67:0/4061266527 shutdown_connections 2024-04-22T23:58:10.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.716+0000 7f007b286700 1 -- 172.21.15.67:0/4061266527 wait complete. 2024-04-22T23:58:10.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.822+0000 7f4bdd193700 1 -- 172.21.15.67:0/1807226065 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bd8100ea0 msgr2=0x7f4bd8101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:10.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.822+0000 7f4bdd193700 1 --2- 172.21.15.67:0/1807226065 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bd8100ea0 0x7f4bd8101280 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f4bc8009910 tx=0x7f4bc8009c20 comp rx=0 tx=0).stop 2024-04-22T23:58:10.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.822+0000 7f4bdd193700 1 -- 172.21.15.67:0/1807226065 shutdown_connections 2024-04-22T23:58:10.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.822+0000 7f4bdd193700 1 --2- 172.21.15.67:0/1807226065 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4bd8101850 0x7f4bd81058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:10.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.822+0000 7f4bdd193700 1 --2- 172.21.15.67:0/1807226065 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bd8100ea0 0x7f4bd8101280 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:10.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.822+0000 7f4bdd193700 1 -- 172.21.15.67:0/1807226065 >> 172.21.15.67:0/1807226065 conn(0x7f4bd80fc750 msgr2=0x7f4bd80feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:58:10.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.822+0000 7f4bdd193700 1 -- 172.21.15.67:0/1807226065 shutdown_connections 2024-04-22T23:58:10.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.822+0000 7f4bdd193700 1 -- 172.21.15.67:0/1807226065 wait complete. 2024-04-22T23:58:10.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.823+0000 7f4bdd193700 1 Processor -- start 2024-04-22T23:58:10.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.823+0000 7f4bdd193700 1 -- start start 2024-04-22T23:58:10.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.823+0000 7f4bdd193700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4bd8100ea0 0x7f4bd819c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:10.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.823+0000 7f4bdd193700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bd8101850 0x7f4bd819cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:10.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.823+0000 7f4bdd193700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd819d2c0 con 0x7f4bd8101850 2024-04-22T23:58:10.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.823+0000 7f4bdd193700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd8196770 con 0x7f4bd8100ea0 2024-04-22T23:58:10.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.824+0000 7f4bd659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bd8101850 0x7f4bd819cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:10.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.824+0000 7f4bd659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bd8101850 0x7f4bd819cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57400/0 (socket says 172.21.15.67:57400) 2024-04-22T23:58:10.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.824+0000 7f4bd659c700 1 -- 172.21.15.67:0/3842937526 learned_addr learned my addr 172.21.15.67:0/3842937526 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:58:10.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.824+0000 7f4bd6d9d700 1 --2- 172.21.15.67:0/3842937526 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4bd8100ea0 0x7f4bd819c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:10.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.824+0000 7f4bd659c700 1 -- 172.21.15.67:0/3842937526 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4bd8100ea0 msgr2=0x7f4bd819c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:10.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.824+0000 7f4bd659c700 1 --2- 172.21.15.67:0/3842937526 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4bd8100ea0 0x7f4bd819c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:10.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.824+0000 7f4bd659c700 1 -- 172.21.15.67:0/3842937526 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc80095f0 con 0x7f4bd8101850 2024-04-22T23:58:10.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.824+0000 7f4bd659c700 1 --2- 172.21.15.67:0/3842937526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bd8101850 0x7f4bd819cc30 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc00c8f0 tx=0x7f4bcc00cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:58:10.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.824+0000 7f4bbffff700 1 -- 172.21.15.67:0/3842937526 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4bcc0078a0 con 0x7f4bd8101850 2024-04-22T23:58:10.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.824+0000 7f4bdd193700 1 -- 172.21.15.67:0/3842937526 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bd8196a50 con 0x7f4bd8101850 2024-04-22T23:58:10.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.824+0000 7f4bbffff700 1 -- 172.21.15.67:0/3842937526 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4bcc00f450 con 0x7f4bd8101850 2024-04-22T23:58:10.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.824+0000 7f4bbffff700 1 -- 172.21.15.67:0/3842937526 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4bcc018610 con 0x7f4bd8101850 2024-04-22T23:58:10.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.824+0000 7f4bdd193700 1 -- 172.21.15.67:0/3842937526 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bd8196fa0 con 0x7f4bd8101850 2024-04-22T23:58:10.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.825+0000 7f4bdd193700 1 -- 172.21.15.67:0/3842937526 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bd8109240 con 0x7f4bd8101850 2024-04-22T23:58:10.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.826+0000 7f4bbffff700 1 -- 172.21.15.67:0/3842937526 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4bcc00f5c0 con 0x7f4bd8101850 2024-04-22T23:58:10.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.826+0000 7f4bbffff700 1 --2- 172.21.15.67:0/3842937526 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4bc006ecc0 0x7f4bc0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:10.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.826+0000 7f4bbffff700 1 -- 172.21.15.67:0/3842937526 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f4bcc08ef40 con 0x7f4bd8101850 2024-04-22T23:58:10.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.826+0000 7f4bd6d9d700 1 --2- 172.21.15.67:0/3842937526 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4bc006ecc0 0x7f4bc0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:10.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.827+0000 7f4bd6d9d700 1 --2- 172.21.15.67:0/3842937526 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4bc006ecc0 0x7f4bc0071180 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f4bc8006010 tx=0x7f4bc801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:58:10.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:10.831+0000 7f4bbffff700 1 -- 172.21.15.67:0/3842937526 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f4bcc0c4150 con 0x7f4bd8101850 2024-04-22T23:58:11.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.048+0000 7f4bdd193700 1 -- 172.21.15.67:0/3842937526 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4bd8197d90 con 0x7f4bc006ecc0 2024-04-22T23:58:11.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.060+0000 7f4bbffff700 1 -- 172.21.15.67:0/3842937526 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f4bd8197d90 con 0x7f4bc006ecc0 2024-04-22T23:58:11.051 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:58:11.051 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (31m) 9m ago 43m 24.4M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:58:11.051 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (43m) 9m ago 43m 10.0M - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:58:11.051 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (42m) 9m ago 42m 10.0M - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:58:11.051 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (29m) 9m ago 43m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:58:11.051 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (29m) 9m ago 42m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:58:11.051 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (31m) 9m ago 42m 81.2M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:58:11.051 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (25m) 9m ago 37m 19.4M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:58:11.051 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (25m) 9m ago 37m 20.3M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:58:11.051 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (24m) 9m ago 37m 20.1M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:58:11.051 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (9m) 9m ago 37m 14.0M - 19.0.0-3252-g43be0201 a0434699c1d7 46d3d7e86fee 2024-04-22T23:58:11.052 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (33m) 9m ago 44m 514M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:58:11.052 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (32m) 9m ago 42m 459M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:58:11.052 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (30m) 9m ago 44m 74.2M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:58:11.052 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (29m) 9m ago 42m 54.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:58:11.052 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (32m) 9m ago 43m 15.2M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:58:11.052 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (32m) 9m ago 42m 16.0M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:58:11.052 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (28m) 9m ago 41m 102M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:58:11.052 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (28m) 9m ago 40m 128M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:58:11.052 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (27m) 9m ago 40m 83.8M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:58:11.052 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (27m) 9m ago 40m 101M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:58:11.052 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (26m) 9m ago 39m 86.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:58:11.052 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (25m) 9m ago 39m 81.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:58:11.052 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (32m) 9m ago 42m 78.7M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:58:11.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.062+0000 7f4bdd193700 1 -- 172.21.15.67:0/3842937526 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4bc006ecc0 msgr2=0x7f4bc0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:11.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.062+0000 7f4bdd193700 1 --2- 172.21.15.67:0/3842937526 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4bc006ecc0 0x7f4bc0071180 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f4bc8006010 tx=0x7f4bc801a040 comp rx=0 tx=0).stop 2024-04-22T23:58:11.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.062+0000 7f4bdd193700 1 -- 172.21.15.67:0/3842937526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bd8101850 msgr2=0x7f4bd819cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:11.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.062+0000 7f4bdd193700 1 --2- 172.21.15.67:0/3842937526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bd8101850 0x7f4bd819cc30 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc00c8f0 tx=0x7f4bcc00cc00 comp rx=0 tx=0).stop 2024-04-22T23:58:11.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.062+0000 7f4bdd193700 1 -- 172.21.15.67:0/3842937526 shutdown_connections 2024-04-22T23:58:11.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.062+0000 7f4bdd193700 1 --2- 172.21.15.67:0/3842937526 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4bc006ecc0 0x7f4bc0071180 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:11.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.062+0000 7f4bdd193700 1 --2- 172.21.15.67:0/3842937526 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4bd8100ea0 0x7f4bd819c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:11.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.062+0000 7f4bdd193700 1 --2- 172.21.15.67:0/3842937526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bd8101850 0x7f4bd819cc30 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:11.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.062+0000 7f4bdd193700 1 -- 172.21.15.67:0/3842937526 >> 172.21.15.67:0/3842937526 conn(0x7f4bd80fc750 msgr2=0x7f4bd80fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:58:11.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.062+0000 7f4bdd193700 1 -- 172.21.15.67:0/3842937526 shutdown_connections 2024-04-22T23:58:11.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.062+0000 7f4bdd193700 1 -- 172.21.15.67:0/3842937526 wait complete. 2024-04-22T23:58:11.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.169+0000 7f3ae2309700 1 -- 172.21.15.67:0/1712490947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3adc100ea0 msgr2=0x7f3adc101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:11.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.169+0000 7f3ae2309700 1 --2- 172.21.15.67:0/1712490947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3adc100ea0 0x7f3adc101280 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f3acc009910 tx=0x7f3acc009c20 comp rx=0 tx=0).stop 2024-04-22T23:58:11.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.169+0000 7f3ae2309700 1 -- 172.21.15.67:0/1712490947 shutdown_connections 2024-04-22T23:58:11.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.169+0000 7f3ae2309700 1 --2- 172.21.15.67:0/1712490947 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3adc101850 0x7f3adc1058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:11.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.169+0000 7f3ae2309700 1 --2- 172.21.15.67:0/1712490947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3adc100ea0 0x7f3adc101280 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:11.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.170+0000 7f3ae2309700 1 -- 172.21.15.67:0/1712490947 >> 172.21.15.67:0/1712490947 conn(0x7f3adc0fc750 msgr2=0x7f3adc0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:58:11.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.170+0000 7f3ae2309700 1 -- 172.21.15.67:0/1712490947 shutdown_connections 2024-04-22T23:58:11.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.170+0000 7f3ae2309700 1 -- 172.21.15.67:0/1712490947 wait complete. 2024-04-22T23:58:11.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.170+0000 7f3ae2309700 1 Processor -- start 2024-04-22T23:58:11.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.170+0000 7f3ae2309700 1 -- start start 2024-04-22T23:58:11.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.171+0000 7f3ae2309700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3adc100ea0 0x7f3adc19c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:11.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.171+0000 7f3ae2309700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3adc101850 0x7f3adc19cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:11.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.171+0000 7f3ae2309700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3adc19d230 con 0x7f3adc100ea0 2024-04-22T23:58:11.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.171+0000 7f3ae2309700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3adc196770 con 0x7f3adc101850 2024-04-22T23:58:11.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.171+0000 7f3adbfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3adc100ea0 0x7f3adc19c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:11.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.171+0000 7f3adbfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3adc100ea0 0x7f3adc19c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57424/0 (socket says 172.21.15.67:57424) 2024-04-22T23:58:11.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.171+0000 7f3adbfff700 1 -- 172.21.15.67:0/1019572239 learned_addr learned my addr 172.21.15.67:0/1019572239 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:58:11.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.171+0000 7f3adb7fe700 1 --2- 172.21.15.67:0/1019572239 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3adc101850 0x7f3adc19cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:11.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.171+0000 7f3adbfff700 1 -- 172.21.15.67:0/1019572239 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3adc101850 msgr2=0x7f3adc19cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:11.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.171+0000 7f3adbfff700 1 --2- 172.21.15.67:0/1019572239 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3adc101850 0x7f3adc19cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:11.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.171+0000 7f3adbfff700 1 -- 172.21.15.67:0/1019572239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3acc0095f0 con 0x7f3adc100ea0 2024-04-22T23:58:11.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.171+0000 7f3adbfff700 1 --2- 172.21.15.67:0/1019572239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3adc100ea0 0x7f3adc19c6f0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f3acc00b5c0 tx=0x7f3acc00be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:58:11.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.171+0000 7f3ad97fa700 1 -- 172.21.15.67:0/1019572239 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3acc01d070 con 0x7f3adc100ea0 2024-04-22T23:58:11.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.171+0000 7f3ae2309700 1 -- 172.21.15.67:0/1019572239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3adc1969f0 con 0x7f3adc100ea0 2024-04-22T23:58:11.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.171+0000 7f3ad97fa700 1 -- 172.21.15.67:0/1019572239 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3acc00f460 con 0x7f3adc100ea0 2024-04-22T23:58:11.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.171+0000 7f3ad97fa700 1 -- 172.21.15.67:0/1019572239 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3acc017610 con 0x7f3adc100ea0 2024-04-22T23:58:11.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.171+0000 7f3ae2309700 1 -- 172.21.15.67:0/1019572239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3adc196ee0 con 0x7f3adc100ea0 2024-04-22T23:58:11.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.173+0000 7f3ae2309700 1 -- 172.21.15.67:0/1019572239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3adc109240 con 0x7f3adc100ea0 2024-04-22T23:58:11.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.173+0000 7f3ad97fa700 1 -- 172.21.15.67:0/1019572239 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3acc0178d0 con 0x7f3adc100ea0 2024-04-22T23:58:11.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.173+0000 7f3ad97fa700 1 --2- 172.21.15.67:0/1019572239 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3ac406ecc0 0x7f3ac4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:11.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.173+0000 7f3ad97fa700 1 -- 172.21.15.67:0/1019572239 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f3acc0905d0 con 0x7f3adc100ea0 2024-04-22T23:58:11.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.173+0000 7f3adb7fe700 1 --2- 172.21.15.67:0/1019572239 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3ac406ecc0 0x7f3ac4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:11.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.174+0000 7f3adb7fe700 1 --2- 172.21.15.67:0/1019572239 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3ac406ecc0 0x7f3ac4071180 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f3ad0005ea0 tx=0x7f3ad0005e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:58:11.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.178+0000 7f3ad97fa700 1 -- 172.21.15.67:0/1019572239 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f3acc05cd40 con 0x7f3adc100ea0 2024-04-22T23:58:11.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.473+0000 7f3ae2309700 1 -- 172.21.15.67:0/1019572239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3adc04f2e0 con 0x7f3adc100ea0 2024-04-22T23:58:11.464 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:58:11.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.473+0000 7f3ad97fa700 1 -- 172.21.15.67:0/1019572239 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f3acc05c8d0 con 0x7f3adc100ea0 2024-04-22T23:58:11.464 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:58:11.464 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:58:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:58:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:58:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:58:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:58:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:58:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:58:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:58:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:58:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:58:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 4 2024-04-22T23:58:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:58:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:58:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 14 2024-04-22T23:58:11.465 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:58:11.465 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:58:11.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.475+0000 7f3ae2309700 1 -- 172.21.15.67:0/1019572239 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3ac406ecc0 msgr2=0x7f3ac4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:11.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.475+0000 7f3ae2309700 1 --2- 172.21.15.67:0/1019572239 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3ac406ecc0 0x7f3ac4071180 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f3ad0005ea0 tx=0x7f3ad0005e30 comp rx=0 tx=0).stop 2024-04-22T23:58:11.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.475+0000 7f3ae2309700 1 -- 172.21.15.67:0/1019572239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3adc100ea0 msgr2=0x7f3adc19c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:11.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.475+0000 7f3ae2309700 1 --2- 172.21.15.67:0/1019572239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3adc100ea0 0x7f3adc19c6f0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f3acc00b5c0 tx=0x7f3acc00be00 comp rx=0 tx=0).stop 2024-04-22T23:58:11.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.476+0000 7f3ae2309700 1 -- 172.21.15.67:0/1019572239 shutdown_connections 2024-04-22T23:58:11.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.476+0000 7f3ae2309700 1 --2- 172.21.15.67:0/1019572239 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3ac406ecc0 0x7f3ac4071180 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:11.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.476+0000 7f3ae2309700 1 --2- 172.21.15.67:0/1019572239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3adc100ea0 0x7f3adc19c6f0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:11.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.476+0000 7f3ae2309700 1 --2- 172.21.15.67:0/1019572239 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3adc101850 0x7f3adc19cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:11.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.476+0000 7f3ae2309700 1 -- 172.21.15.67:0/1019572239 >> 172.21.15.67:0/1019572239 conn(0x7f3adc0fc750 msgr2=0x7f3adc0fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:58:11.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.476+0000 7f3ae2309700 1 -- 172.21.15.67:0/1019572239 shutdown_connections 2024-04-22T23:58:11.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.476+0000 7f3ae2309700 1 -- 172.21.15.67:0/1019572239 wait complete. 2024-04-22T23:58:11.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:58:11.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.583+0000 7f68aa212700 1 -- 172.21.15.67:0/3569330367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68a4101060 msgr2=0x7f68a41047b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:11.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.583+0000 7f68aa212700 1 --2- 172.21.15.67:0/3569330367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68a4101060 0x7f68a41047b0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f6898009910 tx=0x7f6898009c20 comp rx=0 tx=0).stop 2024-04-22T23:58:11.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.583+0000 7f68aa212700 1 -- 172.21.15.67:0/3569330367 shutdown_connections 2024-04-22T23:58:11.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.583+0000 7f68aa212700 1 --2- 172.21.15.67:0/3569330367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68a4101060 0x7f68a41047b0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:11.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.583+0000 7f68aa212700 1 --2- 172.21.15.67:0/3569330367 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f68a41006b0 0x7f68a4100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:11.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.583+0000 7f68aa212700 1 -- 172.21.15.67:0/3569330367 >> 172.21.15.67:0/3569330367 conn(0x7f68a4074e30 msgr2=0x7f68a4075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:58:11.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.583+0000 7f68aa212700 1 -- 172.21.15.67:0/3569330367 shutdown_connections 2024-04-22T23:58:11.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.583+0000 7f68aa212700 1 -- 172.21.15.67:0/3569330367 wait complete. 2024-04-22T23:58:11.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.584+0000 7f68aa212700 1 Processor -- start 2024-04-22T23:58:11.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.584+0000 7f68aa212700 1 -- start start 2024-04-22T23:58:11.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.584+0000 7f68aa212700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f68a41006b0 0x7f68a4101850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:11.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.584+0000 7f68aa212700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68a4101060 0x7f68a4101d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:11.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.584+0000 7f68aa212700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a41022d0 con 0x7f68a4101060 2024-04-22T23:58:11.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.584+0000 7f68aa212700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a4102410 con 0x7f68a41006b0 2024-04-22T23:58:11.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.584+0000 7f68a2ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68a4101060 0x7f68a4101d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:11.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.584+0000 7f68a2ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68a4101060 0x7f68a4101d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57446/0 (socket says 172.21.15.67:57446) 2024-04-22T23:58:11.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.584+0000 7f68a2ffd700 1 -- 172.21.15.67:0/3529327885 learned_addr learned my addr 172.21.15.67:0/3529327885 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:58:11.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.584+0000 7f68a37fe700 1 --2- 172.21.15.67:0/3529327885 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f68a41006b0 0x7f68a4101850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:11.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.585+0000 7f68a2ffd700 1 -- 172.21.15.67:0/3529327885 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f68a41006b0 msgr2=0x7f68a4101850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:11.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.585+0000 7f68a2ffd700 1 --2- 172.21.15.67:0/3529327885 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f68a41006b0 0x7f68a4101850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:11.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.585+0000 7f68a2ffd700 1 -- 172.21.15.67:0/3529327885 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68980095f0 con 0x7f68a4101060 2024-04-22T23:58:11.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.585+0000 7f68a2ffd700 1 --2- 172.21.15.67:0/3529327885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68a4101060 0x7f68a4101d90 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f68980092e0 tx=0x7f689800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:58:11.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.585+0000 7f68a0ff9700 1 -- 172.21.15.67:0/3529327885 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f689801d070 con 0x7f68a4101060 2024-04-22T23:58:11.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.585+0000 7f68a0ff9700 1 -- 172.21.15.67:0/3529327885 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6898005dc0 con 0x7f68a4101060 2024-04-22T23:58:11.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.585+0000 7f68aa212700 1 -- 172.21.15.67:0/3529327885 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68a4102690 con 0x7f68a4101060 2024-04-22T23:58:11.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.585+0000 7f68a0ff9700 1 -- 172.21.15.67:0/3529327885 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6898017400 con 0x7f68a4101060 2024-04-22T23:58:11.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.585+0000 7f68aa212700 1 -- 172.21.15.67:0/3529327885 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68a41a2d50 con 0x7f68a4101060 2024-04-22T23:58:11.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.586+0000 7f68aa212700 1 -- 172.21.15.67:0/3529327885 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68a41090a0 con 0x7f68a4101060 2024-04-22T23:58:11.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.587+0000 7f68a0ff9700 1 -- 172.21.15.67:0/3529327885 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f689800f6f0 con 0x7f68a4101060 2024-04-22T23:58:11.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.587+0000 7f68a0ff9700 1 --2- 172.21.15.67:0/3529327885 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f688c06ecc0 0x7f688c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:11.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.587+0000 7f68a0ff9700 1 -- 172.21.15.67:0/3529327885 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f68980903c0 con 0x7f68a4101060 2024-04-22T23:58:11.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.587+0000 7f68a37fe700 1 --2- 172.21.15.67:0/3529327885 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f688c06ecc0 0x7f688c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:11.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.588+0000 7f68a37fe700 1 --2- 172.21.15.67:0/3529327885 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f688c06ecc0 0x7f688c071180 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f6894005950 tx=0x7f689400b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:58:11.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.592+0000 7f68a0ff9700 1 -- 172.21.15.67:0/3529327885 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f6898059000 con 0x7f68a4101060 2024-04-22T23:58:11.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.847+0000 7f68aa212700 1 -- 172.21.15.67:0/3529327885 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f68a404f2e0 con 0x7f68a4101060 2024-04-22T23:58:11.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.847+0000 7f68a0ff9700 1 -- 172.21.15.67:0/3529327885 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1872 (secure 0 0 0) 0x7f689805c6c0 con 0x7f68a4101060 2024-04-22T23:58:11.838 INFO:teuthology.orchestra.run.smithi067.stdout:e33 2024-04-22T23:58:11.838 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:58:11.838 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:58:11.838 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:58:11.838 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:58:11.838 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:58:11.838 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:58:11.839 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:58:11.839 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:58:11.839 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:58:11.839 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:58:11.839 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:58:11.839 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:58:11.839 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:58:11.839 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:58:11.839 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:58:11.839 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:58:11.839 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:58:11.839 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:58:11.839 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:58:11.839 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:58:11.839 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:58:11.840 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:58:11.840 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:58:11.840 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:58:11.840 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:58:11.840 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:58:11.840 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:58:11.840 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:58:11.840 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:58:11.840 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:58:11.840 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:58:11.840 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:58:11.840 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:58:11.840 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:58:11.840 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:58:11.841 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:58:11.841 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:58:11.841 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:58:11.841 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:58:11.841 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:58:11.841 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44851} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1849648622,v1:172.21.15.134:6827/1849648622] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:58:11.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.850+0000 7f68aa212700 1 -- 172.21.15.67:0/3529327885 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f688c06ecc0 msgr2=0x7f688c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:11.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.850+0000 7f68aa212700 1 --2- 172.21.15.67:0/3529327885 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f688c06ecc0 0x7f688c071180 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f6894005950 tx=0x7f689400b410 comp rx=0 tx=0).stop 2024-04-22T23:58:11.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.850+0000 7f68aa212700 1 -- 172.21.15.67:0/3529327885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68a4101060 msgr2=0x7f68a4101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:11.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.850+0000 7f68aa212700 1 --2- 172.21.15.67:0/3529327885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68a4101060 0x7f68a4101d90 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f68980092e0 tx=0x7f689800bd70 comp rx=0 tx=0).stop 2024-04-22T23:58:11.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.850+0000 7f68aa212700 1 -- 172.21.15.67:0/3529327885 shutdown_connections 2024-04-22T23:58:11.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.850+0000 7f68aa212700 1 --2- 172.21.15.67:0/3529327885 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f688c06ecc0 0x7f688c071180 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:11.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.850+0000 7f68aa212700 1 --2- 172.21.15.67:0/3529327885 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f68a41006b0 0x7f68a4101850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:11.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.850+0000 7f68aa212700 1 --2- 172.21.15.67:0/3529327885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68a4101060 0x7f68a4101d90 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:11.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.850+0000 7f68aa212700 1 -- 172.21.15.67:0/3529327885 >> 172.21.15.67:0/3529327885 conn(0x7f68a4074e30 msgr2=0x7f68a4103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:58:11.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.850+0000 7f68aa212700 1 -- 172.21.15.67:0/3529327885 shutdown_connections 2024-04-22T23:58:11.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.850+0000 7f68aa212700 1 -- 172.21.15.67:0/3529327885 wait complete. 2024-04-22T23:58:11.842 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 33 2024-04-22T23:58:11.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.958+0000 7fe1c032e700 1 -- 172.21.15.67:0/730909040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b8102720 msgr2=0x7fe1b810ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:11.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.958+0000 7fe1c032e700 1 --2- 172.21.15.67:0/730909040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b8102720 0x7fe1b810ac10 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fe1a8009910 tx=0x7fe1a8009c20 comp rx=0 tx=0).stop 2024-04-22T23:58:11.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.958+0000 7fe1c032e700 1 -- 172.21.15.67:0/730909040 shutdown_connections 2024-04-22T23:58:11.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.958+0000 7fe1c032e700 1 --2- 172.21.15.67:0/730909040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b8102720 0x7fe1b810ac10 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:11.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.958+0000 7fe1c032e700 1 --2- 172.21.15.67:0/730909040 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1b8101e00 0x7fe1b81021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:11.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.958+0000 7fe1c032e700 1 -- 172.21.15.67:0/730909040 >> 172.21.15.67:0/730909040 conn(0x7fe1b80fb650 msgr2=0x7fe1b80fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:58:11.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.959+0000 7fe1c032e700 1 -- 172.21.15.67:0/730909040 shutdown_connections 2024-04-22T23:58:11.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.959+0000 7fe1c032e700 1 -- 172.21.15.67:0/730909040 wait complete. 2024-04-22T23:58:11.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.959+0000 7fe1c032e700 1 Processor -- start 2024-04-22T23:58:11.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.959+0000 7fe1c032e700 1 -- start start 2024-04-22T23:58:11.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.959+0000 7fe1c032e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b8101e00 0x7fe1b819c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:11.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.959+0000 7fe1c032e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1b8102720 0x7fe1b819cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:11.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.959+0000 7fe1c032e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b819d1d0 con 0x7fe1b8101e00 2024-04-22T23:58:11.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.959+0000 7fe1c032e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b8196710 con 0x7fe1b8102720 2024-04-22T23:58:11.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.960+0000 7fe1be0ca700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b8101e00 0x7fe1b819c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:11.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.960+0000 7fe1be0ca700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b8101e00 0x7fe1b819c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57458/0 (socket says 172.21.15.67:57458) 2024-04-22T23:58:11.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.960+0000 7fe1be0ca700 1 -- 172.21.15.67:0/805789939 learned_addr learned my addr 172.21.15.67:0/805789939 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:58:11.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.960+0000 7fe1bd8c9700 1 --2- 172.21.15.67:0/805789939 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1b8102720 0x7fe1b819cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:11.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.960+0000 7fe1be0ca700 1 -- 172.21.15.67:0/805789939 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1b8102720 msgr2=0x7fe1b819cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:11.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.960+0000 7fe1be0ca700 1 --2- 172.21.15.67:0/805789939 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1b8102720 0x7fe1b819cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:11.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.960+0000 7fe1be0ca700 1 -- 172.21.15.67:0/805789939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a80095f0 con 0x7fe1b8101e00 2024-04-22T23:58:11.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.960+0000 7fe1bd8c9700 1 --2- 172.21.15.67:0/805789939 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1b8102720 0x7fe1b819cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:58:11.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.960+0000 7fe1be0ca700 1 --2- 172.21.15.67:0/805789939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b8101e00 0x7fe1b819c690 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fe1b400c930 tx=0x7fe1b400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:58:11.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.960+0000 7fe1af7fe700 1 -- 172.21.15.67:0/805789939 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1b4007ad0 con 0x7fe1b8101e00 2024-04-22T23:58:11.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.960+0000 7fe1c032e700 1 -- 172.21.15.67:0/805789939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1b81969f0 con 0x7fe1b8101e00 2024-04-22T23:58:11.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.960+0000 7fe1af7fe700 1 -- 172.21.15.67:0/805789939 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe1b4007c30 con 0x7fe1b8101e00 2024-04-22T23:58:11.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.960+0000 7fe1af7fe700 1 -- 172.21.15.67:0/805789939 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1b4018610 con 0x7fe1b8101e00 2024-04-22T23:58:11.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.960+0000 7fe1c032e700 1 -- 172.21.15.67:0/805789939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1b8196f10 con 0x7fe1b8101e00 2024-04-22T23:58:11.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.961+0000 7fe1c032e700 1 -- 172.21.15.67:0/805789939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1b8108310 con 0x7fe1b8101e00 2024-04-22T23:58:11.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.962+0000 7fe1af7fe700 1 -- 172.21.15.67:0/805789939 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe1b400f450 con 0x7fe1b8101e00 2024-04-22T23:58:11.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.962+0000 7fe1af7fe700 1 --2- 172.21.15.67:0/805789939 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe1a406ecc0 0x7fe1a4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:11.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.962+0000 7fe1bd8c9700 1 --2- 172.21.15.67:0/805789939 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe1a406ecc0 0x7fe1a4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:11.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.962+0000 7fe1af7fe700 1 -- 172.21.15.67:0/805789939 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fe1b408f1f0 con 0x7fe1b8101e00 2024-04-22T23:58:11.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.963+0000 7fe1bd8c9700 1 --2- 172.21.15.67:0/805789939 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe1a406ecc0 0x7fe1a4071180 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fe1a8006010 tx=0x7fe1a801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:58:11.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:11.967+0000 7fe1af7fe700 1 -- 172.21.15.67:0/805789939 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe1b405b960 con 0x7fe1b8101e00 2024-04-22T23:58:12.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.198+0000 7fe1c032e700 1 -- 172.21.15.67:0/805789939 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe1b8197cc0 con 0x7fe1a406ecc0 2024-04-22T23:58:12.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.199+0000 7fe1af7fe700 1 -- 172.21.15.67:0/805789939 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe1b8197cc0 con 0x7fe1a406ecc0 2024-04-22T23:58:12.190 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:58:12.190 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:58:12.190 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:58:12.190 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:58:12.190 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:58:12.190 INFO:teuthology.orchestra.run.smithi067.stdout: "mds", 2024-04-22T23:58:12.190 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:58:12.190 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:58:12.190 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:58:12.191 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:58:12.191 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:58:12.191 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T23:58:12.191 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:58:12.191 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:58:12.191 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:58:12.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.201+0000 7fe1c032e700 1 -- 172.21.15.67:0/805789939 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe1a406ecc0 msgr2=0x7fe1a4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:12.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.201+0000 7fe1c032e700 1 --2- 172.21.15.67:0/805789939 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe1a406ecc0 0x7fe1a4071180 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fe1a8006010 tx=0x7fe1a801a040 comp rx=0 tx=0).stop 2024-04-22T23:58:12.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.201+0000 7fe1c032e700 1 -- 172.21.15.67:0/805789939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b8101e00 msgr2=0x7fe1b819c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:12.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.201+0000 7fe1c032e700 1 --2- 172.21.15.67:0/805789939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b8101e00 0x7fe1b819c690 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fe1b400c930 tx=0x7fe1b400ccf0 comp rx=0 tx=0).stop 2024-04-22T23:58:12.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.201+0000 7fe1c032e700 1 -- 172.21.15.67:0/805789939 shutdown_connections 2024-04-22T23:58:12.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.201+0000 7fe1c032e700 1 --2- 172.21.15.67:0/805789939 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe1a406ecc0 0x7fe1a4071180 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:12.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.201+0000 7fe1c032e700 1 --2- 172.21.15.67:0/805789939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b8101e00 0x7fe1b819c690 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:12.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.201+0000 7fe1c032e700 1 --2- 172.21.15.67:0/805789939 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe1b8102720 0x7fe1b819cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:12.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.201+0000 7fe1c032e700 1 -- 172.21.15.67:0/805789939 >> 172.21.15.67:0/805789939 conn(0x7fe1b80fb650 msgr2=0x7fe1b8105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:58:12.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.201+0000 7fe1c032e700 1 -- 172.21.15.67:0/805789939 shutdown_connections 2024-04-22T23:58:12.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.202+0000 7fe1c032e700 1 -- 172.21.15.67:0/805789939 wait complete. 2024-04-22T23:58:12.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.308+0000 7f101a088700 1 -- 172.21.15.67:0/4239723430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10140ffda0 msgr2=0x7f1014100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:12.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.308+0000 7f101a088700 1 --2- 172.21.15.67:0/4239723430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10140ffda0 0x7f1014100180 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f1004009910 tx=0x7f1004009c20 comp rx=0 tx=0).stop 2024-04-22T23:58:12.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.308+0000 7f101a088700 1 -- 172.21.15.67:0/4239723430 shutdown_connections 2024-04-22T23:58:12.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.308+0000 7f101a088700 1 --2- 172.21.15.67:0/4239723430 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1014100750 0x7f10141047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:12.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.308+0000 7f101a088700 1 --2- 172.21.15.67:0/4239723430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10140ffda0 0x7f1014100180 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:12.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.308+0000 7f101a088700 1 -- 172.21.15.67:0/4239723430 >> 172.21.15.67:0/4239723430 conn(0x7f10140fb650 msgr2=0x7f10140fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:58:12.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.308+0000 7f101a088700 1 -- 172.21.15.67:0/4239723430 shutdown_connections 2024-04-22T23:58:12.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.309+0000 7f101a088700 1 -- 172.21.15.67:0/4239723430 wait complete. 2024-04-22T23:58:12.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.309+0000 7f101a088700 1 Processor -- start 2024-04-22T23:58:12.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.309+0000 7f101a088700 1 -- start start 2024-04-22T23:58:12.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.309+0000 7f101a088700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10140ffda0 0x7f10141954e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:12.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.309+0000 7f101a088700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1014100750 0x7f1014195a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:12.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.309+0000 7f101a088700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1014195f60 con 0x7f10140ffda0 2024-04-22T23:58:12.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.309+0000 7f101a088700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10141960d0 con 0x7f1014100750 2024-04-22T23:58:12.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.310+0000 7f10137fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10140ffda0 0x7f10141954e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:12.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.310+0000 7f10137fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10140ffda0 0x7f10141954e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57484/0 (socket says 172.21.15.67:57484) 2024-04-22T23:58:12.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.310+0000 7f10137fe700 1 -- 172.21.15.67:0/2787731906 learned_addr learned my addr 172.21.15.67:0/2787731906 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:58:12.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.310+0000 7f1012ffd700 1 --2- 172.21.15.67:0/2787731906 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1014100750 0x7f1014195a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:12.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.310+0000 7f10137fe700 1 -- 172.21.15.67:0/2787731906 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1014100750 msgr2=0x7f1014195a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:12.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.310+0000 7f10137fe700 1 --2- 172.21.15.67:0/2787731906 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1014100750 0x7f1014195a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:12.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.310+0000 7f10137fe700 1 -- 172.21.15.67:0/2787731906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10040095f0 con 0x7f10140ffda0 2024-04-22T23:58:12.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.310+0000 7f1012ffd700 1 --2- 172.21.15.67:0/2787731906 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1014100750 0x7f1014195a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:58:12.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.310+0000 7f10137fe700 1 --2- 172.21.15.67:0/2787731906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10140ffda0 0x7f10141954e0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f1004006010 tx=0x7f100400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:58:12.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.310+0000 7f1010ff9700 1 -- 172.21.15.67:0/2787731906 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f100401d070 con 0x7f10140ffda0 2024-04-22T23:58:12.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.310+0000 7f1010ff9700 1 -- 172.21.15.67:0/2787731906 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f100400f460 con 0x7f10140ffda0 2024-04-22T23:58:12.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.310+0000 7f101a088700 1 -- 172.21.15.67:0/2787731906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1014196350 con 0x7f10140ffda0 2024-04-22T23:58:12.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.310+0000 7f1010ff9700 1 -- 172.21.15.67:0/2787731906 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1004017630 con 0x7f10140ffda0 2024-04-22T23:58:12.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.310+0000 7f101a088700 1 -- 172.21.15.67:0/2787731906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10141a2dd0 con 0x7f10140ffda0 2024-04-22T23:58:12.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.311+0000 7f101a088700 1 -- 172.21.15.67:0/2787731906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1014108140 con 0x7f10140ffda0 2024-04-22T23:58:12.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.312+0000 7f1010ff9700 1 -- 172.21.15.67:0/2787731906 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f100400f5d0 con 0x7f10140ffda0 2024-04-22T23:58:12.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.313+0000 7f1010ff9700 1 --2- 172.21.15.67:0/2787731906 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0ffc06ecc0 0x7f0ffc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:12.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.313+0000 7f1010ff9700 1 -- 172.21.15.67:0/2787731906 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f1004090260 con 0x7f10140ffda0 2024-04-22T23:58:12.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.313+0000 7f1012ffd700 1 --2- 172.21.15.67:0/2787731906 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0ffc06ecc0 0x7f0ffc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:12.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.313+0000 7f1012ffd700 1 --2- 172.21.15.67:0/2787731906 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0ffc06ecc0 0x7f0ffc071180 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f1008005950 tx=0x7f10080058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:58:12.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.318+0000 7f1010ff9700 1 -- 172.21.15.67:0/2787731906 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f100405ca50 con 0x7f10140ffda0 2024-04-22T23:58:12.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:12 smithi134 ceph-mon[85630]: from='client.35720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:58:12.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:12 smithi134 ceph-mon[85630]: from='client.35724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:58:12.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:12 smithi134 ceph-mon[85630]: from='client.35728 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:58:12.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:12 smithi134 ceph-mon[85630]: pgmap v956: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:12.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:12 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1019572239' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:58:12.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:12 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3529327885' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:58:12.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:12 smithi067 ceph-mon[112438]: from='client.35720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:58:12.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:12 smithi067 ceph-mon[112438]: from='client.35724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:58:12.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:12 smithi067 ceph-mon[112438]: from='client.35728 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:58:12.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:12 smithi067 ceph-mon[112438]: pgmap v956: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:12.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:12 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1019572239' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:58:12.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:12 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3529327885' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:58:12.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.615+0000 7f101a088700 1 -- 172.21.15.67:0/2787731906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f1014196350 con 0x7f10140ffda0 2024-04-22T23:58:12.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.615+0000 7f1010ff9700 1 -- 172.21.15.67:0/2787731906 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4505 (secure 0 0 0) 0x7f1014196350 con 0x7f10140ffda0 2024-04-22T23:58:12.606 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:58:12.606 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T23:58:12.606 INFO:teuthology.orchestra.run.smithi067.stdout: host smithi134 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-22T23:58:12.606 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T23:58:12.607 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:58:12.607 INFO:teuthology.orchestra.run.smithi067.stdout:Traceback (most recent call last): 2024-04-22T23:58:12.607 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T23:58:12.607 INFO:teuthology.orchestra.run.smithi067.stdout: return _run_code(code, main_globals, None, 2024-04-22T23:58:12.607 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T23:58:12.607 INFO:teuthology.orchestra.run.smithi067.stdout: exec(code, run_globals) 2024-04-22T23:58:12.607 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5642, in 2024-04-22T23:58:12.607 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5630, in main 2024-04-22T23:58:12.607 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 408, in _infer_config 2024-04-22T23:58:12.607 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 323, in _infer_fsid 2024-04-22T23:58:12.607 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 436, in _infer_image 2024-04-22T23:58:12.607 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 310, in _validate_fsid 2024-04-22T23:58:12.607 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 3380, in command_ceph_volume 2024-04-22T23:58:12.607 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T23:58:12.607 INFO:teuthology.orchestra.run.smithi067.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:58:12.608 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:58:12.608 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:58:12.608 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:58:12.608 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:58:12.608 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:58:12.608 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:58:12.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.617+0000 7f101a088700 1 -- 172.21.15.67:0/2787731906 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0ffc06ecc0 msgr2=0x7f0ffc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:12.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.617+0000 7f101a088700 1 --2- 172.21.15.67:0/2787731906 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0ffc06ecc0 0x7f0ffc071180 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f1008005950 tx=0x7f10080058e0 comp rx=0 tx=0).stop 2024-04-22T23:58:12.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.617+0000 7f101a088700 1 -- 172.21.15.67:0/2787731906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10140ffda0 msgr2=0x7f10141954e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:12.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.617+0000 7f101a088700 1 --2- 172.21.15.67:0/2787731906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10140ffda0 0x7f10141954e0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f1004006010 tx=0x7f100400be20 comp rx=0 tx=0).stop 2024-04-22T23:58:12.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.618+0000 7f101a088700 1 -- 172.21.15.67:0/2787731906 shutdown_connections 2024-04-22T23:58:12.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.618+0000 7f101a088700 1 --2- 172.21.15.67:0/2787731906 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0ffc06ecc0 0x7f0ffc071180 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:12.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.618+0000 7f101a088700 1 --2- 172.21.15.67:0/2787731906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10140ffda0 0x7f10141954e0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:12.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.618+0000 7f101a088700 1 --2- 172.21.15.67:0/2787731906 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1014100750 0x7f1014195a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:12.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.618+0000 7f101a088700 1 -- 172.21.15.67:0/2787731906 >> 172.21.15.67:0/2787731906 conn(0x7f10140fb650 msgr2=0x7f10140fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:58:12.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.618+0000 7f101a088700 1 -- 172.21.15.67:0/2787731906 shutdown_connections 2024-04-22T23:58:12.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:12.618+0000 7f101a088700 1 -- 172.21.15.67:0/2787731906 wait complete. 2024-04-22T23:58:13.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:13 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2787731906' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:58:13.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:13 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2787731906' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:58:14.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:14 smithi134 ceph-mon[85630]: from='client.35740 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:58:14.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:14 smithi134 ceph-mon[85630]: pgmap v957: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:14.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:14 smithi067 ceph-mon[112438]: from='client.35740 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:58:14.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:14 smithi067 ceph-mon[112438]: pgmap v957: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:16.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:16 smithi134 ceph-mon[85630]: pgmap v958: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:16.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:16 smithi067 ceph-mon[112438]: pgmap v958: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:18.337 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:18 smithi067 ceph-mon[112438]: pgmap v959: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:18.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:18 smithi134 ceph-mon[85630]: pgmap v959: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:20.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:20 smithi134 ceph-mon[85630]: pgmap v960: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:20.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:20 smithi067 ceph-mon[112438]: pgmap v960: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:21.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:21 smithi134 ceph-mon[85630]: pgmap v961: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:21.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:21 smithi067 ceph-mon[112438]: pgmap v961: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:23.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:23 smithi134 ceph-mon[85630]: pgmap v962: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:23.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:23 smithi067 ceph-mon[112438]: pgmap v962: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:25.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:25 smithi134 ceph-mon[85630]: pgmap v963: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:25.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:25 smithi067 ceph-mon[112438]: pgmap v963: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:26.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:58:26.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:58:27.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:27 smithi134 ceph-mon[85630]: pgmap v964: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:27.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:27 smithi067 ceph-mon[112438]: pgmap v964: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:29.284 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:29 smithi067 ceph-mon[112438]: pgmap v965: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:29.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:29 smithi134 ceph-mon[85630]: pgmap v965: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:31.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:31 smithi134 ceph-mon[85630]: pgmap v966: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:31.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:31 smithi067 ceph-mon[112438]: pgmap v966: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:33.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:33 smithi134 ceph-mon[85630]: pgmap v967: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:33.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:33 smithi067 ceph-mon[112438]: pgmap v967: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:35.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:35 smithi134 ceph-mon[85630]: pgmap v968: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:35.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:35 smithi067 ceph-mon[112438]: pgmap v968: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:37.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:37 smithi134 ceph-mon[85630]: pgmap v969: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:37.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:37 smithi067 ceph-mon[112438]: pgmap v969: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:39.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:39 smithi134 ceph-mon[85630]: pgmap v970: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:39.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:39 smithi067 ceph-mon[112438]: pgmap v970: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:41.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:58:41.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:58:42.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:42 smithi134 ceph-mon[85630]: pgmap v971: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:42.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:42 smithi067 ceph-mon[112438]: pgmap v971: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:42.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.738+0000 7f4d2aefd700 1 -- 172.21.15.67:0/3388224463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d24101850 msgr2=0x7f4d241058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:42.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.738+0000 7f4d2aefd700 1 --2- 172.21.15.67:0/3388224463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d24101850 0x7f4d241058a0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f4d14009910 tx=0x7f4d14009c20 comp rx=0 tx=0).stop 2024-04-22T23:58:42.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.738+0000 7f4d2aefd700 1 -- 172.21.15.67:0/3388224463 shutdown_connections 2024-04-22T23:58:42.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.738+0000 7f4d2aefd700 1 --2- 172.21.15.67:0/3388224463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d24101850 0x7f4d241058a0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:42.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.738+0000 7f4d2aefd700 1 --2- 172.21.15.67:0/3388224463 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4d24100ea0 0x7f4d24101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:42.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.738+0000 7f4d2aefd700 1 -- 172.21.15.67:0/3388224463 >> 172.21.15.67:0/3388224463 conn(0x7f4d240fc750 msgr2=0x7f4d240feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:58:42.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.738+0000 7f4d2aefd700 1 -- 172.21.15.67:0/3388224463 shutdown_connections 2024-04-22T23:58:42.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.739+0000 7f4d2aefd700 1 -- 172.21.15.67:0/3388224463 wait complete. 2024-04-22T23:58:42.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.739+0000 7f4d2aefd700 1 Processor -- start 2024-04-22T23:58:42.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.739+0000 7f4d2aefd700 1 -- start start 2024-04-22T23:58:42.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.739+0000 7f4d2aefd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d24100ea0 0x7f4d2419e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:42.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.739+0000 7f4d2aefd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4d24101850 0x7f4d2419eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:42.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.739+0000 7f4d2aefd700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2419f430 con 0x7f4d24100ea0 2024-04-22T23:58:42.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.739+0000 7f4d2aefd700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d241988e0 con 0x7f4d24101850 2024-04-22T23:58:42.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.740+0000 7f4d28c99700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d24100ea0 0x7f4d2419e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:42.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.740+0000 7f4d28c99700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d24100ea0 0x7f4d2419e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55742/0 (socket says 172.21.15.67:55742) 2024-04-22T23:58:42.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.740+0000 7f4d28c99700 1 -- 172.21.15.67:0/4223981308 learned_addr learned my addr 172.21.15.67:0/4223981308 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:58:42.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.740+0000 7f4d23fff700 1 --2- 172.21.15.67:0/4223981308 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4d24101850 0x7f4d2419eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:42.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.740+0000 7f4d28c99700 1 -- 172.21.15.67:0/4223981308 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4d24101850 msgr2=0x7f4d2419eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:42.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.740+0000 7f4d28c99700 1 --2- 172.21.15.67:0/4223981308 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4d24101850 0x7f4d2419eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:42.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.740+0000 7f4d28c99700 1 -- 172.21.15.67:0/4223981308 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d140095f0 con 0x7f4d24100ea0 2024-04-22T23:58:42.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.740+0000 7f4d28c99700 1 --2- 172.21.15.67:0/4223981308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d24100ea0 0x7f4d2419e860 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c00d900 tx=0x7f4d0c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:58:42.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.740+0000 7f4d21ffb700 1 -- 172.21.15.67:0/4223981308 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d0c0041f0 con 0x7f4d24100ea0 2024-04-22T23:58:42.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.740+0000 7f4d21ffb700 1 -- 172.21.15.67:0/4223981308 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4d0c004cf0 con 0x7f4d24100ea0 2024-04-22T23:58:42.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.740+0000 7f4d2aefd700 1 -- 172.21.15.67:0/4223981308 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d24198bc0 con 0x7f4d24100ea0 2024-04-22T23:58:42.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.740+0000 7f4d21ffb700 1 -- 172.21.15.67:0/4223981308 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d0c00b750 con 0x7f4d24100ea0 2024-04-22T23:58:42.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.741+0000 7f4d2aefd700 1 -- 172.21.15.67:0/4223981308 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d24199110 con 0x7f4d24100ea0 2024-04-22T23:58:42.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.742+0000 7f4d2aefd700 1 -- 172.21.15.67:0/4223981308 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d24105020 con 0x7f4d24100ea0 2024-04-22T23:58:42.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.742+0000 7f4d21ffb700 1 -- 172.21.15.67:0/4223981308 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4d0c004750 con 0x7f4d24100ea0 2024-04-22T23:58:42.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.743+0000 7f4d21ffb700 1 --2- 172.21.15.67:0/4223981308 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4d1006ecc0 0x7f4d10071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:42.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.743+0000 7f4d21ffb700 1 -- 172.21.15.67:0/4223981308 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f4d0c08eed0 con 0x7f4d24100ea0 2024-04-22T23:58:42.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.743+0000 7f4d23fff700 1 --2- 172.21.15.67:0/4223981308 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4d1006ecc0 0x7f4d10071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:42.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.743+0000 7f4d23fff700 1 --2- 172.21.15.67:0/4223981308 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4d1006ecc0 0x7f4d10071180 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f4d240fcee0 tx=0x7f4d1401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:58:42.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.747+0000 7f4d21ffb700 1 -- 172.21.15.67:0/4223981308 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f4d0c05b640 con 0x7f4d24100ea0 2024-04-22T23:58:42.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.979+0000 7f4d2aefd700 1 -- 172.21.15.67:0/4223981308 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4d24199330 con 0x7f4d1006ecc0 2024-04-22T23:58:42.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.980+0000 7f4d21ffb700 1 -- 172.21.15.67:0/4223981308 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f4d24199330 con 0x7f4d1006ecc0 2024-04-22T23:58:42.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.982+0000 7f4d2aefd700 1 -- 172.21.15.67:0/4223981308 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4d1006ecc0 msgr2=0x7f4d10071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:42.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.982+0000 7f4d2aefd700 1 --2- 172.21.15.67:0/4223981308 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4d1006ecc0 0x7f4d10071180 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f4d240fcee0 tx=0x7f4d1401a040 comp rx=0 tx=0).stop 2024-04-22T23:58:42.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.982+0000 7f4d2aefd700 1 -- 172.21.15.67:0/4223981308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d24100ea0 msgr2=0x7f4d2419e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:42.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.982+0000 7f4d2aefd700 1 --2- 172.21.15.67:0/4223981308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d24100ea0 0x7f4d2419e860 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c00d900 tx=0x7f4d0c00dc10 comp rx=0 tx=0).stop 2024-04-22T23:58:42.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.983+0000 7f4d2aefd700 1 -- 172.21.15.67:0/4223981308 shutdown_connections 2024-04-22T23:58:42.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.983+0000 7f4d2aefd700 1 --2- 172.21.15.67:0/4223981308 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4d1006ecc0 0x7f4d10071180 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:42.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.983+0000 7f4d2aefd700 1 --2- 172.21.15.67:0/4223981308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d24100ea0 0x7f4d2419e860 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:42.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.983+0000 7f4d2aefd700 1 --2- 172.21.15.67:0/4223981308 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4d24101850 0x7f4d2419eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:42.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.983+0000 7f4d2aefd700 1 -- 172.21.15.67:0/4223981308 >> 172.21.15.67:0/4223981308 conn(0x7f4d240fc750 msgr2=0x7f4d240fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:58:42.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.983+0000 7f4d2aefd700 1 -- 172.21.15.67:0/4223981308 shutdown_connections 2024-04-22T23:58:42.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:42.983+0000 7f4d2aefd700 1 -- 172.21.15.67:0/4223981308 wait complete. 2024-04-22T23:58:42.983 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:58:43.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.090+0000 7f0559598700 1 -- 172.21.15.67:0/3982269551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05540ffda0 msgr2=0x7f0554100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:43.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.090+0000 7f0559598700 1 --2- 172.21.15.67:0/3982269551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05540ffda0 0x7f0554100180 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f054c00b0a0 tx=0x7f054c00b3b0 comp rx=0 tx=0).stop 2024-04-22T23:58:43.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.090+0000 7f0559598700 1 -- 172.21.15.67:0/3982269551 shutdown_connections 2024-04-22T23:58:43.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.090+0000 7f0559598700 1 --2- 172.21.15.67:0/3982269551 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0554100750 0x7f05541047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:43.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.090+0000 7f0559598700 1 --2- 172.21.15.67:0/3982269551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05540ffda0 0x7f0554100180 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:43.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.091+0000 7f0559598700 1 -- 172.21.15.67:0/3982269551 >> 172.21.15.67:0/3982269551 conn(0x7f05540fb650 msgr2=0x7f05540fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:58:43.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.091+0000 7f0559598700 1 -- 172.21.15.67:0/3982269551 shutdown_connections 2024-04-22T23:58:43.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.091+0000 7f0559598700 1 -- 172.21.15.67:0/3982269551 wait complete. 2024-04-22T23:58:43.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.091+0000 7f0559598700 1 Processor -- start 2024-04-22T23:58:43.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.091+0000 7f0559598700 1 -- start start 2024-04-22T23:58:43.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.092+0000 7f0559598700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f05540ffda0 0x7f0554101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:43.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.092+0000 7f0559598700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0554100750 0x7f0554101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:43.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.092+0000 7f0559598700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05541022c0 con 0x7f0554100750 2024-04-22T23:58:43.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.092+0000 7f0559598700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0554102400 con 0x7f05540ffda0 2024-04-22T23:58:43.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.092+0000 7f05527fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0554100750 0x7f0554101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:43.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.092+0000 7f05527fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0554100750 0x7f0554101d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55750/0 (socket says 172.21.15.67:55750) 2024-04-22T23:58:43.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.092+0000 7f05527fc700 1 -- 172.21.15.67:0/3424746982 learned_addr learned my addr 172.21.15.67:0/3424746982 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:58:43.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.092+0000 7f0552ffd700 1 --2- 172.21.15.67:0/3424746982 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f05540ffda0 0x7f0554101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:43.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.092+0000 7f05527fc700 1 -- 172.21.15.67:0/3424746982 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f05540ffda0 msgr2=0x7f0554101840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:43.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.092+0000 7f05527fc700 1 --2- 172.21.15.67:0/3424746982 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f05540ffda0 0x7f0554101840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:43.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.092+0000 7f05527fc700 1 -- 172.21.15.67:0/3424746982 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054c009d00 con 0x7f0554100750 2024-04-22T23:58:43.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.092+0000 7f0552ffd700 1 --2- 172.21.15.67:0/3424746982 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f05540ffda0 0x7f0554101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:58:43.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.092+0000 7f05527fc700 1 --2- 172.21.15.67:0/3424746982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0554100750 0x7f0554101d80 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f053c00b480 tx=0x7f053c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:58:43.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.092+0000 7f054bfff700 1 -- 172.21.15.67:0/3424746982 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f053c0118c0 con 0x7f0554100750 2024-04-22T23:58:43.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.092+0000 7f054bfff700 1 -- 172.21.15.67:0/3424746982 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f053c010380 con 0x7f0554100750 2024-04-22T23:58:43.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.092+0000 7f0559598700 1 -- 172.21.15.67:0/3424746982 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05541a2c40 con 0x7f0554100750 2024-04-22T23:58:43.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.092+0000 7f054bfff700 1 -- 172.21.15.67:0/3424746982 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f053c00f4f0 con 0x7f0554100750 2024-04-22T23:58:43.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.093+0000 7f0559598700 1 -- 172.21.15.67:0/3424746982 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05541a3050 con 0x7f0554100750 2024-04-22T23:58:43.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.093+0000 7f0559598700 1 -- 172.21.15.67:0/3424746982 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f055404f2e0 con 0x7f0554100750 2024-04-22T23:58:43.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.094+0000 7f054bfff700 1 -- 172.21.15.67:0/3424746982 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f053c0104f0 con 0x7f0554100750 2024-04-22T23:58:43.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.094+0000 7f054bfff700 1 --2- 172.21.15.67:0/3424746982 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f054006ecc0 0x7f0540071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:43.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.095+0000 7f054bfff700 1 -- 172.21.15.67:0/3424746982 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f053c08eb40 con 0x7f0554100750 2024-04-22T23:58:43.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.095+0000 7f0552ffd700 1 --2- 172.21.15.67:0/3424746982 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f054006ecc0 0x7f0540071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:43.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.095+0000 7f0552ffd700 1 --2- 172.21.15.67:0/3424746982 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f054006ecc0 0x7f0540071180 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f054c009480 tx=0x7f054c000f40 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:58:43.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.099+0000 7f054bfff700 1 -- 172.21.15.67:0/3424746982 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f053c057800 con 0x7f0554100750 2024-04-22T23:58:43.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.331+0000 7f0559598700 1 -- 172.21.15.67:0/3424746982 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f05540fc560 con 0x7f054006ecc0 2024-04-22T23:58:43.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.332+0000 7f054bfff700 1 -- 172.21.15.67:0/3424746982 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f05540fc560 con 0x7f054006ecc0 2024-04-22T23:58:43.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.334+0000 7f0559598700 1 -- 172.21.15.67:0/3424746982 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f054006ecc0 msgr2=0x7f0540071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:43.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.334+0000 7f0559598700 1 --2- 172.21.15.67:0/3424746982 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f054006ecc0 0x7f0540071180 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f054c009480 tx=0x7f054c000f40 comp rx=0 tx=0).stop 2024-04-22T23:58:43.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.334+0000 7f0559598700 1 -- 172.21.15.67:0/3424746982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0554100750 msgr2=0x7f0554101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:43.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.334+0000 7f0559598700 1 --2- 172.21.15.67:0/3424746982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0554100750 0x7f0554101d80 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f053c00b480 tx=0x7f053c00b840 comp rx=0 tx=0).stop 2024-04-22T23:58:43.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.334+0000 7f0559598700 1 -- 172.21.15.67:0/3424746982 shutdown_connections 2024-04-22T23:58:43.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.334+0000 7f0559598700 1 --2- 172.21.15.67:0/3424746982 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f054006ecc0 0x7f0540071180 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:43.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.334+0000 7f0559598700 1 --2- 172.21.15.67:0/3424746982 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f05540ffda0 0x7f0554101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:43.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.334+0000 7f0559598700 1 --2- 172.21.15.67:0/3424746982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0554100750 0x7f0554101d80 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:43.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.334+0000 7f0559598700 1 -- 172.21.15.67:0/3424746982 >> 172.21.15.67:0/3424746982 conn(0x7f05540fb650 msgr2=0x7f0554104090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:58:43.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.334+0000 7f0559598700 1 -- 172.21.15.67:0/3424746982 shutdown_connections 2024-04-22T23:58:43.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.334+0000 7f0559598700 1 -- 172.21.15.67:0/3424746982 wait complete. 2024-04-22T23:58:43.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.441+0000 7fcbf4ed4700 1 -- 172.21.15.67:0/347864045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbf0101e00 msgr2=0x7fcbf01021e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:43.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.441+0000 7fcbf4ed4700 1 --2- 172.21.15.67:0/347864045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbf0101e00 0x7fcbf01021e0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0009910 tx=0x7fcbe0009c20 comp rx=0 tx=0).stop 2024-04-22T23:58:43.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.441+0000 7fcbf4ed4700 1 -- 172.21.15.67:0/347864045 shutdown_connections 2024-04-22T23:58:43.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.441+0000 7fcbf4ed4700 1 --2- 172.21.15.67:0/347864045 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcbf0102720 0x7fcbf010ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:43.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.441+0000 7fcbf4ed4700 1 --2- 172.21.15.67:0/347864045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbf0101e00 0x7fcbf01021e0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:43.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.441+0000 7fcbf4ed4700 1 -- 172.21.15.67:0/347864045 >> 172.21.15.67:0/347864045 conn(0x7fcbf00fb650 msgr2=0x7fcbf00fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:58:43.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.441+0000 7fcbf4ed4700 1 -- 172.21.15.67:0/347864045 shutdown_connections 2024-04-22T23:58:43.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.441+0000 7fcbf4ed4700 1 -- 172.21.15.67:0/347864045 wait complete. 2024-04-22T23:58:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.442+0000 7fcbf4ed4700 1 Processor -- start 2024-04-22T23:58:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.442+0000 7fcbf4ed4700 1 -- start start 2024-04-22T23:58:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.442+0000 7fcbf4ed4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbf0101e00 0x7fcbf00ffa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.442+0000 7fcbf4ed4700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcbf0102720 0x7fcbf00fffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.442+0000 7fcbf4ed4700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf01004e0 con 0x7fcbf0101e00 2024-04-22T23:58:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.442+0000 7fcbf4ed4700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf0100620 con 0x7fcbf0102720 2024-04-22T23:58:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.442+0000 7fcbee59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbf0101e00 0x7fcbf00ffa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.442+0000 7fcbee59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbf0101e00 0x7fcbf00ffa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55774/0 (socket says 172.21.15.67:55774) 2024-04-22T23:58:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.442+0000 7fcbee59c700 1 -- 172.21.15.67:0/2772577258 learned_addr learned my addr 172.21.15.67:0/2772577258 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:58:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.442+0000 7fcbedd9b700 1 --2- 172.21.15.67:0/2772577258 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcbf0102720 0x7fcbf00fffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:43.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.443+0000 7fcbee59c700 1 -- 172.21.15.67:0/2772577258 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcbf0102720 msgr2=0x7fcbf00fffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:43.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.443+0000 7fcbee59c700 1 --2- 172.21.15.67:0/2772577258 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcbf0102720 0x7fcbf00fffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:43.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.443+0000 7fcbee59c700 1 -- 172.21.15.67:0/2772577258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe00095f0 con 0x7fcbf0101e00 2024-04-22T23:58:43.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.443+0000 7fcbee59c700 1 --2- 172.21.15.67:0/2772577258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbf0101e00 0x7fcbf00ffa60 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0006010 tx=0x7fcbe000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:58:43.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.443+0000 7fcbdf7fe700 1 -- 172.21.15.67:0/2772577258 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcbe001d070 con 0x7fcbf0101e00 2024-04-22T23:58:43.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.443+0000 7fcbdf7fe700 1 -- 172.21.15.67:0/2772577258 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcbe0005dc0 con 0x7fcbf0101e00 2024-04-22T23:58:43.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.443+0000 7fcbf4ed4700 1 -- 172.21.15.67:0/2772577258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbf01008a0 con 0x7fcbf0101e00 2024-04-22T23:58:43.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.443+0000 7fcbdf7fe700 1 -- 172.21.15.67:0/2772577258 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcbe0017400 con 0x7fcbf0101e00 2024-04-22T23:58:43.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.443+0000 7fcbf4ed4700 1 -- 172.21.15.67:0/2772577258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbf01a2dc0 con 0x7fcbf0101e00 2024-04-22T23:58:43.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.444+0000 7fcbf4ed4700 1 -- 172.21.15.67:0/2772577258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbf0108310 con 0x7fcbf0101e00 2024-04-22T23:58:43.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.444+0000 7fcbdf7fe700 1 -- 172.21.15.67:0/2772577258 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcbe0003a60 con 0x7fcbf0101e00 2024-04-22T23:58:43.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.445+0000 7fcbdf7fe700 1 --2- 172.21.15.67:0/2772577258 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcbd806ecc0 0x7fcbd8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:43.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.445+0000 7fcbdf7fe700 1 -- 172.21.15.67:0/2772577258 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fcbe0090370 con 0x7fcbf0101e00 2024-04-22T23:58:43.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.445+0000 7fcbedd9b700 1 --2- 172.21.15.67:0/2772577258 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcbd806ecc0 0x7fcbd8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:43.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.445+0000 7fcbedd9b700 1 --2- 172.21.15.67:0/2772577258 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcbd806ecc0 0x7fcbd8071180 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fcbf0101250 tx=0x7fcbe4007400 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:58:43.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.450+0000 7fcbdf7fe700 1 -- 172.21.15.67:0/2772577258 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fcbe0059030 con 0x7fcbf0101e00 2024-04-22T23:58:43.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.670+0000 7fcbf4ed4700 1 -- 172.21.15.67:0/2772577258 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fcbf00611d0 con 0x7fcbd806ecc0 2024-04-22T23:58:43.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.682+0000 7fcbdf7fe700 1 -- 172.21.15.67:0/2772577258 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fcbf00611d0 con 0x7fcbd806ecc0 2024-04-22T23:58:43.672 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:58:43.672 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (32m) 10m ago 43m 24.4M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:58:43.672 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (43m) 10m ago 43m 10.0M - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:58:43.672 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (42m) 10m ago 42m 10.0M - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:58:43.673 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (29m) 10m ago 43m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:58:43.673 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (29m) 10m ago 42m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:58:43.673 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (32m) 10m ago 43m 81.2M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:58:43.673 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (26m) 10m ago 38m 19.4M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:58:43.673 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (25m) 10m ago 38m 20.3M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:58:43.673 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (25m) 10m ago 38m 20.1M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:58:43.673 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (10m) 10m ago 38m 14.0M - 19.0.0-3252-g43be0201 a0434699c1d7 46d3d7e86fee 2024-04-22T23:58:43.673 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (34m) 10m ago 45m 514M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:58:43.673 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (33m) 10m ago 42m 459M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:58:43.673 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (30m) 10m ago 45m 74.2M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:58:43.673 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (30m) 10m ago 42m 54.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:58:43.673 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (33m) 10m ago 43m 15.2M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:58:43.673 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (33m) 10m ago 42m 16.0M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:58:43.674 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (29m) 10m ago 41m 102M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:58:43.674 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (28m) 10m ago 41m 128M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:58:43.674 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (28m) 10m ago 41m 83.8M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:58:43.674 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (27m) 10m ago 40m 101M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:58:43.674 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (27m) 10m ago 40m 86.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:58:43.674 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (26m) 10m ago 39m 81.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:58:43.674 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (32m) 10m ago 43m 78.7M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:58:43.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.684+0000 7fcbf4ed4700 1 -- 172.21.15.67:0/2772577258 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcbd806ecc0 msgr2=0x7fcbd8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:43.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.684+0000 7fcbf4ed4700 1 --2- 172.21.15.67:0/2772577258 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcbd806ecc0 0x7fcbd8071180 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fcbf0101250 tx=0x7fcbe4007400 comp rx=0 tx=0).stop 2024-04-22T23:58:43.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.684+0000 7fcbf4ed4700 1 -- 172.21.15.67:0/2772577258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbf0101e00 msgr2=0x7fcbf00ffa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:43.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.684+0000 7fcbf4ed4700 1 --2- 172.21.15.67:0/2772577258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbf0101e00 0x7fcbf00ffa60 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0006010 tx=0x7fcbe000bd70 comp rx=0 tx=0).stop 2024-04-22T23:58:43.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.684+0000 7fcbf4ed4700 1 -- 172.21.15.67:0/2772577258 shutdown_connections 2024-04-22T23:58:43.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.684+0000 7fcbf4ed4700 1 --2- 172.21.15.67:0/2772577258 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcbd806ecc0 0x7fcbd8071180 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:43.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.684+0000 7fcbf4ed4700 1 --2- 172.21.15.67:0/2772577258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbf0101e00 0x7fcbf00ffa60 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:43.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.684+0000 7fcbf4ed4700 1 --2- 172.21.15.67:0/2772577258 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcbf0102720 0x7fcbf00fffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:43.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.684+0000 7fcbf4ed4700 1 -- 172.21.15.67:0/2772577258 >> 172.21.15.67:0/2772577258 conn(0x7fcbf00fb650 msgr2=0x7fcbf0105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:58:43.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.684+0000 7fcbf4ed4700 1 -- 172.21.15.67:0/2772577258 shutdown_connections 2024-04-22T23:58:43.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.684+0000 7fcbf4ed4700 1 -- 172.21.15.67:0/2772577258 wait complete. 2024-04-22T23:58:43.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.789+0000 7f9cbd4db700 1 -- 172.21.15.67:0/4058545297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cb8102720 msgr2=0x7f9cb810ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:43.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.789+0000 7f9cbd4db700 1 --2- 172.21.15.67:0/4058545297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cb8102720 0x7f9cb810ac10 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f9ca8009910 tx=0x7f9ca8009c20 comp rx=0 tx=0).stop 2024-04-22T23:58:43.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.789+0000 7f9cbd4db700 1 -- 172.21.15.67:0/4058545297 shutdown_connections 2024-04-22T23:58:43.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.789+0000 7f9cbd4db700 1 --2- 172.21.15.67:0/4058545297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cb8102720 0x7f9cb810ac10 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:43.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.789+0000 7f9cbd4db700 1 --2- 172.21.15.67:0/4058545297 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9cb8101e00 0x7f9cb81021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:43.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.789+0000 7f9cbd4db700 1 -- 172.21.15.67:0/4058545297 >> 172.21.15.67:0/4058545297 conn(0x7f9cb80fb650 msgr2=0x7f9cb80fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:58:43.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.789+0000 7f9cbd4db700 1 -- 172.21.15.67:0/4058545297 shutdown_connections 2024-04-22T23:58:43.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.789+0000 7f9cbd4db700 1 -- 172.21.15.67:0/4058545297 wait complete. 2024-04-22T23:58:43.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.790+0000 7f9cbd4db700 1 Processor -- start 2024-04-22T23:58:43.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.790+0000 7f9cbd4db700 1 -- start start 2024-04-22T23:58:43.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.790+0000 7f9cbd4db700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9cb8101e00 0x7f9cb819c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:43.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.790+0000 7f9cbd4db700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cb8102720 0x7f9cb819cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:43.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.790+0000 7f9cbd4db700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb819d400 con 0x7f9cb8102720 2024-04-22T23:58:43.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.790+0000 7f9cbd4db700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb81968b0 con 0x7f9cb8101e00 2024-04-22T23:58:43.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.790+0000 7f9cb67fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cb8102720 0x7f9cb819cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:43.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.790+0000 7f9cb67fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cb8102720 0x7f9cb819cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55792/0 (socket says 172.21.15.67:55792) 2024-04-22T23:58:43.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.790+0000 7f9cb67fc700 1 -- 172.21.15.67:0/2730764218 learned_addr learned my addr 172.21.15.67:0/2730764218 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:58:43.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.791+0000 7f9cb6ffd700 1 --2- 172.21.15.67:0/2730764218 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9cb8101e00 0x7f9cb819c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:43.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.791+0000 7f9cb67fc700 1 -- 172.21.15.67:0/2730764218 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9cb8101e00 msgr2=0x7f9cb819c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:43.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.791+0000 7f9cb67fc700 1 --2- 172.21.15.67:0/2730764218 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9cb8101e00 0x7f9cb819c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:43.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.791+0000 7f9cb67fc700 1 -- 172.21.15.67:0/2730764218 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca80095f0 con 0x7f9cb8102720 2024-04-22T23:58:43.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.791+0000 7f9cb6ffd700 1 --2- 172.21.15.67:0/2730764218 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9cb8101e00 0x7f9cb819c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:58:43.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.791+0000 7f9cb67fc700 1 --2- 172.21.15.67:0/2730764218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cb8102720 0x7f9cb819cd70 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f9ca800bd70 tx=0x7f9ca800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:58:43.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.791+0000 7f9c9ffff700 1 -- 172.21.15.67:0/2730764218 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ca801d070 con 0x7f9cb8102720 2024-04-22T23:58:43.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.791+0000 7f9cbd4db700 1 -- 172.21.15.67:0/2730764218 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cb8196b30 con 0x7f9cb8102720 2024-04-22T23:58:43.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.791+0000 7f9c9ffff700 1 -- 172.21.15.67:0/2730764218 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ca800f460 con 0x7f9cb8102720 2024-04-22T23:58:43.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.791+0000 7f9cbd4db700 1 -- 172.21.15.67:0/2730764218 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cb8197020 con 0x7f9cb8102720 2024-04-22T23:58:43.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.791+0000 7f9c9ffff700 1 -- 172.21.15.67:0/2730764218 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ca8017630 con 0x7f9cb8102720 2024-04-22T23:58:43.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.792+0000 7f9cbd4db700 1 -- 172.21.15.67:0/2730764218 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ca4005320 con 0x7f9cb8102720 2024-04-22T23:58:43.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.793+0000 7f9c9ffff700 1 -- 172.21.15.67:0/2730764218 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9ca8003a60 con 0x7f9cb8102720 2024-04-22T23:58:43.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.793+0000 7f9c9ffff700 1 --2- 172.21.15.67:0/2730764218 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9ca006ecc0 0x7f9ca0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:43.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.793+0000 7f9c9ffff700 1 -- 172.21.15.67:0/2730764218 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f9ca8090340 con 0x7f9cb8102720 2024-04-22T23:58:43.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.793+0000 7f9cb6ffd700 1 --2- 172.21.15.67:0/2730764218 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9ca006ecc0 0x7f9ca0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:43.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.794+0000 7f9cb6ffd700 1 --2- 172.21.15.67:0/2730764218 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9ca006ecc0 0x7f9ca0071180 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f9cac007900 tx=0x7f9cac008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:58:43.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:43.798+0000 7f9c9ffff700 1 -- 172.21.15.67:0/2730764218 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f9ca805cb30 con 0x7f9cb8102720 2024-04-22T23:58:44.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.094+0000 7f9cbd4db700 1 -- 172.21.15.67:0/2730764218 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9ca4006200 con 0x7f9cb8102720 2024-04-22T23:58:44.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.095+0000 7f9c9ffff700 1 -- 172.21.15.67:0/2730764218 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f9ca805c6c0 con 0x7f9cb8102720 2024-04-22T23:58:44.086 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:58:44.086 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:58:44.086 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:58:44.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:58:44.086 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:58:44.086 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:58:44.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:58:44.086 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:58:44.086 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:58:44.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:58:44.086 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:58:44.086 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 4 2024-04-22T23:58:44.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:58:44.086 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:58:44.087 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 14 2024-04-22T23:58:44.087 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:58:44.087 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:58:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.097+0000 7f9cbd4db700 1 -- 172.21.15.67:0/2730764218 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9ca006ecc0 msgr2=0x7f9ca0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.097+0000 7f9cbd4db700 1 --2- 172.21.15.67:0/2730764218 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9ca006ecc0 0x7f9ca0071180 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f9cac007900 tx=0x7f9cac008040 comp rx=0 tx=0).stop 2024-04-22T23:58:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.097+0000 7f9cbd4db700 1 -- 172.21.15.67:0/2730764218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cb8102720 msgr2=0x7f9cb819cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.097+0000 7f9cbd4db700 1 --2- 172.21.15.67:0/2730764218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cb8102720 0x7f9cb819cd70 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f9ca800bd70 tx=0x7f9ca800be50 comp rx=0 tx=0).stop 2024-04-22T23:58:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.097+0000 7f9cbd4db700 1 -- 172.21.15.67:0/2730764218 shutdown_connections 2024-04-22T23:58:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.097+0000 7f9cbd4db700 1 --2- 172.21.15.67:0/2730764218 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9ca006ecc0 0x7f9ca0071180 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.097+0000 7f9cbd4db700 1 --2- 172.21.15.67:0/2730764218 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9cb8101e00 0x7f9cb819c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.097+0000 7f9cbd4db700 1 --2- 172.21.15.67:0/2730764218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cb8102720 0x7f9cb819cd70 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.097+0000 7f9cbd4db700 1 -- 172.21.15.67:0/2730764218 >> 172.21.15.67:0/2730764218 conn(0x7f9cb80fb650 msgr2=0x7f9cb8105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:58:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.097+0000 7f9cbd4db700 1 -- 172.21.15.67:0/2730764218 shutdown_connections 2024-04-22T23:58:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.097+0000 7f9cbd4db700 1 -- 172.21.15.67:0/2730764218 wait complete. 2024-04-22T23:58:44.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.205+0000 7f4b225c2700 1 -- 172.21.15.67:0/3332474208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b1c068ea0 msgr2=0x7f4b1c10d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:44.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.205+0000 7f4b225c2700 1 --2- 172.21.15.67:0/3332474208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b1c068ea0 0x7f4b1c10d290 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f4b10009910 tx=0x7f4b10009c20 comp rx=0 tx=0).stop 2024-04-22T23:58:44.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.205+0000 7f4b225c2700 1 -- 172.21.15.67:0/3332474208 shutdown_connections 2024-04-22T23:58:44.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.205+0000 7f4b225c2700 1 --2- 172.21.15.67:0/3332474208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b1c068ea0 0x7f4b1c10d290 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:44.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.205+0000 7f4b225c2700 1 --2- 172.21.15.67:0/3332474208 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b1c068580 0x7f4b1c068960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:44.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.205+0000 7f4b225c2700 1 -- 172.21.15.67:0/3332474208 >> 172.21.15.67:0/3332474208 conn(0x7f4b1c075670 msgr2=0x7f4b1c075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:58:44.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.205+0000 7f4b225c2700 1 -- 172.21.15.67:0/3332474208 shutdown_connections 2024-04-22T23:58:44.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.205+0000 7f4b225c2700 1 -- 172.21.15.67:0/3332474208 wait complete. 2024-04-22T23:58:44.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.206+0000 7f4b225c2700 1 Processor -- start 2024-04-22T23:58:44.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.206+0000 7f4b225c2700 1 -- start start 2024-04-22T23:58:44.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.206+0000 7f4b225c2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b1c068580 0x7f4b1c19e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:44.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.206+0000 7f4b225c2700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b1c068ea0 0x7f4b1c19ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:44.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.206+0000 7f4b225c2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1c19f4b0 con 0x7f4b1c068580 2024-04-22T23:58:44.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.206+0000 7f4b225c2700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1c198960 con 0x7f4b1c068ea0 2024-04-22T23:58:44.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.206+0000 7f4b1bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b1c068580 0x7f4b1c19e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:44.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.206+0000 7f4b1bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b1c068580 0x7f4b1c19e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55804/0 (socket says 172.21.15.67:55804) 2024-04-22T23:58:44.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.206+0000 7f4b1bfff700 1 -- 172.21.15.67:0/2333079832 learned_addr learned my addr 172.21.15.67:0/2333079832 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:58:44.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.206+0000 7f4b1b7fe700 1 --2- 172.21.15.67:0/2333079832 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b1c068ea0 0x7f4b1c19ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:44.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.207+0000 7f4b1bfff700 1 -- 172.21.15.67:0/2333079832 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b1c068ea0 msgr2=0x7f4b1c19ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:44.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.207+0000 7f4b1bfff700 1 --2- 172.21.15.67:0/2333079832 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b1c068ea0 0x7f4b1c19ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:44.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.207+0000 7f4b1bfff700 1 -- 172.21.15.67:0/2333079832 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b100095f0 con 0x7f4b1c068580 2024-04-22T23:58:44.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.207+0000 7f4b1bfff700 1 --2- 172.21.15.67:0/2333079832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b1c068580 0x7f4b1c19e8e0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c00d8d0 tx=0x7f4b0c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:58:44.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.207+0000 7f4b197fa700 1 -- 172.21.15.67:0/2333079832 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b0c0098a0 con 0x7f4b1c068580 2024-04-22T23:58:44.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.207+0000 7f4b197fa700 1 -- 172.21.15.67:0/2333079832 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4b0c010460 con 0x7f4b1c068580 2024-04-22T23:58:44.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.207+0000 7f4b225c2700 1 -- 172.21.15.67:0/2333079832 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b1c198c40 con 0x7f4b1c068580 2024-04-22T23:58:44.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.207+0000 7f4b197fa700 1 -- 172.21.15.67:0/2333079832 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b0c00f5f0 con 0x7f4b1c068580 2024-04-22T23:58:44.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.207+0000 7f4b225c2700 1 -- 172.21.15.67:0/2333079832 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b1c199190 con 0x7f4b1c068580 2024-04-22T23:58:44.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.208+0000 7f4b225c2700 1 -- 172.21.15.67:0/2333079832 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b1c10aa00 con 0x7f4b1c068580 2024-04-22T23:58:44.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.209+0000 7f4b197fa700 1 -- 172.21.15.67:0/2333079832 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4b0c009a00 con 0x7f4b1c068580 2024-04-22T23:58:44.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.209+0000 7f4b197fa700 1 --2- 172.21.15.67:0/2333079832 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4b0406ecc0 0x7f4b04071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:44.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.209+0000 7f4b197fa700 1 -- 172.21.15.67:0/2333079832 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f4b0c08eea0 con 0x7f4b1c068580 2024-04-22T23:58:44.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.209+0000 7f4b1b7fe700 1 --2- 172.21.15.67:0/2333079832 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4b0406ecc0 0x7f4b04071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:44.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.210+0000 7f4b1b7fe700 1 --2- 172.21.15.67:0/2333079832 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4b0406ecc0 0x7f4b04071180 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c101400 tx=0x7f4b1001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:58:44.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.214+0000 7f4b197fa700 1 -- 172.21.15.67:0/2333079832 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f4b0c05b610 con 0x7f4b1c068580 2024-04-22T23:58:44.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.470+0000 7f4b225c2700 1 -- 172.21.15.67:0/2333079832 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4b1c04f2e0 con 0x7f4b1c068580 2024-04-22T23:58:44.461 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:44 smithi067 ceph-mon[112438]: from='client.35748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:58:44.461 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:44 smithi067 ceph-mon[112438]: pgmap v972: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:44.461 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:44 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2730764218' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:58:44.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.470+0000 7f4b197fa700 1 -- 172.21.15.67:0/2333079832 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1872 (secure 0 0 0) 0x7f4b0c05b1a0 con 0x7f4b1c068580 2024-04-22T23:58:44.461 INFO:teuthology.orchestra.run.smithi067.stdout:e33 2024-04-22T23:58:44.461 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:58:44.461 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:58:44.462 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:58:44.462 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:58:44.462 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:58:44.462 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:58:44.462 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:58:44.462 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:58:44.462 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:58:44.462 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:58:44.462 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:58:44.462 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:58:44.462 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:58:44.463 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:58:44.463 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:58:44.463 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:58:44.463 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:58:44.463 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:58:44.463 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:58:44.463 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:58:44.463 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:58:44.463 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:58:44.463 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:58:44.463 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:58:44.463 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:58:44.463 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:58:44.463 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:58:44.464 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:58:44.464 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:58:44.464 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:58:44.464 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:58:44.464 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:58:44.464 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:58:44.464 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:58:44.464 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:58:44.464 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:58:44.464 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:58:44.464 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:58:44.464 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:58:44.464 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:58:44.464 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44851} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1849648622,v1:172.21.15.134:6827/1849648622] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:58:44.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.472+0000 7f4b225c2700 1 -- 172.21.15.67:0/2333079832 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4b0406ecc0 msgr2=0x7f4b04071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:44.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.472+0000 7f4b225c2700 1 --2- 172.21.15.67:0/2333079832 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4b0406ecc0 0x7f4b04071180 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c101400 tx=0x7f4b1001a040 comp rx=0 tx=0).stop 2024-04-22T23:58:44.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.472+0000 7f4b225c2700 1 -- 172.21.15.67:0/2333079832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b1c068580 msgr2=0x7f4b1c19e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:44.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.472+0000 7f4b225c2700 1 --2- 172.21.15.67:0/2333079832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b1c068580 0x7f4b1c19e8e0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c00d8d0 tx=0x7f4b0c00dbe0 comp rx=0 tx=0).stop 2024-04-22T23:58:44.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.472+0000 7f4b225c2700 1 -- 172.21.15.67:0/2333079832 shutdown_connections 2024-04-22T23:58:44.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.472+0000 7f4b225c2700 1 --2- 172.21.15.67:0/2333079832 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4b0406ecc0 0x7f4b04071180 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:44.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.472+0000 7f4b225c2700 1 --2- 172.21.15.67:0/2333079832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b1c068580 0x7f4b1c19e8e0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:44.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.472+0000 7f4b225c2700 1 --2- 172.21.15.67:0/2333079832 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b1c068ea0 0x7f4b1c19ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:44.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.472+0000 7f4b225c2700 1 -- 172.21.15.67:0/2333079832 >> 172.21.15.67:0/2333079832 conn(0x7f4b1c075670 msgr2=0x7f4b1c0fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:58:44.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.472+0000 7f4b225c2700 1 -- 172.21.15.67:0/2333079832 shutdown_connections 2024-04-22T23:58:44.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.472+0000 7f4b225c2700 1 -- 172.21.15.67:0/2333079832 wait complete. 2024-04-22T23:58:44.466 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 33 2024-04-22T23:58:44.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:44 smithi134 ceph-mon[85630]: from='client.35748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:58:44.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:44 smithi134 ceph-mon[85630]: pgmap v972: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:44.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:44 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2730764218' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:58:44.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.580+0000 7f5eb9bde700 1 -- 172.21.15.67:0/497561318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5eb40ffb40 msgr2=0x7f5eb40fffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:44.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.580+0000 7f5eb9bde700 1 --2- 172.21.15.67:0/497561318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5eb40ffb40 0x7f5eb40fffc0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f5ea8009910 tx=0x7f5ea8009c20 comp rx=0 tx=0).stop 2024-04-22T23:58:44.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.580+0000 7f5eb9bde700 1 -- 172.21.15.67:0/497561318 shutdown_connections 2024-04-22T23:58:44.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.580+0000 7f5eb9bde700 1 --2- 172.21.15.67:0/497561318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5eb40ffb40 0x7f5eb40fffc0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:44.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.580+0000 7f5eb9bde700 1 --2- 172.21.15.67:0/497561318 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5eb4105bf0 0x7f5eb4105fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:44.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.580+0000 7f5eb9bde700 1 -- 172.21.15.67:0/497561318 >> 172.21.15.67:0/497561318 conn(0x7f5eb40fb650 msgr2=0x7f5eb40fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:58:44.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.580+0000 7f5eb9bde700 1 -- 172.21.15.67:0/497561318 shutdown_connections 2024-04-22T23:58:44.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.580+0000 7f5eb9bde700 1 -- 172.21.15.67:0/497561318 wait complete. 2024-04-22T23:58:44.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.580+0000 7f5eb9bde700 1 Processor -- start 2024-04-22T23:58:44.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.581+0000 7f5eb9bde700 1 -- start start 2024-04-22T23:58:44.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.581+0000 7f5eb9bde700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5eb40ffb40 0x7f5eb419c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:44.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.581+0000 7f5eb9bde700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5eb4105bf0 0x7f5eb419cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:44.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.581+0000 7f5eb9bde700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb419d190 con 0x7f5eb40ffb40 2024-04-22T23:58:44.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.581+0000 7f5eb9bde700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb4196640 con 0x7f5eb4105bf0 2024-04-22T23:58:44.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.581+0000 7f5eb37fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5eb40ffb40 0x7f5eb419c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:44.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.581+0000 7f5eb37fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5eb40ffb40 0x7f5eb419c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55832/0 (socket says 172.21.15.67:55832) 2024-04-22T23:58:44.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.581+0000 7f5eb37fe700 1 -- 172.21.15.67:0/903448259 learned_addr learned my addr 172.21.15.67:0/903448259 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:58:44.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.581+0000 7f5eb2ffd700 1 --2- 172.21.15.67:0/903448259 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5eb4105bf0 0x7f5eb419cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:44.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.581+0000 7f5eb37fe700 1 -- 172.21.15.67:0/903448259 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5eb4105bf0 msgr2=0x7f5eb419cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:44.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.581+0000 7f5eb37fe700 1 --2- 172.21.15.67:0/903448259 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5eb4105bf0 0x7f5eb419cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:44.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.581+0000 7f5eb37fe700 1 -- 172.21.15.67:0/903448259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea80095f0 con 0x7f5eb40ffb40 2024-04-22T23:58:44.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.581+0000 7f5eb2ffd700 1 --2- 172.21.15.67:0/903448259 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5eb4105bf0 0x7f5eb419cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:58:44.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.582+0000 7f5eb37fe700 1 --2- 172.21.15.67:0/903448259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5eb40ffb40 0x7f5eb419c5c0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f5ea400d8d0 tx=0x7f5ea400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:58:44.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.582+0000 7f5eb0ff9700 1 -- 172.21.15.67:0/903448259 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ea4009940 con 0x7f5eb40ffb40 2024-04-22T23:58:44.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.582+0000 7f5eb9bde700 1 -- 172.21.15.67:0/903448259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eb4196920 con 0x7f5eb40ffb40 2024-04-22T23:58:44.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.582+0000 7f5eb0ff9700 1 -- 172.21.15.67:0/903448259 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5ea4010460 con 0x7f5eb40ffb40 2024-04-22T23:58:44.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.582+0000 7f5eb0ff9700 1 -- 172.21.15.67:0/903448259 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ea400f5f0 con 0x7f5eb40ffb40 2024-04-22T23:58:44.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.582+0000 7f5eb9bde700 1 -- 172.21.15.67:0/903448259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eb4196e70 con 0x7f5eb40ffb40 2024-04-22T23:58:44.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.583+0000 7f5eb9bde700 1 -- 172.21.15.67:0/903448259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eb404f2e0 con 0x7f5eb40ffb40 2024-04-22T23:58:44.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.583+0000 7f5eb0ff9700 1 -- 172.21.15.67:0/903448259 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5ea40105d0 con 0x7f5eb40ffb40 2024-04-22T23:58:44.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.584+0000 7f5eb0ff9700 1 --2- 172.21.15.67:0/903448259 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5e9c06ecc0 0x7f5e9c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:44.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.584+0000 7f5eb2ffd700 1 --2- 172.21.15.67:0/903448259 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5e9c06ecc0 0x7f5e9c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:44.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.584+0000 7f5eb0ff9700 1 -- 172.21.15.67:0/903448259 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f5ea408eff0 con 0x7f5eb40ffb40 2024-04-22T23:58:44.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.584+0000 7f5eb2ffd700 1 --2- 172.21.15.67:0/903448259 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5e9c06ecc0 0x7f5e9c071180 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f5ea8005b40 tx=0x7f5ea801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:58:44.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.589+0000 7f5eb0ff9700 1 -- 172.21.15.67:0/903448259 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f5ea405b760 con 0x7f5eb40ffb40 2024-04-22T23:58:44.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.818+0000 7f5eb9bde700 1 -- 172.21.15.67:0/903448259 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5eb41971b0 con 0x7f5e9c06ecc0 2024-04-22T23:58:44.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.819+0000 7f5eb0ff9700 1 -- 172.21.15.67:0/903448259 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f5eb41971b0 con 0x7f5e9c06ecc0 2024-04-22T23:58:44.810 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:58:44.810 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:58:44.810 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:58:44.810 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:58:44.810 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:58:44.810 INFO:teuthology.orchestra.run.smithi067.stdout: "mds", 2024-04-22T23:58:44.811 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:58:44.811 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:58:44.811 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:58:44.811 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:58:44.811 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:58:44.811 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T23:58:44.811 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:58:44.811 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:58:44.811 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:58:44.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.821+0000 7f5eb9bde700 1 -- 172.21.15.67:0/903448259 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5e9c06ecc0 msgr2=0x7f5e9c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:44.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.822+0000 7f5eb9bde700 1 --2- 172.21.15.67:0/903448259 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5e9c06ecc0 0x7f5e9c071180 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f5ea8005b40 tx=0x7f5ea801a040 comp rx=0 tx=0).stop 2024-04-22T23:58:44.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.822+0000 7f5eb9bde700 1 -- 172.21.15.67:0/903448259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5eb40ffb40 msgr2=0x7f5eb419c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:44.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.822+0000 7f5eb9bde700 1 --2- 172.21.15.67:0/903448259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5eb40ffb40 0x7f5eb419c5c0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f5ea400d8d0 tx=0x7f5ea400dc90 comp rx=0 tx=0).stop 2024-04-22T23:58:44.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.822+0000 7f5eb9bde700 1 -- 172.21.15.67:0/903448259 shutdown_connections 2024-04-22T23:58:44.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.822+0000 7f5eb9bde700 1 --2- 172.21.15.67:0/903448259 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5e9c06ecc0 0x7f5e9c071180 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:44.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.822+0000 7f5eb9bde700 1 --2- 172.21.15.67:0/903448259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5eb40ffb40 0x7f5eb419c5c0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:44.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.822+0000 7f5eb9bde700 1 --2- 172.21.15.67:0/903448259 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5eb4105bf0 0x7f5eb419cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:44.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.822+0000 7f5eb9bde700 1 -- 172.21.15.67:0/903448259 >> 172.21.15.67:0/903448259 conn(0x7f5eb40fb650 msgr2=0x7f5eb4068eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:58:44.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.822+0000 7f5eb9bde700 1 -- 172.21.15.67:0/903448259 shutdown_connections 2024-04-22T23:58:44.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.822+0000 7f5eb9bde700 1 -- 172.21.15.67:0/903448259 wait complete. 2024-04-22T23:58:44.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.928+0000 7faf2aaa2700 1 -- 172.21.15.67:0/3385423539 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf24100ea0 msgr2=0x7faf24101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:44.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.928+0000 7faf2aaa2700 1 --2- 172.21.15.67:0/3385423539 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf24100ea0 0x7faf24101280 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7faf18009910 tx=0x7faf18009c20 comp rx=0 tx=0).stop 2024-04-22T23:58:44.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.928+0000 7faf2aaa2700 1 -- 172.21.15.67:0/3385423539 shutdown_connections 2024-04-22T23:58:44.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.928+0000 7faf2aaa2700 1 --2- 172.21.15.67:0/3385423539 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf24101850 0x7faf241058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:44.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.928+0000 7faf2aaa2700 1 --2- 172.21.15.67:0/3385423539 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf24100ea0 0x7faf24101280 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:44.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.928+0000 7faf2aaa2700 1 -- 172.21.15.67:0/3385423539 >> 172.21.15.67:0/3385423539 conn(0x7faf240fc750 msgr2=0x7faf240feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:58:44.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.928+0000 7faf2aaa2700 1 -- 172.21.15.67:0/3385423539 shutdown_connections 2024-04-22T23:58:44.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.928+0000 7faf2aaa2700 1 -- 172.21.15.67:0/3385423539 wait complete. 2024-04-22T23:58:44.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.929+0000 7faf2aaa2700 1 Processor -- start 2024-04-22T23:58:44.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.929+0000 7faf2aaa2700 1 -- start start 2024-04-22T23:58:44.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.929+0000 7faf2aaa2700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf24100ea0 0x7faf24073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:44.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.929+0000 7faf2aaa2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf24101850 0x7faf240735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:44.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.929+0000 7faf2aaa2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf24073b10 con 0x7faf24101850 2024-04-22T23:58:44.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.929+0000 7faf2aaa2700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf24073c50 con 0x7faf24100ea0 2024-04-22T23:58:44.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.929+0000 7faf23fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf24101850 0x7faf240735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:44.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.929+0000 7faf23fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf24101850 0x7faf240735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55842/0 (socket says 172.21.15.67:55842) 2024-04-22T23:58:44.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.929+0000 7faf23fff700 1 -- 172.21.15.67:0/3476056206 learned_addr learned my addr 172.21.15.67:0/3476056206 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:58:44.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.930+0000 7faf23fff700 1 -- 172.21.15.67:0/3476056206 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf24100ea0 msgr2=0x7faf24073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:44.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.930+0000 7faf23fff700 1 --2- 172.21.15.67:0/3476056206 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf24100ea0 0x7faf24073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:44.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.930+0000 7faf23fff700 1 -- 172.21.15.67:0/3476056206 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf180095f0 con 0x7faf24101850 2024-04-22T23:58:44.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.930+0000 7faf23fff700 1 --2- 172.21.15.67:0/3476056206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf24101850 0x7faf240735d0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7faf1000b480 tx=0x7faf1000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:58:44.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.930+0000 7faf21ffb700 1 -- 172.21.15.67:0/3476056206 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf100118c0 con 0x7faf24101850 2024-04-22T23:58:44.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.930+0000 7faf2aaa2700 1 -- 172.21.15.67:0/3476056206 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf241a2b60 con 0x7faf24101850 2024-04-22T23:58:44.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.930+0000 7faf21ffb700 1 -- 172.21.15.67:0/3476056206 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faf10010380 con 0x7faf24101850 2024-04-22T23:58:44.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.930+0000 7faf21ffb700 1 -- 172.21.15.67:0/3476056206 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf1000f4f0 con 0x7faf24101850 2024-04-22T23:58:44.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.930+0000 7faf2aaa2700 1 -- 172.21.15.67:0/3476056206 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf241a2f70 con 0x7faf24101850 2024-04-22T23:58:44.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.931+0000 7faf2aaa2700 1 -- 172.21.15.67:0/3476056206 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf24109240 con 0x7faf24101850 2024-04-22T23:58:44.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.932+0000 7faf21ffb700 1 -- 172.21.15.67:0/3476056206 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7faf100104f0 con 0x7faf24101850 2024-04-22T23:58:44.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.932+0000 7faf21ffb700 1 --2- 172.21.15.67:0/3476056206 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faf1406ecc0 0x7faf14071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:58:44.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.932+0000 7faf21ffb700 1 -- 172.21.15.67:0/3476056206 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7faf1008ec00 con 0x7faf24101850 2024-04-22T23:58:44.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.932+0000 7faf2883e700 1 --2- 172.21.15.67:0/3476056206 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faf1406ecc0 0x7faf14071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:58:44.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.933+0000 7faf2883e700 1 --2- 172.21.15.67:0/3476056206 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faf1406ecc0 0x7faf14071180 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7faf1800b5c0 tx=0x7faf18005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:58:44.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:44.937+0000 7faf21ffb700 1 -- 172.21.15.67:0/3476056206 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7faf10057840 con 0x7faf24101850 2024-04-22T23:58:45.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:45.233+0000 7faf2aaa2700 1 -- 172.21.15.67:0/3476056206 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7faf24108c20 con 0x7faf24101850 2024-04-22T23:58:45.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:45.234+0000 7faf21ffb700 1 -- 172.21.15.67:0/3476056206 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4505 (secure 0 0 0) 0x7faf1005af00 con 0x7faf24101850 2024-04-22T23:58:45.225 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:58:45.225 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T23:58:45.225 INFO:teuthology.orchestra.run.smithi067.stdout: host smithi134 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-22T23:58:45.225 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T23:58:45.225 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:58:45.225 INFO:teuthology.orchestra.run.smithi067.stdout:Traceback (most recent call last): 2024-04-22T23:58:45.225 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T23:58:45.225 INFO:teuthology.orchestra.run.smithi067.stdout: return _run_code(code, main_globals, None, 2024-04-22T23:58:45.225 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T23:58:45.225 INFO:teuthology.orchestra.run.smithi067.stdout: exec(code, run_globals) 2024-04-22T23:58:45.225 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5642, in 2024-04-22T23:58:45.225 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5630, in main 2024-04-22T23:58:45.226 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 408, in _infer_config 2024-04-22T23:58:45.226 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 323, in _infer_fsid 2024-04-22T23:58:45.226 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 436, in _infer_image 2024-04-22T23:58:45.226 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 310, in _validate_fsid 2024-04-22T23:58:45.226 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 3380, in command_ceph_volume 2024-04-22T23:58:45.226 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T23:58:45.226 INFO:teuthology.orchestra.run.smithi067.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:58:45.226 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:58:45.226 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:58:45.226 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:58:45.226 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:58:45.226 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:58:45.226 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:58:45.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:45.236+0000 7faf2aaa2700 1 -- 172.21.15.67:0/3476056206 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faf1406ecc0 msgr2=0x7faf14071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:45.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:45.236+0000 7faf2aaa2700 1 --2- 172.21.15.67:0/3476056206 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faf1406ecc0 0x7faf14071180 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7faf1800b5c0 tx=0x7faf18005960 comp rx=0 tx=0).stop 2024-04-22T23:58:45.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:45.236+0000 7faf2aaa2700 1 -- 172.21.15.67:0/3476056206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf24101850 msgr2=0x7faf240735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:58:45.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:45.236+0000 7faf2aaa2700 1 --2- 172.21.15.67:0/3476056206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf24101850 0x7faf240735d0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7faf1000b480 tx=0x7faf1000b840 comp rx=0 tx=0).stop 2024-04-22T23:58:45.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:45.236+0000 7faf2aaa2700 1 -- 172.21.15.67:0/3476056206 shutdown_connections 2024-04-22T23:58:45.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:45.236+0000 7faf2aaa2700 1 --2- 172.21.15.67:0/3476056206 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faf1406ecc0 0x7faf14071180 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:45.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:45.236+0000 7faf2aaa2700 1 --2- 172.21.15.67:0/3476056206 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf24100ea0 0x7faf24073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:45.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:45.236+0000 7faf2aaa2700 1 --2- 172.21.15.67:0/3476056206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf24101850 0x7faf240735d0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:58:45.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:45.236+0000 7faf2aaa2700 1 -- 172.21.15.67:0/3476056206 >> 172.21.15.67:0/3476056206 conn(0x7faf240fc750 msgr2=0x7faf240fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:58:45.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:45.236+0000 7faf2aaa2700 1 -- 172.21.15.67:0/3476056206 shutdown_connections 2024-04-22T23:58:45.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:58:45.236+0000 7faf2aaa2700 1 -- 172.21.15.67:0/3476056206 wait complete. 2024-04-22T23:58:45.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:45 smithi134 ceph-mon[85630]: from='client.35752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:58:45.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:45 smithi134 ceph-mon[85630]: from='client.35756 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:58:45.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:45 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2333079832' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:58:45.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:45 smithi067 ceph-mon[112438]: from='client.35752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:58:45.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:45 smithi067 ceph-mon[112438]: from='client.35756 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:58:45.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:45 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2333079832' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:58:46.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:46 smithi134 ceph-mon[85630]: from='client.35768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:58:46.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:46 smithi134 ceph-mon[85630]: pgmap v973: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:46.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:46 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3476056206' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:58:46.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:46 smithi067 ceph-mon[112438]: from='client.35768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:58:46.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:46 smithi067 ceph-mon[112438]: pgmap v973: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:46.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:46 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3476056206' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:58:48.336 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:48 smithi067 ceph-mon[112438]: pgmap v974: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:48.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:48 smithi134 ceph-mon[85630]: pgmap v974: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:50.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:50 smithi134 ceph-mon[85630]: pgmap v975: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:50.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:50 smithi067 ceph-mon[112438]: pgmap v975: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:52.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:52 smithi134 ceph-mon[85630]: pgmap v976: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:52.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:52 smithi067 ceph-mon[112438]: pgmap v976: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:53.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:53 smithi134 ceph-mon[85630]: pgmap v977: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:53.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:53 smithi067 ceph-mon[112438]: pgmap v977: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:55.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:55 smithi134 ceph-mon[85630]: pgmap v978: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:55.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:55 smithi067 ceph-mon[112438]: pgmap v978: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:56.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:58:56.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:58:57.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:57 smithi134 ceph-mon[85630]: pgmap v979: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:57.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:57 smithi067 ceph-mon[112438]: pgmap v979: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:59.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:58:59 smithi134 ceph-mon[85630]: pgmap v980: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:58:59.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:58:59 smithi067 ceph-mon[112438]: pgmap v980: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:01.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:01 smithi134 ceph-mon[85630]: pgmap v981: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:01.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:01 smithi067 ceph-mon[112438]: pgmap v981: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:03.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:03 smithi134 ceph-mon[85630]: pgmap v982: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:03.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:03 smithi067 ceph-mon[112438]: pgmap v982: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:05.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:05 smithi134 ceph-mon[85630]: pgmap v983: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:05.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:05 smithi067 ceph-mon[112438]: pgmap v983: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:07.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:07 smithi134 ceph-mon[85630]: pgmap v984: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:07.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:07 smithi067 ceph-mon[112438]: pgmap v984: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:09.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:09 smithi134 ceph-mon[85630]: pgmap v985: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:09.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:09 smithi067 ceph-mon[112438]: pgmap v985: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:11.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:59:11.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:59:12.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:12 smithi134 ceph-mon[85630]: pgmap v986: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:12.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:12 smithi067 ceph-mon[112438]: pgmap v986: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:13.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:13 smithi134 ceph-mon[85630]: pgmap v987: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:13.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:13 smithi067 ceph-mon[112438]: pgmap v987: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:15.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.357+0000 7f43888bc700 1 -- 172.21.15.67:0/1903166090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4384101850 msgr2=0x7f43841058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:15.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.357+0000 7f43888bc700 1 --2- 172.21.15.67:0/1903166090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4384101850 0x7f43841058a0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f436c009910 tx=0x7f436c009c20 comp rx=0 tx=0).stop 2024-04-22T23:59:15.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.357+0000 7f43888bc700 1 -- 172.21.15.67:0/1903166090 shutdown_connections 2024-04-22T23:59:15.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.357+0000 7f43888bc700 1 --2- 172.21.15.67:0/1903166090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4384101850 0x7f43841058a0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:15.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.357+0000 7f43888bc700 1 --2- 172.21.15.67:0/1903166090 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4384100ea0 0x7f4384101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:15.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.357+0000 7f43888bc700 1 -- 172.21.15.67:0/1903166090 >> 172.21.15.67:0/1903166090 conn(0x7f43840fc750 msgr2=0x7f43840feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:59:15.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.357+0000 7f43888bc700 1 -- 172.21.15.67:0/1903166090 shutdown_connections 2024-04-22T23:59:15.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.358+0000 7f43888bc700 1 -- 172.21.15.67:0/1903166090 wait complete. 2024-04-22T23:59:15.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.358+0000 7f43888bc700 1 Processor -- start 2024-04-22T23:59:15.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.358+0000 7f43888bc700 1 -- start start 2024-04-22T23:59:15.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.359+0000 7f43888bc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4384100ea0 0x7f4384073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:15.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.359+0000 7f43888bc700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4384101850 0x7f43840735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:15.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.359+0000 7f43888bc700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4384073b10 con 0x7f4384100ea0 2024-04-22T23:59:15.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.359+0000 7f43888bc700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4384073c50 con 0x7f4384101850 2024-04-22T23:59:15.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.359+0000 7f438259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4384100ea0 0x7f4384073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:15.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.359+0000 7f438259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4384100ea0 0x7f4384073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32798/0 (socket says 172.21.15.67:32798) 2024-04-22T23:59:15.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.359+0000 7f438259c700 1 -- 172.21.15.67:0/2392965196 learned_addr learned my addr 172.21.15.67:0/2392965196 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:59:15.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.359+0000 7f4381d9b700 1 --2- 172.21.15.67:0/2392965196 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4384101850 0x7f43840735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:15.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.359+0000 7f438259c700 1 -- 172.21.15.67:0/2392965196 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4384101850 msgr2=0x7f43840735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:15.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.359+0000 7f438259c700 1 --2- 172.21.15.67:0/2392965196 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4384101850 0x7f43840735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:15.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.359+0000 7f438259c700 1 -- 172.21.15.67:0/2392965196 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f436c0095f0 con 0x7f4384100ea0 2024-04-22T23:59:15.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.359+0000 7f438259c700 1 --2- 172.21.15.67:0/2392965196 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4384100ea0 0x7f4384073090 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f437c0060b0 tx=0x7f437c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:59:15.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.359+0000 7f437b7fe700 1 -- 172.21.15.67:0/2392965196 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f437c00dda0 con 0x7f4384100ea0 2024-04-22T23:59:15.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.359+0000 7f437b7fe700 1 -- 172.21.15.67:0/2392965196 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f437c00f040 con 0x7f4384100ea0 2024-04-22T23:59:15.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.360+0000 7f43888bc700 1 -- 172.21.15.67:0/2392965196 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43841a2b00 con 0x7f4384100ea0 2024-04-22T23:59:15.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.360+0000 7f437b7fe700 1 -- 172.21.15.67:0/2392965196 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f437c0146f0 con 0x7f4384100ea0 2024-04-22T23:59:15.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.360+0000 7f43888bc700 1 -- 172.21.15.67:0/2392965196 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43841a2f10 con 0x7f4384100ea0 2024-04-22T23:59:15.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.361+0000 7f43888bc700 1 -- 172.21.15.67:0/2392965196 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43841092a0 con 0x7f4384100ea0 2024-04-22T23:59:15.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.361+0000 7f437b7fe700 1 -- 172.21.15.67:0/2392965196 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f437c015400 con 0x7f4384100ea0 2024-04-22T23:59:15.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.362+0000 7f437b7fe700 1 --2- 172.21.15.67:0/2392965196 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f437006ecc0 0x7f4370071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:15.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.362+0000 7f437b7fe700 1 -- 172.21.15.67:0/2392965196 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f437c08e900 con 0x7f4384100ea0 2024-04-22T23:59:15.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.362+0000 7f4381d9b700 1 --2- 172.21.15.67:0/2392965196 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f437006ecc0 0x7f4370071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:15.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.362+0000 7f4381d9b700 1 --2- 172.21.15.67:0/2392965196 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f437006ecc0 0x7f4370071180 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f4384074880 tx=0x7f436c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:59:15.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.367+0000 7f437b7fe700 1 -- 172.21.15.67:0/2392965196 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f437c057670 con 0x7f4384100ea0 2024-04-22T23:59:15.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:15 smithi134 ceph-mon[85630]: pgmap v988: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:15.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:15 smithi067 ceph-mon[112438]: pgmap v988: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:15.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.602+0000 7f43888bc700 1 -- 172.21.15.67:0/2392965196 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f43840611d0 con 0x7f437006ecc0 2024-04-22T23:59:15.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.604+0000 7f437b7fe700 1 -- 172.21.15.67:0/2392965196 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f43840611d0 con 0x7f437006ecc0 2024-04-22T23:59:15.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.606+0000 7f43888bc700 1 -- 172.21.15.67:0/2392965196 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f437006ecc0 msgr2=0x7f4370071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:15.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.606+0000 7f43888bc700 1 --2- 172.21.15.67:0/2392965196 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f437006ecc0 0x7f4370071180 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f4384074880 tx=0x7f436c01a040 comp rx=0 tx=0).stop 2024-04-22T23:59:15.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.606+0000 7f43888bc700 1 -- 172.21.15.67:0/2392965196 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4384100ea0 msgr2=0x7f4384073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:15.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.606+0000 7f43888bc700 1 --2- 172.21.15.67:0/2392965196 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4384100ea0 0x7f4384073090 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f437c0060b0 tx=0x7f437c00d3b0 comp rx=0 tx=0).stop 2024-04-22T23:59:15.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.606+0000 7f43888bc700 1 -- 172.21.15.67:0/2392965196 shutdown_connections 2024-04-22T23:59:15.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.606+0000 7f43888bc700 1 --2- 172.21.15.67:0/2392965196 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f437006ecc0 0x7f4370071180 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:15.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.606+0000 7f43888bc700 1 --2- 172.21.15.67:0/2392965196 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4384100ea0 0x7f4384073090 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:15.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.606+0000 7f43888bc700 1 --2- 172.21.15.67:0/2392965196 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4384101850 0x7f43840735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:15.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.606+0000 7f43888bc700 1 -- 172.21.15.67:0/2392965196 >> 172.21.15.67:0/2392965196 conn(0x7f43840fc750 msgr2=0x7f43840fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:59:15.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.606+0000 7f43888bc700 1 -- 172.21.15.67:0/2392965196 shutdown_connections 2024-04-22T23:59:15.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.606+0000 7f43888bc700 1 -- 172.21.15.67:0/2392965196 wait complete. 2024-04-22T23:59:15.607 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:59:15.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.713+0000 7f6c25696700 1 -- 172.21.15.67:0/3495115241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c20100e90 msgr2=0x7f6c20101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:15.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.713+0000 7f6c25696700 1 --2- 172.21.15.67:0/3495115241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c20100e90 0x7f6c20101270 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f6c10009910 tx=0x7f6c10009c20 comp rx=0 tx=0).stop 2024-04-22T23:59:15.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.713+0000 7f6c25696700 1 -- 172.21.15.67:0/3495115241 shutdown_connections 2024-04-22T23:59:15.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.713+0000 7f6c25696700 1 --2- 172.21.15.67:0/3495115241 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c20101840 0x7f6c20105890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:15.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.713+0000 7f6c25696700 1 --2- 172.21.15.67:0/3495115241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c20100e90 0x7f6c20101270 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:15.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.713+0000 7f6c25696700 1 -- 172.21.15.67:0/3495115241 >> 172.21.15.67:0/3495115241 conn(0x7f6c200fc740 msgr2=0x7f6c200feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:59:15.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.713+0000 7f6c25696700 1 -- 172.21.15.67:0/3495115241 shutdown_connections 2024-04-22T23:59:15.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.713+0000 7f6c25696700 1 -- 172.21.15.67:0/3495115241 wait complete. 2024-04-22T23:59:15.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.714+0000 7f6c25696700 1 Processor -- start 2024-04-22T23:59:15.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.714+0000 7f6c25696700 1 -- start start 2024-04-22T23:59:15.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.714+0000 7f6c25696700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c20100e90 0x7f6c20073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:15.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.714+0000 7f6c25696700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c20101840 0x7f6c200735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:15.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.714+0000 7f6c25696700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c20073b10 con 0x7f6c20101840 2024-04-22T23:59:15.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.714+0000 7f6c25696700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c20073c50 con 0x7f6c20100e90 2024-04-22T23:59:15.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.714+0000 7f6c1e7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c20101840 0x7f6c200735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:15.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.714+0000 7f6c1e7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c20101840 0x7f6c200735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32818/0 (socket says 172.21.15.67:32818) 2024-04-22T23:59:15.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.714+0000 7f6c1e7fc700 1 -- 172.21.15.67:0/1847828610 learned_addr learned my addr 172.21.15.67:0/1847828610 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:59:15.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.714+0000 7f6c1e7fc700 1 -- 172.21.15.67:0/1847828610 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c20100e90 msgr2=0x7f6c20073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:15.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.714+0000 7f6c1effd700 1 --2- 172.21.15.67:0/1847828610 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c20100e90 0x7f6c20073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:15.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.715+0000 7f6c1e7fc700 1 --2- 172.21.15.67:0/1847828610 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c20100e90 0x7f6c20073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:15.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.715+0000 7f6c1e7fc700 1 -- 172.21.15.67:0/1847828610 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c100095f0 con 0x7f6c20101840 2024-04-22T23:59:15.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.715+0000 7f6c1e7fc700 1 --2- 172.21.15.67:0/1847828610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c20101840 0x7f6c200735d0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f6c1400e8f0 tx=0x7f6c1400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:59:15.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.715+0000 7f6c07fff700 1 -- 172.21.15.67:0/1847828610 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c14009790 con 0x7f6c20101840 2024-04-22T23:59:15.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.715+0000 7f6c25696700 1 -- 172.21.15.67:0/1847828610 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c201a2a80 con 0x7f6c20101840 2024-04-22T23:59:15.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.715+0000 7f6c07fff700 1 -- 172.21.15.67:0/1847828610 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6c14004cf0 con 0x7f6c20101840 2024-04-22T23:59:15.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.715+0000 7f6c07fff700 1 -- 172.21.15.67:0/1847828610 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c14005690 con 0x7f6c20101840 2024-04-22T23:59:15.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.715+0000 7f6c25696700 1 -- 172.21.15.67:0/1847828610 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c201a2e90 con 0x7f6c20101840 2024-04-22T23:59:15.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.716+0000 7f6c25696700 1 -- 172.21.15.67:0/1847828610 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c20109230 con 0x7f6c20101840 2024-04-22T23:59:15.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.716+0000 7f6c07fff700 1 -- 172.21.15.67:0/1847828610 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6c14004830 con 0x7f6c20101840 2024-04-22T23:59:15.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.717+0000 7f6c07fff700 1 --2- 172.21.15.67:0/1847828610 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6c0806ecc0 0x7f6c08071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:15.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.717+0000 7f6c07fff700 1 -- 172.21.15.67:0/1847828610 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f6c14014070 con 0x7f6c20101840 2024-04-22T23:59:15.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.717+0000 7f6c1effd700 1 --2- 172.21.15.67:0/1847828610 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6c0806ecc0 0x7f6c08071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:15.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.717+0000 7f6c1effd700 1 --2- 172.21.15.67:0/1847828610 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6c0806ecc0 0x7f6c08071180 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f6c10009910 tx=0x7f6c10009f70 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:59:15.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.722+0000 7f6c07fff700 1 -- 172.21.15.67:0/1847828610 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f6c14058010 con 0x7f6c20101840 2024-04-22T23:59:15.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.950+0000 7f6c25696700 1 -- 172.21.15.67:0/1847828610 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6c200611d0 con 0x7f6c0806ecc0 2024-04-22T23:59:15.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.952+0000 7f6c07fff700 1 -- 172.21.15.67:0/1847828610 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6c200611d0 con 0x7f6c0806ecc0 2024-04-22T23:59:15.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.954+0000 7f6c25696700 1 -- 172.21.15.67:0/1847828610 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6c0806ecc0 msgr2=0x7f6c08071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:15.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.954+0000 7f6c25696700 1 --2- 172.21.15.67:0/1847828610 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6c0806ecc0 0x7f6c08071180 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f6c10009910 tx=0x7f6c10009f70 comp rx=0 tx=0).stop 2024-04-22T23:59:15.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.954+0000 7f6c25696700 1 -- 172.21.15.67:0/1847828610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c20101840 msgr2=0x7f6c200735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:15.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.954+0000 7f6c25696700 1 --2- 172.21.15.67:0/1847828610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c20101840 0x7f6c200735d0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f6c1400e8f0 tx=0x7f6c1400ecb0 comp rx=0 tx=0).stop 2024-04-22T23:59:15.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.954+0000 7f6c25696700 1 -- 172.21.15.67:0/1847828610 shutdown_connections 2024-04-22T23:59:15.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.954+0000 7f6c25696700 1 --2- 172.21.15.67:0/1847828610 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6c0806ecc0 0x7f6c08071180 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:15.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.954+0000 7f6c25696700 1 --2- 172.21.15.67:0/1847828610 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6c20100e90 0x7f6c20073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:15.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.954+0000 7f6c25696700 1 --2- 172.21.15.67:0/1847828610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c20101840 0x7f6c200735d0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:15.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.954+0000 7f6c25696700 1 -- 172.21.15.67:0/1847828610 >> 172.21.15.67:0/1847828610 conn(0x7f6c200fc740 msgr2=0x7f6c200fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:59:15.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.954+0000 7f6c25696700 1 -- 172.21.15.67:0/1847828610 shutdown_connections 2024-04-22T23:59:15.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:15.954+0000 7f6c25696700 1 -- 172.21.15.67:0/1847828610 wait complete. 2024-04-22T23:59:16.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.060+0000 7f1af4f9b700 1 -- 172.21.15.67:0/1606009320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1af0105350 msgr2=0x7f1af0105730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:16.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.060+0000 7f1af4f9b700 1 --2- 172.21.15.67:0/1606009320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1af0105350 0x7f1af0105730 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0009910 tx=0x7f1ae0009c20 comp rx=0 tx=0).stop 2024-04-22T23:59:16.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.061+0000 7f1af4f9b700 1 -- 172.21.15.67:0/1606009320 shutdown_connections 2024-04-22T23:59:16.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.061+0000 7f1af4f9b700 1 --2- 172.21.15.67:0/1606009320 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1af00ff340 0x7f1af00ff7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:16.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.061+0000 7f1af4f9b700 1 --2- 172.21.15.67:0/1606009320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1af0105350 0x7f1af0105730 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:16.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.061+0000 7f1af4f9b700 1 -- 172.21.15.67:0/1606009320 >> 172.21.15.67:0/1606009320 conn(0x7f1af0074e30 msgr2=0x7f1af0075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:59:16.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.061+0000 7f1af4f9b700 1 -- 172.21.15.67:0/1606009320 shutdown_connections 2024-04-22T23:59:16.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.061+0000 7f1af4f9b700 1 -- 172.21.15.67:0/1606009320 wait complete. 2024-04-22T23:59:16.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.061+0000 7f1af4f9b700 1 Processor -- start 2024-04-22T23:59:16.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.061+0000 7f1af4f9b700 1 -- start start 2024-04-22T23:59:16.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.062+0000 7f1af4f9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1af00ff340 0x7f1af0196980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:16.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.062+0000 7f1af4f9b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1af0105350 0x7f1af0196ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:16.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.062+0000 7f1af4f9b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af0197550 con 0x7f1af00ff340 2024-04-22T23:59:16.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.062+0000 7f1af4f9b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af019b360 con 0x7f1af0105350 2024-04-22T23:59:16.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.062+0000 7f1aee59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1af00ff340 0x7f1af0196980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:16.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.062+0000 7f1aee59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1af00ff340 0x7f1af0196980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32842/0 (socket says 172.21.15.67:32842) 2024-04-22T23:59:16.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.062+0000 7f1aee59c700 1 -- 172.21.15.67:0/232683531 learned_addr learned my addr 172.21.15.67:0/232683531 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:59:16.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.062+0000 7f1aedd9b700 1 --2- 172.21.15.67:0/232683531 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1af0105350 0x7f1af0196ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:16.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.062+0000 7f1aee59c700 1 -- 172.21.15.67:0/232683531 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1af0105350 msgr2=0x7f1af0196ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:16.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.062+0000 7f1aee59c700 1 --2- 172.21.15.67:0/232683531 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1af0105350 0x7f1af0196ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:16.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.062+0000 7f1aee59c700 1 -- 172.21.15.67:0/232683531 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae00095f0 con 0x7f1af00ff340 2024-04-22T23:59:16.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.062+0000 7f1aee59c700 1 --2- 172.21.15.67:0/232683531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1af00ff340 0x7f1af0196980 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0006010 tx=0x7f1ae000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:59:16.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.062+0000 7f1adf7fe700 1 -- 172.21.15.67:0/232683531 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ae001d070 con 0x7f1af00ff340 2024-04-22T23:59:16.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.063+0000 7f1adf7fe700 1 -- 172.21.15.67:0/232683531 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ae000f460 con 0x7f1af00ff340 2024-04-22T23:59:16.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.063+0000 7f1af4f9b700 1 -- 172.21.15.67:0/232683531 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1af019b5e0 con 0x7f1af00ff340 2024-04-22T23:59:16.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.063+0000 7f1adf7fe700 1 -- 172.21.15.67:0/232683531 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ae0017630 con 0x7f1af00ff340 2024-04-22T23:59:16.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.063+0000 7f1af4f9b700 1 -- 172.21.15.67:0/232683531 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1af019bad0 con 0x7f1af00ff340 2024-04-22T23:59:16.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.064+0000 7f1af4f9b700 1 -- 172.21.15.67:0/232683531 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1af0109880 con 0x7f1af00ff340 2024-04-22T23:59:16.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.064+0000 7f1adf7fe700 1 -- 172.21.15.67:0/232683531 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1ae000f5d0 con 0x7f1af00ff340 2024-04-22T23:59:16.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.064+0000 7f1adf7fe700 1 --2- 172.21.15.67:0/232683531 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1ad806ecc0 0x7f1ad8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:16.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.065+0000 7f1adf7fe700 1 -- 172.21.15.67:0/232683531 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f1ae00904d0 con 0x7f1af00ff340 2024-04-22T23:59:16.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.065+0000 7f1aedd9b700 1 --2- 172.21.15.67:0/232683531 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1ad806ecc0 0x7f1ad8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:16.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.065+0000 7f1aedd9b700 1 --2- 172.21.15.67:0/232683531 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1ad806ecc0 0x7f1ad8071180 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f1ae4005950 tx=0x7f1ae40058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:59:16.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.069+0000 7f1adf7fe700 1 -- 172.21.15.67:0/232683531 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1ae005cbe0 con 0x7f1af00ff340 2024-04-22T23:59:16.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.289+0000 7f1af4f9b700 1 -- 172.21.15.67:0/232683531 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1af0197da0 con 0x7f1ad806ecc0 2024-04-22T23:59:16.280 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:16 smithi067 ceph-mon[112438]: from='client.35776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:59:16.280 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:16 smithi067 ceph-mon[112438]: from='client.35780 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:59:16.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.301+0000 7f1adf7fe700 1 -- 172.21.15.67:0/232683531 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f1af0197da0 con 0x7f1ad806ecc0 2024-04-22T23:59:16.294 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:59:16.294 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (33m) 10m ago 44m 24.4M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:59:16.294 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (44m) 10m ago 44m 10.0M - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:59:16.295 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (43m) 10m ago 43m 10.0M - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:59:16.295 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (30m) 10m ago 44m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:59:16.295 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (30m) 10m ago 43m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:59:16.295 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (32m) 10m ago 43m 81.2M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:59:16.295 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (26m) 10m ago 38m 19.4M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:59:16.295 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (26m) 10m ago 38m 20.3M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:59:16.295 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (26m) 10m ago 38m 20.1M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:59:16.295 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (10m) 10m ago 38m 14.0M - 19.0.0-3252-g43be0201 a0434699c1d7 46d3d7e86fee 2024-04-22T23:59:16.295 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (34m) 10m ago 45m 514M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:59:16.295 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (33m) 10m ago 43m 459M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:59:16.295 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (31m) 10m ago 45m 74.2M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:59:16.295 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (31m) 10m ago 43m 54.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:59:16.295 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (33m) 10m ago 44m 15.2M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:59:16.296 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (33m) 10m ago 43m 16.0M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:59:16.296 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (29m) 10m ago 42m 102M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:59:16.296 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (29m) 10m ago 42m 128M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:59:16.296 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (28m) 10m ago 41m 83.8M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:59:16.296 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (28m) 10m ago 41m 101M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:59:16.296 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (27m) 10m ago 40m 86.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:59:16.296 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (26m) 10m ago 40m 81.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:59:16.296 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (33m) 10m ago 43m 78.7M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:59:16.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.303+0000 7f1af4f9b700 1 -- 172.21.15.67:0/232683531 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1ad806ecc0 msgr2=0x7f1ad8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:16.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.303+0000 7f1af4f9b700 1 --2- 172.21.15.67:0/232683531 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1ad806ecc0 0x7f1ad8071180 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f1ae4005950 tx=0x7f1ae40058e0 comp rx=0 tx=0).stop 2024-04-22T23:59:16.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.303+0000 7f1af4f9b700 1 -- 172.21.15.67:0/232683531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1af00ff340 msgr2=0x7f1af0196980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:16.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.303+0000 7f1af4f9b700 1 --2- 172.21.15.67:0/232683531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1af00ff340 0x7f1af0196980 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0006010 tx=0x7f1ae000be20 comp rx=0 tx=0).stop 2024-04-22T23:59:16.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.303+0000 7f1af4f9b700 1 -- 172.21.15.67:0/232683531 shutdown_connections 2024-04-22T23:59:16.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.303+0000 7f1af4f9b700 1 --2- 172.21.15.67:0/232683531 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1ad806ecc0 0x7f1ad8071180 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:16.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.303+0000 7f1af4f9b700 1 --2- 172.21.15.67:0/232683531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1af00ff340 0x7f1af0196980 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:16.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.303+0000 7f1af4f9b700 1 --2- 172.21.15.67:0/232683531 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1af0105350 0x7f1af0196ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:16.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.303+0000 7f1af4f9b700 1 -- 172.21.15.67:0/232683531 >> 172.21.15.67:0/232683531 conn(0x7f1af0074e30 msgr2=0x7f1af0103870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:59:16.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.303+0000 7f1af4f9b700 1 -- 172.21.15.67:0/232683531 shutdown_connections 2024-04-22T23:59:16.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.303+0000 7f1af4f9b700 1 -- 172.21.15.67:0/232683531 wait complete. 2024-04-22T23:59:16.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.409+0000 7efd86c4e700 1 -- 172.21.15.67:0/2719707748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd80100750 msgr2=0x7efd801047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:16.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.409+0000 7efd86c4e700 1 --2- 172.21.15.67:0/2719707748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd80100750 0x7efd801047a0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7efd6c009910 tx=0x7efd6c009c20 comp rx=0 tx=0).stop 2024-04-22T23:59:16.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.409+0000 7efd86c4e700 1 -- 172.21.15.67:0/2719707748 shutdown_connections 2024-04-22T23:59:16.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.409+0000 7efd86c4e700 1 --2- 172.21.15.67:0/2719707748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd80100750 0x7efd801047a0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:16.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.409+0000 7efd86c4e700 1 --2- 172.21.15.67:0/2719707748 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd800ffda0 0x7efd80100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:16.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.409+0000 7efd86c4e700 1 -- 172.21.15.67:0/2719707748 >> 172.21.15.67:0/2719707748 conn(0x7efd800fb650 msgr2=0x7efd800fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:59:16.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.409+0000 7efd86c4e700 1 -- 172.21.15.67:0/2719707748 shutdown_connections 2024-04-22T23:59:16.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.409+0000 7efd86c4e700 1 -- 172.21.15.67:0/2719707748 wait complete. 2024-04-22T23:59:16.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.409+0000 7efd86c4e700 1 Processor -- start 2024-04-22T23:59:16.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.410+0000 7efd86c4e700 1 -- start start 2024-04-22T23:59:16.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.410+0000 7efd86c4e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd800ffda0 0x7efd8019c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:16.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.410+0000 7efd86c4e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd80100750 0x7efd8019cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:16.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.410+0000 7efd86c4e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd8019d1b0 con 0x7efd800ffda0 2024-04-22T23:59:16.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.410+0000 7efd86c4e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd801966f0 con 0x7efd80100750 2024-04-22T23:59:16.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.410+0000 7efd849ea700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd800ffda0 0x7efd8019c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:16.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.410+0000 7efd849ea700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd800ffda0 0x7efd8019c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32860/0 (socket says 172.21.15.67:32860) 2024-04-22T23:59:16.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.410+0000 7efd849ea700 1 -- 172.21.15.67:0/3490506449 learned_addr learned my addr 172.21.15.67:0/3490506449 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:59:16.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.410+0000 7efd7ffff700 1 --2- 172.21.15.67:0/3490506449 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd80100750 0x7efd8019cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:16.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.410+0000 7efd849ea700 1 -- 172.21.15.67:0/3490506449 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd80100750 msgr2=0x7efd8019cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:16.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.410+0000 7efd849ea700 1 --2- 172.21.15.67:0/3490506449 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd80100750 0x7efd8019cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:16.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.410+0000 7efd849ea700 1 -- 172.21.15.67:0/3490506449 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd6c0095f0 con 0x7efd800ffda0 2024-04-22T23:59:16.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.410+0000 7efd849ea700 1 --2- 172.21.15.67:0/3490506449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd800ffda0 0x7efd8019c670 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7efd7400c930 tx=0x7efd7400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:59:16.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.411+0000 7efd7dffb700 1 -- 172.21.15.67:0/3490506449 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd74007ad0 con 0x7efd800ffda0 2024-04-22T23:59:16.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.411+0000 7efd7dffb700 1 -- 172.21.15.67:0/3490506449 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efd7400ce80 con 0x7efd800ffda0 2024-04-22T23:59:16.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.411+0000 7efd86c4e700 1 -- 172.21.15.67:0/3490506449 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd801969d0 con 0x7efd800ffda0 2024-04-22T23:59:16.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.411+0000 7efd7dffb700 1 -- 172.21.15.67:0/3490506449 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd74018630 con 0x7efd800ffda0 2024-04-22T23:59:16.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.411+0000 7efd86c4e700 1 -- 172.21.15.67:0/3490506449 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd80196f20 con 0x7efd800ffda0 2024-04-22T23:59:16.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.412+0000 7efd86c4e700 1 -- 172.21.15.67:0/3490506449 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd801091b0 con 0x7efd800ffda0 2024-04-22T23:59:16.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.412+0000 7efd7dffb700 1 -- 172.21.15.67:0/3490506449 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7efd74007c30 con 0x7efd800ffda0 2024-04-22T23:59:16.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.413+0000 7efd7dffb700 1 --2- 172.21.15.67:0/3490506449 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efd7006ecc0 0x7efd70071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:16.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.413+0000 7efd7dffb700 1 -- 172.21.15.67:0/3490506449 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7efd7408f010 con 0x7efd800ffda0 2024-04-22T23:59:16.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.413+0000 7efd7ffff700 1 --2- 172.21.15.67:0/3490506449 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efd7006ecc0 0x7efd70071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:16.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.413+0000 7efd7ffff700 1 --2- 172.21.15.67:0/3490506449 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efd7006ecc0 0x7efd70071180 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7efd80197b60 tx=0x7efd6c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:59:16.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.417+0000 7efd7dffb700 1 -- 172.21.15.67:0/3490506449 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7efd7405b780 con 0x7efd800ffda0 2024-04-22T23:59:16.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:16 smithi134 ceph-mon[85630]: from='client.35776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:59:16.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:16 smithi134 ceph-mon[85630]: from='client.35780 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:59:16.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.721+0000 7efd86c4e700 1 -- 172.21.15.67:0/3490506449 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7efd80197ca0 con 0x7efd800ffda0 2024-04-22T23:59:16.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.722+0000 7efd7dffb700 1 -- 172.21.15.67:0/3490506449 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7efd7405b310 con 0x7efd800ffda0 2024-04-22T23:59:16.712 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:59:16.713 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:59:16.713 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:59:16.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:59:16.713 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:59:16.713 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:59:16.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:59:16.713 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:59:16.713 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:59:16.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:59:16.713 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:59:16.713 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 4 2024-04-22T23:59:16.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:59:16.713 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:59:16.713 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 14 2024-04-22T23:59:16.713 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:59:16.714 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:59:16.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.724+0000 7efd86c4e700 1 -- 172.21.15.67:0/3490506449 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efd7006ecc0 msgr2=0x7efd70071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:16.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.724+0000 7efd86c4e700 1 --2- 172.21.15.67:0/3490506449 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efd7006ecc0 0x7efd70071180 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7efd80197b60 tx=0x7efd6c01a040 comp rx=0 tx=0).stop 2024-04-22T23:59:16.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.724+0000 7efd86c4e700 1 -- 172.21.15.67:0/3490506449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd800ffda0 msgr2=0x7efd8019c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:16.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.724+0000 7efd86c4e700 1 --2- 172.21.15.67:0/3490506449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd800ffda0 0x7efd8019c670 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7efd7400c930 tx=0x7efd7400cc40 comp rx=0 tx=0).stop 2024-04-22T23:59:16.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.724+0000 7efd86c4e700 1 -- 172.21.15.67:0/3490506449 shutdown_connections 2024-04-22T23:59:16.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.724+0000 7efd86c4e700 1 --2- 172.21.15.67:0/3490506449 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7efd7006ecc0 0x7efd70071180 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:16.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.724+0000 7efd86c4e700 1 --2- 172.21.15.67:0/3490506449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd800ffda0 0x7efd8019c670 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:16.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.724+0000 7efd86c4e700 1 --2- 172.21.15.67:0/3490506449 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efd80100750 0x7efd8019cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:16.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.724+0000 7efd86c4e700 1 -- 172.21.15.67:0/3490506449 >> 172.21.15.67:0/3490506449 conn(0x7efd800fb650 msgr2=0x7efd80069150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:59:16.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.724+0000 7efd86c4e700 1 -- 172.21.15.67:0/3490506449 shutdown_connections 2024-04-22T23:59:16.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.724+0000 7efd86c4e700 1 -- 172.21.15.67:0/3490506449 wait complete. 2024-04-22T23:59:16.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.831+0000 7f7fd41d4700 1 -- 172.21.15.67:0/3117453871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fcc102720 msgr2=0x7f7fcc10ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:16.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.831+0000 7f7fd41d4700 1 --2- 172.21.15.67:0/3117453871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fcc102720 0x7f7fcc10ac10 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc009910 tx=0x7f7fbc009c20 comp rx=0 tx=0).stop 2024-04-22T23:59:16.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.831+0000 7f7fd41d4700 1 -- 172.21.15.67:0/3117453871 shutdown_connections 2024-04-22T23:59:16.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.831+0000 7f7fd41d4700 1 --2- 172.21.15.67:0/3117453871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fcc102720 0x7f7fcc10ac10 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:16.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.831+0000 7f7fd41d4700 1 --2- 172.21.15.67:0/3117453871 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7fcc101e00 0x7f7fcc1021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:16.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.831+0000 7f7fd41d4700 1 -- 172.21.15.67:0/3117453871 >> 172.21.15.67:0/3117453871 conn(0x7f7fcc0fb650 msgr2=0x7f7fcc0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:59:16.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.831+0000 7f7fd41d4700 1 -- 172.21.15.67:0/3117453871 shutdown_connections 2024-04-22T23:59:16.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.831+0000 7f7fd41d4700 1 -- 172.21.15.67:0/3117453871 wait complete. 2024-04-22T23:59:16.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.831+0000 7f7fd41d4700 1 Processor -- start 2024-04-22T23:59:16.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.832+0000 7f7fd41d4700 1 -- start start 2024-04-22T23:59:16.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.832+0000 7f7fd41d4700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7fcc101e00 0x7f7fcc0ffa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:16.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.832+0000 7f7fd41d4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fcc102720 0x7f7fcc0fffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:16.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.832+0000 7f7fd41d4700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc1004e0 con 0x7f7fcc102720 2024-04-22T23:59:16.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.832+0000 7f7fd41d4700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc100620 con 0x7f7fcc101e00 2024-04-22T23:59:16.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.832+0000 7f7fd176f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fcc102720 0x7f7fcc0fffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:16.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.832+0000 7f7fd176f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fcc102720 0x7f7fcc0fffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32876/0 (socket says 172.21.15.67:32876) 2024-04-22T23:59:16.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.832+0000 7f7fd176f700 1 -- 172.21.15.67:0/3762613302 learned_addr learned my addr 172.21.15.67:0/3762613302 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:59:16.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.832+0000 7f7fd1f70700 1 --2- 172.21.15.67:0/3762613302 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7fcc101e00 0x7f7fcc0ffa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:16.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.832+0000 7f7fd176f700 1 -- 172.21.15.67:0/3762613302 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7fcc101e00 msgr2=0x7f7fcc0ffa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:16.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.832+0000 7f7fd176f700 1 --2- 172.21.15.67:0/3762613302 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7fcc101e00 0x7f7fcc0ffa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:16.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.832+0000 7f7fd176f700 1 -- 172.21.15.67:0/3762613302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fbc0095f0 con 0x7f7fcc102720 2024-04-22T23:59:16.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.832+0000 7f7fd176f700 1 --2- 172.21.15.67:0/3762613302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fcc102720 0x7f7fcc0fffa0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc00bd70 tx=0x7f7fbc00bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:59:16.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.832+0000 7f7fc2ffd700 1 -- 172.21.15.67:0/3762613302 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7fbc01d070 con 0x7f7fcc102720 2024-04-22T23:59:16.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.833+0000 7f7fc2ffd700 1 -- 172.21.15.67:0/3762613302 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7fbc00f460 con 0x7f7fcc102720 2024-04-22T23:59:16.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.833+0000 7f7fd41d4700 1 -- 172.21.15.67:0/3762613302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fcc1008a0 con 0x7f7fcc102720 2024-04-22T23:59:16.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.833+0000 7f7fc2ffd700 1 -- 172.21.15.67:0/3762613302 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7fbc017630 con 0x7f7fcc102720 2024-04-22T23:59:16.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.833+0000 7f7fd41d4700 1 -- 172.21.15.67:0/3762613302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fcc1a2d70 con 0x7f7fcc102720 2024-04-22T23:59:16.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.834+0000 7f7fd41d4700 1 -- 172.21.15.67:0/3762613302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fcc108310 con 0x7f7fcc102720 2024-04-22T23:59:16.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.834+0000 7f7fc2ffd700 1 -- 172.21.15.67:0/3762613302 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7fbc00fab0 con 0x7f7fcc102720 2024-04-22T23:59:16.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.835+0000 7f7fc2ffd700 1 --2- 172.21.15.67:0/3762613302 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7fb806ecc0 0x7f7fb8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:16.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.835+0000 7f7fd1f70700 1 --2- 172.21.15.67:0/3762613302 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7fb806ecc0 0x7f7fb8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:16.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.835+0000 7f7fc2ffd700 1 -- 172.21.15.67:0/3762613302 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f7fbc0903f0 con 0x7f7fcc102720 2024-04-22T23:59:16.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.835+0000 7f7fd1f70700 1 --2- 172.21.15.67:0/3762613302 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7fb806ecc0 0x7f7fb8071180 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f7fc800ac50 tx=0x7f7fc800a380 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:59:16.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:16.839+0000 7f7fc2ffd700 1 -- 172.21.15.67:0/3762613302 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7fbc0590b0 con 0x7f7fcc102720 2024-04-22T23:59:17.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.095+0000 7f7fd41d4700 1 -- 172.21.15.67:0/3762613302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f7fcc04f2e0 con 0x7f7fcc102720 2024-04-22T23:59:17.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.095+0000 7f7fc2ffd700 1 -- 172.21.15.67:0/3762613302 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1872 (secure 0 0 0) 0x7f7fbc05c770 con 0x7f7fcc102720 2024-04-22T23:59:17.086 INFO:teuthology.orchestra.run.smithi067.stdout:e33 2024-04-22T23:59:17.086 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:59:17.086 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:59:17.087 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:59:17.087 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:59:17.087 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:59:17.087 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:59:17.087 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:59:17.087 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:59:17.087 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:59:17.087 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:59:17.087 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:59:17.087 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:59:17.087 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:59:17.087 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:59:17.087 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:59:17.087 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:59:17.087 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:59:17.088 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:59:17.088 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:59:17.088 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:59:17.088 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:59:17.088 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:59:17.088 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:59:17.088 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:59:17.088 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:59:17.088 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:59:17.088 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:59:17.088 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:59:17.088 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:59:17.088 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:59:17.088 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:59:17.088 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:59:17.089 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:59:17.089 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:59:17.089 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:59:17.089 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:59:17.089 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:59:17.089 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:59:17.089 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:59:17.089 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:59:17.089 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44851} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1849648622,v1:172.21.15.134:6827/1849648622] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:59:17.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.097+0000 7f7fd41d4700 1 -- 172.21.15.67:0/3762613302 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7fb806ecc0 msgr2=0x7f7fb8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:17.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.097+0000 7f7fd41d4700 1 --2- 172.21.15.67:0/3762613302 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7fb806ecc0 0x7f7fb8071180 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f7fc800ac50 tx=0x7f7fc800a380 comp rx=0 tx=0).stop 2024-04-22T23:59:17.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.097+0000 7f7fd41d4700 1 -- 172.21.15.67:0/3762613302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fcc102720 msgr2=0x7f7fcc0fffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:17.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.097+0000 7f7fd41d4700 1 --2- 172.21.15.67:0/3762613302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fcc102720 0x7f7fcc0fffa0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc00bd70 tx=0x7f7fbc00bda0 comp rx=0 tx=0).stop 2024-04-22T23:59:17.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.098+0000 7f7fd41d4700 1 -- 172.21.15.67:0/3762613302 shutdown_connections 2024-04-22T23:59:17.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.098+0000 7f7fd41d4700 1 --2- 172.21.15.67:0/3762613302 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7fb806ecc0 0x7f7fb8071180 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:17.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.098+0000 7f7fd41d4700 1 --2- 172.21.15.67:0/3762613302 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7fcc101e00 0x7f7fcc0ffa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:17.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.098+0000 7f7fd41d4700 1 --2- 172.21.15.67:0/3762613302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fcc102720 0x7f7fcc0fffa0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:17.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.098+0000 7f7fd41d4700 1 -- 172.21.15.67:0/3762613302 >> 172.21.15.67:0/3762613302 conn(0x7f7fcc0fb650 msgr2=0x7f7fcc105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:59:17.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.098+0000 7f7fd41d4700 1 -- 172.21.15.67:0/3762613302 shutdown_connections 2024-04-22T23:59:17.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.098+0000 7f7fd41d4700 1 -- 172.21.15.67:0/3762613302 wait complete. 2024-04-22T23:59:17.091 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 33 2024-04-22T23:59:17.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.205+0000 7f983c7bf700 1 -- 172.21.15.67:0/2560831688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98340ffda0 msgr2=0x7f9834100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:17.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.205+0000 7f983c7bf700 1 --2- 172.21.15.67:0/2560831688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98340ffda0 0x7f9834100180 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f982400b0a0 tx=0x7f982400b3b0 comp rx=0 tx=0).stop 2024-04-22T23:59:17.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.205+0000 7f983c7bf700 1 -- 172.21.15.67:0/2560831688 shutdown_connections 2024-04-22T23:59:17.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.205+0000 7f983c7bf700 1 --2- 172.21.15.67:0/2560831688 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9834100750 0x7f98341047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:17.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.205+0000 7f983c7bf700 1 --2- 172.21.15.67:0/2560831688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98340ffda0 0x7f9834100180 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:17.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.205+0000 7f983c7bf700 1 -- 172.21.15.67:0/2560831688 >> 172.21.15.67:0/2560831688 conn(0x7f98340fb650 msgr2=0x7f98340fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:59:17.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.205+0000 7f983c7bf700 1 -- 172.21.15.67:0/2560831688 shutdown_connections 2024-04-22T23:59:17.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.205+0000 7f983c7bf700 1 -- 172.21.15.67:0/2560831688 wait complete. 2024-04-22T23:59:17.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.206+0000 7f983c7bf700 1 Processor -- start 2024-04-22T23:59:17.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.206+0000 7f983c7bf700 1 -- start start 2024-04-22T23:59:17.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.206+0000 7f983c7bf700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98340ffda0 0x7f9834101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:17.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.206+0000 7f983c7bf700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9834100750 0x7f9834101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:17.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.206+0000 7f983c7bf700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98341022c0 con 0x7f98340ffda0 2024-04-22T23:59:17.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.206+0000 7f983c7bf700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9834102400 con 0x7f9834100750 2024-04-22T23:59:17.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.206+0000 7f983a55b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98340ffda0 0x7f9834101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:17.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.206+0000 7f983a55b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98340ffda0 0x7f9834101840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32884/0 (socket says 172.21.15.67:32884) 2024-04-22T23:59:17.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.206+0000 7f983a55b700 1 -- 172.21.15.67:0/3080695718 learned_addr learned my addr 172.21.15.67:0/3080695718 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:59:17.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.207+0000 7f983a55b700 1 -- 172.21.15.67:0/3080695718 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9834100750 msgr2=0x7f9834101d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:17.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.207+0000 7f9839d5a700 1 --2- 172.21.15.67:0/3080695718 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9834100750 0x7f9834101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:17.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.207+0000 7f983a55b700 1 --2- 172.21.15.67:0/3080695718 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9834100750 0x7f9834101d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:17.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.207+0000 7f983a55b700 1 -- 172.21.15.67:0/3080695718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9824009d00 con 0x7f98340ffda0 2024-04-22T23:59:17.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.207+0000 7f983a55b700 1 --2- 172.21.15.67:0/3080695718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98340ffda0 0x7f9834101840 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f982400b070 tx=0x7f982400cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:59:17.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.207+0000 7f98337fe700 1 -- 172.21.15.67:0/3080695718 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f982400e040 con 0x7f98340ffda0 2024-04-22T23:59:17.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.207+0000 7f98337fe700 1 -- 172.21.15.67:0/3080695718 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9824003de0 con 0x7f98340ffda0 2024-04-22T23:59:17.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.207+0000 7f983c7bf700 1 -- 172.21.15.67:0/3080695718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9834102680 con 0x7f98340ffda0 2024-04-22T23:59:17.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.207+0000 7f98337fe700 1 -- 172.21.15.67:0/3080695718 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f982401d8e0 con 0x7f98340ffda0 2024-04-22T23:59:17.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.207+0000 7f983c7bf700 1 -- 172.21.15.67:0/3080695718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98341a2df0 con 0x7f98340ffda0 2024-04-22T23:59:17.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.208+0000 7f983c7bf700 1 -- 172.21.15.67:0/3080695718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f983404f2e0 con 0x7f98340ffda0 2024-04-22T23:59:17.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.209+0000 7f98337fe700 1 -- 172.21.15.67:0/3080695718 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9824004040 con 0x7f98340ffda0 2024-04-22T23:59:17.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.209+0000 7f98337fe700 1 --2- 172.21.15.67:0/3080695718 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f981c06ecc0 0x7f981c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:17.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.209+0000 7f98337fe700 1 -- 172.21.15.67:0/3080695718 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f9824090800 con 0x7f98340ffda0 2024-04-22T23:59:17.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.209+0000 7f9839d5a700 1 --2- 172.21.15.67:0/3080695718 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f981c06ecc0 0x7f981c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:17.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.210+0000 7f9839d5a700 1 --2- 172.21.15.67:0/3080695718 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f981c06ecc0 0x7f981c071180 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f9828005950 tx=0x7f98280058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:59:17.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.214+0000 7f98337fe700 1 -- 172.21.15.67:0/3080695718 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f9824059440 con 0x7f98340ffda0 2024-04-22T23:59:17.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.449+0000 7f983c7bf700 1 -- 172.21.15.67:0/3080695718 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f98340fc460 con 0x7f981c06ecc0 2024-04-22T23:59:17.440 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:17 smithi067 ceph-mon[112438]: from='client.35784 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:59:17.440 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:17 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3490506449' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:59:17.440 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:17 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3762613302' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:59:17.440 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:17 smithi067 ceph-mon[112438]: pgmap v989: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:17.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.450+0000 7f98337fe700 1 -- 172.21.15.67:0/3080695718 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f98340fc460 con 0x7f981c06ecc0 2024-04-22T23:59:17.441 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:59:17.441 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:59:17.441 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:59:17.441 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:59:17.441 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:59:17.441 INFO:teuthology.orchestra.run.smithi067.stdout: "mds", 2024-04-22T23:59:17.442 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:59:17.442 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:59:17.442 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:59:17.442 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:59:17.442 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:59:17.442 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T23:59:17.442 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:59:17.442 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:59:17.442 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:59:17.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.453+0000 7f983c7bf700 1 -- 172.21.15.67:0/3080695718 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f981c06ecc0 msgr2=0x7f981c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:17.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.453+0000 7f983c7bf700 1 --2- 172.21.15.67:0/3080695718 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f981c06ecc0 0x7f981c071180 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f9828005950 tx=0x7f98280058e0 comp rx=0 tx=0).stop 2024-04-22T23:59:17.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.453+0000 7f983c7bf700 1 -- 172.21.15.67:0/3080695718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98340ffda0 msgr2=0x7f9834101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:17.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.453+0000 7f983c7bf700 1 --2- 172.21.15.67:0/3080695718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98340ffda0 0x7f9834101840 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f982400b070 tx=0x7f982400cdd0 comp rx=0 tx=0).stop 2024-04-22T23:59:17.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.454+0000 7f983c7bf700 1 -- 172.21.15.67:0/3080695718 shutdown_connections 2024-04-22T23:59:17.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.454+0000 7f983c7bf700 1 --2- 172.21.15.67:0/3080695718 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f981c06ecc0 0x7f981c071180 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:17.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.454+0000 7f983c7bf700 1 --2- 172.21.15.67:0/3080695718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98340ffda0 0x7f9834101840 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:17.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.454+0000 7f983c7bf700 1 --2- 172.21.15.67:0/3080695718 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9834100750 0x7f9834101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:17.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.454+0000 7f983c7bf700 1 -- 172.21.15.67:0/3080695718 >> 172.21.15.67:0/3080695718 conn(0x7f98340fb650 msgr2=0x7f9834103fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:59:17.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.454+0000 7f983c7bf700 1 -- 172.21.15.67:0/3080695718 shutdown_connections 2024-04-22T23:59:17.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.454+0000 7f983c7bf700 1 -- 172.21.15.67:0/3080695718 wait complete. 2024-04-22T23:59:17.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:17 smithi134 ceph-mon[85630]: from='client.35784 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:59:17.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:17 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3490506449' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:59:17.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:17 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3762613302' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:59:17.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:17 smithi134 ceph-mon[85630]: pgmap v989: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:17.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.562+0000 7f9cee844700 1 -- 172.21.15.67:0/3060287471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ce8106db0 msgr2=0x7f9ce8107190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:17.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.562+0000 7f9cee844700 1 --2- 172.21.15.67:0/3060287471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ce8106db0 0x7f9ce8107190 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8009910 tx=0x7f9cd8009c20 comp rx=0 tx=0).stop 2024-04-22T23:59:17.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.562+0000 7f9cee844700 1 -- 172.21.15.67:0/3060287471 shutdown_connections 2024-04-22T23:59:17.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.562+0000 7f9cee844700 1 --2- 172.21.15.67:0/3060287471 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ce80ff410 0x7f9ce80ff890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:17.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.562+0000 7f9cee844700 1 --2- 172.21.15.67:0/3060287471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ce8106db0 0x7f9ce8107190 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:17.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.562+0000 7f9cee844700 1 -- 172.21.15.67:0/3060287471 >> 172.21.15.67:0/3060287471 conn(0x7f9ce8074e30 msgr2=0x7f9ce8075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:59:17.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.562+0000 7f9cee844700 1 -- 172.21.15.67:0/3060287471 shutdown_connections 2024-04-22T23:59:17.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.562+0000 7f9cee844700 1 -- 172.21.15.67:0/3060287471 wait complete. 2024-04-22T23:59:17.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.563+0000 7f9cee844700 1 Processor -- start 2024-04-22T23:59:17.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.563+0000 7f9cee844700 1 -- start start 2024-04-22T23:59:17.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.563+0000 7f9cee844700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ce80ff410 0x7f9ce819c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:17.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.563+0000 7f9cee844700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ce8106db0 0x7f9ce819cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:17.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.563+0000 7f9cee844700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce819d370 con 0x7f9ce8106db0 2024-04-22T23:59:17.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.563+0000 7f9cee844700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce8196820 con 0x7f9ce80ff410 2024-04-22T23:59:17.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.563+0000 7f9ce77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ce8106db0 0x7f9ce819cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:17.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.563+0000 7f9ce77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ce8106db0 0x7f9ce819cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32902/0 (socket says 172.21.15.67:32902) 2024-04-22T23:59:17.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.563+0000 7f9ce77fe700 1 -- 172.21.15.67:0/574585960 learned_addr learned my addr 172.21.15.67:0/574585960 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:59:17.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.563+0000 7f9ce7fff700 1 --2- 172.21.15.67:0/574585960 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ce80ff410 0x7f9ce819c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:17.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.563+0000 7f9ce77fe700 1 -- 172.21.15.67:0/574585960 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ce80ff410 msgr2=0x7f9ce819c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:17.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.563+0000 7f9ce77fe700 1 --2- 172.21.15.67:0/574585960 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ce80ff410 0x7f9ce819c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:17.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.563+0000 7f9ce77fe700 1 -- 172.21.15.67:0/574585960 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd80095f0 con 0x7f9ce8106db0 2024-04-22T23:59:17.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.563+0000 7f9ce7fff700 1 --2- 172.21.15.67:0/574585960 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ce80ff410 0x7f9ce819c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:59:17.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.564+0000 7f9ce77fe700 1 --2- 172.21.15.67:0/574585960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ce8106db0 0x7f9ce819cce0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f9cd000e920 tx=0x7f9cd000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:59:17.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.564+0000 7f9ce57fa700 1 -- 172.21.15.67:0/574585960 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9cd0009790 con 0x7f9ce8106db0 2024-04-22T23:59:17.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.564+0000 7f9cee844700 1 -- 172.21.15.67:0/574585960 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ce8196b00 con 0x7f9ce8106db0 2024-04-22T23:59:17.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.564+0000 7f9ce57fa700 1 -- 172.21.15.67:0/574585960 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9cd000cba0 con 0x7f9ce8106db0 2024-04-22T23:59:17.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.564+0000 7f9ce57fa700 1 -- 172.21.15.67:0/574585960 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9cd0018650 con 0x7f9ce8106db0 2024-04-22T23:59:17.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.564+0000 7f9cee844700 1 -- 172.21.15.67:0/574585960 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ce8197050 con 0x7f9ce8106db0 2024-04-22T23:59:17.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.565+0000 7f9cee844700 1 -- 172.21.15.67:0/574585960 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ce8109300 con 0x7f9ce8106db0 2024-04-22T23:59:17.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.565+0000 7f9ce57fa700 1 -- 172.21.15.67:0/574585960 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9cd000cd10 con 0x7f9ce8106db0 2024-04-22T23:59:17.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.566+0000 7f9ce57fa700 1 --2- 172.21.15.67:0/574585960 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9cd406ecc0 0x7f9cd4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:17.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.566+0000 7f9ce57fa700 1 -- 172.21.15.67:0/574585960 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f9cd0014070 con 0x7f9ce8106db0 2024-04-22T23:59:17.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.566+0000 7f9ce7fff700 1 --2- 172.21.15.67:0/574585960 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9cd406ecc0 0x7f9cd4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:17.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.566+0000 7f9ce7fff700 1 --2- 172.21.15.67:0/574585960 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9cd406ecc0 0x7f9cd4071180 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f9cd800b760 tx=0x7f9cd8005940 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:59:17.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.571+0000 7f9ce57fa700 1 -- 172.21.15.67:0/574585960 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f9cd0054150 con 0x7f9ce8106db0 2024-04-22T23:59:17.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.870+0000 7f9cee844700 1 -- 172.21.15.67:0/574585960 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f9ce804f2e0 con 0x7f9ce8106db0 2024-04-22T23:59:17.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.870+0000 7f9ce57fa700 1 -- 172.21.15.67:0/574585960 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4505 (secure 0 0 0) 0x7f9cd005b8b0 con 0x7f9ce8106db0 2024-04-22T23:59:17.861 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:59:17.861 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T23:59:17.861 INFO:teuthology.orchestra.run.smithi067.stdout: host smithi134 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-22T23:59:17.861 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T23:59:17.861 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:59:17.861 INFO:teuthology.orchestra.run.smithi067.stdout:Traceback (most recent call last): 2024-04-22T23:59:17.861 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T23:59:17.862 INFO:teuthology.orchestra.run.smithi067.stdout: return _run_code(code, main_globals, None, 2024-04-22T23:59:17.862 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T23:59:17.862 INFO:teuthology.orchestra.run.smithi067.stdout: exec(code, run_globals) 2024-04-22T23:59:17.862 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5642, in 2024-04-22T23:59:17.862 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5630, in main 2024-04-22T23:59:17.862 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 408, in _infer_config 2024-04-22T23:59:17.862 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 323, in _infer_fsid 2024-04-22T23:59:17.862 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 436, in _infer_image 2024-04-22T23:59:17.862 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 310, in _validate_fsid 2024-04-22T23:59:17.862 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 3380, in command_ceph_volume 2024-04-22T23:59:17.862 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T23:59:17.862 INFO:teuthology.orchestra.run.smithi067.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:59:17.862 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:59:17.862 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:59:17.863 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:59:17.863 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:59:17.863 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:59:17.863 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:59:17.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.872+0000 7f9cee844700 1 -- 172.21.15.67:0/574585960 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9cd406ecc0 msgr2=0x7f9cd4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:17.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.872+0000 7f9cee844700 1 --2- 172.21.15.67:0/574585960 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9cd406ecc0 0x7f9cd4071180 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f9cd800b760 tx=0x7f9cd8005940 comp rx=0 tx=0).stop 2024-04-22T23:59:17.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.872+0000 7f9cee844700 1 -- 172.21.15.67:0/574585960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ce8106db0 msgr2=0x7f9ce819cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:17.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.872+0000 7f9cee844700 1 --2- 172.21.15.67:0/574585960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ce8106db0 0x7f9ce819cce0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f9cd000e920 tx=0x7f9cd000ece0 comp rx=0 tx=0).stop 2024-04-22T23:59:17.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.872+0000 7f9cee844700 1 -- 172.21.15.67:0/574585960 shutdown_connections 2024-04-22T23:59:17.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.872+0000 7f9cee844700 1 --2- 172.21.15.67:0/574585960 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f9cd406ecc0 0x7f9cd4071180 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:17.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.872+0000 7f9cee844700 1 --2- 172.21.15.67:0/574585960 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ce80ff410 0x7f9ce819c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:17.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.872+0000 7f9cee844700 1 --2- 172.21.15.67:0/574585960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ce8106db0 0x7f9ce819cce0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:17.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.872+0000 7f9cee844700 1 -- 172.21.15.67:0/574585960 >> 172.21.15.67:0/574585960 conn(0x7f9ce8074e30 msgr2=0x7f9ce80fd460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:59:17.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.872+0000 7f9cee844700 1 -- 172.21.15.67:0/574585960 shutdown_connections 2024-04-22T23:59:17.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:17.872+0000 7f9cee844700 1 -- 172.21.15.67:0/574585960 wait complete. 2024-04-22T23:59:18.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:18 smithi134 ceph-mon[85630]: from='client.35796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:59:18.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:18 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/574585960' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:59:18.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:18 smithi067 ceph-mon[112438]: from='client.35796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:59:18.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:18 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/574585960' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:59:19.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:19 smithi134 ceph-mon[85630]: pgmap v990: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:19.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:19 smithi067 ceph-mon[112438]: pgmap v990: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:21.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:21 smithi134 ceph-mon[85630]: pgmap v991: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:21.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:21 smithi067 ceph-mon[112438]: pgmap v991: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:23.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:23 smithi134 ceph-mon[85630]: pgmap v992: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:23.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:23 smithi067 ceph-mon[112438]: pgmap v992: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:25.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:25 smithi134 ceph-mon[85630]: pgmap v993: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:25.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:25 smithi067 ceph-mon[112438]: pgmap v993: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:26.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:59:26.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:59:27.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:27 smithi134 ceph-mon[85630]: pgmap v994: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:27.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:27 smithi067 ceph-mon[112438]: pgmap v994: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:29.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:29 smithi134 ceph-mon[85630]: pgmap v995: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:29.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:29 smithi067 ceph-mon[112438]: pgmap v995: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:31.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:31 smithi134 ceph-mon[85630]: pgmap v996: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:31.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:31 smithi067 ceph-mon[112438]: pgmap v996: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:33.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:33 smithi134 ceph-mon[85630]: pgmap v997: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:33.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:33 smithi067 ceph-mon[112438]: pgmap v997: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:35.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:35 smithi134 ceph-mon[85630]: pgmap v998: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:35.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:35 smithi067 ceph-mon[112438]: pgmap v998: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:37.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:37 smithi134 ceph-mon[85630]: pgmap v999: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:37.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:37 smithi067 ceph-mon[112438]: pgmap v999: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:39.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:39 smithi134 ceph-mon[85630]: pgmap v1000: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:39.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:39 smithi067 ceph-mon[112438]: pgmap v1000: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:41.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:59:41.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:59:42.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:42 smithi134 ceph-mon[85630]: pgmap v1001: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:42.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:42 smithi067 ceph-mon[112438]: pgmap v1001: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:43.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:43 smithi134 ceph-mon[85630]: pgmap v1002: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:43.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:43 smithi067 ceph-mon[112438]: pgmap v1002: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:45.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:45 smithi134 ceph-mon[85630]: pgmap v1003: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:45.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:45 smithi067 ceph-mon[112438]: pgmap v1003: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:47.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:47 smithi134 ceph-mon[85630]: pgmap v1004: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:47.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:47 smithi067 ceph-mon[112438]: pgmap v1004: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:47.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:47.993+0000 7ff06c708700 1 -- 172.21.15.67:0/837972007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0640fff50 msgr2=0x7ff064105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:47.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:47.993+0000 7ff06c708700 1 --2- 172.21.15.67:0/837972007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0640fff50 0x7ff064105840 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7ff054009910 tx=0x7ff054009c20 comp rx=0 tx=0).stop 2024-04-22T23:59:47.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:47.993+0000 7ff06c708700 1 -- 172.21.15.67:0/837972007 shutdown_connections 2024-04-22T23:59:47.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:47.993+0000 7ff06c708700 1 --2- 172.21.15.67:0/837972007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0640fff50 0x7ff064105840 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:47.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:47.993+0000 7ff06c708700 1 --2- 172.21.15.67:0/837972007 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff0640ff5a0 0x7ff0640ff980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:47.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:47.993+0000 7ff06c708700 1 -- 172.21.15.67:0/837972007 >> 172.21.15.67:0/837972007 conn(0x7ff064074e30 msgr2=0x7ff064075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:59:47.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:47.993+0000 7ff06c708700 1 -- 172.21.15.67:0/837972007 shutdown_connections 2024-04-22T23:59:47.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:47.993+0000 7ff06c708700 1 -- 172.21.15.67:0/837972007 wait complete. 2024-04-22T23:59:47.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:47.993+0000 7ff06c708700 1 Processor -- start 2024-04-22T23:59:47.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:47.993+0000 7ff06c708700 1 -- start start 2024-04-22T23:59:47.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:47.994+0000 7ff06c708700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff0640ff5a0 0x7ff06419e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:47.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:47.994+0000 7ff06c708700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0640fff50 0x7ff06419ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:47.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:47.994+0000 7ff06c708700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff06419f4f0 con 0x7ff0640fff50 2024-04-22T23:59:47.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:47.994+0000 7ff06c708700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0641989a0 con 0x7ff0640ff5a0 2024-04-22T23:59:47.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:47.994+0000 7ff069ca3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0640fff50 0x7ff06419ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:47.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:47.994+0000 7ff069ca3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0640fff50 0x7ff06419ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52216/0 (socket says 172.21.15.67:52216) 2024-04-22T23:59:47.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:47.994+0000 7ff069ca3700 1 -- 172.21.15.67:0/3353568517 learned_addr learned my addr 172.21.15.67:0/3353568517 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:59:47.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:47.994+0000 7ff06a4a4700 1 --2- 172.21.15.67:0/3353568517 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff0640ff5a0 0x7ff06419e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:47.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:47.994+0000 7ff069ca3700 1 -- 172.21.15.67:0/3353568517 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff0640ff5a0 msgr2=0x7ff06419e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:47.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:47.994+0000 7ff069ca3700 1 --2- 172.21.15.67:0/3353568517 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff0640ff5a0 0x7ff06419e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:47.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:47.994+0000 7ff069ca3700 1 -- 172.21.15.67:0/3353568517 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0540095f0 con 0x7ff0640fff50 2024-04-22T23:59:47.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:47.994+0000 7ff069ca3700 1 --2- 172.21.15.67:0/3353568517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0640fff50 0x7ff06419ee60 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7ff0540092e0 tx=0x7ff05400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:59:47.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:47.995+0000 7ff05b7fe700 1 -- 172.21.15.67:0/3353568517 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff05401d070 con 0x7ff0640fff50 2024-04-22T23:59:47.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:47.995+0000 7ff05b7fe700 1 -- 172.21.15.67:0/3353568517 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff05400f460 con 0x7ff0640fff50 2024-04-22T23:59:47.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:47.995+0000 7ff06c708700 1 -- 172.21.15.67:0/3353568517 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff064198c20 con 0x7ff0640fff50 2024-04-22T23:59:47.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:47.995+0000 7ff05b7fe700 1 -- 172.21.15.67:0/3353568517 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0540175d0 con 0x7ff0640fff50 2024-04-22T23:59:47.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:47.995+0000 7ff06c708700 1 -- 172.21.15.67:0/3353568517 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff064199110 con 0x7ff0640fff50 2024-04-22T23:59:47.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:47.996+0000 7ff06c708700 1 -- 172.21.15.67:0/3353568517 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff064104fc0 con 0x7ff0640fff50 2024-04-22T23:59:47.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:47.997+0000 7ff05b7fe700 1 -- 172.21.15.67:0/3353568517 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff05400f5d0 con 0x7ff0640fff50 2024-04-22T23:59:47.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:47.997+0000 7ff05b7fe700 1 --2- 172.21.15.67:0/3353568517 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff05006ecc0 0x7ff050071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:47.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:47.997+0000 7ff05b7fe700 1 -- 172.21.15.67:0/3353568517 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7ff054090470 con 0x7ff0640fff50 2024-04-22T23:59:47.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:47.997+0000 7ff06a4a4700 1 --2- 172.21.15.67:0/3353568517 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff05006ecc0 0x7ff050071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:47.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:47.998+0000 7ff06a4a4700 1 --2- 172.21.15.67:0/3353568517 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff05006ecc0 0x7ff050071180 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7ff060009a10 tx=0x7ff060009190 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:59:47.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.002+0000 7ff05b7fe700 1 -- 172.21.15.67:0/3353568517 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff05405cbc0 con 0x7ff0640fff50 2024-04-22T23:59:48.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.235+0000 7ff06c708700 1 -- 172.21.15.67:0/3353568517 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff064199380 con 0x7ff05006ecc0 2024-04-22T23:59:48.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.236+0000 7ff05b7fe700 1 -- 172.21.15.67:0/3353568517 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff064199380 con 0x7ff05006ecc0 2024-04-22T23:59:48.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.238+0000 7ff06c708700 1 -- 172.21.15.67:0/3353568517 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff05006ecc0 msgr2=0x7ff050071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:48.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.238+0000 7ff06c708700 1 --2- 172.21.15.67:0/3353568517 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff05006ecc0 0x7ff050071180 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7ff060009a10 tx=0x7ff060009190 comp rx=0 tx=0).stop 2024-04-22T23:59:48.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.238+0000 7ff06c708700 1 -- 172.21.15.67:0/3353568517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0640fff50 msgr2=0x7ff06419ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:48.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.238+0000 7ff06c708700 1 --2- 172.21.15.67:0/3353568517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0640fff50 0x7ff06419ee60 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7ff0540092e0 tx=0x7ff05400bd70 comp rx=0 tx=0).stop 2024-04-22T23:59:48.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.238+0000 7ff06c708700 1 -- 172.21.15.67:0/3353568517 shutdown_connections 2024-04-22T23:59:48.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.238+0000 7ff06c708700 1 --2- 172.21.15.67:0/3353568517 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff05006ecc0 0x7ff050071180 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:48.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.238+0000 7ff06c708700 1 --2- 172.21.15.67:0/3353568517 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff0640ff5a0 0x7ff06419e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:48.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.238+0000 7ff06c708700 1 --2- 172.21.15.67:0/3353568517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0640fff50 0x7ff06419ee60 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:48.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.238+0000 7ff06c708700 1 -- 172.21.15.67:0/3353568517 >> 172.21.15.67:0/3353568517 conn(0x7ff064074e30 msgr2=0x7ff0640fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:59:48.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.238+0000 7ff06c708700 1 -- 172.21.15.67:0/3353568517 shutdown_connections 2024-04-22T23:59:48.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.238+0000 7ff06c708700 1 -- 172.21.15.67:0/3353568517 wait complete. 2024-04-22T23:59:48.240 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-22T23:59:48.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.346+0000 7fca8a49b700 1 -- 172.21.15.67:0/2029701329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca840ffda0 msgr2=0x7fca84100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:48.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.346+0000 7fca8a49b700 1 --2- 172.21.15.67:0/2029701329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca840ffda0 0x7fca84100180 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fca74009910 tx=0x7fca74009c20 comp rx=0 tx=0).stop 2024-04-22T23:59:48.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.346+0000 7fca8a49b700 1 -- 172.21.15.67:0/2029701329 shutdown_connections 2024-04-22T23:59:48.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.346+0000 7fca8a49b700 1 --2- 172.21.15.67:0/2029701329 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fca84100750 0x7fca841047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:48.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.346+0000 7fca8a49b700 1 --2- 172.21.15.67:0/2029701329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca840ffda0 0x7fca84100180 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:48.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.346+0000 7fca8a49b700 1 -- 172.21.15.67:0/2029701329 >> 172.21.15.67:0/2029701329 conn(0x7fca840fb650 msgr2=0x7fca840fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:59:48.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.346+0000 7fca8a49b700 1 -- 172.21.15.67:0/2029701329 shutdown_connections 2024-04-22T23:59:48.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.346+0000 7fca8a49b700 1 -- 172.21.15.67:0/2029701329 wait complete. 2024-04-22T23:59:48.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.347+0000 7fca8a49b700 1 Processor -- start 2024-04-22T23:59:48.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.347+0000 7fca8a49b700 1 -- start start 2024-04-22T23:59:48.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.347+0000 7fca8a49b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca840ffda0 0x7fca8419c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:48.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.347+0000 7fca8a49b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fca84100750 0x7fca8419cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:48.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.347+0000 7fca8a49b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8419d2b0 con 0x7fca840ffda0 2024-04-22T23:59:48.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.347+0000 7fca8a49b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca84196760 con 0x7fca84100750 2024-04-22T23:59:48.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.347+0000 7fca83fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca840ffda0 0x7fca8419c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:48.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.348+0000 7fca83fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca840ffda0 0x7fca8419c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52246/0 (socket says 172.21.15.67:52246) 2024-04-22T23:59:48.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.348+0000 7fca83fff700 1 -- 172.21.15.67:0/172351948 learned_addr learned my addr 172.21.15.67:0/172351948 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:59:48.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.348+0000 7fca83fff700 1 -- 172.21.15.67:0/172351948 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fca84100750 msgr2=0x7fca8419cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:48.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.348+0000 7fca837fe700 1 --2- 172.21.15.67:0/172351948 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fca84100750 0x7fca8419cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:48.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.348+0000 7fca83fff700 1 --2- 172.21.15.67:0/172351948 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fca84100750 0x7fca8419cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:48.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.348+0000 7fca83fff700 1 -- 172.21.15.67:0/172351948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca740095f0 con 0x7fca840ffda0 2024-04-22T23:59:48.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.348+0000 7fca837fe700 1 --2- 172.21.15.67:0/172351948 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fca84100750 0x7fca8419cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T23:59:48.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.348+0000 7fca83fff700 1 --2- 172.21.15.67:0/172351948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca840ffda0 0x7fca8419c6e0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fca74006010 tx=0x7fca7400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:59:48.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.348+0000 7fca817fa700 1 -- 172.21.15.67:0/172351948 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca7401d070 con 0x7fca840ffda0 2024-04-22T23:59:48.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.348+0000 7fca8a49b700 1 -- 172.21.15.67:0/172351948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca841969e0 con 0x7fca840ffda0 2024-04-22T23:59:48.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.348+0000 7fca817fa700 1 -- 172.21.15.67:0/172351948 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fca7400f460 con 0x7fca840ffda0 2024-04-22T23:59:48.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.348+0000 7fca817fa700 1 -- 172.21.15.67:0/172351948 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca74017630 con 0x7fca840ffda0 2024-04-22T23:59:48.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.348+0000 7fca8a49b700 1 -- 172.21.15.67:0/172351948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca84196ed0 con 0x7fca840ffda0 2024-04-22T23:59:48.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.349+0000 7fca8a49b700 1 -- 172.21.15.67:0/172351948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca8404f2e0 con 0x7fca840ffda0 2024-04-22T23:59:48.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.350+0000 7fca817fa700 1 -- 172.21.15.67:0/172351948 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fca7400f5d0 con 0x7fca840ffda0 2024-04-22T23:59:48.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.350+0000 7fca817fa700 1 --2- 172.21.15.67:0/172351948 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fca6c06ecc0 0x7fca6c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:48.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.350+0000 7fca817fa700 1 -- 172.21.15.67:0/172351948 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fca74090470 con 0x7fca840ffda0 2024-04-22T23:59:48.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.350+0000 7fca837fe700 1 --2- 172.21.15.67:0/172351948 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fca6c06ecc0 0x7fca6c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:48.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.351+0000 7fca837fe700 1 --2- 172.21.15.67:0/172351948 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fca6c06ecc0 0x7fca6c071180 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fca78005fd0 tx=0x7fca78005ee0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:59:48.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.355+0000 7fca817fa700 1 -- 172.21.15.67:0/172351948 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fca7405cbe0 con 0x7fca840ffda0 2024-04-22T23:59:48.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.584+0000 7fca8a49b700 1 -- 172.21.15.67:0/172351948 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fca84197160 con 0x7fca6c06ecc0 2024-04-22T23:59:48.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.585+0000 7fca817fa700 1 -- 172.21.15.67:0/172351948 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fca84197160 con 0x7fca6c06ecc0 2024-04-22T23:59:48.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.587+0000 7fca8a49b700 1 -- 172.21.15.67:0/172351948 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fca6c06ecc0 msgr2=0x7fca6c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:48.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.587+0000 7fca8a49b700 1 --2- 172.21.15.67:0/172351948 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fca6c06ecc0 0x7fca6c071180 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fca78005fd0 tx=0x7fca78005ee0 comp rx=0 tx=0).stop 2024-04-22T23:59:48.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.587+0000 7fca8a49b700 1 -- 172.21.15.67:0/172351948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca840ffda0 msgr2=0x7fca8419c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:48.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.587+0000 7fca8a49b700 1 --2- 172.21.15.67:0/172351948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca840ffda0 0x7fca8419c6e0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fca74006010 tx=0x7fca7400be20 comp rx=0 tx=0).stop 2024-04-22T23:59:48.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.587+0000 7fca8a49b700 1 -- 172.21.15.67:0/172351948 shutdown_connections 2024-04-22T23:59:48.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.587+0000 7fca8a49b700 1 --2- 172.21.15.67:0/172351948 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fca6c06ecc0 0x7fca6c071180 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:48.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.587+0000 7fca8a49b700 1 --2- 172.21.15.67:0/172351948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca840ffda0 0x7fca8419c6e0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:48.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.587+0000 7fca8a49b700 1 --2- 172.21.15.67:0/172351948 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fca84100750 0x7fca8419cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:48.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.587+0000 7fca8a49b700 1 -- 172.21.15.67:0/172351948 >> 172.21.15.67:0/172351948 conn(0x7fca840fb650 msgr2=0x7fca84103f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:59:48.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.587+0000 7fca8a49b700 1 -- 172.21.15.67:0/172351948 shutdown_connections 2024-04-22T23:59:48.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.587+0000 7fca8a49b700 1 -- 172.21.15.67:0/172351948 wait complete. 2024-04-22T23:59:48.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.694+0000 7f171248c700 1 -- 172.21.15.67:0/4046724107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f170c100ea0 msgr2=0x7f170c101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:48.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.694+0000 7f171248c700 1 --2- 172.21.15.67:0/4046724107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f170c100ea0 0x7f170c101280 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f16f4009910 tx=0x7f16f4009c20 comp rx=0 tx=0).stop 2024-04-22T23:59:48.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.694+0000 7f171248c700 1 -- 172.21.15.67:0/4046724107 shutdown_connections 2024-04-22T23:59:48.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.694+0000 7f171248c700 1 --2- 172.21.15.67:0/4046724107 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f170c101850 0x7f170c1058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:48.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.694+0000 7f171248c700 1 --2- 172.21.15.67:0/4046724107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f170c100ea0 0x7f170c101280 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:48.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.694+0000 7f171248c700 1 -- 172.21.15.67:0/4046724107 >> 172.21.15.67:0/4046724107 conn(0x7f170c0fc750 msgr2=0x7f170c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:59:48.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.694+0000 7f171248c700 1 -- 172.21.15.67:0/4046724107 shutdown_connections 2024-04-22T23:59:48.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.694+0000 7f171248c700 1 -- 172.21.15.67:0/4046724107 wait complete. 2024-04-22T23:59:48.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.695+0000 7f171248c700 1 Processor -- start 2024-04-22T23:59:48.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.695+0000 7f171248c700 1 -- start start 2024-04-22T23:59:48.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.695+0000 7f171248c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f170c100ea0 0x7f170c073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:48.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.695+0000 7f171248c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f170c101850 0x7f170c0735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:48.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.695+0000 7f171248c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f170c073b10 con 0x7f170c101850 2024-04-22T23:59:48.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.695+0000 7f171248c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f170c073c50 con 0x7f170c100ea0 2024-04-22T23:59:48.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.695+0000 7f170b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f170c101850 0x7f170c0735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:48.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.695+0000 7f170b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f170c101850 0x7f170c0735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52276/0 (socket says 172.21.15.67:52276) 2024-04-22T23:59:48.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.695+0000 7f170b7fe700 1 -- 172.21.15.67:0/3086636564 learned_addr learned my addr 172.21.15.67:0/3086636564 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:59:48.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.696+0000 7f170bfff700 1 --2- 172.21.15.67:0/3086636564 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f170c100ea0 0x7f170c073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:48.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.696+0000 7f170b7fe700 1 -- 172.21.15.67:0/3086636564 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f170c100ea0 msgr2=0x7f170c073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:48.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.696+0000 7f170b7fe700 1 --2- 172.21.15.67:0/3086636564 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f170c100ea0 0x7f170c073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:48.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.696+0000 7f170b7fe700 1 -- 172.21.15.67:0/3086636564 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f40095f0 con 0x7f170c101850 2024-04-22T23:59:48.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.696+0000 7f170b7fe700 1 --2- 172.21.15.67:0/3086636564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f170c101850 0x7f170c0735d0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f16fc00b480 tx=0x7f16fc00b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:59:48.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.696+0000 7f17097fa700 1 -- 172.21.15.67:0/3086636564 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16fc011800 con 0x7f170c101850 2024-04-22T23:59:48.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.696+0000 7f17097fa700 1 -- 172.21.15.67:0/3086636564 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f16fc011e20 con 0x7f170c101850 2024-04-22T23:59:48.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.696+0000 7f171248c700 1 -- 172.21.15.67:0/3086636564 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f170c1a2b60 con 0x7f170c101850 2024-04-22T23:59:48.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.696+0000 7f17097fa700 1 -- 172.21.15.67:0/3086636564 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16fc010ea0 con 0x7f170c101850 2024-04-22T23:59:48.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.696+0000 7f171248c700 1 -- 172.21.15.67:0/3086636564 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f170c1a2f70 con 0x7f170c101850 2024-04-22T23:59:48.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.697+0000 7f171248c700 1 -- 172.21.15.67:0/3086636564 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f170c109240 con 0x7f170c101850 2024-04-22T23:59:48.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.698+0000 7f17097fa700 1 -- 172.21.15.67:0/3086636564 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f16fc010380 con 0x7f170c101850 2024-04-22T23:59:48.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.698+0000 7f17097fa700 1 --2- 172.21.15.67:0/3086636564 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f16f806ecc0 0x7f16f8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:48.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.698+0000 7f170bfff700 1 --2- 172.21.15.67:0/3086636564 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f16f806ecc0 0x7f16f8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:48.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.698+0000 7f17097fa700 1 -- 172.21.15.67:0/3086636564 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f16fc08ea00 con 0x7f170c101850 2024-04-22T23:59:48.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.699+0000 7f170bfff700 1 --2- 172.21.15.67:0/3086636564 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f16f806ecc0 0x7f16f8071180 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f16f4005f50 tx=0x7f16f401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:59:48.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.703+0000 7f17097fa700 1 -- 172.21.15.67:0/3086636564 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f16fc0576c0 con 0x7f170c101850 2024-04-22T23:59:48.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.922+0000 7f171248c700 1 -- 172.21.15.67:0/3086636564 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f170c0611d0 con 0x7f16f806ecc0 2024-04-22T23:59:48.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.933+0000 7f17097fa700 1 -- 172.21.15.67:0/3086636564 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f170c0611d0 con 0x7f16f806ecc0 2024-04-22T23:59:48.926 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T23:59:48.926 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (33m) 11m ago 44m 24.4M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-22T23:59:48.926 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (45m) 11m ago 45m 10.0M - 18.2.1 5be31c24972a df887f62291e 2024-04-22T23:59:48.927 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (44m) 11m ago 44m 10.0M - 18.2.1 5be31c24972a ad89190b5400 2024-04-22T23:59:48.927 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (30m) 11m ago 44m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-22T23:59:48.927 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (30m) 11m ago 43m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-22T23:59:48.927 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (33m) 11m ago 44m 81.2M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-22T23:59:48.927 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (27m) 11m ago 39m 19.4M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-22T23:59:48.927 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (26m) 11m ago 39m 20.3M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-22T23:59:48.928 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (26m) 11m ago 39m 20.1M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-22T23:59:48.928 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (11m) 11m ago 39m 14.0M - 19.0.0-3252-g43be0201 a0434699c1d7 46d3d7e86fee 2024-04-22T23:59:48.928 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (35m) 11m ago 46m 514M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-22T23:59:48.928 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (34m) 11m ago 43m 459M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-22T23:59:48.928 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (32m) 11m ago 46m 74.2M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-22T23:59:48.928 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (31m) 11m ago 43m 54.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-22T23:59:48.928 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (34m) 11m ago 44m 15.2M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-22T23:59:48.929 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (34m) 11m ago 43m 16.0M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-22T23:59:48.929 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (30m) 11m ago 43m 102M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-22T23:59:48.929 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (29m) 11m ago 42m 128M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-22T23:59:48.929 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (29m) 11m ago 42m 83.8M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-22T23:59:48.929 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (28m) 11m ago 41m 101M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-22T23:59:48.929 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (28m) 11m ago 41m 86.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-22T23:59:48.929 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (27m) 11m ago 40m 81.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-22T23:59:48.929 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (33m) 11m ago 44m 78.7M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-22T23:59:48.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.935+0000 7f171248c700 1 -- 172.21.15.67:0/3086636564 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f16f806ecc0 msgr2=0x7f16f8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:48.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.935+0000 7f171248c700 1 --2- 172.21.15.67:0/3086636564 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f16f806ecc0 0x7f16f8071180 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f16f4005f50 tx=0x7f16f401a040 comp rx=0 tx=0).stop 2024-04-22T23:59:48.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.936+0000 7f171248c700 1 -- 172.21.15.67:0/3086636564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f170c101850 msgr2=0x7f170c0735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:48.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.936+0000 7f171248c700 1 --2- 172.21.15.67:0/3086636564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f170c101850 0x7f170c0735d0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f16fc00b480 tx=0x7f16fc00b790 comp rx=0 tx=0).stop 2024-04-22T23:59:48.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.936+0000 7f171248c700 1 -- 172.21.15.67:0/3086636564 shutdown_connections 2024-04-22T23:59:48.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.936+0000 7f171248c700 1 --2- 172.21.15.67:0/3086636564 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f16f806ecc0 0x7f16f8071180 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:48.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.936+0000 7f171248c700 1 --2- 172.21.15.67:0/3086636564 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f170c100ea0 0x7f170c073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:48.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.936+0000 7f171248c700 1 --2- 172.21.15.67:0/3086636564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f170c101850 0x7f170c0735d0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:48.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.936+0000 7f171248c700 1 -- 172.21.15.67:0/3086636564 >> 172.21.15.67:0/3086636564 conn(0x7f170c0fc750 msgr2=0x7f170c0fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:59:48.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.936+0000 7f171248c700 1 -- 172.21.15.67:0/3086636564 shutdown_connections 2024-04-22T23:59:48.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:48.936+0000 7f171248c700 1 -- 172.21.15.67:0/3086636564 wait complete. 2024-04-22T23:59:49.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.041+0000 7f92b31af700 1 -- 172.21.15.67:0/2748947912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92ac101850 msgr2=0x7f92ac1058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:49.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.041+0000 7f92b31af700 1 --2- 172.21.15.67:0/2748947912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92ac101850 0x7f92ac1058a0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f92a0009910 tx=0x7f92a0009c20 comp rx=0 tx=0).stop 2024-04-22T23:59:49.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.041+0000 7f92b31af700 1 -- 172.21.15.67:0/2748947912 shutdown_connections 2024-04-22T23:59:49.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.041+0000 7f92b31af700 1 --2- 172.21.15.67:0/2748947912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92ac101850 0x7f92ac1058a0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:49.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.041+0000 7f92b31af700 1 --2- 172.21.15.67:0/2748947912 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f92ac100ea0 0x7f92ac101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:49.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.041+0000 7f92b31af700 1 -- 172.21.15.67:0/2748947912 >> 172.21.15.67:0/2748947912 conn(0x7f92ac0fc750 msgr2=0x7f92ac0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:59:49.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.041+0000 7f92b31af700 1 -- 172.21.15.67:0/2748947912 shutdown_connections 2024-04-22T23:59:49.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.041+0000 7f92b31af700 1 -- 172.21.15.67:0/2748947912 wait complete. 2024-04-22T23:59:49.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.041+0000 7f92b31af700 1 Processor -- start 2024-04-22T23:59:49.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.042+0000 7f92b31af700 1 -- start start 2024-04-22T23:59:49.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.042+0000 7f92b31af700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92ac100ea0 0x7f92ac19c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:49.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.042+0000 7f92b31af700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f92ac101850 0x7f92ac19cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:49.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.042+0000 7f92b31af700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92ac19d240 con 0x7f92ac100ea0 2024-04-22T23:59:49.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.042+0000 7f92b31af700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92ac196780 con 0x7f92ac101850 2024-04-22T23:59:49.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.042+0000 7f92b0f4b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92ac100ea0 0x7f92ac19c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:49.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.042+0000 7f92b0f4b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92ac100ea0 0x7f92ac19c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56682/0 (socket says 172.21.15.67:56682) 2024-04-22T23:59:49.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.042+0000 7f92b0f4b700 1 -- 172.21.15.67:0/3172494518 learned_addr learned my addr 172.21.15.67:0/3172494518 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:59:49.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.042+0000 7f92abfff700 1 --2- 172.21.15.67:0/3172494518 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f92ac101850 0x7f92ac19cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:49.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.042+0000 7f92b0f4b700 1 -- 172.21.15.67:0/3172494518 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f92ac101850 msgr2=0x7f92ac19cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:49.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.042+0000 7f92b0f4b700 1 --2- 172.21.15.67:0/3172494518 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f92ac101850 0x7f92ac19cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:49.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.042+0000 7f92b0f4b700 1 -- 172.21.15.67:0/3172494518 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a00095f0 con 0x7f92ac100ea0 2024-04-22T23:59:49.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.042+0000 7f92b0f4b700 1 --2- 172.21.15.67:0/3172494518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92ac100ea0 0x7f92ac19c700 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f92a400cfd0 tx=0x7f92a400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:59:49.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.043+0000 7f92a9ffb700 1 -- 172.21.15.67:0/3172494518 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92a400ad70 con 0x7f92ac100ea0 2024-04-22T23:59:49.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.043+0000 7f92a9ffb700 1 -- 172.21.15.67:0/3172494518 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f92a40128e0 con 0x7f92ac100ea0 2024-04-22T23:59:49.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.043+0000 7f92b31af700 1 -- 172.21.15.67:0/3172494518 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92ac196a60 con 0x7f92ac100ea0 2024-04-22T23:59:49.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.043+0000 7f92a9ffb700 1 -- 172.21.15.67:0/3172494518 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92a4011880 con 0x7f92ac100ea0 2024-04-22T23:59:49.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.043+0000 7f92b31af700 1 -- 172.21.15.67:0/3172494518 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92ac196fb0 con 0x7f92ac100ea0 2024-04-22T23:59:49.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.044+0000 7f92b31af700 1 -- 172.21.15.67:0/3172494518 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92ac1092a0 con 0x7f92ac100ea0 2024-04-22T23:59:49.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.044+0000 7f92a9ffb700 1 -- 172.21.15.67:0/3172494518 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f92a4012420 con 0x7f92ac100ea0 2024-04-22T23:59:49.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.044+0000 7f92a9ffb700 1 --2- 172.21.15.67:0/3172494518 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f929406ecc0 0x7f9294071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:49.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.045+0000 7f92a9ffb700 1 -- 172.21.15.67:0/3172494518 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f92a408f070 con 0x7f92ac100ea0 2024-04-22T23:59:49.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.045+0000 7f92abfff700 1 --2- 172.21.15.67:0/3172494518 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f929406ecc0 0x7f9294071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:49.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.045+0000 7f92abfff700 1 --2- 172.21.15.67:0/3172494518 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f929406ecc0 0x7f9294071180 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f92ac197bf0 tx=0x7f92a001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:59:49.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.049+0000 7f92a9ffb700 1 -- 172.21.15.67:0/3172494518 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f92a405b860 con 0x7f92ac100ea0 2024-04-22T23:59:49.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.351+0000 7f92b31af700 1 -- 172.21.15.67:0/3172494518 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f92ac04f2e0 con 0x7f92ac100ea0 2024-04-22T23:59:49.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.351+0000 7f92a9ffb700 1 -- 172.21.15.67:0/3172494518 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f92a405b3f0 con 0x7f92ac100ea0 2024-04-22T23:59:49.343 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:59:49.343 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-22T23:59:49.343 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:59:49.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:59:49.343 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-22T23:59:49.343 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-22T23:59:49.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:59:49.343 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-22T23:59:49.343 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-22T23:59:49.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:59:49.344 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-22T23:59:49.344 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 4 2024-04-22T23:59:49.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-22T23:59:49.344 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-22T23:59:49.344 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 14 2024-04-22T23:59:49.344 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-22T23:59:49.344 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:59:49.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.353+0000 7f92b31af700 1 -- 172.21.15.67:0/3172494518 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f929406ecc0 msgr2=0x7f9294071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:49.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.353+0000 7f92b31af700 1 --2- 172.21.15.67:0/3172494518 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f929406ecc0 0x7f9294071180 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f92ac197bf0 tx=0x7f92a001a040 comp rx=0 tx=0).stop 2024-04-22T23:59:49.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.353+0000 7f92b31af700 1 -- 172.21.15.67:0/3172494518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92ac100ea0 msgr2=0x7f92ac19c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:49.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.353+0000 7f92b31af700 1 --2- 172.21.15.67:0/3172494518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92ac100ea0 0x7f92ac19c700 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f92a400cfd0 tx=0x7f92a400a340 comp rx=0 tx=0).stop 2024-04-22T23:59:49.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.353+0000 7f92b31af700 1 -- 172.21.15.67:0/3172494518 shutdown_connections 2024-04-22T23:59:49.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.353+0000 7f92b31af700 1 --2- 172.21.15.67:0/3172494518 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f929406ecc0 0x7f9294071180 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:49.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.353+0000 7f92b31af700 1 --2- 172.21.15.67:0/3172494518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92ac100ea0 0x7f92ac19c700 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:49.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.353+0000 7f92b31af700 1 --2- 172.21.15.67:0/3172494518 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f92ac101850 0x7f92ac19cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:49.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.353+0000 7f92b31af700 1 -- 172.21.15.67:0/3172494518 >> 172.21.15.67:0/3172494518 conn(0x7f92ac0fc750 msgr2=0x7f92ac0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:59:49.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.354+0000 7f92b31af700 1 -- 172.21.15.67:0/3172494518 shutdown_connections 2024-04-22T23:59:49.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.354+0000 7f92b31af700 1 -- 172.21.15.67:0/3172494518 wait complete. 2024-04-22T23:59:49.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.460+0000 7fa88b89e700 1 -- 172.21.15.67:0/804657107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa884068580 msgr2=0x7fa8840689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:49.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.460+0000 7fa88b89e700 1 --2- 172.21.15.67:0/804657107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa884068580 0x7fa8840689e0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fa880009910 tx=0x7fa880009c20 comp rx=0 tx=0).stop 2024-04-22T23:59:49.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.460+0000 7fa88b89e700 1 -- 172.21.15.67:0/804657107 shutdown_connections 2024-04-22T23:59:49.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.460+0000 7fa88b89e700 1 --2- 172.21.15.67:0/804657107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa884068580 0x7fa8840689e0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:49.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.460+0000 7fa88b89e700 1 --2- 172.21.15.67:0/804657107 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa884106cd0 0x7fa8841070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:49.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.460+0000 7fa88b89e700 1 -- 172.21.15.67:0/804657107 >> 172.21.15.67:0/804657107 conn(0x7fa884075670 msgr2=0x7fa884075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:59:49.452 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:49 smithi067 ceph-mon[112438]: from='client.35804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:59:49.452 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:49 smithi067 ceph-mon[112438]: from='client.35808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:59:49.452 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:49 smithi067 ceph-mon[112438]: from='client.35812 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:59:49.453 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:49 smithi067 ceph-mon[112438]: pgmap v1005: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:49.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.460+0000 7fa88b89e700 1 -- 172.21.15.67:0/804657107 shutdown_connections 2024-04-22T23:59:49.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.460+0000 7fa88b89e700 1 -- 172.21.15.67:0/804657107 wait complete. 2024-04-22T23:59:49.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.461+0000 7fa88b89e700 1 Processor -- start 2024-04-22T23:59:49.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.461+0000 7fa88b89e700 1 -- start start 2024-04-22T23:59:49.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.461+0000 7fa88b89e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa884068580 0x7fa88419c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:49.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.461+0000 7fa88b89e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa884106cd0 0x7fa88419cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:49.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.461+0000 7fa88b89e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa88419d170 con 0x7fa884068580 2024-04-22T23:59:49.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.461+0000 7fa88b89e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8841966b0 con 0x7fa884106cd0 2024-04-22T23:59:49.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.461+0000 7fa88963a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa884068580 0x7fa88419c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:49.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.461+0000 7fa88963a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa884068580 0x7fa88419c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56696/0 (socket says 172.21.15.67:56696) 2024-04-22T23:59:49.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.461+0000 7fa88963a700 1 -- 172.21.15.67:0/2174936692 learned_addr learned my addr 172.21.15.67:0/2174936692 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:59:49.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.461+0000 7fa888e39700 1 --2- 172.21.15.67:0/2174936692 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa884106cd0 0x7fa88419cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:49.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.461+0000 7fa88963a700 1 -- 172.21.15.67:0/2174936692 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa884106cd0 msgr2=0x7fa88419cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:49.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.461+0000 7fa88963a700 1 --2- 172.21.15.67:0/2174936692 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa884106cd0 0x7fa88419cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:49.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.462+0000 7fa88963a700 1 -- 172.21.15.67:0/2174936692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8800095f0 con 0x7fa884068580 2024-04-22T23:59:49.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.462+0000 7fa888e39700 1 --2- 172.21.15.67:0/2174936692 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa884106cd0 0x7fa88419cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T23:59:49.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.462+0000 7fa88963a700 1 --2- 172.21.15.67:0/2174936692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa884068580 0x7fa88419c630 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fa87800c930 tx=0x7fa87800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:59:49.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.462+0000 7fa8767fc700 1 -- 172.21.15.67:0/2174936692 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa878007ad0 con 0x7fa884068580 2024-04-22T23:59:49.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.462+0000 7fa8767fc700 1 -- 172.21.15.67:0/2174936692 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa878007c30 con 0x7fa884068580 2024-04-22T23:59:49.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.462+0000 7fa88b89e700 1 -- 172.21.15.67:0/2174936692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa884196990 con 0x7fa884068580 2024-04-22T23:59:49.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.462+0000 7fa8767fc700 1 -- 172.21.15.67:0/2174936692 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa878018610 con 0x7fa884068580 2024-04-22T23:59:49.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.462+0000 7fa88b89e700 1 -- 172.21.15.67:0/2174936692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa884196ee0 con 0x7fa884068580 2024-04-22T23:59:49.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.463+0000 7fa88b89e700 1 -- 172.21.15.67:0/2174936692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa868005320 con 0x7fa884068580 2024-04-22T23:59:49.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.464+0000 7fa8767fc700 1 -- 172.21.15.67:0/2174936692 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa87800f450 con 0x7fa884068580 2024-04-22T23:59:49.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.464+0000 7fa8767fc700 1 --2- 172.21.15.67:0/2174936692 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa87006ecc0 0x7fa870071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:49.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.464+0000 7fa8767fc700 1 -- 172.21.15.67:0/2174936692 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fa87808f0e0 con 0x7fa884068580 2024-04-22T23:59:49.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.464+0000 7fa888e39700 1 --2- 172.21.15.67:0/2174936692 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa87006ecc0 0x7fa870071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:49.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.465+0000 7fa888e39700 1 --2- 172.21.15.67:0/2174936692 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa87006ecc0 0x7fa870071180 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fa880005b40 tx=0x7fa88001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:59:49.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.469+0000 7fa8767fc700 1 -- 172.21.15.67:0/2174936692 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fa87805b8d0 con 0x7fa884068580 2024-04-22T23:59:49.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:49 smithi134 ceph-mon[85630]: from='client.35804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:59:49.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:49 smithi134 ceph-mon[85630]: from='client.35808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:59:49.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:49 smithi134 ceph-mon[85630]: from='client.35812 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:59:49.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:49 smithi134 ceph-mon[85630]: pgmap v1005: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:49.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.725+0000 7fa88b89e700 1 -- 172.21.15.67:0/2174936692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa868006200 con 0x7fa884068580 2024-04-22T23:59:49.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.725+0000 7fa8767fc700 1 -- 172.21.15.67:0/2174936692 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1872 (secure 0 0 0) 0x7fa87805b460 con 0x7fa884068580 2024-04-22T23:59:49.717 INFO:teuthology.orchestra.run.smithi067.stdout:e33 2024-04-22T23:59:49.717 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T23:59:49.717 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:59:49.717 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-22T23:59:49.717 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:59:49.718 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-22T23:59:49.718 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-22T23:59:49.718 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-22T23:59:49.718 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-22T23:59:49.718 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-22T23:59:49.718 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-22T23:59:49.718 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-22T23:59:49.718 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-22T23:59:49.718 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-22T23:59:49.718 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-22T23:59:49.718 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-22T23:59:49.718 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-22T23:59:49.718 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-22T23:59:49.719 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-22T23:59:49.719 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-22T23:59:49.719 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T23:59:49.719 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-22T23:59:49.719 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-22T23:59:49.719 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-22T23:59:49.719 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-22T23:59:49.719 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-22T23:59:49.719 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-22T23:59:49.719 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-22T23:59:49.719 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-22T23:59:49.719 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-22T23:59:49.719 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-22T23:59:49.719 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-22T23:59:49.720 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-22T23:59:49.720 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-22T23:59:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:59:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:59:49.720 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-22T23:59:49.720 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-22T23:59:49.720 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:59:49.720 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:59:49.720 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:59:49.720 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44851} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1849648622,v1:172.21.15.134:6827/1849648622] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T23:59:49.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.727+0000 7fa88b89e700 1 -- 172.21.15.67:0/2174936692 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa87006ecc0 msgr2=0x7fa870071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:49.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.727+0000 7fa88b89e700 1 --2- 172.21.15.67:0/2174936692 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa87006ecc0 0x7fa870071180 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fa880005b40 tx=0x7fa88001a040 comp rx=0 tx=0).stop 2024-04-22T23:59:49.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.727+0000 7fa88b89e700 1 -- 172.21.15.67:0/2174936692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa884068580 msgr2=0x7fa88419c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:49.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.727+0000 7fa88b89e700 1 --2- 172.21.15.67:0/2174936692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa884068580 0x7fa88419c630 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fa87800c930 tx=0x7fa87800ccf0 comp rx=0 tx=0).stop 2024-04-22T23:59:49.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.728+0000 7fa88b89e700 1 -- 172.21.15.67:0/2174936692 shutdown_connections 2024-04-22T23:59:49.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.728+0000 7fa88b89e700 1 --2- 172.21.15.67:0/2174936692 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa87006ecc0 0x7fa870071180 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:49.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.728+0000 7fa88b89e700 1 --2- 172.21.15.67:0/2174936692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa884068580 0x7fa88419c630 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:49.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.728+0000 7fa88b89e700 1 --2- 172.21.15.67:0/2174936692 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa884106cd0 0x7fa88419cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:49.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.728+0000 7fa88b89e700 1 -- 172.21.15.67:0/2174936692 >> 172.21.15.67:0/2174936692 conn(0x7fa884075670 msgr2=0x7fa8840feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:59:49.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.728+0000 7fa88b89e700 1 -- 172.21.15.67:0/2174936692 shutdown_connections 2024-04-22T23:59:49.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.728+0000 7fa88b89e700 1 -- 172.21.15.67:0/2174936692 wait complete. 2024-04-22T23:59:49.722 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 33 2024-04-22T23:59:49.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.835+0000 7f7a6944f700 1 -- 172.21.15.67:0/1334066553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a64101e00 msgr2=0x7f7a641021e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:49.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.835+0000 7f7a6944f700 1 --2- 172.21.15.67:0/1334066553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a64101e00 0x7f7a641021e0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c00b0a0 tx=0x7f7a5c00b3b0 comp rx=0 tx=0).stop 2024-04-22T23:59:49.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.835+0000 7f7a6944f700 1 -- 172.21.15.67:0/1334066553 shutdown_connections 2024-04-22T23:59:49.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.835+0000 7f7a6944f700 1 --2- 172.21.15.67:0/1334066553 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a64102720 0x7f7a6410ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:49.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.835+0000 7f7a6944f700 1 --2- 172.21.15.67:0/1334066553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a64101e00 0x7f7a641021e0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:49.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.835+0000 7f7a6944f700 1 -- 172.21.15.67:0/1334066553 >> 172.21.15.67:0/1334066553 conn(0x7f7a640fb650 msgr2=0x7f7a640fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:59:49.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.835+0000 7f7a6944f700 1 -- 172.21.15.67:0/1334066553 shutdown_connections 2024-04-22T23:59:49.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.835+0000 7f7a6944f700 1 -- 172.21.15.67:0/1334066553 wait complete. 2024-04-22T23:59:49.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.836+0000 7f7a6944f700 1 Processor -- start 2024-04-22T23:59:49.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.836+0000 7f7a6944f700 1 -- start start 2024-04-22T23:59:49.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.836+0000 7f7a6944f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a64101e00 0x7f7a6419c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:49.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.836+0000 7f7a6944f700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a64102720 0x7f7a6419cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:49.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.836+0000 7f7a6944f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6419d410 con 0x7f7a64101e00 2024-04-22T23:59:49.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.836+0000 7f7a6944f700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a641968c0 con 0x7f7a64102720 2024-04-22T23:59:49.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.836+0000 7f7a62ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a64101e00 0x7f7a6419c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:49.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.836+0000 7f7a62ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a64101e00 0x7f7a6419c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56718/0 (socket says 172.21.15.67:56718) 2024-04-22T23:59:49.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.836+0000 7f7a62ffd700 1 -- 172.21.15.67:0/114093993 learned_addr learned my addr 172.21.15.67:0/114093993 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:59:49.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.837+0000 7f7a627fc700 1 --2- 172.21.15.67:0/114093993 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a64102720 0x7f7a6419cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:49.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.837+0000 7f7a62ffd700 1 -- 172.21.15.67:0/114093993 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a64102720 msgr2=0x7f7a6419cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:49.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.837+0000 7f7a62ffd700 1 --2- 172.21.15.67:0/114093993 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a64102720 0x7f7a6419cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:49.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.837+0000 7f7a62ffd700 1 -- 172.21.15.67:0/114093993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a5c009d00 con 0x7f7a64101e00 2024-04-22T23:59:49.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.837+0000 7f7a62ffd700 1 --2- 172.21.15.67:0/114093993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a64101e00 0x7f7a6419c840 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c000f80 tx=0x7f7a5c00cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:59:49.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.837+0000 7f7a5bfff700 1 -- 172.21.15.67:0/114093993 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a5c00e040 con 0x7f7a64101e00 2024-04-22T23:59:49.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.837+0000 7f7a5bfff700 1 -- 172.21.15.67:0/114093993 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7a5c003de0 con 0x7f7a64101e00 2024-04-22T23:59:49.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.837+0000 7f7a5bfff700 1 -- 172.21.15.67:0/114093993 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a5c01d8e0 con 0x7f7a64101e00 2024-04-22T23:59:49.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.837+0000 7f7a6944f700 1 -- 172.21.15.67:0/114093993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a64196b40 con 0x7f7a64101e00 2024-04-22T23:59:49.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.837+0000 7f7a6944f700 1 -- 172.21.15.67:0/114093993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a64197030 con 0x7f7a64101e00 2024-04-22T23:59:49.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.838+0000 7f7a6944f700 1 -- 172.21.15.67:0/114093993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a64108310 con 0x7f7a64101e00 2024-04-22T23:59:49.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.839+0000 7f7a5bfff700 1 -- 172.21.15.67:0/114093993 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7a5c004820 con 0x7f7a64101e00 2024-04-22T23:59:49.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.840+0000 7f7a5bfff700 1 --2- 172.21.15.67:0/114093993 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7a5006ecc0 0x7f7a50071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:49.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.840+0000 7f7a5bfff700 1 -- 172.21.15.67:0/114093993 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f7a5c0908a0 con 0x7f7a64101e00 2024-04-22T23:59:49.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.840+0000 7f7a627fc700 1 --2- 172.21.15.67:0/114093993 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7a5006ecc0 0x7f7a50071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:49.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.840+0000 7f7a627fc700 1 --2- 172.21.15.67:0/114093993 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7a5006ecc0 0x7f7a50071180 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c006fd0 tx=0x7f7a4c008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:59:49.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:49.845+0000 7f7a5bfff700 1 -- 172.21.15.67:0/114093993 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7a5c05d010 con 0x7f7a64101e00 2024-04-22T23:59:50.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.080+0000 7f7a6944f700 1 -- 172.21.15.67:0/114093993 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7a64197e90 con 0x7f7a5006ecc0 2024-04-22T23:59:50.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.081+0000 7f7a5bfff700 1 -- 172.21.15.67:0/114093993 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f7a64197e90 con 0x7f7a5006ecc0 2024-04-22T23:59:50.073 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-22T23:59:50.073 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-22T23:59:50.073 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-22T23:59:50.074 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-22T23:59:50.074 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-22T23:59:50.074 INFO:teuthology.orchestra.run.smithi067.stdout: "mds", 2024-04-22T23:59:50.074 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-22T23:59:50.074 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-22T23:59:50.074 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-22T23:59:50.074 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-22T23:59:50.074 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-22T23:59:50.074 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "16/23 daemons upgraded", 2024-04-22T23:59:50.074 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-22T23:59:50.074 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-22T23:59:50.075 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-22T23:59:50.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.083+0000 7f7a6944f700 1 -- 172.21.15.67:0/114093993 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7a5006ecc0 msgr2=0x7f7a50071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:50.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.083+0000 7f7a6944f700 1 --2- 172.21.15.67:0/114093993 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7a5006ecc0 0x7f7a50071180 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c006fd0 tx=0x7f7a4c008040 comp rx=0 tx=0).stop 2024-04-22T23:59:50.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.083+0000 7f7a6944f700 1 -- 172.21.15.67:0/114093993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a64101e00 msgr2=0x7f7a6419c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:50.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.083+0000 7f7a6944f700 1 --2- 172.21.15.67:0/114093993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a64101e00 0x7f7a6419c840 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c000f80 tx=0x7f7a5c00cdd0 comp rx=0 tx=0).stop 2024-04-22T23:59:50.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.083+0000 7f7a6944f700 1 -- 172.21.15.67:0/114093993 shutdown_connections 2024-04-22T23:59:50.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.083+0000 7f7a6944f700 1 --2- 172.21.15.67:0/114093993 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7a5006ecc0 0x7f7a50071180 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:50.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.083+0000 7f7a6944f700 1 --2- 172.21.15.67:0/114093993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a64101e00 0x7f7a6419c840 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:50.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.083+0000 7f7a6944f700 1 --2- 172.21.15.67:0/114093993 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7a64102720 0x7f7a6419cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:50.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.083+0000 7f7a6944f700 1 -- 172.21.15.67:0/114093993 >> 172.21.15.67:0/114093993 conn(0x7f7a640fb650 msgr2=0x7f7a64105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:59:50.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.084+0000 7f7a6944f700 1 -- 172.21.15.67:0/114093993 shutdown_connections 2024-04-22T23:59:50.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.084+0000 7f7a6944f700 1 -- 172.21.15.67:0/114093993 wait complete. 2024-04-22T23:59:50.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.190+0000 7f62d7be2700 1 -- 172.21.15.67:0/2385430452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62d0100ea0 msgr2=0x7f62d0101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:50.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.190+0000 7f62d7be2700 1 --2- 172.21.15.67:0/2385430452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62d0100ea0 0x7f62d0101280 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f62cc009910 tx=0x7f62cc009c20 comp rx=0 tx=0).stop 2024-04-22T23:59:50.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.190+0000 7f62d7be2700 1 -- 172.21.15.67:0/2385430452 shutdown_connections 2024-04-22T23:59:50.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.190+0000 7f62d7be2700 1 --2- 172.21.15.67:0/2385430452 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62d0101850 0x7f62d01058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:50.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.190+0000 7f62d7be2700 1 --2- 172.21.15.67:0/2385430452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62d0100ea0 0x7f62d0101280 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:50.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.190+0000 7f62d7be2700 1 -- 172.21.15.67:0/2385430452 >> 172.21.15.67:0/2385430452 conn(0x7f62d00fc750 msgr2=0x7f62d00feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:59:50.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.190+0000 7f62d7be2700 1 -- 172.21.15.67:0/2385430452 shutdown_connections 2024-04-22T23:59:50.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.190+0000 7f62d7be2700 1 -- 172.21.15.67:0/2385430452 wait complete. 2024-04-22T23:59:50.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.191+0000 7f62d7be2700 1 Processor -- start 2024-04-22T23:59:50.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.191+0000 7f62d7be2700 1 -- start start 2024-04-22T23:59:50.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.191+0000 7f62d7be2700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62d0100ea0 0x7f62d019c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:50.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.191+0000 7f62d7be2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62d0101850 0x7f62d019cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:50.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.191+0000 7f62d7be2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d019d2c0 con 0x7f62d0101850 2024-04-22T23:59:50.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.191+0000 7f62d7be2700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d0196770 con 0x7f62d0100ea0 2024-04-22T23:59:50.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.191+0000 7f62d517d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62d0101850 0x7f62d019cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:50.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.191+0000 7f62d517d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62d0101850 0x7f62d019cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56742/0 (socket says 172.21.15.67:56742) 2024-04-22T23:59:50.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.191+0000 7f62d517d700 1 -- 172.21.15.67:0/4274539965 learned_addr learned my addr 172.21.15.67:0/4274539965 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-22T23:59:50.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.192+0000 7f62d597e700 1 --2- 172.21.15.67:0/4274539965 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62d0100ea0 0x7f62d019c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:50.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.192+0000 7f62d517d700 1 -- 172.21.15.67:0/4274539965 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62d0100ea0 msgr2=0x7f62d019c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:50.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.192+0000 7f62d517d700 1 --2- 172.21.15.67:0/4274539965 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62d0100ea0 0x7f62d019c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:50.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.192+0000 7f62d517d700 1 -- 172.21.15.67:0/4274539965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62cc0095f0 con 0x7f62d0101850 2024-04-22T23:59:50.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.192+0000 7f62d517d700 1 --2- 172.21.15.67:0/4274539965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62d0101850 0x7f62d019cc30 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f62c000d900 tx=0x7f62c000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:59:50.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.192+0000 7f62c6ffd700 1 -- 172.21.15.67:0/4274539965 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62c0009900 con 0x7f62d0101850 2024-04-22T23:59:50.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.192+0000 7f62c6ffd700 1 -- 172.21.15.67:0/4274539965 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f62c0010460 con 0x7f62d0101850 2024-04-22T23:59:50.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.192+0000 7f62d7be2700 1 -- 172.21.15.67:0/4274539965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62d0196a50 con 0x7f62d0101850 2024-04-22T23:59:50.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.192+0000 7f62c6ffd700 1 -- 172.21.15.67:0/4274539965 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62c000f5f0 con 0x7f62d0101850 2024-04-22T23:59:50.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.192+0000 7f62d7be2700 1 -- 172.21.15.67:0/4274539965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62d0196fa0 con 0x7f62d0101850 2024-04-22T23:59:50.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.193+0000 7f62d7be2700 1 -- 172.21.15.67:0/4274539965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62d0109240 con 0x7f62d0101850 2024-04-22T23:59:50.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.194+0000 7f62c6ffd700 1 -- 172.21.15.67:0/4274539965 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f62c00105d0 con 0x7f62d0101850 2024-04-22T23:59:50.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.194+0000 7f62c6ffd700 1 --2- 172.21.15.67:0/4274539965 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f62bc06ecc0 0x7f62bc071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T23:59:50.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.194+0000 7f62c6ffd700 1 -- 172.21.15.67:0/4274539965 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f62c008ea80 con 0x7f62d0101850 2024-04-22T23:59:50.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.194+0000 7f62d597e700 1 --2- 172.21.15.67:0/4274539965 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f62bc06ecc0 0x7f62bc071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T23:59:50.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.195+0000 7f62d597e700 1 --2- 172.21.15.67:0/4274539965 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f62bc06ecc0 0x7f62bc071180 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f62cc006010 tx=0x7f62cc01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T23:59:50.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.199+0000 7f62c6ffd700 1 -- 172.21.15.67:0/4274539965 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f62c005b270 con 0x7f62d0101850 2024-04-22T23:59:50.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:50 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3172494518' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:59:50.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:50 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2174936692' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:59:50.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:50 smithi134 ceph-mon[85630]: from='client.35824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:59:50.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.497+0000 7f62d7be2700 1 -- 172.21.15.67:0/4274539965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f62d0197d20 con 0x7f62d0101850 2024-04-22T23:59:50.489 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:50 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3172494518' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T23:59:50.489 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:50 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2174936692' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T23:59:50.489 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:50 smithi067 ceph-mon[112438]: from='client.35824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T23:59:50.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.497+0000 7f62c6ffd700 1 -- 172.21.15.67:0/4274539965 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4505 (secure 0 0 0) 0x7f62c005ae00 con 0x7f62d0101850 2024-04-22T23:59:50.489 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-22T23:59:50.489 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-22T23:59:50.489 INFO:teuthology.orchestra.run.smithi067.stdout: host smithi134 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-22T23:59:50.489 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-22T23:59:50.490 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:59:50.490 INFO:teuthology.orchestra.run.smithi067.stdout:Traceback (most recent call last): 2024-04-22T23:59:50.490 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-22T23:59:50.490 INFO:teuthology.orchestra.run.smithi067.stdout: return _run_code(code, main_globals, None, 2024-04-22T23:59:50.490 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-22T23:59:50.490 INFO:teuthology.orchestra.run.smithi067.stdout: exec(code, run_globals) 2024-04-22T23:59:50.490 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5642, in 2024-04-22T23:59:50.490 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5630, in main 2024-04-22T23:59:50.490 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 408, in _infer_config 2024-04-22T23:59:50.490 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 323, in _infer_fsid 2024-04-22T23:59:50.490 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 436, in _infer_image 2024-04-22T23:59:50.490 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 310, in _validate_fsid 2024-04-22T23:59:50.490 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 3380, in command_ceph_volume 2024-04-22T23:59:50.490 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-22T23:59:50.491 INFO:teuthology.orchestra.run.smithi067.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-22T23:59:50.491 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-22T23:59:50.491 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-22T23:59:50.491 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-22T23:59:50.491 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-22T23:59:50.491 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-22T23:59:50.491 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-22T23:59:50.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.499+0000 7f62d7be2700 1 -- 172.21.15.67:0/4274539965 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f62bc06ecc0 msgr2=0x7f62bc071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:50.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.500+0000 7f62d7be2700 1 --2- 172.21.15.67:0/4274539965 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f62bc06ecc0 0x7f62bc071180 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f62cc006010 tx=0x7f62cc01a040 comp rx=0 tx=0).stop 2024-04-22T23:59:50.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.500+0000 7f62d7be2700 1 -- 172.21.15.67:0/4274539965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62d0101850 msgr2=0x7f62d019cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T23:59:50.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.500+0000 7f62d7be2700 1 --2- 172.21.15.67:0/4274539965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62d0101850 0x7f62d019cc30 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f62c000d900 tx=0x7f62c000dc10 comp rx=0 tx=0).stop 2024-04-22T23:59:50.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.500+0000 7f62d7be2700 1 -- 172.21.15.67:0/4274539965 shutdown_connections 2024-04-22T23:59:50.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.500+0000 7f62d7be2700 1 --2- 172.21.15.67:0/4274539965 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f62bc06ecc0 0x7f62bc071180 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:50.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.500+0000 7f62d7be2700 1 --2- 172.21.15.67:0/4274539965 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f62d0100ea0 0x7f62d019c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:50.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.500+0000 7f62d7be2700 1 --2- 172.21.15.67:0/4274539965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62d0101850 0x7f62d019cc30 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T23:59:50.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.500+0000 7f62d7be2700 1 -- 172.21.15.67:0/4274539965 >> 172.21.15.67:0/4274539965 conn(0x7f62d00fc750 msgr2=0x7f62d00fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T23:59:50.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.500+0000 7f62d7be2700 1 -- 172.21.15.67:0/4274539965 shutdown_connections 2024-04-22T23:59:50.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-22T23:59:50.500+0000 7f62d7be2700 1 -- 172.21.15.67:0/4274539965 wait complete. 2024-04-22T23:59:51.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:51 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/4274539965' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:59:51.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:51 smithi134 ceph-mon[85630]: pgmap v1006: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:51.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:51 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/4274539965' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T23:59:51.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:51 smithi067 ceph-mon[112438]: pgmap v1006: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:53.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:53 smithi134 ceph-mon[85630]: pgmap v1007: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:53.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:53 smithi067 ceph-mon[112438]: pgmap v1007: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:55.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:55 smithi134 ceph-mon[85630]: pgmap v1008: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:55.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:55 smithi067 ceph-mon[112438]: pgmap v1008: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:56.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:59:56.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-22T23:59:57.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:57 smithi134 ceph-mon[85630]: pgmap v1009: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:57.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:57 smithi067 ceph-mon[112438]: pgmap v1009: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:59.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 22 23:59:59 smithi134 ceph-mon[85630]: pgmap v1010: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-22T23:59:59.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 22 23:59:59 smithi067 ceph-mon[112438]: pgmap v1010: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:00.284 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:00 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[112434]: 2024-04-22T23:59:59.999+0000 7f20165b9700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-23T00:00:00.284 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:00 smithi067 ceph-mon[112438]: overall HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-23T00:00:00.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:00 smithi134 ceph-mon[85630]: overall HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-23T00:00:01.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:01 smithi134 ceph-mon[85630]: pgmap v1011: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:01.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:01 smithi067 ceph-mon[112438]: pgmap v1011: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:03.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:03 smithi134 ceph-mon[85630]: pgmap v1012: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:03.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:03 smithi067 ceph-mon[112438]: pgmap v1012: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:05.284 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:05 smithi067 ceph-mon[112438]: pgmap v1013: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:05.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:05 smithi134 ceph-mon[85630]: pgmap v1013: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:07.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:07 smithi134 ceph-mon[85630]: pgmap v1014: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:07.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:07 smithi067 ceph-mon[112438]: pgmap v1014: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:09.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:09 smithi134 ceph-mon[85630]: pgmap v1015: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:09.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:09 smithi067 ceph-mon[112438]: pgmap v1015: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:11.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:00:11.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:00:12.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:12 smithi134 ceph-mon[85630]: pgmap v1016: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:12.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:12 smithi067 ceph-mon[112438]: pgmap v1016: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:13.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:13 smithi134 ceph-mon[85630]: pgmap v1017: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:13.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:13 smithi067 ceph-mon[112438]: pgmap v1017: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:15.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:15 smithi134 ceph-mon[85630]: pgmap v1018: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:15.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:15 smithi067 ceph-mon[112438]: pgmap v1018: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:17.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:17 smithi134 ceph-mon[85630]: pgmap v1019: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:17.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:17 smithi067 ceph-mon[112438]: pgmap v1019: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:19.450 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:19 smithi134 ceph-mon[85630]: pgmap v1020: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:19.450 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:19 smithi067 ceph-mon[112438]: pgmap v1020: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:20.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.612+0000 7fce307ee700 1 -- 172.21.15.67:0/495379181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce28105bd0 msgr2=0x7fce28105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:20.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.612+0000 7fce307ee700 1 --2- 172.21.15.67:0/495379181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce28105bd0 0x7fce28105fb0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fce24009910 tx=0x7fce24009c20 comp rx=0 tx=0).stop 2024-04-23T00:00:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.612+0000 7fce307ee700 1 -- 172.21.15.67:0/495379181 shutdown_connections 2024-04-23T00:00:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.612+0000 7fce307ee700 1 --2- 172.21.15.67:0/495379181 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce280ffc20 0x7fce28100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.612+0000 7fce307ee700 1 --2- 172.21.15.67:0/495379181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce28105bd0 0x7fce28105fb0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.612+0000 7fce307ee700 1 -- 172.21.15.67:0/495379181 >> 172.21.15.67:0/495379181 conn(0x7fce28075670 msgr2=0x7fce28075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:00:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.612+0000 7fce307ee700 1 -- 172.21.15.67:0/495379181 shutdown_connections 2024-04-23T00:00:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.612+0000 7fce307ee700 1 -- 172.21.15.67:0/495379181 wait complete. 2024-04-23T00:00:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.612+0000 7fce307ee700 1 Processor -- start 2024-04-23T00:00:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.612+0000 7fce307ee700 1 -- start start 2024-04-23T00:00:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.613+0000 7fce307ee700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce280ffc20 0x7fce281976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.613+0000 7fce307ee700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce28105bd0 0x7fce28197c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.613+0000 7fce307ee700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce281982f0 con 0x7fce280ffc20 2024-04-23T00:00:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.613+0000 7fce307ee700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce28198460 con 0x7fce28105bd0 2024-04-23T00:00:20.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.613+0000 7fce2e58a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce280ffc20 0x7fce281976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:20.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.613+0000 7fce2e58a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce280ffc20 0x7fce281976d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50324/0 (socket says 172.21.15.67:50324) 2024-04-23T00:00:20.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.613+0000 7fce2e58a700 1 -- 172.21.15.67:0/3962853577 learned_addr learned my addr 172.21.15.67:0/3962853577 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:00:20.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.613+0000 7fce2dd89700 1 --2- 172.21.15.67:0/3962853577 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce28105bd0 0x7fce28197c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:20.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.613+0000 7fce2e58a700 1 -- 172.21.15.67:0/3962853577 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce28105bd0 msgr2=0x7fce28197c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:20.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.613+0000 7fce2e58a700 1 --2- 172.21.15.67:0/3962853577 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce28105bd0 0x7fce28197c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:20.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.613+0000 7fce2e58a700 1 -- 172.21.15.67:0/3962853577 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce240095f0 con 0x7fce280ffc20 2024-04-23T00:00:20.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.613+0000 7fce2e58a700 1 --2- 172.21.15.67:0/3962853577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce280ffc20 0x7fce281976d0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fce24006010 tx=0x7fce2400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:00:20.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.613+0000 7fce1f7fe700 1 -- 172.21.15.67:0/3962853577 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce2401d070 con 0x7fce280ffc20 2024-04-23T00:00:20.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.613+0000 7fce307ee700 1 -- 172.21.15.67:0/3962853577 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce281a0740 con 0x7fce280ffc20 2024-04-23T00:00:20.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.613+0000 7fce1f7fe700 1 -- 172.21.15.67:0/3962853577 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fce24004fb0 con 0x7fce280ffc20 2024-04-23T00:00:20.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.613+0000 7fce1f7fe700 1 -- 172.21.15.67:0/3962853577 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce24022ea0 con 0x7fce280ffc20 2024-04-23T00:00:20.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.613+0000 7fce307ee700 1 -- 172.21.15.67:0/3962853577 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce281a0bb0 con 0x7fce280ffc20 2024-04-23T00:00:20.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.615+0000 7fce307ee700 1 -- 172.21.15.67:0/3962853577 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce281080f0 con 0x7fce280ffc20 2024-04-23T00:00:20.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.615+0000 7fce1f7fe700 1 -- 172.21.15.67:0/3962853577 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fce24022470 con 0x7fce280ffc20 2024-04-23T00:00:20.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.616+0000 7fce1f7fe700 1 --2- 172.21.15.67:0/3962853577 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fce1406ecc0 0x7fce14071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:20.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.616+0000 7fce1f7fe700 1 -- 172.21.15.67:0/3962853577 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fce24090560 con 0x7fce280ffc20 2024-04-23T00:00:20.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.616+0000 7fce2dd89700 1 --2- 172.21.15.67:0/3962853577 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fce1406ecc0 0x7fce14071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:20.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.616+0000 7fce2dd89700 1 --2- 172.21.15.67:0/3962853577 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fce1406ecc0 0x7fce14071180 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fce18005fd0 tx=0x7fce1800a560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:00:20.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.621+0000 7fce1f7fe700 1 -- 172.21.15.67:0/3962853577 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fce24059230 con 0x7fce280ffc20 2024-04-23T00:00:20.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.849+0000 7fce307ee700 1 -- 172.21.15.67:0/3962853577 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fce28068c80 con 0x7fce1406ecc0 2024-04-23T00:00:20.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.851+0000 7fce1f7fe700 1 -- 172.21.15.67:0/3962853577 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fce28068c80 con 0x7fce1406ecc0 2024-04-23T00:00:20.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.853+0000 7fce307ee700 1 -- 172.21.15.67:0/3962853577 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fce1406ecc0 msgr2=0x7fce14071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:20.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.853+0000 7fce307ee700 1 --2- 172.21.15.67:0/3962853577 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fce1406ecc0 0x7fce14071180 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fce18005fd0 tx=0x7fce1800a560 comp rx=0 tx=0).stop 2024-04-23T00:00:20.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.853+0000 7fce307ee700 1 -- 172.21.15.67:0/3962853577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce280ffc20 msgr2=0x7fce281976d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:20.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.853+0000 7fce307ee700 1 --2- 172.21.15.67:0/3962853577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce280ffc20 0x7fce281976d0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fce24006010 tx=0x7fce2400be20 comp rx=0 tx=0).stop 2024-04-23T00:00:20.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.853+0000 7fce307ee700 1 -- 172.21.15.67:0/3962853577 shutdown_connections 2024-04-23T00:00:20.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.853+0000 7fce307ee700 1 --2- 172.21.15.67:0/3962853577 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fce1406ecc0 0x7fce14071180 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:20.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.853+0000 7fce307ee700 1 --2- 172.21.15.67:0/3962853577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce280ffc20 0x7fce281976d0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:20.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.853+0000 7fce307ee700 1 --2- 172.21.15.67:0/3962853577 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce28105bd0 0x7fce28197c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:20.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.853+0000 7fce307ee700 1 -- 172.21.15.67:0/3962853577 >> 172.21.15.67:0/3962853577 conn(0x7fce28075670 msgr2=0x7fce280feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:00:20.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.853+0000 7fce307ee700 1 -- 172.21.15.67:0/3962853577 shutdown_connections 2024-04-23T00:00:20.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.853+0000 7fce307ee700 1 -- 172.21.15.67:0/3962853577 wait complete. 2024-04-23T00:00:20.853 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-23T00:00:20.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.961+0000 7f0acb221700 1 -- 172.21.15.67:0/3834333608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ac4102780 msgr2=0x7f0ac410ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:20.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.961+0000 7f0acb221700 1 --2- 172.21.15.67:0/3834333608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ac4102780 0x7f0ac410ac70 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8009910 tx=0x7f0ab8009c20 comp rx=0 tx=0).stop 2024-04-23T00:00:20.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.961+0000 7f0acb221700 1 -- 172.21.15.67:0/3834333608 shutdown_connections 2024-04-23T00:00:20.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.961+0000 7f0acb221700 1 --2- 172.21.15.67:0/3834333608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ac4102780 0x7f0ac410ac70 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:20.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.961+0000 7f0acb221700 1 --2- 172.21.15.67:0/3834333608 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0ac4101e60 0x7f0ac4102240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:20.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.961+0000 7f0acb221700 1 -- 172.21.15.67:0/3834333608 >> 172.21.15.67:0/3834333608 conn(0x7f0ac40fb650 msgr2=0x7f0ac40fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:00:20.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.961+0000 7f0acb221700 1 -- 172.21.15.67:0/3834333608 shutdown_connections 2024-04-23T00:00:20.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.961+0000 7f0acb221700 1 -- 172.21.15.67:0/3834333608 wait complete. 2024-04-23T00:00:20.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.961+0000 7f0acb221700 1 Processor -- start 2024-04-23T00:00:20.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.962+0000 7f0acb221700 1 -- start start 2024-04-23T00:00:20.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.962+0000 7f0acb221700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0ac4101e60 0x7f0ac419c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:20.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.962+0000 7f0acb221700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ac4102780 0x7f0ac419cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:20.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.962+0000 7f0acb221700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac419d3b0 con 0x7f0ac4102780 2024-04-23T00:00:20.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.962+0000 7f0acb221700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac4196860 con 0x7f0ac4101e60 2024-04-23T00:00:20.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.962+0000 7f0ac3fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ac4102780 0x7f0ac419cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:20.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.962+0000 7f0ac3fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ac4102780 0x7f0ac419cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50350/0 (socket says 172.21.15.67:50350) 2024-04-23T00:00:20.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.962+0000 7f0ac3fff700 1 -- 172.21.15.67:0/1978744539 learned_addr learned my addr 172.21.15.67:0/1978744539 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:00:20.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.962+0000 7f0ac8fbd700 1 --2- 172.21.15.67:0/1978744539 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0ac4101e60 0x7f0ac419c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:20.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.962+0000 7f0ac3fff700 1 -- 172.21.15.67:0/1978744539 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0ac4101e60 msgr2=0x7f0ac419c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:20.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.962+0000 7f0ac3fff700 1 --2- 172.21.15.67:0/1978744539 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0ac4101e60 0x7f0ac419c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:20.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.962+0000 7f0ac3fff700 1 -- 172.21.15.67:0/1978744539 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab80095f0 con 0x7f0ac4102780 2024-04-23T00:00:20.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.962+0000 7f0ac8fbd700 1 --2- 172.21.15.67:0/1978744539 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0ac4101e60 0x7f0ac419c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T00:00:20.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.962+0000 7f0ac3fff700 1 --2- 172.21.15.67:0/1978744539 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ac4102780 0x7f0ac419cd20 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f0ab800bda0 tx=0x7f0ab800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:00:20.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.963+0000 7f0ac1ffb700 1 -- 172.21.15.67:0/1978744539 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ab801d070 con 0x7f0ac4102780 2024-04-23T00:00:20.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.963+0000 7f0acb221700 1 -- 172.21.15.67:0/1978744539 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ac4196ae0 con 0x7f0ac4102780 2024-04-23T00:00:20.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.963+0000 7f0ac1ffb700 1 -- 172.21.15.67:0/1978744539 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0ab8004fb0 con 0x7f0ac4102780 2024-04-23T00:00:20.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.963+0000 7f0ac1ffb700 1 -- 172.21.15.67:0/1978744539 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ab800f480 con 0x7f0ac4102780 2024-04-23T00:00:20.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.963+0000 7f0acb221700 1 -- 172.21.15.67:0/1978744539 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ac4196fd0 con 0x7f0ac4102780 2024-04-23T00:00:20.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.964+0000 7f0acb221700 1 -- 172.21.15.67:0/1978744539 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ac4108300 con 0x7f0ac4102780 2024-04-23T00:00:20.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.964+0000 7f0ac1ffb700 1 -- 172.21.15.67:0/1978744539 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0ab8004af0 con 0x7f0ac4102780 2024-04-23T00:00:20.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.965+0000 7f0ac1ffb700 1 --2- 172.21.15.67:0/1978744539 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0aac06ecc0 0x7f0aac071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:20.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.965+0000 7f0ac1ffb700 1 -- 172.21.15.67:0/1978744539 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f0ab8090520 con 0x7f0ac4102780 2024-04-23T00:00:20.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.965+0000 7f0ac8fbd700 1 --2- 172.21.15.67:0/1978744539 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0aac06ecc0 0x7f0aac071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:20.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.965+0000 7f0ac8fbd700 1 --2- 172.21.15.67:0/1978744539 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0aac06ecc0 0x7f0aac071180 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f0ab40099d0 tx=0x7f0ab4008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:00:20.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:20.970+0000 7f0ac1ffb700 1 -- 172.21.15.67:0/1978744539 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f0ab805cd40 con 0x7f0ac4102780 2024-04-23T00:00:21.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.197+0000 7f0acb221700 1 -- 172.21.15.67:0/1978744539 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0ac4197d70 con 0x7f0aac06ecc0 2024-04-23T00:00:21.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.198+0000 7f0ac1ffb700 1 -- 172.21.15.67:0/1978744539 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f0ac4197d70 con 0x7f0aac06ecc0 2024-04-23T00:00:21.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.202+0000 7f0acb221700 1 -- 172.21.15.67:0/1978744539 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0aac06ecc0 msgr2=0x7f0aac071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:21.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.202+0000 7f0acb221700 1 --2- 172.21.15.67:0/1978744539 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0aac06ecc0 0x7f0aac071180 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f0ab40099d0 tx=0x7f0ab4008040 comp rx=0 tx=0).stop 2024-04-23T00:00:21.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.202+0000 7f0acb221700 1 -- 172.21.15.67:0/1978744539 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ac4102780 msgr2=0x7f0ac419cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:21.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.202+0000 7f0acb221700 1 --2- 172.21.15.67:0/1978744539 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ac4102780 0x7f0ac419cd20 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f0ab800bda0 tx=0x7f0ab800be80 comp rx=0 tx=0).stop 2024-04-23T00:00:21.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.202+0000 7f0acb221700 1 -- 172.21.15.67:0/1978744539 shutdown_connections 2024-04-23T00:00:21.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.202+0000 7f0acb221700 1 --2- 172.21.15.67:0/1978744539 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0aac06ecc0 0x7f0aac071180 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:21.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.202+0000 7f0acb221700 1 --2- 172.21.15.67:0/1978744539 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0ac4101e60 0x7f0ac419c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:21.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.202+0000 7f0acb221700 1 --2- 172.21.15.67:0/1978744539 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ac4102780 0x7f0ac419cd20 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:21.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.202+0000 7f0acb221700 1 -- 172.21.15.67:0/1978744539 >> 172.21.15.67:0/1978744539 conn(0x7f0ac40fb650 msgr2=0x7f0ac4105440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:00:21.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.202+0000 7f0acb221700 1 -- 172.21.15.67:0/1978744539 shutdown_connections 2024-04-23T00:00:21.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.202+0000 7f0acb221700 1 -- 172.21.15.67:0/1978744539 wait complete. 2024-04-23T00:00:21.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.308+0000 7fabbce1e700 1 -- 172.21.15.67:0/378713826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabb8101850 msgr2=0x7fabb81058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:21.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.308+0000 7fabbce1e700 1 --2- 172.21.15.67:0/378713826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabb8101850 0x7fabb81058a0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fabac009910 tx=0x7fabac009c20 comp rx=0 tx=0).stop 2024-04-23T00:00:21.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.308+0000 7fabbce1e700 1 -- 172.21.15.67:0/378713826 shutdown_connections 2024-04-23T00:00:21.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.308+0000 7fabbce1e700 1 --2- 172.21.15.67:0/378713826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabb8101850 0x7fabb81058a0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:21.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.308+0000 7fabbce1e700 1 --2- 172.21.15.67:0/378713826 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabb8100ea0 0x7fabb8101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:21.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.308+0000 7fabbce1e700 1 -- 172.21.15.67:0/378713826 >> 172.21.15.67:0/378713826 conn(0x7fabb80fc750 msgr2=0x7fabb80feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:00:21.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.309+0000 7fabbce1e700 1 -- 172.21.15.67:0/378713826 shutdown_connections 2024-04-23T00:00:21.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.309+0000 7fabbce1e700 1 -- 172.21.15.67:0/378713826 wait complete. 2024-04-23T00:00:21.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.309+0000 7fabbce1e700 1 Processor -- start 2024-04-23T00:00:21.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.309+0000 7fabbce1e700 1 -- start start 2024-04-23T00:00:21.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.309+0000 7fabbce1e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabb8100ea0 0x7fabb819c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:21.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.309+0000 7fabbce1e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabb8101850 0x7fabb819cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:21.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.310+0000 7fabbce1e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb819d2d0 con 0x7fabb8101850 2024-04-23T00:00:21.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.310+0000 7fabbce1e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb8196780 con 0x7fabb8100ea0 2024-04-23T00:00:21.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.310+0000 7fabb5d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabb8101850 0x7fabb819cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:21.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.310+0000 7fabb5d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabb8101850 0x7fabb819cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50364/0 (socket says 172.21.15.67:50364) 2024-04-23T00:00:21.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.310+0000 7fabb5d9b700 1 -- 172.21.15.67:0/3264477032 learned_addr learned my addr 172.21.15.67:0/3264477032 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:00:21.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.310+0000 7fabb659c700 1 --2- 172.21.15.67:0/3264477032 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabb8100ea0 0x7fabb819c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:21.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.310+0000 7fabb5d9b700 1 -- 172.21.15.67:0/3264477032 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabb8100ea0 msgr2=0x7fabb819c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:21.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.310+0000 7fabb5d9b700 1 --2- 172.21.15.67:0/3264477032 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabb8100ea0 0x7fabb819c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:21.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.310+0000 7fabb5d9b700 1 -- 172.21.15.67:0/3264477032 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabac0095f0 con 0x7fabb8101850 2024-04-23T00:00:21.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.310+0000 7fabb5d9b700 1 --2- 172.21.15.67:0/3264477032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabb8101850 0x7fabb819cc40 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fabac0092e0 tx=0x7fabac00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:00:21.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.310+0000 7faba77fe700 1 -- 172.21.15.67:0/3264477032 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabac01d070 con 0x7fabb8101850 2024-04-23T00:00:21.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.310+0000 7fabbce1e700 1 -- 172.21.15.67:0/3264477032 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabb8196a00 con 0x7fabb8101850 2024-04-23T00:00:21.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.310+0000 7faba77fe700 1 -- 172.21.15.67:0/3264477032 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fabac005dc0 con 0x7fabb8101850 2024-04-23T00:00:21.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.310+0000 7faba77fe700 1 -- 172.21.15.67:0/3264477032 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabac017400 con 0x7fabb8101850 2024-04-23T00:00:21.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.310+0000 7fabbce1e700 1 -- 172.21.15.67:0/3264477032 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabb8196ef0 con 0x7fabb8101850 2024-04-23T00:00:21.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.311+0000 7fabbce1e700 1 -- 172.21.15.67:0/3264477032 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabb81092a0 con 0x7fabb8101850 2024-04-23T00:00:21.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.312+0000 7faba77fe700 1 -- 172.21.15.67:0/3264477032 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fabac00f460 con 0x7fabb8101850 2024-04-23T00:00:21.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.312+0000 7faba77fe700 1 --2- 172.21.15.67:0/3264477032 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faba006ecc0 0x7faba0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:21.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.312+0000 7faba77fe700 1 -- 172.21.15.67:0/3264477032 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fabac090420 con 0x7fabb8101850 2024-04-23T00:00:21.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.312+0000 7fabb659c700 1 --2- 172.21.15.67:0/3264477032 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faba006ecc0 0x7faba0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:21.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.313+0000 7fabb659c700 1 --2- 172.21.15.67:0/3264477032 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faba006ecc0 0x7faba0071180 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7faba8005950 tx=0x7faba80058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:00:21.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.317+0000 7faba77fe700 1 -- 172.21.15.67:0/3264477032 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fabac05cc10 con 0x7fabb8101850 2024-04-23T00:00:21.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:21 smithi134 ceph-mon[85630]: from='client.35832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:00:21.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:21 smithi134 ceph-mon[85630]: pgmap v1021: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:21.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.536+0000 7fabbce1e700 1 -- 172.21.15.67:0/3264477032 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fabb8197da0 con 0x7faba006ecc0 2024-04-23T00:00:21.527 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:21 smithi067 ceph-mon[112438]: from='client.35832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:00:21.527 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:21 smithi067 ceph-mon[112438]: pgmap v1021: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:21.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.547+0000 7faba77fe700 1 -- 172.21.15.67:0/3264477032 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fabb8197da0 con 0x7faba006ecc0 2024-04-23T00:00:21.538 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-23T00:00:21.538 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (34m) 11m ago 45m 24.4M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-23T00:00:21.538 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (45m) 11m ago 45m 10.0M - 18.2.1 5be31c24972a df887f62291e 2024-04-23T00:00:21.538 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (44m) 11m ago 44m 10.0M - 18.2.1 5be31c24972a ad89190b5400 2024-04-23T00:00:21.538 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (31m) 11m ago 45m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-23T00:00:21.538 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (31m) 11m ago 44m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-23T00:00:21.538 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (33m) 11m ago 45m 81.2M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-23T00:00:21.538 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (27m) 11m ago 39m 19.4M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-23T00:00:21.538 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (27m) 11m ago 39m 20.3M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-23T00:00:21.539 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (27m) 11m ago 39m 20.1M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-23T00:00:21.539 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (11m) 11m ago 39m 14.0M - 19.0.0-3252-g43be0201 a0434699c1d7 46d3d7e86fee 2024-04-23T00:00:21.539 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (35m) 11m ago 46m 514M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-23T00:00:21.539 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (34m) 11m ago 44m 459M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-23T00:00:21.539 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (32m) 11m ago 47m 74.2M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-23T00:00:21.539 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (32m) 11m ago 44m 54.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-23T00:00:21.539 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (34m) 11m ago 45m 15.2M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-23T00:00:21.539 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (34m) 11m ago 44m 16.0M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-23T00:00:21.539 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (31m) 11m ago 43m 102M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-23T00:00:21.539 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (30m) 11m ago 43m 128M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-23T00:00:21.539 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (29m) 11m ago 42m 83.8M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-23T00:00:21.539 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (29m) 11m ago 42m 101M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-23T00:00:21.539 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (28m) 11m ago 41m 86.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-23T00:00:21.540 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (28m) 11m ago 41m 81.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-23T00:00:21.540 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (34m) 11m ago 44m 78.7M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-23T00:00:21.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.549+0000 7fabbce1e700 1 -- 172.21.15.67:0/3264477032 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faba006ecc0 msgr2=0x7faba0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:21.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.549+0000 7fabbce1e700 1 --2- 172.21.15.67:0/3264477032 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faba006ecc0 0x7faba0071180 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7faba8005950 tx=0x7faba80058e0 comp rx=0 tx=0).stop 2024-04-23T00:00:21.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.549+0000 7fabbce1e700 1 -- 172.21.15.67:0/3264477032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabb8101850 msgr2=0x7fabb819cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:21.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.549+0000 7fabbce1e700 1 --2- 172.21.15.67:0/3264477032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabb8101850 0x7fabb819cc40 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fabac0092e0 tx=0x7fabac00be20 comp rx=0 tx=0).stop 2024-04-23T00:00:21.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.549+0000 7fabbce1e700 1 -- 172.21.15.67:0/3264477032 shutdown_connections 2024-04-23T00:00:21.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.549+0000 7fabbce1e700 1 --2- 172.21.15.67:0/3264477032 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faba006ecc0 0x7faba0071180 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:21.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.549+0000 7fabbce1e700 1 --2- 172.21.15.67:0/3264477032 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabb8100ea0 0x7fabb819c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:21.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.550+0000 7fabbce1e700 1 --2- 172.21.15.67:0/3264477032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabb8101850 0x7fabb819cc40 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:21.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.550+0000 7fabbce1e700 1 -- 172.21.15.67:0/3264477032 >> 172.21.15.67:0/3264477032 conn(0x7fabb80fc750 msgr2=0x7fabb80fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:00:21.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.550+0000 7fabbce1e700 1 -- 172.21.15.67:0/3264477032 shutdown_connections 2024-04-23T00:00:21.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.550+0000 7fabbce1e700 1 -- 172.21.15.67:0/3264477032 wait complete. 2024-04-23T00:00:21.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.659+0000 7fc301f12700 1 -- 172.21.15.67:0/1919430887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2fc0ffc20 msgr2=0x7fc2fc100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:21.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.659+0000 7fc301f12700 1 --2- 172.21.15.67:0/1919430887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2fc0ffc20 0x7fc2fc100080 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4009910 tx=0x7fc2e4009c20 comp rx=0 tx=0).stop 2024-04-23T00:00:21.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.659+0000 7fc301f12700 1 -- 172.21.15.67:0/1919430887 shutdown_connections 2024-04-23T00:00:21.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.659+0000 7fc301f12700 1 --2- 172.21.15.67:0/1919430887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2fc0ffc20 0x7fc2fc100080 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:21.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.659+0000 7fc301f12700 1 --2- 172.21.15.67:0/1919430887 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2fc105bd0 0x7fc2fc105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:21.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.659+0000 7fc301f12700 1 -- 172.21.15.67:0/1919430887 >> 172.21.15.67:0/1919430887 conn(0x7fc2fc075670 msgr2=0x7fc2fc075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:00:21.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.659+0000 7fc301f12700 1 -- 172.21.15.67:0/1919430887 shutdown_connections 2024-04-23T00:00:21.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.659+0000 7fc301f12700 1 -- 172.21.15.67:0/1919430887 wait complete. 2024-04-23T00:00:21.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.659+0000 7fc301f12700 1 Processor -- start 2024-04-23T00:00:21.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.659+0000 7fc301f12700 1 -- start start 2024-04-23T00:00:21.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.660+0000 7fc301f12700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2fc0ffc20 0x7fc2fc101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:21.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.660+0000 7fc301f12700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2fc105bd0 0x7fc2fc101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:21.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.660+0000 7fc301f12700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2fc1022c0 con 0x7fc2fc105bd0 2024-04-23T00:00:21.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.660+0000 7fc301f12700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2fc102400 con 0x7fc2fc0ffc20 2024-04-23T00:00:21.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.660+0000 7fc2faffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2fc105bd0 0x7fc2fc101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:21.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.660+0000 7fc2faffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2fc105bd0 0x7fc2fc101d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50382/0 (socket says 172.21.15.67:50382) 2024-04-23T00:00:21.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.660+0000 7fc2faffd700 1 -- 172.21.15.67:0/3250208262 learned_addr learned my addr 172.21.15.67:0/3250208262 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:00:21.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.660+0000 7fc2fb7fe700 1 --2- 172.21.15.67:0/3250208262 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2fc0ffc20 0x7fc2fc101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:21.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.660+0000 7fc2faffd700 1 -- 172.21.15.67:0/3250208262 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2fc0ffc20 msgr2=0x7fc2fc101840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:21.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.660+0000 7fc2faffd700 1 --2- 172.21.15.67:0/3250208262 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2fc0ffc20 0x7fc2fc101840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:21.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.660+0000 7fc2faffd700 1 -- 172.21.15.67:0/3250208262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e40095f0 con 0x7fc2fc105bd0 2024-04-23T00:00:21.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.660+0000 7fc2faffd700 1 --2- 172.21.15.67:0/3250208262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2fc105bd0 0x7fc2fc101d80 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4006010 tx=0x7fc2e400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:00:21.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.660+0000 7fc2f8ff9700 1 -- 172.21.15.67:0/3250208262 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2e401d070 con 0x7fc2fc105bd0 2024-04-23T00:00:21.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.660+0000 7fc2f8ff9700 1 -- 172.21.15.67:0/3250208262 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc2e4004fb0 con 0x7fc2fc105bd0 2024-04-23T00:00:21.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.660+0000 7fc301f12700 1 -- 172.21.15.67:0/3250208262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2fc102680 con 0x7fc2fc105bd0 2024-04-23T00:00:21.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.661+0000 7fc2f8ff9700 1 -- 172.21.15.67:0/3250208262 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2e4022ea0 con 0x7fc2fc105bd0 2024-04-23T00:00:21.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.661+0000 7fc301f12700 1 -- 172.21.15.67:0/3250208262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2fc1a2d90 con 0x7fc2fc105bd0 2024-04-23T00:00:21.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.661+0000 7fc301f12700 1 -- 172.21.15.67:0/3250208262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2fc04f2e0 con 0x7fc2fc105bd0 2024-04-23T00:00:21.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.662+0000 7fc2f8ff9700 1 -- 172.21.15.67:0/3250208262 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc2e4022470 con 0x7fc2fc105bd0 2024-04-23T00:00:21.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.663+0000 7fc2f8ff9700 1 --2- 172.21.15.67:0/3250208262 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc2e806ecc0 0x7fc2e8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:21.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.663+0000 7fc2f8ff9700 1 -- 172.21.15.67:0/3250208262 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fc2e4090470 con 0x7fc2fc105bd0 2024-04-23T00:00:21.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.663+0000 7fc2fb7fe700 1 --2- 172.21.15.67:0/3250208262 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc2e806ecc0 0x7fc2e8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:21.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.663+0000 7fc2fb7fe700 1 --2- 172.21.15.67:0/3250208262 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc2e806ecc0 0x7fc2e8071180 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fc2f40060b0 tx=0x7fc2f4007b80 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:00:21.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.668+0000 7fc2f8ff9700 1 -- 172.21.15.67:0/3250208262 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc2e4059300 con 0x7fc2fc105bd0 2024-04-23T00:00:21.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.963+0000 7fc301f12700 1 -- 172.21.15.67:0/3250208262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc2fc1a3000 con 0x7fc2fc105bd0 2024-04-23T00:00:21.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.963+0000 7fc2f8ff9700 1 -- 172.21.15.67:0/3250208262 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fc2e4022720 con 0x7fc2fc105bd0 2024-04-23T00:00:21.954 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-23T00:00:21.954 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-23T00:00:21.954 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-23T00:00:21.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-23T00:00:21.955 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-23T00:00:21.955 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-23T00:00:21.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-23T00:00:21.955 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-23T00:00:21.955 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-23T00:00:21.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-23T00:00:21.955 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-23T00:00:21.955 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 4 2024-04-23T00:00:21.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-23T00:00:21.955 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-23T00:00:21.955 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 14 2024-04-23T00:00:21.955 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-23T00:00:21.955 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-23T00:00:21.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.966+0000 7fc301f12700 1 -- 172.21.15.67:0/3250208262 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc2e806ecc0 msgr2=0x7fc2e8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:21.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.966+0000 7fc301f12700 1 --2- 172.21.15.67:0/3250208262 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc2e806ecc0 0x7fc2e8071180 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fc2f40060b0 tx=0x7fc2f4007b80 comp rx=0 tx=0).stop 2024-04-23T00:00:21.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.966+0000 7fc301f12700 1 -- 172.21.15.67:0/3250208262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2fc105bd0 msgr2=0x7fc2fc101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:21.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.966+0000 7fc301f12700 1 --2- 172.21.15.67:0/3250208262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2fc105bd0 0x7fc2fc101d80 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4006010 tx=0x7fc2e400be20 comp rx=0 tx=0).stop 2024-04-23T00:00:21.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.966+0000 7fc301f12700 1 -- 172.21.15.67:0/3250208262 shutdown_connections 2024-04-23T00:00:21.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.966+0000 7fc301f12700 1 --2- 172.21.15.67:0/3250208262 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc2e806ecc0 0x7fc2e8071180 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:21.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.966+0000 7fc301f12700 1 --2- 172.21.15.67:0/3250208262 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc2fc0ffc20 0x7fc2fc101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:21.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.966+0000 7fc301f12700 1 --2- 172.21.15.67:0/3250208262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2fc105bd0 0x7fc2fc101d80 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:21.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.966+0000 7fc301f12700 1 -- 172.21.15.67:0/3250208262 >> 172.21.15.67:0/3250208262 conn(0x7fc2fc075670 msgr2=0x7fc2fc0686a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:00:21.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.966+0000 7fc301f12700 1 -- 172.21.15.67:0/3250208262 shutdown_connections 2024-04-23T00:00:21.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:21.966+0000 7fc301f12700 1 -- 172.21.15.67:0/3250208262 wait complete. 2024-04-23T00:00:22.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.072+0000 7f9766764700 1 -- 172.21.15.67:0/492141390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97600ffc20 msgr2=0x7f9760100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:22.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.072+0000 7f9766764700 1 --2- 172.21.15.67:0/492141390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97600ffc20 0x7f9760100080 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f9750009910 tx=0x7f9750009c20 comp rx=0 tx=0).stop 2024-04-23T00:00:22.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.072+0000 7f9766764700 1 -- 172.21.15.67:0/492141390 shutdown_connections 2024-04-23T00:00:22.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.072+0000 7f9766764700 1 --2- 172.21.15.67:0/492141390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97600ffc20 0x7f9760100080 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:22.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.072+0000 7f9766764700 1 --2- 172.21.15.67:0/492141390 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9760105bd0 0x7f9760105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:22.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.072+0000 7f9766764700 1 -- 172.21.15.67:0/492141390 >> 172.21.15.67:0/492141390 conn(0x7f9760075670 msgr2=0x7f9760075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:00:22.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.072+0000 7f9766764700 1 -- 172.21.15.67:0/492141390 shutdown_connections 2024-04-23T00:00:22.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.072+0000 7f9766764700 1 -- 172.21.15.67:0/492141390 wait complete. 2024-04-23T00:00:22.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.073+0000 7f9766764700 1 Processor -- start 2024-04-23T00:00:22.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.073+0000 7f9766764700 1 -- start start 2024-04-23T00:00:22.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.073+0000 7f9766764700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97600ffc20 0x7f9760101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:22.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.073+0000 7f9766764700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9760105bd0 0x7f9760101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:22.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.073+0000 7f9766764700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97601022c0 con 0x7f97600ffc20 2024-04-23T00:00:22.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.073+0000 7f9766764700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9760102400 con 0x7f9760105bd0 2024-04-23T00:00:22.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.073+0000 7f975ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97600ffc20 0x7f9760101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:22.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.073+0000 7f975ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97600ffc20 0x7f9760101840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50400/0 (socket says 172.21.15.67:50400) 2024-04-23T00:00:22.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.073+0000 7f975ffff700 1 -- 172.21.15.67:0/1623526861 learned_addr learned my addr 172.21.15.67:0/1623526861 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:00:22.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.074+0000 7f975f7fe700 1 --2- 172.21.15.67:0/1623526861 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9760105bd0 0x7f9760101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:22.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.074+0000 7f975ffff700 1 -- 172.21.15.67:0/1623526861 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9760105bd0 msgr2=0x7f9760101d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:22.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.074+0000 7f975ffff700 1 --2- 172.21.15.67:0/1623526861 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9760105bd0 0x7f9760101d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:22.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.074+0000 7f975ffff700 1 -- 172.21.15.67:0/1623526861 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97500095f0 con 0x7f97600ffc20 2024-04-23T00:00:22.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.074+0000 7f975f7fe700 1 --2- 172.21.15.67:0/1623526861 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9760105bd0 0x7f9760101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T00:00:22.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.074+0000 7f975ffff700 1 --2- 172.21.15.67:0/1623526861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97600ffc20 0x7f9760101840 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f975400b4b0 tx=0x7f975400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:00:22.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.074+0000 7f975d7fa700 1 -- 172.21.15.67:0/1623526861 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9754011800 con 0x7f97600ffc20 2024-04-23T00:00:22.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.074+0000 7f9766764700 1 -- 172.21.15.67:0/1623526861 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97601a2b80 con 0x7f97600ffc20 2024-04-23T00:00:22.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.074+0000 7f975d7fa700 1 -- 172.21.15.67:0/1623526861 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9754011e20 con 0x7f97600ffc20 2024-04-23T00:00:22.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.074+0000 7f975d7fa700 1 -- 172.21.15.67:0/1623526861 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f975400f2e0 con 0x7f97600ffc20 2024-04-23T00:00:22.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.074+0000 7f9766764700 1 -- 172.21.15.67:0/1623526861 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97601a2f90 con 0x7f97600ffc20 2024-04-23T00:00:22.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.075+0000 7f9766764700 1 -- 172.21.15.67:0/1623526861 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f976004f2e0 con 0x7f97600ffc20 2024-04-23T00:00:22.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.076+0000 7f975d7fa700 1 -- 172.21.15.67:0/1623526861 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9754011960 con 0x7f97600ffc20 2024-04-23T00:00:22.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.076+0000 7f975d7fa700 1 --2- 172.21.15.67:0/1623526861 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f974806ecc0 0x7f9748071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:22.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.076+0000 7f975f7fe700 1 --2- 172.21.15.67:0/1623526861 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f974806ecc0 0x7f9748071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:22.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.076+0000 7f975d7fa700 1 -- 172.21.15.67:0/1623526861 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f975408eba0 con 0x7f97600ffc20 2024-04-23T00:00:22.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.077+0000 7f975f7fe700 1 --2- 172.21.15.67:0/1623526861 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f974806ecc0 0x7f9748071180 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f975000b5c0 tx=0x7f9750005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:00:22.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.081+0000 7f975d7fa700 1 -- 172.21.15.67:0/1623526861 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f97540577e0 con 0x7f97600ffc20 2024-04-23T00:00:22.284 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:22 smithi067 ceph-mon[112438]: from='client.35836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:00:22.284 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:22 smithi067 ceph-mon[112438]: from='client.35840 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:00:22.284 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:22 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3250208262' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:00:22.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.336+0000 7f9766764700 1 -- 172.21.15.67:0/1623526861 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f97601a31e0 con 0x7f97600ffc20 2024-04-23T00:00:22.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.336+0000 7f975d7fa700 1 -- 172.21.15.67:0/1623526861 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1872 (secure 0 0 0) 0x7f975405aea0 con 0x7f97600ffc20 2024-04-23T00:00:22.327 INFO:teuthology.orchestra.run.smithi067.stdout:e33 2024-04-23T00:00:22.327 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-23T00:00:22.327 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-23T00:00:22.327 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-23T00:00:22.327 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:00:22.327 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-23T00:00:22.327 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-23T00:00:22.327 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-23T00:00:22.328 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-23T00:00:22.328 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-23T00:00:22.328 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-23T00:00:22.328 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-23T00:00:22.328 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-23T00:00:22.328 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-23T00:00:22.328 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-23T00:00:22.328 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-23T00:00:22.328 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-23T00:00:22.328 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-23T00:00:22.328 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-23T00:00:22.328 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-23T00:00:22.328 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-23T00:00:22.328 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-23T00:00:22.328 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-23T00:00:22.329 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-23T00:00:22.329 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-23T00:00:22.329 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-23T00:00:22.329 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-23T00:00:22.329 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-23T00:00:22.329 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-23T00:00:22.329 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-23T00:00:22.329 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-23T00:00:22.329 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-23T00:00:22.329 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-23T00:00:22.329 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-23T00:00:22.329 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:00:22.329 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:00:22.329 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-23T00:00:22.330 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:00:22.330 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T00:00:22.330 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T00:00:22.330 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T00:00:22.330 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44851} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1849648622,v1:172.21.15.134:6827/1849648622] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T00:00:22.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.338+0000 7f9766764700 1 -- 172.21.15.67:0/1623526861 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f974806ecc0 msgr2=0x7f9748071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:22.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.338+0000 7f9766764700 1 --2- 172.21.15.67:0/1623526861 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f974806ecc0 0x7f9748071180 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f975000b5c0 tx=0x7f9750005960 comp rx=0 tx=0).stop 2024-04-23T00:00:22.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.338+0000 7f9766764700 1 -- 172.21.15.67:0/1623526861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97600ffc20 msgr2=0x7f9760101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:22.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.338+0000 7f9766764700 1 --2- 172.21.15.67:0/1623526861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97600ffc20 0x7f9760101840 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f975400b4b0 tx=0x7f975400b870 comp rx=0 tx=0).stop 2024-04-23T00:00:22.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.339+0000 7f9766764700 1 -- 172.21.15.67:0/1623526861 shutdown_connections 2024-04-23T00:00:22.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.339+0000 7f9766764700 1 --2- 172.21.15.67:0/1623526861 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f974806ecc0 0x7f9748071180 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:22.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.339+0000 7f9766764700 1 --2- 172.21.15.67:0/1623526861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97600ffc20 0x7f9760101840 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:22.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.339+0000 7f9766764700 1 --2- 172.21.15.67:0/1623526861 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9760105bd0 0x7f9760101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:22.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.339+0000 7f9766764700 1 -- 172.21.15.67:0/1623526861 >> 172.21.15.67:0/1623526861 conn(0x7f9760075670 msgr2=0x7f97600686e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:00:22.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.339+0000 7f9766764700 1 -- 172.21.15.67:0/1623526861 shutdown_connections 2024-04-23T00:00:22.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.339+0000 7f9766764700 1 -- 172.21.15.67:0/1623526861 wait complete. 2024-04-23T00:00:22.331 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 33 2024-04-23T00:00:22.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.446+0000 7fa2a1c9e700 1 -- 172.21.15.67:0/3472770665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa29c101850 msgr2=0x7fa29c1058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:22.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.446+0000 7fa2a1c9e700 1 --2- 172.21.15.67:0/3472770665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa29c101850 0x7fa29c1058a0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fa28c009910 tx=0x7fa28c009c20 comp rx=0 tx=0).stop 2024-04-23T00:00:22.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.446+0000 7fa2a1c9e700 1 -- 172.21.15.67:0/3472770665 shutdown_connections 2024-04-23T00:00:22.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.446+0000 7fa2a1c9e700 1 --2- 172.21.15.67:0/3472770665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa29c101850 0x7fa29c1058a0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:22.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.446+0000 7fa2a1c9e700 1 --2- 172.21.15.67:0/3472770665 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa29c100ea0 0x7fa29c101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:22.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.446+0000 7fa2a1c9e700 1 -- 172.21.15.67:0/3472770665 >> 172.21.15.67:0/3472770665 conn(0x7fa29c0fc750 msgr2=0x7fa29c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:00:22.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.446+0000 7fa2a1c9e700 1 -- 172.21.15.67:0/3472770665 shutdown_connections 2024-04-23T00:00:22.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.446+0000 7fa2a1c9e700 1 -- 172.21.15.67:0/3472770665 wait complete. 2024-04-23T00:00:22.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.447+0000 7fa2a1c9e700 1 Processor -- start 2024-04-23T00:00:22.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.447+0000 7fa2a1c9e700 1 -- start start 2024-04-23T00:00:22.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.448+0000 7fa2a1c9e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa29c100ea0 0x7fa29c073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:22.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.448+0000 7fa2a1c9e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa29c101850 0x7fa29c0735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:22.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.448+0000 7fa2a1c9e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c073b10 con 0x7fa29c101850 2024-04-23T00:00:22.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.448+0000 7fa2a1c9e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c073c50 con 0x7fa29c100ea0 2024-04-23T00:00:22.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.448+0000 7fa29affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa29c101850 0x7fa29c0735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:22.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.448+0000 7fa29affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa29c101850 0x7fa29c0735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50418/0 (socket says 172.21.15.67:50418) 2024-04-23T00:00:22.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.448+0000 7fa29affd700 1 -- 172.21.15.67:0/4182326514 learned_addr learned my addr 172.21.15.67:0/4182326514 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:00:22.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.448+0000 7fa29b7fe700 1 --2- 172.21.15.67:0/4182326514 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa29c100ea0 0x7fa29c073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:22.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.448+0000 7fa29affd700 1 -- 172.21.15.67:0/4182326514 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa29c100ea0 msgr2=0x7fa29c073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:22.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.448+0000 7fa29affd700 1 --2- 172.21.15.67:0/4182326514 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa29c100ea0 0x7fa29c073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:22.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.448+0000 7fa29affd700 1 -- 172.21.15.67:0/4182326514 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa28c0095f0 con 0x7fa29c101850 2024-04-23T00:00:22.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.448+0000 7fa29b7fe700 1 --2- 172.21.15.67:0/4182326514 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa29c100ea0 0x7fa29c073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T00:00:22.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.448+0000 7fa29affd700 1 --2- 172.21.15.67:0/4182326514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa29c101850 0x7fa29c0735d0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fa28c00bda0 tx=0x7fa28c00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:00:22.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.448+0000 7fa298ff9700 1 -- 172.21.15.67:0/4182326514 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa28c01d070 con 0x7fa29c101850 2024-04-23T00:00:22.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.448+0000 7fa2a1c9e700 1 -- 172.21.15.67:0/4182326514 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa29c073ed0 con 0x7fa29c101850 2024-04-23T00:00:22.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.448+0000 7fa298ff9700 1 -- 172.21.15.67:0/4182326514 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa28c004fb0 con 0x7fa29c101850 2024-04-23T00:00:22.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.448+0000 7fa298ff9700 1 -- 172.21.15.67:0/4182326514 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa28c00f480 con 0x7fa29c101850 2024-04-23T00:00:22.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.448+0000 7fa2a1c9e700 1 -- 172.21.15.67:0/4182326514 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa29c1a2de0 con 0x7fa29c101850 2024-04-23T00:00:22.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.449+0000 7fa2a1c9e700 1 -- 172.21.15.67:0/4182326514 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa29c1092a0 con 0x7fa29c101850 2024-04-23T00:00:22.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.450+0000 7fa298ff9700 1 -- 172.21.15.67:0/4182326514 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa28c022870 con 0x7fa29c101850 2024-04-23T00:00:22.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.450+0000 7fa298ff9700 1 --2- 172.21.15.67:0/4182326514 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa28406ecc0 0x7fa284071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:22.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.450+0000 7fa298ff9700 1 -- 172.21.15.67:0/4182326514 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fa28c090600 con 0x7fa29c101850 2024-04-23T00:00:22.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.451+0000 7fa29b7fe700 1 --2- 172.21.15.67:0/4182326514 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa28406ecc0 0x7fa284071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:22.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.451+0000 7fa29b7fe700 1 --2- 172.21.15.67:0/4182326514 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa28406ecc0 0x7fa284071180 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fa290005fd0 tx=0x7fa290005e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:00:22.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.455+0000 7fa298ff9700 1 -- 172.21.15.67:0/4182326514 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fa28c059370 con 0x7fa29c101850 2024-04-23T00:00:22.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:22 smithi134 ceph-mon[85630]: from='client.35836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:00:22.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:22 smithi134 ceph-mon[85630]: from='client.35840 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:00:22.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:22 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3250208262' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:00:22.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.687+0000 7fa2a1c9e700 1 -- 172.21.15.67:0/4182326514 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa29c0746d0 con 0x7fa28406ecc0 2024-04-23T00:00:22.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.688+0000 7fa298ff9700 1 -- 172.21.15.67:0/4182326514 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa29c0746d0 con 0x7fa28406ecc0 2024-04-23T00:00:22.678 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-23T00:00:22.678 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-23T00:00:22.678 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-23T00:00:22.678 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-23T00:00:22.679 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-23T00:00:22.679 INFO:teuthology.orchestra.run.smithi067.stdout: "mds", 2024-04-23T00:00:22.679 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-23T00:00:22.679 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-23T00:00:22.679 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-23T00:00:22.679 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-23T00:00:22.679 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-23T00:00:22.679 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "16/23 daemons upgraded", 2024-04-23T00:00:22.679 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-23T00:00:22.679 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-23T00:00:22.679 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-23T00:00:22.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.690+0000 7fa2a1c9e700 1 -- 172.21.15.67:0/4182326514 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa28406ecc0 msgr2=0x7fa284071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:22.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.690+0000 7fa2a1c9e700 1 --2- 172.21.15.67:0/4182326514 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa28406ecc0 0x7fa284071180 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fa290005fd0 tx=0x7fa290005e30 comp rx=0 tx=0).stop 2024-04-23T00:00:22.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.690+0000 7fa2a1c9e700 1 -- 172.21.15.67:0/4182326514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa29c101850 msgr2=0x7fa29c0735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:22.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.690+0000 7fa2a1c9e700 1 --2- 172.21.15.67:0/4182326514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa29c101850 0x7fa29c0735d0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fa28c00bda0 tx=0x7fa28c00be80 comp rx=0 tx=0).stop 2024-04-23T00:00:22.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.690+0000 7fa2a1c9e700 1 -- 172.21.15.67:0/4182326514 shutdown_connections 2024-04-23T00:00:22.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.690+0000 7fa2a1c9e700 1 --2- 172.21.15.67:0/4182326514 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa28406ecc0 0x7fa284071180 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:22.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.690+0000 7fa2a1c9e700 1 --2- 172.21.15.67:0/4182326514 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa29c100ea0 0x7fa29c073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:22.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.690+0000 7fa2a1c9e700 1 --2- 172.21.15.67:0/4182326514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa29c101850 0x7fa29c0735d0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:22.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.690+0000 7fa2a1c9e700 1 -- 172.21.15.67:0/4182326514 >> 172.21.15.67:0/4182326514 conn(0x7fa29c0fc750 msgr2=0x7fa29c0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:00:22.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.690+0000 7fa2a1c9e700 1 -- 172.21.15.67:0/4182326514 shutdown_connections 2024-04-23T00:00:22.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.690+0000 7fa2a1c9e700 1 -- 172.21.15.67:0/4182326514 wait complete. 2024-04-23T00:00:22.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.797+0000 7fe892ba2700 1 -- 172.21.15.67:0/4016274428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe88c068580 msgr2=0x7fe88c068a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:22.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.797+0000 7fe892ba2700 1 --2- 172.21.15.67:0/4016274428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe88c068580 0x7fe88c068a00 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fe880009910 tx=0x7fe880009c20 comp rx=0 tx=0).stop 2024-04-23T00:00:22.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.797+0000 7fe892ba2700 1 -- 172.21.15.67:0/4016274428 shutdown_connections 2024-04-23T00:00:22.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.797+0000 7fe892ba2700 1 --2- 172.21.15.67:0/4016274428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe88c068580 0x7fe88c068a00 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:22.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.797+0000 7fe892ba2700 1 --2- 172.21.15.67:0/4016274428 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe88c106a50 0x7fe88c106e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:22.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.797+0000 7fe892ba2700 1 -- 172.21.15.67:0/4016274428 >> 172.21.15.67:0/4016274428 conn(0x7fe88c075460 msgr2=0x7fe88c075870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:00:22.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.797+0000 7fe892ba2700 1 -- 172.21.15.67:0/4016274428 shutdown_connections 2024-04-23T00:00:22.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.797+0000 7fe892ba2700 1 -- 172.21.15.67:0/4016274428 wait complete. 2024-04-23T00:00:22.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.798+0000 7fe892ba2700 1 Processor -- start 2024-04-23T00:00:22.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.798+0000 7fe892ba2700 1 -- start start 2024-04-23T00:00:22.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.798+0000 7fe892ba2700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe88c068580 0x7fe88c19c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:22.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.798+0000 7fe892ba2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe88c106a50 0x7fe88c19cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:22.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.798+0000 7fe892ba2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe88c196600 con 0x7fe88c106a50 2024-04-23T00:00:22.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.798+0000 7fe892ba2700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe88c196770 con 0x7fe88c068580 2024-04-23T00:00:22.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.798+0000 7fe89139f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe88c106a50 0x7fe88c19cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:22.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.798+0000 7fe89139f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe88c106a50 0x7fe88c19cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50432/0 (socket says 172.21.15.67:50432) 2024-04-23T00:00:22.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.798+0000 7fe89139f700 1 -- 172.21.15.67:0/1939984178 learned_addr learned my addr 172.21.15.67:0/1939984178 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:00:22.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.798+0000 7fe891ba0700 1 --2- 172.21.15.67:0/1939984178 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe88c068580 0x7fe88c19c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:22.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.798+0000 7fe89139f700 1 -- 172.21.15.67:0/1939984178 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe88c068580 msgr2=0x7fe88c19c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:22.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.798+0000 7fe89139f700 1 --2- 172.21.15.67:0/1939984178 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe88c068580 0x7fe88c19c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:22.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.798+0000 7fe89139f700 1 -- 172.21.15.67:0/1939984178 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8800095f0 con 0x7fe88c106a50 2024-04-23T00:00:22.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.798+0000 7fe89139f700 1 --2- 172.21.15.67:0/1939984178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe88c106a50 0x7fe88c19cb50 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fe880006010 tx=0x7fe88000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:00:22.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.799+0000 7fe88affd700 1 -- 172.21.15.67:0/1939984178 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe88001d070 con 0x7fe88c106a50 2024-04-23T00:00:22.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.799+0000 7fe88affd700 1 -- 172.21.15.67:0/1939984178 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe880004fb0 con 0x7fe88c106a50 2024-04-23T00:00:22.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.799+0000 7fe892ba2700 1 -- 172.21.15.67:0/1939984178 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe88c1969f0 con 0x7fe88c106a50 2024-04-23T00:00:22.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.799+0000 7fe88affd700 1 -- 172.21.15.67:0/1939984178 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe880022ea0 con 0x7fe88c106a50 2024-04-23T00:00:22.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.799+0000 7fe892ba2700 1 -- 172.21.15.67:0/1939984178 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe88c196ee0 con 0x7fe88c106a50 2024-04-23T00:00:22.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.800+0000 7fe892ba2700 1 -- 172.21.15.67:0/1939984178 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe88c108f90 con 0x7fe88c106a50 2024-04-23T00:00:22.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.800+0000 7fe88affd700 1 -- 172.21.15.67:0/1939984178 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe880022470 con 0x7fe88c106a50 2024-04-23T00:00:22.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.801+0000 7fe88affd700 1 --2- 172.21.15.67:0/1939984178 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe87406ecc0 0x7fe874071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:22.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.801+0000 7fe88affd700 1 -- 172.21.15.67:0/1939984178 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fe880090600 con 0x7fe88c106a50 2024-04-23T00:00:22.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.801+0000 7fe891ba0700 1 --2- 172.21.15.67:0/1939984178 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe87406ecc0 0x7fe874071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:22.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.801+0000 7fe891ba0700 1 --2- 172.21.15.67:0/1939984178 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe87406ecc0 0x7fe874071180 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fe87c009f20 tx=0x7fe87c0094f0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:00:22.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:22.805+0000 7fe88affd700 1 -- 172.21.15.67:0/1939984178 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe880059250 con 0x7fe88c106a50 2024-04-23T00:00:23.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:23.097+0000 7fe892ba2700 1 -- 172.21.15.67:0/1939984178 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe88c197bd0 con 0x7fe88c106a50 2024-04-23T00:00:23.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:23.097+0000 7fe88affd700 1 -- 172.21.15.67:0/1939984178 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4505 (secure 0 0 0) 0x7fe880022720 con 0x7fe88c106a50 2024-04-23T00:00:23.088 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-23T00:00:23.088 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-23T00:00:23.088 INFO:teuthology.orchestra.run.smithi067.stdout: host smithi134 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-23T00:00:23.088 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-23T00:00:23.088 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-23T00:00:23.089 INFO:teuthology.orchestra.run.smithi067.stdout:Traceback (most recent call last): 2024-04-23T00:00:23.089 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-23T00:00:23.089 INFO:teuthology.orchestra.run.smithi067.stdout: return _run_code(code, main_globals, None, 2024-04-23T00:00:23.089 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-23T00:00:23.089 INFO:teuthology.orchestra.run.smithi067.stdout: exec(code, run_globals) 2024-04-23T00:00:23.089 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5642, in 2024-04-23T00:00:23.089 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5630, in main 2024-04-23T00:00:23.089 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 408, in _infer_config 2024-04-23T00:00:23.089 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 323, in _infer_fsid 2024-04-23T00:00:23.089 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 436, in _infer_image 2024-04-23T00:00:23.089 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 310, in _validate_fsid 2024-04-23T00:00:23.089 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 3380, in command_ceph_volume 2024-04-23T00:00:23.089 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-23T00:00:23.089 INFO:teuthology.orchestra.run.smithi067.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-23T00:00:23.090 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-23T00:00:23.090 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-23T00:00:23.090 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-23T00:00:23.090 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-23T00:00:23.090 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-23T00:00:23.090 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-23T00:00:23.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:23.099+0000 7fe892ba2700 1 -- 172.21.15.67:0/1939984178 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe87406ecc0 msgr2=0x7fe874071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:23.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:23.099+0000 7fe892ba2700 1 --2- 172.21.15.67:0/1939984178 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe87406ecc0 0x7fe874071180 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fe87c009f20 tx=0x7fe87c0094f0 comp rx=0 tx=0).stop 2024-04-23T00:00:23.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:23.099+0000 7fe892ba2700 1 -- 172.21.15.67:0/1939984178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe88c106a50 msgr2=0x7fe88c19cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:23.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:23.099+0000 7fe892ba2700 1 --2- 172.21.15.67:0/1939984178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe88c106a50 0x7fe88c19cb50 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fe880006010 tx=0x7fe88000be20 comp rx=0 tx=0).stop 2024-04-23T00:00:23.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:23.100+0000 7fe892ba2700 1 -- 172.21.15.67:0/1939984178 shutdown_connections 2024-04-23T00:00:23.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:23.100+0000 7fe892ba2700 1 --2- 172.21.15.67:0/1939984178 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe87406ecc0 0x7fe874071180 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:23.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:23.100+0000 7fe892ba2700 1 --2- 172.21.15.67:0/1939984178 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe88c068580 0x7fe88c19c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:23.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:23.100+0000 7fe892ba2700 1 --2- 172.21.15.67:0/1939984178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe88c106a50 0x7fe88c19cb50 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:23.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:23.100+0000 7fe892ba2700 1 -- 172.21.15.67:0/1939984178 >> 172.21.15.67:0/1939984178 conn(0x7fe88c075460 msgr2=0x7fe88c0feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:00:23.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:23.100+0000 7fe892ba2700 1 -- 172.21.15.67:0/1939984178 shutdown_connections 2024-04-23T00:00:23.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:23.100+0000 7fe892ba2700 1 -- 172.21.15.67:0/1939984178 wait complete. 2024-04-23T00:00:23.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:23 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1623526861' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-23T00:00:23.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:23 smithi134 ceph-mon[85630]: from='client.35852 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:00:23.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:23 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1939984178' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T00:00:23.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:23 smithi134 ceph-mon[85630]: pgmap v1022: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:23.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:23 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1623526861' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-23T00:00:23.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:23 smithi067 ceph-mon[112438]: from='client.35852 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:00:23.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:23 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1939984178' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T00:00:23.535 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:23 smithi067 ceph-mon[112438]: pgmap v1022: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:25.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:25 smithi134 ceph-mon[85630]: pgmap v1023: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:25.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:25 smithi067 ceph-mon[112438]: pgmap v1023: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:26.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:00:26.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:00:27.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:27 smithi134 ceph-mon[85630]: pgmap v1024: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:27.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:27 smithi067 ceph-mon[112438]: pgmap v1024: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:29.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:29 smithi134 ceph-mon[85630]: pgmap v1025: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:29.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:29 smithi067 ceph-mon[112438]: pgmap v1025: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:31.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:31 smithi134 ceph-mon[85630]: pgmap v1026: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:31.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:31 smithi067 ceph-mon[112438]: pgmap v1026: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:33.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:33 smithi134 ceph-mon[85630]: pgmap v1027: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:33.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:33 smithi067 ceph-mon[112438]: pgmap v1027: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:35.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:35 smithi134 ceph-mon[85630]: pgmap v1028: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:35.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:35 smithi067 ceph-mon[112438]: pgmap v1028: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:37.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:37 smithi134 ceph-mon[85630]: pgmap v1029: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:37.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:37 smithi067 ceph-mon[112438]: pgmap v1029: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:39.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:39 smithi134 ceph-mon[85630]: pgmap v1030: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:39.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:39 smithi067 ceph-mon[112438]: pgmap v1030: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:41.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:00:41.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:00:42.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:42 smithi134 ceph-mon[85630]: pgmap v1031: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:42.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:42 smithi067 ceph-mon[112438]: pgmap v1031: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:43.983 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:43 smithi134 ceph-mon[85630]: pgmap v1032: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:44.034 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:43 smithi067 ceph-mon[112438]: pgmap v1032: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:45.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:45 smithi134 ceph-mon[85630]: pgmap v1033: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:45.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:45 smithi067 ceph-mon[112438]: pgmap v1033: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:47.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:47 smithi134 ceph-mon[85630]: pgmap v1034: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:47.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:47 smithi067 ceph-mon[112438]: pgmap v1034: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:49.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:49 smithi134 ceph-mon[85630]: pgmap v1035: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:49.536 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:49 smithi067 ceph-mon[112438]: pgmap v1035: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:51.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:51 smithi134 ceph-mon[85630]: pgmap v1036: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:51.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:51 smithi067 ceph-mon[112438]: pgmap v1036: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:53.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.222+0000 7f23b60f9700 1 -- 172.21.15.67:0/4092849969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23b0100ea0 msgr2=0x7f23b0101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:53.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.222+0000 7f23b60f9700 1 --2- 172.21.15.67:0/4092849969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23b0100ea0 0x7f23b0101280 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f23a0009910 tx=0x7f23a0009c20 comp rx=0 tx=0).stop 2024-04-23T00:00:53.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.222+0000 7f23b60f9700 1 -- 172.21.15.67:0/4092849969 shutdown_connections 2024-04-23T00:00:53.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.222+0000 7f23b60f9700 1 --2- 172.21.15.67:0/4092849969 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23b0101850 0x7f23b01058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:53.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.222+0000 7f23b60f9700 1 --2- 172.21.15.67:0/4092849969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23b0100ea0 0x7f23b0101280 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:53.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.222+0000 7f23b60f9700 1 -- 172.21.15.67:0/4092849969 >> 172.21.15.67:0/4092849969 conn(0x7f23b00fc750 msgr2=0x7f23b00feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:00:53.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.222+0000 7f23b60f9700 1 -- 172.21.15.67:0/4092849969 shutdown_connections 2024-04-23T00:00:53.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.222+0000 7f23b60f9700 1 -- 172.21.15.67:0/4092849969 wait complete. 2024-04-23T00:00:53.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.223+0000 7f23b60f9700 1 Processor -- start 2024-04-23T00:00:53.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.223+0000 7f23b60f9700 1 -- start start 2024-04-23T00:00:53.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.223+0000 7f23b60f9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23b0100ea0 0x7f23b0073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:53.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.223+0000 7f23b60f9700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23b0101850 0x7f23b00735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:53.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.223+0000 7f23b60f9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b0073b10 con 0x7f23b0100ea0 2024-04-23T00:00:53.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.223+0000 7f23b60f9700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b0073c50 con 0x7f23b0101850 2024-04-23T00:00:53.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.223+0000 7f23af7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23b0100ea0 0x7f23b0073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:53.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.223+0000 7f23af7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23b0100ea0 0x7f23b0073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35546/0 (socket says 172.21.15.67:35546) 2024-04-23T00:00:53.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.223+0000 7f23af7fe700 1 -- 172.21.15.67:0/1025067351 learned_addr learned my addr 172.21.15.67:0/1025067351 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:00:53.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.224+0000 7f23aeffd700 1 --2- 172.21.15.67:0/1025067351 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23b0101850 0x7f23b00735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:53.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.224+0000 7f23af7fe700 1 -- 172.21.15.67:0/1025067351 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23b0101850 msgr2=0x7f23b00735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:53.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.224+0000 7f23af7fe700 1 --2- 172.21.15.67:0/1025067351 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23b0101850 0x7f23b00735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:53.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.224+0000 7f23af7fe700 1 -- 172.21.15.67:0/1025067351 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a00095f0 con 0x7f23b0100ea0 2024-04-23T00:00:53.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.224+0000 7f23af7fe700 1 --2- 172.21.15.67:0/1025067351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23b0100ea0 0x7f23b0073090 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f23a0006010 tx=0x7f23a000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:00:53.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.224+0000 7f23acff9700 1 -- 172.21.15.67:0/1025067351 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23a001d070 con 0x7f23b0100ea0 2024-04-23T00:00:53.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.224+0000 7f23acff9700 1 -- 172.21.15.67:0/1025067351 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f23a000f460 con 0x7f23b0100ea0 2024-04-23T00:00:53.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.224+0000 7f23b60f9700 1 -- 172.21.15.67:0/1025067351 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23b0073ed0 con 0x7f23b0100ea0 2024-04-23T00:00:53.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.224+0000 7f23acff9700 1 -- 172.21.15.67:0/1025067351 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23a0017610 con 0x7f23b0100ea0 2024-04-23T00:00:53.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.224+0000 7f23b60f9700 1 -- 172.21.15.67:0/1025067351 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23b01a2e20 con 0x7f23b0100ea0 2024-04-23T00:00:53.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.225+0000 7f23b60f9700 1 -- 172.21.15.67:0/1025067351 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23b0109240 con 0x7f23b0100ea0 2024-04-23T00:00:53.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.226+0000 7f23acff9700 1 -- 172.21.15.67:0/1025067351 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f23a000f5d0 con 0x7f23b0100ea0 2024-04-23T00:00:53.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.226+0000 7f23acff9700 1 --2- 172.21.15.67:0/1025067351 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f239806ecc0 0x7f2398071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:53.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.226+0000 7f23acff9700 1 -- 172.21.15.67:0/1025067351 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f23a0090510 con 0x7f23b0100ea0 2024-04-23T00:00:53.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.226+0000 7f23aeffd700 1 --2- 172.21.15.67:0/1025067351 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f239806ecc0 0x7f2398071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:53.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.227+0000 7f23aeffd700 1 --2- 172.21.15.67:0/1025067351 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f239806ecc0 0x7f2398071180 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f23b0074880 tx=0x7f23a400a300 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:00:53.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.232+0000 7f23acff9700 1 -- 172.21.15.67:0/1025067351 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f23a0059150 con 0x7f23b0100ea0 2024-04-23T00:00:53.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.466+0000 7f23b60f9700 1 -- 172.21.15.67:0/1025067351 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f23b00611d0 con 0x7f239806ecc0 2024-04-23T00:00:53.457 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:53 smithi067 ceph-mon[112438]: pgmap v1037: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:53.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.467+0000 7f23acff9700 1 -- 172.21.15.67:0/1025067351 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f23b00611d0 con 0x7f239806ecc0 2024-04-23T00:00:53.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.469+0000 7f23b60f9700 1 -- 172.21.15.67:0/1025067351 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f239806ecc0 msgr2=0x7f2398071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:53.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.469+0000 7f23b60f9700 1 --2- 172.21.15.67:0/1025067351 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f239806ecc0 0x7f2398071180 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f23b0074880 tx=0x7f23a400a300 comp rx=0 tx=0).stop 2024-04-23T00:00:53.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.469+0000 7f23b60f9700 1 -- 172.21.15.67:0/1025067351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23b0100ea0 msgr2=0x7f23b0073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:53.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.469+0000 7f23b60f9700 1 --2- 172.21.15.67:0/1025067351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23b0100ea0 0x7f23b0073090 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f23a0006010 tx=0x7f23a000bd70 comp rx=0 tx=0).stop 2024-04-23T00:00:53.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.469+0000 7f23b60f9700 1 -- 172.21.15.67:0/1025067351 shutdown_connections 2024-04-23T00:00:53.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.469+0000 7f23b60f9700 1 --2- 172.21.15.67:0/1025067351 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f239806ecc0 0x7f2398071180 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:53.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.469+0000 7f23b60f9700 1 --2- 172.21.15.67:0/1025067351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23b0100ea0 0x7f23b0073090 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:53.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.469+0000 7f23b60f9700 1 --2- 172.21.15.67:0/1025067351 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23b0101850 0x7f23b00735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:53.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.469+0000 7f23b60f9700 1 -- 172.21.15.67:0/1025067351 >> 172.21.15.67:0/1025067351 conn(0x7f23b00fc750 msgr2=0x7f23b00fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:00:53.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.469+0000 7f23b60f9700 1 -- 172.21.15.67:0/1025067351 shutdown_connections 2024-04-23T00:00:53.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.469+0000 7f23b60f9700 1 -- 172.21.15.67:0/1025067351 wait complete. 2024-04-23T00:00:53.470 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-23T00:00:53.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:53 smithi134 ceph-mon[85630]: pgmap v1037: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:53.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.577+0000 7f1fa801e700 1 -- 172.21.15.67:0/749055963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fa0101850 msgr2=0x7f1fa01058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:53.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.577+0000 7f1fa801e700 1 --2- 172.21.15.67:0/749055963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fa0101850 0x7f1fa01058a0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f1f90009910 tx=0x7f1f90009c20 comp rx=0 tx=0).stop 2024-04-23T00:00:53.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.578+0000 7f1fa801e700 1 -- 172.21.15.67:0/749055963 shutdown_connections 2024-04-23T00:00:53.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.578+0000 7f1fa801e700 1 --2- 172.21.15.67:0/749055963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fa0101850 0x7f1fa01058a0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:53.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.578+0000 7f1fa801e700 1 --2- 172.21.15.67:0/749055963 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1fa0100ea0 0x7f1fa0101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:53.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.578+0000 7f1fa801e700 1 -- 172.21.15.67:0/749055963 >> 172.21.15.67:0/749055963 conn(0x7f1fa00fc750 msgr2=0x7f1fa00feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:00:53.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.578+0000 7f1fa801e700 1 -- 172.21.15.67:0/749055963 shutdown_connections 2024-04-23T00:00:53.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.578+0000 7f1fa801e700 1 -- 172.21.15.67:0/749055963 wait complete. 2024-04-23T00:00:53.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.578+0000 7f1fa801e700 1 Processor -- start 2024-04-23T00:00:53.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.578+0000 7f1fa801e700 1 -- start start 2024-04-23T00:00:53.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.579+0000 7f1fa801e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fa0100ea0 0x7f1fa019c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:53.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.579+0000 7f1fa801e700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1fa0101850 0x7f1fa019cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:53.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.579+0000 7f1fa801e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa019d2d0 con 0x7f1fa0100ea0 2024-04-23T00:00:53.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.579+0000 7f1fa801e700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa0196780 con 0x7f1fa0101850 2024-04-23T00:00:53.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.579+0000 7f1fa5dba700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fa0100ea0 0x7f1fa019c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:53.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.579+0000 7f1fa5dba700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fa0100ea0 0x7f1fa019c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35554/0 (socket says 172.21.15.67:35554) 2024-04-23T00:00:53.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.579+0000 7f1fa5dba700 1 -- 172.21.15.67:0/3096363690 learned_addr learned my addr 172.21.15.67:0/3096363690 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:00:53.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.579+0000 7f1fa55b9700 1 --2- 172.21.15.67:0/3096363690 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1fa0101850 0x7f1fa019cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:53.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.579+0000 7f1fa5dba700 1 -- 172.21.15.67:0/3096363690 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1fa0101850 msgr2=0x7f1fa019cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:53.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.579+0000 7f1fa5dba700 1 --2- 172.21.15.67:0/3096363690 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1fa0101850 0x7f1fa019cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:53.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.579+0000 7f1fa5dba700 1 -- 172.21.15.67:0/3096363690 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f900095f0 con 0x7f1fa0100ea0 2024-04-23T00:00:53.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.579+0000 7f1fa5dba700 1 --2- 172.21.15.67:0/3096363690 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fa0100ea0 0x7f1fa019c700 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c00d8d0 tx=0x7f1f9c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:00:53.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.579+0000 7f1f96ffd700 1 -- 172.21.15.67:0/3096363690 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f9c0098a0 con 0x7f1fa0100ea0 2024-04-23T00:00:53.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.579+0000 7f1f96ffd700 1 -- 172.21.15.67:0/3096363690 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1f9c010460 con 0x7f1fa0100ea0 2024-04-23T00:00:53.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.579+0000 7f1fa801e700 1 -- 172.21.15.67:0/3096363690 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fa0196a60 con 0x7f1fa0100ea0 2024-04-23T00:00:53.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.579+0000 7f1f96ffd700 1 -- 172.21.15.67:0/3096363690 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f9c00f5f0 con 0x7f1fa0100ea0 2024-04-23T00:00:53.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.580+0000 7f1fa801e700 1 -- 172.21.15.67:0/3096363690 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fa0196fb0 con 0x7f1fa0100ea0 2024-04-23T00:00:53.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.580+0000 7f1fa801e700 1 -- 172.21.15.67:0/3096363690 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fa01092a0 con 0x7f1fa0100ea0 2024-04-23T00:00:53.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.581+0000 7f1f96ffd700 1 -- 172.21.15.67:0/3096363690 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1f9c009a00 con 0x7f1fa0100ea0 2024-04-23T00:00:53.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.581+0000 7f1f96ffd700 1 --2- 172.21.15.67:0/3096363690 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1f8c06ecc0 0x7f1f8c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:53.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.581+0000 7f1f96ffd700 1 -- 172.21.15.67:0/3096363690 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f1f9c08eda0 con 0x7f1fa0100ea0 2024-04-23T00:00:53.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.581+0000 7f1fa55b9700 1 --2- 172.21.15.67:0/3096363690 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1f8c06ecc0 0x7f1f8c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:53.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.582+0000 7f1fa55b9700 1 --2- 172.21.15.67:0/3096363690 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1f8c06ecc0 0x7f1f8c071180 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0197bf0 tx=0x7f1f9001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:00:53.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.586+0000 7f1f96ffd700 1 -- 172.21.15.67:0/3096363690 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1f9c05b590 con 0x7f1fa0100ea0 2024-04-23T00:00:53.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.817+0000 7f1fa801e700 1 -- 172.21.15.67:0/3096363690 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1fa0197da0 con 0x7f1f8c06ecc0 2024-04-23T00:00:53.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.818+0000 7f1f96ffd700 1 -- 172.21.15.67:0/3096363690 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f1fa0197da0 con 0x7f1f8c06ecc0 2024-04-23T00:00:53.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.820+0000 7f1fa801e700 1 -- 172.21.15.67:0/3096363690 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1f8c06ecc0 msgr2=0x7f1f8c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:53.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.820+0000 7f1fa801e700 1 --2- 172.21.15.67:0/3096363690 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1f8c06ecc0 0x7f1f8c071180 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0197bf0 tx=0x7f1f9001a040 comp rx=0 tx=0).stop 2024-04-23T00:00:53.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.820+0000 7f1fa801e700 1 -- 172.21.15.67:0/3096363690 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fa0100ea0 msgr2=0x7f1fa019c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:53.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.820+0000 7f1fa801e700 1 --2- 172.21.15.67:0/3096363690 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fa0100ea0 0x7f1fa019c700 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c00d8d0 tx=0x7f1f9c00dbe0 comp rx=0 tx=0).stop 2024-04-23T00:00:53.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.820+0000 7f1fa801e700 1 -- 172.21.15.67:0/3096363690 shutdown_connections 2024-04-23T00:00:53.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.820+0000 7f1fa801e700 1 --2- 172.21.15.67:0/3096363690 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1f8c06ecc0 0x7f1f8c071180 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:53.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.820+0000 7f1fa801e700 1 --2- 172.21.15.67:0/3096363690 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fa0100ea0 0x7f1fa019c700 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:53.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.820+0000 7f1fa801e700 1 --2- 172.21.15.67:0/3096363690 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1fa0101850 0x7f1fa019cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:53.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.820+0000 7f1fa801e700 1 -- 172.21.15.67:0/3096363690 >> 172.21.15.67:0/3096363690 conn(0x7f1fa00fc750 msgr2=0x7f1fa00fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:00:53.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.821+0000 7f1fa801e700 1 -- 172.21.15.67:0/3096363690 shutdown_connections 2024-04-23T00:00:53.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.821+0000 7f1fa801e700 1 -- 172.21.15.67:0/3096363690 wait complete. 2024-04-23T00:00:53.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.927+0000 7f79d64c7700 1 -- 172.21.15.67:0/4281569766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79d0100ea0 msgr2=0x7f79d0101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:53.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.927+0000 7f79d64c7700 1 --2- 172.21.15.67:0/4281569766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79d0100ea0 0x7f79d0101280 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f79c0009910 tx=0x7f79c0009c20 comp rx=0 tx=0).stop 2024-04-23T00:00:53.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.928+0000 7f79d64c7700 1 -- 172.21.15.67:0/4281569766 shutdown_connections 2024-04-23T00:00:53.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.928+0000 7f79d64c7700 1 --2- 172.21.15.67:0/4281569766 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79d0101850 0x7f79d01058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:53.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.928+0000 7f79d64c7700 1 --2- 172.21.15.67:0/4281569766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79d0100ea0 0x7f79d0101280 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:53.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.928+0000 7f79d64c7700 1 -- 172.21.15.67:0/4281569766 >> 172.21.15.67:0/4281569766 conn(0x7f79d00fc750 msgr2=0x7f79d00feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:00:53.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.928+0000 7f79d64c7700 1 -- 172.21.15.67:0/4281569766 shutdown_connections 2024-04-23T00:00:53.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.928+0000 7f79d64c7700 1 -- 172.21.15.67:0/4281569766 wait complete. 2024-04-23T00:00:53.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.928+0000 7f79d64c7700 1 Processor -- start 2024-04-23T00:00:53.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.928+0000 7f79d64c7700 1 -- start start 2024-04-23T00:00:53.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.929+0000 7f79d64c7700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79d0100ea0 0x7f79d0073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:53.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.929+0000 7f79d64c7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79d0101850 0x7f79d00735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:53.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.929+0000 7f79d64c7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d0073b10 con 0x7f79d0101850 2024-04-23T00:00:53.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.929+0000 7f79d64c7700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d0073c50 con 0x7f79d0100ea0 2024-04-23T00:00:53.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.929+0000 7f79cf7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79d0101850 0x7f79d00735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:53.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.929+0000 7f79cf7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79d0101850 0x7f79d00735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35568/0 (socket says 172.21.15.67:35568) 2024-04-23T00:00:53.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.929+0000 7f79cf7fe700 1 -- 172.21.15.67:0/1061072635 learned_addr learned my addr 172.21.15.67:0/1061072635 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:00:53.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.929+0000 7f79cffff700 1 --2- 172.21.15.67:0/1061072635 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79d0100ea0 0x7f79d0073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:53.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.929+0000 7f79cf7fe700 1 -- 172.21.15.67:0/1061072635 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79d0100ea0 msgr2=0x7f79d0073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:53.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.929+0000 7f79cf7fe700 1 --2- 172.21.15.67:0/1061072635 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79d0100ea0 0x7f79d0073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:53.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.929+0000 7f79cf7fe700 1 -- 172.21.15.67:0/1061072635 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c00095f0 con 0x7f79d0101850 2024-04-23T00:00:53.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.929+0000 7f79cf7fe700 1 --2- 172.21.15.67:0/1061072635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79d0101850 0x7f79d00735d0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f79c400d900 tx=0x7f79c400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:00:53.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.930+0000 7f79cd7fa700 1 -- 172.21.15.67:0/1061072635 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79c4009900 con 0x7f79d0101850 2024-04-23T00:00:53.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.930+0000 7f79cd7fa700 1 -- 172.21.15.67:0/1061072635 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f79c4010460 con 0x7f79d0101850 2024-04-23T00:00:53.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.930+0000 7f79d64c7700 1 -- 172.21.15.67:0/1061072635 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79d01a2af0 con 0x7f79d0101850 2024-04-23T00:00:53.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.930+0000 7f79cd7fa700 1 -- 172.21.15.67:0/1061072635 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79c400f5f0 con 0x7f79d0101850 2024-04-23T00:00:53.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.930+0000 7f79d64c7700 1 -- 172.21.15.67:0/1061072635 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79d01a2f00 con 0x7f79d0101850 2024-04-23T00:00:53.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.931+0000 7f79d64c7700 1 -- 172.21.15.67:0/1061072635 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79d0109240 con 0x7f79d0101850 2024-04-23T00:00:53.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.931+0000 7f79cd7fa700 1 -- 172.21.15.67:0/1061072635 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f79c40105d0 con 0x7f79d0101850 2024-04-23T00:00:53.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.932+0000 7f79cd7fa700 1 --2- 172.21.15.67:0/1061072635 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f79b806ecc0 0x7f79b8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:53.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.932+0000 7f79cd7fa700 1 -- 172.21.15.67:0/1061072635 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f79c408eb70 con 0x7f79d0101850 2024-04-23T00:00:53.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.932+0000 7f79cffff700 1 --2- 172.21.15.67:0/1061072635 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f79b806ecc0 0x7f79b8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:53.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.932+0000 7f79cffff700 1 --2- 172.21.15.67:0/1061072635 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f79b806ecc0 0x7f79b8071180 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f79c0006010 tx=0x7f79c001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:00:53.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:53.937+0000 7f79cd7fa700 1 -- 172.21.15.67:0/1061072635 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f79c40577b0 con 0x7f79d0101850 2024-04-23T00:00:54.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.160+0000 7f79d64c7700 1 -- 172.21.15.67:0/1061072635 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f79d00611d0 con 0x7f79b806ecc0 2024-04-23T00:00:54.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.171+0000 7f79cd7fa700 1 -- 172.21.15.67:0/1061072635 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f79d00611d0 con 0x7f79b806ecc0 2024-04-23T00:00:54.161 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-23T00:00:54.161 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (34m) 12m ago 45m 24.4M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-23T00:00:54.161 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (46m) 12m ago 46m 10.0M - 18.2.1 5be31c24972a df887f62291e 2024-04-23T00:00:54.161 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (45m) 12m ago 45m 10.0M - 18.2.1 5be31c24972a ad89190b5400 2024-04-23T00:00:54.161 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (31m) 12m ago 46m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-23T00:00:54.161 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (31m) 12m ago 45m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-23T00:00:54.161 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (34m) 12m ago 45m 81.2M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-23T00:00:54.161 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (28m) 12m ago 40m 19.4M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-23T00:00:54.162 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (27m) 12m ago 40m 20.3M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-23T00:00:54.162 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (27m) 12m ago 40m 20.1M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-23T00:00:54.162 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (12m) 12m ago 40m 14.0M - 19.0.0-3252-g43be0201 a0434699c1d7 46d3d7e86fee 2024-04-23T00:00:54.162 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (36m) 12m ago 47m 514M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-23T00:00:54.162 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (35m) 12m ago 44m 459M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-23T00:00:54.162 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (33m) 12m ago 47m 74.2M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-23T00:00:54.162 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (32m) 12m ago 44m 54.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-23T00:00:54.162 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (35m) 12m ago 45m 15.2M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-23T00:00:54.162 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (35m) 12m ago 44m 16.0M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-23T00:00:54.162 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (31m) 12m ago 44m 102M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-23T00:00:54.162 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (30m) 12m ago 43m 128M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-23T00:00:54.162 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (30m) 12m ago 43m 83.8M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-23T00:00:54.162 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (29m) 12m ago 42m 101M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-23T00:00:54.163 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (29m) 12m ago 42m 86.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-23T00:00:54.163 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (28m) 12m ago 42m 81.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-23T00:00:54.163 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (34m) 12m ago 45m 78.7M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-23T00:00:54.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.173+0000 7f79d64c7700 1 -- 172.21.15.67:0/1061072635 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f79b806ecc0 msgr2=0x7f79b8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:54.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.173+0000 7f79d64c7700 1 --2- 172.21.15.67:0/1061072635 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f79b806ecc0 0x7f79b8071180 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f79c0006010 tx=0x7f79c001a040 comp rx=0 tx=0).stop 2024-04-23T00:00:54.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.173+0000 7f79d64c7700 1 -- 172.21.15.67:0/1061072635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79d0101850 msgr2=0x7f79d00735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:54.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.173+0000 7f79d64c7700 1 --2- 172.21.15.67:0/1061072635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79d0101850 0x7f79d00735d0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f79c400d900 tx=0x7f79c400dc10 comp rx=0 tx=0).stop 2024-04-23T00:00:54.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.173+0000 7f79d64c7700 1 -- 172.21.15.67:0/1061072635 shutdown_connections 2024-04-23T00:00:54.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.173+0000 7f79d64c7700 1 --2- 172.21.15.67:0/1061072635 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f79b806ecc0 0x7f79b8071180 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:54.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.173+0000 7f79d64c7700 1 --2- 172.21.15.67:0/1061072635 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79d0100ea0 0x7f79d0073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:54.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.173+0000 7f79d64c7700 1 --2- 172.21.15.67:0/1061072635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79d0101850 0x7f79d00735d0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:54.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.173+0000 7f79d64c7700 1 -- 172.21.15.67:0/1061072635 >> 172.21.15.67:0/1061072635 conn(0x7f79d00fc750 msgr2=0x7f79d00fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:00:54.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.173+0000 7f79d64c7700 1 -- 172.21.15.67:0/1061072635 shutdown_connections 2024-04-23T00:00:54.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.173+0000 7f79d64c7700 1 -- 172.21.15.67:0/1061072635 wait complete. 2024-04-23T00:00:54.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.280+0000 7f0a680a9700 1 -- 172.21.15.67:0/3602011669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a600ff5a0 msgr2=0x7f0a600ff980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:54.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.280+0000 7f0a680a9700 1 --2- 172.21.15.67:0/3602011669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a600ff5a0 0x7f0a600ff980 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c009910 tx=0x7f0a5c009c20 comp rx=0 tx=0).stop 2024-04-23T00:00:54.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.280+0000 7f0a680a9700 1 -- 172.21.15.67:0/3602011669 shutdown_connections 2024-04-23T00:00:54.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.280+0000 7f0a680a9700 1 --2- 172.21.15.67:0/3602011669 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0a600fff50 0x7f0a60105840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:54.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.280+0000 7f0a680a9700 1 --2- 172.21.15.67:0/3602011669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a600ff5a0 0x7f0a600ff980 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:54.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.280+0000 7f0a680a9700 1 -- 172.21.15.67:0/3602011669 >> 172.21.15.67:0/3602011669 conn(0x7f0a60074e30 msgr2=0x7f0a60075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:00:54.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.280+0000 7f0a680a9700 1 -- 172.21.15.67:0/3602011669 shutdown_connections 2024-04-23T00:00:54.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.280+0000 7f0a680a9700 1 -- 172.21.15.67:0/3602011669 wait complete. 2024-04-23T00:00:54.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.281+0000 7f0a680a9700 1 Processor -- start 2024-04-23T00:00:54.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.281+0000 7f0a680a9700 1 -- start start 2024-04-23T00:00:54.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.281+0000 7f0a680a9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a600ff5a0 0x7f0a6019c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:54.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.281+0000 7f0a680a9700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0a600fff50 0x7f0a6019cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:54.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.281+0000 7f0a680a9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6019d2d0 con 0x7f0a600ff5a0 2024-04-23T00:00:54.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.281+0000 7f0a680a9700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a60196780 con 0x7f0a600fff50 2024-04-23T00:00:54.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.281+0000 7f0a65e45700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a600ff5a0 0x7f0a6019c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:54.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.281+0000 7f0a65e45700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a600ff5a0 0x7f0a6019c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35590/0 (socket says 172.21.15.67:35590) 2024-04-23T00:00:54.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.281+0000 7f0a65e45700 1 -- 172.21.15.67:0/1369558805 learned_addr learned my addr 172.21.15.67:0/1369558805 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:00:54.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.281+0000 7f0a65644700 1 --2- 172.21.15.67:0/1369558805 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0a600fff50 0x7f0a6019cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:54.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.281+0000 7f0a65e45700 1 -- 172.21.15.67:0/1369558805 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0a600fff50 msgr2=0x7f0a6019cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:54.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.281+0000 7f0a65e45700 1 --2- 172.21.15.67:0/1369558805 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0a600fff50 0x7f0a6019cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:54.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.281+0000 7f0a65e45700 1 -- 172.21.15.67:0/1369558805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a5c0095f0 con 0x7f0a600ff5a0 2024-04-23T00:00:54.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.281+0000 7f0a65e45700 1 --2- 172.21.15.67:0/1369558805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a600ff5a0 0x7f0a6019c700 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c005f50 tx=0x7f0a5c00f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:00:54.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.282+0000 7f0a56ffd700 1 -- 172.21.15.67:0/1369558805 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a5c01d070 con 0x7f0a600ff5a0 2024-04-23T00:00:54.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.282+0000 7f0a56ffd700 1 -- 172.21.15.67:0/1369558805 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0a5c00b610 con 0x7f0a600ff5a0 2024-04-23T00:00:54.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.282+0000 7f0a680a9700 1 -- 172.21.15.67:0/1369558805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a60196a00 con 0x7f0a600ff5a0 2024-04-23T00:00:54.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.282+0000 7f0a56ffd700 1 -- 172.21.15.67:0/1369558805 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a5c022470 con 0x7f0a600ff5a0 2024-04-23T00:00:54.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.282+0000 7f0a680a9700 1 -- 172.21.15.67:0/1369558805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a60196ef0 con 0x7f0a600ff5a0 2024-04-23T00:00:54.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.283+0000 7f0a680a9700 1 -- 172.21.15.67:0/1369558805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a60109270 con 0x7f0a600ff5a0 2024-04-23T00:00:54.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.283+0000 7f0a56ffd700 1 -- 172.21.15.67:0/1369558805 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0a5c00fca0 con 0x7f0a600ff5a0 2024-04-23T00:00:54.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.284+0000 7f0a56ffd700 1 --2- 172.21.15.67:0/1369558805 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0a4c06ecc0 0x7f0a4c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:54.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.284+0000 7f0a65644700 1 --2- 172.21.15.67:0/1369558805 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0a4c06ecc0 0x7f0a4c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:54.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.284+0000 7f0a56ffd700 1 -- 172.21.15.67:0/1369558805 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f0a5c090370 con 0x7f0a600ff5a0 2024-04-23T00:00:54.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.284+0000 7f0a65644700 1 --2- 172.21.15.67:0/1369558805 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0a4c06ecc0 0x7f0a4c071180 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f0a60197bf0 tx=0x7f0a50009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:00:54.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.288+0000 7f0a56ffd700 1 -- 172.21.15.67:0/1369558805 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f0a5c05cb60 con 0x7f0a600ff5a0 2024-04-23T00:00:54.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:54 smithi134 ceph-mon[85630]: from='client.35860 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:00:54.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:54 smithi134 ceph-mon[85630]: from='client.35864 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:00:54.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:54 smithi067 ceph-mon[112438]: from='client.35860 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:00:54.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:54 smithi067 ceph-mon[112438]: from='client.35864 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:00:54.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.588+0000 7f0a680a9700 1 -- 172.21.15.67:0/1369558805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0a60197d30 con 0x7f0a600ff5a0 2024-04-23T00:00:54.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.588+0000 7f0a56ffd700 1 -- 172.21.15.67:0/1369558805 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f0a5c05c6f0 con 0x7f0a600ff5a0 2024-04-23T00:00:54.578 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-23T00:00:54.579 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-23T00:00:54.579 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-23T00:00:54.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-23T00:00:54.579 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-23T00:00:54.579 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-23T00:00:54.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-23T00:00:54.579 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-23T00:00:54.579 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-23T00:00:54.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-23T00:00:54.579 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-23T00:00:54.579 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 4 2024-04-23T00:00:54.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-23T00:00:54.579 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-23T00:00:54.579 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 14 2024-04-23T00:00:54.580 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-23T00:00:54.580 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-23T00:00:54.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.590+0000 7f0a680a9700 1 -- 172.21.15.67:0/1369558805 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0a4c06ecc0 msgr2=0x7f0a4c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:54.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.590+0000 7f0a680a9700 1 --2- 172.21.15.67:0/1369558805 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0a4c06ecc0 0x7f0a4c071180 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f0a60197bf0 tx=0x7f0a50009260 comp rx=0 tx=0).stop 2024-04-23T00:00:54.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.590+0000 7f0a680a9700 1 -- 172.21.15.67:0/1369558805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a600ff5a0 msgr2=0x7f0a6019c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:54.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.590+0000 7f0a680a9700 1 --2- 172.21.15.67:0/1369558805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a600ff5a0 0x7f0a6019c700 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c005f50 tx=0x7f0a5c00f740 comp rx=0 tx=0).stop 2024-04-23T00:00:54.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.590+0000 7f0a680a9700 1 -- 172.21.15.67:0/1369558805 shutdown_connections 2024-04-23T00:00:54.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.590+0000 7f0a680a9700 1 --2- 172.21.15.67:0/1369558805 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0a4c06ecc0 0x7f0a4c071180 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:54.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.590+0000 7f0a680a9700 1 --2- 172.21.15.67:0/1369558805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a600ff5a0 0x7f0a6019c700 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:54.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.590+0000 7f0a680a9700 1 --2- 172.21.15.67:0/1369558805 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0a600fff50 0x7f0a6019cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:54.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.590+0000 7f0a680a9700 1 -- 172.21.15.67:0/1369558805 >> 172.21.15.67:0/1369558805 conn(0x7f0a60074e30 msgr2=0x7f0a600fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:00:54.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.591+0000 7f0a680a9700 1 -- 172.21.15.67:0/1369558805 shutdown_connections 2024-04-23T00:00:54.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.591+0000 7f0a680a9700 1 -- 172.21.15.67:0/1369558805 wait complete. 2024-04-23T00:00:54.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.697+0000 7fde6a54d700 1 -- 172.21.15.67:0/428883134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde64100ea0 msgr2=0x7fde64101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:54.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.697+0000 7fde6a54d700 1 --2- 172.21.15.67:0/428883134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde64100ea0 0x7fde64101280 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fde54009910 tx=0x7fde54009c20 comp rx=0 tx=0).stop 2024-04-23T00:00:54.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.697+0000 7fde6a54d700 1 -- 172.21.15.67:0/428883134 shutdown_connections 2024-04-23T00:00:54.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.697+0000 7fde6a54d700 1 --2- 172.21.15.67:0/428883134 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fde64101850 0x7fde641058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:54.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.697+0000 7fde6a54d700 1 --2- 172.21.15.67:0/428883134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde64100ea0 0x7fde64101280 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:54.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.697+0000 7fde6a54d700 1 -- 172.21.15.67:0/428883134 >> 172.21.15.67:0/428883134 conn(0x7fde640fc750 msgr2=0x7fde640feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:00:54.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.697+0000 7fde6a54d700 1 -- 172.21.15.67:0/428883134 shutdown_connections 2024-04-23T00:00:54.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.697+0000 7fde6a54d700 1 -- 172.21.15.67:0/428883134 wait complete. 2024-04-23T00:00:54.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.698+0000 7fde6a54d700 1 Processor -- start 2024-04-23T00:00:54.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.698+0000 7fde6a54d700 1 -- start start 2024-04-23T00:00:54.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.698+0000 7fde6a54d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde64100ea0 0x7fde64073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:54.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.698+0000 7fde6a54d700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fde64101850 0x7fde640735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:54.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.698+0000 7fde6a54d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde64073b10 con 0x7fde64100ea0 2024-04-23T00:00:54.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.698+0000 7fde6a54d700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde64073c50 con 0x7fde64101850 2024-04-23T00:00:54.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.698+0000 7fde63fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde64100ea0 0x7fde64073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:54.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.698+0000 7fde63fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde64100ea0 0x7fde64073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35614/0 (socket says 172.21.15.67:35614) 2024-04-23T00:00:54.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.698+0000 7fde63fff700 1 -- 172.21.15.67:0/774915954 learned_addr learned my addr 172.21.15.67:0/774915954 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:00:54.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.698+0000 7fde637fe700 1 --2- 172.21.15.67:0/774915954 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fde64101850 0x7fde640735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:54.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.698+0000 7fde63fff700 1 -- 172.21.15.67:0/774915954 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fde64101850 msgr2=0x7fde640735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:54.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.698+0000 7fde63fff700 1 --2- 172.21.15.67:0/774915954 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fde64101850 0x7fde640735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:54.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.699+0000 7fde63fff700 1 -- 172.21.15.67:0/774915954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde540095f0 con 0x7fde64100ea0 2024-04-23T00:00:54.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.699+0000 7fde63fff700 1 --2- 172.21.15.67:0/774915954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde64100ea0 0x7fde64073090 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fde54006010 tx=0x7fde5400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:00:54.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.699+0000 7fde617fa700 1 -- 172.21.15.67:0/774915954 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde5401d070 con 0x7fde64100ea0 2024-04-23T00:00:54.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.699+0000 7fde6a54d700 1 -- 172.21.15.67:0/774915954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde64073ed0 con 0x7fde64100ea0 2024-04-23T00:00:54.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.699+0000 7fde617fa700 1 -- 172.21.15.67:0/774915954 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fde5400f460 con 0x7fde64100ea0 2024-04-23T00:00:54.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.699+0000 7fde617fa700 1 -- 172.21.15.67:0/774915954 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde54017630 con 0x7fde64100ea0 2024-04-23T00:00:54.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.699+0000 7fde6a54d700 1 -- 172.21.15.67:0/774915954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde641a2e20 con 0x7fde64100ea0 2024-04-23T00:00:54.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.700+0000 7fde6a54d700 1 -- 172.21.15.67:0/774915954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde64109240 con 0x7fde64100ea0 2024-04-23T00:00:54.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.700+0000 7fde617fa700 1 -- 172.21.15.67:0/774915954 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fde5400f5d0 con 0x7fde64100ea0 2024-04-23T00:00:54.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.701+0000 7fde617fa700 1 --2- 172.21.15.67:0/774915954 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fde5006ecc0 0x7fde50071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:54.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.701+0000 7fde637fe700 1 --2- 172.21.15.67:0/774915954 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fde5006ecc0 0x7fde50071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:54.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.701+0000 7fde617fa700 1 -- 172.21.15.67:0/774915954 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fde54090510 con 0x7fde64100ea0 2024-04-23T00:00:54.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.701+0000 7fde637fe700 1 --2- 172.21.15.67:0/774915954 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fde5006ecc0 0x7fde50071180 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fde64074880 tx=0x7fde4c00a300 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:00:54.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.706+0000 7fde617fa700 1 -- 172.21.15.67:0/774915954 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fde54059150 con 0x7fde64100ea0 2024-04-23T00:00:54.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.961+0000 7fde6a54d700 1 -- 172.21.15.67:0/774915954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fde64074680 con 0x7fde64100ea0 2024-04-23T00:00:54.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.962+0000 7fde617fa700 1 -- 172.21.15.67:0/774915954 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1872 (secure 0 0 0) 0x7fde5405c810 con 0x7fde64100ea0 2024-04-23T00:00:54.952 INFO:teuthology.orchestra.run.smithi067.stdout:e33 2024-04-23T00:00:54.952 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-23T00:00:54.952 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-23T00:00:54.952 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-23T00:00:54.952 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:00:54.952 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-23T00:00:54.952 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-23T00:00:54.952 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-23T00:00:54.953 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-23T00:00:54.953 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-23T00:00:54.953 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-23T00:00:54.953 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-23T00:00:54.953 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-23T00:00:54.953 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-23T00:00:54.953 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-23T00:00:54.953 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-23T00:00:54.953 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-23T00:00:54.953 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-23T00:00:54.953 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-23T00:00:54.953 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-23T00:00:54.953 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-23T00:00:54.953 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-23T00:00:54.953 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-23T00:00:54.954 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-23T00:00:54.954 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-23T00:00:54.954 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-23T00:00:54.954 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-23T00:00:54.954 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-23T00:00:54.954 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-23T00:00:54.954 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-23T00:00:54.954 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-23T00:00:54.954 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-23T00:00:54.954 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-23T00:00:54.954 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-23T00:00:54.954 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:00:54.954 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:00:54.954 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-23T00:00:54.954 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:00:54.955 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T00:00:54.955 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T00:00:54.955 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T00:00:54.955 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44851} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1849648622,v1:172.21.15.134:6827/1849648622] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T00:00:54.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.965+0000 7fde6a54d700 1 -- 172.21.15.67:0/774915954 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fde5006ecc0 msgr2=0x7fde50071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:54.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.965+0000 7fde6a54d700 1 --2- 172.21.15.67:0/774915954 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fde5006ecc0 0x7fde50071180 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fde64074880 tx=0x7fde4c00a300 comp rx=0 tx=0).stop 2024-04-23T00:00:54.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.965+0000 7fde6a54d700 1 -- 172.21.15.67:0/774915954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde64100ea0 msgr2=0x7fde64073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:54.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.965+0000 7fde6a54d700 1 --2- 172.21.15.67:0/774915954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde64100ea0 0x7fde64073090 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fde54006010 tx=0x7fde5400bd70 comp rx=0 tx=0).stop 2024-04-23T00:00:54.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.965+0000 7fde6a54d700 1 -- 172.21.15.67:0/774915954 shutdown_connections 2024-04-23T00:00:54.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.965+0000 7fde6a54d700 1 --2- 172.21.15.67:0/774915954 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fde5006ecc0 0x7fde50071180 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:54.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.965+0000 7fde6a54d700 1 --2- 172.21.15.67:0/774915954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde64100ea0 0x7fde64073090 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:54.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.965+0000 7fde6a54d700 1 --2- 172.21.15.67:0/774915954 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fde64101850 0x7fde640735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:54.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.965+0000 7fde6a54d700 1 -- 172.21.15.67:0/774915954 >> 172.21.15.67:0/774915954 conn(0x7fde640fc750 msgr2=0x7fde640fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:00:54.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.965+0000 7fde6a54d700 1 -- 172.21.15.67:0/774915954 shutdown_connections 2024-04-23T00:00:54.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:54.965+0000 7fde6a54d700 1 -- 172.21.15.67:0/774915954 wait complete. 2024-04-23T00:00:54.956 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 33 2024-04-23T00:00:55.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.075+0000 7fbdd3c56700 1 -- 172.21.15.67:0/2043612160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdcc101850 msgr2=0x7fbdcc1058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:55.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.075+0000 7fbdd3c56700 1 --2- 172.21.15.67:0/2043612160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdcc101850 0x7fbdcc1058a0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8009910 tx=0x7fbdc8009c20 comp rx=0 tx=0).stop 2024-04-23T00:00:55.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.075+0000 7fbdd3c56700 1 -- 172.21.15.67:0/2043612160 shutdown_connections 2024-04-23T00:00:55.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.075+0000 7fbdd3c56700 1 --2- 172.21.15.67:0/2043612160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdcc101850 0x7fbdcc1058a0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:55.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.075+0000 7fbdd3c56700 1 --2- 172.21.15.67:0/2043612160 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbdcc100ea0 0x7fbdcc101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:55.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.075+0000 7fbdd3c56700 1 -- 172.21.15.67:0/2043612160 >> 172.21.15.67:0/2043612160 conn(0x7fbdcc0fc750 msgr2=0x7fbdcc0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:00:55.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.075+0000 7fbdd3c56700 1 -- 172.21.15.67:0/2043612160 shutdown_connections 2024-04-23T00:00:55.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.075+0000 7fbdd3c56700 1 -- 172.21.15.67:0/2043612160 wait complete. 2024-04-23T00:00:55.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.075+0000 7fbdd3c56700 1 Processor -- start 2024-04-23T00:00:55.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.075+0000 7fbdd3c56700 1 -- start start 2024-04-23T00:00:55.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.076+0000 7fbdd3c56700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbdcc100ea0 0x7fbdcc073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:55.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.076+0000 7fbdd3c56700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdcc101850 0x7fbdcc0735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:55.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.076+0000 7fbdd3c56700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdcc073b10 con 0x7fbdcc101850 2024-04-23T00:00:55.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.076+0000 7fbdd3c56700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdcc073c50 con 0x7fbdcc100ea0 2024-04-23T00:00:55.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.076+0000 7fbdd11f1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdcc101850 0x7fbdcc0735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:55.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.076+0000 7fbdd11f1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdcc101850 0x7fbdcc0735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35640/0 (socket says 172.21.15.67:35640) 2024-04-23T00:00:55.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.076+0000 7fbdd11f1700 1 -- 172.21.15.67:0/712259833 learned_addr learned my addr 172.21.15.67:0/712259833 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:00:55.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.076+0000 7fbdd19f2700 1 --2- 172.21.15.67:0/712259833 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbdcc100ea0 0x7fbdcc073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:55.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.076+0000 7fbdd11f1700 1 -- 172.21.15.67:0/712259833 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbdcc100ea0 msgr2=0x7fbdcc073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:55.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.076+0000 7fbdd11f1700 1 --2- 172.21.15.67:0/712259833 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbdcc100ea0 0x7fbdcc073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:55.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.076+0000 7fbdd11f1700 1 -- 172.21.15.67:0/712259833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc80095f0 con 0x7fbdcc101850 2024-04-23T00:00:55.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.076+0000 7fbdd11f1700 1 --2- 172.21.15.67:0/712259833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdcc101850 0x7fbdcc0735d0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fbdc800bd70 tx=0x7fbdc800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:00:55.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.076+0000 7fbdbeffd700 1 -- 172.21.15.67:0/712259833 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbdc801d070 con 0x7fbdcc101850 2024-04-23T00:00:55.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.076+0000 7fbdbeffd700 1 -- 172.21.15.67:0/712259833 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbdc8005dc0 con 0x7fbdcc101850 2024-04-23T00:00:55.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.076+0000 7fbdd3c56700 1 -- 172.21.15.67:0/712259833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdcc073ed0 con 0x7fbdcc101850 2024-04-23T00:00:55.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.076+0000 7fbdbeffd700 1 -- 172.21.15.67:0/712259833 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbdc8017400 con 0x7fbdcc101850 2024-04-23T00:00:55.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.077+0000 7fbdd3c56700 1 -- 172.21.15.67:0/712259833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdcc1a2e30 con 0x7fbdcc101850 2024-04-23T00:00:55.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.078+0000 7fbdd3c56700 1 -- 172.21.15.67:0/712259833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdcc1092a0 con 0x7fbdcc101850 2024-04-23T00:00:55.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.078+0000 7fbdbeffd700 1 -- 172.21.15.67:0/712259833 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbdc800f460 con 0x7fbdcc101850 2024-04-23T00:00:55.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.079+0000 7fbdbeffd700 1 --2- 172.21.15.67:0/712259833 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbdb806ecc0 0x7fbdb8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:55.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.079+0000 7fbdbeffd700 1 -- 172.21.15.67:0/712259833 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fbdc8090440 con 0x7fbdcc101850 2024-04-23T00:00:55.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.079+0000 7fbdd19f2700 1 --2- 172.21.15.67:0/712259833 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbdb806ecc0 0x7fbdb8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:55.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.079+0000 7fbdd19f2700 1 --2- 172.21.15.67:0/712259833 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbdb806ecc0 0x7fbdb8071180 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fbdc0005950 tx=0x7fbdc000b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:00:55.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.083+0000 7fbdbeffd700 1 -- 172.21.15.67:0/712259833 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fbdc8059080 con 0x7fbdcc101850 2024-04-23T00:00:55.284 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:55 smithi067 ceph-mon[112438]: from='client.35868 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:00:55.284 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:55 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1369558805' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:00:55.284 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:55 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/774915954' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-23T00:00:55.284 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:55 smithi067 ceph-mon[112438]: pgmap v1038: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:55.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.315+0000 7fbdd3c56700 1 -- 172.21.15.67:0/712259833 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbdcc0746d0 con 0x7fbdb806ecc0 2024-04-23T00:00:55.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.317+0000 7fbdbeffd700 1 -- 172.21.15.67:0/712259833 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fbdcc0746d0 con 0x7fbdb806ecc0 2024-04-23T00:00:55.307 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-23T00:00:55.307 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-23T00:00:55.307 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-23T00:00:55.307 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-23T00:00:55.307 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-23T00:00:55.307 INFO:teuthology.orchestra.run.smithi067.stdout: "mds", 2024-04-23T00:00:55.307 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-23T00:00:55.307 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-23T00:00:55.308 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-23T00:00:55.308 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-23T00:00:55.308 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-23T00:00:55.308 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "16/23 daemons upgraded", 2024-04-23T00:00:55.308 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-23T00:00:55.308 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-23T00:00:55.308 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-23T00:00:55.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.319+0000 7fbdd3c56700 1 -- 172.21.15.67:0/712259833 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbdb806ecc0 msgr2=0x7fbdb8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:55.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.319+0000 7fbdd3c56700 1 --2- 172.21.15.67:0/712259833 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbdb806ecc0 0x7fbdb8071180 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fbdc0005950 tx=0x7fbdc000b410 comp rx=0 tx=0).stop 2024-04-23T00:00:55.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.319+0000 7fbdd3c56700 1 -- 172.21.15.67:0/712259833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdcc101850 msgr2=0x7fbdcc0735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:55.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.319+0000 7fbdd3c56700 1 --2- 172.21.15.67:0/712259833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdcc101850 0x7fbdcc0735d0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fbdc800bd70 tx=0x7fbdc800bda0 comp rx=0 tx=0).stop 2024-04-23T00:00:55.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.319+0000 7fbdd3c56700 1 -- 172.21.15.67:0/712259833 shutdown_connections 2024-04-23T00:00:55.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.319+0000 7fbdd3c56700 1 --2- 172.21.15.67:0/712259833 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbdb806ecc0 0x7fbdb8071180 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:55.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.319+0000 7fbdd3c56700 1 --2- 172.21.15.67:0/712259833 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbdcc100ea0 0x7fbdcc073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:55.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.319+0000 7fbdd3c56700 1 --2- 172.21.15.67:0/712259833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdcc101850 0x7fbdcc0735d0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:55.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.319+0000 7fbdd3c56700 1 -- 172.21.15.67:0/712259833 >> 172.21.15.67:0/712259833 conn(0x7fbdcc0fc750 msgr2=0x7fbdcc0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:00:55.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.319+0000 7fbdd3c56700 1 -- 172.21.15.67:0/712259833 shutdown_connections 2024-04-23T00:00:55.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.319+0000 7fbdd3c56700 1 -- 172.21.15.67:0/712259833 wait complete. 2024-04-23T00:00:55.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.426+0000 7f13bdbc7700 1 -- 172.21.15.67:0/142659793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13b8105bf0 msgr2=0x7f13b8105fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:55.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.426+0000 7f13bdbc7700 1 --2- 172.21.15.67:0/142659793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13b8105bf0 0x7f13b8105fd0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f13a8009910 tx=0x7f13a8009c20 comp rx=0 tx=0).stop 2024-04-23T00:00:55.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.426+0000 7f13bdbc7700 1 -- 172.21.15.67:0/142659793 shutdown_connections 2024-04-23T00:00:55.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.426+0000 7f13bdbc7700 1 --2- 172.21.15.67:0/142659793 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13b80ffb40 0x7f13b80fffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:55.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.426+0000 7f13bdbc7700 1 --2- 172.21.15.67:0/142659793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13b8105bf0 0x7f13b8105fd0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:55.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.426+0000 7f13bdbc7700 1 -- 172.21.15.67:0/142659793 >> 172.21.15.67:0/142659793 conn(0x7f13b80fb610 msgr2=0x7f13b80fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:00:55.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.426+0000 7f13bdbc7700 1 -- 172.21.15.67:0/142659793 shutdown_connections 2024-04-23T00:00:55.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.426+0000 7f13bdbc7700 1 -- 172.21.15.67:0/142659793 wait complete. 2024-04-23T00:00:55.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.426+0000 7f13bdbc7700 1 Processor -- start 2024-04-23T00:00:55.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.426+0000 7f13bdbc7700 1 -- start start 2024-04-23T00:00:55.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.427+0000 7f13bdbc7700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13b80ffb40 0x7f13b81977b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:55.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.427+0000 7f13bdbc7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13b8105bf0 0x7f13b8197cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:55.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.427+0000 7f13bdbc7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b8198310 con 0x7f13b8105bf0 2024-04-23T00:00:55.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.427+0000 7f13bdbc7700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b8198450 con 0x7f13b80ffb40 2024-04-23T00:00:55.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.427+0000 7f13b6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13b8105bf0 0x7f13b8197cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:55.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.427+0000 7f13b6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13b8105bf0 0x7f13b8197cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35648/0 (socket says 172.21.15.67:35648) 2024-04-23T00:00:55.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.427+0000 7f13b6ffd700 1 -- 172.21.15.67:0/2358874260 learned_addr learned my addr 172.21.15.67:0/2358874260 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:00:55.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.427+0000 7f13b77fe700 1 --2- 172.21.15.67:0/2358874260 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13b80ffb40 0x7f13b81977b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:55.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.427+0000 7f13b6ffd700 1 -- 172.21.15.67:0/2358874260 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13b80ffb40 msgr2=0x7f13b81977b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:55.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.427+0000 7f13b6ffd700 1 --2- 172.21.15.67:0/2358874260 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13b80ffb40 0x7f13b81977b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:55.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.427+0000 7f13b6ffd700 1 -- 172.21.15.67:0/2358874260 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a80095f0 con 0x7f13b8105bf0 2024-04-23T00:00:55.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.427+0000 7f13b6ffd700 1 --2- 172.21.15.67:0/2358874260 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13b8105bf0 0x7f13b8197cf0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f13ac00b4b0 tx=0x7f13ac00b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:00:55.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.427+0000 7f13b4ff9700 1 -- 172.21.15.67:0/2358874260 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13ac011800 con 0x7f13b8105bf0 2024-04-23T00:00:55.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.428+0000 7f13b4ff9700 1 -- 172.21.15.67:0/2358874260 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f13ac011e20 con 0x7f13b8105bf0 2024-04-23T00:00:55.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.428+0000 7f13bdbc7700 1 -- 172.21.15.67:0/2358874260 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13b81a0820 con 0x7f13b8105bf0 2024-04-23T00:00:55.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.428+0000 7f13b4ff9700 1 -- 172.21.15.67:0/2358874260 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13ac00f2e0 con 0x7f13b8105bf0 2024-04-23T00:00:55.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.428+0000 7f13bdbc7700 1 -- 172.21.15.67:0/2358874260 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13b81a0ce0 con 0x7f13b8105bf0 2024-04-23T00:00:55.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.429+0000 7f13bdbc7700 1 -- 172.21.15.67:0/2358874260 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13b8108020 con 0x7f13b8105bf0 2024-04-23T00:00:55.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.429+0000 7f13b4ff9700 1 -- 172.21.15.67:0/2358874260 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f13ac010380 con 0x7f13b8105bf0 2024-04-23T00:00:55.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.430+0000 7f13b4ff9700 1 --2- 172.21.15.67:0/2358874260 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f13a006ecc0 0x7f13a0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:00:55.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.430+0000 7f13b4ff9700 1 -- 172.21.15.67:0/2358874260 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f13ac08eb10 con 0x7f13b8105bf0 2024-04-23T00:00:55.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.430+0000 7f13b77fe700 1 --2- 172.21.15.67:0/2358874260 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f13a006ecc0 0x7f13a0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:00:55.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.430+0000 7f13b77fe700 1 --2- 172.21.15.67:0/2358874260 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f13a006ecc0 0x7f13a0071180 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f13a8006010 tx=0x7f13a801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:00:55.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.435+0000 7f13b4ff9700 1 -- 172.21.15.67:0/2358874260 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f13ac057750 con 0x7f13b8105bf0 2024-04-23T00:00:55.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:55 smithi134 ceph-mon[85630]: from='client.35868 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:00:55.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:55 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1369558805' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:00:55.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:55 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/774915954' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-23T00:00:55.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:55 smithi134 ceph-mon[85630]: pgmap v1038: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:55.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.734+0000 7f13bdbc7700 1 -- 172.21.15.67:0/2358874260 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f13b8107a00 con 0x7f13b8105bf0 2024-04-23T00:00:55.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.735+0000 7f13b4ff9700 1 -- 172.21.15.67:0/2358874260 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4505 (secure 0 0 0) 0x7f13ac05ae10 con 0x7f13b8105bf0 2024-04-23T00:00:55.725 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-23T00:00:55.725 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-23T00:00:55.725 INFO:teuthology.orchestra.run.smithi067.stdout: host smithi134 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-23T00:00:55.725 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-23T00:00:55.725 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-23T00:00:55.726 INFO:teuthology.orchestra.run.smithi067.stdout:Traceback (most recent call last): 2024-04-23T00:00:55.726 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-23T00:00:55.726 INFO:teuthology.orchestra.run.smithi067.stdout: return _run_code(code, main_globals, None, 2024-04-23T00:00:55.726 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-23T00:00:55.726 INFO:teuthology.orchestra.run.smithi067.stdout: exec(code, run_globals) 2024-04-23T00:00:55.726 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5642, in 2024-04-23T00:00:55.726 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5630, in main 2024-04-23T00:00:55.726 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 408, in _infer_config 2024-04-23T00:00:55.726 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 323, in _infer_fsid 2024-04-23T00:00:55.726 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 436, in _infer_image 2024-04-23T00:00:55.726 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 310, in _validate_fsid 2024-04-23T00:00:55.726 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 3380, in command_ceph_volume 2024-04-23T00:00:55.726 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-23T00:00:55.726 INFO:teuthology.orchestra.run.smithi067.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-23T00:00:55.726 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-23T00:00:55.727 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-23T00:00:55.727 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-23T00:00:55.727 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-23T00:00:55.727 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-23T00:00:55.727 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-23T00:00:55.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.737+0000 7f13bdbc7700 1 -- 172.21.15.67:0/2358874260 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f13a006ecc0 msgr2=0x7f13a0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:55.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.737+0000 7f13bdbc7700 1 --2- 172.21.15.67:0/2358874260 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f13a006ecc0 0x7f13a0071180 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f13a8006010 tx=0x7f13a801a040 comp rx=0 tx=0).stop 2024-04-23T00:00:55.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.737+0000 7f13bdbc7700 1 -- 172.21.15.67:0/2358874260 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13b8105bf0 msgr2=0x7f13b8197cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:00:55.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.737+0000 7f13bdbc7700 1 --2- 172.21.15.67:0/2358874260 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13b8105bf0 0x7f13b8197cf0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f13ac00b4b0 tx=0x7f13ac00b7c0 comp rx=0 tx=0).stop 2024-04-23T00:00:55.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.737+0000 7f13bdbc7700 1 -- 172.21.15.67:0/2358874260 shutdown_connections 2024-04-23T00:00:55.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.737+0000 7f13bdbc7700 1 --2- 172.21.15.67:0/2358874260 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f13a006ecc0 0x7f13a0071180 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:55.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.737+0000 7f13bdbc7700 1 --2- 172.21.15.67:0/2358874260 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13b80ffb40 0x7f13b81977b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:55.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.737+0000 7f13bdbc7700 1 --2- 172.21.15.67:0/2358874260 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13b8105bf0 0x7f13b8197cf0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:00:55.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.737+0000 7f13bdbc7700 1 -- 172.21.15.67:0/2358874260 >> 172.21.15.67:0/2358874260 conn(0x7f13b80fb610 msgr2=0x7f13b80fd2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:00:55.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.738+0000 7f13bdbc7700 1 -- 172.21.15.67:0/2358874260 shutdown_connections 2024-04-23T00:00:55.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:00:55.738+0000 7f13bdbc7700 1 -- 172.21.15.67:0/2358874260 wait complete. 2024-04-23T00:00:56.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:56 smithi134 ceph-mon[85630]: from='client.35880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:00:56.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:56 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2358874260' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T00:00:56.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:00:56.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:56 smithi067 ceph-mon[112438]: from='client.35880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:00:56.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:56 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2358874260' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T00:00:56.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:00:57.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:57 smithi134 ceph-mon[85630]: pgmap v1039: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:57.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:57 smithi067 ceph-mon[112438]: pgmap v1039: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:59.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:00:59 smithi134 ceph-mon[85630]: pgmap v1040: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:00:59.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:00:59 smithi067 ceph-mon[112438]: pgmap v1040: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:01.467 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:01 smithi067 ceph-mon[112438]: pgmap v1041: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:01.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:01 smithi134 ceph-mon[85630]: pgmap v1041: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:03.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:03 smithi134 ceph-mon[85630]: pgmap v1042: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:03.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:03 smithi067 ceph-mon[112438]: pgmap v1042: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:05.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:05 smithi134 ceph-mon[85630]: pgmap v1043: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:05.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:05 smithi067 ceph-mon[112438]: pgmap v1043: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:07.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:07 smithi134 ceph-mon[85630]: pgmap v1044: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:07.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:07 smithi067 ceph-mon[112438]: pgmap v1044: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:09.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:09 smithi134 ceph-mon[85630]: pgmap v1045: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:09.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:09 smithi067 ceph-mon[112438]: pgmap v1045: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:11.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:01:11.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:01:12.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:12 smithi134 ceph-mon[85630]: pgmap v1046: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:12.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:12 smithi067 ceph-mon[112438]: pgmap v1046: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:13.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:13 smithi134 ceph-mon[85630]: pgmap v1047: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:13.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:13 smithi067 ceph-mon[112438]: pgmap v1047: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:15.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:15 smithi134 ceph-mon[85630]: pgmap v1048: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:15.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:15 smithi067 ceph-mon[112438]: pgmap v1048: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:17.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:17 smithi134 ceph-mon[85630]: pgmap v1049: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:17.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:17 smithi067 ceph-mon[112438]: pgmap v1049: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:19.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:19 smithi134 ceph-mon[85630]: pgmap v1050: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:19.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:19 smithi067 ceph-mon[112438]: pgmap v1050: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:21.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:21 smithi134 ceph-mon[85630]: pgmap v1051: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:21.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:21 smithi067 ceph-mon[112438]: pgmap v1051: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:23.983 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:23 smithi134 ceph-mon[85630]: pgmap v1052: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:24.033 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:23 smithi067 ceph-mon[112438]: pgmap v1052: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:25.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:25 smithi134 ceph-mon[85630]: pgmap v1053: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:25.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:25 smithi067 ceph-mon[112438]: pgmap v1053: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:25.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:25.858+0000 7ff82cf25700 1 -- 172.21.15.67:0/2399104611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff828100ea0 msgr2=0x7ff828101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:25.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:25.858+0000 7ff82cf25700 1 --2- 172.21.15.67:0/2399104611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff828100ea0 0x7ff828101280 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7ff818009910 tx=0x7ff818009c20 comp rx=0 tx=0).stop 2024-04-23T00:01:25.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:25.859+0000 7ff82cf25700 1 -- 172.21.15.67:0/2399104611 shutdown_connections 2024-04-23T00:01:25.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:25.859+0000 7ff82cf25700 1 --2- 172.21.15.67:0/2399104611 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff828101850 0x7ff8281058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:25.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:25.859+0000 7ff82cf25700 1 --2- 172.21.15.67:0/2399104611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff828100ea0 0x7ff828101280 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:25.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:25.859+0000 7ff82cf25700 1 -- 172.21.15.67:0/2399104611 >> 172.21.15.67:0/2399104611 conn(0x7ff8280fc750 msgr2=0x7ff8280feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:01:25.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:25.859+0000 7ff82cf25700 1 -- 172.21.15.67:0/2399104611 shutdown_connections 2024-04-23T00:01:25.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:25.859+0000 7ff82cf25700 1 -- 172.21.15.67:0/2399104611 wait complete. 2024-04-23T00:01:25.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:25.859+0000 7ff82cf25700 1 Processor -- start 2024-04-23T00:01:25.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:25.860+0000 7ff82cf25700 1 -- start start 2024-04-23T00:01:25.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:25.860+0000 7ff82cf25700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff828100ea0 0x7ff82819c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:01:25.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:25.860+0000 7ff82cf25700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff828101850 0x7ff82819cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:01:25.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:25.860+0000 7ff82cf25700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff82819d2c0 con 0x7ff828100ea0 2024-04-23T00:01:25.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:25.860+0000 7ff82cf25700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff828196770 con 0x7ff828101850 2024-04-23T00:01:25.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:25.860+0000 7ff82659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff828100ea0 0x7ff82819c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:01:25.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:25.860+0000 7ff82659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff828100ea0 0x7ff82819c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60528/0 (socket says 172.21.15.67:60528) 2024-04-23T00:01:25.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:25.860+0000 7ff82659c700 1 -- 172.21.15.67:0/4285884197 learned_addr learned my addr 172.21.15.67:0/4285884197 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:01:25.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:25.860+0000 7ff825d9b700 1 --2- 172.21.15.67:0/4285884197 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff828101850 0x7ff82819cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:01:25.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:25.860+0000 7ff82659c700 1 -- 172.21.15.67:0/4285884197 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff828101850 msgr2=0x7ff82819cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:25.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:25.860+0000 7ff82659c700 1 --2- 172.21.15.67:0/4285884197 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff828101850 0x7ff82819cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:25.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:25.860+0000 7ff82659c700 1 -- 172.21.15.67:0/4285884197 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8180095f0 con 0x7ff828100ea0 2024-04-23T00:01:25.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:25.861+0000 7ff82659c700 1 --2- 172.21.15.67:0/4285884197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff828100ea0 0x7ff82819c6f0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7ff818006010 tx=0x7ff81800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:01:25.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:25.861+0000 7ff81f7fe700 1 -- 172.21.15.67:0/4285884197 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff81801d070 con 0x7ff828100ea0 2024-04-23T00:01:25.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:25.861+0000 7ff81f7fe700 1 -- 172.21.15.67:0/4285884197 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff818005dc0 con 0x7ff828100ea0 2024-04-23T00:01:25.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:25.861+0000 7ff82cf25700 1 -- 172.21.15.67:0/4285884197 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8281969f0 con 0x7ff828100ea0 2024-04-23T00:01:25.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:25.861+0000 7ff81f7fe700 1 -- 172.21.15.67:0/4285884197 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff818017400 con 0x7ff828100ea0 2024-04-23T00:01:25.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:25.861+0000 7ff82cf25700 1 -- 172.21.15.67:0/4285884197 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff828196ee0 con 0x7ff828100ea0 2024-04-23T00:01:25.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:25.862+0000 7ff82cf25700 1 -- 172.21.15.67:0/4285884197 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff828109240 con 0x7ff828100ea0 2024-04-23T00:01:25.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:25.862+0000 7ff81f7fe700 1 -- 172.21.15.67:0/4285884197 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff818003a60 con 0x7ff828100ea0 2024-04-23T00:01:25.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:25.863+0000 7ff81f7fe700 1 --2- 172.21.15.67:0/4285884197 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff81406ecc0 0x7ff814071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:01:25.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:25.863+0000 7ff81f7fe700 1 -- 172.21.15.67:0/4285884197 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7ff818090460 con 0x7ff828100ea0 2024-04-23T00:01:25.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:25.863+0000 7ff825d9b700 1 --2- 172.21.15.67:0/4285884197 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff81406ecc0 0x7ff814071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:01:25.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:25.863+0000 7ff825d9b700 1 --2- 172.21.15.67:0/4285884197 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff81406ecc0 0x7ff814071180 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ff828197be0 tx=0x7ff81000b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:01:25.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:25.868+0000 7ff81f7fe700 1 -- 172.21.15.67:0/4285884197 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff81805cbd0 con 0x7ff828100ea0 2024-04-23T00:01:26.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.105+0000 7ff82cf25700 1 -- 172.21.15.67:0/4285884197 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff828197100 con 0x7ff81406ecc0 2024-04-23T00:01:26.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.106+0000 7ff81f7fe700 1 -- 172.21.15.67:0/4285884197 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff828197100 con 0x7ff81406ecc0 2024-04-23T00:01:26.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.108+0000 7ff82cf25700 1 -- 172.21.15.67:0/4285884197 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff81406ecc0 msgr2=0x7ff814071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:26.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.108+0000 7ff82cf25700 1 --2- 172.21.15.67:0/4285884197 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff81406ecc0 0x7ff814071180 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ff828197be0 tx=0x7ff81000b410 comp rx=0 tx=0).stop 2024-04-23T00:01:26.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.108+0000 7ff82cf25700 1 -- 172.21.15.67:0/4285884197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff828100ea0 msgr2=0x7ff82819c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:26.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.108+0000 7ff82cf25700 1 --2- 172.21.15.67:0/4285884197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff828100ea0 0x7ff82819c6f0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7ff818006010 tx=0x7ff81800bd70 comp rx=0 tx=0).stop 2024-04-23T00:01:26.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.108+0000 7ff82cf25700 1 -- 172.21.15.67:0/4285884197 shutdown_connections 2024-04-23T00:01:26.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.108+0000 7ff82cf25700 1 --2- 172.21.15.67:0/4285884197 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff81406ecc0 0x7ff814071180 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:26.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.108+0000 7ff82cf25700 1 --2- 172.21.15.67:0/4285884197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff828100ea0 0x7ff82819c6f0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:26.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.108+0000 7ff82cf25700 1 --2- 172.21.15.67:0/4285884197 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff828101850 0x7ff82819cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:26.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.108+0000 7ff82cf25700 1 -- 172.21.15.67:0/4285884197 >> 172.21.15.67:0/4285884197 conn(0x7ff8280fc750 msgr2=0x7ff8280fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:01:26.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.108+0000 7ff82cf25700 1 -- 172.21.15.67:0/4285884197 shutdown_connections 2024-04-23T00:01:26.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.108+0000 7ff82cf25700 1 -- 172.21.15.67:0/4285884197 wait complete. 2024-04-23T00:01:26.108 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-23T00:01:26.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.218+0000 7fa058adf700 1 -- 172.21.15.67:0/4023710082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0540ffda0 msgr2=0x7fa054100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:26.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.218+0000 7fa058adf700 1 --2- 172.21.15.67:0/4023710082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0540ffda0 0x7fa054100180 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fa044009910 tx=0x7fa044009c20 comp rx=0 tx=0).stop 2024-04-23T00:01:26.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.218+0000 7fa058adf700 1 -- 172.21.15.67:0/4023710082 shutdown_connections 2024-04-23T00:01:26.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.218+0000 7fa058adf700 1 --2- 172.21.15.67:0/4023710082 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa054100750 0x7fa0541047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:26.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.218+0000 7fa058adf700 1 --2- 172.21.15.67:0/4023710082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0540ffda0 0x7fa054100180 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:26.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.218+0000 7fa058adf700 1 -- 172.21.15.67:0/4023710082 >> 172.21.15.67:0/4023710082 conn(0x7fa0540fb650 msgr2=0x7fa0540fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:01:26.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.219+0000 7fa058adf700 1 -- 172.21.15.67:0/4023710082 shutdown_connections 2024-04-23T00:01:26.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.219+0000 7fa058adf700 1 -- 172.21.15.67:0/4023710082 wait complete. 2024-04-23T00:01:26.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.219+0000 7fa058adf700 1 Processor -- start 2024-04-23T00:01:26.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.219+0000 7fa058adf700 1 -- start start 2024-04-23T00:01:26.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.220+0000 7fa058adf700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0540ffda0 0x7fa05419b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:01:26.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.220+0000 7fa058adf700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa054100750 0x7fa05419ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:01:26.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.220+0000 7fa058adf700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05419c120 con 0x7fa0540ffda0 2024-04-23T00:01:26.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.220+0000 7fa058adf700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0541955d0 con 0x7fa054100750 2024-04-23T00:01:26.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.220+0000 7fa05259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0540ffda0 0x7fa05419b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:01:26.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.220+0000 7fa05259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0540ffda0 0x7fa05419b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60542/0 (socket says 172.21.15.67:60542) 2024-04-23T00:01:26.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.220+0000 7fa05259c700 1 -- 172.21.15.67:0/3949927696 learned_addr learned my addr 172.21.15.67:0/3949927696 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:01:26.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.220+0000 7fa051d9b700 1 --2- 172.21.15.67:0/3949927696 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa054100750 0x7fa05419ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:01:26.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.220+0000 7fa05259c700 1 -- 172.21.15.67:0/3949927696 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa054100750 msgr2=0x7fa05419ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:26.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.220+0000 7fa05259c700 1 --2- 172.21.15.67:0/3949927696 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa054100750 0x7fa05419ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:26.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.220+0000 7fa05259c700 1 -- 172.21.15.67:0/3949927696 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0440095f0 con 0x7fa0540ffda0 2024-04-23T00:01:26.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.220+0000 7fa05259c700 1 --2- 172.21.15.67:0/3949927696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0540ffda0 0x7fa05419b550 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fa04400f690 tx=0x7fa04400f6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:01:26.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.220+0000 7fa0437fe700 1 -- 172.21.15.67:0/3949927696 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa04401d070 con 0x7fa0540ffda0 2024-04-23T00:01:26.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.221+0000 7fa0437fe700 1 -- 172.21.15.67:0/3949927696 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa0440050d0 con 0x7fa0540ffda0 2024-04-23T00:01:26.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.221+0000 7fa058adf700 1 -- 172.21.15.67:0/3949927696 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa054195850 con 0x7fa0540ffda0 2024-04-23T00:01:26.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.221+0000 7fa0437fe700 1 -- 172.21.15.67:0/3949927696 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa044022470 con 0x7fa0540ffda0 2024-04-23T00:01:26.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.221+0000 7fa058adf700 1 -- 172.21.15.67:0/3949927696 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa054195d40 con 0x7fa0540ffda0 2024-04-23T00:01:26.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.222+0000 7fa058adf700 1 -- 172.21.15.67:0/3949927696 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa054108140 con 0x7fa0540ffda0 2024-04-23T00:01:26.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.222+0000 7fa0437fe700 1 -- 172.21.15.67:0/3949927696 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa04400fb20 con 0x7fa0540ffda0 2024-04-23T00:01:26.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.223+0000 7fa0437fe700 1 --2- 172.21.15.67:0/3949927696 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa03c06ecc0 0x7fa03c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:01:26.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.223+0000 7fa051d9b700 1 --2- 172.21.15.67:0/3949927696 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa03c06ecc0 0x7fa03c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:01:26.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.223+0000 7fa0437fe700 1 -- 172.21.15.67:0/3949927696 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fa044090420 con 0x7fa0540ffda0 2024-04-23T00:01:26.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.223+0000 7fa051d9b700 1 --2- 172.21.15.67:0/3949927696 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa03c06ecc0 0x7fa03c071180 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fa054104a20 tx=0x7fa048009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:01:26.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.227+0000 7fa0437fe700 1 -- 172.21.15.67:0/3949927696 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fa04405cb90 con 0x7fa0540ffda0 2024-04-23T00:01:26.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.464+0000 7fa058adf700 1 -- 172.21.15.67:0/3949927696 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa054195f60 con 0x7fa03c06ecc0 2024-04-23T00:01:26.454 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:01:26.454 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:26 smithi067 ceph-mon[112438]: from='client.35888 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:01:26.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.465+0000 7fa0437fe700 1 -- 172.21.15.67:0/3949927696 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa054195f60 con 0x7fa03c06ecc0 2024-04-23T00:01:26.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.467+0000 7fa058adf700 1 -- 172.21.15.67:0/3949927696 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa03c06ecc0 msgr2=0x7fa03c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:26.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.467+0000 7fa058adf700 1 --2- 172.21.15.67:0/3949927696 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa03c06ecc0 0x7fa03c071180 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fa054104a20 tx=0x7fa048009260 comp rx=0 tx=0).stop 2024-04-23T00:01:26.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.467+0000 7fa058adf700 1 -- 172.21.15.67:0/3949927696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0540ffda0 msgr2=0x7fa05419b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:26.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.467+0000 7fa058adf700 1 --2- 172.21.15.67:0/3949927696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0540ffda0 0x7fa05419b550 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fa04400f690 tx=0x7fa04400f6c0 comp rx=0 tx=0).stop 2024-04-23T00:01:26.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.467+0000 7fa058adf700 1 -- 172.21.15.67:0/3949927696 shutdown_connections 2024-04-23T00:01:26.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.467+0000 7fa058adf700 1 --2- 172.21.15.67:0/3949927696 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa03c06ecc0 0x7fa03c071180 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:26.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.467+0000 7fa058adf700 1 --2- 172.21.15.67:0/3949927696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0540ffda0 0x7fa05419b550 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:26.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.467+0000 7fa058adf700 1 --2- 172.21.15.67:0/3949927696 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa054100750 0x7fa05419ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:26.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.467+0000 7fa058adf700 1 -- 172.21.15.67:0/3949927696 >> 172.21.15.67:0/3949927696 conn(0x7fa0540fb650 msgr2=0x7fa0540fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:01:26.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.468+0000 7fa058adf700 1 -- 172.21.15.67:0/3949927696 shutdown_connections 2024-04-23T00:01:26.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.468+0000 7fa058adf700 1 -- 172.21.15.67:0/3949927696 wait complete. 2024-04-23T00:01:26.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:01:26.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:26 smithi134 ceph-mon[85630]: from='client.35888 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:01:26.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.574+0000 7fc380697700 1 -- 172.21.15.67:0/708495537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc378100750 msgr2=0x7fc3781047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:26.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.574+0000 7fc380697700 1 --2- 172.21.15.67:0/708495537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc378100750 0x7fc3781047a0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fc374009910 tx=0x7fc374009c20 comp rx=0 tx=0).stop 2024-04-23T00:01:26.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.574+0000 7fc380697700 1 -- 172.21.15.67:0/708495537 shutdown_connections 2024-04-23T00:01:26.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.574+0000 7fc380697700 1 --2- 172.21.15.67:0/708495537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc378100750 0x7fc3781047a0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:26.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.574+0000 7fc380697700 1 --2- 172.21.15.67:0/708495537 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc3780ffda0 0x7fc378100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:26.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.574+0000 7fc380697700 1 -- 172.21.15.67:0/708495537 >> 172.21.15.67:0/708495537 conn(0x7fc3780fb650 msgr2=0x7fc3780fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:01:26.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.574+0000 7fc380697700 1 -- 172.21.15.67:0/708495537 shutdown_connections 2024-04-23T00:01:26.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.574+0000 7fc380697700 1 -- 172.21.15.67:0/708495537 wait complete. 2024-04-23T00:01:26.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.575+0000 7fc380697700 1 Processor -- start 2024-04-23T00:01:26.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.575+0000 7fc380697700 1 -- start start 2024-04-23T00:01:26.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.575+0000 7fc380697700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3780ffda0 0x7fc37819b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:01:26.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.575+0000 7fc380697700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc378100750 0x7fc37819bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:01:26.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.575+0000 7fc380697700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc37819c1f0 con 0x7fc3780ffda0 2024-04-23T00:01:26.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.575+0000 7fc380697700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3781956a0 con 0x7fc378100750 2024-04-23T00:01:26.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.575+0000 7fc37e433700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3780ffda0 0x7fc37819b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:01:26.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.575+0000 7fc37e433700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3780ffda0 0x7fc37819b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60558/0 (socket says 172.21.15.67:60558) 2024-04-23T00:01:26.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.575+0000 7fc37e433700 1 -- 172.21.15.67:0/1617104082 learned_addr learned my addr 172.21.15.67:0/1617104082 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:01:26.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.575+0000 7fc37dc32700 1 --2- 172.21.15.67:0/1617104082 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc378100750 0x7fc37819bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:01:26.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.575+0000 7fc37e433700 1 -- 172.21.15.67:0/1617104082 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc378100750 msgr2=0x7fc37819bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:26.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.575+0000 7fc37e433700 1 --2- 172.21.15.67:0/1617104082 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc378100750 0x7fc37819bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:26.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.575+0000 7fc37e433700 1 -- 172.21.15.67:0/1617104082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3740095f0 con 0x7fc3780ffda0 2024-04-23T00:01:26.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.576+0000 7fc37e433700 1 --2- 172.21.15.67:0/1617104082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3780ffda0 0x7fc37819b620 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fc368009970 tx=0x7fc368009d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:01:26.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.576+0000 7fc36f7fe700 1 -- 172.21.15.67:0/1617104082 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc368012070 con 0x7fc3780ffda0 2024-04-23T00:01:26.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.576+0000 7fc36f7fe700 1 -- 172.21.15.67:0/1617104082 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc368007af0 con 0x7fc3780ffda0 2024-04-23T00:01:26.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.576+0000 7fc380697700 1 -- 172.21.15.67:0/1617104082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc378195920 con 0x7fc3780ffda0 2024-04-23T00:01:26.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.576+0000 7fc36f7fe700 1 -- 172.21.15.67:0/1617104082 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc368017c60 con 0x7fc3780ffda0 2024-04-23T00:01:26.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.576+0000 7fc380697700 1 -- 172.21.15.67:0/1617104082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc378195e70 con 0x7fc3780ffda0 2024-04-23T00:01:26.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.577+0000 7fc380697700 1 -- 172.21.15.67:0/1617104082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc37804f2e0 con 0x7fc3780ffda0 2024-04-23T00:01:26.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.577+0000 7fc36f7fe700 1 -- 172.21.15.67:0/1617104082 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc368007c60 con 0x7fc3780ffda0 2024-04-23T00:01:26.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.578+0000 7fc36f7fe700 1 --2- 172.21.15.67:0/1617104082 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc36406ecc0 0x7fc364071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:01:26.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.578+0000 7fc36f7fe700 1 -- 172.21.15.67:0/1617104082 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fc3680904e0 con 0x7fc3780ffda0 2024-04-23T00:01:26.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.578+0000 7fc37dc32700 1 --2- 172.21.15.67:0/1617104082 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc36406ecc0 0x7fc364071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:01:26.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.579+0000 7fc37dc32700 1 --2- 172.21.15.67:0/1617104082 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc36406ecc0 0x7fc364071180 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fc374006010 tx=0x7fc37400b540 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:01:26.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.583+0000 7fc36f7fe700 1 -- 172.21.15.67:0/1617104082 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc36805cd80 con 0x7fc3780ffda0 2024-04-23T00:01:26.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.800+0000 7fc380697700 1 -- 172.21.15.67:0/1617104082 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc3781962b0 con 0x7fc36406ecc0 2024-04-23T00:01:26.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.811+0000 7fc36f7fe700 1 -- 172.21.15.67:0/1617104082 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fc3781962b0 con 0x7fc36406ecc0 2024-04-23T00:01:26.800 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-23T00:01:26.801 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (35m) 12m ago 46m 24.4M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-23T00:01:26.801 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (46m) 12m ago 46m 10.0M - 18.2.1 5be31c24972a df887f62291e 2024-04-23T00:01:26.801 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (45m) 12m ago 45m 10.0M - 18.2.1 5be31c24972a ad89190b5400 2024-04-23T00:01:26.801 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (32m) 12m ago 46m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-23T00:01:26.801 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (32m) 12m ago 45m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-23T00:01:26.801 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (34m) 12m ago 46m 81.2M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-23T00:01:26.801 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (28m) 12m ago 40m 19.4M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-23T00:01:26.801 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (28m) 12m ago 40m 20.3M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-23T00:01:26.801 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (28m) 12m ago 40m 20.1M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-23T00:01:26.801 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (13m) 12m ago 40m 14.0M - 19.0.0-3252-g43be0201 a0434699c1d7 46d3d7e86fee 2024-04-23T00:01:26.801 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (36m) 12m ago 47m 514M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-23T00:01:26.801 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (36m) 12m ago 45m 459M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-23T00:01:26.802 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (33m) 12m ago 48m 74.2M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-23T00:01:26.802 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (33m) 12m ago 45m 54.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-23T00:01:26.802 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (35m) 12m ago 46m 15.2M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-23T00:01:26.802 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (35m) 12m ago 45m 16.0M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-23T00:01:26.802 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (32m) 12m ago 44m 102M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-23T00:01:26.802 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (31m) 12m ago 44m 128M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-23T00:01:26.802 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (30m) 12m ago 43m 83.8M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-23T00:01:26.802 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (30m) 12m ago 43m 101M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-23T00:01:26.802 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (29m) 12m ago 42m 86.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-23T00:01:26.802 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (29m) 12m ago 42m 81.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-23T00:01:26.803 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (35m) 12m ago 45m 78.7M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-23T00:01:26.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.813+0000 7fc380697700 1 -- 172.21.15.67:0/1617104082 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc36406ecc0 msgr2=0x7fc364071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:26.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.813+0000 7fc380697700 1 --2- 172.21.15.67:0/1617104082 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc36406ecc0 0x7fc364071180 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fc374006010 tx=0x7fc37400b540 comp rx=0 tx=0).stop 2024-04-23T00:01:26.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.813+0000 7fc380697700 1 -- 172.21.15.67:0/1617104082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3780ffda0 msgr2=0x7fc37819b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:26.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.813+0000 7fc380697700 1 --2- 172.21.15.67:0/1617104082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3780ffda0 0x7fc37819b620 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fc368009970 tx=0x7fc368009d30 comp rx=0 tx=0).stop 2024-04-23T00:01:26.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.813+0000 7fc380697700 1 -- 172.21.15.67:0/1617104082 shutdown_connections 2024-04-23T00:01:26.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.813+0000 7fc380697700 1 --2- 172.21.15.67:0/1617104082 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc36406ecc0 0x7fc364071180 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:26.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.813+0000 7fc380697700 1 --2- 172.21.15.67:0/1617104082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3780ffda0 0x7fc37819b620 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:26.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.813+0000 7fc380697700 1 --2- 172.21.15.67:0/1617104082 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc378100750 0x7fc37819bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:26.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.813+0000 7fc380697700 1 -- 172.21.15.67:0/1617104082 >> 172.21.15.67:0/1617104082 conn(0x7fc3780fb650 msgr2=0x7fc3780fccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:01:26.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.813+0000 7fc380697700 1 -- 172.21.15.67:0/1617104082 shutdown_connections 2024-04-23T00:01:26.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.813+0000 7fc380697700 1 -- 172.21.15.67:0/1617104082 wait complete. 2024-04-23T00:01:26.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.918+0000 7f1397537700 1 -- 172.21.15.67:0/810232868 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13900ffbf0 msgr2=0x7f13900fffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:26.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.918+0000 7f1397537700 1 --2- 172.21.15.67:0/810232868 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13900ffbf0 0x7f13900fffd0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f138c009910 tx=0x7f138c009c20 comp rx=0 tx=0).stop 2024-04-23T00:01:26.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.919+0000 7f1397537700 1 -- 172.21.15.67:0/810232868 shutdown_connections 2024-04-23T00:01:26.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.919+0000 7f1397537700 1 --2- 172.21.15.67:0/810232868 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1390100510 0x7f13901049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:26.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.919+0000 7f1397537700 1 --2- 172.21.15.67:0/810232868 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13900ffbf0 0x7f13900fffd0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:26.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.919+0000 7f1397537700 1 -- 172.21.15.67:0/810232868 >> 172.21.15.67:0/810232868 conn(0x7f13900fb650 msgr2=0x7f13900fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:01:26.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.919+0000 7f1397537700 1 -- 172.21.15.67:0/810232868 shutdown_connections 2024-04-23T00:01:26.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.919+0000 7f1397537700 1 -- 172.21.15.67:0/810232868 wait complete. 2024-04-23T00:01:26.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.919+0000 7f1397537700 1 Processor -- start 2024-04-23T00:01:26.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.919+0000 7f1397537700 1 -- start start 2024-04-23T00:01:26.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.919+0000 7f1397537700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13900ffbf0 0x7f139019cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:01:26.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.919+0000 7f1397537700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1390100510 0x7f139019d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:01:26.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.919+0000 7f1397537700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f139019daf0 con 0x7f1390100510 2024-04-23T00:01:26.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.919+0000 7f1397537700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f139019dc30 con 0x7f13900ffbf0 2024-04-23T00:01:26.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.920+0000 7f1394ad2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1390100510 0x7f139019d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:01:26.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.920+0000 7f1394ad2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1390100510 0x7f139019d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60582/0 (socket says 172.21.15.67:60582) 2024-04-23T00:01:26.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.920+0000 7f1394ad2700 1 -- 172.21.15.67:0/1122147191 learned_addr learned my addr 172.21.15.67:0/1122147191 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:01:26.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.920+0000 7f13952d3700 1 --2- 172.21.15.67:0/1122147191 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13900ffbf0 0x7f139019cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:01:26.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.920+0000 7f1394ad2700 1 -- 172.21.15.67:0/1122147191 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13900ffbf0 msgr2=0x7f139019cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:26.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.920+0000 7f1394ad2700 1 --2- 172.21.15.67:0/1122147191 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13900ffbf0 0x7f139019cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:26.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.920+0000 7f1394ad2700 1 -- 172.21.15.67:0/1122147191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f138c0095f0 con 0x7f1390100510 2024-04-23T00:01:26.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.920+0000 7f1394ad2700 1 --2- 172.21.15.67:0/1122147191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1390100510 0x7f139019d4a0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f138000c960 tx=0x7f138000cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:01:26.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.920+0000 7f13867fc700 1 -- 172.21.15.67:0/1122147191 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1380007a70 con 0x7f1390100510 2024-04-23T00:01:26.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.920+0000 7f13867fc700 1 -- 172.21.15.67:0/1122147191 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1380007bd0 con 0x7f1390100510 2024-04-23T00:01:26.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.920+0000 7f1397537700 1 -- 172.21.15.67:0/1122147191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13901a1a80 con 0x7f1390100510 2024-04-23T00:01:26.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.920+0000 7f13867fc700 1 -- 172.21.15.67:0/1122147191 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13800186c0 con 0x7f1390100510 2024-04-23T00:01:26.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.920+0000 7f1397537700 1 -- 172.21.15.67:0/1122147191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13901a1fd0 con 0x7f1390100510 2024-04-23T00:01:26.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.921+0000 7f1397537700 1 -- 172.21.15.67:0/1122147191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1390102090 con 0x7f1390100510 2024-04-23T00:01:26.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.922+0000 7f13867fc700 1 -- 172.21.15.67:0/1122147191 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f138000faa0 con 0x7f1390100510 2024-04-23T00:01:26.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.922+0000 7f13867fc700 1 --2- 172.21.15.67:0/1122147191 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f137c06ecc0 0x7f137c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:01:26.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.923+0000 7f13867fc700 1 -- 172.21.15.67:0/1122147191 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f138008efe0 con 0x7f1390100510 2024-04-23T00:01:26.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.923+0000 7f13952d3700 1 --2- 172.21.15.67:0/1122147191 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f137c06ecc0 0x7f137c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:01:26.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.923+0000 7f13952d3700 1 --2- 172.21.15.67:0/1122147191 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f137c06ecc0 0x7f137c071180 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f138c006010 tx=0x7f138c01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:01:26.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:26.927+0000 7f13867fc700 1 -- 172.21.15.67:0/1122147191 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f138005b750 con 0x7f1390100510 2024-04-23T00:01:27.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.229+0000 7f1397537700 1 -- 172.21.15.67:0/1122147191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f139004f2e0 con 0x7f1390100510 2024-04-23T00:01:27.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.229+0000 7f13867fc700 1 -- 172.21.15.67:0/1122147191 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f138005b2e0 con 0x7f1390100510 2024-04-23T00:01:27.219 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-23T00:01:27.219 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-23T00:01:27.219 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-23T00:01:27.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-23T00:01:27.220 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-23T00:01:27.220 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-23T00:01:27.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-23T00:01:27.220 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-23T00:01:27.220 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-23T00:01:27.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-23T00:01:27.220 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-23T00:01:27.220 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 4 2024-04-23T00:01:27.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-23T00:01:27.220 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-23T00:01:27.221 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 14 2024-04-23T00:01:27.221 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-23T00:01:27.221 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-23T00:01:27.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.231+0000 7f1397537700 1 -- 172.21.15.67:0/1122147191 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f137c06ecc0 msgr2=0x7f137c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:27.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.231+0000 7f1397537700 1 --2- 172.21.15.67:0/1122147191 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f137c06ecc0 0x7f137c071180 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f138c006010 tx=0x7f138c01a040 comp rx=0 tx=0).stop 2024-04-23T00:01:27.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.231+0000 7f1397537700 1 -- 172.21.15.67:0/1122147191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1390100510 msgr2=0x7f139019d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:27.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.231+0000 7f1397537700 1 --2- 172.21.15.67:0/1122147191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1390100510 0x7f139019d4a0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f138000c960 tx=0x7f138000cd20 comp rx=0 tx=0).stop 2024-04-23T00:01:27.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.231+0000 7f1397537700 1 -- 172.21.15.67:0/1122147191 shutdown_connections 2024-04-23T00:01:27.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.231+0000 7f1397537700 1 --2- 172.21.15.67:0/1122147191 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f137c06ecc0 0x7f137c071180 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:27.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.231+0000 7f1397537700 1 --2- 172.21.15.67:0/1122147191 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13900ffbf0 0x7f139019cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:27.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.231+0000 7f1397537700 1 --2- 172.21.15.67:0/1122147191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1390100510 0x7f139019d4a0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:27.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.231+0000 7f1397537700 1 -- 172.21.15.67:0/1122147191 >> 172.21.15.67:0/1122147191 conn(0x7f13900fb650 msgr2=0x7f1390111060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:01:27.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.232+0000 7f1397537700 1 -- 172.21.15.67:0/1122147191 shutdown_connections 2024-04-23T00:01:27.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.232+0000 7f1397537700 1 -- 172.21.15.67:0/1122147191 wait complete. 2024-04-23T00:01:27.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.339+0000 7fab0f9cf700 1 -- 172.21.15.67:0/3469596259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab08100ea0 msgr2=0x7fab08101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:27.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.339+0000 7fab0f9cf700 1 --2- 172.21.15.67:0/3469596259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab08100ea0 0x7fab08101280 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fab0000b0a0 tx=0x7fab0000b3b0 comp rx=0 tx=0).stop 2024-04-23T00:01:27.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.339+0000 7fab0f9cf700 1 -- 172.21.15.67:0/3469596259 shutdown_connections 2024-04-23T00:01:27.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.339+0000 7fab0f9cf700 1 --2- 172.21.15.67:0/3469596259 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fab08101850 0x7fab081058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:27.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.339+0000 7fab0f9cf700 1 --2- 172.21.15.67:0/3469596259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab08100ea0 0x7fab08101280 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:27.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.339+0000 7fab0f9cf700 1 -- 172.21.15.67:0/3469596259 >> 172.21.15.67:0/3469596259 conn(0x7fab080fc750 msgr2=0x7fab080feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:01:27.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.339+0000 7fab0f9cf700 1 -- 172.21.15.67:0/3469596259 shutdown_connections 2024-04-23T00:01:27.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.339+0000 7fab0f9cf700 1 -- 172.21.15.67:0/3469596259 wait complete. 2024-04-23T00:01:27.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.340+0000 7fab0f9cf700 1 Processor -- start 2024-04-23T00:01:27.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.340+0000 7fab0f9cf700 1 -- start start 2024-04-23T00:01:27.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.340+0000 7fab0f9cf700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fab08100ea0 0x7fab0819c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:01:27.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.340+0000 7fab0f9cf700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab08101850 0x7fab0819cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:01:27.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.340+0000 7fab0f9cf700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab0819d2c0 con 0x7fab08101850 2024-04-23T00:01:27.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.340+0000 7fab0f9cf700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab08196770 con 0x7fab08100ea0 2024-04-23T00:01:27.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.340+0000 7fab0cf6a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab08101850 0x7fab0819cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:01:27.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.340+0000 7fab0cf6a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab08101850 0x7fab0819cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60594/0 (socket says 172.21.15.67:60594) 2024-04-23T00:01:27.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.340+0000 7fab0cf6a700 1 -- 172.21.15.67:0/2316473008 learned_addr learned my addr 172.21.15.67:0/2316473008 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:01:27.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.340+0000 7fab0d76b700 1 --2- 172.21.15.67:0/2316473008 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fab08100ea0 0x7fab0819c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:01:27.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.340+0000 7fab0cf6a700 1 -- 172.21.15.67:0/2316473008 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fab08100ea0 msgr2=0x7fab0819c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:27.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.340+0000 7fab0cf6a700 1 --2- 172.21.15.67:0/2316473008 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fab08100ea0 0x7fab0819c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:27.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.340+0000 7fab0cf6a700 1 -- 172.21.15.67:0/2316473008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab00009d00 con 0x7fab08101850 2024-04-23T00:01:27.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.341+0000 7fab0cf6a700 1 --2- 172.21.15.67:0/2316473008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab08101850 0x7fab0819cc30 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7faafc00d900 tx=0x7faafc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:01:27.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.341+0000 7fab067fc700 1 -- 172.21.15.67:0/2316473008 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faafc009900 con 0x7fab08101850 2024-04-23T00:01:27.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.341+0000 7fab067fc700 1 -- 172.21.15.67:0/2316473008 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faafc010460 con 0x7fab08101850 2024-04-23T00:01:27.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.341+0000 7fab0f9cf700 1 -- 172.21.15.67:0/2316473008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab08196a50 con 0x7fab08101850 2024-04-23T00:01:27.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.341+0000 7fab067fc700 1 -- 172.21.15.67:0/2316473008 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faafc00f5f0 con 0x7fab08101850 2024-04-23T00:01:27.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.341+0000 7fab0f9cf700 1 -- 172.21.15.67:0/2316473008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab08196fa0 con 0x7fab08101850 2024-04-23T00:01:27.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.342+0000 7fab0f9cf700 1 -- 172.21.15.67:0/2316473008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab08109240 con 0x7fab08101850 2024-04-23T00:01:27.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.342+0000 7fab067fc700 1 -- 172.21.15.67:0/2316473008 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7faafc0105d0 con 0x7fab08101850 2024-04-23T00:01:27.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.343+0000 7fab067fc700 1 --2- 172.21.15.67:0/2316473008 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faaf006ecc0 0x7faaf0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:01:27.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.343+0000 7fab067fc700 1 -- 172.21.15.67:0/2316473008 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7faafc08ebe0 con 0x7fab08101850 2024-04-23T00:01:27.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.343+0000 7fab0d76b700 1 --2- 172.21.15.67:0/2316473008 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faaf006ecc0 0x7faaf0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:01:27.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.343+0000 7fab0d76b700 1 --2- 172.21.15.67:0/2316473008 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faaf006ecc0 0x7faaf0071180 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fab0000b830 tx=0x7fab000061f0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:01:27.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.348+0000 7fab067fc700 1 -- 172.21.15.67:0/2316473008 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7faafc05b330 con 0x7fab08101850 2024-04-23T00:01:27.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:27 smithi134 ceph-mon[85630]: from='client.35892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:01:27.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:27 smithi134 ceph-mon[85630]: from='client.35896 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:01:27.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:27 smithi134 ceph-mon[85630]: pgmap v1054: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:27.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:27 smithi067 ceph-mon[112438]: from='client.35892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:01:27.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:27 smithi067 ceph-mon[112438]: from='client.35896 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:01:27.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:27 smithi067 ceph-mon[112438]: pgmap v1054: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:27.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.604+0000 7fab0f9cf700 1 -- 172.21.15.67:0/2316473008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fab08197c80 con 0x7fab08101850 2024-04-23T00:01:27.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.604+0000 7fab067fc700 1 -- 172.21.15.67:0/2316473008 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1872 (secure 0 0 0) 0x7faafc017760 con 0x7fab08101850 2024-04-23T00:01:27.594 INFO:teuthology.orchestra.run.smithi067.stdout:e33 2024-04-23T00:01:27.594 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-23T00:01:27.594 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-23T00:01:27.594 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-23T00:01:27.594 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:01:27.595 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-23T00:01:27.595 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-23T00:01:27.595 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-23T00:01:27.595 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-23T00:01:27.595 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-23T00:01:27.595 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-23T00:01:27.595 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-23T00:01:27.595 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-23T00:01:27.595 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-23T00:01:27.595 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-23T00:01:27.595 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-23T00:01:27.595 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-23T00:01:27.595 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-23T00:01:27.595 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-23T00:01:27.595 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-23T00:01:27.595 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-23T00:01:27.596 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-23T00:01:27.596 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-23T00:01:27.596 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-23T00:01:27.596 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-23T00:01:27.596 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-23T00:01:27.596 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-23T00:01:27.596 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-23T00:01:27.596 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-23T00:01:27.596 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-23T00:01:27.596 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-23T00:01:27.596 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-23T00:01:27.596 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-23T00:01:27.596 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-23T00:01:27.597 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:01:27.597 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:01:27.597 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-23T00:01:27.597 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:01:27.597 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T00:01:27.597 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T00:01:27.597 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T00:01:27.597 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44851} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1849648622,v1:172.21.15.134:6827/1849648622] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T00:01:27.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.606+0000 7fab0f9cf700 1 -- 172.21.15.67:0/2316473008 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faaf006ecc0 msgr2=0x7faaf0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:27.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.607+0000 7fab0f9cf700 1 --2- 172.21.15.67:0/2316473008 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faaf006ecc0 0x7faaf0071180 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fab0000b830 tx=0x7fab000061f0 comp rx=0 tx=0).stop 2024-04-23T00:01:27.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.607+0000 7fab0f9cf700 1 -- 172.21.15.67:0/2316473008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab08101850 msgr2=0x7fab0819cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:27.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.607+0000 7fab0f9cf700 1 --2- 172.21.15.67:0/2316473008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab08101850 0x7fab0819cc30 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7faafc00d900 tx=0x7faafc00dcc0 comp rx=0 tx=0).stop 2024-04-23T00:01:27.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.607+0000 7fab0f9cf700 1 -- 172.21.15.67:0/2316473008 shutdown_connections 2024-04-23T00:01:27.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.607+0000 7fab0f9cf700 1 --2- 172.21.15.67:0/2316473008 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faaf006ecc0 0x7faaf0071180 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:27.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.607+0000 7fab0f9cf700 1 --2- 172.21.15.67:0/2316473008 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fab08100ea0 0x7fab0819c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:27.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.607+0000 7fab0f9cf700 1 --2- 172.21.15.67:0/2316473008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab08101850 0x7fab0819cc30 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:27.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.607+0000 7fab0f9cf700 1 -- 172.21.15.67:0/2316473008 >> 172.21.15.67:0/2316473008 conn(0x7fab080fc750 msgr2=0x7fab080fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:01:27.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.607+0000 7fab0f9cf700 1 -- 172.21.15.67:0/2316473008 shutdown_connections 2024-04-23T00:01:27.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.607+0000 7fab0f9cf700 1 -- 172.21.15.67:0/2316473008 wait complete. 2024-04-23T00:01:27.599 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 33 2024-04-23T00:01:27.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.715+0000 7f953063d700 1 -- 172.21.15.67:0/1778061640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9528105b80 msgr2=0x7f9528068580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:27.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.715+0000 7f953063d700 1 --2- 172.21.15.67:0/1778061640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9528105b80 0x7f9528068580 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f951c009910 tx=0x7f951c009c20 comp rx=0 tx=0).stop 2024-04-23T00:01:27.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.715+0000 7f953063d700 1 -- 172.21.15.67:0/1778061640 shutdown_connections 2024-04-23T00:01:27.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.715+0000 7f953063d700 1 --2- 172.21.15.67:0/1778061640 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9528068ac0 0x7f9528068f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:27.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.715+0000 7f953063d700 1 --2- 172.21.15.67:0/1778061640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9528105b80 0x7f9528068580 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:27.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.715+0000 7f953063d700 1 -- 172.21.15.67:0/1778061640 >> 172.21.15.67:0/1778061640 conn(0x7f9528075670 msgr2=0x7f9528075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:01:27.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.715+0000 7f953063d700 1 -- 172.21.15.67:0/1778061640 shutdown_connections 2024-04-23T00:01:27.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.715+0000 7f953063d700 1 -- 172.21.15.67:0/1778061640 wait complete. 2024-04-23T00:01:27.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.715+0000 7f953063d700 1 Processor -- start 2024-04-23T00:01:27.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.716+0000 7f953063d700 1 -- start start 2024-04-23T00:01:27.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.716+0000 7f953063d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9528068ac0 0x7f952819ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:01:27.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.716+0000 7f953063d700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9528105b80 0x7f952819d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:01:27.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.716+0000 7f953063d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952819da60 con 0x7f9528068ac0 2024-04-23T00:01:27.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.716+0000 7f953063d700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95281a17f0 con 0x7f9528105b80 2024-04-23T00:01:27.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.716+0000 7f952e3d9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9528068ac0 0x7f952819ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:01:27.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.716+0000 7f952e3d9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9528068ac0 0x7f952819ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60618/0 (socket says 172.21.15.67:60618) 2024-04-23T00:01:27.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.716+0000 7f952e3d9700 1 -- 172.21.15.67:0/183807268 learned_addr learned my addr 172.21.15.67:0/183807268 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:01:27.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.716+0000 7f952dbd8700 1 --2- 172.21.15.67:0/183807268 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9528105b80 0x7f952819d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:01:27.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.716+0000 7f952e3d9700 1 -- 172.21.15.67:0/183807268 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9528105b80 msgr2=0x7f952819d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:27.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.716+0000 7f952e3d9700 1 --2- 172.21.15.67:0/183807268 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9528105b80 0x7f952819d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:27.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.716+0000 7f952e3d9700 1 -- 172.21.15.67:0/183807268 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f951c0095f0 con 0x7f9528068ac0 2024-04-23T00:01:27.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.716+0000 7f952e3d9700 1 --2- 172.21.15.67:0/183807268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9528068ac0 0x7f952819ce40 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f951c0092e0 tx=0x7f951c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:01:27.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.717+0000 7f951b7fe700 1 -- 172.21.15.67:0/183807268 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f951c01d070 con 0x7f9528068ac0 2024-04-23T00:01:27.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.717+0000 7f951b7fe700 1 -- 172.21.15.67:0/183807268 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f951c005e00 con 0x7f9528068ac0 2024-04-23T00:01:27.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.717+0000 7f951b7fe700 1 -- 172.21.15.67:0/183807268 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f951c017400 con 0x7f9528068ac0 2024-04-23T00:01:27.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.717+0000 7f953063d700 1 -- 172.21.15.67:0/183807268 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95281a1a70 con 0x7f9528068ac0 2024-04-23T00:01:27.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.717+0000 7f953063d700 1 -- 172.21.15.67:0/183807268 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95281a1f60 con 0x7f9528068ac0 2024-04-23T00:01:27.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.718+0000 7f953063d700 1 -- 172.21.15.67:0/183807268 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95281101f0 con 0x7f9528068ac0 2024-04-23T00:01:27.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.718+0000 7f951b7fe700 1 -- 172.21.15.67:0/183807268 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f951c00f6f0 con 0x7f9528068ac0 2024-04-23T00:01:27.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.719+0000 7f951b7fe700 1 --2- 172.21.15.67:0/183807268 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f951406ecc0 0x7f9514071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:01:27.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.719+0000 7f951b7fe700 1 -- 172.21.15.67:0/183807268 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f951c0903a0 con 0x7f9528068ac0 2024-04-23T00:01:27.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.719+0000 7f952dbd8700 1 --2- 172.21.15.67:0/183807268 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f951406ecc0 0x7f9514071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:01:27.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.719+0000 7f952dbd8700 1 --2- 172.21.15.67:0/183807268 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f951406ecc0 0x7f9514071180 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f952819e460 tx=0x7f952400b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:01:27.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.724+0000 7f951b7fe700 1 -- 172.21.15.67:0/183807268 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f951c059060 con 0x7f9528068ac0 2024-04-23T00:01:27.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.955+0000 7f953063d700 1 -- 172.21.15.67:0/183807268 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f952802cce0 con 0x7f951406ecc0 2024-04-23T00:01:27.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.956+0000 7f951b7fe700 1 -- 172.21.15.67:0/183807268 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f952802cce0 con 0x7f951406ecc0 2024-04-23T00:01:27.946 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-23T00:01:27.946 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-23T00:01:27.946 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-23T00:01:27.946 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-23T00:01:27.946 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-23T00:01:27.946 INFO:teuthology.orchestra.run.smithi067.stdout: "mds", 2024-04-23T00:01:27.947 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-23T00:01:27.947 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-23T00:01:27.947 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-23T00:01:27.947 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-23T00:01:27.947 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-23T00:01:27.947 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "16/23 daemons upgraded", 2024-04-23T00:01:27.947 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-23T00:01:27.947 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-23T00:01:27.947 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-23T00:01:27.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.959+0000 7f953063d700 1 -- 172.21.15.67:0/183807268 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f951406ecc0 msgr2=0x7f9514071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:27.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.959+0000 7f953063d700 1 --2- 172.21.15.67:0/183807268 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f951406ecc0 0x7f9514071180 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f952819e460 tx=0x7f952400b410 comp rx=0 tx=0).stop 2024-04-23T00:01:27.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.959+0000 7f953063d700 1 -- 172.21.15.67:0/183807268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9528068ac0 msgr2=0x7f952819ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:27.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.959+0000 7f953063d700 1 --2- 172.21.15.67:0/183807268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9528068ac0 0x7f952819ce40 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f951c0092e0 tx=0x7f951c00bd70 comp rx=0 tx=0).stop 2024-04-23T00:01:27.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.959+0000 7f953063d700 1 -- 172.21.15.67:0/183807268 shutdown_connections 2024-04-23T00:01:27.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.959+0000 7f953063d700 1 --2- 172.21.15.67:0/183807268 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f951406ecc0 0x7f9514071180 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:27.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.959+0000 7f953063d700 1 --2- 172.21.15.67:0/183807268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9528068ac0 0x7f952819ce40 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:27.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.959+0000 7f953063d700 1 --2- 172.21.15.67:0/183807268 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9528105b80 0x7f952819d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:27.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.959+0000 7f953063d700 1 -- 172.21.15.67:0/183807268 >> 172.21.15.67:0/183807268 conn(0x7f9528075670 msgr2=0x7f95281016d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:01:27.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.959+0000 7f953063d700 1 -- 172.21.15.67:0/183807268 shutdown_connections 2024-04-23T00:01:27.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:27.959+0000 7f953063d700 1 -- 172.21.15.67:0/183807268 wait complete. 2024-04-23T00:01:28.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.065+0000 7f93c5bfb700 1 -- 172.21.15.67:0/3859361816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93c0101e00 msgr2=0x7f93c01021e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:28.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.065+0000 7f93c5bfb700 1 --2- 172.21.15.67:0/3859361816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93c0101e00 0x7f93c01021e0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f93b0009910 tx=0x7f93b0009c20 comp rx=0 tx=0).stop 2024-04-23T00:01:28.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.066+0000 7f93c5bfb700 1 -- 172.21.15.67:0/3859361816 shutdown_connections 2024-04-23T00:01:28.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.066+0000 7f93c5bfb700 1 --2- 172.21.15.67:0/3859361816 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93c0102720 0x7f93c010ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:28.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.066+0000 7f93c5bfb700 1 --2- 172.21.15.67:0/3859361816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93c0101e00 0x7f93c01021e0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:28.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.066+0000 7f93c5bfb700 1 -- 172.21.15.67:0/3859361816 >> 172.21.15.67:0/3859361816 conn(0x7f93c00fb650 msgr2=0x7f93c00fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:01:28.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.066+0000 7f93c5bfb700 1 -- 172.21.15.67:0/3859361816 shutdown_connections 2024-04-23T00:01:28.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.066+0000 7f93c5bfb700 1 -- 172.21.15.67:0/3859361816 wait complete. 2024-04-23T00:01:28.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.066+0000 7f93c5bfb700 1 Processor -- start 2024-04-23T00:01:28.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.066+0000 7f93c5bfb700 1 -- start start 2024-04-23T00:01:28.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.067+0000 7f93c5bfb700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93c0101e00 0x7f93c00ffa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:01:28.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.067+0000 7f93c5bfb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93c0102720 0x7f93c00fffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:01:28.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.067+0000 7f93c5bfb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c01004e0 con 0x7f93c0102720 2024-04-23T00:01:28.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.067+0000 7f93c5bfb700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c0100620 con 0x7f93c0101e00 2024-04-23T00:01:28.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.067+0000 7f93beffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93c0102720 0x7f93c00fffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:01:28.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.067+0000 7f93beffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93c0102720 0x7f93c00fffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60644/0 (socket says 172.21.15.67:60644) 2024-04-23T00:01:28.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.067+0000 7f93beffd700 1 -- 172.21.15.67:0/2532039080 learned_addr learned my addr 172.21.15.67:0/2532039080 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:01:28.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.067+0000 7f93bf7fe700 1 --2- 172.21.15.67:0/2532039080 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93c0101e00 0x7f93c00ffa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:01:28.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.067+0000 7f93beffd700 1 -- 172.21.15.67:0/2532039080 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93c0101e00 msgr2=0x7f93c00ffa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:28.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.067+0000 7f93beffd700 1 --2- 172.21.15.67:0/2532039080 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93c0101e00 0x7f93c00ffa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:28.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.067+0000 7f93beffd700 1 -- 172.21.15.67:0/2532039080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b00095f0 con 0x7f93c0102720 2024-04-23T00:01:28.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.067+0000 7f93beffd700 1 --2- 172.21.15.67:0/2532039080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93c0102720 0x7f93c00fffa0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f93a800b4d0 tx=0x7f93a800b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:01:28.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.067+0000 7f93bcff9700 1 -- 172.21.15.67:0/2532039080 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93a8011800 con 0x7f93c0102720 2024-04-23T00:01:28.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.068+0000 7f93bcff9700 1 -- 172.21.15.67:0/2532039080 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f93a8011e20 con 0x7f93c0102720 2024-04-23T00:01:28.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.068+0000 7f93c5bfb700 1 -- 172.21.15.67:0/2532039080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93c01a2bb0 con 0x7f93c0102720 2024-04-23T00:01:28.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.068+0000 7f93bcff9700 1 -- 172.21.15.67:0/2532039080 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93a8010e70 con 0x7f93c0102720 2024-04-23T00:01:28.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.068+0000 7f93c5bfb700 1 -- 172.21.15.67:0/2532039080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93c01a2f60 con 0x7f93c0102720 2024-04-23T00:01:28.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.069+0000 7f93c5bfb700 1 -- 172.21.15.67:0/2532039080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93c0108310 con 0x7f93c0102720 2024-04-23T00:01:28.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.069+0000 7f93bcff9700 1 -- 172.21.15.67:0/2532039080 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f93a8010380 con 0x7f93c0102720 2024-04-23T00:01:28.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.070+0000 7f93bcff9700 1 --2- 172.21.15.67:0/2532039080 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f93ac06ecc0 0x7f93ac071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:01:28.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.070+0000 7f93bcff9700 1 -- 172.21.15.67:0/2532039080 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f93a808ea10 con 0x7f93c0102720 2024-04-23T00:01:28.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.070+0000 7f93bf7fe700 1 --2- 172.21.15.67:0/2532039080 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f93ac06ecc0 0x7f93ac071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:01:28.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.070+0000 7f93bf7fe700 1 --2- 172.21.15.67:0/2532039080 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f93ac06ecc0 0x7f93ac071180 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f93b0009fd0 tx=0x7f93b001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:01:28.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.074+0000 7f93bcff9700 1 -- 172.21.15.67:0/2532039080 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f93a8057650 con 0x7f93c0102720 2024-04-23T00:01:28.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.375+0000 7f93c5bfb700 1 -- 172.21.15.67:0/2532039080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f93c01007e0 con 0x7f93c0102720 2024-04-23T00:01:28.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.375+0000 7f93bcff9700 1 -- 172.21.15.67:0/2532039080 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4505 (secure 0 0 0) 0x7f93a805ad10 con 0x7f93c0102720 2024-04-23T00:01:28.365 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-23T00:01:28.365 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-23T00:01:28.366 INFO:teuthology.orchestra.run.smithi067.stdout: host smithi134 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-23T00:01:28.366 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-23T00:01:28.366 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-23T00:01:28.366 INFO:teuthology.orchestra.run.smithi067.stdout:Traceback (most recent call last): 2024-04-23T00:01:28.366 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-23T00:01:28.366 INFO:teuthology.orchestra.run.smithi067.stdout: return _run_code(code, main_globals, None, 2024-04-23T00:01:28.366 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-23T00:01:28.366 INFO:teuthology.orchestra.run.smithi067.stdout: exec(code, run_globals) 2024-04-23T00:01:28.366 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5642, in 2024-04-23T00:01:28.366 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5630, in main 2024-04-23T00:01:28.366 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 408, in _infer_config 2024-04-23T00:01:28.366 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 323, in _infer_fsid 2024-04-23T00:01:28.366 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 436, in _infer_image 2024-04-23T00:01:28.366 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 310, in _validate_fsid 2024-04-23T00:01:28.367 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 3380, in command_ceph_volume 2024-04-23T00:01:28.367 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-23T00:01:28.367 INFO:teuthology.orchestra.run.smithi067.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-23T00:01:28.367 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-23T00:01:28.367 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-23T00:01:28.367 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-23T00:01:28.367 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-23T00:01:28.367 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-23T00:01:28.367 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-23T00:01:28.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.377+0000 7f93c5bfb700 1 -- 172.21.15.67:0/2532039080 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f93ac06ecc0 msgr2=0x7f93ac071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:28.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.377+0000 7f93c5bfb700 1 --2- 172.21.15.67:0/2532039080 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f93ac06ecc0 0x7f93ac071180 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f93b0009fd0 tx=0x7f93b001a040 comp rx=0 tx=0).stop 2024-04-23T00:01:28.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.378+0000 7f93c5bfb700 1 -- 172.21.15.67:0/2532039080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93c0102720 msgr2=0x7f93c00fffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:28.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.378+0000 7f93c5bfb700 1 --2- 172.21.15.67:0/2532039080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93c0102720 0x7f93c00fffa0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f93a800b4d0 tx=0x7f93a800b7e0 comp rx=0 tx=0).stop 2024-04-23T00:01:28.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.378+0000 7f93c5bfb700 1 -- 172.21.15.67:0/2532039080 shutdown_connections 2024-04-23T00:01:28.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.378+0000 7f93c5bfb700 1 --2- 172.21.15.67:0/2532039080 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f93ac06ecc0 0x7f93ac071180 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:28.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.378+0000 7f93c5bfb700 1 --2- 172.21.15.67:0/2532039080 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f93c0101e00 0x7f93c00ffa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:28.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.378+0000 7f93c5bfb700 1 --2- 172.21.15.67:0/2532039080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93c0102720 0x7f93c00fffa0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:28.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.378+0000 7f93c5bfb700 1 -- 172.21.15.67:0/2532039080 >> 172.21.15.67:0/2532039080 conn(0x7f93c00fb650 msgr2=0x7f93c0105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:01:28.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.378+0000 7f93c5bfb700 1 -- 172.21.15.67:0/2532039080 shutdown_connections 2024-04-23T00:01:28.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:28.378+0000 7f93c5bfb700 1 -- 172.21.15.67:0/2532039080 wait complete. 2024-04-23T00:01:28.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:28 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1122147191' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:01:28.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:28 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2316473008' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-23T00:01:28.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:28 smithi134 ceph-mon[85630]: from='client.35908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:01:28.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:28 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1122147191' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:01:28.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:28 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2316473008' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-23T00:01:28.534 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:28 smithi067 ceph-mon[112438]: from='client.35908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:01:29.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:29 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2532039080' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T00:01:29.487 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:29 smithi134 ceph-mon[85630]: pgmap v1055: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:29.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:29 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2532039080' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T00:01:30.147 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:29 smithi067 ceph-mon[112438]: pgmap v1055: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:32.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:32 smithi134 ceph-mon[85630]: pgmap v1056: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:32.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:32 smithi067 ceph-mon[112438]: pgmap v1056: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:34.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:34 smithi134 ceph-mon[85630]: pgmap v1057: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:34.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:34 smithi067 ceph-mon[112438]: pgmap v1057: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:36.283 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:36 smithi067 ceph-mon[112438]: pgmap v1058: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:36.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:36 smithi134 ceph-mon[85630]: pgmap v1058: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:38.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:38 smithi134 ceph-mon[85630]: pgmap v1059: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:38.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:38 smithi067 ceph-mon[112438]: pgmap v1059: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:40.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:40 smithi134 ceph-mon[85630]: pgmap v1060: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:40.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:40 smithi067 ceph-mon[112438]: pgmap v1060: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:41.283 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:01:41.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:01:42.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:42 smithi134 ceph-mon[85630]: pgmap v1061: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:42.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:42 smithi067 ceph-mon[112438]: pgmap v1061: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:44.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:44 smithi134 ceph-mon[85630]: pgmap v1062: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:44.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:44 smithi067 ceph-mon[112438]: pgmap v1062: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:46.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:46 smithi134 ceph-mon[85630]: pgmap v1063: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:46.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:46 smithi067 ceph-mon[112438]: pgmap v1063: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:48.335 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:48 smithi067 ceph-mon[112438]: pgmap v1064: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:48.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:48 smithi134 ceph-mon[85630]: pgmap v1064: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:50.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:50 smithi067 ceph-mon[112438]: pgmap v1065: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:50.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:50 smithi134 ceph-mon[85630]: pgmap v1065: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:52.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:52 smithi067 ceph-mon[112438]: pgmap v1066: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:52.535 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:52 smithi134 ceph-mon[85630]: pgmap v1066: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:54.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:54 smithi067 ceph-mon[112438]: pgmap v1067: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:54.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:54 smithi134 ceph-mon[85630]: pgmap v1067: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:56.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:56 smithi067 ceph-mon[112438]: pgmap v1068: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:56.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:01:56.733 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:56 smithi134 ceph-mon[85630]: pgmap v1068: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:56.733 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:01:58.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.498+0000 7fcc82b2c700 1 -- 172.21.15.67:0/2353009851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc7c106cd0 msgr2=0x7fcc7c1070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:58.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.498+0000 7fcc82b2c700 1 --2- 172.21.15.67:0/2353009851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc7c106cd0 0x7fcc7c1070b0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c009910 tx=0x7fcc6c009c20 comp rx=0 tx=0).stop 2024-04-23T00:01:58.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.498+0000 7fcc82b2c700 1 -- 172.21.15.67:0/2353009851 shutdown_connections 2024-04-23T00:01:58.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.498+0000 7fcc82b2c700 1 --2- 172.21.15.67:0/2353009851 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc7c068580 0x7fcc7c0689e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:58.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.498+0000 7fcc82b2c700 1 --2- 172.21.15.67:0/2353009851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc7c106cd0 0x7fcc7c1070b0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:58.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.498+0000 7fcc82b2c700 1 -- 172.21.15.67:0/2353009851 >> 172.21.15.67:0/2353009851 conn(0x7fcc7c075670 msgr2=0x7fcc7c075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:01:58.489 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:58 smithi067 ceph-mon[112438]: pgmap v1069: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:58.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.498+0000 7fcc82b2c700 1 -- 172.21.15.67:0/2353009851 shutdown_connections 2024-04-23T00:01:58.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.498+0000 7fcc82b2c700 1 -- 172.21.15.67:0/2353009851 wait complete. 2024-04-23T00:01:58.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.499+0000 7fcc82b2c700 1 Processor -- start 2024-04-23T00:01:58.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.499+0000 7fcc82b2c700 1 -- start start 2024-04-23T00:01:58.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.499+0000 7fcc82b2c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc7c068580 0x7fcc7c0ffcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:01:58.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.499+0000 7fcc82b2c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc7c106cd0 0x7fcc7c100230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:01:58.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.499+0000 7fcc82b2c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c100770 con 0x7fcc7c068580 2024-04-23T00:01:58.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.499+0000 7fcc82b2c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c1008b0 con 0x7fcc7c106cd0 2024-04-23T00:01:58.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.499+0000 7fcc808c8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc7c068580 0x7fcc7c0ffcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:01:58.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.499+0000 7fcc808c8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc7c068580 0x7fcc7c0ffcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46384/0 (socket says 172.21.15.67:46384) 2024-04-23T00:01:58.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.500+0000 7fcc808c8700 1 -- 172.21.15.67:0/175517028 learned_addr learned my addr 172.21.15.67:0/175517028 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:01:58.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.500+0000 7fcc7bfff700 1 --2- 172.21.15.67:0/175517028 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc7c106cd0 0x7fcc7c100230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:01:58.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.500+0000 7fcc808c8700 1 -- 172.21.15.67:0/175517028 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc7c106cd0 msgr2=0x7fcc7c100230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:58.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.500+0000 7fcc808c8700 1 --2- 172.21.15.67:0/175517028 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc7c106cd0 0x7fcc7c100230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:58.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.500+0000 7fcc808c8700 1 -- 172.21.15.67:0/175517028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc6c0095f0 con 0x7fcc7c068580 2024-04-23T00:01:58.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.500+0000 7fcc808c8700 1 --2- 172.21.15.67:0/175517028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc7c068580 0x7fcc7c0ffcf0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c000c00 tx=0x7fcc6c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:01:58.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.500+0000 7fcc79ffb700 1 -- 172.21.15.67:0/175517028 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc6c01d070 con 0x7fcc7c068580 2024-04-23T00:01:58.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.500+0000 7fcc79ffb700 1 -- 172.21.15.67:0/175517028 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcc6c00f460 con 0x7fcc7c068580 2024-04-23T00:01:58.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.500+0000 7fcc82b2c700 1 -- 172.21.15.67:0/175517028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc7c100b30 con 0x7fcc7c068580 2024-04-23T00:01:58.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.500+0000 7fcc79ffb700 1 -- 172.21.15.67:0/175517028 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc6c017630 con 0x7fcc7c068580 2024-04-23T00:01:58.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.501+0000 7fcc82b2c700 1 -- 172.21.15.67:0/175517028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc7c1a2d90 con 0x7fcc7c068580 2024-04-23T00:01:58.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.502+0000 7fcc82b2c700 1 -- 172.21.15.67:0/175517028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc7c1091f0 con 0x7fcc7c068580 2024-04-23T00:01:58.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.502+0000 7fcc79ffb700 1 -- 172.21.15.67:0/175517028 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcc6c00f5d0 con 0x7fcc7c068580 2024-04-23T00:01:58.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.503+0000 7fcc79ffb700 1 --2- 172.21.15.67:0/175517028 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcc6406ecc0 0x7fcc64071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:01:58.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.503+0000 7fcc79ffb700 1 -- 172.21.15.67:0/175517028 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fcc6c090450 con 0x7fcc7c068580 2024-04-23T00:01:58.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.503+0000 7fcc7bfff700 1 --2- 172.21.15.67:0/175517028 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcc6406ecc0 0x7fcc64071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:01:58.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.503+0000 7fcc7bfff700 1 --2- 172.21.15.67:0/175517028 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcc6406ecc0 0x7fcc64071180 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c1014e0 tx=0x7fcc7000a300 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:01:58.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.508+0000 7fcc79ffb700 1 -- 172.21.15.67:0/175517028 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fcc6c059110 con 0x7fcc7c068580 2024-04-23T00:01:58.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.737+0000 7fcc82b2c700 1 -- 172.21.15.67:0/175517028 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcc7c0611d0 con 0x7fcc6406ecc0 2024-04-23T00:01:58.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.738+0000 7fcc79ffb700 1 -- 172.21.15.67:0/175517028 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fcc7c0611d0 con 0x7fcc6406ecc0 2024-04-23T00:01:58.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.740+0000 7fcc82b2c700 1 -- 172.21.15.67:0/175517028 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcc6406ecc0 msgr2=0x7fcc64071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:58.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.740+0000 7fcc82b2c700 1 --2- 172.21.15.67:0/175517028 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcc6406ecc0 0x7fcc64071180 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c1014e0 tx=0x7fcc7000a300 comp rx=0 tx=0).stop 2024-04-23T00:01:58.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.740+0000 7fcc82b2c700 1 -- 172.21.15.67:0/175517028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc7c068580 msgr2=0x7fcc7c0ffcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:58.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.740+0000 7fcc82b2c700 1 --2- 172.21.15.67:0/175517028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc7c068580 0x7fcc7c0ffcf0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c000c00 tx=0x7fcc6c00bd70 comp rx=0 tx=0).stop 2024-04-23T00:01:58.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.740+0000 7fcc82b2c700 1 -- 172.21.15.67:0/175517028 shutdown_connections 2024-04-23T00:01:58.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.740+0000 7fcc82b2c700 1 --2- 172.21.15.67:0/175517028 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fcc6406ecc0 0x7fcc64071180 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:58.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.740+0000 7fcc82b2c700 1 --2- 172.21.15.67:0/175517028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc7c068580 0x7fcc7c0ffcf0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:58.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.740+0000 7fcc82b2c700 1 --2- 172.21.15.67:0/175517028 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc7c106cd0 0x7fcc7c100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:58.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.740+0000 7fcc82b2c700 1 -- 172.21.15.67:0/175517028 >> 172.21.15.67:0/175517028 conn(0x7fcc7c075670 msgr2=0x7fcc7c0fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:01:58.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.740+0000 7fcc82b2c700 1 -- 172.21.15.67:0/175517028 shutdown_connections 2024-04-23T00:01:58.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.740+0000 7fcc82b2c700 1 -- 172.21.15.67:0/175517028 wait complete. 2024-04-23T00:01:58.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:58 smithi134 ceph-mon[85630]: pgmap v1069: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:01:58.740 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-23T00:01:58.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.848+0000 7faf6291c700 1 -- 172.21.15.67:0/3028602757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf5c100ea0 msgr2=0x7faf5c101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:58.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.848+0000 7faf6291c700 1 --2- 172.21.15.67:0/3028602757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf5c100ea0 0x7faf5c101280 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7faf4c009910 tx=0x7faf4c009c20 comp rx=0 tx=0).stop 2024-04-23T00:01:58.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.848+0000 7faf6291c700 1 -- 172.21.15.67:0/3028602757 shutdown_connections 2024-04-23T00:01:58.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.848+0000 7faf6291c700 1 --2- 172.21.15.67:0/3028602757 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf5c101850 0x7faf5c1058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:58.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.848+0000 7faf6291c700 1 --2- 172.21.15.67:0/3028602757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf5c100ea0 0x7faf5c101280 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:58.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.848+0000 7faf6291c700 1 -- 172.21.15.67:0/3028602757 >> 172.21.15.67:0/3028602757 conn(0x7faf5c0fc750 msgr2=0x7faf5c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:01:58.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.848+0000 7faf6291c700 1 -- 172.21.15.67:0/3028602757 shutdown_connections 2024-04-23T00:01:58.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.848+0000 7faf6291c700 1 -- 172.21.15.67:0/3028602757 wait complete. 2024-04-23T00:01:58.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.849+0000 7faf6291c700 1 Processor -- start 2024-04-23T00:01:58.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.849+0000 7faf6291c700 1 -- start start 2024-04-23T00:01:58.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.850+0000 7faf6291c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf5c100ea0 0x7faf5c19c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:01:58.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.850+0000 7faf6291c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf5c101850 0x7faf5c19cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:01:58.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.850+0000 7faf6291c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf5c19d2c0 con 0x7faf5c100ea0 2024-04-23T00:01:58.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.850+0000 7faf6291c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf5c196770 con 0x7faf5c101850 2024-04-23T00:01:58.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.850+0000 7faf5bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf5c100ea0 0x7faf5c19c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:01:58.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.850+0000 7faf5bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf5c100ea0 0x7faf5c19c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55340/0 (socket says 172.21.15.67:55340) 2024-04-23T00:01:58.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.850+0000 7faf5bfff700 1 -- 172.21.15.67:0/2059239998 learned_addr learned my addr 172.21.15.67:0/2059239998 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:01:58.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.850+0000 7faf5b7fe700 1 --2- 172.21.15.67:0/2059239998 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf5c101850 0x7faf5c19cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:01:58.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.850+0000 7faf5bfff700 1 -- 172.21.15.67:0/2059239998 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf5c101850 msgr2=0x7faf5c19cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:58.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.850+0000 7faf5bfff700 1 --2- 172.21.15.67:0/2059239998 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf5c101850 0x7faf5c19cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:58.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.850+0000 7faf5bfff700 1 -- 172.21.15.67:0/2059239998 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf4c0095f0 con 0x7faf5c100ea0 2024-04-23T00:01:58.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.850+0000 7faf5bfff700 1 --2- 172.21.15.67:0/2059239998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf5c100ea0 0x7faf5c19c6f0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7faf4c006010 tx=0x7faf4c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:01:58.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.850+0000 7faf597fa700 1 -- 172.21.15.67:0/2059239998 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf4c01d070 con 0x7faf5c100ea0 2024-04-23T00:01:58.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.850+0000 7faf6291c700 1 -- 172.21.15.67:0/2059239998 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf5c1969f0 con 0x7faf5c100ea0 2024-04-23T00:01:58.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.850+0000 7faf597fa700 1 -- 172.21.15.67:0/2059239998 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faf4c005dc0 con 0x7faf5c100ea0 2024-04-23T00:01:58.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.850+0000 7faf597fa700 1 -- 172.21.15.67:0/2059239998 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf4c017400 con 0x7faf5c100ea0 2024-04-23T00:01:58.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.850+0000 7faf6291c700 1 -- 172.21.15.67:0/2059239998 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf5c196ee0 con 0x7faf5c100ea0 2024-04-23T00:01:58.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.852+0000 7faf6291c700 1 -- 172.21.15.67:0/2059239998 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf5c109240 con 0x7faf5c100ea0 2024-04-23T00:01:58.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.852+0000 7faf597fa700 1 -- 172.21.15.67:0/2059239998 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7faf4c017560 con 0x7faf5c100ea0 2024-04-23T00:01:58.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.852+0000 7faf597fa700 1 --2- 172.21.15.67:0/2059239998 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faf4406ecc0 0x7faf44071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:01:58.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.853+0000 7faf597fa700 1 -- 172.21.15.67:0/2059239998 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7faf4c090440 con 0x7faf5c100ea0 2024-04-23T00:01:58.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.853+0000 7faf5b7fe700 1 --2- 172.21.15.67:0/2059239998 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faf4406ecc0 0x7faf44071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:01:58.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.853+0000 7faf5b7fe700 1 --2- 172.21.15.67:0/2059239998 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faf4406ecc0 0x7faf44071180 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7faf5c197be0 tx=0x7faf5000b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:01:58.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:58.857+0000 7faf597fa700 1 -- 172.21.15.67:0/2059239998 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7faf4c05cbb0 con 0x7faf5c100ea0 2024-04-23T00:01:59.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.089+0000 7faf6291c700 1 -- 172.21.15.67:0/2059239998 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7faf5c197150 con 0x7faf4406ecc0 2024-04-23T00:01:59.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.090+0000 7faf597fa700 1 -- 172.21.15.67:0/2059239998 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7faf5c197150 con 0x7faf4406ecc0 2024-04-23T00:01:59.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.092+0000 7faf6291c700 1 -- 172.21.15.67:0/2059239998 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faf4406ecc0 msgr2=0x7faf44071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:59.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.092+0000 7faf6291c700 1 --2- 172.21.15.67:0/2059239998 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faf4406ecc0 0x7faf44071180 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7faf5c197be0 tx=0x7faf5000b410 comp rx=0 tx=0).stop 2024-04-23T00:01:59.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.092+0000 7faf6291c700 1 -- 172.21.15.67:0/2059239998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf5c100ea0 msgr2=0x7faf5c19c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:59.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.092+0000 7faf6291c700 1 --2- 172.21.15.67:0/2059239998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf5c100ea0 0x7faf5c19c6f0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7faf4c006010 tx=0x7faf4c00bd70 comp rx=0 tx=0).stop 2024-04-23T00:01:59.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.092+0000 7faf6291c700 1 -- 172.21.15.67:0/2059239998 shutdown_connections 2024-04-23T00:01:59.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.092+0000 7faf6291c700 1 --2- 172.21.15.67:0/2059239998 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7faf4406ecc0 0x7faf44071180 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:59.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.092+0000 7faf6291c700 1 --2- 172.21.15.67:0/2059239998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf5c100ea0 0x7faf5c19c6f0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:59.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.092+0000 7faf6291c700 1 --2- 172.21.15.67:0/2059239998 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faf5c101850 0x7faf5c19cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:59.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.092+0000 7faf6291c700 1 -- 172.21.15.67:0/2059239998 >> 172.21.15.67:0/2059239998 conn(0x7faf5c0fc750 msgr2=0x7faf5c0fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:01:59.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.092+0000 7faf6291c700 1 -- 172.21.15.67:0/2059239998 shutdown_connections 2024-04-23T00:01:59.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.092+0000 7faf6291c700 1 -- 172.21.15.67:0/2059239998 wait complete. 2024-04-23T00:01:59.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.200+0000 7fe585b59700 1 -- 172.21.15.67:0/963867555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5800ffda0 msgr2=0x7fe580100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:59.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.200+0000 7fe585b59700 1 --2- 172.21.15.67:0/963867555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5800ffda0 0x7fe580100180 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fe570009910 tx=0x7fe570009c20 comp rx=0 tx=0).stop 2024-04-23T00:01:59.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.200+0000 7fe585b59700 1 -- 172.21.15.67:0/963867555 shutdown_connections 2024-04-23T00:01:59.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.200+0000 7fe585b59700 1 --2- 172.21.15.67:0/963867555 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe580100750 0x7fe5801047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:59.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.200+0000 7fe585b59700 1 --2- 172.21.15.67:0/963867555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5800ffda0 0x7fe580100180 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:59.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.200+0000 7fe585b59700 1 -- 172.21.15.67:0/963867555 >> 172.21.15.67:0/963867555 conn(0x7fe5800fb650 msgr2=0x7fe5800fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:01:59.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.200+0000 7fe585b59700 1 -- 172.21.15.67:0/963867555 shutdown_connections 2024-04-23T00:01:59.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.200+0000 7fe585b59700 1 -- 172.21.15.67:0/963867555 wait complete. 2024-04-23T00:01:59.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.200+0000 7fe585b59700 1 Processor -- start 2024-04-23T00:01:59.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.200+0000 7fe585b59700 1 -- start start 2024-04-23T00:01:59.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.201+0000 7fe585b59700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5800ffda0 0x7fe580101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:01:59.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.201+0000 7fe585b59700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe580100750 0x7fe580101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:01:59.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.201+0000 7fe585b59700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5801022c0 con 0x7fe5800ffda0 2024-04-23T00:01:59.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.201+0000 7fe585b59700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe580102400 con 0x7fe580100750 2024-04-23T00:01:59.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.201+0000 7fe57f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5800ffda0 0x7fe580101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:01:59.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.201+0000 7fe57f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5800ffda0 0x7fe580101840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55354/0 (socket says 172.21.15.67:55354) 2024-04-23T00:01:59.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.201+0000 7fe57f7fe700 1 -- 172.21.15.67:0/2535962805 learned_addr learned my addr 172.21.15.67:0/2535962805 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:01:59.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.201+0000 7fe57f7fe700 1 -- 172.21.15.67:0/2535962805 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe580100750 msgr2=0x7fe580101d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:59.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.201+0000 7fe576ffd700 1 --2- 172.21.15.67:0/2535962805 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe580100750 0x7fe580101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:01:59.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.201+0000 7fe57f7fe700 1 --2- 172.21.15.67:0/2535962805 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe580100750 0x7fe580101d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:59.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.201+0000 7fe57f7fe700 1 -- 172.21.15.67:0/2535962805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5700095f0 con 0x7fe5800ffda0 2024-04-23T00:01:59.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.201+0000 7fe57f7fe700 1 --2- 172.21.15.67:0/2535962805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5800ffda0 0x7fe580101840 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fe570006010 tx=0x7fe57000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:01:59.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.202+0000 7fe57d7fa700 1 -- 172.21.15.67:0/2535962805 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe57001d070 con 0x7fe5800ffda0 2024-04-23T00:01:59.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.202+0000 7fe57d7fa700 1 -- 172.21.15.67:0/2535962805 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe57000f460 con 0x7fe5800ffda0 2024-04-23T00:01:59.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.202+0000 7fe585b59700 1 -- 172.21.15.67:0/2535962805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe580102680 con 0x7fe5800ffda0 2024-04-23T00:01:59.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.202+0000 7fe57d7fa700 1 -- 172.21.15.67:0/2535962805 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5700175f0 con 0x7fe5800ffda0 2024-04-23T00:01:59.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.202+0000 7fe585b59700 1 -- 172.21.15.67:0/2535962805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5801a2d60 con 0x7fe5800ffda0 2024-04-23T00:01:59.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.203+0000 7fe585b59700 1 -- 172.21.15.67:0/2535962805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe58004f2e0 con 0x7fe5800ffda0 2024-04-23T00:01:59.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.203+0000 7fe57d7fa700 1 -- 172.21.15.67:0/2535962805 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe57000f5d0 con 0x7fe5800ffda0 2024-04-23T00:01:59.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.204+0000 7fe57d7fa700 1 --2- 172.21.15.67:0/2535962805 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe56c06ecc0 0x7fe56c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:01:59.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.204+0000 7fe57d7fa700 1 -- 172.21.15.67:0/2535962805 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fe570090480 con 0x7fe5800ffda0 2024-04-23T00:01:59.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.204+0000 7fe576ffd700 1 --2- 172.21.15.67:0/2535962805 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe56c06ecc0 0x7fe56c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:01:59.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.204+0000 7fe576ffd700 1 --2- 172.21.15.67:0/2535962805 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe56c06ecc0 0x7fe56c071180 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fe580103030 tx=0x7fe568005cf0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:01:59.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.209+0000 7fe57d7fa700 1 -- 172.21.15.67:0/2535962805 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe570059140 con 0x7fe5800ffda0 2024-04-23T00:01:59.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.428+0000 7fe585b59700 1 -- 172.21.15.67:0/2535962805 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe5800fc4a0 con 0x7fe56c06ecc0 2024-04-23T00:01:59.417 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:59 smithi067 ceph-mon[112438]: from='client.35916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:01:59.418 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:01:59 smithi067 ceph-mon[112438]: from='client.35920 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:01:59.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.441+0000 7fe57d7fa700 1 -- 172.21.15.67:0/2535962805 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fe5800fc4a0 con 0x7fe56c06ecc0 2024-04-23T00:01:59.430 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-23T00:01:59.431 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (35m) 13m ago 46m 24.4M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-23T00:01:59.431 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (47m) 13m ago 47m 10.0M - 18.2.1 5be31c24972a df887f62291e 2024-04-23T00:01:59.431 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (46m) 13m ago 46m 10.0M - 18.2.1 5be31c24972a ad89190b5400 2024-04-23T00:01:59.431 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (33m) 13m ago 47m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-23T00:01:59.431 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (32m) 13m ago 46m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-23T00:01:59.431 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (35m) 13m ago 46m 81.2M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-23T00:01:59.431 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (29m) 13m ago 41m 19.4M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-23T00:01:59.431 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (29m) 13m ago 41m 20.3M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-23T00:01:59.431 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (28m) 13m ago 41m 20.1M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-23T00:01:59.431 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (13m) 13m ago 41m 14.0M - 19.0.0-3252-g43be0201 a0434699c1d7 46d3d7e86fee 2024-04-23T00:01:59.431 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (37m) 13m ago 48m 514M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-23T00:01:59.431 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (36m) 13m ago 46m 459M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-23T00:01:59.431 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (34m) 13m ago 48m 74.2M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-23T00:01:59.432 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (33m) 13m ago 45m 54.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-23T00:01:59.432 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (36m) 13m ago 47m 15.2M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-23T00:01:59.432 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (36m) 13m ago 46m 16.0M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-23T00:01:59.432 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (32m) 13m ago 45m 102M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-23T00:01:59.432 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (32m) 13m ago 44m 128M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-23T00:01:59.432 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (31m) 13m ago 44m 83.8M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-23T00:01:59.432 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (30m) 13m ago 43m 101M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-23T00:01:59.432 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (30m) 13m ago 43m 86.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-23T00:01:59.432 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (29m) 13m ago 43m 81.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-23T00:01:59.432 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (36m) 13m ago 46m 78.7M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-23T00:01:59.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.444+0000 7fe585b59700 1 -- 172.21.15.67:0/2535962805 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe56c06ecc0 msgr2=0x7fe56c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:59.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.444+0000 7fe585b59700 1 --2- 172.21.15.67:0/2535962805 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe56c06ecc0 0x7fe56c071180 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fe580103030 tx=0x7fe568005cf0 comp rx=0 tx=0).stop 2024-04-23T00:01:59.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.444+0000 7fe585b59700 1 -- 172.21.15.67:0/2535962805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5800ffda0 msgr2=0x7fe580101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:59.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.444+0000 7fe585b59700 1 --2- 172.21.15.67:0/2535962805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5800ffda0 0x7fe580101840 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fe570006010 tx=0x7fe57000be20 comp rx=0 tx=0).stop 2024-04-23T00:01:59.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.444+0000 7fe585b59700 1 -- 172.21.15.67:0/2535962805 shutdown_connections 2024-04-23T00:01:59.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.444+0000 7fe585b59700 1 --2- 172.21.15.67:0/2535962805 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe56c06ecc0 0x7fe56c071180 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:59.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.444+0000 7fe585b59700 1 --2- 172.21.15.67:0/2535962805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5800ffda0 0x7fe580101840 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:59.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.444+0000 7fe585b59700 1 --2- 172.21.15.67:0/2535962805 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe580100750 0x7fe580101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:59.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.444+0000 7fe585b59700 1 -- 172.21.15.67:0/2535962805 >> 172.21.15.67:0/2535962805 conn(0x7fe5800fb650 msgr2=0x7fe580103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:01:59.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.444+0000 7fe585b59700 1 -- 172.21.15.67:0/2535962805 shutdown_connections 2024-04-23T00:01:59.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.444+0000 7fe585b59700 1 -- 172.21.15.67:0/2535962805 wait complete. 2024-04-23T00:01:59.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.549+0000 7f7e58e2c700 1 -- 172.21.15.67:0/2831517974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e54105350 msgr2=0x7f7e54105730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:59.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.549+0000 7f7e58e2c700 1 --2- 172.21.15.67:0/2831517974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e54105350 0x7f7e54105730 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f7e44009910 tx=0x7f7e44009c20 comp rx=0 tx=0).stop 2024-04-23T00:01:59.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.550+0000 7f7e58e2c700 1 -- 172.21.15.67:0/2831517974 shutdown_connections 2024-04-23T00:01:59.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.550+0000 7f7e58e2c700 1 --2- 172.21.15.67:0/2831517974 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7e540ff340 0x7f7e540ff7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:59.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.550+0000 7f7e58e2c700 1 --2- 172.21.15.67:0/2831517974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e54105350 0x7f7e54105730 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:59.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.550+0000 7f7e58e2c700 1 -- 172.21.15.67:0/2831517974 >> 172.21.15.67:0/2831517974 conn(0x7f7e54074e30 msgr2=0x7f7e54075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:01:59.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.550+0000 7f7e58e2c700 1 -- 172.21.15.67:0/2831517974 shutdown_connections 2024-04-23T00:01:59.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.550+0000 7f7e58e2c700 1 -- 172.21.15.67:0/2831517974 wait complete. 2024-04-23T00:01:59.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.550+0000 7f7e58e2c700 1 Processor -- start 2024-04-23T00:01:59.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.550+0000 7f7e58e2c700 1 -- start start 2024-04-23T00:01:59.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.551+0000 7f7e58e2c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7e540ff340 0x7f7e54196990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:01:59.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.551+0000 7f7e58e2c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e54105350 0x7f7e54196ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:01:59.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.551+0000 7f7e58e2c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e54197560 con 0x7f7e54105350 2024-04-23T00:01:59.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.551+0000 7f7e58e2c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5419b370 con 0x7f7e540ff340 2024-04-23T00:01:59.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.551+0000 7f7e51d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e54105350 0x7f7e54196ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:01:59.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.551+0000 7f7e51d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e54105350 0x7f7e54196ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55368/0 (socket says 172.21.15.67:55368) 2024-04-23T00:01:59.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.551+0000 7f7e51d9b700 1 -- 172.21.15.67:0/3294426907 learned_addr learned my addr 172.21.15.67:0/3294426907 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:01:59.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.551+0000 7f7e5259c700 1 --2- 172.21.15.67:0/3294426907 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7e540ff340 0x7f7e54196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:01:59.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.551+0000 7f7e51d9b700 1 -- 172.21.15.67:0/3294426907 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7e540ff340 msgr2=0x7f7e54196990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:59.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.551+0000 7f7e51d9b700 1 --2- 172.21.15.67:0/3294426907 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7e540ff340 0x7f7e54196990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:59.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.551+0000 7f7e51d9b700 1 -- 172.21.15.67:0/3294426907 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e440095f0 con 0x7f7e54105350 2024-04-23T00:01:59.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.551+0000 7f7e51d9b700 1 --2- 172.21.15.67:0/3294426907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e54105350 0x7f7e54196ed0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f7e4800b4b0 tx=0x7f7e4800b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:01:59.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.551+0000 7f7e437fe700 1 -- 172.21.15.67:0/3294426907 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e48011800 con 0x7f7e54105350 2024-04-23T00:01:59.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.551+0000 7f7e58e2c700 1 -- 172.21.15.67:0/3294426907 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e5419b650 con 0x7f7e54105350 2024-04-23T00:01:59.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.552+0000 7f7e437fe700 1 -- 172.21.15.67:0/3294426907 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7e48011e20 con 0x7f7e54105350 2024-04-23T00:01:59.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.552+0000 7f7e437fe700 1 -- 172.21.15.67:0/3294426907 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e4800f2e0 con 0x7f7e54105350 2024-04-23T00:01:59.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.552+0000 7f7e58e2c700 1 -- 172.21.15.67:0/3294426907 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e5419bba0 con 0x7f7e54105350 2024-04-23T00:01:59.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.553+0000 7f7e58e2c700 1 -- 172.21.15.67:0/3294426907 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e54109880 con 0x7f7e54105350 2024-04-23T00:01:59.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.553+0000 7f7e437fe700 1 -- 172.21.15.67:0/3294426907 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7e48010380 con 0x7f7e54105350 2024-04-23T00:01:59.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.554+0000 7f7e437fe700 1 --2- 172.21.15.67:0/3294426907 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7e3c06ecc0 0x7f7e3c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:01:59.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.554+0000 7f7e437fe700 1 -- 172.21.15.67:0/3294426907 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f7e4808eb90 con 0x7f7e54105350 2024-04-23T00:01:59.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.554+0000 7f7e5259c700 1 --2- 172.21.15.67:0/3294426907 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7e3c06ecc0 0x7f7e3c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:01:59.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.554+0000 7f7e5259c700 1 --2- 172.21.15.67:0/3294426907 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7e3c06ecc0 0x7f7e3c071180 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f7e440059d0 tx=0x7f7e4401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:01:59.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.559+0000 7f7e437fe700 1 -- 172.21.15.67:0/3294426907 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7e4805b260 con 0x7f7e54105350 2024-04-23T00:01:59.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:59 smithi134 ceph-mon[85630]: from='client.35916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:01:59.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:01:59 smithi134 ceph-mon[85630]: from='client.35920 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:01:59.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.865+0000 7f7e58e2c700 1 -- 172.21.15.67:0/3294426907 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7e5404f2e0 con 0x7f7e54105350 2024-04-23T00:01:59.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.865+0000 7f7e437fe700 1 -- 172.21.15.67:0/3294426907 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f7e4805adf0 con 0x7f7e54105350 2024-04-23T00:01:59.854 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-23T00:01:59.854 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-23T00:01:59.855 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-23T00:01:59.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-23T00:01:59.855 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-23T00:01:59.855 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-23T00:01:59.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-23T00:01:59.855 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-23T00:01:59.855 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-23T00:01:59.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-23T00:01:59.855 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-23T00:01:59.855 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 4 2024-04-23T00:01:59.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-23T00:01:59.855 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-23T00:01:59.856 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 14 2024-04-23T00:01:59.856 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-23T00:01:59.856 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-23T00:01:59.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.867+0000 7f7e58e2c700 1 -- 172.21.15.67:0/3294426907 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7e3c06ecc0 msgr2=0x7f7e3c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:59.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.867+0000 7f7e58e2c700 1 --2- 172.21.15.67:0/3294426907 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7e3c06ecc0 0x7f7e3c071180 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f7e440059d0 tx=0x7f7e4401a040 comp rx=0 tx=0).stop 2024-04-23T00:01:59.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.867+0000 7f7e58e2c700 1 -- 172.21.15.67:0/3294426907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e54105350 msgr2=0x7f7e54196ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:59.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.867+0000 7f7e58e2c700 1 --2- 172.21.15.67:0/3294426907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e54105350 0x7f7e54196ed0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f7e4800b4b0 tx=0x7f7e4800b7c0 comp rx=0 tx=0).stop 2024-04-23T00:01:59.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.867+0000 7f7e58e2c700 1 -- 172.21.15.67:0/3294426907 shutdown_connections 2024-04-23T00:01:59.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.867+0000 7f7e58e2c700 1 --2- 172.21.15.67:0/3294426907 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7e3c06ecc0 0x7f7e3c071180 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:59.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.867+0000 7f7e58e2c700 1 --2- 172.21.15.67:0/3294426907 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7e540ff340 0x7f7e54196990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:59.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.867+0000 7f7e58e2c700 1 --2- 172.21.15.67:0/3294426907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e54105350 0x7f7e54196ed0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:59.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.868+0000 7f7e58e2c700 1 -- 172.21.15.67:0/3294426907 >> 172.21.15.67:0/3294426907 conn(0x7f7e54074e30 msgr2=0x7f7e540fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:01:59.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.868+0000 7f7e58e2c700 1 -- 172.21.15.67:0/3294426907 shutdown_connections 2024-04-23T00:01:59.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.868+0000 7f7e58e2c700 1 -- 172.21.15.67:0/3294426907 wait complete. 2024-04-23T00:01:59.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.974+0000 7f98e3647700 1 -- 172.21.15.67:0/4256189903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98dc100750 msgr2=0x7f98dc1047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:59.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.974+0000 7f98e3647700 1 --2- 172.21.15.67:0/4256189903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98dc100750 0x7f98dc1047a0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f98d8009910 tx=0x7f98d8009c20 comp rx=0 tx=0).stop 2024-04-23T00:01:59.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.974+0000 7f98e3647700 1 -- 172.21.15.67:0/4256189903 shutdown_connections 2024-04-23T00:01:59.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.974+0000 7f98e3647700 1 --2- 172.21.15.67:0/4256189903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98dc100750 0x7f98dc1047a0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:59.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.974+0000 7f98e3647700 1 --2- 172.21.15.67:0/4256189903 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98dc0ffda0 0x7f98dc100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:59.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.974+0000 7f98e3647700 1 -- 172.21.15.67:0/4256189903 >> 172.21.15.67:0/4256189903 conn(0x7f98dc0fb650 msgr2=0x7f98dc0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:01:59.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.975+0000 7f98e3647700 1 -- 172.21.15.67:0/4256189903 shutdown_connections 2024-04-23T00:01:59.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.975+0000 7f98e3647700 1 -- 172.21.15.67:0/4256189903 wait complete. 2024-04-23T00:01:59.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.975+0000 7f98e3647700 1 Processor -- start 2024-04-23T00:01:59.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.975+0000 7f98e3647700 1 -- start start 2024-04-23T00:01:59.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.975+0000 7f98e3647700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98dc0ffda0 0x7f98dc19e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:01:59.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.975+0000 7f98e3647700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98dc100750 0x7f98dc19ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:01:59.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.975+0000 7f98e3647700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc19f4a0 con 0x7f98dc100750 2024-04-23T00:01:59.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.975+0000 7f98e3647700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc198950 con 0x7f98dc0ffda0 2024-04-23T00:01:59.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.976+0000 7f98e0be2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98dc100750 0x7f98dc19ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:01:59.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.976+0000 7f98e0be2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98dc100750 0x7f98dc19ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55390/0 (socket says 172.21.15.67:55390) 2024-04-23T00:01:59.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.976+0000 7f98e0be2700 1 -- 172.21.15.67:0/3533646526 learned_addr learned my addr 172.21.15.67:0/3533646526 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:01:59.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.976+0000 7f98e13e3700 1 --2- 172.21.15.67:0/3533646526 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98dc0ffda0 0x7f98dc19e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:01:59.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.976+0000 7f98e0be2700 1 -- 172.21.15.67:0/3533646526 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98dc0ffda0 msgr2=0x7f98dc19e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:01:59.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.976+0000 7f98e0be2700 1 --2- 172.21.15.67:0/3533646526 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98dc0ffda0 0x7f98dc19e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:01:59.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.976+0000 7f98e0be2700 1 -- 172.21.15.67:0/3533646526 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d80095f0 con 0x7f98dc100750 2024-04-23T00:01:59.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.976+0000 7f98e0be2700 1 --2- 172.21.15.67:0/3533646526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98dc100750 0x7f98dc19ee10 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f98d8000c00 tx=0x7f98d800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:01:59.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.976+0000 7f98d27fc700 1 -- 172.21.15.67:0/3533646526 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98d801d070 con 0x7f98dc100750 2024-04-23T00:01:59.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.976+0000 7f98d27fc700 1 -- 172.21.15.67:0/3533646526 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f98d8005dc0 con 0x7f98dc100750 2024-04-23T00:01:59.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.976+0000 7f98e3647700 1 -- 172.21.15.67:0/3533646526 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98dc198bd0 con 0x7f98dc100750 2024-04-23T00:01:59.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.976+0000 7f98d27fc700 1 -- 172.21.15.67:0/3533646526 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98d8017400 con 0x7f98dc100750 2024-04-23T00:01:59.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.976+0000 7f98e3647700 1 -- 172.21.15.67:0/3533646526 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98dc1990c0 con 0x7f98dc100750 2024-04-23T00:01:59.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.977+0000 7f98e3647700 1 -- 172.21.15.67:0/3533646526 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98dc103f20 con 0x7f98dc100750 2024-04-23T00:01:59.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.978+0000 7f98d27fc700 1 -- 172.21.15.67:0/3533646526 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f98d800f460 con 0x7f98dc100750 2024-04-23T00:01:59.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.978+0000 7f98d27fc700 1 --2- 172.21.15.67:0/3533646526 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f98c806ecc0 0x7f98c8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:01:59.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.978+0000 7f98d27fc700 1 -- 172.21.15.67:0/3533646526 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f98d8090460 con 0x7f98dc100750 2024-04-23T00:01:59.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.978+0000 7f98e13e3700 1 --2- 172.21.15.67:0/3533646526 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f98c806ecc0 0x7f98c8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:01:59.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.979+0000 7f98e13e3700 1 --2- 172.21.15.67:0/3533646526 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f98c806ecc0 0x7f98c8071180 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f98cc007be0 tx=0x7f98cc0073d0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:01:59.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:01:59.983+0000 7f98d27fc700 1 -- 172.21.15.67:0/3533646526 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f98d805cbd0 con 0x7f98dc100750 2024-04-23T00:02:00.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.239+0000 7f98e3647700 1 -- 172.21.15.67:0/3533646526 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f98dc199e60 con 0x7f98dc100750 2024-04-23T00:02:00.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.240+0000 7f98d27fc700 1 -- 172.21.15.67:0/3533646526 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1872 (secure 0 0 0) 0x7f98d805c760 con 0x7f98dc100750 2024-04-23T00:02:00.229 INFO:teuthology.orchestra.run.smithi067.stdout:e33 2024-04-23T00:02:00.229 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-23T00:02:00.229 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-23T00:02:00.229 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-23T00:02:00.229 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:02:00.229 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-23T00:02:00.229 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-23T00:02:00.230 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-23T00:02:00.230 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-23T00:02:00.230 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-23T00:02:00.230 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-23T00:02:00.230 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-23T00:02:00.230 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-23T00:02:00.230 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-23T00:02:00.230 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-23T00:02:00.230 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-23T00:02:00.230 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-23T00:02:00.230 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-23T00:02:00.230 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-23T00:02:00.230 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-23T00:02:00.230 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-23T00:02:00.231 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-23T00:02:00.231 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-23T00:02:00.231 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-23T00:02:00.231 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-23T00:02:00.231 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-23T00:02:00.231 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-23T00:02:00.231 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-23T00:02:00.231 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-23T00:02:00.231 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-23T00:02:00.231 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-23T00:02:00.231 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-23T00:02:00.231 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-23T00:02:00.231 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-23T00:02:00.231 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:02:00.231 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:02:00.231 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-23T00:02:00.232 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:02:00.232 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T00:02:00.232 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T00:02:00.232 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T00:02:00.232 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44851} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1849648622,v1:172.21.15.134:6827/1849648622] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T00:02:00.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.243+0000 7f98e3647700 1 -- 172.21.15.67:0/3533646526 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f98c806ecc0 msgr2=0x7f98c8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:02:00.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.243+0000 7f98e3647700 1 --2- 172.21.15.67:0/3533646526 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f98c806ecc0 0x7f98c8071180 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f98cc007be0 tx=0x7f98cc0073d0 comp rx=0 tx=0).stop 2024-04-23T00:02:00.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.243+0000 7f98e3647700 1 -- 172.21.15.67:0/3533646526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98dc100750 msgr2=0x7f98dc19ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:02:00.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.243+0000 7f98e3647700 1 --2- 172.21.15.67:0/3533646526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98dc100750 0x7f98dc19ee10 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f98d8000c00 tx=0x7f98d800bd70 comp rx=0 tx=0).stop 2024-04-23T00:02:00.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.243+0000 7f98e3647700 1 -- 172.21.15.67:0/3533646526 shutdown_connections 2024-04-23T00:02:00.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.243+0000 7f98e3647700 1 --2- 172.21.15.67:0/3533646526 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f98c806ecc0 0x7f98c8071180 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:00.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.243+0000 7f98e3647700 1 --2- 172.21.15.67:0/3533646526 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98dc0ffda0 0x7f98dc19e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:00.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.243+0000 7f98e3647700 1 --2- 172.21.15.67:0/3533646526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98dc100750 0x7f98dc19ee10 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:00.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.243+0000 7f98e3647700 1 -- 172.21.15.67:0/3533646526 >> 172.21.15.67:0/3533646526 conn(0x7f98dc0fb650 msgr2=0x7f98dc0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:02:00.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.243+0000 7f98e3647700 1 -- 172.21.15.67:0/3533646526 shutdown_connections 2024-04-23T00:02:00.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.243+0000 7f98e3647700 1 -- 172.21.15.67:0/3533646526 wait complete. 2024-04-23T00:02:00.233 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 33 2024-04-23T00:02:00.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.353+0000 7fc7639a5700 1 -- 172.21.15.67:0/544327776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc75c101e00 msgr2=0x7fc75c1021e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:02:00.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.353+0000 7fc7639a5700 1 --2- 172.21.15.67:0/544327776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc75c101e00 0x7fc75c1021e0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fc74c009910 tx=0x7fc74c009c20 comp rx=0 tx=0).stop 2024-04-23T00:02:00.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.353+0000 7fc7639a5700 1 -- 172.21.15.67:0/544327776 shutdown_connections 2024-04-23T00:02:00.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.353+0000 7fc7639a5700 1 --2- 172.21.15.67:0/544327776 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc75c102720 0x7fc75c10ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:00.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.353+0000 7fc7639a5700 1 --2- 172.21.15.67:0/544327776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc75c101e00 0x7fc75c1021e0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:00.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.353+0000 7fc7639a5700 1 -- 172.21.15.67:0/544327776 >> 172.21.15.67:0/544327776 conn(0x7fc75c0fb650 msgr2=0x7fc75c0fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:02:00.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.353+0000 7fc7639a5700 1 -- 172.21.15.67:0/544327776 shutdown_connections 2024-04-23T00:02:00.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.353+0000 7fc7639a5700 1 -- 172.21.15.67:0/544327776 wait complete. 2024-04-23T00:02:00.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.353+0000 7fc7639a5700 1 Processor -- start 2024-04-23T00:02:00.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.353+0000 7fc7639a5700 1 -- start start 2024-04-23T00:02:00.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.354+0000 7fc7639a5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc75c101e00 0x7fc75c19c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:02:00.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.354+0000 7fc7639a5700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc75c102720 0x7fc75c19cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:02:00.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.354+0000 7fc7639a5700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc75c19d260 con 0x7fc75c101e00 2024-04-23T00:02:00.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.354+0000 7fc7639a5700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc75c196710 con 0x7fc75c102720 2024-04-23T00:02:00.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.354+0000 7fc761741700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc75c101e00 0x7fc75c19c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:02:00.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.354+0000 7fc761741700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc75c101e00 0x7fc75c19c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55400/0 (socket says 172.21.15.67:55400) 2024-04-23T00:02:00.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.354+0000 7fc761741700 1 -- 172.21.15.67:0/3870611398 learned_addr learned my addr 172.21.15.67:0/3870611398 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:02:00.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.354+0000 7fc760f40700 1 --2- 172.21.15.67:0/3870611398 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc75c102720 0x7fc75c19cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:02:00.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.354+0000 7fc761741700 1 -- 172.21.15.67:0/3870611398 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc75c102720 msgr2=0x7fc75c19cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:02:00.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.354+0000 7fc761741700 1 --2- 172.21.15.67:0/3870611398 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc75c102720 0x7fc75c19cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:00.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.354+0000 7fc761741700 1 -- 172.21.15.67:0/3870611398 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc74c0095f0 con 0x7fc75c101e00 2024-04-23T00:02:00.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.354+0000 7fc761741700 1 --2- 172.21.15.67:0/3870611398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc75c101e00 0x7fc75c19c690 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fc74c005f50 tx=0x7fc74c00f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:02:00.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.354+0000 7fc7527fc700 1 -- 172.21.15.67:0/3870611398 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc74c01d070 con 0x7fc75c101e00 2024-04-23T00:02:00.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.354+0000 7fc7527fc700 1 -- 172.21.15.67:0/3870611398 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc74c00b610 con 0x7fc75c101e00 2024-04-23T00:02:00.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.354+0000 7fc7639a5700 1 -- 172.21.15.67:0/3870611398 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc75c196990 con 0x7fc75c101e00 2024-04-23T00:02:00.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.354+0000 7fc7527fc700 1 -- 172.21.15.67:0/3870611398 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc74c022470 con 0x7fc75c101e00 2024-04-23T00:02:00.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.354+0000 7fc7639a5700 1 -- 172.21.15.67:0/3870611398 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc75c196e80 con 0x7fc75c101e00 2024-04-23T00:02:00.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.355+0000 7fc7639a5700 1 -- 172.21.15.67:0/3870611398 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc75c108310 con 0x7fc75c101e00 2024-04-23T00:02:00.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.356+0000 7fc7527fc700 1 -- 172.21.15.67:0/3870611398 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc74c00fca0 con 0x7fc75c101e00 2024-04-23T00:02:00.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.356+0000 7fc7527fc700 1 --2- 172.21.15.67:0/3870611398 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc74806ecc0 0x7fc748071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:02:00.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.357+0000 7fc760f40700 1 --2- 172.21.15.67:0/3870611398 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc74806ecc0 0x7fc748071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:02:00.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.357+0000 7fc7527fc700 1 -- 172.21.15.67:0/3870611398 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fc74c090370 con 0x7fc75c101e00 2024-04-23T00:02:00.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.357+0000 7fc760f40700 1 --2- 172.21.15.67:0/3870611398 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc74806ecc0 0x7fc748071180 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fc75c197b80 tx=0x7fc758009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:02:00.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.361+0000 7fc7527fc700 1 -- 172.21.15.67:0/3870611398 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc74c05cb60 con 0x7fc75c101e00 2024-04-23T00:02:00.519 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:00 smithi134 ceph-mon[85630]: pgmap v1070: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:00.520 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:00 smithi134 ceph-mon[85630]: from='client.35924 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:02:00.520 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:00 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3294426907' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:02:00.520 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:00 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3533646526' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-23T00:02:00.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:00 smithi067 ceph-mon[112438]: pgmap v1070: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:00.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:00 smithi067 ceph-mon[112438]: from='client.35924 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:02:00.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:00 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3294426907' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:02:00.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:00 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3533646526' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-23T00:02:00.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.588+0000 7fc7639a5700 1 -- 172.21.15.67:0/3870611398 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc75c1970f0 con 0x7fc74806ecc0 2024-04-23T00:02:00.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.590+0000 7fc7527fc700 1 -- 172.21.15.67:0/3870611398 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc75c1970f0 con 0x7fc74806ecc0 2024-04-23T00:02:00.579 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-23T00:02:00.579 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-23T00:02:00.579 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-23T00:02:00.579 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-23T00:02:00.579 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-23T00:02:00.579 INFO:teuthology.orchestra.run.smithi067.stdout: "mds", 2024-04-23T00:02:00.579 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-23T00:02:00.579 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-23T00:02:00.580 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-23T00:02:00.580 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-23T00:02:00.580 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-23T00:02:00.580 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "16/23 daemons upgraded", 2024-04-23T00:02:00.580 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-23T00:02:00.580 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-23T00:02:00.580 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-23T00:02:00.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.592+0000 7fc7639a5700 1 -- 172.21.15.67:0/3870611398 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc74806ecc0 msgr2=0x7fc748071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:02:00.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.592+0000 7fc7639a5700 1 --2- 172.21.15.67:0/3870611398 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc74806ecc0 0x7fc748071180 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fc75c197b80 tx=0x7fc758009260 comp rx=0 tx=0).stop 2024-04-23T00:02:00.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.592+0000 7fc7639a5700 1 -- 172.21.15.67:0/3870611398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc75c101e00 msgr2=0x7fc75c19c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:02:00.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.592+0000 7fc7639a5700 1 --2- 172.21.15.67:0/3870611398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc75c101e00 0x7fc75c19c690 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fc74c005f50 tx=0x7fc74c00f740 comp rx=0 tx=0).stop 2024-04-23T00:02:00.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.592+0000 7fc7639a5700 1 -- 172.21.15.67:0/3870611398 shutdown_connections 2024-04-23T00:02:00.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.592+0000 7fc7639a5700 1 --2- 172.21.15.67:0/3870611398 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc74806ecc0 0x7fc748071180 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:00.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.592+0000 7fc7639a5700 1 --2- 172.21.15.67:0/3870611398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc75c101e00 0x7fc75c19c690 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:00.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.592+0000 7fc7639a5700 1 --2- 172.21.15.67:0/3870611398 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc75c102720 0x7fc75c19cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:00.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.592+0000 7fc7639a5700 1 -- 172.21.15.67:0/3870611398 >> 172.21.15.67:0/3870611398 conn(0x7fc75c0fb650 msgr2=0x7fc75c105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:02:00.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.592+0000 7fc7639a5700 1 -- 172.21.15.67:0/3870611398 shutdown_connections 2024-04-23T00:02:00.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.592+0000 7fc7639a5700 1 -- 172.21.15.67:0/3870611398 wait complete. 2024-04-23T00:02:00.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.698+0000 7fe2952bd700 1 -- 172.21.15.67:0/328805710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe290100750 msgr2=0x7fe2901047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:02:00.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.698+0000 7fe2952bd700 1 --2- 172.21.15.67:0/328805710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe290100750 0x7fe2901047a0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fe284009910 tx=0x7fe284009c20 comp rx=0 tx=0).stop 2024-04-23T00:02:00.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.698+0000 7fe2952bd700 1 -- 172.21.15.67:0/328805710 shutdown_connections 2024-04-23T00:02:00.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.698+0000 7fe2952bd700 1 --2- 172.21.15.67:0/328805710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe290100750 0x7fe2901047a0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:00.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.698+0000 7fe2952bd700 1 --2- 172.21.15.67:0/328805710 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe2900ffda0 0x7fe290100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:00.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.698+0000 7fe2952bd700 1 -- 172.21.15.67:0/328805710 >> 172.21.15.67:0/328805710 conn(0x7fe2900fb650 msgr2=0x7fe2900fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:02:00.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.698+0000 7fe2952bd700 1 -- 172.21.15.67:0/328805710 shutdown_connections 2024-04-23T00:02:00.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.698+0000 7fe2952bd700 1 -- 172.21.15.67:0/328805710 wait complete. 2024-04-23T00:02:00.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.699+0000 7fe2952bd700 1 Processor -- start 2024-04-23T00:02:00.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.699+0000 7fe2952bd700 1 -- start start 2024-04-23T00:02:00.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.699+0000 7fe2952bd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2900ffda0 0x7fe290101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:02:00.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.699+0000 7fe2952bd700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe290100750 0x7fe290101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:02:00.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.699+0000 7fe2952bd700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2901022c0 con 0x7fe2900ffda0 2024-04-23T00:02:00.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.699+0000 7fe2952bd700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe290102400 con 0x7fe290100750 2024-04-23T00:02:00.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.699+0000 7fe28effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2900ffda0 0x7fe290101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:02:00.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.699+0000 7fe28effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2900ffda0 0x7fe290101840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55418/0 (socket says 172.21.15.67:55418) 2024-04-23T00:02:00.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.700+0000 7fe28effd700 1 -- 172.21.15.67:0/1214280544 learned_addr learned my addr 172.21.15.67:0/1214280544 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:02:00.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.700+0000 7fe28e7fc700 1 --2- 172.21.15.67:0/1214280544 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe290100750 0x7fe290101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:02:00.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.700+0000 7fe28effd700 1 -- 172.21.15.67:0/1214280544 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe290100750 msgr2=0x7fe290101d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:02:00.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.700+0000 7fe28effd700 1 --2- 172.21.15.67:0/1214280544 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe290100750 0x7fe290101d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:00.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.700+0000 7fe28effd700 1 -- 172.21.15.67:0/1214280544 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2840095f0 con 0x7fe2900ffda0 2024-04-23T00:02:00.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.700+0000 7fe28effd700 1 --2- 172.21.15.67:0/1214280544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2900ffda0 0x7fe290101840 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fe28000d8d0 tx=0x7fe28000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:02:00.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.700+0000 7fe277fff700 1 -- 172.21.15.67:0/1214280544 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2800098a0 con 0x7fe2900ffda0 2024-04-23T00:02:00.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.700+0000 7fe277fff700 1 -- 172.21.15.67:0/1214280544 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe280010460 con 0x7fe2900ffda0 2024-04-23T00:02:00.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.700+0000 7fe2952bd700 1 -- 172.21.15.67:0/1214280544 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2901a2a70 con 0x7fe2900ffda0 2024-04-23T00:02:00.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.700+0000 7fe277fff700 1 -- 172.21.15.67:0/1214280544 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe28000f5f0 con 0x7fe2900ffda0 2024-04-23T00:02:00.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.700+0000 7fe2952bd700 1 -- 172.21.15.67:0/1214280544 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2901a2e20 con 0x7fe2900ffda0 2024-04-23T00:02:00.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.701+0000 7fe2952bd700 1 -- 172.21.15.67:0/1214280544 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2901091b0 con 0x7fe2900ffda0 2024-04-23T00:02:00.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.702+0000 7fe277fff700 1 -- 172.21.15.67:0/1214280544 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe280009a00 con 0x7fe2900ffda0 2024-04-23T00:02:00.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.702+0000 7fe277fff700 1 --2- 172.21.15.67:0/1214280544 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe27806ecc0 0x7fe278071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:02:00.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.702+0000 7fe277fff700 1 -- 172.21.15.67:0/1214280544 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fe28008eea0 con 0x7fe2900ffda0 2024-04-23T00:02:00.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.702+0000 7fe28e7fc700 1 --2- 172.21.15.67:0/1214280544 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe27806ecc0 0x7fe278071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:02:00.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.703+0000 7fe28e7fc700 1 --2- 172.21.15.67:0/1214280544 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe27806ecc0 0x7fe278071180 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fe284006010 tx=0x7fe28401a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:02:00.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:00.707+0000 7fe277fff700 1 -- 172.21.15.67:0/1214280544 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe280057ae0 con 0x7fe2900ffda0 2024-04-23T00:02:00.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:01.007+0000 7fe2952bd700 1 -- 172.21.15.67:0/1214280544 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe29004f2e0 con 0x7fe2900ffda0 2024-04-23T00:02:00.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:01.008+0000 7fe277fff700 1 -- 172.21.15.67:0/1214280544 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4505 (secure 0 0 0) 0x7fe28005b1a0 con 0x7fe2900ffda0 2024-04-23T00:02:00.997 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-23T00:02:00.997 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-23T00:02:00.997 INFO:teuthology.orchestra.run.smithi067.stdout: host smithi134 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-23T00:02:00.997 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-23T00:02:00.997 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-23T00:02:00.997 INFO:teuthology.orchestra.run.smithi067.stdout:Traceback (most recent call last): 2024-04-23T00:02:00.997 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-23T00:02:00.997 INFO:teuthology.orchestra.run.smithi067.stdout: return _run_code(code, main_globals, None, 2024-04-23T00:02:00.997 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-23T00:02:00.998 INFO:teuthology.orchestra.run.smithi067.stdout: exec(code, run_globals) 2024-04-23T00:02:00.998 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5642, in 2024-04-23T00:02:00.998 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5630, in main 2024-04-23T00:02:00.998 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 408, in _infer_config 2024-04-23T00:02:00.998 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 323, in _infer_fsid 2024-04-23T00:02:00.998 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 436, in _infer_image 2024-04-23T00:02:00.998 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 310, in _validate_fsid 2024-04-23T00:02:00.998 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 3380, in command_ceph_volume 2024-04-23T00:02:00.998 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-23T00:02:00.998 INFO:teuthology.orchestra.run.smithi067.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-23T00:02:00.998 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-23T00:02:00.998 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-23T00:02:00.998 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-23T00:02:00.998 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-23T00:02:00.999 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-23T00:02:00.999 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-23T00:02:00.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:01.010+0000 7fe2952bd700 1 -- 172.21.15.67:0/1214280544 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe27806ecc0 msgr2=0x7fe278071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:02:01.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:01.010+0000 7fe2952bd700 1 --2- 172.21.15.67:0/1214280544 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe27806ecc0 0x7fe278071180 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fe284006010 tx=0x7fe28401a040 comp rx=0 tx=0).stop 2024-04-23T00:02:01.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:01.010+0000 7fe2952bd700 1 -- 172.21.15.67:0/1214280544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2900ffda0 msgr2=0x7fe290101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:02:01.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:01.010+0000 7fe2952bd700 1 --2- 172.21.15.67:0/1214280544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2900ffda0 0x7fe290101840 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fe28000d8d0 tx=0x7fe28000dbe0 comp rx=0 tx=0).stop 2024-04-23T00:02:01.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:01.010+0000 7fe2952bd700 1 -- 172.21.15.67:0/1214280544 shutdown_connections 2024-04-23T00:02:01.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:01.010+0000 7fe2952bd700 1 --2- 172.21.15.67:0/1214280544 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe27806ecc0 0x7fe278071180 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:01.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:01.010+0000 7fe2952bd700 1 --2- 172.21.15.67:0/1214280544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2900ffda0 0x7fe290101840 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:01.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:01.010+0000 7fe2952bd700 1 --2- 172.21.15.67:0/1214280544 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe290100750 0x7fe290101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:01.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:01.010+0000 7fe2952bd700 1 -- 172.21.15.67:0/1214280544 >> 172.21.15.67:0/1214280544 conn(0x7fe2900fb650 msgr2=0x7fe290069150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:02:01.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:01.011+0000 7fe2952bd700 1 -- 172.21.15.67:0/1214280544 shutdown_connections 2024-04-23T00:02:01.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:01.011+0000 7fe2952bd700 1 -- 172.21.15.67:0/1214280544 wait complete. 2024-04-23T00:02:01.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:01 smithi067 ceph-mon[112438]: from='client.35936 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:02:01.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:01 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1214280544' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T00:02:01.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:01 smithi134 ceph-mon[85630]: from='client.35936 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:02:01.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:01 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1214280544' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T00:02:02.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:02 smithi067 ceph-mon[112438]: pgmap v1071: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:02.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:02 smithi134 ceph-mon[85630]: pgmap v1071: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:04.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:04 smithi067 ceph-mon[112438]: pgmap v1072: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:04.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:04 smithi134 ceph-mon[85630]: pgmap v1072: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:06.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:06 smithi067 ceph-mon[112438]: pgmap v1073: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:06.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:06 smithi134 ceph-mon[85630]: pgmap v1073: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:08.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:08 smithi067 ceph-mon[112438]: pgmap v1074: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:08.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:08 smithi134 ceph-mon[85630]: pgmap v1074: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:10.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:10 smithi067 ceph-mon[112438]: pgmap v1075: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:10.627 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:10 smithi134 ceph-mon[85630]: pgmap v1075: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:11.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:02:11.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:02:12.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:12 smithi067 ceph-mon[112438]: pgmap v1076: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:12.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:12 smithi134 ceph-mon[85630]: pgmap v1076: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:14.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:14 smithi067 ceph-mon[112438]: pgmap v1077: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:14.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:14 smithi134 ceph-mon[85630]: pgmap v1077: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:16.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:16 smithi134 ceph-mon[85630]: pgmap v1078: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:16.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:16 smithi067 ceph-mon[112438]: pgmap v1078: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:18.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:18 smithi067 ceph-mon[112438]: pgmap v1079: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:18.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:18 smithi134 ceph-mon[85630]: pgmap v1079: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:20.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:20 smithi134 ceph-mon[85630]: pgmap v1080: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:20.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:20 smithi067 ceph-mon[112438]: pgmap v1080: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:22.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:22 smithi134 ceph-mon[85630]: pgmap v1081: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:22.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:22 smithi067 ceph-mon[112438]: pgmap v1081: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:24.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:24 smithi134 ceph-mon[85630]: pgmap v1082: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:24.783 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:24 smithi067 ceph-mon[112438]: pgmap v1082: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:26.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:26 smithi134 ceph-mon[85630]: pgmap v1083: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:26.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:02:26.783 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:26 smithi067 ceph-mon[112438]: pgmap v1083: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:26.783 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:02:28.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:28 smithi067 ceph-mon[112438]: pgmap v1084: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:28.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:28 smithi134 ceph-mon[85630]: pgmap v1084: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:30.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:30 smithi134 ceph-mon[85630]: pgmap v1085: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:30.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:30 smithi067 ceph-mon[112438]: pgmap v1085: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:31.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.133+0000 7f3f39e76700 1 -- 172.21.15.67:0/2583036574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f340fff50 msgr2=0x7f3f34105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:02:31.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.133+0000 7f3f39e76700 1 --2- 172.21.15.67:0/2583036574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f340fff50 0x7f3f34105840 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f3f24009910 tx=0x7f3f24009c20 comp rx=0 tx=0).stop 2024-04-23T00:02:31.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.133+0000 7f3f39e76700 1 -- 172.21.15.67:0/2583036574 shutdown_connections 2024-04-23T00:02:31.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.133+0000 7f3f39e76700 1 --2- 172.21.15.67:0/2583036574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f340fff50 0x7f3f34105840 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:31.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.133+0000 7f3f39e76700 1 --2- 172.21.15.67:0/2583036574 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3f340ff5a0 0x7f3f340ff980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:31.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.133+0000 7f3f39e76700 1 -- 172.21.15.67:0/2583036574 >> 172.21.15.67:0/2583036574 conn(0x7f3f34074e30 msgr2=0x7f3f34075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:02:31.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.133+0000 7f3f39e76700 1 -- 172.21.15.67:0/2583036574 shutdown_connections 2024-04-23T00:02:31.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.133+0000 7f3f39e76700 1 -- 172.21.15.67:0/2583036574 wait complete. 2024-04-23T00:02:31.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.134+0000 7f3f39e76700 1 Processor -- start 2024-04-23T00:02:31.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.134+0000 7f3f39e76700 1 -- start start 2024-04-23T00:02:31.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.134+0000 7f3f39e76700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3f340ff5a0 0x7f3f3419c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:02:31.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.134+0000 7f3f39e76700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f340fff50 0x7f3f3419cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:02:31.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.134+0000 7f3f39e76700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3419d210 con 0x7f3f340fff50 2024-04-23T00:02:31.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.134+0000 7f3f39e76700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f341966c0 con 0x7f3f340ff5a0 2024-04-23T00:02:31.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.134+0000 7f3f32ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f340fff50 0x7f3f3419cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:02:31.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.134+0000 7f3f32ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f340fff50 0x7f3f3419cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37638/0 (socket says 172.21.15.67:37638) 2024-04-23T00:02:31.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.134+0000 7f3f32ffd700 1 -- 172.21.15.67:0/917332522 learned_addr learned my addr 172.21.15.67:0/917332522 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:02:31.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.134+0000 7f3f337fe700 1 --2- 172.21.15.67:0/917332522 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3f340ff5a0 0x7f3f3419c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:02:31.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.135+0000 7f3f32ffd700 1 -- 172.21.15.67:0/917332522 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3f340ff5a0 msgr2=0x7f3f3419c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:02:31.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.135+0000 7f3f32ffd700 1 --2- 172.21.15.67:0/917332522 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3f340ff5a0 0x7f3f3419c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:31.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.135+0000 7f3f32ffd700 1 -- 172.21.15.67:0/917332522 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f240095f0 con 0x7f3f340fff50 2024-04-23T00:02:31.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.135+0000 7f3f32ffd700 1 --2- 172.21.15.67:0/917332522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f340fff50 0x7f3f3419cb80 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f3f240092e0 tx=0x7f3f2400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:02:31.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.135+0000 7f3f30ff9700 1 -- 172.21.15.67:0/917332522 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f2401d070 con 0x7f3f340fff50 2024-04-23T00:02:31.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.135+0000 7f3f39e76700 1 -- 172.21.15.67:0/917332522 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f34196940 con 0x7f3f340fff50 2024-04-23T00:02:31.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.135+0000 7f3f30ff9700 1 -- 172.21.15.67:0/917332522 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3f24005dc0 con 0x7f3f340fff50 2024-04-23T00:02:31.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.135+0000 7f3f30ff9700 1 -- 172.21.15.67:0/917332522 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f24017400 con 0x7f3f340fff50 2024-04-23T00:02:31.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.135+0000 7f3f39e76700 1 -- 172.21.15.67:0/917332522 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f34196e30 con 0x7f3f340fff50 2024-04-23T00:02:31.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.136+0000 7f3f39e76700 1 -- 172.21.15.67:0/917332522 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f34109200 con 0x7f3f340fff50 2024-04-23T00:02:31.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.137+0000 7f3f30ff9700 1 -- 172.21.15.67:0/917332522 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3f2400f6f0 con 0x7f3f340fff50 2024-04-23T00:02:31.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.137+0000 7f3f30ff9700 1 --2- 172.21.15.67:0/917332522 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3f2006ecc0 0x7f3f20071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:02:31.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.137+0000 7f3f30ff9700 1 -- 172.21.15.67:0/917332522 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f3f240903c0 con 0x7f3f340fff50 2024-04-23T00:02:31.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.137+0000 7f3f337fe700 1 --2- 172.21.15.67:0/917332522 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3f2006ecc0 0x7f3f20071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:02:31.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.138+0000 7f3f337fe700 1 --2- 172.21.15.67:0/917332522 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3f2006ecc0 0x7f3f20071180 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c005950 tx=0x7f3f1c00b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:02:31.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.142+0000 7f3f30ff9700 1 -- 172.21.15.67:0/917332522 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f3f2405cb30 con 0x7f3f340fff50 2024-04-23T00:02:31.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.375+0000 7f3f39e76700 1 -- 172.21.15.67:0/917332522 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3f34197050 con 0x7f3f2006ecc0 2024-04-23T00:02:31.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.376+0000 7f3f30ff9700 1 -- 172.21.15.67:0/917332522 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3f34197050 con 0x7f3f2006ecc0 2024-04-23T00:02:31.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.378+0000 7f3f39e76700 1 -- 172.21.15.67:0/917332522 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3f2006ecc0 msgr2=0x7f3f20071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:02:31.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.378+0000 7f3f39e76700 1 --2- 172.21.15.67:0/917332522 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3f2006ecc0 0x7f3f20071180 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c005950 tx=0x7f3f1c00b410 comp rx=0 tx=0).stop 2024-04-23T00:02:31.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.378+0000 7f3f39e76700 1 -- 172.21.15.67:0/917332522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f340fff50 msgr2=0x7f3f3419cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:02:31.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.378+0000 7f3f39e76700 1 --2- 172.21.15.67:0/917332522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f340fff50 0x7f3f3419cb80 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f3f240092e0 tx=0x7f3f2400bd70 comp rx=0 tx=0).stop 2024-04-23T00:02:31.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.378+0000 7f3f39e76700 1 -- 172.21.15.67:0/917332522 shutdown_connections 2024-04-23T00:02:31.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.378+0000 7f3f39e76700 1 --2- 172.21.15.67:0/917332522 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3f2006ecc0 0x7f3f20071180 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:31.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.378+0000 7f3f39e76700 1 --2- 172.21.15.67:0/917332522 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3f340ff5a0 0x7f3f3419c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:31.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.378+0000 7f3f39e76700 1 --2- 172.21.15.67:0/917332522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f340fff50 0x7f3f3419cb80 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:31.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.378+0000 7f3f39e76700 1 -- 172.21.15.67:0/917332522 >> 172.21.15.67:0/917332522 conn(0x7f3f34074e30 msgr2=0x7f3f340fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:02:31.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.378+0000 7f3f39e76700 1 -- 172.21.15.67:0/917332522 shutdown_connections 2024-04-23T00:02:31.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.378+0000 7f3f39e76700 1 -- 172.21.15.67:0/917332522 wait complete. 2024-04-23T00:02:31.377 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-23T00:02:31.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.487+0000 7f327e9ca700 1 -- 172.21.15.67:0/13989996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3278100750 msgr2=0x7f32781047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:02:31.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.487+0000 7f327e9ca700 1 --2- 172.21.15.67:0/13989996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3278100750 0x7f32781047a0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f3260009910 tx=0x7f3260009c20 comp rx=0 tx=0).stop 2024-04-23T00:02:31.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.487+0000 7f327e9ca700 1 -- 172.21.15.67:0/13989996 shutdown_connections 2024-04-23T00:02:31.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.487+0000 7f327e9ca700 1 --2- 172.21.15.67:0/13989996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3278100750 0x7f32781047a0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:31.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.487+0000 7f327e9ca700 1 --2- 172.21.15.67:0/13989996 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f32780ffda0 0x7f3278100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:31.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.487+0000 7f327e9ca700 1 -- 172.21.15.67:0/13989996 >> 172.21.15.67:0/13989996 conn(0x7f32780fb650 msgr2=0x7f32780fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:02:31.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.487+0000 7f327e9ca700 1 -- 172.21.15.67:0/13989996 shutdown_connections 2024-04-23T00:02:31.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.487+0000 7f327e9ca700 1 -- 172.21.15.67:0/13989996 wait complete. 2024-04-23T00:02:31.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.488+0000 7f327e9ca700 1 Processor -- start 2024-04-23T00:02:31.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.488+0000 7f327e9ca700 1 -- start start 2024-04-23T00:02:31.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.488+0000 7f327e9ca700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32780ffda0 0x7f327819c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:02:31.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.488+0000 7f327e9ca700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3278100750 0x7f327819cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:02:31.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.488+0000 7f327e9ca700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327819d250 con 0x7f32780ffda0 2024-04-23T00:02:31.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.488+0000 7f327e9ca700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3278196700 con 0x7f3278100750 2024-04-23T00:02:31.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.488+0000 7f3277fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32780ffda0 0x7f327819c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:02:31.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.488+0000 7f3277fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32780ffda0 0x7f327819c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37656/0 (socket says 172.21.15.67:37656) 2024-04-23T00:02:31.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.488+0000 7f3277fff700 1 -- 172.21.15.67:0/1984009862 learned_addr learned my addr 172.21.15.67:0/1984009862 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:02:31.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.488+0000 7f32777fe700 1 --2- 172.21.15.67:0/1984009862 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3278100750 0x7f327819cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:02:31.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.488+0000 7f3277fff700 1 -- 172.21.15.67:0/1984009862 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3278100750 msgr2=0x7f327819cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:02:31.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.488+0000 7f3277fff700 1 --2- 172.21.15.67:0/1984009862 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3278100750 0x7f327819cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:31.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.488+0000 7f3277fff700 1 -- 172.21.15.67:0/1984009862 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32600095f0 con 0x7f32780ffda0 2024-04-23T00:02:31.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.488+0000 7f32777fe700 1 --2- 172.21.15.67:0/1984009862 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3278100750 0x7f327819cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T00:02:31.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.489+0000 7f3277fff700 1 --2- 172.21.15.67:0/1984009862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32780ffda0 0x7f327819c680 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f326800c8a0 tx=0x7f326800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:02:31.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.489+0000 7f32757fa700 1 -- 172.21.15.67:0/1984009862 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f326800cea0 con 0x7f32780ffda0 2024-04-23T00:02:31.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.489+0000 7f327e9ca700 1 -- 172.21.15.67:0/1984009862 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32781969e0 con 0x7f32780ffda0 2024-04-23T00:02:31.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.489+0000 7f32757fa700 1 -- 172.21.15.67:0/1984009862 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3268004830 con 0x7f32780ffda0 2024-04-23T00:02:31.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.489+0000 7f327e9ca700 1 -- 172.21.15.67:0/1984009862 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3278196f30 con 0x7f32780ffda0 2024-04-23T00:02:31.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.489+0000 7f32757fa700 1 -- 172.21.15.67:0/1984009862 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32680055b0 con 0x7f32780ffda0 2024-04-23T00:02:31.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.490+0000 7f327e9ca700 1 -- 172.21.15.67:0/1984009862 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f327804f2e0 con 0x7f32780ffda0 2024-04-23T00:02:31.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.490+0000 7f32757fa700 1 -- 172.21.15.67:0/1984009862 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3268007ad0 con 0x7f32780ffda0 2024-04-23T00:02:31.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.491+0000 7f32757fa700 1 --2- 172.21.15.67:0/1984009862 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f326406ecc0 0x7f3264071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:02:31.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.491+0000 7f32757fa700 1 -- 172.21.15.67:0/1984009862 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f326808ed40 con 0x7f32780ffda0 2024-04-23T00:02:31.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.491+0000 7f32777fe700 1 --2- 172.21.15.67:0/1984009862 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f326406ecc0 0x7f3264071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:02:31.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.491+0000 7f32777fe700 1 --2- 172.21.15.67:0/1984009862 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f326406ecc0 0x7f3264071180 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f326000b5c0 tx=0x7f3260005480 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:02:31.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.495+0000 7f32757fa700 1 -- 172.21.15.67:0/1984009862 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f326805b4b0 con 0x7f32780ffda0 2024-04-23T00:02:31.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.728+0000 7f327e9ca700 1 -- 172.21.15.67:0/1984009862 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3278197260 con 0x7f326406ecc0 2024-04-23T00:02:31.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.729+0000 7f32757fa700 1 -- 172.21.15.67:0/1984009862 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3278197260 con 0x7f326406ecc0 2024-04-23T00:02:31.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.731+0000 7f327e9ca700 1 -- 172.21.15.67:0/1984009862 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f326406ecc0 msgr2=0x7f3264071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:02:31.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.731+0000 7f327e9ca700 1 --2- 172.21.15.67:0/1984009862 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f326406ecc0 0x7f3264071180 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f326000b5c0 tx=0x7f3260005480 comp rx=0 tx=0).stop 2024-04-23T00:02:31.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.732+0000 7f327e9ca700 1 -- 172.21.15.67:0/1984009862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32780ffda0 msgr2=0x7f327819c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:02:31.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.732+0000 7f327e9ca700 1 --2- 172.21.15.67:0/1984009862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32780ffda0 0x7f327819c680 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f326800c8a0 tx=0x7f326800cc60 comp rx=0 tx=0).stop 2024-04-23T00:02:31.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.732+0000 7f327e9ca700 1 -- 172.21.15.67:0/1984009862 shutdown_connections 2024-04-23T00:02:31.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.732+0000 7f327e9ca700 1 --2- 172.21.15.67:0/1984009862 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f326406ecc0 0x7f3264071180 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:31.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.732+0000 7f327e9ca700 1 --2- 172.21.15.67:0/1984009862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32780ffda0 0x7f327819c680 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:31.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.732+0000 7f327e9ca700 1 --2- 172.21.15.67:0/1984009862 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3278100750 0x7f327819cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:31.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.732+0000 7f327e9ca700 1 -- 172.21.15.67:0/1984009862 >> 172.21.15.67:0/1984009862 conn(0x7f32780fb650 msgr2=0x7f3278103f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:02:31.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.732+0000 7f327e9ca700 1 -- 172.21.15.67:0/1984009862 shutdown_connections 2024-04-23T00:02:31.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.732+0000 7f327e9ca700 1 -- 172.21.15.67:0/1984009862 wait complete. 2024-04-23T00:02:31.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.838+0000 7f44cfc70700 1 -- 172.21.15.67:0/1521985283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44c80ff5a0 msgr2=0x7f44c80ff980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:02:31.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.838+0000 7f44cfc70700 1 --2- 172.21.15.67:0/1521985283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44c80ff5a0 0x7f44c80ff980 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f44c4009910 tx=0x7f44c4009c20 comp rx=0 tx=0).stop 2024-04-23T00:02:31.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.838+0000 7f44cfc70700 1 -- 172.21.15.67:0/1521985283 shutdown_connections 2024-04-23T00:02:31.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.838+0000 7f44cfc70700 1 --2- 172.21.15.67:0/1521985283 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44c80fff50 0x7f44c8105840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:31.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.838+0000 7f44cfc70700 1 --2- 172.21.15.67:0/1521985283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44c80ff5a0 0x7f44c80ff980 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:31.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.838+0000 7f44cfc70700 1 -- 172.21.15.67:0/1521985283 >> 172.21.15.67:0/1521985283 conn(0x7f44c8074e30 msgr2=0x7f44c8075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:02:31.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.839+0000 7f44cfc70700 1 -- 172.21.15.67:0/1521985283 shutdown_connections 2024-04-23T00:02:31.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.839+0000 7f44cfc70700 1 -- 172.21.15.67:0/1521985283 wait complete. 2024-04-23T00:02:31.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.839+0000 7f44cfc70700 1 Processor -- start 2024-04-23T00:02:31.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.839+0000 7f44cfc70700 1 -- start start 2024-04-23T00:02:31.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.839+0000 7f44cfc70700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44c80ff5a0 0x7f44c8100520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:02:31.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.839+0000 7f44cfc70700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44c80fff50 0x7f44c8100a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:02:31.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.839+0000 7f44cfc70700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c8100fa0 con 0x7f44c80ff5a0 2024-04-23T00:02:31.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.839+0000 7f44cfc70700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c81010e0 con 0x7f44c80fff50 2024-04-23T00:02:31.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.839+0000 7f44cda0c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44c80ff5a0 0x7f44c8100520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:02:31.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.840+0000 7f44cda0c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44c80ff5a0 0x7f44c8100520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37678/0 (socket says 172.21.15.67:37678) 2024-04-23T00:02:31.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.840+0000 7f44cda0c700 1 -- 172.21.15.67:0/1200098305 learned_addr learned my addr 172.21.15.67:0/1200098305 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:02:31.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.840+0000 7f44cd20b700 1 --2- 172.21.15.67:0/1200098305 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44c80fff50 0x7f44c8100a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:02:31.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.840+0000 7f44cda0c700 1 -- 172.21.15.67:0/1200098305 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44c80fff50 msgr2=0x7f44c8100a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:02:31.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.840+0000 7f44cda0c700 1 --2- 172.21.15.67:0/1200098305 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44c80fff50 0x7f44c8100a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:31.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.840+0000 7f44cda0c700 1 -- 172.21.15.67:0/1200098305 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c40095f0 con 0x7f44c80ff5a0 2024-04-23T00:02:31.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.840+0000 7f44cda0c700 1 --2- 172.21.15.67:0/1200098305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44c80ff5a0 0x7f44c8100520 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f44c4006010 tx=0x7f44c400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:02:31.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.840+0000 7f44beffd700 1 -- 172.21.15.67:0/1200098305 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44c401d070 con 0x7f44c80ff5a0 2024-04-23T00:02:31.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.840+0000 7f44beffd700 1 -- 172.21.15.67:0/1200098305 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f44c4004fb0 con 0x7f44c80ff5a0 2024-04-23T00:02:31.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.840+0000 7f44beffd700 1 -- 172.21.15.67:0/1200098305 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44c4022ea0 con 0x7f44c80ff5a0 2024-04-23T00:02:31.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.840+0000 7f44cfc70700 1 -- 172.21.15.67:0/1200098305 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44c8101360 con 0x7f44c80ff5a0 2024-04-23T00:02:31.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.840+0000 7f44cfc70700 1 -- 172.21.15.67:0/1200098305 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44c81a2f10 con 0x7f44c80ff5a0 2024-04-23T00:02:31.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.841+0000 7f44cfc70700 1 -- 172.21.15.67:0/1200098305 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44c8109270 con 0x7f44c80ff5a0 2024-04-23T00:02:31.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.842+0000 7f44beffd700 1 -- 172.21.15.67:0/1200098305 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f44c4022470 con 0x7f44c80ff5a0 2024-04-23T00:02:31.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.842+0000 7f44beffd700 1 --2- 172.21.15.67:0/1200098305 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f44b406ecc0 0x7f44b4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:02:31.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.842+0000 7f44beffd700 1 -- 172.21.15.67:0/1200098305 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f44c4090470 con 0x7f44c80ff5a0 2024-04-23T00:02:31.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.842+0000 7f44cd20b700 1 --2- 172.21.15.67:0/1200098305 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f44b406ecc0 0x7f44b4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:02:31.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.843+0000 7f44cd20b700 1 --2- 172.21.15.67:0/1200098305 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f44b406ecc0 0x7f44b4071180 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f44b8006fd0 tx=0x7f44b8008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:02:31.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:31.847+0000 7f44beffd700 1 -- 172.21.15.67:0/1200098305 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f44c4059300 con 0x7f44c80ff5a0 2024-04-23T00:02:32.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.065+0000 7f44cfc70700 1 -- 172.21.15.67:0/1200098305 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f44c8101b60 con 0x7f44b406ecc0 2024-04-23T00:02:32.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.076+0000 7f44beffd700 1 -- 172.21.15.67:0/1200098305 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f44c8101b60 con 0x7f44b406ecc0 2024-04-23T00:02:32.065 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-23T00:02:32.065 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (36m) 14m ago 47m 24.4M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-23T00:02:32.065 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (47m) 14m ago 47m 10.0M - 18.2.1 5be31c24972a df887f62291e 2024-04-23T00:02:32.065 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (46m) 14m ago 46m 10.0M - 18.2.1 5be31c24972a ad89190b5400 2024-04-23T00:02:32.066 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (33m) 14m ago 47m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-23T00:02:32.066 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (33m) 14m ago 46m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-23T00:02:32.066 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (35m) 14m ago 47m 81.2M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-23T00:02:32.066 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (29m) 14m ago 42m 19.4M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-23T00:02:32.066 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (29m) 14m ago 41m 20.3M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-23T00:02:32.066 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (29m) 14m ago 42m 20.1M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-23T00:02:32.066 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (14m) 14m ago 41m 14.0M - 19.0.0-3252-g43be0201 a0434699c1d7 46d3d7e86fee 2024-04-23T00:02:32.066 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (37m) 14m ago 48m 514M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-23T00:02:32.066 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (37m) 14m ago 46m 459M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-23T00:02:32.066 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (34m) 14m ago 49m 74.2M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-23T00:02:32.066 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (34m) 14m ago 46m 54.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-23T00:02:32.066 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (36m) 14m ago 47m 15.2M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-23T00:02:32.067 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (36m) 14m ago 46m 16.0M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-23T00:02:32.067 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (33m) 14m ago 45m 102M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-23T00:02:32.067 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (32m) 14m ago 45m 128M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-23T00:02:32.067 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (31m) 14m ago 44m 83.8M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-23T00:02:32.067 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (31m) 14m ago 44m 101M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-23T00:02:32.067 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (30m) 14m ago 44m 86.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-23T00:02:32.067 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (30m) 14m ago 43m 81.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-23T00:02:32.067 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (36m) 14m ago 47m 78.7M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-23T00:02:32.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.078+0000 7f44cfc70700 1 -- 172.21.15.67:0/1200098305 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f44b406ecc0 msgr2=0x7f44b4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:02:32.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.078+0000 7f44cfc70700 1 --2- 172.21.15.67:0/1200098305 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f44b406ecc0 0x7f44b4071180 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f44b8006fd0 tx=0x7f44b8008040 comp rx=0 tx=0).stop 2024-04-23T00:02:32.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.078+0000 7f44cfc70700 1 -- 172.21.15.67:0/1200098305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44c80ff5a0 msgr2=0x7f44c8100520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:02:32.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.078+0000 7f44cfc70700 1 --2- 172.21.15.67:0/1200098305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44c80ff5a0 0x7f44c8100520 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f44c4006010 tx=0x7f44c400be20 comp rx=0 tx=0).stop 2024-04-23T00:02:32.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.078+0000 7f44cfc70700 1 -- 172.21.15.67:0/1200098305 shutdown_connections 2024-04-23T00:02:32.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.078+0000 7f44cfc70700 1 --2- 172.21.15.67:0/1200098305 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f44b406ecc0 0x7f44b4071180 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:32.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.078+0000 7f44cfc70700 1 --2- 172.21.15.67:0/1200098305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44c80ff5a0 0x7f44c8100520 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:32.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.078+0000 7f44cfc70700 1 --2- 172.21.15.67:0/1200098305 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44c80fff50 0x7f44c8100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:32.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.078+0000 7f44cfc70700 1 -- 172.21.15.67:0/1200098305 >> 172.21.15.67:0/1200098305 conn(0x7f44c8074e30 msgr2=0x7f44c80fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:02:32.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.078+0000 7f44cfc70700 1 -- 172.21.15.67:0/1200098305 shutdown_connections 2024-04-23T00:02:32.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.078+0000 7f44cfc70700 1 -- 172.21.15.67:0/1200098305 wait complete. 2024-04-23T00:02:32.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.183+0000 7f24d45b8700 1 -- 172.21.15.67:0/3777684107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24cc100ea0 msgr2=0x7f24cc101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:02:32.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.183+0000 7f24d45b8700 1 --2- 172.21.15.67:0/3777684107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24cc100ea0 0x7f24cc101280 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f24c8009910 tx=0x7f24c8009c20 comp rx=0 tx=0).stop 2024-04-23T00:02:32.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.183+0000 7f24d45b8700 1 -- 172.21.15.67:0/3777684107 shutdown_connections 2024-04-23T00:02:32.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.183+0000 7f24d45b8700 1 --2- 172.21.15.67:0/3777684107 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f24cc101850 0x7f24cc1058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:32.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.183+0000 7f24d45b8700 1 --2- 172.21.15.67:0/3777684107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24cc100ea0 0x7f24cc101280 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:32.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.183+0000 7f24d45b8700 1 -- 172.21.15.67:0/3777684107 >> 172.21.15.67:0/3777684107 conn(0x7f24cc0fc750 msgr2=0x7f24cc0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:02:32.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.184+0000 7f24d45b8700 1 -- 172.21.15.67:0/3777684107 shutdown_connections 2024-04-23T00:02:32.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.184+0000 7f24d45b8700 1 -- 172.21.15.67:0/3777684107 wait complete. 2024-04-23T00:02:32.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.184+0000 7f24d45b8700 1 Processor -- start 2024-04-23T00:02:32.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.184+0000 7f24d45b8700 1 -- start start 2024-04-23T00:02:32.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.184+0000 7f24d45b8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24cc100ea0 0x7f24cc19e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:02:32.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.184+0000 7f24d45b8700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f24cc101850 0x7f24cc19ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:02:32.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.184+0000 7f24d45b8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24cc19f4d0 con 0x7f24cc100ea0 2024-04-23T00:02:32.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.184+0000 7f24d45b8700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24cc198980 con 0x7f24cc101850 2024-04-23T00:02:32.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.184+0000 7f24d2354700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24cc100ea0 0x7f24cc19e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:02:32.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.185+0000 7f24d2354700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24cc100ea0 0x7f24cc19e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37686/0 (socket says 172.21.15.67:37686) 2024-04-23T00:02:32.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.185+0000 7f24d2354700 1 -- 172.21.15.67:0/1714486723 learned_addr learned my addr 172.21.15.67:0/1714486723 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:02:32.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.185+0000 7f24d1b53700 1 --2- 172.21.15.67:0/1714486723 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f24cc101850 0x7f24cc19ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:02:32.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.185+0000 7f24d2354700 1 -- 172.21.15.67:0/1714486723 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f24cc101850 msgr2=0x7f24cc19ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:02:32.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.185+0000 7f24d2354700 1 --2- 172.21.15.67:0/1714486723 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f24cc101850 0x7f24cc19ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:32.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.185+0000 7f24d2354700 1 -- 172.21.15.67:0/1714486723 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c80095f0 con 0x7f24cc100ea0 2024-04-23T00:02:32.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.185+0000 7f24d2354700 1 --2- 172.21.15.67:0/1714486723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24cc100ea0 0x7f24cc19e900 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f24c8006010 tx=0x7f24c800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:02:32.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.185+0000 7f24c37fe700 1 -- 172.21.15.67:0/1714486723 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f24c801d070 con 0x7f24cc100ea0 2024-04-23T00:02:32.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.185+0000 7f24c37fe700 1 -- 172.21.15.67:0/1714486723 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f24c8005dc0 con 0x7f24cc100ea0 2024-04-23T00:02:32.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.185+0000 7f24d45b8700 1 -- 172.21.15.67:0/1714486723 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24cc198c00 con 0x7f24cc100ea0 2024-04-23T00:02:32.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.185+0000 7f24c37fe700 1 -- 172.21.15.67:0/1714486723 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f24c8017400 con 0x7f24cc100ea0 2024-04-23T00:02:32.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.185+0000 7f24d45b8700 1 -- 172.21.15.67:0/1714486723 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24cc1990f0 con 0x7f24cc100ea0 2024-04-23T00:02:32.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.186+0000 7f24d45b8700 1 -- 172.21.15.67:0/1714486723 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24cc105020 con 0x7f24cc100ea0 2024-04-23T00:02:32.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.187+0000 7f24c37fe700 1 -- 172.21.15.67:0/1714486723 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f24c8003a60 con 0x7f24cc100ea0 2024-04-23T00:02:32.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.187+0000 7f24c37fe700 1 --2- 172.21.15.67:0/1714486723 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f24b806ecc0 0x7f24b8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:02:32.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.187+0000 7f24c37fe700 1 -- 172.21.15.67:0/1714486723 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f24c8090460 con 0x7f24cc100ea0 2024-04-23T00:02:32.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.187+0000 7f24d1b53700 1 --2- 172.21.15.67:0/1714486723 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f24b806ecc0 0x7f24b8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:02:32.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.188+0000 7f24d1b53700 1 --2- 172.21.15.67:0/1714486723 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f24b806ecc0 0x7f24b8071180 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f24cc0fce90 tx=0x7f24bc00b3c0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:02:32.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.192+0000 7f24c37fe700 1 -- 172.21.15.67:0/1714486723 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f24c805cbd0 con 0x7f24cc100ea0 2024-04-23T00:02:32.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.494+0000 7f24d45b8700 1 -- 172.21.15.67:0/1714486723 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f24cc199ef0 con 0x7f24cc100ea0 2024-04-23T00:02:32.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.494+0000 7f24c37fe700 1 -- 172.21.15.67:0/1714486723 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f24c805c760 con 0x7f24cc100ea0 2024-04-23T00:02:32.483 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-23T00:02:32.484 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-23T00:02:32.484 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-23T00:02:32.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-23T00:02:32.484 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-23T00:02:32.484 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-23T00:02:32.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-23T00:02:32.484 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-23T00:02:32.484 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-23T00:02:32.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-23T00:02:32.484 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-23T00:02:32.485 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 4 2024-04-23T00:02:32.485 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-23T00:02:32.485 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-23T00:02:32.485 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 14 2024-04-23T00:02:32.485 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-23T00:02:32.485 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-23T00:02:32.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.496+0000 7f24d45b8700 1 -- 172.21.15.67:0/1714486723 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f24b806ecc0 msgr2=0x7f24b8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:02:32.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.496+0000 7f24d45b8700 1 --2- 172.21.15.67:0/1714486723 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f24b806ecc0 0x7f24b8071180 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f24cc0fce90 tx=0x7f24bc00b3c0 comp rx=0 tx=0).stop 2024-04-23T00:02:32.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.496+0000 7f24d45b8700 1 -- 172.21.15.67:0/1714486723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24cc100ea0 msgr2=0x7f24cc19e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:02:32.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.496+0000 7f24d45b8700 1 --2- 172.21.15.67:0/1714486723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24cc100ea0 0x7f24cc19e900 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f24c8006010 tx=0x7f24c800bd70 comp rx=0 tx=0).stop 2024-04-23T00:02:32.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.496+0000 7f24d45b8700 1 -- 172.21.15.67:0/1714486723 shutdown_connections 2024-04-23T00:02:32.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.496+0000 7f24d45b8700 1 --2- 172.21.15.67:0/1714486723 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f24b806ecc0 0x7f24b8071180 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:32.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.496+0000 7f24d45b8700 1 --2- 172.21.15.67:0/1714486723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24cc100ea0 0x7f24cc19e900 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:32.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.496+0000 7f24d45b8700 1 --2- 172.21.15.67:0/1714486723 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f24cc101850 0x7f24cc19ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:32.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.496+0000 7f24d45b8700 1 -- 172.21.15.67:0/1714486723 >> 172.21.15.67:0/1714486723 conn(0x7f24cc0fc750 msgr2=0x7f24cc0fddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:02:32.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.496+0000 7f24d45b8700 1 -- 172.21.15.67:0/1714486723 shutdown_connections 2024-04-23T00:02:32.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.496+0000 7f24d45b8700 1 -- 172.21.15.67:0/1714486723 wait complete. 2024-04-23T00:02:32.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.603+0000 7fe99a792700 1 -- 172.21.15.67:0/1421463923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe994073a50 msgr2=0x7fe994111710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:02:32.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.603+0000 7fe99a792700 1 --2- 172.21.15.67:0/1421463923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe994073a50 0x7fe994111710 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fe988009910 tx=0x7fe988009c20 comp rx=0 tx=0).stop 2024-04-23T00:02:32.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.603+0000 7fe99a792700 1 -- 172.21.15.67:0/1421463923 shutdown_connections 2024-04-23T00:02:32.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.603+0000 7fe99a792700 1 --2- 172.21.15.67:0/1421463923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe994073a50 0x7fe994111710 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:32.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.603+0000 7fe99a792700 1 --2- 172.21.15.67:0/1421463923 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe994073130 0x7fe994073510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:32.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.603+0000 7fe99a792700 1 -- 172.21.15.67:0/1421463923 >> 172.21.15.67:0/1421463923 conn(0x7fe9940fc750 msgr2=0x7fe9940feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:02:32.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.603+0000 7fe99a792700 1 -- 172.21.15.67:0/1421463923 shutdown_connections 2024-04-23T00:02:32.592 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:32 smithi067 ceph-mon[112438]: pgmap v1086: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:32.593 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:32 smithi067 ceph-mon[112438]: from='client.35944 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:02:32.593 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:32 smithi067 ceph-mon[112438]: from='client.35948 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:02:32.593 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:32 smithi067 ceph-mon[112438]: from='client.35952 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:02:32.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.603+0000 7fe99a792700 1 -- 172.21.15.67:0/1421463923 wait complete. 2024-04-23T00:02:32.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.603+0000 7fe99a792700 1 Processor -- start 2024-04-23T00:02:32.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.604+0000 7fe99a792700 1 -- start start 2024-04-23T00:02:32.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.604+0000 7fe99a792700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe994073130 0x7fe99419ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:02:32.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.604+0000 7fe99a792700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe994073a50 0x7fe99419d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:02:32.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.604+0000 7fe99a792700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe99419daa0 con 0x7fe994073130 2024-04-23T00:02:32.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.604+0000 7fe99a792700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9941a1830 con 0x7fe994073a50 2024-04-23T00:02:32.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.604+0000 7fe993fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe994073130 0x7fe99419ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:02:32.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.604+0000 7fe993fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe994073130 0x7fe99419ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37712/0 (socket says 172.21.15.67:37712) 2024-04-23T00:02:32.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.604+0000 7fe993fff700 1 -- 172.21.15.67:0/3659659426 learned_addr learned my addr 172.21.15.67:0/3659659426 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:02:32.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.604+0000 7fe9937fe700 1 --2- 172.21.15.67:0/3659659426 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe994073a50 0x7fe99419d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:02:32.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.604+0000 7fe993fff700 1 -- 172.21.15.67:0/3659659426 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe994073a50 msgr2=0x7fe99419d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:02:32.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.604+0000 7fe993fff700 1 --2- 172.21.15.67:0/3659659426 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe994073a50 0x7fe99419d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:32.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.604+0000 7fe993fff700 1 -- 172.21.15.67:0/3659659426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9880095f0 con 0x7fe994073130 2024-04-23T00:02:32.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.605+0000 7fe993fff700 1 --2- 172.21.15.67:0/3659659426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe994073130 0x7fe99419ce80 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fe98400d8d0 tx=0x7fe98400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:02:32.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.605+0000 7fe9917fa700 1 -- 172.21.15.67:0/3659659426 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9840098a0 con 0x7fe994073130 2024-04-23T00:02:32.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.605+0000 7fe9917fa700 1 -- 172.21.15.67:0/3659659426 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe984010460 con 0x7fe994073130 2024-04-23T00:02:32.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.605+0000 7fe99a792700 1 -- 172.21.15.67:0/3659659426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9941a1b10 con 0x7fe994073130 2024-04-23T00:02:32.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.605+0000 7fe9917fa700 1 -- 172.21.15.67:0/3659659426 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe98400f5f0 con 0x7fe994073130 2024-04-23T00:02:32.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.605+0000 7fe99a792700 1 -- 172.21.15.67:0/3659659426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9941a2060 con 0x7fe994073130 2024-04-23T00:02:32.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.606+0000 7fe99a792700 1 -- 172.21.15.67:0/3659659426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe99410ee90 con 0x7fe994073130 2024-04-23T00:02:32.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.606+0000 7fe9917fa700 1 -- 172.21.15.67:0/3659659426 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe984009a00 con 0x7fe994073130 2024-04-23T00:02:32.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.607+0000 7fe9917fa700 1 --2- 172.21.15.67:0/3659659426 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe97c06ecc0 0x7fe97c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:02:32.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.607+0000 7fe9917fa700 1 -- 172.21.15.67:0/3659659426 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fe98408ee80 con 0x7fe994073130 2024-04-23T00:02:32.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.607+0000 7fe9937fe700 1 --2- 172.21.15.67:0/3659659426 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe97c06ecc0 0x7fe97c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:02:32.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.607+0000 7fe9937fe700 1 --2- 172.21.15.67:0/3659659426 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe97c06ecc0 0x7fe97c071180 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fe99419e4a0 tx=0x7fe98801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:02:32.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.612+0000 7fe9917fa700 1 -- 172.21.15.67:0/3659659426 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe984057ac0 con 0x7fe994073130 2024-04-23T00:02:32.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:32 smithi134 ceph-mon[85630]: pgmap v1086: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:32.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:32 smithi134 ceph-mon[85630]: from='client.35944 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:02:32.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:32 smithi134 ceph-mon[85630]: from='client.35948 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:02:32.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:32 smithi134 ceph-mon[85630]: from='client.35952 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:02:32.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.870+0000 7fe99a792700 1 -- 172.21.15.67:0/3659659426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe99404f2e0 con 0x7fe994073130 2024-04-23T00:02:32.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.870+0000 7fe9917fa700 1 -- 172.21.15.67:0/3659659426 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1872 (secure 0 0 0) 0x7fe98405b180 con 0x7fe994073130 2024-04-23T00:02:32.859 INFO:teuthology.orchestra.run.smithi067.stdout:e33 2024-04-23T00:02:32.859 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-23T00:02:32.860 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-23T00:02:32.860 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-23T00:02:32.860 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:02:32.860 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-23T00:02:32.860 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-23T00:02:32.860 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-23T00:02:32.860 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-23T00:02:32.860 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-23T00:02:32.860 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-23T00:02:32.860 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-23T00:02:32.860 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-23T00:02:32.860 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-23T00:02:32.860 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-23T00:02:32.860 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-23T00:02:32.860 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-23T00:02:32.861 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-23T00:02:32.861 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-23T00:02:32.861 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-23T00:02:32.861 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-23T00:02:32.861 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-23T00:02:32.861 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-23T00:02:32.861 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-23T00:02:32.861 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-23T00:02:32.861 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-23T00:02:32.861 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-23T00:02:32.861 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-23T00:02:32.861 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-23T00:02:32.861 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-23T00:02:32.861 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-23T00:02:32.862 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-23T00:02:32.862 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-23T00:02:32.862 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-23T00:02:32.862 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:02:32.862 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:02:32.862 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-23T00:02:32.862 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:02:32.862 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T00:02:32.862 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T00:02:32.862 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T00:02:32.862 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44851} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1849648622,v1:172.21.15.134:6827/1849648622] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T00:02:32.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.872+0000 7fe99a792700 1 -- 172.21.15.67:0/3659659426 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe97c06ecc0 msgr2=0x7fe97c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:02:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.872+0000 7fe99a792700 1 --2- 172.21.15.67:0/3659659426 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe97c06ecc0 0x7fe97c071180 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fe99419e4a0 tx=0x7fe98801a040 comp rx=0 tx=0).stop 2024-04-23T00:02:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.872+0000 7fe99a792700 1 -- 172.21.15.67:0/3659659426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe994073130 msgr2=0x7fe99419ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:02:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.872+0000 7fe99a792700 1 --2- 172.21.15.67:0/3659659426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe994073130 0x7fe99419ce80 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fe98400d8d0 tx=0x7fe98400dbe0 comp rx=0 tx=0).stop 2024-04-23T00:02:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.873+0000 7fe99a792700 1 -- 172.21.15.67:0/3659659426 shutdown_connections 2024-04-23T00:02:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.873+0000 7fe99a792700 1 --2- 172.21.15.67:0/3659659426 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe97c06ecc0 0x7fe97c071180 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.873+0000 7fe99a792700 1 --2- 172.21.15.67:0/3659659426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe994073130 0x7fe99419ce80 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.873+0000 7fe99a792700 1 --2- 172.21.15.67:0/3659659426 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe994073a50 0x7fe99419d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.873+0000 7fe99a792700 1 -- 172.21.15.67:0/3659659426 >> 172.21.15.67:0/3659659426 conn(0x7fe9940fc750 msgr2=0x7fe994103220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:02:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.873+0000 7fe99a792700 1 -- 172.21.15.67:0/3659659426 shutdown_connections 2024-04-23T00:02:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.873+0000 7fe99a792700 1 -- 172.21.15.67:0/3659659426 wait complete. 2024-04-23T00:02:32.864 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 33 2024-04-23T00:02:32.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.980+0000 7fdde9f6c700 1 -- 172.21.15.67:0/2575509445 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdde4101850 msgr2=0x7fdde41058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:02:32.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.980+0000 7fdde9f6c700 1 --2- 172.21.15.67:0/2575509445 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdde4101850 0x7fdde41058a0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fddcc009910 tx=0x7fddcc009c20 comp rx=0 tx=0).stop 2024-04-23T00:02:32.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.980+0000 7fdde9f6c700 1 -- 172.21.15.67:0/2575509445 shutdown_connections 2024-04-23T00:02:32.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.980+0000 7fdde9f6c700 1 --2- 172.21.15.67:0/2575509445 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdde4101850 0x7fdde41058a0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:32.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.980+0000 7fdde9f6c700 1 --2- 172.21.15.67:0/2575509445 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdde4100ea0 0x7fdde4101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:32.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.980+0000 7fdde9f6c700 1 -- 172.21.15.67:0/2575509445 >> 172.21.15.67:0/2575509445 conn(0x7fdde40fc750 msgr2=0x7fdde40feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:02:32.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.980+0000 7fdde9f6c700 1 -- 172.21.15.67:0/2575509445 shutdown_connections 2024-04-23T00:02:32.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.980+0000 7fdde9f6c700 1 -- 172.21.15.67:0/2575509445 wait complete. 2024-04-23T00:02:32.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.981+0000 7fdde9f6c700 1 Processor -- start 2024-04-23T00:02:32.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.981+0000 7fdde9f6c700 1 -- start start 2024-04-23T00:02:32.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.981+0000 7fdde9f6c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdde4100ea0 0x7fdde4073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:02:32.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.981+0000 7fdde9f6c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdde4101850 0x7fdde40735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:02:32.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.981+0000 7fdde9f6c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde4073b10 con 0x7fdde4101850 2024-04-23T00:02:32.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.981+0000 7fdde9f6c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde4073c50 con 0x7fdde4100ea0 2024-04-23T00:02:32.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.982+0000 7fdddaffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdde4101850 0x7fdde40735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:02:32.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.982+0000 7fdddaffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdde4101850 0x7fdde40735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37742/0 (socket says 172.21.15.67:37742) 2024-04-23T00:02:32.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.982+0000 7fdddaffd700 1 -- 172.21.15.67:0/1586475549 learned_addr learned my addr 172.21.15.67:0/1586475549 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:02:32.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.982+0000 7fdde37fe700 1 --2- 172.21.15.67:0/1586475549 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdde4100ea0 0x7fdde4073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:02:32.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.982+0000 7fdddaffd700 1 -- 172.21.15.67:0/1586475549 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdde4100ea0 msgr2=0x7fdde4073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:02:32.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.982+0000 7fdddaffd700 1 --2- 172.21.15.67:0/1586475549 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdde4100ea0 0x7fdde4073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:32.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.982+0000 7fdddaffd700 1 -- 172.21.15.67:0/1586475549 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddcc0095f0 con 0x7fdde4101850 2024-04-23T00:02:32.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.982+0000 7fdddaffd700 1 --2- 172.21.15.67:0/1586475549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdde4101850 0x7fdde40735d0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fddcc0092e0 tx=0x7fddcc00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:02:32.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.982+0000 7fdde17fa700 1 -- 172.21.15.67:0/1586475549 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fddcc01d070 con 0x7fdde4101850 2024-04-23T00:02:32.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.982+0000 7fdde17fa700 1 -- 172.21.15.67:0/1586475549 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fddcc005dc0 con 0x7fdde4101850 2024-04-23T00:02:32.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.982+0000 7fdde9f6c700 1 -- 172.21.15.67:0/1586475549 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdde4073ed0 con 0x7fdde4101850 2024-04-23T00:02:32.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.982+0000 7fdde17fa700 1 -- 172.21.15.67:0/1586475549 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fddcc017400 con 0x7fdde4101850 2024-04-23T00:02:32.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.982+0000 7fdde9f6c700 1 -- 172.21.15.67:0/1586475549 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdde41a2e30 con 0x7fdde4101850 2024-04-23T00:02:32.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.983+0000 7fdde9f6c700 1 -- 172.21.15.67:0/1586475549 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdde41092a0 con 0x7fdde4101850 2024-04-23T00:02:32.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.984+0000 7fdde17fa700 1 -- 172.21.15.67:0/1586475549 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fddcc00f6f0 con 0x7fdde4101850 2024-04-23T00:02:32.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.984+0000 7fdde17fa700 1 --2- 172.21.15.67:0/1586475549 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fddd006ecc0 0x7fddd0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:02:32.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.985+0000 7fdde17fa700 1 -- 172.21.15.67:0/1586475549 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fddcc0903c0 con 0x7fdde4101850 2024-04-23T00:02:32.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.985+0000 7fdde37fe700 1 --2- 172.21.15.67:0/1586475549 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fddd006ecc0 0x7fddd0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:02:32.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.985+0000 7fdde37fe700 1 --2- 172.21.15.67:0/1586475549 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fddd006ecc0 0x7fddd0071180 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fddd4007be0 tx=0x7fddd40073d0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:02:32.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:32.990+0000 7fdde17fa700 1 -- 172.21.15.67:0/1586475549 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fddcc059000 con 0x7fdde4101850 2024-04-23T00:02:33.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.219+0000 7fdde9f6c700 1 -- 172.21.15.67:0/1586475549 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdde40611d0 con 0x7fddd006ecc0 2024-04-23T00:02:33.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.220+0000 7fdde17fa700 1 -- 172.21.15.67:0/1586475549 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fdde40611d0 con 0x7fddd006ecc0 2024-04-23T00:02:33.209 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-23T00:02:33.209 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-23T00:02:33.209 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-23T00:02:33.209 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-23T00:02:33.209 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-23T00:02:33.209 INFO:teuthology.orchestra.run.smithi067.stdout: "mds", 2024-04-23T00:02:33.209 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-23T00:02:33.209 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-23T00:02:33.209 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-23T00:02:33.210 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-23T00:02:33.210 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-23T00:02:33.210 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "16/23 daemons upgraded", 2024-04-23T00:02:33.210 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-23T00:02:33.210 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-23T00:02:33.210 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-23T00:02:33.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.222+0000 7fdde9f6c700 1 -- 172.21.15.67:0/1586475549 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fddd006ecc0 msgr2=0x7fddd0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:02:33.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.222+0000 7fdde9f6c700 1 --2- 172.21.15.67:0/1586475549 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fddd006ecc0 0x7fddd0071180 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fddd4007be0 tx=0x7fddd40073d0 comp rx=0 tx=0).stop 2024-04-23T00:02:33.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.222+0000 7fdde9f6c700 1 -- 172.21.15.67:0/1586475549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdde4101850 msgr2=0x7fdde40735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:02:33.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.222+0000 7fdde9f6c700 1 --2- 172.21.15.67:0/1586475549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdde4101850 0x7fdde40735d0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fddcc0092e0 tx=0x7fddcc00bd70 comp rx=0 tx=0).stop 2024-04-23T00:02:33.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.222+0000 7fdde9f6c700 1 -- 172.21.15.67:0/1586475549 shutdown_connections 2024-04-23T00:02:33.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.222+0000 7fdde9f6c700 1 --2- 172.21.15.67:0/1586475549 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fddd006ecc0 0x7fddd0071180 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:33.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.222+0000 7fdde9f6c700 1 --2- 172.21.15.67:0/1586475549 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdde4100ea0 0x7fdde4073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:33.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.222+0000 7fdde9f6c700 1 --2- 172.21.15.67:0/1586475549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdde4101850 0x7fdde40735d0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:33.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.222+0000 7fdde9f6c700 1 -- 172.21.15.67:0/1586475549 >> 172.21.15.67:0/1586475549 conn(0x7fdde40fc750 msgr2=0x7fdde40fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:02:33.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.222+0000 7fdde9f6c700 1 -- 172.21.15.67:0/1586475549 shutdown_connections 2024-04-23T00:02:33.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.222+0000 7fdde9f6c700 1 -- 172.21.15.67:0/1586475549 wait complete. 2024-04-23T00:02:33.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.333+0000 7ff69b4f5700 1 -- 172.21.15.67:0/928906970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6941027e0 msgr2=0x7ff69410acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:02:33.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.333+0000 7ff69b4f5700 1 --2- 172.21.15.67:0/928906970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6941027e0 0x7ff69410acd0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7ff684009940 tx=0x7ff684009c50 comp rx=0 tx=0).stop 2024-04-23T00:02:33.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.333+0000 7ff69b4f5700 1 -- 172.21.15.67:0/928906970 shutdown_connections 2024-04-23T00:02:33.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.333+0000 7ff69b4f5700 1 --2- 172.21.15.67:0/928906970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6941027e0 0x7ff69410acd0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:33.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.333+0000 7ff69b4f5700 1 --2- 172.21.15.67:0/928906970 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff694101ec0 0x7ff6941022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:33.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.333+0000 7ff69b4f5700 1 -- 172.21.15.67:0/928906970 >> 172.21.15.67:0/928906970 conn(0x7ff6940fb650 msgr2=0x7ff6940fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:02:33.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.333+0000 7ff69b4f5700 1 -- 172.21.15.67:0/928906970 shutdown_connections 2024-04-23T00:02:33.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.333+0000 7ff69b4f5700 1 -- 172.21.15.67:0/928906970 wait complete. 2024-04-23T00:02:33.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.334+0000 7ff69b4f5700 1 Processor -- start 2024-04-23T00:02:33.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.334+0000 7ff69b4f5700 1 -- start start 2024-04-23T00:02:33.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.334+0000 7ff69b4f5700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff694101ec0 0x7ff69419c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:02:33.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.334+0000 7ff69b4f5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6941027e0 0x7ff69419cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:02:33.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.334+0000 7ff69b4f5700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69419d230 con 0x7ff6941027e0 2024-04-23T00:02:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.334+0000 7ff69b4f5700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6941966e0 con 0x7ff694101ec0 2024-04-23T00:02:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.334+0000 7ff698a90700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6941027e0 0x7ff69419cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:02:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.334+0000 7ff698a90700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6941027e0 0x7ff69419cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37754/0 (socket says 172.21.15.67:37754) 2024-04-23T00:02:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.334+0000 7ff698a90700 1 -- 172.21.15.67:0/3292506634 learned_addr learned my addr 172.21.15.67:0/3292506634 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:02:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.334+0000 7ff699291700 1 --2- 172.21.15.67:0/3292506634 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff694101ec0 0x7ff69419c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:02:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.334+0000 7ff698a90700 1 -- 172.21.15.67:0/3292506634 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff694101ec0 msgr2=0x7ff69419c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:02:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.334+0000 7ff698a90700 1 --2- 172.21.15.67:0/3292506634 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff694101ec0 0x7ff69419c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.334+0000 7ff698a90700 1 -- 172.21.15.67:0/3292506634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6840095f0 con 0x7ff6941027e0 2024-04-23T00:02:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.335+0000 7ff698a90700 1 --2- 172.21.15.67:0/3292506634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6941027e0 0x7ff69419cba0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7ff6840055f0 tx=0x7ff6840056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:02:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.335+0000 7ff68a7fc700 1 -- 172.21.15.67:0/3292506634 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff68401d070 con 0x7ff6941027e0 2024-04-23T00:02:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.335+0000 7ff69b4f5700 1 -- 172.21.15.67:0/3292506634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff694196960 con 0x7ff6941027e0 2024-04-23T00:02:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.335+0000 7ff69b4f5700 1 -- 172.21.15.67:0/3292506634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff694196e50 con 0x7ff6941027e0 2024-04-23T00:02:33.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.335+0000 7ff68a7fc700 1 -- 172.21.15.67:0/3292506634 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff68400f460 con 0x7ff6941027e0 2024-04-23T00:02:33.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.335+0000 7ff68a7fc700 1 -- 172.21.15.67:0/3292506634 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff684021620 con 0x7ff6941027e0 2024-04-23T00:02:33.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.336+0000 7ff69b4f5700 1 -- 172.21.15.67:0/3292506634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff694108360 con 0x7ff6941027e0 2024-04-23T00:02:33.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.336+0000 7ff68a7fc700 1 -- 172.21.15.67:0/3292506634 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff68400fa40 con 0x7ff6941027e0 2024-04-23T00:02:33.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.337+0000 7ff68a7fc700 1 --2- 172.21.15.67:0/3292506634 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff68006ecc0 0x7ff680071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:02:33.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.337+0000 7ff68a7fc700 1 -- 172.21.15.67:0/3292506634 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7ff684090220 con 0x7ff6941027e0 2024-04-23T00:02:33.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.337+0000 7ff699291700 1 --2- 172.21.15.67:0/3292506634 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff68006ecc0 0x7ff680071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:02:33.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.337+0000 7ff699291700 1 --2- 172.21.15.67:0/3292506634 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff68006ecc0 0x7ff680071180 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ff690005fd0 tx=0x7ff69000a560 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:02:33.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.342+0000 7ff68a7fc700 1 -- 172.21.15.67:0/3292506634 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff68405ca10 con 0x7ff6941027e0 2024-04-23T00:02:33.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:33 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1714486723' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:02:33.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:33 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3659659426' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-23T00:02:33.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.642+0000 7ff69b4f5700 1 -- 172.21.15.67:0/3292506634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff694107d40 con 0x7ff6941027e0 2024-04-23T00:02:33.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.643+0000 7ff68a7fc700 1 -- 172.21.15.67:0/3292506634 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4505 (secure 0 0 0) 0x7ff68405c5a0 con 0x7ff6941027e0 2024-04-23T00:02:33.632 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-23T00:02:33.632 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-23T00:02:33.632 INFO:teuthology.orchestra.run.smithi067.stdout: host smithi134 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-23T00:02:33.633 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-23T00:02:33.633 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-23T00:02:33.633 INFO:teuthology.orchestra.run.smithi067.stdout:Traceback (most recent call last): 2024-04-23T00:02:33.633 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-23T00:02:33.633 INFO:teuthology.orchestra.run.smithi067.stdout: return _run_code(code, main_globals, None, 2024-04-23T00:02:33.633 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-23T00:02:33.633 INFO:teuthology.orchestra.run.smithi067.stdout: exec(code, run_globals) 2024-04-23T00:02:33.633 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5642, in 2024-04-23T00:02:33.633 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5630, in main 2024-04-23T00:02:33.633 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 408, in _infer_config 2024-04-23T00:02:33.633 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 323, in _infer_fsid 2024-04-23T00:02:33.633 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 436, in _infer_image 2024-04-23T00:02:33.634 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 310, in _validate_fsid 2024-04-23T00:02:33.634 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 3380, in command_ceph_volume 2024-04-23T00:02:33.634 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-23T00:02:33.634 INFO:teuthology.orchestra.run.smithi067.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-23T00:02:33.634 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-23T00:02:33.634 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-23T00:02:33.634 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-23T00:02:33.634 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-23T00:02:33.634 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-23T00:02:33.634 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-23T00:02:33.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.645+0000 7ff69b4f5700 1 -- 172.21.15.67:0/3292506634 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff68006ecc0 msgr2=0x7ff680071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:02:33.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.645+0000 7ff69b4f5700 1 --2- 172.21.15.67:0/3292506634 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff68006ecc0 0x7ff680071180 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ff690005fd0 tx=0x7ff69000a560 comp rx=0 tx=0).stop 2024-04-23T00:02:33.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.645+0000 7ff69b4f5700 1 -- 172.21.15.67:0/3292506634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6941027e0 msgr2=0x7ff69419cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:02:33.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.645+0000 7ff69b4f5700 1 --2- 172.21.15.67:0/3292506634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6941027e0 0x7ff69419cba0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7ff6840055f0 tx=0x7ff6840056d0 comp rx=0 tx=0).stop 2024-04-23T00:02:33.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.645+0000 7ff69b4f5700 1 -- 172.21.15.67:0/3292506634 shutdown_connections 2024-04-23T00:02:33.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.645+0000 7ff69b4f5700 1 --2- 172.21.15.67:0/3292506634 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff68006ecc0 0x7ff680071180 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:33.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.645+0000 7ff69b4f5700 1 --2- 172.21.15.67:0/3292506634 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff694101ec0 0x7ff69419c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:33.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.645+0000 7ff69b4f5700 1 --2- 172.21.15.67:0/3292506634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6941027e0 0x7ff69419cba0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:02:33.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.645+0000 7ff69b4f5700 1 -- 172.21.15.67:0/3292506634 >> 172.21.15.67:0/3292506634 conn(0x7ff6940fb650 msgr2=0x7ff6941054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:02:33.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.645+0000 7ff69b4f5700 1 -- 172.21.15.67:0/3292506634 shutdown_connections 2024-04-23T00:02:33.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:02:33.645+0000 7ff69b4f5700 1 -- 172.21.15.67:0/3292506634 wait complete. 2024-04-23T00:02:33.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:33 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1714486723' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:02:33.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:33 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3659659426' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-23T00:02:34.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:34 smithi134 ceph-mon[85630]: pgmap v1087: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:34.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:34 smithi134 ceph-mon[85630]: from='client.35964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:02:34.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:34 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3292506634' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T00:02:34.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:34 smithi067 ceph-mon[112438]: pgmap v1087: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:34.783 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:34 smithi067 ceph-mon[112438]: from='client.35964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:02:34.783 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:34 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3292506634' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T00:02:36.620 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:36 smithi067 ceph-mon[112438]: pgmap v1088: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:36.731 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:36 smithi134 ceph-mon[85630]: pgmap v1088: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:38.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:38 smithi134 ceph-mon[85630]: pgmap v1089: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:38.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:38 smithi067 ceph-mon[112438]: pgmap v1089: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:40.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:40 smithi134 ceph-mon[85630]: pgmap v1090: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:40.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:40 smithi067 ceph-mon[112438]: pgmap v1090: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:41.675 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:02:41.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:02:42.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:42 smithi134 ceph-mon[85630]: pgmap v1091: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:42.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:42 smithi067 ceph-mon[112438]: pgmap v1091: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:44.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:44 smithi134 ceph-mon[85630]: pgmap v1092: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:44.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:44 smithi067 ceph-mon[112438]: pgmap v1092: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:46.729 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:46 smithi067 ceph-mon[112438]: pgmap v1093: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:46.731 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:46 smithi134 ceph-mon[85630]: pgmap v1093: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:48.731 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:48 smithi134 ceph-mon[85630]: pgmap v1094: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:48.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:48 smithi067 ceph-mon[112438]: pgmap v1094: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:50.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:50 smithi134 ceph-mon[85630]: pgmap v1095: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:50.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:50 smithi067 ceph-mon[112438]: pgmap v1095: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:52.731 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:52 smithi134 ceph-mon[85630]: pgmap v1096: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:52.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:52 smithi067 ceph-mon[112438]: pgmap v1096: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:54.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:54 smithi134 ceph-mon[85630]: pgmap v1097: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:54.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:54 smithi067 ceph-mon[112438]: pgmap v1097: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:56.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:56 smithi134 ceph-mon[85630]: pgmap v1098: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:56.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:02:56.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:56 smithi067 ceph-mon[112438]: pgmap v1098: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:56.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:02:58.731 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:02:58 smithi134 ceph-mon[85630]: pgmap v1099: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:02:58.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:02:58 smithi067 ceph-mon[112438]: pgmap v1099: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:03:00.731 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:00 smithi134 ceph-mon[85630]: pgmap v1100: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:03:00.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:00 smithi067 ceph-mon[112438]: pgmap v1100: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:03:02.731 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:02 smithi134 ceph-mon[85630]: pgmap v1101: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:03:02.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:02 smithi067 ceph-mon[112438]: pgmap v1101: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:03:03.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:03.767+0000 7f7cb6275700 1 -- 172.21.15.67:0/4267756695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cb0101e60 msgr2=0x7f7cb0102240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:03.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:03.767+0000 7f7cb6275700 1 --2- 172.21.15.67:0/4267756695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cb0101e60 0x7f7cb0102240 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f7ca0009910 tx=0x7f7ca0009c20 comp rx=0 tx=0).stop 2024-04-23T00:03:03.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:03.767+0000 7f7cb6275700 1 -- 172.21.15.67:0/4267756695 shutdown_connections 2024-04-23T00:03:03.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:03.767+0000 7f7cb6275700 1 --2- 172.21.15.67:0/4267756695 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7cb0102780 0x7f7cb010ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:03.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:03.767+0000 7f7cb6275700 1 --2- 172.21.15.67:0/4267756695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cb0101e60 0x7f7cb0102240 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:03.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:03.767+0000 7f7cb6275700 1 -- 172.21.15.67:0/4267756695 >> 172.21.15.67:0/4267756695 conn(0x7f7cb00fb650 msgr2=0x7f7cb00fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:03:03.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:03.767+0000 7f7cb6275700 1 -- 172.21.15.67:0/4267756695 shutdown_connections 2024-04-23T00:03:03.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:03.767+0000 7f7cb6275700 1 -- 172.21.15.67:0/4267756695 wait complete. 2024-04-23T00:03:03.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:03.767+0000 7f7cb6275700 1 Processor -- start 2024-04-23T00:03:03.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:03.767+0000 7f7cb6275700 1 -- start start 2024-04-23T00:03:03.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:03.768+0000 7f7cb6275700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cb0101e60 0x7f7cb00ffac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:03.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:03.768+0000 7f7cb6275700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7cb0102780 0x7f7cb0100000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:03.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:03.768+0000 7f7cb6275700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb0100540 con 0x7f7cb0101e60 2024-04-23T00:03:03.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:03.768+0000 7f7cb6275700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb0100680 con 0x7f7cb0102780 2024-04-23T00:03:03.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:03.768+0000 7f7caffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cb0101e60 0x7f7cb00ffac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:03.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:03.768+0000 7f7caffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cb0101e60 0x7f7cb00ffac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33512/0 (socket says 172.21.15.67:33512) 2024-04-23T00:03:03.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:03.768+0000 7f7caffff700 1 -- 172.21.15.67:0/2714046986 learned_addr learned my addr 172.21.15.67:0/2714046986 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:03:03.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:03.768+0000 7f7caf7fe700 1 --2- 172.21.15.67:0/2714046986 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7cb0102780 0x7f7cb0100000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:03.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:03.768+0000 7f7caffff700 1 -- 172.21.15.67:0/2714046986 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7cb0102780 msgr2=0x7f7cb0100000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:03.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:03.768+0000 7f7caffff700 1 --2- 172.21.15.67:0/2714046986 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7cb0102780 0x7f7cb0100000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:03.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:03.768+0000 7f7caffff700 1 -- 172.21.15.67:0/2714046986 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca00095f0 con 0x7f7cb0101e60 2024-04-23T00:03:03.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:03.768+0000 7f7caf7fe700 1 --2- 172.21.15.67:0/2714046986 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7cb0102780 0x7f7cb0100000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T00:03:03.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:03.768+0000 7f7caffff700 1 --2- 172.21.15.67:0/2714046986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cb0101e60 0x7f7cb00ffac0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f7ca000f690 tx=0x7f7ca000f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:03:03.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:03.769+0000 7f7cad7fa700 1 -- 172.21.15.67:0/2714046986 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ca001d070 con 0x7f7cb0101e60 2024-04-23T00:03:03.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:03.769+0000 7f7cad7fa700 1 -- 172.21.15.67:0/2714046986 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7ca00050d0 con 0x7f7cb0101e60 2024-04-23T00:03:03.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:03.769+0000 7f7cad7fa700 1 -- 172.21.15.67:0/2714046986 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ca0022470 con 0x7f7cb0101e60 2024-04-23T00:03:03.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:03.769+0000 7f7cb6275700 1 -- 172.21.15.67:0/2714046986 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cb0100900 con 0x7f7cb0101e60 2024-04-23T00:03:03.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:03.769+0000 7f7cb6275700 1 -- 172.21.15.67:0/2714046986 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cb01a2d20 con 0x7f7cb0101e60 2024-04-23T00:03:03.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:03.770+0000 7f7cb6275700 1 -- 172.21.15.67:0/2714046986 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cb0108310 con 0x7f7cb0101e60 2024-04-23T00:03:03.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:03.770+0000 7f7cad7fa700 1 -- 172.21.15.67:0/2714046986 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7ca0003680 con 0x7f7cb0101e60 2024-04-23T00:03:03.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:03.771+0000 7f7cad7fa700 1 --2- 172.21.15.67:0/2714046986 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7c9806ecc0 0x7f7c98071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:03.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:03.771+0000 7f7cad7fa700 1 -- 172.21.15.67:0/2714046986 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f7ca0090480 con 0x7f7cb0101e60 2024-04-23T00:03:03.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:03.771+0000 7f7caf7fe700 1 --2- 172.21.15.67:0/2714046986 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7c9806ecc0 0x7f7c98071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:03.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:03.771+0000 7f7caf7fe700 1 --2- 172.21.15.67:0/2714046986 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7c9806ecc0 0x7f7c98071180 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0101260 tx=0x7f7ca4009260 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:03:03.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:03.776+0000 7f7cad7fa700 1 -- 172.21.15.67:0/2714046986 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7ca0059170 con 0x7f7cb0101e60 2024-04-23T00:03:03.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.010+0000 7f7cb6275700 1 -- 172.21.15.67:0/2714046986 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7cb00611d0 con 0x7f7c9806ecc0 2024-04-23T00:03:03.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.011+0000 7f7cad7fa700 1 -- 172.21.15.67:0/2714046986 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f7cb00611d0 con 0x7f7c9806ecc0 2024-04-23T00:03:04.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.013+0000 7f7cb6275700 1 -- 172.21.15.67:0/2714046986 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7c9806ecc0 msgr2=0x7f7c98071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:04.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.013+0000 7f7cb6275700 1 --2- 172.21.15.67:0/2714046986 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7c9806ecc0 0x7f7c98071180 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0101260 tx=0x7f7ca4009260 comp rx=0 tx=0).stop 2024-04-23T00:03:04.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.013+0000 7f7cb6275700 1 -- 172.21.15.67:0/2714046986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cb0101e60 msgr2=0x7f7cb00ffac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:04.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.013+0000 7f7cb6275700 1 --2- 172.21.15.67:0/2714046986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cb0101e60 0x7f7cb00ffac0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f7ca000f690 tx=0x7f7ca000f770 comp rx=0 tx=0).stop 2024-04-23T00:03:04.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.013+0000 7f7cb6275700 1 -- 172.21.15.67:0/2714046986 shutdown_connections 2024-04-23T00:03:04.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.013+0000 7f7cb6275700 1 --2- 172.21.15.67:0/2714046986 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7c9806ecc0 0x7f7c98071180 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:04.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.013+0000 7f7cb6275700 1 --2- 172.21.15.67:0/2714046986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cb0101e60 0x7f7cb00ffac0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:04.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.013+0000 7f7cb6275700 1 --2- 172.21.15.67:0/2714046986 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7cb0102780 0x7f7cb0100000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:04.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.013+0000 7f7cb6275700 1 -- 172.21.15.67:0/2714046986 >> 172.21.15.67:0/2714046986 conn(0x7f7cb00fb650 msgr2=0x7f7cb0105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:03:04.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.013+0000 7f7cb6275700 1 -- 172.21.15.67:0/2714046986 shutdown_connections 2024-04-23T00:03:04.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.013+0000 7f7cb6275700 1 -- 172.21.15.67:0/2714046986 wait complete. 2024-04-23T00:03:04.012 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-23T00:03:04.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.120+0000 7f865c1f1700 1 -- 172.21.15.67:0/1737038730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8654102720 msgr2=0x7f865410ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:04.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.120+0000 7f865c1f1700 1 --2- 172.21.15.67:0/1737038730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8654102720 0x7f865410ac10 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f8650009910 tx=0x7f8650009c20 comp rx=0 tx=0).stop 2024-04-23T00:03:04.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.120+0000 7f865c1f1700 1 -- 172.21.15.67:0/1737038730 shutdown_connections 2024-04-23T00:03:04.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.120+0000 7f865c1f1700 1 --2- 172.21.15.67:0/1737038730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8654102720 0x7f865410ac10 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:04.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.120+0000 7f865c1f1700 1 --2- 172.21.15.67:0/1737038730 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8654101e00 0x7f86541021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:04.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.120+0000 7f865c1f1700 1 -- 172.21.15.67:0/1737038730 >> 172.21.15.67:0/1737038730 conn(0x7f86540fb650 msgr2=0x7f86540fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:03:04.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.121+0000 7f865c1f1700 1 -- 172.21.15.67:0/1737038730 shutdown_connections 2024-04-23T00:03:04.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.121+0000 7f865c1f1700 1 -- 172.21.15.67:0/1737038730 wait complete. 2024-04-23T00:03:04.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.121+0000 7f865c1f1700 1 Processor -- start 2024-04-23T00:03:04.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.121+0000 7f865c1f1700 1 -- start start 2024-04-23T00:03:04.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.122+0000 7f865c1f1700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8654101e00 0x7f865419c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:04.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.122+0000 7f865c1f1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8654102720 0x7f865419cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:04.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.122+0000 7f865c1f1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f865419d260 con 0x7f8654102720 2024-04-23T00:03:04.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.122+0000 7f865c1f1700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8654196710 con 0x7f8654101e00 2024-04-23T00:03:04.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.122+0000 7f865978c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8654102720 0x7f865419cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:04.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.122+0000 7f865978c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8654102720 0x7f865419cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33526/0 (socket says 172.21.15.67:33526) 2024-04-23T00:03:04.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.122+0000 7f865978c700 1 -- 172.21.15.67:0/958751448 learned_addr learned my addr 172.21.15.67:0/958751448 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:03:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.122+0000 7f8659f8d700 1 --2- 172.21.15.67:0/958751448 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8654101e00 0x7f865419c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.122+0000 7f865978c700 1 -- 172.21.15.67:0/958751448 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8654101e00 msgr2=0x7f865419c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.122+0000 7f865978c700 1 --2- 172.21.15.67:0/958751448 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8654101e00 0x7f865419c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.122+0000 7f865978c700 1 -- 172.21.15.67:0/958751448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86500095f0 con 0x7f8654102720 2024-04-23T00:03:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.122+0000 7f865978c700 1 --2- 172.21.15.67:0/958751448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8654102720 0x7f865419cbd0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f865000bd70 tx=0x7f865000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:03:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.122+0000 7f864affd700 1 -- 172.21.15.67:0/958751448 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f865001d070 con 0x7f8654102720 2024-04-23T00:03:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.122+0000 7f865c1f1700 1 -- 172.21.15.67:0/958751448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8654196990 con 0x7f8654102720 2024-04-23T00:03:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.122+0000 7f864affd700 1 -- 172.21.15.67:0/958751448 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8650005dc0 con 0x7f8654102720 2024-04-23T00:03:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.122+0000 7f864affd700 1 -- 172.21.15.67:0/958751448 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8650017400 con 0x7f8654102720 2024-04-23T00:03:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.122+0000 7f865c1f1700 1 -- 172.21.15.67:0/958751448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8654196e80 con 0x7f8654102720 2024-04-23T00:03:04.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.123+0000 7f865c1f1700 1 -- 172.21.15.67:0/958751448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8654108310 con 0x7f8654102720 2024-04-23T00:03:04.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.124+0000 7f864affd700 1 -- 172.21.15.67:0/958751448 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f865000f460 con 0x7f8654102720 2024-04-23T00:03:04.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.124+0000 7f864affd700 1 --2- 172.21.15.67:0/958751448 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f864006ecc0 0x7f8640071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:04.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.124+0000 7f864affd700 1 -- 172.21.15.67:0/958751448 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f86500903e0 con 0x7f8654102720 2024-04-23T00:03:04.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.124+0000 7f8659f8d700 1 --2- 172.21.15.67:0/958751448 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f864006ecc0 0x7f8640071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:04.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.125+0000 7f8659f8d700 1 --2- 172.21.15.67:0/958751448 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f864006ecc0 0x7f8640071180 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f8644005fd0 tx=0x7f8644005f60 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:03:04.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.129+0000 7f864affd700 1 -- 172.21.15.67:0/958751448 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f865005cb50 con 0x7f8654102720 2024-04-23T00:03:04.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.359+0000 7f865c1f1700 1 -- 172.21.15.67:0/958751448 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8654197d30 con 0x7f864006ecc0 2024-04-23T00:03:04.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.360+0000 7f864affd700 1 -- 172.21.15.67:0/958751448 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8654197d30 con 0x7f864006ecc0 2024-04-23T00:03:04.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.362+0000 7f865c1f1700 1 -- 172.21.15.67:0/958751448 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f864006ecc0 msgr2=0x7f8640071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:04.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.362+0000 7f865c1f1700 1 --2- 172.21.15.67:0/958751448 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f864006ecc0 0x7f8640071180 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f8644005fd0 tx=0x7f8644005f60 comp rx=0 tx=0).stop 2024-04-23T00:03:04.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.362+0000 7f865c1f1700 1 -- 172.21.15.67:0/958751448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8654102720 msgr2=0x7f865419cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:04.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.362+0000 7f865c1f1700 1 --2- 172.21.15.67:0/958751448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8654102720 0x7f865419cbd0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f865000bd70 tx=0x7f865000be50 comp rx=0 tx=0).stop 2024-04-23T00:03:04.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.362+0000 7f865c1f1700 1 -- 172.21.15.67:0/958751448 shutdown_connections 2024-04-23T00:03:04.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.363+0000 7f865c1f1700 1 --2- 172.21.15.67:0/958751448 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f864006ecc0 0x7f8640071180 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:04.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.363+0000 7f865c1f1700 1 --2- 172.21.15.67:0/958751448 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8654101e00 0x7f865419c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:04.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.363+0000 7f865c1f1700 1 --2- 172.21.15.67:0/958751448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8654102720 0x7f865419cbd0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:04.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.363+0000 7f865c1f1700 1 -- 172.21.15.67:0/958751448 >> 172.21.15.67:0/958751448 conn(0x7f86540fb650 msgr2=0x7f8654105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:03:04.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.363+0000 7f865c1f1700 1 -- 172.21.15.67:0/958751448 shutdown_connections 2024-04-23T00:03:04.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.363+0000 7f865c1f1700 1 -- 172.21.15.67:0/958751448 wait complete. 2024-04-23T00:03:04.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.470+0000 7f131d21c700 1 -- 172.21.15.67:0/2343460330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13180ffda0 msgr2=0x7f1318100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:04.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.470+0000 7f131d21c700 1 --2- 172.21.15.67:0/2343460330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13180ffda0 0x7f1318100180 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f1308009910 tx=0x7f1308009c20 comp rx=0 tx=0).stop 2024-04-23T00:03:04.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.470+0000 7f131d21c700 1 -- 172.21.15.67:0/2343460330 shutdown_connections 2024-04-23T00:03:04.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.470+0000 7f131d21c700 1 --2- 172.21.15.67:0/2343460330 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1318100750 0x7f13181047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:04.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.470+0000 7f131d21c700 1 --2- 172.21.15.67:0/2343460330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13180ffda0 0x7f1318100180 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:04.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.470+0000 7f131d21c700 1 -- 172.21.15.67:0/2343460330 >> 172.21.15.67:0/2343460330 conn(0x7f13180fb650 msgr2=0x7f13180fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:03:04.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.470+0000 7f131d21c700 1 -- 172.21.15.67:0/2343460330 shutdown_connections 2024-04-23T00:03:04.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.470+0000 7f131d21c700 1 -- 172.21.15.67:0/2343460330 wait complete. 2024-04-23T00:03:04.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.470+0000 7f131d21c700 1 Processor -- start 2024-04-23T00:03:04.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.471+0000 7f131d21c700 1 -- start start 2024-04-23T00:03:04.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.471+0000 7f131d21c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13180ffda0 0x7f131819c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:04.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.471+0000 7f131d21c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1318100750 0x7f131819cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:04.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.471+0000 7f131d21c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f131819d2c0 con 0x7f1318100750 2024-04-23T00:03:04.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.471+0000 7f131d21c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1318196770 con 0x7f13180ffda0 2024-04-23T00:03:04.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.471+0000 7f131659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1318100750 0x7f131819cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:04.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.471+0000 7f131659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1318100750 0x7f131819cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33558/0 (socket says 172.21.15.67:33558) 2024-04-23T00:03:04.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.471+0000 7f131659c700 1 -- 172.21.15.67:0/884101060 learned_addr learned my addr 172.21.15.67:0/884101060 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:03:04.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.471+0000 7f1316d9d700 1 --2- 172.21.15.67:0/884101060 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13180ffda0 0x7f131819c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:04.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.471+0000 7f131659c700 1 -- 172.21.15.67:0/884101060 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13180ffda0 msgr2=0x7f131819c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:04.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.471+0000 7f131659c700 1 --2- 172.21.15.67:0/884101060 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13180ffda0 0x7f131819c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:04.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.471+0000 7f131659c700 1 -- 172.21.15.67:0/884101060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13080095f0 con 0x7f1318100750 2024-04-23T00:03:04.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.472+0000 7f131659c700 1 --2- 172.21.15.67:0/884101060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1318100750 0x7f131819cc30 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f130c00d900 tx=0x7f130c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:03:04.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.472+0000 7f12fffff700 1 -- 172.21.15.67:0/884101060 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f130c009900 con 0x7f1318100750 2024-04-23T00:03:04.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.472+0000 7f12fffff700 1 -- 172.21.15.67:0/884101060 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f130c010460 con 0x7f1318100750 2024-04-23T00:03:04.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.472+0000 7f12fffff700 1 -- 172.21.15.67:0/884101060 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f130c00f5f0 con 0x7f1318100750 2024-04-23T00:03:04.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.472+0000 7f131d21c700 1 -- 172.21.15.67:0/884101060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1318196a50 con 0x7f1318100750 2024-04-23T00:03:04.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.472+0000 7f131d21c700 1 -- 172.21.15.67:0/884101060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1318196fa0 con 0x7f1318100750 2024-04-23T00:03:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.473+0000 7f131d21c700 1 -- 172.21.15.67:0/884101060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f131804f2e0 con 0x7f1318100750 2024-04-23T00:03:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.473+0000 7f12fffff700 1 -- 172.21.15.67:0/884101060 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f130c0105d0 con 0x7f1318100750 2024-04-23T00:03:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.474+0000 7f12fffff700 1 --2- 172.21.15.67:0/884101060 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f130006ecc0 0x7f1300071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.474+0000 7f12fffff700 1 -- 172.21.15.67:0/884101060 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f130c08ec40 con 0x7f1318100750 2024-04-23T00:03:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.474+0000 7f1316d9d700 1 --2- 172.21.15.67:0/884101060 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f130006ecc0 0x7f1300071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.474+0000 7f1316d9d700 1 --2- 172.21.15.67:0/884101060 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f130006ecc0 0x7f1300071180 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f1308006010 tx=0x7f130801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:03:04.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.479+0000 7f12fffff700 1 -- 172.21.15.67:0/884101060 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f130c05b310 con 0x7f1318100750 2024-04-23T00:03:04.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.698+0000 7f131d21c700 1 -- 172.21.15.67:0/884101060 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1318197230 con 0x7f130006ecc0 2024-04-23T00:03:04.689 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:04 smithi067 ceph-mon[112438]: pgmap v1102: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:03:04.690 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:04 smithi067 ceph-mon[112438]: from='client.35972 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:03:04.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.709+0000 7f12fffff700 1 -- 172.21.15.67:0/884101060 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f1318197230 con 0x7f130006ecc0 2024-04-23T00:03:04.698 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-23T00:03:04.698 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (36m) 14m ago 47m 24.4M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-23T00:03:04.699 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (48m) 14m ago 48m 10.0M - 18.2.1 5be31c24972a df887f62291e 2024-04-23T00:03:04.699 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (47m) 14m ago 47m 10.0M - 18.2.1 5be31c24972a ad89190b5400 2024-04-23T00:03:04.699 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (34m) 14m ago 48m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-23T00:03:04.699 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (34m) 14m ago 47m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-23T00:03:04.699 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (36m) 14m ago 47m 81.2M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-23T00:03:04.699 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (30m) 14m ago 42m 19.4M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-23T00:03:04.699 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (30m) 14m ago 42m 20.3M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-23T00:03:04.699 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (29m) 14m ago 42m 20.1M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-23T00:03:04.700 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (14m) 14m ago 42m 14.0M - 19.0.0-3252-g43be0201 a0434699c1d7 46d3d7e86fee 2024-04-23T00:03:04.700 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (38m) 14m ago 49m 514M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-23T00:03:04.700 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (37m) 14m ago 47m 459M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-23T00:03:04.700 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (35m) 14m ago 49m 74.2M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-23T00:03:04.700 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (34m) 14m ago 47m 54.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-23T00:03:04.700 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (37m) 14m ago 48m 15.2M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-23T00:03:04.700 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (37m) 14m ago 47m 16.0M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-23T00:03:04.700 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (33m) 14m ago 46m 102M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-23T00:03:04.700 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (33m) 14m ago 45m 128M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-23T00:03:04.701 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (32m) 14m ago 45m 83.8M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-23T00:03:04.701 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (31m) 14m ago 45m 101M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-23T00:03:04.701 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (31m) 14m ago 44m 86.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-23T00:03:04.701 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (30m) 14m ago 44m 81.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-23T00:03:04.701 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (37m) 14m ago 47m 78.7M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-23T00:03:04.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.711+0000 7f131d21c700 1 -- 172.21.15.67:0/884101060 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f130006ecc0 msgr2=0x7f1300071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:04.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.711+0000 7f131d21c700 1 --2- 172.21.15.67:0/884101060 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f130006ecc0 0x7f1300071180 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f1308006010 tx=0x7f130801a040 comp rx=0 tx=0).stop 2024-04-23T00:03:04.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.711+0000 7f131d21c700 1 -- 172.21.15.67:0/884101060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1318100750 msgr2=0x7f131819cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:04.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.711+0000 7f131d21c700 1 --2- 172.21.15.67:0/884101060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1318100750 0x7f131819cc30 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f130c00d900 tx=0x7f130c00dc10 comp rx=0 tx=0).stop 2024-04-23T00:03:04.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.712+0000 7f131d21c700 1 -- 172.21.15.67:0/884101060 shutdown_connections 2024-04-23T00:03:04.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.712+0000 7f131d21c700 1 --2- 172.21.15.67:0/884101060 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f130006ecc0 0x7f1300071180 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:04.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.712+0000 7f131d21c700 1 --2- 172.21.15.67:0/884101060 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f13180ffda0 0x7f131819c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:04.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.712+0000 7f131d21c700 1 --2- 172.21.15.67:0/884101060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1318100750 0x7f131819cc30 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:04.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.712+0000 7f131d21c700 1 -- 172.21.15.67:0/884101060 >> 172.21.15.67:0/884101060 conn(0x7f13180fb650 msgr2=0x7f1318103fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:03:04.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.712+0000 7f131d21c700 1 -- 172.21.15.67:0/884101060 shutdown_connections 2024-04-23T00:03:04.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.712+0000 7f131d21c700 1 -- 172.21.15.67:0/884101060 wait complete. 2024-04-23T00:03:04.731 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:04 smithi134 ceph-mon[85630]: pgmap v1102: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:03:04.731 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:04 smithi134 ceph-mon[85630]: from='client.35972 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:03:04.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.817+0000 7fb2eeae4700 1 -- 172.21.15.67:0/1471231358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2e8105bd0 msgr2=0x7fb2e8105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:04.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.817+0000 7fb2eeae4700 1 --2- 172.21.15.67:0/1471231358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2e8105bd0 0x7fb2e8105fb0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fb2e000b0a0 tx=0x7fb2e000b3b0 comp rx=0 tx=0).stop 2024-04-23T00:03:04.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.817+0000 7fb2eeae4700 1 -- 172.21.15.67:0/1471231358 shutdown_connections 2024-04-23T00:03:04.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.817+0000 7fb2eeae4700 1 --2- 172.21.15.67:0/1471231358 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2e80ffc20 0x7fb2e8100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:04.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.817+0000 7fb2eeae4700 1 --2- 172.21.15.67:0/1471231358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2e8105bd0 0x7fb2e8105fb0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:04.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.817+0000 7fb2eeae4700 1 -- 172.21.15.67:0/1471231358 >> 172.21.15.67:0/1471231358 conn(0x7fb2e8075670 msgr2=0x7fb2e8075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:03:04.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.817+0000 7fb2eeae4700 1 -- 172.21.15.67:0/1471231358 shutdown_connections 2024-04-23T00:03:04.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.817+0000 7fb2eeae4700 1 -- 172.21.15.67:0/1471231358 wait complete. 2024-04-23T00:03:04.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.818+0000 7fb2eeae4700 1 Processor -- start 2024-04-23T00:03:04.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.818+0000 7fb2eeae4700 1 -- start start 2024-04-23T00:03:04.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.818+0000 7fb2eeae4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2e80ffc20 0x7fb2e819c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:04.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.818+0000 7fb2eeae4700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2e8105bd0 0x7fb2e819cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:04.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.818+0000 7fb2eeae4700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e819d1f0 con 0x7fb2e80ffc20 2024-04-23T00:03:04.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.818+0000 7fb2eeae4700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e81966a0 con 0x7fb2e8105bd0 2024-04-23T00:03:04.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.818+0000 7fb2ec880700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2e80ffc20 0x7fb2e819c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:04.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.818+0000 7fb2ec880700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2e80ffc20 0x7fb2e819c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33574/0 (socket says 172.21.15.67:33574) 2024-04-23T00:03:04.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.818+0000 7fb2ec880700 1 -- 172.21.15.67:0/1661756419 learned_addr learned my addr 172.21.15.67:0/1661756419 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:03:04.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.818+0000 7fb2e7fff700 1 --2- 172.21.15.67:0/1661756419 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2e8105bd0 0x7fb2e819cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:04.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.818+0000 7fb2ec880700 1 -- 172.21.15.67:0/1661756419 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2e8105bd0 msgr2=0x7fb2e819cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:04.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.818+0000 7fb2ec880700 1 --2- 172.21.15.67:0/1661756419 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2e8105bd0 0x7fb2e819cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:04.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.818+0000 7fb2ec880700 1 -- 172.21.15.67:0/1661756419 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e0009d00 con 0x7fb2e80ffc20 2024-04-23T00:03:04.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.818+0000 7fb2e7fff700 1 --2- 172.21.15.67:0/1661756419 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2e8105bd0 0x7fb2e819cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T00:03:04.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.818+0000 7fb2ec880700 1 --2- 172.21.15.67:0/1661756419 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2e80ffc20 0x7fb2e819c620 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fb2e0000f80 tx=0x7fb2e000cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:03:04.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.819+0000 7fb2e5ffb700 1 -- 172.21.15.67:0/1661756419 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2e000e040 con 0x7fb2e80ffc20 2024-04-23T00:03:04.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.819+0000 7fb2e5ffb700 1 -- 172.21.15.67:0/1661756419 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb2e0003e60 con 0x7fb2e80ffc20 2024-04-23T00:03:04.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.819+0000 7fb2eeae4700 1 -- 172.21.15.67:0/1661756419 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2e8196920 con 0x7fb2e80ffc20 2024-04-23T00:03:04.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.819+0000 7fb2e5ffb700 1 -- 172.21.15.67:0/1661756419 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2e001d910 con 0x7fb2e80ffc20 2024-04-23T00:03:04.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.819+0000 7fb2eeae4700 1 -- 172.21.15.67:0/1661756419 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2e8196e10 con 0x7fb2e80ffc20 2024-04-23T00:03:04.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.820+0000 7fb2eeae4700 1 -- 172.21.15.67:0/1661756419 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2e804f2e0 con 0x7fb2e80ffc20 2024-04-23T00:03:04.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.820+0000 7fb2e5ffb700 1 -- 172.21.15.67:0/1661756419 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb2e0004820 con 0x7fb2e80ffc20 2024-04-23T00:03:04.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.821+0000 7fb2e5ffb700 1 --2- 172.21.15.67:0/1661756419 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb2d406ecc0 0x7fb2d4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:04.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.821+0000 7fb2e5ffb700 1 -- 172.21.15.67:0/1661756419 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fb2e00907c0 con 0x7fb2e80ffc20 2024-04-23T00:03:04.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.821+0000 7fb2e7fff700 1 --2- 172.21.15.67:0/1661756419 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb2d406ecc0 0x7fb2d4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:04.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.821+0000 7fb2e7fff700 1 --2- 172.21.15.67:0/1661756419 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb2d406ecc0 0x7fb2d4071180 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fb2e80ffa40 tx=0x7fb2d8005f50 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:03:04.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:04.826+0000 7fb2e5ffb700 1 -- 172.21.15.67:0/1661756419 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb2e005cfb0 con 0x7fb2e80ffc20 2024-04-23T00:03:05.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.125+0000 7fb2eeae4700 1 -- 172.21.15.67:0/1661756419 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb2e8197140 con 0x7fb2e80ffc20 2024-04-23T00:03:05.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.125+0000 7fb2e5ffb700 1 -- 172.21.15.67:0/1661756419 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fb2e005cb40 con 0x7fb2e80ffc20 2024-04-23T00:03:05.114 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-23T00:03:05.114 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-23T00:03:05.114 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-23T00:03:05.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-23T00:03:05.114 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-23T00:03:05.114 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-23T00:03:05.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-23T00:03:05.114 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-23T00:03:05.114 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-23T00:03:05.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-23T00:03:05.115 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-23T00:03:05.115 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 4 2024-04-23T00:03:05.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-23T00:03:05.115 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-23T00:03:05.115 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 14 2024-04-23T00:03:05.115 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-23T00:03:05.115 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-23T00:03:05.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.127+0000 7fb2eeae4700 1 -- 172.21.15.67:0/1661756419 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb2d406ecc0 msgr2=0x7fb2d4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:05.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.127+0000 7fb2eeae4700 1 --2- 172.21.15.67:0/1661756419 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb2d406ecc0 0x7fb2d4071180 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fb2e80ffa40 tx=0x7fb2d8005f50 comp rx=0 tx=0).stop 2024-04-23T00:03:05.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.127+0000 7fb2eeae4700 1 -- 172.21.15.67:0/1661756419 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2e80ffc20 msgr2=0x7fb2e819c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:05.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.127+0000 7fb2eeae4700 1 --2- 172.21.15.67:0/1661756419 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2e80ffc20 0x7fb2e819c620 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fb2e0000f80 tx=0x7fb2e000cdd0 comp rx=0 tx=0).stop 2024-04-23T00:03:05.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.127+0000 7fb2eeae4700 1 -- 172.21.15.67:0/1661756419 shutdown_connections 2024-04-23T00:03:05.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.127+0000 7fb2eeae4700 1 --2- 172.21.15.67:0/1661756419 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb2d406ecc0 0x7fb2d4071180 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:05.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.127+0000 7fb2eeae4700 1 --2- 172.21.15.67:0/1661756419 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2e80ffc20 0x7fb2e819c620 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:05.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.127+0000 7fb2eeae4700 1 --2- 172.21.15.67:0/1661756419 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb2e8105bd0 0x7fb2e819cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:05.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.127+0000 7fb2eeae4700 1 -- 172.21.15.67:0/1661756419 >> 172.21.15.67:0/1661756419 conn(0x7fb2e8075670 msgr2=0x7fb2e80686c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:03:05.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.128+0000 7fb2eeae4700 1 -- 172.21.15.67:0/1661756419 shutdown_connections 2024-04-23T00:03:05.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.128+0000 7fb2eeae4700 1 -- 172.21.15.67:0/1661756419 wait complete. 2024-04-23T00:03:05.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.234+0000 7f6694759700 1 -- 172.21.15.67:0/3636547297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f668c0739d0 msgr2=0x7f668c10cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:05.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.234+0000 7f6694759700 1 --2- 172.21.15.67:0/3636547297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f668c0739d0 0x7f668c10cfc0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f667c009910 tx=0x7f667c009c20 comp rx=0 tx=0).stop 2024-04-23T00:03:05.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.234+0000 7f6694759700 1 -- 172.21.15.67:0/3636547297 shutdown_connections 2024-04-23T00:03:05.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.234+0000 7f6694759700 1 --2- 172.21.15.67:0/3636547297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f668c0739d0 0x7f668c10cfc0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:05.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.234+0000 7f6694759700 1 --2- 172.21.15.67:0/3636547297 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f668c0730b0 0x7f668c073490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:05.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.234+0000 7f6694759700 1 -- 172.21.15.67:0/3636547297 >> 172.21.15.67:0/3636547297 conn(0x7f668c0fc750 msgr2=0x7f668c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:03:05.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.234+0000 7f6694759700 1 -- 172.21.15.67:0/3636547297 shutdown_connections 2024-04-23T00:03:05.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.234+0000 7f6694759700 1 -- 172.21.15.67:0/3636547297 wait complete. 2024-04-23T00:03:05.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.234+0000 7f6694759700 1 Processor -- start 2024-04-23T00:03:05.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.234+0000 7f6694759700 1 -- start start 2024-04-23T00:03:05.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.235+0000 7f6694759700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f668c0730b0 0x7f668c19e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:05.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.235+0000 7f6694759700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f668c0739d0 0x7f668c19eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:05.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.235+0000 7f6694759700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f668c19f550 con 0x7f668c0739d0 2024-04-23T00:03:05.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.235+0000 7f6694759700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f668c198a00 con 0x7f668c0730b0 2024-04-23T00:03:05.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.235+0000 7f6691cf4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f668c0739d0 0x7f668c19eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:05.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.235+0000 7f6691cf4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f668c0739d0 0x7f668c19eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33600/0 (socket says 172.21.15.67:33600) 2024-04-23T00:03:05.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.235+0000 7f6691cf4700 1 -- 172.21.15.67:0/3635866675 learned_addr learned my addr 172.21.15.67:0/3635866675 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:03:05.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.235+0000 7f66924f5700 1 --2- 172.21.15.67:0/3635866675 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f668c0730b0 0x7f668c19e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:05.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.235+0000 7f6691cf4700 1 -- 172.21.15.67:0/3635866675 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f668c0730b0 msgr2=0x7f668c19e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:05.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.235+0000 7f6691cf4700 1 --2- 172.21.15.67:0/3635866675 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f668c0730b0 0x7f668c19e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:05.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.235+0000 7f6691cf4700 1 -- 172.21.15.67:0/3635866675 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f667c0095f0 con 0x7f668c0739d0 2024-04-23T00:03:05.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.235+0000 7f66924f5700 1 --2- 172.21.15.67:0/3635866675 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f668c0730b0 0x7f668c19e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T00:03:05.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.235+0000 7f6691cf4700 1 --2- 172.21.15.67:0/3635866675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f668c0739d0 0x7f668c19eec0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f667c00bda0 tx=0x7f667c00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:03:05.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.235+0000 7f66837fe700 1 -- 172.21.15.67:0/3635866675 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f667c01d070 con 0x7f668c0739d0 2024-04-23T00:03:05.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.235+0000 7f6694759700 1 -- 172.21.15.67:0/3635866675 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f668c198c80 con 0x7f668c0739d0 2024-04-23T00:03:05.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.235+0000 7f66837fe700 1 -- 172.21.15.67:0/3635866675 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f667c004fb0 con 0x7f668c0739d0 2024-04-23T00:03:05.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.236+0000 7f66837fe700 1 -- 172.21.15.67:0/3635866675 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f667c00f690 con 0x7f668c0739d0 2024-04-23T00:03:05.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.236+0000 7f6694759700 1 -- 172.21.15.67:0/3635866675 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f668c199170 con 0x7f668c0739d0 2024-04-23T00:03:05.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.236+0000 7f6694759700 1 -- 172.21.15.67:0/3635866675 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f668c10a6c0 con 0x7f668c0739d0 2024-04-23T00:03:05.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.237+0000 7f66837fe700 1 -- 172.21.15.67:0/3635866675 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f667c022470 con 0x7f668c0739d0 2024-04-23T00:03:05.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.237+0000 7f66837fe700 1 --2- 172.21.15.67:0/3635866675 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f667806ecc0 0x7f6678071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:05.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.238+0000 7f66837fe700 1 -- 172.21.15.67:0/3635866675 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f667c0908d0 con 0x7f668c0739d0 2024-04-23T00:03:05.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.238+0000 7f66924f5700 1 --2- 172.21.15.67:0/3635866675 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f667806ecc0 0x7f6678071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:05.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.238+0000 7f66924f5700 1 --2- 172.21.15.67:0/3635866675 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f667806ecc0 0x7f6678071180 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f6688007900 tx=0x7f6688008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:03:05.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.243+0000 7f66837fe700 1 -- 172.21.15.67:0/3635866675 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f667c05cfa0 con 0x7f668c0739d0 2024-04-23T00:03:05.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.506+0000 7f6694759700 1 -- 172.21.15.67:0/3635866675 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f668c199f40 con 0x7f668c0739d0 2024-04-23T00:03:05.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.507+0000 7f66837fe700 1 -- 172.21.15.67:0/3635866675 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1872 (secure 0 0 0) 0x7f667c05cb30 con 0x7f668c0739d0 2024-04-23T00:03:05.496 INFO:teuthology.orchestra.run.smithi067.stdout:e33 2024-04-23T00:03:05.496 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-23T00:03:05.496 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-23T00:03:05.496 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-23T00:03:05.496 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:03:05.496 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-23T00:03:05.496 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-23T00:03:05.496 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 24 2024-04-23T00:03:05.496 INFO:teuthology.orchestra.run.smithi067.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-23T00:03:05.496 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-23T00:03:05.496 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-22T23:32:27.027617+0000 2024-04-23T00:03:05.496 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-23T00:03:05.497 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-23T00:03:05.497 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-23T00:03:05.497 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-23T00:03:05.497 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-23T00:03:05.497 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-23T00:03:05.497 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-23T00:03:05.497 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-23T00:03:05.497 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-23T00:03:05.497 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-23T00:03:05.497 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-23T00:03:05.497 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-23T00:03:05.497 INFO:teuthology.orchestra.run.smithi067.stdout:up {} 2024-04-23T00:03:05.497 INFO:teuthology.orchestra.run.smithi067.stdout:failed 0,1 2024-04-23T00:03:05.497 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-23T00:03:05.497 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-23T00:03:05.498 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-23T00:03:05.498 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-23T00:03:05.498 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-23T00:03:05.498 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-23T00:03:05.498 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-23T00:03:05.498 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-23T00:03:05.498 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 0 members: 2024-04-23T00:03:05.498 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:03:05.498 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:03:05.498 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-04-23T00:03:05.498 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:03:05.498 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{-1:34396} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T00:03:05.498 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{-1:34424} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T00:03:05.498 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{-1:44327} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T00:03:05.498 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{-1:44851} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1849648622,v1:172.21.15.134:6827/1849648622] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T00:03:05.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.509+0000 7f6694759700 1 -- 172.21.15.67:0/3635866675 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f667806ecc0 msgr2=0x7f6678071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:05.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.509+0000 7f6694759700 1 --2- 172.21.15.67:0/3635866675 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f667806ecc0 0x7f6678071180 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f6688007900 tx=0x7f6688008040 comp rx=0 tx=0).stop 2024-04-23T00:03:05.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.509+0000 7f6694759700 1 -- 172.21.15.67:0/3635866675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f668c0739d0 msgr2=0x7f668c19eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:05.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.509+0000 7f6694759700 1 --2- 172.21.15.67:0/3635866675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f668c0739d0 0x7f668c19eec0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f667c00bda0 tx=0x7f667c00be80 comp rx=0 tx=0).stop 2024-04-23T00:03:05.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.509+0000 7f6694759700 1 -- 172.21.15.67:0/3635866675 shutdown_connections 2024-04-23T00:03:05.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.509+0000 7f6694759700 1 --2- 172.21.15.67:0/3635866675 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f667806ecc0 0x7f6678071180 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:05.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.509+0000 7f6694759700 1 --2- 172.21.15.67:0/3635866675 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f668c0730b0 0x7f668c19e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:05.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.509+0000 7f6694759700 1 --2- 172.21.15.67:0/3635866675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f668c0739d0 0x7f668c19eec0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:05.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.509+0000 7f6694759700 1 -- 172.21.15.67:0/3635866675 >> 172.21.15.67:0/3635866675 conn(0x7f668c0fc750 msgr2=0x7f668c107800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:03:05.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.509+0000 7f6694759700 1 -- 172.21.15.67:0/3635866675 shutdown_connections 2024-04-23T00:03:05.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.509+0000 7f6694759700 1 -- 172.21.15.67:0/3635866675 wait complete. 2024-04-23T00:03:05.500 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 33 2024-04-23T00:03:05.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.616+0000 7fee10381700 1 -- 172.21.15.67:0/1301094718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee08101850 msgr2=0x7fee081058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:05.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.616+0000 7fee10381700 1 --2- 172.21.15.67:0/1301094718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee08101850 0x7fee081058a0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fee04009910 tx=0x7fee04009c20 comp rx=0 tx=0).stop 2024-04-23T00:03:05.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.616+0000 7fee10381700 1 -- 172.21.15.67:0/1301094718 shutdown_connections 2024-04-23T00:03:05.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.616+0000 7fee10381700 1 --2- 172.21.15.67:0/1301094718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee08101850 0x7fee081058a0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:05.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.616+0000 7fee10381700 1 --2- 172.21.15.67:0/1301094718 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee08100ea0 0x7fee08101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:05.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.617+0000 7fee10381700 1 -- 172.21.15.67:0/1301094718 >> 172.21.15.67:0/1301094718 conn(0x7fee080fc750 msgr2=0x7fee080feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:03:05.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.617+0000 7fee10381700 1 -- 172.21.15.67:0/1301094718 shutdown_connections 2024-04-23T00:03:05.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.617+0000 7fee10381700 1 -- 172.21.15.67:0/1301094718 wait complete. 2024-04-23T00:03:05.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.617+0000 7fee10381700 1 Processor -- start 2024-04-23T00:03:05.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.617+0000 7fee10381700 1 -- start start 2024-04-23T00:03:05.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.617+0000 7fee10381700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee08100ea0 0x7fee0819c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:05.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.617+0000 7fee10381700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee08101850 0x7fee0819cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:05.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.617+0000 7fee10381700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0819d2d0 con 0x7fee08101850 2024-04-23T00:03:05.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.617+0000 7fee10381700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee08196780 con 0x7fee08100ea0 2024-04-23T00:03:05.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.617+0000 7fee0d91c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee08101850 0x7fee0819cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:05.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.618+0000 7fee0d91c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee08101850 0x7fee0819cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33624/0 (socket says 172.21.15.67:33624) 2024-04-23T00:03:05.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.618+0000 7fee0d91c700 1 -- 172.21.15.67:0/2842885965 learned_addr learned my addr 172.21.15.67:0/2842885965 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:03:05.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.618+0000 7fee0e11d700 1 --2- 172.21.15.67:0/2842885965 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee08100ea0 0x7fee0819c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:05.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.618+0000 7fee0d91c700 1 -- 172.21.15.67:0/2842885965 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee08100ea0 msgr2=0x7fee0819c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:05.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.618+0000 7fee0d91c700 1 --2- 172.21.15.67:0/2842885965 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee08100ea0 0x7fee0819c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:05.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.618+0000 7fee0d91c700 1 -- 172.21.15.67:0/2842885965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee040095f0 con 0x7fee08101850 2024-04-23T00:03:05.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.618+0000 7fee0e11d700 1 --2- 172.21.15.67:0/2842885965 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee08100ea0 0x7fee0819c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T00:03:05.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.618+0000 7fee0d91c700 1 --2- 172.21.15.67:0/2842885965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee08101850 0x7fee0819cc40 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fee040092e0 tx=0x7fee0400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:03:05.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.618+0000 7fedff7fe700 1 -- 172.21.15.67:0/2842885965 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee0401d070 con 0x7fee08101850 2024-04-23T00:03:05.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.618+0000 7fee10381700 1 -- 172.21.15.67:0/2842885965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee08196a00 con 0x7fee08101850 2024-04-23T00:03:05.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.618+0000 7fedff7fe700 1 -- 172.21.15.67:0/2842885965 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fee0400f460 con 0x7fee08101850 2024-04-23T00:03:05.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.618+0000 7fedff7fe700 1 -- 172.21.15.67:0/2842885965 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee040175f0 con 0x7fee08101850 2024-04-23T00:03:05.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.618+0000 7fee10381700 1 -- 172.21.15.67:0/2842885965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee08196ef0 con 0x7fee08101850 2024-04-23T00:03:05.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.619+0000 7fee10381700 1 -- 172.21.15.67:0/2842885965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee081092a0 con 0x7fee08101850 2024-04-23T00:03:05.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.620+0000 7fedff7fe700 1 -- 172.21.15.67:0/2842885965 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fee0400f5d0 con 0x7fee08101850 2024-04-23T00:03:05.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.620+0000 7fedff7fe700 1 --2- 172.21.15.67:0/2842885965 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fedf406ecc0 0x7fedf4071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:05.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.620+0000 7fedff7fe700 1 -- 172.21.15.67:0/2842885965 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fee04090500 con 0x7fee08101850 2024-04-23T00:03:05.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.620+0000 7fee0e11d700 1 --2- 172.21.15.67:0/2842885965 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fedf406ecc0 0x7fedf4071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:05.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.621+0000 7fee0e11d700 1 --2- 172.21.15.67:0/2842885965 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fedf406ecc0 0x7fedf4071180 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fedf8005fd0 tx=0x7fedf8005dc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:03:05.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.625+0000 7fedff7fe700 1 -- 172.21.15.67:0/2842885965 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fee0405cc70 con 0x7fee08101850 2024-04-23T00:03:05.731 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:05 smithi134 ceph-mon[85630]: from='client.35976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:03:05.731 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:05 smithi134 ceph-mon[85630]: from='client.35980 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:03:05.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:05 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1661756419' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:05.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:05 smithi067 ceph-mon[112438]: from='client.35976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:03:05.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:05 smithi067 ceph-mon[112438]: from='client.35980 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:03:05.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:05 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1661756419' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:05.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.859+0000 7fee10381700 1 -- 172.21.15.67:0/2842885965 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fee08197110 con 0x7fedf406ecc0 2024-04-23T00:03:05.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.860+0000 7fedff7fe700 1 -- 172.21.15.67:0/2842885965 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fee08197110 con 0x7fedf406ecc0 2024-04-23T00:03:05.849 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-23T00:03:05.849 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-23T00:03:05.849 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-23T00:03:05.849 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-23T00:03:05.849 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-23T00:03:05.849 INFO:teuthology.orchestra.run.smithi067.stdout: "mds", 2024-04-23T00:03:05.849 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-23T00:03:05.849 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-23T00:03:05.849 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-23T00:03:05.849 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-23T00:03:05.849 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-23T00:03:05.849 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "16/23 daemons upgraded", 2024-04-23T00:03:05.849 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading mds daemons", 2024-04-23T00:03:05.849 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-23T00:03:05.850 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-23T00:03:05.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.862+0000 7fee10381700 1 -- 172.21.15.67:0/2842885965 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fedf406ecc0 msgr2=0x7fedf4071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:05.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.862+0000 7fee10381700 1 --2- 172.21.15.67:0/2842885965 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fedf406ecc0 0x7fedf4071180 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fedf8005fd0 tx=0x7fedf8005dc0 comp rx=0 tx=0).stop 2024-04-23T00:03:05.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.862+0000 7fee10381700 1 -- 172.21.15.67:0/2842885965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee08101850 msgr2=0x7fee0819cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:05.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.862+0000 7fee10381700 1 --2- 172.21.15.67:0/2842885965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee08101850 0x7fee0819cc40 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fee040092e0 tx=0x7fee0400be20 comp rx=0 tx=0).stop 2024-04-23T00:03:05.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.862+0000 7fee10381700 1 -- 172.21.15.67:0/2842885965 shutdown_connections 2024-04-23T00:03:05.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.862+0000 7fee10381700 1 --2- 172.21.15.67:0/2842885965 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fedf406ecc0 0x7fedf4071180 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:05.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.862+0000 7fee10381700 1 --2- 172.21.15.67:0/2842885965 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fee08100ea0 0x7fee0819c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:05.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.862+0000 7fee10381700 1 --2- 172.21.15.67:0/2842885965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee08101850 0x7fee0819cc40 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:05.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.862+0000 7fee10381700 1 -- 172.21.15.67:0/2842885965 >> 172.21.15.67:0/2842885965 conn(0x7fee080fc750 msgr2=0x7fee080fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:03:05.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.863+0000 7fee10381700 1 -- 172.21.15.67:0/2842885965 shutdown_connections 2024-04-23T00:03:05.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.863+0000 7fee10381700 1 -- 172.21.15.67:0/2842885965 wait complete. 2024-04-23T00:03:05.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.969+0000 7f492678c700 1 -- 172.21.15.67:0/1064347215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4920068580 msgr2=0x7f49200689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:05.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.969+0000 7f492678c700 1 --2- 172.21.15.67:0/1064347215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4920068580 0x7f49200689e0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f4914009910 tx=0x7f4914009c20 comp rx=0 tx=0).stop 2024-04-23T00:03:05.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.969+0000 7f492678c700 1 -- 172.21.15.67:0/1064347215 shutdown_connections 2024-04-23T00:03:05.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.969+0000 7f492678c700 1 --2- 172.21.15.67:0/1064347215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4920068580 0x7f49200689e0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:05.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.969+0000 7f492678c700 1 --2- 172.21.15.67:0/1064347215 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4920106cd0 0x7f49201070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:05.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.969+0000 7f492678c700 1 -- 172.21.15.67:0/1064347215 >> 172.21.15.67:0/1064347215 conn(0x7f4920075670 msgr2=0x7f4920075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:03:05.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.970+0000 7f492678c700 1 -- 172.21.15.67:0/1064347215 shutdown_connections 2024-04-23T00:03:05.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.970+0000 7f492678c700 1 -- 172.21.15.67:0/1064347215 wait complete. 2024-04-23T00:03:05.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.970+0000 7f492678c700 1 Processor -- start 2024-04-23T00:03:05.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.970+0000 7f492678c700 1 -- start start 2024-04-23T00:03:05.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.970+0000 7f492678c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4920068580 0x7f49200ffcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:05.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.971+0000 7f492678c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4920106cd0 0x7f4920100230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:05.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.971+0000 7f492678c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4920100770 con 0x7f4920106cd0 2024-04-23T00:03:05.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.971+0000 7f492678c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49201008b0 con 0x7f4920068580 2024-04-23T00:03:05.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.971+0000 7f491f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4920106cd0 0x7f4920100230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:05.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.971+0000 7f491f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4920106cd0 0x7f4920100230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33650/0 (socket says 172.21.15.67:33650) 2024-04-23T00:03:05.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.971+0000 7f491f7fe700 1 -- 172.21.15.67:0/681507475 learned_addr learned my addr 172.21.15.67:0/681507475 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:03:05.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.971+0000 7f491ffff700 1 --2- 172.21.15.67:0/681507475 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4920068580 0x7f49200ffcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:05.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.971+0000 7f491f7fe700 1 -- 172.21.15.67:0/681507475 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4920068580 msgr2=0x7f49200ffcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:05.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.971+0000 7f491f7fe700 1 --2- 172.21.15.67:0/681507475 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4920068580 0x7f49200ffcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:05.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.971+0000 7f491f7fe700 1 -- 172.21.15.67:0/681507475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49140095f0 con 0x7f4920106cd0 2024-04-23T00:03:05.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.971+0000 7f491f7fe700 1 --2- 172.21.15.67:0/681507475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4920106cd0 0x7f4920100230 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f4914006010 tx=0x7f491400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:03:05.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.971+0000 7f491d7fa700 1 -- 172.21.15.67:0/681507475 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f491401d070 con 0x7f4920106cd0 2024-04-23T00:03:05.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.971+0000 7f491d7fa700 1 -- 172.21.15.67:0/681507475 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4914005dc0 con 0x7f4920106cd0 2024-04-23T00:03:05.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.971+0000 7f491d7fa700 1 -- 172.21.15.67:0/681507475 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4914017400 con 0x7f4920106cd0 2024-04-23T00:03:05.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.971+0000 7f492678c700 1 -- 172.21.15.67:0/681507475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4920100b30 con 0x7f4920106cd0 2024-04-23T00:03:05.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.971+0000 7f492678c700 1 -- 172.21.15.67:0/681507475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49201a2d60 con 0x7f4920106cd0 2024-04-23T00:03:05.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.972+0000 7f492678c700 1 -- 172.21.15.67:0/681507475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49201091d0 con 0x7f4920106cd0 2024-04-23T00:03:05.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.973+0000 7f491d7fa700 1 -- 172.21.15.67:0/681507475 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4914003a60 con 0x7f4920106cd0 2024-04-23T00:03:05.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.973+0000 7f491d7fa700 1 --2- 172.21.15.67:0/681507475 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f490806ecc0 0x7f4908071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:05.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.973+0000 7f491d7fa700 1 -- 172.21.15.67:0/681507475 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f4914090450 con 0x7f4920106cd0 2024-04-23T00:03:05.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.974+0000 7f491ffff700 1 --2- 172.21.15.67:0/681507475 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f490806ecc0 0x7f4908071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:05.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.974+0000 7f491ffff700 1 --2- 172.21.15.67:0/681507475 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f490806ecc0 0x7f4908071180 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f4910005950 tx=0x7f491000b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:03:05.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:05.978+0000 7f491d7fa700 1 -- 172.21.15.67:0/681507475 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f4914059090 con 0x7f4920106cd0 2024-04-23T00:03:06.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:06.274+0000 7f492678c700 1 -- 172.21.15.67:0/681507475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f4920108bb0 con 0x7f4920106cd0 2024-04-23T00:03:06.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:06.275+0000 7f491d7fa700 1 -- 172.21.15.67:0/681507475 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4505 (secure 0 0 0) 0x7f491405c750 con 0x7f4920106cd0 2024-04-23T00:03:06.263 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem with deprecated feature inline_data; 1 filesystem is offline 2024-04-23T00:03:06.263 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-23T00:03:06.263 INFO:teuthology.orchestra.run.smithi067.stdout: host smithi134 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-23T00:03:06.263 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-23T00:03:06.264 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-23T00:03:06.264 INFO:teuthology.orchestra.run.smithi067.stdout:Traceback (most recent call last): 2024-04-23T00:03:06.264 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-23T00:03:06.264 INFO:teuthology.orchestra.run.smithi067.stdout: return _run_code(code, main_globals, None, 2024-04-23T00:03:06.264 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-23T00:03:06.264 INFO:teuthology.orchestra.run.smithi067.stdout: exec(code, run_globals) 2024-04-23T00:03:06.264 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5642, in 2024-04-23T00:03:06.264 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5630, in main 2024-04-23T00:03:06.264 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 408, in _infer_config 2024-04-23T00:03:06.264 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 323, in _infer_fsid 2024-04-23T00:03:06.264 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 436, in _infer_image 2024-04-23T00:03:06.264 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 310, in _validate_fsid 2024-04-23T00:03:06.264 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 3380, in command_ceph_volume 2024-04-23T00:03:06.264 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-23T00:03:06.265 INFO:teuthology.orchestra.run.smithi067.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpoc6ppl5t:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-23T00:03:06.265 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-23T00:03:06.265 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is degraded 2024-04-23T00:03:06.265 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-23T00:03:06.265 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-23T00:03:06.265 INFO:teuthology.orchestra.run.smithi067.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-23T00:03:06.265 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-23T00:03:06.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:06.277+0000 7f492678c700 1 -- 172.21.15.67:0/681507475 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f490806ecc0 msgr2=0x7f4908071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:06.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:06.277+0000 7f492678c700 1 --2- 172.21.15.67:0/681507475 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f490806ecc0 0x7f4908071180 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f4910005950 tx=0x7f491000b410 comp rx=0 tx=0).stop 2024-04-23T00:03:06.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:06.277+0000 7f492678c700 1 -- 172.21.15.67:0/681507475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4920106cd0 msgr2=0x7f4920100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:06.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:06.277+0000 7f492678c700 1 --2- 172.21.15.67:0/681507475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4920106cd0 0x7f4920100230 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f4914006010 tx=0x7f491400bd70 comp rx=0 tx=0).stop 2024-04-23T00:03:06.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:06.278+0000 7f492678c700 1 -- 172.21.15.67:0/681507475 shutdown_connections 2024-04-23T00:03:06.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:06.278+0000 7f492678c700 1 --2- 172.21.15.67:0/681507475 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f490806ecc0 0x7f4908071180 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:06.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:06.278+0000 7f492678c700 1 --2- 172.21.15.67:0/681507475 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4920068580 0x7f49200ffcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:06.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:06.278+0000 7f492678c700 1 --2- 172.21.15.67:0/681507475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4920106cd0 0x7f4920100230 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:06.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:06.278+0000 7f492678c700 1 -- 172.21.15.67:0/681507475 >> 172.21.15.67:0/681507475 conn(0x7f4920075670 msgr2=0x7f49200feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:03:06.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:06.278+0000 7f492678c700 1 -- 172.21.15.67:0/681507475 shutdown_connections 2024-04-23T00:03:06.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:06.278+0000 7f492678c700 1 -- 172.21.15.67:0/681507475 wait complete. 2024-04-23T00:03:06.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:06 smithi134 ceph-mon[85630]: pgmap v1103: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:03:06.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:06 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3635866675' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-23T00:03:06.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:06 smithi134 ceph-mon[85630]: from='client.35992 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:03:06.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:06 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/681507475' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T00:03:06.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:06 smithi067 ceph-mon[112438]: pgmap v1103: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:03:06.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:06 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3635866675' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-23T00:03:06.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:06 smithi067 ceph-mon[112438]: from='client.35992 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:03:06.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:06 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/681507475' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T00:03:08.731 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:08 smithi134 ceph-mon[85630]: pgmap v1104: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:03:08.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:08 smithi067 ceph-mon[112438]: pgmap v1104: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:03:10.731 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:10 smithi134 ceph-mon[85630]: pgmap v1105: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:03:10.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:10 smithi067 ceph-mon[112438]: pgmap v1105: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:03:11.731 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:03:11.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:03:12.731 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:12 smithi134 ceph-mon[85630]: pgmap v1106: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:03:12.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:12 smithi067 ceph-mon[112438]: pgmap v1106: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:03:14.731 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:14 smithi134 ceph-mon[85630]: pgmap v1107: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:03:14.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:14 smithi067 ceph-mon[112438]: pgmap v1107: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:03:16.731 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:16 smithi134 ceph-mon[85630]: pgmap v1108: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:03:16.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:16 smithi067 ceph-mon[112438]: pgmap v1108: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:03:18.731 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:18 smithi134 ceph-mon[85630]: pgmap v1109: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:03:18.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:18 smithi067 ceph-mon[112438]: pgmap v1109: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:03:20.731 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:20 smithi134 ceph-mon[85630]: pgmap v1110: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:03:20.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:20 smithi067 ceph-mon[112438]: pgmap v1110: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:03:22.731 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:22 smithi134 ceph-mon[85630]: pgmap v1111: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:03:22.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:22 smithi067 ceph-mon[112438]: pgmap v1111: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:03:24.731 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:24 smithi134 ceph-mon[85630]: pgmap v1112: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:03:24.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:24 smithi067 ceph-mon[112438]: pgmap v1112: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:03:26.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:26 smithi134 ceph-mon[85630]: pgmap v1113: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:03:26.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T00:03:26.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T00:03:26.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:26 smithi134 ceph-mon[85630]: pgmap v1114: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:03:26.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:26 smithi134 ceph-mon[85630]: pgmap v1115: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:03:26.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:26.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T00:03:26.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:26.732 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:26.733 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:26.733 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:26.733 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:26.733 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:26 smithi134 ceph-mon[85630]: Upgrade: Setting container_image for all mds 2024-04-23T00:03:26.733 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:26.733 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi067.aqlxpv"}]: dispatch 2024-04-23T00:03:26.733 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi067.aqlxpv"}]': finished 2024-04-23T00:03:26.733 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi067.xyazmh"}]: dispatch 2024-04-23T00:03:26.733 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi067.xyazmh"}]': finished 2024-04-23T00:03:26.733 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi134.owipad"}]: dispatch 2024-04-23T00:03:26.733 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi134.owipad"}]': finished 2024-04-23T00:03:26.733 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi134.vuxbsj"}]: dispatch 2024-04-23T00:03:26.733 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi134.vuxbsj"}]': finished 2024-04-23T00:03:26.733 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:03:26.733 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:26 smithi134 ceph-mon[85630]: Upgrade: Setting filesystem cephfs Joinable 2024-04-23T00:03:26.734 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-04-23T00:03:26.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:26 smithi067 ceph-mon[112438]: pgmap v1113: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:03:26.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T00:03:26.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T00:03:26.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:26 smithi067 ceph-mon[112438]: pgmap v1114: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:03:26.783 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:26 smithi067 ceph-mon[112438]: pgmap v1115: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-23T00:03:26.783 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:26.783 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T00:03:26.783 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:26.783 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:26.783 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:26.783 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:26.783 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:26.783 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:26 smithi067 ceph-mon[112438]: Upgrade: Setting container_image for all mds 2024-04-23T00:03:26.783 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:26.783 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi067.aqlxpv"}]: dispatch 2024-04-23T00:03:26.783 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi067.aqlxpv"}]': finished 2024-04-23T00:03:26.783 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi067.xyazmh"}]: dispatch 2024-04-23T00:03:26.783 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi067.xyazmh"}]': finished 2024-04-23T00:03:26.784 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi134.owipad"}]: dispatch 2024-04-23T00:03:26.784 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi134.owipad"}]': finished 2024-04-23T00:03:26.784 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi134.vuxbsj"}]: dispatch 2024-04-23T00:03:26.784 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi134.vuxbsj"}]': finished 2024-04-23T00:03:26.784 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:03:26.784 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:26 smithi067 ceph-mon[112438]: Upgrade: Setting filesystem cephfs Joinable 2024-04-23T00:03:26.784 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-04-23T00:03:28.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:28 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-04-23T00:03:28.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:28 smithi134 ceph-mon[85630]: Standby daemon mds.cephfs.smithi067.aqlxpv assigned to filesystem cephfs as rank 0 2024-04-23T00:03:28.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:28 smithi134 ceph-mon[85630]: Standby daemon mds.cephfs.smithi067.xyazmh assigned to filesystem cephfs as rank 1 2024-04-23T00:03:28.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:28 smithi134 ceph-mon[85630]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-23T00:03:28.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:28 smithi134 ceph-mon[85630]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-23T00:03:28.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:28 smithi134 ceph-mon[85630]: Upgrade: Enabling allow_standby_replay on filesystem cephfs 2024-04-23T00:03:28.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:28 smithi134 ceph-mon[85630]: fsmap cephfs:2/2 {0=cephfs.smithi067.aqlxpv=up:replay,1=cephfs.smithi067.xyazmh=up:replay} 2 up:standby 2024-04-23T00:03:28.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:28 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]: dispatch 2024-04-23T00:03:28.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:28 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-04-23T00:03:28.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:28 smithi067 ceph-mon[112438]: Standby daemon mds.cephfs.smithi067.aqlxpv assigned to filesystem cephfs as rank 0 2024-04-23T00:03:28.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:28 smithi067 ceph-mon[112438]: Standby daemon mds.cephfs.smithi067.xyazmh assigned to filesystem cephfs as rank 1 2024-04-23T00:03:28.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:28 smithi067 ceph-mon[112438]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-23T00:03:28.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:28 smithi067 ceph-mon[112438]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-23T00:03:28.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:28 smithi067 ceph-mon[112438]: Upgrade: Enabling allow_standby_replay on filesystem cephfs 2024-04-23T00:03:28.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:28 smithi067 ceph-mon[112438]: fsmap cephfs:2/2 {0=cephfs.smithi067.aqlxpv=up:replay,1=cephfs.smithi067.xyazmh=up:replay} 2 up:standby 2024-04-23T00:03:28.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:28 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]: dispatch 2024-04-23T00:03:29.282 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:29 smithi067 ceph-mon[112438]: pgmap v1116: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.8 MiB/s rd, 1 op/s 2024-04-23T00:03:29.282 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:29 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]': finished 2024-04-23T00:03:29.282 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:29 smithi067 ceph-mon[112438]: fsmap cephfs:2/2 {0=cephfs.smithi067.aqlxpv=up:replay,1=cephfs.smithi067.xyazmh=up:replay} 2 up:standby 2024-04-23T00:03:29.282 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:29 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:29.282 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:29 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:29.282 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:29 smithi067 ceph-mon[112438]: Upgrade: Setting container_image for all rgw 2024-04-23T00:03:29.282 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:29 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:29.283 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:29 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:29.283 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:29 smithi067 ceph-mon[112438]: Upgrade: Setting container_image for all rbd-mirror 2024-04-23T00:03:29.283 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:29 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:29.283 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:29 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:29.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:29 smithi134 ceph-mon[85630]: pgmap v1116: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.8 MiB/s rd, 1 op/s 2024-04-23T00:03:29.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:29 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]': finished 2024-04-23T00:03:29.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:29 smithi134 ceph-mon[85630]: fsmap cephfs:2/2 {0=cephfs.smithi067.aqlxpv=up:replay,1=cephfs.smithi067.xyazmh=up:replay} 2 up:standby 2024-04-23T00:03:29.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:29 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:29.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:29 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:29.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:29 smithi134 ceph-mon[85630]: Upgrade: Setting container_image for all rgw 2024-04-23T00:03:29.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:29 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:29.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:29 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:29.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:29 smithi134 ceph-mon[85630]: Upgrade: Setting container_image for all rbd-mirror 2024-04-23T00:03:29.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:29 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:29.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:29 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:31.082 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:31 smithi067 ceph-mon[112438]: pgmap v1117: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 10 MiB/s rd, 4 op/s 2024-04-23T00:03:31.082 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:31 smithi067 ceph-mon[112438]: Upgrade: Updating ceph-exporter.smithi067 (1/2) 2024-04-23T00:03:31.082 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:31 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:31.082 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:31 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi067", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-23T00:03:31.082 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:31 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T00:03:31.082 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:31 smithi067 ceph-mon[112438]: Deploying daemon ceph-exporter.smithi067 on smithi067 2024-04-23T00:03:31.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:31 smithi134 ceph-mon[85630]: pgmap v1117: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 10 MiB/s rd, 4 op/s 2024-04-23T00:03:31.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:31 smithi134 ceph-mon[85630]: Upgrade: Updating ceph-exporter.smithi067 (1/2) 2024-04-23T00:03:31.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:31 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:31.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:31 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi067", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-23T00:03:31.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:31 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T00:03:31.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:31 smithi134 ceph-mon[85630]: Deploying daemon ceph-exporter.smithi067 on smithi067 2024-04-23T00:03:33.348 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:33 smithi067 ceph-mon[112438]: pgmap v1118: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 22 MiB/s rd, 8 op/s 2024-04-23T00:03:33.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:33 smithi134 ceph-mon[85630]: pgmap v1118: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 22 MiB/s rd, 8 op/s 2024-04-23T00:03:35.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:35 smithi134 ceph-mon[85630]: pgmap v1119: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 29 MiB/s rd, 10 op/s 2024-04-23T00:03:35.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:35 smithi134 ceph-mon[85630]: mds.? [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] up:resolve 2024-04-23T00:03:35.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:35 smithi134 ceph-mon[85630]: mds.? [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] up:resolve 2024-04-23T00:03:35.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:35 smithi134 ceph-mon[85630]: fsmap cephfs:2/2 {0=cephfs.smithi067.aqlxpv=up:resolve,1=cephfs.smithi067.xyazmh=up:resolve} 2 up:standby 2024-04-23T00:03:35.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:35 smithi067 ceph-mon[112438]: pgmap v1119: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 29 MiB/s rd, 10 op/s 2024-04-23T00:03:35.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:35 smithi067 ceph-mon[112438]: mds.? [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] up:resolve 2024-04-23T00:03:35.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:35 smithi067 ceph-mon[112438]: mds.? [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] up:resolve 2024-04-23T00:03:35.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:35 smithi067 ceph-mon[112438]: fsmap cephfs:2/2 {0=cephfs.smithi067.aqlxpv=up:resolve,1=cephfs.smithi067.xyazmh=up:resolve} 2 up:standby 2024-04-23T00:03:36.223 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:36 smithi067 ceph-mon[112438]: reconnect by client.24321 192.168.0.1:0/919997813 after 0.000999978 2024-04-23T00:03:36.223 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:36 smithi067 ceph-mon[112438]: reconnect by client.24321 192.168.0.1:0/919997813 after 0.00199995 2024-04-23T00:03:36.223 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:36 smithi067 ceph-mon[112438]: reconnect by client.14536 192.168.0.1:0/523873030 after 0.00199995 2024-04-23T00:03:36.224 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:36 smithi067 ceph-mon[112438]: reconnect by client.14536 192.168.0.1:0/523873030 after 0.00199995 2024-04-23T00:03:36.224 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:36 smithi067 ceph-mon[112438]: mds.? [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] up:reconnect 2024-04-23T00:03:36.224 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:36 smithi067 ceph-mon[112438]: mds.? [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] up:reconnect 2024-04-23T00:03:36.224 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:36 smithi067 ceph-mon[112438]: fsmap cephfs:2/2 {0=cephfs.smithi067.aqlxpv=up:reconnect,1=cephfs.smithi067.xyazmh=up:reconnect} 2 up:standby 2024-04-23T00:03:36.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.406+0000 7f98366a2700 1 -- 172.21.15.67:0/3503001087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9830106cd0 msgr2=0x7f98301070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:36.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.406+0000 7f98366a2700 1 --2- 172.21.15.67:0/3503001087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9830106cd0 0x7f98301070b0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f9824009910 tx=0x7f9824009c20 comp rx=0 tx=0).stop 2024-04-23T00:03:36.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.406+0000 7f98366a2700 1 -- 172.21.15.67:0/3503001087 shutdown_connections 2024-04-23T00:03:36.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.406+0000 7f98366a2700 1 --2- 172.21.15.67:0/3503001087 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9830068580 0x7f98300689e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:36.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.406+0000 7f98366a2700 1 --2- 172.21.15.67:0/3503001087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9830106cd0 0x7f98301070b0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:36.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.406+0000 7f98366a2700 1 -- 172.21.15.67:0/3503001087 >> 172.21.15.67:0/3503001087 conn(0x7f9830075670 msgr2=0x7f9830075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:03:36.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.406+0000 7f98366a2700 1 -- 172.21.15.67:0/3503001087 shutdown_connections 2024-04-23T00:03:36.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.406+0000 7f98366a2700 1 -- 172.21.15.67:0/3503001087 wait complete. 2024-04-23T00:03:36.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.407+0000 7f98366a2700 1 Processor -- start 2024-04-23T00:03:36.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.407+0000 7f98366a2700 1 -- start start 2024-04-23T00:03:36.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.407+0000 7f98366a2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9830068580 0x7f983019c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:36.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.407+0000 7f98366a2700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9830106cd0 0x7f983019cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:36.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.407+0000 7f98366a2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983019d230 con 0x7f9830068580 2024-04-23T00:03:36.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.407+0000 7f98366a2700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98301966e0 con 0x7f9830106cd0 2024-04-23T00:03:36.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.408+0000 7f982ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9830068580 0x7f983019c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:36.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.408+0000 7f982ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9830068580 0x7f983019c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55584/0 (socket says 172.21.15.67:55584) 2024-04-23T00:03:36.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.408+0000 7f982ffff700 1 -- 172.21.15.67:0/1413273579 learned_addr learned my addr 172.21.15.67:0/1413273579 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:03:36.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.408+0000 7f982ffff700 1 -- 172.21.15.67:0/1413273579 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9830106cd0 msgr2=0x7f983019cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:36.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.408+0000 7f982ffff700 1 --2- 172.21.15.67:0/1413273579 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9830106cd0 0x7f983019cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:36.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.408+0000 7f982ffff700 1 -- 172.21.15.67:0/1413273579 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98240095f0 con 0x7f9830068580 2024-04-23T00:03:36.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.409+0000 7f982ffff700 1 --2- 172.21.15.67:0/1413273579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9830068580 0x7f983019c660 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f9824006010 tx=0x7f982400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:03:36.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.409+0000 7f982d7fa700 1 -- 172.21.15.67:0/1413273579 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f982401d070 con 0x7f9830068580 2024-04-23T00:03:36.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.409+0000 7f98366a2700 1 -- 172.21.15.67:0/1413273579 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9830196960 con 0x7f9830068580 2024-04-23T00:03:36.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.409+0000 7f982d7fa700 1 -- 172.21.15.67:0/1413273579 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f982400f460 con 0x7f9830068580 2024-04-23T00:03:36.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.409+0000 7f982d7fa700 1 -- 172.21.15.67:0/1413273579 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9824017630 con 0x7f9830068580 2024-04-23T00:03:36.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.409+0000 7f98366a2700 1 -- 172.21.15.67:0/1413273579 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9830196e50 con 0x7f9830068580 2024-04-23T00:03:36.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.410+0000 7f98366a2700 1 -- 172.21.15.67:0/1413273579 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98301091f0 con 0x7f9830068580 2024-04-23T00:03:36.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.411+0000 7f982d7fa700 1 -- 172.21.15.67:0/1413273579 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9824003a60 con 0x7f9830068580 2024-04-23T00:03:36.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.411+0000 7f982d7fa700 1 --2- 172.21.15.67:0/1413273579 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f981806ecc0 0x7f9818071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:36.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.411+0000 7f982d7fa700 1 -- 172.21.15.67:0/1413273579 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f9824090480 con 0x7f9830068580 2024-04-23T00:03:36.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.411+0000 7f982f7fe700 1 --2- 172.21.15.67:0/1413273579 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f981806ecc0 0x7f9818071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:36.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.412+0000 7f982f7fe700 1 --2- 172.21.15.67:0/1413273579 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f981806ecc0 0x7f9818071180 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f9820005950 tx=0x7f98200058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:03:36.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.416+0000 7f982d7fa700 1 -- 172.21.15.67:0/1413273579 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f982405cbf0 con 0x7f9830068580 2024-04-23T00:03:36.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:36 smithi134 ceph-mon[85630]: reconnect by client.24321 192.168.0.1:0/919997813 after 0.000999978 2024-04-23T00:03:36.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:36 smithi134 ceph-mon[85630]: reconnect by client.24321 192.168.0.1:0/919997813 after 0.00199995 2024-04-23T00:03:36.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:36 smithi134 ceph-mon[85630]: reconnect by client.14536 192.168.0.1:0/523873030 after 0.00199995 2024-04-23T00:03:36.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:36 smithi134 ceph-mon[85630]: reconnect by client.14536 192.168.0.1:0/523873030 after 0.00199995 2024-04-23T00:03:36.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:36 smithi134 ceph-mon[85630]: mds.? [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] up:reconnect 2024-04-23T00:03:36.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:36 smithi134 ceph-mon[85630]: mds.? [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] up:reconnect 2024-04-23T00:03:36.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:36 smithi134 ceph-mon[85630]: fsmap cephfs:2/2 {0=cephfs.smithi067.aqlxpv=up:reconnect,1=cephfs.smithi067.xyazmh=up:reconnect} 2 up:standby 2024-04-23T00:03:36.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.667+0000 7f98366a2700 1 -- 172.21.15.67:0/1413273579 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9830197070 con 0x7f981806ecc0 2024-04-23T00:03:36.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.668+0000 7f982d7fa700 1 -- 172.21.15.67:0/1413273579 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+441 (secure 0 0 0) 0x7f9830197070 con 0x7f981806ecc0 2024-04-23T00:03:36.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.670+0000 7f98366a2700 1 -- 172.21.15.67:0/1413273579 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f981806ecc0 msgr2=0x7f9818071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:36.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.670+0000 7f98366a2700 1 --2- 172.21.15.67:0/1413273579 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f981806ecc0 0x7f9818071180 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f9820005950 tx=0x7f98200058e0 comp rx=0 tx=0).stop 2024-04-23T00:03:36.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.670+0000 7f98366a2700 1 -- 172.21.15.67:0/1413273579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9830068580 msgr2=0x7f983019c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:36.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.670+0000 7f98366a2700 1 --2- 172.21.15.67:0/1413273579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9830068580 0x7f983019c660 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f9824006010 tx=0x7f982400be20 comp rx=0 tx=0).stop 2024-04-23T00:03:36.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.670+0000 7f98366a2700 1 -- 172.21.15.67:0/1413273579 shutdown_connections 2024-04-23T00:03:36.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.670+0000 7f98366a2700 1 --2- 172.21.15.67:0/1413273579 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f981806ecc0 0x7f9818071180 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:36.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.670+0000 7f98366a2700 1 --2- 172.21.15.67:0/1413273579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9830068580 0x7f983019c660 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:36.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.670+0000 7f98366a2700 1 --2- 172.21.15.67:0/1413273579 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9830106cd0 0x7f983019cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:36.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.670+0000 7f98366a2700 1 -- 172.21.15.67:0/1413273579 >> 172.21.15.67:0/1413273579 conn(0x7f9830075670 msgr2=0x7f98300fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:03:36.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.670+0000 7f98366a2700 1 -- 172.21.15.67:0/1413273579 shutdown_connections 2024-04-23T00:03:36.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.670+0000 7f98366a2700 1 -- 172.21.15.67:0/1413273579 wait complete. 2024-04-23T00:03:36.670 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-23T00:03:36.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.788+0000 7f9460b3a700 1 -- 172.21.15.67:0/3947830777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f945c0ffb40 msgr2=0x7f945c0fffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:36.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.788+0000 7f9460b3a700 1 --2- 172.21.15.67:0/3947830777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f945c0ffb40 0x7f945c0fffc0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f9450009910 tx=0x7f9450009c20 comp rx=0 tx=0).stop 2024-04-23T00:03:36.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.789+0000 7f9460b3a700 1 -- 172.21.15.67:0/3947830777 shutdown_connections 2024-04-23T00:03:36.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.789+0000 7f9460b3a700 1 --2- 172.21.15.67:0/3947830777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f945c0ffb40 0x7f945c0fffc0 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:36.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.789+0000 7f9460b3a700 1 --2- 172.21.15.67:0/3947830777 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f945c105bf0 0x7f945c105fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:36.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.789+0000 7f9460b3a700 1 -- 172.21.15.67:0/3947830777 >> 172.21.15.67:0/3947830777 conn(0x7f945c0fb610 msgr2=0x7f945c0fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:03:36.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.789+0000 7f9460b3a700 1 -- 172.21.15.67:0/3947830777 shutdown_connections 2024-04-23T00:03:36.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.789+0000 7f9460b3a700 1 -- 172.21.15.67:0/3947830777 wait complete. 2024-04-23T00:03:36.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.789+0000 7f9460b3a700 1 Processor -- start 2024-04-23T00:03:36.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.789+0000 7f9460b3a700 1 -- start start 2024-04-23T00:03:36.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.790+0000 7f9460b3a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f945c0ffb40 0x7f945c194420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:36.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.790+0000 7f9460b3a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f945c105bf0 0x7f945c194960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:36.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.790+0000 7f9460b3a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945c196540 con 0x7f945c105bf0 2024-04-23T00:03:36.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.790+0000 7f9460b3a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945c194ea0 con 0x7f945c0ffb40 2024-04-23T00:03:36.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.790+0000 7f9459d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f945c105bf0 0x7f945c194960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:36.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.790+0000 7f9459d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f945c105bf0 0x7f945c194960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55612/0 (socket says 172.21.15.67:55612) 2024-04-23T00:03:36.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.790+0000 7f9459d9b700 1 -- 172.21.15.67:0/390220959 learned_addr learned my addr 172.21.15.67:0/390220959 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:03:36.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.790+0000 7f945a59c700 1 --2- 172.21.15.67:0/390220959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f945c0ffb40 0x7f945c194420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:36.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.791+0000 7f945a59c700 1 -- 172.21.15.67:0/390220959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f945c105bf0 msgr2=0x7f945c194960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:36.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.791+0000 7f945a59c700 1 --2- 172.21.15.67:0/390220959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f945c105bf0 0x7f945c194960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:36.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.791+0000 7f945a59c700 1 -- 172.21.15.67:0/390220959 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94500095f0 con 0x7f945c0ffb40 2024-04-23T00:03:36.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.791+0000 7f9459d9b700 1 --2- 172.21.15.67:0/390220959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f945c105bf0 0x7f945c194960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T00:03:36.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.791+0000 7f945a59c700 1 --2- 172.21.15.67:0/390220959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f945c0ffb40 0x7f945c194420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f944c00e970 tx=0x7f944c00ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:03:36.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.791+0000 7f944b7fe700 1 -- 172.21.15.67:0/390220959 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f944c00c9d0 con 0x7f945c0ffb40 2024-04-23T00:03:36.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.791+0000 7f944b7fe700 1 -- 172.21.15.67:0/390220959 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f944c010430 con 0x7f945c0ffb40 2024-04-23T00:03:36.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.791+0000 7f9460b3a700 1 -- 172.21.15.67:0/390220959 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f945c195180 con 0x7f945c0ffb40 2024-04-23T00:03:36.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.791+0000 7f944b7fe700 1 -- 172.21.15.67:0/390220959 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f944c00c9d0 con 0x7f945c0ffb40 2024-04-23T00:03:36.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.791+0000 7f9460b3a700 1 -- 172.21.15.67:0/390220959 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f945c1a71f0 con 0x7f945c0ffb40 2024-04-23T00:03:36.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.792+0000 7f9460b3a700 1 -- 172.21.15.67:0/390220959 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f943c005320 con 0x7f945c0ffb40 2024-04-23T00:03:36.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.793+0000 7f944b7fe700 1 -- 172.21.15.67:0/390220959 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f944c00cb30 con 0x7f945c0ffb40 2024-04-23T00:03:36.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.794+0000 7f944b7fe700 1 --2- 172.21.15.67:0/390220959 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f944406ed80 0x7f9444071240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:36.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.794+0000 7f944b7fe700 1 -- 172.21.15.67:0/390220959 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f944c014070 con 0x7f945c0ffb40 2024-04-23T00:03:36.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.794+0000 7f9459d9b700 1 --2- 172.21.15.67:0/390220959 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f944406ed80 0x7f9444071240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:36.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.794+0000 7f9459d9b700 1 --2- 172.21.15.67:0/390220959 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f944406ed80 0x7f9444071240 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f945000b760 tx=0x7f9450005a90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:03:36.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:36.799+0000 7f944b7fe700 1 -- 172.21.15.67:0/390220959 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f944c054150 con 0x7f945c0ffb40 2024-04-23T00:03:37.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.041+0000 7f9460b3a700 1 -- 172.21.15.67:0/390220959 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f943c000bf0 con 0x7f944406ed80 2024-04-23T00:03:37.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.042+0000 7f944b7fe700 1 -- 172.21.15.67:0/390220959 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+441 (secure 0 0 0) 0x7f943c000bf0 con 0x7f944406ed80 2024-04-23T00:03:37.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.045+0000 7f9460b3a700 1 -- 172.21.15.67:0/390220959 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f944406ed80 msgr2=0x7f9444071240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:37.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.045+0000 7f9460b3a700 1 --2- 172.21.15.67:0/390220959 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f944406ed80 0x7f9444071240 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f945000b760 tx=0x7f9450005a90 comp rx=0 tx=0).stop 2024-04-23T00:03:37.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.045+0000 7f9460b3a700 1 -- 172.21.15.67:0/390220959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f945c0ffb40 msgr2=0x7f945c194420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:37.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.045+0000 7f9460b3a700 1 --2- 172.21.15.67:0/390220959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f945c0ffb40 0x7f945c194420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f944c00e970 tx=0x7f944c00ec80 comp rx=0 tx=0).stop 2024-04-23T00:03:37.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.045+0000 7f9460b3a700 1 -- 172.21.15.67:0/390220959 shutdown_connections 2024-04-23T00:03:37.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.045+0000 7f9460b3a700 1 --2- 172.21.15.67:0/390220959 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f944406ed80 0x7f9444071240 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:37.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.045+0000 7f9460b3a700 1 --2- 172.21.15.67:0/390220959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f945c0ffb40 0x7f945c194420 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:37.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.045+0000 7f9460b3a700 1 --2- 172.21.15.67:0/390220959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f945c105bf0 0x7f945c194960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:37.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.045+0000 7f9460b3a700 1 -- 172.21.15.67:0/390220959 >> 172.21.15.67:0/390220959 conn(0x7f945c0fb610 msgr2=0x7f945c103a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:03:37.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.045+0000 7f9460b3a700 1 -- 172.21.15.67:0/390220959 shutdown_connections 2024-04-23T00:03:37.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.045+0000 7f9460b3a700 1 -- 172.21.15.67:0/390220959 wait complete. 2024-04-23T00:03:37.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.190+0000 7fa1dd6c5700 1 -- 172.21.15.67:0/814746145 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1d00a4f80 msgr2=0x7fa1d00a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:37.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.190+0000 7fa1dd6c5700 1 --2- 172.21.15.67:0/814746145 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1d00a4f80 0x7fa1d00a5360 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8009910 tx=0x7fa1c8009c20 comp rx=0 tx=0).stop 2024-04-23T00:03:37.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.190+0000 7fa1dd6c5700 1 -- 172.21.15.67:0/814746145 shutdown_connections 2024-04-23T00:03:37.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.190+0000 7fa1dd6c5700 1 --2- 172.21.15.67:0/814746145 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa1d00a5930 0x7fa1d00a9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:37.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.190+0000 7fa1dd6c5700 1 --2- 172.21.15.67:0/814746145 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1d00a4f80 0x7fa1d00a5360 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:37.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.190+0000 7fa1dd6c5700 1 -- 172.21.15.67:0/814746145 >> 172.21.15.67:0/814746145 conn(0x7fa1d001a8a0 msgr2=0x7fa1d001acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:03:37.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.191+0000 7fa1dd6c5700 1 -- 172.21.15.67:0/814746145 shutdown_connections 2024-04-23T00:03:37.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.191+0000 7fa1dd6c5700 1 -- 172.21.15.67:0/814746145 wait complete. 2024-04-23T00:03:37.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.193+0000 7fa1dd6c5700 1 Processor -- start 2024-04-23T00:03:37.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.193+0000 7fa1dd6c5700 1 -- start start 2024-04-23T00:03:37.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.194+0000 7fa1dd6c5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1d00a4f80 0x7fa1d013a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:37.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.194+0000 7fa1dd6c5700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa1d00a5930 0x7fa1d013ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:37.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.194+0000 7fa1d6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1d00a4f80 0x7fa1d013a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:37.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.194+0000 7fa1dd6c5700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d013b210 con 0x7fa1d00a4f80 2024-04-23T00:03:37.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.194+0000 7fa1d6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1d00a4f80 0x7fa1d013a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55638/0 (socket says 172.21.15.67:55638) 2024-04-23T00:03:37.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.194+0000 7fa1d6ffd700 1 -- 172.21.15.67:0/285766289 learned_addr learned my addr 172.21.15.67:0/285766289 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:03:37.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.194+0000 7fa1dd6c5700 1 -- 172.21.15.67:0/285766289 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d013efa0 con 0x7fa1d00a5930 2024-04-23T00:03:37.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.195+0000 7fa1d6ffd700 1 -- 172.21.15.67:0/285766289 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa1d00a5930 msgr2=0x7fa1d013ab30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-23T00:03:37.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.195+0000 7fa1d6ffd700 1 --2- 172.21.15.67:0/285766289 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa1d00a5930 0x7fa1d013ab30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:37.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.195+0000 7fa1d6ffd700 1 -- 172.21.15.67:0/285766289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1c80095f0 con 0x7fa1d00a4f80 2024-04-23T00:03:37.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.195+0000 7fa1d6ffd700 1 --2- 172.21.15.67:0/285766289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1d00a4f80 0x7fa1d013a5f0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8009fd0 tx=0x7fa1c800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:03:37.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.195+0000 7fa1cffff700 1 -- 172.21.15.67:0/285766289 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1c801f070 con 0x7fa1d00a4f80 2024-04-23T00:03:37.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.195+0000 7fa1cffff700 1 -- 172.21.15.67:0/285766289 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa1c801abe0 con 0x7fa1d00a4f80 2024-04-23T00:03:37.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.195+0000 7fa1cffff700 1 -- 172.21.15.67:0/285766289 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1c8004f90 con 0x7fa1d00a4f80 2024-04-23T00:03:37.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.196+0000 7fa1dd6c5700 1 -- 172.21.15.67:0/285766289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1d013f220 con 0x7fa1d00a4f80 2024-04-23T00:03:37.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.196+0000 7fa1dd6c5700 1 -- 172.21.15.67:0/285766289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1d013f620 con 0x7fa1d00a4f80 2024-04-23T00:03:37.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.197+0000 7fa1dd6c5700 1 -- 172.21.15.67:0/285766289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1d00ad2d0 con 0x7fa1d00a4f80 2024-04-23T00:03:37.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.199+0000 7fa1cffff700 1 -- 172.21.15.67:0/285766289 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa1c801a720 con 0x7fa1d00a4f80 2024-04-23T00:03:37.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.199+0000 7fa1cffff700 1 --2- 172.21.15.67:0/285766289 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa1bc06ed20 0x7fa1bc0711e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:37.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.201+0000 7fa1cffff700 1 -- 172.21.15.67:0/285766289 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fa1c805fc30 con 0x7fa1d00a4f80 2024-04-23T00:03:37.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.202+0000 7fa1d67fc700 1 --2- 172.21.15.67:0/285766289 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa1bc06ed20 0x7fa1bc0711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:37.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.202+0000 7fa1d67fc700 1 --2- 172.21.15.67:0/285766289 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa1bc06ed20 0x7fa1bc0711e0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fa1d013bc10 tx=0x7fa1c4006c60 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:03:37.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.211+0000 7fa1cffff700 1 -- 172.21.15.67:0/285766289 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fa1c805b7d0 con 0x7fa1d00a4f80 2024-04-23T00:03:37.282 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:37 smithi067 ceph-mon[112438]: pgmap v1120: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 30 MiB/s rd, 10 op/s 2024-04-23T00:03:37.282 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:37 smithi067 ceph-mon[112438]: mds.? [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] up:rejoin 2024-04-23T00:03:37.282 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:37 smithi067 ceph-mon[112438]: mds.? [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] up:rejoin 2024-04-23T00:03:37.282 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:37 smithi067 ceph-mon[112438]: fsmap cephfs:2/2 {0=cephfs.smithi067.aqlxpv=up:rejoin,1=cephfs.smithi067.xyazmh=up:rejoin} 2 up:standby 2024-04-23T00:03:37.282 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:37 smithi067 ceph-mon[112438]: daemon mds.cephfs.smithi067.xyazmh is now active in filesystem cephfs as rank 1 2024-04-23T00:03:37.282 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:37 smithi067 ceph-mon[112438]: daemon mds.cephfs.smithi067.aqlxpv is now active in filesystem cephfs as rank 0 2024-04-23T00:03:37.418 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:37 smithi134 ceph-mon[85630]: pgmap v1120: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 30 MiB/s rd, 10 op/s 2024-04-23T00:03:37.419 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:37 smithi134 ceph-mon[85630]: mds.? [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] up:rejoin 2024-04-23T00:03:37.419 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:37 smithi134 ceph-mon[85630]: mds.? [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] up:rejoin 2024-04-23T00:03:37.419 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:37 smithi134 ceph-mon[85630]: fsmap cephfs:2/2 {0=cephfs.smithi067.aqlxpv=up:rejoin,1=cephfs.smithi067.xyazmh=up:rejoin} 2 up:standby 2024-04-23T00:03:37.419 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:37 smithi134 ceph-mon[85630]: daemon mds.cephfs.smithi067.xyazmh is now active in filesystem cephfs as rank 1 2024-04-23T00:03:37.419 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:37 smithi134 ceph-mon[85630]: daemon mds.cephfs.smithi067.aqlxpv is now active in filesystem cephfs as rank 0 2024-04-23T00:03:37.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.441+0000 7fa1dd6c5700 1 -- 172.21.15.67:0/285766289 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa1d00024b0 con 0x7fa1bc06ed20 2024-04-23T00:03:37.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.452+0000 7fa1cffff700 1 -- 172.21.15.67:0/285766289 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fa1d00024b0 con 0x7fa1bc06ed20 2024-04-23T00:03:37.441 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-23T00:03:37.441 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (37m) 15m ago 48m 24.4M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-23T00:03:37.442 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (48m) 15m ago 48m 10.0M - 18.2.1 5be31c24972a df887f62291e 2024-04-23T00:03:37.442 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (47m) 15m ago 47m 10.0M - 18.2.1 5be31c24972a ad89190b5400 2024-04-23T00:03:37.442 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (34m) 15m ago 48m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-23T00:03:37.442 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (34m) 15m ago 47m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-23T00:03:37.442 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (36m) 15m ago 48m 81.2M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-23T00:03:37.442 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (31m) 15m ago 43m 19.4M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-23T00:03:37.442 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (30m) 15m ago 43m 20.3M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-23T00:03:37.442 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (30m) 15m ago 43m 20.1M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-23T00:03:37.442 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (15m) 15m ago 43m 14.0M - 19.0.0-3252-g43be0201 a0434699c1d7 46d3d7e86fee 2024-04-23T00:03:37.442 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (39m) 15m ago 50m 514M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-23T00:03:37.442 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (38m) 15m ago 47m 459M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-23T00:03:37.442 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (35m) 15m ago 50m 74.2M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-23T00:03:37.442 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (35m) 15m ago 47m 54.4M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-23T00:03:37.443 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (38m) 15m ago 48m 15.2M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-23T00:03:37.443 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (37m) 15m ago 47m 16.0M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-23T00:03:37.443 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (34m) 15m ago 46m 102M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-23T00:03:37.443 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (33m) 15m ago 46m 128M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-23T00:03:37.443 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (33m) 15m ago 45m 83.8M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-23T00:03:37.443 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (32m) 15m ago 45m 101M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-23T00:03:37.443 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (31m) 15m ago 45m 86.2M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-23T00:03:37.443 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (31m) 15m ago 44m 81.8M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-23T00:03:37.443 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (37m) 15m ago 48m 78.7M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-23T00:03:37.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.455+0000 7fa1dd6c5700 1 -- 172.21.15.67:0/285766289 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa1bc06ed20 msgr2=0x7fa1bc0711e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:37.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.455+0000 7fa1dd6c5700 1 --2- 172.21.15.67:0/285766289 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa1bc06ed20 0x7fa1bc0711e0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fa1d013bc10 tx=0x7fa1c4006c60 comp rx=0 tx=0).stop 2024-04-23T00:03:37.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.455+0000 7fa1dd6c5700 1 -- 172.21.15.67:0/285766289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1d00a4f80 msgr2=0x7fa1d013a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:37.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.455+0000 7fa1dd6c5700 1 --2- 172.21.15.67:0/285766289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1d00a4f80 0x7fa1d013a5f0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8009fd0 tx=0x7fa1c800be20 comp rx=0 tx=0).stop 2024-04-23T00:03:37.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.455+0000 7fa1dd6c5700 1 -- 172.21.15.67:0/285766289 shutdown_connections 2024-04-23T00:03:37.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.455+0000 7fa1dd6c5700 1 --2- 172.21.15.67:0/285766289 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fa1bc06ed20 0x7fa1bc0711e0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:37.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.455+0000 7fa1dd6c5700 1 --2- 172.21.15.67:0/285766289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1d00a4f80 0x7fa1d013a5f0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:37.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.455+0000 7fa1dd6c5700 1 --2- 172.21.15.67:0/285766289 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa1d00a5930 0x7fa1d013ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:37.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.455+0000 7fa1dd6c5700 1 -- 172.21.15.67:0/285766289 >> 172.21.15.67:0/285766289 conn(0x7fa1d001a8a0 msgr2=0x7fa1d00a3660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:03:37.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.455+0000 7fa1dd6c5700 1 -- 172.21.15.67:0/285766289 shutdown_connections 2024-04-23T00:03:37.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.455+0000 7fa1dd6c5700 1 -- 172.21.15.67:0/285766289 wait complete. 2024-04-23T00:03:37.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.570+0000 7fe559c08700 1 -- 172.21.15.67:0/2037009031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5540ffc20 msgr2=0x7fe554100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:37.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.571+0000 7fe559c08700 1 --2- 172.21.15.67:0/2037009031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5540ffc20 0x7fe554100080 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fe548009910 tx=0x7fe548009c20 comp rx=0 tx=0).stop 2024-04-23T00:03:37.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.571+0000 7fe559c08700 1 -- 172.21.15.67:0/2037009031 shutdown_connections 2024-04-23T00:03:37.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.571+0000 7fe559c08700 1 --2- 172.21.15.67:0/2037009031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5540ffc20 0x7fe554100080 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:37.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.571+0000 7fe559c08700 1 --2- 172.21.15.67:0/2037009031 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe554105bd0 0x7fe554105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:37.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.571+0000 7fe559c08700 1 -- 172.21.15.67:0/2037009031 >> 172.21.15.67:0/2037009031 conn(0x7fe554075670 msgr2=0x7fe554075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:03:37.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.571+0000 7fe559c08700 1 -- 172.21.15.67:0/2037009031 shutdown_connections 2024-04-23T00:03:37.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.571+0000 7fe559c08700 1 -- 172.21.15.67:0/2037009031 wait complete. 2024-04-23T00:03:37.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.572+0000 7fe559c08700 1 Processor -- start 2024-04-23T00:03:37.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.572+0000 7fe559c08700 1 -- start start 2024-04-23T00:03:37.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.572+0000 7fe559c08700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe5540ffc20 0x7fe554195550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:37.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.572+0000 7fe559c08700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe554105bd0 0x7fe554195a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:37.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.572+0000 7fe559c08700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe554195fd0 con 0x7fe554105bd0 2024-04-23T00:03:37.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.572+0000 7fe559c08700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe554196140 con 0x7fe5540ffc20 2024-04-23T00:03:37.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.572+0000 7fe552ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe554105bd0 0x7fe554195a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:37.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.572+0000 7fe5537fe700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe5540ffc20 0x7fe554195550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:37.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.572+0000 7fe552ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe554105bd0 0x7fe554195a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55670/0 (socket says 172.21.15.67:55670) 2024-04-23T00:03:37.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.572+0000 7fe552ffd700 1 -- 172.21.15.67:0/3493728713 learned_addr learned my addr 172.21.15.67:0/3493728713 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:03:37.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.573+0000 7fe552ffd700 1 -- 172.21.15.67:0/3493728713 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe5540ffc20 msgr2=0x7fe554195550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:37.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.573+0000 7fe552ffd700 1 --2- 172.21.15.67:0/3493728713 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe5540ffc20 0x7fe554195550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:37.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.573+0000 7fe552ffd700 1 -- 172.21.15.67:0/3493728713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5480095f0 con 0x7fe554105bd0 2024-04-23T00:03:37.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.573+0000 7fe5537fe700 1 --2- 172.21.15.67:0/3493728713 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe5540ffc20 0x7fe554195550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T00:03:37.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.573+0000 7fe552ffd700 1 --2- 172.21.15.67:0/3493728713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe554105bd0 0x7fe554195a90 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fe548006010 tx=0x7fe54800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:03:37.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.573+0000 7fe550ff9700 1 -- 172.21.15.67:0/3493728713 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe54801d070 con 0x7fe554105bd0 2024-04-23T00:03:37.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.573+0000 7fe550ff9700 1 -- 172.21.15.67:0/3493728713 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe548005dc0 con 0x7fe554105bd0 2024-04-23T00:03:37.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.573+0000 7fe550ff9700 1 -- 172.21.15.67:0/3493728713 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe548017400 con 0x7fe554105bd0 2024-04-23T00:03:37.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.573+0000 7fe559c08700 1 -- 172.21.15.67:0/3493728713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5541963c0 con 0x7fe554105bd0 2024-04-23T00:03:37.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.573+0000 7fe559c08700 1 -- 172.21.15.67:0/3493728713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5541a2df0 con 0x7fe554105bd0 2024-04-23T00:03:37.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.575+0000 7fe559c08700 1 -- 172.21.15.67:0/3493728713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5541080d0 con 0x7fe554105bd0 2024-04-23T00:03:37.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.575+0000 7fe550ff9700 1 -- 172.21.15.67:0/3493728713 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe548003a60 con 0x7fe554105bd0 2024-04-23T00:03:37.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.576+0000 7fe550ff9700 1 --2- 172.21.15.67:0/3493728713 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe53c06ecc0 0x7fe53c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:37.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.576+0000 7fe550ff9700 1 -- 172.21.15.67:0/3493728713 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fe548090200 con 0x7fe554105bd0 2024-04-23T00:03:37.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.576+0000 7fe5537fe700 1 --2- 172.21.15.67:0/3493728713 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe53c06ecc0 0x7fe53c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:37.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.577+0000 7fe5537fe700 1 --2- 172.21.15.67:0/3493728713 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe53c06ecc0 0x7fe53c071180 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fe544005950 tx=0x7fe54400b410 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:03:37.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.581+0000 7fe550ff9700 1 -- 172.21.15.67:0/3493728713 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe548058ec0 con 0x7fe554105bd0 2024-04-23T00:03:37.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.886+0000 7fe559c08700 1 -- 172.21.15.67:0/3493728713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe554196390 con 0x7fe554105bd0 2024-04-23T00:03:37.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.887+0000 7fe550ff9700 1 -- 172.21.15.67:0/3493728713 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fe54805c580 con 0x7fe554105bd0 2024-04-23T00:03:37.876 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-23T00:03:37.876 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-23T00:03:37.876 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-23T00:03:37.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-23T00:03:37.876 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-23T00:03:37.877 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-23T00:03:37.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-23T00:03:37.877 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-23T00:03:37.877 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-23T00:03:37.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-23T00:03:37.877 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-23T00:03:37.877 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 4 2024-04-23T00:03:37.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-23T00:03:37.878 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-23T00:03:37.878 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 14 2024-04-23T00:03:37.878 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-23T00:03:37.878 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-23T00:03:37.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.889+0000 7fe559c08700 1 -- 172.21.15.67:0/3493728713 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe53c06ecc0 msgr2=0x7fe53c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:37.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.889+0000 7fe559c08700 1 --2- 172.21.15.67:0/3493728713 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe53c06ecc0 0x7fe53c071180 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fe544005950 tx=0x7fe54400b410 comp rx=0 tx=0).stop 2024-04-23T00:03:37.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.889+0000 7fe559c08700 1 -- 172.21.15.67:0/3493728713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe554105bd0 msgr2=0x7fe554195a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:37.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.889+0000 7fe559c08700 1 --2- 172.21.15.67:0/3493728713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe554105bd0 0x7fe554195a90 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fe548006010 tx=0x7fe54800bd70 comp rx=0 tx=0).stop 2024-04-23T00:03:37.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.889+0000 7fe559c08700 1 -- 172.21.15.67:0/3493728713 shutdown_connections 2024-04-23T00:03:37.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.889+0000 7fe559c08700 1 --2- 172.21.15.67:0/3493728713 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe53c06ecc0 0x7fe53c071180 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:37.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.889+0000 7fe559c08700 1 --2- 172.21.15.67:0/3493728713 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe5540ffc20 0x7fe554195550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:37.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.889+0000 7fe559c08700 1 --2- 172.21.15.67:0/3493728713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe554105bd0 0x7fe554195a90 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:37.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.889+0000 7fe559c08700 1 -- 172.21.15.67:0/3493728713 >> 172.21.15.67:0/3493728713 conn(0x7fe554075670 msgr2=0x7fe5540fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:03:37.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.889+0000 7fe559c08700 1 -- 172.21.15.67:0/3493728713 shutdown_connections 2024-04-23T00:03:37.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:37.889+0000 7fe559c08700 1 -- 172.21.15.67:0/3493728713 wait complete. 2024-04-23T00:03:37.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.002+0000 7fca5e2f1700 1 -- 172.21.15.67:0/2394527065 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca58101840 msgr2=0x7fca58105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:37.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.002+0000 7fca5e2f1700 1 --2- 172.21.15.67:0/2394527065 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca58101840 0x7fca58105890 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fca4c009910 tx=0x7fca4c009c20 comp rx=0 tx=0).stop 2024-04-23T00:03:37.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.002+0000 7fca5e2f1700 1 -- 172.21.15.67:0/2394527065 shutdown_connections 2024-04-23T00:03:37.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.002+0000 7fca5e2f1700 1 --2- 172.21.15.67:0/2394527065 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca58101840 0x7fca58105890 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:37.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.002+0000 7fca5e2f1700 1 --2- 172.21.15.67:0/2394527065 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fca58100e90 0x7fca58101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:37.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.002+0000 7fca5e2f1700 1 -- 172.21.15.67:0/2394527065 >> 172.21.15.67:0/2394527065 conn(0x7fca580fc740 msgr2=0x7fca580feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:03:37.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.002+0000 7fca5e2f1700 1 -- 172.21.15.67:0/2394527065 shutdown_connections 2024-04-23T00:03:37.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.002+0000 7fca5e2f1700 1 -- 172.21.15.67:0/2394527065 wait complete. 2024-04-23T00:03:37.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.002+0000 7fca5e2f1700 1 Processor -- start 2024-04-23T00:03:37.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.002+0000 7fca5e2f1700 1 -- start start 2024-04-23T00:03:37.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.003+0000 7fca5e2f1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca58100e90 0x7fca5819c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:37.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.003+0000 7fca5e2f1700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fca58101840 0x7fca5819cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:37.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.003+0000 7fca5e2f1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5819d240 con 0x7fca58100e90 2024-04-23T00:03:37.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.003+0000 7fca5e2f1700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca581966f0 con 0x7fca58101840 2024-04-23T00:03:37.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.003+0000 7fca57fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca58100e90 0x7fca5819c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:37.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.003+0000 7fca57fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca58100e90 0x7fca5819c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55694/0 (socket says 172.21.15.67:55694) 2024-04-23T00:03:37.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.003+0000 7fca57fff700 1 -- 172.21.15.67:0/2453271858 learned_addr learned my addr 172.21.15.67:0/2453271858 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:03:37.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.003+0000 7fca577fe700 1 --2- 172.21.15.67:0/2453271858 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fca58101840 0x7fca5819cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:37.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.003+0000 7fca57fff700 1 -- 172.21.15.67:0/2453271858 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fca58101840 msgr2=0x7fca5819cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:37.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.003+0000 7fca57fff700 1 --2- 172.21.15.67:0/2453271858 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fca58101840 0x7fca5819cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:37.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.003+0000 7fca57fff700 1 -- 172.21.15.67:0/2453271858 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca4c0095f0 con 0x7fca58100e90 2024-04-23T00:03:37.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.004+0000 7fca57fff700 1 --2- 172.21.15.67:0/2453271858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca58100e90 0x7fca5819c670 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fca4800e920 tx=0x7fca4800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:03:37.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.004+0000 7fca557fa700 1 -- 172.21.15.67:0/2453271858 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca4800c9d0 con 0x7fca58100e90 2024-04-23T00:03:37.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.004+0000 7fca557fa700 1 -- 172.21.15.67:0/2453271858 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fca48010430 con 0x7fca58100e90 2024-04-23T00:03:37.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.004+0000 7fca557fa700 1 -- 172.21.15.67:0/2453271858 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca48018630 con 0x7fca58100e90 2024-04-23T00:03:37.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.004+0000 7fca5e2f1700 1 -- 172.21.15.67:0/2453271858 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca581969d0 con 0x7fca58100e90 2024-04-23T00:03:37.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.004+0000 7fca5e2f1700 1 -- 172.21.15.67:0/2453271858 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca58196ea0 con 0x7fca58100e90 2024-04-23T00:03:37.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.005+0000 7fca5e2f1700 1 -- 172.21.15.67:0/2453271858 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca58109290 con 0x7fca58100e90 2024-04-23T00:03:37.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.006+0000 7fca557fa700 1 -- 172.21.15.67:0/2453271858 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fca480105a0 con 0x7fca58100e90 2024-04-23T00:03:38.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.006+0000 7fca557fa700 1 --2- 172.21.15.67:0/2453271858 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fca4006ecc0 0x7fca40071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:38.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.006+0000 7fca557fa700 1 -- 172.21.15.67:0/2453271858 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fca48014070 con 0x7fca58100e90 2024-04-23T00:03:38.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.006+0000 7fca577fe700 1 --2- 172.21.15.67:0/2453271858 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fca4006ecc0 0x7fca40071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:38.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.007+0000 7fca577fe700 1 --2- 172.21.15.67:0/2453271858 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fca4006ecc0 0x7fca40071180 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fca4c005b40 tx=0x7fca4c005a90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:03:38.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.015+0000 7fca557fa700 1 -- 172.21.15.67:0/2453271858 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fca4805be40 con 0x7fca58100e90 2024-04-23T00:03:38.231 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:38 smithi134 ceph-mon[85630]: from='client.36000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:03:38.231 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:38 smithi134 ceph-mon[85630]: from='client.45375 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:03:38.231 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:38 smithi134 ceph-mon[85630]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-23T00:03:38.231 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:38 smithi134 ceph-mon[85630]: mds.? [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] up:active 2024-04-23T00:03:38.232 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:38 smithi134 ceph-mon[85630]: mds.? [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] up:active 2024-04-23T00:03:38.232 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:38 smithi134 ceph-mon[85630]: fsmap cephfs:2 {0=cephfs.smithi067.aqlxpv=up:active,1=cephfs.smithi067.xyazmh=up:active} 2 up:standby 2024-04-23T00:03:38.232 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:38 smithi134 ceph-mon[85630]: fsmap cephfs:2 {0=cephfs.smithi067.aqlxpv=up:active,1=cephfs.smithi067.xyazmh=up:active} 2 up:standby-replay 2024-04-23T00:03:38.232 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:38 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3493728713' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:38.232 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:38 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:38.232 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:38 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:38.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.280+0000 7fca5e2f1700 1 -- 172.21.15.67:0/2453271858 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fca58197d10 con 0x7fca58100e90 2024-04-23T00:03:38.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:38 smithi067 ceph-mon[112438]: from='client.36000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:03:38.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:38 smithi067 ceph-mon[112438]: from='client.45375 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:03:38.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:38 smithi067 ceph-mon[112438]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-23T00:03:38.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:38 smithi067 ceph-mon[112438]: mds.? [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] up:active 2024-04-23T00:03:38.270 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:38 smithi067 ceph-mon[112438]: mds.? [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] up:active 2024-04-23T00:03:38.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:38 smithi067 ceph-mon[112438]: fsmap cephfs:2 {0=cephfs.smithi067.aqlxpv=up:active,1=cephfs.smithi067.xyazmh=up:active} 2 up:standby 2024-04-23T00:03:38.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:38 smithi067 ceph-mon[112438]: fsmap cephfs:2 {0=cephfs.smithi067.aqlxpv=up:active,1=cephfs.smithi067.xyazmh=up:active} 2 up:standby-replay 2024-04-23T00:03:38.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:38 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3493728713' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:38.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:38 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:38.271 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:38 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:38.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.281+0000 7fca557fa700 1 -- 172.21.15.67:0/2453271858 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 41 v41) v1 ==== 76+0+1966 (secure 0 0 0) 0x7fca4805b9d0 con 0x7fca58100e90 2024-04-23T00:03:38.272 INFO:teuthology.orchestra.run.smithi067.stdout:e41 2024-04-23T00:03:38.272 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-23T00:03:38.272 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-23T00:03:38.273 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-23T00:03:38.273 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:03:38.273 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-23T00:03:38.273 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-23T00:03:38.273 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 41 2024-04-23T00:03:38.273 INFO:teuthology.orchestra.run.smithi067.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-23T00:03:38.273 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-23T00:03:38.274 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-23T00:03:37.116623+0000 2024-04-23T00:03:38.274 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-23T00:03:38.274 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-23T00:03:38.274 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-23T00:03:38.274 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-23T00:03:38.274 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-23T00:03:38.274 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-23T00:03:38.274 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-23T00:03:38.275 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-23T00:03:38.275 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-23T00:03:38.275 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-23T00:03:38.275 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-23T00:03:38.275 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-23T00:03:38.275 INFO:teuthology.orchestra.run.smithi067.stdout:up {0=34396,1=34424} 2024-04-23T00:03:38.275 INFO:teuthology.orchestra.run.smithi067.stdout:failed 2024-04-23T00:03:38.275 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-23T00:03:38.276 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-23T00:03:38.276 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-23T00:03:38.276 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-23T00:03:38.276 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-23T00:03:38.276 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-23T00:03:38.276 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-23T00:03:38.276 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-23T00:03:38.276 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 34396 members: 34424,34396 2024-04-23T00:03:38.276 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{0:34396} state up:active seq 469 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T00:03:38.277 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{0:44327} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T00:03:38.277 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{1:34424} state up:active seq 464 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T00:03:38.277 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{1:44851} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1849648622,v1:172.21.15.134:6827/1849648622] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T00:03:38.277 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:03:38.277 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:03:38.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.284+0000 7fca5e2f1700 1 -- 172.21.15.67:0/2453271858 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fca4006ecc0 msgr2=0x7fca40071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:38.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.284+0000 7fca5e2f1700 1 --2- 172.21.15.67:0/2453271858 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fca4006ecc0 0x7fca40071180 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fca4c005b40 tx=0x7fca4c005a90 comp rx=0 tx=0).stop 2024-04-23T00:03:38.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.284+0000 7fca5e2f1700 1 -- 172.21.15.67:0/2453271858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca58100e90 msgr2=0x7fca5819c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:38.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.284+0000 7fca5e2f1700 1 --2- 172.21.15.67:0/2453271858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca58100e90 0x7fca5819c670 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fca4800e920 tx=0x7fca4800ec30 comp rx=0 tx=0).stop 2024-04-23T00:03:38.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.284+0000 7fca5e2f1700 1 -- 172.21.15.67:0/2453271858 shutdown_connections 2024-04-23T00:03:38.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.284+0000 7fca5e2f1700 1 --2- 172.21.15.67:0/2453271858 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fca4006ecc0 0x7fca40071180 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:38.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.284+0000 7fca5e2f1700 1 --2- 172.21.15.67:0/2453271858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca58100e90 0x7fca5819c670 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:38.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.284+0000 7fca5e2f1700 1 --2- 172.21.15.67:0/2453271858 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fca58101840 0x7fca5819cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:38.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.284+0000 7fca5e2f1700 1 -- 172.21.15.67:0/2453271858 >> 172.21.15.67:0/2453271858 conn(0x7fca580fc740 msgr2=0x7fca580fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:03:38.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.285+0000 7fca5e2f1700 1 -- 172.21.15.67:0/2453271858 shutdown_connections 2024-04-23T00:03:38.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.285+0000 7fca5e2f1700 1 -- 172.21.15.67:0/2453271858 wait complete. 2024-04-23T00:03:38.279 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 41 2024-04-23T00:03:38.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.403+0000 7ff36cf4a700 1 -- 172.21.15.67:0/2903421955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff368102700 msgr2=0x7ff36810abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:38.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.403+0000 7ff36cf4a700 1 --2- 172.21.15.67:0/2903421955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff368102700 0x7ff36810abf0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7ff35c009910 tx=0x7ff35c009c20 comp rx=0 tx=0).stop 2024-04-23T00:03:38.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.403+0000 7ff36cf4a700 1 -- 172.21.15.67:0/2903421955 shutdown_connections 2024-04-23T00:03:38.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.403+0000 7ff36cf4a700 1 --2- 172.21.15.67:0/2903421955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff368102700 0x7ff36810abf0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:38.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.403+0000 7ff36cf4a700 1 --2- 172.21.15.67:0/2903421955 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff368101de0 0x7ff3681021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:38.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.403+0000 7ff36cf4a700 1 -- 172.21.15.67:0/2903421955 >> 172.21.15.67:0/2903421955 conn(0x7ff3680fb610 msgr2=0x7ff3680fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:03:38.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.403+0000 7ff36cf4a700 1 -- 172.21.15.67:0/2903421955 shutdown_connections 2024-04-23T00:03:38.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.403+0000 7ff36cf4a700 1 -- 172.21.15.67:0/2903421955 wait complete. 2024-04-23T00:03:38.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.404+0000 7ff36cf4a700 1 Processor -- start 2024-04-23T00:03:38.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.404+0000 7ff36cf4a700 1 -- start start 2024-04-23T00:03:38.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.404+0000 7ff36cf4a700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff368101de0 0x7ff36819e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:38.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.404+0000 7ff36cf4a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff368102700 0x7ff36819eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:38.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.404+0000 7ff36cf4a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff36819f430 con 0x7ff368102700 2024-04-23T00:03:38.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.404+0000 7ff36cf4a700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3681988e0 con 0x7ff368101de0 2024-04-23T00:03:38.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.404+0000 7ff365d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff368102700 0x7ff36819eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:38.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.404+0000 7ff36659c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff368101de0 0x7ff36819e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:38.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.404+0000 7ff36659c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff368101de0 0x7ff36819e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.67:53146/0 (socket says 172.21.15.67:53146) 2024-04-23T00:03:38.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.404+0000 7ff365d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff368102700 0x7ff36819eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55722/0 (socket says 172.21.15.67:55722) 2024-04-23T00:03:38.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.404+0000 7ff36659c700 1 -- 172.21.15.67:0/1177783497 learned_addr learned my addr 172.21.15.67:0/1177783497 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:03:38.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.405+0000 7ff36659c700 1 -- 172.21.15.67:0/1177783497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff368102700 msgr2=0x7ff36819eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:38.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.405+0000 7ff36659c700 1 --2- 172.21.15.67:0/1177783497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff368102700 0x7ff36819eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:38.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.405+0000 7ff36659c700 1 -- 172.21.15.67:0/1177783497 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff35c0095f0 con 0x7ff368101de0 2024-04-23T00:03:38.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.405+0000 7ff36659c700 1 --2- 172.21.15.67:0/1177783497 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff368101de0 0x7ff36819e860 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff35800e920 tx=0x7ff35800ec30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:03:38.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.405+0000 7ff3577fe700 1 -- 172.21.15.67:0/1177783497 <== mon.1 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff35800c9d0 con 0x7ff368101de0 2024-04-23T00:03:38.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.405+0000 7ff36cf4a700 1 -- 172.21.15.67:0/1177783497 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff368198bc0 con 0x7ff368101de0 2024-04-23T00:03:38.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.405+0000 7ff3577fe700 1 -- 172.21.15.67:0/1177783497 <== mon.1 v2:172.21.15.134:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff358010430 con 0x7ff368101de0 2024-04-23T00:03:38.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.405+0000 7ff3577fe700 1 -- 172.21.15.67:0/1177783497 <== mon.1 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff358018630 con 0x7ff368101de0 2024-04-23T00:03:38.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.405+0000 7ff36cf4a700 1 -- 172.21.15.67:0/1177783497 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3681990e0 con 0x7ff368101de0 2024-04-23T00:03:38.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.406+0000 7ff36cf4a700 1 -- 172.21.15.67:0/1177783497 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff36810a750 con 0x7ff368101de0 2024-04-23T00:03:38.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.407+0000 7ff3577fe700 1 -- 172.21.15.67:0/1177783497 <== mon.1 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff3580105a0 con 0x7ff368101de0 2024-04-23T00:03:38.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.407+0000 7ff3577fe700 1 --2- 172.21.15.67:0/1177783497 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff35006ecc0 0x7ff350071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:38.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.407+0000 7ff3577fe700 1 -- 172.21.15.67:0/1177783497 <== mon.1 v2:172.21.15.134:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7ff358014070 con 0x7ff368101de0 2024-04-23T00:03:38.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.407+0000 7ff365d9b700 1 --2- 172.21.15.67:0/1177783497 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff35006ecc0 0x7ff350071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:38.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.408+0000 7ff365d9b700 1 --2- 172.21.15.67:0/1177783497 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff35006ecc0 0x7ff350071180 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7ff35c0059d0 tx=0x7ff35c005940 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:03:38.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.416+0000 7ff3577fe700 1 -- 172.21.15.67:0/1177783497 <== mon.1 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff35805be40 con 0x7ff368101de0 2024-04-23T00:03:38.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.646+0000 7ff36cf4a700 1 -- 172.21.15.67:0/1177783497 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff368199350 con 0x7ff35006ecc0 2024-04-23T00:03:38.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.648+0000 7ff3577fe700 1 -- 172.21.15.67:0/1177783497 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+441 (secure 0 0 0) 0x7ff368199350 con 0x7ff35006ecc0 2024-04-23T00:03:38.637 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-23T00:03:38.637 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c", 2024-04-23T00:03:38.637 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": true, 2024-04-23T00:03:38.637 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-23T00:03:38.637 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [ 2024-04-23T00:03:38.637 INFO:teuthology.orchestra.run.smithi067.stdout: "mds", 2024-04-23T00:03:38.637 INFO:teuthology.orchestra.run.smithi067.stdout: "crash", 2024-04-23T00:03:38.637 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr", 2024-04-23T00:03:38.637 INFO:teuthology.orchestra.run.smithi067.stdout: "mon", 2024-04-23T00:03:38.638 INFO:teuthology.orchestra.run.smithi067.stdout: "osd" 2024-04-23T00:03:38.638 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-23T00:03:38.638 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": "16/23 daemons upgraded", 2024-04-23T00:03:38.638 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "Currently upgrading ceph-exporter daemons", 2024-04-23T00:03:38.638 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-23T00:03:38.638 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-23T00:03:38.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.650+0000 7ff36cf4a700 1 -- 172.21.15.67:0/1177783497 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff35006ecc0 msgr2=0x7ff350071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:38.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.650+0000 7ff36cf4a700 1 --2- 172.21.15.67:0/1177783497 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff35006ecc0 0x7ff350071180 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7ff35c0059d0 tx=0x7ff35c005940 comp rx=0 tx=0).stop 2024-04-23T00:03:38.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.650+0000 7ff36cf4a700 1 -- 172.21.15.67:0/1177783497 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff368101de0 msgr2=0x7ff36819e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:38.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.650+0000 7ff36cf4a700 1 --2- 172.21.15.67:0/1177783497 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff368101de0 0x7ff36819e860 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff35800e920 tx=0x7ff35800ec30 comp rx=0 tx=0).stop 2024-04-23T00:03:38.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.650+0000 7ff36cf4a700 1 -- 172.21.15.67:0/1177783497 shutdown_connections 2024-04-23T00:03:38.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.650+0000 7ff36cf4a700 1 --2- 172.21.15.67:0/1177783497 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff35006ecc0 0x7ff350071180 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:38.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.650+0000 7ff36cf4a700 1 --2- 172.21.15.67:0/1177783497 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff368101de0 0x7ff36819e860 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:38.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.650+0000 7ff36cf4a700 1 --2- 172.21.15.67:0/1177783497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff368102700 0x7ff36819eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:38.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.650+0000 7ff36cf4a700 1 -- 172.21.15.67:0/1177783497 >> 172.21.15.67:0/1177783497 conn(0x7ff3680fb610 msgr2=0x7ff3681001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:03:38.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.650+0000 7ff36cf4a700 1 -- 172.21.15.67:0/1177783497 shutdown_connections 2024-04-23T00:03:38.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.650+0000 7ff36cf4a700 1 -- 172.21.15.67:0/1177783497 wait complete. 2024-04-23T00:03:38.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.758+0000 7f70837f2700 1 -- 172.21.15.67:0/235317939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f707c105bd0 msgr2=0x7f707c105fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:38.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.758+0000 7f70837f2700 1 --2- 172.21.15.67:0/235317939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f707c105bd0 0x7f707c105fb0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f7078009910 tx=0x7f7078009c20 comp rx=0 tx=0).stop 2024-04-23T00:03:38.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.758+0000 7f70837f2700 1 -- 172.21.15.67:0/235317939 shutdown_connections 2024-04-23T00:03:38.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.758+0000 7f70837f2700 1 --2- 172.21.15.67:0/235317939 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f707c0ffc20 0x7f707c100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:38.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.758+0000 7f70837f2700 1 --2- 172.21.15.67:0/235317939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f707c105bd0 0x7f707c105fb0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:38.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.758+0000 7f70837f2700 1 -- 172.21.15.67:0/235317939 >> 172.21.15.67:0/235317939 conn(0x7f707c075670 msgr2=0x7f707c075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:03:38.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.759+0000 7f70837f2700 1 -- 172.21.15.67:0/235317939 shutdown_connections 2024-04-23T00:03:38.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.759+0000 7f70837f2700 1 -- 172.21.15.67:0/235317939 wait complete. 2024-04-23T00:03:38.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.759+0000 7f70837f2700 1 Processor -- start 2024-04-23T00:03:38.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.759+0000 7f70837f2700 1 -- start start 2024-04-23T00:03:38.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.759+0000 7f70837f2700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f707c0ffc20 0x7f707c19c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:38.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.759+0000 7f70837f2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f707c105bd0 0x7f707c19cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:38.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.759+0000 7f70837f2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f707c19d1f0 con 0x7f707c105bd0 2024-04-23T00:03:38.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.759+0000 7f70837f2700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f707c1966a0 con 0x7f707c0ffc20 2024-04-23T00:03:38.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.760+0000 7f7080d8d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f707c105bd0 0x7f707c19cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:38.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.760+0000 7f7080d8d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f707c105bd0 0x7f707c19cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54326/0 (socket says 172.21.15.67:54326) 2024-04-23T00:03:38.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.760+0000 7f7080d8d700 1 -- 172.21.15.67:0/3458993075 learned_addr learned my addr 172.21.15.67:0/3458993075 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:03:38.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.760+0000 7f708158e700 1 --2- 172.21.15.67:0/3458993075 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f707c0ffc20 0x7f707c19c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:38.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.760+0000 7f7080d8d700 1 -- 172.21.15.67:0/3458993075 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f707c0ffc20 msgr2=0x7f707c19c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:38.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.760+0000 7f7080d8d700 1 --2- 172.21.15.67:0/3458993075 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f707c0ffc20 0x7f707c19c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:38.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.760+0000 7f7080d8d700 1 -- 172.21.15.67:0/3458993075 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70780095f0 con 0x7f707c105bd0 2024-04-23T00:03:38.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.760+0000 7f7080d8d700 1 --2- 172.21.15.67:0/3458993075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f707c105bd0 0x7f707c19cb60 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f706c00d8d0 tx=0x7f706c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:03:38.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.760+0000 7f70727fc700 1 -- 172.21.15.67:0/3458993075 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f706c0098a0 con 0x7f707c105bd0 2024-04-23T00:03:38.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.760+0000 7f70837f2700 1 -- 172.21.15.67:0/3458993075 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f707c196980 con 0x7f707c105bd0 2024-04-23T00:03:38.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.760+0000 7f70727fc700 1 -- 172.21.15.67:0/3458993075 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f706c010460 con 0x7f707c105bd0 2024-04-23T00:03:38.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.760+0000 7f70837f2700 1 -- 172.21.15.67:0/3458993075 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f707c196ed0 con 0x7f707c105bd0 2024-04-23T00:03:38.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.760+0000 7f70727fc700 1 -- 172.21.15.67:0/3458993075 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f706c00f5f0 con 0x7f707c105bd0 2024-04-23T00:03:38.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.761+0000 7f70837f2700 1 -- 172.21.15.67:0/3458993075 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f707c04f2e0 con 0x7f707c105bd0 2024-04-23T00:03:38.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.762+0000 7f70727fc700 1 -- 172.21.15.67:0/3458993075 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f706c00f750 con 0x7f707c105bd0 2024-04-23T00:03:38.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.762+0000 7f70727fc700 1 --2- 172.21.15.67:0/3458993075 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f706806ecc0 0x7f7068071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:03:38.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.762+0000 7f70727fc700 1 -- 172.21.15.67:0/3458993075 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f706c08ed90 con 0x7f707c105bd0 2024-04-23T00:03:38.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.762+0000 7f708158e700 1 --2- 172.21.15.67:0/3458993075 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f706806ecc0 0x7f7068071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:03:38.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.763+0000 7f708158e700 1 --2- 172.21.15.67:0/3458993075 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f706806ecc0 0x7f7068071180 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f7078006010 tx=0x7f707801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:03:38.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:38.767+0000 7f70727fc700 1 -- 172.21.15.67:0/3458993075 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f706c05b580 con 0x7f707c105bd0 2024-04-23T00:03:39.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:39.069+0000 7f70837f2700 1 -- 172.21.15.67:0/3458993075 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f707c197140 con 0x7f707c105bd0 2024-04-23T00:03:39.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:39.069+0000 7f70727fc700 1 -- 172.21.15.67:0/3458993075 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+4283 (secure 0 0 0) 0x7f706c05b110 con 0x7f707c105bd0 2024-04-23T00:03:39.059 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_WARN failed to probe daemons or devices; 1 filesystem with deprecated feature inline_data 2024-04-23T00:03:39.059 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-23T00:03:39.059 INFO:teuthology.orchestra.run.smithi067.stdout: host smithi134 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi134/config 2024-04-23T00:03:39.059 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-23T00:03:39.059 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpqfwjed6d:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-23T00:03:39.059 INFO:teuthology.orchestra.run.smithi067.stdout:Traceback (most recent call last): 2024-04-23T00:03:39.059 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-23T00:03:39.059 INFO:teuthology.orchestra.run.smithi067.stdout: return _run_code(code, main_globals, None, 2024-04-23T00:03:39.059 INFO:teuthology.orchestra.run.smithi067.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-23T00:03:39.059 INFO:teuthology.orchestra.run.smithi067.stdout: exec(code, run_globals) 2024-04-23T00:03:39.059 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5642, in 2024-04-23T00:03:39.059 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 5630, in main 2024-04-23T00:03:39.060 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 408, in _infer_config 2024-04-23T00:03:39.060 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 323, in _infer_fsid 2024-04-23T00:03:39.060 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 436, in _infer_image 2024-04-23T00:03:39.060 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 310, in _validate_fsid 2024-04-23T00:03:39.060 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/__main__.py", line 3380, in command_ceph_volume 2024-04-23T00:03:39.060 INFO:teuthology.orchestra.run.smithi067.stdout: File "/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/cephadm.94efd715a2c841750499b7d3d025171327475e2d4fc96b8d6f7706d699c4cedb/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-23T00:03:39.060 INFO:teuthology.orchestra.run.smithi067.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c -e NODE_NAME=smithi134 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/selinux:/sys/fs/selinux:ro -v /:/rootfs:slave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpqfwjed6d:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c inventory --format=json-pretty --filter-for-batch 2024-04-23T00:03:39.060 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-23T00:03:39.060 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-23T00:03:39.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:39.072+0000 7f70837f2700 1 -- 172.21.15.67:0/3458993075 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f706806ecc0 msgr2=0x7f7068071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:39.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:39.072+0000 7f70837f2700 1 --2- 172.21.15.67:0/3458993075 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f706806ecc0 0x7f7068071180 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f7078006010 tx=0x7f707801a040 comp rx=0 tx=0).stop 2024-04-23T00:03:39.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:39.072+0000 7f70837f2700 1 -- 172.21.15.67:0/3458993075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f707c105bd0 msgr2=0x7f707c19cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:03:39.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:39.072+0000 7f70837f2700 1 --2- 172.21.15.67:0/3458993075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f707c105bd0 0x7f707c19cb60 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f706c00d8d0 tx=0x7f706c00dbe0 comp rx=0 tx=0).stop 2024-04-23T00:03:39.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:39.072+0000 7f70837f2700 1 -- 172.21.15.67:0/3458993075 shutdown_connections 2024-04-23T00:03:39.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:39.072+0000 7f70837f2700 1 --2- 172.21.15.67:0/3458993075 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f706806ecc0 0x7f7068071180 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:39.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:39.072+0000 7f70837f2700 1 --2- 172.21.15.67:0/3458993075 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f707c0ffc20 0x7f707c19c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:39.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:39.072+0000 7f70837f2700 1 --2- 172.21.15.67:0/3458993075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f707c105bd0 0x7f707c19cb60 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:03:39.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:39.072+0000 7f70837f2700 1 -- 172.21.15.67:0/3458993075 >> 172.21.15.67:0/3458993075 conn(0x7f707c075670 msgr2=0x7f707c0686c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:03:39.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:39.072+0000 7f70837f2700 1 -- 172.21.15.67:0/3458993075 shutdown_connections 2024-04-23T00:03:39.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:03:39.072+0000 7f70837f2700 1 -- 172.21.15.67:0/3458993075 wait complete. 2024-04-23T00:03:39.231 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:39 smithi134 ceph-mon[85630]: from='client.36008 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:03:39.231 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:39 smithi134 ceph-mon[85630]: pgmap v1121: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 27 MiB/s rd, 853 B/s wr, 11 op/s 2024-04-23T00:03:39.231 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:39 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2453271858' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-23T00:03:39.231 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:39 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3458993075' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T00:03:39.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:39 smithi067 ceph-mon[112438]: from='client.36008 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:03:39.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:39 smithi067 ceph-mon[112438]: pgmap v1121: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 27 MiB/s rd, 853 B/s wr, 11 op/s 2024-04-23T00:03:39.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:39 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2453271858' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-23T00:03:39.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:39 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3458993075' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T00:03:40.231 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:40 smithi134 ceph-mon[85630]: from='client.45389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:03:40.231 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:40 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:40.231 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:40 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi134", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-23T00:03:40.232 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:40 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T00:03:40.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:40 smithi067 ceph-mon[112438]: from='client.45389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:03:40.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:40 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:40.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:40 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi134", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-23T00:03:40.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:40 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T00:03:41.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:41 smithi134 ceph-mon[85630]: pgmap v1122: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 32 MiB/s rd, 1.7 KiB/s wr, 16 op/s 2024-04-23T00:03:41.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:41 smithi134 ceph-mon[85630]: Upgrade: Updating ceph-exporter.smithi134 (2/2) 2024-04-23T00:03:41.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:41 smithi134 ceph-mon[85630]: Deploying daemon ceph-exporter.smithi134 on smithi134 2024-04-23T00:03:41.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:41.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:03:41.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:41 smithi067 ceph-mon[112438]: pgmap v1122: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 32 MiB/s rd, 1.7 KiB/s wr, 16 op/s 2024-04-23T00:03:41.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:41 smithi067 ceph-mon[112438]: Upgrade: Updating ceph-exporter.smithi134 (2/2) 2024-04-23T00:03:41.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:41 smithi067 ceph-mon[112438]: Deploying daemon ceph-exporter.smithi134 on smithi134 2024-04-23T00:03:41.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:41.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:03:43.231 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:43 smithi134 ceph-mon[85630]: pgmap v1123: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 35 MiB/s rd, 1.7 KiB/s wr, 18 op/s 2024-04-23T00:03:43.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:43 smithi067 ceph-mon[112438]: pgmap v1123: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 35 MiB/s rd, 1.7 KiB/s wr, 18 op/s 2024-04-23T00:03:45.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:45 smithi134 ceph-mon[85630]: pgmap v1124: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 32 MiB/s rd, 8.4 KiB/s wr, 19 op/s 2024-04-23T00:03:45.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:45 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:45.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:45 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:45.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:45 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T00:03:45.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:45 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:45.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:45 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:45.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:45 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:45.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:45 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:45.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:45 smithi067 ceph-mon[112438]: pgmap v1124: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 32 MiB/s rd, 8.4 KiB/s wr, 19 op/s 2024-04-23T00:03:45.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:45 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:45.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:45 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:45.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:45 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T00:03:45.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:45 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:45.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:45 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:45.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:45 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:45.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:45 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:47.413 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:47 smithi067 ceph-mon[112438]: pgmap v1125: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 30 MiB/s rd, 14 KiB/s wr, 20 op/s 2024-04-23T00:03:47.414 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:47 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:47.414 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:47 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:47.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:47 smithi134 ceph-mon[85630]: pgmap v1125: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 30 MiB/s rd, 14 KiB/s wr, 20 op/s 2024-04-23T00:03:47.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:47 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:47.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:47 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:49.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:49 smithi067 ceph-mon[112438]: pgmap v1126: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 26 MiB/s rd, 14 KiB/s wr, 20 op/s 2024-04-23T00:03:49.282 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:49 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:49.282 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:49 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:49.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:49 smithi134 ceph-mon[85630]: pgmap v1126: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 26 MiB/s rd, 14 KiB/s wr, 20 op/s 2024-04-23T00:03:49.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:49 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:49.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:49 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:50.781 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:50 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:50.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:50 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:50.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:50 smithi067 ceph-mon[112438]: pgmap v1127: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 24 MiB/s rd, 13 KiB/s wr, 17 op/s 2024-04-23T00:03:50.981 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:50 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:50.981 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:50 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:50.981 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:50 smithi134 ceph-mon[85630]: pgmap v1127: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 24 MiB/s rd, 13 KiB/s wr, 17 op/s 2024-04-23T00:03:51.731 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:51 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:51.731 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:51 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:52.032 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:51 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:52.032 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:51 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:52.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:52 smithi067 ceph-mon[112438]: pgmap v1128: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 18 MiB/s rd, 12 KiB/s wr, 12 op/s 2024-04-23T00:03:52.981 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:52 smithi134 ceph-mon[85630]: pgmap v1128: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 18 MiB/s rd, 12 KiB/s wr, 12 op/s 2024-04-23T00:03:55.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:55 smithi134 ceph-mon[85630]: pgmap v1129: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 8.6 MiB/s rd, 12 KiB/s wr, 8 op/s 2024-04-23T00:03:55.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:55 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:55.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:55 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:55.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:55 smithi067 ceph-mon[112438]: pgmap v1129: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 8.6 MiB/s rd, 12 KiB/s wr, 8 op/s 2024-04-23T00:03:55.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:55 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:55.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:55 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:56.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:03:56.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:03:57.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:57 smithi134 ceph-mon[85630]: pgmap v1130: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 3.5 MiB/s rd, 5.6 KiB/s wr, 5 op/s 2024-04-23T00:03:57.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:57 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:57.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:57 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:57.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:57 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T00:03:57.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:57 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T00:03:57.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:57 smithi134 ceph-mon[85630]: pgmap v1131: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 14 KiB/s rd, 3 op/s 2024-04-23T00:03:57.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:57 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:57.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:57 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T00:03:57.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:57 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:57.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:57 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:57.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:57 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:57.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:57 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:57.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:57 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:57.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:57 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-04-23T00:03:57.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:57 smithi067 ceph-mon[112438]: pgmap v1130: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 3.5 MiB/s rd, 5.6 KiB/s wr, 5 op/s 2024-04-23T00:03:57.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:57 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:57.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:57 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:57.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:57 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T00:03:57.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:57 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T00:03:57.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:57 smithi067 ceph-mon[112438]: pgmap v1131: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 14 KiB/s rd, 3 op/s 2024-04-23T00:03:57.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:57 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:57.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:57 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T00:03:57.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:57 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:57.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:57 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:57.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:57 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:57.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:57 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:57.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:57 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:57.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:57 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-04-23T00:03:58.335 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: Upgrade: Setting filesystem cephfs Joinable 2024-04-23T00:03:58.336 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: Health check cleared: CEPHADM_REFRESH_FAILED (was: failed to probe daemons or devices) 2024-04-23T00:03:58.336 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-04-23T00:03:58.336 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: fsmap cephfs:2 {0=cephfs.smithi067.aqlxpv=up:active,1=cephfs.smithi067.xyazmh=up:active} 2 up:standby-replay 2024-04-23T00:03:58.336 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:58.336 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:58.336 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:58.336 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:58.336 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: Upgrade: Setting container_image for all ceph-exporter 2024-04-23T00:03:58.336 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:58.336 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi067"}]: dispatch 2024-04-23T00:03:58.336 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi067"}]': finished 2024-04-23T00:03:58.336 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi134"}]: dispatch 2024-04-23T00:03:58.337 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi134"}]': finished 2024-04-23T00:03:58.337 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:58.337 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: Upgrade: Setting container_image for all iscsi 2024-04-23T00:03:58.337 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:58.337 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:58.337 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: Upgrade: Setting container_image for all nfs 2024-04-23T00:03:58.337 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:58.337 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:58.337 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: Upgrade: Setting container_image for all nvmeof 2024-04-23T00:03:58.337 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:58.337 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:58.337 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:58.337 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:58.338 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:58.338 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:58.338 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:58.338 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: Upgrade: Finalizing container_image settings 2024-04-23T00:03:58.338 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-23T00:03:58.338 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-23T00:03:58.338 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-23T00:03:58.338 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-23T00:03:58.338 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-23T00:03:58.338 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-23T00:03:58.338 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-23T00:03:58.338 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-23T00:03:58.339 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-23T00:03:58.339 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-23T00:03:58.339 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-23T00:03:58.339 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-23T00:03:58.339 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-23T00:03:58.339 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-23T00:03:58.339 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-23T00:03:58.339 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-23T00:03:58.339 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-23T00:03:58.339 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-23T00:03:58.339 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-23T00:03:58.339 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-23T00:03:58.339 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-23T00:03:58.340 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-23T00:03:58.340 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-23T00:03:58.340 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-23T00:03:58.340 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-23T00:03:58.340 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-23T00:03:58.340 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-23T00:03:58.340 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-23T00:03:58.340 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-23T00:03:58.340 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-23T00:03:58.340 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-23T00:03:58.340 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T00:03:58.340 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T00:03:58.340 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T00:03:58.340 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:58.341 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T00:03:58.341 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T00:03:58.341 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T00:03:58.341 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:58.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: Upgrade: Setting filesystem cephfs Joinable 2024-04-23T00:03:58.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: Health check cleared: CEPHADM_REFRESH_FAILED (was: failed to probe daemons or devices) 2024-04-23T00:03:58.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-04-23T00:03:58.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: fsmap cephfs:2 {0=cephfs.smithi067.aqlxpv=up:active,1=cephfs.smithi067.xyazmh=up:active} 2 up:standby-replay 2024-04-23T00:03:58.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:58.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:58.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:58.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:58.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: Upgrade: Setting container_image for all ceph-exporter 2024-04-23T00:03:58.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:58.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi067"}]: dispatch 2024-04-23T00:03:58.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi067"}]': finished 2024-04-23T00:03:58.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi134"}]: dispatch 2024-04-23T00:03:58.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi134"}]': finished 2024-04-23T00:03:58.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:58.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: Upgrade: Setting container_image for all iscsi 2024-04-23T00:03:58.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:58.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:58.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: Upgrade: Setting container_image for all nfs 2024-04-23T00:03:58.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:58.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:58.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: Upgrade: Setting container_image for all nvmeof 2024-04-23T00:03:58.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:58.483 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:58.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:58.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:58.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:58.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:58.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:03:58.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: Upgrade: Finalizing container_image settings 2024-04-23T00:03:58.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-23T00:03:58.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-23T00:03:58.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-23T00:03:58.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-23T00:03:58.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-23T00:03:58.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-23T00:03:58.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-23T00:03:58.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-23T00:03:58.484 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-23T00:03:58.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-23T00:03:58.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-23T00:03:58.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-23T00:03:58.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-23T00:03:58.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-23T00:03:58.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-23T00:03:58.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-23T00:03:58.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-23T00:03:58.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-23T00:03:58.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-23T00:03:58.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-23T00:03:58.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-23T00:03:58.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-23T00:03:58.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-23T00:03:58.485 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-23T00:03:58.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-23T00:03:58.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-23T00:03:58.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-23T00:03:58.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-23T00:03:58.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-23T00:03:58.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-23T00:03:58.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-23T00:03:58.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T00:03:58.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T00:03:58.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T00:03:58.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:58.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T00:03:58.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T00:03:58.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T00:03:58.486 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:03:59.351 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:59 smithi134 ceph-mon[85630]: Upgrade: Complete! 2024-04-23T00:03:59.351 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:03:59 smithi134 ceph-mon[85630]: pgmap v1132: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 7.3 KiB/s rd, 4 op/s 2024-04-23T00:03:59.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:59 smithi067 ceph-mon[112438]: Upgrade: Complete! 2024-04-23T00:03:59.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:03:59 smithi067 ceph-mon[112438]: pgmap v1132: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 7.3 KiB/s rd, 4 op/s 2024-04-23T00:04:01.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:01 smithi134 ceph-mon[85630]: pgmap v1133: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-23T00:04:01.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:01 smithi067 ceph-mon[112438]: pgmap v1133: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-23T00:04:02.731 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:02 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:04:02.731 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:02 smithi134 ceph-mon[85630]: pgmap v1134: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-23T00:04:02.781 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:02 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:04:02.782 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:02 smithi067 ceph-mon[112438]: pgmap v1134: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-23T00:04:05.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:05 smithi134 ceph-mon[85630]: pgmap v1135: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-23T00:04:05.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:05 smithi067 ceph-mon[112438]: pgmap v1135: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-23T00:04:07.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:07 smithi134 ceph-mon[85630]: pgmap v1136: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-23T00:04:07.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:07 smithi067 ceph-mon[112438]: pgmap v1136: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-23T00:04:09.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.196+0000 7f0b2e0d7700 1 -- 172.21.15.67:0/759148424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b28100e90 msgr2=0x7f0b28101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:04:09.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.196+0000 7f0b2e0d7700 1 --2- 172.21.15.67:0/759148424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b28100e90 0x7f0b28101270 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f0b18009910 tx=0x7f0b18009c20 comp rx=0 tx=0).stop 2024-04-23T00:04:09.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.196+0000 7f0b2e0d7700 1 -- 172.21.15.67:0/759148424 shutdown_connections 2024-04-23T00:04:09.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.196+0000 7f0b2e0d7700 1 --2- 172.21.15.67:0/759148424 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0b28101840 0x7f0b28105890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:04:09.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.196+0000 7f0b2e0d7700 1 --2- 172.21.15.67:0/759148424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b28100e90 0x7f0b28101270 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:04:09.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.196+0000 7f0b2e0d7700 1 -- 172.21.15.67:0/759148424 >> 172.21.15.67:0/759148424 conn(0x7f0b280fc740 msgr2=0x7f0b280feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:04:09.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.196+0000 7f0b2e0d7700 1 -- 172.21.15.67:0/759148424 shutdown_connections 2024-04-23T00:04:09.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.196+0000 7f0b2e0d7700 1 -- 172.21.15.67:0/759148424 wait complete. 2024-04-23T00:04:09.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.197+0000 7f0b2e0d7700 1 Processor -- start 2024-04-23T00:04:09.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.197+0000 7f0b2e0d7700 1 -- start start 2024-04-23T00:04:09.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.198+0000 7f0b2e0d7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b28100e90 0x7f0b2819e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:04:09.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.198+0000 7f0b2e0d7700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0b28101840 0x7f0b2819edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:04:09.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.198+0000 7f0b2e0d7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2819f440 con 0x7f0b28100e90 2024-04-23T00:04:09.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.198+0000 7f0b2e0d7700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b281988f0 con 0x7f0b28101840 2024-04-23T00:04:09.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.198+0000 7f0b277fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b28100e90 0x7f0b2819e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:04:09.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.198+0000 7f0b277fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b28100e90 0x7f0b2819e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52328/0 (socket says 172.21.15.67:52328) 2024-04-23T00:04:09.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.198+0000 7f0b277fe700 1 -- 172.21.15.67:0/3768332096 learned_addr learned my addr 172.21.15.67:0/3768332096 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:04:09.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.198+0000 7f0b26ffd700 1 --2- 172.21.15.67:0/3768332096 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0b28101840 0x7f0b2819edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:04:09.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.198+0000 7f0b277fe700 1 -- 172.21.15.67:0/3768332096 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0b28101840 msgr2=0x7f0b2819edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:04:09.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.198+0000 7f0b277fe700 1 --2- 172.21.15.67:0/3768332096 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0b28101840 0x7f0b2819edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:04:09.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.198+0000 7f0b277fe700 1 -- 172.21.15.67:0/3768332096 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b180095f0 con 0x7f0b28100e90 2024-04-23T00:04:09.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.198+0000 7f0b277fe700 1 --2- 172.21.15.67:0/3768332096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b28100e90 0x7f0b2819e870 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f0b18006010 tx=0x7f0b1800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:04:09.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.198+0000 7f0b24ff9700 1 -- 172.21.15.67:0/3768332096 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b1801d070 con 0x7f0b28100e90 2024-04-23T00:04:09.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.198+0000 7f0b2e0d7700 1 -- 172.21.15.67:0/3768332096 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b28198b70 con 0x7f0b28100e90 2024-04-23T00:04:09.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.198+0000 7f0b24ff9700 1 -- 172.21.15.67:0/3768332096 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0b1800f460 con 0x7f0b28100e90 2024-04-23T00:04:09.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.198+0000 7f0b24ff9700 1 -- 172.21.15.67:0/3768332096 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b18017630 con 0x7f0b28100e90 2024-04-23T00:04:09.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.198+0000 7f0b2e0d7700 1 -- 172.21.15.67:0/3768332096 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b28199060 con 0x7f0b28100e90 2024-04-23T00:04:09.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.200+0000 7f0b2e0d7700 1 -- 172.21.15.67:0/3768332096 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b28105010 con 0x7f0b28100e90 2024-04-23T00:04:09.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.200+0000 7f0b24ff9700 1 -- 172.21.15.67:0/3768332096 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0b18003a60 con 0x7f0b28100e90 2024-04-23T00:04:09.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.200+0000 7f0b24ff9700 1 --2- 172.21.15.67:0/3768332096 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0b1006ecc0 0x7f0b10071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:04:09.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.201+0000 7f0b24ff9700 1 -- 172.21.15.67:0/3768332096 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f0b18090480 con 0x7f0b28100e90 2024-04-23T00:04:09.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.201+0000 7f0b26ffd700 1 --2- 172.21.15.67:0/3768332096 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0b1006ecc0 0x7f0b10071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:04:09.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.201+0000 7f0b26ffd700 1 --2- 172.21.15.67:0/3768332096 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0b1006ecc0 0x7f0b10071180 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c005950 tx=0x7f0b1c0058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:04:09.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.205+0000 7f0b24ff9700 1 -- 172.21.15.67:0/3768332096 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f0b1805cbf0 con 0x7f0b28100e90 2024-04-23T00:04:09.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.434+0000 7f0b2e0d7700 1 -- 172.21.15.67:0/3768332096 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b28199280 con 0x7f0b1006ecc0 2024-04-23T00:04:09.422 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:09 smithi067 ceph-mon[112438]: pgmap v1137: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:04:09.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.436+0000 7f0b24ff9700 1 -- 172.21.15.67:0/3768332096 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7f0b28199280 con 0x7f0b1006ecc0 2024-04-23T00:04:09.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.438+0000 7f0b2e0d7700 1 -- 172.21.15.67:0/3768332096 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0b1006ecc0 msgr2=0x7f0b10071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:04:09.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.438+0000 7f0b2e0d7700 1 --2- 172.21.15.67:0/3768332096 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0b1006ecc0 0x7f0b10071180 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c005950 tx=0x7f0b1c0058e0 comp rx=0 tx=0).stop 2024-04-23T00:04:09.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.438+0000 7f0b2e0d7700 1 -- 172.21.15.67:0/3768332096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b28100e90 msgr2=0x7f0b2819e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:04:09.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.438+0000 7f0b2e0d7700 1 --2- 172.21.15.67:0/3768332096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b28100e90 0x7f0b2819e870 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f0b18006010 tx=0x7f0b1800be20 comp rx=0 tx=0).stop 2024-04-23T00:04:09.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.438+0000 7f0b2e0d7700 1 -- 172.21.15.67:0/3768332096 shutdown_connections 2024-04-23T00:04:09.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.438+0000 7f0b2e0d7700 1 --2- 172.21.15.67:0/3768332096 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0b1006ecc0 0x7f0b10071180 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:04:09.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.438+0000 7f0b2e0d7700 1 --2- 172.21.15.67:0/3768332096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b28100e90 0x7f0b2819e870 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:04:09.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.438+0000 7f0b2e0d7700 1 --2- 172.21.15.67:0/3768332096 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0b28101840 0x7f0b2819edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:04:09.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.438+0000 7f0b2e0d7700 1 -- 172.21.15.67:0/3768332096 >> 172.21.15.67:0/3768332096 conn(0x7f0b280fc740 msgr2=0x7f0b280fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:04:09.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.438+0000 7f0b2e0d7700 1 -- 172.21.15.67:0/3768332096 shutdown_connections 2024-04-23T00:04:09.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:09.438+0000 7f0b2e0d7700 1 -- 172.21.15.67:0/3768332096 wait complete. 2024-04-23T00:04:09.465 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:09 smithi134 ceph-mon[85630]: pgmap v1137: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:04:10.004 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -e sha1=43be020184947e53516056c9931e1ac5bdbbb1a5 -- bash -c 'ceph orch ps' 2024-04-23T00:04:10.265 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-23T00:04:11.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:11 smithi067 ceph-mon[112438]: from='client.36032 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:04:11.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:11 smithi067 ceph-mon[112438]: pgmap v1138: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:04:11.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:04:11.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:04:11.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:11 smithi134 ceph-mon[85630]: from='client.36032 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:04:11.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:11 smithi134 ceph-mon[85630]: pgmap v1138: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:04:11.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:04:11.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:04:11.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:11.791+0000 7f78d8ffc700 1 -- 172.21.15.67:0/955998020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78d40ffc20 msgr2=0x7f78d4100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:04:11.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:11.791+0000 7f78d8ffc700 1 --2- 172.21.15.67:0/955998020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78d40ffc20 0x7f78d4100080 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f78c4009910 tx=0x7f78c4009c20 comp rx=0 tx=0).stop 2024-04-23T00:04:11.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:11.791+0000 7f78d8ffc700 1 -- 172.21.15.67:0/955998020 shutdown_connections 2024-04-23T00:04:11.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:11.791+0000 7f78d8ffc700 1 --2- 172.21.15.67:0/955998020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78d40ffc20 0x7f78d4100080 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:04:11.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:11.791+0000 7f78d8ffc700 1 --2- 172.21.15.67:0/955998020 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78d4105bd0 0x7f78d4105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:04:11.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:11.791+0000 7f78d8ffc700 1 -- 172.21.15.67:0/955998020 >> 172.21.15.67:0/955998020 conn(0x7f78d4075670 msgr2=0x7f78d4075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:04:11.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:11.791+0000 7f78d8ffc700 1 -- 172.21.15.67:0/955998020 shutdown_connections 2024-04-23T00:04:11.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:11.791+0000 7f78d8ffc700 1 -- 172.21.15.67:0/955998020 wait complete. 2024-04-23T00:04:11.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:11.792+0000 7f78d8ffc700 1 Processor -- start 2024-04-23T00:04:11.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:11.792+0000 7f78d8ffc700 1 -- start start 2024-04-23T00:04:11.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:11.792+0000 7f78d8ffc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78d40ffc20 0x7f78d419e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:04:11.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:11.792+0000 7f78d8ffc700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78d4105bd0 0x7f78d419ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:04:11.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:11.792+0000 7f78d8ffc700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d419f3f0 con 0x7f78d40ffc20 2024-04-23T00:04:11.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:11.792+0000 7f78d8ffc700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d41988a0 con 0x7f78d4105bd0 2024-04-23T00:04:11.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:11.792+0000 7f78d259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78d40ffc20 0x7f78d419e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:04:11.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:11.792+0000 7f78d259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78d40ffc20 0x7f78d419e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52344/0 (socket says 172.21.15.67:52344) 2024-04-23T00:04:11.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:11.792+0000 7f78d259c700 1 -- 172.21.15.67:0/2633811575 learned_addr learned my addr 172.21.15.67:0/2633811575 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:04:11.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:11.793+0000 7f78d1d9b700 1 --2- 172.21.15.67:0/2633811575 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78d4105bd0 0x7f78d419ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:04:11.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:11.793+0000 7f78d259c700 1 -- 172.21.15.67:0/2633811575 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78d4105bd0 msgr2=0x7f78d419ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:04:11.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:11.793+0000 7f78d259c700 1 --2- 172.21.15.67:0/2633811575 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78d4105bd0 0x7f78d419ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:04:11.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:11.793+0000 7f78d259c700 1 -- 172.21.15.67:0/2633811575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c40095f0 con 0x7f78d40ffc20 2024-04-23T00:04:11.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:11.793+0000 7f78d1d9b700 1 --2- 172.21.15.67:0/2633811575 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78d4105bd0 0x7f78d419ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T00:04:11.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:11.793+0000 7f78d259c700 1 --2- 172.21.15.67:0/2633811575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78d40ffc20 0x7f78d419e820 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f78bc00d8d0 tx=0x7f78bc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:04:11.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:11.793+0000 7f78cb7fe700 1 -- 172.21.15.67:0/2633811575 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78bc009940 con 0x7f78d40ffc20 2024-04-23T00:04:11.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:11.793+0000 7f78d8ffc700 1 -- 172.21.15.67:0/2633811575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78d4198b80 con 0x7f78d40ffc20 2024-04-23T00:04:11.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:11.793+0000 7f78cb7fe700 1 -- 172.21.15.67:0/2633811575 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f78bc010460 con 0x7f78d40ffc20 2024-04-23T00:04:11.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:11.793+0000 7f78cb7fe700 1 -- 172.21.15.67:0/2633811575 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78bc00f5f0 con 0x7f78d40ffc20 2024-04-23T00:04:11.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:11.793+0000 7f78d8ffc700 1 -- 172.21.15.67:0/2633811575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78d41990d0 con 0x7f78d40ffc20 2024-04-23T00:04:11.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:11.794+0000 7f78d8ffc700 1 -- 172.21.15.67:0/2633811575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78d404f2e0 con 0x7f78d40ffc20 2024-04-23T00:04:11.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:11.795+0000 7f78cb7fe700 1 -- 172.21.15.67:0/2633811575 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f78bc0105d0 con 0x7f78d40ffc20 2024-04-23T00:04:11.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:11.795+0000 7f78cb7fe700 1 --2- 172.21.15.67:0/2633811575 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f78c006ecc0 0x7f78c0071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:04:11.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:11.795+0000 7f78cb7fe700 1 -- 172.21.15.67:0/2633811575 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f78bc08eff0 con 0x7f78d40ffc20 2024-04-23T00:04:11.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:11.795+0000 7f78d1d9b700 1 --2- 172.21.15.67:0/2633811575 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f78c006ecc0 0x7f78c0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:04:11.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:11.796+0000 7f78d1d9b700 1 --2- 172.21.15.67:0/2633811575 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f78c006ecc0 0x7f78c0071180 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f78c400b5c0 tx=0x7f78c4005960 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:04:11.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:11.800+0000 7f78cb7fe700 1 -- 172.21.15.67:0/2633811575 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f78bc05b760 con 0x7f78d40ffc20 2024-04-23T00:04:12.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:12.020+0000 7f78d8ffc700 1 -- 172.21.15.67:0/2633811575 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f78d4199410 con 0x7f78c006ecc0 2024-04-23T00:04:12.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:12.032+0000 7f78cb7fe700 1 -- 172.21.15.67:0/2633811575 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f78d4199410 con 0x7f78c006ecc0 2024-04-23T00:04:12.020 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-23T00:04:12.020 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (37m) 22s ago 49m 24.4M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-23T00:04:12.020 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (35s) 22s ago 49m 7826k - 19.0.0-3252-g43be0201 a0434699c1d7 edb19108e126 2024-04-23T00:04:12.021 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (28s) 24s ago 48m 7109k - 19.0.0-3252-g43be0201 a0434699c1d7 bf2ea18ce32d 2024-04-23T00:04:12.021 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (35m) 22s ago 49m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-23T00:04:12.021 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (35m) 24s ago 48m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-23T00:04:12.021 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (37m) 22s ago 48m 79.9M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-23T00:04:12.021 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (31m) 22s ago 43m 61.3M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-23T00:04:12.021 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (31m) 22s ago 43m 66.5M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-23T00:04:12.021 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (30m) 24s ago 43m 212M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-23T00:04:12.021 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (15m) 24s ago 43m 206M - 19.0.0-3252-g43be0201 a0434699c1d7 46d3d7e86fee 2024-04-23T00:04:12.021 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (39m) 22s ago 50m 518M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-23T00:04:12.021 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (38m) 24s ago 48m 460M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-23T00:04:12.021 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (36m) 22s ago 50m 88.6M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-23T00:04:12.021 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (35m) 24s ago 48m 61.9M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-23T00:04:12.021 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (38m) 22s ago 49m 15.7M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-23T00:04:12.021 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (38m) 24s ago 48m 17.9M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-23T00:04:12.022 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (34m) 22s ago 47m 109M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-23T00:04:12.022 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (34m) 22s ago 46m 124M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-23T00:04:12.022 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (33m) 22s ago 46m 91.5M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-23T00:04:12.022 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (33m) 24s ago 46m 112M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-23T00:04:12.022 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (32m) 24s ago 45m 95.1M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-23T00:04:12.022 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (31m) 24s ago 45m 91.0M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-23T00:04:12.022 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (38m) 22s ago 48m 92.6M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-23T00:04:12.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:12.034+0000 7f78d8ffc700 1 -- 172.21.15.67:0/2633811575 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f78c006ecc0 msgr2=0x7f78c0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:04:12.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:12.034+0000 7f78d8ffc700 1 --2- 172.21.15.67:0/2633811575 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f78c006ecc0 0x7f78c0071180 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f78c400b5c0 tx=0x7f78c4005960 comp rx=0 tx=0).stop 2024-04-23T00:04:12.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:12.034+0000 7f78d8ffc700 1 -- 172.21.15.67:0/2633811575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78d40ffc20 msgr2=0x7f78d419e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:04:12.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:12.034+0000 7f78d8ffc700 1 --2- 172.21.15.67:0/2633811575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78d40ffc20 0x7f78d419e820 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f78bc00d8d0 tx=0x7f78bc00dc90 comp rx=0 tx=0).stop 2024-04-23T00:04:12.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:12.034+0000 7f78d8ffc700 1 -- 172.21.15.67:0/2633811575 shutdown_connections 2024-04-23T00:04:12.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:12.034+0000 7f78d8ffc700 1 --2- 172.21.15.67:0/2633811575 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f78c006ecc0 0x7f78c0071180 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:04:12.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:12.034+0000 7f78d8ffc700 1 --2- 172.21.15.67:0/2633811575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78d40ffc20 0x7f78d419e820 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:04:12.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:12.034+0000 7f78d8ffc700 1 --2- 172.21.15.67:0/2633811575 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78d4105bd0 0x7f78d419ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:04:12.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:12.034+0000 7f78d8ffc700 1 -- 172.21.15.67:0/2633811575 >> 172.21.15.67:0/2633811575 conn(0x7f78d4075670 msgr2=0x7f78d4068740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:04:12.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:12.034+0000 7f78d8ffc700 1 -- 172.21.15.67:0/2633811575 shutdown_connections 2024-04-23T00:04:12.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:12.034+0000 7f78d8ffc700 1 -- 172.21.15.67:0/2633811575 wait complete. 2024-04-23T00:04:12.685 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -e sha1=43be020184947e53516056c9931e1ac5bdbbb1a5 -- bash -c 'ceph orch upgrade status' 2024-04-23T00:04:12.914 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-23T00:04:13.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:13 smithi067 ceph-mon[112438]: from='client.36036 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:04:13.292 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:13 smithi067 ceph-mon[112438]: pgmap v1139: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:04:13.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:13 smithi134 ceph-mon[85630]: from='client.36036 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:04:13.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:13 smithi134 ceph-mon[85630]: pgmap v1139: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:04:14.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.516+0000 7f3bb1d9c700 1 -- 172.21.15.67:0/1394276449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bac1006b0 msgr2=0x7f3bac100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:04:14.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.516+0000 7f3bb1d9c700 1 --2- 172.21.15.67:0/1394276449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bac1006b0 0x7f3bac100a90 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c009910 tx=0x7f3b9c009c20 comp rx=0 tx=0).stop 2024-04-23T00:04:14.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.516+0000 7f3bb1d9c700 1 -- 172.21.15.67:0/1394276449 shutdown_connections 2024-04-23T00:04:14.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.516+0000 7f3bb1d9c700 1 --2- 172.21.15.67:0/1394276449 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3bac101060 0x7f3bac1047b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:04:14.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.516+0000 7f3bb1d9c700 1 --2- 172.21.15.67:0/1394276449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bac1006b0 0x7f3bac100a90 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:04:14.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.516+0000 7f3bb1d9c700 1 -- 172.21.15.67:0/1394276449 >> 172.21.15.67:0/1394276449 conn(0x7f3bac074e30 msgr2=0x7f3bac075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:04:14.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.516+0000 7f3bb1d9c700 1 -- 172.21.15.67:0/1394276449 shutdown_connections 2024-04-23T00:04:14.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.516+0000 7f3bb1d9c700 1 -- 172.21.15.67:0/1394276449 wait complete. 2024-04-23T00:04:14.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.517+0000 7f3bb1d9c700 1 Processor -- start 2024-04-23T00:04:14.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.517+0000 7f3bb1d9c700 1 -- start start 2024-04-23T00:04:14.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.517+0000 7f3bb1d9c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bac1006b0 0x7f3bac196960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:04:14.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.517+0000 7f3bb1d9c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3bac101060 0x7f3bac196ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:04:14.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.517+0000 7f3bb1d9c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bac197530 con 0x7f3bac1006b0 2024-04-23T00:04:14.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.517+0000 7f3bb1d9c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bac19b340 con 0x7f3bac101060 2024-04-23T00:04:14.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.518+0000 7f3bab7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bac1006b0 0x7f3bac196960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:04:14.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.518+0000 7f3bab7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bac1006b0 0x7f3bac196960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52362/0 (socket says 172.21.15.67:52362) 2024-04-23T00:04:14.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.518+0000 7f3bab7fe700 1 -- 172.21.15.67:0/1999459081 learned_addr learned my addr 172.21.15.67:0/1999459081 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:04:14.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.518+0000 7f3baaffd700 1 --2- 172.21.15.67:0/1999459081 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3bac101060 0x7f3bac196ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:04:14.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.518+0000 7f3bab7fe700 1 -- 172.21.15.67:0/1999459081 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3bac101060 msgr2=0x7f3bac196ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:04:14.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.518+0000 7f3bab7fe700 1 --2- 172.21.15.67:0/1999459081 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3bac101060 0x7f3bac196ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:04:14.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.518+0000 7f3bab7fe700 1 -- 172.21.15.67:0/1999459081 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b9c0095f0 con 0x7f3bac1006b0 2024-04-23T00:04:14.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.518+0000 7f3baaffd700 1 --2- 172.21.15.67:0/1999459081 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3bac101060 0x7f3bac196ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T00:04:14.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.518+0000 7f3bab7fe700 1 --2- 172.21.15.67:0/1999459081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bac1006b0 0x7f3bac196960 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c006010 tx=0x7f3b9c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:04:14.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.518+0000 7f3ba8ff9700 1 -- 172.21.15.67:0/1999459081 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b9c01d070 con 0x7f3bac1006b0 2024-04-23T00:04:14.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.518+0000 7f3ba8ff9700 1 -- 172.21.15.67:0/1999459081 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3b9c004fb0 con 0x7f3bac1006b0 2024-04-23T00:04:14.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.518+0000 7f3ba8ff9700 1 -- 172.21.15.67:0/1999459081 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b9c022ea0 con 0x7f3bac1006b0 2024-04-23T00:04:14.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.518+0000 7f3bb1d9c700 1 -- 172.21.15.67:0/1999459081 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bac19b5c0 con 0x7f3bac1006b0 2024-04-23T00:04:14.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.518+0000 7f3bb1d9c700 1 -- 172.21.15.67:0/1999459081 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bac19bab0 con 0x7f3bac1006b0 2024-04-23T00:04:14.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.520+0000 7f3bb1d9c700 1 -- 172.21.15.67:0/1999459081 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bac109130 con 0x7f3bac1006b0 2024-04-23T00:04:14.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.520+0000 7f3ba8ff9700 1 -- 172.21.15.67:0/1999459081 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3b9c004af0 con 0x7f3bac1006b0 2024-04-23T00:04:14.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.520+0000 7f3ba8ff9700 1 --2- 172.21.15.67:0/1999459081 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3b9406ecc0 0x7f3b94071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:04:14.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.521+0000 7f3ba8ff9700 1 -- 172.21.15.67:0/1999459081 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f3b9c0904c0 con 0x7f3bac1006b0 2024-04-23T00:04:14.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.521+0000 7f3baaffd700 1 --2- 172.21.15.67:0/1999459081 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3b9406ecc0 0x7f3b94071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:04:14.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.521+0000 7f3baaffd700 1 --2- 172.21.15.67:0/1999459081 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3b9406ecc0 0x7f3b94071180 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f3bac197f80 tx=0x7f3ba0008040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:04:14.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.525+0000 7f3ba8ff9700 1 -- 172.21.15.67:0/1999459081 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f3b9c05cd60 con 0x7f3bac1006b0 2024-04-23T00:04:14.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.752+0000 7f3bb1d9c700 1 -- 172.21.15.67:0/1999459081 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3bac197dd0 con 0x7f3b9406ecc0 2024-04-23T00:04:14.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.753+0000 7f3ba8ff9700 1 -- 172.21.15.67:0/1999459081 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7f3bac197dd0 con 0x7f3b9406ecc0 2024-04-23T00:04:14.741 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-23T00:04:14.741 INFO:teuthology.orchestra.run.smithi067.stdout: "target_image": null, 2024-04-23T00:04:14.741 INFO:teuthology.orchestra.run.smithi067.stdout: "in_progress": false, 2024-04-23T00:04:14.741 INFO:teuthology.orchestra.run.smithi067.stdout: "which": "", 2024-04-23T00:04:14.741 INFO:teuthology.orchestra.run.smithi067.stdout: "services_complete": [], 2024-04-23T00:04:14.741 INFO:teuthology.orchestra.run.smithi067.stdout: "progress": null, 2024-04-23T00:04:14.741 INFO:teuthology.orchestra.run.smithi067.stdout: "message": "", 2024-04-23T00:04:14.741 INFO:teuthology.orchestra.run.smithi067.stdout: "is_paused": false 2024-04-23T00:04:14.741 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-23T00:04:14.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.755+0000 7f3bb1d9c700 1 -- 172.21.15.67:0/1999459081 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3b9406ecc0 msgr2=0x7f3b94071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:04:14.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.755+0000 7f3bb1d9c700 1 --2- 172.21.15.67:0/1999459081 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3b9406ecc0 0x7f3b94071180 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f3bac197f80 tx=0x7f3ba0008040 comp rx=0 tx=0).stop 2024-04-23T00:04:14.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.755+0000 7f3bb1d9c700 1 -- 172.21.15.67:0/1999459081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bac1006b0 msgr2=0x7f3bac196960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:04:14.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.755+0000 7f3bb1d9c700 1 --2- 172.21.15.67:0/1999459081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bac1006b0 0x7f3bac196960 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c006010 tx=0x7f3b9c00be20 comp rx=0 tx=0).stop 2024-04-23T00:04:14.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.755+0000 7f3bb1d9c700 1 -- 172.21.15.67:0/1999459081 shutdown_connections 2024-04-23T00:04:14.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.755+0000 7f3bb1d9c700 1 --2- 172.21.15.67:0/1999459081 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3b9406ecc0 0x7f3b94071180 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:04:14.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.755+0000 7f3bb1d9c700 1 --2- 172.21.15.67:0/1999459081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bac1006b0 0x7f3bac196960 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:04:14.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.755+0000 7f3bb1d9c700 1 --2- 172.21.15.67:0/1999459081 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3bac101060 0x7f3bac196ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:04:14.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.755+0000 7f3bb1d9c700 1 -- 172.21.15.67:0/1999459081 >> 172.21.15.67:0/1999459081 conn(0x7f3bac074e30 msgr2=0x7f3bac103c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:04:14.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.756+0000 7f3bb1d9c700 1 -- 172.21.15.67:0/1999459081 shutdown_connections 2024-04-23T00:04:14.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:14.756+0000 7f3bb1d9c700 1 -- 172.21.15.67:0/1999459081 wait complete. 2024-04-23T00:04:15.379 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:15 smithi067 ceph-mon[112438]: pgmap v1140: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:04:15.402 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -e sha1=43be020184947e53516056c9931e1ac5bdbbb1a5 -- bash -c 'ceph health detail' 2024-04-23T00:04:15.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:15 smithi134 ceph-mon[85630]: pgmap v1140: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:04:15.620 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-23T00:04:16.425 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:16 smithi067 ceph-mon[112438]: from='client.36040 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:04:16.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:16 smithi134 ceph-mon[85630]: from='client.36040 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:04:17.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.275+0000 7fb433970700 1 -- 172.21.15.67:0/1852529303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb42c0ffc20 msgr2=0x7fb42c100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:04:17.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.275+0000 7fb433970700 1 --2- 172.21.15.67:0/1852529303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb42c0ffc20 0x7fb42c100080 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fb41c009910 tx=0x7fb41c009c20 comp rx=0 tx=0).stop 2024-04-23T00:04:17.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.275+0000 7fb433970700 1 -- 172.21.15.67:0/1852529303 shutdown_connections 2024-04-23T00:04:17.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.275+0000 7fb433970700 1 --2- 172.21.15.67:0/1852529303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb42c0ffc20 0x7fb42c100080 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:04:17.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.275+0000 7fb433970700 1 --2- 172.21.15.67:0/1852529303 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb42c105bd0 0x7fb42c105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:04:17.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.275+0000 7fb433970700 1 -- 172.21.15.67:0/1852529303 >> 172.21.15.67:0/1852529303 conn(0x7fb42c075670 msgr2=0x7fb42c075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:04:17.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.275+0000 7fb433970700 1 -- 172.21.15.67:0/1852529303 shutdown_connections 2024-04-23T00:04:17.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.275+0000 7fb433970700 1 -- 172.21.15.67:0/1852529303 wait complete. 2024-04-23T00:04:17.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.276+0000 7fb433970700 1 Processor -- start 2024-04-23T00:04:17.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.276+0000 7fb433970700 1 -- start start 2024-04-23T00:04:17.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.276+0000 7fb433970700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb42c0ffc20 0x7fb42c19e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:04:17.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.276+0000 7fb433970700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb42c105bd0 0x7fb42c19ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:04:17.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.276+0000 7fb433970700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c19f400 con 0x7fb42c105bd0 2024-04-23T00:04:17.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.276+0000 7fb433970700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c1988b0 con 0x7fb42c0ffc20 2024-04-23T00:04:17.265 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:17 smithi067 ceph-mon[112438]: pgmap v1141: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:04:17.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.276+0000 7fb430f0b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb42c105bd0 0x7fb42c19ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:04:17.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.276+0000 7fb430f0b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb42c105bd0 0x7fb42c19ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52374/0 (socket says 172.21.15.67:52374) 2024-04-23T00:04:17.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.276+0000 7fb430f0b700 1 -- 172.21.15.67:0/2819650579 learned_addr learned my addr 172.21.15.67:0/2819650579 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:04:17.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.276+0000 7fb43170c700 1 --2- 172.21.15.67:0/2819650579 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb42c0ffc20 0x7fb42c19e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:04:17.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.277+0000 7fb430f0b700 1 -- 172.21.15.67:0/2819650579 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb42c0ffc20 msgr2=0x7fb42c19e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:04:17.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.277+0000 7fb430f0b700 1 --2- 172.21.15.67:0/2819650579 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb42c0ffc20 0x7fb42c19e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:04:17.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.277+0000 7fb430f0b700 1 -- 172.21.15.67:0/2819650579 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb41c0095f0 con 0x7fb42c105bd0 2024-04-23T00:04:17.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.277+0000 7fb430f0b700 1 --2- 172.21.15.67:0/2819650579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb42c105bd0 0x7fb42c19ed70 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fb41c006010 tx=0x7fb41c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:04:17.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.277+0000 7fb4227fc700 1 -- 172.21.15.67:0/2819650579 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb41c01d070 con 0x7fb42c105bd0 2024-04-23T00:04:17.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.277+0000 7fb433970700 1 -- 172.21.15.67:0/2819650579 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb42c198b30 con 0x7fb42c105bd0 2024-04-23T00:04:17.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.277+0000 7fb4227fc700 1 -- 172.21.15.67:0/2819650579 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb41c00f460 con 0x7fb42c105bd0 2024-04-23T00:04:17.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.277+0000 7fb4227fc700 1 -- 172.21.15.67:0/2819650579 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb41c017630 con 0x7fb42c105bd0 2024-04-23T00:04:17.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.277+0000 7fb433970700 1 -- 172.21.15.67:0/2819650579 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb42c199020 con 0x7fb42c105bd0 2024-04-23T00:04:17.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.278+0000 7fb433970700 1 -- 172.21.15.67:0/2819650579 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb42c04f2e0 con 0x7fb42c105bd0 2024-04-23T00:04:17.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.279+0000 7fb4227fc700 1 -- 172.21.15.67:0/2819650579 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb41c00f5d0 con 0x7fb42c105bd0 2024-04-23T00:04:17.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.279+0000 7fb4227fc700 1 --2- 172.21.15.67:0/2819650579 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb41806ecc0 0x7fb418071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:04:17.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.279+0000 7fb4227fc700 1 -- 172.21.15.67:0/2819650579 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fb41c090480 con 0x7fb42c105bd0 2024-04-23T00:04:17.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.279+0000 7fb43170c700 1 --2- 172.21.15.67:0/2819650579 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb41806ecc0 0x7fb418071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:04:17.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.280+0000 7fb43170c700 1 --2- 172.21.15.67:0/2819650579 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb41806ecc0 0x7fb418071180 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fb428005950 tx=0x7fb4280058e0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:04:17.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.284+0000 7fb4227fc700 1 -- 172.21.15.67:0/2819650579 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb41c05cbf0 con 0x7fb42c105bd0 2024-04-23T00:04:17.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:17 smithi134 ceph-mon[85630]: pgmap v1141: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:04:17.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.581+0000 7fb433970700 1 -- 172.21.15.67:0/2819650579 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fb42c199350 con 0x7fb42c105bd0 2024-04-23T00:04:17.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.581+0000 7fb4227fc700 1 -- 172.21.15.67:0/2819650579 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7fb41c05c780 con 0x7fb42c105bd0 2024-04-23T00:04:17.570 INFO:teuthology.orchestra.run.smithi067.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-23T00:04:17.570 INFO:teuthology.orchestra.run.smithi067.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-23T00:04:17.570 INFO:teuthology.orchestra.run.smithi067.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-04-23T00:04:17.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.583+0000 7fb433970700 1 -- 172.21.15.67:0/2819650579 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb41806ecc0 msgr2=0x7fb418071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:04:17.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.583+0000 7fb433970700 1 --2- 172.21.15.67:0/2819650579 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb41806ecc0 0x7fb418071180 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fb428005950 tx=0x7fb4280058e0 comp rx=0 tx=0).stop 2024-04-23T00:04:17.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.583+0000 7fb433970700 1 -- 172.21.15.67:0/2819650579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb42c105bd0 msgr2=0x7fb42c19ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:04:17.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.583+0000 7fb433970700 1 --2- 172.21.15.67:0/2819650579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb42c105bd0 0x7fb42c19ed70 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fb41c006010 tx=0x7fb41c00be20 comp rx=0 tx=0).stop 2024-04-23T00:04:17.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.583+0000 7fb433970700 1 -- 172.21.15.67:0/2819650579 shutdown_connections 2024-04-23T00:04:17.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.583+0000 7fb433970700 1 --2- 172.21.15.67:0/2819650579 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb41806ecc0 0x7fb418071180 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:04:17.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.583+0000 7fb433970700 1 --2- 172.21.15.67:0/2819650579 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb42c0ffc20 0x7fb42c19e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:04:17.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.583+0000 7fb433970700 1 --2- 172.21.15.67:0/2819650579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb42c105bd0 0x7fb42c19ed70 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:04:17.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.583+0000 7fb433970700 1 -- 172.21.15.67:0/2819650579 >> 172.21.15.67:0/2819650579 conn(0x7fb42c075670 msgr2=0x7fb42c068720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:04:17.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.584+0000 7fb433970700 1 -- 172.21.15.67:0/2819650579 shutdown_connections 2024-04-23T00:04:17.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:17.584+0000 7fb433970700 1 -- 172.21.15.67:0/2819650579 wait complete. 2024-04-23T00:04:18.181 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -e sha1=43be020184947e53516056c9931e1ac5bdbbb1a5 -- bash -c 'ceph versions' 2024-04-23T00:04:18.395 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-23T00:04:18.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:18 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2819650579' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T00:04:18.518 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:18 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2819650579' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T00:04:19.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:19 smithi134 ceph-mon[85630]: pgmap v1142: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:04:19.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:19 smithi067 ceph-mon[112438]: pgmap v1142: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:04:20.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.098+0000 7ff35c97c700 1 -- 172.21.15.67:0/2437912998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff358100ea0 msgr2=0x7ff358101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:04:20.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.098+0000 7ff35c97c700 1 --2- 172.21.15.67:0/2437912998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff358100ea0 0x7ff358101280 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7ff348009910 tx=0x7ff348009c20 comp rx=0 tx=0).stop 2024-04-23T00:04:20.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.098+0000 7ff35c97c700 1 -- 172.21.15.67:0/2437912998 shutdown_connections 2024-04-23T00:04:20.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.098+0000 7ff35c97c700 1 --2- 172.21.15.67:0/2437912998 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff358101850 0x7ff3581058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:04:20.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.098+0000 7ff35c97c700 1 --2- 172.21.15.67:0/2437912998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff358100ea0 0x7ff358101280 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:04:20.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.098+0000 7ff35c97c700 1 -- 172.21.15.67:0/2437912998 >> 172.21.15.67:0/2437912998 conn(0x7ff3580fc750 msgr2=0x7ff3580feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:04:20.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.098+0000 7ff35c97c700 1 -- 172.21.15.67:0/2437912998 shutdown_connections 2024-04-23T00:04:20.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.098+0000 7ff35c97c700 1 -- 172.21.15.67:0/2437912998 wait complete. 2024-04-23T00:04:20.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.099+0000 7ff35c97c700 1 Processor -- start 2024-04-23T00:04:20.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.099+0000 7ff35c97c700 1 -- start start 2024-04-23T00:04:20.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.100+0000 7ff35c97c700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff358100ea0 0x7ff358073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:04:20.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.100+0000 7ff35c97c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff358101850 0x7ff3580735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:04:20.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.100+0000 7ff35c97c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff358073b10 con 0x7ff358101850 2024-04-23T00:04:20.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.100+0000 7ff35c97c700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff358073c50 con 0x7ff358100ea0 2024-04-23T00:04:20.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.100+0000 7ff34ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff358101850 0x7ff3580735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:04:20.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.100+0000 7ff34ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff358101850 0x7ff3580735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35126/0 (socket says 172.21.15.67:35126) 2024-04-23T00:04:20.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.100+0000 7ff34ffff700 1 -- 172.21.15.67:0/6806586 learned_addr learned my addr 172.21.15.67:0/6806586 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:04:20.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.100+0000 7ff35659c700 1 --2- 172.21.15.67:0/6806586 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff358100ea0 0x7ff358073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:04:20.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.100+0000 7ff34ffff700 1 -- 172.21.15.67:0/6806586 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff358100ea0 msgr2=0x7ff358073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:04:20.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.100+0000 7ff34ffff700 1 --2- 172.21.15.67:0/6806586 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff358100ea0 0x7ff358073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:04:20.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.100+0000 7ff34ffff700 1 -- 172.21.15.67:0/6806586 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3480095f0 con 0x7ff358101850 2024-04-23T00:04:20.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.100+0000 7ff34ffff700 1 --2- 172.21.15.67:0/6806586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff358101850 0x7ff3580735d0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7ff34000c960 tx=0x7ff34000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:04:20.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.100+0000 7ff34f7fe700 1 -- 172.21.15.67:0/6806586 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff340007a70 con 0x7ff358101850 2024-04-23T00:04:20.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.101+0000 7ff34f7fe700 1 -- 172.21.15.67:0/6806586 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff340007bd0 con 0x7ff358101850 2024-04-23T00:04:20.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.101+0000 7ff35c97c700 1 -- 172.21.15.67:0/6806586 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3581a2af0 con 0x7ff358101850 2024-04-23T00:04:20.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.101+0000 7ff34f7fe700 1 -- 172.21.15.67:0/6806586 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff340018630 con 0x7ff358101850 2024-04-23T00:04:20.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.101+0000 7ff35c97c700 1 -- 172.21.15.67:0/6806586 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3581a2e50 con 0x7ff358101850 2024-04-23T00:04:20.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.101+0000 7ff35c97c700 1 -- 172.21.15.67:0/6806586 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff358109240 con 0x7ff358101850 2024-04-23T00:04:20.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.102+0000 7ff34f7fe700 1 -- 172.21.15.67:0/6806586 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff34000f450 con 0x7ff358101850 2024-04-23T00:04:20.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.103+0000 7ff34f7fe700 1 --2- 172.21.15.67:0/6806586 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff34406ecc0 0x7ff344071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:04:20.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.103+0000 7ff34f7fe700 1 -- 172.21.15.67:0/6806586 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7ff34008eef0 con 0x7ff358101850 2024-04-23T00:04:20.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.103+0000 7ff35659c700 1 --2- 172.21.15.67:0/6806586 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff34406ecc0 0x7ff344071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:04:20.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.103+0000 7ff35659c700 1 --2- 172.21.15.67:0/6806586 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff34406ecc0 0x7ff344071180 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff348006010 tx=0x7ff34801a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:04:20.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.108+0000 7ff34f7fe700 1 -- 172.21.15.67:0/6806586 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7ff340057b30 con 0x7ff358101850 2024-04-23T00:04:20.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.408+0000 7ff35c97c700 1 -- 172.21.15.67:0/6806586 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff35804f2e0 con 0x7ff358101850 2024-04-23T00:04:20.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.408+0000 7ff34f7fe700 1 -- 172.21.15.67:0/6806586 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7ff34005b1f0 con 0x7ff358101850 2024-04-23T00:04:20.397 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-23T00:04:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-23T00:04:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-23T00:04:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-23T00:04:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-23T00:04:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-23T00:04:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-23T00:04:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-23T00:04:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-23T00:04:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-23T00:04:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-23T00:04:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 4 2024-04-23T00:04:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-23T00:04:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-23T00:04:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 14 2024-04-23T00:04:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-23T00:04:20.400 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-23T00:04:20.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.411+0000 7ff35c97c700 1 -- 172.21.15.67:0/6806586 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff34406ecc0 msgr2=0x7ff344071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:04:20.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.411+0000 7ff35c97c700 1 --2- 172.21.15.67:0/6806586 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff34406ecc0 0x7ff344071180 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff348006010 tx=0x7ff34801a040 comp rx=0 tx=0).stop 2024-04-23T00:04:20.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.411+0000 7ff35c97c700 1 -- 172.21.15.67:0/6806586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff358101850 msgr2=0x7ff3580735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:04:20.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.411+0000 7ff35c97c700 1 --2- 172.21.15.67:0/6806586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff358101850 0x7ff3580735d0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7ff34000c960 tx=0x7ff34000cc70 comp rx=0 tx=0).stop 2024-04-23T00:04:20.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.411+0000 7ff35c97c700 1 -- 172.21.15.67:0/6806586 shutdown_connections 2024-04-23T00:04:20.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.411+0000 7ff35c97c700 1 --2- 172.21.15.67:0/6806586 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7ff34406ecc0 0x7ff344071180 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:04:20.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.411+0000 7ff35c97c700 1 --2- 172.21.15.67:0/6806586 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff358100ea0 0x7ff358073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:04:20.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.411+0000 7ff35c97c700 1 --2- 172.21.15.67:0/6806586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff358101850 0x7ff3580735d0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:04:20.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.411+0000 7ff35c97c700 1 -- 172.21.15.67:0/6806586 >> 172.21.15.67:0/6806586 conn(0x7ff3580fc750 msgr2=0x7ff3580fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:04:20.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.411+0000 7ff35c97c700 1 -- 172.21.15.67:0/6806586 shutdown_connections 2024-04-23T00:04:20.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:04:20.411+0000 7ff35c97c700 1 -- 172.21.15.67:0/6806586 wait complete. 2024-04-23T00:04:21.015 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -e sha1=43be020184947e53516056c9931e1ac5bdbbb1a5 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-04-23T00:04:21.231 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-23T00:04:21.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:21 smithi067 ceph-mon[112438]: pgmap v1143: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:04:21.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:21 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/6806586' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:04:21.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:21 smithi134 ceph-mon[85630]: pgmap v1143: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:04:21.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:21 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/6806586' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:04:22.766 INFO:teuthology.orchestra.run.smithi067.stdout:wait for servicemap items w/ changing names to refresh 2024-04-23T00:04:23.402 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:23 smithi067 ceph-mon[112438]: pgmap v1144: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:04:23.403 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -e sha1=43be020184947e53516056c9931e1ac5bdbbb1a5 -- bash -c 'sleep 60' 2024-04-23T00:04:23.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:23 smithi134 ceph-mon[85630]: pgmap v1144: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:04:23.620 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-23T00:04:25.473 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:25 smithi134 ceph-mon[85630]: pgmap v1145: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:04:25.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:25 smithi067 ceph-mon[112438]: pgmap v1145: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:04:26.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:04:26.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:04:27.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:27 smithi134 ceph-mon[85630]: pgmap v1146: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:04:27.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:27 smithi067 ceph-mon[112438]: pgmap v1146: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:04:28.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:28 smithi134 ceph-mon[85630]: pgmap v1147: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:04:28.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:28 smithi067 ceph-mon[112438]: pgmap v1147: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:04:31.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:31 smithi134 ceph-mon[85630]: pgmap v1148: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:04:31.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:31 smithi067 ceph-mon[112438]: pgmap v1148: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:04:33.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:33 smithi067 ceph-mon[112438]: pgmap v1149: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:04:33.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:33 smithi134 ceph-mon[85630]: pgmap v1149: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:04:35.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:35 smithi134 ceph-mon[85630]: pgmap v1150: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:04:35.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:35 smithi067 ceph-mon[112438]: pgmap v1150: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:04:37.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:37 smithi134 ceph-mon[85630]: pgmap v1151: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:04:37.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:37 smithi067 ceph-mon[112438]: pgmap v1151: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:04:39.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:39 smithi134 ceph-mon[85630]: pgmap v1152: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:04:39.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:39 smithi067 ceph-mon[112438]: pgmap v1152: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:04:41.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:41 smithi134 ceph-mon[85630]: pgmap v1153: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:04:41.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:04:41.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:41 smithi067 ceph-mon[112438]: pgmap v1153: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:04:41.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:04:43.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:43 smithi134 ceph-mon[85630]: pgmap v1154: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:04:43.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:43 smithi067 ceph-mon[112438]: pgmap v1154: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:04:45.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:45 smithi134 ceph-mon[85630]: pgmap v1155: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:04:45.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:45 smithi067 ceph-mon[112438]: pgmap v1155: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:04:47.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:47 smithi134 ceph-mon[85630]: pgmap v1156: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:04:47.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:47 smithi067 ceph-mon[112438]: pgmap v1156: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:04:49.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:49 smithi134 ceph-mon[85630]: pgmap v1157: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:04:49.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:49 smithi067 ceph-mon[112438]: pgmap v1157: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:04:51.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:51 smithi134 ceph-mon[85630]: pgmap v1158: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:04:51.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:51 smithi067 ceph-mon[112438]: pgmap v1158: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:04:53.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:53 smithi134 ceph-mon[85630]: pgmap v1159: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:04:53.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:53 smithi067 ceph-mon[112438]: pgmap v1159: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:04:55.231 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:55 smithi134 ceph-mon[85630]: pgmap v1160: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:04:55.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:55 smithi067 ceph-mon[112438]: pgmap v1160: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:04:56.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:04:56.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:04:57.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:57 smithi134 ceph-mon[85630]: pgmap v1161: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:04:57.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:57 smithi067 ceph-mon[112438]: pgmap v1161: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:04:58.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T00:04:58.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:58 smithi134 ceph-mon[85630]: pgmap v1162: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:04:58.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T00:04:58.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T00:04:58.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:04:58 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:04:58.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T00:04:58.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:58 smithi067 ceph-mon[112438]: pgmap v1162: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:04:58.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T00:04:58.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T00:04:58.532 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:04:58 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:05:01.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:01 smithi134 ceph-mon[85630]: pgmap v1163: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:05:01.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:01 smithi067 ceph-mon[112438]: pgmap v1163: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:05:03.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:03 smithi134 ceph-mon[85630]: pgmap v1164: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:05:03.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:03 smithi067 ceph-mon[112438]: pgmap v1164: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:05:05.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:05 smithi134 ceph-mon[85630]: pgmap v1165: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:05:05.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:05 smithi067 ceph-mon[112438]: pgmap v1165: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:05:07.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:07 smithi134 ceph-mon[85630]: pgmap v1166: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:05:07.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:07 smithi067 ceph-mon[112438]: pgmap v1166: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:05:09.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:09 smithi134 ceph-mon[85630]: pgmap v1167: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:05:09.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:09 smithi067 ceph-mon[112438]: pgmap v1167: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:05:11.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:11 smithi134 ceph-mon[85630]: pgmap v1168: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:05:11.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:05:11.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:11 smithi067 ceph-mon[112438]: pgmap v1168: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:05:11.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:05:13.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:13 smithi134 ceph-mon[85630]: pgmap v1169: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:05:13.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:13 smithi067 ceph-mon[112438]: pgmap v1169: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:05:15.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:15 smithi134 ceph-mon[85630]: pgmap v1170: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:05:15.533 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:15 smithi067 ceph-mon[112438]: pgmap v1170: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:05:17.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:17 smithi134 ceph-mon[85630]: pgmap v1171: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:05:17.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:17 smithi067 ceph-mon[112438]: pgmap v1171: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:05:19.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:19 smithi134 ceph-mon[85630]: pgmap v1172: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:05:19.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:19 smithi067 ceph-mon[112438]: pgmap v1172: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:05:21.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:21 smithi134 ceph-mon[85630]: pgmap v1173: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:05:21.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:21 smithi067 ceph-mon[112438]: pgmap v1173: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:05:23.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:23 smithi134 ceph-mon[85630]: pgmap v1174: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:05:23.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:23 smithi067 ceph-mon[112438]: pgmap v1174: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:05:25.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:25 smithi134 ceph-mon[85630]: pgmap v1175: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:05:25.526 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:25 smithi067 ceph-mon[112438]: pgmap v1175: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:05:25.837 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -e sha1=43be020184947e53516056c9931e1ac5bdbbb1a5 -- bash -c 'ceph orch ps' 2024-04-23T00:05:26.061 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-23T00:05:26.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:05:26.511 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:05:27.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:27 smithi134 ceph-mon[85630]: pgmap v1176: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:05:27.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:27 smithi067 ceph-mon[112438]: pgmap v1176: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:05:27.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:27.818+0000 7f14ac68b700 1 -- 172.21.15.67:0/1485410401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14a4068580 msgr2=0x7f14a40689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:27.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:27.818+0000 7f14ac68b700 1 --2- 172.21.15.67:0/1485410401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14a4068580 0x7f14a40689e0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f1498009910 tx=0x7f1498009c20 comp rx=0 tx=0).stop 2024-04-23T00:05:27.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:27.818+0000 7f14ac68b700 1 -- 172.21.15.67:0/1485410401 shutdown_connections 2024-04-23T00:05:27.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:27.818+0000 7f14ac68b700 1 --2- 172.21.15.67:0/1485410401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14a4068580 0x7f14a40689e0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:27.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:27.818+0000 7f14ac68b700 1 --2- 172.21.15.67:0/1485410401 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f14a4106cd0 0x7f14a41070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:27.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:27.818+0000 7f14ac68b700 1 -- 172.21.15.67:0/1485410401 >> 172.21.15.67:0/1485410401 conn(0x7f14a4075670 msgr2=0x7f14a4075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:27.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:27.819+0000 7f14ac68b700 1 -- 172.21.15.67:0/1485410401 shutdown_connections 2024-04-23T00:05:27.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:27.819+0000 7f14ac68b700 1 -- 172.21.15.67:0/1485410401 wait complete. 2024-04-23T00:05:27.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:27.820+0000 7f14ac68b700 1 Processor -- start 2024-04-23T00:05:27.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:27.820+0000 7f14ac68b700 1 -- start start 2024-04-23T00:05:27.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:27.820+0000 7f14ac68b700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f14a4068580 0x7f14a40ffcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:27.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:27.820+0000 7f14ac68b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14a4106cd0 0x7f14a4100230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:27.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:27.820+0000 7f14ac68b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a4100770 con 0x7f14a4106cd0 2024-04-23T00:05:27.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:27.820+0000 7f14ac68b700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a41008b0 con 0x7f14a4068580 2024-04-23T00:05:27.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:27.820+0000 7f14a9c26700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14a4106cd0 0x7f14a4100230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:27.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:27.820+0000 7f14a9c26700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14a4106cd0 0x7f14a4100230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43540/0 (socket says 172.21.15.67:43540) 2024-04-23T00:05:27.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:27.820+0000 7f14a9c26700 1 -- 172.21.15.67:0/2278445783 learned_addr learned my addr 172.21.15.67:0/2278445783 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:27.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:27.820+0000 7f14aa427700 1 --2- 172.21.15.67:0/2278445783 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f14a4068580 0x7f14a40ffcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:27.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:27.821+0000 7f14a9c26700 1 -- 172.21.15.67:0/2278445783 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f14a4068580 msgr2=0x7f14a40ffcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:27.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:27.821+0000 7f14a9c26700 1 --2- 172.21.15.67:0/2278445783 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f14a4068580 0x7f14a40ffcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:27.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:27.821+0000 7f14a9c26700 1 -- 172.21.15.67:0/2278445783 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14980095f0 con 0x7f14a4106cd0 2024-04-23T00:05:27.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:27.821+0000 7f14a9c26700 1 --2- 172.21.15.67:0/2278445783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14a4106cd0 0x7f14a4100230 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f1498006010 tx=0x7f149800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:27.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:27.821+0000 7f14a37fe700 1 -- 172.21.15.67:0/2278445783 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f149801d070 con 0x7f14a4106cd0 2024-04-23T00:05:27.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:27.821+0000 7f14a37fe700 1 -- 172.21.15.67:0/2278445783 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1498005dc0 con 0x7f14a4106cd0 2024-04-23T00:05:27.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:27.821+0000 7f14ac68b700 1 -- 172.21.15.67:0/2278445783 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14a4100b30 con 0x7f14a4106cd0 2024-04-23T00:05:27.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:27.821+0000 7f14ac68b700 1 -- 172.21.15.67:0/2278445783 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14a41a2d60 con 0x7f14a4106cd0 2024-04-23T00:05:27.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:27.821+0000 7f14a37fe700 1 -- 172.21.15.67:0/2278445783 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1498017400 con 0x7f14a4106cd0 2024-04-23T00:05:27.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:27.822+0000 7f14ac68b700 1 -- 172.21.15.67:0/2278445783 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1490005320 con 0x7f14a4106cd0 2024-04-23T00:05:27.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:27.823+0000 7f14a37fe700 1 -- 172.21.15.67:0/2278445783 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f149800f460 con 0x7f14a4106cd0 2024-04-23T00:05:27.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:27.823+0000 7f14a37fe700 1 --2- 172.21.15.67:0/2278445783 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f148c06ecc0 0x7f148c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:27.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:27.824+0000 7f14a37fe700 1 -- 172.21.15.67:0/2278445783 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f14980903b0 con 0x7f14a4106cd0 2024-04-23T00:05:27.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:27.824+0000 7f14aa427700 1 --2- 172.21.15.67:0/2278445783 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f148c06ecc0 0x7f148c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:27.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:27.824+0000 7f14aa427700 1 --2- 172.21.15.67:0/2278445783 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f148c06ecc0 0x7f148c071180 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f1494009550 tx=0x7f14940091f0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:27.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:27.828+0000 7f14a37fe700 1 -- 172.21.15.67:0/2278445783 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1498058ff0 con 0x7f14a4106cd0 2024-04-23T00:05:28.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:28.047+0000 7f14ac68b700 1 -- 172.21.15.67:0/2278445783 --> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1490000bf0 con 0x7f148c06ecc0 2024-04-23T00:05:28.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:28.058+0000 7f14a37fe700 1 -- 172.21.15.67:0/2278445783 <== mgr.34102 v2:172.21.15.67:6800/3392236702 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f1490000bf0 con 0x7f148c06ecc0 2024-04-23T00:05:28.045 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-23T00:05:28.045 INFO:teuthology.orchestra.run.smithi067.stdout:alertmanager.smithi067 smithi067 *:9093,9094 running (39m) 98s ago 50m 24.4M - 0.25.0 c8568f914cd2 c08846ca2469 2024-04-23T00:05:28.045 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi067 smithi067 running (111s) 98s ago 50m 7826k - 19.0.0-3252-g43be0201 a0434699c1d7 edb19108e126 2024-04-23T00:05:28.045 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-exporter.smithi134 smithi134 running (104s) 100s ago 49m 7109k - 19.0.0-3252-g43be0201 a0434699c1d7 bf2ea18ce32d 2024-04-23T00:05:28.045 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi067 smithi067 running (36m) 98s ago 50m 6660k - 19.0.0-3252-g43be0201 a0434699c1d7 a8554b491990 2024-04-23T00:05:28.046 INFO:teuthology.orchestra.run.smithi067.stdout:crash.smithi134 smithi134 running (36m) 100s ago 49m 6665k - 19.0.0-3252-g43be0201 a0434699c1d7 664d5e6f62e6 2024-04-23T00:05:28.046 INFO:teuthology.orchestra.run.smithi067.stdout:grafana.smithi067 smithi067 *:3000 running (38m) 98s ago 50m 79.9M - 9.4.12 2bacad6d85d8 237622a83686 2024-04-23T00:05:28.046 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.aqlxpv smithi067 running (32m) 98s ago 45m 61.3M - 19.0.0-3252-g43be0201 a0434699c1d7 74bfe53aa5d3 2024-04-23T00:05:28.046 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi067.xyazmh smithi067 running (32m) 98s ago 44m 66.5M - 19.0.0-3252-g43be0201 a0434699c1d7 b091af80537e 2024-04-23T00:05:28.046 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.owipad smithi134 running (32m) 100s ago 44m 212M - 19.0.0-3252-g43be0201 a0434699c1d7 58423cf29368 2024-04-23T00:05:28.046 INFO:teuthology.orchestra.run.smithi067.stdout:mds.cephfs.smithi134.vuxbsj smithi134 running (17m) 100s ago 44m 206M - 19.0.0-3252-g43be0201 a0434699c1d7 46d3d7e86fee 2024-04-23T00:05:28.046 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi067.whdcsr smithi067 *:8443,9283,8765 running (40m) 98s ago 51m 518M - 19.0.0-3252-g43be0201 a0434699c1d7 3b67529d3316 2024-04-23T00:05:28.046 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.smithi134.ccxkcj smithi134 *:8443,9283,8765 running (40m) 100s ago 49m 460M - 19.0.0-3252-g43be0201 a0434699c1d7 b5c3693c73f6 2024-04-23T00:05:28.046 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi067 smithi067 running (37m) 98s ago 52m 88.6M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 b879412ee3f1 2024-04-23T00:05:28.046 INFO:teuthology.orchestra.run.smithi067.stdout:mon.smithi134 smithi134 running (37m) 100s ago 49m 61.9M 2048M 19.0.0-3252-g43be0201 a0434699c1d7 20a456e8e562 2024-04-23T00:05:28.046 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi067 smithi067 *:9100 running (39m) 98s ago 50m 15.7M - 1.5.0 0da6a335fe13 4e42ac3c1169 2024-04-23T00:05:28.046 INFO:teuthology.orchestra.run.smithi067.stdout:node-exporter.smithi134 smithi134 *:9100 running (39m) 100s ago 49m 17.9M - 1.5.0 0da6a335fe13 67c5c7506e1f 2024-04-23T00:05:28.046 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (36m) 98s ago 48m 109M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 a74fde1ef34e 2024-04-23T00:05:28.046 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi067 running (35m) 98s ago 48m 124M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 b003cac17eac 2024-04-23T00:05:28.047 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi067 running (34m) 98s ago 47m 91.5M 1216M 19.0.0-3252-g43be0201 a0434699c1d7 76df38d881d7 2024-04-23T00:05:28.047 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi134 running (34m) 100s ago 47m 112M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 5f42b665260c 2024-04-23T00:05:28.047 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi134 running (33m) 100s ago 46m 95.1M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 61da66bd1882 2024-04-23T00:05:28.047 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi134 running (33m) 100s ago 46m 91.0M 2240M 19.0.0-3252-g43be0201 a0434699c1d7 93a955888c84 2024-04-23T00:05:28.047 INFO:teuthology.orchestra.run.smithi067.stdout:prometheus.smithi067 smithi067 *:9095 running (39m) 98s ago 49m 92.6M - 2.43.0 a07b618ecd1d 7feaffc16dcd 2024-04-23T00:05:28.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:28.060+0000 7f14ac68b700 1 -- 172.21.15.67:0/2278445783 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f148c06ecc0 msgr2=0x7f148c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:28.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:28.060+0000 7f14ac68b700 1 --2- 172.21.15.67:0/2278445783 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f148c06ecc0 0x7f148c071180 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f1494009550 tx=0x7f14940091f0 comp rx=0 tx=0).stop 2024-04-23T00:05:28.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:28.060+0000 7f14ac68b700 1 -- 172.21.15.67:0/2278445783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14a4106cd0 msgr2=0x7f14a4100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:28.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:28.060+0000 7f14ac68b700 1 --2- 172.21.15.67:0/2278445783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14a4106cd0 0x7f14a4100230 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f1498006010 tx=0x7f149800bd70 comp rx=0 tx=0).stop 2024-04-23T00:05:28.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:28.060+0000 7f14ac68b700 1 -- 172.21.15.67:0/2278445783 shutdown_connections 2024-04-23T00:05:28.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:28.060+0000 7f14ac68b700 1 --2- 172.21.15.67:0/2278445783 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f148c06ecc0 0x7f148c071180 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:28.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:28.060+0000 7f14ac68b700 1 --2- 172.21.15.67:0/2278445783 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f14a4068580 0x7f14a40ffcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:28.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:28.060+0000 7f14ac68b700 1 --2- 172.21.15.67:0/2278445783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14a4106cd0 0x7f14a4100230 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:28.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:28.060+0000 7f14ac68b700 1 -- 172.21.15.67:0/2278445783 >> 172.21.15.67:0/2278445783 conn(0x7f14a4075670 msgr2=0x7f14a40feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:28.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:28.060+0000 7f14ac68b700 1 -- 172.21.15.67:0/2278445783 shutdown_connections 2024-04-23T00:05:28.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:28.060+0000 7f14ac68b700 1 -- 172.21.15.67:0/2278445783 wait complete. 2024-04-23T00:05:28.390 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:28 smithi067 ceph-mon[112438]: from='client.36052 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:05:28.390 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:28 smithi067 ceph-mon[112438]: pgmap v1177: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:05:28.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:28 smithi134 ceph-mon[85630]: from='client.36052 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T00:05:28.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:28 smithi134 ceph-mon[85630]: pgmap v1177: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:05:28.656 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -e sha1=43be020184947e53516056c9931e1ac5bdbbb1a5 -- bash -c 'ceph versions' 2024-04-23T00:05:28.869 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-23T00:05:30.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.482+0000 7f221362f700 1 -- 172.21.15.67:0/1227744610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f220c068580 msgr2=0x7f220c0689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:30.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.482+0000 7f221362f700 1 --2- 172.21.15.67:0/1227744610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f220c068580 0x7f220c0689e0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f21fc009910 tx=0x7f21fc009c20 comp rx=0 tx=0).stop 2024-04-23T00:05:30.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.482+0000 7f221362f700 1 -- 172.21.15.67:0/1227744610 shutdown_connections 2024-04-23T00:05:30.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.482+0000 7f221362f700 1 --2- 172.21.15.67:0/1227744610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f220c068580 0x7f220c0689e0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:30.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.482+0000 7f221362f700 1 --2- 172.21.15.67:0/1227744610 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f220c106cd0 0x7f220c1070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:30.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.482+0000 7f221362f700 1 -- 172.21.15.67:0/1227744610 >> 172.21.15.67:0/1227744610 conn(0x7f220c075670 msgr2=0x7f220c075a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:30.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.482+0000 7f221362f700 1 -- 172.21.15.67:0/1227744610 shutdown_connections 2024-04-23T00:05:30.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.482+0000 7f221362f700 1 -- 172.21.15.67:0/1227744610 wait complete. 2024-04-23T00:05:30.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.483+0000 7f221362f700 1 Processor -- start 2024-04-23T00:05:30.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.483+0000 7f221362f700 1 -- start start 2024-04-23T00:05:30.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.483+0000 7f221362f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f220c068580 0x7f220c19e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:30.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.483+0000 7f221362f700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f220c106cd0 0x7f220c19ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:30.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.483+0000 7f221362f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220c19f390 con 0x7f220c068580 2024-04-23T00:05:30.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.483+0000 7f221362f700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220c1988d0 con 0x7f220c106cd0 2024-04-23T00:05:30.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.483+0000 7f22113cb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f220c068580 0x7f220c19e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:30.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.484+0000 7f22113cb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f220c068580 0x7f220c19e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42326/0 (socket says 172.21.15.67:42326) 2024-04-23T00:05:30.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.484+0000 7f22113cb700 1 -- 172.21.15.67:0/645548339 learned_addr learned my addr 172.21.15.67:0/645548339 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:30.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.484+0000 7f2210bca700 1 --2- 172.21.15.67:0/645548339 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f220c106cd0 0x7f220c19ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:30.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.484+0000 7f22113cb700 1 -- 172.21.15.67:0/645548339 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f220c106cd0 msgr2=0x7f220c19ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:30.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.484+0000 7f22113cb700 1 --2- 172.21.15.67:0/645548339 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f220c106cd0 0x7f220c19ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:30.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.484+0000 7f22113cb700 1 -- 172.21.15.67:0/645548339 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21fc0095f0 con 0x7f220c068580 2024-04-23T00:05:30.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.484+0000 7f22113cb700 1 --2- 172.21.15.67:0/645548339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f220c068580 0x7f220c19e850 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f220800c930 tx=0x7f220800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:30.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.484+0000 7f22027fc700 1 -- 172.21.15.67:0/645548339 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2208007ad0 con 0x7f220c068580 2024-04-23T00:05:30.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.484+0000 7f221362f700 1 -- 172.21.15.67:0/645548339 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f220c198bb0 con 0x7f220c068580 2024-04-23T00:05:30.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.484+0000 7f22027fc700 1 -- 172.21.15.67:0/645548339 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f220800ce80 con 0x7f220c068580 2024-04-23T00:05:30.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.484+0000 7f22027fc700 1 -- 172.21.15.67:0/645548339 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2208018630 con 0x7f220c068580 2024-04-23T00:05:30.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.484+0000 7f221362f700 1 -- 172.21.15.67:0/645548339 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f220c199100 con 0x7f220c068580 2024-04-23T00:05:30.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.486+0000 7f221362f700 1 -- 172.21.15.67:0/645548339 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f220c04f2e0 con 0x7f220c068580 2024-04-23T00:05:30.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.486+0000 7f22027fc700 1 -- 172.21.15.67:0/645548339 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2208007c30 con 0x7f220c068580 2024-04-23T00:05:30.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.486+0000 7f22027fc700 1 --2- 172.21.15.67:0/645548339 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f21f806ecc0 0x7f21f8071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:30.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.487+0000 7f2210bca700 1 --2- 172.21.15.67:0/645548339 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f21f806ecc0 0x7f21f8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:30.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.487+0000 7f22027fc700 1 -- 172.21.15.67:0/645548339 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f220808f010 con 0x7f220c068580 2024-04-23T00:05:30.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.487+0000 7f2210bca700 1 --2- 172.21.15.67:0/645548339 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f21f806ecc0 0x7f21f8071180 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f220c0fe200 tx=0x7f21fc01a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:30.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.491+0000 7f22027fc700 1 -- 172.21.15.67:0/645548339 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f220805b780 con 0x7f220c068580 2024-04-23T00:05:30.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.793+0000 7f221362f700 1 -- 172.21.15.67:0/645548339 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f220c199380 con 0x7f220c068580 2024-04-23T00:05:30.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.793+0000 7f22027fc700 1 -- 172.21.15.67:0/645548339 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f220805b310 con 0x7f220c068580 2024-04-23T00:05:30.780 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-23T00:05:30.780 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": { 2024-04-23T00:05:30.780 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-23T00:05:30.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-23T00:05:30.781 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": { 2024-04-23T00:05:30.781 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 2 2024-04-23T00:05:30.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-23T00:05:30.781 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": { 2024-04-23T00:05:30.781 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 6 2024-04-23T00:05:30.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-23T00:05:30.781 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-23T00:05:30.781 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 4 2024-04-23T00:05:30.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-23T00:05:30.781 INFO:teuthology.orchestra.run.smithi067.stdout: "overall": { 2024-04-23T00:05:30.781 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)": 14 2024-04-23T00:05:30.781 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-23T00:05:30.781 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-23T00:05:30.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.796+0000 7f221362f700 1 -- 172.21.15.67:0/645548339 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f21f806ecc0 msgr2=0x7f21f8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:30.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.796+0000 7f221362f700 1 --2- 172.21.15.67:0/645548339 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f21f806ecc0 0x7f21f8071180 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f220c0fe200 tx=0x7f21fc01a040 comp rx=0 tx=0).stop 2024-04-23T00:05:30.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.796+0000 7f221362f700 1 -- 172.21.15.67:0/645548339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f220c068580 msgr2=0x7f220c19e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:30.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.796+0000 7f221362f700 1 --2- 172.21.15.67:0/645548339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f220c068580 0x7f220c19e850 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f220800c930 tx=0x7f220800cc40 comp rx=0 tx=0).stop 2024-04-23T00:05:30.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.796+0000 7f221362f700 1 -- 172.21.15.67:0/645548339 shutdown_connections 2024-04-23T00:05:30.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.796+0000 7f221362f700 1 --2- 172.21.15.67:0/645548339 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f21f806ecc0 0x7f21f8071180 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:30.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.796+0000 7f221362f700 1 --2- 172.21.15.67:0/645548339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f220c068580 0x7f220c19e850 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:30.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.796+0000 7f221362f700 1 --2- 172.21.15.67:0/645548339 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f220c106cd0 0x7f220c19ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:30.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.796+0000 7f221362f700 1 -- 172.21.15.67:0/645548339 >> 172.21.15.67:0/645548339 conn(0x7f220c075670 msgr2=0x7f220c0fe9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:30.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.796+0000 7f221362f700 1 -- 172.21.15.67:0/645548339 shutdown_connections 2024-04-23T00:05:30.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:30.796+0000 7f221362f700 1 -- 172.21.15.67:0/645548339 wait complete. 2024-04-23T00:05:31.437 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:31 smithi067 ceph-mon[112438]: pgmap v1178: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:05:31.437 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:31 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/645548339' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:05:31.460 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -e sha1=43be020184947e53516056c9931e1ac5bdbbb1a5 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-04-23T00:05:31.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:31 smithi134 ceph-mon[85630]: pgmap v1178: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:05:31.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:31 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/645548339' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:05:31.676 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-23T00:05:33.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.366+0000 7f30356b7700 1 -- 172.21.15.67:0/3712220292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3030100ea0 msgr2=0x7f3030101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:33.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.366+0000 7f30356b7700 1 --2- 172.21.15.67:0/3712220292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3030100ea0 0x7f3030101280 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f3020009910 tx=0x7f3020009c20 comp rx=0 tx=0).stop 2024-04-23T00:05:33.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.366+0000 7f30356b7700 1 -- 172.21.15.67:0/3712220292 shutdown_connections 2024-04-23T00:05:33.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.366+0000 7f30356b7700 1 --2- 172.21.15.67:0/3712220292 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3030101850 0x7f30301058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:33.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.366+0000 7f30356b7700 1 --2- 172.21.15.67:0/3712220292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3030100ea0 0x7f3030101280 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:33.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.366+0000 7f30356b7700 1 -- 172.21.15.67:0/3712220292 >> 172.21.15.67:0/3712220292 conn(0x7f30300fc750 msgr2=0x7f30300feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:33.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.366+0000 7f30356b7700 1 -- 172.21.15.67:0/3712220292 shutdown_connections 2024-04-23T00:05:33.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.366+0000 7f30356b7700 1 -- 172.21.15.67:0/3712220292 wait complete. 2024-04-23T00:05:33.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.367+0000 7f30356b7700 1 Processor -- start 2024-04-23T00:05:33.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.367+0000 7f30356b7700 1 -- start start 2024-04-23T00:05:33.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.368+0000 7f30356b7700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3030100ea0 0x7f3030196990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:33.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.368+0000 7f30356b7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3030101850 0x7f3030196ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:33.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.368+0000 7f30356b7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3030197560 con 0x7f3030101850 2024-04-23T00:05:33.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.368+0000 7f30356b7700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303019b370 con 0x7f3030100ea0 2024-04-23T00:05:33.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.368+0000 7f302e7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3030101850 0x7f3030196ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:33.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.368+0000 7f302e7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3030101850 0x7f3030196ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42350/0 (socket says 172.21.15.67:42350) 2024-04-23T00:05:33.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.368+0000 7f302e7fc700 1 -- 172.21.15.67:0/3809199068 learned_addr learned my addr 172.21.15.67:0/3809199068 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:33.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.368+0000 7f302effd700 1 --2- 172.21.15.67:0/3809199068 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3030100ea0 0x7f3030196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:33.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.368+0000 7f302e7fc700 1 -- 172.21.15.67:0/3809199068 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3030100ea0 msgr2=0x7f3030196990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:33.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.368+0000 7f302e7fc700 1 --2- 172.21.15.67:0/3809199068 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3030100ea0 0x7f3030196990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:33.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.368+0000 7f302e7fc700 1 -- 172.21.15.67:0/3809199068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30200095f0 con 0x7f3030101850 2024-04-23T00:05:33.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.368+0000 7f302e7fc700 1 --2- 172.21.15.67:0/3809199068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3030101850 0x7f3030196ed0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f302400b480 tx=0x7f302400b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:33.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.368+0000 7f3017fff700 1 -- 172.21.15.67:0/3809199068 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3024011800 con 0x7f3030101850 2024-04-23T00:05:33.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.368+0000 7f3017fff700 1 -- 172.21.15.67:0/3809199068 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3024011e20 con 0x7f3030101850 2024-04-23T00:05:33.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.368+0000 7f30356b7700 1 -- 172.21.15.67:0/3809199068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f303019b650 con 0x7f3030101850 2024-04-23T00:05:33.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.369+0000 7f3017fff700 1 -- 172.21.15.67:0/3809199068 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3024010ea0 con 0x7f3030101850 2024-04-23T00:05:33.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.369+0000 7f30356b7700 1 -- 172.21.15.67:0/3809199068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f303019bba0 con 0x7f3030101850 2024-04-23T00:05:33.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.369+0000 7f30356b7700 1 -- 172.21.15.67:0/3809199068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3030109240 con 0x7f3030101850 2024-04-23T00:05:33.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.370+0000 7f3017fff700 1 -- 172.21.15.67:0/3809199068 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3024010380 con 0x7f3030101850 2024-04-23T00:05:33.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.370+0000 7f3017fff700 1 --2- 172.21.15.67:0/3809199068 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f301806ecc0 0x7f3018071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:33.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.370+0000 7f3017fff700 1 -- 172.21.15.67:0/3809199068 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f302408ea00 con 0x7f3030101850 2024-04-23T00:05:33.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.371+0000 7f302effd700 1 --2- 172.21.15.67:0/3809199068 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f301806ecc0 0x7f3018071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:33.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.371+0000 7f302effd700 1 --2- 172.21.15.67:0/3809199068 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f301806ecc0 0x7f3018071180 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f3020006010 tx=0x7f302001a040 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:33.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.375+0000 7f3017fff700 1 -- 172.21.15.67:0/3809199068 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f302405b1f0 con 0x7f3030101850 2024-04-23T00:05:33.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:33 smithi134 ceph-mon[85630]: pgmap v1179: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:05:33.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:33 smithi067 ceph-mon[112438]: pgmap v1179: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:05:33.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.677+0000 7f30356b7700 1 -- 172.21.15.67:0/3809199068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f303004f2e0 con 0x7f3030101850 2024-04-23T00:05:33.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.678+0000 7f3017fff700 1 -- 172.21.15.67:0/3809199068 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f302405ad80 con 0x7f3030101850 2024-04-23T00:05:33.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.681+0000 7f30356b7700 1 -- 172.21.15.67:0/3809199068 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f301806ecc0 msgr2=0x7f3018071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:33.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.681+0000 7f30356b7700 1 --2- 172.21.15.67:0/3809199068 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f301806ecc0 0x7f3018071180 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f3020006010 tx=0x7f302001a040 comp rx=0 tx=0).stop 2024-04-23T00:05:33.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.681+0000 7f30356b7700 1 -- 172.21.15.67:0/3809199068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3030101850 msgr2=0x7f3030196ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:33.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.681+0000 7f30356b7700 1 --2- 172.21.15.67:0/3809199068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3030101850 0x7f3030196ed0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f302400b480 tx=0x7f302400b790 comp rx=0 tx=0).stop 2024-04-23T00:05:33.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.681+0000 7f30356b7700 1 -- 172.21.15.67:0/3809199068 shutdown_connections 2024-04-23T00:05:33.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.681+0000 7f30356b7700 1 --2- 172.21.15.67:0/3809199068 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f301806ecc0 0x7f3018071180 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:33.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.681+0000 7f30356b7700 1 --2- 172.21.15.67:0/3809199068 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3030100ea0 0x7f3030196990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:33.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.681+0000 7f30356b7700 1 --2- 172.21.15.67:0/3809199068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3030101850 0x7f3030196ed0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:33.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.681+0000 7f30356b7700 1 -- 172.21.15.67:0/3809199068 >> 172.21.15.67:0/3809199068 conn(0x7f30300fc750 msgr2=0x7f30300fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:33.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.681+0000 7f30356b7700 1 -- 172.21.15.67:0/3809199068 shutdown_connections 2024-04-23T00:05:33.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:33.681+0000 7f30356b7700 1 -- 172.21.15.67:0/3809199068 wait complete. 2024-04-23T00:05:33.681 INFO:teuthology.orchestra.run.smithi067.stdout:true 2024-04-23T00:05:34.280 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:34 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3809199068' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:05:34.309 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -e sha1=43be020184947e53516056c9931e1ac5bdbbb1a5 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-04-23T00:05:34.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:34 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3809199068' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:05:34.525 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-23T00:05:35.280 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:35 smithi067 ceph-mon[112438]: pgmap v1180: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:05:35.431 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:35 smithi134 ceph-mon[85630]: pgmap v1180: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:05:36.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.105+0000 7fb584724700 1 -- 172.21.15.67:0/3064884059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb57c0730b0 msgr2=0x7fb57c073490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:36.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.105+0000 7fb584724700 1 --2- 172.21.15.67:0/3064884059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb57c0730b0 0x7fb57c073490 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fb578009910 tx=0x7fb578009c20 comp rx=0 tx=0).stop 2024-04-23T00:05:36.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.105+0000 7fb584724700 1 -- 172.21.15.67:0/3064884059 shutdown_connections 2024-04-23T00:05:36.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.105+0000 7fb584724700 1 --2- 172.21.15.67:0/3064884059 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb57c0739d0 0x7fb57c10cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:36.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.105+0000 7fb584724700 1 --2- 172.21.15.67:0/3064884059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb57c0730b0 0x7fb57c073490 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:36.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.105+0000 7fb584724700 1 -- 172.21.15.67:0/3064884059 >> 172.21.15.67:0/3064884059 conn(0x7fb57c0fc750 msgr2=0x7fb57c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:36.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.105+0000 7fb584724700 1 -- 172.21.15.67:0/3064884059 shutdown_connections 2024-04-23T00:05:36.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.105+0000 7fb584724700 1 -- 172.21.15.67:0/3064884059 wait complete. 2024-04-23T00:05:36.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.105+0000 7fb584724700 1 Processor -- start 2024-04-23T00:05:36.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.105+0000 7fb584724700 1 -- start start 2024-04-23T00:05:36.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.106+0000 7fb584724700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb57c0730b0 0x7fb57c19e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:36.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.106+0000 7fb584724700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb57c0739d0 0x7fb57c19ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:36.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.106+0000 7fb584724700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57c19f380 con 0x7fb57c0730b0 2024-04-23T00:05:36.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.106+0000 7fb584724700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57c1988c0 con 0x7fb57c0739d0 2024-04-23T00:05:36.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.106+0000 7fb5824c0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb57c0730b0 0x7fb57c19e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:36.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.106+0000 7fb5824c0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb57c0730b0 0x7fb57c19e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42370/0 (socket says 172.21.15.67:42370) 2024-04-23T00:05:36.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.106+0000 7fb5824c0700 1 -- 172.21.15.67:0/4091863517 learned_addr learned my addr 172.21.15.67:0/4091863517 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:36.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.106+0000 7fb581cbf700 1 --2- 172.21.15.67:0/4091863517 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb57c0739d0 0x7fb57c19ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:36.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.106+0000 7fb5824c0700 1 -- 172.21.15.67:0/4091863517 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb57c0739d0 msgr2=0x7fb57c19ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:36.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.106+0000 7fb5824c0700 1 --2- 172.21.15.67:0/4091863517 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb57c0739d0 0x7fb57c19ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:36.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.106+0000 7fb5824c0700 1 -- 172.21.15.67:0/4091863517 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5780095f0 con 0x7fb57c0730b0 2024-04-23T00:05:36.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.106+0000 7fb5824c0700 1 --2- 172.21.15.67:0/4091863517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb57c0730b0 0x7fb57c19e840 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fb57800b5c0 tx=0x7fb57800be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:36.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.106+0000 7fb5737fe700 1 -- 172.21.15.67:0/4091863517 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb57801d070 con 0x7fb57c0730b0 2024-04-23T00:05:36.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.106+0000 7fb5737fe700 1 -- 172.21.15.67:0/4091863517 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb57800f460 con 0x7fb57c0730b0 2024-04-23T00:05:36.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.106+0000 7fb5737fe700 1 -- 172.21.15.67:0/4091863517 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb578017610 con 0x7fb57c0730b0 2024-04-23T00:05:36.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.107+0000 7fb584724700 1 -- 172.21.15.67:0/4091863517 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb57c198b40 con 0x7fb57c0730b0 2024-04-23T00:05:36.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.107+0000 7fb584724700 1 -- 172.21.15.67:0/4091863517 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb57c199030 con 0x7fb57c0730b0 2024-04-23T00:05:36.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.107+0000 7fb584724700 1 -- 172.21.15.67:0/4091863517 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb57c10a6c0 con 0x7fb57c0730b0 2024-04-23T00:05:36.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.108+0000 7fb5737fe700 1 -- 172.21.15.67:0/4091863517 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb57800f5d0 con 0x7fb57c0730b0 2024-04-23T00:05:36.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.109+0000 7fb5737fe700 1 --2- 172.21.15.67:0/4091863517 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb56806ecc0 0x7fb568071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:36.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.109+0000 7fb581cbf700 1 --2- 172.21.15.67:0/4091863517 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb56806ecc0 0x7fb568071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:36.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.109+0000 7fb5737fe700 1 -- 172.21.15.67:0/4091863517 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fb578090510 con 0x7fb57c0730b0 2024-04-23T00:05:36.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.109+0000 7fb581cbf700 1 --2- 172.21.15.67:0/4091863517 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb56806ecc0 0x7fb568071180 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fb56c005ea0 tx=0x7fb56c005e30 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:36.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.114+0000 7fb5737fe700 1 -- 172.21.15.67:0/4091863517 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fb57805cd00 con 0x7fb57c0730b0 2024-04-23T00:05:36.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.413+0000 7fb584724700 1 -- 172.21.15.67:0/4091863517 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb57c199dd0 con 0x7fb57c0730b0 2024-04-23T00:05:36.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.413+0000 7fb5737fe700 1 -- 172.21.15.67:0/4091863517 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fb57805c890 con 0x7fb57c0730b0 2024-04-23T00:05:36.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.415+0000 7fb584724700 1 -- 172.21.15.67:0/4091863517 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb56806ecc0 msgr2=0x7fb568071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:36.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.415+0000 7fb584724700 1 --2- 172.21.15.67:0/4091863517 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb56806ecc0 0x7fb568071180 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fb56c005ea0 tx=0x7fb56c005e30 comp rx=0 tx=0).stop 2024-04-23T00:05:36.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.415+0000 7fb584724700 1 -- 172.21.15.67:0/4091863517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb57c0730b0 msgr2=0x7fb57c19e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:36.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.415+0000 7fb584724700 1 --2- 172.21.15.67:0/4091863517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb57c0730b0 0x7fb57c19e840 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fb57800b5c0 tx=0x7fb57800be00 comp rx=0 tx=0).stop 2024-04-23T00:05:36.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.415+0000 7fb584724700 1 -- 172.21.15.67:0/4091863517 shutdown_connections 2024-04-23T00:05:36.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.415+0000 7fb584724700 1 --2- 172.21.15.67:0/4091863517 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fb56806ecc0 0x7fb568071180 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:36.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.415+0000 7fb584724700 1 --2- 172.21.15.67:0/4091863517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb57c0730b0 0x7fb57c19e840 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:36.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.415+0000 7fb584724700 1 --2- 172.21.15.67:0/4091863517 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb57c0739d0 0x7fb57c19ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:36.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.415+0000 7fb584724700 1 -- 172.21.15.67:0/4091863517 >> 172.21.15.67:0/4091863517 conn(0x7fb57c0fc750 msgr2=0x7fb57c107800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:36.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.416+0000 7fb584724700 1 -- 172.21.15.67:0/4091863517 shutdown_connections 2024-04-23T00:05:36.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:36.416+0000 7fb584724700 1 -- 172.21.15.67:0/4091863517 wait complete. 2024-04-23T00:05:36.412 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph version 19.0.0-3252-g43be0201 (43be020184947e53516056c9931e1ac5bdbbb1a5) squid (dev)" 2024-04-23T00:05:37.199 DEBUG:teuthology.parallel:result is None 2024-04-23T00:05:37.200 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-23T00:05:37.257 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi067.front.sepia.ceph.com 2024-04-23T00:05:37.257 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d46c346e-00fd-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-23T00:05:37.474 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/mon.smithi067/config 2024-04-23T00:05:37.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:37 smithi134 ceph-mon[85630]: pgmap v1181: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:05:37.481 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:37 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/4091863517' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:05:37.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:37 smithi067 ceph-mon[112438]: pgmap v1181: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:05:37.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:37 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/4091863517' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T00:05:38.881 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:38 smithi067 ceph-mon[112438]: pgmap v1182: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:05:39.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.121+0000 7f05a81fc700 1 -- 172.21.15.67:0/2845313615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05a0101060 msgr2=0x7f05a01047b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:39.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.121+0000 7f05a81fc700 1 --2- 172.21.15.67:0/2845313615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05a0101060 0x7f05a01047b0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f0590009910 tx=0x7f0590009c20 comp rx=0 tx=0).stop 2024-04-23T00:05:39.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.121+0000 7f05a81fc700 1 -- 172.21.15.67:0/2845313615 shutdown_connections 2024-04-23T00:05:39.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.121+0000 7f05a81fc700 1 --2- 172.21.15.67:0/2845313615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05a0101060 0x7f05a01047b0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:39.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.121+0000 7f05a81fc700 1 --2- 172.21.15.67:0/2845313615 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f05a01006b0 0x7f05a0100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:39.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.121+0000 7f05a81fc700 1 -- 172.21.15.67:0/2845313615 >> 172.21.15.67:0/2845313615 conn(0x7f05a0074e30 msgr2=0x7f05a0075240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:39.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.121+0000 7f05a81fc700 1 -- 172.21.15.67:0/2845313615 shutdown_connections 2024-04-23T00:05:39.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.121+0000 7f05a81fc700 1 -- 172.21.15.67:0/2845313615 wait complete. 2024-04-23T00:05:39.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.122+0000 7f05a81fc700 1 Processor -- start 2024-04-23T00:05:39.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.122+0000 7f05a81fc700 1 -- start start 2024-04-23T00:05:39.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.122+0000 7f05a81fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05a01006b0 0x7f05a019e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:39.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.122+0000 7f05a81fc700 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f05a0101060 0x7f05a019ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:39.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.122+0000 7f05a81fc700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a019f470 con 0x7f05a01006b0 2024-04-23T00:05:39.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.122+0000 7f05a81fc700 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a0198920 con 0x7f05a0101060 2024-04-23T00:05:39.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.122+0000 7f05a5f98700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05a01006b0 0x7f05a019e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:39.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.122+0000 7f05a5f98700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05a01006b0 0x7f05a019e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38734/0 (socket says 172.21.15.67:38734) 2024-04-23T00:05:39.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.122+0000 7f05a5f98700 1 -- 172.21.15.67:0/1638846033 learned_addr learned my addr 172.21.15.67:0/1638846033 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:39.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.122+0000 7f05a5797700 1 --2- 172.21.15.67:0/1638846033 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f05a0101060 0x7f05a019ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:39.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.122+0000 7f05a5f98700 1 -- 172.21.15.67:0/1638846033 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f05a0101060 msgr2=0x7f05a019ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:39.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.122+0000 7f05a5f98700 1 --2- 172.21.15.67:0/1638846033 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f05a0101060 0x7f05a019ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:39.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.122+0000 7f05a5f98700 1 -- 172.21.15.67:0/1638846033 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05900095f0 con 0x7f05a01006b0 2024-04-23T00:05:39.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.122+0000 7f05a5797700 1 --2- 172.21.15.67:0/1638846033 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f05a0101060 0x7f05a019ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T00:05:39.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.122+0000 7f05a5f98700 1 --2- 172.21.15.67:0/1638846033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05a01006b0 0x7f05a019e8a0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f059c00d8d0 tx=0x7f059c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:39.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.123+0000 7f0596ffd700 1 -- 172.21.15.67:0/1638846033 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f059c009940 con 0x7f05a01006b0 2024-04-23T00:05:39.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.123+0000 7f0596ffd700 1 -- 172.21.15.67:0/1638846033 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f059c010460 con 0x7f05a01006b0 2024-04-23T00:05:39.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.123+0000 7f05a81fc700 1 -- 172.21.15.67:0/1638846033 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05a0198c00 con 0x7f05a01006b0 2024-04-23T00:05:39.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.123+0000 7f0596ffd700 1 -- 172.21.15.67:0/1638846033 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f059c00f5f0 con 0x7f05a01006b0 2024-04-23T00:05:39.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.123+0000 7f05a81fc700 1 -- 172.21.15.67:0/1638846033 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05a0199150 con 0x7f05a01006b0 2024-04-23T00:05:39.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.124+0000 7f05a81fc700 1 -- 172.21.15.67:0/1638846033 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05a004f2e0 con 0x7f05a01006b0 2024-04-23T00:05:39.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.124+0000 7f0596ffd700 1 -- 172.21.15.67:0/1638846033 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f059c0105d0 con 0x7f05a01006b0 2024-04-23T00:05:39.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.125+0000 7f0596ffd700 1 --2- 172.21.15.67:0/1638846033 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f058c06ecc0 0x7f058c071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:39.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.125+0000 7f0596ffd700 1 -- 172.21.15.67:0/1638846033 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f059c08eee0 con 0x7f05a01006b0 2024-04-23T00:05:39.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.125+0000 7f05a5797700 1 --2- 172.21.15.67:0/1638846033 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f058c06ecc0 0x7f058c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:39.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.125+0000 7f05a5797700 1 --2- 172.21.15.67:0/1638846033 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f058c06ecc0 0x7f058c071180 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f059000b5c0 tx=0x7f05900053a0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:39.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.129+0000 7f0596ffd700 1 -- 172.21.15.67:0/1638846033 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f059c05b6d0 con 0x7f05a01006b0 2024-04-23T00:05:39.230 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:38 smithi134 ceph-mon[85630]: pgmap v1182: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:05:39.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.387+0000 7f05a81fc700 1 -- 172.21.15.67:0/1638846033 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f05a0199480 con 0x7f05a01006b0 2024-04-23T00:05:39.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.388+0000 7f0596ffd700 1 -- 172.21.15.67:0/1638846033 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 42 v42) v1 ==== 76+0+1966 (secure 0 0 0) 0x7f059c0167c0 con 0x7f05a01006b0 2024-04-23T00:05:39.374 INFO:teuthology.orchestra.run.smithi067.stdout:e42 2024-04-23T00:05:39.375 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-23T00:05:39.375 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-23T00:05:39.375 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-04-23T00:05:39.375 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:05:39.375 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-04-23T00:05:39.375 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-04-23T00:05:39.375 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 42 2024-04-23T00:05:39.375 INFO:teuthology.orchestra.run.smithi067.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-23T00:05:39.375 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-22T23:20:20.823261+0000 2024-04-23T00:05:39.375 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-23T00:03:56.266657+0000 2024-04-23T00:05:39.375 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-04-23T00:05:39.375 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-04-23T00:05:39.375 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 60 2024-04-23T00:05:39.375 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-04-23T00:05:39.376 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-04-23T00:05:39.376 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-04-23T00:05:39.376 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-04-23T00:05:39.376 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-04-23T00:05:39.376 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 90 2024-04-23T00:05:39.376 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-23T00:05:39.376 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 2 2024-04-23T00:05:39.376 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1 2024-04-23T00:05:39.376 INFO:teuthology.orchestra.run.smithi067.stdout:up {0=34396,1=34424} 2024-04-23T00:05:39.376 INFO:teuthology.orchestra.run.smithi067.stdout:failed 2024-04-23T00:05:39.376 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-04-23T00:05:39.376 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-04-23T00:05:39.376 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-04-23T00:05:39.376 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-04-23T00:05:39.376 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data enabled 2024-04-23T00:05:39.376 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-04-23T00:05:39.377 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-04-23T00:05:39.377 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 1 2024-04-23T00:05:39.377 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 34396 members: 34396,34424 2024-04-23T00:05:39.377 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.aqlxpv{0:34396} state up:active seq 469 join_fscid=1 addr [v2:172.21.15.67:6826/789475804,v1:172.21.15.67:6827/789475804] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T00:05:39.377 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.owipad{0:44327} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.134:6824/1473208408,v1:172.21.15.134:6825/1473208408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T00:05:39.377 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi067.xyazmh{1:34424} state up:active seq 464 join_fscid=1 addr [v2:172.21.15.67:6828/927062647,v1:172.21.15.67:6829/927062647] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T00:05:39.377 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.cephfs.smithi134.vuxbsj{1:44851} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.134:6826/1849648622,v1:172.21.15.134:6827/1849648622] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T00:05:39.377 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:05:39.377 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:05:39.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.391+0000 7f05a81fc700 1 -- 172.21.15.67:0/1638846033 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f058c06ecc0 msgr2=0x7f058c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:39.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.391+0000 7f05a81fc700 1 --2- 172.21.15.67:0/1638846033 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f058c06ecc0 0x7f058c071180 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f059000b5c0 tx=0x7f05900053a0 comp rx=0 tx=0).stop 2024-04-23T00:05:39.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.391+0000 7f05a81fc700 1 -- 172.21.15.67:0/1638846033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05a01006b0 msgr2=0x7f05a019e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:39.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.391+0000 7f05a81fc700 1 --2- 172.21.15.67:0/1638846033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05a01006b0 0x7f05a019e8a0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f059c00d8d0 tx=0x7f059c00dc90 comp rx=0 tx=0).stop 2024-04-23T00:05:39.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.391+0000 7f05a81fc700 1 -- 172.21.15.67:0/1638846033 shutdown_connections 2024-04-23T00:05:39.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.391+0000 7f05a81fc700 1 --2- 172.21.15.67:0/1638846033 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f058c06ecc0 0x7f058c071180 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:39.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.391+0000 7f05a81fc700 1 --2- 172.21.15.67:0/1638846033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05a01006b0 0x7f05a019e8a0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:39.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.391+0000 7f05a81fc700 1 --2- 172.21.15.67:0/1638846033 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f05a0101060 0x7f05a019ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:39.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.391+0000 7f05a81fc700 1 -- 172.21.15.67:0/1638846033 >> 172.21.15.67:0/1638846033 conn(0x7f05a0074e30 msgr2=0x7f05a0103790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:39.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.391+0000 7f05a81fc700 1 -- 172.21.15.67:0/1638846033 shutdown_connections 2024-04-23T00:05:39.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:39.391+0000 7f05a81fc700 1 -- 172.21.15.67:0/1638846033 wait complete. 2024-04-23T00:05:39.379 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 42 2024-04-23T00:05:40.001 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:39 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1638846033' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-23T00:05:40.023 INFO:teuthology.run_tasks:Running task fs.post_upgrade_checks... 2024-04-23T00:05:40.035 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T00:05:40.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.159+0000 7f4d4552e640 1 -- 172.21.15.67:0/1237418358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d400ff1a0 msgr2=0x7f4d400ff580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:40.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.159+0000 7f4d4552e640 1 --2- 172.21.15.67:0/1237418358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d400ff1a0 0x7f4d400ff580 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f4d38009880 tx=0x7f4d3802bce0 comp rx=0 tx=0).stop 2024-04-23T00:05:40.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.159+0000 7f4d4552e640 1 -- 172.21.15.67:0/1237418358 shutdown_connections 2024-04-23T00:05:40.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.159+0000 7f4d4552e640 1 --2- 172.21.15.67:0/1237418358 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4d400ffb50 0x7f4d40103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:40.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.159+0000 7f4d4552e640 1 --2- 172.21.15.67:0/1237418358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d400ff1a0 0x7f4d400ff580 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:40.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.159+0000 7f4d4552e640 1 -- 172.21.15.67:0/1237418358 >> 172.21.15.67:0/1237418358 conn(0x7f4d400faa40 msgr2=0x7f4d400fce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:40.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.159+0000 7f4d4552e640 1 -- 172.21.15.67:0/1237418358 shutdown_connections 2024-04-23T00:05:40.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.159+0000 7f4d4552e640 1 -- 172.21.15.67:0/1237418358 wait complete. 2024-04-23T00:05:40.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.160+0000 7f4d4552e640 1 Processor -- start 2024-04-23T00:05:40.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.160+0000 7f4d4552e640 1 -- start start 2024-04-23T00:05:40.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.160+0000 7f4d4552e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d400ff1a0 0x7f4d40195d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:40.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.160+0000 7f4d4552e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4d40196290 0x7f4d4019a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:40.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.160+0000 7f4d4552e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d40196800 con 0x7f4d400ff1a0 2024-04-23T00:05:40.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.160+0000 7f4d4552e640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d40196970 con 0x7f4d40196290 2024-04-23T00:05:40.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.160+0000 7f4d3effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d400ff1a0 0x7f4d40195d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:40.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.160+0000 7f4d3effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d400ff1a0 0x7f4d40195d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38752/0 (socket says 172.21.15.67:38752) 2024-04-23T00:05:40.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.160+0000 7f4d3effd640 1 -- 172.21.15.67:0/3763592098 learned_addr learned my addr 172.21.15.67:0/3763592098 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:40.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.160+0000 7f4d3effd640 1 -- 172.21.15.67:0/3763592098 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4d40196290 msgr2=0x7f4d4019a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:40.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.160+0000 7f4d3effd640 1 --2- 172.21.15.67:0/3763592098 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4d40196290 0x7f4d4019a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:40.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.160+0000 7f4d3effd640 1 -- 172.21.15.67:0/3763592098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d38009560 con 0x7f4d400ff1a0 2024-04-23T00:05:40.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.160+0000 7f4d3effd640 1 --2- 172.21.15.67:0/3763592098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d400ff1a0 0x7f4d40195d50 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f4d38009230 tx=0x7f4d38004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:40.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.161+0000 7f4d37fff640 1 -- 172.21.15.67:0/3763592098 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d380035f0 con 0x7f4d400ff1a0 2024-04-23T00:05:40.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.161+0000 7f4d37fff640 1 -- 172.21.15.67:0/3763592098 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4d38003c10 con 0x7f4d400ff1a0 2024-04-23T00:05:40.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.161+0000 7f4d37fff640 1 -- 172.21.15.67:0/3763592098 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d38034440 con 0x7f4d400ff1a0 2024-04-23T00:05:40.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.161+0000 7f4d4552e640 1 -- 172.21.15.67:0/3763592098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d4019ac90 con 0x7f4d400ff1a0 2024-04-23T00:05:40.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.161+0000 7f4d4552e640 1 -- 172.21.15.67:0/3763592098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d4019b1a0 con 0x7f4d400ff1a0 2024-04-23T00:05:40.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.162+0000 7f4d4552e640 1 -- 172.21.15.67:0/3763592098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d4004e040 con 0x7f4d400ff1a0 2024-04-23T00:05:40.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.162+0000 7f4d37fff640 1 -- 172.21.15.67:0/3763592098 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4d38004fd0 con 0x7f4d400ff1a0 2024-04-23T00:05:40.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.163+0000 7f4d37fff640 1 --2- 172.21.15.67:0/3763592098 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4d2c06ecf0 0x7f4d2c0711b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:40.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.163+0000 7f4d3e7fc640 1 --2- 172.21.15.67:0/3763592098 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4d2c06ecf0 0x7f4d2c0711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:40.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.163+0000 7f4d37fff640 1 -- 172.21.15.67:0/3763592098 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f4d380b1380 con 0x7f4d400ff1a0 2024-04-23T00:05:40.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.163+0000 7f4d3e7fc640 1 --2- 172.21.15.67:0/3763592098 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4d2c06ecf0 0x7f4d2c0711b0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f4d40069820 tx=0x7f4d28005c70 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:40.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.168+0000 7f4d37fff640 1 -- 172.21.15.67:0/3763592098 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f4d3807a040 con 0x7f4d400ff1a0 2024-04-23T00:05:40.230 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:39 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1638846033' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-23T00:05:40.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.358+0000 7f4d4552e640 1 -- 172.21.15.67:0/3763592098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4d4019b410 con 0x7f4d400ff1a0 2024-04-23T00:05:40.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.359+0000 7f4d37fff640 1 -- 172.21.15.67:0/3763592098 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+17185 (secure 0 0 0) 0x7f4d3807d700 con 0x7f4d400ff1a0 2024-04-23T00:05:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:05:40.347 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":94,"fsid":"d46c346e-00fd-11ef-bc93-c7b262605968","created":"2024-04-22T23:13:18.687917+0000","modified":"2024-04-22T23:48:20.827933+0000","last_up_change":"2024-04-22T23:32:23.175924+0000","last_in_change":"2024-04-22T23:18:41.696935+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T23:17:47.912850+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-22T23:20:19.759956+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-22T23:20:20.252379+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"25c9acd1-5743-4dc0-aee0-c7bb8884c0a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":49,"up_from":53,"up_thru":86,"down_at":50,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1540469345},{"type":"v1","addr":"172.21.15.67:6803","nonce":1540469345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1540469345},{"type":"v1","addr":"172.21.15.67:6805","nonce":1540469345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1540469345},{"type":"v1","addr":"172.21.15.67:6809","nonce":1540469345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1540469345},{"type":"v1","addr":"172.21.15.67:6807","nonce":1540469345}]},"public_addr":"172.21.15.67:6803/1540469345","cluster_addr":"172.21.15.67:6805/1540469345","heartbeat_back_addr":"172.21.15.67:6809/1540469345","heartbeat_front_addr":"172.21.15.67:6807/1540469345","state":["exists","up"]},{"osd":1,"uuid":"8be83014-01af-44d3-ab2f-b27840506e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":54,"up_from":63,"up_thru":86,"down_at":55,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1745058179},{"type":"v1","addr":"172.21.15.67:6811","nonce":1745058179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1745058179},{"type":"v1","addr":"172.21.15.67:6813","nonce":1745058179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1745058179},{"type":"v1","addr":"172.21.15.67:6817","nonce":1745058179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1745058179},{"type":"v1","addr":"172.21.15.67:6815","nonce":1745058179}]},"public_addr":"172.21.15.67:6811/1745058179","cluster_addr":"172.21.15.67:6813/1745058179","heartbeat_back_addr":"172.21.15.67:6817/1745058179","heartbeat_front_addr":"172.21.15.67:6815/1745058179","state":["exists","up"]},{"osd":2,"uuid":"ab1cbcda-14dd-4c24-876b-25ad5d4d8563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":67,"up_from":71,"up_thru":86,"down_at":68,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":121705471},{"type":"v1","addr":"172.21.15.67:6819","nonce":121705471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":121705471},{"type":"v1","addr":"172.21.15.67:6821","nonce":121705471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":121705471},{"type":"v1","addr":"172.21.15.67:6825","nonce":121705471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":121705471},{"type":"v1","addr":"172.21.15.67:6823","nonce":121705471}]},"public_addr":"172.21.15.67:6819/121705471","cluster_addr":"172.21.15.67:6821/121705471","heartbeat_back_addr":"172.21.15.67:6825/121705471","heartbeat_front_addr":"172.21.15.67:6823/121705471","state":["exists","up"]},{"osd":3,"uuid":"1a30b6c3-95bf-4b19-883b-1271ff359e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":72,"up_from":76,"up_thru":86,"down_at":73,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":3344693085},{"type":"v1","addr":"172.21.15.134:6801","nonce":3344693085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":3344693085},{"type":"v1","addr":"172.21.15.134:6803","nonce":3344693085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":3344693085},{"type":"v1","addr":"172.21.15.134:6807","nonce":3344693085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":3344693085},{"type":"v1","addr":"172.21.15.134:6805","nonce":3344693085}]},"public_addr":"172.21.15.134:6801/3344693085","cluster_addr":"172.21.15.134:6803/3344693085","heartbeat_back_addr":"172.21.15.134:6807/3344693085","heartbeat_front_addr":"172.21.15.134:6805/3344693085","state":["exists","up"]},{"osd":4,"uuid":"53368fc1-28fd-4622-bc9c-1d8901b06bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":34,"last_clean_end":77,"up_from":81,"up_thru":86,"down_at":78,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":3935208416},{"type":"v1","addr":"172.21.15.134:6809","nonce":3935208416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":3935208416},{"type":"v1","addr":"172.21.15.134:6811","nonce":3935208416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":3935208416},{"type":"v1","addr":"172.21.15.134:6815","nonce":3935208416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":3935208416},{"type":"v1","addr":"172.21.15.134:6813","nonce":3935208416}]},"public_addr":"172.21.15.134:6809/3935208416","cluster_addr":"172.21.15.134:6811/3935208416","heartbeat_back_addr":"172.21.15.134:6815/3935208416","heartbeat_front_addr":"172.21.15.134:6813/3935208416","state":["exists","up"]},{"osd":5,"uuid":"31a885f7-8f91-4b6a-922a-065753f70a95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":82,"up_from":86,"up_thru":86,"down_at":83,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":921342983},{"type":"v1","addr":"172.21.15.134:6817","nonce":921342983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":921342983},{"type":"v1","addr":"172.21.15.134:6819","nonce":921342983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":921342983},{"type":"v1","addr":"172.21.15.134:6823","nonce":921342983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":921342983},{"type":"v1","addr":"172.21.15.134:6821","nonce":921342983}]},"public_addr":"172.21.15.134:6817/921342983","cluster_addr":"172.21.15.134:6819/921342983","heartbeat_back_addr":"172.21.15.134:6823/921342983","heartbeat_front_addr":"172.21.15.134:6821/921342983","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:16:51.179651+0000","dead_epoch":49},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:17:18.694683+0000","dead_epoch":54},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:17:45.368963+0000","dead_epoch":67},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:18:08.585046+0000","dead_epoch":72},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:18:33.412804+0000","dead_epoch":77},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:18:58.909758+0000","dead_epoch":82}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.2","mappings":[{"from":0,"to":2}]},{"pgid":"2.8","mappings":[{"from":0,"to":1}]},{"pgid":"2.b","mappings":[{"from":3,"to":1}]},{"pgid":"2.c","mappings":[{"from":3,"to":1}]},{"pgid":"2.f","mappings":[{"from":0,"to":1}]},{"pgid":"2.16","mappings":[{"from":3,"to":1}]},{"pgid":"3.1","mappings":[{"from":4,"to":1}]},{"pgid":"3.7","mappings":[{"from":4,"to":0}]},{"pgid":"3.c","mappings":[{"from":3,"to":1}]},{"pgid":"3.12","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6828/3606151860":"2024-04-23T23:32:27.027372+0000","172.21.15.134:6824/535957142":"2024-04-23T23:32:27.027388+0000","172.21.15.67:0/1189311107":"2024-04-23T23:15:37.396154+0000","172.21.15.67:6827/2381214817":"2024-04-23T23:20:37.496135+0000","172.21.15.67:6801/2590792187":"2024-04-23T23:15:37.396154+0000","172.21.15.67:6826/2381214817":"2024-04-23T23:20:37.496135+0000","172.21.15.134:0/3526908760":"2024-04-23T23:25:00.458625+0000","172.21.15.134:0/1536532822":"2024-04-23T23:25:00.458625+0000","172.21.15.134:6825/535957142":"2024-04-23T23:32:27.027388+0000","172.21.15.67:0/895378312":"2024-04-23T23:13:56.301645+0000","172.21.15.134:6828/1077503272":"2024-04-23T23:25:00.458625+0000","172.21.15.67:0/1125624760":"2024-04-23T23:15:37.396154+0000","172.21.15.67:0/3387250764":"2024-04-23T23:13:56.301645+0000","172.21.15.67:0/3921081246":"2024-04-23T23:27:55.947537+0000","172.21.15.67:0/2204477458":"2024-04-23T23:23:56.782522+0000","172.21.15.67:0/3730369564":"2024-04-23T23:27:55.947537+0000","172.21.15.134:0/2385923868":"2024-04-23T23:25:00.458625+0000","172.21.15.67:6800/2589747850":"2024-04-23T23:23:56.782522+0000","172.21.15.67:6801/2589747850":"2024-04-23T23:23:56.782522+0000","172.21.15.134:0/2817304592":"2024-04-23T23:25:00.458625+0000","172.21.15.67:6829/3606151860":"2024-04-23T23:32:27.027372+0000","172.21.15.67:0/233140160":"2024-04-23T23:23:56.782522+0000","172.21.15.67:0/3750652317":"2024-04-23T23:23:56.782522+0000","172.21.15.67:6800/2451437776":"2024-04-23T23:13:56.301645+0000","172.21.15.67:0/2665297739":"2024-04-23T23:14:29.214977+0000","172.21.15.67:6800/3568049827":"2024-04-23T23:14:29.214977+0000","172.21.15.67:0/4165654004":"2024-04-23T23:14:29.214977+0000","172.21.15.134:0/3046392865":"2024-04-23T23:25:00.458625+0000","172.21.15.67:0/2052573085":"2024-04-23T23:14:29.214977+0000","172.21.15.134:0/1251980938":"2024-04-23T23:25:00.458625+0000","172.21.15.67:6801/2451437776":"2024-04-23T23:13:56.301645+0000","172.21.15.67:0/4286433529":"2024-04-23T23:27:55.947537+0000","172.21.15.67:0/853292259":"2024-04-23T23:13:56.301645+0000","172.21.15.67:6801/3568049827":"2024-04-23T23:14:29.214977+0000","172.21.15.67:6800/2590792187":"2024-04-23T23:15:37.396154+0000","172.21.15.67:0/1933271294":"2024-04-23T23:23:56.782522+0000","172.21.15.134:6829/1077503272":"2024-04-23T23:25:00.458625+0000","172.21.15.67:0/1737320548":"2024-04-23T23:27:55.947537+0000","172.21.15.67:6800/2306276228":"2024-04-23T23:27:55.947537+0000","172.21.15.67:0/2137374599":"2024-04-23T23:15:37.396154+0000","172.21.15.67:6801/2306276228":"2024-04-23T23:27:55.947537+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T00:05:40.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.361+0000 7f4d4552e640 1 -- 172.21.15.67:0/3763592098 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4d2c06ecf0 msgr2=0x7f4d2c0711b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:40.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.361+0000 7f4d4552e640 1 --2- 172.21.15.67:0/3763592098 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4d2c06ecf0 0x7f4d2c0711b0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f4d40069820 tx=0x7f4d28005c70 comp rx=0 tx=0).stop 2024-04-23T00:05:40.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.361+0000 7f4d4552e640 1 -- 172.21.15.67:0/3763592098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d400ff1a0 msgr2=0x7f4d40195d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:40.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.361+0000 7f4d4552e640 1 --2- 172.21.15.67:0/3763592098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d400ff1a0 0x7f4d40195d50 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f4d38009230 tx=0x7f4d38004040 comp rx=0 tx=0).stop 2024-04-23T00:05:40.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.362+0000 7f4d4552e640 1 -- 172.21.15.67:0/3763592098 shutdown_connections 2024-04-23T00:05:40.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.362+0000 7f4d4552e640 1 --2- 172.21.15.67:0/3763592098 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4d2c06ecf0 0x7f4d2c0711b0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:40.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.362+0000 7f4d4552e640 1 --2- 172.21.15.67:0/3763592098 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4d40196290 0x7f4d4019a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:40.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.362+0000 7f4d4552e640 1 --2- 172.21.15.67:0/3763592098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d400ff1a0 0x7f4d40195d50 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:40.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.362+0000 7f4d4552e640 1 -- 172.21.15.67:0/3763592098 >> 172.21.15.67:0/3763592098 conn(0x7f4d400faa40 msgr2=0x7f4d40103370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:40.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.362+0000 7f4d4552e640 1 -- 172.21.15.67:0/3763592098 shutdown_connections 2024-04-23T00:05:40.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.362+0000 7f4d4552e640 1 -- 172.21.15.67:0/3763592098 wait complete. 2024-04-23T00:05:40.360 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T23:17:47.912850+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-22T23:20:19.759956+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-22T23:20:20.252379+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T00:05:40.360 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T00:05:40.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.478+0000 7f2c62100640 1 -- 172.21.15.67:0/527681521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c5c067f70 msgr2=0x7f2c5c104c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:40.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.478+0000 7f2c62100640 1 --2- 172.21.15.67:0/527681521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c5c067f70 0x7f2c5c104c70 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f2c50009880 tx=0x7f2c5002bce0 comp rx=0 tx=0).stop 2024-04-23T00:05:40.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.478+0000 7f2c62100640 1 -- 172.21.15.67:0/527681521 shutdown_connections 2024-04-23T00:05:40.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.478+0000 7f2c62100640 1 --2- 172.21.15.67:0/527681521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c5c067f70 0x7f2c5c104c70 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:40.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.478+0000 7f2c62100640 1 --2- 172.21.15.67:0/527681521 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c5c0675c0 0x7f2c5c0679a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:40.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.478+0000 7f2c62100640 1 -- 172.21.15.67:0/527681521 >> 172.21.15.67:0/527681521 conn(0x7f2c5c074a50 msgr2=0x7f2c5c074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:40.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.478+0000 7f2c62100640 1 -- 172.21.15.67:0/527681521 shutdown_connections 2024-04-23T00:05:40.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.478+0000 7f2c62100640 1 -- 172.21.15.67:0/527681521 wait complete. 2024-04-23T00:05:40.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.479+0000 7f2c62100640 1 Processor -- start 2024-04-23T00:05:40.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.479+0000 7f2c62100640 1 -- start start 2024-04-23T00:05:40.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.479+0000 7f2c62100640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c5c0675c0 0x7f2c5c195d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:40.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.479+0000 7f2c62100640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c5c1962c0 0x7f2c5c19a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:40.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.479+0000 7f2c62100640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5c1968c0 con 0x7f2c5c0675c0 2024-04-23T00:05:40.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.479+0000 7f2c62100640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5c196a30 con 0x7f2c5c1962c0 2024-04-23T00:05:40.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.479+0000 7f2c5b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c5c0675c0 0x7f2c5c195d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:40.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.479+0000 7f2c5b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c5c0675c0 0x7f2c5c195d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38764/0 (socket says 172.21.15.67:38764) 2024-04-23T00:05:40.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.480+0000 7f2c5b7fe640 1 -- 172.21.15.67:0/4109303560 learned_addr learned my addr 172.21.15.67:0/4109303560 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:40.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.480+0000 7f2c5affd640 1 --2- 172.21.15.67:0/4109303560 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c5c1962c0 0x7f2c5c19a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:40.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.480+0000 7f2c5b7fe640 1 -- 172.21.15.67:0/4109303560 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c5c1962c0 msgr2=0x7f2c5c19a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:40.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.480+0000 7f2c5b7fe640 1 --2- 172.21.15.67:0/4109303560 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c5c1962c0 0x7f2c5c19a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:40.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.480+0000 7f2c5b7fe640 1 -- 172.21.15.67:0/4109303560 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c50009560 con 0x7f2c5c0675c0 2024-04-23T00:05:40.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.480+0000 7f2c5affd640 1 --2- 172.21.15.67:0/4109303560 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c5c1962c0 0x7f2c5c19a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T00:05:40.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.480+0000 7f2c5b7fe640 1 --2- 172.21.15.67:0/4109303560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c5c0675c0 0x7f2c5c195d80 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c00d720 tx=0x7f2c4c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:40.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.480+0000 7f2c58ff9640 1 -- 172.21.15.67:0/4109303560 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c4c0051f0 con 0x7f2c5c0675c0 2024-04-23T00:05:40.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.480+0000 7f2c62100640 1 -- 172.21.15.67:0/4109303560 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c5c19ad20 con 0x7f2c5c0675c0 2024-04-23T00:05:40.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.480+0000 7f2c58ff9640 1 -- 172.21.15.67:0/4109303560 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2c4c009ce0 con 0x7f2c5c0675c0 2024-04-23T00:05:40.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.480+0000 7f2c58ff9640 1 -- 172.21.15.67:0/4109303560 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c4c010460 con 0x7f2c5c0675c0 2024-04-23T00:05:40.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.480+0000 7f2c62100640 1 -- 172.21.15.67:0/4109303560 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c5c19b280 con 0x7f2c5c0675c0 2024-04-23T00:05:40.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.481+0000 7f2c62100640 1 -- 172.21.15.67:0/4109303560 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c5c04e040 con 0x7f2c5c0675c0 2024-04-23T00:05:40.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.482+0000 7f2c58ff9640 1 -- 172.21.15.67:0/4109303560 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2c4c005350 con 0x7f2c5c0675c0 2024-04-23T00:05:40.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.482+0000 7f2c58ff9640 1 --2- 172.21.15.67:0/4109303560 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2c4406ec30 0x7f2c440710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:40.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.482+0000 7f2c58ff9640 1 -- 172.21.15.67:0/4109303560 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f2c4c08eb60 con 0x7f2c5c0675c0 2024-04-23T00:05:40.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.482+0000 7f2c5affd640 1 --2- 172.21.15.67:0/4109303560 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2c4406ec30 0x7f2c440710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:40.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.483+0000 7f2c5affd640 1 --2- 172.21.15.67:0/4109303560 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2c4406ec30 0x7f2c440710f0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f2c50005ec0 tx=0x7f2c50005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:40.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.487+0000 7f2c58ff9640 1 -- 172.21.15.67:0/4109303560 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f2c4c0577a0 con 0x7f2c5c0675c0 2024-04-23T00:05:40.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.674+0000 7f2c62100640 1 -- 172.21.15.67:0/4109303560 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2c5c19b500 con 0x7f2c5c0675c0 2024-04-23T00:05:40.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.674+0000 7f2c58ff9640 1 -- 172.21.15.67:0/4109303560 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2c4c05ae60 con 0x7f2c5c0675c0 2024-04-23T00:05:40.661 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-23T00:05:40.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.677+0000 7f2c62100640 1 -- 172.21.15.67:0/4109303560 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2c4406ec30 msgr2=0x7f2c440710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:40.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.677+0000 7f2c62100640 1 --2- 172.21.15.67:0/4109303560 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2c4406ec30 0x7f2c440710f0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f2c50005ec0 tx=0x7f2c50005fd0 comp rx=0 tx=0).stop 2024-04-23T00:05:40.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.677+0000 7f2c62100640 1 -- 172.21.15.67:0/4109303560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c5c0675c0 msgr2=0x7f2c5c195d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:40.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.677+0000 7f2c62100640 1 --2- 172.21.15.67:0/4109303560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c5c0675c0 0x7f2c5c195d80 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c00d720 tx=0x7f2c4c00dbe0 comp rx=0 tx=0).stop 2024-04-23T00:05:40.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.677+0000 7f2c62100640 1 -- 172.21.15.67:0/4109303560 shutdown_connections 2024-04-23T00:05:40.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.677+0000 7f2c62100640 1 --2- 172.21.15.67:0/4109303560 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f2c4406ec30 0x7f2c440710f0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:40.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.677+0000 7f2c62100640 1 --2- 172.21.15.67:0/4109303560 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c5c1962c0 0x7f2c5c19a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:40.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.677+0000 7f2c62100640 1 --2- 172.21.15.67:0/4109303560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c5c0675c0 0x7f2c5c195d80 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:40.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.677+0000 7f2c62100640 1 -- 172.21.15.67:0/4109303560 >> 172.21.15.67:0/4109303560 conn(0x7f2c5c074a50 msgr2=0x7f2c5c0fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:40.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.677+0000 7f2c62100640 1 -- 172.21.15.67:0/4109303560 shutdown_connections 2024-04-23T00:05:40.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.677+0000 7f2c62100640 1 -- 172.21.15.67:0/4109303560 wait complete. 2024-04-23T00:05:40.675 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-23T00:05:40.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.793+0000 7f55a4ed5640 1 -- 172.21.15.67:0/3871758020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55a00ffb50 msgr2=0x7f55a0103b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:40.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.793+0000 7f55a4ed5640 1 --2- 172.21.15.67:0/3871758020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55a00ffb50 0x7f55a0103b90 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f5588009880 tx=0x7f558802bce0 comp rx=0 tx=0).stop 2024-04-23T00:05:40.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.793+0000 7f55a4ed5640 1 -- 172.21.15.67:0/3871758020 shutdown_connections 2024-04-23T00:05:40.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.793+0000 7f55a4ed5640 1 --2- 172.21.15.67:0/3871758020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55a00ffb50 0x7f55a0103b90 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:40.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.793+0000 7f55a4ed5640 1 --2- 172.21.15.67:0/3871758020 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f55a00ff1a0 0x7f55a00ff580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:40.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.793+0000 7f55a4ed5640 1 -- 172.21.15.67:0/3871758020 >> 172.21.15.67:0/3871758020 conn(0x7f55a00faa20 msgr2=0x7f55a00fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:40.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.793+0000 7f55a4ed5640 1 -- 172.21.15.67:0/3871758020 shutdown_connections 2024-04-23T00:05:40.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.793+0000 7f55a4ed5640 1 -- 172.21.15.67:0/3871758020 wait complete. 2024-04-23T00:05:40.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.794+0000 7f55a4ed5640 1 Processor -- start 2024-04-23T00:05:40.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.794+0000 7f55a4ed5640 1 -- start start 2024-04-23T00:05:40.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.794+0000 7f55a4ed5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55a00ff1a0 0x7f55a0196ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:40.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.794+0000 7f55a4ed5640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f55a0197220 0x7f55a0197680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:40.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.794+0000 7f55a4ed5640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a019fd50 con 0x7f55a00ff1a0 2024-04-23T00:05:40.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.794+0000 7f55a4ed5640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a019fe90 con 0x7f55a0197220 2024-04-23T00:05:40.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.794+0000 7f559e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55a00ff1a0 0x7f55a0196ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:40.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.794+0000 7f559e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55a00ff1a0 0x7f55a0196ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38778/0 (socket says 172.21.15.67:38778) 2024-04-23T00:05:40.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.794+0000 7f559e59c640 1 -- 172.21.15.67:0/254393770 learned_addr learned my addr 172.21.15.67:0/254393770 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:40.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.794+0000 7f559e59c640 1 -- 172.21.15.67:0/254393770 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f55a0197220 msgr2=0x7f55a0197680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:40.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.794+0000 7f559e59c640 1 --2- 172.21.15.67:0/254393770 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f55a0197220 0x7f55a0197680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:40.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.794+0000 7f559e59c640 1 -- 172.21.15.67:0/254393770 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5588009560 con 0x7f55a00ff1a0 2024-04-23T00:05:40.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.794+0000 7f559e59c640 1 --2- 172.21.15.67:0/254393770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55a00ff1a0 0x7f55a0196ce0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f559000b450 tx=0x7f559000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:40.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.795+0000 7f55977fe640 1 -- 172.21.15.67:0/254393770 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5590011860 con 0x7f55a00ff1a0 2024-04-23T00:05:40.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.795+0000 7f55977fe640 1 -- 172.21.15.67:0/254393770 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5590011e80 con 0x7f55a00ff1a0 2024-04-23T00:05:40.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.795+0000 7f55a4ed5640 1 -- 172.21.15.67:0/254393770 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55a01a00f0 con 0x7f55a00ff1a0 2024-04-23T00:05:40.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.795+0000 7f55977fe640 1 -- 172.21.15.67:0/254393770 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f559000f570 con 0x7f55a00ff1a0 2024-04-23T00:05:40.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.795+0000 7f55a4ed5640 1 -- 172.21.15.67:0/254393770 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55a01a0600 con 0x7f55a00ff1a0 2024-04-23T00:05:40.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.796+0000 7f55a4ed5640 1 -- 172.21.15.67:0/254393770 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55a004e040 con 0x7f55a00ff1a0 2024-04-23T00:05:40.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.796+0000 7f55977fe640 1 -- 172.21.15.67:0/254393770 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f55900103e0 con 0x7f55a00ff1a0 2024-04-23T00:05:40.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.797+0000 7f55977fe640 1 --2- 172.21.15.67:0/254393770 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f558c06ec30 0x7f558c0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:40.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.797+0000 7f55977fe640 1 -- 172.21.15.67:0/254393770 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f559008eec0 con 0x7f55a00ff1a0 2024-04-23T00:05:40.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.797+0000 7f5597fff640 1 --2- 172.21.15.67:0/254393770 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f558c06ec30 0x7f558c0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:40.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.797+0000 7f5597fff640 1 --2- 172.21.15.67:0/254393770 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f558c06ec30 0x7f558c0710f0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f5588036040 tx=0x7f5588000bc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:40.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.801+0000 7f55977fe640 1 -- 172.21.15.67:0/254393770 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f5590057b00 con 0x7f55a00ff1a0 2024-04-23T00:05:40.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.988+0000 7f55a4ed5640 1 -- 172.21.15.67:0/254393770 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f55a01074a0 con 0x7f55a00ff1a0 2024-04-23T00:05:40.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.989+0000 7f55977fe640 1 -- 172.21.15.67:0/254393770 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v94) v1 ==== 107+0+11 (secure 0 0 0) 0x7f559005b1c0 con 0x7f55a00ff1a0 2024-04-23T00:05:40.975 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-23T00:05:40.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.990+0000 7f55a4ed5640 1 -- 172.21.15.67:0/254393770 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f558c06ec30 msgr2=0x7f558c0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:40.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.990+0000 7f55a4ed5640 1 --2- 172.21.15.67:0/254393770 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f558c06ec30 0x7f558c0710f0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f5588036040 tx=0x7f5588000bc0 comp rx=0 tx=0).stop 2024-04-23T00:05:40.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.990+0000 7f55a4ed5640 1 -- 172.21.15.67:0/254393770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55a00ff1a0 msgr2=0x7f55a0196ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:40.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.990+0000 7f55a4ed5640 1 --2- 172.21.15.67:0/254393770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55a00ff1a0 0x7f55a0196ce0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f559000b450 tx=0x7f559000b910 comp rx=0 tx=0).stop 2024-04-23T00:05:40.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.990+0000 7f55a4ed5640 1 -- 172.21.15.67:0/254393770 shutdown_connections 2024-04-23T00:05:40.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.990+0000 7f55a4ed5640 1 --2- 172.21.15.67:0/254393770 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f558c06ec30 0x7f558c0710f0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:40.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.990+0000 7f55a4ed5640 1 --2- 172.21.15.67:0/254393770 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f55a0197220 0x7f55a0197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:40.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.991+0000 7f55a4ed5640 1 --2- 172.21.15.67:0/254393770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55a00ff1a0 0x7f55a0196ce0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:40.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.991+0000 7f55a4ed5640 1 -- 172.21.15.67:0/254393770 >> 172.21.15.67:0/254393770 conn(0x7f55a00faa20 msgr2=0x7f55a00fc220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:40.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.991+0000 7f55a4ed5640 1 -- 172.21.15.67:0/254393770 shutdown_connections 2024-04-23T00:05:40.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:40.991+0000 7f55a4ed5640 1 -- 172.21.15.67:0/254393770 wait complete. 2024-04-23T00:05:40.987 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:40 smithi067 ceph-mon[112438]: pgmap v1183: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:05:40.987 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:40 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3763592098' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T00:05:40.988 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:40 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/4109303560' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T00:05:40.989 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-23T00:05:41.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.107+0000 7f1bc6469640 1 -- 172.21.15.67:0/1060140953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bc00ffb50 msgr2=0x7f1bc0103b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:41.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.107+0000 7f1bc6469640 1 --2- 172.21.15.67:0/1060140953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bc00ffb50 0x7f1bc0103b90 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f1bb4009880 tx=0x7f1bb402bce0 comp rx=0 tx=0).stop 2024-04-23T00:05:41.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.107+0000 7f1bc6469640 1 -- 172.21.15.67:0/1060140953 shutdown_connections 2024-04-23T00:05:41.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.107+0000 7f1bc6469640 1 --2- 172.21.15.67:0/1060140953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bc00ffb50 0x7f1bc0103b90 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:41.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.107+0000 7f1bc6469640 1 --2- 172.21.15.67:0/1060140953 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1bc00ff1a0 0x7f1bc00ff580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:41.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.107+0000 7f1bc6469640 1 -- 172.21.15.67:0/1060140953 >> 172.21.15.67:0/1060140953 conn(0x7f1bc00faa20 msgr2=0x7f1bc00fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:41.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.107+0000 7f1bc6469640 1 -- 172.21.15.67:0/1060140953 shutdown_connections 2024-04-23T00:05:41.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.107+0000 7f1bc6469640 1 -- 172.21.15.67:0/1060140953 wait complete. 2024-04-23T00:05:41.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.108+0000 7f1bc6469640 1 Processor -- start 2024-04-23T00:05:41.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.108+0000 7f1bc6469640 1 -- start start 2024-04-23T00:05:41.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.108+0000 7f1bc6469640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1bc00ff1a0 0x7f1bc0195df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:41.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.108+0000 7f1bc6469640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bc0196330 0x7f1bc019a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:41.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.108+0000 7f1bc6469640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc0196930 con 0x7f1bc0196330 2024-04-23T00:05:41.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.108+0000 7f1bc6469640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc0196aa0 con 0x7f1bc00ff1a0 2024-04-23T00:05:41.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.108+0000 7f1bbf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bc0196330 0x7f1bc019a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:41.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.108+0000 7f1bbf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bc0196330 0x7f1bc019a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38804/0 (socket says 172.21.15.67:38804) 2024-04-23T00:05:41.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.108+0000 7f1bbf7fe640 1 -- 172.21.15.67:0/3357089630 learned_addr learned my addr 172.21.15.67:0/3357089630 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:41.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.109+0000 7f1bbffff640 1 --2- 172.21.15.67:0/3357089630 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1bc00ff1a0 0x7f1bc0195df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:41.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.109+0000 7f1bbf7fe640 1 -- 172.21.15.67:0/3357089630 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1bc00ff1a0 msgr2=0x7f1bc0195df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:41.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.109+0000 7f1bbf7fe640 1 --2- 172.21.15.67:0/3357089630 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1bc00ff1a0 0x7f1bc0195df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:41.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.109+0000 7f1bbf7fe640 1 -- 172.21.15.67:0/3357089630 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb4009560 con 0x7f1bc0196330 2024-04-23T00:05:41.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.109+0000 7f1bbf7fe640 1 --2- 172.21.15.67:0/3357089630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bc0196330 0x7f1bc019a790 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f1bb4036040 tx=0x7f1bb402c750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:41.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.109+0000 7f1bbd7fa640 1 -- 172.21.15.67:0/3357089630 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1bb40035f0 con 0x7f1bc0196330 2024-04-23T00:05:41.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.109+0000 7f1bc6469640 1 -- 172.21.15.67:0/3357089630 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bc019ad30 con 0x7f1bc0196330 2024-04-23T00:05:41.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.109+0000 7f1bbd7fa640 1 -- 172.21.15.67:0/3357089630 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1bb4003c10 con 0x7f1bc0196330 2024-04-23T00:05:41.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.109+0000 7f1bbd7fa640 1 -- 172.21.15.67:0/3357089630 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1bb4005630 con 0x7f1bc0196330 2024-04-23T00:05:41.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.109+0000 7f1bc6469640 1 -- 172.21.15.67:0/3357089630 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bc019b290 con 0x7f1bc0196330 2024-04-23T00:05:41.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.110+0000 7f1bc6469640 1 -- 172.21.15.67:0/3357089630 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bc004e040 con 0x7f1bc0196330 2024-04-23T00:05:41.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.111+0000 7f1bbd7fa640 1 -- 172.21.15.67:0/3357089630 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1bb4003750 con 0x7f1bc0196330 2024-04-23T00:05:41.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.111+0000 7f1bbd7fa640 1 --2- 172.21.15.67:0/3357089630 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1ba806ec30 0x7f1ba80710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:41.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.111+0000 7f1bbd7fa640 1 -- 172.21.15.67:0/3357089630 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f1bb40b1310 con 0x7f1bc0196330 2024-04-23T00:05:41.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.111+0000 7f1bbffff640 1 --2- 172.21.15.67:0/3357089630 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1ba806ec30 0x7f1ba80710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:41.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.111+0000 7f1bbffff640 1 --2- 172.21.15.67:0/3357089630 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1ba806ec30 0x7f1ba80710f0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f1bb0009bd0 tx=0x7f1bb0005f90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:41.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.116+0000 7f1bbd7fa640 1 -- 172.21.15.67:0/3357089630 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1bb407a000 con 0x7f1bc0196330 2024-04-23T00:05:41.230 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:40 smithi134 ceph-mon[85630]: pgmap v1183: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:05:41.230 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:40 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3763592098' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T00:05:41.230 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:40 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/4109303560' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T00:05:41.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.305+0000 7f1bc6469640 1 -- 172.21.15.67:0/3357089630 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f1bc00ffb50 con 0x7f1bc0196330 2024-04-23T00:05:41.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.305+0000 7f1bbd7fa640 1 -- 172.21.15.67:0/3357089630 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v94) v1 ==== 107+0+11 (secure 0 0 0) 0x7f1bb407d6c0 con 0x7f1bc0196330 2024-04-23T00:05:41.292 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-23T00:05:41.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.308+0000 7f1bc6469640 1 -- 172.21.15.67:0/3357089630 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1ba806ec30 msgr2=0x7f1ba80710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:41.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.308+0000 7f1bc6469640 1 --2- 172.21.15.67:0/3357089630 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1ba806ec30 0x7f1ba80710f0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f1bb0009bd0 tx=0x7f1bb0005f90 comp rx=0 tx=0).stop 2024-04-23T00:05:41.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.308+0000 7f1bc6469640 1 -- 172.21.15.67:0/3357089630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bc0196330 msgr2=0x7f1bc019a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:41.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.308+0000 7f1bc6469640 1 --2- 172.21.15.67:0/3357089630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bc0196330 0x7f1bc019a790 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f1bb4036040 tx=0x7f1bb402c750 comp rx=0 tx=0).stop 2024-04-23T00:05:41.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.308+0000 7f1bc6469640 1 -- 172.21.15.67:0/3357089630 shutdown_connections 2024-04-23T00:05:41.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.308+0000 7f1bc6469640 1 --2- 172.21.15.67:0/3357089630 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1ba806ec30 0x7f1ba80710f0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:41.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.308+0000 7f1bc6469640 1 --2- 172.21.15.67:0/3357089630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bc0196330 0x7f1bc019a790 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:41.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.308+0000 7f1bc6469640 1 --2- 172.21.15.67:0/3357089630 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1bc00ff1a0 0x7f1bc0195df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:41.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.308+0000 7f1bc6469640 1 -- 172.21.15.67:0/3357089630 >> 172.21.15.67:0/3357089630 conn(0x7f1bc00faa20 msgr2=0x7f1bc01033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:41.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.308+0000 7f1bc6469640 1 -- 172.21.15.67:0/3357089630 shutdown_connections 2024-04-23T00:05:41.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.308+0000 7f1bc6469640 1 -- 172.21.15.67:0/3357089630 wait complete. 2024-04-23T00:05:41.306 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T00:05:41.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.420+0000 7f013d955640 1 -- 172.21.15.67:0/1585003379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0138101260 msgr2=0x7f0138101640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:41.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.420+0000 7f013d955640 1 --2- 172.21.15.67:0/1585003379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0138101260 0x7f0138101640 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f0128009880 tx=0x7f012802bce0 comp rx=0 tx=0).stop 2024-04-23T00:05:41.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.420+0000 7f013d955640 1 -- 172.21.15.67:0/1585003379 shutdown_connections 2024-04-23T00:05:41.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.420+0000 7f013d955640 1 --2- 172.21.15.67:0/1585003379 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0138101b80 0x7f013810a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:41.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.420+0000 7f013d955640 1 --2- 172.21.15.67:0/1585003379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0138101260 0x7f0138101640 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:41.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.420+0000 7f013d955640 1 -- 172.21.15.67:0/1585003379 >> 172.21.15.67:0/1585003379 conn(0x7f01380faa40 msgr2=0x7f01380fce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:41.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.420+0000 7f013d955640 1 -- 172.21.15.67:0/1585003379 shutdown_connections 2024-04-23T00:05:41.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.420+0000 7f013d955640 1 -- 172.21.15.67:0/1585003379 wait complete. 2024-04-23T00:05:41.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.421+0000 7f013d955640 1 Processor -- start 2024-04-23T00:05:41.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.421+0000 7f013d955640 1 -- start start 2024-04-23T00:05:41.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.421+0000 7f013d955640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0138101260 0x7f013819ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:41.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.421+0000 7f013d955640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0138101b80 0x7f013819bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:41.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.421+0000 7f013d955640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0138195a90 con 0x7f0138101260 2024-04-23T00:05:41.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.421+0000 7f013d955640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0138195c00 con 0x7f0138101b80 2024-04-23T00:05:41.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.421+0000 7f0136ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0138101260 0x7f013819ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:41.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.421+0000 7f0136ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0138101260 0x7f013819ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38812/0 (socket says 172.21.15.67:38812) 2024-04-23T00:05:41.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.421+0000 7f0136ffd640 1 -- 172.21.15.67:0/2392095016 learned_addr learned my addr 172.21.15.67:0/2392095016 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:41.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.422+0000 7f01367fc640 1 --2- 172.21.15.67:0/2392095016 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0138101b80 0x7f013819bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:41.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.422+0000 7f0136ffd640 1 -- 172.21.15.67:0/2392095016 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0138101b80 msgr2=0x7f013819bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:41.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.422+0000 7f0136ffd640 1 --2- 172.21.15.67:0/2392095016 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0138101b80 0x7f013819bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:41.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.422+0000 7f0136ffd640 1 -- 172.21.15.67:0/2392095016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0128009560 con 0x7f0138101260 2024-04-23T00:05:41.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.422+0000 7f0136ffd640 1 --2- 172.21.15.67:0/2392095016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0138101260 0x7f013819ba20 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f0128009230 tx=0x7f0128004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:41.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.422+0000 7f013c953640 1 -- 172.21.15.67:0/2392095016 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01280035f0 con 0x7f0138101260 2024-04-23T00:05:41.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.422+0000 7f013c953640 1 -- 172.21.15.67:0/2392095016 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0128003c10 con 0x7f0138101260 2024-04-23T00:05:41.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.422+0000 7f013d955640 1 -- 172.21.15.67:0/2392095016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0138195e80 con 0x7f0138101260 2024-04-23T00:05:41.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.422+0000 7f013c953640 1 -- 172.21.15.67:0/2392095016 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0128005670 con 0x7f0138101260 2024-04-23T00:05:41.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.422+0000 7f013d955640 1 -- 172.21.15.67:0/2392095016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0138196380 con 0x7f0138101260 2024-04-23T00:05:41.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.423+0000 7f013d955640 1 -- 172.21.15.67:0/2392095016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f013804e040 con 0x7f0138101260 2024-04-23T00:05:41.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.424+0000 7f013c953640 1 -- 172.21.15.67:0/2392095016 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0128004dd0 con 0x7f0138101260 2024-04-23T00:05:41.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.424+0000 7f013c953640 1 --2- 172.21.15.67:0/2392095016 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f012006ec90 0x7f0120071150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:41.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.424+0000 7f013c953640 1 -- 172.21.15.67:0/2392095016 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f01280b13e0 con 0x7f0138101260 2024-04-23T00:05:41.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.424+0000 7f01367fc640 1 --2- 172.21.15.67:0/2392095016 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f012006ec90 0x7f0120071150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:41.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.425+0000 7f01367fc640 1 --2- 172.21.15.67:0/2392095016 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f012006ec90 0x7f0120071150 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f012c005fd0 tx=0x7f012c005850 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:41.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.429+0000 7f013c953640 1 -- 172.21.15.67:0/2392095016 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f012807a020 con 0x7f0138101260 2024-04-23T00:05:41.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.657+0000 7f013d955640 1 -- 172.21.15.67:0/2392095016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01381076d0 con 0x7f0138101260 2024-04-23T00:05:41.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.657+0000 7f013c953640 1 -- 172.21.15.67:0/2392095016 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+5248 (secure 0 0 0) 0x7f012807d6e0 con 0x7f0138101260 2024-04-23T00:05:41.644 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:05:41.644 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":42,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T23:20:20.823261+0000","modified":"2024-04-23T00:03:56.266657+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34396,"mds_1":34424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34396":{"gid":34396,"name":"cephfs.smithi067.aqlxpv","rank":0,"incarnation":35,"state":"up:active","state_seq":469,"addr":"172.21.15.67:6827/789475804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":789475804},{"type":"v1","addr":"172.21.15.67:6827","nonce":789475804}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34424":{"gid":34424,"name":"cephfs.smithi067.xyazmh","rank":1,"incarnation":35,"state":"up:active","state_seq":464,"addr":"172.21.15.67:6829/927062647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":927062647},{"type":"v1","addr":"172.21.15.67:6829","nonce":927062647}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_44327":{"gid":44327,"name":"cephfs.smithi134.owipad","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.134:6825/1473208408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":1473208408},{"type":"v1","addr":"172.21.15.134:6825","nonce":1473208408}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_44851":{"gid":44851,"name":"cephfs.smithi134.vuxbsj","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.134:6827/1849648622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":1849648622},{"type":"v1","addr":"172.21.15.134:6827","nonce":1849648622}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":34396,"qdb_cluster":[34396,34424]},"id":1}]} 2024-04-23T00:05:41.644 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 42 2024-04-23T00:05:41.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.660+0000 7f013d955640 1 -- 172.21.15.67:0/2392095016 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f012006ec90 msgr2=0x7f0120071150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:41.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.660+0000 7f013d955640 1 --2- 172.21.15.67:0/2392095016 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f012006ec90 0x7f0120071150 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f012c005fd0 tx=0x7f012c005850 comp rx=0 tx=0).stop 2024-04-23T00:05:41.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.660+0000 7f013d955640 1 -- 172.21.15.67:0/2392095016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0138101260 msgr2=0x7f013819ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:41.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.660+0000 7f013d955640 1 --2- 172.21.15.67:0/2392095016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0138101260 0x7f013819ba20 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f0128009230 tx=0x7f0128004040 comp rx=0 tx=0).stop 2024-04-23T00:05:41.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.660+0000 7f013d955640 1 -- 172.21.15.67:0/2392095016 shutdown_connections 2024-04-23T00:05:41.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.660+0000 7f013d955640 1 --2- 172.21.15.67:0/2392095016 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f012006ec90 0x7f0120071150 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:41.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.660+0000 7f013d955640 1 --2- 172.21.15.67:0/2392095016 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0138101b80 0x7f013819bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:41.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.660+0000 7f013d955640 1 --2- 172.21.15.67:0/2392095016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0138101260 0x7f013819ba20 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:41.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.660+0000 7f013d955640 1 -- 172.21.15.67:0/2392095016 >> 172.21.15.67:0/2392095016 conn(0x7f01380faa40 msgr2=0x7f01381048a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:41.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.660+0000 7f013d955640 1 -- 172.21.15.67:0/2392095016 shutdown_connections 2024-04-23T00:05:41.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.660+0000 7f013d955640 1 -- 172.21.15.67:0/2392095016 wait complete. 2024-04-23T00:05:41.658 DEBUG:tasks.fs:checking fs fscid=1,name=cephfs state = {'epoch': 18, 'max_mds': 2, 'flags': 50} 2024-04-23T00:05:41.659 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 19 2024-04-23T00:05:41.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.775+0000 7f0cfea46640 1 -- 172.21.15.67:0/3411024372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cf80ff260 msgr2=0x7f0cf80ff640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:41.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.775+0000 7f0cfea46640 1 --2- 172.21.15.67:0/3411024372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cf80ff260 0x7f0cf80ff640 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8009880 tx=0x7f0ce802bce0 comp rx=0 tx=0).stop 2024-04-23T00:05:41.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.775+0000 7f0cfea46640 1 -- 172.21.15.67:0/3411024372 shutdown_connections 2024-04-23T00:05:41.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.775+0000 7f0cfea46640 1 --2- 172.21.15.67:0/3411024372 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0cf80ffc10 0x7f0cf8103b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:41.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.775+0000 7f0cfea46640 1 --2- 172.21.15.67:0/3411024372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cf80ff260 0x7f0cf80ff640 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:41.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.775+0000 7f0cfea46640 1 -- 172.21.15.67:0/3411024372 >> 172.21.15.67:0/3411024372 conn(0x7f0cf8074a50 msgr2=0x7f0cf8074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:41.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.775+0000 7f0cfea46640 1 -- 172.21.15.67:0/3411024372 shutdown_connections 2024-04-23T00:05:41.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.775+0000 7f0cfea46640 1 -- 172.21.15.67:0/3411024372 wait complete. 2024-04-23T00:05:41.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.775+0000 7f0cfea46640 1 Processor -- start 2024-04-23T00:05:41.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.775+0000 7f0cfea46640 1 -- start start 2024-04-23T00:05:41.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.776+0000 7f0cfea46640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cf80ff260 0x7f0cf8197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:41.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.776+0000 7f0cfea46640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0cf81984e0 0x7f0cf819c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:41.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.776+0000 7f0cfea46640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf8198ae0 con 0x7f0cf80ff260 2024-04-23T00:05:41.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.776+0000 7f0cfea46640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf8198c50 con 0x7f0cf81984e0 2024-04-23T00:05:41.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.776+0000 7f0cf7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cf80ff260 0x7f0cf8197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:41.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.776+0000 7f0cf7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cf80ff260 0x7f0cf8197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38834/0 (socket says 172.21.15.67:38834) 2024-04-23T00:05:41.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.776+0000 7f0cf7fff640 1 -- 172.21.15.67:0/137691625 learned_addr learned my addr 172.21.15.67:0/137691625 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:41.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.776+0000 7f0cf77fe640 1 --2- 172.21.15.67:0/137691625 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0cf81984e0 0x7f0cf819c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:41.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.776+0000 7f0cf7fff640 1 -- 172.21.15.67:0/137691625 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0cf81984e0 msgr2=0x7f0cf819c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:41.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.776+0000 7f0cf7fff640 1 --2- 172.21.15.67:0/137691625 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0cf81984e0 0x7f0cf819c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:41.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.776+0000 7f0cf7fff640 1 -- 172.21.15.67:0/137691625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce8009560 con 0x7f0cf80ff260 2024-04-23T00:05:41.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.776+0000 7f0cf7fff640 1 --2- 172.21.15.67:0/137691625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cf80ff260 0x7f0cf8197fa0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8009430 tx=0x7f0ce8004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:41.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.776+0000 7f0cf57fa640 1 -- 172.21.15.67:0/137691625 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ce80035f0 con 0x7f0cf80ff260 2024-04-23T00:05:41.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.776+0000 7f0cfea46640 1 -- 172.21.15.67:0/137691625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf819cee0 con 0x7f0cf80ff260 2024-04-23T00:05:41.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.776+0000 7f0cf57fa640 1 -- 172.21.15.67:0/137691625 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0ce8003c10 con 0x7f0cf80ff260 2024-04-23T00:05:41.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.776+0000 7f0cf57fa640 1 -- 172.21.15.67:0/137691625 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ce8005670 con 0x7f0cf80ff260 2024-04-23T00:05:41.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.776+0000 7f0cfea46640 1 -- 172.21.15.67:0/137691625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf819d440 con 0x7f0cf80ff260 2024-04-23T00:05:41.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.778+0000 7f0cfea46640 1 -- 172.21.15.67:0/137691625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cf804e040 con 0x7f0cf80ff260 2024-04-23T00:05:41.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.778+0000 7f0cf57fa640 1 -- 172.21.15.67:0/137691625 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0ce8004dd0 con 0x7f0cf80ff260 2024-04-23T00:05:41.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.778+0000 7f0cf57fa640 1 --2- 172.21.15.67:0/137691625 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0ce006ebe0 0x7f0ce00710a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:41.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.779+0000 7f0cf57fa640 1 -- 172.21.15.67:0/137691625 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f0ce80b13f0 con 0x7f0cf80ff260 2024-04-23T00:05:41.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.779+0000 7f0cf77fe640 1 --2- 172.21.15.67:0/137691625 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0ce006ebe0 0x7f0ce00710a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:41.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.779+0000 7f0cf77fe640 1 --2- 172.21.15.67:0/137691625 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0ce006ebe0 0x7f0ce00710a0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f0cec005fd0 tx=0x7f0cec00b590 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:41.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:41.783+0000 7f0cf57fa640 1 -- 172.21.15.67:0/137691625 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f0ce807a030 con 0x7f0cf80ff260 2024-04-23T00:05:41.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.012+0000 7f0cfea46640 1 -- 172.21.15.67:0/137691625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 19, "format": "json"} v 0) v1 -- 0x7f0cf80ffc10 con 0x7f0cf80ff260 2024-04-23T00:05:41.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.012+0000 7f0cf57fa640 1 -- 172.21.15.67:0/137691625 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 19, "format": "json"}]=0 dumped fsmap epoch 19 v42) v1 ==== 107+0+4892 (secure 0 0 0) 0x7f0ce807d6f0 con 0x7f0cf80ff260 2024-04-23T00:05:41.999 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:05:41.999 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T23:20:20.823261+0000","modified":"2024-04-22T23:22:43.535520+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14484,"mds_1":24273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14484":{"gid":14484,"name":"cephfs.smithi067.xyazmh","rank":0,"incarnation":13,"state":"up:active","state_seq":6,"addr":"172.21.15.67:6829/3606151860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3606151860},{"type":"v1","addr":"172.21.15.67:6829","nonce":3606151860}]},"join_fscid":1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14496":{"gid":14496,"name":"cephfs.smithi067.aqlxpv","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6827/3305304988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3305304988},{"type":"v1","addr":"172.21.15.67:6827","nonce":3305304988}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24273":{"gid":24273,"name":"cephfs.smithi134.owipad","rank":1,"incarnation":6,"state":"up:active","state_seq":4,"addr":"172.21.15.134:6825/535957142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":535957142},{"type":"v1","addr":"172.21.15.134:6825","nonce":535957142}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24289":{"gid":24289,"name":"cephfs.smithi134.vuxbsj","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.134:6827/1954821676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":1954821676},{"type":"v1","addr":"172.21.15.134:6827","nonce":1954821676}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T00:05:41.999 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 19 2024-04-23T00:05:42.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.014+0000 7f0cfea46640 1 -- 172.21.15.67:0/137691625 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0ce006ebe0 msgr2=0x7f0ce00710a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:42.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.014+0000 7f0cfea46640 1 --2- 172.21.15.67:0/137691625 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0ce006ebe0 0x7f0ce00710a0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f0cec005fd0 tx=0x7f0cec00b590 comp rx=0 tx=0).stop 2024-04-23T00:05:42.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.014+0000 7f0cfea46640 1 -- 172.21.15.67:0/137691625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cf80ff260 msgr2=0x7f0cf8197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:42.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.014+0000 7f0cfea46640 1 --2- 172.21.15.67:0/137691625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cf80ff260 0x7f0cf8197fa0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8009430 tx=0x7f0ce8004040 comp rx=0 tx=0).stop 2024-04-23T00:05:42.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.014+0000 7f0cfea46640 1 -- 172.21.15.67:0/137691625 shutdown_connections 2024-04-23T00:05:42.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.014+0000 7f0cfea46640 1 --2- 172.21.15.67:0/137691625 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f0ce006ebe0 0x7f0ce00710a0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:42.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.014+0000 7f0cfea46640 1 --2- 172.21.15.67:0/137691625 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0cf81984e0 0x7f0cf819c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:42.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.014+0000 7f0cfea46640 1 --2- 172.21.15.67:0/137691625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cf80ff260 0x7f0cf8197fa0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:42.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.014+0000 7f0cfea46640 1 -- 172.21.15.67:0/137691625 >> 172.21.15.67:0/137691625 conn(0x7f0cf8074a50 msgr2=0x7f0cf8067be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:42.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.014+0000 7f0cfea46640 1 -- 172.21.15.67:0/137691625 shutdown_connections 2024-04-23T00:05:42.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.014+0000 7f0cfea46640 1 -- 172.21.15.67:0/137691625 wait complete. 2024-04-23T00:05:42.011 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:41 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/254393770' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-23T00:05:42.011 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:05:42.011 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:41 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3357089630' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-23T00:05:42.011 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:41 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2392095016' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T00:05:42.012 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 20 2024-04-23T00:05:42.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.127+0000 7f79a616e640 1 -- 172.21.15.67:0/2302017752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79a0100c50 msgr2=0x7f79a0104c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:42.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.127+0000 7f79a616e640 1 --2- 172.21.15.67:0/2302017752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79a0100c50 0x7f79a0104c90 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f7994009880 tx=0x7f799402bce0 comp rx=0 tx=0).stop 2024-04-23T00:05:42.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.127+0000 7f79a616e640 1 -- 172.21.15.67:0/2302017752 shutdown_connections 2024-04-23T00:05:42.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.127+0000 7f79a616e640 1 --2- 172.21.15.67:0/2302017752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79a0100c50 0x7f79a0104c90 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:42.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.127+0000 7f79a616e640 1 --2- 172.21.15.67:0/2302017752 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79a01002a0 0x7f79a0100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:42.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.127+0000 7f79a616e640 1 -- 172.21.15.67:0/2302017752 >> 172.21.15.67:0/2302017752 conn(0x7f79a00fbb20 msgr2=0x7f79a00fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:42.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.127+0000 7f79a616e640 1 -- 172.21.15.67:0/2302017752 shutdown_connections 2024-04-23T00:05:42.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.127+0000 7f79a616e640 1 -- 172.21.15.67:0/2302017752 wait complete. 2024-04-23T00:05:42.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.128+0000 7f79a616e640 1 Processor -- start 2024-04-23T00:05:42.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.128+0000 7f79a616e640 1 -- start start 2024-04-23T00:05:42.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.128+0000 7f79a616e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79a01002a0 0x7f79a0195d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:42.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.128+0000 7f79a616e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79a01962a0 0x7f79a019a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:42.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.128+0000 7f79a616e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79a01968a0 con 0x7f79a01002a0 2024-04-23T00:05:42.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.128+0000 7f79a616e640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79a0196a10 con 0x7f79a01962a0 2024-04-23T00:05:42.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.128+0000 7f799f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79a01002a0 0x7f79a0195d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:42.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.128+0000 7f799f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79a01002a0 0x7f79a0195d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38852/0 (socket says 172.21.15.67:38852) 2024-04-23T00:05:42.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.128+0000 7f799f7fe640 1 -- 172.21.15.67:0/3575590689 learned_addr learned my addr 172.21.15.67:0/3575590689 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:42.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.129+0000 7f799effd640 1 --2- 172.21.15.67:0/3575590689 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79a01962a0 0x7f79a019a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:42.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.129+0000 7f799f7fe640 1 -- 172.21.15.67:0/3575590689 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79a01962a0 msgr2=0x7f79a019a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:42.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.129+0000 7f799f7fe640 1 --2- 172.21.15.67:0/3575590689 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79a01962a0 0x7f79a019a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:42.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.129+0000 7f799f7fe640 1 -- 172.21.15.67:0/3575590689 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7994009560 con 0x7f79a01002a0 2024-04-23T00:05:42.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.129+0000 7f799f7fe640 1 --2- 172.21.15.67:0/3575590689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79a01002a0 0x7f79a0195d60 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f799000c840 tx=0x7f799000cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:42.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.129+0000 7f799cff9640 1 -- 172.21.15.67:0/3575590689 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79900041f0 con 0x7f79a01002a0 2024-04-23T00:05:42.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.129+0000 7f799cff9640 1 -- 172.21.15.67:0/3575590689 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7990007e70 con 0x7f79a01002a0 2024-04-23T00:05:42.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.129+0000 7f79a616e640 1 -- 172.21.15.67:0/3575590689 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79a019ad00 con 0x7f79a01002a0 2024-04-23T00:05:42.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.129+0000 7f799cff9640 1 -- 172.21.15.67:0/3575590689 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79900049d0 con 0x7f79a01002a0 2024-04-23T00:05:42.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.129+0000 7f79a616e640 1 -- 172.21.15.67:0/3575590689 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79a019b260 con 0x7f79a01002a0 2024-04-23T00:05:42.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.130+0000 7f79a616e640 1 -- 172.21.15.67:0/3575590689 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79a019b540 con 0x7f79a01002a0 2024-04-23T00:05:42.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.131+0000 7f799cff9640 1 -- 172.21.15.67:0/3575590689 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f79900079b0 con 0x7f79a01002a0 2024-04-23T00:05:42.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.131+0000 7f799cff9640 1 --2- 172.21.15.67:0/3575590689 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f798806ec30 0x7f79880710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:42.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.131+0000 7f799effd640 1 --2- 172.21.15.67:0/3575590689 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f798806ec30 0x7f79880710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:42.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.131+0000 7f799cff9640 1 -- 172.21.15.67:0/3575590689 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f799008ee90 con 0x7f79a01002a0 2024-04-23T00:05:42.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.132+0000 7f799effd640 1 --2- 172.21.15.67:0/3575590689 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f798806ec30 0x7f79880710f0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f7994006010 tx=0x7f7994000bc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:42.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.136+0000 7f799cff9640 1 -- 172.21.15.67:0/3575590689 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7990057ad0 con 0x7f79a01002a0 2024-04-23T00:05:42.231 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:41 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/254393770' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-23T00:05:42.231 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:05:42.232 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:41 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3357089630' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-23T00:05:42.232 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:41 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2392095016' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T00:05:42.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.363+0000 7f79a616e640 1 -- 172.21.15.67:0/3575590689 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 20, "format": "json"} v 0) v1 -- 0x7f79a01970b0 con 0x7f79a01002a0 2024-04-23T00:05:42.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.363+0000 7f799cff9640 1 -- 172.21.15.67:0/3575590689 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 20, "format": "json"}]=0 dumped fsmap epoch 20 v42) v1 ==== 107+0+4893 (secure 0 0 0) 0x7f799005b190 con 0x7f79a01002a0 2024-04-23T00:05:42.350 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:05:42.350 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T23:20:20.823261+0000","modified":"2024-04-22T23:23:54.961755+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14484,"mds_1":24273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14484":{"gid":14484,"name":"cephfs.smithi067.xyazmh","rank":0,"incarnation":13,"state":"up:active","state_seq":6,"addr":"172.21.15.67:6829/3606151860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3606151860},{"type":"v1","addr":"172.21.15.67:6829","nonce":3606151860}]},"join_fscid":1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14496":{"gid":14496,"name":"cephfs.smithi067.aqlxpv","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6827/3305304988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3305304988},{"type":"v1","addr":"172.21.15.67:6827","nonce":3305304988}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24273":{"gid":24273,"name":"cephfs.smithi134.owipad","rank":1,"incarnation":6,"state":"up:active","state_seq":4,"addr":"172.21.15.134:6825/535957142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":535957142},{"type":"v1","addr":"172.21.15.134:6825","nonce":535957142}]},"join_fscid":1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24289":{"gid":24289,"name":"cephfs.smithi134.vuxbsj","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.134:6827/1954821676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":1954821676},{"type":"v1","addr":"172.21.15.134:6827","nonce":1954821676}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T00:05:42.350 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 20 2024-04-23T00:05:42.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.365+0000 7f79a616e640 1 -- 172.21.15.67:0/3575590689 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f798806ec30 msgr2=0x7f79880710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:42.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.365+0000 7f79a616e640 1 --2- 172.21.15.67:0/3575590689 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f798806ec30 0x7f79880710f0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f7994006010 tx=0x7f7994000bc0 comp rx=0 tx=0).stop 2024-04-23T00:05:42.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.365+0000 7f79a616e640 1 -- 172.21.15.67:0/3575590689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79a01002a0 msgr2=0x7f79a0195d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:42.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.365+0000 7f79a616e640 1 --2- 172.21.15.67:0/3575590689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79a01002a0 0x7f79a0195d60 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f799000c840 tx=0x7f799000cd00 comp rx=0 tx=0).stop 2024-04-23T00:05:42.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.365+0000 7f79a616e640 1 -- 172.21.15.67:0/3575590689 shutdown_connections 2024-04-23T00:05:42.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.365+0000 7f79a616e640 1 --2- 172.21.15.67:0/3575590689 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f798806ec30 0x7f79880710f0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:42.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.365+0000 7f79a616e640 1 --2- 172.21.15.67:0/3575590689 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f79a01962a0 0x7f79a019a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:42.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.365+0000 7f79a616e640 1 --2- 172.21.15.67:0/3575590689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79a01002a0 0x7f79a0195d60 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:42.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.365+0000 7f79a616e640 1 -- 172.21.15.67:0/3575590689 >> 172.21.15.67:0/3575590689 conn(0x7f79a00fbb20 msgr2=0x7f79a00fd220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:42.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.366+0000 7f79a616e640 1 -- 172.21.15.67:0/3575590689 shutdown_connections 2024-04-23T00:05:42.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.366+0000 7f79a616e640 1 -- 172.21.15.67:0/3575590689 wait complete. 2024-04-23T00:05:42.363 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 21 2024-04-23T00:05:42.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.479+0000 7fbcf9ee9640 1 -- 172.21.15.67:0/3095392933 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf4101220 msgr2=0x7fbcf4101600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:42.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.479+0000 7fbcf9ee9640 1 --2- 172.21.15.67:0/3095392933 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf4101220 0x7fbcf4101600 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fbce4009880 tx=0x7fbce402bce0 comp rx=0 tx=0).stop 2024-04-23T00:05:42.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.479+0000 7fbcf9ee9640 1 -- 172.21.15.67:0/3095392933 shutdown_connections 2024-04-23T00:05:42.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.479+0000 7fbcf9ee9640 1 --2- 172.21.15.67:0/3095392933 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbcf4101b40 0x7fbcf410a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:42.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.479+0000 7fbcf9ee9640 1 --2- 172.21.15.67:0/3095392933 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf4101220 0x7fbcf4101600 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:42.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.479+0000 7fbcf9ee9640 1 -- 172.21.15.67:0/3095392933 >> 172.21.15.67:0/3095392933 conn(0x7fbcf40faa20 msgr2=0x7fbcf40fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:42.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.479+0000 7fbcf9ee9640 1 -- 172.21.15.67:0/3095392933 shutdown_connections 2024-04-23T00:05:42.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.479+0000 7fbcf9ee9640 1 -- 172.21.15.67:0/3095392933 wait complete. 2024-04-23T00:05:42.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.480+0000 7fbcf9ee9640 1 Processor -- start 2024-04-23T00:05:42.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.480+0000 7fbcf9ee9640 1 -- start start 2024-04-23T00:05:42.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.480+0000 7fbcf9ee9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf4101220 0x7fbcf4195d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:42.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.480+0000 7fbcf9ee9640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbcf4101b40 0x7fbcf41962d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:42.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.480+0000 7fbcf9ee9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf4196960 con 0x7fbcf4101220 2024-04-23T00:05:42.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.480+0000 7fbcf9ee9640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf419a730 con 0x7fbcf4101b40 2024-04-23T00:05:42.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.480+0000 7fbcf37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf4101220 0x7fbcf4195d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:42.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.480+0000 7fbcf37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf4101220 0x7fbcf4195d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38858/0 (socket says 172.21.15.67:38858) 2024-04-23T00:05:42.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.480+0000 7fbcf37fe640 1 -- 172.21.15.67:0/3182954 learned_addr learned my addr 172.21.15.67:0/3182954 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:42.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.480+0000 7fbcf37fe640 1 -- 172.21.15.67:0/3182954 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbcf4101b40 msgr2=0x7fbcf41962d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:42.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.480+0000 7fbcf37fe640 1 --2- 172.21.15.67:0/3182954 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbcf4101b40 0x7fbcf41962d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:42.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.480+0000 7fbcf37fe640 1 -- 172.21.15.67:0/3182954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbce4009560 con 0x7fbcf4101220 2024-04-23T00:05:42.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.481+0000 7fbcf37fe640 1 --2- 172.21.15.67:0/3182954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf4101220 0x7fbcf4195d90 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fbce4009230 tx=0x7fbce4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:42.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.481+0000 7fbcf0ff9640 1 -- 172.21.15.67:0/3182954 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbce40037c0 con 0x7fbcf4101220 2024-04-23T00:05:42.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.481+0000 7fbcf9ee9640 1 -- 172.21.15.67:0/3182954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcf419a9b0 con 0x7fbcf4101220 2024-04-23T00:05:42.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.481+0000 7fbcf0ff9640 1 -- 172.21.15.67:0/3182954 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbce4003de0 con 0x7fbcf4101220 2024-04-23T00:05:42.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.481+0000 7fbcf0ff9640 1 -- 172.21.15.67:0/3182954 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbce403e450 con 0x7fbcf4101220 2024-04-23T00:05:42.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.481+0000 7fbcf9ee9640 1 -- 172.21.15.67:0/3182954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcf419af10 con 0x7fbcf4101220 2024-04-23T00:05:42.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.482+0000 7fbcf9ee9640 1 -- 172.21.15.67:0/3182954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcf404e040 con 0x7fbcf4101220 2024-04-23T00:05:42.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.482+0000 7fbcf0ff9640 1 -- 172.21.15.67:0/3182954 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbce4005060 con 0x7fbcf4101220 2024-04-23T00:05:42.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.483+0000 7fbcf0ff9640 1 --2- 172.21.15.67:0/3182954 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbcdc06ec30 0x7fbcdc0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:42.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.483+0000 7fbcf0ff9640 1 -- 172.21.15.67:0/3182954 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fbce40b1c80 con 0x7fbcf4101220 2024-04-23T00:05:42.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.483+0000 7fbcf2ffd640 1 --2- 172.21.15.67:0/3182954 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbcdc06ec30 0x7fbcdc0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:42.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.483+0000 7fbcf2ffd640 1 --2- 172.21.15.67:0/3182954 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbcdc06ec30 0x7fbcdc0710f0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fbcf41973a0 tx=0x7fbce8005c10 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:42.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.488+0000 7fbcf0ff9640 1 -- 172.21.15.67:0/3182954 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fbce407a980 con 0x7fbcf4101220 2024-04-23T00:05:42.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.715+0000 7fbcf9ee9640 1 -- 172.21.15.67:0/3182954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 21, "format": "json"} v 0) v1 -- 0x7fbcf4107650 con 0x7fbcf4101220 2024-04-23T00:05:42.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.716+0000 7fbcf0ff9640 1 -- 172.21.15.67:0/3182954 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 21, "format": "json"}]=0 dumped fsmap epoch 21 v42) v1 ==== 107+0+4892 (secure 0 0 0) 0x7fbce407e040 con 0x7fbcf4101220 2024-04-23T00:05:42.702 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:05:42.703 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T23:20:20.823261+0000","modified":"2024-04-22T23:26:38.553512+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14484,"mds_1":24273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14484":{"gid":14484,"name":"cephfs.smithi067.xyazmh","rank":0,"incarnation":13,"state":"up:active","state_seq":6,"addr":"172.21.15.67:6829/3606151860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3606151860},{"type":"v1","addr":"172.21.15.67:6829","nonce":3606151860}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14496":{"gid":14496,"name":"cephfs.smithi067.aqlxpv","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6827/3305304988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3305304988},{"type":"v1","addr":"172.21.15.67:6827","nonce":3305304988}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24273":{"gid":24273,"name":"cephfs.smithi134.owipad","rank":1,"incarnation":6,"state":"up:active","state_seq":4,"addr":"172.21.15.134:6825/535957142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":535957142},{"type":"v1","addr":"172.21.15.134:6825","nonce":535957142}]},"join_fscid":1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24289":{"gid":24289,"name":"cephfs.smithi134.vuxbsj","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.134:6827/1954821676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":1954821676},{"type":"v1","addr":"172.21.15.134:6827","nonce":1954821676}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T00:05:42.703 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 21 2024-04-23T00:05:42.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.718+0000 7fbcf9ee9640 1 -- 172.21.15.67:0/3182954 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbcdc06ec30 msgr2=0x7fbcdc0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:42.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.718+0000 7fbcf9ee9640 1 --2- 172.21.15.67:0/3182954 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbcdc06ec30 0x7fbcdc0710f0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fbcf41973a0 tx=0x7fbce8005c10 comp rx=0 tx=0).stop 2024-04-23T00:05:42.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.718+0000 7fbcf9ee9640 1 -- 172.21.15.67:0/3182954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf4101220 msgr2=0x7fbcf4195d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:42.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.718+0000 7fbcf9ee9640 1 --2- 172.21.15.67:0/3182954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf4101220 0x7fbcf4195d90 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fbce4009230 tx=0x7fbce4004270 comp rx=0 tx=0).stop 2024-04-23T00:05:42.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.718+0000 7fbcf9ee9640 1 -- 172.21.15.67:0/3182954 shutdown_connections 2024-04-23T00:05:42.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.718+0000 7fbcf9ee9640 1 --2- 172.21.15.67:0/3182954 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbcdc06ec30 0x7fbcdc0710f0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:42.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.718+0000 7fbcf9ee9640 1 --2- 172.21.15.67:0/3182954 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbcf4101b40 0x7fbcf41962d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:42.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.718+0000 7fbcf9ee9640 1 --2- 172.21.15.67:0/3182954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf4101220 0x7fbcf4195d90 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:42.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.718+0000 7fbcf9ee9640 1 -- 172.21.15.67:0/3182954 >> 172.21.15.67:0/3182954 conn(0x7fbcf40faa20 msgr2=0x7fbcf4104820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:42.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.718+0000 7fbcf9ee9640 1 -- 172.21.15.67:0/3182954 shutdown_connections 2024-04-23T00:05:42.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.718+0000 7fbcf9ee9640 1 -- 172.21.15.67:0/3182954 wait complete. 2024-04-23T00:05:42.716 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 22 2024-04-23T00:05:42.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.832+0000 7f592558d640 1 -- 172.21.15.67:0/606085793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5920067f70 msgr2=0x7f5920104c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:42.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.832+0000 7f592558d640 1 --2- 172.21.15.67:0/606085793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5920067f70 0x7f5920104c70 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f5914009880 tx=0x7f591402bce0 comp rx=0 tx=0).stop 2024-04-23T00:05:42.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.832+0000 7f592558d640 1 -- 172.21.15.67:0/606085793 shutdown_connections 2024-04-23T00:05:42.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.832+0000 7f592558d640 1 --2- 172.21.15.67:0/606085793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5920067f70 0x7f5920104c70 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:42.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.832+0000 7f592558d640 1 --2- 172.21.15.67:0/606085793 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59200675c0 0x7f59200679a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:42.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.832+0000 7f592558d640 1 -- 172.21.15.67:0/606085793 >> 172.21.15.67:0/606085793 conn(0x7f5920074a50 msgr2=0x7f5920074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:42.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.833+0000 7f592558d640 1 -- 172.21.15.67:0/606085793 shutdown_connections 2024-04-23T00:05:42.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.833+0000 7f592558d640 1 -- 172.21.15.67:0/606085793 wait complete. 2024-04-23T00:05:42.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.833+0000 7f592558d640 1 Processor -- start 2024-04-23T00:05:42.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.833+0000 7f592558d640 1 -- start start 2024-04-23T00:05:42.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.833+0000 7f592558d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59200675c0 0x7f5920195d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:42.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.833+0000 7f592558d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59201962c0 0x7f592019a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:42.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.833+0000 7f592558d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59201968c0 con 0x7f59200675c0 2024-04-23T00:05:42.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.833+0000 7f592558d640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5920196a30 con 0x7f59201962c0 2024-04-23T00:05:42.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.834+0000 7f591effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59200675c0 0x7f5920195d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:42.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.834+0000 7f591effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59200675c0 0x7f5920195d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38884/0 (socket says 172.21.15.67:38884) 2024-04-23T00:05:42.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.834+0000 7f591effd640 1 -- 172.21.15.67:0/2404259564 learned_addr learned my addr 172.21.15.67:0/2404259564 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:42.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.834+0000 7f591e7fc640 1 --2- 172.21.15.67:0/2404259564 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59201962c0 0x7f592019a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:42.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.834+0000 7f591effd640 1 -- 172.21.15.67:0/2404259564 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59201962c0 msgr2=0x7f592019a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:42.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.834+0000 7f591effd640 1 --2- 172.21.15.67:0/2404259564 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59201962c0 0x7f592019a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:42.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.834+0000 7f591effd640 1 -- 172.21.15.67:0/2404259564 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5914009560 con 0x7f59200675c0 2024-04-23T00:05:42.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.834+0000 7f591effd640 1 --2- 172.21.15.67:0/2404259564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59200675c0 0x7f5920195d80 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f591000c920 tx=0x7f591000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:42.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.834+0000 7f5907fff640 1 -- 172.21.15.67:0/2404259564 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5910007b80 con 0x7f59200675c0 2024-04-23T00:05:42.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.834+0000 7f5907fff640 1 -- 172.21.15.67:0/2404259564 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f59100054e0 con 0x7f59200675c0 2024-04-23T00:05:42.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.834+0000 7f592558d640 1 -- 172.21.15.67:0/2404259564 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f592019ad20 con 0x7f59200675c0 2024-04-23T00:05:42.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.834+0000 7f5907fff640 1 -- 172.21.15.67:0/2404259564 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f591000f450 con 0x7f59200675c0 2024-04-23T00:05:42.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.834+0000 7f592558d640 1 -- 172.21.15.67:0/2404259564 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f592019b280 con 0x7f59200675c0 2024-04-23T00:05:42.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.835+0000 7f592558d640 1 -- 172.21.15.67:0/2404259564 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f592019b560 con 0x7f59200675c0 2024-04-23T00:05:42.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.836+0000 7f5907fff640 1 -- 172.21.15.67:0/2404259564 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5910004f60 con 0x7f59200675c0 2024-04-23T00:05:42.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.836+0000 7f5907fff640 1 --2- 172.21.15.67:0/2404259564 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f590806ec30 0x7f59080710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:42.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.836+0000 7f5907fff640 1 -- 172.21.15.67:0/2404259564 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f591008ee80 con 0x7f59200675c0 2024-04-23T00:05:42.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.836+0000 7f591e7fc640 1 --2- 172.21.15.67:0/2404259564 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f590806ec30 0x7f59080710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:42.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.837+0000 7f591e7fc640 1 --2- 172.21.15.67:0/2404259564 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f590806ec30 0x7f59080710f0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f5914006010 tx=0x7f5914000bc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:42.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:42.841+0000 7f5907fff640 1 -- 172.21.15.67:0/2404259564 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f5910057ac0 con 0x7f59200675c0 2024-04-23T00:05:43.030 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:42 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/137691625' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 19, "format": "json"}]: dispatch 2024-04-23T00:05:43.030 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:42 smithi067 ceph-mon[112438]: pgmap v1184: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:05:43.030 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:42 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3575590689' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 20, "format": "json"}]: dispatch 2024-04-23T00:05:43.030 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:42 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3182954' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 21, "format": "json"}]: dispatch 2024-04-23T00:05:43.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.064+0000 7f592558d640 1 -- 172.21.15.67:0/2404259564 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 22, "format": "json"} v 0) v1 -- 0x7f59201970d0 con 0x7f59200675c0 2024-04-23T00:05:43.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.064+0000 7f5907fff640 1 -- 172.21.15.67:0/2404259564 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 22, "format": "json"}]=0 dumped fsmap epoch 22 v42) v1 ==== 107+0+4891 (secure 0 0 0) 0x7f591005b180 con 0x7f59200675c0 2024-04-23T00:05:43.051 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:05:43.051 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T23:20:20.823261+0000","modified":"2024-04-22T23:27:20.577198+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14484,"mds_1":24273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14484":{"gid":14484,"name":"cephfs.smithi067.xyazmh","rank":0,"incarnation":13,"state":"up:active","state_seq":6,"addr":"172.21.15.67:6829/3606151860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3606151860},{"type":"v1","addr":"172.21.15.67:6829","nonce":3606151860}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14496":{"gid":14496,"name":"cephfs.smithi067.aqlxpv","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6827/3305304988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3305304988},{"type":"v1","addr":"172.21.15.67:6827","nonce":3305304988}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24273":{"gid":24273,"name":"cephfs.smithi134.owipad","rank":1,"incarnation":6,"state":"up:active","state_seq":4,"addr":"172.21.15.134:6825/535957142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":535957142},{"type":"v1","addr":"172.21.15.134:6825","nonce":535957142}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24289":{"gid":24289,"name":"cephfs.smithi134.vuxbsj","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.134:6827/1954821676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":1954821676},{"type":"v1","addr":"172.21.15.134:6827","nonce":1954821676}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T00:05:43.051 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 22 2024-04-23T00:05:43.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.066+0000 7f592558d640 1 -- 172.21.15.67:0/2404259564 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f590806ec30 msgr2=0x7f59080710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:43.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.066+0000 7f592558d640 1 --2- 172.21.15.67:0/2404259564 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f590806ec30 0x7f59080710f0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f5914006010 tx=0x7f5914000bc0 comp rx=0 tx=0).stop 2024-04-23T00:05:43.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.066+0000 7f592558d640 1 -- 172.21.15.67:0/2404259564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59200675c0 msgr2=0x7f5920195d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:43.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.066+0000 7f592558d640 1 --2- 172.21.15.67:0/2404259564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59200675c0 0x7f5920195d80 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f591000c920 tx=0x7f591000cde0 comp rx=0 tx=0).stop 2024-04-23T00:05:43.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.066+0000 7f592558d640 1 -- 172.21.15.67:0/2404259564 shutdown_connections 2024-04-23T00:05:43.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.066+0000 7f592558d640 1 --2- 172.21.15.67:0/2404259564 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f590806ec30 0x7f59080710f0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:43.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.066+0000 7f592558d640 1 --2- 172.21.15.67:0/2404259564 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59201962c0 0x7f592019a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:43.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.067+0000 7f592558d640 1 --2- 172.21.15.67:0/2404259564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59200675c0 0x7f5920195d80 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:43.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.067+0000 7f592558d640 1 -- 172.21.15.67:0/2404259564 >> 172.21.15.67:0/2404259564 conn(0x7f5920074a50 msgr2=0x7f59200fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:43.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.067+0000 7f592558d640 1 -- 172.21.15.67:0/2404259564 shutdown_connections 2024-04-23T00:05:43.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.067+0000 7f592558d640 1 -- 172.21.15.67:0/2404259564 wait complete. 2024-04-23T00:05:43.064 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 23 2024-04-23T00:05:43.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.181+0000 7f6ec85ae640 1 -- 172.21.15.67:0/2495856259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ec01002d0 msgr2=0x7f6ec01006b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:43.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.181+0000 7f6ec85ae640 1 --2- 172.21.15.67:0/2495856259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ec01002d0 0x7f6ec01006b0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc009880 tx=0x7f6ebc02bce0 comp rx=0 tx=0).stop 2024-04-23T00:05:43.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.181+0000 7f6ec85ae640 1 -- 172.21.15.67:0/2495856259 shutdown_connections 2024-04-23T00:05:43.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.181+0000 7f6ec85ae640 1 --2- 172.21.15.67:0/2495856259 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ec0100c80 0x7f6ec0104cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:43.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.181+0000 7f6ec85ae640 1 --2- 172.21.15.67:0/2495856259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ec01002d0 0x7f6ec01006b0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:43.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.181+0000 7f6ec85ae640 1 -- 172.21.15.67:0/2495856259 >> 172.21.15.67:0/2495856259 conn(0x7f6ec00fbb50 msgr2=0x7f6ec00fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:43.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.181+0000 7f6ec85ae640 1 -- 172.21.15.67:0/2495856259 shutdown_connections 2024-04-23T00:05:43.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.181+0000 7f6ec85ae640 1 -- 172.21.15.67:0/2495856259 wait complete. 2024-04-23T00:05:43.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.181+0000 7f6ec85ae640 1 Processor -- start 2024-04-23T00:05:43.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.181+0000 7f6ec85ae640 1 -- start start 2024-04-23T00:05:43.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.182+0000 7f6ec85ae640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ec01002d0 0x7f6ec00725a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:43.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.182+0000 7f6ec85ae640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ec0072ae0 0x7f6ec0072f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:43.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.182+0000 7f6ec85ae640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec01a1fb0 con 0x7f6ec0072ae0 2024-04-23T00:05:43.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.182+0000 7f6ec85ae640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec01a20f0 con 0x7f6ec01002d0 2024-04-23T00:05:43.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.182+0000 7f6ec5b49640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ec0072ae0 0x7f6ec0072f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:43.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.182+0000 7f6ec5b49640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ec0072ae0 0x7f6ec0072f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38898/0 (socket says 172.21.15.67:38898) 2024-04-23T00:05:43.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.182+0000 7f6ec5b49640 1 -- 172.21.15.67:0/478834429 learned_addr learned my addr 172.21.15.67:0/478834429 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:43.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.182+0000 7f6ec634a640 1 --2- 172.21.15.67:0/478834429 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ec01002d0 0x7f6ec00725a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:43.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.182+0000 7f6ec5b49640 1 -- 172.21.15.67:0/478834429 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ec01002d0 msgr2=0x7f6ec00725a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:43.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.182+0000 7f6ec5b49640 1 --2- 172.21.15.67:0/478834429 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ec01002d0 0x7f6ec00725a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:43.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.182+0000 7f6ec5b49640 1 -- 172.21.15.67:0/478834429 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ebc009560 con 0x7f6ec0072ae0 2024-04-23T00:05:43.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.182+0000 7f6ec634a640 1 --2- 172.21.15.67:0/478834429 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ec01002d0 0x7f6ec00725a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T00:05:43.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.182+0000 7f6ec5b49640 1 --2- 172.21.15.67:0/478834429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ec0072ae0 0x7f6ec0072f40 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f6eb000e9a0 tx=0x7f6eb000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:43.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.182+0000 7f6eb77fe640 1 -- 172.21.15.67:0/478834429 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6eb000cc90 con 0x7f6ec0072ae0 2024-04-23T00:05:43.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.182+0000 7f6eb77fe640 1 -- 172.21.15.67:0/478834429 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6eb000cdf0 con 0x7f6ec0072ae0 2024-04-23T00:05:43.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.182+0000 7f6ec85ae640 1 -- 172.21.15.67:0/478834429 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ec01a2230 con 0x7f6ec0072ae0 2024-04-23T00:05:43.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.182+0000 7f6eb77fe640 1 -- 172.21.15.67:0/478834429 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6eb0010430 con 0x7f6ec0072ae0 2024-04-23T00:05:43.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.182+0000 7f6ec85ae640 1 -- 172.21.15.67:0/478834429 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ec01a2790 con 0x7f6ec0072ae0 2024-04-23T00:05:43.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.184+0000 7f6ec85ae640 1 -- 172.21.15.67:0/478834429 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ec004e040 con 0x7f6ec0072ae0 2024-04-23T00:05:43.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.184+0000 7f6eb77fe640 1 -- 172.21.15.67:0/478834429 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6eb0005020 con 0x7f6ec0072ae0 2024-04-23T00:05:43.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.184+0000 7f6eb77fe640 1 --2- 172.21.15.67:0/478834429 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6eac06ec30 0x7f6eac0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:43.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.185+0000 7f6eb77fe640 1 -- 172.21.15.67:0/478834429 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f6eb0014070 con 0x7f6ec0072ae0 2024-04-23T00:05:43.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.185+0000 7f6ec634a640 1 --2- 172.21.15.67:0/478834429 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6eac06ec30 0x7f6eac0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:43.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.185+0000 7f6ec634a640 1 --2- 172.21.15.67:0/478834429 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6eac06ec30 0x7f6eac0710f0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc000c00 tx=0x7f6ebc02cb40 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:43.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.189+0000 7f6eb77fe640 1 -- 172.21.15.67:0/478834429 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f6eb0058140 con 0x7f6ec0072ae0 2024-04-23T00:05:43.230 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:42 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/137691625' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 19, "format": "json"}]: dispatch 2024-04-23T00:05:43.230 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:42 smithi134 ceph-mon[85630]: pgmap v1184: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:05:43.230 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:42 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3575590689' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 20, "format": "json"}]: dispatch 2024-04-23T00:05:43.230 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:42 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3182954' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 21, "format": "json"}]: dispatch 2024-04-23T00:05:43.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.415+0000 7f6ec85ae640 1 -- 172.21.15.67:0/478834429 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 23, "format": "json"} v 0) v1 -- 0x7f6ec0100d20 con 0x7f6ec0072ae0 2024-04-23T00:05:43.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.415+0000 7f6eb77fe640 1 -- 172.21.15.67:0/478834429 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 23, "format": "json"}]=0 dumped fsmap epoch 23 v42) v1 ==== 107+0+3331 (secure 0 0 0) 0x7f6eb005b800 con 0x7f6ec0072ae0 2024-04-23T00:05:43.402 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:05:43.402 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T23:20:20.823261+0000","modified":"2024-04-22T23:32:26.859056+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14484,"mds_1":24273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14484":{"gid":14484,"name":"cephfs.smithi067.xyazmh","rank":0,"incarnation":13,"state":"up:active","state_seq":6,"addr":"172.21.15.67:6829/3606151860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3606151860},{"type":"v1","addr":"172.21.15.67:6829","nonce":3606151860}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24273":{"gid":24273,"name":"cephfs.smithi134.owipad","rank":1,"incarnation":6,"state":"up:active","state_seq":4,"addr":"172.21.15.134:6825/535957142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":535957142},{"type":"v1","addr":"172.21.15.134:6825","nonce":535957142}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":14484,"qdb_cluster":[14484,24273]},"id":1}]} 2024-04-23T00:05:43.402 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 23 2024-04-23T00:05:43.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.417+0000 7f6ec85ae640 1 -- 172.21.15.67:0/478834429 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6eac06ec30 msgr2=0x7f6eac0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:43.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.417+0000 7f6ec85ae640 1 --2- 172.21.15.67:0/478834429 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6eac06ec30 0x7f6eac0710f0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc000c00 tx=0x7f6ebc02cb40 comp rx=0 tx=0).stop 2024-04-23T00:05:43.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.417+0000 7f6ec85ae640 1 -- 172.21.15.67:0/478834429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ec0072ae0 msgr2=0x7f6ec0072f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:43.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.417+0000 7f6ec85ae640 1 --2- 172.21.15.67:0/478834429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ec0072ae0 0x7f6ec0072f40 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f6eb000e9a0 tx=0x7f6eb000ee60 comp rx=0 tx=0).stop 2024-04-23T00:05:43.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.417+0000 7f6ec85ae640 1 -- 172.21.15.67:0/478834429 shutdown_connections 2024-04-23T00:05:43.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.417+0000 7f6ec85ae640 1 --2- 172.21.15.67:0/478834429 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f6eac06ec30 0x7f6eac0710f0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:43.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.417+0000 7f6ec85ae640 1 --2- 172.21.15.67:0/478834429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ec0072ae0 0x7f6ec0072f40 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:43.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.417+0000 7f6ec85ae640 1 --2- 172.21.15.67:0/478834429 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6ec01002d0 0x7f6ec00725a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:43.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.417+0000 7f6ec85ae640 1 -- 172.21.15.67:0/478834429 >> 172.21.15.67:0/478834429 conn(0x7f6ec00fbb50 msgr2=0x7f6ec00fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:43.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.417+0000 7f6ec85ae640 1 -- 172.21.15.67:0/478834429 shutdown_connections 2024-04-23T00:05:43.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.417+0000 7f6ec85ae640 1 -- 172.21.15.67:0/478834429 wait complete. 2024-04-23T00:05:43.415 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 23 2024-04-23T00:05:43.415 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 24 2024-04-23T00:05:43.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.531+0000 7fc53dd27640 1 -- 172.21.15.67:0/2169536227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc538101b20 msgr2=0x7fc53810a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:43.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.531+0000 7fc53dd27640 1 --2- 172.21.15.67:0/2169536227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc538101b20 0x7fc53810a000 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fc528009880 tx=0x7fc52802bd70 comp rx=0 tx=0).stop 2024-04-23T00:05:43.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.531+0000 7fc53dd27640 1 -- 172.21.15.67:0/2169536227 shutdown_connections 2024-04-23T00:05:43.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.531+0000 7fc53dd27640 1 --2- 172.21.15.67:0/2169536227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc538101b20 0x7fc53810a000 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:43.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.531+0000 7fc53dd27640 1 --2- 172.21.15.67:0/2169536227 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc538101200 0x7fc5381015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:43.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.531+0000 7fc53dd27640 1 -- 172.21.15.67:0/2169536227 >> 172.21.15.67:0/2169536227 conn(0x7fc5380faa20 msgr2=0x7fc5380fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:43.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.532+0000 7fc53dd27640 1 -- 172.21.15.67:0/2169536227 shutdown_connections 2024-04-23T00:05:43.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.532+0000 7fc53dd27640 1 -- 172.21.15.67:0/2169536227 wait complete. 2024-04-23T00:05:43.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.532+0000 7fc53dd27640 1 Processor -- start 2024-04-23T00:05:43.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.532+0000 7fc53dd27640 1 -- start start 2024-04-23T00:05:43.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.532+0000 7fc53dd27640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc538101200 0x7fc5380fee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:43.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.532+0000 7fc53dd27640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc538101b20 0x7fc5380ff390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:43.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.532+0000 7fc53dd27640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5380ff8d0 con 0x7fc538101b20 2024-04-23T00:05:43.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.532+0000 7fc53dd27640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5380ffa40 con 0x7fc538101200 2024-04-23T00:05:43.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.533+0000 7fc536ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc538101b20 0x7fc5380ff390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:43.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.533+0000 7fc536ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc538101b20 0x7fc5380ff390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38926/0 (socket says 172.21.15.67:38926) 2024-04-23T00:05:43.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.533+0000 7fc536ffd640 1 -- 172.21.15.67:0/147210049 learned_addr learned my addr 172.21.15.67:0/147210049 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:43.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.533+0000 7fc5377fe640 1 --2- 172.21.15.67:0/147210049 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc538101200 0x7fc5380fee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:43.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.533+0000 7fc536ffd640 1 -- 172.21.15.67:0/147210049 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc538101200 msgr2=0x7fc5380fee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:43.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.533+0000 7fc536ffd640 1 --2- 172.21.15.67:0/147210049 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc538101200 0x7fc5380fee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:43.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.533+0000 7fc536ffd640 1 -- 172.21.15.67:0/147210049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc528009560 con 0x7fc538101b20 2024-04-23T00:05:43.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.533+0000 7fc536ffd640 1 --2- 172.21.15.67:0/147210049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc538101b20 0x7fc5380ff390 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fc528009230 tx=0x7fc52802c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:43.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.533+0000 7fc534ff9640 1 -- 172.21.15.67:0/147210049 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc528004520 con 0x7fc538101b20 2024-04-23T00:05:43.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.533+0000 7fc534ff9640 1 -- 172.21.15.67:0/147210049 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc528003ab0 con 0x7fc538101b20 2024-04-23T00:05:43.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.533+0000 7fc53dd27640 1 -- 172.21.15.67:0/147210049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5380ffc60 con 0x7fc538101b20 2024-04-23T00:05:43.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.533+0000 7fc534ff9640 1 -- 172.21.15.67:0/147210049 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc528005690 con 0x7fc538101b20 2024-04-23T00:05:43.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.533+0000 7fc53dd27640 1 -- 172.21.15.67:0/147210049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5381a2280 con 0x7fc538101b20 2024-04-23T00:05:43.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.534+0000 7fc53dd27640 1 -- 172.21.15.67:0/147210049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc53804e040 con 0x7fc538101b20 2024-04-23T00:05:43.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.535+0000 7fc534ff9640 1 -- 172.21.15.67:0/147210049 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc528004ed0 con 0x7fc538101b20 2024-04-23T00:05:43.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.535+0000 7fc534ff9640 1 --2- 172.21.15.67:0/147210049 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc52406ebe0 0x7fc5240710a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:43.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.535+0000 7fc534ff9640 1 -- 172.21.15.67:0/147210049 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fc5280b1b10 con 0x7fc538101b20 2024-04-23T00:05:43.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.535+0000 7fc5377fe640 1 --2- 172.21.15.67:0/147210049 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc52406ebe0 0x7fc5240710a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:43.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.536+0000 7fc5377fe640 1 --2- 172.21.15.67:0/147210049 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc52406ebe0 0x7fc5240710a0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fc520005fd0 tx=0x7fc520005e20 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:43.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.540+0000 7fc534ff9640 1 -- 172.21.15.67:0/147210049 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc52807a750 con 0x7fc538101b20 2024-04-23T00:05:43.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.766+0000 7fc53dd27640 1 -- 172.21.15.67:0/147210049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 24, "format": "json"} v 0) v1 -- 0x7fc538107670 con 0x7fc538101b20 2024-04-23T00:05:43.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.767+0000 7fc534ff9640 1 -- 172.21.15.67:0/147210049 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 24, "format": "json"}]=0 dumped fsmap epoch 24 v42) v1 ==== 107+0+3298 (secure 0 0 0) 0x7fc528043d00 con 0x7fc538101b20 2024-04-23T00:05:43.753 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:05:43.754 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34380,"name":"cephfs.smithi067.aqlxpv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6827/1310921930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":1310921930},{"type":"v1","addr":"172.21.15.67:6827","nonce":1310921930}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44259,"name":"cephfs.smithi134.vuxbsj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6827/1827288036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":1827288036},{"type":"v1","addr":"172.21.15.134:6827","nonce":1827288036}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T23:20:20.823261+0000","modified":"2024-04-22T23:32:27.027617+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T00:05:43.754 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 24 2024-04-23T00:05:43.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.769+0000 7fc53dd27640 1 -- 172.21.15.67:0/147210049 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc52406ebe0 msgr2=0x7fc5240710a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:43.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.769+0000 7fc53dd27640 1 --2- 172.21.15.67:0/147210049 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc52406ebe0 0x7fc5240710a0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fc520005fd0 tx=0x7fc520005e20 comp rx=0 tx=0).stop 2024-04-23T00:05:43.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.769+0000 7fc53dd27640 1 -- 172.21.15.67:0/147210049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc538101b20 msgr2=0x7fc5380ff390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:43.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.769+0000 7fc53dd27640 1 --2- 172.21.15.67:0/147210049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc538101b20 0x7fc5380ff390 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fc528009230 tx=0x7fc52802c870 comp rx=0 tx=0).stop 2024-04-23T00:05:43.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.769+0000 7fc53dd27640 1 -- 172.21.15.67:0/147210049 shutdown_connections 2024-04-23T00:05:43.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.769+0000 7fc53dd27640 1 --2- 172.21.15.67:0/147210049 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc52406ebe0 0x7fc5240710a0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:43.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.769+0000 7fc53dd27640 1 --2- 172.21.15.67:0/147210049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc538101b20 0x7fc5380ff390 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:43.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.769+0000 7fc53dd27640 1 --2- 172.21.15.67:0/147210049 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc538101200 0x7fc5380fee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:43.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.769+0000 7fc53dd27640 1 -- 172.21.15.67:0/147210049 >> 172.21.15.67:0/147210049 conn(0x7fc5380faa20 msgr2=0x7fc538104840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:43.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.769+0000 7fc53dd27640 1 -- 172.21.15.67:0/147210049 shutdown_connections 2024-04-23T00:05:43.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.769+0000 7fc53dd27640 1 -- 172.21.15.67:0/147210049 wait complete. 2024-04-23T00:05:43.767 DEBUG:tasks.fs:max_mds not reduced in epoch 24 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-23T00:05:43.767 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 24 2024-04-23T00:05:43.767 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 25 2024-04-23T00:05:43.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.897+0000 7fc1979ae640 1 -- 172.21.15.67:0/3950842112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1901002a0 msgr2=0x7fc190100680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:43.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.897+0000 7fc1979ae640 1 --2- 172.21.15.67:0/3950842112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1901002a0 0x7fc190100680 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fc184009880 tx=0x7fc18402bce0 comp rx=0 tx=0).stop 2024-04-23T00:05:43.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.897+0000 7fc1979ae640 1 -- 172.21.15.67:0/3950842112 shutdown_connections 2024-04-23T00:05:43.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.897+0000 7fc1979ae640 1 --2- 172.21.15.67:0/3950842112 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc190100c50 0x7fc190104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:43.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.897+0000 7fc1979ae640 1 --2- 172.21.15.67:0/3950842112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1901002a0 0x7fc190100680 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:43.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.897+0000 7fc1979ae640 1 -- 172.21.15.67:0/3950842112 >> 172.21.15.67:0/3950842112 conn(0x7fc1900fbb20 msgr2=0x7fc1900fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:43.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.897+0000 7fc1979ae640 1 -- 172.21.15.67:0/3950842112 shutdown_connections 2024-04-23T00:05:43.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.897+0000 7fc1979ae640 1 -- 172.21.15.67:0/3950842112 wait complete. 2024-04-23T00:05:43.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.897+0000 7fc1979ae640 1 Processor -- start 2024-04-23T00:05:43.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.898+0000 7fc1979ae640 1 -- start start 2024-04-23T00:05:43.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.898+0000 7fc1979ae640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1901002a0 0x7fc190195de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:43.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.898+0000 7fc1979ae640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc190196320 0x7fc19019a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:43.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.898+0000 7fc1979ae640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc190196920 con 0x7fc190196320 2024-04-23T00:05:43.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.898+0000 7fc1979ae640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc190196a90 con 0x7fc1901002a0 2024-04-23T00:05:43.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.898+0000 7fc194f49640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc190196320 0x7fc19019a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:43.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.898+0000 7fc194f49640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc190196320 0x7fc19019a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38944/0 (socket says 172.21.15.67:38944) 2024-04-23T00:05:43.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.898+0000 7fc194f49640 1 -- 172.21.15.67:0/1910788864 learned_addr learned my addr 172.21.15.67:0/1910788864 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:43.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.898+0000 7fc19574a640 1 --2- 172.21.15.67:0/1910788864 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1901002a0 0x7fc190195de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:43.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.898+0000 7fc194f49640 1 -- 172.21.15.67:0/1910788864 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1901002a0 msgr2=0x7fc190195de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:43.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.898+0000 7fc194f49640 1 --2- 172.21.15.67:0/1910788864 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1901002a0 0x7fc190195de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:43.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.898+0000 7fc194f49640 1 -- 172.21.15.67:0/1910788864 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc184009560 con 0x7fc190196320 2024-04-23T00:05:43.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.898+0000 7fc194f49640 1 --2- 172.21.15.67:0/1910788864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc190196320 0x7fc19019a710 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fc18c00ea30 tx=0x7fc18c00eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:43.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.898+0000 7fc1827fc640 1 -- 172.21.15.67:0/1910788864 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc18c00cdb0 con 0x7fc190196320 2024-04-23T00:05:43.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.898+0000 7fc1979ae640 1 -- 172.21.15.67:0/1910788864 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc19019ad10 con 0x7fc190196320 2024-04-23T00:05:43.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.898+0000 7fc1827fc640 1 -- 172.21.15.67:0/1910788864 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc18c011930 con 0x7fc190196320 2024-04-23T00:05:43.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.898+0000 7fc1827fc640 1 -- 172.21.15.67:0/1910788864 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc18c01a960 con 0x7fc190196320 2024-04-23T00:05:43.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.899+0000 7fc1979ae640 1 -- 172.21.15.67:0/1910788864 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc19019b270 con 0x7fc190196320 2024-04-23T00:05:43.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.899+0000 7fc1979ae640 1 -- 172.21.15.67:0/1910788864 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc19019b550 con 0x7fc190196320 2024-04-23T00:05:43.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.900+0000 7fc1827fc640 1 -- 172.21.15.67:0/1910788864 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc18c01abd0 con 0x7fc190196320 2024-04-23T00:05:43.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.900+0000 7fc1827fc640 1 --2- 172.21.15.67:0/1910788864 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc17c06ec30 0x7fc17c0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:43.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.900+0000 7fc1827fc640 1 -- 172.21.15.67:0/1910788864 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fc18c08f830 con 0x7fc190196320 2024-04-23T00:05:43.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.900+0000 7fc19574a640 1 --2- 172.21.15.67:0/1910788864 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc17c06ec30 0x7fc17c0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:43.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.901+0000 7fc19574a640 1 --2- 172.21.15.67:0/1910788864 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc17c06ec30 0x7fc17c0710f0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fc184006010 tx=0x7fc184000bc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:43.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:43.905+0000 7fc1827fc640 1 -- 172.21.15.67:0/1910788864 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fc18c058470 con 0x7fc190196320 2024-04-23T00:05:44.030 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:43 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2404259564' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-04-23T00:05:44.030 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:43 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/478834429' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-04-23T00:05:44.030 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:43 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/147210049' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-04-23T00:05:44.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.131+0000 7fc1979ae640 1 -- 172.21.15.67:0/1910788864 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 25, "format": "json"} v 0) v1 -- 0x7fc1901085a0 con 0x7fc190196320 2024-04-23T00:05:44.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.131+0000 7fc1827fc640 1 -- 172.21.15.67:0/1910788864 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 25, "format": "json"}]=0 dumped fsmap epoch 25 v42) v1 ==== 107+0+4859 (secure 0 0 0) 0x7fc18c05bb30 con 0x7fc190196320 2024-04-23T00:05:44.118 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:05:44.118 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34380,"name":"cephfs.smithi067.aqlxpv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6827/1310921930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":1310921930},{"type":"v1","addr":"172.21.15.67:6827","nonce":1310921930}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":34388,"name":"cephfs.smithi067.xyazmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6829/2016103304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2016103304},{"type":"v1","addr":"172.21.15.67:6829","nonce":2016103304}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25},{"gid":44259,"name":"cephfs.smithi134.vuxbsj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6827/1827288036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":1827288036},{"type":"v1","addr":"172.21.15.134:6827","nonce":1827288036}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44267,"name":"cephfs.smithi134.owipad","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6825/3366499307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":3366499307},{"type":"v1","addr":"172.21.15.134:6825","nonce":3366499307}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T23:20:20.823261+0000","modified":"2024-04-22T23:32:27.027617+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T00:05:44.118 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 25 2024-04-23T00:05:44.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.133+0000 7fc1979ae640 1 -- 172.21.15.67:0/1910788864 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc17c06ec30 msgr2=0x7fc17c0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:44.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.133+0000 7fc1979ae640 1 --2- 172.21.15.67:0/1910788864 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc17c06ec30 0x7fc17c0710f0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fc184006010 tx=0x7fc184000bc0 comp rx=0 tx=0).stop 2024-04-23T00:05:44.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.133+0000 7fc1979ae640 1 -- 172.21.15.67:0/1910788864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc190196320 msgr2=0x7fc19019a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:44.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.133+0000 7fc1979ae640 1 --2- 172.21.15.67:0/1910788864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc190196320 0x7fc19019a710 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fc18c00ea30 tx=0x7fc18c00eef0 comp rx=0 tx=0).stop 2024-04-23T00:05:44.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.134+0000 7fc1979ae640 1 -- 172.21.15.67:0/1910788864 shutdown_connections 2024-04-23T00:05:44.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.134+0000 7fc1979ae640 1 --2- 172.21.15.67:0/1910788864 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fc17c06ec30 0x7fc17c0710f0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:44.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.134+0000 7fc1979ae640 1 --2- 172.21.15.67:0/1910788864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc190196320 0x7fc19019a710 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:44.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.134+0000 7fc1979ae640 1 --2- 172.21.15.67:0/1910788864 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1901002a0 0x7fc190195de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:44.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.134+0000 7fc1979ae640 1 -- 172.21.15.67:0/1910788864 >> 172.21.15.67:0/1910788864 conn(0x7fc1900fbb20 msgr2=0x7fc1900fd360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:44.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.134+0000 7fc1979ae640 1 -- 172.21.15.67:0/1910788864 shutdown_connections 2024-04-23T00:05:44.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.134+0000 7fc1979ae640 1 -- 172.21.15.67:0/1910788864 wait complete. 2024-04-23T00:05:44.131 DEBUG:tasks.fs:max_mds not reduced in epoch 25 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-23T00:05:44.131 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 25 2024-04-23T00:05:44.131 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 26 2024-04-23T00:05:44.230 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:43 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2404259564' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-04-23T00:05:44.230 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:43 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/478834429' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-04-23T00:05:44.230 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:43 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/147210049' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-04-23T00:05:44.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.247+0000 7fbc95236640 1 -- 172.21.15.67:0/2461579514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc90101200 msgr2=0x7fbc901015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:44.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.247+0000 7fbc95236640 1 --2- 172.21.15.67:0/2461579514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc90101200 0x7fbc901015e0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fbc78009880 tx=0x7fbc7802bd70 comp rx=0 tx=0).stop 2024-04-23T00:05:44.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.247+0000 7fbc95236640 1 -- 172.21.15.67:0/2461579514 shutdown_connections 2024-04-23T00:05:44.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.247+0000 7fbc95236640 1 --2- 172.21.15.67:0/2461579514 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc90101b20 0x7fbc9010a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:44.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.247+0000 7fbc95236640 1 --2- 172.21.15.67:0/2461579514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc90101200 0x7fbc901015e0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:44.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.247+0000 7fbc95236640 1 -- 172.21.15.67:0/2461579514 >> 172.21.15.67:0/2461579514 conn(0x7fbc900faa20 msgr2=0x7fbc900fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:44.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.247+0000 7fbc95236640 1 -- 172.21.15.67:0/2461579514 shutdown_connections 2024-04-23T00:05:44.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.247+0000 7fbc95236640 1 -- 172.21.15.67:0/2461579514 wait complete. 2024-04-23T00:05:44.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.248+0000 7fbc95236640 1 Processor -- start 2024-04-23T00:05:44.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.248+0000 7fbc95236640 1 -- start start 2024-04-23T00:05:44.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.248+0000 7fbc95236640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc90101200 0x7fbc90195d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:44.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.248+0000 7fbc95236640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc90101b20 0x7fbc90196290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:44.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.248+0000 7fbc95236640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc90196890 con 0x7fbc90101b20 2024-04-23T00:05:44.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.248+0000 7fbc95236640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc90196a00 con 0x7fbc90101200 2024-04-23T00:05:44.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.248+0000 7fbc8e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc90101b20 0x7fbc90196290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:44.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.248+0000 7fbc8e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc90101b20 0x7fbc90196290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38970/0 (socket says 172.21.15.67:38970) 2024-04-23T00:05:44.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.248+0000 7fbc8e59c640 1 -- 172.21.15.67:0/3259552340 learned_addr learned my addr 172.21.15.67:0/3259552340 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:44.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.248+0000 7fbc8ed9d640 1 --2- 172.21.15.67:0/3259552340 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc90101200 0x7fbc90195d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:44.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.248+0000 7fbc8e59c640 1 -- 172.21.15.67:0/3259552340 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc90101200 msgr2=0x7fbc90195d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:44.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.248+0000 7fbc8e59c640 1 --2- 172.21.15.67:0/3259552340 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc90101200 0x7fbc90195d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:44.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.248+0000 7fbc8e59c640 1 -- 172.21.15.67:0/3259552340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc78009560 con 0x7fbc90101b20 2024-04-23T00:05:44.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.248+0000 7fbc8ed9d640 1 --2- 172.21.15.67:0/3259552340 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc90101200 0x7fbc90195d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T00:05:44.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.249+0000 7fbc8e59c640 1 --2- 172.21.15.67:0/3259552340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc90101b20 0x7fbc90196290 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fbc8000c8f0 tx=0x7fbc8000cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:44.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.249+0000 7fbc87fff640 1 -- 172.21.15.67:0/3259552340 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc80007b80 con 0x7fbc90101b20 2024-04-23T00:05:44.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.249+0000 7fbc87fff640 1 -- 172.21.15.67:0/3259552340 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbc80007ce0 con 0x7fbc90101b20 2024-04-23T00:05:44.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.249+0000 7fbc95236640 1 -- 172.21.15.67:0/3259552340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc9019a890 con 0x7fbc90101b20 2024-04-23T00:05:44.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.249+0000 7fbc87fff640 1 -- 172.21.15.67:0/3259552340 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc80018830 con 0x7fbc90101b20 2024-04-23T00:05:44.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.249+0000 7fbc95236640 1 -- 172.21.15.67:0/3259552340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc9019adf0 con 0x7fbc90101b20 2024-04-23T00:05:44.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.250+0000 7fbc95236640 1 -- 172.21.15.67:0/3259552340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc9004e040 con 0x7fbc90101b20 2024-04-23T00:05:44.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.250+0000 7fbc87fff640 1 -- 172.21.15.67:0/3259552340 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbc80018aa0 con 0x7fbc90101b20 2024-04-23T00:05:44.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.251+0000 7fbc87fff640 1 --2- 172.21.15.67:0/3259552340 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbc7c06ec30 0x7fbc7c0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:44.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.251+0000 7fbc87fff640 1 -- 172.21.15.67:0/3259552340 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fbc8008f170 con 0x7fbc90101b20 2024-04-23T00:05:44.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.251+0000 7fbc8ed9d640 1 --2- 172.21.15.67:0/3259552340 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbc7c06ec30 0x7fbc7c0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:44.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.251+0000 7fbc8ed9d640 1 --2- 172.21.15.67:0/3259552340 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbc7c06ec30 0x7fbc7c0710f0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fbc78005ec0 tx=0x7fbc78000bc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:44.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.255+0000 7fbc87fff640 1 -- 172.21.15.67:0/3259552340 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fbc80057db0 con 0x7fbc90101b20 2024-04-23T00:05:44.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.484+0000 7fbc95236640 1 -- 172.21.15.67:0/3259552340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 26, "format": "json"} v 0) v1 -- 0x7fbc9010d360 con 0x7fbc90101b20 2024-04-23T00:05:44.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.484+0000 7fbc87fff640 1 -- 172.21.15.67:0/3259552340 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 26, "format": "json"}]=0 dumped fsmap epoch 26 v42) v1 ==== 107+0+4080 (secure 0 0 0) 0x7fbc8005b470 con 0x7fbc90101b20 2024-04-23T00:05:44.471 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:05:44.471 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34388,"name":"cephfs.smithi067.xyazmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6829/2016103304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2016103304},{"type":"v1","addr":"172.21.15.67:6829","nonce":2016103304}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25},{"gid":44259,"name":"cephfs.smithi134.vuxbsj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6827/1827288036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":1827288036},{"type":"v1","addr":"172.21.15.134:6827","nonce":1827288036}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44267,"name":"cephfs.smithi134.owipad","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6825/3366499307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":3366499307},{"type":"v1","addr":"172.21.15.134:6825","nonce":3366499307}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T23:20:20.823261+0000","modified":"2024-04-22T23:32:27.027617+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T00:05:44.471 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-04-23T00:05:44.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.486+0000 7fbc95236640 1 -- 172.21.15.67:0/3259552340 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbc7c06ec30 msgr2=0x7fbc7c0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:44.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.486+0000 7fbc95236640 1 --2- 172.21.15.67:0/3259552340 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbc7c06ec30 0x7fbc7c0710f0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fbc78005ec0 tx=0x7fbc78000bc0 comp rx=0 tx=0).stop 2024-04-23T00:05:44.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.486+0000 7fbc95236640 1 -- 172.21.15.67:0/3259552340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc90101b20 msgr2=0x7fbc90196290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:44.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.486+0000 7fbc95236640 1 --2- 172.21.15.67:0/3259552340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc90101b20 0x7fbc90196290 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fbc8000c8f0 tx=0x7fbc8000cdb0 comp rx=0 tx=0).stop 2024-04-23T00:05:44.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.486+0000 7fbc95236640 1 -- 172.21.15.67:0/3259552340 shutdown_connections 2024-04-23T00:05:44.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.486+0000 7fbc95236640 1 --2- 172.21.15.67:0/3259552340 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbc7c06ec30 0x7fbc7c0710f0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:44.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.486+0000 7fbc95236640 1 --2- 172.21.15.67:0/3259552340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc90101b20 0x7fbc90196290 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:44.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.486+0000 7fbc95236640 1 --2- 172.21.15.67:0/3259552340 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc90101200 0x7fbc90195d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:44.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.486+0000 7fbc95236640 1 -- 172.21.15.67:0/3259552340 >> 172.21.15.67:0/3259552340 conn(0x7fbc900faa20 msgr2=0x7fbc90104840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:44.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.486+0000 7fbc95236640 1 -- 172.21.15.67:0/3259552340 shutdown_connections 2024-04-23T00:05:44.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.487+0000 7fbc95236640 1 -- 172.21.15.67:0/3259552340 wait complete. 2024-04-23T00:05:44.484 DEBUG:tasks.fs:max_mds not reduced in epoch 26 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-23T00:05:44.484 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 26 2024-04-23T00:05:44.484 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 27 2024-04-23T00:05:44.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.601+0000 7fdf9034e640 1 -- 172.21.15.67:0/3680375913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf88100460 msgr2=0x7fdf88103c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:44.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.601+0000 7fdf9034e640 1 --2- 172.21.15.67:0/3680375913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf88100460 0x7fdf88103c10 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fdf78009880 tx=0x7fdf7802bce0 comp rx=0 tx=0).stop 2024-04-23T00:05:44.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.601+0000 7fdf9034e640 1 -- 172.21.15.67:0/3680375913 shutdown_connections 2024-04-23T00:05:44.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.601+0000 7fdf9034e640 1 --2- 172.21.15.67:0/3680375913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf88100460 0x7fdf88103c10 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:44.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.601+0000 7fdf9034e640 1 --2- 172.21.15.67:0/3680375913 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf880ffab0 0x7fdf880ffe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:44.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.601+0000 7fdf9034e640 1 -- 172.21.15.67:0/3680375913 >> 172.21.15.67:0/3680375913 conn(0x7fdf88074250 msgr2=0x7fdf88074660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:44.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.601+0000 7fdf9034e640 1 -- 172.21.15.67:0/3680375913 shutdown_connections 2024-04-23T00:05:44.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.601+0000 7fdf9034e640 1 -- 172.21.15.67:0/3680375913 wait complete. 2024-04-23T00:05:44.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.602+0000 7fdf9034e640 1 Processor -- start 2024-04-23T00:05:44.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.602+0000 7fdf9034e640 1 -- start start 2024-04-23T00:05:44.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.602+0000 7fdf9034e640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf880ffab0 0x7fdf88196c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:44.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.602+0000 7fdf9034e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf881971b0 0x7fdf88197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:44.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.602+0000 7fdf9034e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8819fce0 con 0x7fdf881971b0 2024-04-23T00:05:44.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.602+0000 7fdf9034e640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8819fe20 con 0x7fdf880ffab0 2024-04-23T00:05:44.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.602+0000 7fdf8d8e9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf881971b0 0x7fdf88197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:44.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.602+0000 7fdf8d8e9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf881971b0 0x7fdf88197610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38974/0 (socket says 172.21.15.67:38974) 2024-04-23T00:05:44.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.602+0000 7fdf8d8e9640 1 -- 172.21.15.67:0/48987690 learned_addr learned my addr 172.21.15.67:0/48987690 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:44.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.602+0000 7fdf8e0ea640 1 --2- 172.21.15.67:0/48987690 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf880ffab0 0x7fdf88196c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:44.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.602+0000 7fdf8d8e9640 1 -- 172.21.15.67:0/48987690 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf880ffab0 msgr2=0x7fdf88196c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:44.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.602+0000 7fdf8d8e9640 1 --2- 172.21.15.67:0/48987690 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf880ffab0 0x7fdf88196c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:44.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.602+0000 7fdf8d8e9640 1 -- 172.21.15.67:0/48987690 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf78009560 con 0x7fdf881971b0 2024-04-23T00:05:44.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.602+0000 7fdf8d8e9640 1 --2- 172.21.15.67:0/48987690 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf881971b0 0x7fdf88197610 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fdf78009230 tx=0x7fdf78004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:44.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.603+0000 7fdf7f7fe640 1 -- 172.21.15.67:0/48987690 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf780037c0 con 0x7fdf881971b0 2024-04-23T00:05:44.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.603+0000 7fdf7f7fe640 1 -- 172.21.15.67:0/48987690 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdf78004aa0 con 0x7fdf881971b0 2024-04-23T00:05:44.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.603+0000 7fdf9034e640 1 -- 172.21.15.67:0/48987690 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf8819ffc0 con 0x7fdf881971b0 2024-04-23T00:05:44.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.603+0000 7fdf7f7fe640 1 -- 172.21.15.67:0/48987690 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf78034440 con 0x7fdf881971b0 2024-04-23T00:05:44.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.603+0000 7fdf9034e640 1 -- 172.21.15.67:0/48987690 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf881a0440 con 0x7fdf881971b0 2024-04-23T00:05:44.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.604+0000 7fdf9034e640 1 -- 172.21.15.67:0/48987690 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf8804e040 con 0x7fdf881971b0 2024-04-23T00:05:44.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.604+0000 7fdf7f7fe640 1 -- 172.21.15.67:0/48987690 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdf78004f60 con 0x7fdf881971b0 2024-04-23T00:05:44.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.605+0000 7fdf7f7fe640 1 --2- 172.21.15.67:0/48987690 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdf7406ec30 0x7fdf740710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:44.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.605+0000 7fdf7f7fe640 1 -- 172.21.15.67:0/48987690 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fdf780b1310 con 0x7fdf881971b0 2024-04-23T00:05:44.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.605+0000 7fdf8e0ea640 1 --2- 172.21.15.67:0/48987690 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdf7406ec30 0x7fdf740710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:44.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.605+0000 7fdf8e0ea640 1 --2- 172.21.15.67:0/48987690 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdf7406ec30 0x7fdf740710f0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fdf8400abe0 tx=0x7fdf8400a2d0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:44.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.610+0000 7fdf7f7fe640 1 -- 172.21.15.67:0/48987690 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fdf78079fd0 con 0x7fdf881971b0 2024-04-23T00:05:44.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.837+0000 7fdf9034e640 1 -- 172.21.15.67:0/48987690 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 27, "format": "json"} v 0) v1 -- 0x7fdf881074f0 con 0x7fdf881971b0 2024-04-23T00:05:44.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.837+0000 7fdf7f7fe640 1 -- 172.21.15.67:0/48987690 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 27, "format": "json"}]=0 dumped fsmap epoch 27 v42) v1 ==== 107+0+4924 (secure 0 0 0) 0x7fdf7807d690 con 0x7fdf881971b0 2024-04-23T00:05:44.824 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:05:44.824 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34388,"name":"cephfs.smithi067.xyazmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6829/2016103304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2016103304},{"type":"v1","addr":"172.21.15.67:6829","nonce":2016103304}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25},{"gid":34396,"name":"cephfs.smithi067.aqlxpv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6827/789475804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":789475804},{"type":"v1","addr":"172.21.15.67:6827","nonce":789475804}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":44259,"name":"cephfs.smithi134.vuxbsj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6827/1827288036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":1827288036},{"type":"v1","addr":"172.21.15.134:6827","nonce":1827288036}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44267,"name":"cephfs.smithi134.owipad","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6825/3366499307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":3366499307},{"type":"v1","addr":"172.21.15.134:6825","nonce":3366499307}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T23:20:20.823261+0000","modified":"2024-04-22T23:32:27.027617+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T00:05:44.824 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 27 2024-04-23T00:05:44.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.839+0000 7fdf9034e640 1 -- 172.21.15.67:0/48987690 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdf7406ec30 msgr2=0x7fdf740710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:44.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.839+0000 7fdf9034e640 1 --2- 172.21.15.67:0/48987690 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdf7406ec30 0x7fdf740710f0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fdf8400abe0 tx=0x7fdf8400a2d0 comp rx=0 tx=0).stop 2024-04-23T00:05:44.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.839+0000 7fdf9034e640 1 -- 172.21.15.67:0/48987690 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf881971b0 msgr2=0x7fdf88197610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:44.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.839+0000 7fdf9034e640 1 --2- 172.21.15.67:0/48987690 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf881971b0 0x7fdf88197610 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fdf78009230 tx=0x7fdf78004270 comp rx=0 tx=0).stop 2024-04-23T00:05:44.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.839+0000 7fdf9034e640 1 -- 172.21.15.67:0/48987690 shutdown_connections 2024-04-23T00:05:44.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.839+0000 7fdf9034e640 1 --2- 172.21.15.67:0/48987690 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fdf7406ec30 0x7fdf740710f0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:44.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.839+0000 7fdf9034e640 1 --2- 172.21.15.67:0/48987690 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf881971b0 0x7fdf88197610 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:44.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.839+0000 7fdf9034e640 1 --2- 172.21.15.67:0/48987690 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdf880ffab0 0x7fdf88196c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:44.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.840+0000 7fdf9034e640 1 -- 172.21.15.67:0/48987690 >> 172.21.15.67:0/48987690 conn(0x7fdf88074250 msgr2=0x7fdf880fd3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:44.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.840+0000 7fdf9034e640 1 -- 172.21.15.67:0/48987690 shutdown_connections 2024-04-23T00:05:44.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.840+0000 7fdf9034e640 1 -- 172.21.15.67:0/48987690 wait complete. 2024-04-23T00:05:44.837 DEBUG:tasks.fs:max_mds not reduced in epoch 27 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-23T00:05:44.837 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 27 2024-04-23T00:05:44.837 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 28 2024-04-23T00:05:44.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.953+0000 7f7f469d3640 1 -- 172.21.15.67:0/3095511151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f400ffb50 msgr2=0x7f7f40103b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:44.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.953+0000 7f7f469d3640 1 --2- 172.21.15.67:0/3095511151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f400ffb50 0x7f7f40103b90 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f7f30009880 tx=0x7f7f3002bce0 comp rx=0 tx=0).stop 2024-04-23T00:05:44.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.953+0000 7f7f469d3640 1 -- 172.21.15.67:0/3095511151 shutdown_connections 2024-04-23T00:05:44.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.953+0000 7f7f469d3640 1 --2- 172.21.15.67:0/3095511151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f400ffb50 0x7f7f40103b90 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:44.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.953+0000 7f7f469d3640 1 --2- 172.21.15.67:0/3095511151 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7f400ff1a0 0x7f7f400ff580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:44.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.953+0000 7f7f469d3640 1 -- 172.21.15.67:0/3095511151 >> 172.21.15.67:0/3095511151 conn(0x7f7f400faa20 msgr2=0x7f7f400fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:44.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.953+0000 7f7f469d3640 1 -- 172.21.15.67:0/3095511151 shutdown_connections 2024-04-23T00:05:44.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.953+0000 7f7f469d3640 1 -- 172.21.15.67:0/3095511151 wait complete. 2024-04-23T00:05:44.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.954+0000 7f7f469d3640 1 Processor -- start 2024-04-23T00:05:44.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.954+0000 7f7f469d3640 1 -- start start 2024-04-23T00:05:44.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.954+0000 7f7f469d3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f400ff1a0 0x7f7f40195da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:44.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.954+0000 7f7f469d3640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7f401962e0 0x7f7f4019a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:44.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.954+0000 7f7f469d3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f401968e0 con 0x7f7f400ff1a0 2024-04-23T00:05:44.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.954+0000 7f7f469d3640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f40196a50 con 0x7f7f401962e0 2024-04-23T00:05:44.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.955+0000 7f7f3ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f400ff1a0 0x7f7f40195da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:44.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.955+0000 7f7f3ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f400ff1a0 0x7f7f40195da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38984/0 (socket says 172.21.15.67:38984) 2024-04-23T00:05:44.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.955+0000 7f7f3ffff640 1 -- 172.21.15.67:0/115147932 learned_addr learned my addr 172.21.15.67:0/115147932 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:44.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.955+0000 7f7f3f7fe640 1 --2- 172.21.15.67:0/115147932 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7f401962e0 0x7f7f4019a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:44.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.955+0000 7f7f3ffff640 1 -- 172.21.15.67:0/115147932 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7f401962e0 msgr2=0x7f7f4019a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:44.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.955+0000 7f7f3ffff640 1 --2- 172.21.15.67:0/115147932 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7f401962e0 0x7f7f4019a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:44.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.955+0000 7f7f3ffff640 1 -- 172.21.15.67:0/115147932 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f30009560 con 0x7f7f400ff1a0 2024-04-23T00:05:44.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.955+0000 7f7f3ffff640 1 --2- 172.21.15.67:0/115147932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f400ff1a0 0x7f7f40195da0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f7f2800c920 tx=0x7f7f2800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:44.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.955+0000 7f7f3d7fa640 1 -- 172.21.15.67:0/115147932 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f28007b80 con 0x7f7f400ff1a0 2024-04-23T00:05:44.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.955+0000 7f7f469d3640 1 -- 172.21.15.67:0/115147932 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f4019ad40 con 0x7f7f400ff1a0 2024-04-23T00:05:44.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.955+0000 7f7f3d7fa640 1 -- 172.21.15.67:0/115147932 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f280054e0 con 0x7f7f400ff1a0 2024-04-23T00:05:44.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.955+0000 7f7f3d7fa640 1 -- 172.21.15.67:0/115147932 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f2800f450 con 0x7f7f400ff1a0 2024-04-23T00:05:44.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.955+0000 7f7f469d3640 1 -- 172.21.15.67:0/115147932 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f4019b2a0 con 0x7f7f400ff1a0 2024-04-23T00:05:44.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.956+0000 7f7f469d3640 1 -- 172.21.15.67:0/115147932 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f4019b580 con 0x7f7f400ff1a0 2024-04-23T00:05:44.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.957+0000 7f7f3d7fa640 1 -- 172.21.15.67:0/115147932 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7f28004f60 con 0x7f7f400ff1a0 2024-04-23T00:05:44.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.957+0000 7f7f3d7fa640 1 --2- 172.21.15.67:0/115147932 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7f2c06ec30 0x7f7f2c0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:44.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.957+0000 7f7f3d7fa640 1 -- 172.21.15.67:0/115147932 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f7f2808ee80 con 0x7f7f400ff1a0 2024-04-23T00:05:44.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.957+0000 7f7f3f7fe640 1 --2- 172.21.15.67:0/115147932 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7f2c06ec30 0x7f7f2c0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:44.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.958+0000 7f7f3f7fe640 1 --2- 172.21.15.67:0/115147932 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7f2c06ec30 0x7f7f2c0710f0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f7f30006010 tx=0x7f7f30000bc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:44.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:44.962+0000 7f7f3d7fa640 1 -- 172.21.15.67:0/115147932 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7f28057ac0 con 0x7f7f400ff1a0 2024-04-23T00:05:45.030 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:44 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1910788864' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-04-23T00:05:45.030 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:44 smithi067 ceph-mon[112438]: pgmap v1185: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:05:45.030 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:44 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3259552340' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-04-23T00:05:45.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.187+0000 7f7f469d3640 1 -- 172.21.15.67:0/115147932 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 28, "format": "json"} v 0) v1 -- 0x7f7f4004e040 con 0x7f7f400ff1a0 2024-04-23T00:05:45.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.188+0000 7f7f3d7fa640 1 -- 172.21.15.67:0/115147932 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 28, "format": "json"}]=0 dumped fsmap epoch 28 v42) v1 ==== 107+0+4145 (secure 0 0 0) 0x7f7f2805b180 con 0x7f7f400ff1a0 2024-04-23T00:05:45.174 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:05:45.175 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34396,"name":"cephfs.smithi067.aqlxpv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6827/789475804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":789475804},{"type":"v1","addr":"172.21.15.67:6827","nonce":789475804}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":44259,"name":"cephfs.smithi134.vuxbsj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6827/1827288036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":1827288036},{"type":"v1","addr":"172.21.15.134:6827","nonce":1827288036}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44267,"name":"cephfs.smithi134.owipad","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6825/3366499307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":3366499307},{"type":"v1","addr":"172.21.15.134:6825","nonce":3366499307}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T23:20:20.823261+0000","modified":"2024-04-22T23:32:27.027617+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T00:05:45.175 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 28 2024-04-23T00:05:45.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.190+0000 7f7f469d3640 1 -- 172.21.15.67:0/115147932 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7f2c06ec30 msgr2=0x7f7f2c0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:45.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.190+0000 7f7f469d3640 1 --2- 172.21.15.67:0/115147932 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7f2c06ec30 0x7f7f2c0710f0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f7f30006010 tx=0x7f7f30000bc0 comp rx=0 tx=0).stop 2024-04-23T00:05:45.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.190+0000 7f7f469d3640 1 -- 172.21.15.67:0/115147932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f400ff1a0 msgr2=0x7f7f40195da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:45.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.190+0000 7f7f469d3640 1 --2- 172.21.15.67:0/115147932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f400ff1a0 0x7f7f40195da0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f7f2800c920 tx=0x7f7f2800cde0 comp rx=0 tx=0).stop 2024-04-23T00:05:45.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.190+0000 7f7f469d3640 1 -- 172.21.15.67:0/115147932 shutdown_connections 2024-04-23T00:05:45.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.190+0000 7f7f469d3640 1 --2- 172.21.15.67:0/115147932 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f7f2c06ec30 0x7f7f2c0710f0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:45.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.190+0000 7f7f469d3640 1 --2- 172.21.15.67:0/115147932 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7f401962e0 0x7f7f4019a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:45.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.190+0000 7f7f469d3640 1 --2- 172.21.15.67:0/115147932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f400ff1a0 0x7f7f40195da0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:45.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.190+0000 7f7f469d3640 1 -- 172.21.15.67:0/115147932 >> 172.21.15.67:0/115147932 conn(0x7f7f400faa20 msgr2=0x7f7f40103340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:45.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.190+0000 7f7f469d3640 1 -- 172.21.15.67:0/115147932 shutdown_connections 2024-04-23T00:05:45.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.190+0000 7f7f469d3640 1 -- 172.21.15.67:0/115147932 wait complete. 2024-04-23T00:05:45.187 DEBUG:tasks.fs:max_mds not reduced in epoch 28 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-23T00:05:45.187 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 28 2024-04-23T00:05:45.188 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 29 2024-04-23T00:05:45.230 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:44 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1910788864' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-04-23T00:05:45.230 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:44 smithi134 ceph-mon[85630]: pgmap v1185: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:05:45.230 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:44 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3259552340' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-04-23T00:05:45.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.305+0000 7fafce43a640 1 -- 172.21.15.67:0/3675345212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafc81002d0 msgr2=0x7fafc81006b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:45.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.305+0000 7fafce43a640 1 --2- 172.21.15.67:0/3675345212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafc81002d0 0x7fafc81006b0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fafb0009880 tx=0x7fafb002bce0 comp rx=0 tx=0).stop 2024-04-23T00:05:45.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.305+0000 7fafce43a640 1 -- 172.21.15.67:0/3675345212 shutdown_connections 2024-04-23T00:05:45.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.305+0000 7fafce43a640 1 --2- 172.21.15.67:0/3675345212 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fafc8100c80 0x7fafc8104cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:45.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.305+0000 7fafce43a640 1 --2- 172.21.15.67:0/3675345212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafc81002d0 0x7fafc81006b0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:45.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.305+0000 7fafce43a640 1 -- 172.21.15.67:0/3675345212 >> 172.21.15.67:0/3675345212 conn(0x7fafc80fbb50 msgr2=0x7fafc80fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:45.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.305+0000 7fafce43a640 1 -- 172.21.15.67:0/3675345212 shutdown_connections 2024-04-23T00:05:45.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.305+0000 7fafce43a640 1 -- 172.21.15.67:0/3675345212 wait complete. 2024-04-23T00:05:45.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.306+0000 7fafce43a640 1 Processor -- start 2024-04-23T00:05:45.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.306+0000 7fafce43a640 1 -- start start 2024-04-23T00:05:45.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.306+0000 7fafce43a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafc81002d0 0x7fafc8195df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:45.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.306+0000 7fafce43a640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fafc8196330 0x7fafc819a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:45.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.306+0000 7fafce43a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc8196930 con 0x7fafc81002d0 2024-04-23T00:05:45.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.306+0000 7fafce43a640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc8196aa0 con 0x7fafc8196330 2024-04-23T00:05:45.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.306+0000 7fafc7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafc81002d0 0x7fafc8195df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:45.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.306+0000 7fafc7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafc81002d0 0x7fafc8195df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38992/0 (socket says 172.21.15.67:38992) 2024-04-23T00:05:45.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.306+0000 7fafc7fff640 1 -- 172.21.15.67:0/147771394 learned_addr learned my addr 172.21.15.67:0/147771394 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:45.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.307+0000 7fafc77fe640 1 --2- 172.21.15.67:0/147771394 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fafc8196330 0x7fafc819a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:45.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.307+0000 7fafc7fff640 1 -- 172.21.15.67:0/147771394 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fafc8196330 msgr2=0x7fafc819a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:45.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.307+0000 7fafc7fff640 1 --2- 172.21.15.67:0/147771394 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fafc8196330 0x7fafc819a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:45.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.307+0000 7fafc7fff640 1 -- 172.21.15.67:0/147771394 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb0009560 con 0x7fafc81002d0 2024-04-23T00:05:45.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.307+0000 7fafc77fe640 1 --2- 172.21.15.67:0/147771394 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fafc8196330 0x7fafc819a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T00:05:45.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.307+0000 7fafc7fff640 1 --2- 172.21.15.67:0/147771394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafc81002d0 0x7fafc8195df0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fafb0036040 tx=0x7fafb0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:45.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.307+0000 7fafc57fa640 1 -- 172.21.15.67:0/147771394 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fafb00037c0 con 0x7fafc81002d0 2024-04-23T00:05:45.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.307+0000 7fafce43a640 1 -- 172.21.15.67:0/147771394 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafc819ad30 con 0x7fafc81002d0 2024-04-23T00:05:45.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.307+0000 7fafc57fa640 1 -- 172.21.15.67:0/147771394 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fafb0003de0 con 0x7fafc81002d0 2024-04-23T00:05:45.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.307+0000 7fafc57fa640 1 -- 172.21.15.67:0/147771394 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fafb0034440 con 0x7fafc81002d0 2024-04-23T00:05:45.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.307+0000 7fafce43a640 1 -- 172.21.15.67:0/147771394 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafc819b290 con 0x7fafc81002d0 2024-04-23T00:05:45.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.308+0000 7fafce43a640 1 -- 172.21.15.67:0/147771394 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafc804e040 con 0x7fafc81002d0 2024-04-23T00:05:45.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.309+0000 7fafc57fa640 1 -- 172.21.15.67:0/147771394 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fafb0003920 con 0x7fafc81002d0 2024-04-23T00:05:45.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.309+0000 7fafc57fa640 1 --2- 172.21.15.67:0/147771394 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fafb406ec30 0x7fafb40710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:45.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.309+0000 7fafc77fe640 1 --2- 172.21.15.67:0/147771394 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fafb406ec30 0x7fafb40710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:45.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.309+0000 7fafc57fa640 1 -- 172.21.15.67:0/147771394 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fafb00b1510 con 0x7fafc81002d0 2024-04-23T00:05:45.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.310+0000 7fafc77fe640 1 --2- 172.21.15.67:0/147771394 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fafb406ec30 0x7fafb40710f0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fafb8005fd0 tx=0x7fafb8005ea0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:45.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.314+0000 7fafc57fa640 1 -- 172.21.15.67:0/147771394 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fafb007a210 con 0x7fafc81002d0 2024-04-23T00:05:45.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.539+0000 7fafce43a640 1 -- 172.21.15.67:0/147771394 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 29, "format": "json"} v 0) v1 -- 0x7fafc819b5a0 con 0x7fafc81002d0 2024-04-23T00:05:45.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.540+0000 7fafc57fa640 1 -- 172.21.15.67:0/147771394 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 29, "format": "json"}]=0 dumped fsmap epoch 29 v42) v1 ==== 107+0+4989 (secure 0 0 0) 0x7fafb007d8d0 con 0x7fafc81002d0 2024-04-23T00:05:45.526 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:05:45.527 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34396,"name":"cephfs.smithi067.aqlxpv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6827/789475804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":789475804},{"type":"v1","addr":"172.21.15.67:6827","nonce":789475804}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":34424,"name":"cephfs.smithi067.xyazmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6829/927062647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":927062647},{"type":"v1","addr":"172.21.15.67:6829","nonce":927062647}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":44259,"name":"cephfs.smithi134.vuxbsj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6827/1827288036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":1827288036},{"type":"v1","addr":"172.21.15.134:6827","nonce":1827288036}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44267,"name":"cephfs.smithi134.owipad","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6825/3366499307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":3366499307},{"type":"v1","addr":"172.21.15.134:6825","nonce":3366499307}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T23:20:20.823261+0000","modified":"2024-04-22T23:32:27.027617+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T00:05:45.527 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 29 2024-04-23T00:05:45.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.542+0000 7fafce43a640 1 -- 172.21.15.67:0/147771394 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fafb406ec30 msgr2=0x7fafb40710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:45.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.542+0000 7fafce43a640 1 --2- 172.21.15.67:0/147771394 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fafb406ec30 0x7fafb40710f0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fafb8005fd0 tx=0x7fafb8005ea0 comp rx=0 tx=0).stop 2024-04-23T00:05:45.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.542+0000 7fafce43a640 1 -- 172.21.15.67:0/147771394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafc81002d0 msgr2=0x7fafc8195df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:45.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.542+0000 7fafce43a640 1 --2- 172.21.15.67:0/147771394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafc81002d0 0x7fafc8195df0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fafb0036040 tx=0x7fafb0004270 comp rx=0 tx=0).stop 2024-04-23T00:05:45.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.542+0000 7fafce43a640 1 -- 172.21.15.67:0/147771394 shutdown_connections 2024-04-23T00:05:45.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.542+0000 7fafce43a640 1 --2- 172.21.15.67:0/147771394 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fafb406ec30 0x7fafb40710f0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:45.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.542+0000 7fafce43a640 1 --2- 172.21.15.67:0/147771394 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fafc8196330 0x7fafc819a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:45.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.542+0000 7fafce43a640 1 --2- 172.21.15.67:0/147771394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafc81002d0 0x7fafc8195df0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:45.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.542+0000 7fafce43a640 1 -- 172.21.15.67:0/147771394 >> 172.21.15.67:0/147771394 conn(0x7fafc80fbb50 msgr2=0x7fafc80fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:45.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.542+0000 7fafce43a640 1 -- 172.21.15.67:0/147771394 shutdown_connections 2024-04-23T00:05:45.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.542+0000 7fafce43a640 1 -- 172.21.15.67:0/147771394 wait complete. 2024-04-23T00:05:45.540 DEBUG:tasks.fs:max_mds not reduced in epoch 29 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-23T00:05:45.540 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 29 2024-04-23T00:05:45.540 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 30 2024-04-23T00:05:45.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.655+0000 7f5297bc8640 1 -- 172.21.15.67:0/2073202898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5290101200 msgr2=0x7f52901015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:45.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.655+0000 7f5297bc8640 1 --2- 172.21.15.67:0/2073202898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5290101200 0x7f52901015e0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f528c009880 tx=0x7f528c02bd70 comp rx=0 tx=0).stop 2024-04-23T00:05:45.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.655+0000 7f5297bc8640 1 -- 172.21.15.67:0/2073202898 shutdown_connections 2024-04-23T00:05:45.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.655+0000 7f5297bc8640 1 --2- 172.21.15.67:0/2073202898 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5290101b20 0x7f529010a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:45.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.655+0000 7f5297bc8640 1 --2- 172.21.15.67:0/2073202898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5290101200 0x7f52901015e0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:45.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.655+0000 7f5297bc8640 1 -- 172.21.15.67:0/2073202898 >> 172.21.15.67:0/2073202898 conn(0x7f52900faa20 msgr2=0x7f52900fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:45.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.655+0000 7f5297bc8640 1 -- 172.21.15.67:0/2073202898 shutdown_connections 2024-04-23T00:05:45.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.655+0000 7f5297bc8640 1 -- 172.21.15.67:0/2073202898 wait complete. 2024-04-23T00:05:45.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.656+0000 7f5297bc8640 1 Processor -- start 2024-04-23T00:05:45.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.656+0000 7f5297bc8640 1 -- start start 2024-04-23T00:05:45.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.656+0000 7f5297bc8640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5290101200 0x7f52900fee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:45.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.656+0000 7f5297bc8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5290101b20 0x7f52900ff390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:45.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.656+0000 7f5297bc8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52900ff960 con 0x7f5290101b20 2024-04-23T00:05:45.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.656+0000 7f5297bc8640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52900ffaa0 con 0x7f5290101200 2024-04-23T00:05:45.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.656+0000 7f5295163640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5290101b20 0x7f52900ff390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:45.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.656+0000 7f5295163640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5290101b20 0x7f52900ff390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39020/0 (socket says 172.21.15.67:39020) 2024-04-23T00:05:45.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.656+0000 7f5295163640 1 -- 172.21.15.67:0/1507369538 learned_addr learned my addr 172.21.15.67:0/1507369538 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:45.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.656+0000 7f5295964640 1 --2- 172.21.15.67:0/1507369538 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5290101200 0x7f52900fee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:45.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.656+0000 7f5295163640 1 -- 172.21.15.67:0/1507369538 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5290101200 msgr2=0x7f52900fee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:45.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.656+0000 7f5295163640 1 --2- 172.21.15.67:0/1507369538 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5290101200 0x7f52900fee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:45.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.656+0000 7f5295163640 1 -- 172.21.15.67:0/1507369538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f528c009560 con 0x7f5290101b20 2024-04-23T00:05:45.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.657+0000 7f5295163640 1 --2- 172.21.15.67:0/1507369538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5290101b20 0x7f52900ff390 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f528000b370 tx=0x7f528000b830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:45.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.657+0000 7f5286ffd640 1 -- 172.21.15.67:0/1507369538 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5280011800 con 0x7f5290101b20 2024-04-23T00:05:45.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.657+0000 7f5286ffd640 1 -- 172.21.15.67:0/1507369538 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5280011e20 con 0x7f5290101b20 2024-04-23T00:05:45.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.657+0000 7f5297bc8640 1 -- 172.21.15.67:0/1507369538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52901a2010 con 0x7f5290101b20 2024-04-23T00:05:45.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.657+0000 7f5286ffd640 1 -- 172.21.15.67:0/1507369538 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f528000f2e0 con 0x7f5290101b20 2024-04-23T00:05:45.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.657+0000 7f5297bc8640 1 -- 172.21.15.67:0/1507369538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52901a2440 con 0x7f5290101b20 2024-04-23T00:05:45.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.658+0000 7f5297bc8640 1 -- 172.21.15.67:0/1507369538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f529004e040 con 0x7f5290101b20 2024-04-23T00:05:45.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.658+0000 7f5286ffd640 1 -- 172.21.15.67:0/1507369538 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5280010380 con 0x7f5290101b20 2024-04-23T00:05:45.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.659+0000 7f5286ffd640 1 --2- 172.21.15.67:0/1507369538 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f527c06ec30 0x7f527c0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:45.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.659+0000 7f5286ffd640 1 -- 172.21.15.67:0/1507369538 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f528008eb70 con 0x7f5290101b20 2024-04-23T00:05:45.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.659+0000 7f5295964640 1 --2- 172.21.15.67:0/1507369538 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f527c06ec30 0x7f527c0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:45.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.659+0000 7f5295964640 1 --2- 172.21.15.67:0/1507369538 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f527c06ec30 0x7f527c0710f0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f528c006010 tx=0x7f528c000bc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:45.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.663+0000 7f5286ffd640 1 -- 172.21.15.67:0/1507369538 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f5280057830 con 0x7f5290101b20 2024-04-23T00:05:45.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.892+0000 7f5297bc8640 1 -- 172.21.15.67:0/1507369538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 30, "format": "json"} v 0) v1 -- 0x7f5290107670 con 0x7f5290101b20 2024-04-23T00:05:45.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.892+0000 7f5286ffd640 1 -- 172.21.15.67:0/1507369538 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 30, "format": "json"}]=0 dumped fsmap epoch 30 v42) v1 ==== 107+0+4207 (secure 0 0 0) 0x7f528005aef0 con 0x7f5290101b20 2024-04-23T00:05:45.879 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:05:45.879 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34396,"name":"cephfs.smithi067.aqlxpv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6827/789475804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":789475804},{"type":"v1","addr":"172.21.15.67:6827","nonce":789475804}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":34424,"name":"cephfs.smithi067.xyazmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6829/927062647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":927062647},{"type":"v1","addr":"172.21.15.67:6829","nonce":927062647}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":44259,"name":"cephfs.smithi134.vuxbsj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6827/1827288036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":1827288036},{"type":"v1","addr":"172.21.15.134:6827","nonce":1827288036}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T23:20:20.823261+0000","modified":"2024-04-22T23:32:27.027617+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T00:05:45.879 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 30 2024-04-23T00:05:45.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.895+0000 7f5297bc8640 1 -- 172.21.15.67:0/1507369538 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f527c06ec30 msgr2=0x7f527c0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:45.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.895+0000 7f5297bc8640 1 --2- 172.21.15.67:0/1507369538 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f527c06ec30 0x7f527c0710f0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f528c006010 tx=0x7f528c000bc0 comp rx=0 tx=0).stop 2024-04-23T00:05:45.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.895+0000 7f5297bc8640 1 -- 172.21.15.67:0/1507369538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5290101b20 msgr2=0x7f52900ff390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:45.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.895+0000 7f5297bc8640 1 --2- 172.21.15.67:0/1507369538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5290101b20 0x7f52900ff390 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f528000b370 tx=0x7f528000b830 comp rx=0 tx=0).stop 2024-04-23T00:05:45.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.895+0000 7f5297bc8640 1 -- 172.21.15.67:0/1507369538 shutdown_connections 2024-04-23T00:05:45.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.895+0000 7f5297bc8640 1 --2- 172.21.15.67:0/1507369538 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f527c06ec30 0x7f527c0710f0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:45.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.895+0000 7f5297bc8640 1 --2- 172.21.15.67:0/1507369538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5290101b20 0x7f52900ff390 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:45.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.895+0000 7f5297bc8640 1 --2- 172.21.15.67:0/1507369538 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5290101200 0x7f52900fee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:45.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.895+0000 7f5297bc8640 1 -- 172.21.15.67:0/1507369538 >> 172.21.15.67:0/1507369538 conn(0x7f52900faa20 msgr2=0x7f5290104840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:45.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.895+0000 7f5297bc8640 1 -- 172.21.15.67:0/1507369538 shutdown_connections 2024-04-23T00:05:45.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:45.895+0000 7f5297bc8640 1 -- 172.21.15.67:0/1507369538 wait complete. 2024-04-23T00:05:45.891 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:45 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/48987690' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-04-23T00:05:45.891 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:45 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/115147932' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-04-23T00:05:45.891 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:45 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/147771394' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 29, "format": "json"}]: dispatch 2024-04-23T00:05:45.892 DEBUG:tasks.fs:max_mds not reduced in epoch 30 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-23T00:05:45.892 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 30 2024-04-23T00:05:45.892 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 31 2024-04-23T00:05:45.980 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:45 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/48987690' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-04-23T00:05:45.980 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:45 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/115147932' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-04-23T00:05:45.980 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:45 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/147771394' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 29, "format": "json"}]: dispatch 2024-04-23T00:05:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.008+0000 7f82a82ce640 1 -- 172.21.15.67:0/4035405477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82a0100c50 msgr2=0x7f82a0104c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.008+0000 7f82a82ce640 1 --2- 172.21.15.67:0/4035405477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82a0100c50 0x7f82a0104c90 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f8290009880 tx=0x7f829002bce0 comp rx=0 tx=0).stop 2024-04-23T00:05:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.008+0000 7f82a82ce640 1 -- 172.21.15.67:0/4035405477 shutdown_connections 2024-04-23T00:05:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.008+0000 7f82a82ce640 1 --2- 172.21.15.67:0/4035405477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82a0100c50 0x7f82a0104c90 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.008+0000 7f82a82ce640 1 --2- 172.21.15.67:0/4035405477 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f82a01002a0 0x7f82a0100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.008+0000 7f82a82ce640 1 -- 172.21.15.67:0/4035405477 >> 172.21.15.67:0/4035405477 conn(0x7f82a00fbb20 msgr2=0x7f82a00fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.008+0000 7f82a82ce640 1 -- 172.21.15.67:0/4035405477 shutdown_connections 2024-04-23T00:05:45.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.008+0000 7f82a82ce640 1 -- 172.21.15.67:0/4035405477 wait complete. 2024-04-23T00:05:45.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.008+0000 7f82a82ce640 1 Processor -- start 2024-04-23T00:05:45.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.008+0000 7f82a82ce640 1 -- start start 2024-04-23T00:05:45.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.009+0000 7f82a82ce640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82a01002a0 0x7f82a0195d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:45.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.009+0000 7f82a82ce640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f82a01962c0 0x7f82a019a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:45.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.009+0000 7f82a82ce640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a01968c0 con 0x7f82a01002a0 2024-04-23T00:05:45.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.009+0000 7f82a82ce640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a0196a30 con 0x7f82a01962c0 2024-04-23T00:05:45.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.009+0000 7f82a606a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82a01002a0 0x7f82a0195d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:45.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.009+0000 7f82a606a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82a01002a0 0x7f82a0195d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39040/0 (socket says 172.21.15.67:39040) 2024-04-23T00:05:45.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.009+0000 7f82a606a640 1 -- 172.21.15.67:0/745068370 learned_addr learned my addr 172.21.15.67:0/745068370 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:45.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.009+0000 7f82a5869640 1 --2- 172.21.15.67:0/745068370 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f82a01962c0 0x7f82a019a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:45.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.009+0000 7f82a606a640 1 -- 172.21.15.67:0/745068370 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f82a01962c0 msgr2=0x7f82a019a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:45.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.009+0000 7f82a606a640 1 --2- 172.21.15.67:0/745068370 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f82a01962c0 0x7f82a019a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:45.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.009+0000 7f82a606a640 1 -- 172.21.15.67:0/745068370 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8290009560 con 0x7f82a01002a0 2024-04-23T00:05:45.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.009+0000 7f82a606a640 1 --2- 172.21.15.67:0/745068370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82a01002a0 0x7f82a0195d80 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f829c00d720 tx=0x7f829c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:45.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.009+0000 7f82977fe640 1 -- 172.21.15.67:0/745068370 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f829c0051f0 con 0x7f82a01002a0 2024-04-23T00:05:45.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.009+0000 7f82977fe640 1 -- 172.21.15.67:0/745068370 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f829c009ce0 con 0x7f82a01002a0 2024-04-23T00:05:45.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.009+0000 7f82a82ce640 1 -- 172.21.15.67:0/745068370 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82a019ad20 con 0x7f82a01002a0 2024-04-23T00:05:45.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.009+0000 7f82977fe640 1 -- 172.21.15.67:0/745068370 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f829c010460 con 0x7f82a01002a0 2024-04-23T00:05:45.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.009+0000 7f82a82ce640 1 -- 172.21.15.67:0/745068370 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82a019b280 con 0x7f82a01002a0 2024-04-23T00:05:45.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.010+0000 7f82a82ce640 1 -- 172.21.15.67:0/745068370 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82a004e040 con 0x7f82a01002a0 2024-04-23T00:05:45.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.011+0000 7f82977fe640 1 -- 172.21.15.67:0/745068370 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f829c005350 con 0x7f82a01002a0 2024-04-23T00:05:45.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.011+0000 7f82977fe640 1 --2- 172.21.15.67:0/745068370 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f828c06ec30 0x7f828c0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:45.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.011+0000 7f82977fe640 1 -- 172.21.15.67:0/745068370 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f829c08ead0 con 0x7f82a01002a0 2024-04-23T00:05:45.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.011+0000 7f82a5869640 1 --2- 172.21.15.67:0/745068370 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f828c06ec30 0x7f828c0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:45.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.012+0000 7f82a5869640 1 --2- 172.21.15.67:0/745068370 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f828c06ec30 0x7f828c0710f0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f8290006010 tx=0x7f8290000bc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:46.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.016+0000 7f82977fe640 1 -- 172.21.15.67:0/745068370 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f829c057790 con 0x7f82a01002a0 2024-04-23T00:05:46.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.242+0000 7f82a82ce640 1 -- 172.21.15.67:0/745068370 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 31, "format": "json"} v 0) v1 -- 0x7f82a0197100 con 0x7f82a01002a0 2024-04-23T00:05:46.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.242+0000 7f82977fe640 1 -- 172.21.15.67:0/745068370 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 31, "format": "json"}]=0 dumped fsmap epoch 31 v42) v1 ==== 107+0+5057 (secure 0 0 0) 0x7f829c017090 con 0x7f82a01002a0 2024-04-23T00:05:46.231 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:05:46.231 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34396,"name":"cephfs.smithi067.aqlxpv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6827/789475804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":789475804},{"type":"v1","addr":"172.21.15.67:6827","nonce":789475804}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":34424,"name":"cephfs.smithi067.xyazmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6829/927062647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":927062647},{"type":"v1","addr":"172.21.15.67:6829","nonce":927062647}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":44259,"name":"cephfs.smithi134.vuxbsj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6827/1827288036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":1827288036},{"type":"v1","addr":"172.21.15.134:6827","nonce":1827288036}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44327,"name":"cephfs.smithi134.owipad","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6825/1473208408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":1473208408},{"type":"v1","addr":"172.21.15.134:6825","nonce":1473208408}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T23:20:20.823261+0000","modified":"2024-04-22T23:32:27.027617+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T00:05:46.233 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 31 2024-04-23T00:05:46.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.244+0000 7f82a82ce640 1 -- 172.21.15.67:0/745068370 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f828c06ec30 msgr2=0x7f828c0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:46.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.244+0000 7f82a82ce640 1 --2- 172.21.15.67:0/745068370 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f828c06ec30 0x7f828c0710f0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f8290006010 tx=0x7f8290000bc0 comp rx=0 tx=0).stop 2024-04-23T00:05:46.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.244+0000 7f82a82ce640 1 -- 172.21.15.67:0/745068370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82a01002a0 msgr2=0x7f82a0195d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:46.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.244+0000 7f82a82ce640 1 --2- 172.21.15.67:0/745068370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82a01002a0 0x7f82a0195d80 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f829c00d720 tx=0x7f829c00dbe0 comp rx=0 tx=0).stop 2024-04-23T00:05:46.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.244+0000 7f82a82ce640 1 -- 172.21.15.67:0/745068370 shutdown_connections 2024-04-23T00:05:46.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.244+0000 7f82a82ce640 1 --2- 172.21.15.67:0/745068370 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f828c06ec30 0x7f828c0710f0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:46.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.244+0000 7f82a82ce640 1 --2- 172.21.15.67:0/745068370 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f82a01962c0 0x7f82a019a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:46.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.244+0000 7f82a82ce640 1 --2- 172.21.15.67:0/745068370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82a01002a0 0x7f82a0195d80 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:46.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.244+0000 7f82a82ce640 1 -- 172.21.15.67:0/745068370 >> 172.21.15.67:0/745068370 conn(0x7f82a00fbb20 msgr2=0x7f82a00fd220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:46.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.244+0000 7f82a82ce640 1 -- 172.21.15.67:0/745068370 shutdown_connections 2024-04-23T00:05:46.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.244+0000 7f82a82ce640 1 -- 172.21.15.67:0/745068370 wait complete. 2024-04-23T00:05:46.242 DEBUG:tasks.fs:max_mds not reduced in epoch 31 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-23T00:05:46.242 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 31 2024-04-23T00:05:46.243 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 32 2024-04-23T00:05:46.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.358+0000 7f31eabb2640 1 -- 172.21.15.67:0/3417118006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31e4101200 msgr2=0x7f31e41015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:46.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.358+0000 7f31eabb2640 1 --2- 172.21.15.67:0/3417118006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31e4101200 0x7f31e41015e0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f31dc009c20 tx=0x7f31dc02c230 comp rx=0 tx=0).stop 2024-04-23T00:05:46.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.358+0000 7f31eabb2640 1 -- 172.21.15.67:0/3417118006 shutdown_connections 2024-04-23T00:05:46.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.358+0000 7f31eabb2640 1 --2- 172.21.15.67:0/3417118006 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f31e4101b20 0x7f31e410a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:46.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.358+0000 7f31eabb2640 1 --2- 172.21.15.67:0/3417118006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31e4101200 0x7f31e41015e0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:46.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.358+0000 7f31eabb2640 1 -- 172.21.15.67:0/3417118006 >> 172.21.15.67:0/3417118006 conn(0x7f31e40faa40 msgr2=0x7f31e40fce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:46.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.358+0000 7f31eabb2640 1 -- 172.21.15.67:0/3417118006 shutdown_connections 2024-04-23T00:05:46.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.358+0000 7f31eabb2640 1 -- 172.21.15.67:0/3417118006 wait complete. 2024-04-23T00:05:46.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.358+0000 7f31eabb2640 1 Processor -- start 2024-04-23T00:05:46.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.358+0000 7f31eabb2640 1 -- start start 2024-04-23T00:05:46.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.359+0000 7f31eabb2640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f31e4101200 0x7f31e4195d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:46.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.359+0000 7f31eabb2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31e4101b20 0x7f31e4196280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:46.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.359+0000 7f31eabb2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e4196910 con 0x7f31e4101b20 2024-04-23T00:05:46.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.359+0000 7f31eabb2640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e419a6e0 con 0x7f31e4101200 2024-04-23T00:05:46.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.359+0000 7f31e3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31e4101b20 0x7f31e4196280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:46.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.359+0000 7f31e3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31e4101b20 0x7f31e4196280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39066/0 (socket says 172.21.15.67:39066) 2024-04-23T00:05:46.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.359+0000 7f31e3fff640 1 -- 172.21.15.67:0/3511701670 learned_addr learned my addr 172.21.15.67:0/3511701670 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:46.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.359+0000 7f31e894e640 1 --2- 172.21.15.67:0/3511701670 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f31e4101200 0x7f31e4195d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:46.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.359+0000 7f31e3fff640 1 -- 172.21.15.67:0/3511701670 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f31e4101200 msgr2=0x7f31e4195d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:46.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.359+0000 7f31e3fff640 1 --2- 172.21.15.67:0/3511701670 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f31e4101200 0x7f31e4195d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:46.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.359+0000 7f31e3fff640 1 -- 172.21.15.67:0/3511701670 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31dc009900 con 0x7f31e4101b20 2024-04-23T00:05:46.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.359+0000 7f31e3fff640 1 --2- 172.21.15.67:0/3511701670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31e4101b20 0x7f31e4196280 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f31cc00c8f0 tx=0x7f31cc00cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:46.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.359+0000 7f31e1ffb640 1 -- 172.21.15.67:0/3511701670 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31cc007b80 con 0x7f31e4101b20 2024-04-23T00:05:46.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.359+0000 7f31e1ffb640 1 -- 172.21.15.67:0/3511701670 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f31cc00ae50 con 0x7f31e4101b20 2024-04-23T00:05:46.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.359+0000 7f31eabb2640 1 -- 172.21.15.67:0/3511701670 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31e419a9c0 con 0x7f31e4101b20 2024-04-23T00:05:46.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.359+0000 7f31e1ffb640 1 -- 172.21.15.67:0/3511701670 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31cc004e50 con 0x7f31e4101b20 2024-04-23T00:05:46.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.360+0000 7f31eabb2640 1 -- 172.21.15.67:0/3511701670 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31e419af20 con 0x7f31e4101b20 2024-04-23T00:05:46.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.361+0000 7f31eabb2640 1 -- 172.21.15.67:0/3511701670 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31e404e040 con 0x7f31e4101b20 2024-04-23T00:05:46.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.361+0000 7f31e1ffb640 1 -- 172.21.15.67:0/3511701670 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f31cc004ff0 con 0x7f31e4101b20 2024-04-23T00:05:46.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.361+0000 7f31e1ffb640 1 --2- 172.21.15.67:0/3511701670 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f31d006ebe0 0x7f31d00710a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:46.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.362+0000 7f31e1ffb640 1 -- 172.21.15.67:0/3511701670 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f31cc08eda0 con 0x7f31e4101b20 2024-04-23T00:05:46.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.362+0000 7f31e894e640 1 --2- 172.21.15.67:0/3511701670 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f31d006ebe0 0x7f31d00710a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:46.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.362+0000 7f31e894e640 1 --2- 172.21.15.67:0/3511701670 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f31d006ebe0 0x7f31d00710a0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f31dc006010 tx=0x7f31dc0093a0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:46.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.366+0000 7f31e1ffb640 1 -- 172.21.15.67:0/3511701670 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f31cc0579e0 con 0x7f31e4101b20 2024-04-23T00:05:46.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.598+0000 7f31eabb2640 1 -- 172.21.15.67:0/3511701670 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 32, "format": "json"} v 0) v1 -- 0x7f31e4107670 con 0x7f31e4101b20 2024-04-23T00:05:46.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.598+0000 7f31e1ffb640 1 -- 172.21.15.67:0/3511701670 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 32, "format": "json"}]=0 dumped fsmap epoch 32 v42) v1 ==== 107+0+4275 (secure 0 0 0) 0x7f31cc05b0a0 con 0x7f31e4101b20 2024-04-23T00:05:46.585 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:05:46.585 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34396,"name":"cephfs.smithi067.aqlxpv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6827/789475804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":789475804},{"type":"v1","addr":"172.21.15.67:6827","nonce":789475804}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":34424,"name":"cephfs.smithi067.xyazmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6829/927062647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":927062647},{"type":"v1","addr":"172.21.15.67:6829","nonce":927062647}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":44327,"name":"cephfs.smithi134.owipad","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6825/1473208408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":1473208408},{"type":"v1","addr":"172.21.15.134:6825","nonce":1473208408}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T23:20:20.823261+0000","modified":"2024-04-22T23:32:27.027617+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T00:05:46.585 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 32 2024-04-23T00:05:46.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.600+0000 7f31eabb2640 1 -- 172.21.15.67:0/3511701670 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f31d006ebe0 msgr2=0x7f31d00710a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:46.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.600+0000 7f31eabb2640 1 --2- 172.21.15.67:0/3511701670 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f31d006ebe0 0x7f31d00710a0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f31dc006010 tx=0x7f31dc0093a0 comp rx=0 tx=0).stop 2024-04-23T00:05:46.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.600+0000 7f31eabb2640 1 -- 172.21.15.67:0/3511701670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31e4101b20 msgr2=0x7f31e4196280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:46.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.600+0000 7f31eabb2640 1 --2- 172.21.15.67:0/3511701670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31e4101b20 0x7f31e4196280 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f31cc00c8f0 tx=0x7f31cc00cdb0 comp rx=0 tx=0).stop 2024-04-23T00:05:46.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.600+0000 7f31eabb2640 1 -- 172.21.15.67:0/3511701670 shutdown_connections 2024-04-23T00:05:46.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.600+0000 7f31eabb2640 1 --2- 172.21.15.67:0/3511701670 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f31d006ebe0 0x7f31d00710a0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:46.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.600+0000 7f31eabb2640 1 --2- 172.21.15.67:0/3511701670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31e4101b20 0x7f31e4196280 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:46.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.600+0000 7f31eabb2640 1 --2- 172.21.15.67:0/3511701670 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f31e4101200 0x7f31e4195d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:46.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.600+0000 7f31eabb2640 1 -- 172.21.15.67:0/3511701670 >> 172.21.15.67:0/3511701670 conn(0x7f31e40faa40 msgr2=0x7f31e4104840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:46.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.601+0000 7f31eabb2640 1 -- 172.21.15.67:0/3511701670 shutdown_connections 2024-04-23T00:05:46.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.601+0000 7f31eabb2640 1 -- 172.21.15.67:0/3511701670 wait complete. 2024-04-23T00:05:46.598 DEBUG:tasks.fs:max_mds not reduced in epoch 32 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-23T00:05:46.598 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 32 2024-04-23T00:05:46.598 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 33 2024-04-23T00:05:46.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.715+0000 7fd74c258640 1 -- 172.21.15.67:0/1825884367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd744106070 msgr2=0x7fd744106450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:46.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.715+0000 7fd74c258640 1 --2- 172.21.15.67:0/1825884367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd744106070 0x7fd744106450 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fd740009880 tx=0x7fd74002bce0 comp rx=0 tx=0).stop 2024-04-23T00:05:46.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.715+0000 7fd74c258640 1 -- 172.21.15.67:0/1825884367 shutdown_connections 2024-04-23T00:05:46.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.715+0000 7fd74c258640 1 --2- 172.21.15.67:0/1825884367 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd7440fe730 0x7fd7440feb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:46.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.715+0000 7fd74c258640 1 --2- 172.21.15.67:0/1825884367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd744106070 0x7fd744106450 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:46.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.715+0000 7fd74c258640 1 -- 172.21.15.67:0/1825884367 >> 172.21.15.67:0/1825884367 conn(0x7fd744074250 msgr2=0x7fd744074660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:46.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.715+0000 7fd74c258640 1 -- 172.21.15.67:0/1825884367 shutdown_connections 2024-04-23T00:05:46.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.715+0000 7fd74c258640 1 -- 172.21.15.67:0/1825884367 wait complete. 2024-04-23T00:05:46.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.715+0000 7fd74c258640 1 Processor -- start 2024-04-23T00:05:46.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.716+0000 7fd74c258640 1 -- start start 2024-04-23T00:05:46.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.716+0000 7fd74c258640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7440fe730 0x7fd744195d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:46.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.716+0000 7fd74c258640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd744106070 0x7fd744196280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:46.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.716+0000 7fd74c258640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd744196910 con 0x7fd7440fe730 2024-04-23T00:05:46.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.716+0000 7fd74c258640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74419a6e0 con 0x7fd744106070 2024-04-23T00:05:46.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.716+0000 7fd749ff4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7440fe730 0x7fd744195d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:46.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.716+0000 7fd749ff4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7440fe730 0x7fd744195d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39086/0 (socket says 172.21.15.67:39086) 2024-04-23T00:05:46.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.716+0000 7fd749ff4640 1 -- 172.21.15.67:0/276321109 learned_addr learned my addr 172.21.15.67:0/276321109 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:46.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.716+0000 7fd749ff4640 1 -- 172.21.15.67:0/276321109 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd744106070 msgr2=0x7fd744196280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:46.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.716+0000 7fd7497f3640 1 --2- 172.21.15.67:0/276321109 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd744106070 0x7fd744196280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:46.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.716+0000 7fd749ff4640 1 --2- 172.21.15.67:0/276321109 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd744106070 0x7fd744196280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:46.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.716+0000 7fd749ff4640 1 -- 172.21.15.67:0/276321109 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd740009560 con 0x7fd7440fe730 2024-04-23T00:05:46.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.716+0000 7fd749ff4640 1 --2- 172.21.15.67:0/276321109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7440fe730 0x7fd744195d40 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fd740000c00 tx=0x7fd74002c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:46.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.716+0000 7fd73affd640 1 -- 172.21.15.67:0/276321109 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd740004520 con 0x7fd7440fe730 2024-04-23T00:05:46.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.716+0000 7fd73affd640 1 -- 172.21.15.67:0/276321109 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd740003ab0 con 0x7fd7440fe730 2024-04-23T00:05:46.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.717+0000 7fd74c258640 1 -- 172.21.15.67:0/276321109 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd74419a960 con 0x7fd7440fe730 2024-04-23T00:05:46.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.717+0000 7fd73affd640 1 -- 172.21.15.67:0/276321109 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd740034440 con 0x7fd7440fe730 2024-04-23T00:05:46.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.717+0000 7fd74c258640 1 -- 172.21.15.67:0/276321109 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd74419ae60 con 0x7fd7440fe730 2024-04-23T00:05:46.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.718+0000 7fd74c258640 1 -- 172.21.15.67:0/276321109 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd74404e040 con 0x7fd7440fe730 2024-04-23T00:05:46.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.718+0000 7fd73affd640 1 -- 172.21.15.67:0/276321109 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd740003c20 con 0x7fd7440fe730 2024-04-23T00:05:46.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.718+0000 7fd73affd640 1 --2- 172.21.15.67:0/276321109 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd73006ec30 0x7fd7300710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:46.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.719+0000 7fd73affd640 1 -- 172.21.15.67:0/276321109 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fd7400b1380 con 0x7fd7440fe730 2024-04-23T00:05:46.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.719+0000 7fd7497f3640 1 --2- 172.21.15.67:0/276321109 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd73006ec30 0x7fd7300710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:46.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.719+0000 7fd7497f3640 1 --2- 172.21.15.67:0/276321109 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd73006ec30 0x7fd7300710f0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fd734005fd0 tx=0x7fd734005850 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:46.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.723+0000 7fd73affd640 1 -- 172.21.15.67:0/276321109 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fd740079fc0 con 0x7fd7440fe730 2024-04-23T00:05:46.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.952+0000 7fd74c258640 1 -- 172.21.15.67:0/276321109 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 33, "format": "json"} v 0) v1 -- 0x7fd744108580 con 0x7fd7440fe730 2024-04-23T00:05:46.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.953+0000 7fd73affd640 1 -- 172.21.15.67:0/276321109 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 33, "format": "json"}]=0 dumped fsmap epoch 33 v42) v1 ==== 107+0+5125 (secure 0 0 0) 0x7fd74007d680 con 0x7fd7440fe730 2024-04-23T00:05:46.939 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:05:46.939 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34396,"name":"cephfs.smithi067.aqlxpv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6827/789475804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":789475804},{"type":"v1","addr":"172.21.15.67:6827","nonce":789475804}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":34424,"name":"cephfs.smithi067.xyazmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6829/927062647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":927062647},{"type":"v1","addr":"172.21.15.67:6829","nonce":927062647}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":44327,"name":"cephfs.smithi134.owipad","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6825/1473208408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":1473208408},{"type":"v1","addr":"172.21.15.134:6825","nonce":1473208408}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":44851,"name":"cephfs.smithi134.vuxbsj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6827/1849648622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":1849648622},{"type":"v1","addr":"172.21.15.134:6827","nonce":1849648622}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T23:20:20.823261+0000","modified":"2024-04-22T23:32:27.027617+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T00:05:46.940 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 33 2024-04-23T00:05:46.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.955+0000 7fd74c258640 1 -- 172.21.15.67:0/276321109 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd73006ec30 msgr2=0x7fd7300710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:46.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.955+0000 7fd74c258640 1 --2- 172.21.15.67:0/276321109 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd73006ec30 0x7fd7300710f0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fd734005fd0 tx=0x7fd734005850 comp rx=0 tx=0).stop 2024-04-23T00:05:46.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.955+0000 7fd74c258640 1 -- 172.21.15.67:0/276321109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7440fe730 msgr2=0x7fd744195d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:46.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.955+0000 7fd74c258640 1 --2- 172.21.15.67:0/276321109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7440fe730 0x7fd744195d40 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fd740000c00 tx=0x7fd74002c850 comp rx=0 tx=0).stop 2024-04-23T00:05:46.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.955+0000 7fd74c258640 1 -- 172.21.15.67:0/276321109 shutdown_connections 2024-04-23T00:05:46.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.955+0000 7fd74c258640 1 --2- 172.21.15.67:0/276321109 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd73006ec30 0x7fd7300710f0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:46.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.955+0000 7fd74c258640 1 --2- 172.21.15.67:0/276321109 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd744106070 0x7fd744196280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:46.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.955+0000 7fd74c258640 1 --2- 172.21.15.67:0/276321109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7440fe730 0x7fd744195d40 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:46.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.955+0000 7fd74c258640 1 -- 172.21.15.67:0/276321109 >> 172.21.15.67:0/276321109 conn(0x7fd744074250 msgr2=0x7fd7440fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:46.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.955+0000 7fd74c258640 1 -- 172.21.15.67:0/276321109 shutdown_connections 2024-04-23T00:05:46.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:46.955+0000 7fd74c258640 1 -- 172.21.15.67:0/276321109 wait complete. 2024-04-23T00:05:46.951 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:46 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1507369538' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 30, "format": "json"}]: dispatch 2024-04-23T00:05:46.951 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:46 smithi067 ceph-mon[112438]: pgmap v1186: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:05:46.951 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:46 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/745068370' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 31, "format": "json"}]: dispatch 2024-04-23T00:05:46.952 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:46 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3511701670' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 32, "format": "json"}]: dispatch 2024-04-23T00:05:46.953 DEBUG:tasks.fs:max_mds not reduced in epoch 33 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-23T00:05:46.953 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 33 2024-04-23T00:05:46.953 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 34 2024-04-23T00:05:47.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.068+0000 7f874748f640 1 -- 172.21.15.67:0/3362403479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8740100c50 msgr2=0x7f8740104c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:47.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.068+0000 7f874748f640 1 --2- 172.21.15.67:0/3362403479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8740100c50 0x7f8740104c90 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f8730009880 tx=0x7f873002bce0 comp rx=0 tx=0).stop 2024-04-23T00:05:47.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.068+0000 7f874748f640 1 -- 172.21.15.67:0/3362403479 shutdown_connections 2024-04-23T00:05:47.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.068+0000 7f874748f640 1 --2- 172.21.15.67:0/3362403479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8740100c50 0x7f8740104c90 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:47.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.068+0000 7f874748f640 1 --2- 172.21.15.67:0/3362403479 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f87401002a0 0x7f8740100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:47.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.068+0000 7f874748f640 1 -- 172.21.15.67:0/3362403479 >> 172.21.15.67:0/3362403479 conn(0x7f87400fbb20 msgr2=0x7f87400fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:47.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.068+0000 7f874748f640 1 -- 172.21.15.67:0/3362403479 shutdown_connections 2024-04-23T00:05:47.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.068+0000 7f874748f640 1 -- 172.21.15.67:0/3362403479 wait complete. 2024-04-23T00:05:47.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.069+0000 7f874748f640 1 Processor -- start 2024-04-23T00:05:47.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.069+0000 7f874748f640 1 -- start start 2024-04-23T00:05:47.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.069+0000 7f874748f640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f87401002a0 0x7f8740195d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:47.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.069+0000 7f874748f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87401962c0 0x7f874019a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:47.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.069+0000 7f874748f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87401968c0 con 0x7f87401962c0 2024-04-23T00:05:47.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.069+0000 7f874748f640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8740196a30 con 0x7f87401002a0 2024-04-23T00:05:47.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.069+0000 7f8744a2a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87401962c0 0x7f874019a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:47.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.069+0000 7f8744a2a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87401962c0 0x7f874019a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39108/0 (socket says 172.21.15.67:39108) 2024-04-23T00:05:47.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.069+0000 7f8744a2a640 1 -- 172.21.15.67:0/2312999562 learned_addr learned my addr 172.21.15.67:0/2312999562 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:47.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.069+0000 7f8744a2a640 1 -- 172.21.15.67:0/2312999562 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f87401002a0 msgr2=0x7f8740195d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:47.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.069+0000 7f8744a2a640 1 --2- 172.21.15.67:0/2312999562 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f87401002a0 0x7f8740195d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:47.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.069+0000 7f8744a2a640 1 -- 172.21.15.67:0/2312999562 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8730009560 con 0x7f87401962c0 2024-04-23T00:05:47.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.070+0000 7f8744a2a640 1 --2- 172.21.15.67:0/2312999562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87401962c0 0x7f874019a720 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f8730036040 tx=0x7f873002c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:47.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.070+0000 7f87367fc640 1 -- 172.21.15.67:0/2312999562 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87300035f0 con 0x7f87401962c0 2024-04-23T00:05:47.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.070+0000 7f874748f640 1 -- 172.21.15.67:0/2312999562 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f874019acc0 con 0x7f87401962c0 2024-04-23T00:05:47.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.070+0000 7f87367fc640 1 -- 172.21.15.67:0/2312999562 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8730003c10 con 0x7f87401962c0 2024-04-23T00:05:47.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.070+0000 7f87367fc640 1 -- 172.21.15.67:0/2312999562 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8730034440 con 0x7f87401962c0 2024-04-23T00:05:47.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.070+0000 7f874748f640 1 -- 172.21.15.67:0/2312999562 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f874019b220 con 0x7f87401962c0 2024-04-23T00:05:47.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.071+0000 7f874748f640 1 -- 172.21.15.67:0/2312999562 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f874004e040 con 0x7f87401962c0 2024-04-23T00:05:47.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.071+0000 7f87367fc640 1 -- 172.21.15.67:0/2312999562 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8730003750 con 0x7f87401962c0 2024-04-23T00:05:47.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.072+0000 7f87367fc640 1 --2- 172.21.15.67:0/2312999562 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f872c06ec30 0x7f872c0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:47.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.072+0000 7f87367fc640 1 -- 172.21.15.67:0/2312999562 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f87300b1390 con 0x7f87401962c0 2024-04-23T00:05:47.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.072+0000 7f874522b640 1 --2- 172.21.15.67:0/2312999562 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f872c06ec30 0x7f872c0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:47.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.072+0000 7f874522b640 1 --2- 172.21.15.67:0/2312999562 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f872c06ec30 0x7f872c0710f0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f873c005fd0 tx=0x7f873c005850 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:47.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.076+0000 7f87367fc640 1 -- 172.21.15.67:0/2312999562 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f8730079fd0 con 0x7f87401962c0 2024-04-23T00:05:47.229 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:46 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1507369538' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 30, "format": "json"}]: dispatch 2024-04-23T00:05:47.230 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:46 smithi134 ceph-mon[85630]: pgmap v1186: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:05:47.230 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:46 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/745068370' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 31, "format": "json"}]: dispatch 2024-04-23T00:05:47.230 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:46 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3511701670' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 32, "format": "json"}]: dispatch 2024-04-23T00:05:47.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.301+0000 7f874748f640 1 -- 172.21.15.67:0/2312999562 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 34, "format": "json"} v 0) v1 -- 0x7f87401085a0 con 0x7f87401962c0 2024-04-23T00:05:47.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.302+0000 7f87367fc640 1 -- 172.21.15.67:0/2312999562 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 34, "format": "json"}]=0 dumped fsmap epoch 34 v42) v1 ==== 107+0+5124 (secure 0 0 0) 0x7f873009ce60 con 0x7f87401962c0 2024-04-23T00:05:47.289 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:05:47.289 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34396,"name":"cephfs.smithi067.aqlxpv","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6827/789475804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":789475804},{"type":"v1","addr":"172.21.15.67:6827","nonce":789475804}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":34424,"name":"cephfs.smithi067.xyazmh","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6829/927062647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":927062647},{"type":"v1","addr":"172.21.15.67:6829","nonce":927062647}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":44327,"name":"cephfs.smithi134.owipad","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6825/1473208408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":1473208408},{"type":"v1","addr":"172.21.15.134:6825","nonce":1473208408}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":44851,"name":"cephfs.smithi134.vuxbsj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6827/1849648622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":1849648622},{"type":"v1","addr":"172.21.15.134:6827","nonce":1849648622}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T23:20:20.823261+0000","modified":"2024-04-23T00:03:26.072276+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T00:05:47.289 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 34 2024-04-23T00:05:47.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.304+0000 7f874748f640 1 -- 172.21.15.67:0/2312999562 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f872c06ec30 msgr2=0x7f872c0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:47.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.304+0000 7f874748f640 1 --2- 172.21.15.67:0/2312999562 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f872c06ec30 0x7f872c0710f0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f873c005fd0 tx=0x7f873c005850 comp rx=0 tx=0).stop 2024-04-23T00:05:47.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.304+0000 7f874748f640 1 -- 172.21.15.67:0/2312999562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87401962c0 msgr2=0x7f874019a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:47.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.304+0000 7f874748f640 1 --2- 172.21.15.67:0/2312999562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87401962c0 0x7f874019a720 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f8730036040 tx=0x7f873002c850 comp rx=0 tx=0).stop 2024-04-23T00:05:47.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.304+0000 7f874748f640 1 -- 172.21.15.67:0/2312999562 shutdown_connections 2024-04-23T00:05:47.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.304+0000 7f874748f640 1 --2- 172.21.15.67:0/2312999562 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f872c06ec30 0x7f872c0710f0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:47.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.304+0000 7f874748f640 1 --2- 172.21.15.67:0/2312999562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87401962c0 0x7f874019a720 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:47.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.304+0000 7f874748f640 1 --2- 172.21.15.67:0/2312999562 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f87401002a0 0x7f8740195d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:47.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.304+0000 7f874748f640 1 -- 172.21.15.67:0/2312999562 >> 172.21.15.67:0/2312999562 conn(0x7f87400fbb20 msgr2=0x7f87400fd220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:47.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.304+0000 7f874748f640 1 -- 172.21.15.67:0/2312999562 shutdown_connections 2024-04-23T00:05:47.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.304+0000 7f874748f640 1 -- 172.21.15.67:0/2312999562 wait complete. 2024-04-23T00:05:47.302 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 34 2024-04-23T00:05:47.302 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 35 2024-04-23T00:05:47.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.417+0000 7f7675458640 1 -- 172.21.15.67:0/2515911479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76700ffc10 msgr2=0x7f7670103b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:47.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.417+0000 7f7675458640 1 --2- 172.21.15.67:0/2515911479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76700ffc10 0x7f7670103b70 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f7664009880 tx=0x7f766402bce0 comp rx=0 tx=0).stop 2024-04-23T00:05:47.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.418+0000 7f7675458640 1 -- 172.21.15.67:0/2515911479 shutdown_connections 2024-04-23T00:05:47.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.418+0000 7f7675458640 1 --2- 172.21.15.67:0/2515911479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76700ffc10 0x7f7670103b70 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:47.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.418+0000 7f7675458640 1 --2- 172.21.15.67:0/2515911479 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f76700ff260 0x7f76700ff640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:47.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.418+0000 7f7675458640 1 -- 172.21.15.67:0/2515911479 >> 172.21.15.67:0/2515911479 conn(0x7f7670074a50 msgr2=0x7f7670074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:47.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.418+0000 7f7675458640 1 -- 172.21.15.67:0/2515911479 shutdown_connections 2024-04-23T00:05:47.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.418+0000 7f7675458640 1 -- 172.21.15.67:0/2515911479 wait complete. 2024-04-23T00:05:47.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.418+0000 7f7675458640 1 Processor -- start 2024-04-23T00:05:47.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.418+0000 7f7675458640 1 -- start start 2024-04-23T00:05:47.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.419+0000 7f7675458640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f76700ff260 0x7f7670197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:47.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.419+0000 7f7675458640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76701984a0 0x7f767019c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:47.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.419+0000 7f7675458640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7670198aa0 con 0x7f76701984a0 2024-04-23T00:05:47.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.419+0000 7f7675458640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7670198c10 con 0x7f76700ff260 2024-04-23T00:05:47.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.419+0000 7f766e7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76701984a0 0x7f767019c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:47.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.419+0000 7f766e7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76701984a0 0x7f767019c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39118/0 (socket says 172.21.15.67:39118) 2024-04-23T00:05:47.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.419+0000 7f766e7fc640 1 -- 172.21.15.67:0/794349362 learned_addr learned my addr 172.21.15.67:0/794349362 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:47.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.419+0000 7f766effd640 1 --2- 172.21.15.67:0/794349362 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f76700ff260 0x7f7670197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:47.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.419+0000 7f766e7fc640 1 -- 172.21.15.67:0/794349362 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f76700ff260 msgr2=0x7f7670197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:47.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.419+0000 7f766e7fc640 1 --2- 172.21.15.67:0/794349362 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f76700ff260 0x7f7670197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:47.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.419+0000 7f766e7fc640 1 -- 172.21.15.67:0/794349362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7664009560 con 0x7f76701984a0 2024-04-23T00:05:47.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.419+0000 7f766e7fc640 1 --2- 172.21.15.67:0/794349362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76701984a0 0x7f767019c900 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f7664036040 tx=0x7f766402c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:47.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.419+0000 7f7657fff640 1 -- 172.21.15.67:0/794349362 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76640035f0 con 0x7f76701984a0 2024-04-23T00:05:47.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.419+0000 7f7675458640 1 -- 172.21.15.67:0/794349362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f767019cea0 con 0x7f76701984a0 2024-04-23T00:05:47.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.419+0000 7f7657fff640 1 -- 172.21.15.67:0/794349362 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7664003c10 con 0x7f76701984a0 2024-04-23T00:05:47.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.419+0000 7f7657fff640 1 -- 172.21.15.67:0/794349362 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7664034440 con 0x7f76701984a0 2024-04-23T00:05:47.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.419+0000 7f7675458640 1 -- 172.21.15.67:0/794349362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f767019d400 con 0x7f76701984a0 2024-04-23T00:05:47.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.420+0000 7f7675458640 1 -- 172.21.15.67:0/794349362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f767019d730 con 0x7f76701984a0 2024-04-23T00:05:47.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.421+0000 7f7657fff640 1 -- 172.21.15.67:0/794349362 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7664003750 con 0x7f76701984a0 2024-04-23T00:05:47.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.421+0000 7f7657fff640 1 --2- 172.21.15.67:0/794349362 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f765806ec30 0x7f76580710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:47.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.422+0000 7f7657fff640 1 -- 172.21.15.67:0/794349362 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f76640b1390 con 0x7f76701984a0 2024-04-23T00:05:47.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.422+0000 7f766effd640 1 --2- 172.21.15.67:0/794349362 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f765806ec30 0x7f76580710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:47.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.422+0000 7f766effd640 1 --2- 172.21.15.67:0/794349362 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f765806ec30 0x7f76580710f0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f7668005fd0 tx=0x7f7668009870 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:47.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.426+0000 7f7657fff640 1 -- 172.21.15.67:0/794349362 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f7664079fd0 con 0x7f76701984a0 2024-04-23T00:05:47.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.652+0000 7f7675458640 1 -- 172.21.15.67:0/794349362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 35, "format": "json"} v 0) v1 -- 0x7f76700ffc10 con 0x7f76701984a0 2024-04-23T00:05:47.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.653+0000 7f7657fff640 1 -- 172.21.15.67:0/794349362 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 35, "format": "json"}]=0 dumped fsmap epoch 35 v42) v1 ==== 107+0+5215 (secure 0 0 0) 0x7f766409ce60 con 0x7f76701984a0 2024-04-23T00:05:47.640 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:05:47.640 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44327,"name":"cephfs.smithi134.owipad","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6825/1473208408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":1473208408},{"type":"v1","addr":"172.21.15.134:6825","nonce":1473208408}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":44851,"name":"cephfs.smithi134.vuxbsj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6827/1849648622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":1849648622},{"type":"v1","addr":"172.21.15.134:6827","nonce":1849648622}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T23:20:20.823261+0000","modified":"2024-04-23T00:03:27.072877+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34396,"mds_1":34424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34396":{"gid":34396,"name":"cephfs.smithi067.aqlxpv","rank":0,"incarnation":35,"state":"up:replay","state_seq":1,"addr":"172.21.15.67:6827/789475804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":789475804},{"type":"v1","addr":"172.21.15.67:6827","nonce":789475804}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34424":{"gid":34424,"name":"cephfs.smithi067.xyazmh","rank":1,"incarnation":35,"state":"up:replay","state_seq":1,"addr":"172.21.15.67:6829/927062647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":927062647},{"type":"v1","addr":"172.21.15.67:6829","nonce":927062647}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T00:05:47.641 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 35 2024-04-23T00:05:47.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.655+0000 7f7675458640 1 -- 172.21.15.67:0/794349362 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f765806ec30 msgr2=0x7f76580710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:47.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.655+0000 7f7675458640 1 --2- 172.21.15.67:0/794349362 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f765806ec30 0x7f76580710f0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f7668005fd0 tx=0x7f7668009870 comp rx=0 tx=0).stop 2024-04-23T00:05:47.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.655+0000 7f7675458640 1 -- 172.21.15.67:0/794349362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76701984a0 msgr2=0x7f767019c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:47.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.655+0000 7f7675458640 1 --2- 172.21.15.67:0/794349362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76701984a0 0x7f767019c900 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f7664036040 tx=0x7f766402c850 comp rx=0 tx=0).stop 2024-04-23T00:05:47.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.655+0000 7f7675458640 1 -- 172.21.15.67:0/794349362 shutdown_connections 2024-04-23T00:05:47.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.655+0000 7f7675458640 1 --2- 172.21.15.67:0/794349362 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f765806ec30 0x7f76580710f0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:47.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.655+0000 7f7675458640 1 --2- 172.21.15.67:0/794349362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76701984a0 0x7f767019c900 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:47.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.655+0000 7f7675458640 1 --2- 172.21.15.67:0/794349362 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f76700ff260 0x7f7670197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:47.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.655+0000 7f7675458640 1 -- 172.21.15.67:0/794349362 >> 172.21.15.67:0/794349362 conn(0x7f7670074a50 msgr2=0x7f7670067c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:47.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.655+0000 7f7675458640 1 -- 172.21.15.67:0/794349362 shutdown_connections 2024-04-23T00:05:47.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.655+0000 7f7675458640 1 -- 172.21.15.67:0/794349362 wait complete. 2024-04-23T00:05:47.653 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 35 2024-04-23T00:05:47.654 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 36 2024-04-23T00:05:47.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.768+0000 7f4e6410c640 1 -- 172.21.15.67:0/395662486 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e5c101b20 msgr2=0x7f4e5c10a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:47.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.768+0000 7f4e6410c640 1 --2- 172.21.15.67:0/395662486 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e5c101b20 0x7f4e5c10a000 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c009880 tx=0x7f4e4c02bd70 comp rx=0 tx=0).stop 2024-04-23T00:05:47.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.769+0000 7f4e6410c640 1 -- 172.21.15.67:0/395662486 shutdown_connections 2024-04-23T00:05:47.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.769+0000 7f4e6410c640 1 --2- 172.21.15.67:0/395662486 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e5c101b20 0x7f4e5c10a000 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:47.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.769+0000 7f4e6410c640 1 --2- 172.21.15.67:0/395662486 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e5c101200 0x7f4e5c1015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:47.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.769+0000 7f4e6410c640 1 -- 172.21.15.67:0/395662486 >> 172.21.15.67:0/395662486 conn(0x7f4e5c0faa20 msgr2=0x7f4e5c0fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:47.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.769+0000 7f4e6410c640 1 -- 172.21.15.67:0/395662486 shutdown_connections 2024-04-23T00:05:47.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.769+0000 7f4e6410c640 1 -- 172.21.15.67:0/395662486 wait complete. 2024-04-23T00:05:47.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.769+0000 7f4e6410c640 1 Processor -- start 2024-04-23T00:05:47.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.769+0000 7f4e6410c640 1 -- start start 2024-04-23T00:05:47.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.769+0000 7f4e6410c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e5c101200 0x7f4e5c195d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:47.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.769+0000 7f4e6410c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e5c101b20 0x7f4e5c196290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:47.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.769+0000 7f4e6410c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e5c196920 con 0x7f4e5c101200 2024-04-23T00:05:47.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.769+0000 7f4e6410c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e5c19a6f0 con 0x7f4e5c101b20 2024-04-23T00:05:47.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.770+0000 7f4e61ea8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e5c101200 0x7f4e5c195d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:47.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.770+0000 7f4e61ea8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e5c101200 0x7f4e5c195d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39138/0 (socket says 172.21.15.67:39138) 2024-04-23T00:05:47.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.770+0000 7f4e61ea8640 1 -- 172.21.15.67:0/3036692039 learned_addr learned my addr 172.21.15.67:0/3036692039 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:47.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.770+0000 7f4e616a7640 1 --2- 172.21.15.67:0/3036692039 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e5c101b20 0x7f4e5c196290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.770+0000 7f4e61ea8640 1 -- 172.21.15.67:0/3036692039 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e5c101b20 msgr2=0x7f4e5c196290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.770+0000 7f4e61ea8640 1 --2- 172.21.15.67:0/3036692039 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e5c101b20 0x7f4e5c196290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.770+0000 7f4e61ea8640 1 -- 172.21.15.67:0/3036692039 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e4c009560 con 0x7f4e5c101200 2024-04-23T00:05:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.770+0000 7f4e616a7640 1 --2- 172.21.15.67:0/3036692039 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e5c101b20 0x7f4e5c196290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T00:05:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.770+0000 7f4e61ea8640 1 --2- 172.21.15.67:0/3036692039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e5c101200 0x7f4e5c195d50 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f4e5800d6f0 tx=0x7f4e5800dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.770+0000 7f4e52ffd640 1 -- 172.21.15.67:0/3036692039 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e58009a30 con 0x7f4e5c101200 2024-04-23T00:05:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.770+0000 7f4e52ffd640 1 -- 172.21.15.67:0/3036692039 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4e580054e0 con 0x7f4e5c101200 2024-04-23T00:05:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.770+0000 7f4e6410c640 1 -- 172.21.15.67:0/3036692039 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e5c19a9d0 con 0x7f4e5c101200 2024-04-23T00:05:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.770+0000 7f4e52ffd640 1 -- 172.21.15.67:0/3036692039 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e58010460 con 0x7f4e5c101200 2024-04-23T00:05:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.770+0000 7f4e6410c640 1 -- 172.21.15.67:0/3036692039 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e5c19af30 con 0x7f4e5c101200 2024-04-23T00:05:47.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.771+0000 7f4e6410c640 1 -- 172.21.15.67:0/3036692039 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e5c04e040 con 0x7f4e5c101200 2024-04-23T00:05:47.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.772+0000 7f4e52ffd640 1 -- 172.21.15.67:0/3036692039 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4e58009b90 con 0x7f4e5c101200 2024-04-23T00:05:47.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.772+0000 7f4e52ffd640 1 --2- 172.21.15.67:0/3036692039 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4e4806ec30 0x7f4e480710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:47.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.772+0000 7f4e52ffd640 1 -- 172.21.15.67:0/3036692039 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f4e5808eb80 con 0x7f4e5c101200 2024-04-23T00:05:47.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.772+0000 7f4e616a7640 1 --2- 172.21.15.67:0/3036692039 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4e4806ec30 0x7f4e480710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:47.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.773+0000 7f4e616a7640 1 --2- 172.21.15.67:0/3036692039 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4e4806ec30 0x7f4e480710f0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c197360 tx=0x7f4e4c02cc70 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:47.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:47.777+0000 7f4e52ffd640 1 -- 172.21.15.67:0/3036692039 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f4e580577c0 con 0x7f4e5c101200 2024-04-23T00:05:47.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.006+0000 7f4e6410c640 1 -- 172.21.15.67:0/3036692039 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 36, "format": "json"} v 0) v1 -- 0x7f4e5c107670 con 0x7f4e5c101200 2024-04-23T00:05:47.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.006+0000 7f4e52ffd640 1 -- 172.21.15.67:0/3036692039 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 36, "format": "json"}]=0 dumped fsmap epoch 36 v42) v1 ==== 107+0+5214 (secure 0 0 0) 0x7f4e58017090 con 0x7f4e5c101200 2024-04-23T00:05:47.994 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:05:47.994 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44327,"name":"cephfs.smithi134.owipad","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6825/1473208408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":1473208408},{"type":"v1","addr":"172.21.15.134:6825","nonce":1473208408}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":44851,"name":"cephfs.smithi134.vuxbsj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6827/1849648622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":1849648622},{"type":"v1","addr":"172.21.15.134:6827","nonce":1849648622}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":36,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T23:20:20.823261+0000","modified":"2024-04-23T00:03:27.077964+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34396,"mds_1":34424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34396":{"gid":34396,"name":"cephfs.smithi067.aqlxpv","rank":0,"incarnation":35,"state":"up:replay","state_seq":1,"addr":"172.21.15.67:6827/789475804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":789475804},{"type":"v1","addr":"172.21.15.67:6827","nonce":789475804}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34424":{"gid":34424,"name":"cephfs.smithi067.xyazmh","rank":1,"incarnation":35,"state":"up:replay","state_seq":1,"addr":"172.21.15.67:6829/927062647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":927062647},{"type":"v1","addr":"172.21.15.67:6829","nonce":927062647}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T00:05:47.994 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 36 2024-04-23T00:05:47.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.008+0000 7f4e6410c640 1 -- 172.21.15.67:0/3036692039 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4e4806ec30 msgr2=0x7f4e480710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:47.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.008+0000 7f4e6410c640 1 --2- 172.21.15.67:0/3036692039 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4e4806ec30 0x7f4e480710f0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c197360 tx=0x7f4e4c02cc70 comp rx=0 tx=0).stop 2024-04-23T00:05:47.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.008+0000 7f4e6410c640 1 -- 172.21.15.67:0/3036692039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e5c101200 msgr2=0x7f4e5c195d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:47.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.008+0000 7f4e6410c640 1 --2- 172.21.15.67:0/3036692039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e5c101200 0x7f4e5c195d50 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f4e5800d6f0 tx=0x7f4e5800dbb0 comp rx=0 tx=0).stop 2024-04-23T00:05:47.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.008+0000 7f4e6410c640 1 -- 172.21.15.67:0/3036692039 shutdown_connections 2024-04-23T00:05:47.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.008+0000 7f4e6410c640 1 --2- 172.21.15.67:0/3036692039 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f4e4806ec30 0x7f4e480710f0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:47.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.008+0000 7f4e6410c640 1 --2- 172.21.15.67:0/3036692039 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e5c101b20 0x7f4e5c196290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:47.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.008+0000 7f4e6410c640 1 --2- 172.21.15.67:0/3036692039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e5c101200 0x7f4e5c195d50 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:47.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.008+0000 7f4e6410c640 1 -- 172.21.15.67:0/3036692039 >> 172.21.15.67:0/3036692039 conn(0x7f4e5c0faa20 msgr2=0x7f4e5c104840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:47.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.009+0000 7f4e6410c640 1 -- 172.21.15.67:0/3036692039 shutdown_connections 2024-04-23T00:05:47.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.009+0000 7f4e6410c640 1 -- 172.21.15.67:0/3036692039 wait complete. 2024-04-23T00:05:48.006 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:47 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/276321109' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 33, "format": "json"}]: dispatch 2024-04-23T00:05:48.006 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:47 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2312999562' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 34, "format": "json"}]: dispatch 2024-04-23T00:05:48.006 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:47 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/794349362' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 35, "format": "json"}]: dispatch 2024-04-23T00:05:48.007 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 37 2024-04-23T00:05:48.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.125+0000 7f3a6b896640 1 -- 172.21.15.67:0/2176604682 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a64067f70 msgr2=0x7f3a64104c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:48.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.125+0000 7f3a6b896640 1 --2- 172.21.15.67:0/2176604682 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a64067f70 0x7f3a64104c70 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f3a60009880 tx=0x7f3a6002bce0 comp rx=0 tx=0).stop 2024-04-23T00:05:48.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.125+0000 7f3a6b896640 1 -- 172.21.15.67:0/2176604682 shutdown_connections 2024-04-23T00:05:48.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.125+0000 7f3a6b896640 1 --2- 172.21.15.67:0/2176604682 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a64067f70 0x7f3a64104c70 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:48.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.125+0000 7f3a6b896640 1 --2- 172.21.15.67:0/2176604682 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a640675c0 0x7f3a640679a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:48.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.125+0000 7f3a6b896640 1 -- 172.21.15.67:0/2176604682 >> 172.21.15.67:0/2176604682 conn(0x7f3a64074a50 msgr2=0x7f3a64074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:48.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.125+0000 7f3a6b896640 1 -- 172.21.15.67:0/2176604682 shutdown_connections 2024-04-23T00:05:48.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.125+0000 7f3a6b896640 1 -- 172.21.15.67:0/2176604682 wait complete. 2024-04-23T00:05:48.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.126+0000 7f3a6b896640 1 Processor -- start 2024-04-23T00:05:48.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.126+0000 7f3a6b896640 1 -- start start 2024-04-23T00:05:48.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.126+0000 7f3a6b896640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a640675c0 0x7f3a64195d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:48.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.126+0000 7f3a6b896640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a641962c0 0x7f3a6419a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:48.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.126+0000 7f3a6b896640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a641968c0 con 0x7f3a641962c0 2024-04-23T00:05:48.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.126+0000 7f3a6b896640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a64196a30 con 0x7f3a640675c0 2024-04-23T00:05:48.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.126+0000 7f3a68e31640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a641962c0 0x7f3a6419a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:48.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.126+0000 7f3a68e31640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a641962c0 0x7f3a6419a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39158/0 (socket says 172.21.15.67:39158) 2024-04-23T00:05:48.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.126+0000 7f3a68e31640 1 -- 172.21.15.67:0/1502697406 learned_addr learned my addr 172.21.15.67:0/1502697406 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:48.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.126+0000 7f3a69632640 1 --2- 172.21.15.67:0/1502697406 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a640675c0 0x7f3a64195d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:48.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.126+0000 7f3a68e31640 1 -- 172.21.15.67:0/1502697406 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a640675c0 msgr2=0x7f3a64195d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:48.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.126+0000 7f3a68e31640 1 --2- 172.21.15.67:0/1502697406 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a640675c0 0x7f3a64195d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:48.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.126+0000 7f3a68e31640 1 -- 172.21.15.67:0/1502697406 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a60009560 con 0x7f3a641962c0 2024-04-23T00:05:48.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.126+0000 7f3a68e31640 1 --2- 172.21.15.67:0/1502697406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a641962c0 0x7f3a6419a720 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f3a60036040 tx=0x7f3a6002c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:48.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.127+0000 7f3a5a7fc640 1 -- 172.21.15.67:0/1502697406 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a600035f0 con 0x7f3a641962c0 2024-04-23T00:05:48.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.127+0000 7f3a6b896640 1 -- 172.21.15.67:0/1502697406 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a6419acc0 con 0x7f3a641962c0 2024-04-23T00:05:48.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.127+0000 7f3a5a7fc640 1 -- 172.21.15.67:0/1502697406 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3a60003c10 con 0x7f3a641962c0 2024-04-23T00:05:48.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.127+0000 7f3a6b896640 1 -- 172.21.15.67:0/1502697406 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a6419b220 con 0x7f3a641962c0 2024-04-23T00:05:48.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.127+0000 7f3a5a7fc640 1 -- 172.21.15.67:0/1502697406 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a60046450 con 0x7f3a641962c0 2024-04-23T00:05:48.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.128+0000 7f3a6b896640 1 -- 172.21.15.67:0/1502697406 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a6404e040 con 0x7f3a641962c0 2024-04-23T00:05:48.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.128+0000 7f3a5a7fc640 1 -- 172.21.15.67:0/1502697406 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3a60003750 con 0x7f3a641962c0 2024-04-23T00:05:48.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.129+0000 7f3a5a7fc640 1 --2- 172.21.15.67:0/1502697406 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3a5006ec30 0x7f3a500710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:48.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.129+0000 7f3a5a7fc640 1 -- 172.21.15.67:0/1502697406 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f3a600b1970 con 0x7f3a641962c0 2024-04-23T00:05:48.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.129+0000 7f3a69632640 1 --2- 172.21.15.67:0/1502697406 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3a5006ec30 0x7f3a500710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:48.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.129+0000 7f3a69632640 1 --2- 172.21.15.67:0/1502697406 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3a5006ec30 0x7f3a500710f0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f3a54005fd0 tx=0x7f3a54005850 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:48.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.133+0000 7f3a5a7fc640 1 -- 172.21.15.67:0/1502697406 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f3a6007a630 con 0x7f3a641962c0 2024-04-23T00:05:48.229 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:47 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/276321109' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 33, "format": "json"}]: dispatch 2024-04-23T00:05:48.230 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:47 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2312999562' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 34, "format": "json"}]: dispatch 2024-04-23T00:05:48.230 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:47 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/794349362' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 35, "format": "json"}]: dispatch 2024-04-23T00:05:48.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.363+0000 7f3a6b896640 1 -- 172.21.15.67:0/1502697406 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 37, "format": "json"} v 0) v1 -- 0x7f3a64108580 con 0x7f3a641962c0 2024-04-23T00:05:48.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.363+0000 7f3a5a7fc640 1 -- 172.21.15.67:0/1502697406 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 37, "format": "json"}]=0 dumped fsmap epoch 37 v42) v1 ==== 107+0+5220 (secure 0 0 0) 0x7f3a6007dcf0 con 0x7f3a641962c0 2024-04-23T00:05:48.351 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:05:48.351 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44327,"name":"cephfs.smithi134.owipad","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6825/1473208408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":1473208408},{"type":"v1","addr":"172.21.15.134:6825","nonce":1473208408}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":44851,"name":"cephfs.smithi134.vuxbsj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6827/1849648622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":1849648622},{"type":"v1","addr":"172.21.15.134:6827","nonce":1849648622}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T23:20:20.823261+0000","modified":"2024-04-23T00:03:33.750728+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34396,"mds_1":34424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34396":{"gid":34396,"name":"cephfs.smithi067.aqlxpv","rank":0,"incarnation":35,"state":"up:resolve","state_seq":466,"addr":"172.21.15.67:6827/789475804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":789475804},{"type":"v1","addr":"172.21.15.67:6827","nonce":789475804}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34424":{"gid":34424,"name":"cephfs.smithi067.xyazmh","rank":1,"incarnation":35,"state":"up:resolve","state_seq":461,"addr":"172.21.15.67:6829/927062647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":927062647},{"type":"v1","addr":"172.21.15.67:6829","nonce":927062647}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T00:05:48.351 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 37 2024-04-23T00:05:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.366+0000 7f3a6b896640 1 -- 172.21.15.67:0/1502697406 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3a5006ec30 msgr2=0x7f3a500710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.366+0000 7f3a6b896640 1 --2- 172.21.15.67:0/1502697406 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3a5006ec30 0x7f3a500710f0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f3a54005fd0 tx=0x7f3a54005850 comp rx=0 tx=0).stop 2024-04-23T00:05:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.366+0000 7f3a6b896640 1 -- 172.21.15.67:0/1502697406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a641962c0 msgr2=0x7f3a6419a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.366+0000 7f3a6b896640 1 --2- 172.21.15.67:0/1502697406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a641962c0 0x7f3a6419a720 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f3a60036040 tx=0x7f3a6002c850 comp rx=0 tx=0).stop 2024-04-23T00:05:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.366+0000 7f3a6b896640 1 -- 172.21.15.67:0/1502697406 shutdown_connections 2024-04-23T00:05:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.366+0000 7f3a6b896640 1 --2- 172.21.15.67:0/1502697406 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3a5006ec30 0x7f3a500710f0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.366+0000 7f3a6b896640 1 --2- 172.21.15.67:0/1502697406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a641962c0 0x7f3a6419a720 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.366+0000 7f3a6b896640 1 --2- 172.21.15.67:0/1502697406 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a640675c0 0x7f3a64195d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.366+0000 7f3a6b896640 1 -- 172.21.15.67:0/1502697406 >> 172.21.15.67:0/1502697406 conn(0x7f3a64074a50 msgr2=0x7f3a640fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.366+0000 7f3a6b896640 1 -- 172.21.15.67:0/1502697406 shutdown_connections 2024-04-23T00:05:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.366+0000 7f3a6b896640 1 -- 172.21.15.67:0/1502697406 wait complete. 2024-04-23T00:05:48.366 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 38 2024-04-23T00:05:48.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.497+0000 7f1ed1a69640 1 -- 172.21.15.67:0/3870216291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ecc0ffc10 msgr2=0x7f1ecc103b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:48.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.497+0000 7f1ed1a69640 1 --2- 172.21.15.67:0/3870216291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ecc0ffc10 0x7f1ecc103b70 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f1ec0009880 tx=0x7f1ec002bce0 comp rx=0 tx=0).stop 2024-04-23T00:05:48.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.497+0000 7f1ed1a69640 1 -- 172.21.15.67:0/3870216291 shutdown_connections 2024-04-23T00:05:48.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.497+0000 7f1ed1a69640 1 --2- 172.21.15.67:0/3870216291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ecc0ffc10 0x7f1ecc103b70 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:48.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.497+0000 7f1ed1a69640 1 --2- 172.21.15.67:0/3870216291 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ecc0ff260 0x7f1ecc0ff640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:48.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.497+0000 7f1ed1a69640 1 -- 172.21.15.67:0/3870216291 >> 172.21.15.67:0/3870216291 conn(0x7f1ecc074a50 msgr2=0x7f1ecc074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:48.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.498+0000 7f1ed1a69640 1 -- 172.21.15.67:0/3870216291 shutdown_connections 2024-04-23T00:05:48.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.498+0000 7f1ed1a69640 1 -- 172.21.15.67:0/3870216291 wait complete. 2024-04-23T00:05:48.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.498+0000 7f1ed1a69640 1 Processor -- start 2024-04-23T00:05:48.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.498+0000 7f1ed1a69640 1 -- start start 2024-04-23T00:05:48.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.498+0000 7f1ed1a69640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ecc0ff260 0x7f1ecc196b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:48.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.498+0000 7f1ed1a69640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ecc197070 0x7f1ecc1974d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:48.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.498+0000 7f1ed1a69640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ecc19fba0 con 0x7f1ecc197070 2024-04-23T00:05:48.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.498+0000 7f1ed1a69640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ecc19fce0 con 0x7f1ecc0ff260 2024-04-23T00:05:48.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.499+0000 7f1ecaffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ecc197070 0x7f1ecc1974d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:48.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.499+0000 7f1ecaffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ecc197070 0x7f1ecc1974d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39176/0 (socket says 172.21.15.67:39176) 2024-04-23T00:05:48.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.499+0000 7f1ecaffd640 1 -- 172.21.15.67:0/2703049498 learned_addr learned my addr 172.21.15.67:0/2703049498 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:48.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.499+0000 7f1ecb7fe640 1 --2- 172.21.15.67:0/2703049498 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ecc0ff260 0x7f1ecc196b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:48.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.499+0000 7f1ecaffd640 1 -- 172.21.15.67:0/2703049498 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ecc0ff260 msgr2=0x7f1ecc196b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:48.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.499+0000 7f1ecaffd640 1 --2- 172.21.15.67:0/2703049498 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ecc0ff260 0x7f1ecc196b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:48.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.499+0000 7f1ecaffd640 1 -- 172.21.15.67:0/2703049498 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec0009560 con 0x7f1ecc197070 2024-04-23T00:05:48.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.499+0000 7f1ecaffd640 1 --2- 172.21.15.67:0/2703049498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ecc197070 0x7f1ecc1974d0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f1ec0036040 tx=0x7f1ec0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:48.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.499+0000 7f1ec8ff9640 1 -- 172.21.15.67:0/2703049498 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ec00037c0 con 0x7f1ecc197070 2024-04-23T00:05:48.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.499+0000 7f1ec8ff9640 1 -- 172.21.15.67:0/2703049498 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ec0003de0 con 0x7f1ecc197070 2024-04-23T00:05:48.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.499+0000 7f1ed1a69640 1 -- 172.21.15.67:0/2703049498 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ecc19fe80 con 0x7f1ecc197070 2024-04-23T00:05:48.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.499+0000 7f1ec8ff9640 1 -- 172.21.15.67:0/2703049498 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ec0034440 con 0x7f1ecc197070 2024-04-23T00:05:48.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.499+0000 7f1ed1a69640 1 -- 172.21.15.67:0/2703049498 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ecc1a0350 con 0x7f1ecc197070 2024-04-23T00:05:48.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.500+0000 7f1ed1a69640 1 -- 172.21.15.67:0/2703049498 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ecc04e040 con 0x7f1ecc197070 2024-04-23T00:05:48.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.501+0000 7f1ec8ff9640 1 -- 172.21.15.67:0/2703049498 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1ec0004fb0 con 0x7f1ecc197070 2024-04-23T00:05:48.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.501+0000 7f1ec8ff9640 1 --2- 172.21.15.67:0/2703049498 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1eb406ec30 0x7f1eb40710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:48.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.501+0000 7f1ec8ff9640 1 -- 172.21.15.67:0/2703049498 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f1ec00b1400 con 0x7f1ecc197070 2024-04-23T00:05:48.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.501+0000 7f1ecb7fe640 1 --2- 172.21.15.67:0/2703049498 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1eb406ec30 0x7f1eb40710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:48.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.502+0000 7f1ecb7fe640 1 --2- 172.21.15.67:0/2703049498 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1eb406ec30 0x7f1eb40710f0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc005e10 tx=0x7f1ebc005da0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:48.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.506+0000 7f1ec8ff9640 1 -- 172.21.15.67:0/2703049498 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f1ec007a040 con 0x7f1ecc197070 2024-04-23T00:05:48.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.736+0000 7f1ed1a69640 1 -- 172.21.15.67:0/2703049498 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 38, "format": "json"} v 0) v1 -- 0x7f1ecc107480 con 0x7f1ecc197070 2024-04-23T00:05:48.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.737+0000 7f1ec8ff9640 1 -- 172.21.15.67:0/2703049498 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 38, "format": "json"}]=0 dumped fsmap epoch 38 v42) v1 ==== 107+0+5224 (secure 0 0 0) 0x7f1ec007d700 con 0x7f1ecc197070 2024-04-23T00:05:48.725 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:05:48.725 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44327,"name":"cephfs.smithi134.owipad","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6825/1473208408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":1473208408},{"type":"v1","addr":"172.21.15.134:6825","nonce":1473208408}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":44851,"name":"cephfs.smithi134.vuxbsj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6827/1849648622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":1849648622},{"type":"v1","addr":"172.21.15.134:6827","nonce":1849648622}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":38,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T23:20:20.823261+0000","modified":"2024-04-23T00:03:34.154270+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34396,"mds_1":34424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34396":{"gid":34396,"name":"cephfs.smithi067.aqlxpv","rank":0,"incarnation":35,"state":"up:reconnect","state_seq":467,"addr":"172.21.15.67:6827/789475804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":789475804},{"type":"v1","addr":"172.21.15.67:6827","nonce":789475804}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34424":{"gid":34424,"name":"cephfs.smithi067.xyazmh","rank":1,"incarnation":35,"state":"up:reconnect","state_seq":462,"addr":"172.21.15.67:6829/927062647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":927062647},{"type":"v1","addr":"172.21.15.67:6829","nonce":927062647}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T00:05:48.725 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 38 2024-04-23T00:05:48.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.739+0000 7f1ed1a69640 1 -- 172.21.15.67:0/2703049498 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1eb406ec30 msgr2=0x7f1eb40710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:48.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.739+0000 7f1ed1a69640 1 --2- 172.21.15.67:0/2703049498 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1eb406ec30 0x7f1eb40710f0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc005e10 tx=0x7f1ebc005da0 comp rx=0 tx=0).stop 2024-04-23T00:05:48.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.739+0000 7f1ed1a69640 1 -- 172.21.15.67:0/2703049498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ecc197070 msgr2=0x7f1ecc1974d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:48.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.739+0000 7f1ed1a69640 1 --2- 172.21.15.67:0/2703049498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ecc197070 0x7f1ecc1974d0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f1ec0036040 tx=0x7f1ec0004270 comp rx=0 tx=0).stop 2024-04-23T00:05:48.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.739+0000 7f1ed1a69640 1 -- 172.21.15.67:0/2703049498 shutdown_connections 2024-04-23T00:05:48.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.739+0000 7f1ed1a69640 1 --2- 172.21.15.67:0/2703049498 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f1eb406ec30 0x7f1eb40710f0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:48.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.739+0000 7f1ed1a69640 1 --2- 172.21.15.67:0/2703049498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ecc197070 0x7f1ecc1974d0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:48.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.739+0000 7f1ed1a69640 1 --2- 172.21.15.67:0/2703049498 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1ecc0ff260 0x7f1ecc196b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:48.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.739+0000 7f1ed1a69640 1 -- 172.21.15.67:0/2703049498 >> 172.21.15.67:0/2703049498 conn(0x7f1ecc074a50 msgr2=0x7f1ecc0fdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:48.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.740+0000 7f1ed1a69640 1 -- 172.21.15.67:0/2703049498 shutdown_connections 2024-04-23T00:05:48.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.740+0000 7f1ed1a69640 1 -- 172.21.15.67:0/2703049498 wait complete. 2024-04-23T00:05:48.739 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 39 2024-04-23T00:05:48.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.855+0000 7f5cbd277640 1 -- 172.21.15.67:0/3254218980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cb8101260 msgr2=0x7f5cb8101640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:48.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.855+0000 7f5cbd277640 1 --2- 172.21.15.67:0/3254218980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cb8101260 0x7f5cb8101640 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f5ca80098b0 tx=0x7f5ca802bd60 comp rx=0 tx=0).stop 2024-04-23T00:05:48.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.855+0000 7f5cbd277640 1 -- 172.21.15.67:0/3254218980 shutdown_connections 2024-04-23T00:05:48.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.855+0000 7f5cbd277640 1 --2- 172.21.15.67:0/3254218980 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5cb8101b80 0x7f5cb810a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:48.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.855+0000 7f5cbd277640 1 --2- 172.21.15.67:0/3254218980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cb8101260 0x7f5cb8101640 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:48.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.855+0000 7f5cbd277640 1 -- 172.21.15.67:0/3254218980 >> 172.21.15.67:0/3254218980 conn(0x7f5cb80faa20 msgr2=0x7f5cb80fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:48.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.855+0000 7f5cbd277640 1 -- 172.21.15.67:0/3254218980 shutdown_connections 2024-04-23T00:05:48.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.855+0000 7f5cbd277640 1 -- 172.21.15.67:0/3254218980 wait complete. 2024-04-23T00:05:48.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.855+0000 7f5cbd277640 1 Processor -- start 2024-04-23T00:05:48.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.855+0000 7f5cbd277640 1 -- start start 2024-04-23T00:05:48.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.856+0000 7f5cbd277640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5cb8101260 0x7f5cb8195d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:48.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.856+0000 7f5cbd277640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cb8101b80 0x7f5cb81962d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:48.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.856+0000 7f5cbd277640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb8196960 con 0x7f5cb8101b80 2024-04-23T00:05:48.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.856+0000 7f5cbd277640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb819a730 con 0x7f5cb8101260 2024-04-23T00:05:48.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.856+0000 7f5cb67fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cb8101b80 0x7f5cb81962d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:48.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.856+0000 7f5cb67fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cb8101b80 0x7f5cb81962d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35020/0 (socket says 172.21.15.67:35020) 2024-04-23T00:05:48.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.856+0000 7f5cb67fc640 1 -- 172.21.15.67:0/3311446751 learned_addr learned my addr 172.21.15.67:0/3311446751 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:48.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.856+0000 7f5cb6ffd640 1 --2- 172.21.15.67:0/3311446751 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5cb8101260 0x7f5cb8195d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:48.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.856+0000 7f5cb67fc640 1 -- 172.21.15.67:0/3311446751 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5cb8101260 msgr2=0x7f5cb8195d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:48.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.856+0000 7f5cb67fc640 1 --2- 172.21.15.67:0/3311446751 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5cb8101260 0x7f5cb8195d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:48.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.856+0000 7f5cb67fc640 1 -- 172.21.15.67:0/3311446751 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca8009560 con 0x7f5cb8101b80 2024-04-23T00:05:48.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.856+0000 7f5cb6ffd640 1 --2- 172.21.15.67:0/3311446751 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5cb8101260 0x7f5cb8195d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T00:05:48.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.856+0000 7f5cb67fc640 1 --2- 172.21.15.67:0/3311446751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cb8101b80 0x7f5cb81962d0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f5cac00c8f0 tx=0x7f5cac00cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:48.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.856+0000 7f5c9ffff640 1 -- 172.21.15.67:0/3311446751 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5cac007b80 con 0x7f5cb8101b80 2024-04-23T00:05:48.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.857+0000 7f5cbd277640 1 -- 172.21.15.67:0/3311446751 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cb819aa10 con 0x7f5cb8101b80 2024-04-23T00:05:48.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.857+0000 7f5c9ffff640 1 -- 172.21.15.67:0/3311446751 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5cac0054e0 con 0x7f5cb8101b80 2024-04-23T00:05:48.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.857+0000 7f5c9ffff640 1 -- 172.21.15.67:0/3311446751 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5cac00f450 con 0x7f5cb8101b80 2024-04-23T00:05:48.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.857+0000 7f5cbd277640 1 -- 172.21.15.67:0/3311446751 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cb819af70 con 0x7f5cb8101b80 2024-04-23T00:05:48.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.857+0000 7f5cbd277640 1 -- 172.21.15.67:0/3311446751 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cb804e040 con 0x7f5cb8101b80 2024-04-23T00:05:48.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.858+0000 7f5c9ffff640 1 -- 172.21.15.67:0/3311446751 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5cac004f60 con 0x7f5cb8101b80 2024-04-23T00:05:48.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.858+0000 7f5c9ffff640 1 --2- 172.21.15.67:0/3311446751 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5ca006ec30 0x7f5ca00710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:48.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.859+0000 7f5c9ffff640 1 -- 172.21.15.67:0/3311446751 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f5cac08ee30 con 0x7f5cb8101b80 2024-04-23T00:05:48.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.859+0000 7f5cb6ffd640 1 --2- 172.21.15.67:0/3311446751 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5ca006ec30 0x7f5ca00710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:48.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.859+0000 7f5cb6ffd640 1 --2- 172.21.15.67:0/3311446751 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5ca006ec30 0x7f5ca00710f0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f5ca8005ec0 tx=0x7f5ca8005fd0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:48.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:48.868+0000 7f5c9ffff640 1 -- 172.21.15.67:0/3311446751 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f5cac057a70 con 0x7f5cb8101b80 2024-04-23T00:05:49.030 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:48 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3036692039' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 36, "format": "json"}]: dispatch 2024-04-23T00:05:49.030 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:48 smithi067 ceph-mon[112438]: pgmap v1187: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:05:49.030 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:48 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1502697406' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 37, "format": "json"}]: dispatch 2024-04-23T00:05:49.030 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:48 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2703049498' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 38, "format": "json"}]: dispatch 2024-04-23T00:05:49.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.099+0000 7f5cbd277640 1 -- 172.21.15.67:0/3311446751 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 39, "format": "json"} v 0) v1 -- 0x7f5cb81076d0 con 0x7f5cb8101b80 2024-04-23T00:05:49.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.100+0000 7f5c9ffff640 1 -- 172.21.15.67:0/3311446751 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 39, "format": "json"}]=0 dumped fsmap epoch 39 v42) v1 ==== 107+0+5218 (secure 0 0 0) 0x7f5cac05b130 con 0x7f5cb8101b80 2024-04-23T00:05:49.088 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:05:49.088 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44327,"name":"cephfs.smithi134.owipad","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6825/1473208408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":1473208408},{"type":"v1","addr":"172.21.15.134:6825","nonce":1473208408}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":44851,"name":"cephfs.smithi134.vuxbsj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6827/1849648622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":1849648622},{"type":"v1","addr":"172.21.15.134:6827","nonce":1849648622}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T23:20:20.823261+0000","modified":"2024-04-23T00:03:35.099513+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34396,"mds_1":34424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34396":{"gid":34396,"name":"cephfs.smithi067.aqlxpv","rank":0,"incarnation":35,"state":"up:rejoin","state_seq":468,"addr":"172.21.15.67:6827/789475804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":789475804},{"type":"v1","addr":"172.21.15.67:6827","nonce":789475804}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34424":{"gid":34424,"name":"cephfs.smithi067.xyazmh","rank":1,"incarnation":35,"state":"up:rejoin","state_seq":463,"addr":"172.21.15.67:6829/927062647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":927062647},{"type":"v1","addr":"172.21.15.67:6829","nonce":927062647}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-23T00:05:49.088 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 39 2024-04-23T00:05:49.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.102+0000 7f5cbd277640 1 -- 172.21.15.67:0/3311446751 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5ca006ec30 msgr2=0x7f5ca00710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:49.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.102+0000 7f5cbd277640 1 --2- 172.21.15.67:0/3311446751 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5ca006ec30 0x7f5ca00710f0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f5ca8005ec0 tx=0x7f5ca8005fd0 comp rx=0 tx=0).stop 2024-04-23T00:05:49.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.102+0000 7f5cbd277640 1 -- 172.21.15.67:0/3311446751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cb8101b80 msgr2=0x7f5cb81962d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:49.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.102+0000 7f5cbd277640 1 --2- 172.21.15.67:0/3311446751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cb8101b80 0x7f5cb81962d0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f5cac00c8f0 tx=0x7f5cac00cdb0 comp rx=0 tx=0).stop 2024-04-23T00:05:49.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.102+0000 7f5cbd277640 1 -- 172.21.15.67:0/3311446751 shutdown_connections 2024-04-23T00:05:49.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.102+0000 7f5cbd277640 1 --2- 172.21.15.67:0/3311446751 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5ca006ec30 0x7f5ca00710f0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:49.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.102+0000 7f5cbd277640 1 --2- 172.21.15.67:0/3311446751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cb8101b80 0x7f5cb81962d0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:49.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.102+0000 7f5cbd277640 1 --2- 172.21.15.67:0/3311446751 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5cb8101260 0x7f5cb8195d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:49.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.103+0000 7f5cbd277640 1 -- 172.21.15.67:0/3311446751 >> 172.21.15.67:0/3311446751 conn(0x7f5cb80faa20 msgr2=0x7f5cb81048a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:49.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.103+0000 7f5cbd277640 1 -- 172.21.15.67:0/3311446751 shutdown_connections 2024-04-23T00:05:49.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.103+0000 7f5cbd277640 1 -- 172.21.15.67:0/3311446751 wait complete. 2024-04-23T00:05:49.101 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 40 2024-04-23T00:05:49.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.217+0000 7fabd8ac4640 1 -- 172.21.15.67:0/1398804892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabd40ff260 msgr2=0x7fabd40ff640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:49.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.217+0000 7fabd8ac4640 1 --2- 172.21.15.67:0/1398804892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabd40ff260 0x7fabd40ff640 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fabc4009880 tx=0x7fabc402bce0 comp rx=0 tx=0).stop 2024-04-23T00:05:49.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.217+0000 7fabd8ac4640 1 -- 172.21.15.67:0/1398804892 shutdown_connections 2024-04-23T00:05:49.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.217+0000 7fabd8ac4640 1 --2- 172.21.15.67:0/1398804892 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabd40ffc10 0x7fabd4103b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:49.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.217+0000 7fabd8ac4640 1 --2- 172.21.15.67:0/1398804892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabd40ff260 0x7fabd40ff640 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:49.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.217+0000 7fabd8ac4640 1 -- 172.21.15.67:0/1398804892 >> 172.21.15.67:0/1398804892 conn(0x7fabd4074a50 msgr2=0x7fabd4074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:49.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.217+0000 7fabd8ac4640 1 -- 172.21.15.67:0/1398804892 shutdown_connections 2024-04-23T00:05:49.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.217+0000 7fabd8ac4640 1 -- 172.21.15.67:0/1398804892 wait complete. 2024-04-23T00:05:49.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.218+0000 7fabd8ac4640 1 Processor -- start 2024-04-23T00:05:49.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.218+0000 7fabd8ac4640 1 -- start start 2024-04-23T00:05:49.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.218+0000 7fabd8ac4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabd40ff260 0x7fabd4195d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:49.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.218+0000 7fabd8ac4640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabd41962d0 0x7fabd419a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:49.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.218+0000 7fabd8ac4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd41968d0 con 0x7fabd40ff260 2024-04-23T00:05:49.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.218+0000 7fabd8ac4640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd4196a40 con 0x7fabd41962d0 2024-04-23T00:05:49.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.218+0000 7fabd259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabd40ff260 0x7fabd4195d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:49.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.218+0000 7fabd259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabd40ff260 0x7fabd4195d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35042/0 (socket says 172.21.15.67:35042) 2024-04-23T00:05:49.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.218+0000 7fabd259c640 1 -- 172.21.15.67:0/1232691537 learned_addr learned my addr 172.21.15.67:0/1232691537 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:49.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.218+0000 7fabd1d9b640 1 --2- 172.21.15.67:0/1232691537 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabd41962d0 0x7fabd419a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:49.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.218+0000 7fabd259c640 1 -- 172.21.15.67:0/1232691537 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabd41962d0 msgr2=0x7fabd419a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:49.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.218+0000 7fabd259c640 1 --2- 172.21.15.67:0/1232691537 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabd41962d0 0x7fabd419a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:49.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.218+0000 7fabd259c640 1 -- 172.21.15.67:0/1232691537 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc4009560 con 0x7fabd40ff260 2024-04-23T00:05:49.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.219+0000 7fabd259c640 1 --2- 172.21.15.67:0/1232691537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabd40ff260 0x7fabd4195d90 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fabc4009430 tx=0x7fabc4004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:49.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.219+0000 7fabc37fe640 1 -- 172.21.15.67:0/1232691537 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabc40035f0 con 0x7fabd40ff260 2024-04-23T00:05:49.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.219+0000 7fabd8ac4640 1 -- 172.21.15.67:0/1232691537 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabd419acd0 con 0x7fabd40ff260 2024-04-23T00:05:49.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.219+0000 7fabc37fe640 1 -- 172.21.15.67:0/1232691537 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fabc4003c10 con 0x7fabd40ff260 2024-04-23T00:05:49.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.219+0000 7fabc37fe640 1 -- 172.21.15.67:0/1232691537 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabc4005670 con 0x7fabd40ff260 2024-04-23T00:05:49.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.219+0000 7fabd8ac4640 1 -- 172.21.15.67:0/1232691537 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabd419b230 con 0x7fabd40ff260 2024-04-23T00:05:49.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.220+0000 7fabd8ac4640 1 -- 172.21.15.67:0/1232691537 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabd404e040 con 0x7fabd40ff260 2024-04-23T00:05:49.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.220+0000 7fabc37fe640 1 -- 172.21.15.67:0/1232691537 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fabc4003750 con 0x7fabd40ff260 2024-04-23T00:05:49.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.221+0000 7fabc37fe640 1 --2- 172.21.15.67:0/1232691537 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fabbc06ebe0 0x7fabbc0710a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:49.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.221+0000 7fabc37fe640 1 -- 172.21.15.67:0/1232691537 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fabc40b13c0 con 0x7fabd40ff260 2024-04-23T00:05:49.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.221+0000 7fabd1d9b640 1 --2- 172.21.15.67:0/1232691537 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fabbc06ebe0 0x7fabbc0710a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:49.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.221+0000 7fabd1d9b640 1 --2- 172.21.15.67:0/1232691537 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fabbc06ebe0 0x7fabbc0710a0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fabc8005fd0 tx=0x7fabc8005850 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:49.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.225+0000 7fabc37fe640 1 -- 172.21.15.67:0/1232691537 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fabc407a000 con 0x7fabd40ff260 2024-04-23T00:05:49.229 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:48 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3036692039' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 36, "format": "json"}]: dispatch 2024-04-23T00:05:49.230 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:48 smithi134 ceph-mon[85630]: pgmap v1187: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:05:49.230 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:48 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1502697406' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 37, "format": "json"}]: dispatch 2024-04-23T00:05:49.230 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:48 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2703049498' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 38, "format": "json"}]: dispatch 2024-04-23T00:05:49.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.453+0000 7fabd8ac4640 1 -- 172.21.15.67:0/1232691537 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 40, "format": "json"} v 0) v1 -- 0x7fabd40ffc10 con 0x7fabd40ff260 2024-04-23T00:05:49.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.454+0000 7fabc37fe640 1 -- 172.21.15.67:0/1232691537 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 40, "format": "json"}]=0 dumped fsmap epoch 40 v42) v1 ==== 107+0+5233 (secure 0 0 0) 0x7fabc407d6c0 con 0x7fabd40ff260 2024-04-23T00:05:49.441 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:05:49.441 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44327,"name":"cephfs.smithi134.owipad","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6825/1473208408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":1473208408},{"type":"v1","addr":"172.21.15.134:6825","nonce":1473208408}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31},{"gid":44851,"name":"cephfs.smithi134.vuxbsj","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6827/1849648622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":1849648622},{"type":"v1","addr":"172.21.15.134:6827","nonce":1849648622}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":40,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T23:20:20.823261+0000","modified":"2024-04-23T00:03:37.103874+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34396,"mds_1":34424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34396":{"gid":34396,"name":"cephfs.smithi067.aqlxpv","rank":0,"incarnation":35,"state":"up:active","state_seq":469,"addr":"172.21.15.67:6827/789475804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":789475804},{"type":"v1","addr":"172.21.15.67:6827","nonce":789475804}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34424":{"gid":34424,"name":"cephfs.smithi067.xyazmh","rank":1,"incarnation":35,"state":"up:active","state_seq":464,"addr":"172.21.15.67:6829/927062647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":927062647},{"type":"v1","addr":"172.21.15.67:6829","nonce":927062647}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":34396,"qdb_cluster":[34396,34424]},"id":1}]} 2024-04-23T00:05:49.442 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 40 2024-04-23T00:05:49.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.456+0000 7fabd8ac4640 1 -- 172.21.15.67:0/1232691537 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fabbc06ebe0 msgr2=0x7fabbc0710a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:49.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.456+0000 7fabd8ac4640 1 --2- 172.21.15.67:0/1232691537 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fabbc06ebe0 0x7fabbc0710a0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fabc8005fd0 tx=0x7fabc8005850 comp rx=0 tx=0).stop 2024-04-23T00:05:49.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.456+0000 7fabd8ac4640 1 -- 172.21.15.67:0/1232691537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabd40ff260 msgr2=0x7fabd4195d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:49.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.456+0000 7fabd8ac4640 1 --2- 172.21.15.67:0/1232691537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabd40ff260 0x7fabd4195d90 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fabc4009430 tx=0x7fabc4004040 comp rx=0 tx=0).stop 2024-04-23T00:05:49.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.456+0000 7fabd8ac4640 1 -- 172.21.15.67:0/1232691537 shutdown_connections 2024-04-23T00:05:49.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.456+0000 7fabd8ac4640 1 --2- 172.21.15.67:0/1232691537 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fabbc06ebe0 0x7fabbc0710a0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:49.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.456+0000 7fabd8ac4640 1 --2- 172.21.15.67:0/1232691537 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fabd41962d0 0x7fabd419a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:49.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.456+0000 7fabd8ac4640 1 --2- 172.21.15.67:0/1232691537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabd40ff260 0x7fabd4195d90 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:49.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.456+0000 7fabd8ac4640 1 -- 172.21.15.67:0/1232691537 >> 172.21.15.67:0/1232691537 conn(0x7fabd4074a50 msgr2=0x7fabd41033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:49.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.456+0000 7fabd8ac4640 1 -- 172.21.15.67:0/1232691537 shutdown_connections 2024-04-23T00:05:49.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.456+0000 7fabd8ac4640 1 -- 172.21.15.67:0/1232691537 wait complete. 2024-04-23T00:05:49.455 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 41 2024-04-23T00:05:49.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.570+0000 7f65b8644640 1 -- 172.21.15.67:0/856488415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65b01002a0 msgr2=0x7f65b0100680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:49.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.570+0000 7f65b8644640 1 --2- 172.21.15.67:0/856488415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65b01002a0 0x7f65b0100680 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f65ac009880 tx=0x7f65ac02bce0 comp rx=0 tx=0).stop 2024-04-23T00:05:49.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.570+0000 7f65b8644640 1 -- 172.21.15.67:0/856488415 shutdown_connections 2024-04-23T00:05:49.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.570+0000 7f65b8644640 1 --2- 172.21.15.67:0/856488415 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f65b0100c50 0x7f65b0104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:49.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.570+0000 7f65b8644640 1 --2- 172.21.15.67:0/856488415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65b01002a0 0x7f65b0100680 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:49.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.570+0000 7f65b8644640 1 -- 172.21.15.67:0/856488415 >> 172.21.15.67:0/856488415 conn(0x7f65b00fbb20 msgr2=0x7f65b00fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:49.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.570+0000 7f65b8644640 1 -- 172.21.15.67:0/856488415 shutdown_connections 2024-04-23T00:05:49.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.570+0000 7f65b8644640 1 -- 172.21.15.67:0/856488415 wait complete. 2024-04-23T00:05:49.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.570+0000 7f65b8644640 1 Processor -- start 2024-04-23T00:05:49.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.570+0000 7f65b8644640 1 -- start start 2024-04-23T00:05:49.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.571+0000 7f65b8644640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f65b01002a0 0x7f65b00725a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:49.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.571+0000 7f65b8644640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65b0072ae0 0x7f65b0072f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:49.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.571+0000 7f65b8644640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b01a1fa0 con 0x7f65b0072ae0 2024-04-23T00:05:49.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.571+0000 7f65b8644640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b01a20e0 con 0x7f65b01002a0 2024-04-23T00:05:49.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.571+0000 7f65b5bdf640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65b0072ae0 0x7f65b0072f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:49.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.571+0000 7f65b5bdf640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65b0072ae0 0x7f65b0072f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35046/0 (socket says 172.21.15.67:35046) 2024-04-23T00:05:49.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.571+0000 7f65b5bdf640 1 -- 172.21.15.67:0/2612059763 learned_addr learned my addr 172.21.15.67:0/2612059763 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:49.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.571+0000 7f65b63e0640 1 --2- 172.21.15.67:0/2612059763 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f65b01002a0 0x7f65b00725a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:49.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.571+0000 7f65b5bdf640 1 -- 172.21.15.67:0/2612059763 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f65b01002a0 msgr2=0x7f65b00725a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:49.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.571+0000 7f65b5bdf640 1 --2- 172.21.15.67:0/2612059763 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f65b01002a0 0x7f65b00725a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:49.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.571+0000 7f65b5bdf640 1 -- 172.21.15.67:0/2612059763 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65ac009560 con 0x7f65b0072ae0 2024-04-23T00:05:49.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.571+0000 7f65b5bdf640 1 --2- 172.21.15.67:0/2612059763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65b0072ae0 0x7f65b0072f40 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f65a000d720 tx=0x7f65a000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:49.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.571+0000 7f65a77fe640 1 -- 172.21.15.67:0/2612059763 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65a00099f0 con 0x7f65b0072ae0 2024-04-23T00:05:49.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.571+0000 7f65b8644640 1 -- 172.21.15.67:0/2612059763 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65b01a2220 con 0x7f65b0072ae0 2024-04-23T00:05:49.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.571+0000 7f65a77fe640 1 -- 172.21.15.67:0/2612059763 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f65a0009b50 con 0x7f65b0072ae0 2024-04-23T00:05:49.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.571+0000 7f65a77fe640 1 -- 172.21.15.67:0/2612059763 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65a000f5b0 con 0x7f65b0072ae0 2024-04-23T00:05:49.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.571+0000 7f65b8644640 1 -- 172.21.15.67:0/2612059763 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65b01a2780 con 0x7f65b0072ae0 2024-04-23T00:05:49.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.572+0000 7f65b8644640 1 -- 172.21.15.67:0/2612059763 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65b004e040 con 0x7f65b0072ae0 2024-04-23T00:05:49.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.573+0000 7f65a77fe640 1 -- 172.21.15.67:0/2612059763 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f65a0009cc0 con 0x7f65b0072ae0 2024-04-23T00:05:49.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.573+0000 7f65a77fe640 1 --2- 172.21.15.67:0/2612059763 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f659c06ec30 0x7f659c0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:49.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.573+0000 7f65b63e0640 1 --2- 172.21.15.67:0/2612059763 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f659c06ec30 0x7f659c0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:49.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.574+0000 7f65a77fe640 1 -- 172.21.15.67:0/2612059763 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f65a008ee20 con 0x7f65b0072ae0 2024-04-23T00:05:49.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.574+0000 7f65b63e0640 1 --2- 172.21.15.67:0/2612059763 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f659c06ec30 0x7f659c0710f0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f65ac006010 tx=0x7f65ac000bc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:49.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.578+0000 7f65a77fe640 1 -- 172.21.15.67:0/2612059763 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f65a0057a60 con 0x7f65b0072ae0 2024-04-23T00:05:49.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.807+0000 7f65b8644640 1 -- 172.21.15.67:0/2612059763 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 41, "format": "json"} v 0) v1 -- 0x7f65b0100cf0 con 0x7f65b0072ae0 2024-04-23T00:05:49.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.808+0000 7f65a77fe640 1 -- 172.21.15.67:0/2612059763 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 41, "format": "json"}]=0 dumped fsmap epoch 41 v42) v1 ==== 107+0+5248 (secure 0 0 0) 0x7f65a005b120 con 0x7f65b0072ae0 2024-04-23T00:05:49.796 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:05:49.796 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":41,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T23:20:20.823261+0000","modified":"2024-04-23T00:03:37.116623+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34396,"mds_1":34424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34396":{"gid":34396,"name":"cephfs.smithi067.aqlxpv","rank":0,"incarnation":35,"state":"up:active","state_seq":469,"addr":"172.21.15.67:6827/789475804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":789475804},{"type":"v1","addr":"172.21.15.67:6827","nonce":789475804}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34424":{"gid":34424,"name":"cephfs.smithi067.xyazmh","rank":1,"incarnation":35,"state":"up:active","state_seq":464,"addr":"172.21.15.67:6829/927062647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":927062647},{"type":"v1","addr":"172.21.15.67:6829","nonce":927062647}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_44327":{"gid":44327,"name":"cephfs.smithi134.owipad","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.134:6825/1473208408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":1473208408},{"type":"v1","addr":"172.21.15.134:6825","nonce":1473208408}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_44851":{"gid":44851,"name":"cephfs.smithi134.vuxbsj","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.134:6827/1849648622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":1849648622},{"type":"v1","addr":"172.21.15.134:6827","nonce":1849648622}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":34396,"qdb_cluster":[34424,34396]},"id":1}]} 2024-04-23T00:05:49.796 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 41 2024-04-23T00:05:49.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.810+0000 7f65b8644640 1 -- 172.21.15.67:0/2612059763 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f659c06ec30 msgr2=0x7f659c0710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:49.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.810+0000 7f65b8644640 1 --2- 172.21.15.67:0/2612059763 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f659c06ec30 0x7f659c0710f0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f65ac006010 tx=0x7f65ac000bc0 comp rx=0 tx=0).stop 2024-04-23T00:05:49.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.810+0000 7f65b8644640 1 -- 172.21.15.67:0/2612059763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65b0072ae0 msgr2=0x7f65b0072f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:49.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.810+0000 7f65b8644640 1 --2- 172.21.15.67:0/2612059763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65b0072ae0 0x7f65b0072f40 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f65a000d720 tx=0x7f65a000dbe0 comp rx=0 tx=0).stop 2024-04-23T00:05:49.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.810+0000 7f65b8644640 1 -- 172.21.15.67:0/2612059763 shutdown_connections 2024-04-23T00:05:49.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.810+0000 7f65b8644640 1 --2- 172.21.15.67:0/2612059763 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f659c06ec30 0x7f659c0710f0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:49.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.810+0000 7f65b8644640 1 --2- 172.21.15.67:0/2612059763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65b0072ae0 0x7f65b0072f40 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:49.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.810+0000 7f65b8644640 1 --2- 172.21.15.67:0/2612059763 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f65b01002a0 0x7f65b00725a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:49.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.810+0000 7f65b8644640 1 -- 172.21.15.67:0/2612059763 >> 172.21.15.67:0/2612059763 conn(0x7f65b00fbb20 msgr2=0x7f65b00fd360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:49.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.810+0000 7f65b8644640 1 -- 172.21.15.67:0/2612059763 shutdown_connections 2024-04-23T00:05:49.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:49.810+0000 7f65b8644640 1 -- 172.21.15.67:0/2612059763 wait complete. 2024-04-23T00:05:49.808 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-04-23T00:05:49.825 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-04-23T00:05:49.826 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-23T00:05:49.826 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-23T00:05:49.846 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-23T00:05:49.846 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-23T00:05:49.903 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T00:05:50.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.057+0000 7fd04dd3c640 1 -- 172.21.15.67:0/4018935594 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0480fecf0 msgr2=0x7fd0480ff0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:50.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.057+0000 7fd04dd3c640 1 --2- 172.21.15.67:0/4018935594 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0480fecf0 0x7fd0480ff0d0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fd038009880 tx=0x7fd03802bce0 comp rx=0 tx=0).stop 2024-04-23T00:05:50.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.057+0000 7fd04dd3c640 1 -- 172.21.15.67:0/4018935594 shutdown_connections 2024-04-23T00:05:50.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.057+0000 7fd04dd3c640 1 --2- 172.21.15.67:0/4018935594 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd0480ff610 0x7fd04810c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:50.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.057+0000 7fd04dd3c640 1 --2- 172.21.15.67:0/4018935594 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0480fecf0 0x7fd0480ff0d0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:50.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.057+0000 7fd04dd3c640 1 -- 172.21.15.67:0/4018935594 >> 172.21.15.67:0/4018935594 conn(0x7fd0480fa990 msgr2=0x7fd0480fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:50.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.057+0000 7fd04dd3c640 1 -- 172.21.15.67:0/4018935594 shutdown_connections 2024-04-23T00:05:50.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.057+0000 7fd04dd3c640 1 -- 172.21.15.67:0/4018935594 wait complete. 2024-04-23T00:05:50.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.058+0000 7fd04dd3c640 1 Processor -- start 2024-04-23T00:05:50.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.058+0000 7fd04dd3c640 1 -- start start 2024-04-23T00:05:50.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.058+0000 7fd04dd3c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0480fecf0 0x7fd048197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:50.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.058+0000 7fd04dd3c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd0480ff610 0x7fd048198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:50.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.058+0000 7fd04dd3c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0481989d0 con 0x7fd0480fecf0 2024-04-23T00:05:50.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.058+0000 7fd04dd3c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04819c7a0 con 0x7fd0480ff610 2024-04-23T00:05:50.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.058+0000 7fd0477fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0480fecf0 0x7fd048197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:50.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.058+0000 7fd0477fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0480fecf0 0x7fd048197e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35066/0 (socket says 172.21.15.67:35066) 2024-04-23T00:05:50.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.058+0000 7fd0477fe640 1 -- 172.21.15.67:0/2782649388 learned_addr learned my addr 172.21.15.67:0/2782649388 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:50.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.059+0000 7fd0477fe640 1 -- 172.21.15.67:0/2782649388 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd0480ff610 msgr2=0x7fd048198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:50.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.059+0000 7fd046ffd640 1 --2- 172.21.15.67:0/2782649388 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd0480ff610 0x7fd048198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:50.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.059+0000 7fd0477fe640 1 --2- 172.21.15.67:0/2782649388 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd0480ff610 0x7fd048198340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:50.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.059+0000 7fd0477fe640 1 -- 172.21.15.67:0/2782649388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd038009560 con 0x7fd0480fecf0 2024-04-23T00:05:50.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.059+0000 7fd0477fe640 1 --2- 172.21.15.67:0/2782649388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0480fecf0 0x7fd048197e00 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fd038009230 tx=0x7fd03802ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:50.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.059+0000 7fd044ff9640 1 -- 172.21.15.67:0/2782649388 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd038004520 con 0x7fd0480fecf0 2024-04-23T00:05:50.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.059+0000 7fd044ff9640 1 -- 172.21.15.67:0/2782649388 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd038003ab0 con 0x7fd0480fecf0 2024-04-23T00:05:50.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.059+0000 7fd04dd3c640 1 -- 172.21.15.67:0/2782649388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd04819ca20 con 0x7fd0480fecf0 2024-04-23T00:05:50.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.059+0000 7fd044ff9640 1 -- 172.21.15.67:0/2782649388 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd038005690 con 0x7fd0480fecf0 2024-04-23T00:05:50.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.059+0000 7fd04dd3c640 1 -- 172.21.15.67:0/2782649388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd04819cf80 con 0x7fd0480fecf0 2024-04-23T00:05:50.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.060+0000 7fd04dd3c640 1 -- 172.21.15.67:0/2782649388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd04804e040 con 0x7fd0480fecf0 2024-04-23T00:05:50.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.060+0000 7fd044ff9640 1 -- 172.21.15.67:0/2782649388 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd038004ed0 con 0x7fd0480fecf0 2024-04-23T00:05:50.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.061+0000 7fd044ff9640 1 --2- 172.21.15.67:0/2782649388 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd03006ec90 0x7fd030071150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:50.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.061+0000 7fd046ffd640 1 --2- 172.21.15.67:0/2782649388 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd03006ec90 0x7fd030071150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:50.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.061+0000 7fd044ff9640 1 -- 172.21.15.67:0/2782649388 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fd0380b1b80 con 0x7fd0480fecf0 2024-04-23T00:05:50.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.062+0000 7fd046ffd640 1 --2- 172.21.15.67:0/2782649388 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd03006ec90 0x7fd030071150 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fd03c005fd0 tx=0x7fd03c005850 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:50.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.066+0000 7fd044ff9640 1 -- 172.21.15.67:0/2782649388 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fd03807a7c0 con 0x7fd0480fecf0 2024-04-23T00:05:50.230 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:49 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3311446751' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 39, "format": "json"}]: dispatch 2024-04-23T00:05:50.230 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:49 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1232691537' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 40, "format": "json"}]: dispatch 2024-04-23T00:05:50.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.255+0000 7fd04dd3c640 1 -- 172.21.15.67:0/2782649388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd048109a40 con 0x7fd0480fecf0 2024-04-23T00:05:50.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.256+0000 7fd044ff9640 1 -- 172.21.15.67:0/2782649388 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+17185 (secure 0 0 0) 0x7fd03807de80 con 0x7fd0480fecf0 2024-04-23T00:05:50.244 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:05:50.244 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":94,"fsid":"d46c346e-00fd-11ef-bc93-c7b262605968","created":"2024-04-22T23:13:18.687917+0000","modified":"2024-04-22T23:48:20.827933+0000","last_up_change":"2024-04-22T23:32:23.175924+0000","last_in_change":"2024-04-22T23:18:41.696935+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T23:17:47.912850+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-22T23:20:19.759956+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-22T23:20:20.252379+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"25c9acd1-5743-4dc0-aee0-c7bb8884c0a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":49,"up_from":53,"up_thru":86,"down_at":50,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1540469345},{"type":"v1","addr":"172.21.15.67:6803","nonce":1540469345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1540469345},{"type":"v1","addr":"172.21.15.67:6805","nonce":1540469345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1540469345},{"type":"v1","addr":"172.21.15.67:6809","nonce":1540469345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1540469345},{"type":"v1","addr":"172.21.15.67:6807","nonce":1540469345}]},"public_addr":"172.21.15.67:6803/1540469345","cluster_addr":"172.21.15.67:6805/1540469345","heartbeat_back_addr":"172.21.15.67:6809/1540469345","heartbeat_front_addr":"172.21.15.67:6807/1540469345","state":["exists","up"]},{"osd":1,"uuid":"8be83014-01af-44d3-ab2f-b27840506e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":54,"up_from":63,"up_thru":86,"down_at":55,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1745058179},{"type":"v1","addr":"172.21.15.67:6811","nonce":1745058179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1745058179},{"type":"v1","addr":"172.21.15.67:6813","nonce":1745058179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1745058179},{"type":"v1","addr":"172.21.15.67:6817","nonce":1745058179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1745058179},{"type":"v1","addr":"172.21.15.67:6815","nonce":1745058179}]},"public_addr":"172.21.15.67:6811/1745058179","cluster_addr":"172.21.15.67:6813/1745058179","heartbeat_back_addr":"172.21.15.67:6817/1745058179","heartbeat_front_addr":"172.21.15.67:6815/1745058179","state":["exists","up"]},{"osd":2,"uuid":"ab1cbcda-14dd-4c24-876b-25ad5d4d8563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":67,"up_from":71,"up_thru":86,"down_at":68,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":121705471},{"type":"v1","addr":"172.21.15.67:6819","nonce":121705471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":121705471},{"type":"v1","addr":"172.21.15.67:6821","nonce":121705471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":121705471},{"type":"v1","addr":"172.21.15.67:6825","nonce":121705471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":121705471},{"type":"v1","addr":"172.21.15.67:6823","nonce":121705471}]},"public_addr":"172.21.15.67:6819/121705471","cluster_addr":"172.21.15.67:6821/121705471","heartbeat_back_addr":"172.21.15.67:6825/121705471","heartbeat_front_addr":"172.21.15.67:6823/121705471","state":["exists","up"]},{"osd":3,"uuid":"1a30b6c3-95bf-4b19-883b-1271ff359e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":72,"up_from":76,"up_thru":86,"down_at":73,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":3344693085},{"type":"v1","addr":"172.21.15.134:6801","nonce":3344693085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":3344693085},{"type":"v1","addr":"172.21.15.134:6803","nonce":3344693085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":3344693085},{"type":"v1","addr":"172.21.15.134:6807","nonce":3344693085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":3344693085},{"type":"v1","addr":"172.21.15.134:6805","nonce":3344693085}]},"public_addr":"172.21.15.134:6801/3344693085","cluster_addr":"172.21.15.134:6803/3344693085","heartbeat_back_addr":"172.21.15.134:6807/3344693085","heartbeat_front_addr":"172.21.15.134:6805/3344693085","state":["exists","up"]},{"osd":4,"uuid":"53368fc1-28fd-4622-bc9c-1d8901b06bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":34,"last_clean_end":77,"up_from":81,"up_thru":86,"down_at":78,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":3935208416},{"type":"v1","addr":"172.21.15.134:6809","nonce":3935208416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":3935208416},{"type":"v1","addr":"172.21.15.134:6811","nonce":3935208416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":3935208416},{"type":"v1","addr":"172.21.15.134:6815","nonce":3935208416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":3935208416},{"type":"v1","addr":"172.21.15.134:6813","nonce":3935208416}]},"public_addr":"172.21.15.134:6809/3935208416","cluster_addr":"172.21.15.134:6811/3935208416","heartbeat_back_addr":"172.21.15.134:6815/3935208416","heartbeat_front_addr":"172.21.15.134:6813/3935208416","state":["exists","up"]},{"osd":5,"uuid":"31a885f7-8f91-4b6a-922a-065753f70a95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":82,"up_from":86,"up_thru":86,"down_at":83,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":921342983},{"type":"v1","addr":"172.21.15.134:6817","nonce":921342983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":921342983},{"type":"v1","addr":"172.21.15.134:6819","nonce":921342983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":921342983},{"type":"v1","addr":"172.21.15.134:6823","nonce":921342983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":921342983},{"type":"v1","addr":"172.21.15.134:6821","nonce":921342983}]},"public_addr":"172.21.15.134:6817/921342983","cluster_addr":"172.21.15.134:6819/921342983","heartbeat_back_addr":"172.21.15.134:6823/921342983","heartbeat_front_addr":"172.21.15.134:6821/921342983","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:16:51.179651+0000","dead_epoch":49},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:17:18.694683+0000","dead_epoch":54},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:17:45.368963+0000","dead_epoch":67},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:18:08.585046+0000","dead_epoch":72},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:18:33.412804+0000","dead_epoch":77},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:18:58.909758+0000","dead_epoch":82}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.2","mappings":[{"from":0,"to":2}]},{"pgid":"2.8","mappings":[{"from":0,"to":1}]},{"pgid":"2.b","mappings":[{"from":3,"to":1}]},{"pgid":"2.c","mappings":[{"from":3,"to":1}]},{"pgid":"2.f","mappings":[{"from":0,"to":1}]},{"pgid":"2.16","mappings":[{"from":3,"to":1}]},{"pgid":"3.1","mappings":[{"from":4,"to":1}]},{"pgid":"3.7","mappings":[{"from":4,"to":0}]},{"pgid":"3.c","mappings":[{"from":3,"to":1}]},{"pgid":"3.12","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6828/3606151860":"2024-04-23T23:32:27.027372+0000","172.21.15.134:6824/535957142":"2024-04-23T23:32:27.027388+0000","172.21.15.67:0/1189311107":"2024-04-23T23:15:37.396154+0000","172.21.15.67:6827/2381214817":"2024-04-23T23:20:37.496135+0000","172.21.15.67:6801/2590792187":"2024-04-23T23:15:37.396154+0000","172.21.15.67:6826/2381214817":"2024-04-23T23:20:37.496135+0000","172.21.15.134:0/3526908760":"2024-04-23T23:25:00.458625+0000","172.21.15.134:0/1536532822":"2024-04-23T23:25:00.458625+0000","172.21.15.134:6825/535957142":"2024-04-23T23:32:27.027388+0000","172.21.15.67:0/895378312":"2024-04-23T23:13:56.301645+0000","172.21.15.134:6828/1077503272":"2024-04-23T23:25:00.458625+0000","172.21.15.67:0/1125624760":"2024-04-23T23:15:37.396154+0000","172.21.15.67:0/3387250764":"2024-04-23T23:13:56.301645+0000","172.21.15.67:0/3921081246":"2024-04-23T23:27:55.947537+0000","172.21.15.67:0/2204477458":"2024-04-23T23:23:56.782522+0000","172.21.15.67:0/3730369564":"2024-04-23T23:27:55.947537+0000","172.21.15.134:0/2385923868":"2024-04-23T23:25:00.458625+0000","172.21.15.67:6800/2589747850":"2024-04-23T23:23:56.782522+0000","172.21.15.67:6801/2589747850":"2024-04-23T23:23:56.782522+0000","172.21.15.134:0/2817304592":"2024-04-23T23:25:00.458625+0000","172.21.15.67:6829/3606151860":"2024-04-23T23:32:27.027372+0000","172.21.15.67:0/233140160":"2024-04-23T23:23:56.782522+0000","172.21.15.67:0/3750652317":"2024-04-23T23:23:56.782522+0000","172.21.15.67:6800/2451437776":"2024-04-23T23:13:56.301645+0000","172.21.15.67:0/2665297739":"2024-04-23T23:14:29.214977+0000","172.21.15.67:6800/3568049827":"2024-04-23T23:14:29.214977+0000","172.21.15.67:0/4165654004":"2024-04-23T23:14:29.214977+0000","172.21.15.134:0/3046392865":"2024-04-23T23:25:00.458625+0000","172.21.15.67:0/2052573085":"2024-04-23T23:14:29.214977+0000","172.21.15.134:0/1251980938":"2024-04-23T23:25:00.458625+0000","172.21.15.67:6801/2451437776":"2024-04-23T23:13:56.301645+0000","172.21.15.67:0/4286433529":"2024-04-23T23:27:55.947537+0000","172.21.15.67:0/853292259":"2024-04-23T23:13:56.301645+0000","172.21.15.67:6801/3568049827":"2024-04-23T23:14:29.214977+0000","172.21.15.67:6800/2590792187":"2024-04-23T23:15:37.396154+0000","172.21.15.67:0/1933271294":"2024-04-23T23:23:56.782522+0000","172.21.15.134:6829/1077503272":"2024-04-23T23:25:00.458625+0000","172.21.15.67:0/1737320548":"2024-04-23T23:27:55.947537+0000","172.21.15.67:6800/2306276228":"2024-04-23T23:27:55.947537+0000","172.21.15.67:0/2137374599":"2024-04-23T23:15:37.396154+0000","172.21.15.67:6801/2306276228":"2024-04-23T23:27:55.947537+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T00:05:50.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.258+0000 7fd04dd3c640 1 -- 172.21.15.67:0/2782649388 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd03006ec90 msgr2=0x7fd030071150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:50.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.258+0000 7fd04dd3c640 1 --2- 172.21.15.67:0/2782649388 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd03006ec90 0x7fd030071150 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fd03c005fd0 tx=0x7fd03c005850 comp rx=0 tx=0).stop 2024-04-23T00:05:50.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.258+0000 7fd04dd3c640 1 -- 172.21.15.67:0/2782649388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0480fecf0 msgr2=0x7fd048197e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:50.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.258+0000 7fd04dd3c640 1 --2- 172.21.15.67:0/2782649388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0480fecf0 0x7fd048197e00 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fd038009230 tx=0x7fd03802ce50 comp rx=0 tx=0).stop 2024-04-23T00:05:50.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.258+0000 7fd04dd3c640 1 -- 172.21.15.67:0/2782649388 shutdown_connections 2024-04-23T00:05:50.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.258+0000 7fd04dd3c640 1 --2- 172.21.15.67:0/2782649388 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd03006ec90 0x7fd030071150 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:50.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.258+0000 7fd04dd3c640 1 --2- 172.21.15.67:0/2782649388 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd0480ff610 0x7fd048198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:50.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.258+0000 7fd04dd3c640 1 --2- 172.21.15.67:0/2782649388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0480fecf0 0x7fd048197e00 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:50.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.258+0000 7fd04dd3c640 1 -- 172.21.15.67:0/2782649388 >> 172.21.15.67:0/2782649388 conn(0x7fd0480fa990 msgr2=0x7fd0480fbfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:50.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.258+0000 7fd04dd3c640 1 -- 172.21.15.67:0/2782649388 shutdown_connections 2024-04-23T00:05:50.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.259+0000 7fd04dd3c640 1 -- 172.21.15.67:0/2782649388 wait complete. 2024-04-23T00:05:50.255 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:49 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3311446751' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 39, "format": "json"}]: dispatch 2024-04-23T00:05:50.256 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:49 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1232691537' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 40, "format": "json"}]: dispatch 2024-04-23T00:05:50.257 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T23:17:47.912850+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-22T23:20:19.759956+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-22T23:20:20.252379+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T00:05:50.257 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T00:05:50.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.375+0000 7fd1eb696640 1 -- 172.21.15.67:0/346363380 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1e41002b0 msgr2=0x7fd1e4100690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:50.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.375+0000 7fd1eb696640 1 --2- 172.21.15.67:0/346363380 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1e41002b0 0x7fd1e4100690 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0009880 tx=0x7fd1e002bce0 comp rx=0 tx=0).stop 2024-04-23T00:05:50.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.375+0000 7fd1eb696640 1 -- 172.21.15.67:0/346363380 shutdown_connections 2024-04-23T00:05:50.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.375+0000 7fd1eb696640 1 --2- 172.21.15.67:0/346363380 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd1e4100c60 0x7fd1e4104ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:50.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.375+0000 7fd1eb696640 1 --2- 172.21.15.67:0/346363380 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1e41002b0 0x7fd1e4100690 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:50.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.375+0000 7fd1eb696640 1 -- 172.21.15.67:0/346363380 >> 172.21.15.67:0/346363380 conn(0x7fd1e40fbb50 msgr2=0x7fd1e40fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:50.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.375+0000 7fd1eb696640 1 -- 172.21.15.67:0/346363380 shutdown_connections 2024-04-23T00:05:50.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.375+0000 7fd1eb696640 1 -- 172.21.15.67:0/346363380 wait complete. 2024-04-23T00:05:50.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.375+0000 7fd1eb696640 1 Processor -- start 2024-04-23T00:05:50.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.375+0000 7fd1eb696640 1 -- start start 2024-04-23T00:05:50.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.375+0000 7fd1eb696640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1e41002b0 0x7fd1e40725a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:50.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.375+0000 7fd1eb696640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd1e4072ae0 0x7fd1e4072f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:50.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.375+0000 7fd1eb696640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e41a2020 con 0x7fd1e41002b0 2024-04-23T00:05:50.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.375+0000 7fd1eb696640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e41a2160 con 0x7fd1e4072ae0 2024-04-23T00:05:50.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.376+0000 7fd1e9432640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1e41002b0 0x7fd1e40725a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:50.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.376+0000 7fd1e9432640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1e41002b0 0x7fd1e40725a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35082/0 (socket says 172.21.15.67:35082) 2024-04-23T00:05:50.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.376+0000 7fd1e9432640 1 -- 172.21.15.67:0/3261150512 learned_addr learned my addr 172.21.15.67:0/3261150512 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:50.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.376+0000 7fd1e8c31640 1 --2- 172.21.15.67:0/3261150512 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd1e4072ae0 0x7fd1e4072f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:50.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.376+0000 7fd1e9432640 1 -- 172.21.15.67:0/3261150512 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd1e4072ae0 msgr2=0x7fd1e4072f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:50.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.376+0000 7fd1e9432640 1 --2- 172.21.15.67:0/3261150512 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd1e4072ae0 0x7fd1e4072f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:50.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.376+0000 7fd1e9432640 1 -- 172.21.15.67:0/3261150512 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e0009560 con 0x7fd1e41002b0 2024-04-23T00:05:50.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.376+0000 7fd1e9432640 1 --2- 172.21.15.67:0/3261150512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1e41002b0 0x7fd1e40725a0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0009430 tx=0x7fd1e0005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:50.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.376+0000 7fd1da7fc640 1 -- 172.21.15.67:0/3261150512 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1e00044a0 con 0x7fd1e41002b0 2024-04-23T00:05:50.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.376+0000 7fd1eb696640 1 -- 172.21.15.67:0/3261150512 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1e41a22a0 con 0x7fd1e41002b0 2024-04-23T00:05:50.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.376+0000 7fd1da7fc640 1 -- 172.21.15.67:0/3261150512 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd1e0003ab0 con 0x7fd1e41002b0 2024-04-23T00:05:50.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.376+0000 7fd1da7fc640 1 -- 172.21.15.67:0/3261150512 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1e00055f0 con 0x7fd1e41002b0 2024-04-23T00:05:50.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.376+0000 7fd1eb696640 1 -- 172.21.15.67:0/3261150512 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1e41a2800 con 0x7fd1e41002b0 2024-04-23T00:05:50.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.377+0000 7fd1eb696640 1 -- 172.21.15.67:0/3261150512 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1e404e040 con 0x7fd1e41002b0 2024-04-23T00:05:50.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.378+0000 7fd1da7fc640 1 -- 172.21.15.67:0/3261150512 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd1e0004e60 con 0x7fd1e41002b0 2024-04-23T00:05:50.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.378+0000 7fd1da7fc640 1 --2- 172.21.15.67:0/3261150512 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd1d006ec30 0x7fd1d00710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:50.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.378+0000 7fd1da7fc640 1 -- 172.21.15.67:0/3261150512 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fd1e00b1350 con 0x7fd1e41002b0 2024-04-23T00:05:50.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.378+0000 7fd1e8c31640 1 --2- 172.21.15.67:0/3261150512 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd1d006ec30 0x7fd1d00710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:50.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.379+0000 7fd1e8c31640 1 --2- 172.21.15.67:0/3261150512 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd1d006ec30 0x7fd1d00710f0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fd1e40695f0 tx=0x7fd1d4005f90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:50.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.383+0000 7fd1da7fc640 1 -- 172.21.15.67:0/3261150512 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fd1e007a040 con 0x7fd1e41002b0 2024-04-23T00:05:50.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.569+0000 7fd1eb696640 1 -- 172.21.15.67:0/3261150512 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd1e4100da0 con 0x7fd1e41002b0 2024-04-23T00:05:50.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.570+0000 7fd1da7fc640 1 -- 172.21.15.67:0/3261150512 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd1e007d700 con 0x7fd1e41002b0 2024-04-23T00:05:50.557 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-23T00:05:50.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.572+0000 7fd1eb696640 1 -- 172.21.15.67:0/3261150512 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd1d006ec30 msgr2=0x7fd1d00710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:50.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.572+0000 7fd1eb696640 1 --2- 172.21.15.67:0/3261150512 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd1d006ec30 0x7fd1d00710f0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fd1e40695f0 tx=0x7fd1d4005f90 comp rx=0 tx=0).stop 2024-04-23T00:05:50.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.572+0000 7fd1eb696640 1 -- 172.21.15.67:0/3261150512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1e41002b0 msgr2=0x7fd1e40725a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:50.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.572+0000 7fd1eb696640 1 --2- 172.21.15.67:0/3261150512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1e41002b0 0x7fd1e40725a0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0009430 tx=0x7fd1e0005e50 comp rx=0 tx=0).stop 2024-04-23T00:05:50.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.572+0000 7fd1eb696640 1 -- 172.21.15.67:0/3261150512 shutdown_connections 2024-04-23T00:05:50.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.572+0000 7fd1eb696640 1 --2- 172.21.15.67:0/3261150512 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fd1d006ec30 0x7fd1d00710f0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:50.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.572+0000 7fd1eb696640 1 --2- 172.21.15.67:0/3261150512 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd1e4072ae0 0x7fd1e4072f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:50.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.572+0000 7fd1eb696640 1 --2- 172.21.15.67:0/3261150512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1e41002b0 0x7fd1e40725a0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:50.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.572+0000 7fd1eb696640 1 -- 172.21.15.67:0/3261150512 >> 172.21.15.67:0/3261150512 conn(0x7fd1e40fbb50 msgr2=0x7fd1e40fd340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:50.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.572+0000 7fd1eb696640 1 -- 172.21.15.67:0/3261150512 shutdown_connections 2024-04-23T00:05:50.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.572+0000 7fd1eb696640 1 -- 172.21.15.67:0/3261150512 wait complete. 2024-04-23T00:05:50.570 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-23T00:05:50.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.688+0000 7fbd6ca21640 1 -- 172.21.15.67:0/697114497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd68101b20 msgr2=0x7fbd6810a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:50.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.688+0000 7fbd6ca21640 1 --2- 172.21.15.67:0/697114497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd68101b20 0x7fbd6810a000 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c009880 tx=0x7fbd5c02bd70 comp rx=0 tx=0).stop 2024-04-23T00:05:50.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.688+0000 7fbd6ca21640 1 -- 172.21.15.67:0/697114497 shutdown_connections 2024-04-23T00:05:50.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.688+0000 7fbd6ca21640 1 --2- 172.21.15.67:0/697114497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd68101b20 0x7fbd6810a000 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:50.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.688+0000 7fbd6ca21640 1 --2- 172.21.15.67:0/697114497 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd68101200 0x7fbd681015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:50.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.688+0000 7fbd6ca21640 1 -- 172.21.15.67:0/697114497 >> 172.21.15.67:0/697114497 conn(0x7fbd680faa20 msgr2=0x7fbd680fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:50.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.688+0000 7fbd6ca21640 1 -- 172.21.15.67:0/697114497 shutdown_connections 2024-04-23T00:05:50.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.688+0000 7fbd6ca21640 1 -- 172.21.15.67:0/697114497 wait complete. 2024-04-23T00:05:50.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.689+0000 7fbd6ca21640 1 Processor -- start 2024-04-23T00:05:50.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.689+0000 7fbd6ca21640 1 -- start start 2024-04-23T00:05:50.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.689+0000 7fbd6ca21640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd68101200 0x7fbd68195d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:50.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.689+0000 7fbd6ca21640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd68101b20 0x7fbd68196270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:50.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.689+0000 7fbd6ca21640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd68196900 con 0x7fbd68101200 2024-04-23T00:05:50.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.689+0000 7fbd6ca21640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6819a6d0 con 0x7fbd68101b20 2024-04-23T00:05:50.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.689+0000 7fbd6659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd68101200 0x7fbd68195d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:50.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.689+0000 7fbd6659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd68101200 0x7fbd68195d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35110/0 (socket says 172.21.15.67:35110) 2024-04-23T00:05:50.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.689+0000 7fbd6659c640 1 -- 172.21.15.67:0/3438771760 learned_addr learned my addr 172.21.15.67:0/3438771760 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:50.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.689+0000 7fbd65d9b640 1 --2- 172.21.15.67:0/3438771760 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd68101b20 0x7fbd68196270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:50.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.689+0000 7fbd6659c640 1 -- 172.21.15.67:0/3438771760 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd68101b20 msgr2=0x7fbd68196270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:50.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.690+0000 7fbd6659c640 1 --2- 172.21.15.67:0/3438771760 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd68101b20 0x7fbd68196270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:50.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.690+0000 7fbd6659c640 1 -- 172.21.15.67:0/3438771760 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5c009560 con 0x7fbd68101200 2024-04-23T00:05:50.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.690+0000 7fbd65d9b640 1 --2- 172.21.15.67:0/3438771760 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd68101b20 0x7fbd68196270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T00:05:50.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.690+0000 7fbd6659c640 1 --2- 172.21.15.67:0/3438771760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd68101200 0x7fbd68195d30 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fbd5800d8d0 tx=0x7fbd5800dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:50.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.690+0000 7fbd577fe640 1 -- 172.21.15.67:0/3438771760 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd58009c00 con 0x7fbd68101200 2024-04-23T00:05:50.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.690+0000 7fbd577fe640 1 -- 172.21.15.67:0/3438771760 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbd58009d60 con 0x7fbd68101200 2024-04-23T00:05:50.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.690+0000 7fbd6ca21640 1 -- 172.21.15.67:0/3438771760 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd6819a9b0 con 0x7fbd68101200 2024-04-23T00:05:50.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.690+0000 7fbd577fe640 1 -- 172.21.15.67:0/3438771760 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd580049d0 con 0x7fbd68101200 2024-04-23T00:05:50.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.690+0000 7fbd6ca21640 1 -- 172.21.15.67:0/3438771760 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd6819af10 con 0x7fbd68101200 2024-04-23T00:05:50.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.691+0000 7fbd6ca21640 1 -- 172.21.15.67:0/3438771760 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd6804e040 con 0x7fbd68101200 2024-04-23T00:05:50.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.691+0000 7fbd577fe640 1 -- 172.21.15.67:0/3438771760 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbd58004c90 con 0x7fbd68101200 2024-04-23T00:05:50.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.692+0000 7fbd577fe640 1 --2- 172.21.15.67:0/3438771760 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbd5006ec30 0x7fbd500710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:50.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.692+0000 7fbd577fe640 1 -- 172.21.15.67:0/3438771760 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fbd5808ed40 con 0x7fbd68101200 2024-04-23T00:05:50.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.692+0000 7fbd65d9b640 1 --2- 172.21.15.67:0/3438771760 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbd5006ec30 0x7fbd500710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:50.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.692+0000 7fbd65d9b640 1 --2- 172.21.15.67:0/3438771760 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbd5006ec30 0x7fbd500710f0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c005ec0 tx=0x7fbd5c000bc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:50.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.696+0000 7fbd577fe640 1 -- 172.21.15.67:0/3438771760 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fbd58057980 con 0x7fbd68101200 2024-04-23T00:05:50.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.884+0000 7fbd6ca21640 1 -- 172.21.15.67:0/3438771760 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7fbd68107670 con 0x7fbd68101200 2024-04-23T00:05:50.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.884+0000 7fbd577fe640 1 -- 172.21.15.67:0/3438771760 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v94) v1 ==== 107+0+11 (secure 0 0 0) 0x7fbd5805b040 con 0x7fbd68101200 2024-04-23T00:05:50.873 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-23T00:05:50.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.886+0000 7fbd6ca21640 1 -- 172.21.15.67:0/3438771760 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbd5006ec30 msgr2=0x7fbd500710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:50.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.886+0000 7fbd6ca21640 1 --2- 172.21.15.67:0/3438771760 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbd5006ec30 0x7fbd500710f0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c005ec0 tx=0x7fbd5c000bc0 comp rx=0 tx=0).stop 2024-04-23T00:05:50.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.886+0000 7fbd6ca21640 1 -- 172.21.15.67:0/3438771760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd68101200 msgr2=0x7fbd68195d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:50.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.886+0000 7fbd6ca21640 1 --2- 172.21.15.67:0/3438771760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd68101200 0x7fbd68195d30 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fbd5800d8d0 tx=0x7fbd5800dd90 comp rx=0 tx=0).stop 2024-04-23T00:05:50.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.886+0000 7fbd6ca21640 1 -- 172.21.15.67:0/3438771760 shutdown_connections 2024-04-23T00:05:50.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.886+0000 7fbd6ca21640 1 --2- 172.21.15.67:0/3438771760 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbd5006ec30 0x7fbd500710f0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:50.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.886+0000 7fbd6ca21640 1 --2- 172.21.15.67:0/3438771760 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbd68101b20 0x7fbd68196270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:50.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.886+0000 7fbd6ca21640 1 --2- 172.21.15.67:0/3438771760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd68101200 0x7fbd68195d30 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:50.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.886+0000 7fbd6ca21640 1 -- 172.21.15.67:0/3438771760 >> 172.21.15.67:0/3438771760 conn(0x7fbd680faa20 msgr2=0x7fbd68104840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:50.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.886+0000 7fbd6ca21640 1 -- 172.21.15.67:0/3438771760 shutdown_connections 2024-04-23T00:05:50.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:50.886+0000 7fbd6ca21640 1 -- 172.21.15.67:0/3438771760 wait complete. 2024-04-23T00:05:50.884 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:50 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2612059763' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 41, "format": "json"}]: dispatch 2024-04-23T00:05:50.884 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:50 smithi067 ceph-mon[112438]: pgmap v1188: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:05:50.884 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:50 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2782649388' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T00:05:50.884 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:50 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3261150512' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T00:05:50.886 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-23T00:05:50.979 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:50 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2612059763' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 41, "format": "json"}]: dispatch 2024-04-23T00:05:50.980 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:50 smithi134 ceph-mon[85630]: pgmap v1188: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:05:50.980 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:50 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2782649388' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T00:05:50.980 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:50 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3261150512' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T00:05:50.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.003+0000 7f3ca0a69640 1 -- 172.21.15.67:0/2970706036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c9c0675c0 msgr2=0x7f3c9c0679a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:50.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.003+0000 7f3ca0a69640 1 --2- 172.21.15.67:0/2970706036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c9c0675c0 0x7f3c9c0679a0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f3c94009880 tx=0x7f3c9402bce0 comp rx=0 tx=0).stop 2024-04-23T00:05:50.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.003+0000 7f3ca0a69640 1 -- 172.21.15.67:0/2970706036 shutdown_connections 2024-04-23T00:05:50.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.003+0000 7f3ca0a69640 1 --2- 172.21.15.67:0/2970706036 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c9c067f70 0x7f3c9c104c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:50.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.003+0000 7f3ca0a69640 1 --2- 172.21.15.67:0/2970706036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c9c0675c0 0x7f3c9c0679a0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:50.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.003+0000 7f3ca0a69640 1 -- 172.21.15.67:0/2970706036 >> 172.21.15.67:0/2970706036 conn(0x7f3c9c074a50 msgr2=0x7f3c9c074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:50.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.003+0000 7f3ca0a69640 1 -- 172.21.15.67:0/2970706036 shutdown_connections 2024-04-23T00:05:50.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.003+0000 7f3ca0a69640 1 -- 172.21.15.67:0/2970706036 wait complete. 2024-04-23T00:05:50.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.003+0000 7f3ca0a69640 1 Processor -- start 2024-04-23T00:05:50.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.003+0000 7f3ca0a69640 1 -- start start 2024-04-23T00:05:50.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.004+0000 7f3ca0a69640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c9c0675c0 0x7f3c9c195db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:50.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.004+0000 7f3ca0a69640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c9c1962f0 0x7f3c9c19a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:50.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.004+0000 7f3ca0a69640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9c1968f0 con 0x7f3c9c0675c0 2024-04-23T00:05:50.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.004+0000 7f3ca0a69640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9c196a60 con 0x7f3c9c1962f0 2024-04-23T00:05:50.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.004+0000 7f3c9a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c9c0675c0 0x7f3c9c195db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:50.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.004+0000 7f3c9a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c9c0675c0 0x7f3c9c195db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35126/0 (socket says 172.21.15.67:35126) 2024-04-23T00:05:50.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.004+0000 7f3c9a59c640 1 -- 172.21.15.67:0/4224743149 learned_addr learned my addr 172.21.15.67:0/4224743149 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:50.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.004+0000 7f3c99d9b640 1 --2- 172.21.15.67:0/4224743149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c9c1962f0 0x7f3c9c19a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:50.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.004+0000 7f3c9a59c640 1 -- 172.21.15.67:0/4224743149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c9c1962f0 msgr2=0x7f3c9c19a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:50.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.004+0000 7f3c9a59c640 1 --2- 172.21.15.67:0/4224743149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c9c1962f0 0x7f3c9c19a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:50.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.004+0000 7f3c9a59c640 1 -- 172.21.15.67:0/4224743149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c94009560 con 0x7f3c9c0675c0 2024-04-23T00:05:50.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.004+0000 7f3c99d9b640 1 --2- 172.21.15.67:0/4224743149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c9c1962f0 0x7f3c9c19a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T00:05:50.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.004+0000 7f3c9a59c640 1 --2- 172.21.15.67:0/4224743149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c9c0675c0 0x7f3c9c195db0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f3c94009430 tx=0x7f3c94005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:50.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.004+0000 7f3c8b7fe640 1 -- 172.21.15.67:0/4224743149 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c940044a0 con 0x7f3c9c0675c0 2024-04-23T00:05:50.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.004+0000 7f3ca0a69640 1 -- 172.21.15.67:0/4224743149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c9c19acf0 con 0x7f3c9c0675c0 2024-04-23T00:05:50.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.004+0000 7f3c8b7fe640 1 -- 172.21.15.67:0/4224743149 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3c94003ab0 con 0x7f3c9c0675c0 2024-04-23T00:05:50.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.004+0000 7f3c8b7fe640 1 -- 172.21.15.67:0/4224743149 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c940055f0 con 0x7f3c9c0675c0 2024-04-23T00:05:50.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.005+0000 7f3ca0a69640 1 -- 172.21.15.67:0/4224743149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c9c19b250 con 0x7f3c9c0675c0 2024-04-23T00:05:50.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.005+0000 7f3ca0a69640 1 -- 172.21.15.67:0/4224743149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c9c04e040 con 0x7f3c9c0675c0 2024-04-23T00:05:50.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.006+0000 7f3c8b7fe640 1 -- 172.21.15.67:0/4224743149 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3c94004e60 con 0x7f3c9c0675c0 2024-04-23T00:05:50.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.006+0000 7f3c8b7fe640 1 --2- 172.21.15.67:0/4224743149 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3c8406ec30 0x7f3c840710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:50.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.006+0000 7f3c8b7fe640 1 -- 172.21.15.67:0/4224743149 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f3c940b14a0 con 0x7f3c9c0675c0 2024-04-23T00:05:50.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.006+0000 7f3c99d9b640 1 --2- 172.21.15.67:0/4224743149 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3c8406ec30 0x7f3c840710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:50.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.007+0000 7f3c99d9b640 1 --2- 172.21.15.67:0/4224743149 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3c8406ec30 0x7f3c840710f0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c072e40 tx=0x7f3c90005f90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:50.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.011+0000 7f3c8b7fe640 1 -- 172.21.15.67:0/4224743149 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f3c9407a040 con 0x7f3c9c0675c0 2024-04-23T00:05:51.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.197+0000 7f3ca0a69640 1 -- 172.21.15.67:0/4224743149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f3c9c197100 con 0x7f3c9c0675c0 2024-04-23T00:05:51.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.198+0000 7f3c8b7fe640 1 -- 172.21.15.67:0/4224743149 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v94) v1 ==== 107+0+11 (secure 0 0 0) 0x7f3c9407d700 con 0x7f3c9c0675c0 2024-04-23T00:05:51.185 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-23T00:05:51.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.200+0000 7f3ca0a69640 1 -- 172.21.15.67:0/4224743149 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3c8406ec30 msgr2=0x7f3c840710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:51.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.200+0000 7f3ca0a69640 1 --2- 172.21.15.67:0/4224743149 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3c8406ec30 0x7f3c840710f0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c072e40 tx=0x7f3c90005f90 comp rx=0 tx=0).stop 2024-04-23T00:05:51.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.200+0000 7f3ca0a69640 1 -- 172.21.15.67:0/4224743149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c9c0675c0 msgr2=0x7f3c9c195db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:51.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.200+0000 7f3ca0a69640 1 --2- 172.21.15.67:0/4224743149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c9c0675c0 0x7f3c9c195db0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f3c94009430 tx=0x7f3c94005e50 comp rx=0 tx=0).stop 2024-04-23T00:05:51.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.200+0000 7f3ca0a69640 1 -- 172.21.15.67:0/4224743149 shutdown_connections 2024-04-23T00:05:51.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.200+0000 7f3ca0a69640 1 --2- 172.21.15.67:0/4224743149 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f3c8406ec30 0x7f3c840710f0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:51.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.200+0000 7f3ca0a69640 1 --2- 172.21.15.67:0/4224743149 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c9c1962f0 0x7f3c9c19a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:51.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.200+0000 7f3ca0a69640 1 --2- 172.21.15.67:0/4224743149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c9c0675c0 0x7f3c9c195db0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:51.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.200+0000 7f3ca0a69640 1 -- 172.21.15.67:0/4224743149 >> 172.21.15.67:0/4224743149 conn(0x7f3c9c074a50 msgr2=0x7f3c9c0fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:51.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.200+0000 7f3ca0a69640 1 -- 172.21.15.67:0/4224743149 shutdown_connections 2024-04-23T00:05:51.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.200+0000 7f3ca0a69640 1 -- 172.21.15.67:0/4224743149 wait complete. 2024-04-23T00:05:51.199 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-23T00:05:51.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.314+0000 7fbcee84a640 1 -- 172.21.15.67:0/3570353862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbce80fe770 msgr2=0x7fbce80febb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:51.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.314+0000 7fbcee84a640 1 --2- 172.21.15.67:0/3570353862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbce80fe770 0x7fbce80febb0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fbcd0009880 tx=0x7fbcd002bce0 comp rx=0 tx=0).stop 2024-04-23T00:05:51.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.314+0000 7fbcee84a640 1 -- 172.21.15.67:0/3570353862 shutdown_connections 2024-04-23T00:05:51.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.314+0000 7fbcee84a640 1 --2- 172.21.15.67:0/3570353862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbce80fe770 0x7fbce80febb0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:51.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.314+0000 7fbcee84a640 1 --2- 172.21.15.67:0/3570353862 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbce8104700 0x7fbce8104ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:51.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.314+0000 7fbcee84a640 1 -- 172.21.15.67:0/3570353862 >> 172.21.15.67:0/3570353862 conn(0x7fbce80752f0 msgr2=0x7fbce8075700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:51.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.314+0000 7fbcee84a640 1 -- 172.21.15.67:0/3570353862 shutdown_connections 2024-04-23T00:05:51.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.314+0000 7fbcee84a640 1 -- 172.21.15.67:0/3570353862 wait complete. 2024-04-23T00:05:51.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.315+0000 7fbcee84a640 1 Processor -- start 2024-04-23T00:05:51.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.315+0000 7fbcee84a640 1 -- start start 2024-04-23T00:05:51.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.315+0000 7fbcee84a640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbce80fe770 0x7fbce819b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:51.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.315+0000 7fbcee84a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbce8104700 0x7fbce819bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:51.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.315+0000 7fbcee84a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbce8195a10 con 0x7fbce8104700 2024-04-23T00:05:51.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.315+0000 7fbcee84a640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbce8195b80 con 0x7fbce80fe770 2024-04-23T00:05:51.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.316+0000 7fbce77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbce8104700 0x7fbce819bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:51.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.316+0000 7fbce77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbce8104700 0x7fbce819bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35148/0 (socket says 172.21.15.67:35148) 2024-04-23T00:05:51.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.316+0000 7fbce77fe640 1 -- 172.21.15.67:0/1677549062 learned_addr learned my addr 172.21.15.67:0/1677549062 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:51.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.316+0000 7fbce7fff640 1 --2- 172.21.15.67:0/1677549062 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbce80fe770 0x7fbce819b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:51.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.316+0000 7fbce77fe640 1 -- 172.21.15.67:0/1677549062 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbce80fe770 msgr2=0x7fbce819b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:51.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.316+0000 7fbce77fe640 1 --2- 172.21.15.67:0/1677549062 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbce80fe770 0x7fbce819b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:51.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.316+0000 7fbce77fe640 1 -- 172.21.15.67:0/1677549062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd0009560 con 0x7fbce8104700 2024-04-23T00:05:51.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.316+0000 7fbce77fe640 1 --2- 172.21.15.67:0/1677549062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbce8104700 0x7fbce819bee0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fbcd0009230 tx=0x7fbcd0004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:51.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.316+0000 7fbce57fa640 1 -- 172.21.15.67:0/1677549062 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbcd00035f0 con 0x7fbce8104700 2024-04-23T00:05:51.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.316+0000 7fbce57fa640 1 -- 172.21.15.67:0/1677549062 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbcd0003c10 con 0x7fbce8104700 2024-04-23T00:05:51.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.316+0000 7fbce57fa640 1 -- 172.21.15.67:0/1677549062 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbcd0005670 con 0x7fbce8104700 2024-04-23T00:05:51.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.316+0000 7fbcee84a640 1 -- 172.21.15.67:0/1677549062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbce8195e00 con 0x7fbce8104700 2024-04-23T00:05:51.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.316+0000 7fbcee84a640 1 -- 172.21.15.67:0/1677549062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbce8196300 con 0x7fbce8104700 2024-04-23T00:05:51.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.317+0000 7fbcee84a640 1 -- 172.21.15.67:0/1677549062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbce804e040 con 0x7fbce8104700 2024-04-23T00:05:51.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.318+0000 7fbce57fa640 1 -- 172.21.15.67:0/1677549062 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbcd0004dd0 con 0x7fbce8104700 2024-04-23T00:05:51.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.318+0000 7fbce57fa640 1 --2- 172.21.15.67:0/1677549062 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbcd406ec90 0x7fbcd4071150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:51.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.318+0000 7fbce57fa640 1 -- 172.21.15.67:0/1677549062 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fbcd00b1330 con 0x7fbce8104700 2024-04-23T00:05:51.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.318+0000 7fbce7fff640 1 --2- 172.21.15.67:0/1677549062 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbcd406ec90 0x7fbcd4071150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:51.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.319+0000 7fbce7fff640 1 --2- 172.21.15.67:0/1677549062 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbcd406ec90 0x7fbcd4071150 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8005fd0 tx=0x7fbcd800b490 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:51.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.323+0000 7fbce57fa640 1 -- 172.21.15.67:0/1677549062 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fbcd007a020 con 0x7fbce8104700 2024-04-23T00:05:51.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.524+0000 7fbcee84a640 1 -- 172.21.15.67:0/1677549062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fbce810b1c0 con 0x7fbce8104700 2024-04-23T00:05:51.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.525+0000 7fbce57fa640 1 -- 172.21.15.67:0/1677549062 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 41 entries v94) v1 ==== 81+0+2437 (secure 0 0 0) 0x7fbcd007d6e0 con 0x7fbce8104700 2024-04-23T00:05:51.512 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6828/3606151860 2024-04-23T23:32:27.027372+0000 2024-04-23T00:05:51.512 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.134:6824/535957142 2024-04-23T23:32:27.027388+0000 2024-04-23T00:05:51.513 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/1189311107 2024-04-23T23:15:37.396154+0000 2024-04-23T00:05:51.513 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6827/2381214817 2024-04-23T23:20:37.496135+0000 2024-04-23T00:05:51.513 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6801/2590792187 2024-04-23T23:15:37.396154+0000 2024-04-23T00:05:51.513 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6826/2381214817 2024-04-23T23:20:37.496135+0000 2024-04-23T00:05:51.513 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.134:0/3526908760 2024-04-23T23:25:00.458625+0000 2024-04-23T00:05:51.513 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.134:0/1536532822 2024-04-23T23:25:00.458625+0000 2024-04-23T00:05:51.513 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.134:6825/535957142 2024-04-23T23:32:27.027388+0000 2024-04-23T00:05:51.513 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/895378312 2024-04-23T23:13:56.301645+0000 2024-04-23T00:05:51.513 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.134:6828/1077503272 2024-04-23T23:25:00.458625+0000 2024-04-23T00:05:51.513 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/1125624760 2024-04-23T23:15:37.396154+0000 2024-04-23T00:05:51.513 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/3387250764 2024-04-23T23:13:56.301645+0000 2024-04-23T00:05:51.513 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/3921081246 2024-04-23T23:27:55.947537+0000 2024-04-23T00:05:51.513 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/2204477458 2024-04-23T23:23:56.782522+0000 2024-04-23T00:05:51.513 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/3730369564 2024-04-23T23:27:55.947537+0000 2024-04-23T00:05:51.513 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.134:0/2385923868 2024-04-23T23:25:00.458625+0000 2024-04-23T00:05:51.514 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6800/2589747850 2024-04-23T23:23:56.782522+0000 2024-04-23T00:05:51.514 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6801/2589747850 2024-04-23T23:23:56.782522+0000 2024-04-23T00:05:51.514 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.134:0/2817304592 2024-04-23T23:25:00.458625+0000 2024-04-23T00:05:51.514 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6829/3606151860 2024-04-23T23:32:27.027372+0000 2024-04-23T00:05:51.514 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/233140160 2024-04-23T23:23:56.782522+0000 2024-04-23T00:05:51.514 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/3750652317 2024-04-23T23:23:56.782522+0000 2024-04-23T00:05:51.514 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6800/2451437776 2024-04-23T23:13:56.301645+0000 2024-04-23T00:05:51.514 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/2665297739 2024-04-23T23:14:29.214977+0000 2024-04-23T00:05:51.514 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6800/3568049827 2024-04-23T23:14:29.214977+0000 2024-04-23T00:05:51.514 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/4165654004 2024-04-23T23:14:29.214977+0000 2024-04-23T00:05:51.514 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.134:0/3046392865 2024-04-23T23:25:00.458625+0000 2024-04-23T00:05:51.514 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/2052573085 2024-04-23T23:14:29.214977+0000 2024-04-23T00:05:51.514 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.134:0/1251980938 2024-04-23T23:25:00.458625+0000 2024-04-23T00:05:51.514 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6801/2451437776 2024-04-23T23:13:56.301645+0000 2024-04-23T00:05:51.514 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/4286433529 2024-04-23T23:27:55.947537+0000 2024-04-23T00:05:51.515 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/853292259 2024-04-23T23:13:56.301645+0000 2024-04-23T00:05:51.515 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6801/3568049827 2024-04-23T23:14:29.214977+0000 2024-04-23T00:05:51.515 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6800/2590792187 2024-04-23T23:15:37.396154+0000 2024-04-23T00:05:51.515 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/1933271294 2024-04-23T23:23:56.782522+0000 2024-04-23T00:05:51.515 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.134:6829/1077503272 2024-04-23T23:25:00.458625+0000 2024-04-23T00:05:51.515 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/1737320548 2024-04-23T23:27:55.947537+0000 2024-04-23T00:05:51.515 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6800/2306276228 2024-04-23T23:27:55.947537+0000 2024-04-23T00:05:51.515 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/2137374599 2024-04-23T23:15:37.396154+0000 2024-04-23T00:05:51.515 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6801/2306276228 2024-04-23T23:27:55.947537+0000 2024-04-23T00:05:51.516 INFO:teuthology.orchestra.run.smithi067.stderr:listed 41 entries 2024-04-23T00:05:51.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.527+0000 7fbcee84a640 1 -- 172.21.15.67:0/1677549062 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbcd406ec90 msgr2=0x7fbcd4071150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:51.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.527+0000 7fbcee84a640 1 --2- 172.21.15.67:0/1677549062 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbcd406ec90 0x7fbcd4071150 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8005fd0 tx=0x7fbcd800b490 comp rx=0 tx=0).stop 2024-04-23T00:05:51.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.527+0000 7fbcee84a640 1 -- 172.21.15.67:0/1677549062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbce8104700 msgr2=0x7fbce819bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:51.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.527+0000 7fbcee84a640 1 --2- 172.21.15.67:0/1677549062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbce8104700 0x7fbce819bee0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fbcd0009230 tx=0x7fbcd0004040 comp rx=0 tx=0).stop 2024-04-23T00:05:51.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.527+0000 7fbcee84a640 1 -- 172.21.15.67:0/1677549062 shutdown_connections 2024-04-23T00:05:51.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.527+0000 7fbcee84a640 1 --2- 172.21.15.67:0/1677549062 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fbcd406ec90 0x7fbcd4071150 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:51.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.527+0000 7fbcee84a640 1 --2- 172.21.15.67:0/1677549062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbce8104700 0x7fbce819bee0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:51.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.527+0000 7fbcee84a640 1 --2- 172.21.15.67:0/1677549062 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbce80fe770 0x7fbce819b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:51.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.527+0000 7fbcee84a640 1 -- 172.21.15.67:0/1677549062 >> 172.21.15.67:0/1677549062 conn(0x7fbce80752f0 msgr2=0x7fbce8101900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:51.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.527+0000 7fbcee84a640 1 -- 172.21.15.67:0/1677549062 shutdown_connections 2024-04-23T00:05:51.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.527+0000 7fbcee84a640 1 -- 172.21.15.67:0/1677549062 wait complete. 2024-04-23T00:05:51.525 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-23T00:05:51.525 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-23T00:05:51.542 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T00:05:51.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.695+0000 7fe69683a640 1 -- 172.21.15.67:0/2241214267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe690101b00 msgr2=0x7fe690109fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:51.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.695+0000 7fe69683a640 1 --2- 172.21.15.67:0/2241214267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe690101b00 0x7fe690109fe0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fe680009880 tx=0x7fe68002bd70 comp rx=0 tx=0).stop 2024-04-23T00:05:51.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.695+0000 7fe69683a640 1 -- 172.21.15.67:0/2241214267 shutdown_connections 2024-04-23T00:05:51.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.695+0000 7fe69683a640 1 --2- 172.21.15.67:0/2241214267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe690101b00 0x7fe690109fe0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:51.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.695+0000 7fe69683a640 1 --2- 172.21.15.67:0/2241214267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe6901011e0 0x7fe6901015c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:51.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.695+0000 7fe69683a640 1 -- 172.21.15.67:0/2241214267 >> 172.21.15.67:0/2241214267 conn(0x7fe6900faa20 msgr2=0x7fe6900fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:51.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.695+0000 7fe69683a640 1 -- 172.21.15.67:0/2241214267 shutdown_connections 2024-04-23T00:05:51.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.695+0000 7fe69683a640 1 -- 172.21.15.67:0/2241214267 wait complete. 2024-04-23T00:05:51.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.696+0000 7fe69683a640 1 Processor -- start 2024-04-23T00:05:51.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.696+0000 7fe69683a640 1 -- start start 2024-04-23T00:05:51.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.696+0000 7fe69683a640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe6901011e0 0x7fe69019ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:51.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.696+0000 7fe69683a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe690101b00 0x7fe69019bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:51.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.696+0000 7fe69683a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe690195a90 con 0x7fe690101b00 2024-04-23T00:05:51.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.696+0000 7fe69683a640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe690195c00 con 0x7fe6901011e0 2024-04-23T00:05:51.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.696+0000 7fe68f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe690101b00 0x7fe69019bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:51.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.697+0000 7fe68f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe690101b00 0x7fe69019bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35156/0 (socket says 172.21.15.67:35156) 2024-04-23T00:05:51.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.697+0000 7fe68f7fe640 1 -- 172.21.15.67:0/2271660307 learned_addr learned my addr 172.21.15.67:0/2271660307 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:51.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.697+0000 7fe68ffff640 1 --2- 172.21.15.67:0/2271660307 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe6901011e0 0x7fe69019ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:51.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.697+0000 7fe68f7fe640 1 -- 172.21.15.67:0/2271660307 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe6901011e0 msgr2=0x7fe69019ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:51.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.697+0000 7fe68f7fe640 1 --2- 172.21.15.67:0/2271660307 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe6901011e0 0x7fe69019ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:51.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.697+0000 7fe68f7fe640 1 -- 172.21.15.67:0/2271660307 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe680009560 con 0x7fe690101b00 2024-04-23T00:05:51.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.697+0000 7fe68f7fe640 1 --2- 172.21.15.67:0/2271660307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe690101b00 0x7fe69019bf60 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fe680009230 tx=0x7fe680004a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:51.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.697+0000 7fe68d7fa640 1 -- 172.21.15.67:0/2271660307 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe680004420 con 0x7fe690101b00 2024-04-23T00:05:51.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.697+0000 7fe68d7fa640 1 -- 172.21.15.67:0/2271660307 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe680003ab0 con 0x7fe690101b00 2024-04-23T00:05:51.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.697+0000 7fe69683a640 1 -- 172.21.15.67:0/2271660307 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe690195e80 con 0x7fe690101b00 2024-04-23T00:05:51.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.697+0000 7fe68d7fa640 1 -- 172.21.15.67:0/2271660307 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe680005570 con 0x7fe690101b00 2024-04-23T00:05:51.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.697+0000 7fe69683a640 1 -- 172.21.15.67:0/2271660307 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6901963e0 con 0x7fe690101b00 2024-04-23T00:05:51.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.698+0000 7fe69683a640 1 -- 172.21.15.67:0/2271660307 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe69004e040 con 0x7fe690101b00 2024-04-23T00:05:51.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.699+0000 7fe68d7fa640 1 -- 172.21.15.67:0/2271660307 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe680004dd0 con 0x7fe690101b00 2024-04-23T00:05:51.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.699+0000 7fe68d7fa640 1 --2- 172.21.15.67:0/2271660307 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe67c06ecf0 0x7fe67c0711b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:51.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.699+0000 7fe68d7fa640 1 -- 172.21.15.67:0/2271660307 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7fe6800b1980 con 0x7fe690101b00 2024-04-23T00:05:51.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.699+0000 7fe68ffff640 1 --2- 172.21.15.67:0/2271660307 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe67c06ecf0 0x7fe67c0711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:51.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.700+0000 7fe68ffff640 1 --2- 172.21.15.67:0/2271660307 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe67c06ecf0 0x7fe67c0711b0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fe678009bd0 tx=0x7fe678005f90 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:51.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.704+0000 7fe68d7fa640 1 -- 172.21.15.67:0/2271660307 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7fe68007a5c0 con 0x7fe690101b00 2024-04-23T00:05:51.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.894+0000 7fe69683a640 1 -- 172.21.15.67:0/2271660307 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe69010d2c0 con 0x7fe690101b00 2024-04-23T00:05:51.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.896+0000 7fe68d7fa640 1 -- 172.21.15.67:0/2271660307 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+17185 (secure 0 0 0) 0x7fe68007dc80 con 0x7fe690101b00 2024-04-23T00:05:51.883 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-23T00:05:51.883 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":94,"fsid":"d46c346e-00fd-11ef-bc93-c7b262605968","created":"2024-04-22T23:13:18.687917+0000","modified":"2024-04-22T23:48:20.827933+0000","last_up_change":"2024-04-22T23:32:23.175924+0000","last_in_change":"2024-04-22T23:18:41.696935+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T23:17:47.912850+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-22T23:20:19.759956+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-22T23:20:20.252379+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"25c9acd1-5743-4dc0-aee0-c7bb8884c0a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":49,"up_from":53,"up_thru":86,"down_at":50,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1540469345},{"type":"v1","addr":"172.21.15.67:6803","nonce":1540469345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1540469345},{"type":"v1","addr":"172.21.15.67:6805","nonce":1540469345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1540469345},{"type":"v1","addr":"172.21.15.67:6809","nonce":1540469345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1540469345},{"type":"v1","addr":"172.21.15.67:6807","nonce":1540469345}]},"public_addr":"172.21.15.67:6803/1540469345","cluster_addr":"172.21.15.67:6805/1540469345","heartbeat_back_addr":"172.21.15.67:6809/1540469345","heartbeat_front_addr":"172.21.15.67:6807/1540469345","state":["exists","up"]},{"osd":1,"uuid":"8be83014-01af-44d3-ab2f-b27840506e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":54,"up_from":63,"up_thru":86,"down_at":55,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":1745058179},{"type":"v1","addr":"172.21.15.67:6811","nonce":1745058179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1745058179},{"type":"v1","addr":"172.21.15.67:6813","nonce":1745058179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1745058179},{"type":"v1","addr":"172.21.15.67:6817","nonce":1745058179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":1745058179},{"type":"v1","addr":"172.21.15.67:6815","nonce":1745058179}]},"public_addr":"172.21.15.67:6811/1745058179","cluster_addr":"172.21.15.67:6813/1745058179","heartbeat_back_addr":"172.21.15.67:6817/1745058179","heartbeat_front_addr":"172.21.15.67:6815/1745058179","state":["exists","up"]},{"osd":2,"uuid":"ab1cbcda-14dd-4c24-876b-25ad5d4d8563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":67,"up_from":71,"up_thru":86,"down_at":68,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":121705471},{"type":"v1","addr":"172.21.15.67:6819","nonce":121705471}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":121705471},{"type":"v1","addr":"172.21.15.67:6821","nonce":121705471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":121705471},{"type":"v1","addr":"172.21.15.67:6825","nonce":121705471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":121705471},{"type":"v1","addr":"172.21.15.67:6823","nonce":121705471}]},"public_addr":"172.21.15.67:6819/121705471","cluster_addr":"172.21.15.67:6821/121705471","heartbeat_back_addr":"172.21.15.67:6825/121705471","heartbeat_front_addr":"172.21.15.67:6823/121705471","state":["exists","up"]},{"osd":3,"uuid":"1a30b6c3-95bf-4b19-883b-1271ff359e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":72,"up_from":76,"up_thru":86,"down_at":73,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":3344693085},{"type":"v1","addr":"172.21.15.134:6801","nonce":3344693085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":3344693085},{"type":"v1","addr":"172.21.15.134:6803","nonce":3344693085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":3344693085},{"type":"v1","addr":"172.21.15.134:6807","nonce":3344693085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":3344693085},{"type":"v1","addr":"172.21.15.134:6805","nonce":3344693085}]},"public_addr":"172.21.15.134:6801/3344693085","cluster_addr":"172.21.15.134:6803/3344693085","heartbeat_back_addr":"172.21.15.134:6807/3344693085","heartbeat_front_addr":"172.21.15.134:6805/3344693085","state":["exists","up"]},{"osd":4,"uuid":"53368fc1-28fd-4622-bc9c-1d8901b06bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":34,"last_clean_end":77,"up_from":81,"up_thru":86,"down_at":78,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":3935208416},{"type":"v1","addr":"172.21.15.134:6809","nonce":3935208416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":3935208416},{"type":"v1","addr":"172.21.15.134:6811","nonce":3935208416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":3935208416},{"type":"v1","addr":"172.21.15.134:6815","nonce":3935208416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":3935208416},{"type":"v1","addr":"172.21.15.134:6813","nonce":3935208416}]},"public_addr":"172.21.15.134:6809/3935208416","cluster_addr":"172.21.15.134:6811/3935208416","heartbeat_back_addr":"172.21.15.134:6815/3935208416","heartbeat_front_addr":"172.21.15.134:6813/3935208416","state":["exists","up"]},{"osd":5,"uuid":"31a885f7-8f91-4b6a-922a-065753f70a95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":82,"up_from":86,"up_thru":86,"down_at":83,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":921342983},{"type":"v1","addr":"172.21.15.134:6817","nonce":921342983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":921342983},{"type":"v1","addr":"172.21.15.134:6819","nonce":921342983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":921342983},{"type":"v1","addr":"172.21.15.134:6823","nonce":921342983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":921342983},{"type":"v1","addr":"172.21.15.134:6821","nonce":921342983}]},"public_addr":"172.21.15.134:6817/921342983","cluster_addr":"172.21.15.134:6819/921342983","heartbeat_back_addr":"172.21.15.134:6823/921342983","heartbeat_front_addr":"172.21.15.134:6821/921342983","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:16:51.179651+0000","dead_epoch":49},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:17:18.694683+0000","dead_epoch":54},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:17:45.368963+0000","dead_epoch":67},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:18:08.585046+0000","dead_epoch":72},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:18:33.412804+0000","dead_epoch":77},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T23:18:58.909758+0000","dead_epoch":82}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.2","mappings":[{"from":0,"to":2}]},{"pgid":"2.8","mappings":[{"from":0,"to":1}]},{"pgid":"2.b","mappings":[{"from":3,"to":1}]},{"pgid":"2.c","mappings":[{"from":3,"to":1}]},{"pgid":"2.f","mappings":[{"from":0,"to":1}]},{"pgid":"2.16","mappings":[{"from":3,"to":1}]},{"pgid":"3.1","mappings":[{"from":4,"to":1}]},{"pgid":"3.7","mappings":[{"from":4,"to":0}]},{"pgid":"3.c","mappings":[{"from":3,"to":1}]},{"pgid":"3.12","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6828/3606151860":"2024-04-23T23:32:27.027372+0000","172.21.15.134:6824/535957142":"2024-04-23T23:32:27.027388+0000","172.21.15.67:0/1189311107":"2024-04-23T23:15:37.396154+0000","172.21.15.67:6827/2381214817":"2024-04-23T23:20:37.496135+0000","172.21.15.67:6801/2590792187":"2024-04-23T23:15:37.396154+0000","172.21.15.67:6826/2381214817":"2024-04-23T23:20:37.496135+0000","172.21.15.134:0/3526908760":"2024-04-23T23:25:00.458625+0000","172.21.15.134:0/1536532822":"2024-04-23T23:25:00.458625+0000","172.21.15.134:6825/535957142":"2024-04-23T23:32:27.027388+0000","172.21.15.67:0/895378312":"2024-04-23T23:13:56.301645+0000","172.21.15.134:6828/1077503272":"2024-04-23T23:25:00.458625+0000","172.21.15.67:0/1125624760":"2024-04-23T23:15:37.396154+0000","172.21.15.67:0/3387250764":"2024-04-23T23:13:56.301645+0000","172.21.15.67:0/3921081246":"2024-04-23T23:27:55.947537+0000","172.21.15.67:0/2204477458":"2024-04-23T23:23:56.782522+0000","172.21.15.67:0/3730369564":"2024-04-23T23:27:55.947537+0000","172.21.15.134:0/2385923868":"2024-04-23T23:25:00.458625+0000","172.21.15.67:6800/2589747850":"2024-04-23T23:23:56.782522+0000","172.21.15.67:6801/2589747850":"2024-04-23T23:23:56.782522+0000","172.21.15.134:0/2817304592":"2024-04-23T23:25:00.458625+0000","172.21.15.67:6829/3606151860":"2024-04-23T23:32:27.027372+0000","172.21.15.67:0/233140160":"2024-04-23T23:23:56.782522+0000","172.21.15.67:0/3750652317":"2024-04-23T23:23:56.782522+0000","172.21.15.67:6800/2451437776":"2024-04-23T23:13:56.301645+0000","172.21.15.67:0/2665297739":"2024-04-23T23:14:29.214977+0000","172.21.15.67:6800/3568049827":"2024-04-23T23:14:29.214977+0000","172.21.15.67:0/4165654004":"2024-04-23T23:14:29.214977+0000","172.21.15.134:0/3046392865":"2024-04-23T23:25:00.458625+0000","172.21.15.67:0/2052573085":"2024-04-23T23:14:29.214977+0000","172.21.15.134:0/1251980938":"2024-04-23T23:25:00.458625+0000","172.21.15.67:6801/2451437776":"2024-04-23T23:13:56.301645+0000","172.21.15.67:0/4286433529":"2024-04-23T23:27:55.947537+0000","172.21.15.67:0/853292259":"2024-04-23T23:13:56.301645+0000","172.21.15.67:6801/3568049827":"2024-04-23T23:14:29.214977+0000","172.21.15.67:6800/2590792187":"2024-04-23T23:15:37.396154+0000","172.21.15.67:0/1933271294":"2024-04-23T23:23:56.782522+0000","172.21.15.134:6829/1077503272":"2024-04-23T23:25:00.458625+0000","172.21.15.67:0/1737320548":"2024-04-23T23:27:55.947537+0000","172.21.15.67:6800/2306276228":"2024-04-23T23:27:55.947537+0000","172.21.15.67:0/2137374599":"2024-04-23T23:15:37.396154+0000","172.21.15.67:6801/2306276228":"2024-04-23T23:27:55.947537+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T00:05:51.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.898+0000 7fe69683a640 1 -- 172.21.15.67:0/2271660307 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe67c06ecf0 msgr2=0x7fe67c0711b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:51.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.898+0000 7fe69683a640 1 --2- 172.21.15.67:0/2271660307 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe67c06ecf0 0x7fe67c0711b0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fe678009bd0 tx=0x7fe678005f90 comp rx=0 tx=0).stop 2024-04-23T00:05:51.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.898+0000 7fe69683a640 1 -- 172.21.15.67:0/2271660307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe690101b00 msgr2=0x7fe69019bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:51.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.898+0000 7fe69683a640 1 --2- 172.21.15.67:0/2271660307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe690101b00 0x7fe69019bf60 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fe680009230 tx=0x7fe680004a30 comp rx=0 tx=0).stop 2024-04-23T00:05:51.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.898+0000 7fe69683a640 1 -- 172.21.15.67:0/2271660307 shutdown_connections 2024-04-23T00:05:51.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.898+0000 7fe69683a640 1 --2- 172.21.15.67:0/2271660307 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7fe67c06ecf0 0x7fe67c0711b0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:51.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.898+0000 7fe69683a640 1 --2- 172.21.15.67:0/2271660307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe690101b00 0x7fe69019bf60 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:51.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.898+0000 7fe69683a640 1 --2- 172.21.15.67:0/2271660307 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe6901011e0 0x7fe69019ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:51.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.898+0000 7fe69683a640 1 -- 172.21.15.67:0/2271660307 >> 172.21.15.67:0/2271660307 conn(0x7fe6900faa20 msgr2=0x7fe690104820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:51.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.898+0000 7fe69683a640 1 -- 172.21.15.67:0/2271660307 shutdown_connections 2024-04-23T00:05:51.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:51.898+0000 7fe69683a640 1 -- 172.21.15.67:0/2271660307 wait complete. 2024-04-23T00:05:51.896 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T23:17:47.912850+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-22T23:20:19.759956+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-22T23:20:20.252379+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T00:05:51.897 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T00:05:52.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.014+0000 7f57b7869640 1 -- 172.21.15.67:0/414878963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57b00ffc10 msgr2=0x7f57b0103b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:52.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.014+0000 7f57b7869640 1 --2- 172.21.15.67:0/414878963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57b00ffc10 0x7f57b0103b70 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f57a4009880 tx=0x7f57a402bce0 comp rx=0 tx=0).stop 2024-04-23T00:05:52.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.014+0000 7f57b7869640 1 -- 172.21.15.67:0/414878963 shutdown_connections 2024-04-23T00:05:52.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.014+0000 7f57b7869640 1 --2- 172.21.15.67:0/414878963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57b00ffc10 0x7f57b0103b70 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:52.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.014+0000 7f57b7869640 1 --2- 172.21.15.67:0/414878963 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f57b00ff260 0x7f57b00ff640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:52.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.014+0000 7f57b7869640 1 -- 172.21.15.67:0/414878963 >> 172.21.15.67:0/414878963 conn(0x7f57b0074a50 msgr2=0x7f57b0074e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:52.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.014+0000 7f57b7869640 1 -- 172.21.15.67:0/414878963 shutdown_connections 2024-04-23T00:05:52.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.014+0000 7f57b7869640 1 -- 172.21.15.67:0/414878963 wait complete. 2024-04-23T00:05:52.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.014+0000 7f57b7869640 1 Processor -- start 2024-04-23T00:05:52.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.014+0000 7f57b7869640 1 -- start start 2024-04-23T00:05:52.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.015+0000 7f57b7869640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f57b00ff260 0x7f57b0109ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:52.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.015+0000 7f57b7869640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57b010a420 0x7f57b010a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:52.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.015+0000 7f57b7869640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b01a1fe0 con 0x7f57b010a420 2024-04-23T00:05:52.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.015+0000 7f57b7869640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b01a2120 con 0x7f57b00ff260 2024-04-23T00:05:52.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.015+0000 7f57b4e04640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57b010a420 0x7f57b010a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:52.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.015+0000 7f57b4e04640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57b010a420 0x7f57b010a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35170/0 (socket says 172.21.15.67:35170) 2024-04-23T00:05:52.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.015+0000 7f57b4e04640 1 -- 172.21.15.67:0/1570947989 learned_addr learned my addr 172.21.15.67:0/1570947989 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:52.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.015+0000 7f57b5605640 1 --2- 172.21.15.67:0/1570947989 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f57b00ff260 0x7f57b0109ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:52.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.015+0000 7f57b4e04640 1 -- 172.21.15.67:0/1570947989 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f57b00ff260 msgr2=0x7f57b0109ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:52.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.015+0000 7f57b4e04640 1 --2- 172.21.15.67:0/1570947989 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f57b00ff260 0x7f57b0109ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:52.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.015+0000 7f57b4e04640 1 -- 172.21.15.67:0/1570947989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a4009560 con 0x7f57b010a420 2024-04-23T00:05:52.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.015+0000 7f57b5605640 1 --2- 172.21.15.67:0/1570947989 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f57b00ff260 0x7f57b0109ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T00:05:52.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.015+0000 7f57b4e04640 1 --2- 172.21.15.67:0/1570947989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57b010a420 0x7f57b010a880 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f57a4036040 tx=0x7f57a402c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:52.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.015+0000 7f57ae7fc640 1 -- 172.21.15.67:0/1570947989 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57a40035f0 con 0x7f57b010a420 2024-04-23T00:05:52.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.015+0000 7f57b7869640 1 -- 172.21.15.67:0/1570947989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57b01a2260 con 0x7f57b010a420 2024-04-23T00:05:52.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.015+0000 7f57ae7fc640 1 -- 172.21.15.67:0/1570947989 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f57a4003c10 con 0x7f57b010a420 2024-04-23T00:05:52.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.015+0000 7f57ae7fc640 1 -- 172.21.15.67:0/1570947989 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57a4034440 con 0x7f57b010a420 2024-04-23T00:05:52.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.015+0000 7f57b7869640 1 -- 172.21.15.67:0/1570947989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57b01a2710 con 0x7f57b010a420 2024-04-23T00:05:52.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.016+0000 7f57b7869640 1 -- 172.21.15.67:0/1570947989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57b004e040 con 0x7f57b010a420 2024-04-23T00:05:52.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.017+0000 7f57ae7fc640 1 -- 172.21.15.67:0/1570947989 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f57a4003750 con 0x7f57b010a420 2024-04-23T00:05:52.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.017+0000 7f57ae7fc640 1 --2- 172.21.15.67:0/1570947989 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f579806ec30 0x7f57980710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:52.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.018+0000 7f57ae7fc640 1 -- 172.21.15.67:0/1570947989 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f57a40b13b0 con 0x7f57b010a420 2024-04-23T00:05:52.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.018+0000 7f57b5605640 1 --2- 172.21.15.67:0/1570947989 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f579806ec30 0x7f57980710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:52.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.018+0000 7f57b5605640 1 --2- 172.21.15.67:0/1570947989 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f579806ec30 0x7f57980710f0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f57a0005fd0 tx=0x7f57a0009520 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:52.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.022+0000 7f57ae7fc640 1 -- 172.21.15.67:0/1570947989 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f57a4079ff0 con 0x7f57b010a420 2024-04-23T00:05:52.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.210+0000 7f57b7869640 1 -- 172.21.15.67:0/1570947989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f57b00ffc10 con 0x7f57b010a420 2024-04-23T00:05:52.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.210+0000 7f57ae7fc640 1 -- 172.21.15.67:0/1570947989 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) v1 ==== 93+0+10 (secure 0 0 0) 0x7f57a407d6b0 con 0x7f57b010a420 2024-04-23T00:05:52.197 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-23T00:05:52.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.212+0000 7f57b7869640 1 -- 172.21.15.67:0/1570947989 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f579806ec30 msgr2=0x7f57980710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:52.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.212+0000 7f57b7869640 1 --2- 172.21.15.67:0/1570947989 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f579806ec30 0x7f57980710f0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f57a0005fd0 tx=0x7f57a0009520 comp rx=0 tx=0).stop 2024-04-23T00:05:52.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.212+0000 7f57b7869640 1 -- 172.21.15.67:0/1570947989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57b010a420 msgr2=0x7f57b010a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:52.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.212+0000 7f57b7869640 1 --2- 172.21.15.67:0/1570947989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57b010a420 0x7f57b010a880 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f57a4036040 tx=0x7f57a402c850 comp rx=0 tx=0).stop 2024-04-23T00:05:52.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.212+0000 7f57b7869640 1 -- 172.21.15.67:0/1570947989 shutdown_connections 2024-04-23T00:05:52.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.212+0000 7f57b7869640 1 --2- 172.21.15.67:0/1570947989 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f579806ec30 0x7f57980710f0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:52.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.212+0000 7f57b7869640 1 --2- 172.21.15.67:0/1570947989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57b010a420 0x7f57b010a880 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:52.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.212+0000 7f57b7869640 1 --2- 172.21.15.67:0/1570947989 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f57b00ff260 0x7f57b0109ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:52.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.212+0000 7f57b7869640 1 -- 172.21.15.67:0/1570947989 >> 172.21.15.67:0/1570947989 conn(0x7f57b0074a50 msgr2=0x7f57b00fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:52.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.212+0000 7f57b7869640 1 -- 172.21.15.67:0/1570947989 shutdown_connections 2024-04-23T00:05:52.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.212+0000 7f57b7869640 1 -- 172.21.15.67:0/1570947989 wait complete. 2024-04-23T00:05:52.209 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:51 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3438771760' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-23T00:05:52.209 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:51 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/4224743149' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-23T00:05:52.209 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:51 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1677549062' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-23T00:05:52.210 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-23T00:05:52.230 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:51 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3438771760' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-23T00:05:52.230 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:51 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/4224743149' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-23T00:05:52.230 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:51 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1677549062' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-23T00:05:52.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.328+0000 7f44e1139640 1 -- 172.21.15.67:0/2813934450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44dc0ff1a0 msgr2=0x7f44dc0ff580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:52.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.328+0000 7f44e1139640 1 --2- 172.21.15.67:0/2813934450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44dc0ff1a0 0x7f44dc0ff580 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f44cc009880 tx=0x7f44cc02bce0 comp rx=0 tx=0).stop 2024-04-23T00:05:52.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.328+0000 7f44e1139640 1 -- 172.21.15.67:0/2813934450 shutdown_connections 2024-04-23T00:05:52.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.328+0000 7f44e1139640 1 --2- 172.21.15.67:0/2813934450 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44dc0ffb50 0x7f44dc103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:52.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.328+0000 7f44e1139640 1 --2- 172.21.15.67:0/2813934450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44dc0ff1a0 0x7f44dc0ff580 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:52.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.328+0000 7f44e1139640 1 -- 172.21.15.67:0/2813934450 >> 172.21.15.67:0/2813934450 conn(0x7f44dc0faa20 msgr2=0x7f44dc0fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:52.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.328+0000 7f44e1139640 1 -- 172.21.15.67:0/2813934450 shutdown_connections 2024-04-23T00:05:52.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.328+0000 7f44e1139640 1 -- 172.21.15.67:0/2813934450 wait complete. 2024-04-23T00:05:52.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.329+0000 7f44e1139640 1 Processor -- start 2024-04-23T00:05:52.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.329+0000 7f44e1139640 1 -- start start 2024-04-23T00:05:52.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.329+0000 7f44e1139640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44dc0ff1a0 0x7f44dc195e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:52.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.329+0000 7f44e1139640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44dc196350 0x7f44dc19a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:52.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.329+0000 7f44e1139640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44dc196950 con 0x7f44dc0ff1a0 2024-04-23T00:05:52.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.329+0000 7f44e1139640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44dc196ac0 con 0x7f44dc196350 2024-04-23T00:05:52.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.329+0000 7f44dad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44dc0ff1a0 0x7f44dc195e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:52.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.329+0000 7f44dad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44dc0ff1a0 0x7f44dc195e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35182/0 (socket says 172.21.15.67:35182) 2024-04-23T00:05:52.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.330+0000 7f44dad9d640 1 -- 172.21.15.67:0/3089619875 learned_addr learned my addr 172.21.15.67:0/3089619875 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:52.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.330+0000 7f44d259c640 1 --2- 172.21.15.67:0/3089619875 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44dc196350 0x7f44dc19a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:52.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.330+0000 7f44dad9d640 1 -- 172.21.15.67:0/3089619875 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44dc196350 msgr2=0x7f44dc19a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:52.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.330+0000 7f44dad9d640 1 --2- 172.21.15.67:0/3089619875 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44dc196350 0x7f44dc19a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:52.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.330+0000 7f44dad9d640 1 -- 172.21.15.67:0/3089619875 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44cc009560 con 0x7f44dc0ff1a0 2024-04-23T00:05:52.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.330+0000 7f44dad9d640 1 --2- 172.21.15.67:0/3089619875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44dc0ff1a0 0x7f44dc195e10 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f44cc036040 tx=0x7f44cc02c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:52.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.330+0000 7f44d8d99640 1 -- 172.21.15.67:0/3089619875 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44cc004520 con 0x7f44dc0ff1a0 2024-04-23T00:05:52.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.330+0000 7f44e1139640 1 -- 172.21.15.67:0/3089619875 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44dc19ad00 con 0x7f44dc0ff1a0 2024-04-23T00:05:52.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.330+0000 7f44d8d99640 1 -- 172.21.15.67:0/3089619875 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f44cc003ab0 con 0x7f44dc0ff1a0 2024-04-23T00:05:52.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.330+0000 7f44d8d99640 1 -- 172.21.15.67:0/3089619875 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44cc034440 con 0x7f44dc0ff1a0 2024-04-23T00:05:52.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.330+0000 7f44e1139640 1 -- 172.21.15.67:0/3089619875 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44dc19b260 con 0x7f44dc0ff1a0 2024-04-23T00:05:52.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.331+0000 7f44e1139640 1 -- 172.21.15.67:0/3089619875 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44dc04e040 con 0x7f44dc0ff1a0 2024-04-23T00:05:52.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.332+0000 7f44d8d99640 1 -- 172.21.15.67:0/3089619875 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f44cc003c20 con 0x7f44dc0ff1a0 2024-04-23T00:05:52.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.332+0000 7f44d8d99640 1 --2- 172.21.15.67:0/3089619875 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f44c806ec30 0x7f44c80710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:52.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.332+0000 7f44d259c640 1 --2- 172.21.15.67:0/3089619875 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f44c806ec30 0x7f44c80710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:52.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.332+0000 7f44d8d99640 1 -- 172.21.15.67:0/3089619875 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f44cc0b12d0 con 0x7f44dc0ff1a0 2024-04-23T00:05:52.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.333+0000 7f44d259c640 1 --2- 172.21.15.67:0/3089619875 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f44c806ec30 0x7f44c80710f0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f44c4005fd0 tx=0x7f44c400b390 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:52.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.337+0000 7f44d8d99640 1 -- 172.21.15.67:0/3089619875 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f44cc079fc0 con 0x7f44dc0ff1a0 2024-04-23T00:05:52.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.522+0000 7f44e1139640 1 -- 172.21.15.67:0/3089619875 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f44dc197110 con 0x7f44dc0ff1a0 2024-04-23T00:05:52.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.522+0000 7f44d8d99640 1 -- 172.21.15.67:0/3089619875 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v94) v1 ==== 107+0+11 (secure 0 0 0) 0x7f44cc07d680 con 0x7f44dc0ff1a0 2024-04-23T00:05:52.510 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-23T00:05:52.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.524+0000 7f44e1139640 1 -- 172.21.15.67:0/3089619875 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f44c806ec30 msgr2=0x7f44c80710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:52.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.524+0000 7f44e1139640 1 --2- 172.21.15.67:0/3089619875 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f44c806ec30 0x7f44c80710f0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f44c4005fd0 tx=0x7f44c400b390 comp rx=0 tx=0).stop 2024-04-23T00:05:52.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.524+0000 7f44e1139640 1 -- 172.21.15.67:0/3089619875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44dc0ff1a0 msgr2=0x7f44dc195e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:52.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.524+0000 7f44e1139640 1 --2- 172.21.15.67:0/3089619875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44dc0ff1a0 0x7f44dc195e10 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f44cc036040 tx=0x7f44cc02c850 comp rx=0 tx=0).stop 2024-04-23T00:05:52.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.524+0000 7f44e1139640 1 -- 172.21.15.67:0/3089619875 shutdown_connections 2024-04-23T00:05:52.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.524+0000 7f44e1139640 1 --2- 172.21.15.67:0/3089619875 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f44c806ec30 0x7f44c80710f0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:52.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.524+0000 7f44e1139640 1 --2- 172.21.15.67:0/3089619875 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44dc196350 0x7f44dc19a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:52.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.524+0000 7f44e1139640 1 --2- 172.21.15.67:0/3089619875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44dc0ff1a0 0x7f44dc195e10 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:52.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.524+0000 7f44e1139640 1 -- 172.21.15.67:0/3089619875 >> 172.21.15.67:0/3089619875 conn(0x7f44dc0faa20 msgr2=0x7f44dc103410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:52.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.524+0000 7f44e1139640 1 -- 172.21.15.67:0/3089619875 shutdown_connections 2024-04-23T00:05:52.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.524+0000 7f44e1139640 1 -- 172.21.15.67:0/3089619875 wait complete. 2024-04-23T00:05:52.522 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-23T00:05:52.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.639+0000 7f8a15b24640 1 -- 172.21.15.67:0/843172905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a10101b20 msgr2=0x7f8a1010a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:52.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.639+0000 7f8a15b24640 1 --2- 172.21.15.67:0/843172905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a10101b20 0x7f8a1010a000 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f8a04009880 tx=0x7f8a0402bd70 comp rx=0 tx=0).stop 2024-04-23T00:05:52.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.639+0000 7f8a15b24640 1 -- 172.21.15.67:0/843172905 shutdown_connections 2024-04-23T00:05:52.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.639+0000 7f8a15b24640 1 --2- 172.21.15.67:0/843172905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a10101b20 0x7f8a1010a000 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:52.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.639+0000 7f8a15b24640 1 --2- 172.21.15.67:0/843172905 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a10101200 0x7f8a101015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:52.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.639+0000 7f8a15b24640 1 -- 172.21.15.67:0/843172905 >> 172.21.15.67:0/843172905 conn(0x7f8a100faa20 msgr2=0x7f8a100fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:52.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.639+0000 7f8a15b24640 1 -- 172.21.15.67:0/843172905 shutdown_connections 2024-04-23T00:05:52.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.639+0000 7f8a15b24640 1 -- 172.21.15.67:0/843172905 wait complete. 2024-04-23T00:05:52.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.639+0000 7f8a15b24640 1 Processor -- start 2024-04-23T00:05:52.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.640+0000 7f8a15b24640 1 -- start start 2024-04-23T00:05:52.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.640+0000 7f8a15b24640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a10101200 0x7f8a10195d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:52.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.640+0000 7f8a15b24640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a10101b20 0x7f8a10196290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:52.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.640+0000 7f8a15b24640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a10196890 con 0x7f8a10101200 2024-04-23T00:05:52.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.640+0000 7f8a15b24640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a10196a00 con 0x7f8a10101b20 2024-04-23T00:05:52.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.640+0000 7f8a0f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a10101200 0x7f8a10195d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:52.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.640+0000 7f8a0f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a10101200 0x7f8a10195d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35196/0 (socket says 172.21.15.67:35196) 2024-04-23T00:05:52.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.640+0000 7f8a0f7fe640 1 -- 172.21.15.67:0/1558596298 learned_addr learned my addr 172.21.15.67:0/1558596298 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:52.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.640+0000 7f8a0effd640 1 --2- 172.21.15.67:0/1558596298 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a10101b20 0x7f8a10196290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:52.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.640+0000 7f8a0f7fe640 1 -- 172.21.15.67:0/1558596298 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a10101b20 msgr2=0x7f8a10196290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:52.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.640+0000 7f8a0f7fe640 1 --2- 172.21.15.67:0/1558596298 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a10101b20 0x7f8a10196290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:52.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.640+0000 7f8a0f7fe640 1 -- 172.21.15.67:0/1558596298 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a04009560 con 0x7f8a10101200 2024-04-23T00:05:52.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.640+0000 7f8a0effd640 1 --2- 172.21.15.67:0/1558596298 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a10101b20 0x7f8a10196290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T00:05:52.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.641+0000 7f8a0f7fe640 1 --2- 172.21.15.67:0/1558596298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a10101200 0x7f8a10195d50 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f8a0000c8a0 tx=0x7f8a0000cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:52.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.641+0000 7f8a0cff9640 1 -- 172.21.15.67:0/1558596298 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a00007b80 con 0x7f8a10101200 2024-04-23T00:05:52.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.641+0000 7f8a15b24640 1 -- 172.21.15.67:0/1558596298 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a1019a890 con 0x7f8a10101200 2024-04-23T00:05:52.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.641+0000 7f8a0cff9640 1 -- 172.21.15.67:0/1558596298 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8a00007ce0 con 0x7f8a10101200 2024-04-23T00:05:52.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.641+0000 7f8a0cff9640 1 -- 172.21.15.67:0/1558596298 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a00018810 con 0x7f8a10101200 2024-04-23T00:05:52.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.641+0000 7f8a15b24640 1 -- 172.21.15.67:0/1558596298 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a1019adf0 con 0x7f8a10101200 2024-04-23T00:05:52.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.642+0000 7f8a15b24640 1 -- 172.21.15.67:0/1558596298 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a1004e040 con 0x7f8a10101200 2024-04-23T00:05:52.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.642+0000 7f8a0cff9640 1 -- 172.21.15.67:0/1558596298 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8a00018a80 con 0x7f8a10101200 2024-04-23T00:05:52.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.643+0000 7f8a0cff9640 1 --2- 172.21.15.67:0/1558596298 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f89f806ec30 0x7f89f80710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:52.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.643+0000 7f8a0cff9640 1 -- 172.21.15.67:0/1558596298 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f8a0008f1f0 con 0x7f8a10101200 2024-04-23T00:05:52.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.643+0000 7f8a0effd640 1 --2- 172.21.15.67:0/1558596298 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f89f806ec30 0x7f89f80710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:52.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.643+0000 7f8a0effd640 1 --2- 172.21.15.67:0/1558596298 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f89f806ec30 0x7f89f80710f0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f8a04005ec0 tx=0x7f8a04000bc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:52.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.647+0000 7f8a0cff9640 1 -- 172.21.15.67:0/1558596298 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f8a00057e30 con 0x7f8a10101200 2024-04-23T00:05:52.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.834+0000 7f8a15b24640 1 -- 172.21.15.67:0/1558596298 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f8a101970a0 con 0x7f8a10101200 2024-04-23T00:05:52.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.834+0000 7f8a0cff9640 1 -- 172.21.15.67:0/1558596298 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v94) v1 ==== 107+0+11 (secure 0 0 0) 0x7f8a0005b4f0 con 0x7f8a10101200 2024-04-23T00:05:52.822 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-23T00:05:52.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.837+0000 7f8a15b24640 1 -- 172.21.15.67:0/1558596298 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f89f806ec30 msgr2=0x7f89f80710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:52.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.837+0000 7f8a15b24640 1 --2- 172.21.15.67:0/1558596298 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f89f806ec30 0x7f89f80710f0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f8a04005ec0 tx=0x7f8a04000bc0 comp rx=0 tx=0).stop 2024-04-23T00:05:52.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.837+0000 7f8a15b24640 1 -- 172.21.15.67:0/1558596298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a10101200 msgr2=0x7f8a10195d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:52.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.837+0000 7f8a15b24640 1 --2- 172.21.15.67:0/1558596298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a10101200 0x7f8a10195d50 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f8a0000c8a0 tx=0x7f8a0000cd60 comp rx=0 tx=0).stop 2024-04-23T00:05:52.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.837+0000 7f8a15b24640 1 -- 172.21.15.67:0/1558596298 shutdown_connections 2024-04-23T00:05:52.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.837+0000 7f8a15b24640 1 --2- 172.21.15.67:0/1558596298 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f89f806ec30 0x7f89f80710f0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:52.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.837+0000 7f8a15b24640 1 --2- 172.21.15.67:0/1558596298 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a10101b20 0x7f8a10196290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:52.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.837+0000 7f8a15b24640 1 --2- 172.21.15.67:0/1558596298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a10101200 0x7f8a10195d50 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:52.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.837+0000 7f8a15b24640 1 -- 172.21.15.67:0/1558596298 >> 172.21.15.67:0/1558596298 conn(0x7f8a100faa20 msgr2=0x7f8a10104840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:52.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.837+0000 7f8a15b24640 1 -- 172.21.15.67:0/1558596298 shutdown_connections 2024-04-23T00:05:52.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.837+0000 7f8a15b24640 1 -- 172.21.15.67:0/1558596298 wait complete. 2024-04-23T00:05:52.835 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-23T00:05:52.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.950+0000 7f5d39999640 1 -- 172.21.15.67:0/1463231437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d340ff170 msgr2=0x7f5d340ff550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:52.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.950+0000 7f5d39999640 1 --2- 172.21.15.67:0/1463231437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d340ff170 0x7f5d340ff550 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f5d24009880 tx=0x7f5d2402bce0 comp rx=0 tx=0).stop 2024-04-23T00:05:52.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.950+0000 7f5d39999640 1 -- 172.21.15.67:0/1463231437 shutdown_connections 2024-04-23T00:05:52.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.950+0000 7f5d39999640 1 --2- 172.21.15.67:0/1463231437 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5d340ffb20 0x7f5d34103b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:52.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.950+0000 7f5d39999640 1 --2- 172.21.15.67:0/1463231437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d340ff170 0x7f5d340ff550 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:52.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.950+0000 7f5d39999640 1 -- 172.21.15.67:0/1463231437 >> 172.21.15.67:0/1463231437 conn(0x7f5d340fa9d0 msgr2=0x7f5d340fcdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:52.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.950+0000 7f5d39999640 1 -- 172.21.15.67:0/1463231437 shutdown_connections 2024-04-23T00:05:52.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.950+0000 7f5d39999640 1 -- 172.21.15.67:0/1463231437 wait complete. 2024-04-23T00:05:52.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.951+0000 7f5d39999640 1 Processor -- start 2024-04-23T00:05:52.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.951+0000 7f5d39999640 1 -- start start 2024-04-23T00:05:52.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.951+0000 7f5d39999640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5d340ff170 0x7f5d34195db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:52.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.951+0000 7f5d39999640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d341962f0 0x7f5d3419a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:52.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.951+0000 7f5d39999640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d341968f0 con 0x7f5d341962f0 2024-04-23T00:05:52.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.951+0000 7f5d39999640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d34196a60 con 0x7f5d340ff170 2024-04-23T00:05:52.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.951+0000 7f5d2a7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d341962f0 0x7f5d3419a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:52.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.951+0000 7f5d2a7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d341962f0 0x7f5d3419a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35208/0 (socket says 172.21.15.67:35208) 2024-04-23T00:05:52.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.951+0000 7f5d2a7fc640 1 -- 172.21.15.67:0/1812977082 learned_addr learned my addr 172.21.15.67:0/1812977082 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-23T00:05:52.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.951+0000 7f5d32ffd640 1 --2- 172.21.15.67:0/1812977082 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5d340ff170 0x7f5d34195db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:52.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.952+0000 7f5d2a7fc640 1 -- 172.21.15.67:0/1812977082 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5d340ff170 msgr2=0x7f5d34195db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:52.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.952+0000 7f5d2a7fc640 1 --2- 172.21.15.67:0/1812977082 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5d340ff170 0x7f5d34195db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:52.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.952+0000 7f5d2a7fc640 1 -- 172.21.15.67:0/1812977082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d24009560 con 0x7f5d341962f0 2024-04-23T00:05:52.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.952+0000 7f5d2a7fc640 1 --2- 172.21.15.67:0/1812977082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d341962f0 0x7f5d3419a750 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c00b400 tx=0x7f5d1c00b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:52.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.952+0000 7f5d38997640 1 -- 172.21.15.67:0/1812977082 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d1c011800 con 0x7f5d341962f0 2024-04-23T00:05:52.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.952+0000 7f5d39999640 1 -- 172.21.15.67:0/1812977082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d3419ad50 con 0x7f5d341962f0 2024-04-23T00:05:52.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.952+0000 7f5d38997640 1 -- 172.21.15.67:0/1812977082 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5d1c011e20 con 0x7f5d341962f0 2024-04-23T00:05:52.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.952+0000 7f5d38997640 1 -- 172.21.15.67:0/1812977082 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d1c00f2e0 con 0x7f5d341962f0 2024-04-23T00:05:52.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.952+0000 7f5d39999640 1 -- 172.21.15.67:0/1812977082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d3419b2b0 con 0x7f5d341962f0 2024-04-23T00:05:52.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.953+0000 7f5d39999640 1 -- 172.21.15.67:0/1812977082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d3404e040 con 0x7f5d341962f0 2024-04-23T00:05:52.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.953+0000 7f5d38997640 1 -- 172.21.15.67:0/1812977082 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 40) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5d1c010380 con 0x7f5d341962f0 2024-04-23T00:05:52.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.954+0000 7f5d38997640 1 --2- 172.21.15.67:0/1812977082 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5d2006ecf0 0x7f5d200711b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T00:05:52.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.954+0000 7f5d38997640 1 -- 172.21.15.67:0/1812977082 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6876+0+0 (secure 0 0 0) 0x7f5d1c08ed20 con 0x7f5d341962f0 2024-04-23T00:05:52.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.954+0000 7f5d32ffd640 1 --2- 172.21.15.67:0/1812977082 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5d2006ecf0 0x7f5d200711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T00:05:52.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.954+0000 7f5d32ffd640 1 --2- 172.21.15.67:0/1812977082 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5d2006ecf0 0x7f5d200711b0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f5d24006010 tx=0x7f5d24000bc0 comp rx=0 tx=0).ready entity=mgr.34102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T00:05:52.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:52.959+0000 7f5d38997640 1 -- 172.21.15.67:0/1812977082 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188728 (secure 0 0 0) 0x7f5d1c093070 con 0x7f5d341962f0 2024-04-23T00:05:53.131 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:52 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/2271660307' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T00:05:53.131 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:52 smithi134 ceph-mon[85630]: pgmap v1189: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:05:53.131 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:52 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1570947989' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T00:05:53.132 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:52 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/3089619875' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-23T00:05:53.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:53.159+0000 7f5d39999640 1 -- 172.21.15.67:0/1812977082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f5d34197100 con 0x7f5d341962f0 2024-04-23T00:05:53.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:53.160+0000 7f5d38997640 1 -- 172.21.15.67:0/1812977082 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 41 entries v94) v1 ==== 81+0+2437 (secure 0 0 0) 0x7f5d1c057900 con 0x7f5d341962f0 2024-04-23T00:05:53.148 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6828/3606151860 2024-04-23T23:32:27.027372+0000 2024-04-23T00:05:53.148 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.134:6824/535957142 2024-04-23T23:32:27.027388+0000 2024-04-23T00:05:53.148 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/1189311107 2024-04-23T23:15:37.396154+0000 2024-04-23T00:05:53.148 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6827/2381214817 2024-04-23T23:20:37.496135+0000 2024-04-23T00:05:53.148 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6801/2590792187 2024-04-23T23:15:37.396154+0000 2024-04-23T00:05:53.148 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6826/2381214817 2024-04-23T23:20:37.496135+0000 2024-04-23T00:05:53.149 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.134:0/3526908760 2024-04-23T23:25:00.458625+0000 2024-04-23T00:05:53.149 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.134:0/1536532822 2024-04-23T23:25:00.458625+0000 2024-04-23T00:05:53.149 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.134:6825/535957142 2024-04-23T23:32:27.027388+0000 2024-04-23T00:05:53.149 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/895378312 2024-04-23T23:13:56.301645+0000 2024-04-23T00:05:53.149 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.134:6828/1077503272 2024-04-23T23:25:00.458625+0000 2024-04-23T00:05:53.149 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/1125624760 2024-04-23T23:15:37.396154+0000 2024-04-23T00:05:53.149 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/3387250764 2024-04-23T23:13:56.301645+0000 2024-04-23T00:05:53.149 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/3921081246 2024-04-23T23:27:55.947537+0000 2024-04-23T00:05:53.150 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/2204477458 2024-04-23T23:23:56.782522+0000 2024-04-23T00:05:53.150 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/3730369564 2024-04-23T23:27:55.947537+0000 2024-04-23T00:05:53.150 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.134:0/2385923868 2024-04-23T23:25:00.458625+0000 2024-04-23T00:05:53.150 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6800/2589747850 2024-04-23T23:23:56.782522+0000 2024-04-23T00:05:53.150 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6801/2589747850 2024-04-23T23:23:56.782522+0000 2024-04-23T00:05:53.150 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.134:0/2817304592 2024-04-23T23:25:00.458625+0000 2024-04-23T00:05:53.150 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6829/3606151860 2024-04-23T23:32:27.027372+0000 2024-04-23T00:05:53.151 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/233140160 2024-04-23T23:23:56.782522+0000 2024-04-23T00:05:53.151 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/3750652317 2024-04-23T23:23:56.782522+0000 2024-04-23T00:05:53.151 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6800/2451437776 2024-04-23T23:13:56.301645+0000 2024-04-23T00:05:53.151 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/2665297739 2024-04-23T23:14:29.214977+0000 2024-04-23T00:05:53.151 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6800/3568049827 2024-04-23T23:14:29.214977+0000 2024-04-23T00:05:53.151 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/4165654004 2024-04-23T23:14:29.214977+0000 2024-04-23T00:05:53.151 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.134:0/3046392865 2024-04-23T23:25:00.458625+0000 2024-04-23T00:05:53.151 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/2052573085 2024-04-23T23:14:29.214977+0000 2024-04-23T00:05:53.152 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.134:0/1251980938 2024-04-23T23:25:00.458625+0000 2024-04-23T00:05:53.152 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6801/2451437776 2024-04-23T23:13:56.301645+0000 2024-04-23T00:05:53.152 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/4286433529 2024-04-23T23:27:55.947537+0000 2024-04-23T00:05:53.152 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/853292259 2024-04-23T23:13:56.301645+0000 2024-04-23T00:05:53.152 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6801/3568049827 2024-04-23T23:14:29.214977+0000 2024-04-23T00:05:53.152 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6800/2590792187 2024-04-23T23:15:37.396154+0000 2024-04-23T00:05:53.152 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/1933271294 2024-04-23T23:23:56.782522+0000 2024-04-23T00:05:53.152 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.134:6829/1077503272 2024-04-23T23:25:00.458625+0000 2024-04-23T00:05:53.153 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/1737320548 2024-04-23T23:27:55.947537+0000 2024-04-23T00:05:53.153 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6800/2306276228 2024-04-23T23:27:55.947537+0000 2024-04-23T00:05:53.153 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/2137374599 2024-04-23T23:15:37.396154+0000 2024-04-23T00:05:53.153 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6801/2306276228 2024-04-23T23:27:55.947537+0000 2024-04-23T00:05:53.155 INFO:teuthology.orchestra.run.smithi067.stderr:listed 41 entries 2024-04-23T00:05:53.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:53.162+0000 7f5d39999640 1 -- 172.21.15.67:0/1812977082 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5d2006ecf0 msgr2=0x7f5d200711b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:53.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:53.162+0000 7f5d39999640 1 --2- 172.21.15.67:0/1812977082 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5d2006ecf0 0x7f5d200711b0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f5d24006010 tx=0x7f5d24000bc0 comp rx=0 tx=0).stop 2024-04-23T00:05:53.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:53.162+0000 7f5d39999640 1 -- 172.21.15.67:0/1812977082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d341962f0 msgr2=0x7f5d3419a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T00:05:53.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:53.162+0000 7f5d39999640 1 --2- 172.21.15.67:0/1812977082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d341962f0 0x7f5d3419a750 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c00b400 tx=0x7f5d1c00b8c0 comp rx=0 tx=0).stop 2024-04-23T00:05:53.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:53.162+0000 7f5d39999640 1 -- 172.21.15.67:0/1812977082 shutdown_connections 2024-04-23T00:05:53.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:53.162+0000 7f5d39999640 1 --2- 172.21.15.67:0/1812977082 >> [v2:172.21.15.67:6800/3392236702,v1:172.21.15.67:6801/3392236702] conn(0x7f5d2006ecf0 0x7f5d200711b0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:53.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:53.162+0000 7f5d39999640 1 --2- 172.21.15.67:0/1812977082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d341962f0 0x7f5d3419a750 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:53.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:53.162+0000 7f5d39999640 1 --2- 172.21.15.67:0/1812977082 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5d340ff170 0x7f5d34195db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T00:05:53.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:53.162+0000 7f5d39999640 1 -- 172.21.15.67:0/1812977082 >> 172.21.15.67:0/1812977082 conn(0x7f5d340fa9d0 msgr2=0x7f5d34103340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T00:05:53.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:53.162+0000 7f5d39999640 1 -- 172.21.15.67:0/1812977082 shutdown_connections 2024-04-23T00:05:53.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-23T00:05:53.162+0000 7f5d39999640 1 -- 172.21.15.67:0/1812977082 wait complete. 2024-04-23T00:05:53.159 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:52 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/2271660307' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T00:05:53.159 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:52 smithi067 ceph-mon[112438]: pgmap v1189: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:05:53.159 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:52 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1570947989' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T00:05:53.160 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:52 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/3089619875' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-23T00:05:53.161 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi067.front.sepia.ceph.com... 2024-04-23T00:05:53.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T00:05:53.161 DEBUG:teuthology.orchestra.run.smithi067:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-23T00:05:53.187 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T00:05:54.230 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:53 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1558596298' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-23T00:05:54.230 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:53 smithi134 ceph-mon[85630]: from='client.? 172.21.15.67:0/1812977082' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-23T00:05:54.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:53 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1558596298' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-23T00:05:54.281 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:53 smithi067 ceph-mon[112438]: from='client.? 172.21.15.67:0/1812977082' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-23T00:05:55.230 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:54 smithi134 ceph-mon[85630]: pgmap v1190: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:05:55.280 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:54 smithi067 ceph-mon[112438]: pgmap v1190: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:05:56.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:05:56.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:05:57.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:57 smithi134 ceph-mon[85630]: pgmap v1191: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:05:57.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:57 smithi067 ceph-mon[112438]: pgmap v1191: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:05:59.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:59 smithi134 ceph-mon[85630]: pgmap v1192: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:05:59.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:59 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T00:05:59.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:59 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T00:05:59.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:59 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T00:05:59.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:05:59 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:05:59.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:59 smithi067 ceph-mon[112438]: pgmap v1192: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:05:59.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:59 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T00:05:59.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:59 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T00:05:59.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:59 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T00:05:59.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:05:59 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:06:01.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:06:01 smithi134 ceph-mon[85630]: pgmap v1193: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:06:01.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:06:01 smithi067 ceph-mon[112438]: pgmap v1193: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:06:03.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:06:03 smithi134 ceph-mon[85630]: pgmap v1194: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:06:03.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:06:03 smithi067 ceph-mon[112438]: pgmap v1194: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:06:05.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:06:05 smithi134 ceph-mon[85630]: pgmap v1195: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:06:05.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:06:05 smithi067 ceph-mon[112438]: pgmap v1195: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:06:07.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:06:07 smithi134 ceph-mon[85630]: pgmap v1196: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:06:07.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:06:07 smithi067 ceph-mon[112438]: pgmap v1196: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:06:09.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:06:09 smithi134 ceph-mon[85630]: pgmap v1197: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:06:09.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:06:09 smithi067 ceph-mon[112438]: pgmap v1197: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:06:11.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:06:11 smithi134 ceph-mon[85630]: pgmap v1198: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:06:11.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:06:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:06:11.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:06:11 smithi067 ceph-mon[112438]: pgmap v1198: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:06:11.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:06:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:06:13.482 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:06:13 smithi134 ceph-mon[85630]: pgmap v1199: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:06:13.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:06:13 smithi067 ceph-mon[112438]: pgmap v1199: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:06:15.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:06:15 smithi134 ceph-mon[85630]: pgmap v1200: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:06:15.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:06:15 smithi067 ceph-mon[112438]: pgmap v1200: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:06:17.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:06:17 smithi134 ceph-mon[85630]: pgmap v1201: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:06:17.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:06:17 smithi067 ceph-mon[112438]: pgmap v1201: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:06:19.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:06:19 smithi134 ceph-mon[85630]: pgmap v1202: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:06:19.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:06:19 smithi067 ceph-mon[112438]: pgmap v1202: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:06:20.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:06:20 smithi067 ceph-mon[112438]: pgmap v1203: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:06:20.729 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:06:20 smithi134 ceph-mon[85630]: pgmap v1203: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:06:23.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:06:23 smithi134 ceph-mon[85630]: pgmap v1204: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:06:23.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:06:23 smithi067 ceph-mon[112438]: pgmap v1204: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:06:25.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:06:25 smithi134 ceph-mon[85630]: pgmap v1205: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:06:25.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:06:25 smithi067 ceph-mon[112438]: pgmap v1205: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:06:26.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:06:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:06:26.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:06:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:06:27.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:06:27 smithi134 ceph-mon[85630]: pgmap v1206: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:06:27.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:06:27 smithi067 ceph-mon[112438]: pgmap v1206: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:06:29.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:06:29 smithi134 ceph-mon[85630]: pgmap v1207: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:06:29.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:06:29 smithi067 ceph-mon[112438]: pgmap v1207: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:06:31.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:06:31 smithi134 ceph-mon[85630]: pgmap v1208: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:06:31.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:06:31 smithi067 ceph-mon[112438]: pgmap v1208: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:06:33.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:06:33 smithi134 ceph-mon[85630]: pgmap v1209: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:06:33.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:06:33 smithi067 ceph-mon[112438]: pgmap v1209: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:06:35.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:06:35 smithi134 ceph-mon[85630]: pgmap v1210: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:06:35.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:06:35 smithi067 ceph-mon[112438]: pgmap v1210: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:06:37.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:06:37 smithi134 ceph-mon[85630]: pgmap v1211: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:06:37.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:06:37 smithi067 ceph-mon[112438]: pgmap v1211: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:06:39.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:06:39 smithi134 ceph-mon[85630]: pgmap v1212: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:06:39.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:06:39 smithi067 ceph-mon[112438]: pgmap v1212: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:06:40.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:06:40 smithi134 ceph-mon[85630]: pgmap v1213: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:06:40.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:06:40 smithi067 ceph-mon[112438]: pgmap v1213: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:06:41.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:06:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:06:41.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:06:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:06:42.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:06:42 smithi134 ceph-mon[85630]: pgmap v1214: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:06:42.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:06:42 smithi067 ceph-mon[112438]: pgmap v1214: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:06:45.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:06:45 smithi134 ceph-mon[85630]: pgmap v1215: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:06:45.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:06:45 smithi067 ceph-mon[112438]: pgmap v1215: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:06:47.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:06:47 smithi134 ceph-mon[85630]: pgmap v1216: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:06:47.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:06:47 smithi067 ceph-mon[112438]: pgmap v1216: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:06:49.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:06:49 smithi134 ceph-mon[85630]: pgmap v1217: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:06:49.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:06:49 smithi067 ceph-mon[112438]: pgmap v1217: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:06:51.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:06:51 smithi134 ceph-mon[85630]: pgmap v1218: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:06:51.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:06:51 smithi067 ceph-mon[112438]: pgmap v1218: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:06:53.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:06:53 smithi134 ceph-mon[85630]: pgmap v1219: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:06:53.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:06:53 smithi067 ceph-mon[112438]: pgmap v1219: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:06:55.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:06:55 smithi134 ceph-mon[85630]: pgmap v1220: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:06:55.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:06:55 smithi067 ceph-mon[112438]: pgmap v1220: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:06:56.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:06:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:06:56.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:06:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:06:57.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:06:57 smithi134 ceph-mon[85630]: pgmap v1221: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:06:57.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:06:57 smithi067 ceph-mon[112438]: pgmap v1221: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:06:59.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:06:59 smithi134 ceph-mon[85630]: pgmap v1222: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:06:59.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:06:59 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T00:06:59.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:06:59 smithi067 ceph-mon[112438]: pgmap v1222: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:06:59.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:06:59 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T00:07:00.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:00 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T00:07:00.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:00 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T00:07:00.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:00 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:07:00.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:00 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T00:07:00.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:00 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T00:07:00.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:00 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:07:01.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:01 smithi134 ceph-mon[85630]: pgmap v1223: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:07:01.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:01 smithi067 ceph-mon[112438]: pgmap v1223: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:07:02.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:02 smithi134 ceph-mon[85630]: pgmap v1224: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:07:02.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:02 smithi067 ceph-mon[112438]: pgmap v1224: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:07:05.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:05 smithi134 ceph-mon[85630]: pgmap v1225: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:07:05.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:05 smithi067 ceph-mon[112438]: pgmap v1225: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:07:07.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:07 smithi134 ceph-mon[85630]: pgmap v1226: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:07:07.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:07 smithi067 ceph-mon[112438]: pgmap v1226: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:07:09.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:09 smithi134 ceph-mon[85630]: pgmap v1227: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:07:09.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:09 smithi067 ceph-mon[112438]: pgmap v1227: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:07:11.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:11 smithi134 ceph-mon[85630]: pgmap v1228: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:07:11.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:07:11.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:11 smithi067 ceph-mon[112438]: pgmap v1228: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:07:11.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:07:13.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:13 smithi134 ceph-mon[85630]: pgmap v1229: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:07:13.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:13 smithi067 ceph-mon[112438]: pgmap v1229: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:07:15.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:15 smithi134 ceph-mon[85630]: pgmap v1230: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:07:15.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:15 smithi067 ceph-mon[112438]: pgmap v1230: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:07:17.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:17 smithi134 ceph-mon[85630]: pgmap v1231: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:07:17.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:17 smithi067 ceph-mon[112438]: pgmap v1231: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:07:19.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:19 smithi134 ceph-mon[85630]: pgmap v1232: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:07:19.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:19 smithi067 ceph-mon[112438]: pgmap v1232: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:07:20.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:20 smithi067 ceph-mon[112438]: pgmap v1233: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:07:20.729 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:20 smithi134 ceph-mon[85630]: pgmap v1233: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:07:23.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:23 smithi134 ceph-mon[85630]: pgmap v1234: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:07:23.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:23 smithi067 ceph-mon[112438]: pgmap v1234: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:07:25.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:25 smithi134 ceph-mon[85630]: pgmap v1235: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:07:25.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:25 smithi067 ceph-mon[112438]: pgmap v1235: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:07:26.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:07:26.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:07:27.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:27 smithi134 ceph-mon[85630]: pgmap v1236: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:07:27.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:27 smithi067 ceph-mon[112438]: pgmap v1236: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:07:29.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:29 smithi134 ceph-mon[85630]: pgmap v1237: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:07:29.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:29 smithi067 ceph-mon[112438]: pgmap v1237: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T00:07:31.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:31 smithi134 ceph-mon[85630]: pgmap v1238: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:07:31.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:31 smithi067 ceph-mon[112438]: pgmap v1238: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:07:33.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:33 smithi134 ceph-mon[85630]: pgmap v1239: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:07:33.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:33 smithi067 ceph-mon[112438]: pgmap v1239: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:07:35.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:35 smithi134 ceph-mon[85630]: pgmap v1240: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:07:35.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:35 smithi067 ceph-mon[112438]: pgmap v1240: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:07:36.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:36 smithi134 ceph-mon[85630]: pgmap v1241: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:07:36.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:36 smithi067 ceph-mon[112438]: pgmap v1241: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:07:39.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:39 smithi134 ceph-mon[85630]: pgmap v1242: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:07:39.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:39 smithi067 ceph-mon[112438]: pgmap v1242: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:07:41.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:41 smithi134 ceph-mon[85630]: pgmap v1243: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:07:41.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:07:41.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:41 smithi067 ceph-mon[112438]: pgmap v1243: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:07:41.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:07:43.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:43 smithi134 ceph-mon[85630]: pgmap v1244: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:07:43.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:43 smithi067 ceph-mon[112438]: pgmap v1244: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:07:45.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:45 smithi134 ceph-mon[85630]: pgmap v1245: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:07:45.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:45 smithi067 ceph-mon[112438]: pgmap v1245: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:07:47.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:47 smithi134 ceph-mon[85630]: pgmap v1246: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:07:47.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:47 smithi067 ceph-mon[112438]: pgmap v1246: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:07:49.029 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:48 smithi067 ceph-mon[112438]: pgmap v1247: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:07:49.228 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:48 smithi134 ceph-mon[85630]: pgmap v1247: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:07:51.478 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:51 smithi134 ceph-mon[85630]: pgmap v1248: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:07:51.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:51 smithi067 ceph-mon[112438]: pgmap v1248: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:07:53.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:53 smithi134 ceph-mon[85630]: pgmap v1249: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:07:53.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:53 smithi067 ceph-mon[112438]: pgmap v1249: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:07:55.478 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:55 smithi134 ceph-mon[85630]: pgmap v1250: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:07:55.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:55 smithi067 ceph-mon[112438]: pgmap v1250: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:07:56.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:07:56.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:07:57.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:57 smithi134 ceph-mon[85630]: pgmap v1251: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:07:57.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:57 smithi067 ceph-mon[112438]: pgmap v1251: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:07:58.419 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 23 00:07:58 smithi067 sudo[168643]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T00:07:58.420 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 23 00:07:58 smithi067 sudo[168643]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T00:07:58.420 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 23 00:07:58 smithi067 sudo[168643]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T00:07:58.420 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 23 00:07:58 smithi067 sudo[168643]: pam_unix(sudo:session): session closed for user root 2024-04-23T00:07:58.420 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 23 00:07:58 smithi067 sudo[168646]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T00:07:58.420 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 23 00:07:58 smithi067 sudo[168646]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T00:07:58.420 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 23 00:07:58 smithi067 sudo[168646]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T00:07:58.420 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 23 00:07:58 smithi067 sudo[168646]: pam_unix(sudo:session): session closed for user root 2024-04-23T00:07:58.420 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 23 00:07:58 smithi067 sudo[168637]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T00:07:58.421 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 23 00:07:58 smithi067 sudo[168637]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T00:07:58.421 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 23 00:07:58 smithi067 sudo[168637]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T00:07:58.421 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 23 00:07:58 smithi067 sudo[168637]: pam_unix(sudo:session): session closed for user root 2024-04-23T00:07:58.421 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 23 00:07:58 smithi067 sudo[168640]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T00:07:58.421 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 23 00:07:58 smithi067 sudo[168640]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T00:07:58.421 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 23 00:07:58 smithi067 sudo[168640]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T00:07:58.421 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 23 00:07:58 smithi067 sudo[168640]: pam_unix(sudo:session): session closed for user root 2024-04-23T00:07:58.421 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 23 00:07:58 smithi067 sudo[168649]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T00:07:58.421 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 23 00:07:58 smithi067 sudo[168649]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T00:07:58.421 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 23 00:07:58 smithi067 sudo[168649]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T00:07:58.421 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 23 00:07:58 smithi067 sudo[168649]: pam_unix(sudo:session): session closed for user root 2024-04-23T00:07:58.422 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 23 00:07:58 smithi067 sudo[168652]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T00:07:58.422 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 23 00:07:58 smithi067 sudo[168652]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T00:07:58.422 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 23 00:07:58 smithi067 sudo[168652]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T00:07:58.422 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 23 00:07:58 smithi067 sudo[168652]: pam_unix(sudo:session): session closed for user root 2024-04-23T00:07:58.729 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:58 smithi134 sudo[122072]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T00:07:58.729 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:58 smithi134 sudo[122072]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T00:07:58.730 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:58 smithi134 sudo[122072]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T00:07:58.730 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:58 smithi134 sudo[122072]: pam_unix(sudo:session): session closed for user root 2024-04-23T00:07:58.730 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:58 smithi134 sudo[122075]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T00:07:58.730 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:58 smithi134 sudo[122075]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T00:07:58.731 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:58 smithi134 sudo[122075]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T00:07:58.731 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:58 smithi134 sudo[122075]: pam_unix(sudo:session): session closed for user root 2024-04-23T00:07:58.731 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 23 00:07:58 smithi134 sudo[122060]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T00:07:58.732 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 23 00:07:58 smithi134 sudo[122060]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T00:07:58.732 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 23 00:07:58 smithi134 sudo[122060]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T00:07:58.732 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 23 00:07:58 smithi134 sudo[122060]: pam_unix(sudo:session): session closed for user root 2024-04-23T00:07:58.732 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 23 00:07:58 smithi134 sudo[122063]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T00:07:58.732 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 23 00:07:58 smithi134 sudo[122063]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T00:07:58.733 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 23 00:07:58 smithi134 sudo[122063]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T00:07:58.733 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 23 00:07:58 smithi134 sudo[122063]: pam_unix(sudo:session): session closed for user root 2024-04-23T00:07:58.733 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 23 00:07:58 smithi134 sudo[122054]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T00:07:58.733 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 23 00:07:58 smithi134 sudo[122054]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T00:07:58.734 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 23 00:07:58 smithi134 sudo[122054]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T00:07:58.734 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 23 00:07:58 smithi134 sudo[122054]: pam_unix(sudo:session): session closed for user root 2024-04-23T00:07:58.734 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 23 00:07:58 smithi134 sudo[122057]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T00:07:58.734 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 23 00:07:58 smithi134 sudo[122057]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T00:07:58.734 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 23 00:07:58 smithi134 sudo[122057]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T00:07:58.734 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 23 00:07:58 smithi134 sudo[122057]: pam_unix(sudo:session): session closed for user root 2024-04-23T00:07:58.735 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:07:58 smithi134 sudo[122066]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T00:07:58.735 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:07:58 smithi134 sudo[122066]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T00:07:58.735 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:07:58 smithi134 sudo[122066]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T00:07:58.735 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:07:58 smithi134 sudo[122066]: pam_unix(sudo:session): session closed for user root 2024-04-23T00:07:58.736 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:07:58 smithi134 sudo[122069]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T00:07:58.736 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:07:58 smithi134 sudo[122069]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T00:07:58.736 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:07:58 smithi134 sudo[122069]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T00:07:58.736 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:07:58 smithi134 sudo[122069]: pam_unix(sudo:session): session closed for user root 2024-04-23T00:07:58.779 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:58 smithi067 sudo[168655]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T00:07:58.858 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:58 smithi067 sudo[168655]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T00:07:58.858 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:58 smithi067 sudo[168655]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T00:07:58.859 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:58 smithi067 sudo[168655]: pam_unix(sudo:session): session closed for user root 2024-04-23T00:07:58.859 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:58 smithi067 sudo[168658]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T00:07:58.859 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:58 smithi067 sudo[168658]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T00:07:58.859 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:58 smithi067 sudo[168658]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T00:07:58.859 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:58 smithi067 sudo[168658]: pam_unix(sudo:session): session closed for user root 2024-04-23T00:07:59.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:59 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:07:59.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:59 smithi134 ceph-mon[85630]: pgmap v1252: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:07:59.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:59 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:07:59.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:59 smithi134 ceph-mon[85630]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T00:07:59.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:59 smithi134 ceph-mon[85630]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T00:07:59.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:59 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-23T00:07:59.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:59 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-04-23T00:07:59.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:59 smithi134 ceph-mon[85630]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T00:07:59.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:59 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-23T00:07:59.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:59 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-04-23T00:07:59.480 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:07:59 smithi134 ceph-mon[85630]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T00:07:59.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:59 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:07:59.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:59 smithi067 ceph-mon[112438]: pgmap v1252: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:07:59.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:59 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:07:59.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:59 smithi067 ceph-mon[112438]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T00:07:59.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:59 smithi067 ceph-mon[112438]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T00:07:59.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:59 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-23T00:07:59.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:59 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-04-23T00:07:59.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:59 smithi067 ceph-mon[112438]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T00:07:59.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:59 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi067"}]: dispatch 2024-04-23T00:07:59.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:59 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "mon metadata", "id": "smithi134"}]: dispatch 2024-04-23T00:07:59.531 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:07:59 smithi067 ceph-mon[112438]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T00:08:00.478 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:08:00 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T00:08:00.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:08:00 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T00:08:01.478 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:08:01 smithi134 ceph-mon[85630]: pgmap v1253: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:08:01.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:08:01 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T00:08:01.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:08:01 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T00:08:01.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:08:01 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:08:01.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:08:01 smithi067 ceph-mon[112438]: pgmap v1253: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:08:01.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:08:01 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T00:08:01.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:08:01 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T00:08:01.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:08:01 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:08:03.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:08:03 smithi134 ceph-mon[85630]: pgmap v1254: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 21 KiB/s rd, 87 KiB/s wr, 11 op/s 2024-04-23T00:08:03.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:08:03 smithi067 ceph-mon[112438]: pgmap v1254: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 21 KiB/s rd, 87 KiB/s wr, 11 op/s 2024-04-23T00:08:05.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:08:05 smithi134 ceph-mon[85630]: pgmap v1255: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 21 KiB/s rd, 87 KiB/s wr, 12 op/s 2024-04-23T00:08:05.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:08:05 smithi067 ceph-mon[112438]: pgmap v1255: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 21 KiB/s rd, 87 KiB/s wr, 12 op/s 2024-04-23T00:08:07.478 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:08:07 smithi134 ceph-mon[85630]: pgmap v1256: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 21 KiB/s rd, 87 KiB/s wr, 11 op/s 2024-04-23T00:08:07.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:08:07 smithi067 ceph-mon[112438]: pgmap v1256: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 21 KiB/s rd, 87 KiB/s wr, 11 op/s 2024-04-23T00:08:09.479 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:08:09 smithi134 ceph-mon[85630]: pgmap v1257: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 21 KiB/s rd, 87 KiB/s wr, 12 op/s 2024-04-23T00:08:09.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:08:09 smithi067 ceph-mon[112438]: pgmap v1257: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 21 KiB/s rd, 87 KiB/s wr, 12 op/s 2024-04-23T00:08:11.478 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:08:11 smithi134 ceph-mon[85630]: pgmap v1258: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 21 KiB/s rd, 87 KiB/s wr, 11 op/s 2024-04-23T00:08:11.478 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:08:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:08:11.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:08:11 smithi067 ceph-mon[112438]: pgmap v1258: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 21 KiB/s rd, 87 KiB/s wr, 11 op/s 2024-04-23T00:08:11.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:08:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:08:13.478 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:08:13 smithi134 ceph-mon[85630]: pgmap v1259: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 21 KiB/s rd, 87 KiB/s wr, 11 op/s 2024-04-23T00:08:13.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:08:13 smithi067 ceph-mon[112438]: pgmap v1259: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 21 KiB/s rd, 87 KiB/s wr, 11 op/s 2024-04-23T00:08:14.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:08:14 smithi067 ceph-mon[112438]: pgmap v1260: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:08:14.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:08:14 smithi134 ceph-mon[85630]: pgmap v1260: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:08:17.478 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:08:17 smithi134 ceph-mon[85630]: pgmap v1261: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:08:17.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:08:17 smithi067 ceph-mon[112438]: pgmap v1261: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:08:19.478 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:08:19 smithi134 ceph-mon[85630]: pgmap v1262: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:08:19.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:08:19 smithi067 ceph-mon[112438]: pgmap v1262: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:08:21.478 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:08:21 smithi134 ceph-mon[85630]: pgmap v1263: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:08:21.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:08:21 smithi067 ceph-mon[112438]: pgmap v1263: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:08:23.478 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:08:23 smithi134 ceph-mon[85630]: pgmap v1264: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:08:23.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:08:23 smithi067 ceph-mon[112438]: pgmap v1264: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:08:25.478 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:08:25 smithi134 ceph-mon[85630]: pgmap v1265: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:08:25.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:08:25 smithi067 ceph-mon[112438]: pgmap v1265: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:08:26.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:08:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:08:26.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:08:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:08:27.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:08:27 smithi067 ceph-mon[112438]: pgmap v1266: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:08:27.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:08:27 smithi134 ceph-mon[85630]: pgmap v1266: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:08:28.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:08:28 smithi067 ceph-mon[112438]: pgmap v1267: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:08:28.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:08:28 smithi134 ceph-mon[85630]: pgmap v1267: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:08:31.478 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:08:31 smithi134 ceph-mon[85630]: pgmap v1268: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:08:31.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:08:31 smithi067 ceph-mon[112438]: pgmap v1268: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:08:33.478 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:08:33 smithi134 ceph-mon[85630]: pgmap v1269: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:08:33.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:08:33 smithi067 ceph-mon[112438]: pgmap v1269: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:08:35.478 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:08:35 smithi134 ceph-mon[85630]: pgmap v1270: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:08:35.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:08:35 smithi067 ceph-mon[112438]: pgmap v1270: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:08:37.478 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:08:37 smithi134 ceph-mon[85630]: pgmap v1271: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:08:37.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:08:37 smithi067 ceph-mon[112438]: pgmap v1271: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:08:39.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:08:39 smithi067 ceph-mon[112438]: pgmap v1272: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:08:39.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:08:39 smithi134 ceph-mon[85630]: pgmap v1272: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:08:40.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:08:40 smithi067 ceph-mon[112438]: pgmap v1273: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:08:40.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:08:40 smithi134 ceph-mon[85630]: pgmap v1273: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:08:41.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:08:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:08:41.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:08:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:08:42.512 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:08:42 smithi134 ceph-mon[85630]: pgmap v1274: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:08:42.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:08:42 smithi067 ceph-mon[112438]: pgmap v1274: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:08:45.478 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:08:45 smithi134 ceph-mon[85630]: pgmap v1275: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:08:45.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:08:45 smithi067 ceph-mon[112438]: pgmap v1275: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:08:47.478 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:08:47 smithi134 ceph-mon[85630]: pgmap v1276: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:08:47.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:08:47 smithi067 ceph-mon[112438]: pgmap v1276: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:08:49.478 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:08:49 smithi134 ceph-mon[85630]: pgmap v1277: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:08:49.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:08:49 smithi067 ceph-mon[112438]: pgmap v1277: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:08:51.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:08:51 smithi067 ceph-mon[112438]: pgmap v1278: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:08:51.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:08:51 smithi134 ceph-mon[85630]: pgmap v1278: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:08:52.625 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:08:52 smithi134 ceph-mon[85630]: pgmap v1279: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:08:52.779 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:08:52 smithi067 ceph-mon[112438]: pgmap v1279: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:08:55.478 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:08:55 smithi134 ceph-mon[85630]: pgmap v1280: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:08:55.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:08:55 smithi067 ceph-mon[112438]: pgmap v1280: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:08:56.278 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:08:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:08:56.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:08:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:08:57.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:08:57 smithi067 ceph-mon[112438]: pgmap v1281: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:08:57.682 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:08:57 smithi134 ceph-mon[85630]: pgmap v1281: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:08:59.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:08:59 smithi067 ceph-mon[112438]: pgmap v1282: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:08:59.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:08:59 smithi134 ceph-mon[85630]: pgmap v1282: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:09:00.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:09:00 smithi067 ceph-mon[112438]: pgmap v1283: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:09:00.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:09:00 smithi134 ceph-mon[85630]: pgmap v1283: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:09:01.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:09:01 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T00:09:01.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:09:01 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T00:09:02.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:09:02 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T00:09:02.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:09:02 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T00:09:02.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:09:02 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:09:02.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:09:02 smithi067 ceph-mon[112438]: pgmap v1284: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:09:02.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:09:02 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T00:09:02.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:09:02 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T00:09:02.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:09:02 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:09:02.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:09:02 smithi134 ceph-mon[85630]: pgmap v1284: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:09:05.478 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:09:05 smithi134 ceph-mon[85630]: pgmap v1285: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:09:05.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:09:05 smithi067 ceph-mon[112438]: pgmap v1285: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:09:07.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:09:07 smithi067 ceph-mon[112438]: pgmap v1286: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:09:07.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:09:07 smithi134 ceph-mon[85630]: pgmap v1286: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:09:09.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:09:09 smithi067 ceph-mon[112438]: pgmap v1287: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:09:09.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:09:09 smithi134 ceph-mon[85630]: pgmap v1287: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:09:11.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:09:11 smithi067 ceph-mon[112438]: pgmap v1288: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:09:11.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:09:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:09:11.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:09:11 smithi134 ceph-mon[85630]: pgmap v1288: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:09:11.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:09:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:09:12.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:09:12 smithi067 ceph-mon[112438]: pgmap v1289: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:09:12.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:09:12 smithi134 ceph-mon[85630]: pgmap v1289: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:09:15.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:09:15 smithi067 ceph-mon[112438]: pgmap v1290: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:09:15.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:09:15 smithi134 ceph-mon[85630]: pgmap v1290: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:09:17.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:09:17 smithi067 ceph-mon[112438]: pgmap v1291: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:09:17.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:09:17 smithi134 ceph-mon[85630]: pgmap v1291: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:09:19.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:09:19 smithi067 ceph-mon[112438]: pgmap v1292: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:09:19.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:09:19 smithi134 ceph-mon[85630]: pgmap v1292: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:09:20.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:09:20 smithi134 ceph-mon[85630]: pgmap v1293: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:09:20.778 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:09:20 smithi067 ceph-mon[112438]: pgmap v1293: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:09:23.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:09:23 smithi067 ceph-mon[112438]: pgmap v1294: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:09:23.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:09:23 smithi134 ceph-mon[85630]: pgmap v1294: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:09:25.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:09:25 smithi067 ceph-mon[112438]: pgmap v1295: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:09:25.530 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:09:25 smithi134 ceph-mon[85630]: pgmap v1295: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:09:26.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:09:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:09:26.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:09:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:09:27.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:09:27 smithi067 ceph-mon[112438]: pgmap v1296: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:09:27.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:09:27 smithi134 ceph-mon[85630]: pgmap v1296: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:09:28.478 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:09:28 smithi134 ceph-mon[85630]: pgmap v1297: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:09:28.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:09:28 smithi067 ceph-mon[112438]: pgmap v1297: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:09:31.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:09:31 smithi067 ceph-mon[112438]: pgmap v1298: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:09:31.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:09:31 smithi134 ceph-mon[85630]: pgmap v1298: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:09:33.478 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:09:33 smithi134 ceph-mon[85630]: pgmap v1299: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:09:33.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:09:33 smithi067 ceph-mon[112438]: pgmap v1299: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:09:34.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:09:34 smithi134 ceph-mon[85630]: pgmap v1300: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:09:34.778 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:09:34 smithi067 ceph-mon[112438]: pgmap v1300: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:09:37.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:09:37 smithi067 ceph-mon[112438]: pgmap v1301: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:09:37.727 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:09:37 smithi134 ceph-mon[85630]: pgmap v1301: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:09:39.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:09:39 smithi067 ceph-mon[112438]: pgmap v1302: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:09:39.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:09:39 smithi134 ceph-mon[85630]: pgmap v1302: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:09:40.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:09:40 smithi134 ceph-mon[85630]: pgmap v1303: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:09:40.778 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:09:40 smithi067 ceph-mon[112438]: pgmap v1303: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:09:41.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:09:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:09:41.779 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:09:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:09:42.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:09:42 smithi134 ceph-mon[85630]: pgmap v1304: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:09:42.778 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:09:42 smithi067 ceph-mon[112438]: pgmap v1304: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:09:45.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:09:45 smithi067 ceph-mon[112438]: pgmap v1305: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:09:45.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:09:45 smithi134 ceph-mon[85630]: pgmap v1305: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:09:47.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:09:47 smithi067 ceph-mon[112438]: pgmap v1306: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:09:47.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:09:47 smithi134 ceph-mon[85630]: pgmap v1306: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:09:48.430 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:09:48 smithi067 ceph-mon[112438]: pgmap v1307: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:09:48.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:09:48 smithi134 ceph-mon[85630]: pgmap v1307: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:09:51.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:09:51 smithi067 ceph-mon[112438]: pgmap v1308: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:09:51.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:09:51 smithi134 ceph-mon[85630]: pgmap v1308: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:09:52.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:09:52 smithi134 ceph-mon[85630]: pgmap v1309: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:09:52.778 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:09:52 smithi067 ceph-mon[112438]: pgmap v1309: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:09:55.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:09:55 smithi067 ceph-mon[112438]: pgmap v1310: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:09:55.529 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:09:55 smithi134 ceph-mon[85630]: pgmap v1310: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:09:56.530 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:09:56 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:09:56.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:09:56 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:09:57.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:09:57 smithi067 ceph-mon[112438]: pgmap v1311: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:09:57.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:09:57 smithi134 ceph-mon[85630]: pgmap v1311: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:09:58.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:09:58 smithi067 ceph-mon[112438]: pgmap v1312: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:09:58.727 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:09:58 smithi134 ceph-mon[85630]: pgmap v1312: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:10:00.478 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:00 smithi134 ceph-mon[85630]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-23T00:10:00.478 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:00 smithi134 ceph-mon[85630]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-23T00:10:00.478 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:00 smithi134 ceph-mon[85630]: fs cephfs has deprecated feature inline_data enabled. 2024-04-23T00:10:00.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:00 smithi067 ceph-mon[112438]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-04-23T00:10:00.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:00 smithi067 ceph-mon[112438]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-04-23T00:10:00.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:00 smithi067 ceph-mon[112438]: fs cephfs has deprecated feature inline_data enabled. 2024-04-23T00:10:01.477 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:01 smithi134 ceph-mon[85630]: pgmap v1313: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:10:01.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:01 smithi067 ceph-mon[112438]: pgmap v1313: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:10:02.477 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:02 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T00:10:02.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:02 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T00:10:03.419 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:03 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T00:10:03.420 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:03 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T00:10:03.420 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:03 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:10:03.420 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:03 smithi134 ceph-mon[85630]: pgmap v1314: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:10:03.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:03 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T00:10:03.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:03 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T00:10:03.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:03 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' 2024-04-23T00:10:03.529 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:03 smithi067 ceph-mon[112438]: pgmap v1314: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:10:05.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:05 smithi067 ceph-mon[112438]: pgmap v1315: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:10:05.727 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:05 smithi134 ceph-mon[85630]: pgmap v1315: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:10:06.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:06 smithi067 ceph-mon[112438]: pgmap v1316: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:10:06.727 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:06 smithi134 ceph-mon[85630]: pgmap v1316: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:10:09.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:09 smithi067 ceph-mon[112438]: pgmap v1317: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:10:09.727 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:09 smithi134 ceph-mon[85630]: pgmap v1317: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:10:10.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:10 smithi067 ceph-mon[112438]: pgmap v1318: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:10:10.727 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:10 smithi134 ceph-mon[85630]: pgmap v1318: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:10:11.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:11 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:10:11.727 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:11 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:10:12.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:12 smithi067 ceph-mon[112438]: pgmap v1319: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:10:12.727 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:12 smithi134 ceph-mon[85630]: pgmap v1319: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:10:14.728 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:14 smithi134 ceph-mon[85630]: pgmap v1320: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:10:14.778 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:14 smithi067 ceph-mon[112438]: pgmap v1320: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:10:16.727 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:16 smithi134 ceph-mon[85630]: pgmap v1321: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:10:16.778 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:16 smithi067 ceph-mon[112438]: pgmap v1321: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:10:18.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:18 smithi067 ceph-mon[112438]: pgmap v1322: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:10:18.590 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:18 smithi134 ceph-mon[85630]: pgmap v1322: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:10:20.727 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:20 smithi134 ceph-mon[85630]: pgmap v1323: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:10:20.778 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:20 smithi067 ceph-mon[112438]: pgmap v1323: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:10:22.727 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:22 smithi134 ceph-mon[85630]: pgmap v1324: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:10:22.778 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:22 smithi067 ceph-mon[112438]: pgmap v1324: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:10:24.727 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:24 smithi134 ceph-mon[85630]: pgmap v1325: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:10:24.778 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:24 smithi067 ceph-mon[112438]: pgmap v1325: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:10:26.477 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:26 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:10:26.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:26 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:10:27.477 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:27 smithi134 ceph-mon[85630]: pgmap v1326: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:10:27.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:27 smithi067 ceph-mon[112438]: pgmap v1326: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:10:28.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:28 smithi067 ceph-mon[112438]: pgmap v1327: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:10:28.704 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:28 smithi134 ceph-mon[85630]: pgmap v1327: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:10:30.727 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:30 smithi134 ceph-mon[85630]: pgmap v1328: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:10:30.778 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:30 smithi067 ceph-mon[112438]: pgmap v1328: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:10:32.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:32 smithi067 ceph-mon[112438]: pgmap v1329: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:10:32.727 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:32 smithi134 ceph-mon[85630]: pgmap v1329: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:10:34.727 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:34 smithi134 ceph-mon[85630]: pgmap v1330: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:10:34.778 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:34 smithi067 ceph-mon[112438]: pgmap v1330: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:10:36.727 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:36 smithi134 ceph-mon[85630]: pgmap v1331: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:10:36.778 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:36 smithi067 ceph-mon[112438]: pgmap v1331: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:10:38.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:38 smithi067 ceph-mon[112438]: pgmap v1332: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:10:38.727 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:38 smithi134 ceph-mon[85630]: pgmap v1332: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:10:40.727 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:40 smithi134 ceph-mon[85630]: pgmap v1333: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:10:40.778 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:40 smithi067 ceph-mon[112438]: pgmap v1333: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:10:41.730 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:41 smithi134 ceph-mon[85630]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:10:41.778 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:41 smithi067 ceph-mon[112438]: from='mgr.34102 172.21.15.67:0/1772269179' entity='mgr.smithi067.whdcsr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T00:10:42.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:42 smithi067 ceph-mon[112438]: pgmap v1334: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:10:42.727 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:42 smithi134 ceph-mon[85630]: pgmap v1334: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:10:44.727 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:44 smithi134 ceph-mon[85630]: pgmap v1335: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:10:44.778 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:44 smithi067 ceph-mon[112438]: pgmap v1335: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:10:46.727 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:46 smithi134 ceph-mon[85630]: pgmap v1336: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:10:46.778 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:46 smithi067 ceph-mon[112438]: pgmap v1336: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:10:48.528 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:48 smithi067 ceph-mon[112438]: pgmap v1337: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:10:48.727 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:48 smithi134 ceph-mon[85630]: pgmap v1337: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-23T00:10:50.727 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:50 smithi134 ceph-mon[85630]: pgmap v1338: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:10:50.778 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:50 smithi067 ceph-mon[112438]: pgmap v1338: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-23T00:10:52.259 ERROR:tasks.cephfs.fuse_mount:process failed to terminate after unmount. This probably indicates a bug within ceph-fuse. 2024-04-23T00:10:52.260 ERROR:teuthology.run_tasks:Manager failed: ceph-fuse Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_43be020184947e53516056c9931e1ac5bdbbb1a5/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph-c_43be020184947e53516056c9931e1ac5bdbbb1a5/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-23T00:10:52.262 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-23T00:10:52.287 INFO:tasks.cephadm:Teardown begin 2024-04-23T00:10:52.287 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_43be020184947e53516056c9931e1ac5bdbbb1a5/qa/tasks/cephadm.py", line 2246, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_43be020184947e53516056c9931e1ac5bdbbb1a5/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph-c_43be020184947e53516056c9931e1ac5bdbbb1a5/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-23T00:10:52.288 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T00:10:52.322 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T00:10:52.355 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-23T00:10:52.355 DEBUG:teuthology.orchestra.run.smithi067:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-23T00:10:52.379 DEBUG:teuthology.orchestra.run.smithi134:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-23T00:10:52.411 INFO:tasks.cephadm:Stopping all daemons... 2024-04-23T00:10:52.412 INFO:tasks.cephadm.mon.smithi067:Stopping mon.smithi067... 2024-04-23T00:10:52.412 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-d46c346e-00fd-11ef-bc93-c7b262605968@mon.smithi067 2024-04-23T00:10:52.727 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:52 smithi134 ceph-mon[85630]: pgmap v1339: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:10:52.749 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:52 smithi067 ceph-mon[112438]: pgmap v1339: 65 pgs: 65 active+clean; 310 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-23T00:10:52.749 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:52 smithi067 systemd[1]: Stopping Ceph mon.smithi067 for d46c346e-00fd-11ef-bc93-c7b262605968... 2024-04-23T00:10:53.028 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:52 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[112434]: 2024-04-23T00:10:52.765+0000 7f20195bf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi067 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T00:10:53.028 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:52 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067[112434]: 2024-04-23T00:10:52.765+0000 7f20195bf700 -1 mon.smithi067@0(leader) e3 *** Got Signal Terminated *** 2024-04-23T00:10:53.028 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:52 smithi067 podman[169135]: 2024-04-23 00:10:52.806415247 +0000 UTC m=+0.174781974 container died b879412ee3f1581858c53ba16eaa5c05ccd18c2352d65fa5cfd8fe8a49196ebc (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.vendor=CentOS) 2024-04-23T00:10:53.500 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:53 smithi067 podman[169135]: 2024-04-23 00:10:53.281921986 +0000 UTC m=+0.650288709 container cleanup b879412ee3f1581858c53ba16eaa5c05ccd18c2352d65fa5cfd8fe8a49196ebc (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5) 2024-04-23T00:10:53.500 INFO:journalctl@ceph.mon.smithi067.smithi067.stdout:Apr 23 00:10:53 smithi067 bash[169135]: ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi067 2024-04-23T00:10:53.639 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d46c346e-00fd-11ef-bc93-c7b262605968@mon.smithi067.service' 2024-04-23T00:10:53.687 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T00:10:53.687 INFO:tasks.cephadm.mon.smithi067:Stopped mon.smithi067 2024-04-23T00:10:53.687 INFO:tasks.cephadm.mon.smithi134:Stopping mon.smithi134... 2024-04-23T00:10:53.688 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ceph-d46c346e-00fd-11ef-bc93-c7b262605968@mon.smithi134 2024-04-23T00:10:53.977 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:53 smithi134 systemd[1]: Stopping Ceph mon.smithi134 for d46c346e-00fd-11ef-bc93-c7b262605968... 2024-04-23T00:10:54.445 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:54 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi134[85626]: 2024-04-23T00:10:54.087+0000 7f0e131a1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi134 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T00:10:54.445 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:54 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi134[85626]: 2024-04-23T00:10:54.087+0000 7f0e131a1700 -1 mon.smithi134@1(peon) e3 *** Got Signal Terminated *** 2024-04-23T00:10:54.445 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:54 smithi134 podman[122549]: 2024-04-23 00:10:54.180007972 +0000 UTC m=+0.281155880 container died 20a456e8e56257e85251cd69af703c3ae98985766033c93d6397da443ebc0ff6 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi134, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T00:10:54.727 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:54 smithi134 podman[122549]: 2024-04-23 00:10:54.652943142 +0000 UTC m=+0.754091091 container cleanup 20a456e8e56257e85251cd69af703c3ae98985766033c93d6397da443ebc0ff6 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi134, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-23T00:10:54.727 INFO:journalctl@ceph.mon.smithi134.smithi134.stdout:Apr 23 00:10:54 smithi134 bash[122549]: ceph-d46c346e-00fd-11ef-bc93-c7b262605968-mon-smithi134 2024-04-23T00:10:55.066 DEBUG:teuthology.orchestra.run.smithi134:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d46c346e-00fd-11ef-bc93-c7b262605968@mon.smithi134.service' 2024-04-23T00:10:55.113 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T00:10:55.113 INFO:tasks.cephadm.mon.smithi134:Stopped mon.smithi134 2024-04-23T00:10:55.113 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-23T00:10:55.114 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-d46c346e-00fd-11ef-bc93-c7b262605968@osd.0 2024-04-23T00:10:55.420 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 23 00:10:55 smithi067 systemd[1]: Stopping Ceph osd.0 for d46c346e-00fd-11ef-bc93-c7b262605968... 2024-04-23T00:10:55.778 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 23 00:10:55 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0[123959]: 2024-04-23T00:10:55.436+0000 7fb564753700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T00:10:55.778 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 23 00:10:55 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0[123959]: 2024-04-23T00:10:55.436+0000 7fb564753700 -1 osd.0 94 *** Got signal Terminated *** 2024-04-23T00:10:55.778 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 23 00:10:55 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0[123959]: 2024-04-23T00:10:55.436+0000 7fb564753700 -1 osd.0 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T00:11:01.197 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 23 00:11:00 smithi067 podman[169265]: 2024-04-23 00:11:00.823084505 +0000 UTC m=+5.493226101 container died a74fde1ef34edcffb85b2a275ec89aef7fcb531d7ae1abe3035519a261f301e6 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0, ceph=True, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-23T00:11:01.528 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 23 00:11:01 smithi067 podman[169265]: 2024-04-23 00:11:01.303378405 +0000 UTC m=+5.973520009 container cleanup a74fde1ef34edcffb85b2a275ec89aef7fcb531d7ae1abe3035519a261f301e6 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-23T00:11:01.528 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 23 00:11:01 smithi067 bash[169265]: ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0 2024-04-23T00:11:01.889 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 23 00:11:01 smithi067 podman[169283]: 2024-04-23 00:11:01.563095974 +0000 UTC m=+0.732782761 container remove a74fde1ef34edcffb85b2a275ec89aef7fcb531d7ae1abe3035519a261f301e6 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-23T00:11:02.166 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 23 00:11:01 smithi067 podman[169378]: 2024-04-23 00:11:01.809409677 +0000 UTC m=+0.018852784 image pull a0434699c1d76af4eea1fc95ff0a22ea73d1e04e609464fbf658d6f39bc7f2fc quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c 2024-04-23T00:11:02.166 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 23 00:11:01 smithi067 podman[169378]: 2024-04-23 00:11:01.930816902 +0000 UTC m=+0.140260044 container create 52900e185c44c88dc77da7558f33165a0aa9c6dbb5661fb8676f6e55a6f11705 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T00:11:02.167 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 23 00:11:02 smithi067 podman[169378]: 2024-04-23 00:11:02.09151273 +0000 UTC m=+0.300955827 container init 52900e185c44c88dc77da7558f33165a0aa9c6dbb5661fb8676f6e55a6f11705 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True) 2024-04-23T00:11:02.167 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 23 00:11:02 smithi067 podman[169378]: 2024-04-23 00:11:02.094661406 +0000 UTC m=+0.304104504 container start 52900e185c44c88dc77da7558f33165a0aa9c6dbb5661fb8676f6e55a6f11705 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240416) 2024-04-23T00:11:02.167 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 23 00:11:02 smithi067 podman[169378]: 2024-04-23 00:11:02.183500794 +0000 UTC m=+0.392943894 container attach 52900e185c44c88dc77da7558f33165a0aa9c6dbb5661fb8676f6e55a6f11705 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, CEPH_POINT_RELEASE=) 2024-04-23T00:11:02.777 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 23 00:11:02 smithi067 podman[169404]: 2024-04-23 00:11:02.442540447 +0000 UTC m=+0.019462372 container died 52900e185c44c88dc77da7558f33165a0aa9c6dbb5661fb8676f6e55a6f11705 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-23T00:11:03.073 INFO:journalctl@ceph.osd.0.smithi067.stdout:Apr 23 00:11:03 smithi067 podman[169404]: 2024-04-23 00:11:03.070671359 +0000 UTC m=+0.647593314 container remove 52900e185c44c88dc77da7558f33165a0aa9c6dbb5661fb8676f6e55a6f11705 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-0-deactivate, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5) 2024-04-23T00:11:03.078 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d46c346e-00fd-11ef-bc93-c7b262605968@osd.0.service' 2024-04-23T00:11:03.121 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T00:11:03.121 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-23T00:11:03.122 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-23T00:11:03.122 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-d46c346e-00fd-11ef-bc93-c7b262605968@osd.1 2024-04-23T00:11:03.363 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 23 00:11:03 smithi067 systemd[1]: Stopping Ceph osd.1 for d46c346e-00fd-11ef-bc93-c7b262605968... 2024-04-23T00:11:03.778 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 23 00:11:03 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1[129933]: 2024-04-23T00:11:03.481+0000 7fd3896a8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T00:11:03.778 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 23 00:11:03 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1[129933]: 2024-04-23T00:11:03.481+0000 7fd3896a8700 -1 osd.1 94 *** Got signal Terminated *** 2024-04-23T00:11:03.778 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 23 00:11:03 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1[129933]: 2024-04-23T00:11:03.481+0000 7fd3896a8700 -1 osd.1 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T00:11:09.141 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 23 00:11:08 smithi067 podman[169483]: 2024-04-23 00:11:08.896644805 +0000 UTC m=+5.507908358 container died b003cac17eac384fe520e1d5c967a6d6de8d1707adebf2ea974c9ac43c834373 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416) 2024-04-23T00:11:09.467 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 23 00:11:09 smithi067 podman[169483]: 2024-04-23 00:11:09.223646963 +0000 UTC m=+5.834910515 container cleanup b003cac17eac384fe520e1d5c967a6d6de8d1707adebf2ea974c9ac43c834373 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T00:11:09.467 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 23 00:11:09 smithi067 bash[169483]: ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1 2024-04-23T00:11:09.777 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 23 00:11:09 smithi067 podman[169511]: 2024-04-23 00:11:09.484160536 +0000 UTC m=+0.582259596 container remove b003cac17eac384fe520e1d5c967a6d6de8d1707adebf2ea974c9ac43c834373 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T00:11:10.154 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 23 00:11:09 smithi067 podman[169604]: 2024-04-23 00:11:09.81263317 +0000 UTC m=+0.018975105 image pull a0434699c1d76af4eea1fc95ff0a22ea73d1e04e609464fbf658d6f39bc7f2fc quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c 2024-04-23T00:11:10.154 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 23 00:11:09 smithi067 podman[169604]: 2024-04-23 00:11:09.909937637 +0000 UTC m=+0.116279613 container create f1072a44a48d557943e9a000a81b1638d48fc1ec94b5cc73feb07094d7f020eb (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-22-1618-43be020) 2024-04-23T00:11:10.154 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 23 00:11:10 smithi067 podman[169604]: 2024-04-23 00:11:10.096048137 +0000 UTC m=+0.302390074 container init f1072a44a48d557943e9a000a81b1638d48fc1ec94b5cc73feb07094d7f020eb (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T00:11:10.154 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 23 00:11:10 smithi067 podman[169604]: 2024-04-23 00:11:10.099220155 +0000 UTC m=+0.305562090 container start f1072a44a48d557943e9a000a81b1638d48fc1ec94b5cc73feb07094d7f020eb (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416) 2024-04-23T00:11:10.154 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 23 00:11:10 smithi067 podman[169604]: 2024-04-23 00:11:10.170750341 +0000 UTC m=+0.377092281 container attach f1072a44a48d557943e9a000a81b1638d48fc1ec94b5cc73feb07094d7f020eb (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-23T00:11:10.528 INFO:journalctl@ceph.osd.1.smithi067.stdout:Apr 23 00:11:10 smithi067 podman[169604]: 2024-04-23 00:11:10.404710477 +0000 UTC m=+0.611052425 container died f1072a44a48d557943e9a000a81b1638d48fc1ec94b5cc73feb07094d7f020eb (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T00:11:10.939 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d46c346e-00fd-11ef-bc93-c7b262605968@osd.1.service' 2024-04-23T00:11:10.980 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T00:11:10.980 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-23T00:11:10.980 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-23T00:11:10.980 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-d46c346e-00fd-11ef-bc93-c7b262605968@osd.2 2024-04-23T00:11:11.278 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 23 00:11:11 smithi067 systemd[1]: Stopping Ceph osd.2 for d46c346e-00fd-11ef-bc93-c7b262605968... 2024-04-23T00:11:11.778 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 23 00:11:11 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2[135515]: 2024-04-23T00:11:11.369+0000 7fadf964e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T00:11:11.778 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 23 00:11:11 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2[135515]: 2024-04-23T00:11:11.369+0000 7fadf964e700 -1 osd.2 94 *** Got signal Terminated *** 2024-04-23T00:11:11.778 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 23 00:11:11 smithi067 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2[135515]: 2024-04-23T00:11:11.369+0000 7fadf964e700 -1 osd.2 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T00:11:17.028 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 23 00:11:16 smithi067 podman[169710]: 2024-04-23 00:11:16.732095764 +0000 UTC m=+5.497728505 container died 76df38d881d7d98b46ac93191d7249eed8201ea900c90a4b5df239e9055803a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-23T00:11:17.346 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 23 00:11:17 smithi067 podman[169710]: 2024-04-23 00:11:17.145166446 +0000 UTC m=+5.910799187 container cleanup 76df38d881d7d98b46ac93191d7249eed8201ea900c90a4b5df239e9055803a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T00:11:17.346 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 23 00:11:17 smithi067 bash[169710]: ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2 2024-04-23T00:11:17.770 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 23 00:11:17 smithi067 podman[169728]: 2024-04-23 00:11:17.363115709 +0000 UTC m=+0.631308252 container remove 76df38d881d7d98b46ac93191d7249eed8201ea900c90a4b5df239e9055803a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-23T00:11:18.028 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 23 00:11:17 smithi067 podman[169822]: 2024-04-23 00:11:17.690418285 +0000 UTC m=+0.019093280 image pull a0434699c1d76af4eea1fc95ff0a22ea73d1e04e609464fbf658d6f39bc7f2fc quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c 2024-04-23T00:11:18.028 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 23 00:11:17 smithi067 podman[169822]: 2024-04-23 00:11:17.805721271 +0000 UTC m=+0.134396238 container create 871b2b2f5c39af82ddc04064f0c33974057633d62388e7d07b99ac1dbdfdba10 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-23T00:11:18.028 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 23 00:11:18 smithi067 podman[169822]: 2024-04-23 00:11:18.00029503 +0000 UTC m=+0.328970008 container init 871b2b2f5c39af82ddc04064f0c33974057633d62388e7d07b99ac1dbdfdba10 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS) 2024-04-23T00:11:18.028 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 23 00:11:18 smithi067 podman[169822]: 2024-04-23 00:11:18.00362678 +0000 UTC m=+0.332301747 container start 871b2b2f5c39af82ddc04064f0c33974057633d62388e7d07b99ac1dbdfdba10 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.schema-version=1.0) 2024-04-23T00:11:18.292 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 23 00:11:18 smithi067 podman[169822]: 2024-04-23 00:11:18.083775625 +0000 UTC m=+0.412450593 container attach 871b2b2f5c39af82ddc04064f0c33974057633d62388e7d07b99ac1dbdfdba10 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS) 2024-04-23T00:11:18.572 INFO:journalctl@ceph.osd.2.smithi067.stdout:Apr 23 00:11:18 smithi067 podman[169822]: 2024-04-23 00:11:18.309443672 +0000 UTC m=+0.638118642 container died 871b2b2f5c39af82ddc04064f0c33974057633d62388e7d07b99ac1dbdfdba10 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-23T00:11:18.874 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d46c346e-00fd-11ef-bc93-c7b262605968@osd.2.service' 2024-04-23T00:11:18.914 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T00:11:18.914 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-23T00:11:18.914 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-23T00:11:18.914 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ceph-d46c346e-00fd-11ef-bc93-c7b262605968@osd.3 2024-04-23T00:11:19.227 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 23 00:11:18 smithi134 systemd[1]: Stopping Ceph osd.3 for d46c346e-00fd-11ef-bc93-c7b262605968... 2024-04-23T00:11:19.727 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 23 00:11:19 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3[95082]: 2024-04-23T00:11:19.480+0000 7f2ebfd77700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T00:11:19.727 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 23 00:11:19 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3[95082]: 2024-04-23T00:11:19.480+0000 7f2ebfd77700 -1 osd.3 94 *** Got signal Terminated *** 2024-04-23T00:11:19.727 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 23 00:11:19 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3[95082]: 2024-04-23T00:11:19.480+0000 7f2ebfd77700 -1 osd.3 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T00:11:22.477 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 23 00:11:22 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3[95082]: 2024-04-23T00:11:22.085+0000 7f2eb8363700 -1 osd.3 94 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-04-23T00:10:57.971660+0000 front 2024-04-23T00:10:57.971733+0000 (oldest deadline 2024-04-23T00:11:22.071543+0000) 2024-04-23T00:11:23.477 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 23 00:11:23 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3[95082]: 2024-04-23T00:11:23.072+0000 7f2eb8363700 -1 osd.3 94 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-04-23T00:10:57.971660+0000 front 2024-04-23T00:10:57.971733+0000 (oldest deadline 2024-04-23T00:11:22.071543+0000) 2024-04-23T00:11:24.477 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 23 00:11:24 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3[95082]: 2024-04-23T00:11:24.089+0000 7f2eb8363700 -1 osd.3 94 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-04-23T00:10:57.971660+0000 front 2024-04-23T00:10:57.971733+0000 (oldest deadline 2024-04-23T00:11:22.071543+0000) 2024-04-23T00:11:25.198 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 23 00:11:24 smithi134 podman[122684]: 2024-04-23 00:11:24.917819727 +0000 UTC m=+5.794390481 container died 5f42b665260c42d6c323239ebc3be9b4d9ce2dd63c783bea2fc448f9b4ae2e89 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-23T00:11:25.477 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:25 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[105317]: 2024-04-23T00:11:25.410+0000 7f0df0b07700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-04-23T00:10:57.859194+0000 front 2024-04-23T00:10:57.859226+0000 (oldest deadline 2024-04-23T00:11:24.859323+0000) 2024-04-23T00:11:25.478 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 23 00:11:25 smithi134 podman[122684]: 2024-04-23 00:11:25.305237018 +0000 UTC m=+6.181807769 container cleanup 5f42b665260c42d6c323239ebc3be9b4d9ce2dd63c783bea2fc448f9b4ae2e89 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3, org.label-schema.build-date=20240416, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True) 2024-04-23T00:11:25.478 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 23 00:11:25 smithi134 bash[122684]: ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3 2024-04-23T00:11:25.832 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 23 00:11:25 smithi134 podman[122701]: 2024-04-23 00:11:25.548586881 +0000 UTC m=+0.633345213 container remove 5f42b665260c42d6c323239ebc3be9b4d9ce2dd63c783bea2fc448f9b4ae2e89 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-23T00:11:26.160 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 23 00:11:25 smithi134 podman[122796]: 2024-04-23 00:11:25.751577204 +0000 UTC m=+0.017727194 image pull a0434699c1d76af4eea1fc95ff0a22ea73d1e04e609464fbf658d6f39bc7f2fc quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c 2024-04-23T00:11:26.160 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 23 00:11:25 smithi134 podman[122796]: 2024-04-23 00:11:25.874559785 +0000 UTC m=+0.140709817 container create 9cf6463fa5662a215e45de5785682e0d5d2569a3dad1c5d0afba920e3a2bb724 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3-deactivate, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-23T00:11:26.160 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 23 00:11:26 smithi134 podman[122796]: 2024-04-23 00:11:26.093569216 +0000 UTC m=+0.359719205 container init 9cf6463fa5662a215e45de5785682e0d5d2569a3dad1c5d0afba920e3a2bb724 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-22-1618-43be020) 2024-04-23T00:11:26.160 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 23 00:11:26 smithi134 podman[122796]: 2024-04-23 00:11:26.096877079 +0000 UTC m=+0.363027070 container start 9cf6463fa5662a215e45de5785682e0d5d2569a3dad1c5d0afba920e3a2bb724 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240416, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T00:11:26.161 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 23 00:11:26 smithi134 podman[122796]: 2024-04-23 00:11:26.176909154 +0000 UTC m=+0.443059146 container attach 9cf6463fa5662a215e45de5785682e0d5d2569a3dad1c5d0afba920e3a2bb724 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-23T00:11:26.477 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 23 00:11:26 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4[100010]: 2024-04-23T00:11:26.390+0000 7fcfc94cb700 -1 osd.4 94 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-04-23T00:10:59.480484+0000 front 2024-04-23T00:10:59.480552+0000 (oldest deadline 2024-04-23T00:11:25.880525+0000) 2024-04-23T00:11:26.477 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:26 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[105317]: 2024-04-23T00:11:26.403+0000 7f0df0b07700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-04-23T00:10:57.859194+0000 front 2024-04-23T00:10:57.859226+0000 (oldest deadline 2024-04-23T00:11:24.859323+0000) 2024-04-23T00:11:26.478 INFO:journalctl@ceph.osd.3.smithi134.stdout:Apr 23 00:11:26 smithi134 podman[122796]: 2024-04-23 00:11:26.389186679 +0000 UTC m=+0.655336675 container died 9cf6463fa5662a215e45de5785682e0d5d2569a3dad1c5d0afba920e3a2bb724 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-3-deactivate, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-23T00:11:26.954 DEBUG:teuthology.orchestra.run.smithi134:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d46c346e-00fd-11ef-bc93-c7b262605968@osd.3.service' 2024-04-23T00:11:26.991 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T00:11:26.992 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-23T00:11:26.992 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-23T00:11:26.992 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ceph-d46c346e-00fd-11ef-bc93-c7b262605968@osd.4 2024-04-23T00:11:27.227 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 23 00:11:27 smithi134 systemd[1]: Stopping Ceph osd.4 for d46c346e-00fd-11ef-bc93-c7b262605968... 2024-04-23T00:11:27.727 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 23 00:11:27 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4[100010]: 2024-04-23T00:11:27.374+0000 7fcfc94cb700 -1 osd.4 94 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-04-23T00:10:59.480484+0000 front 2024-04-23T00:10:59.480552+0000 (oldest deadline 2024-04-23T00:11:25.880525+0000) 2024-04-23T00:11:27.727 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 23 00:11:27 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4[100010]: 2024-04-23T00:11:27.375+0000 7fcfcced7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T00:11:27.727 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 23 00:11:27 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4[100010]: 2024-04-23T00:11:27.375+0000 7fcfcced7700 -1 osd.4 94 *** Got signal Terminated *** 2024-04-23T00:11:27.727 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 23 00:11:27 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4[100010]: 2024-04-23T00:11:27.376+0000 7fcfcced7700 -1 osd.4 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T00:11:27.728 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:27 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[105317]: 2024-04-23T00:11:27.410+0000 7f0df0b07700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-04-23T00:10:57.859194+0000 front 2024-04-23T00:10:57.859226+0000 (oldest deadline 2024-04-23T00:11:24.859323+0000) 2024-04-23T00:11:28.727 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 23 00:11:28 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4[100010]: 2024-04-23T00:11:28.338+0000 7fcfc94cb700 -1 osd.4 94 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-04-23T00:10:59.480484+0000 front 2024-04-23T00:10:59.480552+0000 (oldest deadline 2024-04-23T00:11:25.880525+0000) 2024-04-23T00:11:28.727 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:28 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[105317]: 2024-04-23T00:11:28.422+0000 7f0df0b07700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-04-23T00:10:57.859194+0000 front 2024-04-23T00:10:57.859226+0000 (oldest deadline 2024-04-23T00:11:24.859323+0000) 2024-04-23T00:11:29.628 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:29 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[105317]: 2024-04-23T00:11:29.423+0000 7f0df0b07700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-04-23T00:10:57.859194+0000 front 2024-04-23T00:10:57.859226+0000 (oldest deadline 2024-04-23T00:11:24.859323+0000) 2024-04-23T00:11:29.628 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 23 00:11:29 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4[100010]: 2024-04-23T00:11:29.326+0000 7fcfc94cb700 -1 osd.4 94 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-04-23T00:10:59.480484+0000 front 2024-04-23T00:10:59.480552+0000 (oldest deadline 2024-04-23T00:11:25.880525+0000) 2024-04-23T00:11:29.628 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 23 00:11:29 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4[100010]: 2024-04-23T00:11:29.326+0000 7fcfc94cb700 -1 osd.4 94 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-04-23T00:11:06.381407+0000 front 2024-04-23T00:11:06.381502+0000 (oldest deadline 2024-04-23T00:11:29.281251+0000) 2024-04-23T00:11:30.640 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 23 00:11:30 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4[100010]: 2024-04-23T00:11:30.327+0000 7fcfc94cb700 -1 osd.4 94 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-04-23T00:10:59.480484+0000 front 2024-04-23T00:10:59.480552+0000 (oldest deadline 2024-04-23T00:11:25.880525+0000) 2024-04-23T00:11:30.640 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 23 00:11:30 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4[100010]: 2024-04-23T00:11:30.327+0000 7fcfc94cb700 -1 osd.4 94 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-04-23T00:11:06.381407+0000 front 2024-04-23T00:11:06.381502+0000 (oldest deadline 2024-04-23T00:11:29.281251+0000) 2024-04-23T00:11:30.640 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:30 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[105317]: 2024-04-23T00:11:30.378+0000 7f0df0b07700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-04-23T00:10:57.859194+0000 front 2024-04-23T00:10:57.859226+0000 (oldest deadline 2024-04-23T00:11:24.859323+0000) 2024-04-23T00:11:30.640 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:30 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[105317]: 2024-04-23T00:11:30.378+0000 7f0df0b07700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-04-23T00:11:04.859748+0000 front 2024-04-23T00:11:04.859736+0000 (oldest deadline 2024-04-23T00:11:30.159687+0000) 2024-04-23T00:11:31.727 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:31 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[105317]: 2024-04-23T00:11:31.418+0000 7f0df0b07700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-04-23T00:10:57.859194+0000 front 2024-04-23T00:10:57.859226+0000 (oldest deadline 2024-04-23T00:11:24.859323+0000) 2024-04-23T00:11:31.727 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:31 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[105317]: 2024-04-23T00:11:31.418+0000 7f0df0b07700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-04-23T00:11:04.859748+0000 front 2024-04-23T00:11:04.859736+0000 (oldest deadline 2024-04-23T00:11:30.159687+0000) 2024-04-23T00:11:31.727 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 23 00:11:31 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4[100010]: 2024-04-23T00:11:31.358+0000 7fcfc94cb700 -1 osd.4 94 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-04-23T00:10:59.480484+0000 front 2024-04-23T00:10:59.480552+0000 (oldest deadline 2024-04-23T00:11:25.880525+0000) 2024-04-23T00:11:31.727 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 23 00:11:31 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4[100010]: 2024-04-23T00:11:31.358+0000 7fcfc94cb700 -1 osd.4 94 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-04-23T00:11:06.381407+0000 front 2024-04-23T00:11:06.381502+0000 (oldest deadline 2024-04-23T00:11:29.281251+0000) 2024-04-23T00:11:32.727 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 23 00:11:32 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4[100010]: 2024-04-23T00:11:32.363+0000 7fcfc94cb700 -1 osd.4 94 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-04-23T00:10:59.480484+0000 front 2024-04-23T00:10:59.480552+0000 (oldest deadline 2024-04-23T00:11:25.880525+0000) 2024-04-23T00:11:32.727 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 23 00:11:32 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4[100010]: 2024-04-23T00:11:32.363+0000 7fcfc94cb700 -1 osd.4 94 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-04-23T00:11:06.381407+0000 front 2024-04-23T00:11:06.381502+0000 (oldest deadline 2024-04-23T00:11:29.281251+0000) 2024-04-23T00:11:32.727 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:32 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[105317]: 2024-04-23T00:11:32.451+0000 7f0df0b07700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-04-23T00:10:57.859194+0000 front 2024-04-23T00:10:57.859226+0000 (oldest deadline 2024-04-23T00:11:24.859323+0000) 2024-04-23T00:11:32.727 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:32 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[105317]: 2024-04-23T00:11:32.451+0000 7f0df0b07700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-04-23T00:11:04.859748+0000 front 2024-04-23T00:11:04.859736+0000 (oldest deadline 2024-04-23T00:11:30.159687+0000) 2024-04-23T00:11:33.110 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 23 00:11:32 smithi134 podman[122901]: 2024-04-23 00:11:32.850619423 +0000 UTC m=+5.577799746 container died 61da66bd188211774132ac6d83cc81c2bfb0ff2e79b05f481c19da2281f379a1 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-23T00:11:33.386 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 23 00:11:33 smithi134 podman[122901]: 2024-04-23 00:11:33.209168326 +0000 UTC m=+5.936348656 container cleanup 61da66bd188211774132ac6d83cc81c2bfb0ff2e79b05f481c19da2281f379a1 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416) 2024-04-23T00:11:33.386 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 23 00:11:33 smithi134 bash[122901]: ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4 2024-04-23T00:11:33.727 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 23 00:11:33 smithi134 podman[122929]: 2024-04-23 00:11:33.498607529 +0000 UTC m=+0.649717391 container remove 61da66bd188211774132ac6d83cc81c2bfb0ff2e79b05f481c19da2281f379a1 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-23T00:11:33.727 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:33 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[105317]: 2024-04-23T00:11:33.402+0000 7f0df0b07700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-04-23T00:10:57.859194+0000 front 2024-04-23T00:10:57.859226+0000 (oldest deadline 2024-04-23T00:11:24.859323+0000) 2024-04-23T00:11:33.727 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:33 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[105317]: 2024-04-23T00:11:33.402+0000 7f0df0b07700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-04-23T00:11:04.859748+0000 front 2024-04-23T00:11:04.859736+0000 (oldest deadline 2024-04-23T00:11:30.159687+0000) 2024-04-23T00:11:34.152 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 23 00:11:33 smithi134 podman[123023]: 2024-04-23 00:11:33.738938282 +0000 UTC m=+0.018259720 image pull a0434699c1d76af4eea1fc95ff0a22ea73d1e04e609464fbf658d6f39bc7f2fc quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c 2024-04-23T00:11:34.153 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 23 00:11:33 smithi134 podman[123023]: 2024-04-23 00:11:33.853149082 +0000 UTC m=+0.132470516 container create b4336cb0d18482458304746b1324bb202143d54f1a36babeed3ae3b21cb04cfa (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-23T00:11:34.153 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 23 00:11:34 smithi134 podman[123023]: 2024-04-23 00:11:34.05605806 +0000 UTC m=+0.335379505 container init b4336cb0d18482458304746b1324bb202143d54f1a36babeed3ae3b21cb04cfa (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T00:11:34.153 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 23 00:11:34 smithi134 podman[123023]: 2024-04-23 00:11:34.059503273 +0000 UTC m=+0.338824708 container start b4336cb0d18482458304746b1324bb202143d54f1a36babeed3ae3b21cb04cfa (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-22-1618-43be020, io.buildah.version=1.33.5, org.label-schema.build-date=20240416) 2024-04-23T00:11:34.153 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 23 00:11:34 smithi134 podman[123023]: 2024-04-23 00:11:34.169564004 +0000 UTC m=+0.448885451 container attach b4336cb0d18482458304746b1324bb202143d54f1a36babeed3ae3b21cb04cfa (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS) 2024-04-23T00:11:34.477 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 23 00:11:34 smithi134 podman[123023]: 2024-04-23 00:11:34.352856085 +0000 UTC m=+0.632177520 container died b4336cb0d18482458304746b1324bb202143d54f1a36babeed3ae3b21cb04cfa (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-23T00:11:34.477 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:34 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[105317]: 2024-04-23T00:11:34.359+0000 7f0df0b07700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-04-23T00:10:57.859194+0000 front 2024-04-23T00:10:57.859226+0000 (oldest deadline 2024-04-23T00:11:24.859323+0000) 2024-04-23T00:11:34.477 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:34 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[105317]: 2024-04-23T00:11:34.359+0000 7f0df0b07700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-04-23T00:11:04.859748+0000 front 2024-04-23T00:11:04.859736+0000 (oldest deadline 2024-04-23T00:11:30.159687+0000) 2024-04-23T00:11:34.911 DEBUG:teuthology.orchestra.run.smithi134:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d46c346e-00fd-11ef-bc93-c7b262605968@osd.4.service' 2024-04-23T00:11:34.935 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 23 00:11:34 smithi134 podman[123049]: 2024-04-23 00:11:34.901129707 +0000 UTC m=+0.540276761 container remove b4336cb0d18482458304746b1324bb202143d54f1a36babeed3ae3b21cb04cfa (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-23T00:11:34.935 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 23 00:11:34 smithi134 systemd[1]: ceph-d46c346e-00fd-11ef-bc93-c7b262605968@osd.4.service: Deactivated successfully. 2024-04-23T00:11:34.935 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 23 00:11:34 smithi134 systemd[1]: Stopped Ceph osd.4 for d46c346e-00fd-11ef-bc93-c7b262605968. 2024-04-23T00:11:34.935 INFO:journalctl@ceph.osd.4.smithi134.stdout:Apr 23 00:11:34 smithi134 systemd[1]: ceph-d46c346e-00fd-11ef-bc93-c7b262605968@osd.4.service: Consumed 17.478s CPU time. 2024-04-23T00:11:34.951 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T00:11:34.951 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-23T00:11:34.951 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-23T00:11:34.952 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ceph-d46c346e-00fd-11ef-bc93-c7b262605968@osd.5 2024-04-23T00:11:35.227 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:35 smithi134 systemd[1]: Stopping Ceph osd.5 for d46c346e-00fd-11ef-bc93-c7b262605968... 2024-04-23T00:11:35.727 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:35 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[105317]: 2024-04-23T00:11:35.287+0000 7f0df4513700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T00:11:35.727 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:35 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[105317]: 2024-04-23T00:11:35.287+0000 7f0df4513700 -1 osd.5 94 *** Got signal Terminated *** 2024-04-23T00:11:35.727 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:35 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[105317]: 2024-04-23T00:11:35.287+0000 7f0df4513700 -1 osd.5 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T00:11:35.727 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:35 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[105317]: 2024-04-23T00:11:35.400+0000 7f0df0b07700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-04-23T00:10:57.859194+0000 front 2024-04-23T00:10:57.859226+0000 (oldest deadline 2024-04-23T00:11:24.859323+0000) 2024-04-23T00:11:35.727 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:35 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[105317]: 2024-04-23T00:11:35.400+0000 7f0df0b07700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-04-23T00:11:04.859748+0000 front 2024-04-23T00:11:04.859736+0000 (oldest deadline 2024-04-23T00:11:30.159687+0000) 2024-04-23T00:11:36.727 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:36 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[105317]: 2024-04-23T00:11:36.447+0000 7f0df0b07700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-04-23T00:10:57.859194+0000 front 2024-04-23T00:10:57.859226+0000 (oldest deadline 2024-04-23T00:11:24.859323+0000) 2024-04-23T00:11:36.727 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:36 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[105317]: 2024-04-23T00:11:36.447+0000 7f0df0b07700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-04-23T00:11:04.859748+0000 front 2024-04-23T00:11:04.859736+0000 (oldest deadline 2024-04-23T00:11:30.159687+0000) 2024-04-23T00:11:37.727 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:37 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[105317]: 2024-04-23T00:11:37.453+0000 7f0df0b07700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-04-23T00:10:57.859194+0000 front 2024-04-23T00:10:57.859226+0000 (oldest deadline 2024-04-23T00:11:24.859323+0000) 2024-04-23T00:11:37.727 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:37 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[105317]: 2024-04-23T00:11:37.453+0000 7f0df0b07700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-04-23T00:11:04.859748+0000 front 2024-04-23T00:11:04.859736+0000 (oldest deadline 2024-04-23T00:11:30.159687+0000) 2024-04-23T00:11:37.727 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:37 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[105317]: 2024-04-23T00:11:37.453+0000 7f0df0b07700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.67:6822 osd.2 since back 2024-04-23T00:11:13.560965+0000 front 2024-04-23T00:11:13.561008+0000 (oldest deadline 2024-04-23T00:11:37.060741+0000) 2024-04-23T00:11:38.727 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:38 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[105317]: 2024-04-23T00:11:38.437+0000 7f0df0b07700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-04-23T00:10:57.859194+0000 front 2024-04-23T00:10:57.859226+0000 (oldest deadline 2024-04-23T00:11:24.859323+0000) 2024-04-23T00:11:38.727 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:38 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[105317]: 2024-04-23T00:11:38.437+0000 7f0df0b07700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-04-23T00:11:04.859748+0000 front 2024-04-23T00:11:04.859736+0000 (oldest deadline 2024-04-23T00:11:30.159687+0000) 2024-04-23T00:11:38.727 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:38 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[105317]: 2024-04-23T00:11:38.437+0000 7f0df0b07700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.67:6822 osd.2 since back 2024-04-23T00:11:13.560965+0000 front 2024-04-23T00:11:13.561008+0000 (oldest deadline 2024-04-23T00:11:37.060741+0000) 2024-04-23T00:11:39.727 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:39 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[105317]: 2024-04-23T00:11:39.422+0000 7f0df0b07700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-04-23T00:10:57.859194+0000 front 2024-04-23T00:10:57.859226+0000 (oldest deadline 2024-04-23T00:11:24.859323+0000) 2024-04-23T00:11:39.727 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:39 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[105317]: 2024-04-23T00:11:39.422+0000 7f0df0b07700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.67:6814 osd.1 since back 2024-04-23T00:11:04.859748+0000 front 2024-04-23T00:11:04.859736+0000 (oldest deadline 2024-04-23T00:11:30.159687+0000) 2024-04-23T00:11:39.727 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:39 smithi134 ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5[105317]: 2024-04-23T00:11:39.422+0000 7f0df0b07700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.67:6822 osd.2 since back 2024-04-23T00:11:13.560965+0000 front 2024-04-23T00:11:13.561008+0000 (oldest deadline 2024-04-23T00:11:37.060741+0000) 2024-04-23T00:11:40.921 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:40 smithi134 podman[123128]: 2024-04-23 00:11:40.663079617 +0000 UTC m=+5.468604739 container died 93a955888c845793c0aa293386bde864bf7d19270c244d21c2a477733e0ae3d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416) 2024-04-23T00:11:41.227 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:41 smithi134 podman[123128]: 2024-04-23 00:11:41.046313022 +0000 UTC m=+5.851838151 container cleanup 93a955888c845793c0aa293386bde864bf7d19270c244d21c2a477733e0ae3d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T00:11:41.227 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:41 smithi134 bash[123128]: ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5 2024-04-23T00:11:41.635 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:41 smithi134 podman[123145]: 2024-04-23 00:11:41.322820138 +0000 UTC m=+0.661285430 container remove 93a955888c845793c0aa293386bde864bf7d19270c244d21c2a477733e0ae3d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS) 2024-04-23T00:11:41.917 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:41 smithi134 podman[123239]: 2024-04-23 00:11:41.555874071 +0000 UTC m=+0.018166932 image pull a0434699c1d76af4eea1fc95ff0a22ea73d1e04e609464fbf658d6f39bc7f2fc quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c 2024-04-23T00:11:41.917 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:41 smithi134 podman[123239]: 2024-04-23 00:11:41.65688211 +0000 UTC m=+0.119174958 container create a8d37640f279b36b067388639787dd92d15ff32a42417831a6d71822a4fac30d (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416) 2024-04-23T00:11:41.917 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:41 smithi134 podman[123239]: 2024-04-23 00:11:41.85102889 +0000 UTC m=+0.313321737 container init a8d37640f279b36b067388639787dd92d15ff32a42417831a6d71822a4fac30d (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-23T00:11:41.917 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:41 smithi134 podman[123239]: 2024-04-23 00:11:41.85410489 +0000 UTC m=+0.316397746 container start a8d37640f279b36b067388639787dd92d15ff32a42417831a6d71822a4fac30d (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-23T00:11:41.917 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:41 smithi134 podman[123239]: 2024-04-23 00:11:41.93420834 +0000 UTC m=+0.396501199 container attach a8d37640f279b36b067388639787dd92d15ff32a42417831a6d71822a4fac30d (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T00:11:42.227 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:42 smithi134 podman[123239]: 2024-04-23 00:11:42.144994698 +0000 UTC m=+0.607287546 container died a8d37640f279b36b067388639787dd92d15ff32a42417831a6d71822a4fac30d (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T00:11:42.695 INFO:journalctl@ceph.osd.5.smithi134.stdout:Apr 23 00:11:42 smithi134 podman[123265]: 2024-04-23 00:11:42.696432539 +0000 UTC m=+0.541399810 container remove a8d37640f279b36b067388639787dd92d15ff32a42417831a6d71822a4fac30d (image=quay.ceph.io/ceph-ci/ceph@sha256:b8bc99cfccdf4e584cbe106d55b5c5dc774312148ba54d6ca5b660445cf4506c, name=ceph-d46c346e-00fd-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-adk-testing-2024-04-22-1618-43be020, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-23T00:11:42.700 DEBUG:teuthology.orchestra.run.smithi134:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d46c346e-00fd-11ef-bc93-c7b262605968@osd.5.service' 2024-04-23T00:11:42.737 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T00:11:42.737 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-23T00:11:42.737 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d46c346e-00fd-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-23T00:11:42.900 INFO:teuthology.orchestra.run.smithi067.stdout:Deleting cluster with fsid: d46c346e-00fd-11ef-bc93-c7b262605968 2024-04-23T00:11:47.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi067.stderr:ceph-fuse[75366]: fuse finished with error 0 and tester_r 0 2024-04-23T00:12:10.796 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d46c346e-00fd-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-23T00:12:10.949 INFO:teuthology.orchestra.run.smithi134.stdout:Deleting cluster with fsid: d46c346e-00fd-11ef-bc93-c7b262605968 2024-04-23T00:12:26.106 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T00:12:26.139 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T00:12:26.165 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-23T00:12:26.167 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/adking-2024-04-22_22:45:33-orch:cephadm-wip-adk-testing-2024-04-22-1618-distro-default-smithi/7669190/remote/smithi067/crash 2024-04-23T00:12:26.168 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash -- . 2024-04-23T00:12:26.207 INFO:teuthology.orchestra.run.smithi067.stderr:tar: /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-23T00:12:26.207 INFO:teuthology.orchestra.run.smithi067.stderr:tar: Error is not recoverable: exiting now 2024-04-23T00:12:26.209 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/adking-2024-04-22_22:45:33-orch:cephadm-wip-adk-testing-2024-04-22-1618-distro-default-smithi/7669190/remote/smithi134/crash 2024-04-23T00:12:26.210 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash -- . 2024-04-23T00:12:26.232 INFO:teuthology.orchestra.run.smithi134.stderr:tar: /var/lib/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-23T00:12:26.232 INFO:teuthology.orchestra.run.smithi134.stderr:tar: Error is not recoverable: exiting now 2024-04-23T00:12:26.233 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-23T00:12:26.234 DEBUG:teuthology.orchestra.run.smithi067:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-04-23T00:12:26.285 INFO:teuthology.orchestra.run.smithi067.stdout:2024-04-22T23:30:00.000197+0000 mon.smithi067 (mon.0) 352 : cluster [WRN] osd.1 (root=default,host=smithi067) is down 2024-04-23T00:12:26.285 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-04-23T00:12:26.286 INFO:tasks.cephadm:Compressing logs... 2024-04-23T00:12:26.286 DEBUG:teuthology.orchestra.run.smithi067:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-23T00:12:26.328 DEBUG:teuthology.orchestra.run.smithi134:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-23T00:12:26.351 INFO:teuthology.orchestra.run.smithi134.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/ceph-client.1.log 2024-04-23T00:12:26.351 INFO:teuthology.orchestra.run.smithi134.stderr:: No such file or directory 2024-04-23T00:12:26.351 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-mgr.smithi134.ccxkcj.log 2024-04-23T00:12:26.351 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph.log 2024-04-23T00:12:26.351 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.1.log: /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-mgr.smithi134.ccxkcj.log: gzip -5 --verbose -- /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-23T00:12:26.356 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-client.ceph-exporter.smithi134.log 2024-04-23T00:12:26.356 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-23T00:12:26.356 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-client.ceph-exporter.smithi134.log: gzip -5 --verbose -- /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-23T00:12:26.356 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-23T00:12:26.356 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-mon.smithi134.log 2024-04-23T00:12:26.357 INFO:teuthology.orchestra.run.smithi067.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-23T00:12:26.357 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.log 2024-04-23T00:12:26.357 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-mon.smithi067.log 2024-04-23T00:12:26.357 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph.log 2024-04-23T00:12:26.357 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.log: /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-mon.smithi067.log: gzip -5 --verbose -- /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-23T00:12:26.357 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-23T00:12:26.357 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-client.ceph-exporter.smithi067.log 2024-04-23T00:12:26.357 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-osd.2.log: /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-23T00:12:26.357 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-client.ceph-exporter.smithi067.log: gzip -5 --verbose -- /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-23T00:12:26.357 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi067.xyazmh.log 2024-04-23T00:12:26.363 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-osd.4.log: 89.5% -- replaced with /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-23T00:12:26.363 INFO:teuthology.orchestra.run.smithi134.stderr: 94.6% -- replaced with /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-client.ceph-exporter.smithi134.log.gz 2024-04-23T00:12:26.363 INFO:teuthology.orchestra.run.smithi134.stderr: 91.5% -- replaced with /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-23T00:12:26.363 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-23T00:12:26.363 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-mon.smithi134.log: gzip -5 --verbose -- /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi134.vuxbsj.log 2024-04-23T00:12:26.364 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-23T00:12:26.364 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi134.vuxbsj.log: 85.8% -- replaced with /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-23T00:12:26.364 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi134.owipad.log 2024-04-23T00:12:26.365 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-osd.0.log: 89.5% -- replaced with /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-23T00:12:26.365 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi067.aqlxpv.log 2024-04-23T00:12:26.365 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi067.xyazmh.log: 94.5% -- replaced with /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-client.ceph-exporter.smithi067.log.gz 2024-04-23T00:12:26.365 INFO:teuthology.orchestra.run.smithi067.stderr: 91.4% -- replaced with /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-23T00:12:26.374 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-23T00:12:26.374 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi067.aqlxpv.log: gzip -5 --verbose -- /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-mgr.smithi067.whdcsr.log 2024-04-23T00:12:26.375 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph.cephadm.log: 85.9% -- replaced with /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-23T00:12:26.375 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-23T00:12:26.381 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-volume.log: 91.0% -- replaced with /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-23T00:12:26.382 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-23T00:12:26.388 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-mgr.smithi067.whdcsr.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-23T00:12:26.408 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi134.owipad.log: /var/log/ceph/cephadm.log: 92.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-23T00:12:26.408 INFO:teuthology.orchestra.run.smithi134.stderr: 90.0% -- replaced with /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-mgr.smithi134.ccxkcj.log.gz 2024-04-23T00:12:26.426 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/cephadm.log: 91.0% -- replaced with /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-23T00:12:26.475 INFO:teuthology.orchestra.run.smithi067.stderr: 90.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-23T00:12:27.096 INFO:teuthology.orchestra.run.smithi067.stderr: 89.5% -- replaced with /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-mgr.smithi067.whdcsr.log.gz 2024-04-23T00:12:27.226 INFO:teuthology.orchestra.run.smithi134.stderr: 92.6% -- replaced with /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-mon.smithi134.log.gz 2024-04-23T00:12:28.834 INFO:teuthology.orchestra.run.smithi067.stderr: 89.9% -- replaced with /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-mon.smithi067.log.gz 2024-04-23T00:12:30.712 INFO:teuthology.orchestra.run.smithi134.stderr: 93.7% -- replaced with /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-23T00:12:31.398 INFO:teuthology.orchestra.run.smithi067.stderr: 93.6% -- replaced with /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-23T00:12:33.580 INFO:teuthology.orchestra.run.smithi067.stderr: 93.7% -- replaced with /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-23T00:12:33.824 INFO:teuthology.orchestra.run.smithi067.stderr: 93.6% -- replaced with /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-23T00:12:33.859 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-23T00:12:34.624 INFO:teuthology.orchestra.run.smithi134.stderr: 95.1% -- replaced with /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi134.vuxbsj.log.gz 2024-04-23T00:12:34.875 INFO:teuthology.orchestra.run.smithi134.stderr: 93.7% -- replaced with /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-23T00:12:36.194 INFO:teuthology.orchestra.run.smithi067.stderr: 95.1% -- replaced with /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi067.aqlxpv.log.gz 2024-04-23T00:12:42.099 INFO:teuthology.orchestra.run.smithi134.stderr:gzip: /var/log/ceph/ceph-client.1.log: file size changed while zipping 2024-04-23T00:12:42.099 INFO:teuthology.orchestra.run.smithi134.stderr: 93.2% -- replaced with /var/log/ceph/ceph-client.1.log.gz 2024-04-23T00:12:42.745 INFO:teuthology.orchestra.run.smithi067.stderr: 93.2% -- replaced with /var/log/ceph/ceph-client.0.log.gz 2024-04-23T00:13:10.607 INFO:teuthology.orchestra.run.smithi134.stderr: 93.4% -- replaced with /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi134.owipad.log.gz 2024-04-23T00:13:10.609 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-04-23T00:13:10.609 INFO:teuthology.orchestra.run.smithi134.stderr:real 0m44.268s 2024-04-23T00:13:10.609 INFO:teuthology.orchestra.run.smithi134.stderr:user 1m26.757s 2024-04-23T00:13:10.609 INFO:teuthology.orchestra.run.smithi134.stderr:sys 0m2.629s 2024-04-23T00:13:16.764 INFO:teuthology.orchestra.run.smithi067.stderr: 93.3% -- replaced with /var/log/ceph/d46c346e-00fd-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi067.xyazmh.log.gz 2024-04-23T00:13:16.766 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-04-23T00:13:16.766 INFO:teuthology.orchestra.run.smithi067.stderr:real 0m50.426s 2024-04-23T00:13:16.767 INFO:teuthology.orchestra.run.smithi067.stderr:user 1m36.490s 2024-04-23T00:13:16.767 INFO:teuthology.orchestra.run.smithi067.stderr:sys 0m2.890s 2024-04-23T00:13:16.767 INFO:tasks.cephadm:Archiving logs... 2024-04-23T00:13:16.767 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/log/ceph to /home/teuthworker/archive/adking-2024-04-22_22:45:33-orch:cephadm-wip-adk-testing-2024-04-22-1618-distro-default-smithi/7669190/remote/smithi067/log 2024-04-23T00:13:16.768 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-23T00:13:29.157 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/log/ceph to /home/teuthworker/archive/adking-2024-04-22_22:45:33-orch:cephadm-wip-adk-testing-2024-04-22-1618-distro-default-smithi/7669190/remote/smithi134/log 2024-04-23T00:13:29.158 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-23T00:13:41.246 INFO:tasks.cephadm:Removing cluster... 2024-04-23T00:13:41.246 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d46c346e-00fd-11ef-bc93-c7b262605968 --force 2024-04-23T00:13:41.406 INFO:teuthology.orchestra.run.smithi067.stdout:Deleting cluster with fsid: d46c346e-00fd-11ef-bc93-c7b262605968 2024-04-23T00:13:41.769 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d46c346e-00fd-11ef-bc93-c7b262605968 --force 2024-04-23T00:13:41.915 INFO:teuthology.orchestra.run.smithi134.stdout:Deleting cluster with fsid: d46c346e-00fd-11ef-bc93-c7b262605968 2024-04-23T00:13:42.263 INFO:tasks.cephadm:Removing cephadm ... 2024-04-23T00:13:42.263 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-23T00:13:42.280 DEBUG:teuthology.orchestra.run.smithi134:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-23T00:13:42.294 INFO:tasks.cephadm:Teardown complete 2024-04-23T00:13:42.295 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-23T00:13:42.307 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_43be020184947e53516056c9931e1ac5bdbbb1a5/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph-c_43be020184947e53516056c9931e1ac5bdbbb1a5/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-23T00:13:42.308 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-23T00:13:42.308 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-23T00:13:42.322 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-23T00:13:42.428 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-23T00:13:42.439 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-23T00:13:42.439 DEBUG:teuthology.orchestra.run.smithi067:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T00:13:42.442 DEBUG:teuthology.orchestra.run.smithi134:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T00:13:42.455 INFO:teuthology.orchestra.run.smithi067.stderr:bash: line 1: ntpq: command not found 2024-04-23T00:13:42.455 INFO:teuthology.orchestra.run.smithi134.stderr:bash: line 1: ntpq: command not found 2024-04-23T00:13:42.458 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T00:13:42.458 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2024-04-23T00:13:42.458 INFO:teuthology.orchestra.run.smithi067.stdout:^- hv01.front.sepia.ceph.com 2 7 377 116 +623us[ +619us] +/- 581ms 2024-04-23T00:13:42.459 INFO:teuthology.orchestra.run.smithi067.stdout:^* hv02.front.sepia.ceph.com 3 8 377 49 -185us[ -189us] +/- 65ms 2024-04-23T00:13:42.459 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 46 +213us[ +213us] +/- 88ms 2024-04-23T00:13:42.459 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T00:13:42.460 INFO:teuthology.orchestra.run.smithi134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T00:13:42.460 INFO:teuthology.orchestra.run.smithi134.stdout:=============================================================================== 2024-04-23T00:13:42.460 INFO:teuthology.orchestra.run.smithi134.stdout:^- hv01.front.sepia.ceph.com 2 6 377 51 +632us[ +632us] +/- 582ms 2024-04-23T00:13:42.460 INFO:teuthology.orchestra.run.smithi134.stdout:^* hv02.front.sepia.ceph.com 3 6 377 51 -175us[ -179us] +/- 65ms 2024-04-23T00:13:42.460 INFO:teuthology.orchestra.run.smithi134.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 43 +248us[ +248us] +/- 88ms 2024-04-23T00:13:42.461 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T00:13:42.461 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-23T00:13:42.479 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-23T00:13:42.480 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-23T00:13:42.549 DEBUG:teuthology.orchestra.run.smithi067:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-23T00:13:42.634 DEBUG:teuthology.orchestra.run.smithi134:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-23T00:13:42.706 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T00:13:42.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T00:13:42.735 DEBUG:teuthology.orchestra.run.smithi134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T00:13:42.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T00:13:42.763 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-23T00:13:42.775 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-23T00:13:42.806 INFO:teuthology.task.internal:Duration was 4015.466715 seconds 2024-04-23T00:13:42.807 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-23T00:13:42.843 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-23T00:13:42.843 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-23T00:13:42.846 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-23T00:13:42.881 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T00:13:42.881 INFO:teuthology.orchestra.run.smithi134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T00:13:43.075 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-23T00:13:43.075 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi067.front.sepia.ceph.com 2024-04-23T00:13:43.075 DEBUG:teuthology.orchestra.run.smithi067:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-23T00:13:43.139 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi134.front.sepia.ceph.com 2024-04-23T00:13:43.139 DEBUG:teuthology.orchestra.run.smithi134:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-23T00:13:43.159 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-23T00:13:43.159 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-23T00:13:43.182 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-23T00:13:43.223 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-23T00:13:43.223 DEBUG:teuthology.orchestra.run.smithi067:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-23T00:13:43.248 DEBUG:teuthology.orchestra.run.smithi134:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-23T00:13:44.463 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-23T00:13:44.475 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-23T00:13:44.475 DEBUG:teuthology.orchestra.run.smithi067:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-23T00:13:44.500 DEBUG:teuthology.orchestra.run.smithi134:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-23T00:13:44.528 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-23T00:13:44.539 DEBUG:teuthology.orchestra.run.smithi067:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-23T00:13:44.542 DEBUG:teuthology.orchestra.run.smithi134:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-23T00:13:44.562 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = core 2024-04-23T00:13:44.593 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = core 2024-04-23T00:13:44.904 DEBUG:teuthology.orchestra.run.smithi067:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-23T00:13:44.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T00:13:44.920 DEBUG:teuthology.orchestra.run.smithi134:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-23T00:13:44.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T00:13:44.962 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-23T00:13:44.974 INFO:teuthology.task.internal:Transferring archived files... 2024-04-23T00:13:44.975 DEBUG:teuthology.misc:Transferring archived files from smithi067:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/adking-2024-04-22_22:45:33-orch:cephadm-wip-adk-testing-2024-04-22-1618-distro-default-smithi/7669190/remote/smithi067 2024-04-23T00:13:44.975 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-23T00:13:45.037 DEBUG:teuthology.misc:Transferring archived files from smithi134:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/adking-2024-04-22_22:45:33-orch:cephadm-wip-adk-testing-2024-04-22-1618-distro-default-smithi/7669190/remote/smithi134 2024-04-23T00:13:45.037 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-23T00:13:45.079 INFO:teuthology.task.internal:Removing archive directory... 2024-04-23T00:13:45.079 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-23T00:13:45.082 DEBUG:teuthology.orchestra.run.smithi134:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-23T00:13:45.130 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-23T00:13:45.144 INFO:teuthology.task.internal:Not uploading archives. 2024-04-23T00:13:45.144 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-23T00:13:45.219 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-23T00:13:45.219 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-23T00:13:45.222 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-23T00:13:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: 262213 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 23 00:13 /home/ubuntu/cephtest 2024-04-23T00:13:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: 548396 4 d--------- 2 ubuntu ubuntu 4096 Apr 22 23:20 /home/ubuntu/cephtest/mnt.0 2024-04-23T00:13:45.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T00:13:45.235 INFO:teuthology.orchestra.run.smithi067.stderr:find: ‘/home/ubuntu/cephtest/mnt.0’: Permission denied 2024-04-23T00:13:45.235 INFO:teuthology.orchestra.run.smithi067.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-04-23T00:13:45.236 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_43be020184947e53516056c9931e1ac5bdbbb1a5/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/git.ceph.com_ceph-c_43be020184947e53516056c9931e1ac5bdbbb1a5/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi067 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-04-23T00:13:45.236 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-23T00:13:45.249 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-23T00:13:45.296 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-23T00:13:45.296 INFO:teuthology.nuke:Checking targets against current locks 2024-04-23T00:13:45.331 DEBUG:teuthology.nuke:shortname: smithi067 2024-04-23T00:13:45.331 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-23T00:13:45.373 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-04-22_22:45:33-orch:cephadm-wip-adk-testing-2024-04-22-1618-distro-default-smithi/7669190', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 23:01:16.930821', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ5hEEgSAMmspQN5NnJQsa+hsJAfHYTu3I5IW2nL2wQlYZzcUSeO0DE0hSGXpgY+MuX3h1ZRYwV4X9rxoiwDYN4='} 2024-04-23T00:13:45.383 DEBUG:teuthology.nuke:shortname: smithi134 2024-04-23T00:13:45.383 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-23T00:13:45.412 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi134.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-04-22_22:45:33-orch:cephadm-wip-adk-testing-2024-04-22-1618-distro-default-smithi/7669190', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 23:01:16.932839', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCEGBOx3MuAKppZlNWhmQElTN2uNaQUyAfgTd0/Cox0zmELbuVIEtalkaxr05afBn9GowE0EaPYgpdlHG/jSdAc='} 2024-04-23T00:13:45.461 INFO:teuthology.orchestra.console.smithi067:Power off 2024-04-23T00:13:45.461 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-23T00:13:45.510 INFO:teuthology.orchestra.console.smithi134:Power off 2024-04-23T00:13:45.510 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-23T00:13:45.521 DEBUG:teuthology.orchestra.console.smithi067:power off output: Chassis Power Control: Down/Off 2024-04-23T00:13:45.522 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T00:13:45.561 DEBUG:teuthology.orchestra.console.smithi134:power off output: Chassis Power Control: Down/Off 2024-04-23T00:13:45.561 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T00:13:45.573 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is on 2024-04-23T00:13:45.610 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2024-04-23T00:13:49.574 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T00:13:49.611 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T00:13:49.684 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is on 2024-04-23T00:13:49.722 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2024-04-23T00:13:53.690 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T00:13:53.723 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T00:13:53.800 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is on 2024-04-23T00:13:53.832 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2024-04-23T00:13:57.805 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T00:13:57.833 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T00:13:57.916 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is off 2024-04-23T00:13:57.942 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is off 2024-04-23T00:13:58.018 INFO:teuthology.orchestra.console.smithi067:Power off completed 2024-04-23T00:13:58.043 INFO:teuthology.orchestra.console.smithi134:Power off completed 2024-04-23T00:13:58.148 INFO:teuthology.run:Summary data: description: orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 4015.46671462059 failure_reason: reached maximum tries (51) after waiting for 300 seconds flavor: default owner: scheduled_adking@teuthology status: fail success: false 2024-04-23T00:13:58.148 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-23T00:13:58.242 INFO:teuthology.run:FAIL